# A Hybrid Approach for Approximating the Ideal Observer for Joint Signal
Detection and Estimation Tasks by Use of Supervised Learning and Markov-Chain
Monte Carlo Methods

Kaiyan Li, , Weimin Zhou, Hua Li, and Mark A. Anastasio  Kaiyan Li is with the
Department of Bioengineering, University of Illinois at Urbana-Champaign,
Urbana, IL, 61801 USA (Email: kaiyanl2@illinois.edu). Weimin Zhou was with the
Department of Electrical and Systems Engineering, Washington University in St.
Louis, St. Louis, MO, 63130 USA. He is now with the Department of
Psychological and Brain Sciences, University of California, Santa Barbara,
Santa Barbara, CA 93106 USA (Email: weiminzhou@ucsb.edu). Hua Li is with the
Department of Radiation Oncology, Washington University School of Medicine,
Saint Louis, MO, She was with the Department of Bioengineering, University of
Illinois at Urbana-Champaign, Urbana, IL 61801 USA<EMAIL_ADDRESS>&
huali19@illinois.edu). Mark A. Anastasio is with the Department of
Bioengineering, University of Illinois at Urbana-Champaign, Urbana, IL, 61801
USA (Email: maa@illinois.edu). This work was supported in part by NIH awards
R01EB020604, R01EB023045, R01NS102213, R01CA233873, and R21CA223799.
(Corresponding authors: Mark A. Anastasio & Hua Li).

###### Abstract

The ideal observer (IO) sets an upper performance limit among all observers
and has been advocated for assessing and optimizing imaging systems. For
general joint detection and estimation (detection-estimation) tasks,
estimation ROC (EROC) analysis has been established for evaluating the
performance of observers. However, in general, it is difficult to accurately
approximate the IO that maximizes the area under the EROC curve. In this
study, a hybrid method that employs machine learning is proposed to accomplish
this. Specifically, a hybrid approach is developed that combines a multi-task
convolutional neural network and a Markov-Chain Monte Carlo (MCMC) method in
order to approximate the IO for detection-estimation tasks. Unlike traditional
MCMC methods, the hybrid method is not limited to use of specific utility
functions. In addition, a purely supervised learning-based sub-ideal observer
is proposed. Computer-simulation studies are conducted to validate the
proposed method, which include signal-known-statistically/background-known-
exactly and signal-known-statistically/background-known-statistically tasks.
The EROC curves produced by the proposed method are compared to those produced
by the MCMC approach or analytical computation when feasible. The proposed
method provides a new approach for approximating the IO and may advance the
application of EROC analysis for optimizing imaging systems.

###### Index Terms:

Numerical observers, Ideal Observer, joint signal detection and estimation
tasks, estimation receiver operating characteristic curve, task-based image
quality assessment, deep learning.

##  I Introduction

Objective, or task-based, measures of image quality (IQ) are advocated for use
in the assessment and optimization of medical imaging systems [1, 2, 3, 4, 5].
Unlike traditional physical measures of IQ, task-based measures of IQ quantify
the ability of an observer to perform a specific task such as detection or
estimation of a signal. To compute such measures of IQ when developing and
refining new imaging technologies, numerical observers (NOs) have been widely
employed [4, 2]. This enables the exploration of large parameter spaces when
optimizing system performance. Such NOs can be designed to estimate an upper
bound on the possible performance of any observer for a given task and
collection of image data. Such models are referred to as _ideal observers_
(IOs). Alternatively, anthropomorphic NOs can be designed to mimic the
performance of a human observer, which is generally sub-optimal. The focus of
this work will be on the computation of IOs.

The Bayesian IO is a numerical observer (NO) that achieves the performance of
the optimal decision maker acting on given measured data [3, 6, 7]. The IO
performance is a task-based image quality measure that depends on the data and
the task (e.g., lesion detection) and not the capabilities of a human
observer, the quality of feature extraction, or a particular classification
scheme. As such, it is a metric of fundamental importance in the objective
assessment of imaging hardware and data-acquisition designs [1]. Knowledge of
the IO performance is also important because it can reveal when task-related
information is readily extracted by a human observer or another sub-optimal
NO. This can permit the identification of opportunities for improved image
processing or other methodological changes that lead to improved task-
performance.

Much of the literature on IO approximation has focused on binary signal
detection tasks. The IO test statistic in this case is generally a non-linear
function of the image data and, except in some special cases, cannot be
determined analytically. Because of this, sampling-based methods that employ
Markov-chain Monte Carlo (MCMC) techniques and supervised learning have been
developed to approximate the IO test statistic for medical image applications
[8, 9, 10].

Detection-estimation tasks that involve the detection of a signal and the
subsequent estimation of a set of parameters for the signal present decisions
are relevant to many medical imaging applications [6, 11, 12, 13, 14, 15]. The
estimation receiver operating characteristic (EROC) curve [6] can be employed
to assess the performance of an observer on detection-estimation tasks and the
area under the EROC curve (AEROC) can be utilized as a figure-of-merit (FOM).
Similar to the case of binary detection tasks, the IO decision strategy for
detection-estimation tasks is analytically intractable except in special cases
[16]. However, unlike the of case binary detection tasks for which MCMC
methods or supervised learning methods can be employed to establish NOs, [17,
9], there is a lack of available NOs for approximating the IO for detection-
estimation tasks.

To address this need, in this study a hybrid method is developed for
approximating the IO for a wide class of detection-estimation tasks. The
proposed method combines deep learning (DL) and a Markov-Chain Monte Carlo
(MCMC) method in order to implement the known IO decision strategy [6]. Unlike
traditional MCMC methods for approximating the IO, the hybrid method is not
limited to use of specific utility functions. In addition, a purely supervised
learning-based sub-ideal observer is designed, which avoids the need to
involve the MCMC method. Computer-simulation studies are conducted to validate
the performance of the proposed methods, which address signal-known-
statistically/background-known-exactly (SKS/BKE) and signal-known-
statistically/background-known-statistically (SKS/BKS) tasks. The proposed
methods provide a new capability for approximating the IO for detection-
estimation tasks and may advance the application of EROC analysis for
optimizing imaging systems.

The remainder of the paper is organized as follows. Section II provides
salient information regarding signal detection-estimation theory. The proposed
hybrid IO approximation method and the supervised learning-based sub-ideal
observer are described in Section III. The numerical studies and the results
are provided in Sections IV and V, respectively. Finally, the article
concludes with a discussion in Section VI.

##  II Background

A linear digital imaging system can be described as a continuous-to-discrete
(C-D) mapping process [3]:

$\mathbf{g}=\mathcal{H}{f(\mathbf{r})+\mathbf{n}},$ (1)

where $\mathbf{g}\in\mathbb{R}^{N\times 1}$ is the measured image vector,
$f(\mathbf{r})$ denotes the object function that is dependent on the
coordinate $\mathbf{r}\in\mathbb{R}^{k\times 1}$, $k\geq 2$, $\mathcal{H}$
denotes a linear imaging operator that maps $\mathbb{L}_{2}(\mathbb{R}^{k})$
to $\mathbb{R}^{N\times 1}$, and $\mathbf{n}\in\mathbb{R}^{N\times 1}$ denotes
the measurement noise. The imaging process described in Eqn. (1) can be
expressed as [3]:

$[\mathbf{g}]_{m}\equiv\int_{V}d\mathbf{r}\,f(\mathbf{r})\,h_{m}(\mathbf{r})+[\mathbf{n}]_{m},$
(2)

where $[\mathbf{g}]_{m}$ and $[\mathbf{n}]_{m}$ denote the $m^{th}$ component
of $\mathbf{g}$ and $\mathbf{n}$, respectively, $V$ denotes the support of
$f(\mathbf{r})$, and $h_{m}(\mathbf{r})$ is the point response function (PRF)
of the imaging system [3].

###  II-A General detection-estimation tasks

In a binary detection-estimation task, each image is either signal-absent or
contains a signal that is specified by a parameter vector $\bm{\theta}$. The
imaging processes under these hypotheses can be expressed as:

$H_{0}:\mathbf{g}=\mathcal{H}\mathbf{f_{b}}+\mathbf{n}=\mathbf{b+n},$ (3a)
$H_{1}:\mathbf{g}=\mathcal{H}\mathbf{(f_{b}+f_{s(\bm{\theta})})}+\mathbf{n}=\mathbf{b+s(\bm{\theta})+n},$
(3b)

where $\mathbf{f_{s(\bm{\theta})}}$ and $\mathbf{f_{b}}$ denote the signal and
background, respectively, and
$\mathbf{s}\equiv\mathcal{H}\mathbf{f_{s(\bm{\theta})}}$ and
$\mathbf{b}\equiv\mathcal{H}\mathbf{f_{b}}$ denote the signal and background
images. To perform this task, a deterministic observer first computes a test
statistic $T(\mathbf{g})$ that maps the measured image $\mathbf{g}$ to a real-
valued scalar variable. The value of $T(\mathbf{g})$ is then compared to a
predetermined threshold $\tau$ to determine if $\mathbf{g}$ satisfies $H_{0}$
or $H_{1}$. Subsequently, an estimate $\bm{\hat{\theta}}(\mathbf{g})$ is given
if the observer decides that the signal is present [6, 16].

An EROC curve is generated by plotting the expected utility of
$\bm{\hat{\theta}}(\mathbf{g})$ for the true positive (TP) decisions versus
the false-positive (FP) fraction as $\tau$ is varied. The expected utility of
$\bm{\hat{\theta}}(\mathbf{g})$ for a true positive (TP) decision (referred to
as the expected utility) is defined as [6]

$U_{TP}(\tau)=E[u(\hat{\bm{\theta}}(\mathbf{g}),\bm{\theta})step[T(\mathbf{g})-\tau]|H_{1}],$
(4)

where $U_{TP}(\tau)$ is the expected utility,
$u(\hat{\bm{\theta}}(\mathbf{g}),\bm{\theta})$ is a utility function for the
parameter estimate, $E[\cdot]$ is the mathematical expectation operator, and
$step[\cdot]$ is the Heaviside step function. In general, the utility function
should be designed to return a high value when $\hat{\bm{\theta}}(\mathbf{g})$
is close to $\bm{\theta}$ and a low value otherwise. Each point on the EROC
curve gives the expected utility of the estimated parameter for the TP
decisions at a given false-positive fraction. Because the utility function may
be negative, unlike the traditional ROC curve, the EROC curve is not always an
increasing as a function of FPF.

###  II-B Ideal observer for detection-estimation tasks

The IO test statistic $T_{I}(\mathbf{g})$ and estimator
$\hat{\bm{\theta}}_{I}(\mathbf{g})$ can be computed as [6]:

$T_{I}(\mathbf{g})=\max_{\hat{\bm{\theta}}}\left\\{\int
p(\bm{\theta})\Lambda(\mathbf{g}|\bm{\theta})u(\bm{\theta},\hat{\bm{\theta}})d\bm{\theta}\right\\},$
(5)

and

$\hat{\bm{\theta}}_{I}(\mathbf{g})=\operatorname*{argmax}_{\hat{\bm{\theta}}}\left\\{\int
p(\bm{\theta})\Lambda(\mathbf{g}|\bm{\theta})u(\bm{\theta},\hat{\bm{\theta}})d\bm{\theta}\right\\},$
(6)

where the quantity $\Lambda(\mathbf{g}|\bm{\theta})$ is the
$\bm{\theta}-$conditional likelihood ratio:

$\Lambda(\mathbf{g}|\bm{\theta})=\frac{p(\textbf{g}|\bm{\theta},H_{1})}{p(\textbf{g}|H_{0})}.$
(7)

Equation (5) implies that the IO test statistic can be written as [6]:

$T_{I}(\mathbf{g})=\int
p(\bm{\theta})\Lambda(\mathbf{g}|\bm{\theta})u(\bm{\theta},\hat{\bm{\theta}}_{I}(\mathbf{g}))d\bm{\theta},$
(8)

where $\hat{\bm{\theta}}_{I}(\mathbf{g})$ is the ideal estimator defined in
Eqn. (6).

###  II-C Scanning linear observer

The scanning linear observer (SLO) [12] is a sub-optimal linear observer that
can be employed with detection-estimation tasks when the IO is intractable
[14, 13]. The SLO is designed to approximate the mode of the posterior density
and perform pseudo maximum a posteriori (MAP) estimation [12, 14] as:

$\displaystyle\hat{\bm{\theta}}_{SLO}(\mathbf{g})=$
$\displaystyle\operatorname*{argmax}_{\hat{\bm{\theta}}}\\{\mathbf{\bar{g}(\hat{\bm{\theta}})}^{T}\mathbf{\bar{K}_{g}}^{-1}\mathbf{g}-\frac{1}{2}\mathbf{\bar{g}(\hat{\bm{\theta}})}^{T}\mathbf{\bar{K}_{g}}^{-1}\mathbf{\bar{g}(\hat{\bm{\theta}})}$
(9) $\displaystyle+\ln p(\hat{\bm{\theta}})\\},$

where $\mathbf{\bar{g}}(\hat{\bm{\theta}})$ is the mean image averaged over
the parameters and $\mathbf{\bar{K}_{g}}^{-1}$ is the inverse covariance
matrix for $(\mathbf{g}-\mathbf{\bar{g}(\hat{\bm{\theta}})})$ with
approximation of slowly varying parameters. The corresponding SLO test
statistic is given by

$\displaystyle T_{SLO}(\mathbf{g})=$
$\displaystyle\mathbf{\bar{g}}(\hat{\bm{\theta}}_{SLO}(\mathbf{g}))^{T}\mathbf{\bar{K}_{g}}^{-1}\mathbf{g}$
(10)
$\displaystyle-\frac{1}{2}\mathbf{\bar{g}}(\hat{\bm{\theta}}_{SLO}(\mathbf{g}))^{T}\mathbf{\bar{K}_{g}}^{-1}\mathbf{\bar{g}}(\hat{\bm{\theta}}_{SLO}(\mathbf{g}))$
$\displaystyle+\ln p(\hat{\bm{\theta}}_{SLO}(\mathbf{g})).$

##  III Approximating the IO for signal detection-estimation tasks

###  III-A Approximating the IO using a hybrid supervised learning-MCMC method

The supervised learning-based methods employed in previous studies [17, 9, 10]
cannot be utilized directly to approximate the IO test statistic for
detection-estimation tasks. One reason is that the test statistic
$T_{I}(\mathbf{g})$ in Eqn. (8) involves an integral that depends on the ideal
estimator $\hat{\bm{\theta}}_{I}(\mathbf{g})$. To circumvent this, a novel
hybrid supervised learning-Markov-Chain Monte Carlo (MCMC) method is described
below for IO approximation.

Considering that the signal parameter vector $\bm{\theta}$ is estimated only
when the signal is determined to be present,
$p(\bm{\theta})=p(\bm{\theta}|H_{1})$ holds true [6, 16]. Thus, the test
statistic $T_{I}(\mathbf{g})$ can be expressed as

$T_{I}(\mathbf{g})=\int
p(\bm{\theta}|H_{1})\frac{p(\mathbf{g}|\bm{\theta},H_{1})}{p(\mathbf{g}|H_{0})}u(\hat{\bm{\theta}}_{I}(\mathbf{g}),\bm{\theta})d\bm{\theta}.$
(11)

Using Bayes’s rule, $T_{I}(\mathbf{g})$ can then be decomposed as [7]

$\displaystyle T_{I}(\mathbf{g})$
$\displaystyle=\int\frac{pr(\mathbf{g}|H_{1})}{pr(\mathbf{g}|H_{0})}pr(\bm{\theta}|\mathbf{g},H_{1})u(\hat{\bm{\theta}}_{I}(\mathbf{g}),\bm{\theta})d\bm{\theta}$
(12) $\displaystyle\equiv\Lambda(\mathbf{g})U(\mathbf{g}),$

where
$\Lambda(\mathbf{g})\equiv\frac{p(\mathbf{g}|H_{1})}{p(\mathbf{g}|H_{0})}$ is
the likelihood ratio and

$U(\mathbf{g})\equiv\int
pr(\bm{\theta}|\mathbf{g},H_{1})u(\hat{\bm{\theta}}_{I}(\mathbf{g}),\bm{\theta})d\bm{\theta}$
(13)

is the utility weighted posterior mean. Equation (12) is central to the
methodology described below.

The proposed hybrid supervised learning-MCMC strategy is summarized in Fig. 1.
First, a multi-task CNN is employed to approximate the likelihood ratio
$\Lambda(\mathbf{g})$ and the ideal estimate
$\hat{\bm{\theta}}_{I}(\mathbf{g})$. Second, $U(\mathbf{g})$ is approximated
by use of a MCMC technique. Finally, the IO test statistic is obtained by
multiplying $\Lambda(\mathbf{g})$ and $U(\mathbf{g})$ according to Eqn. (12).
The corresponding details are described below.

Figure 1: A schematic of the hybrid method that combines a multi-task CNN and
the MCMC method. The Shared Conv and Estimation Conv are two blocks that
comprise several convolutional layers. The convolutional layers in the Shared
Conv are shared by the detection and estimation sub-networks (i.e., the bottom
and top network branches). The layers in the Estimation Conv are employed by
the estimation sub-network only.

####  III-A1 Approximating the likelihood ratio and ideal estimate using
multi-task CNNs

The likelihood ratio $\Lambda(\mathbf{g})$ and the ideal estimate
$\hat{\bm{\theta}}_{I}(\mathbf{g})$ can be approximated by use of CNNs.
Specifically, as shown in Fig. 1, a multi-task CNN architecture is employed in
which several convolutional layers in the Shared Conv block are shared by the
detection and estimation sub-networks (i.e., the bottom and top network
branches in Fig. 1). Several additional convolutional layers in the Estimation
Conv block are employed by the estimation sub-network only. For use in
training the detection network, the loss function is defined as the binary
cross entropy loss function to approximate the posterior probability
$p(H_{1}|\mathbf{g})$, which is a monotonic transformation of
$\Lambda(\mathbf{g})$ [9]. For use in training the estimation sub-networks,
the loss function is defined as the negative of the utility function [6, 18].
The derivation of the loss function for estimation tasks is provided in
Appendix A.

Consider that a training dataset $\\{\mathbf{g}_{i}\\}_{1}^{2J}$ that contains
$2J$ independent measured images with the first $J$ images satisfying the
$H_{1}$ hypothesis and the rest satisfying the $H_{0}$ hypothesis is employed
to train the multi-task CNN. Let $y\in\\{0,1\\}$ denote the class label, where
$y=0$ and $y=1$ correspond to the hypothesis $H_{0}$ and $H_{1}$,
respectively. The corresponding class labels and target parameter vectors for
the signal present cases are denoted as $\\{y_{i}\\}_{1}^{2J}$ and
$\\{\bm{\theta}_{i}\\}_{1}^{J}$, respectively. Let the vector
$\mathbf{w}=[\mathbf{w_{1}},\mathbf{w_{2}}]$ denote the trainable parameters
of the multi-task CNN, where $\mathbf{w_{1}}$ and $\mathbf{w_{2}}$ denote the
weights of the detection and estmation sub-networks, respectively. In terms of
these quantities, the loss functions employed for training the detection and
estimation sub-networks can be expressed as:

$\mathcal{L}_{\text{Detection}}(\mathbf{w}_{1})=-\sum_{j=1}^{2J}p(y_{i}|\mathbf{g}_{i},\mathbf{w}_{1})\log
p(y_{i}|\mathbf{g}_{i},\mathbf{w}_{1}),$ (14a)
$\mathcal{L}_{\text{Estimation}}(\mathbf{w}_{2})=-\frac{1}{J}\sum_{j=1}^{J}u(\hat{\bm{\theta}}_{j}(\mathbf{w}_{2}),\bm{\theta}_{j}).$
(14b)

The multi-task CNN can be trained by minimizing the loss functions described
in Eqn. (14) in an alternating fashion. At each iteration of the training
process, Eqn. (14b) is minimized first and Eqn. (14a) is minimized
subsequently. Additional details relevant to a specific implementation of this
procedure are described in Sec. IV-D.

####  III-A2 Approximating the utility weighted posterior mean using MCMC
techniques

According to Eqn. (12), the utility weighted posterior mean $U(\mathbf{g})$ in
Eqn. (13) is required when computing $T_{I}(\mathbf{g})$. For a SKS/BKE task,
$U(\mathbf{g})$ can be approximated via Monte Carlo integration as:

$\hat{U}(\mathbf{g})=\frac{1}{J}\sum_{j=1}^{J}u(\hat{\bm{\theta}}_{I}(\mathbf{g}),\bm{\theta}^{(j)}),$
(15)

where $\bm{\theta}^{(j)}$ is sampled from the posterior distribution
$p(\bm{\theta}|\mathbf{g},H_{1})$. A Markov chain with an initial parameter
vector $\bm{\theta}^{(0)}$ and a proposal density
$q(\bm{\theta}|\bm{\theta}^{(j)})$ can be constructed to generate
$\bm{\theta}^{(j)}$, which is described below.

Given $\bm{\theta}^{(i)}$, a candidate parameter vector $\bm{\theta}^{\ast}$
is drawn from the proposal density and is accepted to the Markov chain with an
acceptance probability:

$\displaystyle p_{a}(\bm{\theta}^{\ast}|$
$\displaystyle\bm{\theta^{(i)}},\mathbf{g})=$ (16)
$\displaystyle\min\left[1,\frac{p(\mathbf{g}|\mathbf{s}(\bm{\theta}^{\ast}),H_{1})p(\bm{\theta}^{\ast})q(\bm{\theta}^{\ast}|\bm{\theta}^{(i)})}{p(\mathbf{g}|\mathbf{s}(\bm{\theta}^{(i)}),H_{1})p(\bm{\theta}^{(i)})q(\bm{\theta}^{(i)}|\bm{\theta}^{\ast})}\right],$

where $p(\bm{\theta})$ is the distribution of the parameter vector. The signal
images $\mathbf{s}(\bm{\theta}^{(i)})$ and $\mathbf{s}(\bm{\theta}^{\ast})$
are specified by $\bm{\theta}^{(i)}$ and $\mathbf{s}(\bm{\theta}^{\ast})$,
respectively. The vector $\bm{\theta}^{(i+1)}\equiv\bm{\theta}^{\ast}$ if the
candidate is accepted; otherwise $\bm{\theta}^{(i+1)}\equiv\bm{\theta}^{(i)}$.
When a random walk Metropolis-Hastings (RWMH) [19] is employed, the proposal
density $q(\bm{\theta}^{\ast}|\bm{\theta}^{(i)})$ is a simple Gaussian
density:
$q(\bm{\theta}^{\ast}|\bm{\theta}^{(i)})\propto\exp[-\frac{1}{2}(\bm{\theta}^{\ast}-\bm{\theta}^{(i)})^{T}\mathbf{K}^{-1}(\bm{\theta}^{\ast}-\bm{\theta}^{(i)})]$.

When background variability is considered, $U(\mathbf{g})$ can still be
approximated according to Eqn. (15) if it is assumed that the background can
be described by a stochastic object model (SOM) that is parameterized by the
parameter vector $\bm{\alpha}$, i.e.,
$\mathbf{b}\equiv\mathbf{b}(\bm{\alpha})$. The corresponding acceptance rate
is:

$\displaystyle
p_{a}(\bm{\theta}^{\ast},\mathbf{b}(\bm{\alpha}^{\ast})|\bm{\theta^{(i)}},\mathbf{b}$
$\displaystyle(\bm{\alpha}^{(i)}),\mathbf{g})=$ (17)
$\displaystyle\min\left[1,\frac{p_{a}(\bm{\theta}^{\ast},\mathbf{b}(\bm{\alpha}^{\ast}))}{p_{a}(\bm{\theta^{(i)}},\mathbf{b}(\bm{\alpha}^{(i)}))}\right],$

where

$\displaystyle p_{a}(\bm{\theta}^{\ast},\mathbf{b}(\bm{\alpha}^{\ast}))=$
$\displaystyle
p(\mathbf{g}|\mathbf{s}(\bm{\theta}^{\ast}),\mathbf{b}(\bm{\alpha}^{\ast}),H_{1})p(\bm{\theta}^{\ast})p(\bm{\alpha}^{\ast})$
(18a) $\displaystyle
q_{1}(\bm{\theta}^{\ast}|\bm{\theta}^{(i)})q_{2}(\bm{\alpha}^{\ast}|\bm{\alpha}^{(i)}),$
$\displaystyle p_{a}(\bm{\theta^{(i)}},\mathbf{b}(\bm{\alpha}^{(i)}))=$
$\displaystyle
p(\mathbf{g}|\mathbf{s}(\bm{\theta}^{(i)}),\mathbf{b}(\bm{\alpha}^{(i)}),H_{1})p(\bm{\theta}^{(i)})$
(18b) $\displaystyle
p(\bm{\alpha}^{(i)})q_{1}(\bm{\theta}^{(i)}|\bm{\theta}^{\ast})q_{2}(\bm{\alpha}^{(i)}|\bm{\alpha}^{\ast}).$

Here, $\mathbf{b}(\bm{\alpha}^{(i)})$ and $\mathbf{b}(\bm{\alpha}^{\ast})$ are
the background images determined by the parameter vectors $\bm{\alpha}^{(i)}$
and $\bm{\alpha}^{\ast}$, respectively. The quantity $p(\hat{\bm{\alpha}})$ is
the corresponding probability density, and $q_{1}(\cdot)$ and $q_{2}(\cdot)$
are the proposal densities for the signal and background parameters,
respectively. The pair $(\bm{\theta}^{(i)},\bm{\alpha}^{(i)})$ are sampled
from the distribution
$p(\bm{\theta},\mathbf{b}(\bm{\alpha})|\mathbf{g},H_{1})$, and
$(\bm{\theta}^{\ast},\bm{\alpha}^{\ast})$ are candidate parameter vectors that
are drawn from the proposal densities. A Markov chain for generating
$(\bm{\theta}^{(i)},\bm{\alpha}^{(i)})$ can then be established in a similar
way as described above.

###  III-B Supervised learning-based sub-ideal NO

As described above, the proposed hybrid method can approximate the IO for
detection-estimation tasks. However, the MCMC technique is a component of the
hybrid method, which requires knowledge of background density $p(\bm{\alpha})$
if background variability is considered. This currently limits the application
of the hybrid method to certain object models [9]. To circumvent this, a
purely supervised learning-based sub-optimal NO can be employed.

Figure 2: A schematic of the multi-task CNN employed to form the sub-ideal NO.
The Shared Conv and Estimation Conv are two blocks that comprise several
convolutional layers. The convolutional layers in the Shared Conv are shared
by the detection and estimation sub-networks (i.e., the bottom and top network
branches). The layers in the Estimation Conv are employed by the estimation
sub-network only.

A sub-optimal NO that is established by use of only supervised-learning can be
readily obtained by eliminating the MCMC method from the hybrid method as
depicted in Fig. 2. This modification removes the influence of the estimation
result on the detection performance. The corresponding test statistic and
estimator are given by:

$T_{Sub}(\mathbf{g})=\frac{p(\mathbf{g}|H_{1})}{p(\mathbf{g}|H_{0})}=\Lambda(\mathbf{g})$
(19a)

$\hat{\bm{\theta}}_{Sub}(\mathbf{g})=\operatorname*{argmax}_{\hat{\bm{\theta}}}\left\\{\int
p(\bm{\theta})\Lambda(\mathbf{g}|\bm{\theta})u(\bm{\theta},\hat{\bm{\theta}})d\bm{\theta}\right\\},$
(19b)

where $T_{Sub}(\mathbf{g})$ is IO test statistic for binary signal detection
tasks and $\hat{\bm{\theta}}_{Sub}(\mathbf{g})$ IO estimator for detection-
estimation tasks.

##  IV Numerical studies

Computer-simulation studies were performed to investigate the proposed NOs for
detection-estimation tasks. The considered signal detection-estimation tasks
included both BKE and BKS tasks. A lumpy background (LB) model [8] and a
clustered lumpy background (CLB) model [20] were employed in the BKS tasks.

The imaging system considered was an idealized parallel-hole collimator system
that was specified by a linear C-D mapping with Gaussian point response
functions (PRFs) [8]:

$h_{m}(\mathbf{r})=\frac{h}{2\pi
w_{m}^{2}}\exp\left[-\frac{(\mathbf{r-r}_{m})^{T}(\mathbf{r-r}_{m})}{2w_{m}^{2}}\right],$
(20)

where $h$ and $w_{m}$ are the height and width of the PRFs, respectively.

The signal to be detected and estimated was modeled by a 2D Gaussian function:

$f_{s}(\mathbf{r})={A_{s}}\exp\left[-\frac{(\mathbf{r-r}_{s})^{T}(\mathbf{r-r}_{s})}{2w_{s}^{2}}\right],$
(21)

where $A_{s}$ is the signal amplitude, $w_{s}$ is the signal width and
$\mathbf{r_{s}}$ is the center of signal. These signal parameters can be
random when a detection-estimation task is specified. Considering the
specified imaging system in Eqn. (20), the $m^{th}$ element $[\mathbf{s}]_{m}$
of the signal image $\mathbf{s}$ is given by:

$[\mathbf{s}]_{m}=\frac{{A_{s}}{h}{w_{s}^{2}}}{w_{m}^{2}+w_{s}^{2}}\exp\left[-\frac{(\mathbf{r}_{m}-\mathbf{r}_{s})^{T}(\mathbf{r}_{m}-\mathbf{r}_{s})}{2(w_{m}^{2}+w_{s}^{2})}\right].$
(22)

In the studies described below, the AEROC was employed to quantify observer
performance. The AEROC was estimated by use of a nonparametric estimator [11]
and the uncertainty in the estimates was conveyed by use of a 90% confidence
interval.

###  IV-A SKS/BKE signal detection-estimation task

In the SKS/BKE case considered, the task was to detect a random signal and
estimate its amplitude. This task can be viewed as a surrogate for tumor
detection in positron emission tomography (PET) [21]. In this task, the
dimensions of $\mathbf{g}$, $\mathbf{b}$, and $\mathbf{n}$ were $64\times 64$
pixels. Without loss of generality, $\mathbf{b=0}$. The signal defined in Eqn.
(21) was employed with $w_{s}=1$ and $\mathbf{r}_{s}=[32;32]^{T}$. The to-be-
estimated signal amplitude was sampled from a Gaussian distribution with mean
$\mu_{A}=9$ and standard deviation $\sigma_{A}=4$, i.e.,
$A_{s}\sim\mathcal{N}(9,4^{2})$. The assumed parameters of the imaging system
defined in Eqn. (20) were $h=16$ and $w_{m}=3.87$. The standard deviation
$\sigma_{n}$ of Gaussian noise was set to 40. To define an EROC curve, a
Gaussian utility function,
$u(\hat{A_{s}},A_{s})=\exp[-(\hat{A_{s}}-A_{s})^{2}/(2\sigma_{u}^{2})]$, was
employed where $\sigma_{u}=3$.

Wunderlich et al.[16] provided the optimal decision rule for this case that is
expressed as:

$\displaystyle\hat{A}_{I}(\mathbf{g})=\frac{\sigma_{A}^{2}\mathbf{s}_{ref}^{T}\mathbf{g}+\sigma_{n}^{2}\mu_{A}}{\sigma_{n}^{2}+\sigma_{A}^{2}\mathbf{s}_{ref}^{T}\mathbf{s}_{ref}}$
(23) $\displaystyle
T_{I}(\mathbf{g})=\mu_{A}(\mathbf{s}_{ref}^{T}\mathbf{g})+\frac{\sigma_{A}^{2}}{2\sigma_{n}^{2}}(\mathbf{s}_{ref}^{T}\mathbf{g})^{2}.$

Here, $\mathbf{s}_{ref}$ is a reference signal whose components are defined as

$[\mathbf{s}_{ref}]_{m}=\frac{{h}{w_{s}^{2}}}{w_{m}^{2}+w_{s}^{2}}\exp\left[-\frac{(\mathbf{r}_{m}-\mathbf{r}_{s})^{T}(\mathbf{r}_{m}-\mathbf{r}_{s})}{2(w_{m}^{2}+w_{s}^{2})}\right].$
(24)

This analytic IO decision strategy was implemented as a reference method that
the proposed hybrid method and sub-ideal NO were compared against. The
Gaussian proposal density in the MCMC method that was employed to estimate
$U(\mathbf{g})$ was chosen to have a standard deviation of 3.

###  IV-B SKS/BKS signal detection-estimation tasks with a lumpy background
model

In the first BKS task considered, the Gaussian signal defined in Eqn. (21) was
employed with $A_{s}=6$ and $w_{s}=3$. The signal location $\mathbf{r_{s}}$
was a two-dimensional random vector whose components were independently
sampled from a uniform distribution on the interval $(16,48)$. A quadratic
utility function
$u_{1}(\mathbf{\hat{r}_{s}},\mathbf{r_{s}})=1-\frac{1}{\epsilon_{1}}\|\mathbf{\hat{r}_{s}}-\mathbf{r_{s}}\|_{2}^{2}$
and an $l^{1}$-norm based utility function
$u_{2}(\mathbf{\hat{r}_{s}},\mathbf{r_{s}})=1-\frac{1}{\epsilon_{2}}\|\mathbf{\hat{r}_{s}}-\mathbf{r_{s}}\|_{1}$
were employed, where $\epsilon_{1}=100$ and 200, and $\epsilon_{2}=20$,
respectively. A quadratic utility function focuses on large errors while an
$l^{1}$-norm based one emphasizes more on small errors. The different values
$\epsilon_{1}$ were employed to investigate the influences of estimation tasks
on the test statistics according to Eqn. (12). Compared with previously
investigated detection-localization tasks [22, 10], the task in this study can
be considered as a straightforward generalization.

To emulate background variability, a lumpy object model [23] was utilized:

$f_{b}(\mathbf{r})=\sum^{N_{b}}_{n=1}l(\mathbf{r-r}_{n}|a,w_{b}),$ (25)

where $N_{b}\sim P(\bar{N})$ denotes the number of the lumps with $P(\bar{N})$
denoting a Poisson distribution with the mean $\bar{N}=5$. The lump function
$l(\mathbf{r-r}_{n}|a,w_{b})$ was modeled by a 2D Gaussian function with lump
amplitude $a=10$ and lump width $w_{b}=7$:

$l(\mathbf{r-r}_{n}|a,w_{b})=a\exp(-\frac{(\mathbf{r-r}_{n})^{T}(\mathbf{r-r}_{n})}{2w^{2}_{b}}).$
(26)

Here, $\mathbf{r}_{n}$ denotes the center location of the $n^{th}$ lump that
was sampled from a uniform distribution over the spatial support of the image.
The dimensions of $\mathbf{s}$, $\mathbf{b}$, $\mathbf{n}$ and $\mathbf{g}$ in
Eqn. (3) were $64\times 64$. The imaging system was specified by $h=40$ and
$w_{m}=0.5$. Given the assumed imaging system, the $m^{th}$ element
$[\mathbf{b}]_{m}$ of the background image $\mathbf{b}$ is given by:

$[\mathbf{b}]_{m}=\frac{{a}{h}{w_{b}^{2}}}{w_{m}^{2}+w_{b}^{2}}\sum_{n=1}^{N_{b}}\exp\left[-\frac{(\mathbf{r}_{n}-\mathbf{r}_{m})^{T}(\mathbf{r}_{n}-\mathbf{r}_{m})}{2(w_{m}^{2}+w_{b}^{2})}\right].$
(27)

The measurement noise was described by i.i.d. Gaussian random variables with a
mean of 0 and a standard deviation of 320. One realization of the signal image
$\mathbf{s}$, the background image $\mathbf{b}$, and the corresponding signal-
present noisy measurement $\mathbf{g}$ are shown in Figure 3.

Because the IO decision rule in this case cannot be analytically computed, an
MCMC-based IO approximation method was established as a reference method.
Unlike the proposed hybrid method, this MCMC-based method requires a strong
constraint on the utility function. The details of the MCMC approximated IO
(MCMC-IO) are provided in Appendix B. The performances of the proposed hybrid
method and sub-ideal NO were compared to that of MCMC-IO when the quadratic
utility function was employed. When the $l^{1}$-norm based utility function
was considered, the MCMC-IO cannot be employed and the SLO described in Sec.
II-C was utilized as the reference observer. To implement the SLO, the
covariance matrix was estimated by use of the covariance matrix decomposition
method [3, 13] and 4,000 signal-present and 4,000 signal-absent noiseless
images were utilized. The Gaussian proposal density in the MCMC method that
was employed to estimate $U(\mathbf{g})$ was chosen to have a standard
deviation of 4 for each location coordinate.

Figure 3: Realizations that depict (a) a possible signal $\mathbf{s}$, (b) a
background image $\mathbf{b}$, and (c) the corresponding noisy measurement
$\mathbf{g}$ for the SKS/BKS signal detection-estimation tasks with a LB
model. The dimensions of the images were $64\times 64$.

###  IV-C SKS/BKS signal detection-estimation tasks with a clustered lumpy
background model

In the second BKS task considered, the signal image $\mathbf{s}$ had an
amplitude of 0.05 and signal location $\mathbf{r}_{s}=[32;32]^{T}$. The width
of $\mathbf{s}$ was a random vector and was sampled from a uniform
distribution on the interval $(1,6)$. A Gaussian utility function,
$u(\hat{w},w)=\exp[-(\hat{w}-w)^{2}/(2\sigma_{u}^{2})]$, was employed with
$\sigma_{u}=3$.

A clustered lumpy background (CLB) model [20] was employed to emulate
background variability. The $m^{th}$ element $[\mathbf{b}]_{m}$ of the
background image $\mathbf{b}$ was computed as [20]:

$[\mathbf{b}]_{m}=\sum_{k=1}^{K}\sum_{n=1}^{N_{k}}l(\mathbf{r}_{m}-\mathbf{r}_{k}-\mathbf{r}_{kn}|\mathbf{R}_{\theta_{kn}}).$
(28)

Here, $K\sim P(\bar{K})$ denotes the number of the clusters with $P(\bar{K})$
denoting a Poisson distribution with the mean $\bar{K}$, $N_{k}$ denotes the
number of blobs in the $k^{th}$ cluster that was sampled from a Poisson
distribution with the mean of $\bar{N}$: $N_{k}\sim P(\bar{N})$,
$\mathbf{r}_{k}$ denotes the center location of the $k^{th}$ cluster that was
sampled from a uniform distribution over the spatial support of the image, and
$\mathbf{r}_{kn}$ denotes the center location of the $n^{th}$ blob in the
$k^{th}$ cluster that was sampled from a Gaussian distribution with the center
of $\mathbf{r}_{k}$ and standard deviation of $\sigma$. The blob function
$l(\mathbf{r}|\mathbf{R}_{\theta_{kn}})$ was specified as:

$l(\mathbf{r}|\mathbf{R}_{\theta_{kn}})=\exp\left(-\alpha\frac{\|\mathbf{R}_{\theta_{kn}}\mathbf{r}\|^{\beta}}{L(\mathbf{R}_{\theta_{kn}}\mathbf{r})}\right).$
(29)

Here, $L(\mathbf{r})$ is computed as the “radius” of the ellipse with half-
axes $L_{x}$ and $L_{y}$, and $\mathbf{R}_{\theta_{kn}}$ is the rotation
matrix corresponding to the angle $\theta_{kn}$ that was sampled uniformly
between 0 and $2\pi$. The generated background images were normalized to the
range between 0 and 1. The parameters that specify the CLB model employed in
this study are summarized in Table. I.

TABLE I: Parameters for generating CLB images $\bar{K}$ | $\bar{N}$ | $L_{x}$ | $L_{y}$ | $\alpha$ | $\beta$ | $\sigma$
---|---|---|---|---|---|---
70 | 20 | 5 | 2 | 2.1 | 0.5 | 12

The measurement noise was described by i.i.d. Gaussian random variables with a
mean of 0 and a standard deviation of 0.33. One realization of the signal
image $\mathbf{s}$, the background image $\mathbf{b}$, and the corresponding
signal-present noisy measurement $\mathbf{g}$ are shown in Figure 4.

Figure 4: Images that depict (a) a possible signal $\mathbf{s}$, (b) a
background image $\mathbf{b}$, and (c) the corresponding noisy measurement
$\mathbf{g}$ for the SKS/BKS signal detection-estimation tasks with a CLB
model. The dimensions of the images were $64\times 64$.

The SLO described in Sec. II-C was utilized as the reference observer. The
details regarding the implementation are described in Sec. IV-A.

###  IV-D Multi-task CNN training details

Details regarding the implementation of multi-task CNN described in Sec.
III-A1 are described here. Each convolutional layer in the Shared Conv block
comprised 64 filters with $5\times 5$ spatial support followed by a Leaky ReLU
activation function [24]. In the detection sub-network, a max-pooling layer
[25] was employed to subsample the feature maps and the last layer was fully
connected (FC) and employed a sigmoid activation function for estimation of
the posterior probability. In the estimation sub-network, the Estimation Conv
block included additional convolutional layers to account for the fact that
estimation can be a more complicated task. The architecture for both blocks
were identical. A max-pooling layer and a FC layer were added to compute the
estimate.

The train-validation-test scheme [26] was employed to train and evaluate
multi-task CNNs. The initial training dataset included 150,000 signal-present
images and 150,000 signal-absent images for the BKS detection-estimation task
with the lumpy background. For tasks that involved the clustered lumpy
background, 200,000 signal-present images and 200,000 signal-absent images
were employed. To mitigate overfitting, a “semi-online learning” method [9,
10] was employed in which measurement noise was generated on-the-fly and added
to noiseless images drawn from the finite initial training dataset. The
validation datatset included 1000 signal-present images and 1000 signal absent
images. Finally, the test dataset comprised 1000 signal-present images and
1000 signal-absent images.

The multi-task CNN was trained by minimizing the loss functions described in
Eqn. (14) on mini-batches in an alternating fashion. At each iteration of the
training process, Eqn. (14b) was minimized first and Eqn. (14a) was minimized
subsequently. Multi-task CNNs comprising different number of layers were
trained for 200,000 mini-batches. Each mini-batch contained 200 signal-absent
images and 200 signal-present images randomly selected from the training data
set. The Adam optimizer [27] with a learning rate of 0.00001 was employed for
model training.

In order to accurately approximate the IO, the CNN architecture needs to
possess sufficient capacity [9]. For determining the optimal architecture of
the multi-task CNN, the training process started from an architecture with one
convolutional layer in both the Shared Conv and Estimation Conv blocks to
formulate baseline detection and estimation sub-networks. More layers were
added gradually as described below. The optimal number of convolutional layers
for the detection sub-network was determined by adding more layers in the
Shared Conv block until the cross-entropy on the validation dataset did not
significantly decrease. The loss functions were considered as significantly
decreased if its decrements are at least $1.0\%$ of that produced by multi-
task CNN with one less convolutional layer. After the number of layers for the
detection sub-network was determined, the optimal number of layers for
estimation sub-network was specified by increasing the number of convolutional
layers in the Estimation Conv block. The training and implementation of the
multi-task CNN were performed using Tensorflow [28].

##  V Results

###  V-A SKS/BKE signal detection-estimation tasks

The optimal network architecture was determined to contain three convolutional
layers in the Shared Conv block and two layers in the Estimation Conv block.
In Figure 5, the EROC curves produced by the approximated IO (black solid
curve) and the sub-ideal NO (yellow dashed curve) are compared with that
produced by the analytical computation (red dashed curve). The AEROC values
were $0.565\pm 0.010$, $0.565\pm 0.010$, and $0.570\pm 0.010$ corresponding to
the approximated IO, the sub-ideal NO, and the analytical computation,
respectively. The corresponding EROC curves were statistically equivalent in
this SKE/BKE case.

Figure 5: The EROC curves produced by the approximated IO (black), the
analytical computation (red), and sub-ideal NO (yellow) for the BKE task were
statistically equivalent.

###  V-B SKS/BKS signal detection-estimation tasks with a lumpy background
model

The optimal network architecture was determined to contain seven convolutional
layers in the Shared Conv block and three layers in the Estimation Conv block.
To compare the proposed methods with the IO approximated by the MCMC method, a
quadratic utility function described in Sec. IV-B with $\epsilon_{1}=200$ was
employed. For this case, the EROC curves generated by use of the approximated
IO (black solid curve) and the sub-ideal NO (yellow dashed curve) are compared
with that produced by the MCMC method (red dashed curve) in Figure 6. The
AEROC values were $0.697\pm 0.021$, $0.686\pm 0.024$, and $0.708\pm 0.027$
corresponding to the approximated IO, the sub-ideal NO, and the MCMC-IO,
respectively. The corresponding EROC curves were statistically equivalent in
this BKS signal detection-estimation task.

Figure 6: The EROC curves produced by the approximated IO (black), the MCMC-IO
(red), and sub-ideal NO (yellow) for the BKS task with the lumpy background
model are compared. The corresponding AEROC values were statistically
equivalent in this case. The quadratic utility function described in Sec. IV-B
was employed with $\epsilon_{1}=200$.

A quadratic utility function described in Sec. IV-B with $\epsilon_{1}=100$
was employed to investigate a situation where the performance of sub-ideal NO
is worse than that of the approximated IO. Compared with the case addressed in
Figure 6, the variance of $U(\mathbf{g})$ increased from 0.1708 to 0.7295,
indicating that the estimation performance has larger influence on the test
statistic according to Eqn. (12). Because of this increase in variance, the
performance of the hybrid approximated IO became better than that of the sub-
ideal NO. In Figure 7, the EROC curves generated by use of the approximated IO
(black solid curve) and the sub-ideal NO (yellow dashed curve) are compared
with that produced by the MCMC method (red dashed curve). The AEROC values
were $0.545\pm 0.036$, $0.486\pm 0.042$, and $0.553\pm 0.031$ corresponding to
the approximated IO, the sub-ideal NO, and the MCMC-IO, respectively. The EROC
curves corresponding to the approximated IO and the MCMC-IO are in close
agreement in this SKS/BKS signal detection-estimation task, while the
difference between the approximated IO and the sub-ideal NO is statistically
significant.

Figure 7: The EROC curves produced by the approximated IO (black), the MCMC-IO
(red), and the sub-ideal NO (yellow) for the BKS task with the lumpy
background model are compared. The quadratic utility function described in
Sec. IV-B was employed with $\epsilon_{1}=100$.

Intuition for this result can be gained by noting that, when the variance of
$U(\mathbf{g})$ is small, Eqn. (12) can be approximated as
$T_{I}(\mathbf{g})\approx\Lambda(\mathbf{g})\langle{U}(\mathbf{g})\rangle$,
where $\langle{U}(\mathbf{g})\rangle$ is the mean of $U(\mathbf{g})$. In this
case, because their test statistics differ by (approximately) a constant, the
hybrid approximated IO and sub-ideal NO will perform similarly. However, this
is not to be expected when the variance of $U(\mathbf{g})$ is large because
the test statistics $T_{I}(\mathbf{g})$ and $\Lambda(\mathbf{g})$ will not
simply differ by a constant and they will not generally be related by a
monotonic transformation.

To demonstrate a case where the MCMC-IO cannot be applied but the hybrid
method can, the $l^{1}$-norm based utility function described in Sec. IV-B was
employed. In this case, the SLO was employed as the reference observer.
Compared with a quadratic utility function, small errors weight more in an
$l^{1}$-norm based one. As shown in Figure 8, the EROC curves generated by use
of the approximated IO (black solid curve) and the sub-ideal NO (red dashed
curve) are compared with that produced by the SLO (yellow dashed curve). The
AEROC values corresponding to the approximated IO, the sub-ideal NO, and the
SLO were $0.643\pm 0.018$, $0.633\pm 0.018$, and $0.294\pm 0.023$,
respectively. The EROC curves corresponding to the approximated IO and the
sub-ideal NO were in close agreement in this task. As expected, the AEROC
value produced by the SLO was significantly smaller. The results demonstrated
that the performance of sub-ideal NO was significantly better than that of the
SLO.

Figure 8: The EROC curves produced by the approximated IO (black), the sub-
ideal NO (red), and the SLO (yellow) for the BKS task with the lumpy
background model are compared. The $l^{1}$-norm based utility function
described in Sec. IV-B was employed with $\epsilon_{2}=20$.

###  V-C BKS signal detection-estimation tasks with a CLB model

To investigate a case where only the sub-ideal purely supervised NO could be
applied, the task involving the CLB background model was employed. The optimal
network architecture was determined to contain eight convolutional layers in
the Shared Conv block and three layers in the Estimation Conv block.
Considering that current MCMC methods have not been applied to the CLB object
model, the hybrid method was not employed and the SLO was employed as the
reference method. The EROC curves corresponding to the sub-ideal NO (black
dashed curve) are compared with that generated by SLO (red dashed curve) in
Figure 9. The AEROC value corresponding to the sub-ideal purely supervised NO
was $0.601\pm 0.012$, which was larger than the $0.538\pm 0.013$ produced by
the SLO.

Figure 9: The EROC curves corresponding to the sub-ideal NO (black) are
compared with that generated by SLO (red) for the BKS task with the clustered
lumpy background model. The AEROC value produced by the sub-ideal NO was
larger than that produced by the SLO.

##  VI Summary

General signal detection-estimation tasks are frequently considered in medical
imaging. For detection-estimation tasks, the EROC curve has been proposed for
evaluating the performance of observers. However, in practice, it is difficult
to accurately approximate the IO that maximizes the AEROC for a general
detection-estimation task. In this work, a hybrid approach was developed that
combines a multi-task CNN and a MCMC method in order to approximate the IO for
detection-estimation tasks. Compared with the MCMC-IO, the hybrid method is
not limited to use of specific utility functions. Additionally, a supervised
learning-based sub-ideal NO was designed for signal detection-estimation
tasks. Both SKS/BKE and SKS/BKS tasks were considered and computer-simulation
studies were conducted to validate the proposed methods. The proposed hybrid
method provides a new approach for approximating the IO and may advance the
application of EROC analysis for optimizing imaging systems.

The proposed methods possess certain limitations. The MCMC methods are
employed in the hybrid framework for IO approximation, which makes the
limitations of MCMC methods also exist in the hybrid methods. Numerous
practical issues such as the design of proposal densities from which the
Markov chain can be efficiently generated need to be addressed. Because of
this, the hybrid method has been limited to relatively simple object models.
An advanced method called MCMC-GAN [29] can potentially solve this problem. By
replacing the classic MCMC methods in the hybrid method with MCMC-GAN, the IO
approximation method can be extended to more complicated background models. In
addition, the supervised learning-based method may require a large amount of
training data to accurately train multi-task CNNs. To address this limitation,
one may establish a stochastic object model (SOM) from experimental data by
training an AmbientGAN [30, 31]. Given a well-established SOM, one can produce
large amount of training samples to train the multi-task CNNs.

There remains several other topics for future investigation. The proposed
methods should be further validated by use of a variety of image data and
detection-estimation tasks that address real-world problems. Additionally, it
will be important to quantify the effect of insufficient training data on the
proposed methods. Finally, it will be interesting to employ the proposed
methods to perform task-based performance evaluation of deep learning-based
image restoration [32, 33, 34, 35] and image reconstruction [36, 37, 38]
techniques.

##  Appendix A Approximating the ideal estimate using multi-task CNNs

According to Eqn. (6), the ideal estimate can be defined as:

$\hat{\bm{\theta}}_{I}(\mathbf{g})=\Lambda(\mathbf{g})\left\\{\operatorname*{argmax}_{\hat{\bm{\theta}}}\int
pr(\bm{\theta}|\mathbf{g},H_{1})u(\hat{\bm{\theta}},\bm{\theta})d\bm{\theta}\right\\},$
(30)

where $\Lambda(\mathbf{g})$ is the likelihood ratio. Considering that
$\Lambda(\mathbf{g})$ is independent to $\hat{\bm{\theta}}$, Eqn. (30) can be
written as:

$\hat{\bm{\theta}}_{I}(\mathbf{g})=\operatorname*{argmax}_{\hat{\bm{\theta}}}\int
pr(\bm{\theta}|\mathbf{g},H_{1})u(\hat{\bm{\theta}},\bm{\theta})d\bm{\theta}=\operatorname*{argmax}_{\hat{\bm{\theta}}}U(\mathbf{g}).$
(31)

A supervised learning-based method can be employed to approximate
$\hat{\bm{\theta}}_{I}(\mathbf{g})$ that maximizes the expectation
$E[U(\mathbf{g})]$ over an ensemble of training data, where the expectation is
taken over $\mathbf{g}$. Assuming $\mathbf{g}$ is signal-present
$p(\mathbf{g})=p(\mathbf{g}|H_{1})$, $E[U(\mathbf{g})]$ can be depicted as:

$\displaystyle E[U(\mathbf{g})]$ $\displaystyle=\iint
pr(\bm{\theta}|\mathbf{g},H_{1})u(\hat{\bm{\theta}},\bm{\theta})p(\mathbf{g}|H_{1})d\bm{\theta}d\mathbf{g}$
(32) $\displaystyle=\iint
pr(\bm{\theta},\mathbf{g}|H_{1})u(\hat{\bm{\theta}},\bm{\theta})d\bm{\theta}d\mathbf{g}=E[u(\hat{\bm{\theta}},\bm{\theta})],$

where the expectation is taken over the random variables $\mathbf{g}$,
$\mathbf{\hat{\bm{\theta}}}$, and $\bm{\theta}$. Considering a training
dataset $\\{\mathbf{g}_{j},\bm{\theta}_{j}\\}_{1}^{J}$ that contains $J$
independent signal-present images with corresponding parameter vectors for
joint tasks. Similar to Wunderlich et al.[11],
$E[u(\hat{\bm{\theta}},\bm{\theta})]$ can be estimated with the unbiased
estimators:

$\hat{E}[u(\hat{\bm{\theta}},\bm{\theta})]=\frac{1}{J}\sum_{j=1}^{J}u(\bm{\theta}_{j},\hat{\bm{\theta}}_{j}),$
(33)

where $\hat{\bm{\theta}}_{j}$ is the observer’s estimate of $\bm{\theta}_{j}$.
Let $\mathbf{w}_{2}$ denotes the weight vector that parameterizes an
estimation sub-network and $\hat{\bm{\theta}}(\mathbf{w}_{2})$ is the signal
parameter vector estimated by the CNN. The desired weight vector
$\mathbf{w}_{I}$ for the estimation sub-network can be determined by
optimizing the following loss function:

$\mathbf{w}_{I}=\operatorname*{argmin}_{\mathbf{w}_{2}}\frac{1}{J}\sum_{j=1}^{J}-u(\hat{\bm{\theta}}(\mathbf{w}_{2}),\bm{\theta}),$
(34)

which leads to the loss function introduced in Eqn. (14b).

##  Appendix B Approximating the IO by use of MCMC methods

In this section, a reference method based on MCMC techniques is described.
This method can approximate the IO for a general signal detection-estimation.
The difference between the hybrid method and the MCMC-based method is that the
latter one employ MCMC techniques to approximate the likelihood ratio
$\Lambda(\mathbf{g})$ and ideal estimate $\hat{\bm{\theta}}_{I}(\mathbf{g})$
while the former one use a multi-task CNN instead. The steps for approximating
$\Lambda(\mathbf{g})$ and $\hat{\bm{\theta}}_{I}(\mathbf{g})$ are described
below.

###  B-A Approximating the likelihood ratio

Kupinski et al. proposed a method to approximate $\Lambda(\mathbf{g})$
numerically by employing MCMC techniques [8] and Park et al. extended the MCMC
approach to signal-known-statistically (SKS) signal detection tasks [39].
Using the notations defined in Sec. III, the likelihood ratio
$\Lambda(\mathbf{g})=\frac{p(\mathbf{g}|H_{1})}{p(\mathbf{g}|H_{0})}$ can be
written as:

$\displaystyle\Lambda(\mathbf{g})$ $\displaystyle=\frac{\int
d\mathbf{b}(\bm{\alpha})\int
d\mathbf{s}(\bm{\theta})p(\mathbf{b}(\bm{\alpha}))p(\mathbf{g|b(\bm{\alpha}),s(\bm{\theta})},H_{1})p(\mathbf{s}(\bm{\theta}))}{\int
d\mathbf{b^{\prime}(\bm{\alpha})}p({\mathbf{b^{\prime}(\bm{\alpha})}})p(\mathbf{g|b^{\prime}(\bm{\alpha})},H_{0})}$
(35) $\displaystyle\equiv\int d\bm{\theta}\int
d\bm{\alpha}\Lambda_{BSKE}(\mathbf{g|b(\bm{\alpha}),s(\bm{\theta})})p(\mathbf{\bm{\alpha}|g},H_{0})p(\mathbf{\bm{\theta}}),$

where
$\Lambda_{BSKE}(\mathbf{g|b,s})=\frac{p(\mathbf{g|b,s},H_{1})}{p(\mathbf{g|b},H_{0})}$
is the background and signal-known exactly (BSKE) likelihood ratio and
$p(\mathbf{b|g},H_{0})=\frac{p(\mathbf{g|b},H_{0})p(\mathbf{b})}{\int
d\mathbf{b^{\prime}}p(\mathbf{g|b^{\prime}},H_{0})p(\mathbf{b^{\prime}})}$.
The likelihood ratio can be subsequently approximated as:

$\Lambda(\mathbf{g})=\frac{1}{J}\sum_{j=1}^{J}\Lambda_{BSKE}(\mathbf{g|b(\bm{\alpha^{(j)}}),s(\bm{\theta^{(j)}})}),$
(36)

where $(\bm{\theta}^{(j)},\bm{\alpha}^{(j)})$ are sampled from the
distribution $p(\bm{\alpha}|\mathbf{g},H_{0})p(\bm{\theta})$. A Markov chain
with initial parameter vectors $(\bm{\theta}^{(0)},\bm{\alpha}^{(0)})$ and
proposal densities $q_{1}(\bm{\theta}|\bm{\theta}^{(j)})$ and
$q_{2}(\bm{\alpha}|\bm{\alpha}^{(j)})$ is constructed to generate
$(\bm{\theta}^{(j)},\bm{\alpha}^{(j)})$, which is described below. Given
$(\bm{\theta}^{(i)},\bm{\alpha}^{(i)})$, candidate parameter vectors
$(\bm{\theta}^{\ast},\bm{\alpha}^{\ast})$ are drawn from the proposal density
and are accepted to the Markov chain with the acceptance probability:

$\displaystyle
p_{a}(\bm{\theta}^{\ast},\bm{\alpha}^{\ast}|\bm{\theta^{(i)}},\bm{\alpha}^{(i)},\mathbf{g})=$
(37)
$\displaystyle\min\left[1,\frac{p(\mathbf{g}|\mathbf{b}(\bm{\alpha}^{\ast}),H_{0})p(\bm{\theta}^{\ast})p(\bm{\alpha}^{\ast})q_{1}(\bm{\theta}^{\ast}|\bm{\theta}^{(i)})q_{2}(\bm{\alpha}^{\ast}|\bm{\alpha}^{(i)})}{p(\mathbf{g}|\mathbf{b}(\bm{\alpha}^{(i)}),H_{0})p(\bm{\theta}^{(i)})p(\bm{\alpha}^{(i)})q_{1}(\bm{\theta}^{(i)}|\bm{\theta}^{\ast})q_{2}(\bm{\alpha}^{(i)}|\bm{\alpha}^{\ast})}\right].$

The vectors
$(\bm{\theta}^{(i+1)},\bm{\alpha}^{(i+1)})\equiv(\bm{\theta}^{\ast},\bm{\alpha}^{\ast})$
if the candidates are accepted; otherwise
$(\bm{\theta}^{(i+1)},\bm{\alpha}^{(i+1)})\equiv(\bm{\theta}^{(i)},\bm{\alpha}^{(i)})$.
When a random walk Metropolis-Hastings (RWMH) [19] is used here, the proposal
density $q(\bm{\theta}|\bm{\theta}^{(j)})$ can be a simple Gaussian density.
Therefore, $\Lambda(\mathbf{g})$ can be approximated.

###  B-B Approximating the ideal estimate

Here, the utility function is assumed to be a quadratic utility function
$u(\hat{\bm{\theta}},\bm{\theta})=1-\frac{1}{\epsilon}\|\hat{\bm{\theta}}-\bm{\theta}\|_{2}^{2}$.
According to Eqn. (6), the ideal estimator can be written as:

$\hat{\bm{\theta}}_{I}(\mathbf{g})=\operatorname*{argmax}_{\hat{\bm{\theta}}}\left\\{\int
p(\bm{\theta})\frac{p(\textbf{g}|\bm{\theta},H_{1})}{p(\textbf{g}|H_{0})}u(\bm{\theta},\hat{\bm{\theta}})d\bm{\theta}\right\\}.$
(38)

when background variability is considered, given that the density of
background is independent to the hypothesis, i.e.,
$p(\mathbf{b})=p(\mathbf{b}|H_{1})$ and $p(\bm{\theta})=p(\bm{\theta}|H_{1})$,
the optimization can be written as following by using the Bayes’ rule after
term cancellations:

$\hat{\bm{\theta}}_{I}(\mathbf{g})=\operatorname*{argmax}_{\hat{\bm{\theta}}}\left\\{\int
d\bm{\theta}\int
d\bm{\alpha}p(\bm{\theta},\mathbf{b}(\bm{\alpha})|\mathbf{g},H_{1})u(\bm{\theta},\hat{\bm{\theta}})\right\\}.$
(39)

Considering a quadratic based utility function:

$\hat{\bm{\theta}}_{I}(\mathbf{g})=\operatorname*{argmin}_{\hat{\bm{\theta}}}\left\\{\int
d\bm{\theta}\int
d\bm{\alpha}p(\bm{\theta},\mathbf{b}(\bm{\alpha})|\mathbf{g},H_{1})\|\hat{\bm{\theta}}-\bm{\theta}\|_{2}^{2}\right\\}.$
(40)

By taking the derivative of Eqn. (40) with respect to ${\hat{\bm{\theta}}}$
and setting the derivative to zero, $\hat{\bm{\theta}}_{I}(\mathbf{g})$ can be
expressed as:

$\hat{\bm{\theta}}_{I}(\mathbf{g})=\iint\bm{\theta}p(\bm{\theta},\mathbf{b}(\bm{\alpha})|\mathbf{g},H_{1})d\bm{\theta}d\bm{\alpha}.$
(41)

$\hat{\bm{\theta}}_{I}(\mathbf{g})$ can be approximated by using Monte Carlo
integration subsequently:

$\hat{\bm{\theta}}_{I}(\mathbf{g})=\frac{1}{J}\sum_{j=1}^{J}\bm{\theta}^{(j)}.$
(42)

Here, $(\bm{\theta}^{(j)},\bm{\alpha}^{(j)})$ are sampled from the posterior
distribution $p(\bm{\theta},\mathbf{b}(\bm{\alpha})|\mathbf{g},H_{1})$. A
Markov chain with the initial parameter vectors
$(\bm{\theta}^{(0)},\bm{\alpha}^{(0)})$ and the proposal densities
$q_{1}(\bm{\theta}|\bm{\theta}^{(j)})$ and
$q_{2}(\bm{\alpha}|\bm{\alpha}^{(j)})$ is constructed to generate
$(\bm{\theta}^{(j)},\bm{\alpha}^{(j)})$, which is described below. Given
$(\bm{\theta}^{(i)},\bm{\alpha}^{(i)})$, candidate parameter vectors
$(\bm{\theta}^{\ast},\bm{\alpha}^{\ast})$ are drawn from the proposal density
and are accepted to the Markov chain with the acceptance probability:

$\displaystyle
p_{a}(\bm{\theta}^{\ast},\mathbf{b}(\bm{\alpha}^{\ast})|\bm{\theta^{(i)}},\mathbf{b}(\bm{\alpha}^{(i)}),\mathbf{g})=$
(43)
$\displaystyle\min\left[1,\frac{p(\mathbf{g}|\bm{\theta}^{\ast},\mathbf{b}(\bm{\alpha}^{\ast}),H_{1})p(\bm{\theta}^{\ast})p(\bm{\alpha}^{\ast})q_{1}(\bm{\theta}^{\ast}|\bm{\theta}^{(i)})q_{2}(\bm{\alpha}^{\ast}|\bm{\alpha}^{(i)})}{p(\mathbf{g}|\bm{\theta}^{(i)},\mathbf{b}(\bm{\alpha}^{(i)}),H_{1})p(\bm{\theta}^{(i)})p(\bm{\alpha}^{(i)})q_{1}(\bm{\theta}^{(i)}|\bm{\theta}^{\ast})q_{2}(\bm{\alpha}^{(i)}|\bm{\alpha}^{\ast})}\right].$

The vectors
$(\bm{\theta}^{(i+1)},\bm{\alpha}^{(i+1)})\equiv(\bm{\theta}^{\ast},\bm{\alpha}^{\ast})$
if the candidates are accepted; otherwise
$(\bm{\theta}^{(i+1)},\bm{\alpha}^{(i+1)})\equiv(\bm{\theta}^{(i)},\bm{\alpha}^{(i)})$.
When a random walk Metropolis-Hastings (RWMH) [19] is used here, the proposal
density can be a simple Gaussian density. Therefore,
$\hat{\bm{\theta}}_{I}(\mathbf{g})$ can be approximated.

## References

  * [1] C. E. Metz, R. F. Wagner, K. Doi, D. G. Brown, R. M. Nishikawa, and K. J. Myers, “Toward consensus on quantitative assessment of medical imaging systems,” _Medical physics_ , vol. 22, no. 7, pp. 1057–1061, 1995.
  * [2] H. H. Barrett, J. Yao, J. P. Rolland, and K. J. Myers, “Model observers for assessment of image quality,” _Proceedings of the National Academy of Sciences_ , vol. 90, no. 21, pp. 9758–9765, 1993.
  * [3] H. H. Barrett and K. J. Myers, _Foundations of image science_. John Wiley & Sons, 2013.
  * [4] X. He and S. Park, “Model observers in medical imaging research,” _Theranostics_ , vol. 3, no. 10, p. 774, 2013.
  * [5] C. K. Abbey and F. Bochud, “Modeling visual detection tasks in correlated image noise with linear model observers,” _Handbook of Medical Imaging_ , vol. 1, pp. 629–654, 2000.
  * [6] E. Clarkson, “Estimation receiver operating characteristic curve and ideal observers for combined detection/estimation tasks,” _JOSA A_ , vol. 24, no. 12, pp. B91–B98, 2007.
  * [7] K. Li, W. Zhou, H. Li, and M. A. Anastasio, “Supervised learning-based ideal observer approximation for joint detection and estimation tasks,” in _Medical Imaging 2021: Image Perception, Observer Performance, and Technology Assessment_ , vol. 11599. International Society for Optics and Photonics, 2021, p. 115990F.
  * [8] M. A. Kupinski, J. W. Hoppin, E. Clarkson, and H. H. Barrett, “Ideal-observer computation in medical imaging with use of markov-chain Monte Carlo techniques,” _JOSA A_ , vol. 20, no. 3, pp. 430–438, 2003.
  * [9] W. Zhou, H. Li, and M. A. Anastasio, “Approximating the ideal observer and Hotelling observer for binary signal detection tasks by use of supervised learning methods,” _IEEE transactions on medical imaging_ , vol. 38, no. 10, pp. 2456–2468, 2019.
  * [10] W. Zhou, H. Li, and M. A. Anastasio, “Approximating the ideal observer for joint signal detection and localization tasks by use of supervised learning methods,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 12, pp. 3992–4000, 2020.
  * [11] A. Wunderlich and B. Goossens, “Nonparametric estimation receiver operating characteristic analysis for performance evaluation on combined detection and estimation tasks,” _Journal of Medical Imaging_ , vol. 1, no. 3, p. 031002, 2014.
  * [12] M. K. Whitaker, E. Clarkson, and H. H. Barrett, “Estimating random signal parameters from noisy images with nuisance parameters: linear and scanning-linear methods,” _Optics express_ , vol. 16, no. 11, pp. 8150–8173, 2008.
  * [13] C.-J. Lee, M. A. Kupinski, and L. Volokh, “Assessment of cardiac single-photon emission computed tomography performance using a scanning linear observer,” _Medical physics_ , vol. 40, no. 1, p. 011906, 2013.
  * [14] H.-W. Tseng, J. Fan, and M. A. Kupinski, “Assessing computed tomography image quality for combined detection and estimation tasks,” _Journal of Medical Imaging_ , vol. 4, no. 4, p. 045503, 2017.
  * [15] L. Zhang, B. Goossens, C. Cavaro-Ménard, P. Le Callet, and D. Ge, “Channelized model observer for the detection and estimation of signals with unknown amplitude, orientation, and size,” _JOSA A_ , vol. 30, no. 11, pp. 2422–2432, 2013.
  * [16] A. Wunderlich, B. Goossens, and C. K. Abbey, “Optimal joint detection and estimation that maximizes ROC-type curves,” _IEEE transactions on medical imaging_ , vol. 35, no. 9, pp. 2164–2173, 2016.
  * [17] M. A. Kupinski, D. C. Edwards, M. L. Giger, and C. E. Metz, “Ideal observer approximation using bayesian classification neural networks,” _IEEE transactions on medical imaging_ , vol. 20, no. 9, pp. 886–899, 2001.
  * [18] E. Clarkson, “Risk analysis, ideal observers, and receiver operating characteristic curves for tasks that combine detection and estimation,” _Journal of Medical Imaging_ , vol. 6, no. 1, p. 015502, 2019.
  * [19] M. Pereyra, P. Schniter, E. Chouzenoux, J.-C. Pesquet, J.-Y. Tourneret, A. O. Hero, and S. McLaughlin, “A survey of stochastic simulation and optimization methods in signal processing,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 10, no. 2, pp. 224–241, 2015.
  * [20] F. O. Bochud, C. K. Abbey, and M. P. Eckstein, “Statistical texture synthesis of mammographic images with clustered lumpy backgrounds,” _Optics express_ , vol. 4, no. 1, pp. 33–43, 1999.
  * [21] P. Rezai, M. J. Pisaneschi, C. Feng, and V. Yaghmai, “A radiologist’s guide to treatment response criteria in oncologic imaging: Functional, molecular, and disease-specific imaging biomarkers,” _American Journal of Roentgenology_ , vol. 201, no. 2, pp. 246–256, 2013.
  * [22] P. Khurd and G. Gindi, “Decision strategies that maximize the area under the LROC curve,” _IEEE transactions on medical imaging_ , vol. 24, no. 12, pp. 1626–1636, 2005.
  * [23] M. A. Kupinski and H. H. Barrett, _Small-animal SPECT imaging_. Springer, 2005, vol. 233.
  * [24] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” _arXiv preprint arXiv:1412.6806_ , 2014.
  * [25] D. Scherer, A. Müller, and S. Behnke, “Evaluation of pooling operations in convolutional architectures for object recognition,” in _International conference on artificial neural networks_. Springer, 2010, pp. 92–101.
  * [26] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, _Deep learning_. MIT press Cambridge, 2016, vol. 1, no. 2.
  * [27] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [28] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin _et al._ , “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
  * [29] W. Zhou and M. A. Anastasio, “Markov-chain Monte Carlo approximation of the ideal observer using generative adversarial networks,” in _Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment_ , vol. 11316. International Society for Optics and Photonics, 2020, p. 113160D.
  * [30] A. Bora, E. Price, and A. G. Dimakis, “Ambientgan: Generative models from lossy measurements,” in _International Conference on Learning Representations_ , 2018.
  * [31] W. Zhou, S. Bhadra, F. J. Brooks, H. Li, and M. A. Anastasio, “Progressively-growing ambientgans for learning stochastic object models from imaging measurements,” in _Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment_ , vol. 11316. International Society for Optics and Photonics, 2020, p. 113160Q.
  * [32] V. A. Kelkar, X. Zhang, J. Granstedt, H. Li, and M. A. Anastasio, “Task-based evaluation of deep image super-resolution in medical imaging,” in _Medical Imaging 2021: Image Perception, Observer Performance, and Technology Assessment_ , vol. 11599. International Society for Optics and Photonics, 2021, p. 115990X.
  * [33] K. Li, W. Zhou, H. Li, and M. A. Anastasio, “Task-based performance evaluation of deep neural network-based image denoising,” in _Medical Imaging 2021: Image Perception, Observer Performance, and Technology Assessment_ , vol. 11599. International Society for Optics and Photonics, 2021, p. 115990L.
  * [34] ——, “Assessing the impact of deep neural network-based image denoising on binary signal detection tasks,” _IEEE Transactions on Medical Imaging_ , 2021\.
  * [35] C. You, Q. Yang, H. Shan, L. Gjesteby, G. Li, S. Ju, Z. Zhang, Z. Zhao, Y. Zhang, W. Cong _et al._ , “Structurally-sensitive multi-scale deep neural network for low-dose ct denoising,” _IEEE Access_ , vol. 6, pp. 41 839–41 855, 2018.
  * [36] V. A. Kelkar, S. Bhadra, and M. A. Anastasio, “Compressible latent-space invertible networks for generative model-constrained image reconstruction,” _IEEE Transactions on Computational Imaging_ , vol. 7, pp. 209–223, 2021\.
  * [37] B. Kelly, T. P. Matthews, and M. A. Anastasio, “Deep learning-guided image reconstruction from incomplete data,” _arXiv preprint arXiv:1709.00584_ , 2017.
  * [38] G. Wang, J. C. Ye, and B. De Man, “Deep learning for tomographic image reconstruction,” _Nature Machine Intelligence_ , vol. 2, no. 12, pp. 737–748, 2020.
  * [39] S. Park, M. A. Kupinski, E. Clarkson, and H. H. Barrett, “Ideal-observer performance under signal and background uncertainty,” in _Biennial International Conference on Information Processing in Medical Imaging_. Springer, 2003, pp. 342–353.