# Gain-assisted quantum heat engine based on electromagnetically induced
transparency

Laraib Niaz  Quantum Optics Lab, Department of Physics, COMSATS University
Islamabad, Pakistan  You-Lin Chuang  Physics Division, National Center for
Theoretical Sciences, Taipei, 10617, Taiwan  Fazal Badshah
<EMAIL_ADDRESS>School of Electrical and Information Engineering,
Hubei University of Automotive Technology, Shiyan, 442000, People’s Republic
of China  Rahmatullah<EMAIL_ADDRESS>Quantum Optics Lab, Department
of Physics, COMSATS University Islamabad, Pakistan

###### Abstract

We propose a scheme to realize a gain-assisted quantum heat engine (QHE) based
on electromagnetically induced transparency (EIT). The QHE consists of a
three-level $\Lambda$-type atomic system that interacts with two thermal
reservoirs and a coupling field. The gain without inversion is induced in the
system via spontaneously generated coherence (SGC) between two lower levels.
The SGC has a significant effect on the system’s dynamics, resulting in an
enhancement of the emission cross-section and spectral brightness of the QHE.

##  I Introduction

The QHE has attracted considerable interest due to its potential applications
as well as for investigating the quantum limit of classical thermodynamics
such as the limit of the Carnot heat engine Kosloff and Levy (2014). In 1824,
Sadi Carnot proposed the idea of an ideal heat engine that operates by
extracting heat from a hot reservoir, converting a portion of it into
mechanical work, and then transferring the remaining heat to a cold reservoir.
Decades later, with the discovery of the maser, Scovil and Schulz-DuBois
introduced the concept of a QHE and demonstrated that its efficiency could be
described by the Carnot efficiency Boukobza and Tannor (2007). Both the Carnot
and the QHE require at least two reservoirs with differing temperatures,
$T_{C}<T_{H}$, where $T_{C}(T_{H})$ is the temperature of cold (hot)
reservoir) and no mechanical work can be done if $T_{C}=T_{H}$. A novel QHE
extracting work even from a single heat bath, i.e., $T_{C}=T_{H}$ was proposed
by Scully and his coworkers Scully  _et al._ (2003). They introduced a small
amount of coherence in a three-level atomic system and initially prepared the
atoms in the superposition of the lower two levels. Since then, there has been
a significant advancement in the implementation of QHEs by utilizing various
techniques such as the optomechanical effectZhang  _et al._ (2014); Gelbwaser-
Klimovsky  _et al._ (2013), squeezed thermal reservoir Roßnagel  _et al._
(2016), and superconducting resonators Hardal  _et al._ (2017). Clearly, this
is significant progress in the field of thermodynamics as it opens up new
avenues for developing efficient heat engines capable of converting heat into
mechanical work with improved efficiency and performance.

The efficiency of the QHE in multilevel atomic systems can be increased via
quantum coherence Uzdin  _et al._ (2015); Dorfman  _et al._ (2013) which is
the main mechanism behind EIT. EIT refers to a process in which the absorption
of a probe field in an optical medium is reduced or suppressed by the presence
of a strong control field. As a result, the system appears transparent to the
probe field, even though it would normally be optically thick. Recently, a new
scheme for QHE was proposed by Harris, based on the concept of EIT Harris
(2016). This proposal was soon confirmed experimentally through the
demonstration of EIT-based QHE in cooled ${}^{85}\text{Rb}$ atoms Zou  _et
al._ (2017). In this model, a three-level $\Lambda$-type atomic system having
a ground, a meta-stable, and an excited state coupled with two thermal
reservoirs and one coupling field. A thermal reservoir at temperature $T_{13}$
interacts with the transition from the ground to the excited state. The second
thermal reservoir at temperature $T_{23}$ and a coupling field are applied to
transition from an excited to a meta-stable state. Initially, a photon is
absorbed from reservoir $T_{13}$, and a photon is generated in $T_{23}$
reservoir and the population of the meta-stable state is increased by one
unit. Following that, a photon from a coupling field is absorbed and a photon
is emitted on the transition from excited to ground level, and the last
transition accounts for the mechanical work. The QHE using EIT is also called
nontraditional QHE because we can take either $T_{13}$=$T_{23}$ or
$T_{13}>T_{23}$ ($T_{13}<T_{23}$). The details working mechanism of EIT-based
QHE is given in Ref. Harris (2016). The performance of nontraditional QHE can
also be improved by applying a microwave field between ground and meta-stable
state Zhang  _et al._ (2019). A recent investigation of QHE in a Doppler-
broadened atomic medium demonstrates that hot atoms can also be used to
construct non-traditional QHE Zhang  _et al._ (2021).

Another type of coherence called SGC can emerge in a $\Lambda$-type atomic
system when two degenerate lower levels possess non-orthogonal dipole moments
Menon and Agarwal (1998). This type of coherence is a result of the quantum
mechanical interaction between the two degenerate states, leading to the
generation of coherence without any external driving fields Menon and Agarwal
(1998). SGC has been observed to produce some fascinating outcomes, including
the elimination of spectral lines Zhu and Scully (1996), enhancement in
quantum interference Yannopapas  _et al._ (2009) and gain without inversion
Zhou and Swain (1997). These results highlight the importance of SGC in the
field of quantum optics and its potential applications in areas such as
spectroscopy and quantum information processing. Furthermore, the study of SGC
can provide new insights into the behavior of quantum systems and inform the
design of quantum devices such as quantum heat engines.

In this paper, we suggest a modified version of the nontraditional QHE by
incorporating SGC between two lower levels. The emission and absorption cross-
sections, which are directly linked to the output field, are found to be
significantly influenced by the SGC coefficient. The use of SGC compensates
for the losses in the EIT medium and improves the spectral brightness and
emission cross-section of the QHE. With the effect of SGC, we demonstrate that
the performance of the QHE can be greatly enhanced compared to traditional
thermal engines. Furthermore, the role of SGC in improving the power output of
the QHE shows its importance in the development of quantum technologies for
energy conversion and power generation.

##  II Atomic Model and Equations

The schematic of the proposed EIT-based QHE model is shown in Fig. 1(a). We
consider a three-level $\Lambda$-type atomic configuration in which $\ket{1}$
$\rightarrow$ $\ket{2}$ transition is meta-stable whose transition frequency
is $\omega_{12}$. A coupling field of frequency $\omega_{\text{c}}$ is applied
between transition $\ket{2}\rightarrow\ket{3}$. The Rabi frequency of the
coupling field is defined as
$\Omega_{\text{c}}=|\vec{d}_{32}|E_{\text{c}}/2\hbar$, where $E_{\text{c}}$ is
the amplitude of the coupling field and $|\vec{d}_{32}|$ is the dipole matrix
element. The incoherent pumping rates ( black-body radiations) $R_{13}$ and
$R_{23}$ at temperature $T_{13}$ and $T_{23}$ interact with $\ket{1}$ to
$\ket{3}$ and $\ket{2}$ to $\ket{3}$ transitions respectively. The quantum
heat cycle can be described as
$\ket{1}\xrightarrow{T_{13}}\ket{3}\xrightarrow{T_{23}}\ket{2}\xrightarrow{\Omega_{\text{c}}}\ket{3}\xrightarrow{\omega}\ket{1}$,
the overall process is as follows: At first, the photon is absorbed from
$T_{13}$ reservoir and electron from ground state $\ket{1}$ excites to the
excited state $\ket{3}$, as a result, a photon is produced on the $T_{23}$
reservoir, therefore the population of state $\ket{2}$ is increased by one
unit. By absorbing a photon from the coupling laser, the atom is excited again
from the meta state $\ket{2}$ to the upper level $\ket{3}$. In the final step,
a photon having $\omega$ is emitted on the $\ket{3}\rightarrow\ket{1}$
transition, which represents the output of the QHE.

Figure 1: (a) The schematic of a three-level atomic system. The atomic
transition $\ket{1}\rightarrow\ket{3}$($\ket{2}\rightarrow\ket{3}$) interacts
with black-body radiations at temperature $T_{13}$($T_{23}$). The
$\ket{2}\rightarrow\ket{3}$ transition is coupled with a monochromatic laser
or control field with a Rabi frequency $\Omega_{\text{c}}$. The SGC is
generated between $\ket{1}\rightarrow\ket{2}$ transition. As a result of the
EIT phenomenon, a strong and spectrally narrow emission is observed in the
$z$-direction. This emission has a temperature that is higher than that of any
of the reservoirs. (b) The non-orthogonal configuration of dipole moments
$\vec{d}_{31}$ and $\vec{d}_{32}$.

Hamiltonian for the given system under the rotating wave and dipole
approximation is given by

$\displaystyle H$ $\displaystyle=$
$\displaystyle\hbar\Delta\omega_{31}\ket{3}\bra{3}+\hbar\Delta\omega_{21}\ket{2}\bra{2}-\frac{\hbar}{2}(\text{g}\ket{3}\bra{1}$
(1) $\displaystyle+\Omega_{\text{c}}\ket{3}\bra{2}+\text{H.c.}),$

where $\Delta\omega_{31}=\omega_{31}-\omega$ and
$\Delta\omega_{21}=\omega_{21}+\omega_{\text{c}}-\omega$ are the detunings of
the radiating and control fields, respectively and $\omega_{31}(\omega_{21})$
is the transition frequency between the levels $\ket{3}$ and $\ket{1}$
($\ket{2}$ and $\ket{1}$). The parameter g is the coupling coefficient between
the emitted field and atomic transition $\ket{3}\rightarrow\ket{1}$. We follow
the density matrix approach as all the decay processes are radiative and apply
the von Neumann equation to get the equation of motions for density matrix
elements.

$\dot{\rho}=-\frac{\dot{\iota}}{\hbar}[H,\rho]+\mathscr{L}(\rho),$ (2)

where $\mathscr{L}(\rho)$ is the Liouvillean operator and defined as

$\displaystyle\mathscr{L}(\rho)$ $\displaystyle=$ $\displaystyle
R_{13}(\sigma_{13}\rho\sigma_{31}+\sigma_{31}\rho\sigma_{13}-(\sigma_{33}-\sigma_{11})\rho)$
(4)
$\displaystyle+R_{23}(\sigma_{23}\rho\sigma_{32}+\sigma_{32}\rho\sigma_{23}-(\sigma_{33}-\sigma_{22})\rho)$
$\displaystyle+\Gamma_{31}(\sigma_{13}\rho\sigma_{31}-\sigma_{33}\rho)$
$\displaystyle+\Gamma_{32}(\sigma_{23}\rho\sigma_{32}-\sigma_{33}\rho).$

Where $\sigma_{ij}=\ket{i}\bra{j}(i.j=1,2,3)$, $\Gamma_{31}$ and $\Gamma_{32}$
are the spontaneous emission rates from the excited state $\ket{3}$ to lower
states $\ket{1}$ and $\ket{2}$, respectively. The pumping rates $R_{13}$ and
$R_{23}$ can be expressed as

$R_{23}=\Gamma_{32}n_{23}=\Gamma_{32}\\{\exp{[\hbar\omega_{23}/k_{b}T_{23}}]-1\\}^{-1},$
(5)
$R_{13}=\Gamma_{31}n_{13}=\Gamma_{31}\\{\exp{[\hbar\omega_{13}/k_{b}T_{13}}]-1\\}^{-1},$
(6)

where $T_{13}$ and $T_{23}$ are the ambient temperatures of the thermal
radiation fields. The $n_{13}$ and $n_{23}$ are the average number of photons
per mode.

The equations of motion for density matrix elements can be written as

$\dot{\rho}_{11}=\Gamma_{31}\rho_{33}+\frac{1}{2}\dot{\iota}\text{g}(\rho_{31}-\rho_{13})+R_{13}(\rho_{33}-\rho_{11}),$
(7)
$\dot{\rho}_{22}=\Gamma_{32}\rho_{33}+\frac{1}{2}\dot{\iota}\Omega_{\text{c}}(\rho_{32}-\rho_{23})+R_{23}(\rho_{33}-\rho_{22}),$
(8) $\displaystyle\dot{\rho}_{33}$ $\displaystyle=$
$\displaystyle-(\Gamma_{31}+\Gamma_{32})\rho_{33}+\frac{1}{2}\dot{\iota}\text{g}(\rho_{13}-\rho_{31})+\frac{1}{2}\dot{\iota}\Omega_{\text{c}}(\rho_{23}-\rho_{32})$
(9) $\displaystyle-R_{23}(\rho_{33}-\rho_{22})-R_{13}(\rho_{33}-\rho_{11}),$
$\displaystyle\dot{\rho}_{12}$ $\displaystyle=$
$\displaystyle-(\frac{1}{2}\gamma_{21}-\dot{\iota}\Delta\omega_{21})\rho_{12}-\frac{1}{2}\dot{\iota}\Omega_{\text{c}}\rho_{13}+\frac{1}{2}\dot{\iota}\text{g}\rho_{32}$
(10) $\displaystyle+\gamma_{s}\rho_{33},$ $\displaystyle\dot{\rho}_{23}$
$\displaystyle=$
$\displaystyle-(\frac{1}{2}\gamma_{32}-\dot{\iota}(\Delta\omega_{31}-\Delta\omega_{21}))\rho_{23}+\frac{1}{2}\dot{\iota}\Omega_{\text{c}}(\rho_{33}-\rho_{22})$
(11) $\displaystyle-\frac{1}{2}\dot{\iota}\text{g}\rho_{21},$
$\displaystyle\dot{\rho}_{13}$ $\displaystyle=$
$\displaystyle-(\frac{1}{2}\gamma_{31}-\dot{\iota}\Delta\omega_{31})\rho_{13}-\frac{1}{2}\dot{\iota}\text{g}(\rho_{33}-\rho_{11})$
(12) $\displaystyle-\frac{1}{2}\dot{\iota}\Omega_{\text{c}}\rho_{12},$

The dephasing rates $\gamma_{21}$, $\gamma_{31}$ and $\gamma_{32}$ are related
to decays $\Gamma_{32}$ and $\Gamma_{31}$. For each transition, the dephasing
rate is defined as

$\begin{split}\gamma_{21}=R_{13}+R_{23},\\\
\gamma_{31}=\Gamma_{31}+\Gamma_{32}+R_{23}+2R_{13},\\\
\gamma_{32}=\Gamma_{31}+\Gamma_{32}+2R_{23}+R_{13}.\\\ \end{split}$ (13)

Here we consider SGC between two ground states $\ket{1}$ and $\ket{2}$
Javanainen (1992); Berman (2005); Shui  _et al._ (2015). The factor
$\gamma_{s}$ in equation (10) represents the coefficient of SGC. The SGC
coherence appears due to the cross-coupling between spontaneous emission paths
$\ket{3}$ $\rightarrow$ $\ket{1}$ and $\ket{3}$ $\rightarrow$ $\ket{2}$ for
non-orthogonal dipole moments. The $\gamma_{s}$ is defined as Menon and
Agarwal (1998)

$\gamma_{s}=\frac{|\vec{d}_{31}.\vec{d}_{13}|}{|\vec{d}_{32}||\vec{d}_{31}|}\sqrt{\Gamma_{31}\Gamma_{32}}/2=\text{p}\sqrt{\Gamma_{31}\Gamma_{32}}/2,$
(14)

here $|\vec{d}_{31}|$ and $|\vec{d}_{32}|$ are the dipole moments and p=
$\cos{\theta}$, where $\theta$ is the angle between these dipole moments, see
Fig. 1(b). SGC effect disappears when these dipole moments are orthogonal
Javanainen (1992); Niu and Gong (2006). In our proposed system, we vary the
values of the parameter p, which is equal to the $\cos{\theta}$. By doing so,
we investigate the impact of different values of p on the dynamics of the QHE.

In the absence of any reflection or scattering to other modes, the spectral
brightness as a function of $z$ an be written as Harris (2016)

$\frac{dB(\omega,z)}{dz}+N[\sigma_{A}\rho_{11}-\sigma_{E}(\rho_{22}+\rho_{33})]B(\omega,z)=\sigma_{E}(\rho_{22}+\rho_{33}).$
(15)

Here, $N$ is a number density, $\sigma_{A}$ and $\sigma_{E}$ are absorption
and emission cross-sections, respectively. The brightness is zero at $z=0$,
and becomes maximum $B_{\text{black}}(\omega)$ at a certain value of $z$,
where all relevant spectral components are fully absorbed. The brightness for
each spectral component can be determined by evaluating the steady state
solution of Eq. (15) ($\frac{dB(\omega,z)}{dz}=0$), as expressed by the
following equation

$B_{\text{black}}(\omega)=\frac{\Lambda\sigma_{E}}{\sigma_{A}-\Lambda\sigma_{E}},$
(16)

here, $\Lambda$ is defined as the ”ratio of the number of atoms in the upper
state to those in the ground state”. In the weak field limit, i.e.,
$\text{g}<\Omega_{c}$, $\Lambda$ can be defined as
$\Lambda={\rho_{22}^{(0)}+\rho_{33}^{(0)}}/{\rho_{11}^{(0)}}$, where
$\rho_{ii}^{(0)}(i=1,2,3)$ is the first solution given in Appendix A. The
absorption $\sigma_{A}$ and emission $\sigma_{E}$ cross sections can be
obtained by solving the density matrix equations (Eqs. (7-12) under steady
state condition. The $\sigma_{A}$ and $\sigma_{E}$ are related to the
imaginary part of the first order solution of $\rho_{13}^{(1)}$ Imamoglu
(1991), i.e.,

$Im(\text{g}^{*}\rho_{13}^{(1)})=-\sigma_{A}\rho_{11}^{(0)}+\sigma_{E}(\rho_{22}^{(0)}+\rho_{33}^{(0)}),$
(17)

where the expression for $\rho^{(1)}_{13}$ is given in appendix B.

Figure 2: (a) Gain profile Im$[\rho^{(0)}_{13}]$ and (b) excited-state
population $\rho^{(1)}_{33}$ is plotted against detuning $\Delta\omega_{31}$
for different values of p. The remaining parameters are
$\Gamma_{31}=10^{7},\Gamma_{32}=6\times 10^{7},\Omega_{\text{c}}=5\times
10^{7},\omega_{13}=4\times 10^{15},\omega_{12}=10^{15},\Delta\omega_{21}=0$,
$T_{23}=5778K$, $T_{13}=3778K$ and $\Lambda=0.019$.

First, we consider the role of SGC on the coherence between the ground state
($|1\rangle$) and the excited state ($|3\rangle$) and the population of the
upper level $|3\rangle$. In Fig. 2, we plot Im$[\rho^{(0)}_{13}]$ and
$\rho^{(1)}_{33}$ against detuning for different values of p. The remaining
parameters are $\Gamma_{31}=10^{7},\Gamma_{32}=6\times
10^{7},\Omega_{\text{c}}=5\times 10^{7},\omega_{13}=4\times
10^{15},\omega_{12}=10^{15},\Delta\omega_{21}=0$, $\Lambda=0.019$ and
$T_{23}=5778K$, $T_{13}=3778K$. The Fig. 2(a) illustrates the gain
(Im$[\rho^{(0)}_{13}]<0$) in the system prior to the generation of the output
field and this gain increases with the increase in SGC. The graph demonstrates
a continuous enhancement in gain at resonance $\Delta\omega_{31}=0$ as the
value of the parameter p increases from $0.1$ to $0.7$. The gain is induced in
the system without inversion as most of the atoms are in the ground state and
with only a few atoms found in the excited state $|3\rangle$ as shown in Fig.
2(b). In fact, even for p=$0.7$, the population of the excited state
$\rho^{(1)}_{33}$ is low as $1.2\times 10^{-3}$, as illustrated with green
curve in Fig. 2(b). The gain that arises due to SGC is crucial for the
operation of the EIT-based QHE. The SGC plays a significant role in enhancing
the emission and spectral brightness of the system by generating coherence
between lower levels.

##  III Absorption and Emission cross-sections

In this section, we analyze the effect of the SGC on the absorption and
emission cross-section. The SGC coherence arises due spontaneous decay rates
$\Gamma_{31}$ and $\Gamma_{32}$ from the excited state $\ket{3}$ to lower
states $\ket{1}$ and $\ket{2}$, see Eq. (14). It only affects the spontaneous
emission of the field. The absorption cross-section $\sigma_{A}$ depends on
the ground state population $\sigma_{A}\rho_{11}^{(0)}$ while emission-cross
$\sigma_{E}$ is related to the excited state population as given in Eq. (17).
The SGC has no effect on the absorption of the medium and most of the atoms
remain in the ground state. We obtain the same result for absorption cross-
section as given in Harris (2016). However, SGC amplifies the generated
spontaneous radiation and significantly changes the emission cross-section.
The expression for emission cross-section is lengthy and complex, making it
unsuitable to present here. It is provided in Eq. 30 of Appendix B. However,
we can write $\sigma_{E}$ as

$\sigma_{E}/\sigma_{0}=\sigma_{EIT}+\sigma_{SGC},$ (18)

where, $\sigma_{0}=(2\text{g}|d_{13}|^{2})/(\epsilon c\hbar\gamma_{13})$,
$\sigma_{EIT}$ is same as given in Harris (2016). The $\sigma_{SGC}$ (see Eq.
30) shows a quadratic enhancement with respect to SGC coefficient
$\gamma_{s}$. This suggests a non-linear dependence between the SGC
coefficient $\gamma_{s}$ and $\sigma_{SGC}$ which shows that the $\gamma_{s}$
has a greater impact on $\sigma_{SGC}$ at higher values.

In Fig. 3, we plot the emission cross section against the detuning for four
different values of p, i.e, p=$0.1$, p=$0.3$, p=$0.5$ and p=$0.7$. The
remaining parameters are the same as given in Fig. 2.

Figure 3: Emission cross section is plotted against the detuning
$\Delta\omega_{31}$ for p=0.1 (blue curve), p=0.3 (purple curve), p=0.5 (red
curve) and p=0.7 (green curve). (b) Normalized emission cross-section vs p
(spontaneously generated factor). Other parameters are the same as given in
Fig. 2.

Our results demonstrate a significant enhancement in the emission cross-
section as the value of p is varied as shown in Fig 3. At a value of p=0.1,
the emission cross-section exhibits a value of $0.56\sigma_{0}$. Upon
increasing the value of p from 0.1, there is a corresponding increase in the
emission cross-section. For example, at p=0.3 the emission cross-section
reaches $\sigma_{0}$ depicted by a purple line. Further increases in p to 0.5
result in a further increase in the emission profile, reaching $1.8\sigma_{0}$
shown by the red curve. Finally, when p=0.7, the peak value of the emission
cross-section approaches $3\sigma_{0}$. The fact that the emission cross-
section can be made greater than 1 through tuning p indicates the presence of
a gain in the system. For further validation, the emission cross-section as a
function of p is plotted 3 (b). An analysis of the normalized emission cross-
section in relation to the parameter p demonstrates a gradual rise in the
emission spectrum from its initial value as p increases. The emission spectrum
attains its peak value of $3\sigma_{0}$ at p=0.7, demonstrating a strong
correlation between the emission cross-section and p, which is very helpful
for enhancing the brightness (output) of QHE.

Figure 4: (a) Normalized brightness is plotted against detuning for different
values of p such as p=0.1 (blue curve), p=0.3 (purple curve), p=0.5 (red
curve), and p=0.7 (green curve). The interpretation of the graphs shows that
for increasing values of p, we get enhancement in the brightness profile. The
inset is the zoomed-in region from $\Delta\omega=-0.5\gamma_{31}$ to
$\Delta\omega=0.5\gamma_{31}$. (b) Normalized spectral brightness is plotted
as a function of p. Other parameters are the same as given in Fig. 2.

##  IV Spectral Brightness

Now, we investigate the spectral brightness which is related to the output
power of the QHE. The results are depicted in Fig. 4(a), where the brightness
spectrum is plotted as a function of detuning $\Delta\omega_{31}$ for four
distinct values of the parameter p: p=0.1, p=0.3, p=0.5, and p=0.7. Fig. 4
shows that an increase in the value of p leads to a sharp peak in the
brightness spectrum. Additionally, the intensity of the spectral brightness
experiences a significant increase as p increases. The spectral brightness
reaches a maximum value of $370n_{13}$ when p=0.7, as indicated in Fig. 4 (a).
Moreover, a graph is plotted to investigate the correlation between normalized
spectral brightness and the parameter p, see Fig. 4 (b). The results indicate
an increase in spectral brightness with a corresponding increase in the value
of p. These results are significant as they demonstrate the potential for a
gain-assisted QHE to produce a large output through the implementation of SGC.
They also provide evidence that by inducing coherence, the performance of a
QHE can be enhanced to a certain extent.

Figure 5: (a) A density plot of spectral brightness is plotted as a function
of two variables, p, and $\Omega_{\text{c}}$. (b) A 3D plot of spectral
brightness is generated against the same variables, p, and
$\Omega_{\text{c}}$, while the remaining parameters are same as given in Fig.
2.

Next In Fig. 5, we analyze the dependence of the spectral brightness on the
parameters p and control field $\Omega_{\text{c}}$. A three-dimensional
density graph and a three-dimensional plot are generated to visualize this
relationship. The results clearly demonstrate that the spectral brightness
increases in tandem with the increase of both p and $\Omega_{\text{c}}$. The
graphical representation provides a comprehensive understanding of the
dependence of spectral brightness on the two parameters. The results from Fig.
5 (a) indicate that the atomic system exhibits its maximum spectral brightness
of $370n_{13}$ when we set p=0.7 and the control field
$\Omega_{\text{c}}=0.7\gamma_{31}$. These results were further supported by
the 3D plot in Fig. 5 (b), which demonstrates that the highest spectral
brightness is achieved at the same values of
$\Omega_{\text{c}}=0.7\gamma_{31}$ and p=0.7. Moreover, these two graphs
support the fact that the brightness and emission cross-section at p = 0, is
consistent with a typical Harris heat engine.

The efficiency of our proposed gain-assisted QHE is the same as EIT-based QHE
described in Ref. Harris (2016). However, due to the effect of the SGC, the
brightness, which is related to the power of the QHE, is greatly enhanced.
These particular QHEs can be extremely important in many different ways. They
might be utilized to move energy in nano-scale devices or to cool down quantum
systems also they can be use in other fields such as quantum sensing, quantum
communication, quantum information processing, and quantum metrology. In our
situation, the typical Harris heat engine’s power can be boosted up to a
certain point, and this power can be used to harness work for potential
applications.

##  V Conclusion

We have proposed a scheme to realize a gain-assisted QHE based on EIT. We
utilized the SGC which is generated between two lower levels due to the cross-
coupling between the two spontaneous pathways from an excited state. The
emission cross-section and brightness strongly depend on the SGC. The SGC
induced gain without inversion in the system and as a result, both the
emission-cross section and brightness are enhanced. In summary, the output
power in our proposed EIT-Based QHE can be increased with the aid of SGC.

##  Appendix A Zero and First-order Solutions

To calculate the absorption and emission cross-sections, the density matrix
equations are solved in a perturbative manner. In the linear regime, we take
$\rho_{ij}=\rho^{(0)}_{ij}+\lambda\rho^{(1)}_{ij}(i,j=1,2,3)$, where $\lambda$
is a small number representing the interaction order. We assume that radiating
field is weak and choose $\text{g}=\lambda\text{g}$ and consider the resonant
interaction of the coupling field with atom $\Delta\omega_{21}=0$. The steady-
state solution $\dot{\rho}_{ij}=0$ of the density matrix elements (Eqs
(7)-(12)) to all order of $\Omega_{c}$ and zero-order of g are given by

$\displaystyle\rho^{(0)}_{11}=\frac{(R_{13}+\Gamma_{31})(R_{23}\gamma_{32}+\Omega_{\text{c}}^{2})}{R_{23}\Gamma_{31}\gamma_{32}+R_{13}\Gamma_{32}(3R_{23}+\Gamma_{32})+(3R_{13}+\Gamma_{31})\Omega_{\text{c}}^{2}},$
(19)
$\displaystyle\rho^{(0)}_{22}=\frac{R_{13}(\Gamma_{32}(R_{23}+\Gamma_{32})+\Omega_{\text{c}}^{2})}{(R_{23}\Gamma_{31}\Gamma_{32}+R_{13}\Gamma_{32}(3R_{23}+\Gamma_{32})+(3R_{13}+\
\Gamma_{31})\Omega{23}^{2})},$ (20)

$\displaystyle\rho^{(0)}_{33}=\frac{R_{13}(R_{23}\gamma_{32}+\Omega_{\text{c}}^{2})}{R_{23}\Gamma_{31}\gamma_{32}+R_{13}\gamma_{32}(3R_{23}+\Gamma_{32})+(3R_{13}+\
\Gamma_{31})\Omega_{\text{c}}^{2}},$ (21)
$\displaystyle\rho^{(0)}_{23}=\frac{-\dot{\iota}R13\Gamma_{32}\Omega_{\text{c}}}{R_{23}\Gamma_{31}\gamma_{32}+R_{13}\Gamma_{32}(3R_{23}+\Gamma_{32})+(3R_{13}+\
\Gamma_{31})\Omega_{\text{c}}^{2}},$ (22)
$\rho^{(0)}_{12}=\frac{2R_{13}\gamma_{s}(\gamma_{31}-2\dot{\iota}\Delta\omega_{31})(R_{23}\gamma_{32}+\
\Omega_{\text{c}}^{2})}{((\gamma_{21}-2\dot{\iota}\Delta\omega_{31})(\gamma_{31}-2\dot{\iota}\Delta\omega_{31})+\Omega_{\text{c}}^{2})(R_{23}\Gamma_{31}\gamma_{32}+R_{13}\gamma_{32}(3R_{23}+\Gamma_{32})+(3R_{13}+\
\Gamma_{31})\Omega_{\text{c}}^{2})},$ (23)
$\rho^{(0)}_{13}=\frac{-2\dot{\iota}R_{13}\gamma_{s}\Omega_{\text{c}}(R_{23}\gamma_{32}+\
\Omega_{\text{c}}^{2})}{((\gamma_{21}-2\dot{\iota}\Delta\omega_{31})(\gamma_{31}-2\dot{\iota}\Delta\omega_{31})+\Omega_{\text{c}}^{2})(R23\
\Gamma_{31}\gamma_{32}+R_{13}\gamma_{32}(3R_{23}+\Gamma_{32})+(3R_{13}+\
\Gamma_{31})\Omega_{\text{c}}^{2})}.$ (24)

Express $\rho^{(0)}_{13}$ and $\rho^{(0)}_{12}$ in terms of $\rho^{(0)}_{33}$
to facilitate the calculation of emission cross-section and spectral
brightness.

$\displaystyle\rho^{(0)}_{12}=\frac{2\gamma_{s}\rho^{(0)}_{33}(\gamma_{31}-2\dot{\iota}\Delta\omega_{31})}{[(\gamma_{31}-2\dot{\iota}\Delta\omega_{31})(\gamma_{21}-2\dot{\iota}\Delta\omega_{31})+\Omega_{\text{c}}]},$
(25)
$\displaystyle\rho^{(0)}_{13}=\frac{-2\dot{\iota}\gamma_{s}\rho^{(0)}_{33}\Omega_{\text{c}}}{[(\gamma_{31}-2\dot{\iota}\Delta\omega_{31})(\gamma_{21}-2\dot{\iota}\Delta\omega_{31})+\Omega_{\text{c}}]},$
(26)

The steady-state solution for $\rho_{33}$ and $\rho_{13}$ to all order of
$\Omega_{c}$ and first order of g are obtained as

$\displaystyle\rho^{(1)}_{33}=\rho^{(0)}_{33}+\frac{(\rho^{(0)}_{21}+\rho^{(0)}_{12})R_{13}\Omega_{\text{c}}\text{g}+\dot{\iota}(\rho^{(0)}_{13}-\rho^{(0)}_{31})(R_{23}\gamma_{32}+\Omega_{\text{c}}^{2})\text{g}}{2\gamma_{32}(R_{23}\Gamma_{31}+R_{13}(3R_{23}+\Gamma_{32}))+2(3R_{13}+\Gamma_{31})\Omega_{\text{c}}^{2}},$
(27)
$\displaystyle\rho^{(1)}_{13}=\frac{\dot{\iota}\text{g}\Omega_{\text{c}}^{2}(\rho^{(0)}_{22}-\rho^{(0)}_{33})+2\dot{\iota}\Omega_{\text{c}}\gamma_{s}\gamma_{32}\rho^{(1)}_{33}+\dot{\iota}\gamma_{32}(\gamma_{21}-2\dot{\iota}\Delta\omega_{31})\text{g}(\rho^{(0)}_{33}-\rho^{(0)}_{11})}{[(\gamma_{31}-2\dot{\iota}\Delta\omega_{31})(\gamma_{21}-2\dot{\iota}\Delta\omega_{31})+\Omega_{\text{c}}^{2}]}.$
(28)

##  Appendix B Expression for Emission Cross-Section

The detailed and step-by-step calculation for absorption and emission cross-
sections without SGC can be found in Ref. Imamoglu (1991). Here, these cross-
sections can be obtained from the imaginary part of $\rho^{(1)}_{13}$, see Eq.
(17). By substituting Eqs. (25)-(27) in Eq. (28) and and performing algebraic
manipulations, one can express Im$[\rho^{(1)}_{13}]$ as a function of the
diagonal terms, $\rho^{(0)}_{11}$, $\rho^{(0)}_{22}$ and $\rho^{(0)}_{33}$.
The coefficient of $\rho^{(0)}_{11}$ gives the absorption cross-section which
is the same as given in Harris (2016). The coefficient associated with
$(\rho_{22}^{(0)}+\rho_{33}^{(0)})$ corresponds to the emission cross-section
that consists of two parts. The first part represents the emission of EIT-
based QHE as described in the work of Harris Harris (2016). The second part
represents the emission cross-section incorporating the effect of SGC. The
combination of these two emission cross-sections provides the overall
expression for gain-assisted QHE based on EIT.

$\sigma_{E}/\sigma_{0}=\sigma_{EIT}+\sigma_{SGC},$ (29)

where, $\sigma_{0}=(2\text{g}|d_{13}|^{2})/(\epsilon c\hbar\gamma_{13})$ and

$\displaystyle\sigma_{SGC}=\frac{\gamma_{31}}{\text{g}}\left[\gamma_{s}^{2}\left(2\Omega_{\text{c}}\left(\left(R_{23}\gamma_{32}+\Omega_{\text{c}}^{2}\right)\left(\text{g}\Omega_{\text{c}}\mathscr{A}+\text{g}\Omega_{\text{c}}\left(\mathscr{B}+\mathscr{C}\right)\right)\right)\right)+\gamma_{s}\mathscr{D}\right],$
(30)

where

$\mathscr{A}=(R_{13}\gamma_{31}+R_{23}\gamma_{32}+\Omega_{\text{c}}^{2})((\gamma_{21}^{2}+4\Delta\omega^{2}_{31})(\gamma_{31}^{2}+4\Delta\omega^{2}_{31})+2(\gamma_{21}\gamma_{31}-4\Delta\omega^{2}_{31})\Omega_{\text{c}}^{2}+\Omega_{\text{c}}^{4}),$

$\mathscr{B}=(R_{23}\gamma_{32}+\Omega_{\text{c}}^{2})(\gamma_{21}\gamma_{31}-2(\gamma_{21}+\gamma_{31})\Delta\omega_{31}-4\Delta\omega^{2}_{31}+\Omega_{\text{c}}^{2})(\gamma_{21}\gamma_{31}+2(\gamma_{21}+\gamma_{31})\Delta\omega_{31}-4\Delta\omega^{2}_{31}+\Omega_{\text{c}}^{2}),$
$\displaystyle\mathscr{C}$ $\displaystyle=$ $\displaystyle
R_{13}(\gamma_{21}^{2}\gamma_{31}^{3}-12\gamma_{21}^{2}\gamma_{31}\Delta\omega^{2}_{31}-24\gamma_{21}\gamma_{31}^{2}\Delta\omega^{2}_{31}-4\gamma_{31}^{3}\Delta\omega^{2}_{31}+32\Gamma_{21}\Delta\Omega^{4}_{31}+48\Gamma_{31}\Delta\omega^{4}_{31}$
$\displaystyle+2(\gamma_{21}\gamma_{31}^{2}-4(\gamma_{21}+2\gamma_{31})\Delta\omega^{2}_{31})\Omega_{\text{c}}^{2}+\gamma_{31}\Omega_{\text{c}}^{4}),$
$\mathscr{D}=\frac{2\Omega_{\text{c}}(R_{23}\gamma_{32}+\Omega_{\text{c}}^{2})(\gamma_{21}\gamma_{31}-4\Delta\omega^{2}_{31}+\Omega_{\text{c}}^{2})}{(2R_{23}\gamma_{32}+\gamma_{32}\gamma_{32}+2\Omega_{\text{c}}^{2})((\gamma_{21}^{2}+4\Delta\omega^{2}_{31})(\gamma_{31}^{2}+4\Delta\omega^{2}_{31})+2(\gamma_{21}\gamma_{31}-4\Delta\omega^{2}_{31})\Omega_{\text{c}}^{2}+\Omega_{\text{c}}^{4})}.$

The expression for brightness is complex and is not provided in this paper,
but it can be determined by utilizing Eq. 16.

## Acknowledgments

The research presented in this work has been conducted under the support of
the Higher Education Commission (HEC) of Pakistan’s National Research Program
for Universities (NRPU) funded project entitled ”Realization of Optical
Devices using Quantum Dots Rydberg Atoms and 2D Graphene structure” with grant
agreement No. 10692/Balochistan/NRPU/R and D/HEC/2017.

## References

  * Kosloff and Levy (2014) R. Kosloff and A. Levy, Annual Review of Physical Chemistry 65, 365 (2014).
  * Boukobza and Tannor (2007) E. Boukobza and D. Tannor, Physical review letters 98, 240601 (2007).
  * Scully  _et al._ (2003) M. O. Scully, M. S. Zubairy, G. S. Agarwal, and H. Walther, Science 299, 862 (2003).
  * Zhang  _et al._ (2014) K. Zhang, F. Bariani, and P. Meystre, Physical review letters 112, 150602 (2014).
  * Gelbwaser-Klimovsky  _et al._ (2013) D. Gelbwaser-Klimovsky, R. Alicki, and G. Kurizki, Physical Review E 87, 012140 (2013).
  * Roßnagel  _et al._ (2016) J. Roßnagel, S. T. Dawkins, K. N. Tolazzi, O. Abah, E. Lutz, F. Schmidt-Kaler, and K. Singer, Science 352, 325 (2016).
  * Hardal  _et al._ (2017) A. Ü. Hardal, N. Aslan, C. Wilson, and Ö. E. Müstecaplıoğlu, Physical Review E 96, 062120 (2017).
  * Uzdin  _et al._ (2015) R. Uzdin, A. Levy, and R. Kosloff, Physical Review X 5, 031044 (2015).
  * Dorfman  _et al._ (2013) K. E. Dorfman, D. V. Voronine, S. Mukamel, and M. O. Scully, Proceedings of the National Academy of Sciences 110, 2746 (2013).
  * Harris (2016) S. Harris, Physical Review A 94, 053859 (2016).
  * Zou  _et al._ (2017) Y. Zou, Y. Jiang, Y. Mei, X. Guo, and S. Du, Physical Review Letters 119, 050602 (2017).
  * Zhang  _et al._ (2019) X.-J. Zhang, Z.-H. Shi, H.-H. Wang, Z.-H. Kang, and J.-H. Wu, Physical Review A 99, 033817 (2019).
  * Zhang  _et al._ (2021) X.-J. Zhang, G. La Rocca, M. Artoni, H.-H. Wang, and J.-H. Wu, Physical Review A 103, 062205 (2021).
  * Menon and Agarwal (1998) S. Menon and G. Agarwal, Physical Review A 57, 4014 (1998).
  * Zhu and Scully (1996) S.-Y. Zhu and M. O. Scully, Physical review letters 76, 388 (1996).
  * Yannopapas  _et al._ (2009) V. Yannopapas, E. Paspalakis, and N. V. Vitanov, Physical review letters 103, 063602 (2009).
  * Zhou and Swain (1997) P. Zhou and S. Swain, Physical review letters 78, 832 (1997).
  * Javanainen (1992) J. Javanainen, EPL (Europhysics Letters) 17, 407 (1992).
  * Berman (2005) P. Berman, Physical Review A 72, 035801 (2005).
  * Shui  _et al._ (2015) T. Shui, Z. Wang, and B. Yu, JOSA B 32, 210 (2015).
  * Niu and Gong (2006) Y. Niu and S. Gong, Physical Review A 73, 053811 (2006).
  * Imamoglu (1991) A. Imamoglu,  _Lasers without inversion: theory and experiment_ (Stanford University, 1991).


# Improving Object Detection in Medical Image Analysis through Multiple Expert
Annotators: An Empirical Investigation

Hieu H. Pham
Coordinated Science Laboratory, UIUC
VinUni-Illinois Smart Health Center & CECS, VinUniversity
<EMAIL_ADDRESS>Khiem H. Le
VinUniversity
<EMAIL_ADDRESS>Tuan V. Tran
VinUniversity
<EMAIL_ADDRESS>Ha Q. Nguyen
Vingroup Big Data Institute
<EMAIL_ADDRESS>
##  1 Introduction

The work discusses the use of machine learning algorithms for anomaly
detection in medical image analysis and how the performance of these
algorithms depends on the number of annotators and the quality of labels. To
address the issue of subjectivity in labeling with a single annotator, we
introduce a simple and effective approach that aggregates annotations from
multiple annotators with varying levels of expertise. We then aim to improve
the efficiency of predictive models in abnormal detection tasks by estimating
hidden labels from multiple annotations and using a re-weighted loss function
to improve detection performance. Our method is evaluated on a real-world
medical imaging dataset and outperforms relevant baselines that do not
consider disagreements among annotators. This work111This is a short version
submitted to the Midwest Machine Learning Symposium (MMLS 2023), Chicago, IL,
USA. has been accepted for publication by IEEE Access and its full form can be
found in [7]. We also released the dataset and the code used in this
study222https://vindr.ai/datasets/cxr..

##  2 Learning from multiple annotators

Existing works that are highly related to our work, including learning from
multiple annotators [35, 25, 5, 29, 8, 33, 24, 6, 4, 27, 31, 21, 20, 22, 34,
29] and weighted training techniques [3, 26, 23, 9, 2, 32, 1]. Unlike any
approaches above, we aggregated annotations from multiple annotators and
propose a re-weighted loss function that assigns more weights to more
confident examples that determine by the consensus of multiple annotators.
Figure 1 shows an overview of our method. Specifically, we first estimate the
actual labels using WBF algorithm [28]. They are then used to train a typical
object detector with a re-weighted loss function. Note a general form of the
loss function for those detectors can be written as

$\displaystyle\begin{aligned}
\mathcal{L}\left(p,p^{*},t,t^{*}\right)&=\mathcal{L}_{cls}\left(p,p^{*}\right)+\beta
I(t)\mathcal{L}_{loc}\left(t,t^{*}\right)\\\
I(t)&=\left\\{\begin{array}[]{ll}1&\text{if}\operatorname{IoU}\left\\{a,a^{*}\right\\}>\eta\\\
0&\text{ otherwise}.\end{array}\right.\end{aligned}$ (1)

where $t$ and $t^{*}$ are the predicted and ground truth box coordinates, $p$
and $p^{*}$ are the class category probabilities, respectively;
$\operatorname{IoU}\left\\{a,a^{*}\right\\}$ denotes the Intersection over
Union (IoU) between the anchor $a$ and its ground truth $a^{*}$; $\eta$ is an
IoU threshold for objectness, i.e. the confidence score of whether there is an
object or not; $\beta$ is a constant for balancing two loss terms
$\mathcal{L}_{cls}$ and $\mathcal{L}_{loc}$ [36]. We use fused boxes
confidence scores $c^{i}_{k}$ obtained from WBF algorithm [28] to get a re-
weighted loss function that emphasizes boxes with high annotators agreement.
The new loss function can now be written as

$\mathcal{L}\left(p,p^{*},t,t^{*}\right)=c\mathcal{L}_{cls}\left(p,p^{*}\right)+c\beta
I(t)\mathcal{L}_{loc}\left(t,t^{*}\right).$ (2)

Figure 1: Illustration of the proposed approach.

##  3 Experiments and Results

We validated our method on VinDr-CXR [11], a real-world chest X-ray dataset
with labels provided by multiple radiologists for a typical task medical
imaging [13, 30, 17, 14, 18, 10, 12, 15, 19, 16]. It consists of 18,000 chest
X-ray scans, with 15,000 for training and 3,000 for testing sets. We compared
the performance of the proposed method against (1) Baseline using all
annotations as the ground truth; models trained on annotations provided by
each annotator independently (Annotators #1, #2, #3) and (3) an ensemble of
independent models trained on separate radiologists’ annotation sets. Table 1
reports the experimental results in<EMAIL_ADDRESS>score. Our method outperforms the
baselines, individual models, and even the ensemble of individual experts’
models. Experimental results validated the effectiveness of the proposed
method.

Table 1:  Experimental results on the VinDr-CXR dataset.
Method |<EMAIL_ADDRESS>
---|---
Baseline | 0.148
Annotator #1 | 0.121
Annotator #2 | 0.132
Annotator #3 | 0.124
Ensemble of all annotators | 0.154
Ours | 0.158

##  4 Discussions and Conclusion

The proposed method is the first effort to train an image detector from labels
provided by multiple annotators. We empirically showed a notable improvement
in terms of mAP scores by estimating the true labels and then integrating the
implicit annotators’ agreement into the loss function to emphasize the
accurate bounding boxes over the imprecise ones. The idea is simple but
effective, allowing the overall framework can be applied in training any
machine learning-based detectors. However, the overall architecture is not
end-to-end. It may not fully exploit the benefits of combining truth inference
and training the desired image detector.

## References

  * [1] Devansh Arpit, Stanisław Jastrzundefinedbski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien.  A closer look at memorization in deep networks.  In Proceedings of the 34th International Conference on Machine Learning, ICML’17, page 233–242. JMLR.org, 2017.
  * [2] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 41–48, New York, NY, USA, 2009. Association for Computing Machinery.
  * [3] N. Dalal and B. Triggs.  Histograms of oriented gradients for human detection.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, pages 886–893 vol. 1, 2005.
  * [4] A. Dawid and A. Skene.  Maximum likelihood estimation of observer error‐rates using the em algorithm.  Journal of The Royal Statistical Society Series C-applied Statistics, 28:20–28, 1979.
  * [5] Yuan Jin, Mark Carman, Ye Zhu, and Yong Xiang.  A technical survey on statistical modelling and design methods for crowdsourcing quality control.  Artificial Intelligence, page 103351, 2020.
  * [6] David Karger, Sewoong Oh, and Devavrat Shah.  Iterative learning for reliable crowdsourcing systems.  Advances in Neural Information Processing Systems, 24, 2011.
  * [7] Khiem H Le, Tuan V Tran, Hieu H Pham, Hieu T Nguyen, Tung T Le, and Ha Q Nguyen.  Learning from multiple expert annotators for enhancing anomaly detection in medical image analysis.  IEEE Access, 11:14105–14114, 2023.
  * [8] Jingzheng Li, Hailong Sun, Jiyi Li, Zhijun Chen, Renshuai Tao, and Yufei Ge.  Learning from multiple annotators by incorporating instance features, 2021\.
  * [9] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2980–2988, 2017.
  * [10] Huy Ngoc Nguyen, Hieu Pham, Thanh Tran, Tuan Nguyen, and Quy Ha Nguyen.  Vindr-pcxr: An open, large-scale chest radiograph dataset for interpretation of thoracic diseases in children.  medRxiv, pages 2022–03, 2022.
  * [11] Ha Q Nguyen, Khanh Lam, Linh T Le, Hieu H Pham, Dat Q Tran, Dung B Nguyen, Dung D Le, Chi M Pham, Hang TT Tong, Diep H Dinh, et al.  VinDr-CXR: An open dataset of chest X-rays with radiologist’s annotations.  arXiv preprint arXiv:2012.15029, 2020.
  * [12] Hieu Trung Nguyen, Ha Quy Nguyen, Hieu Huy Pham, Khanh Lam, Linh Tuan Le, Minh Dao, and Van Vu.  Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography.  MedRxiv, pages 2022–03, 2022.
  * [13] Hieu T Nguyen, Hieu H Pham, Nghia T Nguyen, Ha Q Nguyen, Thang Q Huynh, Minh Dao, and Van Vu.  Vindr-spinexr: A deep learning framework for spinal lesions detection and classification from radiographs.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24, pages 291–301. Springer, 2021.
  * [14] Ngoc Huy Nguyen, Ha Quy Nguyen, Nghia Trung Nguyen, Thang Viet Nguyen, Hieu Huy Pham, and Tuan Ngoc-Minh Nguyen.  A clinical validation of vindr-cxr, an ai system for detecting abnormal chest radiographs.  arXiv preprint arXiv:2104.02256, 2021.
  * [15] Ngoc Huy Nguyen, Ha Quy Nguyen, Nghia Trung Nguyen, Thang Viet Nguyen, Hieu Huy Pham, and Tuan Ngoc-Minh Nguyen.  Deployment and validation of an ai system for detecting abnormal chest radiographs in clinical settings.  Frontiers in Digital Health, page 130, 2022.
  * [16] Thao Nguyen, Tam M Vo, Thang V Nguyen, Hieu H Pham, and Ha Q Nguyen.  Learning to diagnose common thorax diseases on chest radiographs from radiology reports in vietnamese.  Plos one, 17(10):e0276545, 2022.
  * [17] Hieu H Pham, Dung V Do, and Ha Q Nguyen.  Dicom imaging router: An open deep learning framework for classification of body parts from dicom x-ray scans.  medRxiv, 2021.
  * [18] Hieu H Pham, Tung T Le, Dat Q Tran, Dat T Ngo, and Ha Q Nguyen.  Interpreting chest X-rays via CNNs that exploit hierarchical disease dependencies and uncertainty labels.  Neurocomputing, 437:186–194, 2021.
  * [19] Hieu H Pham, Ha Q Nguyen, Hieu T Nguyen, Linh T Le, and Lam Khanh.  An accurate and explainable deep learning system improves interobserver agreement in the interpretation of chest radiograph.  arXiv preprint arXiv:2208.03545, 2022.
  * [20] Anil Ramakrishna, Rahul Gupta, Ruth B Grossman, and Shrikanth S Narayanan.  An expectation maximization approach to joint modeling of multidimensional ratings derived from multiple annotators.  In Interspeech, pages 1555–1559, 2016.
  * [21] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Anna Jerebko, Charles Florin, Gerardo Hermosillo Valadez, Luca Bogoni, and Linda Moy.  Supervised learning from multiple experts: Whom to trust when everyone lies a bit.  In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 889–896, New York, NY, USA, 2009. Association for Computing Machinery.
  * [22] Filipe Rodrigues and Francisco Pereira.  Deep learning from crowds.  In Proceedings of the Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence, volume 32, 2018\.
  * [23] Robert E Schapire.  Explaining AdaBoost.  In Empirical Inference, pages 37–52. Springer, 2013.
  * [24] Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis.  Get another label? Improving data quality and data mining using multiple, noisy labelers.  In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’08, page 614–622, New York, NY, USA, 2008. Association for Computing Machinery.
  * [25] Victor S. Sheng and Jing Zhang.  Machine learning with crowdsourcing: A brief summary of the past research and future directions.  Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):9837–9843, Jul. 2019.
  * [26] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick.  Training region-based object detectors with online hard example mining.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 761–769, 2016.
  * [27] Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi.  Inferring ground truth from subjective labelling of venus images.  In Proceedings of the 7th International Conference on Neural Information Processing Systems, NIPS’94, page 1085–1092, Cambridge, MA, USA, 1994. MIT Press.
  * [28] Roman Solovyev, Weimin Wang, and Tatiana Gabruseva.  Weighted boxes fusion: Ensembling boxes from different object detection models.  Image and Vision Computing, 107:104117, Mar 2021.
  * [29] Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silberman.  Learning from noisy labels by regularized estimation of annotator confusion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11244–11253, 2019.
  * [30] Thanh T Tran, Hieu H Pham, Thang V Nguyen, Tung T Le, Hieu T Nguyen, and Ha Q Nguyen.  Learning to automatically diagnose multiple diseases in pediatric chest radiographs using deep convolutional neural networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3314–3323, 2021.
  * [31] Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo.  Whose vote should count more: Optimal integration of labels from labelers of unknown expertise.  Advances in Neural Information Processing Systems, 22:2035–2043, 2009.
  * [32] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  Communications of the ACM, 64(3):107–115, 2021.
  * [33] Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Chen Jin, Joseph Jacob, Olga Cicarrelli, Frederik Barkhof, and Daniel Alexander.  Disentangling human error from ground truth in segmentation of medical images.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15750–15762. Curran Associates, Inc., 2020.
  * [34] Le Zhang, Ryutaro Tanno, Mou-Cheng Xu, Chen Jin, Joseph Jacob, Olga Cicarrelli, Frederik Barkhof, and Daniel Alexander.  Disentangling human error from ground truth in segmentation of medical images.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15750–15762. Curran Associates, Inc., 2020.
  * [35] Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng.  Truth inference in crowdsourcing: Is the problem solved?  Proc. VLDB Endow., 10(5):541–552, Jan. 2017.
  * [36] Zhengxia Zou, Zhenwei Shi, Yuhong Guo, and Jieping Ye.  Object detection in 20 years: A survey.  arXiv preprint arXiv:1905.05055, 2019.


# Fidelity of Ensemble Aggregation for Saliency Map Explanations using
Bayesian Optimization Techniques

Yannik Mahlau111Corresponding Author. This work was created as part of an
internship at dSPACE GmbH.
Leibniz Universität Hannover
<EMAIL_ADDRESS>Christian Nolde
dSPACE GmbH

(July, 2022)

###### Abstract

In recent years, an abundance of feature attribution methods for explaining
neural networks have been developed. Especially in the field of computer
vision, many methods for generating saliency maps providing pixel attributions
exist. However, their explanations often contradict each other and it is not
clear which explanation to trust. A natural solution to this problem is the
aggregation of multiple explanations. We present and compare different pixel-
based aggregation schemes with the goal of generating a new explanation, whose
fidelity to the model’s decision is higher than each individual explanation.
Using methods from the field of Bayesian Optimization, we incorporate the
variance between the individual explanations into the aggregation process.
Additionally, we analyze the effect of multiple normalization techniques on
ensemble aggregation.

##  1 Introduction

Neural Networks in computer vision applications, like the popular ResNet18
[12] architecture, often have millions of parameters and are therefore
inherently difficult to interpret. This is a big deterrent for the use of
neural networks in real world applications [23], as understanding is vital for
creating trust in the machine learning model [22]. Therefore, the topic of
explainable machine learning has drawn a lot of attention in the research
community [26]. Even though we focus on the application of neural networks in
computer vision tasks, the same is true for all fields of machine learning
involving neural nets. Joshi et al. [16] provide an overview of explainability
in different machine learning fields. As a result of this interest, a lot of
different explanation schemes have been developed. However, their application
in real-world tasks is challenging, because it is not clear which explanation
represents the models decision best. The quality of an explanation regarding
the models decision is called fidelity [23].

A comparison between explanations is problematic, because no ground truth data
exists for the models decision process. Additionally, visual comparison by
humans has been proven to be precarious [1] since humans inherently have a
confirmation bias of their own belief. As a result, various metrics for
measuring fidelity have been developed [41]. Most commonly used are the
insertion and deletion metric [25], even though valid criticism regarding the
issue of baseline choice and implicit independence assumptions can be voiced
[37].

Instead of developing more algorithms for generating individual explanation,
we believe it is pertinent to use multiple explanations and aggregate them
into a new and better explanation. For the aggregation of multiple
explanations, we use acquisition functions from the field of Bayesian
Optimization as they are a tool for estimating the importance of a data point
(pixel). In contrast to Bayesian Optimization, we utilize acquisition
functions for identifying data points with low disagreement (low standard
deviation). That is, because a pixel is most important if all individual
explanations agree on its importance. One further difference to Bayesian
Optimization is the absence of a sequential process, because the individual
explanations yield fixed data. Therefore, we only evaluate the acquisition
function once for every pixel.

The main contributions of our work are:

  * •

We introduce different aggregation functions for ensemble explanations.
Empirically, we evaluate their fidelity on two different datasets.

  * •

For aggregation methods with adjustable hyperparameters, we test their
sensitivity.

  * •

We compare different normalization techniques and empirically stress their
importance.

##  2 Related Work

In recent years, a number of pixel-attribution (saliency map) methods for
explaining computer vision classification models have been developed. They can
be broadly categorized into model-agnostic (black-box) and model-specific
(white-box) methods [23].

Model-agnostic interpretation schemes do not consider the internal structure
of a model, but only the relation of input and output. In RISE [25], for
example, part of the input image is occluded and the resulting change in
output probability is observed.

In contrast, model-specific methods make use of the internal activations,
weights and gradients of a model. Therefore, these methods have more
information available, but are also not universally applicable anymore.
GradCAM [31], for instance, linearly weighs the activations of the last
convolutional layers with the gradient of fully connected layers. In the
extension PolyCAM [8], all activations of convolutional layers are aggregated
instead of just the last layer. SmoothGrad [32] follows a different idea by
averaging the gradients of the model output with regard to input images
perturbed by small random noise. Integrated Gradients [36] does not only
consider a single gradient per image, but rather integrates all gradients
along an interpolation path from a baseline image to the original. Integrated
Gradients and SmoothGrad can be combined to form Expected Gradients [9].
Integrated Gradient Optimized Saliency (iGOS++) [17] utilizes the integrated
gradients to optimize a soft mask using stochastic gradient descent.

Figure 1: Comparison of base attribution methods for ResNet18 trained on
ImageNet. Attributions are linearly normalized to the full color range.

In Figure 1, we display the attributions of all mentioned methods on an
example image. We normalize the attributions to the full color range for
better visualization in all figures throughout this work. A common observation
comparing different saliency methods is that they disagree with each other.
Krishna et al. [19] developed multiple metrics measuring this disagreement.
The question arises, if a combination of multiple methods may produce a better
saliency map than each individual one. Bhatt et al. [4] developed aggregation
schemes minimizing explanation sensitivity by using Shapley Values.
Additionally, they minimized explanation complexity by gradient-descent and
region shrinking.

Rieger et al. [27] proved that an average of multiple attribution methods is
more robust to adversarial attacks than a single explanation. Incorporating
confidence, this method can also be used to enhance fidelity [28].
Correspondingly, a Restricted Boltzmann Machine [33] trained on an ensemble of
explanations, using unsupervised contrastive divergence learning [13], is
robust against noisy and unfaithful explanations [5]. Additionally, it is
possible to achieve high fidelity using this ensemble aggregation.

##  3 Method

In the context of computer vision, we consider a machine learning model for
classification $F:\mathbb{R}^{m\times n\times d}\rightarrow\mathbb{R}^{|C|}$,
which maps an input image $I\in\mathbb{R}^{m\times n\times d}$ to a
probability vector representing the prediction for each class $c\in C$. The
input image has a spatial resolution of $m\times n$ pixels with $d$ color
channels.

An explanation of a single prediction $e(F,I,c)\in\mathbb{R}^{m\times n}$
yields an attribution score $e_{i,j}$ for every pixel $(i,j)$ of the input
image. For better readability, we omit the parameters $F$, $I$ and $c$. An
ensemble $E=\\{e^{1},...,e^{k}\\}$ is a set of attributions. We generate the
members $e^{i}$ of an ensemble using different attribution methods and call
them base attributions. The base attributions of an ensemble can be merged
into a new explanation $\hat{e}\in\mathbb{R}^{m\times n}$ using an aggregation
function $g:E\mapsto\hat{e}$. We present multiple aggregation functions in
Section 3.2.

Note that our methods could easily be extended to object detection. However,
we only consider image classification as there are many more base attribution
methods available.

###  3.1 Normalization

Normalization of the base attributions is necessary, because they do not share
the same scale. For example, the pixel attributions of RISE represent a
probability difference in the interval $[0,1]$, but the attributions of
PolyCAM are a product of activations, which can take arbitrary (real) positive
values. Conversely, the gradients of SmoothGrad can theoretically be any real
number, but are practically often small values around zero. If one would
aggregate an ensemble without normalization, the high attribution values of
PolyCAM would dominate the small values of SmoothGrad. Images are often
linearly mapped to the interval $[0,1]$ to utilize the full color or greyscale
range [11]. Equivalently, we can compute a normalized attribution (see
Equation 1). Another normalization technique is Z-score normalization [18].
Here, we assume that the distribution of the attribution is Gaussian with mean
$\mu(e)$ and standard deviation $\sigma(e)$ over the pixels values. This
distribution can then be normalized to a standard normal distribution by
Equation 2. Additionally, we experimented with the $l_{1}$\- and
$l_{2}$-normalization (see Equation 3 and 4). This gives us the following
normalization methods:

$\displaystyle\tilde{e}_{lin}$
$\displaystyle=\frac{e-\min(e)}{\max(e)-\min(e)},$ (1)
$\displaystyle\tilde{e}_{z}$ $\displaystyle=\frac{e-\mu(e)}{\sigma(e)},$ (2)

$\displaystyle\tilde{e}_{1}$ $\displaystyle=\frac{e}{\lVert e\rVert_{1}},$ (3)
$\displaystyle\tilde{e}_{2}$ $\displaystyle=\frac{e}{\lVert e\rVert_{2}}.$ (4)

Figure 2 displays the average attribution (i.e. mean of the normalized base
attributions) using the presented normalization techniques. See Equation 5 for
more details on average aggregation. As in all figures, the displayed
aggregated attributions are linearly normalized to the full color range for
visualization purposes. Without normalization, PolyCAM dominates all other
base attributions of the ensemble (see Figure 1 for comparison). This effect
diminishes when using linear normalization. However, base attributions with
many high pixel attributions like GradCAM or RISE contribute more. That is,
because in linear normalization only the minimum and maximum values are used.
In contrast, Z-score-, $l_{1}$\- and $l_{2}$-normalization prohibit this issue
by considering the full distribution for normalization. We perform an
empirical comparison of all mentioned normalization techniques in Section 4.3.

Figure 2: Comparison of normalization techniques on average aggregation. For
the base attributions, see Figure 1. The explained model is a ResNet18 trained
on ImageNet. Attributions are linearly normalized to the full color range
after aggregation.

###  3.2 Aggregation

We present a number of different aggregation functions for ensemble
explanations. The simplest is the average of base attributions:

$g_{avg}\left(E\right)=\frac{1}{|E|}\sum_{e\in E}{e}.$ (5)

A different, non-linear aggregation method is the median, or a general
percentile. We denote this aggregation as $g_{k}(E)$, where
$k\in\mathbb{R},\;0\leq k\leq 100$ is the percentage of the percentile. For
example, $g_{50}(E)$ is the median aggregation and $g_{25}(E)$ the lower
quartile. We use linear interpolation between data points, but other methods
like nearest neighbor or midpoint are also viable.

Additionally, it might be helpful to balance the mean and standard deviation
of base attributions. If a pixel is important, the mean attribution of the
pixel should be high and the standard deviation low. The mean attribution
$\mu(E_{i,j})$ of a pixel $(i,j)$ and its standard deviation $\sigma(E_{i,j})$
can be calculated by:

$\displaystyle\mu\left(E_{i,j}\right)$
$\displaystyle=g_{avg}\left(E_{i,j}\right)=\frac{1}{|E|}\sum_{e\in E}e_{i,j},$
(6) $\displaystyle\sigma\left(E_{i,j}\right)$
$\displaystyle=\sqrt{\frac{1}{|E|}\sum_{e\in
E}\left(e_{i,j}-\mu\left(E_{i,j}\right)\right)^{2}}.$ (7)

Note that this is a mean and standard deviation over multiple attributions for
a single pixel. In contrast, Z-score normalization uses the global mean and
standard deviation over all pixels of a single attribution.

Methods balancing mean and standard deviations can be found by the name
"acquisition function" in the field of Bayesian Optimization [2]. They assign
a score to every data point, which indicates the value of a sample at this
position in the search space. Our motivation for using acquisition functions
is that this value transfers to the pixel importance when applied to an
ensemble of saliency maps.

Three commonly used acquisition functions are Upper Confidence Bounds (UCB)
[3], Probability of Improvement (PI) [20] and Expected Improvement (EI) [15].
They balance mean and standard deviation with the exploration rate $\epsilon$.
The name "exploration rate" originates from the setting of Bayesian
Optimization, where it balances between exploration and exploitation in the
search space. They are calculated as:

$\displaystyle g_{ucb}\left(E_{i,j},\epsilon\right)$
$\displaystyle=\mu\left(E_{i,j}\right)+\varepsilon\sigma\left(E_{i,j}\right),$
(8) $\displaystyle g_{pi}\left(E_{i,j},\epsilon\right)$
$\displaystyle=\Phi\left(Z\right),$ (9) $\displaystyle
g_{ei}\left(E_{i,j},\epsilon\right)$
$\displaystyle=\Phi\left(Z\right)\left(\mu\left(E_{i,j}\right)-E^{+}-\epsilon\right)+\sigma\left(E_{i,j}\right)\phi\left(Z\right).$
(10)

$Z=\frac{\mu(E_{i,j})-E^{+}-\epsilon}{\sigma(E_{i,j})}$ is a random variable
and $E^{+}=\max_{i,j}\left(\mu\left(E_{i,j}\right)\right)$ is the approximate
attribution value of the most important pixel. $\phi$ denotes the probability
distribution function of a standard normal distribution and $\Phi$ the
respective cumulative distribution function. All three aggregation functions
are highly dependent on the exploration rate $\epsilon$. However, it is often
not clear what exploration rate leads to the best result. To alleviate this
issue, Jasrasaria et al. [14] propose to use

$\epsilon=\frac{1}{mnE^{+}}\sum_{i=1}^{m}\sum_{j=1}^{n}\sigma(E_{i,j})^{2}$
(11)

in Equation 10. They call their method Contextual Improvement (CI), which we
denote by the function $g_{ci}(E)$.

Additionally, we propose to use the average attribution for different
exploration rates. Naturally, this is only viable for PI and EI. For UCB, this
method would simply lead to the same result as using the average exploration
rate, due to the linear nature of UCB. For an interval
$[a,b]\subseteq\mathbb{R}$ we define Average Probability of Improvement (API)
and Average Expected Improvement (AEI) of an ensemble as:

$\displaystyle g_{api}\left(E_{i,j},a,b\right)$
$\displaystyle=\frac{1}{b-a}\int_{a}^{b}g_{pi}(E_{i,j},\epsilon)\;\mathrm{d}\epsilon,$
(12) $\displaystyle g_{aei}\left(E_{i,j},a,b\right)$
$\displaystyle=\frac{1}{b-a}\int_{a}^{b}g_{ei}(E_{i,j},\epsilon)\;\mathrm{d}\epsilon.$
(13)

We approximate the average using $N$ discrete samples evenly spaced in the
interval $[a,b]$:

$\displaystyle g_{api}\left(E_{i,j},a,b\right)$
$\displaystyle\approx\frac{1}{N}\sum_{k=0}^{N-1}g_{pi}\left(E_{i,j},\;\frac{k(b-a)}{N-1}+a\right),$
(14) $\displaystyle g_{aei}\left(E_{i,j},a,b\right)$
$\displaystyle\approx\frac{1}{N}\sum_{k=0}^{N-1}g_{ei}\left(E_{i,j},\;\frac{k(b-a)}{N-1}+a\right).$
(15)

Lastly, we also consider the aggregation functions presented in earlier
literature. The aggregation using a Restricted Boltzmann Machine (RBM) [5] is
denoted by $g_{rbm}(E_{i,j},\alpha,k)$ with learning rate
$\alpha\in\mathbb{R}$ and $k\in\mathbb{N}$ iterations of training in the
contrastive learning algorithm.

Furthermore, we also consider the aggregation introduced by Rieger et al.
[28], which they denote Variance-Aggregation (VAR). We extend their concept by
the parameter $\epsilon$ controlling the balance between mean and standard
deviation:

$\displaystyle
g_{var}\left(E_{i,j},\epsilon,\delta\right)=\frac{1}{|E|}\sum_{e\in
E}\frac{e_{i,j}}{\epsilon\sigma\left(E_{i,j}\right)+\delta}.$ (16)

Figure 3: Comparison of Aggregation Methods. The explained model is a ResNet18
trained on ImageNet. The base attributions are normalized using Z-score
normalization. After aggregation, the results are linearly normalized to the
full color range.

In Figure 3, all mentioned aggregation functions are shown. The attributions
are linearly normalized to the full color range as in all other figures. The
mean and median saliency map inherently produce very similar results. UCB
includes the standard deviation such that more details are visible. PI and CI
yield sparse masks displaying only the most important pixels, for example the
eye of the ostrich. The attribution of EI and AEI separate important and non-
important regions, e.g. the edge between the ostrich and the background. The
saliency map of API is similar, but more scattered. RBM produces a saliency
map with approximated binary values. The explanation of VAR is intrinsically
close to the average aggregation. The results of methods using hyperparameters
(exploration rate, …) are highly dependent on these hyperparameters. For more
details on the effect of hyperparameters, see Section 4.2.

##  4 Experiments

We perform different experiments to evaluate the theory presented. To measure
the quality of an explanation, we utilize the insertion and deletion metrics
[25]. The idea of the insertion metric is to incrementally insert the most
important pixels into a baseline image. Similarly, the deletion metric
replaces the most important pixels from the original image with the baseline.
For every step, the model output of the class of interest is measured. A good
explanation results in a steeply rising insertion and steeply falling deletion
curve. Therefore, the area under the curve (AUC) should be maximized in
insertion and minimized in the deletion metric. We use 1000 increments in our
experiments. Therefore, in the ImageNet-dataset about 64 pixels are inserted
or deleted each increment. For GTSRB, the size of an increment is about 16
pixels.

The choice of baseline is generally important for many feature attribution
methods and metrics [35]. On the one hand it has to contain no information,
i.e. the prediction of the model changes in comparison to the original. On the
other hand, it cannot be too far from the training distribution, i.e. an out
of distribution sample. We employ a random normal baseline as a solution. More
precisely, we estimate mean and standard deviation of each color channel and
sample new values for every pixel.

There exist many metrics similar to the insertion and deletion metric, namely
IROF [29], pixel flipping [30], masking top pixels [6], infidelity [38] and
many more not listed here. For simplicity, we do not consider these.

Another popular metric measuring explanation fidelity is the pointing game
[40]. However, as discussed by Khorram et al. [17], it relies on human-
annotated ground truth boxes. Therefore, the metric does not measure the
models decision process well. We do not consider the pointing game for this
reason.

For all experiments, the ensemble consists of the following base attributions.
For the detailed hyperparameter configuration refer to Section A in the
appendix.

  * •

SmoothGrad [32]

  * •

Integrated Gradients [36]

  * •

Expected Gradients [9]

  * •

GradCAM [31]

  * •

PolyCAM [8]

  * •

iGOS++ [17]

  * •

RISE [25]

The experiments were performed on a Intel(R) Xeon(R) W-2145 CPU (3.70GHz, 8
Core, 16 Threads), Nivida RTX 6000 GPU and 32GB RAM. We average our results
over 2000 sample images from the respective validation sets. The methods were
implemented using TensorFlow 2.7.1 [21] and Scikit-learn 1.0.2 [24].

###  4.1 Aggregation Comparison

We compare the different ensemble aggregations on two different datasets: the
ImageNet-dataset [7] and the German Traffic Sign Recognition Benchmark (GTSRB)
[34]. The model architecture is a ResNet18 [12]. The ImageNet-dataset contains
over 14 million images from 1000 classes with a resolution of $224\times 224$
pixels. The GTSRB-dataset contains more than 50000 images from 43 classes with
a resolution of $128\times 128$ pixels.

(a) Insertion Curves

(b) Deletion Curves

Figure 4: Insertion and deletion curves for base attributions and average
aggregation. The explained model is a ResNet18 trained on the ImageNet-
dataset. Results are averaged over 2000 sample images.

In Figure 4, we display the insertion and deletion curves for the first
experimental setup using a ResNet-18 trained on the ImageNet dataset. The
average attribution achieves a better insertion score than each individual
base attribution. It utilizes the initial steep ascend of iGOS++ as well as
the continuous ascend of RISE and GradCAM. Note that the S-shaped curve of
iGOS++ results from the sparse saliency map. iGOS++ assigns an attribution
score of zero to many pixels, which are in turn inserted or deleted in random
order.

The deletion score for the average attribution is not better than all base
attributions, but close to the best. Additionally, the base attributions which
achieve a better deletion score, namely Integrated Gradients, Expected
Gradients and iGOS++, perform much worse overall in the insertion metric.
Overall we can conclude that the average attribution is better than each
individual base attribution.

Aggregation Method | ImageNet | GTSRB
---|---|---
| Insertion | Deletion | Insertion | Deletion
Average | 0.3808 | 0.0296 | 0.6740 | 0.0667
RBM $(\alpha=10^{-4},k=100)$ | 0.3827 | 0.0301 | 0.6312 | 0.0744
VAR $(\epsilon=10^{-7})$ | 0.3819 | 0.0296 | 0.6739 | 0.0671
UCB $(\epsilon=-0.5)$ | 0.3896 | 0.0298 | 0.6798 | 0.0657
Percentile $(k=25)$ | 0.3713 | 0.0295 | 0.6720 | 0.0675
PI $(\epsilon=-4)$ | 0.3098 | 0.0304 | 0.6306 | 0.0817
EI $(\epsilon=-6)$ | 0.3397 | 0.0298 | 0.6681 | 0.0673
CI | 0.2646 | 0.0324 | 0.5908 | 0.0827
API $([a,b]=[-7,2])$ | 0.3546 | 0.0299 | 0.6726 | 0.0672
AEI $([a,b]=[-10,5])$ | 0.3692 | 0.0298 | 0.6721 | 0.0672
Table 1: Performance of Ensemble Aggregation Methods. Base Attributions are
normalized using Z-score normalization. Hyperparameters were tuned on ImageNet
and kept constant for GTSRB. Results are averaged over 2000 samples.

In Table 1, we list the performance of all aggregation functions on both
datasets. We tuned the hyperparameters of all methods on the ImageNet dataset
and used the same values for GTSRB. This tests the general applicability of
the aggregations.

Across both datasets, the simple average aggregation achieved surprisingly
high performance. Many methods were not able to achieve higher scores in any
metric on any dataset. Most drastically, CI performed the worst in all
experiments. That is, because the approximated best value for the exploration
rate $\epsilon$ in Bayesian Optimization does not transfer to explainability.
In the context of Bayesian Optimization, a sparse score distribution is
desirable, but this directly leads to a bad insertion and deletion score. PI
fails for the same reason, even though not as drastically. EI, API and AEI are
able to achieve scores nearly as good as the average attribution, but always a
little worse. Therefore, we deem CI, PI, EI, API and AEI not applicable in the
context of explainability ensembles.

RBM achieves a higher insertion score than average aggregation on the ImageNet
dataset. However, the hyperparameter configuration did not transfer well to
GTSRB, where the insertion score is much lower. In general, it was very
challenging to find good hyperparameters for learning rate $\alpha$ and number
of iterations $k$. Borisov et al. [5] reported deletion scores significantly
lower than average aggregation, but we were not able to reproduce these
results using our models and datasets.

Variance aggregation (VAR) has a better insertion score than average
aggregation on ImageNet, but this small advantage diminishes on GTSRB.

The lower quartile aggregation (percentile with $k=25$) was able to achieve
consistent good scores and even managed to beat the average in the deletion
metric on ImageNet. However, the differences in the deletion scores on
ImageNet are so small that we do not judge the lower quartile to be better
than average aggregation overall.

The best performance was achieved by UCB. It earned the best score on three
experiments and was very close on the fourth one. Admittedly, the difference
between average and UCB is small, but the hyperparameter configuration of
$\epsilon=-0.5$ seems to transfer well to other datasets. Therefore, we
believe that incorporating the uncertainty through UCB is an uncomplicated
method of enhancing the average aggregation.

###  4.2 Hyperparameter Study

We compare the hyperparameters of percentile aggregation, UCB, PI, EI and
variance aggregation. All experiments presented in this section are performed
using a ResNet18 trained on the ImageNet-dataset.

####  4.2.1 Percentile Aggregation

The percentile aggregation $g_{k}(E)$ has the percentage parameter $k$. We
compare the insertion and deletion score for different values of k. The
aggregation with $k=0$ corresponds to the minimum, $k=50$ to the median and
$k=100$ to the maximum value. Figure 5 displays the saliency maps for
different values of $k$ as well as their insertion and deletion scores. The
best insertion score is achieved at $k=40$ and the best deletion score at
$k=10$. Interestingly, a good insertion score seems to lead to a bad deletion
score.

Percentile aggregation manages to achieve a higher insertion score than the
best base attributions over a large range of $k$. However, it is never able to
beat the average aggregation. The order of magnitude is much smaller for the
deletion score (note the different y-axis-scale).

(a) Insertion

(b) Deletion

Figure 5: Insertion and deletion scores for different percentages of
percentile aggregation. Base attributions are normalized using Z-score. The
explained model is a ResNet18 trained on ImageNet. Results are averaged over
2000 samples.

####  4.2.2 Upper Confidence Bound

For Upper Confidence Bounds, the exploration rate $\epsilon$ controls the
balance between mean and standard deviation. An important pixel should score
high attributions across all base attributions, such that its mean value is
high and standard deviation low. Therefore, we expect that subtracting the
standard deviation from the average attribution should increase the
performance of the explanation. Figure 6 displays the insertion and deletion
AUC-score for varying values of $\epsilon\in[-2,2]$. The experiment confirms
our hypothesis as the best insertion score is achieved at $\epsilon=-0.6$. For
a large range of $\epsilon$, the insertion score of UCB is higher than the
best base attributions and, for a smaller range, better than the average.

The deletion score is not significantly affected by the parameter $\epsilon$
(note the different y-axis-scale), but the best score is achieved at
$\epsilon=-1.5$.

(a) Insertion AUC

(b) Deletion AUC

Figure 6: Insertion and deletion scores for Upper Confidence Bound (UCB) with
varying exploration rate. Base attributions are normalized using Z-score. The
explained model is a ResNet18 trained on ImageNet. Results are averaged over
2000 samples.

####  4.2.3 Probability of Improvement

Probability of Improvement yields an explanation separating important and non-
important regions by nearly binary values. This is called an approximated hard
explanation and inherently hurts the insertion and deletion score, because the
metrics only consider the order of pixel attributions. Nonetheless, sparse and
hard masks are more interpretable for humans [39].

The exploration rate $\epsilon$ controls the size of the approximated hard
saliency map. Figure 7 visualizes this effect and displays the corresponding
insertion and deletion scores. The performance of PI measured by insertion and
deletion metric is worse than the best base attribution and average
aggregation throughout the whole range.

(a) Insertion AUC

(b) Deletion AUC

Figure 7: Insertion and deletion scores for Probability of Improvement (PI)
with varying exploration rate. Base attributions are normalized using Z-score.
The explained model is a ResNet18 trained on ImageNet. Results are averaged
over 2000 samples.

####  4.2.4 Expected Improvement

Figure 8 displays the performance of EI for different exploration rates. For
small exploration rate, the performance converges to the average attribution.
For larger values, using EI only decreases the performance. Analyzing Equation
10, we quickly discover the reason for the convergence to average aggregation.
For large negative $\epsilon$, the probability distribution function of a
standard normal distribution goes to zero and the cumulative distribution
function to one. Therefore, the only terms left are the average aggregation
and a constant shift of $-E^{+}-\epsilon$. The shift does not change the order
of pixel attributions and therefore neither insertion nor deletion score.

(a) Insertion AUC

(b) Deletion AUC

Figure 8: Insertion and deletion scores for Expected Improvement (EI) with
varying exploration rate. Base attributions are normalized using Z-score. The
explained model is a ResNet18 trained on ImageNet. Results are averaged over
2000 samples.

####  4.2.5 Variance Aggregation

In Figure 9, we analyze the fidelity of variance aggregation (VAR) for
different values of $\epsilon$. The range of interesting values
($[10^{-10},10^{-2}]$) is far from the original paper [28] due to Z-score
normalization of the base attributions. For very small values of $\epsilon$,
the insertion and deletion score is close to the average attribution.

Using variance aggregation, it is possible to achieve scores better than the
simple average. For insertion, $\epsilon=10^{-7}$ achieves the best result.
For deletion, the score becomes better with larger values of $\epsilon$.
However, the difference is very small as for all experiments using ImageNet.
The best balance is achieved at $\epsilon=10^{-7}$.

(a) Insertion AUC

(b) Deletion AUC

Figure 9: Insertion and deletion scores for Variance Aggregation (VAR) with
varying exploration rate. Base attributions are normalized using Z-score. The
explained model is a ResNet18 trained on ImageNet. Results are averaged over
2000 samples.

###  4.3 Effect of Normalization

As already discussed above, it is necessary to normalize the base attributions
before aggregation. However, it is not clear which normalization technique
leads to the best result. We compare the aforementioned methods of linear,
Z-score, $l_{1}$\- and $l_{2}$-normalization against no normalization using
the average ensemble. Figure 10 displays the insertion and deletion curves for
this experimental setup. Interestingly, linear normalization achieves, despite
its theoretical downside of ignoring the full distribution, the best insertion
score. It is closely followed by Z-score normalization. The $l_{1}$\- and
$l_{2}$-normalization achieve the best deletion score, again closely followed
by Z-score normalization. We conclude that Z-score normalization leads to the
best balance between a good insertion and deletion score, even though it does
not perform best in either metric.

(a) Insertion

(b) Deletion

Figure 10: Insertion and deletion curves for different norms using average
aggregation. The explained model is a ResNet18 trained on ImageNet. Results
are averaged over 2000 samples.

###  4.4 Ablation Study

An important question is the sensitivity of the aggregation methods to
unfaithful or noisy base attributions. Borisov et al. [5] included multiple
base attributions consisting of only random noise and visually compared the
results. However, this experiment does not reflect reality well. We focus on
the question if the worst base attributions of an ensemble contribute
positively to the aggregated result.

(a) Insertion

(b) Deletion

Figure 11: Insertion and deletion for different ensemble sizes. Ensembles
contain the top-k base attributions regarding the measured metric. The
explained model is a ResNet18 trained on ImageNet. Results are averaged over
2000 samples.

To answer this question, we start with the best base attribution and
iteratively insert the other base attributions in descending order of their
performance. We consider the performance regarding both metrics individually,
such that the order is different for each metric. Figure 11 displays the
results of this experiment. Using the insertion metric, we observe a negative
impact when inserting the fourth, fifth and seventh base attribution into the
ensemble. This indicates that better performance is possible if fewer, high
performing base attributions are used. However, the drop in performance is
less pronounced in UCB, compared to average attribution, due to the inclusion
of the standard deviation between base attributions.

Again, the order of magnitude for the insertion score is much smaller (note
the different y-axis-scale), but best performance is achieved when only using
two ensembles. Again, for smaller ensembles average attribution performs
better than UCB, but for larger ensembles the opposite is true.

Even though performance decreases for both metrics when including weaker base
attributions, the drop in performance is relatively small. In a real-world
scenario, determining the performance of every base attribution and finding
the best ensemble would be a tedious task. Therefore, we argue that using all
available base attributions and aggregating the ensemble with UCB is the best
option.

Another interesting observation regarding both metrics is the increasing
difference in performance between UCB and the simple average when using larger
ensembles. This makes sense, as the standard deviation between base
attributions becomes more meaningful in larger ensembles.

##  5 Conclusion and Future Work

We introduced multiple aggregation functions originating in the context of
Bayesian Optimization and compared their fidelity to existing aggregation
functions. The empirical comparison revealed that incorporating confidence
using Upper Confidence Bounds yields the best saliency maps.

Additionally, normalization of the base attributions is very important. We
compared multiple normalization techniques and found Z-score normalization
most effective.

As discussed above, aggregation functions like Expected Improvement and
Probability of Improvement assign an attribution of zero to many pixels, which
inherently hurts the insertion and deletion scores. We tried to solve this
issue along with the problem of finding the best exploration rate by averaging
over multiple exploration rates. This solution only achieved limited results,
but in future work we may examine other techniques. For example, one could
successively mask the most important pixels and compute new attribution scores
in every iteration. In addition to solving the problem of attribution scores
of zero, this solution also models the sequential process of Bayesian
Optimization. Therefore, it may yield a better theoretical foundation and
empirical results. However, this is beyond the scope of this work.

Many base attributions have a high number of hyperparameters, which
significantly affect the resulting explanation. For example, iGOS++ [17] has
12 hyperparameters. Finding the best hyperparameter configuration can be a
difficult challenge [10]. As discussed by Borisov et al. [5], aggregating
explanations from multiple hyperparameter configurations alleviates this issue
and might even produce better results. Further directions of research may
investigate the applicability of the presented methods in this context.

Furthermore, all aggregation methods presented are pixel-level aggregation,
i.e. the aggregated attribution is calculated independently for each pixel.
The only exception is RBM, which can be seen as a global aggregation since it
considers all pixels during training time. Future work could also explore
aggregations based on a local neighborhood.

## References

  * [1] Julius Adebayo et al.  “Sanity Checks for Saliency Maps”  arXiv, 2018  DOI: 10.48550/ARXIV.1810.03292
  * [2] Apoorv Agnihotri and Nipun Batra  “Exploring Bayesian Optimization”  In _Distill_ , 2020  DOI: 10.23915/distill.00026
  * [3] Peter Auer  “Using Confidence Bounds for Exploitation-Exploration Trade-Offs”  In _Journal of Machine Learning Research_ 3 JMLR.org, 2003, pp. 397–422
  * [4] Umang Bhatt, Adrian Weller and José M.. Moura  “Evaluating and Aggregating Feature-based Model Explanations”  arXiv, 2020  DOI: 10.48550/ARXIV.2005.00631
  * [5] Vadim Borisov et al.  “A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines”  arXiv, 2021  DOI: 10.48550/ARXIV.2111.07379
  * [6] Jianbo Chen, Le Song, Martin J. Wainwright and Michael I. Jordan  “L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data”  arXiv, 2018  DOI: 10.48550/ARXIV.1808.02610
  * [7] J. Deng et al.  “ImageNet: A Large-Scale Hierarchical Image Database”  In _CVPR09_ , 2009
  * [8] Alexandre Englebert, Olivier Cornu and Christophe De Vleeschouwer  “PolyCAM: High resolution class activation map for convolutional neural networks”  arXiv, 2022  DOI: 10.48550/ARXIV.2204.13359
  * [9] Gabriel Erion et al.  “Improving performance of deep learning models with axiomatic attribution priors and expected gradients”  arXiv, 2019  DOI: 10.48550/ARXIV.1906.10670
  * [10] Matthias Feurer and Frank Hutter  “Hyperparameter Optimization”  In _Automated Machine Learning_ Springer, Cham, 2019
  * [11] Rafael C. Gonzales and Richard E. Woods  “Digital Image Processing”  Prentice Hall, 2007, pp. 74
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep Residual Learning for Image Recognition”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [13] Geoffrey Hinton, Simon Osindero and Yee-Whye Teh  “A Fast Learning Algorithm for Deep Belief Nets”  In _Neural computation_ 18, 2006, pp. 1527–54  DOI: 10.1162/neco.2006.18.7.1527
  * [14] Dipti Jasrasaria and Edward O. Pyzer-Knapp  “Dynamic Control of Explore/Exploit Trade-Off In Bayesian Optimization”  arXiv, 2018  DOI: 10.48550/ARXIV.1807.01279
  * [15] Donald R. Jones, Matthias Schonlau and William J. Welch  “Efficient Global Optimization of Expensive Black-Box Functions”  In _Journal of Global Optimization_ 13, 1998, pp. 455–492
  * [16] Gargi Joshi, Rahee Walambe and Ketan Kotecha  “A Review on Explainability in Multimodal Deep Neural Nets”  In _IEEE Access_ 9, 2021, pp. 59800–59821  DOI: 10.1109/ACCESS.2021.3070212
  * [17] Saeed Khorram, Tyler Lawson and Fuxin Li  “iGOS++: Integrated Gradient Optimized Saliency by Bilateral Perturbations”  arXiv, 2020  DOI: 10.48550/ARXIV.2012.15783
  * [18] Herber Kreyszig and Erwin Kreyszig  “Advanced Engineering Mathematics”  John WileySons Inc, 2011, pp. 1014
  * [19] Satyapriya Krishna et al.  “The Disagreement Problem in Explainable Machine Learning: A Practitioner’s Perspective”  arXiv, 2022  DOI: 10.48550/ARXIV.2202.01602
  * [20] Harold J. Kushner  “A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise”  In _Journal of Basic Engineering_ 86, 1964, pp. 97–106
  * [21] Martín Abadi et al.  “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems” Software available from tensorflow.org, 2015  URL: https://www.tensorflow.org/
  * [22] Tim Miller  “Explanation in artificial intelligence: Insights from the social sciences”  In _Artificial Intelligence_ 267, 2019, pp. 1–38  DOI: https://doi.org/10.1016/j.artint.2018.07.007
  * [23] Christoph Molnar  “Interpretable Machine Learning”  Bookdown, 2022  URL: https://christophm.github.io/interpretable-ml-book
  * [24] F. Pedregosa et al.  “Scikit-learn: Machine Learning in Python”  In _Journal of Machine Learning Research_ 12, 2011, pp. 2825–2830
  * [25] Vitali Petsiuk, Abir Das and Kate Saenko  “RISE: Randomized Input Sampling for Explanation of Black-box Models”  arXiv, 2018  DOI: 10.48550/ARXIV.1806.07421
  * [26] Gabrielle Ras, Ning Xie, Marcel Gerven and Derek Doran  “Explainable Deep Learning: A Field Guide for the Uninitiated”  arXiv, 2020  DOI: 10.48550/ARXIV.2004.14545
  * [27] Laura Rieger and Lars Kai Hansen  “A simple defense against adversarial attacks on heatmap explanations”  arXiv, 2020  DOI: 10.48550/ARXIV.2007.06381
  * [28] Laura Rieger and Lars Kai Hansen  “Aggregating explanation methods for stable and robust explainability”  arXiv, 2019  DOI: 10.48550/ARXIV.1903.00519
  * [29] Laura Rieger and Lars Kai Hansen  “IROF: a low resource evaluation metric for explanation methods”  arXiv, 2020  DOI: 10.48550/ARXIV.2003.08747
  * [30] Wojciech Samek et al.  “Evaluating the visualization of what a Deep Neural Network has learned”  arXiv, 2015  DOI: 10.48550/ARXIV.1509.06321
  * [31] Ramprasaath R. Selvaraju et al.  “GradCAM: Visual Explanations from Deep Networks via Gradient-Based Localization”  In _International Journal of Computer Vision_ 128.2 Springer ScienceBusiness Media LLC, 2019, pp. 336–359  DOI: 10.1007/s11263-019-01228-7
  * [32] Daniel Smilkov et al.  “SmoothGrad: removing noise by adding noise”  arXiv, 2017  DOI: 10.48550/ARXIV.1706.03825
  * [33] Paul Smolensky  “Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations”  MIT Press, 1986, pp. 194–281
  * [34] J. Stallkamp, M. Schlipsing, J. Salmen and C. Igel  “Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition” Selected Papers from IJCNN 2011  In _Neural Networks_ 32, 2012, pp. 323–332  DOI: https://doi.org/10.1016/j.neunet.2012.02.016
  * [35] Pascal Sturmfels, Scott Lundberg and Su-In Lee  “Visualizing the Impact of Feature Attribution Baselines”  In _Distill_ , 2020  DOI: 10.23915/distill.00022
  * [36] Mukund Sundararajan, Ankur Taly and Qiqi Yan  “Axiomatic Attribution for Deep Networks”  arXiv, 2017  DOI: 10.48550/ARXIV.1703.01365
  * [37] Yipei Wang and Xiaoqian Wang  “A Unified Study of Machine Learning Explanation Evaluation Metrics”  arXiv, 2022  DOI: 10.48550/ARXIV.2203.14265
  * [38] Chih-Kuan Yeh et al.  “On the (In)fidelity and Sensitivity of Explanations”  In _Advances in Neural Information Processing Systems_ 32 Curran Associates, Inc., 2019  URL: https://proceedings.neurips.cc/paper/2019/file/a7471fdc77b3435276507cc8f2dc2569-Paper.pdf
  * [39] Hao Yuan, Haiyang Yu, Shurui Gui and Shuiwang Ji  “Explainability in Graph Neural Networks: A Taxonomic Survey”  arXiv, 2020  DOI: 10.48550/ARXIV.2012.15445
  * [40] Jianming Zhang et al.  “Top-down Neural Attention by Excitation Backprop”  arXiv, 2016  DOI: 10.48550/ARXIV.1608.00507
  * [41] Jianlong Zhou, Amir H. Gandomi, Fang Chen and Andreas Holzinger  “Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics”  In _Electronics_ 10.5, 2021  DOI: 10.3390/electronics10050593

##  Appendix A Hyperparameter of Base Attributions

The hyperparameters of the base attributions used for all ensembles in the
experiments are listed below:

  * •

SmoothGrad [32]

    * –

256 samples

    * –

Standard deviation $\sigma=0.1$

  * •

Integrated Gradients [36] with blur baseline ($\sigma=20$ and kernel size
$25\times 25$)

    * –

Baseline: blurred image with $\sigma=20$ and kernel size $25\times 25$

    * –

30 interpolation steps

  * •

Expected Gradients [9]

    * –

20 samples

    * –

30 interpolation steps

    * –

Baseline: Normal

  * •

GradCAM [31] (has no parameters)

  * •

PolyCAM [8]

    * –

Channel-wise variation of confidence as activation map weights (see original
paper for more information)

  * •

iGOS++ [17]

    * –

Mask resolution $50\times 50$

    * –

Sparsity regularization parameter $\lambda_{1}=20$

    * –

Smoothness regularization parameter $\lambda_{2}=30$

    * –

20 iterations

    * –

25 interpolation steps

    * –

Maximum step size $\alpha_{u}=10$

    * –

Minimum step size $\alpha_{l}=10^{-5}$

    * –

Armijo condition constant $\beta=2$

    * –

Bilateral Total Variation Norm $\beta=10^{-4}$

    * –

Bilateral Total Variation Smoothing $\sigma=0.01$

    * –

decay $\eta=0.2$

    * –

Baseline: Normal

  * •

RISE [25]

    * –

5000 samples

    * –

Mask resolution $10\times 10$

Normal baseline refers to an estimation of mean and standard deviation of each
color channel. Afterwards, new pixel values are sampled from the resulting
normal distribution.