# Photon-jet events as a probe of axion-like particles at the LHC

Daohan Wang<EMAIL_ADDRESS>Department of Physics and Institute of
Theoretical Physics, Nanjing Normal University, Nanjing, 210023, China  CAS
Key Laboratory of Theoretical Physics, Institute of Theoretical Physics,
Chinese Academy of Sciences, Beijing 100190, China  School of Physical
Sciences, University of Chinese Academy of Sciences, Beijing 100049, China
Lei Wu  corresponding author<EMAIL_ADDRESS>Department of Physics and
Institute of Theoretical Physics, Nanjing Normal University, Nanjing, 210023,
China  Jin Min Yang<EMAIL_ADDRESS>CAS Key Laboratory of Theoretical
Physics, Institute of Theoretical Physics, Chinese Academy of Sciences,
Beijing 100190, China  School of Physical Sciences, University of Chinese
Academy of Sciences, Beijing 100049, China  Mengchao Zhang  corresponding
author<EMAIL_ADDRESS>Department of Physics and Siyuan Laboratory, Jinan
University, Guangzhou 510632, P.R. China

(August 27, 2024)

###### Abstract

Axion-like particles (ALPs) are predicted by many extensions of the Standard
Model (SM). When ALP mass lies in the range of MeV to GeV, the cosmology and
astrophysics will be largely irrelevant. In this work, we investigate such
light ALPs through the ALP-strahlung process $pp\to Va(\to\gamma\gamma)$ at
the 14 TeV LHC with an integrated luminosity of 3000 fb-1 (HL-LHC). With the
photon-jet algorithm, we demonstrate that our approach can probe the mass
range of ALPs, which is inaccessible to previous LHC experiments. The obtained
result can surpass the existing limits on ALP-photon coupling in the ALP mass
range from 0.3 GeV to 10 GeV.

###### pacs:

Valid PACS appear here

##  I Introduction

Searching for new particles is one of the crutial tasks in the LHC experiment.
Light pseudo-scalars, such as axion-like particles (ALPs), are theoretically
well-motivated. They generically appear in models with the spontaneous
breaking of a global symmetry Peccei and Quinn (1977); Weinberg (1978);
Wilczek (1978); Kim (1979) or in the compactifications of string theory Svrcek
and Witten (2006); Arvanitaki et al. (2010); Cicoli et al. (2012). Besides,
ALPs may have connections with electroweak phase transition Ballesteros et al.
(2017) and play a key role in solving the hierarchy problem Graham et al.
(2015).

In general, the ALP masses and couplings to SM particles are independent
parameters. When ALP masses are below the MeV scale, they are already subject
to many constraints from cosmological and astrophysical observations Raffelt
(1990); Marsh (2016).Besides, such light ALPs can serve as cold dark matter
(DM) Preskill et al. (1983); Abbott and Sikivie (1983); Dine and Fischler
(1983) and be explored by various astrophysical and terrestrial anomalies
Arias et al. (2012); Jaeckel et al. (2014); Athron et al. (2021); Gao et al.
(2020).While for ALPs in the mass range of MeV to GeV, the above cosmological
and astrophysical bounds will vanish. But ALPs can have sizable contributions
to low-energy observables in particle physics. Recently, many works have been
paid to searching for ALPs in intensity frontiers Izaguirre et al. (2017);
Dolan et al. (2017); Bauer et al. (2020); Banerjee et al. (2020); Gu et al.
(2021).

On the other hand, the ALPs can be directly produced at high energy colliders
Mimasu and Sanz (2015); Knapen et al. (2017); Barrie et al. (2016); Bauer et
al. (2017a); Brivio et al. (2017); Bauer et al. (2019); Ebadi et al. (2019);
Wang et al. (2021); Ren et al. (2021). A considerable region of parameter
space of ALPs has been constrained by the LEP and LHC data. For instance, ALPs
can be searched for from the processes $e^{+}e^{-}\to\gamma
a(\to\gamma\gamma)$ and $Z\to a\gamma$ at LEP Jaeckel and Spannowsky (2016).
The non-resonant production process of ALPs, such as $pp\to ZZ$, has been
proposed at the LHC Gavela et al. (2020). The rare decays of Higgs boson $h\to
Za(\to\gamma\gamma)$ and $h\to a(\to\gamma\gamma)a(\to\gamma\gamma)$ have been
proposed to probe the ALP coupling to the photon in terms of the ALP mass on a
future complete LHC Run-3 dataset in the future Bauer et al. (2017b).

Figure 1: Feynman diagrams of ALP-strahlung process $pp\to
Va(\to\gamma\gamma)$ at the LHC, where $V=W,Z$.

However, current LHC analyses are sensitive to the ALPs in the parameter
region $m_{a}\gtrsim 10$ GeV. When ALPs become light, they can be highly
boosted, and thus the two photons from the ALP decay are recognized as one
object in the detector. This will lead to an interesting signature “photon-
jet” at the LHC. In this work, we will use jet substructure variables to
analyze these “photon-jet” events from the decay of ALPs, as the method
proposed in Ref. Ellis et al. (2013a, b). We will focus on ALPs that only
couple to the electroweak vector bosons with a mass in MeV to GeV range. Such
light electroweak ALPs have obtained an increasing amount of interest due to
current collider data. In this work, there is no ALP-gluon coupling so that
the dominant production channel of electroweak ALPs at the LHC is ALP-
strahlung process $pp\to Va$, where $V=W,Z$ (see Fig. 1). We find that our
proposal can extend the LHC sensitivity to parameter space unreachable in
previous studies.

##  II Model and Photon-jet

The relevant effective interactions of ALP with electroweak gauge bosons up to
dimension-5 is given by Brivio et al. (2017)

$\displaystyle\mathcal{L}_{\text{eff}}$ $\displaystyle\supseteq$
$\displaystyle\frac{1}{2}(\partial^{\mu}a)(\partial_{\mu}a)-\frac{1}{2}m_{a}^{2}a^{2}-C_{BB}\frac{a}{f_{a}}B_{\mu\nu}\tilde{B}^{\mu\nu}$
(1) $\displaystyle-C_{WW}\frac{a}{f_{a}}W^{i}_{\mu\nu}\tilde{W}^{\mu\nu,i},$

where the ALP field is denoted by $a$, and the field strengths for the SM
gauge groups are denoted as
$V_{\mu\nu}\equiv\partial_{\mu}V_{\nu}-\partial_{\nu}V_{\mu}$ and
$\tilde{V}_{\mu\nu}\equiv\epsilon_{\mu\nu\rho\sigma}V^{\rho\sigma}$. The
$W_{\mu\nu}$ and $B_{\mu\nu}$ are field strengths for $SU(2)_{L}$ and
$U(1)_{Y}$, respectively. Both $C_{WW}$ and $C_{BB}$ contribute to the
interaction of the ALP with two photons. The dimentionful coupling
$g_{a\gamma\gamma}$, $g_{aWW}$, $g_{aZZ}$, $g_{a\gamma Z}$ are given by

$\displaystyle g_{a\gamma\gamma}$ $\displaystyle=$
$\displaystyle\frac{4}{f_{a}}(C_{BB}\cos\theta_{W}^{2}+C_{WW}\sin\theta_{W}^{2})$
(2) $\displaystyle g_{aWW}$ $\displaystyle=$
$\displaystyle\frac{4}{f_{a}}C_{WW}$ (3) $\displaystyle g_{aZZ}$
$\displaystyle=$
$\displaystyle\frac{4}{f_{a}}(C_{BB}\sin\theta_{W}^{2}+C_{WW}\cos\theta_{W}^{2})$
(4) $\displaystyle g_{a\gamma Z}$ $\displaystyle=$
$\displaystyle\frac{8}{f_{a}}\sin\theta_{W}\cos\theta_{W}(C_{WW}-C_{BB})$ (5)

where $\theta_{W}$ is the Weinberg angle. For simplicity, we set
$C_{WW}=C_{BB}$ in our study. We consider two ALP-strahlung processes $pp\to
W^{\pm}(\to\ell^{\pm}\nu)a(\to\gamma\gamma)$ and $pp\to
Z(\to\ell^{+}\ell^{-})a(\to\gamma\gamma)$ as our signals. The dominant SM
background processes include $V\gamma$, $Vj$, and QCD di-jets. In the LHC
experiment, when $m_{a}\gtrsim 10$ GeV, the two photons from ALP decay can be
separated enough and identified as $2\gamma$ events. On the other hand, if ALP
mass is lighter than a few hundred MeV, those photons are highly collimated so
that they will be detected as single photon events. While between these two
mass limits, the two photons will be seen like a “photon-jet”. In this case,
we can use jet substructure techniques to discriminate photon-jets from single
photons and QCD jets Dobrescu et al. (2001); Chang et al. (2007); Toro and
Yavin (2012); Draper and McKeen (2012); Ellis et al. (2013a, b).

We implement the lagrangian of Eq. 1 in FeynRules Alloul et al. (2014) to
generate the corresponding UFO model file. We use MadGraph5 aMC@NLO Alwall et
al. (2014) to calculate the production cross sections and generate the signal
and background events. Then the parton level events are showered and
hadronized by Pythia8 Sjöstrand et al. (2015). The detector simulation is
performed by Delphes de Favereau et al. (2014). FastJet Cacciari et al. (2012)
is used for jet clustering. The electron and muon identification efficiencies
are taken as default values. Based on the energy-flow algorithm CMS (2009), we
cluster EflowPhotons, EflowNeutralHadrons and ChargedHadrons into jets by
using the anti-$k_{t}$ algorithm Cacciari et al. (2008) with $R_{j}=0.4$. Only
the leading jet with $p_{T}$ > 50 GeV for each event is retained for further
analysis. Then we recluster the rest energy deposits in each jet using $k_{t}$
algorithm Catani et al. (1993); Ellis and Soper (1993), which determines a
recombination tree for the jets. Besides, we also consider the pileup
interaction effect. The low-$Q^{2}$ soft QCD pile-up events are generated by
Pythia8 and then simulated by Delphes. We take the default parametrization in
the CMS card to distribute the minimum-bias pile-up events and hard scattering
events in time and $z$ positions. The average amount of pileup events per
bunch-crossing is considered to be 40. Besides, it should be noted that the
calibration of jet energy scale is one of the main uncertainties in the jet
analysis at the LHC. Since the traditional photon isolation criterion is not
used, the new method of calibrating the photon-jets energy scale based on a
full simulation of detector and real data is needed. This is challenging for a
phenomenological study and is beyond the scope of our work.

To discriminate the signal events from the SM backgrounds, we use jet
substructure algorithm to select our photon-jet events. We generate $pp\to
Z(\to\nu\bar{\nu})a$, $pp\to Z(\to\nu\bar{\nu})j$ and $pp\to
Z(\to\nu\bar{\nu})\gamma$ as the training samples of photon-jet, QCD jet, and
single photon events, respectively. According to Ref. Ellis et al. (2013a, b),
the following variables are used in our substructure analysis:

Figure 2: Distributions of jet substructure variables (log$\theta_{J}$,
$N_{\text{track}}$) for single photon, photon-jet and QCD jet events.

Figure 3: Distributions of N-subjettiness variables (log$\tau_{1}$,
$\tau_{2}/\tau_{1}$, $\tau_{3}/\tau_{1}$, $\tau_{3}/\tau_{2}$) for single
photon, photon-jet and QCD jet events.

Figure 4: Distributions of jet substructure variables ($\lambda_{J}$,
$\epsilon_{J}$) for single photon, photon-jet, and QCD jet events.

Hadronic energy fraction of a jet, $\theta_{J}$, which is defined as the
energy fraction carried by a jet’s constituents that belong to the hadronic
calorimeter (HCal), including the EFlowNeutralHadrons and ChargedHadrons in
the energy-flow algorithm:

$\theta_{J}\ =\ \frac{1}{E_{J}}(\sum_{i}E_{i}+\sum_{j}E_{j}),$ (6)

where $E_{J}$ is the total energy of the leading jet, $E_{i}$ and $E_{j}$ are
the energy of the $i$-th EFlowNeutralHadron and the energy of the $j$-th
ChargedHadron respectively that are constituents of the jet. Due to isospin
symmetry, a QCD-jet typically contains about 2/3 charged pions and 1/3 neutral
pions, which will decay to a pair of photons. Thus we expect to have a peak at
$\theta_{J}\sim$ 2/3 (log$\theta_{J}\sim$ -0.2) for QCD jets. On the other
hand, most of the energy of single photons and photon-jets are deposited in
Electromagnetic Calorimeter (ECal). So the log$\theta_{J}$ of them are much
smaller than that of QCD jets. In Fig. 2, we show the distribution of
log$\theta_{J}$ for QCD jets, single photons, and photon-jets in our
simulation data.

The number of charged tracks in a jet, $N_{track}$, which means the number of
charged particles inside a jet. We calculate the angular distance $\Delta R$
between the leading jet and all the tracks with $p_{T}$ > 2 GeV. When $\Delta
R<R_{j}$, the corresponding track is considered to be inside the leading jet.
As mentioned before, a QCD-jet typically contains several charged pions while
single photons and photon-jets leave no tracks in the Tracker. Therefore, the
number of tracks associated with QCD-jets varies over a broad range, however,
the single photon and photon-jet samples are dominated by jets without
associated tracks. In Fig. 2, we show the distribution of $N_{track}$ per jet
for QCD jet, single photon and photon-jet samples. It can be seen that the
variable $N_{track}$ has a good discrimination power of QCD-jets and single
photons/photon-jets, but cannot distinguish the single photons from photon-
jets.

N-subjettiness, which is a measure of the number of energetic subjets inside a
jet Thaler and Van Tilburg (2011, 2012). Given a set of N-axes, we can define

$\tau_{N}=\frac{\sum_{k}p_{T_{k}}\times\text{min}\bigl{\\{}\Delta
R_{1,k},\cdots,\Delta R_{N,k}\bigr{\\}}}{\sum_{k}p_{T_{k}}\times R_{j}}\;,$
(7)

where $k$ runs over all the constituents of the jet. $p_{T_{k}}$ is the
transverse momentum of the $k$-th constituent, and $\Delta R_{l,k}$ is the
angular distance between the $l$-th subjet and the $k$-th constituent of the
jet. When $N=1$, N-subjettiness describes the energy distribution of the jet.
In Fig. 3, it can be seen that the single photon and photon-jet samples peak
at lower values of $\log\tau_{1}$, as a comparison with the QCD jets. Besides,
$\tau_{N}$ will decrease rapidly as $N$ increasing. Thus, the ratio of two
N-subjettiness, $\tau_{a}/\tau_{b}$, can be used to separate the signal and
backgrounds. From Fig. 3, we can see that the photon-jet and single photon
events have smaller values of $\tau_{2}/\tau_{1}$, $\tau_{3}/\tau_{1}$ and
$\tau_{3}/\tau_{2}$ than the QCD-jets.

Functions of energy and $p_{T}$ of subjets We use the Jet Filtering
Butterworth et al. (2008) in FastJet to recluster the leading jet’s
constituents using $k_{T}$ algorithm. After reclustering, we obtain $N$
exclusive $k_{T}$-subjets. We set $N=5$ and take the three hardest subjets to
construct two jet substructure variables. One is the fraction of the jet
$p_{T}$ carried by the leading subjet $\lambda_{J}$,

$\lambda_{J}\ =\ \log\Bigl{(}1-\frac{p_{T_{L}}}{p_{T_{J}}}\Bigr{)}\;,$ (8)

and the other is the energy correlation function of the three hardest subjets
$\epsilon_{J}$,

$\epsilon_{J}\ =\ \frac{1}{E_{J}^{2}}\sum_{(i>j)\in
N_{\text{hard}}}E_{i}E_{j}\,,$ (9)

where $p_{T_{L}}$ and $E_{J}$ are the total momentum and energy of a given
jet, respectively. $E_{i}$ is the energy of the $i$-th subjet. For the single
photon and photon-jet, the leading subjet carries most of the energy.
Therefore, their distributions of $\lambda_{J}$ have smaller values than QCD-
jets. As for $\epsilon_{J}$, among the 3 hardest jets, $\epsilon_{J}$
increases with the number of subjets with similar energy. The photon-jet and
QCD-jets have a larger values of $\epsilon_{J}$ than the single photon. Fig. 4
presents the corresponding distributions of $\epsilon_{J}$ and $\lambda_{J}$
for signal and backgrounds.

In the following we implement the above variables in a Boosted Decision Tree
(BDT) Roe et al. (2005) to enhance the ability of distinguishing signal from
background, as Ref. Ellis et al. (2013a, b). In practice, we use the Toolkit
for Multivariate Analysis (TMVA) Hocker et al. (2007) package and the “BDTD”
option to book BDTs. We uses 200 trees ensemble that requires a minimum
training events in each node of $2.5\%$ and a maximum tree depth of 3. Other
variables are set at their default values. It is trained using the half of the
signal and background events and is tested on the rest of the events. We also
demand the Kolmogorov-Smirnov test of the BDT analysis to be greater than 0.01
to avoid overtraining.

From Fig. 2, we can see that $\theta_{J}$ and $N_{track}$ offer a good
discrimination between photon-jets and QCD jets. The other six variables can
be used for discrimination between photon-jets and single photons. In order to
achieve simultaneous separation of photon-jets, single photons, and QCD-jets,
we take the $\theta_{J}$ and $N_{track}$ of photon-jets as signal samples and
the $\theta_{J}$ and $N_{track}$ of QCD-jets as background samples to train
the first BDT, namely BDT-1. The other six jet substructure variables of
photon-jets and single photons, as signal samples and background samples
respectively, are used to train the second BDT, namely BDT-2. In other words,
we discriminate photon-jet and single photon from QCD-jet by using BDT-1, and
then discriminate photon-jet from single photon by using BDT-2.

After training, BDT can map an event with two sets of variables
{log$\theta_{J}$, $N_{T}$} and {$\log\tau_{1}$, $\tau_{2}/\tau_{1}$,
$\tau_{3}/\tau_{1}$, $\tau_{3}/\tau_{2}$, $\lambda_{J}$, $\epsilon_{J}$} into
two BDT responses.

Figure 5: BDT response distributions of photon-jet as signal and single photon
and QCD-jet as backgrounds respectively

In Fig. 5 we show the BDT responses for $m_{a}$ = 3 GeV. In the BDT-1
analysis, photon-jet-like events and photon-like events tend to get high BDT
response while a QCD jet-like events tend to get low response. In the BDT-2, a
photon-jet-like event tends to get high BDT response while a photon-like event
tends to get a low BDT response. In BDT-1, we identify a jet with BDT response
larger than 0 as a photon-jet or a single photon, while in BDT-2, we tag a jet
with BDT response larger than 0.4 as a photon-jet. Since the jet substructure
variables are sensitive to the ALP mass, we choose different BDT response cuts
for different ALP masses to optimize the search ability.

##  III ALP-strahlung process at the LHC

For the ALP-strahlung process $pp\to W^{\pm}a$, the final states are
characterized by an isolated lepton and one photon-jet. We consider the SM
backgrounds: QCD di-jet, $W^{\pm}j$, $W^{\pm}\gamma$, $t\bar{t}$ and $tj$.
Based on the above analysis, we impose the following cuts to discriminate the
signal and backgrounds: (i) Exactly one isolated lepton (electron or muon)
with $p_{\text{T}}>20$ GeV and $|\eta|<2.5$; (ii) The hardest jet with
$p_{\text{T}}>50$ GeV and $|\eta|<2.5$; (iii) The hardest jet’s BDT-1 response
is larger than 0 and BDT-2 response is larger than the corresponding BDT
response cut for different ALP masses.

For the ALP-strahlung process $pp\to Za$, the final states are characterized
by the opposite sign and same flavor charged lepton pair and a photon-jet. The
main SM backgrounds include $Z\gamma$ and $Zj$. According to our analysis, we
discriminate the signal and backgrounds by imposing the following cuts: (i)
Exactly two leptons with $p_{\text{T}}>20$ GeV and $|\eta|<2.5$; (ii) The
invariant mass of the oppositely charged lepton pair with same flavor must be
within $|m_{ll}-m_{Z}|$<20 GeV; (iii) The hardest jet with $p_{\text{T}}>50$
GeV and $|\eta|<2.5$; (iv) The hardest jet’s BDT-1 response is larger than 0
and BDT-2 response is larger than the corresponding BDT response cut for each
ALP mass.

cut flow |  | signal
---
| $jj$
---
| $W^{\pm}\gamma$
---
| $W^{\pm}j$
---
| $t\bar{t}$
---
| $tj$
---
| 1 lepton with
---
$p_{\text{T}}>20$ GeV
and $|\eta|<2.5$
36.24 | 19357.15 | 12.31 | 4448.53 | 151.86 | 29.39
| The hardest jet
---
with $p_{\text{T}}>50$ GeV
and $|\eta|<2.5$
23.31 | 12893 | 2.54 | 1605.92 | 136.41 | 18.90
| The hardest jet’s
---
BDT-1 > 0
and BDT-2 > 0.4
4.14 | 3.30 | 0.001 | 0.77 | 0.094 | 0.012
Table 1: The cut-flow of the cross sections (in units of pb) of the ALP-strahlung production process $pp\to aW^{\pm}$ and the corresponding backgrounds at the 14 TeV LHC. The benchmark point is chosen as $m_{a}$=3 GeV and $g_{a\gamma\gamma}=16$ TeV-1.  cut flow |  signal |  | $Z\gamma$
---
| $Zj$
---
| 2leptons with
---
$p_{\text{T}}>20$ GeV
and $|\eta|<2.5$
2.65 | 1.93 | 279.90
| oppositely charged lepton
---
pair with same flavor
and $|m_{ll}-m_{Z}|$<20 GeV
2.55 | 1.85 | 275.08
| The hardest jet
---
with $p_{\text{T}}>50$ GeV
and $|\eta|<2.5$
1.79 | 0.44 |  104.476
| The hardest jet’s
---
BDT-1 > 0
and BDT-2 > 0.4
0.29 |  0.0003 | 0.027
Table 2: Same as Table I, but for the ALP-strahlung production process $pp\to
aZ$ and the corresponding backgrounds.

As an example, we consider a benchmark signal point with $m_{a}=3$ GeV and
$g_{a\gamma\gamma}=16$ TeV-1. The cut-flows of this benchmark point and the
main SM backgrounds are shown in Tables 1 and 2. It can be seen that after the
BDT response cut, all the backgrounds are reduced dramatically. At the end of
the cut flow, the largest backgrounds for the processes $pp\to W^{\pm}a$ and
$pp\to Za$ signal are QCD di-jet and $Zj$, respectively.

Figure 6: The $2\sigma$ exclusion limits on the plane of the ALP mass $m_{a}$
versus the coupling $g_{a\gamma\gamma}$ from the ALP-strahlung production
process $pp\to aW^{\pm}/Z$ at the LHC with luminosity 3000 fb-1. Other limits
from the LEP Mimasu and Sanz (2015); Jaeckel and Spannowsky (2016), the
SN1987a Payez et al. (2015); Jaeckel et al. (2018), the Beam Dump Döbrich et
al. (2016, 2019), the ATLAS/CMS Aad et al. (2013); Jaeckel et al. (2013);
Jaeckel and Spannowsky (2016); Knapen et al. (2017), the rare $K$ and $B$
meson decays Izaguirre et al. (2017) are also shown. It should be noted that
the limits involving the ALP-gluon couplings are not shown but discussed in
the context.

In Fig. 6, we present the $2\sigma$ bound from our ALP-strahlung production
process $pp\to aW^{\pm}/Z$ on the plane of the ALP coupling
$g_{a\gamma\gamma}$ versus the ALP mass $m_{a}$. We use the Poisson formula
$\sqrt{2\left[(\mathcal{S}+\mathcal{B}){\rm
ln}(1+\mathcal{S}/\mathcal{B})-\mathcal{S}\right]}$ to estimate the
sensitivity at the LHC with the luminosity of 3000 fb-1, where $\mathcal{S}$
and $\mathcal{B}$ are the number of signal and background events. As the
comparison with our results, we also show other relevant bounds on the ALP-
photon coupling. Since we only focus on the ALP-EW boson couplings in this
work, the limits involving the ALP-gluon couplings are not shown in Fig. 6.

It can be seen that our photon-jet method can cover the ALP mass region of 0.3
GeV $<m_{a}<$ 10 GeV at the LHC, which can extend the current LHC sensitivity
of searching for the diphoton resonances in photon fusion and vector boson
fusion processes Aad et al. (2013); Jaeckel et al. (2013); Jaeckel and
Spannowsky (2016); Knapen et al. (2017). In addition, our bound on the ALP-
photon coupling $g_{a\gamma\gamma}$ is stronger than that derived from the
null results of searching for the ALPs in $e^{+}e^{-}\to 2\gamma/3\gamma$
processes at the LEP Jaeckel and Spannowsky (2016). For example,
$g_{a\gamma\gamma}$ in our study is constrained to be less than 2.4 TeV-1 at
$m_{a}=0.3$ GeV and 1.5 TeV-1 at $m_{a}=10$ GeV. On the other hand, we note
that searching for $3\gamma$ events from the process $e^{+}e^{-}\to
a(\to\gamma\gamma)\gamma$ and the rare meson decays, such as
$K_{L}\to\pi^{0}a$, $K^{\pm}\to\pi^{\pm}a$ and $B\to K^{(*)}a$, in the
existing and future low energy experiments may provide more stringent
constraints than ours when $m_{a}\lesssim 5$ GeV Izaguirre et al. (2017).

Besides, if the ALP-gluon couplings are present and not suppressed by at least
two orders of magnitude relative to the electroweak ones, the current and
future measurements of $B$ meson decays at the LHCb and B-factories can also
give strong exclusion limits for $m_{a}<10$ GeV Cid Vidal et al. (2019); Lees
et al. (2011). However, it should be mentioned that this result is not
applicable to our case because the ALP-gluon couplings are absent in our
scenario. For the same reason, we do not show the constraints from isolated
and energetic photons from hadronic decay of $Z$ boson at L3 Adriani et al.
(1992) and the searches of ATLAS and CMS for $\gamma\gamma$ resonances in the
gluon fusion process Mariotti et al. (2018).

##  IV Conclusions

In this work we extended the current LHC sensitivity of probing the
electroweak ALP via the ALP-strahlung production processes $pp\to aW^{\pm}/Z$
in the mass range of 0.3 GeV $<m_{a}<$ 10 GeV at the 14 TeV HL-LHC. Since the
two photons from ALP decay are not well separated for such a light ALP, we
used jet substructure variables and proposed photon-jet method to discriminate
our signal from QCD jets and single photons backgrounds. With the help of BDT
technique, we obtained the $2\sigma$ bounds on the ALP-photon coupling
$g_{a\gamma\gamma}$ as a function of $m_{a}$. The coupling
$g_{a\gamma\gamma}>$ 2.4 TeV-1 at $m_{a}=0.3$ GeV and $g_{a\gamma\gamma}>$ 1.5
TeV-1 at $m_{a}=10$ GeV can be excluded at $2\sigma$ level at the 14 TeV HL-
LHC. This demonstrates that our approach can not only cover the mass gap 0.3
GeV $<m_{a}<$ 10 GeV at the LHC, but also surpass the existing LEP bounds.
Although the measurements of rare meson decays may also exclude the ALP in
MeV-scale mass range, a direct LHC search would provide an independent probe
of this parameter space.

##  V acknowledgments

This work is supported by the National Natural Science Foundation of China
(NNSFC) under grant Nos. 11805161, 12075300, 11821505, 11947118, by Peng-Huan-
Wu Theoretical Physics Innovation Center (12047503), by the CAS Center for
Excellence in Particle Physics (CCEPP), by the CAS Key Research Program of
Frontier Sciences, by a Key R&D Program of Ministry of Science and Technology
under number 2017YFA0402204, and by the Key Research Program of the Chinese
Academy of Sciences, grant No. XDPB15.

## References

  * Peccei and Quinn (1977) R. D. Peccei and H. R. Quinn, Phys. Rev. Lett. 38, 1440 (1977).
  * Weinberg (1978) S. Weinberg, Phys. Rev. Lett. 40, 223 (1978).
  * Wilczek (1978) F. Wilczek, Phys. Rev. Lett. 40, 279 (1978).
  * Kim (1979) J. E. Kim, Phys. Rev. Lett. 43, 103 (1979).
  * Svrcek and Witten (2006) P. Svrcek and E. Witten, JHEP 06, 051 (2006), eprint hep-th/0605206.
  * Arvanitaki et al. (2010) A. Arvanitaki, S. Dimopoulos, S. Dubovsky, N. Kaloper, and J. March-Russell, Phys. Rev. D 81, 123530 (2010), eprint 0905.4720.
  * Cicoli et al. (2012) M. Cicoli, M. Goodsell, and A. Ringwald, JHEP 10, 146 (2012), eprint 1206.0819.
  * Ballesteros et al. (2017) G. Ballesteros, J. Redondo, A. Ringwald, and C. Tamarit, Phys. Rev. Lett. 118, 071802 (2017), eprint 1608.05414.
  * Graham et al. (2015) P. W. Graham, D. E. Kaplan, and S. Rajendran, Phys. Rev. Lett. 115, 221801 (2015), eprint 1504.07551.
  * Raffelt (1990) G. G. Raffelt, Phys. Rept. 198, 1 (1990).
  * Marsh (2016) D. J. E. Marsh, Phys. Rept. 643, 1 (2016), eprint 1510.07633.
  * Preskill et al. (1983) J. Preskill, M. B. Wise, and F. Wilczek, Phys. Lett. B 120, 127 (1983).
  * Abbott and Sikivie (1983) L. F. Abbott and P. Sikivie, Phys. Lett. B 120, 133 (1983).
  * Dine and Fischler (1983) M. Dine and W. Fischler, Phys. Lett. B 120, 137 (1983).
  * Arias et al. (2012) P. Arias, D. Cadamuro, M. Goodsell, J. Jaeckel, J. Redondo, and A. Ringwald, JCAP 06, 013 (2012), eprint 1201.5902.
  * Jaeckel et al. (2014) J. Jaeckel, J. Redondo, and A. Ringwald, Phys. Rev. D 89, 103511 (2014), eprint 1402.7335.
  * Athron et al. (2021) P. Athron et al., JHEP 05, 159 (2021), eprint 2007.05517.
  * Gao et al. (2020) C. Gao, J. Liu, L.-T. Wang, X.-P. Wang, W. Xue, and Y.-M. Zhong, Phys. Rev. Lett. 125, 131806 (2020), eprint 2006.14598.
  * Izaguirre et al. (2017) E. Izaguirre, T. Lin, and B. Shuve, Phys. Rev. Lett. 118, 111802 (2017), eprint 1611.09355.
  * Dolan et al. (2017) M. J. Dolan, T. Ferber, C. Hearty, F. Kahlhoefer, and K. Schmidt-Hoberg, JHEP 12, 094 (2017), eprint 1709.00009.
  * Bauer et al. (2020) M. Bauer, M. Neubert, S. Renner, M. Schnubel, and A. Thamm, Phys. Rev. Lett. 124, 211803 (2020), eprint 1908.00008.
  * Banerjee et al. (2020) D. Banerjee et al. (NA64), Phys. Rev. Lett. 125, 081801 (2020), eprint 2005.02710.
  * Gu et al. (2021) Y. Gu, L. Wu, and B. Zhu (2021), eprint 2105.07232.
  * Mimasu and Sanz (2015) K. Mimasu and V. Sanz, JHEP 06, 173 (2015), eprint 1409.4792.
  * Knapen et al. (2017) S. Knapen, T. Lin, H. K. Lou, and T. Melia, Phys. Rev. Lett. 118, 171801 (2017), eprint 1607.06083.
  * Barrie et al. (2016) N. D. Barrie, A. Kobakhidze, M. Talia, and L. Wu, Phys. Lett. B 755, 343 (2016), eprint 1602.00475.
  * Bauer et al. (2017a) M. Bauer, M. Neubert, and A. Thamm, JHEP 12, 044 (2017a), eprint 1708.00443.
  * Brivio et al. (2017) I. Brivio, M. B. Gavela, L. Merlo, K. Mimasu, J. M. No, R. del Rey, and V. Sanz, Eur. Phys. J. C 77, 572 (2017), eprint 1701.05379.
  * Bauer et al. (2019) M. Bauer, M. Heiles, M. Neubert, and A. Thamm, Eur. Phys. J. C 79, 74 (2019), eprint 1808.10323.
  * Ebadi et al. (2019) J. Ebadi, S. Khatibi, and M. Mohammadi Najafabadi, Phys. Rev. D 100, 015016 (2019), eprint 1901.03061.
  * Wang et al. (2021) D. Wang, L. Wu, and M. Zhang, Phys. Rev. D 103, 115017 (2021), eprint 2007.09722.
  * Ren et al. (2021) J. Ren, D. Wang, L. Wu, J. M. Yang, and M. Zhang (2021), eprint 2106.07018.
  * Jaeckel and Spannowsky (2016) J. Jaeckel and M. Spannowsky, Phys. Lett. B 753, 482 (2016), eprint 1509.00476.
  * Gavela et al. (2020) M. B. Gavela, J. M. No, V. Sanz, and J. F. de Trocóniz, Phys. Rev. Lett. 124, 051802 (2020), eprint 1905.12953.
  * Bauer et al. (2017b) M. Bauer, M. Neubert, and A. Thamm, Phys. Rev. Lett. 119, 031802 (2017b), eprint 1704.08207.
  * Ellis et al. (2013a) S. D. Ellis, T. S. Roy, and J. Scholtz, Phys. Rev. Lett. 110, 122003 (2013a), eprint 1210.1855.
  * Ellis et al. (2013b) S. D. Ellis, T. S. Roy, and J. Scholtz, Phys. Rev. D 87, 014015 (2013b), eprint 1210.3657.
  * Dobrescu et al. (2001) B. A. Dobrescu, G. L. Landsberg, and K. T. Matchev, Phys. Rev. D 63, 075003 (2001), eprint hep-ph/0005308.
  * Chang et al. (2007) S. Chang, P. J. Fox, and N. Weiner, Phys. Rev. Lett. 98, 111802 (2007), eprint hep-ph/0608310.
  * Toro and Yavin (2012) N. Toro and I. Yavin, Phys. Rev. D 86, 055005 (2012), eprint 1202.6377.
  * Draper and McKeen (2012) P. Draper and D. McKeen, Phys. Rev. D 85, 115023 (2012), eprint 1204.1061.
  * Alloul et al. (2014) A. Alloul, N. D. Christensen, C. Degrande, C. Duhr, and B. Fuks, Comput. Phys. Commun. 185, 2250 (2014), eprint 1310.1921.
  * Alwall et al. (2014) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, JHEP 07, 079 (2014), eprint 1405.0301.
  * Sjöstrand et al. (2015) T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen, and P. Z. Skands, Comput. Phys. Commun. 191, 159 (2015), eprint 1410.3012.
  * de Favereau et al. (2014) J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi (DELPHES 3), JHEP 02, 057 (2014), eprint 1307.6346.
  * Cacciari et al. (2012) M. Cacciari, G. P. Salam, and G. Soyez, Eur. Phys. J. C 72, 1896 (2012), eprint 1111.6097.
  * CMS (2009) (2009).
  * Cacciari et al. (2008) M. Cacciari, G. P. Salam, and G. Soyez, JHEP 04, 063 (2008), eprint 0802.1189.
  * Catani et al. (1993) S. Catani, Y. L. Dokshitzer, M. H. Seymour, and B. R. Webber, Nucl. Phys. B 406, 187 (1993).
  * Ellis and Soper (1993) S. D. Ellis and D. E. Soper, Phys. Rev. D 48, 3160 (1993), eprint hep-ph/9305266.
  * Thaler and Van Tilburg (2011) J. Thaler and K. Van Tilburg, JHEP 03, 015 (2011), eprint 1011.2268.
  * Thaler and Van Tilburg (2012) J. Thaler and K. Van Tilburg, JHEP 02, 093 (2012), eprint 1108.2701.
  * Butterworth et al. (2008) J. M. Butterworth, A. R. Davison, M. Rubin, and G. P. Salam, Phys. Rev. Lett. 100, 242001 (2008), eprint 0802.2470.
  * Roe et al. (2005) B. P. Roe, H.-J. Yang, J. Zhu, Y. Liu, I. Stancu, and G. McGregor, Nucl. Instrum. Meth. A 543, 577 (2005), eprint physics/0408124.
  * Hocker et al. (2007) A. Hocker et al. (2007), eprint physics/0703039.
  * Payez et al. (2015) A. Payez, C. Evoli, T. Fischer, M. Giannotti, A. Mirizzi, and A. Ringwald, JCAP 02, 006 (2015), eprint 1410.3747.
  * Jaeckel et al. (2018) J. Jaeckel, P. C. Malta, and J. Redondo, Phys. Rev. D 98, 055032 (2018), eprint 1702.02964.
  * Döbrich et al. (2016) B. Döbrich, J. Jaeckel, F. Kahlhoefer, A. Ringwald, and K. Schmidt-Hoberg, JHEP 02, 018 (2016), eprint 1512.03069.
  * Döbrich et al. (2019) B. Döbrich, J. Jaeckel, and T. Spadaro, JHEP 05, 213 (2019), [Erratum: JHEP 10, 046 (2020)], eprint 1904.02091.
  * Aad et al. (2013) G. Aad et al. (ATLAS), JHEP 01, 086 (2013), eprint 1211.1913.
  * Jaeckel et al. (2013) J. Jaeckel, M. Jankowiak, and M. Spannowsky, Phys. Dark Univ. 2, 111 (2013), eprint 1212.3620.
  * Cid Vidal et al. (2019) X. Cid Vidal, A. Mariotti, D. Redigolo, F. Sala, and K. Tobioka, JHEP 01, 113 (2019), [Erratum: JHEP 06, 141 (2020)], eprint 1810.09452.
  * Lees et al. (2011) J. P. Lees et al. (BaBar), Phys. Rev. Lett. 107, 221803 (2011), eprint 1108.3549.
  * Adriani et al. (1992) O. Adriani et al. (L3), Phys. Lett. B 292, 472 (1992).
  * Mariotti et al. (2018) A. Mariotti, D. Redigolo, F. Sala, and K. Tobioka, Phys. Lett. B 783, 13 (2018), eprint 1710.01743.


# Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models

Konstantinos Vilouras  0009-0003-5910-9748 Pedro Sanchez  0000-0003-2435-3049
Alison Q. O’Neil  0000-0001-8371-0603 and Sotirios A. Tsaftaris
0000-0002-8795-9294 Senior Member, IEEE Manuscript submitted for review on 15
March 2024. This work was supported by the University of Edinburgh, the Royal
Academy of Engineering and Canon Medical Research Europe by PhD studentships
to Konstantinos Vilouras and Pedro Sanchez. S.A. Tsaftaris also acknowledges
the support of Canon Medical and the Royal Academy of Engineering and the
Research Chairs and Senior Research Fellowships scheme (grant RCSRF1819\8\25),
and the UK’s Engineering and Physical Sciences Research Council (EPSRC)
support via grant EP/X017680/1.Konstantinos Vilouras, Pedro Sanchez, Sotirios
A. Tsaftaris are with the School of Engineering, University of Edinburgh,
Edinburgh EH9 3FB, United Kingdom (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>s.tsaftaris@ed.ac.uk). Alison O’Neil is with Canon
Medical Research Europe Ltd., Edinburgh EH6 5NP, United Kingdom (e-mail:
<EMAIL_ADDRESS>and with the School of Engineering, University
of Edinburgh, Edinburgh EH9 3FB, United Kingdom.

###### Abstract

Localizing the exact pathological regions in a given medical scan is an
important imaging problem that requires a large amount of bounding box ground
truth annotations to be accurately solved. However, there exist alternative,
potentially weaker, forms of supervision, such as accompanying free-text
reports, which are readily available. The task of performing localization with
textual guidance is commonly referred to as phrase grounding. In this work, we
use a publicly available Foundation Model, namely the Latent Diffusion Model,
to solve this challenging task. This choice is supported by the fact that the
Latent Diffusion Model, despite being generative in nature, contains
mechanisms (cross-attention) that implicitly align visual and textual
features, thus leading to intermediate representations that are suitable for
the task at hand. In addition, we aim to perform this task in a zero-shot
manner, i.e., without any further training on target data, meaning that the
model’s weights remain frozen. To this end, we devise strategies to select
features and also refine them via post-processing without extra learnable
parameters. We compare our proposed method with state-of-the-art approaches
which explicitly enforce image-text alignment in a joint embedding space via
contrastive learning. Results on a popular chest X-ray benchmark indicate that
our method is competitive wih SOTA on different types of pathology, and even
outperforms them on average in terms of two metrics (mean IoU and AUC-ROC).
Source code will be released upon acceptance.

Deep Learning, Diffusion Models, Medical Imaging, Phrase Grounding, Zero-shot
learning

##  1 Introduction

The rapid success of deep learning over the last few years has led to powerful
data-driven models being deployed already in real-world scenarios. Recently,
by taking advantage of the scaling properties of popular deep learning methods
both in terms of learnable parameters and training data, we witness the era of
Foundation Models (FMs) [1], i.e., large-scale neural networks that were
trained on massive amounts of data. FMs have unprecedented capabilities: they
can be readily applied to a wide variety of tasks as standalone solutions, or
they can serve as a robust basis for training models on unknown tasks and,
plausibly, modalities (e.g., transferring knowledge from natural images to the
medical domain). Among their many benefits, FMs provide machine learning
practitioners and researchers with a universal tool that enables the
widespread application of data-driven solutions to multiple scientific fields,
as well as the development of a sound theoretical framework around well-known
deep learning methodologies.

Figure 1: High-level description of the zero-shot phrase grounding task. Given
input pairs of an image (chest X-ray) and its accompanying text prompt, our
goal is to leverage feature alignment mechanisms within a frozen Latent
Diffusion Model (LDM) to extract heatmaps, which indicate the regions where
image and text are maximally aligned. Then, we evaluate the generated heatmaps
based on ground truth bounding boxes (shown in green) for pathology detection.
Our method, thus, is an illustration of using pre-trained LDMs for downstream
applications in a zero-shot setting.

In this work, we investigate a specific type of FM: the Latent Diffusion
Model. We draw inspiration from research studies showing that diffusion models
can solve downstream tasks such as classification [2, 3] and segmentation [4]
with little to no additional fine-tuning on target data. Here, we attempt to
validate this finding, that is the re-usability of diffusion models for
downstream tasks, in the medical imaging domain, focusing on the phrase
grounding task where the goal is to localize the pathology in an input scan
(e.g., chest X-ray) given an accompanying textual description provided by
expert clinicians. A high-level overview of our method for the phrase
grounding task is shown in Fig. 1.

We perform extensive experiments on an established phrase grounding benchmark
dataset, i.e., MS-CXR [5]. The results suggest that, despite being inherently
a generative model, the Latent Diffusion Model has learned high quality
features for the task at hand. Our proposed approach, which tackles the
extreme case where no additional fine-tuning is performed (we refer to it as a
zero-shot scenario), yields, perhaps surprisingly, a highly competitive method
that proves to be state-of-the-art in terms of two metrics (mean IoU and AUC-
ROC) on average across 8 pathology labels.

Overall, our contributions can be summarized as follows:

  * •

Building on the Latent Diffusion Model architecture, we propose a method to
gather semantically meaningful visual features both from multiple timesteps of
the reverse diffusion process and also from various cross-attention layers. We
target those layers since they inherently align information from the visual
and textual stream, thus being suitable for the phrase grounding task.

  * •

We perform extensive experiments on a medical dataset with ground truth
bounding boxes and test our method against several strong baselines. Results
indicate that our method shows competitive performance against, and even
exceeds in the case of two metrics, the current state-of-the-art without any
fine-tuning strategies.

  * •

We provide an ablation study to justify the different hyperparameter choices
in our system.

  * •

We further analyze qualitatively our proposed method, as well as the strongest
available baseline, to provide useful insights for both approaches.

##  2 Related Work

We now summarize the most relevant research. We start by mentioning prior
influential works showing that diffusion models can be effectively applied to
multiple downstream tasks in a zero- or few-shot setting. Then, we briefly
discuss some of the most popular methods for phrase grounding in natural RGB
images. Finally, we shift our focus to approaches related to the medical
imaging domain, against which we compare our proposed method.

###  2.1 Downstream Application of Diffusion Models

Recent studies have revealed the ability of diffusion models to perform fairly
well on downstream tasks. For example, for classification tasks it has been
shown that the posterior $p(\mathbf{c}|\mathbf{x})$ for all candidate classes
$\mathbf{c}$ can be estimated from a diffusion model’s residual errors at a
given timestep without any requirement for further hyperparameters or training
[2, 3]. Similarly, for segmentation, [4] use intermediate visual
representations extracted from a diffusion model. In this case, a few
additional labelled images are required to train a shallow network that
outputs pixel-wise predictions. In another work, [6] experiment with the text
feature extraction pipeline, as well as the choice of intermediate visual
features, and train lightweight task-specific models for segmentation and
depth estimation, respectively.

###  2.2 Visual Phrase Grounding

Phrase grounding is a cross-modal reasoning task referring to the spatial
localization of objects present in an image given a related text description.
Due to the widespread availability of image-text data sources, there already
exist large-scale end-to-end models for the natural image domain. In a
discriminative learning scenario, ViLBERT [7] introduces the concept of co-
attention, i.e., exchanging information between modalities within the
transformer layers. Early fusion strategies are also adopted by MDETR [8]
which enforces image-text alignment with appropriate learning objectives, and
GLIP [9] in which image and language encoders are simultaneously trained to
correctly assign a word to a specific image region. [10] follow a generative
learning approach by training a text-guided diffusion model to gradually
recover ground-truth bounding boxes from their noise-perturbed versions.

###  2.3 Medical Phrase Grounding

In the medical imaging domain, phrase grounding is considered a difficult task
due to the inherent variation in textual information (radiologists, e.g.,
commonly use domain-specific terms, negations, or phrases that convey a level
of uncertainty). Earlier works have taken different approaches. [11] extract
pneumonia-related attributes from radiology reports, while a pre-trained
bounding box detector is used to extract regions of interest (ROIs) and their
associated features. Then, using both streams of information, their system is
trained to correctly classify attributes from visual features, as well as to
maximize the similarity for a given image-text pair. End-to-end discriminative
methods to date are largely fully supervised [12, 13] or rely on self-
supervised contrastive formulations [14, 5, 15]. Specifically, [12] train a
vision-language Transformer model to directly predict bounding boxes, whereas
[13] gather publicly available labelled chest X-ray datasets and train a
single model via multi-task learning. LIMITR [14] is a self-supervised method
that aligns local cross-modal representations that are further weighted via
learnable significance scores. Lastly, BioViL [5] is an end-to-end model with
a BERT text encoder finetuned on radiology reports that is optimized via both
global and local cross-modal contrastive losses. [15] extend the BioViL model
to support longitudinal information across patients, and the resulting system
(BioViL-T) achieves state-of-the-art performance on medical phrase grounding.

In contrast, we adopt a generative approach to this task. A concurrent work
also applies diffusion models to phrase grounding [16]. Our approach differs
in several ways: First, we use a pre-trained, publicly available diffusion
model and propose mechanisms to perform phrase grounding whilst keeping the
model frozen, whereas [16] focuses on training the model from scratch. Second,
to define these mechanisms, we pay attention to how we select cross-attention
layers and timesteps, whereas [16] simply average across both time and layers.
Finally, we compare our method with more recent state-of-the-art and other,
recently proposed, baselines. Nevertheless, the findings of [16] are
beneficial for our work. They show that learning both text and visual encoders
simultaneously severely degrades performance. In addition, regarding the
textual information that is available for phrase grounding, they show that
using sentences from the original radiology reports (as is the case, for
example, for the MS-CXR [5] dataset) leads to the best overall results,
outperforming both synthetically generated text (with ChatGPT) and the
simplest case of using the class string as the input prompt.

##  3 Methodology

Figure 2: Overview of our proposed phrase grounding pipeline based on the
Latent Diffusion Model architecture [17]. The input image $x$ is first passed
through the encoder $E$. The resulting feature vector $z$ is gradually mapped
into an isotropic Gaussian after $T$ time steps of the forward diffusion
process. Then, a denoising U-Net $\boldsymbol{\epsilon_{\theta}}$ is tasked to
sequentially transform noisy latents in reverse order, i.e., from step $t$ to
$t-1$, $t=1,...,T$. To further aid this task, text embeddings (generated from
the encoder $\boldsymbol{\tau_{\theta}}$) are also introduced in the U-Net’s
cross-attention blocks. We intervene on this setup by gathering the $N$
attention maps from all cross-attention blocks within the U-Net
$\boldsymbol{\epsilon_{\theta}}$. Thus, by averaging a total of $N\cdot T$
attention maps, we end up with an activation heatmap $\mathbf{h}$ for the
given input.

We adopt the following notation: x is a scalar, $x$ denotes a vector,
$\mathbf{x}$ is a tensor, $\mathbf{f_{\theta}}$ refers to the total number of
a network’s $\mathbf{f}$ learnable parameters (weights), $\mathcal{X}$ stands
for a vector space, and $\mathbf{X}:\mathcal{X}\rightarrow\mathcal{X}$ is a
neural network that performs a mapping between two vector spaces. We also
consider a dataset $\mathcal{D}=\\{\mathbf{x_{i}},p_{i}\\}_{i=1}^{N}$ with
image-text pairs, where $N$ is the total number of samples. For brevity, we
will next discuss our pipeline while considering a single input
$(\mathbf{x},p)$; the extension to a batch of samples is trivial.

###  3.1 Overview of the Latent Diffusion Model

The Latent Diffusion Model (LDM) [17] belongs to the class of probabilistic
models and is considered one of the most groundbreaking methods for image
synthesis. The versatile design of the LDM has significantly contributed
towards its success, as external sources of information (e.g., text,
segmentation masks, or any other type of a representation) can be easily
incorporated into the model without any architectural changes. Here, we follow
the standard setup of the LDM architecture as proposed in [17]. More
specifically, via the encoder $\mathbf{E}:\mathcal{X}\rightarrow\mathcal{Z}$,
the input image $\mathbf{x}$ is first encoded into a latent feature vector $z$
which, in turn, is progressively corrupted with Gaussian noise via a
deterministic forward diffusion process for a total of T timesteps. Then, for
each timestep t, a denoising U-Net
$\boldsymbol{\epsilon_{\theta}}:\mathcal{Z}\rightarrow\mathcal{Z}$ is applied
to the noisy latent $z_{t}$ to recover the cleaner estimate at the previous
step $z_{t-1}$. After the reverse diffusion process, the recovered global
latent $z$ is mapped to the original pixel space via a decoder
$\mathbf{D}:\mathcal{Z}\rightarrow\mathcal{X}$, leading to the reconstructed
input $\mathbf{\tilde{x}}$. An overview of the LDM pipeline is depicted in
Fig. 2.

###  3.2 Image-Text Alignment via Cross-Attention

To aid reconstruction, textual information about the image $\mathbf{x}$ is
also injected in the U-Net model $\boldsymbol{\epsilon_{\theta}}$ in the form
of a natural language prompt. Concretely, the prompt $p$ that accompanies the
input sample $\mathbf{x}$ is projected into an embedding via a pre-trained
text encoder $\boldsymbol{\tau_{\theta}}$. The resulting text feature vector
serves as context in each cross-attention block of the U-Net, i.e., it is used
to generate keys $\mathbf{k}$ and values $\mathbf{v}$ for the self-attention
operation as defined below, where $\mathbf{q}$ refers to the queries extracted
from the visual features, $\mathbf{P_{K}}$ and $\mathbf{P_{V}}$ are projection
layers for the keys and values, respectively, and $d$ is a scaling factor,

$\displaystyle\mathbf{k}=\mathbf{P_{K}}(\tau_{\theta}(p))$
$\displaystyle\qquad\mathbf{v}=\mathbf{P_{V}}(\tau_{\theta}(p))$ (1)
$\displaystyle\text{Attention}(\mathbf{q},\mathbf{k},\mathbf{v})$
$\displaystyle=\text{softmax}(\frac{\mathbf{q}\mathbf{k}^{\top}}{\sqrt{d}})\cdot\mathbf{v}.$

From Eq. (1), we observe that the LDM architecture contains a straightforward
mechanism that aligns visual and textual features. Moreover, the model’s
design provides access to information at different levels of granularity,
since there are multiple cross-attention blocks in the U-Net, while the
generated attention maps also vary across timesteps. Let N denote the total
number of cross-attention blocks in the U-Net model and T the number of
timesteps used for the diffusion process. That leaves us with $N\cdot T$
attention maps in total, where each one varies both on the content of the
visual features (low-level features or higher-level semantics) or at the
timestep of the reverse diffusion process, which directly affects the noise
level of each latent $z_{t}$.

###  3.3 Zero-shot Phrase Grounding with the LDM

The trained LDM can be readily used to perform phrase grounding in a zero-shot
setting. After gathering the $N\cdot T$ attention maps, we need to aggregate
them to extract a single activation heatmap $\mathbf{h}$ that matches the
original input’s, $\mathbf{x}$, spatial dimensions (after appropriate up-
scaling). In the ideal case of perfect image-text alignment, $\mathbf{h}$ will
be highly activated on the regions of interest specified by the prompt. In
this paper, we simply average the gathered attention maps across layers and
timesteps, experimenting with the selection of cross-attention blocks and
timesteps to optimise phrase grounding performance. We note that more
elaborate methods might be devised, however we leave this for future work. The
red block in Fig. 1 shows the process for generating the heatmap $\mathbf{h}$.
Furthermore, as detailed in section 4.5, results can be refined by applying
additional post-processing techniques on the generated heatmap $\mathbf{h}$
that do not involve any learnable parameters or any form of supervision, thus
not violating the zero-shot scenario.

##  4 Experimental Results

###  4.1 LDM Pre-training Setup

First, we briefly discuss the LDM pre-training stage. Here, we use an open-
source implementation of the LDM along with a provided checkpoint; thus, we do
not train the LDM ourselves. For our experiments, we use the LDM architecture
as defined in [18]. The model is pre-trained on MIMIC-CXR [19], a large-scale
dataset of chest radiographs accompanied by free-text reports. The training
set consists of 368,960 chest X-ray images, whereas the text prompt for each
image is a randomly sampled sentence from the corresponding radiology report,
from either the Impressions or Findings section. Note that the pre-trained
weights, as well as the configuration files with all training-related details,
are publicly available in the MONAI Generative framework [20].

###  4.2 Phrase Grounding Setup

The phrase grounding task is performed with the following setup: The sampled
input image is first transformed into a fixed spatial dimension of $512\times
512$. For the diffusion process, we set the scheduler’s total timesteps to
T$=300$. In terms of text processing, we use the standard CLIP text encoder
with frozen weights. Prior to encoding, the input prompt is tokenized and then
either padded or truncated to match the maximum sequence length ($S=77$
tokens). Since the CLIP text encoder is not trained on radiology reports,
domain-specific terms such as “atelectasis” may be out-of-vocabulary (OOV).
The CLIP tokenizer handles any unknown OOV token by splitting it into multiple
known sub-tokens.

The size of the gathered attention maps is $\mathbb{R}^{S\times B\times
B\times L\times T}$, where $L=11$ is the total number of cross-attention
layers in the U-Net and $B$ is the spatial size (either 16 or 32, depending on
the layer’s level in the U-Net architecture). We first average across the
token sequence dimension $S$, excluding any padding tokens. Then, the
intensities within the spatial maps are normalized to span the range $[0,1]$,
and the maps are resized to the original input’s dimension with bi-linear
interpolation. After averaging across the layer $L$ and timestep $T$
dimensions, following [5, 15], the resulting heatmap with resolution
$512\times 512$ is convolved with a Gaussian kernel ($\sigma=2.5$).

To further optimise performance, we experiment with the choice of layers and
timesteps, as well as with post-processing techniques to enhance the resulting
heatmap’s quality. Our best-performing setup is reported in Subsection 4.5,
and our choices are justified via ablations found in Subsection 4.9.

###  4.3 Evaluation Dataset

We evaluate our proposed system on the MS-CXR benchmark [5] which consists of
1,153 image-sentence pairs with ground truth bounding boxes indicating the
pathology. Note that this dataset is extracted from the official MIMIC-CXR
test set. We further pre-process the dataset by merging entries corresponding
to the same patient and the same sentence, i.e., an image might have more than
one ground truth bounding box as reference for a given text prompt, where the
pathology appears in multiple locations. We made the choice to compute metrics
on a per-image basis (and not per bounding box) as this matched the
performance reported by [5, 15].

###  4.4 Baselines

We compare our proposed method to 4 state-of-the-art baselines trained in a
discriminative manner, using either fully supervised or self-supervised
learning. Fully supervised networks (MedRPG [12], OmniFM-DR [13]) are trained
to correctly predict the ground truth bounding box, whereas the self-
supervised networks (BioViL [5], BioViL-T [15]) are trained to maximize the
cosine similarity between visual and textual features without any further
ground truth annotations in a self-supervised manner. We evaluate our system
against these 4 methods only since they were shown to outperform previous
approaches; in fact, BioViL-T [15], a discriminative self-supervised model,
sets the current state-of-the-art on phrase grounding. For all baseline
methods considered in this work, we use the publicly available pre-trained
model checkpoints provided by the respective authors.

To the best of our knowledge, a generative baseline does not yet exist for the
downstream task of medical phrase grounding. Note that, at the time of
writing, the concurrent work [16] has not yet released weights for their own
version of the LDM.

###  4.5 Our Method

We first report the hyperparameter choices leading to our best performing
setup. Inspired by a prior work that performs semantic segmentation with
diffusion models [4], we gather attention maps only from the middle timesteps
and middle cross-attention blocks. This choice is motivated by the following
two observations. During the initial timesteps of the reverse diffusion
process, the reconstructed sample’s global structure has not yet been modelled
properly, thus attention masks do not convey any semantic meaning; on the
contrary, towards the final timesteps, fine-grained details are added to the
generated sample which, according to [4], hurt segmentation performance.
Therefore, using the middle timesteps strikes a balance. Correspondingly,
earlier U-Net blocks contain more information about small objects in the
image, while deeper blocks are more informative for larger objects. This, in
turn, means that in the former case the generated heatmap would highly
activate over the entire image area, whereas in the latter case we would get
high activations on large anatomical structures. Therefore, focusing on middle
layers strikes a balance between these two extreme cases.

Similarly, we find that optimal results are achieved when attention maps are
collected from $L=4$ layers (i.e., from the $3^{\text{rd}}$, $4^{\text{th}}$,
$6^{\text{th}}$ and $7^{\text{th}}$ cross-attention blocks), which
automatically sets their spatial size to $B=16$, and also from 60 different
timesteps (i.e., from timestep 120 to 180) out of $T=300$ steps in total.
Note, however, that we did not perform an exhaustive search to find the
optimal combination of selected layers and timesteps; in fact, it is possible
that different settings per pathology might lead to better overall
performance. We leave this for future work.

Furthermore, based on our empirical observation that the resulting heatmap
activates widely over the entire input area, we apply binary-Otsu thresholding
[21] as a post-processing step to separate the more strongly activating
foreground from more weakly activating background. The output binary mask is
then applied to the heatmap to suppress weak signals while leaving the
foreground activations unaffected.

###  4.6 Metrics

Given a predicted heatmap $H$ (with values in the $[0,1]$ range) and ground
truth binary segmentation mask $M_{GT}$ which has ones within each bounding
box area and zero otherwise, phrase grounding performance is measured via the
following evaluation metrics:

####  4.6.1 Mean Intersection over Union (mIoU)

mIoU is a standard metric to evaluate segmentation performance. The predicted
binary mask at threshold $t$ is defined as $M_{H}=\\{h\in H:h>t\\}$. Then, IoU
at given threshold $t$ is calculated as

$IoU_{@t}=\frac{|M_{H}\cap M_{GT}|}{|M_{H}\cup M_{GT}|}.$ (2)

Here, following [5], we calculate mIoU as the average over 5 different
thresholds $t\in[0.1,0.2,0.3,0.4,0.5]$.

####  4.6.2 Area Under ROC Curve (AUC-ROC)

The area under the receiver operating characteristic curve is another method
to estimate segmentation performance on a per-pixel basis given $H$ and
$M_{GT}$. Here, we compute an estimate of AUC-ROC as the mean across 5
thresholds $t\in[0,0.25,0.5,0.75,1.0]$.

####  4.6.3 Contrast-to-Noise Ratio (CNR)

CNR as used in [5] is a threshold-agnostic measure that reflects the
distribution of raw heatmap activations over the entire input. Let $A$ denote
the area within each bounding box, whereas $\bar{A}$ is the rest of the
heatmap, i.e., $H=A\cup\bar{A}$. Then, after calculating the mean $\mu$ and
variance $\sigma^{2}$ of the raw heatmap scores for each area $A$ and
$\bar{A}$, respectively, CNR is defined as

$CNR=\frac{\mu_{A}-\mu_{\bar{A}}}{\sqrt{\sigma_{A}^{2}+\sigma_{\bar{A}}^{2}}}.$
(3)

Note that this definition will penalize the case where
$\mu_{A}<\mu_{\bar{A}}$. However, following [5], we also provide results for
the absolute CNR evaluated as

$|CNR|=\frac{|\mu_{A}-\mu_{\bar{A}}|}{\sqrt{\sigma_{A}^{2}+\sigma_{\bar{A}}^{2}}}.$
(4)

###  4.7 Evaluation Protocol

To ensure a fair comparison across all approaches, we adopt the following
protocol:

  * •

All metrics are computed on the original image dimensions. Therefore, for
heatmap-based methods such as [5, 15], and our own method, we also perform
nearest neighbor interpolation with appropriate zero padding to match the
original image resolution. Note that each of those methods initially generates
a fixed size heatmap which matches the respective input size: for BioViL [5],
the resolution is $480\times 480$, for BioViL-T [15] it is equal to $448\times
448$, while our LDM-based approach outputs a $512\times 512$ heatmap per
image. For methods [12, 13] that directly predict bounding box coordinates,
those are simply resized based on the width and height of the original image.

  * •

Since heatmaps generated with BioViL [5] and BioViL-T [15] methods are on
$[-1,1]$ range (due to cosine similarity), whereas our method yields heatmaps
in the $[0,1]$ range, we set all negative values for the aforementioned
baselines [5, 15] to 0.

  * •

Both MedRPG [12] and OmniFM-DR [13] are Transformer-based models that directly
predict bounding box coordinates. Thus, we evaluate those two methods only in
terms of mIoU.

  * •

MedRPG [12] does not support the case where more than one bounding box exists
per image. Therefore, for those images, we evaluate it on each bounding box
separately and only keep the maximum value of mIoU reported for each image.

###  4.8 Results and Discussion

In this section we report results using the 4 aforementioned metrics (mIoU,
AUC-ROC, $|$CNR$|$ and CNR), and also further discuss implications. In
Subsection 4.9 we illustrate strengths in our chosen approach via a separate
ablation study.

Table 1: Phrase grounding results on MS-CXR dataset. Results are reported across the 8 pathologies of interest and also averaged (Avg). Both $|$CNR$|$ and CNR metrics are unbounded, i.e., in $(-\infty,\infty)$ range. mIoU and AUC-ROC are reported in (%). Higher is better for all metrics ($\uparrow$). Note that, due to implementation details, only mIoU can be computed for methods [12] and [13]. Best metrics are highlighted with bold. Second best metrics are underlined. Method | Metric | Pneumonia | Pneumothorax | Consolidation | Atelectasis | Edema | Cardiomegaly | Lung Opacity | Pleural Eff. | Avg
---|---|---|---|---|---|---|---|---|---|---
BioViL [5] |  $|$CNR$|$ | 1.56 | 0.78 | 1.79 | 1.37 | 0.85 | 0.81 | 1.24 | 1.38 | 1.22
CNR | 1.49 | 0.63 | 1.73 | 1.28 | 0.77 | 0.73 | 1.18 | 1.33 | 1.14
mIoU | 27.3 | 10.2 | 31.8 | 24.1 | 21.3 | 22.0 | 15.0 | 20.4 | 21.5
AUC-ROC | 76.5 | 66.3 | 83.5 | 76.4 | 65.1 | 64.5 | 68.9 | 76.6 | 72.2
MedRPG [12] | mIoU | 11.3 | 5.8 | 6.1 | 12.2 | 5.1 | 16.3 | 6.5 | 11.7 | 9.4
OmniFM-DR [13] | mIoU | 12.6 | 7.0 | 24.7 | 5.4 | 17.0 | 30.9 | 15.4 | 5.1 | 14.8
BioViL-T [15] |  $|$CNR$|$ | 1.70 | 1.01 | 1.79 | 1.47 | 0.84 | 1.06 | 1.59 | 1.55 | 1.38
CNR | 1.66 | 0.91 | 1.74 | 1.45 | 0.77 | 1.05 | 1.54 | 1.53 | 1.33
mIoU | 29.0 | 12.6 | 30.2 | 24.7 | 19.0 | 23.5 | 17.5 | 18.6 | 21.9
AUC-ROC | 80.1 | 70.1 | 83.9 | 76.4 | 63.0 | 66.1 | 76.3 | 73.8 | 73.7
Ours |  $|$CNR$|$ | 1.01 | 0.46 | 1.17 | 1.05 | 0.75 | 0.91 | 1.12 | 0.88 | 0.92
CNR | 1.01 | -0.07 | 1.14 | 1.04 | 0.70 | 0.90 | 1.08 | 0.86 | 0.83
mIoU | 23.7 | 5.3 | 24.5 | 24.8 | 30.4 | 37.1 | 16.6 | 19.2 | 22.6
AUC-ROC | 78.7 | 49.7 | 80.9 | 78.8 | 72.2 | 75.2 | 79.4 | 75.2 | 73.8

Overall phrase grounding results on the MS-CXR database are reported in Table
1. Note that error bars are not included in our analysis since we run all
models on evaluation mode, meaning that results are deterministic.

We now provide an interpretation of the results shown in Table 1. More
specifically, we draw the following conclusions:

First, our proposed method outperforms both supervised baselines MedRPG [12]
and OmniFM-DR [13] by a large margin. This is also the case for self-
supervised baselines [5, 15] that are trained on image-text pairs. This
observation suggests that free-text reports provide a rich source of
information that can be used as an alternative to ground truth bounding box
annotations.

Second, our method is competitive with both BioViL variants [5, 15] for most
pathologies on the MS-CXR dataset. In fact, our pipeline based on the LDM sets
a new state-of-the-art in terms of both mIoU (0.7 $\%$ relative improvement to
BioViL-T [15]) and AUC-ROC (0.1 $\%$ relative increase to BioViL-T [15])
metrics averaged across all classes.

Moreover, given both definitions of the CNR metric presented in Eqs. (3) and
(4), respectively, our method remains fairly robust between the two for most
pathology classes. More specifically, our approach yields the lowest
difference between $|$CNR$|$ and CNR for 7 out of 8 classes. Concretely, this
highlights that $|$CNR$|$, which is reported in [5, 15], overestimates
performance, thus making it less reliable.

We also observe that every method performs poorly on the Pneumothorax class
(our approach leads to a negative CNR value). We note that pneumothorax causes
a dark air space (i.e., a region with low intensity pixels) in the position of
the collapsed lung, as opposed to the other pathologies which manifest as
“bright” regions. These dark regions may be more difficult to differentiate
from normal lung.

###  4.9 Ablation Study

In Table 2 we provide an ablation study showing how different hyperparameters
affect phrase grounding performance. To speed up these experiments, we set the
total number of timesteps to $T=100$ (yet the empirical observations are
expected to be similar for larger $T$ as well). Starting from the initial
setup of collecting all attention maps across all timesteps ($L=11$, $T=100$),
we gradually focus only on middle cross-attention blocks ($L=6$) and middle
timesteps ($T=20$). Note that, although we got the exact same results for
$T=40$, we chose $T=20$ since it speeds up inference. We see that the choice
of middle timesteps (L=11, T=20) has a positive impact on both metrics.
Furthermore, although the choice of middle cross-attention blocks (L=6, T=100)
does not provide significant improvements on its own, we observe that the
combination thereof (L=6, T=20) yields a substantial boost in performance
compared to the initial setup (L=11, T=100). We also show the effect of post-
processing the heatmap with binary Otsu thresholding. In fact, although we
notice a slight decrease in CNR, mIoU is increased by 3.5$\%$. Note also how
different pathology classes might benefit from different setups, meaning that
there is room for improvement.

Table 2: Ablation study on the choice of layers $L$ and timesteps $T$, as well as the effect of post-processing on the resulting heatmap. Note that: the total number of timesteps is fixed here to $T_{total}=100$, while $T=20$ refers to the $[40,60]$ range of steps. CNR metric is unbounded, mIoU is in ($\%$). Best metrics are highlighted with bold. Setup | Metric | Pneumonia | Pneumothorax | Consolidation | Atelectasis | Edema | Cardiomegaly | Lung Opacity | Pleural Eff. | Avg
---|---|---|---|---|---|---|---|---|---|---
L=11, T=100 | CNR | 0.86 | -0.17 | 0.89 | 0.96 | 0.55 | 0.52 | 0.88 | 0.90 | 0.67
mIoU | 14.7 | 5.4 | 16.1 | 16.1 | 25.0 | 22.8 | 10.6 | 13.6 | 15.5
L=11, T=20 | CNR | 0.87 | -0.16 | 0.91 | 0.94 | 0.63 | 0.60 | 0.89 | 0.90 | 0.70
mIoU | 16.0 | 5.3 | 17.2 | 17.0 | 25.3 | 24.3 | 11.6 | 14.1 | 16.3
L=6, T=100 | CNR | 0.86 | -0.17 | 0.89 | 0.97 | 0.56 | 0.52 | 0.89 | 0.90 | 0.68
mIoU | 14.8 | 5.4 | 16.2 | 16.2 | 25.0 | 22.9 | 10.7 | 13.7 | 15.6
L=6, T=20 | CNR | 0.90 | -0.09 | 0.99 | 1.01 | 0.61 | 0.77 | 0.98 | 0.90 | 0.76
mIoU | 15.6 | 5.5 | 17.1 | 16.9 | 26.2 | 26.7 | 11.3 | 14.1 | 16.7
L=6, T=20, Otsu | CNR | 0.87 | -0.15 | 0.98 | 0.98 | 0.59 | 0.80 | 0.94 | 0.88 | 0.74
mIoU | 20.3 | 5.1 | 21.2 | 22.1 | 28.2 | 33.0 | 14.1 | 18.1 | 20.2

###  4.10 Qualitative Analysis

Fig. 3 depicts examples of generated heatmaps for BioViL-T [15] and our
proposed pipeline for randomly sampled image-text pairs from the MS-CXR
dataset. We observe that BioViL-T provides more densely localized results
compared to our method which activates on larger input areas. However, unlike
BioViL-T, our method does not miss an area of interest (cf. sub-figures 3(a)
and 3(e) for pathology labels Pneumonia and Edema, respectively). We note that
our LDM-based method could also be focusing on less relevant anatomical
regions in some cases (cf. the right column for sub-figures 3(b), 3(d), 3(g),
respectively) that, however, can be categorized as easy false positives.

(a) Basal consolidations could represent atelectasis versus
aspiration/pneumonia

(b) fluid level posteriorly, which represents a loculated hydropneumothorax

(c) patchy consolidation of the right lung base

(d) bibasilar atelectasis noted

(e) peribronchial cuffing consistent with mild pulmonary edema

(f) enlarged cardiac silhouette (class: Cardiomegaly)

(g) Minimal residual right middle lobe opacity, otherwise clear lungs

(h) Moderate right pleural fluid remains (class: Pleural Effusion)

Figure 3: Qualitative results for the phrase grounding task. For each image-
text pair, we overlay the generated heatmaps from BioViL-T [5] (left column of
each pair) and our own method (right column of each pair) on top of the
original images. The grey shaded area in each figure shows the center crop
that is used as input for each method. We provide the accompanying text prompt
for each image as a caption in each sub-figure. Ground truth classes are
highlighted in bold. For completeness, in cases where the ground truth class
is not explicitly mentioned in the prompt, we refer to it in parentheses.
Ground-truth bounding boxes are depicted in green. The samples are randomly
drawn such that each pathology class is represented once. For each image, we
also provide the reported $|$CNR$|$ and mIoU metrics per method (shown on top
of each plot). Best viewed in colour.

##  5 Conclusion

In this work we have presented a novel approach for performing phrase
grounding with a pre-trained Latent Diffusion Model. In fact, we draw on the
parts of the model that integrate visual and textual features, namely the
cross-attention blocks. These blocks, as evidenced by our results, provide a
rich source of information that can be readily used to solve downstream tasks.
Our proposed method does not alter the backbone generative model in any way,
thus operating on a zero-shot setting. We believe that further improvements on
the generative aspect of the LDM, i.e., forcing the model to rely less on
spurious features during reconstruction (e.g. by generating appropriate
synthetic data to counteract biases [22]), will also bring a positive effect
on the model’s downstream performance. In terms of this work’s broader impact,
our proposed framework might be used to automatically link reports to the
relevant image locations, allowing fast inclusion of key images and easy
navigation when reviewing a previous scan. We might also extend to the task of
diagnosis by creating text prompts such as "Where is {pathology_label}?", to
achieve an off-the-shelf detector that provides automated diagnosis (beyond
the scope of this paper).

## References

  * [1] Rishi Bommasani et al.  “On the opportunities and risks of foundation models”  In _arXiv preprint arXiv:2108.07258_ , 2021
  * [2] Alexander C Li et al.  “Your diffusion model is secretly a zero-shot classifier”  In _arXiv preprint arXiv:2303.16203_ , 2023
  * [3] Kevin Clark and Priyank Jaini  “Text-to-Image Diffusion Models are Zero Shot Classifiers”  In _Advances in Neural Information Processing Systems_ 36, 2024
  * [4] Dmitry Baranchuk et al.  “Label-efficient semantic segmentation with diffusion models”  In _arXiv preprint arXiv:2112.03126_ , 2021
  * [5] Benedikt Boecking et al.  “Making the most of text semantics to improve biomedical vision–language processing”  In _European conference on computer vision_ , 2022, pp. 1–21  Springer
  * [6] Wenliang Zhao et al.  “Unleashing text-to-image diffusion models for visual perception”  In _arXiv preprint arXiv:2303.02153_ , 2023
  * [7] Jiasen Lu, Dhruv Batra, Devi Parikh and Stefan Lee  “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks”  In _Advances in neural information processing systems_ 32, 2019
  * [8] Aishwarya Kamath et al.  “Mdetr-modulated detection for end-to-end multi-modal understanding”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1780–1790
  * [9] Liunian Harold Li et al.  “Grounded language-image pre-training”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10965–10975
  * [10] Sijia Chen and Baochun Li  “Language-Guided Diffusion Model for Visual Grounding”  In _arXiv preprint arXiv:2308.09599_ , 2023
  * [11] Riddhish Bhalodia et al.  “Improving pneumonia localization via cross-attention on medical images and reports”  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24_ , 2021, pp. 571–581  Springer
  * [12] Zhihao Chen et al.  “Medical phrase grounding with region-phrase context contrastive alignment”  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2023, pp. 371–381  Springer
  * [13] Lijian Xu et al.  “Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation”  In _arXiv preprint arXiv:2311.01092_ , 2023
  * [14] Gefen Dawidowicz, Elad Hirsch and Ayellet Tal  “LIMITR: Leveraging Local Information for Medical Image-Text Representation”  In _arXiv preprint arXiv:2303.11755_ , 2023
  * [15] Shruthi Bannur et al.  “Learning to exploit temporal structure for biomedical vision-language processing”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 15016–15027
  * [16] Mischa Dombrowski et al.  “Trade-offs in Fine-tuned Diffusion Models Between Accuracy and Interpretability”, 2023  arXiv:2303.17908 [cs.CV]
  * [17] Robin Rombach et al.  “High-resolution image synthesis with latent diffusion models”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10684–10695
  * [18] Walter HL Pinaya et al.  “Brain imaging generation with latent diffusion models”  In _MICCAI Workshop on Deep Generative Models_ , 2022, pp. 117–126  Springer
  * [19] Alistair EW Johnson et al.  “MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs”  In _arXiv preprint arXiv:1901.07042_ , 2019
  * [20] Walter HL Pinaya et al.  “Generative ai for medical imaging: extending the monai framework”  In _arXiv preprint arXiv:2307.15208_ , 2023
  * [21] Ping-Sung Liao, Tse-Sheng Chen and Pau-Choo Chung  “A fast algorithm for multilevel thresholding”  In _J. Inf. Sci. Eng._ 17.5, 2001, pp. 713–727
  * [22] Fernando Pérez-García et al.  “RadEdit: stress-testing biomedical vision models via diffusion image editing”  In _arXiv preprint arXiv:2312.12865_ , 2023