# Naked Singularity as a Possible Source of Ultra-High Energy Cosmic Rays

Kauntey Acharya<EMAIL_ADDRESS>International Centre for Space
and Cosmology, School of Arts and Sciences, Ahmedabad University,
Ahmedabad-380009 (Gujarat), India  Kshitij Pandey<EMAIL_ADDRESS>PDPIAS, Charusat University, Anand-388421 (Gujarat), India  International
Center for Cosmology, Charusat University, Anand-388421 (Gujarat), India
Parth Bambhaniya<EMAIL_ADDRESS>PDPIAS, Charusat University,
Anand-388421 (Gujarat), India  International Center for Cosmology, Charusat
University, Anand-388421 (Gujarat), India  Pankaj S. Joshi
<EMAIL_ADDRESS>International Centre for Space and Cosmology, School of
Arts and Sciences, Ahmedabad University, Ahmedabad-380009 (Gujarat), India
International Center for Cosmology, Charusat University, Anand-388421
(Gujarat), India  Vishva Patel<EMAIL_ADDRESS>International Center
for Cosmology, Charusat University, Anand-388421 (Gujarat), India

###### Abstract

The source of Ultra-High Energy Cosmic Rays (UHECRs) remains one of the
greatest mysteries in astrophysics. Their possible source can be the galactic
nuclei, where the ultra-high gravity region plays a crucial role. Cosmic rays
are extremely energetic particles that travel through space with energies
exceeding $10^{20}eV$, but their origin is still a mystery despite years of
studies and observations. In view of this, in this work, we studied the Joshi-
Malafarina-Narayan (JMN-1) naked singularity as a natural particle
accelerator. We derived the necessary expressions to find center of mass
energy when two particles collide. We have obtained results showing that
center of mass energy of the two particles will reach to Planck energy scale.
This will form a microscopic black hole which will decay in Hawking radiation,
having energy on the order of $10^{26}eV$ from the ultra-high gravity region
of Sgr A*. These outgoing highly energetic particles from the naked
singularity could be the possible sources of UHECRs.

Key words: Ultra-compact objects, Naked singularity, Black holes, Particle
acceleration, Cosmic rays, Hawking radiation.

##  I Introduction

Recent investigations have shown that a significant number of events have been
observed so far that exceed $10^{19}eV$ energy scale CTAConsortium:2013tmf .
However, there are no evidences of the predicted Greisen-Zatsepin-Kuzmin (GZK)
cutoff Aloisio:2009sj . The GZK cutoff is a feature of high-energy cosmic
rays, which are charged particles that originate from far-off astrophysical
sources and are propelled to exceptionally high energies. As these cosmic rays
travel through the universe, they interact with photons of the cosmic
microwave background radiation, which is a relic radiation left over from the
Big Bang Bergman:2006vt . These interactions can cause the cosmic rays to lose
energy, particularly at very high energies. The GZK cutoff is the energy
threshold above which cosmic rays are expected to be strongly attenuated due
to these interactions Haungs:2003jv .

The absence of the GZK cutoff is surprising as it should be present if the
ultra-high energy particles are from extra-galactic sources. Cosmic ray
protons with energies above a few $10^{19}eV$ lose energy through photopion
production off the cosmic microwave background (CMB) and cannot originate
beyond 50 $Mpc$ from the Earth Aloisio:2009sj . This motivates us to study the
UHECRs production phenomena within 50 $Mpc$.
The first feature, known as the pair-production dip, is a relatively weak
signal from tesulting from the pair production process and can be expressed as

$\displaystyle p+\gamma_{CMB}\longrightarrow e^{+}+e^{-}+p.$ (1)

The second feature is a pronounced steepening of the spectrum, known as the
GZK cutoff, which is caused by pion photo-production and is expressed as:

$\displaystyle p+\gamma_{CMB}\longrightarrow\pi+p.$ (2)

The existence of ultra-compact objects such as black holes, naked
singularities, gravastars and many others are yet to be proven
observationally. On the other hand, it is known about the visible and hidden
singularities that might form from the continuous gravitational collapse joshi
; goswami ; mosani1 ; mosani2 ; mosani3 ; mosani4 ; Deshingkar:1998ge ;
Jhingan:2014gpa ; Joshi:2011zm . Regarding the observational aspects, the
recent findings of the Event Horizon Telescope (EHT) suggest that JMN-1 naked
singularity is the best Schwarzschild black hole mimicker
EventHorizonTelescope:2022xqj . The study of accretion discs, shadow
properties, tidal forces, gravitational lensing, energy extractions, timelike
and null geodesics around such compact objects are eisxplored in various
literature Page:1974he ; Liu:2020vkh ; Liu:2021yev ; Joshi:2013dva ;
Bambhaniya:2021ugr ; Tahelyani:2022uxw ; Rahaman:2021kge ; Harko:2008vy ;
Harko:2009xf ; Kovacs:2010xm ; Guo:2020tgv ; Chowdhury:2011aa ;
Lattimer:1976kbf ; Chicone:2004pv ; Madan:2022spd ; Shaikh:2019jfr ;
Shaikh:2018oul ; Paul:2020ufc ; Virbhadra:2007kw ; Gyulchev:2008ff ;
Kala:2020prt ; Sahu:2012er ; Martinez ; Madan:2022spd ; Eva1 ; Eva2 ; tsirulev
; Joshi:2019rdo ; Bambhaniya:2019pbr ; Dey:2019fpv ; Bam2020 ;
Bambhaniya:2022xbz ; Gralla:2019xty ; Vagnozzi:2019apd ; Chen:2022nbb ;
Dey:2020haf ; atamurotov_2015 ; abdujabbarov_2015b ; Li:2021 ; Hu:2020usx ;
Patel:2022vlu ; Kaur:2021wgy ; Saurabh:2020zqg ; Pugliese:2022oes ;
Vagnozzi:2022moj ; Saurabh:2022jjv ; Patel:2022jbk ; Patel:2023efv .

The collision of Ultra-high energy particles near the ultra-compact object
through particle acceleration can be one of the most prominent sources of
these cosmic rays. Therefore, in this paper, we aim to explore the mechanisms
behind particle acceleration in the vicinity of ultra-compact objects. In
particular, we focus on timelike geodesics to understand the acceleration
process. Our approach does not consider the role of radiation or fields in the
acceleration process, allowing us to isolate and analyze the effect of
geodesic motions on particle acceleration.

Several papers have shown the particle acceleration phenomena around ultra-
compact objects. In Patil:2011yb , authors conclude that for Sgr A*, the
collisional energy of two particles can reach up to $10^{3}\,TeV$.
Furthermore, particle acceleration in Schwarzschild and Kerr black holes has
been shown Banados:2009pr . The study of Janis-Newman-Winicour, Resinner-
Nordstrom and Kerr naked singularities as particle accelerators are discussed
in Patil:2011aa ; Patil:2011uf ; Patil:2011ya . Due to the high gravity
region, ultra-compact objects can accelerate the particles up to the Planck
scale physics as shown in Patil:2011aw .

Initially, the phenomenon of particle collisions around black holes was known
as the Banados-Silk-West effect. However, this effect required many finely-
tuned scenarios for the collision of particles to release energy, which makes
it less appealing as a mechanism for explaining high-energy astrophysical
phenomena. Following this, authors in Patil:2011aa ; Patil:2011uf ;
Patil:2011ya  studied the particle acceleration process in the vicinity of the
naked singularity which requires less fine tuning for the possibility of high
energy collisions of particles.

Thus, in this work, we consider particle acceleration as a potential mechanism
to explain the high energy phenomena by considering the collision of particles
in the vicinity of JMN-1 naked singularity. We investigate the behaviour of
energy of colliding particles in the centre of mass frame $(E_{CM})$ with
respect to radial distance as well as for the angular momenta of two
particles. Further, we also derive the numerical value of the centre of mass
energy $(E_{CM})$ by considering JMN-1 naked singularity at the milky-way
galactic center.

This paper is organised as follows: The JMN-1 geometry is discussed in section
(II). The geodesic motion of a particle and the center of mass energy around
the JMN-1 naked singularity are derived in section (III). In the section (IV),
we try to explain the possibility of formation of microscopic black holes.
This possibility is described with a detailed discussion of Planck scale
energy and Hawking radiation for the high center of mass energy coming from
the particle collision. After that, the energy of outgoing particles is
discussed in section (IV.1). Finally, we conclude and discuss the results of
this work in section (V). Only when necessary, we defined the gravitational
constant (G) and speed of light (c) values, otherwise defined as the
geometrized units. The signature of the metric is considered as (-,+,+,+).

##  II Joshi–Malafarina–Narayan (JMN-1) spacetime

The very first model of gravitational collapse of a spherically symmetric and
homogeneous dust cloud was proposed by Oppenheimer, Snyder, and Dutt, (OSD)
which indeed indicates the formation of a black hole as an end state
Oppenheimer:1939ue . In this OSD model of gravitational collapse, trapped
surfaces form around the centre before the formation of the central space-like
singularity. As a result, the central singularity is causally disconnected
from other points of spacetime, indicating the presence of a horizon in the
spacetime structure. However, the OSD model assumes homogeneous density and
zero pressure within the massive collapsing star. Thus it is considered to be
an idealistic scenario of gravitational collapse. Despite the fact that of
Cosmic Censorship Conjecture (CCC) by Roger Penrose, several investigations
have been carried out to understand the process of gravitational collapse in
more realistic physical scenarios, including the inhomogeneous distribution of
matter in the collapsing cloud and non-zero pressure profiles Joshi:2011zm .

When an inhomogeneous matter cloud was considered i.e. higher density at the
centre which represents a more realistic physical condition, studies suggested
that an end state can be locally or globally visible strong curvature
singularity following certain equilibrium conditions Mosani:2022nsp ; mosani3
. In Joshi:2011zm , authors have extensively studied equilibrium
configurations of the collapsing cloud under the influence of gravity with
zero radial pressure and non-zero tangential pressure i.e. presence of an-
isotropic fluid. Non-zero tangential pressure can prevent the formation of
trapped surfaces around the central region. An end state of a massive star
under such conditions can lead to the formation of JMN-1 naked singularity.
The line element of the JMN-1 naked singularity spacetime is,

$ds^{2}=-(1-M_{0})\left(\frac{r}{Rb}\right)^{\frac{M_{0}}{1-M_{0}}}dt^{2}+\frac{1}{(1-M_{0})}dr^{2}+\,r^{2}d\Omega^{2},$
(3)

where, $d\Omega^{2}=d\theta^{2}+\,sin^{2}\,\theta\,d\phi^{2}$. $M_{0}$ and
$R_{b}$ are positive constants. $M_{0}$ can have any value within the range
$0<M_{0}\,<\,1$. $R_{b}$ is the radius of matter distributed around the
central singularity.

The spacetime metric is modelled by considering a high-density compact region
in a vacuum, which means that the spacetime configuration should be
asymptotically flat. For this purpose, the JMN-1 spacetime is matched with an
exterior Schwarzschild spacetime at $r=R_{b}$ radius. The line element of this
exterior spacetime can be written as,

$ds^{2}=-\left(1-\frac{M_{0}R_{b}}{r}\right)dt^{2}+\frac{dr^{2}}{\left(1-\frac{M_{0}R_{b}}{r}\right)}+r^{2}d\Omega^{2},$
(4)

where, Schwarzschild radius is $R_{s}=M_{0}R_{b}$ and the Schwarzschild mass
is given by, $M=\frac{M_{0}R_{b}}{2}.$ Now for matching of two spacetimes,
there are two junction conditions which suggest that (i) extrinsic curvatures
of both spacetimes should smoothly match at a null hyper-surface and (ii)
induced metrics of exterior and interior geometries be equivalent on the
hyper-surface where matching is considered. Since the radial pressure is zero
in JMN-1 naked singularity spacetime, the extrinsic curvatures of the interior
JMN-1 spacetime and the exterior Schwarzschild spacetime are smoothly match at
$r=R_{b}$. It is also important to note that all the energy conditions are
satisfied in the above spacetime structure. In the next section (III), we
consider collisions of particles in the vicinity of the above discussed
compact object with a central high-density region of JMN-1 spacetime and outer
Schwarzschild metric.

##  III Particle acceleration near a JMN-1 naked singularity

In this section, we describe the mechanism of particle acceleration by
investigating collisions of particles near the central singularity of JMN-1
spacetime. We first consider the geodesic motion of colliding particles with
turning points and derive the expression of center of mass energy. For this,
we also study the allowed values of metric parameter $M_{0}$ for which energy
collisions are possible because of the existence of turning points.

###  III.1 Geodesic motions

We first describe the motion of the massive particles of mass $m$ moving with
the four velocities $U^{\mu}$ in time-like geodesics in the JMN-1 spacetime.
For simplicity, we consider that particles move in an equatorial plane
($\theta\,=\,\pi/2$), which suggests that $U^{\theta}=0$. The equations of
motion of particles can be written by using constants of motion, i.e.
conserved energy and conserved angular momentum per unit mass of the particle.
From these constants of motion, the expressions of four velocity are written
as,

$\displaystyle
u^{t}=\frac{e}{(1-M_{0})\left(\frac{r}{Rb}\right)^{\frac{M_{0}}{1-M_{0}}}},$
(5) $\displaystyle u^{\phi}=\frac{L}{r^{2}}.$ (6)

The radial component of the four velocity can be written as follows using the
normalization condition $u^{\mu}u_{\mu}=-1$,

$u^{r}=\pm\bigg{[}e^{2}-\frac{1}{2}\bigg{(}(1\;-\;M_{o})\bigg{(}\frac{r}{R_{b}}\bigg{)}^{\frac{\;M_{o}}{1\;-\;M_{o}}}\bigg{(}1+\frac{L^{2}}{r^{2}}\bigg{)}-1\bigg{)}\bigg{]}^{\frac{1}{2}},$
(7)

Here $\pm$ represents the motion of particle in radial outward and inward
directions, respectively. from which one can write the effective potential and
total energy as,

$(u^{r})^{2}+V_{eff}(r)=E,$ (8)

where $E=\frac{e^{2}-1}{2}$ is the total energy and $V_{eff}(r)$ is the
effective potential as a function of the radial coordinate for a massive
particle which can be expressed as,

$V_{eff}(r)=\frac{1}{2}\bigg{[}(1\;-\;M_{o})\bigg{(}\frac{r}{R_{b}}\bigg{)}^{\frac{\;M_{o}}{1\;-\;M_{o}}}\bigg{(}1+\frac{L^{2}}{r^{2}}\bigg{)}-1\bigg{]}.$
(9)

Now we consider a scenario where non-relativistic particles at infinity fall
inward under the influence of gravity near the JMN-1 naked singularity.
However, the exterior region of the JMN-1 naked singularity is represented by
the Schwarzschild metric. Thus it is important to consider equations of motion
for the exterior Schwarzschild metric as it is considered that colliding
particles are coming from infinity. The expressions of four velocity of a test
particle using constants of motion can be written as,

$\displaystyle u^{t}=\frac{e}{\left(1-\frac{M_{0}R_{b}}{r}\right)},$ (10)
$\displaystyle u^{\phi}=\frac{L}{r^{2}}.$ (11)

Using the normalization condition $u^{\mu}u_{\mu}=-1$ for timelike geodesics,
the radial component of the four velocity in exterior Schwarzschild geometry
can be written as,

$(u^{r})^{2}=\left[e^{2}-\frac{1}{2}\bigg{[}\left(1-\frac{M_{0}R_{b}}{r}\right)\bigg{(}1+\frac{L^{2}}{r^{2}}\bigg{)}-1\bigg{]}\right],$
(12)

and the expression of an effective potential for Schwarzschild spacetime is,

$V_{eff}(r)=\frac{1}{2}\bigg{[}\left(1-\frac{M_{0}R_{b}}{r}\right)\bigg{(}1+\frac{L^{2}}{r^{2}}\bigg{)}-1\bigg{]}.$
(13)

Now for particles at infinity, as $r\rightarrow\infty$, $U^{r}\rightarrow 0$,
the conserved energy per unit rest mass of the particles should be $e=1$.
Since total energy of the particle is $E=\frac{e^{2}-1}{2}$, $E=0$ and hence
the radial component of the four velocity can be written as,

$(U^{r})^{2}=\left[1-\left(1-\frac{M_{0}R_{b}}{r}\right)\left(1+\frac{L^{2}}{r^{2}}\right)\right],$
(14)

In general for high energy collisions, the process must happen near the
central singularity so high gravitational effects can be considered. For this,
one of the particles must turn back before it reaches near the singularity.
For a particle to turn back at any given radial distance $r$, $U^{r}=0$ and
$E=1$. From which one can write the expression of required angular momentum
$L$ for the particle to turn back at $r$ in the exterior Schwarzschild
geometry,

$L=\pm\left(\frac{rM_{0}R_{b}}{r-M_{0}R_{b}}\right)^{\frac{1}{2}}.$ (15)

However to consider the maximum energy extraction from the collision of
particles, one needs to consider this phenomenon of collisions near
singularity. Since the central high-density region is represented by JMN-1
spacetime metric, we consider the collision of particles within this
spacetime. Here again, it is considered that particles must turn back before
it reaches the singularity as shown in Fig. (2). In this geometry, for a
particle to turn back at any given radial distance, $U^{r}=0$ and $E=1$. The
expression of required angular momentum $L$ for the particle to turn back at
any radial distance $r$ in the JMN-1 spacetime can be written as,

(a) $M_{0}$ = 0.55, $R_{b}$ = 3.64, $L_{1}$= 4.05, $L_{2}$= -4.05.

(b) The bar on right side of the figure represents the different values of
parameter $M_{0}$.

Figure 1: The change in $E_{cm}$ for the radial distance is shown in Fig.
1(a). Fig. 1(b) shows the change in angular momentum ($L(r)$) of a particle
with respect to the radial distance ($r$).

$L=\pm\left[\frac{R_{0}^{\frac{M_{0}}{1-M_{0}}}}{\left(1-M_{0}\right)r^{\frac{M_{0}}{1-M_{0}}}}-1\right]^{-\frac{1}{2}}.$
(16)

The plot of the above expression is shown in Fig. 1(b). It represents the
variation in angular momentum for particle to turn back at any radial distance
with respect to radius in JMN-1 geometry for different values of $M_{0}$
parameter between $1/2\,<\,M_{0}\,<\,3/4$. Other values of $M_{0}$ are not
considered because there does not exist any turning points for the
corresponding values. It should be noted that for $M_{0}\,<\,1/2$, the radius
at which the collision would take place is greater than the boundary radius
$R_{b}$. Since the minima in the graph of $L\rightarrow r$ is outside the
boundary radius for JMN-1 metric which is not possible. While for
$M_{0}\,>\,2/3$, the radius at which we get the minimum angular momentum which
is imaginary. This can also be seen from the corresponding lines representing
the angular momentum of a particle with respect to distance $r$ that there are
no minima in the angular momentum plot for $M_{0}=0.7,0.75$, which suggest
that the particle does not turn back at any value of radial distance and hence
collision is not possible. Thus the particle collisions need to be considered
only between the range $1/2\,\leq\,M_{0}\,\leq\,2/3$. Corresponding to these
values of $M_{0}$, the radial distance at which the collisions would occur can
be found using eq.(16), which is $r=4.00M$ for $M_{0}=0.50$, $r=3.23M$ for
$M_{0}=0.55$, $r=2.43M$ for $M_{0}=0.60$, and $r=1.31M$ for $M_{0}=0.65$.
Here, $M$ is the mass of the central compact object. For the case of
$M_{0}<\frac{1}{2}$, the radius at which turning occurs outside the boundary
radius $R_{b}$ and for $M_{0}>\frac{2}{3}$ the particle will plunge inside the
singularity.

Figure 2: The figure shows that particle 1 is turned at a turning point and
collides head-on with particle 2 at a radial distance of $r=3.2M$ from the
singularity, corresponding to $M_{0}=0.55$ case. Here $r_{max}$ and $r_{min}$
are the maximum and minimum possible radius for the turning point.

Now the range of possible values of angular momentum for the collision to
occur is known. Using this information, we can find out energy particles after
the collision. However, since the collision is taking place in curved
spacetime, there is no fixed coordinate system or reference frame for which
the energy of the particles can be defined. Thus we consider a centre of mass
frame of two particles as a reference frame for this purpose.

###  III.2 Center of mass energy in JMN-1 naked singularity

To define the center of mass energy of colliding particles, we first consider
a general curved spacetime defined by $g_{\mu\nu}$. In this spacetime, we can
consider any arbitrary point as a lab frame where the spacetime can be
considered to be locally flat and has a form of Minkowski metric. The local
lab frame can be described by basis vectors $\hat{e_{a}}$ for which
$\hat{e_{a}}\hat{e_{b}}=\eta_{ab}$. Here for any given basis, the coordinate
basis can be written as $\vec{e_{\mu}}=e^{a}_{\mu}\hat{e_{a}}$. Now if the
curved spacetime metric $g_{\mu\nu}$ is given, then the matrix $e^{a}_{\mu}$
can be uniquely defined and thus, from the information about the world-line
history $x^{\mu}(\tau)$ and four velocity $u^{\mu}$, three velocity of any
particle in that local lab frame can be written as,

(a) $M_{0}$ = 0.50, $R_{b}$ = 4.00.

(b) $M_{0}$ = 0.55, $R_{b}$ = 3.64.

(c) $M_{0}$ = 0.60, $R_{b}$ = 3.33.

(d) $M_{0}$ = 0.65, $R_{b}$ = 3.08.

Figure 3: The above figures shows the change in center of mass energy
($E_{cm}$) per unit rest mass with respect to the different angular momentum
of two colliding particle. The bar on right side of the figures shows the
numerical value of $E_{cm}$.

$v^{(i)}=\frac{e^{(i)}_{\mu}u^{\mu}}{e^{(0)}_{\mu}u^{\mu}},$ (17)

The energy of center of mass in this frame can be expressed as,

$E_{cm}^{2}=2m^{2}\left(1-\eta_{\mu\nu}u^{\mu}_{(1)}u^{\nu}_{(2)}\right)^{\frac{1}{2}}.$
(18)

The above equation can be written for a generally curved spacetime metric from
$g_{\mu\nu}=e^{a}_{\mu}e^{b}_{\nu}\eta_{ab}$ as,

$E_{cm}^{2}=2m^{2}\left(1-g_{\mu\nu}u^{\mu}_{(1)}u^{\nu}_{(2)}\right)^{\frac{1}{2}}.$
(19)

Now we can use the above equation for exterior Schwarzschild metric and get
the following expression of center of mass energy,

$\displaystyle\left(\frac{E_{cm}^{2}}{2m^{2}}\right)_{SCH}$
$\displaystyle=1+\frac{1}{\left(1-\frac{M_{0}R_{b}}{r}\right)}-\frac{1}{\left(1-\frac{M_{0}R_{b}}{r}\right)}\times$
$\displaystyle\left(1-\left(1-\frac{M_{0}R_{b}}{r}\right)\left(1+\frac{L_{1}^{2}}{r^{2}}\right)\right)^{\frac{1}{2}}\left(1-\left(1-\frac{M_{0}R_{b}}{r}\right)\left(1+\frac{L_{2}^{2}}{r^{2}}\right)\right)^{\frac{1}{2}}-\frac{L_{1}L_{2}}{r^{2}}.$
(20)

However, as we consider that the collision could also occur at $r<R_{b}$, we
need to find the expression of $E_{cm}$ for JMN-1 naked singularity spacetime
which can be written as,

$\displaystyle\left(\frac{E_{cm}^{2}}{2m^{2}}\right)_{JMN-1}$
$\displaystyle=1+\frac{1}{(1-M_{0})\left(\frac{r}{Rb}\right)^{\frac{M_{0}}{1-M_{0}}}}-\frac{1}{1-M_{0}}\times$
$\displaystyle\left(1-(1-M_{0})\left(\frac{r}{Rb}\right)^{\frac{M_{0}}{1-M_{0}}}\left(1+\frac{L_{1}^{2}}{r^{2}}\right)\right)^{\frac{1}{2}}\left(1-(1-M_{0})\left(\frac{r}{Rb}\right)^{\frac{M_{0}}{1-M_{0}}}\left(1+\frac{L_{2}^{2}}{r^{2}}\right)\right)^{\frac{1}{2}}-\frac{L_{1}L_{2}}{r^{2}}$
(21)

From this expression, the energy of both particles after collision in the
JMN-1 geometry from the center of mass frame can be evaluated.

At first, to understand the behavior of energy of particle per unit rest mass
from center of mass frame of reference $E_{cm}$ with radial distance
coordinate, we consider particle collision for $M_{0}=0.55$ (which is in the
range of values of $M_{0}$ for which particle collision can be considered as
the turning point exists at $r<R_{b}$) and get the Fig. 1(a). The graph shows
that as we move away from the singularity, the energy of particle per unit
rest mass from the center of mass frame of reference decreases.

However, as we have considered earlier, collision does just take place where
the angular momentum of the particles possess values in a certain range, thus
it is important to see the behavior of $E_{cm}$ with angular momenta $L_{1}$
and $L_{2}$ of two particles. As stated previously, we can consider the
collision of particles only when $1/2\,\leq\,M_{0}\,\leq\,2/3$ for the
corresponding limiting values of angular momentum. Considering those values,
we get plots of energy of two particles per unit rest mass from the center of
mass frame with angular momenta of two particles $L_{1}$ and $L_{2}$ which can
be seen in Fig. (3). From this Fig. (3), one can see that the energy from
center of mass frame is maximum when both particles possess opposite values of
angular momenta. This is because the head-on collision is taking place between
the particles as they are approaching each other. The scenario refers to a
situation where a particle is moving towards a singularity and another
particle is moving away from the same singularity. The two particles collide
at the turning point.

Also, for the similar values of angular momenta, which suggest the collision
of two particles moving in the same direction, $E_{cm}$ is very small.
Furthermore, $E_{cm}$ is zero when the magnitude of $L_{1}$ and $L_{2}$ is
maximum from the theoretically obtained possible range of angular momenta.
Note that as the value of $M_{0}$ parameter increases, the maximal $E_{cm}$
also increases between $\frac{1}{2}\leq M_{0}\leq\frac{2}{3}$. We evaluate the
$E_{cm}$ using Eq.21 and Eq.16 by considering mass of a neutron as mass
$m=1.67\times 10^{-27}$ kg of colliding particles and the mass of Sgr A
$M=4.1\times 10^{6}M_{\odot}$ as mass of JMN-1 naked singularity. For the
value of parameter $M_{0}=0.5$, the corresponding value of centre of mass
energy $E_{CM}$ of colliding particles is in the order of $10^{28}eV$, which
is Planck energy scale.

##  IV Formation of Microscopic Black Holes

According to the theory of general relativity, all forms of energy, including
momentum, generate gravity. As a result, it’s believed that gravity will
become a significant factor at very high energies, specifically when the
center of mass energies approach the Planck scale. At Planck energy, the
collision of two particles can even create a microscopic black hole
Choptuik:2009ww .

The four dimensional Planck scale energy is $10^{19}GeV$ or $10^{28}eV$.
According to hoop conjecture given by Kip Thorne in 1972, if a large amount of
matter and energy, represented by $E$, are compressed into a region, and if a
hoop with proper circumference $2\pi R$ can fully encircle this matter in all
directions. Then a black hole will form if the resulting Schwarzschild radius
(calculated as $R_{s}=2GE/c^{4}$) is larger than the value of $R$. The
parameters used in this calculation include Newton’s constant ($G$) and the
speed of light ($c$) Choptuik:2009ww . Since the energy of the collision
reaches a Planck scale, it can form a microscopic black hole. However, these
microscopic black holes are not stable as they decay very rapidly through
Hawking radiation CMS:2010oej , Alok:2022xiy .

The particles that makeup Hawking radiation are generated by the intense
gravitational forces that exist around a black hole. These forces are strong
enough to separate virtual particles, which are pairs of particles constantly
popping in and out of existence in empty space. When one of these virtual
particles is separated from its partner and falls into the black hole, the
other particle is free to escape and create Hawking radiation. Thus through
Hawking Radiation, these micro black holes will decay instantaneously
Kovacik:2021qms .

It is generally believed that miniature black holes decay by releasing
elementary particles in the form of a spectrum of energy consistent with that
of a black body Anchordoqui:2002cp . The energy of these particles can be
calculated through Hawking temperature. The energy $E$ of Hawking radiation is
given by

$\displaystyle E=\frac{\hbar c^{3}}{16\pi GM},$ (22)

where $\hbar$ is the reduced Planck constant, $c$ is the speed of light, and
$M$ is the mass of the black hole.

###  IV.1 Energy of the outgoing particles

The particles that will be coming from the decay of microscopic black holes
are of great interest. The energy of primary particles coming out of this
microscopic black hole can be obtained using Hawking Temperature, as mentioned
above. Here the mass of the black hole can be obtained using
$M=\frac{E_{CM}}{c^{2}}$. Since the $E_{CM}$ is equal to the Planck energy,
the mass of the microscopic black hole that we obtain will be Planck mass
which is in the order of $10^{19}GeV$. Considering all the above-given
parameters, the energy of these primary particles in the local frame will be
in the order of $10^{26}eV$. However, the end results of Hawking emission are
not the elementary particles only released by BHs. Others only exist in
composite states, while some of them are unstable and decay (hadrons)
Arbey:2019mbc . After all the decay, only the stable particle with a very
large half-life will be visible to an asymptotic observer. The energy that we
are getting near JMN-1 is significantly higher than the UHECRs that we detect
on Earth. However, here we have not considered any other interaction such as
coulomb interaction, Brehmstrahlung radiation (a type of electromagnetic
radiation that is emitted when charged particles, such as electrons, are
accelerated or decelerated by a strong electric field), and many other types
of interactions. Furthermore, for an asymptotic observer like on Earth, the
gravitational redshift will also play a significant role in downscaling the
energy. In a physical scenario, these particles’ interactions and redshift
combine will down-scale the energy scale of particles, which leads us to the
possibility of forming UHECRs. The detailed investigation of the down-scaling
of the energy due to various interactions is left for future work, which will
consider the hadronization of Hawking radiation.

##  V Discussion and Conclusions

In this paper, we have considered the phenomena of collision of particles in
the vicinity of the JMN-1 naked singularity. We have also derived a numerical
value of energy of particles after collision corresponding to $M_{0}=0.5$. Our
investigation leads to the following conclusion:

  * •

We considered collision of particles with turning points in this study. For
this, we define a range of metric parameter $M_{0}$ for which particles turn
back in the vicinity of the singularity. We find turning of particle is
possible within the range $\frac{1}{2}\leq M_{0}\leq\frac{2}{3}$. For the case
of $M_{0}>\frac{2}{3}$, the radius at which particle turns back is imaginary,
which suggests that particle plunges inside the singularity. For
$M_{0}\,<\frac{1}{2}$, the radius of turning point is outside the boundary
radius $R_{b}$.

  * •

We find values of radial distance coordinate at which particle turns for
different values of $M_{0}$. This radial distance is $r=4.00M$ for
$M_{0}=0.50$, $r=3.23M$ for $M_{0}=0.55$, $r=2.43M$ for $M_{0}=0.60$, and
$r=1.31M$ for $M_{0}=0.65$. Here, $r_{min}$ would be the smallest value of
radial distance coordinate at which particle can turn back. For $r<r_{min}$,
there does not exist turning points as particles plunge inside the
singularity. We considered collision of particles for these values of radial
distance. It is important to note here that, in case of Schwarzschild black
hole, the collisions can be defined for values of radial distance $r>2M$ as
there is event horizon at $r_{s}=2M$. However in JMN-1 naked singularity case,
the collisions can be defined at distances closer to singularity compared to
black hole case.

  * •

From the range of JMN-1 metric parameter $\frac{1}{2}\leq
M_{0}\leq\frac{2}{3}$ and values of radial distance coordinates, we define
corresponding values of angular momentum $L$ of two particles approaching the
central high density region moving along the geodesic. Finally we studied
collision of two particles between the range $\frac{1}{2}\leq
M_{0}\leq\frac{2}{3}$ for corresponding values of angular momentum $L$ and
radial distance coordinate $r$. In Fig. 1(a) the behavior of centre of mass
energy $E_{CM}$ of particles after collision with respect to the radial
distance is shown. It can be seen that centre of mass energy decreases as the
radial distance increases which is because the gravitational influence of
singularity is weak at large distances.

  * •

Another point which should be noted is, in case of black holes, the presence
of event horizon leads to requirement of comparatively more fine tuning
conditions for the phenomena of particle collision in the vicinity of the high
curvature region. However in this case, it is easier to theoretically show the
possibility of collision of particles in the vicinity of the singularity
because of absence of horizon.

  * •

We derive the numerical value of centre of mass energy by considering the mass
of colliding particles as a neutron. The mass of JMN-1 naked singularity is
considered to equivalent to Sgr A. Here, for $M_{0}=0.5$, the boundary radius
is $R_{b}=2.419\times\,10^{10}m$. For this, we get the centre of mass energy
$E_{CM}$ of particles after collision in the order of $10^{28}eV$, which is in
Planck scale. According to Hoop conjecture, collision at these energy scales
will form a microscopic black hole, whose mass is given by $m=E_{CM}/c^{2}$.
The microscopic black hole will decay instantaneously through Hawking
radiation.

  * •

The energy of the particles coming out of the Hawking radiation near JMN-1
naked singularity will be given by $E=\frac{\hbar c^{3}}{16\pi GM}$. For the
given parameters, the energy of the initial particles will be $10^{26}eV$.
However, most of the particles coming out of the Hawking radiation will decay
either due to a lesser half-life or interaction with other particles.
Moreover, for an asymptotic observer like on earth, the effect of
gravitational redshift will also play a role. The combination of interactions
among particles and their redshift can cause a reduction in the energy scale
of particles, which in turn increases the likelihood of producing UHECRs.

Massive compact objects can act as natural particle accelerators; not only can
they accelerate the particles at an energy that is next to impossible on
earth, but also these accelerators can answer some of the most underlying
questions about the ultra-high energy phenomena. For future work, we will deal
with the phenomenology of these particles and how their energy will change
with various interactions and processes.

##  VI ACKNOWLEDGMENTS

The authors would like to express their gratitude towards Prof. P C
Vinodkumar, Dr. Tapobroto Bhanja and Mr. Saurabh for their valuable
suggestions and comments. V. P. would like to acknowledge the support of the
SHODH fellowship (ScHeme Of Developing High quality research - MYSY).

## References

  * (1) H. Sol et al. [CTA Consortium], Astropart. Phys. 43 (2013), 215-240.
  * (2) R. Aloisio, V. Berezinsky and A. Gazizov, Astropart. Phys. 34 (2011), 620-626.
  * (3) D. R. Bergman [HiRes], Nucl. Phys. B Proc. Suppl. 165, 19-26 (2007).
  * (4) A. Haungs, H. Rebel and M. Roth, Rept. Prog. Phys. 66, 1145-1206 (2003).
  * (5) P. S. Joshi and I. H. Dwivedi, Phys. Rev. D 47, 5357 (1993).
  * (6) R. Goswami and P. S. Joshi, Classical Quantum Gravity 21, 3645 (2004).
  * (7) K. Mosani, D. Dey, and P. S. Joshi, Phys. Rev. D 101, 044052 (2020).
  * (8) K. Mosani, D. Dey and P. S. Joshi, Phys. Rev. D 102, no.4, 044037 (2020).
  * (9) K. Mosani, D. Dey and P. S. Joshi, Monthly Notices of the Royal Astronomical Society,  504, 4, 4743–4750 (2021).
  * (10) D. Dey, P. S. Joshi, K. Mosani and V. Vertogradov, Eur. Phys. J. C 82 (2022) no.5, 431.
  * (11) S. S. Deshingkar, S. Jhingan and P. S. Joshi, Gen. Rel. Grav. 30 (1998), 1477-1499.
  * (12) S. Jhingan and S. Kaushik, Phys. Rev. D 90 (2014) no.2, 024009.
  * (13) P. S. Joshi, D. Malafarina and R. Narayan, Class. Quant. Grav. 28 (2011), 235018.
  * (14) K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 930 (2022) no.2, L17.
  * (15) D. N. Page and K. S. Thorne, Astrophys. J. 191 (1974), 499-506.
  * (16) C. Liu, T. Zhu and Q. Wu, Chin. Phys. C 45 (2021) no.1, 015105.
  * (17) C. Liu, S. Yang, Q. Wu and T. Zhu, JCAP 02 (2022) no.02, 034.
  * (18) P. S. Joshi, D. Malafarina and R. Narayan, Class. Quant. Grav. 31 (2014), 015002.
  * (19) P. Bambhaniya, S. K, K. Jusufi and P. S. Joshi, Phys. Rev. D 105 (2022) no.2, 023021.
  * (20) D. Tahelyani, A. B. Joshi, D. Dey and P. S. Joshi, Phys. Rev. D 106 (2022) no.4, 044036.
  * (21) F. Rahaman, T. Manna, R. Shaikh, S. Aktar, M. Mondal and B. Samanta, Nucl. Phys. B 972 (2021), 115548.
  * (22) T. Harko, Z. Kovacs and F. S. N. Lobo, Phys. Rev. D 78 (2008), 084005.
  * (23) T. Harko, Z. Kovacs and F. S. N. Lobo, Phys. Rev. D 79 (2009), 064001.
  * (24) Z. Kovacs and T. Harko, Phys. Rev. D 82 (2010), 124047.
  * (25) J. Q. Guo, P. S. Joshi, R. Narayan and L. Zhang, Class. Quant. Grav. 38 (2021) no.3, 035012.
  * (26) A. N. Chowdhury, M. Patil, D. Malafarina and P. S. Joshi, Phys. Rev. D 85 (2012), 104031.
  * (27) J. M. Lattimer and D. N. Schramm, Astrophys. J. 210 (1976), 549.
  * (28) C. Chicone and B. Mashhoon, Annalen Phys. 14 (2005), 290-308.
  * (29) S. Madan and P. Bambhaniya, [arXiv:2201.13163 [gr-qc]].
  * (30) R. Shaikh, P. Banerjee, S. Paul and T. Sarkar, JCAP 07 (2019), 028.
  * (31) R. Shaikh, P. Banerjee, S. Paul and T. Sarkar, Phys. Lett. B 789 (2019), 270-275 [erratum: Phys. Lett. B 791 (2019), 422-423]
  * (32) S. Paul, Phys. Rev. D 102 (2020) no.6, 064045.
  * (33) K. S. Virbhadra and C. R. Keeton, Phys. Rev. D 77 (2008), 124014.
  * (34) G. N. Gyulchev and S. S. Yazadjiev, Phys. Rev. D 78 (2008), 083004.
  * (35) S. Kala, Saurabh, H. Nandan and P. Sharma, Int. J. Mod. Phys. A 35 (2020), no.28, 2050177.
  * (36) S. Sahu, M. Patil, D. Narasimha and P. S. Joshi, Phys. Rev. D 86 (2012), 063010.
  * (37) C. Martínez, N. Parra, N. Valdés and J. Zanelli, Phys. Rev. D 100, no.2, 024026 (2019).
  * (38) E. Hackmann, V. Kagramanova2, J. Kunz2 and C. Lämmerzahl1 Published 13 November 2009 • Europhysics
  * (39) Eva Hackmann, Claus Lämmerzahl, Yuri N. Obukhov, Dirk Puetzfeld, and Isabell Schaffer Phys. Rev. D 90, 064035 – Published 22 September 2014
  * (40) I. M. PotashovJu. V. TchemarinaA. N. Tsirulev The European Physical Journal C, August 2019, 79:709
  * (41) A. B. Joshi, P. Bambhaniya, D. Dey and P. S. Joshi, arXiv:1909.08873 [gr-qc].
  * (42) P. Bambhaniya, A. B. Joshi, D. Dey and P. S. Joshi, Phys. Rev. D 100, no.12, 124020 (2019).
  * (43) D. Dey, P. S. Joshi, A. Joshi and P. Bambhaniya, Int. J. Mod. Phys. D 28, no. 14, 1930024 (2019).
  * (44) P. Bambhaniya, D. N. Solanki, D. Dey, A. B. Joshi, P. S. Joshi and V. Patel, Eur. Phys. J. C 81, no.3, 205 (2021).
  * (45) P. Bambhaniya, A. B. Joshi, D. Dey, P. S. Joshi, A. Mazumdar, T. Harada and K. i. Nakao, [arXiv:2209.12610 [gr-qc]].
  * (46) S. E. Gralla, D. E. Holz and R. M. Wald, Phys. Rev. D 100, no. 2, 024018 (2019).
  * (47) S. Vagnozzi and L. Visinelli, Phys. Rev. D 100, no. 2, 024020 (2019).
  * (48) Y. Chen, R. Roy, S. Vagnozzi and L. Visinelli, arXiv:2205.06238 [astro-ph.HE] (2022).
  * (49) D. Dey, R. Shaikh and P. S. Joshi, Phys. Rev. D 102, no.4, 044042 (2020).
  * (50) F. Atamurotov, B. Ahmedov, and A. Abdujabbarov, Phys. Rev. D 92, 084005 (2015).
  * (51) A. A. Abdujabbarov, L. Rezzolla, and B. J. Ahmedov, Mon. Not. R. Astron. Soc. 454, 2423 (2015).
  * (52) G. P. Li and K. J. He, JCAP 06, 037 (2021).
  * (53) Z. Hu, Z. Zhong, P. C. Li, M. Guo and B. Chen, Phys. Rev. D 103, no.4, 044057 (2021).
  * (54) V. Patel, D. Tahelyani, A. B. Joshi, D. Dey and P. S. Joshi, Eur. Phys. J. C 82 (2022) no.9, 798.
  * (55) K. P. Kaur, P. S. Joshi, D. Dey, A. B. Joshi and R. P. Desai, arXiv:2106.13175 [gr-qc], (2021).
  * (56) K. Saurabh and K. Jusufi, Eur. Phys. J. C 81, no.6, 490 (2021).
  * (57) D. Pugliese and Z. Stuchlík, Phys. Rev. D 106, no.12, 124034 (2022).
  * (58) S. Vagnozzi, R. Roy, Y. D. Tsai, L. Visinelli, M. Afrin, A. Allahyari, P. Bambhaniya, D. Dey, S. G. Ghosh and P. S. Joshi, et al. [arXiv:2205.07787 [gr-qc]].
  * (59) Saurabh, P. Bambhaniya and P. S. Joshi, [arXiv:2202.00588 [gr-qc]].
  * (60) V. Patel, K. Acharya, P. Bambhaniya and P. S. Joshi, Universe 8 (2022) no.11, 571.
  * (61) V. Patel, K. Acharya, P. Bambhaniya and P. S. Joshi, [arXiv:2301.11052 [gr-qc]].
  * (62) M. Patil and P. S. Joshi, Phys. Rev. D 84 (2011), 104001.
  * (63) M. Banados, J. Silk and S. M. West, Phys. Rev. Lett. 103 (2009), 111102.
  * (64) M. Patil and P. S. Joshi, Phys. Rev. D 85 (2012), 104014
  * (65) M. Patil, P. S. Joshi, M. Kimura and K. i. Nakao, Phys. Rev. D 86 (2012), 084023.
  * (66) M. Patil and P. S. Joshi, Class. Quant. Grav. 28 (2011), 235012.
  * (67) M. Patil, P. S. Joshi and D. Malafarina, Phys. Rev. D 83 (2011), 064007.
  * (68) J. R. Oppenheimer and H. Snyder, Phys. Rev. 56, 455-459 (1939).
  * (69) K. Mosani, [arXiv:2211.06604 [gr-qc]].
  * (70) M. W. Choptuik and F. Pretorius, Phys. Rev. Lett. 104 (2010), 111101.
  * (71) M. W. Choptuik and F. Pretorius, Phys. Rev. Lett. 104 (2010), 111101
  * (72) V. Khachatryan et al. [CMS], Phys. Lett. B 697 (2011), 434-453.
  * (73) A. K. Alok, T. Sarkar and S. Yadav, Eur. Phys. J. C 82 (2022) no.8, 711.
  * (74) S. Kováčik, Phys. Dark Univ. 34 (2021), 100906.
  * (75) L. Anchordoqui and H. Goldberg, Phys. Rev. D 67 (2003), 064010.
  * (76) A. Arbey and J. Auffinger, Eur. Phys. J. C 79 (2019) no.8, 693.


# Benchmarking and Boosting Radiology Report Generation for 3D High-Resolution
Medical Images

Che Liu1∗, Zhongwei Wan2, Yuqi Wang3, Hui Shen2, Haozhe Wang4,
Kangyu Zheng2, Mi Zhang2, Rossella Arcucci1
1 Imperial College London 2 The Ohio State University
3 University of Hong Kong
4 Hong Kong University of Science and Technology
<EMAIL_ADDRESS>$\\*$ Equal Contribution.

###### Abstract

Automatic radiology report generation can significantly benefit the labor-
intensive process of report writing by radiologists, especially for 3D
radiographs like CT scans, which are crucial for broad clinical diagnostics
yet underexplored compared to 2D radiographs. Existing methods often handle 3D
volumes either slice-wise or with aggressive downsampling due to current GPU
memory limitations, which results in a loss of the inherent 3D nature and
critical details. To overcome these issues, we introduce a novel framework
that efficiently and effectively generates radiology reports for high-
resolution (HR) 3D volumes, based on large language models (LLMs).
Specifically, our framework utilizes low-resolution (LR) visual tokens as
queries to mine information from HR tokens, preserving detailed HR information
while reducing computational costs by only processing HR informed LR visual
queries. Further benefiting the field, we curate and release BIMCV-RG, a new
dataset with 5,328 HR 3D volumes and paired reports, establishing the first
benchmarks for report generation from 3D HR medical images. Our method
consistently surpasses existing methods on this benchmark across three
different settings: normal-resolution, high-resolution inputs, and zero-shot
domain transfer, all at an acceptable computational cost, trainable on a
single A100-80G GPU.

##  1 Introduction

Figure 1:  Relationship between the number of visual tokens processed by the
LLMs and report generation performance with the BIMCV-RG dataset. Left: For
RadFM and M3D wu2023towards ; bai2024m3d , the number of visual tokens
processed by the LLM increases with the input 3D volume resolution. In
contrast, our method maintains a constant token count, ensuring computational
efficiency. Right: Our method consistently improves report generation
performance with increasing 3D volume resolution, while competing methods
struggle at $256\times 256\times 256$ resolutions due to out-of-memory (OOM)
from high token counts when $\textrm{batchsize}=1$. Refer to Sect 5.2 for
details.

Radiology reports are essential for clinical decision making, yet their manual
creation is labor-intensive and time-consuming bastawrous2017improving ;
rimmer2017radiologist ; rosenkrantz2016us ; wan2024electrocardiogram ;
chen2024bimcv ; liu2024zero . This has necessitated the automation of
radiology report generation. Existing work on radiology report generation
primarily focuses on 2D images such as chest X-rays tanida2023interactive ;
li2023unify ; wan2023med . However, compared to 3D images, 2D images offer
limited diagnostic information and therefore fall short in identification of
some of the most critical and life-threatening diseases such as pulmonary
opacities and early-stage cancer bradley2019sensitivity ; self2013high .

In 3D medical image analysis, high-resolution (HR) images are more clinically
informative than their low-resolution (LR) counterparts, as certain diseases,
such as pulmonary nodules, are difficult to detect in LR images
liu2021application . Despite the clinical advantages of HR 3D medical images,
current studies downsample HR volumes (e.g., $512\times 512\times 256$) to LR
($256\times 256\times 32$) for radiology report generation with large language
models (LLMs) wu2023towards ; bai2024m3d . This results in significant
information loss ($512\hskip 2.84526ptvs.\hskip 2.84526pt256$, $256\hskip
2.84526ptvs.\hskip 2.84526pt32$) and could miss lesions that are challenging
to detect in LR views. The main reason for downsampling is the increased
computational cost when processing the larger number of visual tokens from HR
3D volumes by LLMs, as illustrated in Fig 1 (left). For instance, a standard
CT scan singh2009computer with dimensions of $512\times 512\times 256$
generates 4,096 visual tokens, each measuring $32\times 32\times 16$.
Processing these tokens with common multimodal large language model (MLLM)
architectures, such as LLaVA liu2023visual combined with Llama3-8B
llama3modelcard , often leads to out-of-memory (OOM) issues during training on
singe A100-80GB GPU, as depicted in Fig 1 (right). Another approach involves
processing the 3D volume slice-by-slice ct2rep , potentially losing crucial 3D
context and consistency along the transverse axis. This leads to a critical
challenge: Can we generate radiology reports for HR 3D medical images using
LLMs in a compute-efficient manner while preserving HR information?

To address this challenge, our work makes three contributions:

  * •

We introduce High-resolution Informing Low-resolution Tokens (HILT), which
effectively mines HR information through LR visual queries. This approach
allows LLMs to process costs based solely on the number of LR queries instead
of the increasing number of HR tokens with higher resolution inputs, while
still preserving HR information for enhanced performance, as shown in Fig 1.

  * •

To thoroughly benchmark the 3D High-Resolution Medical Image Report Generation
(3DHRG) task, we curate a new large-scale dataset based on publicly accessible
sources, consisting of 5,328 paired HR 3D CT volumes and radiology reports,
named BIMCV-RG111This dataset will be released post-acceptance. This dataset
allows the community to assess the generalizability of the 3DHRG task under
zero-shot domain transfer scenarios, a significant addition given there was
only one large-scale dataset previously available for this task.

  * •

We establish the first benchmark for the 3DHRG task across two large datasets
and three settings: normal-resolution, high-resolution inputs, and zero-shot
domain transfer. Our method consistently achieves superior performance across
all the settings for both datasets with a fixed computational cost for LLMs
processing visual tokens.

##  2 Related Works

Multimodal Large Language Models. The development of Multimodal Large Language
Models (MLLMs) combines computer vision and natural language processing to
enhance visual and linguistic capabilities. This integration is crucial for
tasks that require both visual perception and language comprehension. Models
like CLIP radford2019language , Flamingo alayrac2022flamingo , and BLIP
li2023blip have improved the alignment between these modalities by using
extensive image-text data samples, resulting in significant performance gains.
LLaVA liu2023visual further simplifies this approach by using a basic linear
projector for aligning visual features to language space, involving only a few
learnable parts and tailored instruction data to fully leverage the model’s
strong capabilities. Despite their effectiveness, these approaches focus
mainly on 2D images. To explore the underexplored area of 3D medical images,
such as CT scans, we extend the LLaVA framework to incorporate our customized
3D vision encoder and HILT. Our aim is to address the challenge of generating
radiology reports for 3D medical images.

Radiology Report Generation for 2D Medical Images. Radiology report generation
for 2D medical images involves three primary approaches: (1) Template
Selection and Generation, which focuses on sentence crafting and
differentiating regions using dual writers li2018hybrid ; jing2017automatic ;
wan2023efficient . (2) Data Integration and Coherence, that integrates
knowledge for consistency and highlights abnormal features liu2021exploring ;
ma2021contrastive . (3) Cross-Modal alignment, using attention mechanisms to
align textual and visual features and improve semantic accuracy gu2024complex
; tanida2023interactive ; li2023unify . However, these methods are limited in
2D imaging, such as chest X-rays, and adapting them to 3D medical imaging like
CT scans is challenging. This difficulty arises from the inherent differences
between the 3D context of volumetric data and 2D multichannel images, as shown
in Fig 2 (a). For example, in CT scans, each slice captures unique patterns:
upper slices might show the lungs and heart, while lower ones display
abdominal organs like the liver, underscoring the unique spatial information
in each slice. In contrast, 2D images exhibit only variations in intensity
across uniform spatial structures in all channels adegun2021deep ;
puttagunta2021medical ; shen2016reappraisal ; ker2017deep ; singh20203d .

Radiology Report Generation for 3D Medical Images. Although 3D medical imaging
is crucial for clinical decision making, only a few studies such as CT2Rep
hamamci2024ct2rep , RadFM wu2023towards , and M3D bai2024m3d have explored the
task of report generation for such images and provided official code for
reimplementations. However, these methods have limitations: as depicted in Fig
2 (b), CT2Rep processes HR 3D volumes slice by slice, sacrificing the 3D
context and potentially losing consistency across the transverse axis. In Fig
2 (c), RadFM downsamples HR inputs to LR views, losing critical information
from original HR 3D volumes, while M3D not only downsamples HR inputs but also
applies spatial pooling to visual tokens, further reducing spatial
information. Additionally, all three methods process visual tokens with LLMs
for report generation, resulting in increased computational costs as volume
resolution increases. Maintaining HR information in medical imaging is
crucial, as reducing the coronal and sagittal resolution could corrupt crucial
subtle abnormal patterns bradley2019sensitivity ; self2013high , diminishing
the quality of the generated reports. Additionally, clinical practice often
prefers 256-slice CT scans over those with fewer slices for their enhanced
detail and diagnostic value klass2010prospectively ; chua2013diagnostic , as
downsampling on the transverse axis reduces the model’s ability to capture
these 3D patterns, thereby limiting its clinical applicability. In this work,
we design a method to generate radiology reports for 3D HR medical images that
preserves HR information without downsampling or pooling. This approach keeps
computational costs constant for processing visual tokens with LLMs,
regardless of volume resolution. It consistently outperforms existing methods
across resolutions from $128\times 128\times 64$ to $512\times 512\times 256$,
as shown in Figure 1.

##  3 Generating Radiology Report for HR 3D Medical Images

Our framework is straightforward yet effective: it utilizes a single vision
encoder to extract visual embeddings simultaneously from both LR and HR views
of a given 3D volume. Subsequently, we introduce High-resolution Informing
Low-resolution Tokens (HILT), which treats LR visual tokens as queries and
enhances them by integrating mined regional information from all HR visual
tokens. Moreover, the LLMs exclusively process HILT for report generation,
rather than the entire set of HR visual tokens. This approach ensures
efficiency by limiting the computational cost to a constant as the number of
LR queries, achieved without resorting to brute-force downsampling and spatial
pooling, thereby preserving HR information.

Figure 2:  Mainstream radiology report generation methods for 3D medical
images using LLMs. We use red and green box to highlight existing
deficiencies, and denote our method’s advantages in bleudefrance. We omit text
instructions input branch for all methods for simplicity. (a): Methods
liu2023visual ; wang2023r2gengpt ; zhang2023pmc process 3D volumes as 2D
inputs, losing the inherent 3D nature. (b): CT2Rep ct2rep processes 3D volumes
in slice-wise using a shared encoder. The LLM processes all slice visual
tokens for generating reports, leading to increased computational costs as the
resolution increases. (c): RadFM and M3D wu2023towards ; bai2024m3d downsample
3D HR inputs to LR views, risking loss of critical spatial details. M3D
further pools LR visual tokens, reducing spatial information in the latent
space. Both methods process all visual tokens with the LLM, increasing
computational costs with higher resolutions. (d): Our method simultaneously
obtains LR and HR visual tokens with a single vision encoder, using HR tokens
to inform LR tokens. This preserves HR detail while only processing LR queries
through the LLM, minimizing computational costs.

###  3.1 Unified Processing of LR and HR 3D Volumes Using Single Visual
Encoder.

In our framework, the standard ViT dosovitskiy2020image serves as the 3D
vision encoder, represented by $E_{V}$, with a 3D visual tokenizer for
tokenizing 3D volume input into patches. This 3D vision encoder optimizes
computational efficiency by simultaneously processing both LR and HR views of
3D volumes. As illustrated in Fig 2 (d), the HR 3D volume, denoted as
$X^{\textrm{HR}}\in\mathbb{R}^{H\times W\times D}$, where $H$, $W$, and $D$
represent the dimensions of height, width, and depth (number of slices),
respectively. The LR view is obtained from the HR input volume using bilinear
interpolation and is represented as
$X^{\textrm{LR}}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times D^{\prime}}$,
where $H^{\prime}\leq H$, $W^{\prime}\leq W$, and $D^{\prime}\leq D$ indicate
the reduced dimensions of the LR view. Both HR and LR views are encoded using
the same vision encoder. The positional embeddings are initialized based on
the dimensions of the HR volume and interpolated to match the LR volume
dimensions during the encoding of the LR view. The resulting HR visual
embeddings, denoted as $Z^{\textrm{HR}}\in\mathbb{R}^{N\times C}$, where $N$
represents the number of HR visual tokens, and $C$ represents the embedding
dimensions. Similarly, the LR visual embeddings are represented as
$Z^{\textrm{LR}}\in\mathbb{R}^{N^{\prime}\times C}$, where $N^{\prime}$ is the
number of LR visual tokens. The visual encoding process is formulated as
follows:

$\displaystyle Z^{\textrm{HR}}=E_{V}(X^{\textrm{HR}})$ $\displaystyle,\hskip
5.69054ptZ^{\textrm{LR}}=E_{V}(X^{\textrm{LR}})$ (1) $\displaystyle
Z^{\textrm{HR}}\in\mathbb{R}^{N\times C}$ $\displaystyle,\hskip
5.69054ptZ^{\textrm{LR}}\in\mathbb{R}^{N^{\prime}\times C}$ (2)

###  3.2 HILT: High-resolution Informing Low-resolution Tokens.

We utilize the previously generated HR features, $Z^{\textrm{HR}}$, to enrich
LR features, $Z^{\textrm{LR}}$. This process creates HR-informed LR visual
tokens, integrating HR information into LR tokens without corrupting HR
spatial information through downsampling or spatial pooling. Specifically, LR
visual tokens, $Z^{\textrm{LR}}$, serve as queries,
$Q\in\mathbb{R}^{N^{\prime}\times C}$. Simultaneously, HR visual tokens,
$Z^{\textrm{HR}}$, act as both keys, $K\in\mathbb{R}^{N\times C}$, and values,
$V\in\mathbb{R}^{N\times C}$. This process is depicted in Fig 2 (d). The
operation of HR informing LR tokens is mathematically described as:

$Z^{\textrm{HILT}}=\mathcal{F}\left(Q+\operatorname{Softmax}\left(QK^{T}\right)V\right),\hskip
5.69054ptZ^{\textrm{HILT}}\in\mathbb{R}^{N^{\prime}\times C}$ (3)

Here, $\mathcal{F}$ represents the two-layer projector. We ablate the number
of layers of the projector in Sec A.5 and show it in Fig 8.

During this operation, each LR query interacts with all regional HR visual
features, preserving these HR features in the relevant LR query. Additionally,
we merge the LR queries that have not interacted with HR visual tokens with
those that have, ensuring that global information is retained. This prevents
the loss of global context from LR queries after interactions with regional HR
features. Consequently, $Z^{\textrm{HILT}}$ includes both global and regional
features, essential for generating radiology reports that require both
comprehensive descriptions and specific regional details. This method
compresses a vast number of HR tokens into fewer LR queries, ensuring that the
number of visual tokens processed by the LLM is consistent with the number of
LR queries, even as the HR input resolution increases. This approach maintains
computational efficiency and preserves HR information in $Z^{\textrm{HILT}}$,
allowing us to manage very high-resolution inputs at an acceptable cost. It’s
worth noting that we ablate the LR queries by randomly initialized queries,
similar to BLIP li2023blip in Tab 3, and observe a significant performance
drop, as illustrated in Sec 5.5.

###  3.3 Generating Radiology Reports.

We concatenate $Z^{\textrm{HILT}}$, with input textual instruction tokens
$Z^{\textrm{T}}$ for LLMs processing, facilitating auto-regressive report
generation, similar to the strategy employed in LLaVA liu2023visual . The
$Z^{\textrm{HILT}}$ serve as the visual features in the LLM input, with their
count always matching the number of LR queries. This strategy effectively
reduces computational costs associated with an increasing number of HR visual
tokens, particularly as the resolution of 3D volumes rises, without
sacrificing critical HR information from the original HR 3D volumes.

$\textrm{Generated
Report}=\textrm{LLM}(\text{concat}(Z^{\textrm{HILT}},Z^{\textrm{T}}))$ (4)

##  4 Benchmarking Radiology Report Generation on HR 3D Medical Image

###  4.1 Datasets

CT-RATE 222https://huggingface.co/datasets/ibrahimhamamci/CT-RATE. This
dataset consists of 50,188 3D chest CT volumes, each paired with a radiology
text report. After filtering out duplicate reports, we obtain 24,284 unique CT
volume-report pairs. To create training, validation, and testing sets, we
randomly split the data into 70%, 10%, and 20%, respectively. Following
previous report generation studies wang2023r2gengpt , we use only the
‘FINDINGS’ section of the reports for our generation tasks.
BIMCV-RG. As CT-RATE is currently the only large-scale publicly accessible
dataset for the 3DHRG task, it is insufficient for establishing a
comprehensive benchmark. To better assess generalizability of methods, we
curated a new dataset based on the BIMCV bimcv . Details of the curation steps
are provided in Sec A.3. After data curation, we compiled a dataset named
BIMCV-RG 333This dataset will be released upon acceptance., consisting of
5,328 samples, each comprising one HR 3D volume and a unique radiology report.
We divided BIMCV-RG into training, validation, and testing subsets, comprising
70%, 10%, and 20% of the data, respectively. Since the dataset does not
provide separate ‘FINDINGS’ and ‘IMPRESSIONS’, we use the entire report for
all experiments.

###  4.2 Evaluation Metrics

The evaluation of all methods is conducted using nine metrics: BLEU 1-4
papineni2002bleu , METEOR banerjee2005meteor , ROUGE 1-2 and L lin2004rouge ,
and CIDEr vedantam2015cider . It should be noted that the the original BIMCV
bimcv dataset lacks abnormality annotations for each radiology report,
therefore, clinical efficacy tanida2023interactive are not applicable in the
evaluations for this work.

###  4.3 Baselines on Radiology Report Generation

To our knowledge, no existing benchmark evaluates report generation for 3D HR
medical images. Therefore, we select five methods processing HR 3D volumes to
establish baselines for 3D High-Resolution Medical Image Report Generation
(3DHRG) tasks. We graphically describe these baselines in Fig 2. For fair and
consistent comparison, we utilize Llama3-8B-Instruct llama3modelcard as the
language decoder for all compared methods.

Baselines Using 2D Inputs:
R2GenGPT wang2023r2gengpt . This method extracts visual tokens with a 2D
visual encoder, projected into the LLM space for report generation. As the
model doesn’t accept 3D inputs, we treat the 3D volume as 2D multi-channel
images, modifying only the image tokenizer. Since this work originally
designed for chest X-ray images wang2023r2gengpt , we’ve reimplemented their
official code 444https://github.com/wang-zhanyu/R2GenGPT and training strategy
for both datasets in this work.
MedVInT zhang2023pmc . This method, similar to R2GenGPT, processes 3D inputs
as 2D multi-channel images. We adapt the image tokenizer to accommodate 3D
inputs, utilizing the publicly available MedVInT-TD checkpoint. Fine-tuning
followed procedures outlined in the official code
555https://github.com/xiaoman-zhang/PMC-VQA.
Baselines Using 3D Slice Window Inputs:
CT2Rep ct2rep . This method divides the 3D volume into multiple windows, each
containing several slices. The visual encoder processes each window
sequentially and concatenates all slice visual tokens for input to the
language decoders, which generate the report. While capable of accepting 3D
input, it still processes the entire volume slice-wise, potentially
compromising its 3D nature and introducing inconsistent artifacts. For fair
comparison, we adopted the official code
666https://github.com/ibrahimethemhamamci/CT2Rep.
Baselines Using 3D Volume Inputs:
RadFM wu2023towards . This method is able to processes 3D volume inputs but is
restricted to downsampled versions of original HR 3D medical images, as
explained in their original publication wu2023towards . They directly process
all 3D visual tokens with LLM to generate radiology reports, incurring high
computational costs as the number of tokens increases with HR volume inputs.
We utilize their released checkpoints 777https://github.com/chaoyi-wu/RadFM
for the 3D vision encoder for fair comparison with the original work.
M3D bai2024m3d . M3D is a MLLM designed to process HR 3D volumes, utilizing
downsampling at the input level and learnable spatial pooling at the latent
level to decrease computational demands bai2024m3d . However, the use of small
patch sizes introduces heavy computational costs due to the large number of
visual tokens. We reimplement their approach on CT-RATE and BIMCV-RG datasets
using their official code 888https://github.com/BAAI-DCAI/M3D.

##  5 Experiments and Evaluation

###  5.1 Implementation Details.

We utilize the standard ViT-Base architecture for our 3D vision encoder,
training it from scratch with a patch size of $32\times 32\times 16$. We fixed
the LR input size to $128\times 128\times 64$ to minimize computational costs.
Additionally, we conduct ablation experiments on the LR input resolution, as
shown in Tab 4 and discussed further in Sec A.5. The langauge decoder employed
is Llama3-8B-Instruct llama3modelcard . Ablation studies on both the ViT and
LLM at various scales are detailed in Sec 5.5. For different LLMs, ablation
results are specifically discussed in Sec A.5. The details of training are
detailed in Sec A.6 and listed in Tab 5. For data preprocessing, we detail our
strategy in Sec A.4, applying the same processing techniques across all
methods for a fair comparison. An exception is made for methods based on 2D
inputs; here, we convert the 3D volumes to 2D multichannel images to
accommodate volume inputs.

Table 1:  Average metrics are computed across nine metrics from Fig 3 for
3DHRG tasks on the CT-RATE and BIMCV-RG datasets, using both normal-resolution
and high-resolution settings for input volumes. The best results are
highlighted in bold. ‘Number of Visual Tokens’ refers to the quantity of
visual tokens processed by LLMs for report generation. ‘OOM’ indicates the
method cannot be trained on an 80 GB A100-80G GPU due to out-of-memory issues
when $\textrm{batchsize}=1$. The light gray color indicates that input 3D
volumes are converted to non-3D format to match the configurations of the
compared methods.

Methods | 3D Volume | Input Image | Number of | Large Language | CT-RATE | BIMCV-RG
---|---|---|---|---|---|---
| Input? | Resolution | Visual Tokens | Model | Averaged Metric | Averaged Metric
Normal-Resolution Setting
R2GenGPT wang2023r2gengpt | ✗ | $256\times 256\times 128$ | 256 | Llama3-8B-Instruct | 34.16 | 31.17
MedVInT zhang2023pmc | ✗ | $256\times 256\times 128$ | 256 | Llama3-8B-Instruct | 34.31 | 31.89
CT2Rep ct2rep | ✗ | $256\times 256\times 128$ | 1066 | Llama3-8B-Instruct | 37.02 | 34.14
RadFM wu2023towards | ✓ | $256\times 256\times 128$ | 512 | Llama3-8B-Instruct | 39.96 | 37.74
M3D bai2024m3d | ✓ | $256\times 256\times 128$ | 1024 | Llama3-8B-Instruct | 41.59 | 37.91
HILT(Ours) | ✓ | $256\times 256\times 128$ | 64 | Llama3-8B-Instruct | 43.83 | 38.79
High-Resolution Setting
R2GenGPT wang2023r2gengpt | ✗ | $512\times 512\times 256$ | 1024 | Llama3-8B-Instruct | 42.86 | 34.08
MedVInT zhang2023pmc | ✗ | $512\times 512\times 256$ | 1024 | Llama3-8B-Instruct | 43.13 | 35.84
CT2Rep ct2rep | ✗ | $512\times 512\times 256$ | 9408 | Llama3-8B-Instruct | OOM | OOM
RadFM wu2023towards | ✓ | $512\times 512\times 256$ | 4096 | Llama3-8B-Instruct | OOM | OOM
M3D bai2024m3d | ✓ | $512\times 512\times 256$ | 8192 | Llama3-8B-Instruct | OOM | OOM
HILT(Ours) | ✓ | $512\times 512\times 256$ | 64 | Llama3-8B-Instruct | 46.05 | 40.96

Figure 3: Results of 3DHRG tasks on BIMCV-RG and CT-RATE datasets are
evaluated using nine metrics mentioned in Sec 4.2. ‘OOM’ indicates that the
method cannot be trained at the setting on a single A100-80G GPU when
$\textrm{batchsize}=1$. Top: Methods with an input resolution of $256\times
256\times 128$. Bottom: Methods with an input resolution of $512\times
512\times 256$.

###  5.2 Main Results

Normal Resolution.  In Tab 1, our method is compared with existing approaches
across both normal and high-resolution settings, as well as 2D and 3D input
formats. Notably, LR queries derived from an LR view of the 3D volume, which
has dimensions of $128\times 128\times 64$, result in 64 LR queries for
generating $Z^{\textrm{HILT}}$. The averaged metric across nine metrics
described in Sec 4.2 is reported in Tab 1, with detailed comparisons for the
CT-RATE and BIMCV-RG datasets displayed in Fig. 3 top panel. This highlights
our method’s consistent superiority over existing methods across all metrics.
From the efficiency aspect, the LLM in our method processes only 64 tokens
from HILT for report generation, significantly fewer than those handled by the
compared approaches, underscoring the superior performance achieved with fewer
visual tokens and highlighting the effectiveness of our method in both
efficacy and efficiency.

Existing methods using 3D input outperform those using non-3D input,
suggesting the necessity of 3D input for 3DHRG tasks. 2D input methods process
fewer tokens but sacrifice the inherent 3D nature of the volume, leading to
sub-optimal performance. Hence, preserving 3D context is crucial for better
performance in 3DHRG tasks.

High Resolution.  As suggested by clinical standards bradley2019sensitivity ;
self2013high ; chua2013diagnostic , HR 3D medical images provide richer
information compared to lower variants. Thus, we increase the input volume
resolution from $256\times 256\times 128$ to $512\times 512\times 256$.
However, this increase in volume resolution leads to a higher number of visual
tokens processed by LLMs in existing methods, resulting in OOM issues for
CT2Rep, RadFM, and M3D hamamci2024ct2rep ; wu2023towards ; bai2024m3d ,
rendering them untrainable on a single A100-80GB GPU following their official
configurations. Only 2D input-based methods are available under HR settings.

Despite the increased resolution of the input volume, our method maintains a
constant number of visual tokens processed by the LLM, set at 64 as the number
of LR queries. In our method, LLM only processes the HILT rather than all HR
visual tokens, ensuring computational efficiency. Our method reaches the
highest performance across all metrics as depicted in Fig 3 bottom panel, and
with the lowest cost from the number of LLM processing visual tokens, as shown
in Tab 1 HR setting. Demonstrating its effectiveness in handling HR 3D medical
images for 3DHRG tasks.

###  5.3 Zero-shot Domain Transferring

Methods |  CT-RATE $\rightarrow$ BIMCV-RG |  BIMCV-RG $\rightarrow$ CT-RATE
---|---|---
| Averaged Metric | Averaged Metric
R2GenGPT wang2023r2gengpt | 30.26 | 32.64
MedVInT zhang2023pmc | 30.19 | 32.89
CT2Rep ct2rep | 32.73 | 35.14
RadFM wu2023towards | 33.91 | 35.92
M3D bai2024m3d | 34.22 | 36.34
HILT(Ours) | 36.54 | 40.21

Table 2:  Results of the 3DHRG task under zero-shot domain transfer scenario.
Reported results represent the average of nine metrics described in Sec 4.2.
‘CT-RATE $\rightarrow$ BIMCV-RG’ indicates training on CT-RATE and testing on
BIMCV-RG, and vice versa. Best results are highlighted in bold.

To establish a comprehensive benchmark for 3DHRG, we explore an extreme
scenario: zero-shot domain transfer. In this setup, models are trained on one
dataset and tested on another, simulating real clinical applications where
test data distribution is typically unknown and differs from the training set.
We utilize CT-RATE and our proposed dataset BIMCV-RG for this benchmark. These
datasets are sourced from distinct populations (Turkish vs. Spanish) and
settings (single center vs. multiple centers), underlining the challenge of
this scenario and the significance of BIMCV-RG. Previously, only the CT-RATE
dataset was available for 3DHRG, limiting the feasibility of zero-shot domain
transfer. Tab 2 presents the results for all baseline methods under this
scenario. We restrict the input resolutions to $256\times 256\times 128$ due
to OOM issues encountered by CT2Rep ct2rep , RadFM wu2023towards , and M3D
bai2024m3d with higher resolutions. Our method consistently outperforms all
existing methods under the zero-shot domain transfer scenario across both
datasets, showcasing the robustness and generalizability of our approach.

Figure 4:  Qualitative comparison between ground truth and reports generated
by existing methods and ours. Blue indicates imaging technology. Green
indicates normal information. Brown indicates abnormality. Red highlights
incorrect content. Except for the ‘HR’ setting, methods process inputs at
normal resolution, as shown in Tab 1. ‘HR’ denotes high-resolution input.

###  5.4 Visualization of Qualitative Results

In Fig 4, we present generated reports alongside ground truth from BIMCV-RG,
focusing on three aspects: (1) imaging technology, (2) normal patterns, and
(3) abnormal patterns. In RadFM wu2023towards and M3D bai2024m3d , both
methods concentrate solely on the CT modality, neglecting other imaging
details and providing very few accurate normal and abnormal contents, often
leading to incorrect interpretations. For instance, RadFM incorrectly mentions
‘pleural effusion’, ‘no other consolidation’, and ‘not suggestive of COVID’,
despite the ground truth indicating the opposite. Similarly, M3D struggles to
identify COVID-19, disregarding clear indications from the ground truth, while
also disregarding most normal and abnormal pattern information, resulting in
irrelevant content. These deficiencies arise from downsampling and spatial
pooling strategies that compromise crucial spatial information from the 3D
medical images.

In contrast, our method produces accurate and detailed content, such as ‘CT
angiography of pulmonary arteries and veins of the lower limbs,’ accurately
capturing imaging technology. Our method also correctly identifies normal and
abnormal patterns, for instance, ‘without pericardial effusion’ and ‘faint
ground-glass opacities.’ As input resolution increases, our method captures
subtler information, like ‘venography up to the popliteal region,’ specifying
the CT scan area. ‘Central airway of normal appearance’ suggests subtle
regional pathology patterns, which are hard to observe in lower resolution
input. Similarly, ‘Subtle ground-glass opacities in both upper lobes’ pinpoint
abnormality and location, benefiting from HR input. These findings highlight
our method’s superior capability to generate precise and comprehensive
radiology reports from 3D medical images compared to existing approaches at
normal resolution setting. Additionally, our method enhances report quality in
high-resolution setting, contrasting with existing methods encountering OOM
issues.

Table 3:  Results of ablation studies with each component. The baseline model,
LLaVA-1.5 liu2023visual , processes 3D input by converting 3D volume to 2D
images, with the transverse axis as the channel axis. ‘3D Vision Encoder’
indicates directly processing input as a 3D volume. ‘Rand. Init. Queries’
indicates using randomly initialized queries to compress HR visual tokens
following BLIP li2023blip strategy. ‘HILT’ indicates our method uses LR
queries to mine HR information and compress HR visual tokens. ‘Number of
Visual Tokens’ represents the number of visual tokens processed by LLMs. ‘-’
indicates no LR input in this setting. Best results are in bold.

Methods | Input Image | Input Image | Number of | CT-RATE | BIMCV-RG
---|---|---|---|---|---
| (HR) | (LR) | Visual Tokens | Averaged Metric | Averaged Metric
Normal-Resolution Setting
Baseline (LLaVA-1.5) | $256\times 256\times 128$ | - | 256 | 33.42 | 30.76
+3D Vision Encoder | $256\times 256\times 128$ | - | 512 | 36.52 | 35.19
+Rand. Init. Queries | $256\times 256\times 128$ | - | 64 | 36.23 | 34.79
+HILT (Ours) | $256\times 256\times 128$ | $128\times 128\times 64$ | 64 | 43.83 | 38.69
High Resolution Setting
Baseline (LLaVA-1.5) | $512\times 512\times 256$ | - | 1024 | 35.63 | 33.17
+3D Vision Encoder | $512\times 512\times 256$ | - | 4096 | OOM | OOM
+Rand. Init. Queries | $512\times 512\times 256$ | - | 64 | 41.14 | 36.21
+HILT (Ours) | $512\times 512\times 256$ | $128\times 128\times 64$ | 64 | 46.05 | 40.96

Figure 5:  Results of ablation studies across various factors: (a): Averaged
metrics for both datasets, varying spatial resolution (sagittal and coronal
axes) from $128\times 128$ to $512\times 512$, with a fixed number of 256
slices. (b): Averaged metrics for both datasets with various number of slices
and a fixed spatial resolution of $256\times 256$. (c): Averaged metrics for
both datasets using different scales of 3D vision encoders. (d): Averaged
metrics for both datasets using various scales of LLMs, limited to 8B and 70B
scales provided by Llama3 llama3modelcard .

###  5.5 Analysis

In this section, we evaluate various components, input resolutions, and scales
of vision and language models. All ablation studies use five unique random
seeds, and we report averages and standard deviations across five runs for
nine metrics, as detailed in Sec 4.2.

Ablation Studies on Each Components.  In Tab 3, we set LLaVA-1.5 liu2023visual
as our baseline, as our method builds upon its architecture. Across both
resolution settings, variants with 3D input consistently outperform baselines,
highlighting the necessity of 3D context in 3DHRG tasks. While compressing the
number of HR visual tokens with randomly initialized queries can reduce
computational costs, it leads to poorer performance, even falling short of
directly processing all HR visual tokens under normal-resolution settings. Our
method, HILT, utilizing LR queries to compress HR visual tokens, significantly
outperforms variants using randomly initialized queries in both resolution
settings. This can be attributed to each LR query indicating a specific region
in the 3D volume and interacting with all regional HR features, thereby
preserving HR information in LR tokens along with spatial relations.
Conversely, randomly initialized queries lose spatial information from the HR
volume, as they do not reflect any spatial relation to the original 3D
volumes.
Affect of Resolution of 3D Medical Image.  We analyze the impact of varying
spatial resolutions along the sagittal and coronal axes, as well as the number
of slices along the transverse axis, as shown in Fig 5 (a) and (b). We find
that increased resolution across all three axes enhances the 3DHRG task,
demonstrating the benefits of HR 3D volumes for radiology report generation.
With our efficient method, we can handle HR input at an acceptable cost while
preserving HR information.
Scalability Analysis.  We evaluate the scalability of our method on vision and
language models in Fig 5 (c) and (d), respectively. For the vision encoder, we
employ three scales: ViT-Tiny, ViT-Small, and ViT-Base dosovitskiy2020image .
Regarding language models, we scale from 8B to 70B, as Llama3-Instruct
llama3modelcard offers versions only in these sizes. Notably, when training
with the 80B version, we utilize QLoRA dettmers2024qlora with int4 precision
due to hardware limitations. As depicted in Fig 5 (c) and (d), our method
consistently improves in performance with both vision and language model
scaling. This indicates the scalability of our model and the potential
effectiveness of large-scale models for the 3DHRG task.

##  6 Conclusion

In this work, we introduce a framework for generating radiology reports from
HR 3D medical images, maintaining acceptable costs while preserving HR
details. At the central of our approach is the High-resolution Informing Low-
resolution Tokens (HILT), which inegrates HR information into LR queries,
enhancing efficiency and effectiveness for handling 3D volumes up to
$512\times 512\times 256$. We establish the first benchmark for the 3DHRG
task, compare it with existing methods, and introduce the BIMCV-RG dataset for
comprehensive benchmarking across three settings: normal and high-resolution
inputs, and zero-shot domain transfer. Thanks to HILT, our method demonstrates
superior performance on two datasets in these settings at an acceptable cost.
We anticipate that our benchmarks and dataset will significantly advance the
field of radiology report generation for 3D medical images.

## References

  * (1) C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, “Towards generalist foundation model for radiology,” _arXiv preprint arXiv:2308.02463_ , 2023.
  * (2) F. Bai, Y. Du, T. Huang, M. Q.-H. Meng, and B. Zhao, “M3d: Advancing 3d medical image analysis with multi-modal large language models,” _arXiv preprint arXiv:2404.00578_ , 2024.
  * (3) S. Bastawrous and B. Carney, “Improving patient safety: avoiding unread imaging exams in the national va enterprise electronic health record,” _Journal of digital imaging_ , vol. 30, pp. 309–313, 2017.
  * (4) A. Rimmer, “Radiologist shortage leaves patient care at risk, warns royal college,” _BMJ: British Medical Journal (Online)_ , vol. 359, 2017.
  * (5) A. B. Rosenkrantz, D. R. Hughes, and R. Duszak Jr, “The us radiologist workforce: an analysis of temporal and geographic variation by using large national datasets,” _Radiology_ , vol. 279, no. 1, pp. 175–184, 2016.
  * (6) Z. Wan, C. Liu, X. Wang, C. Tao, H. Shen, Z. Peng, J. Fu, R. Arcucci, H. Yao, and M. Zhang, “Electrocardiogram instruction tuning for report generation,” _arXiv preprint arXiv:2403.04945_ , 2024.
  * (7) Y. Chen, C. Liu, X. Liu, R. Arcucci, and Z. Xiong, “Bimcv-r: A landmark dataset for 3d ct text-image retrieval,” _arXiv preprint arXiv:2403.15992_ , 2024.
  * (8) C. Liu, Z. Wan, C. Ouyang, A. Shah, W. Bai, and R. Arcucci, “Zero-shot ecg classification with multimodal learning and test-time clinical knowledge enhancement,” _arXiv preprint arXiv:2403.06659_ , 2024.
  * (9) T. Tanida, P. Müller, G. Kaissis, and D. Rueckert, “Interactive and explainable region-guided radiology report generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 7433–7442.
  * (10) Y. Li, B. Yang, X. Cheng, Z. Zhu, H. Li, and Y. Zou, “Unify, align and refine: Multi-level semantic alignment for radiology report generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 2863–2874.
  * (11) Z. Wan, C. Liu, M. Zhang, J. Fu, B. Wang, S. Cheng, L. Ma, C. Quilodrán-Casas, and R. Arcucci, “Med-unic: Unifying cross-lingual medical vision-language pre-training by diminishing bias,” _arXiv preprint arXiv:2305.19894_ , 2023.
  * (12) S. H. Bradley, S. Abraham, M. E. Callister, A. Grice, W. T. Hamilton, R. R. Lopez, B. Shinkins, and R. D. Neal, “Sensitivity of chest x-ray for detecting lung cancer in people presenting with symptoms: a systematic review,” _British Journal of General Practice_ , vol. 69, no. 689, pp. e827–e835, 2019.
  * (13) W. H. Self, D. M. Courtney, C. D. McNaughton, R. G. Wunderink, and J. A. Kline, “High discordance of chest x-ray and computed tomography for detection of pulmonary opacities in ed patients: implications for diagnosing pneumonia,” _The American journal of emergency medicine_ , vol. 31, no. 2, pp. 401–405, 2013.
  * (14) S. Liu, H. Liu, P. Li, and L. Jiang, “Application of high-resolution ct images information in complicated infection of lung tumors,” _Journal of Infection and Public Health_ , vol. 14, no. 3, pp. 418–422, 2021.
  * (15) M. Singh and H. Singh, “Computer assisted imaging for nodule detection on a multi-slice ct image,” in _World Congress on Medical Physics and Biomedical Engineering, September 7-12, 2009, Munich, Germany: Vol. 25/5 Information and Communication in Medicine, Telemedicine and e-Health_. Springer, 2009, pp. 301–304.
  * (16) H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” _arXiv preprint arXiv:2304.08485_ , 2023.
  * (17) AI@Meta, “Llama 3 model card,” 2024. [Online]. Available: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
  * (18) I. E. Hamamci, S. Er, and B. Menze, “Ct2rep: Automated radiology report generation for 3d medical imaging,” _arXiv preprint arXiv:2403.06801_ , 2024.
  * (19) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * (20) J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds _et al._ , “Flamingo: a visual language model for few-shot learning,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 23 716–23 736, 2022.
  * (21) J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” _arXiv preprint arXiv:2301.12597_ , 2023.
  * (22) Y. Li, X. Liang, Z. Hu, and E. P. Xing, “Hybrid retrieval-generation reinforced agent for medical image report generation,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * (23) B. Jing, P. Xie, and E. Xing, “On the automatic generation of medical imaging reports,” _arXiv preprint arXiv:1711.08195_ , 2017.
  * (24) Z. Wan, X. Wang, C. Liu, S. Alam, Y. Zheng, J. Liu, Z. Qu, S. Yan, Y. Zhu, Q. Zhang, M. Chowdhury, and M. Zhang, “Efficient large language models: A survey,” 2023.
  * (25) F. Liu, X. Wu, S. Ge, W. Fan, and Y. Zou, “Exploring and distilling posterior and prior knowledge for radiology report generation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 13 753–13 762.
  * (26) X. Ma, F. Liu, C. Yin, X. Wu, S. Ge, Y. Zou, P. Zhang, and X. Sun, “Contrastive attention for automatic chest x-ray report generation,” _arXiv preprint arXiv:2106.06965_ , 2021.
  * (27) T. Gu, D. Liu, Z. Li, and W. Cai, “Complex organ mask guided radiology report generation,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2024, pp. 7995–8004.
  * (28) A. A. Adegun, S. Viriri, and R. O. Ogundokun, “Deep learning approach for medical image analysis,” _Computational Intelligence and Neuroscience_ , vol. 2021, pp. 1–9, 2021.
  * (29) M. Puttagunta and S. Ravi, “Medical image analysis based on deep learning approach,” _Multimedia tools and applications_ , vol. 80, no. 16, pp. 24 365–24 398, 2021.
  * (30) X.-H. Shen, B.-Y. Su, J.-J. Liu, G.-M. Zhang, H.-D. Xue, Z.-Y. Jin, S. A. Mirjalili, and C. Ma, “A reappraisal of adult thoracic and abdominal surface anatomy via ct scan in chinese population,” _Clinical anatomy_ , vol. 29, no. 2, pp. 165–174, 2016.
  * (31) J. Ker, L. Wang, J. Rao, and T. Lim, “Deep learning applications in medical image analysis,” _Ieee Access_ , vol. 6, pp. 9375–9389, 2017.
  * (32) S. P. Singh, L. Wang, S. Gupta, H. Goli, P. Padmanabhan, and B. Gulyás, “3d deep learning on medical images: a review,” _Sensors_ , vol. 20, no. 18, p. 5097, 2020.
  * (33) I. E. Hamamci, S. Er, and B. Menze, “Ct2rep: Automated radiology report generation for 3d medical imaging,” _arXiv preprint arXiv:2403.06801_ , 2024.
  * (34) O. Klass, M. Walker, A. Siebach, T. Stuber, S. Feuerlein, M. Juchems, and M. H. Hoffmann, “Prospectively gated axial ct coronary angiography: comparison of image quality and effective radiation dose between 64-and 256-slice ct,” _European radiology_ , vol. 20, pp. 1124–1131, 2010.
  * (35) S.-K. Chua, H.-F. Hung, J.-J. Cheng, M.-T. Tseng, W.-Y. Law, C.-J. Kuo, C.-Z. Chiu, C.-M. Chang, S.-H. Lee, H.-M. Lo _et al._ , “Diagnostic performance of 64-versus 256-slice computed tomography coronary angiography compared with conventional coronary angiography in patients with suspected coronary artery disease,” _Acta Cardiologica Sinica_ , vol. 29, no. 2, p. 151, 2013.
  * (36) Z. Wang, L. Liu, L. Wang, and L. Zhou, “R2gengpt: Radiology report generation with frozen llms,” _Meta-Radiology_ , vol. 1, no. 3, p. 100033, 2023.
  * (37) X. Zhang, C. Wu, Z. Zhao, W. Lin, Y. Zhang, Y. Wang, and W. Xie, “Pmc-vqa: Visual instruction tuning for medical visual question answering,” _arXiv preprint arXiv:2305.10415_ , 2023.
  * (38) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * (39) M. D. L. I. Vayá, J. M. Saborit, J. A. Montell, A. Pertusa, A. Bustos, M. Cazorla, J. Galant, X. Barber, D. Orozco-Beltrán, F. García-García _et al._ , “Bimcv covid-19+: a large annotated dataset of rx and ct images from covid-19 patients,” _arXiv preprint arXiv:2006.01174_ , 2020.
  * (40) K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , 2002, pp. 311–318.
  * (41) S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , 2005, pp. 65–72.
  * (42) C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in _Text summarization branches out_ , 2004, pp. 74–81.
  * (43) R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 4566–4575.
  * (44) T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * (45) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale _et al._ , “Llama 2: Open foundation and fine-tuned chat models, 2023,” _URL https://arxiv. org/abs/2307.09288_ , 2023.
  * (46) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier _et al._ , “Mistral 7b,” _arXiv preprint arXiv:2310.06825_ , 2023.
  * (47) E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen _et al._ , “Lora: Low-rank adaptation of large language models,” in _International Conference on Learning Representations_ , 2021.

##  Appendix A Appendix / supplemental material

###  A.1 Limitations and Future Work

In this work, we propose an effective and efficient method for generating
radiology reports from high-resolution 3D medical images. Since this is a
generative task, the content produced may sometimes be inaccurate. Clinically,
radiologists do not only rely on radiographs to write reports; they also
consider electronic health records (EHR) and other clinical examinations,
which currently limits the clinical application of our method. In the future,
we aim to integrate additional modalities of data, such as EHR and
electrocardiograms, to enhance the quality of the generated reports.

###  A.2 Broader Impacts

In this work, we introduce an effective and efficient method for automatic
radiology report generation from 3D HR medical images, aiming to alleviate the
burden on radiologists in report writing and expedite clinical decision-making
processes. Additionally, we establish the first benchmark for radiology report
generation from 3D high-resolution medical images, comprising three different
settings. We also propose a new dataset to construct this comprehensive
benchmark. This benchmark and dataset have the potential to significantly
advance the research community in this field.

###  A.3 Details of BIMCV-RG Dataset

In the original BIMCV dataset bimcv , there are 8,069 radiology reports, each
paired with a HR 3D CT scan. These reports are written in Spanish. To ensure
consistency in language for the 3DHRG task with zero-shot domain transfer, all
Spanish reports were automatically translated into English using GPT-4, then
manually corrected with the assistance of medical experts. Since the original
dataset provides reports in their entirety without specific delineation of
‘FINDINGS’ and ‘IMPRESSION’ sections, we translated and used the entire report
for 3DHRG. After translation, the BIMCV-RG dataset was constructed following
these steps: After translation, we build BIMCV-RG dataset followed these
steps:

  1. 1.

Filtering out samples with reports shorter than 10 tokens or longer than 512
tokens.

  2. 2.

Removing samples with duplicated reports.

  3. 3.

Excluding samples that have a report but no associated available CT scans.

  4. 4.

Removing samples with reports where the 3D volumes have dimensions width and
height less than 512, and number of slices less than 256.

Basic statistics of the text in the BIMCV-RG dataset are visualized in Fig 6.
Exemplars from the curated BIMCV-RG dataset are shown in Fig 7.

Figure 6: Left: Analysis of Clinical-Related Word Frequency. Right: Word Cloud
Analysis of Clinical-Related Terms. Figure 7: Examples of reports from the
BIMCV-RG dataset.

###  A.4 Data Preprocessing

For each dataset, The Hounsfield Units (HU) of CT scans were clipped to a
range of $[-1000\mathrm{HU},+200\mathrm{HU}]$, reflecting the usable
diagnostic range of HU values. Volumes were resized to the following
specifications:

  1. 1.

A resolution of $512\times 512$ on the sagittal and coronal axes, with the
transverse axis comprising 256 slices.

  2. 2.

A resolution of $256\times 256$ on the sagittal and coronal axes, with 128
slices on the transverse axis.

The resized volumes were adjusted to maintain uniform slice spacing of
$0.75\mathrm{mm}$ on the sagittal and coronal axes and $1.5\mathrm{mm}$ on the
transverse axis. Voxel values across all scans were normalized to a 0 to 1
range. To preserve the integrity of spatial relationships, which is crucial in
interpreting 3D medical images, augmentation techniques such as flipping or
rotating were not employed. This decision helps avoid potential ambiguities in
spatial orientation, such as given radiological reports that specify locations
like the ‘left lung’ and ‘right lung.’

###  A.5 Extra Ablation Studies

Figure 8:  Results from varying numbers of layers in projectors and different
LLMs. Left: Results from different numbers of layers in projectors applied to
both datasets, with an input resolution of $512\times 512\times 256$ and using
Llama3 as the LLM. Right: Results from using different numbers of LLMs on both
datasets, each with an input resolution of $512\times 512\times 256$.

Affect of Number of Layers of Projectors We evaluate the effect of varying the
number of projectors, $\mathcal{F}$, from 1 to 3, across two datasets as shown
in Fig 8 (left). The results indicate that performance saturates with two-
layer projectors and slightly decreases with three layers. Consequently, we
select two layers for our model to achieve optimal performance. However, the
relatively stable performance across different numbers of layers demonstrates
that our method is not sensitive to this factor.

Effect of Different LLMs We also investigate the impact of different Large
Language Models (LLMs) on performance, as depicted in Fig 8 (right), including
Llama2-7B-chat-hf touvron2023llama , Llama3-8B-Instruct llama3modelcard , and
Mistral-7B-Instruct-v0.2 jiang2023mistral . We achieve optimal performance
using Llama3-8B-Instruct, which is currently the most powerful LLM in the
7B-8B scale. This suggests that a stronger foundational LLM leads to better
performance in report generation tasks due to enhanced language understanding
capabilities. Although Llama3 reaches the highest performance, our method
shows minimal fluctuation when LLMs change, indicating robustness across
different LLMs.

Table 4: Results of different LR input resolutions are presented. For both LR
input resolutions, the HR input resolution is fixed at $512\times 512\times
256$. The highest result and the lowest number of visual tokens processed by
LLMs are indicated in bold.

LR Input | Number of | CT-RATE | BIMCV-RG
---|---|---|---
Resolution | Visual Tokens | Averaged Metric | Averaged Metric
$128\times 128\times 64$ | 64 | 46.05 | 40.96
$256\times 256\times 128$ | 512 | 46.21 | 40.85

Effect of LR Input Resolution We explore the effect of two different LR input
resolutions, $128\times 128\times 64$ and $256\times 256\times 128$, to
generate LR queries and HILT. For both LR input resolutions, we use the HR
input resolution $512\times 512\times 256$. As shown in Table 4, a higher LR
input resolution leads to an eightfold increase in the number of visual tokens
processed by LLMs, indicating a significant increase in computational cost.
However, the performance on both datasets does not show the same trend as the
computational cost increase. Hence, this suggests that increasing the LR input
resolution is not necessary, as we mine HR information from HR views and
integrate it into LR queries. Increasing the LR input resolution would not
introduce new information but would mainly increase the computational cost.

###  A.6 Hyper-parameters of Training

Table 5: Hyper-parameters for our methods.

Hyperparameters |
---|---
Mixed precision | bf16
Instruction tuning epochs | 100
LoRA alpha | 64
LoRA rank | 128
LoRA dropout | 0.1
Total batch size | 4
Gradient accumulation | 2
Maximum sequence length | 512
Learning rate | 1e-4
Learning rate Optimizer | AdamW
Schedule | linear
Warm-up ratio | 0.03
Weight decay | 0.0

In this study, we implement the Low-Rank Adaptation (LoRA) hu2021lora for the
3DHRG task. We configure the LoRA parameters with an alpha of 64 and a rank of
128, and apply a dropout rate of 0.1. Instruction tuning is performed over 100
epochs. All experiments are conducted on a single A100-80G GPU.

For computational efficiency, we employ mixed precision at bf16, as detailed
in Tab 5. The training process includes a total batch size of 4 with a
gradient accumulation factor of 2, and we restrict the maximum sequence length
to 512 tokens.

Regarding learning rates, we use $1\times 10^{-4}$ for our models, optimized
via the AdamW algorithm. The learning rate schedule is linear with a warm-up
ratio of 0.03, and the weight decay is set to 0.0.

We utilize the same configuration for all existing methods if they do not
specify these hyperparameters in their official codebase.