# Two and One-dimensional Honeycomb Structure of Boron Nitride

M. Topsakal  UNAM-Institute of Materials Science and Nanotechnology, Bilkent
University, Ankara 06800, Turkey  E. Aktürk  UNAM-Institute of Materials
Science and Nanotechnology, Bilkent University, Ankara 06800, Turkey  S.
Ciraci<EMAIL_ADDRESS>UNAM-Institute of Materials Science and
Nanotechnology, Bilkent University, Ankara 06800, Turkey  Department of
Physics, Bilkent University, Ankara 06800, Turkey

###### Abstract

This paper presents a systematic study of two and one dimensional honeycomb
structure of boron nitride (BN) using first-principles plane wave method. Two-
dimensional (2D) graphene like BN is a wide band gap semiconductor with ionic
bonding. Phonon dispersion curves demonstrate the stability of 2D BN flakes.
Quasi 1D armchair BN nanoribbon are nonmagnetic semiconductors with edge
states. Upon passivation of B and N with hydrogen atoms these edge states
disappear and band gap increases. Bare zigzag BN nanoribbons are metallic, but
become a ferromagnetic semiconductor when their both edges are passivated with
hydrogen. However, their magnetic ground state, electronic band structure and
band gap are found to be strongly dependent on whether B- or N-edge of the
ribbon is saturated with hydrogen. Vacancy defects in armchair and zigzag
nanoribbons affects also magnetic state and electronic structure. In order to
reveal dimensionality effects these properties are contrasted with those of
various 3D BN crystals and 1D BN atomic chain.

###### pacs:

73.22.-f, 75.75.+a, 63.22.-m

##  I introduction

Synthesis of a single atomic plane of graphite, i.e. Graphene with covalently
bonded honeycomb lattice has been a breakthrough for several reasons novo ;
zhang ; berger . Firstly, electrons behaving as if massless Dirac Fermions
have made the observation of several relativistic effects possible. Secondly,
stable graphene has disproved previous theories, which were concluded that
two-dimensional structures cannot be stable. Graphene displaying exceptional
properties, such as high mobility even at room temperature, ambipolar effect,
Klein tunelling, anomalous quantum hall effect etc. seems to offer novel
applications in various fields graphene_applications1 . Not only 2D graphene,
but also its quasi 1D forms, such as armchair and zigzag nanoribbons have
shown novel electronic and magnetic properties graphene_applications2 ;
graphene_applications3 ; graphene_applications4 , which can lead to important
applications in nanotechnology. As a result, 2D honeycomb structures derived
from Group IV elements and Group III-V and II-VI compounds are currently
generating significant interest owing to their unique properties.

Boron-Nitride (BN) in ionic honeycomb lattice which is the Group III-V
analogue of graphene have also been produced having desired insulator
characteristics bn-synthesis . Nanosheets bn-nanosheets1 ; bn-nanosheets2 ,
nanocones bn-nanocones , nanotubes bn-nanotubes , nanohorns bn-nanohorns ,
nanorods bn-nanorods  and nanowires bn-nanowires  of BN have already been
synthesized and these systems might hold promise for novel technological
applications. Among all these different structures, BN nanoribbons, where the
charge carriers are confined in two dimension and free to move in third
direction, are particularly important due to their well defined geometry and
possible ease of manipulation.

BN nanoribbons posses different electronic and magnetic properties depending
on their size and edge termination. Recently, the variation of band gaps of BN
nanoribbons with their widths and Stark effect due to applied electric field
have been studied Guo ; louie-nanoletter . Recently the magnetic properties of
zigzag BN nanoribbons have been investigated barone . Half-metallic properties
have been revealed from these studies which might be important for spintronic
applications. Production of graphene nanoribbons as small as 10 nm in width
has been achieved dai1 ; dai2  and similar techniques are expected to be
developed for BN nanoribbons.

A thorough understanding of 2D BN honeycomb structure and their various
nanoribbons is important for further study of this graphene like compounds. BN
by itself provides with very interesting chemical and physical properties,
which may lead to important applications. In this paper, we present a detailed
ab-initio study of electronic, magnetic and elastic properties of 2D (graphene
like) BN and bare and hydrogen passivated, quasi 1D BN nanoribbons (BNNRs). We
also investigated the effect of the vacancy defects on these properties. To
reveal the dimensionality effects we include also a short discussion regarding
3D BN bulk crystals and 1D BN atomic chains. We found that 2D BN is a
nonmagnetic, wide band gap semiconductor. The ionic bonding due to significant
amount of charge from B to N atom opens a gap and hence dominates electronic
structure. Calculated phonon dispersion curves provide a clear evidence that
2D BN flakes is stable. The armchair and zigzag nanoribbons of BN display even
more interesting electronic and magnetic properties. Bare and hydrogen
passivated armchair BN nanoribbons (A-BNNR) are nonmagnetic wide band gap
semiconductor. The value of band gap of A-BNNR having width $w>10$ Å is
practically independent from the width of nanoribbons. While the bare zigzag
BN nanoribbons (Z-BNNR) are ferromagnetic metal, they become nonmagnetic
semiconductor upon the passivation of both edges. We found that 2D BN and its
nanoribbons have properties, which are complementary to graphene.

##  II Model and Methodology

We have performed first-principles plane wave calculations within density
functional theory (DFT) using PAW potentials paw . The exchange correlation
potential has been approximated by generalized gradient approximation (GGA)
using PW91 pw91  functional both for spin-polarized and spin-unpolarized
cases. All structures have been treated within supercell geometry using the
periodic boundary conditions. A plane-wave basis set with kinetic energy
cutoff of 500 eV has been used. In the self-consistent potential and total
energy calculations the Brillouin zone (BZ) is sampled by special k-points.
The numbers of these k-points are (15x15x15) for bulk BN, (25x25x1) for 2D BN
and (25x1x1) for nanoribbons, respectively, and are scaled according to the
size of superlattices. All atomic positions and lattice constants are
optimized by using the conjugate gradient method, where the total energy and
atomic forces are minimized. The convergence for energy is chosen as 10-5 eV
between two steps, and the maximum Hellmann-Feynman forces acting on each atom
is less than 0.02 eV /Å upon ionic relaxation. A large spacing ($\sim$ 10 Å)
between monolayers has been taken to prevent interactions between them. The
pseudopotentials having 3 and 5 valence electrons for the B (B: $2s^{2}$
$2p^{1}$) and N ions (N: $2s^{2}$ $2p^{3}$) were used. Numerical calculations
have been performed by using VASP package vasp1 ; vasp2 . The phonon
dispersion curves are calculated within density functional perturbation theory
(DFPT) using plane wave methods as implemented in PWSCF software pwscf .

##  III 3D BN Crystals and 1D atomic chain

In this section, we present our theoretical calculations on 3D bulk BN
crystals and truly 1D BN atomic chain. Earlier these 3D bulk crystals
BNcrystal1 ; BNcrystal2 ; BNcrystal3 ; BNcrystal4 ; BNcrystal5  and 1D atomic
chains BNchain  have been studied theoretically by using different methods.
Our purpose in including these crystals of BN in different dimensionalities is
to contrast their properties with those of 2D and quasi 1D honeycomb
structures of BN and also reveal dimensionality effects.

###  III.1 3D Bulk BN crystals

Three dimensional bulk crystals include hexagonal-layered BN (h-BN), wurtzite
BN (wz-BN) and zincblende BN (zb-BN) structures. Their atomic configurations
and primitive unit cells are described in Fig. 1. By using the expression,

$E_{C}={E[BN]}-E[B]-E[N]$ (1)

where E[BN] is the total energy per B-N pair of the optimized structure of BN
crystal; E[B] and E[N] are the total energies of free B and N atoms; we
calculated the equilibrium cohesive energies of h-BN, wz-BN and zb-BN crystals
as -17.65, -17.45 and -17.49 eV per B-N pair, respectively. Accordingly, h-BN,
which is the analogue of graphite, is the most energetic bulk structure. On
the other hand, the cubic BN structure is known to be the second hardest
material of all.

The lattice constants of the optimized structures in equilibrium are $a=2.511$
Å, $c/a=2.66$ and the distance between the nearest B and N atoms is $d=1.450$
Åfor h-BN layered crystal. For wz-BN, optimized values of $a$, $c/a$ and $d$
are calculated to be 2.542 Å, 1.64, and 1.561 Å, respectively. The zincblende
structure has lattice constant $a=2.561$ Å and $d=1.568$ Å. All our results
related with the structural parameters are in good agreement with the
experimental and theoretical values BNcrystal1 ; BNcrystal2 ; BNcrystal3 ;
BNcrystal4 ; bulk-references  within the average error of $\sim 1\%$.

The calculated electronic band structure, total and partial (or orbital
projected) density of states (DOS) of 3D crystals are presented in Fig. 1.
These h-BN, wz-BN and zb-BN crystals are indirect band gap semiconductors with
calculated band gaps being $E_{G}$=4.47, 5.72, and 4.50 eV, respectively. The
calculated values of $E_{G}$ differ from the earlier ones depending on the
method used DFTgap . For h-BN having 2D BN atomic layers in the (x,y)-plane.
The band structure is composed from the band structures of these individual
atomic layers with hexagonal symmetry, which are slightly split due to weak
coupling between them. Highest valence band has N-$p_{z}$ character; the
states of the lowest conduction band is formed from B-$p_{z}$ orbitals (the z
direction corresponds to “c” in Fig. 1 ). Overall features of the total
density of states (TDOS) are similar for three 3D crystal structures. Valence
band consists of two parts separated by a wide intra band gap. The lower part
at $\sim$ -20 eV is projected mainly to N-$s$ and partly to N-$p$ and B-$s$
orbitals. The upper part is due to mainly N-$p$ and partly B-$p$ orbitals and
has similarities in both zb-BN and wz-BN crystals. As for the lower part of
the conduction band it is derived mainly from B-$p$ orbitals. The differences
of three 3D crystals are pronounced in the lower part of the conduction band.

Figure 1: (Color online) Optimized atomic structure, energy bands, total
(TDOS) and orbital projected density of states (PDOS) of various 3D crystals
of BN. (a) Hexagonal (h-BN) whose B(N) atoms are on top of the N(B) atoms in
the consecutive layer; (b) wurtzite (wz-BN); and (c) zincblende (zb-BN)
crystals. Dark-green and light-gray balls represent B and N atoms,
respectively. The band gaps between conduction and valence bands are
highlighted. The orbital character of states are indicated for the conduction
and valence band edges. The zero of energy is set to the Fermi energy EF. All
structures are fully optimized.

We calculate the amount of charge on constituent B and N atoms in 3D crystals
by performing the Löwdin lowdin  analysis in terms of the projection of plane-
waves into atomic orbitals. By subtracting the valencies of free B and N atoms
from the calculated charge values on the same atoms in 3D crystals we obtain
the charge transfer, $\Delta Q$. The calculated values of $\Delta Q$ for h-BN,
wz-BN and zb-BN are 0.416, 0.342, 0.334 electrons, respectively. The fact that
$\Delta Q$ of zb-BN and wz-BN have almost equal values, but $\Delta Q$ of h-BN
crystal is significantly larger related to the shorter B-N bond length in h-BN
crystal.

###  III.2 1D BN Atomic chain

BN forms stable segments of linear atomic chain BNchain  like carbon tongay .
This situation is in contrast to second and third row elements (such as Si and
Ge) and III-V compounds and metals (such as Al, Au etc) which can form stable
zigzag chain structures instead of linear chain structures. Our results on
optimized chain structure yield the cohesive energy $E_{C}$=16.04 eV per B-N
pair, the B-N distance $d=$1.307 Å, the indirect band gap E${}_{G}=$3.99 eV
and charge transfer from B to N, $\Delta Q=$0.511 electrons. Hence the double
bond between B and N is ionic.

##  IV 2D Honeycomb Structure of BN

Having discussed the overall structural and elastic properties of 3D and 1D
BN, we now consider 2D BN with hexagonal symmetry. The atomic structure of 2D
BN is similar to the honeycomb structure of graphene, except that the
constituent atoms of the former are from III and V columns of the Periodic
Table. Normally, the bond between nearest B and N atoms is formed from the
bonding combination of B-$sp^{2}$ and N-$sp^{2}$ orbitals. However, owing to
the electronegativity difference between B and N atoms electrons are
transferred from B to N. As a result, in contrast to purely covalent bond in
graphene the bonding between B and N gains an ionic character. The charge
transfer from B to N dominates several properties of 2D BN including the
opening of the band gap. In this respect the BN honeycomb structure is
complementary to graphene.

###  IV.1 Charge density analysis and electronic structure

The atomic structure, atomic charge, charge transfer from B to N and the
electronic structure of 2D BN are presented in Fig. 2. Contour plots of total
charge indicates high density around N atoms. The difference charge density is
calculated by subtracting charge densities of free B and N atoms from the
charge density of 2D BN, i.e. $\Delta\rho=\rho_{BN}-\rho_{B}-\rho_{N}$. High
density contour plots around N atoms protruding towards the B-N bonds indicate
charge transfer from B to N atoms. This way the B-N bonds achieve an ionic
character. The amount of transfer of charge is calculated by Löwdin analysis
to be $\Delta Q$=0.429 electrons. Interestingly, $\Delta Q$ is slightly larger
than that calculated for h-BN, but significant larger than those calculated
for wz-BN and zb-BN crystals.

2D BN is a semiconductor. Calculated electronic energy bands are similar to
those calculated for h-BN crystal. The $\pi$\- and $\pi^{*}$\- bands of
graphene which cross at the K- and K∗-points of the BZ open a gap in 2D BN as
a bonding and antibonding combination of N-$p_{z}$ and B-$p_{z}$ orbitals. The
contribution of N-$p_{z}$ is pronounced for the filled band at the edge of
valence band. The calculated band gap is indirect and $E_{G}=$4.64 eV. TDOS
and partial density of states show also similarity to those of h-BN layered
crystal presented in Fig. 1.

Figure 2: (Color online) (a) Primitive unit cell of the honeycomb structure of
2D BN together with Bravais lattice vectors. Calculated total charge density
$\rho_{BN}$ and difference charge density $\Delta\rho$, are also shown in the
same panel. (b) Calculated electronic structure of 2D BN honeycomb crystal
together with total, TDOS and partial density of states, PDOS on B and N
atoms. The orbital character of the states are also indicated.

###  IV.2 Phonon spectrum

Even if the structure optimization resulting in the honeycomb structure in
Fig. 2 can be taken as an indication for the stability of 2D BN, calculation
of phonon dispersion curves through the diagonalization of dynamical matrix
provides a more stringent test for stability. One of acoustical branches for
$\Gamma$ to $K$ curves taking negative value even at a small region of BZ
indicates the instability of the structure. There have been a number of
experimental Rokuta  and theoretical studies of phonon spectrum of 2D Wirtz
and 3D honeycomb BN Kern ; Yu ; Serrano ; Solozhenkan ; Miyamoto . Here, the
phonon dispersion curves of h-BN, 2D BN and 1D BN chain and density of states
together with the infrared (IR) and Raman (R) active modes of 2D BN and h-BN
at $\Gamma$-point have been calculated by using density functional
perturbation theory (DFPT) as implemented in PWSCF software pwscf . For the
DFPT phonon calculation of bulk h-BN, we used a four atom primitive cell,
which yield 12 phonon branches at the center of BZ in Fig. 3 (a). The symmetry
point group is calculated as D6h (space group P6/mmm). The irreducible
representations at $\Gamma$ is 2 E2g\+ 2 B2g+2 A2u+2 E1u. While the modes E1u
and E2g are doubly degenerate, B2g and A2u are non degenerate. The modes E1u
and A2u are IR active, the E2g is Raman active. B2g is an inactive mode. Our
results are in agreement with previously calculated and experimental data, but
differ slightly from those of Serrano et al.Serrano . While present GGA
calculations predict B2g mode as an inactive mode, LDA calculations by Serrano
et al. found B1g as an inactive mode. Most of the phonon bands of h-BN are
degenerate. This indicates that the coupling between BN layers in h-BN is
weak. However, it is well known that the BN is polar material with long range
dipole-dipole interaction. This gives rise to the splitting between
longitudinal optical (LO) and transverse optical (TO) mode at $\Gamma$ point.
The lowest transverse acoustical mode has parabolic dispersion as k
$\rightarrow$ 0 owing to rapidly decaying interatomic forces for transversal
displacements decay . Another feature is the overlap of the lowest transversal
optical mode with the acoustical modes.

In Fig. 3 (b) we show the phonon dispersion curve of BN atomic chain. Two TA
modes have low frequency and get very small but negative values near the zone
center. This indicates structural instabilty as $\lambda\rightarrow\infty$.
However, the linear segments of BN atomic chain can be stable. Similar to
h-BN, the doubly degenerate TO branch overlaps with the LA branch.

For 2D BN honeycomb structure, the unit cell consists of two atoms.
Accordingly, there are three acoustical and three optical branches in Fig. 3
(c). The symmetry point group is D3h (space group (P-62m)). Optical phonon
modes at the $\Gamma$-point is given by A${}^{{}^{\prime\prime}}_{2}$+2
E${}^{{}^{\prime}}$. The mode A${}^{{}^{\prime\prime}}_{2}$ is IR active and
the E${}^{{}^{\prime}}$ mode is both IR and Raman active. The similarity
between calculated phonon dispersion curves of h-BN and 2D-BN is remarkable.

We also calculate the phonon dispersion curves of 2D BN honeycomb structure by
using PAW potentials paw  as implemented in VASP vasp1  for further checking
of the results of our phonon calculation. Force constants are determined from
the $(8\times 8\times 1)$ supercells. The phonon modes were calculated by
using the direct method as implemented in the PHON alfe  software. The
calculated phonon frequencies are almost identical with those calculated by
DFPT method. In Fig. 3 (d), we present the phonon density of states calculated
for 2D BN honeycomb structure. Note that both calculations yield that TA (or
ZA) mode displaying parabolic dispersion gets negative frequencies as
$k\rightarrow 0$. similar to BN atomic chains, this indicates structural
instability as $\lambda\rightarrow\infty$. Accordingly, finite size of 2D BN
flakes are expected to have stable structure.

Figure 3: (Color online) Calculated phonon frequencies versus k-vectors. (a)
h-BN crystal. (b) 1D BN atomic chain. (c) 2D BN honeycomb structure. Phonon
modes calculated by force constant direct method are shown by the blue-dashed
curve. (d) Density of phonon frequencies (DOS) for the 2D BN honeycomb
structure.

##  V 1D BN Nanoribbons

Similar to graphene graphene-nanoribbons , two unique orientation in 2D BN
yield nanoribbons with uniform edges: These are armchair (A-BNNR) and zigzag
(Z-BNNR) nanoribbons. The profile of the atomic configuration at both edges of
the nanoribbon determines their electronic and magnetic properties. The
properties can be modified by the passivation of dangling bond of edge atoms
by hydrogen. Because of their interesting electronic and spintronic
properties, BN nanoribbons are attractive nanostructures for various device
applications. Electronic properties of BN nanoribbons have been investigated
in recent papers Guo ; louie-nanoletter ; barone . Present study is
complementary to previous studies.

###  V.1 Electronic structure

Here we present the results of our study on the electronic and magnetic
properties of bare and hydrogen passivated A-BNNR and Z-BNNRs. Bare and
hydrogen passivation A-BNNR are wide band gap semiconductors. Similarly,
hydrogen passivated Z-BNNRs are also semiconductor. The band gaps of these BN
nanoribbons depend on the width of the nanoribbons $w$ or the numbers of BN
pairs, $n$ in the primitive unit cell. The variation of the band gap EG as a
function of $n$ is given in Fig. 4. Normally, the properties of nanoribbons
approaches to those of 2D honeycomb structure as the width
$n\rightarrow\infty$. However, due to the localized edge states the band gap
of Z-BNNR approaches to a gap smaller than that of 2D BN honeycomb structure
louie-nanoletter . For narrow ($n<8$) bare and hydrogen passivated A-BNNRs the
band gaps vary with $n$, but they are practically unaltered for $n>8$. For
$n>8$ the band gap of bare A-BNNR is 0.4 eV smaller than that of hydrogen
passivated A-BNNR. The band gap of hydrogen passivated Z-BNNR is 4.5 eV for
$n=3$, but decrease to 3.8 eV for $n=16$. However, its variation with $n$ is
not monotonic for $5<n<13$, it rather display family dependent oscillatory
variation with changes as large as 0.4 eV between two consecutive values of
$n$. On the other hand, bare Z-BNNRs are found to be metallic.

Figure 4: (Color online). Energy band gap versus the width of the nanoribbons
given in terms of the number of B-N atom pairs in the primitive unit cell,
$n$. Bare armchair nanoribbons A-BNNR, hydrogen passivated A-BNNR, and
hydrogen passivated zigzag nanoribbons Z-BNNR. Dotted line indicates the bulk
band gap of 2D-BN.  Figure 5: (Color online) (a) Energy band structure of bare
armchair nanoribbon A-BNNR having $n=12$ B-N pairs in the primitive unit cell.
At the right hand side of bands, the schematic description of atomic structure
with primitive unit cell delineated by dotted lines and isosurface charge
distribution of specific states are shown. (b) Same as (a) but the dangling
bonds at both edges are passivated by hydrogen atoms.

The atomic and electronic structure of bare and hydrogen passivated A-BNNR are
described in Fig. 5 for $n=12$. The atoms at the edges of the bare A-BNNR are
reconstructed; while one edge atom, B is lowering, adjacent edge atom, N is
raised. Two bands of edge states occur below the conduction band edge. These
bands are normally degenerate for large $n$, but split around the center of BZ
due to their coupling. The bands of edge states occur $\sim$-1 eV below the
top of the valance band edge. Normal states, on the other hand, have charge
distributed uniformly in the ribbon. Because of the edge states the band gap
is indirect and is $\sim$4.22 eV wide. Upon passivation of the dangling bonds
of B and N atoms situated at the edges with hydrogen atoms, these edge state
bands are discarded from the band gap and reconstruction of edge atoms
disappear. At the end, the band gap of H-passivated A-BNNR becomes direct and
increases by $\sim$0.3 eV.

Figure 6: (Color online) Top panels: Atomic structures of zigzag nanoribbons
(Z-BNNR). The primitive unit cell has $n=6$ B-N pairs delineated by dotted
lines. The unit cell is doubled due to antiferromagnetic interaction between
adjacent N atoms. Middle panels: Isosurface plots of difference charge density
between up spin and down spin states,
$\Delta\rho=\rho(\uparrow)-\rho(\downarrow)$. Bottom panels: Energy band
structure with dotted (blue) and solid (red) lines showing spin up and spin
down states, respectively. (a) Bare Z-BNNR; (b) B-side free, but N-side is
passivated by hydrogen atoms; (c) N-side free, but B-side is saturated by
hydrogen atoms; (d) Both sides are saturated by hydrogen atoms. The bands in
(a), (b), and (c) are calculated using double cell.

The electronic and magnetic states of Z-BNNR depend on whether their edges are
passivated with hydrogen atoms. While a bare Z-BNNR is magnetic and metallic,
it becomes nonmagnetic and a wide band gap semiconductor upon the passivation
of B and N atoms at both edges. Moreover, its electronic and magnetic
properties depend on whether only B- or N-side is passivated with hydrogen
atoms. Accordingly, Z-BNNRs provide us for several alternatives for different
electronic and magnetic properties barone . However, different magnetic states
corresponding to different edge configuration, namely bare or hydrogen
passivated, are very sensitive to the parameters of calculation. In Fig. 6 we
present the calculated electronic structures of a Z-BNNR with $n$=6 B-N pairs
in a primitive unit cell for four different cases. These are both side free,
only N-side is passivated with hydrogen, only B-side is passivated with
hydrogen and both edges are passivated with hydrogen.

Bare Z-BNNR having both edges are free display different magnetic states
(magnetic order), which are close in energy. Moreover, the ordering of these
magnetic states with respect to their energy is sensitive to the criterion of
energy convergence. To ensure the antiferromagnetic (AFM) order at edges, we
considered double cells. The possible magnetic states are spin-up, spin-down
for adjacent B atoms at one side and spin-up, spin-up for the adjacent N atoms
at the other side; namely $\uparrow\downarrow$ / $\uparrow\uparrow$ spin
configuration. Other possible spin configurations are $\uparrow\uparrow$ /
$\downarrow\downarrow$; $\uparrow\uparrow$ / $\uparrow\uparrow$ ;
$\uparrow\uparrow$ / $\uparrow\downarrow$; $\uparrow\downarrow$ /
$\uparrow\uparrow$. We found that the spin configuration, $\uparrow\downarrow$
/ $\uparrow\uparrow$ for Z-BNNR having 12 B-N pairs in double unit cell
corresponds to the ground state. The other excited configurations,
$\uparrow\uparrow$ / $\downarrow\downarrow$; $\uparrow\uparrow$ /
$\uparrow\uparrow$ ; $\uparrow\uparrow$ / $\uparrow\downarrow$;
$\uparrow\downarrow$ / $\uparrow\downarrow$, have 6,7,35,131 meV higher
energies than ground state. The ordering of these configuration is slightly
different from that reported earlier barone . Nevertheless, the difference
between the earlier and present ground state energies are within the accuracy
limits of DFT calculations. The ground state spin configuration
$\uparrow\downarrow$ / $\uparrow\uparrow$ of the bare Z-BNNR having both edges
free is found to be ferrimagnetic metal with $\mu=1.77$ $\mu_{B}$ per double
cell. Whereas the excited magnetic state with configuration $\uparrow\uparrow$
/ $\downarrow\downarrow$ is half metallic.

In Fig. 6 (b), Z-BNNR with N-edge passivated with hydrogen atoms is an AFM
semiconductor. The AFM edge state is localized at the B-side. When only the
B-side is passivated with hydrogen atoms the magnetic edge state is, this
time, localized at the N-side of the ribbon. As seen in Fig. 6 (c) the ground
state of Z-BNNR is ferromagnetic with $\mu$ =2 $\mu_{B}$ per double cell. Our
calculations suggest that the nearest neighbor N-N interaction is
ferromagnetic, the B-B interaction is antiferromagnetic. Finally, the Z-BNNR
becomes non magnetic, when the atoms at both edges are passivated with
hydrogen atoms. Earlier, Hwan and Louie louie-nanoletter  studied hydrogen
passivated A-BNNRs and Z-BNNRs with widths up to 10 nm. Our results for
hydrogen passivated nanoribbons are in good agreement with their results,
except that our results for zigzag ribbons obtained using GGA as well as LDA
exhibit family dependent oscillations for $5<n<13$.

###  V.2 Elastic properties

The elastic properties of BNNRs are examined through the variation of the
total energy $E_{T}$ with respect to the applied uniaxial strain
$\epsilon=\Delta c/c$, $c$ being the lattice constant along the nanoribbon
axis. Owing to ambiguities in defining the cross section of the ribbon one
cannot determine the Young’s modulus rigorously. Instead we calculate
$\kappa=\partial^{2}E_{T}/\partial c^{2}$ from the variation of $E_{T}$ to
specify the elastic properties of quasi 1D nanoribbons. In Fig. 7 (a) we show
the variation of the total energy $E_{T}$ versus $\epsilon$. In order to lift
the constraints imposed by periodic boundary conditions, calculations are
performed for a supercell comprising five primitive unit cells having lattice
constant $c_{s}=5c$. For $\epsilon<0.10$, the variation of $E_{T}(\epsilon)$
is parabolic, and hence $\kappa$ is independent of $\epsilon$. For
$\epsilon>10$ $E_{T}(\epsilon)$ curve deviates from parabola and becomes
anharmonic. For higher values of strain in the plastic region, the ribbon
undergoes structural transformation. For example, such a transformation
occurred at $\epsilon=0.24$ with a sudden change in $E_{T}(\epsilon)$ curve.
The corresponding structure is illustrated as inset. The lattice constant
$c_{s}$ increased from the initial value 21.5 Å to 27.4 Åcorresponding to
$\epsilon=0.27$.

Figure 7: (Color online) (a) Variation of total energy of hydrogen saturated
A-BNNR with strain , $\epsilon$ is shown by dashed curve with large black dots
indicating the calculated data points ($c_{s}=5c$ and $n$ = 9). Harmonic,
anharmonic and plastic regions are distinguished. The harmonic part is fitted
to a parabola presented by red-solid curve. Atomic structure shown by filled,
empty and very small empty circles represent B, N, and H atoms. Supercell
comprising five primitive unit cells are shown in the harmonic and plastic
regions. (b) Variation of $\kappa=\partial^{2}E_{T}/\partial c^{2}$ versus
ribbon width $n$ calculated for bare and hydrogen passivated A-BNNR.

In Fig. 7 (b) $\kappa$ versus the width of the ribbon in terms of the number
of B-N pair in the primitive unit cell $n$ is plotted for bare and hydrogen
passivated A-BNNR. $\kappa(n)$ shows an approximately linear variation
indicating that the force constant is directly proportional to the width of
the ribbon. One also sees that the strength of the ribbon increases upon
passivation with hydrogen.

The behavior of bare and hydrogen passivated Z-BNNR under uniaxial tensile
stress is similar to that of A-BNNR. In Fig. 8 three regions, namely elastic
harmonic, elastic-anharmonic and plastic regions are seen. The sudden change
in the $E_{T}(\epsilon)$ curve at $\epsilon\sim 0.23$ indicates a structural
phase transformation, where the lattice constant $c_{s}$ elongates from the
initial $\epsilon=0$ value of 19.8 Å to 25.7 Å corresponding to
$\epsilon=0.3$. The structure of hydrogen passivated Z-BNNR before and after
the structural transformation are shown as inset. Variation of $\kappa$ versus
the ribbon width $n$ is calculated for bare and hydrogen passivated Z-BNNR
show an overall linear behavior as presented in Fig. 7 (b).

Figure 8: (Color online) Variation of total energy of hydrogen saturated
Z-BNNR shown by dashed curve with large black dots indicating the calculated
data points. Harmonic, anharmonic and plastic regions are distinguished. The
harmonic part is fitted to a parabola presented by a red solid curve. Atomic
structure of the ribbon in a supercell comprising eight unit cells ($c_{s}=8c$
and $n$ = 6) are shown before and after structural transformation as inset.

###  V.3 Vacancy and antisite defects

It is known that the vacancy defect in 2D graphene esquinazi ; Iijima ; yazyev
; guinea ; brey2  and graphene nanoribbons brey1 ; delik  give rise to crucial
changes in the electronic and magnetic structure. According to Lieb’s theorem
lieb , the net magnetic moment per cell is determined with the difference in
the number of atoms belonging to different sublattices, i.e.
$\mu=(N_{B}-N_{N})\mu_{B}$. While DFT calculations on vacancies in 2D graphene
and armchair graphene nanoribbons confirmed Lieb’s theorem, results are
diversified for vacancies in zigzag graphene nanoribbons brey1 ; delik .
Therefore, the effect of vacancy defects on the properties of BNNRs is of
interest.

Earlier activation energies and reaction paths for diffusion and nucleation
mono and divacancy in h-BN layers have been investigated by using density
functional tight-binding method zobelli . The formation energies were
calculated to be 11.22 eV and 8.91 eV, for a B- and N-vacancy, respectively.
The possible magnetism induced by nonmagnetic impurities and vacancy defects
in a BN sheet have been investigated from the first-principles. The magnetic
moment associated by nonmagnetic atoms substituting B or N has been calculated
to be $1\mu_{B}$ liu . Based on first-principles calculations, the magnetic
moment of a N-vacancy in a 2D BN sheet has been predicted to be 1 $\mu_{B}$.
In the case of a B-vacancy, three neighboring N atoms are displaced further
apart from each other and the net magnetic moment is predicted to be 3
$\mu_{B}$ si . Another calculation of defects in a BN monolayer found that
three dangling bonds associated with a B-vacancy lead to total spin S=3/2, i.e
3 $\mu_{B}$ azeveda .

Figure 9: (Color online) Relaxed atomic structures and corresponding energy
bands of hydrogen passivated A-BNNR with $n=12$ having a point defect located
periodically in every four primitive cell. Blue filled, empty and small
circles represent B, N, and H atoms, respectively. Blue-dark and yellow-light
isosurface plots are for spin-up and spin-down states. (a) Single B-vacancy;
(b) single N vacancy; (c) B-N divacancy; (d) Antisite defect.

The effects of vacancies of BN nanoribbons have not been treated yet. Here we
investigated the effect of B-, N-, B+N-divacancy and B+N-anti site on the
electronic and magnetic properties of A- and Z-BNNR. Within periodic boundary
conditions, a vacancy defect in an A-BNNR of width $n=12$ is repeated in every
5th primitive unit cell to yield minute defect-defect coupling. As shown in
Fig. 9 (a) A-BNNR with B-vacancy becomes ferromagnetic with a net magnetic
moment of $\mu=1$ $\mu_{B}$ per unit cell. Similarly, a N vacancy gives rise
to a net magnetic moment of $\mu=1$ $\mu_{B}$ per unit cell. A-BNNR having
either periodic B+N-divacancy or anti site defect for every five unit cell
remain nonmagnetic. The calculated values of magnetic moments are in
compliance with Lieb’s theorem. We found that the structural relaxation is
crucial to obtain correct values of magnetic moments. In particular, initially
we calculated $\mu=3$ $\mu_{B}$ for relaxed structure of the B-vacancy.
However, the neighboring N atoms distorted slightly from their equilibrium,
the structure is relaxed further and had lowered the total energy. As a
result, the magnetic moment was calculated as $\mu=1$ $\mu_{B}$. The energy
band structures in Fig. 9 (a)-(d) are calculated for periodic vacancy defects
repeating in every four primitive cell. The Fermi levels are assigned
according to the occupancy of vacancy states. We note that the empty state
associated with the B-vacancy in Fig. 9 is hole like. The states associated
with the N-vacancy occur near the edge of the conduction band are donor like.

Figure 10: (Color online) Relaxed atomic structures of hydrogen passivated
Z-BNNR with $n=6$ having a vacancy defect located periodically in every eight
primitive cell ($c_{s}\approx 8c$ and $n$ = 6). Filled, empty and small
circles represent B, N, and H atoms, respectively. Blue and pink isosurface
plots are spin up and spin down states, respectively. (a) Single B-vacancy;
(b) single N vacancy; (c) B+N divacancy; (d) anti site defect.

The situation with Z-BNNR is similar to that in A-BNNR discussed above, since
hydrogen passivated Z-BNNR is nonmagnetic as A-BNNR. A periodic B- or
N-vacancy repeated in every eight unit cell of hydrogen saturated Z-BNNR with
$n$=6 has a net magnetic moment of $\mu=1$ $\mu_{B}$ per supercell. Whereas
Z-BNNR passivated with hydrogen atoms at both edges and having either periodic
B+N-divacancy or anti site defect repeating in every eight unit cell is
nonmagnetic. The type of the periodic vacancy defect modifies the band gap of
Z-BNNR from 4.2 eV to 2.21 eV for divacancy, but to 2.8 eV for anti site. The
calculated magnetic moment of hydrogen passivated Z-BNNR are in agreement with
Lieb’s theorem. We note that in zigzag graphene nanoribbons magnetic edge
states survive even after hydrogen passivation, and interact with the magnetic
moments of vacancies delik . This interaction causes deviation from the
prediction of Lieb’s theorem.

##  VI Discussion and Conclusions

In various allotropic forms of BN the dimensionality play a crucial role. For
the sake of comparison, we present the calculated values of BN for different
allotropic forms of BN in different dimensionalities. One sees that the B-N
double bond of 1D BN atomic chain is shortest and is 1.31 Å. The s$p^{2}$ bond
of h-BN and 2D BN has intermediate value of 1.45 Å. Therefore, h-BN can
considered to be quasi two dimensional. Three dimensional wurtzite and
zincblende BN crystal have s$p^{3}$-bonding with $d=1.56$ Å, which is largest
among the allotropic forms studied here. According to GGA results the cohesive
energy of 2D BN is 3 meV larger than that of h-BN. This is due to fact that
the GGA calculation cannot account the van der Waals interaction between
atomic layers of h-BN. However, the calculations using LDA, where the van der
Waals interactions are better accounted, yield the cohesive energy of h-BN is
$\sim$57 meV larger than that of 2D BN as one expects. The charge transfer
$\Delta Q$ from B to N atom increases with decreasing dimensionality. This due
to fact that $d$ decreases with decreasing dimensionality. As for the
coordination number increases with increasing dimension.

In 2D BN honeycomb structures and in its zigzag and armchair nanoribbons, the
B-N bond formed from the bonding s$p^{2}$ hybrid orbitals from B and N atoms
is essential. Owing to the transfer of charge from B to N the B-N bond
acquires an ionic character, which underlies the semiconducting properties
with wide band gap.

Table 1: Values of the bond length $d$ in Å, cohesive energy $E_{C}$ in eV per B-N pair, band gap EG in eV, charge transfer from B to N ($\Delta Q$) in electrons and lattice constants (a,c) in Åcalculated for various allotropic forms of BN in different dimensionality. | $d$ | $E_{C}$ | EG |  $\Delta$ Q | Lattice
---|---|---|---|---|---
1D Chain | 1.307 | -16.04 | 3.99 | 0.511 | a=2.614
2D BN | 1.452 | -17.65 | 4.64 | 0.429 | a=2.511
h-BN | 1.450 | -17.65 | 4.47 | 0.416 | a=2.511 , c/a=2.66
Wurtzite | 1.561 | -17.45 | 5.726 | 0.342 | a=2.542 , c/a=1.63
Zincblende | 1.568 | -17.49 | 4.50 | 0.334 | a=2.561

Bare armchair nanoribbon of 2D BN is again a nonmagnetic wide band gap
semiconductor, the band gap of which is practically unaltered with width
$n>8$. Upon passivation with hydrogen band gap of the ribbon increase by 0.3
eV. As for zigzag nanoribbons, they provide a number of interesting
properties. When its both edges are bare, it is ferromagnetic metal. When its
N-edge is passivated with hydrogen, it becomes an antiferromagnetic
semiconductor. In the reverse case, namely when B-side is passivated, it
becomes a ferromagnetic semiconductor. When both edges are passivated, it
becomes a nonmagnetic, wide band gap semiconductor. The band gap as well as
the magnetic state of a ribbon can be modified by periodic vacancy defects.
Finally, BN nanoribbons have been found to be strong, quasi one dimensional
and stable structures. They can sustain up to high strains, and they stretch
in the plastic region with structural transformations.

Briefly, the calculated electronic, magnetic and mechanical properties of 2D
BN honeycomb structure and its nanoribbons present interesting but some
differences from graphene. In this respect BN honeycomb structure and its
nanoribbons are complimentary to graphene. The properties of 2D BN honeycomb
structure can be changed upon functionalization with foreign atoms.
Interesting quantum structures (such as single and series quantum dots,
resonant tunneling double barriers and multiple quantum well structures) based
on heterostructures and core shell structures of lattice matched graphene and
BN can be formed, since the band gap of BN is much larger than that of
graphene.

###### Acknowledgements.

Part of the computational resources have been provided by UYBHM at Istanbul
Technical University through grant Grant No. 2-024-2007. E. Aktürk gratefully
acknowledgments the receipt of a BIDEB Postdoctoral Fellowship from TUBITAK.

## References

  * (1) K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, Y. Zhang, S. V. Dubonos, I. V. Grigorieva, A. A. Firsov, Science 306, 666 (2004).
  * (2) Y. Zhang, Y.-W. Tan, H. L. Stormer, P. Kim, Nature 438, 201 (2005).
  * (3) C. Berger, Z. Song, X. Li, X. Wu, N. Brown, C. Naud, D. Mayou, T. Li, J. Hass, A. N. Marchenkov, E. H. Conrad, P. N. First, W. A. de Heer, Science 312, 1191 (2006).
  * (4) A. K. Geim, K. S. Novoselov, Nature Materials 6, 183 (2007).
  * (5) Y.-W. Son, M.L. Cohen and S.G. Louie, Nature 444, (2006).
  * (6) H. Sevinçli, M. Topsakal, E. Durgun, S. Ciraci, Phys. Rev. B 77, 195434 (2008).
  * (7) M. Topsakal; H. Sevinçli, S. Ciraci, Appl. Phys. Lett. 92, 173118 (2008).
  * (8) K. S. Novoselov, D. Jiang, F. Schedin, T. Booth , V. V. Khotkevich, S. Morozov, A. K. Geim, Proc. Natl. Acad. Sci. U.S.A. 102, 10451 (2005)
  * (9) D. Pacilé, J. C. Meyer, Ç. Ö. Girit, and A. Zettl, Appl. Phys. Lett. 92, 133107 (2008).
  * (10) A. Nagashima, N. Tejima, Y. Gamou, T. Kawai, and C. Oshima, Phys. Rev. Lett. 75, 3918 (1995).
  * (11) L. Bourgeois, Y. Bando, W. Q. Han, and T. Sato, Phys. Rev. B 61, 7686 (2000)
  * (12) N. G. Chopra, R. J. Luyken, K. Cherrey, V. H. Crespi, M. L. Cohen, S. G. Louie, A. Zettl, Science 1995, 269, 966.
  * (13) C. Zhi, Y. Bando, C. Tang, and D. Golberg, Appl. Phys. Lett. 87, 063107 (2005).
  * (14) D. Golberg, A. Rode, Y. Bando, M. Mitome, E. Gamaly, and B. Luther-Davies, Diamond Relat. Mater. 12, 1269 (2003).
  * (15) Y. J. Chen , H. Z. Zhang, Y. Chen, Nanotechnology 17, 786 (2006).
  * (16) Z. Zhang and W. Guo, Phys. Rev. B 77, 075403 (2008).
  * (17) Cheol-Hwan Park and Steven G. Louie, Nano Lett. 8, 2200 (2008).
  * (18) V. Barone, J. E. Peralta, Nano Lett., 8, 2210 (2008)
  * (19) X. Li, L. Zhang, S. Lee, H. Dai, Science 319, 1229 (2008).
  * (20) X. Wang, Y. Ouyang, X. Li, H. Wang, J. Guo, H. Dai, Phys. Rev. Lett. 100, 206803 (2008).
  * (21) P. E. Blöchl, Phys. Rev. B 50, 17953 (1994).
  * (22) J. P. Perdew, J. A. Chevary, S. H. Vosko, K. A. Jackson, M. R. Pederson, D. J. Singh, C. Fiolhais, Phys. Rev. B 46, 6671 (1992).
  * (23) G. Kresse, J. Hafner, Phys. Rev. B 47, 558 (1993).
  * (24) G. Kresse, J. Furthmüller, Phys. Rev. B 54, 11169 (1996).
  * (25) S. Baroni, A. Del Corso, S. Girancoli and P. Giannozzi, http:/www.pwscf.org/
  * (26) A. Catellani, M. Posternak, A. Baldereschi, A. J. Freeman, Phys. Rev. B 36, 6105 (1987).
  * (27) L. Liu, Y. P. Feng, and Z. X. Shen, Phys. Rev. B 68, 104102 (2003).
  * (28) N. Ooi, V. Rajan, J. Gottlieb, Y. Catherine, and J. B. Adams, Modell. Simul. Mater. Sci. Eng. 14, 515 (2006).
  * (29) K. Shimada, T. Sota, and K. Suzuki, J. Apl. Phys. 84, 9 (1998).
  * (30) M. P. Surh, S. G. Louie, and M. L. Cohen, Phys. Rev. B 43, 9126 (1991).
  * (31) R.T. Senger, S. Tongay, E. Durgun and S. Ciraci, Phys. Rev. B 72, 075419 (2005).
  * (32) A.V. Kurdyumov, V.L. Solozhenko, and W.B. Zelyavski, J. Appl. Crystallogr. 28, 540 (1995).
  * (33) Caution has to be taken in using the values of band gap calculated with DFT as in the present study, since the value of band gap is usually underestimated by DFT.
  * (34) The calculations of $\Delta Q$ have been carried out by using PWSCF software pwscf .
  * (35) S. Tongay, R.T. Senger, S. Dag and S. Ciraci, Phys. Rev. Lett. 93, 136404 (2004).
  * (36) E. Rokuta, Y. Hasegawa, K. Suzuki, Y. Gamou, C. Oshima and A. Nagashima, Phys. Rev. Lett. 79, 4609 (1997).
  * (37) L. Wirtz, A.Rubio, R.A. delaConcha, A. Loiseau, Phys. Rev. B 68,045425 (2003).
  * (38) G. Kern, G. Kresse and J. Hafner, Phys. Rev. B 59, 8551 (1999).
  * (39) W.J. Yu, W.M. Lau, S.P.Chan, Z.F. Liu, Q.Q. Zheng, Phys. Rev. B 67, 014108 (2003).
  * (40) J. Serrano, A. Bosak, R. Arenal, M. Krisch, K. Watanabe, T. Taniguchi, H. Kanda, A. Rubio, and L. Wirtz, Phys. Rev. Lett. 98, 095503 (2007).
  * (41) V. L. Solozhenkan, G. Will and F. Elf, Solid State Commun. 96,1 (1995).
  * (42) Y. Miyamoto, M. L. Cohen, and S. G. Louie, Phys. Rev. B 52, 14971 (1995).
  * (43) F. Liu, P. Ming and J. Li, Phys. Rev. 76, 064120 (2007).
  * (44) D. Alfè, http://chianti.geol.ucl.ac.uk/ dario.
  * (45) V. Barone, O. Hod and G.E. Scuseria, Nano Lett. 6, 2748 (2006).
  * (46) P. Esquinazi , D. Spemann, R. Höhne, A. Setzer, K.-H. Han, T. Butz, Phys. Rev. Lett. 91, 227201 (2003).
  * (47) A. Hashimoto,K. Suenaga, A. Gloter, K. Urita, S. Iijima, Nature 430, 870 (2004).
  * (48) O. V. Yazyev and L. Helm, Phys. Rev. B 75, 125408 (2007).
  * (49) M.A.H. Vozmediano, M.P. Lopez-Sancho, T. Stauber and F. Guinea, Phys. Rev. B, 72, 155121 (2005).
  * (50) L. Brey, H.A. Fertig, and S. Das Sarma, Phys. Rev. Lett. 99, 116802 (2007).
  * (51) J.J. Palacios, J. Fernandez-Rossier, and L. Brey, Phys. Rev. B. 77, 195428 (2008).
  * (52) M. Topsakal, E. Aktürk and S. Ciraci. Phys. Rev. B 78,xxxxx (2008).
  * (53) E.H. Lieb, Phys. Rev. Lett. 62, 1201 (1989).
  * (54) A. Zobelli, C.P. Ewels, A.Gloter and G. Seifert, Phys. Rev. B 75, 094104 (2007).
  * (55) R.F. Liu and C. Cheng, Phys. Rev. B 76, 014405 (2007).
  * (56) M.S. Si and D.S. Xue, Phys. Rev. B 75, 193409 (2007).
  * (57) S. Azeveda, J.R. Kashny, C.M.C. de Castilhoand F. De Brito, Nanotechnology 18, 495707 (2007).


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Stanford University, Palo Alto CA 94305 USA 22institutetext:
MBZUAI, Abu Dhabi, UAE 33institutetext: University of Technology Sydney,
Ultimo NSW 2007 Australia 44institutetext: School of Information Science and
Technology, University of Science and Technology of China
44email<EMAIL_ADDRESS><EMAIL_ADDRESS>
44email<EMAIL_ADDRESS>
# Contrastive Learning with Counterfactual Explanations for Radiology Report
Generation

Mingjie Li 11 Haokun Lin 22 Liang Qiu 11 Xiaodan Liang Corresponding Author.
Code is available at: https://github.com/mlii0117/CoFE22 Ling Chen 33
Abdulmotaleb Elsaddik 22 Xiaojun Chang 4422

###### Abstract

Due to the common content of anatomy, radiology images with their
corresponding reports exhibit high similarity. Such inherent data bias can
predispose automatic report generation models to learn entangled and spurious
representations resulting in misdiagnostic reports. To tackle these, we
propose a novel CounterFactual Explanations-based framework (CoFE) for
radiology report generation. Counterfactual explanations serve as a potent
tool for understanding how decisions made by algorithms can be changed by
asking “what if” scenarios. By leveraging this concept, CoFE can learn non-
spurious visual representations by contrasting the representations between
factual and counterfactual images. Specifically, we derive counterfactual
images by swapping a patch between positive and negative samples until a
predicted diagnosis shift occurs. Here, positive and negative samples are the
most semantically similar but have different diagnosis labels. Additionally,
CoFE employs a learnable prompt to efficiently fine-tune the pre-trained large
language model, encapsulating both factual and counterfactual content to
provide a more generalizable prompt representation. Extensive experiments on
two benchmarks demonstrate that leveraging the counterfactual explanations
enables CoFE to generate semantically coherent and factually complete reports
and outperform in terms of language generation and clinical efficacy metrics.

###### Keywords:

Medical report generation Counterfactual explanation Contrastive learning

##  1 Introduction

Automatically generating reports can reduce the load on radiologists and
potentially increase the accuracy and consistency of interpretations. This is
achieved by translating intricate radiology images into semantically coherent
and clinically reliable free texts. However, in comparison to generic
captioning tasks, Radiology Report Generation (RRG) presents a significant
challenge, often yielding unsatisfactory performance when employing direct
captioning methods [42, 30] in the field of radiology. The difficulty arises
due to the severe data bias within the limited image-report pair data
available, a challenge that has been extensively acknowledged and discussed
[27, 4, 23, 36, 45, 16].

Given the shared anatomical content, radiology images tend to display
significant similarity to one another, with abnormal or lesioned areas
typically occupying minor portions of the images [24, 44]. This similarity
also extends to the accompanying reports, where several sentences often
describe normal tissues. However, the clinical usefulness of radiology reports
hinges on the accurate depiction of abnormalities. This intrinsic data bias
tends to lead models to learn spurious and intertwined visual features,
resulting in the generation of inaccurate diagnostic reports. To mitigate data
bias, various successful concepts have been proposed by existing methods to
enhance learning representations, such as employing contrastive learning [23,
28], incorporating medical knowledge [27, 49], and implementing relational
memory [4] etc.

Recently, Tanida et al. [36] achieved promising performances by detecting
abnormal regions using a pre-trained detector with pseudo labels. They then
utilized these features to guide a pre-trained large language model (LLM) in
generating reports. Identifying critical regions that cover abnormalities or
lesions not only enhances non-spurious visual representation but also improves
the explainability of RRG models. Ideally, the method would employ golden
annotations to train a lesion detector capable of accurately localizing
abnormal regions or lesions. However, existing RRG benchmarks lack such
annotations. Relying on weakly supervised signals from pseudo labels [36, 46]
can result in misalignment. Furthermore, the limited size of available medical
data may prevent the full unleashing of the potential of LLMs.

To address these challenges, we introduce a novel concept: counterfactual
explanation (CE). The concept of CE [12, 43] has surfaced in machine learning
as an insightful tool to comprehend how models’ decisions can be changed. CE
offers a hypothetical alternative to the observed data, allowing for the
assessment and comprehension of models through ’what if’ scenarios. This
technique has been integrated into diagnostic models to not only improve
diagnostic accuracy but also enhance explainability[37, 5]. For example,
Tanyel et al. [37] proposed a CE to identify the minimal feature change and
effectively demonstrated which features are more informative in
differentiating two tumor types from MRI brains. Inspired by this, we aim to
make this progress further interactive and explainable by explaining the
global feature change in specific local regions. In particular, we propose a
CE as ‘what if we exchange the patch between two images, will the diagnosis
shift?’ to identify critical regions within images that may cover
abnormalities or lesions, providing insights into the diagnosis process. For
instance, as illustrated in Figure.1, we generate a counterfactual image by
iteratively swapping a patch between semantically similar images with
different diagnosis labels until a shift in predicted diagnosis is achieved.
Due to aforementioned similarities, exchanging a patch in the same position
between two radiology images, particularly those that are semantically similar
but carry different labeled diagnoses, does not disrupt the anatomical
content. Notably, previous methods only integrate the CE into the decision-
making process, lacking the ability to convey factual or counterfactual
information effectively. In contrast, we translate our CE into a prompt that
can present the key concept of CE and encapsulate the factual and
counterfactual content. This prompt can yield more comprehensive instructions
to LLMs and facilitate the elicitation of their knowledge.

Figure 1: A conceptual overview of our proposed counterfactual explanations is
presented. Such CEs help to construct a counterfactual image by iteratively
exchanging a patch between factual (positive) and negative images until the
predicted diagnosis shift occurs. In this instance, the box in red covering
the heart is identified as the critical region that causes the diagnosis
shift.

In this paper, we propose a CounterFactual Explanations-based framework (CoFE)
for radiology report generation. CoFE is capable of learning non-spurious
visual representations and effectively leverage the capabilities of LLMs.
First, we introduce a novel type of CEs for RRG tasks and propose a
counterfactual generation process through contrastive learning, which
constructs a counterfactual image and a learnable prompt. Specifically, we
adopt a negative sampling strategy to discover the most semantically similar
negative sample from the data bank, based on text similarity and diagnosis
label. By iteratively exchanging patches between factual (positive) and
negative samples until a predicted diagnosis change occurs, we pinpoint the
critical region and create a counterfactual image. We then employ contrastive
learning within a joint optimization framework to differentiate
representations between factual and counterfactual samples, enabling the model
to learn non-spurious visual representations. Subsequently, we employ a
pretrained LLM, GPT-2 Medium [34], as a decoder to generate reports. To fine-
tune the LLM efficiently, we propose a learnable prompt that encapsulates both
factual and counterfactual content. This prompt can elicit the embedded
knowledge within the LLM, which is helpful to generate semantically coherent
and factually complete reports.

We evaluate our proposed method on two benchmarks, IU-Xray [6] and MIMIC-CXR
[18] respectively. Extensive experiments demonstrate that our approach can
outperform previous competitive methods in metrics that measure descriptive
accuracy and clinical correctness. It indicates that leveraging CEs to learn
non-spurious representations and prompt the generation process can improve the
quality of predicted reports.

##  2 Related Work

###  2.1 Medical Report Generation

The pursuit of automating medical report generation through machine learning
aims to alleviate the workload on radiologists. Numerous effective concepts
exist to learn non-spurious representations to mitigate inherent data bias.
Relation memory [4, 3] can prompt enhancement by boosting interaction
efficiency of cross-modal memory network. Integrating medical knowledge is
another solution, researchers utilize graph structural data [48, 23] or
medical tags [22, 17] to incorporate prior knowledge into image encoding.
Additional models [50, 47] also enhance performance by integrating knowledge
information, with strategies including multi-modal semantic alignment and
multi-label classification pre-training. To identify the abnormalities, PPKED
[27] employs a unique architecture to mimic human learning processes. Tanida
et al. [36] utilize a lesion detector pre-trained by pseudo labels to attain
the non-spurious features and lead a pretrained GPT-2 [34]. Due to the lack of
annotations, weakly supervised signals from pseudo labels may lead to
misalignment. Although, large pretrained models showcase the adaptability in
learning medical representations [31], the scarcity of data may limit the
potential of LLMs. In this paper, our method concentrates on learning non-
spurious representations by identifying critical regions and employing a
robust prompt to fine-tune the LLM efficiently.

###  2.2 Counterfactual Explanations Reasoning

The advent of counterfactual explanations (CEs) construction has driven
significant innovations, particularly in computer vision applications,
enhancing both accuracy and interpretability. CEs have the potential to
relieve existing methodologies from the reliance on extensive training data
and meticulous annotations, by asking ‘what if’ scenario to explore self-
supervised signals. Fang et al. [8] and Kim et al. [19] have introduced
systems and frameworks, such as Counterfactual Generative Networks (CGN),
designed to augment interpretability and resilience to CEs inputs without
compromising accuracy. Similarly, CPL [12] proficiently generates
counterfactual features and has exhibited remarkable efficacy in tasks like
image-text matching and visual question answering. Ji et al. [15] specifically
target video relationship detection, constructing CEs to elucidate their
influence on factual scenario predictions. Further, the studies by Yang et al.
[51] on PubMedQA highlight the crucial role of CEs, generated via ChatGPT, in
learning causal features in counterfactual classifiers, demonstrating the
versatility and broad applicability of counterfactual methods across various
domains. In this paper, we employ CEs to enhance the RRG models, especially
where acquiring a substantial amount of golden annotations is prohibitively
expensive.

###  2.3 Prompt Tuning

Prompt tuning is a method in natural language processing (NLP) used to
efficiently modify the behavior of a pre-trained LLM, based on specific
prompts or trigger phrases. This approach involves fine-tuning the model on a
set of prompts or queries and their corresponding responses, allowing the
model to respond more accurately or appropriately to those or similar prompts
[35, 54]. For example, Guo et al.[11] applied Q-Learning to optimize soft
prompts, while PTuning v2 [29] demonstrated that continuous prompt tuning
could match the performance of fine-tuning in various scenarios. This
technique has also garnered significant interest in the field of computer
vision. CoOp [33] introduced a strategy for continuous prompt optimization to
negate the need for prompt design, and CoCoOp [53] expanded upon this by
learning an instance-conditional network to generate a unique input-
conditional token for each image. Fischer et al. [9] also prove the
adaptability of prompt tuning in medical image segmentation tasks. However,
the reliance on empirical risk minimization presents challenges, necessitating
advancements to avoid spurious representations. In this paper, we aim to
propose a generalizable prompt incorporating factual and counterfactual
content to efficiently refine the medical LLMs.

##  3 Methodology

In this section, we introduce the detailed implementations of our proposed
Coun-terFactual Explanations-based framework (CoFE). As shown in Fig.2, our
CoFE mainly consists of two uni-modal encoders, one cross-modal encoder, a
language decoder, and a counterfactual generation module with four training
objectives. We first introduce the backbone of CoFE and then describe the
counterfactual generation process in detail.

###  3.1 Set Up

In this work, we aim to integrate counterfactual explanations into report
generation models to learn non-spurious visual representations and efficiently
generate high-quality reports. Radiology report generation tasks require a
model to translate a complex radiology image $I$ into a generic report
$T=\\{y_{1},y_{2},\dots,y_{n}\\}$. We denote the target report by
$\hat{T}=\\{\hat{y}_{1},\hat{y}_{2},\dots,\hat{y}_{\hat{n}}\\}$. $n$ and
$\hat{n}$ represent the number of tokens in a report. In addition to
corresponding reports, we also utilize the diagnosis label $C$ for each
examination in this work. Since not all existing benchmarks provide such
annotations, we use the CheXPert labeling tool [13] to label ground truth
reports with 14 different medical terminologies. Notably, we assign the label
“No Finding" when CheXPert does not extract any terminologies. For instances
with multiple labels, we adopt a strategy during training where only one label
is randomly retained for each case.

Figure 2: Illustration of our proposed CounterFactual Explanations-based
framework (CoFE). CoFE consists of two unimodal encoders, one cross-modal
encoder, one language decoder, and our proposed counterfactual generation
module that can construct a counterfactual image and a learnable prompt,
respectively. The entire framework is trained through joint optimization,
mainly employing contrastive learning paradigms for radiology report
generation.

Automatic report generation systems are typically based on encoder-decoder
frameworks. The encoder generally aims to convert the given image $I$ into
dense visual vectors $f_{v}(I)$. The decoder is usually a sequence processing
network, which translates $f_{v}(I)$ to a report $T$. To generate the CE and
calculate the image report contrastive loss, we involve an additional text
encoder following the BLIP[21] architecture, drawing inspiration from the
successful concepts found in [28, 23].

###  3.2 Encoders

Image encoder. Different from prior works employing CNNs, we use a pre-trained
ViT [7]-S as the image encoder $f_{v}(\cdot)$. ViT enables finer semantic
feature extraction by dividing images into more patches, whose resolution is
16$\times$16, compared to conventional CNNs 7$\times$7\. A [CLS] token is also
prepended before input to the encoder layers. The encoder layer process,
$\mathbf{f}_{e}(\cdot)$, is defined as:

$\displaystyle\mathbf{f}_{e}(x)$
$\displaystyle=\text{LN}(\text{FFN}(e_{attn})+e_{attn}),$ (1) $\displaystyle
e_{attn}$ $\displaystyle=\text{LN}(\text{MHA}(x)+x),$ (2)

where FFN and LN represent Feed Forward Network [40] and Layer Normalization
operation [1], respectively. MHA [40] (multi-head attention) splits attention
into $n$ heads, with each head, $\text{Att}(\cdot)$, defined as:

$\displaystyle\text{Att}(x)=\text{softmax}(\frac{\mathbf{Q}^{x}(\mathbf{K}^{x})^{\top}}{\sqrt{d}})\mathbf{V}^{x}.$
(3)

with $d=384$ being the embedding space dimension, and
$\\{\mathbf{Q},\mathbf{K^{*}},\mathbf{V^{*}}\\}$ representing the
corresponding Query, Key, Value vectors. The resulting output, the encoded
visual vectors $\mathbf{f}_{I}$, will be used for report generation.

Text encoder. We employ a PubMedBERT [10], pre-trained with abstracts and full
texts from PubMed111https://pubmed.ncbi.nlm.nih.gov, as our text encoder
$f_{t}(\cdot)$. It extracts textual representations $f_{t}(T)$ from positive
and negative reports, which will be utilized for calculating the image-report
contrastive (IRC) loss to facilitate learning robust and generalizable medical
visual and textual representations. We utilize momentum image and text
encoders to extract positive and negative data representations in a batch.
Then we first calculate the softmax-normalized image-to-report similarity
$f_{m}^{\mathrm{i2t}}(I)$ and the report-to-image similarity
$f_{m}^{\mathrm{t2i}}(T)$ for the image $I$ and its paired report $T$ by
$f_{m}^{i2r}(I)=\frac{\exp s(I,T_{m})\//\tau}{\sum_{m=1}^{M}\exp
s(I,T_{m})\//\tau}$, with $\tau$ as a learnable temperature parameter. The IRC
loss can be written as:

$\displaystyle\mathcal{L}_{\mathrm{IRC}}=\frac{1}{2}(\mathcal{L}_{\mathrm{ce}}(g^{t2i}(T),f^{t2i}(T))+\mathcal{L}_{\mathrm{ce}}(g^{i2t}(I),f^{i2t}(I))).$
(4)

where $g(\cdot)$ denotes the ground truth of similarity.

###  3.3 Decoding Process

Recognizing the advanced capabilities of Large Language Models (LLMs) in
various language generation tasks, we utilize a GPT-2 Medium model, which is
pre-trained on the PubMed dataset, as our language decoder. In contrast to
R2GenGPT, which employs a frozen LLaMa-7B [38], our choice of GPT-2, with its
355 million parameters, allows for full fine-tuning. This adaptability enables
GPT-2 to more effectively cater to the nuances of report generation tasks.
GPT-2, an auto-regressive model leveraging self-attention, conditions each
output token in a sequence on its previous tokens for report generation. The
entire process can be represented as:

$\displaystyle p(T|I)=\prod_{t=1}^{n}p(y_{t}|y_{1},\dots,y_{t-1},I).$ (5)

Here, $y_{t}$ is the input token at time step $t$. The typical objective for
report generation is minimizing cross-entropy loss between the predicted and
ground truth token sequences. With ground truth report $\hat{R}$, all modules
are optimized to maximize $p(\mathbf{y}|I)$ by minimizing:

$\displaystyle\mathcal{L}_{\mathrm{RG}}=-\sum_{t=1}^{\hat{n}}\log
p(\hat{y}_{t}|\hat{y}_{1},\cdots,\hat{y}_{t-1},I).$ (6)

###  3.4 Counterfactual Generation

In this section, we will explain how to generate counterfactual features,
encompassing a counterfactual image and a learnable prompt in detail.
Counterfactual images are pivotal, allowing the model to discern non-spurious
features through contrasting representations between factual and
counterfactual images. The learnable prompt encapsulates both factual and
counterfactual contents and then efficiently refines the pre-trained LLM.

Figure 3: Illustration of negative sampling strategy. The objective is to
select a negative sample that is mostly similar in semantics but carries a
different diagnostic label from the data bank.

Negative sampling strategy. Counterfactual features combine features derived
from both factual (positive) and negative data. We first propose a negative
sampling strategy to select the negative data from a data bank. Such negative
data should have different labels and are difficult to distinguish from
factual data. To implement this, we first construct a data bank, denoted by
$D$, containing candidate data, each instance $d_{i}$ is annotated with
$\\{$Image $I_{i}$, Report $T_{i}$, Label $C_{i}\\}$, maintaining the balanced
distribution of diagnostic labels within the data bank. Next, we select the
negative data from the data bank, as
$d^{-}=\operatorname*{arg\,max}_{i}\text{BLEUScore}(T,T_{i})$ and $C\neq
C_{i}$. The BLEUScore function calculates the BLEU [32] score, setting the
factual report as a reference and the negative data as a candidate. The so-
selected $d^{-}={I^{-},T^{-},C^{-}}$ is earmarked as negative data, exhibiting
textual semantics similar to the original data but possessing distinct labels,
emphasizing their inherent dissimilarity. The entire procedure is visually
depicted in Fig. 3.

Counterfactual image. After selecting the negative data from candidates, we
proceed to generate counterfactual images combining factual and negative
images, thereby enhancing non-spurious representations through contrastive
learning. As shown in Fig.4, a factual image, $I$, is presented in the form of
$n$ patches: $I={p_{1},p_{2},...,p_{n}}$; its corresponding negative image is
represented as $I^{-}={p^{-}_{1},p^{-}_{2},...,p^{-}_{n}}$. From the 1st to
the $n$-th patch, each patch of the negative image replaces the patch of the
factual image at the corresponding position. The modified image is denoted by
$I^{\prime}=(1-u)*I+u*I^{-}$, where $u$ is a one-hot vector to present the
index of the replaced patch. Subsequent to each replacement, the modified
image is fed into a pre-trained and frozen discriminator composed of the image
encoder and a Multilayer Perceptron (MLP) to predict the logits for the
diagnostic label $C^{\prime}$. The replacement process ceases once
$C^{\prime}\neq C$, culminating in the acquisition of the counterfactual image
$I^{\prime}$. This methodology enables the identification of critical regions
that prompt models to alter the predicted diagnosis. In essence, such regions
contain pivotal information pertinent to the examination. It helps to mitigate
inherent data bias and facilitates the model’s focus on these critical
regions, learning non-spurious and robust visual representations.

Figure 4: Illustration of the counterfactual generation process, including a
counterfactual image and a learnable prompt.

Learnable prompt. Another key component of our counterfactual features is a
learnable prompt, designed to elicit knowledge and leash the potential of pre-
trained LLMs. Frequently used prompts in caption tasks, such as “the caption
is…” or “describe [visual tokens]”, clarify the task but often lack
comprehensive instructions. To rectify this deficiency, we embed both factual
and counterfactual content within the learnable prompt to attain more
generalizable representations. As suggested by [39], our prompt incorporates
detailed instructions and is formulated by concatenating the factual visual
tokens, factual label, counterfactual label, and the index of the patch with
supplementary text. The training prompt is articulated as “The $u$ patch of
image contains critical features for diagnosing $C$. Generate a diagnostic
report for the image by describing critical entities including tubes,
pneumothorax, pleural effusion, lung opacity, cardiac silhouette, hilar
enlargement, and mediastinum.”

###  3.5 Joint optimization

In addition to the image-report contrastive loss, and report generation loss,
we introduce a novel contrastive loss aimed at amplifying the proficiency of
visual representation learning. Specifically, the factual image feature
$f_{v}(I)$, text feature $f_{t}(T)$, and counterfactual image feature
$f_{v}(I^{\prime})$ are employed to compute the counterfactual loss,
$\mathbf{L}_{cf}$, thereby extending the divergence between the counterfactual
features and the features of the original data. This can be represented as:

$\mathcal{L}_{CF}=-\log\frac{e^{\frac{f_{v}(I)@f_{t}(T)}{\tau}}}{e^{\frac{f_{v}(I)@f_{t}(T)}{\tau}}+e^{\frac{f_{v}(I^{\prime})@f_{t}(T)}{\tau}}}$
(7)

Here, $@$ denotes cosine similarity. The total training loss is written as:

$\mathcal{L}=\mathcal{L}_{IRC}+\lambda_{rg}\mathcal{L}_{RG}+\lambda_{cf}\mathcal{L}_{CF}$
(8)

where $\lambda_{cf}$ and $\lambda_{rg}$ denote the loss weights, we assign a
value of 1.2 to $\lambda_{cf}$ and 1.5 to $\lambda_{rg}$ based on performance
on the validation set.

###  3.6 Inference

Our inference methodology is streamlined, as the counterfactual generation
module is operational exclusively during the training phase, thereby enhancing
the learning of both visual and textual representations. Given an image, the
hidden embeddings $f_{V}(I)$ are seamlessly concatenated with the prompt and
fed into the LLM to generate diagnostic reports. In a similar vein, the prompt
is refined to “Generate a diagnostic report for the image by describing
critical entities including tubes, pneumothorax, pleural effusion, lung
opacity, cardiac silhouette, hilar enlargement, and mediastinum.”

##  4 Experiments

###  4.1 Datasets, Evaluation Metrics and Settings

Datasets. We validate the efficacy of our proposed CoFE using the IU-Xray [6]
and MIMIC-CXR [18] benchmarks. The settings adopted by [4] are utilized to
uniformly split and preprocess the datasets and reports, ensuring a fair
comparison. IU-Xray[6], a prevalent benchmark for evaluating RRG systems,
comprises 3,955 reports and 7,470 images. After excluding cases with only one
image as per [4, 20], 2069/296/590 cases are allocated for
training/validation/testing respectively. We utilize CheXPert to extract
terminologies from reports and assign labels to each examination. MIMIC-
CXR[18], the most extensive radiology dataset publicly available, includes
368,960 images and 222,758 reports. It has officially segmented subsets and
has spurred the development of structurally explorative child datasets like
RadGraph [14].

Metrics. We employ two types of metrics to evaluate the quality of our
predicted reports. First, natural language generation (NLG) are employed to
assess the descriptive precision of the predicted reports, with CIDEr [41] and
BLEU [32] being primary. BLEU is primarily designed for machine translation,
evaluating word n-gram overlap between reference and candidate, repeating
frequent sentences can also achieve high scores. Conversely, CIDEr, developed
for captioning systems, rewards topic terms and penalizes frequent ones, thus
is more fitting for evaluating reports in RRG tasks. Additionally, ROUGE-L
[25] and METEOR [2] are also considered for comprehensive comparison. Lastly,
clinical efficacy metrics, a more recent innovation, ascertain the clinical
accuracy of reports by using the CheXPert labeling tool to annotate predicted
reports. Subsequent classification measurements like F1-Score, Precision, and
Recall assess the aptness of the generated reports in describing
abnormalities.

Experimental settings. For both datasets, we only utilize the front view
examinations. We first pre-train the ViT-S for 10 epochs using diagnosis
labels. Given the distinct domain difference between medical and general
texts, a pretrained PubMedBert [10] is utilized as both a tokenizer and a text
encoder. The training is conducted on 4 NVIDIA 2080 Ti GPUs, spanning 50
epochs with batch sizes of 8. The model checkpoint achieving the highest CIEDr
metric is selected for testing. An Adam optimizer, with a learning rate of
1e-4 and a weight decay of 0.02, is applied. The learning rate drops 10 every
2 epochs and stops at 1e-6. We set the size of data bank to 1,380. Note that
all encoded vectors are projected by a linear transformation layer into a
dimension of $d=384$.

###  4.2 Main Results

Table 1: The performance in NLG metrics of our proposed method compared to
other competitive methods on the IU-Xray and MIMIC-CXR datasets. The highest
figures in each column are highlighted in bold.

IU-Xray |  |  |  |  | MIMIC-CXR |  |  |  |
---|---|---|---|---|---|---|---|---|---
Methods | CIDEr | BLEU-4 | ROUGE-L | METEOR | Methods | CIDEr | BLEU-4 | ROUGE-L | METEOR
R2Gen | 0.398 | 0.165 | 0.371 | 0.187 | R2Gen | 0.253 | 0.103 | 0.277 | 0.142
KERP | 0.280 | 0.162 | 0.339 | - | CMN | - | 0.106 | 0.278 | 0.142
HRGP | 0.343 | 0.151 | 0.322 | - | TopDown | 0.073 | 0.092 | 0.267 | 0.129
MKG | 0.304 | 0.147 | 0.367 | - | PPKED | 0.237 | 0.106 | 0.284 | 0.149
PPKED | 0.351 | 0.168 | 0.376 | 0.190 | RGRG | 0.495 | 0.126 | 0.264 | 0.168
MGSK | 0.382 | 0.178 | 0.381 | - | MGSK | 0.203 | 0.115 | 0.284 | -
CMCL | - | 0.162 | 0.378 | 0.186 | CMCL | - | 0.097 | 0.281 | 0.133
DCL | 0.586 | 0.163 | 0.383 | 0.193 | DCL | 0.281 | 0.109 | 0.284 | 0.150
CoFE | 0.731 | 0.175 | 0.438 | 0.202 | CoFE | 0.453 | 0.125 | 0.304 | 0.176

Descriptive Accuracy. We compare our CoFE with several competitive RRG methods
on two benchmarks. R2Gen [4] and CMN [3] are two widely-used baseline models
implementing relation memory. KERP [20], PPKED [27], MKG [52] and MGSK [48]
are proposed to integrate medical knowledge with typical RRG backbones. CMCL
[26] and DCL [23] employ contrastive learning to further improve performance.
As presented in Table.1, our method notably outperforms all competing
approaches, attaining the highest figures across almost all the metrics, with
a CIDEr score of 0.731 and a BLEU-4 score of 0.175 on IU-xray. Similarly, our
method demonstrates competitive performance on the MIMIC-CXR dataset,
achieving the highest ROUGE-L score of 0.304 and METEOR score of 0.176. These
results showcase the superior capability of our method in generating matched
and semantically similar reports.

Clinical Correctness. We also evaluate our method by Clinical Efficacy (CE)
metrics on the MIMIC-CXR dataset to evaluate the clinical correctness of our
predicted reports. In Table. 2, we compare the performance against several
baseline models, DCL, R2Gen, and MKSG, respectively. Most notably, our CoFE
achieves the SOTA performance across all the clinical efficacy metrics, with a
Precision of 0.489, Recall of 0.370, and F1-score of 0.405. This performance-
boosting underscores the effectiveness of integrating counterfactual
explanations, enabling the model to generate more clinically correct and
relevant reports.

###  4.3 Analysis

Table 2: The comparison of the clinical efficacy metrics on the MIMIC-CXR
dataset.

Methods | Precision | Recall | F1-score
---|---|---|---
DCL | 0.471 | 0.352 | 0.373
R2Gen | 0.333 | 0.273 | 0.276
MKSG | 0.458 | 0.348 | 0.371
Base | 0.325 | 0.271 | 0.273
\+ LLMs | 0.396 | 0.323 | 0.317
\+ prompt | 0.463 | 0.355 | 0.366
\+ $\mathcal{L}_{CF}$ (full) | 0.489 | 0.370 | 0.405

In this section, we conduct ablation studies and a case study on IU-Xray and
MIMIC-CXR datasets to investigate the proficiency of each key component in
CoFE. Specifically, Table. 3 presents the quantitative analysis of CoFE on IU-
Xray measuring descriptive accuracy. In addition, clinical correctness
evaluation is reported in Table. 2. We employ a BLIP without the cross-modal
encoder as our base model. We found that abandoning this module can save
computation sources and achieve similar performances.

Effect of pre-trained LLMs. Compared with the base model in setting (a),
illustrated in Table 3, where we utilize a pre-trained PubMedBert and a
355M-parameter GPT-2 Meduium as the text encoder and language decoder, there
is a significant enhancement in all metrics, with CIDEr improving from 0.363
to 0.510, emphasizing the impactful role of LLMs in enhancing the report
generation performance. Specifically, PubMedBert can encode the reports into
better textual representations, while GPT-2 has the capability to generate
semantically coherent and logically consistent reports.

Non-spurious Representation Learning. The primary motivation for integrating
counterfactual explanations is to enhance non-spurious visual representations
by contrasting the representations between factual and counterfactual images.
When comparing setting (c) to Setting (a) and the full model to setting (b), a
significant performance boost is observable across all metrics. For instance,
CIDEr elevates from 0.510 to 0.678, and from 0.698 to 0.731, respectively.
Additionally, the final BLEU-4 metrics reach 0.175, achieving the SOTA
performances. These notable elevations highlight the importance of non-
spurious representation learning capabilities in radiology report generation
tasks.

Effect of Prompt Tuning. To fully elicit pre-trained knowledge and unleash the
potential of LLMs, we propose a learnable prompt that encapsulates both
factual and counterfactual content to refine the LLMs. Observing setting (a)
vs (b) and (c) vs the full model, it is evident that our proposed prompt can
further augment performance, especially evident in ROUGE-L, which elevates
from 0.355 to 0.373 and from 0.381 to 0.428, respectively. This increment
underscores the effectiveness of our prompt in refining the model’s natural
language generation capability. Furthermore, as shown in Table.2, this prompt
can also increase the clinical correctness of the predicted reports.

Negative Sampling Strategy. The key point to constructing a counterfactual
image is selecting negative data which have different labels and are difficult
to be distinguished from the factual data. To verify this, we employ a random
sampling strategy in which candidate data are indiscriminately selected as the
negative sample. The incorporation of this random sampling strategy in
settings (c) and (d) results in a discernible degeneration in the model’s
capability to generate high-quality reports. For instance, the CIDEr metrics
drop from 0.678 to 0.653, while Bleu-4 scores decrease to 0.156. This slight
decline across almost all performance metrics elucidates the influential role
of our negative sampling strategy in pinpointing the most suitable negative
data.

Table 3: Quantitative analysis of our proposed method on the IU-Xray dataset.
We employ a vanilla BLIP without loading pre-trained parameters as the base
model.

Settings | LLMs | Prompt | $\mathcal{L}_{CF}$ | Random Sampling | CIDEr | BLEU-4 | ROUGE-L | METEOR
---|---|---|---|---|---|---|---|---
Base |  |  |  |  | 0.363 | 0.132 | 0.248 | 0.154
(a) | ✓ |  |  |  | 0.510 | 0.151 | 0.355 | 0.180
(b) | ✓ | ✓ |  |  | 0.698 | 0.160 | 0.373 | 0.183
(c) | ✓ |  | ✓ |  | 0.678 | 0.164 | 0.381 | 0.191
(d) | ✓ |  | ✓ | ✓ | 0.653 | 0.156 | 0.392 | 0.186
(e) | ✓ | ✓ | ✓ | ✓ | 0.706 | 0.166 | 0.407 | 0.196
CoFE | ✓ | ✓ | ✓ |  | 0.731 | 0.175 | 0.428 | 0.202

Figure 5: Illustration of reports generated by R2Gen, DCL and our CoFE. The
text in blue demonstrates the ground truth diagnosis labels. The red text
represents the accurately matched abnormalities.

Qualitative Analysis. In Figure.5, we present two samples from MIMIC-CXR and
corresponding reports generated by R2Gen, DCL and our CoFE. R2Gen seems to
lack specificity and detailed insights, providing a more generalized statement
about the conditions and missing several key abnormalities mentioned in the
ground truth, such as pulmonary nodules and pleural effusion. The DCL model is
somewhat more aligned with the ground truth, acknowledging the unchanged
appearance of the cardiac silhouette and the presence of extensive bilateral
parenchymal opacities. However, it fails to mention the presence of pulmonary
nodules and the pleural effusion in the right middle fissure specifically. In
contrast, CoFE addresses pleural effusion, atelectasis, and the absence of
pneumonia and pneumothorax, making it more in alignment with certain elements
of the ground truth. These observations prove that our CoFE can generate
factual complete and consistent reports.

Figure 6: Heatmaps that illustrate the frequency at which individual patches
were replaced during the training process of two distinct medical imaging
datasets: IU-Xray on the left and MIMIC-CXR on the right.

Frequency of Replaced Patches: In Figure 6, we present two heatmaps that
illustrate the frequency at which specific patches were replaced to construct
counterfactual images during the training process on the IU-Xray and MIMIC-CXR
datasets. The color gradient in each patch ranges from dark to light, with the
lightest shade denoting the highest replacement frequency. The observed
patterns across the heatmaps indicate that patches associated with critical
anatomical regions, such as the heart, lungs, and air spaces, are replaced
more frequently. This suggests that during training, these areas are
particularly targeted as they likely contain critical diagnostic information
(counterfactual contexts), underlining their clinical significance in disease
detection or condition assessment. The distribution of frequency substantiates
our methodology for constructing counterfactual images, which effectively
identifies and emphasizes the most pertinent regions, thereby potentially
enhancing the representation learning process.

##  5 Conclusion

In this paper, we present a novel framework, Counterfactual Explanations-based
Framework (CoFE), designed for radiology report generation. To address the
inherent data bias, we introduce a novel counterfactual concept, allowing CoFE
to identify critical regions and construct a counterfactual image during
training. By contrasting the representations between factual and
counterfactual features, CoFE is adept at learning non-spurious visual
representations. Subsequently, we summarize the counterfactual generation
process into a learnable prompt, enabling the efficient fine-tuning of a pre-
trained LLM. Experiments on two widely-recognized benchmarks verify the
efficacy of our approach in generating factual, comprehensive, and coherent
reports.

Acknowledge This work is supported by ARC DP210101347.

## References

  * [1] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
  * [2] Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72 (2005)
  * [3] Chen, Z., Shen, Y., Song, Y., Wan, X.: Cross-modal memory networks for radiology report generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 5904–5914 (2021)
  * [4] Chen, Z., Song, Y., Chang, T., Wan, X.: Generating radiology reports via memory-driven transformer. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (2020)
  * [5] Dai, X., Keane, M.T., Shalloo, L., Ruelle, E., Byrne, R.M.J.: Counterfactual explanations for prediction and diagnosis in XAI. In: Conitzer, V., Tasioulas, J., Scheutz, M., Calo, R., Mara, M., Zimmermann, A. (eds.) AIES ’22: AAAI/ACM Conference on AI, Ethics, and Society, Oxford, United Kingdom, May 19 - 21, 2021. pp. 215–226. ACM (2022). https://doi.org/10.1145/3514094.3534144, https://doi.org/10.1145/3514094.3534144
  * [6] Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association 23(2), 304–310 (2016)
  * [7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2020)
  * [8] Fang, Z., Kong, S., Fowlkes, C., Yang, Y.: Modularized textual grounding for counterfactual resilience. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6378–6388 (2019)
  * [9] Fischer, M., Bartler, A., Yang, B.: Prompt tuning for parameter-efficient medical image segmentation. CoRR abs/2211.09233 (2022). https://doi.org/10.48550/arXiv.2211.09233, https://doi.org/10.48550/arXiv.2211.09233
  * [10] Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing (2020)
  * [11] Guo, H., Tan, B., Liu, Z., Xing, E.P., Hu, Z.: Efficient (soft) q-learning for text generation with limited good data. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. pp. 6969–6991. Association for Computational Linguistics (2022). https://doi.org/10.18653/v1/2022.findings-emnlp.518, https://doi.org/10.18653/v1/2022.findings-emnlp.518
  * [12] He, X., Yang, D., Feng, W., Fu, T.J., Akula, A., Jampani, V., Narayana, P., Basu, S., Wang, W.Y., Wang, X.: Cpl: Counterfactual prompt learning for vision and language models. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 3407–3418 (2022)
  * [13] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 590–597 (2019)
  * [14] Jain, S., Agrawal, A., Saporta, A., Truong, S., Bui, T., Chambon, P., Zhang, Y., Lungren, M.P., Ng, A.Y., Langlotz, C., et al.: Radgraph: Extracting clinical entities and relations from radiology reports. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) (2021)
  * [15] Ji, X., Chen, J., Wu, X.: Counterfactual inference for visual relationship detection in videos. In: 2023 IEEE International Conference on Multimedia and Expo (ICME). pp. 162–167. IEEE (2023)
  * [16] Jin, H., Che, H., Lin, Y., Chen, H.: Promptmrg: Diagnosis-driven prompts for medical report generation. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 2607–2615. No. 3 (2024)
  * [17] Jing, B., Xie, P., Xing, E.: On the automatic generation of medical imaging reports. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2577–2586 (2018)
  * [18] Johnson, A.E.W., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., ying Deng, C., Mark, R.G., Horng, S.: Mimic-cxr: A large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019)
  * [19] Kim, J., Kim, M., Ro, Y.M.: Interpretation of lesional detection via counterfactual generation. In: 2021 IEEE International Conference on Image Processing (ICIP). pp. 96–100. IEEE (2021)
  * [20] Li, C.Y., Liang, X., Hu, Z., Xing, E.P.: Knowledge-driven encode, retrieve, paraphrase for medical image report generation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 6666–6673 (2019)
  * [21] Li, J., Li, D., Xiong, C., Hoi, S.C.H.: BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., Sabato, S. (eds.) International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. Proceedings of Machine Learning Research, vol. 162, pp. 12888–12900. PMLR (2022), https://proceedings.mlr.press/v162/li22n.html
  * [22] Li, M., Cai, W., Verspoor, K., Pan, S., Liang, X., Chang, X.: Cross-modal clinical graph transformer for ophthalmic report generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20656–20665 (2022)
  * [23] Li, M., Lin, B., Chen, Z., Lin, H., Liang, X., Chang, X.: Dynamic graph enhanced contrastive learning for chest x-ray report generation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 3334–3343. IEEE (2023). https://doi.org/10.1109/CVPR52729.2023.00325, https://doi.org/10.1109/CVPR52729.2023.00325
  * [24] Li, M., Liu, R., Wang, F., Chang, X., Liang, X.: Auxiliary signal-guided knowledge encoder-decoder for medical report generation. World Wide Web pp. 1–18 (2022)
  * [25] Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. Association for Computational Linguistics (Jul 2004)
  * [26] Liu, F., Ge, S., Wu, X.: Competence-based multimodal curriculum learning for medical report generation. arXiv preprint arXiv:2206.14579 (2022)
  * [27] Liu, F., Wu, X., Ge, S., Fan, W., Zou, Y.: Exploring and distilling posterior and prior knowledge for radiology report generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13753–13762 (2021)
  * [28] Liu, F., Yin, C., Wu, X., Ge, S., Zhang, P., Sun, X.: Contrastive attention for automatic chest x-ray report generation. In: Findings of the Association for Computational Linguistics. pp. 269–280 (2021)
  * [29] Liu, X., Ji, K., Fu, Y., Du, Z., Yang, Z., Tang, J.: P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR abs/2110.07602 (2021), https://arxiv.org/abs/2110.07602
  * [30] Lu, J., Xiong, C., Parikh, D., Socher, R.: Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. pp. 3242–3250. IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.345, https://doi.org/10.1109/CVPR.2017.345
  * [31] Mohsan, M.M., Akram, M.U., Rasool, G., Alghamdi, N.S., Abdullah-Al-Wadud, M., Abbas, M.: Vision transformer and language model based radiology report generation. IEEE Access 11, 1814–1824 (2023). https://doi.org/10.1109/ACCESS.2022.3232719, https://doi.org/10.1109/ACCESS.2022.3232719
  * [32] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (Jul 2002)
  * [33] Peng, Z., Hui, K.M., Liu, C., Zhou, B.: Learning to simulate self-driven particles system with coordinated policy optimization. Advances in Neural Information Processing Systems 34 (2021)
  * [34] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [35] Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020)
  * [36] Tanida, T., Müller, P., Kaissis, G., Rueckert, D.: Interactive and explainable region-guided radiology report generation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 7433–7442. IEEE (2023). https://doi.org/10.1109/CVPR52729.2023.00718, https://doi.org/10.1109/CVPR52729.2023.00718
  * [37] Tanyel, T., Ayvaz, S., Keserci, B.: Beyond known reality: Exploiting counterfactual explanations for medical research. CoRR abs/2307.02131 (2023). https://doi.org/10.48550/arXiv.2307.02131, https://doi.org/10.48550/arXiv.2307.02131
  * [38] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [39] Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P., Carroll, A., Lau, C., Tanno, R., Ktena, I., Mustafa, B., Chowdhery, A., Liu, Y., Kornblith, S., Fleet, D.J., Mansfield, P.A., Prakash, S., Wong, R., Virmani, S., Semturs, C., Mahdavi, S.S., Green, B., Dominowska, E., y Arcas, B.A., Barral, J.K., Webster, D.R., Corrado, G.S., Matias, Y., Singhal, K., Florence, P., Karthikesalingam, A., Natarajan, V.: Towards generalist biomedical AI. CoRR abs/2307.14334 (2023). https://doi.org/10.48550/arXiv.2307.14334, https://doi.org/10.48550/arXiv.2307.14334
  * [40] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [41] Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)
  * [42] Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015. pp. 3156–3164. IEEE Computer Society (2015). https://doi.org/10.1109/CVPR.2015.7298935, https://doi.org/10.1109/CVPR.2015.7298935
  * [43] Virgolin, M., Fracaros, S.: On the robustness of sparse counterfactual explanations to adverse perturbations. Artif. Intell. 316, 103840 (2023). https://doi.org/10.1016/j.artint.2022.103840, https://doi.org/10.1016/j.artint.2022.103840
  * [44] Voutharoja, B.P., Wang, L., Zhou, L.: Automatic radiology report generation by learning with increasingly hard negatives. CoRR abs/2305.07176 (2023). https://doi.org/10.48550/arXiv.2305.07176, https://doi.org/10.48550/arXiv.2305.07176
  * [45] Wang, Z., Liu, L., Wang, L., Zhou, L.: Metransformer: Radiology report generation by transformer with multiple learnable expert tokens. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. pp. 11558–11567. IEEE (2023). https://doi.org/10.1109/CVPR52729.2023.01112, https://doi.org/10.1109/CVPR52729.2023.01112
  * [46] Wang, Z., Zhou, L., Wang, L., Li, X.: A self-boosting framework for automated radiographic report generation. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. pp. 2433–2442. Computer Vision Foundation / IEEE (2021). https://doi.org/10.1109/CVPR46437.2021.00246, https://openaccess.thecvf.com/content/CVPR2021/html/Wang_A_Self-Boosting_Framework_for_Automated_Radiographic_Report_Generation_CVPR_2021_paper.html
  * [47] Xu, D., Zhu, H., Huang, Y., Jin, Z., Ding, W., Li, H., Ran, M.: Vision-knowledge fusion model for multi-domain medical report generation. Inf. Fusion 97, 101817 (2023). https://doi.org/10.1016/j.inffus.2023.101817, https://doi.org/10.1016/j.inffus.2023.101817
  * [48] Yang, S., Wu, X., Ge, S., Zhou, S., Xiao, L.: Knowledge matters: Chest radiology report generation with general and specific knowledge. Medical Image Analysis 80, 102510–102510 (2022)
  * [49] Yang, S., Wu, X., Ge, S., Zheng, Z., Zhou, S.K., Xiao, L.: Radiology report generation with a learned knowledge base and multi-modal alignment. Medical Image Anal. 86, 102798 (2023). https://doi.org/10.1016/j.media.2023.102798, https://doi.org/10.1016/j.media.2023.102798
  * [50] Yang, S., Wu, X., Ge, S., Zheng, Z., Zhou, S.K., Xiao, L.: Radiology report generation with a learned knowledge base and multi-modal alignment. Medical Image Anal. 86, 102798 (2023). https://doi.org/10.1016/j.media.2023.102798, https://doi.org/10.1016/j.media.2023.102798
  * [51] Yang, Z., Liu, Y., Ouyang, C., Ren, L., Wen, W.: Counterfactual can be strong in medical question and answering. Inf. Process. Manag. 60(4), 103408 (2023). https://doi.org/10.1016/j.ipm.2023.103408, https://doi.org/10.1016/j.ipm.2023.103408
  * [52] Zhang, Y., Wang, X., Xu, Z., Yu, Q., Yuille, A., Xu, D.: When radiology report generation meets knowledge graph. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 12910–12917 (2020)
  * [53] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Conditional prompt learning for vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16816–16825 (2022)
  * [54] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International Journal of Computer Vision 130(9), 2337–2348 (2022)