# Comparative analysis of structural, elastic, electronic, phonon, thermal and
optical properties of two Na6Ge2Se6 phases from first principles calculations

Qi Zhang  Department of Physics, Missouri University of Science and
Technology, Missouri, Rolla 65401, USA  Amitava Choudhury  Department of
Chemistry, Missouri University of Science and Technology, Missouri, Rolla
65401, USA  Aleksandr Chernatynskiy  Department of Physics, Missouri
University of Science and Technology, Missouri, Rolla 65401, USA

###### Abstract

The demand for new alkali metal chalcogenide materials is continuously
increasing due to their potential applications across various technological
fields. Recently, a new compound, Na6Ge2Se6, was computationally predicted,
representing a new phase distinct from the experimentally observed Na6Ge2Se6
reported in 1985. Notably, this newly predicted phase displays a lower total
energy compared to the previously known experimental phase, as determined by
first-principles calculations. In this study, we undertake a thorough
comparative analysis of the structural, elastic, electronic, phonon, thermal,
and optical properties of these two Na6Ge2Se6 phases. Our results show that
both phases meet mechanical and dynamical stability criteria. The electronic
band structure analysis confirms the semiconducting nature of both materials,
with a 2.97 eV indirect band gap for the predicted phase and a 2.93 eV direct
band gap for the observed phase. Optically, both phases exhibit strong
absorption in the ultraviolet region. Thermal properties analysis reveals that
the predicted phase is more thermodynamically stable below 907 K, while the
observed phase shows greater thermodynamic stability above this temperature.

Figure 1: Crystal structures of both Na6Ge2Se6 phases. (a) Predicted new
Na6Ge2Se6 phase. (b) Observed Na6Ge2Se6 phase. (c) One Na atom and five Se
atoms form a [NaSe5] pyramid. (d) One Na atom and six Se atoms form a [NaSe6]
distorted octahedron.

##  I introduction

Ternary alkali metal-based chalcogenides are characterized by the general
formula ABZ where A signifies an alkali metal (Li, Na, K), B includes main
group elements (e.g., Ge, Bi) or transition metals (e.g., Fe, Cu), while Z
represents chalcogen elements (e.g., S, Se). These compounds have gained
considerable interest recently for their potential applications in energy
conversion/storage, optoelectronics, and thermoelectrics. These areas are
vital for supporting the advancement of current technologies and industries,
especially in addressing the global energy challenge as society aims to
transition from fossil fuels to renewable, efficient, and eco-friendly energy
solutions. In response to these demands, numerous ABZ compounds have been
synthesized and explored.

In the quest for improved energy storage solutions, the focus on ion-battery
anodes is crucial due to their significant impact on battery performance. ABZ
materials like LiLnSe2 and NaFeS2 have shown high storage capacities and
efficient energy cycling in recent studies [1, 2], with NaFeS2 showcasing
excellent cycling stability as a Li-ion battery anode, suggesting potential
for enhanced battery lifespan and reliability. Optoelectronically, ABZ
compounds such as NaSbS2 offer strong absorption in the visible spectrum, and
their affordability, abundance, and non-toxicity make them attractive for
solar energy applications[3]. Furthermore, in thermoelectric applications, ABZ
materials like CsAg5Te3 exhibit promising mid-temperature performance with
high $ZT$ values[4], thanks to their intrinsically low thermal conductivity[5,
6, 7]. Ternary chalcogenides have also played a crucial role in enabling the
synthesis of more complex multinary chalcogenides in a rational manner[8].
Many of these compounds have demonstrated significant technological potential.
For instance, outstanding electric thermal properties have been observed in
materials like $\text{Na}_{0.95}\text{Pb}_{19}\text{SbTe}_{22}$, achieving a
$ZT$ value greater than 1 across one of the broadest temperature ranges (475
to 650 K) reported for any single material[9]. Additionally, a study on
$\text{Na}_{1-x}\text{K}_{x}\text{AsQ}_{2}$ $(\text{Q}=\text{S,Se})$
highlighted its potential applications in signal processing and data
transmission, attributed to its nonlinear optical properties[10].

Although many ABZ compounds have been synthesised, the existing collection of
these ternary ABZ compounds still lacks many possible combinations. The
process of discovering new materials through purely experimental means is
often prolonged, as the growth and characterization of their properties can be
time-consuming. With the increase of computational power, simulations are
increasingly employed in material discovery, for both screening of the
properties of already known materials as well as prediction of the new
compounds. Recently, a new structure of the chalcogenide, Na6Ge2Se6 was
computationally predicted using simulated annealing and first-principles
methods[11]. This compound represents a new phase distinct from the
experimentally observed compound reported in 1985 with the same composition
(Na6Ge2Se6)[12]. The latter has been utilized recently in synthesizing a
potential nonlinear optical material, Na8Mn2(Ge2Se6)2[8]. Notably, the newly
predicted Na6Ge2Se6 phase shows a lower total energy compared to the observed
phase from first-principles calculations, suggesting it as a potential meta-
stable state of Na6Ge2Se6 that has not been reported before. Furthermore, the
observed Na6Ge2Se6 phase has received little attention, with no comprehensive
analysis conducted to the best of our knowledge. This gap in research has
motivated us to perform a comparative analysis of the structural, elastic,
electronic, phonon, thermal, and optical properties of these two compounds,
which could serve as a valuable reference for future studies.

##  II Computational methodology

In this study, we examined the ground state properties of both predicted and
observed Na6Ge2Se6 phases using density functional theory (DFT) as implemented
in the Vienna Ab initio Simulation Package (VASP)[13, 14]. This involved
solving the Kohn-Sham equations[15] to determine the ground state energies of
a crystalline system. Given the close proximity of the ground state energies
between the predicted and observed phases, calculations were performed using
two distinct approaches for the exchange-correlation potential: the
Generalized Gradient Approximation (GGA) within the Perdew-Burke-Ernzerhof
framework (GGA-PBE)[16] and the hybrid functionals from Heyd-Scuseria-
Ernzerhof (HSE06)[17]. In both Na6Ge2Se6 structures, Integration over the
Brillouin zone was executed via the tetrahedron method with Gaussian smearing,
employing a $5\times 5\times 5$ Monkhorst–Pack k-point mesh[18]. The energy
cutoff for the plane wave basis set was set to 600 eV. The Ge
$3d^{10}4s^{3}4p^{2}$, Se $4s^{2}p^{4}$, Na $2p^{6}3s^{1}$ electrons are
treated as valence electrons. convergence thresholds were
$1\text{\times}{10}^{-8}$ eV for the electronic self-consistent loop and
$1\text{\times}{10}^{-7}$ eV for structure optimization.

Further, we explored the elastic and dynamical properties of optimized
structures using first-principle calculations with the GGA-PBE functional
within VASP. This analysis allowed for the calculation of elastic constants
$C_{ij}$ and several elastic properties, including the bulk modulus $B$,
Young’s modulus $E$ and shear modulus $G$, along with optical properties such
as the dielectric function $\epsilon(\omega)$, absorption coefficient
$\alpha(\omega)$ and conductivity $\sigma(\omega)$ using the VASPKIT[19] tool.
The Phonopy package[20, 21] facilitated the calculation of phonon dispersion
relations, phonon density of states, and related thermal properties.
Considering the tendency of GGA-PBE to underestimate band gaps in
semiconductors and insulators, the HSE06 functional[17] was applied to
investigate electronic and optical properties. The electronic band structures
and density of states for both phases were visualized using the open-source
Python library Pymatgen[22].

Figure 2:  Electron localization function (ELF) (a) Predicted Na6Ge2Se6 phase.
(b) Observed Na6Ge2Se6 phase.

##  III Crystal structure

We analyzed the Na6Ge2Se6 structures using two different exchange-correlation
energy functionals: GGA-PBE and the HSE06 hybrid functional. The calculated
energies per atom are presented in Table 1. Both PBE and HSE06 functionals
indicated lower energies for the predicted Na6Ge2Se6 phase by 16 meV/atom. To
confirm the convergence of our DFT calculations, we conducted two additional
structure relaxations with the GGA-PBE functional: one with a 400 $eV$ energy
cutoff and a $3\times 3\times 3$ k-point mesh, and another with a 500 $eV$
energy cutoff and a $4\times 4\times 4$ k-point mesh. For the predicted phase,
both of these settings produced energy per atom within 0.006% of the most
accurate result listed in the Table 1. For the observed phase, the difference
is even smaller, within 0.002%. The energy difference between the predicted
and observed phases listed above is an order of magnitude larger than 0.06%,
thus numerical accuracy is sufficient to make conclusions about the energy
order of two structures. This leads us to conclude that the predicted
Na6Ge2Se6 phase is energetically more favorable than the observed one at 0 K.

The predicted Na6Ge2Se6 crystallizes in the space group $R\mkern
1.5mu\overline{\mkern-1.5mu3\mkern-1.5mu}\mkern 1.5mu$ (No.148), whereas the
observed phase is found in the space group $P2_{1}/c$ (No.14). Both structures
exhibit identical ethane-like selenide [Ge2Se6] units, featuring a Ge-Ge bond
length of 2.49 Å for the predicted phase and 2.47 Å for the observed phase,
and an average Ge-Se bond length of 2.39 Å as detailed in Table 2. These bond
lengths are slightly greater than the originally observed Na6Ge2Se6 data[12],
which recorded a Ge-Ge bond length of 2.43 Å and a Ge-Se bond length of 2.33
Å. The [Ge2Se6] ethane-like dimer configuration bears similarity to the
[P2Se6] in Mg2P2Se6[23], and [Si2Se6] in Na4MgSi2Se6[24]. For both Na6Ge2Se6
phases, the formed [Ge2Se6] dimers are isolated from each other, similar to
the cases in K6Ge2Se6[25] and Cs6Ge2Se6[26]. The predicted Na6Ge2Se6 phase
exhibits cell parameters of $a=11.349$ Å, $b=11.349$ Å, $c=10.900$ Å,
$\alpha=90\degree$, $\beta=90\degree$ and $\gamma=120\degree$, while the
observed Na6Ge2Se6 has $a=8.445$ Å, $b=12.038$ Å, $c=8.316$ Å,
$\alpha=90\degree$, $\beta=118.877\degree$ and $\gamma=90\degree$ with a
smaller unit cell volume. The predicted Na6Ge2Se6 features only one
crystallographically unique Na atom, one Ge atom, and one Se atom, contrasting
with the observed Na6Ge2Se6, which has three unique Na atoms, one Ge atom, and
three Se atoms. In the predicted Na6Ge2Se6, each Na atom coordinates with five
Se atoms to form a [NaSe5] pyramid. This pyramid shares corners with its
neighboring [NaSe5] pyramids, whereas in the observed phase, each Na atom
bonds with six Se atoms to form a [NaSe6] slightly distorted octahedron. These
octahedra are connected by face-sharing with their neighboring [NaSe6]
octahedron, as illustrated in Fig. 1(c) and Fig. 1(d). The electron
localization function (ELF) provides a visual representation of valence
electron distribution[27], highlighting features that correspond to chemical
bonding characteristics. Fig. 2 shows that areas near the center of Se atoms
and between Ge/Ge atoms exhibit ELF values close to 1.

Table 1: : Crystal data and DFT evaluated energy of predicted and observed Na6Ge2Se6 | Predicted | Observed
---|---|---
Empirical formula |  Na6Ge2Se6 |  Na6Ge2Se6
Crystal system | Trigonal | Monoclinic
Space group | $R\mkern 1.5mu\overline{\mkern-1.5mu3\mkern-1.5mu}\mkern 1.5mu$ | $P2_{1}/c$
Unit cell |  $a=11.349$Å |  $a=8.445$Å
|  $b=11.349$Å |  $b=12.038$Å
|  $c=10.900$Å |  $c=8.316$Å
| $\alpha=90\degree$ | $\alpha=90\degree$
| $\beta=90\degree$ | $\beta=118.877\degree$
| $\gamma=120\degree$ | $\gamma=90\degree$
Volume (Å3) | $V=405.27$ | $740.20$
$Z$ | 1 | 2
Density($\rho$) | 3.075 $\text{g/cm}^{-3}$ | 3.367 $\text{g/cm}^{-3}$
Energy (eV/atom) |  |
GGA-PBE | $-3.598$ | $-3.582$
HSE06 | $-4.268$ | $-4.252$
Table 2: : Bond distance within the [Ge2Se6] block for predicted and observed Na6Ge2Se6 Predicted | Observed
---|---
Ge-Ge | 2.49 Å | Ge-Ge | 2.47 Å
Ge-Se | 2.39 Å | Ge-Se | 2.38 Å

##  IV Elastic properties

Figure 3: 3D representations of the spatial dependence of the Young’s modulus
and linear compressibility for (a) and (b) the predicted Na6Ge2Se6 phase, and
(c) and (d) the observed Na6Ge2Se6 phase. Their cross-sections in the x-, y-,
and z-directions are shown below.

Analyzing the elastic properties offers insights into the mechanical behavior
of compounds, including their stability, ductility, and brittleness. Such
information is especially valuable in industrial and device manufacturing
sectors, enhancing our understanding of the forces acting in solids. To
calculate elastic constants, we employed the energy-strain method[28], as
facilitated by VASPKIT[19] based on DFT calculations with GGA-PBE functional.

For the predicted Na6Ge2Se6 phase, which crystallizes in the trigonal crystal
system, There are five independent elastic constants: $C_{11}$, $C_{33}$,
$C_{44}$, $C_{12}$, $C_{13}$ with the added relation

$C_{66}=(C_{11}-C_{12})/2$ (1)

This contrasts with its observed counterpart, which crystallizes in monoclinic
crystal system with lower symmetry that has thirteen independent elastic
constants: $C_{11}$, $C_{12}$, $C_{13}$, $C_{15}$, $C_{22}$, $C_{23}$,
$C_{25}$, $C_{33}$, $C_{35}$ , $C_{44}$, $C_{46}$ ,$C_{55}$ and $C_{66}$. The
configurations of these constants constitute the elastic matrices for each
respective phase as follows:

$C_{predicted}=\begin{pmatrix}32.06&10.86&13.31&\cdot&\cdot&\cdot\\\
\cdot&32.06&13.31&\cdot&\cdot&\cdot\\\ \cdot&\cdot&21.06&\cdot&\cdot&\cdot\\\
\cdot&\cdot&\cdot&12.09&\cdot&\cdot\\\ \cdot&\cdot&\cdot&\cdot&12.09&\cdot\\\
\cdot&\cdot&\cdot&\cdot&\cdot&10.60\end{pmatrix}$
$C_{observed}=\begin{pmatrix}28.28&10.82&15.82&\cdot&2.45&\cdot\\\
\cdot&32.08&15.21&\cdot&1.55&\cdot\\\ \cdot&\cdot&40.55&\cdot&4.76&\cdot\\\
\cdot&\cdot&\cdot&10.43&\cdot&2.81\\\ \cdot&\cdot&\cdot&\cdot&10.67&\cdot\\\
\cdot&\cdot&\cdot&\cdot&\cdot&10.75\end{pmatrix}$

Table 3: : Calculated mechanical properties of polycrystal for both predicted and observed Na6Ge2Se6 (all in $GPa$) | $B$ | $E$ | $G$ | $\nu$ | $B/G$ | $\zeta$
---|---|---|---|---|---|---
Predicted | 17.543 | 24.407 | 9.623 | 0.268 | 1.82 | 0.59
Observed | 19.770 | 25.731 | 10.027 | 0.283 | 1.97 | 0.65
Figure 4: Electronic band structures (BS) and density of states (DOS) of both
Na6Ge2Se6 compounds. (a) BS and DOS for predicted Na6Ge2Se6 phase. (b) BS and
PDOS for observed Na6Ge2Se6 phase

These elastic constants are pivotal for assessing other elastic properties. A
crystal must satisfy the Born-Huang criteria[29] to be deemed mechanically
stable. For the predicted structure, the necessary and sufficient Born-Huang
criteria include:

$C_{11}>|C_{12}|,C_{44}>0,C_{66}>0$ (2) $2C_{13}^{2}<C_{33}(C_{11}+C_{12})$
(3)

Due to the intricacy of these criteria for the monoclinic system, their
detailed equations are not listed here. Crucially, the calculated elastic
constants for both Na6Ge2Se6 phases fulfill their respective stability
criteria, indicating mechanical stability for both the predicted and observed
phases.

Constants $C_{11}$, $C_{22}$ and $C_{33}$ quantify the response to uniaxial
strain along the three principal axes. The differences between these constants
measure the extent of anisotropy in linear compressibility. From matrix
$C_{predicted}$ and $C_{observed}$, we can see that for predicted phase,
$C_{11}=C_{22}>C_{33}$ which means linear compressibility is isotropic in the
x- and y-directions and greater than in the z-direction. For the observed
phase, $C_{33}>C_{22}>C_{11}$, indicating that the observed Na6Ge2Se6 phase is
characterized by anisotropic linear compressibility. While both phases have
similar resistance to deformation along the x- and y-directions, the observed
phase is significantly stiffer in the z-direction.

On the other hand, $C_{44}$, $C_{55}$ and $C_{66}$ assess the material’s
resistance to shear deformation about these axes. The elastic constant matrix
for the predicted Na6Ge2Se6 phase exhibits $C_{44}=C_{55}$ and slightly
smaller $C_{66}$, indicating its shear modulus is weakly anisotropic. For this
phase, the resistance to shear deformation is isotropic about the x- and
y-directions and greater than that in the z-direction. Comparing both phases,
the observed phase shows slightly higher resistance to shear deformation about
the principal axes.

The off-diagonal terms indicate coupling between different deformation modes.
For instance, $C_{12}$ correlates the stress applied in the x-direction to the
resultant strain in the y-direction. Analysis of the matrices suggests that
both phases have a similar response to deformation in the y-direction when
stress is applied in the x-direction. However, the predicted phase shows
greater shear strain in the z-direction induced by stress in both the x- and
y-directions.

Figure 5: The partial density of states (PDOS) on orbitals and elements for
(a) the predicted Na6Ge2Se6 and (b) the observed Na6Ge2Se6. Figure 6: Phonon
density of states and band structure of both Na6Ge2Se6 compounds. (a)
Predicted new Na6Ge2Se6 phase. (b) Observed Na6Ge2Se6 phase.

Utilizing the independent elastic constants, we can derive key mechanical
properties such as the bulk modulus ($B$), shear modulus ($G$), Young’s
modulus ($E$), and Poisson’s ratio ($\nu$) through the following relations,
with the calculated values presented in Table 3:

$B=\frac{B_{V}+B_{R}}{2}$ (4) $G=\frac{G_{V}+G_{R}}{2}$ (5)
$E=\frac{9BG}{3B+G}$ (6) $\nu=\frac{3B-2G}{2(3B+G)}$ (7)

Here, $B_{V}$, $B_{R}$ and $G_{V}$, $G_{R}$ represent the Voigt[30] and
Reuss[31] values of the bulk modulus and shear modulus respectively.
Furthermore, we evaluated Kleinman’s parameter ($\zeta$) using VASPKIT[19].

The bulk and shear moduli offer insights into a crystal’s mechanical behavior,
indicating its resistance to volumetric and shape deformations, respectively.
The data in the Table 3 reveal that the observed Na6Ge2Se6 phase possesses a
greater resistance to deformation. This is expected as the unit-cell volume of
the predicted phase is greater than that of the observed phase. Pugh’s ratio
($B/G$) serves as a gauge for a material’s ductility or brittleness, with a
threshold value of 1.75 distinguishing between the two behaviors where the
material is claimed to be brittle, if $B/G<1.75$ and classified to be ductile
if $B/G>1.75$[32]. Our findings suggest that both phases exhibit ductility.
The stiffness and thermal shock resistance of these materials are inferred
from Young’s modulus ($E$)[33], indicating superior stiffness in the observed
phase. Conversely, a lower Young’s modulus hints at enhanced thermal shock
resistance[34] for the predicted phase. Poisson’s ratio ($\nu$) sheds light on
the nature of bonding forces within a crystal and aids in evaluating its
mechanical properties, including stability against shear[35]. Values for the
predicted and observed phases are 0.268 and 0.283, respectively, falling
within the central force solids’ expected range of 0.25 to 0.50[36]. This
metric also predicts the material’s ductility or brittleness, classifying
those with $\nu>0.26$ as ductile and those below 0.26 as brittle[37, 34].
Thus, our analysis corroborates both phases’ ductility, aligning with
conclusions drawn from Pugh’s ratio. Lastly, the Kleinman parameter ($\zeta$),
varying between 0 and 1, indicates the dominance of bond stretching or bending
in resisting external stress[38] where the lower value of the Kleinman
parameter $\zeta$ indicates a minimal role of bond bending in resisting
external stress, whereas a higher value suggests a negligible contribution of
bond stretching or contracting to resist externally applied stress. For both
Na6Ge2Se6 phases, a $\zeta>0.5$ indicates a major role for bond bending in its
mechanical strength.

Fig. 3 shows the 3D and 2D representations of Young’s modulus and linear
compressibility for both Na6Ge2Se6 phases. For the predicted phase, the
Young’s modulus and linear compressibility in the xy-plane form a circle, as
shown in Figures 3(e) and (f), suggesting both properties are isotropic within
the xy-plane. However, both properties show significant deviation from a
circular form within the yz- and xz-planes, indicating that Young’s modulus
and linear compressibility are anisotropic overall for the predicted
Na6Ge2Se6. For the observed phase, Figures 3(g) and (h) show that both
properties are anisotropic, with the 2D representation of Young’s modulus in
the xy- and yz-planes and linear compressibility in the xy-plane more closely
resembling a circle. This suggests relative isotropy for both properties
within the corresponding planes.

Figure 7: Thermal properties comparison between predicted and observed
Na6Ge2Se6 phases. (a) Entropy. (b) Heat capacity. (c) Helmholtz free energy in
the temperature range from 905$K$ to 912$K$. (d) Helmholtz free energy in the
temperature range from 0$K$ to 1000$K$ Figure 8: (a)(b)(c)(d)(e)(f) are the
optical properties for predicted Na6Ge2Se6, (g)(h)(i)(j)(k)(l) are the optical
properties for observed Na6Ge2Se6. (a)(g): Absorption coefficient $\alpha$,
(b)(h): Conductivity $\sigma$, (c)(i): Dielectric function $\epsilon$, (d)(j):
Energy-loss function $L$, (e)(k): Reflectivity $R$, (f)(l): Refractive index
$n$.

##  V Electronic Properties

We present the electronic band structure along with the partial and total
density of states (DOS) in Fig. 4. These calculations were performed using
optimized lattice parameters along high-symmetry directions in the first
Brillouin zone. The partial density of states (PDOS) on orbitals and elements
are depicted in Fig. 5. The band structures and DOS for both Na6Ge2Se6 phases
were determined using the HSE06 hybrid exchange-correlation functional[17] to
ensure enhanced accuracy[39]. The energy reference point is set at the top of
the valence band. The band structure analysis reveals that the predicted
Na6Ge2Se6 phase possesses an indirect band gap, with the conduction band
minimum (CBM) located at the $X$-point and the valence band maximum (VBM) at
the $Z$-point, resulting in a band gap of 2.972 eV. In contrast, the observed
Na6Ge2Se6 phase exhibits a direct band gap of 2.932 eV, with both the CBM and
VBM situated at the $Y_{2}$-point.

From Fig. 5, we observe that both phases share similar orbital and element
contributions to the total DOS. Regions close to the Fermi level are
predominantly influenced by contributions from Se-p and Ge-s orbitals, while
Na’s involvement near the Fermi level is minimal. This observation suggests
that the band gaps in both Na6Ge2Se6 phases are largely attributable to the
[Ge2Se6] dimers. Within the valence band, ranging from -4 to 0 eV, the primary
contributions come from Se-p and Ge-p orbitals in both phases. This trend
extends to the lower regions of the conduction band, from 4 to 8 eV, where
Na-p displays a slight contribution to the total DOS in this region.

##  VI Phonon and thermal properties

Investigating phonon properties is fundamental for understanding crystalline
materials, as it provides insights into structural stability, phase
transitions, and how vibrations influence their thermal behavior. In our study
of the phonon dispersion and phonon band structures for both Na6Ge2Se6 phases,
we employed the Phonopy package[20, 21] to create a series of $2\times 2\times
2$ supercell structures with various displacements. We then conducted force
calculations using VASP, employing the GGA-PBE exchange-correlation
functional. The phonon frequencies and eigenvectors were determined from the
dynamical matrices, which were calculated based on the force constants derived
using Phonopy.

The phonon dispersion relations along high-symmetry paths for both phases are
depicted in Fig. 6. The unit cell of the predicted phase comprises 14 atoms,
resulting in 42 phonon branches, while the observed phase has 28 atoms,
resulting in 82 branches, which include three acoustic and the rest optical
branches. The frequency spectrum of these modes spans from 0 to 10 THz,
without any discernible gap between the acoustic and optical modes for either
phase. In the lower frequency domain, below 7 THz, Na and Se atoms
predominantly contribute to the optical branches. In the higher frequency
range, the contributions mainly stem from Ge and Se. This distribution
underscores the strong bonding present within the [Ge2Se6] dimers, reflecting
their structural integrity across both phases of Na6Ge2Se6. The lack of
negative frequency branches within the dispersion plots affirms the dynamic
stability of these phases at zero pressure.

Table 4: : Calculated $v_{t}$, $v_{l}$, $v_{m}$ and $\theta_{D}$ for both predicted and observed Na6Ge2Se6 |  $v_{t}($m/s$)$ |  $v_{l}($m/s$)$ |  $v_{m}($m/s$)$ |  $\theta_{D}$ ($K$)
---|---|---|---|---
Predicted | 1761.41 | 3129.30 | 1959.71 | 190.0
Observed | 1725.51 | 3136.88 | 1923.28 | 191.7

The Debye temperature ($\theta_{D}$) represents the temperature corresponding
to the highest energy vibrational mode in a solid, derived from the
equation[40]:

$\theta_{D}=\frac{h}{k_{B}}\left[\frac{3n}{4{\pi}V_{0}}\right]^{\frac{1}{3}}v_{m}$
(8)

Here, $h$ is the Planck’s constant, $k_{B}$ is the Boltzmann’s constant, $n$
is the number of atoms in the unit cell, $V_{0}$ is the unit cell’s
equilibrium volume, and $v_{m}$ is the material’s average speed of sound. The
average speed of sound $v_{m}$ can be determined from the material’s mass
density ($\rho$), along with its bulk ($B$) and shear modulus ($G$), through
the equations for longitudinal ($v_{l}$) and transverse ($v_{t}$) sound
speeds:

$v_{m}=\left[\frac{1}{3}\left(\frac{3}{v_{t}^{3}}+\frac{1}{v_{l}^{3}}\right)\right]^{\frac{1}{3}}$
(9) $v_{t}=\sqrt{\frac{G}{\rho}}$ (10) $v_{l}=\sqrt{\frac{3B+4G}{3\rho}}$ (11)

The calculated values for $v_{t}$, $v_{l}$, $v_{m}$ and $\theta_{D}$ are
provided in Table 4. These values indicate that the predicted Na6Ge2Se6 phase
exhibits a slower longitudinal speed of sound but a higher transverse one than
observed phase, although the values differ only slightly. The Debye
temperature of the predicted phase is approximately 0.89% lower than that of
its counterpart.

Phonon calculations offer insights into additional thermal properties such as
entropy ($S$), constant-volume heat capacity ($C_{v}$), and Helmholtz free
energy ($F$), which are depicted in Fig. 7. The heat capacity curves, as one
can see from Fig. 7(b), are almost indistinguishable as one expects from
similar speeds of sound and Debye temperatures. They show standard Debye
theory behavior with the increasing temperature: quantum $~{}T^{3}$ at low
temperature, saturating to the constant classical Dulong-Petit limit[41] above
Debye temperature.

The calculation of the Helmholtz free energy ($F$) within the harmonic
approximation[42] facilitates an evaluation of the thermal contributions to
the relative stability of two phases, as described by the following equation:

$\begin{split}F=&E_{total}+\frac{1}{2}\sum_{q\nu}\hbar\omega(q\nu)\\\
&+k_{B}T\sum_{q\nu}\ln{\left[1-exp(-\hbar\omega(q\nu)/k_{B}T\right]}\end{split}$
(12)

Here, $E_{total}$ denotes the total energy of the crystal, available in Table
1. The summed terms represent the Helmholtz free energy attributable to
phonons[43], with the initial sum reflecting the zero-point energy (ZPE) that
is independent of temperature. The predicted phase’s ZPE is approximately 1
meV/atom higher than that of the observed phase. The subsequent sum accounts
for the temperature-dependent term referring to the thermally induced
occupation of phonon modes. The Helmholtz free energies, plotted in Fig. 7(d)
for temperatures ranging from 0 to 1000 K, with a closer look at the range of
905 to 912 K in Fig. 7(c), illustrate that the predicted phase possesses lower
free energy from 0 to approximately 907 K. Since its rate of decrease with
temperature is more gradual than that of the observed phase, the free energy
of the observed phase becomes lower when the temperature exceeds 907 K. This
suggests that the predicted phase is more thermodynamically stable than the
observed phase in the temperature range from 0 to 907 K, indicating that this
predicted phase is very likely to be experimentally observable.

##  VII Optical properties

A material’s optical behavior is characterized by several energy/frequency-
dependent parameters, such as the dielectric function $\epsilon(\omega)$,
absorption coefficient $\alpha(\omega)$, conductivity $\sigma(\omega)$,
energy-loss function $L(\omega)$, reflectivity $R(\omega)$ and refractive
index $n(\omega)$. The outcomes of these calculated properties are depicted in
Fig. 8, based on formulas integrated within VASPKIT[19].

$\epsilon(\omega)=\epsilon_{1}(\omega)+i\epsilon_{2}(\omega)$ (13)
$\alpha(\omega)=\frac{\sqrt{2}\omega}{c}\left[\sqrt{\epsilon_{1}^{2}+\epsilon_{2}^{2}}-\epsilon_{1}\right]^{\frac{1}{2}}$
(14) $L(\omega)=\frac{\epsilon_{2}}{\epsilon_{1}^{2}+\epsilon_{2}^{2}}$ (15)
$n(\omega)=\left[\frac{\sqrt{\epsilon_{1}^{2}+\epsilon_{2}^{2}}+\epsilon_{1}}{2}\right]^{\frac{1}{2}}$
(16) $R(\omega)=\frac{(n-1)^{2}+k^{2}}{(n+1)^{2}+k^{2}}$ (17)

To examine the anisotropic characteristics, we calculated the optical
parameters along the three principal axes and presented the optical spectra
for incident photon energies up to 25 eV. For the predicted phase, these
parameters are isotropic in the x- and y-directions. Therefore, Fig. 8
displays the optical parameters and the corresponding derived optical
properties plotted only in the y- and z-directions for the predicted phase.

The complex dielectric function is essential for describing a material’s
optical properties, as it serves as the foundation from which other energy-
dependent optical constants are derived. We illustrate the dielectric
functions for both predicted and observed Na6Ge2Se6 phases in Fig. 8(c) and
(i), respectively. The real part of the dielectric function ($\epsilon_{1}$)
initially increases, reaches a peak, then sharply decreases, eventually
dropping below zero. The imaginary part ($\epsilon_{2}$), indicative of
dielectric loss, closely aligns with the optical absorption coefficient. From
the figure we can see $\epsilon_{1}$ and $\epsilon_{2}$ peak in different
directions. For the predicted phase, $\epsilon_{1}$ peaks in z-direction, and
$\epsilon_{2}$ peaks in y-direction. This contrasts with the observed phase,
where both $\epsilon_{1}$ and $\epsilon_{2}$ peak in the same y-direction. The
material’s electronic characteristics can be inferred from the absorption
coefficient ($\alpha(\omega)$), depicted in Fig. 8(a) and (g). Both phases
begin to absorb at photon energies of approximately 3.1 eV and 2.9 eV
respectively, indicating their optical band gaps. The absorption coefficients
in three different directions follow a similar trend, with peaks at 7.4 eV for
the predicted phase and 9.5 eV for the observed phase, both within the
ultraviolet spectrum. The material’s reflection and absorption properties,
related to its loss function, are illustrated in Fig. 8(e)(k) and (d)(j). The
edge trailing of the material’s reflection and absorption spectra matches the
peak in the loss function, showing a sharp increase around 12 eV for both
phases. This spike indicates enhanced absorption and reflection of
electromagnetic waves in the ultraviolet region, with the predicted phase
exhibiting much notable anisotropy along y- and z-directions with the maximum
appearing in y-direction for both phase. Optical conductivity which represents
electrical conductivity over specific photon energy ranges is shown in Fig.
8(b) and (h). Conductivity sharply increases from zero at photon energies of
3.1 eV and 2.9 eV for the predicted and observed Na6Ge2Se6 phases
respectively, mirroring the optical band gap observed in the absorption curves
in Fig. 8(a)(g). In summary, the predicted and observed phases display similar
optical properties, with the predicted phase exhibiting a higher degree of
optical anisotropy in the y- and z-directions, while being isotropic in the x-
and y-directions.

##  VIII Conclusion

A comprehensive comparison was conducted between two Na6Ge2Se6 phases: a
computationally predicted phase and an experimentally observed phase. This
comparison spans their structural, elastic, electronic, phonon, thermal, and
optical properties, employing first-principles methods.

The predicted Na6Ge2Se6 phase is the energetically favored structure at zero
pressure and temperature based on first-principles calculations. Both phases
feature the identical [Ge2Se6] dimer within their crystal structures, with
significant differences observed in the bonding environment between Se and Na
atoms (five Na atoms bond with one Se atom in the predicted phase, whereas six
Na atoms bond with one Se atom in the observed phase).

Regarding elastic properties, both phases meet the criteria for mechanical
stability and share similar characteristics. These include their resistance to
volumetric and shape deformations, ductility, nature of bonding forces
(central force), and bond bending as a major role in resisting external
stress. The differences arise from their anisotropic characteristics. While
the predicted phase is isotropic in the x- and y-directions, it exhibits
greater anisotropy in the y- and z-directions compared to the observed phase.

Electronic structure analyses reveal a closely matched energy band gap between
the two phases, primarily contributed by the [Ge2Se6] dimers, with only a 1.3%
difference. However, the electronic band structure indicates that the
predicted phase exhibits an indirect band gap, while the observed phase has a
direct band gap.

The optical behavior between the phases is similarly aligned, with both
exhibiting a comparable optical band gap (3.1 eV for the predicted phase and
2.9 eV for the observed phase) and enhanced absorption and reflection of
electromagnetic waves in the ultraviolet region. Although the predicted phase
is isotropic in the x- and y-directions, it displays a greater degree of
optical anisotropy in the y- and z-directions, aligning with the anisotropic
nature identified in the elastic properties.

Phonon dispersion analyses affirm the dynamic stability of both phases.
Further calculations of thermal properties indicate that the predicted phase
is more thermodynamically stable in the temperature range from 0 to 907 K,
suggesting a high likelihood of experimental observation.

##  IX Acknowledgements

The authors thank NSF, USA (grant No. DMR1809128) for the funding of this
project. The authors also acknowledge the usage of the HPC cluster ”Foundry”
at Missouri S&T funded by NSF award OAC-1919789.

##  X data avaiability statement

The data that support the findings of this study are available upon request.

## References

  * Jia  _et al._ [2019] N. Jia, M. Zhang, B. Li, C. Li, Y. Liu, Y. Zhang, T. Yu, Y. Liu, D. Cui, and X. Tao, Electrochimica Acta 320, 134562 (2019).
  * Zhang  _et al._ [2022] J. Zhang, T. Li, B. Li, S. Zhang, Y. Dou, Q. Yuan, Y. Wu, and J. Han, ACS Sustainable Chemistry & Engineering 10, 10666 (2022).
  * Sun  _et al._ [2018] W.-C. Sun, S. U. Rahayu, and M.-W. Lee, IEEE Journal of Photovoltaics 8, 1011 (2018).
  * Lin  _et al._ [2016] H. Lin, G. Tan, J.-N. Shen, S. Hao, L.-M. Wu, N. Calta, C. Malliakas, S. Wang, C. Uher, C. Wolverton, _et al._ , Angewandte Chemie International Edition 55, 11431 (2016).
  * Pei  _et al._ [2016] Y. Pei, C. Chang, Z. Wang, M. Yin, M. Wu, G. Tan, H. Wu, Y. Chen, L. Zheng, S. Gong, _et al._ , Journal of the American Chemical Society 138, 16364 (2016).
  * Ma  _et al._ [2020] N. Ma, Y.-Y. Li, L. Chen, and L.-M. Wu, Journal of the American Chemical Society 142, 5293 (2020).
  * Ma  _et al._ [2019] N. Ma, F. Jia, L. Xiong, L. Chen, Y.-Y. Li, and L.-M. Wu, Inorganic Chemistry 58, 1371 (2019).
  * Balijapelly  _et al._ [2022] S. Balijapelly, A. J. Craig, J. B. Cho, J. I. Jang, K. Ghosh, J. A. Aitken, A. V. Chernatynskiy, and A. Choudhury, Journal of Alloys and Compounds 900, 163392 (2022).
  * Poudeu  _et al._ [2006] P. F. Poudeu, J. D’Angelo, A. D. Downey, J. L. Short, T. P. Hogan, and M. G. Kanatzidis, Angewandte Chemie International Edition 45, 3835 (2006).
  * Iyer  _et al._ [2021] A. K. Iyer, J. B. Cho, H. R. Byun, M. J. Waters, S. Hao, B. M. Oxley, V. Gopalan, C. Wolverton, J. M. Rondinelli, J. I. Jang, _et al._ , Journal of the American Chemical Society 143, 18204 (2021).
  * [11] A. C. Qi Zhang, Amitava Choudhury, submitted for publication to J. Phys. Cond. Matter. When it will be published we can update the citation. .
  * Eisenmann  _et al._ [1985] B. Eisenmann, J. Hansa, _et al._ , Materials research bulletin 20, 1339 (1985).
  * Kresse and Furthmüller [1996a] G. Kresse and J. Furthmüller, Computational materials science 6, 15 (1996a).
  * Kresse and Furthmüller [1996b] G. Kresse and J. Furthmüller, Physical review B 54, 11169 (1996b).
  * Kohn and Sham [1965] W. Kohn and L. J. Sham, Physical review 140, A1133 (1965).
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Physical review letters 77, 3865 (1996).
  * Krukau  _et al._ [2006] A. V. Krukau, O. A. Vydrov, A. F. Izmaylov, and G. E. Scuseria, The Journal of chemical physics 125 (2006).
  * Monkhorst and Pack [1976] H. J. Monkhorst and J. D. Pack, Physical review B 13, 5188 (1976).
  * Wang  _et al._ [2021] V. Wang, N. Xu, J.-C. Liu, G. Tang, and W.-T. Geng, Computer Physics Communications 267, 108033 (2021).
  * Togo  _et al._ [2023] A. Togo, L. Chaput, T. Tadano, and I. Tanaka, J. Phys. Condens. Matter 35, 353001 (2023).
  * Togo [2023] A. Togo, J. Phys. Soc. Jpn. 92, 012001 (2023).
  * Ong  _et al._ [2013] S. P. Ong, W. D. Richards, A. Jain, G. Hautier, M. Kocher, S. Cholia, D. Gunter, V. L. Chevrier, K. A. Persson, and G. Ceder, Computational Materials Science 68, 314 (2013).
  * Joergens and Mewis [2004] S. Joergens and A. Mewis, Zeitschrift für anorganische und allgemeine Chemie 630, 51 (2004).
  * Wu  _et al._ [2015] K. Wu, Z. Yang, and S. Pan, Inorganic Chemistry 54, 10108 (2015).
  * Eisenmann  _et al._ [1984] B. Eisenmann, E. Kieselbach, H. Schäfer, and H. Schrod, Zeitschrift fuer Anorganische und Allgemeine Chemie (1950) 516 (1984).
  * Schlirf and Deiseroth [2001] J. Schlirf and H. J. Deiseroth, Zeitschrift für Kristallographie - New Crystal Structures 216, 27 (2001).
  * Becke and Edgecombe [1990] A. D. Becke and K. E. Edgecombe, The Journal of chemical physics 92, 5397 (1990).
  * Le Page and Saxe [2001] Y. Le Page and P. Saxe, Physical Review B 63, 174103 (2001).
  * Mouhat and Coudert [2014] F. Mouhat and F.-X. Coudert, Physical review B 90, 224104 (2014).
  * Voigt [1910] W. Voigt,  _Lehrbuch der kristallphysik:(mit ausschluss der kristalloptik)_ , Vol. 34 (BG Teubner, 1910).
  * Reuß [1929] A. Reuß, ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik 9, 49 (1929).
  * Pugh [1954] S. Pugh, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 45, 823 (1954).
  * Kittel and McEuen [2018] C. Kittel and P. McEuen,  _Introduction to solid state physics_ (John Wiley & Sons, 2018).
  * Karim  _et al._ [2018] A. T. Karim, M. Hadi, M. Alam, F. Parvin, S. Naqib, and A. Islam, Journal of Physics and Chemistry of Solids 117, 139 (2018).
  * Ravindran  _et al._ [1998] P. Ravindran, L. Fast, P. A. Korzhavyi, B. Johansson, J. Wills, and O. Eriksson, Journal of Applied Physics 84, 4891 (1998).
  * Anderson and Demarest Jr [1971] O. L. Anderson and H. H. Demarest Jr, Journal of Geophysical Research 76, 1349 (1971).
  * Vaitheeswaran  _et al._ [2007] G. Vaitheeswaran, V. Kanchana, A. Svane, and A. Delin, Journal of Physics: Condensed Matter 19, 326214 (2007).
  * Kleinman [1962] L. Kleinman, Physical Review 128, 2614 (1962).
  * Heyd  _et al._ [2003] J. Heyd, G. E. Scuseria, and M. Ernzerhof, The Journal of chemical physics 118, 8207 (2003).
  * Anderson [1963] O. L. Anderson, Journal of Physics and Chemistry of Solids 24, 909 (1963).
  * Fitzgerel and Verhoek [1960] R. K. Fitzgerel and F. H. Verhoek, The law of dulong and petit (1960).
  * Fultz [2010] B. Fultz, Progress in Materials Science 55, 247 (2010).
  * Togo and Tanaka [2015] A. Togo and I. Tanaka, Scripta Materialia 108, 1 (2015).


# ViLaM: A Vision-Language Model with Enhanced Visual Grounding and
Generalization Capability

Xiaoyu Yang 1,2, Lijian Xu 2,3 (🖂) , Hongsheng Li 3,4& Shaoting Zhang 2
1 College of Electronics and Information Engineering, Tongji University,
Shanghai
2 Shanghai Artificial Intelligence Laboratory, Shanghai
3 Centre for Perceptual and Interactive Intelligence, the Chinese University
of Hong Kong, Hong Kong
4 Department of Electronic Engineering, the Chinese University of Hong Kong,
Hong Kong

###### Abstract

Vision-language models have revolutionized human-computer interaction and
shown significant progress in multi-modal tasks. However, applying these
models to complex visual tasks like medical image analysis remains
challenging. In this study, we propose ViLaM, a unified Vision-Language
transformer model that integrates instruction tuning predicated on a large
language model. This approach enables us to optimally utilize the knowledge
and reasoning capacities of large pre-trained language models for an array of
tasks encompassing both language and vision. We employ frozen pre-trained
encoders to encode and align both image and text features, enabling ViLaM to
handle a variety of visual tasks following textual instructions. Besides,
we’ve designed cycle training for referring expressions to address the need
for high-quality, paired referring expression datasets for training large
models in terms of both quantity and quality. We evaluated ViLaM’s exceptional
performance on public general datasets and further confirmed its
generalizability on medical datasets. Importantly, we’ve observed the model’s
impressive zero-shot learning ability, indicating the potential future
application of ViLaM in the medical field.

_K_ eywords Visual Grounding $\cdot$ Vision-language Models $\cdot$ Generalist
Models

##  1 Introduction

The world-beating development of large language models and large visual models
has spawned thinking about the generalist vision-language models. How to
activate orientation in a vision-language model, finding the location based on
the provided referring expression and having smooth human-computer
interaction, is of great interest to the community.

Based on vast amounts of text data, Large Language Models (LLMs) have acquired
the ability to generate human-like answers and can solve a variety of tasks,
such as language translation, question answering, and text generation. This
advancement provides a new paradigm for human-computer interaction. However,
unlike the cross-domain portability observed in the language, “pure” visual
large models trained on natural images often struggle to achieve expected
performance when faced with images that exhibit significant distribution
shifts. Furthermore, visual large models require prompts in the form of
points, boxes, or masks, and the precision of these prompts significantly
impacts the performance. Generating such prompts requires expert prior
knowledge, thus raising the bar for interaction between ordinary individuals
and models.

Building upon LLMs, multi-modality large models like BLIP2 [1] and MiniGPT4
[2] utilize pre-trained image encoder and text encoder then align vision-
language features with simple linear layers. The stronger generalizable
language features are leveraged to guide the extraction of visual features.
These models demonstrate impressive joint understanding capabilities of
language and images, allowing users to give instructions in natural language
to perform specific tasks. Models that are primarily developed for language-
specific tasks often struggle with image processing tasks like detection.
Current methodologies encounter inconsistencies in input, output, and training
procedures when attempting to bridge the gap between visual tasks and language
tasks, which impedes effective integration. Moreover, the exclusive reliance
on textual outputs can limit the model’s answer capabilities and
interpretability.

In this study, we introduce ViLaM, a unified transformer model designed
specifically for multi-modality tasks. ViLaM incorporates customized
instruction tuning to fully harness the visual capabilities of Language-Only
Pre-trained Models. To achieve this, we utilize frozen pre-trained visual
encoders and LLMs, which encode and align features from both images and text.
This enables ViLaM to effectively handle various language and vision tasks
based on instructions, producing diverse and intricate output results.
Leveraging the advantages of pre-trained language models and the mutual
guidance between tasks, ViLaM excels in continuous question-answering and can
provide visual explanations of answers during conversations. This capability
is particularly valuable in safety-critical domains such as medical diagnosis.
Cycle training of referring expressions is a method designed to address the
challenges of scarcity and quality in paired referring expression datasets,
which are critical for training large models.

To summarize, our contributions are three-fold: (1) We incorporate the large
language models into multi-modality systems, utilizing instruction tuning to
maximize the use of the knowledge and inferential abilities of these pre-
trained language models for intricate visual grounding tasks. (2) We design
the cycle training of referring expressions, satisfying the requirements of
paired referring expression datasets for the large model training, both in
quantity and quality. (3) We assess the superior performance of ViLaM in
public general datasets, and demonstrate its generalization in medical
datasets. Besides, we observe the excellent zero-shot capability of the
proposed method, suggesting the potential application of ViLaM in the medical
field. Our code is public at https://github.com/AnonymGiant/ViLaM

##  2 Related Work

###  2.1 LLMs and Multi-modality Pre-training

Large Language Models (LLMs) have recently significantly impacted the field of
natural language processing. Through alignment techniques such as supervised
learning and reinforcement learning with human feedback, LLMs can effectively
generalize to perform a wide range of tasks, even with limited training data.
A remarkable application of LLM is ChatGPT, which presents an amazing ability
to interact with humans. OpenAI’s ChatGPT and GPT4 are prime examples of the
impact that AI can have, and there have been extensive open-source efforts to
replicate their success, such as OPT [3], BLOOM [4], PALM [5], LLaMA [6].

Multi-modality models have further promoted the development of the generalist
model. CLIP [7] was introduced to separately extract features from different
encoders and combine them using contrastive learning. Building on CLIP, GLIP
[8] was developed to learn object-level, language-aware, and semantic-rich
visual representations, unifying object detection and phrase grounding for
pre-training. Different from the contrastive method, Flamingo [9] aligned a
pre-trained vision encoder and language model using gated cross-attention,
demonstrated impressive few-shot learning capabilities. BLIP2 [1] was
subsequently introduced, and it employed a Flan-T5 [10] along with a Q-Former
to effectively align visual features with the language model. GPT-4V [11, 12]
has recently shown unprecedented ability in understanding and processing an
arbitrary mix of input images and texts. On the other hand, preliminary
experiments show that visual grounding accuracy is still limited in the
comprehensive scene, like the medical field.

Figure 1: The workflow of our methodology. We design the task of cycle
referring expression to activate orientation of the generalist vision-language
model, which combines the referring expression generation and referring
expression comprehension. Coordinates will cycle through two subtasks. To
reduce the calculation scale and expand the dataset, only referring expression
comprehension participates in training. The pre-trained visual model and large
language model are frozen in the training.

###  2.2 Visual Grounding

Two-stage and One-stage Methods Early pioneers typically used a two-stage
approach to tackle visual grounding tasks. The initial step involves
extracting interest regions, which are subsequently prioritized based on their
similarity scores with the language query. VILLA [13] introduces large-scale
adversarial training to vision-language representation learning, adding
adversarial perturbations in the embedding space of multi-modalities. CM-Att-
Erase [14] devises an erasing approach guided by cross-modal attention,
selectively removing the prominent information from either the textual or
visual domains and generating training samples. Another line of work advocates
a one-stage pipeline based on dense anchors. MDETR [15] is an end-to-end
modulated detector derived from the detection framework, and performs object
detection in conjunction with natural language understanding. TransVG [16]
uses a simple stack of transformer encoders to perform the multi-modality
fusion and reasoning for the disease localization task, instead of leveraging
complex manually designed fusion modules. Other Transformer models like SeqTR
[17] and VGTR [18] in Vision-Language Tasks are subsequently proposed for the
visual grounding task and achieved satisfactory performance.

Generalist Model Recently, the potential of generalist models has been
increasingly explored, garnering considerable attention from the research
community. Among these, OFA [19] integrates a diverse set of cross-modal and
uni-modal tasks within a simple sequence-to-sequence learning framework. It
adheres to instruction-based learning in both pre-training and fine-tuning
stages, negating the need for additional task-specific layers for downstream
tasks. Besides, mPLUG-2 [20] presents a multi-module composition network that
utilizes shared universal modules for modality collaboration and separates
distinct modality modules to address modality entanglement.

##  3 Methodology

In this section, we first introduce the architecture of our vision-language
model with the LLM. Then, the robust activation of coordinates in the
generalist vision-language model is explored. On the basis of outputting
object coordinates robustly, the cycle training of referring expressions is
presented to reinforce the link between coordinates and referring expressions
of objects.

###  3.1 Architecture

Image encoder:  With an input image $x_{i}\in\mathbb{R}^{H\times W}$, visual
features are extracted by image encoder and further projected to feature
dimension:

$v_{i}=P_{img}(E_{img}(x_{i}))\in\mathbb{R}^{(h_{f}\times w_{f})\times d}$ (1)

where $h_{f}$ and $w_{f}$ are the output size of visual features, and $d$
represents the feature dimension. $E_{img}$ can be any common visual backbones
and we use Vit-Large in our case. Then by using $P_{img}$, which is composed
of two linear layers, visual features are projected to feature dimension.

Language encoder:  With any processed input instruction sequence $t_{i}$, text
features are extracted by language encoder:

$l_{i}=E_{txt}(t_{i})\in\mathbb{R}^{n_{t}\times d}$ (2)

where $n_{t}$ is the number of input tokens and $d$ represents the feature
dimension. In our case, Bert [21] is used as the language encoder.

Multi-modality module:  This module follows an encoder-decoder architecture
format. Given the input visual features $v_{i}$ and text features $l_{i}$, we
first generate fused multi-modality representations by combining the image and
text embeddings. These fused features serve as the keys and values in the
cross-attention blocks in the decoder. By conditioning on the partial sequence
$y_{i,<j}$ predicted so far, the decoder recursively makes predictions for the
token at position $j$, effectively generating aligned descriptions across
modalities.

$y_{i,j}=D_{mm}(E_{mm}(concat(v_{i},l_{i})),y_{i,<j})\in\mathbb{R}^{1\times
d}$ (3)

###  3.2 Activation of Coordinates

Leveraging its emergent capabilities, the generalist vision-language model
exhibits remarkable versatility in scenarios and tasks related to orientation.
Initially, the task of activating coordinates is transmuted into conventional
object detection within the framework of the generalist vision-language model,
devoid of referring expressions. Subsequently, extensive object detection
datasets, such as COCO 2014 and COCO 2017, are integrated into the activation
procedure. The considerable quantity of data facilitates the generalist model
in producing coordinates with enhanced robustness and precision.

Subsequently, to reconcile the divergence between semantic and linguistic
coordinates, we establish a linguistic representation of coordinates within
the large language model: $[x_{1},y_{1},x_{2},y_{2}]$. Here, $x$ denotes
horizontal coordinates, while $y$ signifies longitude. The pair
$(x_{1},y_{1})$ designates the upper-left point, and $(x_{2},y_{2})$
corresponds to the lower-right point. All coordinates adopt relative
positions, normalization to 1000, and rounding.

We employ the captioning task to prompt our model to output coordinates that
express orientation, owing to its proven effectiveness in capturing
information in knowledge-intensive scenarios [22]. During training, we utilize
the captioning format as follows:

_find the $<$object$>$ in the region of $[x_{1},y_{1},x_{2},y_{2}]$._

Due to the absence of referring expressions, more than one coordinate may
correspond to multiple objects in the image. The captioning form has more
feasibility and practicability for the generalist vision-language model to
establish links between orientation and linguistic coordinates robustly,
without inferring in the prompt.

In the training of captioning, the generalist model is expected to output
image captions containing object-related coordinates and compute the loss. For
activation of coordinates, we optimize using cross-entropy loss:

$\mathcal{L}=-\sum_{i=1}^{n}\sum_{j=1}^{|y|}\log
P_{\theta}(y_{i,j}|y_{i,<j},x_{i},t_{i})$ (4)

where $n$ is the batch size, $\theta$ represents the model parameters, $x_{i}$
represents the input image, $t_{i}$ stands for the input instruction, and
$y_{i,j}$ denotes the output token at position $j$ for the $ith$ sample at
each batch. We follow the training strategy of BLIP2, which only trains the
alignment layer and freezes the pre-trained visual model and large language
model. To enhance the quality of generation during inference, we employ
various decoding techniques, such as beam search.

(a) Multiple Persons

(b) Overlapping Persons

(c) Incomplete Objects

(d) Animals of Different Sizes

(e) Containers with Different Dishes

(f) Aircraft with Special Markings

Figure 2: Results of visual grounding in RefCOCO with the OPT6.7b as the large
language model. (a)-(c) are from testA split containing only people, while
(d)-(f) from testB consisting of only non-people. We display typical cases of
referring expressions, especially with common indications of orientation,
size, color, attachment and markings. The referring expressions of the object
are presented in the text box with two coordinates, where R (red) denotes
grounding truth and Y (yellow) symbolizes the prediction. The red and yellow
bounding boxes are also depicted in the image, respectively.

###  3.3 Cycle Training of Referring Expressions

Obtaining the ability to output coordinates in the generalist vision-language
model, we design the cycle training of referring expressions to incorporate
referring expressions with coordinates, as shown in Fig.1. Inspired by Cycle-
GAN [23], the cycle training of referring expressions is expected to learn
alignment relationships between two domains $X$ and $Y$ given training samples
$\\{x_{i}\\}_{i=1}^{N}\in X$ and $\\{y_{j}\\}_{j=1}^{M}\in Y$ in the large
language model, where $X$ denotes the visual grounding features and $Y$
represents the linguistic referring expression.

The cycle referring expression consists of two subtasks: referring expression
generation (REG) represented as $G:X\rightarrow Y$ and referring expression
comprehension (REC) formulated as $F:Y\rightarrow X$. The form of visual
question answering (VQA) is suitable for the referring expression task with
inferring and understanding. By posing questions with coordinates, REG detects
the object and generates related detailed referring expressions. The generated
referring description of the coordinate-related object is expected to be
unique in the image, as shown in the following:

_Question: What is in the region of [122, 366, 393, 898] ?_

_Answer: A man on a skateboard wearing a plaid shirt in the region of [122,
366, 393, 898]._

Depending on the specific and elaborate description of the object, we cycle
through questions about the coordinates of the object via REC. By asking for
the coordinates of the object with characteristics in REC, the result is
expected to be similar to the coordinates entered by the REG as follows:

_Question: where is a man on a skateboard wearing a plaid shirt in the image?_

_Answer: In the region of [150, 366, 393, 898]._

With the above form of VQA, the bounding box and referring expression are
cycled training in the large language model. We argue that the learned
alignment relationships should be cycle-consistent: for every bounding box $x$
belonging to domain $X$, the cycle-referring expression should possess the
capability to restore $x$ to its nearby coordinates, indicated $x\rightarrow
G(x)\rightarrow F(G(x))\approx x$. Similarly, for each referring expression
$y$ from domain $Y$, $y$ should be reduced to its original form, i.e.
$y\rightarrow F(y)\rightarrow G(F(y))\approx y$. The cycle referring
expression can be incentivized by the consistency loss:

$\mathcal{L}_{\mathrm{cyc}}(G,F)=\mathbb{E}_{x\sim
p(x)}\left[\|F(G(x))-x\|\right]+\mathbb{E}_{y\sim
p(y)}\left[\|G(F(y))-y\|\right].$ (5)

Besides, image-text loss is applied to both mapping functions, including
image-text contrastive loss (ITC), image-grounded text generation loss (ITG)
and image-text matching loss (ITM), as denoted as
$\mathcal{L}_{\mathrm{align}}$. For the mapping of REG $G:X\rightarrow Y$, we
express the image-text loss as: $\mathcal{L}_{\mathrm{align}}(G,X,Y)$. For the
REC $F:Y\rightarrow X$, the image-text loss is represented as
$\mathcal{L}_{\mathrm{align}}(F,X,Y)$. Consequently, the full criterion is
presented as follows:

$\mathcal{L}(G,F,X,Y)=\mathcal{L}_{\mathrm{align}}(G,X,Y)+\mathcal{L}_{\mathrm{align}}(F,X,Y)+\mathcal{L}_{\mathrm{cyc}}(G,F)$
(6)

Benefiting from the cycle training, more normal object detection datasets
without referring expressions could be expanded for the visual grounding
training, such as COCO 2014 and COCO 2017. REG generates referring expressions
from bounding boxes in the object detection dataset, and REC then inferences
the bounding box from the generated referring expression. In the training, the
questions of VQA are involved in the generalist vision-language model as the
prompt of the large language model. The question is exclusive from the
computation of the loss. Model updates are driven by the loss of answers in
VQA.

##  4 Experiments

###  4.1 Experimental Settings

Datasets RefCOCO [24], RefCOCO+ [24], and RefCOCOg [25] are three visual
grounding datasets that utilize images sourced from MSCOCO [26]. In line with
previous approaches, we adopt the train/validation/testA/testB split for both
RefCOCO and RefCOCO+ datasets, where testA and testB sets contain only people
and only non-people respectively. The split of RefCOCOg-umd [27] on RefCOCOg
refers to the splits as the val-u, and test-u<EMAIL_ADDRESS><EMAIL_ADDRESS>is used
to measure the performance of the visual grounding task, which is right if the
IoU between the grounding-truth box and the predicted bounding box is larger
than 0.5.

For applications in the medical field, we utilize public datasets for foreign
object detection and disease identification in chest X-ray images. The Object-
CXR [28] dataset is designed for the automatic detection of foreign objects in
chest X-rays. It consists of 5,000 frontal chest X-ray images with foreign
objects and 5,000 images without foreign objects. These X-ray images were
captured and collected from approximately 300 township hospitals in China. The
ChestXray14 dataset [29] contains 112,120 chest X-ray images with labels for
14 common diseases. Among these, 984 images feature eight key findings with
hand-labelled bounding boxes. The RSNA Pneumonia dataset [30] is a binary
classification chest X-ray dataset consisting of 26,683 images. Each
radiograph is categorized as either pneumonia or normal. The TBX11K dataset
[31] is a large collection comprising 11,000 chest X-ray images, each with
corresponding bounding box annotations for tuberculosis areas. Take note that
the ChestXray14 dataset does not provide an official distribution ratio for
training/validation/test in the disease localization task, and the Object-CXR,
RSNA Pneumonia, and TBX11K lack ground truth bounding boxes for their test
sets. Consequently, we arbitrarily divide the official training sets of these
datasets into training/validation/test sets at a ratio of 7:1:2 for the
subsequent fine-tuning experiments.

Implementation Details For the language-guided image tokenizer, we adopt Bert
[32] and Vit as the text encoder and visual encoder, respectively. We set the
number of queries to 10, and the number of encoder/decoder layers to 12.
Unless otherwise specified, the training runs for 20 epochs on 4 × 8 NVIDIA
V100 GPUs. AdamW is used as the optimizer, with one sample per GPU. We employ
the cosine annealing schedule as the learning policy, with an initial learning
rate of $2\times 10^{-5}$.

###  4.2 Accurate Visual Grounding in COCO dataset

Table 1: Evaluation results on visual grounding (RefCOCO, RefCOCO+ and RefCOCOg). Red indicates the method with the best indicators, and blue with the second-best<EMAIL_ADDRESS>is applied to evaluate the performance of different methods. Three main types of visual grounding methods are used for comparison, namely two-stage, one-stage and generalist model. |  | RefCOCO | RefCOCO+ | RefCOCOg
---|---|---|---|---
Models | Type | val | testA | testB | val | testA | testB | val-u | test-u
VILLA [13] |  | 82.39 | 87.48 | 74.84 | 76.17 | 81.54 | 66.84 | 76.18 | 76.71
DGA [33] |  | 86.34 | 86.64 | 84.79 | 73.56 | 78.31 | 68.15 | 80.21 | 80.26
CM-Att-Erase [14] |  | 87.47 | 88.12 | 86.32 | 73.74 | 77.58 | 68.85 | 80.23 | 80.37
NMTREE [34] | Two-stage | 85.65 | 85.63 | 85.08 | 72.84 | 75.74 | 67.62 | 78.57 | 78.21
TransVG [16] |  | 81.02 | 82.72 | 78.35 | 64.82 | 70.70 | 56.94 | 68.67 | 67.73
MDETR [15] |  | 86.75 | 89.58 | 81.41 | 79.52 | 84.09 | 70.62 | 81.64 | 80.89
SeqTR [17] |  | 83.72 | 86.51 | 81.24 | 71.45 | 76.26 | 64.88 | 74.86 | 74.21
VGTR [18] | One-stage | 79.30 | 82.16 | 74.38 | 64.40 | 70.85 | 55.84 | 66.83 | 67.28
OFA [19] |  | 92.04 | 94.03 | 88.44 | 87.86 | 91.70 | 80.71 | 88.07 | 88.78
mPLUG-2 [20] |  | 90.33 | 92.80 | 86.05 | - | - | - | 84.70 | 85.14
Shikra-13B [35] |  | 87.83 | 91.11 | 81.81 | 82.89 | 87.79 | 74.41 | 82.64 | 83.16
COMM-7B [36] |  | 91.73 | 94.06 | 88.85 | 87.21 | 91.74 | 81.39 | 87.32 | 88.33
$\textbf{Ours}_{\textbf{(Flan-t5-xl)}}$ |  | 42.77 | 43.14 | 44.53 | 43.84 | 43.85 | 45.22 | 50.55 | 49.71
$\textbf{Ours}_{\textbf{(OPT6.7b)}}$ |  | 88.79 | 91.44 | 86.30 | 85.86 | 90.84 | 83.09 | 88.62 | 87.22
$\textbf{Ours}_{\textbf{(Vicuna-7b)}}$ | Generalist Model | 92.99 | 95.90 | 90.39 | 90.96 | 94.78 | 86.93 | 90.05 | 89.51

Qualitatively, Fig.2 exhibits the visual grounding results of RefCOCO, where
Fig.2(a), 2(b) and 2(c) are from the testA split containing only people, and
Fig.2(d), 2(e) and 2(f) are from the testB split containing only non-people,
respectively. For multiple persons, Fig.2(a) demonstrates the great ability of
the model to detect locations based on orientation indications, such as left,
right, and middle. Significantly, our model accurately interprets the
referring expression even when multiple orientation words are present, such as
"left skier not center". Additionally, overlapping individuals are well-
recognized by our model, as illustrated in Fig.2(b). The man referred to as
"guy with glasses white shirt" is almost within the bounding box of the "woman
in front", and our model carefully distinguishes between them. Concerning
incomplete objects depicted in Fig.2(c), they are effectively identified by
our trained generalist vision-language model. Moreover, the model manages
relative positional relationships, such as the "left shoulder" of the "person
on right". In conclusion, our generalist vision-language model exhibits
considerable proficiency in comprehending referring expressions of persons.

For non-human images, our model also demonstrates excellent performance.
Fig.2(d) shows that our model can accurately recognize animals. Importantly,
the large language model effectively handles size descriptors such as
"largest" and "little", aligning well with the image features extracted by the
visual encoder. In more complex scenarios with multiple objects, as shown in
Fig.2(e), our model accurately distinguishes them. The term "with" is
correctly interpreted as denoting containment in the phrase "back bowl with
cucumbers in it". Concurrently, our trained vision-language model robustly
detects marked objects, such as the aircraft in Fig.2(f). The two planes,
marked ’CANADA’ and ’EXPRESS’ respectively, are precisely identified.

Table.1 further compares the quantitative results between our model and
various types of visual grounding models, including two-stage, one-stage, and
generalist models. Meanwhile, different large language models are supported in
our framework, where Flan-t5 [10], OPT [3], and Vicuna [37] are selected to
demonstrate the scalability and superiority of the proposed method. Table.1
reveals that our presented method with Vicuna-7b achieves state-of-the-art
results in all test datasets. Especially in the testB of RefCOCO and RefCOCO+,
we significantly outperform other methods, indicating the superiority in
referring expression compression task of non-people.

(a) VQA in chest X-ray without foreign objects.

(b) VQA in chest X-ray with foreign objects.

Figure 3: The zero-shot results of visual question answering for foreign
objects detection in chest X-ray images. (a) Chest X-ray image without foreign
objects. Our model accurately states that there are no visible external
foreign objects and points out the possible abnormality. (b) Chest X-ray image
with foreign objects. The presence of foreign objects is accurately detected
by giving its coordinates. Particularly, the model can deduce that the foreign
object is metal or plastic by asking to recognize the foreign object.

###  4.3 General Visual Grounding in Medical Foreign Object Detection

We evaluate ViLaM’s generalizability on the Object-CXR dataset [28] for
foreign object detection in chest X-rays. Vicuna-7b is utilized as the large
language model in our framework. Without fine-tuning, ViLaM accurately answers
questions about foreign objects in sample images as presented in Fig.3. It
correctly states no foreign objects are present in Fig.3(a), while providing
possible explanations. In Fig.3(b), it localizes the foreign object and
deduces it could be metal/plastic debris.

In Fig.3(a), we pose the question, "Is there anything foreign in this x-ray
that is not part of the patient’s body?". Our model accurately identifies the
absence of any foreign objects, responding with, "No, there is nothing visible
in the x-ray that is not part of the patient’s body.". Particularly,
leveraging the expansive generalizability of the large language model,
detailed descriptions explain why no foreign objects were found, and also
point out other abnormalities that are not foreign objects, as highlighted in
red. Furthermore, Fig.3(b) shows a case with foreign objects, by inquiring
about their presence. Our model accurately identifies the foreign object and
provides its localization coordinates, thus demonstrating the model’s ability
to visually detect foreign objects and generalize in a zero-shot setting.
Further demonstrating ViLaM’s generalization capabilities, the model can
recognize the detected foreign object upon inquiry. It goes beyond mere
recognition by deducing that the object is likely made of metal or plastic
debris. ViLaM also exhibits an ability to infer the potential origin or source
of the debris, leveraging its extensive language understanding capacity.

Subsequently, we fine-tune our model on the object-CXR dataset, which further
demonstrates its generalization capabilities. We use 8,000 training samples of
chest X-ray images for fine-tuning, half of which contain foreign objects. Our
generalist model achieves an AUC of 93.1%, surpassing the JF Healthcare
baseline [28] of 92.1%. This result investigates the scalability and
generalizability of our approach, which extends well to medically relevant
tasks through large language models. Notably, when compared with classical and
dedicated object detection methods, such as Fast-RCNN [38] of 95.7% and
YOLO-v3 of 89.7% [39], our generalist vision-language model achieves a similar
performance. This provides further validation of our model’s generalizability,
showing that a generalist model can achieve comparable results to a
specialized model in certain tasks.

(a) VQA in chest X-ray for the tuberculosis localization in the TBX11K
dataset, without referring expressions.

(b) VQA in chest X-ray for the pneumonia localization in the RSNA dataset,
with the orientation-related referring expression.

Figure 4: The 20-shot results of disease localization in chest X-ray images. The red box denotes the grounding truth, and the yellow box represents the prediction. (a) Tuberculosis detection in the TBX11K dataset. (b) Pneumonia detection in the RSNA dataset.  Table 2: Comparison with other state-of-the-art methods of disease localization task with 20-shot setting on six disease labels from the chest X-ray datasets<EMAIL_ADDRESS>is applied to evaluate the methods. Datasets | TBX11K | RSNA | ChestXray14
---|---|---|---
Diseases | Tuberculosis | Pneumonia | Atelectasis | Infiltration | Pneumonia | Pneumothorax
VGTR [18] | 1.99 | 4.67 | 3.70 | 6.67 | 3.23 | 0
OFA [19] | 20.40 | 14.67 | 3.90 | 8.78 | 22.57 | 12.49
Ours | 30.84 | 28.00 | 11.11 | 8.89 | 32.26 | 20.83

###  4.4 Scalable Visual Grounding in Disease Localization on Chest X-ray
Datasets

To examine the model’s scalability to the medical field, we conduct
preliminary experiments on three typical chest X-ray datasets, namely, TBX11K,
RSNA Pneumonia, and ChestXray14. As illustrated in Table. 2, ViLaM
consistently outperforms the classical vision-language models. When fine-tuned
on the downstream dataset with 20 shots for each label, ViLaM achieves an
average<EMAIL_ADDRESS>of 30.84% and 28% for the TBX11K and RSNA Pneumonia datasets,
respectively, exceeding other methods by 10%. When fine-tuned with the full
data, ViLaM maintains the highest<EMAIL_ADDRESS>of 71.46% and 42% on the TBX11K and
RSNA Pneumonia datasets, respectively. Fig.4 illustrates the application of
VQA in chest X-ray analysis for the localization of tuberculosis and
pneumonia. This demonstrates that our generalist model effectively scales to
the medical field, and it can adapt to medical disease localization tasks,
with or without the use of referring expressions.

For the ChestXray14 dataset, we selected four disease labels (i.e.,
Atelectasis, Infiltration, Pneumonia, and Pneumothorax) for demonstration
purposes. Our model consistently outperforms other approaches across most
disease categories. Notably, we observed improvements of 10% in detecting
Pneumonia and Pneumothorax compared to alternative methods.

###  4.5 Ablation Study

Table 3: Ablation evaluation results on visual grounding of RefCOCO dataset.  Conditions | RefCOCO
---|---
| Coordinates
---
Activation
| Cycle
---
Training
| Data
---
Augment.
val | testA | testB
✓ |  |  | 79.95 | 79.24 | 80.99
✓ | ✓ |  | 85.59 | 87.54 | 82.60
✓ | ✓ | ✓ | 92.99 | 95.90 | 90.39

We conduct ablation studies to verify the influence of each component in our
model, namely coordinates activation, cycle training, and data augmentation.
It’s crucial to note that coordinate activation serves as the foundation for
subsequent cycle training, while data augmentation depends on the generation
of referring expressions during cycle training. Therefore, the ablation
studies follow a sequence from coordinate activation, to cycle training, and
finally to data augmentation.

Table.3 presents the ablation study results for our model’s visual grounding
task using the RefCOCO dataset. When only the coordinates are activated, the
model demonstrates strong performance, achieving an<EMAIL_ADDRESS>of 79.95% in val,
79.24% in testA, and 80.99% in testB. Benefiting from the impressive
comprehension and inference capabilities of the LLM, it can effectively handle
referring expressions, even though it was trained solely on normal object
detection datasets. This strong performance lays a solid foundation for
subsequent cycle training. The generated referring expressions, used as
pseudo-labels, are particularly beneficial for other object detection datasets
that lack referring expressions.

With the cycle training, our model achieves<EMAIL_ADDRESS>of 85.59% in val, 87.54%
in testA, and 82.60% in testB, which presents a significant improvement from
the coordinates activation. Furthermore, the cycle training in the form of
VQA, based on the captioning task of coordinate activation, facilitates a
better understanding of referring expressions. Besides, our model leverages
data augmentation using the COCO 2014 and COCO 2017 datasets, which achieves
an<EMAIL_ADDRESS>of 92.99% in val, 95.90% in testA, and 90.39% in testB. These
results suggest that utilizing large amounts of data can significantly enhance
a large model’s performance.

##  5 Conclusion

We have developed a vision-language model, ViLaM, that enhances visual
grounding capabilities and generalization performance based on the foundations
of a large language model. We evaluated ViLaM’s visual grounding capabilities
on multiple natural datasets and found that it outperformed existing methods,
illustrating the enhanced visual capabilities offered by our proposed
approach. Importantly, ViLaM also exhibits strong visual grounding
capabilities and generalizability in medical tasks, indicating potential for
future clinical applications.

## References

  * [1] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
  * [2] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
  * [3] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022.
  * [4] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.  Bloom: A 176b-parameter open-access multilingual language model.  arXiv preprint arXiv:2211.05100, 2022.
  * [5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * [7] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * [8] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded language-image pre-training, 2022.
  * [9] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  * [10] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling Instruction-Finetuned Language Models.
  * [11] OpenAI.  Gpt-4v(ision) system card, 2023.
  * [12] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning, 2023.
  * [13] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.  Large-scale adversarial training for vision-and-language representation learning.  Advances in Neural Information Processing Systems, 33:6616–6628, 2020.
  * [14] Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, and Hongsheng Li.  Improving Referring Expression Grounding With Cross-Modal Attention-Guided Erasing.  pages 1950–1959.
  * [15] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.  Mdetr-modulated detection for end-to-end multi-modal understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780–1790, 2021.
  * [16] Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li.  Transvg: End-to-end visual grounding with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1769–1779, October 2021.
  * [17] Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji.  SeqTR: A simple yet universal network for visual grounding.  In Lecture Notes in Computer Science, pages 598–615. Springer Nature Switzerland, 2022.
  * [18] Ye Du, Zehua Fu, Qingjie Liu, and Yunhong Wang.  Visual grounding with transformers.  In Proceedings of the International Conference on Multimedia and Expo, 2022.
  * [19] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework, 2022.
  * [20] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, and Jingren Zhou.  mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video.
  * [21] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [22] Ander Salaberria, Gorka Azkune, Oier Lopez de Lacalle, Aitor Soroa, and Eneko Agirre.  Image captioning for effective use of language models in knowledge-based visual question answering.  212:118669.
  * [23] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2223–2232.
  * [24] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg.  Modeling Context in Referring Expressions.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, Lecture Notes in Computer Science, pages 69–85. Springer International Publishing.
  * [25] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy.  Generation and comprehension of unambiguous object descriptions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11–20.
  * [26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common Objects in Context.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, Lecture Notes in Computer Science, pages 740–755. Springer International Publishing.
  * [27] Varun K. Nagaraja, Vlad I. Morariu, and Larry S. Davis.  Modeling Context Between Objects for Referring Expression Understanding.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, Lecture Notes in Computer Science, pages 792–807. Springer International Publishing.
  * [28] JF Healthcare.  Object-cxr - automatic detection of foreign objects on chest x-rays.
  * [29] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017.
  * [30] George Shih, Carol C Wu, Safwan S Halabi, Marc D Kohli, Luciano M Prevedello, Tessa S Cook, Arjun Sharma, Judith K Amorosa, Veronica Arteaga, Maya Galperin-Aizenberg, et al.  Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia.  Radiology: Artificial Intelligence, 1(1):e180041, 2019.
  * [31] Yun Liu, Yu-Huan Wu, Yunfeng Ban, Huifang Wang, and Ming-Ming Cheng.  Rethinking computer-aided tuberculosis diagnosis.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2646–2655, 2020.
  * [32] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
  * [33] Sibei Yang, Guanbin Li, and Yizhou Yu.  Dynamic Graph Attention for Referring Expression Comprehension.  pages 4644–4653.
  * [34] Daqing Liu, Hanwang Zhang, Feng Wu, and Zheng-Jun Zha.  Learning to Assemble Neural Module Tree Networks for Visual Grounding.  pages 4673–4682.
  * [35] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.  Shikra: Unleashing multimodal llm’s referential dialogue magic, 2023.
  * [36] Dongsheng Jiang, Yuchen Liu, Songlin Liu, Xiaopeng Zhang, Jin Li, Hongkai Xiong, and Qi Tian.  From clip to dino: Visual encoders shout in multi-modal large language models, 2023.
  * [37] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [38] Ross Girshick.  Fast R-CNN.  pages 1440–1448.
  * [39] Joseph Redmon and Ali Farhadi.  YOLOv3: An Incremental Improvement.