# Pressure dependence of solvation of non-polar solute in simple model of
water††thanks: Dedicated to Professor Yurii Kalyuzhnyi on the occasion of his
70th birthday.

T. Urbic  $|$

(Received May 03, 2021, in final form July 21, 2021)

###### Abstract

We modelled the aqueous solvation of a nonpolar solute as a function of the
radius, temperature and pressure. In this study a simple two-dimensional
Mercedes-Benz (MB) water model was used in NPT Monte Carlo simulations. This
model has previously been shown to qualitatively predict the volume anomalies
of pure water and the free energy, enthalpy, entropy, heat capacity, and
volume change in order to insert a nonpolar solute into water. Here, we
extended the studies of solvation of nonpolar solute to examine the pressure
dependence and broader range of temperature and size dependence. The model
shows two different mechanisms, one for the solvation of large nonpolar
solutes bigger than water and the second for smaller solutes.

Key words: Monte Carlo, solvation, nonpolar solute

###### Abstract

Ó ñòàòò çìîäåëüîâàíî âîäíó ñîëüâàòàöþ íåïîëÿðíî¿ ðîçèíåíî¿ ðåîâèíè ÿê ôóíêöþ
ðàäóñó àñòèíîê, òåìïåðàòóðè òà òèñêó. Äëÿ ìîäåëþâàííÿ Ìîíòå Êàðëî â NPT-
àíñàìáë âèêîðèñòîâóòüñÿ ïðîñòà äâîâèìðíà ò. çâ. ‘‘Ìåðñåäåñ-Áåíö’’ ìîäåëü âîäè.
Ðàíøå áóëî ïîêàçàíî, ùî öÿ ìîäåëü çäàòíà ÿêñíî ïåðåäáààòè îá’ìí àíîìàë¿ èñòî¿
âîäè, à òàêîæ âëüíó åíåðãþ, åíòàëüïþ, åíòðîïþ, òåïëîìíñòü òà çìíè îá’ìó ïðè
ââåäåíí ó âîäó íåïîëÿðíî¿ ðîçèíåíî¿ ðåîâèíè. Ó äàíé ðîáîò ïðîâåäåíî áëüø
äåòàëüíå âèâåííÿ ñîëüâàòàö¿ íåïîëÿðíèõ ðîçèíåíèõ ðåîâèí ç ìåòîþ ïåðåâðêè
çàëåæíîñò öüîãî ïðîöåñó âä òèñêó òà øèðøîãî äàïàçîíó òåìïåðàòóð ðîçìðâ
àñòèíîê. Äàíà ìîäåëü äåìîíñòðó äâà ðçí ìåõàíçìè: îäèí — äëÿ ñîëüâàòàö¿ âåëèêèõ
íåïîëÿðíèõ ðåîâèí, ðîçìðè àñòèíîê ÿêèõ áëüø, íæ ó âîäè, òà äðóãèé — äëÿ ìåíøèõ
àñòèíîê.

Ключов слова: Ìîíòå Êàðëî, ñîëüâàòàöÿ, íåïîëÿðí ðîçèíåí ðåîâèíè

##  1 Introduction

Understanding solute-solvent interactions in aqueous solutions is important in
biology and chemistry. A lot of works and several important review papers have
been published on this subject [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]. Various
water models with different degrees of sophistication have been developed to
capture the anomalous properties of water and to study aqueous solutions (for
review see [12, 13]). In spite of huge time spent studying the water and its
solvation properties, the water is considered to be a poorly understood liquid
[14]. The most important anomalous properties of water are maximum in density,
negative expansion coefficient, low compressibility, high and almost constant
heat capacity [11]. The principal challenge in modelling the physics of water
comes from the orientation-dependent formation of hydrogen bonds.

Many properties of the water and aqueous solutions can be captured by simple
models that lack atomic details [15, 16, 17]. One of the simplest model for
water is the Mercedes-Benz (MB) model [18] which was originally proposed by
Ben-Naim in 1971 [19, 20]. Each molecule is described as disk with Lennard-
Jones (LJ) interactions and three radial arms arranged as in the MB logo to
mimic the formation of hydrogen bonds (HB). This HB interaction is an
orientation-dependent interaction.

Simplified models are of interest due to the insights they offer that are not
obtainable from all-atom computer simulations. Simpler models are more
flexible in providing insights and illuminating concepts, and do not require
big computer resources. Second, simple models can explore a much broader range
of conditions and external variables. Meanwhile, simulating a detailed model
may predict the behaviour at a single temperature and pressure, and a simpler
model can be used to study a whole phase diagram of temperatures and pressures
much faster. Third, analytical models can provide functional relationships for
engineering applications and lead to improved models of greater computational
efficiency. Fourth, simple models can be used as a polygon to develop and
study theoretical methods. Our interest in using the MB model is that it
serves as one of the simplest models of an orientationally dependent liquid.
Thus, it can serve as a testbed for developing analytical theories that might
ultimately be useful for more realistic models. Another advantage of the MB
model, compared to more realistic water models, is that the underlying
physical principles can be more readily explored and visualised in two
dimensions.

For the MB model, NPT Monte Carlo simulations have shown that the MB model
qualitatively predicts the density anomaly, the minimum in isothermal
compressibility as a function of temperature, a large heat capacity, as well
as the experimental trends for thermodynamic properties of solvation of
nonpolar solutes [18, 21, 22, 23, 24, 25, 26] and cold denaturation of
proteins [27]. The 2D MB model was also extended to 3D by Bizjak et al. [28,
29] and Dias et al. [26, 30] and was studied using computer simulations [28,
29, 26, 27]. The 2D model was also extensively studied using analytical
methods such as integral equation and thermodynamic perturbation theory [31,
32, 33, 34, 35, 36, 37] and statistical mechanic modelling [38, 39, 40].
Recently, phase diagram of liquid part and percolation curve of the model was
also calculated and reported [41]. The MB model has also been used to study
the systems with water molecules confined in partially quenched disordered
matrix [34, 42, 43] and within small geometric spaces [37, 44]. Non-
equilibrium Monte Carlo and molecular dynamics simulations were used to study
the effect of translational and rotational degrees of freedom on the
structural and thermodynamic properties of the simple Mercedes-Benz water
model [45]. By holding one of the temperatures constant and varying the other
one, the effect of faster motion in the corresponding degrees of freedom on
the properties of the simple water model was investigated. For this case, we
also developed an analytical theory for studying rotational and translational
degrees of freedom, by applying integral equation theory and thermodynamic
perturbation theory [46, 47]. In this work we extended the studies of
solvation of nonpolar solute to study the effect of size, pressure and
temperature as well as the depth of LJ potential between water and solute.

##  2 Model

Figure 1: The MB particles.

In the MB model, the water molecules are modelled as two-dimensional disks
with Lennard-Jones potential and three arms placed so that the angle between
them is 120∘ [19, 18] to mimic the formation of hydrogen bonds between
molecules. The interaction between two molecules $i$ and $j$ is the sum of
Lennard-Jones (LJ) and hydrogen-bonding (HB) term

$U(\vec{X}_{i},\vec{X}_{j})=U_{\rm{LJ}}(r_{ij})+U_{\rm{HB}}(\vec{X}_{i},\vec{X}_{j})$
(2.1)

and depends on the distance between molecules as well as on their relative
orientation ($\vec{X}_{i}$ is the vector of the position and orientation of
the $i$-th particle). The Lennard-Jones part of the potential is calculated in
a standard way as follows:

$U_{\rm{LJ}}(r_{ij})=4\varepsilon_{\rm{LJ}}\left[\left(\frac{\sigma_{\rm{LJ}}}{r_{ij}}\right)^{12}-\left(\frac{\sigma_{\rm{LJ}}}{r_{ij}}\right)^{6}\right].$
(2.2)

$\varepsilon_{\rm{LJ}}$ is the depth and $\sigma_{\rm{LJ}}$ is the contact
distance of the LJ potential. The term that describes hydrogen bonds is a sum
of all interactions $U_{\rm{HB}}^{kl}$ between the arms $k$ and $l$ of
molecules $i$ and $j$, respectively

$U_{\rm{HB}}(\vec{X}_{i},\vec{X}_{j})=\sum_{k,l=1}^{3}U_{\rm{HB}}^{kl}(r_{ij},\theta_{i},\theta_{j}),$
(2.3)

where $\theta_{i}$ is the orientation of $i$-th molecule, interaction between
two arms is modelled with Gaussian function depending on the distance between
the interacting molecules and their orientation.

$U_{\rm{HB}}^{kl}(r_{ij},\theta_{i},\theta_{j})=\varepsilon_{\rm{HB}}G(r_{ij}-r_{\rm{HB}})G(\vec{i}_{k}\vec{u}_{ij}-1)G(\vec{j}_{l}\vec{u}_{ij}+1)\\\
=\varepsilon_{\rm{HB}}G(r_{ij}-r_{\rm{HB}})G(\cos(\theta_{i}+{\scriptstyle\frac{2\piup}{3}}(k-1))-1)G(\cos(\theta_{j}+{\scriptstyle\frac{2\piup}{3}}(l-1))+1).$
(2.4)

$G(x)$ is an unnormalized Gaussian function:

$G(x)=\exp{\left(-{\frac{x^{2}}{2\sigma^{2}}}\right)},$ (2.5)

$\varepsilon_{\rm{HB}}$ is the HB energy and $r_{\rm{HB}}$ is a HB distance.
$\vec{u}_{ij}$ is the unit vector along $\vec{r}_{ij}$ and $\vec{i}_{k}$ is
the unit vector representing the $k$-th arm of the $i$-th particle. When the
arms on two molecules are parallel, pointing toward the center of the other
molecule and the distance between molecules is $r_{\rm{HB}}$, the strongest
possible hydrogen bond is formed. Units used are the same as in previous
studies: energies were expressed in $|\varepsilon_{\rm{HB}}|$ and lengths in
$r_{\rm{HB}}$. Hence, parameter for hydrogen-bond energy,
$\varepsilon_{\rm{HB}}$, equalled $-1$, and hydrogen-bond length equalled $1$.
LJ interaction potential parameters were set to:
$\varepsilon_{\rm{LJ}}=0.1|\varepsilon_{\rm{HB}}|$ and
$\sigma_{\rm{LJ}}=0.7r_{\rm{HB}}$. The width of Gaussian
($\sigma=0.085r_{\rm{HB}}$) is small enough, so that a direct hydrogen bond is
more favourable than a bifurcated one.

Hydrophobic solute was modelled as a disk with Lennard-Jones potential. The
depth of solute’s LJ potential was the same as depth of water’s LJ potential
for most calculations, while the $\sigma_{\rm{LJ}}$, of solute’s LJ potential
were varied to represent solutes of various sizes (from 0.1 to 5.00). When the
effect of depth of LJ potential was studied, the $\varepsilon_{\rm{LJ}}$ was
varied from 0.01 to 0.5. When calculating the interaction between solute and
water molecule, Lorentz-Berthelot mixing rules were used [48].

(a) (b)

(c) (d)

Figure 2: (Colour online) Temperature dependence of (a) the density, (b) the
isothermal compressibility multiplied by pressure, (c) the thermal expansion
coefficient and (d) the heat capacity for various pressures.

##  3 Monte Carlo simulation details

(a) (b)

(c) (d)

Figure 3: (Colour online) Pressure dependence of (a) the density, (b) the
isothermal compressibility multiplied by pressure, (c) the thermal expansion
coefficient and (d) the heat capacity for various temperatures.

(a) (b)

Figure 4: (Colour online) Pair correlation functions for various pressures at
(a) low temperature $T^{*}=0.18$, and (b) high temperature $T^{*}=0.24$.

(a)

(b)

(c)

Figure 5: (Colour online) Size dependence of transfer free energy and transfer
enthalpy of nonpolar solute for pressures (a) $p^{*}=0.08$, (b) $p^{*}=0.19$,
and (c) $p^{*}=0.32$ for various temperatures for depth of solute-solute LJ
potential equal to $\varepsilon_{\rm{LJ}}=0.1$

(a)

(b)

Figure 6: (Colour online) Size dependence of transfer free energy and transfer
enthalpy of nonpolar solute for temperatures (a) $T^{*}=0.18$, and (b)
$T^{*}=0.24$ for various pressures for depth of solute-solute LJ potential
equal to $\varepsilon_{\rm{LJ}}=0.1$

(a)

(b)

(c)

Figure 7: (Colour online) Temperature dependence of transfer free energy and
transfer enthalpy of nonpolar solute for sizes of the solution (a)
$\sigma_{\rm{LJ}}=0.3$, (b) $\sigma_{\rm{LJ}}=0.7$, and (c)
$\sigma_{\rm{LJ}}=1.5$ for various pressures for depth of solute-solute LJ
potential equal to $\varepsilon_{\rm{LJ}}=0.1$

(a)

(b)

(c)

Figure 8: (Colour online) Solute-water pair correlation functions for various
pressures for sizes of the solution (a) $\sigma_{\rm{LJ}}=0.3$, (b)
$\sigma_{\rm{LJ}}=0.7$, and (c) $\sigma_{\rm{LJ}}=1.5$ for various pressures
for depth of solute-solute LJ potential equal to $\varepsilon_{\rm{LJ}}=0.1$
for high temperature $T^{*}=0.24$ (right-hand) and for low temperature
$T^{*}=0.18$ (left-hand).

(a)

(b)

(c)

Figure 9: (Colour online) Temperature dependence of transfer free energy and
transfer enthalpy of nonpolar solute for different depths of solute-solute LJ
potential $\varepsilon_{\rm{LJ}}$ for sizes of the solution (a)
$\sigma_{\rm{LJ}}=0.3$, (b) $\sigma_{\rm{LJ}}=0.7$, and (c)
$\sigma_{\rm{LJ}}=1.5$ for pressure $p^{*}=0.19$.

In order to determine the solvation of nonpolar solute in 2D MB model of water
we performed Monte Carlo computer simulations in the isothermal-isobaric
ensemble with constant $N$, $p$, $T$ [49, 50]. In order to mimic an infinite
size of system of particles we used the periodic boundary conditions and the
minimum image convention. All starting configurations were selected at random.
In each move, we randomly tried to translate or rotate the random particle or
to change the volume. Probabilities for translation and rotation, change of
volume and exchange of particles were the same. In one cycle we tried to
translate and rotate each particle once on average and one change of volume
per move of particles. The simulations were first equilibrated for as many
cycles until equilibration was reached. The number of cycles depended on
pressure and temperature. After equilibration, production runs were taken for
minimum 20 series consistent with as many cycles as required to obtain well
converged results. In the system we had from 100 to 500 particles depending on
the density of the system. Thermodynamic quantities such as energy were
calculated as statistical averages over the course of the simulations [50].
Cut off of the potential was half-length of the simulation box. Increasing the
number of particles had no significant effect on the calculated quantities.
The cluster and bond analyses use an energy criteria wherein water molecules
are considered bonded when their hydrogen bonded energy is less than $-0.05$.
Small variations of this energy cutoff did not account for significant
differences in the bonding state of the system. The mechanical properties such
as enthalpy and volume were calculated as the statistical averages of these
quantities over the course of the simulations [50]. The heat capacity,
$C_{p}$, the isothermal compressibility, $\kappa$, and the thermal expansion
coefficient, $\alpha$, are computed from the fluctuations [18]. The
thermodynamic properties of a solvation process were computed using the
Widom’s particle insertion method [51]. At each cycle, a ghost solute particle
is randomly placed into the box and is not allowed to interact with the
solvent particles. We then calculate the hypothetical interaction of the ghost
particle with the solvent, and its weighted average. Thermodynamic quantities
describing the solvation (free energy $\Delta G$, enthalpy $\Delta H$, entropy
$\Delta S$ and heat capacity $\Delta c_{p}$) and the change in volume ($\Delta
V$) follow from [18]:

$\displaystyle\Delta G$ $\displaystyle=$
$\displaystyle-\beta^{-1}\ln\left[\left\langle
V\right\rangle_{N}^{-1}\left\langle V\exp\left(-\beta
U\right)\right\rangle_{N}\right],$ (3.1) $\displaystyle\Delta H$
$\displaystyle=$ $\displaystyle\frac{\left\langle H_{N+1}\exp\left(-\beta
U\right)\right\rangle_{N}}{\left\langle\exp\left(-\beta
U\right)\right\rangle_{N}}-\left\langle H_{N}\right\rangle_{N},$ (3.2)
$\displaystyle T\Delta S$ $\displaystyle=$ $\displaystyle\Delta H-\Delta G,$
(3.3) $\displaystyle k_{b}T^{2}\Delta c_{p}$ $\displaystyle=$
$\displaystyle\frac{\left\langle H_{N+1}^{2}\exp\left(-\beta
U\right)\right\rangle_{N}}{\left\langle\exp\left(-\beta
U\right)\right\rangle_{N}}-\frac{\left\langle H_{N+1}\exp\left(-\beta
U\right)\right\rangle_{N}^{2}}{\left\langle\exp\left(-\beta
U\right)\right\rangle^{2}_{N}}-\left\langle
H_{N}^{2}\right\rangle_{N}+\left\langle H_{N}\right\rangle_{N}^{2},$ (3.4)
$\displaystyle\Delta V$ $\displaystyle=$ $\displaystyle\frac{\left\langle
V\exp\left(-\beta U\right)\right\rangle_{N}}{\left\langle\exp\left(-\beta
U\right)\right\rangle_{N}}-\left\langle V\right\rangle_{N},$ (3.5)

where $\beta=\left(k_{b}T\right)^{-1}$. Angle brackets $<...>_{N}$ denote the
averages throughout the run over all insertions. $U$ is the interaction of
ghost particle with the system and $H_{N+1}$ represents the enthalpy of the
system with ghost particles. $\left\langle H_{N}\right\rangle_{N}$ and
$\left\langle H_{N}^{2}\right\rangle_{N}$ stand for the average enthalpy and
the average of squared enthalpy, respectively, of the system without the ghost
particle.

##  4 Results

All the results are given in reduced units; the excess internal energy and
temperature are normalized to the HB interaction parameter
$\varepsilon_{\rm{HB}}$ ($E^{*}={E}/{\varepsilon_{\rm{HB}}}$,
$T^{*}={k_{\rm{B}}T/{\varepsilon_{\rm{HB}}}}$) and all the distances are
scaled to the HB characteristic length $r_{\rm{HB}}$
($r^{*}={r}/{r_{\rm{HB}}}$).

In figure 2, we calculated the temperature dependence of the density,
$\rho^{*}$, the isothermal compressibility, $\kappa^{*}$, the thermal
expansion coefficient, $\alpha^{*}$, the heat capacity, $c_{p}^{*}$ for bulk
water for different pressures. Since $\kappa^{*}$ behaves as $1/p^{*}$ at high
temperatures, we normalized the curves by multiplying them with $p^{*}$ so
that all have the same high temperature limit. The lowest pressure $0.01$ is
at below gas-liquid critical pressure, so we are in the region of phase
transition which we can see as a jump for all 4 quantities. All other
pressures are above the critical point, so we can see only the maxima in
curves. These maxima in all three quantities are moving to higher temperatures
as we increase pressure. The highest pressure $0.32$ is already so high that
we no longer have density anomaly present for this model. The model has an
area of anomalous behaviour where the density has maxima similar to real water
for different pressures [11]. We can observe the density anomaly for pressures
between $p^{*}=0.01$ and $p^{*}=0.3$ for this model. Here, we can also observe
negative thermal expansion coefficients. In figure 3, we calculated the
pressure dependence of the same quantities ($\rho^{*}$, $\kappa^{*}$,
$\alpha^{*}$, $c_{p}^{*}$) for bulk water for different temperatures in the
region of density anomaly and outside the region. For density dependence we
can see crossover of the curves in the region of density anomaly. We can also
observe negative thermal expansion coefficients for these points. There are no
much data available for pressure dependence in literature. We did some
analytical modelling and we can observe the same trends in MB model as in
analytical modelling of experimental water [52]. As last, we also checked the
pair correlations function between two water molecules plotted in figure 4. We
presented the behaviour for high and low temperature for various pressures. At
both temperatures we observe an increase of the pressure lower peak at
distance 1 (HB peak). We can explain this as melting of hydrogen bonded
structures. At the same time, we can notice that the peak at 0.7 is increasing
(Lennard-Jones contact peak). At a very high pressure 1.0 both peaks have
about the same size and the long range order is different, so we think that we
have a different kind of liquid in comparison with lower pressures.

Next, we studied the solvation of nonpolar solutes in this MB model of water.
First we calculated the transfer free enthalpy and transfer enthalpy of
nonpolar solute modelled as Lennard-Jones disk of various sizes. We first
check the size dependence for isotherms at different pressures (figure 5) and
isobars at different temperatures (figure 6). We can see the following form
the curves: (1) bigger solutes are easier to inset at higher temperatures,
position crossover depends on the temperature and pressure; (2) smaller
solutes have linear dependence of the free enthalpy (up to size 1.0), bigger
solutes have quadratic dependence; (3) transfer enthalpy has the lowest value
for solutes of the size 1.0 (size of MB water, $r_{\rm{HB}}=1.0$; and (4) for
high pressures ($p^{*}=1.0$) there is only one regime, only quadratic
behaviour of transfer free enthalpy; transfer enthalpy also does not exhibit
minima for reasons mentioned in the results of pure MB water. We can see that
size dependence of solvation is similar to real 3D water. Bigger solutes are
more solvable at high temperatures [53].

Next, we selected three different sizes of solutes from different regions;
solutes smaller than water particles $\sigma_{\rm{LJ}}=0.3$ from the region of
linear dependence of transfer free enthalpy, solutes of the same size as a
water particles $\sigma_{\rm{LJ}}=0.7$ (this is the region of change from
linear to quadratic behaviour of transfer free enthalpy), and solutes bigger
than water molecules $\sigma_{\rm{LJ}}=1.5$ from region of quadratic
behaviour. We checked how pressure affects the transfer quantities for the
same type of solutes studied earlier [18]. The temperature dependence of the
transfer quantities for different isobars is presented in figure 7. For all
pressures, the transfer free energy is positive and we have the limit at high
temperature to the value that it is proportional to the product of the
pressure and volume of the solute. The dependence has maxima at a smaller
pressure which is more pronounced at lower pressures. The position of the
maxima depends on the solute sizes and shifts to lower temperatures as the
size of the solute increases. The transfer enthalpy is negative and increases
up to the maxima which is positive and then goes toward a limiting value which
depends on pressure. Each curve has temperature at which the transfer enthalpy
changes its sign. This temperature increases with an increase of the pressure
and decreases with an increase of the size of the solute. These quantities
behave differently at higher pressures. The transfer free enthalpy no longer
has maxima. For smaller sizes, it increases toward the limit, for bigger sizes
it decreases toward the limit. The behaviour for the same size as water has
minima at temperature $T^{*}=0.18$ which is temperature where there is anomaly
present at lower pressures. The transfer enthalpy is positive for all
temperatures for higher pressures. The reason is that at these pressures the
liquid is very dense and there are no cavities present for insertion of
nonpolar solute. Thus, in order to insert the solute, we first need to spend
energy to make a cavity and then the interaction energy between nonpolar
solute and water is released, but all in all the change is positive. Next, we
also checked solute-water pair correlation functions for different pressures
(in figure 8) for high and low temperature. For big solute and high pressure,
the statistics were not good, so data are not present. The reason is high
density and insufficiently big cavities to get good statistics. We can see
that with an increase of the pressure, the contact peak increases, which is
more pronounced at higher temperatures.

Finally, we also checked how the depth of LJ potential between solutes changes
the thermodynamics of solvation. In all previous studies, the value of this
parameter was always kept equal to the value of MB. In figure 9 we plotted the
temperature dependence of transfer free energy, $\Delta G^{*}$, and transfer
enthalpy, $\Delta H^{*}$, for various depths at a standard pressure of MB
water model $p^{*}=0.19$. For small values of parameter
$\varepsilon_{\rm{LJ}}$, the transfer free energy is positive and has maxima.
At maxima, the solubility is the lowest. With an increase of the depth, the
maxima first move to higher temperatures and later disappear and the values
become negative. The transfer free enthalpy also decreases which means that
solubility of the solute increases. We can see the same trend for all sizes,
only that for bigger solutes it shifts to bigger depths. The transfer enthalpy
has similar characteristics, but the shape is different. For smaller solutes,
the transfer enthalpy is negative for all depths at low temperatures and
increases with an increase of the temperature up to maxima and then tends
toward the limiting value at high temperatures. This limiting value depends on
the size of the solute. With an increase of the depth, the maxima disappear,
but they are more pronounced for bigger solutes and shift toward lower
temperatures. The maxima disappear at higher depths.

##  5 Conclusion

In this work we have studied the aqueous solvation of a nonpolar solute. We
used 2D MB model for water and LJ disks as nonpolar solutes. We studied the
structure and thermodynamic properties as a function of radius of nonpolar
solute and solute-solute attraction, temperature and pressure in NPT Monte
Carlo simulations. At low pressures, the model shows two different mechanisms,
one for the solvation of large nonpolar solutes bigger than water and the
second for smaller solutes. At higher pressures, we have only one kind of
mechanism. The change of Lennard-Jones depth between solutes also leads to a
stronger attraction with water and a negative transfer of free enthalpy and
enthalpy.

##  6 Acknowledgements

The financial support of the Slovenian Research Agency through Grant P1-0201
as well as to projects J7-1816, J1-1708, N1-0186 and N2-0067 is acknowledged
as well as National Institutes for Health RM1 award RM1GM135136.

## References

  * [1] Eisenberg D., Kauzmann W., The Structure and Properties of Water, Oxford University Press, Oxford, 1969.
  * [2] Franks F. (Ed.), Water, a Comprehensive Treatise, Vol. 1–7, Plenum Press, New York, 1972–1980.
  * [3] Stillinger F. H., Science, 1980, 209, 451–457, doi:10.1126/science.209.4455.451.
  * [4] Tanford C., The Hydrophobic Effect: Formation of Micelles and Biological Membranes, 2nd ed., Wiley, New York, 1980.
  * [5] Robinson G., Zhu S.-B., Singh S., Evans M., Water in Biology, Chemistry and Physics: Experimental Overviews and Computational Methodologies, World Scientific, Singapore, 1996.
  * [6] Pratt L. R., Annu. Rev. Phys. Chem., 2002, 53, 409, doi:10.1146/annurev.physchem.53.090401.093500.
  * [7] Widom B., Bhimalapuram P., Koga K., Phys. Chem. Chem. Phys., 2003, 5, 3085–3093, doi:10.1039/B304038K.
  * [8] Griffith J. H., Scheraga H. A., J. Mol. Struct. THEOCHEM, 2004, 682, 97–113,
doi:10.1016/j.theochem.2004.06.003.

  * [9] Huang D. M., Chandler D., J. Phys. Chem. B, 2002, 106, 2047–2053, doi:10.1021/jp013289v.
  * [10] Předota M., Ben-Naim A., Nezbeda I., J. Chem. Phys., 2003, 118, 6446, doi:10.1063/1.1559687.
  * [11] Gallo P., Amann-Winkel K., Angell C. A., Anisimov M. A., Caupin F., Chakravarty C., Lascaris E., Loerting T., Panagiotopoulos A. Z., Russo J., Sellberg J. A., Stanley H. E., Tanaka H., Vega C., Xu L., Pettersson L. G. M., Chem. Rev., 2016, 116, 7463–7500, doi:10.1021/acs.chemrev.5b00750.
  * [12] Nezbeda I., J. Mol. Liq., 1997, 73/74, 317–336, doi:10.1016/S0167-7322(97)00076-7.
  * [13] Smith D. E., Haymet A. D. J., In: Reviews in Computational Chemistry, Lipkowitz K. B., Larter R.,
Cundari T. R. (Eds.), Wiley-VCH, New York, 2003.

  * [14] Dill K. A., Truskett T. M., Vlachy V., Hribar-Lee B., Annu. Rev. Biophys. Biomol. Struct., 2005, 34, 173,
doi:10.1146/annurev.biophys.34.040204.144517.

  * [15] Truskett T.M., Debenedetti P.G., Sastry S., Torquato S., J. Chem. Phys., 1999, 111, 2647, doi:10.1063/1.479540.
  * [16] Truskett T.M., Dill K. A., J. Phys. Chem. B, 2002, 106, 11829–11842, doi:10.1021/jp021418h.
  * [17] Williamson C. H., Hall J. R., Fennell C. J., J. Mol. Liq., (in press).
  * [18] Silverstein K. A. T., Haymet A. D. J., Dill K. A., J. Am. Chem. Soc., 1998, 120, 3166–3175,
doi:10.1021/ja973029k.

  * [19] Ben-Naim A., J. Chem. Phys., 1971, 54, 3682, doi:10.1063/1.1675414.
  * [20] Ben-Naim A., Mol. Phys., 1972, 24, 705–721, doi:10.1080/00268977200101851.
  * [21] Andaloro G., Sperandeo-Mineo R. M., Eur. J. Phys., 1990, 11, 275, doi:10.1088/0143-0807/11/5/004.
  * [22] Silverstein K. A. T., Dill K. A., Haymet A. D. J., Fluid Phase Equilib., 1998, 150-151, 83–90,
doi:10.1016/S0378-3812(98)00278-7.

  * [23] Southall N. T., Dill K. A., J. Phys. Chem. B, 2000, 104, 1326–1331, doi:10.1021/jp992860b.
  * [24] Silverstein K. A. T., Dill K. A., Haymet A. D. J., J. Chem. Phys., 2001, 114, 6303, doi:10.1063/1.1355997.
  * [25] Hribar B., Southall N. T., Vlachy V., Dill K. A., J. Am. Chem. Soc., 2002, 124, 12302–12311,
doi:10.1021/ja026014h.

  * [26] Dias C. L., Hynninen T., Ala-Nissila T., Foster A. S., Karttunen M., J. Chem. Phys., 2011, 134, 65106,
doi:10.1063/1.3537734.

  * [27] Dias C. L., Phys. Rev. Lett., 2012, 109, 048104, doi:10.1103/PhysRevLett.109.048104.
  * [28] Bizjak A., Urbic T., Vlachy V., Dill K. A., Acta Chim. Slov., 2007, 54, 532–537.
  * [29] Bizjak A., Urbic T., Vlachy V., Dill K. A., J. Chem. Phys., 2009, 131, 194504, doi:10.1063/1.3259970.
  * [30] Dias C. L., Ala-Nissila T., Grant M., Karttunen M., J. Chem. Phys., 2009, 131, 054505, doi:10.1063/1.3183935.
  * [31] Urbič T., Vlachy V., Kalyuzhnyi Yu. V., Southall N. T., Dill K. A., J. Chem. Phys., 2000, 112, 2843, doi:10.1063/1.480928.
  * [32] Urbič T., Vlachy V., Kalyuzhnyi Yu. V., Southall N. T., Dill K. A., J. Chem. Phys., 2002, 116, 723, doi:10.1063/1.1427307.
  * [33] Urbič T., Vlachy V., Kalyuzhnyi Yu. V., Dill K. A., J. Chem. Phys., 2003, 118, 5516, doi:10.1063/1.1556754.
  * [34] Urbič T., Vlachy V., Pizio O., Dill K. A., J. Mol. Liq., 2004, 112, 71–80, doi:10.1016/j.molliq.2003.12.001.
  * [35] Urbič T., Vlachy V., Kalyuzhnyi Yu. V., Dill K. A., J. Chem. Phys., 2007, 127, 174511, doi:10.1063/1.2784124.
  * [36] Urbič T., Vlachy V., Kalyuzhnyi Yu. V., Dill K. A., J. Chem. Phys., 2007, 127, 174505, doi:10.1063/1.2779329.
  * [37] Urbič T., Holovko M. F., J. Chem. Phys., 2011, 135, 134706, doi:10.1063/1.3644934.
  * [38] Urbič T., Dill K. A., J. Chem. Phys., 2010, 132, 224507, doi:10.1063/1.3454193.
  * [39] Urbič T., Phys. Rev. E, 2012, 85, 061503, doi:10.1103/PhysRevE.85.061503.
  * [40] Urbič T., Phys. Rev. E, 2016, 94, 042126, doi:10.1103/PhysRevE.94.042126.
  * [41] Urbič T., Phys. Rev. E, 2017, 96, 032122, doi:10.1103/PhysRevE.96.032122.
  * [42] Kurtjak M., Urbič T., Mol. Phys., 2014, 112, 1132–1148, doi:10.1080/00268976.2013.836608.
  * [43] Kurtjak M., Urbič T., Mol. Phys., 2015, 113, 727–738, doi:10.1080/00268976.2014.973919.
  * [44] Urbič T., Vlachy V., Dill K. A., J. Phys. Chem. B, 2006, 110, 4963–4970, doi:10.1021/jp055543f.
  * [45] Urbič T., Mohoric T., J. Chem. Phys., 2017, 146, 094505, doi:10.1063/1.4977214.
  * [46] Ogrin P., Urbič T., J. Mol. Liq., 2021, 146, 114880, doi:10.1016/j.molliq.2020.114880.
  * [47] Ogrin P., Urbič T., J. Mol. Liq., 2021, 327, 114880, doi:10.1016/j.molliq.2020.114880.
  * [48] Delhommelle J., Millié P., Mol. Phys., 2001, 99, 619–625, doi:10.1080/00268970010020041.
  * [49] Hansen J. P., McDonald I. R., Theory of Simple Liquids, Academic, London, 1986.
  * [50] Frenkel D., Smit B., Molecular Simulation: From Algorithms to Applications, Academic Press, New York, 2000.
  * [51] Widom B., J. Chem. Phys., 1963, 39, 2808, doi:10.1063/1.1734110.
  * [52] Urbič T., Dill K. A., J. Am. Chem. Soc., 2018, 140, 17106–17113, doi:10.1021/jacs.8b08856.
  * [53] Brini E., Fennell C. J., Fernandez-Serra M., Hribar-Lee B., Lukšič M., Dill K. A., Chem. Rev., 2017, 117, 12385–12414, doi:10.1021/acs.chemrev.7b00259.

Ukrainian 0

Çàëåæíñòü ñîëüâàòàö¿ âä òèñêó äëÿ íåïîëÿðíèõ ðîçèíåíèõ ðåîâèí ó ïðîñòèõ
ìîäåëÿõ âîäè Ò. Óðá

Ôàêóëüòåò õì¿ òà õìíèõ òåõíîëîãé óíâåðñèòåòó Ëþáëÿíè, âóë. Âåíà 113, SI-1000,
Ëþáëÿíà, Ñëîâåíÿ


# Nested-TNT: Hierarchical Vision Transformers with Multi-Scale Feature
Processing

Yuang Liu1,*, Zhiheng Qiu1,*, Xiaokai Qin1,*,†

###### Abstract

Transformer has been applied in the field of computer vision due to its
excellent performance in natural language processing, surpassing traditional
convolutional neural networks and achieving new state-of-the-art. ViT divides
an image into several local patches, known as ”visual sentences”. However, the
information contained in the image is vast and complex, and focusing only on
the features at the ”visual sentence” level is not enough. The features
between local patches should also be taken into consideration. In order to
achieve further improvement, the TNT model is proposed, whose algorithm
further divides the image into smaller patches, namely ”visual words,”
achieving more accurate results. The core of Transformer is the Multi-Head
Attention mechanism, and traditional attention mechanisms ignore interactions
across different attention heads. In order to reduce redundancy and improve
utilization, we introduce the nested algorithm and apply the Nested-TNT to
image classification tasks. The experiment confirms that the proposed model
has achieved better classification performance over ViT and TNT, exceeding
2.25%, 1.1% on dataset CIFAR10 and 2.78%, 0.25% on dataset FLOWERS102
respectively.

## Introduction

The birth of Transformer (Vaswani et al. 2017) opens up the era of large
models and accelerates the development of the field of natural language
processing. The proposal of Transformer solves the drawbacks of traditional
convolutional neural networks (CNNs) (Lecun et al. 1998) that are weak in
capturing global features, and improves the generalisation ability of the
model. Compared with traditional recurrent neural networks, Transformer allows
parallel computation, which improves computational efficiency. The core of the
Transformer network is the self-attention mechanism, which focuses on the
dependency of each element on the global, i.e., it enables the model to pay
attention to the contextual information and allows bidirectional inference.

Inspired by Transformer’s great success in the field of natural language
processing, Vision Transformer (ViT) (Dosovitskiy et al. 2020) was born. As
the name suggests, ViT transfers the idea of Transformer to the field of
computer vision by segmenting an image into ”visual sentences” to be fed into
the Transformer network. ViT is widely used in the fields of image
classification, image segmentation, target detection and image generation. For
example, in the field of autonomous driving (Ando et al. 2023), ViT has been
applied to detect obstacles and traffic markers on the road. ViT has also been
applied to image restoration and resolution enhancement (Lu et al. 2022),
i.e., resolving a high-resolution version of a low-pixel image. ViT is also
commonly used for target detection and target recognition in the medical field
(Al-Hammuri et al. 2023), biometrics field (Sun and Tzimiropoulos 2022) and
video processing field (Arnab et al. 2021).

As mentioned earlier, ViT only splits images to the visual sentence level,
which is still a coarse-grained approach. Therefore, ViT has limited accuracy
in processing images, which can lead to loss of detailed features. In order to
solve this problem and retain as much information as possible in the image,
Transformer iN Transformer(TNT) (Han et al. 2021) was proposed. In TNT, image
patches are further divided into ”visual words”, which provide a fine-grained
approach to feature extraction.

On the other hand, the core of the Transformer architecture is a Multi-Head
Attention (MHA) mechanism where each head is used to capture image features.
In order to capture as many features as possible, a number of complex
attention paradigms have been born under the existing TNT framework. In order
to improve the efficiency of parameter utilization to reduce the model
complexity, Nested Vision Transformer (Nested ViT) (Peng et al. 2023) is
proposed. Nested ViT maximizes the efficiency of parameter utilization by
connecting the attention logistics of neighbouring layers of Transformers so
that each head focuses on complementary information.

TNT and Nested ViT optimize some of the shortcomings of ViT, respectively. In
order to combine the advantages of both we have designed a new method capable
of simultaneously improving task accuracy and reducing parameter redundancy,
called Nested-TNT. Nested-TNT refers to the idea of TNT and adds a layer of
fine-grained Transformer Block on the basis of ViT to complete the extraction
of image features in more details. In the original coarse-grained Transformer
Block, the connection between different layers is established by MHA
mechanism, which achieves the purpose of improving the efficiency of parameter
utilization and reducing redundancy. We apply Nested-TNT to an image
classification task and demonstrate its effectiveness experimentally.

## Related Works

Just like a transformer, Vision Transformer (ViT) divides the input image into
equally-sized patches, such as 16x16 pixels, which are analogous to
”sequences” in a text sequence. Each patch is then transformed into a vector
of a specific dimension through a linear projection, effectively creating an
embedding for each image patch. At the start of the sequence, a
”classification token” is introduced, and the ensuing sequence of vectors is
then processed through a typical Transformer encoder. However, since the
Transformer architecture inherently lacks the capability to process sequential
data, positional embeddings are added to these patch embeddings. This step is
crucial because it injects information into the model about the location of
each patch in the image. Following this, the combined embeddings (patch and
positional) are fed into several layers of the Transformer encoder. Within
each layer, a multi-head self-attention mechanism and a multi-layer perceptron
are employed. The output from the final Transformer encoder layer represents
the aggregated output of the entire ViT model. It outperforms the
convolutional neural networks (CNNs) on downstream tasks.

There is a lot of work dedicated to improving the performance and efficiency
of ViT. For example, one team combined Convolutions and Vision Transformers to
generate a new network CvT (Wu et al. 2021), which improves the performance
compared to CNN-based models (e.g. ResNet) and prior Transformer-based models
(e.g. ViT, DeiT) while utilising fewer FLOPS and parameters. Another example
is LeViT (Graham et al. 2021), which provides a better trade-off between
accuracy and inference time, especially for small and medium-sized
architectures. There also a team proposed the design of the Multiscale Visual
Transformer (MViT) (Fan et al. 2021), which introduces a multiscale feature
hierarchy to the transformer. It uses the Multiple Heads Pooling Attention
(MHPA) mechanism to allow flexible resolution modelling in the converter
block. There are more relevant studies, such as Mvitv2 (Li et al. 2022), YOLOS
(Fang et al. 2021), Pyramid vision transformer (Wang et al. 2021), Reversible
vision transformers (Mangalam et al. 2022), Scaling vision transformers (Zhai
et al. 2022), Segmenter (Strudel et al. 2021) etc., which will not be expanded
in the space limit.

Besides, among related studies, Transformer iN Transformer and Nested Vision
Transformer caught our attention.

In ViT, the image is divided into local patches of 16x16 pixels, this coarse-
grained division ignores more features and details. In order to obtain more
accurate results, the Transformer iN Transformer (TNT) architecture is
proposed. TNT further divides each local patch into smaller patches (4x4
pixels), i.e., each visual sentence is divided into visual words. Visual
sentence inside the visual words, and then performs feature extraction and
training in the next dimension at the visual sentence level. The algorithm
achieves results beyond ViT.

The Multi-Head Attention (MHA) mechanism is the basis of the Transformer and
Vision Transformer. As the network structure continues to be reconstructed and
overlaid, the number of head parameters becomes increasingly large. However,
it is not always necessary to have so many header parameters. Peng’s team
(Peng et al. 2023) noticed that if it is possible to make each head pay
attention to as much complementary information as possible, we can maximize
the efficiency of parameter utilization and reduce the redundancy. Based on
the ”explaining-away” effect, they found a way to make the features of each
head complementary called Nested Vision Transformer (Nested ViT). They
established connections between layers of the Transformer to enable the
network to capture both detailed and global features, and experimentally
demonstrated that this approach effectively reduces parameter redundancy.

Figure 1: Nested Multi-head Attention TNT Structure

## Methodology

Firstly, we will provide a brief outline of the structure underlying multi-
head attention. Multi-Head Attention algorithm sets independent queries, keys
and values matrices for every single head and executes the attention procedure
simultaneously across these heads. Subsequently, it combines the output values
from each head through concatenation, followed by linear projection to
generate the ultimate output. If we use $Q_{i}$, $K_{i}$, $V_{i}$ and $H_{i}$
to represent queries, keys, values and the output of a single head, then the
calculation process is shown in the following formula:

$H_{i}=softmax\left(\frac{Q_{i}K_{i}^{T}}{\sqrt{d}}\right)V(1)$

Through linear projection, we can reach the final output of multi-head
attention: $\sum_{i=1}^{h}H_{i}W_{i}^{o}$, where $W_{i}^{o}$ are learnable
weights.

An overview of our model is depicted in Figure 1. Our usage of transformer is
partly based on TNT (Han et al. 2021). In total, there are L layers. Each
layer comprises multiple inner transformers, treating words as basic units,
along with an outer transformer focusing on sentences as basic units. Standard
learnable 1D position encodings are applied both in word embeddings and
sentence embeddings to keep spatial information. Sentence position encoding
can maintain the global spatial information, while word position encoding is
used for preserving the local relative position, and the word position
encodings are shared across sentences.

To be more specific, the input picture is initially divided into n patches as
ViT (Dosovitskiy et al. 2020) does and each patch is regarded as a visual
sentence. Then each visual sentence is split into a sequence of visual words.
After word embedding, the vectors are sent to inner transformer blocks, whose
purpose is to learn the relationships among visual words. For an embedded
visual sentence $Y_{l}^{i}=[y_{l}^{i,1},y_{l}^{i,2},...,y_{l}^{i,m}]$,
$l=1,2,...,L$ is the index of the block, where $i$ is the index of visual
sentence and there are $m$ visual words in the sentence. The calculation
process of the inner transformer can be expressed by the following formula,
where $MSA()$ is the multi-head attention process, $LN()$ is the layer
normalization process and $MLP()$ stands for the layer applied between self-
attention layers for feature transformation and non-linearity.

${Y^{{}^{\prime}}}_{l}^{i}=Y_{l-1}^{i}+MSA(LN(Y_{l-1}^{i}))(2)$
$Y_{l}^{i}={Y^{{}^{\prime}}}_{l}^{i}+MLP(LN(Y_{l-1}^{i}))(3)$

After transformation of inner transformers, we have the expression for all
word embeddings:$\gamma_{l}=[Y_{l}^{1},Y_{l}^{2},...,Y_{l}^{n}]$.

At sentence level, we do sentence embedding and the procedure can be depicted
as $\beta_{0}=[Z_{class},Z_{0}^{1},...,Y_{0}^{n}]$ where $Z_{class}$ is the
class token. Before entering outer transformer, the sentence embedding should
be augmented by word-level features learned by inner transformers:

$Z_{l-1}^{i}=FC(Vec(Y_{l}^{i}))(4)$

$FC()$ stands for fully connected layer and $Vec(\textperiodcentered)$ is the
vectorization operation. Then we use outer transformer block to deal with the
sentence embeddings:

${Z^{{}^{\prime}}}_{l}=Z_{l-1}+MSA^{{}^{\prime}}(LN(Z_{l-1}))(5)$
$Z_{l}={Z^{{}^{\prime}}}_{l}+MLP(LN(Z_{l-1}))(6)$

where $MSA^{{}^{\prime}}()$ stands for nested multi-head attention mechanism.
The mechanism creates a new data path so that the attention logits of adjacent
layers’ outer transformer blocks can communicate directly. The addition of
this mechanism was inspired by Peng’s team(Peng et al. 2023), who also pointed
out the impact of the explaining-away effects when stacking transformer
blocks. Specifically, when an attention head focuses on parts of the input
that are highly relevant to the output, it prevents other heads from doing the
same things. On the contrary, it encourages them to focus on supplementary
information. This means that proper connections between the heads will help
reduce the redundancy and improve the efficiency of model parameters. Multi-
head attention mechanism provides communication methods for different
attention heads within each layer, but the connection between heads from
different layers is still weak. The data path uses a multi-layer perceptron
layer to fuse the attention logits from adjacent layers:

$A_{outer}^{l}=softmax(z^{l}_{outer}+MLP(z^{l}_{outer},z^{l-1}_{outer}))(7)$

where $z^{l}_{outer}$ stands for the attention logits of the out transformer
of layer l and $A_{outer}^{l}$ is the attention weights of the out transformer
of layer l. During training, softmax cross-entropy loss function and gradient
descent optimizer are applied to update the parameters.

To sum up, according to the structure shown in Figure 1 and the formulas
listed above, the inputs and outputs of Nested TNT can be formulated as

$\gamma_{l},\beta_{l},z_{outer}^{l}=Nested\\_TNT(\gamma_{l-1},\beta_{l-1},z_{outer}^{l-1})(8)$

## Experiments

In this section, we conduct experiments on several visual datasets to evaluate
the classification ability of the proposed Nested Multi-head Attentions TNT
architecture. Moreover, two baseline models (ViT(Dosovitskiy et al. 2020), TNT
(Han et al. 2021)) are chosen as comparison targets and also adopted with the
same experiments.

### Datasets

CIFAR100, CIFAR10 (Krizhevsky, Hinton et al. 2009), are two image
classification benchmarks including 50000 train and 10000 test images
belonging to 100 and 10 classes respectively. The images included in these two
datasets are in the same size, 32x32 with 3 color channels. All the classes
have same number of images, which helps avoid bias in the training process.
CIFAR100 is used as pre-train dataset while CIFAR10 is used as fine-tune
dataset.

In addition to CIFAR10, Flowers102 (Gogul and Kumar 2017) is the other fine-
tune dataset, where some flower categories are very similar in appearance.
This is a challenge for classification algorithms and requires models that can
capture and learn subtle feature differences.

Adopted data augmentation strategies are similar to those in Nested-Attention
ViT (Peng et al. 2023) implementation, e.g., random crop, random flip, mixup
and bicubic train-interpolation(Chen et al. 2023). The details of visual
datasets are shown in Table 1.

Dataset | Type | Train Size | Test Size | Classes
---|---|---|---|---
Cifar100 | Pre-train | 50000 | 10000 | 100
Cifar10 | Fine-tune | 50000 | 10000 | 10
Flowers102 | Fine-tune | 6149 | 2040 | 102
Table 1: Details of Used Datasets.

### Experimental Settings

The 3 models chosen in our experiment are ViT-S, TNT-S, and small size of the
proposed architecture, which all process images at 224x224-pixels resolution.
Moreover, the main settings for hyper-parameters in pre-training and fine-
tuning are shown in tables 2 and 3 respectively.

Epochs | Optimizer | Batch Size
---|---|---
150 | SGD | 32
Learning Rate | Learning Decay | Weight Decay
5e-2 | Cosine | 1e-4
Warmup epochs | Label Smooth | train-interpolation
3 | 0.1 | bicubic
Table 2: Pre-Train Hyper-Parameters. Epochs | Optimizer | Batch Size
---|---|---
100 | SGD | 32
Learning Rate | Learning Decay | Weight Decay
5e-2 | Cosine | 1e-4
Warmup epochs | Label Smooth | train-interpolation
3 | 0.1 | bicubic
Table 3: Fine-Tune Hyper-Parameters.

### Results

The parameter complexity for three models is shown in Table 4. Nested-TNT
model has the highest parameter complexity, but only a little higher than that
of TNT. And it is obviously more complex than that of VIT, which is one of the
sacrifices our model makes for the following advantages.

Models | ViT | TNT | Nested-TNT
---|---|---|---
Params(M) | 21.67 | 23.41 | 23.42
Table 4: Parameters for Three Models.

The results of classification accuracy are shown in Table 5. The accuracy
values are all in top-1 accuracy form. The Nested-TNT outperforms the other
two visual transformer models in terms of final accuracy, although it is
reached at the expense of number of images processed per second. For CIFAR100
and CIFAR10, Nested-Attention TNT’s better performance comes from the
introduced Nested-Attention, which enhances links between global modules.
Then, better links create better recognition.

For Flowers102, the higher accuracy means Nested-Attention TNT has better
performance in fine-grained analysis. This comes from the added inner blocks
in TNT, which helps captures localized details of an image. As a result, it
gets a result that is similar to that of TNT and 3% higher than that of ViT.

Model | Cifar100 | CIFAR10 | Flower | im/sec
---|---|---|---|---
ViT | 66.98 | 91.28 | 90.49 | 522
TNT | 69.63 | 92.43 | 93.02 | 190
Nested-TNT | 70.59 | 93.53 | 93.27 | 137
Table 5: Accuracy for Three Models.

## Conclusion

We propose a new Vision Transformer model, Nested-TNT, based on nested multi-
head attention mechanism and principle of Transformer iN Transformer (TNT)
model. The nested multi-head attention mechanism improves independence between
different attention heads, while the TNT model cuts the image patches into
smaller patches. Then, the experiment results show that Nested-TNT performs
better on image classification task. It demonstrates its ability to enhance
both the detailed and the global features at the same time, which satisfies
our basic targets.

## Limitation and Future Work

Although our proposed model has a better performance on the image
classification accuracy, the image processing speed and parameter complexity
are not so optimistic. Also, because of the time and computing limitation, the
amount of pre-train is insufficient.

Thus, in the future, firstly, the extension of the Nested-TNT can be focused
on the simplification of the algorithms and the optimization of connection
layers. After that, we can apply more pre-train to get better performance. The
pre-train dataset can be switched to a larger one, e.g., ImageNet.
Furthermore, the experiments were limited to categorization tasks, which can
be extended to Objective Detection and Semantic Segmentation fields. Ablation
study can be taken into consideration to check each part’s ability as well.

## References

  * Al-Hammuri et al. (2023) Al-Hammuri, K.; Gebali, F.; Kanan, A.; and Chelvan, I. T. 2023.  Vision transformer architecture and applications in digital health: a tutorial and survey.  _Visual Computing for Industry, Biomedicine, and Art_ , 6(1): 14.
  * Ando et al. (2023) Ando, A.; Gidaris, S.; Bursuc, A.; Puy, G.; Boulch, A.; and Marlet, R. 2023.  RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5240–5250.
  * Arnab et al. (2021) Arnab, A.; Dehghani, M.; Heigold, G.; Sun, C.; Lučić, M.; and Schmid, C. 2021.  Vivit: A video vision transformer.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 6836–6846.
  * Chen et al. (2023) Chen, J.; Hu, S.; Pan, L.; and Wang, X. 2023.  Nonuniform quadrupolar orders in the spin-3/2 generalized Heisenberg chain.  arXiv:2311.08099.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2020.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  _CoRR_ , abs/2010.11929.
  * Fan et al. (2021) Fan, H.; Xiong, B.; Mangalam, K.; Li, Y.; Yan, Z.; Malik, J.; and Feichtenhofer, C. 2021.  Multiscale vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 6824–6835.
  * Fang et al. (2021) Fang, Y.; Liao, B.; Wang, X.; Fang, J.; Qi, J.; Wu, R.; Niu, J.; and Liu, W. 2021.  You only look at one sequence: Rethinking transformer in vision through object detection.  _Advances in Neural Information Processing Systems_ , 34: 26183–26197.
  * Gogul and Kumar (2017) Gogul, I.; and Kumar, V. S. 2017.  Flower species recognition system using convolution neural networks and transfer learning.  In _2017 fourth international conference on signal processing, communication and networking (ICSCN)_ , 1–6. IEEE.
  * Graham et al. (2021) Graham, B.; El-Nouby, A.; Touvron, H.; Stock, P.; Joulin, A.; Jégou, H.; and Douze, M. 2021.  Levit: a vision transformer in convnet’s clothing for faster inference.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 12259–12269.
  * Han et al. (2021) Han, K.; Xiao, A.; Wu, E.; Guo, J.; Xu, C.; and Wang, Y. 2021.  Transformer in Transformer.  _CoRR_ , abs/2103.00112.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Lecun et al. (1998) Lecun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11): 2278–2324.
  * Li et al. (2022) Li, Y.; Wu, C.-Y.; Fan, H.; Mangalam, K.; Xiong, B.; Malik, J.; and Feichtenhofer, C. 2022.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4804–4814.
  * Lu et al. (2022) Lu, Z.; Li, J.; Liu, H.; Huang, C.; Zhang, L.; and Zeng, T. 2022.  Transformer for single image super-resolution.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 457–466.
  * Mangalam et al. (2022) Mangalam, K.; Fan, H.; Li, Y.; Wu, C.-Y.; Xiong, B.; Feichtenhofer, C.; and Malik, J. 2022.  Reversible vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10830–10840.
  * Peng et al. (2023) Peng, J.; Li, C.; , Y. Z.; Lin, Y.; fang, x.; and Gong, J. 2023.  Improving Vision Transformers with Nested Multi-head Attentions.  In _ICME_.
  * Strudel et al. (2021) Strudel, R.; Garcia, R.; Laptev, I.; and Schmid, C. 2021.  Segmenter: Transformer for semantic segmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 7262–7272.
  * Sun and Tzimiropoulos (2022) Sun, Z.; and Tzimiropoulos, G. 2022.  Part-based face recognition with vision transformers.  _arXiv preprint arXiv:2212.00057_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2021) Wang, W.; Xie, E.; Li, X.; Fan, D.-P.; Song, K.; Liang, D.; Lu, T.; Luo, P.; and Shao, L. 2021.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 568–578.
  * Wu et al. (2021) Wu, H.; Xiao, B.; Codella, N.; Liu, M.; Dai, X.; Yuan, L.; and Zhang, L. 2021.  Cvt: Introducing convolutions to vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 22–31.
  * Zhai et al. (2022) Zhai, X.; Kolesnikov, A.; Houlsby, N.; and Beyer, L. 2022.  Scaling vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 12104–12113.

## About Authors

This research was jointly completed by three individuals.

Yuang Liu has a bachelor degree in Electronics and Computer Science from
University of Edinburgh. Now he is pursuing his Master’s degree in Artificial
Intelligence at Nanyang Technological University.

Zhiheng Qiu has a Bachelor’s Degree in Computer Science and Technology from
Zhejiang University, and pursuing a Master’s degree in Artificial Intelligence
at Nanyang Technological University currently.

Xiaokai Qin has a B.Sc. degree in Computer Science from University of
Liverpool and B.Sc. degree in Information and Computing Science from Xi’an
Jiaotong Liverpool University. Presently, he is pursuing his Master’s degree
in Artificial Intelligence at Nanyang Technological University, Singapore.


$\displaystyle
dP_{\varphi(t+h)}-dP_{\varphi(t)}=\int_{\varphi(t)}^{\varphi(t+h)}\dot{\rho}_{s,u}\,ds$

for any $u\in\mathbb{S}^{d-1}$. Using this result, the chi-squared divergence
can be written out as follows:

$\displaystyle\chi^{2}(P_{\varphi(t+h)}\|P_{\varphi(t)})$
$\displaystyle=\int\frac{\big{(}dP_{\varphi(t+h)}-dP_{\varphi(t)}\big{)}^{2}}{dP_{\varphi(t)}}\,d\nu$
$\displaystyle=\int\frac{\big{(}\int_{\varphi(t)}^{\varphi(t+h)}\dot{\rho}_{s,u}\,ds\big{)}^{2}}{dP_{\varphi(t)}}\,d\nu$
$\displaystyle=\int\frac{\left\\{\int_{0}^{1}(\nabla\varphi(t+h\widetilde{s})h)^{\top}\dot{\rho}_{\varphi(t+h\widetilde{s}),u}\,d\widetilde{s}\right\\}^{2}}{dP_{\varphi(t)}(x)}\,d\nu$
$\displaystyle=h^{\top}\left(\int\frac{\int_{0}^{1}\int_{0}^{1}\nabla\varphi^{\top}(t+h\widetilde{s}_{1})\dot{\rho}_{\varphi(t+h\widetilde{s}_{1}),u}\dot{\rho}^{\top}_{\varphi(t+h\widetilde{s}_{2}),u}\nabla\varphi(t+h\widetilde{s}_{2})\,d\widetilde{s}_{1}\,d\widetilde{s}_{2}}{dP_{\varphi(t)}}\,d\nu\right)h.$

The last steps use the change of variables. By the uniform integrability
assumption and the continuity of $\dot{\rho}_{t}$, both asserted by (A1), it
follows that

$\displaystyle\chi^{2}(P_{\varphi(t+h)}\|P_{\varphi(t)})$
$\displaystyle=h^{\top}\nabla\varphi(t)^{\top}\left(\int\frac{\nabla\dot{\rho}_{\varphi(t)}\nabla\dot{\rho}^{\top}_{\varphi(t)}}{dP_{\varphi(t)}}\,d\nu\right)\nabla\varphi(t)h+o(\|h\|_{2}^{2})$

by the dominated convergence. As the middle term in the parenthesis is finite
under (A1), Lemma 6 of Gassiat and Stoltz,  (2024) implies that the path
$dP^{1/2}_{\varphi(t)}$ is Hellinger differentiable with the gradient

$\dot{\xi}_{\varphi(t)}:=\frac{1}{2}\frac{\nabla\dot{\rho}_{\varphi(t)}}{dP^{1/2}_{\varphi(t)}}I(dP_{\varphi(t)}>0).$

We can use this to show that:

$\displaystyle\mathcal{I}(\varphi(t))=4\int\dot{\xi}_{\varphi(t)}\dot{\xi}_{\varphi(t)}^{\top}\,d\nu=\left(\int\frac{\nabla\dot{\rho}_{\varphi(t)}\nabla\dot{\rho}^{\top}_{\varphi(t)}}{dP_{\varphi(t)}}\,d\nu\right)$

and this leads us to conclude

$\displaystyle\chi^{2}(P_{\varphi(t+h)}\|P_{\varphi(t)})$
$\displaystyle=h^{\top}\varphi(t)^{\top}\mathcal{I}(\varphi(t))\nabla\varphi(t)h+o(\|h\|_{2}^{2}).$

as desired. ∎

######  Lemma 20.

Assuming that the collection of paths $h\mapsto dP_{\varphi(t+h)}$ is
Hellinger differentiable at $h=0$ and $\nabla\varphi$ is continuous at $t$,

$\displaystyle H^{2}\left(P_{\varphi(t+h)},P_{\varphi(t)}\right)$
$\displaystyle=\frac{1}{4}h^{\top}\nabla\varphi(t)^{\top}\mathcal{I}(\varphi(t))\nabla\varphi(t)h^{\top}+o(\|h\|_{2}^{2})$

as $\|h\|_{2}\longrightarrow 0$.

###### Proof of Lemma 20.

By the definition of the Hellinger differentiability, we have

$\displaystyle H^{2}\left(P_{\varphi(t+h)},P_{\varphi(t)}\right)$
$\displaystyle=\int\left(dP^{1/2}_{\varphi(t+h)}-dP^{1/2}_{\varphi(t)}\right)^{2}$
$\displaystyle=\int\left(dP^{1/2}_{\varphi(t)+\varphi(t+h)-\varphi(t)}-dP^{1/2}_{\varphi(t)}\right)^{2}$
$\displaystyle=\int\left((\varphi(t+h)-\varphi(t))^{\top}\dot{\xi}_{\varphi(t)}\right)^{2}+o(\|\varphi(t+h)-\varphi(t)\|_{2}^{2})$
$\displaystyle=\int\left((\nabla\varphi(t+\lambda
h)h)^{\top}\dot{\xi}_{\varphi(t)}\right)^{2}+o(\|\nabla\varphi(t+\lambda
h)h)\|_{2}^{2})$ $\displaystyle=h^{\top}\nabla\varphi^{\top}(t+\lambda
h)\left(\int\dot{\xi}_{\varphi(t)}\dot{\xi}_{\varphi(t)}^{\top}\,d\nu\right)\nabla\varphi(t+\lambda
h)h^{\top}+o(\|\nabla\varphi(t+\lambda h)h)\|_{2}^{2})$

for some constant $\lambda\in[0,1]$ possibley depending on $t$. Since the
Fisher information is defined as
$4\int\dot{\xi}_{\varphi(t)}\dot{\xi}_{\varphi(t)}^{\top}\,d\nu$ and
$\nabla\varphi(t+\lambda h)\longrightarrow\nabla\varphi(t)$ as
$\|h\|_{2}\longrightarrow 0$ by the continuity, we conclude that

$\displaystyle H^{2}\left(P_{\varphi(t+h)},P_{\varphi(t)}\right)$
$\displaystyle=\frac{1}{4}h^{\top}\nabla\varphi(t)^{\top}\mathcal{I}(\varphi(t))\nabla\varphi(t)h^{\top}+o(\|h\|_{2}^{2})$

as $\|h\|_{2}\longrightarrow 0$.

∎

######  Lemma 21.

Assuming that the collection of paths $h\mapsto P_{\varphi(t+h)}$ satisfies
(A2),

$\displaystyle\chi^{2}(\widetilde{\mathbb{P}}^{n}_{h}\|\lambda\widetilde{\mathbb{P}}^{n}_{h}+(1-\lambda)\widetilde{\mathbb{P}}^{n}_{0})$
$\displaystyle\qquad=(1-\lambda)^{2}h^{\top}\left\\{\mathcal{I}(Q)+n\int\nabla\varphi(t)^{\top}\,\left(\mathcal{I}(\varphi(t))+\frac{1-4\lambda}{4\lambda}\mathcal{I}^{\circ}(\varphi(t))\right)\,\nabla\varphi(t)\,dQ(t)\right\\}h+o(\|h\|_{2}^{2}).$

###### Proof of Lemma 21.

The analogous proof can be found in Section 7.14 of Polyanskiy and Wu,
(2022). To begin, we observe that the $\chi^{2}$-divergence can be written out
as

$\displaystyle\chi^{2}(\widetilde{\mathbb{P}}^{n}_{h}\|\lambda\widetilde{\mathbb{P}}^{n}_{h}+(1-\lambda)\widetilde{\mathbb{P}}^{n}_{0})$
$\displaystyle=\iint_{\mathcal{X}\times\Theta}\frac{\\{d\widetilde{\mathbb{P}}^{n}_{h}-(\lambda\widetilde{\mathbb{P}}^{n}_{h}+(1-\lambda)\widetilde{\mathbb{P}}^{n}_{0})\\}^{2}}{\lambda
d\widetilde{\mathbb{P}}^{n}_{h}+(1-\lambda)d\widetilde{\mathbb{P}}^{n}_{0}}$
$\displaystyle=(1-\lambda)^{2}\iint_{\mathcal{X}\times\Theta}\frac{(d\widetilde{\mathbb{P}}^{n}_{h}-d\widetilde{\mathbb{P}}^{n}_{0})^{2}}{\lambda
d\widetilde{\mathbb{P}}^{n}_{h}+(1-\lambda)d\widetilde{\mathbb{P}}^{n}_{0}}.$

We will consider the function

$\phi:=f\mapsto\iint_{\mathcal{X}\times\Theta}\frac{f^{2}-d\widetilde{\mathbb{P}}^{n}_{0}}{\\{\lambda
f^{2}+(1-\lambda)d\widetilde{\mathbb{P}}^{n}_{0}\\}^{1/2}}.$

It can then be seen that
$\chi^{2}(\widetilde{\mathbb{P}}^{n}_{h}\|\lambda\widetilde{\mathbb{P}}^{n}_{h}+(1-\lambda)\widetilde{\mathbb{P}}^{n}_{0})=(1-\lambda)^{2}\iint_{\mathcal{X}\times\Theta}\phi(d\widetilde{\mathbb{P}}^{n/2}_{h})^{2}$.
When $\phi$ is Lipshitz continuous and $d\widetilde{\mathbb{P}}^{n/2}_{h}$ is
absolutely continuous with the gradient $t\mapsto\dot{\gamma}_{t,u}$, then the
composition function $\phi(d\widetilde{\mathbb{P}}^{n/2}_{h})$ is also
absolutely continuous with the gradient
$t\mapsto\phi^{\prime}(d\widetilde{\mathbb{P}}^{n/2}_{t})\dot{\gamma}_{t,u}$.
In Section 7.14 of Polyanskiy and Wu,  (2022), it is shown that $\phi$ is
indeed Lipshitz continuous with the constant
$(2-\lambda)/\\{(1-\lambda)\lambda^{1/2}\\}$. Additionally, $\phi$ has a
continuous derivative and we have

$\displaystyle\phi^{\prime}(d\widetilde{\mathbb{P}}^{n}_{0}):=\begin{cases}2,&\mbox{if}\quad(x,t):d\widetilde{\mathbb{P}}^{n}_{0}(x,t)>0\\\
\lambda^{-1/2}&\mbox{if}\quad(x,t):d\widetilde{\mathbb{P}}^{n}_{0}(x,t)=0.\end{cases}$

Putting together, we have

$\lim_{t\longrightarrow
0}\phi^{\prime}(d\widetilde{\mathbb{P}}^{n/2}_{t})\dot{\gamma}_{t,u}=\dot{\gamma}_{0,u}\left(2I(d\widetilde{\mathbb{P}}^{n}_{0}>0)+\frac{1}{\lambda^{1/2}}I(d\widetilde{\mathbb{P}}^{n}_{0}=0)\right).$

Noting that $\phi(d\widetilde{\mathbb{P}}^{n/2}_{0})=0$, it follows that

$\displaystyle\chi^{2}(\widetilde{\mathbb{P}}^{n}_{h}\|\lambda\widetilde{\mathbb{P}}^{n}_{h}+(1-\lambda)\widetilde{\mathbb{P}}^{n}_{0})$
$\displaystyle=(1-\lambda)^{2}\iint_{\mathcal{X}\times\Theta}\phi(d\widetilde{\mathbb{P}}^{n/2}_{h})^{2}$
$\displaystyle=(1-\lambda)^{2}\iint_{\mathcal{X}\times\Theta}\left(\phi(d\widetilde{\mathbb{P}}^{n/2}_{h})-\phi(d\widetilde{\mathbb{P}}^{n/2}_{0})\right)^{2}$
$\displaystyle=(1-\lambda)^{2}\iint_{\mathcal{X}\times\Theta}\left(\int_{0}^{h}\phi^{\prime}(d\widetilde{\mathbb{P}}^{n/2}_{s})\dot{\gamma}_{s,u}\,ds\right)^{2}$
$\displaystyle=(1-\lambda)^{2}h^{\top}\iint_{\mathcal{X}\times\Theta}\left(\int_{0}^{1}\int_{0}^{1}\phi^{\prime}(d\widetilde{\mathbb{P}}^{n/2}_{hs_{1}})\phi^{\prime}(d\widetilde{\mathbb{P}}^{n/2}_{hs_{2}})\dot{\gamma}_{hs_{1},u}\dot{\gamma}^{\top}_{hs_{2},u}\,ds_{1}\,ds_{1}\right)h.$

Since $\phi^{\prime}$ is continuous and $\dot{\gamma}$ is the uniform
integrability, asserted by (A2), it follows that

$\displaystyle\chi^{2}(\widetilde{\mathbb{P}}^{n}_{h}\|\lambda\widetilde{\mathbb{P}}^{n}_{h}+(1-\lambda)\widetilde{\mathbb{P}}^{n}_{0})$
$\displaystyle\qquad=(1-\lambda)^{2}h^{\top}\left\\{\iint_{\mathcal{X}\times\Theta}\dot{\gamma}_{0}\dot{\gamma}^{\top}_{0}\left(4I(d\widetilde{\mathbb{P}}^{n}_{0}>0)+\frac{1}{\lambda}I(d\widetilde{\mathbb{P}}^{n}_{0}=0)\right)\right\\}h+o(\|h\|_{2}^{2})$
$\displaystyle\qquad=(1-\lambda)^{2}h^{\top}4\left\\{\iint_{\mathcal{X}\times\Theta}\dot{\gamma}_{0}\dot{\gamma}^{\top}_{0}+\iint_{\mathcal{X}\times\Theta}\dot{\gamma}_{0}\dot{\gamma}^{\top}_{0}\left(\frac{1-4\lambda}{4\lambda}I(d\widetilde{\mathbb{P}}^{n}_{0}=0)\right)\right\\}h+o(\|h\|_{2}^{2})$

as $\|h\|_{2}\longrightarrow 0$ by the dominated convergence theorem. Finally,
Lemma 23 states that the Hellinger differentiablity is implied by (A2), and
that the corresponding Fisher information can be computed using
$4\iint_{\mathcal{X}\times\Theta}\dot{\gamma}_{0}\dot{\gamma}^{\top}_{0}$. By
the definition of the Hellinger differentiability, it follows that

$\displaystyle
H^{2}(\widetilde{\mathbb{P}}^{n}_{h},\widetilde{\mathbb{P}}^{n}_{0})=\int\left(d\widetilde{\mathbb{P}}_{h}^{n/2}-d\widetilde{\mathbb{P}}_{0}^{n/2}\right)^{2}=h^{\top}\left(\iint_{\mathcal{X}\times\Theta}\dot{\gamma}_{0}\dot{\gamma}_{0}^{\top}\right)h+o(\|h\|_{2}^{2})$

as $\|h\|_{2}\longrightarrow 0$. Thus in view of Lemma 22, we have

$\iint_{\mathcal{X}\times\Theta}\dot{\gamma}_{0}\dot{\gamma}_{0}^{\top}=\frac{1}{4}\left(\mathcal{I}(Q)+n\int_{\mathbb{R}^{d}}\nabla\varphi(t)^{\top}\,\mathcal{I}(\varphi(t))\,\nabla\varphi(t)\,dQ(t)\right)$

and similarly

$\iint_{\mathcal{X}\times\Theta}\dot{\gamma}_{0}\dot{\gamma}_{0}^{\top}I(d\widetilde{\mathbb{P}}^{n}_{0}=0)=\frac{1}{4}\left(\mathcal{I}^{\circ}(Q)+n\int_{\mathbb{R}^{d}}\nabla\varphi(t)^{\top}\,\mathcal{I}^{\circ}(\varphi(t))\,\nabla\varphi(t)\,dQ(t)\right)$

where
$\mathcal{I}^{\circ}(0):=4\int\dot{\gamma}_{0}\,\dot{\gamma}_{0}^{\top}I(d\widetilde{\mathbb{P}}^{n}_{0}=0)$
is the Fisher defect. When the prior $Q$ has a continuously differentiable
density $q$, it must follow that $\nabla q(t)=0$ whenever $q(t)=0$. This
implies that the Fisher defect $\mathcal{I}^{\circ}(Q)$ is zero. Thereby, we
conclude that

$\displaystyle\chi^{2}(\widetilde{\mathbb{P}}^{n}_{h}\|\lambda\widetilde{\mathbb{P}}^{n}_{h}+(1-\lambda)\widetilde{\mathbb{P}}^{n}_{0})$
$\displaystyle\qquad=(1-\lambda)^{2}h^{\top}\left\\{\mathcal{I}(Q)+n\int\nabla\varphi(t)^{\top}\,\left(\mathcal{I}(\varphi(t))+\frac{1-4\lambda}{4\lambda}\mathcal{I}^{\circ}(\varphi(t))\right)\,\nabla\varphi(t)\,dQ(t)\right\\}h+o(\|h\|_{2}^{2}).$

as desired. ∎

######  Lemma 22.

Assuming $Q$ has a density function that is bounded and continuously
differentiable, and $P_{t}$ is Hellinger differentiable at $t=\varphi(t)$ such
that $\|\nabla\varphi\|_{\infty}<C$,

$\displaystyle
H^{2}(\widetilde{\mathbb{P}}^{n}_{h},\widetilde{\mathbb{P}}^{n}_{0})=\frac{1}{4}h^{\top}\left(\mathcal{I}(Q)+n\int_{\mathbb{R}^{d}}\nabla\varphi(t)^{\top}\,\mathcal{I}(\varphi(t))\,\nabla\varphi(t)\,dQ(t)\right)h+o(\|h\|_{2}^{2})$

###### Proof of Lemma 22.

First, the Hellinger distance between $\widetilde{\mathbb{P}}^{n}_{h}$ and
$\widetilde{\mathbb{P}}^{n}_{0}$ can be written out as

$\displaystyle H^{2}(\widetilde{\mathbb{P}}_{h},\widetilde{\mathbb{P}}_{0})$
$\displaystyle=2-2\iint
dP^{n/2}_{\varphi(t+h)}\,dQ^{1/2}(t+h)\,dP^{n/2}_{\varphi(t)}\,dQ^{1/2}(t)\,$
$\displaystyle=2-2\int_{\mathbb{R}^{d}}dQ^{1/2}(t+h)dQ^{1/2}(t)\int_{\mathcal{X}}dP^{n/2}_{\varphi(t+h)}\,dP^{n/2}_{\varphi(t)}$
$\displaystyle=2-2\int_{\mathbb{R}^{d}}dQ^{1/2}(t+h)\,dQ^{1/2}(t)\left(1-\frac{1}{2}H^{2}(P^{n}_{\varphi(t+h)},P^{n}_{\varphi(t)})\right)$
$\displaystyle=H^{2}(Q_{h},Q)+\int_{\mathbb{R}^{d}}\,dQ^{1/2}(t+h)dQ^{1/2}(t)\,H^{2}(P^{n}_{\varphi(t+h)},P^{n}_{\varphi(t)}).$

When the density function $Q$ is continuously differentiable, the induced
location family is differentiable in quadratic mean (Example 7.8 of Van der
Vaart,  (2000)), which implies the Hellinger differentiability. By the
analogous result to Lemma 20 for $\varphi(t)=t$, we have

$\displaystyle
H^{2}(Q_{h},Q)=\frac{1}{4}h^{\top}\mathcal{I}(Q)h+o(\|h\|_{2}^{2}).$

Next, using the tensorization property of the Hellinger distance as well as
Lemma 20, it implies

$\displaystyle
H^{2}\left(P^{n}_{\varphi(t)},P^{n}_{\varphi(t+h)}\right)=2-2\left(1-\frac{H^{2}(P_{\varphi(t)},P_{\varphi(t+h)})}{2}\right)^{n}=2-2\left(1-h^{\top}Ch+o(\|h\|_{2}^{2})\right)^{n}$

as $\|h\|_{2}\longrightarrow 0$ where
$C:=\frac{1}{8}u^{\top}\nabla\varphi(t)^{\top}\mathcal{I}(\varphi(t))\nabla\varphi(t)u$.
By treating $n\geq 1$ as a fixed constant as $\|h\|_{2}\longrightarrow 0$, we
have

$\displaystyle
2-2\left(1-h^{\top}Ch+o(\|h\|_{2}^{2})\right)^{n}=2-2(1-nh^{\top}Ch+o(n\|h\|_{2}^{2}))=2nh^{\top}Ch+o(n\|h\|_{2}^{2}),$

by a first-order Taylor expansion. Putting them together, we have

$\displaystyle
H^{2}(\widetilde{\mathbb{P}}^{n}_{h},\widetilde{\mathbb{P}}^{n}_{0})=\frac{1}{4}h^{\top}\left(\mathcal{I}(Q)+n\int_{\mathbb{R}^{d}}\nabla\varphi(t)^{\top}\,\mathcal{I}(\varphi(t))\,\nabla\varphi(t)\,dQ(t)\right)h+o(\|h\|_{2}^{2}).$

where we invoke the dominated convergence theorem to interchange the limit and
integration operations, which follows by the boundedness of $\nabla\varphi$
and $\mathcal{I}(\cdot)$. ∎

######  Lemma 23.

If $\\{P_{t}:t\in\Theta\\}$ satisfies (A2) for some $\delta>0$, it is
Hellinger differentiable at $t=0$ with the corresponding Fisher information is
given by

$\displaystyle\mathcal{I}(0)=4\int\dot{\gamma}_{0}\dot{\gamma}_{0}^{\top}\,d\nu$

###### Proof of Lemma 23.

First, we claim that the regularity (A2) implies the Hellinger
differentiability. By the absolute continuity of the square root density, it
follows

$\displaystyle
dP_{t}^{1/2}-dP_{0}^{1/2}=\int_{0}^{t}\dot{\gamma}_{s,\mathrm{sign}(t)}\,ds=\int_{0}^{1}\dot{\gamma}_{t\widetilde{s},\mathrm{sign}(t)}t\,d\widetilde{s}$

and hence we have

$\displaystyle\int\left(dP_{t}^{1/2}-dP_{0}^{1/2}\right)^{2}\,d\nu$
$\displaystyle=\int\left(\int_{0}^{1}\dot{\gamma}_{t\widetilde{s},\mathrm{sign}(t)}t\,d\widetilde{s}\right)^{2}\,d\nu$
$\displaystyle=t^{\top}\left\\{\int\left(\int_{0}^{1}\int_{0}^{1}\dot{\gamma}_{t\widetilde{s}_{1},\mathrm{sign}(t)}\dot{\gamma}_{t\widetilde{s}_{2},\mathrm{sign}(t)}^{\top}\,d\widetilde{s}_{1}\,d\widetilde{s}_{2}\right)\,d\nu\right\\}\,t.$

By the assumption that $\lim_{s\longrightarrow
0}\dot{\gamma}_{s,u}=\dot{\gamma}_{0}$ for $\nu$-almost where and uniform
integrability $s\mapsto\dot{\gamma}_{s,u}$, we conclude

$\displaystyle\int\left(dP_{t}^{1/2}-dP_{0}^{1/2}\right)^{2}\,d\nu=t^{\top}\left(\int\dot{\gamma}_{0}\dot{\gamma}_{0}^{\top}\,d\nu\right)t+o(\|t\|_{2}^{2})$

as $\|t\|_{2}\longrightarrow 0$. This concludes the claim that the local path
$t\mapsto P_{t}$ is Hellinger differentiable at $t=0$ with
$\dot{\xi}_{0}=\dot{\gamma}_{0}$, $\nu$-almost everywhere. As a result, we
have

$\mathcal{I}(0)=4\int\dot{\xi}_{0}\dot{\xi}_{0}^{\top}\,d\nu=4\int\dot{\gamma}_{0}\dot{\gamma}_{0}^{\top}\,d\nu.$

∎