present address]MPA-Q, Los Alamos National Laboratory, Los Alamos, 87545 New
Mexico, USA

# Field-induced magnetic structures in the chiral polar antiferromagnet
Ni2InSbO6

Y. Ihara<EMAIL_ADDRESS>R. Hiyoshi  M. Shimohashi  R. Kumar
Department of Physics, Faculty of Science, Hokkaido University, Sapporo
060-0810, Japan  T. Sasaki  M. Hirata  [  Institute for Materials Research,
Tohoku University, Sendai, 980-8577, Japan  Y. Araki  Y. Tokunaga  Department
of Advanced Materials Science, University of Tokyo, Kashiwa 277-8561, Japan
T. Arima  Department of Advanced Materials Science, University of Tokyo,
Kashiwa 277-8561, Japan  RIKEN Center for Emergent Matter Science (CEMS), Wako
351-0198, Japan

###### Abstract

We have performed 115In-NMR spectroscopy for Ni2InSbO6 with corundum-related
crystal structure to reveal magnetic structures that develop in high magnetic
fields. At low fields Ni2InSbO6 shows a helical magnetic order with a long
wavelength because of its chiral and polar crystal structure. The field-
induced magnetic state was not investigated by microscopic experiment because
an extremely high magnetic field is required to modify the
antiferromagnetically coupled helical structure. From the analysis of our
115In-NMR spectra obtained at high magnetic fields, we confirm that the canted
antiferromagnetic structure appears in fields applied in the $[110]$ direction
and the propagation vector of magnetic helix is rotated toward the field
direction for fields in the $[001]$ direction. We discuss the effect of
magnetic field that modifies the magnetic structure of an antiferromagnetic
chiral magnet.

##  I Introduction

The correlation between the space-inversion breaking and magnetism, leading
linear magnetoelectric (ME) effect between the magnetic (electric) field and
electric (magnetic) polarization, has long been attracting much attention
because of their fundamental interest and technological applications. The
antiferromagnetic (AFM) order in corundum-type Cr2O3 simultaneously breaks the
space-inversion and time-reversal symmetries, which triggers the linear ME
effect. [1, 2] Dzyaloshinskii and Moriya succeeded in elucidating the origin
of a weak spontaneous magnetization in corundum type Fe2O3 [3, 4] by
introducing the antisymmetric exchange term, which sometimes induces a helical
magnetic order with a long modulation period. The magnetic helix can be
further modulated by the application of a magnetic field. The response to the
external magnetic field should be explored to understand and control the ME
effect potentially applicable to novel devices.

In chiral magnets, the application of magnetic fields gradually modifies the
magnetic helix at zero field into a magnetic chiral soliton lattice in CrNb3S6
[5] and the nanometric magnetic swirling object termed skyrmion in MnSi
assisted by the thermal fluctuations. [6] In contrast to these prototypal
chiral magnets with ferromagnetic (FM) symmetric exchange interactions, the
magnetic field effects on the ME materials with AFM interaction have not been
investigated in detail, because the critical fields required to modify the AFM
structure are much higher than those in the cases of FM chiral magnets. In the
AFM chiral magnets, thus, it is not obvious if the application of a magnetic
field can actually modify the magnetic helix. As the theoretical prediction of
the field effect on the AFM structure is difficult when nearly isotropic
magnetic moments reside in the chiral and polar crystal structure, it is
essential to experimentally reveal a change in AFM structure by magnetic
fields for unveiling the origin of the ME effect in AFM chiral magnets.

Figure 1:  Magnetic phase diagram of Ni2InSbO6 for (a) $\bm{H}\parallel[110]$
and (b) $\bm{H}\parallel[001]$ directions. [8] The magnetic structure at small
magnetic fields is a proper helical with long wavelength (LF phase). [7] (a)
In the fields along $[110]$, soliton excitations are introduced near the
metamagnetic fields of $B_{m}^{a}=14$ T. Above $B_{m}^{a}$, canted AFM state
(HFa phase) is stabilized. (b) In the fields along $[001]$, the propagation
vector of helical structure flops to $[001]$ direction above of
$B_{m}^{c}=19.2$ T (HFc phase). Hysteresis between LF and HFc phases was
observed in the magnetization and electric polarization measured in pulsed
magnetic fields.

As represented by Cr2O3 and Fe2O3, magnetic oxide compounds of corundum-
related structures provide a good playground for investigating the AFM state
emerging on the space-inversion-symmetry broken crystal structure.
Particularly, if the cation sites are periodically occupied by more than one
element, the cation ordering may cause a further lowering of symmetry.
Ni2InSbO6 is such a compound, which crystallizes in a hettotype structure. [7]
The cation sites of corundum form honeycomb layers perpendicular to the three-
fold axis. In Ni2InSbO6, Ni-In and Ni-Sb honeycomb layers are alternately
stacked. Ni and In or Sb are alternately arranged in each layer as in
hexagonal boron nitride. As a consequence, the centers of inversion and glide
mirrors in the corundum (space group $R\bar{3}c$) are completely broken, and
the space group becomes chiral and polar $R3$. As is expected, helical
magnetic order with an in-plane small wave vector (LF phase) has been
confirmed below $T_{N}=78$ K by neutron studies at zero magnetic field. [7] It
is proposed that the spin spiral plane should be perpendicular to the wave
vector, which implies weak magnetic anisotropy in the Ni2+ moments. [8] The
magnetic transition is also accompanied by a kink anomaly in the temperature
dependence of electric polarization, which suggests that the magnetic ordering
on the space-inversion-broken crystal structure has introduced the ME
response. [8]

Moreover, the measurements of magnetization and electric polarization in a
pulsed high magnetic field have revealed metamagnetic transitions with a
fairly large ME effect. [8] At the critical fields $B_{m}^{a}\simeq 14$ T for
$\bm{H}\parallel[110]$ and $B_{m}^{c}\simeq 19$ T for $\bm{H}\parallel[001]$,
electric polarization shows an anomaly, which suggests the modification of ME
coupling tensor by the field-induced magnetic structure transition. Various
magnetic structures can be stabilized by magnetic fields in Ni2InSbO6 with
small magnetic anisotropy and DM and AFM interactions. Previous study
suggested a canted AFM and a Q-flopped magnetic structures for the high-field
phases in $\bm{H}\parallel[110]$ (HFa phase) and $\bm{H}\parallel[001]$ (HFc
phase), respectively, as summarized in Fig. 1. [8] In order to solve the
magnetic structure in high magnetic fields microscopic experiments, worthy of
addressing the relative orientations of magnetic moments, are desirable. It
is, however, not easy to perform a neutron diffraction and a resonant x-ray
scattering measurements in a pulsed high magnetic field. [10, 11, 12] We must
utilize a complementary experimental technique that allows us to probe the
microscopic magnetic structure and can be performed in very high magnetic
fields. Therefore, in this study, we measured the nuclear magnetic resonance
(NMR) spectrum in fields up to 24 T to analyze the field-induced magnetic
structures in Ni2InSbO6. From our 115In-NMR spectrum measurement we observed
the internal fields at In sites, which are generated by the ordered Ni2+
moments, and thus their strength depends on the local spin configurations
around the In sites. We estimate the microscopic parameters that describe the
magnetic coupling strength between Ni2+ moments and In nuclear spins from the
analyses in the high-temperature paramagnetic state and LF phase. Using these
parameters, we determined the magnetic structures in the field-induced
magnetic state.

##  II Experimental

Single crystals of Ni2InSbO6 were grown with the chemical vapor transport
method [13] and the dimension of the sample used for NMR measurement is
$1\times 1\times 0.5$ mm3. We confirmed by the Laue photography that a domain
that has the $a$ axis rotated by 30 degrees about the $c$ axis is absent. In
the $R3$ space group without inversion and mirror symmetries, chiral and polar
domains exist inevitably. The domain formation does not affect the analyses of
NMR spectra, because we focus only on the local magnetic fields around the In
sites. A high-field NMR spectrum measurement was performed with a cryogen-free
superconducting magnet at the Institute for Materials Research (IMR), Tohoku
University. [14] Field-sweep NMR spectra were reconstructed from the transient
Fourie transform (FT) spectra obtained during the field sweep at a constant
sweep rate. [15] Broad NMR spectra in the ordered state were measured at
several frequencies to follow the field variation of spectral shape. Field-
dependence of NMR spectra was measured without changing the radio-frequency
(rf) coil for the rf tank circuit. As all the NMR spectra in one field
orientation were measured in situ, we can exclude the effect of change in the
field orientation for the field-induced modification of spectral structure.
For a high-temperature measurement at a fixed field of 13.0248 T, we measured
only a sharp central peak, which allows us to cover full spectrum by the
frequency window (typically $\sim 500$ kHz) for a single FT spectrum.

Figure 2:  Ni sites around the NMR-target In site on the three-fold axis. (a)
Eight Ni sites within a distance of $4$ Å. The cartesian coordinate system for
the hyperfine coupling tensor is indicated as $xyz$ axes. (b) NiO6 octahedra
around the In site. Ni2+ moments are coupled with In nuclear spin through the
Ni-O-In exchange paths. Ni2-c, Ni1, and Ni2-f sites have one, two and three
exchange paths, respectively.

##  III results and discussions

###  III.1 NMR parameters

NMR spectroscopy is a useful microscopic technique to investigate the magnetic
structure in magnetic fields. In the magnetically ordered state the ordered
moments create internal fields at the target nuclear sites, which shift the
peak positions of the NMR spectra. We compute the internal fields for several
model magnetic structures, and compare the result with those estimated from
the experimentally obtained NMR spectra. The nuclear spins are magnetically
coupled with the ordered moments through the dipole-dipole and hyperfine
interactions. The dipole fields can be directly calculated from the crystal
and magnetic structures. To estimate the hyperfine fields we need to determine
experimentally the hyperfine coupling strength $A_{\rm hf}$ between Ni2+
moments and In nuclear spins. In general the hyperfine coupling constant is
written in a tensor form as

$\displaystyle\widehat{A}_{\rm
hf}=\left(\begin{array}[]{ccc}A_{xx}&A_{xy}&A_{xz}\\\ A_{yx}&A_{yy}&A_{yz}\\\
A_{zx}&A_{zy}&A_{zz}\\\ \end{array}\right),$ (4)

where $z$ and $x$ directions are parallel to crystallographic $[001]$ and
$[110]$ directions and $y$ is perpendicular to both $x$ and $z$. [Fig. 2(a)]
The hyperfine field $\bm{B}_{\rm hf}$ at the In sites located on a three-fold
axis is written with the three Ni2+ moments ($\bm{M}_{1},\bm{M}_{2}$,
$\bm{M}_{3}$) reproduced by the three-fold operation and $\widehat{A}_{\rm
hf}$ for each Ni2+ moment rotated together by the three-fold rotation tensor
$\widehat{R}_{z}$ as

$\displaystyle\bm{B}_{\rm hf}$
$\displaystyle=\widehat{R}_{z}^{\mathsf{T}}\widehat{A}_{\rm
hf}\widehat{R}_{z}\bm{M}_{1}+\widehat{A}_{\rm
hf}\bm{M}_{2}+\widehat{R}_{z}\widehat{A}_{\rm
hf}\widehat{R}_{z}^{\mathsf{T}}\bm{M}_{3}$ $\displaystyle=3\widehat{A}_{\rm
tri}\bm{M}.$ (5)

In the paramagnetic state, where all Ni2+ moments are polarized uniformly to
the external field direction, that is, $\bm{M}_{1}=\bm{M}_{2}=\bm{M}_{3}$, the
total hyperfine coupling tensor becomes

$\displaystyle\widehat{A}_{\rm tri}$
$\displaystyle=\left(\begin{array}[]{ccc}\frac{1}{2}\left(A_{xx}+A_{yy}\right)&\frac{1}{2}\left(A_{xy}-A_{yx}\right)&0\\\
-\frac{1}{2}\left(A_{xy}-A_{yx}\right)&\frac{1}{2}\left(A_{xx}+A_{yy}\right)&0\\\
0&0&A_{zz}\end{array}\right).$ (9)

Here, we note that the off-diagonal components between $xy$ and $z$ directions
are canceled by the three-fold symmetry. On the other hand, an external field
in the $y$ ($x$) direction contributes to the hyperfine field in the $x$ ($y$)
direction through the off-diagonal components. These hyperfine coupling
constants are determined experimentally by measuring the Knight shift $K$ in
corresponding external field directions.

Figure 3:  High temperature 115In-NMR spectra in fields along [110] (top) and
[001] (bottom) directions. The measurement frequency is $150.51$ MHz for both
directions. The 115In spectrum is split into 9 peaks by the electric
quadrupolar interaction. The NQR frequency of 4.35 MHz was determined by the
peak separation in $\bm{H}\parallel[001]$.

Before measuring the temperature dependence of Knight shift, we need to
estimate the nuclear quadrupole resonance (NQR) frequency because the nuclear
spin of In is $I=9/2$, and thus interacts with electric field gradient (EFG).
Figure 3 shows the 115In-NMR spectrum at temperature $T=150$ K $>T_{N}$ in
fields applied parallel (bottom) and perpendicular (top) to the [001] axis.
Together with the sharp central peak from $m=-1/2\leftrightarrow 1/2$
transition broad satellite lines were observed. The resonant peaks from the
highest order satellites were not clearly observed because of the spectral
broadening originating from the local lattice distortion. By the three-fold
local symmetry of the In site, the EFG tensor has axial symmetry around three-
fold axis and the main principal axis of EFG is along the [001] axis. The NQR
frequency along the [001] axis is determined by the peak separation as 4.35
MHz.

Figure 4:  Temperature dependence of Knight shift plotted with bulk
susceptibility for fields along (a) [110] and (b) [001]. Susceptibility is
plotted against the right scale. (c) Knight shift is plotted against
susceptibility using temperature as a implicit parameter.

We focus on the sharp central peak to measure the temperature dependence of
Knight shift $K(T)$, which is shown in Figs. 4(a) and (b) together with the
bulk susceptibility $\chi(T)$ for corresponding field direction. In the
paramagnetic state, $K$ is proportional to the bulk susceptibility, thus
$K(T)=A_{\rm hf}\chi(T)$ for a certain field direction. Then by plotting
$K(T)$ as a function of $\chi(T)$ using the temperature as an implicit
parameter ($K-\chi$ plot), we can estimate $A_{\rm hf}$ from the slope of
their linear relation. Figure 4 (c) shows the $K-\chi$ plots for
$\bm{H}\parallel[110]$ and $\bm{H}\parallel[001]$. The finite intercept
$K_{0}$ originates from the quadrupolar shift. Although the nuclear
quadrupolar interaction does not modify $m=-1/2\leftrightarrow 1/2$ transition
by the first order perturbation, the second order contribution gives a
constant shift to the central peak. The linear relation between $K$ and $\chi$
was found at temperatures higher than 100 K for $\bm{H}\parallel[110]$ and 130
K for $\bm{H}\parallel[001]$. From the high-temperature linear relation the
total coupling constants are obtained as $A^{\rm total}_{[110]}=-399$
mT/$\mu_{B}$ and $A^{\rm total}_{[001]}=-431$ mT/$\mu_{B}$. The dipole
contributions are calculated from the crystal structure as $A^{\rm
dip}_{[110]}=-43$ mT/$\mu_{B}$ and $A^{\rm dip}_{[001]}=86$ mT/$\mu_{B}$. By
subtracting the dipole contributions from $A_{\rm hf}^{\rm total}$, hyperfine
coupling constants originating from the spin transfer interactions are
estimated as $A^{\rm hf}_{[110]}=-356$ mT/$\mu_{B}$ and $A^{\rm
hf}_{[001]}=-517$ mT/$\mu_{B}$. These angle dependent hyperfine coupling
constants are decomposed into the isotropic term $A_{\rm iso}=(2A^{\rm
hf}_{[110]}+A^{\rm hf}_{[001]})/3=-409$ mT/$\mu_{B}$ and the anisotropic term
$A_{\rm ani}=(A^{\rm hf}_{[110]}-A^{\rm hf}_{[001]})/3=54$ mT/$\mu_{B}$. The
anisotropic term originates from small population of In $5p$ electrons. Since
$A_{\rm iso}$ is one order of magnitude larger than $A_{\rm ani}$, we ignore
the anisotropic term for the analyses of NMR spectrum in the ordered state.

At lower temperatures near $T_{N}$ the Knight shift deviates from the bulk
susceptibility, as shown in Figs. 4(a) and (b), because of the development of
short-range correlations, which disturbs the uniform polarization of
paramagnetic spins and violates the linear relationship between uniform
susceptibility and the microscopic internal fields at the In sites. The NMR
intensity once disappears at $T_{N}$ because of the significant shortening of
nuclear spin-spin relaxation time by critical fluctuations. The NMR intensity
appears again when the critical fluctuations are suppressed at very low
temperatures. In the ordered state below $T_{N}$, the Ni2+ moments around the
In sites create the spontaneous internal fields dominantly through the
isotropic hyperfine interaction. The internal-field contribution from the
direct dipole interaction was calculated to be less than $10$ mT in the
magnetic structures studied here, which is much smaller than the internal
fields created by the hyperfine interaction and observed actually from NMR
measurement. Therefore, to understand the magnetic structure in the ordered
state, we need to estimate the hyperfine fields from each Ni2+ moment around
the In site.

In the crystal structure of Ni2InSbO6, Ni2+ ions occupy two crystallographic
sites at Ni1:$(0,0,-0.0139)$ and Ni2:$(0,0,0.4798)$. [7] As shown in Fig.
2(a), we find three Ni1 (Ni2) sites below (above) the target In nuclear spins.
The three Ni sites around the In sites are reproduced by the three-fold
operation. We label the nearest three Ni2 sites as Ni2-c to differentiate
another Ni2 site (Ni2-f) located on the three-fold axis. The Ni2-f site has a
significant contribution to the hyperfine interaction because it is connected
to the In site through three Ni-O-In paths. [Fig. 2(b)] Since the NiO6
octahedron shares a face with the neighboring InO6 octahedron, the coupling
with the Ni2-f site should be the largest. Ni1 and Ni2-c sites are connected
with the In site through two and one Ni-O-In paths, respectively. The angle of
the Ni-O-In path is close to 90 degrees for Ni2-f and Ni1 sites, while that
for Ni2-c site is 120 degrees as listed in Table I. This structural feature
suggests that the sign of hyperfine coupling constant for Ni2-c sites is
opposite to those for Ni2-f and Ni1 sites. As a result the total hyperfine
coupling constant $A_{\rm hf}=A_{2f}+3A_{1}+3A_{2c}$ is partly canceled in the
paramagnetic state.

To roughly estimate the hyperfine coupling strength for each site, we refer to
the exchange interaction calculated for Ni3TeO6 [16], in which In and Sb are
replaced with Ni (Ni3 sites) and Te, respectively, and thus the exchange
interactions between Ni3 sites and Ni1, Ni2-f, and Ni2-c sites are mediated by
the exchange paths similar to Ni-O-In bonds in Ni2InSbO6. (Table I) We
underline that the exchange interaction $J$ is negative only for Ni2-c sites,
confirming the negative sign for $A_{2c}$. The relative values between these
results weakly depend on the on-site Coulomb interaction. Assuming a similar
ratio for the hyperfine coupling strength for Ni2InSbO6, we can estimate
$(A_{1},A_{2f},A_{2c})=(+0.11,+0.56,-0.43)$ T/$\mu_{B}$.

Table 1: Exchange paths for Ni2InSbO6 (Ni$i$-O-In) and Ni3TeO6 (Ni$i$-O-Ni3)[16]. The last column shows the exchange interactions $J$ between Ni moments for each path. $J$ was calculated only for Ni3TeO6 [16].  Exchange path | Bond length (Å) | Angle (deg.) |  $J$ (meV)
---|---|---|---
Ni2InSbO6
(Ni2-c)-O-In | 4.003 | 121.2 | –
Ni1-O1-In | 4.293 | 91.8 | –
Ni1-O2-In | 4.287 | 92.0
(Ni2-f)-O-In | 4.252 | 86.3 | –
Ni3TeO6
(Ni2-c)-O-Ni3 | 4.167 | 124.24 | -1.48
Ni1-O1-Ni3 | 4.195 | 91.28 | 0.94
Ni1-O2-Ni3 | 4.186 | 91.55
(Ni2-f)-O-Ni3 | 4.273 | 81.07 | 4.55

###  III.2 NMR spectrum at low magnetic fields

With the microscopic parameters estimated above, we analyze the NMR spectrum
in the ordered state, which is broadened magnetically by the spontaneous
internal fields from the ordered Ni2+ moments. Figure 5(a) shows the 115In-NMR
spectrum for the LF phase measured in low magnetic fields applied in the [001]
direction. In the LF phase the number of peaks becomes more than nine, which
is the number of transitions for one nuclear spin $I=9/2$, and each peak has
an asymmetric shape like sawtooth. This spectral shape is explained by the
spatially modulated internal fields typically observed for an incommensurate
spin-density-wave state. The sinusoidal modulation of internal fields along
the external field direction constructs a double-horn structure as shown in
Fig. 5(b). By imposing the same spectral broadening for all the transition
lines and superposing these nine spectra, we successfully simulate the NMR
spectrum in the ordered state, as shown by the red solid line in Fig. 5(a).

Figure 5:  115In-NMR spectrum in the LF phase at 1.5 K. Magnetic fields were
applied in the [001] direction. Many peaks of saw-tooth shape are observed.
(a) The experimental data (blue shadow) is consistently explained by the
simulation (red solid line). The positions of two peaks split from the central
$-1/2\leftrightarrow 1/2$ transition as indicated by downward arrows. The
sharp peaks of metallic 65Cu and 63Cu come from the rf coil. (b) The simulated
spectral shape of only the central transition, $m=+1/2\leftrightarrow-1/2$.
The sharp peak in the paramagnetic state turns into a two-horn structure in
the ordered state with spatial modulation of ordered moments.

From the separation of two horns we can estimate the maximum value of the
internal fields at the In site as $B_{\rm int}=1.14$ T. This internal field is
calculated by the following formula,

$\displaystyle\bm{B}_{\rm
LF}=A_{2f}\bm{M}^{2f}+A_{1}\sum_{i}\bm{M}_{i}^{1}+A_{2c}\sum_{i}\bm{M}_{i}^{2c},$
(10)

where $\bm{M}$ is the ordered Ni2+ moments forming the proper helical
structure with propagation vector $\bm{k}=0.029b^{\star}$ and the index $i$
runs over three Ni sites around the three-fold axis. From the previous neutron
diffraction study the size of the ordered moment was measured as $1.89\mu_{B}$
and the Ni2+ moments at Ni1 and Ni2 sites were suggested to be
antiferromagnetically coupled. [7] In this spin configuration, the Ni2+ moment
at a position $(x,y,z)$ in the cartesian coordinate system defined in Fig.
2(a) is written as

$\displaystyle\bm{M}(x,y,z)=M_{0}\left(\begin{array}[]{c}\sin(2\pi
ky+\phi_{p})\\\ 0\\\ \cos(2\pi ky+\phi_{p})\\\ \end{array}\right).$ (14)

Here, $M_{0}=1.89\mu_{B},k=|\bm{k}|=0.029$, and $\phi_{p}$ is an arbitrary
phase factor for Ni1 ($p=1$) and Ni2 ($p=2$) sites. For the AFM spin
configuration between Ni1 and Ni2, which is realized by choosing
$\phi_{1}=\phi_{2}+\pi$, the maximum internal field in the $z$ direction is
computed as 2.0 T using the isotropic hyperfine coupling constants estimated
above. As this estimate is larger than the experimentally obtained internal
field of $B_{\rm int}=1.14$ T, we need to refine the Ni-site dependent
hyperfine coupling constants. We note that the relative change between
$A_{2f}$ and $A_{2c}$ is compensated in eq. (10) by the constraint that the
total hyperfine coupling constant must be consistent with the experimentally
determined $A_{\rm iso}=-409$ mT/$\mu_{B}$. Therefore, we tuned the ratio
between the hyperfine coupling constants to the Ni1 sites ($A_{1}$) and those
for Ni2 sites ($A_{2f}$ and $A_{2c}$). As the result, we obtained
$(A_{1},A_{2f},A_{2c})=(+0.034,+0.38,-0.30)$ T/$\mu_{B}$. The relative
decrease in $A_{1}$ with respect to $A_{2f}$ and $A_{2c}$ is consistent with
the increase in the ratio for the corresponding bond lengths. For example, the
ratio of Ni1-O1-Ni3 to (Ni2-c)-O-Ni3 bonds in Ni3TeO6 is 1.007, while the
corresponding ratio of Ni1-O1-In to (Ni2-c)-O-In bonds in Ni2InSbO6 is 1.072.

###  III.3 NMR spectra in high magnetic fields

We measured the field dependence of the NMR spectra at 4 K for both fields
parallel to [001] and [110] directions. Figure 6 shows the field-sweep NMR
spectra at several fixed frequencies. An apparent change in the spectral shape
is observed at $\mu_{0}H=14.2$ T for $\bm{H}\parallel[110]$ and
$\mu_{0}H=19.2$ T for $\bm{H}\parallel[001]$, both of which coincide with the
field-induced anomalies detected by bulk measurements, [8] suggesting that a
change in magnetic structure affects the NMR spectral structures. In addition
to the 115In-NMR signals, we observed the NMR spectra from the metallic
63Cu/65Cu of the rf coil and from 121Sb in the sample. The reference fields at
each measurement frequency are determined by the gyromagnetic ratio of each
nuclear spins $\gamma$ as $f_{n}=\gamma_{n}H$ ($n=^{63}$Cu, 65Cu, 121Sb) and
are represented by the dashed lines in Fig. 6. A broad 121Sb-NMR spectrum was
observed only in the high field phases for both field orientations. This is
because the optimal spin-echo pulse condition to obtain the largest 121Sb-NMR
signal is different between LF and high-field phases. We optimized the pulse
conditions to maximize the 115In-NMR signal intensity and focus only on the
115In-NMR spectra.

Figure 6:  Field dependence of 115In-NMR spectra measured in the ordered
states. The external fields were applied along (a) [110] and (b) [001]
directions. The baseline was offset according to the measurement frequency for
a better visibility. The reference fields for 63Cu, 65Cu, 121Sb, and 115In are
indicated by dashed lines. Change in the spectral shape was observed above the
metamagnetic critical fields $B_{m}^{a}$ and $B_{m}^{c}$

To extract the information about the internal fields at the In site, we
convert the horizontal axis of NMR spectrum at each frequency to $\Delta
B=B_{0}-B_{\rm obs}$, where $B_{0}$ is a reference field determined by the
measurement frequency $f_{0}$ as $B_{0}=f_{0}/\gamma$. For
$\bm{H}\parallel[110]$, NMR spectrum at 118.51 MHz is in the LF phase and
those above 131.51 MHz are in the HFa phase. The structureless broad spectra
in the LF phase change to the multi-peak structure in the HFa phase. The
spectral shape in the HFa phase is understood as the quadrupolar-split peaks
with small magnetic broadening. The peak separation of $\sim 0.23$ T in the
HFa phase is consistent with those observed at high temperature in Fig. 3. The
considerably small magnetic spectral width in the HFa phase suggests that the
internal field at the In sites becomes spatially uniform after the
metamagnetic transition. In contrast, the internal fields created in the LF
phase with proper helical structure have spatial distribution, which leads to
a spectrum broadening.

In the LF phase, as the proper helical spin configuration does not have a spin
component parallel to the field direction, [8] the internal field along the
external field direction is produced by the off-diagonal terms of the
hyperfine coupling tensor. We introduced a hyperfine coupling tensor
$\widehat{A}_{\rm tri}$ in eq. (9) for the paramagnetic state, where all spins
are aligned to the same direction. When spins take a proper helical structure
with long wavelength, $\widehat{A}_{\rm tri}$ is modified because one of three
spins is rotated by $9^{\circ}$ around the propagation direction. After a
rotation of $\theta$ around $y$ axis, the magnetic moment
$\bm{M}=(m_{x},m_{y},m_{z})$ becomes

$\displaystyle\bm{M}(\theta)$
$\displaystyle=\left(\begin{array}[]{c}m_{x}\cos\theta-m_{z}\sin\theta\\\
m_{y}\\\ m_{x}\sin\theta+m_{z}\cos\theta\\\ \end{array}\right)$ (18)
$\displaystyle\simeq\bm{M}(0)+\left(\begin{array}[]{c}-m_{z}\\\ 0\\\ m_{x}\\\
\end{array}\right)\sin\theta.$ (22)

Here, we approximated $\cos 9^{\circ}=0.99\simeq 1$. The additional second
term modifies $\widehat{A}_{\rm tri}$ by contributing to the off-diagonal
($xz$ and $zx$) components. Nevertheless, as these components are multiplied
by a small factor of $\sin 9^{\circ}=0.157$, we can treat the helical
structure as a minor modification to the paramagnetic spin configuration at a
microscopic length scale. Thus, the $z$ component of Ni2+ moment does not
produce any sizable internal fields in the $xy$ plane in the LF phase. When
the external field is applied in the $y$ direction, the internal field in the
$y$ direction is created mainly by the $x$ component of the Ni2+ moments
through the $xy$ component of $\widehat{A}_{\rm tri}$. The spatial modulation
of $m_{x}=M_{0}\sin(2\pi ky+\phi)$ in the helical structure leads to the
distribution of the internal fields, which then contributes to the large
magnetic broadening of the NMR spectrum below $B_{m}^{a}$. The internal-field
distribution from helical modulation is eliminated in the HFa phase with the
canted AFM spin configuration, because $m_{x}$ is uniform over a large length
scale when both Ni1 and Ni2 moments are fixed to a certain crystallographic
orientation. [Fig. 8(b)] The field-induced NMR spectrum narrowing observed in
the HFa phase evidences the canted AFM structure, which was proposed from the
previous bulk measurements. [8]

Figure 7:  Internal-field representation of the 115In-NMR spectra. The
horizontal axis $\Delta B$ is the shift from the reference field at each
measurement frequency. The external field directions are (a) [110] and (b)
[001]. Spectral width becomes narrow in HFa phase above $B_{m}^{a}$. In the
field parallel to [001] direction, the saw-tooth shape in the LF phase
disappears above $B_{m}^{c}$ and quadrupolar-split NMR peaks are observed in
the HFc phase. The peak position for the quadrupolar-split peaks are indicated
by the downward arrow at the top of each figure.

In the magnetic fields parallel to [001], the reduction of spectral width was
also observed in high-field (HFc) phase. The sawtooth structure in the LF
phase is smeared at high fields and transforms to symmetric peaks above $19.2$
T. The separation of these symmetric peaks coincides with the quadrupolar
splitting observed in the paramagnetic state (Fig. 3). The peak positions of
quadrupolar-split 9-peaks for $\bm{H}\parallel[001]$ are indicated by downward
arrows at the top of Fig. 7(b). As the NMR spectral shape is dominated by the
quadrupolar splitting, we conclude that the magnetic broadening is much
smaller than the quadrupolar interaction in the HFc phase as in the case of
$\bm{H}\parallel[110]$. The full spectral shape for HFc phase was not observed
because a part of the 115In-NMR spectrum at large internal fields ($\Delta
B>2$ T) is overlapped with NMR signal coming from 121Sb nuclear spins in the
sample. The other end ($\Delta B<0$ T at 222.51 MHz) was limited by the
maximum magnetic fields currently accessible at IMR.

Figure 8:  Magnetic structures suggested for (a) LF, (b) HFa and (c) HFc
phases. (a) Ni spins in the triangular layer forms proper helical structure
with a long wavelength. Ni1 and Ni2 spin magnetic moments are
antiferromagnetically aligned. (b) In the HFa phase with fields applied in the
[110] direction, antiferomagnetically coupled Ni1 and Ni2 spin moments are
canted toward the field direction. The spatial modulation of helical structure
is removed by the ferromagnetic configuration of Ni spins on the same
triangular layer. (c) In the HFc phase, propagation vector of the helical
structure flops to the [001] direction. As the spins are nearly perpendicular
to the external field direction, the internal fields along [001] direction is
small.

The reduction of magnetic spectral width is consistently explained by the
Q-flop transition at 19.2 T. At higher magnetic fields, the propagation vector
points to the field direction $\parallel[001]$ and the Ni2+ moments are
aligned in the $(001)$ plane to form a proper helical structure along $[001]$
direction. [Fig. 8(c)] Here, we assume that Ni2+ moments on the same plane are
ferromagnetically aligned. This type of helical magnetic structure is realized
in $Re$Be13 series [17]. In this spin configuration, the hyperfine coupling
tensor is written as eq. (9), as the Ni2+ moments around the three-fold axis
are aligned in parallel. Then, as the off-diagonal components between $z$ and
$xy$ directions are zero, the ordered Ni2+ moments aligned in the $(001)$
plane do not create any hyperfine fields along the [001] direction. As a
consequence, the magnetic broadening is suppressed in the HFc phase. We note
that in addition to the reduction of spectral broadening, the peak positions
shift with fields, as indicated by tilted dashed line above 19 T in Fig. 7(b).
This shift is interpreted as the partial spin polarization in the [001]
direction after the Q-flop transition, which results in the conical spin
structure.

From recent neutron study a similar Q-flop transition was observed in a sister
compound Ni2ScSbO6 by substituting a small amount of Co for Ni. [9] The
pristine Ni2ScSbO6 shows a helical magnetic ordering with long wavelength, as
in the case of Ni2InSbO6. The magnetic propagation vector changes its
orientation from in-plane to the $c$ direction in Ni2-xCoxScSbO6 with $x>0.5$.
As the Co2+ moments have a magnetic anisotropy stronger than Ni2+ moments, the
Q-flop transition can be attributed to the introduction of magnetic
anisotropy. Assuming that the external magnetic fields also introduce the
magnetic anisotropy, forcing magnetic moments to direct perpendicular to the
external field direction, the HFc phase in Ni2InSbO6 would have the magnetic
structure similar to those in Ni2-xCoxScSbO6. This similarity leads us to an
intuitive understanding for the effect of magnetic fields in an AFM chiral
magnet with competing interactions. Interestingly, the wavelength of helix in
Ni2-xCoxScSbO6 is commensurate to the crystal structure and becomes short at
small Co doping. In contrast to the Co doping, magnetic fields do not create
any structural defects, and thus the Q-flop transition occurs in relatively
uniform background. The wavelength and commensurability in HFc phase should be
revealed to address the difference between the effects of magnetic field and
Co doping. From the present NMR measurement, however, the wavelength in the
HFc phase cannot be determined because NMR experiment detects only the local
magnetic fields at the In sites but cannot measure the relative orientations
between the neighboring Ni2+ moments. The neutron diffraction measurement
should be performed above $B_{m}^{c}=19.2$ T in future.

##  IV Conclusion

We have investigated the magnetic structure of corundum-related
antiferromagnet Ni2InSbO6 by means of 115In-NMR spectrum measurement. We
observed the spectral broadening in the LF phase at low magnetic fields, which
is ascribed to the spatial modulation of internal field induced by the helical
magnetic structure with a long wavelength. The spectral width becomes narrower
in high magnetic fields above the critical fields $B_{m}^{a}$ and $B_{m}^{c}$.
Our microscopic analyses for the local magnetic fields at the In site lead us
to determine the field-induced magnetic structure as a canted AFM for
$\bm{H}\parallel[111]$ and Q-flopped helix for $\bm{H}\parallel[001]$. These
magnetic structures are consistent with those proposed from the previous bulk
measurements. [8] By comparing the similar Q-flop transition in the Co-
substituted Ni2-xCoxScSbO6, [9] we suggest that the high magnetic field
modifies the magnetic structure by introducing the magnetic anisotropy.
Present NMR results provide experimental evidence that high magnetic fields
actually modify the helical magnetic structure even in the AFM chiral magnet.
Microscopic identification of the field-induced magnetic structure establishes
the field control of functional ME effects in the AFM chiral magnet and
motivates technically challenging neutron diffraction measurements in high
magnetic fields.

###### Acknowledgements.

The high-field NMR measurements were performed at the High Field Laboratory
for Superconducting Materials, Institute for Materials Research, Tohoku
University (Project Nos. 20H0601, 202110-HMCPB-0423) This work was partially
supported by JSPS KAKENHI (Grants Nos. 19H01832, 21H01035).

## References

  * [1] D. N. Astrov, Sov. Phys. JETP 11, 708 (1960).
  * [2] V. J. Folen, G. T. Rado, and E. W. Stalder, Phys. Rev. Lett. 6, 607 (1961).
  * [3] I. Dzyaloshinsky, J. Phys. Chem. Solids 4, 241 (1958).
  * [4] T. Moriya, Phys. Rev. 120, 91 (1960).
  * [5] Y. Togawa, T. Koyama, K. Takayanagi, S. Mori, Y. Kousaka, J. Akimitsu, S. Nishihara, K. Inoue, A. S. Ovchinnikov, and J. Kishine, Phys. Rev. Lett. 108, 107202 (2012).
  * [6] S. Mühlbauer, B. Binz, F. Jonietz, C. Pfleiderer, A. Rosch, A. Neubauer, R. Georgii, P. Böni, Science 323, 915 (2009).
  * [7] S. A. Ivanov, R. Mathieu, P. Nordblad, R. Tellgren C. Ritter, E. Politova, G. Kaleva, A. Mosunov, S. Stefanovich, and M. Weil, Chem. Mater. 25, 935 (2013).
  * [8] Y. Araki, T. Sato, Y. Fujima, N. Abe, M. Tokunaga, S. Kimura, D. Morikawa, V. Ukleev, Y. Yamasaki, C. Tabata, H. Nakao, Y. Murakami, H. Sagayama, K. Ohishi, Y. Tokunaga, and T. Arima, Phys. Rev. B 102, 054409 (2020).
  * [9] K.-L. Ji, E. Solana-Madruga, A. M. Arevalo-Lopez, P. Manuel, C. Ritter, A. Senyshyn, and J. P. Attfield, Chem. Commun. 54 12523 (2018).
  * [10] K. Ohoyama, N. Katoh, H. Nojiri, Y. H. Matsuda, H. Hiraka, K. Ikeda, H. M. Shimizu, J. Magn. Magn. Mater. 310, e974 (2007).
  * [11] H. Nojiri, S. Yoshii, M. Yasui, K. Okada, M. Matsuda, J.-S. Jung, T. Kimura, L. Santodonato, G. E. Granroth, K. A. Ross, J. P. Carlo, and B. D. Gaulin, Phys. Rev. Lett. 106, 237202 (2011).
  * [12] F. Duc, X. Tonon, J. Billette, B. Rollet, W. Knafo, F. Bourdarot, J. Béard, F. Mantegazza, B. Longuet, J. E. Lorenzo, E. Leliévre-Berna, P. Frings, Rev. Sci. Instrum. 89, 053905 (2018).
  * [13] M. Weil, R. Mathieu, P. Nordblad, and S. Ivanov, Cryst. Res. Technol. 49, 142 (2014).
  * [14] S. Awaji, K. Watanabe, H. Oguro, S. Hanai, H. Miyazaki, M. Takahashi, S. Ioka, M. Sugimoto, H. Tsubouchi, S. Fujita, M. Daibo, Y. Iijima, and H. Kumakura, IEEE Trans. Appl. Supercond. 24, 4302005 (2014).
  * [15] W. G. Clark, M. E. Hanson, F. Lefloch, P. Ségransan, Rev. Sci. Instrum. 66, 2453 (1995).
  * [16] F. Wu, E. Kan, C. Tian, and M.-H. Whangbo, Inorg. Chem. 2010, 49 7545-7548.
  * [17] P. J. Becker, M. Bonnet, and F. Vigneron, Mol. Cryst. Liq. Cryst. 125, 405 (1985).


# Von Neumann entropy in a dispersive cavity

Ram Narayan Deb<EMAIL_ADDRESS><EMAIL_ADDRESS>Chandernagore
College, Chandernagore, Hooghly, PIN-712136, West Bengal, India

###### Abstract

We study the von Neumann entropy of the partial trace of a system of two two-
level atoms (qubits) in a dispersive cavity where the atoms are interacting
collectively with a single mode electromagnetic field in the cavity. We make a
contrast of this entanglement entropy with the spin squeezing property of the
system. We find a close relationship between this von Neumann entropy and the
spin squeezing of the system. We observe that this entanglement entropy
attains its maximum value when the spin squeezing in the system goes to its
maximum and is zero when the spin squeezing in the system vanishes.

Keywords: Von Neumann entropy, two-level atom, dispersive cavity

##  I I. INTRODUCTION

Von Neumann entropy, also called the entanglement entropy, forms an important
measure of quantum entanglement in bipartite states of quantum systems Nielsen
.

Optical cavity, in which the atoms interact with the electromagnetic field, is
an important experimental tool through which we can study the various aspects
of the atom-field interaction and its dynamics. Cavity QED and cavity related
systems are widely studied for producing correlated photons or two-level atoms
or both. Dispersive cavity plays a significant role in this field $[2-14]$.

Spin squeezing of a system of two-level atoms is an important experimentally
measurable quantity that depends on the amount of quantum correlations
established among the atoms Kitagawa , Wineland . So, there exists a close
relationship between the von Neumann entropy of the partial trace of a system
of two two-level atoms and spin squeezing in that system. The problem of
finding the entropy for the interaction of a single atom and a quantized field
in a dispersive cavity has been considered in Ref. Moya .

In this paper we consider a system of two identical two-level atoms each of
frequency $\omega_{0}$ interacting collectively with a single mode
electromagnetic field in a dispersive cavity whose characteristic frequency
nearest to $\omega_{0}$ is $\omega_{c}$ Agarwal . The cavity temperature is
related with the average number of thermal photons $\bar{n}$ present in the
cavity. The photon lifetime is represented by $\frac{1}{2k}$, where $k$ is the
cavity bandwidth. If $\hat{J}_{+}$, $\hat{J}_{-}$ and $\hat{J}_{z}$ are the
atomic operators and $\hat{a}$, $\hat{a}^{\dagger}$ are the bosonic operators
of the cavity field mode then the atom-field interaction is governed by the
Hamiltonian Agarwal

$\hat{H}_{af}=\hbar\omega_{0}\hat{J}_{z}+\hbar\omega_{c}\hat{a}^{\dagger}\hat{a}+\hbar
g(\hat{J}_{+}\hat{a}+\hat{a}^{\dagger}\hat{J}_{-}),$ (1)

where $g$ is the atom-field coupling constant.

We consider the cavity to be highly detuned, that is,
$\delta=\omega_{c}-\omega_{0}$ is very large satisfying
$|i\delta+k|>>g\sqrt{2}$. The solution to the master equation for a quantized
cavity mode has been described in Ref. Arevalo . Quantum state reconstruction
in the presence of dissipation has been described in Ref. Moya1 , that takes
into account cavity losses. Now, for the present case the derivation of the
effective Hamiltonian describing the dynamics of the atoms in the cavity,
using the master equation approach, has been done by Agarwal et al. in Ref.
Agarwal . If $\delta>>k$, then the contribution due to damping is very small
and the effective Hamiltonian takes the form Agarwal

$\displaystyle\hat{H}_{eff}=\hbar\Delta_{0}[\hat{J}_{+}\hat{J}_{-}+2\bar{n}\hat{J}_{z}]=\hbar\Delta_{0}[\hat{J}^{2}-\hat{J}z^{2}+2\bar{n}\hat{J}_{z}],$
(2)

where

$\Delta_{0}=\frac{g^{2}\delta}{k^{2}+\delta^{2}}.$ (3)

This Hamiltonian is quadratic in population inversion operator $\hat{J}_{z}$.
The temperature dependent term in the Hamiltonian produces a simple rotation
of the quantum state that has no physical significance. Therefore, we drop it
by setting $\bar{n}=0$. So, the effective Hamiltonian reduces to

$\hat{H}_{eff}=\hbar\Delta_{0}(\hat{J}^{2}-\hat{J}_{z}^{2}).$ (4)

We consider the initial state of the two two-level atoms, before they enter
the cavity, to be an atomic coherent state Radcliffe , Arecchi  and after they
enter the dispersive cavity the time evolution of the state is governed by the
above Hamiltonian in Eq. (4). The resulting state is an atomic
Schr$\ddot{o}$dinger cat state Agarwal . In this paper we investigate the
dynamics of the von Neumann entropy of partial trace of this state and make a
contrast with the spin squeezing dynamics of the system.

The organization of the paper is as follows. In Sec. II we study the von
Neumann entropy of the partial traces of the two two-level atoms in the
dispersive cavity. In Sec. III we make a contrast of this entanglement entropy
with the spin squeezing dynamics of the whole system.

##  II II. Von Neumann entropy of partial traces of two two-level atoms in a
dispersive cavity

We assume the initial state of the two two-level atoms, before they enter the
dispersive cavity, to be an atomic coherent state which is represented as

$|j,\chi\rangle=\frac{1}{(1+\chi^{2})^{j}}\sum_{n=0}^{2j}\sqrt{{}^{2j}C_{n}}\chi^{n}|j,j-n\rangle.$
(5)

The quantum number $j=N/2$, where $N$ is the number of atoms. Since, in this
case the number of atoms $N=2$, the quantum number $j=\frac{N}{2}=1$. So, the
above state reduces to

$|1,\chi\rangle=\frac{1}{(1+\chi^{2})}\big{[}|1,1\rangle+\sqrt{2}\chi|1,0\rangle+\chi^{2}|1,-1\rangle\big{]}.$
(6)

Here $\chi=\tan(\theta/2)e^{i\phi}$. $\chi=0$ represents all the atoms in
their upper states. In the following we assume $\phi=0$ as it has no effect on
the problem we study here. This state is achieved by, first, sending the atoms
in their upper states through an auxillary cavity. The quantity $\theta$ is
decided by the duration of atom-field interaction in this cavity. These atoms
then enter the dispersive cavity where their time evolution is governed by the
Hamiltonian in Eq. (4).

In the $\\{m_{1},m_{2}\\}$ representation, the time evolved state (with
$\hbar=1$) is

$\displaystyle|1,\chi,t\rangle$ $\displaystyle=$
$\displaystyle\frac{1}{(1+|\chi|^{2})}\bigg{[}e^{-i\Delta_{0}t}\bigg{|}\frac{1}{2},\frac{1}{2}\bigg{\rangle}+\chi
e^{-2i\Delta_{0}t}\bigg{|}\frac{1}{2},-\frac{1}{2}\bigg{\rangle}+\chi
e^{-2i\Delta_{0}t}\bigg{|}-\frac{1}{2},\frac{1}{2}\bigg{\rangle}$ (7)
$\displaystyle+$
$\displaystyle\chi^{2}e^{-i\Delta_{0}t}\bigg{|}-\frac{1}{2},-\frac{1}{2}\bigg{\rangle}\bigg{]}.$

Now, if $\hat{J}_{1_{x}}$, $\hat{J}_{1_{y}}$ and $\hat{J}_{1_{z}}$ are the
operators of atom 1 and $\hat{J}_{2_{x}}$, $\hat{J}_{2_{y}}$ and
$\hat{J}_{2_{z}}$ are the operators of atom 2, then, the mean spin vectors of
atoms 1 and 2 are

$\langle\hat{\overrightarrow{J}}_{1}\rangle=\langle\hat{J}_{1_{x}}\rangle\hat{i}+\langle\hat{J}_{1_{y}}\rangle\hat{j}+\langle\hat{J}_{1_{z}}\rangle\hat{k}$
(8)

and

$\langle\hat{\overrightarrow{J}}_{2}\rangle=\langle\hat{J}_{2_{x}}\rangle\hat{i}+\langle\hat{J}_{2_{y}}\rangle\hat{j}+\langle\hat{J}_{2_{z}}\rangle\hat{k},$
(9)

where $\hat{i}$, $\hat{j}$ and $\hat{k}$ are the unit vectors along the
positive $x$, $y$ and $z$ axes respectively.

Now, the density matrix for a pair of qubits $A$ and $B$ can be written in the
Bloch-Fano decomposition as $[22-25]$

$\displaystyle\rho_{AB}=\frac{1}{4}\bigg{(}{\bf 1}\otimes{\bf
1}+\overrightarrow{\sigma}^{A}.{\bf u}\otimes{\bf 1}+{\bf
1}\otimes\overrightarrow{\sigma}^{B}.{\bf
v}+\sum_{i,j=1}^{3}\beta_{ij}\sigma_{i}\sigma_{j}\bigg{)},$ (10)

where

$\displaystyle{\bf u}$ $\displaystyle=$
$\displaystyle\langle\sigma_{x}^{A}\rangle\hat{i}+\langle\sigma_{y}^{A}\rangle\hat{j}+\langle\sigma_{z}^{A}\rangle\hat{k},$
(11) $\displaystyle{\bf v}$ $\displaystyle=$
$\displaystyle\langle\sigma_{x}^{B}\rangle\hat{i}+\langle\sigma_{y}^{B}\rangle\hat{j}+\langle\sigma_{z}^{B}\rangle\hat{k}.$
(12)

From Eq. (10), we can obtain two reduced density matrices

$\displaystyle\rho_{A}$ $\displaystyle=$ $\displaystyle
Tr_{B}(\rho_{AB})=\frac{1}{2}({\bf 1}+\overrightarrow{\sigma}^{A}.{\bf u})$
(13) $\displaystyle\rho_{B}$ $\displaystyle=$ $\displaystyle
Tr_{A}(\rho_{AB})=\frac{1}{2}({\bf 1}+\overrightarrow{\sigma}^{B}.{\bf v}).$
(14)

For pure states $|{\bf u}|=|{\bf v}|$. The well known measure of concurrence,
for pure states, can be calculated as

$\displaystyle
C=\sqrt{2[1-Tr{(\rho^{A})}^{2}]}=\sqrt{2[1-Tr{(\rho^{B})}^{2}]}=\sqrt{1-|{\bf
u}|^{2}}=\sqrt{1-|{\bf v}|^{2}}.$ (15)

The entanglement of formation is given by

$\displaystyle
F(C)=H\bigg{(}\frac{1+\sqrt{1-C^{2}}}{2}\bigg{)}=H\bigg{(}\frac{1+\sqrt{|{\bf
u}|^{2}}}{2}\bigg{)},$ (16)

where

$H(x)=-xlog_{2}x-(1-x)log_{2}(1-x).$ (17)

Now, for pure states the entanglement of formation is same as the von Neumann
entropy of the reduced density matrix of either $A$ or $B$. So, using Eqs.
(16) and (17) the von Neumann entropy of the reduced density matrix takes the
form

$\displaystyle E=-\bigg{(}\frac{1}{2}+\frac{1}{2}|{\bf
u}|\bigg{)}log_{2}\bigg{(}\frac{1}{2}+\frac{1}{2}|{\bf
u}|\bigg{)}-\bigg{(}\frac{1}{2}-\frac{1}{2}|{\bf
u}|\bigg{)}log_{2}\bigg{(}\frac{1}{2}-\frac{1}{2}|{\bf u}|\bigg{)}.$ (18)

Now, the mean spin vector for atom 1 is related with
$\langle\overrightarrow{\sigma}_{1}\rangle$ (with $\hbar=1$) as

$\displaystyle\langle\hat{\overrightarrow{J_{1}}}\rangle=\langle\hat{J}_{1_{x}}\rangle\hat{i}+\langle\hat{J}_{1_{y}}\rangle\hat{j}+\langle\hat{J}_{1_{z}}\rangle\hat{k}=\frac{1}{2}\bigg{(}\langle{\sigma}_{1_{x}}\rangle\hat{i}+\langle{\sigma}_{1_{y}}\rangle\hat{j}+\langle{\sigma}_{1_{z}}\rangle\hat{k}\bigg{)}=\frac{1}{2}\langle\overrightarrow{\sigma}_{1}\rangle,$
(19)

So, the von Neumann entropy of the partial traces of the system can be
represented in terms of these mean spin vectors of either atom 1 or atom 2 and
is given as $[9-12]$

$\displaystyle
E=-\bigg{(}\frac{1}{2}+|\langle\hat{\overrightarrow{J}}_{1}\rangle|\bigg{)}log_{2}\bigg{(}\frac{1}{2}+|\langle\hat{\overrightarrow{J}}_{1}\rangle|\bigg{)}-\bigg{(}\frac{1}{2}-|\langle\hat{\overrightarrow{J}}_{1}\rangle|\bigg{)}log_{2}\bigg{(}\frac{1}{2}-|\langle\hat{\overrightarrow{J}}_{1}\rangle|\bigg{)}$
(20)

or,

$\displaystyle
E=-\bigg{(}\frac{1}{2}+|\langle\hat{\overrightarrow{J}}_{2}\rangle|\bigg{)}log_{2}\bigg{(}\frac{1}{2}+|\langle\hat{\overrightarrow{J}}_{2}\rangle|\bigg{)}-\bigg{(}\frac{1}{2}-|\langle\hat{\overrightarrow{J}}_{2}\rangle|\bigg{)}log_{2}\bigg{(}\frac{1}{2}-|\langle\hat{\overrightarrow{J}}_{2}\rangle|\bigg{)}.$
(21)

Figure 1: [Variation of von Neumann entropy of the partial traces of the
system of two two-level atoms with the dimensionless interaction time
$\Delta_{0}t$. We plot the von Neumann entropy and the dimensionless
interaction time $\Delta_{0}t$ along y-axis and x-axis respectively. The thin
line curve (in the bottom) is for $\theta=\pi/6$. The dotted curve (at the
top) is for $\theta=\pi/2$ and the thick solid curve (at the middle) is for
$\theta=2\pi/3$.]

Since, in the quantum state of Eq. (7), both the atoms are on equal footing,
the mean spin vectors of both the atoms for the quantum state in Eq. (7) are
equal and has the magnitude (with $\chi=\tan(\theta/2)$)

$\displaystyle|\langle\hat{\overrightarrow{J}}_{1}\rangle|=|\langle\hat{\overrightarrow{J}}_{2}\rangle|=\frac{1}{2}\sqrt{1-\sin^{2}(\Delta_{0}t)\sin^{4}\theta}.$
(22)

So, using Eqs. (20), (21) and (22), the von Neumann entropy of the partial
traces of this system is given as

$\displaystyle E$ $\displaystyle=$
$\displaystyle-\frac{1}{2}\Big{[}1+\sqrt{1-\sin^{2}(\Delta_{0}t)\sin^{4}\theta}\Big{]}log_{2}\bigg{[}\frac{1}{2}\Big{(}1+\sqrt{1-\sin^{2}(\Delta_{0}t)\sin^{4}\theta}\Big{)}\bigg{]}$
$\displaystyle-$
$\displaystyle\frac{1}{2}\Big{[}1-\sqrt{1-\sin^{2}(\Delta_{0}t)\sin^{4}\theta}\Big{]}log_{2}\bigg{[}\frac{1}{2}\Big{(}1-\sqrt{1-\sin^{2}(\Delta_{0}t)\sin^{4}\theta}\Big{)}\bigg{]}.$

We show in Fig. 1 the time evolution of this entanglement entropy for various
values $\theta$. We observe that the von Neumann entropy of the partial traces
increases with $\Delta_{0}t$, reaches a maximum and then decreases to zero.
This behaviour is repeated with the further increase in $\Delta_{0}t$. This
shows that the quantum entanglement shows oscillatroy behaviour with the
dimensionless interaction time $\Delta_{0}t$ inside the dispersive cavity. The
quantum entanglement is maximum when $\Delta_{0}t=(2n+1)\pi/2$, where n=0, 1,
2, 3…. This means that the entanglement entropy is maximum when

$\frac{g^{2}\delta}{k^{2}+\delta^{2}}t=(2n+1)\frac{\pi}{2}.$ (24)

Fig. 1 also shows that the entanglement entropy takes the maximum value 1,
when $\theta=\pi/2$.

The angle $\theta$ is related to the probabilities of finding the two atoms in
their respective upper and lower states as discussed below.

From the expression of the quantum state given in Eq. (7) we observe that the
probability of finding both the atoms in their respective upper states is

$P_{1}=\frac{1}{{(1+|\chi|^{2})}^{2}}=\cos^{4}\frac{\theta}{2}.$ (25)

Similarly, the probabilities of finding the first atom in its upper state and
the second atom in its lower state and vice-versa is

$P_{2}=P_{3}=\frac{|\chi|^{2}}{{(1+|\chi|^{2})}^{2}}=\frac{1}{4}\sin^{2}\theta.$
(26)

The probability of finding both the atoms in their lower states is

$P_{4}=\frac{|\chi|^{4}}{{(1+|\chi|^{2})}^{2}}=\sin^{4}\frac{\theta}{2}.$ (27)

These probabilities for $\theta=\pi/6$, $\pi/2$, and $2\pi/3$ are given in the
following table.

Table 1: Values of $\theta$, $P_{1}$, $P_{2}$, $P_{3}$ and $P_{4}$ $\theta$ |  $P_{1}$ |  $P_{2}$ |  $P_{3}$ |  $P_{4}$
---|---|---|---|---
$\pi/6$ | 0.87 | 0.0625 | 0.0625 | 0.00448
$\pi/2$ | 0.25 | 0.25 | 0.25 | 0.25
$2\pi/3$ | 0.0625 | 0.1875 | 0.1875 | 0.5625
Figure 2: [Variation of von Neumann entropy of the partial traces of the
system of two two-level atoms with the angle $\theta$. We plot the von Neumann
entropy and the angle $\theta$ along y-axis and x-axis respectively. The
dimensionless interaction time $\Delta_{0}t=\pi/2$.]

We observe from Fig. 1 and Table I, that the von Neumann entropy attains the
top most value when $\theta=\pi/2$ and $P_{1}=P_{2}=P_{3}=P_{4}=0.25$.

In Fig. 2 we show the variation of the von Neumann entropy of the partial
traces with the angle $\theta$, keeping fixed the dimensionless interaction
time $\Delta_{0}t$ at $\pi/2$. We see that von Neumann entropy increases with
increase in $\theta$, goes to a maximum, and then decreases to zero. This
behaviour is repeated with further increase in $\theta$. So, quantum
entanglement shows oscillatory behaviour with the angle $\theta$.

We, now, proceed to make a contrast of this entanglement entropy with the spin
squeezing dynamics of the system.

##  III III. Von Neumann entropy of the partial traces versus spin squeezing

The spin squeezing of the system of two-level atoms is defined as follows.

If $\hat{J}_{x}$, $\hat{J}_{y}$ and $\hat{J}_{z}$ are the collective spin
operators of a system of two-level atoms, then the collective mean spin vector
is

$\langle\hat{\bf
J}\rangle=\langle\hat{J}_{x}\rangle\hat{i}+\langle\hat{J}_{y}\rangle\hat{j}+\langle\hat{J}_{z}\rangle\hat{k}.$
(28)

Now, if $\hat{J}_{1_{\perp}}$ and $\hat{J}_{2_{\perp}}$ are the two mutually
orthogonal components of $\hat{\bf J}$ in a plane perpendicular to
$\langle{\hat{\bf J}}\rangle$, satisfying

$\Delta J_{1_{\perp}}\Delta J_{2_{\perp}}\geq\frac{|\langle\hat{\bf
J}\rangle|}{2},$ (29)

we say that there is spin squeezing in the system if $\sqrt{2}\Delta
J_{1_{\perp}}/\sqrt{|\langle\hat{\bf J}\rangle|}<1$ at the expense of $\Delta
J_{2_{\perp}}$ or, $\sqrt{2}\Delta J_{2_{\perp}}/\sqrt{|\langle\hat{\bf
J}\rangle|}<1$ at the expense of $\Delta J_{1_{\perp}}$.

Now, in general $\langle\hat{\bf J}\rangle$ points in an arbitrary direction.
So, conventionally we rotate our coordinate system $\\{x,y,z\\}$ to
$\\{x^{\prime},y^{\prime},z^{\prime}\\}$ such that the mean spin vector
$\langle\hat{\bf J}\rangle$ points along the $z^{\prime}$ axis. In that case
we say that there is spin squeezing in the system if

$S_{x}=\frac{\sqrt{2}\Delta J_{x^{\prime}}}{\sqrt{|\langle\hat{\bf
J}\rangle|}}<1$ (30)

or,

$S_{y}=\frac{\sqrt{2}\Delta J_{y^{\prime}}}{\sqrt{|\langle\hat{\bf
J}\rangle|}}<1.$ (31)

Figure 3: [Variation of spin squeezing parameters $S_{x}$, $S_{y}$ and
entanglement entropy $E$ with the dimensionless interaction time
$\Delta_{0}t$. We plot $S_{x}$, $S_{y}$ and $E$ along $y$ axis and
$\Delta_{0}t$ along $x$ axis. The thick solid curve is E, the thin solid curve
is $S_{x}$ and the dotted curve is $S_{y}$.]

Spin squeezing properties of this system has been studied in Ref. Ram . In
this paper we study the dynamics of the von Neumann entropy of this system and
make a contrast with the spin squeezing.

Now, in Fig. 3, we plot the spin squeezing parameters $S_{x}$, $S_{y}$ and the
entanglement entropy $E$ with respect to the dimensionless interaction time
$\Delta_{0}t$. We observe that when the entanglement entropy goes to maximum,
$S_{x}$ becomes very much less than 1, implying maximum spin squeezing in the
$x^{\prime}$ quadrature. The quantity $S_{y}$ is never squeezed and it is
always greater than 1. $S_{y}$ goes to maximum when the entanglement entropy
$E$ goes to maximum.

We conclude that when the correlations among the two atoms is maximum, showing
maximum squeezing, the entanglement entropy is also maximum. Thus, there is a
close relationship between spin squeezing and the quantum entanglement.

## References

  * (1) M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information Cambridge University Press, Cambridge, U.K., 2000.
  * (2) J. M. Raimond, M. Brune, and S. Haroche, Rev. Mod. Phys. 73, 565 (2001);
  * (3) E. Solano, G. S. Agarwal, and H. Walther, Phys. Rev. Lett. 90, 027903 (2003).
  * (4) G. S. Agarwal, R. R. Puri, and R. P. Singh, Phys. Rev. A 56, 2249 (1997).
  * (5) M. Brune, S. Haroche, J. M. Raimond, L. Davidovich, and N. Zagury, Phys. Rev. A 45, 5193 (1992).
  * (6) M. Boissonneault, J. M. Gambetta, and A. Blais, Phys. Rev. A 77, 060305 (2008).
  * (7) H. Walther, B. T. H. Varcoe, B. G. Englert, and T. Becker, Rep. Prog. Phys. 69, 1325 (2006).
  * (8) R. Miller, T. E. Northup, K. M. Birnbaum, A. Boca, A. D. Boozer, and H. J. Kimble, J. Phys. B: At. Mol. Opt. Phys. 38, S551 (2005).
  * (9) P. Ginzburg, Rev. Phys. 1, 120 (2006).
  * (10) P. J. Leek, M. Baur, J. M. Fink, R. Bianchetti, L. Steffen, S. Filipp, and A. Wallraff, Phys. Rev. Lett. 104, 100504 (2010).
  * (11) I. B. Mekhov, C. Maschler, and H. Ritsch, Nat. Phys. 3, 319 (2007).
  * (12) D. Meiser, and P. Meystre, Phys. Rev. A 74, 065801 (2006).
  * (13) D. I. Schuster, A. Fragner, M. I. Dykman, S. A. Lyon, and R. J. Schoelkopf, Phys. Rev. Lett. 105, 040503 (2010).
  * (14) Y. Guo, R. M. Kroeze, V. D. Vaidya, J. Keeling, and B. L. Lev, Phys. Rev. Lett. 122, 193601 (2019).
  * (15) M. Kitagawa, M. Ueda, Phys. Rev. A 47, 5138 (1993)
  * (16) D. J. Wineland, J. J. Bollinger, W. M. Itano, Phys. Rev. A 50, 67 (1994).
  * (17) H. Moya-Cessa, Int. J. Quant. Inf. 5, 149 (2007).
  * (18) L. M. Arevalo-Aguilar and H. Moya-Cessa, Quant. Sem. Opt. 10, 671 (1998).
  * (19) H. Moya-Cessa, S. M. Dutra, J. A. Roversi, A. Vidiella-Barranco, J. Mod. Opt. 46, 555 (1999).
  * (20) J. M. Radcliffe, J. Phys. A 4, 313 (1971).
  * (21) F. T. Arecchi, E. Courtens, R. Gilmore, and H. Thomas, Phys. Rev. A 6, 2211 (1972).
  * (22) W. K. Wootters, Phys. Rev. Lett. 80, 2245 (1998).
  * (23) H. J. Kummer, Int. J. Theor. Phys. 40, 1071 (2001).
  * (24) J. L. Chen, L. Fu, A. A. Ungar, and X. G. Zhao, Phys. Rev. A 65, 044303 (2002).
  * (25) S. P. Walborn, P. H. Souto Ribeiro, L. Davidovich, F. Mintert, and A. Buchleitner, Phys. Rev. A 75, 032338 (2007).
  * (26) R. N. Deb, M. S. Abdalla, S. S. Hassan, and N. Nayak, Phys. Rev. A 73, 053817 (2006).


# Irrelevant Pixels are Everywhere: Find and Exclude Them for More Efficient
Computer Vision
††thanks: This project is supported in part by NSF OAC-2104709 and NSF
OAC-2107020. Any opinions, findings, conclusions, or recommendations expressed
in this material by the authors do not necessarily reflect the views of the
sponsors.

Caleb Tung1, Abhinav Goel1, Xiao Hu1, Nick Eliopoulos1, Emmanuel S. Amobi2,
George K. Thiruvathukal2, Vipin Chaudhary3, Yung-Hsiang Lu1 1Purdue
University, 2Loyola University Chicago, 3Case Western Reserve University
{tung3, goel39, hu440, neliopou<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Computer vision is often performed using Convolutional Neural Networks (CNNs).
CNNs are compute-intensive and challenging to deploy on power-contrained
systems such as mobile and Internet-of-Things (IoT) devices. CNNs are compute-
intensive because they indiscriminately compute many features on all pixels of
the input image. We observe that, given a computer vision task, images often
contain pixels that are irrelevant to the task. For example, if the task is
looking for cars, pixels in the sky are not very useful. Therefore, we propose
that a CNN be modified to only operate on relevant pixels to save computation
and energy. We propose a method to study three popular computer vision
datasets, finding that 48% of pixels are irrelevant. We also propose the
focused convolution to modify a CNN’s convolutional layers to reject the
pixels that are marked irrelevant. On an embedded device, we observe no loss
in accuracy, while inference latency, energy consumption, and multiply-add
count are all reduced by about 45%.

###### Index Terms:

computer vision, low-power devices, embedded systems, datasets

##  I Introduction

Convolutional Neural Networks (CNNs) are known for their high accuracy at many
computer vision tasks. However, this accuracy comes at a cost: CNNs are
compute-intensive. ResNet, a popular computer vision CNN for performing the
relatively simple task of image classification, needs to compute nearly 30
million parameters across all the pixels in the input image [1]. This high
compute requirement is typically satisfied by running the CNN on powerful
Graphics Processing Units (GPUs) or other hardware accelerators. However, low-
power systems (i.e., Internet-of-Things (IoT), mobile, and embedded devices)
often impose power and memory constraints that make it challenging to deploy
CNNs on them [1].

To lessen the computation of a CNN on low-power systems, many methods opt to
reduce the sheer magnitude of CNN parameters. These include pruning [2] and
quantization [3] to cut away redundant parameters or reduce precision, and
further include knowledge distillation [4] and neural architecture search [5]
to train small CNNs with fewer parameters.

These methods all improve efficiency by changing the computer vision model
itself; we instead propose changing the input to the model. CNNs operate
indiscriminately on every single pixel in the input; therefore, if we reduce
the input, we reduce the computation.

In this paper, we propose that input images have many pixels that can be
deleted, and that by doing so, a CNN would save energy. We confirm this idea
by creating methods to (1) identify that three different popular computer
vision datasets all contain many such pixels and (2) demonstrate the energy
and inference speed improvements possible by using our focused convolution to
delete those pixels.

An irrelevant pixel (Figure 1) (formally defined in Section II) is one that is
not useful for the computer vision task (e.g., a building’s pixels are not
useful when looking for cars). We use depth maps to convert ground truth
labels into irrelevant pixel maps, showing that in the Microsoft Common
Objects in Context (COCO, 164,000+ images) [6], Multi-Object Tracking
Challenge (MOT Challenge, 5,500 images) [7], and PASCAL VOC [8] (17,900+
images) datasets, roughly 48% of pixels are irrelevant.

Given the irrelevant pixels, we further experiment to find that explicitly
excluding those pixels during inference significantly reduces a CNN’s compute
expenses, saving energy and speeding inference. Our proposed focused
convolution technique modifies a CNN such that the model itself can exclude
pixels marked as irrelevant while still using the same parameters and General
Matrix-Multiplication convolutional technique. Similarly inspired work
includes Uber’s SBNet and its variations; SBNet does convolution in ResNet-
sized blocks and tries to fit the blocks into the relevant pixels [9, 10].
Those other methods still require the model to be changed and re-trained, ours
does not. Replacing normal convolution with the focused convolution on the
three datasets reduces multiply-add operations, energy consumption, and
inference latency by about 45% in two popular CNNs.

(a)

(b)

Figure 1: Example of irrelevant pixels in a PASCAL VOC dataset image. (a) The
shuttle buses are highlighted in red. (b) Our method shows that a significant
number of pixels (black) are irrelevant, providing little utility, even though
CNNs waste computation on them.

This paper’s contributions: (1) use depth maps to find how many pixels are
irrelevant in three popular computer vision datasets, and (2) demonstrate that
CNNs can save on inference latency and energy consumption by excluding
irrelevant pixels from computation and only performing operations on relevant
ones, via our focused convolution.

##  II Finding Irrelevant Pixels in Datasets

We propose a method to study datasets for irrelevant pixels. We find that
irrelevant pixels are commonly found in the COCO, PASCAL VOC, and MOTChallenge
datasets.

###  II-A Counting Irrelevant Pixels with Depth Maps

Our study defines a dataset’s relevant pixels as: all pixels that comprise the
dataset ground truth objects and their associated depth levels. In Figure 1a,
the ground truth pixels are represented by the red area around the shuttle
buses. The final pixelwise map of relevant pixels, shown in Figure 1b, is
generated using depth level thresholding (explained below) and includes
additional pixels.

Ground truth pixels alone do not represent all relevant pixels. A typical CNN
uses not only the pixels comprising the object (Figure 2a), but also the
pixels from the surrounding area of an object (Figure 2b) to extract features
that contextualize and understand the object [11]. To properly include these
contextual pixels, our method uses depth maps. Shown in Figure 3a, an image’s
depth map represents each pixel in the image with one value, referred to as
the pixel’s depth level, denoting how far away from the camera that pixel is.

Pixels that have similar depth levels to a given object on the ground are
known to provide useful, contextual pixels that surround the object [12]. For
example, a car will be at a similar depth level as its context: the road on
which it drives. Therefore, by thresholding the depth level appropriately, we
can capture all the relevant pixels: both the contextual pixels and their
associated objects.

(a)

(b)

Figure 2: Ground truth pixels alone (a) are not always a sufficient amount of
relevant pixels for a CNN to make good predictions (red box is inaccurate,
misses second bus); contextual pixels at similar depth levels are also needed
(b) for CNN accuracy (two good detections).

We generate depth maps with “MiDaS,” a depth estimation model by Ranftl, et
al. [13] MiDaS is chosen from among other monocular depth estimation
techniques because it is uniquely trained using a mix of diverse datasets,
dramatically improving its performance on Microsoft COCO, our largest target
dataset.

Our method thresholds the depth map to create a binary, pixelwise mask of
relevant and irrelevant pixels, such that the relevant pixel region
encapsulates the ground truth (see Figure 3). Using the observation that
camera focal lengths tend to cause pictures to be taken at mid-range depth
levels [11], we threshold an image’s MiDaS depth map according to the mid-
range of its distribution, filtering out pixels at short-range and long-range
depth levels. The filtered pixels are considered irrelevant and the remaining
pixels are considered relevant. We verify this binary, pixelwise mask against
the ground truth labels: if the mask’s relevant-marked region contain the
ground truth, then we assume all ground truth and associated contextual pixels
are captured, and we record the number of irrelevant VS relevant pixels. If
not, the threshold is expanded and the process is repeated.

(a)

(b)

Figure 3: This is how to count the number of irrelevant pixels in datasets:
First, use MiDaS to generate the image’s depth map (a). Next, threshold the
depth map, producing a small Relevant Pixels region (b). Verify against ground
truth to ensure all important data is captured. Figure 4:  Average percentage
of pixels that are relevant. Less than 1% of images contain 10-30% or 90-100%
useful pixels and are not shown on the graph. As shown, most images contain
50%-60% relevant (40-50% irrelevant pixels). This means that in most images,
40-50% of pixels do not contribute to useful CNN computations and, therefore,
can be excluded from processing.

###  II-B Exploring Datasets for Irrelevant Pixels

In our study, we use our depth map method to count irrelevant pixels in each
image in the three datasets, totaling nearly 200,000 images. Results are shown
in Figure 4. Irrelevant pixels are quite common in the three datasets. On
average, 42% of all pixels are irrelevant in PASCAL VOC, 49% are irrelevant in
MOT Challenge, and 45% are irrelevant in COCO. This suggests that CNNs are
wasting significant amounts of compute resources on embedded devices by
computing convolutions on all those irrelevant pixels.

##  III Excluding Irrelevant Pixels in Datasets

To demonstrate that being able to exclude irrelevant pixels can significantly
save computer vision energy consumption and inference latency, we propose the
focused convolution technique. In typical CNN computer vision models, the
2D-convolutional layer is responsible for up to 80% of the computations
performed by the neural network [1]. Standard convolution layers are not
designed to exclude irrelevant pixels. Instead, they operate on all the input
pixels, wasting computation on irrelevant pixels. Because CNNs have multiple
convolutional layers in succession, this waste is repeated across multiple
layers, compounding the negative impact on the model’s energy use and speed.
The focused convolution reduces this waste by excluding any pixels marked
irrelevant by some assumed oracle - in this case, our relevant VS irrelevant
pixelwise masks generated in Section II-B.

###  III-A Using Focused Convolutions to Exclude Irrelevant Pixels

Our focused convolution improves the popular General Matrix Multiplication
(GEMM) technique [14]. The GEMM technique reduces a sequential, sliding-window
2D-convolution to a parallelized, matrix-multiplication operation; matrix
multiplications are heavily optimized on modern computers [14]. GEMM (Figure
5a) uses a procedure called im2col to convert patches of the 3D input tensor
(e.g., an RGB image) into the columns of a 2D matrix, and then multiplies the
matrix by the convolutional weights to retrieve the results of convolution.

The focused convolution enables the im2col procedure to accommodate a
pixelwise mask (such as those generated in Section II-B) indicating whether a
pixel is relevant or not. If a patch of pixels is labeled irrelevant, then the
modified im2col will not convert the patch into a column of the matrix. Thus,
those irrelevant pixels get excluded entirely by the matrix multiplication, as
shown in Figure5b.

Because the focused convolution is simply designed to accommodate the
pixelwise masks as inputs, it does not need to change the model itself. It can
be used in any pretrained CNN, requiring no additional training. It can
replace existing convolution layers without changing the weights and biases.
If no pixelwise mask is supplied, the focused convolution behaves identically
to a standard convolution.

(a)

(b)

Figure 5: (a) Traditional GEMM convolution uses the im2col procedure to
convert a $1\times 4\times 6\times 1$ input tensor (light gray), assuming a
stride-1, $3\times 3$ weight kernel (dark gray), to a $9\times 8$ matrix for
matrix multiplication. Each $9\times 1$ column of the matrix (e.g., red, blue
dashed line) represents one $3\times 3$ patch from the input tensor. (b) The
proposed focused convolution modifies im2col exclude patches deemed irrelevant
by an oracle-generated pixelwise mask. Thus, the final $9\times 6$ matrix will
have fewer columns than that of traditional GEMM, resulting in less
computation on embedded devices.

For a given image and convolutional layer, we can count the number of
operations needed to complete a GEMM convolution’s matrix multiplication.
Assume the input has dimensions $B_{I}\times C_{I}\times H_{I}\times W_{I}$
(batch, channels, height, width), and the convolutional weights has dimensions
$S_{W}\times S_{W}$ (side lengths) with stride $s$ and no padding. A normal
im2col would convert each channel of the input to
$\frac{H_{I}-S_{W}}{s}\frac{W_{I}-S_{W}}{s}$ columns of $S_{W}S_{W}$ pixels.
That amounts to multiplying the weights with a $S_{W}S_{W}\times
C_{I}\frac{(H_{I}-S_{W})}{s}\frac{W_{I}-S_{W}}{s}$ matrix, $B_{I}$ times. That
is a total of
$B_{I}\frac{C_{I}(H_{I}-S_{W})}{s}\frac{W_{I}-S_{W}}{s}S_{W}S_{W}$
multiply/add operations.

If we convert the GEMM convolution to a focused convolution, we save matrix
multiplication operations by excluding columns of irrelevant pixels. As we
discovered in Section II-B, an average of 48% of pixels are irrelevant in
popular datasets. If the region of relevant pixels is a rectangle that takes
up $p$% of the pixels, then the focused convolution would produce only
$p\%\frac{H_{I}-S_{W}}{s}\frac{W_{I}-S_{W}}{s}$ columns, resulting in a
$100-p$% reduction in operations. Therefore, the focused convolution’s energy
and latency improvements is a function of the linear relationship between
number of irrelevant pixels and image size.

###  III-B Evaluating the Compute Savings of Focused Convolutions

We implement and test the focused convolution using PyTorch on a Raspberry Pi
3 (average power 5 W). We compare the focused convolution with a normal
PyTorch GEMM convolution. Excluding pixels is beneficial on more powerful
hardware, too - we compare the focused convolution with an Intel MKL-optimized
convolution [15] on a Intel Core i7 CPU (average power 28 W). For input
images, we use the MOT Challenge, COCO, and PASCAL VOC. We exclude the pixels
deemed irrelevant by our method from Section II-A.

Our tests comprise of two popular computer vision CNNs designed for use on
low-power devices: EfficientDet [16] and SSD-Lite [17]. We use PyTorch’s
pretrained weights, and we replace each CNN’s convolutional layer with a
focused convolutional layer. The pixelwise, binary Relevant-VS-Irrelevant
masks we generated in Section II-B are propagated through the CNN by scaling
the mask’s size for each layer.

Table I summarizes the observed energy consumption/inference latency
improvements from the focused convolutions. As shown, we see that equipping a
CNN with focused convolution allows it to dramatically reduce its computation
expense by excluding irrelevant pixels on images from popular datasets. On
Intel CPU, the MKL optimizations allow a fullsize, “wasteful” convolution to
operate comparably to the focused convolution, but such optimizations are
unavailable on low-power devices, so the focused convolution is preferable.

Finally, we verify the focused convolution-equipped CNN’s detection accuracy
remains identical to the original CNN.

###  III-C Assumptions, Challenges, and Opportunities

This paper assumes that including all pixels at the same depth level as ground
truth accurately captures all related pixels. This may not be true when a
camera’s focus is extended (e.g. wide, scenic shots). Still, our technique is
demonstrably sufficient for three major datasets. Additionally, depth mapping
is computationally intensive, so future irrelevant pixel generation will need
to either be offloaded or rapidly generated.

| MOT2015 | COCO | PASCAL VOC
---|---|---|---
ED | SL | ED | SL | ED | SL
Number of Mult-Add Operations (M/inference)
Normal | 384.5 | 483.6 | 384.5 | 483.6 | 384.5 | 483.6
Focused | 196.1 | 246.8 | 211.4 | 266.0 | 223.0 | 280.4
Inference Latency (s/inference)
RPi (5W) | Normal | 2.10 | 2.26 | 2.00 | 2.33 | 2.06 | 2.29
Focused | 1.11 | 1.30 | 1.33 | 1.51 | 1.47 | 1.56
Intel (28W) | Normal | 0.25 | 0.28 | 0.25 | 0.29 | 0.25 | 0.28
MKL | 0.18 | 0.19 | 0.18 | 0.20 | 0.18 | 0.20
Focused | 0.17 | 0.18 | 0.18 | 0.20 | 0.19 | 0.20
Energy Consumption (J/inference)
RPi (5W) | Normal | 10.22 | 11.80 | 10.15 | 11.81 | 10.20 | 10.90
Focused | 5.60 | 6.11 | 6.71 | 7.50 | 7.44 | 7.80
Intel (28W) | Normal | 6.61 | 7.39 | 6.45 | 7.42 | 6.69 | 7.81
MKL | 5.18 | 5.09 | 5.09 | 5.61 | 5.23 | 5.60
Focused | 4.76 | 5.04 | 5.10 | 5.60 | 5.29 | 5.62
Table I: By excluding irrelevant pixels from popular datasets (MOTChallenge,
COCO, PASCAL VOC), CNNs equipped with our Focused convolutions outperform
those without (Normal). On an Intel CPU, MKL-optimized convolutions are
comparable in performance to focused convolutions.

RPi: Raspberry Pi 3, Intel: Intel Core i7 CPU, ED: EfficientDet, SL: SSD-Lite,
MKL: using Intel MKL optimized convolution.

##  IV Conclusion

We observe that in computer vision, CNNs are compute-heavy because they waste
time looking at irrelevant pixels. We propose a thresholded depth mapping
technique to study modern computer vision datasets for irrelevant pixels. We
find that an average of 42%, 49%, and 45% of pixels per image are irrelevant,
for three popular datasets (PASCAL VOC, MOT Challenge, and COCO,
respectively). We further propose significantly reducing a convolutional
layer’s multiply-add operations, energy consumption, and inference latency
with our focused convolution: this modifies the layer’s standard GEMM
operations to ignore irrelevant pixels. Ignoring the irrelevant pixels we
computed on the three datasets, we find that multiply-add operations are
reduced by an average of 48.3%, energy consumption by 42.7%, and inference
latency by 47.4% for two popular object detection CNNs on an embedded device.

## References

  * [1] George K. Thiruvathukal  “Low-Power Computer Vision: Improve the Efficiency of Artificial Intelligence”  CRC Press, 2022
  * [2] H. Wang  “Structured Pruning for Efficient Convolutional Neural Networks via Incremental Regularization”  In _IEEE JSTSP_ , 2020
  * [3] Matthieu Courbariaux  “Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1”  In _arXiv:1602.02830_ , 2016
  * [4] B. Peng  “Correlation Congruence for Knowledge Distillation”  In _2019 IEEE/CVF ICCV_
  * [5] M. Tan  “MnasNet: Platform-Aware Neural Architecture Search for Mobile”  In _2019 IEEE/CVF CVPR_
  * [6] Tsung-Yi Lin  “Microsoft COCO: Common Objects in Context”  In _ECCV 2014_
  * [7] Laura Leal-Taixé  “MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking”  In _arXiv:1504.01942_ , 2015
  * [8] Mark Everingham  “The Pascal Visual Object Classes (VOC) Challenge”  In _Springer IJCV_ , 2010
  * [9] M. Ren  “SBNet: Sparse Blocks Network for Fast Inference”  In _2018 IEEE/CVF CVPR_
  * [10] T. Verelst  “Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference”  In _2020 IEEE/CVF CVPR_
  * [11] Kent Gauen  “Comparison of Visual Datasets for Machine Learning”  In _2017 IEEE IRI_
  * [12] Shuran Song  “Semantic Scene Completion From a Single Depth Image”  In _2017 IEEE/CVF CVPR_
  * [13] Rene Ranftl  “Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer”  In _IEEE TPAMI_ , 2020
  * [14] Andrew Anderson  “Low-memory GEMM-based convolution for deep neural networks”  In _arXiv:1709.03395_ , 2017
  * [15] Endong Wang  “Intel Math Kernel Library”  In _High-Performance Computing on the Intel® Xeon Phi™: How to Fully Exploit MIC Architectures_ Springer International Publishing, 2014
  * [16] Mingxing Tan, Ruoming Pang and Quoc V. Le  “EfficientDet: Scalable and Efficient Object Detection”  In _2020 IEEE/CVF CVPR_
  * [17] Mark Sandler  “MobileNetV2: Inverted Residuals and Linear Bottlenecks”  In _2018 IEEE/CVF CVPR_ , 2018