# Anisotropy-assisted magnon condensation in ferromagnetic thin films

Therese Frostad  Center for Quantum Sprintronics, Department of Physics,
Norwegian University of Science and Technology, NO-7491 Trondheim, Norway
Philipp Pirro  Alexander A. Serga  Burkard Hillebrands  Dapartment of Physics
and Research Center OPTIMAS, Rhainland-Pfälzische Technische Universität
Kaiserslautern-Landau, 67663 Kaiserslautern, Germany  Arne Brataas  Alireza
Qaiumzadeh  Center for Quantum Sprintronics, Department of Physics, Norwegian
University of Science and Technology, NO-7491 Trondheim, Norway

###### Abstract

We theoretically demonstrate that adding an easy-axis magnetic anisotropy
facilitates magnon condensation in thin yttrium iron garnet (YIG) films.
Dipolar interactions in a quasi-equilibrium state stabilize room-temperature
magnon condensation in YIG. Even though the out-of-plane easy-axis anisotropy
generally competes with the dipolar interactions, we show that adding such
magnetic anisotropy may assist the generation of the magnon condensation
electrically, via the spin transfer torque mechanism. We use analytical
calculations and micromagnetic simulations to illustrate this effect. Our
results may explain the recent experiment on Bi-doped YIG and open a new
pathway toward application of current-driven magnon condensation in quantum
spintronics.

Introduction—. Magnon condensation with nonzero momentum at room temperature
[1] is a fascinating phenomenon first observed in 2006. The condensed magnons
were observed at the two degenerate magnon band minima of yttrium iron garnet
(YIG), and easy-plane ferrimagnetic insulator with very low magnetic
dissipation [2, 3], as the spontaneous formation of a quasi-equilibrium and
coherent magnetization dynamics in the momentum space [4]. To generate
condensate magnons, magnon must be pumped into the system by an incoherent
stimulus such as parametric pumping [5, 6, 1, 7, 8, 9, 10, 11, 12, 13, 14]
and/or spin-transfer torque [15, 16, 17, 18, 19]. The system may thermalize
above a critical magnon density to form a quasi-equilibrium magnon
condensation state at the bottom of magnon bands. The study of magnon
condensation is not only interesting from an academic point of view, but it is
also of great importance in various areas of quantum technology and applied
spintronics [20, 21, 11, 22].

At high magnon densities, the relevant regime for the magnon condensation
state and nonlinear magnon-magnon interactions becomes important. A
(meta)stable and steady quasi-equilibrium magnon condensation requires an
effective repulsive interaction between magnon quasiparticles. It was shown
that in a system mainly influenced by exchange interaction, magnons are
attractive, but dipolar interactions in YIG may change the sign of nonlinear
magnon interactions and thus are crucial for the creation of a (meta)stable
condensate magnon state [23, 24, 8, 25, 9, 10, 26, 27, 28, 29].

Recently, it was shown that the thermalization time of magnon condensation is
reduced in confined nanoscopic systems [30]. It was also demonstrated that the
lateral confinement in YIG enhances the dipolar interaction along the
propagation direction and causes a deeper band depth, i.e., the difference
between ferromagnetic resonance (FMR) and magnon band minima. Increasing the
magnon condensation lifetime was attributed to this enhancement of the band
depth [30].

In another recent achievement in magnon condensation experiments, Divinsky et
al. [31] found evidence of condensation of magnons by spin-transfer torque
mechanism. They introduced a small perpendicular magnetocrystalline anisotropy
(PMA) through bismuth doping in the thin film of YIG, while the magnetic
ground state still resides within the plane. This discovery opens a new route
toward electronic control of magnon condensation.

However, the interplay between the dipolar interactions, which was previously
shown to be essential for the stability and thermalization of magnon
condensation, and the counteracting out-of-plane easy-axis magnetic
anisotropy, is so far uncharted. This article studies the nonlinear magnon
interactions by analyzing the mechanism behind the anisotropy-assisted
formation of the magnon condensate. We present simulations within the Landau-
Lifshitz-Gilbert framework [32, 33, 34] that support analytical calculations.

Model—. We consider a thin ferromagnetic film in the $y-z$ plane to model YIG.
The magnetic moments are directed along the $z$ direction by an external
magnetic field of strength $H_{0}$. The magnetic potential energy of the film
contains contributions from the isotropic exchange interaction
$\mathcal{H}_{\text{ex}}$, Zeeman interaction $\mathcal{H}_{\text{Z}}$,
dipolar interaction $\mathcal{H}_{\text{dip}}$, and additionally a PMA energy
$\mathcal{H}_{\text{an}}$ in the $x$ direction, normal to the film plane. YIG
has a weak in-plane easy-axis that can be neglected compared to the other
energy scales in the system. The total spin Hamiltonian of the system reads,

$\mathcal{H}=\mathcal{H}_{\text{ex}}+\mathcal{H}_{\text{Z}}+\mathcal{H}_{\text{dip}}+\mathcal{H}_{\text{an}}.$
(1)

The PMA energy is given by,

$\mathcal{H}_{\text{an}}=-K_{\text{an}}\sum_{j}(\bm{S}_{j}\cdot\hat{x})^{2},$
(2)

where $K_{\text{an}}>0$ is the easy-axis energy, $\hbar\bm{S}_{j}$ is the
vector of spin operator at site $j$, with $\hbar$ is the reduced Planck
constant. Details of the Hamiltonian can be found in the Supplemental Material
(SM) [35].

The Holstein-Primakoff spin-boson transformation [36] allows us to express the
spin Hamiltonian in terms of the magnon creation and annihilation operators.
The amplitude of the effective spin per unit cell in YIG at room temperature
is large $\hbar S\approx 14.3\hbar$, [27, 37, 38], and thus we can expand the
spin Hamiltonian in the inverse powers of the spin $S$. Up to the lowest order
in nonlinear terms, the magnon Hamiltonian $\mathcal{H}$ of a YIG thin film
can be expressed as the sum of two components: $\mathcal{H}_{2}$ and
$\mathcal{H}_{4}$. The former represents a noninteracting magnon gas
comprising quadratic magnon operators. The latter, on the other hand,
constitutes nonlinear magnon interactions characterized by quartic magnon
operators; see the SM for details [35]. Note that three-magnon interactions
are forbidden in our geometry by the conservation laws [39]

Magnon dispersion of YIG with a finite PMA—. The magnon dispersion in YIG is
well known and has been studied extensively in both experimental and
theoretical works [40, 41, 2]. Magnons travelling in the direction of the
external magnetic field have the lowest energy. These so-called backward
volume magnetostatic (BVM) magnons have a dispersion with a double degenerate
minimum at finite wavevectors $q_{z}=\pm Q$. When pumping magnons into the
thin film, the magnons may thermalize and eventually form a condensate state
in these two degenerate minima with opposite wavevectors.

The noninteracting magnon Hamiltonian and the dispersion of BVM magnons, along
the $z$ direction, in the presence of a finite PMA reads,

$\displaystyle\mathcal{H}_{2}$
$\displaystyle=\sum{q_{z}}\hbar\omega_{q_{z}}\hat{c}_{q_{z}}^{\dagger}\hat{c}_{q_{z}},$
(3a) $\displaystyle\hbar\omega_{q_{z}}$
$\displaystyle=\sqrt{A_{q_{z}}^{2}-B_{q_{z}}^{2}},$ (3b)

where $\hat{c}_{q_{z}}^{\dagger}(\hat{c}_{q_{z}})$ are the magnon creation
(annihilation) operators, and

$\displaystyle A_{q_{z}}$
$\displaystyle=D_{\text{ex}}q_{z}^{2}+\gamma(H_{0}+2\pi
M_{S}f_{q})-K_{\text{an}}S,$ (4a) $\displaystyle B_{q_{z}}$
$\displaystyle=2\pi M_{S}f_{q}-K_{\text{an}}S.$ (4b)

Here, $D_{\rm{ex}}$ is the exchange stiffness, $M_{S}=\gamma\hbar S/a^{3}$ is
the saturation magnetization, with
$\gamma=$1.2\text{\times}{10}^{-5}\text{\,}\mathrm{eV}\text{\,}{\mathrm{Oe}}^{-1}$$
is the gyromagnetic ratio, and $a=12.376$Å is the lattice constant of YIG. The
form factor $f_{q}=(1-e^{-|q_{z}|L_{x}})/(|q_{z}|L_{x})$ stems from dipolar
interactions in a thin magnetic film with thickness $L_{x}$ [42, 43].

Figure 1:  The dispersion of noninteracting BVM magnons in a YIG thin film for
various PMA strengths. The inset shows the depth of the band minima as a
function of the PMA strength. We set $L_{x}=$50\text{\,}\mathrm{nm}$$ and
$H_{0}=$1\text{\,}\mathrm{kOe}$$

Fig. 1 shows the effect of PMA on the magnon dispersion of YIG. PMA decreases
both the ferromagnetic resonance (FMR), and the magnon band gap at the
$\Gamma$ point $\omega_{{q_{z}}=0}$, in addition to a greater decrease in the
magnon band gap at the band minima $\omega_{{q_{z}}=\pm Q}$. Therefore the
band depth $\Delta\omega=\omega_{{q_{z}}=0}-\omega_{{q_{z}}=\pm Q}$ is
increased. The position of the band minima at $q_{z}=\pm Q$ is also shifted to
larger momenta. In addition, the curvature of the minima increases as a
function of the anisotropy strength. Above a critical PMA,
$K^{c_{2}}_{\text{an}}$, the magnetic ground state is destabilized and the in-
plane magnetic state becomes out-of-plane. We are interested in the regime in
which the magnetic ground state remains in-plane, and thus the effective
saturation magnetization is positive
$M_{\rm{eff}}=M_{S}-2K_{\text{an}}/(\mu_{0}M_{S})>0$

The effect of PMA on magnon dispersion resembles the effect of confinement in
the magnon spectra of YIG. In Ref. 30, it was shown that transverse
confinement in a YIG thin film leads to an increase of the FMR frequency, the
band depth, as well as shifting the band minima to higher momenta while the
magnon band gap at the band minima is also increased. It was shown that this
change of the spectrum in confined systems increases the magnon condensate
lifetime. Therefore, we expect, in a similar way, PMA increases the magnon
condensate lifetime and assists the generation of magnon condensation.

Nonlinear magnon interactions in the presence of PMA—. Magnons are considered
quasiparticles that exhibit weak interactions in the low-density regime, but
their intensity of nonlinear interactions increases as their density
increases. Repulsive interactions are essential for thermalizing injected
nonequilibrium magnons and creating a metastable magnon condensation at a
steady and quasi-equilibrium state. Since the discovery of magnon
condensation, there has been a long debate over the origin of magnon
thermalization [26, 8, 9, 27, 28, 44].

The nonlinear interaction of condensate magnons at the two degenerate minima,
$q_{z}=\pm Q$, consists of intra- and inter-band contributions,
$\mathcal{H}_{4}=\mathcal{H}_{4}^{\text{intra}}+\mathcal{H}_{4}^{\text{inter}}$,
where

$\displaystyle\mathcal{H}_{4}^{\text{intra}}=A(\hat{c}_{Q}^{\dagger}\hat{c}_{Q}^{\dagger}\hat{c}_{Q}\hat{c}_{Q}+\hat{c}_{-Q}^{\dagger}\hat{c}_{-Q}^{\dagger}\hat{c}_{-Q}\hat{c}_{-Q}),$
(5a)
$\displaystyle\mathcal{H}_{4}^{\text{inter}}=2B(\hat{c}_{Q}^{\dagger}\hat{c}_{-Q}^{\dagger}\hat{c}_{Q}\hat{c}_{-Q})+C(\hat{c}_{Q}^{\dagger}\hat{c}_{-Q}\hat{c}_{Q}\hat{c}_{-Q}$
$\displaystyle\hat{c}_{-Q}^{\dagger}\hat{c}_{-Q}\hat{c}_{-Q}\hat{c}_{Q}+\text{h.c.})+D(\hat{c}_{Q}^{\dagger}\hat{c}_{Q}^{\dagger}\hat{c}_{-Q}^{\dagger}\hat{c}_{-Q}^{\dagger}+\text{h.c.}).$
(5b)

The interaction amplitudes are given by,

$\displaystyle A=$ $\displaystyle-\frac{\gamma\pi
M_{S}}{SN}\big{[}(\alpha_{1}+\alpha_{3})f_{Q}-2\alpha_{2}(1-f_{2Q})\big{]}$
$\displaystyle-\frac{D_{\rm{ex}}Q^{2}}{2SN}(\alpha_{1}-4\alpha_{2})+\frac{K_{\rm{an}}}{2N}(\alpha_{1}+\alpha_{3}),$
(6a) $\displaystyle B=$ $\displaystyle\frac{\gamma 2\pi
M_{S}}{SN}\big{[}(\alpha_{1}-\alpha_{2})(1-f_{2Q})-(\alpha_{1}-\alpha_{3})f_{Q})\big{]}$
$\displaystyle+\frac{D_{\rm{ex}}Q^{2}}{2SN}(\alpha_{1}-2\alpha_{2})+\frac{K_{\rm{an}}}{N}(\alpha_{1}+\alpha_{3}),$
(6b) $\displaystyle C=$ $\displaystyle\frac{\gamma\pi
M_{S}}{2SN}\big{[}(3\alpha_{1}+3\alpha_{2}+4\alpha_{3})f_{Q}-\frac{8}{3}\alpha_{3}(1-f_{2Q})\big{]}$
$\displaystyle+\frac{D_{\rm{ex}}Q^{2}}{3SN}\alpha_{3}+\frac{K_{\rm{an}}}{4N}(3\alpha_{1}+3\alpha_{2}+4\alpha_{3}),$
(6c) $\displaystyle D=$ $\displaystyle\frac{\gamma\pi
M_{S}}{2SN}\big{[}(3\alpha_{1}+3\alpha_{2}+4\alpha_{3})f_{Q}-2\alpha_{2}(1-f_{2Q})\big{]}$
$\displaystyle+\frac{D_{\rm{ex}}Q^{2}}{2SN}\alpha_{2}+\frac{K_{\rm{an}}}{2N}(3\alpha_{2}+\alpha_{3}).$
(6d)

Here, $N$ is the total number of spin sites. The dimensionless parameters
$\alpha_{1}$, $\alpha_{2}$, and $\alpha_{3}$ are related to the Bogoliubov
transformation coefficients, listed in the SM [35].

An off-diagonal long-rage order characterizes the condensation state. The
condensate state is a macroscopic occupation of the ground state and can be
represented by a classical complex field. Therefore, to analyze the stability
of the magnon condensate state, we perform the Madelung’s transform
$\hat{c}_{\pm Q}\rightarrow\sqrt{N_{\pm Q}}e^{i\phi_{\pm Q}}$, in which the
macroscopic condensate magnon state is described with a coherent phase
$\phi_{\pm Q}$ and a population number $N_{\pm Q}$ [27, 28]. The total number
of condensed magnons is $N_{c}=N_{+Q}+N_{-Q}$, while the distribution
difference is $\delta=N_{+Q}-N_{-Q}$. We also define the total phase as
$\Phi=\phi_{+Q}+\phi_{-Q}$.
Finally, the macroscopic four-magnon interaction energy of condensed magnons
is expressed as ,

$\displaystyle\mathcal{V}_{4}(\delta,\Phi)=$
$\displaystyle\frac{N_{c}^{2}}{2}\big{[}A+B+2C\cos{\Phi}\sqrt{1-\frac{\delta^{2}}{N_{c}^{2}}}$
$\displaystyle+D\cos{2\Phi}-\big{(}B-A+D\cos{2\Phi}\big{)}\frac{\delta^{2}}{N_{c}^{2}}\big{]}.$
(7)

Without PMA, this expression is reduced to the one derived in [45].

Figure 2:  The total nonlinear magnon interaction energy, Eq. 7, as a function
of the PMA strength. $N$ and $N_{c}$ are the total number of spins and
condensate magnons, respectively. $K^{c_{1}}_{\rm{an}}$ represents the
critical value of the PMA at which the sign of nonlinear interactions is
changed. On the other hand, $K^{c_{2}}_{\rm{an}}$ corresponds to the critical
value of PMA at which the in-plane magnetic ground state becomes unstable. We
set $L_{x}=$50\text{\,}\mathrm{nm}$$ and $H_{0}=$1\text{\,}\mathrm{kOe}$$.
$K^{sim}_{\rm{an}}=$0.5\text{\,}\mu\mathrm{eV}$$ denotes the PMA used in our
micromagnetic simulations.

Now, we can look at the total interaction energy and interaction amplitudes in
more detail. Figure 2 shows the effective interaction potential as a function
of the PMA. In a critical PMA strength, $K^{c_{1}}_{\text{an}}$, the sign of
the interaction changes from repulsive to attractive. This critical anisotropy
is well below the critical magnetic anisotropy strength
$K^{c_{2}}_{\text{an}}$ that destabilizes the inplane magnetic ground state.

The necessary condition to reach a steady-state quasi-equilibrium magnon
condensation is the presence of repulsive interactions between magnons; thus,
in the following, we consider a PMA strength below the critical anisotropy
$K_{\text{an}}<K^{c_{1}}_{\text{an}}$. I this regime, the intraband
interaction is attractive, and thus interband contributions are important.

The interacting potential energy, Eq. (7), has five extrema at,

$\displaystyle\delta_{1}=0,\Phi=0;$ (8a) $\displaystyle\delta_{2}=0,\Phi=\pi;$
(8b) $\displaystyle\delta_{3}=0,\Phi=\cos^{-1}(-\frac{C}{D});$ (8c)
$\displaystyle\delta_{4}=N_{c}\big{[}1-(\frac{C}{B-A+D})^{2}\big{]}^{\frac{1}{2}},\Phi=0;$
(8d) $\displaystyle\delta_{5}=\delta_{4},\Phi=\pi.$ (8e)

Whether these extrema represent minima of the interacting potential energy
relies on the system thickness $L_{x}$ and the strength of the applied
magnetic field $H_{0}$.

Parameter | Symbol | Value
---|---|---
Saturation magnetization | $4\pi M_{S}$ | $1.75\text{\,}\mathrm{kOe}$
Effective spin | $S$ | 14.3
Exchange stiffness | $D_{\rm ex}$ | $0.64\text{\times}{10}^{-20}\text{\,}\mathrm{eV}\text{\,}{\mathrm{m}}^{2}$
Gilbert damping parameter | $\alpha$ | $1\text{\times}{10}^{-3}$
Table 1: The material parameters used in the micromagnetic simulations.

Phase diagram for magnon condensate—. Now, we explore the stability of the
magnon condensate as a function of the thickness of the film $L_{x}$ and the
strength of the external magnetic field $H_{0}$, using the typical YIG
parameters, see Table 1.

(a)

(b)

Figure 3:  The phase diagram pf the condensate magnon in the absence (a) and
presence (b) of PMA. We plot the magnon interaction energy
$\mathcal{V}_{4}/N_{c}^{2}$, Eq. 7, as a function of the film thickness
$L_{x}$ and external magnetic field strength $H_{0}$. The dashed black lines
indicate the boundaries between the different condensate phases, Eq. 8. We set
$K_{an}=$0.5\text{\,}\mathrm{\SIUnitSymbolMicro eV}$$ in (b).

First, we present the phase diagram for magnon condensation in YIG, in the
absence of PMA, in Fig. 3a. The thinnest films are expected to have a
symmetric distribution of magnon condensation between the two minima. This
phase diagram is in agreement with previous studies [27, 45].

Next, we add a PMA, with strength $K_{\rm an}=$0.5\text{\,}\mu\mathrm{eV}$$,
and plot the phase diagram of the magnon condensate in Fig. 3b for different
thicknesses. For the selected material parameters, PMA tends to push the
magnon condensate towards a more asymmetric population distribution between
the two magnon band minima. Since both minima are degenerate thus there is an
oscillation of magnon condensate between these two minima. The asymmetry of
condensate magnon populations agrees with our previous analysis of interaction
amplitudes. Within out parameters for thickness and PMA strength, the
intraband interaction $A$ is attractive, while the interband interactions are
still repulsive.

This phase diagram shows that in the presence of a PMA, condensate magnon can
still be a metastable state. In addition, as we discussed earlier, a PMA
increases the band depth and reduces the curvature of noninteracting magnon
dispersion, see Fig. 1, which leads to an enhancement of the condensate magnon
lifetime. Thus, we expect that introducing a small PMA into a thin film of YIG
facilitates the magnon condensation process.

(a) 0.1in0.1in

(a)

(b) 0.1in0.1in

(b)

(c) 0.1in0.1in

(c)

(d) 0.1in0.1in

(d)

Figure 4:  Magnon distribution from micromagnetic simulations of a
$50\text{\,}\mathrm{nm}$ thick YIG film at an external magnetic field strength
$H_{0}=$1\text{\,}\mathrm{kOe}$$. In the absence of the PMA, $K_{\text{an}}=0$
: (a) and (b) show magnon distributions of initial nonequilibrium excited
magnons and final quasi-equilibrium magnon condensate steady state,
respectively. In the presence of the PMA,
$K_{\text{an}}=$0.5\text{\,}\mathrm{\SIUnitSymbolMicro eV}$$ : (c) and (d)
show magnon distributions of initial nonequilibrium excited magnons and final
quasi-equilibrium magnon condensate steady state, respectively. The dotted
line indicates the analytical dispersion relation of noninteracting magnons,
Eq. 3b. Because of nonlinear magnon interactions, there is a spectrum shift in
the simulated magnon dispersion compared to the noninteracting result.
Although the duration of magnon pumping by spin-transfer torque is the same in
the absence or presence of the PMA, the critical torque amplitude is lower in
the presence of PMA.

Micromagnetic simulation of magnon condensate—. To validate our theoretical
predictions and demonstrate the facilitation of condensate formation by
including a PMA, we conducted a series of micromagnetic simulations using the
LLG framework [46]. We simulate a ferromagnetic system where the magnons are
excited by a spin-transfer torque. We perform calculations at zero
temperature; thus, the system has no thermal magnons. Nonequilibrium magnons
in the magnetic thin film are excited by a spin-transfer torque mechanism
through injection of a spin current on the entire sample surface [31]. The
sign of the spin torque and its amplitude should be chosen so that the
injected magnon population reaches the condensation critical density, we refer
to the SM for simulation details [35]. With the spin-transfer torque
mechanism, we expect nonequilibrium magnons with different wavevectors and
frequencies to be excited. A fraction of these magnons will eventually be
thermalized via repulsive nonlinear magnon-magnon interactions and form a
steady and quasi-equilibrium state of condensate magnons at the bottom of
magnon band dispersion, see Fig. 4.

The numerical simulations confirm the supportive role of PMA in the
condensation process. First, there is a reduction in the threshold of spin
transfer torque necessary to inject the critical magnon density into the
system enabling the system to attain said critical magnon density even at
lower torque amplitudes. Second, final condensate magnons in the presence of
the PMA are more localized around the band minima than in the case where PMA
is absent. Simulations also indicate that PMA shifts the population of
condensate magnons from a symmetric distribution between two band minima to an
asymmetric distribution, Fig. 4. This is in agreement with the analytical
phase diagram in Fig. 3b.

Summary and concluding remarks—. The thermalization of nonequilibrium-magnons
and the stability of the condensate require a repulsive sign for effective
magnon-magnon interactions. This typically requires the presence of strong
dipolar interactions. The presence of PMA is expected to counteract dipolar
interactions. We show that even at intermediate strengths of the PMA field,
the magnon interactions are still repulsive, and the magnon condensate can be
created as a metastable state. We note that the anisotropy increases the band
depth and curvature of the magnon dispersion. These adjustments to the spectra
shape are expected to benefit the condensate formation. From the calculations
of effective magnon-magnon interactions at the band minima, we present a
classification diagram predicting whether the relative number of condensate
magnons in the two degenerate minima might be symmetric. The presence of PMA
strength, in a certain range, will tend to push the condensate toward a more
uneven population distribution between two degenerate band minima.
Micromagnetic simulations, within the LLG framework, confirms our analytical
results and analyses.

## Acknowledgements

The authors thank Anne Louise Kristoffersen for helpful discussions. We
acknowledge financial support from the Research Council of Norway through its
Centers of Excellence funding scheme, project number 262633, ”QuSpin”. A. Q.
was supported by the Norwegian Financial Mechanism Project No.
2019/34/H/ST/00515, ”2Dtronics”.

## References

  * Demokritov  _et al._ [2006] S. O. Demokritov, V. E. Demidov, O. Dzyapko, G. A. Melkov, A. A. Serga, B. Hillebrands, and A. N. Slavin, Bose–Einstein condensation of quasi-equilibrium magnons at room temperature under pumping, Nature 443, 430 (2006).
  * Cherepanov  _et al._ [1993] V. Cherepanov, I. Kolokolov, and V. L’vov, The saga of YIG: Spectra, thermodynamics, interaction and relaxation of magnons in a complex magnet, Phys. Rep. 229, 81 (1993).
  * Onbasli  _et al._ [2014] M. Onbasli, A. Kehlberger, D. H. Kim, G. Jakob, M. Kläui, A. V. Chumak, B. Hillebrands, and C. A. Ross, Pulsed laser deposition of epitaxial yttrium iron garnet films with low Gilbert damping and bulk-like magnetization, APL Mater. 2, 106102 (2014).
  * Demokritov [2022] S. Demokritov, Comment on “Bose–Einstein Condensation and Spin Superfluidity of Magnons in a Perpendicularly Magnetized Yttrium Iron Garnet Film”, JETP Lett. 115, 691 (2022).
  * Anderson and Suhl [1955] P. Anderson and H. Suhl, Instability in the motion of ferromagnets at high microwave power levels, Phys. Rev. 100, 1788 (1955).
  * Suhl [1956] H. Suhl, The non-linear behaviour of ferrites at high signal level, Proceedings of the IRE 44, 1270 (1956).
  * Demidov  _et al._ [2008a] V. Demidov, O. Dzyapko, M. Buchmeier, T. Stockhoff, G. Schmitz, G. Melkov, and S. Demokritov, Magnon kinetics and Bose-Einstein condensation studied in phase space, Phys. Rev. Lett. 101, 257201 (2008a).
  * Rezende [2009a] S. M. Rezende, Theory of coherence in Bose-Einstein condensation phenomena in a microwave-driven interacting magnon gas, Phys. Rev. B 79, 174411 (2009a).
  * Nowik-Boltyk  _et al._ [2012] P. Nowik-Boltyk, O. Dzyapko, V. Demidov, N. Berloff, and S. Demokritov, Spatially non-uniform ground state and quantized vortices in a two-component Bose-Einstein condensate of magnons, Sci. Rep. 2, 482 (2012).
  * Serga  _et al._ [2014] A. A. Serga, V. S. Tiberkevich, C. W. Sandweg, V. I. Vasyuchka, D. A. Bozhko, A. V. Chumak, T. Neumann, B. Obry, G. A. Melkov, A. N. Slavin, _et al._ , Bose–Einstein condensation in an ultra-hot gas of pumped magnons, Nat. Commun. 5, 3452 (2014).
  * Bozhko  _et al._ [2016] D. A. Bozhko, A. A. Serga, P. Clausen, V. I. Vasyuchka, F. Heussner, G. A. Melkov, A. Pomyalov, V. S. L’vov, and B. Hillebrands, Supercurrent in a room-temperature Bose–Einstein magnon condensate, Nat. Phys. 12, 1057 (2016).
  * Sun  _et al._ [2016] C. Sun, T. Nattermann, and V. L. Pokrovsky, Unconventional superfluidity in yttrium iron garnet films, Phys. Rev. Lett. 116, 257205 (2016).
  * Dzyapko  _et al._ [2016] O. Dzyapko, P. Nowik-Boltyk, B. Koene, V. E. Demidov, J. Jersch, A. Kirilyuk, T. Rasing, and S. O. Demokritov, High-resolution magneto-optical Kerr-effect spectroscopy of magnon Bose–Einstein condensate, IEEE Magn. Lett. 7, 1 (2016).
  * Borisenko  _et al._ [2020] I. Borisenko, B. Divinskiy, V. Demidov, G. Li, T. Nattermann, V. Pokrovsky, and S. Demokritov, Direct evidence of spatial stability of Bose-Einstein condensate of magnons, Nat. Commun. 11, 1691 (2020).
  * Demidov  _et al._ [2011] V. E. Demidov, S. Urazhdin, E. Edwards, M. D. Stiles, R. D. McMichael, and S. O. Demokritov, Control of magnetic fluctuations by spin current, Phys. Rev. Lett. 107, 107204 (2011).
  * Demidov  _et al._ [2017] V. Demidov, S. Urazhdin, G. De Loubens, O. Klein, V. Cros, A. Anane, and S. Demokritov, Magnetization oscillations and waves driven by pure spin currents, Phys. Rep. 673, 1 (2017).
  * Bender  _et al._ [2012] S. A. Bender, R. A. Duine, and Y. Tserkovnyak, Electronic pumping of quasiequilibrium Bose-Einstein-condensed magnons, Phys. Rev. Lett. 108, 246601 (2012).
  * Bender  _et al._ [2014] S. A. Bender, R. A. Duine, A. Brataas, and Y. Tserkovnyak, Dynamic phase diagram of dc-pumped magnon condensates, Phy. Rev. B 90, 094409 (2014).
  * Tserkovnyak  _et al._ [2016] Y. Tserkovnyak, S. A. Bender, R. A. Duine, and B. Flebus, Bose-Einstein condensation of magnons pumped by the bulk spin Seebeck effect, Phys. Rev. B 93, 100402 (2016).
  * Andrianov and Moiseev [2014] S. Andrianov and S. Moiseev, Magnon qubit and quantum computing on magnon Bose-Einstein condensates, Phys. Rev. A 90, 042303 (2014).
  * Mohseni  _et al._ [2022] M. Mohseni, V. I. Vasyuchka, V. S. L’vov, A. A. Serga, and B. Hillebrands, Classical analog of qubit logic based on a magnon Bose–Einstein condensate, Commun. Phys. 5, 196 (2022).
  * Bunkov  _et al._ [2021] Y. M. Bunkov, A. N. Kuzmichev, T. R. Safin, P. M. Vetoshko, V. I. Belotelov, and M. S. Tagirov, Quantum paradigm of the foldover magnetic resonance, Sci. Rep. 11, 7673 (2021).
  * Demokritov  _et al._ [2008] S. Demokritov, V. Demidov, O. Dzyapko, G. Melkov, and A. Slavin, Quantum coherence due to Bose–Einstein condensation of parametrically driven magnons, New J. Phys. 10, 045029 (2008).
  * Demidov  _et al._ [2008b] V. Demidov, O. Dzyapko, S. Demokritov, G. Melkov, and A. Slavin, Observation of spontaneous coherence in Bose-Einstein condensate of magnons, Phys. Rev. Lett. 100, 047205 (2008b).
  * Rezende [2009b] S. M. Rezende, Theory of microwave superradiance from a Bose-Einstein condensate of magnons, Phys. Rev. B 79, 060410 (2009b).
  * Tupitsyn  _et al._ [2008] I. Tupitsyn, P. Stamp, and A. Burin, Stability of bose-einstein condensates of hot magnons in yttrium iron garnet films, Phys. Rev. Lett. 100, 257202 (2008).
  * Li  _et al._ [2013] F. Li, W. M. Saslow, and V. L. Pokrovsky, Phase diagram for magnon condensate in yttrium iron garnet film, Sci. Rep. 3, 1372 (2013).
  * Salman  _et al._ [2017] H. Salman, N. G. Berloff, and S. O. Demokritov, Microscopic theory of Bose-Einstein condensation of magnons at room temperature, in  _Universal Themes of Bose-Einstein Condensation_, edited by N. P. Proukakis, D. W. Snoke, and P. B. Littlewood (Cambridge University Press, 2017) Chap. 25, p. 493.
  * Hick  _et al._ [2010] J. Hick, F. Sauli, A. Kreisel, and P. Kopietz, Bose-Einstein condensation at finite momentum and magnon condensation in thin film ferromagnets, Eur. Phys. J. B 78, 429 (2010).
  * Mohseni  _et al._ [2020] M. Mohseni, A. Qaiumzadeh, A. A. Serga, A. Brataas, B. Hillebrands, and P. Pirro, Bose–Einstein condensation of nonequilibrium magnons in confined systems, New J. Phys. 22, 083080 (2020).
  * Divinskiy  _et al._ [2021] B. Divinskiy, H. Merbouche, V. E. Demidov, K. Nikolaev, L. Soumah, D. Gouéré, R. Lebrun, V. Cros, J. B. Youssef, P. Bortolotti, A. Anane, and D. S. O., Evidence for spin current driven Bose-Einstein condensation of magnons, Nat. Commun. 12, 6541 (2021).
  * Lakshmanan [2011] M. Lakshmanan, The fascinating world of the Landau–Lifshitz–Gilbert equation: an overview, Philos. Trans. R. Soc. A 369, 1280 (2011).
  * Landau and Lifshitz [1992] L. Landau and E. Lifshitz, 3 - On the theory of the dispersion of magnetic permeability in ferromagnetic bodies, in  _Perspectives in Theoretical Physics_, edited by L. P. Pitaevski (Pergamon, Amsterdam, 1992) pp. 51–65.
  * Gilbert [2004] T. L. Gilbert, A phenomenological theory of damping in ferromagnetic materials, IEEE Trans. Magn. 40, 3443 (2004).
  * [35] See supplementary material for more details on the magnon hamiltonian and simulation.
  * Holstein and Primakoff [1940] T. Holstein and H. Primakoff, Field dependence of the intrinsic domain magnetization of a ferromagnet, Phys. Rev. 58, 1098 (1940).
  * Streib  _et al._ [2019] S. Streib, N. Vidal-Silva, K. Shen, and G. E. Bauer, Magnon-phonon interactions in magnetic insulators, Phys. Rev. B 99, 184442 (2019).
  * Maier-Flaig  _et al._ [2017] H. Maier-Flaig, S. Klingler, C. Dubs, O. Surzhenko, R. Gross, M. Weiler, H. Huebl, and S. T. Goennenwein, Temperature-dependent magnetic damping of yttrium iron garnet spheres, Phys. Rev. B 95, 214423 (2017).
  * Boardman and Nikitov [1988] A. Boardman and S. Nikitov, Three-and four-magnon decay of nonlinear surface magnetostatic waves in thin ferromagnetic films, Phys. Rev. B 38, 11444 (1988).
  * Princep  _et al._ [2017] A. J. Princep, R. A. Ewings, S. Ward, S. Tóth, C. Dubs, D. Prabhakaran, and A. T. Boothroyd, The full magnon spectrum of yttrium iron garnet, npj Quantum Mater. 2, 63 (2017).
  * Serga, AA and Chumak, AV and Hillebrands, B [2010] Serga, AA and Chumak, AV and Hillebrands, B, Yig magnonics, J. Phys. D: Appl. Phys. 43, 264002 (2010).
  * Buijnsters  _et al._ [2018] F. J. Buijnsters, L. J. Van Tilburg, A. Fasolino, and M. I. Katsnelson, Two-dimensional dispersion of magnetostatic volume spin waves, J. Phys. Condens. Matter. 30, 255803 (2018).
  * Zabel and Farle [2012] H. Zabel and M. Farle,  _Magnetic nanostructures: spin dynamics and spin transport_ , Vol. 246 (Springer, 2012).
  * Demidov  _et al._ [2007] V. Demidov, O. Dzyapko, S. Demokritov, G. Melkov, and A. Slavin, Thermalization of a parametrically driven magnon gas leading to Bose-Einstein condensation, Phys. Rev. Lett. 99, 037205 (2007).
  * Sun  _et al._ [2017] C. Sun, T. Nattermann, and V. L. Pokrovsky, Bose–Einstein condensation and superfluidity of magnons in yttrium iron garnet films, J. Phys. D: Appl. Phys. 50, 143002 (2017).
  * Vansteenkiste  _et al._ [2014] A. Vansteenkiste, J. Leliaert, M. Dvornik, M. Helsen, F. Garcia-Sanchez, and B. Van Waeyenberge, The design and verification of MuMax3, AIP Adv. 4, 107133 (2014).
  * Kreisel  _et al._ [2009] A. Kreisel, F. Sauli, L. Bartosch, and P. Kopietz, Microscopic spin-wave theory for yttrium-iron garnet films, Eur. Phys. J. B 71, 59 (2009).
  * Frostad  _et al._ [2022] T. Frostad, H. L. Skarsvåg, A. Qaiumzadeh, and A. Brataas, Spin-transfer-assisted parametric pumping of magnons in yttrium iron garnet, Phys. Rev. B 106, 024423 (2022).

## Supplemental Material

###  .1 Diagonalization of Magnon Hamiltonian

The total spin Hamiltonian of a thin film, in the $y-z$ plane with a small
perpendicular anisotropy along the $\mathbf{\hat{\textbf{x}}}$ direction
reads,

$\mathcal{H}=\mathcal{H}_{\text{ex}}+\mathcal{H}_{\text{Z}}+\mathcal{H}_{\text{dip}}+\mathcal{H}_{\text{an}}.$
(9)

The exchange energy between neighboring spins reads

$\mathcal{H}_{\rm
ex}=-\frac{1}{2}J_{\text{ex}}\sum_{i,j}\bm{S}_{i}\cdot\bm{S}_{j},$ (10)

where $J_{\text{ex}}>0$ is the ferromagnetic exchange constant. The Zeeman
energy due to an inplane external magnetic field of strength $H_{0}$ along the
$\mathbf{\hat{\textbf{z}}}$-direction reads,

$\mathcal{H}_{\text{Z}}=-g\mu_{B}H_{0}\sum_{j}\bm{S}_{j}^{z},$ (11)

where $\mu_{B}$ is the Bohr magneton and $g$ is the the effective Landé
g-factor. The dipolar field is expressed as [47],

$\displaystyle\mathcal{H}_{\rm dip}$
$\displaystyle=-\frac{1}{2}\sum_{i,j}\sum_{\alpha,\beta}D_{i,j}^{\alpha,\beta}S_{i}^{\alpha}S_{j}^{\beta},$
(12a) $\displaystyle D_{i,j}^{\alpha,\beta}$
$\displaystyle=(g\mu_{B})^{2}(1-\delta_{i,j})\frac{\partial^{2}}{\partial
r_{ij}^{\alpha}\partial r_{ij}^{\beta}}\frac{1}{|\bm{r}_{ij}|},$ (12b)

where $\alpha,\beta$ denote the spatial components $x,y$ and $z$; and
$\bm{r}_{ij}$ is the distance vector between the spin sites $i$ and $j$.
Finally, the PMA anisotropy is given by,

$\mathcal{H}_{\text{an}}=-K_{\text{an}}\sum_{j}(\bm{S}_{j}\cdot\hat{x})^{2}.$
(13)

The Holstein-Primakoff transformation allows us to express the spin operators
in terms of bosonic creation and annihilation operators $\hat{a}^{\dagger}$
and $\hat{a}$ respectively. Using the large-$S$ approximation, we have,
$S^{+}\approx\hbar\sqrt{2S}(\hat{a}-\hat{a}^{\dagger}\hat{a}\hat{a}/(4S))$ ,
$S^{-}\approx\hbar\sqrt{2S}(\hat{a}^{\dagger}-\hat{a}^{\dagger}\hat{a}^{\dagger}\hat{a}/(4S))$,
and $S^{z}=\hbar(S-\hat{a}^{\dagger}\hat{a})$ .

The corresponding noniteracting boson Hamiltonian in the Fourier space reads,

$\mathcal{H}_{2}=\sum_{\bm{q}}A_{\bm{q}}\hat{a}_{\bm{q}}^{\dagger}\hat{a}_{\bm{q}}+\frac{1}{2}B_{\bm{q}}\hat{a}_{\bm{q}}\hat{a}_{\bm{-q}}+\frac{1}{2}B_{\bm{q}}^{\ast}\hat{a}_{\bm{q}}^{\dagger}\hat{a}_{\bm{-q}}^{\dagger},$
(14)

where
$\hat{a}_{j}^{\dagger}=\frac{1}{N}\sum_{\bm{q}}e^{i\bm{k}\cdot\bm{r}_{j}}\hat{a}_{\bm{q}}^{\dagger}$
and $A_{\bm{q}}$ ($B_{\bm{q}}$) is presented in Eq. 4b. We utilize the
Bogoliubov transformation to diagonalize this bosonic Hamiltonian and find the
corresponding noninteracting magnon Hamiltonian,

$\hat{a}_{\bm{q}}=u_{\bm{q}}\hat{c}_{\bm{q}}+v_{\bm{q}}\hat{c}_{\bm{-q}}^{\dagger}$
$\hat{a}_{\bm{q}}^{\dagger}=u_{\bm{q}}^{\dagger}\hat{c}_{\bm{q}}+v_{\bm{q}}\hat{c}_{\bm{-q}}$.
Here, $u_{\bm{q}}=u_{\bm{-q}}$ and $v_{\bm{q}}=v_{\bm{-q}}$ are the Bogoliubov
coefficients,
$u_{\bm{q}}=\big{(}(A_{\bm{q}}+2\hbar\omega_{\bm{q}})/(2\hbar\omega_{\bm{q}})\big{)}^{1/2}$,
$v_{\bm{q}}=\text{sgn}(B_{\bm{q}})\big{(}B_{\bm{q}}-2\hbar\omega_{\bm{q}})/(2\hbar\omega_{\bm{q}})\big{)}^{1/2}$

It can be shown that the off-diagonal terms ($\alpha\neq\beta$) of the dipolar
interaction vanish in the uniform mode approximation for a thin film of
infinite lateral lengths [29, 47]. In this case, the dipolar interaction
contains no three-magnon operator terms. We omit any renormalization
correction to the noninteracting magnon Hamiltonian, as they are of the order
of $1/S$ and small. We define the following parameters in the 4-magnon
interaction amplitudes, introduced in Eq. (6) [27, 28].

$\displaystyle\alpha_{1}$
$\displaystyle=u_{\bm{Q}}^{4}+v_{\bm{Q}}^{4}+4u_{\bm{Q}}^{2}v_{\bm{Q}}^{2},$
(15a) $\displaystyle\alpha_{2}$ $\displaystyle=2u_{\bm{Q}}^{2}v_{\bm{Q}}^{2},$
(15b) $\displaystyle\alpha_{3}$
$\displaystyle=3u_{\bm{Q}}v_{\bm{Q}}(u_{\bm{Q}}^{2}+v_{\bm{Q}}^{2}).$ (15c)

###  .2 Micromagnetic Simulations

Parameter | Value
---|---
Excitation time, first interval | $100\text{\,}\mathrm{ns}$
Excitation time, second interval | $200\text{\,}\mathrm{ns}$
Excitation strength, first interval | $-$1.2\text{\times}{10}^{10}\text{\,}\mathrm{A}\text{\,}{\mathrm{m}}^{-2}$$
Excitation strength, second interval | $-$0.35\text{\times}{10}^{10}\text{\,}\mathrm{A}\text{\,}{\mathrm{m}}^{-2}$$
$(K_{\rm an}=0)$ |
Excitation strength, second interval | $-$0.2\text{\times}{10}^{10}\text{\,}\mathrm{A}\text{\,}{\mathrm{m}}^{-2}$$
$(K_{\rm an}=$0.5\text{\,}\mathrm{\SIUnitSymbolMicro eV}$)$ |
Table 2: Simulation parameters

We perform simulations of magnon creation by spin-transfer torque with and
without out-of-plane anisotropy. We define an initial state in which the
spins, on average, point along the $\mathbf{\hat{\textbf{z}}}$-direction. We
introduce a random noise in the spin direction to mimic the thermal noise as
the initial condition of our simulation. Next, we excite magnons in the
ferromagnetic thin film by applying a spin torque to the entire film surface.
The film is discretized in the lateral directions and uniform in the
$\mathbf{\hat{\textbf{x}}}$ direction. The lateral dimensions of the ilm are
large compared to the film thickness. In this way, the film is effectively a
2D spin system. The LLG equation is solved for each successive time step,
using the open-source software mumax3 [46]. We refer to Ref. 48 for examples
that illustrate the simulation of magnon creation via STT.

The spin accumulation determines the strength of the spin torque, see Ref.
[48]. We start the simulations by exciting magnons with a strong spin torque
(interval $I_{1}$), before lowering the torque strength to keep the
magnetization dynamics in a semi-stable state where the total number of
magnons does not change dramatically over time (interval $I_{2}$). In this
semistable state, the two magnon populations may interact with each other, and
the density of magnons in both minima may oscillate in time. However, the
total number of magnons remains relatively unchanged. The current strength and
time duration of the two intervals are listed in Table 2. The magnetization
data in Fig. 4 is from the last $50\text{\,}\mathrm{ns}$ of the intervals.

The magnon density in the film is proportional to the longitudinal component
of the average magnetization, $\eta=1-\langle m_{z}\rangle$ [48].

The torque strength, or spin accumulation due to the injected current,
determines the number of magnons created. The anisotropy lowers the spin-wave
spectra, meaning that one can use a weaker torque for the stronger anisotropy.
In Fig. 4 we choose the torque strength so that $\eta\approx$0.01$$.

The deviation of spin from the ground-state direction is proportional to the
magnon density. The magnon distribution $\xi$ can be found by performing a
Fourier transform of the transverse magnetization components,
$\xi(q_{y},q_{z})=|\mathcal{F}[{m}_{x}(y,z)]|^{2}+|\mathcal{F}[{m}_{y}(y,z)]|^{2}$
[48].

To reduce the consequences of the finite-size effect in our results, we
analyze the magnetization data in the middle region of the film,
$y,z\in[L/8,7L/8]$, where $L$ is the lateral dimension of the square film.


#  Pay Attention to Convolution Filters:
Towards Fast and Accurate Fine-Grained Transfer Learning ††thanks: CS 194
Final Project Report. Team name: “compressed r-cnn”

Simon Mo Ryan Cheng Ian Fang
UC Berkeley
{xmo,chengruizhe,tinyik} @berkeley.edu

###### Abstract

We propose an efficient transfer learning method for adapting ImageNet pre-
trained Convolutional Neural Network (CNN) to fine-grained image
classification task. Conventional transfer learning methods typically face the
trade-off between training time and accuracy. By adding “attention module” to
each convolutional filters of the pre-trained network, we are able to rank and
adjust the importance of each convolutional signal in an end-to-end pipeline.
In this report, we show our method can adapt a pre-trianed ResNet50 for a
fine-grained transfer learning task within 10 epochs and achieve accuracy
above conventional transfer learning methods and close to models trained from
scratch. Our model also offer interpretable result because the rank of the
convolutional signal shows which convolution channels are utilized and
amplified to achieve better classification result, as well as which signal
should be treated as noise for the specific transfer learning task, which
could be pruned to lower model size.

##  1 Introduction

Our work combines three different topics in deep learning––transfer learning,
fine-grained image classification and attention methods—to solve a practical
problem. In this section, we will first introduce the problem we are trying to
solve and its implication. We will then introduce the three separate topics
that inspired our works. Lastly, we will briefly summarize our contribution
and result.

###  1.1 Problem and Implication

Most image classification problems in real life are fine-grained
classification problems. For example, self-driving cars need classification
models that correctly classify different kinds of street signs; medical
imaging application requires model to distinguish different types of cells.
However, most pre-trained image classification models were trained on ImageNet
dataset. The dataset is very coarse grained; e.g. models were tasked to
distinguish between trucks and cats. It is widely believed that these models
constitute as common feature extractors that can be adapted for specific
tasks. _How can we efficiently take advantage of pre-trained network and adapt
them for fine-grained image classification problem?_

Our objective is two-fold: (1) We want the training process to be as fast as
possible. Ideally, a training process should take less than 10 epochs. This
can be done via taking advantages of learned weights in the pre-trained
network. (2) We want the model accuracy to be as high as possible. It should
perform reasonably well compared to a model that’s trained from scratch.

We hope our work can benefit machine learning practitioners in industry. Using
our methods, a small network that learned to classify fine-grained objects
well can be trained and iterated quickly. For example, an “expert network”
that is able to distinguish very similar classes can serve as part of a larger
ensemble [21] or inference cascade [23] to improve overall accuracy.

###  1.2 Transfer Learning

High accuracy image classification models like ResNet [8] and Inception-V3
[22] that are trained on ImageNet [4] have been made widely available across
deep learning frameworks [20]. Transfer learning attempts to use these pre-
trained models as starting point to adapt the network for a specific task. A
general approach [11] is to freeze the weights of some layers, typically low-
level features, and re-train the high level features and fully-connected
layers. If the new dataset’s distribution is similar to that of ImageNet, it
is recommended to only re-train the fully connected layer. If the new
dataset’s distribution is different from ImageNet, it is recommended to only
keep the low level features and re-train layers that correspond to high level
features.

###  1.3 Fine-Grained Image Classification

Fine-grained image classification tasks are tasks where the objects in the
source images are very similar and requires fine-grained features: for
example, classifying animal species [9] requires the network to pick up
features like color patterns of frog skins or specific shapes of bird beaks.
To solve this problem, the academia has moved away from training model from
scratch. Many researchers have started working on adapting learned features
efficiently to identity important details. Methods like subset feature
learning [7], mixture of convolutional networks [6], or adding visual
attention to image [15] have been proven to be effective. However, many of
these methods are very domain specific, and most importantly, their training
still takes a long time.

###  1.4 Attention Methods

Attention is first introduced in the setting of Neural Machine Translation
[3]. Bahdanau et al. applied attention to each state of RNN to jointly produce
a weighted vector that captures the latent semantic meaning of the sentence
being translated. Xu et al. borrows the same idea and applies it to the task
of image captioning [24]. By using attention mechanism, they achieved state-
of-the-art performance while purposing a novel way of visualizing the regions
in the image that are most heavily weighted when the network is generating
each word in the caption.

###  1.5 Our Contribution

Our work ties together all three ideas to attack the problem of transfer
learning for fine-grained image classification. In particular, we draw
inspiration from attention method in text models and image model to rank
convolution filters in a pre-trained network regarding a specific fine-grained
transfer learning task. Our goal is to explore a method that strikes a good
balance between speed and validation accuracy.

Our final model follows from the intuition that pre-trained models are over-
parameterized. By adding trainable “attention” weights to each convolution
channel and optimize for the classification loss, these “attention” weights
will diverge from their initial states (which are just $1$, mapping identical
filters). After few iterations, attention weights can either amplify a
convolution channel or lower the activation of it. For example, if a
convolution channel detecting the vertical stripes pattern from species A
contributes greatly to the success of correctly classifying species A, we
expect the network would learn to “pay more attention” to this channel by
amplifying the attention weight corresponds to that channel.

We also expect the network will learn to “pay less attention” to features that
are irrelevant/less important to the fine-grained dataset. For example, high
level features (like floppy ears of dogs or the shape of a semi-trailer truck)
in a ImageNet trained model (like Inception-V3) [18] are not really useful for
a fine-grained dataset.

###  1.6 Structure of the Report

This report will be structured as followed. Section 2 will identify several
key literature on which we ground our assumptions and methods. Section 3 will
discuss our experimental setup and baseline models. Section 4 will discuss
detail implementation of our models and training scheme. Section 5 will
identify the set of experiments we ran in order to explore many facets of
efficent fine-grained transfer learning. Finally, in section 6 and 7, we will
summarize our result and explore future works.

##  2 Related Work

This section is organized by a series of key assumptions we made.

###  2.1 Key Assumption 1: Large Networks are Over-Parametrized

Although almost a consensus among deep learning community, we found
interesting theoretical and experimental results from Frankle et al [5].
Frankle et al. explore so-called “The Lottery Ticket Hypothesis” and realized
that the success of many large network can be attributed to the large number
of layers and parameters that made possible for successful sub-networks to
appear, which are usually discovered via pruning.

###  2.2 Key Assumption 2: ImageNet Pre-Trained Convolution Channels are Good
Feature Extractors

In particular, we need to assume that the base network that we adapt for
transfer learning purpose is a high quality network in terms of its
convolution channels. We assume the features learned from training well-design
networks like ResNet and Inception using ImageNet dataset are generally
transferable. This assumption is confirmed by the work “What makes ImageNet
good for transfer learning?” from Huh et al. [10]. ImageNet is confirmed to be
a good dataset to produce general convolution feature extractors.

###  2.3 Key Assumption 3: Convolution Channels Can be Ranked and Pruned

In particular, we are assuming that not all the convolution channels are
useful. Works on model compression and network pruning provide for us a solid
experimental data for this assumption.

Most of the work in convolution channel pruning involves a heuristic defined
channel ranking methods, for examples: ranking channels by their $ell_{1}$
norms [13], ranking channels by combinatorially determine their effect on
validation score [2], and the state-of-the-art ranking methods is first-order
Taylor expansion with respect to the channel to closely match the cost
function of original network [16].

Our approach is different. Instead of a human defined heuristic about image
norm or Taylor expansion, we want the network to figure out how to rank
filters by itself. We only provide the network with the an objective of
classification score and a set of resource (convolution filters).

###  2.4 Key Assumption 4: Attention Should Be Regularized

If we just assign the weight of $1$ to each convolution filters, the network
will move very slowly or even remain stagnant. An important lesson we take
from Kim et al. about visual attention is the importance of regularized
attention [12]. Although the work is about applying visual attention to self-
driving cars, it provides important intuitions about regularizing visual
attention and forcing it to look for new objects by tuning $\lambda$ as the
strength of regularization.

Our key take-away is just that adding attention in visual model is not enough.
Especially in our model, where the attention is about _the hierarchical
relationship and strength of the convolution channels_ , just adding a scalar
weight of $1$ and freeze the convolution filter weights are not good enough.
We need to add some prior about the attention weights. We further experiment
with and discuss the different forms of regularization in section 5.1.1.

##  3 Experimental Setup

###  3.1 Data Source and Pre-Processing

We chose a subset of the INaturalist 2018 Dataset, in which there are 11,156
images of 144 species of amphibians. This dataset resembles a specific fine-
grained classification dataset typically encountered in real life. It has
quite a significant class imbalance for some classes, and significantly
different data distribution from that of ImageNet. The amphibians share many
common features and can be extremely similar among different species. Since
they were usually photoed in their natural habitats, it is indeed a very hard
dataset, yet general and diverse enough to represent a specific fine-grained
classification task (example: Figure 1).

Figure 1: Example of iNaturalist Dataset

The images originally come in different sizes and shapes, and all of them are
reshaped without cropping to 224x224 for ResNet models and 299x299 for
Inception-V3. They are then normalized according to standard ImageNet
practice, with $\mu=\begin{bmatrix}0.485&0.456&0.406\end{bmatrix}$ and
$\sigma=\begin{bmatrix}0.229&0.224&0.225\end{bmatrix}$

Data augmentation was used to solved the problem of class imbalance and over-
fitting in some experiments, which will be further discussed in section 5.1.2.

Data is split 7-1.5-1.5 for training, validation, and testing respectively.

###  3.2 Tools

We used PyTorch[19] as our Machine Learning library for this project. We chose
PyTorch over TensorFlow or other popular library because of its flexibility.
PyTorch uses dynamic computational graph rather than a static one.

In addition, PyTorch provides a simple way to customize pre-trained models to
facilitate our experiment. For example, to freeze/unfreeze a specific variable
inside the model, one can simply set .requires_grad to desired boolean value;
adding Attention weight to a specific channel also only requires minor changes
to the existing pre-trained models.

All the pre-trained models used in our experiments (ResNet-{34,50,101},
Inception-V3) come from PyTorch Model Zoo [20].

Models were mostly trained on EC2 instances using Tesla K80 GPUs, and some
were trained on Pascal Titan X.

###  3.3 Baseline Model

We attempted different popular models pre-trained on ImageNet. Most
experiments were done using ResNet 50, but ResNet 34, ResNet 101 and
Inception-V3 were also used to investigate compatibility of our method with
different pre-trained model structures, and model depths. More on this in
section 5. We showed that our attention module and training strategies are
generally applicable to models with different architectures and depths.

###  3.4 Evolution of our model

In order to solve the problem of filter redundancy in transfer learning on
specific datasets, we first explored the idea of pruning, which ranks the
convolutional filters based on some heuristic algorithms, and removes ones
with less activation. Such a heuristic algorithm was proposed in [16], which
uses Taylor expansion to approximate the loss function and compares the loss
when a certain filter is masked out to zero. This gives a heuristic estimation
of the impact of this filter. However, we want to propose an end-to-end method
that uses attention weights as a measure of usefulness of a convolutional
filter. The soft attention amplifies useful signals and suppresses redundant
ones, which achieves a similar effect to pruning.

##  4 Final Model

Model Architecture

The final model has attention modules attached at the output of each of the
convolutional layers of the pre-trained models. This is done by multiplying
each pre-trained convolution filter by a trainable scalar weight initialized
at $1$.

One to two epochs of the FC layers were trained to adjust the classifier to
the fine-grained dataset, so that errors can be better back propagated to the
attention modules. Next we fixed the entire network, and only trained the
attention modules to identify the feature extractors in the pre-trained
network that are useful for the specific fine-grained task. Training of the
batchnorm layers was interspersed within training of the attention modules, in
order to fight overfitting and adjust for variance shift.

###  4.1 Pseudocode

The core of our model boils down to the pseodo-code shown below:

    # We initialize attention weights
    # as follows and fill it with 1
    attn_weights = Parameter(FloatTensor(
    out_channels, in_channels, 1,1))
    ...
    # And we modify the "forward" function
    # in 2D convolution module
    attn_paid = conv_weight*attn_weights
    return F.conv2d(input, attn_paid, ...)

##  5 Experiment Result and Discussion

In this section, we describe a set of experiments we ran concerning many
facets of fine-grained transfer learning and the specifics of our method. For
each experiment, we will describe our setup, our result, and our findings.

The investigation section contains experiments that were successfully ran and
worked as expected. The visualization contains result from our experiment with
interpretability of attention models. The surprise section consists of
experiments that either gave us surprising results or just did not work out.

###  5.1 Investigation

####  5.1.1 What is the best attention loss/regularization?

We wanted to identify useful filters from redundant ones, and thus applied
three different loss for attention weights. Initially, most attention weights
clustered near 1, and were reluctant to differentiate filters. L1 or L2 norms
of the attention weights were added to the loss function as regularization. L1
norm resulted in a sparser and more polarized distribution of the attention
weights, but L2 converged slightly faster. Both achieved similar performance
in terms of accuracy.

$\ell_{1}(a_{j})=\|a_{j}\|_{1}$ (1)

$\ell_{2}(a_{j})=\|a_{j}\|_{2}$ (2)

In order to further help the attention weights diverge, we proposed a modified
L2 loss that penalizes the distance of each attention weight to 1, which also
serves as regularization. This penalty scheme was most commonly used in our
model.

$\ell_{3}(a_{j})=-\|a_{j}-1\|_{1}^{2}$ (3)

The loss function is thus:

$\mathcal{L}=-\sum_{i=1}^{n}y^{(i)}\log(p)^{(i)}+\lambda\cdot\sum_{j=1}^{F}\ell(a_{j})$
(4)

where F is the number of filters in the convolutional layers, $a_{j}$ is the
attention weight vector for filter j, and $\lambda$ is a hyper-parameter.

Loss |  Top 1 Val |  Top 3 Val |  Best Epoch
---|---|---|---
L1 |  38.6% |  60% |  8
L2 |  42.7% |  63.4% |  10
Table 1: Result from different loss functions, using same FFAAABAAABAA
training scheme. F: fully connected layers; A: attention layers; E: all layers
other than attention layers; B: batch norm layers

Figure 2: Different Attention Regularization, the wider the better

As figure 2 shows, both regularization scheme/prior lead to expected result
distribution. The $\ell_{1}$ regularization put a sparsity inducing prior on
attention weights; the result shows many weights are reduced close to zero.
Surprisingly, there are few attention weights that are still amplified (value
$>1$). We expect these weights help identify the subset of convolution
channels that are useful for fine-grained classification with our dataset. The
histogram for $\ell_{2}$ attention loss show similar but smoother result. The
distribution of weights gradually diverge from $1$ and shift towards either
$0$ or $2$ according to their significance in this particular fine-grained
classification task.

Our result is as shown in Table 1.

####  5.1.2 Does data augmentation help?

Data augmentation was done with a weighted sampler that samples from each
class with probability inverse to the number of training images in that class.
This solves the problem of class imbalance and also serves as a regularizer to
combat overfitting.

Training Scheme |  Top 1 Val |  Top 3 Val |  Epoch |  Data Aug
---|---|---|---|---
FC*10 |  28.9% |  46.6% |  10 |  No
FC*10 |  28.5% |  48.4% |  10 |  Yes
FC+Att |  42.7% |  63.4% |  12 |  No
FC+Att |  50.5% |  70.0% |  12 |  Yes
Table 2: We used some fixed training schemes with good results for controlled
experiment.* FC+Att: FFAAABAAABAA

Our result is as shown in Table 2. Using data augmentation does improve the
performance and it works particularly well with attention modules.

####  5.1.3 How Does Different Model Architecture Affect the Score?

Our result is as shown in Table 3. We expect the validation accuracy to be
about the same across different depth of the network as well as different
model architecture.

As the depth of ResNet increase, we see increase in validation accuracy; this
result is justified by the fact that there are much more convolution channel
as the network goes deeper as well as more interesting intermediate features
ready to be combined.

If we switch the base architecture to a different comparable ImageNet model,
Inception V3, we see the performance is not degraded. This show the
compatibility of our method with any ImageNet models; as long as it contains
convolution.

Model |  Top 1 Val |  Top 3 Val |  Best Epoch
---|---|---|---
ResNet34 |  39% |  60% |  9
RestNet50 |  43% |  63% |  10
RestNet101 |  47% |  66% |  10
Inception V3 |  41% |  62.7% |  5
Table 3: Result from different model architecture, using same FFAAABAAABAA
training scheme.

####  5.1.4 Is The Accuracy Good Enough (Compare with 100 hours models)?

There is a baseline model [1] trained from scratch using Inception-V3 on a
larger and a more complete dataset of animal and plant species, of which our
amphibian dataset is a subset. It took 100 epochs for the model to achieve a
final top 3 accuracy of $77\%$.

In comparison, our model, though trained on a more specific dataset, were able
to achieve the highest $70\%$ top 3 validation accuracy in just under 15
epochs, which took around _13 minutes_ on a Pascal Titan X GPU to train.

Thus, the attention model is more suitable for classification tasks where high
accuracy is not the absolute priority but require fast iteration of training
and decent results. This makes it possible to train a large number of
personalized models with personal data.

###  5.2 Interpretability and Visualization

In this section, we present two visualizations that showcase the effectiveness
of our method for picking up good signals.

Figure 3: Direct Feature Amplified by Attention Figure 4: Auxiliary Feature
Amplified by Attention

We visualize two channels (Figure 3, Figure 4) in the third to last
convolution layer of ResNet 50 (res5c_branch2a). The channels are the top-2
channels ranked by the “attention” weights on such channels after training
attention weights based on ResNet50 for few iterations using $\ell_{2}$
penalty to regularize the attention weights. Convolution features are
visualized by optimizing random noise image to maximize the activation of a
convolution channel (activation measured using mean of the image, applied
Gaussian blur or Bilateral Filter, described in [17]). The images are sampled
from top-10 images that maximally activate the channels ranked by $\ell_{1}$
norm of the feature map, the metric as inspired by convolution filter pruning
method in [14].

_What does these visualizations shows?_

Figure 3 shows a high level feature of spines of animals. The attention method
is able to pickup the specific feature that significantly helps classification
for this fine-tuned dataset. We also included a negative image sample. The
leaf structure strongly resembles the spine structure therefore also activates
this filters.

Figure 4 shows another high level feature. However, this feature is an
auxiliary feature that helps identify the surrounding environment of different
types of frogs. The vertical leaf structure was picked up as a strong signal
that can assists the network in identifying species.

###  5.3 Surprise

####  5.3.1 Attention shape

To answer this question, we ran two sets of controlled experiments. For the
”Out Only” method, we assigned the same attention weight to each input channel
in the same 3D convolutional filter. For the ”In $\times$ Out” method, we
assigned a different weight to each input channel.

As a concrete example, the first convolution layer of the ResNet-Family model
takes as input a depth 3 RGB representation of the image and outputs a depth
64 feature map, for method 1 ”Out Only” there are 64 attention weights in this
layer while method 2 ”In $\times$ Out” have 3 $\times$ 64 attention weights.

Attention Channel |  Top 1 Val |  Top 3 Val |  Best Epoch
---|---|---|---
Out Only |  41.2% |  60.3% |  5
In $\times$ Out |  42.7% |  63.4% |  10
Table 4: Result from different attention shape, using same FFAAABAAABAA
training scheme.

Comparing the number of parameters in either model (in Table 4), we initially
expect the second method to outperform the first one by a large advantage.
However, our experiment result only reveals a relatively small boost of
performance. This is surprising because we thought a more sophisticated
attention scheme will result in a drastically higher performance.

One possible explanation for this surprising result is the hierarchical
connection between a single $H\times W$ convolution filter and it’s filter
group, where all filters in the group collectively result in just one single
feature map:

$\text{out}(N_{i},C_{out_{j}})=\sum_{k=0}^{C_{in}-1}\text{weight}(C_{out_{j}},k)\star\text{input}(N_{i},k)$

where $\star$ is the convolution operator, $C_{out_{j}}$ is the $j$th feature
map. A single feature map is produced by adding the convoluted result from
each filter in the group via $\sum_{k=0}^{C_{in}-1}$. _We hypothesize there is
a strong connection between different filters in the same group_ , making
attention weight less effective.

Despite of their similar performance, the ”Out Only” method converges to the
best model much faster than the ”In $\times$ Out” method (5 versus 10 epochs).

Our result is as shown in Table 4.

####  5.3.2 Do Traditional Transfer Learning Methods Still Work?

We also explored different levels of ’cut’, which are widely used in
traditional transfer learning methods. We define the level of ’cut’ as the
number of layers that were not frozen when training. We trained different
blocks of the pre-trained network for different number of epochs, including
(1) Fully Connected Layers, (2) Block 4 of ResNet, and (3) the entire network
end to end.

Method (1) gives us a good linear transformation from fixed high level feature
(after average pooling) to the probability of classes. However it is prone to
over-fit within few epochs. The best top 3 validation score we got is 46.6%.

Method (2) unfreezes the last large block of ResNet and attempts to re-train
high level features. We found the accuracy to be slightly higher: 56.6%

Surprisingly, method (3) uses the entire pre-trained network as
initialization. The final best validation score is 65.05%, which is better
than attention methods. However, this can be justified by the fact that all
elements in every convolution filter are subject to optimization. Amplifying a
convolution activation by a scalar weight has less complexity than fine-tuning
every filter.

##  6 Lesson Learned

We investigated the effectiveness of channel wise attention in the context of
transfer learning for fine-grained specific classification tasks. Compared to
just re-training the fully connected layers, the attention modules are
surprisingly effective in quickly identifying useful in and out convolution
channels, and it raises the performance of the model to nearly $70\%$ top 3
validation accuracy in under 12 epochs. An inception-V3 model trained with 100
epochs achieved $77\%$ top 3 accuracy. Our attention modules indeed provides a
fast and relatively high performance model for transfer learning.

We did different investigations to find out the best training strategy,
attention penalty, level of cut, as well as the effect of data augmentation.
We used different training strategies and used different combinations of FC,
Attention, and BN layers. We discovered that FC layers quickly adjusted the
classifier to the fine-grained data distribution but overfits after a few
epochs, whereas attention layers effectively boosts the accuracy in a few
epochs after the FC layers are trained. After the attention layers, however,
the output feature maps are no longer well-behaved. When training batchnorm
layers that are interspersed in between attention layers, accuracy drops
slightly due to its regularization effect, but they help attention modules to
achieve better results.

Furthermore, the attention weights would cluster around 1 when only using
cross entropy loss, but the addition of attention penalty encourages them to
diverge and form different distributions, which yields roughly equally good
performance. We also found the effectiveness of attention in picking
interpretable image features that can help explain the network’s decision.

Our project provides an intuitive approach for fine-grained transfer learning.
We still have a long way to go in exploiting pre-trained networks and adapting
them for fine-grained or even personalized computer vision models.

##  7 Future Work

There are multiple lines of future work possible beyond our current stage.

  1. 1.

A natural continuation of the project is to prune convolution channels that
have attention weights smaller than a certain threshold to reduce the size of
the final model as well as to improve accuracy. Attention weights provide us
with a good ranking of features available for pruning.

  2. 2.

Another direction is to further exploit the benefit of interpretability that
attention scheme brings to the table. The attention weights are highly
interpretable.

  3. 3.

In our current approach, we have to experiment with different architectures
and training schemes to find a satisfiable configuration. A natural step
beyond this project will be to use reinforcement learning, especially meta
learning strategies to let the model to learn the best way of integrating
attention modules given a set of pre-trianed filters with the objective to
classify fine-grained data correctly.

##  8 Team Contributions

Work was evenly split among the group. Simon came up the initial idea, the
group refined the idea and designed experiment setups together. Each of us
experimented with multiple controlled experiments so that we can efficiently
allocate GPU usage. We all worked on the poster, final report and code.
111https://github.com/simon-mo/attn-attn

## References

  * [1] Anonymous.  Pretrain Inception V-3 Model on iNaturalist 2018 DataSet: https://github.com/macaodha/inat_comp_2018.
  * [2] S. Anwar, K. Hwang, and W. Sung.  Structured pruning of deep convolutional neural networks.  CoRR, abs/1512.08571, 2015.
  * [3] D. Bahdanau, K. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate.  CoRR, abs/1409.0473, 2014.
  * [4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In CVPR09, 2009.
  * [5] J. Frankle and M. Carbin.  The lottery ticket hypothesis: Training pruned neural networks.  CoRR, abs/1803.03635, 2018.
  * [6] Z. Ge, A. Bewley, C. McCool, B. Upcroft, P. Corke, and C. Sanderson.  Fine-grained classification via mixture of deep convolutional neural networks.  CoRR, abs/1511.09209, 2015.
  * [7] Z. Ge, C. McCool, C. Sanderson, and P. I. Corke.  Subset feature learning for fine-grained category classification.  CoRR, abs/1505.02269, 2015.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [9] G. V. Horn, O. Mac Aodha, Y. Song, A. Shepard, H. Adam, P. Perona, and S. J. Belongie.  The inaturalist challenge 2017 dataset.  CoRR, abs/1707.06642, 2017.
  * [10] M. Huh, P. Agrawal, and A. A. Efros.  What makes imagenet good for transfer learning?  CoRR, abs/1608.08614, 2016.
  * [11] A. Karpathy.  CS231n: Transfer Learning: http://cs231n.github.io/transfer-learning/.  2018\.
  * [12] J. Kim and J. F. Canny.  Interpretable learning for self-driving cars by visualizing causal attention.  CoRR, abs/1703.10631, 2017.
  * [13] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf.  Pruning filters for efficient convnets.  CoRR, abs/1608.08710, 2016.
  * [14] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf.  Pruning filters for efficient convnets.  CoRR, abs/1608.08710, 2016.
  * [15] X. Liu, T. Xia, J. Wang, and Y. Lin.  Fully convolutional attention localization networks: Efficient attention localization for fine-grained recognition.  CoRR, abs/1603.06765, 2016.
  * [16] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz.  Pruning convolutional neural networks for resource efficient transfer learning.  CoRR, abs/1611.06440, 2016.
  * [17] C. Olah, A. Mordvintsev, and L. Schubert.  Feature visualization.  Distill, 2017.  https://distill.pub/2017/feature-visualization.
  * [18] C. Olah, A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Ye, and A. Mordvintsev.  The building blocks of interpretability.  Distill, 2018.  https://distill.pub/2018/building-blocks.
  * [19] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  2017\.
  * [20] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  PyTorch Model Zoo: https://pytorch.org/docs/master/torchvision/models.html.  2018\.
  * [21] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. V. Le, G. E. Hinton, and J. Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  CoRR, abs/1701.06538, 2017.
  * [22] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.  Rethinking the inception architecture for computer vision.  CoRR, abs/1512.00567, 2015.
  * [23] X. Wang, Y. Luo, D. Crankshaw, A. Tumanov, and J. E. Gonzalez.  IDK cascades: Fast deep learning by learning not to overthink.  CoRR, abs/1706.00885, 2017.
  * [24] K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  CoRR, abs/1502.03044, 2015.