# Anomaly-induced Chiral Mixing in Cold and Dense Matter

Chihiro Sasaki  Institute of Theoretical Physics, University of Wroclaw,
PL-50204 Wroclaw, Poland

###### Abstract

We construct the spectral functions for light vector mesons at finite density
and temperature in the presence of a novel mixing between parity partners,
induced by baryon density via the Wess-Zumino-Witten action. As the main
origin of in-medium broadening, a set of baryon resonances that strongly
couple to the vector mesons and the modifications of kaons and anti-kaons due
to the Kaplan-Nelson term are included. It is shown that the vector spectra,
even with the broadening effects, exhibit sizable signatures of chiral
symmetry restoration thanks to the chiral mixing depending on three-momenta
carried by the vector mesons. Those spectral functions are used to calculate
the integrated production rates of lepton pairs, and a proper binning in
momenta and potential decrease in the vector-meson masses due to chiral
symmetry restoration are discussed in quantifying the signatures.

##  I Introduction

Light vector mesons, especially their dynamical properties arising from
spontaneously broken chiral symmetry in QCD, have been studied extensively in
various approaches Rapp ; HH ; RWvH ; Rapp:2011zz ; FS . In-medium
modifications of vector spectral functions were anticipated due to the
interactions with a hadronic medium, which carry imprints of restoration of
chiral symmetry at high temperature and/or baryon density. Strong
modifications have been indeed measured via dilepton production in heavy-ion
collisions NA60 ; vHR , whereas it remains inconclusive how to quantify those
modifications of vector mesons as the direct consequence of chiral symmetry
restoration since the vector spectra themselves are not the order parameter.

The ideal way is to measure the spectral functions both in the vector and its
parity partner, axial-vector channels to examine if the two spectra are nearly
degenerate as expected with unbroken chiral symmetry. Given the difficulty
that it is elusive to construct the axial-vector spectrum in heavy-ion
experiments, the key phenomenon is that the vector mesons mix with the axial-
vector mesons at finite temperature and density, known as chiral mixing. The
chiral mixing effect induced by pions can be quantified in a model-independent
way at low temperature/density DEI ; Krippa98 ; Chanfray . Its extrapolation
toward the chiral symmetry restoration requires a caution. There exists a
systematic calculation HRhot  to construct the in-medium axial-vector spectral
function at finite temperature from the vector correlator that successfully
describes the dilepton data, via the Weinberg sum rules WSR . It was reported
there that the $a_{1}$ meson mass drops substantially toward the $\rho$ mass
together with a width broadening. The observed trend is consistent with
another study on the chiral mixing in which the Nambu-Goldstone bosons and
vector mesons with opposite parity were explicitly included in a chiral
Lagrangian to perform a systematic computation at loop level HSWhot . It was
shown that the chiral symmetry restoration forces the chiral mixing to vanish,
which ensures the vector and axial-vector spectra degenerate while it is
totally distinct from the result of a naive extrapolation of the mixing
theorem leading to a maximal mixing UBW .

There exists a novel class of chiral mixing induced by baryon density via
anomaly in a dense medium. It was first recognized in a holographic QCD with
the Chern-Simons action and the emergent mixing operator results in the
modified dispersion relations of transverse polarizations of vector and axial-
vector states DHholo . The low-energy Lagrangian was then reformulated in the
standard 4-dimensional chiral effective theory and a brief estimate of the
mixing strength as a function of baryon density was given in the mean field
approximation HSdense . Although the resultant mixing is smaller than the
holographic estimate by one order of magnitude, the chiral mixing is
reinforced with decreasing mass difference between parity partners, i.e. the
onset of chiral symmetry restoration, and leads to the structural change of
the vector spectra and the dilepton production rates Sasaki .

We are yet to extend the previous calculations of vector spectra in the
presence of density-induced chiral mixing. The mixing effect results in the
spectral function somewhat broadened since the transverse modes obey the non-
standard dispersion relations leading to a downward shift of their masses and
the final spectrum as a superposition of all the polarization states is no
longer in the Breit-Wigner distribution HSdense ; Sasaki  where the main
origin of standard broadening due to baryon resonances that strongly couple to
the vector mesons in nuclear matter was not included.

In this paper, we will introduce those baryon resonances into the computation
of in-medium vector spectral functions for the $\rho/\omega$ states at finite
density and low temperature. We will also include modifications of kaons and
anti-kaons in nuclear matter due to the Kaplan-Nelson term kaon  to extend our
previous calculation of the $\phi$ spectral function. The updated set of
spectral functions will be used to calculate the production rates of dilepton
to examine to what extent the signatures of chiral symmetry restoration via
the chiral mixing would stay in such observable quantities in heavy-ion
collisions.

##  II Chiral mixing via the Wess-Zumino-Witten term

Non-vanishing baryon chemical potential violates explicitly charge-conjugation
invariance. Consequently, effective Lagrangians contain a set of new operators
that are prohibited in matter-free space. In the system composed of the Nambu-
Goldstone bosons as well as the lowest vector $V_{\mu}$ and axial-vector
$A_{\mu}$ mesons, there exists an operator leading to a direct mixing of
vector with axial-vector states, the chiral mixing, in the form of

${\mathcal{L}}_{\rm
mix}=2c\,\epsilon^{0\mu\nu\lambda}\mbox{tr}\left[\partial_{\mu}V_{\nu}\cdot
A_{\lambda}+\partial_{\mu}A_{\nu}\cdot V_{\lambda}\right]\,,$ (1)

with the total anti-symmetric tensor $\epsilon^{0123}=1$ and a parameter $c$
to be fixed later. This term can be deduced from the Chern-Simons action in a
model based on AdS/CFT correspondence DHholo , and alternatively from the
Wess-Zumino-Witten (WZW) action KM . The mixing term (1) modifies dispersion
relations for the transverse polarizations as

$p_{0}^{2}-\vec{p}^{2}=\frac{1}{2}\left[m_{V}^{2}+m_{A}^{2}\pm\sqrt{(m_{A}^{2}-m_{V}^{2})^{2}+16c^{2}\vec{p}^{2}}\right]\,,$
(2)

where the lower sign is for the vector and the upper one for the axial-vector
mesons. The longitudinal polarizations obey the standard dispersion relation,
$p_{0}^{2}-\vec{p}^{2}=m_{V,A}^{2}$.

The holographic model DHholo  yields a rather strong density dependence in the
mixing parameter, $c\simeq 1$ GeV at the normal nuclear matter density
$\rho_{0}$. This, however, results in the onset of vector meson condensation
only slightly above $\rho_{0}$, which is an apparent drawback of the large
$N_{c}$ prescription that the approaches with AdS/CFT depend on. In contrast,
the standard chiral approach with the gauged WZW action HSdense  yields a
moderate density-effect, with the expectation value of iso-scalar vector meson
$\langle\omega_{0}\rangle=g_{\omega NN}\cdot\rho_{B}/m_{\omega}^{2}$, leading
to $c=g_{\omega\rho a_{1}}\langle\omega_{0}\rangle\simeq 0.1$ GeV at
$\rho_{0}$.

Whereas the weak mixing is negligible in the propagators, thus in vector
spectral functions as well, the key feature is that the mixing strength
linearly increases with baryon density $\rho_{B}$. In Ref. Sasaki , it has
been shown that the restoration of chiral symmetry enhances the chiral mixing
and the structural changes in vector spectra serve as a clear signature of
chiral restoration in cold and dense matter. Such a drastic deformation
further yields a non-negligible contribution to dilepton production rates, and
thus in principle it is measurable in heavy-ion collisions at FAIR, NICA and
J-PARC. Our previous calculations Sasaki  were performed without explicit
baryonic resonances arising from the direct coupling of vector mesons to the
surrounding nucleons. Those resonances are known to generate significant
modifications in self-energies in the $\rho/\omega$ sectors. In the invariant-
mass region relevant to the $\phi$ meson, there are no $\phi N$ resonances but
the kaon and anti-kaon change their properties in nuclear matter kaon . In the
next section, we will introduce those characteristic effects to estimate the
interplay between the vector mesons and the chiral mixing induced by density.

##  III Vector spectral functions

We begin with the current-current correlation functions in matter;

$G_{V,A}^{\mu\nu}(p_{0},\vec{p})=P_{L}^{\mu\nu}G_{V,A}^{L}(p_{0},\vec{p})+P_{T}^{\mu\nu}G_{V,A}^{T}(p_{0},\vec{p})\,,$
(3)

with the polarization tensors

$\displaystyle P_{T,\mu\nu}$ $\displaystyle=$ $\displaystyle g_{\mu
i}\left(\delta_{ij}-\frac{\vec{p}_{i}\vec{p}_{j}}{\vec{p}^{2}}\right)g_{j\nu}\,,$
$\displaystyle P_{L,\mu\nu}$ $\displaystyle=$
$\displaystyle-\left(g_{\mu\nu}-\frac{p_{\mu}p_{\nu}}{p^{2}}\right)-P_{T,\mu\nu}\,.$
(4)

Formulating the chiral Lagrangian for pions, vector and axial-vector mesons in
the non-linear realization BKY ; HSghls , the longitudinal and transverse
parts read HSdense

$\displaystyle
G_{V}^{L}=\left(\frac{g_{V}}{m_{V}}\right)^{2}\frac{-s}{D_{V}^{L}}\,,\quad
G_{V}^{T}=\left(\frac{g_{V}}{m_{V}}\right)^{2}\frac{-sD_{A}^{T}+4c^{2}\vec{p}^{2}}{D_{V}^{T}D_{A}^{T}-4c^{2}\vec{p}^{2}}\,,$
$\displaystyle
G_{A}^{L}=\left(\frac{g_{A}}{m_{A}}\right)^{2}\frac{-s}{D_{A}^{L}}\,,\quad
G_{A}^{T}=\left(\frac{g_{A}}{m_{A}}\right)^{2}\frac{-sD_{V}^{T}+4c^{2}\vec{p}^{2}}{D_{V}^{T}D_{A}^{T}-4c^{2}\vec{p}^{2}}\,,$
(5)

with $s=p_{0}^{2}-\vec{p}^{2}$ and the coupling of the vector/axial-vector
meson to the corresponding current $g_{V,A}$ as well as the propagator inverse
without the mixing $D_{V,A}^{L,T}=s-m_{V,A}^{2}-\Sigma_{V,A}^{L,T}(s)$. The
self-energies $\Sigma_{V}^{L,T}$ will be computed in the presence of baryon
resonances in the $\rho$-meson sector and the modified kaons in the
$\phi$-meson sector. The spin-averaged correlators are given by
$G_{V,A}=\frac{1}{3}\left(G_{V,A}^{L}+2G_{V,A}^{T}\right)$. The labels $(V,A)$
refer to the iso-vector $(\rho,a_{1})$ and the iso-singlet
$(\omega,f_{1}(1285))$ and $(\phi,f_{1}(1420))$ mesons. The coupling constants
to the vector current are related to $g_{\rho}=0.119$ GeV2 via chiral
symmetry, so that

$g_{\omega}=\frac{1}{3}\frac{m_{\omega}^{2}}{m_{\rho}^{2}}g_{\rho}\,,\quad
g_{\phi}=\frac{\sqrt{2}}{3}\frac{m_{\phi}^{2}}{m_{\rho}^{2}}g_{\rho}\,.$ (6)

We shall focus on the vector spectral functions at chiral symmetry restoration
since the mixing effect becomes maximal there. We will also neglect meson-loop
effects which are small at low temperature, $T\ll m_{\pi}$, relevant to our
study. The in-medium masses are nearly degenerate, $m_{V}\simeq m_{A}$,
leading to $G_{V}\simeq G_{A}$. As demonstrated in Ref. Sasaki , the central
quantities, i.e. the mass difference between vector and axial-vector mesons
and the mixing strength, were estimated at temperature $T=50$ MeV, and found
to be $\delta m=47$ MeV and $c=98$ MeV with the chiral crossover
$\rho_{B}^{\rm crit}/\rho_{0}=2.5$, within an extended parity-doublet model
that describes not only the properties of nuclear ground states in symmetric
matter but also those of neutron star matter Marczenko . Below, we will assume
for simplicity the vector-meson mass being independent of density and
temperature.

###  III.1 $\rho$/$\omega$ mesons

The importance of $\rho N$ interactions in nuclear matter was first discussed
in Ref. FP  where two p-wave states, $N(1720)$ and $\Delta(1905)$, were
considered because of their large branching ratios into $\rho$ and $N$. Due to
the $\rho N$ interactions, the expected three states, the $\rho$ meson and two
resonance-hole states, are all mixed, and lead to the spectral function
strongly smeared.

Since the photo-absorption data requires the inclusion of additional low-lying
resonances, we will include a larger set of resonances following Ref. RUBW ,
summarized in Table 1.

B | $l_{\rho N}$ | SI |  $\Gamma^{0}_{\rho N}$ [MeV] | $\left({f^{2}_{\rho BN}}/{4\pi}\right)^{2}$ |  $\Gamma^{\rm med}$ [MeV]
---|---|---|---|---|---
$\Delta(1232)$ | $p$ | $16/9$ | $-$ | $23.2$ | $15$
$N(1520)$ | $s$ | $8/3$ | $24$ | $5.5$ | $250$
$\Delta(1620)$ | $s$ | $8/3$ | $22.5$ | $0.7$ | 50
$\Delta(1700)$ | $s$ | $16/9$ | $45$ | $1.2$ | $50$
$N(1720)$ | $p$ | $8/3$ | $105$ | $9.2$ | $50$
$\Delta(1905)$ | $p$ | $4/5$ | $210$ | $18.5$ | $50$
Table 1:  Set of baryon resonances and their properties RUBW .

Employing the interaction Lagrangians for those resonances given in FP , the
in-medium self-energy of transverse and longitudinal $\rho$ mesons, to the
lowest order in density, read

$\displaystyle\Sigma_{\rho}^{T}(p_{0},\vec{p})$ $\displaystyle=$
$\displaystyle\sum_{\rm p-wave}SI\,\frac{f^{2}_{\rho
BN}}{m_{\rho}^{2}}\,F(\vec{p}^{2})\,\vec{p}^{2}\,\rho_{B}\,\frac{(E_{p}^{B}-m_{N})}{p_{0}^{2}-(E_{p}^{B}-m_{N})^{2}}$
$\displaystyle+$ $\displaystyle\sum_{\rm s-wave}SI\,\frac{f^{2}_{\rho
BN}}{m_{\rho}^{2}}\,F(\vec{p}^{2})\,p_{0}^{2}\,\rho_{B}\,\frac{(E_{p}^{B}-m_{N})}{p_{0}^{2}-(E_{p}^{B}-m_{N})^{2}}\,,$
$\displaystyle\Sigma_{\rho}^{L}(p_{0},\vec{p})$ $\displaystyle=$
$\displaystyle\sum_{\rm s-wave}SI\,\frac{f^{2}_{\rho
BN}}{m_{\rho}^{2}}\,F(\vec{p}^{2})\,p_{0}^{2}\,\rho_{B}\,\frac{(E_{p}^{B}-m_{N})}{p_{0}^{2}-(E_{p}^{B}-m_{N})^{2}}\,,$

where

$\displaystyle E_{p}^{B}(p_{0},\vec{p})$ $\displaystyle=$
$\displaystyle\sqrt{\vec{p}^{2}+m_{B}}-\frac{i}{2}\Gamma_{B}(p_{0},\vec{p})\,,$
$\displaystyle\Gamma_{B}(p_{0},\vec{p})$ $\displaystyle=$
$\displaystyle\Gamma_{B\to\rho N}^{0}(p_{0},\vec{p})+\Gamma_{B}^{\rm
med}\,\frac{\rho_{B}}{\rho_{0}}\,,$ (8)

and a form factor $F(\vec{p}^{2})=\Lambda^{2}/(\Lambda^{2}+\vec{p}^{2})$ with
$\Lambda=600$ MeV RUBW . The spin-isospin factors $SI$ and the parameters
$f_{\rho BN}$ for the resonances included are summarized in Table 1.

The $\Gamma_{B\to\rho N}^{0}$ in the above equation represents the full width
modified by the phase space for resonances. We will follow the prescription
given in Ref. FP  to introduce the energy and momentum dependence in the
static approximation for the nucleon;

$\Gamma_{B\to\rho N}^{0}(p_{0},\vec{p})=\Gamma_{B\to\rho
N}^{0}\,\left(\frac{p}{p_{B}}\right)^{3}\,,$ (9)

where

$\displaystyle p$ $\displaystyle=$
$\displaystyle\sqrt{\left[(\bar{s}-m_{N}^{2}-m_{\pi}^{2})^{2}-4m_{N}^{2}m_{\pi}^{2}\right]/4\bar{s}}\,,$
$\displaystyle p_{B}$ $\displaystyle=$
$\displaystyle\sqrt{\left[(m_{B}^{2}-m_{N}^{2}-m_{\pi}^{2})^{2}-4m_{N}^{2}m_{\pi}^{2}\right]/4m_{B}^{2}}\,,$
(10)

with $\bar{s}=(p_{0}+m_{N})^{2}-\vec{p}^{2}$. The final self-energy of $\rho$
meson reads $\Sigma_{\rho,{\rm
tot}}^{T,L}=-im_{\rho}\Gamma_{\rho}+\Sigma_{\rho}^{T,L}$ with the vacuum
$\rho$-meson width $\Gamma_{\rho}(s)$.

The $\omega$ meson in nuclear matter, similarly to the $\rho$ meson, gets
broader but with little mass shift, and its spectral function still exhibits a
distinct maximum Rapp ; RWvH ; Rapp:2011zz . Therefore, we shall assume for
simplicity that the in-medium width at chiral restoration is larger than its
vacuum value by a constant factor, $7$, reproducing the value $\sim 60$ MeV at
$\rho_{B}/\rho_{0}=2$ found in a coupled channel approach implementing $\omega
N$ resonances giessen .

Fig. 1 represents the spectral function of $\rho/\omega$ states at the chiral
crossover at $T=50$ MeV with selected three-momenta, $\vec{p}=0.1,0.5,1.0$
GeV.

Figure 1: The $\rho/\omega$ spectral function at chiral crossover
$\rho_{B}^{\rm crit}/\rho_{0}=2.5$ and $T=50$ MeV with finite three-momenta,
$\vec{p}=0.1,0.5,1.0$ GeV, from left to right. The curves without chiral
symmetry restoration (CSR) were computed such that the mass difference was
kept to be its vacuum value, i.e. $\delta m=0.49$ GeV at any $\rho_{B}$ and
$T$. The curves ”w/o B” represent the results without baryon resonances, taken
from our earlier study in Ref. Sasaki .

The three s-wave states in Table 1 carry negative parity, and they are
expected to decrease their masses toward the corresponding parity partners
near the chiral symmetry restoration (CSR), just as the lowest-lying baryons
do fastsum . We thus assume that the masses of the three resonances with
negative parity are reduced by a similar amount, 25%.

For the smallest momentum $\vec{p}=0.1$ GeV, the mixing effect has little
importance. The difference between the two lines is due to the s-wave baryonic
resonances whose masses are reduced when CSR is realized in dense matter. With
increasing $\vec{p}$, the p-wave states come into the spectral function. The
spectrum with $\vec{p}=1$ GeV is an admixture of shifted transverse and
unshifted longitudinal polarization states of vector and axial-vector mesons
via chiral mixing, on top of the in-medium broadening induced dominantly by
the p-wave resonances. When CSR is not imposed, the former effect, the chiral
mixing, is negligible so that the spectrum is broadened just due to the latter
effect. The importance of including baryon resonances is evident: the emergent
bumpy structure due to CSR found in our earlier study Sasaki  is smeared.

The production rate of a lepton pair emitted from dense matter via a virtual
photon is readily calculated. The differential rate at finite $T$ and
$\mu_{B}$ is given in terms of the imaginary part of the vector current
correlator RWvH  by

$\frac{dN}{d^{4}p}(p_{0},\vec{p};T,\mu_{B})=\frac{\alpha^{2}}{\pi^{3}s}\frac{\mbox{Im}G_{V}(p_{0},\vec{p};T,\mu_{B})}{e^{p_{0}/T}-1}\,,$
(11)

with $\alpha=e^{2}/4\pi$ the electromagnetic coupling constant. The three-
momentum integrated rate reads

$\frac{dN}{ds}(s;T,\mu_{B})=\int\frac{d^{3}\vec{p}}{2p_{0}}\frac{dN}{d^{4}p}(p_{0},\vec{p};T,\mu_{B})\,.$
(12)

In Fig. 2, the integrated rate in the range of $0\leq|\vec{p}|\leq 2$ GeV at
chiral crossover with $T=50$ MeV is presented.

Figure 2: The dilepton production rates at chiral crossover $\rho_{B}^{\rm
crit}/\rho_{0}=2.5$ and $T=50$ MeV, integrated over three-momenta in the rage
of $0\leq|\vec{p}|\leq 2$ GeV. The curve labeled ”CSR-0” was calculated with
the mass of vector meson being the same value as in matter-free space, whereas
the curve labeled ”CSR-1” was obtained by implementing a reduction of vector
meson mass to $550$ MeV as suggested in Ref. Kim:2021xyp

The result obtained from the vector spectra shown in Fig. 1, labeled ”CSR-0”,
exhibits a substantial broadening effect mainly due to the p-wave baryon
resonances, and consequently it does not differ much from the baseline without
CSR. We have assumed so far that the vector meson mass at chiral crossover
remains its vacuum value, which can be relaxed. In fact, the recent study
using QCD sum rules applied to the vacuum with unbroken chiral symmetry
suggests that the degenerate parity partners of vector states carry the mass
of $500$-$600$ MeV Kim:2021xyp . Introducing such a reduction into the vector
spectra results in a much larger contribution in the production rate below
$\sqrt{s}=0.7$ GeV, labeled ”CSR-1”. In practice, the signatures of CRS in the
rates are to a large extent diminished by the p-wave baryon resonances and it
would be hard to measure them in heavy-ion experiments, unless the masses of
vector states would decrease near the chiral symmetry restoration by $20$-$30$
%.

###  III.2 $\phi$ meson

The $\phi$ mesons predominantly decay into $K\bar{K}$ pairs, and the dressing
of kaon cloud mainly modifies the $\phi$ meson in nuclear matter. The $SU(3)$
chiral Lagrangian in the mean field approximation gives in-medium masses of
kaon and anti-kaon as kaon

$\displaystyle m_{K}^{\ast}$ $\displaystyle=$
$\displaystyle\left[m_{K}^{2}-a_{K}\rho_{S}+\left(b_{K}\rho_{B}\right)^{2}\right]^{1/2}+b_{K}\rho_{B}\,,$
$\displaystyle m_{\bar{K}}^{\ast}$ $\displaystyle=$
$\displaystyle\left[m_{K}^{2}-a_{\bar{K}}\rho_{S}+\left(b_{K}\rho_{B}\right)^{2}\right]^{1/2}-b_{K}\rho_{B}\,,$
(13)

where $\rho_{S}$ represents the nuclear scalar density and the three
parameters are given by $a_{K}=a_{\bar{K}}=\Sigma_{KN}/f_{\pi}^{2}$ with the
kaon-nucleon sigma term $\Sigma_{KN}$ and $b_{K}=3/(8f_{\pi}^{2})$,
respectively. Given the fact that there exist large uncertainties in
$\Sigma_{KN}$ and difficulties in dealing with higher-order corrections
systematically, the parameters $a_{K}$ and $a_{\bar{K}}$ have been determined
from the kaon production data in heavy-ion collisions and found $a_{K}=0.22$
GeV2fm3 and $a_{\bar{K}}=0.45$ GeV2fm3, respectively Li:1997tz . One finds the
in-medium kaon masses to be $m_{K}^{\ast}=560$ MeV and
$m_{\bar{K}}^{\ast}=230$ MeV at the chiral crossover, $\rho_{B}^{\rm
crit}/\rho_{0}=2.5$.

Accordingly, the decay width of $\phi$ meson is modified as well. We shall
adopt the form Chung:1998ev ,

$\displaystyle\Gamma_{\phi}(s)$ $\displaystyle=$
$\displaystyle\frac{g^{2}_{\phi K\bar{K}}}{3\pi}\frac{k(s)^{3}}{s}\,,$
$\displaystyle k(s)$ $\displaystyle=$
$\displaystyle\frac{1}{2\sqrt{s}}\left[\left(s-(m_{K}^{\ast}+m_{\bar{K}}^{\ast})^{2}\right)\left(s-(m_{K}^{\ast}-m_{\bar{K}}^{\ast})^{2}\right)\right]^{1/2}\,,$

with the coupling constant $g_{\phi K\bar{K}}^{2}/4\pi=1.69$. Assuming that
the $\phi$ meson mass is little shifted in nuclear matter as studied in the
$SU(3)$ coupled channel approach Oset:2000eg , the kaon and anti-kaon obeying
the above relations (13) lead to the effective width
$\Gamma_{\phi}(s=m_{\phi}^{2})=61$ MeV.

Given the modified width of $\phi$ meson, the spectral function can be readily
computed at chiral crossover, for a fixed three-momentum $\vec{p}$, as
displayed in Fig. 3.

Figure 3: The $\phi$ spectral function at chiral crossover $\rho_{B}^{\rm
crit}/\rho_{0}=2.5$ and $T=50$ MeV with finite three-momenta,
$\vec{p}=0.1,0.5,1.0$ GeV, from left to right.

Recall that the chiral mixing effect (2) is always accompanied by finite
$\vec{p}$. The signatures of CSR thus become stronger for larger $\vec{p}$,
whose contributions are more Boltzmann suppressed in dilepton rates though.
Therefore, a careful binning of dilepton data in three-momenta, say
$|\vec{p}|>0.5$ GeV, would be crucial to extract the signals of restored
chiral symmetry.

The corresponding production rates of lepton pairs integrated over $\vec{p}$
above $0.5$ GeV up to $2$ GeV are shown in Fig. 4.

Figure 4: The dilepton production rates at chiral crossover $\rho_{B}^{\rm
crit}/\rho_{0}=2.5$ and $T=50$ MeV, integrated over three-momenta in the rage
of $0.5\leq|\vec{p}|\leq 2$ GeV. The $\phi$ meson mass was assumed to be the
same as in vacuum (left), reduced by $5$% (middle) and $10$% (right).

The clear peak due to the chiral mixing seen at $\sqrt{s}\simeq 1.1$ GeV
Sasaki  where no modifications of kaons were included is now smeared because
of in-medium broadening. When the mass of $\phi$ meson decreases toward CSR,
even by a few percent, the mixing effect from the $\phi$ and its counterpart
becomes enhanced and generates a sizable difference from the result without
CSR.

So far, our calculations were performed with the mixing strength estimated in
the mean field approximation that also yields a specific density of chiral
crossover, $2.5\,\rho_{0}$ Sasaki . Since the chiral mixing is linearly
proportional to the net baryon density $\rho_{B}$, one can accommodate
potential options, e.g. a stronger mixing and/or higher critical density of
chiral symmetry restoration, into our computation by tuning the mixing
strength. In Fig. 5, we show the dilepton rates at $T=50$ MeV for several
values of the mixing parameter in dense matter.

Figure 5: The same as in Fig. 4 (left), but for various strengths of chiral
mixing.

The curve labeled ”$1.2\,\times$ mixing” represents the result at the same
chiral crossover $\rho_{B}^{\rm crit}=2.5\,\rho_{0}$ with a stronger mixing by
$20$%, or alternatively at a higher crossover density $3\,\rho_{0}$ with the
same mixing strength. Similarly, the curve labeled ”$1.4\,\times$ mixing”
depicts two scenarios, i.e. a stronger mixing by $40$% alternating with a
higher critical density $3.5\,\rho_{0}$. The signatures of CSR become stronger
with larger baryon density leading to either stronger mixing or higher
critical density, or even both. To better quantify the in-medium effects and
their consequences on dilepton production, a more refined approach beyond the
mean field approximation would be required.

##  IV Conclusions

We have carried out the computation of in-medium spectral functions for the
light vector mesons in the presence of novel chiral mixing induced by density,
by implementing relevant baryon resonances and modifications of kaons and
anti-kaons which yield significant broadening of those spectra in dense
nuclear matter. The emergence of chiral symmetry restoration (CSR) can be
found maximally via the chiral mixing Sasaki , and this property stays with
the baryon resonances and modified kaons.

The $\rho/\omega$ spectra at chiral crossover exhibit distinct features from
those without CSR in a wide range of three momenta $\vec{p}$; For small
$\vec{p}$, the mixing effect is negligible but the s-wave resonances, which
carry negative parity, change their masses because of CSR and generate a clear
difference from the scenario without CSR. For large $\vec{p}$, the p-wave
states play the major role to broaden the spectra, whereas the chiral mixing
effect becomes enlarged as well. The $\phi$ spectra at chiral crossover are
not altered much compared with the $\rho/\omega$, but those carrying higher
$\vec{p}$ show stronger signatures of CSR as naturally expected from the
modified dispersion relations due to the chiral mixing.

The observable quantities, such as dilepton production rates, are rather
insensitive to CSR since they are integrated over $\vec{p}$. This is
particularly the case in the $\rho/\omega$ sector where the signatures of CSR
are to a large extent diminished, unless the vector meson masses would
decrease by a few hundred MeV as suggested in the recent analysis based on QCD
sum rules Kim:2021xyp . The rates from $\phi$ mesons also show a strong
sensitivity to the $\phi$ meson mass at chiral crossover, even with a slight
decrease by a few percent leading to a sizable difference from the ”no CSR”
scenario around the invariance mass of $1$ GeV.

A higher crossover density and/or stronger chiral mixing are regarded as
theoretical options in dense nuclear matter. If either one would be realized,
the signals of CSR both in the vector spectra and dilepton rates become
stronger, although the quantitative estimate will rely on a precise value of
the chiral mixing that requires a more refined prescription beyond the
approximations employed in this study.

Chiral spin symmetry has been recently shown to emerge in a temperature range
of $T/T_{\rm ch}=2-3$ with chiral crossover temperature $T_{\rm ch}$
Glozman:2022lda , in which the new symmetry manifests itself in the nucleon
parity doublet. Its phenomenological consequences via parity doubling will be
explored in the domain of dense QCD.

### Acknowledgments

The author acknowledges stimulating discussions with T. Galatyuk and K.
Redlich. This work has been partly supported by the Polish Science Foundation
(NCN) under OPUS Grant No. 2018/31/B/ST2/01663.

## References

  * (1) R. Rapp, Phys. Rev. C 63, 054907 (2001).
  * (2) R. S. Hayano and T. Hatsuda, Rev. Mod. Phys. 82, 2949 (2010).
  * (3) R. Rapp, J. Wambach and H. van Hees, Landolt-Bornstein 23, 134 (2010).
  * (4) R. Rapp, B. Kampfer, A. Andronic, D. Blaschke, C. Fuchs, M. Harada, T. Hilger, M. Kitazawa, T. Kunihiro and P. Petreczky, et al. Lect. Notes Phys. 814, 335-529 (2011).
  * (5) K. Fukushima and C. Sasaki, Prog. Part. Nucl. Phys. 72, 99 (2013).
  * (6) R. Arnaldi et al. [NA60 Collaboration], Phys. Rev. Lett. 96, 162302 (2006).
  * (7) H. van Hees and R. Rapp, Phys. Rev. Lett. 97, 102301 (2006).
  * (8) M. Dey, V. L. Eletsky and B. L. Ioffe, Phys. Lett. B 252, 620 (1990).
  * (9) B. Krippa, Phys. Lett. B 427, 13 (1998).
  * (10) G. Chanfray, J. Delorme and M. Ericson, Nucl. Phys. A 637, 421 (1998).
  * (11) P. M. Hohler and R. Rapp, Phys. Lett. B 731, 103 (2014).
  * (12) S. Weinberg, Phys. Rev. Lett. 18, 507 (1967).
  * (13) M. Harada, C. Sasaki and W. Weise, Phys. Rev. D 78, 114003 (2008).
  * (14) M. Urban, M. Buballa and J. Wambach, Phys. Rev. Lett. 88, 042002 (2002).
  * (15) S. K. Domokos and J. A. Harvey, Phys. Rev. Lett. 99, 141602 (2007).
  * (16) M. Harada and C. Sasaki, Phys. Rev. C 80, 054912 (2009).
  * (17) C. Sasaki, Phys. Lett. B 801, 135172 (2020).
  * (18) D. B. Kaplan and A. E. Nelson, Phys. Lett. B 175, 57-63 (1986).
  * (19) N. Kaiser and U. G. Meissner, Nucl. Phys. A 519, 671 (1990).
  * (20) M. Bando, T. Kugo and K. Yamawaki, Phys. Rept. 164, 217 (1988).
  * (21) M. Harada and C. Sasaki, Phys. Rev. D 73, 036001 (2006).
  * (22) M. Marczenko, D. Blaschke, K. Redlich and C. Sasaki, Universe 5, no. 8, 180 (2019).
  * (23) B. Friman and H. J. Pirner, Nucl. Phys. A 617, 496-509 (1997).
  * (24) R. Rapp, M. Urban, M. Buballa and J. Wambach, Phys. Lett. B 417, 1-6 (1998).
  * (25) P. Muehlich, V. Shklyar, S. Leupold, U. Mosel and M. Post, Nucl. Phys. A 780, 187-205 (2006).
  * (26) G. Aarts, C. Allton, S. Hands, B. Jäger, C. Praki and J. I. Skullerud, Phys. Rev. D 92, no.1, 014503 (2015); G. Aarts, C. Allton, D. De Boni, S. Hands, B. Jäger, C. Praki and J. I. Skullerud, JHEP 06, 034 (2017); G. Aarts, C. Allton, D. De Boni and B. Jäger, Phys. Rev. D 99, no.7, 074503 (2019).
  * (27) J. Kim and S. H. Lee, Phys. Rev. D 105, no.1, 014014 (2022).
  * (28) G. Q. Li, C. H. Lee and G. E. Brown, Phys. Rev. Lett. 79, 5214-5217 (1997).
  * (29) W. S. Chung, C. M. Ko and G. Q. Li, Nucl. Phys. A 641, 357-378 (1998).
  * (30) E. Oset and A. Ramos, Nucl. Phys. A 679, 616-628 (2001).
  * (31) L. Y. Glozman, O. Philipsen and R. D. Pisarski, [arXiv:2204.05083 [hep-ph]].


# Knowledge Transfer with Jacobian Matching

Suraj Srinivas  François Fleuret

###### Abstract

Classical distillation methods transfer representations from a “teacher”
neural network to a “student” network by matching their output activations.
Recent methods also match the Jacobians, or the gradient of output activations
with the input. However, this involves making some ad hoc decisions, in
particular, the choice of the loss function.

In this paper, we first establish an equivalence between Jacobian matching and
distillation with input noise, from which we derive appropriate loss functions
for Jacobian matching. We then rely on this analysis to apply Jacobian
matching to transfer learning by establishing equivalence of a recent transfer
learning procedure to distillation.

We then show experimentally on standard image datasets that Jacobian-based
penalties improve distillation, robustness to noisy inputs, and transfer
learning.

Machine Learning, ICML

###  1 Introduction

Consider that we are given a neural network $\mathcal{A}$ trained on a
particular dataset, and want to train another neural network $\mathcal{B}$ on
a similar (or related) dataset. Is it possible to leverage $\mathcal{A}$ to
train $\mathcal{B}$ more efficiently? We call this the problem of knowledge
transfer. Distillation (Hinton et al., 2015) is a form of knowledge transfer
where $\mathcal{A}$ and $\mathcal{B}$ are trained on the same dataset, but
have different architectures. Transfer Learning (Pan & Yang, 2010) is another
form of knowledge transfer where $\mathcal{A}$ and $\mathcal{B}$ are trained
on different (but related) datasets. If the architectures are the same, we can
in both cases simply copy the weights from $\mathcal{A}$ to $\mathcal{B}$. The
problem becomes more challenging when $\mathcal{A}$ and $\mathcal{B}$ have
different architectures.

A perfect distillation method would enable us to transform one neural network
architecture into another, while preserving generalization. This would allow
us to easily explore the space of neural network architectures, which can be
used for neural network architecture search, model compression, or creating
diverse ensembles. A perfect transfer learning method, on the other hand,
would use little data to train $\mathcal{B}$, optimally using the limited
samples at it’s disposal.

This paper deals with the problem of knowledge transfer based on matching the
Jacobian of the network’s output which, like the output itself, has a
dimension independent of the network’s architecture. This approach has also
been recently explored for the case of distillation by Czarnecki et al.
(2017), who considered the general idea of matching Jacobians, and by
Zagoruyko & Komodakis (2017) who view Jacobians as attention maps. However, in
both of these cases, it was not clear what loss function must be used to match
Jacobians. It was also unclear how these methods relate to classical
distillation approaches (Ba & Caruana, 2014; Hinton et al., 2015).

Recently Li & Hoiem (2016) proposed a distillation-like approach to perform
transfer learning. While this approach works well in practice, it was not
clear how this exactly relates to regular distillation. It is also not clear
how this applies to the challenging setting of transfer learning where the
architectures of both networks $\mathcal{A}$ and $\mathcal{B}$ can be
arbitrary.

The overall contributions of our paper are:

  1. 1.

We show that matching Jacobians is a special case of classical distillation,
where noise is added to the inputs.

  2. 2.

We show that a recent transfer learning method (LwF by Li & Hoiem, 2016) can
be viewed as distillation, which allows us to match Jacobians for this case.

  3. 3.

We provide methods to match Jacobians of practical deep networks, where
architecture of both networks are arbitrary.

We experimentally validate these results by providing evidence that Jacobian
matching helps both regular distillation and transfer learning, and that
Jacobian-norm penalties learn models robust to noise.

###  2 Related Work

Several Jacobian-based regularizers have been proposed in recent times.
Sobolev training (Czarnecki et al., 2017), showed that using higher order
derivatives along with the targets can help in training with less data. This
work is similar to ours. While we also make similar claims, we clarify the
relationship of this method with regular distillation based on matching
activations, and show how it can help. Specifically, we show how the loss
function used for activation matching also decides the loss function we use
for Jacobian matching. Similarly, Wang et al. (2016) use the Jacobian for
distillation and show that it helps improve performance. Zagoruyko & Komodakis
(2017) introduce the idea of matching attention maps. The Jacobian was also
considered as one such attention map. This work also finds that combining both
activation matching and Jacobian matching is helpful.

Jacobian-norm regularizers were used in early works by Drucker & Le Cun
(1992), where they looked at penalizing the Jacobian norm. The intuition was
to make the model more robust to small changes in the input. We find that this
conforms to our analysis as well.

Knowledge Distillation (Hinton et al., 2015) first showed that one can use
softmax with temperature to perform knowledge transfer with neural nets. Ba &
Caruana (2014) found that squared error between logits worked better than the
softmax method, and they used this method to train shallow nets with
equivalent performance to deep nets. Romero et al. (2014) and Zagoruyko &
Komodakis (2017) showed how to enhance distillation by matching intermediate
features along with the outputs, but use different methods to do so. Sau &
Balasubramanian (2016) found that adding noise to logits helps during teacher-
student training. We show that the use of the Jacobian can be interpreted as
adding such noise to the inputs analytically.

###  3 Jacobians of Neural Networks

Let us consider the first order Taylor series expansion of a function
$f:\mathbb{R}^{D}\rightarrow\mathbb{R}$ around a small neighborhood
$\\{\mathbf{x}+\Delta\mathbf{x}:\|\Delta\mathbf{x}\|\leq\epsilon\\}$. It can
be written as

$\displaystyle f(\mathbf{x}+\Delta\mathbf{x})$
$\displaystyle=f(\mathbf{x})+\nabla_{x}f(\mathbf{x})^{T}(\Delta\mathbf{x})+\mathcal{O}(\epsilon^{2})$
(1)

We can apply this linearization to neural nets. The source of non-linearity
for neural nets lie in the elementwise non-linear activations (like ReLU,
sigmoid) and pooling operators. It is easy to see that to linearize the entire
neural network, one must linearize all such non-linearities in the network.

####  3.1 Special case: ReLU and MaxPool

For the ReLU nonlinearity, the Taylor approximation is locally exact and
simple to compute, as the derivative $\frac{\mathrm{d}\sigma(z)}{\mathrm{d}z}$
is either 0 or 1 (except at $z=0$, where it is undefined). A similar statement
holds for max-pooling. Going back to the definition in Equation 1, for
piecewise linear nets there exist $\epsilon>0$ such that the super-linear
terms are exactly zero, i.e.;
$f(\mathbf{x}+\Delta\mathbf{x})=f(\mathbf{x})+\nabla_{x}f(\mathbf{x})^{T}(\Delta\mathbf{x})$,
i.e.; $\mathbf{x}$ and $\Delta\mathbf{x}$ lie on the same linear surface.

####  3.2 Invariance to weight and architecture specification

One useful property of the Jacobian is that its size does not depend on the
network architecture. For $k$ output classes, and input dimension $D$ , the
Jacobian of a neural network is of dimension $D\times k$. This means that one
can compare Jacobians of different architectures.

Another useful property is that for a given neural network architecture,
different weight configurations can lead to the same Jacobian. One simple
example of this is permutation symmetry of neurons in intermediate hidden
layers. It is easy to see different permutations of neurons leave the Jacobian
unchanged (as they have the same underlying function). In general, because of
redundancy of neural network models and non-convexity of the loss surface,
several different weight configurations can end up having similar Jacobians.

Given these desirable properties, we consider using the Jacobian to perform
knowledge transfer between neural networks of different architectures. Note
that these properties hold trivially for output activations also. Thus it
seems sensible that both these quantities must be used for knowledge transfer.
However, the important practical question remains: how exactly should this be
done?

###  4 Distillation

We consider the problem of improving distillation using Jacobians. This
problem of distillation may be posed as follows: given a teacher network
$\mathcal{T}$ which is trained on a dataset $\mathcal{D}$, we wish to enhance
the training of a student network $\mathcal{S}$ on $\mathcal{D}$ using “hints”
from $\mathcal{T}$. Classically, such “hints” involve activations of the
output layer or some intermediate layers. Recent works (Czarnecki et al.,
2017; Zagoruyko & Komodakis, 2017) sought to match the Jacobians of
$\mathcal{S}$ and $\mathcal{T}$. However, two aspects are not clear in these
formalisms: (i) what penalty term must be used between Jacobians, and (ii) how
this idea of matching Jacobians relates to simpler methods such as activation
matching (Ba & Caruana, 2014; Hinton et al., 2015). To resolve these issues,
we make the following claim.

######  Claim.

Matching Jacobians of two networks is equivalent to matching soft targets with
noise added to the inputs during training.

Figure 1: Illustration of teacher-student learning in a simple 1D case. Here,
x-axis is the input data, and y-axis denotes function outputs. Given a limited
number of data points, there exist multiple student functions consistent with
the data. How do we select the hypothesis closest to the teacher’s? There are
two equivalent solutions: either by augmenting the data set by adding noise to
the inputs or by directly matching slopes (Jacobians) of the function at the
data points.

More concretely, we make the following proposition.

######  Proposition 1.

Consider the squared error cost function for matching soft targets of two
neural networks with $k$-length targets ($\in\mathbb{R}^{k}$), given by
$\ell(\mathcal{T}(\mathbf{x}),\mathcal{S}(\mathbf{x}))=\sum_{i=1}^{k}(\mathcal{T}^{i}(\mathbf{x})-\mathcal{S}^{i}(\mathbf{x}))^{2}$,
where $\mathbf{x}\in\mathbb{R}^{D}$ is an input data point. Let
$\bm{\xi}~{}(\in\mathbb{R}^{D})=\sigma~{}\mathbf{z}$ be a scaled version of a
unit normal random variable $\mathbf{z}~{}\in\mathbb{R}^{D}$ with scaling
factor $\sigma\in\mathbb{R}$. Then the following is true.

$\displaystyle\mathbb{E}_{\bm{\xi}}\left[\sum_{i=1}^{k}\left(\mathcal{T}^{i}(\mathbf{x}+\bm{\xi})-\mathcal{S}^{i}(\mathbf{x}+\bm{\xi})\right)^{2}\right]$
$\displaystyle=\sum_{i=1}^{k}\left(\mathcal{T}^{i}(\mathbf{x})-\mathcal{S}^{i}(\mathbf{x})\right)^{2}+\sigma^{2}\sum_{i=1}^{k}\|\nabla_{x}\mathcal{T}^{i}(\mathbf{x})-\nabla_{x}\mathcal{S}^{i}(\mathbf{x})\|_{2}^{2}$
$\displaystyle+\mathcal{O}(\sigma^{4})$

Notice that in this expression, we have decomposed the loss function into two
components: one representing the usual distillation loss on the samples, and
the second regularizer term representing the Jacobian matching loss. The
higher order error terms are small for small $\sigma$ and can be ignored. The
above proposition is a simple consequence of using the first-order Taylor
series expansion around $x$. Note that the error term is exactly zero for
piecewise-linear nets. An analogous statement is true for the case of cross
entropy error between soft targets, leading to:

$\displaystyle\mathbb{E}_{\bm{\xi}}\left[-\sum_{i=1}^{k}\mathcal{T}^{i}_{s}(\mathbf{x}+\bm{\xi})\log\left(\mathcal{S}^{i}_{s}(\mathbf{x}+\bm{\xi})\right)\right]$
(2)
$\displaystyle\approx-\sum_{i=1}^{k}\mathcal{T}^{i}_{s}(\mathbf{x})\log(\mathcal{S}^{i}_{s}(\mathbf{x}))~{}-~{}\sigma^{2}\sum_{i=1}^{k}\frac{\nabla_{x}\mathcal{T}^{i}_{s}(\mathbf{x})^{T}\nabla_{x}\mathcal{S}^{i}_{s}(\mathbf{x})}{\mathcal{S}^{i}_{s}(\mathbf{x})}$

where $\mathcal{T}^{i}_{s}(\mathbf{x})$ denotes the same network
$\mathcal{T}^{i}(\mathbf{x})$ but with a softmax or sigmoid (with temperature
parameter $T$ if needed) added at the end. We do not write the super-linear
error terms for convenience. This shows that the Jacobian matching loss
depends crucially on the loss used to match activations. This observation can
be used in practice to pick appropriate loss function by choosing a specific
noise model of interest.

To summarize, these statements show that matching Jacobians is a natural
consequence of matching not only the raw CNN outputs at the given data points,
but also at the infinitely many data points nearby. This is illustrated in
Figure 1, which shows that by matching on a noise-augmented dataset, the
student is able to mimic the teacher better.

We can use a similar idea to derive regularizers for the case of regular
neural network training as well. These regularizers seek to make the
underlying model robust to small amounts of noise added to the inputs.

######  Proposition 2.

Consider the squared error cost function for training a neural network with
$k$ targets, given by
$\ell(y(\mathbf{x}),\mathcal{S}(\mathbf{x}))=\sum_{i=1}^{k}(y^{i}(\mathbf{x})-\mathcal{S}^{i}(\mathbf{x}))^{2}$,
where $\mathbf{x}\in\mathbb{R}^{D}$ is an input data point, and
$y^{i}(\mathbf{x})$ is the $i^{th}$ target output. Let
$\bm{\xi}~{}(\in\mathbb{R}^{D})=\sigma~{}\mathbf{z}$ be a scaled version of a
unit normal random variable $\mathbf{z}~{}\in\mathbb{R}^{D}$ with scaling
factor $\sigma\in\mathbb{R}$. Then the following is true.

$\displaystyle\mathbb{E}_{\bm{\xi}}\left[\sum_{i=1}^{k}\left(y^{i}(\mathbf{x})-\mathcal{S}^{i}(\mathbf{x}+\bm{\xi})\right)^{2}\right]$
$\displaystyle=\sum_{i=1}^{k}\left(y^{i}(\mathbf{x})-\mathcal{S}^{i}(\mathbf{x})\right)^{2}+\sigma^{2}\sum_{i=1}^{k}\|\nabla_{x}\mathcal{S}^{i}(\mathbf{x})\|_{2}^{2}+\mathcal{O}(\sigma^{4})$

A statement similar to Proposition 2 has been previously derived by Bishop
(1995), who observed that the regularizer term for linear models corresponds
exactly to the well-known Tikhonov regularizer. This regularizer was also
proposed by Drucker & Le Cun (1992). The $\ell_{2}$ weight decay regularizer
for neural networks can be derived by applying this regularizer layer-wise
separately. However, we see here that a more appropriate way to ensure noise
robustness is to penalize the norm of the Jacobian rather than weights. We can
derive a similar result for the case of cross-entropy error as well, which is
given by -

$\displaystyle\mathbb{E}_{\bm{\xi}}\left[-\sum_{i=1}^{k}y^{i}(\mathbf{x})\log(\mathcal{S}^{i}_{s}(\mathbf{x}+\bm{\xi}))\right]$
(3)
$\displaystyle\approx-\sum_{i=1}^{k}y^{i}(\mathbf{x})\log(\mathcal{S}^{i}_{s}(\mathbf{x}))~{}+~{}\sigma^{2}\sum_{i=1}^{k}y^{i}(\mathbf{x})\frac{\|\nabla_{x}\mathcal{S}^{i}_{s}(\mathbf{x})\|_{2}^{2}}{\mathcal{S}^{i}_{s}(\mathbf{x})^{2}}$

We notice here again that the regularizer involves
$\mathcal{S}^{i}_{s}(\mathbf{x})$, which has the sigmoid / softmax
nonlinearity applied on top of the final layer of
$\mathcal{S}^{i}(\mathbf{x})$. Deriving all the above results is a simple
matter of using first-order Taylor series expansions, and a second-order
expansion in the case of Equation 3. Proof is provided in the supplementary
material.

In all cases above we see that the regularizers for cross-entropy error term
seem more unstable when compared to those for squared error. We find that this
is true experimentally as well. As a result, we use squared error loss for
distillation.

####  4.1 Approximating the Full Jacobian

One can see that both in the case of Proposition 1 and 2, we are required to
compute the full Jacobian. This is computationally expensive, and sometimes
unnecessary. For example, Equation 3 requires only the terms where
$y^{i}(\mathbf{x})$ is non-zero.

In general, we can approximate the summation of Jacobian terms with the one
with largest magnitude. However, we cannot estimate this without computing the
Jacobians themselves. As a result, we use a heuristic where the only output
variable involving the correct answer $c\in[1,k]$ is used for computing the
Jacobian. This corresponds to the case of Equation 3. Alternately, if we do
not want to use the labels, we may instead use the output variable with the
largest magnitude, as it often corresponds to the right label (for good
models).

###  5 Transfer Learning

We now use our Jacobian matching machinery to transfer learning problems. In
computer vision, transfer learning is often done by fine-tuning (Yosinski et
al., 2014), where models pre-trained on a large dataset $\mathcal{D}_{l}$,
such as Imagenet (Russakovsky et al., 2015), are used as initialization for
training on another smaller dataset $\mathcal{D}_{s}$. We can also refer to
these as the source dataset and the target dataset respectively. Practically,
this means that the architecture used for fine-tuning must be the same as that
of the pre-trained network, which is restrictive. We would like to develop
transfer learning methods where the architectures of the pre-trained network
and target “fine-tuned” network can be arbitrarily different.

Figure 2: Illustration of our proposed method for transfer learning. We match
the output activations of a pre-trained Imagenet network similar to LwF (Li &
Hoiem, 2016). We also match aggregated activations or “attention” maps between
networks, similar to the work of Zagoruyko & Komodakis (2017). We propose to
match Jacobians of (aggregated) attention maps w.r.t. inputs.

One way to achieve this is by distillation: we match output activations of a
pre-trained teacher network and an untrained student network. However, this
procedure is not general as the target dataset may not share the same label
space as the large source dataset. This leads to size mismatch between output
activations of the teacher and the student. To overcome this, we can design
the student network to have two sets of outputs (or two output “branches”),
one with the label space of the smaller target dataset, while the other with
that of the larger source dataset. This leads to the method proposed by Li &
Hoiem (2016), called “Learning without Forgetting” (LwF). Note that similar
methods were concurrently developed by Jung et al. (2016) and Furlanello et
al. (2016). In this method, the student network is trained with a composite
loss function involving two terms, one in each output branch. The two
objectives are (1) matching ground truth labels on the target dataset, and (2)
matching the activations of the student network and a pre-trained teacher
network on the target dataset. This is illustrated in Figure 2. Crucially,
these losses are matched only on the target dataset, and the source data is
untouched. This is conceptually different from distillation, where the teacher
network is trained on the dataset being used. In LwF, the pre-trained teacher
is not trained on the target dataset.

This makes it problematic to apply our Jacobian matching framework to LwF. For
distillation, it is clear that adding input noise (or Jacobian matching) can
improve overall matching as shown in Figure 1. For the case of LwF, it is not
clear whether improving matching between teacher and student will necessarily
improve transfer learning performance. This is especially because the teacher
is not trained on the target dataset, and can potentially produce noisy or
incorrect results on this data. To resolve this ambiguity, we shall now
connect LwF with distillation.

####  5.1 LwF as Distillation

In the discussion below we shall only consider the distillation-like loss of
LwF, and ignore the branch which matches ground truth labels. For LwF to work
well, it must be the case that the activations of the pre-trained teacher
network on the target dataset must contain information about the source
dataset (i.e.; Imagenet). The attractiveness of LwF lies in the fact that this
is done without explicitly using Imagenet. Here, we make the claim that LwF
approximates distillation on (a part of) Imagenet.

Let $f(\cdot)$ be an untrained neural network, $g(\cdot)$ be a pre-trained
network, $\mathbf{x},\mathbf{y}$ be the input image and corresponding ground
truth label respectively. Let $|\mathcal{D}|$ be the size of the dataset
$\mathcal{D}$. Let us denote
$\rho(\mathbf{x})=\ell(f(\mathbf{x}),g(\mathbf{x}))$ for convenience. Assume
Lipschitz continuity for $\rho(\mathbf{x})$ with Lipschitz constant
$\mathrm{K}$, and distance metric $\psi_{\mathbf{x}}$ in the input space

$\displaystyle\|\rho(\mathbf{x}_{1})-\rho(\mathbf{x}_{2})\|\leq\mathrm{K}\psi_{\mathbf{x}}(\mathbf{x}_{1},\mathbf{x}_{2})$
(4)

Note here that the distance in the input space need not be in terms of
pixelwise distances, but can also be neural net feature distance, for example.
Let us also define an assymetric version of the Hausdorff distance between two
sets $A,B$:

$\displaystyle\mathcal{H}_{a}(A,B)=\sup_{a\in A}\inf_{b\in
B}\psi_{\mathbf{x}}(a,b).$ (5)

Note that this is no longer a valid distance metric unlike the Hausdorff.
Given these assumptions, we are now ready to state our result.

######  Proposition 3.

Given the assumptions and notations described above, we have

$\displaystyle\frac{1}{|\mathcal{D}_{l}|}\sum_{\mathbf{x}\sim\mathcal{D}_{l}}\ell(f(\mathbf{x}),g(\mathbf{x}))$
$\displaystyle\leq\max_{\mathbf{x}\sim\mathcal{D}_{s}}\ell(f(\mathbf{x}),g(\mathbf{x}))$
(6) $\displaystyle+\mathrm{K}\mathcal{H}_{a}(\mathcal{D}_{l},\mathcal{D}_{s})$
(7)

On the left side of 6 we have the distillation loss on the source dataset, and
on the right we have a max-loss term on the target dataset. Note that the LwF
loss is an average loss on the target dataset. As expected, the slack terms in
the inequality depends on the distance between the source and target datasets
(7). This bounds a loss related to the LwF loss (i.e. max-loss instead of
average) with the distillation loss. If the Hausdorff distance is small, then
reducing the max-loss would reduce the distillation loss as well. A similar
theory was previously presented by Ben-David et al. (2010), but with different
formalisms. Our formalism allows us to connect with Jacobian matching, which
is our primary objective.

In practice, the target dataset is often much smaller than the Imagenet and
has different overall statistics. For example, the target dataset could be a
restricted dataset of flower images. In such a case, we can restrict the
source dataset to its “best” subset, in particular with all the irrelevant
samples (those far from target dataset) removed. This would make the Hausdorff
distance smaller, and provide a tighter bound. In our example, this involves
keeping only flowers from Imagenet.

This makes intuitive sense as well: if the source and target datasets are
completely different, we do not expect transfer learning (and thus LwF) to
help. The greater the overlap between source and target datasets, the smaller
is the Hausdorff distance, the tighter is the bound, and the more we expect
knowledge transfer to help. Our results capture this intuition in a rigorous
manner. In addition, this predicts that Lipschitz-smooth teacher neural nets
that produce small feature distance between source and target images are
expected to do well in transfer learning. Lipschitz-smoothness of models has
been previously related to adversarial noise robustness (Cisse et al., 2017),
and to learning theory as a sufficient condition for generalization (Xu &
Mannor, 2012). It is interesting that this relates to transfer learning as
well.

More importantly, this establishes LwF as a distillation method. The following
result motivates input noise added to the target dataset.

######  Corollary.

For any superset $\mathrm{D^{\prime}}_{s}\supseteq\mathcal{D}_{s}$ of the
target dataset,
$\mathcal{H}_{a}(\mathcal{D}_{l},\mathcal{D^{\prime}}_{s})\leq\mathcal{H}_{a}(\mathcal{D}_{l},\mathcal{D}_{s})$

Thus if we augment the target dataset $\mathcal{D}_{s}$ by adding noise, we
expect the bound in Proposition 3 to get tighter. This is because when we add
noise to points in $\mathcal{D}_{s}$, the minimum distance between points from
$\mathcal{D}_{l}$ to $\mathcal{D}_{s}$ decreases. Proofs are provided in the
supplementary material.

To summarize, we have showed that a loss related to the LwF loss (max-loss) is
an upper bound on the true distillation loss. Thus by minimizing the upper
bound, we can expect to reduce the distillation loss also.

#####  5.1.1 Incorporating Jacobian matching

Now that input noise and thus Jacobian matching is well motivated, we can
incorporate these losses into LwF. When we implemented this for practical deep
networks we found that the optimizer wasn’t able to reduce the Jacobian loss
at all. We conjecture that it might be because of a vanishing gradient effect
/ network degeneracy on propagation of second order gradients through the
network (and not the first). As a result, we need an alternative way to match
Jacobians.

####  5.2 Matching attention maps

It is often insufficient to match only output activations between a teacher
and a student network, especially when both networks are deep. In such cases
we can consider matching intermediate feature maps as well. In general it is
not possible to match the full feature maps between an arbitrary teacher and
student network as they may have different architectures, and features sizes
may never match at any layer. However, for modern convolutional architectures,
spatial sizes of certain features often match across architectures even when
the number of channels doesn’t. Zagoruyko & Komodakis (2017) noticed that it
in such cases it helps to match channelwise aggregated activations, which they
call attention maps. Specifically, this aggregation is performed by summing
over squared absolute value of channels of a feature activation $Z$, and is
given by -

$A=AttMap(Z)=\sum_{\mathclap{{i\in channels}}}|Z_{i}|^{2}$ (8)

Further, the loss function used to match these activations is

$\mathrm{Match~{}Activations}=\left|\left|\frac{A_{t}}{\|A_{t}\|_{2}}-\frac{A_{s}}{\|A_{s}\|_{2}}\right|\right|_{2}$
(9)

Here, $A_{t},A_{s}$ are the attention maps of the teacher and student
respectively. Zagoruyko & Komodakis (2017) note that this choice of loss
function is especially crucial.

#####  5.2.1 Incorporating Jacobian loss

As the activation maps have large spatial dimensions, it is computationally
costly to compute the full Jacobians. We hence resort to computing
approximating Jacobians in the same manner as previously discussed. In this
case, this leads to picking the pixel in the attention map with the largest
magnitude, and computing the Jacobian of this quantity w.r.t. the input. We
compute the index $(i,j)$ of this maximum-valued pixel for the teacher network
and use the same index to compute the student’s Jacobian. We then use a loss
function similar to Equation 9, given by

$\mathrm{Match~{}Jacobians}=\left|\left|\frac{\nabla_{x}f(\mathbf{x})}{\|\nabla_{x}f(\mathbf{x})\|_{2}}-\frac{\nabla_{x}g(\mathbf{x})}{\|\nabla_{x}g(\mathbf{x})\|_{2}}\right|\right|_{2}^{2}$
(10)

#####  5.2.2 Justification for Jacobian loss

We can show that the above loss term corresponds to adding a noise term
$\bm{\xi}_{f}\propto\|\nabla_{x}f(\mathbf{x})\|^{-1}_{2}$ for $f(\mathbf{x})$
and $\bm{\xi}_{g}\propto\|\nabla_{x}g(\mathbf{x})\|^{-1}_{2}$ for
$g(\mathbf{x})$ for the distillation loss. From the first order Taylor series
expansion, we see that
$g(x+\bm{\xi})=g(x)+\bm{\xi}_{g}\nabla_{x}g(\mathbf{x})$. Thus for networks
$f(\cdot)$ and $g(\cdot)$ with different Jacobian magnitudes, we expect
different responses for the same noisy inputs. Specifically, we see that
$\mathbb{E}_{\bm{\xi}_{g}}\|g(x+\bm{\xi}_{g})-g(x)\|^{2}_{2}=\sigma_{g}^{2}\|\nabla_{x}g(\mathbf{x})\|^{2}_{2}=\sigma^{2}\frac{\|\nabla_{x}g(\mathbf{x})\|^{2}_{2}}{\|\nabla_{x}g(\mathbf{x})\|^{2}_{2}}=\sigma^{2}$
for a gaussian model with covariance matrix being $\sigma$ times the identity.

###  6 Experiments

We perform three experiments to show the effectiveness of using Jacobians.
First, we perform distillation in a limited data setting on the CIFAR100
dataset (Krizhevsky & Hinton, 2009). Second, we show on that same dataset that
penalizing Jacobian norm increases stability of networks to random noise.
Finally, we perform transfer learning experiments on the MIT Scenes dataset
(Quattoni & Torralba, 2009).

####  6.1 Distillation

For the distillation experiments, we use VGG-like (Simonyan & Zisserman, 2014)
architectures with batch normalization. The main difference is that we keep
only the convolutional layers and have only one fully connected layer rather
than three. Our workflow is as follows. First, a 9-layer “teacher” network is
trained on the full CIFAR100 dataset. Then, a smaller 4-layer “student”
network is trained, but this time on small subsets rather than the full
dataset. As the teacher is trained on much more data than the student, we
expect distillation to improve the student’s performance.

A practical scenario where this would be useful is the case of architecture
search and ensemble training, where we require to train many candidate neural
network architectures on the same task. Distillation methods can help speed up
such methods by using already trained networks to accelerate training of newer
models. One way to achieve acceleration is by using less data to train the
student dataset.

We compare our methods against the following baselines. (1): Cross-Entropy
(CE) training – Here we train the student using only the ground truth (hard
labels) available with the dataset without invoking the teacher network. (2):
CE + match activations – This is the classical form of distillation, where the
activations of the teacher network are matched with that of the student. This
is weighted with the cross-entropy term which uses ground truth targets. (3):
Match activations only – Same as above, except that the cross-entropy term is
not used in the loss function.

We compare these methods by appending the Jacobian matching term in each case.
In all cases, we use the squared-error distillation loss shown in Proposition
1. We found that squared loss worked much better than the cross-entropy loss
for distillation and it was much easier to tune.

From Table 1 we can conclude that (1) it is generally beneficial to do any
form of distillation to improve performance, (2) matching Jacobians along with
activations outperforms matching only activations in low-data settings, (3)
not matching Jacobians is often beneficial for large data.

One interesting phenomenon we observe is that having a cross-entropy (CE)
error term is often not crucial to maintain good performance. It is only
slightly worse than using ground truth labels.

For Table 1, we see that when training with activations, Jacobians and regular
cross-entropy training (fourth row), we reach an accuracy of $52.43\%$ when
training with 100 data points per class. We observe that the overall accuracy
of raw training using the full dataset is $54.28\%$. Thus we are able to reach
close to the full training accuracy using $1/5^{th}$ of the data requirement.

Table 1: Distillation performance on the CIFAR100 dataset. Table shows test accuracy (%). We find that matching both activations and Jacobians along with cross-entropy error performs the best for limited-data settings. The student network is VGG-4 while the teacher is a VGG-9 network which achieves $64.78\%$ accuracy. # of Data points per class $\rightarrow$ | 1 | 5 | 10 | 50 | 100 | 500 (full)
---|---|---|---|---|---|---
Cross-Entropy (CE) training | 5.69 | 13.9 | 20.03 | 37.6 | 44.92 | 54.28
CE + match activations | 12.13 | 26.97 | 33.92 | 46.47 | 50.92 | 56.65
CE + match Jacobians | 6.78 | 23.94 | 32.03 | 45.71 | 51.47 | 53.44
CE + match {activations + Jacobians} | 13.78 | 33.39 | 39.55 | 49.49 | 52.43 | 54.57
Match activations only | 10.73 | 28.56 | 33.6 | 45.73 | 50.15 | 56.59
Match {activations + Jacobians} | 13.09 | 33.31 | 38.16 | 47.79 | 50.06 | 51.33

####  6.2 Noise robustness

We perform experiments where we penalize the Jacobian norm to improve
robustness of models to random noise. We train 9-layer VGG networks on
CIFAR100 with varying amount of the regularization strength ($\lambda$), and
measure their classification accuracy in presence of noise added to the
normalized images. From Table 2 we find that using higher regularization
strengths is better for robustness, even when the initial accuracy at the
zero-noise case is lower. This aligns remarkably well with theory.
Surprisingly, we find that popular regularizers such as $\ell_{2}$
regularization and dropout (Srivastava et al., 2014) are not robust.

Table 2: Robustness of various VGG-9 models to gaussian noise added to CIFAR100 images at test time. Table shows test accuracy (%). $\lambda$ is the regularization strength of the Jacobian-norm penalty regularizer. $\gamma$ is the $\ell_{2}$ regularization strength and $p$ is the dropout value. We see that the Jacobian-norm penalty can be remarkably robust to noise, unlike $\ell_{2}$ regularization and dropout.  Noise std. dev. $\rightarrow$ | 0 | 0.1 | 0.2 | 0.3 | 0.4
---|---|---|---|---|---
$\lambda=0$ | 64.78 | $61.9\pm 0.07$ | $47.53\pm 0.23$ | $29.63\pm 0.16$ | $17.69\pm 0.17$
$\lambda=0.1$ | 65.62 | $\bm{63.36\pm 0.18}$ | $53.57\pm 0.23$ | $37.38\pm 0.18$ | $23.99\pm 0.19$
$\lambda=1.0$ | 63.59 | $62.66\pm 0.13$ | ${57.53\pm 0.17}$ | ${47.48\pm 0.14}$ | ${35.43\pm 0.11}$
$\lambda=10.0$ | 61.37 | $60.78\pm 0.05$ | $\bm{58.28\pm 0.13}$ | $\bm{52.82\pm 0.10}$ | $\bm{44.96\pm 0.19}$
$\ell_{2}$ regularization ($\gamma=1e-3$) | 66.92 | $60.41\pm 0.27$ | $39.93\pm 0.28$ | $23.32\pm 0.25$ | $13.76\pm 0.16$
Dropout $(p=0.25)$ | 66.8 | $61.53\pm 0.10$ | $44.34\pm 0.19$ | $26.70\pm 0.24$ | $15.77\pm 0.11$

####  6.3 Transfer Learning

For transfer learning, our objective is to improve training on the target
dataset (MIT Scenes) by using Imagenet pre-trained models. Crucially, we want
our MIT Scenes model to have a different architecture than the Imagenet model.
The teacher model we use is a ResNet-34 (He et al., 2016) pre-trained on
Imagenet, while the student model is an untrained VGG-9 model with batch
normalization. We choose VGG-9 because its architecture is fundamentally
different from a ResNet. In principle we could use any architecture for the
student. We modify this VGG-9 architecture such that it has two sets of
outputs, one sharing the label space with Imagenet (1000 classes), and another
with MIT Scenes (67 classes, $\sim$ 6k images). The pre-final layer is common
to both outputs.

Our baselines are the following. (1): Cross-Entropy (CE) training of student
with ground truth – Here we ignore the VGG-9 branch with 1000 classes and
optimize the cross-entropy error on MIT Scenes data. The loss function on this
branch is always the same for all methods. (2): CE on pre-trained network –
This is exactly the same as the first baseline, except that the VGG-9 model is
initialized from Imagenet pre-trained weights. We consider this as our
“oracle” method and strive to match it’s performance. (3): CE + match
activations (LwF) – This corresponds to the method of Li & Hoiem (2016), where
the Imagenet branch output activations of the student are matched with those
of the teacher. (4): CE + match { activations + attention}  – This corresponds
to the method of Zagoruyko & Komodakis (2017), where attention maps are
matched between some intermediate layers.

We add our Jacobian matching terms to the baselines 3 and 4. We provide our
results in Table 3. In all cases, we vary the number of images per class on
MIT Scenes to observe the performance on low-data settings as well. In this
case we average our results over two runs by choosing different random
subsets.

Table 3: Transfer Learning from Imagenet to MIT Scenes dataset. Table shows test accuracy (%). The student network (VGG9) is trained from scratch unless otherwise mentioned. The teacher network used is a pre-trained ResNet34. Results are averaged over two runs. # of Data points per class $\rightarrow$ | 5 | 10 | 25 | 50 | Full
---|---|---|---|---|---
Cross-Entropy (CE) training on untrained student network | 11.64 | 20.30 | 35.19 | 46.38 | 59.33
CE on pre-trained student network (Oracle) | 25.93 | 43.81 | 57.65 | 64.18 | 71.42
CE + match activations (Li & Hoiem, 2016) | 17.08 | 27.13 | 45.08 | 55.22 | 65.22
CE + match {activations + Jacobians} | 17.88 | 28.25 | 45.26 | 56.49 | 66.04
CE + match {activations + attention} (Zagoruyko & Komodakis, 2017) | 16.53 | 28.35 | 46.01 | 57.80 | 67.24
CE + match {activations + attention + Jacobians} | 18.02 | 29.25 | 47.31 | 58.35 | 67.31
Table 4: Ablation experiments over choice of feature matching depth. ($\mathcal{T}$, $\mathcal{S}$) denotes teacher (ResNet34) and student (VGG9) feature depths. These pairs are chosen such that resulting features have same spatial dimensions. We see that matching the shallowest feature works the best. Results are averaged over two runs. | Feature matching
---
depth ($\mathcal{T}$, $\mathcal{S}$)
(7, 2) | (15, 4) | (27, 6) | (33, 8)
Accuracy (%) | 22.39 | 21.98 | 20.45 | 20.03
| Jacobian loss
---
reduction (%)
25.88 | 15.59 | 11.55 | 1.25
Table 5: Ablation experiments over the computation of Jacobian. Here, $s$ is the size of the attention map. “Full” is global average pooling, and “None” is no average pooling. We see that using average pooling while computing Jacobians helps performance. Results are averaged over two runs. | Average Pool
---
Window size
Full | $s/3$ | $s/5$ | $s/7$ |  | None
---
Accuracy (%) | 20.00 | 21.20 | 21.87 | 21.09 | 19.74

Experiments show that matching activations and attention maps increases
performance at all levels of data size. It also shows that Jacobians improve
performance of all these methods. However, we observe that none of the methods
match the oracle performance of using a pre-trained model. The gap in
performance is especially large at intermediate data sizes of $10$ and $25$
images per class.

#####  6.3.1 Which features to match?

An important practical question is the choice of intermediate features to
compute attention maps for matching. The recipe followed by Zagoruyko &
Komodakis (2017) for ResNets is to consider features at the end of a residual
block. 111A residual block is the set of all layers in between two consecutive
max-pooling layers in a ResNet. All features in a block have the same
dimensions. As there are typically 3-5 max-pooling layers in most modern
networks, we have 3-5 intermediate features to match between any typical
teacher and student network. Note that we require the attention maps
(channelwise aggregated features) to be of similar spatial size to match.
Zagoruyko & Komodakis (2017) match at all such possible locations, and we use
the same approach.

However, computing Jacobians at all such locations is computationally
demanding and perhaps unnecessary. We observe that if we compute Jacobians at
later layers, we are still not able to reduce training Jacobian loss, possibly
due to a “second-order” vanishing gradient effect. At suitable intermediate
layers, we see that loss reduction occurs. This is reflected in Table 4, where
we vary the feature matching depth and observe performance. We observe that
the Jacobian loss reduction (during training) is highest for the shallowest
layers, and this corresponds to good test performance as well. These ablation
experiments are done on the MIT Scenes dataset picking only $10$ points per
class.

#####  6.3.2 Feature Pooling to compute Jacobians

Instead of considering a single pixel per attention map to compute Jacobians,
we can aggregate a large number of large-magnitude pixels. One way to do this
is by average pooling over the attention map, and then picking the maximum
pixel over the average pooled map. In Table 5 we vary the window size for
average pooling and observe performance. We observe that it is beneficial to
do average pooling, we find that using a window size of
$\mathrm{(feature~{}size)}/5$ works the best. These ablation experiments are
done on the MIT Scenes dataset picking only $10$ points per class.

###  7 Conclusion

In this paper we considered matching Jacobians of deep neural networks for
knowledge transfer. Viewing Jacobian matching as a form of data augmentation
with gaussian noise motivates their usage, and also informs us about the loss
functions to use. We also connected a recent transfer learning method (LwF) to
distillation, enabling us to incorporate Jacobian matching.

Despite our advances, there is still a large gap between distillation-based
methods and the oracle method of using pre-trained nets for transfer learning.
Future work can focus on closing this gap by considering more structured forms
of data augmentation than simple noise addition.

### References

  * Ba & Caruana (2014) Ba, LJ and Caruana, R.  Do deep networks really need to be deep.  _Advances in neural information processing systems_ , 27:1–9, 2014.
  * Ben-David et al. (2010) Ben-David, Shai, Blitzer, John, Crammer, Koby, Kulesza, Alex, Pereira, Fernando, and Vaughan, Jennifer Wortman.  A theory of learning from different domains.  _Machine learning_ , 79(1-2):151–175, 2010.
  * Bishop (1995) Bishop, Chris M.  Training with noise is equivalent to tikhonov regularization.  _Neural Computation_ , 1995.
  * Cisse et al. (2017) Cisse, Moustapha, Bojanowski, Piotr, Grave, Edouard, Dauphin, Yann, and Usunier, Nicolas.  Parseval networks: Improving robustness to adversarial examples.  In _International Conference on Machine Learning_ , pp. 854–863, 2017.
  * Czarnecki et al. (2017) Czarnecki, Wojciech Marian, Osindero, Simon, Jaderberg, Max, Świrszcz, Grzegorz, and Pascanu, Razvan.  Sobolev training for neural networks.  _NIPS_ , 2017.
  * Drucker & Le Cun (1992) Drucker, Harris and Le Cun, Yann.  Improving generalization performance using double backpropagation.  _IEEE Transactions on Neural Networks_ , 1992.
  * Furlanello et al. (2016) Furlanello, Tommaso, Zhao, Jiaping, Saxe, Andrew M, Itti, Laurent, and Tjan, Bosco S.  Active long term memory networks.  _arXiv preprint arXiv:1606.02355_ , 2016.
  * He et al. (2016) He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hinton et al. (2015) Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff.  Distilling the knowledge in a neural network.  _NIPS Deep Learning Workshop_ , 2015.
  * Jung et al. (2016) Jung, Heechul, Ju, Jeongwoo, Jung, Minju, and Kim, Junmo.  Less-forgetting learning in deep neural networks.  _arXiv preprint arXiv:1607.00122_ , 2016.
  * Krizhevsky & Hinton (2009) Krizhevsky, Alex and Hinton, Geoffrey.  Learning multiple layers of features from tiny images.  2009\.
  * Li & Hoiem (2016) Li, Zhizhong and Hoiem, Derek.  Learning without forgetting.  In _European Conference on Computer Vision_ , pp. 614–629. Springer, 2016.
  * Pan & Yang (2010) Pan, Sinno Jialin and Yang, Qiang.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ , 22(10):1345–1359, 2010.
  * Quattoni & Torralba (2009) Quattoni, A and Torralba, A.  Recognizing indoor scenes.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009.
  * Romero et al. (2014) Romero, Adriana, Ballas, Nicolas, Kahou, Samira Ebrahimi, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua.  Fitnets: Hints for thin deep nets.  _arXiv preprint arXiv:1412.6550_ , 2014.
  * Russakovsky et al. (2015) Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, et al.  Imagenet large scale visual recognition challenge.  _International Journal of Computer Vision_ , 115(3):211–252, 2015.
  * Sau & Balasubramanian (2016) Sau, Bharat Bhusan and Balasubramanian, Vineeth N.  Deep model compression: Distilling knowledge from noisy teachers.  _arXiv preprint arXiv:1610.09650_ , 2016.
  * Simonyan & Zisserman (2014) Simonyan, Karen and Zisserman, Andrew.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Srivastava et al. (2014) Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan.  Dropout: A simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ , 15(1):1929–1958, 2014.
  * Wang et al. (2016) Wang, Shengjie, Mohamed, Abdel-rahman, Caruana, Rich, Bilmes, Jeff, Plilipose, Matthai, Richardson, Matthew, Geras, Krzysztof, Urban, Gregor, and Aslan, Ozlem.  Analysis of deep neural networks with extended data jacobian matrix.  In _International Conference on Machine Learning_ , pp. 718–726, 2016.
  * Xu & Mannor (2012) Xu, Huan and Mannor, Shie.  Robustness and generalization.  _Machine learning_ , 86(3):391–423, 2012.
  * Yosinski et al. (2014) Yosinski, Jason, Clune, Jeff, Bengio, Yoshua, and Lipson, Hod.  How transferable are features in deep neural networks?  In _Advances in neural information processing systems_ , pp. 3320–3328, 2014.
  * Zagoruyko & Komodakis (2017) Zagoruyko, Sergey and Komodakis, Nikos.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  _ICLR_ , 2017.

## Supplementary Material

###  8 Proof for Proposition 1

######  Proposition 1.

Consider the squared error cost function for matching soft targets of two
neural networks with $k$-length targets ($\in\mathbb{R}^{k}$), given by
$\ell(\mathcal{T}(\mathbf{x}),\mathcal{S}(\mathbf{x}))=\sum_{i=1}^{k}(\mathcal{T}^{i}(\mathbf{x})-\mathcal{S}^{i}(\mathbf{x}))^{2}$,
where $\mathbf{x}\in\mathbb{R}^{D}$ is an input data point. Let
$\bm{\xi}~{}(\in\mathbb{R}^{D})=\sigma~{}\mathbf{z}$ be a scaled version of a
unit normal random variable $\mathbf{z}~{}\in\mathbb{R}^{D}$ with scaling
factor $\sigma\in\mathbb{R}$. Then the following is locally true.

$\displaystyle\mathbb{E}_{\bm{\xi}}\left[\sum_{i=1}^{k}\left(\mathcal{T}^{i}(\mathbf{x}+\bm{\xi})-\mathcal{S}^{i}(\mathbf{x}+\bm{\xi})\right)^{2}\right]$
$\displaystyle=\sum_{i=1}^{k}\left(\mathcal{T}^{i}(\mathbf{x})-\mathcal{S}^{i}(\mathbf{x})\right)^{2}+\sigma^{2}\sum_{i=1}^{k}\|\nabla_{x}\mathcal{T}^{i}(\mathbf{x})-\nabla_{x}\mathcal{S}^{i}(\mathbf{x})\|_{2}^{2}$
$\displaystyle+\mathcal{O}(\sigma^{4})$

###### Proof.

There exists $\sigma$ and $\bm{\xi}$ small enough that first-order Taylor
series expansion holds true

$\displaystyle\mathbb{E}_{\bm{\xi}}\left[\sum_{i=1}^{k}\left(\mathcal{T}^{i}(\mathbf{x}+\bm{\xi})-\mathcal{S}^{i}(\mathbf{x}+\bm{\xi})\right)^{2}\right]$
$\displaystyle=$
$\displaystyle~{}\mathbb{E}_{\bm{\xi}}\left[\sum_{i=1}^{k}\left(\mathcal{T}^{i}(\mathbf{x})+\bm{\xi}^{T}\nabla_{x}\mathcal{T}^{i}(\mathbf{x})-\mathcal{S}^{i}(\mathbf{x})-\bm{\xi}^{T}\nabla_{x}\mathcal{S}^{i}(\mathbf{x})\right)^{2}\right]$
$\displaystyle+$ $\displaystyle\mathcal{O}(\sigma^{4})$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{k}\left(\mathcal{T}^{i}(\mathbf{x})-\mathcal{S}^{i}(\mathbf{x})\right)^{2}$
$\displaystyle+$
$\displaystyle\mathbb{E}_{\bm{\xi}}\left[\sum_{i=1}^{k}\left[\bm{\xi}^{T}\left(\nabla_{x}\mathcal{T}^{i}(\mathbf{x})-\nabla_{x}\mathcal{S}^{i}(\mathbf{x})\right)\right]^{2}\right]+\mathcal{O}(\sigma^{4})$
(11)

To get equation 11, we use the fact that mean of $\bm{\xi}$ is zero. To
complete the proof, we use the diagonal assumption on the covariance matrix of
$\bm{\xi}$.

∎

Proofs of other statements are similar. For proof for cross-entropy loss of
Proposition 2, use a second order Taylor series expansion of $\log(\cdot)$ in
the first step.

###  9 Proof for Proposition 3

######  Proposition 2.

From the notations in the main text, we have

$\displaystyle\frac{1}{|\mathcal{D}_{l}|}\sum_{\mathbf{x}\sim\mathcal{D}_{l}}\ell(f(\mathbf{x}),g(\mathbf{x}))$
$\displaystyle\leq\max_{\mathbf{x}\sim\mathcal{D}_{s}}\ell(f(\mathbf{x}),g(\mathbf{x}))$
$\displaystyle+\mathrm{K}\mathcal{H}_{a}(\mathcal{D}_{l},\mathcal{D}_{s})$

###### Proof.

Let us denote $\rho(\mathbf{x})=\ell(f(\mathbf{x}),g(\mathbf{x}))$ for
convenience. Assume Lipschitz continuity for $\rho(\mathbf{x})$ with Lipschitz
constant $\mathrm{K}$, and distance metric $\psi_{\mathbf{x}}(\cdot,\cdot)$ in
the input space -

$\displaystyle\|\rho(\mathbf{x}_{1})-\rho(\mathbf{x}_{2})\|_{1}\leq\mathrm{K}\psi_{\mathbf{x}}(\mathbf{x}_{1},\mathbf{x}_{2})$
$\displaystyle\implies\rho(\mathbf{x}_{1})\leq\rho(\mathbf{x}_{2})+\mathrm{K}\psi_{\mathbf{x}}(\mathbf{x}_{1},\mathbf{x}_{2})$

Assuming that $\rho(\mathbf{x}_{1})\geq\rho(\mathbf{x}_{2})$. Note that it
holds even otherwise, but is trivial.

Now, for every datapoint $\mathbf{x}_{l}\in\mathcal{D}_{l}$, there exists a
point $\mathbf{x}_{s}\in\mathcal{D}_{s}$ such that
$\psi_{\mathbf{x}}(\mathbf{x}_{s},\mathbf{x}_{l})$ is the smallest among all
points in $\mathcal{D}_{s}$. In other words, we look at the point in
$\mathcal{D}_{s}$ closest to each point $\mathbf{x}_{l}$. Note that in this
process only a subset of points $\mathrm{d}_{s}\subseteq\mathcal{D}_{s}$ are
chosen, and individual points can be chosen multiple times. For these points,
we can write

$\displaystyle\rho(\mathbf{x}_{l})\leq\rho(\mathbf{x}_{s})+\mathrm{K}\psi_{\mathbf{x}}(\mathbf{x}_{s},\mathbf{x}_{l})$
$\displaystyle\implies\frac{1}{|\mathcal{D}_{l}|}\sum_{\mathbf{x}_{l}\sim\mathcal{D}_{l}}\rho(\mathbf{x}_{l})\leq\frac{1}{|\mathcal{D}_{l}|}\sum_{\mathbf{x}_{s}\mathrm{~{}closest~{}to~{}}\mathbf{x}_{l}}$
$\displaystyle\rho(\mathbf{x}_{s})$
$\displaystyle+\frac{1}{|\mathcal{D}_{l}|}\sum_{\mathbf{x}_{s}\mathrm{~{}closest~{}to~{}}\mathbf{x}_{l}}$
$\displaystyle\mathrm{K}\psi_{\mathbf{x}}(\mathbf{x}_{s},\mathbf{x}_{l})$

We see that
$\frac{1}{|\mathcal{D}_{l}|}\sum_{\mathbf{x}_{s}}\rho(\mathbf{x}_{s})\leq\max_{\mathbf{x}\sim\mathrm{d}_{s}}\rho(\mathbf{x})\leq\max_{\mathbf{x}\sim\mathcal{D}_{s}}\rho(\mathbf{x})$,
which is a consequence of the fact that the max is greater than any convex
combination of elements.

Also, we have
$\psi_{\mathbf{x}}(\mathbf{x}_{l},\mathbf{x}_{s})\leq\mathcal{H}_{a}(\mathcal{D}_{l},\mathcal{D}_{s})$,
which is the maximum distance between any two ‘closest’ points from
$\mathcal{D}_{l}$ to $\mathcal{D}_{s}$ (by definition).

Applying these bounds, we have the final result.

∎

####  9.1 Proof for Corollary

######  Corollary.

For any superset $\mathrm{D^{\prime}}_{s}\supseteq\mathcal{D}_{s}$ of the
target dataset,
$\mathcal{H}_{a}(\mathcal{D}_{l},\mathcal{D^{\prime}}_{s})\leq\mathcal{H}_{a}(\mathcal{D}_{l},\mathcal{D}_{s})$

###### Proof.

From the previous proof, we have
$\rho(\mathbf{x}_{l})\leq\rho(\mathbf{x}_{s})+\mathrm{K}\psi_{\mathbf{x}}(\mathbf{x}_{s},\mathbf{x}_{l})$
for an individual point $\mathbf{x}_{l}$. Now if we have
$\mathrm{D^{\prime}}_{s}\supseteq\mathcal{D}_{s}$, then we have
$\rho(\mathbf{x}_{l})\leq\rho(\mathbf{x}^{\prime}_{s})+\mathrm{K}\psi_{\mathbf{x}}(\mathbf{x}^{\prime}_{s},\mathbf{x}_{l})$,
where $\mathbf{x}^{\prime}_{s}$ is the new point closest to $\mathbf{x}_{l}$.
It is clear that
$\psi_{\mathbf{x}}(\mathbf{x}^{\prime}_{s},\mathbf{x}_{l})\leq\psi_{\mathbf{x}}(\mathbf{x}_{s},\mathbf{x}_{l})$
for all $\mathbf{x}_{l}$. Hence it follows that
$\mathcal{H}_{a}(\mathcal{D}_{l},\mathcal{D^{\prime}}_{s})\leq\mathcal{H}_{a}(\mathcal{D}_{l},\mathcal{D}_{s})$.

∎

###  10 Experimental details

####  10.1 VGG Network Architectures

The architecture for our networks follow the VGG design philosophy.
Specifically, we have blocks with the following elements:

  * •

$3\times 3$ conv kernels with $c$ channels of stride 1

  * •

Batch Normalization

  * •

ReLU

Whenever we use Max-pooling (M), we use stride $2$ and window size $2$.

The architecture for VGG-9 is - $[64-M-128-M-256-256-M-512-512-M-512-512-M]$.
Here, the number stands for the number of convolution channels, and $M$
represents max-pooling. At the end of all the convolutional and max-pooling
layers, we have a Global Average Pooling (GAP) layer, after which we have a
fully connected layer leading up to the final classes. Similar architecture is
used for the case of both CIFAR and MIT Scene experiments.

The architecture for VGG-4 is - $[64-M-128-M-512-M]$.

####  10.2 Loss function

The loss function for distillation experiments use the following form.

$\ell(\mathcal{S},\mathcal{T})=\alpha\times\mathrm{(CE)}+\beta\times\mathrm{(Match~{}Activations)}+\gamma\times\mathrm{(Match~{}Jacobians)}$

In our experiments, $\alpha,\beta,\gamma$ are either set to $1$ or $0$. In
other words, all regularization constants are $1$.

Here, ‘CE’ refers to cross-entropy with ground truth labels. ‘Match
Activations’ refers to squared error term over pre-softmax activations of the
form $(y_{s}-y_{t})^{2}$. ‘Match Jacobians’ refers to the same squared error
term, but for Jacobians.

For the MIT Scene experiments, $\alpha,\beta,\gamma$ are either set to $10$ or
$0$, depending on the specific method. To compute the Jacobian, we use average
pooling over a $feature~{}size/5$ window with a stride of $1$. We match the
Jacobian after the first residual block for resnet, and after the second max-
pool for VGG. This corresponds to feature level “1” in the ablation
experiments.

####  10.3 Optimization

For CIFAR100 experiments, we run optimization for $500$ epochs. We use the
Adam optimizer, with an initial learning rate of $1e-3$, and a single learning
rate annealing (to $1e-4$) at $400$ epochs. We used a batch size of $128$.

For MIT Scenes, we used SGD with momentum of $0.9$, for $75$ epochs. The
initial learning rate is $1e-3$, and it is reduced $10$ times after $40$ and
$60$ epochs. We used batch size $8$. This is because the Jacobian computation
is very memory intensive.