Two-loop gauge coupling $\beta$-function in a four-dimensional framework: the
Standard Model case

Adriano Cherchiglia1

1 Centro de Ciêcias Naturais e Humanas, Universidade Federal do ABC, Santo
André, Brazil

*<EMAIL_ADDRESS>
15th International Symposium on Radiative Corrections:
Applications of Quantum Field Theory to Phenomenology,
FSU, Tallahasse, FL, USA, 17-21 May 2021

## Abstract

At present, the gauge coupling $\beta$-function in the Standard Model (SM) is
known up to four-loop order. As most SM calculations, dimensional
regularization was employed. Despite its striking success, other
regularization schemes have emerged, which aim to stay in the physical
dimension as far as possible. In this contribution, we apply Implicit
Regularization, a scheme defined in momentum space that complies with BPHZ, to
obtain the two-loop gauge coupling $\beta$-function in the Standard Model. We
reproduce the same result obtained when applying dimensional regularization.

###### Contents

  1. 1 Introduction
  2. 2 The Implicit Regularization formalism
  3. 3 Gauge coupling $\beta$-function in the Standard Model to two-loop order
  4. 4 Conclusion

##  1 Introduction

Renormalization group functions are essential ingredients to precisely probe
the Standard Model and its extensions. They allow experimental results
performed at different scales to be matched to theoretical predictions
stemming from the knowledge of divergent intermediate pieces. Given the level
of accuracy and accordance among experiment and theory, they can actually be
seen as a triumph of the renormalization program implemented in the context of
quantum field theory. The current state-of-the-art comprises the knowledge of
the gauge coupling $\beta$-function in the SM at four-loop order [1, 2], an
achievement only possible after some recent developments regarding the
treatment of the $\gamma_{5}$ matrix [3]. As customary, such calculations have
been performed by relying on dimensional regularization, which may suffer from
inconsistencies when dealing with dimensional specific objects.

As an effort to circumvent these problems, other regularization methods have
been proposed, which aim to stay in the physical dimension as far as possible
(see [4, 5] for a review). However, as recently pointed out [6], they may also
suffer from similar issues in general (see also [7, 8, 9] for particular
examples). Nevertheless, since some of these methods aim to perform
regularization/renormalization at integrand level, they may be more effective
and prone to a numerical approach. One promising candidate is Implicit
Regularization [10] which is defined in momentum space and complies with BPHZ
[11]. In order to sketch the potential of this method, some working examples
can be envisaged, one of those being the determination of the gauge coupling
$\beta$-function in the SM at two-loop order. This constitutes a good check
since the coefficients up to this loop order are universal as long as the
subtraction schemes employed are mass-independent as is the case here.
Moreover, this calculation may serve as a backbone to a more complete study of
the RGE functions, which are regularization dependent at higher loop order in
general, allowing the establishment of transition rules among methods. Given
these prospects, in this contribution we obtain the gauge coupling
$\beta$-function in the SM at two-loop order in the context of Implicit
Regularization, confirming that it complies with known results.

We organize our work as follows: in section 2 we present a very brief overview
of the IREG formalism, while in section 3 we present our main results.
Finally, we conclude in section 4.

##  2 The Implicit Regularization formalism

The Implicit Regularization (IREG) method, first presented in [10], is based
on the idea that the UV divergent part of a general Feynman diagram can be
isolated in integrals void from physical parameters (masses/external momenta).
This is accomplished by applying a mathematical identity at integrand level
which allows a separation between a UV only divergent integral (free of
external momenta/masses) and a finite part. All in all, the UV divergent part
of a general 1-loop Feynman amplitude will be proportional to

$\displaystyle I_{log}(\lambda^{2})$ $\displaystyle\equiv$
$\displaystyle\int_{k}\frac{1}{(k^{2}-\lambda^{2})^{2}},$ (1)

where $\lambda$ plays the role of the renormalization group scale in the
method. By performing subtractions of $I_{log}(\lambda^{2})$ one defines a
minimal subtraction scheme in the context of IREG. Therefore, the 1-loop
renormalization functions $Z_{i}$ will be proportional to it as well. Since
derivatives of $Z_{i}$ account for renormalization group functions,
derivatives of $I_{log}(\lambda^{2})$ will be needed as well

$\lambda^{2}\frac{\partial
I_{log}(\lambda^{2})}{\partial\lambda^{2}}=-b,\quad\quad
b=\frac{i}{(4\pi)^{2}}.$ (2)

At higher loop order a similar program can be envisaged in a way compatible
with Bogoliubov’s recursion formula [11]. Since we are adopting a mass-
independent subtraction scheme, the UV divergent part will only contain a
logarithmic dependence, allowing us to define

$\displaystyle I_{log}^{(l)}(\lambda^{2})$
$\displaystyle\equiv\int\limits_{k_{l}}\frac{1}{(k_{l}^{2}-\lambda^{2})^{n}}\ln^{l-1}{\left(-\frac{k_{l}^{2}-\lambda^{2}}{\lambda^{2}}\right)},$
(3)

whose derivatives with respect to $\lambda$ can be straightforwardly obtained

$\displaystyle\lambda^{2}\frac{\partial
I_{log}^{(l)}(\lambda^{2})}{\partial\lambda^{2}}$
$\displaystyle=-(l-1)\,I_{log}^{(l-1)}(\lambda^{2})-b\,\,(l-1)!\,\quad\mbox{where}\quad
l>1.$ (4)

Given these definitions, we can express the UV divergent part of a general
Feynman amplitude at two-loop level as

$\displaystyle\mathcal{A}=c_{1}\;I_{log}^{(2)}(\lambda^{2})+c_{2}\;\left[I_{log}(\lambda^{2})\right]^{2}+c_{3}\;I_{log}(\lambda^{2})$
(5)

where $c_{i}$ are coefficients that may depend on masses/external momenta.

This brief description is aimed to provide the reader with the most basic and
necessary ingredients of the IREG rules, easing the presentation of the
results of upcoming sections. For further details we refer, for instance, to
[12].

##  3 Gauge coupling $\beta$-function in the Standard Model to two-loop order

As a non-trivial example of the application of IREG we will pursue the
computation of the two-loop coefficients for the gauge couplings
$\beta$-function in the Standard Model. This improves the analysis performed
in [13] where the $\beta$-function for an abelian (QED) and non-abelian (QCD)
theory was studied.

For this end, we will adopt the conventions of [14] and use the background
field method [15, 16]. This allows us to only compute two-point functions,
instead of three-point ones needed in a standard computation. The generic
topologies we are interested at can be seen in fig. 1.

Figure 1: Two-loop topologies for two-point functions

whose amplitudes are of the form

$\displaystyle\mathcal{A_{\text{T1}}}$
$\displaystyle\propto\int_{k,l}\frac{\mathcal{F}_{T1}(l,k,p)}{k^{2}(k-p)^{2}l^{2}(l-k)^{2}},$
$\displaystyle\mathcal{A_{\text{T2}}}$
$\displaystyle\propto\int_{k,l}\frac{\mathcal{F}_{T2}(l,k,p)}{k^{2}(k-p)^{2}(k-l)^{2}l^{2}(l-p)^{2}},$
$\displaystyle\mathcal{A_{\text{T3}}}$
$\displaystyle\propto\int_{k,l}\frac{\mathcal{F}_{T3}(l,k,p)}{k^{4}(k-p)^{2}l^{2}(l-k)^{2}},$
$\displaystyle\mathcal{A_{\text{T4}}}$
$\displaystyle\propto\int_{k,l}\frac{\mathcal{F}_{T4}(l,k,p)}{k^{2}(k-p)^{2}l^{2}(l-p)^{2}},$
$\displaystyle\mathcal{A_{\text{T5}}}$
$\displaystyle\propto\int_{k,l}\frac{\mathcal{F}_{T5}(l,k,p)}{k^{2}l^{2}(l-k+p)^{2}}.$
(6)

Notice that we are considering massless propagators since we will only be
interested in the UV divergent part of each amplitude.

The generic topologies will be filled with the particle content of the
Standard Model by using the model file already implemented in FeynArts [17]
supplemented with an extension related to the QCD background field method. A
drawback of this approach is that only the theory on the spontaneously broken
phase is available. Therefore, we will actually compute the 2-loop corrections
to $\tilde{A}\tilde{A}$, $\tilde{A}\tilde{Z}$, $\tilde{Z}\tilde{Z}$,
$\tilde{G}\tilde{G}$ where $\tilde{A}$, $\tilde{Z}$, $\tilde{G}$ are the
photon, Z-boson, and gluon background fields. The connection to the
$\tilde{B}$ and $\tilde{W}$ fields is straightforward [14]

$\displaystyle\Pi_{\tilde{B}\tilde{B}}=\cos^{2}\theta_{W}^{\text{bare}}\Pi_{\tilde{A}\tilde{A}}+2\sin\theta_{W}^{\text{bare}}\cos\theta_{W}^{\text{bare}}\Pi_{\tilde{A}\tilde{Z}}+\sin^{2}\theta_{W}^{\text{bare}}\Pi_{\tilde{Z}\tilde{Z}}$
(7)
$\displaystyle\Pi_{\tilde{W}\tilde{W}}=\cos^{2}\theta_{W}^{\text{bare}}\Pi_{\tilde{A}\tilde{A}}-2\sin\theta_{W}^{\text{bare}}\cos\theta_{W}^{\text{bare}}\Pi_{\tilde{A}\tilde{Z}}+\sin^{2}\theta_{W}^{\text{bare}}\Pi_{\tilde{Z}\tilde{Z}}$
(8)

As a check, we have explictly obtained $\tilde{W}\tilde{W}$ as well, which is
identical to the value obtained from the knowledge of
$\Pi_{\tilde{A}\tilde{A}}$, $\Pi_{\tilde{A}\tilde{Z}}$,
$\Pi_{\tilde{Z}\tilde{Z}}$. To perform the computation, we made use of
FormCalc [18] as well as Package-X [19], which we embedded in our own
routines. Since the number of diagrams is substantial, we refrain to present
individual results, and we just quote the end result for the renormalization
function of the background fields obtained in the context of IREG

$\displaystyle Z_{\tilde{B}\tilde{B}}=\;$ $\displaystyle
1-\frac{\alpha_{1}}{4\pi}\left[\frac{1}{10}+\frac{4}{3}n_{f}\right]\frac{I_{log}(\lambda^{2})}{b}+\frac{\alpha_{1}}{(4\pi)^{2}}\left[-\frac{9\alpha_{1}}{50}-\frac{9\alpha_{2}}{10}+\frac{17\text{tr}\hat{T}}{10}+\frac{\text{tr}\hat{B}}{2}+\frac{3\text{tr}\hat{L}}{2}\right.$
$\displaystyle\left.\quad-
n_{f}\left(\frac{19\alpha_{1}}{5}+\frac{9\alpha_{2}}{5}+\frac{44\alpha_{3}}{15}\right)\right]\frac{I_{log}(\lambda^{2})}{b}$
(9) $\displaystyle Z_{\tilde{W}\tilde{W}}=\;$ $\displaystyle
1-\frac{\alpha_{2}}{4\pi}\left[-\frac{43}{6}+\frac{4}{3}n_{f}\right]\frac{I_{log}(\lambda^{2})}{b}+\frac{\alpha_{2}}{(4\pi)^{2}}\left[-\frac{3\alpha_{1}}{10}\frac{I_{log}(\lambda^{2})}{b}\right.$
$\displaystyle\left.\quad+\frac{\alpha_{2}}{6b^{2}}\left(144(I_{log}^{2}(\lambda^{2})-2bI_{log}^{(2)}(\lambda^{2}))+547bI_{log}(\lambda^{2})\right)\right.$
$\displaystyle\left.\quad+\left(\frac{3\text{tr}\hat{T}}{2}+\frac{3\text{tr}\hat{B}}{2}+\frac{\text{tr}\hat{L}}{2}\right)\frac{I_{log}(\lambda^{2})}{b}-n_{f}\left(\frac{3\alpha_{1}}{5}+49\alpha_{2}+4\alpha_{3}\right)\frac{I_{log}(\lambda^{2})}{b}\right]$
(10) $\displaystyle Z_{\tilde{G}\tilde{G}}=\;$ $\displaystyle
1-\frac{\alpha_{3}}{4\pi}\left[-11+\frac{4}{3}n_{f}\right]\frac{I_{log}(\lambda^{2})}{b}+\frac{\alpha_{3}}{(4\pi)^{2}}\left[\frac{6\alpha_{3}}{b^{2}}\left(9(I_{log}^{2}(\lambda^{2})-2bI_{log}^{(2)}(\lambda^{2}))+35bI_{log}(\lambda^{2})\right)\right.$
$\displaystyle\left.+\left(2\text{tr}\hat{T}+2\text{tr}\hat{B}\right)\frac{I_{log}(\lambda^{2})}{b}-n_{f}\left(\frac{11\alpha_{1}}{10}+\frac{9\alpha_{2}}{2}+\frac{38\alpha_{3}}{3}\right)\frac{I_{log}(\lambda^{2})}{b}\right]$
(11)

We have adopted the following conventions when writing our results[14]

$\displaystyle\alpha_{1}=\frac{5}{3}\frac{\alpha_{\text{QED}}}{\cos^{2}\theta_{W}},\quad\alpha_{2}=\frac{\alpha_{\text{QED}}}{\sin^{2}\theta_{W}},\quad\alpha_{3}=\alpha_{s},\quad\text{tr}\hat{X}=\frac{\alpha_{\text{QED}}}{2\sin^{2}\theta_{W}M_{W}^{2}}\sum_{i=1}^{n_{f}}m_{Xi}^{2},$
(12)

where $\alpha_{\text{QED}}$ is the fine structure constant, $\theta_{W}$ is
the Weinberg angle, $M_{W}$ is the W-boson mass, $\alpha_{s}$ is the QCD
strong coupling constant, $m_{Xi}$ is the mass of the $X$-type fermion from
the $i$-th generation, and $n_{f}$ is the number of generations. We have also
applied DREG and DRED, whose results agree with previous calculations [14].
Nevertheless, for comparison with IREG, we quote it here as well

$\displaystyle Z_{\tilde{B}\tilde{B}}=$
$\displaystyle\;1-\frac{\alpha_{1}}{4\pi}\left[\frac{1}{10}+\frac{4}{3}n_{f}\right]\frac{1}{\epsilon}$
$\displaystyle+\frac{\alpha_{1}}{(4\pi)^{2}}\left[-\frac{9\alpha_{1}}{100}-\frac{9\alpha_{2}}{20}+\frac{17\text{tr}\hat{T}}{20}+\frac{\text{tr}\hat{B}}{4}+\frac{3\text{tr}\hat{L}}{4}-n_{f}\left(\frac{19\alpha_{1}}{10}+\frac{9\alpha_{2}}{10}+\frac{22\alpha_{3}}{15}\right)\right]\frac{1}{\epsilon}$
(13) $\displaystyle Z_{\tilde{W}\tilde{W}}=$
$\displaystyle\;1-\frac{\alpha_{2}}{4\pi}\left[-\frac{43}{6}+\frac{4}{3}n_{f}\right]\frac{1}{\epsilon}$
$\displaystyle+\frac{\alpha_{2}}{(4\pi)^{2}}\left[-\frac{3\alpha_{1}}{20}+\frac{259\alpha_{2}}{12}+\frac{3\text{tr}\hat{T}}{4}+\frac{3\text{tr}\hat{B}}{4}+\frac{\text{tr}\hat{L}}{4}-n_{f}\left(\frac{3\alpha_{1}}{10}+\frac{49\alpha_{2}}{2}+2\alpha_{3}\right)\right]\frac{1}{\epsilon}$
(14) $\displaystyle Z_{\tilde{G}\tilde{G}}=$
$\displaystyle\;1-\frac{\alpha_{3}}{4\pi}\left[-11+\frac{4}{3}n_{f}\right]\frac{1}{\epsilon}$
$\displaystyle+\frac{\alpha_{3}}{(4\pi)^{2}}\left[51\alpha_{3}+\text{tr}\hat{T}+\text{tr}\hat{B}-n_{f}\left(\frac{11\alpha_{1}}{20}+\frac{9\alpha_{2}}{4}+\frac{17\alpha_{3}}{3}\right)\right]\frac{1}{\epsilon}$
(15)

Some comments are in order: 1) when extracting the renormalization constants,
we obtained that the end result is transverse, illustrating that IREG complies
with non-abelian gauge invariance (see [13] for a more detailed analysis); 2)
DREG and DRED differs in intermediate terms, but the final result is identical
to each other as it should be (both subtraction schemes DR and MS are mass-
independent [20]); 3) counterterms for quantum fields are not needed as first
discussed in [15], however, since we adopted the Feynman gauge, the
renormalization for the gauge fixing was required.

Regarding the treatment of chiral fermions, we have applied two procedures
available within FormCalc. The first was just the naive scheme for
$\gamma_{5}$, which implies that the $\gamma_{5}$ matrix could be anticommuted
freely. In the second procedure we replaced $\gamma_{5}$ by its definition as

$\gamma_{5}=-\frac{i}{4!}\epsilon_{abcd}{\gamma}^{a}{\gamma}^{b}{\gamma}^{c}{\gamma}^{d},$
(16)

performed Dirac algebra together with contractions of the Levi-Civitá symbols
that appear. Given the increased number of Dirac matrices, the second
procedure demands more computational power. Both procedures amount to the same
end results. We emphasize that none of the procedures is completely consistent
in general. For the first procedure this is clear; for the second one the
reason boils down to the use of identities involving Levi-Civitá symbols which
in the context of IREG generates terms like $k^{2}$. However, since symmetric
integration is not allowed in general in the method, such terms should be
written as ${g}_{ab}k^{a}k^{b}$ instead which stands for enforcing the
regularization before performing Lorentz contractions. For a more detailed
discussion of these points we refer the reader to [6, 21]. Finally, we would
like to mention that ambiguities in the gauge couplings $\beta$-function in
the Standard Model may arise only at four-loop level, at least in the context
of dimensional methods [22]. Since the issues that IREG must face in the
presence of $\gamma_{5}$ are similar to the ones encountered in dimensional
methods, we also expect that only at higher loop order (potential) ambiguities
may arise.

To conclude this section, we present the gauge coupling $\beta$-function up to
two-loop order obtained within the IREG framework. We adopt the same
conventions of [14]

$\displaystyle\beta_{i}=\lambda^{2}\frac{d}{d\lambda^{2}}\frac{\alpha_{i}}{\pi}=-\frac{\alpha_{i}}{\pi}\lambda^{2}\frac{d}{d\lambda^{2}}\ln
Z_{\alpha_{i}}=\frac{\alpha_{i}}{\pi}\lambda^{2}\frac{d}{d\lambda^{2}}\ln
Z_{X_{i}}$ (17)

where $X_{i}$ denotes the background field corresponding to the coupling
$\alpha_{i}$. Moreover, we can write

$\displaystyle Z_{X_{i}}$
$\displaystyle=1+\frac{\alpha_{i}}{4\pi}A_{i}+\sum_{j}\frac{\alpha_{i}\alpha_{j}}{(4\pi)^{2}}A_{ij}$
(18) $\displaystyle\beta_{i}$
$\displaystyle=\left(\frac{\alpha_{i}}{4\pi}\right)^{2}\left[\beta_{i}+\sum_{j}\frac{\alpha_{j}}{4\pi}\beta_{ij}\right]$
(19)

It is straightforward to obtain the relations, valid for IREG [13],

$\displaystyle\beta_{i}=4\lambda^{2}\frac{d}{d\lambda^{2}}A_{i};\quad\beta_{ij}=4\lambda^{2}\frac{d}{d\lambda^{2}}A_{ij}$
(20)

Finally, using Eqs. 9-11, yields

$\displaystyle\beta_{1}$
$\displaystyle=\frac{\alpha_{1}^{2}}{(4\pi)^{2}}\left[\frac{2}{5}+\frac{16}{3}n_{f}\right]+\frac{\alpha_{1}^{2}}{(4\pi)^{3}}\left[\frac{18\alpha_{1}}{25}+\frac{18\alpha_{2}}{5}-\frac{34\text{tr}\hat{T}}{5}-2\text{tr}\hat{B}-6\text{tr}\hat{L}\right.$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\left.+n_{f}\left(\frac{76\alpha_{1}}{15}+\frac{12\alpha_{2}}{5}+\frac{176\alpha_{3}}{15}\right)\right]$
(21) $\displaystyle\beta_{2}$
$\displaystyle=\frac{\alpha_{2}^{2}}{(4\pi)^{2}}\left[-\frac{86}{3}+\frac{16}{3}n_{f}\right]+\frac{\alpha_{2}^{2}}{(4\pi)^{3}}\left[\frac{6\alpha_{1}}{5}-\frac{518\alpha_{2}}{3}-6\text{tr}\hat{T}-6\text{tr}\hat{B}-2\text{tr}\hat{L}\right.$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\left.+n_{f}\left(\frac{4\alpha_{1}}{5}+\frac{196\alpha_{2}}{3}+16\alpha_{3}\right)\right]$
(22) $\displaystyle\beta_{3}$
$\displaystyle=\frac{\alpha_{3}^{2}}{(4\pi)^{2}}\left[-44+\frac{16}{3}n_{f}\right]+\frac{\alpha_{3}^{2}}{(4\pi)^{3}}\left[-408\alpha_{3}-8\text{tr}\hat{T}-8\text{tr}\hat{B}\right.$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\left.+n_{f}\left(\frac{22\alpha_{1}}{15}+6\alpha_{2}+\frac{304\alpha_{3}}{3}\right)\right]$
(23)

As can be seen, we recover known results obtained in the context of
dimensional methods in a minimal subtraction scheme [14]. This was expected
since in both cases a subtraction scheme independent of the mass was used,
which implies that up to two-loop order the gauge coupling $\beta$-function is
renormalization scheme independent [20].

##  4 Conclusion

In the last years we have witnessed an unprecedented amount of data collected
in the LHC. Since no clear sign of Beyond Standard Model (BSM) Physics has
emerged, it is required to know the Standard Model predictions with great
precision to clear disentangle the possible BSM contributions. In order to
tackle this task, innovative techniques have emerged, some of those proposing
distinct regularization methods which aim to stay in the physical dimension as
far as possible. Among those, Implicit Regularization is a promising
candidate, aiming to perform regularization at integrand level. It has been
shown to comply with unitarity, Lorentz invariance, causality, and abelian
gauge symmetry in general, while it complies with non-abelian gauge invariance
in working examples. In this contribution, we apply the technique for the
first time in the Standard Model, which is a spontaneously broken chiral non-
abelian theory. We have obtained the two-loop gauge coupling $\beta$-function,
reproducing the results known when applying dimensional regularizations. This
proves as a consistent test of the technique, since up to two-loop level the
gauge coupling $\beta$-function is universal to any mass-independent
subtraction scheme.

### Funding information

The author acknowledges support from Fundação para a Ciência e Tecnologia
(FCT) through the project CERN/FIS-COM/0035/2019, and the networking support
by the COST Action CA16201.

## References

  * [1] J. Davies, F. Herren, C. Poole, M. Steinhauser and A. E. Thomsen, Phys. Rev. Lett. 124, no.7, 071803 (2020) doi:10.1103/PhysRevLett.124.071803 [arXiv:1912.07624 [hep-ph]].
  * [2] A. Bednyakov and A. Pikelner, Phys. Rev. Lett. 127, no.4, 041801 (2021) doi:10.1103/PhysRevLett.127.041801 [arXiv:2105.09918 [hep-ph]].
  * [3] C. Poole and A. E. Thomsen, Phys. Rev. Lett. 123, no.4, 041602 (2019) doi:10.1103/PhysRevLett.123.041602 [arXiv:1901.02749 [hep-th]].
  * [4] C. Gnendiger, A. Signer, D. Stöckinger, A. Broggio, A. L. Cherchiglia, F. Driencourt-Mangin, A. R. Fazio, B. Hiller, P. Mastrolia and T. Peraro, et al. Eur. Phys. J. C 77, no.7, 471 (2017) doi:10.1140/epjc/s10052-017-5023-2 [arXiv:1705.01827 [hep-ph]].
  * [5] W. J. Torres Bobadilla, G. F. R. Sborlini, P. Banerjee, S. Catani, A. L. Cherchiglia, L. Cieri, P. K. Dhani, F. Driencourt-Mangin, T. Engel and G. Ferrera, et al. Eur. Phys. J. C 81, no.3, 250 (2021) doi:10.1140/epjc/s10052-021-08996-y [arXiv:2012.02567 [hep-ph]].
  * [6] A. M. Bruque, A. L. Cherchiglia and M. Pérez-Victoria, JHEP 08, 109 (2018) doi:10.1007/JHEP08(2018)109 [arXiv:1803.09764 [hep-ph]].
  * [7] A. C. D. Viglioni, A. L. Cherchiglia, A. R. Vieira, B. Hiller and M. Sampaio, Phys. Rev. D 94, no.6, 065023 (2016) doi:10.1103/PhysRevD.94.065023 [arXiv:1606.01772 [hep-th]].
  * [8] J. S. Porto, A. R. Vieira, A. L. Cherchiglia, M. Sampaio and B. Hiller, Eur. Phys. J. C 78, no.2, 160 (2018) doi:10.1140/epjc/s10052-018-5648-9 [arXiv:1706.01001 [hep-th]].
  * [9] Y. R. Batista, B. Hiller, A. Cherchiglia and M. Sampaio, Phys. Rev. D 98, no.2, 025018 (2018) doi:10.1103/PhysRevD.98.025018 [arXiv:1805.08225 [hep-th]].
  * [10] O. A. Battistel, A. L. Mota and M. C. Nemes, Mod. Phys. Lett. A 13, 1597-1610 (1998) doi:10.1142/S0217732398001686
  * [11] A. L. Cherchiglia, M. Sampaio and M. C. Nemes, Int. J. Mod. Phys. A 26, 2591-2635 (2011) doi:10.1142/S0217751X11053419 [arXiv:1008.1377 [hep-th]].
  * [12] D. C. Arias-Perdomo, A. Cherchiglia, B. Hiller and M. Sampaio, Symmetry 13, no.6, 956 (2021) doi:10.3390/sym13060956 [arXiv:2105.08085 [hep-th]].
  * [13] A. Cherchiglia, D. C. Arias-Perdomo, A. R. Vieira, M. Sampaio and B. Hiller, Eur. Phys. J. C 81, no.5, 468 (2021) doi:10.1140/epjc/s10052-021-09259-6 [arXiv:2006.10951 [hep-ph]].
  * [14] L. N. Mihaila, J. Salomon and M. Steinhauser, Phys. Rev. D 86, 096008 (2012) doi:10.1103/PhysRevD.86.096008 [arXiv:1208.3357 [hep-ph]].
  * [15] L. F. Abbott, Nucl. Phys. B 185, 189-203 (1981) doi:10.1016/0550-3213(81)90371-0
  * [16] A. Denner, G. Weiglein and S. Dittmaier, Nucl. Phys. B 440, 95-128 (1995) doi:10.1016/0550-3213(95)00037-S [arXiv:hep-ph/9410338 [hep-ph]].
  * [17] T. Hahn, Comput. Phys. Commun. 140, 418-431 (2001) doi:10.1016/S0010-4655(01)00290-9 [arXiv:hep-ph/0012260 [hep-ph]].
  * [18] T. Hahn and M. Perez-Victoria, Comput. Phys. Commun. 118, 153-165 (1999) doi:10.1016/S0010-4655(98)00173-8 [arXiv:hep-ph/9807565 [hep-ph]].
  * [19] H. H. Patel, Comput. Phys. Commun. 218, 66-70 (2017) doi:10.1016/j.cpc.2017.04.015 [arXiv:1612.00009 [hep-ph]].
  * [20] D. Espriu and R. Tarrach, Phys. Rev. D 25, 1073 (1982) doi:10.1103/PhysRevD.25.1073
  * [21] A. Cherchiglia, [arXiv:2106.14039 [hep-ph]].
  * [22] A. V. Bednyakov and A. F. Pikelner, Phys. Lett. B 762, 151-156 (2016) doi:10.1016/j.physletb.2016.09.007 [arXiv:1508.02680 [hep-ph]].


# Some remarks on rotation theorems
for complex polynomials

V.N. Dubinin111E-mail address<EMAIL_ADDRESS>
Far Eastern Federal University (FEFU), 8, Sukhanova Street, Vladivostok,
690950, Russia
Institute of Applied Mathematics, FEBRAS, 7, Radio Street, Vladivostok,
690041, Russia

###### Abstract

For any complex polynomial $P(z)=c_{0}+c_{1}z+...+c_{n}z^{n},\;\;c_{n}\not=0,$
having all its zeros in the unit disk $|z|\leq 1,$ we consider the behavior of
the function (arg$P(e^{i\theta}))^{\prime}_{\theta}$ when the real argument
$\theta$ changes. We give some sharp estimates of this function involving of
the values of $P(e^{i\theta}),$ arg$P(e^{i\theta})$ or the coefficients
$c_{k},\;k=0,1,n-1,n.$

_Keywords:_ complex polynomials, rotation theorems, inequalities, boundary
Schwarz lemma, rational functions.

MSC2010: 30A10; 30C10; 30C15; 30C80.

##  1 Introduction

Let all zeros of the polynomial

$P(z)=c_{0}+c_{1}z+...+c_{n}z^{n},\;c_{n}\not=0,$ (1)

lie in the unit disk $|z|\leq 1,\;c_{k}\in{\mathbb{C}},\;k=0,1,...,n,\;n\geq
1.$ In the theory of polynomials, inequality

${\rm Re}\;\frac{zP^{\prime}(z)}{P(z)}\geq\frac{n}{2},$ (2)

is well known, which is valid for all points $z$ on the circle $|z|=1$ that
are different from the zeros of the polynomial $P$. The equality in (2) takes
place if and only if all zeros of $P$ lie on $|z|=1$ (see, for example, [1, p.
439]). Inequality (2) can be interpreted as a rotation theorem for the complex
polynomial $P$ on the circle $|z|=1$:

$({\rm arg}P(z))^{\prime}_{\theta}\geq\frac{n}{2},$ (3)

$z=e^{i\theta},\;0\leq\theta\leq 2\pi.$ In a series of papers beginning with
[2], this author developed the geometric function theory approach to
inequalities for polynomials and rational functions. This is described in
detail in the survey article [6] and some subsequent publications of the
author. In particular, as an example of the application of the boundary
Schwarz lemma, the following strengthening of inequality (3) is given in [5]

$({\rm
arg}P(z))^{\prime}_{\theta}\geq\frac{n}{2}+\frac{|c_{n}|-|c_{0}|}{2(|c_{n}|+|c_{0}|)}\geq\frac{n}{2},$
(4)

$|z|=1,\;P(z)\not=0.$ Equality in (4) holds for polynomials $P$ with zeros
lying on the unit circle $|z|=1$ and for any $z,\;|z|=1,\;P(z)\not=0$ [5,
inequality (10)]. Note that if the polynomial $P(z)$ of the form (1) has no
zeros in the disk $|z|<1$, then the zeros of the polynomial $z^{n}P(1/z)$ lie
in the disk $|z|\leq 1$. Applying (4) to the last polynomial, we obtain the
inequality

$({\rm
arg}P(z))^{\prime}_{\theta}\leq\frac{n}{2}+\frac{|c_{n}|-|c_{0}|}{2(|c_{n}|+|c_{0}|)}\leq\frac{n}{2},$

for all $z$ on $|z|=1$ for which $P(z)\not=0$. Earlier [2], a weakened version
of inequality (4) was established, in which the right-hand side of (4) is
replaced by the quantity

$\frac{n}{2}+\frac{\sqrt{|c_{n}|}-\sqrt{|c_{0}|}}{2\sqrt{|c_{n}|}}\geq\frac{n}{2},$

[2, Theorem 4]. At present, the application of the boundary Schwarz lemma to
inequalities for complex polynomials had raised considerable interest (see,
for example, [11], [12], [14], [18] – [23], [25] and references therein ). As
regards inequality (4) directly, we note the following results. In the papers
of Govil, Kumar [10] and Rather, Dar, Iqbal [19] inequality (4) is generalized
in different directions and another proofs of the above inequality are given.
Gulzar, Zargar, Akhter [11], Hussain, Ahmad [12], and Milovanovi${\acute{\rm
c}}$, Mir, and Ahmad [18] used (4) to generalize and strengthen the well-known
inequalities for polynomials. In the paper of Rather, Dar, Iqbal [20] the
weakened version (4) was used.

In this note, we, first, refine inequality (4) taking into account the values
of the polynomial $P$ on the circle $|z|=1$. Further, the strengthening of (4)
with the involvement of the coefficients $c_{1}$ and $c_{n-1}$ of the
polynomial $P$ is obtained. In addition, we give an upper bound for the left-
hand side of (3) in the form of a finite increment theorem. The proofs of all
theorems are carried out in a unified manner and are based on the geometric
function theory approach [6]. In the final Section 3, some remarks are made
concerning the estimation of the value $({\rm arg}R(z))^{\prime}_{\theta}$ for
the rational function $R$.

##  2 Rotation theorems

For points $z=e^{i\theta}$ different from the zeros of a polynomial $P$, we
introduce the notation

$\Lambda(P,z)=2({\rm arg}P(e^{i\theta}))^{\prime}_{\theta}-n.$

In view of (3)

$\Lambda(P,z)\geq 0$

on the circle $|z|=1$. It is natural to pose the question of estimating the
rotation speed $P(z)$ when $z$ rotates on the circle $|z|=1$, depending on the
value of $P(z)$ itself. Some progress in the solution of this question is
given by

######  Theorem 1.

If all zeros of a polynomial $P$ of the form (1) lie in the disc $|z|\leq 1$,
then for any point $z$ a on the circle $|z|=1$ that is different from zero of
the polynomial $P$, inequality

$\Lambda(P,z)\geq\left|(\Lambda(P,z)+1)\;\frac{\overline{c}_{0}P(z)}{c_{n}z^{n}\overline{P(z)}}-1\right|$
(5)

holds. Equality in (5) at the point $z=1$ is attained for a polynomial $P$ of
the form

$P(z)=(z-\alpha_{1})\prod_{k=2}^{n}(z-\alpha_{k}),$ (6)

where $\alpha_{k},\;k=1,...,n,$ are arbitrary numbers that satisfy the
relations $|\alpha_{1}|<1,\;|\alpha_{k}|=1,\;\alpha_{k}\not=1,\;k=2,...,n.$

Proof. Let

$P(z)=c_{n}\prod_{k=1}^{n}(z-\alpha_{k}),$

and let $\alpha_{k}\not=1,\;k=1,...,n.$ In [8] (see also [9, Theorem 1]),
among other results, Goryainov obtained inequality

$\left|f^{\prime}(0)-\frac{1}{f^{\prime}(1)}\right|\leq
1-\frac{1}{f^{\prime}(1)},$ (7)

which is valid for any holomorphic map of the disk $|z|<1$ into itself,
normalized by conditions $f(0)=0,\;f(1)=1$ and $f^{\prime}(1)\not=\infty$.
Here $f(1)$ is the angular limit of $f$ when $z\to 1$, and $f^{\prime}(1)$ is
the angular derivative of the function $f$ at the point $z=1$. It is well
known that $f^{\prime}(1)\geq 1.$ Equality in (7) is attained for the function

$f^{*}(z)=z\frac{1-\overline{a}}{1-a}\frac{z-a}{1-\overline{a}z}$

for any $a,\;|a|<1.$ Consider the Blaschke product

$f(z)=\frac{\overline{c}_{n}}{c_{n}}\left[\prod_{k=1}^{n}\frac{1-\overline{\alpha}_{k}}{1-\alpha_{k}}\right]\frac{P(z)}{z^{n-1}\overline{P(1/\overline{z})}}=z\left[\prod_{k=1}^{n}\frac{1-\overline{\alpha}_{k}}{1-\alpha_{k}}\right]\prod_{k=1}^{n}\frac{z-\alpha_{k}}{1-\overline{\alpha}_{k}z}.$

If $|\alpha|=1$, then

$\frac{z-\alpha}{1-\overline{\alpha}z}=\frac{1}{\overline{\alpha}}\cdot\frac{z-\alpha}{1/\overline{\alpha}-z}=-\frac{1}{\overline{\alpha}},$

and for $|\alpha|<1$ the function

$w=\frac{z-\alpha}{1-\overline{\alpha}z}$

is a linear-fractional mapping of the disk $|z|<1$ onto the disk $|w|<1,$ such
that the circle $|z|=1$ turns to the circle $|w|=1$. Thus, the function $f$ is
holomorphic in the disk $|z|<1$ and satisfies the conditions:

$|f(z)|<1\;\mbox{when}\;|z|<1,\;f(0)=0,\;f(1)=1\;\mbox{and}\;f^{\prime}(1)\geq
1.$

Taking into account the geometric meaning of the derivative, we conclude that
at the points of the circle $|z|=1$, different from the zeros of the
polynomial $P$,

$|f^{\prime}(z)|=\frac{zf^{\prime}(z)}{f(z)}=\frac{z^{n}\overline{P(1/\overline{z})}}{P(z)}\cdot\frac{z^{n-1}P^{\prime}(z)\overline{P(1/\overline{z})}-P(z)[(n-1)z^{n-2}\overline{P(1/\overline{z})}-z^{n-3}\overline{P^{\prime}(1/\overline{z})}]}{(z^{n-1}\overline{P(1/\overline{z})})^{2}}=$
$=\frac{zP^{\prime}(z)}{P(z)}-n+1+\frac{\overline{P^{\prime}(z)}}{z\overline{P(z)}}=2{\rm
Re}\frac{zP^{\prime}(z)}{P(z)}-n+1$

is satisfied.

Goryainov’s inequality (7) applied to the function $f$ gives

$\left|\frac{c_{0}\overline{P(1)}}{\overline{c}_{n}P(1)}\left[2{\rm
Re}\frac{P^{\prime}(1)}{P(1)}-n+1\right]-1\right|\leq 2{\rm
Re}\frac{P^{\prime}(1)}{P(1)}-n.$

In general, if point $z_{0},\;|z_{0}|=1,$ is different from the zeros of the
polynomial $P$, then the point $z=1$ is different from the zeros of the
polynomial $\tilde{P}(z):=P(zz_{0})$. Applying what was proved above to the
polynomial $\tilde{P}$, we have

$\left|\frac{c_{0}\overline{P(z_{0})}}{\overline{c}_{n}\overline{z}^{n}_{0}P(z_{0})}\left[2{\rm
Re}\frac{z_{0}P^{\prime}(z_{0})}{P(z_{0})}-n+1\right]-1\right|\leq 2{\rm
Re}\frac{z_{0}P^{\prime}(z_{0})}{P(z_{0})}-n.$

The resulting inequality coincides with inequality (5) for $z=z_{0}.$ If now
$P(z)$ has the form (6), then

$f(z)=f^{*}(z),\;\mbox{where}\;a=\alpha_{1}.$

Therefore, equality is attained in (7). Hence the equality holds in (5). This
completes the proof of Theorem 1.

Inequality (5) is stronger than (4). Indeed,

$\Lambda(P,z)\geq 1-(\Lambda(P,z)+1)\left|\frac{c_{0}}{c_{n}}\right|$

is fulfilled from (5). Therefore,

$\Lambda(P,z)\geq\frac{|c_{n}|-|c_{0}|}{|c_{n}|+|c_{0}|},$

which is equivalent to inequality (4).

######  Theorem 2.

Let all zeros of a polynomial $P$ of the form (1) lie in the disc $|z|\leq 1$.
Then for any point $z$ on the circle $|z|=1$, different from the zeros of the
polynomial $P$, inequality

$\Lambda(P,z)\geq\frac{2(|c_{0}|-|c_{n}|)^{2}}{|c_{n}|^{2}-|c_{0}|^{2}+|\overline{c}_{n}c_{1}-c_{0}\overline{c}_{n-1}|}$
(8)

holds222In the case of $|c_{0}|=|c_{n}|,$ the expression on the right-hand
side of (8) means zero.. Equality in (8) is attained for polynomials $P$ with
zeros on the unit circle $|z|=1$ for any $z,\;|z|=1,\;P(z)\not=0$.

Proof. The equality $|c_{0}|=|c_{n}|$ holds if and only if all zeros of the
polynomial $P$ lie on the circle $|z|=1$. Therefore, the case of equality in
(8) is obvious. Further, we consider that $|c_{0}|\not=|c_{n}|$. We need the
following inequality (7) from [4]:

$|f^{\prime}(z)|\geq
1+\frac{2(1-|f^{\prime}(0)|)^{2}}{1-|f^{\prime}(0)|^{2}+|f^{\prime\prime}(0)/2|}.$
(9)

Here $f$ is a holomorphic map of the disk $|z|<1$ into itself, normalized by
the conditions: $f(0)=0,\;|f^{\prime}(0)|\not=1$ and $z$ is an arbitrary point
on the circle $|z|=1$, in which the derivative $f^{\prime}(z)$ exists and
$|f(z)|=1$. In [16], Mercer presented a direct proof of (9) using Rogosinski’s
lemma (see also [17, inequality (15)]). Consider the Blaschke product

$f(z)=\frac{P(z)}{z^{n-1}\overline{P(1/\overline{z})}}=\frac{c_{n}z}{\overline{c}_{n}}\prod_{k=1}^{n}\frac{z-\alpha_{k}}{1-\overline{\alpha}_{k}z},$

where $P(z)=c_{n}\prod_{k=1}^{n}(z-\alpha_{k}).$ As above, we see that $f$ is
a holomorphic map of the disk $|z|<1$ into itself, $f(0)=0,$ and at each point
$z$ on the circle $|z|=1$, nonzero of the polynomial $P$, $|f(z)|=1,$ and
there is a derivative $f^{\prime}(z)$. Moreover,

$f^{\prime}(0)=\frac{c_{n}}{\overline{c}_{n}}\prod_{k=1}^{n}(-\alpha_{k})=\frac{c_{0}}{\overline{c}_{n}},$

so $|f^{\prime}(0)|\not=1.$ Note that the derivative is

$\left(\prod_{k=1}^{n}\frac{z-\alpha_{k}}{1-\overline{\alpha}_{k}z}\right)^{\prime}=\left(\prod_{k=1}^{n}\frac{z-\alpha_{k}}{1-\overline{\alpha}_{k}z}\right)\sum_{k=1}^{n}\left(\frac{1}{z-\alpha_{k}}+\frac{\overline{\alpha}_{k}}{1-\overline{\alpha}_{k}z}\right).$

In view of this and Vieta’s formulas, we have

$f^{\prime\prime}(0)=2\frac{c_{n}}{\overline{c}_{n}}\prod_{k=1}^{n}(-\alpha_{k})\sum_{k=1}^{n}\left(\frac{1}{-\alpha_{k}}+\overline{\alpha}_{k}\right)=\frac{2c_{0}}{\overline{c}_{n}}\left[\frac{c_{1}}{c_{0}}-\frac{\overline{c}_{n-1}}{\overline{c}_{n}}\right].$

Substituting the found values of the derivatives into (9), taking into account
the calculations of the $|f^{\prime}(z)|$ in the proof of Theorem 1, we arrive
at inequality (8). The theorem is proved.

As noted by Merser [17, Remark 3.2] for the function $f$ from (9)

$|f^{\prime\prime}(0)|\leq 2(1-|f^{\prime}(0)|^{2})$

is fulfilled. This attracts

$|c_{1}\overline{c}_{n}-c_{0}\overline{c}_{n-1}|\leq|c_{n}|^{2}-|c_{0}|^{2}.$

Hence it is easy to see that inequality (8) is stronger (4).

######  Theorem 3.

Let all zeros of a polynomial $P$ of degree n lie in the disc $|z|\leq 1$.
Suppose that for some point $z_{0}$ on the circle $|z|=1$, the arc of this
circle

$\gamma(z_{0},\alpha):=\\{z:\;|z|=1,\;|{\rm arg}z-{\rm
arg}z_{0}|<\alpha\\},\;\;0<\alpha<\pi,$

does not contain the zeros of the polynomial $P$ and the increment of the
value

$2\;{\rm arg}P(z)-n\;{\rm arg}z$

along any curve on the arc $\gamma(z_{0},\alpha)$ with an end point at $z_{0}$
does not exceed in absolute value $\beta,\;0<\beta<\pi$. Then

$\Lambda(P,z_{0})\leq\frac{{\rm tg}\frac{\beta}{2}}{{\rm
tg}\frac{\alpha}{2}}.$ (10)

Equality in (10) is attained for a polynomial $P$ of the form

$P(z)=c_{n}z(z-\alpha_{2})...(z-\alpha_{n}),$ (11)

$|\alpha_{k}|=1,\;\alpha_{k}\not=1,\;k=2,...,n,$ point $z_{0}=1$, any
$\alpha,\;0<\alpha<\pi,$ for which the $\gamma(1,\alpha)$ does not contain
$\alpha_{k},\;k=2,...,n$ and $\beta=\alpha$.

Proof. We can assume that the point $z_{0}=1$. In [7], among other results,
inequality

$|f^{\prime}(1)|\leq\frac{{\rm tg}\frac{\beta}{2}}{{\rm tg}\frac{\alpha}{2}}.$
(12)

was established. Here f is a holomorphic self-mapping of the unit disk
$|z|<1$, $f(\gamma(1,\alpha))\subset\gamma(1,\beta)$ for some $\alpha$ and
$\beta$, $0<\alpha<\pi,\;0<\beta<\pi,$ and $f$ has an angular limit $f(1)=1$
and a finite angular derivative $f^{\prime}(1)$. Equality in (12) is attained
for the function $f(z)\equiv z$ and $\alpha=\beta$. We put as above

$P(z)=c_{n}\prod_{k=1}^{n}(z-\alpha_{k}).$

Let us show that the Blaschke product

$f(z):=\frac{\overline{c}_{n}}{c_{n}}\left[\prod_{k=1}^{n}\frac{1-\overline{\alpha}_{k}}{1-\alpha_{k}}\right]\frac{P(z)}{z^{n}\overline{P(1/\overline{z})}}=\left[\prod_{k=1}^{n}\frac{1-\overline{\alpha}_{k}}{1-\alpha_{k}}\right]\prod_{k=1}^{n}\frac{z-\alpha_{k}}{1-\overline{\alpha}_{k}z}.$

satisfies the conditions for inequality (12). Indeed, $f$ is a holomorphic
function mapping the disk $|z|<1$ into itself. In addition, $f$ is
differentiable on the arc $\gamma(1,\alpha),\;f(1)=1$ and $|f(z)|=1$ on
$\gamma(1,\alpha)$. In view of the hypothesis of Theorem 3, the increment of
the argument of the function $f$ along any curve $\gamma$ on
$\gamma(1,\alpha)$ with an end point at the point $z=1$ does not exceed in
absolute value $\beta$:

$|\Delta_{\gamma}{\rm arg}f(z)|=|\Delta_{\gamma}(2{\rm arg}P(z)-n\;{\rm
arg}z)|\leq\beta.$

Hence, $f(\gamma(1,\alpha))\subset\gamma(1,\beta).$ Following the calculations
carried out in the proof of Theorem 1, we see that

$|f^{\prime}(1)|=2{\rm Re}\frac{P^{\prime}(1)}{P(1)}-n.$

Inequality (12) gives

$2{\rm Re}\frac{P^{\prime}(1)}{P(1)}-n\leq\frac{{\rm tg}\frac{\beta}{2}}{{\rm
tg}\frac{\alpha}{2}},$

which is equivalent to (10) for $z_{0}=1$. If now $P(z)$ has the form (11),
then $f(z)\equiv z.$ In this case, $f^{\prime}(1)=1$ and

$|\Delta_{\gamma}(2{\rm arg}P(z)-n\;{\rm arg}z)|=|\Delta_{\gamma}{\rm
arg}z|\leq\alpha,\;\;\gamma\subset\gamma(1,\alpha).$

and you can take $\beta=\alpha$. This gives equality in (12) and hence
equality in (10). The theorem is proved.

##  3 On rational functions

In conclusion we consider an application of the function theory to
inequalities for rational functions with prescribed poles:

$R(z)=\frac{P(z)}{\prod_{k=1}^{n}(z-a_{k})},$

where $P(z)$ is an algebraic polynomial of degree $m$ and
$|a_{k}|>1,\;k=1,...,n.$ In some problems, extremal functions are related to
the Blaschke product

$B(z)=\prod_{k=1}^{n}\frac{1-\overline{a}_{k}z}{z-a_{k}},$

which is in general defined for any system of poles
$(a_{1},...,a_{n}),\;|a_{k}|\not=1,\;k=1,...,n.$

The following analogue of the polynomial inequality (3) is known. If a
rational function $R$ has exactly $m$ zeros (counting multiplicities)
belonging to the disk $|z|\leq 1$, then

$({\rm arg}R(z))^{\prime}_{\theta}\geq\frac{1}{2}(m-n+({\rm
arg}B(z))^{\prime}_{\theta})$ (13)

for all points on the circle $|z|=1$ different from zeros of $R$. If all zeros
of $R$ lie in the complement to the disk $|z|<1$, then the reverse inequality
is valid:

$({\rm arg}R(z))^{\prime}_{\theta}\leq\frac{1}{2}(m-n+({\rm
arg}B(z))^{\prime}_{\theta}).$ (14)

Equality in (13), (14) holds for $R(z)=\alpha B(z)+\beta$ with
$|\alpha|=|\beta|$. The proof of inequalities (13) and (14) can be found in
[15, Lemma 4], [3, Lemma 3 and Theorem 4], and in [3] the geometric function
theory was first applied to such a range of problems. Using various versions
of the boundary Schwarz lemma, Wali and Shah [24], [26] strengthened (13),
(14) in different directions. Kalmykov [13] recently proved two- and three-
point distortion theorems for rational functions that generalize some known
results on Bernstein-type inequalities for polynomials and rational functions.
The rational functions under study have either majorants or restrictions on
location of their zeros. The proofs in [13] are based on the new version of
the Schwarz Lemma and univalence condition for holomorphic functions.

In this regard, it can be assumed that the application of the methods of
geometric function theory will lead to new rotation theorems for rational
functions with prescribed poles, in particular, to theorems generalizing
Theorems 1–3 of this article.

Funding: This work was supported by the Russian Basic Research Fund [grant
number 20-01-00018].

## References

  * [1] P. Borwein, T. Erd′elyi, Polynomials and polynomial inequalities, Grad. Texts in Math., 161, Springer-Verlag, New York, 1995, x+480 pp.
  * [2] V.N. Dubinin, Distortion theorems for polynomials on a circle, Sb. Math., 191(12) (2000), 1797–1807.
  * [3] V.N. Dubinin, On an application of conformal maps to inequalities for rational functions, Izv. Math., 66(2) (2002), 285–297.
  * [4] V.N. Dubinin, On the Shwarz inequality on the boundary for functions regular in the disk, J. Math. Sci. (N. Y.), 122(6) (2004), 3623–3629.
  * [5] V.N. Dubinin, Applications of the Schwarz lemma to inequalities for entire functions with constraints on zeros, J. Math. Sci. (N. Y.), 143(3) (2007), 3069–3076.
  * [6] V.N. Dubinin, Methods of geometric function theory in classical and modern problems for polynomials, Russian Math. Surveys, 67(4) (2012), 599–684.
  * [7] V.N. Dubinin, On holomorphic self-mappings of the unit disk, Siberian Electronic Mathematical Reports, 16 (2019), 1633–1639.
  * [8] V. V. Goryainov, Holomorphic mappings of the unit disc into itself with two fixed points, Sb. Math., 208(3) (2017), 360–376.
  * [9] V.V. Goryainov, Some inequalities for holomorphic self-maps of the unit disc with two fixed points, Contemporary Mathematics, 699 (2017), 129–136. http://dx.doi.org/10.1090/conm/699/14085.
  * [10] N.K. Govil, P. Kumar, On sharpening of an inequality of Tur$\acute{\rm a}$n, Appl. Anal. Discrete Math., 13 (2019), 711–720.
  * [11] M.H. Gulzar, B. A. Zargar, R. Akhter, Inequalities for the polar derivative of a polynomial, The Journal of Analysis, 28 (2020), 923–929. https://doi.org/10.1007/s41478-020-00222-4.
  * [12] A. Hussain, A. Ahmad, Generalizations of some Bernstein-type inequalities for the polar derivative of a polynomial, ANNALI DELL’UNIVERSITA’ DI FERRARA 66 (2020), 371–379. https://doi.org/10.1007/s11565-020-00341-7.
  * [13] S.I. Kalmykov, About multipoint distortion theorems for rational functions, Siberian Mathematical Journal, 61(1) (2020), 85–94. https://doi.org/10.1134/S0037446620010073.
  * [14] P. Kumar, On the Inequalities Concerning Polynomials, Complex Analysis and Operator Theory. Complex Analysis, 14(65) (2020). https://doi.org/10.1007/s11785-020-01023-0.
  * [15] X. Li, R.N. Mohapatra and R.S. Rodriguez, Bernstein-type inequalities for rational functions with prescribed poles, J. London. Math. Soc., 51(3) (1995), 523–531.
  * [16] P.R. Mercer, Boundary Schwarz inequalities arising from Rogosinski’s Lemma, Journal of Classical Analysis, 12(2) (2018), 93–97. https://doi.org/10.7153/jca-2018-12-08.
  * [17] P.R. Mercer, An improved Schwarz Lemma at the boundary, Open Math., 16 (2018), 1140–1144. https://doi.org/10.1515/math-2018-0096.
  * [18] G.V. Milovanovi$\acute{\rm c}$, A. Mir, and A. Ahmad, Certain estimates of Tur$\acute{\rm a}$n’s-type for the maximum modulus of the polar derivative of a polynomial, PUBLICATIONS DE L’INSTITUT MATH$\acute{\rm E}$MATIQUE, Nouvelle s$\acute{\rm e}$rie, 108(122) (2020), 121–130. https://doi.org/10.2298/PIM2022121M.
  * [19] N.A. Rather, I. Dar, A. Iqbal, Some inequalities for polynomials with restricted zeros. Annali dell’Universit$\acute{\rm a}$ di Ferrara. Sezione 7: Scienze matematiche, 2020. https://doi.org/ 10.1007/s11565-020-00353-3.
  * [20] N.A. Rather, I. Dar, A. Iqbal, Some extensions of a theorem of paul Tur$\acute{\rm a}$n concerning polynomials, Kragujevac Journal of Mathematics, 46(6) (2022), 969–979.
  * [21] N.A. Rather, I.A. Dar, Some applications of the Boundary Schwarz lemma for Polynomials with restricted zeros, Applied Mathematics E-Notes, 20 (2020), 422–431. http://www.math.nthu.edu.tw/amen/.
  * [22] N.A. Rather, I. Dar, and A. Iqbal, Inequalities for the derivative of polynomials with restricted zeros, Korean J. Math., 28(4) (2020), 931–942. http://dx.doi.org/10.11568/kjm.2020.28.4.931.
  * [23] N.A. Rather, A. Iqbal and I. Dar, Inequalities concerning the polar derivatives of polynomials with restricted zeros, Nonlinear Functional Analysis and Applications 24(4) (2019), 813–826.
  * [24] S.L. Wali, W.M. Shah, Applications of the Schwarz lemma to inequalities for rational functions with prescribed poles. J. Anal., 25(1) (2017), 43–53. https://doi.org/10.1007/s41478-016-0025-2.
  * [25] S.L. Wali, A. Liman, Refinements of some polynomial inequalities with restricted zeros. J. Anal., 25(2) (2017), 291–299. https://doi.org/10.1007/s41478-017-0057-2.
  * [26] S.L. Wali, W.M. Shah, Some applications of Dubinin’s lemma to rational functions with prescribed poles, Journal of Mathematical Analysis and Applications, 450(1) (2017), 769–779. https://doi.org/10.1016/j.jmaa.2017.01.069.


# A Closer Look at Knowledge Distillation with Features, Logits, and Gradients

Yen-Chang Hsu1, James Smith2, Yilin Shen1, Zsolt Kira2, Hongxia Jin1

###### Abstract

Knowledge distillation (KD) is a substantial strategy for transferring learned
knowledge from one neural network model to another. A vast number of methods
have been developed for this strategy. While most method designs a more
efficient way to facilitate knowledge transfer, less attention has been put on
comparing the effect of knowledge sources such as features, logits, and
gradients. This work provides a new perspective to motivate a set of knowledge
distillation strategies by approximating the classical KL-divergence criteria
with different knowledge sources, making a systematic comparison possible in
model compression and incremental learning. Our analysis indicates that logits
are generally a more efficient knowledge source and suggests that having
sufficient feature dimensions is crucial for the model design, providing a
practical guideline for effective KD-based transfer learning.

## Introduction

Knowledge distillation transfers knowledge from one neural network model to
another by matching different information sources from the models: logits
(Hinton, Vinyals, and Dean 2014), features (Romero et al. 2015), or gradients
(Srinivas and Fleuret 2018). It has found wide applications in such areas as
model compression (Hinton, Vinyals, and Dean 2014; Wang and Yoon 2020; Ba and
Caruana 2014; Urban et al. 2017), incremental learning (Li and Hoiem 2017; Lee
et al. 2019), privileged learning (Lopez-Paz et al. 2016), adversarial defense
(Papernot et al. 2016), and learning with noisy data (Li et al. 2017). This
knowledge transfer usually follows the teacher-student scheme, in that a high-
performing teacher model provides knowledge sources for a student model to
match.

Which part of the teacher provides a more informative knowledge source for a
student to distill? Heo et al. (2019b) shows the features are more effective;
Tian, Krishnan, and Isola (2020) observes that logits are generally better,
but matching the pairwise correlation with features outperforms logits.
Recently, Kim et al. (2021) reports that logits achieve a better result when
using L2 loss instead of KL-divergence. This puzzling inconsistency is due to
the differences of both optimization criteria and the knowledge sources used
in different methods, making a fair comparison across knowledge sources
impossible.

This work provides a systematic approach to analyzing the above puzzle,
providing a closer look at transferring knowledge in features, logits, and
gradients. Specifically, we propose a new perspective to reinterpret the
classical KD objective, using Taylor expansion to approximate the KL-
divergence with different knowledge sources. This novel perspective leads to a
generalized divergence that allows us to use logits and features under the
same optimization criteria, while the gradients provide information about the
importance of each feature for distillation. We therefore based on the
mathematical insight as a unified framework, instantiating varied methods to
distill features, logits, and gradients. Interestingly, when we instantiate
our variants with the simplest design choices, the resulting methods are
similar to previous knowledge transfer techniques. The similarity demonstrates
the generalizability of our framework. Nevertheless, there are still nuanced
differences between our variants and prior methods to ensure a fair comparison
across knowledge sources, discussed later.

We explore two aspects that have never been jointly investigated with
knowledge sources. The first is observing whether the trend of effectiveness
is consistent across different transfer learning tasks. The two most popular
KD tasks are included: model compression and incremental learning. The second
aspect is to investigate the impact of model design and identify the key
factors affecting knowledge sources’ effectiveness. In summary, this work
provides the following contributions and findings:

  * •

We provide a new perspective to interpret the classical KD, showing that a
single framework can describe the distillation with different knowledge
sources. To the best of our knowledge, this perspective has not been
presented.

  * •

The new perspective leads to a new strategy for improving feature-based KD by
weighing the importance of features based on gradients from the teacher.

  * •

Our systematic comparison shows that logits generally is the more effective
knowledge source, followed by the features weighted by gradients and the plain
features. This trend is consistent in both model compression and incremental
learning.

  * •

We further use a new metric with a normalized basis to analyze the factors
that affect the above trend, pointing out that the feature size of the
penultimate layer plays a crucial role in impacting different knowledge
sources’ effectiveness.

## Rethinking Knowledge Distillation

Our observation starts with the generic knowledge distillation criteria
$\mathcal{L}_{KD}$ for classification (Hinton, Vinyals, and Dean 2014). The
criteria includes cross-entropy loss $\mathcal{L}_{CE}$ for a student model to
learn from ground-truth label $y^{*}$, and $D_{\mathrm{KL}}$ for minimizing
the difference between the predicted class distribution $p^{s}$ (from the
student) and $p^{t}$ (from the teacher). The latter term makes the knowledge
transfer happen, for which the coefficient $\lambda$ controls the intensity:

$\mathcal{L}_{KD}=\mathcal{L}_{CE}(p^{s},y^{*})+\lambda
D_{\mathrm{KL}}(p^{t}||p^{s}),$ (1)
$D_{\mathrm{KL}}(p^{t}||p^{s})=\sum_{y}p^{t}_{y}\log
p^{t}_{y}-\sum_{y}p^{t}_{y}\log p^{s}_{y}$ (2)

The next step is to expand $D_{\mathrm{KL}}$. Here, we are more interested in
how intermediate outputs $z$ from the teacher ($z^{t}=g^{t}(x)$) and student
($z^{s}=g^{s}(x)$) affect $D_{\mathrm{KL}}$; therefore, we treat the $z$ as
the only variable in $p^{t}_{y}=f(y;z^{t})$ and $p^{s}_{y}=f(y;z^{s})$,
leaving the parameters (if any) of the softmax-based classifier $f$ as
constants and the same $f$ is used for both the teacher and student. By taking
the Taylor expansion around $z^{t}$ for the second term of equation 2 and
using the notation $dz=z^{s}-z^{t}$, $D_{\mathrm{KL}}$ becomes:

$\displaystyle D_{\mathrm{KL}}(p^{t}||p^{s})=$ $\displaystyle\sum p^{t}\log
p^{t}-\sum p^{t}\log p^{t}$ $\displaystyle-dz^{T}\sum p^{t}\frac{d}{dz}\log
p^{t}$ $\displaystyle-\frac{1}{2}dz^{T}(\sum p^{t}\frac{d^{2}}{dz^{2}}\log
p^{t})dz+\epsilon$ (3) $\displaystyle=$ $\displaystyle\frac{1}{2}dz^{T}\sum
p^{t}(\frac{d}{dz}\log p^{t})(\frac{d}{dz}\log p^{t})^{T}dz+\epsilon$ (4)

In equation 3, the first-order term is zero, while the second-order has a form
of Fisher information matrix $F(z^{t})$ at its middle. Details of derivation
are available in the Supplementary. The above equations omit the $y$ for
conciseness. Lastly, by ignoring the higher-order term $\epsilon$,
$D_{\mathrm{KL}}$ can be rewritten as:

$D_{\mathrm{KL}}(p^{t}||p^{s})\approx\frac{1}{2}(z^{s}-z^{t})^{T}F(z^{t})(z^{s}-z^{t})$
(5)

Although equation 5 has a simple quadratic form, it provides two key insights:

  1. 1.

Minimizing the difference between the student’s and teacher’s intermediate
representations reduces the KL-divergence.

  2. 2.

The Fisher information $F(z^{t})$, which leverages the gradients regarding the
teacher’s intermediate representation, provides a weighting mechanism for the
importance of features.

## The Generalized Divergence

This section discusses the simplest design choices to turn equation 5 into a
framework that is easy to implement and instantiate its variants. There are
two empirical considerations in applying equation 5, both stemming from the
challenges of computing $F(z^{t})$. The first is its $O(|z|^{2})$ complexity.
The computation could be expensive when $z$ has a large dimension (e.g.the
flattened feature map from a convolution neural network based on image
inputs). Here we follow the common simplification used by EWC (Kirkpatrick et
al. 2017) and Adam (Kingma and Ba 2015) in which only the diagonal of Fisher
information matrix is considered, reducing the complexity to $O(|z|)$. Second,
marginalizing over $y$ for accumulating the gradients could be time-consuming;
therefore, there is a need to use an alternative loss function to collect
gradients of $z^{t}$.

To adopt the above considerations, we define a generalized divergence $D_{G}$,
making equation 5 a special case of the generalized form:

$D_{G}(z^{t},z^{s})=\alpha(z^{s}-z^{t})^{T}W(z^{t})(z^{s}-z^{t})$ (6)

The coefficient $\alpha$ is a scaling factor that can be absorbed by
$\lambda$. The $W(z^{t})$ is still an n-by-n weighting matrix like $F(z^{t})$,
given $z^{t}$’s dimension $n$. The calculation of $W$ is still based on
gradients from the teacher, specifically:

$W(z^{t})=diag((\frac{d}{dz}\mathcal{L}_{*})(\frac{d}{dz}\mathcal{L}_{*})^{T})$
(7)

The function $diag$ casts all off-diagonal elements to be zero. The
$\mathcal{L}_{*}$ is $\log p^{t}$ when computing the Fisher information. Here
we consider two design choices for $\mathcal{L}_{*}$ to avoid the need of
marginalizing over $y$:

$\displaystyle\mathcal{L}_{E}$ $\displaystyle=\log p^{t}_{y^{*}}$ (8)
$\displaystyle\mathcal{L}_{H}$
$\displaystyle=\frac{1}{k}\sum_{y=1}^{k}(l^{t}_{y})^{2}$ (9)

$\mathcal{L}_{E}$ is related to the empirical Fisher which requires knowing
the ground-truth class $y^{*}$. $\mathcal{L}_{H}$ is a heuristic criteria by
using the mean-squared logits ($l_{y}$) over $k$ classes. $\mathcal{L}_{H}$
does not require labels, but captures the gradients that lead to a large
change in logits. $\mathcal{L}_{H}$ is useful when the student (and its
training data) has a different set of classes from the teacher, which is a
case that $\mathcal{L}_{E}$ is not applicable.

Overall, our full criteria $\mathcal{L}_{KD-G}$ with the generalized
divergence has the form:

$\mathcal{L}_{KD-G}=\mathcal{L}_{CE}(p^{s},y^{*})+\lambda D_{G}(z^{t},z^{s})$
(10)

Note $z$ can be the logits or features. Besides, the knowledge in the
teacher’s gradients are transferred to the student via $W(z^{t})$. Therefore
$\mathcal{L}_{KD-G}$ provides a unified framework for comparing the
effectiveness of each knowledge source by instantiating it in different ways.
This is one of the main contributions of this paper, as our formulation allows
an explicit fair comparison across knowledge sources within a unified
framework. Below we elaborate the cases when $z$ is features or logits, and
additionally extend the discussion to a model’s parameters, which is a popular
knowledge source in the incremental learning.

Figure 1: The schematic illustration for the variants that use features. The
example vectors demonstrate how the gradients weigh the features. A darker
color in the example vector indicates the feature is more influential. Note
that the operation $a^{2}$ means all elements in the vector are squared. SE is
the abbreviation of squared error.

### Features

When $z$ is features, there are rich choices for computing $W$. One can use
$\mathcal{L}_{E}$ when the ground-truth class $y^{*}$ is available. Such a
condition is usually true in the case of model compression. In contrast, there
may not be a valid label for the teacher model (as we will see in the later
section on incremental learning). In such cases, we use $\mathcal{L}_{H}$ for
$W$. Furthermore, $W$ can also be an identity matrix, which reduces $D_{G}$ to
a simple squared error. This case leads to the same optimization objective
used in FitNet (Romero et al. 2015). Therefore, we include $W=\mathbb{I}$ as
one of the variants in our framework while simplifying its deployment with a
principled normalization to make it amicable to KD tasks other than model
compression, described in the later sections.

### Logits

When $z$ is logits, its Hessian matrix (as appears in equation 3) obtained
through $\mathcal{L}_{H}$ is an identity matrix ($W=\mathbb{I}$ with a scaling
factor that can be absorbed by the coefficient $\lambda$), indicating that a
weighting mechanism based on gradients is redundant for this case. As a
result, $D_{G}$ reduces to a simple squared error with logits. The result has
the same formula as when computing the KL-divergence between the output
probabilities with the logits being divided by a large temperature (Hinton,
Vinyals, and Dean 2014; Kim et al. 2021), indicating this simple variant is
worth more attention in a comparative study.

### Parameters

In an alternative perspective, which switches the focus from intermediate
representations $z$ to model’s parameters $\theta^{t}$ (from $g^{t}$) and
$\theta^{s}$ (from $g^{s}$), a derivation similar to equation 3 will lead to
the parameter-based regularization. Note that our process derives it from a
different motivation, but reaches the same formulation of EWC (Kirkpatrick et
al. 2017), an importance-weighted parameter regularization method:

$\mathcal{L}_{EWC}=\mathcal{L}_{CE}(p^{s},y^{*})+\frac{\lambda}{2}(\theta^{s}-\theta^{t})^{T}F(\theta^{t})(\theta^{s}-\theta^{t})$
(11)

Although $\mathcal{L}_{KD-G}$ and $\mathcal{L}_{EWC}$ share a similar form,
they have four fundamental differences: (1) $\mathcal{L}_{KD-G}$ is for
features or logits, while $\mathcal{L}_{EWC}$ is only for parameters. (2) Our
derivation starts from KL-divergence between categorical distributions, while
EWC starts from the normal approximation of the posterior for parameters. (3)
EWC requires the teacher and student model to be the same. As a result, EWC is
not considered as a KD method and is not applicable to model compression. (4)
$\mathcal{L}_{KD-G}$ achieves significantly better results than
$\mathcal{L}_{EWC}$ in the task-incremental learning (shown in the experiment
section).

### The Four Instantiations

Based on the above discussion, we create four $\mathcal{L}_{KD-G}$
instantiations to enable a systematic comparison. The first two (WeightedE
Features-SE and WeightedH Features-SE) are novel KD methods derived from our
framework, while the latter two (Features-SE and Logits-SE) have close
alternatives in previous works. The illustration of the first three variants
is shown in Figure 1. Their names are listed below with a description of how
their $W$ is implemented for the $\mathcal{L}_{KD-G}$ (Note: SE means squared
error):

  * •

WeightedE Features-SE: $W$ uses $\mathcal{L}_{E}$

  * •

WeightedH Features-SE: $W$ uses $\mathcal{L}_{H}$

  * •

Features-SE: $W=\mathbb{I}$

  * •

Logits-SE: $W=\mathbb{I}$

## Experiments

### Model Compression (MC)

Model compression is the primary task where knowledge distillation techniques
are applied heavily. In this case, the student model has a smaller capacity
than the teacher yet is asked to match the teacher’s outputs. If the target
for matching contains a great deal of information that is not crucial for a
downstream task, the student is more likely to waste its limited capacity on
matching unimportant information, capping the student’s ability to reach the
teacher’s performance. This argument gives an intuition of why one can expect
the weighted features to perform better than an unweighted one. A similar
argument may apply to the logits since the linear layer before the logits
imposes weights on the features. This section provides empirical support for
the arguments with our unified framework.

#### Applying our framework

Two empirical considerations need to be addressed for applying equation 10 to
model compression. The first one is that a large numerical range can result
from $D_{G}$, potentially making the optimization unstable. In the KD training
procedure, the student is initialized randomly and is directly optimized from
scratch with the knowledge distillation criteria. The random student model
could make the squared difference between $z^{s}$ and $z^{t}$ unbounded. This
issue can be addressed by normalization. Specifically, we make $\hat{z}^{s}$
and $\hat{z}^{t}$ unit vectors:

$\hat{z}^{t}=\frac{z^{t}}{||z^{t}||},\hat{z}^{s}=\frac{z^{s}}{||z^{s}||}$ (12)

The second empirical consideration is the mismatched dimensions between
$z^{s}$ and $z^{t}$ when they are features. This case happens when the student
and teacher have different types of neural network architectures or when the
student has a smaller model width. We add a linear transformation $r$ on the
outputs of $g^{s}(x)$ to match the teacher’s dimension:

$z^{s}=r(g^{s}(x))$ (13)

The parameters of $r$ are also optimized by the customized $D_{G}$ for this
section:

$D_{G}^{MC}=(\hat{z}^{s}-\hat{z}^{t})^{T}W(z^{t})(\hat{z}^{s}-\hat{z}^{t})$
(14)

Note that $r$ is only involved during training and is removed from testing;
thus, the student model’s design has no dependency on $r$. Additionally, $r$
is only used when z is features, since the logits layer always has the same
dimensionality (number of classes) between the teacher and student.

Teacher model |  | resnet32x4 | vgg13 | WRN-40-2 | WRN-40-2 | resnet110
---|---|---|---|---|---|---
Student model | . | resnet8x4 | vgg8 | WRN-16-2 | WRN-40-1 | resnet32
Teacher acc. |  | 79.42 | 74.64 | 75.61 | 75.61 | 74.31
Student acc. |  | 72.50 | 70.36 | 73.26 | 71.98 | 71.14
Method | Knowledge source |  |  |  |  |
HKD (Hinton 2014) | L | 73.33 $\pm$ 0.25 | 72.98 $\pm$ 0.19 | 74.92 $\pm$ 0.28 | 73.54 $\pm$ 0.20 | 73.08 $\pm$ 0.18
FitNet (Romero 2015) | F | 73.50 $\pm$ 0.28 | 71.02 $\pm$ 0.31 | 73.58 $\pm$ 0.32 | 72.24 $\pm$ 0.24 | 71.06 $\pm$ 0.13
AT (Zagoruyko 2017) | F | 73.44 $\pm$ 0.19 | 71.43 $\pm$ 0.09 | 74.08 $\pm$ 0.25 | 72.77 $\pm$ 0.10 | 72.31 $\pm$ 0.08
FT (Kim 2018) | F | 72.86 $\pm$ 0.12 | 70.58 $\pm$ 0.08 | 73.25 $\pm$ 0.20 | 71.59 $\pm$ 0.15 | 72.37 $\pm$ 0.31
SP (Tung 2019) | F+ | 72.94 $\pm$ 0.23 | 72.68 $\pm$ 0.19 | 73.83 $\pm$ 0.12 | 72.43 $\pm$ 0.27 | 72.69 $\pm$ 0.41
CRD (Tian 2020) | F+ | 75.51 $\pm$ 0.18 | 73.94 $\pm$ 0.22 | 75.48 $\pm$ 0.09 | 74.14 $\pm$ 0.22 | 73.48 $\pm$ 0.13
CRD+HKD | L + F+ | 75.46 $\pm$ 0.25 | 74.29 $\pm$ 0.12 | 75.64 $\pm$ 0.21 |  74.38 $\pm$ 0.11 |  73.75 $\pm$ 0.24
(A) Features-SE | F | 74.83 $\pm$ 0.15 | 73.08 $\pm$ 0.26 | 75.35 $\pm$ 0.20 | 74.06 $\pm$ 0.43 | 73.13 $\pm$ 0.18
(B) WeightedE Features-SE | F + G | 75.20 $\pm$ 0.15 | 73.57 $\pm$ 0.22 | 75.33 $\pm$ 0.30 | 73.87 $\pm$ 0.39 | 73.37 $\pm$ 0.26
(C) Logits-SE | L | 76.29 $\pm$ 0.16 |  74.45 $\pm$ 0.19 | 75.67 $\pm$ 0.25 |  74.38 $\pm$ 0.16 | 73.21 $\pm$ 0.31
(B+C) | L + F + G |  76.66 $\pm$ 0.22 | 74.28 $\pm$ 0.14 |  75.74 $\pm$ 0.20 | 74.29 $\pm$ 0.11 | 73.48 $\pm$ 0.24
Table 1:  Test accuracy (%) of student models on CIFAR100 of many distillation methods. The teacher-student pair is from the same architecture family. The variants from our framework are labeled with (A), (B), (C), and (B+C). The results of other methods are inherited from CRD (a state-of-the-art method) since we strictly follow its training and testing procedure and can reproduce its reported results. The type of knowledge sources is categorized by: logits (L), features (F), higher-order features (i.e., instance-wise correlation, F+), and gradients (G).  Teacher model |  | resnet32x4 | resnet32x4 | WRN-40-2 | ResNet50 | ResNet50 | vgg13
---|---|---|---|---|---|---|---
Student model |  | ShuffleNetV1 | ShuffleNetV2 | ShuffleNetV1 | vgg8 | MobileNetV2 | MobileNetV2
Teacher acc. |  | 79.42 | 79.42 | 75.61 | 79.34 | 79.34 | 74.64
Student acc. |  | 70.5 | 71.82 | 70.5 | 70.36 | 64.6 | 64.6
Method | Source |  |  |  |  |  |
HKD (Hinton 2014) | L | 74.07 $\pm$ 0.19 | 74.45 $\pm$ 0.27 | 74.83 $\pm$ 0.17 | 73.81 $\pm$ 0.13 | 67.35 $\pm$ 0.32 | 67.37 $\pm$ 0.32
FitNet (Romero 2015) | F | 73.59 $\pm$ 0.15 | 73.54 $\pm$ 0.22 | 73.73 $\pm$ 0.32 | 70.69 $\pm$ 0.22 | 63.16 $\pm$ 0.47 | 64.14 $\pm$ 0.50
AT (Zagoruyko 2017) | F | 71.73 $\pm$ 0.31 | 72.73 $\pm$ 0.09 | 73.32 $\pm$ 0.35 | 71.84 $\pm$ 0.28 | 58.58 $\pm$ 0.54 | 59.40 $\pm$ 0.20
FT (Kim 2018) | F | 71.75 $\pm$ 0.20 | 72.50 $\pm$ 0.15 | 72.03 $\pm$ 0.16 | 70.29 $\pm$ 0.19 | 60.99 $\pm$ 0.37 | 61.78 $\pm$ 0.33
SP (Tung 2019) | F+ | 73.48 $\pm$ 0.42 | 74.56 $\pm$ 0.22 | 74.52 $\pm$ 0.24 | 73.34 $\pm$ 0.34 | 68.08 $\pm$ 0.38 | 66.30 $\pm$ 0.38
CRD (Tian 2020) | F+ | 75.11 $\pm$ 0.32 | 75.65 $\pm$ 0.10 | 76.05 $\pm$ 0.14 | 74.30 $\pm$ 0.14 | 69.11 $\pm$ 0.28 | 69.73 $\pm$ 0.42
CRD+HKD | L + F+ | 75.12 $\pm$ 0.35 | 76.05 $\pm$ 0.09 | 76.27 $\pm$ 0.29 | 74.58 $\pm$ 0.27 | 69.54 $\pm$ 0.39 |  69.94 $\pm$ 0.05
(A) Features-SE | F | 75.47 $\pm$ 0.13 | 75.61 $\pm$ 0.25 | 75.91 $\pm$ 0.33 | 73.78 $\pm$ 0.31 | 68.38 $\pm$ 0.80 | 68.24 $\pm$ 0.13
(B) WE Features-SE | F + G | 76.24 $\pm$ 0.29 | 76.14 $\pm$ 0.23 | 76.30 $\pm$ 0.15 | 74.42 $\pm$ 0.19 | 69.29 $\pm$ 0.21 | 68.36 $\pm$ 0.30
(C) Logits-SE | L | 75.90 $\pm$ 0.47 |  76.64 $\pm$ 0.18 | 76.36 $\pm$ 0.24 | 75.02 $\pm$ 0.35 | 69.95 $\pm$ 0.18 | 68.99 $\pm$ 0.52
(B+C) | L + F + G |  76.99 $\pm$ 0.43 | 76.56 $\pm$ 0.35 |  76.91 $\pm$ 0.29 |  75.08 $\pm$ 0.28 |  70.58 $\pm$ 0.26 | 69.79 $\pm$ 0.33
Table 2:  Test accuracy (%) of student models on CIFAR100 with transfer across very different teacher and student architectures.  Teacher model |  | ResNet50 | ResNet34
---|---|---|---
Student model |  | ResNet18 | ResNet18
Method | Knowledge source | Top 1 | Top 5 | Top 1 | Top 5
Teacher | - | 62.22 | 83.52 | 57.86 | 80.39
Student (baseline) | - | 53.15 | 75.97 | 53.15 | 75.97
HKD (Hinton 2014) | L | 58.26 $\pm$ 0.34 | 80.92 $\pm$ 0.25 | 58.54 $\pm$ 0.24 | 81.07 $\pm$ 0.12
AT (Zagoruyko 2017) | F | 55.67 $\pm$ 0.32 | 79.40 $\pm$ 0.22 | 56.04 $\pm$ 0.24 | 79.09 $\pm$ 0.35
SP (Tung 2019) | F+ | 55.48 $\pm$ 0.33 | 78.34 $\pm$ 0.31 | 55.28 $\pm$ 0.32 | 78.46 $\pm$ 0.29
PKT (Passalis 2018) | F+ | 54.34 $\pm$ 0.20 | 77.49 $\pm$ 0.24 | 54.45 $\pm$ 0.42 | 77.48 $\pm$ 0.21
RKD (Park 2019) | F+ | 54.30 $\pm$ 0.44 | 77.31 $\pm$ 0.43 | 54.37 $\pm$ 0.22 | 77.40 $\pm$ 0.30
(A) Features-SE | F | 56.86 $\pm$ 0.21 | 79.44 $\pm$ 0.38 | 56.57 $\pm$ 0.40 | 79.43 $\pm$ 0.22
(B) WeightedE Features-SE | F + G | 57.15 $\pm$ 0.29 | 79.55 $\pm$ 0.24 | 57.08 $\pm$ 0.17 | 79.54 $\pm$ 0.36
(C) Logits-SE | L | 58.95 $\pm$ 0.34 | 81.14 $\pm$ 0.12 | 58.53 $\pm$ 0.32 | 80.80 $\pm$ 0.31
(B+C) | L + F + G |  59.30 $\pm$ 0.20 |  81.46 $\pm$ 0.15 |  58.79 $\pm$ 0.15 |  81.09 $\pm$ 0.06
Table 3:  Test accuracy (%) of student models on Tiny-ImageNet. The type of
knowledge sources is categorized by: logits (L), features (F), higher-order
features (i.e., instance-wise correlation, F+), and gradients (G). The value
is averaged with 5 repeats.

#### Implementation Details

The procedure of the experiments here closely follows the model compression
benchmark (Tian, Krishnan, and Isola 2020). The experiments include a large
number of combinations between the teacher and student models. The teacher-
student pairs include the models from the same architectural family but with
different depth or width, and the models from different architectures that
result in different sizes of features. The list of neural network
architectures includes ResNet (He et al. 2016), WideResNet (Zagoruyko and
Komodakis 2016), VGG (Simonyan and Zisserman 2015), MobileNet (Howard et al.
2017), and ShuffleNet (Zhang et al. 2018). We use the feature map output of
the last convolutional block for the features and the last linear layer’s
outputs for the logits. When the student’s feature map is different from the
teacher’s, the transformation function $r$ resizes the feature maps spatially
with PyTorch’s pooling operation (Paszke et al. 2019). Then, the student’s
number of channels is linearly projected by a 1x1-conv layer ($r$) to match
the teacher’s channel number. Lastly, the resulting feature map is flattened
for $z$. For consistency, we include $r$ in all our variants that use features
(but not logits), regardless of whether the feature maps have the same size or
not.

For learning with the CIFAR100 dataset, the models have an initial learning
rate of 0.05, decayed by 0.1 every 30 epochs after the first 150 epochs until
it reaches 240. For MobileNetV2, ShuffleNetV1 and ShuffleNetV2, the initial
learning rate is 0.01 as suggested by (Tian, Krishnan, and Isola 2020). All
the methods use SGD with a momentum of 0.9 and a batch size of 64. In short,
we follow the benchmark settings (Tian 2020) and use the same teacher models
provided to conduct all the experiments, ensuring a fair comparison between
all methods.

Hyper-parameter selection could have a profound effect on most knowledge
distillation-based model compression. We follow the benchmark protocol (Tian,
Krishnan, and Isola 2020), which selects the hyper-parameters based on only
one teacher-student pair (we use resnet32x4/resnet8x4), then apply it to all
other cases. Therefore, a method has to be robust to the hyper-parameters
choice to perform well in all cases.

We make a step further to align the hyperparameter $\lambda$ used in Features-
SE and WeightedE Features-SE. This can be achieved by normalizing the
$W(z^{t})$’s outputs to make its diagonal to have a mean of one (like the
identity matrix $\mathbb{I}$) and unit variance. This normalization makes the
features have an expected importance of 1 no matter how $W$ is computed,
leaving the gradient-based weighting the only factor to affect the performance
between the two cases. As a result, we can use the coefficient
$\lambda=\lambda_{F}=3$ in all cases for features. Additionally,
$\lambda=\lambda_{L}=15$ when we use logits. Lastly, we add an extra setting
by combining features and logits. It leads to the case of ”B+C” in Tables 1
and 2 with the custmoized $D_{G}$:

$\displaystyle D_{G-BC}^{MC}$
$\displaystyle=\lambda_{L}(\hat{l}_{s}-\hat{l}_{t})^{T}(\hat{l}_{s}-\hat{l}_{t})$
$\displaystyle+\lambda_{F}(\hat{z}_{s}-\hat{z}_{t})^{T}W_{E}(z_{t})(\hat{z}_{s}-\hat{z}_{t})$
(15)

Note that $\hat{l}$ is the normalized logits and $\hat{z}$ is the normalized
features. We use $\lambda=1$ for $D_{G-BC}^{MC}$.

#### MC Benchmark Results

First of all, we emphasize that our focus is on evaluating our general
formulation and keeping their instantiations in the simplest form, revealing
the intrinsic trend of KD with different knowledge sources. Our goal here is
not beating state-of-the-art (e.g.CRD), although our methods (B and C in
Tables 1 and 2) achieve comparable or better performance. In our comparison,
Tables 1 and 2 statistically agree with the arguments made at the beginning of
the MC section: First, the weighted features (B) performs better than the
unweighted one (A) in 9 out of 11 cases. Second, the logits (C) performs
better than features (A and B) in 9 out of 11 cases. This result suggests the
rank of ”logits $>$ weighted features $>$ plain features” in their KD
efficiency. Besides, our variants (A, B, C) outperform most of the previous KD
methods, showing that our methods are efficient in extracting out the
knowledge, and making the observed ranks more representative.

#### Experiment on ImageNet

We use a larger and harder dataset for replicating the experiments of Table 1
with the standard ResNet. Table 3 uses the subset of ImageNet images
(TinyImageNet 2017) , showing the same trend of ”logits $>$ weighted features
$>$ plain features”, and combing all (B+C) leads to the best result. Tiny-
ImageNet has 200 classes sub-sampled from ImageNet. Each class has 500
training images and testing images with size 64x64. All the methods in Table 3
use the same hyperparameters as in Tables 1 and 2. The training configuration
is similar to Tables 1, except that the initial learning rate is 0.1, and is
decayed by the factor of 0.1 at 50% and 75% of total (100) epochs. The weight-
decay is 0.005. The models (ResNet-18/34/50) are the default models defined in
the PyTorch. All students are trained from scratch (with random
initialization). The teacher’s weights were initialized with an
ImageNet(full)-pretrained model in PyTorch model zoo, then is fine-tuned with
Tiny-ImageNet.

Figure 2:  The analysis of feature size versus sources of knowledge. The
performance gain due to model capacity has been subtracted. This figure
highlights the changes in the ranking of knowledge sources. The raw value of
each bar is averaged with 5 repeats and is available in Supplementary.

#### Key factor analysis

This section investigates the factor that affects the ranking of knowledge
sources. The clue comes from Tables 1 and 2, in which Table 2 has a larger
difference on (B-A) and (C-B) than in Table 1. One possible factor is that the
students in Table 1 generally have a smaller feature size (e.g., resnet32: 64
channels; WRN-40-1: 64; WRN-40-2: 128) than the students in Table 2 (e.g.,
MobileNetV2: 160; ShuffleNetV1: 960; ShuffleNetV2: 464). To investigate
whether the student’s feature size has a substantial impact, we vary the
feature size within the same model architecture. However, changing feature
size inevitably changes the model’s capacity, and a larger capacity helps the
student more easily match the teacher. Thus, comparing the absolute accuracy
can not make a conclusive analysis.

We therefore make an additional contribution in addressing the above issue for
the analysis, applying two strategies to minimize the capacity effect: (1)
only the number of channels for the last convolutional block is changed, and
(2) we propose a recovered performance ratio to measure the relative
performance gain by subtracting the performance of a vanilla student (trained
without a teacher). The vanilla student has its performance increased along
with the capacity; therefore, subtracting its accuracy ($Acc^{s}_{vanilla}$)
excludes the capacity effect. The recovered performance ratio (RPR) is
computed by:

$RPR=(Acc^{s}_{KD}-Acc^{s}_{vanilla})/(Acc^{t}_{vanilla}-Acc^{s}_{vanilla}).$
(16)

Figure 2 uses RPR to examine various student feature sizes. The teacher and
student models are resnet32x4 and resnet8x4, correspondingly. The student
model’s last convolutional block has its width (number of channels) configured
to be between 64 to 1024. The result confirms feature size’s impact on the
ranking:

  * •

A larger student feature (e.g.$\geq 256$) leads to a consistent ranking of
”logits $>$ weighted features $>$ plain features”. A smaller feature (e.g.64)
breaks the trend.

We additionally subtract a stronger baseline (HKD (Hinton, Vinyals, and Dean
2014)) by replacing equation 16’s $Acc^{s}_{vanilla}$ with $Acc^{s}_{HKD}$ in
Supplementary. Its trend is still the same as Figure 2, providing extra
support for the observation. The result suggests the design guideline:

  * •

Having a larger student feature size can benefit KD. Note that using larger
features does not conflict with the goal of model compression. ShuffleNet
(Zhang et al. 2018) in Table 2 is a positive case that has a relatively
smaller model while still having a sufficient feature size.

### Incremental Learning (IL)

Incremental learning is a problem setting where KD is often applied. The
setting has its model exposed to a sequence of tasks. These tasks have
differences in either their input distribution, label distribution, or both.
The model has no access to the training data of previous tasks when learning a
new task. The shift of distributions among tasks introduces a significant
interference to the learned parameters, largely undermining previous tasks’
performance. This phenomenon is called catastrophic forgetting. A popular
strategy is to regularize the model’s parameters to mitigate the forgetting,
reducing drift from its previously learned parameters. However, when the
regularization is too strong, the model will not have sufficient plasticity to
learn a new task well. Thus, there is a trade-off between minimizing
forgetting and maximizing plasticity. A good trade-off strategy keeps
important knowledge while allowing the less important ones to be overwritten
by the new tasks. The parameter-based regularization dominates this line of
strategy. Previous works (Kirkpatrick et al. 2017; Zenke, Poole, and Ganguli
2017; Aljundi et al. 2018) select important parameters based on gradients and
avoid those parameters from changing too much. It is a setting that
complements model compression, providing an excellent opportunity to compare
not only features, logits, and gradients, but also the model parameters for
transferring the knowledge.

#### Applying our framework

We consider the task-incremental learning (Hsu et al. 2018; van de Ven and
Tolias 2019) setting for our experiments. This setting has exclusive sets of
classes in a sequence of classification tasks. The model learns each
classification task sequentially with only access to the training data of the
current task. During the learning curriculum, the model regularly adds an
output head (as a linear layer) for a new classification task, while
inheriting all the parts learned in the previous tasks. As a result, the model
has multiple heads (one for each task), and it requires $D_{G}$ to sum over
all previous tasks’ output heads to regularize the model drifting.
Specifically, $D_{G}$ is customized by:

$D_{G-logits}^{IL}=\sum_{j}(l^{s}_{[j]}-l^{t}_{[j]})^{T}(l^{s}_{[j]}-l^{t}_{[j]})$
(17)
$D_{G-features}^{IL}=\sum_{j}(z^{s}-z^{t})^{T}W_{[j]}(z^{t})(z^{s}-z^{t})$
(18)
$W_{[j]}(z^{t})=diag((\frac{d}{dz}\frac{1}{k}\sum_{y=1}^{k}(l^{t}_{[j],y})^{2})(\frac{d}{dz}\frac{1}{k}\sum_{y=1}^{k}(l^{t}_{[j],y})^{2})^{T})$
(19)

The $l_{[j]}$ is the logits from the $j$th task. The regularization term sums
over the tasks except the current task $T_{current}$ (i.e.task index
$j=\\{1..T_{current}-1\\}$). Note that when $T_{current}=2$, everything here
(equations 17 to 19) falls back to equations 6 and 7. The only difference is
that the current task’s labels are out-of-scope for the previous (teacher)
model. In other words, this is the case that $y^{*}$ is not valid for the
teacher; therefore, equation 19 uses $\mathcal{L}_{H}$ to collect the
gradients. In this section, our Logits-SE uses $D_{G-logits}^{IL}$, WeightedH
Features-SE uses $D_{G-features}^{IL}$, and Features-SE has its
$W_{[j]}(z^{t})=\mathbb{I}$. We additionally add three EWC variants for the
comparison. SI (Zenke, Poole, and Ganguli 2017) accumulates the gradients
along the optimization trajectory to replace the $F(\theta^{t})$ in equation
11. MAS (Aljundi et al. 2018) uses $\mathcal{L}_{H}$ to compute the gradients
for a weighting matrix similar to our $W$. The L2 sets its
$F(\theta^{t})=\mathbb{I}$ in equation 11. All experiments here closely follow
the implementation and evaluation protocol described in the popular benchmark
(Hsu 2018). More details are in Supplementary.

Method | Reg | S-CIFAR10 | S-CIFAR100
---|---|---|---
Upper-bound | - | 97.6 $\pm$ 0.1 | 84.3 $\pm$ 0.4
Baseline | - | 63.5 $\pm$ 1.7 | 30.5 $\pm$ 0.6
L2 | P | 74.2 $\pm$ 0.6 | 51.7 $\pm$ 1.3
EWC (Kirkpatrick 2017) | P | 84.4 $\pm$ 2.1 | 61.1 $\pm$ 1.4
SI (Zenke 2017) | P | 79.1 $\pm$ 1.3 | 64.8 $\pm$ 1.0
MAS (Aljundi 2018) | P | 78.3 $\pm$ 0.7 | 64.8 $\pm$ 0.8
Features-SE | F | 77.4 $\pm$ 3.6 | 70.5 $\pm$ 0.6
WeightedH Features-SE | F | 93.3 $\pm$ 0.5 | 73.7 $\pm$ 0.6
Logits-SE | L |  95.3 $\pm$ 0.2 |  78.3 $\pm$ 0.3
Table 4: The averaged classification accuracy of task-incremental learning.
The S-CIFAR10/100 datasets split their classes into 5 tasks for a model to
learn sequentially. The regularization targets (Reg) are parameters (P),
features (F), and logits (L). The upper-bound performance is from non-
incremental learning (learn 5 tasks simultaneously).

#### IL Benchmark Results

Table 4 shows that the effectiveness of knowledge sources is ranked: logits
(L) $>$ features (F) $>$ parameters (P). Although the number of classes
imposes a very different difficulty to the problem (2 classes per task in
S-CIFAR10 versus 20 in S-CIFAR100), the methods noted with ”P” performs
significantly worse than ”F” and ”L” on both datasets, suggesting that
regularizing the outputs generally strikes a better balance between forgetting
and plasticity. Furthermore, the comparison between WeightedH Features-SE
versus Features-SE shows that having the squared error weighted by gradients
is very helpful. Both above observations are consistent with the trends in
model compression.

## Related Work

We categorize KD methods by their knowledge sources and discuss the most
related works. First, the features-based methods generally make a small
student match a large teacher’s features without selection. One exception is
(Heo et al. 2019a), which selects useful features by using margin ReLU with a
per-feature threshold. However, its heuristic nature is significantly
different from our gradient-driven approach (i.e., Weighted Features-SE ). It
is also worth noting that our Features-SE is closely related FitNet (Romero et
al. 2015) and FT (Kim, Park, and Kwak 2018). FitNet, FT, and our method align
the dimension of the features between the teacher and student, then use
squared error to match the features. However, FitNet adds a convolutional
regressor (with non-linearity) for the student to do the matching, and trains
the student with squared error loss and cross-entropy loss in two separate
stages. In FT, it uses two small auto-encoders to transform the features from
both the teacher and student. Therefore, both FitNet and FT have a more
complicated design than our linear transformation function $r$ and our one-
stage training procedure. Second, in previous gradient-based methods (Srinivas
and Fleuret 2018; Zagoruyko and Komodakis 2017), they directly match the
teacher’s and student’s Jacobian, which requires double backpropagation to
optimize its loss function. In contrast, our weighted features-SE does not use
Jacobian, avoiding the heavy overhead in optimization. Lastly, the logits-
based methods (Hinton, Vinyals, and Dean 2014; Li and Hoiem 2017) have been
discussed in previous sections. Other logits-based strategies such as early
stopping (Cho and Hariharan 2019) and teacher assistant (Mirzadeh et al. 2020)
are orthogonal approaches to our work and can be applied jointly.

## Conclusion

We present a new perspective that can utilize different knowledge sources
under a unified KD framework. This framework leads to a new KD method that
prioritizes the distillation of important features based on gradients, and
provides a new justification on how simple squared error approximates the
classical KD criteria. We instantiate our framework based on the type of
knowledge sources utilized, finding that logits is generally more efficient
than features, while gradients can help the latter. Furthermore, our analysis
points out that a student’s feature size is impactful to the KD efficiency. We
hope the new insights, new methods, and new findings will inspire more works
in this field.

## References

  * Aljundi et al. (2018) Aljundi, R.; Babiloni, F.; Elhoseiny, M.; Rohrbach, M.; and Tuytelaars, T. 2018\.  Memory Aware Synapses: Learning what (not) to forget.  In _ECCV_.
  * Ba and Caruana (2014) Ba, J.; and Caruana, R. 2014.  Do deep nets really need to be deep?  In _Advances in neural information processing systems_ , 2654–2662.
  * Cho and Hariharan (2019) Cho, J. H.; and Hariharan, B. 2019.  On the efficacy of knowledge distillation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 4794–4802.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Heo et al. (2019a) Heo, B.; Kim, J.; Yun, S.; Park, H.; Kwak, N.; and Choi, J. Y. 2019a.  A comprehensive overhaul of feature distillation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 1921–1930.
  * Heo et al. (2019b) Heo, B.; Lee, M.; Yun, S.; and Choi, J. Y. 2019b.  Knowledge transfer via distillation of activation boundaries formed by hidden neurons.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Hinton, Vinyals, and Dean (2014) Hinton, G.; Vinyals, O.; and Dean, J. 2014.  Distilling the knowledge in a neural network.  _NIPS Deep Learning Workshop_.
  * Howard et al. (2017) Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_.
  * Hsu (2018) Hsu, Y.-C. 2018.  https://github.com/GT-RIPL/Continual-Learning-Benchmark.
  * Hsu et al. (2018) Hsu, Y.-C.; Liu, Y.-C.; Ramasamy, A.; and Kira, Z. 2018.  Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines.  In _NeurIPS Continual learning Workshop_.
  * Kim, Park, and Kwak (2018) Kim, J.; Park, S.; and Kwak, N. 2018.  Paraphrasing complex network: Network compression via factor transfer.  In _Advances in neural information processing systems_.
  * Kim et al. (2021) Kim, T.; Oh, J.; Kim, N.; Cho, S.; and Yun, S.-Y. 2021.  Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation.  In _IJCAI_.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A method for stochastic optimization.  _ICLR_.
  * Kirkpatrick et al. (2017) Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_.
  * Lee et al. (2019) Lee, K.; Lee, K.; Shin, J.; and Lee, H. 2019.  Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 312–321.
  * Li et al. (2017) Li, Y.; Yang, J.; Song, Y.; Cao, L.; Luo, J.; and Li, L.-J. 2017.  Learning from noisy labels with distillation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 1910–1918.
  * Li and Hoiem (2017) Li, Z.; and Hoiem, D. 2017.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(12): 2935–2947.
  * Lopez-Paz et al. (2016) Lopez-Paz, D.; Bottou, L.; Schölkopf, B.; and Vapnik, V. 2016.  Unifying distillation and privileged information.  _International Conference on Learning Representations_.
  * Mirzadeh et al. (2020) Mirzadeh, S. I.; Farajtabar, M.; Li, A.; Levine, N.; Matsukawa, A.; and Ghasemzadeh, H. 2020.  Improved knowledge distillation via teacher assistant.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Papernot et al. (2016) Papernot, N.; McDaniel, P.; Wu, X.; Jha, S.; and Swami, A. 2016.  Distillation as a defense to adversarial perturbations against deep neural networks.  In _2016 IEEE Symposium on Security and Privacy (SP)_ , 582–597. IEEE.
  * Park et al. (2019) Park, W.; Kim, D.; Lu, Y.; and Cho, M. 2019.  Relational knowledge distillation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 3967–3976.
  * Passalis and Tefas (2018) Passalis, N.; and Tefas, A. 2018.  Learning Deep Representations with Probabilistic Knowledge Transfer.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Kopf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; and Chintala, S. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 32_ , 8024–8035. Curran Associates, Inc.
  * Romero et al. (2015) Romero, A.; Ballas, N.; Kahou, S. E.; Chassang, A.; Gatta, C.; and Bengio, Y. 2015\.  Fitnets: Hints for thin deep nets.  _ICLR_.
  * Simonyan and Zisserman (2015) Simonyan, K.; and Zisserman, A. 2015.  Very deep convolutional networks for large-scale image recognition.  _ICLR_.
  * Srinivas and Fleuret (2018) Srinivas, S.; and Fleuret, F. 2018.  Knowledge Transfer with Jacobian Matching.  In _International Conference on Machine Learning_ , 4723–4731.
  * Tian (2020) Tian, Y. 2020.  https://github.com/HobbitLong/RepDistiller.
  * Tian, Krishnan, and Isola (2020) Tian, Y.; Krishnan, D.; and Isola, P. 2020.  Contrastive Representation Distillation.  In _International Conference on Learning Representations_.
  * TinyImageNet (2017) TinyImageNet. 2017.  https://www.kaggle.com/c/tiny-imagenet.
  * Tung and Mori (2019) Tung, F.; and Mori, G. 2019.  Similarity-preserving knowledge distillation.  In _Proceedings of the IEEE International Conference on Computer Vision_.
  * Urban et al. (2017) Urban, G.; Geras, K. J.; Kahou, S. E.; Aslan, O.; Wang, S.; Caruana, R.; Mohamed, A.; Philipose, M.; and Richardson, M. 2017.  Do deep convolutional nets really need to be deep and convolutional?  _ICLR_.
  * van de Ven and Tolias (2019) van de Ven, G. M.; and Tolias, A. S. 2019.  Three scenarios for continual learning.  _arXiv preprint arXiv:1904.07734_.
  * Wang and Yoon (2020) Wang, L.; and Yoon, K.-J. 2020.  Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks.  _arXiv preprint arXiv:2004.05937_.
  * Zagoruyko and Komodakis (2016) Zagoruyko, S.; and Komodakis, N. 2016.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_.
  * Zagoruyko and Komodakis (2017) Zagoruyko, S.; and Komodakis, N. 2017.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  _ICLR_.
  * Zenke, Poole, and Ganguli (2017) Zenke, F.; Poole, B.; and Ganguli, S. 2017.  Continual Learning Through Synaptic Intelligence.  In _International Conference on Machine Learning_.
  * Zhang et al. (2018) Zhang, X.; Zhou, X.; Lin, M.; and Sun, J. 2018.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 6848–6856.

A Closer Look at Knowledge Distillation with Features, Logits, and Gradients
-Supplementary Materials-

Paper ID 2350
---

## Detailed Derivations of Equation 3

This section provides the detailed steps that are not included in the main
text for deriving equation 5 from equation 3.

(1) The first-order term in equation 3 is zero:

$\displaystyle-dz^{T}\sum_{y}p^{t}_{y}\frac{d}{dz}\log p^{t}_{y}$
$\displaystyle=-dz^{T}\sum_{y}\frac{d}{dz}p^{t}_{y}$
$\displaystyle=-dz^{T}(\frac{d}{dz}\sum_{y}p^{t}_{y})$ $\displaystyle=0$

(2) The second-order term in equation 3 has a form of Fisher information
matrix $F(z^{t})$ at its middle:

$\displaystyle-$
$\displaystyle\frac{1}{2}dz^{T}(\sum_{y}p^{t}_{y}\frac{d^{2}}{dz^{2}}\log
p^{t}_{y})dz$
$\displaystyle=-\frac{1}{2}dz^{T}\sum_{y}p^{t}_{y}\frac{d}{dz}(\frac{1}{p^{t}_{y}}\frac{dp^{t}_{y}}{dz})dz$
$\displaystyle=-\frac{1}{2}dz^{T}\sum_{y}p^{t}_{y}\left[\frac{1}{p^{t}_{y}}\frac{d^{2}p^{t}_{y}}{dz^{2}}-(\frac{1}{p^{t}_{y}}\frac{dp^{t}_{y}}{dz})(\frac{1}{p^{t}_{y}}\frac{dp^{t}_{y}}{dz})^{T}\right]dz$
$\displaystyle=-\frac{1}{2}dz^{T}\left[\sum_{y}\frac{d^{2}p^{t}_{y}}{dz^{2}}-\sum_{y}p^{t}_{y}(\frac{1}{p^{t}_{y}}\frac{dp^{t}_{y}}{dz})(\frac{1}{p^{t}_{y}}\frac{dp^{t}_{y}}{dz})^{T}\right]dz$
$\displaystyle=-\frac{1}{2}dz^{T}\left[\frac{d^{2}}{dz^{2}}\sum_{y}p^{t}_{y}-\sum_{y}p^{t}_{y}(\frac{1}{p^{t}_{y}}\frac{dp^{t}_{y}}{dz})(\frac{1}{p^{t}_{y}}\frac{dp^{t}_{y}}{dz})^{T}\right]dz$
$\displaystyle=-\frac{1}{2}dz^{T}\left[-\sum_{y}p^{t}_{y}(\frac{1}{p^{t}_{y}}\frac{dp^{t}_{y}}{dz})(\frac{1}{p^{t}_{y}}\frac{dp^{t}_{y}}{dz})^{T}\right]dz$
$\displaystyle=\frac{1}{2}dz^{T}\left[\sum_{y}p^{t}_{y}(\frac{d}{dz}\log
p^{t}_{y})(\frac{d}{dz}\log p^{t}_{y})^{T}\right]dz$
$\displaystyle=\frac{1}{2}dz^{T}F(z^{t})dz$

## Additional analysis of Figure 2

Figure A: The analysis of feature size versus sources of knowledge. The
performance gain due to model capacity has been more aggressively removed by
subtracting the performance of HKD (Hinton, Vinyals, and Dean 2014) in the RPR
(equation 16). The ranking of methods is still consistent with Figure 2.

Figure A has the same experiment as in Figure 2, but the performance gain due
to model capacity has been more aggressively removed by subtracting the
performance of HKD (Hinton, Vinyals, and Dean 2014) in the RPR (equation 16).
This figure has a trend similar to Figure 2, strengthening the argument that
the student’s feature size has a crucial impact on knowledge distillation.
Table A provides the raw accuracy used in both Figures 2 and A.

#channel | Baseline | HKD | (A) Features-SE | (B) Weighted Feature-SE | (C) Logits-SE | (B+C)
---|---|---|---|---|---|---
64 | 68.12 | 67.87 $\pm$ 0.18 | 69.79 $\pm$ 0.36 | 68.96 $\pm$ 0.47 | 68.77 $\pm$ 0.14 | 68.67 $\pm$ 0.49
128 | 70.55 | 71.34 $\pm$ 0.43 | 72.52 $\pm$ 0.25 | 72.88 $\pm$ 0.12 | 73.82 $\pm$ 0.28 | 73.73 $\pm$ 0.24
256 | 72.50 | 73.33 $\pm$ 0.25 | 74.83 $\pm$ 0.15 | 75.20 $\pm$ 0.15 | 76.29 $\pm$ 0.16 | 76.66 $\pm$ 0.22
512 | 73.62 | 74.98 $\pm$ 0.17 | 76.22 $\pm$ 0.10 | 76.66 $\pm$ 0.13 | 77.47 $\pm$ 0.25 | 78.29 $\pm$ 0.20
1024 | 74.40 | 75.94 $\pm$ 0.24 | 77.38 $\pm$ 0.20 | 77.64 $\pm$ 0.19 | 78.82 $\pm$ 0.08 | 79.47 $\pm$ 0.25
Table A: The raw accuracy (CIFAR100) used in Figures 2 and A. The baseline is the student model trained without knowledge distillation.  Teacher model | resnet32x4 | vgg13 | WRN-40-2 | WRN-40-2 | resnet110
---|---|---|---|---|---
Student model | resnet8x4 | vgg8 | WRN-16-2 | WRN-40-1 | resnet32
Teacher size (M) | 7.43 | 9.46 | 2.26 | 2.26 | 1.74
Student size (M) | 1.23 | 3.96 | 0.70 | 0.57 | 0.47
Compression rate | 0.17 | 0.42 | 0.31 | 0.25 | 0.27
Table B:  The size of models used in Table 1 Teacher model | resnet32x4 | resnet32x4 | WRN-40-2 | ResNet50 | ResNet50 | vgg13
---|---|---|---|---|---|---
Student model | ShuffleNetV1 | ShuffleNetV2 | ShuffleNetV1 | vgg8 | MobileNetV2 | MobileNetV2
Teacher size (M) | 7.43 | 7.43 | 2.26 | 23.71 | 23.71 | 9.46
Student size (M) | 0.95 | 1.36 | 0.95 | 3.96 | 0.81 | 0.81
Compression rate | 0.13 | 0.18 | 0.42 | 0.17 | 0.03 | 0.09
Table C:  The size of models used in Table 2

## Model Compression Size

Tables B and C provide the size of each model used in Tables 1 and 2.

## Implementation Details of IL Experiment Incremental Learning (IL)

Our task-incremental learning experiment follows the implementation and
evaluation protocol described in a popular continual learning benchmark (Hsu
et al. 2018). The benchmark splits the image datasets CIFAR10 and CIFAR100
into 5 tasks; thus, each task has 2 and 20 classes, correspondingly. The
evaluation is performed at the end of the learning curriculum, and the
averaged classification accuracy of all tasks is reported with testing data.
The regularization coefficient $\lambda$ of all methods (except the baseline
and non-incremental learning) are selected by a grid search with 20% of the
dataset. We follow all the default training and testing configurations
provided by the public benchmark (Hsu 2018) to conduct the experiments.
Lastly, In our methods, the features used in WeightedH Feature-SE and Feature-
SE are the flattened outputs from the last convolutional block of
WideResNet-28-2 (Zagoruyko and Komodakis 2016), which has the output dimension
of 2048 (CxWxH=128x4x4). Note that we do not use the linear transformation
function $r$ in Section Incremental Learning (IL) since the teacher and
student models always have the same feature dimensions.

## Additional discussion: Model Compression versus Incremental Learning

This section highlights three differences between Model Compression (MC) and
Incremental Learning (IL) in their problem settings. First, MC has its teacher
share the same training data with the student. In contrast, the student in
task-incremental learning has no access to the teacher’s training data and is
exposed to a new set of classes in the new task.

Second, MC only applies the KD process once, while the task-IL repeats the KD
process multiple times based on the length of the task sequence.

Lastly, the task-incremental learning setting obeys closely the assumptions
made in Section Rethinking Knowledge Distillation, while MC relaxes those a
little bit. Although MC’s formulation (equation 14) has the same form as
equation 10, it deviates from the assumptions made in equation 4 in three
ways: (1) the $dz$ might not be small, (2) the $z$ is normalized and linearly
transformed, and (3) the softmax-based classifier $f$ is not forced to be the
same between the teacher and student. Note that (2) and (3) happens when z is
features but not logits, and the impact of (1) has been reduced by the
normalization (equation 12).