Quantum tomography with $\tau$ leptons at the FCC-ee
Entanglement, Bell inequality violation, $\sin\theta_{W}$ and anomalous
copulings

M. Fabbrichesia, and L. Marzolab,c

aINFN, Sezione di Trieste, Via Valerio 2, I-34127 Trieste, Italy
bLaboratory of High-Energy and Computational Physics, NICPB, Rävala 10, 10143
Tallinn, Estonia
cInstitute of Computer Science, University of Tartu, Narva mnt 18, 51009
Tartu, Estonia

ABSTRACT

The Future Circular Collider (FCC)—in its first incarnation as a lepton
collider—will produce, according to the proposed design, more than 100 billion
pairs of $\tau$ leptons after working for four years at the energy of the
$Z$-boson resonance. The $\tau$ lepton is special because its relatively long
lifetime makes it possible to reconstruct the momenta of neutrinos emitted in
the single pion decay mode. The resulting large number of events is an ideal
source for a full quantum tomography of the process that will test quantum
entanglement and the violation of Bell inequality with unprecedented
precision. In addition, the study of polarizations and spin correlations can
provide a competitive determination of the Weinberg angle $\theta_{W}$ and
constrain possible anomalous couplings in the neutral electroweak current. We
utilize analytic results and Monte Carlo simulations to explore to what extent
these goals might be accomplished.

❏

###### Contents

  1. 1 Motivations
  2. 2 Methods
    1. 2.1 Quantum tomography at work
    2. 2.2 Entanglement and Bell inequality violation
    3. 2.3 Monte Carlo simulation
      1. 2.3.1 Neutrino momenta reconstruction
      2. 2.3.2 Detector response and initial state radiation
  3. 3 Results
    1. 3.1 Theoretical quantum tomography: analytic results
    2. 3.2 Tomography with Monte Carlo simulation
      1. 3.2.1 Monte Carlo simulation with detector and ISR effects
      2. 3.2.2 Background and systematic uncertainties
    3. 3.3 Polarizations
      1. 3.3.1 The interplay between polarizations and entanglement
      2. 3.3.2 Weinberg angle
    4. 3.4 Anomalous couplings
      1. 3.4.1 Observables
      2. 3.4.2 Limits
  4. 4 Outlook

##  1 Motivations

T he study of the $\tau$ lepton is interesting for at least three reasons. For
a start, it is the heaviest among the leptons and for this reason alone a
particle that deserves a detailed study. Next, an exact reconstruction of the
neutrino momentum—and thereby that of the $\tau$ itself— is made possible by
the relatively long lifetime that yields a measurable impact parameter for the
charged pions produced in the semi-leptonic decay. Moreover, it decays into
hadrons whose angular distribution of momenta allows for the reconstruction of
the $\tau$ leptons polarization. The last two of these features make the
quantum tomography of the corresponding polarization density matrix not only
possible but also very precise.

The FCC-ee [1] is expected to produce $3\times 10^{12}$ visible $Z$ after
working for 4 years a the $Z$-boson resonance—of these $1.3\times 10^{11}$
will decay into $\tau$-lepton pairs. Given the branching ratio of $10.82$% for
the single pronged charged pion plus a neutrino decay mode of the $\tau$, we
expect $1.7\times 10^{9}$ events originated by the produced $\tau$ pairs.
These events are the simplest to analyze, have very little if any background
and also provide optimal resolution to analyze the polarization of the
decaying leptons. For comparison, and to have a sense of the possible
accomplishment, SuperKEKB [2] will produce by the end of its present run
$5\times 10^{10}$ $\tau$ lepton pairs [3], while LEP produced about 137 000 of
them [4]. For these reasons, the FCC-ee is the ideal laboratory for a detailed
study of the $\tau$ lepton and of its properties fully accessible via quantum
tomography.

The production of the $\tau$ pairs at the $Z$-boson resonance unveils the
physics of the electroweak interactions and, in particular, the parity
violating interactions yielding non-vanishing polarizations in the final
state. These polarizations vanish in the lower energy regime where the photon
dominates the production process. The interplay between polarizations and the
other entries in the density matrix shows the origin, and the limited
validity, of the maximum entanglement principle featured in purely
electromagnetic processes—when only leptons are present. It also makes it
possible to determine the Weinberg angle with great precision.

Finally, the definition of appropriated operators in terms of the polarization
density matrix, leads to strong constraints on the size of the form factors of
all operators up to dimension five affecting the electroweak neutral current.
In the low-energy limit, these form factors correspond to the effective radius
and magnetic and electric dipole momenta of the $\tau$ lepton.

Together with the above, perhaps more mundane goals, the FCC could push the
study of quantum mechanics to regimes where it has yet to be fully
investigated, namely at very high energies and in the presence of strong and
electroweak interactions, opening the possibility of unexpected new
fundamental discoveries.

##  2 Methods

T he density matrix of a quantum state can be fully reconstructed via a
procedure dubbed quantum tomography. The $\tau$ leptons—whose spin states are
represented by qubits, that is, two-level quantum states—act as their own
polarimeters and the full polarization density matrix of a $\tau$-lepton pair
can be reconstructed by studying the angular distribution of suitable decay
products. The simplest and most powerful—in terms of polarization resolution—
of these decay modes is the single-prong $\tau\to\pi\nu_{\tau}$ decay. Other
decay channels, including those yielding three pions or $\rho$ or $a_{1}$
mesons, can also be used for quantum tomography purposes.

###  2.1 Quantum tomography at work

The density matrix describing the polarization state of a quantum system
composed by two fermions can be written as

$\rho=\frac{1}{4}\quantity[\mathbb{1}\otimes\mathbb{1}+\sum_{i}\operatorname{B}_{i}^{+}\,\quantity(\sigma_{i}\otimes\mathbb{1})+\sum_{j}\operatorname{B}_{j}^{-}\,\quantity(\mathbb{1}\otimes\sigma_{j})+\sum_{i,j}\operatorname{C}_{ij}\,\quantity(\sigma_{i}\otimes\sigma_{j})],$
(2.1)

with $i,j=r,\,n,\,k$ and $\sigma_{i}$ being the Pauli matrices. The
decomposition refers to a right-handed orthonormal basis, $\\{{\bf n},{\bf
r},{\bf k}\\}$ and the quantization axis for the polarization is taken along
${\bf k}$, so that $\sigma_{k}\equiv\sigma_{3}$. In the fermion-pair center of
mass frame we have

${\bf n}=\frac{1}{\sin\Theta}\quantity({\bf p}\times{\bf k}),\quad{\bf
r}=\frac{1}{\sin\Theta}\quantity({\bf p}-{\bf k}\cos\Theta)\,,$ (2.2)

where ${\bf k}$ is the direction of the $\tau^{+}$ momentum and $\Theta$ is
the scattering angle. We take ${\bf p}\cdot{\bf k}=\cos\Theta$, with ${\bf p}$
being the direction of the incoming $e^{+}$.

The coefficients $\operatorname{B}^{\pm}_{i}$ in Eq. (2.1) give the
polarizations of the individual fermions, whereas the coefficients
$\operatorname{C}_{ij}$ encode the spin correlations. By using
$\mbox{Tr}\,(\sigma_{i}\sigma_{j})=2\delta_{ij}$ and
$\mbox{Tr}\,(\sigma_{i})=0$, we have:

$\operatorname{B}_{i}^{+}=\mbox{Tr}\,[\rho\quantity(\sigma_{i}\otimes\mathbb{1})]\,,\quad\operatorname{B}_{i}^{-}=\mbox{Tr}\,[\rho\quantity(\mathbb{1}\otimes\sigma_{i})]\,,\quad\operatorname{C}_{ij}=\mbox{Tr}\,[\rho\quantity(\sigma_{i}\otimes\sigma_{j})]\,.$
(2.3)

More details on these definitions and quantum tomography in general can be
found in [5].

The quantum tomography of the polarization density matrix is completed once
the coefficients $\operatorname{B}_{i}$ and $\operatorname{C}_{ij}$ have been
found. Given a Lagrangian for a theory, these quantities can be computed from
the scattering amplitudes describing the underlying process. Experimentally,
or in Monte Carlo simulations, they can instead be reconstructed by tracking
the angular distribution of suitable $\tau$-pair decay products. In
particular, for events where each $\tau$ lepton decays to a single pion and a
neutrino, we have

$\frac{1}{\sigma}\,\frac{\differential\sigma}{\differential\cos\theta^{\pm}_{i}}=\frac{1}{2}\,\quantity(1+\operatorname{B}^{\pm}_{i}\cos\theta^{\pm}_{i})\,,$
(2.4)

$\frac{1}{\sigma}\,\frac{\differential\sigma}{\differential\cos\theta^{+}_{i}\,\differential\cos\theta^{-}_{j}}=\frac{1}{4}\,\left(1+\operatorname{C}_{ij}\,\cos\theta^{+}_{i}\,\cos\theta^{-}_{j}\right)\,,$
(2.5)

in which $\cos\theta^{\pm}_{i}$ are the projections of the $\pi^{\pm}$
momentum direction on the $\\{{\bf n},{\bf r},{\bf k}\\}$ basis, as computed
in the rest frame of the decaying $\tau^{\pm}$. Figures 2.1 and 2.2 show the
normalized distributions obtained for the values of the $\operatorname{B}_{i}$
and $\operatorname{C}_{ij}$ from the actual Monte Carlo simulation we have
used in this work. The central values of the Gaussian fits—dashed
lines—indicate the values of the coefficients themselves.

Figure 2.1: Distributions obtained for the coefficients
$\operatorname{B}_{i}^{\pm}$ in the Monte Carlo simulation of the
$\tau$-lepton pairs decaying into a charged pion plus a neutrino, the momentum
of which is reconstructed. The dotted curves represent the Gaussian fit of the
histograms.

Whenever the average of the product of two cosine (in the
$\operatorname{C}_{ij}$ coefficients) differ from the product of the averages
of the single cosines (in the $\operatorname{B}_{i}$ coefficients) we have non
vanishing correlation and, possibly, quantum entanglement.

Figure 2.2: Distribution of the values of the coefficients
$\operatorname{C}_{ij}$ in the Monte Carlo simulation of the $\tau$-lepton
pairs decaying into a charged pion plus a neutrino, the momentum of which is
reconstructed. The dotted curves represent the Gaussian fit of the histograms.

###  2.2 Entanglement and Bell inequality violation

Given a two-qubit, $4\times 4$ density matrix $\rho$ as in (2.1), its
concurrence can be explicitly constructed by using the auxiliary matrix

$R=\rho\,(\sigma_{y}\otimes\sigma_{y})\,\rho^{*}\,(\sigma_{y}\otimes\sigma_{y})\,,$
(2.6)

where $\rho^{*}$ denotes the matrix with complex conjugated entries. Although
non-Hermitian, the matrix $R$ has non-negative eigenvalues; denoting with
$r_{i}$, $i=1,2,3,4$, their square roots and assuming $r_{1}$ to be the
largest, the concurrence $\mathscr{C}$ can be expressed as [6]

$\mathscr{C}=\max\big{(}0,r_{1}-r_{2}-r_{3}-r_{4}\big{)}\ .$ (2.7)

The concurrence is a quantitative estimate of the amount of entanglement in
the two-qubit system. Entanglement is present if $\mathscr{C}>0$ and maximal
for $\mathscr{C}=1$.

To assess the possible violation of Bell inequalities we resort instead to the
matrix $\operatorname{C}$ in Eq. (2.5) and build, with its transpose
$\operatorname{C}^{T}$, the symmetric, positive, $3\times 3$ matrix
$M=\operatorname{C}\operatorname{C}^{T}$; its eigenvalues $m_{1}$, $m_{2}$,
$m_{3}$ can be ordered in increasing order: $m_{1}\geq m_{2}\geq m_{3}$. Then,
the two-spin state $\rho$ in (2.1) violates the Bell inequality [7], in its
original as well as in the equivalent Clauser-Horne-Shimony-Holt (CHSH) [8]
form (for details see [5]) if and only if the sum of the two greatest
eigenvalues of $M$ is strictly larger than 1, that is (Horodecki’s condition
[9])

$\mathfrak{m}_{12}\equiv m_{1}+m_{2}>1\,.$ (2.8)

We take this condition as our test of the violation of the Bell inequality. It
has the advantage of automatically maximize the amount of violation without
having to worry about a specific choice of basis for the polarization vectors.

Figure 2.3: Numerical value of $\mathfrak{m}_{12}$ and its uncertainty
obtained for the spin correlations of the $\tau$-lepton pairs as the size $N$
of the sample is varied. The dashed horizontal line corresponds to the
analytic value. The bias is manifest for samples containing 100 events because
$\mathfrak{m}_{12}$ must always be less than 2.

The Horodecki’s condition in Eq. (2.8) may show a bias toward positive values
when evaluated numerically on samples with a restricted statistics. The issue
was addressed in [10] by correcting for it in the analysis of the violation of
Bell inequalities in top-quark production at the LHC, a case in which the
number of events is limited and $\mathfrak{m}_{12}$ is biased. In this work,
as in [11], such a correction is not necessary because, as shown in Fig 2.3,
the bias is negligible for samples containing at least $10^{4}$ events.

###  2.3 Monte Carlo simulation

The Monte Carlo simulation provides all the inputs required for the quantum
tomography of the process, which we perform as follows.

We generate 10 million events, containing each a $\tau$ lepton pair decaying
into two opposite charged pions plus the $\tau$ neutrinos, by means of
MadGraph5 [12] and the TauDecay library [13]. The events are generated at the
tree level in the electroweak interaction and, therefore, the cross section
and the remaining observables are expected to assume their tree-level values.

To make the simulation closer to the analysis with actual data, we replace the
Monte Carlo truth $\tau$ lepton momenta with those obtained from the neutrino
momenta reconstruction. We then sort the events into 50 independent samples
(each corresponding to a pseudo-experiments with $2\times 10^{5}$ events) and
reconstruct the $\operatorname{B}^{\pm}_{i}$ and $\operatorname{C}_{ij}$
coefficients for each sample. The resulting Gaussian distributions, shown in
Figures 2.1 and 2.2, give the central values and the statistical uncertainties
of these parameters.

To mimic real data even further, we repeat the analysis by modelling the
efficiency and uncertainty in the charged pion tracks and vertex determination
of the detector prior to performing the neutrino momenta reconstruction. More
details are given below.

####  2.3.1 Neutrino momenta reconstruction

The eight unknown components of the neutrino momenta can be reconstructed by
means of eight equations: four from the sum of the $\tau$-lepton momenta,
which is constrained to satisfy

$p_{\tau^{+}}^{\mu}+p_{\tau^{-}}^{\mu}=p^{\mu}_{e^{+}e^{-}}\,,$ (2.9)

and four from the mass-shell conditions

$\displaystyle(p_{\tau^{+}}-p_{\pi^{+}})^{2}=m_{\nu}^{2}=0\quad$ and
$\displaystyle\quad(p_{\tau^{-}}-p_{\pi^{-}})^{2}=m_{\nu}^{2}=0$ (2.10)
$\displaystyle p_{\tau^{+}}^{2}=m_{\tau}^{2}\quad$ and $\displaystyle\quad
p_{\tau^{-}}^{2}=m_{\tau}^{2}\,.$ (2.11)

The system of equations is second order and a two-fold degeneracy arises (see
the Appendix in [14]).

As opposed to other processes, like top quark and $W$-boson decays, the
reconstruction of the neutrino momenta and, therefore, of the $\tau$ momenta
is almost perfect. The reason is that the $\tau$ lepton lives long enough to
give a decay vertex that can be distinguished from the collision point.
Consequently, the vector of closest approach, identified from the continuation
of the trajectories of the pions emitted in the decay, can be measured and
used as to resolve the two-fold degeneracy arising from the momenta
reconstruction. Following [15], we then define the directions of the two
charged pions as

${\mathbf{n}_{-}}=\frac{{\mathbf{p}}_{-}}{|{\mathbf{p}}_{-}|}\quad\text{and}\quad{\bf
n_{+}}=\frac{{\mathbf{p}}_{+}}{|{\mathbf{p}}_{+}|}\,,$ (2.12)

in which ${\mathbf{p}}_{\pm}$ are their momenta. The distance between the two
decay vertices ${\bf v_{\pm}}$ of the $\tau^{\pm}$ leptons is

${\bf d}={\bf v_{+}}-{\bf v_{-}}\,.$ (2.13)

The vector of closest approach connecting the backward continuations of the
two charged pion tracks is then given by

${\bf d}_{min}={\bf d}+\frac{[({\bf d}\cdot{\bf n_{+}})({\bf n_{-}}\cdot{\bf
n_{+}})-{\bf d}\cdot{\bf n_{-}}]\,{\bf n_{-}}+[({\bf d}\cdot{\bf n_{-}})({\bf
n_{-}}\cdot{\bf n_{+}})-{\bf d}\cdot{\bf n_{+}}]\,{\bf n_{+}}}{1-({\bf
n_{-}}\cdot{\bf n_{+}})^{2}}\,.$ (2.14)

The correct kinematic reconstruction of the $\tau$ momenta is then selected by
computing ${\bf d}_{min}$ for the two solutions and comparing the results with
the measured value.

####  2.3.2 Detector response and initial state radiation

The performance of what will be the actual detectors of FCC-ee can only be
presumed from the specifications detailed in the current experimental
proposal.

The pairs of $\tau$ leptons must be identified from the charged pions
appearing in the final state. Even though the efficiency is high but not 100%,
this is not really a problem given the very large number of events available.

We model the detector resolution with the following uncertainties:

$\frac{\sigma_{p_{T}}}{p_{T}}=3\times 10^{-5}\oplus 0.6\times
10^{-3}\frac{p_{T}}{\rm GeV}\quad\text{and}\quad\sigma_{\theta,\phi}=0.1\times
10^{-3}\,\text{rad}$ (2.15)

for the tracks proper and

$\sigma_{b}=3\,\mu{\rm m}\oplus\frac{15\,\mu{\rm
m}}{\sin^{2/3}\Theta}\frac{\rm GeV}{p_{T}}$ (2.16)

for the impact parameters and for the reconstruction of the vector of closest
approach. These values are taken from the envisaged IDEA tracking detector
[1]; a constant smearing originates in imperfections in the detector or
anomalies in signal collection, whereas the term scaling with the momentum is
a noise that comes from the imperfection of the readouts. In the simulation we
retain the leading contributions in the quoted uncertainties and account for
the related detector effects by performing a Gaussian smearing of the Monte
Carlo truth pion momenta and closest approach vector taking the nominal
resolutions above as standard deviations.

Notice that the uncertainties are small if typical pion momenta of the order
of 10 GeV and $\tau$-lepton times of flight of the order of 0.1 mm are
considered. Furthermore, the decay of the $\tau$ pairs will generally occur
inside the beam pipe (of order 1 cm), thereby ensuring the absence of further
interactions with the detector material that could quench the spin correlation
under study.

Figure 2.4: Distribution of the $\tau$-lepton pair events in invariant mass
after including the initial state radiation of the electrons. The last four
bins, from 89 to 91 GeV, contain 99% of the events.

We also include in our Monte Carlo simulation the effect of Initial state
radiation (ISR), which shifts the beam—and the actual center of mass
(CM)—energy as shown in Fig. 2.4. The plot is obtained by using Pythia 8 [16]
and following the indicated statistics we pollute our dataset with events
characterized by a lower CM energy up to $\sqrt{s}=89$ GeV, corresponding to
the last three bins of Fig. 2.4 which comprise 99% of the events.

The angular distributions of the pion momenta in the rest frame of the
$\tau$-lepton pairs give, as the desired, the coefficients of the polarization
density matrix. In the following we compare the results obtained from
analytical computations to the values indicated by the Monte Carlo simulations
performed with and without the ISR and detector resolution effects.

##  3 Results

Q uantum tomography gives us the means to study entanglement and Bell
inequality violation in the process

$e^{+}e^{-}\to
Z\to\tau^{+}\tau^{-}\to\pi^{+}\pi^{-}\nu_{\tau}\,\bar{\nu}_{\tau}\,$ (3.1)

as it will be available at the FCC-ee.

###  3.1 Theoretical quantum tomography: analytic results

Let us first look at the analytic results for the parton level process
$e^{+}e^{-}\to Z\to\tau^{+}\tau^{-}$. This study already provides most of the
information about the actual physical process, that is, the one inclusive of
the hadronic decays of the $\tau$ leptons.

Figure 3.1: Analytic solutions for the concurrence (left) and the Bell
inequality violation (right) in the parton level process
$e^{+}e^{-}\to\tau^{+}\tau^{-}$. The presence of entanglement is signaled by
$\mathcal{C}>0$ and the violation of Bell inequalities by
$\mathfrak{m}_{12}>1$.  Figure 3.2: Values of the concurrence $\mathscr{C}$
and the Horodecki’s condition $\mathfrak{m}_{12}$ as functions of the
scattering angle in the parton level process $e^{+}e^{-}\to\tau^{+}\tau^{-}$.
The presence of entanglement is signaled by $\mathcal{C}>0$ and the violation
of Bell inequalities by $\mathfrak{m}_{12}>1$.

The values of the concurrence $\mathscr{C}$ and the Horodecki’s condition
$\mathfrak{m}_{12}$ for energies that range from the production threshold up
to 350 GeV are shown in Fig. 3.1. The result for $\sqrt{s}=10$ GeV have been
already discussed in [11] and applied to superKEKB and the physics at Belle
II. Here we study the region at the $Z$-boson resonance. In Fig. 3.2 the C M
energy is set at $\sqrt{s}=91.19$ GeV and the concurrence and the Horodecki’s
condition are shown as functions of the scattering angle $\Theta$, formed by
the $e^{+}$ and $\tau^{+}$ directions.

As for the polarization of the $\tau$ pair, we find
$\operatorname{B}_{i}^{+}=\operatorname{B}_{i}^{-}=\operatorname{B}_{i}$ for
$i=r,\,n,\,k$. Contrary to most of the other process that have been utilized
in studies of entanglement, we have non vanishing $\operatorname{B}_{i}$
coefficients at the leading order even though the initial beam is made of
unpolarized electrons and positrons. This feature is due to the parity
violating electroweak interactions and makes the study of the $\tau$ leptons
at this energy particularly interesting. The two plots in Fig. 3.3 show the
behavior of the coefficients $\operatorname{B}_{i}^{\pm}$ as functions of the
scattering angle $\Theta$ at $\sqrt{s}=91.19$ GeV, as well as in a broader
energy range.

Figure 3.3: Coefficients
$\operatorname{B}_{i}=\operatorname{B}_{i}^{+}=\operatorname{B}_{i}^{-}$ as
functions of the scattering angle in the parton level process
$e^{+}e^{-}\to\tau^{+}\tau^{-}$ at $\sqrt{s}=91.19$ GeV (left) and
$|\operatorname{B}|=\sqrt{\operatorname{B}_{k}^{2}+\operatorname{B}_{n}^{2}+\operatorname{B}_{n}^{2}}$
across a broader kinematic space (right).

The coefficients $\operatorname{C}_{ij}$ and $\operatorname{B}_{i}$ averaged
over the angular distribution of the $\tau$ pair are found to be

$\operatorname{C}=\begin{pmatrix}0.4878&\phantom{-}0&0&\\\ 0&-0.4878&0.0011\\\
0&\phantom{-}0.0011&1\end{pmatrix}\quad\quad\operatorname{B}^{+}=\operatorname{B}^{-}=\begin{pmatrix}0\\\
0.0001\\\ 0.2194\end{pmatrix}\,.$ (3.2)

These values can be directly compared with the results of the Monte Carlo
simulations. We find that the system is entangled and that the weighted
angular averaged value of the concurrence is

$\mathscr{C}=0.4878\,.$ (3.3)

As shown in Fig 3.2 the entanglement depends on the kinematic region and is
maximal at $\Theta=\pi/2$.

Similarly, the violation of the Bell inequality is given by

$\mathfrak{m}_{12}=1.238\,,$ (3.4)

which again, is the result for the angular average, and can be enhanced by
choosing a kinematic region around $\Theta=\pi/2$ (see Fig. 3.2).

While the analytic results show evidence for entanglement and Bell inequality
violation, a quantitative study requires an estimate of the related
uncertainties inherent to the experimental analyses. For these, we turn to the
Monte Carlo simulation.

###  3.2 Tomography with Monte Carlo simulation

As explained in Section 2, we generate a total of $10^{7}$ events containing
$\tau$ pairs that result in two opposite charged pions and missing energy in
the final states. For each of the events we reconstruct the rest frames of the
$\tau^{\pm}$ leptons and compute the projection of their spin, indicated by
the directions of the pions, on the $\\{{\bf r},{\bf n},{\bf k}\\}$ basis in
these frames of reference. The statistical uncertainty is estimated by running
50 independent pseudo-experiments each consisting of a sample of 200 000
events drawn from the full dataset of $10^{7}$ Monte Carlo generated events.

Omitting, for the moment, the effects of ISR and finite detector resolution,
the procedure leads to the following determination of the coefficients
$\operatorname{C}_{ij}$ and their statistical uncertainty:

$\operatorname{C}=\begin{pmatrix}\phantom{-}0.4810\pm 0.0061&-0.0089\pm
0.0067&-0.0001\pm 0.0080\\\ -0.0074\pm 0.0068&-0.4806\pm
0.0065&\phantom{-}0.0017\pm 0.0059\\\ \phantom{-}0.0004\pm
0.0076&\phantom{-}0.0005\pm 0.0069&\phantom{-}0.9982\pm
0.0073\end{pmatrix}\,.$ (3.5)

For the coefficients $\operatorname{B}_{i}$ we have:

$\operatorname{B}^{+}=\begin{pmatrix}-0.0023\pm 0.0037\\\ -0.0001\pm 0.0039\\\
\phantom{-}0.2173\pm
0.0040\end{pmatrix}\quad\quad\operatorname{B}^{-}=\begin{pmatrix}-0.0037\pm
0.0038\\\ \phantom{-}0.0007\pm 0.0035\\\ \phantom{-}0.2186\pm
0.0035\end{pmatrix}\,.$ (3.6)

The concurrence is found to be

$\mathscr{C}=0.481\pm 0.004\,,$ (3.7)

and the Horodecki’s condition for the violation of the Bell inequality is
given by

$\mathfrak{m}_{12}=1.233\pm 0.014\,.$ (3.8)

The values in Eqs. (3.7)–(3.8) are in excellent agreement with the analytic
results, proving that the size of the samples used in the analysis yields
sufficient precision.

####  3.2.1 Monte Carlo simulation with detector and ISR effects

To have a better sense of the actual experimental uncertainties, we run a
second Monte Carlo simulation including the smearing of the charged pion
momenta and of the closest approach vector discussed in Section 2.3, and
contaminating the sample with events at lower $\sqrt{s}$ to include the effect
of ISR. This procedure gives

$\operatorname{C}=\begin{pmatrix}\phantom{-}0.4819\pm 0.0079&-0.0073\pm
0.0082&-0.0016\pm 0.0089\\\ -0.0066\pm 0.0082&-0.4784\pm 0.0084&0.0016\pm
0.0070\\\ -0.0002\pm 0.0080&-0.0004\pm 0.0087&\phantom{-}1.000\pm
0.0074\end{pmatrix}$ (3.9)

$\operatorname{B}^{+}=\begin{pmatrix}-0.0028\pm 0.0042\\\ -0.0001\pm 0.0049\\\
\phantom{-}0.2198\pm 0.0044\
\end{pmatrix}\quad\quad\operatorname{B}^{-}=\begin{pmatrix}-0.0039\pm
0.0048\\\ \phantom{-}0.0017\pm 0.0049\\\ \phantom{-}0.2207\pm
0.0044\end{pmatrix}\,.$ (3.10)

According to the above results for the coefficients $\operatorname{C}_{ij}$
and $\operatorname{B}_{i}$, we find that the concurrence is given now by

$\boxed{\mathscr{C}=0.4805\pm 0.0063}$ (3.11)

and the Horodecki’s condition by

$\boxed{\mathfrak{m}_{12}=1.239\pm 0.017\,.}$ (3.12)

The significance of the Bell inequality violation is about 14 standard
deviations. Equations Eqs. (3.11)–(3.12) are the main result of the present
work. The significance can be increased by means of kinematic cuts that select
events with scattering angles close to $\Theta=\pi/2$, for which the values of
$\mathcal{C}$ and $\mathfrak{m}_{12}$ are larger (see Fig. 3.2).

####  3.2.2 Background and systematic uncertainties

The dominant background arises from misconstruction of the $\tau$ decay
channel. This is very small for the single pion channel, much smaller than for
other decay channels as that into two pions. A potentially large background
comes from the presence of electron and positrons in the final state, but it
can be controlled by using the impact parameter. Backgrounds arising from the
process $e^{+}e^{-}\to q\bar{q}$ and from other sources are even smaller. We
therefore feel that it is not necessary to estimate their effect for the
purpose of the present analysis.

The absolute value of the difference between the central values of the
concurrence obtained in the Monte Carlo simulation without, see Eq. (3.7), and
with detector effects and ISR, see Eq. (3.11), is

$\delta\mathscr{C}|_{\rm syst}=\left|0.481-0.485\right|=0.004\,;$ (3.13)

similarly, for the Horodecki condition signaling the violation of the Bell
inequality (cf. Eqs. (3.8)–(3.12)) we have

$\delta\mathfrak{m}_{12}|_{\rm syst}=\left|1.233-1.239\right|=0.006\,.$ (3.14)

These differences provide an estimate of the upper bound in the systematic
errors of these two observables. They are subdominant with respect to the
statistical errors in Eq. (3.11) and Eq. (3.12).

To find the expected uncertainty at the FCC-ee—where about $10^{9}$ events are
to be collected in four years of operation—we must rescale our uncertainties
obtained by running pseudo-experiments containing 200 000 event each by a
factor $10\,\sqrt{50}$. This rescaling will make the statistical uncertainties
smaller than the upper limits on the systematic errors, which at that point
will come to dominate the overall uncertainty unless reduced and brought under
control. We expect that indeed this will be the case and that the systematic
error will be under control after the operating of the machine and the
detectors are comprehensively understood.

###  3.3 Polarizations

The coefficients $\operatorname{B}_{i}^{\pm}$ are directly related to the
polarizations of the $\tau$ leptons. Analytically, we find that these
coefficients are equal, $\operatorname{B}^{+}_{i}=\operatorname{B}^{-}_{i}$
for $i=r,\,n,\,k$.

From the Monte Carlo simulation with detector effects and ISR included, we
take the polarization to be the arithmetic average and obtain

$\boxed{\langle
P\rangle_{\tau}=\frac{1}{2}(\operatorname{B}_{k}^{+}+\operatorname{B}_{k}^{-})=0.2203\pm
0.0044\,,}$ (3.15)

while $\operatorname{B}_{n}^{\pm}$ and $\operatorname{B}_{r}^{\pm}$ are
consistent with their vanishing analytic value, and the overall sign depends
on the frame of reference. The polarization in Eq. (3.15) corresponds to the
tree-level value, the measured value being $0.1476\pm 0.0108$ [4].

####  3.3.1 The interplay between polarizations and entanglement

The components $\operatorname{B}_{i}$ of the polarization and the coefficients
$\operatorname{C}_{ij}$ of the spin correlations are closely related, for they
both enter the density matrix which must satisfy positivity and normalization
conditions. The entanglement is the largest for vanishing polarizations and
vice versa. This relationship explains the maximum entanglement shown by
amplitudes in parity conserving theories, such as QED [17], that have
vanishing polarizations. This also happens in electroweak processes if the
parity violating structure of the lepton current is suppressed, as it is known
to be for the special, and unphysical, choice $\sin\theta_{W}=0.5$ [17]. For
this value, the vector coupling of leptons vanishes and the current conserves
parity. The cancellation is not possible for any common value of the Weinberg
angle once quark currents are involved because of the different charges. For
this reason, the requirement of maximum entanglement (or the vanishing of the
polarizations, which amounts to the same), though appealing, can hardly be
promoted to a general principle from which to constrain a Standard Model
parameters like the Weinberg angle itself.

####  3.3.2 Weinberg angle

The Weinberg angle can be obtained from the polarization vector $P_{\tau}$
using the relation [18]

$P_{\tau}(\cos\Theta)=\frac{{\cal
A}_{\tau}\,\left(1+\cos^{2}\Theta\right)+2\cos\Theta\,{\cal
A}_{e}}{1+\cos^{2}\Theta+2\cos\Theta\,{\cal A}_{\tau}{\cal A}_{e}}\,,$ (3.16)

with,

${\cal A}={\cal A}_{e}={\cal
A}_{\tau}=\frac{2\,(1-4\sin^{2}\theta_{W})}{1+(1-4\sin^{2}\theta_{W})^{2}}\,,$
(3.17)

assuming lepton flavor universality.

We compute the Weinberg angle by taking the weighted angular average of Eq.
(3.16) and finding ${\cal A}$ by solving the equation after inserting the
value of $\langle P\rangle_{\tau}$ from Eq. (3.15). Inserting the value of
${\cal A}$ so obtained into Eq. (3.17) yields the equation

$0.2190=\frac{2\,(1-4\sin^{2}\theta_{W})}{1+(1-4\sin^{2}\theta_{W})^{2}}\,,$
(3.18)

which shows that the uncertainty in the coefficient $\operatorname{B}_{i}$ is
reduced by a factor of about 8 in the error propagation. The central value for
the Weinberg angle that solves Eq. (3.18) corresponds to the tree level value:

$\boxed{\sin^{2}\theta_{W}=0.2223\pm 0.0006\,,}$ (3.19)

consistently with the inputs of our Monte Carlo simulation.

Assuming that the systematic uncertainties will eventually be under control,
we rescale the uncertainty of our sample to account for the expected FCC-ee
yield and obtain an uncertainty of $7.8\times 10^{-6}$ on the determination of
the tree-level Weinberg angle. This uncertainty is two order of magnitude
better than the current value [19]. The report [1] cites uncertainties in the
Weinberg angle determination between $10^{-5}$ and $10^{-6}$ as one of the
goals for the FCC-ee machine.

###  3.4 Anomalous couplings

The $\tau$ lepton—the heaviest among the leptons, as the top quark among the
quarks—could be the likeliest to show a behavior departing from that described
by the Standard Model. The most general electroweak Lorentz-invariant vertex
$\Gamma^{\mu}$ between the $Z$-boson and the $\tau$ lepton up to dimension
five operators can be written as

$i\,\frac{g}{2\,\cos\theta_{W}}\,\bar{\tau}\,\Gamma^{\mu}(q^{2})\,\tau\,Z_{\mu}(q)=\\\
i\,\,\frac{g}{2\,\cos\theta_{W}}\,\bar{\tau}\left[\gamma^{\mu}F_{1}^{V}(q^{2})+\gamma^{\mu}\gamma_{5}F_{1}^{A}(q^{2})+\frac{i\sigma^{\mu\nu}q_{\nu}}{2m_{\tau}}F_{2}(q^{2})+\frac{\sigma^{\mu\nu}\gamma_{5}q_{\nu}}{2m_{\tau}}F_{3}(q^{2})\right]\tau\,Z_{\mu}(q)\,,$
(3.20)

with $F_{1}^{V}(0)=g_{V}=-1/2+2\,\sin^{2}\theta_{W}$ and
$F_{1}^{A}(0)=-g_{A}=1/2$. $F_{2}(0)$ is the magnetic and $F_{3}(0)$ the
electric dipole moment. We parametrize the first two form factors in terms of
a Taylor expansion as

$F_{1}^{V,A}(q^{2})=F_{1}^{V,A}(0)+\frac{q^{2}}{m_{Z}^{2}}C_{1}^{V,A}\,,$
(3.21)

and give limits on the coefficients $C_{1}^{V,A}$ at $q^{2}=m_{Z}^{2}$. These
form factors act as anomalous couplings of the $\tau$ lepton to the $Z$ boson.

####  3.4.1 Observables

Our strategy to constrain the form factors in Eq. (3.20) exploits the
polarization density matrix, which can be experimentally reconstructed through
quantum tomography and gives a bird’s-eye view of the possible observables
available for a given process.

The method has been previously used to constrain physics beyond the SM
affecting the top-quark [20, 21] and $\tau$ pair [21] production at the LHC
and at superKEKB [22], or yielding Higgs anomalous couplings to $\tau$ leptons
[14] and gauge bosons [23, 24, 25].

For the present case, we use the entries of the polarization density matrix to
define three observables that provide the means to best constrain the
parameters in Eq. (3.20): one observable is the concurrence $\mathscr{C}$
defined in Eq. (2.7) and it measures the entanglement in the spin states of
the produced $\tau$ pairs, another— specific to the CP-violating
contributes—is related to triple products involving one momentum and the spin
vectors of the $\tau$ leptons:

$\mathscr{C}_{odd}=\sum_{\begin{subarray}{c}i<j\end{subarray}}\Big{|}\operatorname{C}_{ij}-\operatorname{C}_{ji}\Big{|}\,,$
(3.22)

and the third is the total cross section:

$\sigma_{T}=\frac{1}{64\pi^{2}\,s}\int\mbox{d}\Omega\frac{|{\cal
M}|^{2}}{4}\sqrt{1-\frac{4m_{\tau}^{2}}{s}}\,,$ (3.23)

in which we neglect the electron mass and take $\sqrt{s}=91.19$ GeV.

####  3.4.2 Limits

Figure 3.4: $\chi^{2}$ test for the form factor $F_{2}(m_{Z}^{2})$ and
$F_{3}^{V}(m_{Z}^{2})$. The limits for $F_{2}(m_{Z}^{2})$ are obtained by
means of the concurrence, those for the form factor $F_{3}^{V}(m_{Z}^{2})$ by
means of the operator $\mathscr{C}_{odd}$.

Figure 3.5: $\chi^{2}$ test for the form factor $F_{1}^{V}(m_{Z}^{2})$ (left)
and $F_{1}^{A}(m_{Z}^{2})$ (right). Both limits are obtained by means of the
cross section.

For a sample of 200.000 events, the expected value and uncertainty on the
cross section is $(22.83\pm 0.05)$ pb. For the same sample, from Section
3.2.1, we find an uncertainty on $\operatorname{C}_{ij}$, and therefore on
$\mathscr{C}_{odd}$, of at most 0.009, and on the concurrence $\mathscr{C}$ of
0.006. We use these uncertainties in the determination of the limits on the
form factors with the understanding that they can be improved by rescaling the
number of events to that actually expected at the FCC-ee.

The operators introduced in the previous section, generically denoted here as
$\mathscr{O}_{a}[F_{i}(m_{Z}^{2})]$, depend on the electroweak form factors
$F_{i}(m_{Z}^{2})$ and $\mathscr{O}_{SM}$ are the values of the operators for
the Standard Model. To constrain the couplings, we introduce a $\chi^{2}$ test
set for a (68.3) 95% C.L.

$\left[\frac{\mathscr{O}_{a}[F_{i}(m_{Z}^{2})]-\mathscr{O}_{SM}}{\sigma_{a}}\right]^{2}\leq(1.00)\;3.84\,,$
(3.24)

in which we set the uncertainties $\sigma_{a}$ for the operators
$\mathscr{O}_{a}$ at the values discussed in the previous paragraph.

The bounds on each coupling can be extracted from Fig. 3.4 and 3.5 under the
assumption that the parameters are independent. With that, we neglect possible
correlations. The values obtained from the 95% confidence interval are
reported in Table 3.1.

$\mathscr{O}_{a}$ |  $\sigma_{a}$ |  form factor limits
---|---|---
$\mathscr{C}$ | 0.006 | $-0.002\leq F_{2}(m_{Z}^{2})\leq 0.003$
$\mathscr{C}_{odd}$ | 0.009 | $-0.001\leq F_{3}(m_{Z}^{2})\leq 0.001$
$\sigma_{T}$ | 0.05 pb | $-0.009\leq C_{1}^{V}\leq 0.010$
$\sigma_{T}$ | 0.05 pb | $-0.001\leq C_{1}^{A}\leq 0.001$
Table 3.1:  Bounds obtained at the 95% confidence level for the form factors,
as shown in Figures 3.4 and 3.5, neglecting correlations. The values refer to
a sample of 200 000 events; limits for larger samples can be obtained by
rescaling these values by the square root of the ratio of the corresponding
numbers of events.

In Table 3.1 we report the limits as obtained with the most effective
operator. For the form factor $F_{2}(m_{Z}^{2})$, we could have included those
obtained with the cross section instead of the concurrence. The cross section
provides a slightly stronger bound because it has a smaller relative
uncertainty. This does not mean that the cross section is a more sensitive
observable than the concurrence. Fig. 3.6 shows that the limits from the
concurrence are stronger than those from the cross section when a common
relative uncertainty is considered.

Figure 3.6: The $\chi^{2}$ tests for the form factor $F_{2}^{V}(m_{Z}^{2})$
using the cross section ($\sigma_{T}$) and the concurrence ($\mathscr{C}$)
show that the latter provides more stringent limits when equal relative
uncertainties are considered.

The limit on $F_{3}(m_{Z}^{2})$ in Table 3.1 corresponds to a limit on the CP
violating weak dipole moment of $5.6\times 10^{-18}$ e cm for the $\tau$
lepton. The corresponding limit from LEP is equal to $1.4\times 10^{-17}$ [26,
27]. These limits are comparable since they are obtained with a similar number
of events and utilize similar observables built out of a CP-odd triple
product. Rescaling our limits to the FCC-ee luminosity, we find
$|F_{3}(m_{Z}^{2})|\leq 2.0\times 10^{-19}$, which is one order of magnitude
better than the current limit: $3.6\times 10^{-18}$ e cm [19].

Table 3.1 shows the bounds obtained with pseudo-experiments containing each
200 000 events. Corresponding projections base on the full dataset of $10^{9}$
expected at the FCC-ee can be derived (considering only the statistical
errors) upon a rescaling by a factor $10\sqrt{50}$.

##  4 Outlook

T he FCC-ee can, after 4 years of operation at the energy of the $Z$-boson
mass, test quantum entanglement and Bell inequality violation in $\tau$-lepton
pairs with extraordinary precision. Already with the reduced sample we have
used in our analysis the significance is well in excess of five standard
deviations and it is expected to reach many times that value with the full
expected dataset of $10^{9}$ events—provided the systematic uncertainties are
under control. We believe that the results we have presented make a convincing
case for including the quantum tomography of the $\tau$ leptons in the physics
goals of the FCC-ee. The work also proves that the same quantum tomography
technique can provide new and stringent bounds on the form factors entering
the electroweak neutral current of the $\tau$ leptons.

## Acknowledgements

The authors thank Michele Pinamonti and Christian Veelken for discussions.
L.M. is supported by the Estonian Research Council grants PRG803, RVTT3 and by
the CoE program grant TK202 “Fundamental Universe”.

## References

  * [1] FCC Collaboration, A. Abada et al., FCC-ee: The Lepton Collider: Future Circular Collider Conceptual Design Report Volume 2, Eur. Phys. J. ST 228 (2019), no. 2 261–623.
  * [2] SuperKEKB Collaboration, SuperKEKB Collider, Nucl. Instrum. Meth. A 907 (2018) 188, [arXiv:1809.01958].
  * [3] Belle-II Collaboration, W. Altmannshofer et al., The Belle II Physics Book, PTEP 2019 (2019), no. 12 123C01, [arXiv:1808.10567]. [Erratum: PTEP 2020, 029201 (2020)].
  * [4] L3 Collaboration, M. Acciarri et al., Measurement of tau polarization at LEP, Phys. Lett. B 429 (1998) 387–398.
  * [5] A. J. Barr, M. Fabbrichesi, R. Floreanini, E. Gabrielli, and L. Marzola, Quantum entanglement and Bell inequality violation at colliders, arXiv:2402.07972.
  * [6] W. K. Wootters, Entanglement of formation of an arbitrary state of two qubits, Phys. Rev. Lett. 80 (Mar, 1998) 2245–2248.
  * [7] J. Bell, On the Einstein Podolsky Rosen paradox, Physics Physique Fizika 1 (1964) 195.
  * [8] J. F. Clauser, M. A. Horne, A. Shimony, and R. A. Holt, Proposed experiment to test local hidden variable theories, Phys. Rev. Lett. 23 (1969) 880–884.
  * [9] R. Horodecki, P. Horodecki, and M. Horodecki, Violating bell inequality by mixed spin-1/2 states: necessary and sufficient conditions, Phys. Lett. A 200 (1995) 340.
  * [10] M. Fabbrichesi, R. Floreanini, and G. Panizzo, Testing Bell Inequalities at the LHC with Top-Quark Pairs, Phys. Rev. Lett. 127 (2021), no. 16 161801, [arXiv:2102.11883].
  * [11] K. Ehatäht, M. Fabbrichesi, L. Marzola, and C. Veelken, Probing entanglement and testing Bell inequality violation with $e^{+}e^{-}\to\tau^{+}\tau^{-}$ at Belle II, Phys. Rev. D 109 (2024), no. 3 032005, [arXiv:2311.17555].
  * [12] J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations, JHEP 07 (2014) 079, [arXiv:1405.0301].
  * [13] K. Hagiwara, T. Li, K. Mawatari, and J. Nakamura, TauDecay: a library to simulate polarized tau decays via FeynRules and MadGraph5, Eur. Phys. J. C 73 (2013) 2489, [arXiv:1212.6247].
  * [14] M. M. Altakach, P. Lamba, F. Maltoni, K. Mawatari, and K. Sakurai, Quantum information and CP measurement in $H\to\tau^{-}\tau^{+}$ at future lepton colliders, Phys. Rev. D 107 (2023), no. 9 093002, [arXiv:2211.10513].
  * [15] J. H. Kuhn, Tau kinematics from impact parameters, Phys. Lett. B 313 (1993) 458–460, [hep-ph/9307269].
  * [16] C. Bierlich et al., A comprehensive guide to the physics and usage of PYTHIA 8.3, SciPost Phys. Codeb. 2022 (2022) 8, [arXiv:2203.11601].
  * [17] A. Cervera-Lierta, J. I. Latorre, J. Rojo, and L. Rottoli, Maximal Entanglement in High Energy Physics, SciPost Phys. 3 (2017), no. 5 036, [arXiv:1703.02989].
  * [18] S. Jadach and E. Was, The $\tau$ polarization measurement, in Z PHYSICS AT LEP-1 (G. Altarelli, R. Kleiss, and C. Verzegnassi, eds.), p. 235.  CERN-EP-89-129, 1989.
  * [19] Particle Data Group Collaboration, R. L. Workman and Others, Review of Particle Physics, PTEP 2022 (2022) 083C01.
  * [20] R. Aoude, E. Madge, F. Maltoni, and L. Mantani, Quantum SMEFT tomography: Top quark pair production at the LHC, Phys. Rev. D 106 (2022), no. 5 055007, [arXiv:2203.05619].
  * [21] M. Fabbrichesi, R. Floreanini, and E. Gabrielli, Constraining new physics in entangled two-qubit systems: top-quark, tau-lepton and photon pairs, Eur. Phys. J. C 83 (2023), no. 2 162, [arXiv:2208.11723].
  * [22] M. Fabbrichesi and L. Marzola, Dipole momenta and compositeness of the $\tau$ lepton at Belle II, arXiv:2401.04449.
  * [23] M. Fabbrichesi, R. Floreanini, E. Gabrielli, and L. Marzola, Stringent bounds on HWW and HZZ anomalous couplings with quantum tomography at the LHC, JHEP 09 (2023) 195, [arXiv:2304.02403].
  * [24] A. Bernal, P. Caban, and J. Rembieliński, Entanglement and Bell inequalities violation in $H\rightarrow ZZ$ with anomalous coupling, Eur. Phys. J. C 83 (2023), no. 11 1050, [arXiv:2307.13496].
  * [25] R. Aoude, E. Madge, F. Maltoni, and L. Mantani, Probing new physics through entanglement in diboson production, JHEP 12 (2023) 017, [arXiv:2307.09675].
  * [26] OPAL Collaboration, P. D. Acton et al., Test of CP invariance in $e^{+}e^{-}\to\tau^{+}\tau^{-}$ and a limit on the weak dipole moment of the tau lepton, Phys. Lett. B 281 (1992) 405–415.
  * [27] ALEPH Collaboration, D. Buskulic et al., Search for CP violation in $Z\to\tau\tau$, Phys. Lett. B 297 (1992) 459–468.


# Review helps learn better: Temporal Supervised Knowledge Distillation

Dongwei Wang1, 2, Zhi Han1*, Yanmei Wang1, 2, Xiai Chen1, Baichen Liu1, 2,
Yandong Tang1

###### Abstract

Reviewing plays an important role when learning knowledge. The knowledge
acquisition at a certain time point may be strongly inspired with the help of
previous experience. Thus the knowledge growing procedure should show strong
relationship along the temporal dimension. In our research, we find that
during the network training, the evolution of feature map follows temporal
sequence property. A proper temporal supervision may further improve the
network training performance. Inspired by this observation, we propose
Temporal Supervised Knowledge Distillation (TSKD). Specifically, we extract
the spatiotemporal features in the different training phases of student by
convolutional Long Short-term memory network (Conv-LSTM). Then, we train the
student net through a dynamic target, rather than static teacher network
features. This process realizes the refinement of old knowledge in student
network, and utilizes it to assist current learning. Extensive experiments
verify the effectiveness and advantages of our method over existing knowledge
distillation methods, including various network architectures and different
tasks (image classification and object detection) .

## Introduction

Deep learning (LeCun, Bengio, and Hinton 2015) has brought significant boosts
to a series of pattern recognition tasks, such as image classification
(Simonyan and Zisserman 2014; Krizhevsky, Sutskever, and Hinton 2017; You et
al. 2020; Zhang et al. 2018), object detection (Girshick 2015; Redmon et al.
2016; He et al. 2017) and semantic segmentation (Zhao et al. 2017; Chen et al.
2017). As powerful networks usually grow deeper and wider (Xie et al. 2017;
Dosovitskiy et al. 2020), knowledge distillation is an effective way to train
superior small models.

The concept of knowledge distillation (KD) was firstly proposed in (Hinton,
Vinyals, and Dean 2015), where the student model can achieve better
performance by learning the output probability distributions of the teacher
model (Fig. 2(a)). Existing distillation works can be divided into two
categories: logits-based (Hinton, Vinyals, and Dean 2015; Zhao et al. 2022)
and features-based (Romero et al. 2014; Zagoruyko and Komodakis 2016a; Tian,
Krishnan, and Isola 2019; Heo et al. 2019; Chen et al. 2021; Ji, Heo, and Park
2021). Since FitNets (Romero et al. 2014), research has mainly focused on
distilling intermediate features which contain plentiful spatial knowledge
(Fig. 2(b)). However, though two kinds of methods have shown their excellent
performance, they tend to neglect the fact that student networks have
different _learning process_ on the same data due to structural discrepancy
with teachers. Using teacher outputs (either logits or features) as the
consistent learning goals may not be the best choice for students to converge.
Little research considers supervising the temporal learning process of
students.

Figure 1: The ARIMA analysis of the fully connected network. We record the
output of a neuron in FC1 in the first 30 epochs, and use ARIMA to predict its
output in the next 10 epochs. The results show that the prediction can match
well with the real outputs.

(a) Classical KD (b) Feature-based KD (c) Our TSKD

Figure 2: Illustration of classical KD, feature-based KD and our TSKD. Instead
of distilling between corresponding Teacher-Student outputs, we aim to make
teacher supervise the temporal learning process of student.

In this paper, we propose to distill along the temporal dimension. Our
motivation comes from the human learning process, where _reviewing_ not only
deepens our memory of old knowledge, but also more importantly, inspires us to
learn the new knowledge. We believe that the learning process of neural
networks also possesses strong temporal relationship.

To verify this hypothesis, we conduct time series prediction analysis on a
fully connected network using Autoregressive Integrated Moving Average model
(ARIMA) (Box and Pierce 1970). Specifically, we train the network to fit a
quadratic function and use ARIMA to model the intermediate outputs through
different epochs. As shown in Fig. 1, the fitted ARIMA model can provide an
approximate prediction to the real training process, which indicates the
temporal sequence property of network learning. Based on this discovery, we
come up with two questions: (1) Can networks utilize the old knowledge to
assist current learning like humans? (2) How to apply positive supervision to
the temporal learning process of network?

To solve these two questions, we propose a novel knowledge distillation
framework named Temporal Supervised Knowledge Distillation (TSKD). Different
from existing feature-based distillation methods that distill knowledge from
spatial features, we attempt to extract more knowledge along the temporal
dimension (Fig. 2(c)). Firstly, we modify the student network training as a
memorize-review mode, which imitates the human learning and reviewing
knowledge. Moreover, we design a dynamic learning target with teacher features
to train the temporal feature extractor, which enables teacher to guide the
learning process of students. A powerful student network can be obtained by
the temporal supervision of the teacher network.

Overall, our contributions are summarized as follows:

$\bullet$ We find that the knowledge in the network training grows regularly
over time. There exists exploitable information in the temporal dimension.

$\bullet$ We establish a new training paradigm by planning the network
training as memorize-review mode. This makes it possible for the student
network to review old knowledge and utilize it to assist current learning.

$\bullet$ We propose a novel knowledge distillation framework by supervising
the student network along the temporal dimension. Specifically, we use Conv-
LSTM network to extract temporal features in the student learning and train it
with a dynamic target.

$\bullet$ We achieve competitive performance against representative feature-
based distillation works on various network architectures and different
computer vision tasks.

## Related Work

The concept of knowledge distillation was firstly proposed by Hinton _et al._
in (Hinton, Vinyals, and Dean 2015). As an efficient manner to train smaller
student networks with the guidance of bigger teacher networks, it is applied
to various downstream tasks (Li, Jin, and Yan 2017; Li et al. 2021).

Previous research mainly focus on matching the output distributions of two
models (Hinton, Vinyals, and Dean 2015; Mirzadeh et al. 2020; Cho and
Hariharan 2019; Furlanello et al. 2018). As intermediate features contain more
valuable information, FitNets (Romero et al. 2014) was proposed to transfer
the knowledge from teacher network features to student network features using
$\mathcal{L}_{2}$ distance as a constraint. Following FitNets, most of the
research attention has been drawn to utilize the knowledge within intermediate
features and feature-based methods have achieved state-of-the-art distillation
performance. Representative works can be categorized to two branches: design
new transformation and loss functions (Zagoruyko and Komodakis 2016a; Tian,
Krishnan, and Isola 2019; Heo et al. 2019); optimize the matching relationship
between teacher and student feature candidates (Chen et al. 2021; Ji, Heo, and
Park 2021)

Most works distill between the corresponding teacher-student outputs directly.
Though ReviewKD (Chen et al. 2021) proposed to perform “review” in the
distillation, its main idea is to utilize multi-level information of the
teacher to guide one-level of the student, which realized multiscale spatial
knowledge transfer. This paper focused on supervising the student along the
temporal dimension.

## Method

### Notations and definitions

Let $S_{i}$ denote the student network at the $i$th training epoch and $T$
denote the teacher network. Given the same input data $X$, we denote the
intermediate features of teacher layer $t_{l}$ and student layer $s_{l}$ as
$F_{t_{l}}$ and $F_{s_{l}}^{i}$ respectively. Previous works have mainly
focused on transferring spatial knowledge from $F_{t_{l}}$ to $F_{s_{l}}^{i}$,
usually by reducing the distance between the two in the transformation space.
The adding loss term for $S_{i}$ can be written as follows:

$\displaystyle
L_{spatial}=\mathcal{D}\Big{(}Map(F_{s_{l}}^{i}),Map(F_{t_{l}})\Big{)}$ (1)

where $Map(\cdot)$ is the transformation that transfers the feature map to a
more representative space, and $\mathcal{D}(\cdot)$ is the distance
measurement function. Similarly, multi-layers distillation is written as:

$\displaystyle
L_{spatial}=\sum_{(s_{l},t_{l})\in\mathcal{C}}\mathcal{D}\Big{(}Map(F_{s_{l}}^{i}),Map(F_{t_{l}})\Big{)}$
(2)

where $\mathcal{C}$ stores the layers of features to transfer knowledge. And
the loss function of feature-based KD is:

$\displaystyle L_{student}=L_{task}+\lambda L_{spatial}$ (3)

Figure 3: An overview of the proposed Temporal Supervised Knowledge
Distillation (TSKD). During training at a review node, the same data will be
input into the $k$ memory networks and the current student network. After
transforming their outputs feature maps to attention maps, knowledge sequences
are obtained by connecting the increments between two adjacency states.
Finally, $\mathcal{L}_{temporal}$ is calculated between Conv-LSTM’s prediction
and the absolute increment. _BP_ and _FP_ denote back propagation and forward
propagation respectively.

The core idea is to use teacher features as auxiliary optimization goals.
Although $L_{spatial}$ brings improvements to students, these methods neglect
the progressive learning process. In our method, we exploit the temporal
information in student learning and use the teacher model to guide the whole
learning process. Specifically, we view the training of the student network as
a temporal process and plan it as a memorize-review mode (shown in Fig. 4).
Here we give some definitions for better explaining our method. The replanned
training is also shown more clearly in Algorithm 1.

_Action 1 (Memorize)_ : As the training progresses, the network will gradually
converge. We hope that the network can memorize the current state at some
points for future review. This action is implemented by saving the current
model.

_Action 2 (Train)_ : This action is the same as the general data-based
training and continues throughout the entire training process.

_Action 3 (Review)_ : Review the knowledge learned in $k$ previous memory
nodes and utilize it to assist the current training. The implementation
details are given in the following section.

_Memory nodes_ : Perform Action 1 and 2. The set of memory nodes is denoted by
$\mathcal{M}$.

_General nodes_ : Only perform Action 2. The set of general nodes is denoted
by $\mathcal{G}$.

_Review nodes_ : Perform Action 2 and 3.The set of review nodes is denoted by
$\mathcal{R}$.

_Memory interval_ : The number of general nodes between memory nodes, denoted
by $\delta$.

Figure 4: The replanned training process. Each node corresponds to an epoch in
training.

### Review Mechanism

Given input data $X$, the student model will have outputs at different layers,
which contain plentiful knowledge. The outputs evolve as training progresses.
For example, the network will gradually focus on the class discriminative
regions, which is called “attention” in previous works (Zagoruyko and
Komodakis 2016a; Guo et al. 2023). Take layer $s_{l}$ as an example, the
_knowledge increment_ between two temporally adjacent models $S_{i-1}$ and
$S_{i}$ can be represented as:

$\displaystyle\Delta^{i-1,i}_{s_{l}}=\Big{|}Map(F_{s_{l}}^{i})-Map(F_{s_{l}}^{i-1})\Big{|}$
(4)

Intuitively, $\Delta^{i-1,i}_{s_{l}}$ represents the cognitive differences
between $S_{i}$ and $S_{i-1}$ for the same input $X$.

When the training reaches a review node $S_{t},t\in\mathcal{R}$, we calculate
increments among $k$ previous memory nodes and $S_{t}$, and connect them to
obtain a length $k$ knowledge sequence:

$\displaystyle
knowledge\\_seq=(\Delta^{t-k\delta,t-(k-1)\delta}_{s_{l}},\dots,\Delta^{t-\delta,t}_{s_{l}})$
(5)

where $\delta$ denotes the memory interval. The value of $\delta$ indicates
the number of general nodes between two memory nodes. As the knowledge
sequence briefly summarizes the learning process during the period, we use a
simple Conv-LSTM network to extract temporal features in it and obtain a
prediction:

$\displaystyle\Delta_{s_{l}}^{pred}=ConvLSTM(knowledge\\_seq)$ (6)

Specifically, the sequence can be seen as “what has been learnt” (old
knowledge) in previous training, and Conv-LSTM gives a prediction of “what to
learn next” based on the sequence. Now the problem is the training of the
Conv-LSTM network. Due to the complexity of neural network and data
distribution, it is hard to find an optimal solution for the whole learning
process of student. However, we already have a well-trained teacher network
whose outputs can be used as strong outlines. Thus, we design the following
distillation mechanism. Specifically, we calculate the increment between
$S_{t}$ and $T$ and use it as Conv-LSTM’s target:

$\displaystyle\Delta^{abs}_{s_{l},t_{l}}=\Big{|}Map(F_{t_{l}})-Map(F_{s_{l}}^{t})\Big{|}$
(7)

We call $\Delta^{abs}_{s_{l},t_{l}}$ as absolute increment because it implies
“what needs to be learnt”. More importantly, $\Delta^{abs}_{s_{l},t_{l}}$ is a
dynamic learning goal that changes as student outputs gradually approximate
teacher outputs during training. This is more suitable for training in a
memorize-review cycle.

In conclusion, for the current review node, the temporal loss can be
calculated as :

$\displaystyle
L_{temporal}=\Big{\|}\Delta_{s_{l}}^{pred}-\Delta^{abs}_{s_{l},t_{l}}\Big{\|}_{2}$
(8)

Similarly, equation 8 can also be written in multi-layers style:

$\displaystyle
L_{temporal}=\sum_{(s_{l},t_{l})\in\mathcal{C}}\Big{\|}\Delta_{s_{l}}^{pred}-\Delta^{abs}_{s_{l},t_{l}}\Big{\|}_{2}$
(9)

where $\mathcal{C}$ stores the layers of features to perform _review_. The
update of Conv-LSTM will also bring gradients to student based on chain rule ,
which can be represented as:

$\displaystyle\theta_{S_{t+1}}=\theta_{S_{t}}-\alpha\frac{\partial
L_{temporal}}{\partial\theta_{lstm}}\frac{\partial\theta_{lstm}}{\partial\theta_{S_{t}}},$
(10)

where $\theta_{S_{t}}$ and $\theta_{lstm}$ denote the parameters of student
and Conv-LSTM respectively. The overall loss function of student network at a
review node is:

$\displaystyle L_{student}=L_{task}+\lambda L_{temporal}$ (11)

To get a better understanding of our method, we provide detailed description
in Fig. 3 and Algorithm 1.

Algorithm 1 Temporal Supervised Knowledge Distillation

Input: A pre-trained teacher model with parameters $\theta_{t}$; A student
model with initialized parameters $\theta_{s}$; A Conv-LSTM model with
initialized parameters $\theta_{lstm}$; Training dataset $\mathcal{D}$;
Initialization: Set memory interval $\delta$; Divide the training into three
nodes sets $\mathcal{M},\mathcal{G}$ and $\mathcal{R}$;
Output: A well-trained student model;

1: while $\theta_{s}$ not converged do

2: Sample a mini-batch $\mathcal{B}$ from $\mathcal{D}$.

3: if current node $\in$ $\mathcal{M}$ then

4: Freeze $\theta_{lstm}$;

5: Perform general _train_ on $\mathcal{B}$ and update $\theta_{s}$ with
$L_{task}$;

6:  _Memorize_ the current state.

7: else if current node $\in$ $\mathcal{G}$ then

8: Freeze $\theta_{lstm}$;

9: Perform general _train_ on $\mathcal{B}$ and update $\theta_{s}$ with
$L_{task}$.

10: else if current node $\in$ $\mathcal{R}$ then

11: Unfreeze $\theta_{lstm}$;

12: Forward propagation $\mathcal{B}$ into $\theta_{t}$ and $\theta_{s}$ to
obtain $F_{t_{l}}$ and $F_{s_{l}}$ across layers;

13: Construct $knowledge\\_seq$ and forward it into $\theta_{lstm}$ to obtain
$\Delta^{pred}_{s_{l}}$, calculate $L_{temporal}$;

14: Update $\theta_{lstm}$ with $L_{temporal}$, update $\theta_{s}$ with
$L_{task}$ and $L_{temporal}$,

15: end if

16: end while

Map function. We choose Attention Transfer ($AT(\cdot)$) as map function in
our method, which utilizes the insight of (Zagoruyko and Komodakis 2016a). It
is a transformation function that maps a 3D feature map tensor
$F\in\mathbb{R}^{C\times H\times W}$ to a 2D attention map
$F_{sum}\in\mathbb{R}^{H\times W}$. In our method, $F$ is flattened by summing
the squares along the channel dimension, which can be denoted as:

$\displaystyle F_{sum}=\textstyle\sum_{n=1}^{C}|F_{n}|^{2}$ (12)

Although various transformation functions have been proposed (Tian, Krishnan,
and Isola 2019; Heo et al. 2019) to map the features maps to a more knowledge-
transferable space, we choose $AT(\cdot)$ because it is simple and intuitive.
More importantly, it does not bring disruption to the temporal pattern
contained in the original feature maps. The distribution of values in
$F_{sum}$ reflects the spatial attention of the network clearly.
Temporal feature extractor. In the proposed method, KD is reformulated as a
time series prediction problem. Since ARIMA (Box and Pierce 1970) can not deal
with high-dimension data and transformer (Vaswani et al. 2017) may contribute
to higher computational cost, we choose Conv-LSTM (Shi et al. 2015) as the
temporal feature extractor. Conv-LSTM is a variation of general LSTM, which
replaces the element-wise operations in calculation by convolutional
operations. Thus, Conv-LSTM is able to handle spatiotemporal data. The
detailed information of the designed Conv-LSTM in TSKD is attached in
supplement due to the page limit.

## Experiments

Distillation Mechanism | Teacher | ResNet56 | ResNet32x4 | WRN40-2 | WRN40-2 | ResNet110 | VGG13
---|---|---|---|---|---|---|---
acc | 72.34 | 79.42 | 75.61 | 75.61 | 72.99 | 74.64
Student | ResNet20 | ResNet8x4 | WRN16-2 | WRN40-1 | ResNet32 | VGG8
acc | 69.06 | 72.50 | 73.26 | 71.98 | 71.14 | 70.36
Logits | KD | 70.66 | 73.33 | 74.92 | 73.54 | 73.08 | 72.98
Spatial feature | FitNets | 69.21 | 73.50 | 73.58 | 72.24 | 71.06 | 71.02
PKT | 70.34 | 73.64 | 74.54 | 73.54 | 72.61 | 72.88
RKD | 69.61 | 71.90 | 73.35 | 72.22 | 71.82 | 71.48
CRD | 71.16 | 75.51 | 75.48 | 74.14 | 73.48 | 73.94
AT | 70.55 | 73.44 | 74.08 | 72.77 | 72.31 | 71.43
VID | 70.38 | 73.09 | 74.11 | 73.30 | 72.61 | 71.23
OFD | 70.98 | 74.95 | 75.24 | 74.33 | 73.23 | 73.95
SP | 69.97 | 72.94 | 73.83 | 72.43 | 72.69 | 72.68
CC | 69.63 | 72.97 | 73.56 | 72.21 | 71.48 | 70.71
Temporal feature | TSKD | 71.76 | 75.73 | 75.77 | 74.49 | 73.81 | 74.75
$\Delta$ | +2.7 | +3.23 | +2.51 | +2.51 | +2.67 | +4.39
Table 1: Top-1 accuracy of the student network on CIFAR-100. And $\Delta$
represents the performance improvement over the vanilla student.

We evaluate our TSKD on various networks architectures e.g., VGG (Simonyan and
Zisserman 2014), ResNet (He et al. 2016), Wide ResNet (Zagoruyko and Komodakis
2016b) and MobileNet (Sandler et al. 2018). We compare TSKD with existing
distillation methods of different categories: logits-based: KD (Hinton,
Vinyals, and Dean 2015), DKD (Zhao et al. 2022); spatial feature-based:
FitNets (Romero et al. 2014), PKT (Passalis and Tefas 2018), RKD (Park et al.
2019), CRD (Tian, Krishnan, and Isola 2019), AT (Zagoruyko and Komodakis
2016a), VID (Ahn et al. 2019), OFD (Heo et al. 2019), SP (Tung and Mori 2019),
CC (Peng et al. 2019), CAT-KD (Guo et al. 2023) and ReviewKD (Chen et al.
2021).

Hyperparameters. The loss weight $\lambda$, the number of memory nodes $k$ and
the memory interval $\delta$ are important hyper-parameters in the review
stage. For image classification, we set $\lambda=1,k=3$ and $\delta=5$. For
object detection, we set $\lambda=0.4,k=3$ and $\delta=1$. The influence
brought by the different settings of these hyperparameters are explored
further in ablation studies.

Datasets. (1) CIFAR-100 (Krizhevsky, Hinton et al. 2009) comprises of 50,000
training images, with 500 images per class, and 10,000 test images. (2)
ImageNet (Deng et al. 2009) is considered the most challenging dataset for
classification, offering 1.2 million images for training and 50,000 images for
validation across 1,000 classes. (3) MS-COCO (Lin et al. 2014) is an
80-category general object detection dataset. The train2017 and val2017
contain 118k and 5k images respectively.

Implementation Details. Our implementation for CIFAR-100 and ImageNet strictly
follows (Chen et al. 2021; Zhao et al. 2022). Specifically, for CIFAR-100, we
train all models for 240 epochs with batch size 128 and decay the learning
rate by 0.1 for every 30 epochs after the first 150 epochs using SGD. The
initial learning rate is set to 0.1. For ImageNet, we adopt the standard
training process, which involves training the model for 100 epochs with batch
size 256 and decaying the learning rate every 30 epochs (initial learning rate
is 0.1).

For MS-COCO object detection, we take the most popular open-source report
Detectron2111https://github. com/facebookresearch/detectron2 as our strong
baseline. We train the student models using the standard training policy
following tradition (Wang et al. 2019).

For fairness, previous method results are either reported in previous papers
or obtained using author-released codes with our training settings.

### Main Results

Distillation Mechanism | Spatial feature | Logits | Temporal feature
---|---|---|---
| Teacher | Student | CRD | AT | OFD | ReviewKD | CAT-KD | KD | DKD | TSKD
top-1 | 76.16 | 68.87 | 71.37 | 69.56 | 71.25 | 72.56 | 72.24 | 68.58 | 72.05 | 72.22
top-5 | 92.86 | 88.76 | 90.41 | 89.33 | 90.34 | 91.00 | 91.13 | 88.98 | 91.05 | 90.99
Table 2: Top-1 and top-5 accuracy (%) on the ImageNet validation. We set ResNet-50 as the teacher and MobileNet-V2 as the student. Distillation Mechanism | Spatial feature | Logits | Temporal feature
---|---|---|---
| Teacher | Student | CRD | AT | OFD | ReviewKD | CAT-KD | KD | DKD | TSKD
top-1 | 73.31 | 69.75 | 71.17 | 70.69 | 70.81 | 71.61 | 71.26 | 70.66 | 71.70 | 71.90
top-5 | 91.41 | 89.07 | 90.13 | 90.01 | 89.98 | 90.51 | 90.45 | 89.88 | 90.41 | 90.76
Table 3: Top-1 and top-5 accuracy (%) on the ImageNet validation.We set
ResNet-34 as the teacher and ResNet-18 as the student.

CIFAR-100 classification. Table 1 presents the results on CIFAR-100. We have
categorized previous works into different groups based on the main idea. In
contrast, our method utilizes the spatiotemporal feature extracted in the
training process. Our method achieves $3\sim 5\%$ improvements on different
teacher-student pairs, which strongly demonstrates the effectiveness of TSKD.

ImageNet classification. We also conducted additional experiments on ImageNet
to further validate our approach. Specifically, we experimented with two
distillation settings: from ResNet50 to MobileNet and from ResNet34 to
ResNet18. The experimental results are reported in Table 2 and Table 3. Our
method achieves competitive results. In the setting from ResNet34 to ResNet18,
the gap between the student and the teacher had already been reduced to a very
small value of 1.61 by the previous best method. Nevertheless, we were able to
further reduce this gap to 1.41, resulting in a 12% relative performance
improvement.

MS-COCO object detection. In addition to classification, we also apply our
method to object detection task. For this task, we distill the output features
of the teacher and student’s backbones, following a similar procedure as in
the classification task. We use the best pre-trained models provided by
Detectron2 as teachers.

However, we find that the number of epochs for training a classical detection
network is relatively fewer compared to the classification networks (usually
hundreds). This makes it difficult for deploying the memorize-review pipeline.
Singly applying TSKD can hardly achieve outstanding performance. Thus, we
introduce ReviewKD (Chen et al. 2021) as our strong baseline to obtain
satisfactory results. It can be observed that our TSKD brings a further boost
to the AP metrics, even the performance of ReviewKD is relatively high.

| Method | mAP | AP50 | AP75 | AP1 | APm | APs
---|---|---|---|---|---|---|---
Teacher | Faster R-CNN w/R101-FPN | 42.04 | 62.48 | 45.88 | 54.60 | 45.55 | 25.22
Student | Faster R-CNN w/R18-FPN | 33.26 | 53.61 | 35.26 | 43.16 | 35.68 | 18.96
| w/KD | 33.97(+0.61) | 54.66 | 36.62 | 44.14 | 36.67 | 18.71
| w/FitNets | 34.13(+0.87) | 54.16 | 36.71 | 44.69 | 36.50 | 18.88
| w/FGFI | 35.44(+2.18) | 55.51 | 38.17 | 47.34 | 38.29 | 19.04
| w/ReviewKD | 36.75(+3.49) | 56.72 | 34.00 | 49.58 | 39.51 | 19.42
| w/ours+ReviewKD | 36.82(+3.56) | 56.85 | 39.70 | 49.63 | 39.73 | 19.20
Teacher | Faster R-CNN w/R101-FPN | 42.04 | 62.48 | 45.88 | 54.60 | 45.55 | 25.22
Student | Faster R-CNN w/R50-FPN | 37.93 | 58.84 | 41.05 | 49.10 | 41.14 | 22.44
| w/KD | 38.35(+0.42) | 59.41 | 41.71 | 49.48 | 41.80 | 22.73
| w/FitNets | 38.76(+0.83) | 59.62 | 41.80 | 50.70 | 42.20 | 22.32
| w/FGFI | 39.44(+1.51) | 60.27 | 43.04 | 51.97 | 42.51 | 22.89
| w/ReviewKD | 40.36(+2.43) | 60.97 | 44.08 | 52.87 | 43.81 | 23.60
| w/ours+ReviewKD | 40.65(+2.72) | 61.23 | 44.43 | 53.38 | 43.91 | 23.81
Teacher | Faster R-CNN w/R50-FPN | 40.22 | 61.02 | 43.81 | 51.98 | 43.53 | 24.16
Student | Faster R-CNN w/MV2-FPN | 29.47 | 48.87 | 30.90 | 38.86 | 30.77 | 16.33
| w/KD | 30.13(+0.66) | 50.28 | 31.35 | 39.56 | 31.91 | 16.69
| w/FitNets | 30.20(+0.73) | 49.8 | 31.69 | 39.69 | 31.64 | 16.39
| w/FGFI | 31.16(+1.69) | 50.68 | 32.92 | 42.12 | 32.63 | 16.73
| w/ReviewKD | 33.71(+4.24) | 53.15 | 36.13 | 46.47 | 35.81 | 16.77
| w/ours+ReviewKD | 34.03(+4.56) | 53.67 | 36.33 | 45.72 | 36.53 | 18.03
Table 4: Results on object detection. We use AP on different settings to
evaluate results. R101 represents using ResNet-101 as backbone, and MV2 stands
for MobileNetV2.

### Ablation Studies

Feature maps as knowledge sequence. In our distillation method, we extract
spatiotemporal features from the increment sequence rather than feature map
sequence. The reason we choose the increment is that it can filter the
irrelevant information in feature maps. The network will pay more attention to
the new content in the progressive learning. The experiments show that using
feature maps themselves as sequences also brings improvements, but increment
groups have better performance (Table 5). In fact, first-order differential is
a common way to process raw data in classical time series analysis problem,
which will enhance the temporal property of the sequence. The experimental
results are consistent with this theory.

T-S Structure | fm. seq | inc. seq
---|---|---
ResNet-56 to ResNet-20 | 71.19 | 71.76
WRN40-2 to WRN40-1 | 74.30 | 74.49
Table 5: Comparison results on different types of sequence. incre. and fm.
denote increment and feature map respectively.

Effects of memory interval. The memory interval $\delta$ determines how many
epochs of general training are performed between memory nodes. When $\delta$
is relatively high, the learning period recorded in the knowledge increment
sequence will become longer. This may make the review more difficult.
Different settings of $\delta$ are explored in Table 6.

Interval $\delta$ | ACC($\%$)
---|---
5 | 71.76(+2.70)
10 | 71.28(+2.22)
15 | 71.19(+2.13)
Table 6: Ablation studies on memory interval $\delta$ on CIFAR-100. The
student and teacher models are ResNet-56 and ResNet-20 respectively.

Effects of memory nodes. In order to investigate how many memory nodes are
proper in one review, we compare different settings of $k$ in Table 7.
Obviously, the knowledge sequence will become longer as the number of memory
nodes increases. However, given the same training time, the frequency of
review will also decrease accordingly. On the other hand, too few memory nodes
can also make the temporal property in the sequence not clear enough. The
R56-R20 experiment shows the highest accuracy when $k=6$.

| Value of $k$ | ACC($\%$)
---|---|---
| 3 | 71.76(+2.70)
| 6 | 71.81(+2.75)
| 9 | 71.78(+2.72)
Table 7: Ablation studies on the number of memory nodes on CIFAR-100. The
student and teacher models are ResNet-56 and ResNet-20 respectively.

### Extensions

Visualizations. We visualize the deep features of student network WRN-16-2
(distilled from WRN-40-2 on CIFAR-100). The t-SNE (Fig. 5) results show that
representations of TSKD are more separable than general KD, which proves that
student trained by TSKD benefits from the discriminability of features.

Figure 5: t-SNE of features learned by KD (left) and TSKD (right).

Efficiency. We compare the training cost of the proposed TSKD with state-of-
the-art feature-based distillation methods. Specifically, we record the
average training time per batch in a whole memory-review cycle. Different from
existing methods which perform distillation throughout the entire training
process, our TSKD, instead, only needs to distill in review nodes. Therefore,
the computational costs of training student networks have been reduced. As the
results reported in Fig. 5, our method has better trade-off between training
time and accuracy.

Figure 6: Training time (per batch) vs. accuracy on CIFAR-100. We set
ResNet32x4 as the teacher and ResNet8x4 as the student.

Transferability. We evaluate the generalizability of distilled student
network. A primary goal of representation learning is to acquire general
knowledge that is not only useful in the current dataset, but also in
dataset/tasks from other domains. Therefore, we test if the knowledge
distilled by TSKD transfer well. In the experiment, we use WRN-16-2 as the
frozen representation extractors, which is either trained from scratch on
CIFAR-100 or distilled from WRN-40-2 with various KD methods. We then perform
linear probing tasks on TinyImageNet222https://www.kaggle.com/c/tiny-imagenet.
As the results reported in Table 8, TSKD outperforms other methods by a
certain margin, demonstrating its strong generalizability.

baseline | KD | FitNets | CRD | ReviewKD | DKD | TSKD
---|---|---|---|---|---|---
33.7 | 33.9 | 33.5 | 35.6 | 36.6 | 37.1 | 38.5
Table 8: Comparison with previous methods on transferring representations from
CIAFR-100 to TinyImageNet.

## Conclusion

This paper deals with the problem of knowledge distillation from a novel
perspective. We find that there exists temporal pattern in the evolution of
network knowledge. Motivated by this observation, we propose Temporal
Supervised Knowledge Distillation (TSKD) to solve two questions: (1) How to
utilize the old knowledge to assist current learning? and (2) How to supervise
the temporal learning process of network? The student trained by TSKD achieves
significant improvements on CIFAR-100, ImageNet and MS-COCO datasets for image
classification and object detection tasks. Besides, TSKD also shows
superiority on training efficiency and knowledge transferability. We hope our
work will help future research on knowledge distillation and interpretable
deep learning.

## References

  * Ahn et al. (2019) Ahn, S.; Hu, S. X.; Damianou, A.; Lawrence, N. D.; and Dai, Z. 2019.  Variational information distillation for knowledge transfer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 9163–9171.
  * Box and Pierce (1970) Box, G. E.; and Pierce, D. A. 1970.  Distribution of residual autocorrelations in autoregressive-integrated moving average time series models.  _Journal of the American statistical Association_ , 65(332): 1509–1526.
  * Chen et al. (2017) Chen, L.-C.; Papandreou, G.; Schroff, F.; and Adam, H. 2017.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_.
  * Chen et al. (2021) Chen, P.; Liu, S.; Zhao, H.; and Jia, J. 2021.  Distilling knowledge via knowledge review.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5008–5017.
  * Cho and Hariharan (2019) Cho, J. H.; and Hariharan, B. 2019.  On the efficacy of knowledge distillation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 4794–4802.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Furlanello et al. (2018) Furlanello, T.; Lipton, Z.; Tschannen, M.; Itti, L.; and Anandkumar, A. 2018.  Born again neural networks.  In _International Conference on Machine Learning_ , 1607–1616. PMLR.
  * Girshick (2015) Girshick, R. 2015.  Fast r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , 1440–1448.
  * Guo et al. (2023) Guo, Z.; Yan, H.; Li, H.; and Lin, X. 2023.  Class Attention Transfer Based Knowledge Distillation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 11868–11877.
  * He et al. (2017) He, K.; Gkioxari, G.; Dollár, P.; and Girshick, R. 2017.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , 2961–2969.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Heo et al. (2019) Heo, B.; Kim, J.; Yun, S.; Park, H.; Kwak, N.; and Choi, J. Y. 2019.  A comprehensive overhaul of feature distillation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 1921–1930.
  * Hinton, Vinyals, and Dean (2015) Hinton, G.; Vinyals, O.; and Dean, J. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Ji, Heo, and Park (2021) Ji, M.; Heo, B.; and Park, S. 2021.  Show, attend and distill: Knowledge distillation via attention-based feature matching.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 7945–7952.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Krizhevsky, Sutskever, and Hinton (2017) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2017.  Imagenet classification with deep convolutional neural networks.  _Communications of the ACM_ , 60(6): 84–90.
  * LeCun, Bengio, and Hinton (2015) LeCun, Y.; Bengio, Y.; and Hinton, G. 2015.  Deep learning.  _nature_ , 521(7553): 436–444.
  * Li, Jin, and Yan (2017) Li, Q.; Jin, S.; and Yan, J. 2017.  Mimicking very efficient network for object detection.  In _Proceedings of the ieee conference on computer vision and pattern recognition_ , 6356–6364.
  * Li et al. (2021) Li, Z.; Ye, J.; Song, M.; Huang, Y.; and Pan, Z. 2021.  Online knowledge distillation for efficient pose estimation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 11740–11750.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , 740–755. Springer.
  * Mirzadeh et al. (2020) Mirzadeh, S. I.; Farajtabar, M.; Li, A.; Levine, N.; Matsukawa, A.; and Ghasemzadeh, H. 2020.  Improved knowledge distillation via teacher assistant.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, 5191–5198.
  * Park et al. (2019) Park, W.; Kim, D.; Lu, Y.; and Cho, M. 2019.  Relational knowledge distillation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3967–3976.
  * Passalis and Tefas (2018) Passalis, N.; and Tefas, A. 2018.  Probabilistic knowledge transfer for deep representation learning.  _CoRR, abs/1803.10837_ , 1(2): 5.
  * Peng et al. (2019) Peng, B.; Jin, X.; Liu, J.; Li, D.; Wu, Y.; Liu, Y.; Zhou, S.; and Zhang, Z. 2019\.  Correlation congruence for knowledge distillation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5007–5016.
  * Redmon et al. (2016) Redmon, J.; Divvala, S.; Girshick, R.; and Farhadi, A. 2016.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 779–788.
  * Romero et al. (2014) Romero, A.; Ballas, N.; Kahou, S. E.; Chassang, A.; Gatta, C.; and Bengio, Y. 2014\.  Fitnets: Hints for thin deep nets.  _arXiv preprint arXiv:1412.6550_.
  * Sandler et al. (2018) Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C. 2018.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4510–4520.
  * Shi et al. (2015) Shi, X.; Chen, Z.; Wang, H.; Yeung, D.-Y.; Wong, W.-K.; and Woo, W.-c. 2015.  Convolutional LSTM network: A machine learning approach for precipitation nowcasting.  _Advances in neural information processing systems_ , 28.
  * Simonyan and Zisserman (2014) Simonyan, K.; and Zisserman, A. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Tian, Krishnan, and Isola (2019) Tian, Y.; Krishnan, D.; and Isola, P. 2019.  Contrastive representation distillation.  _arXiv preprint arXiv:1910.10699_.
  * Tung and Mori (2019) Tung, F.; and Mori, G. 2019.  Similarity-preserving knowledge distillation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 1365–1374.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2019) Wang, T.; Yuan, L.; Zhang, X.; and Feng, J. 2019.  detectron.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4933–4942.
  * Xie et al. (2017) Xie, S.; Girshick, R.; Dollár, P.; Tu, Z.; and He, K. 2017.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1492–1500.
  * You et al. (2020) You, S.; Huang, T.; Yang, M.; Wang, F.; Qian, C.; and Zhang, C. 2020.  Greedynas: Towards fast one-shot nas with greedy supernet.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1999–2008.
  * Zagoruyko and Komodakis (2016a) Zagoruyko, S.; and Komodakis, N. 2016a.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  _arXiv preprint arXiv:1612.03928_.
  * Zagoruyko and Komodakis (2016b) Zagoruyko, S.; and Komodakis, N. 2016b.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_.
  * Zhang et al. (2018) Zhang, X.; Zhou, X.; Lin, M.; and Sun, J. 2018.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 6848–6856.
  * Zhao et al. (2022) Zhao, B.; Cui, Q.; Song, R.; Qiu, Y.; and Liang, J. 2022.  Decoupled knowledge distillation.  In _Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition_ , 11953–11962.
  * Zhao et al. (2017) Zhao, H.; Shi, J.; Qi, X.; Wang, X.; and Jia, J. 2017.  Pyramid scene parsing network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2881–2890.