# Do short range correlations inhibit the appearance of the nuclear pasta?

M. R. Pelicer1, D. P. Menezes1, M. Dutra2, and O. Lourenço2 1Depto de Física -
CFM - Universidade Federal de Santa Catarina Florianópolis - SC - CP. 476 -
CEP 88.040 - 900 - Brazil
2Departamento de Física, Instituto Tecnológico de Aeronáutica, DCTA,
12228-900, São José dos Campos, SP, Brazil

###### Abstract

It is well known that strongly correlated neutron-proton pairs, the short-
range correlations (SRC), can modify many of the nuclear properties. In this
work we have introduced, for the first time, short range correlations in the
calculation of the nuclear pasta phase at zero temperature and checked how
they affect its size and internal structure. We have used two different
parameterizations of relativistic models in a mean field approximation and the
coexistence phase approximation as a first estimation of the effects. We have
seen that for very asymmetric neutron-proton-electon matter, the pasta phase
shrinks considerably as compared with the results without SRC and all internal
structures vanish, except the simple spherically symmetric one, the droplets.
Our results indicate a possible disappearance of these complicated structures
as the temperature increases.

Introduction.—Nonspherical complex structures that appear due to a frustration
in sub-saturation nuclear densities are believed to be present in the inner
crust of neutron stars and in core-collapse supernova cores Ravenhall et al.
(1983); Horowitz et al. (2004). Although possibly present only in a small
range of densities and temperatures, different studies suggest that these
exotic structures may have considerable impact on different astrophysical
phenomena and on the magnetic evolution of neutron stars Pons et al. (2013).
Some of the consequences of the existence of the pasta phase are expected to
leave signatures in quasiperiodic oscillations observed in soft-gamma ray
repeaters Sotani (2011) and magnetar giant flares Horowitz (2010). More
recently, the detection of late-time neutrinos from a Milky Way core-collapse
supernova seems to be approaching reality Li et al. (2021) and it was claimed
that neutrino diffusion seems to be affected by the pasta phase in
protoneutron stars in such a way that its signal can be greatly enhanced after
core collapse Horowitz et al. (2016).

From the theoretical point of view, there are some models and simulations that
predict that the pasta structures can be very complex indeed. Calculations
departed from the more traditional 3D, 2D and 1D geometries in a single unit
cell to density fluctuations Barros et al. (2020); Pelicer et al. (2021) as
well as to structures resembling waffle, parking garage and TPMS Fattoyev et
al. (2017); Schneider et al. (2018); Schuetrumpf et al. (2019). Different
calculations also show that the pasta phase size decreases as temperature
increases and may even occupy just a small portion in between two homogeneous
phases at certain temperatures Avancini et al. (2008, 2010, 2012).

Most of the pasta phase calculations depend on equations of state (EOS)
parameterized to satisfy nuclear matter bulk properties. One missing
ingredient in all pasta calculations is the inclusion of short-range
correlations (SRC). Strongly correlated neutron-proton pairs can modify the
internal structure of the nucleus and generate a series of non trivial
consequences Hen et al. (2014); CLAS Collaboration (2018, 2019); Schmidt et
al. (2020); Hen et al. (2017); Duer et al. (2019); Li et al. (2008); Cai and
Li (2016); Guo et al. (2021); Cai and Li (2022a, b); Souza et al. (2020);
Lourenço et al. (2022a, b); Dutra et al. (2022). In a seminal paper on cold
dense matter obtained with the inclusion of SRC Subedi et al. (2008), the
authors claimed that the difference between the types of pairs (proton-proton,
neutron-neutron and proton-neutron) is due to the nature of the strong force
and has implications for understanding cold dense nuclear systems such as
neutron stars. Hence, the introduction of SRC in the pasta phase structure is
long due.

So…what if the complicated pasta structures are simply not there when SRC are
taken into account? As a first study, we investigate the effects of the SRC on
the pasta phase and its structure with a simple prescription, the coexistence
phase approximation (CPA) Avancini et al. (2008) within relativistic mean
field models at zero temperature.

SRC (homogeneous phase).— We start by studying the effects of the SRC on the
homogeneous phase with two different parametrizations, namely IUFSU Fattoyev
et al. (2010) and FSU2R Tolos et al. (2017). When SRC are included, the
momentum ($k$) distribution present in the kinetic terms of the energy
density, pressure and scalar density is modified by the inclusion of a “high
momentum tail” proportional to $k^{-4}$ and, as a consequence, the model has
to be reparameterized in order to reproduce nuclear bulk properties. The
Lagrangian density used in the present work, the new expressions for the
quantities necessary to compute the EOS and the new parameters for the FSU2R
model are given in Souza et al. (2020); Lourenço et al. (2022a, b); Dutra et
al. (2022). For the IUFSU, the constants for the model with and without SRC
are given in Table LABEL:tabconst.

Table 1: Coupling constants of the IUFSU parametrization with and without SRC included. $M_{\mbox{\tiny nuc}}=939$ MeV is the nucleon rest mass. In both cases, the last constant is fixed in $C=5.000\times 10^{-3}$. coupling | IUFSU | IUFSU-SRC
---|---|---
$g_{\sigma}$ | $9.971$ | $10.132$
$g_{\omega}$ | $13.032$ | $11.867$
$g_{\rho}$ | $13.590$ | $15.551$
$A/M_{\mbox{\tiny nuc}}$ | $1.785$ | $2.956$
$B$ | $0.488$ | $-29.880$
$\alpha_{3}^{\prime}$ ($\times 10^{-2}$) | $9.200$ | $1.094$

As the results obtained with the two models are qualitatively similar, we
display the graphs for the IUFSU parameterization only. We have chosen to show
results for symmetric matter (proton fraction $Y_{p}=0.5$), $Y_{p}=0.35$,
usually considered in core-collapse supernova and $Y_{p}=0.1$, a quantity that
relates to the amount of protons in neutron star matter.

In Fig. 1, we compare the proton and neutron chemical potentials with and
without SRC for different proton fractions. It is notorious that very
asymmetric matter is more susceptible to changes than symmetric nuclear
matter.

Figure 1: Chemical potential of protons (top) and neutrons (bottom) for
homogeneous matter with the IUFSU parametrization for different proton
fractions, both without (dashed curves) and with (full curves) short range
correlations.

As the main objective of this work is to investigate the influence of SRC on
the pasta phase and the pressure is an important quantity in its construction,
in Fig. 2 we also show how the pressure behaves as a function of the density.
The differences at very low densities, the region important in the present
study, are shown in the inset, corroborating the statement that very
asymmetric matter feels the effect of SRC more strongly.

Figure 2: Pressure of homogeneous baryon matter for different proton fractions
with IUFSU without (dashed curves) and with (full curves) short range
correlations for different proton fractions.

SRC (pasta phase).— Once we have understood how the homogeneous matter is
affected, we can obtain the pasta phase and check its structure. In these
calculations, as generally done when homogeneous matter is compared with the
pasta phase, neutron-proton-electron (npe) matter is considered, i.e., not
only nucleons, but also electrons that are responsible for a charge neutral
matter, are taken into account.

As already mentioned, we use the simplest possible prescription, in which
Gibbs conditions are enforced and the electron gas remains in the background.
The two phases are generally referred as liquid (the denser one) and gas
phase. As the proton fraction decreases, a gas richer in neutrons starts to be
contaminated with protons as well.

The Gibbs conditions that have to be satisfied by the CPA method are given in
Avancini et al. (2008). The surface tension coefficient is a crucial quantity
in this formalism and we have used a fitting to the Thomas-Fermi calculation
as a function of the proton fraction calculated in Avancini et al. (2012). For
the FSU2R functional, we used a fixed surface tension $\sigma_{0}=1.2$ MeV
fm-2. The geometries considered are droplets and bubbles (3D), rods and tubes
(2D) and slabs (1D). In this work, fluctuations are not taken into account as
done in Barros et al. (2020); Pelicer et al. (2021), but an extension to take
them into account is planned for a future work. Hence, to obtain the
preferential ground state and its corresponding geometry at each density, the
homogeneous phase is compared with the pasta phase for each internal
structure. The lower one in free energy is the preferential matter.

In Fig. 3, our results are shown for the IUFSU model without (left) and with
(right) SRC. As a consequence of the already observed differences in
homogeneous matter, the larger the asymmetry, the stronger the effects of the
SRC. For symmetric matter, the results are very similar, both as far as the
size of the pasta phase as well as its internal structure. For $Y_{p}=0.35$,
the similarities disappear: not only the pasta phase shrinks when SRC are
included, but one of its internal structure, the tubes, vanishes. When we
analyse $Y_{p}=0.1$, the most important result for the neutron star inner
crust, the pasta phase diminishes even more and only droplets survive.

Figure 3: Baryon densities where each pasta geometry is dominant for the IUFSU
(left) and IUFSU-SRC (right) for proton fractions $Y_{p}=$0.5, 0.35 and 0.1
(left, center and right, respectively).

For the sake of completeness, we also show the free energy density for
different proton fractions without and with SRC in Fig. 4, from where the
densities related to each pasta structure are obtained.

At this point, a word of caution is necessary: more robust calculations in
which the surface tension can be self-consistently calculated, as explained in
Avancini et al. (2010, 2012), must be carried out when the SRC are included.
The modifications with respect to the fitting obtained without SRC are
certainly minor, but the quantitative results can be slightly modified.

Figure 4: Free energy density per baryon of homogeneous npe matter (dashed
curves) and pasta (full curves) with (top) and without (bottom) short range
correlations.

Analysing the present results in face of the knowledge we have accumulated
from previous works, we can guess two important outcomes, at least for the
inner crust of neutron stars: 1) as the temperature increases just to a few
keV, probably no pasta phase will survive if SRC are considered and if so 2)
the calculation of transport properties Yakovlev (2015); Pelicer et al.  can
be greatly simplified, as the breaking of the clusters spherical symmetry that
could lead to anisotropies in the collision frequencies would be greatly
inhibited. These features will be analysed in future works.

Acknowledgements.— This work is a part of the project INCT-FNA Proc. No.
464898/2014-5. D.P.M. is partially supported by Conselho Nacional de
Desenvolvimento Científico e Tecnológico (CNPq/Brazil) under grant
301155.2017-8 and M.R.P. is supported with a doctorate scholarship by
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Capes/Brazil).
The work is also supported by CNPq under Grants No. 312410/2020-4 (O.L.) and
No. 308528/2021-2 (M.D.). O.L. and M.D. also acknowledge Fundação de Amparo à
Pesquisa do Estado de São Paulo (FAPESP/Brazil) under Thematic Project
2017/05660-0 and Grant No. 2020/05238-9.

## References

  * Ravenhall et al. (1983) D. G. Ravenhall, C. J. Pethick, and J. R. Wilson, Phys. Rev. Lett. 50, 2066 (1983), URL https://link.aps.org/doi/10.1103/PhysRevLett.50.2066.
  * Horowitz et al. (2004) C. J. Horowitz, M. A. Pérez-García, J. Carriere, D. K. Berry, and J. Piekarewicz, Phys. Rev. C 70, 065806 (2004), URL https://link.aps.org/doi/10.1103/PhysRevC.70.065806.
  * Pons et al. (2013) J. Pons, D. Viganó, and N. Rea, Nature Physics pp. 431–434 (2013).
  * Sotani (2011) H. Sotani, Monthly Notices of the Royal Astronomical Society: Letters 417, L70 (2011), ISSN 1745-3925.
  * Horowitz (2010) C. J. Horowitz, Phys. Rev. D 81, 103001 (2010), URL https://link.aps.org/doi/10.1103/PhysRevD.81.103001.
  * Li et al. (2021) S. W. Li, L. F. Roberts, and J. F. Beacom, Phys. Rev. D 103, 023016 (2021), URL https://link.aps.org/doi/10.1103/PhysRevD.103.023016.
  * Horowitz et al. (2016) C. J. Horowitz, D. K. Berry, M. E. Caplan, T. Fischer, Z. Lin, W. G. Newton, E. O’Connor, and L. F. Roberts, _Nuclear pasta and supernova neutrinos at late times_ (2016), URL https://arxiv.org/abs/1611.10226.
  * Barros et al. (2020) C. C. Barros, D. P. Menezes, and F. Gulminelli, Phys. Rev. C 101, 035211 (2020), URL https://link.aps.org/doi/10.1103/PhysRevC.101.035211.
  * Pelicer et al. (2021) M. R. Pelicer, D. P. Menezes, C. C. Barros, and F. Gulminelli, Phys. Rev. C 104, L022801 (2021), URL https://link.aps.org/doi/10.1103/PhysRevC.104.L022801.
  * Fattoyev et al. (2017) F. J. Fattoyev, C. J. Horowitz, and B. Schuetrumpf, Phys. Rev. C 95, 055804 (2017), URL https://link.aps.org/doi/10.1103/PhysRevC.95.055804.
  * Schneider et al. (2018) A. S. Schneider, M. E. Caplan, D. K. Berry, and C. J. Horowitz, Phys. Rev. C 98, 055801 (2018), URL https://link.aps.org/doi/10.1103/PhysRevC.98.055801.
  * Schuetrumpf et al. (2019) B. Schuetrumpf, G. Martínez-Pinedo, M. Afibuzzaman, and H. M. Aktulga, Phys. Rev. C 100, 045806 (2019), URL https://link.aps.org/doi/10.1103/PhysRevC.100.045806.
  * Avancini et al. (2008) S. S. Avancini, D. P. Menezes, M. D. Alloy, J. R. Marinelli, M. M. W. Moraes, and C. Providência, Phys. Rev. C 78, 015802 (2008), URL https://link.aps.org/doi/10.1103/PhysRevC.78.015802.
  * Avancini et al. (2010) S. S. Avancini, S. Chiacchiera, D. P. Menezes, , and C. Providência, Phys. Rev. C 82, 055807 (2010), URL https://link.aps.org/doi/10.1103/PhysRevC.82.055807.
  * Avancini et al. (2012) S. S. Avancini, C. C. Barros, L. Brito, S. Chiacchiera, D. P. Menezes, and C. Providência, Phys. Rev. C 85, 035806 (2012), URL https://link.aps.org/doi/10.1103/PhysRevC.85.035806.
  * Hen et al. (2014) O. Hen et al., Science 346, 614 (2014), URL https://www.science.org/doi/abs/10.1126/science.1256785.
  * CLAS Collaboration (2018) CLAS Collaboration, Nature 560, 617 (2018).
  * CLAS Collaboration (2019) CLAS Collaboration, Nature 566, 354 (2019).
  * Schmidt et al. (2020) A. Schmidt et al., Nature 578, 540 (2020).
  * Hen et al. (2017) O. Hen, G. A. Miller, E. Piasetzky, and L. B. Weinstein, Rev. Mod. Phys. 89, 045002 (2017), URL https://link.aps.org/doi/10.1103/RevModPhys.89.045002.
  * Duer et al. (2019) M. Duer et al., Physics Letters B 797, 134792 (2019).
  * Li et al. (2008) B.-A. Li, L.-W. Chen, and C. M. Ko, Physics Reports 464, 113 (2008), ISSN 0370-1573, URL https://www.sciencedirect.com/science/article/pii/S0370157308001269.
  * Cai and Li (2016) B.-J. Cai and B.-A. Li, Phys. Rev. C 93, 014619 (2016), URL https://link.aps.org/doi/10.1103/PhysRevC.93.014619.
  * Guo et al. (2021) W.-M. Guo, B.-A. Li, and G.-C. Yong, Phys. Rev. C 104, 034603 (2021), URL https://link.aps.org/doi/10.1103/PhysRevC.104.034603.
  * Cai and Li (2022a) B.-J. Cai and B.-A. Li, Phys. Rev. C 105, 064607 (2022a), URL https://link.aps.org/doi/10.1103/PhysRevC.105.064607.
  * Cai and Li (2022b) B.-J. Cai and B.-A. Li, Annals of Physics 444, 169062 (2022b), ISSN 0003-4916, URL https://www.sciencedirect.com/science/article/pii/S000349162200183X.
  * Souza et al. (2020) L. A. Souza, M. Dutra, C. H. Lenzi, and O. Lourenço, Phys. Rev. C 101, 065202 (2020), URL https://link.aps.org/doi/10.1103/PhysRevC.101.065202.
  * Lourenço et al. (2022a) O. Lourenço, T. Frederico, and M. Dutra, Phys. Rev. D 105, 023008 (2022a), URL https://link.aps.org/doi/10.1103/PhysRevD.105.023008.
  * Lourenço et al. (2022b) O. Lourenço, C. H. Lenzi, T. Frederico, and M. Dutra, Phys. Rev. D 106, 043010 (2022b), URL https://link.aps.org/doi/10.1103/PhysRevD.106.043010.
  * Dutra et al. (2022) M. Dutra, C. H. Lenzi, and O. Lourenço, Monthly Notices of the Royal Astronomical Society 517, 4265 (2022), ISSN 0035-8711, URL https://doi.org/10.1093/mnras/stac2986.
  * Subedi et al. (2008) R. Subedi et al., Science 320, 1476 (2008), URL https://www.science.org/doi/abs/10.1126/science.1156675.
  * Fattoyev et al. (2010) F. J. Fattoyev, C. J. Horowitz, J. Piekarewicz, and G. Shen, Phys. Rev. C 82, 055803 (2010), URL https://link.aps.org/doi/10.1103/PhysRevC.82.055803.
  * Tolos et al. (2017) L. Tolos, M. Centelles, and A. Ramos, Publications of the Astronomical Society of Australia 34, e065 (2017), URL https://doi.org/10.1017/pasa.2017.60.
  * Yakovlev (2015) D. G. Yakovlev, Monthly Notices of the Royal Astronomical Society 453, 581 (2015), ISSN 0035-8711, eprint https://academic.oup.com/mnras/article-pdf/453/1/581/4929833/stv1642.pdf, URL https://doi.org/10.1093/mnras/stv1642.
  * (35) M. R. Pelicer, M. Antonelli, D. P. Menezes, and F. Gulminelli, _in preparation_.


# Relational Representation Distillation

Nikolaos Giakoumoglou
Imperial College London
London, UK, SW7 2AZ
<EMAIL_ADDRESS>Tania Stathaki
Imperial College London
London, UK, SW7 2AZ
<EMAIL_ADDRESS>
###### Abstract

Knowledge distillation (KD) is an effective method for transferring knowledge
from a large, well-trained teacher model to a smaller, more efficient student
model. Despite its success, one of the main challenges in KD is ensuring the
efficient transfer of complex knowledge while maintaining the student’s
computational efficiency. Unlike previous works that applied contrastive
objectives promoting explicit negative instances, we introduce Relational
Representation Distillation (RRD). Our approach leverages pairwise
similarities to explore and reinforce the relationships between the teacher
and student models. Inspired by self-supervised learning principles, it uses a
relaxed contrastive loss that focuses on similarity rather than exact
replication. This method aligns the output distributions of teacher samples in
a large memory buffer, improving the robustness and performance of the student
model without the need for strict negative instance differentiation. Our
approach demonstrates superior performance on CIFAR-100, outperforming
traditional KD techniques and surpassing 13 state-of-the-art methods. It also
transfers successfully to other datasets like Tiny ImageNet and STL-10. The
code will be made public soon.

##  1 Introduction

Knowledge distillation (KD) is a technique that facilitates the transfer of
knowledge from a larger, well-trained model (teacher) to a smaller, more
efficient model (student). This is achieved by minimizing the Kullback-Leibler
(KL) divergence between their outputs, allowing the student model to
approximate the performance of the teacher model while maintaining lower
computational complexity. This process is particularly beneficial for
deployment in resource-constrained environments. A critical aspect of KD is
representation learning, which enables the student model to acquire meaningful
feature representations that capture the underlying data distribution.
Effective representation learning in KD can significantly boost the
performance of the student model across various domains, such as natural
language processing, computer vision, and speech recognition [10, 22, 15].
Despite these advantages, a major challenge in KD is the efficient transfer of
complex knowledge from the teacher to the student model. Ensuring that the
student model captures the abstract features and nuanced information present
in the teacher model, without the need for similar computational capacity,
remains a significant bottleneck.

Recent advancements in KD have introduced various strategies to enhance the
transfer process, addressing the bottlenecks associated with representation
learning. Techniques such as adversarial training, attention transfer, and
contrastive representation distillation have been proposed to further align
the representations learned by the student model with those of the teacher
model. Adversarial training involves using adversarial examples to improve the
robustness of the student model, while attention transfer focuses on aligning
the attention maps of the teacher and student models to ensure that both
models focus on similar regions of the input data [32, 26]. Contrastive
representation distillation aims to improve the quality of learned
representations by encouraging the student model to produce similar
representations for similar inputs while differentiating dissimilar inputs.
Additionally, methods like BookKD reduce distillation costs by decoupling
knowledge generation and learning processes, leading to improved performance
with minimal resource consumption [36]. Other approaches like structured KD
focus on training compact networks by distilling structured knowledge from
cumbersome networks [15]. These methods aim to capture richer information from
the teacher model, leading to more robust and generalized student models. The
continuous evolution of KD techniques underscores its pivotal role in
developing efficient deep learning models capable of high performance with
lower computational costs, while overcoming the challenges of effective
knowledge transfer.

Our proposed method, Relational Representation Distillation (RRD), introduces
a novel approach to address these challenges by maintaining relational
consistency between the teacher and student models. By leveraging a large
memory buffer of teacher samples to align their output distributions, our
method ensures consistent relational structures, thereby enhancing the
robustness and performance of the student model.

Our contributions are threefold:

  1. 1.

We propose a novel relational consistency framework for KD that leverages a
memory buffer to maintain relational structures between teacher and student
models.

  2. 2.

We introduce a relational consistency loss that aligns the similarity
distributions of teacher and student outputs, improving the robustness and
generalization of the student model.

  3. 3.

We validate the effectiveness of RRD hthrough comprehensive testing on
standard benchmarks, showcasing considerable gains in both accuracy and
robustness. RRD surpasses other methods with a 19.22% relative
improvement111Average relative improvement is calculated as:
$\frac{1}{N}\sum_{i=1}^{N}\frac{\text{Acc}^{i}{\text{RRD}}-\text{Acc}^{i}{\text{KD}}}{\text{Acc}^{i}{\text{KD}}-\text{Acc}^{i}{\text{van}}}$,
where $\text{Acc}^{i}{\text{RRD}}$, $\text{Acc}^{i}{\text{KD}}$, and
$\text{Acc}^{i}_{\text{van}}$ represent the accuracies of RRD, KD, and vanilla
training of the $i$-th student model, respectively [27]. over conventional KD.
When integrated with KD, it demonstrates a 55.18% relative improvement over
standard KD, underscoring its robust potential in enhancing model performance
significantly.

The rest of this paper is organized as follows. Section 2 reviews related work
in KD and contrastive learning. Section 3 details our proposed methodology.
Section 4 presents our experimental setup and results, and Section 5 concludes
the paper.

##  2 Related Work

The study of [10] on KD introduced the idea of transferring knowledge from
large, complex models to smaller, more efficient models without losing their
generalization capabilities. This technique involves temperature scaling in
the softmax outputs to effectively capture and convey the knowledge from the
teacher model. Numerous enhancements to this method have been proposed. For
example, using intermediate representations or "hints" to guide the learning
process [22], and aligning the attention maps of teacher and student models to
ensure they focus on similar areas during training [32]. Other methods
preserve relational knowledge between samples [28] or align the correlation
structures between teacher and student models [20]. Additionally, some
techniques use variational inference to improve the knowledge transfer process
[1]. Further developments include focusing on structural relationships between
data points to ensure the student model learns relational information [18],
and maintaining the internal dynamics of neural networks during distillation
[19, 9]. Other notable techniques include network compression through factor
transfer [12], optimizing networks for better transfer learning [31],
promoting selectivity in the distillation process [11], and using contrastive
learning objectives to enhance representation learning [27].

Self-supervised learning has significantly impacted representation learning by
leveraging unlabeled data. Methods such as SimCLR [3] and MoCo [7] use
contrastive losses to learn useful representations by maximizing agreement
between different augmented views of the same data point. ReSSL [35]
introduces relational self-supervised learning, which explores the
relationships between data points. These approaches have inspired various KD
methods, including our proposed method, which adapts relational consistency
from self-supervised learning to the KD framework.

Our method, RRD, differentiates itself from state-of-the-art methods by
focusing on maintaining relational consistency between the teacher and student
models. Unlike traditional KD methods that often rely on direct alignment of
logits or intermediate features, RRD leverages a large memory buffer of
teacher samples to align the relational structures of the output
distributions. This approach not only enhances the robustness and performance
of the student model but also provides a more flexible and scalable solution
for KD.

##  3 Methodology

This section presents our methodology to improve the efficiency and accuracy
of KD. Our method, Relational Representation Distillation (RRD), focuses on
maintaining relational consistency between the teacher and student models by
leveraging a large memory buffer of teacher samples to align their output
distributions. By ensuring consistent relational structures, RRD enhances the
robustness and performance of the student model. Figure 1 shows an overview of
the proposed RRD method.

Figure 1: Overview of the RRD method. The student network processes input
$x_{i}$ to produce embeddings $z_{i}^{S}$, while the teacher network,
indicated as frozen by a snowflake, processes the same input to generate
embeddings $z_{i}^{T}$. The embeddings are stored in a memory buffer to align
their output distributions via softmax layers. The relational consistency
between the teacher and student models is enforced by leveraging the stored
embeddings to train the student network.

###  3.1 Preliminary

KD involves transferring knowledge from a high-capacity teacher neural
network, denoted as $f^{T}$, to a more compact student neural network,
$f^{S}$. Consider $x_{i}$ as the input to these networks, typically an image.
We represent the outputs at the penultimate layer (just before the final
classification layer, or logits) as $z_{i}^{T}=f^{T}(x_{i})$ and
$z_{i}^{S}=f^{S}(x_{i})$ for the teacher and student models, respectively. The
primary objective of KD is to enable the student model to approximate the
performance of the teacher model while leveraging the student’s computational
efficiency. The overall distillation process can be mathematically expressed
as:

$\mathcal{L}=\mathcal{L}_{\text{sup}}(y_{i},z_{i}^{S})+\lambda\cdot\mathcal{L}_{\text{distill}}(z_{i}^{T},z_{i}^{S})$
(1)

where $y_{i}$ represents the true label for the input $x_{i}$ and $\lambda$ is
a hyperparameter that balances the supervised loss and the distillation loss.
The loss $\mathcal{L}_{\text{sup}}$ is the alignment error between the network
prediction and the annotation. For example, in the image classification task
[17, 24, 21, 4], it is normally a cross-entropy loss. For object detection
[14, 2], it includes bounding box regression as well. The loss
$\mathcal{L}_{\text{distill}}$ is the mimic error of the student network
towards a pre-trained teacher network, typically implemented as KL divergence
between student and teacher outputs [10].

###  3.2 Relational consistency

Our method is inspired by self-supervised learning. Traditional contrastive
self-supervised learning uses instance discrimination as a pretext task and
relies on ($K+1$)-softmax classification, where different instances are pushed
apart, and augmented views of the same instance are expected to have identical
features. Applying such properties to KD imposes overly strict constraints,
where a contrastive loss encourages the representations from the teacher and
student models for the same input data to be similar, while simultaneously
pushing apart representations from different data inputs:

$\mathcal{L}_{\text{contrast}}(z_{i}^{T},z_{i}^{S})=-\log\frac{\exp(\phi(z_{i}^{T},z_{i}^{S})/\tau)}{\sum_{j=1}^{M}\exp(\phi(z_{i}^{S},z_{j}^{T})/\tau)}$
(2)

where $\mathcal{L}_{\text{contrast}}$ is the InfoNCE [29] loss, typically
employed in self-supervised methods, $\phi$ is a similarity function, $\tau$
is a temperature parameter, and $M$ is the number of negative samples.

In this way, we do not encourage explicit negative instances (those to be
pushed away) for each instance; instead, we leverage the pairwise similarities
to explore their relationships. We pull the features of the student $f^{S}$
and teacher $f^{T}$. As a result, our method relaxes Equation 2, where
different teacher and student outputs do not always need to be pushed away
from each other; and teacher and student need to share similar but not exactly
the same features.

Concretely, given an input image $x_{i}$, and the outputs
$z_{i}^{T}=f^{T}(x_{i})$ and $z_{i}^{S}=f^{S}(x_{i})$ for the teacher and
student models, respectively, we calculate their similarity measured by
$\phi(z_{i}^{T},z_{i}^{S})$. A softmax layer can be adopted to process the
calculated similarities, which then produces a relationship distribution:

$p_{i}^{T}=\frac{\exp(\phi(z_{i}^{T},z_{j}^{T})/\tau_{t})}{\sum_{k=1}^{M}\exp(\phi(z_{i}^{T},z_{k}^{T})/\tau_{t})}$
(3)

where $\tau_{t}$ is the temperature parameter for the teacher network. At the
same time, we can calculate the relationship distribution for the student
model as:

$p_{i}^{S}=\frac{\exp(\phi(z_{i}^{S},z_{j}^{S})/\tau_{s})}{\sum_{k=1}^{M}\exp(\phi(z_{i}^{S},z_{k}^{S})/\tau_{s})}$
(4)

where $\tau_{s}$ is a different temperature parameter for the student network.
We propose to push the relational consistency between $p_{i}^{T}$ and
$p_{i}^{S}$, similar to [35], by minimizing the KL divergence, which can be
formulated as:

$\mathcal{L}_{\text{relational}}(z_{i}^{T},z_{i}^{S})=\textit{KL}(p_{i}^{T}\parallel
p_{i}^{S})=H(p_{i}^{T},p_{i}^{S})-H(p_{i}^{T})$ (5)

where KL denotes the KL divergence between $p_{i}^{T}$ and $p_{i}^{S}$. Since
$p_{i}^{T}$ will be used as a target, the gradient will be clipped here to
avoid the model collapsing, thus we only minimize the cross-entropy term
$H(p_{i}^{T},p_{i}^{S})$ in our implementation.

However, the quality of the target similarity distribution $p_{i}^{T}$ is
crucial for reliable and stable training. To ensure this, we maintain a large
memory buffer $Q$, storing the feature embeddings from teacher batches. The
relational consistency between the teacher and student models is enforced by
aligning the similarity distributions of their outputs using the KL
divergence.

We normalize the outputs $z_{i}^{T}$ and $z_{i}^{S}$ before computing the
loss, ensuring that the representations lie on a unit hypersphere. To compute
$\mathcal{L}_{\text{relational}}$, we further encode $z_{i}^{T}$ and
$z_{i}^{S}$ using a projection head to match the dimensions. This ensures that
the representations from both models are compatible for comparison and
alignment.

The final objective function, which includes the supervised loss and standard
KL divergence, is given by:

$\mathcal{L}=\mathcal{L}_{\text{sup}}(y_{i},z_{i}^{S})+\lambda\cdot\mathcal{L}_{\text{distill}}(z_{i}^{T},z_{i}^{S})+\beta\cdot\mathcal{L}_{\text{relational}}(z_{i}^{T},z_{i}^{S})$
(6)

where $\beta$ is a hyperparameter that balances the KD loss
$\mathcal{L}_{\text{relational}}$. We experiment with $\beta$ in ablation
studies to understand its impact on the final performance.

##  4 Experiments

We evaluate our RRD framework in the KD task of model compression of a large
network to a smaller one.

Datasets. (1) CIFAR-100 [13] contains 50,000 training images with 500 images
per class and 10,000 test images. (2) STL-10 [5] consists of a training set of
5,000 labeled images from 10 classes and 100,000 unlabeled images, and a test
set of 8,000 images. (3) Tiny ImageNet (TIN-200) [6] has 200 classes, each
with 500 training images and 50 validaton images.

Setup. We experiment on CIFAR-100 with student-teacher combinations of various
capacity, such as ResNet [8] or Wide ResNet (WRN) [33], VGG [25], MobileNet
[23], and ShuffleNet [34, 16] (more details about the network architectures
are described in the supplementary material). We use $M=16384$ negative
samples and set the temperature parameter of the student to $\tau_{s}=0.04$
and of the teacher to $\tau_{t}=0.07$. Both the student and teacher outputs
are projected to a 128-dimensional space. We use a projection head of a single
linear layer, followed by $\ell_{2}$ normalization. We train for a total of
240 epochs. More details on the training details are described in the
supplementary material.

###  4.1 Results on CIFAR-100

Table 1 and Table 2 present the top-1 accuracies of student networks trained
using different distillation techniques across various teacher-student
architectural pairings. Table 1 examines pairings where both student and
teacher models share similar architectural styles, while Table 2 focuses on
cross-architecture distillations. Our proposed loss consistently outperforms
the conventional KD technique, as indicated by the green upward arrows
($\uparrow$). While the standalone performance of our method is comparable to
CRD, its integration with KD not only achieves higher accuracies but in some
cases, surpasses the performance of the teacher networks, such as in the
distillation of WRN-40-2 to ShuffleNet-v1. The enhanced performance of our
distillation method can be credited to multiple factors that collectively
improve the transfer of knowledge from the teacher to the student model. Our
approach uses a unique loss function that complements KD’s primary focus on
matching the softened output logits of the teacher and student. We introduce
an additional layer of representational alignment that ensures not only the
final outputs but also the intermediate feature representations of the student
closely match those of the teacher. This dual focus allows the student model
to mimic the teacher’s outputs and develop more robust and generalizable
internal representations.

Teacher | WRN-40-2 | WRN-40-2 | resnet-56 | resnet-110 | resnet-110 | resnet-32x4 | VGG-13
---|---|---|---|---|---|---|---
Student | WRN-16-2 | WRN-40-1 | resnet-20 | resnet-20 | resnet-32 | resnet-8x4 | VGG-8
Teacher | 75.61 | 75.61 | 72.34 | 74.31 | 74.31 | 79.42 | 74.64
Student | 73.26 | 71.98 | 69.06 | 69.06 | 71.14 | 72.50 | 70.36
KD | 74.92 | 73.54 | 70.66 | 70.67 | 73.08 | 73.33 | 72.98
FitNet | 73.58 ($\downarrow$) | 72.24 ($\downarrow$) | 69.21 ($\downarrow$) | 68.99 ($\downarrow$) | 71.06 ($\downarrow$) | 73.50 ($\downarrow$) | 71.02 ($\downarrow$)
AT | 74.08 ($\downarrow$) | 72.77 ($\downarrow$) | 70.55 ($\downarrow$) | 70.22 ($\downarrow$) | 72.31 ($\downarrow$) | 73.44 ($\downarrow$) | 71.43 ($\downarrow$)
SP | 73.83 ($\downarrow$) | 72.43 ($\downarrow$) | 69.67 ($\downarrow$) | 70.04 ($\downarrow$) | 72.69 ($\downarrow$) | 72.94 ($\downarrow$) | 72.68 ($\downarrow$)
CC | 73.56 ($\downarrow$) | 72.21 ($\downarrow$) | 69.63 ($\downarrow$) | 69.48 ($\downarrow$) | 71.48 ($\downarrow$) | 72.97 ($\downarrow$) | 70.81 ($\downarrow$)
VID | 74.11 ($\downarrow$) | 73.30 ($\downarrow$) | 70.38 ($\downarrow$) | 70.16 ($\downarrow$) | 72.61 ($\downarrow$) | 73.09 ($\downarrow$) | 71.23 ($\downarrow$)
RKD | 73.35 ($\downarrow$) | 72.22 ($\downarrow$) | 69.61 ($\downarrow$) | 69.25 ($\downarrow$) | 71.82 ($\downarrow$) | 71.90 ($\downarrow$) | 71.48 ($\downarrow$)
PKT | 74.54 ($\downarrow$) | 73.45 ($\downarrow$) | 70.34 ($\downarrow$) | 70.25 ($\downarrow$) | 72.61 ($\downarrow$) | 73.64 ($\uparrow$) | 72.88 ($\downarrow$)
AB | 72.50 ($\downarrow$) | 72.38 ($\downarrow$) | 69.47 ($\downarrow$) | 69.53 ($\downarrow$) | 70.98 ($\downarrow$) | 73.17 ($\downarrow$) | 70.94 ($\downarrow$)
FT | 73.25 ($\downarrow$) | 71.59 ($\downarrow$) | 69.84 ($\downarrow$) | 70.22 ($\downarrow$) | 72.37 ($\downarrow$) | 72.86 ($\downarrow$) | 70.58 ($\downarrow$)
FSP | 72.91 ($\downarrow$) | n/a | 69.95 ($\downarrow$) | 70.11 ($\downarrow$) | 71.89 ($\downarrow$) | 72.62 ($\downarrow$) | 70.33 ($\downarrow$)
NST | 73.68 ($\downarrow$) | 72.24 ($\downarrow$) | 69.60 ($\downarrow$) | 69.53 ($\downarrow$) | 71.96 ($\downarrow$) | 73.30 ($\downarrow$) | 71.53 ($\downarrow$)
CRD | 75.48 ($\uparrow$) | 74.14 ($\uparrow$) | 71.16 ($\uparrow$) | 71.46 ($\uparrow$) | 73.48 ($\uparrow$) |  75.51 ($\uparrow$) | 73.94 ($\uparrow$)
CRD+KD | 75.64 ($\uparrow$) |  74.38 ($\uparrow$) | 71.63 ($\uparrow$) | 71.56 ($\uparrow$) |  73.75 ($\uparrow$) | 75.46 ($\uparrow$) | 74.29 ($\uparrow$)
RRD | 75.01 ($\uparrow$) | 73.55 ($\uparrow$) | 70.71 ($\uparrow$) | 70.72 ($\uparrow$) | 73.10 ($\uparrow$) | 74.21 ($\uparrow$) | 73.99 ($\uparrow$)
RRD+KD | 75.66 ($\uparrow$) | 73.77 ($\uparrow$) | 71.72 ($\uparrow$) | 71.62($\uparrow$) | 73.48 ($\uparrow$) | 74.86 ($\uparrow$) | 74.32 ($\uparrow$)
Table 1: Test top-1 accuracy (%) of student networks on CIFAR-100, comparing students and teachers of the same architecture using various distillation methods. $\uparrow$ denotes outperformance over KD and $\downarrow$ denotes underperformance. Results adapted from [27]. The citations and abbreviations of other methods are described in the supplementary material. The values in bold indicate the maximum of each column. Teacher | VGG-13 | ResNet-50 | ResNet-50 | ResNet-32x4 | ResNet-32x4 | WRN-40-2
---|---|---|---|---|---|---
Student | MobileNet-v2 | MobileNet-v2 | VGG-8 | ShuffleNet-v1 | ShuffleNet-v2 | ShuffleNet-v1
Teacher | 74.64 | 79.34 | 79.34 | 79.42 | 79.42 | 75.61
Student | 64.6 | 64.6 | 70.36 | 70.5 | 71.82 | 70.5
KD | 67.37 | 67.35 | 73.81 | 74.07 | 74.45 | 74.83
FitNet | 64.14 ($\downarrow$) | 63.16 ($\downarrow$) | 70.69 ($\downarrow$) | 73.59 ($\downarrow$) | 73.54 ($\downarrow$) | 73.73 ($\downarrow$)
AT | 59.40 ($\downarrow$) | 58.58 ($\downarrow$) | 71.84 ($\downarrow$) | 71.73 ($\downarrow$) | 72.73 ($\downarrow$) | 73.32 ($\downarrow$)
SP | 66.30 ($\downarrow$) | 68.08 ($\downarrow$) | 73.34 ($\downarrow$) | 73.48 ($\downarrow$) | 74.56 ($\uparrow$) | 74.52 ($\downarrow$)
CC | 64.86 ($\downarrow$) | 65.43 ($\downarrow$) | 70.25 ($\downarrow$) | 71.14 ($\downarrow$) | 71.29 ($\downarrow$) | 71.38 ($\downarrow$)
VID | 65.56 ($\downarrow$) | 67.57 ($\downarrow$) | 70.30 ($\downarrow$) | 73.38 ($\downarrow$) | 73.40 ($\downarrow$) | 73.61 ($\downarrow$)
RKD | 64.52 ($\downarrow$) | 64.43 ($\downarrow$) | 71.50 ($\downarrow$) | 72.28 ($\downarrow$) | 73.21 ($\downarrow$) | 72.21 ($\downarrow$)
PKT | 67.13 ($\downarrow$) | 66.52 ($\downarrow$) | 73.01 ($\downarrow$) | 74.10 ($\uparrow$) | 74.69 ($\uparrow$) | 73.89 ($\downarrow$)
AB | 66.06 ($\downarrow$) | 67.20 ($\downarrow$) | 70.65 ($\downarrow$) | 73.55 ($\downarrow$) | 74.31 ($\downarrow$) | 73.34 ($\downarrow$)
FT | 61.78 ($\downarrow$) | 60.99 ($\downarrow$) | 70.29 ($\downarrow$) | 71.75 ($\downarrow$) | 72.50 ($\downarrow$) | 72.03 ($\downarrow$)
NST | 58.16 ($\downarrow$) | 64.96 ($\downarrow$) | 71.28 ($\downarrow$) | 74.12 ($\uparrow$) | 74.68 ($\uparrow$) | 76.09 ($\uparrow$)
CRD | 69.73 ($\uparrow$) | 69.11 ($\uparrow$) | 74.30 ($\uparrow$) | 75.11 ($\uparrow$) | 75.65 ($\uparrow$) | 76.05 ($\uparrow$)
CRD+KD | 69.94 ($\uparrow$) |  69.54 ($\uparrow$) |  74.58 ($\uparrow$) |  75.12 ($\uparrow$) |  76.05 ($\uparrow$) | 76.27 ($\uparrow$)
RRD | 67.93 ($\uparrow$) | 68.84 ($\uparrow$) | 74.01 ($\uparrow$) | 74.11 ($\uparrow$) | 74.64 ($\uparrow$) | 74.98 ($\uparrow$)
RRD+KD | 69.98 ($\uparrow$) | 69.13 ($\uparrow$) | 74.26 ($\uparrow$) | 74.78 ($\uparrow$) | 75.78 ($\uparrow$) | 76.31 ($\uparrow$)
Table 2: Test top-1 accuracy (%) of student networks on CIFAR-100 involving
students and teachers from different architectures, using various distillation
methods. $\uparrow$ denotes outperformance over KD and $\downarrow$ denotes
underperformance. Results adapted from [27]. The citations and abbreviations
of other methods are described in the supplementary material. The values in
bold indicate the maximum of each column.

###  4.2 Capturing inter-class correlations

Cross-entropy loss overlooks the relationships among class logits in a teacher
network, often resulting in less effective knowledge transfer. Distillation
techniques that use "soft targets", such as those described by [10], have
successfully captured these relationships, improving student model
performance. Figure 4 assesses the effectiveness of different distillation
methods on the CIFAR-100 KD task using WRN-40-2 as the teacher and WRN-40-1 as
the student. We compare students trained without distillation, with attention
transfer [32], with KL divergence [10], and with our proposed RRD method. Our
findings show that RRD achieves close alignment between teacher and student
logits, as evidenced by reduced differences in their correlation matrices.
While RRD does not match CRD [27] in terms of exact correlation alignment, it
significantly enhances learning efficiency and reduces error rates. The
smaller discrepancies between teacher and student logits indicate that the RRD
objective captures a substantial portion of the correlation structure in the
logits, resulting in lower error rates, though CRD achieves a slightly closer
match.

(a) Student: vanilla

(b) Student: AT

(c) Student: KD

(d) Student: RRD (ours)

Figure 2: Comparison of correlation matrix differences between teacher and
student logits across various distillation methods on the CIFAR-100 task.
Subfigures show results for (a) students trained without distillation, (b)
with attention transfer [32], (c) with KL divergence [10], and (d) with our
RRD method, highlighting better matching between student’s and teacher’s
correlations. Results have been re-implemented according to [27].

###  4.3 Transferability of representations

Our study investigates how knowledge is transferred from a larger teacher
network (WRN-40-2) to a smaller student network (WRN-16-2), aiming to develop
versatile representations suitable for a range of tasks and datasets. We apply
this technique by having the student network either learn directly from the
CIFAR-100 dataset or via distillation. In this setup, the student network is
employed as a static feature extractor for images from the STL-10 and TIN-200
datasets, both adjusted to a $32\times 32$ resolution. We evaluate the
adaptability of these features by training a linear classifier on the final
feature layer to conduct classifications with 10 categories for STL-10 and 200
categories for TIN-200. We document the impact of various distillation
approaches on the transferability of these features in Table 3. Our findings
reveal that, except for the FitNet method, all distillation techniques
significantly improve the feature transferability on both datasets. Notably,
while the teacher network achieves the highest performance on CIFAR-100, its
features show the least effective transfer, likely due to training data
specificity. Conversely, the student network using a combination of RRD and KD
distillation techniques not only equates to the teacher’s CIFAR-100
performance but also surpasses it in transferability, showing improvements of
3.4% and 2.1% in STL-10 and TIN-200 respectively.

| Teacher | Student | KD | AT | FitNet | CRD | CRD+KD | RRD | RRD+KD
---|---|---|---|---|---|---|---|---|---
CIFAR-100$\rightarrow$STL-10 | 68.6 | 69.7 | 70.9 | 70.7 | 70.3 | 71.6 | 72.2 | 71.2 | 72.0
CIFAR-100$\rightarrow$TIN-200 | 31.5 | 33.7 | 33.9 | 34.2 | 33.5 | 35.6 | 35.5 | 34.8 | 35.0
Table 3: Test top-1 accuracy (%) of WRN-16-2 (student) distilled from WRN-40-2
(teacher). In this setup, the representations learned from the CIFAR-100
dataset are transferred to the STL-10 and TIN-200 datasets. The network is
frozen, and a linear classifier is trained on the last feature layer to
perform classification with 10 classes (STL-10) or 200 classes (TIN-200).
Results adapted from [27]. The values in bold indicate the maximum of each
row.

###  4.4 Visualization of t-SNE embeddings

We provide t-SNE [30] visualizations to compare the embeddings generated by
various KD methods and the teacher network on the CIFAR-100 dataset. Figure 3
displays the embeddings from the teacher network, a WRN-40-2, and the student
network, WRN-40-1, under standard training as well as distillation using AT
and RRD where we limit the dataset to the first 10 classes of CIFAR-100 to
offer a clearer understanding of the embedding space. We observe improved
consistency in the embedding distributions between the teacher and student
networks, indicating that RRD effectively transfers the knowledge of the
teacher’s feature space to the student. Our relational consistency approach
ensures that the spatial relationships in the embedding spaces of both the
student and teacher models are preserved. This alignment not only enhances the
student’s performance but also maintains the integrity of the feature
representations learned by the teacher.

(a) Teacher

(b) Student: vanilla

(c) Student: AT

(d) Student: RRD (ours)

Figure 3: t-SNE visualizations of embeddings from the teacher network and
student networks trained using different distillation techniques on the first
10 classes of the CIFAR-100 dataset.

###  4.5 Ablation study

There are two main hyperparameters in our objective: the number of negative
samples $M$ in the memory buffer $\mathcal{Q}$ of Equation 3 and Equation 4;
and temperature parameters $\tau_{s}$ and $\tau_{t}$ of Equation 3 and
Equation 4 that modulate the softmax probability. We also ablate the
hyperparameter $\beta$ that balances the KD loss. For the ablation study, we
adopt WRN-40-2 as the teacher and WRN-16-2 as the student. Experiments are
conducted on CIFAR-100, and the results are shown in Figure 4.

Number of negatives $\bm{M}$. We validated different values for $M$: 256,
1024, 2048, 4096, 8192, 16384, and 32768. As shown in Figure 4(a), increasing
$M$ leads to improved performance. However, the difference in error rate
between $M=4096$ and $M=16384$ is less 0.5%. Therefore, we use $M=16384$ for
reporting the accuracy, while in practice lower $M$ should suffice. Going
beyond $M=16384$ proves to harms performance.

Temperature $\bm{\tau_{s}}$ and $\bm{\tau_{t}}$. We varied $\tau_{s}$ and
$\tau_{t}$, considering all the permutations of the numbers 0.04, 0.07, and
0.2 taken two at a time, considering that order matters. As Figure 4(b)
illustrates, both extremely high or low temperatures lead to a sub-optimal
solution. Also, as expected, a lower $\tau_{s}$ than $\tau_{t}$ leads to
better performance. This improvement is attributed to the sharper predictions
from the student model with a lower temperature, resulting in more confident
and distinct class probabilities. This sharpness helps the student model align
better with the teacher’s guidance, enhance generalization, provide a more
informative learning signal, and reduce overfitting. The sweep spot lies at
$\tau_{s}=0.04$ and $\tau_{t}=0.07$ (red area of Figure 4(b)).

Loss coefficient $\bm{\beta}$. We varied $\beta$ from 0.1 to 100. As Figure
4(c) illustrates, both extremely high or low $\beta$ lead to a sub-optimal
solution. In general, $\beta$ between 1 and 2 works well on CIFAR-100.

(a) Effect of varying $M$

(b) Effect of varying $\tau_{s}$ and $\tau_{t}$

(c) Effect of varying loss coefficient $\beta$

Figure 4: Ablation study results on CIFAR-100 using WRN-40-2 as the teacher
and WRN-16-2 as the student. (a) Effect of the number of negatives $M$ on
performance. Increasing $M$ generally improves performance, but the difference
becomes negligible beyond $M=4096$. (b) Effect of temperature parameters
$\tau_{s}$ and $\tau_{t}$. Lower $\tau_{s}$ than $\tau_{t}$ results in better
performance, with extremely high or low temperatures leading to sub-optimal
solutions. (c) Effect of loss coefficient $\beta$ on performance (logarithmic
scale). Optimal $\beta$ values range between 1 and 2.

Computational Cost. The RRD loss method introduces an additional 5.24 MFLOPs
to the computational workload of a ResNet-50 model. This represents
approximately 0.262% of the original 2 GFLOPs computational baseline. Despite
the added computational complexity, the impact on training time is minimal,
allowing us to maintain efficiency in practical training scenarios such as on
CIFAR-100. The memory bank for RRD stores all 128-dimensional features of up
to 16,384 CIFAR-100 images, consuming about 8 MB of GPU memory.

##  5 Conclusions

Our method offers a significant advancement in KD by maintaining relational
consistency between teacher and student models. RRD leverages a large memory
buffer of teacher samples to align their output distributions, ensuring
consistent relational structures throughout the learning process. Unlike
traditional approaches, RRD uses pairwise similarities and a relaxed
contrastive loss to enhance the robustness and performance of the student
model without explicit negative instances. Through comprehensive testing on
CIFAR-100, TIN-200, and STL-10 datasets, RRD consistently outperforms
traditional KD techniques and state-of-the-art methods, demonstrating
substantial improvements in accuracy, robustness, and transferability of
learned representations. Our experiments on model compression highlight RRD’s
ability to provide a scalable solution.

## References

  * [1] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai.  Variational information distillation for knowledge transfer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9163–9171, 2019.
  * [2] Gongfan Chen, Yuting Wang, Jiajun Xu, Zhe Du, Qionghai Dai, Shiyang Geng, and Tao Mei.  Learning efficient object detection models with knowledge distillation.  In Advances in Neural Information Processing Systems, pages 742–751, 2017.
  * [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations, 2020.
  * [4] Jang Hyun Cho and Bharath Hariharan.  On the efficacy of knowledge distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4794–4802, 2019.
  * [5] Adam Coates and Andrew Y. Ng.  The importance of encoding versus training with sparse coding and vector quantization.  In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, page 921–928, Madison, WI, USA, 2011. Omnipress.
  * [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [7] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning, 2020.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [9] Byeongho Heo, Minsik Lee, Seong Joon Yun, Jin Young Choi, and In So Kweon.  Knowledge transfer via distillation of activation boundaries formed by hidden neurons.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3779–3787, 2019.
  * [10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network, 2015.
  * [11] Zehao Huang and Naiyan Wang.  Like what you like: Knowledge distill via neuron selectivity transfer.  In Advances in Neural Information Processing Systems, pages 185–195, 2017.
  * [12] Jangho Kim, Seongwon Park, and Nojun Kwak.  Paraphrasing complex network: Network compression via factor transfer.  In Advances in Neural Information Processing Systems, pages 2760–2769, 2018.
  * [13] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  pages 32–33, 2009.
  * [14] Wei Liu, Andrew Rabinovich, and Alexander C Berg.  Learning efficient single-stage pedestrian detectors by asymptotic localization fitting.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 618–634, 2018.
  * [15] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang.  Structured knowledge distillation for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2604–2613, 2019.
  * [16] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 116–131, 2018.
  * [17] Arun Mishra and Debbie Marr.  Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy.  In International Conference on Learning Representations, 2017.
  * [18] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019.
  * [19] Nikolaos Passalis and Anastasios Tefas.  Learning deep representations with probabilistic knowledge transfer.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 268–284, 2018.
  * [20] Baoyun Peng, Xi Li, Yifan Wu, Yizhou Fan, Bo Wang, Qi Tian, and Jun Liang.  Correlation congruence for knowledge distillation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5007–5016, 2019.
  * [21] Antonio Polino, Razvan Pascanu, and Dan Alistarh.  Model compression via distillation and quantization.  In International Conference on Learning Representations, 2018.
  * [22] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  In Proceedings of the 4th International Conference on Learning Representations, 2014.
  * [23] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [24] Li Shen and Marios Savvides.  Amalgamating knowledge towards comprehensive classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1687–1696, 2020.
  * [25] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015.
  * [26] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding, 2020.
  * [27] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation, 2022.
  * [28] Frederick Tung and Greg Mori.  Similarity-preserving knowledge distillation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1365–1374, 2019.
  * [29] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding, 2019.
  * [30] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(Nov):2579–2605, 2008.
  * [31] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133–4141, 2017.
  * [32] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  In Proceedings of the 5th International Conference on Learning Representations, 2016.
  * [33] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks, 2017.
  * [34] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018.
  * [35] Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu.  Ressl: Relational self-supervised learning with weak augmentation, 2021.
  * [36] Songling Zhu, Ronghua Shang, Ke Tang, Songhua Xu, and Yangyang Li.  Bookkd: A novel knowledge distillation for reducing distillation costs by decoupling knowledge generation and learning.  Knowledge-Based Systems, 263:110916, 2023.