2021

[1]Ashesh Chattopadhyay

[2]Ruoying He

[1]Applied Mathematics, University of California, Santa Cruz, Santa Cruz,
95060, California, United States

2]Marine, Earth & Atmospheric Sciences, North Carolina State University,
Raleigh, 27695, North Carolina, United States

# OceanNet: A principled neural operator-based digital twin for regional
oceans

<EMAIL_ADDRESS>Michael Gray<EMAIL_ADDRESS>Tianning Wu<EMAIL_ADDRESS>Anna B. Lowe<EMAIL_ADDRESS><EMAIL_ADDRESS>*  [

###### Abstract

While data-driven approaches demonstrate great potential in atmospheric
modeling and weather forecasting, ocean modeling poses distinct challenges due
to complex bathymetry, land, vertical structure, and flow non-linearity. This
study introduces OceanNet, a principled neural operator-based digital twin for
ocean circulation. OceanNet uses a Fourier neural operator and predictor-
evaluate-corrector integration scheme to mitigate autoregressive error growth
and enhance stability over extended time scales. A spectral regularizer
counteracts spectral bias at smaller scales. OceanNet is applied to the
northwest Atlantic Ocean western boundary current (the Gulf Stream), focusing
on the task of seasonal prediction for Loop Current eddies and the Gulf Stream
meander. Trained using historical sea surface height (SSH) data, OceanNet
demonstrates competitive forecast skill by outperforming SSH predictions by an
uncoupled, state-of-the-art dynamical ocean model forecast, reducing
computation by 500,000 times. These accomplishments demonstrate the potential
of physics-inspired deep neural operators as cost-effective alternatives to
high-resolution numerical ocean models.

###### keywords:

Data-driven model, ocean forecasting, neural operator, spectral bias

##  1 Introduction

An essential component of the Earth’s ocean circulation is the northern
Atlantic Ocean’s western boundary current (WBC) system, which includes the
Loop Current (LC), Loop Current eddies (LCE), the Gulf Stream (GS), and Gulf
Stream meander (GSM). The WBC plays a critical role in the transport of heat,
salt, and nutrients, and strongly influences global weather, climate, and
marine ecosystems. Recent studies have shown the promise of using data-driven
approaches to model and predict atmospheric circulation, but complexities in
the ocean system present additional challenges when building data-driven
models pathak2022fourcastnet ; lam2022graphcast ; bi2022pangu . One challenge
is the complex land boundaries, such as in the Gulf of Mexico (GoM). Equally
difficult to model, the GS frequently sheds eddies as a result of nonlinear
flow and the interactions of cool, subpolar circulation from the north and
warm, subtropical circulation from the south. Predicting LCEs in the GoM and
GS meander and eddy separation have been long-standing challenges for
numerical ocean models dengo1993problem ; chassignet2008gulf ;
ezer2016revisiting . The separation mechanism involves nonlinear barotropic
and baroclinic instabilities of ocean currents, complex flow-topography
interactions, and, in the case of the GS, interaction with the Deep Western
Boundary Current spall1996dynamics ; zhang2007role ; hurlburt2008gulf ;
schoonover2017local . These challenging features require carefully calibrated,
high-resolution numerical models and tremendous computing resources to resolve
andres2016recent ; gangopadhyay2019observed ; silver2023increased .

Recent efforts in emulating ocean dynamics with deep learning-based approaches
have primarily focused on predicting large-scale circulation features, such as
those resolved by empirical orthogonal functions (EOFs) wang2019medium  or on
constructing low-dimensional representations agarwal2021comparison . However,
there has yet to be a comprehensive data-driven model for the ocean akin to
those used in global atmospheric modeling.

Data-driven modeling in Earth sciences has focused on predicting global
atmospheric dynamics chattopadhyay2019analog ; chattopadhyay2020deep ;
pathak2022fourcastnet ; lam2022graphcast ; bi2022pangu ; weather prediction
has been used as a benchmark. The primary advantage of data-driven models lies
in their computational efficiency, enabling the execution of a large number of
ensembles. This facilitates seamless and efficient data assimilation leading
to improved estimates of extreme weather events. Several recent studies
pathak2022fourcastnet ; lam2022graphcast ; bi2022pangu  have demonstrated that
these data-driven models can achieve accuracy comparable to, and often
surpassing, state-of-the-art numerical weather prediction models such as the
Integrated Forecasting System (IFS), at a significantly reduced computational
cost. This achievement holds substantial promise for incorporating deep
learning-based data-driven models into the suite of operational weather
forecasting models.

A significant limitation of these models arises when they are integrated over
longer time scales ($\geq$ 2 weeks), leading to instability and the emergence
of nonphysical features (see Fig. 1 in Chattopadhyay et al.
chattopadhyay2023long ). The cause of this instability was identified as
“spectral bias” chattopadhyay2023long , an inductive bias in all deep neural
networks that hinders their ability to capture small-scale features in
turbulent flows. A potential solution was proposed in chattopadhyay2023long
in the form of a framework to construct long-term stable digital twins for
atmospheric dynamics.

Research into data-driven digital twins for both global and regional ocean
modeling is still in its infancy and has not been explored to the same extent
as in atmospheric modeling. The development of long-term stable, accurate
data-driven ocean models that maintain computational efficiency is essential
for realizing data-driven digital twins for the entire Earth system.

We present here a principled neural operator-based digital twin for the
northwest Atlantic Ocean’s western boundary current, built upon the same
principles as the FouRK model introduced in Chattopadhyay et al.
chattopadhyay2023long . The proposed model, OceanNet, relies on a Fourier
neural operator (FNO), which incorporates a predictor-evaluate-corrector (PEC)
integration scheme to suppress autoregressive error growth. Additionally, a
spectral regularizer is employed to mitigate spectral bias at small scales.

OceanNet is trained on historical sea surface height (SSH) data from a high-
resolution northwest Atlantic Ocean reanalysis and demonstrates remarkable
stability and competitive skills. It often outperforms SSH prediction made by
the state-of-the-art Regional Ocean Modeling System (ROMS) while maintaining a
computational cost that is 500,000 cheaper. This marks the first instance of a
fully data-driven ocean model exhibiting competitive or superior performance
in predicting high-resolution SSH fields compared to a state-of-the-art
numerical ocean model. This achievement underscores the potential of
structured, physics-inspired deep neural operators as a cost-effective
alternative to computationally expensive high-resolution numerical models.

Figure 1:  The domain for the reanalysis data covering the northwestern
Atlantic (a). The two subdomains used to develop OceanNet, specifically (b)
the GS separation from the central US east coast to 60∘W and (c) tthe LC eddy-
shedding region in the eastern GoM, extending from 92∘W into the Atlantic
75∘W. The mean SSH from from 1993-2020 in the reanalysis data is shown in (a),
while (b) and (c) depict daily mean SSH on May 1, 2019, and July 10, 2019,
respectively. All three domains share the same color scale.

##  2 Results

Mesoscale ocean circulation dynamics can be well represented by the spatio-
temporal evolution of SSH fields. In this section we conduct a comprehensive
comparison of SSH predictions generated by OceanNet and the dynamical ROMS
forecast with independent reanalysis data. To assess the performance
rigorously, we employ both qualitative and quantitative measures. The metrics
for evaluating predictive accuracy of SSH include root mean squared error
(RMSE) and correlation coefficient (CC), which are widely recognized and
employed in forecasting pathak2022fourcastnet ; bi2022pangu ; lam2022graphcast
; chattopadhyay2023long . In addition, we incorporate a specialized object-
tracking metric to evaluate the prediction of major ocean features delineated
by SSH contours: the modified Hausdorff distance (MHD, dukhovskoy2015skill ).
MHD quantifies the comparison of predicted eddy/meander structures to their
counterparts in the reanalysis data; identical shapes have an MHD of zero. To
provide a comprehensive assessment, we also present qualitative snapshots of
the predicted SSH fields generated by OceanNet, ROMS dynamical forecast, and
the independent SSH field derived from the reanalysis.

###  2.1 Performance of OceanNet in the GoM

Figure 2: Prediction performance of OceanNet. (a) SSH fields from the ocean
reanalysis. (b) Predicted SSH generated by the OceanNet. (c) ROMS dynamical
forecasts. Each row in the OceanNet and dynamical model predictions was
initialized with the corresponding reanalysis data in the left column. The
predictions display SSH forecasts for 30, 60, and 90 days. To evaluate the
predictions, we can examine a diagonal comparison with the reanalysis SSH, as
indicated by the black arrows in (b). The same diagonal comparison can be
conducted with the ocean reanalysis data for (c).

We present an evaluation of OceanNet’s seasonal (up to 120 days) forecasting
performance compared to that of a ROMS forecast for two major GoM LCE shedding
events: Eddy Sverdrup (Jul 2019-Jan 2020) and Eddy Thor (Jan 2020 - Sep 2020).
The target LC dynamics, represented by spatio-temporal evolutions of SSH, are
from the reanalysis dataset. Specifics regarding the generation and structure
of the reanalysis dataset are covered in section 4.1.

A qualitative assessment (Fig. 2) reveals that OceanNet excels in capturing
the dynamics of LC eddy shedding and reattachment, outperforming the ROMS
dynamical forecasts for these two events. Furthermore, OceanNet maintains both
stability and physical consistency for at least 120 days. For this domain,
OceanNet is trained with a $5$-day lead, consistent with the LC and LCE
evolution timescale. ROMS predictions were initialized five days apart to
allow for fair comparisons between the two models. OceanNet predictions have a
“wall-clock” execution time of microseconds, approximately 500,000 times
faster than ROMS dynamical forecasts.

Figure 3: OceanNet performance metrics in the GoM: RMSE (a-c), CC (d-f), and
MHD (g-i). In the left column, probability density functions (PDFs) are
presented for 1,000 random pairs sourced from the training data spanning
1993-2018 (means shown by black horizontal lines). The performance statistics,
calculated based on forecasts of 0-120 days for eddies Sverdrup (middle
column) and Thor (right column), are displayed as mean values (lines) with
standard deviations (shading). The black horizontal dashed lines in these
columns denote saturation values determined as 95% of the means derived from
the random pairs. In the middle and right columns, solid blue lines are
OceanNet, dashed red lines are the ROMS dynamical forecast model, and gray
dots are persistence. These representations illustrate how each method’s
statistics compare with the target SSH from the reanalysis dataset

An examination of RMSE, CC, and MHD for SSH, over a 120-day period (Fig. 3)
demonstrates that OceanNet is consistently competitive or better than ROMS
dynamical forecasts, and outperforms random predictions (indicated by the
horizontal dashed lines). These random predictions are derived from 1,000
random pairs of samples from the training data covering the period of
1993-2018 to calculate the saturation values, which quantify the intrinsic
predictability of the ocean system dalcher1987error ;
delsole2004predictability ; dukhovskoy2023assessment . Comparing OceanNet and
ROMS with persistence forecasting, we see that ROMS suffers from inaccuracies
in predicting the timing and configuration of the eddy-shedding and
reattachment process, and thus underperforms persistence forecast on several
occasions. This is not the case for OceanNet.

We conducted a further comparison (Supplementary Information) of OceanNet with
previous deep learning-based models that have been successfully used in
weather prediction, such as the model introduced by Weyn et al.
weyn2020improving . The spectral characteristics of these models exhibit
deviations in the high wavenumbers, rendering their predictions nunphysical
beyond a $2$-week forecasting horizon (Fig. S1). A geostrophic constraint
within the loss function was also tested (Fig. S2), but OceanNet in its
current design performed better.

###  2.2 Performance of OceanNet in the North Atlantic GS Region

We present qualitative and quantitative evaluations of OceanNet’s forecasting
performance compared to the ROMS dynamical model forecast for GSM in the
northwest Atlantic. OceanNet was specifically trained for this domain with a
lead time of $4$ days to resolve the evolution of the northern boundary of the
GS. To ensure alignment with the initialization dates for ROMS (which were
separated by $5$ days), we show the $20$th, $40$th, and $60$th-day forecasts,
as opposed to the $30$th, $60$th, and $90$th-day forecasts used for the LC.

Figure 4: Performance of OceanNet for GS prediction. (a) SSH fields from the
ocean reanalysis. (b) Predicted SSH generated by OceanNet. (c) ROMS dynamical
model forecasts. In both OceanNet and dynamical model predictions, each row
was initialized with the corresponding reanalysis data in the left column. SSH
forecasts are provided for 20, 40, and 60 days. To evaluate the predictions,
we can perform a diagonal comparison with the reanalysis SSH, as indicated by
the black arrows in (b). The same diagonal comparison can also be conducted
with the ocean reanalysis data for (c).

A qualitative assessment reveals that OceanNet effectively captures the SSH
propagation of undulations in the northern boundary of the GS. Moreover, it
skillfully captures large-scale eddies traveling into and out of the domain,
even without receiving any boundary information (Fig. 4). In contrast, ROMS
dynamical model forecast tends to overpredict extreme values of SSH and the
meridional amplitude of the northern boundary. While it is sensitive to
initial conditions, OceanNet remains physically consistent over long-term
forecasts in this region as well. As observed in the GoM, OceanNet provides
stable and physically reasonable SSH predictions for the GSM for at least 120
days (not shown for brevity).

Similar to the GoM, OceanNet consistently outperforms ROMS in RMSE, CC, and
MHD computed between the predicted SSH values and the reanalysis SSH over 120
days (Fig. 5). Persistence forecasting also fares reasonably well in this
region due to the background state of the GSM; however, OceanNet still
outperforms persistence in all three metrics over 120 days.

Figure 5: OceanNet’s performance metrics in the northwest Atlantic: RMSE
(top), CC (middle), and MHD (bottom), compared to the persistence forecast and
ROMS dynamical model forecast. In the left column, probability density
functions (PDFs) are presented, derived from 1,000 random pairs sourced from
the training data spanning 1993-2018 (means shown by black horizontal lines).
The performance statistics, calculated based on forecasts of 0-120 days, are
displayed as mean values (lines) with standard deviations (shading). The black
horizontal dashed lines denote saturation values, which are determined as 95%
of the means derived from the random pairs. Solid blue lines are OceanNet,
dashed red lines are the ROMS dynamical forecast model, and gray dots are
persistence. These representations illustrate how each method’s statistics
compare with the target SSH from the reanalysis dataset

##  3 Discussion

In this study, we introduce OceanNet, a deep neural operator, with principled
design structures within the architecture, to function as a digital twin for
long-term (up to 120 days) mesoscale ocean circulation forecasting. Our major
technical contributions encompass the development of an FNO and the
application of a PEC-based convergent integration scheme, along with a
spectral regularizer. Together, these techniques mitigate autoregressive error
growth and the small-scale spectral bias frequently observed in data-driven
models of multi-scale dynamical systems.

By training the model on 26 years (1993-2018) of high-resolution regional
ocean reanalysis data for the northwest Atlantic, we evaluate OceanNet’s
ability to predict mesoscale ocean processes such as the GoM LC eddy shedding
and GSM over a $90$-$120$ day forecasting horizon. OceanNet demonstrates
remarkable stability and consistently outperforms the ROMS dynamical ocean
forecast while being $500,000$ times faster. These results demonstrate the
potential of utilizing scientific machine learning to develop long-term
stable, and accurate data-driven ocean models of great computational
efficiency, paving the way for realizing a data-driven digital twin
encompassing the entire climate system.

Despite the promising forecast accuracy observed with reanalysis data, our
algorithm possesses several limitations. First, this study exclusively trained
and tested OceanNet using reanalysis data pertaining to mesoscale ocean eddies
and meanders. Real-world ocean forecasting systems operate with dynamical
ocean models and real-time ocean observational data, covering dynamic
processes across diverse spatial scales. The disparities between these data
sources and scales necessitate further investigation into OceanNet’s
performance across various ocean applications. Second, OceanNet’s focus is
solely on SSH. This study did not explore other ocean state variables, such as
currents and temperature. The omission of these variables may restrict the
model’s capacity to accurately predict small-scale ocean circulation
phenomena, such as shelf break jets and frontal currents. Third, AI-based
methods frequently produce smoother forecast results than dynamical models,
which can potentially lead to an underestimation of the magnitudes of extreme
ocean events. Therefore, additional research is imperative to assess
OceanNet’s performance under extreme ocean conditions, e.g., during severe
storms.

Significant opportunities exist for improvement in both AI-based methods and
dynamical model-based ocean forecasting. In the AI domain, potential
advancements involve the integration of subsurface ocean states and additional
ocean variables, the incorporation of temporal dimensions through the training
of four-dimensional deep networks, and the exploration of more complex network
architectures with increased depth and breadth. In the realm of numerical
ocean forecast modeling, the development of pre- and post-processing
techniques can help mitigate the inherent biases found in ocean models. We
expect that a hybrid approach, combining data-driven and dynamical numerical
models will play a pivotal role in pushing the boundaries of excellence in
ocean prediction.

##  4 Data and Methods

###  4.1 Reanalysis Data

We utilized high-resolution northwest Atlantic regional ocean reanalysis data
to train OceanNet. This reanalysis was generated using ROMS with the ensemble
Kalman filter data assimilation method (EnKF). It features a horizontal
resolution of 4 km with 50 vertical layers. For surface atmospheric forcing,
we employed data from the European Center for Medium-Range Weather Forecasting
(ECMWF) Reanalysis v5 (ERA5), while open boundary conditions were derived from
the Copernicus Global Ocean Physics Reanalysis (GLORYS). Ten major tidal
constituents from the OSU TPXO tide database were used. The model incorporated
120 river inputs, sourced from the National Water Model and climatology
datasets. The temporal scope of the reanalysis spans from January 1, 1993 to
December 31, 2020, with daily output.

The assimilated observations encompass a variety of sources, including AVHRR
and MODIS Terra sea surface temperature, AVISO along-track sea surface height
anomaly, glider temperature/salinity observations from the Integrated Ocean
Observing System (IOOS), and the EN4 dataset which aggregates data from Argo
floats, shipboard surveys, drifters, moorings, and other sources. EnKF
assimilates data with no influence or retainment of previous timesteps, as
opposed to 3D/4D-var methods, which include forward and backward passes over
the results to ensure continuity. The resulting data assimilative ocean
reanalysis allows us to train OceanNet with a time-space continuous SSH
dataset with no external knowledge (as would be the case for observations).

OceanNet is trained on 5-day mean SSH data with a 5-day lead time for the GoM
domain and 4-day lead time for the GSM domain. Both training datasets consist
of 26 years of reanalysis data from 1993 through 2018. SSH Forecasting was
conducted for 2019-2020. Further details on OceanNet’s training process are
provided in section 4.2.

###  4.2 OceanNet: Configuration and Design

The OceanNet model (Fig. 6) has three major components. the Fourier Neural
Operator (FNO), which is used to learn the residual, similar to Krishnapriyan
et al. krishnapriyan2022learning  and Chattopadhyay et al.
chattopadhyay2023long ; the PEC integrator; and the spectral regularizer,
which alleviates spectral bias.

Figure 6: (a) A schematic of the OceanNet model. (b) The Fourier Neural
Operator, depicted as $\mathcal{N}$. (c) 2-time-step training scheme. (d) The
loss function used. Here, $u$ is the state of the system that is predicted,
and $\mathbf{H}$ is the PEC-based convergent integration scheme used.

####  4.2.1 Fourier Neural Operator (FNO)

OceanNet is built upon the FNO li2020fourier . Training utilizes labeled pairs
of historical 5-day mean SSH data in the GoM (section 4.1), $X(t)$ (input),
$X(t+5\Delta t)$ (label), and $X(t+10\Delta t)$ (label), where $\Delta t=1$
day. For training OceanNet in the northwest Atlantic region, the labels are
$X(t+4\Delta t)$ and $X(t+8\Delta t)$. The training assumes the governing
partial differential equation for the reduced ocean system involves ocean
states $X(t)$:

$\displaystyle\frac{d\mathbf{X}}{dt}=\mathbf{F}\left(\mathbf{X}\left(t\right)\right).$
(1)

To integrate the system from the initial condition, $X(t)$, we represent Eq.
(1) in its discrete form:

$\displaystyle\mathbf{X}(t+5\Delta
t)=\underbrace{\mathbf{X}(t)+\int_{t}^{t+5\Delta
t}\underbrace{\mathbf{F}\left(\mathbf{X}\left(t\right)\right)dt}_{\mathcal{N[\circ,\theta]}}}_{\mathbf{H[\circ]}}.$
(2)

Here, $\mathcal{N}$ is an FNO that parameterizes $\mathbf{F}$ with four
Fourier layers, similar to Li et al. li2020fourier , each layer retaining $64$
Fourier modes. $\theta$ represents the $\approx 80\times 10^{6}$ the trainable
parameters of the FNO. $\mathbf{H}$ is a higher-order
predictor–evaluate–corrector (PEC) integrator (section 4.2.2).

####  4.2.2 Predictor-Evaluate-Corrector Integration Scheme

Similar to the higher-order integration scheme in the form of fourth-order
Runge-Kutta (RK4) (Chattopadhyay et al. chattopadhyay2023long ), we implement
a PEC scheme in OceanNet, represented by the operator, $\mathbf{H}$. The
operations in $\mathbf{H}$ are given by:

$\displaystyle i_{1}$
$\displaystyle=\mathcal{N}\left[\mathbf{X}\left(t\right),\theta\right],$ (3a)
$\displaystyle z$
$\displaystyle=\mathbf{X}\left(t\right)+\mathcal{N}\left[\mathbf{X}\left(t\right)+\frac{1}{2}i_{1},\theta\right].$
(3b)

The predicted state is given by
$z=\mathbf{H}\left[\mathcal{N}\left(\mathbf{X}\left(t\right)\right),\theta\right]$.

Although most of the higher-order integration schemes, including RK4,
demonstrate good performance for this problem, we have identified PEC as the
most effective choice. A theoretical study of the effect of each integration
scheme on the inductive bias of the trained $\mathcal{N}$ is an active area of
research krishnapriyan2022learning , especially for understanding the role it
plays on the subsequent spectral bias (section 4.2.3).

####  4.2.3 Fourier Regularizer and 2-time-step Loss Function

In OceanNet’s loss function, we incorporate a spectral regularizer based on
Fourier transforms introduced in Chattopadhyay et al. chattopadhyay2023long .
This is in addition to the standard mean squared error loss (MSE) function,
which is computed exclusively for grid points located over the ocean.

The spectral regularizer penalizes deviations in the absolute value of the
latitude-averaged zonal Fourier spectrum (Fourier spectrum from now on) of the
SSH field at small wavenumbers. Such deviations arise due to spectral bias
xu2019frequency ; chattopadhyay2023long , which represents an inherent
inductive bias in deep neural networks. This bias is responsible for their
limitations in learning the fine-scale dynamics of turbulent flow.

Furthermore, the combined loss function, which integrates both MSE and the
spectral regularizer is evaluated over two time steps, namely $t+5\Delta t$
($t+4\Delta t$ for GS) and $t+10\Delta t$ ($t+8\Delta t$ for GS), to extend
the stability horizon. This two-time-step loss function had remarkable success
in the weather prediction model, FourCastNet pathak2022fourcastnet . The
spectral regularizers for time $t+5\Delta t$ and $t+10\Delta t$ are defined as
follows:

$\displaystyle\mu\left(t+5\Delta
t,\theta\right)=\sum_{t=0}^{t=T}\biggr{\|}\widehat{\mathbf{X}}\left(t+5\Delta
t\right)\biggr{\rvert}_{{k_{x}\geq
k_{T}}}-\widehat{\mathbf{H}}\left[\mathcal{N}\left(\mathbf{X}\left(t\right),\theta\right)\right]\biggr{\rvert}_{{k_{x}\geq
k_{T}}}\biggr{\|}_{2}^{2}.$ (4)

$\displaystyle\mu\left(t+10\Delta
t,\theta\right)=\sum_{t=0}^{t=T}\biggr{\|}\widehat{\mathbf{X}}\left(t+10\Delta
t\right)\biggr{\rvert}_{{k_{x}\geq
k_{T}}}-\widehat{\mathbf{H}}\left[\mathbf{H}\left[\mathcal{N}\left(\mathbf{X}\left(t\right),\theta\right)\right]\right]\biggr{\rvert}_{{k_{x}\geq
k_{T}}}\biggr{\|}_{2}^{2}.$ (5)

Here, the $\hat{\left[\circ\right]}$ operator is the latitude-averaged zonal
Fourier transform of both the predicted and the target SSH fields. $k_{T}$ is
a cutoff wavenumber chosen to be $100$ after significant trial and error. The
loss functions for $t+5\Delta t$ and $t+10\Delta t$ are given by
$\mathbf{L}\left(t+5\Delta t,\theta\right)$:

$\displaystyle\mathbf{L}\left(t+5\Delta
t,\theta\right)=\sum_{t=0}^{t=T}\|\mathbf{X}\left(t+5\Delta
t\right)-\mathbf{H}\left[\mathcal{N}\left(\mathbf{X}\left(t\right),\theta\right)\right]\|_{2}^{2}+\lambda\mu(t+5\Delta
t,\theta).$ (6)

$\displaystyle\mathbf{L}\left(t+10\Delta
t,\theta\right)=\sum_{t=0}^{t=T}\|\mathbf{X}\left(t+10\Delta
t\right)-\mathbf{H}\left[\mathbf{H}\left[\mathcal{N}\left(\mathbf{X}\left(t\right),\theta\right)\right]\right]\|_{2}^{2}+\lambda\mu(t+10\Delta
t,\theta).$ (7)

The total loss is given by:

$\displaystyle\mathbf{L}\left(\theta\right)=\mathbf{L}\left(t+5\Delta
t,\theta\right)+\mathbf{L}\left(t+10\Delta t,\theta\right).$ (8)

The spectral regularizer ensures that the high wavenumbers in the Fourier
spectrum of SSH remain consistent with the target Fourier spectrum. This
measure is effective in reducing spectral bias throughout the training
process. The two-time-step loss function enhances the stability horizon of the
model. The number of time steps over which the loss is calculated can
potentially be extended. However, with each increase in the number of time
steps, the memory requirement for the subsequent backpropagation process
during training grows exponentially.

Acknowledgments We thank Subhashis Hazarika and Maria Molina for the
insightful discussions and Jennifer Warrillow for editorial assistance. MG,
AL, TW, and RH, were supported by NSF awards 2019758 and 2331908. AC performed
partial work at thePalo Alto Research Center, SRI International. All authors
contributed equally to the research conducted and the writing of the paper.
The codes used in this study will be made available upon publication.

## References

  *   * (1) Pathak, J., Subramanian, S., Harrington, P., Raja, S., Chattopadhyay, A., Mardani, M., Kurth, T., Hall, D., Li, Z., Azizzadenesheli, K., et al.: FourCastNet: A global data-driven high-resolution weather model using adaptive Fourier neural operators. arXiv preprint arXiv:2202.11214 (2022)
  * (2) Lam, R., Sanchez-Gonzalez, A., Willson, M., Wirnsberger, P., Fortunato, M., Pritzel, A., Ravuri, S., Ewalds, T., Alet, F., Eaton-Rosen, Z., et al.: Graphcast: Learning skillful medium-range global weather forecasting. arXiv preprint arXiv:2212.12794 (2022)
  * (3) Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., Tian, Q.: Accurate medium-range global weather forecasting with 3d neural networks. Nature, 1–6 (2023)
  * (4) Dengo, J.: The problem of gulf stream separation: A barotropic approach. Journal of Physical oceanography 23(10), 2182–2200 (1993)
  * (5) Chassignet, E., Marshall, D.: Gulf Stream Separation in Numerical Ocean Models. American Geophysical Union, ??? (2008)
  * (6) Ezer, T.: Revisiting the problem of the gulf stream separation: on the representation of topography in ocean models with different types of vertical grids. Ocean Modelling 104, 15–27 (2016)
  * (7) Spall, M.A.: Dynamics of the gulf stream/deep western boundary current crossover. part i: Entrainment and recirculation. Journal of physical oceanography 26(10), 2152–2168 (1996)
  * (8) Zhang, R., Vallis, G.K.: The role of bottom vortex stretching on the path of the north atlantic western boundary current and on the northern recirculation gyre. Journal of Physical Oceanography 37(8), 2053–2080 (2007)
  * (9) Hurlburt, H.E., Hogan, P.J.: The gulf stream pathway and the impacts of the eddy-driven abyssal circulation and the deep western boundary current. Dynamics of Atmospheres and Oceans 45(3-4), 71–101 (2008)
  * (10) Schoonover, J., Dewar, W.K., Wienders, N., Deremble, B.: Local sensitivities of the gulf stream separation. Journal of Physical Oceanography 47(2), 353–373 (2017)
  * (11) Andres, M.: On the recent destabilization of the gulf stream path downstream of cape hatteras. Geophysical Research Letters 43(18), 9836–9842 (2016)
  * (12) Gangopadhyay, A., Gawarkiewicz, G., Silva, E.N.S., Monim, M., Clark, J.: An observed regime shift in the formation of warm core rings from the gulf stream. Scientific reports 9(1), 12319 (2019)
  * (13) Silver, A., Gangopadhyay, A., Gawarkiewicz, G., Fratantoni, P., Clark, J.: Increased gulf stream warm core ring formations contributes to an observed increase in salinity maximum intrusions on the northeast shelf. Scientific Reports 13(1), 1–9 (2023)
  * (14) Wang, J.L., Zhuang, H., Chérubin, L.M., Ibrahim, A.K., Muhamed Ali, A.: Medium-term forecasting of loop current eddy cameron and eddy darwin formation in the gulf of mexico with a divide-and-conquer machine learning approach. Journal of Geophysical Research: Oceans 124(8), 5586–5606 (2019)
  * (15) Agarwal, N., Kondrashov, D., Dueben, P., Ryzhov, E., Berloff, P.: A comparison of data-driven approaches to build low-dimensional ocean models. Journal of Advances in Modeling Earth Systems 13(9), 2021–002537 (2021)
  * (16) Chattopadhyay, A., Nabizadeh, E., Hassanzadeh, P.: Analog forecasting of extreme-causing weather patterns using deep learning. Journal of Advances in Modeling Earth Systems 12(2), 2019–001958 (2020)
  * (17) Chattopadhyay, A., Mustafa, M., Hassanzadeh, P., Kashinath, K.: Deep spatial transformers for autoregressive data-driven forecasting of geophysical turbulence. In: Proceedings of the 10th International Conference on Climate Informatics, pp. 106–112 (2020)
  * (18) Chattopadhyay, A., Hassanzadeh, P.: Long-term instabilities of deep learning-based digital twins of the climate system: The cause and a solution. arXiv preprint arXiv:2304.07029 (2023)
  * (19) Dukhovskoy, D.S., Ubnoske, J., Blanchard-Wrigglesworth, E., Hiester, H.R., Proshutinsky, A.: Skill metrics for evaluation and comparison of sea ice models. Journal of Geophysical Research: Oceans 120(9), 5910–5931 (2015)
  * (20) Dalcher, A., Kalnay, E.: Error growth and predictability in operational ecmwf forecasts. Tellus A 39(5), 474–491 (1987)
  * (21) DelSole, T.: Predictability and information theory. part i: Measures of predictability. Journal of the atmospheric sciences 61(20), 2425–2440 (2004)
  * (22) Dukhovskoy, D.S., Chassignet, E.P., Bozec, A., Morey, S.L.: Assessment of predictability of the loop current in the gulf of mexico from observing system experiments and observing system simulation experiments. Frontiers in Marine Science 10, 1153824 (2023)
  * (23) Weyn, J.A., Durran, D.R., Caruana, R.: Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere. Journal of Advances in Modeling Earth Systems 12(9), 2020–002109 (2020)
  * (24) Krishnapriyan, A.S., Queiruga, A.F., Erichson, N.B., Mahoney, M.W.: Learning continuous models for continuous physics. arXiv preprint arXiv:2202.08494 (2022)
  * (25) Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., Anandkumar, A.: Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 (2020)
  * (26) Xu, Z.-Q.J., Zhang, Y., Luo, T., Xiao, Y., Ma, Z.: Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint arXiv:1901.06523 (2019)


[1]Chae Jung Park

[1]Division of Cancer Data Science, Cancer Big Data and Artificial
Intelligence, National Cancer Center, Ilsan-ro, Goyang-si, 10408, Gyeonggi-do,
South Korea

#  Retro: Reusing teacher projection head for efficient embedding distillation
on Lightweight Models via Self-supervised Learning

Khanh-Binh Nguyen<EMAIL_ADDRESS><EMAIL_ADDRESS>*

###### Abstract

Self-supervised learning (SSL) is gaining attention for its ability to learn
effective representations with large amounts of unlabeled data. Lightweight
models can be distilled from larger self-supervised pre-trained models using
contrastive and consistency constraints. Still, the different sizes of the
projection heads make it challenging for students to mimic the teacher’s
embedding accurately. We propose Retro, which reuses the teacher’s projection
head for students, and our experimental results demonstrate significant
improvements over the state-of-the-art on all lightweight models. For
instance, when training EfficientNet-B0 using ResNet-50/101/152 as teachers,
our approach improves the linear result on ImageNet to $66.9\%$, $69.3\%$, and
$69.8\%$, respectively, with significantly fewer parameters.

###### keywords:

Self-supervised learning, light-weight model, contrastive learning.

##  1 Introduction

Deep learning has achieved remarkable success in various visual tasks, such as
image classification, object detection, and semantic segmentation, thanks to
the availability of large-scale annotated datasets. However, acquiring labeled
data is time-consuming and expensive, making it crucial to explore better ways
to utilize unlabeled data. Self-supervised learning (SSL) has emerged as an
effective method to learn useful representations on unlabeled data, resulting
in an outstanding performance on downstream tasks [11, 18, 7, 21, 3, 4, 12,
14].

Despite its effectiveness, most SSL methods require large networks, and the
performance deteriorates when the model size is reduced. To address this
issue, [9] proposed SEED, a self-supervised representation distillation method
that distills the knowledge of larger pre-trained models into lightweight
models via self-supervised learning. Similarly, CompRess [16] mimics the
similarity score distribution between a teacher and a student over a
dynamically maintained queue. [10] suggests incorporating consistency
constraints between teacher and student embeddings to alleviate the Distilling
Bottleneck problem via DisCo. BINGO [27] aims to transfer the relationship
learned by the teacher to the student by leveraging a set of similar samples
constructed by the teacher and grouped within a bag.

Figure 1: ImageNet top-1 linear evaluation accuracy on different network
architectures. Our method significantly exceeds the result of using MoCo-V2
directly and surpasses the state-of-the-art DisCo by a large margin.
Particularly, the result of EfficientNet-B0 is quite close to the teacher
ResNet-50, while the number of parameters of EfficientNet-B0 is only 16.3% of
ResNet-50. The improvement brought by Retro is compared to the MoCo-V2
baseline.

Despite achieving state-of-the-art results across multiple tasks with high
performance, some concerns remain. First, [10] discovered that expanding the
dimension of the hidden layer in the MLP (projection head) could alleviate the
Distilling Bottleneck problem. However, this approach is trivial since
determining the size of the dimension and how large it should be remains
unanswered. Second, because the student is lightweight with limited
capability, it is challenging to accurately mimic the teacher from the encoder
to the projection head. For example, in the DisCo study [10], they expanded
the dimension to $2048$, which is the projection head dimension of
ResNet-50/101/152. Consequently, this approach is equivalent to increasing the
capability of mimicking the teacher, resulting in improved performance.
However, when using ResNet-50$\times$2 with a dimension of $8192$ as a
teacher, the performance on MobileNet-v3-Large and EfficientNet-B1 drops
significantly and is inferior to the previous method [9]. Moreover, the
feature distributions of the teacher and student models are statistically
different and cannot be directly compared in practice, even if their
dimensions are the same. Therefore, the optimal dimension for the projection
head and how to efficiently distill the teacher embedding remain unanswered
questions.

In this study, we propose a novel approach for improving the performance of
distilling lightweight models through SSL. Specifically, we suggest reusing
the pre-trained teacher projection head for students, instead of mimicking it
during training. This is based on the hypothesis that the most valuable
knowledge is contained in the projection head, and it should be retained
during distillation. Our proposed “teacher projection head reusing strategy”
involves replacing the student projection head with the pre-trained one from a
teacher, which is a large dimension MLP layer that has been optimized. This
enables direct reuse of the projection head, without the need for heuristic
selection of the dimension size via trial and error. Additionally, a
“dimension adapter” is inserted between the student encoder and the teacher
projection head to align the dimension.

Our approach simplifies the training objective from mimicking the
representation and the embedding to aligning the representation with the
optimal embedding. Our experiments show that the proposed method, named Retro,
outperforms the existing DisCo method by a significant margin when using the
same architecture on various downstream tasks. Moreover, Retro achieves state-
of-the-art SSL results on all lightweight models, including ResNet-18/34,
EfficientNet-B0/B1, and MobileNetV3. Notably, the linear evaluation results of
EfficientNet-B0 on ImageNet are comparable to ResNet-50/ResNet-101, despite
having only a fraction of the parameters. On the COCO and PASCAL VOC datasets,
Retro also achieves more than $3\%$ mAP improvement across different pre-
trained models.

##  2 Related Work

Self-supervised learning and knowledge distillation have emerged as crucial
research areas in machine learning, attracting significant attention in recent
years. In this section, we present a review of some of the key works in these
fields.

###  2.1 Self-supervised Learning

Self-supervised learning is an essential branch of unsupervised learning that
automatically generates supervisory signals from unlabeled data. One of the
earliest and most effective techniques used in self-supervised learning is the
autoencoder, which compresses the input data and reconstructs it. Contrastive
learning is another popular self-supervised learning method that enables the
model to differentiate between similar and dissimilar pairs of examples.

Recent studies have demonstrated the efficacy of contrastive-based techniques
in self-supervised representation learning, where different perspectives of
the same input are encouraged to be closer in feature space [3, 4, 5, 6, 12,
14]. For instance, SimCLR [3, 4] has proved that using strong data
augmentation, larger batch sizes of negative samples, and including a
projection head (MLP) after global average pooling can boost self-supervised
learning. However, the performance of SimCLR is dependent on very large batch
sizes and may not be feasible in real-world scenarios.

MoCo [6, 14], on the other hand, uses a memory bank to maintain consistent
representations of negative samples. It considers contrastive learning as a
look-up dictionary, enabling it to achieve superior performance without large
batch sizes, making it more practical. BYOL [12] introduces a predictor to one
branch of the network to prevent trivial solutions and break the symmetry.
DINO [2] applies contrastive learning to vision transformers with self-
distillation intuition.

(a) CompRess

(b) SEED

(c) DisCo

(d) Proposed

Figure 2: Comparison with existing self-supervised distillers. $x$ is the
input image. The orange arrow indicates the knowledge transfer direction. Both
2a CompRess [16] and 2b SEED [9] transfer the knowledge of the similarity
between a sample and a negative memory bank. 2c DisCo [10] constrains the last
embedding of the student to be consistent with that of the teacher. 2d Our
Retro improves DisCo by reusing the teacher projection head for the student,
which has a higher capability to generate generalized embedding. ’Adt.’
indicates the adapter layer.

###  2.2 Knowledge Distillation

Knowledge distillation is a powerful technique used for transferring knowledge
from a large, complex model (known as the teacher) to a smaller, simpler model
(known as the student) to improve its performance.

The idea of knowledge distillation was first proposed by [15], which transfers
knowledge from a large teacher to a smaller student by minimizing the Kulback-
Leibler (KL) divergence between the outputs of the two models. Attention
Transfer (AT) is introduced to transfer the spatial attention of the teacher
to the student by minimizing the mean squared error (MSE) between the feature
maps of the two models. This method guides the student to focus on relevant
regions of the input image, improving its performance on small datasets.

FitNets [23] is another method of knowledge distillation that transfers
knowledge from the intermediate layers of a deep and thin teacher to a deeper
but thinner student. The intermediate layers learned by the teacher are
treated as hints, and the student is trained to mimic them using mean squared
error loss. Relation Knowledge Distillation (RKD) [20] is a method that
transfers the mutual relationship between the samples in a batch from the
teacher to the student. RKD uses distance-wise and angle-wise distillation
loss to transfer the relationship between the samples to the student.

##  3 Method

###  3.1 Self-supervised Learning and Knowledge Distillation

In recent years, there has been a growing interest in combining knowledge
distillation and self-supervised learning to improve the learning process.
Some recent works, such as CRD [25] and SSKD [26], have used self-supervision
as an auxiliary task to enhance knowledge distillation in fully supervised
settings by transferring relationships between different modalities or
mimicking transformed data and self-supervision tasks.

On the other hand, CompRess [16] and SEED [9] have focused on improving self-
supervised visual representation learning on small models by incorporating
knowledge distillation. They leverage the memory bank of MoCo [14] to maintain
the consistency of the student’s distribution with that of the teacher.
Meanwhile, DisCo [10] proposes to align the final embedding of the lightweight
student with that of the teacher, exploiting the student’s learning ability to
maximize knowledge. They also increase the dimension of the student’s
projection head to better mimic the teacher’s embedding.

Figure 3: The pipeline of the proposed Retro technique. Two different data
augmentation techniques first transform a single image into two views. A self-
supervised pre-trained teacher is added in addition to the original
contrastive SSL component, and the final embeddings generated by the learnable
student and the frozen teacher must be consistent for each view. The
contrastive is conducted with both views, and the projection head is frozen
and reused in the student model.

However, the questions of which knowledge is essential for the student and how
to efficiently distill it remain unanswered. Moreover, previous approaches
focused only on making the student mimic the teacher instead of exploiting the
student’s learning ability. Our proposed method aims to enhance the self-
supervised representation learning ability of lightweight models by aligning
the student encoder with the teacher’s projection head instead of merely
mimicking the teacher. Figure 2 illustrates the differences between our
proposed method and CompRess, SEED, and DisCo.

In this section, we will provide a detailed description of our proposed
method, Retro. We will start by reviewing the preliminary concepts of
contrastive-learning-based SSL. Next, we will discuss the overall framework of
Retro and explain how it works. Finally, we will introduce the objective of
Retro and describe the process of updating its parameters.

###  3.2 Preliminary on Contrastive Learning Based SSL

####  3.2.1 Contrastive Learning Based SSL

In contrastive-learning-based SSL, the goal is to predict whether a pair of
instances belong to the same class or different classes. The two instances are
obtained by applying different data augmentation techniques to the same input
image $x$, resulting in two augmented views $v$ and $v^{\prime}$ of the same
instance. The objective is to make the two views similar while views of
different instances should be dissimilar. Each view is then passed through two
encoders $f_{s}\left(\cdot\right)$ and $f_{t}\left(\cdot\right)$ to obtain the
corresponding representations $z_{s}$ and $z_{t}$.

To map the high-dimensional representations to a lower-dimensional embedding,
a projection head $g\left(\cdot\right)$, which is a non-linear MLP, is used.
Specifically, $g$ takes the representation $z$ as input and maps it to an
embedding $E$ as $E=g\left(z\right)=g\circ f\left(x\right)$. These embeddings
are then used to estimate the similarity in contrastive learning. The
projection head is crucial to the success of self-supervised learning, as
demonstrated in prior works such as MoCo [14] and SimCLR [3]. The encoder can
be any network architecture, such as ResNet or EfficientNet.

The projection head consists of two linear layers followed by a non-linear
activation function such as ReLU. The output dimension of the projection head
is typically set to a smaller value, such as $128$, to obtain a low-
dimensional embedding.

####  3.2.2 DisCo

In DisCo, the input $x$ is transformed into two views $v$ and $v^{\prime}$
using two different augmentation strategies in each iteration. The views are
fed into both the student encoder $f_{s}$ and the teacher encoder $f_{t}$ to
create four representations $z_{s}$, $z^{\prime}_{s}$, $z_{t}$, and
$z^{\prime}_{t}$. These representations are then projected using two different
projection heads $g_{s}$ and $g_{t}$ to produce low-dimensional embeddings
$E_{s}$, $E^{\prime}_{s}$, $E_{t}$, and $E^{\prime}_{t}$, respectively. The
same process is also applied with the mean student, resulting in
representations $z_{m}$, $z^{\prime}_{m}$ and embeddings $E_{m}$,
$E^{\prime}_{m}$. The embeddings are then used to compute the contrastive
learning loss using InfoNCE loss [19], similar to MoCo [14], as follows:

$\mathcal{L}_{\text{con}}=-\log\frac{\exp\left(\mathbf{q}\cdot\mathbf{k}^{+}/\tau\right)}{\sum_{i=0}^{K}\exp\left(\mathbf{q}\cdot\mathbf{k}_{i}/\tau\right)},$
(1)

where $\mathbf{q}$ is the embedding $E_{s}$ of the student on view $v$,
$\mathbf{k}$ is the embedding $E^{\prime}_{m}$ of the mean student on view
$v^{\prime}$, $\tau$ is the temperature, and $K$ is the size of the memory
bank. Additionally, the embeddings are used to compute a consistency loss
using cosine similarity, which is represented using mean squared error (MSE)
as follows:

$\begin{gathered}\mathcal{L}_{\text{dis}}=\|E_{s}-E_{t}\|^{2}_{2}+\|E^{\prime}_{s}-E^{\prime}_{t}\|^{2}_{2}\end{gathered}$
(2)

###  3.3 Retro

The overall framework of Retro is illustrated in Figure 3. Retro comprises a
lightweight student $s\left(\cdot\right)$, a mean student
$m\left(\cdot\right)$, and a pre-trained frozen teacher $t\left(\cdot\right)$,
which is similar to the DisCo framework [10]. However, unlike DisCo, we
propose that the pre-trained teacher projection head can be used directly for
the students since it contains the most valuable knowledge. Therefore, the
objective is to train the student encoder to align the representation with the
teacher projection head, instead of learning to mimic the teacher’s behavior.
In addition, we leverage the power of the multi-view strategy by inputting
both views into the mean student and enforcing the similarity constraint on
pairs of embeddings between the student and the mean student.

To achieve this, we replace the student projection head with the teacher’s
projection head $g$, ensuring the consistency of all projection heads.
However, since the input dimension of the student projection head is smaller
than that of the teacher, we place an adapter $a\left(\cdot\right)$ between
the encoder and projection head to align the dimension. This process can be
formulated as $E_{s}=g\circ a\left(z_{s}\right)$, $E^{\prime}_{s}=g\circ
a\left(z^{\prime}_{s}\right)$, $E_{t}=g\left(z_{t}\right)$,
$E^{\prime}_{t}=g\left(z^{\prime}_{t}\right)$, $E_{m}=g^{\prime}\circ
a^{\prime}\left(z_{m}\right)$, and $E^{\prime}_{m}=g^{\prime}\circ
a^{\prime}\left(z^{\prime}_{m}\right)$, where $a^{\prime}\left(\cdot\right)$
and $g^{\prime}\left(\cdot\right)$ are the mean adapter and mean projection
head, respectively. These embeddings are then used to compute the contrastive
loss and consistency loss.

###  3.4 Loss function and parameter update process

We follow BYOL [12] to symmetrize the contrastive loss in Retro as follows:

$\displaystyle\mathcal{L}_{\text{con}}=\frac{1}{2}\left(-\log\frac{\exp\left(\mathbf{q}\cdot\mathbf{k^{{}^{\prime}}}_{+}/\tau\right)}{\sum_{i=0}^{K}\exp\left(\mathbf{q}\cdot\mathbf{k^{{}^{\prime}}}_{i}/\tau\right)}\right)+\frac{1}{2}\left(-\log\frac{\exp\left(\mathbf{q^{{}^{\prime}}}\cdot\mathbf{k}_{+}/\tau\right)}{\sum_{i=0}^{K}\exp\left(\mathbf{q^{{}^{\prime}}}\cdot\mathbf{k}_{i}/\tau\right)}\right)$
(3)

Here, $q$ and $q^{\prime}$ are the embeddings from the student, while $k$ and
$k^{\prime}$ are the embeddings from the mean student. We use two different
memory banks for the two different views $v$ and $v^{\prime}$, respectively.

The overall loss function of Retro is formulated as follows:

$\mathcal{L}=\mathcal{L}_{\text{dis}}+\gamma\mathcal{L}_{\text{con}}$ (4)

where $\mathcal{L}_{\text{dis}}$ is the consistency loss, and
$\mathcal{L}_{\text{con}}$ is the contrastive loss of the conventional SSL
method. $\gamma$ is a hyperparameter used to control the weight of the
contrastive loss, which is typically set to $1$. The parameters of the student
encoder are optimized using the objective from Equation 4, while the
parameters of the entire mean student are updated using the exponential moving
average strategy as follows:

$\theta_{k}\leftarrow m\theta_{k}+\left(1-m\right)\theta_{q}$ (5)

Here, $m\in[0,1)$ is the momentum coefficient, which is typically set to
$0.999$, and $\theta$ represents the model parameters.

##  4 Experiments

###  4.1 Implementation Details

We first pre-train the self-supervised teacher models on the ImageNet dataset
[24], which contains $1.3$ million training images and $50,000$ validation
images with $1,000$ categories, using the MoCo-V2 [14] framework. We use
ResNet as the backbone for the teacher models with different widths/depths,
such as ResNet-50 (22.4M), ResNet-101 (40.5M), ResNet-152 (55.4M), and
ResNet-50$\times$2 (94M). We pre-train ResNet-50/101 using the MoCo-V2
framework for $200$ epochs, ResNet-152 for $800$ epochs, while
ResNet-50$\times$2 is pre-trained using the SwAV [1] method for $400$ epochs.

We evaluate our Retro method on five lightweight networks as students,
including EfficientNet-B0 (4.0M), EfficientNet-B1 (6.4M), MobileNet-v3-Large
(4.2M), ResNet-18 (10.7M), and ResNet-34 (20.4M). We use the same learning
rate and optimizer as MoCo-V2 and train all student models for $200$ epochs.
During distillation, the teacher’s and student’s projection heads are frozen
for Retro, while SEED, DisCo, and BINGO only freeze the teacher. As a result,
Retro has fewer trainable parameters and simpler training objectives. Note
that SEED trains the models using a SwAV pre-trained teacher for $400$ epochs,
and BINGO uses CutMix regularization [28] and more positive samples
($5\times$) during training, resulting in a higher benchmark score.
Additionally, BINGO is not an end-to-end framework.

We later fine-tune the self-supervised distillation models for linear
evaluation on ImageNet for $100$ epochs. We set the initial learning rate to
$3$ for EfficientNet-B0/B1 and MobileNet-v3-Large, and $30$ for ResNet-18/34.
The learning rate is scheduled to decrease by a factor of $10$ at $60$ and
$80$ epochs, and we use SGD as the optimizer. We follow the other
hyperparameters strictly as in MoCo-V2 [14].

###  4.2 Linear Evaluation

The results presented in Table 1 demonstrate that students distilled by Retro
outperform their counterparts pre-trained by MoCo-V2 (Baseline), and also
outperform the prior state-of-the-art DisCo by a significant margin. However,
we have not included CompRess in our comparison since it uses a teacher that
was trained for $600$ epochs longer and distills for $400$ epochs longer than
SEED, DisCo, and Retro. Therefore, it would be unfair to compare Retro to
CompRess in this context.

Table 1: ImageNet Test Accuracy (%) using Linear Classification on Different
Student Architectures. In the table, $\lozenge$ indicates that the teacher and
students are pre-trained with MoCo-V2, while $\dagger$ indicates that the
teacher is pre-trained by SwAV. SEED distilled for $800$ epochs using
R-50$\times$2 as the teacher, while DisCo, BINGO, and Retro distilled for
$200$ epochs. "T" denotes the teacher, and "S" denotes the student. The
subscript in green represents the improvement compared to the MoCo-V2
baseline.

Method |  T S | Eff-b0 | Eff-b1 | Mob-v3 | R-18 | R-34
---|---|---|---|---|---|---
T-1 | T-5 | T-1 | T-5 | T-1 | T-5 | T-1 | T-5 | T-1 | T-5
Supervised | 77.1 | 93.3 | 79.2 | 94.4 | 75.2 | - | 72.1 | - | 75.0 | -
Self-supervised |  |  |  |  |  |  |  |  |  |
MoCo-V2 (Baseline)$\lozenge$ | 46.8 | 72.2 | 48.4 | 73.8 | 36.2 | 62.1 | 52.2 | 77.6 | 56.8 | 81.1
SSL Distillation |  |  |  |  |  |  |  |  |  |  |
SEED [9] | R-50 (67.4)$\lozenge$ | 61.3 | 82.7 | 61.4 | 83.1 | 55.2 | 80.3 | 57.6 | 81.8 | 58.5 | 82.6
DisCo [10] | R-50 (67.4)$\lozenge$ | 66.5 | 87.6 | 66.6 | 87.5 | 64.4 | 86.2 | 60.6 | 83.7 | 62.5 | 85.4
BINGO [27] | R-50 (67.4)$\lozenge$ | - | - | - | - | - | - | 61.4 | 84.3 | 63.5 | 85.7
Retro | R-50 (67.4)$\lozenge$ | 66.9 | 88.2 | 67.1 | 88.4 | 66.2 | 87.2 | 62.9 | 85.4 | 64.1 | 86.8
|  | ($20.1\uparrow$) | ($16.0\uparrow$) | ($18.7\uparrow$) | ($14.6\uparrow$) | ($30.0\uparrow$) | ($25.1\uparrow$) | ($10.7\uparrow$) | ($7.8\uparrow$) | ($7.3\uparrow$) | ($5.7\uparrow$)
SEED [9] | R-101 (70.3) | 63.0 | 83.8 | 63.4 | 84.6 | 59.9 | 83.5 | 58.9 | 82.5 | 61.6 | 84.9
DisCo [10] | R-101 (69.1)$\lozenge$ | 68.9 | 88.9 | 69.0 | 89.1 | 65.7 | 86.7 | 62.3 | 85.1 | 64.4 | 86.5
Retro | R-101 (70.3) | 69.3 | 89.8 | 69.9 | 89.9 | 67.5 | 88.6 | 64.8 | 86.6 | 66.1 | 87.9
|  | ($22.5\uparrow$) | ($17.6\uparrow$) | ($21.5\uparrow$) | ($16.1\uparrow$) | ($31.3\uparrow$) | ($26.5\uparrow$) | ($12.6\uparrow$) | ($9.0\uparrow$) | ($9.3\uparrow$) | ($6.8\uparrow$)
SEED [9] | R-152 (74.2) | 65.3 | 86.0 | 67.3 | 86.9 | 61.4 | 84.6 | 59.5 | 83.3 | 62.7 | 85.8
DisCo [10] | R-152 (74.1)$\lozenge$ | 67.8 | 87.0 | 73.1 | 91.2 | 63.7 | 84.9 | 65.5 | 86.7 | 68.1 | 88.6
BINGO [27] | R-152 (74.1)$\lozenge$ | - | - | - | - | - | - | 65.9 | 87.1 | 69.1 | 88.9
Retro | R-152 (74.1)$\lozenge$ | 69.8 | 90.2 | 73.7 | 91.4 | 68.0 | 86.2 | 66.9 | 88.1 | 69.4 | 89.9
|  | ($23.0\uparrow$) | ($18.0\uparrow$) | ($25.3\uparrow$) | ($17.6\uparrow$) | ($31.8\uparrow$) | ($24.1\uparrow$) | ($14.7\uparrow$) | ($10.5\uparrow$) | ($12.6\uparrow$) | ($8.8\uparrow$)
SEED [9] | R-50$\times$2 (77.3)$\dagger$ | 67.6 | 87.4 | 68.0 | 87.6 | 68.2 | 88.2 | 63.0 | 84.9 | 65.7 | 86.8
DisCo [10] | R-50$\times$2 (77.3)$\dagger$ | 69.1 | 88.9 | 64.0 | 84.6 | 58.9 | 81.4 | 65.2 | 86.8 | 67.6 | 88.6
BINGO [27] | R-50$\times$2 (77.3)$\dagger$ | - | - | - | - | - | - | 65.5 | 87.0 | 68.9 | 89.0
Retro | R-50$\times$2 (77.3)$\dagger$ | 70.2 | 90.4 | 73.8 | 91.6 | 70.1 | 89.2 | 65.9 | 87.1 | 68.9 | 89.7
|  | ($23.4\uparrow$) | ($18.2\uparrow$) | ($25.4\uparrow$) | ($17.8\uparrow$) | ($33.9\uparrow$) | ($27.1\uparrow$) | ($13.7\uparrow$) | ($9.5\uparrow$) | ($12.1\uparrow$) | ($8.6\uparrow$)

The results in Table 1 demonstrate that Retro outperforms prior methods SEED,
DisCo, and BINGO across all benchmarked models. Notably, when using ResNet-50
as the teacher, Retro achieves state-of-the-art top-1 accuracy on all student
models. Moreover, using ResNet-152 instead of ResNet-50 as the teacher leads
to a significant improvement in the performance of ResNet-34, from $56.8\%$ to
$69.4\%$. It is worth noting that when using Retro with ResNet-50/101 as the
teacher, the linear evaluation result of EfficientNet-B0 is very close to that
of the teacher, despite EfficientNet-B0 having only $9.4\%/16.3\%$ of the
parameters of ResNet-50/101.

###  4.3 Semi-supervised Linear Evaluation

We also evaluate our method in semi-supervised scenarios, following previous
methodologies. We use $1\%$ and $10\%$ sampled subsets of the ImageNet
training data (i.e., $13$ and $128$ samples per class, respectively) to fine-
tune the student models. As shown in Table 2, our Retro approach consistently
outperforms the baseline under any quantity of labeled data. Notably, our
method achieves these results while strictly following the settings from SEED
[9] and DisCo [10], whereas BINGO uses a higher learning rate ($10$) for the
classifier layer.

Moreover, our experiments demonstrate that Retro is stable under varying
percentages of annotations, indicating that students always benefit from being
distilled by larger teacher models. The results also suggest that having more
labeled data can help improve the final performance of the student models.

Table 2: Semi-supervised learning by fine-tuning 1% and 10% images on ImageNet using ResNet-18. Method | T | 1% labels | 10% labels
---|---|---|---
MoCo-V2 (Baseline) | - | 30.9 | 45.8
SEED[9] | R-50 (67.4) | 39.1 | 50.2
DisCo[10] | R-50 (67.4) | 39.2 | 50.1
BINGO[27] | R-50 (67.4) | 42.8 | 57.5
Retro | R-50 (67.4) | 43.1 | 57.9
SEED[9] | R-101 (70.3) | 41.4 | 54.8
DisCo[10] | R-101 (69.1) | 47.8 | 54.7
Retro | R-101 (70.3) | 49.2 | 60.5
SEED[9] | R-152 (74.1) | 44.3 | 54.8
DisCo[10] | R-152 (74.1) | 47.1 | 54.7
BINGO[27] | R-152 (74.1) | 50.3 | 61.2
Retro | R-152 (74.1) | 50.9 | 62.0
BINGO[27] | R-50$\times$2 (77.3) | 48.2 | 60.2
Retro | R-50$\times$2 (77.3) | 50.6 | 61.9
Table 3: Object detection and instance segmentation results on VOC-07 test and
COCO val2017 using ResNet-34 as the backbone. The subscript in green
represents the improvement compared to the MoCo-V2 baseline.

S | T | Method | Object Detection | Instance Segmentation
---|---|---|---|---
VOC | COCO | COCO
$AP^{bb}$ | $AP^{bb}_{50}$ | $AP^{bb}_{75}$ | $AP^{bb}$ | $AP^{bb}_{50}$ | $AP^{bb}_{75}$ | $AP^{mk}$ | $AP^{mk}_{50}$ | $AP^{mk}_{75}$
R-34 | $\times$ | MoCo-V2 | 53.6 | 79.1 | 58.7 | 38.1 | 56.8 | 40.7 | 33.0 | 53.2 | 35.3
R-50 | SEED [9] | 53.7 | 79.4 | 59.2 | 38.4 | 57.0 | 41.0 | 33.3 | 53.2 | 35.3
DisCo [10] | 56.5 | 80.6 | 62.5 | 40.0 | 59.1 | 43.4 | 34.9 | 56.3 | 37.1
Retro | 57.2 | 81.4 | 63.3 | 41.5 | 60.2 | 45.3 | 35.9 | 57.8 | 38.8
| ($3.6\uparrow$) | ($2.3\uparrow$) | ($4.6\uparrow$) | ($3.4\uparrow$) | ($3.4\uparrow$) | ($4.6\uparrow$) | ($2.9\uparrow$) | ($4.6\uparrow$) | ($3.5\uparrow$)
R-101 | SEED [9] | 54.1 | 79.8 | 59.1 | 38.5 | 57.3 | 41.4 | 33.6 | 54.1 | 35.6
DisCo [10] | 56.1 | 80.3 | 61.8 | 40.0 | 59.1 | 43.2 | 34.7 | 55.9 | 37.4
Retro | 57.3 | 81.8 | 63.5 | 41.5 | 60.3 | 45.4 | 36.0 | 57.8 | 38.9
| ($3.7\uparrow$) | ($2.7\uparrow$) | ($4.8\uparrow$) | ($3.4\uparrow$) | ($3.5\uparrow$) | ($4.7\uparrow$) | ($3.0\uparrow$) | ($4.6\uparrow$) | ($3.6\uparrow$)
R-152 | SEED [9] | 54.4 | 80.1 | 59.9 | 38.4 | 57.0 | 41.0 | 33.3 | 53.7 | 35.3
DisCo [10] | 56.6 | 80.8 | 63.4 | 39.4 | 58.7 | 42.7 | 34.4 | 55.4 | 36.7
BINGO [27] | - | - | - | 39.9 | 59.4 | 43.5 | 35.7 | 56.5 | 38.2
Retro | 57.5 | 81.9 | 64.1 | 41.4 | 60.6 | 45.4 | 36.1 | 57.3 | 39.2
|  | ($3.9\uparrow$) | ($2.8\uparrow$) | ($5.4\uparrow$) | ($3.3\uparrow$) | ($3.8\uparrow$) | ($4.7\uparrow$) | ($3.1\uparrow$) | ($4.1\uparrow$) | ($3.8\uparrow$)

###  4.4 Transfer to CIFAR-10/CIFAR-100

We conducted further evaluations to assess the generalization of
representations obtained by Retro on CIFAR-10 and CIFAR-100 datasets, using
ResNet-18/EfficientNet-B0 as a student and ResNet-50/ResNet-101/ResNet-152 as
a teacher. The models were fine-tuned for $100$ epochs, with an initial
learning rate of $3$ and the learning rate scheduler decreasing by a factor of
$10$ at $60$ and $80$ epochs. All images were resized to $224\times 224$,
following the methodology from [9] since the original image resolution of the
CIFAR dataset is $32\times 32$. The results presented in Figure 4 show that
Retro outperforms prior methods SEED and DisCo across the datasets.
Furthermore, the improvement brought by Retro becomes more apparent as the
quality of the teacher models improves.

(a) ResNet-18 (b) EfficientNet-B0 (c) ResNet-18 (d) EfficientNet-B0

Figure 4: Top-1 accuracy for transfer learning on CIFAR-10 (4a, 4b) and
CIFAR-100 (4c, 4d) dataset.

###  4.5 Transfer to Detection and Segmentation

For segmentation and downstream detection tasks, we adopt the fine-tuning
methodology used in SEED [9] and DisCo [10], where all parameters are fine-
tuned. For the detection task on VOC [8], the model is initialized with an
initial learning rate of $0.1$, with $200$ warm-up iterations, and the
learning rate is decreased by a factor of $10$ at $18$k and $22.2$k steps. The
detector is trained for $48k$ steps, with a total batch size of $32$. During
training, the input is randomly scaled from $400$ to $800$, and set to $800$
during inference. For detection and segmentation on COCO [17], the model is
trained for $180k$ iterations with an initial learning rate of $0.11$. During
training, the image scale is randomized from $600$ to $800$.

We also performed tests on detection and segmentation tasks for generalization
analysis. Faster R-CNN [22] based on C4 is used for object detection for VOC
and R-CNN Mask [13] is used for object detection and version segmentation for
COCO. The results are displayed in Table 3. In object detection, our method
can yield clear improvements for both VOC and COCO datasets. Also, as claimed
by SEED [9], the COCO training dataset has 118k images, while VOC has only
16.5k training images, so the improvement of COCO is relatively small compared
to VOC. Therefore, the gain from initialization weight is relatively small.
Retro also has an advantage when it comes to instance segmentation tasks.

###  4.6 Impact of each contribution:

In this section, we report the effectiveness of each of our contributions. ①
Reusing teacher projection head and ② symmetric contrastive learning loss. We
verify this via students that are trained with ResNet-50 as a teacher.

Table 4: ImageNet top-1 accuracy (%) using linear classification on different strategies. Setting Student Model | Eff-b0 | Eff-b1 | Mob-v3 | R-18 | R-34
---|---|---|---|---|---
MoCo-V2 (Baseline) | 46.8 | 48.4 | 36.2 | 52.2 | 56.8
DisCo [10] | 66.5 | 66.6 | 64.4 | 60.6 | 62.5
\+ ① | 66.7 | 66.9 | 65.8 | 62.5 | 63.4
\+ ② (Retro) | 66.9 | 67.1 | 66.2 | 62.9 | 64.1

###  4.7 Computational Complexity:

As illustrated in Figure 3, the computational cost of Retro is higher compared
to SEED [9] and DisCo [10] due to the additional forward propagation required
for the mean student. The total number of forward propagation is $6$, which is
three times higher than SEED and MoCo-V2. However, these additional forward
propagations are not used during inference, so there is no overhead at
inference time. The results in Table 5 show that Retro has a lower number of
learnable parameters than DisCo. Therefore, the run-time overhead of Retro is
small and negligible compared to DisCo and BINGO. It should be noted that
BINGO requires a KNN run to create a bag of positive samples, while Retro is
an end-to-end approach.

Table 5: Comparison for the number of learnable parameters between DisCo and Retro. Method | Eff-b0 | Eff-b1 | Mob-v3 | R-18 | R-34
---|---|---|---|---|---
DisCo [10] | 6.57M | 8.96M | 6.76M | 11.91M | 21.55M
Retro | 6.32M | 8.71M | 6.51M | 11.66M | 21.30M
| ($\downarrow 0.25$M) | ($\downarrow 0.25$M) | ($\downarrow 0.25$M) | ($\downarrow 0.25$M) | ($\downarrow 0.25$M)

###  4.8 Comparison with other Distillation:

Table 6: Top-1 linear classification accuracy on ImageNet utilizing various distillation techniques on the ResNet-18 student model (ResNet-50 is used as teacher model). Method | Top-1
---|---
MoCo-V2 (Baseline) [14] | 52.2
MoCo-V2 + KD [9] | 55.3
MoCo-V2 + RKD [20] | 61.6
DisCo + KD [10] | 60.6
DisCo + RKD [10] | 60.6
BINGO [27] | 61.4
Retro | 62.9

For further verifying the strengths of Retro, we conducted the comparison
against several different distillation strategies. We include feature-based
distillation (KD) and relation-based distillation (RKD), following DisCo [10]
and BINGO [27]. As shown in Table 6, Retro shows superior performance compared
with other distillation methods and surpasses them by a large margin.

###  4.9 Adapter settings

We visualize the adapter architecture for different student networks in Figure
5. The adapter for ResNet, EfficientNet, and MobileNet-v3 is a 1-D convolution
layer that receives output from the student encoder ($D_{s}$) and aligns that
for the teacher projection head ($D_{t}$), follows by a batch normalization
and a non-linear activation layer. Note that the adapter is placed right
before the last pooling layer.

(a) ResNet (b) EfficientNet (c) MobileNet-v3

Figure 5: Adapter structure for different student networks.

##  5 Frozen vs unfrozen projection head

We conduct the comparison between training Retro for $200$ epochs with frozen
projection head and training for $170$ epochs with frozen and $30$ epochs with
unfrozen projection head. As we can see from Table 7, the latter training
scheme produces better performance.

Table 7: Comparison for the training scheme of Retro. Method | Eff-b0 | Eff-b1 | Mob-v3 | R-18 | R-34
---|---|---|---|---|---
$170$/$30$ frozen/unfrozen epochs | 66.2 | 66.7 | 65.8 | 62.1 | 63.8
$200$ frozen epochs | 66.9 | 67.1 | 66.2 | 62.9 | 64.1

##  6 Conclusion

In this paper, we introduce the Reusing Teacher Projection head strategy
(Retro), a novel approach for efficiently distilling self-supervised pre-
trained teachers on lightweight models. Additionally, we impose symmetry
contrastive learning to improve the representation between the student and
mean student model. Despite its simplicity, our method outperforms prior
methods by a large margin, demonstrating the importance of the projection head
in distillation on lightweight models with fewer learnable parameters. Retro
does not introduce any overhead during the inference phase. Our experiments
show its superior performance across a range of architectures and tasks.

##  7 Funding

This work was supported by the National Cancer Center Grant (NCC-2311350-2).

##  8 Ethical Statement

This study does not contain any studies with human or animal subjects
performed by any of the authors.

##  9 Conflicts of Interest

The authors declare that the research was conducted in the absence of any
commercial or financial relationships that could be construed as a potential
conflict of interest.

##  10 Data Availability Statement

The datasets ImageNet for this study can be downloaded at: https://www.image-
net.org/.

## References

  *   * Caron et al [2020] Caron M, Misra I, Mairal J, et al (2020) Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems 33:9912–9924
  * Caron et al [2021] Caron M, Touvron H, Misra I, et al (2021) Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 9650–9660
  * Chen et al [2020a] Chen T, Kornblith S, Norouzi M, et al (2020a) A simple framework for contrastive learning of visual representations. In: International conference on machine learning, PMLR, pp 1597–1607
  * Chen et al [2020b] Chen T, Kornblith S, Swersky K, et al (2020b) Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems 33:22243–22255
  * Chen and He [2021] Chen X, He K (2021) Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 15750–15758
  * Chen et al [2020c] Chen X, Fan H, Girshick R, et al (2020c) Improved baselines with momentum contrastive learning. arXiv preprint arXiv:200304297
  * Doersch et al [2015] Doersch C, Gupta A, Efros AA (2015) Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE international conference on computer vision, pp 1422–1430
  * Everingham et al [2015] Everingham M, Eslami SA, Van Gool L, et al (2015) The pascal visual object classes challenge: A retrospective. International journal of computer vision 111:98–136
  * Fang et al [2021] Fang Z, Wang J, Wang L, et al (2021) Seed: Self-supervised distillation for visual representation. arXiv preprint arXiv:210104731
  * Gao et al [2022] Gao Y, Zhuang JX, Lin S, et al (2022) Disco: Remedying self-supervised learning on lightweight models with distilled contrastive learning. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI, Springer, pp 237–253
  * Gidaris et al [2018] Gidaris S, Singh P, Komodakis N (2018) Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:180307728
  * Grill et al [2020] Grill JB, Strub F, Altché F, et al (2020) Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems 33:21271–21284
  * He et al [2017] He K, Gkioxari G, Dollár P, et al (2017) Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969
  * He et al [2020] He K, Fan H, Wu Y, et al (2020) Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9729–9738
  * Hinton et al [2015] Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:150302531
  * Koohpayegani et al [2020] Koohpayegani SA, Tejankar A, Pirsiavash H (2020) Compress: Self-supervised learning by compressing representations. Advances in neural information processing systems
  * Lin et al [2014] Lin TY, Maire M, Belongie S, et al (2014) Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, Springer, pp 740–755
  * Noroozi and Favaro [2016] Noroozi M, Favaro P (2016) Unsupervised learning of visual representations by solving jigsaw puzzles. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, Springer, pp 69–84
  * Oord et al [2018] Oord Avd, Li Y, Vinyals O (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:180703748
  * Park et al [2019] Park W, Kim D, Lu Y, et al (2019) Relational knowledge distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3967–3976
  * Pathak et al [2016] Pathak D, Krahenbuhl P, Donahue J, et al (2016) Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2536–2544
  * Ren et al [2015] Ren S, He K, Girshick R, et al (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28
  * Romero et al [2014] Romero A, Ballas N, Kahou SE, et al (2014) Fitnets: Hints for thin deep nets. arXiv preprint arXiv:14126550
  * Russakovsky et al [2015] Russakovsky O, Deng J, Su H, et al (2015) Imagenet large scale visual recognition challenge. International journal of computer vision 115:211–252
  * Tian et al [2019] Tian Y, Krishnan D, Isola P (2019) Contrastive representation distillation. arXiv preprint arXiv:191010699
  * Xu et al [2020] Xu G, Liu Z, Li X, et al (2020) Knowledge distillation meets self-supervision. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX, Springer, pp 588–604
  * Xu et al [2021] Xu H, Fang J, Zhang X, et al (2021) Bag of instances aggregation boosts self-supervised distillation. arXiv preprint arXiv:210701691
  * Yun et al [2019] Yun S, Han D, Oh SJ, et al (2019) Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6023–6032