#  Ca2Lib: Simple and Accurate LiDAR-RGB Calibration
using Small Common Markers

Emanuele Giacomini* Leonardo Brizi* Luca Di Giammarino Omar Salem

Patrizio Perugini Giorgio Grisetti  All authors are with the Department of
Computer, Control, and Management Engineering “Antonio Ruberti”, Sapienza
University of Rome, Italy, Email: {giacomini, brizi, digiammarino, salem,
<EMAIL_ADDRESS>work has been supported by PNRR MUR project
PE0000013-FAIR* The authors contributed equally.

###### Abstract

In many fields of robotics, knowing the relative position and orientation
between two sensors is a mandatory precondition to operate with multiple
sensing modalities. In this context, the pair LiDAR-RGB cameras offer
complementary features: LiDARs yield sparse high quality range measurements,
while RGB cameras provide a dense color measurement of the environment.

Existing techniques often rely either on complex calibration targets that are
expensive to obtain, or extracted virtual correspondences that can hinder the
estimate’s accuracy.

In this paper we address the problem of LiDAR-RGB calibration using typical
calibration patterns (i.e. A3 chessboard) with minimal human intervention. Our
approach exploits the planarity of the target to find correspondences between
the sensors measurements, leading to features that are robust to LiDAR noise.
Moreover, we estimate a solution by solving a joint non-linear optimization
problem.

We validated our approach by carrying on quantitative and comparative
experiments with other state-of-the-art approaches. Our results show that our
simple schema performs on par or better than other approches using complex
calibration targets. Finally, we release an open-source C++ implementation at
https://github.com/srrg-sapienza/ca2lib

##  I Introduction

The ability to fuse readings from heterogeneous sensors is often beneficial in
many robotics and perception applications. In particular, LiDAR and RGB
sensors exhibit a strong compatibility: the former being able to capture high
precision sparse range readings while the latter measure dense color intensity
measurements.

These properties makes integration between the two sensors suited for the task
of _depth estimation_. Historically, stereo based solutions leverage the known
relative offset between two cameras, along with concepts from epipolar
geometry to estimate a depth value for every pixel in a image. Albeit its
popularity, due to its optical nature, these approaches suffer in texture-less
regions and in areas where the depth exceed a maximum value determined by the
baseline of the stereo. While the texture-less problem has been partially
solved by the usage of active stereo sensors (i.e. Realsense D435, Kinect),
the maximum range still poses a challenge. On the contrary, LiDARs operates
using Time of Flight (TOF) principle, which is proven to be robust at
detecting range measurements on non-reflective surfaces with accuracy even at
high distances.

These considerations lead the community to investigate the problem of _depth-
completion_ , namely estimating a dense depth image by superimposing an
accurate sparse depth measurement along an intensity RGB image. Multiple
publicly available datasets like KITTI and VOID [20][21] allowed the community
to tackle this problem either by fully leveraging the sparse depth measurement
(unguided) or by fusing RGB features (guided). Furthermore, in the field of 3D
reconstruction, recent findings show that coupling the two sensors may lead to
a more robust and accurate trajectory estimate [2].

Besides, to accomplish any of these tasks, one would require to know the
relative offset between the two sensors.

This work aims at solving the task of LiDAR-RGB calibration, namely,
estimating the relative offset (extrinsic parameters) between the two sensors,
using their raw measurements.

Figure 1: Reprojection of a LiDAR point cloud on a fisheye RGB camera rigidly
attached to the former. The offset between the sensors leads to shadows on
parts of the image.

The core idea behind this multi-modal calibration is to find spatial
correspondences between the sensors measurements. Most approaches rely on one
or more calibration patterns to establish common features between the sensors.
These patterns are often complex or expensive to produce [1]. The main
contribution of this paper is a versatile calibration toolbox that allows to
estimate the extrinsic parameters between LiDAR and RGB with minimal user
intervention, using a simple calibration checkerboard target. We leverage a
joint non-linear formulation of the problem to achieve high accuracy results
even with a minimum of three measurements. The requirement for our method is
to use a calibration pattern (e.g. Checkerboard, ChAruCO [5]) that must be
observed by both sensors during the acquisition. We exploit the planarity of
the target to find a common observation used to estimate the extrinsic
parameters. Moreover, we release an open-source implementation of our toolbox.

##  II Related Work

This paper’s section delves into LiDAR-RGB calibration and explores the two
main classes of approaches: _target-based_ and _target-less_. As the name
suggests, target-based approaches require the user to place artificial markers
that both the camera and LiDAR can easily detect. This contrasts with target-
less methods that free the use from this task. The core idea of calibration is
common in the two classes of approaches: computing common features between
heterogeneous measurements and estimating the transformation that minimizes
the distance between corresponding features.

First, an overview of _target-less_ approaches is presented: Pandey _et al._
presents an automatic data-driven approach based upon the maximization of
mutual information between the sensor-measured surface intensities [13]. The
authors exploits the correlation coefficient for the reflectivity and
intensity values of many scan-image pairs using different calibration
parameters. However, shadows of objects or colored surfaces that completely
absorb infrared lights might result in weaker correlation between scan-image
pairs. Yoon _et al._ proposes a calibration method using region-based object
pose estimation. Objects are segmented in both measurements, then a 3D mesh is
generated from the LiDAR measurements, while images are used to reconstruct
the scene using Structure from Motion (SfM). The two models are then
registered together to acquire an initial guess on the relative pose. The
final solution is obtained iteratively by finding correspondences between the
reconstructed objects from both measurements [22]. In recent years, the
development of learning based methods have also spanned in this field: Lv _et
al._ proposes a real-time self-calibration network that predict the extrinsic
parameters by constructing a cost volume between RGB and LiDAR features [11],
while Sun _et al._ first estimates an initial guess by solving an hand-eye
calibration method [18]. Moreover, the guess is fine-tuned by segmenting the
image-cloud pair and by aligning the distances between centroids. The
advantage of target-less method is that they can be used without preparing the
environment. This comes at a cost of a lower accuracy and robustness when
compared to their target-based counterpart.

_Target-based_ methods estimate the relative pose using an observed known
structure. Given the difference of resolution for the two sensors, it is high
unlikely that correspondences within the measurements can be established
directly. For this reason, _point-to-point_ methods tends to process LiDAR
measurements to implicitly obtain virtual points111Points that are not
explicitly detected, but estimated from the LiDAR measurement. easily
detectable from an RGB sensor. For instance, Park _et al._ utilizes a
specially designed polygonal planar calibration board with known lengths of
adjacent sides [14]. By estimating the 3D corresponding points from the LiDAR,
vertices of the board can be determined as the meeting points of two projected
sides. The vertices, along with the corresponding points detected from the
color image, are used for calibration. Pusztai _et al._ introduces a
methodology that utilizes cubic objects with predetermined side lengths [15,
16]. The corners of the cubes are estimated by initially detecting each side
of the box and subsequently determining their intersection points.
Furthermore, the corners along with their corresponding RGB image are employed
to calibrate the system by solving Iterative Corresponding Point (ICP). _Zhou_
_et al._ proposes a single-shot calibration method requiring a checkerboard
[23]. The target is detected both in the RGB image, and LiDAR measurement,
using RANSAC[4] for the latter. Furthermore, the four edges of the
checkerboard are estimated and aligned to compute the relative offset between
the two sensors. Tóth _et al._ introduces a fully automatic calibration
technique that leverages the utilization of spheres, enabling accurate
detection in both point clouds and camera images [19]. Upon successful
detection, the algorithm aligns the set of sphere centers using SVD. Beltrán
_et al._ presents a methodology that utilizes a custom calibration target
equipped with 4 holes and AruCO markers specifically designed for monocular
detection [1]. The methodology employs a set of techniques for each sensor to
estimate the center points of the holes. Subsequently, the relative offset
between sensors are determined by aligning the set of centers obtained from
each sensor, Li _et al._ adopt a similar approach while using a checkerboard
with 4 holes [10]. Fan _et al._ propose a two-stage calibration method using
an auxiliary device with distinctive geometric features [3]. The method
extracts lines from images and LiDAR point clouds, providing an initial
estimation of the external parameters. Nonlinear optimization is then applied
to refine these parameters. In the work of Singandhupe _et al._ , the authors
first extract planar information from RGB and LiDAR measurements, then, two
grid of points are extracted from the computed planar patches and aligned
using a customized ICP algorithm [17].

Albeit these approaches provides relatively accurate results with few
measurements, care should be taken during the estimation of virtual
correspondences, as they can cause significant errors in the estimation step.
Moreover these custom targets often requires precise construction or expensive
manufacturing.

Another group of approaches does not directly solve the calibration problem
using point-to-point correspondences, but rather exploit the planarity of the
target to reduce the feasible set of solutions using _plane-to-plane_
constraints. Mirzaei _et al._ addresses the challenge of accurate initial
estimates by dividing the problem into two sub-problems and analytically
solving each to obtain precise initial estimates [12]. The authors then refine
these estimates through iterative minimization. They also discuss the
identifiability conditions for accurate parameter estimation. Finally, a
method similar to our proposal, Kim _et al._ combine observed normals to first
estimate the relative orientation with SVD and then iteratively estimates an
initial guess of the relative translation by minimizing the pairwise planar
distances between measurements [8]. Finally, the translation is refined using
a non-linear optimization problem using Levenberg-Marquardt (LM). Despite its
simplicity, this method decouples the estimation of orientation and
translation, thus leading to potential losses in accuracy while also
increasing the number of required measurements.

Compared with the state of the art, we propose:

  * •

a formulation for joint nonlinear optimization that couples relative rotation
and translation using a plane-to-plane metric;

  * •

an extensible framework that decouples the optimization from target detection.
Currently supports Checkerboard and _ChARuCO_ patterns of typical A3-A4 sizes,
easily obtainable from commercial printers;

  * •

the possibility to handle different camera models and distortion;

  * •

an open-source C++ implementation.

##  III Our Approach

In this section, we will provide a detailed and comprehensive description of
our method. First we describe the preliminaries required to understand our
approach, then every component of the pipeline is described, following the
procedure from the acquisition of the measurements up to the computation of
the relative poses between the two sensors (extrinsic parameter).

Plane Representation: Let $\pi=(\hat{\mathbf{n}},d)$ be a 3D plane, where
$\hat{\mathbf{n}}\in\mathbb{S}^{2}$ represents the unit vector orthogonal to
the plane and $d\in\mathbb{R}$ is the shortest distance of the plane respect
to the origin. Applying a transform $\mathbf{X}\in\mathbb{SE}(3)$ to a plane
$\pi$ yields new coefficients $\pi^{\prime}$: as follows:

$\mathbf{X}\pi=\left\\{\begin{array}[]{ll}\hat{\mathbf{n}}^{\prime}=&\mathbf{R}\mathbf{n}\\\
d^{\prime}=&d+(\mathbf{R}\mathbf{n})^{t}\mathbf{t}\end{array}\right.$ (1)

Here $\mathbf{X}=\left<\mathbf{R};\mathbf{t}\right>$ is represented by a
rotation matrix $\mathbf{R}\in\mathbb{SO}(3)$, and the translation vector
$\mathbf{t}\in\mathbb{R}^{3}$.

If the transformation is modified by a small local perturbation
$\mathbf{\Delta X}=(\mathbf{\Delta R}|\mathbf{\Delta t})$ then we can rewrite:

$(\mathbf{X}\boxplus\mathbf{\Delta}\mathbf{X})\pi=\left\\{\begin{array}[]{ll}\tilde{\mathbf{n}}=&\mathbf{\Delta
R}\mathbf{R}\mathbf{n}\\\
\tilde{d}=&d^{\prime}+\mathbf{n}^{t}\mathbf{R}^{t}\mathbf{\Delta
R}^{t}\mathbf{\Delta t}\end{array}\right.$ (2)

Deriving the result with respect to $\mathbf{\Delta}\mathbf{X}$ leads to the
following Jacobian:

$\frac{\partial(\mathbf{X}\boxplus\mathbf{\Delta}\mathbf{X})\pi}{\partial\mathbf{\Delta}\mathbf{X}}=\begin{bmatrix}&0_{3\times
3}&-{\lfloor{\mathbf{R}\mathbf{n}}\rfloor}_{\times}\\\
&\mathbf{n}^{t}\mathbf{R}^{t}&0_{1\times 3}\end{bmatrix}_{4\times 6}$ (3)

The distance between two planes depends both on the difference between their
normals and the signed distance of the planes from the origin. These
quantities can be captured by a 4D error vector $e_{p}$ expressing the _plane-
to-plane_ error metric:

$\displaystyle\mathbf{p}(\pi_{k})$ $\displaystyle=-\mathbf{n}_{k}d_{k}$ (4)
$\displaystyle e_{p}(\pi_{i},\pi_{j})$
$\displaystyle=\begin{bmatrix}\mathbf{n}_{i}^{t}(\mathbf{p}(\pi_{i})-\mathbf{p}(\pi_{j}))\\\
\mathbf{n}_{j}-\mathbf{n}_{i}\end{bmatrix}.$ (5)

Here $\mathbf{p}(\pi_{k})$ is the point on the plane closest to the origin of
the reference system, and it is obtained by taking a point along the normal
direction $\mathbf{n}$ at distance $d$.

Pinhole Model (RGB): Let $\mathbf{p}$ be a point expressed in camera frame and
$\mathbf{K}$ be t camera matrix. Assuming any lens distortion effect have been
previously corrected, then the projection on the image plane of $\mathbf{p}$
is computed as

$\displaystyle\pi_{\mathrm{c}}(\mathbf{p})$
$\displaystyle=\phi(\mathbf{K}\mathbf{p})$ (6) $\displaystyle\mathbf{K}$
$\displaystyle=\begin{bmatrix}f_{x}&0&c_{x}\\\ 0&f_{y}&c_{y}\\\
0&0&1\end{bmatrix}$ (7) $\displaystyle\phi(v)$
$\displaystyle=\frac{1}{v_{z}}\begin{bmatrix}v_{x}\\\ v_{y}\end{bmatrix}$ (8)

where $\phi(v)$ represents the homogeneous division and
$\pi_{\mathrm{c}}(\mathbf{p})$ the pinhole projection function. For
simplicity, we detail only the pinhole camera projection, however the same
principle applies for more complex camera models.

Projection by ID (LiDAR): Let $\mathbf{p}$ be a point detected by the LiDAR
and expressed in its frame. Its projection is computed as:

$\displaystyle\pi_{\mathrm{l}}(\mathbf{p})$
$\displaystyle=\mathbf{A}\psi(\mathbf{p})$ (9) $\displaystyle\mathbf{A}$
$\displaystyle=\begin{bmatrix}f_{x}&0&c_{x}\\\ 0&1&0\end{bmatrix}$ (10)
$\displaystyle\psi(v)$
$\displaystyle=\begin{bmatrix}\operatorname*{atan2}(v_{y},v_{x})\\\
\textit{ring}(v)\\\ 1\end{bmatrix}$ (11)

where $f_{x}$ represent the azimuth resolution of the LiDAR, while $c_{x}$
denotes the offset in pixels. The $\textit{ring}(v)$ function is either
obtained directly from the LiDAR sensor, which augment every measured point
with a number that represents the beam that detected it or, assuming the cloud
is ordered, by dividing the point index by the horizontal resolution of the
sensor. Compared with the classical spherical projection, the projection by ID
does not preserve the geometric consistency of the scene. Still, it provides
an image with no holes, which is preferred for computer-vision applications.

First, we process the incoming raw LiDAR and RGB measurements to acquire
planar information. Assuming the scene to remain static throughout the
acquisition of a single joint measurement, the LiDAR cloud is embedded in an
image using the projection by ID. Moreover, the system awaits the user
interaction to guess the position of the calibration target on the LiDAR
image.

A parametric circular patch around the user’s selection is used to estimate a
plane using RANSAC and, concurrently, the calibration target detection is
attempted on the RGB image. Once the target is detected, the RGB plane is
computed by solving the ICP. If the user is satisfied with both LiDAR and RGB
planes, they are stored for processing.

Whereas a straightforward rank analysis of the Jacobians reveals that just 3
measurments are sufficient to constrain a solution, it is well known from the
estimation theory that the accuracy grows with the number of measurements.

Once the set of measurements are acquired, we jointly estimate the relative
orientation and translation of the LiDAR with respect to the RGB sensor
$\mathbf{X}\in\mathbb{SE}(3)$ by solving the following nonlinear minimization
problem:

$\mathbf{X}=\operatorname*{argmin}_{\mathbf{X}\in\mathbb{SE}(3)}\sum_{i\in\mathbf{Z}}\lVert\underbrace{\mathbf{X}\pi^{i}_{\mathrm{l}}-\pi^{i}_{\mathrm{c}}}_{e_{p}}\rVert^{2}$
(12)

where $e_{p}$ represent the plane-to-plane error.

During acquisition, it may happen that the user accept one or more wrongly
estimated measurements. Due to the quadratic nature of the error terms, these
_outliers_ are often over-accounted, resulting in wrong estimations. To
account for this factor, as described in [6], we employ an Huber _M-estimator_
$\rho(\cdot)$ that treats differently measurements based on their error. We
rewrite Eq. (12) as follows:

$\mathbf{X}=\operatorname*{argmin}_{\mathbf{X}\in\mathbb{SE}(3)}\sum_{i\in\mathbf{Z}}\rho(\lVert\mathbf{X}\pi^{i}_{\mathrm{l}}-\pi^{i}_{\mathrm{c}}\rVert).$
(13)

To resolve Eq. (13) we employ the Gauss-Newton (GN) algorithm implemented in
the srrg2_solver[6].

##  IV Experimental Evaluation

|  $\sigma_{\mathrm{l}}=0$ $\sigma_{\mathrm{c}}=0$ |  $\sigma_{\mathrm{l}}=8e^{-3}$ $\sigma_{\mathrm{c}}=7e^{-3}$ |  $\sigma_{\mathrm{l}}=16e^{-3}$ $\sigma_{\mathrm{c}}=14e^{-3}$
---|---|---|---
| mean | stdev | mean | stdev | mean | stdev
No. Measurements |  |  |  |  |  |
3 | 41.761 | 104.362 | 20.790 | 25.124 | 57.849 | 112.365
4 | 10.872 | 17.941 | 12.206 | 12.363 | 14.940 | 11.681
5 | 6.492 | 7.997 | 8.350 | 9.076 | 9.115 | 5.675
10 | 4.591 | 3.458 | 5.759 | 4.974 | 5.849 | 1.989
20 | 2.575 | 1.981 | 3.646 | 2.564 | 4.123 | 1.139
30 | 2.673 | 1.263 | 2.867 | 1.659 | 3.735 | 0.878
39 | 2.091 | 0.883 | 2.666 | 1.206 | 3.261 | 0.413
TABLE I: Average translation error in millimeters with different noise levels
and number of measurements.

In this section, we describe the experiments we conducted to establish the
quality of our calibration toolbox. We perform quantitative experiments in the
simulated environment provided by [1] to compare our estimates with
groundtruth while we also conduct qualitative and quantitive experiments on
real scenarios using our acquisition system. We directly compare our results
with [8], as it is the work which is closest to ours. In addition, we compare
to [1] that produce accurate results relying on a very complex target (CNC
printed).

###  IV-A Synthetic Case

We conducted experiments on _Gazebo_ simulator [9] to evaluate the accuracy
and robustness of our approach, injecting different noise figures to the
sensor measurements. We also experiment how the number of observations affect
the final results. The setup of the scene includes a Velodyne HDL-64 LiDAR, a
BlackFly-S RGB sensor and a $6\times 8$ checkerboard target with corner size
of $0.2$ meters. We randomly generate and acquire $53$ valid222A valid
measurement is one for which both LiDAR and RGB sensor are able to detect the
target measurements.

To quantify the impact of the number of measurements on the accuracy of our
approach, we run the calibration procedure with an increasing number of
measurements $w_{s}=[3\dots 39]$ and at three different LiDAR noise levels
$\sigma_{\mathrm{l}}$ ($0$ mm, $7$ mm and $14$ mm). For every $w_{s}$, we
sample $40$ sets of measurements.

From Tab. I, we observe a steady decrease of error for every noise level,
reaching an average of $2.6$mm translation error in the intermediate noise
case. In case of $3$ measurements the high uncertainty is due to the
potentially poorly conditioned system when using planes that have similar
normals. Nonetheless, we compare our best result with $3$ measurements against
the best results of the methods presented in [1] and [8]. Tab. II shows the
results.

Method |  $e_{t}$ (cm) |  $e_{r}$ ($10^{-2}$ rad)
---|---|---
_Beltrán_ _et al._[1] | 0.82 | 0.50
_Kim_ _et al._[8] | 10.2 | 129.56
Ours | 0.11 | 0.25
TABLE II: Best solutions obtained by calibration using $N=3$ measurements.

###  IV-B Real Case

Figure 2: Acquisition system used for the Real Case experiments.

In this section, we describe the experiments conducted on real measurements.
We perform a quantitative test on our acquisition system shown in Fig. 2 that
is equipped with a Ouster OS0-128 LiDAR with a resolution of $128\times 1024$,
a RealSense T-265 stereo camera and two Manta-G145 RGB cameras arranged in a
wide horizontal stereo configuration.

Since no groundtruth information is available, we take advantage of the stereo
extrinsics to provide an estimate of the calibration error. The offset between
multiple camera is measured using optical calibration procedures which
typically reach subpixel precision.

In the first experiment, we consider the LiDAR and the Realsense T-265 sensor
which provides factory calibrated intrinsic/extrinsic parameters for both
cameras. The task of the experiment is to demonstrate the accuracy of the
calibrator in real case scenarios and to understand how the number of
measurements considered affects the quality of the solution.

As for the synthetic case, we first acquire a set of $17$ cloud-image LiDAR
RGB measurements for both cameras. Moreover we perform $40$ calibrations with
$w_{s}$ randomly selected measurements with $w_{s}\in\\{3,15\\}$. Finally, for
every $w_{s}$, we combine the computed extrinsics for both cameras to obtain
an estimate stereo transform. Assuming approximately symmetrical errors in the
two cameras, Fig. 3 shows the results of this experiment. We were able to
obtain at best an average error of $7.1$ mm in translation and $0.01$ rads in
orientation.

Figure 3: Average camera-wise calibration error in the LiDAR-T265 case.

The second experiment is conducted using the wide stereo setup for which we
also calibrate the intrinsics and extrinsics of the cameras, providing
expected results in a typical scenario. The acquisition procedure is the same
as in the first experiment and Fig. 4 shows the experimental result, where we
obtain the best solution with $4.6$ mm and $0.002$ rads in orientation.

Figure 4: Average camera-wise calibration error in the LiDAR-Manta case.

Moreover, Fig. 1 and Fig. 5 show the reprojection onto the right camera
respectively of the fisheye and wide baseline RGB sensor. In the latter, the
large parallax between the sensors leads to strong occlusions effects, that
have been mitigated with an hidden point removal algorithm [7].

Figure 5: Qualitative samples showing LiDAR cloud projection on RGB image.

In summary, our evaluation indicates that our method is capable of generating
extrinsic estimates that are comparable or superior to those obtained using
other state-of-the-art approaches. It is important to note that careful
consideration is required when selecting a minimal number of measurements.
However, our experiments clearly demonstrate that the accuracy of these
estimates improves as the number of measurements increases.

##  V Conclusion

In summary, our paper introduces a simple and effective method for accurately
estimating extrinsic parameters between LiDARs and RGB sensors. By leveraging
the inherent planarity of standard calibration patterns, we establish common
observations between these sensors, greatly simplifying the calibration
procedure. Our experiments show that planar features mitigate the LiDAR noise,
leading to accurate results even with common A3/A4 calibration patterns.
Finally, we also release an open source C++ implementation to benefit the
community.

## References

  * [1] Jorge Beltrán, Carlos Guindel, Arturo de la Escalera, and Fernando García.  Automatic extrinsic calibration method for lidar and camera sensor setups.  IEEE Trans. on Intelligent Transportation Systems (ITS), 23(10):17677–17689, 2022.
  * [2] Luca Di Giammarino, Emanuele Giacomini, Leonardo Brizi, Omar Salem, and Giorgio Grisetti.  Photometric lidar and rgb-d bundle adjustment.  IEEE Robotics and Automation Letters (RA-L), 2023.
  * [3] Song Fan, Ying Yu, Maolin Xu, and Longhai Zhao.  High-precision external parameter calibration method for camera and lidar based on a calibration device.  IEEE Access, 11:18750–18760, 2023.
  * [4] Martin A. Fischler and Robert C. Bolles.  Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.  Commun. ACM, 24(6):381–395, jun 1981.
  * [5] S. Garrido-Jurado, R. Muñoz-Salinas, F.J. Madrid-Cuevas, and M.J. Marín-Jiménez.  Automatic generation and detection of highly reliable fiducial markers under occlusion.  Pattern Recognition, 47(6):2280–2292, 2014.
  * [6] Giorgio Grisetti, Tiziano Guadagnino, Irvin Aloise, Mirco Colosi, Bartolomeo Della Corte, and Dominik Schlegel.  Least squares optimization: From theory to practice.  9(3), 2020.
  * [7] Sagi Katz, Ayellet Tal, and Ronen Basri.  Direct visibility of point sets.  ACM Trans. Graph., 26(3):24–es, jul 2007.
  * [8] Eung-su Kim and Soon-Yong Park.  Extrinsic calibration between camera and lidar sensors by matching multiple 3d planes.  IEEE Sensors Journal, 20(1), 2020.
  * [9] Nathan Koenig and Andrew Howard.  Design and use paradigms for gazebo, an open-source multi-robot simulator.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), volume 3, pages 2149–2154. IEEE, 2004.
  * [10] Xingxing Li, Feiyang He, Shengyu Li, Yuxuan Zhou, Chunxi Xia, and Xuanbin Wang.  Accurate and automatic extrinsic calibration for a monocular camera and heterogenous 3d lidars.  IEEE Sensors Journal, 22(16):16472–16480, 2022.
  * [11] Xudong Lv, Boya Wang, Ziwen Dou, Dong Ye, and Shuo Wang.  Lccnet: Lidar and camera self-calibration using cost volume network.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 2894–2901, 2021.
  * [12] Faraz M Mirzaei, Dimitrios G Kottas, and Stergios I Roumeliotis.  3d lidar–camera intrinsic and extrinsic calibration: Identifiability and analytical least-squares-based initialization.  Intl. Journal of Robotics Research (IJRR), 31(4):452–467, 2012\.
  * [13] Gaurav Pandey, James R McBride, Silvio Savarese, and Ryan M Eustice.  Automatic extrinsic calibration of vision and lidar by maximizing mutual information.  32(5):696–722, 2015.
  * [14] Yoonsu Park, Seokmin Yun, Chee Sun Won, Kyungeun Cho, Kyhyun Um, and Sungdae Sim.  Calibration between color camera and 3d lidar instruments with a polygonal planar board.  IEEE Sensors Journal, 14(3):5333–5353, 2014.
  * [15] Zoltan Pusztai and Levente Hajder.  Accurate calibration of lidar-camera systems using ordinary boxes.  In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pages 394–402, 2017.
  * [16] Zoltán Pusztai, Iván Eichhardt, and Levente Hajder.  Accurate calibration of multi-LiDAR-multi-camera systems.  18(7):2139.
  * [17] Ashutosh Singandhupe, Hung Manh La, and Quang Phuc Ha.  Single frame lidar-camera calibration using registration of 3d planes.  In 2022 Sixth IEEE International Conference on Robotic Computing (IRC), pages 395–402, 2022.
  * [18] Chao Sun, Zhijie Wei, Wenyi Huang, Qianfei Liu, and Bo Wang.  Automatic targetless calibration for lidar and camera based on instance segmentation.  IEEE Robotics and Automation Letters (RA-L), 8(2):981–988, 2022\.
  * [19] Tekla Tóth, Zoltán Pusztai, and Levente Hajder.  Automatic lidar-camera calibration of extrinsic parameters using a spherical target.  In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), pages 8580–8586, 2020.
  * [20] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger.  Sparsity invariant cnns, 2017.
  * [21] Alex Wong, Xiaohan Fei, Stephanie Tsuei, and Stefano Soatto.  Unsupervised depth completion from visual inertial odometry.  IEEE Robotics and Automation Letters (RA-L), 5(2):1899–1906, 2020\.
  * [22] Byung-Hyun Yoon, Hyeon-Woo Jeong, and Kang-Sun Choi.  Targetless multiple camera-lidar extrinsic calibration using object pose estimation.  In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), pages 13377–13383. IEEE, 2021.
  * [23] Lipu Zhou, Zimo Li, and Michael Kaess.  Automatic extrinsic calibration of a camera and a 3d lidar using line and plane correspondences.  pages 5562–5569, 2018.


# Distilling Calibrated Student from an Uncalibrated Teacher

Ishan Mishra
Indian Institute of Technology, Jodhpur
Jodhpur, Rajasthan, India
<EMAIL_ADDRESS>Sethu Vamsi Krishna
Indian Institute of Technology, Jodhpur
Jodhpur, Rajasthan, India
<EMAIL_ADDRESS>Deepak Mishra
Indian Institute of Technology, Jodhpur
Jodhpur, Rajasthan, India
<EMAIL_ADDRESS>
###### Abstract

Knowledge distillation is a common technique for improving the performance of
a shallow student network by transferring information from a teacher network,
which in general, is comparatively large and deep. These teacher networks are
pre-trained and often uncalibrated, as no calibration technique is applied to
the teacher model while training. Calibration of a network measures the
probability of correctness for any of its predictions, which is critical in
high-risk domains. In this paper, we study how to obtain a calibrated student
from an uncalibrated teacher. Our approach relies on the fusion of the data-
augmentation techniques, including but not limited to cutout, mixup, and
CutMix, with knowledge distillation. We extend our approach beyond traditional
knowledge distillation and find it suitable for Relational Knowledge
Distillation and Contrastive Representation Distillation as well. The novelty
of the work is that it provides a framework to distill a calibrated student
from an uncalibrated teacher model without compromising the accuracy of the
distilled student. We perform extensive experiments to validate our approach
on various datasets, including CIFAR-10, CIFAR-100, CINIC-10 and TinyImageNet,
and obtained calibrated student models. We also observe robust performance of
our approach while evaluating it on corrupted CIFAR-100C data.

##  1 Introduction

Recent advances in deep learning have made deep neural networks (DNNs) [9, 11,
25] achieve human-like performance in terms of accuracy. However, these models
are often overconfident [7, 16] and deploying them in high-risk domains like
medical, astronomical, autonomous vehicles, etc., becomes unsafe. Thus, the
calibration of DNNs becomes critical for trustworthy AI, which helps in
aligning the model’s confidence with its accuracy. Various techniques like
vector scaling [7, 20], matrix scaling [7], temperature scaling [6, 20, 12],
label smoothing [4, 19], mixup [38, 26], CutMix [34] etc. help in calibrating
the DNNs. Moreover, techniques like cutout, mixup, and CutMix, categorised
under data-augmentation techniques, have also shown to make the DNN robust.
However, training deep architectures with these techniques incurs overhead,
while shallow architectures cannot match the accuracy of their deeper
counterparts, even with augmentation. Thus, a method to transfer the knowledge
from deep architectures to a shallow architecture while also improving its
calibration is desirable.

Knowledge distillation (KD) [10] is a technique that enhances the performance
of a student network by incorporating the knowledge from a teacher network.
Generally, the student network is shallow, and the teacher network is a deeper
network pre-trained on a large-scale dataset like ImageNet. KD is used
extensively in real-life scenarios where computational complexity is a
critical factor. The teacher model being a deep architecture is often
expensive to deploy on a local machine, while a shallow student architecture,
trained from scratch, does not deliver the same performance. Thus, KD helps in
increasing the accuracy of the shallow student network. Various modifications
over traditional KD are reported in past to boost the performance of the
student network, for example, Relational KD (RKD) [21], Contrastive
Representation Distillation (CRD) [27], etc. However, improving the accuracy
of the student without paying attention to its confidence makes it
uncalibrated. Stanton _et al_. [24] analyzed the accuracy and fidelity of the
student in the view of KD. Fig. 1 shows the poor calibration of the student
distilled using conventional KD.

Some works [18, 5] show the distillation of a well-calibrated student from an
already-calibrated teacher. However, large models with millions of parameters,
quite common these days, are often unclibrated and calibrating such pre-
trained model is expensive due to the involvement of huge datasets. We,
therefore, ask a question - can a calibrated student be distilled from an
uncalibrated teacher?, and try to find its answer through our approach.

Following are the main contributions and findings of our paper:

  * •

We develop a simple approach to distilled a well-calibrated student network
from an uncalibrated teacher. We use scaled version of KL-divergence loss with
the augmentation loss to improve the calibration of the student.

  * •

We propose a generalised framework that can easily be integrated with a
variety of distillation and augmentation techniques. We perform experiments
using three different augmentation techniques namely mixup, CutMix, cutout and
with different distillation techniques like RKD and CRD.

  * •

We perform experiments on various datasets, including CIFAR-10, CIFAR-100,
CINIC-10, and TinyImageNet and obtain a student which is even better than
teacher in terms of calibration.

  * •

We perform experiments on corrupted (CIFAR-100C) to demonstrate the robustness
of the distilled student network using our approach.

##  2 Related Work

###  2.1 Knowledge Distillation

The first instance of knowledge transfer was reported by Bucilua _et al_. [3],
where a single student is distilled from an ensemble of networks. Ba and
Caruana [1] used the knowledge of a deep neural network to increase the
accuracy of a shallow network. The term “Knowledge Distillation” was coined by
Hinton _et al_. [10] with the idea of minimizing the KL loss between the
softened probabilities from the last layer of the teacher and student model.
Broadly, KD is classified into offline, online [8, 29, 42], and self
distillation [30, 41, 17] based on the pre-trained teacher. Researchers [22,
36] have also explored minimizing the loss between the intermediate layers
rather than the last layer, which finds the similarity between the feature
representation of the teacher and student. Yim _et al_. [33] proposed a method
of distilling relational knowledge from the teacher by using Gram matrix
between the feature maps of first and last layers of the teacher model. Park
_et al_. [21] proposed RKD aiming to transfer knowledge by finding the
relation between different instances. They introduced two new losses,
distance-wise and angle-wise loss, to minimize the structural difference in
relations. Beyer _et al_. [2] introduced the concept of function matching to
improve the transfer of knowledge from teacher to student by making both see
the consistent input and training the students using a long training schedule.
Shen _et al_. [23] used the relabelling technique of Yun _et al_. [35] along
with data augmentation to boost the performance of the student network in an
efficient manner. Li _et al_. [13] and Yang _et al_. [32] distilled the
student using self-knowledge fused with mixup at the feature level, i.e. the
feature maps were mixed using the mixup technique. Malinin _et al_. [15]
proposed an ensemble approach to improve the calibration of the model by using
the prior networks [14] in the distillation framework. There has been a lot of
research in KD [43, 2, 37] to improve the accuracy of the student in different
distillation modes.

(a) Teacher

(b) Student

(c) KD

Figure 1: (a), (b) and (c) show the scatter plots of accuracy vs confidence
for teacher, student and student distilled using conventional KD on CIFAR-100
dataset respectively. A well calibrated model will have most of its density
along the $x=y$ line. We observe that (i) the teacher is not calibrated but
accurate, (ii) the student model trained from scratch is poorly calibrated and
less accurate than the teacher, (iii) by performing distillation, we get a
student that is still not well calibrated.

###  2.2 Knowledge distillation with data augmentation

Wang _et al_. [28] demonstrated the role of data augmentation in KD. They
passed both the original and the augmented data through the teacher network
while training the student network. Zhao _et al_. [44] proposed similarity
transfer for KD. They use mixup technique to generate virtual samples forming
a new dataset, which distills knowledge from the teacher to the student. Zhang
_et al_. [39] proposed an augmentation-based distillation technique that
focuses on improving the accuracy of the student network. They perform cross-
distillation using mixup and CutMix augmentation techniques. Xu _et al_. [31]
proposed a computationally efficient KD by combining uncertainty and mixup
with KD. It was discussed in [45] that a better teacher does not necessarily
make a better student and there may also be a degradation in the accuracy of
the distilled student. Muhammad _et al_. [18] proposed mixup-based robustness
transfer via distillation of activated channel maps (MixACM) that makes the
student robust by distilling knowledge from the adversarially trained robust
teacher using mixup augmentation.

The advantage of using data augmentation and KD in an integrated framework for
calibrating the student network is an open problem. Researchers have mainly
focused on improving the accuracy of the student model while overlooking the
calibration aspect. Our approach addresses this gap by keeping its prime focus
on improving the calibration of the student network without using a robust
teacher. To the best of our knowledge, none of the previous works addresses
the problem of distilling a calibrated student from an uncalibrated teacher in
the image domain. In cases where an improvement in calibration of the student
has been reported, the teacher for such approaches was either adversarially
trained or a certain technique was used to make the teacher robust.
Subsequently, a calibrated teacher resulted in a calibrated student with
standard KD. This differs from our approach as we do not make teachers
calibrated or robust and use it the way it is.

##  3 Proposed Method

In this section, we first revisit the concept of distillation followed by data
augmentation techniques and then present the generalised framework of our
approach. Notation: Let $S$ be the student model and $T$ be the teacher model.
This teacher model is pre-trained on the data $\mathcal{D}$. Let $f_{t}$ and
$f_{s}$ represent the output (logits) of the teacher and student network
respectively. Let $X$ represent the original data and $y$ represent its
corresponding label.

(a) KD

(b) Scaled KD

Figure 2: (a) and (b) show the calibration plots for vanilla KD and scaled KD
with $\alpha=0.5$. Scaled KD has a better calibration than vanilla KD but with
decreased accuracy.

###  3.1 Knowledge distillation

The primary goal of KD is to distill the dark knowledge from the teacher
network into the student network by comparing the temperature-scaled softmax
logits of the teacher and student. The loss term generally consists of two
losses, KL divergence loss ($\mathcal{L}_{div}$) and a task-specific loss
($\mathcal{L}_{task}$).

$\mathcal{L}_{div}=\sum_{x_{i}\in X}\text{KL}[f_{s}(x_{i})||(f_{t}(x_{i})]$
(1) $\mathcal{L}_{task}=\sum_{i=1}^{N}-y_{i}\text{ log }f_{s}(x_{i})$ (2)

where $N$ is the total number of samples in the dataset. KL loss is the
Kullback-Leibler Divergence score which describes the difference between the
probability distribution of soft labels of teacher $T$ and student $S$. In
conventional KD [10], the KL-divergence loss and the task-specific loss are
linearly combined as follows:

$\mathcal{L}_{HKD}=\alpha*\mathcal{L}_{div}+(1-\alpha)*\mathcal{L}_{task}$ (3)

where $\alpha$ is a balancing factor. Generally, the value of $\alpha$ is kept
high to ensure high knowledge transfer from the teacher to the student,
however, doing so makes the student network highly uncalibrated as shown in
Fig. 1. Therefore, we scale the KL-divergence loss by taking mean across the
dimension of the logits (i.e. dividing it by the logit size, which is same as
number of classes). Using this scaled KL-divergence loss we observe a
improvement in the calibration and a drop in accuracy, as shown in Fig. 2. We
encounter a trade-off between accuracy and calibration error in the framework
of KD111For detailed experiments on KD (without scaling) and without using
ground truth, please refer to the supplementary material.. Our approach
leverages the distillation framework to take care of the accuracy while
improving the calibration with the help of data augmentation techniques.

###  3.2 Distillation with Augmentation: Integrated Framework for Calibration

In this section, we discuss a generic framework that makes our approach work
on any distillation technique with any augmentation technique. Let $A_{KD}$ be
a distillation scheme having its loss as $l_{A}$ and $G_{aug}$ be an
augmentation scheme that generates $x_{G}$ samples using the original data
samples $x$ and having its own loss as $l_{G}$. We replace the task-specific
loss with the loss corresponding to the augmentation scheme. If the
augmentation scheme does not have its own loss, we use the standard cross
entropy loss as a task-specific loss. In our framework, we pass the original
samples through both the teacher and student networks and obtain the
logits/features corresponding to the original data as,

$\mathcal{L}_{KD}=l_{A}(f_{t}(x),f_{s}(x))$ (4)

We compute the distillation loss $\mathcal{L}_{KD}$ between the outputs of
teacher and student using the distillation-specific loss function $l_{A}$ as
shown in Eq. 4. Our framework is independent of the nature of this loss, i.e.,
whether it is applied at the feature level or logit level. Simultaneously, we
pass the augmented samples through the student network (only) and apply the
augmentation-specific loss.

$\mathcal{L}_{Aug}=l_{G}(y,z_{s}^{aug})$ (5)

Here $l_{G}$ is the augmentation-specific loss function, $y$ is the true label
corresponding to the augmented sample $x_{G}$ and $z_{s}^{aug}=f_{s}(x_{G})$
is the logit vector corresponding to the augmented input $x_{G}$. As the
teacher is uncalibrated, there is no benefit of passing the augmented input
through it. The overall loss is represented as follows:

$\mathcal{L}=\alpha_{A}*\mathcal{L}_{KD}+(1-\alpha_{A})*\mathcal{L}_{Aug}$ (6)

The balancing factor $\alpha_{A}$ is a hyper-parameter corresponding to the
$A_{KD}$ distillation scheme. This hyper-parameter regulates the trade-off
between the calibration and accuracy of the student model.

###  3.3 Improving calibration w/o accuracy degradation

Calibration implies that the model is generalized and does not give
overconfident predictions. Various data-augmentation schemes like mixup and
CutMix are theoretically proven or empirically shown in the literature to
improve the calibration and accuracy of the standalone networks. In our
framework, the student learns the knowledge from two sources: distillation and
augmentation. Distillation loss targets accuracy, while augmentation-specific
loss targets the calibration of the student model. In conventional KD, higher
weightage is given to the distillation loss, however, to improve the
calibration of the student, the augmentation-specific loss needs more
weightage. As the teacher is uncalibrated, improvement in calibration is more
likely to come from the augmentation.

Figure 3: Illustration of our proposed framework. The original image is passed
through the teacher $f_{t}$ and the student $f_{s}$ network to compute the KL
divergence or distillation loss $\mathcal{L}_{KD}$ between the soft logits of
teacher and student. Since the teacher is uncalibrated, the augmented image
$x_{aug}$ is passed through the student network only. The augmentation-based
loss $\mathcal{L}_{aug}$ is then applied to the logits $z^{aug}_{s}$ and true
labels.

#### Calibrated distillation using mixup

Mixup is a data-augmentation technique developed by [38]. It is a linear
interpolation of the images and labels to generate the new augmented data.
Thulasidasan _et al_. [26] acknowledged the calibration and generalization
property of mixup. Zhang _et al_. [40] theoretically proved the improvement in
calibration metrics when the model is trained using mixup. In addition, mixup
is also shown to reduce the overconfidence error of the model. Consider two
random samples from the dataset, $(x_{i},y_{i})$ and $(x_{j},y_{j})$. Mixup
generates a new data point in the vicinity of the original data distribution
by using these two data points controlled by hyperparameter “$\lambda$”.
Mathematically, mixup is defined as follows:

$\displaystyle x_{mix}=\lambda*x_{i}+(1-\lambda)*x_{j}$ (7) $\displaystyle
y_{mix}=\lambda*y_{j}+(1-\lambda)*y_{j}$ (8)

The empirical Dirac delta distribution centered at $(x_{i},y_{i})$

$P_{\delta}(x,y)=\frac{1}{n}\sum_{i}^{n}\delta(x=x_{i},y=y_{i})$ (9)

is replaced with the empirical vicinal distribution

$P_{v}(x_{mix},y_{mix})=\frac{1}{n}\sum_{i}^{n}\nu(x_{mix},y_{mix}|x_{i},y_{i})$
(10)

where $(x_{mix},y_{mix})$ is the mixup generated data point, and $\nu$ is the
vicinal distribution that gives the probability of the new sample in the
vicinity of the original data distribution. This vicinity distribution gives
the probability $P_{v}$ of finding an augmented sample. Usually $\lambda$ is
sampled from a beta distribution $\beta(a,a)$ where $a\in(0,\infty)$. We will
not use soft label $y_{mix}$ in future operations; therefore, we skip this
computation, instead we use hard labels. The mixup loss is defined as follows:

$\mathcal{L}_{CE}=-[\lambda*y_{i}*log(f_{s}(x_{mix}))+(1-\lambda)*y_{j}*log(f_{s}(x_{mix}))]$
(11)

where $y_{i}$ and $y_{j}$ are the true labels corresponding to the image
$x_{i}$ and $x_{j}$. Since mixup uses two images to generate an augmented
image, we modify the distillation loss as follows:

$\mathcal{L}_{KD}=\frac{1}{n_{c}}(KL[f_{t}(x_{i})||f_{s}(x_{i})]+KL[f_{t}(x_{j})||f_{s}(x_{j})])$
(12)

where $n_{c}$ is the number of classes. The distillation objective function
takes the following form:

$\displaystyle\mathcal{L}=\alpha\mathcal{L}_{KD}+(1-\alpha)\mathcal{L}_{CE}$
(13)

The hyper-parameter $\alpha$ signifies the amount of knowledge to be gained by
the student from the teacher and by itself via training using augmented data.
Our integrated approach using mixup is shown in Fig. 3. Similar to mixup, we
apply our technique to calibrate student models using cutout and CutMix222For
details and implementation please refer to supplementary material.

##  4 Experiments

We present our experiments on the calibration of the distilled classifiers. We
take mean of the KL-divergence loss by dividing it by the size of the logit
vector and refer to this scaled version as conventional KD unless stated
otherwise. We conduct several experiments to observe the student network
calibration using the popular data augmentation techniques such as cutout,
mixup, and CutMix. We use datasets such as CIFAR-10/100, CINIC-10, and
TinyImageNet for image classification experiments and report the calibration
metrics (ECE and OE) corresponding to best validation accuracy. For
understanding the robustness of our approach we use CIFAR-100C dataset333We
perform experiments on OOD data also and include our findings in the
supplementary material..

###  4.1 Calibration Metrics

In the experiments, we use two calibration metrics namely ECE (expected
calibration error) and OE (overconfidence error) defined as follows:

$\textbf{ECE}=\sum^{B}_{b=1}\frac{n_{b}}{N}|acc(b)-conf(b)|$ (14)
$\textbf{OE}=\sum^{B}_{b=1}\frac{n_{b}}{N}[conf(b)\times
max(conf(b)-acc(b),0)]$ (15)

where $n_{b}$ is the number of predictions in bin $b$, $N$ is the number of
testing samples, $B$ is the total number of bins, $acc(b)$ and $conf(b)$ are
the accuracy and average confidence corresponding to bin $b$. We follow
Thulasidasan _et al_. [26] implementation of ECE and OE.

(a) Teacher

(b) Student

(c) KD

(d) KD + cutout

(e) KD + mixup

(f) KD + CutMix

Figure 4: Calibration plot for CIFAR-100 dataset. (a), (b) and (c) are identical to Fig. 1(a), 1(b) and 2(b). They are included here only for completeness. (d), (e) and (f) are calibration plots for KD + cutout, KD + mixup and KD + CutMix respectively. The teacher is ResNet32x4 and the student is ShuffleNetV1. The graph shows a scatterplot of accuracy vs confidence. We observe that KD + mixup and KD + CutMix shows a considerable improvement in the calibration of the student. Teacher/ | Metrics | Teacher | Student | KD | RKD | CRD | KD + | KD + | KD +
---|---|---|---|---|---|---|---|---|---
Student |  |  |  |  |  |  | cutout | mixup | CutMix
WRN-40-2/ ShuffleNetV1 | Acc | 77.09 | 70.58 | 71.05 | 73.41 | 75.28 | 72.46 | 74.65 | 74.94
ECE | 0.103 | 0.121 | 0.108 | 0.126 | 0.106 | 0.090 |  0.049 | 0.046
OE | 0.085 | 0.093 | 0.084 | 0.100 | 0.087 | 0.067 | 0.002 |  0.031
ResNet32x4/ ShuffleNetV2 | Acc | 79.21 | 73.19 | 72.58 | 73.68 | 75.14 | 74.05 | 75.25 | 76.64
ECE | 0.068 | 0.104 | 0.104 | 0.122 | 0.105 | 0.087 | 0.042 |  0.061
OE | 0.052 | 0.083 | 0.082 | 0.098 | 0.087 | 0.066 | 0.003 |  0.044
ResNet50/ MobileNetV2 | Acc | 78.98 | 63.2 | 63.92 | 63.81 | 67.53 | 64.03 | 65.94 | 67.32
ECE | 0.104 | 0.169 | 0.135 | 0.167 | 0.134 | 0.097 |  0.051 | 0.037
OE | 0.086 | 0.129 | 0.101 | 0.128 | 0.103 | 0.068 | 0.001 |  0.019
ResNet50/ VGG8 | Acc | 78.98 | 70.3 | 70.79 | 71.4 | 73.27 | 71.47 | 71.94 | 72.3
ECE | 0.104 | 0.122 | 0.091 | 0.103 | 0.101 | 0.087 |  0.050 | 0.031
OE | 0.086 | 0.096 | 0.070 | 0.077 | 0.080 | 0.063 | 0.002 |  0.011
ResNet32x4/ ShuffleNetV1 | Acc | 79.21 | 70.58 | 71.86 | 71.96 | 73.98 | 73.23 | 74.68 | 74.63
ECE | 0.068 | 0.121 | 0.096 | 0.130 | 0.110 | 0.087 |  0.051 | 0.036
OE | 0.052 | 0.093 | 0.077 | 0.103 | 0.089 | 0.065 | 0.001 |  0.023
VGG13/ MobileNetV2 | Acc | 76.36 | 63.2 | 64.13 | 63.44 | 67.35 | 64.03 | 66.48 | 66.44
ECE | 0.073 | 0.169 | 0.130 | 0.167 | 0.114 | 0.103 |  0.047 | 0.037
OE | 0.057 | 0.129 | 0.097 | 0.127 | 0.086 | 0.071 | 0.002 |  0.022
Table 1: Experiment result on CIFAR-100 dataset. The table shows the result of our framework on six teacher-student pairs. We observe a significant improvement in the calibration of the student for all the teacher-student pairs considered for CIFAR-100 dataset. The best scores are highlighted in bold and second best are underlined. Teacher | VGG13 | ResNet50 | VGG13
---|---|---|---
Student | MobileNetV2 | MobileNetV2 | VGG11
Metrics | Accuracy | ECE | OE | Accuracy | ECE | OE | Accuracy | ECE | OE
Teacher | 93.92 | 0.039 | 0.034 | 95.51 | 0.027 | 0.024 | 93.92 | 0.039 | 0.034
Student | 95.01 | 0.031 | 0.027 | 95.01 | 0.031 | 0.027 | 92.4 | 0.047 | 0.041
KD | 95.19 | 0.034 | 0.031 | 95.26 | 0.035 | 0.031 | 92.84 | 0.050 | 0.044
KD + cutout | 96.25 | 0.025 | 0.022 | 96 | 0.021 |  0.018 | 93.38 | 0.040 | 0.035
KD + mixup | 95.83 |  0.021 | 0.013 | 95.75 | 0.023 | 0.012 | 93.35 |  0.028 | 0.015
KD + CutMix | 96.3 | 0.019 |  0.014 | 96.22 |  0.021 | 0.018 | 93.64 | 0.022 |  0.016
Table 2: Experiments on CIFAR-10 dataset: Our approach improves the
calibration error of the student network as compared to traditional KD. Among
various augmentation techniques, CutMix augmentation integrated using our
approach has distilled a calibrated as well as accurate student.

###  4.2 Setup

For CIFAR-10/100 and CINIC-10 dataset experiments, we use SGD optimizer having
momentum of 0.9, weight decay of $5\times 10^{-4}$ and initial learning rate
of 0.1 (0.01 for MobileNet, ShuffleNet architectures). Similar to [27], we
train networks for 240 epochs with batch size of 64 for CIFAR10/100 and 256
for CINIC-10 datasets. We use CosineAnnealingLR as a learning rate scheduler
with $T_{max}=$240\. For TinyImageNet dataset experiments, we use the same
optimizer except a weight decay of $10^{-4}$. We train the networks for 200
epochs with batch size of 128 and scheduled the learning rate using
MutliStepLR having milestones at 150 and 180 epochs and decay factor of 0.1.
For RKD and CRD experiments, we use author provided hyper-parameters for their
respective loss functions. For all the experiments, conventional KD’s
Temperature, mixup $\beta$ distribution parameter ($a$), and loss balancing
factor ($\alpha$) are fixed as 50, 0.4, and 0.5 respectively unless otherwise
stated. For the experiments involving cutout augmentation technique, the
number of holes is 1 and size of masked region is 16 $\times$ 16\. We use the
standard hyper-parameters for CutMix augmentation technique (number of holes
=1, CutMix probability=0.5). In all the experiments, standard augmentations
like random crop, horizontal flip, and normalization are applied to the
images.

###  4.3 Image Classification

#### CIFAR-100

The CIFAR-100 dataset consists of 60,000 RGB images of size 32 $\times$ 32\.
The dataset has 100 classes where each class consists of 500 train images and
100 test images. In our experiments , we use six (teacher, student) pairs as
shown in Tab. 1. For CIFAR-100 experiments, the teacher and student networks
have different architectures. We refer scaled KD as KD in all our experiments
as scaling helps in calibration (refer Fig. 2). We use KD (scaled), RKD [21]
and CRD [27] as baselines for comparison. Tab. 1 shows that the students
distilled through our integrated framework are better calibrated with
comparable accuracy and reduced overconfidence. Note that unlike the well-
engineered and advance distillation approaches, RKD and CRD, we use
conventional KD while distilling a calibrated student through our approach.
Later in Sec. 4.5, we show that RKD and CRD can also be improved through
proposed framework.

Out of the 3 augmentation techniques considered in our experiments, CutMix and
mixup calibrates the student model better than cutout. CutMix, which takes the
advantage of both cutout and mixup does better than mixup in most cases. Our
approach has reduced the ECE of the student model up to 5 times when compared
with that of the student trained from scratch. In all the teacher-student
configurations, it is observed that the student model has an improved
calibration (ECE) than the corresponding teacher network. Mixup [26] targets
the overconfidence of the model, therefore KD + mixup drastically reduces the
OE of the model which makes it suitable for high risk domains. The calibration
plots for teacher-student pair: ResNet32x4 and ShuffleNetV1 are shown in Fig.
4. We observe that KD and KD + cutout slightly improves the calibration of the
student network while KD + mixup and KD + CutMix shows significant improvement
in the calibration by aligning the confidence of the model with the accuracy.

Teacher | VGG13 | ResNet50 | VGG13
---|---|---|---
Student | MobileNetV2 | MobileNetV2 | VGG11
Metrics | Accuracy | ECE | OE | Accuracy | ECE | OE | Accuracy | ECE | OE
Teacher | 84.68 | 0.105 | 0.095 | 88.2 | 0.085 | 0.078 | 84.68 | 0.105 | 0.095
Student | 86.43 | 0.092 | 0.083 | 86.43 | 0.092 | 0.083 | 81.62 | 0.125 | 0.112
KD | 87.05 | 0.094 | 0.087 | 86.95 | 0.101 | 0.094 | 82.54 | 0.126 | 0.115
KD + cutout | 88.1 | 0.075 | 0.067 | 88.04 | 0.075 | 0.067 | 83.22 | 0.106 | 0.093
KD + mixup | 87.24 | 0.044 |  0.035 | 87.64 |  0.048 |  0.041 | 82.31 |  0.093 |  0.080
KD + CutMix | 87.75 |  0.045 | 0.003 | 87.69 | 0.030 | 0.018 | 83.12 | 0.025 | 0.011
Table 3: Experiments on CINIC-10 dataset. Our approach outperforms the
conventional KD in the calibration metrics by reducing the ECE to
approximately half when compared with the student trained from scratch.

#### CIFAR-10 and CINIC-10

The CIFAR-10 consists of 60,000 RGB images of size 32 $\times$ 32\. The
dataset has 10 classes where each class consists of 5000 train images and 1000
test images. We perform experiments on various architectures to show the
effectiveness of our approach in calibrating the student network. The results
of our experiments are shown in Table 2. Since the dataset is small, we
observe a good ECE for teachers. In the (VGG13, MobileNetV2) combination, we
observe improvement in calibration when the teacher has higher ECE than
student. In other combinations, the teacher network has lower ECE compared to
the student. Our approach outperforms the conventional KD in terms of ECE and
OE. The student network distilled using our approach is well-calibrated and
even more accurate. Among the augmentations considered, KD + CutMix approach
results in comparatively better calibration similar to the previous
experiments. We observe that the KD + mixup, similar to previous experiments,
decreases the overconfidence by a large factor, and KD + cutout approach also
gives a better-calibrated student when compared with the student (trained from
scratch) and KD.

#####

We extend our observation to CINIC-10 dataset which is downsampled from both
CIFAR-10 and ImageNet. It consists of 270,000 colored images of size 32x32.
The dataset contains training, validation and test set with each set having
90,000 images. The results for CINIC-10 are shown in Tab. 3. We observe a
similar trend as in CIFAR-10 with improvement in the calibration of the
student model trained using our framework. For CINIC-10, CutMix approach has
performed comparatively better in terms of calibration.

Metrics | Accuracy | ECE | OE
---|---|---|---
Teacher | 57.28 | 0.140 | 0.100
Student | 55.76 | 0.146 | 0.103
KD | 56.22 | 0.103 | 0.072
KD + cutout | 58.6 | 0.102 | 0.070
KD + mixup | 58.11 |  0.074 | 0.004
KD + CutMix | 60.99 | 0.049 |  0.026
Table 4: Experiments on TinyImageNet Dataset. We report the calibration metrics of the best validation accuracy of the model obtained through training. Metrics | Accuracy | ECE | OE
---|---|---|---
RKD-DA | 72.81 | 0.113 | 0.089
RKD-DA + cutout | 73.7 | 0.102 | 0.080
RKD-DA + mixup | 75.87 | 0.041 |  0.002
RKD-DA + CutMix | 74.81 |  0.046 | 0.001
CRD | 75.03 | 0.083 | 0.064
CRD + cutout | 75.84 | 0.080 | 0.063
CRD + mixup | 76.56 |  0.065 | 0.002
CRD + CutMix | 76.41 | 0.046 |  0.015
Table 5: Experiment results of RKD and CRD distillation techniques on CIFAR-100. | Frost | Snow | Gaussian Blur | Motion Blur
---|---|---|---|---
Metrics | Acc | ECE | OE | Acc | ECE | OE | Acc | ECE | OE | Acc | ECE | OE
Teacher | 51.77 | 0.177 | 0.132 | 57.33 | 0.128 | 0.095 | 54.51 | 0.176 | 0.132 | 57.9 | 0.140 | 0.105
Student | 51.36 | 0.200 | 0.146 | 53.53 | 0.185 | 0.134 | 50.09 | 0.224 | 0.163 | 51.17 | 0.212 | 0.155
KD | 51.03 | 0.188 | 0.139 | 53.56 | 0.166 | 0.121 | 51.33 | 0.183 | 0.132 | 52.99 | 0.175 | 0.128
KD + cutout | 44.91 | 0.204 | 0.141 | 51.33 | 0.166 | 0.115 | 47.88 | 0.198 | 0.135 | 51.17 | 0.181 | 0.128
KD + mixup | 56.44 | 0.049 | 0.018 | 60.24 | 0.037 | 0.007 | 51.38 | 0.076 | 0.036 | 53.64 | 0.049 | 0.022
KD + CutMix | 49.97 |  0.132 |  0.090 | 55.82 |  0.091 |  0.061 | 51.41 |  0.132 |  0.086 | 52.26 |  0.127 |  0.086
Table 6: Experiment results on CIFAR-100C. We evaluated our models trained on
CIFAR-100 dataset on the corrupted data. We observe a better calibration for
the corrupted dataset.

#### TinyImageNet

TinyImageNet is a subset of the ImageNet dataset. It consists 200 class images
of resolution 64x64x3 with a training set consisting of 100,000 images and a
validation set consisting of 10,000 images. In this experiment, we consider
ResNet-34 and ResNet18 as teacher and student respectively. The results on the
validation set are reported in Tab. 4. The teacher and student have high ECE
and OE, therefore are poorly calibrated. We observe an considerable
improvement in the calibration of the student network for all augmentations
considered. KD + mixup approach has reduced the overconfidence error by a
factor of 50 and KD + CutMix reduced the ECE by nearly one-third when compared
with the student trained from scratch. The accuracy of models distilled using
our mixup and cutmix approach has surpassed the accuracy of the teacher which
is an additional benefit. This shows the effectiveness of our framework even
on a TinyImageNet which is considerably a large and complex dataset.

###  4.4 Testing Robustness

#### CIFAR-100C

CIFAR-100C is a corrupted dataset of CIFAR-100 with several noises. It
consists of 15 different corruption schemes, with each corruption having 5
severity levels. Each corruption set consists of 50,000 test images. In this
experiment we consider four corruption schemes, namely frost, snow, gaussian
blur and motion blur. We use ResNet32x4 as teacher and ShuffleNetV1 as student
which are trained on original CIFAR-100 to evaluate the robustness of the
proposed method. The results are averaged over all severity levels and
reported in Tab. 6. In this experiment, we measure the accuracy and
calibration of the model when it is exposed to noisy inputs. We observe that
in case of snow and frost noise, KD + mixup outperforms the performance of
teacher, student and conventional distillation both in terms of accuracy and
calibration. However, the cutout integration does not perform well as the
important regions are blacked out i.e. information is lost when we apply
cutout augmentation technique and the model finds it difficult for high
severity levels thereby decreasing the overall accuracy. KD + cutmix also
shows am improvement in accuracy, ECE and OE when compared with conventional
distillation. This shows that our framework helps in the making the student
robust for corrupted images. The choice of data-augmentation and distillation
technique has an effect over the robustness of distilled student. In our
experiments, we observe that KD + mixup surpasses the accuracy and ECE of
conventional KD.

###  4.5 Extension to other KD techniques

In this section, we explore the integration of other distillation techniques
within our framework on CIFAR-100 dataset. We extend our approach to RKD[21]
and CRD[27]. The teacher-student pair for this framework is (WideResNet-40-2,
ShuffleNetV1). The experiment results are reported in Tab. 5. We observe an
improvement in the calibration metrics of these approaches along with better
accuracy. For RKD and CRD, we replace the distillation loss $\mathcal{L}_{KD}$
in equation by DA (distance-angle) and CRD loss respectively. We have not used
any KL-divergence loss in this experiment. The experiments are performed on
the hyperparameters suggested by the respective authors except balancing
factor which is 0.5 in this experiment. We observe that mixup, CutMix and
cutout integrated distillation shows a better calibration and accuracy.

##  5 Conclusions

In this work, we successfully distilled a well-calibrated student from an
uncalibrated teacher. Although augmentation techniques have been utilized in
past for accuracy improvement while distillation but their benefit on
calibration has remained unexplored. We bring forward this advantage and show
it with the help of different distillation and different augmentation
techniques. Our approach can be easily integrated to any existing or
subsequently developed distillation techniques. We used different teacher-
student combinations for different experiments to show the variability and
generalizability of our proposed method. In future, we can explore a similar
approach for data-free knowledge distillation.

## References

  * [1] Jimmy Ba and Rich Caruana.  Do deep nets really need to be deep?  Advances in neural information processing systems, 27, 2014.
  * [2] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10925–10934, June 2022.
  * [3] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil.  Model compression.  In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541, 2006.
  * [4] Blair Chen, Liu Ziyin, Zihao Wang, and Paul Pu Liang.  An investigation of how label smoothing affects generalization.  arXiv preprint arXiv:2010.12648, 2020.
  * [5] Deepan Das, Haley Massa, Abhimanyu Kulkarni, and Theodoros Rekatsinas.  An empirical analysis of the impact of data augmentation on knowledge distillation.  arXiv preprint arXiv:2006.03810, 2020.
  * [6] Zhipeng Ding, Xu Han, Peirong Liu, and Marc Niethammer.  Local temperature scaling for probability calibration.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6889–6899, 2021.
  * [7] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In International conference on machine learning, pages 1321–1330. PMLR, 2017.
  * [8] Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo.  Online knowledge distillation via collaborative learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11020–11029, 2020.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [10] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7), 2015.
  * [11] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [12] Ananya Kumar, Percy S Liang, and Tengyu Ma.  Verified uncertainty calibration.  Advances in Neural Information Processing Systems, 32, 2019.
  * [13] Xingjian Li, Haoyi Xiong, Chengzhong Xu, and Dejing Dou.  Smile: Self-distilled mixup for efficient transfer learning.  arXiv preprint arXiv:2103.13941, 2021.
  * [14] Andrey Malinin and Mark Gales.  Predictive uncertainty estimation via prior networks.  Advances in neural information processing systems, 31, 2018.
  * [15] Andrey Malinin, Bruno Mlodozeniec, and Mark Gales.  Ensemble distribution distillation.  arXiv preprint arXiv:1905.00076, 2019.
  * [16] Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic.  Revisiting the calibration of modern neural networks.  Advances in Neural Information Processing Systems, 34:15682–15694, 2021.
  * [17] Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett.  Self-distillation amplifies regularization in hilbert space.  Advances in Neural Information Processing Systems, 33:3351–3361, 2020.
  * [18] Awais Muhammad, Fengwei Zhou, Chuanlong Xie, Jiawei Li, Sung-Ho Bae, and Zhenguo Li.  Mixacm: Mixup-based robustness transfer via distillation of activated channel maps.  Advances in Neural Information Processing Systems, 34, 2021.
  * [19] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  Advances in neural information processing systems, 32, 2019.
  * [20] Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran.  Measuring calibration in deep learning.  In CVPR Workshops, volume 2, 2019.
  * [21] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019.
  * [22] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550, 2014.
  * [23] Zhiqiang Shen and Eric Xing.  A fast knowledge distillation framework for visual recognition.  arXiv preprint arXiv:2112.01528, 2021.
  * [24] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson.  Does knowledge distillation really work?  Advances in Neural Information Processing Systems, 34:6906–6919, 2021.
  * [25] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [26] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.  On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  Advances in Neural Information Processing Systems, 32, 2019.
  * [27] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation.  In International Conference on Learning Representations, 2020.
  * [28] Huan Wang, Suhas Lohit, Michael Jeffrey Jones, and Yun Fu.  What makes a ”good” data augmentation in knowledge distillation - a statistical perspective.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [29] Guile Wu and Shaogang Gong.  Peer collaborative learning for online knowledge distillation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10302–10310, 2021.
  * [30] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy.  Knowledge distillation meets self-supervision.  In European Conference on Computer Vision (ECCV), 2020.
  * [31] Guodong Xu, Ziwei Liu, and Chen Change Loy.  Computation-efficient knowledge distillation via uncertainty-aware mixup.  arXiv preprint arXiv:2012.09413, 2020.
  * [32] Chuanguang Yang, Zhulin An, Helong Zhou, Linhang Cai, Xiang Zhi, Jiwen Wu, Yongjun Xu, and Qian Zhang.  Mixskd: Self-knowledge distillation from mixup for image recognition.  arXiv preprint arXiv:2208.05768, 2022.
  * [33] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4133–4141, 2017.
  * [34] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019.
  * [35] Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun.  Re-labeling imagenet: from single to multi-labels, from global to localized labels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2340–2350, 2021.
  * [36] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  arXiv preprint arXiv:1612.03928, 2016.
  * [37] Hailin Zhang, Defang Chen, and Can Wang.  Confidence-aware multi-teacher knowledge distillation.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4498–4502. IEEE, 2022.
  * [38] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In International Conference on Learning Representations, 2018.
  * [39] Ke Zhang, Jin Fan, Shaoli Huang, Yongliang Qiao, Xiaofeng Yu, and Feiwei Qin.  Cekd: Cross ensemble knowledge distillation for augmented fine-grained data.  Applied Intelligence, pages 1–11, 2022.
  * [40] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou.  When and how mixup improves calibration.  In International Conference on Machine Learning, pages 26135–26160. PMLR, 2022.
  * [41] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.  Be your own teacher: Improve the performance of convolutional neural networks via self distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3713–3722, 2019.
  * [42] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu.  Deep mutual learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4320–4328, 2018.
  * [43] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang.  Decoupled knowledge distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11953–11962, 2022.
  * [44] Haoran Zhao, Kun Gong, Xin Sun, Junyu Dong, and Hui Yu.  Similarity transfer for knowledge distillation.  arXiv preprint arXiv:2103.10047, 2021.
  * [45] Yichen Zhu and Yi Wang.  Student customized knowledge distillation: Bridging the gap between student and teacher.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5037–5046, 2021.