# Dust Temperature of Compact Star-forming Galaxies at $z\sim 1-3$ in
3D-HST/CANDELS

Zuyi Chen  School of Mathematics and Physics, Anqing Normal University, Anqing
246011, People’s Republic of China  CAS Key Laboratory for Research in
Galaxies and Cosmology, Department of Astronomy, University of Science and
Technology of China, Hefei 230026, People’s Republic of China  School of
Astronomy and Space Science, University of Science and Technology of China,
Hefei 230026, People’s Republic of China  Guanwen Fang  Guanwen Fang and Zuyi
Chen contributed equally to this work.  School of Mathematics and Physics,
Anqing Normal University, Anqing 246011, People’s Republic of China  Zesen Lin
CAS Key Laboratory for Research in Galaxies and Cosmology, Department of
Astronomy, University of Science and Technology of China, Hefei 230026,
People’s Republic of China  School of Astronomy and Space Science, University
of Science and Technology of China, Hefei 230026, People’s Republic of China
Hongxin Zhang  CAS Key Laboratory for Research in Galaxies and Cosmology,
Department of Astronomy, University of Science and Technology of China, Hefei
230026, People’s Republic of China  School of Astronomy and Space Science,
University of Science and Technology of China, Hefei 230026, People’s Republic
of China  Guangwen Chen  CAS Key Laboratory for Research in Galaxies and
Cosmology, Department of Astronomy, University of Science and Technology of
China, Hefei 230026, People’s Republic of China  School of Astronomy and Space
Science, University of Science and Technology of China, Hefei 230026, People’s
Republic of China  Xu Kong  CAS Key Laboratory for Research in Galaxies and
Cosmology, Department of Astronomy, University of Science and Technology of
China, Hefei 230026, People’s Republic of China  School of Astronomy and Space
Science, University of Science and Technology of China, Hefei 230026, People’s
Republic of China

(Received *** **, ****; Revised *** **, ****; Accepted *** **, ****)

###### Abstract

Recent simulation studies suggest that the compaction of star-forming galaxies
(SFGs) at high redshift might be a critical process, during which the central
bulge is being rapidly built, followed by quenching of the star formation. To
explore dust properties of SFGs with compact morphology, we investigate the
dependence of dust temperature, $T_{\rm{dust}}$, on their size and star
formation activity, using a sample of massive SFGs with
$\log(M_{\ast}/M_{\odot})>10$ at $1<z<3$, drawn from the 3D-HST/CANDELS
database in combination with deep Herschel observations. $T_{\rm{dust}}$ is
derived via fitting the mid-to-far-infrared photometry with a mid-infrared
power law and a far-infrared modified blackbody. We find that both extended
and compact SFGs generally follow a similar $T_{\rm{dust}}-z$ evolutionary
track as that of the main-sequence galaxies. The compact SFGs seem to share
similar dust temperature with extended SFGs. Despite the frequent occurrence
of AGNs in compact SFGs, we do not observe any effect on dust caused by the
presence of AGN in these galaxies during the compaction. Our results disfavor
different ISM properties between compact and extended SFGs, suggesting that a
rapid and violet compaction process might be not necessary for the formation
of compact SFGs.

Galaxy evolution (594), High-redshift galaxies (734), Galaxy structure (622),
Star formation (1569)

††journal: ApJ

##  1 Introduction

The presence of massive quiescent galaxies (QGs) and star-forming galaxies
(SFGs) with extremely small sizes in both observations and simulations (so-
called red and blue “nuggets”, respectively; van Dokkum et al. 2008; Damjanov
et al. 2009; Barro et al. 2013; Dekel & Burkert 2014; Zolotov et al. 2015;
Tacchella et al. 2016a, b) has spurred extensive discussion about a possible
“compaction and quenching” evolutionary scenario. During the compaction
process, galaxies would acquire significant amount of central stellar mass in
short timescale, followed by the cessation of star formation. Evidences of
such a two-step quenching mode have been found in local and distant galaxies
(Fang et al., 2013; Barro et al., 2013; Fang et al., 2015; Barro et al., 2017;
Wang et al., 2018; Woo & Ellison, 2018). Possible mechanisms accounting for
the compaction include mergers or/and violent disk instabilities, inducing gas
inflows and resulting in active star formation in the central region (Hopkins
et al., 2008; Dekel & Burkert, 2014; Zolotov et al., 2015; Tacchella et al.,
2016b). When the compact center of a galaxy is being rapidly built, the galaxy
enters a starburst phase associated with short gas depletion timescale
($230^{+90}_{-120}$ Myr; Barro et al. 2016) as well as the triggering of
active galactic nuclei (AGNs)/stellar feedback that initialize the quenching
process of the galaxy (Cai et al., 2013; Zolotov et al., 2015). Therefore, in
massive SFGs where gas replenishment is usually suppressed (Tacchella et al.,
2016b), the compaction will end up with the formation of quiescent galaxies
(Tacchella et al., 2016b; Barro et al., 2017).

Nevertheless, it remains to be explored which mechanisms drive the compaction.
Recent studies reveal a dust-obscured nature for the compact star-forming
galaxies (cSFGs; e.g., Barro et al., 2014), which would be the first step of
the proposed “compaction and quenching” scenario. These galaxies appear to be
dusty according to their locations on the popular UVJ diagram with many of
them being detected by Herschel. Furthermore, a much larger AGN fraction is
found in cSFGs than in extended star-forming galaxies (eSFGs) and QGs along
the “compaction and quenching” sequence, suggesting that AGNs might play a
role during the compaction and quenching process (Barro et al., 2014; Kocevski
et al., 2017). But how AGNs help to shape the physical properties and fueling
process of their hosting cSFGs still remain to be fully explored (Kocevski et
al., 2017).

The infrared (IR) emission of active SFGs primarily traces the dust-obscured
star formation. Dust properties (e.g., the dust mass and temperature), as
usually constrained by a modified blackbody fitting to the FIR spectral energy
distribution (SED), are linked to the physical conditions within which star
formation occurs. Thus, dust in the interstellar medium (ISM) provides a
unique perspective to study properties of host galaxies such as gas content
and the star formation efficiency (Magnelli et al., 2014; Schreiber et al.,
2016, 2018; Ma et al., 2019; Liang et al., 2019). Besides star formation, AGN
activity may also contribute to dust heating. A harder radiation field may
lead to a more efficient dust heating and thus a higher dust temperature. This
is particularly relevant for cSFGs which have a frequent occurrence of AGNs
(Barro et al., 2014; Kocevski et al., 2017). Therefore, the AGN effect may be
uncovered through the analysis of dust content.

By investigating dust properties of cSFGs and eSFGs as well as the potential
effect of AGNs on those, the work presented here is an attempt to gain new
insights into the physical mechanisms involved in the compaction process.
Based on the wealth of multi-wavelength photometry collected from the HST,
Spitzer, and Herschel observations in the Cosmic Assembly Near-infrared Deep
Extragalactic Legacy Survey fields (CANDELS; Grogin et al. 2011; Koekemoer et
al. 2011), we determine the dust properties of a large sample of massive SFGs
at $1<z<3$ and constrain the nature of cSFGs. We present the galaxy sample and
their multi-wavelength data in Section 2, while their $T_{\rm dust}$ and star
formation rate (SFR) are derived in Section 3. Our results, including a
comparative analysis of the dust properties between extended and compact
galaxies, are presented in Section 4, followed by a summary in Section 5.

Throughout the paper, a Chabrier (2003) initial mass function (IMF) and a
standard $\Lambda$CDM cosmology with $\Omega_{\rm m}=0.3,\
\Omega_{\rm{\Lambda}}=0.7$ and $H_{\rm{0}}=70\
\rm{km~{}s}^{-1}~{}\rm{Mpc}^{-1}$ are adopted.

##  2 Data and Sample

Four of the CANDELS fields are selected to conduct this work, where both
robust rest-frame optical size measurements and deep mid-IR (MIR) to far-IR
(FIR) photometries are available, i.e., the Great Observatories Origins Survey
(GOODS) Northern and Southern fields (GOODS-N and GOODS-S), the Cosmic
Evolution Survey field (COSMOS), and the All-wavelength Extended Groth Strip
International Survey fields (AEGIS). These valuable sizes measurements and
infrared observations ensure a census of dust temperature variation during the
compaction process of SFGs at the peak of the cosmic star formation history.

###  2.1 3D-HST Survey

Our sample is primarily based on data products from the 3D-HST spectroscopic
survey (Brammer et al., 2012) in combination with the CANDELS program (Grogin
et al., 2011; Koekemoer et al., 2011).

We use the astrometry and photometry from the 3D-HST Release v4.1 (Skelton et
al., 2014) but adopt the updated physical parameters from the Data Release
v4.1.5 111http://3dhst.research.yale.edu/Data.html (Momcheva et al., 2016) to
select massive galaxies with $\log(M_{\ast}/M_{\odot})>10$ at $1<z<3$. Only
those with use_phot = 1 in the 3D-HST catalogs are considered, i.e., galaxies
with reliable photometric measurements. We adopt the “best” galaxy redshift,
$z_{\rm{best}}$, created by Momcheva et al. (2016). This $z_{\rm{best}}$ is
determined by combing the grism redshift, if robust, with the spectroscopic
and photometric redshifts compiled by Skelton et al. (2014). Using
$z_{\rm{best}}$, Momcheva et al. (2016) re-derive other auxiliary stellar
population parameters, rest-frame colors, and SFRs from spectral energy
modeling with FAST (Kriek et al., 2009).

Furthermore, the circularized radius, $r_{\rm{e}}$, is derived using
$r_{\rm{e}}=R_{\rm{e}}\times\sqrt{q}$, where $R_{\rm{e}}$ is the effective
radius along the major axis and $q$ the axis ratio, as measured by van der Wel
et al. (2014) from CANDELS HST/WFC3 imaging mosaics with GALFIT (Peng et al.,
2002). We correct them to the rest-frame optical size at 5000 Å using Equation
(2) in van der Wel et al. (2014) to achieve a uniform measurement. We exclude
galaxies with inaccurate or no size measurement, i.e., with Flag = 2 or 3 in
the catalog of van der Wel et al. (2014).

In total, $6303$ galaxies satisfy our mass and redshift cuts as well as
use_phot = 1, $5700$ of which have robust size measurement and would be
further considered in the next section.

###  2.2 MIR to FIR Photometry and X-ray Data

The four CANDELS fields used here are all covered by the PACS Evolutionary
Probe (PEP222http://www.mpe.mpg.de/ir/Research/PEP/DR1; Lutz et al. 2011;
Elbaz et al. 2011; Magnelli et al. 2013) survey at 100, and 160 $\mu$m (and 70
$\mu$m in GOODS-S) and by the Herschel Multi-tiered Extragalactic Survey
(HerMES333http://hedam.lam.fr/HerMES; Roseboom et al. 2010, 2012; Oliver et
al. 2012) at 250, 350, and 500 $\mu$m. Our MIR to FIR photometry is retrieved
from the PEP and HerMES catalogs, which were extracted at the positions of the
Spitzer/MIPS 24 $\mu$m sources.

To identify AGNs in our sample, we use X-ray observations from the Chandra
X-ray Observatory (Chandra; Weisskopf et al. 2000). The X-ray source catalogs
for 2 Ms exposure of the Chandra Deep Field-North (Xue et al., 2016) and 7 Ms
exposure of the Chandra Deep Field-South (Luo et al., 2017) are adopted for
the GOODS-N and GOODS-S fields, respectively. In these two catalogs, AGNs are
already identified via several X-ray-based criteria (Xue et al., 2016; Luo et
al., 2017). For the COSMOS and AEGIS fields, we use the X-ray source catalogs
from the Chandra COSMOS Legacy survey (Civano et al., 2016; Marchesi et al.,
2016) and the AEGIS-X Deep survey (Brightman et al., 2014; Nandra et al.,
2015), respectively. AGNs in these two fields are identified by either an
intrinsic X-ray luminosity of $L_{\rm{0.5-7keV}}\geq 3\times 10^{42}\ \rm{erg\
s^{-1}}$ or an intrinsic photon index of $\Gamma\leq 1.0$ (obscured AGNs),
which are consistent with the criteria adopted by Xue et al. (2016) and Luo et
al. (2017) in the GOODS-N and GOODS-S fields, respectively. Because
$L_{\rm{0.5-7keV}}$ is not given in the catalogs of the COSMOS and AEGIS
fields, we converse the provided $L_{\rm{2-10keV}}$ to $L_{\rm{0.5-7keV}}$ via

$L_{\rm{0.5-7keV}}=L_{\rm{2-10keV}}/0.72,$ (1)

assuming an intrinsic photon index of $\Gamma=1.8$.

For each field, we use the PEP catalog as the reference one, and subsequently
cross-match it with the 3D-HST catalog, the HerMES catalog, and the X-ray
source catalog utilizing the method described below. Briefly, for every source
in the reference catalogue (i.e., the PEP catalogue), we search for the
nearest counterpart within an optimised matching radius. Here, we consider a
range of matching radii from 0$\farcs$1 to 5″ with a step of 0$\farcs$1, among
which the optimised one is set to the largest radius satisfying the following
two criteria: (1) the fake matched fraction, defined as the ratio of the fake
matched number (see below) to the successful matched number, is smaller than
5%; (2) the differential successful matched number, i.e., the new successful
matched number with increasing matching radius, is larger than the
differential fake matched number. To estimate the fake matched number given
two catalogs, we shift one of them by 30″ in random directions 16 times and
take the average of the 16 “successful” matched numbers as the fake matched
number. Clearly, the optimised matching radius depends on the number densities
of the matched catalogs, and thus varies between catalogs even for the same
field. However, we note that most of the resulting optimised matching radii
are smaller than 1$\farcs$5.

Since both the 3D-HST catalogs and the X-ray source catalogs provide redshift
information, we require that $z_{\rm{best}}$ from the 3D-HST catalogs and
$z_{\rm{X}}$ (redshifts of optical counterparts of X-ray sources) from the
X-ray source catalogs should be in good agreement, i.e., $\mid
z_{\rm{best}}-z_{\rm{X}}\mid/z_{\rm{best}}\leq 0.2$ (only one source was
excluded by this criterion). In total, the cross-match yields a sample of 2403
galaxies with at least one Herschel measurement and a robust HST/size
measurement.

Furthermore, to ensure an accurate measurement of dust temperature, we remove
sources with less than three detections at rest-frame wavelength longer than
50 $\mu$m. After these selections, we obtain a MIR-to-FIR catalog including
521 galaxies, 109 of which have AGN counterparts.

###  2.3 Sample Selection

Figure 1: The UVJ diagram (left panel) and mass-size (right panel)
distributions of our FIR-detected massive SFGs at $1<z<3$ with
$\log(M_{\ast}/M_{\odot})>10$ and with an accurate measurement of their dust
temperature. The Williams et al. (2009) criteria (black broken line) adopted
to select SFGs are shown on the UVJ diagram, together with the direction and
amplitude of extinction (vector) implied by the Calzetti et al. (2000)
extinction law. The SFGs are further divided into extended SFGs (blue points)
and compact SFGs (red points), according to the compaction criteria of
$\log(M_{\ast}/r_{{\rm{e}}}^{1.5}/[M_{\odot}\ \rm{kpc^{-1.5}}])=10.45$ (gray
solid line) and 10.3 (gray dashed line) for galaxies at $1<z\leq 2$ and
$2<z<3$, respectively. Black circles indicate X-ray-identified AGNs.

We use the UVJ criteria of Williams et al. (2009) to select SFGs amongst our
catalog of 521 galaxies, resulting in 488 FIR detected massive SFGs with
$\log(M_{\ast}/M_{\odot})>10$ at $1<z<3$. To ensure a robust analysis of dust
temperature, sources with catastrophic MIR-to-FIR SED fitting (see Section
3.1) are also removed from our final sample. We end up with a sample of 459
SFGs within which 95 galaxies are identified as X-ray AGN hosts. Note that
approximately $64\%$ of these galaxies have spectroscopic or grism redshift,
which is key for accurate dust temperature measurements.

Finally, to identify cSFG from eSFGS, we use the compactness criteria defined
by Barro et al. (2013), i.e.,
$\Sigma_{\rm{1.5}}=\log(M_{\ast}r_{{\rm{e}}}^{-1.5}/[M_{\odot}\
\rm{kpc}^{\rm{{-1.5}}}])$. Since the mass-size relation of SFGs evolves with
redshift (van der Wel et al., 2014), cSFGs are identified as
$\Sigma_{\rm{1.5}}\geq 10.3$ at $1<z\leq 2$ (Barro et al., 2013) and
$\Sigma_{\rm{1.5}}\geq 10.45$ at $2<z<3$ (Barro et al., 2014), respectively.
In total, there are 61 cSFGs.

Figure 1 shows the distributions on the $UVJ$ and the mass-size diagrams for
our final sample. We observe a higher occurrence of X-ray AGNs in cSFG than in
eSFGs, i.e., 37% versus 18%, respectively. This finding is consistent with the
ubiquitous AGNs in cSFGs reported by Barro et al. (2016) and Kocevski et al.
(2017). More than 80% of our SFGs have $V-J$ colors redder than 1.2, a
criterion used for selecting dusty SFGs in Spitler et al. (2014). Such red
$V-J$ colors of our sample indicate that they are dust-obscured, which agrees
with their Herschel detections.

##  3 Dust Temperature and Star Formation rate

###  3.1 Dust Temperature

Following Casey (2012) and using the corresponding IDL code cmcirsed, we model
the MIR to FIR (i.e., 24 to $500\ \mu\rm{m}$) SED of our SFGs, with a MIR
power law plus a FIR greybody with a single dust temperature, $T_{\rm dust}$.
The power-law component accounts for the MIR emission of small clumps
associated with hot dust or for AGN radiation, while the greybody accounts for
the cold dust component emission (Casey, 2012; Casey et al., 2014). For
galaxies lacking sufficient FIR detection, the peak of the greybody emission
will be poorly constrained, resulting in an unreliable estimate on
$T_{\rm{dust}}$. To reduce such effect, only galaxies with more than three
data points at $\lambda_{\rm rest}>50\ \mu$m (i.e., where the greybody
emission dominates) are included in our analysis (see Section 2.2). This extra
selection criteria effectively translates into having for most of our galaxies
more than four bands covering their MIR to FIR emission (76$\%$ have $\geq 5$
bands), providing us with a very accurate description of their infrared
emission.

We fix the dust emissivity to a standard reference value of 1.5 (Chapman et
al., 2005; Casey et al., 2009), leave the power-law slope ($0.5<\alpha<5.5$)
as a free parameter (Casey, 2012). We first perform an automatic fitting
process assuming an optically thin dust model. Then we visually inspect the
best-fit SEDs for individual sources in order to ensure acceptable best-fit
results. For a few cases that the model fails to fit the data, we rerun the
fit by manually adjusting (and fixing) $\alpha$ values. For 25 galaxies, we
are able to better constrain $T_{\rm dust}$ with manual fitting, and the new
best-fit results are adopted. However, there are still 29 of the original 488
FIR detected massive SFGs that fail to achieve an acceptable fit even though
manual fitting process is implemented, due to either insufficient wavelength
coverage that leads the peak emission unconstrained, or the fact that the data
apparently deviate a lot from the assumption of MIR power law plus FIR
greybody. These galaxies with failed determination of $T_{\rm dust}$ are
already excluded from our final sample (see Section 2.3).

While dust temperature can be defined in different ways, such as mass-weighted
or light-weighted, the light-weighted dust temperature derived as here using
the FIR peak emission is what can be best constrained observationally (Casey
et al., 2014; Liang et al., 2019). These light-weighted dust temperatures
reflect the condition of the ISM surrounding massive and thus luminous star-
forming regions. They are also less dependent on the assumed underlying
templates and allow for a direct comparison with previous works.

###  3.2 Star Formation Rate

Figure 2:  SFR–$M_{\ast}$ or main-sequence relations for our extended SFGs
(blue points) and compact SFGs (red points) at $1<z\leq 2$ (left panel) and
$2<z<3$ (right panel), compared to the main-sequence relations of SFGs from
Whitaker et al. (2014, thick gray dashed lines plus thin ones for typical 0.3
dex scatter). The SFGs associated with X-ray-identified AGNs are marked with
black circles. Compared with main-sequence galaxies, a significant number of
our Herschel-detected compact and extended SFGs show elevated star formation
activity, especially in the higher redshift bin.

The SFRs in the 3D-HST catalogs are derived using rest-frame UV luminosity,
$L_{\rm{UV}}$, in combination with the Spitzer/MIPS 24 $\mu$m based total IR
luminosities, $L_{\rm{IR}}$, as described in Whitaker et al. (2014):

$\displaystyle\rm{SFR}$ $\displaystyle=$
$\displaystyle\rm{SFR}_{\rm{UV}}+\rm{SFR}_{\rm{IR}},$ (2) $\displaystyle=$
$\displaystyle 1.09\times 10^{-10}(2.2L_{\rm{UV}}+L_{\rm{IR}}).$

We only keep the UV-based SFRs but recalculate $\rm{SFR}_{\rm{IR}}$ based on
our MIR to FIR photometries. We make use of the total IR luminosity,
$L_{\rm{IR}}$, integrating the SED output by cmcirsed from the rest-frame 8 to
1000 $\mu$m, and correct AGN contribution, $L_{\rm{IR,AGN}}$, with the method
presented in Dai et al. (2018). Briefly, we convert the 2–10 keV luminosity of
our X-ray AGNs to AGN emission at 6 $\mu$m using the well calibrated X-ray to
MIR relation of Stern (2015)

$\log(L_{2-10\rm{keV}}/[\mathrm{erg\ s^{-1}}])=40.981+1.024x-0.047x^{2},$ (3)

where $x=\log(\nu L_{\nu}(6\ \mu\rm{m})/[10^{41}\ \rm{erg\ s^{-1}]})$.
$L_{\rm{IR,AGN}}$ is then extrapolated by multiplying $\nu L_{\nu}(6\ \mu m)$
by a factor of 2.5, derived from the AGN template of Dai et al. (2012). We
find generally small corrections for AGN contribution in our sample with a
median value of $L_{\rm{IR,AGN}}/L_{\rm{IR}}\simeq 9\%$. Note that for
galaxies without X-ray AGN detection, the median of $L_{\rm{IR}}$ based on our
FIR SED fitting is only larger by 0.07 dex than, and therefore broadly
consistent with, that derived by Whitaker et al. (2014).

The SFR–$M_{*}$ distributions for our SFGs are shown in Figure 2 along with
the main-sequence of SFGs from Whitaker et al. (2014). Compared to the main-
sequence of SFGs, a significant number of our SFGs have enhanced star
formation activity, especially in the high redshift bin. By requiring FIR
detection, our SFGs are biased to higher SFR.

##  4 Results

###  4.1 Dust Temperature for Extended and Compact Galaxies

Figure 3:  Top: normalized number distribution (i.e., the total area under the
histogram is 1) of compact SFGs (red line) and extended ones (blue line),
together with the fraction of compact SFGs at different redshift (black dashed
line). Bottom: evolution of the dust temperature, $T_{\rm dust}$, with
redshift for our extended SFGs (blue points) and compact SFGs (red points),
compared to the overall redshift evolution of $T_{\rm{dust}}$ for the main-
sequence galaxies (Magnelli et al. 2014; thick magenta dashed line). The SFGs
associated with X-ray-detected AGNs are marked with black circles. The typical
error of $\sim 5.5$ K for our sample is illustrated by the green error bar.
Generally, the evolution of $T_{\rm{dust}}$ with redshift for both our
extended and compact SFGs is similar to that of the main-sequence galaxies.

Dust temperature of SFGs is known to evolve with redshift in the sense that
typical SFGs exhibit hotter dust temperature at high redshift than in the
local universe, possibly due to a higher star formation efficiency or lower
metallicity (Magnelli et al., 2014; Schreiber et al., 2018; Liang et al.,
2019). Such an evolution cannot be neglected especially for our sample
spanning a large redshift range. Therefore, Figure 3 shows the dust
temperature versus redshift for both eSFGs and cSFGs compared to that of main-
sequence galaxies inferred by Magnelli et al. (2014). The median dust
temperatures of eSFGs and cSFGs are also summarized in Table 1. We found that
the number of cSFGs relatively to the number of eSFGs significantly decreases
between $z\sim 2.5$ and $z\sim 1.5$ (upper panel of Figure 3). This is
consistent with Barro et al. (2013) in which the decrease is explained by the
fact that cSFGs are continuously formed at redshift 2–3 and gradually fade
into QGs at later times.

Table 1:  The source number, dust temperature, and dust mass for eSFGs, cSFGs, and SFGs hosting AGNs in two redshift bins. Sample | $N$ |  $T_{\rm dust}$/K | $\log(M_{\rm dust}/M_{\odot})$
---|---|---|---
$1<z\leq 2$ |  |  |
eSFGs | 337 | 31.6 $\pm$ 0.5 | 8.1 $\pm$ 0.1
cSFGs | 34 | 31.8 $\pm$ 1.8 | 8.4 $\pm$ 0.1
AGNs | 67 | 33.3 $\pm$ 1.2 | 8.3 $\pm$ 0.1
$2<z<3$ |  |  |
eSFGs | 62 | 38.3 $\pm$ 1.9 | 8.5 $\pm$ 0.1
cSFGs | 26 | 38.8 $\pm$ 2.9 | 8.7 $\pm$ 0.1
AGNs | 28 | 38.7 $\pm$ 2.4 | 8.7 $\pm$ 0.1

It is clear that both eSFGs and cSFGs generally follow the $T_{\rm{dust}}-z$
evolutionary track defined by the main-sequence SFGs presented in Magnelli et
al. (2014), especially for galaxies at $z\lesssim 2$. From $z\sim 2.5$ to
$\sim 1.5$, the dust temperature of cSFGs drops by $\mathbf{7.0\pm 3.4}$ K. In
the same redshift range, the dust temperature of eSFGs drops by
$\mathbf{6.7\pm 2.0}$ K. The redshift evolution of dust temperatures for cSFGs
and eSFGs are thus fully consistent within the uncertainties.

In the higher redshift bin, $T_{\rm{dust}}$ of both cSFGs and eSFGs are about
5 K higher than that inferred by Magnelli et al. (2014). This trend can,
however, be explained by their higher SFRs as revealed in Figure 2.

Similarly, $T_{\rm{dust}}$ for AGN-hosting SFGs are also uniformly distributed
around the $T_{\rm{dust}}$–$z$ track of main-sequence SFGs (see the open
circle in Figure 3 and Table 1). No AGN effect on the temperature of the cold
dust is observed for our sample. However, further interpretation is limited by
both the small sample size and the large uncertainty involved during the
$T_{\rm dust}$ calculation.

Based on our FIR SED fits, we also infer dust masses ($M_{\rm{dust}}$) by
extrapolating the rest-frame 850 $\mu$m emission of our galaxies and assuming
a dust absorption coefficient of $\kappa_{850}=0.15\ \rm{m^{2}\ kg^{-1}}$
(Casey, 2012). The resulting $M_{\rm{dust}}$ is given in Table 1. It can be
seen that a huge amount of dust is enclosed in our SFGs, which is expected
from their FIR detections. Though cSFGs might have more dust compared to
eSFGs, it should be treated with caution since the reliability of
$M_{\rm{dust}}$ measurement is limited by the lack of data at longer
wavelengths. Our measurement is consistent with those of high-redshift dusty
SFGs or sub-millimeter galaxies with $T_{\rm{dust}}\approx 20-50$ K and
$\log(M_{\rm{dust}}/M_{\odot})\approx 8-9.3$ (Kovács et al., 2006; Casey,
2012; Casey et al., 2014). Furthermore, the huge amount of dust in cSFGs also
directly confirms the observations of Barro et al. (2014) that SFGs become
more compact before they lose their gas and dust.

Figure 4:  Relation between $T_{\rm{dust}}$ and $\Sigma_{\rm{1.5}}$ for our
SFGs, color-coded by their sSFR, at $1<z\leq 2$ (left panel) and $2<z<3$
(right panel). In each panel, the green solid line is the running medians of
$T_{\rm{dust}}$ for each subsample, while the green hatched shadow indicates
the $1\sigma$ uncertainties of the running medians estimated via the
bootstrapping method. The yellow shadows show the running standard deviations
of $T_{\rm{dust}}$ for the corresponding subsamples. The AGN hosts are marked
by the black circles. The gray dashed lines indicate the cSFG criterion, i.e.,
$\Sigma_{1.5}=10.3$ and 10.5 for $1<z\leq 2$ and $2<z<3$, respectively.

To reveal the role of compactness in dust heating, we plot the
$\Sigma_{1.5}$–$T_{\rm dust}$ relation, color-coded by specific star formation
rates (${\rm sSFR}\equiv{\rm SFR}/M_{\ast}$), in Figure 4. Obviously, there is
no significant trend between $\Sigma_{1.5}$ and $T_{\rm dust}$ in both
redshift bins, while the Pearson linear (or Spearman rank) correlation
coefficients are close to zero for both subsamples. This result suggests a
similar dust temperature for cSFGs and eSFGs, which is consistent with Gómez-
Guijarro et al. (2019) in which the authors found that the ISM properties of
cSFGs are similar to those of eSFGs located slightly above the main sequence.
Moreover, no significant feature of sSFR on the $\Sigma_{1.5}$–$T_{\rm dust}$
plane is observed in both redshift bins, although we will show in Section 4.2
that cSFGs tends to have smaller sSFR compared to eSFGs at $2<z<3$.

###  4.2 Connection with Star Formation

In this section, we focus on the effect of star formation activities on dust,
with both the specific star formation rates and the star formation rate
surface densities ($\Sigma_{{\rm SFR}}={{\rm SFR}/2\pi
r_{\rm{e}}^{2}}$)444Note that $\Sigma_{\rm SFR}$ used in this work is defined
by rest-frame optical size $r_{\rm e}$, the star formation-related size,
however, is found to be smaller than the optical stellar size for SFGs at this
redshift (e.g., Elbaz et al. 2018; Fujimoto et al. 2018; Puglisi et al. 2019;
Chen et al. 2020). Given the lack of the measurement of star formation-related
sizes of these galaxies, we will use $\Sigma_{\rm SFR}$ defined by optical
size to describe the concentration of star formation, but one should keep in
mind that the derived values might be the lower limits of the true ones..

Figure 5:  The $T_{\rm dust}$–sSFR (top) and $T_{\rm
dust}$–$\Sigma_{\rm{SFR}}$ (bottom) relations for our SFGs, color-coded by
their compactness ($\Sigma_{\rm{1.5}}$), at $1<z\leq 2$ (left panel) and
$2<z<3$ (right panel). The magenta dashed lines are the $T_{\rm{dust}}$–sSFRs
relations at $z=1.5$ (left panel) and $2.5$ (right panel) from Magnelli et al.
(2014). Other symbols are similar to those in Figure 4. The normalized
distributions of sSFR and $\Sigma_{\rm{SFR}}$ for eSFGs (blue) and cSFGs (red)
are also given in the corresponding panel.

Dust temperature has been found to correlate with sSFR, better than other
quantities like SFRs or $L_{\rm IR}$ (Magnelli et al., 2014). In Figure 5 we
show the relation between $T_{\rm dust}$ and sSFR, color-coded by their
compactness ($\Sigma_{\rm 1.5}$). The whole sample is divided into two
redshift bins ($1<z\leq 2$ and $2<z<3$). The running medians of $T_{\rm dust}$
for two subsamples are shown in green solid lines, while the uncertainties of
the running medians estimated via the bootstrapping method (green hatched
shadows) and the running standard deviations (yellow shadows) are also plotted
in Figure 5.

In each redshift bin, $T_{\rm dust}$ of our sample is roughly consistent with
that of the main-sequence galaxies at similar redshift (magenta dashed line;
Magnelli et al. 2014) within the large scatters. At $1<z\leq 2$, a weak
positive correlation with a linear correlation coefficient of
$\mathbf{r=0.34\pm 0.06}$ between sSFR and $T_{\rm dust}$ is found, which
supports that active star formation heats dust intensively. However, at
$2<z<3$, due to the lack of small-$T_{\rm dust}$ SFGs in the low-sSFR regime,
such a relation is more ambiguous, and $T_{\rm dust}$ seems to show no
correlation with sSFR ($\mathbf{r=0.16\pm 0.09}$). The lack of small-$T_{\rm
dust}$ SFGs with relatively low sSFR at $2<z<3$ might be due to the selection
effect, i.e., our FIR selection bias to galaxies with more intense star
formation compared to the main-sequence SFGs at similar redshift (Whitaker et
al., 2014), as suggested by the right panel of Figures 2 and the large void
region below the Magnelli et al. (2014) relation at $z\sim 2.5$ in Figure 3.

In the low-redshift bin, SFGs with different compactness are well mixed, which
is supported by the similar distributions of sSFR for eSFGs and cSFGs given in
Figure 5. However, both the distribution of individual galaxies and the
comparison of the sSFR distributions between the eSFG and cSFG subsamples
imply that more compact SFGs (redder color in Figure 5) seem to possess lower
sSFR at $2<z<3$, which might be a signature of suppression of star formation.
However, as shown in the right panel of Figure 2, our FIR selected galaxies
form an SFR detection limit of $\sim 100~{}M_{\odot}$ yr-1, which bias our
sample to more star-bursting galaxies at low-$M_{*}$ end. Given that eSFGs in
this subsample tend to be less massive than cSFGs,555Figure 2 clearly shows
that less cSFGs are observed at the low-$M_{*}$ ends in both redshift bins. In
fact, our FIR selection should bias our sample to more compact galaxies due to
their compact morphology and thus possibly higher IR surface brightness.
Therefore, given an IR detection limit and a fixed SFR, cSFGs should be much
easier to be detected, which conflicts with our observations. The physical
reason for this conflict is still unknown and might be worth further study.
eSFGs thus might be more star-bursting (higher sSFR) than cSFGs. In other
words, the observed lower sSFR of cSFGs at $2<z<3$ might result from the FIR
selection combining with the lack of cSFGs at the low-$M_{*}$ end. Further
$M_{*}$-controlled comparison reveals that cSFGs and eSFGs have similar sSFR
within the same $M_{*}$ bins, supporting that the apparently lower sSFR of
cSFGs might be just a selection effect.

High-redshift cSFGs also seem to have slightly hotter dust than that predicted
by the main-sequence SFGs (Magnelli et al., 2014). This might suggest
differences in the ISM conditions of these compact objects with respect to
typical SFGs with similar sSFR. However, considering the low number statistic
and the aforementioned selection effect, this result should be treated with
caution.

Similar plots for the $T_{\rm dust}$–$\Sigma_{\rm SFR}$ relation are also
shown in Figure 5. SFGs with greater compactness (gray to red points) have
significantly larger $\Sigma_{\rm SFR}$ than eSFGs, suggesting that size
shrinkage is more dramatic than SFR changes. The linear correlation
coefficients between $T_{\rm dust}$ and $\Sigma_{\rm SFR}$ are
$\mathbf{0.30\pm 0.06}$ and $\mathbf{0.24\pm 0.08}$ for the $1<z\leq 2$ and
$2<z<3$ subsamples, respectively, indicating a weak relation in both redshift
bins.

From Figures 5, we find that AGN hosts are distributed in a similar way as
those without AGN. The two populations share the same dependence of
$T_{\rm{dust}}$ on star formation activities, in terms of sSFR and
${\Sigma_{\rm{SFR}}}$. Star formation thus might be the main heating source of
dust in these galaxies.

In short, at $1<z\leq z$, cSFGs seem to be indistinguishable from eSFGs except
for their compact stellar morphology, while an apparently suppression of star
formation is observed for cSFGs at $2<z<3$, however, this might be a selection
effect.

###  4.3 Role of cSFGs

Given the current data, we only observed an apparently suppression of star
formation for our selected cSFGs compared to eSFGs at $2<z<3$. The role of
these cSFGs in the picture of galaxy evolution is still unclear.

Elbaz et al. (2018) compiled a sample of $z\sim 2$ Herschel-selected galaxies,
which resemble the massive end ($M_{\ast}\gtrsim 10^{11}~{}M_{\odot}$) of our
high-redshift subsample,666Note that seven out of eight Herschel-selected
galaxies in Elbaz et al. (2018) have redshift of $\gtrsim 2$. and found that
galaxies located within the scatter of the main sequence tend to have compact
star formation and short depletion timescales. The authors argued that these
galaxies could be at the last step of star formation. Although we do not have
measurement of star formation-related size for our sample, the most massive
galaxies at $2<z<3$ located within the scatter of the main sequence also tend
to have compact stellar morphology (see Figure 2), possibly suggesting a
similar properties to those galaxies hidden in the main sequence in the Elbaz
et al. (2018) sample.

Furthermore, based on observations of galaxies at $0.5<z<3$, Gómez-Guijarro et
al. (2019) reported that cSFGs and eSFGs exhibit similar star formation
efficiencies, while the ISM properties (e.g., CO excitation and local physical
conditions of the neutral gas) of cSFGs are also similar to those of eSFGs
located slightly above the main sequence. Besides the above ISM properties,
our results (i.e., Table 1 and Figure 4) provide further supplement related to
dust that no significant differences in $T_{\rm dust}$ and $M_{\rm dust}$ are
observed between cSFGs and eSFGs for both redshift subsamples.

Therefore, for cSFGs in both redshift bins, because of their similarity to
eSFGs, it is possible that these galaxies form their compact stellar cores via
slow secular evolution rather than a rapid compaction process, as suggested by
Gómez-Guijarro et al. (2019).

##  5 Summary

In this work, we investigate how dust temperature depends on the compactness
and the star formation activity, using a sample of massive high-redshift SFGs
with FIR observations from Herschel in four of 3D-HST/CANDELS fields. Our
findings are summarized as follows.

  * •

Dust temperature of both eSFGs and cSFGs agree, within the uncertainties, with
the $T_{\rm{dust}}-z$ evolutionary track of the main-sequence SFGs found by
Magnelli et al. (2014).

  * •

No significant difference in dust temperature is found between cSFGs and
eSFGs.

  * •

While similar sSFR is found for eSFGs and cSFGs at $1<z\leq 2$, cSFGs tend to
show smaller sSFR at $2<z<3$, which results from our FIR selection and the
lack of low-$M_{*}$ cSFGs. Compared to eSFGs, cSFGs are observed to have
larger $\Sigma_{\rm SFR}$, which is consistent with their remarkable size
shrinkage.

  * •

$T_{\rm{dust}}$ seems to slightly correlate in a positive way with sSFR at
$1<z\leq 2$, but such trends disappears at $2<z<3$, which might cause by FIR
selection effects.

  * •

No difference of dust temperature is observed whether AGN activities are
ignited or not, though AGNs have higher occurrence in cSFGs that is consistent
with the literature (Barro et al., 2013; Fang et al., 2015; Kocevski et al.,
2017).

  * •

Our findings, together with other ISM study in the literature, suggest similar
ISM properties between eSFGs and cSFGs, which implies that a rapid and violent
compaction process might be not necessary for the formation of cSFGs.

The intense star formation in the central bulge and strong AGN feedback may
lead to extreme properties of the ISM in the compact phase. However, our
results disfavor different ISM properties in cSFGs compared to eSFGs,
questioning the existence of a rapid compaction process to form such compact
galaxies. Our sample size is limited by both precise size measurement and the
needs for multi-wavelength Herschel detection. Future work is still required
to disclose the detail physics of the dynamic ISM in the compact phase, in
order to complete the picture of “compaction and quenching” scenario.

We are grateful to the anonymous referee whose constructive and detailed
comments helped us improve this paper. We thank Zhenyi Cai for help polishing
the manuscript. This work is supported by the National Natural Science
Foundation of China (NSFC, Nos. 11673004, 1320101002, 11433005, and 11421303)
and the National Basic Research Program of China (973 Program) (2015CB857004).
Z. Lin gratefully acknowledges support from the China Scholarship Council (No.
201806340211). G.W.F acknowledge support from Yunnan young and middle-aged
academic and technical leaders reserve talent program (No. 201905C160039) and
Yunnan Applied Basic Research Projects (2019FB007). This work is based on
observations taken by the 3D-HST Treasury Program (GO 12177 and 12328) with
the NASA/ESA HST, which is operated by the Association of Universities for
Research in Astronomy, Inc., under NASA contract NAS5-26555. This research has
made use of data from HerMES project (http://hermes.sussex.ac.uk/). HerMES is
a Herschel Key Programme utilising Guaranteed Time from the SPIRE instrument
team, ESAC scientists and a mission scientist. The HerMES data was accessed
through the Herschel Database in Marseille (HeDaM - http://hedam.lam.fr)
operated by CeSAM and hosted by the Laboratoire d’Astrophysique de Marseille.

## References

  * Barro et al. (2013) Barro, G., Faber, S. M., Pérez-González, P. G., et al. 2013, ApJ, 765, 104.  http://adsabs.harvard.edu/abs/2013ApJ...765..104B
  * Barro et al. (2014) —. 2014, ApJ, 791, 52.  http://adsabs.harvard.edu/abs/2014ApJ...791...52B
  * Barro et al. (2016) Barro, G., Kriek, M., Pérez-González, P. G., et al. 2016, ApJ, 827, L32.  http://adsabs.harvard.edu/abs/2016ApJ...827L..32B
  * Barro et al. (2017) Barro, G., Faber, S. M., Koo, D. C., et al. 2017, The Astrophysical Journal, 840, 47.  http://stacks.iop.org/0004-637X/840/i=1/a=47
  * Brammer et al. (2012) Brammer, G. B., van Dokkum, P. G., Franx, M., et al. 2012, ApJS, 200, 13.  http://stacks.iop.org/0067-0049/200/i=2/a=13
  * Brightman et al. (2014) Brightman, M., Nandra, K., Salvato, M., et al. 2014, MNRAS, 443, 1999.  http://adsabs.harvard.edu/abs/2014MNRAS.443.1999B
  * Cai et al. (2013) Cai, Z.-Y., Lapi, A., Xia, J.-Q., et al. 2013, ApJ, 768, 21.  https://ui.adsabs.harvard.edu/abs/2013ApJ...768...21C
  * Calzetti et al. (2000) Calzetti, D., Armus, L., Bohlin, R. C., et al. 2000, ApJ, 533, 682.  https://ui.adsabs.harvard.edu/abs/2000ApJ...533..682C
  * Casey (2012) Casey, C. M. 2012, MNRAS, 425, 3094.  http://adsabs.harvard.edu/abs/2012MNRAS.425.3094C
  * Casey et al. (2014) Casey, C. M., Narayanan, D., & Cooray, A. 2014, Phys. Rep., 541, 45.  http://adsabs.harvard.edu/abs/2014PhR...541...45C
  * Casey et al. (2009) Casey, C. M., Chapman, S. C., Beswick, R. J., et al. 2009, MNRAS, 399, 121.  https://ui.adsabs.harvard.edu/abs/2009MNRAS.399..121C
  * Chabrier (2003) Chabrier, G. 2003, PASP, 115, 763.  http://adsabs.harvard.edu/abs/2003PASP..115..763C
  * Chapman et al. (2005) Chapman, S. C., Blain, A. W., Smail, I., & Ivison, R. J. 2005, ApJ, 622, 772.  https://ui.adsabs.harvard.edu/abs/2005ApJ...622..772C
  * Chen et al. (2020) Chen, C.-C., Harrison, C. M., Smail, I., et al. 2020, A&A, 635, A119.  https://ui.adsabs.harvard.edu/abs/2020A&A...635A.119C
  * Civano et al. (2016) Civano, F., Marchesi, S., Comastri, A., et al. 2016, ApJ, 819, 62.  http://adsabs.harvard.edu/abs/2016ApJ...819...62C
  * Dai et al. (2018) Dai, Y. S., Wilkes, B. J., Bergeron, J., et al. 2018, MNRAS, 478, 4238.  http://adsabs.harvard.edu/abs/2018MNRAS.478.4238D
  * Dai et al. (2012) Dai, Y. S., Bergeron, J., Elvis, M., et al. 2012, ApJ, 753, 33.  http://adsabs.harvard.edu/abs/2012ApJ...753...33D
  * Damjanov et al. (2009) Damjanov, I., McCarthy, P. J., Abraham, R. G., et al. 2009, ApJ, 695, 101.  http://adsabs.harvard.edu/abs/2009ApJ...695..101D
  * Dekel & Burkert (2014) Dekel, A., & Burkert, A. 2014, MNRAS, 438, 1870.  http://adsabs.harvard.edu/abs/2014MNRAS.438.1870D
  * Elbaz et al. (2011) Elbaz, D., Dickinson, M., Hwang, H. S., et al. 2011, ArXiv e-prints, arXiv:1105.2537.  http://adsabs.harvard.edu/abs/2011arXiv1105.2537E
  * Elbaz et al. (2018) Elbaz, D., Leiton, R., Nagar, N., et al. 2018, A&A, 616, A110.  https://ui.adsabs.harvard.edu/abs/2018A&A...616A.110E
  * Fang et al. (2015) Fang, G., Ma, Z., Kong, X., & Fan, L. 2015, ApJ, 807, 139.  http://adsabs.harvard.edu/abs/2015ApJ...807..139F
  * Fang et al. (2013) Fang, J. J., Faber, S. M., Koo, D. C., & Dekel, A. 2013, ApJ, 776, 63.  http://adsabs.harvard.edu/abs/2013ApJ...776...63F
  * Fujimoto et al. (2018) Fujimoto, S., Ouchi, M., Kohno, K., et al. 2018, ApJ, 861, 7.  https://ui.adsabs.harvard.edu/abs/2018ApJ...861....7F
  * Gómez-Guijarro et al. (2019) Gómez-Guijarro, C., Magdis, G. E., Valentino, F., et al. 2019, ApJ, 886, 88\.  https://ui.adsabs.harvard.edu/abs/2019ApJ...886...88G
  * Grogin et al. (2011) Grogin, N. A., Kocevski, D. D., Faber, S. M., et al. 2011, ApJS, 197, 35.  https://ui.adsabs.harvard.edu/abs/2011ApJS..197...35G
  * Hopkins et al. (2008) Hopkins, P. F., Hernquist, L., Cox, T. J., & Kereš, D. 2008, ApJS, 175, 356\.  http://adsabs.harvard.edu/abs/2008ApJS..175..356H
  * Kocevski et al. (2017) Kocevski, D. D., Barro, G., Faber, S. M., et al. 2017, ApJ, 846, 112.  http://adsabs.harvard.edu/abs/2017ApJ...846..112K
  * Koekemoer et al. (2011) Koekemoer, A. M., Faber, S. M., Ferguson, H. C., et al. 2011, ApJS, 197, 36.  https://ui.adsabs.harvard.edu/abs/2011ApJS..197...36K
  * Kovács et al. (2006) Kovács, A., Chapman, S. C., Dowell, C. D., et al. 2006, ApJ, 650, 592.  http://adsabs.harvard.edu/abs/2006ApJ...650..592K
  * Kriek et al. (2009) Kriek, M., van Dokkum, P. G., Labbé, I., et al. 2009, ApJ, 700, 221.  http://adsabs.harvard.edu/abs/2009ApJ...700..221K
  * Liang et al. (2019) Liang, L., Feldmann, R., Kereš, D., et al. 2019, arXiv e-prints, arXiv:1902.10727.  http://adsabs.harvard.edu/abs/2019arXiv190210727L
  * Luo et al. (2017) Luo, B., Brandt, W. N., Xue, Y. Q., et al. 2017, ApJS, 228, 2.  http://adsabs.harvard.edu/abs/2017ApJS..228....2L
  * Lutz et al. (2011) Lutz, D., Poglitsch, A., Altieri, B., et al. 2011, A&A, 532, A90.  http://adsabs.harvard.edu/abs/2011A%26A...532A..90L
  * Ma et al. (2019) Ma, X., Hayward, C. C., Casey, C. M., et al. 2019, arXiv e-prints, arXiv:1902.10152.  http://adsabs.harvard.edu/abs/2019arXiv190210152M
  * Magnelli et al. (2013) Magnelli, B., Popesso, P., Berta, S., et al. 2013, A&A, 553, A132.  http://adsabs.harvard.edu/abs/2013A%26A...553A.132M
  * Magnelli et al. (2014) Magnelli, B., Lutz, D., Saintonge, A., et al. 2014, A&A, 561, A86.  http://adsabs.harvard.edu/abs/2014A%26A...561A..86M
  * Marchesi et al. (2016) Marchesi, S., Civano, F., Elvis, M., et al. 2016, ApJ, 817, 34.  http://adsabs.harvard.edu/abs/2016ApJ...817...34M
  * Momcheva et al. (2016) Momcheva, I. G., Brammer, G. B., van Dokkum, P. G., et al. 2016, ApJS, 225, 27\.  http://adsabs.harvard.edu/abs/2016ApJS..225...27M
  * Nandra et al. (2015) Nandra, K., Laird, E. S., Aird, J. A., et al. 2015, ApJS, 220, 10.  http://adsabs.harvard.edu/abs/2015ApJS..220...10N
  * Oliver et al. (2012) Oliver, S. J., Bock, J., Altieri, B., et al. 2012, MNRAS, 424, 1614.  http://adsabs.harvard.edu/abs/2012MNRAS.424.1614O
  * Peng et al. (2002) Peng, C. Y., Ho, L. C., Impey, C. D., & Rix, H.-W. 2002, AJ, 124, 266.  http://adsabs.harvard.edu/abs/2002AJ....124..266P
  * Puglisi et al. (2019) Puglisi, A., Daddi, E., Liu, D., et al. 2019, ApJ, 877, L23.  https://ui.adsabs.harvard.edu/abs/2019ApJ...877L..23P
  * Roseboom et al. (2010) Roseboom, I. G., Oliver, S. J., Kunz, M., et al. 2010, MNRAS, 409, 48.  http://adsabs.harvard.edu/abs/2010MNRAS.409...48R
  * Roseboom et al. (2012) Roseboom, I. G., Ivison, R. J., Greve, T. R., et al. 2012, MNRAS, 419, 2758.  http://adsabs.harvard.edu/abs/2012MNRAS.419.2758R
  * Schreiber et al. (2018) Schreiber, C., Elbaz, D., Pannella, M., et al. 2018, A&A, 609, A30.  http://adsabs.harvard.edu/abs/2018A%26A...609A..30S
  * Schreiber et al. (2016) —. 2016, A&A, 589, A35.  http://adsabs.harvard.edu/abs/2016A%26A...589A..35S
  * Skelton et al. (2014) Skelton, R. E., Whitaker, K. E., Momcheva, I. G., et al. 2014, ApJS, 214, 24\.  http://adsabs.harvard.edu/abs/2014ApJS..214...24S
  * Spitler et al. (2014) Spitler, L. R., Straatman, C. M. S., Labbé, I., et al. 2014, ApJ, 787, L36.  https://ui.adsabs.harvard.edu/abs/2014ApJ...787L..36S
  * Stern (2015) Stern, D. 2015, ApJ, 807, 129.  http://adsabs.harvard.edu/abs/2015ApJ...807..129S
  * Tacchella et al. (2016a) Tacchella, S., Dekel, A., Carollo, C. M., et al. 2016a, MNRAS, 458, 242.  http://adsabs.harvard.edu/abs/2016MNRAS.458..242T
  * Tacchella et al. (2016b) —. 2016b, MNRAS, 457, 2790.  http://adsabs.harvard.edu/abs/2016MNRAS.457.2790T
  * van der Wel et al. (2014) van der Wel, A., Franx, M., van Dokkum, P. G., et al. 2014, ApJ, 788, 28.  http://adsabs.harvard.edu/abs/2014ApJ...788...28V
  * van Dokkum et al. (2008) van Dokkum, P. G., Franx, M., Kriek, M., et al. 2008, ApJ, 677, L5.  http://adsabs.harvard.edu/abs/2008ApJ...677L...5V
  * Wang et al. (2018) Wang, E., Kong, X., & Pan, Z. 2018, ArXiv e-prints, arXiv:1808.05929.  http://adsabs.harvard.edu/abs/2018arXiv180805929W
  * Weisskopf et al. (2000) Weisskopf, M. C., Tananbaum, H. D., Van Speybroeck, L. P., & O’Dell, S. L. 2000, in Proc. SPIE, Vol. 4012, X-Ray Optics, Instruments, and Missions III, ed. J. E. Truemper & B. Aschenbach, 2–16.  https://ui.adsabs.harvard.edu/abs/2000SPIE.4012....2W
  * Whitaker et al. (2014) Whitaker, K. E., Franx, M., Leja, J., et al. 2014, ApJ, 795, 104.  http://adsabs.harvard.edu/abs/2014ApJ...795..104W
  * Williams et al. (2009) Williams, R. J., Quadri, R. F., Franx, M., van Dokkum, P., & Labbé, I. 2009, ApJ, 691, 1879.  http://adsabs.harvard.edu/abs/2009ApJ...691.1879W
  * Woo & Ellison (2018) Woo, J., & Ellison, S. L. 2018, arXiv e-prints, arXiv:1810.12298.  http://adsabs.harvard.edu/abs/2018arXiv181012298W
  * Xue et al. (2016) Xue, Y. Q., Luo, B., Brandt, W. N., et al. 2016, ApJS, 224, 15.  http://adsabs.harvard.edu/abs/2016ApJS..224...15X
  * Zolotov et al. (2015) Zolotov, A., Dekel, A., Mandelker, N., et al. 2015, MNRAS, 450, 2327.  http://adsabs.harvard.edu/abs/2015MNRAS.450.2327Z


# Knowledge Distillation for Detection Transformer with Consistent
Distillation Points Sampling

Yu Wang Xin Li Shengzhao Wen Fukui Yang Wanping Zhang Gang Zhang
Haocheng Feng Junyu Han Errui Ding
Baidu VIS
{wangyu106, lixin68, wenshengzhao, yangfukui, zhangwanping, zhanggang03,
fenghaocheng, hanjunyu<EMAIL_ADDRESS>
###### Abstract

DETR is a novel end-to-end transformer architecture object detector, which
significantly outperforms classic detectors when scaling up the model size. In
this paper, we focus on the compression of DETR with knowledge distillation.
While knowledge distillation has been well-studied in classic detectors, there
is a lack of researches on how to make it work effectively on DETR. We first
provide experimental and theoretical analysis to point out that the main
challenge in DETR distillation is the lack of consistent distillation points.
Distillation points refer to the corresponding inputs of the predictions for
student to mimic, and reliable distillation requires sufficient distillation
points which are consistent between teacher and student. Based on this
observation, we propose a general knowledge distillation paradigm for DETR(KD-
DETR) with consistent distillation points sampling. Specifically, we decouple
detection and distillation tasks by introducing a set of specialized object
queries to construct distillation points. In this paradigm, we further propose
a general-to-specific distillation points sampling strategy to explore the
extensibility of KD-DETR. Extensive experiments on different DETR
architectures with various scales of backbones and transformer layers validate
the effectiveness and generalization of KD-DETR. KD-DETR boosts the
performance of DAB-DETR with ResNet-18 and ResNet-50 backbone to 41.4$\%$,
45.7$\%$ mAP, respectively, which are 5.2$\%$, 3.5$\%$ higher than the
baseline, and ResNet-50 even surpasses the teacher model by $2.2\%$.

##  1 Introduction

Object detection, as a fundamental task in computer vision, has long been
dominated by detectors based on convolutional neural networks
(CNNs)[9][5][48]. In recent years, [3] propose the novel end-to-end detector
Detection Transformer (DETR) which eliminates the need for hand-crafted
anchors and non-maximum suppression (NMS). [49][16][21][44]further make
remarkable stride towards the scalability and potential of DETR, significantly
outperforming classical detectors.

Figure 1: ResuIts Overview. Improvement of KD-DETR on DAB-DETR, Deformable
DETR, DAB-Deformable and DINO. Results are evaluated on MS COCO2017. ResNet-50
is used as teacher for ResNet-18, and ResNet101 for ResNet-50. Note that the
student model can even exceed the teacher significantly in the first three
architectures.

Different from classic detectors, DETR interprets object detection as an end-
to-end set prediction problem with bipartite matching. A set of learnable
object queries are introduced, each responsible for a certain instance. The
object queries interact with features extracted from the encoder to make final
predictions of box locations and categories. Despite the impressive
performance, the growing model scale prevents DETR from being deployed to
real-world applications with urgent computation budget requirement.

(a) Distillation Points in Classic Detector

(b) Distillation Points in DETR

(c) Distillation Points in KD-DETR

Figure 2: Schematic Illustrations of Distillation Points in Different
Architecture: (a) Two-Stage Detector: both positive and negetive proposals in
RPN and RoI are consistent distillation points with strict one-to-one
correspondence between teacher and student model; (b) DETR: Object queries
lacks spatial or semantic relationship between teacher and student model,
resulting in inconsistent distillation points; (c) KD-DETR: A set of special
object queries is introduced to construct consistent distillation points for
DETR distillation

To address this problem, current works have made efforts in designing
efficient DETR architectures, reducing the encoder tokens utilized in cross-
attention module to decrease the computation cost [27], and leveraging dense
prior from RPN to downsize the decoder layers [39]. In this work, we
concentrate on compressing the large-scale DETR model by knowledge
distillation [13] approaches, which is a promising technique for model
compression and accuracy boosting. Knowledge distillation can transfer the
knowledge learned from large and cumbersome DETR models to small and efficient
ones by forcing the student to mimic the predictions from teacher, whether
logits or internal activations. However, modern knowledge distillation methods
are designed under CNN-based detectors, and researches on expanding it to
general DETR compression are limited.

We start from experiments of applying classic logit-based distillation[4] to
DETR, which unfortunately leads to degradation of student performance. With
these experiments, we observe that the critical challenge lies in the
different formulations of DETR and classic detectors. Compared with classic
detectors, the set-prediction formulation of DETR naturally contains fewer
consistent distillation points. We use distillation points to denote the
corresponding input of the prediction for mimicking in knowledge distillation,
and the sufficiency and consistency of distillation points form the foundation
of knowledge distillation. Specifically, abundant distillation points which
are kept consistent between teacher and student models are essential for
effective distillation. As shown in Fig 2(a), classic detectors make
predictions for a set of region proposals generated from the sliding window
locations on the images with handcraft scales. This pattern naturally ensures
a strict spatial correspondence between the large number of proposals made by
teacher and student models, even for those negative ones with low confidence,
thus providing a sufficient number of consistent distillation points for
mimicking. In DETR, as is shown in Fig 2(b), the distillation points actually
consist of both image and object queries. However, the object queries from
teacher and student models are egocentric and even differ in number, thus
lacking definite correspondence, especially those redundant negative queries
in bipartite matching. As the distillation points in DETR are inconsistent and
insufficient, the predictions acquired from teacher are not reliable or
informative for student to mimic.

The observation above raises the issue: how to obtain sufficient and
consistent distillation points for DETR distillation. To address this issue,
we focus on the object queries, and propose a general knowledge distillation
paradigm for DETR(KD-DETR) with consistent distillation points sampling. In
KD-DETR, as illustrated in Fig 2(c), we decouple detection task and
distillation task by introducing a set of specialized object queries to
construct distillation points. The distillation points are unlearnable and
shared between teacher and student models, probing the “dark knowledge” in
teacher model. In this way, consistent distillation points with customized
quantities become available. With the paradigm of KD-DETR, we propose a
general-to-specific distillation points sampling strategy to probe
comprehensive knowledge in teacher model. For general sampling, we randomly
sample a set of object queries which are uniformly distributed to coarsely
scan the whole feature maps and probe more general knowledge from teacher
model. For specific sampling, the well-optimized object queries in teacher
model are used to construct distillation points, as they represent the
specific knowledge the teacher model concerns. Considering the imbalance of
positive and negative regions in object detection, we further reweigh the
importance of each distillation point with its foreground probability
predicted by the teacher model.

To the best of our knowledge, this is the first work to propose a general
knowledge distillation paradigm for DETR. In this paper, we first provide a
thorough experimental and theoretical analysis of the degraded failure of
knowledge distillation in DETR. Based on the analysis, we design a novel KD-
DETR that significantly improves the performance of DETR distillation. KD-DETR
has both flexibility to different DETR architectures, even for teacher and
student with different object query settings, and immense potential for
scalability. To demonstrate the effectiveness and generalization of KD-DETR,
we conduct extensive experiments on the MS COCO2017[20] dataset on various
DETR methods, compressing the scale of backbone and the number of transformer
layers. As is shown in Fig 1, our method significantly boosts the performance
of student models. DAB-DETR with ResNet-18 and ResNet-50 backbone achieves
41.4$\%$, 45.7$\%$ mAP, respectively, which are 5.2$\%$, 3.5$\%$ higher than
the baseline, and ResNet-50 even surpasses the teacher model of ResNet-101 by
2.2$\%$; and Deformable DETR with ResNet-18 reaches 43.0$\%$ mAP, 2.9$\%$
higher than the baseline.

##  2 Related Work

###  2.1 Classic Object Detection

Classic detectors with CNN view object detection as a verification task with a
sliding window on the image to generate anchors. The mainstream detectors can
be divided into one-stage detectors[30][18] and two-stage detectors[26][2].
One-stage detectors, such as Retinanet[19], YOLO[25], and FCOS[30], directly
predict the category and regression of anchors on each pixel of the feature
maps. While two-stage detectors such as Faster-RCNN[26] and its
variants[24][35][42] introduce a Region Proposal Networks(RPN) to generate
proposals, and a ROIPool or ROIAlign[33] to extract features of each region
proposal for further classification and regression refinement. Both one-stage
and two-stage detectors require post-processing, such as NMS, to remove
duplicate predictions.

###  2.2 Detection Transformer

[3] first propose an end-to-end transformer-based detector without any post-
processing. Different from classic object detection, DETR interprets object
detection as a set-prediction problem with bipartite matching.

Lots of follow-up focus on the slow convergence of DETR[5][28][8][43].
Deformable DETR[49] focuses on the multi-head attention, and introduces a
deformable attention module by generating reference points for query elements.
Each reference point will only concentrate on a small number of locations on
the whole feature map. An alternative way is to add more prior information to
the object queries in the decoder. Conditional-DETR[23] decouples the context
and position features in object queries and generates position features by
spatial location. DAB-DETR[21] further introduces the width and height
information to the positional features. Anchor DETR[34] also encodes the
anchor points as object queries with multiple patterns, and further designs a
row-column decouple attention to reduce memory cost. The recent work of
DINO[44] draws the existing novel techniques, and further exerts the potential
of DETR by enlarging the scale of model and datasets.

Besides, another problem in DETR is the scale and computation cost of the
model. Current works solving this problem by designing more efficient DETR
architecture. Sparse DETR[27] reduces the computation cost by sparsifying
encoder tokens. Efficient DETR[39] otherwise introduce RPN to generate object
queries and eliminate the cascading decoder layers in DETR. PnP DETR[32]
shorten the length of sampled feature with a poll and pool sampling module.

###  2.3 Knowledge Distillation

Knowledge distillation is a widely-used method for model compression and
accuracy boosting by transferring the knowledge in a large cumbersome teacher
model to a small student. [13] first propose the concept of knowledge
distillation, where the student mimics the soft predictions from teacher.
Knowledge distillation has been utilized in various fields including
classification[47][29][1], object detection[37][46], semantic
segmentation[36][22]. According to the objective of mimicking, knowledge
distillation can be divided into three categories: response-based[47],
feature-based[12][38] and relation-based[40][41], which distill with logits,
intermediate activations and the relation of features in different layers
respectively.

Recently, several works focus on applying knowledge distillation to object
detection[10][4][14]. [4] successfully distill the features on the neck, the
classification head, and the regression head, while [17] choose to distill the
logits and features from the RPN head. To overcome the imbalance of foreground
and background, [33] introduce fine-grained mask to focus on the region close
to ground-truth bounding boxes, [6] pay more attention to the regions where
teacher and student are divided in predictions. However, the modern knowledge
distillation methods for object detection are built upon the architecture of
CNN-based detectors, and are not suitable for DETR due to the completely
different transformer architecture.

Since [7] introduce transformer to vision task, several works[31][45] have
verified the effectiveness of knowledge distillation on ViT. However, these
methods perform trivially or even harmfully on DETR, and there is still a lack
of research on DETR distillation.

##  3 A Closer Look at DETR Distillation

In this section, we first revisit the DETR architecture briefly. Then we
conduct a series of classic knowledge distillation experiments on DETR to
reveal that the core of DETR distillation is to obtain sufficient and
consistent distillation points.

###  3.1 Revisiting DETR

DETR is built upon the encoder-decoder architecture of transformer. The
encoder takes pixels of the feature map from backbone as input for multi-head
self-attention to extract context features $X\in\mathbb{R}^{HW\times D}$,
where $HW$ denotes the resolution of the feature map, and $D$ denotes the
feature dimension. The decoder takes the features from encoder and a set of
learnable object queries $\mathbf{Q}=\\{q_{i}\in\mathbb{R}^{D}|i=1,...,N\\}$
as input, where N denotes the number of queries. Each object query is an
abstract feature describing a certain instance, and will probe and pool the
features from encoder through cross-attention to make predictions of category
$\mathbf{C}=\\{\mathbf{c}_{i}\in\mathbb{R}^{K}|i=1,...,N\\}$ and location
$\mathbf{B}=\\{\mathbf{b}_{i}=(bx_{i},by_{i},bw_{i},bh_{i})|i=1,...,N\\}$,
where K denotes the number of categories. Finally, the Hungarian algorithm is
used to find a bipartite matching between ground truth and predictions of
object queries.

Strategy | AP | $AP_{50}$ | $AP_{75}$
---|---|---|---
Baseline | 36.2 | 56.1 | 44.7
Inconsistent | 35.1 | 55.2 | 36.7
Similar Foreground | 37.2 | 57.4 | 39.9
Similar General | 37.4 | 58.0 | 40.6
Table 1: Distillation with Different Distillation Points. Inconsistent
distillation points are harmful for distillation.

###  3.2 Consistent Distillation Points

The core idea of knowledge distillation is forcing the student to mimic the
prediction of teacher, which can be interpreted as matching the mapping
function of student and teacher with a set of distillation points.
Distillation points refer to the corresponding input $\mathbf{x}$ of the
predictions, as $\mathbf{y}=f(\mathbf{x})$, where $f$ represents the model. In
this view, the distillation points should kept sufficient in quantity and
consistent between teacher and student models for effective and reliable
matching. However, comparing the formulation of CNN detector and DETR, a
critical challenge of DETR distillation lies in the lacking of consistent
distillation points.

Figure 3: KD-DETR architecture KD-DETR decouples the distillation and
detection tasks by introducing a set of distillation points shared between
teacher and student models. For distillation, the distillation points are
served as the query of the transformer decoders for both teacher and student
models, and the student will mimic teacher’s classification and box location
predictions. The weight of each distillation point is measured by its
foreground probability predicted by teacher. For detection task, the original
object queries are processed by the student decoder for final prediction.

Classic detector degrades object detection to a verification problem which
combines classification and regression, and introduces a set of anchors to
specify the region for verification. In this way, the formulation of
distillation points consists of the image and the location and scale of anchor
$\mathbf{x}=(\mathbf{I},\mathbf{anchor})$. As the anchors are generated
through the sliding window strategy with handcrafted shapes, the locations and
scales of anchors are implicitly embedded in the model architecture as prior
information. Since student and teacher models share the same or similar
architectures, a large number of object proposals generated by teacher and
student models naturally have strict spatial correspondence, even for those
background regions with low confidence. With the spatial correspondence which
can be viewed as an inductive bias of CNN, classic detector naturally
guarantees a sufficient number of consistent distillation points.

In contrast, the DETR formulates object detection as a set-prediction problem.
The distillation points, therefore, become the combination of the image and
the object queries $\mathbf{x}=(\mathbf{I},\mathbf{q})$. However, the object
queries in different models are egocentric, as they are initialized and
optimized by themselves independently. Since object queries play the role of
probing and pooling the features of certain instances, they have inconsistent
concentration preferences in different models. Consequently, the formulation
of DETR naturally lacks the ability to provide sufficient distillation points
with strict consistency between teacher and student models, and the
predictions acquired from teacher are not informative or reliable for student
to mimic.

###  3.3 Distillation with Inconsistent Distillation Points

To validate the analysis above that the sufficiency and consistency of
distillation points is the essential challenge in DETR distillation, we start
with applying the original logit-based distillation method[4] in classic
detector to DETR, which mimics the category and box location logits
predictions of teacher model. We examine three distillation points strategies:
Inconsistent, Similar Foreground, and Similar General. In Inconsistent, all
the object queries are viewed as distillation points with their original
permutation; In Similar Foreground, only object queries matched to ground
truth in bipartite matching will be used as distillation points, and will
permute with the same order of ground truth label; Similar General further
increase the number of distillation points by viewing the average of negative
object queries in bipartite matching as a general background distillation
point. Experiments are conducted on DAB-DETR and evaluated on MS COCO2017,
with ResNet18 as student and ResNet-50 as teacher.

Models |  | Epochs | AP | $AP_{50}$ | $AP_{75}$ | $AP_{S}$ | $AP_{M}$ | $AP_{L}$ | FPS | Params
---|---|---|---|---|---|---|---|---|---|---
DAB-DETR | ResNet-50(T) | 50 | 42.1 | 63.1 | 44.7 | 21.5 | 45.7 | 69.3 | 32 | 44M
| ResNet-18(S) | 50 | 36.2 | 56.1 | 37.9 | 16.9 | 39.0 | 53.5 | 76 | 31M
| Ours | 50 | 41.4 | 61.4 | 44.2 | 21.2 | 44.7 | 58.7 | 76 | 31M
| Gains |  | +5.2 | +5.3 | +6.3 | +4.3 | +5.7 | +5.2 |  |
| ResNet-34(S) | 50 | 40.6 | 60.9 | 43.3 | 21.4 | 44.3 | 56.3 | 51 | 42M
| Ours | 50 | 44.1 | 64.4 | 47.3 | 24.6 | 47.8 | 61.0 | 51 | 42M
| Gains |  | +3.5 | +3.5 | +4.0 | +3.2 | +3.5 | +4.7 |  |
DAB-DETR | ResNet-101(T) | 50 | 43.5 | 63.9 | 44.6 | 23.6 | 47.3 | 61.5 | 19 | 63M
| ResNet-50(S) | 50 | 42.1 | 63.1 | 44.7 | 21.5 | 45.7 | 60.3 | 32 | 44M
| Ours | 50 | 45.7 | 66.3 | 49.4 | 26.4 | 49.8 | 62.7 | 32 | 44M
| Gains |  | +3.2 | +3.2 | +4.7 | +4.9 | +4.1 | +2.4 |  |
Deformable-DETR | ResNet-50(T) | 50 | 43.8 | 62.6 | 47.7 | 26.4 | 47.1 | 58.0 | 22 | 40M
| ResNet-18(S) | 50 | 40.1 | 58.1 | 43.7 | 22.4 | 42.8 | 54.2 | 40 | 27M
| Ours | 50 | 43.4 | 61.8 | 47.4 | 25.2 | 46.6 | 57.7 | 40 | 27M
| Gains |  | +3.3 | +3.7 | +3.7 | +2.8 | +3.8 | +3.5 |  |
Deformable-DETR | ResNet-101(T) | 50 | 44.6 | 63.7 | 48.5 | 26.3 | 48.2 | 58.7 | 15 | 59M
| ResNet-50(S) | 50 | 43.8 | 62.6 | 47.7 | 26.4 | 47.1 | 58.0 | 22 | 47M
| Ours | 50 | 46.1 | 65.6 | 50.6 | 28.3 | 50.0 | 60.7 | 22 | 47M
| Gains |  | +2.3 | +3.0 | +2.9 | +1.9 | +2.9 | +2.7 |  |
DAB-Deformable | ResNet-50(T) | 50 | 46.8 | 66.0 | 50.4 | 29.1 | 49.8 | 62.3 | 19 | 47M
| ResNet-18(S) | 50 | 45.8 | 63.5 | 50.1 | 29.6 | 48.5 | 59.0 | 30 | 34M
| Ours | 50 | 47.2 | 65.4 | 51.7 | 31.5 | 50.3 | 60.6 | 30 | 34M
| Gains |  | +1.4 | +1.9 | +1.6 | +1.9 | +1.8 | +1.6 |  |
DINO | ResNet-50(T) | 36 | 50.9 | 69.0 | 55.3 | 34.6 | 54.1 | 64.6 | 18 | 47M
| ResNet-18(S) | 12 | 44.0 | 61.2 | 48.1 | 27.4 | 46.9 | 56.9 | 29 | 34M
| Ours | 12 | 45.4 | 62.2 | 49.3 | 27.3 | 48.2 | 59.0 | 29 | 34M
| Gains |  | +1.4 | +1.0 | +1.2 | -0.1 | +1.3 | +2.1 |  |
Table 2: Results of the proposed KD-DETR with different DETR detectors and
backbones with various scale.

As shown in Table 1, Inconsistent distillation points results in great
degradation of the student model with unreliable knowledge from teacher model;
Similar Foreground with semantic-similar foreground distillation points
alleviate the problem; and Similar General achieves further improvement by
increasing the number of distillation points with general background features.
The preliminary experiments validate that the sufficiency and consistency of
distillation points are of prime importance to improve the performance of
student model in DETR distillation.

##  4 KD-DETR

To address the lack of consistent distillation points in DETR, we propose a
general knowledge distillation paradigm for DETR with consistent distillation
points sampling, called KD-DETR. As illustrated in Fig 3, KD-DETR introduces a
set of specialized object queries $\mathbf{\tilde{q}}$ shared between teacher
and student models to construct distillation points. Decoupling the
distillation task and detection task, KD-DETR can provide sufficient and
consistent distillation points. We denote the original input and sampled
distillation points as
$\mathbf{x}=\\{\mathbf{I},\mathbf{q}\\},\mathbf{\tilde{x}}=\\{\mathbf{I},\mathbf{\tilde{q}}\\}$
respectively.

For detection task, the student model is first optimized by its original
detection loss: the original input $\mathbf{x}$ is fed into student model to
make category and box location predictions, which will be assigned to the
ground truth with bipartite matching and calculate detection loss.

For distillation task, the sampled distillation points $\mathbf{\tilde{x}}$
will be fed into both student and teacher to make category and location
predictions $\mathbf{c},\mathbf{b}$:

$\displaystyle\mathbf{c^{s}},\mathbf{b^{s}}=f^{s}(\mathbf{I},\mathbf{\tilde{q}}),$
(1)
$\displaystyle\mathbf{c^{t}},\mathbf{b^{t}}=f^{t}(\mathbf{I},\mathbf{\tilde{q}}),$
(2)

where $f^{s},f^{t}$ refer to student and teacher model respectively. The
distillation loss is calculated with following form:

$\begin{split}\mathcal{L}_{distill}=&\sum_{i=1}^{M}[\lambda_{cls}\mathcal{L}_{KL}(\mathbf{\hat{c}}^{t}_{i}\|\mathbf{\hat{c}}^{s}_{i})+\lambda_{L1}\mathcal{L}_{L1}(\mathbf{b}_{i}^{s},\mathbf{b}_{i}^{t}),\\\
+&\lambda_{GIoU}\mathcal{L}_{GIoU}(\mathbf{b}_{i}^{s},\mathbf{b}_{i}^{t})],\end{split}$
(3)

where M denotes the number of distillation points. For classification, we
choose the KL-divergence $\mathcal{L}_{KL}$ as distillation loss with
temperature T: $\mathbf{\hat{c}}=SoftMax(\frac{\mathbf{c}}{T})$. For box
regression, $\mathcal{L}_{L1}$ and $\mathcal{L}_{GIoU}$ represent the L1 and
GIoU loss for location distillation, which have the same formulation with
detection loss. $\lambda_{cls},\lambda_{L1}$ and $\lambda_{GIoU}$ represent
the coefficient of corresponding loss.

###  4.1 Distillation Points Sampling

Generally, object queries are a set of abstract features responsible for
certain objects by probing and pooling the context features from encoder.
Existing works interpret object queries as anchors or reference points,
revealing that each object query is sensitive to a particular region on the
feature maps. Following this perspective, we provide a comprehensive general-
to-specific sampling strategies for distillation points sampling:
$\mathbf{\tilde{q}}=\\{\mathbf{q_{g}},\mathbf{q_{s}}\\}$.

General Sampling with Random Initialized Queries

In general sampling, we hope to probe teacher’s general responses on different
locations of the features by sparsely scanning the whole feature maps.
Therefore, we randomly initialize a set of object queries, which are uniformly
distributed on the features, to construct the general distillation points:
$\mathbf{q_{g}}=\\{\mathbf{q_{i}}\sim\mathcal{U}(0,1)|i=1,...,M_{g}\\}$, where
$M_{g}$ denotes the number of general distillation points. To learn more
general knowledge from teacher, we leave these distillation points unlearnable
during training.

Specific Sampling with Teacher Queries

While the general sampling provides a global retrieval of the features, we
further propose a specific sampling strategy, focusing on those regions where
teacher pays more attention. An intuitive way for specific sampling is to
directly reuse the well-optimized object queries in teacher
model:$\mathbf{q_{s}}=\mathbf{q_{teacher}}$. While teacher model is learned to
concentrate more on these object queries, the predictions in these areas are
more precise and informative.

Foreground Attention Weight

The imbalance between foreground and background regions is one critical
problem in object detection distillation, not special in DETR. An intuitive
way is to adopt the classification scores predicted by teacher model as the
attention weights for distillation points. Concretely, those distillation
points with higher scores when classified to a certain object are regarded as
foreground distillation points, and are more important in knowledge
distillation.

$w_{i}=\max_{c\in[0,K]}p^{t}(y_{c}|\mathbf{q}_{i}),$ (4)

where $w_{i}$ denotes the attention weight of $\mathbf{q}_{i}$, and
$p^{t}(y_{c}|\mathbf{q}_{i})$ denotes the probability of $\mathbf{q}_{i}$
assigned to category $c$ predicted by teacher model. In this way, the
distillation loss in Eq. (3) will be writen as follow:

$\begin{split}\mathcal{L}_{distill}=&\sum_{i=1}^{M}w_{i}[\lambda_{cls}\mathcal{L}_{KL}(\mathbf{\hat{c}}^{t}_{i}\|\mathbf{\hat{c}}^{s}_{i})\\\
+&\lambda_{L1}\mathcal{L}_{L1}(\mathbf{b}_{i}^{s},\mathbf{b}_{i}^{t})+\lambda_{GIoU}\mathcal{L}_{GIoU}(\mathbf{b}_{i}^{s},\mathbf{b}_{i}^{t})]\end{split}$
(5)

##  5 Experiment

To validate the effectiveness and generalization of KD-DETR, we evaluate it on
different DETR architectures, with various scales of backbones and the number
of transformer layers. To support our analysis of consistent distillation
points, we further conduct an extensive ablation study.

###  5.1 Experimental Settings

Datasets: All the proposed experiments are evaluated on MS COCO2017[20]
spanning 80 categories, with the default split of 117k training images for
training and 5k validate images for testing. Standard COCO evaluation metrics
are adopted for validation.

Implementation Details: As KD-DETR is a plug-and-play distillation module, we
follow the original settings of hyper-parameters, learning rate scheduler and
optimizer of each model for training. We choose ResNet[11] as backbone, which
are pre-trained on ImageNet1K[15]. For distillation task, we set hyper-
parameters of the coefficient of the distillation loss as
$\lambda_{kl}=1,\lambda_{L1}=5,\lambda_{GIoU}=2$. The number of General
distillation points is $300,300,300,900$ for DAB-DETR, Deformable DETR, DAB-
Deformable-DETR, and DINO. We train our model on Nvidia A100 GPUs with batch
size set to 16 in total. The reported FPS is measured on a single Nvidia A100
GPU.

###  5.2 Distillation on Different DETR benchmarks

In this paper, we evaluate our method on four typical DETR architectures: DAB-
DETR[21], which is a single-scale DETR with special object query settings;
Deformable-DETR[49], which is a multi-scale DETR with deformable attention
module; and the combination DAB-Deformable-DETR[21]. We additionally provide
an additional experiment on DINO[44], which is the SOTA DETR model combining a
series of novel techniques including deformable attention, two-stage object
queries settings, and de-noising module, to evaluate our effectiveness on
well-optimized model with high accuracy.

The results are illustrated in Table2. For DAB-DETR, KD-DETR significantly
boosts the performance of ResNet-18, ResNet-34, and ResNet-50 with $5.2\%$,
$3.5\%$, and $3.2\%$ mAP improvement respectively. Note that student with
ResNet-34 and ResNet-50 even surpass the teacher by $2.0\%$ and $2.2\%$ mAP.
Deformable DETR with ResNet-18 and ResNet-50, KD-DETR achieve $3.3\%$ and
$2.5\%$ mAP gains, and student with ResNet-50 also exceed the teacher with
$1.5\%$ mAP. For DAB-Deformable-DETR, KD-DETR achieves $1.4\%$ mAP higher than
the baseline model with ResNet-18, and surpasses the teacher model with
ResNet-50 by $0.4\%$. For DINO with ResNet-18, KD-DETR also improves $1.4\%$
mAP with 12 epoch training scheduler.

###  5.3 Distillation on the Transformer Layers

Besides the scale of backbone, the layer number of transformer encoder and
decoder is also an important factor of the model size and computation cost in
DETR. In this paper, we also conduct experiments to compress the layer numbers
of transformer to validate the scalability of KD-DETR,

Table 3 shows the results of KD-DETR on DAB-DETR, with backbone of ResNet-50
as teacher and ResNet-18 as student. While decreasing the number of
transformer layers will cause great degradation in the performance, KD-DETR
can significantly boost the student model. For example, when compressing the
layer number of decoder to 4, KD-DETR can still improve the performance of
student to $41.0\%$, and the student model with 2 encoder layers and 6 decoder
layers can outperform the full-scale model for $2.8\%$ mAP with 1.2x FPS
improvement.

Enc/Dec | AP | $AP_{50}$ | $AP_{75}$ | FPS | Params
---|---|---|---|---|---
6/6 | 36.2 | 56.1 | 37.9 | 76 | 31M
Ours | 41.4(+5.2) | 61.4 | 44.2 | 76 | 31M
2/6 | 36.2 | 56.3 | 38.4 | 102 | 27M
Ours | 39.0(+2.8) | 58.9 | 41.7 | 102 | 27M
6/4 | 35.7 | 55.1 | 37.8 | 79 | 28M
Ours | 41.0(+5.3) | 60.8 | 43.7 | 79 | 28M
6/2 | 31.8 | 49.8 | 33.5 | 82 | 25M
Ours | 38.9(+7.0) | 58.0 | 41.6 | 82 | 25M
4/4 | 35.8 | 54.7 | 38.1 | 91 | 24M
Ours | 38.8(+3.0) | 58.7 | 41.1 | 91 | 24M
2/2 | 29.3 | 46.6 | 37.0 | 113 | 17M
Ours | 32.8(+3.5) | 52.6 | 34.2 | 113 | 17M
Table 3: Distillation on Transformer Layers: Compressing the number of encoder
layers and decoder layers with KD-DETR

###  5.4 Ablation Study

The ablation study is conducted on MS COCO2017 with DAB-DETR with backbone of
ResNet50 as teacher and ResNet18 as student.

Analysis on the Benefit of Knowledge Distillation

In KD-DETR, we introduce a set of specialized distillation points in the
training process to the original DETR architecture. [44][16] have proven that
the number of object queries will affect the performance of model. To validate
that the benefit is from knowledge distillation rather than the increase of
object queries, we conduct an ablation experiment by applying an additional
set of object queries with the same number of distillation points.

As shown in Table 4, simply increasing the number of object queries only
achieves trivial gains, and the main contribution of boosting the model’s
performance is from the knowledge transferred from teacher with distillation
points.

Object Queries |  Distillation Points | AP | $AP_{50}$ | $AP_{75}$
---|---|---|---|---
300 | - | 36.2 | 56.1 | 37.9
300+300 | - | 37.7 | 58.0 | 41.1
300 | 300 | 40.2 | 60.7 | 42.8
Table 4: Benefit of Knowledge Distillation: Simply increasing the set of
object queries leads to trivial improvement

Analysis on the Distillation Points Sampling Strategy

A comprehensive general-to-specific distillation points sampling scheme is
introduced in this paper, including three strategies: General Sampling,
Specific Sampling, and Foreground Attention Weight.

The results are illustrated in Table 5. The general sampling strategy with
randomly initialized queries can boost the performance of student model for
$2.5\%$ mAP, while the specific sampling strategy with teacher queries
achieves an improvement of $3.8\%$. When further refining the general
distillation points with foreground attention weight, the general sampling
strategy and specific sampling strategy yield larger gains for $3.7\%$ and
$4.0\%$, respectively.

It is also important to note that the performance of general sampling with
foreground attention weight is fairly close to the specific sampling. Since
the teacher queries in specific sampling are well-optimized and concentrate
more on the foreground regions, such phenomena validate that the general
sampling with randomly initialized queries can probe the whole feature maps
evenly, and foreground attention weight can help the student focus more on the
foreground regions. In addition, the combination of the three strategies
further promotes the performance for $5.2\%$, indicating that the specific
distillation points bring more information from the concentration of teacher
model.

General | Specific | FGW | AP | $AP_{50}$ | $AP_{75}$
---|---|---|---|---|---
|  |  | 36.2 | 56.1 | 37.9
$\surd$ |  |  | 38.7 | 59.1 | 40.7
$\surd$ |  | $\surd$ | 39.9 | 60.2 | 42.5
| $\surd$ |  | 40.0 | 60.5 | 42.4
| $\surd$ | $\surd$ | 40.2 | 60.7 | 42.8
$\surd$ | $\surd$ | $\surd$ | 41.4 | 61.4 | 44.2
Table 5: Distillation with Different Distillation Points Sampling Strategies.
FGW refers to Foreground Attention Weight.

Number of General Distillation Points

We investigate the influence of the number of general distillation points by
varying the number from 10 to 900. As is shown in Table 6, the improvement of
performance is significant when increasing the distillation points from 10 to
50, while gradually saturating when continuing to increase to 300. There is
even a slight degradation in performance when further increasing the number.
These phenomena validate that the general distillation points can effectively
probe the knowledge from teacher’s attention in different regions when
sparsely distributed on the feature maps. However, dense distillation points
will introduce more noise of background information, and will be harmful for
distillation.

Number | 10 | 50 | 100 | 300 | 900
---|---|---|---|---|---
AP | 38.6 | 39.3 | 39.5 | 39.9 | 39.5
$AP_{50}$ | 58.3 | 59.2 | 59.3 | 60.2 | 60.4
$AP_{75}$ | 41.0 | 41.3 | 41.8 | 42.5 | 41.7
$AP_{S}$ | 19.5 | 20.5 | 20.3 | 20.6 | 20.7
$AP_{M}$. | 41.8 | 42.6 | 42.7 | 42.9 | 42.6
$AP_{L}$. | 55.5 | 55.6 | 56.5 | 57.0 | 56.5
Table 6: Distillation with Different Number of General Distillation Points

Visualization of the Distillation Points

(a) Teacher Model.

(b) Student Model.

(c) KD-DETR.

Figure 4: Attention Map Visualization of Specific Distillation Points: Images
from left to right are from teacher, original student and student with KD-
DETR. The corresponding instance of the distillation points are marked with
red rectangles.

Figure 5: Attention Map Visualization of General Distillation Points: Teacher
model’s attention of general distillation points focuses more on the
background regions, providing more information for student to mimic.

In order to understand what knowledge has been transferred from teacher to
student better, we visualize the attention map of different kinds of
distillation points. Fig 4 illustrate the attention of the same specific
distillation points from teacher model, student model and KD-DETR. It can be
seen that the specific distillation points focus more on the features of
foreground regions. While the attention of the original student model without
distillation is different from the teacher model, KD-DETR can align the
concentration of teacher and student models. On the contrary, the general
distillation points, as illustrated in Fig 5, concentrate more on the
background regions, and objects that are not included in the ground truth
annotations, which can provide more additional semantic information to the
student. In this way, the combination of general and specific distillation
points can provide more comprehensive knowledge.

##  6 Conclusion

In this paper, we study the compression of DETR with knowledge distillation.
We provide a thorough experimental and theoretical analysis of the failure of
knowledge distillation on DETR. Based on the analysis, we propose the first
general knowledge distillation paradigm for DETR(KD-DETR), together with a
comprehensive general-to-specific consistent distillation sampling scheme. We
demonstrate the flexibility, generalization, and extensibility of KD-DETR on
various DETR architectures, compressing both the scale of backbone and
transformer layers, significantly boosting the performance of student model.

## References

  * [1] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10925–10934, 2022.
  * [2] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: high quality object detection and instance segmentation.  IEEE transactions on pattern analysis and machine intelligence, 43(5):1483–1498, 2019.
  * [3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [4] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker.  Learning efficient object detection models with knowledge distillation.  Advances in neural information processing systems, 30, 2017.
  * [5] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang.  Dynamic head: Unifying object detection heads with attentions.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7373–7382, 2021.
  * [6] Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou.  General instance distillation for object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7842–7851, 2021.
  * [7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [8] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.  Fast convergence of detr with spatially modulated co-attention.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3621–3630, 2021.
  * [9] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2918–2928, 2021.
  * [10] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu.  Distilling object detectors via decoupled features.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2154–2164, 2021.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [12] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi.  A comprehensive overhaul of feature distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921–1930, 2019.
  * [13] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7), 2015.
  * [14] Zijian Kang, Peizhen Zhang, Xiangyu Zhang, Jian Sun, and Nanning Zheng.  Instance-conditional knowledge distillation for object detection.  Advances in Neural Information Processing Systems, 34:16468–16480, 2021.
  * [15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [16] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang.  Dn-detr: Accelerate detr training by introducing query denoising.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13619–13627, 2022.
  * [17] Quanquan Li, Shengying Jin, and Junjie Yan.  Mimicking very efficient network for object detection.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 6356–6364, 2017.
  * [18] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang.  Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.  Advances in Neural Information Processing Systems, 33:21002–21012, 2020.
  * [19] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [21] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  Dab-detr: Dynamic anchor boxes are better queries for detr.  arXiv preprint arXiv:2201.12329, 2022.
  * [22] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang.  Structured knowledge distillation for semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2604–2613, 2019.
  * [23] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang.  Conditional detr for fast training convergence.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3651–3660, 2021.
  * [24] Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin.  Libra r-cnn: Towards balanced learning for object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 821–830, 2019.
  * [25] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [26] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [27] Byungseok Roh, JaeWoong Shin, Wuhyun Shin, and Saehoon Kim.  Sparse detr: Efficient end-to-end object detection with learnable sparsity.  In International Conference on Learning Representations, 2021.
  * [28] Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris M Kitani.  Rethinking transformer-based set prediction for object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 3611–3620, 2021.
  * [29] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive representation distillation.  arXiv preprint arXiv:1910.10699, 2019.
  * [30] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627–9636, 2019.
  * [31] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
  * [32] Tao Wang, Li Yuan, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan.  Pnp-detr: Towards efficient visual analysis with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4661–4670, 2021.
  * [33] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng.  Distilling object detectors with fine-grained feature imitation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4933–4942, 2019.
  * [34] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun.  Anchor detr: Query design for transformer-based detector.  In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2567–2575, 2022.
  * [35] Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin.  Meta r-cnn: Towards general solver for instance-level low-shot learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9577–9586, 2019.
  * [36] Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang.  Cross-image relational knowledge distillation for semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12319–12328, 2022.
  * [37] Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan.  Focal and global knowledge distillation for detectors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4643–4652, 2022.
  * [38] Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan.  Masked generative distillation.  arXiv preprint arXiv:2205.01529, 2022.
  * [39] Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang.  Efficient detr: improving end-to-end object detector with dense prior.  arXiv preprint arXiv:2104.01318, 2021.
  * [40] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4133–4141, 2017.
  * [41] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  arXiv preprint arXiv:1612.03928, 2016.
  * [42] Gongjie Zhang, Shijian Lu, and Wei Zhang.  Cad-net: A context-aware detection network for objects in remote sensing imagery.  IEEE Transactions on Geoscience and Remote Sensing, 57(12):10015–10024, 2019.
  * [43] Gongjie Zhang, Zhipeng Luo, Yingchen Yu, Kaiwen Cui, and Shijian Lu.  Accelerating detr convergence via semantic-aligned matching.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 949–958, 2022.
  * [44] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  arXiv preprint arXiv:2203.03605, 2022.
  * [45] Jinnian Zhang, Houwen Peng, Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan.  Minivit: Compressing vision transformers with weight multiplexing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12145–12154, 2022.
  * [46] Linfeng Zhang and Kaisheng Ma.  Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors.  In International Conference on Learning Representations, 2020.
  * [47] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang.  Decoupled knowledge distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11953–11962, 2022.
  * [48] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Probabilistic two-stage detection.  arXiv preprint arXiv:2103.07461, 2021.
  * [49] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.