# The effects of a magnetar engine on the gamma-ray burst-associated
supernovae: Application to double-peaked SN 2006aj

Zhen-Dong Zhang  Institute of Astrophysics, Central China Normal University,
Wuhan 430079, China<EMAIL_ADDRESS>Yun-Wei Yu  Institute of Astrophysics,
Central China Normal University, Wuhan 430079, China<EMAIL_ADDRESS>Key
Laboratory of Quark and Lepton Physics (Central China Normal University),
Ministry of Education, Wuhan 430079, China  Liang-Duan Liu  Institute of
Astrophysics, Central China Normal University, Wuhan 430079, China;
<EMAIL_ADDRESS>Key Laboratory of Quark and Lepton Physics (Central China
Normal University), Ministry of Education, Wuhan 430079, China

###### Abstract

A millisecond magnetar engine has been widely suggested to exist in gamma-ray
burst (GRB) phenomena, in view of its substantial influences on the GRB
afterglow emission. In this paper, we investigate the effects of the magnetar
engine on the supernova (SN) emission which is associated with long GRBs and,
specifically, confront the model with the observational data of SN 2006aj/GRB
060218. SN 2006aj is featured by its remarkable double-peaked ultraviolet-
optical (UV-opt) light curves. By fitting these light curves, we demonstrate
that the first peak can be well accounted for by the breakout emission of the
shock driven by the magnetar wind, while the primary supernova emission is
also partly powered by the energy injection from the magnetar. The magnetic
field strength of the magnetar is constrained to be $\sim 10^{15}$ G, which is
in good agreement with the common results inferred from the afterglow emission
of long GRBs. In more detail, it is further suggested that the UV excess in
the late emission of the supernova could also be due to the leakage of the
non-thermal emission of the pulsar wind nebula (PWN), if some special
conditions can be satisfied. The consistency between the model and the SN
2006aj observation indicates that the magnetar engine is likely to be
ubiquitous in the GRB phenomena and even further intensify their connection
with the phenomena of superluminous supernovae.

Gamma-ray bursts (629) — Magnetars (992) — Supernovae (1668)

##  1 Introduction

One of the unresolved mysteries of gamma-ray burst (GRB) phenomena is the
nature of their central engines, which could be a black hole (BH) or a neutron
star (NS; Woosley, 1993; Usov, 1992; Duncan & Thompson, 1992). Since the
discovery of the shallow-decay and plateau afterglows and, in particular,
afterglow flares, it has been widely suggested that a rapidly rotating and
highly magnetized remnant NS (i.e., millisecond magnetar) should play a
crucial role in causing these afterglow features (Dai & Lu, 1998a, b; Zhang &
Mészáros, 2001; Fan & Xu, 2006; Dai et al., 2006; Yu & Dai, 2007; de Pasquale
et al., 2007; Metzger et al., 2007; Troja et al., 2007; Lyons et al., 2010; Yu
et al., 2010). It is worth to mention that millisecond magnetars could also be
able to drive an unusual type of supernovae called super-luminous supernovae
(SLSNe; at least, a part of them), which are about $10-100$ times brighter
than normal SNe (Kasen & Bildsten, 2010; Woosley, 2010; Gal-Yam, 2012; Inserra
et al., 2013; Yu et al., 2017; Liu et al., 2017). Then, in view of that the
long GRBs are also originated from the collapse of massive stars, it is of
significance to ask what imprints the post-GRB magnetars can leave in their
supernova counterparts. In other words, what we concern in this paper is,
besides the GRB afterglow features, what else signatures of the post-GRB
magnetars exist in the GRB-associated supernovae (GRBSNe).

Generally speaking, the magnetic field strengths of the magnetars in long GRBs
are usually found to be about an order of magnitude higher than those in the
SLSN cases (Yu et al., 2017). Then, the spin-down timescale of these magnetars
is usually much shorter than the diffusion timescale of the SN ejecta.
Therefore, different from SLSNe, it is expected that GRBSNe cannot be
brightened to be much more luminous than normal core-collapse supernovae and,
instead, the majority of the spin-down energy is primarily converted into the
kinetic energy of the supernova ejecta. In observations, since the discovery
of the GRB 980425/SN 1998bw association event (Galama et al., 1998; Iwamoto et
al., 1998), several tens of GRBSNe have been discovered (see, Hjorth & Bloom
2012; Cano et al. 2017; Lü et al. 2018 and the references therein). On the one
hand, the spectral features of these GRBSNe indicate they belong to broad-
lined Type Ic supernovae (SNe Ic-BL; Woosley & Bloom, 2006; Cano et al.,
2017), which hints the GRBSNe are indeed given huge kinetic energy. In the
light of statistics, it is found that the kinetic energy of GRBSNe is
clustered around $E_{\rm k}\sim(1-2)\times 10^{52}\ \rm{erg}$ (Mazzali et al.,
2014), which is in good agreement with the typical rotational energy of a
millisecond magnetar. On the other hand, although the GRBSNe are not expected
to be as luminous as SLSNe111SN 2011kl is an exception, which was associated
with ultra-long GRB 111209A and simultaneously can be classified as a SLSN
(Gendre et al., 2013; Stratta et al., 2013). This somehow indicates that SLSNe
and long GRBs could have a unified origin and there is a continuous transition
between the two phenomena (Yu et al., 2017). GRB 111209A/SN 2011kl is just a
sample on the critical line., their luminosities are still found to be
somewhat higher than those of normal supernovae and thus a more powerful
energy source is usually required. The most direct explanation of such an
energy source is a relatively high mass of $\rm{}^{56}Ni$ (Drout et al., 2011;
Lyman et al., 2016; Prentice et al., 2016; Lü et al., 2018), which could in
principle be synthesized during the shock acceleration of the SN ejecta due to
the energy injection from the magnetar (Suwa & Tominaga, 2015; Nishimura et
al., 2015). In summary, the basic features of GRBSNe can generally be
consistent with the millisecond magnetar engine model.

To be specific, the energy injection from a remnant magnetar to SN ejecta is
through a relativistic leptonic wind, which can lastingly push the ejecta and
drive a pair of shocks. While the reverse shock terminates the injecting wind
material persistently, the forward shock would quickly cross the ejecta. Then,
the emission due to these shocks could provide potential observational
signatures of the existence of the remnant magnetar. First of all, the
breakout of the forward shock emission could produce an early peak in the
supernova light curve before its primary peak, which has indeed been found in
several SLSNe recently (Kasen et al., 2016; Liu et al., 2021). So, considering
that the spin-down luminosity of GRB magnetars can be much higher than those
of the SLSNe, it is natural to expect that a more significant shock breakout
(SBO) emission peak can also appear in the GRBSN light curves. This
expectation leads us to recall the famous GRBSN: SN 2006aj, which was
associated with GRB 060218 and impressive with its double-peaked light curves
(Campana et al., 2006; Mazzali et al., 2006; Pian et al., 2006; Soderberg et
al., 2006; Kaneko et al., 2007). The primary purpose of this paper is to test
whether the early UV-optical peak of SN 2006aj can be contributed by the SBO
due to the injection of a magnetar wind.

The paper is organized as follows. The basic properties of SN 2006aj are
introduced in the next section. In Section 3, we reproduce the double-peaked
light curves of SN 2006aj by using the model including a wind-driven SBO
emission and a magnetar-aided supernova emission. In Section 4, for a more
elaborate fitting, we further consider the possible leakage of the non-thermal
emission from the shocked wind region which is termed as usual as a pulsar
wind nebula (PWN). Finally, we give a summary and conclusions in Section 5.

Figure 1: The schematic diagram of the model (not to scale).

##  2 GRB 060218/SN 2006aj

GRB 060218 was detected on 2006 February 18 by _Swift_ satellite (Gehrels et
al., 2004), which is famous by its ultra-long and X-ray prompt emission
including a blackbody component. Thanks to the long duration and the near
distance of a redshift $z\approx 0.0335$ (Cusumano et al., 2006; Mirabal et
al., 2006), its multi-wavelength afterglows and as well as the associated
supernova SN 2006aj had been monitored carefully and deeply. One of the most
mysterious features of the afterglow emission is the appearance of a big bump
in the UV-optical light curves before the normal supernova peak (Campana et
al., 2006; Ferrero et al., 2006; Modjaz et al., 2006; Sollerman et al., 2006;
Šimon et al., 2010; Emery et al., 2019). As the first impression, this UV-
optical peak is considered to be the tail of the blackbody component of the
prompt X-ray emission (Campana et al., 2006). However, Ghisellini et al.
(2007) found that it is difficult to connect the UV-optical peak with the
thermal X-rays by a naturally evolving blackbody emission. Alternatively, they
explained the overall spectrum from the optical-UV to the X-ray by a
synchrotron self-Compton model, which is however disfavored by the radio
observations (Soderberg et al., 2006; Irwin & Chevalier, 2016).

In principle, the breakout emission of a supernova bounce shock could be a
promising origin for the early UV-optical peak. However, the early peak in the
GRB 060218/SN 2006aj event is obviously too bright to be explained by a normal
supernova SBO. Then, alternatively, it was suggested that the UV-optical peak
could be contributed by a shock-heated wind (Waxman et al., 2007; Irwin &
Chevalier, 2016) or a SBO driven by a jet that was choked in a low-mass and
extended envelope around the SN ejecta (Nakar, 2015). However, both of these
scenarios require an elaborate progenitor structure. For GRB 060218/SN 2006aj,
the mass of its progenitor star is constrained to be around $\sim 20\
M_{\odot}$ (Mazzali et al., 2006), which is likely to give birth to a NS
remnant rather than a BH. Furthermore, a NS remnant is also required to
account for the production of a relatively high mass ($\sim 0.05\ M_{\odot}$)
of neutron-rich $\rm{}^{58}Ni$ (Maeda et al., 2007), which is inferred from
the corresponding emission line. Therefore, it is reasonable and necessary to
investigate whether the SBO emission driven by a magnetar wind can provide an
explanation for the UV-optical peak of SN 2006aj and, moreover, what
constraints can be obtained on the magnetar parameters.

##  3 The magnetar wind-driven SBO

###  3.1 Basic equations

Homologous expanding supernova ejecta is considered to surround a remnant NS.
The spin-down of the NS drives a relativistic wind to collide with the
supernova ejecta. As a result, a forward shock is formed to propagate into the
supernova ejecta, while the wind material is terminated by the reverse shock.
The SBO emission is produced when the forward shock is very close to the outer
surface of the supernova ejecta. Our model is depicted in Figure 1.

Here we adopt the model developed in Li & Yu (2016) to describe the dynamical
evolution and emission of the SBO and supernova. According to the energy
conservation, the dynamical evolution of the forward shock is determined by

$\displaystyle\frac{dv_{\rm sh}}{dt}$ $\displaystyle=$
$\displaystyle{1\over{M_{\rm sh}v_{\rm sh}}}\left[(L_{\rm inj}-L_{\rm
sbo}-L_{\rm sn})\right.$ (1) $\displaystyle\left.-{1\over 2}\left(v_{\rm
sh}^{2}-v_{\rm ej}^{2}\right)\frac{dM_{\rm sh}}{dt}-\frac{dU}{dt}\right],$
$\displaystyle\frac{dM_{\rm sh}}{dt}$ $\displaystyle=$ $\displaystyle 4\pi
R_{\rm sh}^{2}\left(v_{\rm sh}-v_{\rm ej}\right)\rho_{\rm ej},$ (2)

where $v_{\rm sh}$ and $R_{\rm sh}$ are the velocity and radius of the forward
shock, $M_{\rm sh}$ is the total mass of the shocked ejecta, $L_{\rm inj}$ is
the power of the energy sources, $L_{\rm sbo}$ and $L_{\rm sn}$ are the
luminosities of the SBO emission and supernova emission, $v_{\rm ej}$ and
$\rho_{\rm ej}$ are the velocity and density of the ejecta material just in
front of the shock. $U$ is the total internal energy of the supernova ejecta,
which can be divided into the shock-accumulated part ($U_{\rm sh}$) and the
remaining part ($\tilde{U}=U-U_{\rm sh}$). The evolution of these internal
energies can be coupled with the dynamical quantities by the following two
equations:

$\displaystyle\frac{dU_{\rm sh}}{dt}$ $\displaystyle=$ $\displaystyle H_{\rm
sh}-{P_{\rm sh}}\frac{d(\epsilon V)}{dt}-L_{\rm sbo},$ (3)
$\displaystyle\frac{d\tilde{U}}{dt}$ $\displaystyle=$ $\displaystyle L_{\rm
inj}-\tilde{P}\frac{dV}{dt}-L_{\rm sn},$ (4)

where the shock heating rate is given by $H_{\rm sh}={1\over 2}\left[v_{\rm
sh}-v_{\rm ej}(r_{\rm sh},t)\right]^{2}(dM_{\rm sh}/dt)$, the pressures read
$P_{\rm sh}=U_{\rm sh}/(3\epsilon V)$ and $\tilde{P}=\tilde{U}/3V$, and $V$ is
the volume of the shocked ejecta. The parameter $\epsilon$ represents the
volume fraction of the shock heating region in the entire shocked ejecta,
which is considered to be extremely small.

Finally, the emission luminosity of the SBO and the supernova can be given by
(Li & Yu, 2016):

$L_{\rm sbo}=\frac{R_{\rm max}^{2}U_{\rm sh}c}{\epsilon R_{\rm sh}^{3}+R_{\rm
max}^{3}-R_{\rm sh}^{3}}\left[\frac{1-e^{-(\epsilon\tau_{\rm sh}+\tau_{\rm
un})}}{\epsilon\tau_{\rm sh}+\tau_{\rm un}}\right],$ (5)

and

$L_{\rm sn}=\frac{\tilde{U}c}{R_{\rm max}}\left[\frac{1-e^{-(\tau_{\rm
sh}+\tau_{\rm un})}}{\tau_{\rm sh}+\tau_{\rm un}}\right],$ (6)

respectively, where $\tau_{\rm sh}={3\kappa M_{\rm ej}}/{4\pi R_{\rm sh}^{2}}$
and $\tau_{\rm un}=\int_{R_{\rm sh}}^{R_{\max}}\kappa\rho_{\rm ej}dr$ are the
optical depths of the shocked and unshocked ejecta, and $\kappa=0.1\rm
cm^{2}g^{-1}$ is the opacity. In order to calculate the multi-band light
curves, we simply assume a blackbody spectrum for the SBO and supernova
emission, which has a temperature of

$T_{\rm{eff}}=\left(\frac{L_{\rm e}}{4\pi\sigma_{\rm SB}R_{\rm
ph}^{2}}\right)^{1/4},$ (7)

where $L_{\rm e}=L_{\rm sbo}+L_{\rm sn}$, $\sigma_{\rm SB}$ is the Stephan-
Boltzmann constant, and $R_{\rm ph}$ is the photosphere radius where the
electron-scattering optical depth satisfies

$\tau_{\rm ph}=\int_{R_{\rm ph}}^{R_{\max}}\kappa\rho_{\rm ej}dr={2\over 3}.$
(8)

If $R_{\rm ph}<R_{\rm sh}$, we simply set $R_{\rm ph}=R_{\rm sh}$.

###  3.2 Physical inputs

Here we consider that GRBSNe could be powered by the radioactivity of 56Ni and
the spin-down of a remnant NS. The heating power due to the radioactive decays
can be written as (Arnett, 1982; Valenti et al., 2008):

$L_{\rm r}(t)=M_{\rm Ni}\left[(\epsilon_{\rm Ni}-\epsilon_{\rm
Co})e^{-t/\tau_{\rm Ni}}+\epsilon_{\rm Co}e^{-t/\tau_{\rm Co}}\right],$ (9)

where $M_{\rm Ni}$ is the total mass of 56Ni, $\epsilon_{\rm Ni}=3.90\times
10^{10}\ \rm{erg\ s^{-1}\ g^{-1}}$, $\epsilon_{\rm Co}=6.78\times 10^{9}\
\rm{erg\ s^{-1}\ g^{-1}}$, $\tau_{\rm Ni}=8.77\ \rm{days}$, and $\tau_{\rm
Co}=111.3\ \rm{days}$. Meanwhile, the spin-down power of the remnant NS by the
dipole radiation is written as:

$L_{\rm sd}(t)=L_{\rm sd,0}\left(1+\frac{t}{t_{\rm sd}}\right)^{-2},$ (10)

where $L_{\rm sd,0}$ is the initial value of the luminosity and $t_{\rm sd}$
represents the spin-down timescale. In our calculations, we take $L_{\rm
inj}=L_{\rm r}+L_{\rm sd}$.

For the SN ejecta, we take a density distribution as (Liu et al., 2018):

$\rho_{\rm ej}(x,t)=\rho_{\rm 0}\eta(x)\left[\frac{R_{\rm max,0}-R_{\rm
min,0}}{R_{\rm max}(t)-R_{\rm min}(t)}\right]^{3},$ (11)

where $R_{\rm min}(t)=R_{\rm min,0}+v_{\rm min}t$ and $R_{\rm max}(t)=R_{\rm
max,0}+v_{\rm max}t$ are the evolving minimum and maximum radius of the ejecta
with $v_{\rm min}$ and $v_{\max}$ being the corresponding velocities and the
subscript $\mathrm{0}$ represents their initial values. The density profile is
described by a broken power-law as

$\eta(x)=\left\\{\begin{array}[]{rcl}\left(\frac{x}{x_{0}}\right)^{-n},&&{x_{0}\leqslant
x\leqslant 1},\\\ \left(\frac{x}{x_{0}}\right)^{-\sigma},&&{0\leqslant
x<x_{0}},\\\ \end{array}\right.$ (12)

with

$x=\frac{R-R_{\rm min}}{R_{\rm max}-R_{\rm min}},$ (13)

where the power-law indices satisfy $n>5$ and $\sigma<3$. Denoting the mass
and kinetic energy of the ejecta by $M_{\rm ej}$ and $E_{\rm k}$, we can
obtain (Vinkó et al., 2004; Liu et al., 2018)

$\rho_{\rm 0}={M_{\rm ej}\over 4\pi I_{\rm m}R_{\rm max,0}^{3}},$ (14)

and

$v_{\max}=\sqrt{\frac{2E_{\rm k}I_{\rm m}}{M_{\rm ej}I_{\rm k}}},$ (15)

where

$\displaystyle I_{\rm m}$ $\displaystyle=$
$\displaystyle\frac{1}{3-\sigma}x_{0}^{3}+\frac{1}{3-n}(x_{0}^{n}-x_{0}^{3}),$
(16) $\displaystyle I_{\rm k}$ $\displaystyle=$
$\displaystyle\frac{1}{5-\sigma}x_{0}^{5}+\frac{1}{5-n}(x_{0}^{n}-x_{0}^{5}),$
(17)

and $x_{0}=0.1$ is taken in our calculations. The values of $R_{\rm min,0}$
and $v_{\rm min}$, as they are sufficiently small, will not significantly
influence the following calculation results.

Figure 2: Posterior distributions of parameters for the fittings of SN 2006aj.

###  3.3 Results and discussion

Parameter | Value | Best-fit | Range
---|---|---|---
$M_{\rm ej}\ (M_{\odot})$ | $3.62_{-0.10}^{+0.10}$ | 3.64 | (1,10)
$\log_{10}E_{\rm k}\ (\rm{erg})$ | $50.80_{-0.04}^{+0.04}$ | 50.82 | (50,52)
$\log_{10}R_{\rm{max,0}}\ (\rm{cm})$ | $13.79_{-0.02}^{+0.01}$ | 13.80 | (11,13.8)
$\sigma$ | $1.38_{-0.26}^{+0.34}$ | 1.09 | (1,3)
$n$ | $6.24_{-0.16}^{+0.12}$ | 6.38 | (5,10)
$\log_{10}t_{\rm sd}\ (\rm{day})$ | $-2.18_{-0.02}^{+0.02}$ | -2.21 | (-4,0)
$\log_{10}L_{\rm sd,0}\ (\rm{erg/s})$ | $49.54_{-0.02}^{+0.02}$ | 49.57 | (47,50)
$t_{\rm gap}\ (\rm{day})$ | $0.27_{-0.01}^{+0.01}$ | 0.27 | (0,2)
$M_{\rm Ni}\ (M_{\odot})$ | 0.1 (fixed) |  |
$E_{\rm sd}\ (\rm{erg})$ |  | $1.988\times 10^{52}$ |
$P_{0}\ (\rm{ms})$ |  | 1.003 |
$B_{\rm p}\ (10^{14}\rm{G})$ |  | 9.694 |

Table 1: Parameters estimated from the modeling of SN 2006aj by MCMC-sampling.

Figure 3: The best fit of the multi-wavelength double-peaked light curves of
SN 2006aj by using the model including a magnetar wind-driven SBO emission and
a magnetar-aided supernova emission, for the parameter values listed in Table
1. The left and right panels are shown in a linear and logarithmic-scale
x-axis, respectively. The observational data are taken from Nakar (2015) and
Campana et al. (2006). Figure 4: The bolometric light curves of the SBO and
supernova emission in comparison with the luminosity evolution of the heating
powers. The parameter values are same to Figure 3. The zero-time in this
figure is $t_{\rm mag,0}$.

By substituting the heating powers and the ejecta profile into the model
equations, we obtain the temporal evolution of $L_{\rm sbo}$ and $L_{\rm sn}$
and then fit the multi-band light curves of SN 2006aj. Then, the model
parameters are constrained by MCMC sampling with emcee package (Foreman-Mackey
et al., 2013). The parameter constraints with $1\sigma$ confidence level are
presented in Figure 2 and Table 1. The corresponding fitting results are shown
in Figure 3 and, furthermore, we plot the bolometric light curve for the best-
fit parameters in Figure 4. As shown, the peak bolometric luminosity of the
SBO emission $L_{\rm sbo}$ could reach $\sim 10^{46}\ \rm{erg\ s^{-1}}$, which
is concentrated within the UV band ($\sim 60$ eV). The X-ray ($0.3$ keV) flux
of the SBO emission is about $0.002\mu$Jy, which can be easily submerged by
the bright X-ray afterglow emission of the GRB of $\sim 0.1\mu$Jy. The
Rayleigh-Jeans tail of the SBO emission produced the first peak in the UV-
optical light curves of SN 2006aj, which can be much higher than or comparable
to the luminosity of the main supernova peak. This result is very different
from the situation of SLSNe, where the wind-driven SBO emission is always
dimmer than the supernova emission (Kasen et al., 2016; Liu et al., 2021).
This difference can be understood by realizing that the remnant magnetars in
long GRBs and SLSNe can distinguish each other by their different magnetic
fields (Yu et al., 2017). According to the obtained value of $L_{\rm sd,0}$
and $t_{\rm sd}$ for SN 2006aj, the magnetic field strength and initial spin
period of its remnant magnetar can be derived to $B_{\rm p}=1.0\times 10^{15}$
G and $P_{0}=1.0$ ms, which are both typical for long GRB magnetars as
revealed from the plateau afterglow fittings (Lü & Zhang, 2014; Metzger et
al., 2015; Zou et al., 2021).

The mass of the SN ejecta is constrained to be $\sim 3.6M_{\odot}$, which is
somewhat higher than the result ($\sim 2M_{\odot}$) obtained by Mazzali et al.
(2006). Meanwhile, as a result of the injection of the spin-down energy, the
velocity of the supernova ejecta can finally approach to (Li & Yu, 2016)

$v_{\rm ej}\sim\left({2L_{\rm sd,0}t_{\rm sd}\over M_{\rm
ej}}\right)^{1/2}=23,000~{}\rm km~{}s^{-1},$ (18)

which is also roughly consistent with the velocity inferred by Mazzali et al.
(2006) from the supernova spectra. Nevertheless, the difference is that
Mazzali et al. (2006) considered only a small fraction of ejecta has such a
high velocity, but our result indicates this high velocity could be an average
value of the entire ejecta, which could correspond to an uncommon velocity
profile for SN 2006aj (Nakar, 2015). For the adopted opacity $\kappa=0.1\rm
cm^{2}g^{-1}$, the obtained mass and velocity of the supernova ejecta
determine a photon diffusion timescale as

$t_{\rm diff}\sim\left({\kappa M_{\rm ej}\over 4\pi cv_{\rm
ej}}\right)^{1/2}=10.5\rm~{}day,$ (19)

which is much longer than the obtained spin-down timescale of the magnetar.
Therefore, the injected energy is primarily converted into the kinetic energy
of the ejecta rather than contributing to the supernova emission, as supposed
in Eq. (18). This is the basic reason leading the GRBSN to be different from
SLSNe (Yu et al., 2017).

Nevertheless, Figure 4 also shows that, although the spin-down luminosity had
decreased significantly at the diffusion timescale, the contribution of the
magnetar to the supernova emission can still be comparable to and even higher
than the radioactive power for the adopted $M_{\rm Ni}=0.1M_{\odot}$. In the
traditional radioactivity model for GRBSNe, a relatively high mass of
$\rm{}^{56}Ni\gtrsim 0.2\ M_{\odot}$ is usually required to explain GRBSNe (Lü
et al., 2018), since their luminosities are usually somewhat higher than
normal core-collapse supernovae (Lyman et al., 2016; Prentice et al., 2016).
However, in this scenario, how such a huge amount of $\rm{}^{56}Ni$ can be
synthesized is an open question (see, Maeda & Tominaga 2009; Suwa & Tominaga
2015; Nishimura et al. 2015 and the references therein). In the magnetar
engine model, in principle, the $\rm{}^{56}Ni$ synthesization can be enhanced
by the wind-driven shock (Suwa & Tominaga, 2015), which however usually
requires a too high magnetic field to be consistent with the results inferred
from the GRB afterglows. Therefore, as shown by our result, another possible
explanation can be that the GRBSN emission is powered by substantial hybrid
energy sources including the radioactivity and the magnetar engine, as also
suggested by Wang et al. (2016, 2017) for SNe Ic-BL. In this case, we only
need to invoke a relatively normal nickel mass.

Finally, two special parameter values are worth to be discussed. One is the
initial value of the maximum radius of the ejecta which is found to be close
to $R_{\rm max,0}\sim 10^{14}$ cm, and another one is the gap $t_{\rm gap}$
between the time zeros of our calculation and the GRB trigger. The relatively
large value of $R_{\rm max,0}$, which had also been realized in previous
studies (Chevalier & Fransson, 2008; Nakar, 2015; Irwin & Chevalier, 2016),
may indicate the progenitor star of SN 2006aj ejecta has a special extended
structure. Alternatively, this may hint that it needs to spend a period of
time to become a magnetar for the remnant NS (e.g., Cheng & Yu 2014). So,
during this period, the supernova ejecta can expand significantly. No matter
whether this time gap is true or not, a time gap between the magnetar
formation and the GRB trigger is definitely necessary for reproducing the
rising phase of the SBO peak of the light curves. Specifically, the GRB
trigger (i.e., the time zero of the data) is found to lag behind the magnetar
formation (i.e., the time zero of our calculation) by a time of

$t_{\rm gap}=t_{\rm GRB,0}-t_{\rm mag,0}=0.27\rm~{}day.$ (20)

The possible existence of a time lag between the associated supernova and GRB
had been previously noticed by some authors. For example, Fryer & Mészáros
(2003) expected this time lag to be at least larger than $\sim 30\
\mathrm{s}$, Norris & Bonnell (2004) suggested to be about $\pm 0.5\
\mathrm{day}$, and Iwamoto et al. (1998) constrained it to be within $-0.7/+2\
\mathrm{days}$ for GRB 980425/SN 1998bw. Our result demonstrates that the
observations of the SBO rise can provide a practicable method to measure this
time gap, which is crucial for testing the GRB mechanisms. However, in most
cases, the UV-optical breakout can be easily overwhelmed by the GRB afterglow
emission. For GRB 060218/SN 2006aj, its relatively faint afterglow and near
distance make it possible to probe the detail of the UV-optical peak at very
early time.

##  4 Possible non-thermal PWN emission

Figure 5: The same as Figure 3 but with an extra non-thermal PWN emission
component shown in dashed lines, the thermal component is shown in dotted
lines.

Although the fittings presented in the previous section account for the basic
features of SN 2006aj, the observational data exceed the theoretical curves
significantly for relatively high frequencies, especially in the UV bands.
Such a discrepancy between the model and data suggests that the spectra of SN
2006aj deviate from an ideal black body in the UV bands, which becomes more
serious at later time. This UV excess could be a result of frequency-dependent
radiative transfer and could also indicate an extra non-thermal emission
component. In the magnetar engine model, such a non-thermal emission is most
likely to originate from the relativistic wind of the magnetar behind the
supernova ejecta. When the wind collides with the supernova ejecta, a
termination shock can be driven and propagate into the wind. As usual, we
termed the shocked wind region as a pulsar wind nebula (PWN), which can in
principle contribute to a significant non-thermal emission after the supernova
ejecta gradually becomes transparent (Kotera et al., 2013; Yu et al., 2019; Wu
et al., 2021). Then, in this section, we try to refit the observational data
of SN 2006aj by invoking such a non-thermal PWN emission.

We calculate the PWN emission by using the model presented in Yu et al.
(2019), where the most crucial step is to determine the comoving density of
the internal energy of the PWN as

$e^{\prime}_{\rm pwn}=4\Gamma_{\rm ts}^{\prime\ 2}n^{\prime}_{\rm
w}m_{e}c^{2},$ (21)

where $\Gamma^{\prime}_{\rm ts}=\frac{1}{2}\Gamma_{\rm w}$ is the Lorentz
factor of the termination shock relative to the unshocked wind of a Lorentz
factor $\Gamma_{\rm w}$ and

$n^{\prime}_{\rm w}=\frac{L_{\rm sd}}{4\pi R_{\rm ts}^{2}\Gamma_{\rm
w}^{2}m_{e}c^{3}}$ (22)

is the comoving number density of the unshocked wind which is considered to
purely consist of electron and positron pairs. Here, the luminosity of the
relativistic wind is simply given by the spin-down luminosity of the magnetar,
and $R_{\rm ts}$ is the radius of the termination shock. In our calculation,
the value of $e^{\prime}_{\rm pwn}$ can be directly obtained from the
mechanical equilibrium between the PWN and the supernova ejecta, i.e.,

$e^{\prime}_{\rm pwn}=e^{\prime}_{\rm ej}={3\tilde{U}\over 4\pi R_{\rm
sh}^{3}}.$ (23)

Then, by introducing equipartition factors $\epsilon_{\rm B}$ and
$\epsilon_{\rm e}$ for the magnetic field and electrons in the PWN, and
assuming the electron energy distribution with a power-law index $p$. Thus we
can calculate the synchrotron emission of these electrons (see Sari et al.
1998; Yu et al. 2019 for details).

For a PWN synchrotron luminosity of $L_{\nu,\rm pwn}$, the luminosity of the
leaking non-thermal component appearing in the supernova emission can be given
by

$L_{\nu,\rm nth}=L_{\nu,\rm pwn}{\rm e}^{-\tau_{\nu}}.$ (24)

Simultaneously, the power absorbed by the supernova ejecta can be written as

$L_{\rm sd,inj}=\int L_{\nu,\rm{pwn}}(1-e^{-\tau_{\nu}})d\nu,$ (25)

which would be used in the previous section for calculating the dynamical
evolution and emission of the supernova ejecta. Here, the optical depth
includes both contributions from the shocked and unshocked supernova ejecta,
which reads

$\displaystyle\tau_{\nu}$ $\displaystyle=$ $\displaystyle\tau_{\rm
sh,\nu}+\tau_{\rm un,\nu}$ (26) $\displaystyle=$
$\displaystyle\frac{3\kappa_{\nu}M_{\rm ej}}{4\pi R_{\rm sh}^{2}}+\int_{R_{\rm
sh}}^{R_{\max}}\kappa_{\nu}\rho_{\rm ej}dr,$

where the frequency dependence of the opacity is taken into account because
the non-thermal PWN emission can appear in a wide range of electromagnetic
bands. By fitting the numerical result presented in Kotera et al. (2013), Yu
et al. (2019) suggested

$\displaystyle\kappa_{\nu}$ $\displaystyle=$ $\displaystyle
C_{1}\left(\frac{h\nu}{1\
\rm{keV}}\right)^{-3}+C_{2}\left[\left(\frac{h\nu}{50\
\rm{keV}}\right)^{-0.36}\right.$ (27)
$\displaystyle\left.+\left(\frac{h\nu}{50\
\rm{keV}}\right)^{0.65}\right]^{-1}+C_{3}$

for $h\nu\gtrsim 0.1\ \rm{keV}$, where the coefficients are $C_{1}=2.0\
\rm{cm^{2}g^{-1}}$, $C_{2}=0.7\ \rm{cm^{2}g^{-1}}$, and $C_{3}=0.012\
\rm{cm^{2}g^{-1}}$.

Figure 6: Posterior distributions of parameters for the fittings of SN 2006aj with the model in Section 4. Parameter | Value | Best-fit | Range
---|---|---|---
$M_{\rm ej}\ (M_{\odot})$ | $2.05_{-0.14}^{+0.12}$ | 1.92 | (1,10)
$\log_{10}E_{\rm k}\ (\rm{erg})$ | $50.88_{-0.07}^{+0.05}$ | 50.89 | (50,52)
$\log_{10}R_{\rm{max,0}}\ (\rm{cm})$ | $13.77_{-0.05}^{+0.02}$ | 13.78 | (11,13.8)
$\sigma$ | $1.79_{-0.44}^{+0.39}$ | 1.78 | (1,3)
$n$ | $5.97_{-0.29}^{+0.26}$ | 6.04 | (5,11)
$\log_{10}t_{\rm sd}\ (\rm{day})$ | $-2.50_{-0.06}^{+0.06}$ | -2.56 | (-4,0)
$\log_{10}L_{\rm sd,0}\ (\rm{erg/s})$ | $49.85_{-0.06}^{+0.06}$ | 49.92 | (47,50)
$t_{\rm gap}\ (\rm{day})$ | $0.16_{-0.01}^{+0.01}$ | 0.14 | (0,2)
$\log_{10}\Gamma_{\rm w}$ | $4.58_{-0.29}^{+0.26}$ | 4.69 | (2,7)
$\log_{10}\epsilon_{\rm B}$ | $-2.80_{-0.54}^{+0.74}$ | -2.46 | (-6,-1)
$p$ | $2.20_{-0.03}^{+0.03}$ | 2.16 | (2,3)
$\log_{10}{t_{\rm tr}}\ (\rm{s})$ | $4.17_{-0.05}^{+0.06}$ | 4.09 | (4,6)
$t_{\rm sd,1}\ (\rm{day})$ | $47.17_{-3.65}^{+2.10}$ | 49.10 | (1,50)
$C_{\rm\tau}$ | $0.01_{-0.00}^{+0.00}$ | 0.01 | (0,1)
$M_{\rm Ni}\ (M_{\odot})$ | 0.1 (fixed) |  |
$E_{\rm sd}\ (\rm{erg})$ |  | $1.979\times 10^{52}$ |
$p_{\rm sd,0}\ (\rm{ms})$ |  | 1.022 |
$B_{\rm p}\ (10^{14}\rm{G})$ |  | 15.058 |

Table 2: Parameters estimated from modeling the double-peaked light curve of
SN 2006aj with the model in Section 4 by MCMC-sampling. Where $t_{\rm sd,1}$
represents the spin-down timescale after the magnetic dipole radiation
suppression.
Figure 7: The temporal evolution of the spin-down luminosity of the mangetar,
which is required by the consistency of the non-thermal PWN emission with the
observational data of SN 2006aj. The zero-time in this figure is $t_{\rm
mag,0}$.

Combining Eqs. (5), (6) and (24), we refit the light curves of SN 2006aj and
present the result in Figure 5. The corresponding parameter values are shown
in Table 2 and Figure 6. As shown, a plausible fit can be obtained if two ad
hoc assumptions are made, where the excesses in the UV bands are successfully
accounted for and the supernova’s rising phase is better fitted. On the one
hand, it is assumed the magnetic dipole radiation of the magnetar is suddenly
suppressed at a time of $t_{\rm tr}$ due to an unknown reason, the luminosity
evolution of which is shown in Figure 7. Although it seems unusual, such a
sudden change has been previously suggested to explain the overall evolution
of short GRB 170817A and its associated kilonova AT 2017gfo (Yu et al., 2018).
In principle, the suppression of the magnetic dipole radiation could be caused
by a fallback accretion that buries the surface dipolar magnetic field of the
magnetar (Li et al., 2021). On the other hand, an artificial suppression on
the optical depth $\tau_{\nu,\mathrm{sup}}=C_{\tau}\tau_{\nu}$ should be
invoked in Eq. (24) and (25), so that the PWN emission can leak not too late.
This hints the first leaked PWN emission could come from a radius much larger
than $R_{\rm ts}$ and very close to the surface of the supernova ejecta. Such
a situation could appear if the supernova ejecta is highly anisotropic, which
seems not strange in the extreme stellar explosive phenomena such as GRBs. In
addition, as illustrated in Figure 1, the early leakage of the PWN emission
could also be due to turbulence of the interface between the PWN region and
the supernova ejecta, which is subject to the Rayleigh-Taylor instabilities
(Bucciantini et al., 2004; Chevalier, 2005; Slane, 2017). In this case, a
remarkable fraction of the PWN material (the ‘mushroom finger’) can penetrate
into the supernova ejecta deeply.

##  5 Conclusion and discussion

In this paper, we revisit the famous double-peaked GRBSN, SN 2006aj, to
investigate its consistency with a millisecond magnetar engine. The effects of
the magnetar on the supernova emission can be in principle shown in the
following three aspects. (i) The powerful magnetar wind can drive a forward
shock crossing the supernova ejecta, which can accumulate a remarkable amount
of internal energy in the region immediately behind the shock front. As the
ejecta material outside the shock becomes transparent, the shock breaks out
and produces a transient emission. The Rayleigh-Jeans tail of this SBO
emission can well account for the first peaks in the UV-optical light curves
of SN 2006aj, which is the most remarkable feature of this supernova. (ii)
Besides the traditional radioactive power, the primary peak of the supernova
emission can also be powered by the energy injection from the magnetar wind,
although the spin-down luminosity had been reduced significantly at $t_{\rm
diff}$ which is much longer than $t_{\rm sd}$. The extra power leads the GRBSN
to be somewhat more luminous than normal core-collapse supernovae, even though
not too many 56Ni is synthesized. (iii) As a result of the interaction between
the magnetar wind and the supernova ejecta, the energy carried by the wind
would be first converted into PWN emission, which can be effectively absorbed
by the supernova ejecta at early times. Nevertheless, as the expansion of the
supernova ejecta, the PWN emission can finally leak from the ejecta to be
observed, which then contributes a non-thermal component to the late supernova
emission. This component could provide an explanation for the UV excess in the
SN 2006aj emission.

The good agreement between the theoretical expectation and observational
features demonstrates that the central engine of GRB 060218/SN 2006aj is very
likely to be a magnetar, with a magnetic field of $B_{\rm p}\sim 10^{15}$ G.
This magnetic field strength is a typical value for usual GRB magnetars as
inferred from afterglow emission, which indicates the self-consistence of the
magnetar engine model. Meanwhile, this magnetic field strength is also the key
to understand the difference between the GRBSNe and the SLSNe. In addition,
two ad hoc treatments in our calculations need to be emphasized. On the one
hand, it is strongly suggested that time gaps can exist between the supernova
explosion, the magnetar formation, and the GRB trigger, which can offer
crucial information for exploring the GRB mechanisms. In the future, besides
to find more double-peaked GRBSNe like SN 2006aj, multi-messenger observations
including gravitational wave and neutrino emission would make us be able to
more directly measure the time lags (Fryer et al., 2002; Blondin et al., 2003;
Mészáros et al., 2004; Yu et al., 2008). On the other hand, the geometry of
the GRBSN ejecta could be very different from a simple isotropic structure and
the interface between the magnetar wind and the ejecta could also seriously
deviate from a simple spherical surface (e.g., due to the Rayleigh-Taylor
instabilities). Only invoking these complexities, the non-thermal component
appearing in the SN 2006aj emission at the late time can be explained by the
leakage of the PWN emission.

This work is supported by the National Key R&D Program of China
(2021YFA0718500), the China Manned Spaced Project (CMS-CSST-2021-A12), the
National SKA program of China (2020SKA0120300), and the National Natural
Science Foundation of China (Grant No. 11822302 and 11833003).

## References

  * Arnett (1982) Arnett, W. D. 1982, ApJ, 253, 785, doi: 10.1086/159681
  * Blondin et al. (2003) Blondin, J. M., Mezzacappa, A., & DeMarino, C. 2003, ApJ, 584, 971, doi: 10.1086/345812
  * Bucciantini et al. (2004) Bucciantini, N., Amato, E., Bandiera, R., Blondin, J. M., & Del Zanna, L. 2004, A&A, 423, 253, doi: 10.1051/0004-6361:20040360
  * Campana et al. (2006) Campana, S., Mangano, V., Blustin, A. J., et al. 2006, Nature, 442, 1008, doi: 10.1038/nature04892
  * Cano et al. (2017) Cano, Z., Wang, S.-Q., Dai, Z.-G., & Wu, X.-F. 2017, Advances in Astronomy, 2017, 8929054, doi: 10.1155/2017/8929054
  * Cheng & Yu (2014) Cheng, Q., & Yu, Y.-W. 2014, ApJ, 786, L13, doi: 10.1088/2041-8205/786/2/L13
  * Chevalier (2005) Chevalier, R. A. 2005, ApJ, 619, 839, doi: 10.1086/426584
  * Chevalier & Fransson (2008) Chevalier, R. A., & Fransson, C. 2008, ApJ, 683, L135, doi: 10.1086/591522
  * Cusumano et al. (2006) Cusumano, G., Moretti, A., Tagliaferri, G., Kennea, J., & Burrows, D. 2006, GRB Coordinates Network, 4786, 1
  * Dai & Lu (1998a) Dai, Z. G., & Lu, T. 1998a, A&A, 333, L87.  https://arxiv.org/abs/astro-ph/9810402
  * Dai & Lu (1998b) —. 1998b, MNRAS, 298, 87, doi: 10.1046/j.1365-8711.1998.01681.x
  * Dai et al. (2006) Dai, Z. G., Wang, X. Y., Wu, X. F., & Zhang, B. 2006, Science, 311, 1127, doi: 10.1126/science.1123606
  * de Pasquale et al. (2007) de Pasquale, M., Oates, S. R., Page, M. J., et al. 2007, MNRAS, 377, 1638, doi: 10.1111/j.1365-2966.2007.11724.x
  * Drout et al. (2011) Drout, M. R., Soderberg, A. M., Gal-Yam, A., et al. 2011, ApJ, 741, 97, doi: 10.1088/0004-637X/741/2/97
  * Duncan & Thompson (1992) Duncan, R. C., & Thompson, C. 1992, ApJ, 392, L9, doi: 10.1086/186413
  * Emery et al. (2019) Emery, S. W. K., Page, M. J., Breeveld, A. A., et al. 2019, MNRAS, 484, 5484, doi: 10.1093/mnras/stz373
  * Fan & Xu (2006) Fan, Y.-Z., & Xu, D. 2006, MNRAS, 372, L19, doi: 10.1111/j.1745-3933.2006.00217.x
  * Ferrero et al. (2006) Ferrero, P., Kann, D. A., Zeh, A., et al. 2006, A&A, 457, 857, doi: 10.1051/0004-6361:20065530
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. 2013, PASP, 125, 306, doi: 10.1086/670067
  * Fryer et al. (2002) Fryer, C. L., Holz, D. E., & Hughes, S. A. 2002, ApJ, 565, 430, doi: 10.1086/324034
  * Fryer & Mészáros (2003) Fryer, C. L., & Mészáros, P. 2003, ApJ, 588, L25, doi: 10.1086/375412
  * Gal-Yam (2012) Gal-Yam, A. 2012, Science, 337, 927, doi: 10.1126/science.1203601
  * Galama et al. (1998) Galama, T. J., Vreeswijk, P. M., van Paradijs, J., et al. 1998, Nature, 395, 670, doi: 10.1038/27150
  * Gehrels et al. (2004) Gehrels, N., Chincarini, G., Giommi, P., et al. 2004, ApJ, 611, 1005, doi: 10.1086/422091
  * Gendre et al. (2013) Gendre, B., Stratta, G., Atteia, J. L., et al. 2013, ApJ, 766, 30, doi: 10.1088/0004-637X/766/1/30
  * Ghisellini et al. (2007) Ghisellini, G., Ghirlanda, G., & Tavecchio, F. 2007, MNRAS, 375, L36, doi: 10.1111/j.1745-3933.2006.00270.x
  * Hjorth & Bloom (2012) Hjorth, J., & Bloom, J. S. 2012, The Gamma-Ray Burst - Supernova Connection, 169–190
  * Inserra et al. (2013) Inserra, C., Smartt, S. J., Jerkstrand, A., et al. 2013, ApJ, 770, 128, doi: 10.1088/0004-637X/770/2/128
  * Irwin & Chevalier (2016) Irwin, C. M., & Chevalier, R. A. 2016, MNRAS, 460, 1680, doi: 10.1093/mnras/stw1058
  * Iwamoto et al. (1998) Iwamoto, K., Mazzali, P. A., Nomoto, K., et al. 1998, Nature, 395, 672, doi: 10.1038/27155
  * Kaneko et al. (2007) Kaneko, Y., Ramirez-Ruiz, E., Granot, J., et al. 2007, ApJ, 654, 385, doi: 10.1086/508324
  * Kasen & Bildsten (2010) Kasen, D., & Bildsten, L. 2010, ApJ, 717, 245, doi: 10.1088/0004-637X/717/1/245
  * Kasen et al. (2016) Kasen, D., Metzger, B. D., & Bildsten, L. 2016, ApJ, 821, 36, doi: 10.3847/0004-637X/821/1/36
  * Kotera et al. (2013) Kotera, K., Phinney, E. S., & Olinto, A. V. 2013, MNRAS, 432, 3228, doi: 10.1093/mnras/stt680
  * Li & Yu (2016) Li, S.-Z., & Yu, Y.-W. 2016, ApJ, 819, 120, doi: 10.3847/0004-637X/819/2/120
  * Li et al. (2021) Li, S.-Z., Yu, Y.-W., Gao, H., & Zhang, B. 2021, ApJ, 907, 87, doi: 10.3847/1538-4357/abcc70
  * Liu et al. (2021) Liu, L.-D., Gao, H., Wang, X.-F., & Yang, S. 2021, ApJ, 911, 142, doi: 10.3847/1538-4357/abf042
  * Liu et al. (2017) Liu, L.-D., Wang, S.-Q., Wang, L.-J., et al. 2017, ApJ, 842, 26, doi: 10.3847/1538-4357/aa73d9
  * Liu et al. (2018) Liu, L.-D., Zhang, B., Wang, L.-J., & Dai, Z.-G. 2018, ApJ, 868, L24, doi: 10.3847/2041-8213/aaeff6
  * Lü et al. (2018) Lü, H.-J., Lan, L., Zhang, B., et al. 2018, ApJ, 862, 130, doi: 10.3847/1538-4357/aacd03
  * Lü & Zhang (2014) Lü, H.-J., & Zhang, B. 2014, ApJ, 785, 74, doi: 10.1088/0004-637X/785/1/74
  * Lyman et al. (2016) Lyman, J. D., Bersier, D., James, P. A., et al. 2016, MNRAS, 457, 328, doi: 10.1093/mnras/stv2983
  * Lyons et al. (2010) Lyons, N., O’Brien, P. T., Zhang, B., et al. 2010, MNRAS, 402, 705, doi: 10.1111/j.1365-2966.2009.15538.x
  * Maeda & Tominaga (2009) Maeda, K., & Tominaga, N. 2009, MNRAS, 394, 1317, doi: 10.1111/j.1365-2966.2009.14460.x
  * Maeda et al. (2007) Maeda, K., Kawabata, K., Tanaka, M., et al. 2007, ApJ, 658, L5, doi: 10.1086/513564
  * Mazzali et al. (2014) Mazzali, P. A., McFadyen, A. I., Woosley, S. E., Pian, E., & Tanaka, M. 2014, MNRAS, 443, 67, doi: 10.1093/mnras/stu1124
  * Mazzali et al. (2006) Mazzali, P. A., Deng, J., Nomoto, K., et al. 2006, Nature, 442, 1018, doi: 10.1038/nature05081
  * Mészáros et al. (2004) Mészáros, P., Kobayashi, S., Razzaque, S., & Zhang, B. 2004, Baltic Astronomy, 13, 317.  https://arxiv.org/abs/astro-ph/0305066
  * Metzger et al. (2015) Metzger, B. D., Margalit, B., Kasen, D., & Quataert, E. 2015, MNRAS, 454, 3311, doi: 10.1093/mnras/stv2224
  * Metzger et al. (2007) Metzger, B. D., Thompson, T. A., & Quataert, E. 2007, ApJ, 659, 561, doi: 10.1086/512059
  * Mirabal et al. (2006) Mirabal, N., Halpern, J. P., An, D., Thorstensen, J. R., & Terndrup, D. M. 2006, ApJ, 643, L99, doi: 10.1086/505177
  * Modjaz et al. (2006) Modjaz, M., Stanek, K. Z., Garnavich, P. M., et al. 2006, ApJ, 645, L21, doi: 10.1086/505906
  * Nakar (2015) Nakar, E. 2015, ApJ, 807, 172, doi: 10.1088/0004-637X/807/2/172
  * Nishimura et al. (2015) Nishimura, N., Takiwaki, T., & Thielemann, F.-K. 2015, ApJ, 810, 109, doi: 10.1088/0004-637X/810/2/109
  * Norris & Bonnell (2004) Norris, J. P., & Bonnell, J. T. 2004, in American Institute of Physics Conference Series, Vol. 727, Gamma-Ray Bursts: 30 Years of Discovery, ed. E. Fenimore & M. Galassi, 412–415, doi: 10.1063/1.1810876
  * Pian et al. (2006) Pian, E., Mazzali, P. A., Masetti, N., et al. 2006, Nature, 442, 1011, doi: 10.1038/nature05082
  * Prentice et al. (2016) Prentice, S. J., Mazzali, P. A., Pian, E., et al. 2016, MNRAS, 458, 2973, doi: 10.1093/mnras/stw299
  * Sari et al. (1998) Sari, R., Piran, T., & Narayan, R. 1998, ApJ, 497, L17, doi: 10.1086/311269
  * Slane (2017) Slane, P. 2017, Pulsar Wind Nebulae, ed. A. W. Alsabti & P. Murdin, 2159, doi: 10.1007/978-3-319-21846-5_95
  * Soderberg et al. (2006) Soderberg, A. M., Kulkarni, S. R., Nakar, E., et al. 2006, Nature, 442, 1014, doi: 10.1038/nature05087
  * Sollerman et al. (2006) Sollerman, J., Jaunsen, A. O., Fynbo, J. P. U., et al. 2006, A&A, 454, 503, doi: 10.1051/0004-6361:20065226
  * Stratta et al. (2013) Stratta, G., Gendre, B., Atteia, J. L., et al. 2013, ApJ, 779, 66, doi: 10.1088/0004-637X/779/1/66
  * Suwa & Tominaga (2015) Suwa, Y., & Tominaga, N. 2015, MNRAS, 451, 282, doi: 10.1093/mnras/stv901
  * Troja et al. (2007) Troja, E., Cusumano, G., O’Brien, P. T., et al. 2007, ApJ, 665, 599, doi: 10.1086/519450
  * Usov (1992) Usov, V. V. 1992, Nature, 357, 472, doi: 10.1038/357472a0
  * Valenti et al. (2008) Valenti, S., Benetti, S., Cappellaro, E., et al. 2008, MNRAS, 383, 1485, doi: 10.1111/j.1365-2966.2007.12647.x
  * Vinkó et al. (2004) Vinkó, J., Blake, R. M., Sárneczky, K., et al. 2004, A&A, 427, 453, doi: 10.1051/0004-6361:20040272
  * Šimon et al. (2010) Šimon, V., Pizzichini, G., & Hudec, R. 2010, A&A, 523, A56, doi: 10.1051/0004-6361/201014469
  * Wang et al. (2016) Wang, L.-J., Han, Y.-H., Xu, D., et al. 2016, ApJ, 831, 41, doi: 10.3847/0004-637X/831/1/41
  * Wang et al. (2017) Wang, L. J., Yu, H., Liu, L. D., et al. 2017, ApJ, 837, 128, doi: 10.3847/1538-4357/aa5ff5
  * Waxman et al. (2007) Waxman, E., Mészáros, P., & Campana, S. 2007, ApJ, 667, 351, doi: 10.1086/520715
  * Woosley (1993) Woosley, S. E. 1993, ApJ, 405, 273, doi: 10.1086/172359
  * Woosley (2010) —. 2010, ApJ, 719, L204, doi: 10.1088/2041-8205/719/2/L204
  * Woosley & Bloom (2006) Woosley, S. E., & Bloom, J. S. 2006, ARA&A, 44, 507, doi: 10.1146/annurev.astro.43.072103.150558
  * Wu et al. (2021) Wu, G.-L., Yu, Y.-W., & Zhu, J.-P. 2021, A&A, 654, A124, doi: 10.1051/0004-6361/202141325
  * Yu et al. (2019) Yu, Y.-W., Chen, A., & Li, X.-D. 2019, ApJ, 877, L21, doi: 10.3847/2041-8213/ab1f85
  * Yu et al. (2010) Yu, Y.-W., Cheng, K. S., & Cao, X.-F. 2010, ApJ, 715, 477, doi: 10.1088/0004-637X/715/1/477
  * Yu & Dai (2007) Yu, Y. W., & Dai, Z. G. 2007, A&A, 470, 119, doi: 10.1051/0004-6361:20077053
  * Yu et al. (2008) Yu, Y. W., Dai, Z. G., & Zheng, X. P. 2008, MNRAS, 385, 1461, doi: 10.1111/j.1365-2966.2008.12924.x
  * Yu et al. (2018) Yu, Y.-W., Liu, L.-D., & Dai, Z.-G. 2018, ApJ, 861, 114, doi: 10.3847/1538-4357/aac6e5
  * Yu et al. (2017) Yu, Y.-W., Zhu, J.-P., Li, S.-Z., Lü, H.-J., & Zou, Y.-C. 2017, ApJ, 840, 12, doi: 10.3847/1538-4357/aa6c27
  * Zhang & Mészáros (2001) Zhang, B., & Mészáros, P. 2001, ApJ, 552, L35, doi: 10.1086/320255
  * Zou et al. (2021) Zou, L., Liang, E.-W., Zhong, S.-Q., et al. 2021, MNRAS, 508, 2505, doi: 10.1093/mnras/stab2766


# Self-training For Pre-training Language Models

Tong Guo<EMAIL_ADDRESS>China

(2018)

###### Abstract.

Language model pre-training has proven to be useful in many language
understanding tasks. In this paper, we investigate whether it is still helpful
to add the self-training method in the pre-training step and the fine-tuning
step. Towards this goal, we propose a learning framework that making best use
of the unlabel data on the low-resource and high-resource labeled dataset. In
industry NLP applications, we have large amounts of data produced by users or
customers. Our learning framework is based on this large amounts of unlabel
data. First, We use the model fine-tuned on manually labeled dataset to
predict pseudo labels for the user-generated unlabeled data. Then we use the
pseudo labels to supervise the task-specific training on the large amounts of
user-generated data. We consider this task-specific training step on pseudo
labels as a pre-training step for the next fine-tuning step. At last, we fine-
tune on the manually labeled dataset upon the pre-trained model. In this work,
we first empirically show that our method is able to solidly improve the
performance by 3.6%, when the manually labeled fine-tuning dataset is
relatively small. Then we also show that our method still is able to improve
the performance further by 0.2%, when the manually labeled fine-tuning dataset
is relatively large enough. We argue that our method make the best use of the
unlabel data, which is superior to either pre-training or self-training alone.

pre-training, self-training, text classification, named entity recognition

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: Woodstock ’18: ACM Symposium on Neural
Gaze Detection; June 03–05, 2018; Woodstock, NY††booktitle: Woodstock ’18: ACM
Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Natural
language processing††ccs: Computing methodologies Knowledge representation and
reasoning

##  1\. Introduction

Figure 1. The learning framework of our method: (1) We use the domain-specific
data to pre-train on MaskLM task and get the pre-trained Model-A. This step is
a classic pre-training step. (2) We use the manually labeled data to fine-tune
Model-A to get Model-B. This step is a classic fine-tuning step (3) We use
Model-B to predict pseudo labels for all the unlabeled data. (4) We train the
model using the pseudo-label and task-specific loss to get Model-C, this step
is also considered as a pre-training step for the next step. This step is
viewed as the step of self-training as pre-training (5) We fine-tune Model-C
to get the final Model-D. The fine-tuning data do not mix the high-score
pseudo-label data.

Deep neural networks often require large amounts of labeled data to achieve
good performance. However, acquiring labels is a costly process, which
motivates research on methods that can effectively utilize unlabeled data to
improve performance. Towards this goal, semi-supervised learning (Chapelle et
al., 2009) and pre-training are proposed to take advantage of both labeled and
unlabeled data. Self-training (Scudder, 1965; Yarowsky, 1995) is a semi-
supervised method which uses a teacher model, trained using labeled data, to
create pseudo labels for unlabeled data. Then, a student model is trained with
this new training set to yield the final model. Meanwhile, language model pre-
training has been shown to be effective for improving many natural language
processing (NLP) tasks (Devlin et al., 2019; Brown et al., 2020; Peters et
al., 2018; Radford et al., 2019, 2018). Previous studies (Gururangan et al.,
2020; Lee et al., 2020; Beltagy et al., 2019) also have shown the benefit of
continued pre-training on domain-specific unlabeled data. Then our direct
motivation is to answer the question: Do pre-training combined with self-
training further improves the fine-tuning performance? Are they complementary?
In this paper, we try to answer this question by using the pseudo-label data
for pre-training language model.

We are one of the largest Chinese dining and delivery platform. The report
shows that the total number of online food orders reaches millions in a day.
In our industry NLP applications, we gather large amounts of unlabeled data
produced by users or customers. So we have large amounts of data in the food
domain, which allows us to use the large amounts of unlabel data. We follow
the self-training idea that we can use the fine-tuned model to predict a
pseudo-label. We explore the question whether the data with pseudo-label bring
further improvement for BERT. We design a learning framework to explore the
effect of self-training for pre-training. The whole learning framework is
shown in Figure 1.

In this paper, we evaluate the performance of our learning framework on our
item (a specific food) type text classification task and item property named
entity recognition (NER) task. For the item type text classification task, our
goal is to predict the 1 food type in 311 classes/types for all the 500
million items, given the item name, item short tag written by POI (Point of
Interest, a specific store or restaurant) and item’s POI name. For the item
property NER task, our goal is to extract item property (such as benefit for
stomach, sweet and chewy, cool and refreshing) from item description. The task
detail is described in section 3.

As it is shown in Figure 1, we have several terminologies to introduce:

In-domain data / domain-specific data: The large amounts of data in our
database. Our manually labeled dataset is sampled from this in-domain data.
This in-domain data is produced by our user or customer.

Self-training: We use a simple self-training method inspired by Algorithm 1.
First, a teacher model is trained on the manually labeled data. Then the
teacher model generates pseudo labels on unlabeled data (e.i., all the data in
our database). Finally, a student is trained to optimize the loss on human
labels and pseudo labels jointly.

Pre-training: This term means the training step that use hundred millions
data. The step 1 of Figure 1 is training on the unlabel data. The step 1 of
Figure 1 is training on the pseudo-label data. The next step of this training
step is the fine-tuning step using the manually labeled data.

Self-training as pre-training / task-specific training: This term refers to
the step 4 of Figure 1, which means the training step that use pseudo-label.
We consider this training step as a pre-training step for the next fine-tuning
step on manually labeled data.

Self-training for fine-tuning / task-specific fine-tuning: This term refers to
the step 5 of Figure 1, which means the fine-tuning step that use both the
pseudo-label data and the manually labeled data.

In detail, we aim to answer the questions: How much does self-training as pre-
training perform in low-resource NLP task and high-resource NLP task? To the
best of our knowledge, this is the first study exploring the improvement of
self-training as pre-training (i.e., using the pseudo-label data for pre-
training language models) for natural language understanding. And this is the
first study exploring the self-training improvement based on the step of self-
training as pre-training (i.e., adding the high-confidence-score pseudo-label
data for fine-tuning based on the pre-trained model). This is also the first
work exploring the combination of self-training and pre-training when the
manually labeled fine-tuning dataset is relatively large (2000K).

In summary, our contributions include:

• We explore the improvement of pre-training combined with self-training. We
reveal that pre-training combined with self-training improves the performance
stably, when the fine-tuning dataset is relatively small (100K). We find that
using the pseudo-label data for fine-tuning do not improve the performance
further, when the fine-tuning dataset is relatively large enough. But using
the manually labeled data in the fine-tuning step without pseudo-label data
improve the performance, when the fine-tuning dataset is relatively large
enough (2000K).

• We argue that our learning framework is the best combination of self-
training and pre-training method to make use of a large amounts of unlabeled
data. Even when the fine-tuning dataset is relatively large enough (2000K),
Our method, which is corresponding to Figure 1 and Figure 4-5, still is able
to improve the performance. In our experiments based on our dataset, the
previous methods (i.e., the classic self-training) is not able to improve the
performance when the fine-tuning dataset is relatively large enough (2000K).

• We explore different experimental factors on our text classification dataset
and NER dataset. The experiment results prove that the pre-training with task-
specific loss and no-MaskLM (masked language model) loss is the best way to
make use of the unlabel data. We also find that pre-training using KL-
divergence loss with pre-softmax logits is better than cross-entropy loss with
one-hot pseudo-label, which is corresponding to the step 4 of Figure 1.

##  2\. Related Work

Algorithm 1 Classic Self-training

1:Train a base model $f_{{\bm{\theta}}}$ on
$L=\\{{\bm{x}}_{i},{\bm{y}}_{i}\\}_{i=1}^{l}$

2:repeat

3: Apply $f_{{\bm{\theta}}}$ to the unlabeled instances $U$

4: Select a subset
$S\subset\\{({\bm{x}},f_{{\bm{\theta}}}({\bm{x}}))|{\bm{x}}\in U\\}$

5: Train a new model $f_{{\bm{\theta}}}$ on $S\cup L$

6:until convergence or maximum iterations are reached

There is a long history of pre-training language representations(Brown et al.,
1992; Ando et al., 2005; Blitzer et al., 2006; Pennington et al., 2014;
Mikolov et al., 2013; Turian et al., 2010; Mnih and Hinton, 2009; Kiros et
al., 2015; Logeswaran and Lee, 2018; Jernite et al., 2017; Hill et al., 2016),
and we briefly review the most widely-used approaches in this section.

BERT (Devlin et al., 2019) is based on the multi-layer transformer encoder
(Vaswani et al., 2017), which produces contextual token representations that
have been pre-trained from unlabeled text and fine-tuned for a supervised
downstream task. BERT achieved previously state-of-the-art results on many
sentence-level tasks from the GLUE benchmark (Wang et al., 2018). There are
two steps in BERT’s framework: pre-training and fine-tuning. During pre-
training, the model is trained on unlabeled data by using masked language
model task and next sentence prediction task. Apart from output layers, the
same architectures are used in both pre-training and fine-tuning. The same
pre-trained model parameters are used to initialize models for different down-
stream tasks.

Semi-supervised learning (Zhu and Goldberg, 2009; Zhu, 2005; Chapelle et al.,
2009) solve the problem that making best use of a large amounts of unlabeled
data. These works include UDA (Xie et al., 2020a), Mixmatch (Berthelot et al.,
2019b), Fixmatch (Sohn et al., 2020), Remixmatch (Berthelot et al., 2019a).
These works design the unsupervised loss to add to the supervised loss. These
works prove that domain-specific unlabel data is able to improve the
performance, especially in low-resource manually labeled dataset.

Self-training (Blum and Mitchell, 1998; Zhou and Goldman, 2004; Zhou and Li,
2005) is one of the earliest and simplest semi-supervised methods. As shown in
Algorithm 1, Self-training first uses labeled data to train a good teacher
model, then use the teacher model to label unlabeled data and finally use the
labeled data and unlabeled data to jointly train a student model. Some early
work have successfully applied self-training to word sense
disambiguation(Yarowsky, 1995) and parsing(Huang and Harper, 2009; Reichart
and Rappoport, 2007; McClosky et al., 2006). In recent years, these works
include self-training for natural language processing(Du et al., 2020; He et
al., 2019), self-training for computer vision (Xie et al., 2020b; Zoph et al.,
2020), self-training for speech recognition (Kahn et al., 2020) and back-
translation for machine translation (Bojar and Tamchyna, 2011; Sennrich et
al., 2015; Edunov et al., 2018). (Zoph et al., 2020) reveals some generality
and flexibility of self-training combined with pre-training in computer
vision. (Du et al., 2020) study the self-training improvement on the fine-
tuning step, based on data augmentation. But (Du et al., 2020) restrict to the
size of the specific in-domain data and manually labeled data.

##  3\. Our Dataset

In this section, we describe our datasets. The item examples are shown in
Table 1. The task examples are shown in Table 2. The data size information is
shown in Table 3.

Table 1. Some examples of the food/items.  Item Name | Item Short Tag | Item POI Name | Item Description
---|---|---|---
Yuxiang shredded pork | Hot dishes | The Sichuan restaurant |  6cmIt contains salty, sweet, sour, hot, and fresh tastes, making the food more delicious.
Kung Pao chicken | Hot dishes | The Sichuan restaurant |  6cmIt is a spicy, stir-fried Chinese dish made with cubes of chicken, peanuts and vegetables.
Congee with Pork and Century Eggs | Congee series | The congee restaurant |  6cmIt is benefit for stomach.
Knife-sliced noodle | Noodle series | The noodle restaurant |  6cmThe noodles going directly into the boiling water. This makes the noodles very fresh
Table 2. Some examples of the text classification task and the NER task, corresponding to Table 1. Item Name | Item Classification Food Type | Item NER results
---|---|---
Yuxiang shredded pork | stir-fried food | fresh tastes
Kung Pao chicken | stir-fried food | (no result)
Congee with Pork and Century Eggs | meat congee | benefit for stomach
Knife-sliced noodle | noodle with soup | very fresh
Table 3. The data amount and average token number for each experiment step.  | Text Classification Task | NER Task
---|---|---
Experiment Step | Data Size | Average Length | Data Size | Average Length
Domain-Specific Pre-training | 500 million | 28 | 200 million | 50
Task-Specific Pre-training | 500 million | 28 | 200 million | 50
Fine-Tune Training | 2,000,000 | 28 | 47,500 | 50
Fine-Tune Test | 40,000 | 28 | 2,500 | 50

###  3.1. Chinese Text Classification Dataset

This task is to predict the item type or category, given the item name, item
short tag, item POI name. We define 311 classes/types for all the items. The
model inputs are item name, item short tag given by POI and the POI name of
item. We manually labeled 2,040,000 data. The total item number is 500
million.

###  3.2. Chinese NER Dataset

This task is to extract all the item properties from item description. The
item description is a short paragraph written by users. We manually labeled
50,000 data. The total item description number is 200 million. There are 500
million items in total, in which 200 million items have the their
descriptions.

##  4\. Our Method

In this section, we describe our method in detail. As the Figure 1 shows, our
framework includes 5 steps, so we separate this section into 5 subsections: In
subsection 4-1, we describe the domain-specific pre-training with unlabeled
data. In subsection 4-2, we describe the task-specific fine-tuning with the
manually labeled data. In subsection 4-3, we describe the inference step by
the fine-tuned model of last step. In subsection 4-4, we describe the task-
specific pre-training with the pseudo-label predicted by the fine-tuned model.
In this paper, the step that training the model with the task-specific loss on
pseudo-label data is also considered as a pre-training step for the next step.
In subsection 4-5, we describe the task-specific fine-tuning which is almost
same to subsection 4-2 to get the final model.

###  4.1. Domain-Specific Pre-training

This is the first step of our method. This step is almost the same to the
origin BERT’s (Devlin et al., 2019) pre-training step except the data
preparation. We use all the in-domain data in our database for pre-training.

For our Chinese text classification task, the model’s 3 inputs are item name,
item short tag and item’s POI name. We follow the origin BERT (Devlin et al.,
2019) setting and use the character-level word masking. In detail we concat
the 3 inputs string and mask 15% characters. The max sequence length is 64. We
follow the RoBERTa (Liu et al., 2019) setting and remove the next-sentence-
predict loss. For efficiency reason, we use 3-layer-BERT in our experiment
because we need to inference more than hundred millions data in our
application. We extract 3 layers from the origin official pre-trained BERT
(Devlin et al., 2019) as the initialized parameter. The total pre-training
data number is 500 million.

For our Chinese NER task, the model’s input is the item description short
paragraph. We follow the RoBERTa setting and use the character-level word
masking in the short paragraph. The max sequence length is 128 and we remove
the next-sentence-predict loss. For efficiency reason, we use 3-layer-BERT in
our experiment. We extract 3 layers from the origin official pre-trained BERT
as the initialized parameter. The total pre-training data number is 200
million.

Figure 2. The task-specific pre-training for item type text classification
task (i.e., the ablation experiment for step 4 of Figure 1). The main
difference to the origin BERT pre-training is we replace the next-sentence-
predict loss with the classification loss from the pseudo-label. In this
picture, we mask 15% characters of the text as the model input. Figure 3. The
task-specific pre-training for item property NER task (i.e., the ablation
experiment for step 4 of Figure 1). The main difference to the origin BERT
pre-training is we add the NER loss from the pseudo-label. In this picture, we
mask 15% characters of the text as the model input. Figure 4. The task-
specific pre-training for item type text classification task (i.e., step 4 of
Figure 1). The main difference to the origin BERT pre-training is we replace
the next-sentence-predict loss with the classification loss from the pseudo-
label. In this picture, we do not mask 15% characters of the text as the model
input. Figure 5. The task-specific pre-training for item property NER task
(i.e., step 4 of Figure 1). The main difference to the origin BERT pre-
training is we replace the MaskLM loss with the NER loss from the pseudo-
label. In this picture, we do not mask 15% characters of the text as the model
input.

###  4.2. Task-Specific Fine-Tuning

This is the second step of our method. We use the model parameter of the last
step to initialize this step’s model.

For our Chinese text classification task, we follow the origin BERT (Devlin et
al., 2019) setting and use the cross-entropy loss. The total fine-tuning data
number is our 2,040,000 manually labeled data. We select 40,000 as test
dataset. Then we sampled 100K, 400K and 2000K data from the rest 2000K data
separately.

For our Chinese NER task, we use the CRF(Conditional Random Field) loss. The
total fine-tuning data number is our 50,000 manually labeled data. We split
all the data to 19:1 as training dataset and test dataset.

###  4.3. Inference Step

This is the third step of our method. This step is to use the fine-tuned model
of last subsection to predict a pseudo-label for all the in-domain data.

For our Chinese text classification task, we use the fine-tuned model of the
last subsection to predict the classification pseudo-label for all the 500
million unlabeled data. Then we get the 500 million classification data with
pseudo-label. The pseudo-label could be the one-hot label or the pre-softmax
logits.

For our Chinese NER task, we use the fine-tuned model of the last subsection
to predict the sequence tagging pseudo-label for all the 200 million unlabeled
data. Then we get the 200 million NER data with pseudo-label. The pseudo-label
could be the one-hot label sequence or the pre-softmax logits.

###  4.4. Task-specific Pre-training

This is the fourth step of our method. We use the pseudo-label of last step as
the pseudo ground truth label for training. We consider this training step as
the pre-training step for the next step.

For our Chinese text classification task, the model input is the masked text,
which is the concat of item name, item short tag and item’s POI name. There
are four kinds of experiment: 1) The sum of the classification cross-entropy
loss and the MaskLM loss. 2) The cross-entropy loss only. 3) The sum of the
classification KL-divergence loss on the pre-softmax logits and the MaskLM
loss. 4) The KL-divergence loss on the pre-softmax logits only. In detail, we
use the [CLS] token’s output of BERT for computing the classification cross-
entropy loss on one-hot pseudo-label or the KL-divergence loss on the pre-
softmax logits. The detail is shown in Figure 2 and Figure 4.

For our Chinese NER task, the model input is masked text, which is the item
description short paragraph. There are four kinds of experiment: 1) The sum of
the CRF loss and the MaskLM loss. 2) The CRF loss only. 3) The sum of the KL-
divergence loss on the pre-softmax logits and the MaskLM loss. 4) The KL-
divergence loss on the pre-softmax logits only. In detail, we use all the
tokens’ output sequence of BERT for computing the CRF loss and MaskLM loss.
The detail is shown in Figure 3 and Figure 5.

###  4.5. Final Task-specific Fine-tuning

This is the final step of our method. We find that fine-tuning on the manually
labeled dataset without pseudo-label high-score data is better. For the
classic self-training comparison experiment, we randomly sampled the addition
data from each class of the pseudo-label high-score data. We use the pre-
trained model of subsection 4-4 for fine-tuning.

Table 4. Experiment results for the NER task and the text classification task separated to fine-tuning data with different size. The BERT-Base baselines are initialized by the origin BERT’s (Devlin et al., 2019). Classic self-training follow the Algorithm 1 and mix the pseudo-label data and manually labeled data. Our method is pre-training using KL-divergence loss on the pre-softmax logits without the MaskLM loss and fine-tuning on dataset that do not mix the high-score pseudo-label data, which is corresponding to Figure 1. Accuracy scores are reported for the text classification task. F1 scores are reported for NER task.  | Text Classification Test Accuracy | NER
---|---|---
Model | 100K data | 400K data | 2000K data | Test F1
BERT-Base-3layer baseline | 87.0 | 89.2 | 90.8 | 88.6
BERT-Base-12layer baseline | 87.5 | 89.4 | 91.0 | 88.7
Classic Self-training upon BERT-Base-3layer | 88.0 | 89.8 | 91.0 | 89.0
Model-B | 88.2 | 90.1 | 91.7 | 88.8
Classic Self-training upon Model-A | 89.3 | 90.4 | 91.6 | 89.2
Classic Self-training upon Model-C | 90.2 | 90.9 | 91.7 | 89.4
Our Method (Model-D) | 90.6 | 91.1 | 91.9 | 89.6

##  5\. Experiments

In this section we describe our experimental setup and experimental results.

###  5.1. Experimental Setup

In this section we describe the parameters in our experiments. In all the
experiments except the baselines, we use 3-layer-BERT with 768 hidden size and
12 self-attention heads. (Total Parameters = 40M). For efficiency reason, we
use only 3 layers of BERT, because it is an industry application and we only
have limited computation resource to inference hundred millions data. And
3-layer-BERT is 200%-300% faster than 12-layer-BERT in inference time. The max
sequence length for text classification task is 64. The max sequence length
for the NER task is 128. The total data size are presented in Table 3. For the
text classification task, we setup 3 experimental groups with different fine-
tuning data size separately. The 3 different fine-tuning dataset are sampled
from the total 2,000,000 data and run the learning framework independently.

####  5.1.1. Text Classification Domain-Specific Pre-training

We use a batch size of 64 * 4-GPU and pre-train for 3,000,000 steps. We use
Adam (Kingma and Ba, 2014) with learning rate of 5e-5, beta1 = 0.9, beta2 =
0.999, L2 weight decay of 0.01, learning rate warmup over the first 10,000
steps, and linear decay of the learning rate. We use a dropout (Srivastava et
al., 2014) probability of 0.1 on all layers. The 3-layer-BERT is initialized
by the origin BERT’s (Devlin et al., 2019) 3 layers.

####  5.1.2. NER Domain-Specific Pre-training

The NER domain-specific pre-training is almost the same to the text
classification domain-specific pre-training. The pre-training data is item
description sentence for NER and the pre-training data is concatenation of
multi words (item name, item short tag, item poi name) for text
classification.

####  5.1.3. Text Classification Fine-Tuning

We use a batch size of 64 * 1-GPU and fine-tune for 7 epochs. We use Adam with
learning rate of 1e-5. The dropout probability is 0.1 on all layers. The fine-
tuning data with different size in Table 2 is sampled from all the 2,000,000
data and the 40,000 test data is fixed.

####  5.1.4. NER Fine-Tuning

We use a batch size of 64 * 1-GPU and fine-tune for 3 epochs. We use Adam with
learning rate of 1e-5. The dropout probability is 0.1 on all layers.

####  5.1.5. Text Classification Task-Specific Pre-training

We use a batch size of 64 * 4-GPU and pre-train for 3,000,000 steps. We use
Adam with learning rate of 5e-5, beta1 = 0.9, beta2 = 0.999, L2 weight decay
of 0.01, learning rate warmup over the first 10,000 steps, and linear decay of
the learning rate. We use a dropout probability of 0.1 on all layers. The
3-layer-BERT is initialized by the origin BERT’s (Devlin et al., 2019) 3
layers. For fair comparison, we do not initialize this step of pre-training by
the result model of domain-specific pre-training step.

####  5.1.6. NER Task-Specific Pre-training

We use a batch size of 64 * 4-GPU and pre-train for 3,000,000 steps. We use
Adam with learning rate of 5e-5, beta1 = 0.9, beta2 = 0.999, L2 weight decay
of 0.01, learning rate warmup over the first 10,000 steps, and linear decay of
the learning rate. We use a dropout probability of 0.1 on all layers. The
3-layer-BERT is initialized by the origin BERT’s (Devlin et al., 2019) 3
layers. For fair comparison, we do not initialize this step of pre-training by
the result model of domain-specific pre-training step.

Table 5. Ablation over the pre-training step on in-domain data without task-specific loss, corresponding to the step 1 of Figure 1, which shows the improvement of in-domain pre-training on MaskLM task. Model-B is corresponding to Figure 1. | Text Classification Test Accuracy | NER
---|---|---
Model | 100K data | 400K data | 2000K data | Test F1
BERT-Base-3layer baseline | 87.0 | 89.2 | 90.8 | 88.6
BERT-Base-12layer baseline | 87.5 | 89.4 | 91.0 | 88.7
Model-B | 88.2 | 90.1 | 91.7 | 88.8
Table 6. Ablation over the pre-training step on in-domain unlabel data and pseudo-label data, which shows the improvement of pre-training on pseudo-label data. Model-B and Model-D are corresponding to Figure 1. Model-B is upon the in-domain MaskLM-based pre-training on unlabel data. Model-D is upon the task-specific pre-training on pseudo-label data. | Text Classification Test Accuracy | NER
---|---|---
Model | 100K data | 400K data | 2000K data | Test F1
Model-B | 88.2 | 90.1 | 91.7 | 88.8
Model-D | 90.6 | 91.1 | 91.9 | 89.6
Table 7. Ablation over the task-specific pre-training step between logits-based loss and one-hot label-based loss, corresponding to the step 4 of Figure 1. Model-D is corresponding to Figure 1. (label) means we use the one-hot pseudo-label for the supervised pre-training. (logits) means we use the pre-softmax logits for the supervised pre-training. | Text Classification Test Accuracy | NER
---|---|---
Model | 100K data | 400K data | 2,000K data | Test F1
Model-D (label) | 90.4 | 91.0 | 91.7 | 89.3
Model-D (logits) | 90.6 | 91.1 | 91.9 | 89.6
Table 8. Ablation over the task-specific pre-training step between masked noisy input text and and the origin input text, corresponding to the step 4 of Figure 1. Model-D is corresponding to Figure 1. (masked) means we masked 15% tokens of the input text. (nomask) means we input the origin text. | Text Classification Test Accuracy | NER
---|---|---
Model | 100K data | 400K data | 2000K data | Test F1
Model-D (masked / MaskLM loss + Task-specific loss) | 90.2 | 90.9 | 91.6 | 89.3
Model-D (nomask / Task-specific loss) | 90.6 | 91.1 | 91.9 | 89.6
Table 9. The comparison of absolute improvement upon baseline (the first row of table), which proves the conclusion again: Continued pre-training on in-domain unlabeled data benefits the performance. | Text Classification Test Accuracy | NER
---|---|---
Factor | 100K data | 400K data | 2000K data | Test F1
BERT-Base-3layer baseline | - | - | - | -
BERT-Base-12layer | +0.5% | +0.2% | +0.2% | +0.1%
\+ self-training | +1.0% | +0.6% | +0.2% | +0.4%
In-domain pre-training with MaskLM task | +1.2% | +0.9% | +0.9% | +0.2%
Table 10. The comparison of absolute improvement upon baseline (the first row of table), which reveals the conclusion: Self-training as pre-training get bigger gain than in-domain pre-training. | Text Classification Test Accuracy | NER
---|---|---
Factor | 100K data | 400K data | 2000K data | Test F1
In-domain pre-training with MaskLM task | - | - | - | -
\+ self-training | +0.5% | +0.3% | -0.1% | +0.4%
pre-training with task-specific task and MaskLM task | +1.0% | +0.8% | -0.1% | +0.5%
pre-training with task-specific task | +1.4% | +1.1% | +0.2% | +0.8%
Table 11. The comparison of absolute improvement upon baseline (the first row of table), which reveals the conclusion: For high-resource manually labeled dataset, pre-training with task-specific loss without masking the input text is the only way to further improve the performance. | Text Classification Test Accuracy | NER
---|---|---
Factor | 100K data | 400K data | 2000K data | Test F1
pre-training with task-specific task | - | - | - | -
\+ self-training | +0.4% | +0.2% | -0.2% | +0.2%

###  5.2. Baseline Setup

In this section we describe the baselines in Table 2.

####  5.2.1. BERT-Base-3layer baseline

For the BERT-Base-3layer baseline, we extract 3 layers from the origin
12-layer BERT-Base (Devlin et al., 2019) and fine-tune on the manually labeled
dataset.

####  5.2.2. BERT-Base-12layer baseline

For the BERT-Base-12layer baseline, we use the origin 12-layer BERT-Base
(Devlin et al., 2019) and fine-tune on the manually labeled dataset.

####  5.2.3. Classic Self-training baseline

For the Classic Self-training baseline, we use a simple self-training method
inspired by Algorithm 1. First, a teacher model is trained on the manually
labeled data. Then the teacher model generates pseudo labels on unlabeled data
(e.i., all the data in our database). Finally, a student is trained to
optimize the loss on human labels and pseudo labels jointly. The pseudo labels
data with the high confidence score are averagely sampled from each class of
all the data for fine-tuning upon the origin BERT-Base-12layer, Model-A and
Model-C.

###  5.3. Experimental Results

In this section, we present experiment results on the text classification task
and the NER task. The detail results are presented in Table 4-11. The text
classification accuracy means the exact match of predicted class and the
ground truth. There are 311 classes in the task. The NER F1 is same to the
definition of CoNLL (Sang and De Meulder, 2003). The BERT-Base baseline’s pre-
trained model is from the origin official Github repository. The BERT-
Base-3layer is extracted from the origin official BERT-Base-12layer.

###  5.4. Analysis

In this section, we analysis our experiment results on Table 4. we focus on
the effect of two factors: the size of the fine-tuning dataset, the data
content of the pre-training dataset.

Fine-tuning data size. In low-resource dataset, the whole gain of our
framework is 3.6% (from 87.0% to 90.6%) for the text classification task. In
high-resource dataset, the whole gain of our framework is 1.1% (from 90.8% to
91.9%) for the text classification task. We also observe that pre-training
with pseudo-label data (i.e., step 4 of Figure 1) is able to improve the
performance in high-resource dataset while fine-tuning with manually labeled
data and pseudo-label data is not able to improve the performance in high-
resource dataset. In the NER task, the stable improvement of in-domain pre-
training (i.e., step 1 of Figure 1) is only 0.2% (from 88.6% to 88.8%),
compared to the whole 1.0% (from 88.6% to 89.6%) improvement of the learning
framework in the NER task. While the improvement of in-domain pre-training for
the text classification task is 0.9% (from 90.8% to 91.9%). And the whole
improvement is 1.1% (from 90.8% to 91.9%) for the text classification task.

In-domain data and out-domain data. The domain-specific pre-training (i.e.,
the step 1 of Figure 1) gain of the text classification task is more than the
gain of the NER task. The gain of in-domain pre-training (i.e., the step 1 of
Figure 1) for text classification task is 0.7% (from 87.5% to 88.2%). The gain
of in-domain pre-training (i.e., the step 1 of Figure 1) for the NER task is
0.1% (from 88.7% to 88.8%). The reason is that the input of the text
classification task is the concatenation of multi words (item name, item short
tag, item poi name), which is not consistent to the origin BERT’s (Devlin et
al., 2019) pre-training. The origin BERT’s pre-training uses the Chinese
Wikipedia natural language sentences. And the NER task’s in-domain pre-
training uses natural language sentences in our database, which is not that
much different to the data format of origin BERT.

###  5.5. Ablation Studies

In this section, we perform ablation experiments over a number of facets in
order to better understand their relative importance, which is corresponding
to Table 5-8.

In Table 5, we show the ablation experiment results over in-domain pre-
training corresponding to step 1 of Figure 1. The gain of the text
classification task (from 87.0% to 88.2%) shows the power of in-domain
continued pre-training on unlabel data. Although the gain of NER task (from
88.7% to 88.8%) is small, we speedup the model inference by 200%-300% as we
replace the 12-layer BERT to 3-layer BERT.

In Table 6, we show the ablation experiment results between unlabel pre-
training and pseudo-label pre-training, corresponding to step 1 and 4 of
Figure 1. The gain of the text classification task (from 91.7% to 91.9%)
proves that pseudo-label pre-training (i.e., the step 4 of Figure 1) can still
improve the performance, even the fine-tuning dataset is very large.

In Table 7, we show the ablation experiment results between logits-based loss
and one-hot label-based loss. For the task-specific pre-training step on text
classification (i.e., the step 4 of Figure 1), using the KL-divergence loss
with pre-softmax logits get more gain than the cross-entropy loss with one-hot
pseudo-label. We think the reason is that the pre-softmax logits contains more
pre-training information than one-hot pseudo-label.

In Table 8, we show the ablation experiment results between masked text and
the origin text in the task-specific pre-training step. For the task-specific
pre-training step (i.e., the step 4 of Figure 1), adding the MaskLM loss or
masking 15% tokens of the input text do not get more gain, compared to the no-
mask or no-noisy text input. We also observer that no-mask or no-noisy text
input for pre-training with pseudo-label is able to improve the performance,
even when is fine-tuning dataset is relatively large (2000K).

##  6\. Conclusion

In summary, we reveal these conclusions (detail conclusions are shown in Table
9-11):

1) For low-resource manually label dataset, the combination of all the steps
of our learning framework improve the performance most, which means pre-
training with no-masked logits-based pseudo-label data.

2) For high-resource manually labeled dataset, pre-training with task-specific
loss (i.e., step 4 of Figure 1) without masking the input text is the only way
to further improve the performance.

3) For pre-training with task-specific loss (i.e., step 4 of Figure 1), input
the text without masking is better than masking 15% tokens of the text.

4) For pre-training with task-specific loss (i.e., step 4 of Figure 1), using
the KL-divergence loss computed on pre-softmax logits is better than cross-
entropy loss computed on one-hot pseudo-label.

5) More labeled data diminishes the value of self-training. But pre-training
still has big gain to performance.

6) Self-training as pre-training (i.e., step 4 of Figure 1) get bigger gain
than in-domain pre-training (i.e., step 1 of Figure 1).

In the end, our main contribution is we answered the question: Combining pre-
training and self-training to make best use of large amounts of unlabeled data
improves the fine-tuning performance. We propose a learning framework using
no-masked logits-based pseudo-label data for pre-training, which is superior
to either pre-training or self-training alone. The experiment result shows
that our learning framework make the best use of the unlabel data even when is
fine-tuning dataset is relatively large.

## References

  * (1)
  * Ando et al. (2005) Rie Kubota Ando, Tong Zhang, and Peter Bartlett. 2005\.  A framework for learning predictive structures from multiple tasks and unlabeled data.  _Journal of Machine Learning Research_ 6, 11 (2005).
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3606–3611.
  * Berthelot et al. (2019a) David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. 2019a.  Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  _arXiv preprint arXiv:1911.09785_ (2019).
  * Berthelot et al. (2019b) David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. 2019b.  Mixmatch: A holistic approach to semi-supervised learning. In _Advances in Neural Information Processing Systems_. 5049–5059.
  * Blitzer et al. (2006) John Blitzer, Ryan McDonald, and Fernando Pereira. 2006\.  Domain adaptation with structural correspondence learning. In _Proceedings of the 2006 conference on empirical methods in natural language processing_. 120–128.
  * Blum and Mitchell (1998) Avrim Blum and Tom Mitchell. 1998.  Combining labeled and unlabeled data with co-training. In _Proceedings of the eleventh annual conference on Computational learning theory_. 92–100.
  * Bojar and Tamchyna (2011) Ondřej Bojar and Aleš Tamchyna. 2011.  Improving translation model by monolingual data. In _Proceedings of the Sixth Workshop on Statistical Machine Translation_. 330–336.
  * Brown et al. (1992) Peter F Brown, Vincent J Della Pietra, Peter V Desouza, Jennifer C Lai, and Robert L Mercer. 1992\.  Class-based n-gram models of natural language.  _Computational linguistics_ 18, 4 (1992), 467–480.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ (2020).
  * Chapelle et al. (2009) Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. 2009\.  Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews].  _IEEE Transactions on Neural Networks_ 20, 3 (2009), 542–542.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. 4171–4186.
  * Du et al. (2020) Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, and Alexis Conneau. 2020\.  Self-training Improves Pre-training for Natural Language Understanding.  _arXiv preprint arXiv:2010.02194_ (2020).
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018\.  Understanding back-translation at scale.  _arXiv preprint arXiv:1808.09381_ (2018).
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020.  Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks.  _arXiv preprint arXiv:2004.10964_ (2020).
  * He et al. (2019) Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2019\.  Revisiting self-training for neural sequence generation.  _arXiv preprint arXiv:1909.13788_ (2019).
  * Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016\.  Learning distributed representations of sentences from unlabelled data.  _arXiv preprint arXiv:1602.03483_ (2016).
  * Huang and Harper (2009) Zhongqiang Huang and Mary Harper. 2009.  Self-training PCFG grammars with latent annotations across languages. In _Proceedings of the 2009 conference on empirical methods in natural language processing_. 832–841.
  * Jernite et al. (2017) Yacine Jernite, Samuel R Bowman, and David Sontag. 2017\.  Discourse-based objectives for fast unsupervised sentence representation learning.  _arXiv preprint arXiv:1705.00557_ (2017).
  * Kahn et al. (2020) Jacob Kahn, Ann Lee, and Awni Hannun. 2020.  Self-training for end-to-end speech recognition. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7084–7088.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015.  Skip-thought vectors.  _arXiv preprint arXiv:1506.06726_ (2015).
  * Lee et al. (2020) J Lee, W Yoon, S Kim, D Kim, S Kim, CH So, and J Kang. 2020\.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics (Oxford, England)_ 36, 4 (2020), 1234\.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Logeswaran and Lee (2018) Lajanugen Logeswaran and Honglak Lee. 2018.  An efficient framework for learning sentence representations.  _arXiv preprint arXiv:1803.02893_ (2018).
  * McClosky et al. (2006) David McClosky, Eugene Charniak, and Mark Johnson. 2006\.  Effective self-training for parsing. In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_. 152–159.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _arXiv preprint arXiv:1310.4546_ (2013).
  * Mnih and Hinton (2009) Andriy Mnih and Geoffrey E Hinton. 2009.  A scalable hierarchical distributed language model. In _Advances in neural information processing systems_. Citeseer, 1081–1088.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations. In _Proceedings of NAACL-HLT_. 2227–2237.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  (2018).
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Reichart and Rappoport (2007) Roi Reichart and Ari Rappoport. 2007.  Self-training for enhancement and domain adaptation of statistical parsers trained on small datasets. In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_. 616–623.
  * Sang and De Meulder (2003) Erik Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_. 142–147.
  * Scudder (1965) H Scudder. 1965\.  Probability of error of some adaptive pattern-recognition machines.  _IEEE Transactions on Information Theory_ 11, 3 (1965), 363–371.
  * Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015\.  Improving neural machine translation models with monolingual data.  _arXiv preprint arXiv:1511.06709_ (2015).
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. 2020.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _arXiv preprint arXiv:2001.07685_ (2020).
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ 15, 1 (2014), 1929–1958.
  * Turian et al. (2010) Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010\.  Word representations: a simple and general method for semi-supervised learning. In _Proceedings of the 48th annual meeting of the association for computational linguistics_. 384–394.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018\.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. 353–355.
  * Xie et al. (2020a) Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020a.  Unsupervised data augmentation for consistency training.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Xie et al. (2020b) Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020b.  Self-training with noisy student improves imagenet classification. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10687–10698.
  * Yarowsky (1995) David Yarowsky. 1995\.  Unsupervised word sense disambiguation rivaling supervised methods. In _33rd annual meeting of the association for computational linguistics_. 189–196.
  * Zhou and Goldman (2004) Yan Zhou and Sally Goldman. 2004.  Democratic co-learning. In _16th IEEE International Conference on Tools with Artificial Intelligence_. IEEE, 594–602.
  * Zhou and Li (2005) Zhi-Hua Zhou and Ming Li. 2005.  Tri-training: Exploiting unlabeled data using three classifiers.  _IEEE Transactions on knowledge and Data Engineering_ 17, 11 (2005), 1529–1541.
  * Zhu and Goldberg (2009) Xiaojin Zhu and Andrew B Goldberg. 2009.  Introduction to semi-supervised learning.  _Synthesis lectures on artificial intelligence and machine learning_ 3, 1 (2009), 1–130.
  * Zhu (2005) Xiaojin Jerry Zhu. 2005\.  Semi-supervised learning literature survey.  (2005).
  * Zoph et al. (2020) Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. 2020.  Rethinking pre-training and self-training.  _arXiv preprint arXiv:2006.06882_ (2020).