11institutetext: Institut d’Astrophysique de Paris, CNRS, Sorbonne Université,
UMR7095, 98bis bd Arago, 75014 Paris, France 22institutetext: Institute of
Astronomy and Kavli Institute for Cosmology, University of Cambridge,
Madingley Road, Cambridge CB3 0HA, UK 33institutetext: AIM, CEA, CNRS,
Université Paris-Saclay, Université Paris Diderot, Sorbonne Paris Cité, 91191
Gif-sur-Yvette, France 44institutetext: SOFIA Science Center, USRA, NASA Ames
Research Center, M.S. N232-12 Moffett Field, CA 94035, USA 55institutetext:
Department of Physics and Astronomy, University of Kentucky, 505 Rose Street,
Lexington, KY 40506, USA

# AGN jets do not prevent the suppression of conduction by the heat buoyancy
instability in simulated galaxy clusters

Ricarda S. Beckmann<EMAIL_ADDRESS>Yohan Dubois 11 Alisson
Pellissier 33 Fiorella L. Polles 44 Valeria Olivares 55

(Accepted . Received ; in original form )

Centres of galaxy clusters must be efficiently reheated to avoid a cooling
catastrophe. One potential reheating mechanism is anisotropic thermal
conduction, which could transport thermal energy from intermediate radii to
the cluster centre. However, if fields are not re-randomised, anisotropic
thermal conduction drives the heat buoyancy instability (HBI) which reorients
magnetic field lines and shuts off radial heat fluxes. We revisit the
efficiency of thermal conduction under the influence of spin-driven AGN jets
in idealised magneto-hydrodynamical simulations with anisotropic thermal
conduction. Despite the black hole spin’s ability to regularly re-orientate
the jet so that the jet-induced turbulence is driven in a quasi-isotropic
fashion, the HBI remains efficient outside the central 50 kpc of the cluster,
where the reservoir of heat is the largest. As a result, conduction plays no
significant role in regulating the cooling of the intra-cluster medium if
central active galactic nuclei are the sole source of turbulence. Whistler-
wave driven saturation of thermal conduction reduces the magnitude of the HBI
but does not prevent it.

###### Key Words.:

galaxies: clusters: intracluster medium – methods: numerical – galaxies:
magnetic fields – instabilities – galaxies: jets

##  1 Introduction

The hot intracluster medium (ICM) is a substantial thermal energy reservoir.
If this energy can be efficiently transported to the cluster centre via
thermal conduction, it could offset some of the centre’s radiative cooling
(Fabian 1994; Binney & Tabor 1995), and thereby contribute to the long-term
thermal stability of the cluster (Tucker & Rosner 1983; Santos 2000; Narayan &
Medvedev 2001; Ruszkowski & Begelman 2002).

The conductive heat flux $\@vec{F}_{\rm cond}$ takes the form (Spitzer & Härm
1953)

$\@vec{F}_{\rm cond}=-\kappa_{\rm
e}\@vec{\hat{b}}(\@vec{\hat{b}}\cdot\nabla)T_{\rm e}$ (1)

where $\kappa_{\rm e}$ is the Spitzer conductivity for electrons,
$\@vec{\hat{b}}$ is the unit vector along the magnetic field and $T_{\rm e}$
is the electron temperature. Assuming a predominantly radial temperature
gradient, as seen in galaxy clusters,

$\@vec{F}_{\rm cond,r}=-\kappa_{\rm e}\mbox{$f_{\rm c}$\leavevmode\nobreak\
}\frac{\partial T_{\rm e}(r)}{\partial r}$ (2)

where $f_{\rm c}$ parametrizes the effective strength of the conductive
heating in comparison to the Spitzer value. As thermal conductivity is high
along magnetic field lines but effectively zero across them (Spitzer & Härm
1953), the effective efficiency of thermal conduction, $f_{\rm c}$ , depends
strongly on magnetic field morphology. It is equal to $\mbox{$f_{\rm
c}$\leavevmode\nobreak\ }=b_{r}^{2}$, where
$b_{r}=\@vec{\hat{b}}\cdot\mathbf{\hat{r}}$ is the magnetic field unit vector
in the radial direction. A tangled magnetic field has $\mbox{$f_{\rm
c}$\leavevmode\nobreak\ }=1/3$, as the magnetic field is equally likely to be
oriented in each of the three dimensions, whereas a radial field has
$\mbox{$f_{\rm c}$\leavevmode\nobreak\ }=1$. The required values of $f_{\rm
c}$ to offset radiative cooling depends on the cluster (Jacob & Pfrommer
2017).

In the presence of thermal conduction, the Heat-Buoyancy instability (HBI)
(Quataert 2008; Parrish et al. 2009) can reorient the cluster magnetic field.
It acts when the temperature increases with height ($\mathbf{g}\cdot\nabla
T_{\rm e}<0$, where $\mathbf{g}$ is the gravitational acceleration). Left
unchecked, the HBI will rearrange magnetic fields in galaxy cluster centres to
a tangential configuration, suppressing conductive heat fluxes (Parrish et al.
2009; Bogdanović et al. 2009). Turbulence can counteract the HBI and re-
randomise the magnetic field (Ruszkowski & Oh 2010).

Thermal conduction has the potential to reduce cluster cooling flows
(Ruszkowski et al. 2011) and the total energy required for Active Galactic
Nuclei (AGN) to regulate cluster cooling flows (Kannan et al. 2016; Barnes et
al. 2019), but its efficiency depends on the efficiency of the HBI. This, in
turn, depends on the relative magnitude of turbulent and buoyant timescales
(McCourt et al. 2011). Volume-filling turbulence of $50-100\rm\ kms^{-1}$ can
suppressed the HBI and allow for $\mbox{$f_{\rm c}$\leavevmode\nobreak\ }=0.5$
(Ruszkowski & Oh 2010). This is a level of turbulence that can be delivered by
a simplified AGN-based turbulence model (Parrish et al. 2012), but more recent
simulations employing fixed direction AGN jets found that the HBI remains
active outside the jet cone (Karen Yang & Reynolds 2016; Su et al. 2019),
which limits thermal conduction.

If AGN are only able to re-randomise magnetic fields around the jet cone, the
jet direction is a key variable that determines the ability of the central AGN
to prevent the HBI. In this paper, we revisit whether an AGN jet can offset
the HBI and allow for efficient thermal conduction using a more self-
consistent treatment of jet direction. The paper is structured as follows: the
simulation setup is laid out in Section 2, insights on the evolution of the
HBI are presented in Section 3.1, the cooling flow is analysed in Section 3.2
and conclusions are summarised in Section 4.

##  2 Simulations

All simulations presented in this paper are part of the same suite as those
presented in Beckmann et al. (2022). We briefly summarise the setup here but
refer the reader to Beckmann et al. (2022) for further details.

In this paper we present a set of magneto-hydrodynamical (MHD) simulations of
isolated galaxy clusters with and without thermal conduction. The cluster
simulations are run with the adaptive mesh refinement code ramses (Teyssier
2002) that solves for the MHD equations with separate ion-electron
temperatures (Dubois & Commerçon 2016), including the anisotropic conductive
heat flux $\@vec{F}_{\rm cond}=-\kappa_{\rm
e}\@vec{\hat{b}}(\@vec{\hat{b}}\cdot\nabla)T_{\rm e}$ (see Eq. (1)). This
includes the Spitzer conductivity,

$\kappa_{\rm e}=f_{\rm sat}\kappa_{\rm sp}=f_{\rm sat}n_{\rm e}k_{\rm B}D_{\rm
cond}$ (3)

where $n_{\rm e}$ is the electron number density, $k_{\rm B}$ the Boltzmann
constant and $D_{\rm cond}$ the thermal diffusivity. The conductive flux
saturates once the characteristic scale length of the gradient of temperature
$\ell_{T_{\rm e}}=T_{\rm e}/\nabla T_{\rm e}$ is comparable or shorter than
the mean free path of electrons $\lambda_{\rm e}$ (Cowie & McKee 1977). We
follow Sarazin (1986) and introduce an effective conductivity that
approximates the solution in the unsaturated and saturated regime by

$f_{\rm sat}=\frac{1}{1+4.2\lambda_{\rm e}/\ell_{T_{\rm e}}}\,.$ (4)

When taken into account, the whistler instability (e.g. Roberg-Clark et al.
2016; Komarov et al. 2018) can reduce the saturation coefficient to

$f_{\rm sat,whist}=\frac{1}{1+(4.2+\beta/3)\lambda_{\rm e}/\ell_{T_{\rm
e}}}\,$ (5)

for high plasma-$\beta$. Our simulations use Eq. (4), except for AGN_whistler,
which uses Eq. (5) (See Table 1).

The induction equation is solved with constrained transport (Teyssier et al.
2006), which guarantees $\nabla\cdot\@vec{B}=0$ at machine precision. The MHD
system equations are solved using the MUSCL-Hancock scheme (Fromang et al.
2006), a minmod total variation diminishing scheme, and the HLLD Riemann
solver (Miyoshi & Kusano 2005). The flux for the anisotropic thermal
conduction is solved with an implicit method (Dubois & Commerçon 2016; Dashyan
& Dubois 2020) using a minmod slope limiter on the transverse component of the
face-oriented flux (Sharma et al. 2007).

###  2.1 Initial conditions and Refinement

Clusters are initialised with cored NFW profiles (Navarro et al. 1997) (gas
and dark matter) with a total mass of $8\times 10^{14}\ \rm M_{\odot}$, a core
radius of $13\rm\ kpc$, a concentration parameter of $c_{\rm 200}=4.41$
(Maccio et al. 2007) and a gas fraction of $0.103$ (Andreon et al. 2017). The
gas is initialised in hydrostatic equilibrium and dark matter is modelled as a
fixed background potential. A black hole sink particle of mass $1.65\times
10^{10}\rm\ M_{\odot}$ (Phipps et al. 2019) is placed at the centre of the
box. The magnetic field is initialised in a tangled configuration on a
characteristic length scale of $10\rm\ kpc$. It is scaled with the initial gas
density profile $\rho(r)$ as $B(\rm r)=B_{0}(\rho(r)/\rho_{0})^{2/3}$ where
$\rm B_{0}=20\rm\ \mu G$ and $\rho_{0}$ is the central density of the cluster.

Radiative cooling is calculated according to Sutherland & Dopita (1993) for
temperatures above $T>10^{4}\,\rm K$, with values extended below $10^{4}$ K
using Rosen & Bregman (1995). Metallicity is treated as a passive scalar
advected with the flow. It is initialised as
$Z=\min\left(0.45,\max\left(0.22,0.15\left(\frac{r}{r_{200}}\right)^{-0.28}\right)\right)$
using limits from Leccardi & Molendi (2008) and Urban et al. (2017). Star
formation proceeds in cells with hydrogen number density $n_{\rm H}>0.1\,\rm
H\,cm^{-3}$ and temperature $T<10^{4}\,\rm K$, at a stellar mass resolution of
$m_{*}=3.89\times 10^{5}\,\mbox{$\rm M_{\odot}$}$. Stellar feedback includes
type II supernovae only, using the energy-momentum model of Kimm et al. (2015)
using an efficiency of $\eta_{\rm SN}=0.2$ and a metal yield of 0.1

The black hole accrets according to the Bondi-Hoyle-Lyttleton accretion rate,
limited to a maximum Eddington fraction of 0.01 (Dubois et al. 2012). The
black hole spin is initialised at zero (spin parameter $a=0$), and evolves
throughout the simulation assuming a magnetically arrested disc McKinney et
al. (2012). A fraction $\epsilon_{\rm MAD}(a)$ of the accreted mass is
returned as AGN bipolar kinetic outflows aligned with the black hole spin
axis, using the implementation from Dubois et al. (2014, 2021). As this spin
vector naturally evolves under the influence of chaotic cold accretion on the
black hole (Gaspari et al. 2013), there is no need to add further jet
precession (Beckmann et al. 2019). In AGN_cr, a fraction $f_{\rm cr}=0.1$ of
the jet energy is injected into cosmic rays (see Beckmann et al. 2022, for
details). During AGN feedback, a passive scalar is injected at the jet base
which is used for refinement only. It decays with a decay time of 10 Myr.

The cluster profile is truncated at the virial radius ($r_{200}=1.9\rm\ Mpc$),
and embedded in a box of size 8.7 Mpc. Simulations were performed on a root
grid of $64^{3}$, adaptively refined to a maximum resolution of $\Delta
x=531\rm\ pc$ if any of the following criteria are met: 1) The gas mass in a
cell exceeds $[27089,8713,4098,1621,461,152,59,12,12]\times 1.47\times
10^{6}\rm\ M_{\odot}$ (levels 6 to 14). 2) The cell is located within $4\Delta
x$ of the black hole. 3) The cell density of the passive scalar injected by
the jet exceeds $\rho_{\rm scalar}/\rho_{\rm gas}>10^{-4}$ and its gradient
exceeds 10%.

Simulation parameters are summarised in Table 1.

Simulation | AGN | conduction | $f_{\rm c}$ | saturation
---|---|---|---|---
noAGN_nocond | no | no | - | -
noAGN_cond | no | anisotropic | from $\@vec{\hat{b}}$ | $f_{\rm sat}$
AGN_nocond | jet | no | - | -
AGN_cond | jet | anisotropic | from $\@vec{\hat{b}}$ | $f_{\rm sat}$
AGN_third | jet | anisotropic | 1/3 | $f_{\rm sat}$
AGN_iso | jet | isotropic | 1 | $f_{\rm sat}$
AGN_whistler | jet | anisotropic | from $\@vec{\hat{b}}$ | $f_{\rm sat,whist}$
AGN_cr* |  jet, $f_{\rm cr}=0.1$ | anisotropic | from $\@vec{\hat{b}}$ | $f_{\rm whist}$
Table 1: Simulation parameters. $f_{\rm c}$ is the fraction of Spitzer
conductivity used in the conductive heating rate of the simulation.
*Simulation AGN_cr has cosmic rays, and is identical to CRc_dsh_weak from
Beckmann et al. (2022).

##  3 Results

###  3.1 AGN jets and the HBI

|
---|---
|
|
Figure 1: Line integral convolution of the magnetic field, in the plane of the
image. Colourmaps are for visualization only. The black hole is marked as a
black cross, and the instantaneous jet direction by an orange line. The white
circle have radii of 50 (dashed) and 100 (solid) kpc respectively. In the
presence of conduction, the HBI re-orients the magnetic field in a tangential
configuration outside the central region.

As can be seen in the magnetic field morphology shown in Fig. 1, the presence
of a self-regulating AGN only affects the magnetic field orientation in the
central 50 kpc of the cluster. Without AGN, the central magnetic fields tend
to be radial, while with AGN they are effective randomised by the jet and
resulting turbulence. At larger radii, conduction drives the HBI to
tangentialize the magnetic field while in the absence of conduction, the
magnetic field remains tangled (with an AGN) or develops radial features
(without an AGN).

Figure 2: Radial profiles of $<b_{r}^{2}>$ for different simulations at two
different points in time. The initial conditions for all simulations are shown
as a black dotted line, and have on average $<b_{r}^{2}>$ $=1/3$, as expected
for a tangled magnetic field.

This evolution is shown quantitatively in Fig. 2 in the form of volume-
weighted average radial profiles of the radial component of the magnetic field
unit vector, $<b_{r}^{2}>$ , for concentric radial shells. For all
simulations, the initial conditions (dotted black line) are consistent with a
tangled magnetic field ($\mbox{$<b_{r}^{2}>$\leavevmode\nobreak\ }=1/3$). From
there, the evolution diverges: without conduction, $<b_{r}^{2}>$ increases
(noAGN_nocond and AGN_nocond). With conduction (noAGN_cond and AGN_cond,
AGN_whistler and AGN_cr), $<b_{r}^{2}>$ decreases out to the radius where the
cluster temperature profile turns over (grey vertical line, $r=358\rm\ kpc$ at
$t=0\rm\ Gyr$), i.e. within the region where the HBI can act.

Figure 3: Evolution of the jet direction for all AGN simulations over 3 Gyr of
evolution. Limits of the parameter space shown account for the symmetric
nature of the jet. All jets show significant reorientation over $3\rm\ Gyr$ of
evolution. A dot marks the jets final position at $3\rm Gyr$.

Low $<b_{r}^{2}>$ is a clear sign of the HBI in action, which in our
simulation continues to decrease $<b_{r}^{2}>$ over the full $3\rm\ Gyr$ of
evolution. This is despite the fact that, unlike in Karen Yang & Reynolds
(2016) and Su et al. (2019), our jets sweep out a significant volume of the
cluster centre over the $3\rm\ Gyr$ evolution studied here. This can be seen
in Fig. 3, which shows the direction of jet axis as a function of time. While
this more isotropic injection of turbulence is effective at counteracting the
HBI within the cluster centre, the limited extent of the jets meas values of
$<b_{r}^{2}>$ still fall as low as 0.1 by $r=100\rm\ kpc$ , which produces an
effective barrier to heat fluxes to the cluster centre. In our model, the jet
has a comparatively low average spin value of about $a=0.1$ (see Beckmann et
al. 2019, for a discussion of the BH spin evolution), which means the spin
direction is more easily adjusted as for a faster spinning BH. It is possible
that a jet with a more fixed direction would be able to inject turbulence to
larger radii, but this would come at the cost of affecting a smaller fraction
of the core volume. We will leave an investigation into the impact of the jet
model on the evolution of the magnetic field morphology to future work.

Another possibility to stabilize the HBI would be via strong magnetic fields,
as the HBI growth rate is damped on scales above $\rm H\gtrsim 20\lambda_{\rm
e}\beta$ (Quataert 2008, Eq. 33), where $\beta$ is the plasma-$\beta$. In our
clusters, the magnetic field strength after $3\rm\,Gyr$ of evolution ranges
from $4-7.5\,\mu\rm G$ in the centre to $0.5-1\,\mu\rm G$ at $r=100\rm\,kpc$.
This means $H\geq\sim 640\rm\,kpc$ in the centre and increases with radius, so
the HBI is not suppressed by magnetic tension on scales relevant to the
cluster cooling flow.

Figure 4: Volume-weighted radial profiles of the mean plasma beta (top) and
saturation coefficient $f_{\rm sat}$ (bottom) for two the two choices of
$f_{\rm sat}$ at three different points in time. At late times, conduction
saturates at a lower value in the presence of whistler waves than in their
absence.

Finally, whistler-wave modulated conduction could significantly delay the HBI.
As can be seen in Fig. 2, $<b_{r}^{2}>$ does indeed decrease more slowly for
AGN_whistler than for for AGN_cond, but there is still a significant reduction
in comparison to the random initial conditions. The presence of Whistler waves
therefore slows down, but does not eliminate, the evolution of the HBI in
galaxy clusters. This is due to the fact that for realistic magnetic field
strengths of a few micro Gauss in the cluster centre, the plasma-$\beta$ in
the cluster is sufficiently low such that $f_{\rm sat,whistler}\approx f_{\rm
sat}$ early on, as can be seen in Fig. 4. It is only as the magnetic field
decays away slowly throughout the simulation due to the lack of volume-filling
turbulence that plasma$-\beta$ increases and $f_{\rm sat,whistler}$ drops at
late times.

The conclusion that Whistler-limited saturation makes only a small difference
to the evolution of the HBI is in agreement with work by Berlok et al. (2021),
who studied the other conduction-driven instability, the magneto-thermal
instability, which is active in cluster outskirts, and also concluded that
whistler-based suppression of thermal conduction has only a small impact. We
note that using a reduced saturation-coefficient is a simplified approach. A
more complete treatment would be to self-consistently modelling the whistler-
wave energy density, source and loss terms (Drake et al. 2021), and that doing
so is likely to significantly change the conclusions on the impact of whistler
waves on conduction and the HBI in galaxy clusters.

$<b_{r}^{2}>$ increases again at small radii for both noAGN_cond and AGN_cond
but for different reasons: in AGN_cond, the AGN randomises the magnetic field
in the centre for an average $\mbox{$<b_{r}^{2}>$\leavevmode\nobreak\ }\sim
1/3$. In noAGN_cond, a strong cooling flow develops, which causes radial
inflows. Cooling flows are also responsible for the high values of
noAGN_nocond at all radii, and the late increase of $<b_{r}^{2}>$ for
AGN_nocond (See Section 3.2).

###  3.2 Thermal conduction and the cooling flow

Figure 5: Total cold gas mass ($T<10^{7}\rm\,K$, top) and cumulative AGN
energy (bottom) as a function of time. The inset shows a timeseries of
$\chi=\sum\rm E_{\rm AGN,nocond}/\sum\rm E_{\rm AGN,cond}$, where $\sum\rm
E_{\rm AGN,nocond}$ is the cumulative energy injected by the AGN up to time
$t$ in the simulation without conduction, while $\sum\rm E_{\rm AGN,cond}$ is
the same for simulations with different conduction prescriptions. The grey
shaded region reporting results from Kannan et al. (2016) and Barnes et al.
(2019). The presence of an AGN strongly reduces the cooling flow, but
conduction only makes a significant difference when fully isotropic.

The fact that both noAGN_nocond and noAGN_cond develop a strong cooling flow
can be seen in Fig. 5. Both simulations without AGN build up unrealistically
large quantities of cold gas in the cluster centre as the cooling of the hot
ICM proceeds unimpeded. noAGN_cond cools more slowly than noAGN_nocond, but
the reduction in total gas mass at 3 Gyr, $1.0\times 10^{11}\rm\,M_{\odot}$
for noAGN_cond versus $1.4\times 10^{11}\rm\,M_{\odot}$ for noAGN_nocond, is
insignificant to the overall cooling flow.

To further understand the potential impact of thermal conduction, we add three
simulations with a fixed $f_{\rm c}$ : noAGN_iso and AGN_iso have
$\mbox{$f_{\rm c}$\leavevmode\nobreak\ }=1$, i.e. conduction is fully
isotropic. AGN_third also has isotropic conduction but uses $\mbox{$f_{\rm
c}$\leavevmode\nobreak\ }=1/3$, equivalent to a fully tangled magnetic field.
When $f_{\rm c}$ is constant, any reorientation of the magnetic field will
have no impact on the heat flux. It is only when conduction is isotropic
(noAGN_iso), and therefore not influenced by the HBI, that conduction
significantly reduces the cooling flow. This is in good agreement with work by
Wagh et al. (2014), who also showed that isotropic thermal conduction
significantly reduces the formation of cold gas in galaxy clusters. However,
even isotropic thermal conduction is less efficient at regulating the cooling
flow than AGN feedback, as can be seen by comparing noAGN_iso to any of the
simulations with an AGN.

In the presence of AGN jets, cooling flows are strongly reduced as cold gas
building-up in the cluster activates the AGN which prevents further cooling
(see e.g. Li et al. 2015; Prasad et al. 2015; Li et al. 2017; Karen Yang &
Reynolds 2016; Beckmann et al. 2019, for more details on AGN regulation of
cluster cooling flows). Comparing AGN_nocond and AGN_cond in Fig. 5 shows that
while the AGN effectively regulates cluster cooling and prevents a run-away
cooling flow (top panel), anisotropic conduction makes little difference to
the long-term evolution of the cluster.

To compare the impact of thermal conduction on the ability of the AGN to self-
regulate the cluster, we define $\chi=\sum\rm E_{\rm AGN,nocond}/\sum\rm
E_{\rm AGN,cond}$ (inset, Fig. 5), where $\sum\rm E_{\rm AGN}(t)$ is the
cumulative energy injected by the AGN up to time $t$. In our simulations,
$\chi\geq 1.2$, i.e. the simulation without conduction requires at least 20%
more cumulative AGN energy than the case with conduction, for all simulations
only while $t<\rm\ 1\rm Gyr$, during which the cluster is still evolving away
from the initial conditions. At late times, at an average value of $\chi=1.24$
at $t>2\rm\ Gyr$, $\chi$ remains elevated for AGN_third, but drops to $\sim 1$
for AGN_cond and AGN_whistler. This is in contrast to Kannan et al. (2016) and
Barnes et al. (2019), who report $\chi=1.2-1.3$ at all times. One possibility
is that the HBI in Kannan et al. (2016) and Barnes et al. (2019) is being
offset by turbulence injected by the large-scale cosmological environment.
Support for this theory comes from the fact that fully tangled fields
(AGN_third) in our simulations also show $\chi=1.2-1.3$. However, Karen Yang &
Reynolds (2016) used isolated clusters and also report $\chi\approx 1.5$.
Another possibility is that as $\chi$ is very sensitive to the cluster mass
(Karen Yang & Reynolds 2016) so the different $\chi$ could be due to different
cluster masses or cluster profiles. A final possibility is that the higher
resolution in our simulation ($\Delta x=531\rm\ pc$ compared to $1.5-2\rm\
kpc$ for other studies) is responsible for the higher $f_{\rm c}$ (
$\mbox{$f_{\rm c}$\leavevmode\nobreak\ }\sim 0.2$ in Karen Yang & Reynolds
(2016) versus $\mbox{$f_{\rm c}$\leavevmode\nobreak\ }\sim 0.1$ here) and the
resulting difference in $\chi$.

Injecting a small fraction of AGN energy into cosmic rays (AGN_cr) increases
the efficiency of AGN jets in self-regulating clusters ($\rm\sum E_{\rm AGN}$
is reduced) but the cooling flow is only mildly affected. Further details on
the impact of cosmic rays on galaxy cluster cooling flows can be found in
Beckmann et al. (2022). It is only when the magnetic field becomes
preferentially radial ($\mbox{$f_{\rm c}$\leavevmode\nobreak\ }\rightarrow 1$)
that thermal conduction significantly reduces cold gas mass and cumulative AGN
energy. This is in agreement with Jacob & Pfrommer (2017) who also reported
values of $\mbox{$f_{\rm c}$\leavevmode\nobreak\ }>1/3$ for cosmic ray and
thermal conduction regulated steady-state solutions.

Figure 6: Net mass accretion rate through concentric radial shells averaged
over $3\rm\ Gyr$ of evolution in $500\rm\ Myr$ bins. The simulations without
AGN show steady, smooth inflows at all radii. Simulations with AGN show on
average lower accretion rates and more disturbed accretion patterns due to the
large-scale shocks propagating outwards from the central AGN.

Such cooling flows result in significant radial mass fluxes, as can be seen in
Fig. 6. Simulations without AGN show steady inflows at all radii, while those
with AGN show lower net accretion rates with more disturbed patterns.The flow
in simulations with AGN are less smooth as large-scale shocks propagating
outwards from the central AGN disturb the contraction of the ICM as it looses
thermal pressure support. In the very centre ($r<\rm 50\ kpc$), inflows in AGN
simulations are highly variable due to the turbulence and the multiphase
nature of the gas (Beckmann et al. 2019).

It is radial mass flows that create the high values of $<b_{r}^{2}>$ for
noAGN_nocond in Fig. 2. For ideal MHD, field-lines are ”frozen in”, i.e.
advected with the gas flow. As material falls onto the cluster centre, these
mass fluxes will drag the magnetic field in a radial configuration, which
increases $<b_{r}^{2}>$ . This radialisation of magnetic field lines proceeds
unimpeded without conduction (noAGN_nocond) but competes with the
tangentialisation due to the HBI in the presence of conduction. Comparing Fig.
6 with Fig. 2 shows that simulations with strong mass fluxes have on average
higher $<b_{r}^{2}>$ than equivalent simulations without strong mass fluxes.
For example, due to the absence of a strong cooling flow, $<b_{r}^{2}>$ is
lower at 100 kpc in AGN_cond than without AGN feedback (noAGN_cond) despite
the HBI being very active in both. Cooling flows also explain the late rise in
$<b_{r}^{2}>$ for AGN_nocond due to an excess of cold (Fig. 5), inflowing
(Fig. 6) gas at $t>2.5\rm\ Gyr$. Mass fluxes are high for noAGN_iso because
thermal conduction efficiently extracts thermal energy energy from
intermediate radii which causes the gas to loses pressure support and
contract.

##  4 Conclusions

In this paper, we investigated whether the presence of a black hole spin-
driven AGN jet can counteract the HBI in the centre of massive galaxy clusters
and allow for efficient thermal conduction to aid in the long-term self-
regulation of cluster cooling flows. We conclude that

  * •

Spin-driven AGN feedback is able to randomize the magnetic field in the
central $50\rm\ kpc$ of the cluster but not outside this region.

  * •

Whether an AGN is included or not, the HBI remains very active in the region
50 - 300 kpc from the cluster centre, which reduces the effective conductivity
to values as low as $\mbox{$f_{\rm c}$\leavevmode\nobreak\ }=0.1$ of Spitzer
conductivity.

  * •

Such low levels of thermal conduction have no significant influence on the
cluster cooling flow, or the AGN self-regulation thereof.

  * •

Whistler-wave driven saturation of thermal conduction reduces the magnitude of
the HBI in galaxy clusters, but does not prevent it.

  * •

Even if the HBI was inefficient, and the magnetic field remained tangled, the
resulting effective conductivity of $\mbox{$f_{\rm c}$\leavevmode\nobreak\
}=1/3$ Spitzer is not sufficiently high to influence the AGN self-regulated
cooling flow.

  * •

Only very high values of $f_{\rm c}$ , which would require predominantly
radial magnetic fields, transport sufficient thermal energy to reduce the
cluster cooling flow.

Our set of idealised simulations have demonstrated the inability of AGN jets
alone to re-randomise the magnetic field after it has been tangentially
aligned by the HBI, and, hence, to restore significant thermal conduction on
cluster scales. We have shown that thermal conduction does affect the
morphology of the magnetic field, which in turn could have important
consequences for where cosmic ray energy is deposited in the cluster (see
Beckmann et al. 2022). How the magnetic field evolves depends strongly on how
conduction is modelled, and on the initial conditions of the magnetic field.
We defer a more detailed study of the impact of magnetic field morphology on
the evolution of galaxy clusters to future work, together with a study of the
contribution of other sources of turbulence beyond AGN jets, such as large-
scale inflows, stirring by satellites and other substructure (Ruszkowski & Oh
2011; Bourne & Sijacki 2017). Other phenomena to be considered are anisotropic
thermal pressure due to Braginskii viscosity (Kunz et al. 2011).

###### Acknowledgements.

RSB And YD designed the project, RSB, YD and AP developed the code,
interpreted results and wrote the paper. RSB designed, executed and processed
the suite of simulations. FLP and VO contributed to discussion and
interpretation of results. This work was supported by the ANR grant LYRICS
(ANR-16-CE31-0011) and was granted access to the HPC resources of CINES under
the allocations A0080406955 and A0100406955 made by GENCI. This work has made
use of the Infinity Cluster hosted by Institut d’Astrophysique de Paris. We
thank Stéphane Rouberol for smoothly running this cluster for us.
Visualisations in this paper were produced using the yt project (Turk et al.
2011) and cartopy (Met Office 2015).

## References

  * Andreon et al. (2017) Andreon, S., Wang, J., Trinchieri, G., Moretti, A., & Serra, A. L. 2017, A&A, 606, A24
  * Barnes et al. (2019) Barnes, D. J., Kannan, R., Vogelsberger, M., et al. 2019, MNRAS, 488, 3003
  * Beckmann et al. (2019) Beckmann, R. S., Dubois, Y., Guillard, P., et al. 2019, A&A, 631, A60
  * Beckmann et al. (2022) Beckmann, R. S., Dubois, Y., Pellisier, A., et al. 2022, Preprint [arXiv:2204.03629]
  * Berlok et al. (2021) Berlok, T., Quataert, E., Pessah, M. E., & Pfrommer, C. 2021, MNRAS, 504, 3435
  * Binney & Tabor (1995) Binney, J. & Tabor, G. 1995, MNRAS, 276, 663
  * Bogdanović et al. (2009) Bogdanović, T., Reynolds, C. S., Balbus, S. A., & Parrish, I. J. 2009, ApJ, 704, 211
  * Bourne & Sijacki (2017) Bourne, M. A. & Sijacki, D. 2017, MNRAS, 472, 4707
  * Cowie & McKee (1977) Cowie, L. L. & McKee, C. F. 1977, ApJ, 211, 135
  * Dashyan & Dubois (2020) Dashyan, G. & Dubois, Y. 2020, A&A, 638, A123
  * Drake et al. (2021) Drake, J. F., Pfrommer, C., Reynolds, C. S., et al. 2021, ApJ, 923, 245
  * Dubois et al. (2021) Dubois, Y., Beckmann, R., Bournaud, F., et al. 2021, A&A, 651, A109
  * Dubois & Commerçon (2016) Dubois, Y. & Commerçon, B. 2016, A&A, 585, A138
  * Dubois et al. (2012) Dubois, Y., Devriendt, J., Slyz, A., & Teyssier, R. 2012, MNRAS, 420, 2662
  * Dubois et al. (2014) Dubois, Y., Volonteri, M., Silk, J., Devriendt, J., & Slyz, A. 2014, MNRAS, 440, 2333
  * Fabian (1994) Fabian, A. C. 1994, ARA&A, 32, 277
  * Fromang et al. (2006) Fromang, S., Hennebelle, P., & Teyssier, R. 2006, A&A, 457, 371
  * Gaspari et al. (2013) Gaspari, M., Ruszkowski, M., & Oh, S. P. 2013, MNRAS, 432, 3401
  * Jacob & Pfrommer (2017) Jacob, S. & Pfrommer, C. 2017, MNRAS, 467, stx131
  * Kannan et al. (2016) Kannan, R., Vogelsberger, M., Pfrommer, C., et al. 2016, ApJ, 837, L18
  * Karen Yang & Reynolds (2016) Karen Yang, H.-Y. & Reynolds, C. S. 2016, ApJ, 818, 181
  * Kimm et al. (2015) Kimm, T., Cen, R., Devriendt, J., Dubois, Y., & Slyz, A. 2015, MNRAS, 451, 2900
  * Komarov et al. (2018) Komarov, S., Schekochihin, A., Churazov, E., & Spitkovsky, A. 2018, J. Plasma Phys., 84, 905840305
  * Kunz et al. (2011) Kunz, M. W., Schekochihin, A. A., Cowley, S. C., Binney, J. J., & Sanders, J. S. 2011, MNRAS, 410, 2446
  * Leccardi & Molendi (2008) Leccardi, A. & Molendi, S. 2008, A&A, 487, 461
  * Li et al. (2015) Li, Y., Bryan, G. L., Ruszkowski, M., et al. 2015, ApJ, 811, 73
  * Li et al. (2017) Li, Y., Ruszkowski, M., & Bryan, G. G. L. 2017, ApJ, 847, 106
  * Maccio et al. (2007) Maccio, A. V., Dutton, A. A., Van Den Bosch, F. C., et al. 2007, MNRAS, 378, 55
  * McCourt et al. (2011) McCourt, M., Parrish, I. J., Sharma, P., & Quataert, E. 2011, MNRAS, 413, 1295
  * McKinney et al. (2012) McKinney, J. C., Tchekhovskoy, A., & Blandford, R. D. 2012, MNRAS, 423, 3083
  * Met Office (2015) Met Office. 2015, Cartopy: a cartographic python library with a Matplotlib interface
  * Miyoshi & Kusano (2005) Miyoshi, T. & Kusano, K. 2005, J. Comput. Phys., 208, 315
  * Narayan & Medvedev (2001) Narayan, R. & Medvedev, M. V. 2001, ApJ, 562, L129
  * Navarro et al. (1997) Navarro, J. F., Frenk, C. S., & White, S. D. M. 1997, ApJ, 490, 493
  * Parrish et al. (2012) Parrish, I. J., McCourt, M., Quataert, E., & Sharma, P. 2012, MNRAS, 422, 704
  * Parrish et al. (2009) Parrish, I. J., Quataert, E., & Sharma, P. 2009, ApJ, 703, 96
  * Phipps et al. (2019) Phipps, F., Bogdán, Á., Lovisari, L., et al. 2019, ApJ, 875, 141
  * Prasad et al. (2015) Prasad, D., Sharma, P., & Babul, A. 2015, ApJ, 811, 108
  * Quataert (2008) Quataert, E. 2008, ApJ, 673, 758
  * Roberg-Clark et al. (2016) Roberg-Clark, G. T., Drake, J. F., Reynolds, C. S., & Swisdak, M. 2016, ApJ, 830, L9
  * Rosen & Bregman (1995) Rosen, A. & Bregman, J. N. 1995, ApJ, 440, 634
  * Ruszkowski & Begelman (2002) Ruszkowski, M. & Begelman, M. C. 2002, ApJ, 581, 223
  * Ruszkowski et al. (2011) Ruszkowski, M., Lee, D., Brüggen, M., Parrish, I., & Oh, S. P. 2011, ApJ, 740, 81
  * Ruszkowski & Oh (2010) Ruszkowski, M. & Oh, S. P. 2010, ApJ, 713, 1332
  * Ruszkowski & Oh (2011) Ruszkowski, M. & Oh, S. P. 2011, MNRAS, 414, 1493
  * Santos (2000) Santos, S. D. 2000, MNRAS, 323, 930
  * Sarazin (1986) Sarazin, C. L. 1986, Rev. Mod. Phys., 58, 1
  * Sharma et al. (2007) Sharma, P., Quataert, E., Hammett, G. W., & Stone, J. M. 2007, ApJ, 667, 714
  * Spitzer & Härm (1953) Spitzer, L. & Härm, R. 1953, Phys. Rev., 89, 977
  * Su et al. (2019) Su, K.-Y., Hopkins, P. F., Hayward, C. C., et al. 2019, MNRAS, 487, 4393
  * Sutherland & Dopita (1993) Sutherland, R. S. & Dopita, M. A. 1993, Astrophys. J. Suppl. Ser., 88, 253
  * Teyssier (2002) Teyssier, R. 2002, A&A, 385, 337
  * Teyssier et al. (2006) Teyssier, R., Fromang, S., & Dormy, E. 2006, J. Comput. Phys., 218, 44
  * Tucker & Rosner (1983) Tucker, W. H. & Rosner, R. 1983, ApJ, 267, 547
  * Turk et al. (2011) Turk, M. J., Smith, B. D., Oishi, J. S., et al. 2011, Astrophys. J. Suppl. Ser., 192, 9
  * Urban et al. (2017) Urban, O., Werner, N., Allen, S. W., Simionescu, A., & Mantz, A. 2017, MNRAS, 470, 4583
  * Wagh et al. (2014) Wagh, B., Sharma, P., & McCourt, M. 2014, MNRAS, 439, 2822


# DoubleMatch:
Improving Semi-Supervised Learning with Self-Supervision

Erik Wallin1,2, Lennart Svensson2, Fredrik Kahl2, Lars Hammarstrand2 1Saab AB,
2Chalmers University of Technology
<EMAIL_ADDRESS>
###### Abstract

Following the success of supervised learning, semi-supervised learning (SSL)
is now becoming increasingly popular. SSL is a family of methods, which in
addition to a labeled training set, also use a sizable collection of unlabeled
data for fitting a model. Most of the recent successful SSL methods are based
on pseudo-labeling approaches: letting confident model predictions act as
training labels. While these methods have shown impressive results on many
benchmark datasets, a drawback of this approach is that not all unlabeled data
are used during training. We propose a new SSL algorithm, DoubleMatch, which
combines the pseudo-labeling technique with a self-supervised loss, enabling
the model to utilize all unlabeled data in the training process. We show that
this method achieves state-of-the-art accuracies on multiple benchmark
datasets while also reducing training times compared to existing SSL methods.
Code is available at https://github.com/walline/doublematch.

##  I Introduction

Supervised learning has gained much attention in recent years because of
remarkable achievements in fields such as image classification [1], object
detection [2], and natural language processing [3]. The impressive results are
typically fueled by vast amounts of labeled data with datasets such as
ImageNet [4] and COCO [5]. In many practical applications, however, labeled
data might be scarce or require expert domain knowledge to attain. In
contrast, _unlabeled data_ are often much easier to acquire, e.g., through web
scraping or unsupervised sensor recordings, thus creating a natural demand for
methods that can successfully learn from data _without_ prior labels.

Semi-supervised learning is the procedure of combining unlabeled data with a
(typically) much smaller set of labeled data for fitting a model. There are
two main ideas behind the most well-performing methods for semi-supervised
learning: _consistency regularization_ and _pseudo-labeling_. The former means
encouraging consistent predictions on unlabeled data across different views of
the same sample, e.g., through domain-specific data augmentation. Pseudo-
labeling, on the other hand, involves using confident model predictions on
unlabeled data as de facto training labels.

UDA [6] and FixMatch [7] recently gained wide recognition because of their
simple yet powerful frameworks for combining consistency regularization and
pseudo-labeling in semi-supervised learning. Among their reported results is,
e.g., an impressive classification accuracy of 94.93% on CIFAR-10 [8] using
only 250 labeled images for training [7]. However, despite their excellent
performance, these methods have the drawback of only enforcing consistency
regularization on unlabeled data with confident model predictions while harder
data samples are essentially discarded. For challenging datasets, in
particular, this means the model only uses a smaller part of unlabeled data
during training (the relatively easier part). We show that this inefficient
and incomplete use of unlabeled data leads to unnecessarily long training
times and, for some datasets, reductions in classification accuracy.

Figure 1: A unification of semi- and self-supervised frameworks. While many
existing methods for semi-supervised learning operate in the label space,
DoubleMatch operates in both the label and feature spaces for a more efficient
use of unlabeled data. Red arrows mark the flow of gradients.

An alternative to semi-supervised training is to instead train fully without
label information. This practice is referred to as self-supervised learning,
and the goal is not to predict classes but instead to learn feature
representations of the data that can be used for downstream tasks [9, 10, 11].
By focusing on learning the fundamental representations of the data, these
methods can more fully exploit the unlabeled training data and are in this
respect more efficient than many semi-supervised methods. However, if the end
task is classification, these methods do not extend the training data with
task-specific information as in the case of, e.g., UDA and FixMatch.

Analogously to semi-supervised learning, self-supervised learning is heavily
based on consistency regularization, which makes the two fields closely
related. Generally, methods for both semi- and self-supervised learning employ
a setting where a _teacher model_ predicts optimization targets for a _student
model_ , making gradients flow only through the student predictions. The
teacher and student models often share parameters, either by making the
teacher a moving average of the student or by using the same parameters.

We propose DoubleMatch, an extension of FixMatch, which solves its inefficient
use of unlabeled data by taking inspiration from research on self-supervised
learning. Fig. 1 illustrates how DoubleMatch unifies the typical setups for
semi- and self-supervised learning by operating both in the label and feature
space for unlabeled data. In the label space, as with FixMatch for confident
data, the model is evaluated based on predicted class distributions (_pseudo-
label loss_). Whereas, in the feature space, the model is assessed based on
the similarity of the predicted feature representations (_self-supervised
loss_). More specifically, with motivation from methods such as [11, 12], we
suggest adding a self-supervised feature loss to the FixMatch framework by
enforcing consistency regularization on _all_ unlabeled data in the feature
space. Moreover, we can add this term with minimal computational overhead
because we do not require additional augmentations or model predictions.

Our main contributions of this work are:

  1. 1.

We propose adding a self-supervised loss to the FixMatch framework.

  2. 2.

We demonstrate that our method leads to faster training times and increased
classification accuracy than previous SOTA across multiple datasets.

  3. 3.

We analyze the importance of a pseudo-labeling loss for different sizes of the
labeled training set.

##  II Related work

Research in semi-supervised learning dates back to the 1960s [13, 14, 15].
Over the last couple of years, research in semi-supervised learning has been
dominated by methods combining the use of consistency regularization and
pseudo-labeling [7, 6]. One of these methods is FixMatch [7], which this work
proposes an extension of.

###  II-A FixMatch

FixMatch [7] made a big impact on the field of semi-supervised learning
because of its well-performing yet simple method for combining consistency
regularization and pseudo-labeling. FixMatch is similar to the marginally
earlier UDA [6] although slightly less complex.

As proposed by ReMixMatch [16], FixMatch employs consistency regularization
through a setup with weak and strong data augmentations. The weak augmentation
consists of a random horizontal flip followed by a random image translation,
while the strong augmentation utilizes sharp image-transformations such as
RandAugment [17], CTAugment [16], and CutOut [18]. The strong domain-specific
augmentations for consistency regularization are highlighted as a crucial
component in the performance gain compared to many previous methods, which
use, e.g., domain-agnostic perturbations [19, 20, 21, 22] or MixUp-
augmentations [23, 24, 25].

FixMatch also makes use of pseudo-labeling, which can be compactly described
as the practice of using model predictions as training labels. While there are
many variations of this technique [26, 27, 28, 29], FixMatch follows the
confidence-based pseudo-labels introduced by UDA. The confidence-based pseudo-
labels in FixMatch are designed such that only predictions on unlabeled data
above a predetermined confidence-threshold are used for training. The
artificial label used in the cross-entropy loss is then the argmax of the
confident prediction.

The proposed method in this article is an extension of the FixMatch framework,
following both its augmentation scheme and pseudo-labeling strategy.

###  II-B Extensions of FixMatch

As a consequence of the simplicity and high performance of FixMatch, there
have been several subsequent studies trying to extend and improve on this
framework [30, 31, 32, 33, 34]. For example, FlexMatch [30] and Dash [31]
propose two different ways of introducing dynamic confidence-thresholds,
replacing the constant threshold of FixMatch. DP-SSL [32] generates pseudo-
labels using a data programming scheme with an ensemble of labeling functions,
each specialized in a subset of the classes in the classification problem. In
contrast to our work, methods such as [30, 31, 32, 34] aim to improve FixMatch
through alternative strategies for pseudo-label selections. Our proposed
method instead focuses on improving the consistency regularization of FixMatch
and thus could potentially be further improved by using pseudo-labeling
strategies proposed by [30, 31, 32, 34].

Figure 2: Graph showing loss evaluation for unlabeled images. Double slash
marks a stop-gradient operation.

###  II-C Self-supervised learning

This section gives a very brief overview of research in self-supervised
learning that has inspired our work. Fully without labels, the goal of self-
supervised learning is not to predict classes, but instead to learn
_representations_ of data that can be used for downstream tasks. Similarly to
semi-supervised learning, recent advances in self-supervised learning have
been largely reliant on consistency regularization through heavy use of
domain-specific data augmentation [9, 10, 12, 11, 35].

Influential works such as [9, 10] make use of the compute-heavy contrastive
loss for self-supervised learning, not only maximizing similarities between
augmentations of the same data but also _minimizing_ the agreement between
different data. However, more recent works [12, 11, 35] show that a
contrastive loss is not required to achieve competitive self-learning
performance. Instead, they show that it is sufficient to, as in, e.g.,
FixMatch, focus on maximizing similarities between augmentations of the same
data (consistency regularization). This means that self-supervised learning
losses similar to those in [12, 11, 35] can be integrated into, e.g., FixMatch
with a minimal additional computational load. For this work, we take
particular inspiration from SimSiam [11] by incorporating a similar self-
supervised loss into our method.

###  II-D Self-supervision in semi-supervised learning

Lastly, similarly to this paper, there has been some other work covering self-
supervision techniques in semi-supervised learning [36, 16, 37, 38, 39]. For
example, EnAET [36], ReMixMatch [16], and S4L [37] incorporate a self-
supervised task that involves predicting the parameters of some stochastic
image transformation that is applied to an unlabeled image. USADTM [38]
proposes a self-supervised loss that involves evaluating similarities between
_three_ different views of unlabeled data. Contrary to our proposed method,
the self-supervision in these methods requires introducing additional data
augmentations and, as in case of USADTM, a more complex pseudo-label
selection, all of which increases the computational load.

##  III Method

This section describes DoubleMatch, our proposed method. Many existing methods
for semi-supervised learning have two central terms in their loss functions: a
supervised loss term for labeled data and a pseudo-label loss term for
unlabeled data [7, 6, 27]. We propose to add a third, self-supervised loss
term, to fully utilize unlabeled data for faster training and increased
accuracy. This term acts as a natural extension with minimal computational
overhead to methods that utilize consistency regularization through data
augmentation. We choose to apply our idea to the FixMatch [7] framework but
the method proposed can easily be applied to similar frameworks such as UDA
[6]. The training algorithm of DoubleMatch for a single batch is summarized in
Algorithm 1 and detailed below.

For a $C$-class image classification problem, the supervised loss in FixMatch
is the standard cross-entropy loss given by

$l_{l}=\frac{1}{B}\sum_{i=1}^{B}H(y_{i},p_{i}),$ (1)

where $B$ is the batch size, $y_{i}\in\mathbb{R}^{C\times 1}$ is the ground-
truth one-hot label for image $i$, $p_{i}\in\mathbb{R}^{C\times 1}$ is the
predicted probability distribution for image $i$, and $H(x,y)=-\sum x\log y$
is the cross-entropy between two probability distributions.

The supervised loss is supplemented with a pseudo-label loss on unlabeled
data:

$l_{p}=\frac{1}{\mu B}\sum_{i=1}^{\mu
B}\mathds{1}\\{\max(w_{i})>\tau\\}H(\text{argmax}(w_{i}),q_{i}).$ (2)

Here, $\mu\in\mathbb{Z}^{+}$ and $\tau\in[0,1]$ are hyperparameters where
$\mu$ determines the ratio of labeled to unlabeled data in a batch and $\tau$
sets the confidence threshold for assigning pseudo-labels. The probability
distributions $w_{i}$ and $q_{i}$ are the predictions for a _weak_ and a
_strong_ augmentation, respectively, of the unlabeled sample $i$. We let
$\text{argmax}:\mathbb{R}^{C\times 1}\rightarrow\\{0,1\\}^{C\times 1}$ so that
it returns a one-hot vector, and $\mathds{1}\\{\cdot\\}$ is the indicator
function. Finally, we let the prediction on the weakly augmented image act as
the teacher, meaning we consider $w_{i}$ to be constant when back-propagating
through this loss term.

The two loss terms from (1) and (2) constitute the key components of the
FixMatch method. In our method, we add an additional consistency
regularization between the two different augmentations of unlabeled data. This
consistency regularization is enforced not on the predicted probability
distributions but on the _feature vectors_ from the penultimate layer of the
classification network. This added loss improves the data efficiency of the
method by operating on all unlabeled data, and not only for data with
confident predictions as in (2). Inspired by the SimSiam [11] method for self-
supervised learning, we use the negative cosine similarity for this loss.
Similarly to [11], in order to allow for differences between the feature
representations of weakly and strongly augmented versions of the same image,
we feed the feature representation of the strongly augmented image through a
trainable linear _projection head_ before evaluating the cosine similarity. We
end up with our proposed additional loss term:

$l_{s}=-\frac{1}{\mu B}\sum_{i=1}^{\mu B}\frac{h(v_{i})\cdot
z_{i}}{\|h(v_{i})\|\|z_{i}\|}=-\frac{1}{\mu B}\sum_{i=1}^{\mu
B}\cos(h(v_{i}),z_{i}).$ (3)

In this expression, $z_{i}\in\mathbb{R}^{d\times 1}$ is the output from the
penultimate layer of the classification network for the weakly augmented
version of image $i$. The dimension of this vector, $d$, is determined by the
network architecture. Similarly, $v_{i}\in\mathbb{R}^{d\times 1}$ is the
output from the penultimate layer for the strongly augmented version of image
$i$. However, $v_{i}$ is also fed through the trainable linear projection
head, $h:\mathbb{R}^{d\times 1}\rightarrow\mathbb{R}^{d\times 1}$. Again, the
prediction on the weakly augmented image acts as the teacher, so we consider
$z_{i}$ as constant when evaluating the gradient w.r.t. this loss term.
Finally, $\|\cdot\|$ is the $l^{2}$ norm.

For each batch, we let the full loss be

$l=l_{l}+l_{p}+w_{s}l_{s},$ (4)

where $w_{s}\in\mathbb{R}^{+}$ is a hyperparameter determining the weight of
the self-supervised loss term. We empirically find that it is important to
obtain a good balance between $l_{l}$ and $w_{s}l_{s}$ in this loss, meaning
that a well-tuned value for $w_{s}$ will be largely correlated with the number
of labeled training data (see Section IV-D). Note that our loss function is
identical to that used in FixMatch when $w_{s}=0$. A graph showing the loss
evaluation for unlabeled images is displayed in Fig. 2 where we divide the
network into three parts:

  * •

$f$, backbone: the network up to the final layer,

  * •

$g$, prediction head: the final layer of the classification network,

  * •

$h$, projection head: a single dimension-preserving linear layer for
transforming features of strongly augmented unlabeled images.

###  III-A Data augmentation

Data augmentation has proved to be a central component of self- and semi-
supervised learning [7, 10]. We follow one of the augmentation schemes used in
FixMatch [7] where the weak augmentation is a horizontal flip with probability
0.5 followed by a random translation with maximum distance 0.125 of the image
height. The strong augmentation stacks the weak augmentation, CTA [16], and
Cutout [18].

###  III-B Optimizer and regularization

For optimization, we stay close to the FixMatch settings and use SGD with
Nesterov momentum [40]. The learning rate, $\eta$, is set to follow a cosine
scheme [41] given by

$\eta=\eta_{0}\cos\left(\gamma\frac{\pi k}{2K}\right)$ (5)

where $\eta_{0}$ is the initial learning rate, $k$ is the current training
step and $K$ is the total number of training steps. We define one training
step as one gradient update in the SGD optimization. The decay rate is
determined by the hyperparameter $\gamma\in(0,1)$. Contrary to FixMatch which
uses a constant $\gamma$, we suggest tuning $\gamma$ for different datasets in
order to minimize overfitting.

Finally, we add weight-decay regularization to the loss as

$l_{w}=w_{d}\frac{1}{2}\left(\|\theta_{f}\|^{2}+\|\theta_{g}\|^{2}+\|\theta_{h}\|^{2}\right),$
(6)

where $\theta_{f}$, $\theta_{g}$ and $\theta_{h}$ are the vectors of
parameters in the backbone, prediction head and projection head, respectively,
and $w_{d}$ is a hyperparameter controlling the weight of this regularization
term. The weight-decay is identical to FixMatch with the exception that
DoubleMatch has the additional parameters from the projection head,
$\theta_{h}$.

1:Strong augmentation $\beta$, weak augmentation $\alpha$, labeled batch
$\\{(x_{1},y_{1}),\cdots,(x_{B},y_{B})\\}$, unlabeled batch
$\\{\tilde{x}_{1},\cdots,\tilde{x}_{\mu B}\\}$, unsupervised loss weight
$w_{s}$, weight decay parameter $w_{d}$, confidence threshold $\tau$, backbone
model $f$, prediction layer $g$, projection layer $h$

2:

3:$\triangleright$ Cross-entropy loss for (weakly augmented) labeled data

4:for $i=1,\cdots,B$ do

5: $p_{i}=g\circ f(\alpha(x_{i}))$

6:end for

7:$l_{l}=\frac{1}{B}\sum_{i=1}^{B}H(y_{i},p_{i})$

8:

9:$\triangleright$ Predictions on unlabeled data

10:for $i=1,\cdots,\mu B$ do

11: $z_{i}=f(\alpha(\tilde{x}_{i}))$ $\triangleright$ Weak augmentation

12: $v_{i}=f(\beta(\tilde{x}_{i}))$ $\triangleright$ Strong augmentation

13: $q_{i}=g(v_{i})$

14: $w_{i}=\texttt{stopgrad}(g(z_{i}))$

15:end for

16:$\triangleright$ Self-supervised cosine similarity

17:$l_{s}=-\frac{1}{\mu B}\sum_{i=1}^{\mu
B}\cos(h(v_{i}),\texttt{stopgrad}(z_{i}))$

18:$\triangleright$ Cross-entropy with pseudo-labels

19:$l_{p}=\frac{1}{\mu B}\sum_{i=1}^{\mu
B}\mathds{1}\\{\max(w_{i})>\tau\\}H(\text{argmax}(w_{i}),q_{i})$

20:return
$l_{l}+l_{p}+w_{s}l_{s}+w_{d}\frac{1}{2}\left(\|\theta_{f}\|^{2}+\|\theta_{g}\|^{2}+\|\theta_{h}\|^{2}\right)$

Algorithm 1 DoubleMatch algorithm TABLE I: Error rates on different datasets using different sizes for the labeled training set. DoubleMatch achieves state-of-the-art results on many combinations. | CIFAR-10 | CIFAR-100 | SVHN | STL-10
---|---|---|---|---
Method | 40 labels | 250 labels | 4000 labels | 400 labels | 2500 labels | 10000 labels | 40 labels | 250 labels | 1000 labels | 1000 labels
MixMatch [7, 24] |  47.54$\pm$11.50 |  11.05$\pm$0.86 |  6.42$\pm$0.10 |  67.61$\pm$1.32 |  39.94$\pm$0.37 |  28.31$\pm$0.33 |  42.55$\pm$14.53 |  3.98$\pm$0.23 |  3.50$\pm$0.28 |  10.41$\pm$0.61
UDA [7, 6] |  29.05$\pm$5.93 |  8.82$\pm$1.08 |  4.88$\pm$0.18 |  59.28$\pm$0.88 |  33.13$\pm$0.22 |  24.50$\pm$0.25 |  52.63$\pm$20.51 |  5.69$\pm$2.76 |  2.46$\pm$0.24 |  7.66$\pm$0.56
ReMixMatch [7, 16] |  19.10$\pm$9.64 |  5.44$\pm$0.05 |  4.72$\pm$0.13 |  44.28$\pm$2.06 |  27.43$\pm$0.31 |  23.03$\pm$0.56 |  3.34$\pm$0.20 |  2.92$\pm$0.48 |  2.65$\pm$0.08 |  5.23$\pm$0.45
FixMatch (CTA) [7] |  11.39$\pm$3.35 |  5.07$\pm$0.33 |  4.31$\pm$0.15 |  49.95$\pm$3.01 |  28.64$\pm$0.24 |  23.18$\pm$0.11 |  7.65$\pm$7.65 |  2.64$\pm$0.64 |  2.36$\pm$0.19 |  5.17$\pm$0.63
DP-SSL [32] | 6.54$\pm$0.98 | 4.78$\pm$0.26 |  4.23$\pm$0.20 |  43.17$\pm$1.29 |  28.00$\pm$0.79 |  22.24$\pm$0.31 | 2.98$\pm$0.86 | 2.16$\pm$0.36 | 1.99$\pm$0.18 |  4.97$\pm$0.42
Dash (CTA) [31] |  9.16$\pm$4.31 | 4.78$\pm$0.12 | 4.13$\pm$0.06 |  44.83$\pm$1.36 |  27.85$\pm$0.19 |  22.77$\pm$0.21 |  3.14$\pm$1.60 |  2.38$\pm$0.29 |  2.14$\pm$0.09 | 3.96$\pm$0.25
DoubleMatch (last 20) |  14.02$\pm$5.71 |  5.72$\pm$0.51 |  4.83$\pm$0.17 | 42.61$\pm$1.15 | 27.48$\pm$0.19 | 21.69$\pm$0.26 |  16.50$\pm$13.73 |  2.58$\pm$0.53 |  2.25$\pm$0.09 |  4.46$\pm$0.20
DoubleMatch (min) |  13.59$\pm$5.60 |  5.56$\pm$0.42 |  4.65$\pm$0.17 | 41.83$\pm$1.22 | 27.07$\pm$0.26 | 21.22$\pm$0.17 |  15.37$\pm$11.81 |  2.37$\pm$0.35 |  2.10$\pm$0.07 |  4.35$\pm$0.20

##  IV Experiments/results

In this section we evaluate our method on a set of benchmark datasets for
image classification. We use the datasets CIFAR-10 [8], CIFAR-100 [8], SVHN
[42] and STL-10 [43] with different sizes of the labeled training set. We
compare our results with reported error rates from similar works. For fair
comparisons, we choose works that state using similar experimental setups as
us with the same data folds and the same architectures [7, 31, 32]. The
results are shown in Table I where we present mean and standard deviation of
the error rate on the test set using five different data folds. We choose to
report both the minimum error rate for the full training run, and the median
for the 20 last evaluations. FixMatch [7] uses the median of the last
evaluations for their results, while others are not clear on this point.

We use an exponential moving average of the model parameters (with momentum
0.999) to evaluate the performance on the test set. We train DoubleMatch using
$352,000$ training steps with batch size $B=64$ in all our experiments. The
method is implemented in TensorFlow and is based on the FixMatch codebase. All
experiments are carried out on Nvidia A100 GPUs. The full list of
hyperparameters can be found in Section IV-D.

###  IV-A Classification results

####  IV-A1 CIFAR-10 and SVHN

The datasets CIFAR-10 and SVHN both consist of color images of size $32\times
32$. Both datasets contain ten classes where CIFAR-10 has classes such as dog,
horse and ship while the classes in SVHN are the ten digits. CIFAR-10 has a
test set of 10,000 images and a training set of 50,000 images. SVHN has 26,032
images for testing and 73,257 for training. For these datasets we use a Wide
ResNet-28-2 [44] with 1.5M parameters. This architecture makes the dimension
of our feature vectors ${d=128}$. Even though we obtain competitive results on
many of the splits, we do perform worse than SOTA, especially in the very-low
label regime.

####  IV-A2 CIFAR-100

CIFAR-100 is similar to CIFAR-10 in that it consists of color images of size
$32\times 32$ with training and test sets of size 50,000 and 10,000
respectively. However, CIFAR-100 contains 100 classes, making it a much more
challenging classification problem. For this dataset, we use the Wide
ResNet-28-8 architecture with 24M parameters, resulting in $d=512$. On this
dataset, we achieve SOTA results across all splits, not only beating FixMatch
and ReMixMatch, but also the more recent methods DP-SSL and Dash.

####  IV-A3 STL-10

The dataset STL-10 comprises color images of size $96\times 96$ belonging to
ten classes. It has a labeled training set of 5,000 images and a unlabeled
training set of 100,000 images. Its distribution of unlabeled data is wider
than the labeled distribution, meaning that the unlabeled set contains classes
that are not present in the labeled set. Here we use a Wide ResNet-37-2 with
6M parameters, making $d=256$. On this dataset, we achieve a very competitive
error rate, surpassed only by the result reported by Dash.

###  IV-B Training speed

Running FixMatch for its full training duration on CIFAR-100 using a single
A100 GPU takes 5 days of wall-time. DoubleMatch reduces these long training
times by more efficiently making use of unlabeled data through the added self-
supervised loss. While the methods we use as comparison in Table I run for
little more than 1M training steps, we run DoubleMatch for only roughly a
third of that. A clear illustration of our increase in training speed is seen
in Fig. 3 where we compare DoubleMatch to FixMatch during training runs on
CIFAR-100 with 10,000 labeled training data and on STL-10 with 1,000 labeled
data.

$74$$78$AccuracyCIFAR-100$0$$0.5$$1$$\cdot 10^{6}$$94$$96$Training
stepAccuracySTL-10DoubleMatchFixMatch Figure 3: Comparing FixMatch to
DoubleMatch on training runs on CIFAR-100 and STL-10. We see that DoubleMatch
reaches higher accuracies with roughly one third of the training steps.

###  IV-C Discussion

We present an extension of FixMatch, which outperforms the original work in
terms of prediction accuracy and training efficiency on CIFAR-100, STL-10, and
2/3 splits of SVHN. We note a particularly high performance on CIFAR-100 (new
SOTA) and STL-10. Our hypothesis for the results on these two datasets is that
the self-supervised loss contributes to the biggest improvement when it is
difficult to reach high classification accuracies on the unlabeled training
set. Low accuracies on the unlabeled set could be either due to 1) the
difficulty of the classification problem, as in the case with CIFAR-100, or 2)
due to the unlabeled data coming from a wider distribution than the labeled
training set, as in the case with STL-10. For CIFAR-10 and SVHN, it is
generally possible to reach high accuracies on the unlabeled training set so
the quality of the pseudo-labels can be very high, making a self-supervised
loss less important.

In Table I, we include Dash [31] and DP-SSL [32]. These are recent methods
that, similarly to us, aim to improve the FixMatch framework. However,
contrary to DoubleMatch, these methods focus on improving the pseudo-labeling
aspect: Dash through a dynamic confidence threshold and DP-SSL by an ensemble
of labeling functions. Both of these methods report improvements compared to
FixMatch. A future line of work could be combining these improved methods for
pseudo-label selections with the self-supervised loss from DoubleMatch.

###  IV-D Hyperparameters

For CIFAR-10, SVHN and STL-10 we use $\gamma=7/8$ and $w_{d}=0.0005$. For
CIFAR-100 we use $\gamma=5/8$ and $w_{d}=0.001$. For $w_{s}$, we use $10,5,2$
for CIFAR-100 with 10,000, 2,500 and 400 labels respectively; $5,1,0.5$ for
CIFAR-10 with 4,000, 250 and 40 labels; $0.05,0.05,0.001$ for SVHN with 1,000,
250 and 40 labels; and $1.0$ for STL-10 with 1,000 labels. We use
$\eta_{0}=0.3$, $\mu=7$, $B=64$, $\tau=0.95$ and SGD momentum 0.9 for all
datasets. We run all experiments for 352,000 training steps.

##  V Ablation

In this section we cover two ablation studies related to our method. First, we
show results from experiments where the cosine similarity in our self-
supervised loss is replaced by other similarity functions. The second ablation
study regards the importance of the pseudo-labeling loss for different sizes
of the labeled training set.

###  V-A Self-supervised loss functions

In DoubleMatch we use a cosine similarity for the self-supervised loss term
according to (3). We have conducted experiments with other loss functions. One
alternative to the cosine similarity is a simple mean squared error, as used
in, e.g., [20]:

$l_{s}=\frac{1}{\mu Bd}\sum_{i=1}^{\mu
B}(h(v_{i})-z_{i})^{T}(h(v_{i})-z_{i}).$ (7)

Another alternative, as done in, e.g., [35] is to apply the softmax function
to the feature vectors and then calculate the cross-entropy loss as

$l_{s}=\frac{1}{\mu B}\sum_{i=1}^{\mu
B}H(\sigma(h(v_{i})),\sigma(z_{i}/\lambda)).$ (8)

Here, $\lambda$ is a parameter that can be used to sharpen the resulting
distribution for the prediction of weakly augmented data. The standard softmax
function, $\sigma(\cdot)$, is given by

$\sigma_{k}(x)=\frac{e^{x_{k}}}{\sum_{j=1}^{D}e^{x_{j}}}\quad\text{for}\quad
x\in\mathbb{R}^{D\times 1},$ (9)

where $\sigma_{k}$ and $x_{k}$ are the k:th elements of $\sigma$ and $x$
respectively.

We have evaluated the different loss function on CIFAR-100 with 10,000 labels,
where we made training runs using 1) MSE 2) Softmax with $\lambda=1$ and 3)
Softmax with $\lambda=0.1$. The self-supervised weight, $w_{s}$, is re-tuned
for every loss function. The results are shown in Table II. When comparing to
the other functions, we note that a considerably lower error rate is reached
using the cosine similarity, indicating that this is indeed the correct choice
for DoubleMatch.

TABLE II: Evaluations of different functions for the self-supervised loss in DoubleMatch on CIFAR-100 with 10,000 labels. Loss function | Error rate | $w_{s}$
---|---|---
Cosine |  21.22$\pm$0.17 | 10.0
MSE | 23.91 | 0.25
Softmax ($\lambda=1$) | 23.23 | 1.0
Softmax ($\lambda=0.1$) | 23.57 | 0.5

###  V-B Importance of pseudo-labels

Lastly, we analyze the importance of pseudo-labels in DoubleMatch for
different sizes of the labeled training set. These experiments are conducted
on CIFAR-100. Here, we evaluate the difference in accuracy between DoubleMatch
with and without the pseudo-labeling loss, ($l_{p}$ in (2)). The weight for
the self-supervised loss, $w_{s}$, is re-tuned for each split after removing
$l_{p}$. The drop in accuracy by removing $l_{p}$ for different numbers of
labeled training data is shown in Table III. We note that there is nearly no
loss in performance by removing $l_{p}$ when using 10,000 labels. However,
when moving towards fewer labels, this reduction in accuracy increases
monotonously to roughly 8.5 for 400 labels.

While DoubleMatch still reaches SOTA results for CIFAR-100 with 400 labels,
these findings seem to be in line with our poor results on CIFAR-10 and SVHN
with 40 labels, indicating that our added self-supervised loss does not do
well on its own in the low-label regime. It is also consistent with results
reported from Dash [31] and DP-SSL [32] indicating that improved pseudo-label
selections seem to be more important in the low-label regime. However, with
enough labels, the pseudo-label loss can be completely replaced by a self-
supervised loss with little to no loss in performance.

TABLE III: Reduction in test accuracy on CIFAR-100 by removing the pseudo-label loss from DoubleMatch for different sizes of the labeled training set. | Nr. of labeled training data
---|---
| 400 | 1,000 | 2,500 | 10,000
Reduction | 8.46 | 3.81 | 2.20 | 0.39

##  VI Conclusion

This paper shows that using a self-supervised loss in semi-supervised learning
can lead to reduced training times and increased test accuracies for multiple
datasets. In particular, we present new SOTA results on CIFAR-100 using
different sizes of the labeled training set while using fewer training steps
than existing methods. Additionally, we present interesting findings showing
that, with enough labeled training data, the pseudo-labeling loss can be
removed with no performance reduction in the presence of a self-supervised
loss.

## Acknowledgment

This work was supported by Saab AB, the Swedish Foundation for Strategic
Research, and Wallenberg AI, Autonomous Systems and Software Program (WASP)
funded by the Knut and Alice Wallenberg Foundation. The experiments were
enabled by resources provided by the Swedish National Infrastructure for
Computing (SNIC) at Chalmers Centre for Computational Science and Engineering
(C3SE), and National Supercomputer Centre (NSC) at Linköping University.

## References

  * [1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021\.
  * [2] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _IEEE conference on computer vision and pattern recognition_ , 2016.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017.
  * [4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in _IEEE conference on computer vision and pattern recognition_ , 2009.
  * [5] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _European conference on computer vision_ , 2014.
  * [6] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, “Unsupervised data augmentation for consistency training,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [7] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “FixMatch: Simplifying semi-supervised learning with consistency and confidence,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [8] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” University of Toronto, Tech. Rep., 2009.
  * [9] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [10] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_ , 2020.
  * [11] X. Chen and K. He, “Exploring simple siamese representation learning,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [12] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent - a new approach to self-supervised learning,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [13] H. Scudder, “Probability of error of some adaptive pattern-recognition machines,” _IEEE Transactions on Information Theory_ , vol. 11, no. 3, pp. 363–371, 1965.
  * [14] S. Fralick, “Learning to recognize patterns without a teacher,” _IEEE Transactions on Information Theory_ , vol. 13, no. 1, pp. 57–64, 1967.
  * [15] A. Agrawala, “Learning with a probabilistic teacher,” _IEEE Transactions on Information Theory_ , vol. 16, no. 4, pp. 373–379, 1970.
  * [16] D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, “ReMixMatch: Semi-supervised learning with distribution matching and augmentation anchoring,” in _International Conference on Learning Representations_ , 2020.
  * [17] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “RandAugment: Practical automated data augmentation with a reduced search space,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020.
  * [18] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _arXiv preprint arXiv:1708.04552_ , 2017.
  * [19] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, “Semi-supervised learning with ladder networks,” in _Advances in Neural Information Processing Systems_ , 2015.
  * [20] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” in _International Conference on Learning Representations_ , 2017.
  * [21] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [22] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 8, pp. 1979–1993, 2018.
  * [23] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _International Conference on Learning Representations_ , 2018.
  * [24] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “MixMatch: A holistic approach to semi-supervised learning,” in _Advances in Neural Information Processing Systems_ , 2019.
  * [25] V. Verma, A. Lamb, J. Kannala, Y. Bengio, and D. Lopez-Paz, “Interpolation consistency training for semi-supervised learning,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , 2019.
  * [26] D. Yarowsky, “Unsupervised word sense disambiguation rivaling supervised methods,” in _33rd annual meeting of the association for computational linguistics_ , 1995.
  * [27] D.-H. Lee _et al._ , “Pseudo-Label: The simple and efficient semi-supervised learning method for deep neural networks,” in _Workshop on challenges in representation learning, ICML_ , 2013.
  * [28] H. Pham, Z. Dai, Q. Xie, and Q. V. Le, “Meta pseudo labels,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [29] M. N. Rizve, K. Duarte, Y. S. Rawat, and M. Shah, “In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning,” in _International Conference on Learning Representations_ , 2021.
  * [30] B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki, “FlexMatch: Boosting semi-supervised learning with curriculum pseudo labeling,” in _Advances in Neural Information Processing Systems_ , 2021\.
  * [31] Y. Xu, L. Shang, J. Ye, Q. Qian, Y.-F. Li, B. Sun, H. Li, and R. Jin, “Dash: Semi-supervised learning with dynamic thresholding,” in _International Conference on Machine Learning_ , 2021.
  * [32] Y. Xu, J. Ding, L. Zhang, and S. Zhou, “DP-SSL: Towards robust semi-supervised learning with a few labeled samples,” in _Advances in Neural Information Processing Systems_ , 2021.
  * [33] Z. Hu, Z. Yang, X. Hu, and R. Nevatia, “SimPLE: Similar pseudo label exploitation for semi-supervised classification,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [34] I. Nassar, S. Herath, E. Abbasnejad, W. Buntine, and G. Haffari, “All labels are not created equal: Enhancing semi-supervision via label grouping and co-training,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [35] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [36] X. Wang, D. Kihara, J. Luo, and G.-J. Qi, “EnAET: A self-trained framework for semi-supervised and supervised learning with ensemble transformations,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 1639–1647, 2020.
  * [37] X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer, “S4L: Self-supervised semi-supervised learning,” in _IEEE/CVF International Conference on Computer Vision_ , 2019.
  * [38] T. Han, J. Gao, Y. Yuan, and Q. Wang, “Unsupervised semantic aggregation and deformable template matching for semi-supervised learning,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [39] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton, “Big self-supervised models are strong semi-supervised learners,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [40] Y. E. Nesterov, “A method for solving the convex programming problem with convergence rate O(1/k2),” in _Dokl. akad. nauk Sssr_ , 1983.
  * [41] I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in _International Conference on Learning Representations_ , 2017\.
  * [42] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in _NIPS Workshop on Deep Learning and Unsupervised Feature Learning_ , 2011.
  * [43] A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , 2011.
  * [44] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in _Proceedings of the British Machine Vision Conference (BMVC)_ , 2016.
  * [45] M. Larsson, E. Stenborg, C. Toft, L. Hammarstrand, T. Sattler, and F. Kahl, “Fine-grained segmentation networks: Self-supervised segmentation for improved long-term visual localization,” in _IEEE/CVF conference on computer vision and pattern recognition_ , 2019.