# Polarization of Intrabinary Shock Emission in Spider Pulsars

Andrew G. Sullivan Kavli Institute for Particle Astrophysics and Cosmology,
Department of Physics, Stanford University, Stanford, CA 94305, USA  Roger W.
Romani Kavli Institute for Particle Astrophysics and Cosmology, Department of
Physics, Stanford University, Stanford, CA 94305, USA

###### Abstract

In ‘spider’ pulsars, the X-ray band is dominated by Intrabinary Shock (IBS)
synchrotron emission. While the double-peaked X-ray light curves from these
shocks have been well characterized in several spider systems (both black
widows and redbacks), the polarization of this emission is yet to be studied.
Motivated by the new polarization capability of the Imaging X-ray Polarization
Explorer (IXPE) and the confirmation of highly ordered magnetic fields in
pulsar wind nebulae, we model the IBS polarization, employing two potential
magnetic field configurations: toroidal magnetic fields imposed by the pre-
shock pulsar wind, and tangential shock-generated fields, which follow the
post-shock flow. We find that if IBSs host ordered magnetic fields, the
synchrotron X-rays from spider binaries can display a high degree of
polarization ($\gtrsim 50\%$), while the polarization angle variation provides
a good probe of the binary geometry and the magnetic field structure. Our
results encourage polarization observational studies of spider pulsars, which
can distinguish the proposed magnetic models and better constrain unique
properties of these systems.

Pulsars (1306) – Binary pulsars (153)

##  1 Introduction

Spider binaries contain a millisecond pulsar and a low-mass companion star in
a tight orbit with period $P_{b}\lesssim 1$ day. Spiders are often classed as
black widows (Fruchter et al., 1988), with sub-stellar $M_{c}<<0.1$ M⊙
companions or as redbacks (Roberts, 2013), with $M_{c}\approx 0.1-0.4$ M⊙. In
these systems, the pulsar irradiates the companion and drives a stellar wind
(Kluzniak et al., 1988; van Paradijs et al., 1988). The relativistic pulsar
wind and massive companion wind collide, forming an intrabinary shock (IBS).
These sources emit across the electromagnetic spectrum, with radio and gamma-
ray emission from the pulsar itself, optical emission from the companion star
and X-ray emission dominated by the IBS. In redbacks, the companion wind
momentum dominates that of the pulsar, causing the IBS to wrap around the
pulsar, while in black widows, the IBS wraps around the companion (Romani &
Sanchez, 2016).

The pulsar wind is strongly magnetized, so the shock-accelerated particles
emit prominent synchrotron X-rays in the post-shock flow (Arons & Tavani,
1993; Kandel et al., 2019; van der Merwe et al., 2020; Kandel et al., 2021).
This flow accelerates to mildly relativistic speeds, so that the non-thermal
IBS orbital light curves often display two caustic peaks, associated with
Doppler-beamed emission from relativistic particles traveling tangent to the
instantaneous line of sight. The light curve is quite sensitive to the
geometry of the IBS, with the wind momentum ratio and observer viewing angle
determining the separation of the two peaks. Conversely, the IBS spectrum
reveals much about the acceleration and cooling of the particles in the post-
shock flow.

Synchrotron emission from organized fields is polarized. While synchrotron
X-ray light curve and spectral analyses have been conducted on a number of
redbacks and black widows (e.g. Gentile et al., 2014; Romani & Sanchez, 2016;
Kandel et al., 2019, 2021; Perez et al., 2023), the X-ray polarization of
these sources has not yet been studied. New X-ray capabilities such as IXPE
(Weisskopf et al., 2016, 2022) may make this possible, affording new
information on the IBS magnetic field geometry. IXPE has already demonstrated
that pulsar wind nebulae (PWNe) have very high synchrotron polarization (Xie
et al., 2022; Bucciantini et al., 2023), suggesting similar features in
spiders, which have even harder spectra.

In this paper, we present a pilot study of IBS polarization, exploring some
simple field models (Sec. 2), and computing the expected polarization
signatures in realistic spider geometries (sec. 3). Our models use a semi-
analytic thin shock treatment, with analytic expressions for the IBS contact
discontinuity shape as well as particle radiation and cooling, as in Kandel et
al. (2019). This allows for rapid model generation to explore parameter space
and fit data. These models capture well the observed IBS pulse shapes and
spectra and fits can constrain a system’s geometrical parameters, although
omitting the detailed post-shock spreading that would be captured in
magnetohydrodynamic (MHD) models. The prospects for detection and the
potential for deeper probes of the IBS structure are briefly discussed in Sec.
4.

##  2 IBS Synchrotron Polarization

In the IBS, shocked pulsar wind electrons and positrons become accelerated to
very high energies (Sironi & Spitkovsky, 2011; Cortés & Sironi, 2022) and flow
along the shock surface (Bogovalov et al., 2008; Bosch-Ramon et al., 2015;
Malkov & Lemoine, 2023). The magnetic fields, either remnants of the shock-
compressed, incompletely cancelled striped wind fields, or MHD instability-
generated fields, stretched along the post-shock flow, allow particle cooling
via synchrotron radiation. If sufficiently uniform, this field induces high
polarization (Westfold, 1959; Rybicki & Lightman, 1979) perpendicular to the
projection of the magnetic field on the sky. While the complex 3-D structure
of the IBS means that such field directions vary, the beaming of the caustic
peaks guarantees that a sub-set of these directions dominates at a given
orbital phase. This raises the prospect of substantial net polarization and
motivates a detailed computation of the IBS synchrotron emission.

###  2.1 Polarized Synchrotron Radiation

The total synchrotron power spectrum per particle is (Rybicki & Lightman,
1979; Kandel et al., 2019, e.g.)

$P(\omega,\gamma)=\frac{\sqrt{3}q^{3}B\sin\alpha}{2\pi
mc^{2}}F\left(\frac{\omega}{\omega_{c}(\gamma)}\right),$ (1)

where $q$ and $m$ are the charge and mass of the radiating particle, $B$ is
the magnetic field strength, $\alpha$ is the pitch angle between the particle
velocity and the magnetic field, $\omega_{c}(\gamma)\equiv
3qB\gamma^{2}\sin\alpha/2mc$ is the characteristic synchrotron frequency,
$\gamma$ is the particle Lorentz factor, $F(x)\equiv
x\int_{x}^{\infty}K_{5/3}(y)dy$, and $K_{n}(x)$ is the modified Bessel
function of order $n$. The power radiated along and perpendicular to the
magnetic field direction are

$P_{||}(\omega,\gamma)=\frac{\sqrt{3}q^{3}B\sin\alpha}{4\pi
mc^{2}}\left[F\left(\frac{\omega}{\omega_{c}}\right)-G\left(\frac{\omega}{\omega_{c}(\gamma)}\right)\right],$
(2a) $P_{\perp}(\omega,\gamma)=\frac{\sqrt{3}q^{3}B\sin\alpha}{4\pi
mc^{2}}\left[F\left(\frac{\omega}{\omega_{c}}\right)+G\left(\frac{\omega}{\omega_{c}(\gamma)}\right)\right],$
(2b)

where $G(x)\equiv xK_{2/3}(x)$. The emission projected to the sky direction
$\vec{n}$ will be polarized with polarization vector
$\vec{e}=\vec{n}\times\vec{b}$, where $\vec{b}$ is the magnetic field
direction vector. For an angle $\chi$ between the magnetic field and a
particular reference direction in the plane of the sky, the Stokes parameters
for linear polarization from an individual emission zone (Trippe, 2014, e.g.,)
are

$Q=I(0)-I\left(\frac{\pi}{2}\right)=(P_{\perp}-P_{||})(\sin^{2}\chi-\cos^{2}\chi),$
(3a)
$\begin{split}U=&I\left(\frac{\pi}{4}\right)-I\left(\frac{3\pi}{4}\right)\\\
&=(P_{\perp}-P_{||})\left[\sin^{2}\left(\chi-\frac{\pi}{4}\right)-\cos^{2}\left(\chi-\frac{\pi}{4}\right)\right],\end{split}$
(3b)

where $I(\theta)$ is the intensity along the direction with angle $\theta$ to
the reference direction. The polarization degree $\Pi$ in this case is
(Rybicki & Lightman, 1979, e.g.)

$\Pi=\frac{\sqrt{Q^{2}+U^{2}}}{I_{tot}}=\frac{P_{\perp}(\omega)-P_{||}(\omega)}{P_{\perp}(\omega)+P_{||}(\omega)},$
(4)

where $I_{tot}$ is the total intensity.

When the bulk velocity of the emitting region is nonzero, the emission will be
relativistically boosted. The boosting also affects the polarization
direction. For an emitting particle population traveling in direction
$\hat{v}$ with bulk Lorentz factor $\Gamma$, the radiated power will be
boosted by

$P_{obs}(\omega)=D^{-3}P\left(D\omega\right),$ (5)

with $D\equiv\Gamma\left[1-(1-\Gamma^{-2})^{1/2}\hat{v}\cdot\vec{n}\right]$.
$P_{\perp}$ and $P_{||}$ are boosted in the same manner. The polarization
vector in the observer frame $\vec{e}_{obs}$ is transformed to (Blandford &
Königl, 1979; Lyutikov et al., 2003; Peirson & Romani, 2018)

$\vec{e}_{obs}=\frac{\vec{n}\times\vec{q}}{\sqrt{\vec{q}^{2}-(\vec{n}\cdot\vec{q})^{2}}},$
(6)

where
$\vec{q}\equiv\vec{b}+\vec{n}\times[(1-\Gamma^{-2})^{1/2}\hat{v}\times\vec{b}]$
for $\vec{b}$ defined in the observer frame.

###  2.2 Intrabinary Shock Model

We adopt the IBS model of Kandel et al. (2019) and add polarization as
outlined above. This model is semi-analytic and designed to capture the impact
of geometry on the resulting emission. The primary geometry of the IBS is
governed by the stellar wind to pulsar wind momentum ratio
$\beta=\dot{M_{w}}v_{w}c/\dot{E}_{PSR}$ (Romani & Sanchez, 2016; Kandel et
al., 2019). The $\beta>1$ case generally corresponds to redbacks, while the
$\beta<1$ case to black widows. The exact shape of the contact discontinuity
between the shocks may also depend on the latitudinal distribution of the
pulsar wind (Kandel et al., 2019). The simplest case occurs when the pulsar
wind is spherical, and the geometry of the IBS is given by Cantó et al.
(1996); if the pulsar wind is equatorial, the appropriate formulae are given
by Kandel et al. (2019). The shape of the IBS is also distorted by sweepback
due to the companion’s orbital motion. This effect is parameterized by
$f_{v}=v_{w}/v_{orb}$, where $v_{orb}$ is the orbital speed (Romani & Sanchez,
2016). For small $f_{v}$, the shock contact discontinuity will trace out an
Archimedean spiral (Parkin & Pittard, 2008; Lamberts et al., 2012; Bosch-Ramon
et al., 2015).

We extend the ICARUS IBS code (Breton et al., 2012; Romani & Sanchez, 2016;
Kandel et al., 2019) to include synchrotron polarization. Computationally, the
IBS is assumed to be a thin shock along the contact discontinuity, which is
divided into triangular tiles of constant angular size as viewed from the
pulsar, representing different zones from which synchrotron radiation is
emitted. In this paper, we assume a spherical pulsar wind for simplicity. At
the IBS, the pulsar wind injects an electron and positron population with
energy spectrum in the flow frame

$N(\gamma_{e})d\gamma_{e}=N_{0}\gamma_{e}^{-p}d\gamma_{e},$ (7)

where $\gamma_{e}$ is the electron/positron Lorentz factor in range
$\gamma_{min}<\gamma_{e}<\gamma_{max}$, $N_{0}$ (in e/${\rm cm^{2}/s}$) is a
global normalization coefficient and $p$ depends on the particle acceleration
mechanism. $N_{0}$ is typically a free parameter in IBS fits to data; the
relative normalization of each tile is $N_{0,j}\propto 1/r_{j}^{2}$, where
$r_{j}$ is the distance between the $j$th tile and the pulsar. When the
corresponding energy flux of the power law is integrated over solid angle, it
can be usefully compared with the pulsar spin-down power
${\dot{E}}=I\Omega{\dot{\Omega}}$. After injection, the particles duct from an
individual tile downstream and radiatively cool. The bulk velocity of the
electron/positron population is approximated with direction $\hat{v}$ tangent
to the contact discontinuity. Bulk Lorentz factor increases along the shock,
approximated as

$\Gamma_{B}(s)=\Gamma_{0}\left(1+k\frac{s}{r_{0}}\right),$ (8)

where $s$ is the arclength from the nose to a given tile, $r_{0}$ is the nose-
standoff distance from the pulsar, $\Gamma_{0}$ is the Lorentz factor at the
nose, and $k$ is a scaling parameter that controls the flow velocity increase.
The magnetic field $\vec{B}(r)$ is defined at each tile of the IBS as a
function of distance from the pulsar and boosted to the flow frame at each
tile when computing the emission. We defer further discussion of the magnetic
field configuration to sec. 2.3.

We compute the residence time of the bulk flow in each tile in the flow frame

$\tau_{j}=\frac{d_{j}}{c\Gamma_{j}\sqrt{1-\Gamma_{j}^{-2}}},$ (9)

where $d_{j}$ is the physical length of the tile. Synchrotron cooling evolves
the flow frame particle spectrum as (Rybicki & Lightman, 1979; Kandel et al.,
2019)

$\gamma_{e}(t)=\gamma_{e,0}\left(1+\frac{2e^{4}B^{2}\sin^{2}\alpha}{3m^{3}c^{5}}\gamma_{e,0}t\right).$
(10)

We assume the time spent by the electrons/positrons in a particular tile is
evenly distributed over $(0,\tau_{j})$. Since the tiles are triangular, we add
half of the final cooled particle spectrum after $\tau_{j}$ to the freshly
injected electron spectrum of each of the two downstream ‘daughter’ tiles. In
our thin shock approximation, we do not include possible adiabatic losses.

The synchrotron spectrum projected in a given sky direction from tile $j$ is
computed as

$L_{j}(\omega)=D_{j}^{-3}\int_{\gamma_{min}}^{\gamma_{max}}N_{j}(\gamma_{e})P(D_{j}\omega,\gamma_{e})d\gamma_{e},$
(11)

with $P(\omega,\gamma_{e})$ from Eq. 1 and $D$ defined above. The injected
$e^{\pm}$ power law (Eq. 7) gives rise to a photon spectrum ${\rm
d}N_{\gamma}/{\rm d}E\sim E^{-\Gamma_{x}}\sim E^{-(p-1)/2}$; more generally
$N_{j}(\gamma_{e})$ for a tile is the full electron population, including
cooled electrons advected from upstream tiles. For the emission polarized
perpendicular and parallel to the projected magnetic field on the sky in a
given sky direction $\vec{n}$, one replaces $P$ in Eq. 1 with $P_{\perp}$ and
$P_{||}$ of Eq. 2. The total power on the sky in a given band emitted by tile
$j$ is simply

$L_{j}=\int_{\omega_{min}}^{\omega_{max}}L_{j}(\omega)d\omega,$ (12)

where $\omega_{min}$ and $\omega_{max}$ are the minimum and maximum frequency
in the spectral band of interest.

The observed polarization vector from each tile is given by Eq. 6. We set
$\hat{l}_{proj}$, the orbital angular momentum projected on the sky, as our
reference direction. The Stokes $Q$ and $U$ from each tile are then

$\begin{split}Q_{j}=&\left(L_{\perp,j}-L_{||,j}\right)\\\
&\times\left[(\vec{e}_{obs,j}\cdot\hat{l}_{proj})^{2}-(\vec{e}_{obs,j}\cdot[\vec{n}\times\hat{l}_{proj}])^{2}\right],\end{split}$
(13a) $\begin{split}U_{j}=&\left(L_{\perp,j}-L_{||,j}\right)\\\
&\times\left[(\vec{e}_{obs,j}\cdot\hat{l}_{proj,45^{\circ}})^{2}-(\vec{e}_{obs,j}\cdot[\vec{n}\times\hat{l}_{proj,45^{\circ}}])^{2}\right],\end{split}$
(13b)

where $\hat{l}_{proj,45^{\circ}}$ is the unit vector rotated counter clockwise
$45^{\circ}$ from $\hat{l}_{proj}$. The total luminosity and Stokes parameters
of the IBS are

$L=\sum_{j}L_{j},$ (14a) $Q=\sum_{j}Q_{j},$ (14b) $U=\sum_{j}U_{j}.$ (14c)

The polarization degree and angle on the sky are then given by

$\Pi_{tot}=\frac{\sqrt{Q^{2}+U^{2}}}{L},$ (15a)
$\chi_{tot}=\frac{1}{2}\arctan_{2}{\left(\frac{U}{Q}\right)}.$ (15b)

Note that $\chi_{tot}$ here is the polarization angle of the summed emission,
while $\chi$ in eq. 3b is the single zone polarization angle. Evaluating Eqs.
14 for all $\vec{n}$ gives skymaps (see Fig. 3 of Kandel et al., 2019) of
flux, $Q$, and $U$. Selecting a particular inclination angle $i$ between
$\vec{n}$ and the binary orbital angular momentum gives the phase-resolved
light curve, polarization degree, and polarization angle. Light curves have
orbital phases $0\leq\Phi<1$, with $\Phi=0$ the pulsar ascending node (pulsar
inferior conjunction at $\Phi=0.75$ has the pulsar between us and the
companion).

###  2.3 Magnetic field geometry

Outside the light cylinder, the pulsar wind magnetic field should be toroidal
(i.e. $B(r)\propto r^{-1}$) and takes the form of an MHD wind with stripes of
alternating polarity separated by current sheets (Coroniti, 1990; Cerutti et
al., 2015). At the small IBS distance, we expect stronger incident fields than
those of PWNe termination shocks (e.g. Reynolds et al., 2012). At the shock,
magnetic reconnection likely occurs, accelerating shocked particles; a
residual magnetic field remains in the IBS after the stripes annihilate
(Lyubarsky, 2003; Pétri & Lyubarsky, 2007; Sironi & Spitkovsky, 2011). The
very hard spectra observed in spiders (i.e. $\Gamma_{x}\approx 1$) (Bogdanov
et al., 2014; Kandel et al., 2021; Perez et al., 2023) supports this picture
(Cortés & Sironi, 2022; Zhang et al., 2023). The toroidal field remaining
after pulsar wind annihilation represents a promising candidate for the
magnetic field geometry, but post-shock dynamics such as turbulence may
produce differently ordered fields (e.g. Goldsmith & Pittard, 2016).

Figure 1: A 3-D visualization of the IBS flow and magnetic structure we
consider in this paper. Here the pulsar (red dot), in this case a redback, is
wrapped by the IBS (grey fading surface). The IBS is viewed from inclination
$i=90^{\circ}$ at orbital phase $\Phi=0.58$. Blue arrows mark the bulk flow
directions $\hat{v}$ and the green arrows show the magnetic field vectors
$\vec{b}$ for the shock-modified cylindrical model. For the flow model, the
blue arrows also represent the magnetic field vectors. Green arrows with heads
ending on the blue vectors are from the far side, viewed through the IBS. The
lower right shows the pre-shock cylindrical field around the pulsar. The
magenta rings denote the magnetic field lines while the black arrow represents
the orbital angular momentum axis; these strongly recycled spider pulsars are
assumed to be spin-aligned.

We assume here that the field magnitude immediately pre-shock is

$B(r)=B_{0}\left(\frac{r_{0}}{r}\right),$ (16)

where $B_{0}$ is the value of the magnetic field at the IBS nose, $r_{0}$ is
the distance from the pulsar to the nose, and $r$ is the distance from the
pulsar to a point on the IBS. For typical inferred $B_{0}$, synchrotron losses
can be significant for energetic particles radiating at high energies as they
flow along the shock. There can also be spectral features associated with the
electron spectrum cutoff at $\gamma_{\rm max}$. These features are often in
the hard X-ray/soft $\gamma$-ray band for typical spider parameters (Kandel et
al., 2019).

The magnetic field structure inside an IBS is not yet understood. To
illustrate the range of uncertain IBS magnetic field structures, we present
two simple model geometries: 1) a shock-modified cylindrical field and 2) a
flow geometry. The cylindrical magnetic field geometry is inspired by the
toroidal structure already observed in PWNe (Xie et al., 2022; Bucciantini et
al., 2023) and corresponds to the toroidal field imprinted on the shock by the
partially annihilated striped wind (Coroniti, 1990; Bogovalov, 1999; Pétri &
Lyubarsky, 2007). In this model, we set $\vec{b}=\hat{\phi}$ in the pre-shock
wind with $\hat{\phi}$ azimuthal in a cylindrical system centered on the
pulsar with cylindrical axis along the orbital angular momentum (see the inset
in Fig. 1). Note that the IBS shock, unlike a PWN shock, is oblique in many
regions. In the immediate post-shock IBS, the component of $\vec{B}$ parallel
to the shock surface is magnified by a factor of $3$ (for an ultra-
relativistic shock with adiabatic index $\gamma=4/3$), while the normal
component is unaltered. This drives ${\vec{b}}$ closer to parallel in the
shocked wind. We assume here that the radiation arises in this immediate post-
shock zone.

Alternatively, in the flow model, we envision a scenario in which field lines
become stretched along the motion of particles flowing in the shock. This is
motivated by the radially stretched magnetic field structure inferred in young
supernova remnants (Dickel & Milne, 1976; Dubner & Giacani, 2015; Vink et al.,
2022) as well as the field advection seen in IBS particle-in-cell simulations
(e.g. Cortés & Sironi, 2022). Geometrically, we assume that the magnetic field
follows the IBS flow so that $\vec{b}=\hat{v}$, while the magnitude is the
pre-shock value given by eq. 16 multiplied by 3 to account for shock
compression. We show a 3-D visualization of the IBS and the magnetic field
configurations in Fig. 1. Note that these geometries are specified in the lab
frame. The fields are boosted to the flow frame when evaluating eq. 1. These
two magnetic field geometries have maximal projected angle differences toward
the orbital poles. While phenomenological, they illustrate a wide range of
possible polarization behavior.

##  3 Model Results

We will illustrate the IBS peak and polarization pattern with a ‘redback’
geometry. In this case the IBS lies close to the pulsar and the wind
compresses and accelerates the flow, leading to more prominent IBS peaks.
Redback sources generally also have larger X-ray fluxes (e.g. Koljonen &
Linares, 2023), so they will be prime targets for IBS polarization studies.
Black widows should have broadly similar polarization behavior, except with
peak features centered on orbital phase $\Phi=0.25$ rather than $\Phi=0.75$.
The larger obliquity across much of the black widow IBS will cause some
differences for the cylindrical model.

In figs. 2 and 3, we show the predicted polarization properties with the
cylindrical and flow magnetic field models for a range of inclination angles.
In both cases, the model parameters are chosen so that the emission is from
the uncooled power law spectral component, as typically appropriate for the
soft X-ray band (Kandel et al., 2019). The polarization profiles of these two
models are notably different. Most prominently, the polarization degree is
substantially higher in the cylindrical model. The polarization degree is
highest around the flux minimum but can be substantial between the caustic
peaks. The total polarization degree increases with $i$, as the viewing angle
more closely coincides with the equatorial plane in which most field lines
lie. The orbital variation in the electric vector polarization angle (EVPA)
decreases with $i$, as polarization direction aligns more closely with the
orbital angular momentum. The fastest EVPA sweep and minimum polarization
degree occur across the two caustic peaks. Note that the polarization degree
minima lie outside the flux peaks for small $i$, inside for large $i$.

Figure 2: Plots of the IBS flux (top), polarization degree (middle), and
polarization angle (bottom) for $i=40^{\circ}$ (left), $i=60^{\circ}$
(center), and $i=80^{\circ}$ (right) for the cylindrical magnetic field model
in the uncooled power law part of the spectrum. Polarization angle $0^{\circ}$
corresponds to the polarization vector $\vec{e}$ aligned with the binary
orbital angular momentum vector projected onto the sky plane. The IBS model
was computed with $\beta=5$, $f_{v}=\infty$, $p=1$, $\Gamma_{0}=1.1$, and
$k=0.2$. Figure 3: Plots of the IBS flux (top), polarization degree (middle),
and polarization angle (bottom) for $i=40^{\circ}$ (left), $i=60^{\circ}$
(center), and $i=80^{\circ}$ (right) for the flow magnetic field model. The
conventions and model parameters are the same as those in Fig. 2

The orbital peak emission comes from IBS zones with flow lines near-tangent to
the line of sight, due to relativistic beaming. For the flow model, the
position angle of the projection of the local magnetic field on the sky varies
rapidly about this tangent point, both on the IBS surface and with binary
phase. This causes the rapid EVPA sweep near the peaks, and strong
depolarization of the integrated emission. Fig. 4 highlights the phase region
surrounding the first peak in the flow model, showing the polarization
behavior. Notice that the polarization degree minimum is slightly behind the
first peak phase (and slightly ahead of the second peak). Thus the phase where
the polarized flux most nearly cancels is offset from the flux peaks in this
model as well, although the shifts are more subtle. In detail, the offset is
sensitive to the IBS surface curvature, field structure, and emissivity in the
zones where the caustic peaks form. Since Doppler boosting at bulk Lorentz
factor $\Gamma_{B}$ mixes caustic emission over a beaming angle
$1/\Gamma_{B}$, the maximum phase offset scales with $P_{b}/\pi\Gamma_{B}$.

Figure 4: Plots of the flow magnetic field model flux, polarization degree,
and polarization angle as in Fig. 3, zoomed in around the first peak. The
dashed line in each subplot marks the orbital phase of the flux peak.

While the detailed profiles depend on the peculiarities of the IBS structure
and particle flow, a few basic physical effects can distort the peaks in
interesting ways. Orbital motion can impact the light curves and polarization
profiles for low companion wind speeds. Lower values of $f_{v}$ generate
asymmetry in the height of the two light curve caustic peaks as the IBS is
swept back in an Archimedean spiral (Romani & Sanchez, 2016; Kandel et al.,
2019, e.g.). The phase-resolved polarization profile exhibits similar features
with higher polarization around the higher peaked caustics. We show an example
light curve and phase-resolved polarization profile for the flow magnetic
field model with different values of $f_{v}$ and $i=60^{\circ}$ in Fig. 5. For
the cylindrical model the polarization changes little from the $f_{v}=\infty$
case.

Figure 5: Sweepback-induced distortion of IBS light curves (left),
polarization degree (middle), and polarization angle (right) with the same
parameters (except $f_{v}$) as Fig. 3. $f_{v}$ describes the companion’s wind
speed and, hence, is inversely proportional to the sweepback distortion.
Curves show the flow magnetic field model at $i=60^{\circ}$.

The IBS spectrum, peak shape, and polarization are also affected by
synchrotron cooling and the upper cutoff $\gamma_{\rm max}$. While the
uncooled power law population dominates the spectrum at low energies, the
spectrum has a cooling break after the particles flow a distance $s$ with
$\Gamma_{B}\sim 1.2$ at $E\simeq 10\left(\frac{B}{10\text{
G}}\right)^{-3}\left(\frac{s}{5\text{ R}_{\odot}}\right)^{-2}$ keV. In
addition, the spectrum cuts off exponentially above $E\simeq
170\left(\frac{\gamma_{max}}{10^{6}}\right)^{2}\left(\frac{B}{10\text{
G}}\right)$ keV due to the maximum particle energy. In Fig. 6, we compare
light curves and polarization profiles for the three spectral regimes for one
IBS geometry. The peak separation widens while the peaks themselves narrow
with energy (Kandel et al., 2019). The polarization structure remains similar
across the three regimes, but, interestingly, the polarization degree
increases at the higher energies. In the cooled power law regime, the
polarization degree increases from $50\%$ to 65%, while in the exponential
tail, the polarization degree reaches 80%. This arises from
$F(\omega/\omega_{c})$ and $G(\omega/\omega_{c})$ for $\omega>\omega_{c}$. The
EVPA sweep loses the reversal at energies above the cooling break.

Figure 6: Effects of synchrotron spectral features on the light curve (left),
polarization degree (middle), and polarization angle (right), here for the
cylindrical model at $i=60^{\circ}$ with the same parameters as Fig. 2. The
line types show spectral bands dominated by three spectral regimes: the low
energy uncooled power law (PL), the intermediate range where a cooled power
law (cPL) dominates, and a high energy range sensitive to the exponential tail
(ET) of the IBS spectrum. Note that the polarization degree grows as the
spectrum steepens at high energy.

##  4 Discussion and Conclusion

The high polarization we predict for the IBS synchrotron emission makes spider
pulsars interesting polarization targets. As the IBS emission dominates
X-rays, IXPE may be able to probe the polarization for X-ray bright spiders.
The polarization levels predicted here might be considered upper limits, since
strong turbulence will decrease the observed polarization degree. Since IXPE
has shown that PWN termination shocks have polarizations approaching the
maximum turbulence-free levels allowed for synchrotron emission, however, the
same may apply here. Conversely the IBS components of spider pulsars, while
dominant in the X-rays, are faint compared to typical accretion powered X-ray
sources; only a few spiders reach the $f_{2-8{\rm keV}}\approx
10^{-12}{\rm\text{ }erg/cm^{2}/s}$ fluxes required for high-significance IXPE
measurements in plausible Ms exposure times.

In IXPE’s 2-8 keV bandwidth, we should typically be mapping uncooled
synchrotron emission, although some spider parameters imply $B_{0}>100$ G,
moving the cooling break into the soft X-ray band. Future Compton
polarimeters, sensitive to hard X-rays and low energy gamma-rays (Lei et al.,
1997) may also probe the cooled population and possibly the exponential tail,
especially given the increasingly high polarization expected. NuSTAR hard
X-ray observations of some redbacks have shown double peak structure
consistent with the IBS (e.g. Kong et al., 2018; Perez et al., 2023), making
targeted polarization measurements of the IBS in this band appealing.

The uncooled power law spectrum of the IBS can extend down to the optical
band. In such cases, a non-thermal flux component, most prominent at bluer
wavelengths, adds to the companion thermal emission. Such non-thermal fluxes
have already been observed in some spiders (e.g. Nieder et al., 2020). As the
companion thermal emission is unpolarized, the polarized IBS contribution will
generally be strongly diluted, but may be accessible with high precision
optical polarimetry.

Our calculations may also be relevant to other binaries showing hard-spectrum
IBS emission, such as high mass $\gamma$-ray binaries containing energetic
pulsars. Unlike spiders, where our light curve and polarization computations
are for orbital phase-varying views of a stationary IBS, these systems are
typically eccentric, with the IBS structure modulated by changing orbital
separation and/or equatorial disk crossings. This complicates the orbital
light curve and polarization modulation; however, if the post-shock magnetic
field takes on the organized structures assumed in this paper, some phases may
also show high polarization.

Our models do not exhaust the possible field structures and other effects may
have a significant impact on polarization behavior. For example, field
annihilation in the pulsar wind may be a strong function of pulsar co-latitude
and the residual field strength may be sensitive to pulsar spin-orbit
misalignment (Bogovalov, 1999). Additionally, turbulence and disordered
magnetic fields can naturally decrease the overall polarization. More
realistic magnetic field models which account for these behaviors will require
detailed MHD simulations. We defer discussion of more detailed field
configurations to future work. The models presented here do, however,
illustrate possible phase-resolved polarization degree and EVPA features. In
conclusion, the highly ordered fields seen in PWNe suggest the same may be
true for IBSs; the resulting high polarization can provide a powerful
diagnostic of magnetic reconnection and particle acceleration in these
systems.

## Acknowledgements

The authors are grateful to Roger Blandford for useful discussions and the
anonymous referee for a prompt, useful review. This work was supported in
parts by NASA grants NMM17AA26C and 80NSS229K1506. A.S. acknowledges the
support of the Stanford University Physics Department Fellowship and the
National Science Foundation Graduate Research Fellowship Program.

## References

  * Arons & Tavani (1993) Arons, J., & Tavani, M. 1993, ApJ, 403, 249, doi: 10.1086/172198
  * Blandford & Königl (1979) Blandford, R. D., & Königl, A. 1979, ApJ, 232, 34, doi: 10.1086/157262
  * Bogdanov et al. (2014) Bogdanov, S., Esposito, P., Crawford, Fronefield, I., et al. 2014, ApJ, 781, 6, doi: 10.1088/0004-637X/781/1/6
  * Bogovalov (1999) Bogovalov, S. V. 1999, A&A, 349, 1017, doi: 10.48550/arXiv.astro-ph/9907051
  * Bogovalov et al. (2008) Bogovalov, S. V., Khangulyan, D. V., Koldoba, A. V., Ustyugova, G. V., & Aharonian, F. A. 2008, MNRAS, 387, 63, doi: 10.1111/j.1365-2966.2008.13226.x
  * Bosch-Ramon et al. (2015) Bosch-Ramon, V., Barkov, M. V., & Perucho, M. 2015, A&A, 577, A89, doi: 10.1051/0004-6361/201425228
  * Breton et al. (2012) Breton, R. P., Rappaport, S. A., van Kerkwijk, M. H., & Carter, J. A. 2012, ApJ, 748, 115, doi: 10.1088/0004-637X/748/2/115
  * Bucciantini et al. (2023) Bucciantini, N., et al. 2023, Nature Astronomy, 7, 602, doi: 10.1038/s41550-023-01936-8
  * Cantó et al. (1996) Cantó, J., Raga, A. C., & Wilkin, F. P. 1996, ApJ, 469, 729, doi: 10.1086/177820
  * Cerutti et al. (2015) Cerutti, B., Philippov, A., Parfrey, K., & Spitkovsky, A. 2015, MNRAS, 448, 606, doi: 10.1093/mnras/stv042
  * Coroniti (1990) Coroniti, F. V. 1990, ApJ, 349, 538, doi: 10.1086/168340
  * Cortés & Sironi (2022) Cortés, J., & Sironi, L. 2022, ApJ, 933, 140, doi: 10.3847/1538-4357/ac74b2
  * Dickel & Milne (1976) Dickel, J. R., & Milne, D. K. 1976, Australian Journal of Physics, 29, 435, doi: 10.1071/PH760435
  * Dubner & Giacani (2015) Dubner, G., & Giacani, E. 2015, A&A Rev., 23, 3, doi: 10.1007/s00159-015-0083-5
  * Fruchter et al. (1988) Fruchter, A. S., Stinebring, D. R., & Taylor, J. H. 1988, Nature, 333, 237, doi: 10.1038/333237a0
  * Gentile et al. (2014) Gentile, P. A., Roberts, M. S. E., McLaughlin, M. A., et al. 2014, ApJ, 783, 69, doi: 10.1088/0004-637X/783/2/69
  * Goldsmith & Pittard (2016) Goldsmith, K. J. A., & Pittard, J. M. 2016, MNRAS, 461, 578, doi: 10.1093/mnras/stw1365
  * Kandel et al. (2019) Kandel, D., Romani, R. W., & An, H. 2019, ApJ, 879, 73, doi: 10.3847/1538-4357/ab24d9
  * Kandel et al. (2021) —. 2021, ApJ, 917, L13, doi: 10.3847/2041-8213/ac15f7
  * Kluzniak et al. (1988) Kluzniak, W., Ruderman, M., Shaham, J., & Tavani, M. 1988, Nature, 334, 225, doi: 10.1038/334225a0
  * Koljonen & Linares (2023) Koljonen, K. I. I., & Linares, M. 2023, MNRAS, 525, 3963, doi: 10.1093/mnras/stad2485
  * Kong et al. (2018) Kong, A. K. H., Takata, J., Hui, C. Y., et al. 2018, MNRAS, 478, 3987, doi: 10.1093/mnras/sty1459
  * Lamberts et al. (2012) Lamberts, A., Dubus, G., Lesur, G., & Fromang, S. 2012, A&A, 546, A60, doi: 10.1051/0004-6361/201219006
  * Lei et al. (1997) Lei, F., Dean, A. J., & Hills, G. L. 1997, Space Sci. Rev., 82, 309, doi: 10.1023/A:1005027107614
  * Lyubarsky (2003) Lyubarsky, Y. E. 2003, MNRAS, 345, 153, doi: 10.1046/j.1365-8711.2003.06927.x
  * Lyutikov et al. (2003) Lyutikov, M., Pariev, V. I., & Blandford, R. D. 2003, ApJ, 597, 998, doi: 10.1086/378497
  * Malkov & Lemoine (2023) Malkov, M., & Lemoine, M. 2023, Phys. Rev. E, 107, 025201, doi: 10.1103/PhysRevE.107.025201
  * Nieder et al. (2020) Nieder, L., Clark, C. J., Kandel, D., et al. 2020, ApJ, 902, L46, doi: 10.3847/2041-8213/abbc02
  * Parkin & Pittard (2008) Parkin, E. R., & Pittard, J. M. 2008, MNRAS, 388, 1047, doi: 10.1111/j.1365-2966.2008.13511.x
  * Peirson & Romani (2018) Peirson, A. L., & Romani, R. W. 2018, ApJ, 864, 140, doi: 10.3847/1538-4357/aad69d
  * Perez et al. (2023) Perez, K. I., Bogdanov, S., Halpern, J. P., & Gajjar, V. 2023, ApJ, 952, 150, doi: 10.3847/1538-4357/acdc23
  * Pétri & Lyubarsky (2007) Pétri, J., & Lyubarsky, Y. 2007, A&A, 473, 683, doi: 10.1051/0004-6361:20066981
  * Reynolds et al. (2012) Reynolds, S. P., Gaensler, B. M., & Bocchino, F. 2012, Space Sci. Rev., 166, 231, doi: 10.1007/s11214-011-9775-y
  * Roberts (2013) Roberts, M. S. E. 2013, in Neutron Stars and Pulsars: Challenges and Opportunities after 80 years, ed. J. van Leeuwen, Vol. 291, 127–132, doi: 10.1017/S174392131202337X
  * Romani & Sanchez (2016) Romani, R. W., & Sanchez, N. 2016, ApJ, 828, 7, doi: 10.3847/0004-637X/828/1/7
  * Rybicki & Lightman (1979) Rybicki, G. B., & Lightman, A. P. 1979, Radiative processes in astrophysics
  * Sironi & Spitkovsky (2011) Sironi, L., & Spitkovsky, A. 2011, ApJ, 741, 39, doi: 10.1088/0004-637X/741/1/39
  * Trippe (2014) Trippe, S. 2014, Journal of Korean Astronomical Society, 47, 15, doi: 10.5303/JKAS.2014.47.1.15
  * van der Merwe et al. (2020) van der Merwe, C. J. T., Wadiasingh, Z., Venter, C., Harding, A. K., & Baring, M. G. 2020, ApJ, 904, 91, doi: 10.3847/1538-4357/abbdfb
  * van Paradijs et al. (1988) van Paradijs, J., Allington-Smith, J., Callanan, P., et al. 1988, Nature, 334, 684, doi: 10.1038/334684a0
  * Vink et al. (2022) Vink, J., et al. 2022, ApJ, 938, 40, doi: 10.3847/1538-4357/ac8b7b
  * Weisskopf et al. (2016) Weisskopf, M. C., Ramsey, B., O’Dell, S., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9905, Space Telescopes and Instrumentation 2016: Ultraviolet to Gamma Ray, ed. J.-W. A. den Herder, T. Takahashi, & M. Bautz, 990517, doi: 10.1117/12.2235240
  * Weisskopf et al. (2022) Weisskopf, M. C., Soffitta, P., Baldini, L., et al. 2022, Journal of Astronomical Telescopes, Instruments, and Systems, 8, 026002, doi: 10.1117/1.JATIS.8.2.026002
  * Westfold (1959) Westfold, K. C. 1959, ApJ, 130, 241, doi: 10.1086/146713
  * Xie et al. (2022) Xie, F., et al. 2022, Nature, 612, 658, doi: 10.1038/s41586-022-05476-5
  * Zhang et al. (2023) Zhang, H., Sironi, L., Giannios, D., & Petropoulou, M. 2023, arXiv e-prints, arXiv:2302.12269, doi: 10.48550/arXiv.2302.12269


# Unidirectional Thin Adapter for Efficient Adaptation of Deep Neural Networks

Han Gyel Sun, Hyunjae Ahn, HyunGyu Lee, Injung Kim
Handong Global University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

In this paper, we propose a new adapter network for adapting a pre-trained
deep neural network to a target domain with minimal computation. The proposed
model, unidirectional thin adapter (UDTA), helps the classifier adapt to new
data by providing auxiliary features that complement the backbone network.
UDTA takes outputs from multiple layers of the backbone as input features but
does not transmit any feature to the backbone. As a result, UDTA can learn
without computing the gradient of the backbone, which saves computation for
training significantly. In addition, since UDTA learns the target task without
modifying the backbone, a single backbone can adapt to multiple tasks by
learning only UDTAs separately. In experiments on five fine-grained
classification datasets consisting of a small number of samples, UDTA
significantly reduced computation and training time required for
backpropagation while showing comparable or even improved accuracy compared
with conventional adapter models.

##  1 Introduction

Deep neural networks (DNN) have shown outstanding performance in diverse
fields and are developing rapidly. However, DNNs require a large amount of
data to learn a target task. There has been a growing body of research that
alleviates this issue by adapting a model pre-trained on a large dataset, such
as ImageNet, to a small target dataset [1, 2]. The ‘pre-training followed by
fine-tuning’ strategy is effective in improving performance for tasks of
insufficient data. In particular, a few recent works propose minimizing or
omitting the adaptation step by learning abundant knowledge from a large
amount of data with large-scale models such as vision transformer (ViT) and
DALL-E [3, 4].

However, large-scale DNNs require excessive computational resources for use on
low-resource devices and are vulnerable to overfitting when fine-tuned with a
small amount of data. Due to these issues, when adapting to new data in an
environment with low computing power or insufficient data, many previous works
freeze the backbone network and learn only the classifier using transfer
learning or few-shot learning [2, 5].

Despite the aforementioned issues, the demand for learning or adapting DNNs in
the low-resource environment, such as an edge device, is increasing. Training
in the client device can improve the accuracy of the target task by further
optimizing the pre-trained model with the target data of the user. We can
extend this advantage to federated learning that improves a shared model
without transferring private data by integrating only the training results of
multiple client devices [6, 7]. In particular, in certain fields, such as
fine-grained classification, adapting the pre-trained feature extractor layers
to the target data is crucial because it is difficult to achieve high
performance in such fields with only the features learned from a general
dataset.

To train a DNN in a low-resource environment, it is essential to reduce the
computational cost for learning. In previous work, diverse approaches have
been proposed to reduce the size and computation of the DNN. Recently, there
has been much progress to tackle this issue with lightweight models such as
MobileNetV1 and V2 [8, 9], ShuffleNetV1 and V2 [10, 11], EfficientNetV1 and V2
[12, 13]. However, even with the lightweight models, training the entire model
in a low-resource environment requires additional computation reduction.

Figure 1: Unidirectional thin adapter (UDTA) combined with a backbone network
(MobileNetV2). UDTA adapts the pre-trained backbone network by providing
auxiliary features to complement the backbone features. Since the encoders
reduce feature dimension, the thin blocks are substantially narrower than IR
blocks.

On the other hand, a few previous studies have proposed adapter networks to
learn DNNs with only a small amount of data [14, 15, 16, 17, 18]. The adapter
improves performance by adjusting the activation of intermediate backbone
layers or by providing complement features, without modifying the backbone
substantially. Composed of a small number of parameters, the adapters are
effective in reducing overfitting and preserving the pre-trained knowledge of
the backbone. However, the existing adapters do not reduce computation for
training because they still require the gradient of the backbone while
training.

To overcome this limitation, in this paper, we propose a unidirectional thin
adapter (UDTA) that adapts the pre-trained backbone network to the target task
with minimal computation and a small number of target data. UDTA, as shown in
Figure 1, is composed of thin and lightweight blocks that can be trained with
minimal computation and a small amount of data.

Similar to the existing adapters, UDTA utilizes the intermediate features of
the backbone to produce features effective for the target task. However,
unlike conventional adapters, UDTA has only unidirectional connections from
the backbone and does not transmit its features to the backbone. As a result,
UDTA does not require the gradient of the backbone during backpropagation,
thereby can be trained with minimal computation. In addition, we reduce the
dimension of the feature maps delivered from the backbone by autoencoders,
further reducing the size and computation of UDTA without significant loss of
information. Since UDTA perfectly preserves the pre-trained knowledge of the
backbone, a single backbone can conduct multiple tasks by combining a
collection of per-task UDTAs. In experiments on five fine-grained
classification datasets containing a small number of data, UDTA reduced the
computation and time for backpropagation by 86.36% and 77.78% compared with
model patches [14], while exhibiting comparable or improved accuracy compared
with the other adapters.

Our main contributions include 1) a theoretical analysis of the reason why the
existing adapter networks fail to reduce computation for training, 2) a novel
unidirectional thin adapter (UDTA) to adapt a pre-trained DNN to a new dataset
with minimal computation and data, 3) a lightweight adapter structure and
reduced data transfer due to dimensionality reduction, and 4) comparable or
improved accuracy compared with existing adapter networks for five small-size
datasets for fine-grained classification.

Section 2 of this paper introduces related work. Section 3 analyzes the
problems with conventional adapter networks in terms of computational
efficiency in backpropagation and proposes the unidirectional thin adapter to
overcome the limitation. Section 4 and Section 5 present experimental results
and conclusions, respectively.

##  2 Related Work

Figure 2: Training of residual adapters and model patches. The red arrows
illustrate the backpropagation flow. Both adapters require the gradient of the
backbone blocks.

Pre-training followed by fine-tuning strategy In recent years, there has been
an increasing interest in domain adaptation by fine-tuning a pre-trained DNN
model. Specifically, the ‘pre-training followed by fine-tuning’ strategy, that
first pre-trains a DNN with a large volume of data, such as ImageNet, and then
fine-tunes it with a small amount of target data, has been widely adopted and
demonstrated its effectiveness in improving performance for the tasks with
insufficient training data. [2] proposes a meta transfer learning method for
effectively fine-tuning a DNN pre-trained on a large dataset for multiple few-
shot tasks. [5] applies a pre-trained convolutional neural network (CNN) to
map samples to a latent space, and then classifies the latent feature using a
simple model, such as k-NN classifier and Gaussian mixture models.

Lightweight network models MobileNetV2 [9], ShuffleNetV2 [11], and
EfficientNetV2 [13] propose lightweight DNN models with reduced size and
computation. TinyTL [19] proposes a memory-efficient model that learns only
the bias without storing intermediate activation. Besides the above models,
researchers have attempted to reduce the computation or the number of
parameters of the DNN. Lightlayers [20] and batch ensemble [21] propose
network modules as to reduce computation. Lightlayers decompose convolutional
layers and fully connected layers into a combination of minor operations. In
contrast, batch ensemble approximates a weight matrix by the product of a
single row and a single column.

We built our backbone network based on MobileNetV2 [9]. The main building
block of MobileNetV2 is the inverted residual (IR) block that was designed to
minimize computational requirement and the number of parameters preserving
performance. The IR block is composed of a 1x1 convolution that increases the
number of channels, a 3x3 depth-wise convolution, and another 1x1 convolution
that decreases the number of channels. The third 1x1 convolution layer omits
ReLU activation to reduce information loss.

Figure 3: Training of UDTA. The red arrows illustrate the backpropagation
flow. UDTA learns without the gradient of the backbone.

Adapter networks Numerous previous studies on domain adaptive learning have
proposed adapter networks to adapt a pre-trained DNN to a new target domain
efficiently [22, 14, 15, 16, 17, 18]. Composed of a small number of
parameters, the adapter networks can be trained with a small number of target
data.

Some previous works have applied partial learning to reduce the number of
parameters [22, 14]. [14] proposes to adapt a pre-trained DNN to a new task by
learning only a small number of layers, fixing the other parts. They
demonstrate that training only a few layers with a small number of parameters,
such as scale-and-bias patch and depthwise-convolution patch, can
significantly improve performance for a new target task. [15, 16] proposes a
residual adapter that reduces the number of parameters to train in domain
adaptation. While [14] reuses the layers of the pre-trained network for model
patches, [15, 16] adapt the backbone by combining it with a collection of
adapter networks and learning only the adapters. The adapter network is
composed of layers with a small number of parameters, such as 1x1 convolution
and batch normalization. They combine residual adapters to all 3x3 convolution
layers of the backbone network. Model patches and residual adapters help DNNs
adapt to a new domain with a small number of training samples. However, they
do not reduce the computational cost for training, as analyzed in the next
section.

[17, 18] propose a budget-aware adapter ($BA^{2}$) and FLUTE to reduce the
computational cost. Budget-aware adapter selects the most relevant feature
channels and saves the computational cost by activating only the selected
channels. [18] proposes a method for few-shot learning with a universal
template (FLUTE), a partial model that can define a wide array of dataset-
specialized models by plugging in appropriate components. They are effective
in reducing computation for forward propagation, but cannot be used to reduce
computation for training.

##  3 Efficient Domain Adaptation with Unidirectional Thin Adapter

###  3.1 Backpropagation of Conventional Adapter Networks

Figure 4: Training of UDTA. (a) and (b) are conducted on a server, while (c)
is conducted on the client device.

The gradient descent algorithm optimizes the neural network by adjusting model
parameters to minimize loss function as Eq. (1), where $w^{l}$, $L$, and
$\eta$ denote the weights of the $l$-th layer, loss function, and learning
rate, respectively.

$w^{l}\leftarrow w^{l}-\eta\frac{\partial L}{\partial w^{l}}$ (1)

As shown in Eq. (1), the gradient descent algorithm requires the gradient
$\frac{\partial L}{\partial w^{l}}$. The backpropagation algorithm computes
the gradient sequentially from the top to the bottom layer.

Residual adapters are combined with the backbone layers in a parallel or
serial way. Figure 2(a) displays parallel residual adapters attached to the
backbone network. In training procedure, the backbone blocks are fixed and
only the adapter blocks are trained with the target data. To train the weights
of the residual adapter $w_{r}^{l}$, we need to compute its gradient
$\frac{\partial L}{\partial w_{r}^{l}}$.

The red arrows in Figure 2(a) illustrate the backpropagation path to learn the
residual adapters. Applying the chain rule, $\frac{\partial L}{\partial
w_{r}^{l}}$ is decomposed as Eq. (2), where $x^{l}$ and $r^{l}$ denote
activation of the $l$-th block in the backbone and the residual adapter,
respectively.

$\frac{\partial L}{\partial w_{r}^{l}}=\frac{\partial L}{\partial
r^{l}}\frac{\partial r^{l}}{\partial w_{r}^{l}}=\frac{\partial L}{\partial
x^{l}}\frac{\partial x^{l}}{\partial r^{l}}\frac{\partial r^{l}}{\partial
w_{r}^{l}}$ (2)

$\frac{\partial L}{\partial x^{l}}$ in Eq. (2) is computed by combining the
gradient from $x^{l+1}$ and that from $r^{l+1}$ as Eq. (3). The recursive
expansion of Eq. (3) includes $\frac{\partial L}{\partial
x^{n}}(\prod_{k=l}^{n-1}\frac{\partial x^{k+1}}{\partial x^{k}})$.

$\frac{\partial L}{\partial x^{l}}=\frac{\partial L}{\partial
x^{l+1}}\frac{\partial x^{l+1}}{\partial x^{l}}+\frac{\partial L}{\partial
r^{l+1}}\frac{\partial r^{l+1}}{\partial x^{l}}$ (3)

As shown in Eq.(2) and (3), computing $\frac{\partial L}{\partial w_{r}^{l}}$
requires the gradients $\frac{\partial L}{\partial x^{k}}$ for all $k\geq l$.
Consequently, training the parallel residual adapter requires backpropagation
through the entire network including the backbone. The computation of the
gradients to learn serial adapters is similar to that with model patches
explained below.

Figure 2(b) illustrates the forward and backward propagation of a DNN with
model patches. The gradient for the weight $w_{p}^{l}$ of the $l$-th patch
layer is computed as Eq.(4), where $x^{l}$ is the activation of a patch layer
and $x^{k}$s are the activation of either a patch layer or a backbone layer.

$\frac{\partial L}{\partial w_{p}^{l}}=\frac{\partial L}{\partial
x^{l}}\frac{\partial x^{l}}{\partial w_{p}^{l}}=\frac{\partial L}{\partial
x^{n}}(\prod_{k=l}^{n-1}\frac{\partial x^{k+1}}{\partial x^{k}})\frac{\partial
x^{l}}{\partial w_{p}^{l}}$ (4)

Eq. (4) shows that the training of model patches also requires backpropagation
through the entire network.

###  3.2 Structure of Unidirectional Thin Adapter

Figure 1 illustrates the structure of UDTA combined with a backbone network.
In Figure 1, $w$, $h$, and $c$ denote the width, height, and the number of
channels of the backbone feature map, respectively. $d>1$, $t>1$ and $u>1$
denote the dimensionality reduction factor of the encoder, channel expansion
factor of the IR block, and channel expansion factor of thin block,
respectively.

The backbone network is pre-trained with a large-scale dataset and extracts
high-level features from the input. The UDTA is composed of a stack of thin
blocks. The dimension of the UDTA feature map is lower than that of the
backbone feature map at the same level. Each thin block takes as input the
feature map of the corresponding backbone block, and propagates features to
upper blocks to produce auxiliary features for the classifier. In training, we
freeze the backbone and train only UDTA with the target data to produce
auxiliary features that complement the backbone features, thereby improving
the performance for the target task. In other words, the backbone learns
general knowledge from a large-scale pre-training dataset, while UDTA learns
domain-specific knowledge from the target data to complement the backbone
features. The classifier combines the features from the backbone and UDTA and
outputs the classification result.

The encoders between the backbone and the UDTA blocks reduce the dimension of
the features transmitted from the backbone, which allows further reducing the
size and computation of UDTA. We use autoencoders to train UDTA encoders to
reduce feature dimension while minimizing information loss. As a result, the
UDTA comprises a small number of parameters and requires a small amount of
computation for both forward and backward propagation.

The full training of the entire network requires a large amount of
computation. On the other hand, freezing the backbone and training only the
classifier reduce computation significantly but exhibit limited performance
because of the lack of domain-specific knowledge. UDTA can make a good trade-
off between performance and computational cost. In addition, UDTA perfectly
preserves the pre-training knowledge of the backbone since it learns without
modifying the backbone. Therefore, it is possible to perform multiple tasks
with a single backbone network by adapting only UDTAs to the target tasks.
This is an important advantage, especially in the low-resource environment.

The most prominent advantage of UDTA over existing adapter networks is that we
can train it with a minimal amount of computation. In Section 3.1, we have
shown that residual adapters and model patches cannot reduce computation for
training, because training them requires the gradient of the backbone which
demands heavy computation. To learn without the gradient of the backbone, we
designed UDTA to be connected to the backbone in only one direction. While it
receives the intermediate features from multiple backbone blocks, it does not
provide any feature to the backbone blocks.

Training UDTA requires the gradient $\frac{\partial L}{\partial w_{u}^{l}}$,
where $w_{u}^{l}$ denotes the weights of the $l$-th thin block. The gradient
is computed as Eq. (5).

$\displaystyle\frac{\partial L}{\partial w_{u}^{l}}$
$\displaystyle=\frac{\partial L}{\partial u^{l}}\frac{\partial u^{l}}{\partial
w_{u}^{l}}=\frac{\partial L}{\partial u^{n}}\frac{\partial u^{n}}{\partial
u^{l}}\frac{\partial u^{l}}{\partial w_{u}^{l}}$ (5)
$\displaystyle=\frac{\partial L}{\partial
u^{n}}(\prod_{k=l}^{n-1}\frac{\partial u^{k+1}}{\partial u^{k}})\frac{\partial
u^{l}}{\partial w_{u}^{l}}$

Note that Eq. (5) does not require the gradient of the backbone
$\frac{\partial L}{\partial x^{l}}$ for any $l$. Figure 3 illustrates the
backpropagation path to compute gradient $\frac{\partial L}{\partial
w_{u}^{l}}$ to train UDTA. The red arrow starts from the loss and goes through
the thin blocks. However, it does not pass any backbone block. Consequently,
UDTA can learn without backpropagation through the entire backbone network,
and therefore, can save computation for training substantially.

The structure of UDTA is based on the inverted residual block [9], as shown in
Figure 1. The first $1\times 1$ convolution increases the number of channels,
and the $3\times 3$ depth-wise convolution extracts spatial information from
the input feature. The last 1 × 1 convolution decreases the number of
channels. The width and depth of UDTA can be flexibly decided according to the
characteristics of the target data and the backbone network. The autoencoder
combines an encoder and a decoder, each of which is composed of a $1\times 1$
convolution layer. The input dimension of the autoencoder is the same as the
output dimension of the backbone layer, while its hidden dimension is the same
as the input dimension of the UDTA layer.

###  3.3 Learning of Recognition System Using Unidirectional Thin Adapter

We train the entire network in three steps. In the first step, the backbone
network learns abundant general knowledge that is useful for many target
tasks, from a large-scale dataset such as ImageNet on a server, as shown in
Figure 4(a).

Figure 5: FGVC-Aircraft image samples. Only minor differences distinguish the
aircraft types. Table 1: The number of classes and samples in the datasets for
fine-grained classification. (Air: FGVC-Aircraft, Cars: Stanford Cars, DTD:
Describable Textures, Dogs: Stanford Dogs, Flow: Flowers-102)

| Classes | Training Samples | Test Samples | Total Samples
---|---|---|---|---
Air | 100 | 6,667 | 3,333 | 10,000
Cars | 196 | 8,144 | 8,041 | 16,185
DTD | 47 | 3,760 | 1,880 | 5,640
Dogs | 102 | 2,040 | 6,149 | 8,189
Flow | 120 | 12,000 | 8,580 | 20,580

Table 2: Experimental settings for backbone pre-training, autoencoder
training, and domain adaptation

| Epoch | Learning rate | Batch | Optimizer | Loss function
---|---|---|---|---|---
Pre-training | 120 | 0.01 | 256 |  | ADAM
---
(a=0.9, b= 0.999)
Cross Entropy
AE training | 120 | 0.001 | 256 |  | ADAM
---
(a=0.9, b= 0.999)
MSE
Domain adap. | 120 | 0.001 | 256 |  | ADAM
---
(a=0.9, b= 0.999)
Cross Entropy

The second step is to learn the encoders for dimensionality reduction. Similar
to the first step, we also train the autoencoder with a large-scale pre-
training dataset on a server, as shown in Figure 4(b). To reduce feature
dimension while minimizing the loss of information, we train the autoencoder
by the $L_{2}$ reconstruction loss. We do not train the backbone and the
autoencoders at the same time because, before the backbone training is
complete, learning to compress the immature features is meaningless. After the
second step, we use the encoder for dimensionality reduction and discard the
decoder.

In the final step, we freeze the backbone and autoencoder, and train only UDTA
and the classifier with the target dataset. Due to the encoder, the feature
map of UDTA has significantly lower dimension compared with that of the
backbone, and therefore, UDTA can learn from a small amount of data with a
minimal amount of computation. Furthermore, we combine only a few top layers
with thin layers, further saving parameters and computation. However, for the
backbone block with low dimensionality, we directly connect the thin block to
the backbone block without encoder.

##  4 Experiment

###  4.1 Datasets and Experimental Settings

Datasets We used the ImageNet dataset [23] for the pre-training of the
backbone network and the autoencoders. For the target data, we used five
small-size datasets designed for fine-grained classification: FGVC-Aircraft
[24], Stanford Cars [25], DTD [26], Oxford 102 Flowers [27], and Stanford Dogs
[28] datasets. They consist of a small number of samples per class, and the
difference between class images in those datasets is small compared with
ordinary datasets. Figure 5 displays a few samples in FGVC-Aircraft set. To
obtain high performance for those datasets, it is essential to learn
discriminative features by domain adaptation. The number of classes and
samples are presented in Table 1.

Table 3: Computation, time, and accuracy of UDTA compared with other models.
(Scratch: backbone trained from scratch, Full-FT: fully fine-tuned backbone,
Top-FT: backbone with fine-tuned top layer, MP: model patch [14], RA: residual
adapters [15, 16], and UDTA: proposed method)

| F.FLOPs | B.FLOPs | F.Time | B.Time | Acc. | Params
---|---|---|---|---|---|---
Scratch | 85,322M | 170,645M | 8.2ms | 12.6ms | 63.57 | 2352K
Full-FT | 85,322M | 170,645M | 8.2ms | 12.6ms | 81.03 | 2352K
Top-FT | 85,322M | 66M | 7.8ms | 0.4ms | 41.91 | 129K
MP | 85,322M | 165,096M | 8.2ms | 11.7ms | 66.17 | 156K
RA | 92,611M | 171,966M | 10.4ms | 12.0ms | 62.19 | 163K
UDTA | 97,357M | 22,499M | 9.7ms | 2.6ms | 67.56 | 865K

Experimental Environment and Settings We conducted experiments on a PC with
RTX-3090 GPU, AMD Threadripper 1900x CPU, and 32GB of RAM. We implemented the
models on Ubuntu Linux using PyTorch (1.7.1v) and Torchvision (0.8.2v). The
experimental settings and hyper-parameters are presented in Table 2.

Metrics We measured classification performance by top-1 accuracy on the test
data, the amount of computation by FLOPs (floating point operations), and time
by msec. In each table, F.FLOPs and B.FLOPs denote the computation for forward
and backward propagation, and F.Time and B.Time denote the time for forward
and backward propagation, respectively. Additionally, we present the number of
parameters to train in each model.

(a) Accuracy (blue), computation for forward propagation (red) and
backpropagation (orange)

(b) Accuracy (blue), time for forward propagation (red) and backpropagation
(orange)

Figure 6: Accuracy, computation, and time for forward/backward propagation of
UDTA compared with other models

###  4.2 Comparison with Other Adapter Networks

In the first experiment, we compared UDTA with five other models on the FGVC-
Aircraft dataset. The experimental results are presented in Table 3 and Figure
6, where ‘Scratch’ denotes the backbone network only trained with the target
datasets from scratch, ‘Full-FT’ denotes the backbone pre-trained with
ImageNet and then fully fine-tuned with the target dataset. ‘Top-FT’ is the
backbone network in which only the top layer was fine-tuned after pre-
training. ‘MP’, ‘RA’, and ‘UDTA’ denote model patch, residual adapter, and the
proposed adapter, respectively.

Without pre-training, the backbone trained from scratch (‘Scratch’) was slow
and exhibited a poor accuracy. The full fine-tuning of the pre-training
backbone (‘Full-FT’) exhibited the best accuracy but required heavy
computation for backpropagation. On the other hand, fine-tuning only the top
layer of the pre-trained backbone (‘Top-FT’) was the fastest but exhibited the
lowest accuracy. Applying model patches (‘MP’) and residual adapters (‘RA’)
improved accuracy compared with the ’TOP-FT’ model with learning a minimal
number of parameters. However, they have little effect in reducing the
computation and time for training. The proposed model, UDTA, exhibited higher
accuracy than other adapters, while reducing the computation and time for
backward propagation by 86.36% from 165,096 MFLOPs to 22,499 MFLOPs, and
77.78% from 11.7 msec to 2.6 msec compared with model patches. In return, UDTA
increased the computation and time for forward propagation by 15.88% and
18.29%, respectively.

Table 4: Computation, time, and accuracy according to the number of thin
blocks B

B. | F.FLOPs | B.FLOPs | F.Time | B.Time | Acc. | Params
---|---|---|---|---|---|---
1 | 91,364M | 12,148M | 8.2ms | 1.3ms | 65.01 | 634K
3 | 97,357M | 22,499M | 9.7ms | 2.6ms | 67.56 | 865K
5 | 102,138M | 30,782M | 11.1ms | 3.7ms | 63.51 | 1002K
7 | 113,233M | 51,870M | 11.7ms | 4.3ms | 60.24 | 1007K

Table 5: Computation, time, and accuracy according to expansion factor $u$

$u$ | F.FLOPs | B.FLOPs | F.Time | B.Time | Acc. | Params
---|---|---|---|---|---|---
2 | 90,475M | 8,570M | 8.8ms | 2.1ms | 60.06 | 433K
4 | 94,674M | 16,967M | 9.3ms | 2.5ms | 65.22 | 705K
6 | 97,357M | 22,499M | 9.7ms | 2.6ms | 67.56 | 865K
8 | 103,072M | 33,763M | 9.8ms | 3.3ms | 67.56 | 1249K

Table 6: Domain adaptation performance (Top-1 accuracy) on multiple datasets

| Air | Cars | DTD | Dogs | Flow
---|---|---|---|---|---
Scratch | 63.57% | 72.80% | 39.94% | 55.65% | 59.39%
Full-FT | 81.03% | 88.62% | 69.14% | 82.59% | 95.65%
TOP-FT | 41.91% | 50.20% | 67.44% | 80.17% | 90.82%
MP | 66.17% | 76.33% | 68.61% | 81.62% | 93.78%
RA | 62.19% | 74.94% | 67.58% | 77.87% | 92.60%
UDTA | 67.56% | 81.21% | 69.89% | 79.94% | 93.54%

To find the optimal depth and width of UDTA, we measured computation, time,
and accuracy on the FGVC-Aircraft dataset changing the number of thin blocks
and the expansion factor $u$ in Figure 1. The results are presented in Table 4
and 5. As the width and depth of UDTA increased, the amount of computation and
time increased. The accuracy increased with width but was not proportional to
depth. We found the optimal size of UDTA, in terms of the trade-off between
computation and accuracy, with depth three and expansion factor six.

We also evaluated UDTA and existing models on multiple fine-grained
classification datasets. Table 6 measures the accuracy of the models. The
‘Full-FT’ model exhibited the highest accuracy. The accuracy of UDTA was
comparable or higher than the other adapter networks.

##  5 Conclusion

We analyzed why conventional adapter networks fail to reduce the amount of
computation for training and proposed a unidirectional thin adapter (UDTA) for
efficient domain adaption of the DNN. UDTA improves the performance for the
target task by providing auxiliary features to complement the features of the
backbone network, which was trained from a general pre-training dataset.
Unlike conventional adapter networks, UDTA is connected to the backbone
network in one way, not requiring the gradient of the backbone network for
training, which substantially reduces computation for backpropagation. In
experiments on five small-size datasets for fine-grained classification, UDTA
reduced the amount of computation and time for backpropagation by 86.36% and
77.78%, respectively, compared with model patches [14], while exhibiting
comparable or even improved accuracy.

## References

  * [1] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [2] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele.  Meta-transfer learning for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computaer Vision and Pattern Recognition, pages 403–412, 2019.
  * [3] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  2021\.
  * [4] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation, 2021.
  * [5] Tomáš Chobola, Daniel Vašata, and Pavel Kordík.  Transfer learning based few-shot classification using optimal transport mapping from preprocessed latent space of backbone neural network.  arXiv preprint arXiv:2102.05176, 2021.
  * [6] Jakub Konečnỳ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon.  Federated learning: Strategies for improving communication efficiency.  arXiv preprint arXiv:1610.05492, 2016.
  * [7] Wei Yang Bryan Lim, Nguyen Cong Luong, Dinh Thai Hoang, Yutao Jiao, Ying-Chang Liang, Qiang Yang, Dusit Niyato, and Chunyan Miao.  Federated learning in mobile edge networks: A comprehensive survey.  IEEE Communications Surveys & Tutorials, 22(3):2031–2063, 2020\.
  * [8] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [9] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [10] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018.
  * [11] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018.
  * [12] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International Conference on Machine Learning, pages 6105–6114. PMLR, 2019.
  * [13] Mingxing Tan and Quoc V Le.  Efficientnetv2: Smaller models and faster training.  arXiv preprint arXiv:2104.00298, 2021.
  * [14] Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, and Andrew Howard.  K for the price of 1: Parameter-efficient multi-task and transfer learning.  arXiv preprint arXiv:1810.10703, 2018.
  * [15] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Learning multiple visual domains with residual adapters.  arXiv preprint arXiv:1705.08045, 2017.
  * [16] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Efficient parametrization of multi-domain deep neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8119–8127, 2018.
  * [17] Rodrigo Berriel, Stephane Lathuillere, Moin Nabi, Tassilo Klein, Thiago Oliveira-Santos, Nicu Sebe, and Elisa Ricci.  Budget-aware adapters for multi-domain learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 382–391, 2019.
  * [18] Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin.  Learning a universal template for few-shot dataset generalization.  arXiv preprint arXiv:2105.07029, 2021.
  * [19] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han.  Tinytl: Reduce memory, not parameters for efficient on-device learning.  arXiv preprint arXiv:2007.11622, 2020.
  * [20] Debesh Jha, Anis Yazidi, Michael A Riegler, Dag Johansen, Håvard D Johansen, and Pål Halvorsen.  Lightlayers: Parameter efficient dense and convolutional layers for image classification.  In International Conference on Parallel and Distributed Computing: Applications and Technologies, pages 285–296. Springer, 2020.
  * [21] Yeming Wen, Dustin Tran, and Jimmy Ba.  Batchensemble: an alternative approach to efficient ensemble and lifelong learning.  arXiv preprint arXiv:2002.06715, 2020.
  * [22] Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia.  A brief review of domain adaptation.  Advances in Data Science and Information Engineering, pages 877–894, 2021.
  * [23] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [24] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [26] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606–3613, 2014.
  * [27] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008.
  * [28] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.  Novel dataset for fine-grained image categorization: Stanford dogs.  In Proc. CVPR Workshop on Fine-Grained Visual Categorization (FGVC), volume 2. Citeseer, 2011.

##  Appendix A Training procedure

The training of UDTA is composed of three steps: pre-training step,
autoencoder training step, and domain adaptation step.

  1. 1.

Pre-train the backbone network using the source dataset which consists of a
large amount of image data.

  2. 2.

Train the autoencoders using the source dataset. Each autoencoder is trained
by the output features of the backbone network blocks which are connected to
the autoencoders.

  3. 3.

Adapt the image recognition model(backbone network + encoder + UDTA) to a new
domain using fine-grained classification dataset

    1. (a)

Connect UDTA to the backbone network by the encoder.

    2. (b)

Train UDTA for domain adaptation while freezing the backbone network blocks.

Algorithm 1 Pre-training algorithm

Source dataset: $X_{s}=\\{x_{s}^{1},x_{s}^{2},...x_{s}^{n}\\}$ where $n$ is
the total number of the source data

Class labels of source dataset: $Y_{s}=\\{y_{s}^{1},y_{s}^{2},...y_{s}^{n}\\}$

Batch size: $m$

Backbone network parameters: $\theta_{b}$

Classification loss function: $L$

Learning rate: $\eta$

for $i=1$ to number of epochs do

for $j=1$ to $n/m$ do

Sample $m$-size batch from $X_{s}$

$k$ is the index of the data in the batch

Gradient
$\hat{g}\leftarrow\frac{1}{m}\nabla_{\theta_{b}}\Sigma_{k}L(f(x_{s}^{k};\theta_{b}),y_{s}^{k})$

$\theta_{b}\leftarrow\theta_{b}-\eta\hat{g}$

end for

end for

Algorithm 2 Autoencoder training algorithm

Source dataset: $X_{s}=\\{x_{s}^{1},x_{s}^{2},...x_{s}^{n}\\}$ where $n$ is
the total number of the source data

List of parameters of autoencoders:
$\Theta_{AE}=\\{\theta_{ae}^{1},\theta_{ae}^{2},...\theta_{ae}^{h}\\}$ where
$h$ is the number of autoencoders

Batch size: $m$

Backbone network parameters: $\theta_{b}$

Reconstruction loss function: $L_{rec}$

Learning rate: $\eta$

for $i=1$ to number of epochs do

for $j=1$ to $n/m$ do

Sample $m$-size batch from $X_{s}$

$k$ is the index of the data in the batch

for $l=1$ to $h$ do

$o^{l}(x_{s}^{k};\theta_{b})$ is the $l$-th output feature of backbone network
starting from the first block connected to the autoencoder

$f(o^{l}(x_{s}^{k};\theta_{b});\theta_{ae}^{l})$ is the output of $l$-th
autoencoder

Gradient
$\hat{g}^{l}\leftarrow\frac{1}{m}\nabla_{\theta_{ae}^{l}}\Sigma_{k}L_{rec}(f(o^{l}(x_{s}^{k};\theta_{b});\theta_{ae}^{l}),o^{l}(x_{s}^{k};\theta_{b}))$

$\theta_{ae}^{l}\leftarrow\theta_{ae}^{l}-\eta\hat{g}^{l}$

end for

end for

end for

Algorithm 3 Domain adaptation algorithm

Target dataset: $X_{t}=\\{x_{t}^{1},x_{t}^{2},...x_{t}^{n}\\}$ where $n$ is
the total number of the target data

Class labels of target dataset: $Y_{t}=\\{y_{t}^{1},y_{t}^{2},...y_{t}^{n}\\}$

Batch size: $m$

Backbone network parameters: $\theta_{b}$

Encoder parameters: $\theta_{e}$

UDTA parameters: $\theta_{u}$

Classification loss function: $L$

Learning rate: $\eta$

for $i=1$ to number of epochs do

for $j=1$ to $n/m$ do

Sample $m$-size batch from $X_{t}$

$k$ is the index of the data in the batch

Gradient
$\hat{g}\leftarrow\frac{1}{m}\nabla_{\theta_{u}}\Sigma_{k}L(f(x_{t}^{k};\theta_{b},\theta_{e},\theta_{u}),y_{t}^{k})$

$\theta_{u}\leftarrow\theta_{u}-\eta\hat{g}$

end for

end for

##  Appendix B Specification of UDTA attached to MobileNetV2

Table 7: Structure of (a) MobileNetV2, (b) Encoders and (c) UDTA.

Input Operator Expansion Output Repetition Stride $224^{2}$ X $3$ Conv2d -
$112^{2}$ X $32$ 1 2 $112^{2}$ X $32$ IR block 1 $112^{2}$ X $16$ 1 1
$112^{2}$ X $16$ IR block 6 $56^{2}$ X $24$ 2 2 $56^{2}$ X $24$ IR block 6
$28^{2}$ X $32$ 3 2 $28^{2}$ X $32$ IR block 6 $14^{2}$ X $64$ 4 2 $14^{2}$ X
$64$ IR block 6 $14^{2}$ X $96$ 3 1 $14^{2}$ X $96$ IR block 6 $7^{2}$ X $160$
3 2 $7^{2}$ X $160$ IR block 6 $7^{2}$ X $320$ 1 1 $7^{2}$ X $320$ Conv2d 1x1
- $7^{2}$ X $1280$ 1 1 $7^{2}$ X $1280$ Avgpool 7x7 - $1^{2}$ X $1280$ 1 -

(a) MobileNetV2

Input Operator Output Stride $7^{2}$ X $160$ Conv2d 1x1 $7^{2}$ x $80$ 1
$7^{2}$ X $320$ Conv2d 1x1 $7^{2}$ x $160$ 1

(b) Encoders

Input Operator Expansion Output Stride $14^{2}$ X $96$ IR block 6 $7^{2}$ x
$80$ 2 $7^{2}$ X $80$ IR block 6 $7^{2}$ x $160$ 1 $7^{2}$ X $160$ IR block 6
$7^{2}$ x $320$ 1 $7^{2}$ X $320$ Avgpool 7x7 - $1$x$1$ x $320$ -

(c) UDTA

In table (a), each line describes a sequence of one or more repeated
operators. inverted residual block (IR block) consists of 1 x 1 convolution, 3
x 3 depth-wise convolution and 1 x 1 convolution layers. The depth-wise
convolution in the first repetition of the sequence uses a stride represented
in the tables and all others use a stride of 1 [9]. The encoder consists of a
1 x 1 convolution layer.

##  Appendix C Experiments on training encoders separately

We conducted an experiment on training UDTA and encoder during domain
adaptation, comparing with solely training UDTA to scrutinize the
effectiveness of training encoder separately. The separate training of the
encoder achieves significantly increased accuracy with fewer parameters and
less computation.

Table 8: Computation, time, and accuracy of training only UDTA compared with
training UDTA and encoder during domain adaptation. The separate training of
autoencoder reduced the computation for backward propagation by 6.78% from
24,135 MFLOPs to 22,499 MFLOPs and the number of parameters by 6.89% from 929K
to 865K. The accuracy increased from 65.43% to 67.56%

Methods | F.FLOPs | B.FLOPs | F.Time | B.Time | Acc. | Params
---|---|---|---|---|---|---
UDTA + Enc | 97,357M | 24,135M | 9.7ms | 2.6ms | 65.43 | 929K
UDTA | 97,357M | 22,499M | 9.7ms | 2.6ms | 67.56 | 865K

##  Appendix D Sample datasets

(a) Oxford 102 Flowers image samples.

(b) DTD image samples.

(c) Stanford Cars image samples.

(d) Stanford Dogs image samples.

Figure 7: Besides FGVC-Aircraft shown in Figure 5, (a) Oxford 102 Flowers, (b)
DTD, (c) Stanford Cars, and (d) Stanford Dogs are used for domain adaptation
experiments. They are fine-grained datasets consisting of a small number of
images. Fine-grained classification datasets are composed of limited
categories with barely discernible details which make them difficult to be
distinguished. It is even more challenging for DNN to learn enough knowledge
of those domains by a small number of images.