#  Bright X-ray pulsars as sources of MeV neutrinos in the sky

Aman Asthana1, Alexander A. Mushtukov2, Alexandra A. Dobrynina3 and Igor S.
Ognev3
1 Coppell High School, 185 W Parkway Blvd, Coppell, TX 75019 Coppell, USA
2 Astrophysics, Department of Physics, University of Oxford, Denys Wilkinson
Building, Keble Road, Oxford OX1 3RH, UK
3 P.G. Demidov Yaroslavl State University, Sovietskaya 14, 150003 Yaroslavl,
Russia
E-mail<EMAIL_ADDRESS>(AA)E-mail:
<EMAIL_ADDRESS>(AAM)

###### Abstract

High mass accretion rate onto strongly magnetised neutron stars results in the
appearance of accretion columns supported by the radiation pressure and
confined by the strong magnetic field of a star. At mass accretion rates above
$\sim 10^{19}\,{\rm g\,s^{-1}}$, accretion columns are expected to be
advective. Under such conditions, a noticeable part of the total energy
release can be carried away by neutrinos of a MeV energy range. Relying on a
simple model of the neutrino luminosity of accreting strongly magnetised
neutron stars, we estimate the neutrino energy fluxes expected from six ULX
pulsars known up to date and three brightest Be X-ray transits hosting
magnetised neutron stars. Despite the large neutrino luminosity expected in
ULX pulsars, the neutrino energy flux from the Be X-ray transients of our
Galaxy, SMC and LMC is dominant. However, the neutrino flux from the brightest
X-ray transients is estimated to be below the isotropic background by two
orders of magnitude at least, which makes impossible direct registration of
neutrino emission from accreting strongly magnetised neutron stars nowadays.

###### keywords:

accretion – accretion discs – X-rays: binaries – stars: neutron – stars:
oscillations

††pubyear: 2023††pagerange:  Bright X-ray pulsars as sources of MeV neutrinos
in the sky –References

##  1 Introduction

X-ray pulsars (XRPs) are strongly magnetised neutron stars (NSs) in close
binary systems (see, e.g., Mushtukov & Tsygankov 2022). Their emission in
X-rays is caused by the accretion of matter from a companion star, which loses
its mass via stellar wind or through the inner Lagrange point. A strong
magnetic field in XRPs affects the geometry of accretion flow directing it
towards small regions located close to magnetic poles of a NS and details of
the interaction between material and radiation (Harding & Lai, 2006). The
apparent luminosity of XRPs covers orders of magnitude from $10^{32}\,{\rm
erg\ \rm s^{-1}}$ to $10^{41}\,{\rm erg\ \rm s^{-1}}$. The brightest XRPs
belong to the recently discovered class of pulsating ultra-luminous X-ray
sources (ULXs, see Bachetti et al. 2014; Israel et al. 2017a and Fabrika et
al. 2021 for review). The relation between the actual and apparent luminosity
in the brightest ULXs is still under debate. On one hand, extreme accretion
onto a NS can cause strong outflows from accretion disc (Shakura & Sunyaev,
1973), which naturally results in geometrical collimation of X-ray radiation
(Poutanen et al., 2007; King, 2009; King et al., 2017). According to some
authors, the collimation of X-ray radiation can lead to the fact that in
certain directions, the apparent luminosity of the ULXs has orders of
magnitude higher than the actual one (King et al., 2017; King & Lasota, 2020).
On the other hand, the high pulsed fractions observed in pulsating ULXs
largely excludes strong geometrical beaming (Mushtukov et al., 2021; Mushtukov
& Portegies Zwart, 2023), which says in favour of a relatively small
difference between the luminosities. The possible similarity between the
actual and apparent luminosity in ULX hosting NSs is in agreement with the
recent result of population synthesis models (Kuranov et al., 2020) and the
analysis based on the observed luminosity function of high mass X-ray binaries
(Mushtukov et al., 2015b).

High mass accretion rates ($\gtrsim 10^{17}\,{\rm g\,s^{-1}}$) onto the
surface of magnetised NSs results in the appearance of radiation pressure
dominated shock above the surface and accretion columns (Basko & Sunyaev,
1976; Wang & Frank, 1981; Mushtukov et al., 2015a, b) \- structures supported
by radiation pressure and confined by a strong magnetic field.

Under conditions of very high mass accretion rates ($\gtrsim 10^{19}\,{\rm
g\,s^{-1}}$), accretion columns turn into advection dominated state when the
typical time of photon diffusion from the flow becomes comparable or larger
than the dynamical timescale of accretion process (Mushtukov et al., 2018). In
this case, a large fraction of energy can turn into the production of
electron-positron pairs (Mushtukov et al., 2019) and further neutrino emission
due to the pair’s annihilation (Kaminker et al., 1992b). As a result, the
brightest XRPs can be comparably bright in neutrinos.

The theory of neutrino emission from bright XRPs requires the development of
an accurate model of accretion column, which accounts for mechanisms of
magnetic opacity (Suleimanov et al., 2022, 2023) and possibly deviations of
accreting matter state from the thermodynamic equilibrium (see Discussion in
Mushtukov et al. 2019). Nonetheless, some order of magnitude estimations can
be done in the base of the toy model of the accretion column with advection
(Mushtukov et al., 2018). The most promising candidates for the role of
neutrino pulsars are pulsating ULXs (assuming their apparent luminosity is
close to the actual one) and bright Be X-ray transients, which reach
luminosity $\sim 10^{39}\,{\rm erg\ \rm s^{-1}}$ at the peak of their
outbursts (see Reig 2011 for review).

In this paper, we consider six pulsating ULXs known up to date - M82 X-2
(Bachetti et al., 2014), NGC 5907 X-1 (Israel et al., 2017a), NGC 7793 P13
(Fürst et al., 2016; Israel et al., 2017b), NGC 300 X-1 (Carpano et al.,
2018), NGC 1313 X-2 (Sathyaprakash et al., 2019), and M51 X-7 (Rodríguez
Castillo et al., 2020) \- and three of the brightest Be X-ray transients:
Swift J0243.6+6124 (Doroshenko et al., 2020), SMC X-3 (Tsygankov et al.,
2017), and RX J0209.6-7427 (Vasilopoulos et al., 2020; Hou et al., 2022).
Using a toy model of neutrino emission from accreting strongly magnetised NSs
proposed by Mushtukov et al. 2018, we estimate the neutrino luminosity of
considered objects and their corresponding neutrino energy flux at Earth. We
compare the estimated flux level with the flux of neutrino background and
sensitivity of neutrino telescopes.

Table 1:  The table represents the maximal apparent X-ray luminosity of the objects $L_{X}$, expected neutrino luminosity $L_{\nu}$, distance to the source, and corresponding neutrino energy flux $F_{\nu}$ at Earth from six pulsating ULXs discovered up to date and three bright Be XRPs. Estimations of neutrino luminosity are performed under the assumption that the apparent X-ray luminosity is equal to the actual one. NS mass and radius are taken to be $M=1.4M_{\odot}$ and $10^{6}\,{\rm cm}$ respectively.  Name |  $L_{X}$ (max) | $L_{\nu}$ | $D$ | $F_{\nu}$
---|---|---|---|---
| $[{\rm erg\ \rm s^{-1}}]$ | $[{\rm erg\ \rm s^{-1}}]$ | $[{\rm ly}]$ | $[{\rm MeV\,cm^{-2}\,s^{-1}}]$
NGC 5907 X-1 | $1.5\times 10^{41}$ | $3.5\times 10^{41}$ | $5.4\times 10^{7}$ | $6.7\times 10^{-6}$
NGC 1313 X-2 | $2\times 10^{40}$ | $1.3\times 10^{40}$ | $1.3\times 10^{7}$ | $4.3\times 10^{-6}$
M82 X-2 | $1.8\times 10^{40}$ | $9\times 10^{39}$ | $1.1\times 10^{7}$ | $3.9\times 10^{-6}$
M51 X-7 | $7\times 10^{39}$ | $8.4\times 10^{38}$ | $2.5\times 10^{8}$ | $7.4\times 10^{-10}$
NGC 7793 P13 | $5\times 10^{39}$ | $4\times 10^{38}$ | $1.2\times 10^{7}$ | $1.5\times 10^{-7}$
NGC 300 X-1 | $5\times 10^{39}$ | $4\times 10^{38}$ | $6\times 10^{6}$ | $6\times 10^{-7}$
SMC X-3 | $2.5\times 10^{39}$ | $9\times 10^{37}$ | $2\times 10^{5}$ | $1.3\times 10^{-4}$
Swift J0243.6+6124 | $2\times 10^{39}$ | $5.7\times 10^{37}$ | $1.64\times 10^{4}$ | $1.2\times 10^{-2}$
RX J0209.6-7427 | $1.1\times 10^{39}$ | $1.7\times 10^{37}$ | $2\times 10^{5}$ | $2.3\times 10^{-5}$

##  2 Model

Figure 1:  Dependence of X-ray (black solid line) and neutrino (red dashed
line) luminosity on the mass accretion rate onto NS surface (Mushtukov et al.,
2018). The solid orange line approximates neutrino luminosity according to
(1). Total luminosity is given by the dotted black line. One can see that the
total luminosity is dominated by photons at mass accretion rates $\lesssim
2\times 10^{20}\,{\rm g\,s^{-1}}$. At higher mass accretion rates, neutrino
luminosity becomes comparable to the photon luminosity and dominates at
$\dot{M}\gtrsim 10^{21}\,{\rm g\,s^{-1}}$.

The estimation of neutrino luminosity and flux is based on the model proposed
by Mushtukov et al. 2018. The model considers super-critical X-ray pulsars
(Basko & Sunyaev, 1976), where the luminosity of an accreting NS is high
enough to support radiation pressure dominated accretion columns above NS
magnetic poles. If the mass accretion rate onto a NS is high enough, the
optical thickness of the accretion column prevents photon excitation from the
accretion flow making the flow advective. Under these conditions, the internal
temperature can increase up to a few hundred keV or even a MeV. It results in
the production of electron-positron pairs and further neutrino emission due to
the pair’s annihilation $e^{-}e^{+}\to\nu\,\overline{\nu}$. Neutrinos produced
due to the pair annihilation belong to the MeV energy band and are dominated
initially (before accounting for neutrino oscillation) by electron neutrinos
and anti-neutrinos. Neutrino synchrotron emission: $e^{\mp}\to
e^{\mp}\nu\,\overline{\nu}$ also contributes to the total neutrino luminosity,
but its contribution is expected to be relatively small (Kaminker et al.,
1992a). The major uncertainty of the model is due to the fact that the model
relies on the assumption that the concentration of electron-positron pairs is
close to the equilibrium one. This condition may not be fulfilled in the
accretion column because of short dynamical times of accretion ($\sim
10^{-4}\,{\rm s}$), which may be insufficient for the equilibrium
concentration of the pairs (see discussion in Section 2 of Mushtukov et al.
2019). If the concentration of pairs is lower than the equilibrium
concentration, the neutrino emission losses are also lower. Therefore, the
estimates on which we rely should be regarded as optimistic. Uncertainties
associated with the geometry of the accretion channel at the NS surface lead
to uncertainties of the neutrino luminosity determination within a magnitude
of $2-3$.

The expected neutrino luminosity calculated under the assumption of the
equilibrium pair concentration in the accretion channel can be approximated as

$\displaystyle L_{\nu}\approx 0.64\,L_{\rm tot}\,\arctan\left(\frac{L_{\rm
tot}}{8\times 10^{40}\,{\rm erg\ \rm s^{-1}}}\right),$ (1)

where $L_{\rm tot}=L_{\rm X}+L_{\nu}$ is the total luminosity of the source
due to photon and neutrino emission (see Fig. 1 and more accurate results in
Mushtukov et al. 2018).

Both photon and neutrino luminosity experience gravitational redshift. The
luminosity in the observer’s reference frame $L_{\infty}$ is related to the
luminosity at the surface of a NS of mass $M$ and radius $R$ as

$\displaystyle L_{\infty}=L(1-u),$ (2)

where the compactness $u=R_{\rm S}/R$, the Schwarzschild radius of a NS
$R_{\rm S}={2GM}/{c^{2}}\approx 3\times 10^{5}({M}/{M_{\odot}})\,{\rm cm}$,
and $M_{\odot}$ is the mass of the Sun. The dependence of neutrino luminosity
$L_{\nu}$ on the total luminosity $L_{\rm tot}$ is not linear (see Fig. 1).
Thus, the procedure of neutrino luminosity estimation is divided into a few
steps: (i) using the apparent photon luminosity of a NS we get the photon
luminosity at the NS surface using (2), (ii) the photon luminosity at the NS
surface is converted into neutrino luminosity at the NS using approximation
(1) or more accurate estimations (Mushtukov et al., 2018), (iii) we get
neutrino luminosity at the infinity accounting for the gravitational redshift
(2). Because the apparent luminosity can be different from the actual one due
to the geometrical beaming (King et al., 2017; Mushtukov et al., 2021), we
illustrate the influence of this uncertainty assuming that the difference
between two photon luminosities is within a factor of 2. With the known
neutrino luminosity corrected for the gravitational redshift and distance to
the source, we can estimate the expected neutrino energy flux at Earth.

##  3 Estimations of neutrino luminosity

Estimating neutrino luminosity, we assume NS mass and radius to be
$M=1.4M_{\odot}$ and $R=10^{6}\,{\rm cm}$ respectively. All ULX pulsars are
known to be strongly variable (see discussion on the observed variability in
M82 X-2 by Tsygankov et al. 2016, in NGC 5907 X-1 by Israel et al. 2017a, in
NGC 1313 X-2, M51 X-7, NGC 7793 P13 and NGC 300 X-1 by Gúrpide et al. 2021)
and our estimations of their neutrino luminosity and corresponding neutrino
energy flux are based on the maximal detected X-ray luminosity.

The estimated neutrino luminosity and neutrino energy flux $F_{\nu}$ in six
pulsating ULXs and in three bright Be X-ray transients at the peak of their
outbursts are represented in Table 1. Despite relatively large neutrino
luminosity expected in the ULXs, due to the difference in distance to the
sources, the neutrino energy flux at Earth is below $10^{-5}\,{\rm
MeV\,cm^{-2}\,s^{-1}}$, which is smaller than the neutrino flux expected from
considered Be X-ray transients.

By using recent observations of three bright X-ray transients - Swift
J0243.6+6124 , SMC X-3 and RX J0209.6-7427 - during their outbursts, we have
obtained neutrino light curves during the outbursts (see blue squares in Fig.
2, 3 and 4). Because the neutrino luminosity is strongly dependent on the
total luminosity of accreting NS, the uncertainties in estimations of actual
X-ray luminosity result in significant uncertainties in estimations of
neutrino luminosity and energy flux. Blue regions in Fig. 2, 3, 4 illustrate
the uncertainty in neutrino luminosity due to the uncertainty in photons
luminosity within a factor of 2. The majority of neutrinos are emitted during
the main peak of the outbursts in Be XRPs. When integrating the neutrino light
curve obtained for three Be transients, we estimated the total energy emitted
with neutrinos during the outbursts: $\sim 5\times 10^{43}\,{\rm erg}$ in
Swift J0243.6+6124 , $\sim 1.7\times 10^{44}\,{\rm erg}$ in SMC X-3, and $\sim
4.5\times 10^{43}\,{\rm erg}$ in RX J0209.6-7427 .

The estimations of neutrino luminosity are dependent on NS mass and radius
because of different contributions of the gravitational redshift. However, the
uncertainties due to the unknown mass and radius of a NS are much smaller than
the uncertainties due to the relation between the actual and apparent
luminosity of the objects.

Figure 2:  Black circles represent the photon light curve of Swift
J0243.6+6124 during its outburst in 2017-2018 reproduced on the base of data
from Doroshenko et al. 2020, while blue squares represent neutrino light curve
calculated on the base of Mushtukov et al. 2018.  Figure 3:  Black circles
represent the photon light curve of SMC X-3 during its outburst in 2016-2017
reproduced on the base of data from Tsygankov et al. 2017, while blue squares
represent neutrino light curve calculated on the base of Mushtukov et al.
2018.  Figure 4:  Black circles represent the photon light curve of RX
J0209.6-7427 during its outburst in 2019-2020 reproduced on the base of data
from Hou et al. 2022 , while blue squares represent neutrino light curve
calculated on the base of Mushtukov et al. 2018.

##  4 Neutrino flux property and detection

###  4.1 Neutrino composition and energy spectrum

The possibility of neutrino detection at Earth is determined not only by the
neutrino flux $F_{\nu}$ but also by neutrino species composition and their
spectral distribution. The dominant process of neutrino emission from
relatively hot electron-positron plasma is annihilation of electron-positron
pairs (see, e. g., Kaminker et al. 1992a). The result of this reaction is
production of neutrino and antineutrino of one of three flavors: electron
($\nu_{e}$, $\overline{\nu}_{e}$), muon ($\nu_{\mu}$, $\overline{\nu}_{\mu}$),
or tau ($\nu_{\tau}$, $\overline{\nu}_{\tau}$). In the case of non-
relativistic plasma and under the condition of extremely strong magnetic field
$B\gg B_{0}\approx 4.41\times 10^{13}$ G, neutrino emissivity due to the
annihilation is proportional to $c_{V}^{2}+c_{A}^{2}$, while in the case of
$B\ll B_{0}$, the emissivity of proportional to $c_{V}^{2}$ (Kaminker et al.,
1992b). Here $c_{V}=\pm 1/2+2\sin^{2}\theta_{W}$ and $c_{A}=\pm 1/2$ are the
vector and axial constants of the charged lepton current, and
$\sin^{2}\theta_{W}\approx 0.2312$, where $\theta_{W}$ is the Weinberg angle
(Workman & Others, 2022). The upper (plus) signs in $c_{V}$ and $c_{A}$
correspond to the electron flavor, while the lower (minus) signs correspond to
muon and tau species. So we could reconstruct the initial contributions of a
different flavor to summarize neutrino luminosity $L_{\nu}$. The initial
electron neutrino and antineutrino luminosities are

$\displaystyle L_{\nu_{e}}\\!\\!=L_{\overline{\nu}_{e}}\\!\\!\equiv
f_{0e}\,L_{\nu}$ (3)

and the initial muon/tau neutrino and antineutrino luminosities are

$\displaystyle
L_{\nu_{\mu}}\\!\\!=L_{\overline{\nu}_{\mu}}\\!\\!=L_{\nu_{\tau}}\\!\\!=L_{\overline{\nu}_{\tau}}\\!\\!\equiv
f_{0x}\,L_{\nu},$ (4)

where $f_{0e}\approx 0.35$, $f_{0x}\approx 0.075$ for the case of magnetar-
lile magnetic field strength $B\gg B_{0}$, and $f_{0e}\approx 0.4985$,
$f_{0x}\approx 0.00075$ for the case of relatively weak magnetic fields $B\ll
B_{0}$ (Kaminker et al., 1992b). We neglect neutrino oscillation in the matter
of X-ray source due to the essential uncertainty of accretion column geometry
and its small extent. Gravitational redshift does not affect the flavor
composition of neutrino flux.

However, neutrino propagation over the distance between the source and
observer affects the composition of the flux due to the vacuum neutrino
oscillations. The initial and final flavor fractions are related as (Mena et
al., 2007):

$\displaystyle\begin{gathered}f_{e}=f_{0e}-(f_{0e}-f_{0x})\,\sin^{2}(2\theta_{12})/2,\\\
f_{x}=f_{0x}-(f_{0e}-f_{0x})\,\sin^{2}(2\theta_{12})/4,\end{gathered}$ (7)

where $\sin^{2}\theta_{12}\approx 0.307$ (Workman & Others, 2022) is a squared
sin of the neutrino mixing angle. Hence we obtain $f_{e}\approx 0.23$,
$f_{x}\approx 0.135$ for magnetar-like accretor and $f_{e}\approx 0.29$,
$f_{x}\approx 0.105$ for the accretor of weaker surface magnetic field. These
values are equal approximately, so we used the following averages:

$\displaystyle f_{e}=1/4,\quad f_{x}=1/8.$ (8)

To restore the neutrino spectrum, we follow the result of paper by Misiaszek
et al. 2006. Possibility of neutrino detection could be obtained from the
specific neutrino flux $\varphi_{\nu}$, which has the dimension
$[\varphi_{\nu}]=\rm{[cm^{-2}s^{-1}MeV^{-1}]}$. The specific flux is related
to neutrino energy flux $F_{\nu}$ as

$\displaystyle
F_{\nu}=\int\limits_{0}^{\infty}\varepsilon\,\varphi_{\nu}(\varepsilon)\,d\varepsilon,$
(9)

where $\varepsilon$ is neutrino energy measured in MeV. Following the result
of Misiaszek et al. 2006, we approximate neutrino flux at Earth from an X-ray
source as

$\displaystyle\varphi_{\nu}(\varepsilon)=\frac{s^{s}F_{\nu}}{\varGamma(s)\;\overline{\varepsilon}^{2}}\left(\frac{\varepsilon}{\overline{\varepsilon}}\right)^{s-1}\exp\left(-s\frac{\varepsilon}{\overline{\varepsilon}}\right),$
(10)

where the parameter $s\approx 4.181$ and the mean neutrino energy
$\overline{\varepsilon}=m_{e}+3/2\,T_{X}$, $T_{X}$ is the temperature of
neutrino emitted medium and $m_{\rm e}\approx 0.511$ MeV is electron mass.
Note, that the spectra of different neutrino flavors are the same due to the
kinematics of the annihilation process, and functions $\varphi_{\nu_{i}}$
corresponding to different flavors are differed from $\varphi_{\nu}$ by a
factor $f_{e}$ or $f_{x}$ only (8).

Figure 5:  Set electron antineutrino fluxes $\varphi_{\overline{\nu}_{e}}$ as
a function of energy. Swift J0243.6+6124: curves for different temperatures of
neutrino emitted area $T_{X}\\!=\\!0$ MeV (blue short dashed),
$T_{X}\\!=\\!0.25$ MeV (green dashed), $T_{X}\\!=\\!0.5$ MeV (orange long
dashed) Diffusion supernova background: pure neutron star $T_{\rm SN}\\!=\\!4$
MeV (red dot and long dash) and pure black hole $T_{\rm SN}\\!=\\!6$ MeV
(purple dot and dash) forming. Borexino model-independent limit on electron
antineutrino fluxes from unknown sources (brown solid).

The dependence of the neutrino spectrum on temperature is shown in Fig. 5. One
can see that the spectrum of neutrino emitted by hotter material is wider but
lower at its maximum value.

Figure 6:  The energy spectrum of neutrino background at Earth summed over
flavors in the energy range from $10\,{\rm eV}$ to $10^{12}\,{\rm eV}$ (taken
from Vitagliano et al. 2020 and Brocato et al. 1998). The background is
composed of solar thermal (A∗) and nuclear neutrinos (B∗), neutrinos from
nuclear reactors (C∗), geoneutrinos (D∗), isotropic component of neutrino flux
from stars of the Galaxy (E), neutrinos from old supernovae (F), and
atmospheric neutrinos (G). Dashed lines represent non-isotropic neutrino
background, while the background represented by solid lines is expected to be
isotropic. Note, that the total neutrino flux due to the stars of the Galaxy
is not isotropic and expected to be larger (by a factor $\sim 8$ in comparison
to the isotropic component) in the direction towards the Galactic center (see
Fig. 8 in Brocato et al. 1998). Two rectangles show the area, where we expect
neutrino flux from three bright BeXRPs at the peak of their outbursts (solid
blue) and from six pulsating ULXs (dashed black).

###  4.2 Neutrinos detection

As seen from Table 1, pulsating ULXs have higher neutrino luminosity in
comparison to three bright Be XRPs considered in the paper. However, the
neutrino energy flux at the Earth is smaller due to the large distance to
ULXs. Below we discuss the possibility of neutrino detection Swift
J0243.6+6124 , which neutrino energy flux at the Earth is expected to the the
highest one.

The observability of accreting NSs in neutrinos is determined by the
sensitivity of neutrino telescopes and the ratio of neutrino flux from sources
to the background. The sources of neutrino background can be isotropic and
non-isotropic. Non-isotropic neutrino background in the energy band from
$0.01$ to $100\,{\rm MeV}$ is dominated by solar thermal and nuclear neutrinos
(Fig. 6, A∗ and B∗ components), neutrinos from the ground based reactors (Fig.
6, C∗) and geoneutrinos (Fig. 6, D∗). The isotropic neutrino flux in the same
energy band is due to neutrino emission of the nearest stars of the Galaxy
(Fig. 6, E), old core-collapse supernova explosions (Fig. 6, F) and
atmospheric neutrinos (Fig. 6, G).

Non-isotropic neutrino background is dominated over the isotropic background
at a MeV energy range. Solar nuclear neutrino flux is estimated to be

$F_{\odot}\gtrsim 10^{8}\,{\rm MeV\,cm^{-2}\,s^{-1}}.$

Decay of radioactive isotopes in the Earth provides the flux of terrestrial
antineutrinos (Dye, 2012)

$F_{\oplus}\sim 10^{6}\,{\rm MeV\,cm^{-2}\,s^{-1}}.$

Nuclear reactors release a few percents of their total energy production in
the form of MeV-range antineutrinos and cause the flux (Mueller et al., 2011)

$F_{\rm nucl}\sim F_{\oplus}\sim 10^{6}\,{\rm MeV\,cm^{-2}\,s^{-1}}.$

Isotropic neutrino background (its components are shown by solid lines in Fig.
6) is a few orders of magnitude lower in comparison to the non-isotropic one.
Ancient core collapse supernovae provide the neutrino flux

$F_{\rm SN}\sim 1\,{\rm MeV\,cm^{-2}\,s^{-1}}.$

Neutrino flux from stars of the Galaxy is strongly dependent on energy (see
curve E in Fig. 6) and direction: the isotropic component due to the neutrino
emission by the nearest stars is estimated to be almost an order of magnitude
smaller than the flux in the direction towards the Galactic center (Brocato et
al., 1998). The composition of Galactic neutrinos is similar to the
composition of neutrino flux from the Sun. The atmospheric neutrino and
antineutrino flux due to interaction of cosmic rays with the atmosphere of
Earth is estimated to be

$F_{\rm atm}<10^{-2}\,{\rm MeV\,cm^{-2}\,s^{-1}}$

at a MeV energy range (Honda et al., 2015).

Detecting electron neutrinos from an X-ray source is difficult due to
significant background from the Sun and Galactic stars. Thus, observation of
accreting NSs is more preferred in the other species of neutrino and
antineutrino ($\nu_{\mu}\,/\overline{\nu}_{\mu}$ or
$\nu_{\tau}/\overline{\nu}_{\tau}$). The most significant isotropic neutrino
background of mixed flavours is provided by old core-collapse supernovae
explosions, which are sources of all neutrino species. Although this
background has an average energy of $\sim\\!\\!6$ MeV, its estimated neutrino
flux (Vitagliano et al., 2020):

$\displaystyle\begin{aligned} \varphi_{\rm
SN}(\varepsilon)\approx\,&5.12\left(\frac{4\,\text{MeV}}{T_{\rm
SN}}\right)^{2}\arctan\left[3\left(\frac{\varepsilon}{T_{\rm
SN}}\right)^{3/2}\right]\\\ &\times e^{-{1.03\varepsilon}/{T_{\rm
SN}}}\end{aligned}$ (11)

exceed value corresponding to Swift J0243.6+6124. Supernova neutrino flavor
composition is expected to be different for the explosions leading to
production of NSs and black holes (Sumiyoshi et al., 2006; Walk et al., 2020).
In the case of NS remnant, the flavour composition is given by
$f_{e}\\!\approx\\!1/4$, $f_{x}\\!\approx\\!1/8$, and $T_{\rm SN}\approx
4\,{\rm MeV}$, while in the case of black hole production neutrinos are
dominated by the electron flavour: $f_{e}\\!\approx\\!2/5$,
$f_{x}\\!\approx\\!1/20$, and $T_{\rm SN}\approx 6\,{\rm MeV}$ (Vitagliano et
al., 2020). As one can see from the Fig. 5, the diffusion supernova background
is in excess of the electron antineutrino emission from Swift J0243.6+6124
even in a MeV energy band. A similar result is for other neutrino species.
Thus, old core-collapse supernova could produce significant background even
for the brightest in neutrino X-ray source Swift J0243.6+6124.

Detection of MeV neutrinos and antineutrinos requires use of underground
detectors. Further, we estimate the possibility of neutrino detection from
X-ray sources using data from the Borexino underground detector (Alimonti et
al., 2009). We use a model-independent upper limit in the energy range
$1.8-16.8\,{\rm MeV}$ on electron antineutrino fluxes from unknown sources
Agostini et al. (2021). In this paper, the monochromatic neutrinos model with
1 MeV wide bins was used, so the upper limit is a piecewise function. The
result is shown in Fig. 5. As one can see from the picture, the Borexino
model-independent limit exceeds electron antineutrino flux from Swift
J0243.6+6124 by more than eight orders of magnitude. Note that the Borexino
limit obtains through detector neutrino events of inverse beta decay of the
free proton. The energy threshold of this reaction is 1.8 MeV, so the maximum
neutrino spectrum from X-ray sources occurs in the lower energy range.
Therefore the reaction of neutrino electron scattering is preferable for the
detection of neutrinos from an X-ray source. Moreover, all neutrino species
are involved in this reaction, in contrast to the inverse beta decay, where
only the electron antineutrino participates. All backgrounds except old
supernovae are electron neutrinos or antineutrinos. Thus, muon and tau
(anti)neutrinos detection is the preferable strategy for the detection of
neutrino signal from bright XRPs.

As discussed above, even neutrino brightest X-ray pulsar Swift J0243.6+6124
has the flux (i) under the background of diffuse neutrino from old supernovae,
and (ii) significantly less than the reliable threshold of neutrino detection
by existent detectors. Finally, we estimate the possibility of neutrino
detection from near the Earth but unknown at present X-ray sources. For this
purpose, we obtain the following simple approximation data from Table 1:
$L_{\nu}\approx 1.6\times 10^{39}({L_{X}}/{10^{40}\,\text{erg s}^{-1}})^{2}.$
Note that approximation values differ from data less than 50 %. Hence we
obtain an estimation of the distance on which neutrinos from the X-ray sources
could be detected at the maximum of their spectrum:

$\displaystyle
D_{*}\sim\left(\frac{10^{5}\,\text{cm${}^{-2}$s${}^{-1}$MeV${}^{-1}$}}{\varphi_{th}}\right)^{1/2}\left(\frac{L_{\rm
X}}{10^{39}\,{\rm erg\ \rm s^{-1}}}\right)\,{\rm ly},$ (12)

where $\varphi_{th}$ is threshold of detection neutrino flux. It follows from
the estimation that Be XRPs, able to be detected by existent neutrino
telescope, should be located among the nearest stars. Whereas pulsating ULXs
could be detected from a distance of a few hundred light years. Note that a
special method of selecting neutrino events in the detector, such as taking
into account the direction of X-ray sources, detection muon and tau
(anti)neutrinos, could decrease the detection threshold and increase the
distance to the detectable X-ray objects.

##  5 Summary

In this paper, we have considered accreting strongly magnetized NSs as
possible sources of neutrino emission. Applying the model proposed by
Mushtukov et al. 2018 to six ULX pulsars and three bright Be XRPs, we have
estimated neutrino luminosity and neutrino energy flux from these objects (see
Table 1). Despite the large total neutrino luminosity of ULX pulsars, the
neutrino energy flux from the Be X-ray transients of our Galaxy (Swift
J0243.6+6124 ) and Magellanic Clouds (SMC X-3 and RX J0209.6-7427 ) is
expected to be dominant due to a relatively small distance from them. For the
case of Be XRPs, we have provided the expected neutrino light curve during
their recent outbursts (see Fig. 2, 3 and 4). The total energy release due to
the neutrino emission during the outbursts was estimated to be at the level of
$10^{44}\,{\rm erg}$.

Because the major process of neutrino emission in bright XRPs is the
annihilation of electron-positron pairs, the flux is composed equally of
neutrinos and antineutrinos of a MeV energy range, where about a half of
particles belong to the electron flavour. In a MeV energy range, the neutrino
background is dominated by non-isotropic sources: solar neutrinos,
antineutrinos from nuclear reactors and terrestrial antineutrinos due to the
decay of radioactive isotopes (B∗,C∗,D∗ curves in Fig. 6 respectively).
Isotropic neutrino background is a few orders of magnitude lower and in a MeV
energy range dominated by neutrinos/antineutrinos from old core-collapse
supernova explosions and neutrinos from the nearest stars of the Galaxy (F and
E curves in Fig. 6 respectively). Because there is a lack of antineutrinos in
the flux from the Sun and stars of the Galaxy, it is preferable to observe
accreting NSs in antineutrinos. However, the flux from bright Be X-ray
transients considered in our paper is still more than two orders of magnitude
below the isotropic neutrino background due to old supernova explosions (see
Fig. 6). Therefore, direct registration of neutrino emission from accreting
NSs seems impossible at present and confirmation of significant energy losses
with neutrino by NSs requires the development of indirect methods.

## Acknowledgements

AAM thanks UKRI Stephen Hawking fellowship. We are grateful to an anonymous
referee for useful comments and suggestions.

## Data availability

The calculations presented in this paper were performed using a private code
developed and owned by the corresponding author. All the data appearing in the
figures are available upon request.

## References

  * Agostini et al. (2021) Agostini M., et al., 2021, Astropart. Phys., 125, 102509
  * Alimonti et al. (2009) Alimonti G., et al., 2009, Nuclear Instruments and Methods in Physics Research A, 600, 568
  * Bachetti et al. (2014) Bachetti M., et al., 2014, Nature, 514, 202
  * Basko & Sunyaev (1976) Basko M. M., Sunyaev R. A., 1976, MNRAS, 175, 395
  * Brocato et al. (1998) Brocato E., Castellani V., degl’Innocenti S., Fiorentini G., Raimondo G., 1998, A&A, 333, 910
  * Carpano et al. (2018) Carpano S., Haberl F., Maitra C., Vasilopoulos G., 2018, MNRAS, 476, L45
  * Doroshenko et al. (2020) Doroshenko V., et al., 2020, MNRAS, 491, 1857
  * Dye (2012) Dye S. T., 2012, Reviews of Geophysics, 50, RG3007
  * Fabrika et al. (2021) Fabrika S. N., Atapin K. E., Vinokurov A. S., Sholukhova O. N., 2021, Astrophysical Bulletin, 76, 6
  * Fürst et al. (2016) Fürst F., et al., 2016, ApJ, 831, L14
  * Gúrpide et al. (2021) Gúrpide A., Godet O., Koliopanos F., Webb N., Olive J. F., 2021, A&A, 649, A104
  * Harding & Lai (2006) Harding A. K., Lai D., 2006, Reports on Progress in Physics, 69, 2631
  * Honda et al. (2015) Honda M., Athar M. S., Kajita T., Kasahara K., Midorikawa S., 2015, Phys. Rev. D, 92, 023004
  * Hou et al. (2022) Hou X., et al., 2022, ApJ, 938, 149
  * Israel et al. (2017a) Israel G. L., et al., 2017a, Science, 355, 817
  * Israel et al. (2017b) Israel G. L., et al., 2017b, MNRAS, 466, L48
  * Kaminker et al. (1992a) Kaminker A. D., Levenfish K. P., Yakovlev D. G., Amsterdamski P., Haensel P., 1992a, Phys. Rev. D, 46, 3256
  * Kaminker et al. (1992b) Kaminker A. D., Gnedin O. Y., Yakovlev D. G., Amsterdamski P., Haensel P., 1992b, Phys. Rev. D, 46, 4133
  * King (2009) King A. R., 2009, MNRAS, 393, L41
  * King & Lasota (2020) King A., Lasota J.-P., 2020, MNRAS, 494, 3611
  * King et al. (2017) King A., Lasota J.-P., Kluźniak W., 2017, MNRAS, 468, L59
  * Kuranov et al. (2020) Kuranov A. G., Postnov K. A., Yungelson L. R., 2020, Astronomy Letters, 46, 658
  * Mena et al. (2007) Mena O., Mocioiu I., Razzaque S., 2007, Phys. Rev. D, 75, 063003
  * Misiaszek et al. (2006) Misiaszek M., Odrzywolek A., Kutschera M., 2006, Phys. Rev. D, 74, 043006
  * Mueller et al. (2011) Mueller T. A., et al., 2011, Phys. Rev. C, 83, 054615
  * Mushtukov & Portegies Zwart (2023) Mushtukov A. A., Portegies Zwart S., 2023, MNRAS, 518, 5457
  * Mushtukov & Tsygankov (2022) Mushtukov A., Tsygankov S., 2022, arXiv e-prints, p. arXiv:2204.14185
  * Mushtukov et al. (2015a) Mushtukov A. A., Suleimanov V. F., Tsygankov S. S., Poutanen J., 2015a, MNRAS, 447, 1847
  * Mushtukov et al. (2015b) Mushtukov A. A., Suleimanov V. F., Tsygankov S. S., Poutanen J., 2015b, MNRAS, 454, 2539
  * Mushtukov et al. (2018) Mushtukov A. A., Tsygankov S. S., Suleimanov V. F., Poutanen J., 2018, MNRAS, 476, 2867
  * Mushtukov et al. (2019) Mushtukov A. A., Ognev I. S., Nagirner D. I., 2019, MNRAS, 485, L131
  * Mushtukov et al. (2021) Mushtukov A. A., Portegies Zwart S., Tsygankov S. S., Nagirner D. I., Poutanen J., 2021, MNRAS, 501, 2424
  * Poutanen et al. (2007) Poutanen J., Lipunova G., Fabrika S., Butkevich A. G., Abolmasov P., 2007, MNRAS, 377, 1187
  * Reig (2011) Reig P., 2011, Ap&SS, 332, 1
  * Rodríguez Castillo et al. (2020) Rodríguez Castillo G. A., et al., 2020, ApJ, 895, 60
  * Sathyaprakash et al. (2019) Sathyaprakash R., et al., 2019, MNRAS, 488, L35
  * Shakura & Sunyaev (1973) Shakura N. I., Sunyaev R. A., 1973, A&A, 24, 337
  * Suleimanov et al. (2022) Suleimanov V. F., Mushtukov A. A., Ognev I., Doroshenko V. A., Werner K., 2022, MNRAS, 517, 4022
  * Suleimanov et al. (2023) Suleimanov V. F., Mushtukov A., Ognev I., Doroshenko V. A., Werner K., 2023, IAU Symposium, 363, 327
  * Sumiyoshi et al. (2006) Sumiyoshi K., Yamada S., Suzuki H., Chiba S., 2006, Phys. Rev. Lett., 97, 091101
  * Tsygankov et al. (2016) Tsygankov S. S., Mushtukov A. A., Suleimanov V. F., Poutanen J., 2016, MNRAS, 457, 1101
  * Tsygankov et al. (2017) Tsygankov S. S., Doroshenko V., Lutovinov A. A., Mushtukov A. A., Poutanen J., 2017, A&A, 605, A39
  * Vasilopoulos et al. (2020) Vasilopoulos G., et al., 2020, MNRAS, 494, 5350
  * Vitagliano et al. (2020) Vitagliano E., Tamborra I., Raffelt G., 2020, Rev. Mod. Phys., 92, 45006
  * Walk et al. (2020) Walk L., Tamborra I., Janka H.-T., Summa A., Kresse D., 2020, Phys. Rev. D, 101, 123013
  * Wang & Frank (1981) Wang Y. M., Frank J., 1981, A&A, 93, 255
  * Workman & Others (2022) Workman R. L., Others 2022, PTEP, 2022, 083C01


# Multi-Task Learning for Screen Content Image Coding

Rashid Zamanshoar Heris and Ivan V. Bajić  School of Engineering Science,
Simon Fraser University, Burnaby, BC, V5A 1S6, Canada

###### Abstract

With the rise of remote work and collaboration, compression of screen content
images (SCI) is becoming increasingly important. While there are efficient
codecs for natural images, as well as codecs for purely-synthetic images,
those SCIs that contain both synthetic and natural content pose a particular
challenge. In this paper, we propose a learning-based image coding model
developed for such SCIs. By training an encoder to provide a latent
representation suitable for two tasks – input reconstruction and
synthetic/natural region segmentation – we create an effective SCI image codec
whose strong performance is verified through experiments. Once trained, the
second task (segmentation) need not be used; the codec still benefits from the
segmentation-friendly latent representation.

###### Index Terms:

Image compression, screen content image, learning-based compression, image
segmentation

##  I Introduction

Traditional image compression standards such as JPEG [1] and JPEG2000 [2] were
developed mostly with natural images in mind. However, with the rise of remote
work and collaboration, transmission of screen content images (SCI) has become
important. Recognizing this, a screen content coding extension [3] was
developed for High Efficiency Video Coding (HEVC). More recently, low-level
coding techniques tailored to screen content were introduced into the
Versatile Video Coding (VVC) standard [4, 5, 6]. Due to the unique signal
characteristics of SCIs, such as sharp edges, repetitive patterns, and the
absence of noise, many encoding techniques that are effective for natural
images are either ineffective or less efficient for SCIs [7]. As an example,
Fig. 1 shows one natural and one screen content image along with their
luminance histograms. It is easy to see that the histograms are very
different. Also the effect of coding artifacts in the two types of images can
be different [8]. For instance, quality loss near the edges is a common side
effect of compressing natural images; yet, if done right, it may be
imperceptible. However, in synthetic images, blurring of sharp edges (e.g., in
SCIs showing text) can be quite noticeable and annoying, and may affect the
ability of the viewer to understand the text.

Figure 1: Top: a natural image from the CLIC dataset [9] and its luma
histogram. Bottom: a SCI from the SCID dataset [10] and its luma histogram.

In the meantime, steady progress has been made in learning-based image
compression [11, 12, 13, 14, 15, 16, 17], however, the focus has been mostly
on natural images. When used on SCIs, coding models trained on natural images
tend to be less effective, as will be seen in our results. Therefore, in this
paper, we develop a learning-based coding model targeted at SCI. Note that
SCIs can contain both synthetic and natural content. In fact, one of the key
challenges in SCI compression is to adapt coding to the nature of the content
(synthetic vs. natural), because these have different statistics and therefore
may require different coding tools. We leverage the fact that a learning-based
model can be trained to distinguish natural from synthetic content, as well as
perform compression, in order to boost the coding performance on SCI.

The paper is structured as follows. In Section II, we provide a brief overview
of SCI compression and learning-based image compression, and outline our
contribution. The proposed SCI coding model is presented in Section III.
Experiments are described in Section IV followed by conclusions in Section V.

##  II Related work and our contribution

###  II-A Traditional screen content image compression

High-Efficiency Video Coding (HEVC) standard contains specialized tools for
screen content compression, which have been wrapped into its screen content
coding extension (HEVC-SCC) [3, 4]. These include pallete mode [18], sample-
based prediction [19], and DPCM-based edge prediction [20]. Line-by-line
template matching, which is more flexible than HEVC’s native intra-block-copy
mode, has been proposed in [21]. In [22], a layer-based method is presented
for extracting the synthetic data from the screen content image. While the
remaining natural video is compressed using HEVC-SCC, detected text and
graphical structures are encoded using a customized compression strategy. In
[23], a pre-processing step is suggested to alter parts of the original image
data based on the utility of the related image region in order to decrease the
bitrate. Additionally, [24] and [25] have examined the quality of screen
content images when using lossy compression. Versatile Video Coding (VVC)
standard [7] also includes low-level coding tools that are suitable for screen
content coding.

###  II-B Learning-based image compression

Learning-based image compression has made rapid progress in recent years.
Early works [26, 27, 28] employed Recurrent Neural Networks (RNNs) to model
spatial relationships in an image. More recent models are based Convolutional
Neural Networks (CNNs) and have a structure shown in the top part of Fig. 2.
An image $X$ is subject to a nonlinear analysis transform [11, 12], composed
of convolutional layers, nonlinear activations, and normalizations, among
which generalized divisive normalization (GDN) [29] is a popular choice. The
latent representation $Y$ is then quantized to $\widehat{Y}$ and entropy-coded
to produce the compressed bit stream. An important part of the learning
process is estimating the rate $R$, i.e., modeling the entropy of
$\widehat{Y}$. Entropy models have become more sophisticated in recent years,
involving factorized priors and hyperpriors [13], autoregressive and
hierarchical priors [14], channel-wise autoregressive models [15], and
Gaussian mixtures [16]. Most recent work has explored advanced learned
transforms such as normalizing flows [30] and transformers [31]. Common to all
end-to-end learning-based codecs is that differentiable estimates of rate $R$
and distortion $D$ are use to optimize the whole system [17].

Learning-based image compression has largely focused on natural images so far.
However, recently, [32] presented a learned image codec that incorporates the
concept of transform skip into the end-to-end pipeline to improve SCI
compression. The authors also showed that retraining an end-to-end learned
pipeline on SCI would improve SCI compression performance, indicating the
effect of “domain shift” between natural and screen content images. Our paper
provides further quantification of the effects of this domain shift, and
proposes an alternative training strategy to improve SCI coding performance.

###  II-C Our contribution

Our contributions can be summarized as follows:

  * •

We quantify the performance gap of two learning-based codecs [13, 15] due to
“domain shift” from natural to synthetic and mixed natural-synthetic content.

  * •

We develop a multi-task SCI coding approach that is able to distinguish
natural from synthetic content in SCI, to help boost coding performance.

  * •

We demonstrate the strong coding performance of the proposed approach by
comparing it against relevant benchmarks, both handcrafted and learned.

Figure 2: Top: high-level structure of a learning-based image codec. Bottom:
proposed approach. Figure 3: An example of a SCI used in our experiments
(left), with its ground-truth synthetic/natural (S/N) segmentation map
(right).

##  III Proposed methods

###  III-A Multi-task coding model

The main idea behind the proposed SCI codec is that we want to create a latent
representation that is both compressible and has sufficient information about
synthetic and natural parts of the input image, to help the codec adjust to
different statistics of these regions. To accomplish this, we propose a multi-
task architecture shown in the bottom of Fig. 2. Image $X$ is input to
learnable analysis transform $T_{a}$ to produce the latent representation $Y$
which is then quantized into $\widehat{Y}$:

$Y=T_{a}(X),\qquad\widehat{Y}=Q(Y).$ (1)

The quantized representation $\widehat{Y}$ is entropy coded to produce the
compressed bit stream. At the decoder, the bit stream is entropy decoded to
recover $\widehat{Y}$, which is then passed to two synthesis transforms,

$\widehat{X}=T_{s}^{(1)}(\widehat{Y}),\qquad\widehat{S}=T_{s}^{(2)}(\widehat{Y}),$
(2)

to generate the reconstructed input image $\widehat{X}$ and the
synthetic/natural (S/N) segmentation map $\widehat{S}$, which is supposed to
separate synthetic from natural parts of the SCI image $X$. An example of such
a segmentation map is shown in Fig. 3 next to its corresponding input image.
We describe later in this section how we generate input data (both SCIs and
S/N segmentation maps) for training the proposed system.

Following [13], the rate is estimated as

$R=\mathbb{E}\left[-\textup{log}_{2}p_{\widehat{Y}}(\widehat{Y})\right]+\mathbb{E}\left[-\textup{log}_{2}p_{\widehat{Z}}(\widehat{Z})\right],$
(3)

where $\widehat{Z}$ is the quantized hyperprior used in entropy modeling,
$p_{\widehat{Y}}$ and $p_{\widehat{Z}}$ are the probability distributions of
$\widehat{Y}$ and $\widehat{Z}$, respectively, and the expectation is taken
over input data. The loss function used in the training is

$\mathcal{L}=R+\lambda\cdot D_{X}+\varphi\cdot D_{S},$ (4)

where $D_{X}$ and $D_{S}$ are distortion measures for input reconstruction and
S/N segmentation, respectively. We used $D_{X}=\text{MSE}(X,\widehat{X})$ and
$D_{S}=\text{MSE}(S,\widehat{S})$ in the experiments, but other choices are
possible, such as SSIM [33] for reconstruction or Intersection-over-Union for
segmentation. Once the model is trained, the segmentation output need not be
used; it’s purpose was to make the system aware of synthetic/natural parts of
the input.

Note that any transforms, quantizers, and entropy models from learning-based
coding pipelines can be used to construct the proposed multi-task codec. To
illustrate this, we conducted our experiments with components from two
pipelines, namely [13] and [15], and show that in both cases, the proposed
multi-task approach improves SCI coding performance compared to the default
pipeline.

###  III-B Dataset

The proposed multi-task model requires ground-truth S/N segmentation masks for
training, to be able to compute $D_{S}$. Hence, we constructed such a
dataset111https://github.com/SFU-Multimedia-Lab/MLSCIC from synthetic and
natural images. For this purpose, we used synthetic SCIs from [10] and [34],
and natural images from [9]. The dataset is constructed as follows. First, a
pair of synthetic and natural images is randomly chosen. Then, a random patch
(with randomly chosen size, such that width and height are between 128 and 192
pixels) is extracted from the natural image and inserted into a randomly
chosen location in the synthetic image. At the same time, the size and
location of the natural patch within the synthetic image are used to construct
the binary S/N segmentation map, as shown in Fig. 3. This way, a dataset of
3,100 images and their corresponding S/N segmentation masks is constructed,
both with size 512 $\times$ 512\. Of these, 100 randomly chosen images are
used for testing and the remaining 3,000 images are used for training.

##  IV Experiments

###  IV-A Setup

We construct the multi-task codec using two learning-based image compression
pipelines – [13] and [15] – referred to as “bmshj2018” and “ms2020”,
respectively. In each case, another synthesis transform block, architecturally
the exact copy of the existing synthesis block, is added to the decoder to
provide the S/N segmentation output. The only difference is that the output
layer of the added synthesis block is changed to produce a 1-channel (binary)
image as the segmentation map, instead of the 3-channel RGB image.

Both default pipelines are trained on the CLIC dataset [9] for 1,000 epochs
using the Adam optimizer [35] with an initial learning rate of $10^{-4}$, to
provide a baseline trained on natural images. These are called X-1-Decoder-
Natural, where X $\in$ {“bmshj2018”, “ms2020”}. Further, the same pipelines
are trained in the same way on synthetic images from [10] and [34] to provide
codecs called X-1-Decoder-Synthetic, and on our synthetic/natural dataset for
300 epochs to provide codecs called X-1-Decoder-S/N. Finally, our multi-task
codecs are trained on our dataset for 300 epochs, and are called X-2-Decoder.
The following $(\lambda,\varphi)$ pairs were used in (4) to train five multi-
task codecs: $(\lambda,\varphi)\,\in\,\\{(1,10^{-5}),\,(10^{-1},10^{-5}),\\\
(10^{-2},10^{-6}),(10^{-3},10^{-7}),(10^{-4},10^{-7})\\}$. The single-task
codecs used only the $\lambda$ values from this list, without the last term in
the loss function (4). Two image quality metrics are used in the experiments:
Peak Signal to Noise Ratio (PSNR), and Gradient Magnitude Similarity Deviation
(GMSD) [36], which is considered suitable for SCIs.

TABLE I: BD-Rates (%) for PSNR and GMSD metrics relative to bmshj2018-1-Decoder-Natural for bmshj2018-based codecs and ms2020-1-Decoder-Natural for ms2020-based codecs Codec | PSNR | GMSD
---|---|---
bmshj2018-1-Decoder-Synthetic | –20.45 | –20.83
bmshj2018-1-Decoder-S/N | –25.87 | –31.08
bmshj2018-2-Decoder | –36.14 | –34.48
ms2020-1-Decoder-Synthetic | –17.39 | 0.15
ms2020-1-Decoder-S/N | –39.97 | –43.68
ms2020-2-Decoder | –61.57 | –70.95

###  IV-B Quantifying the domain shift

Table I quantifies various domain shifts for the two coding pipelines in terms
of the BR-Rate [37], for both PSNR and GMSD. For each pipeline, X-1-Decoder-
Natural, where X $\in$ {“bmshj2018”, “ms2020”}, is used as the anchor, and
best results for each pipeline are indicated in bold. The results show that
merely retraining a coding model on synthetic images will reduce the BD-Rate
by 17-20% when the quality metric is PSNR. This provides additional support to
the results from [32], which showed similar trend. Moreover, our results show
that retraining the pipeline on combined synthetic/natural images provides
20-40% BD-Rate gain for PSNR. Finally, the multi-task system provides 36-62%
BD-Rate gain for PSNR.

The comparison between X-1-Decoder-S/N and X-2-Decoder is particularly
interesting. Both these types of codecs are trained our synthetic/natural
dataset, and the only difference is that X-2-Decoder is performing two tasks
instead of one. Hence, this comparison shows the effect of multi-tasking. We
see from Table I that adding the second task (S/N segmentation) improves BD-
Rate by 10-20% for PSNR, and 3-35% for GMSD, depending on the pipeline X. We
hypothesize that this is because the encoder in a X-2-Decoder system is more
aware of where natural vs. synthetic parts of the input are, so it is able to
adjust its entropy model parameters more quickly to the different statistics
of these regions.

###  IV-C Screen content image coding performance

Next we test the SCI image coding performance on our test set. Since the
“ms2020” pipeline showed better performance in the experiments described
above, here we use ms2020-1-Decoder-Natural as the anchor, and examine the
coding performance of other versions of this pipeline. Further, we also test
HEVC Intra (HM 16.26rc1), HEVC-SCC Intra, and VVC Intra (VTM 18.2) with screen
content coding options turned on. Table II shows the BD-Rate relative to the
ms2020-1-Decoder-Natural anchor. As seen in the table, among the conventional
codecs, VVC is the best with about 46% BD rate gain over the anchor for both
PSNR and GMSD. Both HEVC-SCC and ms2020-1-Decoder-S/N are close behind it for
GMSD, with a slightly larger gap for PSNR. However, our ms2020-2-Decoder
provides the best performance, with over 60% (resp. 70%) BD-Rate gain for PSNR
(resp. GMSD).

TABLE II: BD-Rates (%) for PSNR and GMSD metrics relative to ms2020-1-Decoder-Natural Codec | PSNR | GMSD
---|---|---
HEVC (HM 16.26rc1) | –10.37 | –28.31
HEVC-SCC | –37.61 | –43.92
VVC (VTM 18.2) | –46.31 | –46.02
ms2020-1-Decoder-Synthetic | –17.39 | 0.15
ms2020-1-Decoder-S/N | –39.97 | –43.68
ms2020-2-Decoder | –61.57 | –70.95

Figure 4: Visual example of SCI coding. Top-left: HEVC (0.0604 bpp, PSNR =
25.81 dB, GMSD = 0.1593). Top-right: HEVC-SCC (0.0581 bpp, PSNR = 27.84 dB,
GMSD = 0.1344). Bottom-left: VVC (0.0569 bpp, PSNR = 28.13 dB, GMSD = 0.1192).
Bottom-right: proposed ms2020-2-Decoder (0.0380 bpp, PSNR = 29.12 dB, GMSD =
0.1019). With GMSD, the lower the better.

Fig. 4 shows an example of a SCI coded with HEVC, HEVC-SCC, VVC, and the
proposed codec. From HEVC (shown in the top-right) to the proposed (bottom-
left), the bitrate decreases but quality increases. All codecs handle large
text reasonably well. However, details of both the small text and the natural
part of the SCI are preserved much better with the proposed codec compared to
the other codecs.

In order to further explain the effectiveness of the proposed multi-task
approach, we measure its coding performance on the natural and synthetic parts
of the test SCI images. Specifically, we measure the bit rate and PSNR
separately on these two portions of the input image, and then compute BD-Rate
with respect to the anchor ms2020-1-Decoder-Natural. The results are shown in
Table III. Recall that ms2020-1-Decoder-Synthetic is trained on the pure
synthetic training set, and ms2020-2-Decoder is trained on our
synthetic/natural training set. The results show that ms2020-1-Decoder-
Synthetic has lost about 118% BD-Rate on the natural portion compared to the
anchor, but has gained slightly (about 17% BD-Rate) on the synthetic portion
compared to the anchor. Hence, improved coding of synthetic portion has been
paid for by the lower efficiency on the natural portion of the SCI. However,
the proposed multi-task system loses only about 26% BD-Rate compared to the
anchor on the natural portion while gaining about 69% on the synthetic
portion. Hence, in the case of the multi-task codec, the large coding gain on
the synthetic portion is paid for by a much smaller loss on the natural
portion, because the codec is better able to recognize the two regions and
adjust its coding accordingly.

Although segmentation is not the main goal of the proposed codec, it is
interesting to see how the system performs on this task as well. Fig. 5 shows
a SCI that was not in our dataset, and has multiple circular natural inserts.
Our multi-task system is still able to identify these natural portions
reasonably well.

TABLE III: BD-Rates (%) for PSNR in natural and synthetic parts of SCI relative to ms2020-1-Decoder-Natural Codec | Natural | Synthetic
---|---|---
ms2020-1-Decoder-Synthetic | 117.63 | –16.66
ms2020-2-Decoder | 25.61 | –68.67

Figure 5: Example of segmentation for a SCI with multiple circular natural
components.

##  V Conclusion

We presented a learning-based multi-task codec for screen content images
(SCI). Besides coding and reconstructing the input image, the multi-task codec
was trained to perform an additional task: segment synthetic vs. natural
portions of the input image. This helps it adjust its coding to the two
regions, which have different statistics. The performance of the proposed
codec was compared against relevant benchmarks, both handcrafted and learning-
based, and its superior SCI coding performance was demonstrated through
extensive experiments.

## References

  * [1] Gregory K Wallace,  “The JPEG still picture compression standard,”  Communications of the ACM, vol. 34, no. 4, pp. 30–44, 1991.
  * [2] Majid Rabbani and Rajan Joshi,  “An overview of the JPEG 2000 still image compression standard,”  Signal Processing: Image Communication, vol. 17, no. 1, pp. 3–48, 2002.
  * [3] Jizheng Xu, Rajan Joshi, and Robert A. Cohen,  “Overview of the emerging hevc screen content coding extension,”  IEEE Transactions on Circuits and Systems for Video Technology, vol. 26, no. 1, pp. 50–62, 2016.
  * [4] Wen-Hsiao Peng, Frederick G Walls, Robert A Cohen, Jizheng Xu, Jörn Ostermann, Alexander MacInnis, and Tao Lin,  “Overview of screen content video coding: Technologies, standards, and beyond,”  IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 6, no. 4, pp. 393–408, 2016.
  * [5] Benjamin Bross, Ye-Kui Wang, Yan Ye, Shan Liu, Jianle Chen, Gary J Sullivan, and Jens-Rainer Ohm,  “Overview of the versatile video coding (VVC) standard and its applications,”  IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3736–3764, 2021.
  * [6] Xiaozhong Xu and Shan Liu,  “Overview of screen content coding in recently developed video coding standards,”  IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 2, pp. 839–852, 2021.
  * [7] Tung Nguyen, Xiaozhong Xu, Felix Henry, Ru-Ling Liao, Mohammed Golam Sarwer, Marta Karczewicz, Yung-Hsuan Chao, Jizheng Xu, Shan Liu, Detlev Marpe, and Gary J. Sullivan,  “Overview of the screen content support in VVC: Applications, coding tools, and performance,”  IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3801–3817, 2021.
  * [8] Pooryaa Cheraaqee, Zahra Maviz, Azadeh Mansouri, and Ahmad Mahmoudi-Aznaveh,  “Quality assessment of screen content images in wavelet domain,”  IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 2, pp. 566–578, 2022.
  * [9] George Toderici, Wenzhe Shi, Radu Timofte, Lucas Theis, Johannes Balle, Eirikur Agustsson, Nick Johnston, and Fabian Mentzer,  “Workshop and challenge on learned image compression (clic2020),” 2020\.
  * [10] Zhangkai Ni, Lin Ma, Huanqiang Zeng, Jing Chen, Canhui Cai, and Kai-Kuang Ma,  “Esim: Edge similarity for screen content image quality assessment,”  IEEE Transactions on Image Processing, vol. 26, no. 10, pp. 4818–4831, 2017.
  * [11] Johannes Ballé, Valero Laparra, and Eero P Simoncelli,  “End-to-end optimization of nonlinear transform codes for perceptual quality,”  in 2016 Picture Coding Symposium (PCS). IEEE, 2016, pp. 1–5.
  * [12] Johannes Ballé, Valero Laparra, and Eero P Simoncelli,  “End-to-end optimized image compression,”  in ICLR, 2017.
  * [13] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston,  “Variational image compression with a scale hyperprior,”  in ICLR, 2018.
  * [14] Johannes Ballé, David Minnen, and George Toderici,  “Joint autoregressive and hierarchical priors for learned image compression,”  in NeurIPS, 2018, vol. 31, pp. 10794–10803.
  * [15] David Minnen and Saurabh Singh,  “Channel-wise autoregressive entropy models for learned image compression,”  in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 3339–3343.
  * [16] Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto,  “Learned image compression with discretized gaussian mixture likelihoods and attention modules,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7939–7948.
  * [17] Yueyu Hu, Wenhan Yang, Zhan Ma, and Jiaying Liu,  “Learning end-to-end lossy image compression: A benchmark,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 8, pp. 4194–4211, 2022.
  * [18] Wei Pu, Marta Karczewicz, Rajan Joshi, Vadim Seregin, Feng Zou, Joel Sole, Yu-Chen Sun, Tzu-Der Chuang, Polin Lai, Shan Liu, et al.,  “Palette mode coding in HEVC screen content coding extension,”  IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 6, no. 4, pp. 420–432, 2016.
  * [19] Je-Won Kang, Soo-Kyung Ryu, Na-Young Kim, and Min-Joo Kang,  “Efficient residual dpcm using an $l\\_1$ robust linear prediction in screen content video coding,”  IEEE Transactions on Multimedia, vol. 18, no. 10, pp. 2054–2065, 2016.
  * [20] Victor Sanchez, Francesc Auli-Llinas, and Joan Serra-Sagrista,  “Dpcm-based edge prediction for lossless screen content coding in hevc,”  IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 6, no. 4, pp. 497–507, 2016.
  * [21] Xiulian Peng and Jizheng Xu,  “Hash-based line-by-line template matching for lossless screen image coding,”  IEEE Transactions on Image Processing, vol. 25, no. 12, pp. 5601–5609, 2016.
  * [22] Iulia Mitrica, Eric Mercier, Christophe Ruellan, Attilio Fiandrotti, Marco Cagnazzo, and Béatrice Pesquet-Popescu,  “Very low bitrate semantic compression of airplane cockpit screen content,”  IEEE Transactions on Multimedia, vol. 21, no. 9, pp. 2157–2170, 2019.
  * [23] Shiqi Wang, Xinfeng Zhang, Xianming Liu, Jian Zhang, Siwei Ma, and Wen Gao,  “Utility-driven adaptive preprocessing for screen content video compression,”  IEEE Transactions on Multimedia, vol. 19, no. 3, pp. 660–667, 2016\.
  * [24] Huan Yang, Yuming Fang, and Weisi Lin,  “Perceptual quality assessment of screen content images,”  IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 4408–4421, 2015.
  * [25] Ke Gu, Shiqi Wang, Huan Yang, Weisi Lin, Guangtao Zhai, Xiaokang Yang, and Wenjun Zhang,  “Saliency-guided quality assessment of screen content images,”  IEEE Transactions on Multimedia, vol. 18, no. 6, pp. 1098–1110, 2016.
  * [26] George Toderici, Sean M O’Malley, Sung Jin Hwang, Damien Vincent, David Minnen, Shumeet Baluja, Michele Covell, and Rahul Sukthankar,  “Variable rate image compression with recurrent neural networks,”  in ICLR, 2016.
  * [27] David Minnen, George Toderici, Michele Covell, Troy Chinen, Nick Johnston, Joel Shor, Sung Jin Hwang, Damien Vincent, and Saurabh Singh,  “Spatially adaptive image compression using a tiled deep network,”  in 2017 IEEE International Conference on Image Processing (ICIP), 2017, pp. 2796–2800.
  * [28] Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang, Joel Shor, and George Toderici,  “Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks,”  in Proc. IEEE/CVF CVPR, 2018, pp. 4385–4393.
  * [29] Johannes Ballé, Valero Laparra, and Eero P Simoncelli,  “Density modeling of images using a generalized normalization transformation,”  in ICLR, 2016.
  * [30] Yung-Han Ho, Chih-Chun Chan, Wen-Hsiao Peng, Hsueh-Ming Hang, and Marek Domański,  “Anfic: Image compression using augmented normalizing flows,”  IEEE Open Journal of Circuits and Systems, vol. 2, pp. 613–626, 2021.
  * [31] Binglin Li, Jie Liang, and Jingning Han,  “Variable-rate deep image compression with vision transformers,”  IEEE Access, vol. 10, pp. 50323–50334, 2022.
  * [32] Meng Wang, Kai Zhang, Li Zhang, Yaojun Wu, Yue Li, Junru Li, and Shiqi Wang,  “Transform skip inspired end-to-end compression for screen content image,”  in Proc. IEEE ICIP, 2022, pp. 3848–3852.
  * [33] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli,  “Image quality assessment: from error visibility to structural similarity,”  IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.
  * [34] Shan Cheng, Huanqiang Zeng, Jing Chen, Junhui Hou, Jianqing Zhu, and Kai-Kuang Ma,  “Screen content video quality assessment: Subjective and objective study,”  IEEE Transactions on Image Processing, vol. 29, pp. 8636–8651, 2020\.
  * [35] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [36] Wufeng Xue, Lei Zhang, Xuanqin Mou, and Alan C Bovik,  “Gradient magnitude similarity deviation: A highly efficient perceptual image quality index,”  IEEE Transactions on Image Processing, vol. 23, no. 2, pp. 684–695, 2013.
  * [37] Gisle Bjontegaard,  “Calculation of average PSNR differences between RD-curves,”  VCEG-M33, 2001.