# Results on direct $C\\!P$ violation in $B$ decays in LHCb

T.M. Karbach aaaon behalf of the LHCb collaboration

I present three studies from the LHCb experiment on the subject of direct
$C\\!P$ violation in $B^{0}$ and $B^{0}_{s}$ decays. First, we measure the
$C\\!P$ asymmetry in $B^{\pm}\rightarrow\psi K^{\pm}$ decays, with
$\psi={J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu},\psi(2S)$, using
$0.35\mbox{\,fb}^{-1}$ of $pp$ collisions at
$\sqrt{s}=7\mathrm{\,Te\kern-0.80005ptV}$. We find no evidence for $C\\!P$
violation. Second, using the same data sample, we see the first evidence of
$C\\!P$ violation in the decays of $B^{0}_{s}$ mesons to $K^{\pm}\pi^{\mp}$
pairs, $A_{C\\!P}(B^{0}_{s}\rightarrow K\pi)=0.27\pm 0.08\mathrm{(stat)}\pm
0.02\mathrm{(syst)}$ ($3.3\sigma$). Third, using $1.0\mbox{\,fb}^{-1}$ of
data, measurements of $C\\!P$ sensitive observables of the $B^{\pm}\rightarrow
DK^{\pm}$ system are presented. They include the first observation of the
suppressed mode $B^{\pm}\rightarrow[\pi^{\pm}K^{\mp}]_{D}K^{\pm}$. Combining
several $D$ final states, $C\\!P$ violation in $B^{\pm}\rightarrow DK^{\pm}$
decays is observed with a significance of $5.8\sigma$.

##  1 Measurement of $C\\!P$ asymmetries in $B^{\pm}\rightarrow\psi h^{\pm}$
decays

The $B^{\pm}\rightarrow\psi h^{\pm}$ decays, with
$\psi=({J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu},\psi(2S))$ and $h=K,\pi$,
receive contributions from both tree and penguin diagrams. If these
contributions have different weak phases, direct $C\\!P$ violation may occur.
The Standard Model predicts that for $b\rightarrow c\bar{c}s$ decays the tree
and penguin contributions have the same weak phase and thus no direct $C\\!P$
violation is expected in $B^{\pm}\rightarrow\psi K^{\pm}$. For $b\rightarrow
c\bar{c}d$ transitions, however, both contributions have different weak
phases, and $C\\!P$ violation in $B^{\pm}\rightarrow\psi\pi^{\pm}$ decays may
occur. Their branching fractions are expected to be about 5% of the favoured
$B^{\pm}\rightarrow\psi K^{\pm}$ modes. In our paper $\\!{}^{{\bf?}}$ we
analyse a data sample of $0.35\mbox{\,fb}^{-1}$ of $pp$ collisions at
$\sqrt{s}=7\mathrm{\,Te\kern-1.00006ptV}$, taken in 2011 with the LHCb
detector. We define the $C\\!P$ asymmetry and the charge-averaged ratio of
branching ratios as

$A^{\psi\pi}=\frac{{\cal B}(B^{-}\rightarrow\psi\pi^{-})-{\cal
B}(B^{+}\rightarrow\psi\pi^{+})}{{\cal B}(B^{-}\rightarrow\psi\pi^{-})+{\cal
B}(B^{+}\rightarrow\psi\pi^{+})}~{},\quad R^{\psi}=\frac{{\cal
B}(B^{\pm}\rightarrow\psi\pi^{\pm})}{{\cal B}(B^{\pm}\rightarrow\psi
K^{\pm})}~{}.$ (1)

The $\psi$ resonance is reconstructed in the $\mu^{+}\mu^{-}$ final state, and
the well known and abundant decay
$B^{\pm}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}K^{\pm}$ is
used as a control channel. It is crucial to control its cross feed into the
$B^{+}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}\pi^{+}$ channel.
Here we benefit from LHCb’s two ring imaging Cherenkov (RICH) detectors that
provide strong $K/\pi$ separation. We obtain the signal yields from a
simultaneous fit to the $B$ candidate invariant mass distribution in eight
independent subsamples, defined by the charge ($\times 2$), the $\psi$ state
($\times 2$) and the flavour of the bachelor hadron ($K,\pi$, $\times 2$). The
fit projections for the $\psi(2S)$ subsamples are shown in Figure 1. The
measured ratios of branching fractions are
$R^{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}}=(3.83\pm 0.11\pm 0.07)\times
10^{-2}$ and $R^{\psi(2S)}=(3.95\pm 0.40\pm 0.12)\times 10^{-2}$, where the
first uncertainty is statistical and the second systematic. $R^{\psi(2S)}$ is
compatible with the one existing measurement $\\!{}^{{\bf?}}$, $(3.99\pm
0.36\pm 0.17)\times 10^{-2}$. The measurement of
$R^{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}}$ is $3.2\sigma$ lower than
the current world average $\\!{}^{{\bf?}}$, $(5.2\pm 0.4)\times 10^{-2}$.
Using the established measurements of the Cabibbo-favoured branching fractions
$\\!{}^{{\bf?}}$, we deduce ${\cal
B}(B^{\pm}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip
2.0mu}\pi^{\pm})=(3.88\pm 0.11\pm 0.15)\times 10^{-5}$, ${\cal
B}(B^{\pm}\rightarrow\psi(2S)\pi^{\pm})=(2.52\pm 0.26\pm 0.15)\times 10^{-5}$.
The measured $C\\!P$ asymmetries,

$\displaystyle A^{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}\pi}_{C\\!P}$
$\displaystyle=0.005\pm 0.027\pm 0.011~{},$ (2) $\displaystyle
A^{\psi(2S)\pi}_{C\\!P}$ $\displaystyle=0.048\pm 0.090\pm 0.011~{},$ (3)
$\displaystyle A^{\psi(2S)K}_{C\\!P}$ $\displaystyle=0.024\pm 0.014\pm
0.008~{},$ (4)

have comparable or better precision than previous results, and no evidence of
direct $C\\!P$ violation is seen.

Figure 1: $B^{\pm}\rightarrow\Psi(2S)h^{\pm}$ invariant mass distributions,
overlaid by the total fitted PDF (thin line). Pion-like events are
reconstructed as ${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}\pi^{\pm}$ and
enter in the top plots. All other events are reconstructed as
${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}K^{\pm}$ enter the bottom plots
(shown in logarithmic scale). $B^{-}$ decays are shown on the left, $B^{+}$ on
the right. The dark [red] curve shows the
$B^{\pm}\rightarrow\psi(2S)\pi^{\pm}$ component, the light [green] curve
represents $B^{\pm}\rightarrow\psi(2S)K^{\pm}$. Partially reconstructed
backgrounds are shaded.

##  2 Direct $C\\!P$ violation in $B^{0}(B^{0}_{s})\rightarrow K^{-}\pi^{+}$
decays

$C\\!P$ violation is well established in the $K^{0}$ and $B^{0}$ meson
systems. Recent results from LHCb have also provided evidence for $C\\!P$
violation in the $D^{0}$ system $\\!{}^{{\bf?}}$. In our paper
$\\!{}^{{\bf?}}$ we report evidence of direct $C\\!P$ violation in the last
neutral meson system, the $B^{0}_{s}$ system. We reconstruct both
$B^{0}\rightarrow K^{+}\pi^{-}$ and $B^{0}_{s}\rightarrow K^{-}\pi^{+}$ decays
in $0.35~{}\mathrm{fb}^{-1}$ of data collected with the LHCb detector in 2011.
The considered decays have contributions from both tree and penguin diagrams,
and are sensitive to contribution of new physics in the penguins. The $C\\!P$
asymmetry in the $B^{0}\rightarrow K^{+}\pi^{-}$ is well established
$\\!{}^{{\bf?}}$. The probability or a $b$ quark to decay as
$B^{0}_{s}\rightarrow K\pi$ is about 14 times smaller than that to decay as
$B^{0}\rightarrow K\pi$. However, both tree and penguin diagrams are roughly
of the same magnitude, so $C\\!P$ violation effects can potentially be large.
We define the $C\\!P$ asymmetries as

$A_{C\\!P}(B^{0}_{(s)})=\frac{\Gamma(\overline{B}^{0}_{(s)}\rightarrow\bar{f}_{(s)})-\Gamma(B^{0}_{(s)}\rightarrow
f_{(s)})}{\Gamma(\overline{B}^{0}_{(s)}\rightarrow\bar{f}_{(s)}+\Gamma(B^{0}_{(s)}\rightarrow
f_{(s)})}~{},$ (5)

with $f=K^{+}\pi^{-}$ and $f_{s}=K^{-}\pi^{+}$. To distinguish the
$K^{+}\pi^{-}$ and $K^{-}\pi^{+}$ final states we rely on the RICH particle
identification system. We carefully control the efficiencies and
misidentification rates from data, through large control samples of
$D^{*}\rightarrow D\pi\rightarrow(K\pi)_{D}\pi$ and $\Lambda_{b}\rightarrow
p\pi$ decays. There are cross feeds from $B^{0}\rightarrow\pi^{+}\pi^{-}$ and
$B^{0}_{s}\rightarrow K^{+}K^{-}$ decays, whose line shape we predict from
simulation. We compute a raw asymmetry from the yields of a fit to the
invariant mass distribution in the positive charge and negative charge
subsamples. Figure 2 shows the projections. This raw asymmetry needs to be
corrected for two effects: an inherent detector charge asymmetry (which we
estimate from our $D^{*}$ control samples) and a non-zero production asymmetry
that is further diluted by $B$ mixing (thus it mostly affects the $B^{0}$
channel due to its much slower $B^{0}$–$\kern
1.79993pt\overline{\kern-1.79993ptB}{}^{0}$ oscillation). The total
corrections to the raw asymmetry are $\Delta A_{C\\!P}(B^{0})=-0.007\pm 0.006$
and $\Delta A_{C\\!P}(B^{0}_{s})=0.010\pm 0.002$, where the errors are
statistical. The systematic uncertainty of $A_{C\\!P}(B^{0})$ is dominated by
uncertainties due to instrumentation and production asymmetry, while the
systematic uncertainty of $A_{C\\!P}(B^{0}_{s})$ receives a leading
contribution from the combinatorial background description. In conclusion we
obtain the following measurements of the $C\\!P$ asymmetries:

$A_{C\\!P}(B^{0}\rightarrow K\pi)=-0.088\pm 0.011\,\mathrm{(stat)}\pm
0.008\,\mathrm{(syst)}$

and

$A_{C\\!P}(B^{0}_{s}\rightarrow K\pi)=0.27\pm 0.08\,\mathrm{(stat)}\pm
0.02\,\mathrm{(syst)}.$

The result for $A_{C\\!P}(B^{0}\rightarrow K\pi)$ constitutes the most precise
measurement available to date. It is in good agreement with the current world
average $\\!{}^{{\bf?}}$. The significance of the measured deviation from zero
exceeds $6\sigma$. The result for $A_{C\\!P}(B^{0}_{s}\rightarrow K\pi)$ is in
agreement with the only measurement previously available $\\!{}^{{\bf?}}$. The
significance computed for $A_{C\\!P}(B^{0}_{s}\rightarrow K\pi)$ is
3.3$\sigma$, making this the first evidence for $C\\!P$ violation in the
decays of $B^{0}_{s}$ mesons.

Figure 2: $K\pi$ invariant mass spectra obtained using the event selection
adopted for the best sensitivity on (a, b) $A_{C\\!P}(B^{0}\rightarrow K\pi)$
and (c, d) $A_{C\\!P}(B^{0}_{s}\rightarrow K\pi)$. Plots (a) and (c) represent
the $K^{+}\pi^{-}$ invariant mass whereas plots (b) and (d) represent the
$K^{-}\pi^{+}$ invariant mass. The results of the unbinned maximum likelihood
fits are overlaid. The main components contributing to the fit model are also
shown.

##  3 Observation of $C\\!P$ violation in $B^{\pm}\rightarrow DK^{\pm}$

The CKM angle $\gamma=\arg\left(-V_{ud}V_{ub}^{*}/V_{cd}V_{cb}^{*}\right)$ is
the least well known angle of the corresponding unitarity triangle of the CKM
matrix $V$. The angle $\gamma$ can be measured in $B^{\pm}\rightarrow
DK^{\pm}$ decays where the $D$ signifies a $D^{0}$ or $\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}$ meson. The amplitude for the
$D^{0}$ contribution is proportional to $V_{cb}$ whilst the $\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}$ amplitude depends on $V_{ub}$. If
the $D$ final state is accessible for both $D^{0}$ and $\kern
1.99997pt\overline{\kern-1.99997ptD}{}^{0}$ mesons, the two amplitudes
interfere and give rise to observables that are sensitive to $\gamma$. Many
different $D$ final states can be used. In our analysis $\\!{}^{{\bf?}}$ of
1.0 $\mbox{\,fb}^{-1}$ of $\sqrt{s}=7~{}\mathrm{\,Te\kern-1.00006ptV}$ data
collected by LHCb in 2011, we use the $C\\!P$ eigenstates $D\rightarrow
K^{+}K^{-}$, $\pi^{+}\pi^{-}$ (often referred to as “GLW” modes
$\\!{}^{{\bf?},{\bf?}}$), and the flavour eigenstate
$D\rightarrow\pi^{-}K^{+}$ (labelled “ADS” mode $\\!{}^{{\bf?},{\bf?}}$). The
latter requires the favoured, $b\rightarrow c$ decay to be followed by a
doubly Cabibbo-suppressed $D$ decay, and the suppressed $b\rightarrow u$ decay
to be followed by a favoured $D$ decay. As a consequence, the interfering
amplitudes are of similar magnitude and hence large interference can occur. In
total, 13 observables are measured: three ratios of partial widths

$R_{K/\pi}^{f}=\frac{\Gamma(B^{-}\rightarrow[f]_{D}K^{-})+\Gamma(B^{+}\rightarrow[f]_{D}K^{+})}{\Gamma(B^{-}\rightarrow[f]_{D}\pi^{-})+\Gamma(B^{+}\rightarrow[f]_{D}\pi^{+})}~{},$
(6)

where $f$ represents $KK$, $\pi\pi$ and the favoured $K\pi$ mode, six $C\\!P$
asymmetries

$A_{h}^{f}=\frac{\Gamma(B^{-}\rightarrow[f]_{D}h^{-})-\Gamma(B^{+}\rightarrow[f]_{D}h^{+})}{\Gamma(B^{-}\rightarrow[f]_{D}h^{-})+\Gamma(B^{+}\rightarrow[f]_{D}h^{+})}~{},$
(7)

and four charge-separated partial widths of the $ADS$ mode relative to the
favoured mode

$R_{h}^{\pm}=\frac{\Gamma(B^{\pm}\rightarrow[\pi^{\pm}K^{\mp}]_{D}h^{\pm})}{\Gamma(B^{\pm}\rightarrow[K^{\pm}\pi^{\mp}]_{D}h^{\pm})}~{}.$
(8)

Similar analyses have found evidence of the
$B^{\pm}\rightarrow[\pi^{\pm}K^{\mp}]_{D}K^{\pm}$ decay
$\\!{}^{{\bf?},{\bf?},{\bf?}}$. The abundant $B^{-}\rightarrow D\pi^{-}$
decays have limited sensitivity to $\gamma$ and provide a large control sample
from which probability density functions are shaped. The analysis method
benefits greatly from a boosted decision tree, which combines 20 kinematic
variables to effectively suppress combinatorial backgrounds. Charmless
backgrounds are suppressed by exploiting the large forward boost of the $D$
meson through a cut on its flight distance. The signal yields are estimated by
a simultaneous fit to 16 independent subsamples, defined by the charges
($\times 2$), the $D$ final states ($\times 4$), and the $K$ or $\pi$ nature
of the bachelor hadron ($\times 2$). Figures 3 and 4 show the projections of
the $\pi^{+}\pi^{-}$ and suppressed $\pi^{\pm}K^{\mp}$ subsamples,
respectively. It is crucial to control the cross feed of the abundant
$B^{-}\rightarrow D\pi^{-}$ decays into the signal decays. For this we rely on
the two RICH detectors, which allow to place particle identification cuts on
the bachelor hadron. These cuts are $87.6\%$ efficient for kaons at a rate of
$3.8\%$ misidentified pions. Many systematic uncertainties cancel in the
ratios Eqns. 6-8. The remaining systematic uncertainties are dominated by an
intrinsic charge asymmetry of the detector, and by the uncertainty on the
particle identification. From the measured 13 observables the following
established quantities can be deduced (the full set is contained in our
paper$\\!{}^{{\bf?}}$):

$\displaystyle R_{C\\!P+}$ $\displaystyle=\phantom{-}1.007\pm 0.038\pm
0.012~{},$ $\displaystyle A_{C\\!P+}$ $\displaystyle=\phantom{-}0.145\pm
0.032\pm 0.010~{},$ $\displaystyle R_{K}^{-}$
$\displaystyle=\phantom{-}0.0073\pm 0.0023\pm 0.0004~{},$ $\displaystyle
R_{K}^{+}$ $\displaystyle=\phantom{-}0.0232\pm 0.0034\pm 0.0007~{},$

where the first error is statistical and the second systematic; $R_{C\\!P+}$
is computed from $R_{C\\!P+}\approx\langle
R_{K/\pi}^{KK},R_{K/\pi}^{\pi\pi}\rangle/R_{K/\pi}^{K\pi}$ with an additional
$1\%$ systematic uncertainty assigned to account for the approximation;
$A_{C\\!P+}$ is computed as $A_{C\\!P+}=\langle
A_{K}^{KK},A_{K}^{\pi\pi}\rangle$. From the $R_{K}^{\pm}$ we also compute

$\displaystyle R_{ADS(K)}$ $\displaystyle=\phantom{-}0.0152\pm 0.0020\pm
0.0004~{},$ $\displaystyle A_{ADS(K)}$ $\displaystyle=-0.52\pm 0.15\pm
0.02~{},$

as $R_{ADS(K)}=(R_{K}^{-}+R_{K}^{+})/2$ and
$A_{ADS(K)}=(R_{K}^{-}-R_{K}^{+})/(R_{K}^{-}+R_{K}^{+})$.

To summarise, the $B^{\pm}\rightarrow DK^{\pm}$ $ADS$ mode is observed with
$\approx 10\sigma$ statistical significance when comparing the maximum
likelihood to that of the null hypothesis. This mode displays evidence
($4.0\sigma$) of a large negative asymmetry, consistent with previous
experiments $\\!{}^{{\bf?},{\bf?},{\bf?}}$. The combined asymmetry
$A_{C\\!P+}$ is smaller than (but compatible with) previous measurements
$\\!{}^{{\bf?},{\bf?}}$. It is $4.5\sigma$ significant. We compare the maximum
likelihood with that under the null-hypothesis in all three $D$ final states
where the bachelor is a kaon, diluted by the non-negligible correlated
systematic uncertainties. From this we observe, with a total significance of
$5.8\sigma$, direct $C\\!P$ violation in $B^{\pm}\rightarrow DK^{\pm}$ decays.

Figure 3: The invariant mass distribution of selected
$B^{\pm}\rightarrow[\pi^{+}\pi^{-}]_{D}h^{\pm}$ candidates. The left plots are
$B^{-}$ candidates, $B^{+}$ are on the right. In the top plots, the bachelor
track passes the kaon RICH cut and the $B$ candidates are reconstructed
assigning this track the kaon mass. The remaining events are placed in the
bottom row and are reconstructed with a pion mass hypothesis. The dark (red)
curve represents the $B\rightarrow DK^{\pm}$ events, the light (green) curve
is $B\rightarrow D\pi^{\pm}$. The shaded contribution are partially
reconstructed events and the thin line shows the total PDF which also includes
a linear combinatoric component.  Figure 4: The invariant mass distribution of
selected $B^{\pm}\rightarrow[\pi^{\pm}K^{\pm}]_{D}h^{\pm}$ candidates. See the
caption of Fig. 3 for a full description. The broken line here represents the
partially reconstructed, but Cabibbo favoured, $B^{0}_{s}\rightarrow
D^{0}K^{+}\pi^{-}$ decays where the pion is lost.

## References

## References

  * [1] LHCb collaboration, R. Aaij et al., Measurements of the branching fractions and CP asymmetries of $B^{+}\rightarrow J/\psi\pi^{+}$ and $B^{+}\rightarrow\psi(2S)\pi^{+}$ decays, arXiv:1203.3592
  * [2] Belle Collaboration, V. Bhardwaj et al., Observation of $B^{\pm}\rightarrow\psi(2S)\pi^{\pm}$ and search for direct CP-violation, Phys. Rev. D78 (2008) 051104, arXiv:0807.2170
  * [3] Particle Data Group, K. Nakamura et al., Review of particle physics, J. Phys. G G37 (2010) 075021
  * [4] LHCb Collaboration, R. Aaij et al., Evidence for CP violation in time-integrated $D^{0}\rightarrow h^{-}h^{+}$ decay rates, Phys. Rev. Lett. 108 (2012) 111602, arXiv:1112.0938
  * [5] LHCb collaboration, R. Aaij et al., First evidence of direct CP violation in charmless two-body decays of $B_{s}$ mesons, arXiv:1202.6251
  * [6] Heavy Flavor Averaging Group, D. Asner et al., Averages of b-hadron, c-hadron, and tau-lepton properties, arXiv:1010.1589, Updates available online at http://www.slac.stanford.edu/xorg/hfag
  * [7] CDF collaboration, T. Aaltonen et al., Measurements of direct CP violating asymmetries in charmless decays of strange bottom mesons and bottom baryons, Phys. Rev. Lett. 106 (2011) 181802, arXiv:1103.5762
  * [8] LHCb collaboration, R. Aaij et al., Observation of CP violation in $B^{+}\rightarrow DK^{+}$ decays, arXiv:1203.3662
  * [9] M. Gronau and D. London, How to determine all the angles of the unitarity triangle from $B_{d}^{0}\rightarrow DK^{0}_{\rm\scriptscriptstyle S}$ and $B_{s}^{0}\rightarrow D\phi$, Phys. Lett. B253 (1991) 483
  * [10] M. Gronau and D. Wyler, On determining a weak phase from $C\\!P$ asymmetries in charged $B$ decays, Phys. Lett. B265 (1991) 172
  * [11] D. Atwood, I. Dunietz, and A. Soni, Enhanced CP violation with $B\rightarrow KD^{0}(\kern 1.99997pt\overline{\kern-1.99997ptD}{}^{0})$ modes and extraction of the CKM angle $\gamma$, Phys. Rev. Lett. 78 (1997) 3257, arXiv:hep-ph/9612433
  * [12] D. Atwood, I. Dunietz, and A. Soni, Improved methods for observing CP violation in $B^{\pm}\rightarrow KD$ and measuring the CKM phase $\gamma$, Phys. Rev. D63 (2001) 036005, arXiv:hep-ph/0008090
  * [13] Belle collaboration, Y. Horii et al., Evidence for the suppressed decay $B^{-}\rightarrow DK^{-},D\rightarrow K^{+}\pi^{-}$, Phys. Rev. Lett. 106 (2011) 231803, arXiv:1103.5951
  * [14] Babar collaboration, P. del Amo Sanchez et al., Search for $b\rightarrow u$ transitions in $B^{-}\rightarrow DK^{-}$ and $B^{-}\rightarrow D^{*}K^{-}$ decays, Phys. Rev. D82 (2010) 072006, arXiv:1006.4241
  * [15] CDF collaboration, T. Aaltonen et al., Measurements of branching fraction ratios and CP-asymmetries in suppressed $B^{-}\rightarrow D(\rightarrow K^{+}\pi^{-})K^{-}$ and $B^{-}\rightarrow D(\rightarrow K^{+}\pi^{-})\pi^{-}$ decays, Phys. Rev. D84 (2011) 091504, arXiv:1108.5765
  * [16] Babar collaboration, P. del Amo Sanchez et al., Measurement of CP observables in $B^{\pm}\rightarrow D_{CP}K^{\pm}$ decays and constraints on the CKM angle $\gamma$, Phys. Rev. D82 (2010) 072004, arXiv:1007.0504
  * [17] CDF collaboration, T. Aaltonen et al., Measurements of branching fraction ratios and CP asymmetries in $B^{\pm}\rightarrow D_{CP}K^{\pm}$ decays in hadron collisions, Phys. Rev. D81 (2010) 031105, arXiv:0911.0425


# Scalar modes of polarization and speed of gravitational waves in $f(R)$
gravity

Utkal Keshari Dash<EMAIL_ADDRESS>Bal Krishna Yadav
<EMAIL_ADDRESS>Murli Manohar Verma<EMAIL_ADDRESS>Department of
Physics, University of Lucknow, Lucknow 226 007, India

###### Abstract

We explore the gravitational waves (GWs) within the framework of the $f(R)$
gravity model represented by $f(R)=R^{1+\delta}/R^{\delta}_{c}$ in the weak
field approximation. In this scenario, gravitational waves exhibit an
additional polarization mode beyond the standard transverse-traceless (TT)
tensor modes. We show that the polarization characteristics of these waves are
connected to the scalaron mass and the effective potential derived from the
function $f(R)$. Furthermore, the study of the speed of gravitational waves
($c_{g}$) within the Horndeski theory, particularly using the $f(R)$ model,
reveals an intriguing feature about the equality of the speed of gravitational
waves to that that of electromagnetic waves. This equivalence arises due to
the modification introduced in the Ricci scalar within the $f(R)$ model.

##  I Introduction

Einstein’s General Relativity (GR) has evidently stood as an invaluable tool
for addressing a wide range of cosmological issues related to gravitation. We
have amassed a wealth of observational evidence supporting the existence of
dark energy, such as data from sources like Supernovae type Ia, Baryon
Acoustic Oscillation (BAO), Cosmic Microwave Background (CMB) anisotropies,
and weak gravitational lensing, among others [1, 2, 3, 4]. Dark energy, though
not yet well known, is largely understood to be responsible for the ongoing
accelerated expansion of the universe.

Efforts to modify GR can be broadly categorized into two classes: (i) modified
gravity models and (ii) modified matter models. In modified gravity models [5,
6, 7, 8, 9], the gravitational aspect of the Einstein-Hilbert action is
altered in various ways. Numerous modified gravity models, such as $f(R)$
theory, Scalar-Tensor theories, Braneworld models, and Gauss-Bonnet dark
energy models, have been proposed and explored. Conversely, modified matter
models [10, 11, 12, 13] introduce an additional component of matter into the
energy-momentum tensor of GR. Quintessence [14], k-essence, and phantom dark
energy models are some examples of modified matter models.

The standard model of cosmology, often referred to as the Lambda Cold Dark
Matter ($\Lambda$CDM) model, provides a robust framework for explaining the
late-time cosmic acceleration [15, 16, 17]. However, it faces challenges
related to the cosmological constant $\Lambda$. Modified gravity models,
particularly $f(R)$ gravity models, have emerged as successful alternatives
for addressing both the dark matter and dark energy problems by leveraging the
concept of the scalaron mass. This mass exhibits dark matter-like effects on
local scales and behaves as dark energy on cosmological scales[18].

Among the various $f(R)$ models, the present paper focuses on our previous
model $f(R)=R^{1+\delta}/R^{\delta}_{c}$ as discussed in detail in our work at
small and large scales [19, 20, 21]. This model provides a sophisticated
explanation of the two distinct phases of the universe. Specifically, it
delves into how the mass of the scalaron depends on the $\delta$ parameter at
galactic scales, revealing a sharp decrease for smaller values of $\delta$ and
a subsequent increase for larger values. This insight enhances our
understanding of the dynamical behaviour of the universe at both, local and
cosmological, scales.

The significant progress in addressing cosmological challenges owes much to
the introduction of gravitational waves (GWs), a ground breaking discovery
made in 2023 when the NANOGrav Collaboration [22] successfully detected
gravitational waves through variations in Pulsar timing arrays [23]. Several
findings suggest that these waves hold tremendous potential for shedding light
on various cosmological questions, including the early inflationary epoch,
where we have already shown the scalar spectral index and tensor-to-scalar
ratio being very close to the recent observational data [20], age of the
universe, the Hubble parameter, dark matter and dark energy. This also proves
to be an asset for detection of black holes (BHs), Super Massive Black
Holes(SMBHs) and their dynamical evolution such as spin etc [24].

In the framework of Einstein’s General Relativity (GR), the plane wave
solution for gravitational waves encompasses two distinct polarization modes
as plus (+) mode polarization and cross($\times$) mode polarizations, which
shows the behaviour of gravitational waves in weak field limit in GR.

This paper aims to focus on the gravitational waves in the weak field
approximation limit in the modified $f(R)$ gravity model and investigate the
modes of polarisation that exist due to the effective generalisation of the
Ricci scalar. It has been argued that the energy carried by these additional
modes would cause perturbation in the angular momentum of the inspiralling
compact binary system and change the phase evolution of tensor modes. We hope
that if the phase correction due to scalar modes can be measured through their
distinct waveforms, that would indicate the existence of these modes and would
be able to constrain $f(R)$ model parameters. The observations of GW170814 and
GW170817 also put some constraints on the energy flux of the scalar modes
comparable in magnitude to that of the tensor modes [25].

Another important aspect discussed in this paper is the speed of gravitational
waves $c_{g}$. To understand the puzzling features of the universe like the
events occuring beyond the event horizon of the black hole and supermassive
black holes, pre-big-bang evolution, and other unsolved cosmological problems,
we need to have a clear knowledge of the speed of GWs along with its
counterpart electromagnetic (EM) waves. From theoretical point of view, if the
speed of GWs exceeds the speed of EM radiations, then the GWs can show a
distinct behaviour as they propagate inside the highly dense objects like SMB
or as stochastic waves in post-inflationary era. Further, if we find the speed
as scale dependent, it can also lead to significant features at quantum scales
too [26].

Using the more generalised Horndeski theory, we can estimate the speed of GWs
[28, 27]. There are various models like Quintessence and k-essence, Brans
Dicke model, Covariant Galileons, Derivative couplings, Gauss-Bonnet couplings
etc. but in this paper we have used f(R) theory as a special case of Horndeski
theory, to calculate the speed of gravitational waves and compare it to the EM
counterpart.

Thus, the present paper is organized in four sections. In Section II we
discuss the background of field equations in $f(R)$ gravity and conformal
transformation from the Jordan frame to the Einstein frame including an
introduction of GWs in the field equations. Section III covers solution to
waves equations in the $f(R)$ gravity model and examination of the resulting
polarisation modes. In Section IV, we explore the speed of these waves in our
specific model. Section V encapsulates the summary and conclusion.

##  II  $f(R)$ gravity and linearization of gravitational waves

In this section, we discuss about the modification made in geometric part of
the Einstein’s field equations.

We begin with the 4-dimensional action in the general $f(R)$ gravity model
which contains the modification of Ricci scalar in some functional form. Thus,
the action is modified as

$\mathcal{A}=\frac{1}{2\kappa^{2}}\int
d^{x}\sqrt{-g}f(R)+\mathcal{A}_{m}(g_{\mu\nu},\psi_{m}),$ (1)

where $\kappa^{2}=8\pi G$ and $\mathcal{A}_{m}$ is the action of the matter
field $\psi_{m}$. The Friedmann-Lemaitre-Robertson-Walker (FLRW) spacetime is
given as

$\quad ds^{2}=-dt^{2}+a^{2}(t)[dr^{2}+r^{2}(d\theta^{2}+sin^{2}\theta
d\phi^{2})],$ (2)

where $a(t)$ is the time dependent scale factor and speed of light $c$ = 1.
The Einstein’s field equations then can be obtained as

$F(R)R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}f(R)+g_{\mu\nu}\Box F(R)$ $\hskip
85.35826pt-\nabla_{\mu}\nabla_{\nu}F(R)=\kappa^{2}T_{\mu\nu},$ (3)

where $F(R)=df/dR$ and $T_{\mu\nu}$ is the energy-momentum tensor contributed
by matter.

Taking the trace of (3) in vacuum $(T_{\mu}^{\mu}=0)$, we obtain

$3\Box F(R)+RF(R)-2f(R)=0.$ (4)

By conformally transforming from the Jordan frame to the Einstein frame, the
new action can be written as [29, 30, 33]

$\mathcal{A}=\int
d^{4}x\sqrt{-\tilde{g}}\left[\frac{1}{2\kappa^{2}}\tilde{R}-\frac{1}{2}\tilde{g}^{\mu\nu}\partial_{\mu}\phi\partial_{\nu}\phi-V(\phi)\right]$
$\hskip 170.71652pt+\mathcal{A}_{m},$ (5)

where

$V(\phi)=\frac{FR-f}{2\kappa^{2}F^{2}}.$ (6)

represents the potential in a general class of $f(R)$ models showing the
emergence of a new scalar degree of freedom and an overhead tilde denotes the
quantities in the Einstein frame.

In this modified gravity background, we consider dark matter as scalaron
particle corresponding to field $\phi$ which is related to $F(R)$ in the
Einstein frame as

$\Omega^{2}=F(R)=e^{-2Q\kappa\phi},$ (7)

where $Q$ is the strength of coupling between scalar field $\phi$ and non-
relativistic matter and is given as

$Q=-\frac{1}{\sqrt{6}}.$ (8)

Thus, by varying the action (5) with respect to scalar field $\phi$, the
Klein-Gordon equation for effective scalar field $\phi$ is obtained as [30,
29, 33]

$\tilde{\Box}\phi=\frac{dV(\phi)}{d\phi}+\frac{\kappa}{\sqrt{6}}\tilde{T}\equiv\frac{dV_{eff}(\phi)}{d\phi},$
(9)

where $V_{eff}(\phi)$ is the effective potential whose second derivative
provides scalar field a mass as
$m^{2}_{\phi}=d^{2}V_{eff}(\phi_{min})/{d\phi^{2}}$, with $V_{eff}(\phi)$
having the minimum value at $\phi_{min}$.

The gravitational waves are extremely weak at the point very far from the
sources, and so they can be linearized. Thus, the metric can be characterized
in covariant form as given below.

$g_{\mu\nu}=\eta_{\mu\nu}+h_{\mu\nu}.$ (10)

where $g_{\mu\nu}$ is the metric, $\eta_{\mu\nu}$ is the background metric and
$h_{\mu\nu}$ is the small perturbation in the background metric with the
condition $|{h_{\mu\nu}}|<<1$.

The contravariant counterpart of the metric in (10) can be written as

$g^{\mu\nu}=\eta^{\mu\nu}-h^{\mu\nu}.$ (11)

The linearization of the modified Einstein field equations leads to

$\displaystyle
F(R)R_{\mu\nu}+\frac{1}{2}(\eta_{\mu\nu}+h_{\mu\nu})f(R)+(\eta_{\mu\nu}+h_{\mu\nu})\Box
F(R)$ $\hskip 113.81102pt-\nabla_{\mu}\nabla_{\nu}F(R)=0.$ (12)

It is assumed that the sources of field for this condition are very far from
the point where gravitational waves are detected, so
$\mathcal{A}_{m}=0\implies T_{\mu\nu}=0$ [30, 34].

##  III Effect of scalaron mass in gravitational waves for
$f(R)=R^{1+\delta}/R^{\delta}_{c}$ type models

The constant tangential velocity condition of the test particle specifies that
the motion of the test particle is stable in the orbits of spiral galaxies,
which indicates that $f(R)$ is proportional to $R^{1+\delta}$, where
$\delta<<1$. A generic $f(R)$ model gives rise to the scalar degree of freedom
and, in particular, can be represented as

$f(R)=\frac{R^{1+\delta}}{R^{\delta}_{c}},$ (13)

where $R_{c}$ is a constant having dimensions of Ricci Scalar $R$, $\delta$ is
the model parameter and $\delta<<1$. The derivative of $f(R)$ in equation (13)
is given as

$F(R)=(1+\delta)\frac{R^{\delta}}{R^{\delta}_{c}}.$ (14)

Now, from equation (3), the field equations can be written as

$G_{\mu\nu}\equiv\tilde{R}_{\mu\nu}-\frac{1}{2}g_{\mu\nu}\tilde{R}$
$=\frac{R^{\delta}_{c}}{R^{\delta}}\left[\frac{g_{\mu\nu}}{2}\left(\frac{R^{1+\delta}}{R^{\delta}_{c}}-(1+\delta)\frac{R^{\delta}.R}{R^{\delta}_{c}}\right)+\left({\frac{R^{\delta}}{R^{\delta}_{c}}}\right)_{,\mu,\nu}\right]$
$-\frac{R^{\delta}_{c}}{R^{\delta}}\left[g_{\mu\nu}\frac{\Box{R^{\delta}}}{R^{\delta}_{c}}\right]=0.$
(15)

The equation (15) further leads to the following form

$G_{\mu\nu}=\tilde{R}_{\mu\nu}-\frac{\tilde{R}}{2}g_{\mu\nu}=\frac{R^{\delta}_{c}}{R^{\delta}}\left[\left({\frac{R^{\delta}}{R^{\delta}_{c}}}\right)_{,\mu,\nu}-g_{\mu\nu}\frac{\Box{R^{\delta}}}{R^{\delta}_{c}}\right]$
$-\frac{R^{\delta}_{c}}{R^{\delta}}\left[{\frac{1}{2}\eta_{\mu\nu}(\delta
R^{\delta+1})}\right]=0.$ (16)

Recalling conformal transformation from equation (7)

$\phi=\sqrt{\frac{2}{3}}\ln F.$ (17)

The Ricci scalar then gets transformed to

$R=R_{c}\left[\frac{e^{\sqrt{\frac{2}{3}}\kappa\phi}}{1+\delta}\right]^{\frac{1}{\delta}}.$
(18)

The modified Einstein’s field equations in the Einstein frame are given by

$\tilde{R}_{\mu\nu}-\frac{1}{2}g_{\mu\nu}\tilde{R}={\sqrt{\frac{2}{3}}\kappa}\left[\partial_{\mu}\partial_{\nu}\phi-\eta_{\mu\nu}\Box{\phi}\right]$
$-\frac{1}{2}\eta_{\mu\nu}\delta
R_{c}\left[\frac{e^{\sqrt{\frac{2}{3}}\kappa\phi}}{1+\delta}\right]^{\frac{1}{\delta}}=0.$
(19)

As gravitational waves coupled with scalar field component propagate, the
later gets perturbed by a small change given as $\Delta{\phi}$.

$\phi=\phi_{0}+\Delta{\phi}.$ (20)

Using $h_{s}=\frac{\Delta{\phi}}{\phi_{0}}$, we obtain

$\tilde{R}_{\mu\nu}-\frac{1}{2}g_{\mu\nu}\tilde{R}={\sqrt{\frac{2}{3}}\kappa\phi_{0}}\left(\partial_{\mu}\partial_{\nu}h_{s}-\eta_{\mu\nu}\Box{h_{s}}\right)$
$-\frac{1}{2}\eta_{\mu\nu}\delta
R_{c}\left[\frac{e^{\sqrt{\frac{2}{3}}\kappa\phi}}{1+\delta}\right]^{\frac{1}{\delta}}=0,$
(21)

where $h_{s}$ is the massive mode polarization term which has been generated
due to the scalar field $\phi$ which produces the longitudinal modes in
traceless transverse modes.

Using equations (4)\- (9) in previous Section, we get

$m^{2}_{\phi}\left[\frac{3\delta(1+\delta)}{1-\delta}\right]=R_{c}\left[\frac{e^{\sqrt{\frac{2}{3}}\kappa\phi}}{1+\delta}\right]^{\frac{1}{\delta}},$
(22)

where $m_{\phi}$ term represents the mass of the scalaron and is given as [19]

$m^{2}_{\phi}=\frac{(1-\delta)^{\frac{2\delta}{1+\delta}}}{3\delta(1+\delta)}(R_{c})^{\frac{2\delta}{1+\delta}}(\kappa^{2}\rho)^{\frac{1-\delta}{1+\delta}},$
(23)

which also includes the source $T_{\mu\nu}$ whose trace is $\rho$.

Using equation (22) in equation (21) we obtain

$\Box
h_{s}=\sqrt{\frac{3}{2}}\frac{\delta^{2}(1+\delta)}{(\delta-1)\phi_{0}}m^{2}_{\phi}.$
(24)

From the above equation, it is evident that the massive mode polarisation
depends on the mass of the scalaron which itself is dependent on $\delta$. So,
the massive field shows complete dependence on the model. The plane wave
equation for transverse mode of polarisation is written as

$\Box h_{\mu\nu}=0.$ (25)

On solving equations (23) and (24) together we obtain [30]

$h_{\mu\nu}=B_{\mu\nu}(\textbf{p})e^{ik^{\alpha}x_{\alpha}}$ (26)

and

$h_{s}=H(\textbf{q})e^{iq^{\alpha}x_{\alpha}}$ (27)

are added to the plus and cross polarization terms are added, which is now
given as [30]

$\bar{h}_{\mu\nu}=h_{+}e^{(+)}_{\mu\nu}+h_{\times}e^{(\times)}_{\mu\nu}+h_{s}e^{(s)}_{\mu\nu}.$
(28)

Since the polarized wave is travelling along the Z-direction, these
polarizations modify to

$\bar{h}_{\mu\nu}=\begin{bmatrix}1&0&0&0\\\ 0&h_{+}&h_{\times}&0\\\
0&h_{\times}&-h_{+}&0\\\ 0&0&0&h_{s}\end{bmatrix}.$ (29)

The group velocity of $h_{s}$ can be given as

$v_{g}=\frac{p}{\omega_{m}}$ (30)
$v_{g}=\sqrt{1-\left(\frac{m_{\phi}}{\omega_{m}}\right)^{2}},$ (31)

where $\omega_{m}=\sqrt{m_{\phi}^{2}+{p}^{2}}$ is the angular velocity of the
massive gravitational wave propagating in the longitudinal direction. Thus, by
using the constraints on the scalaron mass from cosmological evolution and
consistent with dark matter in our earlier work [19], we can constrain
$v_{g}$. It may be compared to the constraints obtained on the velocity
through the mass of gravitons $m_{g}$ as
$\left(m_{g}/\omega_{orbital}\right)^{2}<0.003$ by consideration of energy
loss in form of emission of massive gravitons from compact binary systems [31,
32]. In our model, using equation (23) with (31), the propagation velocity is
given as

$v_{g}=\sqrt{1-\frac{(1-\delta)^{\frac{2\delta}{1+\delta}}}{12\pi^{2}n^{2}\delta(1+\delta)}R_{c}^{\frac{2\delta}{1+\delta}}(\kappa^{2}\rho)^{\frac{1-\delta}{1+\delta}}},$
(32)

where $n$ is the wave frequency. Clearly, $v_{g}$ is sensitive to $R_{c}$,
$\delta$ for a given matter background (both of which together determine the
scalaron mass), and wave frequency depending on the dynamically evolving
sources of GWs (which includes a stochastic GW background). This can be used
as an effective tool to distinguish the cosmological models beyond GR from the
GR-based models over a wide range of frequencies in the future GW detectors.
We explore the behaviour of the velocity $v_{g}$ on our model parameters as
the waves propagate through the matter background.

Figure 1: Variation of $v_{g}$ vs $\delta$ for $R_{c}$ = $\Lambda$
(cosmological constant = $10^{-84}(GeV)^{2}$), $10^{-78}(GeV)^{2}$ and
$1(GeV)^{2}$, shown as black dotted, red and blue dashed curves, respectively.
Value of energy density of background matter $\rho=10^{-42}(GeV)^{4}$ is
chosen at the galactic scales and frequency is chosen as the lowest
($10^{-18}$ Hz) in the ELF band to see its comparable effect with respect to
$R_{c}$ in the $v_{g}$. Figure 2: Plot of $v_{g}$ vs $\delta$ for frequency
$=10^{-8}$Hz relevant to NANOGrav band. Velocity approaches $1$ for
$\delta\sim 10^{-24}$. $R_{c}$ = $\Lambda$ (cosmological constant =
$10^{-84}(GeV)^{2}$) and $\rho=10^{-42}(GeV)^{4}$.  Figure 3: Plot of $v_{g}$
vs $\delta$ for frequency = $10^{-2}$Hz of the expected range of LISA.
Velocity approaches $1$ for $\delta\sim 10^{-36}$. $R_{c}$ = $\Lambda$
(cosmological constant = $10^{-84}(GeV)^{2}$) and $\rho=10^{-42}(GeV)^{4}$.
Figure 4: Plot of $v_{g}$ vs $\delta$ with for frequency = $100$ Hz of LIGO
window. Velocity approaches $1$ for extremely low value of $\delta\sim
10^{-44}$. $R_{c}$ = $\Lambda$ (cosmological constant = $10^{-84}(GeV)^{2}$)
and $\rho=10^{-42}(GeV)^{4}$.  Figure 5: Multi-frequency variation of $v_{g}$
vs $\delta$ for a set of different frequencies. Brown curve represents
frequency $10^{-18}$ Hz, while the green dashed, yellow dashed and red dashed
curves correspond to the frequencies $10^{-8}$ Hz, $10^{-2}$ Hz and $100$ Hz,
respectively. $R_{c}$ = $\Lambda$ (cosmological constant =
$10^{-84}(GeV)^{2}$) and $\rho=10^{-42}(GeV)^{4}$.  Figure 6: This plot shows
$v_{g}$ vs $\rho$ for $R_{c}$ = $\Lambda$ (cosmological constant =
$10^{-84}(GeV)^{2}$). Smooth red, deep rainbow, black, green and blue dashed
lines correspond to $\delta$= $2\times 10^{-5}$, $10^{-4}$, $10^{-3}$,
$10^{-2}$ and $0.1$, respectively. Frequency is chosen as the lowest
($10^{-18}$Hz) in the ELF band.

Figure 1 describes the role of parameter $\delta$ in variation $v_{g}$ of GWs.
It shows a range of $\delta$ up to an order of $10^{-4}$. This speed
asymptotically approaches the value of speed of light beyond this value of
$\delta$. This indicates that if the mass of the scalaron is heavier then it
would more strongly impede the propagation of GWs through galactic background
($\rho=10^{-42}(GeV)^{4}$), and the longitudinal mode would tend to disappear.
This is a manifestation of the chameleon mechanism in the higher density
regions where mass of scalaron $m_{\phi}$ becomes too large in response to the
matter environment and consequently, the Compton wavelength of scalaron
becomes extremely small. This leads to the collapse of scalar field and the
gravitational wave loses the scalar longitudinal mode. The plot also shows
that the speed of the gravitational waves does not change much for different
values of $R_{c}$. This is because the value of $R_{c}$ is extremely small
compared to the much higher contribution from frequency (which is actually the
lowest, as taken here) in the extremely low frequency (ELF) band
$10^{-15}-10^{-18}$ Hz, where gravitational wavelength is $\pi$ times the
Hubble distance producing quadrupolar anisotropies in CMB radiation. Of
course, when we choose still higher frequencies corresponding to Laser
Interferometer Gravitational Wave observatory (LIGO), future Laser
Interferometer Space Antenna (LISA), NANOGrav etc. and carry out a similar
analysis, we find the increasing independence from $R_{c}$.

In addition to frequency $10^{-18}$ Hz taken in Figure 1, further, the
variation of $v_{g}$ is plotted for a set of other frequencies ($10^{-8}$ Hz,
$10^{-2}$ Hz and $100$ Hz) in Figures 2, 3, and 4, respectively. Here, we use
a single value of $R_{c}=\Lambda$ and a common matter density background of
galaxies $\rho=10^{-42}(GeV)^{4}$, and attempt to examine the effects on these
vastly different frequencies which arise from different sources and fall
within the detection range of the present or future detectors. It can be
clearly seen that for higher frequencies, $v_{g}$ gets more sharply closer to
$c=1$ for successively lower values of model parameter $\delta$. This combined
multi-frequency behaviour over the large range of $\delta=0-10^{-4}$ is shown
in Figure 5, where only $10^{-18}$ Hz coming from the stochastic gravitational
wave background displays a distinct variation, while at all other chosen
frequencies, velocity $v_{g}$ is uniformly equal to that of light.

In Figure 6, we observe a significant variation in $v_{g}$ with the background
matter density for different values of $\delta$. When $\delta$ is of very low
order $\sim 10^{-5}$, the value of $v_{g}$ drops sharply with matter density,
but when it increases further, then close to $\delta=0.001$ and $0.1$, the
curve approximates speed of light $c=1$ for the given range of $\rho$. This is
due to rising mass of scalaron with decreasing value of $\delta$. We find that
$\rho$ is also dominant for the value of $\delta<10^{-3}$ or lower, and
becomes successively less effective for the higher values of $\delta$. Thus,
smaller values of $\delta$, and higher background densities, both provide a
higher mass to scalarons, and so, both are favourable for reducing the speed
of the longitudinal mode. At the given galactic background density, the degree
of elimination of the longitudinal mode will grow for smaller values of
$\delta$ and may escape detection. Thus, using the cosmological constraints on
mass of scalaron from the consideration of age of the universe etc., we can
conclude that the longitudinal mode of GWs does not propagate at the speed of
light when $\delta$ is very small and mass of scalarons becomes high.

##  IV Speed of gravitational waves in Horndeski theory using $f(R)$ gravity

The detection of gravitational waves from the event GW170817 provided the
possibility of determination of their velocity. Several efforts, such as in
Horndeski theory, emerged with an extended scope of study [35, 28, 27, 36].
The Lagrangian from the generalised action

$A=\int d^{4}x\sqrt{-g}L$ (33)

can be written using this theory as [35]

$L=G_{2}(\phi,X)+G_{3}(\phi,X)\Box\phi+G_{4}(\phi,X)R+G_{4,X}(\phi,X)[(\Box\phi)^{2}$
$-(\nabla_{\mu}\nabla_{\nu}\phi)(\nabla^{\mu}\nabla^{\nu}\phi)]+G_{5}(\phi,X)G_{\mu\nu}\nabla^{\mu}\nabla^{\nu}\phi$
$-\frac{1}{6}G_{5,X}(\phi,X)[(\Box\phi)^{3}-3(\Box\phi)(\nabla_{\mu}\nabla_{\nu}\phi)(\nabla^{\mu}\nabla^{\nu}\phi)$
$+2(\nabla^{\mu}\nabla_{\alpha}\phi)(\nabla^{\alpha}\nabla_{\beta}\phi)(\nabla^{\beta}\nabla_{\mu}\phi)],$
(34)

where $G_{2,3,4,5}$ depend on $\phi$ and $X$, and $G_{i,\phi}\equiv{\partial
G_{i}}/{\partial\phi}$ and $G_{i,X}\equiv{\partial G_{i}}/{\partial X}$ with

$X=-\frac{1}{2}\nabla_{\mu}\phi\nabla^{\mu}\phi.$ (35)

Using $f(R)$ gravity model we can write action (33) in terms of equation (1).
Thus, the Lagrangian can be transformed accordingly with the choice of
$G_{2},G_{3},G_{4},G_{5}$, and all these parameters take the form

$G_{2}=RF-f,\hskip 5.69046ptG_{3}=0,\hskip 5.69046ptG_{4}=F,\hskip
5.69046ptG_{5}=0.$ (36)

Using the equivalence principle, we can transform the values of the above
parameters in terms of $\phi$ which leads to the following equations,

$G_{2}=\delta
R^{1+\delta}\left(e^{\sqrt{\frac{2}{3}}\kappa\phi}\right)^{\frac{1+\delta}{\delta}},G_{3}=0,G_{4}=e^{\sqrt{\frac{2}{3}}\kappa\phi},$
$G_{5}=0.$ (37)

The speed of the gravitational waves $c_{g}$ propagating over a cosmological
matter background depends on the scalar field which can be written in the
generalised form using Horndeski theory as

$c^{2}_{g}=\frac{2G_{4}-(\dot{\phi})^{2}G_{5,\phi}-\dot{\phi}^{2}\ddot{\phi}G_{5,X}}{2G_{4}-(2\dot{\phi})^{2}G_{4,X}+(\dot{\phi})^{2}G_{5,\phi}-H(\dot{\phi})^{3}G_{5,X}},$
(38)

where

$X=-\frac{1}{2}[\dot{\phi}^{2}-{\nabla\phi}^{2}]$ (39)

$H$ is the Hubble parameter. Using equation (37) in equation (38) we obtain

$c^{2}_{g}=\frac{2G_{4}}{2G_{4}-(2\dot{\phi})^{2}G_{4,X}}.$ (40)

Further, we consider the second term of the denominator of equation (38) as

$G_{4,X}=\frac{\partial G}{\partial X}=\frac{\partial
e^{\sqrt{\frac{2}{3}}\kappa\phi}}{\partial[\dot{\phi}^{2}-{\nabla\phi}^{2}]}=0.$
(41)

From equations (40) and (41), we obtain $c^{2}_{g}=1$ This implies that the
GWs would propagate with the speed of light in this case.

##  V Discussion and conclusion

In the present paper, we investigated the gravitational waves in an $f(R)$
gravity model and found that the conventional transverse polarisation modes
(plus and cross) are accompanied by a massive polarisation mode along the
Z-direction of the wave propagation, which is produced due to the scalar
degree of freedom associated with the modification in the geometrical part of
the Einstein-Hilbert action. The scalar field seems to play no direct role in
the transverse modes of polarisation, but causes the creation and propagation
of this massive longitudinal mode. This $h_{s}$ mode varies with $m_{\phi}$,
and thus, is sensitive to even an extremely small variation in the powers of
Ricci scalar. The velocity curves plotted in Figures show that this
longitudinal mode of gravitational wave cannot propagate when $\delta=0$.
However, depending on frequency ($10^{-18}$ Hz, $10^{-8}$ Hz, $10^{-2}$ Hz and
$100$ Hz corresponding to stochastic GW background imprinted on CMB, NANOGrav,
future LISA and present LIGO detectors, respectively), the speed of
propagation in galactic matter background picks up sharply and jumps to speed
of light as $\delta$ is switched on. Thus, for example, we find that for a
$100$ Hz astrophysical GW longitudinal mode, its velocity jumps very sharply
to speed of light even as $\delta$ slightly increases beyond $10^{-44}$.

Thus, it can be interpreted that in GR limit where mass is infinite, the
longitudinal modes of vibration cannot be formed. On the other hand, when we
calculate the speed of the gravitational waves using Horndeski theory, it is
found exactly equal to the speed of light. Using the $f(R)$ model as a special
case in Horndeski theory, we have only the Ricci curvature in the field
equations which contributes to the conformal part, but when we consider the
Riemannian part we find an inequality in the speed of GWs and speed of light.
This anomalous speed of GWs is due to the involvement of Weyl curvature tensor
when $G_{5}$ becomes non zero. These consideration allow us to formulate a
Weyl criterion for anamolous speed of spin-2 GWs [26, 37] using the Newman-
Penrose formalism which will be discussed in our subsequent work.

## Acknowledgments

Authors thank the Inter-University Centre for Astronomy and Astrophysics
(IUCAA), Pune for the facilities where this work was partially done under the
visiting associateship programme.

## References

  * [1] A. G. Riess et al., (Supernova Search Team Collaboration), Astron. J. 116, 1009 (1998).
  * [2] B. Schmidt et al., Astrophys. J. 507, 46 (1998).
  * [3] S. Perlmutter et al., Astrophys. J. 517, 565 (1999).
  * [4] D. N. Spergel et al., Astrophys. J. Suppl. 148, 97 (2003).
  * [5] M. Kunz and D. Sapone, Phys. Rev. Lett. 98, 121301 (2007).
  * [6] L. Amendola, R. Gannouji, D. Polarski and S. Tsujikawa, Phys. Rev. D 75, 083504 (2007).
  * [7] V. Sahni and Y. Shtanov, JCAP 0311, 014 (2003).
  * [8] A. A. Starobinski, Phys. Lett. B 91, 99 (1980).
  * [9] S. Nojiri and S. D. Odintsov, Phys. Rept. 505, 59 (2011).
  * [10] R. R. Caldwell, R. Dave, and P. J. Steinhardt, Phys. Rev. Lett. 80, 1582 (1998).
  * [11] S. Capozziello, Int. J. Mod. Phys. D 11, 483 (2002)
  * [12] T. Chiba, T. Okabe, and M. Yamaguchi, Phys. Rev. D 62, 023511 (2000).
  * [13] R. R. Caldwell, Phys. Lett. B 545, 23 (2002).
  * [14] V. Sahni and L. Wang, Phys. Rev. D 62, 103517 (2000).
  * [15] S. M. Carroll, Living Rev. Relativ. 4, 1 (2001).
  * [16] P. J. E. Peebles and B. Ratra, Rev. Mod. Phys. 75, 559-606 (2003).
  * [17] A. Borriello and P. Salucci, Mon. Not. R. Astron. Soc. 323, 285 (2001).
  * [18] T. Katsuragawa and S. Matsuzaki, Phys. Rev. D 97, 064037, (2018).
  * [19] B. K. Yadav and M. M. Verma, JCAP 10  052 (2019).
  * [20] A. K. Sharma and M. M. Verma, Astrophys. J. 926, 29 (2022).
  * [21] A. K. Sharma and M. M. Verma, Astrophys. J. 934, 13 (2022).
  * [22] G. Agazie et al., [NANOGrav], Astrophys. J. Lett. 951, L8 (2023).
  * [23] B. Das, N. Jaman, and M. Sami, Phys. Rev. D 108, 103510 (2023).
  * [24] S. Jana, S. J. Kapadia, T. Venumadhav, and P. Ajith, Phys. Rev. Lett 130, 261401 (2023).
  * [25] H. Takeda, Y. Manita, H. Omiya1 , and T. Tanaka, Prog. Theor. Exp. Phys. 073E01 (2023) .
  * [26] D. Bettoni et al., Phys. Rev. D 95, 084029 (2017).
  * [27] P. I. Dyadina, Jour of Exp. and Theor. Phys. 135, (2022).
  * [28] T. Kobayashi, Rep. Prog. Phys. 82, 086901 (2019).
  * [29] A. D. Felice and S. Tsuzikawa, Living Rev. Relativity 13, 3, (2010).
  * [30] S. Capozziello, C. Corda, and M. F. De Laurentis, Phys. Let. B 669, 5 (2008).
  * [31] L. S. Finn and P. J. Sutton, Phys. Rev. D 65, 044022, (2002).
  * [32] A. Nishizawa et al., Phys. Rev. D 79, 082002, (2009).
  * [33] H. R. Kausar, L. Philippoz, and P. Jetzer, Phys. Rev. D 93, 124071 (2016).
  * [34] D. M. Eardley, D. L. Lee, and A. P. Lightman, Phys. Rev. D 8, 10 (1973).
  * [35] R. Kase and S. Tsujikawa, Int. J. Mod. Phys. D 28, 1942005, (5) (2019).
  * [36] P. Creminelli and F. Vernizzi, Phys. Rev. Lett. 119, 251302 (2017).
  * [37] S. Capozziello and M. Capriolo, Class. Quantum Grav. 38, 175008,(2021).


# Toward Interactive Modulation for Photo-Realistic Image Restoration

Haoming Cai${}^{1\ *}$ Jingwen He1 Yu Qiao1,2 Chao Dong1,3
1Key Laboratory of Human-Machine Intelligence-Synergy Systems Denotes equal
contributionDenotes corresponding author (e-mail<EMAIL_ADDRESS>
Shenzhen Institutes of Advanced Technology  Chinese Academy of Sciences
2Shanghai AI Lab  Shanghai  China
3SIAT Branch  Shenzhen Institute of Artificial Intelligence and Robotics for
Society
{hm.cai, jw.he, yu.qiao<EMAIL_ADDRESS>
###### Abstract

Modulating image restoration level aims to generate a restored image by
altering a factor that represents the restoration strength. Previous works
mainly focused on optimizing the mean squared reconstruction error, which
brings high reconstruction accuracy but lacks finer texture details. This
paper presents a Controllable Unet Generative Adversarial Network (CUGAN) to
generate high-frequency textures in the modulation tasks. CUGAN consists of
two modules - base networks and condition networks. The base networks comprise
a generator and a discriminator. In the generator, we realize the interactive
control of restoration levels by tuning the weights of different features from
different scales in the Unet architecture. Moreover, we adaptively modulate
the intermediate features in the discriminator according to the severity of
degradations. The condition networks accept the condition vector (encoded
degradation information) as input, then generate modulation parameters for
both the generator and the discriminator. During testing, users can control
the output effects by tweaking the condition vector. We also provide a smooth
transition between GAN and MSE effects by a simple transition method.
Extensive experiments demonstrate that the proposed CUGAN achieves excellent
performance on image restoration modulation tasks.

##  1 Introduction

Figure 1: The input image is with blur $r2$ (kernel width=2) and $\sigma 30$.
Through the modulation on deblurring and denoising, users could obtain a well
restored image with rich texture details. Besides, users are allowed to
modulate on the green sliding bar to achieve smooth transition between GAN and
MSE effects.

Deep learning methods have achieved great success in many low-level vision
tasks, such as image denoising, deblurring, and super-resolution. Various
network architectures and training strategies have been continuously improving
the reconstruction quality (e.g., PSNR). Later on, to pursue visually pleasing
results, the generative adversarial network (GAN) [13] is introduced to
encourage the network to produce natural looking images. For most image
restoration tasks, a deep model learns a deterministic mapping and outputs a
fixed result for a pre-determined degradation type/level. For instance, many
image restoration deep models are trained for a single level (e.g., Gaussian
noise $\sigma$50). However, this deterministic mapping is not flexible, as
users cannot continuously modify the restoration effect or strength based on
personal preference. Moreover, when the degradation model mismatches the
degraded input, the output image will contain severe artifacts, especially for
GAN-based models, as shown in Figure 2. Therefore, developing modulation
models that can flexibly handle a range of degradations by sliding bars is
essential and practical.

In recent years, several modulation methods [16, 31, 30, 11] have been
proposed to adapt existing deep models to other objectives. Specifically, they
can generate continuous restoration results between the pre-defined start
level and end level (e.g, denoising $\sigma 15\rightarrow\sigma 50$).
Furthermore, CResMD [17] proposes a multi-dimension modulation framework that
allows jointly modulation for different degradations. However, the above
modulation methods are all PSNR-oriented, which will produce over-smoothed
results without sufficient high-frequency details. To obtain modulation
outputs with photo-realistic effect, this work focuses on interactive
modulation for GAN-based image restoration.

The main challenges behind PSNR-oriented and GAN-based modulation are
different. For PSNR-oriented modulation methods, such as CResMD, severe and
mild degradations have different magnitude orders on the MSE loss, which will
lead to the ”unbalanced learning” problem. This phenomenon encourages models
to focus on restoring images with severe degradations while ignoring the mild
ones. On the contrary, in GAN-based modulation, the generator may ignore those
severe degradations if we directly apply a vanilla GAN. Specifically, for a
vanilla discriminator, an image restored from severe degradations will look
like the fake one compared with that from mild ones. This incorrect judgment
will lead to the vanishing generator gradient on those severe degradations.
Therefore, we need specific discriminator to clarify images restored from
various degradations. This discriminator could adjust its judgment criterion
based on the degradation of the input image.

Figure 2: Artifacts incurred by applying GAN-based and PSNR-oriented model
with mismatched restoration level. Models are trained on blur $r2$+noise
$\sigma 30$, while the input image is with blur $r1$+noise $\sigma 50$.

In this paper, we introduce a novel GAN-based modulation framework for photo-
realistic image restoration. The proposed controllable Unet generative
adversarial network (CUGAN) adopts the Unet architecture. We realize the
interactive control of the reconstruction result by tuning the weights of
different features from different scales or within one scale in the Unet
framework. To tackle vanishing generator gradient on severe degradations, we
apply global feature modulation (GFM) [18] in the discriminator. Based on the
severity of degradations, the global feature modulation adaptively modulates
intermediate features of the discriminator. The above modulation is achieved
by the condition networks embedded in the CUGAN. The condition networks accept
a condition vector that encodes the degradation information for each input
image. Then, the condition networks generate the tuning weights for the
generator and the parameters of global feature modulation (GFM) for the
discriminator. During testing, users can control the restoration strength by
tweaking the condition vector. We also provide the trade-off method between
GAN and MSE effect, allowing users to adjust the output effect in another
dimension. Different from DNI [31], our interpolation works on both the base
network $G$ and the condition network $C_{G}$. Figure 1 shows a qualitative
example for the whole modulation process.

To verify the effectiveness of the proposed method, we conduct extensive
experiments on modulation for image restoration with multiple degradations.
Experimental results show that the proposed CUGAN could achieve modulation
with high visual quality, high reconstruction accuracy, or a compromise
between them. We also demonstrate its effectiveness on image restoration in
real-world scenarios. In this work, our main contributions can be summarized
as follows:

  * $\bullet$

We propose a novel GAN-based modulation framework, named CUGAN, to obtain
photo-realistic restoration results by interactive modulation.

  * $\bullet$

We propose a discriminator equipped with global feature modulation to ease the
vanishing generator gradient on severe degradations.

  * $\bullet$

We test the restoration performance on selected datasets and real-world
images. Extensive experiments show the effectiveness of CUGAN in modulated
image restoration.

##  2 Related Work

Perceptual Image Restoration. With the rapid development of deep learning in
recent years, deep learning techniques have been widely explored to tackle
image restoration problems, such as image super-resolution, denoising,
deblurring, and compression artifacts reduction [34, 36, 15, 10, 37, 14].
Focusing on optimizing PSNR, the above methods tend to generate blurry images.
In contrast, perceptual image restoration targets to obtain better perceptual
results with more texture details by appling GAN[13]. Ledig et al. [24]
propose SRGAN that could generate photo-realistic images in SR task. In the
PIRM2018-SR Challenge [3], ESRGAN [32] achieves state-of-the-art performance
by improving the network architecture and loss functions. Benefiting from a
learnable ranker, RankSRGAN [39] could generate visually pleasant images that
favor different perceptual metrics.

Explorative Image Restoration. Despite that deep-learning-based methods have
achieved high qualitative performance, most of them deal with image
restoration problem by learning a deterministic mapping. To allow users
adjusting the restoration effects, Bahat et al.[2] propose an editing module
that could iteractively influence the texture or brightness. With normalizing
flows [9, 23], SR-Flow [27] takes a step forward to model the conditional
distribution of all possible SR reconstructions given an LR facial input.
Furthermore, DeepSEE [5] also adopts normalizing flow to leverage semantic
maps for explorative facial super-resolution.

Modulation for Image Restoration. Although existing methods could allow user
to adjust restored effect, most image restoration deep networks are trained on
one specific degradation level. Therefore, it is tiring to train $N$ various
models for $N$ various degradations. DNI [31] and AdaFM [16] find the high
similarity on kernels between models trained on various levels. Based on this
observation, DNI directly interpolate parameters between two related networks
to attain a smooth control of diverse imagery effects. While AdaFM adopts a
more efficient way: utilize depth-wise convolution layers to modulate the
intermediate features instead of changing all convolution filters. Different
from above two interpolation-based methods, the work in CFSnet [30] adaptively
learns the interpolation coefficients and uses them to couple intermediate
features from the main branch and tuning branch.

However, In real-world scenarios, images contain multiple degradations, such
as blur and noise. Since different types of degradations are coherently
related, modulation for multiple degradations should be conducted jointly, not
independently. To address this, CResMD [17] regards the modulation as a
conditional image restoration problem, and proposed a framework that accepts
both corrupted images and their degradation information as input. Thus, image
restoration is conditioned on the restoration/degradation information.
Modulation for image restoration is essential in practical usages not merely
because that it enables a unified framework to handle multiple degradations
with arbitrary levels. More importantly, it provides a flexible interaction on
the restoration strength.

Figure 3: The framework of CUGAN. CUGAN consists of 2 base networks as well as
2 condition networks. The 2 base networks are the generator $G$ and the
discrimiantor $D$. The 2 condition networks are condition network for
generator ($C_{G}$) and condtion network for discriminator ($C_{D}$). During
training and testing process, the condition networks $C_{G}$ and $C_{D}$
accept degradation information and generate tunning weights for $G$ and $D$.
In the modulation Unet, we provide Modulated Residual Block (MRB) and
Modulated Scale Fusion (MSF) to modulate features from different scales or
within one scale. As for the modulation discriminator, we provide Global
Feature Modulation (GFM) to scale and shift intermediate feature maps.

##  3 Methods

Our goal is to design a GAN-based restoration model that takes in both the
degraded image and desired restoration information as inputs and outputs
visually pleasing restored image. The restoration information, which is equal
to the degradation information, can be regarded as sliding bars for users to
modulate during testing. We achieve such a photo-realistic modulation model by
the following approach. Given any corrupted image $I^{dis}_{i}$ distorted with
some degradations, we want to restore it to a clear image $I^{res}_{i}$, which
is close to the ground truth image $I^{gt}$. To allow image restoration for
different degradations, we accept a condition vector $z$ ($z^{k}\in[0,1]$). In
particular, each dimension of $z$ represents a certain degradation type while
its value encodes the degradation level. Therefore, our task is to find a
function $F$, such that: $F(I^{dis}_{i},z_{i})\rightarrow I^{gt}$, where
$i=1,2\ldots,N$, and $N$ indicates the number of all predefined degradations.
In addition, we introduce a conditional discriminator, that aims to
distinguish between the restored image $I^{res}_{i}$ and the ground truth
image $I^{gt}$ under condition of $z_{i}$. In particular, the discriminator
accepts $I^{res}_{i}$, $I^{gt}$, and $z_{i}$ as inputs, and outputs the two
probabilities for restored result and ground truth image, respectively.

###  3.1 Base Networks

The base networks consist of two parts: generator $G$ and discriminator $D$.
The generator $G$ accepts the input image and outputs the restored result,
while $D$ aims to discriminate the restored result from the ground truth.

Generator. The architecture of the generator $G$ is shown in Figure 3.
Basically, it is a Unet framework incorporated with Residual Blocks. $G$
consists of three scales, namely $scale_{1}$, $scale_{2}$, and $scale_{3}$.
This multi-scale architecture could help achieve better results for GAN-based
modulation compared with a plain network (Please see the experimental results
in Ablation Study). For each scale, there are two residual blocks in the left
and the other two in the right. Each residual block contains two $3\times 3$
convolution layers, and a ReLU activation function between them. In
$scale_{1}$, there are two $3\times 3$ convolution layers at two ends. As for
$scale_{2}$ and $scale_{3}$, we begin by using $2\times 2$ strided convolution
to downscale the features and end up with $2\times 2$ transposed convolution
for upscaling. The number of channels for each layer from $scale_{1}$,
$scale_{2}$, $scale_{3}$ are set to 64, 128, and 256, respectively. Besides,
each scale has a short connection that could skip its successive scale. Note
that we also add a global connection to ease the difficulty for restoration on
mild degradations.

Discriminator. The discriminator $D$ is to discriminate ground truth images
from restored images. It accepts 64$\times$64 image patches, and outputs the
predictions. The architecture of $D$ is shown in Figure 3. We follow the work
in [24], and design a discriminator that progressively down-sample the feature
maps to a feature vector. Specifically, we use several strided convolution
layers to half the sizes of feature maps each time. On the other hand, we
increase the channels of feature maps by using convolution layers with doubled
number of filters. There are 10 convolution layers in total. LeakyReLU [33]
activation is adopted between convolution layers. After we obtaining the
feature maps with 512 channels, we use global average pooling and two fully-
connected layers (regression) to output the final probability.

###  3.2 Condition networks

The condition networks mainly contain two parts, one for the generator and the
other for the discriminator, namely as $C_{G}$ and $C_{D}$, respectively. Each
condition network accepts a condition vector that encodes the degradation
information for the input image, and outputs the parameters of modulation
operations that will be used to modulate the intermediate feature in the base
networks $G$ and $D$.

First, the degradation information of each corrupted image $x$ should be
encoded into the condition vector $z$. Specifically, for each degradation
type, the corresponding degradation level is scaled to a value within range
$[0,1]$. For instance, given an input image with blur level $r=1$
($r\in[0,4]$) and noise level $\sigma=10$ ($\sigma\in[0,50]$), we could obtain
a corresponding condition vector $z=[0.25,0.20]$ by computing $[1/4,10/50]$.

The architectures of condition networks are shown in Figure 3. Each condition
network consists of several independent fully-connected layers (FC). To
generate the tuning weights for the generator, the $i$-th fully-connected
layer transforms the condition vector $z$ of the input image $x$ to the tuning
weight $w_{i}$ for the $i$-th modulation module in the generator. The
formulation can be written as follows:

$\displaystyle w_{i}=F^{w}_{i}(z),$

where $F^{w}_{i}(\cdot)$ denotes the function of $i$-th fully-connected layer.
The dimension of $w_{i}$ is the same as the channel number of the
corresponding feature maps to be modulated.

As for the $i$-th modulation module in the discriminator, we use two
independent fully-connected layers to respectively generate the parameters of
scaling and shifting operations for global feature modulation (GFM). In
particular, we have:

$\displaystyle\alpha_{i}=F^{\alpha}_{i}(z),\beta_{i}=F^{\beta}_{i}(z),$

where $F^{\alpha}_{i}(\cdot)$ and $F^{\beta}_{i}(\cdot)$ denote the $i$-th
fully-connected layers for generating parameters of scaling ($\alpha$) and
shifting ($\beta$) operators, respectively. Note that the dimensions of
$\alpha_{i}$ and $\beta_{i}$ are equal to the channel number of the
intermediate feature maps in the $i$-th modulation module.

###  3.3 Modulation in Base Networks

In this section, we introduce our modulation strategies adopted in the
generator and discriminator, which are illustrated in Figure 3. The parameters
of modulation operations are generated by the condition networks.

For the generator, we impose controls on different features from different
scales as well as within one scale. As we mentioned above, there are three
scales in the generator. In different scales, the feature maps have different
spatial sizes from high resolution to low resolution. In particular, for two
successive scales: $scale_{m}$ and $scale_{m+1}$, we have:

$\displaystyle x_{m}^{\prime}=F^{G}_{m+1}(x_{m})+x_{m},$

where $F^{G}_{m+1}(\cdot)$ denotes the transformation for $scale_{m+1}$ in the
generator $G$, $x_{m}$ is the feature maps in $scale_{m}$ before entering the
$scale_{m+1}$, and $x_{m}^{\prime}$ is the features maps obtained after the
addition of the feature maps from $scale_{m+1}$.

Then, we formulate the modulated scale fusion (MSF) on features from
$scale_{m}$ and $scale_{m+1}$:

$\displaystyle x_{m}^{\prime}=w_{i}*F^{G}_{m+1}(x_{m})+x_{m},$

where $w_{i}$ is the tuning weight generated by the $i$-th fully connected
layer of the condition network for generator. Besides, the dimension of
$w_{i}$ is the same as the number of feature maps. This controlling strategy
could dynamically change the weights for lower-resolution features.
Intuitively, when the degradation information of the input image is extremely
mild, the tuning weights will be changed close to zero, since there is no need
to receive rich contextual information from low resolutions.

Within one scale, we formulate modulated residual block (MRB) to achieve
control. To be more exact, we have:

$\displaystyle x^{\prime}_{i}=w_{i}*f(x_{i})+x_{i},$

where $f(\cdot)$ denotes the transformation in the residual branch, $x_{i}$
and $x^{\prime}_{i}$ are the original and modulated feature maps,
respectively. $w_{i}$ is the corresponding tuning weight.

In the discriminator, we adopt global feature modulation (GFM) to modulate the
intermediate feature maps. To be more specific, we scale and then shift the
intermediate features based on the condition. Practically, we perform
modulation after batch normalization. In the $i$-th modulation module, we
obtain the modulated feature maps by:

$\displaystyle x^{\prime}_{i}=\alpha_{i}*x_{i}+\beta_{i},$

where $x_{i}$ and $x^{\prime}_{i}$ are the original and modulated feature
maps. The multiplier $\alpha_{i}$ and addition operator $\beta_{i}$ are
respectively generated by two independent fully connected layers from the
condition network for discriminator.

Degradation Settings | Single Degradation | Two Degradations
---|---|---
blur |  | 0 | 0 | 2 | 4 |  |  | 1 | 1 | 2 | 2 | 4 | 4 |
noise |  | 30 | 50 | 0 | 0 |  |  | 15 | 30 | 30 | 50 | 30 | 50 |
condition vector |  | [0.0, 0.6] | [0.0, 1.0] | [0.5, 0.0] | [1.0, 0.0] |  |  | [0.15, 0.30] | [0.15, 0.60] | [0.50, 0.60] | [0.50, 1.00] | [1.00, 0.60] | [1.00, 1.00] |
Metric | Model Name |
| UGAN |  | 0.0490 | 0.0957 | 0.0484 | 0.1331 |  |  | 0.0697 | 0.1140 | 0.2068 | 0.2557 | 0.3154 | 0.3510 |
LPIPS $\downarrow$ | CUGAN |  | 0.0522 | 0.0966 | 0.0525 | 0.1463 |  |  | 0.0714 | 0.1178 | 0.2019 | 0.2498 | 0.3075 | 0.3350 |
4-7 10-15 |  |  distance $\downarrow$ | 0.0032 | 0.0009 | 0.0041 | 0.0132 |  |  | 0.0017 | 0.0038 | -0.0049 | -0.0059 | -0.0079 | -0.0160 |
| UGAN |  | 0.0603 | 0.0901 | 0.0499 | 0.1013 |  |  | 0.0716 | 0.0987 | 0.1430 | 0.1705 | 0.1994 | 0.2110 |
DISTS $\downarrow$ | CUGAN |  | 0.0639 | 0.0964 | 0.0525 | 0.1081 |  |  | 0.0727 | 0.1003 | 0.1392 | 0.1657 | 0.1982 | 0.2162 |
4-7 10-15 |  |  distance $\downarrow$ | 0.0036 | 0.0063 | 0.0026 | 0.0068 |  |  | 0.0011 | 0.0016 | -0.0038 | -0.0048 | -0.0012 | 0.0052 |

Table 1: Quantitative LPIPS/DISTS results for GAN-based modulation testing on LIVE1. $\downarrow$ means the lower the better. LPIPS/DISTS distance are bolded to stress the modulation results.  | Single Degradation | Two Degradations
---|---|---
blur | 0 | 0 | 2 | 4 | 1 | 1 | 2 | 4
noise | 30 | 50 | 0 | 0 | 15 | 30 | 50 | 50
Unet (1.364M) | 30.62 | 28.28 | 30.24 | 26.85 | 29.12 | 27.41 | 24.57 | 23.03
CUnet (1.370M) | 30.50 | 28.16 | 20.07 | 28.63 | 29.04 | 27.35 | 24.55 | 23.02
PSNR Distance $\downarrow$ | 0.12 | 0.12 | 0.17 | 0.22 | 0.08 | 0.06 | 0.02 | 0.01
CResMD (2.50M) | 30.43 | 28.06 | 30.09 | 26.53 | 29.00 | 27.28 | 24.48 | 22.95
Table 2: Quantitative PSNR results for MSE-based modulation testing on CBSD68.
The total parameter of the model is presented in the bracket. $\downarrow$
means the lower the better. PSNR distance are bolded to stress the modulation
results.

###  3.4 Interpolation between modulation models.

The proposed CUGAN can achieve modulation for photo-realistic image
restoration. To meet different user flavors, we use the deep network
interpolation strategy (DNI) [31] to achieve smooth transition between GAN and
MSE effects. Specifically, we first train a PSNR-oriented modulation model
with MSE loss, and obtain the networks $G_{MSE}$ and $C_{MSE}$. For GAN
training, we directly finetune all parameters of the PSNR-oriented modulation
model, and obtain the networks $G_{GAN}$ and $C_{GAN}$. Then, we interpolate
all the corresponding parameters of these two modulation models to obtain an
interpolated modulation model, whose parameters are:

$\displaystyle\theta^{interp}_{G}=(1-\alpha)\theta^{GAN}_{G}+\alpha\theta^{MSE}_{G},$
$\displaystyle\theta^{interp}_{C}=(1-\alpha)\theta^{GAN}_{C}+\alpha\theta^{MSE}_{C},$

where $\theta^{interp}_{G}$, $\theta^{MSE}_{G}$ and $\theta^{GAN}_{G}$ are the
parameters of the base networks $G_{interp}$, $G_{MSE}$ and $G_{GAN}$,
respectively, $\theta^{interp}_{C}$, $\theta^{PSNR}_{C}$ and
$\theta^{GAN}_{C}$ are the parameters of the condition networks $C_{interp}$,
$C_{PSNR}$ and $C_{GAN}$, respectively, and $\alpha\in[0,1]$ is the
interpolation coefficient.

For a given corrupted image $x$, we first specify an appropriate condition
vector for photo-realistic image restoration. Then, we change the coefficient
$\alpha$ from $0$ to $1$ to obtain smooth transition between GAN and MSE
effects without artifacts (see Figure 1, 4).

Figure 4: Qualitative results of Modulation for Image Restoration. For each
group of images, three rows illustrate operations of denoising, deblurring,
and interpolation, respectively. The colored-boxed images are the chosen
results of the corresponding modulation process.

##  4 Experiments

###  4.1 Implementation Details

We use DIV2K training dataset[1], which is a high-quality dataset and widely
used in image restoration. We follow the work in [17] to generate the degraded
input images. Specifically, we add Gaussian blur and Gaussian noise
sequentially to every training image in DIV2K dataset with random levels. The
covariance range of Gaussian noise is $\sigma\in[0,50]$, and the range of
kernel widths for $21\times 21$ Gaussian blur is $r\in[0,4]$. We uniformly
sample the degradation levels with a stride of 0.1 and 1 for blur and noise,
respectively. Besides, the degradation information for each degraded image is
encoded into condition vector $z$.

The training process consists of two stages. First, we pre-train the PSNR-
oriented modulation model by only using MSE loss, named as CUnet. The base
network $G$ and the corresponding condition network $C_{G}$ are jointly
trained. The learning rate is initialized as $5\times 10^{-4}$ and decayed by
a factor of 2 every $2\times 10^{5}$ iterations of update. The well-trained
model CUnet will serve as the starting point for the next GAN training. In the
second stage, the base networks, $G$ and $D$, as well as the condition
networks, $C_{G}$ and $C_{D}$ are jointly trained with the objective:

$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{percep}+0.005\mathcal{L}_{GAN}+0.01\mathcal{L}_{MSE}$

where $\mathcal{L}_{MSE}$ is the MSE loss, $\mathcal{L}_{GAN}$ denotes the
standard GAN loss, $\mathcal{L}_{percep}$ represents the perceptual loss. The
perceptual loss is the L1 distance between two activated features obtained
from VGG19-54111We use pre-trained 19-layer VGG network, where 54 indicates
features output by the 4th convolution before the 5th maxpooling layer. The
initial learning rate is set to $5\times 10^{-4}$ and will be halved at
$[50k,100k,200k,300,400k]$. The number of mini-batch is set to 16, and the
crop size is 64. For optimization, we use Adam [22] with $\beta_{1}$ = 0.9 and
$\beta_{2}$ = 0.999. Moreover, we augment the training data with random
horizontal flips and 90-degree rotations. All the models are implemented with
the PyTorch framework and trained on NVIDIA 1080Ti GPUs.

Figure 5: The qualitative comparison for blind SR task on DPED dataset [19].
The images in the first row show the modulation process of CUGAN. The
corresponding condition vectors and interpolation coefficients $\alpha$ are
denoted. The red-boxed image is the visually best result found by modulation.
The number of total parameters of each model is presented in the bracket.

###  4.2 Evaluation for Modulation Performance

Settings. LIVE1 [29] and CBSD68 [28] are chosen to evaluate our models. We
evaluate the modulation results on single degradation and two degradations.
For single degradation, we choose denoising $\sigma 30$, $\sigma 50$ and
deblurring $r2$, $r4$. For two degradations, we choose deblurring+denoising,
[$r1$, $\sigma 15$] [$r2$, $\sigma 15$], [$r2$, $\sigma 30$], [$r2$, $\sigma
50$], [$r4$, $\sigma 30$], [$r4$, $\sigma 50$]. We train two baseline models,
baseline-MSE named Unet (CUnet w/o condition network) and baseline-GAN named
UGAN (CUGAN w/o condition networks), for each of chosen degradations.
Specifically, we remove the condition networks and only use the base networks
to train these baseline models. The implementation details remain the same
with CUnet and CUGAN, except that every baseline model is only trained on one
specific degradation level.

Based on the evaluation results in PIPAL dataset[21], LPIPS [38] and DISTS [8]
have higher consistency with human ratings than NIQE, PI, and MA on images
from the GAN model. Therefore, in our experiments, PSNR is used to evaluate
the performance of PSNR-oriented models, while DISTS and LPIPS are used to
evaluate the GAN-based models. Given the ground truth images, we calculate the
PSNR of CUnet and Unet and compute the PSNR distance. Then we calculate LPIPS
and DISTS of CUGAN and UGAN and compute the LPIPS distance as well as the
DISTS distance. Lower PSNR/LPIPS/DISTS distance indicates better modulation
results.

Modulation for Image Restoration. Here, we evaluate the modulation performance
of the proposed CUGAN quantitatively and qualitatively. The ideal performance
of CUGAN and CUnet is to approach the performance of baseline models on every
degradation level as close as possible. In Table 1, we evaluate the GAN-based
modulation across various degradations on LIVE1 dataset. For the single
degradation, the LPIPS/DISTS distances are all smaller than 0.0041/0.0068.
This result demonstrates that our CUGAN could achieve modulation with high
quality. As for two degradations (blur+noise), the modulation performance of
CUGAN is even better than UGAN on several degradations: [$r2$, $\sigma 30$],
[$r2$, $\sigma 50$], [$r4$, $\sigma 30$], [$r4$, $\sigma 50$]. The
quantitative results indicate that our CUGAN has better potential and ability
on image restoration when degradations are more complex and severe. The first
two rows of Figure 4 show the qualitative results, which present the smooth
and stable transitions by modulation through the condition vector. In Table 2,
we provide the PSNR-oriented modulation evaluated on the CBSD68 dataset. The
PSNR distances for all degradations are below 0.22dB. Furthermore, CUnet
outperforms CResMD on almost all degradations. Specifically, CUnet surpasses
CResMD on deblurring $r4$ with 0.32dB, and denoising $\sigma 50$ with 0.22dB,
which indicates a significant improvement. Moreover, CResMD contains 2.5
million parameters while ours contains 1.37 million parameters. These results
prove the effectiveness and superiority of our proposed multi-dimension
modulation framework.

methods | a = 15 | a = 25 | a = 30 | a = 50
---|---|---|---|---
BM3D | 31.08 | 28.57 | 27.76 | 25.62
TNRD | 31.42 | 28.92 | 27.66 | 25.97
DnCNN-B | 31.61 | 29.16 | 28.36 | 26.23
IRCNN | 31.63 | 29.15 | 28.26 | 26.19
FFDNet | 31.63 | 29.19 | 28.26 | 26.29
CUnet | 34.53 | 31.80 | 30.89 | 28.48
Table 3: Comparison with different image denoising methods in term of PSNR on
gray CBSD68. The highest PSNR on various noise degrees are bolded.

Smooth transition between MSE and GAN effects. The proposed CUGAN provides the
default modulation results with GAN effects. Meanwhile, CUnet provides
reconstruction style–MSE effect, which is consistent with high reconstruction
accuracy (high PSNR). We apply linear interpolation for all parameters of
CUnet and CUGAN to achieve transition between GAN and MSE effect. In Figure 4,
the third row illustrates the trade-off between these two effects. After we
have obtained a photo-realistic restored image by modulation through the
condition vector, we could further smooth the image by changing an
interpolation coefficient $\alpha$ from $0$ to $1$. In the transition, users
could make the unpleasing textures/artifacts gradually disappear to obtain an
artifact-free output.

###  4.3 Comparison on Image Denoising.

To show the competitive denoising ability of the proposed model, we compare
CUnet trained for Section 4.2 with several state-of-the-art denoising methods:
BM3D[7], TNRD[6], DnCNN[34], IRCNN[35] , FFDNet[36]. We test them on gray
CBSD68 using the mean PSNR as the quantitative metric. To obtain degraded
images, gaussian noise of different levels (e.g., 15, 25, 30, 40, and 50) are
added to clean images. Then, we modulate the condition vector corresponding to
the noise level to obtain restored images. Table 3 provides the quantitative
comparison, which presents the competitive performance of our method on image
denoising. CUnet provides competitive PSNR performance comparing with those
state-of-the-art denoising methods.

Figure 6: Modulation for image restoration in real-world images. The input
image with unknown noise is from NIND [4] data set. Two rows present denoising
and debluring, respectively.

###  4.4 Modulation on Real-World Images

To further validate the generalization ability of the proposed CUGAN, we
evaluate the effectiveness of modulation on real-world images SR and
denoising.

Qualitative Results on DPED. In real-world image SR, the LR images usually
have unknown degradations, such as complicated downsampling kernel and noise.
Therefore, the non-blind GAN-based SR methods fail to produce satisfactory
results. The proposed CUGAN could deal with this situation through modulation.
In Figure 5, given an LR input from DPED [19], we first up-sample it by
bicubic interpolation. Then, we obtain the restored result by changing the
condition vector and interpolation coefficient. CUGAN has a competitive
performance compared with those winner methods [12, 20] in the AIM2019 [26]
and NTIRE2020 [25] real-world image SR challenge with fewer parameters.

Qualitative Results on NIND. As for real-world image denoising, visual results
are provided in Figure 6. The input image from a real-world denoising dataset
NIND [4] contains noticeable but unknown noise. We first use our CUGAN to
eliminate the noise by changing the condition vector from $[0.0,0.0]$ to
$[0.0,0.8]$. The restored image is noise-free but slightly over-smoothing.
Then, we sharpen this restored image by changing the first element from $0.0$
to $0.3$. After the modulation process, we can obtain a noise-free as well as
a non-blurry restored result.

###  4.5 Ablation Study

Effect of the multi-scale architecture in the generator. To verify the
effectiveness of the proposed multi-scale architecture, we train three models
CUGAN-1 (Plain Struture), CUGAN-2, and CUGAN-3 (Ours) under the same
experimental settings. Specifically, these three models have one scale, two
scales, and three scales, respectively. Each model has 12 residual blocks, and
each scale has the same number of residual blocks. The convergence curves on
LIVE1 dataset for degradations $[r1,\sigma 30]$ and $[r2,\sigma 15]$ are
presented in Figure 7. We use LPIPS to evaluate the performance. In Figure 7,
we can observe that CUGAN-3 (Ours) surpasses CUGAN-1 and CUGAN-2 by a large
margin on those degradations. Therefore, the multi-scale architecture is more
effective in GAN-based modulation compared with plain networks. More
convergence curves are in the supplementary.

Figure 7: Convergence curves achieved by different models with different
number of scale in term of LPIPS on LIVE1 dataset.

Effect of modulation in the discriminator. In this section, we investigate the
effectiveness of the modulation in the discriminator. Specifically, we train
the GAN-based modulation model without the global feature modulation (GFM) in
the discriminator and compare it with our CUGAN on different degradations
evaluated by LIVE1 dataset. The performance of these two models in terms of
LPIPS is shown in Figure 8. We can observe that our CUGAN equipped with GFM
outperforms the other one on all those degradations. It is consistent with our
assumption that the condition information is essential for the discriminator
to make proper discrimination on restored images from different degradations.

Figure 8: Results achived by CUGAN and CUGAN without GFM.

##  5 Conclusion

In this work, we propose a novel GAN-based modulation framework named CUGAN.
CUGAN consists of a generator and a discriminator, which are all controlled by
conditions vector. The key idea of the CUGAN is introducing a modulation for
users to freely control the strength of restoration and texture reconstruction
with a photo-realistic effect. We apply modulated scale fusion (MSF) and
modulated residual block (MRB) in the generator to achieve interactive
modulation. Moreover, the Global Feature Modulation (GFM) is brought into the
discriminator to ease the vanishing generator gradient. Although CUGAN could
realize modulation across multiple degradations, the modulation strategy can
be more effective and efficient. Better solutions are expected for future
research.

Acknowledgements. This work was supported in part by the Shanghai Committee of
Science and Technology, China (Grant No. 20DZ1100800), in part by the Na-
tional Natural Science Foundation of China under Grant (61906184), Science and
Technology Service Network Ini- tiative of Chinese Academy of Sciences
(KFJSTSQYZX- 092), Shenzhen Institute of Artificial Intelligence and Robotics
for Society.

## References

  * [1] Eirikur Agustsson and Radu Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 126–135, 2017.
  * [2] Yuval Bahat and Tomer Michaeli.  Explorable super resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2716–2725, 2020.
  * [3] Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor.  The 2018 pirm challenge on perceptual image super-resolution.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 0–0, 2018.
  * [4] Benoit Brummer and Christophe De Vleeschouwer.  Natural image noise dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [5] Marcel C Buhler, Andrés Romero, and Radu Timofte.  Deepsee: deep disentangled semantic explorative extreme super-resolution.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [6] Yunjin Chen and Thomas Pock.  Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration.  IEEE transactions on pattern analysis and machine intelligence, 39(6):1256–1272, 2016.
  * [7] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Image denoising by sparse 3-d transform-domain collaborative filtering.  IEEE Transactions on image processing, 16(8):2080–2095, 2007.
  * [8] Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli.  Image quality assessment: Unifying structure and texture similarity.  arXiv preprint arXiv:2004.07728, 2020.
  * [9] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  arXiv preprint arXiv:1605.08803, 2016.
  * [10] Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang.  Compression artifacts reduction by a deep convolutional network.  In Proceedings of the IEEE International Conference on Computer Vision, pages 576–584, 2015.
  * [11] Qingnan Fan, Dongdong Chen, Lu Yuan, Gang Hua, Nenghai Yu, and Baoquan Chen.  Decouple learning for parameterized image operators.  2018\.
  * [12] Manuel Fritsche, Shuhang Gu, and Radu Timofte.  Frequency separation for real-world super-resolution.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3599–3608. IEEE, 2019.
  * [13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [14] Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong.  Blind super-resolution with iterative kernel correction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [15] Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Toward convolutional blind denoising of real photographs.  2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [16] Jingwen He, Chao Dong, and Yu Qiao.  Modulating image restoration with continual levels via adaptive feature modification layers.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [17] Jingwen He, Chao Dong, and Yu Qiao.  Interactive multi-dimension modulation with dynamic controllable residual learning for image restoration, 2020.
  * [18] Jingwen He, Yihao Liu, Yu Qiao, and Chao Dong.  Conditional sequential modulation for efficient global image retouching.  In European Conference on Computer Vision, pages 679–695. Springer, 2020.
  * [19] Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool.  Dslr-quality photos on mobile devices with deep convolutional networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3277–3285, 2017.
  * [20] Xiaozhong Ji, Yun Cao, Ying Tai, Chengjie Wang, Jilin Li, and Feiyue Huang.  Real-world super-resolution via kernel estimation and noise injection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 466–467, 2020.
  * [21] Gu Jinjin, Cai Haoming, Chen Haoyu, Ye Xiaoxing, Jimmy S Ren, and Dong Chao.  Pipal: a large-scale image quality assessment dataset for perceptual image restoration.  In European Conference on Computer Vision, pages 633–651. Springer, 2020.
  * [22] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] Diederik P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  arXiv preprint arXiv:1807.03039, 2018.
  * [24] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Photo-realistic single image super-resolution using a generative adversarial network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [25] Andreas Lugmayr, Martin Danelljan, and Radu Timofte.  Ntire 2020 challenge on real-world image super-resolution: Methods and results.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 494–495, 2020.
  * [26] Andreas Lugmayr, Martin Danelljan, Radu Timofte, Manuel Fritsche, Shuhang Gu, Kuldeep Purohit, Praveen Kandula, Maitreya Suin, AN Rajagoapalan, Nam Hyung Joon, et al.  Aim 2019 challenge on real-world image super-resolution: Methods and results.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3575–3583. IEEE, 2019.
  * [27] Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Srflow: Learning the super-resolution space with normalizing flow.  In European Conference on Computer Vision, pages 715–732. Springer, 2020.
  * [28] Stefan Roth and Michael J Black.  Fields of experts: A framework for learning image priors.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 2, pages 860–867. IEEE, 2005.
  * [29] Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik.  A statistical evaluation of recent full reference image quality assessment algorithms.  IEEE Transactions on image processing, 15(11):3440–3451, 2006.
  * [30] Wei Wang, Ruiming Guo, Yapeng Tian, and Wenming Yang.  Cfsnet: Toward a controllable feature space for image restoration.  In The IEEE International Conference on Computer Vision (ICCV), October 2019.
  * [31] Xintao Wang, Ke Yu, Chao Dong, Xiaoou Tang, and Chen Change Loy.  Deep network interpolation for continuous imagery effect transition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1692–1701, 2019.
  * [32] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In The European Conference on Computer Vision Workshops (ECCVW), September 2018.
  * [33] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li.  Empirical evaluation of rectified activations in convolutional network.  arXiv preprint arXiv:1505.00853, 2015.
  * [34] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  IEEE transactions on image processing, 26(7):3142–3155, 2017.
  * [35] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang.  Learning deep cnn denoiser prior for image restoration.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3929–3938, 2017.
  * [36] Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Ffdnet: Toward a fast and flexible solution for cnn-based image denoising.  IEEE Transactions on Image Processing, 27(9):4608–4622, 2018.
  * [37] Kai Zhang, Wangmeng Zuo, and Lei Zhang.  Deep plug-and-play super-resolution for arbitrary blur kernels.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1671–1681, 2019.
  * [38] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [39] Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao.  Ranksrgan: Generative adversarial networks with ranker for image super-resolution.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.