# Wall mode dynamics and transition to chaos in magnetoconvection with a
vertical magnetic field

Matthew McCormack1  Andrei Teimurazov2  Olga Shishkina2  Moritz Linkmann1
<EMAIL_ADDRESS>1School of Mathematics and Maxwell Institute for
Mathematical Sciences, University of Edinburgh, UK2Max Planck Institute for
Dynamics and Self-Organization, 37077 Göttingen, Germany

###### Abstract

Quasistatic magnetoconvection of a low Prandtl number fluid ($\Pr=0.025)$ with
a vertical magnetic field is considered in a unit aspect ratio box with no-
slip boundaries. At high relative magnetic field strengths, given by the
Hartmann number Ha, the onset of convection is known to result from a sidewall
instability giving rise to the wall mode regime. Here, we carry out 3D direct
numerical simulations of unprecedented length to map out the parameter space
at $\mbox{\rm{Ha}}=200,500,1000$, varying the Rayleigh number (Ra) between
$6\times 10^{5}\lesssim\mbox{\rm{Ra}}\lesssim 5\times 10^{8}$. We track the
development of stable equilibria produced by this primary instability,
identify bifurcations leading to limit cycles, and eventually to chaotic
dynamics. At $\mbox{\rm{Ha}}=200$, the steady wall mode solution undergoes a
symmetry-breaking bifurcation producing a state featuring a coexistence
between wall modes and a large-scale roll in the centre of the domain which
persists to higher Ra. However, under a stronger magnetic field at
$\mbox{\rm{Ha}}=1000$, the steady wall mode solution undergoes a Hopf
bifurcation producing a limit cycle which further develops to solutions that
shadow an orbit homoclinic to a saddle point. Upon a further increase in Ra,
the system undergoes a subsequent symmetry break producing a coexistence
between wall modes and a large-scale roll, although the large-scale roll
exists only for a small range of Ra, and chaotic dynamics primarily arise due
to a mixture of chaotic wall mode dynamics and arrays of cellular structures.

###### keywords:

Enter online.

##  1 Introduction

Convective flows of electrically conducting fluids influenced by magnetic
fields are readily found throughout nature and in many industrial processes.
Examples of such naturally occurring flows arise in the study of stellar
convection zones and liquid metal planetary cores (Jones, 2011), with
industrial applications including liquid metal batteries, casting, semi-
conductor crystal growth as well as liquid-metal blanket cooling systems for
nuclear fusion reactors (Davidson, 1999). Despite the physical relevance of
such systems, our understanding of magnetoconvection has been limited due to
numerous experimental and computational difficulties (Schumacher, 2022).
Experiments, which often require the use of opaque liquid metals, are unable
to use commonly available optical imaging techniques, and numerical studies
usually require substantial computational resource. To mitigate these
challenges, either greatly simplified systems are used or the simulations are
evolved only for short times. The result is that little is known about the
often slowly evolving spatiotemporal dynamics in such systems, especially at
high magnetic field strengths.

The onset of magnetoconvection with a vertical magnetic field has been studied
in a few isolated scenarios through linear stability analysis: in the case of
an infinite/periodic plane layer by Chandrasekhar (1961); and in the case of a
semi-infinite sidewall by Busse (2008). Such analysis reveals that the
sidewalls in a system are responsible for the onset of convection at high
relative magnetic field strengths (typically characterised by the Hartmann
number Ha), particularly relevant to industrial processes that often occur in
closed vessels. Indeed, an analytic-numerical hybrid analysis performed by
Houchens et al. (2002) for the special case of a cylindrical geometry showed
that sidewalls were responsible for the onset, producing thin convective
layers attached to these sidewalls, commonly referred to as _wall modes_.
However, knowledge about nonlinear effects and the later stages of transition
to turbulence are still limited.

Wall modes in quasistatic magnetoconvection with a vertical magnetic field
were first examined in the full system numerically by Liu et al. (2018) who
conducted direct numerical simulations in a box of aspect ratio $\Gamma=4$ up
to $\mbox{\rm{Ha}}=2000$ with no-slip boundaries, confirming that onset
occurred below the linear stability threshold of the infinite plane layer.
Wall modes were seen to have a 3D spatial structure, featuring thin rolls
pressed against the sidewalls in an alternating pattern of positive and
negative rotation. Thus, between each set of rolls lay alternating structures
with positive and negative vertical velocities which were additionally seen to
have protrusions that extended into the bulk of the domain. At increased
levels of thermal driving (typically characterised by the Rayleigh number Ra),
more chaotic cellular style regimes were observed, although details of the
transition were not characterised.

Although hints of wall modes were present in the experiments of Cioni et al.
(2000), wall modes were first confirmed experimentally in a cylinder by Zürner
et al. (2020), who additionally characterised the cellular regime more
carefully, with various numbers of cells being observed at different regions
of the $(\mbox{\rm{Ra}},\mbox{\rm{Ha}})$ parameter space. Further simulations
in a cylinder were performed by Akhmedagaev et al. (2020) as well as more
recently by Xu et al. (2023) who combined new experimental and numerical
results. From the current data, it appears that the wall mode protrusions grow
with increased Ra, extending towards the bulk of the domain exhibiting quasi-
steady dynamics. Once the critical Ra of the bulk onset is reached, i.e. the
onset in the infinite plane layer (Chandrasekhar, 1961), the cellular regime
ensues producing a more chaotic variation in system observables such as the
dimensionless heat transport (Nusselt number Nu). For $\mbox{\rm{Ha}}<300$ at
mostly higher Ra, Zürner et al. (2020) observed large-scale rolls, i.e. the
large-scale circulation (LSC) that is commonly found in classical Rayleigh-
Bénard convection (RBC), and thus it is assumed that at sufficiently large Ra
magnetoconvection will approach more familiar solutions of RBC.

In summary, although some information about the transition to turbulence is
known in magnetoconvection, many questions are still open, mostly concerning
the various dynamics admissible by this system. Namely, it is currently not
well understood how the transition between steady wall mode solutions and the
more chaotic cellular regime takes place or potentially what states exist in
between. In particular, it is relevant to try and characterise the series of
bifurcations that occur upon increased Ra in this system and how heavily this
depends on the relative magnetic field strength Ha. To address this in the
present study, we have performed an array of long numerical simulations in an
aspect ratio $\Gamma=1$ box, in many instances on the order of 1000 free-fall
times, to study this transition and the associated spatiotemporal dynamics.

Wall modes also occur in rotating Rayleigh-Bénard convection (RRBC). There are
similarities between wall modes in magnetoconvection and in RRBC, especially
near onset, where both types of wall modes have a similar two-layer structure.
This is likely attributed to the extensive similarities in the linear theory
between the two systems (Busse, 2008; Herrmann & Busse, 1993). However, a
number of differences exist between the two systems in terms of their
secondary instabilities and further nonlinear effects. In the rotating case, a
Hopf bifurcation leads to an azimuthal precession of the wall modes (Ecke et
al., 1992), which is not seen to occur in magnetoconvection. Concerning the
spatial structure, striations found in the RRBC wall modes at higher
supercriticalities, as observed by Ecke et al. (2022), have not been found in
magnetoconvection. With that being said, some nonlinear effects do have
similarities between the two systems such as wall mode protrusions extending
into the bulk, sometimes identified as plume-like jets in the rotating case.
It is yet to be seen if somewhat similar dynamics to those presented here
could exists in some regimes of RRBC.

##  2 Formulation

We consider a three-dimensional flow of electrically conducting fluid driven
by an imposed vertical temperature difference between the top and bottom of
the domain, and in the presence of a vertical magnetic field
$\boldsymbol{B}=B_{0}\boldsymbol{e}_{z}$. Under the quasistatic approximation,
that is, for magnetic Reynolds number $\mbox{\rm{Rm}}=U\ell/\eta\ll 1$ and
magnetic Prandtl number $\mbox{\rm{Pm}}=\nu/\eta\ll 1$, where $U$ and $\ell$
are characteristic velocity and length scales, $\eta$ is the magnetic
diffusivity, and $\nu$ is the kinematic viscosity, and the Oberbeck-
Boussinessq approximation, the velocity field $\boldsymbol{u}$ and the
temperature field $T$ evolve from suitable initial conditions according to the
following equations,

$\displaystyle\partial_{t}\boldsymbol{u}+\boldsymbol{u}\bcdot\bnabla\boldsymbol{u}+\bnabla
p$
$\displaystyle=\sqrt{\frac{\Pr}{\mbox{\rm{Ra}}}}\big{[}\nabla^{2}\boldsymbol{u}+\mbox{\rm{Ha}}^{2}(\boldsymbol{j}\times\boldsymbol{e}_{z})\big{]}+T\boldsymbol{e}_{z},$
(1a) $\displaystyle\partial_{t}T+\boldsymbol{u}\bcdot\bnabla T$
$\displaystyle=\frac{1}{\sqrt{\mbox{\rm{Ra}}\Pr}}\nabla^{2}T,$ (1b)
$\displaystyle\bnabla\bcdot\boldsymbol{u}=0,\quad\boldsymbol{j}=-\bnabla\phi$
$\displaystyle+(\boldsymbol{u}\times\boldsymbol{e}_{z}),\quad\nabla^{2}\phi=\bnabla\bcdot(\boldsymbol{u}\times\boldsymbol{e}_{z}),$
(1c)

where $p$ is the kinematic pressure, $\boldsymbol{j}$ is the electric current
density, $\phi$ is the electric field potential, and $\boldsymbol{e}_{z}$ is
the unit vector that points vertically, opposed to gravity. Here, variables
have been made dimensionless by using the container height $H$, and the free-
fall velocity $u_{f}=(\alpha gH\delta T)^{1/2}$ to construct length, velocity
and time scales, while the temperature difference between the bottom and top
plates $\delta T=T_{+}-T_{-}$, and the applied magnetic field strength $B_{0}$
have been used to construct the dimensionless temperature and magnetic field
strength respectively. In turn, the control parameters of the governing
equations (equations 1) are the Rayleigh number $\mbox{\rm{Ra}}={\alpha
g\delta TH^{3}}/{\kappa\nu}$, the Prandtl number $\Pr={\nu}/{\kappa}$, and the
Hartmann number $\mbox{\rm{Ha}}=B_{0}H({{\sigma}/{\rho\nu}})^{1/2}$, where
$\sigma$ is the electrical conductivity, $\rho$ is the mass density, $\alpha$
is the thermal expansion coefficient, $g$ is the acceleration due to gravity,
and $\kappa$ is the thermal diffusivity. No slip boundary conditions are
applied to all boundaries, as well as constant temperatures $T_{+}$ and
$T_{-}$ applied to the bottom $(z=0)$ and top $(z=H)$ plates respectively. The
domain is equipped with adiabatic side walls $\partial
T/\partial\boldsymbol{n}=0$, where $\boldsymbol{n}$ is the vector orthogonal
to the surface, and electrically insulating boundaries
$\partial\phi/\partial\boldsymbol{n}=0$.

These equations have been solved numerically at a range of Ra and Ha for a
fluid with $\Pr=0.025$ (such as the GaInSn alloy) in a cubic domain of unit
aspect ratio, i.e. $\Gamma=L/H=W/H=1$ where $L$ and $W$ are the length and
width of the domain perpendicular to $\boldsymbol{e}_{z}$ respectively. This
has been done using the direct numerical solver goldfish (Reiter et al.,
2022), which has been widely used in previous studies of convective flows.
This version (Teimurazov et al., 2023) uses a fourth-order finite-volume
discretisation on staggered grids, and a third-order Runge-Kutta time marching
scheme which has been extended to simulate magnetoconvective flows using a
consistent and conservative scheme (Ni & Li, 2012). A compromise had to be
made in terms of the grid resolution due to the high computational cost. The
flows are resolved on non-uniform grids with $220^{2}$ points in the cross-
plane direction and with either 300 or 350 points in the vertical direction,
sufficient to resolve the Hartmann, Shercliff and thermal boundary layers,
with at least 5, 15 and 20 grid points within each layer, respectively. The
ratio between the largest finite volume cell length and the Kolmogorov scale,
calculated from the mean dissipation rate, is less than 3 in the worst case,
with the vast majority of simulations being less than 2. Changes in grid
resolution were not seen to qualitatively change the observed dynamics. Most
simulations have been initialised from a lower resolution simulation with
$\mbox{\rm{Ha}}=0$ with the magnetic field strength being progressively
increased, although a few of the solutions have been continued from one
another. In this sense, we try here to broadly classify the states that exist
in this flow rather than tracking a detailed route to turbulence.

##  3 Numerical Results

###  3.1 Comparison to linear stability theory

Figure 1: Studied parameter space given by the Hartmann (Ha) and Rayleigh (Ra)
numbers compared to the critical Rayleigh number for onset of the bulk
$\mbox{\rm{Ra}}_{c,b}$, and wall modes $\mbox{\rm{Ra}}_{c,w}$.

A number of linear stability results are relevant for the considered system.
Each of these results are derived from perturbations to a static flow field
$\boldsymbol{u}=0$, and linear temperature profile. The dispersion relation
for the case of an infinite plane layer/periodic sidewalls and no-slip upper
and lower boundaries was derived by Chandrasekhar (1961). To apply this result
to our cube $(\Gamma=1)$, the dispersion relation is minimised for each value
of Ha over the discrete horizontal wavenumbers admitted by the domain. This
results in the critical Rayleigh number in the bulk $\mbox{\rm{Ra}}_{c,b}$
which is plotted in figure 1. At the values of Ha considered, the wavelength
of the most unstable mode in the infinite plane layer is smaller than the size
of our domain, and continues to decrease with Ha. Thus, the bulk onset is seen
to be nearly identical in our closed geometry compared to the infinite plane
layer.

Additionally, the influence of no-slip sidewalls is seen to give rise to
another linear instability which leads to the formation of thin convective
zones close to these sidewalls, known as wall modes (Busse, 2008; Liu et al.,
2018). An asymptotic linear stability result to second order, derived by Busse
(2008) for a semi-infinite side-wall bounded domain, gives a relation for the
critical Rayleigh number for the side walls denoted by $\mbox{\rm{Ra}}_{c,w}$,
which has also been plotted in figure 1. The most unstable mode in this case
is seen to spatially decay away from the wall with $\mbox{\rm{Ha}}^{-1/2}$ at
leading order, so becomes increasingly thin compared to our domain with
increased magnetic field strength.

In both cases, the linear stability results have been derived for the full
magnetohydrodynamic (MHD) system. However, the results are seen in both cases
to be independent of the Prandtl number $\Pr$, and the magnetic Prandtl number
Pm. Thus, both results hold in the quasistatic MHD case which we examine here.
The critical Rayleigh numbers for both bulk and wall mode onset have been
compared to an overview of our computational data set, shown by the blue
markers in figure 1. Differences in the velocity boundary conditions are
expected to have a small effect on the value of $\mbox{\rm{Ra}}_{c}$ in both
cases, with free-slip boundary conditions typically reducing the value of the
critical Rayleigh number.

Notably, in the regime covered by our data $(200\leq\mbox{\rm{Ha}}\leq 1000)$,
the minimum critical Rayleigh number from linear theory is that of the wall
mode onset $\mbox{\rm{Ra}}_{c,w}$ which is lower than that of the bulk onset,
and thus, we expect the onset in our system to occur in the form of a wall
mode instability. At low values of Ra close to onset we observe equilibrium
solutions showing clear evidence of wall modes at each value of Ha. Our data
set at $\mbox{\rm{Ha}}=500$ has been extended to below the predicted wall mode
onset of Busse (2008) to as low as $\mbox{\rm{Ra}}=6\times 10^{5}$.

Figure 2: Overview of the equilibrium solutions at $\textrm{Ha}=500$.
$(a$-$d)$ Vertical velocity isosurfaces $u_{z}=\pm 7\times 10^{-3}$
(pink/blue) and instantaneous streamlines (black) from the top view. $(e)$
Comparison of data points to linear theory.

Comparing this to the linear stability results (figure 2$(a,e)$), the onset of
the wall mode regime appears to be lower than the asymptotic solution derived
in the semi-infinite sidewall case by Busse (2008) with the convective part of
the heat transport being approximately 3% of the conductive heat transport at
$\mbox{\rm{Ra}}=6\times 10^{5}$ (i.e. $\mbox{{Nu}}=1.03$). This indicates that
the presence of multiple sidewalls has a significant effect on the critical
Rayleigh number, at least at this value of Ha. Although the onset of
convection occurs beneath $\mbox{\rm{Ra}}_{c,w}$, the onset here is in line
with linear theory, with exponential growth being observed from a perturbed
laminar state at $\mbox{\rm{Ra}}=8\times 10^{5}$ in figure 3$(a)$, and there
is no evidence to suggest a subcritical transition. In some instances, we
additionally observe equilibrium solutions above the bulk onset
$\mbox{\rm{Ra}}_{c,b}$, as shown by the equilibrium solution at
$\mbox{\rm{Ra}}=3\times 10^{6}$ (figure 2$(d,e)$). In this case, it is likely
that perturbations about the basic state used in the linear stability analysis
are not representative of perturbations about a wall-mode dominated flow field
and thus the analysis of Chandrasekhar (1961) is not valid in this regime.

An overview of the equilibrium solutions found at $\mbox{\rm{Ha}}=500$ are
shown in figure 2, revealing clear wall mode structures very similar to those
found in Liu et al. (2018). At $\mbox{\rm{Ra}}=6\times 10^{5}$, convective
zones are nearly entirely restricted to the near sidewall region, with
individual wall modes pinned to each of the four corners from the top view
(figure 2$(a)$) and the velocity isosurfaces being near uniform in the
vertical direction. On each flat sidewall, a roll exists between each wall
mode structure, confined to this near-wall region. Small vertical counter-flow
structures sit closer to the centre of the domain, in front of each wall mode.
As Ra is increased, regions of high vertical velocity begin to protrude from
the centre of each wall mode, extending into the domain. This results in a
bending of the counter-flow structures at $\mbox{\rm{Ra}}=8\times 10^{5}$
(figure 2$(b)$) subsequently splitting into two distinct counter flow
structures by $\mbox{\rm{Ra}}=2\times 10^{6}$ (figure 2$(c)$). At this point,
the wall mode protrusions have extended into the central part of the domain
which had near zero velocity at $\mbox{\rm{Ra}}=6\times 10^{5}$. At
$\mbox{\rm{Ra}}=3\times 10^{6}$ (figure 2$(d)$), the protrusions have fully
extended into the central part of the domain, and show clear interaction with
the wall mode in the opposing corner.

It currently appears that nonlinear effects have a significant influence on
the final structure of these wall mode solutions. As can be seen from the time
evolution of Nu from the laminar state perturbed by a cross-flow for
$\mbox{\rm{Ha}}=500$ and $\mbox{\rm{Ra}}=8\times 10^{5}$ in figure 3(a), the
Nusselt number grows exponentially after a short transient. Hence, the
snapshot shown in figure 3(b), taken during this phase of exponential growth,
corresponds to the most unstable mode. In comparison with the nonlinear
solution shown in figure 2(b), we observe a slightly different symmetry in the
velocity fields. Concerning the heat transport, we currently observe a
deviation from exponential growth at Nu $\approx 1.03$ with the final
equilibrium solution having Nu $\approx 1.28$.

Figure 3: $(a)$ Temporal evolution of the dimensionless heat transport Nu from
the laminar state perturbed by a cross-flow at $\mbox{\rm{Ha}}=500$,
$\mbox{\rm{Ra}}=8\times 10^{5}$. Exponential growth
$\mbox{{Nu}}-1\sim\exp(\lambda t)$ is seen, with $\lambda\approx 0.07889$, for
a short time before nonlinear growth is observed. $(b)$ Vertical velocity
isosurfaces $u_{z}=\pm 4\times 10^{-4}$ (pink/blue) and instantaneous
streamlines (black) from the top view showing the exponentially growing mode.

Before discussing secondary instabilities, and in view of the aforementioned
similarities between wall moded in magnetoconvection and RRBC, we briefly
discuss linear stability for RRBC, where more refined analyses have been
carried out (Liao et al., 2006; Zhang & Liao, 2009). Asymptotic analyses for
no-slip and stress-free side walls differ at second order in the expansions
with the critical Rayleigh number at the onset of near-wall convection is
$\mbox{\rm{Ra}}_{w}\approx 31.8\mbox{\rm{Ek}}^{-1}+46.49\mbox{\rm{Ek}}^{-2/3}$
in the no-slip case, where Ek is the Ekman number, and
$\mbox{\rm{Ra}}_{w}\approx
31.8\mbox{\rm{Ek}}^{-1}-25.25\mbox{\rm{Ek}}^{-2/3}$, in the free-slip case for
duct (two rigid walls and one periodic direction) (Liao et al., 2006) and
cylindrical geometries (Zhang & Liao, 2009). In the former case, the
asymptotic solutions were compared against a numerical linear stability
analysis, assuming two unstable modes, one with retrograde and one with
prograde rotation. For the stress-free case, the asymptotics overestimate the
critical Rayleigh number for $\mbox{\rm{Ek}}\geqslant 10^{-2}$, followed by an
underestimation in the range $10^{-2}\leqslant\mbox{\rm{Ek}}\leqslant
10^{-6}$. A similar trend is observed for the no-slip case, with
overestimation for $\mbox{\rm{Ek}}\geqslant 10^{-2}$, followed by
underestimation in the range $10^{-2}\leqslant\mbox{\rm{Ek}}\leqslant
10^{-4}$. Apart the asymptotic result reached at lower Ek, the
underestimations are less pronounced in the no slip case. Further differences
may arise for closed rectangular systems.

###  3.2 Beyond the primary instability

We now turn our attention to the dynamics of the flow past the primary wall
mode instability. It is observed that distinct transition processes occur at
varying Ha and thus the strength of the magnetic field plays a pivotal role in
the transition to turbulence in this system.

At all values of Ha considered, the basic wall mode equilibrium solution
features a 4-fold symmetry from the top view (90° rotation and flip), and a
vertical symmetry about the midplane rotated 180° about $\boldsymbol{e}_{z}$
(see figures 2, 5$(a)$, 8$(a)$). More precisely, the discrete rotational
symmetries of the vertical velocity field form a group isomorphic to
$\mathbb{Z}_{4}$ generated by the following symmetry operation,

$\mathbb{Z}_{4}:\>u_{z}(x,y,z)=-\mathcal{R}_{\pi/2}u_{z}(x,y,-z),$ (2)

which in turn produces the additional symmetry observed
$(u_{z}(x,y,z)=\mathcal{R}_{\pi}u_{z}(x,y,z))$, where $\mathcal{R}_{\beta}$
represents a rotation of $\beta$ about $\boldsymbol{e}_{z}$, and the
coordinates $\boldsymbol{x}=(x,y,z)$ have their origin at the centre of the
domain. Another common symmetry observed in the vertical velocity fields is a
similar 2-fold symmetry seen in various states whose rotational symmetry forms
a group isomorphic to $\mathbb{Z}_{2}$ generated by the following symmetry
operation,

$\mathbb{Z}_{2}:\>u_{z}(x,y,z)=-\mathcal{R}_{\pi}u_{z}(x,y,-z).$ (3)

Figure 4: Overview of the transition at $\textrm{Ha}=200$. $(a$-$h)$ Vertical
velocity isosurfaces $u_{z}=\pm 0.1$ (pink/blue) and instantaneous streamlines
(black) from the top view. $(e)$ Comparison of data points to linear theory

Figure 5: Phase portrait consisting of thermal energy $E_{T}$, vertical
kinetic energy $E_{w}$ and cross-plane kinetic energy $E_{u}+E_{v}$, and
dimensionless heat transport Nu time series data for each value of Ra
considered at $\mbox{\rm{Ha}}=200$. Here the colours highlight different parts
of the Nu time series for easy comparison with the corresponding phase
portrait.

At $\mbox{\rm{Ha}}=200$, wall mode equilibrium solutions are observed at
$\mbox{\rm{Ra}}=4\times 10^{5}$ to $\mbox{\rm{Ra}}=6\times 10^{5}$ (see phase
portrait and time series data in figure 5$(a)$) with a $\mathbb{Z}_{4}$
symmetry (figure 5$(a)$). With increased Ra, the wall mode protrusions extend
into the centre of the domain, similar to the equilibria at
$\mbox{\rm{Ha}}=500$ (§3.1). However, by $\mbox{\rm{Ra}}=1\times 10^{6}$, a
symmetry-breaking bifurcation has occurred resulting in a state with a
$\mathbb{Z}_{2}$ symmetry (figure 5$(b)$), which produces a periodic solution
(figure 5$(b)$) with a period $\mathcal{T}=82.8$, and marks the breakdown of
the rolls near the side walls which existed between the wall modes.
Physically, the wall modes at $\mbox{\rm{Ra}}=1\times 10^{6}$ consist of two
vortices, one near the corner of the domain and one near the outer tip of the
wall mode, the outermost of which is seen to contract and extend away from and
into the centre of the domain, periodically switching between being biased to
one side of the diagonal and then the other (animated versions of figures 5
and 8 can be found in the supplementary materials). Of note, is that wall
modes at $\mbox{\rm{Ra}}=1\times 10^{6}$ coexist with a large-scale roll in
the centre of the domain, orientated about the diagonal as shown in figure 6.
At these values of Ra, the orientation of the large-scale roll is seen to
oscillate back and forth with the wall modes as they contract and expand.
However, as Ra is increased to $2\times 10^{6}$, the large-scale roll begins
to dominate the dynamics, maintaining a fixed diagonal orientation and pinning
the wall modes into the position shown in figure 5$(c)$. The wall modes, now
pinned close to the corners, produce higher frequency oscillations due to a
rapid churning of the wall mode vortex closest to the wall. The combined
effect of these processes is a weakly aperiodic signal shown in figure 5$(c)$.
Upon further increase of Ra, the $\mathbb{Z}_{2}$ symmetry is seen to break,
and aperiodic solutions are observed at $\mbox{\rm{Ra}}=5\times 10^{6}$ and
$\mbox{\rm{Ra}}=1\times 10^{7}$ (figure 5$(d)$). These solutions exhibit
chaotic behaviour in the velocity fields and are seen to contain more small-
scale structures (figure 5$(d)$), which appear to be associated with more
chaotic wall mode dynamics that invade the bulk of the domain. Although masked
by the small-scale chaotic dynamics in the instantaneous velocity fields,
clear evidence of a large-scale roll can still be seen in the mean flow at
these higher Rayleigh numbers (see figure 9$(a)$ for example).

Figure 6: Vertical velocity isosurfaces $u_{z}=\pm 0.1$ (pink/blue) and
instantaneous streamlines (black) showing coexistence of wall modes and large-
scale circulation at $Ha=200$, $Ra=10^{6}$ from $(a)$ top view, $(b)$ 3d view
and $(c)$ angled side view.

Figure 7: Overview of the transition at $\textrm{Ha}=1000$. Vertical velocity
isosurfaces $(a\textrm{-}e)$ $u_{z}=\pm 0.01$, $(f\textrm{-}g)$ $u_{z}=\pm
0.025$ (pink/blue) and instantaneous streamlines (black) from the top view.
$(i)$ Comparison of data points to linear theory.

Figure 8: Phase portrait consisting of thermal energy $E_{T}$, vertical
kinetic energy $E_{w}$ and cross-plane kinetic energy $E_{u}+E_{v}$, and
dimensionless heat transport Nu time series data for each value of Ra
considered at $\mbox{\rm{Ha}}=1000$. Here the colours highlight different
parts of the Nu time series for easy comparison with the corresponding phase
portrait.

At $\mbox{\rm{Ha}}=1000$, the wall mode equilibrium solution appears to have a
similar structure to those at lower magnetic field strengths, featuring the
same $\mathbb{Z}_{4}$ symmetry, although the characteristic length scale of
the wall modes is affected by the magnetic field. The result of this is that
the wall-mode protrusions become shorter and thinner compared to the length
scale along the sidewall (compare figure 8$(a)$ to figure 2$(c)$ for example).
This is consistent with the linear theory which suggests that the most
unstable sidewall mode spatially decays into the bulk with
$\mbox{\rm{Ha}}^{-1/2}$ at leading order, thus producing a thinner sidewall
layer at higher Ha (Busse, 2008). At increased Ra, the equilibrium solution
undergoes a Hopf bifurcation, resulting in the simple limit cycle oscillation
($\mathcal{T}=13.8$) observed at $\mbox{\rm{Ra}}=7\times 10^{6}$ (figure
8$(b)$) which maintains the $\mathbb{Z}_{4}$ symmetry. This periodic signal
arises due to an oscillation of the wall mode protrusions themselves which
synchronously flap back and forth across the diagonal (figure 8$(b)$). As Ra
is further increased to $\mbox{\rm{Ra}}=8\times 10^{6}$, the amplitude of the
wall mode nose oscillations increases as the protrusions extend further into
the domain (figure 8$(c)$), appearing to violently crash into the sidewall
rolls between the wall modes, which modulates the amplitude of the flapping,
damping the wall mode protrusions to a straighter position, closer to the
equilibrium solution seen at $\mbox{\rm{Ra}}=5\times 10^{6}$. The protrusions
then undergo an instability and quickly return to a large amplitude flapping
motion, which damps again, continuing the cycle. Autocorrelation of the Nu
time series exhibits decays that scale as $\exp({-\lambda\tau})$ with
$\lambda\approx 0.041$ for timelag $\tau$. The mathematical interpretation of
this behaviour is made clearer by the phase portrait and time series data in
figure 8$(c)$ where the solution appears to decay along the stable manifold of
a saddle point before being ejected along the unstable manifold of the saddle
i.e. the solution appears to shadow an orbit homoclinic to a saddle focus with
the saddle point having a similar flow structure to that of the equilibrium
found at $\mbox{\rm{Ra}}=5\times 10^{6}$. This behaviour is reminiscent of the
Shilnikov phenomenon (Wiggins, 1988), arising in the third-order system first
studied by Shilnikov (1965). By $\mbox{\rm{Ra}}=9\times 10^{6}$, the ejections
are more frequent (figure 8$(d)$), with Nu autocorrelations decaying with
$\lambda\approx 0.078$, and the protrusions exhibit stronger and more complex
flapping behaviour, as seen in figure 8$(d)$. At $\mbox{\rm{Ra}}=1\times
10^{7}$, the flow is observed undergoing a break in symmetry, resulting in a
bursting limit cycle with period $\mathcal{T}=147.2$ (figure 8$(e)$). The
velocity field reveals that three wall mode structures move across opposing
sidewalls, which when coinciding with the end sidewall produces a large flap
from the end wall mode in opposing corners, as shown in figure 8$(e)$, which
then decays to the original flow field. This large flap produces a large
increase in the dimensionless heat transport resulting in the bursting
behaviour observed in the Nusselt number time series data shown in figure
8$(e)$. At $\mbox{\rm{Ra}}=1.5\times 10^{7}$, the $\mathbb{Z}_{4}$ symmetry
has fully broken and has been replaced by the $\mathbb{Z}_{2}$ symmetry. This
state sustains large periodic wall mode flapping in two opposing corners which
converges to the limit cycle ($\mathcal{T}=31.2$) shown in figure 8$(f)$.
Interestingly, this state also features steady convection in the bulk which
occurs at a Rayleigh number higher than $\mbox{\rm{Ra}}_{c,b}$ (figure
8$(i)$), and features a thin roll, aligned along the diagonal, in the centre
of the domain (figure 8$(f)$). Additionally, as Ra is increased further to
$2\times 10^{7}$, this large-scale roll appears in the mean flow field (figure
9$(b)$), and the bulk is dominated by vertical flow structures which interact
heavily with now chaotically flapping wall mode protrusions which extend far
into the domain and shed large-scale structures into the bulk as shown at
$\mbox{\rm{Ra}}=2\times 10^{7}$ in figure 8$(g)$. However, by
$\mbox{\rm{Ra}}=5\times 10^{7}$, no evidence of such large-scale roll is
observed in the instantaneous or mean flow (figure 9$(c)$) unlike the
$\mbox{\rm{Ha}}=200$ case. Chaotic solutions ensue, with a combination of
unsteady bulk dynamics in the form of interconnected columnar structures and
the remnants of more chaotic wall mode dynamics in the corners similar to the
cellular regimes observed previously (Liu et al., 2018; Zürner et al., 2020;
Akhmedagaev et al., 2020; Xu et al., 2023) . The destruction of the large-
scale roll at $\mbox{\rm{Ha}}=1000$ has occurred at most by
$\mbox{\rm{Ra}}/\mbox{\rm{Ra}}_{c,b}\approx 4.84$ ($\mbox{\rm{Ra}}=5\times
10^{7}$), whereas at $\mbox{\rm{Ha}}=200$, the large-scale roll as been seen
to persist in our simulations up to at least
$\mbox{\rm{Ra}}/\mbox{\rm{Ra}}_{c,b}\approx 218.36$ ($\mbox{\rm{Ra}}=1\times
10^{8}$). This shows the clear impact of the magnetic field on the large-scale
circulation.

The transition process near onset at $\mbox{\rm{Ha}}=500$ is seen to be
qualitatively more similar to the transition at $\mbox{\rm{Ha}}=1000$, with a
Hopf bifurcation leading to a limit cycle with a $\mathbb{Z}_{4}$ symmetry,
which subsequently undergoes a break in symmetry to a $\mathbb{Z}_{2}$ state.
However, the highest Ra simulations ($\mbox{\rm{Ra}}=5\times 10^{8}$) carried
out at $\mbox{\rm{Ha}}=500$ show a large-scale roll in the mean flow like in
the $\mbox{\rm{Ha}}=200$ case at high supercriticalities, and different from
$\mbox{\rm{Ha}}=1000$ case where a cellular regime is observed at high Ra.

Figure 9: Mean vertical velocity isosurfaces $(a)$ $u_{z}=\pm 0.1$, $(b,c)$
$u_{z}=\pm 0.05$ (pink/blue) and streamlines of mean flow (black) from the
angled side view.

##  4 Conclusions

We have studied three-dimensional quasistatic magnetoconvection for Hartmann
numbers between $200\leq\mbox{\rm{Ha}}\leq 1000$ in a cube $(\Gamma=1)$ with
no-slip boundaries, tracking the base state to states exhibiting chaotic
multi-scale dynamics. In line with linear theory in semi-infinite domains, we
confirm that the primary instability in this system comes from the sidewalls,
giving rise to an equilibrium wall mode solution for all values of Ha
considered. However, the later stages of the transition to chaos past this
primary instability are seen to vary with Ha. At $\mbox{\rm{Ha}}=200$, the
basic wall mode state undergoes a symmetry breaking from a $\mathbb{Z}_{4}$ to
a $\mathbb{Z}_{2}$ symmetry, giving rise to limit cycles involving a large-
scale roll which re-orientates in time with wall mode oscillations. On
increased Rayleigh numbers, the large-scale roll is seen to persist
accompanied by more chaotic dynamics from the wall modes. However, at
$\mbox{\rm{Ha}}=1000$, the equilibrium wall mode state undergoes a Hopf
bifurcation resulting in limit cycles involving synchronous oscillations of
the wall mode protrusions, which further evolves to states that shadow an
orbit homoclinic to a saddle focus involving more complex oscillatory
behaviour. The system then undergoes a $\mathbb{Z}_{4}$ to $\mathbb{Z}_{2}$
symmetry breaking bifurcation producing limit cycles featuring a large-scale
roll which is dominated by more vigorous wall mode oscillations, which
subsequently begin to shed large-scale structures into the bulk. At higher Ra
this develops to form the cellular regime, and no large-scale roll persists.

The degree to which the results of this study generalise to other geometries,
aspect ratios and magnetic boundary conditions is not currently clear. In
larger aspect ratio boxes, it is expected that similar dynamics could occur,
and indeed some evidence of potential wall mode nose flapping can be seen in
some flow snapshots in the $\Gamma=4$ system of Liu et al. (2018), although
short run times of these simulations mean that this cannot be confirmed, and
further, whether such oscillations are synchronised between the larger number
of wall modes. In cylindrical containers there is also some evidence of
extended wall mode protrusions and potential dynamics in the presented flow
snapshots of Zürner et al. (2020) and Xu et al. (2023), although the
continuous symmetry in this geometry appears to result in less strict wall
mode symmetries in some instances and dynamics such as those described here
have not yet been identified. It is currently unclear whether wall modes
persist in more generalised geometries. However, wall modes in rotating
convection have been seen to persist independently of the geometry in both
numerical and experimental studies (Favier & Knobloch, 2020; Ecke et al.,
2022). With respect to magnetic boundary conditions, we expect that the
results here would generalise to low but finitely conducting boundaries, such
as those found in experiments. The reason being that wall modes arise due to a
suppression of the Lorentz force near the boundary, largely due to
electrically insulated boundaries. However, an investigation into the
subsequent bifurcations from the basic wall mode state and dynamics of the
ensuing states as discussed here has not yet been carried out experimentally.

Here, we focused on the identification of a large number of flow states
including equilibria, limit cycles and chaotic dynamics to map out the
parameter space and to compare with results from linear stability of wall-mode
and bulk onset. Further analysis is currently underway to describe the
transition to turbulence, for instance through a series of bifurcations.
Preliminary results suggest the presence of multiple states and hysteresis. In
the context of pattern formation, the possibility of hysteresis in this system
has been suggested in experimental studies (Zürner et al., 2020).

[Supplementary data]Supplementary material and movies are available at …

[Acknowledgements]The authors thank G. M. Vasil for fruitful discussions.

[Funding]This work was supported by the Deutsche Forschungsgemeinschaft
(SPP1881 ”Turbulent Superstructures” grants Sh405/7, Sh405/16 and Li3694/1)
and used the ARCHER2 UK National Supercomputing Service
(https://www.archer2.ac.uk) with resources provided by the UK Turbulence
Consortium (EPSRC grants EP/R029326/1 and EP/X035484/1).

[Declaration of interests]The authors report no conflict of interest.

## References

  * Akhmedagaev et al. (2020) Akhmedagaev, R., Zikanov, O., Krasnov, D. & Schumacher, J. 2020 Turbulent Rayleigh–Bénard convection in a strong vertical magnetic field. J. Fluid Mech. 895, R4.
  * Busse (2008) Busse, F. H. 2008 Asymptotic theory of wall-attached convection in a horizontal fluid layer with a vertical magnetic field. Phy. Fluids 20 (2).
  * Chandrasekhar (1961) Chandrasekhar, S. 1961 Hydrodynamic and hydromagnetic stability. Int. Ser. Monogr. on Phys. .
  * Cioni et al. (2000) Cioni, S., Chaumat, S. & Sommeria, J. 2000 Effect of a vertical magnetic field on turbulent Rayleigh-Bénard convection. Phys. Rev. E 62 (4), R4520.
  * Davidson (1999) Davidson, P. A. 1999 Magnetohydrodynamics in materials processing. Annu. Rev. Fluid Mech. 31, 273–300.
  * Ecke et al. (2022) Ecke, R. E., Zhang, X. & Shishkina, O. 2022 Connecting wall modes and boundary zonal flows in rotating rayleigh-bénard convection. Phys. Rev. Fluids 7 (1), L011501.
  * Ecke et al. (1992) Ecke, R. E., Zhong, F. & Knobloch, E. 1992 Hopf bifurcation with broken reflection symmetry in rotating rayleigh-bénard convection. Europhysics Letters 19 (3), 177.
  * Favier & Knobloch (2020) Favier, B. & Knobloch, E. 2020 Robust wall states in rapidly rotating Rayleigh–Bénard convection. J. Fluid Mech. 895, R1.
  * Herrmann & Busse (1993) Herrmann, J. & Busse, F. H. 1993 Asymptotic theory of wall-attached convection in a rotating fluid layer. J. Fluid Mech. 255, 183–194.
  * Houchens et al. (2002) Houchens, B. C., Witkowski, L. M. & Walker, J. S. 2002 Rayleigh–Bénard instability in a vertical cylinder with a vertical magnetic field. J. Fluid Mech. 469, 189–207.
  * Jones (2011) Jones, C. A. 2011 Planetary magnetic fields and fluid dynamos. Annu. Rev. Fluid Mech. 43, 583–614.
  * Liao et al. (2006) Liao, X., Zhang, K. & Chang, Y. 2006 On boundary-layer convection in a rotating fluid layer. J. Fluid Mech. 549, 375–384.
  * Liu et al. (2018) Liu, W., Krasnov, D. & Schumacher, J. 2018 Wall modes in magnetoconvection at high Hartmann numbers. J. Fluid Mech. 849, R21–R212.
  * Ni & Li (2012) Ni, M. J. & Li, J. F. 2012 A consistent and conservative scheme for incompressible MHD flows at a low magnetic Reynolds number. Part III: On a staggered mesh. J. Comp. Phys. 231 (2), 281–298.
  * Reiter et al. (2022) Reiter, P., Zhang, X. & Shishkina, O. 2022 Flow states and heat transport in Rayleigh–Bénard convection with different sidewall boundary conditions. J. Fluid Mech. 936, A32.
  * Schumacher (2022) Schumacher, J. 2022 The various facets of liquid metal convection. J. Fluid Mech. 946, F1.
  * Shilnikov (1965) Shilnikov, L. P. 1965 A case of the existence of a denumerable set of periodic motions. In Doklady Akademii Nauk, , vol. 160, pp. 558–561. Russian Academy of Sciences.
  * Teimurazov et al. (2023) Teimurazov, A., McCormack, M., Linkmann, M. & Shishkina, O. 2023 Unifying heat transport model for the transition between buoyancy-dominated and Lorentz-force-dominated regimes in quasistatic magnetoconvection. arXiv preprint arXiv:2308.01748 .
  * Wiggins (1988) Wiggins, S. 1988 Global bifurcations and chaos: analytical methods. Springer.
  * Xu et al. (2023) Xu, Y., Horn, S. & Aurnou, J. M. 2023 The transition from wall modes to multimodality in liquid gallium magnetoconvection. arXiv preprint arXiv:2303.08966 .
  * Zhang & Liao (2009) Zhang, K. & Liao, X. 2009 The onset of convection in rotating circular cylinders with experimental boundary conditions. J. Fluid Mech. 622, 63–73.
  * Zürner et al. (2020) Zürner, T., Schindler, F., Vogt, T., Eckert, S. & Schumacher, J. 2020 Flow regimes of Rayleigh–Bénard convection in a vertical magnetic field. J. Fluid Mech. 894, A21.


# CU-Mamba: Selective State Space Models with Channel Learning for Image
Restoration

Rui Deng  Institute of Computational Mathematical Engineering
Stanford University
Palo Alto, United States
<EMAIL_ADDRESS>Tianpei Gu  Applied AI Research
KREA AI
San Francisco, United States
<EMAIL_ADDRESS>
###### Abstract

Reconstructing degraded images is a critical task in image processing.
Although CNN and Transformer-based models are prevalent in this field, they
exhibit inherent limitations, such as inadequate long-range dependency
modeling and high computational costs. To overcome these issues, we introduce
the Channel-Aware U-Shaped Mamba (CU-Mamba) model, which incorporates a dual
State Space Model (SSM) framework into the U-Net architecture. CU-Mamba
employs a Spatial SSM module for global context encoding and a Channel SSM
component to preserve channel correlation features, both in linear
computational complexity relative to the feature map size. Extensive
experimental results validate CU-Mamba’s superiority over existing state-of-
the-art methods, underscoring the importance of integrating both spatial and
channel contexts in image restoration.

###### Index Terms:

Image Restoration, Structured State Space Model, Mamba U-Net, Channel Learning

##  I Introduction

Image restoration is a fundamental task in digital image processing, aiming to
reconstruct high-quality images from those compromised by various degradations
such as noise, blur, and rain streaks. Recent advancements have highlighted
the effectiveness of Convolutional Neural Networks (CNNs)[1, 2, 3], and
Transformer-based models[4, 5, 6, 7], in this domain. CNNs utilize a
hierarchical structure that excels in capturing spatial hierarchies within
images. Transformer-model are originally designed for natural language
processing, but have shown promising result for visual understanding, such as
Vision Transformer[8]. Transformer models employ self-attention mechanisms,
making them particularly effective at modeling long-range dependencies. Both
approaches have achieved state-of-the-art results in numerous image
restoration tasks[9, 10, 11].

However, both CNNs and Transformer-based models have their limitations. CNNs,
despite their effectiveness in local feature extraction, often struggle with
capturing long-range dependencies within images due to their limited receptive
fields. In contrast, while Transformers mitigate this issue through their
global attention modules, they incur quadratic computational costs relative to
the size of the feature map. Moreover, Transformers may neglect fine-grained
local details that are critical for effective image restoration.

Figure 1: The Overall pipeline of CU-Mamba. Each CU-Mamba Block consists of a
Spatial SSM block (as explained in $(1)$) followed by a Channel SSM block (as
detailed in $(2)$). The structure of SelectiveSSM block is explained in Fig. 2

To address these limitations, recent advances have introduced structured state
space models (SSMs), particularly the Mamba model[12, 13], as an effective
building block for image recognition networks[14, 15]. By efficiently
compressing the global context through input-dependent selective SSMs[13],
Mamba maintains the benefits of global receptive fields while operating with
linear complexity relative to input tokens. This approach has demonstrated
superior performance in various language and visual tasks, surpassing both CNN
and Transformer-based models[13]. However, most visual Mamba models apply SSM
blocks independently to each feature channel, leading to a potential loss of
information flow across channels[16], which is particularly crucial for
compressing and reconstructing image details in image restoration.

To solve the above challenges, we propose the Channel-Aware U-Shaped Mamba
(CU-Mamba) model for image restoration. On top of the traditional U-Net
structure[17] in image restoration, CU-Mamba attains global receptive fields
with Mamba blocks while preserving channel-specific features. We utilize a
Spatial State Space Model block within our architecture to effectively capture
long-range dependencies in images with linear computational complexity,
ensuring a comprehensive understanding of global context. Additionally, we
implement a Channel State Space Model component to enhance feature mixing
across channels during feature map compression and subsequent upsampling
within the U-Net. This dual approach enables the CU-Mamba model to achieve a
delicate balance between capturing extensive spatial details and preserving
complicated channel-wise correlation, thereby significantly enhancing the
quality and accuracy of the restored images.

Overall, the main contributions of this work are:

  * •

We introduce the Channel-Aware U-Shaped Mamba (CU-Mamba) model, incorporating
a dual State Space Model (SSM) to enrich U-Net with global context and
channel-specific features for image restoration.

  * •

We validate the effectiveness of both the Spatial and Channel SSM modules
through detailed ablation studies.

  * •

Our experiments demonstrate that the CU-Mamba model achieves promising
performance on various image restoration datasets, surpassing current SOTA
methods while maintaining a lower computational cost.

.

##  II Related Work

CNN-Based Approaches: CNN-based models[11, 3] have served as a fundamental
architecture for image restoration over the past few years. These models
demonstrated substantial improvements over traditional techniques[18], which
relied heavily on hand-crafted features and prior knowledge. Among CNN-based
models, the U-shaped encoder-decoder networks with skip connection[19] have
demonstrated strong competence in various image restoration tasks due to their
hierarchical multi-scale architecture and residual feature representation.

Transformer-Based Approaches: The inherently local receptive fields in CNNs
posed limitations for capturing long-range dependencies. This challenge leads
to the adoption of Transformer models[20, 8], which utilize a global self-
attention mechanism to encapsulate long-range interactions across the image.
Transformer models are now widely applied in low-level vision tasks such as
super-resolution[7], image denoising[21], deblurring[22], and deraining[23].
To reduce the quadratic computation complexity in the attention mechanism,
self-attention is performed over local windows[24] or channel-dimensions[25].
Despite the architectural design, the computational overhead remains high
given the intrinsic mechanism of self-attention modules.

Visual Structured State Space Models: Recent innovations include the
integration of State Space Models (SSMs)[12, 13] into the image recognition
pipeline, as demonstrated by Vision Mamba[15]. SSMs provide a novel method for
capturing long-range dependencies with linear computational complexity, thus
addressing the computational inefficiencies inherent in Transformers while
retaining their global contextual modeling capabilities. U-Mamba[26] and VM-
UNet[27] introduce the Mamba block to the U-net structure, targeting
biomedical image segmentation problems. To facilitate the channel-wise
information flow, MambaMixer[16] introduces channel-mixing Mamba blocks to
image recognition and time series forecasting. Still, the existing U-shaped
Mamba architecture does not integrate channel SSM modules, which are vital for
compressing and reconstructing features given the rich context in channel
dimensions. In this work, we propose an efficient and effective dual-
directional Mamba U-Net that accounts for both the global context and channel
correlation during image restoration.

##  III Method

We aim to develop an effective U-Net that focuses on long-range spatial and
channel correlations in image restoration. We propose the CU-Mamba model,
which applies the Spatial and Channel SSM blocks to learn global context and
channel features with only linear complexity. In this section, we first go
through the overall pipeline of our U-Net design, and then we dive into its
components by explaining: the selective SSM framework, our Spatial SSM block,
and our Channel SSM block. We finally analyze the computation cost of our
model to demonstrate its efficiency.

###  III-A Overall Pipeline

Fig. 1 demonstrates the overall framework of CU-Mamba. Given a degraded image
$I\in\mathbb{R}^{H\times W\times 3}$, it first goes through a $3\times 3$
convolution to obtain low-level features $X_{0}\in\mathbb{R}^{H\times W\times
C}$. $X_{0}$ is then feed through a 4-level symmetrical encoder-decoder U-Net
structure to formulate fine-grained, high-quality features. At each level $l$,
the encoder consists of $N_{l}$ CU-Mamba Blocks and a downsampling layer.
Specifically, each CU-Mamba Block contains one Spatial SSM block followed by
one Channel SSM block, as demonstrated in Fig. 1 (1) and (2). The downsampling
operator hierarchically reduces spatial size and expanding the number of
channels, forming the feature map
$X_{l}\in\mathbb{R}^{\frac{H}{2^{l}}\times\frac{W}{2^{l}}\times 2^{l}C}$.

During feature reconstruction, the low-resolution latent feature
$X_{4}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times 8C}$ is passed into
the decoder, which consists of $N_{l}$ CU-Mamba Blocks symmetrical to the
encoder and an upsampling layer. The upsampling operator doubles the size of
the feature map while reducing the channel length to a half. To facilitate
feature reconstruction, we follow [5, 4] to use a skip-connection by
concatenating the encoder features with the corresponding decoder features.
The final decoded feature is then passed into the output projection block and
reshaped back to $R\in\mathbb{R}^{H\times W\times 3}$. We obtain the final
restored image: $I^{\prime}=I+R$. We train the CU-Mamba following the loss
function of previous works[5, 17, 28]:

$\displaystyle\mathcal{L}(I^{\prime},\hat{I})$
$\displaystyle=\sqrt{\|I^{\prime}-\hat{I}\|_{2}+\epsilon}+\lambda\|\mathcal{F}(I^{\prime})-\mathcal{F}(\hat{I})\|_{1}$
(1)

, where $\hat{I}$ is the ground-truth image, $\mathcal{F}$ is the Fourier
transform to the frequency domain. We set $\epsilon=10^{-3}$ and $\lambda=0.1$
in the experiments.

###  III-B Selective SSM Framework

We provide a simple overview of the selective SSM (Mamba) mechanism[13]
employed in our framework.

Structured state space sequence models (SSMs) map a 1D sequential input
$x(t)\in\mathbb{R}\to y(t)\in\mathbb{R}$ through an implicit latent state
$h(t)\in\mathbb{R}^{N}$. A SSM is defined by four parameters $(\Delta,A,B,C)$
with the following operations:

$\displaystyle h_{t}$ $\displaystyle=\bar{A}h_{t-1}+\bar{B}x_{t}$ (2)
$\displaystyle y_{t}$ $\displaystyle=Ch_{t}$ (3)

, where $(\bar{A},\bar{B})$ are discrete version of $(A,B)$ from fixed
transformations $\bar{A}=f_{A}(\Delta,A)$ and $\bar{B}=f_{B}(\Delta,A,B)$.
Various discretization rules can be used in a SSM block, and discretization
allows efficient parallelizable training through global convolution.

Despite the efficiency brought by discretization, parameters
$(\Delta,\bar{A},\bar{B},C)$ in SSMs are data-independent and time-invariant,
limiting the expressiveness of the hidden state to compress seen context.
Selective SSM (or Mamba) introduces data-dependent parameters $(B,C,\Delta)$
that effectively select relevant information in $x_{t}$:
$B_{t}=\texttt{Linear}_{B}(x_{t}),C_{t}=\texttt{Linear}_{C}(x_{t}),\Delta_{t}=\texttt{SoftPlus}(\texttt{Linear}_{\Delta}(x_{t}))$.

Through hardware-aware optimizations[29, 30], selective SSM consumes linear
computation and memory complexity w.r.t. sequence lengths, while effectively
compressing relevant context over the global input sequence. The optimized
selective SSM (Mamba) architecture[13] is shown in Fig. 2.

Figure 2: The structure of SelectiveSSM block. On top of the traditional SSM
blocks, the selective SSM adds a SiLU activation similar to the Gated MLP[31].
This Gated design allows the model to fuse and select information across
tokens. On the other hand, the Linear and Conv layers allows the model to
learn input-dependent parameters.

###  III-C Global Learning Block: Spatial SSM

The success of the transformer architectures[4, 22] shows that integrating the
global context through the hierarchical structure of the U-Net is crucial for
high-quality image restoration. However, such global receptive field comes at
the cost of quadratic computational complexity[20]. We thus design a global
learning block that effectively compresses the long-range context with the
selective SSM framework, which only requires linear computational complexity.

Given a layer-normalized input tensor $X\in\mathbb{R}^{H\times W\times C}$, we
first apply $1\times 1$ convolutions to gather context across different
channels at the pixel level, and then use $3\times 3$ depth-wise convolutions
to capture the spatial context through channels. We then flatten the feature
map into $\hat{X}\in\mathbb{R}^{L\times C}$, where $L=H\times W$, to construct
a sequence of feature patches. We encode the global context of $\hat{X}$
through:

$\displaystyle\hat{X^{\prime}}$ $\displaystyle=\texttt{SelectiveSSM}(\hat{X})$
(4)

, where the SelectiveSSM block is demonstrated and explained in Fig. 2. We can
interpret this operation as linearly scanning the feature map of tensor $X$
from the upper-left to the bottom-right corner, where each pixel in the map
learns its hidden representation from all previously seen context. The final
representation $\hat{X^{\prime}}$ is reshaped to
$\hat{X^{\prime}}\in\mathbb{R}^{H\times W\times C}$, and it encodes the long-
range dependency within its $H\times W$ dimensions.

Figure 3: The visualization of image denosing result in SIDD[32] dataset.

###  III-D Channel Learning Block: Channel SSM

In U-Net architectures, the channel-wise features in the downsampling and
upsampling path is essential for compressing and reconstructing image context
and structures. One problem of the existing Mamba-based U-Net is that while
the global context is captured by scanning image feature map, the channel
information is usually overlooked.[16] To learn the dependencies of features
across channels, we introduce the selective SSM mechanism to the channel
dimension.

Similar to the Sptial SSM block, given a layer-normalized input tensor
$X\in\mathbb{R}^{H\times W\times C}$, we use $1\times 1$ convolutions followed
by $3\times 3$ depth-wise convolutions to preprocess the local context. We
then transpose $X$ to $X^{T}\in\mathbb{R}^{C\times H\times W}$ and flatten to
$\hat{X^{T}}\in\mathbb{R}^{C\times L}$. This can be viewed as using flattened
feature pixels as the channel representation. We then apply the selective SSM
by:

$\displaystyle\hat{X^{T^{\prime}}}$
$\displaystyle=\texttt{SelectiveSSM}(\hat{X^{T}})$ (5)

This operation effectively mixes and memorizes the channel-wise feature by
scanning the channel map from top to bottom. The final feature
$\hat{X^{T^{\prime}}}$ is reshaped and transposed back to
$\hat{X^{*}}\in\mathbb{R}^{H\times W\times C}$. It is then passed to 2 blocks
of $3\times 3$ depth-wise convolutions with LeakyReLU activation to smooth the
local representations.

###  III-E Computational Complexity of CU-Mamba

We follow the complexity analysis from [13] and [16]. Denote the batch size as
$B$, the input sequence length as $L$ (Here, $L=H\times W$), the channel
dimension as $C$, and the expansion factor as $E$ ($E=2$ throughout our
implementation). With the efficient parallel scan algorithm[30], the
computational complexity is $\mathcal{O}(BLE+EC)$[13] for the spatial SSM
block and $\mathcal{O}(BCE+EL)$ for the channel SSM block. The total
complexity is thus $\mathcal{O}(BE(L+C))$, which is linear in the sequence
length and channel dimensions.

TABLE I: Real Noise Removal Experimental Results.

| SIDD[32] | DND[33]
---|---|---
Model | PSNR | SSIM | PSNR | SSIM
BM3D [34] | 25.65 | 0.685 | 34.51 | 0.851
RIDNet [35] | 38.71 | 0.914 | 39.26 | 0.953
DAGL [36] | 38.94 | 0.953 | 39.77 | 0.956
DANet+ [37] | 39.47 | 0.918 | 39.59 | 0.955
MPRNet [38] | 39.71 | 0.958 | 39.80 | 0.954
Uformer [5] | 39.77 | 0.959 | 39.96 | 0.956
HINet [39] | 39.99 | 0.958 | - | -
Restormer [4] | 40.02 | 0.960 | 40.03 | 0.956
NAFNet [40] | 40.30 | 0.962 | - | -
CU-Mamba | 40.22 | 0.962 | 40.34 | 0.960

TABLE II: Image Motion Deblurring Experimental Results.

| GoPro[41] | HIDE[42] | RealBlur-R[43] | RealBlur-J[43]
---|---|---|---|---
Model | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Xu et al. [44] | 21.00 | 0.741 | - | - | 34.46 | 0.937 | 27.14 | 0.830
Nah et al. [41] | 29.08 | 0.914 | 25.73 | 0.874 | 32.51 | 0.841 | 27.87 | 0.827
DeblurGAN-v2 [45] | 29.55 | 0.934 | 26.61 | 0.875 | 35.26 | 0.944 | 28.70 | 0.866
DMPHN [46] | 31.20 | 0.940 | 29.09 | 0.924 | 35.70 | 0.948 | 28.42 | 0.860
SPAIR [47] | 32.06 | 0.953 | 30.29 | 0.931 | - | - | 28.81 | 0.875
MPRNet [38] | 32.66 | 0.959 | 30.96 | 0.939 | 35.99 | 0.952 | 28.70 | 0.873
Restormer [4] | 32.92 | 0.961 | 31.22 | 0.942 | 36.19 | 0.957 | 28.96 | 0.879
Uformer [5] | 32.97 | 0.967 | 30.83 | 0.952 | 36.22 | 0.957 | 29.06 | 0.884
Stripformer [22] | 33.08 | 0.962 | 31.03 | 0.940 | 39.84 | 0.974 | 32.48 | 0.929
FSNet [2] | 33.29 | 0.963 | 31.05 | 0.941 | 35.84 | 0.952 | - | -
MRLPFNet [1] | 33.50 | 0.965 | 31.63 | 0.946 | 40.92 | 0.975 | 33.19 | 0.936
CU-Mamba | 33.53 | 0.965 | 31.47 | 0.944 | 41.01 | 0.976 | 33.21 | 0.936

##  IV Experiments

Figure 4: The visualization of image motion deblurring result in GoPro[41]
dataset.

We first explain the details of our experimental settings. Then, we
demonstrate the strong performance of CU-Mamba through extensive experiments
in image denoising and image deblurring. We finally conduct ablation studies
to validate the effectiveness of each module in the CU-Mamba model.

###  IV-A Experimental Settings

Parameter Settings: Following the previous training setting[5], we preprocess
the training samples by randomly flipping the image horizontally and rotating
the images by $90^{\circ}$, $180^{\circ}$, or $270^{\circ}$. During training,
we use the AdamW optimizer[48] with the momentum $\beta_{1}=0.9$ and
$\beta_{2}=0.999$. To stabilize the training process, we set the initial
learning rate to $5e^{-5}$ and gradually decrease to $1e^{-6}$ with the cosine
annealing strategy[49]. The channel width $C$ passed to the initial SSM block
is set to 32.

Evaluation Metrics: To evaluate the restoration quality, we use the PSNR (Peak
Signal-to-Noise Ratio) and SSIM (Structural Similarity Index)[50] metrics
following previous works. PSNR measures the quality of a reconstructed image
by calculating the ratio between the maximum possible signal power and the
power of corrupting noise, while SSIM compares the structural similarity
between the reconstructed and the original image. We calculate these metrics
under the RGB color space.

###  IV-B Image Denoising Results

We evaluated the performance of our method for real-world noise removal using
the SIDD[32] and DND[33] datasets. For evaluation of the DND dataset, we
follow the common strategy of previous works[5]: train our model using the
SIDD dataset and test our model in DND’s online server. Table I presents a
comparative analysis of our proposed method with state-of-the-art approaches
on the SIDD and DND datasets. Notably, our approach outperforms existing CNN-
based methods (RIDNet[35], MPRNet[38], HINet[39]) and Transformer-based
methods (Uformer[5], Restormer[4]), showcasing its efficacy in handling real-
world noise. Fig. 3 showcases the qualitative results of our method compared
to ground truth images on the SIDD dataset. We can observe that our method
reconstruct the noisy image with more exact details similar to the ground-
truth image.

###  IV-C Image Deblurring Results

We test the motion blur removal performance of CU-Mamba in four datasets.
Following the previous methods[38], we train our model using the training set
of the GoPro dataset[41], and then we test our model on two synthetic (Test
set of GoPro[41] and HIDE[42]) and two real-world datasets (RealBlur-R and
RealBlur-J[43]). As demonstrated in Table II, our method outperforms the
current state-of-the-art MRLPFNet[1], achieving a 0.09 dB improvement for
real-world image deblurring on the RealBlur-R dataset. We also test and
compare the inference time of our model with previous works in table III. Our
model exhibits a $4\times$ faster inference speed comparing to the
transformer-based model Restormer[4], additionally achieving a 0.87 dB
performance gain in PSNR. This proves the efficiency of linear complexity in
our selective SSM blocks over the quadratic cost of self-attention modules.
Moreover, through qualitative comparisons with other existing methods in Fig.
4, we highlight the effectiveness of CU-Mamba in producing more realistic
deblurred images that aligns closely to the ground-truth label.

TABLE III: The inference time comparison with existing models using GoPro test
dataset[41]. FLOPs is calculated with input image of size $256\times 256$.

| Time (s) | FLOPs (G) | PSNR
---|---|---|---
DBGAN[51] | 1.447 | 759.85 | 31.10
DMPHN[46] | 0.405 | - | 31.20
MPRNet[38] | 1.148 | 777.01 | 32.66
Restormer[4] | 1.218 | 140.99 | 32.92
FSNet[2] | 0.362 | 111.14 | 33.29
CU-Mamba | 0.305 | 43.10 | 33.53

TABLE IV: Ablation Study of CU-Mamba Components. PSNR and SSIM is calculated
using GoPro test dataset[41].

Architecture | # Params (M) | PSNR | SSIM
---|---|---|---
UNet with Resblocks[17] | 16.1 | 32.45 | 0.957
✓Spatial SSM | 15.6 | 33.31 | 0.963
✓Channel SSM | 16.3 | 33.07 | 0.962
✓✓Spatial + Channel SSM | 19.7 | 33.53 | 0.965

###  IV-D Ablation Studies

We conduct ablation studies for the spatial and channel SSM modules of CU-
Mamba to analyze their impacts. Table IV shows the number of model parameter,
SSIM, and PSNR of each ablation setting in the GoPro test dataset[41]. The
spatial SSM block, when applied independently, exhibited a more effective
improvement in model performance compared to the channel SSM block alone. This
suggests that the long-range dependency plays a crucial role in image
restoration, and the channel SSM alone is not able to encode global
information. The channel SSM, on the other hand, facilitates the information
mixing across the channel direction, thus it also shows 0.62 dB increase in
PSNR over the UNet baseline. Still, when we apply both the spatial and channel
blocks, CU-Mamba demonstrates the best performance of 33.53 dB in PSNR. This
highlights the effectiveness of integrating both types of selective SSM blocks
to fully exploit the model’s capacity of learning complex, hierarchical image
representations.

##  V Conclusion

In conclusion, we introduce the Channel-Aware U-Shaped Mamba (CU-Mamba) model,
which improves image restoration by combining a U-Net framework with dual-
direction selective State Space Models to better understand and reconstruct
images. Extensive experiments show that CU-Mamba performs well compared to
existing methods, demonstrating the efficiency and effectiveness of our
approach. This work offers a new perspective on the U-Net architecture in
image restoration, showcasing the importance of both the spatial and channel
context in feature reconstruction and opening a promising direction for
further research and practical applications.

## References

  * [1] J. Dong, J. Pan, Z. Yang, and J. Tang, “Multi-scale residual low-pass filter network for image deblurring,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 12 345–12 354.
  * [2] Y. Cui, W. Ren, X. Cao, and A. Knoll, “Image restoration via frequency selection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [3] Q. Yang, Y. Liu, J. Tang, and T. Ku, “Residual and dense unet for under-display camera restoration,” in _European Conference on Computer Vision_. Springer, 2020, pp. 398–408.
  * [4] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 5728–5739.
  * [5] Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li, “Uformer: A general u-shaped transformer for image restoration,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 17 683–17 693.
  * [6] J. Ke, Q. Wang, Y. Wang, P. Milanfar, and F. Yang, “Musiq: Multi-scale image quality transformer,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 5148–5157.
  * [7] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 1833–1844.
  * [8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021.
  * [9] F. Yu, J. Gu, Z. Li, J. Hu, X. Kong, X. Wang, J. He, Y. Qiao, and C. Dong, “Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild,” 2024.
  * [10] K. Zhang, J. Liang, L. Van Gool, and R. Timofte, “Designing a practical degradation model for deep blind image super-resolution,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 4791–4800.
  * [11] X. Wang, L. Xie, C. Dong, and Y. Shan, “Real-esrgan: Training real-world blind super-resolution with pure synthetic data,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 1905–1914.
  * [12] A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,” 2022.
  * [13] A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” _arXiv preprint arXiv:2312.00752_ , 2023.
  * [14] E. Nguyen, K. Goel, A. Gu, G. Downs, P. Shah, T. Dao, S. Baccus, and C. Ré, “S4nd: Modeling images and videos as multidimensional signals with state spaces,” _Advances in neural information processing systems_ , vol. 35, pp. 2846–2861, 2022.
  * [15] L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, “Vision mamba: Efficient visual representation learning with bidirectional state space model,” _arXiv preprint arXiv:2401.09417_ , 2024.
  * [16] A. Behrouz, M. Santacatterina, and R. Zabih, “Mambamixer: Efficient selective state space models with dual token and channel selection,” _arXiv preprint arXiv:2403.19888_ , 2024.
  * [17] S.-J. Cho, S.-W. Ji, J.-P. Hong, S.-W. Jung, and S.-J. Ko, “Rethinking coarse-to-fine approach in single image deblurring,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 4641–4650.
  * [18] K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 33, no. 12, pp. 2341–2353, 2010.
  * [19] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18_. Springer, 2015, pp. 234–241.
  * [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [21] M. Li, J. Liu, Y. Fu, Y. Zhang, and D. Dou, “Spectral enhanced rectangle transformer for hyperspectral image denoising,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 5805–5814.
  * [22] F.-J. Tsai, Y.-T. Peng, Y.-Y. Lin, C.-C. Tsai, and C.-W. Lin, “Stripformer: Strip transformer for fast image deblurring,” in _European Conference on Computer Vision_. Springer, 2022, pp. 146–162.
  * [23] X. Chen, H. Li, M. Li, and J. Pan, “Learning a sparse transformer network for effective image deraining,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 5896–5905.
  * [24] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 10 012–10 022.
  * [25] M. Ding, B. Xiao, N. Codella, P. Luo, J. Wang, and L. Yuan, “Davit: Dual attention vision transformers,” in _European conference on computer vision_. Springer, 2022, pp. 74–92.
  * [26] J. Ma, F. Li, and B. Wang, “U-mamba: Enhancing long-range dependency for biomedical image segmentation,” _arXiv preprint arXiv:2401.04722_ , 2024.
  * [27] J. Ruan and S. Xiang, “Vm-unet: Vision mamba unet for medical image segmentation,” _arXiv preprint arXiv:2402.02491_ , 2024.
  * [28] L. Kong, J. Dong, J. Ge, M. Li, and J. Pan, “Efficient frequency domain-based transformers for high-quality image deblurring,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 5886–5895.
  * [29] G. E. Blelloch, “Prefix sums and their applications,” 1990.
  * [30] J. T. Smith, A. Warrington, and S. W. Linderman, “Simplified state space layers for sequence modeling,” _arXiv preprint arXiv:2208.04933_ , 2022.
  * [31] H. Liu, Z. Dai, D. So, and Q. V. Le, “Pay attention to mlps,” _Advances in neural information processing systems_ , vol. 34, pp. 9204–9215, 2021.
  * [32] A. Abdelhamed, S. Lin, and M. S. Brown, “A high-quality denoising dataset for smartphone cameras,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 1692–1700.
  * [33] T. Plotz and S. Roth, “Benchmarking denoising algorithms with real photographs,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1586–1595.
  * [34] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” _IEEE Transactions on image processing_ , vol. 16, no. 8, pp. 2080–2095, 2007.
  * [35] S. Anwar and N. Barnes, “Real image denoising with feature attention,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 3155–3164.
  * [36] C. Mou, J. Zhang, and Z. Wu, “Dynamic attentive graph learning for image restoration,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 4328–4337.
  * [37] Z. Yue, Q. Zhao, L. Zhang, and D. Meng, “Dual adversarial network: Toward real-world noise removal and noise generation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16_. Springer, 2020, pp. 41–58.
  * [38] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 14 821–14 831.
  * [39] L. Chen, X. Lu, J. Zhang, X. Chu, and C. Chen, “Hinet: Half instance normalization network for image restoration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 182–192.
  * [40] L. Chen, X. Chu, X. Zhang, and J. Sun, “Simple baselines for image restoration,” in _European conference on computer vision_. Springer, 2022, pp. 17–33.
  * [41] S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3883–3891.
  * [42] Z. Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and L. Shao, “Human-aware motion deblurring,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5572–5581.
  * [43] J. Rim, H. Lee, J. Won, and S. Cho, “Real-world blur dataset for learning and benchmarking deblurring algorithms,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16_. Springer, 2020, pp. 184–201.
  * [44] L. Xu, S. Zheng, and J. Jia, “Unnatural l0 sparse representation for natural image deblurring,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 1107–1114.
  * [45] O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, “Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 8878–8887.
  * [46] H. Zhang, Y. Dai, H. Li, and P. Koniusz, “Deep stacked hierarchical multi-patch network for image deblurring,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 5978–5986.
  * [47] K. Purohit, M. Suin, A. Rajagopalan, and V. N. Boddeti, “Spatially-adaptive image restoration using distortion-guided networks,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 2309–2319.
  * [48] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [49] ——, “Sgdr: Stochastic gradient descent with warm restarts,” _arXiv preprint arXiv:1608.03983_ , 2016.
  * [50] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [51] K. Zhang, W. Luo, Y. Zhong, L. Ma, B. Stenger, W. Liu, and H. Li, “Deblurring by realistic blurring,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2737–2746.