# GroupCDL: Interpretable Denoising and Compressed Sensing MRI via Learned
Group-Sparsity and Circulant Attention

Nikola Janjušević1,3, Amirhossein Khalilian-Gourtani2, Adeen Flinker2, Li
Feng3, Yao Wang1 1New York University Tandon School of Engineering, Electrical
and Computer Engineering Department, Brooklyn, NY 11201, USA. New York
University Grossman School of Medicine, {2Neurology Department, 3Radiology
Department}, New York, NY 10016, USA. Please send all correspondence regarding
to this manuscript to N. Janjušević (email:nikola@nyu.edu).

###### Abstract

Nonlocal self-similarity within images has become an increasingly popular
prior in deep-learning models. Despite their successful image restoration
performance, such models remain largely uninterpretable due to their black-box
construction. Our previous studies have shown that interpretable construction
of a fully convolutional denoiser (CDLNet), with performance on par with
state-of-the-art black-box counterparts, is achievable by unrolling a
convolutional dictionary learning algorithm. In this manuscript, we seek an
interpretable construction of a convolutional network with a nonlocal self-
similarity prior that performs on par with black-box nonlocal models. We show
that such an architecture can be effectively achieved by upgrading the
$\ell_{1}$ sparsity prior (soft-thresholding) of CDLNet to an image-adaptive
group-sparsity prior (group-thresholding). The proposed learned group-
thresholding makes use of nonlocal attention to perform spatially varying
soft-thresholding on the latent representation. To enable effective training
and inference on large images with global artifacts, we propose a novel
_circulant-sparse attention_. We achieve competitive natural-image denoising
performance compared to black-box nonlocal DNNs and transformers. The
interpretable construction of our network allows for a straightforward
extension to Compressed Sensing MRI (CS-MRI), yielding state-of-the-art
performance. Lastly, we show robustness to noise-level mismatches between
training and inference for denoising and CS-MRI reconstruction.

###### Index Terms:

Deep-learning, interpretable neural network, nonlocal self-similarity, group-
sparsity, unrolled network, convolutional dictionary learning, image
denoising, compressed-sensing, MRI

##  I Background and Introduction

Nonlocal self-similarity (NLSS) of natural images has proven to be a powerful
signal prior for classical and deep-learning based image restoration. However,
state-of-the-art NLSS deep-learning methods are widely constructed as black-
boxes, often rendering their analysis and improvement beholden to trial and
error. Additionally, current implementations of the NLSS prior in deep-
learning separately process overlapping image windows, falsely neglecting the
dependency between these overlaps. Here, we address these two shortcomings of
nonlocal deep neural networks (DNNs) from the perspective of interpretable
architecture design and sparse array arithmetic. This construction allows us
to extend the formulation to compressed sensing magnetic resonance imaging
(CS-MRI) without use of tricks common in adapting black-box methods to
inverse-problems.

A growing literature of DNNs, derived as direct parameterizations of classical
image restoration algorithms, perform on par with state-of-the-art black-box
fully convolutional neural networks, without employing common deep-learning
tricks (such as batch-normalization, residual learning, and feature domain
processing). This interpretable construction has been shown to be instrumental
in obtaining parameter and dataset efficiency[1, 2, 3, 4, 5]. Our previous
work, CDLNet [1], introduced a unique interpretable construction, based on
convolutional dictionary learning, and achieved novel robustness to mismatches
in observed noise-level during training and inference. In this work, we
incorporate the NLSS prior into CDLNet and demonstrate the first instance of
an interpretable network bridging the performance gap to state-of-the-art
nonlocal black-box methods for image denoising. Additionally, the extension of
our method to CS-MRI outperforms existing state-of-the-art methods.

Nonlocal layers in DNNs attempt to model long-range image dependencies by
computing pixel-wise self-similarity metrics. To tame the quadratic
computational complexity of this operation, image-restoration DNNs generally
rely on computing dense similarities on small overlapping patches of the
input, which are processed independently by the layer/network and subsequently
averaged on overlapping regions to produce the final output [2, 6, 7].
Naturally, such patch-based dense attention may incur a runtime penalty by
redundant overlap processing and a restoration penalty due to the disregard
for the correlation among these overlapping regions. In this work, we propose
a novel circulant-sparse attention operation (CircAtt) that addresses these
shortcomings in a computationally efficient manner. CircAtt allows for
seamless training and inference on large images, enabling more effective
restoration of global degradation such as aliasing-artifacts encountered in
CS-MRI.

Previous work used a patch-based group-sparsity prior in an interpretably
constructed DNN [2]. However, this approach did not achieve competitive
performance with black-box nonlocal DNNs. In contrast, we enforce pixel-wise
group-sparsity of a latent convolutional sparse representation, with nonlocal
attention performed in a compressed subband domain to aid performance and
inference speed. We also propose the novel CircAtt operation, together
achieving denoising performance on par with the state-of-the-art methods.

We highlight the following contributions:

  * •

a novel and efficient nonlocal self-similarity operation (CircAtt) which
addresses the modeling and computational shortcomings of common patch-based
dense attention implementations.

  * •

a novel thresholding operation, inspired by a group-sparsity prior, which
utilizes a reduced channel dimension of the latent space to achieve
competitive inference speeds.

  * •

an interpretable nonlocal CNN with competitive denoising and CS-MRI
performance to state-of-the-art black-box models.

  * •

an open-source implementation111 https://github.com/nikopj/GroupCDL,
https://github.com/nikopj/CirculantAttention.jl.  in the Julia programming
language [8].

The remainder of the manuscript is organized as follows: in Section II, we
introduce the mathematics and notation behind classical convolutional
dictionary learning and group-sparse representation. We also provide context
for related black-box and interpretable deep-learning methods. In Section III,
we introduce our nonlocal CNN derived from group-sparse convolutional
dictionary learning, dubbed GroupCDL. We additionally introduce GroupCDL’s
extension for CS-MRI reconstruction and detail our novel CircAtt operation. In
Section IV, we show experimental results that compare GroupCDL to state-of-
the-art deep learning methods for both denoising and CS-MRI reconstruction.

##  II Preliminaries and Related Work

TABLE I: Notation

$\bm{x}\in\mathbb{R}^{NC}$ | a vector valued image with $N=N_{1}\times N_{2}$ pixels,
---|---
and vectorized channels,
$\bm{x}=[\bm{x}_{1}^{T},\,\cdots,\,\bm{x}_{C}^{T}]^{T}$.
$\bm{x}_{c}\in\mathbb{R}^{N}$ | the $c$-th subband/feature-map/channel of $\bm{x}$.
$\bm{x}[n]\in\mathbb{R}^{C}$ | the $n$-th pixel of $\bm{x}$, $n\in[1,\,N]$.
$\bm{x}_{c}[n]\in\mathbb{R}$ | the $n$-th pixel of the $c$-th channel of $\bm{x}$.
$\vec{n}\in[1,N_{1}]\times[1,N_{2}]$ | the spatial coordinates of the $n$-th pixel of $\bm{x}$.
$\bm{u}\circ\bm{v}\in\mathbb{R}^{N}$ | the element-wise product of two vectors.
$\bm{D}\in\mathbb{R}^{NC\times QM}$ | a 2D $M$ to $C$ channel synthesis convolution
operator with stride $s_{c}$, where $Q=N/s_{c}^{2}$.
$\bm{D}^{T}\in\mathbb{R}^{QM\times NC}$ | a 2D $C$ to $M$ channel analysis convolution
operator with stride $s_{c}$, where $Q=N/s_{c}^{2}$.
$\bm{U}\in\mathbb{R}^{Q\times N}$ | a $Q\times N$ matrix with elements $\bm{U}_{ij}\in\mathbb{R}$.
$\bm{U}_{i:}\in\mathbb{R}^{N},~{}\bm{U}_{:j}\in\mathbb{R}^{Q}$ | the $i$-th row, $j$-th column of matrix $\bm{U}$.
$\bm{U}\otimes\bm{V}\in\mathbb{R}^{QM\times NC}$ | Kronecker product of $\bm{U}\in\mathbb{R}^{Q\times N}$ and $\bm{V}\in\mathbb{R}^{M\times C}$,
i.e. the block matrix with $\bm{V}$s scaled by $\bm{U}_{ij}~{}\forall~{}i,j$.
$\bm{I}_{N}\in\mathbb{R}^{N\times N}$ | the $N$ by $N$ identity matrix.
$\bm{y}=\underline{\bm{U}}\bm{x}\equiv(\bm{I}_{C}\otimes\bm{U})\bm{x}$ | the matrix $\bm{U}$ applied channel-wise,
i.e. $\bm{y}_{c}=\bm{U}\bm{x}_{c}\in\mathbb{R}^{Q},~{}\forall~{}1\leq c\leq
C$.
$\bm{y}=\overline{\bm{V}}\bm{x}\equiv(\bm{V}\otimes\bm{I}_{N})\bm{x}$ | the matrix $\bm{V}$ applied pixel-wise,
i.e. $\bm{y}[i]=\bm{V}\bm{x}[i]\in\mathbb{R}^{M},~{}\forall~{}1\leq i\leq N$.
$\bm{S}\in\mathbb{B}_{W}^{N\times N}$ | a real-valued $N\times N$ matrix with
BCCB sparsity pattern of windowsize $W$.

###  II-A Dictionary Learning and Group-Sparse Representation

We consider the observation model of additive white Gaussian noise (AWGN),

$\bm{y}=\bm{x}+\bm{\nu},\quad\textnormal{where}\quad\bm{\nu}\sim\mathcal{N}(\bm{0},\sigma^{2}\bm{I}).$
(1)

Here, the ground-truth image $\bm{x}\in\mathbb{R}^{NC}$ is contaminated with
AWGN of noise-level $\sigma$, resulting in observed image
$\bm{y}\in\mathbb{R}^{NC}$. For convenience and clarity of notation, we denote
images in the vectorized form, and any linear operation on an image as a
matrix vector multiplication (see Table I for details). In implementation,
fast algorithms are used and these matrices are not actually formed, except
when explicitly mentioned.

We frame our signal-recovery problem in terms of a (given) $s_{c}$-strided
convolutional dictionary $\bm{D}\in\mathbb{R}^{NC\times QM}$, with
$Q=N/s_{c}^{2}$, i.e. the columns of $\bm{D}$ are formed by integer translates
of a set of $M$ (vectorized) 2D convolutional filters, each having $C$
channels. We assume
$\exists\,\bm{z}\in\mathbb{R}^{QM}\,\operatorname{\mathrm{s.t.}\,}\,\bm{x}\approx\bm{D}\bm{z}$.
The rich works of sparse-representation and compressed-sensing provide
guarantees based on assumptions of sparsity in $\bm{z}$ and regularity on the
columns of $\bm{D}$ [9]. We refer to $\bm{z}$ as our sparse-code, latent-
representation, or subband-representation of $\bm{x}$.

A popular classical paradigm for estimating $\bm{x}$ from an observed noisy
$\bm{y}$ is the Basis Pursuit DeNoising (BPDN) model,

$\underset{\bm{z}}{\mathrm{minimize}}~{}\frac{1}{2}\lVert\bm{y}-\bm{D}\bm{z}\rVert_{2}^{2}+\lambda\psi(\bm{z}),$
(2)

where $\psi:\mathbb{R}^{QM}\rightarrow\mathbb{R}_{+}$ is a chosen
regularization function. The Lagrange-multiplier term $\lambda>0$ provides a
trade-off between satisfying observation consistency and obeying the prior-
knowledge encoded by $\psi$. A popular approach to solving (2) is the
proximal-gradient method (PGM) [10], involving the proximal-operator of
$\psi$, defined as

$\operatorname{\mathbf{prox}}_{\tau\psi}(\bm{v})\coloneqq\operatorname*{arg\,min\,}_{\bm{x}}\tau\psi(\bm{x})+\frac{1}{2}\lVert\bm{x}-\bm{v}\rVert_{2}^{2},\quad\tau>0.$
(3)

PGM can be understood as a fixed point iteration involving the iterative
application of a gradient-descent step on the observation consistency term of
(2) followed by application of the proximal operator of $\psi$,

$\bm{z}^{(k+1)}=\operatorname{\mathbf{prox}}_{\tau\psi}(\bm{z}^{(k)}-\eta\bm{D}^{T}(\bm{D}\bm{z}^{(k)}-\bm{y})),$
(4)

where $\tau=\eta\lambda$, and $\eta>0$ is a step-size parameter.

When $\psi$ is the sparsity-promoting $\ell_{1}$-norm, the proximal operator
is given in closed-form by element-wise soft-thresholding [10],

$\operatorname{\mathrm{ST}}_{\tau}(\bm{z})=\bm{z}\circ\left(1-\frac{\tau}{\lvert\bm{z}\rvert}\right)_{+},$
(5)

where $(\cdot)_{+}$ denotes projection onto the positive orthant
$\mathbb{R}_{+}$. The resulting PGM iterations are commonly referred to as the
Iterative Soft-Thresholding Algorithm (ISTA) [10].

More sophisticated priors ($\psi$) can be used to obtain better estimates of
our desired ground-truth image by exploiting correlations between “related”
image-pixels. One such prior is group-sparsity,

$\displaystyle\psi(\bm{z})$ $\displaystyle=\sum_{\begin{subarray}{c}m=1\\\
i=1\end{subarray}}^{M,Q}\sqrt{\sum_{j=1}^{Q}\bm{\Gamma}_{ij}\bm{z}_{m}[j]^{2}}=\lVert\sqrt{(\bm{I}_{M}\otimes\bm{\Gamma})\bm{z}^{2}}\rVert_{1},$
(6)

where $\bm{\Gamma}\in\mathbb{R}_{+}^{Q\times Q}$ is a row-normalized adjacency
matrix (i.e. $\lVert\bm{\Gamma}_{i:}\rVert_{1}=1$), and $\cdot^{2}$ and
$\sqrt{\cdot}$ are taken element-wise. The $i$th adjacency row
($\bm{\Gamma}_{i:}$) measures closeness between the $i$th latent pixel
($\bm{z}[i]$) and every other latent pixel. Hence, group-sparse regularization
may be understood as encouraging similar latent-pixels to share the same
channel-wise sparsity pattern, and has been shown to improve denoising
performance under classical patch-based sparse coding methods [11], as well as
recent interpretably constructed DNNs [2].

In general, the group-sparse regularizer (6) does not have a closed form
proximal-operator. Motivated by the operator proposed in [2], we propose an
approximate solution, group-thresholding,

$\displaystyle\operatorname{\mathrm{GT}}_{\tau}(\bm{z};\,\bm{\Gamma})$
$\displaystyle=\bm{z}\circ\left(1-\frac{\tau}{\sqrt{(\bm{I}_{M}\otimes\bm{\Gamma})\bm{z}^{2}}}\right)_{+}.$
(7)

Note that the thresholding operator in Lecouat et. al [2] is equivalent to (7)
when the adjacency matrix in their formulation is row-normalized. This
approximate solution has the desirable property of reducing to element-wise
soft-thresholding (5) when $\bm{\Gamma}$ is the identity matrix. From the
perspective of deep-learning, GT involves an “attention” operation: a signal
transformation via a similarity/adjacency matrix. Black-box attention networks
(e.g. transformers) use attention to generate features for the next layer via
matrix-vector multiplication. In contrast, GT performs attention to obtain an
adjacency weighted signal energy which informs a spatially varying soft-
thresholding, i.e. group-thresholding may be rewritten as,

$\operatorname{\mathrm{GT}}_{\tau}(\bm{z};\bm{\Gamma})=\operatorname{\mathrm{ST}}_{\frac{\tau}{\bm{\xi}}\lvert\bm{z}\rvert}(\bm{z}),\quad\bm{\xi}=\sqrt{(\bm{I}_{M}\otimes\bm{\Gamma})\bm{z}^{2}}.$

The BPDN (2) problem can be made more expressive by opting to learn an optimal
dictionary from a dataset of noisy images $\mathcal{D}=\\{\bm{y}\\}$. We
express the (convolutional) dictionary learning problem as,

$\underset{\\{\bm{z}\\},\bm{D}\in\mathcal{C}}{\mathrm{minimize}}~{}\sum_{\bm{y}\in\mathcal{D}}\frac{1}{2}\lVert\bm{y}-\bm{D}\bm{z}\rVert_{2}^{2}+\lambda\psi(\bm{z}),$
(8)

where constraint set
$\mathcal{C}=\\{\bm{D}\,:\,\lVert\bm{D}_{:j}\rVert_{2}^{2}\leq
1~{}\forall\,j\\}$ ensures that the regularization term is not rendered
useless by an arbitrary scaling of latent coefficients. Solving (8) generally
involves alternating sparse-pursuit (ex. (4)) and a dictionary update with
fixed sparse-codes (ex. projected gradient descent) [12].

###  II-B Unrolled and Dictionary Learning Networks

Approaches in [13, 14, 15] explore the construction of DNNs as unrolled
proximal gradient descent machines with proximal-operators that are
implemented by a black-box CNN, learned end-to-end. Although these methods
contribute to more principled DNN architecture design in image-processing,
their use of black-box neural networks, such as UNets [16] and ResNets [17],
ultimately side-step the goal of full interpretability. In contrast, our
previous work CDLNet [1] introduces a CNN as a direct parameterization of
convolutional PGM (4) with an $\ell_{1}$ prior, with layers defined as,

$\begin{gathered}\bm{z}^{(0)}=\bm{0},\quad\text{for }~{}k=0,1,\dots,K-1,\\\
\bm{z}^{(k+1)}=\operatorname{\mathrm{ST}}_{\bm{\tau}^{(k)}}\left(\bm{z}^{(k)}-{\bm{A}^{(k)}}^{T}(\bm{B}^{(k)}\bm{z}^{(k)}-\bm{y})\right),\\\
\bm{\tau}^{(k)}=\bm{\tau}^{(k)}_{0}+\hat{\sigma}\bm{\tau}^{(k)}_{1},\quad\hat{\bm{x}}=\bm{D}\bm{z}^{(K)}.\end{gathered}$
(9)

Parameters
$\Theta=\\{\bm{D},\,\\{{\bm{A}^{T}}^{(k)},\,\bm{B}^{(k)},\,\bm{\tau}_{0}^{(k)},\bm{\tau}^{(k)}_{1}\\}_{0=1}^{K-1}\\}$
are optimized by back-propagation of a supervised or unsupervised loss
function. In this manuscript, we extend the formulation and direct
parameterization of CDLNet by introducing a novel implementation of the group-
sparsity prior, embodied in the proposed GroupCDL architecture (see Section
III). We also show that the noise-adaptive thresholding of CDLNet, derived
from BPDN (2), extends to GroupCDL in both image denoising and joint denoising
and CS-MRI (see Section IV-E).

Zheng et. al [18] propose a DNN architecture based on a classical dictionary
learning formulation of image denoising. However, this network heavily employs
black-box models such as UNets [16] and multi-layer perceptrons (MLPs). Our
proposed method differentiates itself by using a direct parameterization of
variables present in the classical proximal gradient method (4) with a group-
sparsity regularizer (6). This construction offers an alternative to the use
of black-boxes, yielding great learned parameter count efficiency, novel
generalization capabilities, and easy extensibility.

Directly parameterized dictionary learning networks [1, 3, 2, 19, 4, 5] have
gained some popularity in recent years due to their simple design and strong
structural similarities to popular ReLU-activation DNNs. This connection was
first established by the seminal work of Gregor et. al [20] in the form of the
Learned Iterative Shrinkage Thresholding Algorithm (LISTA). Here, we build
upon this literature by proposing a learned approximate proximal operator for
the convolutional dictionary learning formulation of a DNN, derived from a
group-sparsity prior (see Section III-B). We demonstrate that such a network
can compete well with and outperform state-of-the-art methods, without
sacrificing interpretability.

Lecouat et. al [2] propose a nonlocal CNN derived from a patch-based
dictionary learning algorithm with a group-sparsity prior, dubbed GroupSC. It
is well established that the independent processing of image patches and
subsequent overlap and averaging (patch-processing) is inherently suboptimal
to the convolutional model, due to the lack of consensus between pixels in
overlapping patch regions [4]. Our method is in-part inspired by GroupSC, but
is adapted to the convolutional sparse coding with groups defined over sliding
windows via CircAtt.

###  II-C Nonlocal Networks

The nonlocal self-similarity prior in image-restoration DNNs is commonly
formulated with patch-based dense attention to manage quadratic complexity [6,
2]. Patch overlap is often used to ensure that artifacts do not occur on
local-window boundaries. Despite many such networks being formulated as CNNs,
their patch-based inference ultimately diminishes the powerful shift-
invariance prior and increases computational cost due to additional processing
of overlapping regions (see Section III-D).

Liu et. al [21] proposed an alternate form of patch-based dense attention via
the Shifted-Window Vision Transformer architecture (SwinViT), demonstrated on
image classification, segmentation, and detection tasks. Their so-called Swin
attention mechanism makes use of patching without overlap, using alternating
“shifted” patch boundaries between patch-based dense attention layers. This
allows some cross over of information of neighboring patches [21]. Liu et.
al’s ablation studies considered use of a sliding-window attention mechanism,
similar to our CircAtt, but ultimately abandoned it due to similar performance
and slower inference compared to Swin attention on their examined tasks. Liang
et. al [7] (SwinIR) adapted the SwinViT architecture for image denoising and
other restoration tasks with local degradation operators (ex. super-
resolution, JPEG artifact removal). Our proposed method re-examines sliding-
window attention in the context of image restoration and shows favorable
performance over SwinIR in terms of learned parameter count efficiency.

Zamir et. al [22] proposed Restormer, a multi-resolution transformer model for
image restoration. They demonstrated results on image denoising and local
degradation operator tasks. Restormer’s core (transformer) block does not use
nonlocal attention, but instead computes similarities between entire feature
maps (known as “transposed attention”). In contrast, the proposed GroupCDL
method operates on a single resolution and implements nonlocal attention.

To correctly account for dependencies between neighboring local windows, we
propose a sliding-window NLSS (CircAtt), enabled by sparse matrix arithmetic.
To the best of our knowledge, we are the first to examine a sliding-window
attention mechanism for image restoration. Recent works have proposed other
so-called “sparse attention” mechanisms, however, they have either not been in
the context of image restoration [23], not employed a sliding-window [24], or
have employed a complicated hashing algorithm to exploit extremely long-range
dependencies [25].

##  III Proposed Method

Our proposed framework derives nonlocal attention from the group-sparsity
prior (6). We thus begin by introducing our Circulant-Sparse Attention
mechanism (Section III-A), which is integral to implementing group-
thresholding on convolutional sparse codes, in addition to enabling unified
training and inference on large images. In Section III-B, we propose our
GroupCDL architecture to tackle the problem of image restoration under the
AWGN model (1). AWGN is a popular and successful model for camera noise
following white-balance and gamma-correction [26]. Denoising also serves as a
fundamental building block (ex. proximal operators) and subproblem in various
inverse-problem approaches [14]. Hence, our proposed method has potential for
application beyond camera image denoising. In Section III-C, we demonstrate a
significant appeal of interpretable construction by extending the proposed
method to Compressed Sensing MRI. Sections III-D and III-E give discussion on
the relation between our interpretably constructed nonlocal network and common
black-box approaches.

###  III-A Circulant-Sparse Attention

(a) Patch-based Dense Attention

(b) Circulant-Sparse Attention (CircAtt)

Figure 1:  (a) To enable dense attention, the input image is divided into
overlapping windows (of size $W\times W$ and with window-stride $s_{w}\times
s_{w}$), processed independently via a DNN with dense self-attention. The
denoised windows are then placed in their original positions and averaged on
their overlaps. (b) CircAtt processes the entire image jointly in a single
forward pass. The adjacency matrix has a block-circulant with circulant blocks
(BCCB) sparsity pattern, where the number of non-zeros in each row/column is
at most $W^{2}$. The adjacency matrix is computed in the subband domain, with
spatial dimension $Q=N/s_{c}^{2}$, where $s_{c}$ is the convolution stride.
Hence, the effective image-domain window-size is $s_{c}W\times s_{c}W$.

1 function  _CircDistSim(_$\bm{k}\in\mathbb{C}^{QM}$ ,
$\bm{q}\in\mathbb{C}^{QM}$; $W\in\mathbb{Z}_{+}$_)_:

2
$\bm{S}_{ij}=\begin{cases}-\tfrac{1}{2}\lVert\bm{k}[i]-\bm{q}[j]\rVert_{2}^{2},&\lVert\vec{i}-\vec{j}\rVert_{\infty}\leq
W\\\ -\infty,&\text{otherwise}\end{cases}\quad\forall~{}(i,j)$

3

4 function  _bwd(_$\differential\bm{S}\in\mathbb{B}_{W}^{Q\times Q}$_)_:

5
$\differential{\bm{k}}=\underline{\differential\bm{S}}\bm{q}-(\underline{\differential\bm{S}}\bm{1})\circ\bm{k}$

6
$\differential{\bm{q}}=\underline{\differential\bm{S}}^{T}\bm{k}-(\underline{\differential\bm{S}}^{T}\bm{1})\circ\bm{q}$

7 return $\differential{\bm{k}}$, $\differential{\bm{q}}$

8

9 return $\bm{S}$, bwd

10

Algorithm 1 Circulant-Sparse Distance-Similarity

1 function  _CircAtt(_$\bm{\Gamma}\in\mathbb{B}_{W}^{Q\times Q}$ ,
$\bm{x}\in\mathbb{C}^{QM}$_)_:

2 $\bm{y}=\underline{\bm{\Gamma}}\bm{x}$

3

4 function  _bwd(_$\differential\bm{y}\in\mathbb{C}^{QM}$_)_:

5
$\differential{\bm{\Gamma}}=\operatorname{\mathbf{CircDotSim}}(\differential\bm{y},\bm{x};W)$

6 $\differential{\bm{x}}=\underline{\bm{\Gamma}}^{T}\differential\bm{y}$

7 return $\differential{\bm{\Gamma}}$, $\differential{\bm{x}}$

8

9 return $\bm{y}$, bwd

10

Algorithm 2 Circulant-Sparse Attention

1 function  _CircDotSim(_$\bm{k}\in\mathbb{C}^{QM}$ ,
$\bm{q}\in\mathbb{C}^{QM}$; $W\in\mathbb{Z}_{+}$_)_:

2
$\bm{S}_{ij}=\begin{cases}\operatorname{Re}\\{\bm{q}[j]^{H}\bm{k}[i]\\},&\lVert\vec{i}-\vec{j}\rVert_{\infty}\leq
W\\\ -\infty,&\text{otherwise}\end{cases}\quad\forall~{}(i,j)$

3

4 function  _bwd(_$\differential\bm{S}\in\mathbb{B}_{W}^{Q\times Q}$_)_:

5 $\differential{\bm{k}}=\underline{\differential\bm{S}}\bm{q}$

6 $\differential{\bm{q}}=\underline{\differential\bm{S}}^{T}\bm{k}$

7 return $\differential{\bm{k}}$, $\differential{\bm{q}}$

8

9 return $\bm{S}$, bwd

10

Algorithm 3 Circulant-Sparse Dot-Similarity

The nonlocal self-similarity prior assumes that additional information may be
extracted from images by considering different image pixels, patches, or
latent pixels, jointly. In practice, for an image with $N=N_{1}\times N_{2}$
pixels, this prior is implemented via formation of an adjacency matrix
$\bm{\Gamma}\in\mathbb{R}_{+}^{N\times N}$ (i.e. a row-normalized similarity
matrix), where the $i$th adjacency row ($\bm{\Gamma}_{i:}$) measures closeness
between a pixel/patch/latent at image position $i$ and every other image
pixel/path/latent. For example, we may reshape a latent representation
$\bm{z}\in\mathbb{R}^{NC}$ as $\bm{Z}\in\mathbb{R}^{N\times C}$ to compute the
dense adjacency $\bm{\Gamma}=\operatorname{\mathbf{row-
sm}}(\bm{Z}\bm{Z}^{T})$. Here, $\operatorname{\mathbf{row-sm}}$ refers to the
softmax function applied row-wise to a matrix, $\operatorname{\mathbf{row-
sm}}(\bm{S})_{ij}=\frac{e^{\bm{S}_{ij}}}{\sum_{j}e^{\bm{S}_{ij}}}.$

For computational and memory reasons, it is infeasible to form this dense
adjacency matrix even for reasonably small image sizes222Memory and
computation scale with side-length to the fourth power. A dense Float32
adjacency matrix for a square images of side lengths 256 and 512 would require
17 and 275 GB of memory, respectively. It is thus common and appropriate to
spatially limit our computations to more reasonably sized windows. Ideally, we
seek to limit similarity computations for the $i$th pixel to pixels $j$ within
a distance $W$, i.e. for all $j$ such that $\lVert\vec{i}-\vec{j}\rVert\leq W$
where $\vec{\cdot}$ notation denotes the conversion from linear to cartesian
indexing (see Table I).

Previous architectures have ignored this ideal case for a more practical
solution: independently processing overlapping image patches for the entire
network [2, 6], or latent patches in each layer [7]. After this processing,
image/latent patches may be combined to re-form the image/latent via averaging
patches overlapping regions. We illustrate this “Patch-based Dense Attention”
in Figure 1(a). This patching operation follows arbitrary borders and
necessarily neglects dependencies between pixels of neighboring patches,
despite their proximity.

We propose a return to the original goal of a spatially-limited nonlocal
attention. This goal is achieved by three observations. First, spatially-
limited attention defines a Block-Circulant with Circulant Blocks (BCCB)
sparsity pattern for the adjacency matrix (when circular boundary conditions
are imposed)333This is the sparsity pattern of a 2D single-channel convolution
matrix, as seen illustrated in Figure 1(b). As each row of the BCCB-sparse
matrix (with windowsize $W\times W$) contains $W^{2}$ elements, the non-zero
elements are easily arranged in a matrix $\tilde{\bm{S}}\in\mathbb{R}^{N\times
W^{2}}$. This sparse matrix may be populated with similarity computations in
parallel, and a row-normalization may be performed on $\tilde{\bm{S}}$ using
existing fast softmax algorithms.

Second, the now formed BCCB-sparse adjacency matrix may be applied to a
vectorized image using existing fast sparse-dense matrix multiplication
algorithms.

Third, the back-propagation rules for BCCB-sparse similarity computations and
BCCB-sparse attention make use of and maintain the original BCCB sparsity
pattern. Memory explosion from considering a $N\times N$ adjacency matrix is
not possible, as the sparsity pattern remains fixed. This allows for training
with circulant-sparse attention, enabling effective tackling of image
restoration with global artifacts/degradation operators, as explored in
Section IV-E.

The proposed GroupCDL makes use of a circulant-sparse distance similarity and
circulant-sparse attention, detailed in Algorithms 1, 2. In Section IV-D we
also consider the use of circulant-sparse dot-product similarity (Algorithm
3), and this algorithm is also necessary during the back-propagation of
circulant-sparse attention (regardless of which similarity algorithm is used
in the forward pass; see Supplementary Material). In each of these algorithms,
a backward-pass function is returned which produces the input gradients (ex.
$\differential{\bm{k}},\differential{\bm{q}},\differential{\bm{\Gamma}},\differential{\bm{x}}$)
given output gradients (ex. $\differential\bm{S},\differential\bm{y}$).
Derivations of these back-propagation rules are given in the Supplmentary
Material. Our Circulant-Sparse Attention is implemented in Julia’s CUDA.jl
[27] and is publicly
available444https://github.com/nikopj/CirculantAttention.jl.

###  III-B The GroupCDL Architecture

Figure 2: The GroupCDL Architecture for Denoising: The network begins with no
prior of group-sparsity ($\bm{\Gamma}^{(0)}=\bm{I}$). In the second layer, and
each subsequent $\Delta K$ layers, the adjacency matrix $\bm{\Gamma}^{(k)}$ is
updated by a Circulant(-Sparse) Self-Adjacency computation on the latent
representation $\bm{z}^{(k)}$. Expanded views of GT and Circulant Self-
Adjacency blocks are given in Fig. 3.

(a) Group-Thresholding

(b) Circulant Self-Adjacency

Figure 3:  Expanded block diagrams of GT and Circulant Self-Adjacency from the
GroupCDL Architecture (Fig. 2). Group-Thresholding may be viewed as an image-
adaptive spatially-varying soft-thresholding, informed by adjacency matrix
$\bm{\Gamma}$ and the current features.

We propose a neural network architecture as a direct parameterization of PGM
(4) on the convolutional BPDN problem with a group-sparsity prior, dubbed
GroupCDL. The GroupCDL architecture is equivalent to replacing the CDLNet [1]
architecture’s soft-thresholding (9) with a learned group-thresholding w.r.t
an image-adaptive adjacency matrix $\bm{\Gamma}$, as described in Algorithm 5
and shown in Figure 2. Noise-adaptive thresholds are computed using learned
parameters $\bm{\tau}_{0},\,\bm{\tau}_{1}\in\mathbb{R}^{M}_{+}$ (via
$\bm{\tau}=\bm{\tau}_{0}+\hat{\sigma}\bm{\tau}_{1}$).
${\bm{A}^{T}}^{(k)},\bm{B}^{(k)}$ are 2D ($C$ to $M$ channel, stride-$s_{c}$)
analysis and ($M$ to $C$ channel, stride-$s_{c}$) synthesis convolutions,
respectively. $\bm{D}$ is our 2D ($M$ to $C$ channel, stride-$s_{c}$)
synthesis convolutional dictionary. For an input noisy image
$\bm{y}\in\mathbb{R}^{NC}$, our latent representation is of the form
$\bm{z}\in\mathbb{R}^{QM}$, where $Q=N/s_{c}^{2}$.

The adjacency matrix of the group-sparsity prior
($\bm{\Gamma}\in\mathbb{R}^{Q\times Q}_{+}$) encodes similarity between latent
subband pixels $\bm{z}[i],\,\bm{z}[j]~{}\forall\,i,\,j$. To manage
computational complexity while staying true to the convolutional inductive
bias of the network, we form this adjacency using a sliding-window of size
$W\times W$ (with circular boundary conditions). Specifically, we compute a
circulant-sparse adjacency on a latent representation ($\bm{z}$) of the
network as

$\bm{\Gamma}=\operatorname{\mathbf{row-
sm}}(\operatorname{\mathbf{CircDistSim}}(\overline{\bm{W}}_{\theta}\bm{z},\,\overline{\bm{W}}_{\phi}\bm{z};\,W)),$
(10)

via Algorithm 1, where
$\bm{W}_{\theta},\bm{W}_{\phi}\in\mathbb{R}^{M_{h}\times M}$ are learned
pixel-wise transforms shared across all layers, and $M_{h}$ is a hyper-
parameter. The similarity matrix is normalized via a row-wise softmax
operation ($\operatorname{\mathbf{row-sm}}$). To reduce computational
complexity, we only compute similarity every $\Delta K$ layers. To ensure
smooth adjacency matrix updates, a convex combination of the computed
adjacency and the adjacency of the previous layer is employed via a learned
parameter $\gamma\in[0,1]$ (see Alg. 4).

Mimicking the use of subband transforms in the similarity computation (10), we
introduce two additional subband transforms,
$\bm{W}_{\alpha}\in\mathbb{R}^{M\times
M_{h}},\bm{W}_{\beta}\in\mathbb{R}_{+}^{M\times M_{h}}$, into the group-
thresholding operation,

$\begin{gathered}\operatorname{\mathrm{GT}}_{\bm{\tau}}(\bm{z};\,\bm{\Gamma})=\bm{z}\circ\left(1-\frac{\bm{\tau}}{\bm{\xi}}\right)_{+},\\\
\bm{\xi}=\overline{\bm{W}_{\beta}}\sqrt{(\bm{I}_{M_{h}}\otimes\bm{\Gamma})(\overline{\bm{W}^{T}_{\alpha}}\bm{z})^{2}},\\\
\end{gathered}$ (11)

where $\bm{\xi}\in\mathbb{R}_{+}^{QM}$ informs an image-adaptive spatially-
varying threshold. Here, $\overline{\bm{W}}$ refers to pixel-wise application
of a matrix $\bm{W}$ (see Table I). In contrast to (7), $\bm{W}_{\alpha}$
allows the adjacency-weighted energy of the latent representation to be
computed in a compressed subband domain (by setting $M_{h}<<M$). Then,
$\bm{W}_{\beta}$ maps this energy back to the uncompressed subband domain ($M$
channels), pixel-wise. In Section IV-D, we empirically show that the use of a
compressed subband domain has a positive impact on denoising performance and
an even greater impact on reducing inference time. Figure 3 illustrates the
proposed learned group-thresholding operation in block-diagram form.

1 Input: $\bm{\Gamma}^{(k-1)}$, $\bm{z}^{(k)}$, $\gamma\in[0,1]$, window-size
$W>1$

$\bm{S}=\operatorname{\mathbf{CircDistSim}}(\overline{\bm{W}}_{\theta}\bm{z}^{(k)},\,\overline{\bm{W}}_{\phi}\bm{z}^{(k)};\,W)$
//Alg. 1

2 if  _$k=1$_ then

3 $\bm{\Gamma}^{(1)}=\operatorname{\mathbf{row-sm}}(\bm{S})$

4

5else if  _$\mathrm{mod}(k+1,\,\Delta K)=0$_ then

6 $\bm{\Gamma}^{(k)}=\gamma\operatorname{\mathbf{row-
sm}}(\bm{S})+(1-\gamma)\bm{\Gamma}^{(k-1)}$

7else

8 $\bm{\Gamma}^{(k)}=\bm{\Gamma}^{(k-1)}$

9

10Output: $\bm{\Gamma}^{(k)}$

Algorithm 4 Adjacency Matrix Update Forward Pass

1 Input: noisy image $\bm{y}$, estimated noise-level $\hat{\sigma}$

2 Parameters:
$\Theta=\\{\gamma,\,\bm{W}_{\\{\theta,\phi,\alpha,\beta\\}},\,\bm{D},\,\\{{\bm{A}^{T}}^{(k)},\,\bm{B}^{(k)},\,\bm{\tau}^{(k)}_{\\{0,1\\}}\\}_{0=1}^{K-1}\\}$

3 Preprocess: $\tilde{\bm{y}}=\bm{y}-\mu$, $\mu=\mathrm{mean}(\bm{y})$

4 Initialize: $\bm{z}^{(0)}=\mathbf{0}$, $\bm{\Gamma}^{(0)}=\bm{I}$,
$\bm{\tau}^{(k)}=\bm{\tau}^{(k)}_{0}+\hat{\sigma}\bm{\tau}^{(k)}_{1}\,\,\forall\,\,k$

5 for  _$k=0,1,\dots,K-1$_ do

//Update Adj. using Alg 4

6 $\bm{\Gamma}^{(k)}\leftarrow\mathrm{\bf
AdjUpdate}(\bm{\Gamma}^{(k-1)},\bm{z}^{(k)},\gamma)$

7 $\bm{v}={\bm{A}^{T}}^{(k)}(\bm{B}^{(k)}\bm{z}^{(k)}-\tilde{\bm{y}})$

$\bm{z}^{(k+1)}=\operatorname{\mathrm{GT}}_{\bm{\tau}^{(k)}}\left(\bm{z}^{(k)}-\bm{v};\bm{\Gamma}^{(k)}\right)$
//Eq. (11)

8

9Output: $\hat{\bm{x}}=\bm{D}\bm{z}^{(K)}+\mu$

Algorithm 5 GroupCDL Denoising Forward Pass

###  III-C Extending the GroupCDL Architecture for CS-MRI

1 Input: zero-filled Fourier domain $\bm{y}$, estimated noise-level
$\hat{\sigma}$, Fourier domain mask operator $\bm{M}$, coil sensitivity
operator $\bm{R}$

2 Parameters:
$\Theta=\\{\gamma,\,\bm{W}_{\\{\theta,\phi,\alpha,\beta\\}},\,\bm{D},\,\\{{\bm{A}^{H}}^{(k)},\,\bm{B}^{(k)},\,\bm{\tau}^{(k)}_{\\{0,1\\}}\\}_{0=1}^{K-1}\\}$

3 Preprocess: $\tilde{\bm{y}}=\bm{R}^{H}\bm{F}^{H}\bm{y}-\mu$,
$\mu=\mathrm{mean}(\bm{R}^{H}\bm{F}^{H}\bm{y})$

4 Initialize: $\bm{z}^{(0)}=\mathbf{0}$, $\bm{\Gamma}^{(0)}=\bm{I}$,
$\bm{\tau}^{(k)}=\bm{\tau}^{(k)}_{0}+\hat{\sigma}\bm{\tau}^{(k)}_{1}\,\,\forall\,\,k$

5 for  _$k=0,1,\dots,K-1$_ do

//Update Adj. using Alg 4

6 $\bm{\Gamma}^{(k)}\leftarrow\mathrm{\bf
AdjUpdate}(\bm{\Gamma}^{(k-1)},\,\bm{z}^{(k)},\,\gamma)$

7
$\bm{v}={\bm{A}^{H}}^{(k)}(\bm{R}^{H}\underline{\bm{F}^{H}\bm{M}\bm{F}}\bm{R}\bm{B}^{(k)}\bm{z}^{(k)}-\tilde{\bm{y}})$

$\bm{z}^{(k+1)}=\operatorname{\mathrm{GT}}_{\bm{\tau}^{(k)}}\left(\bm{z}^{(k)}-\bm{v};\bm{\Gamma}^{(k)}\right)$
//Eq. (11)

8

9Output: $\hat{\bm{x}}=\bm{D}\bm{z}^{(K)}+\mu$

Algorithm 6 GroupCDL CS-MRI Forward Pass Figure 4:  Block diagram of a single
GroupCDL layer in Fig. 2 extended to general linear inverse problem
$\bm{y}=\bm{H}\bm{x}+\nu$. The layer differs from the denoising GroupCDL by
adjoint operator preprocessing and use of the Gram Operator $\bm{H}^{H}\bm{H}$
in each layer. In the case of CS-MRI, the observation operator is a masked
multi-coil Fourier transform following a sensitivity map operator,
$\bm{H}=\underline{\bm{M}\bm{F}}\bm{R}$. Other blocks correspond with Figure
2.

Parallel (or multi-coil) compressed sensing magnetic resonance imaging (CS-
MRI) may in general be described by the following observation model,

$\bm{y}=\underline{\bm{M}}(\underline{\bm{F}}\bm{R}\bm{x}+\bm{\nu}),\quad\bm{\nu}\sim\mathcal{CN}(0,\overline{\bm{\Sigma}}).$
(12)

where $\bm{x}\in\mathbb{C}^{N}$ is the ground-truth image-domain signal and
$\bm{y}\in\mathbb{C}^{NC}$ is a complex-valued measurement vector with
$C$-channels acquired in a masked Fourier domain (where $C$ is the number of
scanner coils). Here, $\bm{M}=\operatorname{\mathbf{diag}}(\bm{m})$ is
channel-wise mask operator with binary mask $\bm{m}\in\\{0,1\\}^{N}$
indicating the position of acquired Fourier domain samples. $\bm{F}$
represents the $N$-dimensional DFT matrix, which is applied channel-wise. The
coil-sensitivities are encoded in the sensitivity operator
$\bm{R}=\begin{bmatrix}\operatorname{\mathbf{diag}}(\bm{r}_{1})&\cdots&\operatorname{\mathbf{diag}}(\bm{r}_{C})\end{bmatrix}^{T}$
via sensitivity map $\bm{r}\in\mathbb{C}^{NC}$. The acquired samples are well
modeled as being contaminated by i.i.d. complex additive Gaussian noise with
noise-covariance matrix $\bm{\Sigma}\in\mathbb{C}^{C\times C}$. This
observation model may be simplified by considering a coil-whitening pre-
processing transformation, resulting in the noise following a diagonal
covariance matrix of equal power in each coil, i.e.
$\bm{\nu}\sim\mathcal{CN}(0,\sigma^{2}\bm{I})$. After coil-whitening, the CS-
MRI convolutional dictionary BPDN functional takes the form,

$\underset{\bm{z}}{\mathrm{minimize}}~{}\frac{1}{2}\lVert\bm{y}-\underline{\bm{M}\bm{F}}\bm{R}\bm{D}\bm{z}\rVert_{2}^{2}+\lambda\psi(\bm{z}).$
(13)

We derive the CS-MRI GroupCDL (and CS-MRI CDLNet) architectures as a directly
parameterized unrolling of PGM applied to (13), employing complex-valued
filter-banks/dictionaries. Inverse DFT and coil-combination are applied to the
input Fourier domain signal, along with mean-subtraction pre-processing.
Correspondingly, the observation Gram operator
$\bm{R}^{H}\underline{\bm{F}^{H}\bm{M}\bm{F}}\bm{R}$ is inserted after
synthesis convolution in each layer. These changes are summarized in Algorithm
6 and Figure 4.

Whereas the standard i.i.d. denoising problem involves a purely local (pixel-
wise) image degradation, CS-MRI represents a global image degradation via
aliasing artifacts. Circulant-Sparse Attention is especially well suited for
tackling the CS-MRI as global aliasing artifacts can be taken into account via
joint inference over the entire image with attention computations seamlessly
centered on each pixel of interest. In contrast, patch-based dense attention
arbitrarily divides its field of view, limiting its modeling capabilities for
global degradation operators.

###  III-D Computational Burden of Patch-based Dense Attention

The circulant-sparse attention employed by GroupCDL is favorable to the patch-
based dense attention employed by GroupSC [2] and black-box DNNs [6, 7],
because it naturally encourages agreement on overlapping regions and centers
the attention windows on each pixel. As shown in Figure 1(a), patch-based
dense attention may additionally incur computational overhead by processing
overlapping pixels multiple times. Let $N=N_{1}\times N_{2}$ be the number of
pixels in an image processed by window-size $W\times W$ (for both patch-
attention and CircAtt), and denote window-stride $s_{w}\times s_{w}$. We
express the burden factor as a ratio of the number of pixels processed by a
single patch-based dense attention layer over a CircAtt layer,

$\frac{N_{1}/s_{w}\times N_{2}/s_{w}\times W^{2}}{N_{1}\times
N_{2}}=\frac{W^{2}}{s_{w}^{2}}.$ (14)

Common nonlocal window sizes, $45\times 45$, and window-strides, $7\times 7$,
such as used by NLRN [6], make this burden factor 41 times the computational
complexity of an equivalent CircAtt implementation. GroupCDL’s use of strided
convolution may add an additional $s_{c}^{2}\times$ computational benefit
compared to common NLSS implementations by computing similarities over a
reduced spatial dimension $Q=N/s_{c}^{2}$.

Note that shifted-window (Swin) attention (employed by SwinIR [7]) side-steps
this burden factor by patching without overlap ($s_{w}=W$) twice
(successively) with shifted patch boundaries. Swin attention may also enjoy
enhanced processing speeds due to its use of dense-array arithmetic vs.
CircAtt’s use of sparse-array arithmetic. Yet, Swin attention does not address
the modeling deficiencies inherent with patch-based dense attention. This is
demonstrated in experimentally by GroupCDL’s on-par performance with SwinIR at
a fraction of its learned parameter count, in Section IV-B.

We further explore the relation between computation time and denoising
performance of patch-based dense attention vs CircAtt in the Supplementary
Material.

###  III-E Group-Thresholding vs. Black-box Attention

Nonlocal self-similarity is used across domains in DNNs, from transformer
architectures [24] to nonlocal image restoration networks [6, 7]. The
underlying formula behind these methods is most commonly dot-product attention
(DPA), easily expressed in terms of the reshaped latent
$\bm{Z}\in\mathbb{R}^{N\times M_{\mathrm{in}}}$ as,

$\begin{gathered}\bm{K}=\bm{Z}^{(k)}\bm{W}_{k}^{T},\quad\bm{Q}=\bm{Z}^{(k)}\bm{W}_{q}^{T},\quad\bm{V}=\bm{Z}^{(k)}\bm{W}_{v}^{T}\\\
\bm{\Gamma}=\operatorname{\mathbf{row-sm}}(\bm{K}\bm{Q}^{T})\\\
\bm{Z}^{(k+1)}=\bm{\Gamma}\bm{V}\end{gathered}$ (15)

where $\bm{W}_{q},\bm{W}_{k},\in\mathbb{R}^{M_{h}\times M_{\mathrm{in}}}$,
$\bm{W}_{v}\in\mathbb{R}^{M_{\mathrm{out}}\times M_{\mathrm{in}}}$, and
$\bm{Z}^{(k+1)}\in\mathbb{R}^{N\times M_{\mathrm{out}}}$.

Both DPA and the proposed GT (11) make use of a normalized adjacency matrix
($\bm{\Gamma}$), computed in an asymmetric feature
domain555($\bm{W}_{k}\neq\bm{W}_{q}$, $\bm{W}_{\theta}\neq\bm{W}_{\phi}$).
Both use this adjacency to weight the current spatial features, identically
over channels. However, in DPA, the weighting directly results in the layer’s
output (via matrix multiplication), whereas in GT this weighting informs a
spatially adaptive soft-thresholding (see Figure 3).

The proposed GT’s decoupling of adjacency application and output dimension is
key in allowing group-thresholding to be computationally efficient, as the
adjacency matrix-vector multiplication can be performed over transformed
features with a reduced dimension. In contrast, DPA operating with reduced
channel dimensions ($M_{\mathrm{out}}<<M_{\mathrm{in}}$) would harm the
capacity of the network’s latent representation. In Section IV-D we show
empirical evidence for favoring the distance similarity in GT over the dot-
product similarity of DPA.

##  IV Experimental Results

###  IV-A Natural Image Denoising Experimental Setup

Architecture: We denote the network detailed in Algorithm 5 as GroupCDL.
GroupCDL and CDLNet are trained with noise-adaptive thresholds
($\bm{\tau}^{(k)}=\bm{\tau}^{(k)}_{0}+\hat{\sigma}\bm{\tau}^{(k)}_{1}$) unless
specified using the -B suffix, indicating the models are noise-blind
($\bm{\tau}^{(k)}=\bm{\tau}^{(k)}_{0}$). The hyperparameters for these
architectures are given in Table II, unless otherwise specified.

TABLE II: Architectures of the GroupCDL models, CDLNet models, and variants
presented in the experimental section. Conv-stride $s_{c}=2$ for all models.

Name | Task | $p$ | $K$ | $M$ | $M_{h}$ | $W$ | $\Delta K$
---|---|---|---|---|---|---|---
CDLNet | Denoise | 7 | 30 | 169 | - | - | -
GroupCDL | Denoise, CS-MRI | 7 | 30 | 169 | 64 | 35 | 5
Big-GroupCDL | Denoise | 9 | 40 | 448 | 128 | 45 | 10

Dataset and Training: Let $f_{\Theta}$ denote the GroupCDL DNN as a function
of parameters $\Theta$. Let $\mathcal{D}=\\{(\bm{y},\sigma,\bm{x})\\}$ denote
a dataset of noisy and ground-truth natural image pairs, with noise-level
$\sigma$. Grayscale image denoising GroupCDL models were trained on the BSD432
[28] dataset. Grayscale Big-GroupCDL models were additionally trained on the
Waterloo Exploration dataset and DIV2K dataset. All models were trained with a
supervised mean squared error (MSE) loss,

$\underset{\begin{subarray}{c}\bm{W}_{\theta},~{}\bm{W}_{\phi},~{}\bm{W}_{\alpha},\\\
\bm{W}_{\beta}\geq 0,~{}\gamma\in[0,1],\\\
\bm{D}\in\mathcal{C},~{}\\{\bm{\tau}^{(k)}\geq 0\\}_{k=0}^{K-1},\\\
\\{\bm{A}^{(k)}\in\mathcal{C},~{}\bm{B}^{(k)}\in\mathcal{C}\\}_{k=0}^{K-1}\end{subarray}}{\mathrm{minimize}}\quad\sum_{\\{\bm{y},\sigma,\bm{x}\\}\in\mathcal{D}}\lVert
f_{\Theta}(\bm{y},\sigma)-\bm{x}\rVert_{2}^{2},$ (16)

where $\mathcal{C}=\\{\bm{D}\,:\,\lVert\bm{D}_{:j}\rVert_{2}^{2}\leq
1~{}\forall\,j\\}$. We use the Adam optimizer with default parameters [29],
and project the network parameters onto their constraint sets after each
gradient step. The dataset is generated online from clean images via random
crops, rotations, flips, and AWGN of noise-level $\sigma$ sampled uniformly
within $\sigma^{\mathrm{train}}$ for each mini-batch element. A mini-batch
size of 12 was used. Initial learning-rates of $5\times 10^{-4}$ and $3\times
10^{-4}$ were used for GroupCDL and Big-GroupCDL, respectively, with cosine-
annealing [30] to a final value of $2\times 10^{-6}$. Each network was trained
for $\approx 600k$ gradient steps. Additional training hyperparamter details
(such as back-tracking) follow the CDLNet setup [1].

Test and validation performance is evaluated on several datasets. The dataset
name, along with (arithmetic) average dimensions, are provided to better
understand reported inference timings: Set12 (362 $\times$ 362), CBSD68 [28]
(481 $\times$ 321), Urban100 [31] (1030 $\times$ 751), and NikoSet10666see
https://github.com/nikopj/GroupCDL. (1038 $\times$ 779).

Training Initialization: GroupCDL models are initialized as
ISTA777$(\bm{A}^{(k)}=\bm{B}^{(k)}=\bm{D}$ $\forall$ $k$) with
$\bm{\tau}_{0}=10^{-3},~{}\bm{\tau}_{1}=0$, and a base dictionary $\bm{D}$
that has been spectrally normalized. Details are given in [1]. Pixel-wise
transforms $\bm{W}_{\\{\theta,\phi,\alpha,\beta\\}}$ are initialized with the
same weights drawn from a standard uniform distribution and spectrally-
normalized. We initialize parameter $\gamma=0.8$.

Hardware: All models were trained on a single core Intel Xeon CPU with 2.90
GHz clock and a single NVIDIA A100 GPU. GroupCDL training takes approximately
48 hours. For code-base compatibility reasons, inference timings for methods
(GroupCDL, GroupSC, NLRN, GCDN) in Tables III were determined by running
models an NVIDIA Quadro RTX-8000, whereas for other listed methods (Big-
GroupCDL, SwinIR, Restormer, DCDicl) we report NVIDIA A100 inference time.

###  IV-B Single Noise-Level Performance

In this section, we demonstrate competitive denoising performance of the
proposed GroupCDL. All models are trained on a single noise-level and tested
at the same noise-level ($\sigma^{\mathrm{train}}=\sigma^{\mathrm{test}}$). We
compare GroupCDL to its fully convolutional counterpart (CDLNet), patch-
processing dictionary learning based nonlocal DNN (GroupSC) [2], and state-of-
the-art black-box DNNs [32, 6, 7, 22, 18].

TABLE III:  Grayscale denoising performance (PSNR (dB)/ $100\times$SSIM) and
GPU inference runtimes. All learned methods are trained for individual noise-
levels ($\sigma=\sigma^{\mathrm{train}}=\sigma^{\mathrm{test}}$). Learned
parameter counts are displayed below the method names.

Dataset |  Noise $\sigma$ |  |  CDLNet 507k [1] |  GroupSC 68k [2] |  NLRN 340k [6] |  GroupCDL 550k |  GCDN 6M [32] |  |  SwinIR 11M [7] |  Restormer 25M [22] |  DCDicL 33M [18] |  Big-GroupCDL 3M
---|---|---|---|---|---|---|---|---|---|---|---|---
Set12 | 15 |  | 32.87/90.43 | 32.85/90.63 | 33.16/90.70 | 33.05/90.73 | 33.14/90.72 |  | 33.36/91.11 | 33.42/91.28 | 33.34/91.15 | 33.24/91.02
25 |  | 30.52/86.55 | 30.44/86.42 | 30.80/86.89 | 30.75/86.93 | 30.78/86.87 |  | 31.00/87.38 | 31.08/87.64 | 31.03/87.48 | 30.91/87.35
50 |  | 27.42/79.41 | 27.14/77.97 | 27.64/79.80 | 27.63/80.04 | 27.60/79.57 |  | 27.87/80.84 | 27.95/81.18 | 28.00/81.22 | 27.85/80.82
time (s) |  |  | 0.019 | 22.07 | 25.62 | 0.68 | 405 |  | 1.88 | 0.13 | 0.42 | 1.60
BDS68 [28] | 15 |  | 31.74/89.18 | 31.70/89.63 | 31.88/89.32 | 31.82/89.41 | 31.70/89.63 |  | 31.96/89.60 | 31.96/89.65 | 31.95/89.57 | 31.90/89.57
25 |  | 29.26/83.06 | 29.20/83.36 | 29.41/83.31 | 29.38/83.51 | 29.35/83.32 |  | 29.50/83.77 | 29.52/83.90 | 29.52/83.79 | 29.46/83.76
50 |  | 26.35/72.69 | 26.17/71.83 | 26.47/72.98 | 26.47/73.32 | 26.38/73.89 |  | 26.57/73.79 | 26.60/73.99 | 26.63/73.95 | 26.57/73.83
time (s) |  |  | 0.022 | 23.63 | 26.66 | 0.65 | 540 |  | 1.43 | 0.14 | 0.38 | 1.67
Urban100 [31] | 15 |  | 32.59/92.85 | 32.72/93.08 | 33.42/93.48 | 33.07/93.40 | 33.47/93.58 |  | 33.75/93.98 | 33.79/94.09 | 33.59/93.88 | 33.56/93.89
25 |  | 30.03/89.00 | 30.05/89.12 | 30.88/90.03 | 30.61/90.03 | 30.95/90.20 |  | 31.30/90.94 | 31.46/91.29 | 31.30/91.08 | 31.20/90.93
50 |  | 26.66/81.11 | 26.43/80.02 | 27.40/82.44 | 27.29/83.05 | 27.41/81.60 |  | 28.05/84.84 | 28.29/85.60 | 28.24/85.49 | 28.03/84.83
time (s) |  |  | 0.090 | 93.33 | 135.8 | 3.56 | 1580 |  | 4.98 | 0.60 | 1.75 | 8.32

Table III shows the grayscale denoising performance and inference speed of the
aforementioned models across several common datasets. We include a learned
parameter count as a crude measure of expressivity of the model. The group-
sparsity prior of GroupCDL significantly increases denoising performance
compared to the unstructured sparsity prior of CDLNet. We observe that
GroupCDL has denoising performance superior to other dictionary learning based
networks using group sparsity prior (GroupSC) and competitive performance with
black-box nonlocal methods GCDN and NLRN. GroupCDL also significantly
outperforms these methods in inference time due to the computational
efficiency of CircAtt over overlapping patch-based dense attention. Big-
GroupCDL is shown to outperform GCDN and be competitive with state-of-the-art
black-box networks SwinIR, Restormer, and DCDicl, with on par inference time
and only a fraction of the learned parameters.

(a) GT
PSNR/$100\times$SSIM

(b) Noisy ($\sigma=50$)
14.86/35.51

(c) SwinIR [7]
25.12/79.18

(d) Restormer [22]
25.17/79.72

(e) Big-GroupCDL-S
25.07/79.25

(f) GT
PSNR/$100\times$SSIM

(g) Noisy ($\sigma=50$)
15.24/28.80

(h) SwinIR [7]
26.49/81.69

(i) Restormer [22]
26.55/82.06

(j) Big-GroupCDL-S
26.50/81.78

(k) GT
PSNR/$100\times$SSIM

(l) Noisy ($\sigma=50$)
15.30/24.49

(m) SwinIR [7]
25.03/82.73

(n) Restormer [22]
25.11/83.28

(o) Big-GroupCDL-S
24.95/82.25

Figure 5:  Visual comparison of large paramter count deep denoisers.

Figure 5 highlights the qualitative differences between Big-GroupCDL and
state-of-the-art black-box networks (SwinIR [7], Restormer [22]). We observe
that Big-GroupCDL is able to produce denoised images on par with SwinIR and
Restormer (Fig. 5 first row red box), and may even retain a greater amount of
detail information where SwinIR and Restormer appear to over-smooth detailed
regions (Fig. 5 second row red box, third row red box) or hallucinate image
structures (Fig. 5 second row blue box).

###  IV-C Noise-Level Generalization

Figure 6:  Left: Noise-level generalization of different grayscale denoising
networks tested on BSD68 [28]. GroupCDL-S is trained at
$\sigma^{\mathrm{test}}$ for each point on the graph. All other networks are
trained on $\sigma^{\mathrm{train}}=[20,30]$. Right: Visualization of
normalized adjacency $\bm{\Gamma}^{(K)}_{i:}$ for $i$th pixel of input image
$(\bm{y})$ at noise-levels $\sigma=10,25,50$. Catastrophic failure of
GroupCDL-B model is observed for $\sigma>\sigma^{\mathrm{train}}$.

Figure 6 shows that the proposed novel group-thresholding scheme (11) is able
to obtain near-perfect noise-level generalization (w.r.t GroupCDL-S
performance). This serves as empirical evidence for the interpretation of the
unrolled network as performing some approximate/accelerated group-sparse BPDN,
as the noise-adaptive thresholds
($\bm{\tau}=\bm{\tau}_{0}+\hat{\sigma}\bm{\tau}_{1}$; $\hat{\sigma}$ estimated
following [1]) appear to correspond very well to their classical counter-parts
from which they are derived.

Figure 6 also shows a single input nonlocal window $\bm{y}$ across noise-
levels $\sigma$ and the computed adjacency values of the three types of
GroupCDL models (GroupCDL-S, GroupCDL-B, GroupCDL). The adjacency
visualizations of GroupCDL-B show a catastrophic failure in the similarity
computations of the network above $\sigma^{\mathrm{train}}$, as no structure
is found. Similar patterns are seen in the GroupCDL models (with noise-
adaptive thresholds) compared to those of GroupCDL-S.

###  IV-D Ablation Studies

In this section we examine the denoising and inference time performance of the
GroupCDL model under different hyperparameters associated with the proposed
group-thresholding operation (11), (10). Table IV shows the effect of the
update-interval parameter ($\Delta K$), which determines how often we update
the adjacency $\bm{\Gamma}^{(k)}$ in the network (every $\Delta K$ layers, see
Alg. 5). We observe that decreasing the update interval increases denoising
performance, with diminishing returns, at cost to the inference speed of the
network.

TABLE IV: Effect of update interval ($\Delta K$). Grayscale denoising performance averaged over the NikoSet10 dataset ($\sigma^{\mathrm{train}}=\sigma^{\mathrm{test}}=25$). $\Delta K$ | PSNR/$100\times$SSIM | time (s)
---|---|---
2 | 30.22/86.88 | 4.01
3 | 30.22/86.87 | 4.72
5 | 30.21/86.87 | 3.47
10 | 30.21/86.87 | 3.24
15 | 30.19/86.79 | 3.18
TABLE V: Effect of NLSS feature compression. Grayscale denoising performance averaged over the NikoSet10 dataset ($\sigma^{\mathrm{train}}=\sigma^{\mathrm{test}}=25$). feature-compression | $M_{h}$ | sim_fun | PSNR/$100\times$SSIM | time (s)
---|---|---|---|---
none | n/a | distance | 30.19/86.79 | 8.64
$\bm{W}_{\\{\theta,\phi\\}}$ | 64 | distance | 30.21/86.86 | 8.13
$\bm{W}_{\\{\theta,\phi,\alpha,\beta\\}}$ | 169 | distance | 30.21/86.89 | 8.89
$\bm{W}_{\\{\theta,\phi,\alpha,\beta\\}}$ | 64 | distance | 30.21/86.87 | 3.47
$\bm{W}_{\\{\theta,\phi,\alpha,\beta\\}}$ | 32 | distance | 30.20/86.84 | 2.00
$\bm{W}_{\\{\theta,\phi,\alpha,\beta\\}}$ | 64 | dot | 30.11/86.55 | 3.43

Table V shows the effect of employing learned pixel-wise transforms in the
similarity computation (10) ($\bm{W}_{\theta}$, $\bm{W}_{\phi}$) and group-
thresholding (11) ($\bm{W}_{\alpha}$, $\bm{W}_{\beta}$). The table also shows
the effect of employing channel reduction in these transforms ($M_{h}<<M=169$)
and a comparison of using the distance similarity (Alg. 1) vs. dot-product
similarity (Alg. 3). We observe that pixel-wise transforms increases denoising
performance. Furthermore, setting $M_{h}<<M$ roughly preserves the
performance, while greatly reducing the inference time, with the reduction
roughly equal to the channel reduction ratio $M/M_{h}=169/64\approx
8.89/3.47$. This demonstrates one advantage of group-thresholding over black-
box dot-product attention: the dimension for similarity calculation and
attention operation ($M_{h}$) is decoupled from the layer’s output channel
dimension and may be tuned to achieve a better trade-off between speed and
performance. Distance-similarity is also empirically shown to achieve better
performance over dot-product similarity.

###  IV-E Application to Compressed-Sensing MRI

TABLE VI: CS-MRI reconstruction performance (PSNR (dB)/$100\times$SSIM) . All
learned methods are trained on the MoDL Brain dataset [36]. Learned parameter
counts are displayed below the method names. ∗Numbers reported in [37].

Accel. | Zero-Filled | MoDL∗ [36] | ISTA-Net$+$∗ [38] | VS-Net∗ [39] | HFMRI∗ [37] | CDLNet | GroupCDL
---|---|---|---|---|---|---|---
- | 5.6M | 368k | 1.0M | 1.1M | 1.0M | 1.1M
4x | 25.60/80.61 | 28.89/80.1 | 32.26/91.6 | 32.18/91.4 | 33.20/93.8 |  36.20/96.01 |  36.55/95.97
8x | 23.24/72.91 | 25.28/71.1 | 27.48/83.1 | 27.56/83.3 | 27.65/85.3 |  29.93/89.47 |  30.49/89.94

(a) GT
PNSR/$100\times$SSIM

(b) Zero-Filled
23.11/74.84

(c) HFMRI$\dagger$ [37]
33.20/93.80

(d) CDLNet
32.22/92.87

(e) GroupCDL
33.34/93.17

(f) Zero-Filled
21.01/66.30

(g) HFMRI$\dagger$ [37]
27.65/85.30

(h) CDLNet
28.56/86.93

(i) GroupCDL
29.27/88.15

Figure 7:  Visual comparison of multi-coil CS-MRI reconstruction models on a
selected crop of slice 111 from the MODL Brain test set [36]. Magnitude error
maps shown below each crop. Top-row: 4x acceleration. Bottom-row: 8x
acceleration. †figure and numbers taken from [37].

(a) GT
PNSR/$100\times$SSIM

(b)

(c) Zero-Filled
22.58/74.34

(d) GroupCDL-B
29.62/82.12

(e) GroupCDL
30.23/86.85

(f) Zero-Filled
20.86/58.34

(g) GroupCDL-B
24.66/72.25

(h) GroupCDL
25.37/77.13

Figure 8:  Visual comparison of Joint-Denoising and CS-MRI reconstruction
models with (GroupCDL) and without noise-adaptive thresholds (GroupCDL-B) on
MODL Brain test set [36]. Each model was trained for 4$\times~{}$acceleration
on a noise-range $\sigma^{\mathrm{train}}\in[0.08,0.12]$. Top-row:
$\sigma^{\mathrm{test}}=0.02$. Bottom-row: $\sigma^{\mathrm{test}}=0.25$.

In this section, we apply the GroupCDL architecture (detailed in Section
III-C) to supervised training for the CS-MRI problem, with and without noise
contamination.
Experimental Setup: We use the fully-sampled MoDL multi-coil brain dataset
[36], which was acquired using a 3D T2 CUBE sequence with an isotropic 1 mm
resolution, a 210 mm $\times$ 210 mm field-of-view, and echo-time of 84 ms.
The dataset contains 360 slices for training and 164 for testing. Sensitivity
maps are provided by the dataset. We follow the experimental setup of [37] by
using random Cartesian Fourier domain subsampling (in a single readout
direction) for 4$\times~{}$and 8$\times~{}$acceleration with $8\%$ and $4\%$
of center lines preserved, respectively.
Training: We train CDLNet and GroupCDL models for CS-MRI using identical model
and training hyperparameters as the grayscale denoising models (see Table II),
though each filter is now complex-valued (nearly doubling the learned
parameter count). Following [37], we replace the standard MSE loss function
with an $\ell_{1}-\mathrm{ssim}$ loss function for fair comparison. PSNR and
SSIM computations use a peak-value of $1.0$, following [37].

No Noise: Table VI shows the results of training the proposed GroupCDL against
state-of-the-art deep-learning CS-MRI reconstruction, without the presence of
additive noise. Both GroupCDL and CDLNet perform significantly better than
state-of-the-art methods. GroupCDL distinguishes itself from CDLNet at the
higher acceleration reconstruction. These observations are bolstered in Figure
7 where the black-box DNN HFMRI [37] is shown to have significantly more
structure present in their reconstruction error maps compared to GroupCDL.

With Noise: Figure 8 shows the results of CS-MRI reconstruction training in
the presence of Fourier domain AWGN, across noise-levels. The proposed CS-MRI
GroupCDL model was trained for 4$\times~{}$reconstruction on an intermediate
noise-range ($\sigma^{\mathrm{train}}=[0.08,0.12]$)888with an image-domain
maximum magnitude of $1.6$. Such a training may be encountered in practice
where ground-truth images are unavailable and the limited available data is
corrupted with varying noise-levels. Figure 8(b) shows that GroupCDL’s noise-
adaptive thresholds enable effective inference both above and below the
training noise-range. Figure 8’s visual examples show that GroupCDL-B is
unable to adequately recover features/textures below the training noise-range,
and introduces severe artifacts above it. In contrast, GroupCDL (with noise-
adaptive thresholds) is able to recover additional image details below the
training range, and generalize gracefully above it.

##  V Discussion and Conclusion

In GroupCDL, we adapt the classical, patch-processing based, group-sparsity
prior to convolutional sparse coding (CSC) and apply it to the direct-
parametrization unrolling frame-work of CDLNet [1]. In doing so, we arrive at
a sliding-window NLSS consistent with the CSC model (CircAtt), which addresses
the modeling pitfalls of patch-based dense attention. We demonstrate
competitive denoising performance to state-of-the-art black-box networks with
notable efficiency in the number of learned parameters. We show that the
CircAtt operation allows GroupCDL to be easily and effectively extended to
global degradation operator inverse-problems such as CS-MRI. GroupCDL brings
substantial gain over current state-of-the-art black-box nonlocal networks for
CS-MRI in the case of zero-noise reconstruction.

Additionally, GroupCDL inherits the noise-level generalization capabilities of
CDLNet [1]. This robustness to training-inference mismatch may be of special
interest to practical scenarios, where ground-truth is unavailable and
inference noise-levels may deviate from what is present in a training dataset.
We leverage GroupCDL’s noise-adaptive thresholds to recover image details
especially well in such cases. To the best of our knowledge, robustness to
training-inference noise-level mismatch for CS-MRI has not been considered in
prior literature.

When scaled up, GroupCDL performs very competitively with state-of-the-art
black-box DNNs, even without incorporating existing state-of-the-art modeling
choices such as multi-resolution processing and transposed-attention. Future
work may consider additionally incorporating these features into the GroupCDL
framework to enable even better performance and inference speeds. We believe
GroupCDL’s interpretability and robustness are well suited to tackle other
large signal reconstruction problems with nonlocal image-domain artifacts,
especially in the unsupervised learning regime.

## Acknowledgments

The authors would like to thank NYU HPC for its computing resources and
technical support. The authors are grateful to Che Maria Baez for her
linguistic revisions on a preliminary draft of this manuscript.

## References

  * [1] N. Janjušević, A. Khalilian-Gourtani, and Y. Wang, “CDLNet: Noise-adaptive convolutional dictionary learning network for blind denoising and demosaicing,” _IEEE Open Journal of Signal Processing_ , vol. 3, pp. 196–211, 2022.
  * [2] B. Lecouat, J. Ponce, and J. Mairal, “Fully trainable and interpretable non-local sparse models for image restoration,” in _European Conference on Computer Vision (ECCV)_ , 2020.
  * [3] N. Janjušević, A. Khalilian-Gourtani, and Y. Wang, “Gabor is enough: Interpretable deep denoising with a gabor synthesis dictionary prior,” in _2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP)_ , 2022, pp. 1–5.
  * [4] D. Simon and M. Elad, “Rethinking the CSC model for natural images,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 2274–2284.
  * [5] M. Scetbon, M. Elad, and P. Milanfar, “Deep k-svd denoising,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 5944–5955, 2019.
  * [6] D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang, “Non-local recurrent network for image restoration,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 1680–1689.
  * [7] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_ , October 2021, pp. 1833–1844.
  * [8] J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, “Julia: A fresh approach to numerical computing,” _SIAM Review_ , vol. 59, no. 1, pp. 65–98, 2017. [Online]. Available: https://doi.org/10.1137/141000671
  * [9] S. Mallat, _A Wavelet Tour of Signal Processing: The Sparse Way_. Elsevier Science, 2008.
  * [10] A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” _SIAM Journal on Imaging Sciences_ , vol. 2, pp. 183–202, 01 2009.
  * [11] J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman, “Non-local sparse models for image restoration,” in _Proceedings of 12th IEEE International Conference on Computer Vision (ICCV)_ , 2009, pp. 2272–2279.
  * [12] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online dictionary learning for sparse coding,” in _Proceedings of the 26th International Conference on Machine Learning (ICML)_ , 2009, pp. 689–696.
  * [13] G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, “Deep learning techniques for inverse problems in imaging,” _IEEE Journal on Selected Areas in Information Theory_ , vol. 1, no. 1, pp. 39–56, 2020.
  * [14] D. Gilton, G. Ongie, and R. Willett, “Neumann networks for linear inverse problems in imaging,” _IEEE Transactions on Computational Imaging_ , vol. 6, pp. 328–343, 2019.
  * [15] ——, “Deep equilibrium architectures for inverse problems in imaging,” _IEEE Transactions on Computational Imaging_ , vol. 7, pp. 1123–1133, 2021\.
  * [16] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” _Medical Image Computing and Computer-Assisted Intervention_ , p. 234–241, 2015.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [18] H. Zheng, H. Yong, and L. Zhang, “Deep convolutional dictionary learning for image denoising,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 630–641.
  * [19] H. Sreter and R. Giryes, “Learned convolutional sparse coding,” in _Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 2191–2195.
  * [20] K. Gregor and Y. LeCun, “Learning fast approximations of sparse coding,” in _Proceedings of the 27th International Conference on Machine Learning (ICML)_ , 2010, pp. 399–406.
  * [21] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * [22] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in _CVPR_ , 2022.
  * [23] R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” _URL https://openai.com/blog/sparse-transformers_ , 2019.
  * [24] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” in _Advances in Neural Information Processing Systems_ , 2022.
  * [25] Y. Mei, Y. Fan, and Y. Zhou, “Image super-resolution with non-local sparse attention,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 3517–3526.
  * [26] D. Khashabi, S. Nowozin, J. Jancsary, and A. W. Fitzgibbon, “Joint demosaicing and denoising via learned nonparametric random fields,” _IEEE Transactions on Image Processing_ , vol. 23, pp. 4968–4981, 2014.
  * [27] T. Besard, C. Foket, and B. De Sutter, “Effective extensible programming: Unleashing Julia on GPUs,” _IEEE Transactions on Parallel and Distributed Systems_ , 2018.
  * [28] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in _Proceedings of 8th IEEE International Conference on Computer Vision (ICCV)_ , vol. 2, 2001, pp. 416–423.
  * [29] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _International Conference on Learning Representations (ICLR)_ , 2015.
  * [30] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” _arXiv preprint arXiv:1608.03983_ , 2016.
  * [31] J.-B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015.
  * [32] D. Valsesia, G. Fracastoro, and E. Magli, “Deep graph-convolutional image denoising,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 8226–8237, 2020.
  * [33] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” _IEEE Transactions on Image Processing_ , vol. 16, no. 8, pp. 2080–2095, 2007.
  * [34] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising,” _IEEE Transactions on Image Processing_ , vol. 26, no. 7, p. 3142–3155, 2017.
  * [35] P. Liu, H. Zhang, K. Zhang, L. Lin, and W. Zuo, “Multi-level wavelet-cnn for image restoration,” _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , Jun 2018. [Online]. Available: http://dx.doi.org/10.1109/CVPRW.2018.00121
  * [36] H. K. Aggarwal, M. P. Mani, and M. Jacob, “Modl: Model-based deep learning architecture for inverse problems,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 2, pp. 394–405, 2019.
  * [37] J. Liu, C. Qin, and M. Yaghoobi, “High-fidelity mri reconstruction using adaptive spatial attention selection and deep data consistency prior,” _IEEE Transactions on Computational Imaging_ , vol. 9, pp. 298–313, 2023\.
  * [38] J. Zhang and B. Ghanem, “Ista-net: Interpretable optimization-inspired deep network for image compressive sensing,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1828–1837.
  * [39] J. Duan, J. Schlemper, C. Qin, C. Ouyang, W. Bai, C. Biffi, G. Bello, B. Statton, D. P. O’Regan, and D. Rueckert, “Vs-net: Variable splitting network for accelerated parallel mri reconstruction,” in _Medical Image Computing and Computer Assisted Intervention – MICCAI 2019_ , D. Shen, T. Liu, T. M. Peters, L. H. Staib, C. Essert, S. Zhou, P.-T. Yap, and A. Khan, Eds. Cham: Springer International Publishing, 2019, pp. 713–722.