# DURRNet: Deep Unfolded Single Image Reflection Removal Network

Jun-Jie Huang, Tianrui Liu, Zhixiong Yang, Shaojing Fu, Wentao Zhao, and Pier
Luigi Dragotti

###### Abstract

Single image reflection removal problem aims to divide a reflection-
contaminated image into a transmission image and a reflection image. It is a
canonical blind source separation problem and is highly ill-posed. In this
paper, we present a novel deep architecture called deep unfolded single image
reflection removal network (DURRNet) which makes an attempt to combine the
best features from model-based and learning-based paradigms and therefore
leads to a more interpretable deep architecture. Specifically, we first
propose a model-based optimization with transform-based exclusion prior and
then design an iterative algorithm with simple closed-form solutions for
solving each sub-problems. With the deep unrolling technique, we build the
DURRNet with ProxNets to model natural image priors and ProxInvNets which are
constructed with invertible networks to impose the exclusion prior.
Comprehensive experimental results on commonly used datasets demonstrate that
the proposed DURRNet achieves state-of-the-art results both visually and
quantitatively.

##  1 Introduction

Single image reflection removal (SIRR) is a typical blind image separation
problem. It aims to decompose an image, which is captured through a glass and
is associated with reflections, into a transmission image and a reflection
image. The transmission image refers to the image content of the target scene
on the other side of the glass, and the reflection image refers to the image
content from another scene reflected by the glass. This is a highly ill-posed
problem and requires high-level understanding of the scene.

A reflection-contaminated color image $\mathbf{I}\in\mathbb{R}_{+}^{W\times
H\times 3}$ is usually assumed to be a linear combination of a transmission
image $\mathbf{T}\in\mathbb{R}_{+}^{W\times H\times 3}$ and a reflection image
$\mathbf{R}\in\mathbb{R}_{+}^{W\times H\times 3}$, i.e.,
$\mathbf{I}=\mathbf{T}+\mathbf{R}$, where $W$ and $H$ are the width and height
of the image, respectively. Decomposing $\mathbf{I}$ into $\mathbf{T}$ and
$\mathbf{R}$ is a highly ill-posed problem since there are infinite number of
feasible decomposition in the form of
$\mathbf{I}=\left(\mathbf{T}+\mathbf{Q}\right)+\left(\mathbf{R}-\mathbf{Q}\right)$,
where $\mathbf{Q}$ is the shared image content between $\mathbf{T}$ and
$\mathbf{R}$. The purpose of image reflection removal is therefore to minimize
the shared image contents on the decomposed images, and at the same time
maintain the natural aspect of the estimated images.

In order to perform effective reflection removal, suitable priors should be
exploited to constrain the problem effectively. Model-based methods
levin2004separating ; levin2007user ; li2014single ; ghost_cues_2015 ;
reflect_suppression2017 ; fast_convex_2019  formulate the image reflection
removal problem as an optimization problem with explicitly defined image
priors, for example, the gradient sparsity prior. Model-based methods lead to
highly interpretable mathematical formulations and optimization algorithms
though the end result may not be satisfactory when strong and complex
reflections are present. On the other hand, methods based on deep learning
generic_smooth_2017 ; wan2017benchmarking ; perceptual_loss_2018 ;
yang2018seeing ; beyond_linear_2019 ; wei2019single ; cascaded_refine_2020
design task specific deep network structures and loss functions to exploit
data-driven priors. These priors can be learned from large-scale real training
data or from the generation of faithful synthetic training data. However, the
deep-learning based methods are difficult to interpret and a more principled
approach to design the network structures is needed.

Figure 1: The proposed Deep Unfolded Reflection Removal Layer (DURRLayer)
based on deep unfolding. It consists of a transmission estimation network and
a reflection estimation network. For each estimation network, a proxyNet
updates the features and the proxInvNet imposes exclusion condition on two
estimated images.

In this paper, we propose a model-inspired deep network architecture for the
image separation task using deep unrolling technique. We first formulate the
single image reflection removal problem as a convolutional sparse coding
problem with sparsity priors and an exclusion prior, then we propose an
iterative algorithm based on proximal gradient descent to solve the problem.
By using the unfolding technique, we unroll an iteration of the proposed
iterative algorithm into a Deep Unfolded Reflection Removal Layer (DURRLayer)
as shown in Fig. 1. A model-driven multi-scale Deep Unfolded Reflection
Removal Network (DURRNet) is then constructed with DURRLayers in a multi-
resolution fashion. Facilitated by the model-driven deep network structure,
the proposed DURRNet is not only more interpretable, but also achieves high
quality reflection removal results.

The contribution of this paper is three-fold:

  * •

We propose a single image reflection removal convolutional sparse coding model
by exploiting the formation model of reflection-contaminated image and a
transform-based exclusion loss. Based on proximal gradient descent, we propose
an iterative algorithm with simple computations.

  * •

Based on the proposed iterative algorithm, we design a new deep network
architecture for single image reflection removal by unrolling the algorithm
into a deep network with learnable parameters. The proposed DURRNet consists
of multiple scales of DURRLayers which has an exact step-by-step relationship
with the corresponding optimization algorithm, therefore, is of high
interpretability.

  * •

Through extensive experiments, we demonstrate that the proposed DURRNet is
able to achieve effective single image reflection removal and obtains highly
competitive results compared to both the model-based and deep-learning based
single image removal methods.

The rest of the paper is organized as follows: Section 2 reviews the related
single image removal methods and algorithm unfolding. Section 3 presents the
model formulation, optimization algorithm design and the deep network
architecture of the proposed DURRNet. Section 4 demonstrates the experimental
results and comparisons. Section 5 concludes the paper.

##  2 Related Works

Model-based SIRR Methods levin2004separating ; levin2007user ; li2014single ;
ghost_cues_2015 ; reflect_suppression2017 ; fast_convex_2019  formulate the
image reflection removal problem as an optimization problem and solve it with
optimization tools. The gradient sparsity prior of natural images has been
exploited in levin2004separating ; levin2007user  to obtain decomposition with
minimal edges and local features. The relative smoothness prior has been
proposed in li2014single  since the reflected image is usually more blurred.
In fast_convex_2019 , a convex model which implies a partial differential
equation with gradient thresholding is used to suppress the reflection from a
single input image. The Laplacian fidelity prior and the $l_{0}$ gradient
sparsity prior have been used in reflect_suppression2017  to formulate the
optimization problem for reflection suppression. In ghost_cues_2015 , Gaussian
Mixture Model (GMM) has been applied for modelling patch prior to exploit the
ghosting effects on reflection.

Deep-Learning-based SIRR Methods generic_smooth_2017 ; wan2017benchmarking ;
perceptual_loss_2018 ; yang2018seeing ; beyond_linear_2019 ; wei2019single ;
cascaded_refine_2020 ; hu2021trash  solve the reflection removal problem by
designing proper deep network architectures, loss functions and exploiting
external real or synthetically generated training datasets. The Cascaded Edge
and Image Learning Network (CEILNet) generic_smooth_2017  consists of two
cascaded CNN networks, i.e., E-CNN and I-CNN for edge prediction and image
reconstruction, respectively. In perceptual_loss_2018 , exclusion loss,
perceptual loss and adversarial loss are proposed to regularize the learning
of the reflection separation network. In yang2018seeing , a bidirectional
network (BDN) which consists of a cascaded deep network has been proposed to
estimate the reflection image and use it to improve the estimation of the
transmission image. ERRNet wei2019single  proposes to utilize misaligned
training data with an alignment-invariant loss. In cascaded_refine_2020 , an
Iterative Boost Convolutional LSTM Network (IBCLN) has been proposed to
progressively separate the reflection-contaminated image into two image
layers. In hu2021trash , a dual-stream decomposition network has been proposed
to enable information exchange at different branches and achieved state-of-
the-art single image reflection removal performance.

Deep Unfolding monga2021algorithm  aims to merge model-based and deep-learning
based approaches for solving inverse problems (e.g., image restoration
problems). The general idea is to design an iterative algorithm for the
problem at hand and then convert certain steps of the iterative algorithm into
learnable parameters. In the seminal work gregor2010learning , Gregor and
LeCun proposed to convert the iterative shrinkage-thresholding algorithm
(ISTA) into a deep network by setting the dictionaries in ISTA as learnable
parameters. In yang2016deep , ADMM-Net has been proposed to unfold the
Alternating Direction Method of Multipliers (ADMM) algorithm for compressive
sensing Magnetic Resonance Imaging (MRI) reconstruction. In zhang2020deep , a
deep unfolding network for single image super-resolution has been proposed by
unfolding Maximum-a-Posteriori (MAP) formulation via a half-quadratic
splitting algorithm and interpreting the prior term as a denoiser. A Deep
Unrolling for Blind Deblurring (DUBLID) network li2020efficient  unfolds a
total variation based blind deconvolution algorithm and contains a very small
number of learnable parameters. In model_driven_rain_2020 , Wang et al.
proposed a model-inspired rain removal deep unfolding network based on
proximal gradient descent to simplify computations. Recently, Pu et al.
pu2022mixed  proposed a self-supervised deep unfolding network for separating
X-Ray images of Artworks.

##  3 Proposed Method

In this section, we will first introduce the proposed model-based optimization
formulation for single image reflection removal and then we solve the
optimization using an iterative algorithm based on proximal gradient descent.
Finally we present the proposed Deep Unfolded Reflection Removal Network
(DURRNet) architecture based on the designed iterative algorithm and detail
the training strategy.

###  3.1 Model Formulation

A reflection-contaminated color image $\mathbf{I}\in\mathbb{R}_{+}^{W\times
H\times 3}$ can be expressed as a linear combination of a transmission image
and a reflection image reflect_suppression2017 . Therefore, we can represent
the observed reflection-contaminated color image as
$\mathbf{I}=\mathbf{T}+\mathbf{R}$, where
$\mathbf{T}\in\mathbb{R}_{+}^{W\times H\times 3}$ and
$\mathbf{R}\in\mathbb{R}_{+}^{W\times H\times 3}$ are the transmission image
and the reflection image, respectively. $W$ and $H$ are the width and height
of the image.

The reflection image is usually considered as a blurred version of the
reflected scene due to the effect of the glass. With different
characteristics, $\mathbf{T}$ and $\mathbf{R}$ are assumed to have two
different representations over a transmission dictionary $\mathbf{D}_{T}$ and
a reflection dictionary $\mathbf{D}_{R}$, respectively. Based on the
Convolutional Sparse Coding (CSC) model papyan2017convolutional ;
bristow2013fast , we propose to formulate the reflection removal problem as:

$\displaystyle\underset{\mathbf{z}_{T},\mathbf{z}_{R}}{\min}$
$\displaystyle\frac{1}{2}\|\mathbf{I}-\sum_{i=1}^{N}\mathbf{D}_{T}^{i}\otimes\mathbf{z}_{T}^{i}-\sum_{j=1}^{N}\mathbf{D}_{R}^{j}\otimes\mathbf{z}_{R}^{j}\|_{F}^{2}+\lambda_{T}p_{T}(\mathbf{z}_{T})+\lambda_{R}p_{R}(\mathbf{z}_{R}),$
(1)

where $\mathbf{D}_{T}=[\mathbf{D}_{T}^{1},\cdots,\mathbf{D}_{T}^{N}]$ and
$\mathbf{D}_{R}=[\mathbf{D}_{R}^{1},\cdots,\mathbf{D}_{R}^{N}]$ are the
transmission convolutional dictionary and the reflection dictionary and
$\mathbf{z}_{T}=[\mathbf{z}_{T}^{1},\cdots,\mathbf{z}_{T}^{N}]$ and
$\mathbf{z}_{R}=[\mathbf{z}_{R}^{1},\cdots,\mathbf{z}_{R}^{N}]$ are the
features corresponding to $\mathbf{T}$ and $\mathbf{R}$, respectively. Here
$\otimes$ denotes the convolution operator and $N$ is the number of filters.
Moreover, $\lambda_{T}$, $\lambda_{R}$ are regularization parameters, and
$p_{T}(\cdot)$ and $p_{R}(\cdot)$ represents the prior term for the feature of
$\mathbf{T}$ and $\mathbf{R}$, respectively.

The exclusion loss perceptual_loss_2018  is based on the idea that if two
images do not contain shared contents, then their edges and their contours
will only overlap in a small region. In perceptual_loss_2018 , the exclusion
loss is applied as a training loss function to facilitate the training of the
image reflection network. It measures the degree of edge overlapping of two
images in a multi-scale manner and can be expressed as:

$\displaystyle\mathcal{L}_{\text{e}}=\sum_{j=1}^{J}||\Psi(f^{\downarrow
j}(\mathbf{T}),f^{\downarrow j}(\mathbf{R}))||_{F},$ (2)

where
$\Psi(\mathbf{T},\mathbf{R})=\tanh(\beta_{T}|\nabla\mathbf{T}|)\odot\tanh(\beta_{R}|\nabla\mathbf{R}|)$,
$\beta_{T}$ and $\beta_{R}$ are normalization factors, moreover, $\odot$
denotes element-wise multiplication, $\nabla\mathbf{T}$ and $\nabla\mathbf{R}$
denote the gradients of $\mathbf{T}$ and $\mathbf{R}$, respectively. Finally,
$f^{\downarrow j}(\cdot)$ denotes the downsampling operation by a factor
$2^{j-1}$ with bilinear interpolation.

In our model, we aim to explicitly include the exclusion constraint into the
optimization formulation for reflection removal, however, Eq. (2) does not
lead to easy to compute solutions. Inspired by kamilov2016parallel  which
proposed a proximal-gradient algorithm for minimizing Total Variation
regularized least-squares cost functional, a transform-based exclusion loss
has been proposed in pu2022mixed :

$\mathcal{L}_{\text{te}}(\mathbf{T},\mathbf{R})=\sum_{m=1}^{M}\|\left(\mathbf{W}_{m}\otimes\mathbf{T}\right)\odot\left(\mathbf{W}_{m}\otimes\mathbf{R}\right)\|_{1},$
(3)

where $\mathbf{W}=[\mathbf{W}_{1},\cdots,\mathbf{W}_{M}]$ denotes the high-
pass filters of a transform with $\mathbf{W}_{m}$ being the $m$-th filter.
This new formulation uses high-pass filters of a transform to extract high-
frequency information from the image and measures the element-wise correlation
between each pair of “edge” images in $l_{1}$ norm. This enables simple
closed-form solution for the optimization problem.

Based on Eq. (1) and Eq. (3), we propose to formulate the reflection removal
problem as a convolutional sparse coding problem:

$\displaystyle\underset{\mathbf{z}_{T},\mathbf{z}_{R}}{\min}$
$\displaystyle\frac{1}{2}\|\mathbf{I}-\mathbf{D}_{T}\otimes\mathbf{z}_{T}-\mathbf{D}_{R}\otimes\mathbf{z}_{R}\|_{F}^{2}+\lambda_{T}p_{T}(\mathbf{z}_{T})+\lambda_{R}p_{R}(\mathbf{z}_{R})$
(4)
$\displaystyle+\kappa\mathcal{L}_{\text{te}}(\mathbf{D}_{T}\otimes\mathbf{z}_{T},\mathbf{D}_{R}\otimes\mathbf{z}_{R}),$

where with a slight abuse of notation, we denote
$\mathbf{D}_{T}\otimes\mathbf{z}_{T}=\sum_{i=1}^{N}\mathbf{D}_{T}^{i}\otimes\mathbf{z}_{T}^{i}$
and
$\mathbf{D}_{R}\otimes\mathbf{z}_{R}=\sum_{i=1}^{N}\mathbf{D}_{R}^{i}\otimes\mathbf{z}_{R}^{i}$,
and $\kappa$ is the regularization parameter for the exclusion term.

In Eq. (4), the transmission image and the reflection image are modelled as a
linear combination of atoms from the transmission dictionary and the
reflection dictionary; the data fidelity term ensures the estimated
transmission image and the reflection image contain sufficient information of
the observed image; the two prior terms, $p_{T}(\mathbf{z}_{T})$ and
$p_{R}(\mathbf{z}_{R})$ regularize the features for the transmission and the
reflection image, and the transform-based exclusion term
$\mathcal{L}_{\text{te}}$ is used to further facilitate the separation of
image contents on the two images.

###  3.2 Optimization Algorithm

Based on the model formulation defined in Eq. (4), in this section, we design
an algorithm which solves iteratively simpler sub-problems for which we can
provide close-form solutions. Since the features $\bm{z}_{T}$ and $\bm{z}_{R}$
appear in the data fidelity term, the prior terms and the exclusion terms, it
is difficult to optimize all these terms jointly. Therefore, we introduce two
auxiliary parameters $\hat{\mathbf{T}}=\mathbf{D}_{T}\otimes\mathbf{z}_{T}$
and $\hat{\mathbf{R}}=\mathbf{D}_{R}\otimes\mathbf{z}_{R}$. With Half-
Quadratic Splitting (HQS) algorithm, Eq. (4) can then be reformulated as:

$\displaystyle\underset{\mathbf{z}_{T},\mathbf{z}_{R},\hat{\mathbf{T}},\hat{\mathbf{R}}}{\min}$
$\displaystyle\frac{1}{2}\|\mathbf{I}-\hat{\mathbf{T}}-\hat{\mathbf{R}}\|_{F}^{2}+\frac{\tau}{2}\|\hat{\mathbf{T}}-\mathbf{D}_{T}\otimes\mathbf{z}_{T}\|_{F}^{2}+\frac{\tau}{2}\|\hat{\mathbf{R}}-\mathbf{D}_{R}\otimes\mathbf{z}_{R}\|_{F}^{2}$
(5)
$\displaystyle+\lambda_{T}p_{T}(\mathbf{z}_{T})+\lambda_{R}p_{R}(\mathbf{z}_{R})+\kappa\sum_{m=1}^{M}\|(\mathbf{W}_{m}\otimes\hat{\mathbf{T}})\odot(\mathbf{W}_{m}\otimes\hat{\mathbf{R}})\|_{1},$

where $\tau$ is a regularization parameter. This formulation minimizes over
features $\mathbf{z}_{T},\mathbf{z}_{R}$ and two auxiliary parameters
$\hat{\mathbf{T}},\hat{\mathbf{R}}$. Based on Proximal Gradient Descent (PGD)
beck2009fast ; model_driven_rain_2020 , we propose an iterative algorithm to
sequentially update
$\mathbf{z}_{T},\mathbf{z}_{R},\hat{\mathbf{T}},\hat{\mathbf{R}}$ with simple
computations.

Updating $\mathbf{z}_{T}$: The sub-problem corresponding to $\mathbf{z}_{T}$
can be solved using quadratic approximation:

$\underset{\mathbf{z}_{T}}{\min}\frac{1}{2}\|\mathbf{z}_{T}-\left(\mathbf{z}_{T}^{(k)}-\eta_{1}\nabla
f(\mathbf{z}_{T}^{(k)})\right)\|_{F}^{2}+\frac{\eta_{1}\lambda_{T}}{\tau}p_{T}(\mathbf{z}_{T}),$
(6)

where $\eta_{1}$ denotes the step-size for updating, the superscript $(k)$
denotes the results from the $k$-th iteration, and
$f(\mathbf{z}_{T})=\frac{1}{2}\|\hat{\mathbf{T}}-\mathbf{D}_{T}\otimes\mathbf{z}_{T}|_{F}^{2}$.
Therefore, its solution can be expressed as:

$\mathbf{z}_{T}^{(k+1)}=\text{prox}_{\eta_{1}\lambda_{T}/\tau}\left(\mathbf{z}_{T}^{(k)}-\eta_{1}\nabla
f(\mathbf{z}_{T}^{(k)})\right),$ (7)

where $\text{prox}_{\eta_{1}\lambda_{T}/\tau}(\cdot)$ is the proximal operator
corresponding to the prior term $p_{T}(\cdot)$, $\nabla
f(\mathbf{z}_{T}^{(k)})=-\mathbf{D}_{T}^{(k)}\otimes^{T}(\hat{\mathbf{T}}-\mathbf{D}_{T}^{(k)}\otimes\mathbf{z}_{T}^{(k)})$,
and $\otimes^{T}$ denotes the transposed convolution111The operation
$\otimes^{T}$ can be implemented using the function
“torch.nn.ConvTransposed2d” in PyTorch..

Updating $\mathbf{z}_{R}$: The updating rule of $\mathbf{z}_{R}$ is similar to
that of $\mathbf{z}_{T}$ and can be expressed as:

$\mathbf{z}_{R}^{(k+1)}=\text{prox}_{\eta_{2}\lambda_{R}/\tau}\left(\mathbf{z}_{R}^{(k)}-\eta_{2}\nabla
h(\mathbf{z}_{R}^{(k)})\right),$ (8)

where $\eta_{2}$ denotes the step-size for updating,
$\text{prox}_{\eta_{2}\lambda_{R}/\tau}(\cdot)$ is the proximal operator
corresponding to the prior term $p_{R}(\cdot)$, $\nabla
h(\mathbf{z}_{R}^{(k)})=-\mathbf{D}_{R}^{(k)}\otimes^{T}(\hat{\mathbf{R}}-\hat{\mathbf{T}}-\mathbf{D}_{R}^{(k)}\otimes\mathbf{z}_{R}^{(k)})$.

Updating $\hat{\mathbf{T}}$: The sub-problem with respect to
$\hat{\mathbf{T}}$ can be expressed as:

$\displaystyle\underset{\hat{\mathbf{T}}}{\min}$
$\displaystyle\frac{1}{2}\|\mathbf{I}-\hat{\mathbf{T}}-\hat{\mathbf{R}}\|_{F}^{2}+\frac{\tau}{2}\|\hat{\mathbf{T}}-\mathbf{D}_{T}\otimes\mathbf{z}_{T}\|_{F}^{2}+\kappa\sum_{m=1}^{M}\|(\mathbf{W}_{m}\otimes\hat{\mathbf{T}})\odot(\mathbf{W}_{m}\otimes\hat{\mathbf{R}})\|_{1}.$
(9)

The quadratic approximation of Eq. (9) can similarly be expressed as:

$\displaystyle\underset{\hat{\mathbf{T}}}{\min}$
$\displaystyle\frac{1}{2}\|\hat{\mathbf{T}}-(\hat{\mathbf{T}}^{(k)}-\eta_{3}\nabla
u(\hat{\mathbf{T}}^{(k)}))\|_{F}^{2}+\kappa\sum_{m=1}^{M}\|(\mathbf{W}_{m}\otimes\hat{\mathbf{R}}^{(k)})\odot(\mathbf{W}_{m}\otimes\hat{\mathbf{T}})\|_{1},$
(10)

where
$u(\hat{\mathbf{T}})=\frac{1}{2}\|\mathbf{I}-\hat{\mathbf{T}}-\hat{\mathbf{R}}^{(k)}\|_{F}^{2}+\frac{\tau}{2}\|\hat{\mathbf{T}}-\mathbf{D}_{T}\otimes\mathbf{z}_{T}^{(k+1)}\|_{F}^{2}$.
Therefore $\nabla
u(\hat{\mathbf{T}})=-(\mathbf{I}-\hat{\mathbf{R}}^{(k)}-\hat{\mathbf{T}})+\tau(\hat{\mathbf{T}}-\mathbf{D}_{T}\otimes\mathbf{z}_{T}^{(k+1)})$.

When optimizing with respect to $\hat{\mathbf{T}}$, the estimated reflection
image $\hat{\mathbf{R}}$ is assumed to be fixed. Therefore, the transform
coefficients of the reflection image $\mathbf{W}_{m}\otimes\hat{\mathbf{R}}$
in the proposed transform-based exclusion loss can be treated as an element-
wise regularization parameter for the transform coefficients
$\mathbf{W}_{m}\otimes\hat{\mathbf{T}}$ of the transmission image.
Consequently, the solution to Eq. (9) can be expressed in terms of the
proximal operator for the proposed transform-based exclusion loss:

$\hat{\mathbf{T}}^{(k+1)}=\sum_{m=1}^{M}\mathbf{W}_{m}^{\dagger}\otimes\mathcal{S}_{\kappa|\mathbf{W}_{m}\otimes\hat{\mathbf{R}}^{(k)}|}(\mathbf{W}_{m}\otimes\phi(\hat{\mathbf{T}}^{(k)})),$
(11)

where $\phi(\hat{\mathbf{T}}^{(k)})=\hat{\mathbf{T}}^{(k)}-\eta_{3}\nabla
u(\hat{\mathbf{T}}^{(k)})$ and $\mathbf{W}_{m}^{\dagger}$ denotes the inverse
filter of $\mathbf{W}_{m}$.

The proximal operator is the soft-thresholding operator performed on the
transform coefficients of $\phi(\hat{\mathbf{T}})^{(k)}$. The soft-thresholds
${\kappa|\mathbf{W}_{m}\otimes\hat{\mathbf{R}}^{(k)}|}$ is position dependent
and based on the transform coefficients of the estimated reflection image
$\hat{\mathbf{R}}^{(k)}$. After soft-thresholding, the updated transmission
image is reconstructed using inverse transform with the soft-thresholded
transform coefficients.

Updating $\hat{\mathbf{R}}$: Similar to the updating rule for
$\hat{\mathbf{T}}$, we can express the solution to the sub-problem
corresponding to $\hat{\mathbf{R}}$ as follows:

$\hat{\mathbf{R}}^{(k+1)}=\sum_{m=1}^{M}\mathbf{W}_{m}^{\dagger}\otimes\mathcal{S}_{\kappa|\mathbf{W}_{m}\otimes\hat{\mathbf{T}}^{(k+1)}|}(\mathbf{W}_{m}\otimes\psi(\hat{\mathbf{R}}^{(k)})),$
(12)

where $\psi(\hat{\mathbf{R}}^{(k)})=\hat{\mathbf{R}}^{(k)}-\eta_{4}\nabla
v(\hat{\mathbf{R}}^{(k)})$ and $\nabla
v(\hat{\mathbf{R}})=-(\mathbf{I}-\hat{\mathbf{R}}-\hat{\mathbf{T}})+\tau(\hat{\mathbf{R}}-\mathbf{D}_{R}\otimes\mathbf{z}_{R}^{(k+1)})$.

###  3.3 Deep Unfolded Reflection Removal Network (DURRNet)

Figure 2: The proposed Deep Unfolded Reflection Removal Network (DURRNet). It
consists of $S$ scales of DURRLayers to gradually estimate the transmission
and the reflection images from low-resolution scales to the resolution of the
input image. At each scale, there are $K$ stages of DURRLayers. $\downarrow 2$
and $\uparrow 2$ denotes bilinear interpolation by a factor of 0.5 and 2,
respectively.

In this section, by using the unfolding technique, we construct a model-driven
multi-scale Deep Unfolded Reflection Removal Network (DURRNet) with multiple
Deep Unfolded Reflection Removal Layers (DURRLayers). Each DURRLayer unrolls
an iteration of the proposed iterative algorithm for single image reflection
removal.

Overall Architecture: As shown in Fig. 2, the proposed DURRNet is designed in
a multi-resolution fashion. There are $S$ scales of DURRLayers to effectively
exploit information at different scales for separating the input image into a
transmission image and a reflection image. Each scale consists of $K$
DURRLayers. At the lowest scale, the initial transmission image
$\mathbf{T}_{S}$, reflection image $\mathbf{R}_{S}$ and features
$\mathbf{z}_{T,S},\mathbf{z}_{R,S}$ are initialized based on the down-sampled
input image and the the hyper-column feature perceptual_loss_2018 ;
hariharan2015hypercolumns  of the input image using bilinear interpolation by
a factor $2^{S-1}$, respectively. At an upper scale, the transmission and
reflection images are initialized based on the $2$ times up-sampled version
estimated from its lower scale, and the features are initialized based on the
down-sampled hyper-column feature of the input image and the up-sampled
feature estimated from the lower scale. The multi-scale architecture performs
image separation in a coarse-to-fine manner and can therefore effectively
combine information from different scales.

DURRLayer: Fig. 1 shows the network structure for the proposed Deep Unfolded
Reflection Removal Layer (DURRLayer) which corresponds to one iteration of the
proposed iterative algorithm. The model-inspired DURRLayer enables that the
estimated transmission and reflection image can well reconstruct the input
image and the prior information can be properly imposed. For each image layer,
a proximal network ProxNet is used to impose the prior for the feature, and a
proximal network based on invertible network ProxInvNet is proposed to impose
the exclusion prior for each estimated image.

ProxNet: Similar to model_driven_rain_2020 , the proximal operators for
$\mathbf{z}_{T}$ and $\mathbf{z}_{R}$ in Eq. (7) and (8) are represented by
two deep convolutional networks
$\text{ProxNet}_{\mathbf{\theta}_{\mathbf{z}_{T}}}(\cdot)$ and
$\text{ProxNet}_{\mathbf{\theta}_{\mathbf{z}_{R}}}(\cdot)$ whose parameters
are learned from the training dataset to well capture the prior information.
The updating rule for $\mathbf{z}_{T}$ and $\mathbf{z}_{R}$ can be therefore
expressed as:

$\begin{cases}\nabla
f(\mathbf{z}_{T}^{(k)})=-\mathbf{K}_{T}^{(k)}\otimes^{T}\left(\hat{\mathbf{T}}^{(k)}-\mathbf{D}_{T}^{(k)}\otimes\mathbf{z}_{T}^{(k)}\right),\\\
\mathbf{z}_{T}^{(k+1)}=\text{ProxNet}_{\mathbf{\theta}_{\mathbf{z}_{T}}}\left(\mathbf{z}_{T}^{(k)}-\nabla
f(\mathbf{z}_{T}^{(k)})\right),\\\ \end{cases}$ (13) $\begin{cases}\nabla
h(\mathbf{z}_{R}^{(k)})=-\mathbf{K}_{R}^{(k)}\otimes^{T}\left(\hat{\mathbf{R}}^{(k)}-\mathbf{D}_{R}^{(k)}\otimes\mathbf{z}_{R}^{(k)}\right),\\\
\mathbf{z}_{R}^{(k+1)}=\text{ProxNet}_{\mathbf{\theta}_{\mathbf{z}_{R}}}\left(\mathbf{z}_{R}^{(k)}-\nabla
f(\mathbf{z}_{R}^{(k)})\right),\end{cases}$ (14)

where the convolutional dictionaries $\mathbf{D}_{T}^{(k)}$,
$\mathbf{D}_{R}^{(k)}$, $\mathbf{K}_{T}^{(k)}$, and $\mathbf{K}_{R}^{(k)}$ and
the parameters of the proximal networks $\mathbf{\theta}_{\mathbf{z}_{T}}$ and
$\mathbf{\theta}_{\mathbf{z}_{R}}$ are learnable parameters.

Figure 3: The proposed Proximal Invertible Network
$\text{ProxInvNete}_{\mathbf{\theta}_{\mathbf{T}}}(\cdot,\cdot)$. The
invertible network (invNet) serves as an invertible transform to transform
images to coefficient domain using its forward pass then transform the
coefficients back to image domain using its backward pass.

ProxInvNet: For the proximal operators for the transform-based exclusion term,
a direct option is to apply wavelet transform to extract edge information, use
soft-thresholding operator to suppress common content and then reconstruct the
image using the inverse wavelet transform. However, the fixed transform may
not be sufficiently flexible to handle complex reflections. Inspired by the
invertible networks as a learnable invertible transform huang2021linn ;
huang2021winnet , we propose to use the invertible networks to construct a
learnable proximal operator $\text{ProxInvNete}_{\mathbf{\theta}}(\cdot)$ for
imposing the exclusion condition. The forward pass of the invertible networks
serves as the forward transform, and the backward pass of the invertible
networks then serves as the corresponding inverse transform. The updating rule
for $\hat{\mathbf{T}}$ and $\hat{\mathbf{R}}$ can be expressed as:

$\begin{cases}\mathcal{E}_{\mathbf{T}}^{(k+1)}=\hat{\mathbf{T}}^{(k)}-\mathbf{D}_{T}^{(k)}\otimes\mathbf{z}_{T}^{(k+1)},\\\
\phi(\hat{\mathbf{T}}^{(k)})=\hat{\mathbf{T}}^{(k)}+\eta_{T}\left((\mathbf{I}-\hat{\mathbf{R}}^{(k)}-\hat{\mathbf{T}}^{(k)})-\tau_{T}\mathcal{E}_{\mathbf{T}}^{(k+1)}\right),\\\
\hat{\mathbf{T}}^{(k+1)}=\text{ProxInvNet}_{\mathbf{\theta}_{\mathbf{T}}}\left(\phi(\hat{\mathbf{T}}^{(k)}),\hat{\mathbf{R}^{(k)}}\right),\\\
\end{cases}$ (15)
$\begin{cases}\mathcal{E}_{\mathbf{R}}^{(k+1)}=\hat{\mathbf{R}}-\mathbf{D}_{R}^{(k)}\otimes\mathbf{z}_{R}^{(k+1)},\\\
\psi(\hat{\mathbf{R}}^{(k)})=\hat{\mathbf{R}}^{(k)}+\eta_{R}\left((\mathbf{I}-\hat{\mathbf{R}}^{(k)}-\hat{\mathbf{T}}^{(k+1)})-\tau_{R}\mathcal{E}_{\mathbf{R}}^{(k+1)}\right),\\\
\hat{\mathbf{R}}^{(k+1)}=\text{ProxInvNet}_{\mathbf{\theta}_{\mathbf{R}}}\left(\psi(\hat{\mathbf{R}}^{(k)}),\hat{\mathbf{T}}^{(k+1)}\right),\end{cases}$
(16)

where the convolutional dictionaries $\mathbf{D}_{T}^{(k)}$ and
$\mathbf{D}_{R}^{(k)}$, and step size parameter $\eta_{T}$, $\eta_{R}$,
$\tau_{T}$, and $\tau_{R}$ and the parameters of the proximal invertible
networks $\mathbf{\theta}_{\mathbf{T}}$ and $\mathbf{\theta}_{\mathbf{R}}$ are
learnable parameters.

Fig. 3 shows the diagram for
$\text{ProxInvNete}_{\mathbf{\theta}_{\mathbf{T}}}(\cdot,\cdot)$. The forward
pass of the invertible networks is applied as the forward transform to extract
features from $\phi(\hat{\mathbf{T}}^{(k)})$ and $\hat{\mathbf{R}^{(k)}}$. In
the Threshold Network (ThreNet), the feature of $\hat{\mathbf{R}}$ will be
concatenated with that of $\phi(\hat{\mathbf{T}}^{(k)})$ and then they pass
through a convolutional network with residual blocks to generate corrections
for the feature of $\hat{\mathbf{T}}$. The updated feature of
$\hat{\mathbf{T}}$ will then be converted back to image domain using the
backward pass of the invertible networks. Similar operations can be performed
when updating $\hat{\mathbf{R}}$. The forward and backward pass of the
invertible networks are constructed by the same set of $P$ pairs of prediction
and updater networks (PUNet), for details please refer to huang2021winnet .

Figure 4: The network architectures for ProxNet, PUNet and ThreNet used to
construct the proposed DURRLayer. The blue and green blocks represent
convolutional layers and ReLU activation layers, respectively. The yellow
blocks represent a residual block.

###  3.4 Training Details

Apart from the proposed exclusion loss we introduced in Section 3.1, we adopt
the reconstruction loss and the perceptual loss perceptual_loss_2018  for
training:

$\mathcal{L}=\mathcal{L}_{{r}}+\lambda_{e}\mathcal{L}_{{e}}+\lambda_{p}\mathcal{L}_{{p}},$
(17)

where $\lambda_{e}=0.01$ and $\lambda_{p}=0.01$ are regularization parameters.
The reconstruction loss $\mathcal{L}_{\text{r}}$ is applied to the estimated
transmission image $\widehat{\mathbf{T}}$ and reflection image
$\widehat{\mathbf{R}}$ as well as the reconstructed image based on the final
features:

$\displaystyle\mathcal{L}_{{r}}=$
$\displaystyle\|\mathbf{T}-\widehat{\mathbf{T}}\|_{2}^{2}+\|\mathbf{T}-\mathbf{D}_{T}\otimes\mathbf{z}_{T}\|_{2}^{2}+\|\mathbf{R}-\hat{\mathbf{R}}\|_{2}^{2}+\|\mathbf{R}-\mathbf{D}_{R}\otimes\mathbf{z}_{R}\|_{2}^{2}.$
(18)

Perceptual loss perceptual_loss_2018  is used to regularize the estimated
images with high perceptual quality by minimizing the $l_{1}$ difference
between the VGG features of the estimated and the ground-truth images:

$\mathcal{L}_{{p}}=\|\tau(\mathbf{T})-\tau(\widehat{\mathbf{T}})\|_{1}+\|\tau(\mathbf{R})-\tau(\widehat{\mathbf{R}})\|_{1},$
(19)

where $\tau(\cdot)$ denotes the features of the VGG-19 model pretrained on
ImageNet dataset.

##  4 Experimental Results

###  4.1 Implementation Details

The proposed method is implemented with Pytorch, and the models are optimized
with Adam optimizer with initial learning rate $10^{-4}$ which are decayed at
epoch 10, 15, and 20 with learning rate decay 0.5. The total number of epochs
is 25. The early stop strategy is used. The experiments were performed on a
computer with a RTX 3090 Ti GPU.

The number of scales $S$ in DURRNet is set to 4 and the number of DURRLayer
stages in each scale is set to 2. The number of feature channels is set to 64.
The forward pass and backward pass of the invertible networks consists of
$P=2$ pairs of PUNets. The network architectures of ProxNet, PUNet, ThreNet
used to construct DURRLayer are illustrated in Fig. 4. All the networks are
constructed using convolutional layers, ReLU layers and Residual blocks.

Table 1: Quantitative comparisons on Real20 testing dataset perceptual_loss_2018  of different methods. (The best scores are in bold.) Metrics |  CEILNet |  Zhang et al. |  BDN |  IBCLN |  YTMT |  DURRNet
---|---|---|---|---|---|---
PSNR |  18.45 |  22.55 |  18.41 |  21.86 |  23.26 |  23.61
SSIM |  0.690 |  0.788 |  0.726 |  0.762 |  0.806 |  0.804
Table 2: Quantitative comparisons on Nature testing dataset cascaded_refine_2020  of different methods. (The best scores are in bold.) Metrics |  CEILNet-F |  Zhang et al. |  BDN-F |  IBCLN |  YTMT |  DURRNet
---|---|---|---|---|---|---
PSNR |  19.33 |  19.56 |  18.92 |  23.57 |  23.85 |  24.29
SSIM |  0.745 |  0.736 |  0.737 |  0.783 |  0.810 |  0.806

(a) Zhang et al.

(b) BDN

(c) IBCLN

(d) DURRNet

(e) GT

Figure 5: Visual comparisons on the estimated transmission image (row 1, 3 and
5) and the estimated reflection image (row 2, 4 and 6) of different single
image reflection methods on Real20 dataset perceptual_loss_2018 . The last
column shows the ground-truth transmission image and reflection image for the
reference.

###  4.2 Comparison with State-of-the-arts Methods

In this section, we quantitatively and visually compare our DURRNet with other
single image reflection removal methods including CEILNet method
generic_smooth_2017 , Zhang et al.’s method perceptual_loss_2018 , BDN method
yang2018seeing , IBCLN method cascaded_refine_2020  and YTMT method
hu2021trash .

Table 1 shows the quantitative evaluation results of different single image
reflection removal methods evaluated on Real20 dataset perceptual_loss_2018 .
The training datasets consist of synthetically generated reflection-
contaminated images using 7643 image pairs from PASCAL VOC dataset by
following the settings in CEILNet generic_smooth_2017  and 90 pairs of real
images from perceptual_loss_2018 . The testing datasets contain 20 images from
Real20 perceptual_loss_2018 . From Table 1, we can see that on Real20 dataset
the proposed DURRNet achieves significantly better PSNR compared to other
methods and achieves a similar SSIM results as YTMT method.

Table 2 shows the quantitative comparison results on Nature testing dataset
cascaded_refine_2020 . The comparison follows the settings in
cascaded_refine_2020 . Additional 200 training image pairs from the Nature
training dataset cascaded_refine_2020  were used for training and other models
(with a suffix “-F") were fine-tuned on the Nature training dataset for fair
comparisons. We can see that the proposed DURRNet achieves the highest PSNR
value and the second best SSIM value among all the methods.

For visual comparisons, Fig. 5 shows the estimated transmission and reflection
images by different methods on 3 exemplar images from Real20 dataset
perceptual_loss_2018 . This dataset is a challenging dataset since the input
images contain different reflection patterns and the region of overlap is
large. From Fig. 5, we can see that the proposed DURRNet is able to recover
natural looking transmission and reflection images. This could be due to the
fact that the proposed deep unfolded network architecture takes the image
formation model into consideration and prior information has been properly
imposed into the network architecture. For comparison methods, Zhang et al.’s
method is able to well separate most reflections in the input, but may
generate images with visible artifacts, BDN method did not successfully remove
strong reflections and usually generates reflection images with too much
transmission image content, and IBCLN method struggle to separate large
overlapping reflections.

Fig. 6 further shows the visual comparisons on Real45 dataset
generic_smooth_2017  which does not contain ground-truth images for reference.
We can see that the proposed DURRNet is able to properly separate the
reflection image content from the input reflection-contaminated image and the
separated reflection images contain little information from the transmission
image.

(a) Input

(b) Zhang et al.

(c) BDN

(d) IBCLN

(e) DURRNet

Figure 6: Visual comparisons of different single image reflection methods on
Real45 dataset generic_smooth_2017 . Row 1 and 3 show the estimated
transmission images, row 2 and 4 show the estimated reflection image.

###  4.3 Ablation Studies

The effectiveness of ProxNet/ProxInvNet: In the proposed DURRLayer, the main
network components are the ProxNet and ProxInvNet which is used to impose
natural image prior and exclusion prior, respectively. To understand their
functionalities, we perform ablation studies on these network components.

Table 3: Quantitative performance of the proposed DURRNet with different variations. The performance of different models are evaluated on Real20 dataset perceptual_loss_2018 . Settings |  DURRNet |  w/o ProxNet | w/o ProxInvNet | $(S,K)=(1,8)$ | $(S,K)=(2,4)$
---|---|---|---|---|---
PSNR |  23.61 |  22.63 | 22.61 | 22.47 | 22.74
SSIM |  0.803 |  0.787 | 0.788 | 0.787 | 0.794

From Table 3, we can see that when ProxNets or ProxInvNets are removed from
DURRNet there is approximately a 1 dB drop in PSNR. Therefore they are both
essential components of the proposed DURRNet. To further visualize the
functionality of ProxNet and ProxInvNet, Fig. 7 shows the single image
reflection removal results of the DURRLayer w/o ProxNet, DURRLayer w/o
ProxInvNet, and the complete model of DURRNet. We can see that when either
ProxNets or ProxInvNets are disabled, the model can still produce relatively
good results. This could be due to the network architecture design for ProxNet
which includes a global skip connection, and ProxInvNet which adopts
invertible networks as learnable transforms. In Fig. 7 (b) when ProxNets are
disabled, the model would have difficulty to localize the reflection region,
and in Fig. 7 (c) when ProxInvNets are disabled, the model would have
difficulty at dealing with the contour regions of the reflections.

(a) Input

(b) w/o ProxNet

(c) w/o ProxInvNet

(d) DURRNet

Figure 7: Visualization of the different effects of ProxNets and ProxInvNets
in the proposed DURRNet.

The effectiveness of Multi-scale Architecture: As shown in Fig. 2, the
proposed DURRNet consists of $S$ scales of DURRLayers to progressively
estimate the transmission image and the reflection image from low-resolution
scales to high-resolution scales. In Table 3 , we further analyzed the
effectiveness of the multi-scale architecture. From the table, we can see that
when the same total number of DURRLayer stages are fixed, i.e., $S\times K=8$,
the proposed DURRNet (with $(S,K)=(4,2)$) achieves the best performance
compared to other configurations, e.g., $(S,K)=(1,8)$ and $(S,K)=(2,4)$. This
indicates that the multi-scale architecture can effectively and efficiently
integrate information from different scales.

##  5 Conclusions

In this paper, we proposed a novel model-inspired single image reflection
removal network named Deep Unfolded Reflection Removal Network (DURRNet). The
proposed DURRNet is designed using deep unfolding technique and has clear
interpretation. The image formation model and priors have been explicitly
embedded into the design of the DURRNet architecture. Within each DURRLayer,
ProxNets are used to model natural image priors and ProxInvNets which are
constructed with invertible networks are used to impose the exclusion prior.
From experimental results, the proposed DURRNet is able to recover high-
quality transmission and reflection images both quantitatively and visually.

## References

  * [1] N. Arvanitopoulos, R. Achanta, and S. Susstrunk.  Single image reflection suppression.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4498–4506, 2017.
  * [2] A. Beck and M. Teboulle.  A fast iterative shrinkage-thresholding algorithm for linear inverse problems.  SIAM journal on imaging sciences, 2(1):183–202, 2009.
  * [3] H. Bristow, A. Eriksson, and S. Lucey.  Fast convolutional sparse coding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 391–398, 2013.
  * [4] Q. Fan, J. Yang, G. Hua, B. Chen, and D. Wipf.  A generic deep architecture for single image reflection removal and image smoothing.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3238–3247, 2017.
  * [5] K. Gregor and Y. LeCun.  Learning fast approximations of sparse coding.  In Proceedings of the 27th international conference on international conference on machine learning, pages 399–406, 2010.
  * [6] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik.  Hypercolumns for object segmentation and fine-grained localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 447–456, 2015.
  * [7] Q. Hu and X. Guo.  Trash or treasure? an interactive dual-stream strategy for single image reflection separation.  Advances in Neural Information Processing Systems, 34, 2021.
  * [8] J.-J. Huang and P. L. Dragotti.  LINN: Lifting inspired invertible neural network for image denoising.  In 2021 29th European Signal Processing Conference (EUSIPCO), pages 636–640, 2021.
  * [9] J.-J. Huang and P. L. Dragotti.  WINNet: Wavelet-inspired invertible network for image denoising.  arXiv preprint arXiv:2109.06381, 2021.
  * [10] U. S. Kamilov.  Parallel proximal methods for total variation minimization.  In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4697–4701. IEEE, 2016.
  * [11] A. Levin and Y. Weiss.  User assisted separation of reflections from a single image using a sparsity prior.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(9):1647–1654, 2007.
  * [12] A. Levin, A. Zomet, and Y. Weiss.  Separating reflections from a single image using local features.  In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 1, pages I–I. IEEE, 2004.
  * [13] C. Li, Y. Yang, K. He, S. Lin, and J. E. Hopcroft.  Single image reflection removal through cascaded refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3565–3574, 2020.
  * [14] Y. Li and M. S. Brown.  Single image layer separation using relative smoothness.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2752–2759, 2014.
  * [15] Y. Li, M. Tofighi, J. Geng, V. Monga, and Y. C. Eldar.  Efficient and interpretable deep blind image deblurring via algorithm unrolling.  IEEE Transactions on Computational Imaging, 6:666–681, 2020.
  * [16] V. Monga, Y. Li, and Y. C. Eldar.  Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing.  IEEE Signal Processing Magazine, 38(2):18–44, 2021.
  * [17] V. Papyan, Y. Romano, and M. Elad.  Convolutional neural networks analyzed via convolutional sparse coding.  The Journal of Machine Learning Research, 18(1):2887–2938, 2017\.
  * [18] W. Pu, J.-J. Huang, B. Sober, N. Daly, C. Higgitt, I. Daubechies, P. L. Dragotti, and M. Rodigues.  Mixed x-ray image separation for artworks with concealed designs.  arXiv preprint arXiv:2201.09167, 2022.
  * [19] Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman.  Reflection removal using ghosting cues.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3193–3201, 2015.
  * [20] R. Wan, B. Shi, L.-Y. Duan, A.-H. Tan, and A. C. Kot.  Benchmarking single-image reflection removal algorithms.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3922–3930, 2017.
  * [21] H. Wang, Q. Xie, Q. Zhao, and D. Meng.  A model-driven deep neural network for single image rain removal.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3100–3109, 2020.
  * [22] K. Wei, J. Yang, Y. Fu, D. Wipf, and H. Huang.  Single image reflection removal exploiting misaligned training data and network enhancements.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8178–8187, 2019.
  * [23] Q. Wen, Y. Tan, J. Qin, W. Liu, G. Han, and S. He.  Single image reflection removal beyond linearity.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3771–3779, 2019.
  * [24] J. Yang, D. Gong, L. Liu, and Q. Shi.  Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal.  In Proceedings of the european conference on computer vision (ECCV), pages 654–669, 2018.
  * [25] Y. Yang, W. Ma, Y. Zheng, J.-F. Cai, and W. Xu.  Fast single image reflection suppression via convex optimization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8141–8149, 2019.
  * [26] Y. Yang, J. Sun, H. Li, and Z. Xu.  Deep ADMM-net for compressive sensing mri.  In Proceedings of the 30th international conference on neural information processing systems, pages 10–18, 2016.
  * [27] K. Zhang, L. V. Gool, and R. Timofte.  Deep unfolding network for image super-resolution.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3217–3226, 2020.
  * [28] X. Zhang, R. Ng, and Q. Chen.  Single image reflection separation with perceptual losses.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4786–4794, 2018.