# Diving Deep into Regions: Exploiting Regional Information Transformer for
Single Image Deraining

Baiang Li1, Zhao Zhang1 , Huan Zheng1, Xiaogang Xu2,3, Yanyan Wei1, Jingyi
Zhang1,
Jicong Fan4, Meng Wang1,
1Hefei University of Technology, Hefei, China,
2Zhejiang Lab, Hangzhou, China,
3Zhejiang University, Hangzhou, China
4The Chinese University of Hong Kong, Shenzhen, Shen Zhen, China
Corresponding author

###### Abstract

Transformer-based Single Image Deraining (SID) methods have achieved
remarkable success, primarily attributed to their robust capability in
capturing long-range interactions. However, we’ve noticed that current methods
handle rain-affected and unaffected regions concurrently, overlooking the
disparities between these areas, resulting in confusion between rain streaks
and background parts, and inabilities to obtain effective interactions,
ultimately resulting in suboptimal deraining outcomes. To address the above
issue, we introduce the Region Transformer (Regformer), a novel SID method
that underlines the importance of independently processing rain-affected and
unaffected regions while considering their combined impact for high-quality
image reconstruction. The crux of our method is the innovative Region
Transformer Block (RTB), which integrates a Region Masked Attention (RMA)
mechanism and a Mixed Gate Forward Block (MGFB). Our RTB is used for attention
selection of rain-affected and unaffected regions and local modeling of mixed
scales. The RMA generates attention maps tailored to these two regions and
their interactions, enabling our model to capture comprehensive features
essential for rain removal. To better recover high-frequency textures and
capture more local details, we develop the MGFB as a compensation module to
complete local mixed scale modeling. Extensive experiments demonstrate that
our model reaches state-of-the-art performance, significantly improving the
image deraining quality. Our code and trained models will be publicly
available at https://github.com/ztMotaLee/Regformer.

##  1 Introduction

Single Image Deraining (SID) remains a critical and persistently challenging
task in the field of computer vision, primarily due to its extensive
applications in fields such as surveillance Dey & Bhattacharjee (2020),
autonomous driving Sun et al. (2019), and remote sensing Li et al. (2020). The
inclusion of rain in images not only diminishes visual quality but also exerts
a substantial influence on the effectiveness of numerous high-level vision
tasks Chen et al. (2023a); Zhou et al. (2023). Eliminating rain streaks from
rainy images poses an inherent challenge owing to the ill-posed nature of this
problem, intricate patterns, occlusions, and diverse densities of raindrops,
all of which collectively contribute to the complexity of the deraining
process.

The emergence of deep learning has spearheaded significant progress in SID,
with a multitude of Convolutional Neural Networks (CNN)-based methods being
introduced Hu et al. (2019); Li et al. (2018); Yang et al. (2020). These
methods have delivered promising outcomes across different scenarios. Beyond
CNN-based approaches, Generative Adversarial Network (GAN)-based methods Qian
et al. (2018); Zhang et al. (2019) have demonstrated their effectiveness in
addressing the SID task. Through the use of adversarial loss, these models can
produce visually appealing results, frequently surpassing traditional CNN-
based techniques in preserving image details and structure. More recently, the
remarkable success of transformer architecture in a broad range of computer
vision tasks has prompted researchers to investigate its potential for SID
Chen et al. (2023b); Jiang et al. (2022); Xiao et al. (2022).

| (CVPR 22)(CVPR 23)(TPAMI 22)(Ours)(CVPR 22)(CVPR 23)(Ours)(TPAMI 22)
---|---
(a) (b) |  (c)
Figure 1: Sub-figures (a) and (b): Schematic diagrams illustrating the process
of handling images in previous methods (left) and our Regformer (right).
Different from previous works, our method applies three distinct masks to the
attention phase, resulting in three different attentional results for SID. Our
approach addresses the significant disparities between the rain-affected and
unaffected regions of an image, as demonstrated in the output results of our
approach. Sub-figure (c): Performance comparison between our method and other
Transformer-based methods in terms of PSNR, computational complexity (GFLOPs),
and model parameters. In both graphs, our method outperforms the others by
achieving the highest PSNR while maintaining a reasonable balance of model
complexity and computational cost. This showcases the efficiency and
effectiveness of our Regformer for SID.

Despite the substantial strides made with both transformer-based and CNN-based
models in SID, when dealing with the complex and changeable real-world SID
problem, especially when the rain streaks are very similar to the light spots
in the image content, these methods often yield unsatisfactory results, as can
be seen from Fig 2. Upon a closer examination, it can be observed that current
models do not differentiate between the rain-affected and unaffected during
processing. However, there are significant disparities between the unaffected
regions of an image and the rain-affected areas, so it’s crucial to exploit
the unique feature information inherent in these separate regions. Therefore,
we believe that the distinct processing requirements of rain-affected and
unaffected regions in images necessitate a more nuanced treatment than
provided by current methods, as shown in Figure 1(a) and (b).

Drawing inspiration from these insights, we introduce a novel SID approach
called Region Transformer (Regformer). Regformer is meticulously designed to
process rain-affected and unaffected regions separately while simultaneously
considering their interaction to enhance image reconstruction. The cornerstone
of our Regformer model lies in the innovative Region Transformer Block (RTB),
which incorporates a Region Masked Attention (RMA) mechanism. RMA applies
selective masking to the query (Q) and key (K) values during the attention
process, thus generating attention feature maps exclusive to both the rain-
affected and unaffected image areas. This design empowers our Regformer model
to efficiently capture a more comprehensive set of features essential for
efficient deraining while minimizing interference from extraneous or
irrelevant features.

rainy img_140 Wang et al. (2019) GT (PSNR/SSIM) RESCAN (25.17/0.9084) Li et
al. (2018) Restormer (21.56/0.9041) Zamir et al. (2022) DualGCN (23.82/0.9291)
Fu et al. (2021) SPDNet (23.51/0.9109) Yi et al. (2021) IDT (20.90/0.9125)
Xiao et al. (2022) DRSformer (19.38/0.8894) Chen et al. (2023b) Ours
(33.96/0.9682)
---
Figure 2: Visual comparison of different methods on SPA-Data Wang et al.
(2019) dataset. Clearly, our method can better preserve the original
unaffected details of the input image compared to other methods.

In addition, we introduce a novel module named Mixed Gate Forward Block (MGFB)
to refine and enhance our model’s feature extraction capabilities. MGFB is
expertly designed to process features across varying receptive fields. This
technique models local interactions more effectively, thereby enabling the
model to gain a comprehensive understanding of the image context by
considering multiple-scale local features. With the embedded Mixed-Scale
Modeling process, MGFB can better recover textures and preserve local details.
The outcome is a highly refined feature set, resulting in more robust and
detailed representations. In conjunction, the RTB with its RMA mechanism and
the MGFB module empowers our Regformer model to overcome the constraints faced
by existing methods, further leading to state-of-the-art performance, as shown
in Figure 1(c).

In summary, the contributions of our work are three-fold.

  * •

We present the innovative Regformer model, which introduces a novel region
attention mechanism within the transformer to acquire the essential features
for SOTA SID.

  * •

We propose the innovative RTB with the RMA mechanism, enabling the model to
efficiently capture comprehensive features from rain-affected, unaffected
image areas, and their interactions, which are essential for effective
deraining. Besides, the newly designed MGFB can perceive the local mixed-scale
features between adjacent pixels in the image to complete local mixed-scale
modeling.

  * •

Our model achieves state-of-the-art performance on six public datasets,
including four synthetic datasets and two real-world datasets. Our pretrained
models and code will be publicly available.

##  2 Related Work

In this section, we provide a comprehensive review of the literature on image
deraining, specifically focusing on rain streak removal and raindrop removal
techniques. Additionally, we discuss the development of Vision Transformers
(ViT) and their applications in various vision tasks.

###  2.1 Rain Streak and Raindrop Removal

In recent years, deep learning-based approaches have outperformed their
conventional counterparts in rain streak removal. These methods employ semi-
supervised methods Wei et al. (2019), self-supervised module Huang et al.
(2021), patch-based priors for both the unaffected and rain layers Huang et
al. (2022), recurrent squeeze-and-excitation contextual dilated networks Li et
al. (2018), conditional GAN-based models Zhang et al. (2019), and two-step
rain removal techniques Ahn et al. (2022); Lin et al. (2020) to achieve
promising results. As a unique rain degradation phenomenon, raindrop removal
presents unique challenges that most rain streak removal algorithms cannot
address directly. Consequently, researchers have proposed dedicated techniques
for raindrop detection and removal, e.g., attentive GAN-based networks Qian et
al. (2018); Zhang et al. (2019); Wei et al. (2021), CNN-based networks with
shape-driven attention Quan et al. (2019), and two-stage video-based raindrop
removal approaches Chen et al. (2022); Yan et al. (2022).

###  2.2 Vision Transformers in image deraining

Following the Transformer network Vaswani et al. (2017), researchers have
endeavored to replace CNNs with Transformers as the backbone structure for
vision tasks. Dosovitskiy et al. Dosovitskiy et al. (2021) introduced the ViT
by dividing an image into non-overlapping small patches and treating each
patch as a token in the original transformer. Liu et al. Liu et al. (2021)
proposed the Swin Transformer to further reduce the computational complexity
of ViT. The Swin Transformer employs multi-headed self-attention in local
windows rather than the global approach used in ViT. This modification reduces
computation cost, while the shifted window approach compensates for any loss
of global information.

In the domain of image rain removal, several notable contributions have been
made. For instance, Jiang et al. Jiang et al. (2022) proposed a dynamic-
associated deraining network that integrates self-attention into a Transformer
alongside an unaffected recovery network. Following this, Xiao et al. Xiao et
al. (2022) introduced the Image Deraining Transformer (IDT), which employs
both window-based and spatial-based dual Transformers, achieving remarkable
results. Further advancements were made by Zamir et al. Zamir et al. (2022),
who presented an efficient transformer model named Restormer. By incorporating
innovative designs in the building blocks, Restormer can capture long-range
pixel interactions while maintaining applicability to large images, which
significantly contributes to subsequent image restoration. Building upon the
foundation laid by Restormer, Chen et al. Chen et al. (2023b) put forward a
sparse transformer. This model further explores useful self-attention values
for feature aggregation and introduces a learnable top-k selection operator to
procure more detailed features. Despite these efforts, current methodologies
often stumble in comprehensively addressing the distinct regions within an
image and their interactions, curtailing image deraining quality, as depicted
in Figure 2. These limitations serve as the impetus for our Regformer model.

Figure 3: This is an overview of our proposed Regformer, which is comprised of
Region Transformer Block (RTB) and Region Transformer Cascade (RTC) consisting
of three RTBs (each implementing a different mask mechanism). In sub-figure
(a), RTBs with different colors represent the use of varying mask strategies,
and we use a full mask strategy in the encoder part. We calculate the mask by
utilizing the shallow features derived from the input image post a $3\times 3$
convolution and various downsampling stages, along with the features obtained
from the RTC (more details can be seen in Eq. 2 and Fig. 4).  Sub-figure (a)
depicts the overall framework of our model, sub-figure (b) demonstrates our
RMA mechanism, and sub-figure (c) illustrates our MGFB mechanism.

##  3 Method

In this section, we present Region Transformer (Regformer) model for SID.
First, we briefly introduce the overall architecture of the Regformer,
followed by a detailed description of the critical components, i.e., the
Region Transformer Block (RTB) with the Region Masked Attention (RMA)
mechanism, and the Mixed Gate Forward Block (MGFB).

###  3.1 Overall Architecture

The overall pipeline of our Regformer model, as shown in Figure 3(a), employs
an encoder-decoder architecture. Given an input image of size
$H$$\times$$W$$\times$$C$, where $H$, $W$, and $C$ represent the height,
width, and channel number of the image, respectively, we first preprocess the
image using a 3$\times$3 convolution to extract the shallow feature. Note that
we save a feature map during this stage, which is later utilized in the
decoder part to obtain the mask. Subsequently, the feature map is passed
through the RTB to extract global image features. Following each RTB in the
encoder, we perform downsampling using 1$\times$1 convolutions and pixel-
unshuffle operations, similar to methods Chen et al. (2023b); Wang et al.
(2022); Xiao et al. (2022); Zamir et al. (2022). Concurrently, we introduce
skip connections to bridge intermediate features and maintain network
stability. Each RTB comprises one RMA, one MGFB, and two normalization layers.
_Note that we directly use a tensor with all 1 values as a mask at the encoder
stage, since we have not yet accessed the restored feature maps._

In the decoder, we use $1\times 1$ convolution to reduce channels after
processing the skip connections. Our Region Transformer Cascade (RTC) in the
decoder consists of three RTBs and a $1\times 1$ convolution. Specifically, we
duplicate the input feature maps and feed them into two RTBs separately.
Within these two RTBs, we perform rain-affected and unaffected attention
masking, resulting in two feature maps corresponding to the rain-affected and
unaffected regions. A residual connection is then applied to avoid feature
loss. We concatenate this resulting matrix and pass it through a $1\times 1$
convolution to reduce the channels. Finally, an RTB without a masking
mechanism is used to integrate the three regional feature maps, yielding a
comprehensive map containing rich feature information from the three distinct
regions. The above process can be expressed as:

$F_{final}={RTB}({Conv}_{1\times 1}(Concat({RTB}_{U}(I),{RTB}_{R}(I),I))),$
(1)

where $I$ denotes the input feature map, $RTB_{U}(\cdot)$ and $RTB_{R}(\cdot)$
represent the RTB process with unaffected region mask and rain region mask,
respectively. The $Concat(\cdot)$ means using concatenation, and
$Conv_{1\times 1}(\cdot)$ represents using 1$\times$1 convolution. In decoders
with different depths, we process masks to varying extents. For the third
layer decoder, we directly send the feature map extracted and saved from the
encoder. For the first and second layer decoders, we need to downsample the
feature map. Notably, the downsampling process shares parameters with the
first two downsampling operations in the encoder part, as shown in Figure 3
with shared parameters indicated by the same number.

In the final stage, we use an RTB without masked regions to synthesize
features and reconstruct the image, followed by a 3$\times$3 convolution to
adjust the channels. We then add a residual connection between the original
input image and the output to preserve features during network processing.

###  3.2 Region Transformer Block

The Region Transformer Block (RTB) is composed of one RMA, one MGFB and two LN
layers, as depicted in Figure 3(a). Specifically, we first pass the input
feature map through the LayerNorm layer. Subsequently, we obtain the attention
weight feature map using our RMA mechanism and apply a residual connection to
preserve features. Following this, we perform normalization processing and
comprehensive extraction of regional features through our MGFB module. The
details of RMA and MGFB will be elaborated in the following subsections.

####  3.2.1 Region Masked Attention

Figure 4: Illustration of the Region Mask generation. Here, $I$ and
$I^{\prime}$ represent the shallow features and the restored features in RTC,
respectively. The ForeGround Mask focuses on the rain-affected region, and the
BackGround Mask highlights the unaffected area.  For ease of understanding ,
we show images mapped to RGB space rather than feature maps.

Previous works did not fully consider the unique characteristics of different
regions within an image. This oversight often resulted in a lack of
differentiation between the rain-affected and unaffected areas, which in turn
limited the effectiveness of their deraining outcomes. To address the
limitation of previous works, we propose region masked attention, as
illustrated in Figure 3(b). We first pass the input feature map through a 1x1
convolution to obtain the query (Q), key (K), and value (V) matrices. We then
apply a 3 $\times$3 depth-wise separable convolution as a preprocessing step.
After getting the processed result, we divide Q, K, and V to generate three
independent projection matrices of key-value pairs. It is essential to use
masking to Q and K after this step.
Embedded Region Mask. At different stages of our Regformer model, we employ
various masking strategies to effectively differentiate between the rain-
affected and unaffected regions of the input feature maps. As mentioned
earlier, these masking approaches are tailored to suit the specific needs of
different positions within the network architecture. The mask acquisition
method is illustrated in Figure 4. We denote the feature map after the
3$\times$3 convolution as $I$ and downsample the feature map output, which is
saved in the encoder part, to different degrees. In the decoder part, we
denote the feature map before entering the next RTC as $I^{\prime}$. Our rain
mask (ForeGround Mask) $R$ and unaffected region mask (BackGround Mask) $U$
can be obtained using the following formula:

$\displaystyle\left\\{\begin{array}[]{ll}R=Binarize(T(I-I^{\prime}))*Conv_{1\times
1}(Rep_{C}(1)),\\\ U=1-R,\end{array}\right.$ (2)

where $I$ and $I^{\prime}$ denote the shallow features and the restored
features in RTC, respectively; $R$ and $U$ denote the rain masks and
unaffected region masks, respectively; $T$ signifies the application of
dynamic thresholds; the $Binarize(\cdot)$ performs binarization, and
${Rep}_{C}(1)$ indicates the process of copying a single channel into C
channels to align them. We use a dynamic threshold, to allow the network to
adaptively distinguish the rain region and the unaffected region, and we
binarize the segmentation result into a matrix of 0 or 1. In this case, the
area where the value is set to 0 is not easily selected when processing the
attention mechanism. However, this threshold cannot completely distinguish the
rain region and unaffected region, which is shown in figure 6 and 7, so the
RTB with the background mask is also needed. As a result, we derive the
desired masks for rain-affected and unaffected regions.
Region Masked Attention. We then perform element-wise multiplication of the
$Q$ and $K$ key-value pairs with the corresponding region mask to obtain the
masked $Q$ and $K$ key-value pairs. Subsequently, we reshape $K$ to enable
matrix multiplication, resulting in a region mask matrix with dimensions equal
to the number of channels ($\hat{C}$). By performing matrix multiplication
with this matrix and the $V$ key-value pair, we obtain the reconstructed
attention map. The above process can be expressed as:

$\displaystyle\left\\{\begin{array}[]{lll}&Q^{\prime}=Q\otimes M,\\\
&K^{\prime}=Reshape(K\otimes M),\\\ &Attention(Q,K,V,M)=Conv_{1\times
1}(Q^{\prime}K^{\prime}\otimes V),\end{array}\right.$ (3)

where $Q$, $K$, and $V$ represent the query, key, and value, respectively. The
$M$ represents the Region Mask mechanism, and the $\otimes$ here represents
the element-wise multiplication. Hence, RMA effectively utilizes the rain mask
and unaffected region mask, enabling our model to selectively focus on crucial
information from both regions. Consequently, it ensures the preservation of
critical information from both the rain-affected and unaffected regions while
considering their combined perspective for high-quality image reconstruction,
ultimately yielding a feature map with region-specific features. Finally, we
use one 1$\times$1 convolution to adjust the feature map’s channels.

####  3.2.2 Mixed Gate Forward Block

In the field of SID, one key point is how to conduct broader modeling for rain
streaks of different scales. However, existing methods have not paid much
attention to this Zamir et al. (2022), or are only limited to a specific scale
modeling Chen et al. (2023b); Xiao et al. (2022), which will make the model
perform poorly when retaining rain-like features, as can be seen from figure
2. In order to solve this problem, we design the Mixed Gate Forward Block
(MGFB) with a Mixed Scale Modeling process to effectively perceive the local
mixed-scale features in adjacent pixel areas, as depicted in Figure 3(c).
Specifically, the input fused feature map is first passed through a 1$\times$1
convolution to expand the channels, and we use a 3$\times$3 depth-wise
convolution to complete the preliminary feature extraction. This can be
described as:

$M=DWConv_{3\times 3}(Conv_{1\times 1}(I)),$ (4)

where $I$ and $M$ represent the input feature map and the middle feature map,
respectively. To complete Mixed Scale Modeling, we divide this feature map
into multiple parts according to the channels and use depth-wise convolutions
with different kernel sizes to fully extract features on different receptive
fields. We added residual connections to avoid the loss of original features.
The depth-wise convolutions with different kernels are used to capture the
local features of rain streaks at varying scales without incurring a high
computational cost, allowing the image deraining process to be guided
simultaneously in different scales. Subsequently, the resulting feature maps
are combined through element-wise matrix addition. This approach enables each
level to focus on unique, detailed aspects complementary to other levels. This
Mixed Scale Modeling process can be expressed as:

$F=Activation(DWConv_{k_{1}\times
k_{1}}(M)+M)\odot\prod_{i=2}^{n}(DWConv_{k_{i}\times k_{i}}(M)+M),$ (5)

where $F$ denotes the final feature map, $DWConv(\cdot)$ represents using
depth-wise convolution, the $Activation$ refers to the activation function,
the product symbol and $\odot$ here represents the element-wise
multiplication. The $n$ and $k_{i}$ here are all hyperparameters. Lastly, we
use a $1\times 1$ convolution to adjust the number of output channels, which
can be expressed as follows:

$O=Conv_{1\times 1}(F),$ (6)

where $O$ represents the output feature map, and $Conv_{1\times 1}$ represents
using 1$\times$1 convolution. By incorporating the MGFB, our model effectively
processes the three-region features extracted by the RMA, while completing
local mixed-scale modeling. This results in a more comprehensive understanding
of the image context and improved image deraining performance.

rainy img_465 Wang et al. (2019) GT (PSNR/SSIM) RESCAN (24.04/0.9024) Li et
al. (2018) Restormer (23.48/0.9102) Zamir et al. (2022) DualGCN (22.09/0.8831)
Fu et al. (2021) SPDNet (20.46/0.0.8593) Yi et al. (2021) IDT (27.60/0.9273)
Xiao et al. (2022) DRSformer (23.79/0.9140) Chen et al. (2023b) Ours
(38.26/0.9775)
---

Figure 5: Visual comparison on SPA-Data Wang et al. (2019) dataset. To
facilitate the display of the results, we rotate the local graph 90 degrees
clockwise. Unlike previous methods, our method successfully preserves the
original details of the input image, compared to the GT.

##  4 Experiments

###  4.1 Experimental Settings

Evaluated Datasets. We evaluate the performance of our model on several public
benchmarks, including Rain200L Yang et al. (2017), Rain200H Yang et al.
(2017), DID-Data Zhang & Patel (2018), DDN-Data Fu et al. (2017) and SPA-Data
Wang et al. (2019). Specifically, Rain200L and Rain200H consist of 1,800
synthetic rainy image pairs for training and 200 synthetic test image pairs.
DID-Data and DDN-Data contain 12,000 and 12,600 synthetic rainy image pairs
for training, respectively, along with 1,200 and 1,400 rainy image pairs for
testing. Meanwhile, SPA-Data is a real-world dataset that includes 638,492
training image pairs and 1,000 testing image pairs. We further evaluate
Regformer on a real-world raindrop dataset AGAN-Data Qian et al. (2018), which
includes 1,110 image pairs.

Compared Methods. For rain streak removal, we compare our Regformer model with
a variety of state-of-the-art methods, including prior knowledge-based
approaches like DSC Luo et al. (2015) and GMM Li et al. (2016), CNN-based
methods such as DDN Fu et al. (2017), RESCAN Li et al. (2018), PReNet Ren et
al. (2019), MSPFN Jiang et al. (2020), RCDNet Wang et al. (2020), MPRNet Zamir
et al. (2021), DualGCN Fu et al. (2021) and SPDNet Yi et al. (2021). In
addition, we consider Transformer-based methods, including Uformer Wang et al.
(2022), Restormer Zamir et al. (2022), IDT Xiao et al. (2022) and DRSformer
Chen et al. (2023b). We refer to the experimental data provided by DRSformer
and adhere to their original experimental settings for further analysis. Our
experimental results are presented in Table 1. In addition, we also test our
model on one real-world raindrop dataset AGAN-Data Qian et al. (2018) to
measure our model performance more broadly. We compare our model with five
methods, including Eigen et al. Eigen et al. (2013), Pix2Pix Isola et al.
(2017), AttentGAN Qian et al. (2018), Quan et al. Quan et al. (2019) and IDT
Xiao et al. (2022). Our experimental results are presented in Table 2. We
employ PSNR (Peak Signal-to-Noise Ratio) Q. & M. (2008) and SSIM (Structural
Similarity Index Measure) Wang et al. (2004) as the primary evaluation
metrics. Note that we calculate the metrics on the Y channel of YCbCr like the
previous works Chen et al. (2023b); Wang et al. (2020); Yang et al. (2017).

Implementation and Training Details. In our Regformer model, we configure the
layers L0, L1, L2 and L3 to 4, 4, 4, 4, and the corresponding attention heads
to 6, 6, 6, 6. Additionally, we set the initial channel to 48 for Regformer.
During training, we employ the AdamW Loshchilov & Hutter (2019) optimizer with
parameters $\beta 1$ = 0.9, $\beta 2$ = 0.999, and weight decay = 1e-4. In the
experiment part, we set $n$ = 2, $k_{1}$ = 3 and $k_{2}$ = 5 in the MGFB.
Please note that in different tasks or different datasets, we may modify these
hyperparameters to achieve better results. Concurrently, we follow DRSformer’s
settings Chen et al. (2023b), which utilize the L1 loss as the loss function.
We set the initial learning rate to 3e-4 and apply a cosine annealing
algorithm to gradually decay it to 1e-6. The patch size is set to $128\times
128$, and we perform a total of 300K iterations during training using two
NVIDIA GeForce RTX 3090 GPUs with 24GB memory each. Any settings that are not
explicitly mentioned in this paper keep consistent with those used in
DRSformer Chen et al. (2023b).

###  4.2 Deraining Performance Evaluations

Table 1: Quantitative comparison (PSNR/SSIM) with state-of-the-art methods for
single image rain streak removal. The best and second best results are colored
with red and blue.

Datasets | Rain200L | Rain200H | DID-Data | DDN-Data | SPA-Data
---|---|---|---|---|---
Metrics | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
DSC | 27.16 | 0.8663 | 14.73 | 0.3815 | 24.24 | 0.8279 | 27.31 | 0.8373 | 34.95 | 0.9416
GMM | 28.66 | 0.8652 | 14.50 | 0.4164 | 25.81 | 0.8344 | 27.55 | 0.8479 | 34.30 | 0.9428
DDN | 34.68 | 0.9671 | 26.05 | 0.8056 | 30.97 | 0.9116 | 30.00 | 0.9041 | 36.16 | 0.9457
RESCAN | 36.09 | 0.9697 | 26.75 | 0.8353 | 33.38 | 0.9417 | 31.94 | 0.9345 | 38.11 | 0.9707
PReNet | 37.80 | 0.9814 | 29.04 | 0.8991 | 33.17 | 0.9481 | 32.60 | 0.9459 | 40.16 | 0.9816
MSPFN | 38.58 | 0.9827 | 29.36 | 0.9034 | 33.72 | 0.9550 | 32.99 | 0.9333 | 43.43 | 0.9843
RCDNet | 39.17 | 0.9885 | 30.24 | 0.9048 | 34.08 | 0.9532 | 33.04 | 0.9472 | 43.36 | 0.9831
MPRNet | 39.47 | 0.9825 | 30.67 | 0.9110 | 33.99 | 0.9590 | 33.10 | 0.9347 | 43.64 | 0.9844
DualGCN | 40.73 | 0.9886 | 31.15 | 0.9125 | 34.37 | 0.9620 | 33.01 | 0.9489 | 44.18 | 0.9902
SPDNet | 40.50 | 0.9875 | 31.28 | 0.9207 | 34.57 | 0.9560 | 33.15 | 0.9457 | 43.20 | 0.9871
Uformer | 40.20 | 0.9860 | 30.80 | 0.9105 | 35.02 | 0.9621 | 33.95 | 0.9545 | 46.13 | 0.9913
Restormer | 40.99 | 0.9890 | 32.00 | 0.9329 | 35.29 | 0.9641 | 34.20 | 0.9571 | 47.98 | 0.9921
IDT | 40.74 | 0.9884 | 32.10 | 0.9344 | 34.89 | 0.9623 | 33.84 | 0.9549 | 47.35 | 0.9930
DRSformer | 41.23 | 0.9894 | 32.17 | 0.9326 | 35.35 | 0.9646 | 34.35 | 0.9588 | 48.54 | 0.9924
Regformer (ours) | 41.51 | 0.9900 | 32.46 | 0.9353 | 35.43 | 0.9651 | 34.38 | 0.9591 | 48.60 | 0.9941

(1) Quantitative Results. The comparison results of our model and other
related methods on different

Metrics | PSNR | SSIM
---|---|---
Eigen et al. Eigen et al. (2013) | 21.31 | 0.7572
Pix2Pix Isola et al. (2017) | 27.20 | 0.8359
AttentGAN Qian et al. (2018) | 31.59 | 0.9170
Quan et al. Quan et al. (2019) | 31.37 | 0.9183
IDT Xiao et al. (2022) | 31.87 | 0.9313
Regformer (ours) | 32.51 | 0.9725

Table 2: Quantitative comparison with other related methods for single image
raindrop removal on AGAN-Data Qian et al. (2018). The best and second best
results are colored with red and blue.

datasets are shown in Table 1 and Table 2. Specifically, in Table 1, we show
the results on rain streak datasets, while Table 2 shows our results on the
raindrop dataset. For rain streak removal, we can clearly see that the PSNR
and SSIM values of our method on each dataset exceed the existing solutions,
or maintain a high competitive level. Especially on the Rain200L and Rain200H
datasets, our method has greatly improved the restored results, and
outperformed current methods by up to 0.26 dB and 0.28 dB. On other datasets,
PSNR far exceeds the baseline Zamir et al. (2022), and at the same time
refreshes the current SOTA level. This clearly demonstrates the effectiveness
of our innovative approach to Regformer. In addition, it can be seen from the
chart that the CNN-based method still maintains high competitiveness compared
with the current Transformer-based method, but the performance of methods
based on prior knowledge, such as DSC Luo et al. (2015) and GMM Li et al.
(2016), is relatively backward. Although DRSformer also performs effective
attention mechanism exploration based on Restormer, they lag behind our
Regformer on both synthetic and real datasets since they do not explore the
difference between different regions. In addition, for raindrop removal, we
can clearly see that our method far exceeds current related methods by up to
0.64 dB, and the error rate decreased by nearly 60% (the ground truth’s SSIM
is 1.0000).
(2) Visualization Results. We present our rain streak removal results in
detail in Figures 2 and 5. Notably, we present a visualization of our model’s
ideas in Figure 1, 6, and 7, which focus on showing why and how our model
works. Figures 2, 5 show the comparison results of ours and other existing
representative methods on the real dataset SPA-Data, respectively. Clearly, we
can see that in some areas, other compared methods do not remove the rain
streak information well, or produce image distortion problems while removing
the rain streak information. Different from other methods, our method
preserves the original details of the image content area while removing the
rain streaks, which fully demonstrates the superiority of our method. Due to
space limitations, more visualization results will be shown in the appendix
section.

###  4.3 Ablation Studies

In the ablation study, we utilize the Rain200L dataset to train and evaluate
our network. We take Restormer Zamir et al. (2022) as the Baseline and modify
some of its parameters to enhance its performance. In Table 3, Baseline1 (v1)
denotes the initial Restormer; baseline2 (v2) denotes the Restormer with
optimized parameters; w/RTC1 (v3) refers to the addition of RTC without
employing the RegionMask mechanism (i.e., no mask operation); and w/RTC2 (v4)
signifies the incorporation of the RegionMask mechanism. The term w/MGFB (v5)
represents the integration of the MGFB. This ablation study aims to provide a
clear understanding of the individual component’s contributions and the
overall effectiveness of our proposed Regformer model.
(1) Effectiveness of Training Settings. Initially, we train our model using
the default settings of

| Variants | PSNR | SSIM
---|---|---|---
v1 | Restormer 1 | 40.99 | 0.9890
v2 | Baseline 2 | 41.24 | 0.9897
v3 | w/RTC1 | 41.25 | 0.9897
v4 | w/RTC2 | 41.42 | 0.9899
v5 | w/MGFB | 41.26 | 0.9898
v6 | v5+FGMask | 41.46 | 0.9899
v7 | v5+BGMask | 41.43 | 0.9899
v8 | Regformer | 41.51 | 0.9900
Table 3: Ablation study on Rain200L dataset. It reflects the role of each
component in our Regformer.

Restormer (v1) and subsequently retrain it by adjusting some of its parameters
(v2). The results can be seen in Table 3. For specific parameter details,
please refer to the training details section. We fine-tune the number of
layers and attention heads within each module to better adapt our model to the
target task and establish a solid foundation for further modifications. The
modifications in training settings serve to enhance our model’s performance,
making it more effective in handling SID task.
(2) Effectiveness of RTC and Region Mask Mechanism. Table 3 presents the
results of incorporating the RTC and region mask mechanisms into our model.
Clearly, adding the RTC module without the region mask mechanism does not
significantly improve the model’s performance (compared to v2, v3 exhibits
only a 0.01 dB increase). We add foreground mask (v6) and background mask
(v7), respectively, and the performance improves by 0.20dB and 0.17dB,
respectively. However, upon integrating the entire Region Mask mechanism, the
network effectively captures feature information from different regions. This
allows the model to efficiently comprehend the features of distinct regions
and their fused attributes, thereby achieving a substantial performance
enhancement (compared to v3, the performance of v4 improves by 0.17 dB; and
relative to v5, the performance of Regformer is boosted by 0.25 dB). These
results highlight the crucial role the region mask mechanism plays in
Regformer, demonstrating its effectiveness in handling the single image
deraining task.
(3) Effectiveness of MGFB. Table 3 illustrates the impact of incorporating
MGFB into our model. As evident from the table, adding MGFB without
introducing RTC (v5) does not significantly enhance the model’s restoration
performance (compared to v2, the PSNR of v5 increases by a mere 0.02 dB). This
outcome can be attributed to MGFB being primarily designed as a mixed-scale
feature extraction mechanism for different Region Mask attention mechanisms,
intended to thoroughly extract features across various scales and receptive
fields in distinct regions. Without employing the Region Mask attention
mechanism, MGFB is restricted to extracting information from a single-layer
region, which limits its performance. However, when RTC2 is integrated, we see
a notable performance improvement (Regformer’s PSNR is enhanced by 0.09 dB
compared to v4), which directly substantiates the validity and rationality of
our approach. This highlights the effectiveness of MGFB within our proposed
framework for SID.

##  5 Conclusion

In this paper, we introduced Regformer, a novel SID model that combines the
advantages of both the rain-affected and rain-unaffected regions. The
Regformer model efficiently integrates the local and non-local features to
effectively reconstruct high-quality images while preserving essential details
in both the rain-affected and unaffected regions. Our proposed embedded region
mask mechanism allows the model to selectively focus on crucial information
from both regions. By employing rain masks and unaffected region masks, the
model can better differentiate between the two regions and ultimately yield a
feature map with region-specific features. Extensive experiments on rain
streak and raindrop removal demonstrated the effectiveness of our model. The
results showed that our model outperforms current methods in terms of both
quantitative and qualitative evaluations. In conclusion, Regformer is a
promising SID model, achieving superior performance by leveraging the power of
Transformer and U-Net architectures while incorporating an effective region
mask mechanism. Future work may involve exploring additional masking
strategies for further improvement and extending the model to other image
restoration tasks, including dehazing and denoising.

#### Acknowledgments

This work is supported by the National Natural Science Foundation of China
(62072151), Anhui Provincial Natural Science Fund for the Distinguished Young
Scholars (2008085J30), Open Foundation of Yunnan Key Laboratory of Software
Engineering (2023SE103), CCF-Baidu Open Fund and CAAI-Huawei MindSpore Open
Fund.

## References

  * Ahn et al. (2022) Woo Jin Ahn, Tae Koo Kang, Hyun Duck Choi, and Myo Taeg Lim.  Remove and recover: Deep end-to-end two-stage attention network for single-shot heavy rain removal.  _Neurocomputing_ , 481:216–227, 2022.
  * Chen et al. (2023a) Keyan Chen, Xiaolong Jiang, Yao Hu, Xu Tang, Yan Gao, Jianqi Chen, and Weidi Xie.  Ovarnet: Towards open-vocabulary object attribute recognition.  In _CVPR_ , pp. 23518–23527, 2023a.
  * Chen et al. (2022) Wei-Ting Chen, Zhi-Kai Huang, Cheng-Che Tsai, Hao-Hsiang Yang, Jian-Jiun Ding, and Sy-Yen Kuo.  Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: Toward a unified model.  In _CVPR_ , pp. 17632–17641, 2022.
  * Chen et al. (2023b) Xiang Chen, , Hao Li, Mingqiang Li, and Jinshan Pan.  Learning a sparse transformer network for effective image deraining.  In _CVPR_ , pp. 1–10, 2023b.
  * Dey & Bhattacharjee (2020) Ratnadeep Dey and Debotosh Bhattacharjee.  Single image de-raining using gan for accurate video surveillance.  _Intelligence Enabled Research: DoSIER_ , pp. 7–11, 2020.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , pp. 1–21, 2021.
  * Eigen et al. (2013) David Eigen, Dilip Krishnan, and Rob Fergus.  Restoring an image taken through a window covered with dirt or rain.  In _ICCV_ , pp. 633–640, 2013.
  * Fu et al. (2017) Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley.  Removing rain from single images via a deep detail network.  In _CVPR_ , pp. 3855–3863, 2017.
  * Fu et al. (2021) Xueyang Fu, Qi Qi, Zheng-Jun Zha, Yurui Zhu, and Xinghao Ding.  Rain streak removal via dual graph convolutional network.  In _AAAI_ , pp. 1352–1360, 2021.
  * Hu et al. (2019) Xiaowei Hu, Chi-Wing Fu, Lei Zhu, and Pheng-Ann Heng.  Depth-attentional features for single-image rain removal.  In _CVPR_ , pp. 8022–8031, 2019.
  * Huang et al. (2021) Huaibo Huang, Aijing Yu, and Ran He.  Memory oriented transfer learning for semi-supervised image deraining.  In _CVPR_ , pp. 7732–7741, 2021.
  * Huang et al. (2022) Shuying Huang, Yating Xu, Mingyang Ren, Yong Yang, and Weiguo Wan.  Rain removal of single image based on directional gradient priors.  _Applied Sciences_ , 12(22):11628, 2022.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In _CVPR_ , pp. 1125–1134, 2017.
  * Jiang et al. (2020) Kui Jiang, Zhongyuan Wang, Peng Yi, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang.  Multi-scale progressive fusion network for single image deraining.  In _CVPR_ , pp. 8346–8355, 2020.
  * Jiang et al. (2022) Kui Jiang, Zhongyuan Wang, Chen Chen, Zheng Wang, Laizhong Cui, and Chia-Wen Lin.  Magic ELF: image deraining meets association learning and transformer.  In _ACM MM_ , pp. 827–836, 2022.
  * Li et al. (2020) Jun Li, Yanqiu Pei, Shaohua Zhao, Rulin Xiao, Xiao Sang, and Chengye Zhang.  A review of remote sensing for environmental monitoring in china.  _Remote Sensing_ , 12(7):1130, 2020.
  * Li et al. (2018) Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha.  Recurrent squeeze-and-excitation context aggregation net for single image deraining.  In _ECCV_ , pp. 254–269, 2018.
  * Li et al. (2016) Yu Li, Robby T Tan, Xiaojie Guo, Jiangbo Lu, and Michael S Brown.  Rain streak removal using layer priors.  In _CVPR_ , pp. 2736–2744, 2016.
  * Lin et al. (2020) Chih-Yang Lin, Zhuang Tao, Ai-Sheng Xu, Li-Wei Kang, and Fityanul Akhyar.  Sequential dual attention network for rain streak removal in a single image.  _IEEE TIP_ , 29:9250–9265, 2020.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , pp. 10012–10022, 2021.
  * Loshchilov & Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _ICLR_ , pp. 1–18, 2019.
  * Luo et al. (2015) Yu Luo, Yong Xu, and Hui Ji.  Removing rain from a single image via discriminative sparse coding.  In _ICCV_ , pp. 3397–3405, 2015.
  * Q. & M. (2008) Huynh-Thu Q. and Ghanbari M.  Scope of validity of psnr in image/video quality assessment.  _Electronics Letters_ , 44(13):800–801, 2008.
  * Qian et al. (2018) Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu.  Attentive generative adversarial network for raindrop removal from a single image.  In _CVPR_ , pp. 2482–2491, 2018.
  * Quan et al. (2019) Yuhui Quan, Shijie Deng, Yixin Chen, and Hui Ji.  Deep learning for seeing through window with raindrops.  In _ICCV_ , pp. 2463–2471, 2019.
  * Ren et al. (2019) Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng.  Progressive image deraining networks: A better and simpler baseline.  In _CVPR_ , pp. 3937–3946, 2019.
  * Sun et al. (2019) Hao Sun, Marcelo H Ang, and Daniela Rus.  A convolutional network for joint deraining and dehazing from a single image for autonomous driving in rain.  In _IROS_ , pp. 962–969, 2019.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , pp. 1–11, 2017.
  * Wang et al. (2020) Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng.  A model-driven deep neural network for single image rain removal.  In _CVPR_ , pp. 3103–3112, 2020.
  * Wang et al. (2019) Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, and Rynson WH Lau.  Spatial attentive single-image deraining with a high quality real rain dataset.  In _CVPR_ , pp. 12270–12279, 2019.
  * Wang et al. (2022) Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li.  Uformer: A general u-shaped transformer for image restoration.  In _CVPR_ , pp. 17683–17693, 2022.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _TIP_ , 13(4):600–612, 2004.
  * Wei et al. (2019) Wei Wei, Deyu Meng, Qian Zhao, Zongben Xu, and Ying Wu.  Semi-supervised transfer learning for image rain removal.  In _CVPR_ , pp. 3877–3886, 2019.
  * Wei et al. (2021) Yanyan Wei, Zhao Zhang, Yang Wang, Mingliang Xu, Yi Yang, Shuicheng Yan, and Meng Wang.  Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking.  _IEEE TIP_ , pp. 4788–4801, 2021.
  * Xiao et al. (2022) Jie Xiao, Xueyang Fu, Aiping Liu, Feng Wu, and Zheng-Jun Zha.  Image de-raining transformer.  _IEEE TPAMI_ , pp. 1–18, 2022.
  * Yan et al. (2022) Wending Yan, Lu Xu, Wenhan Yang, and Robby T Tan.  Feature-aligned video raindrop removal with temporal constraints.  _IEEE TIP_ , 31:3440–3448, 2022.
  * Yang et al. (2017) Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan.  Deep joint rain detection and removal from a single image.  In _CVPR_ , pp. 1357–1366, 2017.
  * Yang et al. (2020) Wenhan Yang, Robby T Tan, Shiqi Wang, Yuming Fang, and Jiaying Liu.  Single image deraining: From model-based to data-driven and beyond.  _IEEE TPAMI_ , 43(11):4059–4077, 2020.
  * Yi et al. (2021) Qiaosi Yi, Juncheng Li, Qinyan Dai, Faming Fang, Guixu Zhang, and Tieyong Zeng.  Structure-preserving deraining with residue channel prior guidance.  In _ICCV_ , pp. 4238–4247, 2021.
  * Zamir et al. (2021) Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Multi-stage progressive image restoration.  In _CVPR_ , pp. 14821–14831, 2021.
  * Zamir et al. (2022) Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Restormer: Efficient transformer for high-resolution image restoration.  In _CVPR_ , pp. 5728–5739, 2022.
  * Zhang & Patel (2018) He Zhang and Vishal M Patel.  Density-aware single image de-raining using a multi-stream dense network.  In _CVPR_ , pp. 695–704, 2018.
  * Zhang et al. (2019) He Zhang, Vishwanath Sindagi, and Vishal M Patel.  Image de-raining using a conditional generative adversarial network.  _IEEE TCSVT_ , 30(11):3943–3956, 2019.
  * Zhou et al. (2023) Xingyi Zhou, Anurag Arnab, Chen Sun, and Cordelia Schmid.  How can objects help action recognition?  In _CVPR_ , pp. 2353–2362, 2023.

##  6 Appendix

In this section, we mainly show the visual results of using the Mask mechanism
to process images, and more visual results of image deraining on Rain200H Yang
et al. (2017) and SPA-Data Wang et al. (2019) datasets.

Input

Rain Mask

Unaffected Mask

img_1

img_2

img_3

img_4

img_5

img_6

Figure 6: The visualization image results of the rain mask and unaffected
region mask we generated on Rain200L dataset Yang et al. (2017). As can be
seen in the figure, most of the rain regions and the unaffected regions have
been successfully distinguished.

Input

Rain Mask

Unaffected Mask

img_1

img_2

img_3

img_4

img_5

img_6

Figure 7: The visualization image results of the rain mask and unaffected
region mask we generated on DDN dataset Fu et al. (2017). They are one image
with rain content added in different directions and different levels. As can
be seen in the figure, most of the rain regions and the unaffected regions
have been successfully distinguished.

rainy img_124 GT (PSNR/SSIM) DDN (26.09/0.7232) Restormer (29.72/0.8532)
DualGCN (27.87/0.8330) SPDNet (29.72/0.8532) IDT (28.21/0.8535) DRSformer
(31.00/0.8966) Ours (31.69/0.9024)
---

Figure 8: Visual comparison of each method on Rain200H Yang et al. (2017)
dataset. Clearly, our Regformer model can perform more accurately in detail
and texture recovery than other methods.

Rainy img_137 GT (PSNR/SSIM) Restormer (20.23/0.8619) DualGCN (26.40/0.9507)
SPDNet (24.73/0.8988) IDT (23.88/0.9279) DRSformer (19.34/0.8539) Ours
(36.00/0.9762)

Figure 9: Visual comparison of each method on SPA-Data Wang et al. (2019)
dataset. As can be seen in the figure, our model preserves more precise
details while removing rain streaks.

Input

Rain Mask

Figure 10: These mask feature maps on the Rain200L dataset Yang et al. (2017)
represent the masks as they are processed within the network during inference.
To effectively illustrate our high-quality mask, we have averaged the masks
across their respective channels and transformed them into RGB images for a
more direct and visual representation.


$\displaystyle\forall g\in UC,h\in H,w\in W$ (16b) $\displaystyle
x^{commit}_{g,h,w}\geq\sum_{h\in(h-\tau^{+}_{g}:h)}x^{start}_{g,h,w}$
$\displaystyle\forall g\in UC,h\in H,w\in W$ (16c)

#### Time Wrapping and Coupling

Our modeling approach does not assumes exogenous initial conditions for any of
the time related components (e.g. unit commitment, ramping, energy storage
balance, etc). Instead, we wrap-up initial and final conditions by setting the
previous hour to the first hour of the first sub-period ($h-1,w|h=1,w=1$) to
be equal to the last hour of the last sub-period ($h,w|h=H,w=W$) in the time
horizon –e.g. the storage level at the end of the planning horizon is equal to
the initial condition for the first hour of the planning horizon.
Additionally, our methodology keeps the chronological coupling across
different sub-periods $w$ (e.g. weeks) ensuring the operation is consistent
and optimized simultaneously for the full year. This is done by setting the
previous hour to the first hour of each sub-period
($h-1,w|h=1,w\in\\{2,..,W\\}$) to be equal to the last hour of the previous
sub-period ($h,w-1|h=H,w\in\\{2,..,W\\}$) –e.g. the commitment state at the
end of a sub-period is the initial commitment state for the next sub-period.

## References

  * [1] A. D. Mills, T. Levin, R. Wiser, J. Seel, A. Botterud, Impacts of variable renewable energy on wholesale markets and generating assets in the United States: A review of expectations and evidence, Renewable and Sustainable Energy Reviews 120 (2020) 109670.  doi:10.1016/j.rser.2019.109670.
URL https://www.sciencedirect.com/science/article/pii/S1364032119308755

  * [2] D. S. Mallapragada, N. A. Sepulveda, J. D. Jenkins, Long-run system value of battery energy storage in future grids with increasing wind and solar generation, Applied Energy 275 (2020) 115390.  doi:https://doi.org/10.1016/j.apenergy.2020.115390.
URL https://www.sciencedirect.com/science/article/pii/S0306261920309028

  * [3] N. A. Sepulveda, J. D. Jenkins, A. Edington, D. S. Mallapragada, R. K. Lester, The design space for long-duration energy storage in decarbonized power systems, Nature Energy (Mar. 2021).  doi:10.1038/s41560-021-00796-8.
URL https://doi.org/10.1038/s41560-021-00796-8

  * [4] T. T. Mai, P. Jadun, J. S. Logan, C. A. McMillan, M. Muratori, D. C. Steinberg, L. J. Vimmerstedt, B. Haley, R. Jones, B. Nelson, Electrification Futures Study: Scenarios of Electric Technology Adoption and Power Consumption for the United States, NREL (2018).  doi:10.2172/1459351.
URL https://www.osti.gov/biblio/1459351

  * [5] IRENA, Hydrogen from renewable power: Technology outlook for the energy transition, Tech. rep., International Renewable Energy Agency, Abu Dhabi (2018).
URL https://www.irena.org/publications/2018/Sep/Hydrogen-from-renewable-power

  * [6] D. Wang, M. Muratori, J. Eichman, M. Wei, S. Saxena, C. Zhang, Quantifying the flexibility of hydrogen production systems to support large-scale renewable energy integration, Journal of Power Sources 399 (2018) 383–391.  doi:10.1016/j.jpowsour.2018.07.101.
URL https://www.sciencedirect.com/science/article/pii/S0378775318308267

  * [7] J. Wohland, D. Witthaut, C.-F. Schleussner, Negative Emission Potential of Direct Air Capture Powered by Renewable Excess Electricity in Europe, Earth’s Future 6 (10) (2018) 1380–1384, publisher: John Wiley & Sons, Ltd.  doi:10.1029/2018EF000954.
URL https://doi.org/10.1029/2018EF000954

  * [8] IRENA, Demand-side flexibility for power sector transformation, Tech. rep., International Renewable Energy Agency (2019).
URL
https://www.irena.org/-/media/Files/IRENA/Agency/Publication/2019/Dec/IRENA-
Demand-side-flexibility-2019.pdf

  * [9] J. H. Williams, R. A. Jones, B. Haley, G. Kwok, J. Hargreaves, J. Farbes, M. S. Torn, Carbon‐neutral pathways for the united states, AGU Advances 2 (March 2021).  doi:10.1029/2020AV000284.
URL https://doi.org/10.1029/2020AV000284

  * [10] M. Ghorbanian, S. H. Dolatabadi, P. Siano, I. Kouveliotis-Lysikatos, N. D. Hatziargyriou, Methods for Flexible Management of Blockchain-Based Cryptocurrencies in Electricity Markets and Smart Grids, IEEE Transactions on Smart Grid 11 (5) (2020) 4227–4235.  doi:10.1109/TSG.2020.2990624.
  * [11] A. A. Atia, V. Fthenakis, Active-salinity-control reverse osmosis desalination as a flexible load resource, Desalination 468 (2019) 114062.  doi:10.1016/j.desal.2019.07.002.
URL https://www.sciencedirect.com/science/article/pii/S0011916419306770

  * [12] K. Oikonomou, M. Parvania, Optimal Participation of Water Desalination Plants in Electricity Demand Response and Regulation Markets, IEEE Systems Journal 14 (3) (2020) 3729–3739.  doi:10.1109/JSYST.2019.2943451.
  * [13] G. Fridgen, R. Keller, M.-F. Körner, M. Schöpf, A holistic view on sector coupling, Energy Policy 147 (2020) 111913.
  * [14] J. Ramsebner, R. Haas, A. Ajanovic, M. Wietschel, The sector coupling concept: A critical review, Wiley Interdisciplinary Reviews: Energy and Environment 10 (4) (2021) e396.
  * [15] B. R. de Vasconcelos, J.-M. Lavoie, Recent advances in power-to-x technology for the production of fuels and chemicals, Frontiers in Chemistry 392 (2019).
URL https://doi.org/10.3389/fchem.2019.00392

  * [16] Z. Chehade, C. Mansilla, P. Lucchese, S. Hilliard, J. Proost, Review and analysis of demonstration projects on power-to-x pathways in the world, International Journal of Hydrogen Energy 44 (51) (2019) 27637–27655.  doi:https://doi.org/10.1016/j.ijhydene.2019.08.260.
URL https://www.sciencedirect.com/science/article/pii/S0360319919333142

  * [17] C. Wulf, P. Zapp, A. Schreiber, Review of power-to-x demonstration projects in europe, Frontiers in Energy Research 8 (2020).
URL https://doi.org/10.3389/fenrg.2020.00191

  * [18] R. Daiyan, I. MacGill, R. Amal, Opportunities and Challenges for Renewable Power-to-X, ACS Energy Letters 5 (12) (2020) 3843–3847, publisher: American Chemical Society.  doi:10.1021/acsenergylett.0c02249.
URL https://doi.org/10.1021/acsenergylett.0c02249

  * [19] M. Hermesmann, K. Grübel, L. Scherotzki, T. Müller, Promising pathways: The geographic and energetic potential of power-to-x technologies based on regeneratively obtained hydrogen, Renewable and Sustainable Energy Reviews 138 (2021) 110644.  doi:https://doi.org/10.1016/j.rser.2020.110644.
URL https://www.sciencedirect.com/science/article/pii/S136403212030928X

  * [20] J. C. Koj, C. Wulf, P. Zapp, Environmental impacts of power-to-x systems-a review of technological and methodological choices in life cycle assessments, Renewable and Sustainable Energy Reviews 112 (2019) 865–879.
  * [21] T. Brown, D. Schlachtberger, A. Kies, S. Schramm, M. Greiner, Synergies of sector coupling and transmission reinforcement in a cost-optimised, highly renewable European energy system, Energy 160 (2018) 720–739.  doi:10.1016/j.energy.2018.06.222.
URL http://www.sciencedirect.com/science/article/pii/S036054421831288X

  * [22] J. Gea-Bermúdez, I. G. Jensen, M. Münster, M. Koivisto, J. G. Kirkerud, Y.-k. Chen, H. Ravn, The role of sector coupling in the green transition: A least-cost energy system development in northern-central europe towards 2050, Applied Energy 289 (2021) 116685.
  * [23] J. D. Jenkins, N. A. Sepulveda, Enhanced Decision Support for a Changing Electricity Landscape : The GenX Configurable Electricity Resource Capacity Expansion Model Revision 1.0, Working Paper, MIT Energy Initiative (2017).
  * [24] H. Eshraghi, A. R. de Queiroz, J. F. DeCarolis, Us energy-related greenhouse gas emissions in the absence of federal climate policy, Environmental science & technology 52 (17) (2018) 9595–9604.
  * [25] N. A. Sepulveda, J. D. Jenkins, F. J. de Sisternes, R. K. Lester, The Role of Firm Low-Carbon Electricity Resources in Deep Decarbonization of Power Generation, Joule 2 (11) (2018) 2403–2420.  doi:10.1016/j.joule.2018.08.006.
URL http://www.sciencedirect.com/science/article/pii/S2542435118303866

  * [26] IEA, The Future of Hydrogen, Tech. rep., IEA, Paris (2019).
URL https://www.iea.org/reports/the-future-of-hydrogen

  * [27] A. Palzer, Sektorübergreifende Modellierung und Optimierung eines zukünftigen deutschen Energiesystems unter Berücksichtigung von Energieeffizienzmaßnahmen im Gebäudesektor, Ph.D. thesis, Karlsruher Institut für Technologie (KIT) (2016).
  * [28] M. Fasihi, O. Efimova, C. Breyer, Techno-economic assessment of CO2 direct air capture plants, Journal of Cleaner Production 224 (2019) 957–980.  doi:10.1016/j.jclepro.2019.03.086.
URL http://www.sciencedirect.com/science/article/pii/S0959652619307772

  * [29] Wilcox, Jen, Worcester Polytechnic Institute, Direct Air Capture (Jul. 2019).
URL
https://usea.org/sites/default/files/event-/Wilcox_RD%20Needs%20for%20Procces%20Configurations.pdf

  * [30] G. Realmonte, L. Drouet, A. Gambhir, J. Glynn, A. Hawkes, A. C. Köberle, M. Tavoni, An inter-model assessment of the role of direct air capture in deep mitigation pathways, Nature Communications 10 (1) (2019) 3277.  doi:10.1038/s41467-019-10842-5.
URL https://doi.org/10.1038/s41467-019-10842-5

  * [31] Marsidi, Marc, Electric Industrial Boiler, Factsheet, Netherlands Organisation for Applied Scientific Research (TNO) (May 2019).
URL https://energy.nl/wp-content/uploads/2018/12/Technology-Factsheet-
Electric-industrial-boiler-1.pdf

  * [32] Blockchain.com, Total Circulating Bitcoin, Online, Blockchain.com (2021).
URL https://www.blockchain.com/charts/total-bitcoins

  * [33] Cambridge Center for Alternative Finance, Cambridge Bitcoin Electricity Consumption Index, Online, Cambridge Center for Alternative Finance (2021).
URL https://cbeci.org/

  * [34] Y. Ghalavand, M. S. Hatamipour, A. Rahimi, A review on energy consumption of desalination processes, Desalination and Water Treatment 54 (6) (2015) 1526–1541, publisher: Taylor & Francis.  doi:10.1080/19443994.2014.892837.
URL https://doi.org/10.1080/19443994.2014.892837

  * [35] N. Ghaffour, T. M. Missimer, G. L. Amy, Technical review and evaluation of the economics of water desalination: Current and future challenges for better water supply sustainability, Elsevier BV, 2013, iSSN: 00119164 Publication Title: Desalination.  doi:10.1016/j.desal.2012.10.015.
URL http://hdl.handle.net/10754/562573

  * [36] Andlinger Center for Energy and the Environment (ACEE), High Meadows Environmental Institute, Larson, Eric, Jenkins, Jesse D., Greig, Chris, Pacala, Steve, Socolow, Robert H., Williams, Robert, Mayfield, Erin, Net-Zero America by 2050: Potential Pathways, Infrastructure, and Impacts, Tech. rep., Princeton University (Dec. 2020).
URL https://netzeroamerica.princeton.edu/

  * [37] BloombergNEF, Hydrogen Economy Outlook, Tech. rep., Bloomberg Finance L.P. (2020).
URL https://data.bloomberglp.com/professional/sites/24/BNEF-Hydrogen-Economy-
Outlook-Key-Messages-30-Mar-2020.pdf

  * [38] D. Sauer, P. Elsner, I. Arzberger, H. Bolt, C. Doetsch, R. Salinger, J. Tübke, E. Weidner, M. Weinhold, J. Burfeind, M. Czyperek, M. Eck, R.-A. Eichel, J. Krassowski, M. Linder, B. Lunz, A. Wörner, S. Zunft, B. Erlach, M. Merzkirch, Flexibility Concepts for the German Power Supply in 2050 - Ensuring Stability in the Age of Renewable Energies, Tech. rep., RWTH Aachen University (Nov. 2015).  doi:10.13140/RG.2.1.1478.4409.
  * [39] IRENA, Hydrogen: A renewable energy perspective, Tech. rep., International Renewable Energy Agency, Abu Dhabi (2019).
URL https://www.irena.org/publications/2019/Sep/Hydrogen-A-renewable-energy-
perspective

  * [40] K. Nagasawa, F. T. Davidson, A. C. Lloyd, M. E. Webber, Impacts of renewable hydrogen production from wind energy in electricity markets on potential hydrogen demand for light-duty vehicles, Applied Energy 235 (2019) 1001–1016.  doi:10.1016/j.apenergy.2018.10.067.
URL http://www.sciencedirect.com/science/article/pii/S0306261918316350

  * [41] N. Kaufman, A. R. Barron, W. Krawczyk, P. Marsters, H. McJeon, A near-term to net zero alternative to the social cost of carbon for setting carbon prices, Nature Climate Change 10 (11) (2020) 1010–1014.  doi:10.1038/s41558-020-0880-3.
URL https://doi.org/10.1038/s41558-020-0880-3

  * [42] K. Schaber, Integration of Variable Renewable Energies in the European power system: a model-based analysis of transmission grid extensions and energy sector coupling, Ph.D. thesis, Technischen Universität München (2014).
  * [43] Connolly, David, Mathiesen, Brian Vad, A Technical and Economic Analysis of One Potential Pathway to a 100% Renewable Energy System, International Journal of Sustainable Energy Planning and Management 1 (2014) 7–28.  doi:https://doi.org/10.5278/ijsepm.2014.1.2.
URL https://journals.aau.dk/index.php/sepm/article/view/497

  * [44] H. Lund, B. Möller, B. Mathiesen, A. Dyrelund, The role of district heating in future renewable energy systems, Energy 35 (3) (2010) 1381–1390.  doi:10.1016/j.energy.2009.11.023.
URL https://www.sciencedirect.com/science/article/pii/S036054420900512X

  * [45] EIA, Annual Energy Outlook, Tech. rep., U.S. Energy Information Administration (Feb. 2021).
URL https://www.eia.gov/outlooks/aeo/

  * [46] IEA, Global Energy & CO2 Status Report 2019, Technical, IEA, Paris (2019).
URL https://www.iea.org/reports/global-energy-co2-status-report-2019/emissions

  * [47] NREL, Annual Technology Baseline, Tech. rep., National Renewable Energy Laboratory (2020).
  * [48] J. Squalli, Renewable energy, coal as a baseload power source, and greenhouse gas emissions: Evidence from us state-level data, Energy 127 (2017) 479–488.
  * [49] M. Á. Lynch, S. Nolan, M. T. Devine, M. O’Malley, The impacts of demand response participation in capacity markets, Applied Energy 250 (2019) 444–451.
  * [50] F. J. De Sisternes, J. D. Jenkins, A. Botterud, The value of energy storage in decarbonizing the electricity sector, Applied Energy 175 (2016) 368–379.
  * [51] E. Baik, K. P. Chawla, J. D. Jenkins, C. Kolster, N. S. Patankar, A. Olson, S. M. Benson, J. C. Long, What is different about different net-zero carbon electricity systems?, Energy and Climate Change 2 (2021) 100046.
  * [52] P. Feron, A. Cousins, K. Jiang, R. Zhai, R. Thiruvenkatachari, K. Burnard, et al., Towards zero emissions from fossil fuel power stations, International Journal of Greenhouse Gas Control 87 (2019) 188–202.
  * [53] E. Leslie, A. Pascale, J. Jenkins, Wind and solar candidate project areas for princeton repeat, Zenodo (2021).  doi:10.5281/zenodo.5021146.
URL https://doi.org/10.5281/zenodo.5021146

  * [54] G. Schivley, E. Welty, N. Patankar, PowerGenome/PowerGenome: v0.4.0 (Jan. 2021).  doi:10.5281/zenodo.4426097.
URL https://doi.org/10.5281/zenodo.4426097

  * [55] Catalyst-Cooperative, Public utility data liberation (pudl), Zenodo (2020).  doi:10.5281/zenodo.3672068.
URL https://doi.org/10.5281/zenodo.3404014

  * [56] C. T. Clack, A. Alexander, A. Choukulkar, A. E. MacDonald, Demonstrating the effect of vertical and directional shear for resource mapping of wind power, Wind Energy 19 (9) (2016) 1687–1697.
  * [57] C. T. Clack, Modeling solar irradiance and solar pv power output to create a resource assessment using linear multiple multivariate regression, Journal of Applied Meteorology and Climatology 56 (1) (2017) 109–125.
  * [58] S. Pfenninger, I. Staffell, Long-term patterns of european pv output using 30 years of validated hourly reanalysis and satellite data, Energy 114 (2016) 1251–1265.
  * [59] Palmintier, Bryan S., Incorporating Operational Flexibility Into Electric Generation Planning: Impacts and Methods for System Design and Policy Analysis, PhD Dissertation, Massachusetts Institute of Technology (2013).
URL https://dspace.mit.edu/handle/1721.1/79147

  * [60] Palmintier, Bryan S., Webster, Mort D., Impact of Operational Flexibility on Electricity Generation Planning with Renewable and Carbon Targets, IEEE Transactions on Sustainable Energy 7 (2) (2016) 672–684.  doi:10.1109/TSTE.2015.2498640.
  * [61] Jenkins, Jesse D., Electricity system planning with distributed energy resources : new methods and insights for economics, regulation, and policy, Doctoral Dissertation, Massachusetts Institute of Technology (2018).
URL https://dspace.mit.edu/handle/1721.1/120611

  * [62] Jenkins, Jesse D., Luke, Max, Thernstrom, Samuel, Getting to Zero Carbon Emissions in the Electric Power Sector, Joule 2 (12) (2018) 2498–2510.  doi:10.1016/j.joule.2018.11.013.
URL https://doi.org/10.1016/j.joule.2018.11.013