# FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining

Zhen Zou<EMAIL_ADDRESS>University of Science and Technology of
ChinahefeianhuiChina ,  Hu Yu<EMAIL_ADDRESS>University of Science
and Technology of ChinahefeianhuiChina and  Feng Zhao<EMAIL_ADDRESS>University of Science and Technology of ChinahefeianhuiChina

###### Abstract.

Images corrupted by rain streaks often lose vital frequency information for
perception, and image deraining aims to solve this issue which relies on
global and local degradation modeling. Recent studies have witnessed the
effectiveness and efficiency of Mamba for perceiving global and local
information based on its exploiting local correlation among patches, however,
rarely attempts have been explored to extend it with frequency analysis for
image deraining, limiting its ability to perceive global degradation that is
relevant to frequency modeling (e.g. Fourier transform). In this paper, we
propose FreqMamba, an effective and efficient paradigm that leverages the
complementary between Mamba and frequency analysis for image deraining. The
core of our method lies in extending Mamba with frequency analysis from two
perspectives: extending it with frequency-band for exploiting frequency
correlation, and connecting it with Fourier transform for global degradation
modeling. Specifically, FreqMamba introduces complementary triple interaction
structures including spatial Mamba, frequency band Mamba, and Fourier global
modeling. Frequency band Mamba decomposes the image into sub-bands of
different frequencies to allow 2D scanning from the frequency dimension.
Furthermore, leveraging Mamba’s unique data-dependent properties, we use rainy
images at different scales to provide degradation priors to the network,
thereby facilitating efficient training. Extensive experiments show that our
method outperforms state-of-the-art methods both visually and quantitatively.

Image deraining, Frequency analysis, State space model

††conference: Make sure to enter the correct conference title from your rights
confirmation emai; June 03–05, 2018; Woodstock, NY

##  1\. Introduction

Figure 1. Comparison of different modeling methods. Our FreqMamba enhances
Mamba’s 2D global perception capability from the frequency perspective.
Meanwhile, Mamba modeling in frequency dimension is introduced to realize the
seamless transition between the spatial and frequency domains.

Frequency Scan.

Images taken in rainy conditions suffer from significant quality degradation
in terms of object details and contrast caused by raindrops in the air, which
results in unpleasant visual results and loss of frequency information. Such
degradation has a serious adverse impact on the performance of advanced visual
tasks such as image classification and target detection. Therefore, image
deraining is an important task in the low-level vision field. However,
recovering clear images from those with severe raindrop degradation is pretty
a difficult job due to the complex coupling of raindrops to the background and
the loss of important perception frequency information.

Traditional methods have relied on prior knowledge to dissect the physical
properties of rain and background layers, introducing various methods to
distinguish rain streaks from clean images. However, these priors, being based
on specific observations, might not be reliable for modeling intrinsic
features of images or estimating transmission maps in physical models. The
advent of deep learning has heralded new directions in rain removal
techniques. Many methods are proposed to improve the performance of rain
removal methods from different perspectives(Ren et al., 2019; Fu et al., 2020;
Xiao et al., 2022; Guo et al., 2022). Especially, effective global degradation
modeling has proved crucial for addressing intricate challenges of image
deraining. For instance, the attention mechanism in Transformer achieves great
success with the capacity to model image-internal correlations. While,
attention mechanisms face scalability challenges due to their quadratic
complexity, posing a significant challenge when addressing large images.
Recently, an improved structured state-space sequence model (S4) with a
selective scanning mechanism, Mamba, stands out due to its ability to model
long-range sequence relationships with linear complexity. Specifically,
Mamba’s selective methodology can explicitly build the correlation among image
patches, further enabling the guidance of clean to degraded areas. However, it
is noteworthy that Mamba’s approach to processing visual tasks is
fundamentally based on pixel sequences. While this allows it to model long-
range dependencies, the selective modeling of one-dimensional sequences limits
its ability in ”global degradation perception” like the Fourier transform.
Recognizing this, we attempt to extend it from the perspective of frequency
analysis for image deraining.

We integrate two typical frequency analysis techniques into our method:
Fourier transform (FT) and wavelet packet transform (WPT). It is well known
that the Fourier transform has effective global modeling capabilities. In
addition, it has good degradation separation properties (Fig. 2). This unique
perspective is invaluable for tasks such as image deraining, where
understanding the entire image structure can significantly enhance the quality
of the restoration result. However, there are inherent gaps between different
domains, so simply combining the Fourier transform with Mamba modeling can be
quite blunt. We need an intermediate state for a seamless transition. The
wavelet packet transform decomposes the image into a series of sub-bands with
varying frequencies in the spatial domain, encapsulating both frequency
analysis and spatial information simultaneously. By utilizing the wavelet
domain as the intermediate state between the spatial and Fourier domains, we
establish a smoother transition, enhancing the overall effectiveness of the
analysis.

Recognizing the potential for perceiving global degradation via Frequency
analysis and Mamba’s ability to capture regional correlations within the
spatial domain, we introduce FreqMamba. This effective and efficient paradigm
utilizes the complementarity between Mamba and frequency analysis for image
deraining. The cornerstone of our approach lies in a three-branch structure.

Figure 2. Observation of the spectrum exchange of the Discrete Fourier
Transform(DFT). The degradation is mainly in the amplitude component, and the
Fourier transform can disentangle image content and degradation to some
extent.

fourier

  * •

Spatial Mamba: This branch operates on the original image features. It
extracts details and correlations within the image, providing crucial insights
into degradation patterns.

  * •

Frequency Band Mamba: This branch employs the WPT to dissect the input
features into a spectrum of features spread across various frequency bands.
Arranging these low-resolution features back to the original scale in
frequency order, we perform a frequency dimension scanning from low to high
frequency as well as in reverse as shown in Fig. 3. This strategy not only
enriches the model’s analytical breadth but also acts as a pivotal bridge,
melding the spatial and Fourier domains into a unified analysis framework,
offering a novel perspective on modeling.

  * •

Fourier Modeling: This branch equips the model with the ability to conduct
global analysis by leveraging the Fourier Transform to process the input. It
captures the overarching degradation patterns that affect the image, offering
a panoramic view of the frequency spectrum. This global modeling capability is
instrumental in comprehensively understanding and mitigating the effects of
degradation, ensuring an in-depth removal of rain streaks from the image for a
cleaner visual outcome.

Together, these branches form a robust architecture for tackling the challenge
of image deraining. The Spatial Scanning Mamba analyzes intricate spatial
details, while the Fourier Modeling branch offers a holistic view, enabling
the model to understand global degradation phenomena. The Frequency Band
Scanning Mamba explores the frequency dimension and provides a new perspective
for 2D modeling. Furthermore, leveraging the unique data-dependent
characteristics of Mamba, we apply this methodology to rainy images across
different scales to derive attention maps based on degradation prior. These
attention maps are then integrated into the backbone network to assist in
effective training.

##  2\. RELATED WORK

###  2.1. Image Deraining

Image deraining has witnessed substantial evolution, transitioning from early
model-based strategies to advanced data-driven techniques. Initially, model-
driven methods(Kang et al., 2011; Luo et al., 2015), separate rain streaks
from the background using hand-crafted features and physical priors. These
approaches, while insightful, often struggle with complex rain patterns and
diverse real-world scenarios, leading to limitations in their practical
applicability and performance. The advent of deep learning ushered in a new
era for image deraining, with data-driven approaches demonstrating remarkable
adeptness in extracting and learning features directly from data. The
introduction of CNNs marked a significant advancement, enabling more nuanced
and adaptive handling of rain streaks across a wide array of images (Yang et
al., 2017; Fu et al., 2017b; Zhang and Patel, 2018). Besides, the development
of architectures incorporating attention mechanisms (Wang et al., 2022;
Valanarasu et al., 2022; Xiao et al., 2022) has further refined the capacity
to discern and eliminate rain components, addressing previous shortcomings in
model generalization and detail preservation. In this work, we propose a novel
baseline with a novel triple interaction block to improve the deraining
performance jointly.

###  2.2. Frequency Analysis

The Fourier Transform is a fundamental technique in frequency domain analysis,
enabling the conversion of signals to a domain where their global statistical
properties can be more easily analyzed. This capability is extensively
leveraged across various computer vision tasks. By adept modeling of global
domain information, the FT facilitates advancements in various areas (Pratt et
al., 2017; Lee et al., 2018; Huang et al., 2022b; Zhou et al., 2022; Li et
al., 2023; Yang and Soatto, 2020; Xu et al., 2021). In terms of image
restoration, FECNet(Huang et al., 2022a) highlights the utility of Fourier
feature amplitudes in isolating global lightness components, thus improving
the aesthetic appeal and clarity of images. Similarly, FSDGN (Yu et al., 2022)
reveals how the amplitude of Fourier features serves as a key indicator of
global haze information in image dehazing tasks.

Despite these advances, the inherent constraints of FT in signal processing
hint at untapped potential, suggesting that the efficacy of these
methodologies could be augmented further. Beyond Fourier transform, the
Wavelet Transform (WT) is also a mathematical tool used in the analysis of
signals and images, offering a complementary perspective to FT. Unlike FT,
which excels in capturing frequency information, Wavelet transform provides a
multi-resolution analysis that is particularly effective in detecting and
representing localized variations in signals.

###  2.3. State Space Models

State Space Models (SSMs)(Gu et al., 2021; Smith et al., 2022) have garnered
significant attention in recent years for their ability to effectively model
long-range dependencies while exhibiting linear scalability with sequence
length. The foundational work of S4 introduced by (Gu et al., 2021) laid the
groundwork for deep state-space modeling, demonstrating promising linear
scaling properties. A recent innovation, (Mehta et al., 2022) enhances SSMs’
capabilities by integrating gating units. Furthermore, Mamba (Gu and Dao,
2023), a data-dependent SSM featuring a selective mechanism and efficient
hardware design, has emerged as a standout performer, surpassing Transformers
in natural language tasks while maintaining linear scalability with input
length. The applicability of SSMs extends beyond NLP, with pioneering works
leveraging Mamba for various vision tasks, including image classification (Liu
et al., 2024), biomedical image segmentation (Ma et al., 2024; Gong et al.,
2024), and others (Qiao et al., 2024; Wang et al., 2024; Bhirangi et al.,
2024).

##  3\. METHOD

In this section, we first introduce the motivation of our proposed method,
then introduce the preliminaries of frequency analysis and SSMs. Finally, we
outline the overall framework of FreqMamba.

###  3.1. Motivation

Degraded images, particularly those afflicted by rain, suffer from both global
degradation and local detail destruction. Frequency-based methods are utilized
for mitigating global degradation, leveraging the significant correlation
between raindrop effects and the Fourier amplitude spectrum (refer to Fig. 2).
Nonetheless, due to the inherent gap between the frequency and spatial domain,
the global nature of frequency domain operations makes it impossible to model
local dependencies of the spatial domain. Mamba is a state space model
distinguished by its selective scanning mechanism, which skillfully
facilitates the explicit modeling of interactions among sequences with linear
complexity. Applied to vision tasks, it can effectively establish correlations
between areas in 2D images.

Recognizing the complementary strengths of frequency-based methods and the
Mamba model in addressing different aspects of image degradation, we introduce
the FreqSSM Block. This novel structure features a three-branch design
engineered to transition smoothly from the correction of global degradation to
the refinement of local content. Further leveraging Mamba’s distinctive data-
dependent properties, we employ degraded images across various scales to
derive attention maps and add them in the encoder stage, thereby enhancing the
efficiency of training. Expanding upon these foundations, we propose
FreqMamba, a specialized solution designed specifically to solve the single-
image deraining task.

###  3.2. Preliminaries

####  3.2.1. Frequency Analysis in Digital Imaging

The Fourier Transform is a mathematical technique for transforming a signal
from its original domain (often time or space) into a representation in the
frequency domain and vice versa with the inverse Fourier Transform (iFT).
Specifically for a single-channel image $x$ with size of $H\times W$, the
Discrete Fourier Transform is defined as:

(1)
$\mathcal{F}(x)(u,v)=\sum_{h=0}^{H-1}\sum_{w=0}^{W-1}x(h,w)e^{-j2\pi(\frac{h}{H}u+\frac{w}{W}v)}.$

Figure 3. The comparison between (a) the vanilla scanning strategy employed by
VMamba (Liu et al., 2024) and (b) our frequency dimension strategy. Utilizing
k-level wavelet packet transform, we decompose the input into $4^{k}$ ($k$=2
in the figure) frequency bands. It is then scanned along the frequency
dimension in the spatial domain. This strategy introduces a new dimension to
2D-Mamba, allowing it to capture complex image details at different
frequencies. It is noteworthy that we use a local scanning strategy similar to
LocalMamba(Huang et al., 2024), which is more consistent with the form of WPT
and allows strictly frequency-ordered scanning.

Frequency Scan.

This process converts spatial features into a complex component, illustrating
the image’s frequency components. The frequency domain representation
$\mathcal{F}(x)$ can be decomposed into real $\mathcal{R}(x)$ and imaginary
$\mathcal{I}(x)$ parts, leading to amplitude spectrum and phase spectrum,
describing the image’s frequency content:

(2) $\displaystyle\mathcal{A}(x)(u,v)$
$\displaystyle=\sqrt{\mathcal{R}^{2}(x)(u,v)+\mathcal{I}^{2}(x)(u,v)},$
$\displaystyle\mathcal{P}(x)(u,v)$
$\displaystyle=\arctan{[\frac{\mathcal{I}(x)(u,v)}{\mathcal{R}(x)(u,v)}]}.$

Another tool, the Discrete Wavelet Transform (DWT), decomposes an image
$I\in\mathcal{R}^{H\times W\times C}$ into four sub-bands representing low-
frequency approximations and high-frequency details:

(3) $I_{LL},I_{LH},I_{HL},I_{HH}=DWT(I).$

Each sub-band, corresponding to different directional details, reduces the
original image’s dimensions by half. In contrast, the Wavelet Packet Transform
offers a more detailed frequency content analysis by further decomposing all
sub-bands at each level.

Figure 4.  The detailed architecture of our FreqMamba. The three-branch
FreqSSM forms the basic block of the u-net architecture for global and local
modeling. Multi-scale degradation priors are introduced into the training
process at the encoder stage.

####  3.2.2. State Space Models

State Space Models (SSMs) are fundamental in translating one-dimensional
inputs into outputs through latent states, utilizing a framework of linear
ordinary differential equations. For a system with input $x(t)$ and output
$y(t)$, the model dynamics are described by:

(4) $\displaystyle h^{\prime}(t)$
$\displaystyle=\textbf{A}h(t)+\textbf{B}x(t),$ $\displaystyle y(t)$
$\displaystyle=\textbf{C}h(t)+\textbf{D}x(t),$

where $\textbf{A},\textbf{B},\textbf{C}$, and D are the model parameters. The
discrete versions of these models, such as Mamba, incorporate a discretization
step through the zero-order hold (ZOH) method while allowing the models to
scan and adjust to the input data adaptively through a selective scanning
mechanism. This adaptability is particularly useful in complex applications
like image restoration, where understanding contextual relationships between
different image regions is crucial.

###  3.3. Architecture

We show our proposed method in Fig.4, which uses a multi-scale U-Net
architecture with three-branch Frequency-SSM Blocks as the core components. At
the same time, the input-dependent nature of Mamba is used to generate
attention maps using degradation images of different scales, allowing the
model to capture degradation distributions at different scales. Now we
introduce the details of the different components of the network respectively.

####  3.3.1. Frequency-SSM Block.

The frequency-SSM block is designed to address complex challenges by
leveraging the synergy of three distinct branches: the Fourier Modeling
Branch, the Spatial Branch, and the Frequency Band Branch.

Fourier Modeling Branch. For the Fourier Modeling Branch, the input feature
$F_{in}$ first undergoes a convolution layer to produce $F_{0}$. $F_{0}$ is
transformed into its Fourier spectrum via the Fast Fourier Transform (FFT),
which is then decomposed into amplitude spectrum $\mathcal{A}(F_{0})$ and
phase spectrum $\mathcal{P}(F_{0})$. The amplitude and phase spectrum are
refined through convolution blocks in the frequency domain and finally
returned to the spatial domain through iFFT:

(5) $\displaystyle F_{P}$
$\displaystyle=ConvBlock(W_{1}(\mathcal{A}(F_{0}))),$ $\displaystyle F_{A}$
$\displaystyle=ConvBlock(W_{1}(\mathcal{P}(F_{0}))),$ $\displaystyle F_{f}$
$\displaystyle=\mathcal{F}^{-1}(F_{P},F_{A}),$

where $W_{1}(\cdot)$ donates a $1\times 1$ convolution operation and
$ConvBlock(\cdot)$ signifies a series of convolution operations and activation
functions. This branch captures and processes the frequency domain
representation of features, mastering the global recovery of images.

Spatial Branch. We first use Layernorm to process the input features $F_{in}$
to get $F_{LN}$. The features then pass through two parallel sub-branches. The
first sub-branch simply SiLU activates them. The other sub-branch performs
spatial Mamba on features after $1\times 1$ convolution. Spatial Mamba
consists of sequence : DWConv $\rightarrow$ SiLU $\rightarrow$ Mamba-scan
$\rightarrow$ LN, where Mamba-scan means vanilla 2D Mamba-scan shown in Fig. 3
(a). The outputs of two sub-branches are then element-wise multiplied to yield
the spatial output $F_{s}$:

(6) $F_{s}=MambaScan(W_{1}(F_{LN}))\odot SiLU(F_{LN}),$

where $\odot$ donates element-wise multiplication.

Figure 5. Visualization of degradation prior attention maps and three-branch
features. (a) Rainy images. (b) Attention maps. (c), (d), and (e) are the
feature maps of the spatial, frequency band, and Fourier modeling branch. The
spatial branch (c) comprehensively recognizes raindrops but the boundaries are
blurry. The Fourier branch (e) outputs high-contrast features and focuses more
on larger rain streaks. Overall the frequency band branch (d) is somewhere in
between.

Frequency Band Branch. $F_{LN}$ is subjected to a 2-layer wavelet packet
transform to capture the information of different frequency bands with the
size of $\frac{H}{4},\frac{W}{4}$. They are arranged back to the original
resolution from top left to bottom right, then processed by frequency Mamba,
which is unique in its frequency scan as shown in Fig. 3 (b). Since the
strategy in Fig. 3 (a) cannot completely match WPT along the low to high
frequency, we use a strategy similar to LocalMamba (Huang et al., 2024). We
divide the wavelet features into four blocks (red line in Fig. 3 (b)) and scan
separately block by block.

The output is returned to the spatial domain after the corresponding k-layer
wavelet packet inverse transformation and lastly multiplied element by element
with $SiLU(F_{LN})$ to obtain the frequency band scanning output $F_{b}$. This
procedure is encapsulated as:

(7) $F_{b}=IDWT(FreqScan(DWT(F_{LN})))\odot SiLU(F_{LN}),$

where $WPT$ and $IWPT$ donate discrete wavelet packet transform and inverse
discrete wavelet packet transformation, respectively. We regard the frequency
dimension of Mamba modeling as a transition between Mamba and Fourier modeling
to achieve global degradation processing and local detail restoration
seamlessly.

$F_{in}$ is added with the output of the spatial branch using a residual
connection. By concatenating the output features of three branches and
applying a final $1\times 1$ convolution operation, the block achieves a
harmonized synthesis of the features. In Fig. 5, we present visualizations of
the features from different branches to illustrate the variations in rain
capture across each branch.

Figure 6. Error map (c) is the difference between the restored image (b) by
Restormer(Zamir et al., 2022) and the ground truth (d). There are
significantly larger errors in mountainous areas with dense rain streaks and
complex backgrounds, reflecting the uneven recovery difficulty and, more
importantly, the need to model degradation priors. Error map (f) is the
difference between our restoration result and GT with fewer huge error values.

1:

####  3.3.2. Data-dependent Degradation Prior Attention Map.

In various regions of the degraded image, the challenge of recovery varies
significantly, influenced by the degradation distribution and the image
background’s complexity. As illustrated in Fig. 6, the restoration performed
by Restormer (Zamir et al., 2022) exhibits a huge error with GT in areas such
as mountains, characterized by dense rain patterns and intricate backgrounds.
This discrepancy stems from the absence of explicit modeling of the
degradation in the rainy images.

To address this issue, we leverage the unique data-dependent property of the
Mamba module, which allows it to dynamically focus on or disregard specific
input features based on their significance. We introduce an innovative
approach wherein we utilize the Mamba module to generate degradation priors,
thereby enhancing our model’s ability to discern and address the varying
degrees of degradation across different regions of the image. This process
involves the generation of degradation priors through spatial 2D scanning of
input images at multiple scales. These degradation priors are then element-
wise multiplied with features corresponding to the same scale and subsequently
summed with the features:

(8) $\displaystyle M_{atten}$ $\displaystyle=Mamba(W_{1}(I_{in})),$
$\displaystyle F$ $\displaystyle=F\odot M_{atten}+F,$

where $I_{in}$ donates low-resolution rainy image and $F$ donates feature at
the same scale. By incorporating input features of different scales, our model
can effectively delineate the possible locations of degradation at various
levels of granularity. As shown in Fig. 5, we show some rainy images and the
corresponding degradation attention maps. The rain streaks are highlighted in
the attention maps.

In essence, this approach harnesses the inherent adaptability of the Mamba
module to dynamically allocate attention to different regions of the input
image, thereby enabling our model to better address the intricate of
degradation distribution across the image.

Table 1. Quantitative comparison (PSNR/SSIM) for Image Deraining on five
benchmark datasets. The highest and second-highest performances are marked in
bold and underlined. ’-’ indicates the result is not available.

Method | Venue |  Rain100H (Yang et al., 2017) |  Rain100L (Yang et al., 2017) |  Test2800 (Fu et al., 2017b) |  Test1200 (Zhang and Patel, 2018) | Param(M) | GFlops
---|---|---|---|---|---|---|---
PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$
DerainNet (Fu et al., 2017a) | TIP’17 | 14.92 | 0.592 | 27.03 | 0.884 | 24.31 | 0.861 | 23.38 | 0.835 | 0.058 | 1.453
UMRL (Yasarla and Patel, 2019) | CVPR’19 | 26.01 | 0.832 | 29.18 | 0.923 | 29.97 | 0.905 | 30.55 | 0.910 | 0.98 | -
RESCAN (Li et al., 2018) | ECCV’18 | 26.36 | 0.786 | 29.80 | 0.881 | 31.29 | 0.904 | 30.51 | 0.882 | 1.04 | 20.361
PreNet (Ren et al., 2019) | CVPR’19 | 26.77 | 0.858 | 32.44 | 0.950 | 31.75 | 0.916 | 31.36 | 0.911 | 0.17 | 73.021
MSPFN (Jiang et al., 2020) | CVPR’20 | 28.66 | 0.860 | 32.40 | 0.933 | 32.82 | 0.930 | 32.39 | 0.916 | 13.22 | 604.70
SPAIR (Purohit et al., 2021) | ICCV’21 | 30.95 | 0.892 | 36.93 | 0.969 | 33.34 | 0.936 | 33.04 | 0.922 | - | -
MPRNet (Zamir et al., 2021) | CVPR’21 | 30.41 | 0.890 | 36.40 | 0.965 | 33.64 | 0.938 | 32.91 | 0.916 | 3.64 | 141.28
Restormer (Zamir et al., 2022) | CVPR’22 | 31.46 | 0.904 | 38.99 | 0.978 | 34.18 | 0.944 | 33.19 | 0.926 | 24.53 | 174.7
Fourmer (Zhou et al., 2023) | ICML’23 | 30.76 | 0.896 | 37.47 | 0.970 | - | - | 33.05 | 0.921 | 0.4 | 16.753
IR-SDE (Luo et al., 2023) | ICML’23 | 31.65 | 0.904 | 38.30 | 0.980 | 30.42 | 0.891 | - | - | 135.3 | 119.1
MambaIR (Guo et al., 2024) | arxiv’24 | 30.62 | 0.893 | 38.78 | 0.977 | 33.58 | 0.927 | 32.56 | 0.923 | 31.51 | 80.64
VMambaIR (Shi et al., 2024) | arxiv’24 | 31.66 | 0.909 | 39.09 | 0.979 | 34.01 | 0.944 | 33.33 | 0.926 | - | -
FreqMamba (Ours) | - | 31.74 | 0.912 | 39.18 | 0.981 | 34.25 | 0.951 | 33.36 | 0.931 | 14.52 | 36.49

Figure 7. Visual quality comparison on an image from Rain100L(Yang et al.,
2017). Zoom in for better visualization.

Visual quality comparison on an image from Rain100L(Yang et al., 2017).

###  3.4. Loss Function

In addition to new modules and degradation priors, we also introduce new loss
functions to optimize the training process of the network to achieve good
results in both spatial and frequency domains. The loss function consists of
three parts: a spatial domain loss, a phase spectrum loss, and an amplitude
spectrum loss.

In the spatial domain, we use L1 loss between the final output and GT to
achieve supervision. At the same time, to achieve better global information
reconstruction, we calculate the L1 loss on the amplitude spectrum and phase
spectrum respectively to obtain the amplitude and phase spectrum loss,
expressed as:

(9) $\displaystyle L_{spa}$ $\displaystyle=\|Y_{out}-X_{gt}\|_{1},$
$\displaystyle L_{amp}$
$\displaystyle=\|\mathcal{A}(Y_{out})-\mathcal{A}(X_{gt})\|_{1},$
$\displaystyle L_{pha}$
$\displaystyle=\|\mathcal{P}(Y_{out})-\mathcal{P}(X_{gt})\|_{1},$

Finally, the overall composition of the loss function can be briefly expressed
as follows:

(10) $\displaystyle L_{total}$ $\displaystyle=L_{spa}+\alpha L_{amp}+\beta
L_{pha},$

where $\alpha$ and $\beta$ are both empirically set to 0.05 in our
implementation.

##  4\. EXPERIMENTS

In this section, we evaluate our method through extensive experiments. First,
We describe the experimental setup. Then we present the comparison results
between our method and state-of-the-art approaches qualitatively and
quantitatively. Lastly, a detailed ablation analysis of our proposed method is
conducted.

Figure 8. Visual quality comparison on an image from Rain100H(Yang et al.,
2017). Zoom in for better visualization.

Visual quality comparison on an image from Rain100H. Please zoom in for better
visualization.

###  4.1. Datasets and Implementation

Datasets. For validation, we train and validate our model on the widely-used
datasets Rain100H(Yang et al., 2017), Rain100L(Yang et al., 2017),
Test1200(Zhang and Patel, 2018), and Test2800(Fu et al., 2017b) datasets.
Rain100L is selected from BSD200 (Martin et al., 2001) with only one type of
rain streaks, which consists of 200 image pairs for training and 100 image
pairs for testing. Rain100H contains 1800 image pairs for training and 100
image pairs for testing with five types of streak directions. test1200(Zhang
and Patel, 2018) has three groups of 12,000 light, medium, and heavy rainy
images for training, with each group 4,000 images. The same three groups of
1,200 images are used for testing. Test2800(Fu et al., 2017b) contains 14000
image pairs, of which 12,600 pairs are used for training and 1,400 for
testing.

Implementation details. Our model was implemented using the Pytorch framework
and executed on NVIDIA RTX 3090 GPUs. The number of blocks in each layer
affects the amount of model parameters and rain removal performance. After
weighing the balance, we set blocks of each layer to [2, 3, 3, 4, 3, 3, 2],
which can achieve good performance with a reasonable amount of parameters. We
use the progressive training strategy. Taking the RAIN100L dataset as an
example, we set the total number of iterations to 75000 and set the image size
to [160, 256, 320, 384] while the corresponding batch sizes were [8, 4, 2, 1].
We use the Adam(Kingma and Ba, 2015) optimizer with default parameters to
train the network by minimizing the loss function $L_{total}$. The initial
learning rate is set to $3\times e{-4}$ and then gradually decays to $1\times
e{-6}$ using the cosine annealing strategy.

###  4.2. Comparison with State-of-the-art Methods

In this section, we compare our method with state-of-the-art deraining
approaches: DerainNet (Fu et al., 2017a), UMRL (Yasarla and Patel, 2019),
RESCAN (Li et al., 2018), PreNet (Ren et al., 2019), MSPFN (Jiang et al.,
2020) , SPAIR (Purohit et al., 2021) , MPRNet (Zamir et al., 2021) ,
Restormer(Zamir et al., 2022), Fourmer (Zhou et al., 2023), IR-SDE (Luo et
al., 2023), MambaIR (Guo et al., 2024), and VMambaIR (Shi et al., 2024) .

Quantitative Comparison. Following (Zamir et al., 2022), we compute Peak
Signal-to-Noise Ratio(PSNR) and Structural Similarity(SSIM) scores using the Y
channel in YCbCr color space. Tab. 1 reports the performance evaluation on the
four datasets. As can be seen, our method achieves the best performance among
all the baseline algorithms.

Qualitative Comparison. To demonstrate the enhanced fidelity and level of
detail exhibited by images generated by our proposed FreqMamba model for the
image rain removal task, we compare the visual quality of challenging degraded
images from the Rain100L in Fig. 7 and Rain100H datasets in Fig. 8. When faced
with complex or very severe rain streaks, our method achieves almost perfect
results. Compared with previous methods, our FreqMamba achieves the perfect
performance of global and local recovery. For example, by zooming in the red
box region in Fig. 8, our method removes more rain streak residue while better
restoring texture details. We provide more visualization results in the
supplementary material.

###  4.3. Ablation Studies

We performed ablations on different components of the model on the Rain100L
dataset. More results are provided in the supplementary.

Table 2. Ablation study for investigating the components of FreqMamba. The former three columns represent the three branches. Map refers to the attention map. Fourier | Frequency Band | Spatial | Map | PSNR | SSIM
---|---|---|---|---|---
| ✓ | ✓ | ✓ | 38.85 | 0.9782
✓ |  | ✓ | ✓ | 39.08 | 0.9801
✓ | ✓ |  | ✓ | 37.33 | 0.9758
✓ | ✓ | ✓ |  | 39.11 | 0.9803
✓ | ✓ | ✓ | ✓ | 39.18 | 0.9814

Investigation of space mamba branch. To verify the efficacy of the space mamba
branch, we replace Mamba blocks with standard convolution layers and keep
other parts unchanged. The result presented in Tab.2 shows that the
performance has dropped significantly, as the limited receptive field of
standard convolution results in its poor modeling capability.

Investigation of Fourier branch. We remove the Fourier branch, leaving the
spatial and frequency band mamba branch. The lack of global overall modeling
ability reduces model performance by 0.33 dB, which shows the importance of
modeling global information in the Fourier domain. It’s consistent with our
original intention.

Investigation of frequency band branch. We remove the frequency branch,
leaving the spatial mamba and Fourier branch. The model performance has
dropped by 0.1 dB, which shows the importance of the new frequency dimension.

Investigation of attention map. Our degradation prior attention strategy
adaptively learns degradation distributions based on rainy images at different
scales. To verify its effectiveness, we conduct ablation experiments in Tab.
2. It can be clearly seen that when equipped with the attention map, the
algorithm gives better results than that without using degradation prior.
Further, we show an error map in Fig. 9 with or without the attention map,
which demonstrates the effectiveness of our degradation prior attention
strategy. More comparisons can be found in the supplementary.

Investigation of loss function. The frequency loss aims to directly emphasize
global frequency information optimization. We remove it and its two components
in Tab. 3 respectively to check the validity. The results show that removing
it decreases performance metrics, indicating its importance.

Table 3. Ablation study of the contribution of loss functions. Loss function | PSNR $\uparrow$ | SSIM $\uparrow$
---|---|---
w/o $L_{freq}$ | 39.09 | 0.9795
w/o $L_{amp}$ | 39.12 | 0.9809
w/o $L_{pha}$ | 39.12 | 0.9807
Full Loss | 39.18 | 0.9814
Figure 9. The visualization of the errors between the restored image and
corresponding GT. With the employment of the degradation prior attention map,
the errors are greatly reduced.

###  4.4. Extensions on Other Tasks

To demonstrate the potential of our FreqMamba, we extend it to low-light image
enhancement and Real-world image dehazing.

Table 4. Quantitative results of different methods on the LOL-V1 and
LOL-V2-Syn dataset.

Method | LOL-V1 (Yang et al., 2017) | LOL-V2-Syn (Yang et al., 2017)
---|---|---
PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$
RetinexNet (Wei et al., 2018) | 18.38 | 0.7756 | 19.92 | 0.8847
KinD (Zhang et al., 2019) | 20.38 | 0.8248 | 22.62 | 0.9041
ZeroDCE (Guo et al., 2020) | 16.80 | 0.5573 | 17.53 | 0.6072
KinD++ (Zhang et al., 2021) | 21.30 | 0.8226 | 21.17 | 0.8814
URetinex-Net(Wu et al., 2022) | 21.33 | 0.8348 | 22.89 | 0.8950
FECNet (Huang et al., 2022b) | 22.24 | 0.8372 | 22.57 | 0.8938
SNR-Aware (Xu et al., 2022) | 23.38 | 0.8441 | 24.12 | 0.9222
FreqMamba (Ours) | 23.57 | 0.8453 | 24.46 | 0.9355

Extension on low-light image enhancement. Low-light image enhancement mainly
focuses on illuminating the darkness of the scene and removing amplified
noise. Our three-branch structure copes with this scenario well. We adopt
LOL-V1 and LOL-V2-Synthetic datasets to evaluate the performance of our
method. Several low-light image enhancement methods are selected for
comparison: RetinexNet (Wei et al., 2018), KinD (Zhang et al., 2019), ZeroDCE
(Guo et al., 2020), KinD ++ (Zhang et al., 2021), URetinex-Net (Wu et al.,
2022), FECNet (Huang et al., 2022b), and SNR-Aware (Xu et al., 2022). Tab. 4
shows the quantitative comparison.

Extension on real-world image dehazing. Real-world dehazing aims to restore a
clean scene from a real-world hazy image. For this task, we apply two
datasets: Dense-Haze (Ancuti et al., 2019)and NH-HAZE (Ancuti et al., 2020).
We compare our method with others including DCP (He et al., 2010), DehazeNet
(Cai et al., 2016), GridNet(Liu et al., 2019), MSBDN (Dong et al., 2020), and
AECR-Net (Wu et al., 2021). We present the quantitative comparison in Tab. 5.

Table 5. Quantitative results of different methods on the Dense-Haze and NH-HAZE dataset. Method | Dense-Haze (Ancuti et al., 2019) | NH-HAZE (Ancuti et al., 2020)
---|---|---
PNSR $\uparrow$ | SSIM $\uparrow$ | PNSR $\uparrow$ | SSIM $\uparrow$
DCP (He et al., 2010) | 10.06 | 0.3856 | 10.57 | 0.5196
DehazeNet (Cai et al., 2016) | 13.84 | 0.4252 | 16.62 | 0.5238
GridNet (Liu et al., 2019) | 13.31 | 0.3681 | 13.80 | 0.5370
MSBDN (Dong et al., 2020) | 15.37 | 0.4858 | 19.23 | 0.7056
AECR-Net (Wu et al., 2021) | 15.80 | 0.4660 | 19.88 | 0.7173
FreqMamba (Ours) | 17.35 | 0.5827 | 19.93 | 0.7372

##  5\. Conclusion

In this work, we introduce FreqMamba, an innovative deraining network that
seamlessly integrates spatial domain sequence modeling and frequency domain
global modeling to address the challenge of image deraining. The core of the
FreqMamba network utilizes the unique frequency SSM block. Meanwhile we
leverage Mamba’s input dependency properties to generate attention maps across
multiple scales. This integration enables a nuanced understanding and
processing of rain streaks, distinguishing our approach from existing methods.
Our comprehensive experiments on various datasets highlight the effectiveness
and efficiency of FreqMamba. Notably, the model is not only good at removing
rain from images but also preserves the integrity and detail of the underlying
scene. In applications where final image quality is critical, the balance of
performance and fidelity is critical. Furthermore, FreqMamba has applications
beyond rain removal, demonstrating its versatility in a variety of image
restoration tasks. This adaptability illustrates the robustness of the
underlying architecture and its potential as a foundational model for future
image restoration research.

## References

  * (1)
  * Ancuti et al. (2019) Codruta O Ancuti, Cosmin Ancuti, Mateu Sbert, and Radu Timofte. 2019.  Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. In _2019 IEEE international conference on image processing (ICIP)_. IEEE, 1014–1018.
  * Ancuti et al. (2020) Codruta O Ancuti, Cosmin Ancuti, and Radu Timofte. 2020.  NH-HAZE: An image dehazing benchmark with non-homogeneous hazy and haze-free images. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_. 444–445.
  * Bhirangi et al. (2024) Raunaq Bhirangi, Chenyu Wang, Venkatesh Pattabiraman, Carmel Majidi, Abhinav Gupta, Tess Hellebrekers, and Lerrel Pinto. 2024.  Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling.  _arXiv preprint arXiv:2402.10211_ (2024).
  * Cai et al. (2016) Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, and Dacheng Tao. 2016.  Dehazenet: An end-to-end system for single image haze removal.  _IEEE transactions on image processing_ 25, 11 (2016), 5187–5198.
  * Dong et al. (2020) Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming-Hsuan Yang. 2020.  Multi-scale boosted dehazing network with dense feature fusion. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 2157–2167.
  * Fu et al. (2017a) Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley. 2017a.  Clearing the skies: A deep network architecture for single-image rain removal.  _IEEE TIP_ (2017).
  * Fu et al. (2017b) Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. 2017b.  Removing rain from single images via a deep detail network. In _CVPR_.
  * Fu et al. (2020) Xueyang Fu, Borong Liang, Yue Huang, Xinghao Ding, and John Paisley. 2020.  Lightweight Pyramid Networks for Image Deraining.  _IEEE Transactions on Neural Networks and Learning Systems_ 31, 6 (2020), 1794–1807.  https://doi.org/10.1109/TNNLS.2019.2926481
  * Gong et al. (2024) Haifan Gong, Luoyao Kang, Yitao Wang, Xiang Wan, and Haofeng Li. 2024.  nnmamba: 3d biomedical image segmentation, classification and landmark detection with state space model.  _arXiv preprint arXiv:2402.03526_ (2024).
  * Gu and Dao (2023) Albert Gu and Tri Dao. 2023.  Mamba: Linear-time sequence modeling with selective state spaces.  _arXiv preprint arXiv:2312.00752_ (2023).
  * Gu et al. (2021) Albert Gu, Karan Goel, and Christopher Ré. 2021.  Efficiently modeling long sequences with structured state spaces.  _arXiv preprint arXiv:2111.00396_ (2021).
  * Guo et al. (2020) Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. 2020.  Zero-reference deep curve estimation for low-light image enhancement. In _IEEE Conf. Comput. Vis. Pattern Recog._ 1780–1789.
  * Guo et al. (2024) Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. 2024.  MambaIR: A Simple Baseline for Image Restoration with State-Space Model.  _arXiv preprint arXiv:2402.15648_ (2024).
  * Guo et al. (2022) Xin Guo, Xueyang Fu, Man Zhou, Zhen Huang, Jialun Peng, and Zheng-Jun Zha. 2022.  Exploring Fourier Prior for Single Image Rain Removal.. In _IJCAI_. 935–941.
  * He et al. (2010) Kaiming He, Jian Sun, and Xiaoou Tang. 2010.  Single image haze removal using dark channel prior.  _IEEE transactions on pattern analysis and machine intelligence_ 33, 12 (2010), 2341–2353.
  * Huang et al. (2022a) Jie Huang, Yajing Liu, Xueyang Fu, Man Zhou, Yang Wang, Feng Zhao, and Zhiwei Xiong. 2022a.  Exposure Normalization and Compensation for Multiple-Exposure Correction. In _IEEE Conf. Comput. Vis. Pattern Recog._ 6043–6052.
  * Huang et al. (2022b) Jie Huang, Yajing Liu, Feng Zhao, Keyu Yan, Jinghao Zhang, Yukun Huang, Man Zhou, and Zhiwei Xiong. 2022b.  Deep Fourier-Based Exposure Correction Network with Spatial-Frequency Interaction. In _Eur. Conf. Comput. Vis._ Springer, 163–180.
  * Huang et al. (2024) Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. 2024.  LocalMamba: Visual State Space Model with Windowed Selective Scan.  _arXiv preprint arXiv:2403.09338_ (2024).
  * Jiang et al. (2020) Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. 2020.  Multi-scale progressive fusion network for single image deraining. In _CVPR_.
  * Kang et al. (2011) Li-Wei Kang, Chia-Wen Lin, and Yu-Hsiang Fu. 2011.  Automatic single-image-based rain streaks removal via image decomposition.  _IEEE transactions on image processing_ 21, 4 (2011), 1742–1755.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR_.
  * Lee et al. (2018) Jae-Han Lee, Minhyeok Heo, Kyung-Rae Kim, and Chang-Su Kim. 2018.  Single-image depth estimation based on Fourier domain analysis. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 330–339.
  * Li et al. (2023) Chongyi Li, Chun-Le Guo, Man Zhou, Zhexin Liang, Shangchen Zhou, Ruicheng Feng, and Chen Change Loy. 2023.  Embedding fourier for ultra-high-definition low-light image enhancement.  _arXiv preprint arXiv:2302.11831_ (2023).
  * Li et al. (2018) Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, and Hongbin Zha. 2018.  Recurrent squeeze-and-excitation context aggregation net for single image deraining. In _ECCV_.
  * Liu et al. (2019) Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. 2019.  Griddehazenet: Attention-based multi-scale network for image dehazing. In _Proceedings of the IEEE/CVF international conference on computer vision_. 7314–7323.
  * Liu et al. (2024) Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. 2024.  Vmamba: Visual state space model.  _arXiv preprint arXiv:2401.10166_ (2024).
  * Luo et al. (2015) Yu Luo, Yong Xu, and Hui Ji. 2015.  Removing rain from a single image via discriminative sparse coding. In _Proceedings of the IEEE international conference on computer vision_. 3397–3405.
  * Luo et al. (2023) Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. 2023.  Image Restoration with Mean-Reverting Stochastic Differential Equations.  _International Conference on Machine Learning_ (2023).
  * Ma et al. (2024) Jun Ma, Feifei Li, and Bo Wang. 2024.  U-mamba: Enhancing long-range dependency for biomedical image segmentation.  _arXiv preprint arXiv:2401.04722_ (2024).
  * Martin et al. (2001) David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. 2001.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In _Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001_ , Vol. 2. IEEE, 416–423.
  * Mehta et al. (2022) Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. 2022.  Long range language modeling via gated state spaces.  _arXiv preprint arXiv:2206.13947_ (2022).
  * Pratt et al. (2017) Harry Pratt, Bryan Williams, Frans Coenen, and Yalin Zheng. 2017.  FCNN: Fourier Convolutional Neural Networks. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 786–798.
  * Purohit et al. (2021) Kuldeep Purohit, Maitreya Suin, AN Rajagopalan, and Vishnu Naresh Boddeti. 2021.  Spatially-adaptive image restoration using distortion-guided networks. In _ICCV_.
  * Qiao et al. (2024) Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. 2024.  VL-Mamba: Exploring State Space Models for Multimodal Learning.  _arXiv preprint arXiv:2403.13600_ (2024).
  * Ren et al. (2019) Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. 2019.  Progressive image deraining networks: A better and simpler baseline. In _CVPR_.
  * Shi et al. (2024) Yuan Shi, Bin Xia, Xiaoyu Jin, Xing Wang, Tianyu Zhao, Xin Xia, Xuefeng Xiao, and Wenming Yang. 2024.  VmambaIR: Visual State Space Model for Image Restoration.  _arXiv preprint arXiv:2403.11423_ (2024).
  * Smith et al. (2022) Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. 2022.  Simplified state space layers for sequence modeling.  _arXiv preprint arXiv:2208.04933_ (2022).
  * Valanarasu et al. (2022) Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M Patel. 2022.  Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2353–2363.
  * Wang et al. (2024) Chloe Wang, Oleksii Tsepa, Jun Ma, and Bo Wang. 2024.  Graph-mamba: Towards long-range graph sequence modeling with selective state spaces.  _arXiv preprint arXiv:2402.00789_ (2024).
  * Wang et al. (2022) Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. 2022.  Uformer: A general u-shaped transformer for image restoration. In _CVPR_.
  * Wei et al. (2018) Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. 2018.  Deep retinex decomposition for low-light enhancement.  _arXiv preprint arXiv:1808.04560_ (2018).
  * Wu et al. (2021) Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. 2021.  Contrastive learning for compact single image dehazing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10551–10560.
  * Wu et al. (2022) Wenhui Wu, Jian Weng, Pingping Zhang, Xu Wang, Wenhan Yang, and Jianmin Jiang. 2022.  Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 5901–5910.
  * Xiao et al. (2022) Jie Xiao, Xueyang Fu, Aiping Liu, Feng Wu, and Zheng-Jun Zha. 2022.  Image de-raining transformer.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2022).
  * Xu et al. (2021) Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. 2021.  A fourier-based framework for domain generalization. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 14383–14392.
  * Xu et al. (2022) Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and Jiaya Jia. 2022.  SNR-aware low-light image enhancement. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 17714–17724.
  * Yang et al. (2017) Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. 2017.  Deep joint rain detection and removal from a single image. In _CVPR_.
  * Yang and Soatto (2020) Yanchao Yang and Stefano Soatto. 2020.  FDA: Fourier Domain Adaptation for Semantic Segmentation.
  * Yasarla and Patel (2019) Rajeev Yasarla and Vishal M Patel. 2019.  Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining. In _CVPR_.
  * Yu et al. (2022) Hu Yu, Naishan Zheng, Man Zhou, Jie Huang, Zeyu Xiao, and Feng Zhao. 2022.  Frequency and spatial dual guidance for image dehazing. In _European Conference on Computer Vision_. Springer, 181–198.
  * Zamir et al. (2022) Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022.  Restormer: Efficient transformer for high-resolution image restoration. In _CVPR_.
  * Zamir et al. (2021) Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021.  Multi-stage progressive image restoration. In _CVPR_.
  * Zhang and Patel (2018) He Zhang and Vishal M Patel. 2018.  Density-aware single image de-raining using a multi-stream dense network. In _CVPR_.
  * Zhang et al. (2021) Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. 2021.  Beyond brightening low-light images.  _International Journal of Computer Vision_ 129 (2021), 1013–1037.
  * Zhang et al. (2019) Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. 2019.  Kindling the darkness: A practical low-light image enhancer. In _ACM Int. Conf. Multimedia_. 1632–1640.
  * Zhou et al. (2023) Man Zhou, Jie Huang, Chun-Le Guo, and Chongyi Li. 2023.  Fourmer: An efficient global modeling paradigm for image restoration. In _International Conference on Machine Learning_. PMLR, 42589–42601.
  * Zhou et al. (2022) Man Zhou, Jie Huang, Chongyi Li, Hu Yu, Keyu Yan, Naishan Zheng, and Feng Zhao. 2022.  Adaptively learning low-high frequency information integration for pan-sharpening. In _Proceedings of the 30th ACM International Conference on Multimedia_. 3375–3384.


utf8

# Violet: A Vision-Language Model for Arabic Image Captioning with Gemini
Decoder

Abdelrahman Mohamedξ Fakhraddin Alwajihλ El Moatez Billah Nagoudiλ
Alcides Alcoba Inciarteλ Muhammad Abdul-Mageedλ,ξ
λ Deep Learning & Natural Language Processing Group, The University of British
Columbia
ξDepartment of Natural Language Processing & Department of Machine Learning,
MBZUAI
<EMAIL_ADDRESS>
###### Abstract

Although image captioning has a vast array of applications, it has not reached
its full potential in languages other than English. Arabic, for instance,
although the native language of more than 400 million people, remains largely
underrepresented in this area. This is due to the lack of labeled data and
powerful Arabic generative models. We alleviate this issue by presenting a
novel vision-language model dedicated to Arabic, dubbed Violet. Our model is
based on a vision encoder and a Gemini text decoder that maintains generation
fluency while allowing fusion between the vision and language components. To
train our model, we introduce a new method for automatically acquiring data
from available English datasets. We also manually prepare a new dataset for
evaluation. Violet performs sizeably better than our baselines on all of our
evaluation datasets. For example, it reaches a CIDEr score of $61.2$ on our
manually annotated dataset and achieves an improvement of $13$ points on
Flickr8k.

##  1 Introduction

(a) امرأة وفتاة يلعبان بالفريزبي
على العشب>

(b) كلب أبيض وأسود يسبح
في الماء>

(c) قطة مستلقية بجانب جهاز
تحكم عن بعد>

(d) كعكة عيد ميلاد مع هاتف
محمول عليها>

Figure 1: Examples of captions generated by our model.

Captioning images involves describing the visual elements of a picture using
natural language. This requires a system that combines the strengths of two
models: one that can represent the visual elements of an image, and another
that can translate this representation into natural language. The latter
employs a language model to produce fluent (i.e., grammatically accurate) and
adequate (i.e., capturing sufficient semantic information) descriptions. In
recent years, research on vision language models (VLMs) and their applications
has boomed Alayrac et al. (2022); Wang et al. (2022); Huang et al. (2023).
Owing to the rapid advancements in large language models (LLMs), the
performance of VLMs has improved dramatically. More concretely, VLMs have
progressed from merely providing descriptions that vaguely resemble a given
image Vinyals et al. (2015) to accurately describing complex visual cues
within the image. The pretraining-then-finetuning paradigm also plays a
significant role in achieving such impressive results, as it allows models to
first grasp general language structures and then specialize in the specific
task of image captioning Gan et al. (2022).

Progress in VLMs, however, has been witnessed thus far primarily on English
Awais et al. (2023). This leaves behind a large number of other languages for
which no sufficient image captioning data or language models exist. Arabic is
a case in point where image captioning lags far behind Elbedwehy and Medhat
(2023). Similar to other low-resource languages, progress in Arabic image
captioning has been hampered by the lack of publicly available datasets and
limited efforts in creating any such data. Manual creation of image datasets,
after all, requires a huge amount of time and labor. Again, the unavailability
of powerful Arabic language models that understands the structure of the
language and can capture its rich morphology has also caused a delay in the
development of VLMs. Given the rapid progress in vision language technologies
and their wide applications in society, limited progress in this area can have
negative consequences for the Arabic-speaking world.

Figure 2: Performance of our model compared to previous works on Flickr8k
using CIDEr metric.

To address this important issue, we introduce a novel Arabic image captioning
model dubbed Violet. Our new model is comprised of two main components: a
vision encoder and a text decoder. For the vision encoder, we employ an object
detector network based on FasterRCNN Ren et al. (2015) to extract visual
features that are then passed to a compact transformer encoder. At the decoder
side, we leverage the recently developed generative pretrained model JASMINE
Nagoudi et al. (2022). Taking inspiration from Yu et al. (2022), we split our
text decoder into two halves: the first half functions as a text decoder,
whereas the second incorporates cross-attention layers, effectively serving as
a fusion decoder. Given the dual nature of our decoder, we refer to it as
Gemini. Drawing parallels with VisualGPT Chen et al. (2022) and the meshed
transformer Cornia et al. (2020), we also adopt a meshed connection between
the transformer vision encoder and the text decoder to foster enhanced
communication between the encoder and decoder layers.

Figure 3: The architecture and output generated by our model. We use an object
detection network to extract K object features (K equal $50$ in our case) from
an image. After projecting to a lower dimension, the features are fed to an
L-layer (three-layer in our architecture) transformer encoder. Meshed
connection is employed between the encoder and decoder layers, where each
encoder layer contributes to the cross-attention output. Our text decoder is
split into two halves, the first half is the standard frozen pretrained text
decoder layers, while the second half has cross-attention layers inserted
after each self-attention layer. We call this design a Gemini decoder. We
employ a gating mechanism through $\pi_{t}$ and $\pi_{m}$ that controls the
flow of information from the vision and language sides. The final input to the
feed forward network in each cross-attention layer is the weighted sum of each
encoder-decoder attention controlled by the $\alpha$ parameters.

The other major challenge we face in our work is the unavailability of native
Arabic captioning data. We alleviate this challenge by introducing a method
for automatically acquiring captions that is based on first employing a
powerful machine translation model followed by a quality assurance mechanism
for removing poor captions. For evaluation, in addition to reporting on Arabic
translated dataset, we task five human annotators to manually caption an image
dataset. Compared to previous works and baselines, our novel model excels in
captioning images in fluent Arabic. Figure 1 offers four examples of fluent
Arabic captions generated by our novel model. Figure 2 shows a comparison of
our model performance with prior research on Flickr8k in CIDEr score.

In summary, our contributions are as follows:

  * •

We present a novel image captioning model that employs an effective pretrained
Arabic decoder capable of outputting rich captions.

  * •

Our model achieves competitive performance for Arabic image captioning on both
the MSCOCO Lin et al. (2014) and Flicker8k Jia et al. (2014) datasets,
establishing a new state-of-the-art in this area.

  * •

In the process of developing our new model, we release a translated version of
MSCOCO dataset that has gone through our quality assurance pipeline. Our
released dataset can help further advance research in Arabic VLMs.

  * •

We also release our manually captioned dataset, a subset of MSCOCO test set,
that we dub AraCOCO.

##  2 Related Work

Image captioning. Early methods for image captioning involve either retrieving
descriptions Karpathy et al. (2014) or using template filling combined with
manually designed natural language generation techniques Yang et al. (2011);
Li et al. (2011). However, modern image captioning primarily relies on deep
learning models. In early work, image captioning is framed as an image-to-
sequence task using encoder-decoder models, with Convolutional Neural Networks
(CNNs) as encoders and Recurrent Neural Networks (RNNs) as decoders while
incorporating attention mechanisms Xu et al. (2015); You et al. (2016); Huang
et al. (2019). Soon after, using a transformer architecture of a vision
encoder with a text decoder became the defacto direction towards solving the
problem of image captioning Stefanini et al. (2022). Some approaches use a
detection model to extract visual features and then pass it to a transformer
text decoder as in Oscar Li et al. (2020); Chen et al. (2022), while others
like CoCa Yu et al. (2022) train a transformer vision encoder with a text
decoder from scratch on a large-scale dataset.

More recently, there has been a shift towards using pre-trained LLMs and
vision models. Generative Image-to-text Transformer (GIT) Wang et al. (2022)
is a decoder-only transformer that utilizes a CLIP Radford et al. (2021)
visual encoder to incorporate both visual and textual inputs. Another method
to consider is VisualGPT Chen et al. (2022) which uses a pretrained FasterCNN
to extract visual features that it passes to a small vision encoder. For the
decoder side, it uses the text-pretrained model GPT2 Radford et al. (2019).

Arabic image captioning. Arabic poses significant challenges to image
captioning. This is due to the lack of native Arabic captioning datasets in
the public domain, the morphological complexity of Arabic, and the large
number of diverse dialects Attai and Elnagar (2020). However, a number of
Arabic image captioning works exist. For instance, approaches such as root-
word based RNNs and deep neural networks are used for direct Arabic caption
generation Jindal (2017). Al-Muzaini et al. (2018) employ a generative merge
model with three components: an LSTM-based language model, a CNN-based image
feature extraction model, and a decoder that processes outputs from the first
two models. ElJundi et al. (2020) introduce an Arabic captioning model trained
on a translated Flickr8K dataset, discussing issues related to translation.
Afyouni et al. (2021) present AraCap, a hybrid design that combines a CNN with
object detection using attention mechanisms and produces captions through an
LSTM. They train their model on MSCOCO and Flickr30k Plummer et al. (2015)
datasets and test on an Arabic translated subset of MSCOCO. Lasheen and
Barakat (2022) propose an encoder-decoder structure, incorporating attention
mechanisms with CNN encoding and LSTM decoding. In another study Emami et al.
(2022), various Arabic image captioning models are formulated and assessed
using standard metrics. The authors use transformers pretrained on diverse
Arabic datasets following the architecture and training method introduced in
OSCAR Li et al. (2020). Elbedwehy and Medhat (2023) present a model employing
transformers for both encoding and decoding. It uses feature extraction from
images in the encoding stage and a pretrained word embedding model in the
decoding stage, all tested on the Arabic-translated Flickr8k dataset in
ElJundi et al. (2020). This work is closest to ours in that we also utilize
transformer encoders and decoders. However, we use a GPT-styled decoder that
endows our approach with high Arabic fluency.

##  3 Approach

###  3.1 Model Architecture

Our model is a vision-encoder-decoder architecture. For the vision encoder
part, we employ an object detection network Anderson et al. (2018) and a
three-layer transformer. For the text decoder, we use the pretrained
transformer decoder JASMINE Nagoudi et al. (2022). To align visual and textual
features, we utilize cross-attention. In standard attention, also known as
self-attention, the attention output is computed using three matrices derived
from the same input: the query matrix $Q$, the key matrix $K$, and the value
matrix $V$. More concretely, given an input sequence represented as a matrix
$S_{t}$, where each row corresponds to a vector in the sequence, the attention
is calculated as:

$\displaystyle
Attn(S_{t})=\text{softmax}\left(\frac{S_{t}W_{q}(S_{t}W_{k})^{T}}{\sqrt{d_{k}}}\right)S_{t}W_{v}$
(1)

Where $W_{q}$, $W_{k}$, and $W_{v}$ are the learnable weight matrices for the
query $Q$, key $K$, and value $V$ respectively. $d_{k}$ is the dimensionality
of the query/key vectors. The division by $\sqrt{d_{k}}$ is a scaling factor
to ensure the dot products don’t grow too large as the dimensionality
increases.

In the case of cross-attention, the query is derived from the output of the
text decoder’s self-attention, while the key and value are sourced from the
vision encoder. Mathematically, given image visual features output $S_{m}$,
and the textual features $S_{t}$, the formula becomes:

$\displaystyle XAttn(S_{t},S_{m})$
$\displaystyle=\text{softmax}\left(\frac{(S_{t}W_{q})(S_{m}W_{k})^{T}}{\sqrt{d_{k}}}\right)$
(2) $\displaystyle\times S_{m}W_{v}$

Now that the attention mechanism foundations are laid out, we describe our
vision encoder and text decoder in detail.

####  3.1.1 Vision Encoder

Our vision encoder consists of two components: a pretrained object detection
network, and a three-layer transformer encoder. For the object detection
network, we employ bottom-up attention network Anderson et al. (2018). In our
initial experiments, it results in superior visual features compared to using
the vanilla FasterRCNN model Ren et al. (2015). Previous works Li et al.
(2020); Cornia et al. (2020); Chen et al. (2022) also show the effectiveness
of this network in feature extraction. The transformer encoder, on the other
hand, is a three-layer standard transformer architecture that takes the output
of the detection network to further refine the visual features. For each
image, the detection network detects the potential objects and extracts the
visual features from their bounding boxes.111A bounding box is a region in the
image that contains the object. These visual features are passed through a
projection layer and then fed to the three-layer transformer encoder as input.
We adapt meshed connection Cornia et al. (2020) in our architecture between
the encoder layers and the text decoder. This allows all the encoder layers to
contribute to the input of the cross-attention rather than using only the
output of the last encoder layer. The contribution of each encoder layer is
determined by the learnable parameters matrix $\alpha$. For each layer $i$,
$\alpha_{i}$ is calculated as:

$\displaystyle\alpha_{i}=\sigma(W_{i}[S_{t}\parallel
XAttn(S_{m_{i}},S_{t})+b_{i}])$ (3)

Where $S_{t}$ is the input sequence of each decoder layer, $\sigma$ is the
sigmoid activation function, $W_{i}$ is a learnable weight matrix, $b_{i}$ is
a bias term and $\parallel$ indicates concatenation. This measures the
relevance between the input for each decoder layer $S_{t}$, and the output of
each encoder layer.

####  3.1.2 Gemini Decoder

We employ the pretrained Arabic decoder JASMINE Nagoudi et al. (2022) as our
text decoder. JASMINE is a decoder-based transformer that follows GPTNeo
architecture Black et al. (2021). JASMINE models range in complexity from
$300$ million to $13$ billion parameters and are trained on a text dataset of
approximately $400$GB, covering diverse Arabic varieties from multiple
domains. We utilize the JASMINE base variant in our architecture, which is a
12-layer transformer decoder with a $768$-dimensional embedding.

Although the meshed connection introduced in Cornia et al. (2020) proved to
have positive improvements on performance due to the richer visual features,
calculating the cross-attention of each encoder layer with each decoder layer
is computationally expensive. Inspired by Yu et al. (2022), we split our
pretrained text decoder into two parts. The first part acts as a vanilla text
decoder, while the second part acts as a fusion decoder that aligns visual and
textual features. This design choice serves two purposes. First, it reduces
the computations and the number of parameters by removing cross-attention
layers and the mesh connections in the first half of the decoder. Second,
having its first half intact acting as a vanilla text decoder, allows our
decoder to keep its innate generative capabilities, while also enabling
smoother convergence.

As shown in Figure 3, the first half has only the pretrained self-attention
layers of JASMINE. While the second half got cross-attention blocks inserted
in-between each layer, acting as a fusion decoder. To ensure maintaining the
functionality of our pretrained decoder, we freeze the first part that acts as
the text decoder. This modification not only decreases computational cost but
also positively impacts overall performance. In order to further enhance the
quality of the features generated by both the vision encoders and the text
decoder, we employ self-resurrecting activation unit (SRAU) introduced in Chen
et al. (2022). The process of generating a caption relies on visual cues to
convey the image’s content and textual cues to provide relationships between
words for a coherent and fluent output. To allow the important information to
flow without distortion, SRAU selectively permits the activation above a
certain threshold through a gating mechanism. This effectively filters out any
weak signal produced by either the vision or language part.

Concretely, as shown in Figure 3, for each encoder-decoder connection, the
output $Z_{i}$ to the feedforward layer is calculated as:

$Z_{i}=\pi_{m}\otimes XAttn(S_{t},S_{m_{i}})+\pi_{t}\otimes Attn(S_{t}),$ (4)

in which $\pi_{m}$ is the gating parameter for the vision part and $\pi_{t}$
for the text part, calculated as:

$\displaystyle\pi_{m}$
$\displaystyle=\sigma(A_{n})\mathbbm{1}(\sigma(A_{n})>\tau),$
$\displaystyle\forall n\in Attn(S_{t})$ $\displaystyle\pi_{t}$
$\displaystyle=(1-\sigma(A_{n}))\mathbbm{1}(1-\sigma(A_{n})>\tau)$
$\displaystyle\forall n\in Attn(S_{t})$

where $\sigma$ is the sigmoid function, $A_{n}$ is an element in the attention
matrix, $\mathbbm{1}$ is an indicator function that equals one if the
condition is true and zero otherwise, and $\tau$ is a hyperparameter. This
negates any disturbance caused by weak activations below the threshold $\tau$
by zeroing them out. The final output $Z$ to the feedforward layer will be the
sum of each encoder-decoder connection weighted by the learned parameter
$\alpha$ introduced earlier, mathematically:

$Z=\frac{1}{\sqrt{L}}\sum_{i=1}^{L}\alpha_{i}Z_{i}$ (5)

Where $L$ is the number of encoder layers, set to three in our architecture.

Figure 4: A comparison between the translations produced by Google translate
API and NLLB for MSCOCO dataset. Unlike NLLB, Google API tends to give literal
translations without incorporating the context.

###  3.2 Data Collection

Owing to the unavailability of high-quality Arabic captioning training data,
we first start by creating a training dataset for our model. Manually labeling
and creating a new dataset would be both time-consuming and expensive;
therefore, we opt for translating the commonly used captioning dataset
Microsoft Common Objects in Context (MSCOCO) Lin et al. (2014). There are two
famous training/validation splits for this dataset, the 2014 Karpathy’s split,
and the 2017 split. Both splits contain the same images and only differ in the
split ratio. The dataset covers around $80$ different objects in a total of
$123$k images with $5$ captions per image. The dataset is annotated manually,
which makes it suitable for evaluation. We create our dataset in two steps,
(i) translating the English MSCOCO, followed by (ii) a quality assurance step
to filter poor translations.

####  3.2.1 Machine Translation

In all of the previous attempts at Arabic image captioning pretraining ElJundi
et al. (2020); Sabri (2021); Emami et al. (2022), Google translate API Google
(2023) was used for translating the datasets. However, the quality of the
translations produced by it is not satisfactory. In Sabri (2021) it is
reported that from a random sample of $150$ examples, a whooping $46$% of the
translations obtained by Google API are unintelligible. Motivated by that, we
investigate Meta’s No Language Left Behind model (NLLB) model Costa-jussà et
al. (2022) for translation. Figure 4 illustrates a comparison between the
translations produced by the Google Translate API and NLLB for four sentences
sampled from MSCOCO dataset.

We conduct our comparison between the two translation models, Google Translate
API222The Google Translate API, integrated into Google Sheets, was used to
translate the subset of data utilized in the comparison. and NLLB, on two
aspects. First, we manually check the quality of $200$ sentences translated by
both models. Second, we calculate the perplexity of the translations of both
models using our JASMINE decoder. Perplexity calculates the probability of a
given sequence, providing insight into the fluency of the output translations.
Lower perplexity scores indicate better fluency, while higher scores indicate
poor fluency. This metric helps us to quantitatively gauge how good the
translations are, supplementing our manual evaluation to offer a comprehensive
understanding of the models’ performance. Subsequently, our observations
reveal that the Google API tends to provide a more literal translation in
comparison to NLLB. Empirically speaking, we find that $42$% of Google’s
translations are unintelligible, a stark contrast to the mere $15$% from NLLB.
Interestingly, this observation is consistent with findings presented in Sabri
(2021). Furthermore, when pitted against ChatGPT Ouyang et al. (2022), the
latter displays an impressive error rate of only $7$% in its translations.
However, we opted for NLLB due to its open-source nature.

English Caption |  NLLB Translation |  AraCOCO
---|---|---
An airport with large jetliners and a bus traveling on a tarmac. |  مطار مع طائرات كبيرة وحافلة |
تسافر على المدرج> |  أشجار النخيل أمام مطار به طائرتان كبيرتان |
للركاب وحافلات مكوكية.> |  |
a group of buses driving around at the airport |  مجموعة من الحافلات تسير في المطار> |  مجموعة من الحافلات تتجول
في المطار> |  |
Airplanes sit at the gate as transportation vehicles move about. |  الجيران تجلس عند البوابة بينما |
تتحرك مركبات النقل.> |  طائرات متوقفة عند بوابة |
المطار وهناك مركبات نقل> |  |
A busy runway with buses and luggage carts driving around |  مدرج مزدحم مع الحافلات وعربات |
الأمتعة التي تقود حولها> |  مدرج مزدحم مع حافلات وعربات |
أمتعة تتجول> |  |
An airplane and busses are lined up at the airport. |  طائرة وحافلات منتظمة |
في المطار> |  طائرة وحافلات منتظمة |
في المطار > |  |

Table 1: A comparison between original MSCOCO captions (first column), their
NLLB translations (second column), and AraCOCO captions (third column) for the
image in Figure 6.  Figure 5: Examples of the rejected translations from the
dataset and their semantic similarity to the English caption. Where orange
highlighting refers to poor translation, and red highlighting refers to poor
original caption.

####  3.2.2 Data Quality Assurance

Although NLLB in general provides better translations compared to that of
Google API, it can still output ‘hallucinations’ and ultimately poor
translations. This can be seen in the orange highlighted instances in Figure
5. Moreover, our manual inspection reveals that some English captions in the
original dataset are indeed incorrect. The MSCOCO training set can have
incomprehensible samples, typos, and even unrelated captions. Examples
highlighted in red in Figure 5 illustrate these poor cases. To mitigate this
issue, we employ a simple method based on semantic similarity that allows us
to identify and reject any such examples.

The semantic similarity of two sentences, as the term suggests, is an
indicator of the extent to which these two sentences align. A simple
comparison between the embeddings of the two sentences can be obtained by
passing each of them through a model and a metric such as cosine similarity
can be calculated to determine how alike the two embeddings are. The smaller
the angle between the two vectors, the higher the similarity score, indicating
that the sentences are closer in meaning. When the sentences are in different
languages, it is crucial to employ a multilingual model to generate accurate
embeddings, ensuring the semantic comparison remains valid across languages.
In our experiments, we employ sentence-BERT Reimers and Gurevych (2019) to
calculate the semantic similarity between each original caption and its
translation. We empirically chose a similarity score threshold of $0.6$,
rejecting all captions below that threshold. This results in removing a total
of $60$K samples from the whole dataset, which amounts to approximately $10$%
of the data.

####  3.2.3 AraCOCO Evaluation Dataset

Evaluating the performance of an Arabic captioning model presents a
significant challenge due to the limited availability of human captioned data.
To tackle this issue, we manually annotate a subset of $500$ images from the
MSCOCO test set, dubbing our resulting dataset AraCOCO. For each of the $500$
images, we acquire five distinct captions.

Figure 6: A sample from MSCOCO included in our AraCOCO.

To ensure diversity of image descriptions, we acquire captions from five
native Arabic-speaking annotators. The human labeling process is carried out
using Label Studio, a platform designed for such tasks. Each annotator is
presented with the same set of images and is asked to write an Arabic caption
describing the image given a unique English caption as a reference. We
encourage annotators to provide an Arabic caption that is more descriptive
whenever possible. That is, in cases where the English caption is not
capturing all details in the image, annotators are encouraged to capture these
lacking details in their Arabic captions. Each annotator gets to provide only
one caption per image, This approach ensures having multiple perspectives to
the captions on the same image. We provide an example image from AraCOCO in
Figure 6, along with five different captions each acquired from one annotator
in Table 1.

Model |  BLEU-1 $\uparrow$ |  BLEU-4 $\uparrow$ |  Rouge $\uparrow$ |  CIDEr $\uparrow$
---|---|---|---|---
VisualGPT |  $56.2$ |  $21.4$ |  $44.1$ |  $82.1$
Violet (w/o Gemini) |  $45.1$ |  $11.3$ |  $34.1$ |  $41.2$
Violet (w/ Gemini) |  $59.2$ |  $21.5$ |  $46.3$ |  $83.2$
Violet (w/ Gemini)  |  $\bf{60.3}$ |  $\bf{24.8}$ |  $\bf{47.2}$ |  $\bf{84.9}$
Table 2: Results on the translated MSCoco test set. VisualGPT is trained by us
on the MSCOCO dataset, and the outputs were translated using NLLB Costa-jussà
et al. (2022). (w/o Gemini) means using a normal text decoder with meshed
cross-attention in each layer.  indicates freezing the first part of the text
decoder.

##  4 Experiments

We analyze the performance of three variations of our architecture: (i) using
the normal decoder with cross-attention in each layer, (ii) using Gemini
decoder without freezing the text part, and (iii) using Gemini decoder while
freezing the text part. As a baseline, we train a VisualGPT model Chen et al.
(2022) on the English MSCOCO training set then translate output into Arabic
using NLLB. Our trained VisualGPT achieves a $117.8$ CIDEr score on the
English MSCOCO validation set. We conduct our experiments on three datasets,
as follows:

(i) Our translated MSCOCO: Following the Karpathy split, our translated and
filtered MSCOCO contains $543,817$ samples for training (Train), $22,845$
samples for validation (Dev), and $22,912$ samples for testing (Test). We
refer to this dataset as MSCOCO.

(ii) Translated Flickr8K: Similar to the original Flickr8k, the translated
dataset introduced in ElJundi et al. (2020) consists of $6,000$ images for
Train, $1,000$ images for Dev, and $1,000$ for Test. Each Image has three
captions, all translated using Google translate API. We refer to this dataset
simply as Flickr8K.

(iii) AraCOCO: As described in Section 3.2.3, AraCOCO consists of $500$ images
from Karpathy test split. Each image has five captions, all obtained from
human annotators.

###  4.1 Implementation Details

We use JASMINE base ($300$m) as our text decoder. While for the detection
network, following previous works Li et al. (2020); Cornia et al. (2020); Chen
et al. (2022), we employ bottom-up attention network Anderson et al. (2018)
based on Resnet-101 backbone He et al. (2016) with $2,048$ output features. We
also limit the maximum number of detections per image to $50$ bounding boxes.
The three-layer transformer encoder contains $12$ attention heads per layer
with $768$ embeddings dimension.

As we utilize the JASMINE decoder Nagoudi et al. (2022), we adopt its byte-
pair encoding (BPE) vocabulary where frequent character pairs are merged to
form subwords. This vocabulary encompasses $63,999$ tokens. For data
preprocessing, we employ a custom normalizer that removes punctuation and
repeated characters.

For the optimization part, in all experiments, we use AdamW Loshchilov and
Hutter (2019) with a learning rate of $1e^{-4}$, and empirically set $\tau$ to
$0.3$. The model is trained using a batch size of $60$ for $20$ epochs while
employing early stopping with a patience of $5$ on the validation loss. For
Flickr8k, we use our MSCOCO-pretrained model and only finetune it for one
epoch on Flickr8k’s training data. We employ a cross-entropy loss and train
the model in an auto-regressive manner, where the decoder predicts the next
token given the visual features and the previously generated textual tokens.

###  4.2 Results and Discussion

We evaluate the performance of our models against previous methods on the
popular evaluation metrics BLEU Papineni et al. (2002), ROUGE Lin (2004), and
CIDEr Vedantam et al. (2015). The results of our models on our MSCOCO dataset
are displayed in Table 2. Our Gemini decoder with six frozen layers (last row
in Table 2) achieves better performance while having fewer computations than
the unfrozen counterpart. Furthermore, it achieves around three points higher
CIDEr score compared to the translated VisualGPT outputs (first row in Table
2). The poor performance observed using the full decoder with cross-attention
layers (second row in Table 2), compared to other variants may be due to
sensitivity of the decoder parameters which end up being changed significantly
with full cross-attention across all its layers.

Model |  BLEU-1 |  BLEU-4 |  Rouge |  CIDEr
---|---|---|---|---
Elbedwehy and Medhat (2023) |  $\bf{58.7}$ |  $\bf{16.5}$ |  $\underline{38.0}$ |  $\underline{46.9}$
Emami et al. (2022) |  $39.0$ |  $09.0$ |  $33.4$ |  $42.3$
Violet |  $\underline{44.2}$ |  $\underline{13.0}$ |  $\bf{38.4}$ |  $\bf{60.1}$

Table 3: Results on Flickr8k test set from ElJundi et al. (2020). The results
are taken from the respective papers.

To compare our Arabic captioning model with previously published Arabic
models, we evaluate our model on the Flickr8k test set from ElJundi et al.
(2020). As shown in Table 3, our model achieves $2$ points better score on the
ROUGE metric, while having a substantial improvement over previous published
results in the CIDEr metric. Our model scores $13$ points higher than the best
model of the two previous models. On the other hand, our model falls behind in
BLEU score against Elbedwehy and Medhat (2023). It is worth noting, however,
that we are only comparing to published results of Elbedwehy and Medhat (2023)
since their model is not available (i.e., not released). They have also used
the validation set of Flickr8k in their training, and applied self-critical
Rennie et al. (2017) with no mention of the target data, thus giving their
model an advantage over our own model. Regardless, for image captioning, it is
known that the CIDEr (where our model excels) is a more relevant evaluation
metric than BLEU.

Model |  BLEU-1 |  BLEU-4 |  Rouge |  CIDEr
---|---|---|---|---
VisualGPT |  $52.7$ |  $17.6$ |  $40.2$ |  $58.5$
Violet |  $\bf{54.5}$ |  $\bf{19.0}$ |  $\bf{41.8}$ |  $\bf{61.2}$

Table 4: Results of our model against translated outputs of VisaulGPT on
AraCOCO.

Finally, we score our model on our manually annotated dataset, AraCOCO. As
shown in Table 4, our model again exhibits sizeable gains compared to our
baseline model (i.e., the translated output of VisualGPT). This means that we
cannot expect a satisfactory performance by simply taking output from a VLM
trained on English data and translating it into Arabic, further corroborating
our previous findings and motivating future work on developing VLM models that
natively tailored to Arabic language.

##  5 Conclusion

In this paper, we introduced Violet, an Arabic image captioning model
leveraging the pretrained text decoder JASMINE. Our results demonstrated the
efficacy of our Gemini decoder in enhancing performance while simultaneously
reducing the number of model parameters and computations. We also presented a
new method that is effective for acquiring Arabic captioning data from
available English data. In addition, we manually annotated a new dataset for
evaluating Arabic image captioning models. Our model outperforms all of our
baselines and promises to enable benchmarking in this area. We will release
our model and datasets to advance Arabic vison-language research.

##  6 Limitations

Similar to other image detection-based captioning models, the dependence on an
external network to provide the visual features introduces an additional layer
of complexity to the model. Since the model is not trained end to end, during
inference, the visual features must first be obtained from the detection
network before passing it to the vision encoder. Another limitation arises
from the constraints of the training data. Since MSCOCO focuses solely on $80$
class objects, the model’s applicability in real-world scenarios is
restricted. In our future work, we aim to address both of these limitations to
enhance Arabic models’ efficiency and broaden their practical usage.

##  7 Ethics Statement and Broad Impact

##### Bridging the Gap in Multilingual Image Captioning.

Image captioning serves as a crucial bridge between vision and language, with
its applications touching numerous domains such as accessibility, education,
and search engines. For a long time, the privilege of these advancements has
been constrained to a handful of languages, primarily due to the lack of
necessary datasets and dedicated research in other languages. Arabic, with its
vast speakers and rich history, has unfortunately been left behind in this
domain. Our work with Violet seeks to rectify this disparity, providing a
robust foundation for Arabic image captioning. By releasing Violet and the
datasets, we aim to invigorate research in this direction, promoting
inclusivity and equal opportunity in NLP and computer vision advancements
across languages.

##### Automated Data Acquisition and Transparency.

To overcome the challenge of limited labeled data for Arabic image captioning,
we employed a novel method for data acquisition using available English
datasets. While this approach provides a solution, it also warrants a
discussion on the accuracy, bias, and quality of the automatically acquired
data. We emphasize that while our method provides a foundational dataset,
manual annotations and human evaluations remain paramount for ensuring data
quality and avoiding propagation of errors.

##### Acknowledgment of Data Sources and Fair Credit.

Similar to ensuring proper credit assignment for benchmarking tasks, we
emphasize the importance of acknowledging the original data sources we
leveraged, especially in the context of automated data acquisition. Users and
researchers utilizing our datasets and model are encouraged to cite and
acknowledge the original datasets and sources. This practice ensures that
original creators receive the recognition they deserve and promotes a culture
of transparency and fairness in the research community.

## Acknowledgments

We acknowledge support from Canada Research Chairs (CRC), the Natural Sciences
and Engineering Research Council of Canada (NSERC; RGPIN-2018-04267), the
Social Sciences and Humanities Research Council of Canada (SSHRC;
435-2018-0576; 895-2020-1004; 895-2021-1008), Canadian Foundation for
Innovation (CFI; 37771), Digital Research Alliance of
Canada,333https://alliancecan.ca and UBC ARC-
Sockeye.444https://arc.ubc.ca/ubc-arc-sockeye We also thank Samar Magdy, Ahmed
Omar, and Karima Kadaoui for their help in creating the manually captioned
subset of MSCOCO, which we refer to as AraCOCO.

## References

  * Afyouni et al. (2021) Imad Afyouni, Imtinan Azhar, and Ashraf Elnagar. 2021.  Aracap: A hybrid deep learning architecture for arabic image captioning.  _Procedia Computer Science_ , 189:382–389.
  * Al-Muzaini et al. (2018) Huda A Al-Muzaini, Tasniem N Al-Yahya, and Hafida Benhidour. 2018.  Automatic arabic image captioning using rnn-lstm-based language model and cnn.  _International Journal of Advanced Computer Science and Applications_ , 9(6).
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6077–6086.
  * Attai and Elnagar (2020) Anfal Attai and Ashraf Elnagar. 2020.  A survey on arabic image captioning systems using deep learning models.  In _2020 14th International Conference on Innovations in Information Technology (IIT)_ , pages 114–119. IEEE.
  * Awais et al. (2023) Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. 2023.  Foundational models defining a new era in vision: A survey and outlook.  _arXiv preprint arXiv:2307.13721_.
  * Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  GPT-Neo: Large scale autoregressive language modeling with meshtensorflow.
  * Chen et al. (2022) Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. 2022.  Visualgpt: Data-efficient adaptation of pretrained language models for image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18030–18040.
  * Cornia et al. (2020) Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020.  Meshed-memory transformer for image captioning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10578–10587.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Elbedwehy and Medhat (2023) Samar Elbedwehy and T Medhat. 2023.  Improved arabic image captioning model using feature concatenation with pre-trained word embedding.  _Neural Computing and Applications_ , pages 1–17.
  * ElJundi et al. (2020) Obeida ElJundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem M Hajj, and Daniel C Asmar. 2020.  Resources and end-to-end neural network models for arabic image captioning.  In _VISIGRAPP (5: VISAPP)_ , pages 233–241.
  * Emami et al. (2022) Jonathan Emami, Pierre Nugues, Ashraf Elnagar, and Imad Afyouni. 2022.  Arabic image captioning using pre-training of deep bidirectional transformers.  In _Proceedings of the 15th International Conference on Natural Language Generation_ , pages 40–51.
  * Gan et al. (2022) Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. 2022\.  Vision-language pre-training: Basics, recent advances, and future trends.  _Foundations and Trends® in Computer Graphics and Vision_ , 14(3–4):163–352.
  * Google (2023) Google. 2023.  Google translate api.  Accessed: 15/07/2023.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Huang et al. (2019) Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. 2019.  Attention on attention for image captioning.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 4634–4643.
  * Huang et al. (2023) Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023.  Language is not all you need: Aligning perception with language models.  _arXiv preprint arXiv:2302.14045_.
  * Jia et al. (2014) Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. 2014.  Caffe: Convolutional architecture for fast feature embeding.  _arXiv preprint_ , arXiv:1408.5093.
  * Jindal (2017) Vasu Jindal. 2017.  A deep learning approach for arabic caption generation using roots-words.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Karpathy et al. (2014) Andrej Karpathy, Armand Joulin, and Li F Fei-Fei. 2014.  Deep fragment embeddings for bidirectional image sentence mapping.  _Advances in neural information processing systems_ , 27.
  * Lasheen and Barakat (2022) Moaz T Lasheen and Nahla H Barakat. 2022.  Arabic image captioning: the effect of text pre-processing on the attention weights and the bleu-n scores.  _Int J Adv Comput Sci Appl_ , 13(7):11.
  * Li et al. (2011) Siming Li, Girish Kulkarni, Tamara Berg, Alexander Berg, and Yejin Choi. 2011.  Composing simple image descriptions using web-scale n-grams.  In _Proceedings of the fifteenth conference on computational natural language learning_ , pages 220–228.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16_ , pages 121–137. Springer.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  _In ICLR_.
  * Nagoudi et al. (2022) El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, AbdelRahim Elmadany, Alcides Alcoba Inciarte, and Md Tawkat Islam Khondaker. 2022.  Jasmine: Arabic gpt models for few-shot learning.  _arXiv preprint arXiv:2212.10755_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Plummer et al. (2015) Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2641–2649.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28.
  * Rennie et al. (2017) Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017.  Self-critical sequence training for image captioning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7008–7024.
  * Sabri (2021) Sabri Monaf Sabri. 2021.  _Arabic image captioning using deep learning with attention_.  Ph.D. thesis, University of Georgia.
  * Stefanini et al. (2022) Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. 2022.  From show to tell: A survey on deep learning-based image captioning.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(1):539–559.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3156–3164.
  * Wang et al. (2022) Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022.  Git: A generative image-to-text transformer for vision and language.  _arXiv preprint arXiv:2205.14100_.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , pages 2048–2057. PMLR.
  * Yang et al. (2011) Yezhou Yang, Ching Teo, Hal Daumé III, and Yiannis Aloimonos. 2011.  Corpus-guided sentence generation of natural images.  In _Proceedings of the 2011 conference on empirical methods in natural language processing_ , pages 444–454.
  * You et al. (2016) Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016.  Image captioning with semantic attention.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4651–4659.
  * Yu et al. (2022) Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_.