(cvpr) Package cvpr Warning: Package ‘hyperref’ is not loaded, but highly
recommended for camera-ready version

# Defense against Adversarial Cloud Attack on Remote Sensing
Salient Object Detection

Huiming Sun1, Lan Fu2, Jinlong Li1, Qing Guo3,
Zibo Meng2, Tianyun Zhang1, Yuewei Lin4, Hongkai Yu1

1Cleveland State University. 2OPPO US Research Center.
3A*STAR SINGA. 4Brookhaven National Laboratory

###### Abstract

Detecting the salient objects in a remote sensing image has wide applications
for the interdisciplinary research. Many existing deep learning methods have
been proposed for Salient Object Detection (SOD) in remote sensing images and
get remarkable results. However, the recent adversarial attack examples,
generated by changing a few pixel values on the original remote sensing image,
could result in a collapse for the well-trained deep learning based SOD model.
Different with existing methods adding perturbation to original images, we
propose to jointly tune adversarial exposure and additive perturbation for
attack and constrain image close to cloudy image as Adversarial Cloud. Cloud
is natural and common in remote sensing images, however, camouflaging cloud
based adversarial attack and defense for remote sensing images are not well
studied before. Furthermore, we design DefenseNet as a learn-able pre-
processing to the adversarial cloudy images so as to preserve the performance
of the deep learning based remote sensing SOD model, without tuning the
already deployed deep SOD model. By considering both regular and generalized
adversarial examples, the proposed DefenseNet can defend the proposed
Adversarial Cloud in white-box setting and other attack methods in black-box
setting. Experimental results on a synthesized benchmark from the public
remote sensing SOD dataset (EORSSD) show the promising defense against
adversarial cloud attacks.

Figure 1: (a) Illustration of the proposed defense against the adversarial
cloud attacks for remote sensing salient object detection. (b) Performance (F
Measure) of the proposed DefenseNet against different adversarial cloud
attacks. Bigger area means better defense.

##  1 Introduction

The cross-domain research of computer vision and remote sensing has wide
applications in the real world, such as hyperspectral image classification [1,
2], cross-view geolocation [3, 4], scene classification [5, 6], change
detection [7, 8], aerial-view object detection [9, 10], and so on. Salient
Object Detection (SOD) in remote sensing images is to extract the salient
objects in a satellite or drone image, which might benefit many research works
mentioned above.

Some existing methods have been proposed for the SOD task in remote sensing
images [11, 12] using Convolutional Neural Network (CNN) based network
architecture, whose efforts are mainly focused on multi-scale feature
aggregation [11] and representative context feature learning [12]. However, in
some scenarios, these deep learning based remote sensing SOD models might
suffer from the attacks by the adversarial examples on deep neural networks.
Recent research [13] shows that the adversarial noises can be added to fool
the deep learning based SOD models, leading to the low SOD performance. For
example, by adding a small portion of adversarial noises on the original
remote sensing image between the image acquisition and data processing,
_e.g_., during the communication, the salient objects in the remote sensing
image might be hided or missed to some extents by the deep SOD model. This
kind of malicious attack exposes a potential security threat to the remote
sensing.

Many researches have been proposed for the adversarial examples based attack
and defense in deep learning [14, 15, 16, 17]. Meanwhile, some attack and
defense researches have been proposed for remote sensing tasks, such as the
remote sensing scene classification [18]. Different with existing methods
adding the perturbation on the original image, we propose to generate
Adversarial Cloud as attack to the deep learning based remote sensing SOD
model. Cloud is widely common in remote sensing images [19]. However, cloud
based adversarial attack and defense for remote sensing images has not been
well studied. The proposed Adversarial Cloud has realistic appearance close to
a normal cloud, which might be difficult to be perceived but will be malicious
in the remote sensing applications.

In this paper, we propose a novel DenfenseNet to defend the proposed
Adversarial Cloud attack to preserve the advanced SOD performance. In general,
the adversarial attack and defense networks will be trained with an
adversarial deep learning by iteratively training the Adversarial Cloud and
DenfenseNet. However, the already deployed deep remote sensing SOD model is
kept unchanged to simplify the real-world setting. Thus, the proposed
DefenseNet is designed as a learn-able pre-processing technique to preserve
the SOD performance. In specific, the adversarial examples will go through the
DefenseNet to become clean examples as the input to SOD models. Based on the
publicized remote sensing SOD dataset (EORSSD [12]), we build a benchmark by
synthesizing the Adversarial Cloud to test the performance of attack and
defense for the SOD problem in the remote sensing images. As shown in Fig. 1
(b), our proposed method could defend different adversarial attack methods.
Experimental results on the built benchmark show the effectiveness and
accuracy of the proposed method. The contributions of this paper are
summarized as follows.

  * •

This paper proposes a novel attack method by jointly tuning adversarial
exposure and additive perturbation and constraining image close to cloudy
image as Adversarial Cloud for the SOD in remote sensing images.

  * •

This paper proposes a novel DefenseNet as learn-able pre-processing against
the adversarial cloud attack for the safety-ensured SOD in remote sensing
images, without tuning the already deployed deep learning based SOD model.

  * •

By considering both regular and generalized adversarial examples, the proposed
DefenseNet can defend the proposed Adversarial Cloud in white-box setting and
other attack methods in black-box setting.

##  2 Related Work

###  2.1 Salient Object Detection for Remote Sensing

Salient object detection (SOD) is to automatically extract the salient objects
in an image. Many existing methods have been proposed for SOD in natural
images, while the SOD in optical remote sensing images is more challenging due
to the unique, complex and diverse environments [11]. SOD in satellite or
drone images has wide applications in remote sensing, such as building
extraction [20], Region-of-Interest extraction [21], airport detection [22],
oil tank detection [23], ship detection [24], _etc_.

Some traditional methods have been proposed for SOD in remote sensing images
by employing the bottom-up SOD models [25, 26, 21, 27, 28]. Recently, more
deep learning based SOD methods are proposed for the optical remote sensing
images [11, 29, 30, 12, 31]. The efforts of these deep learning based methods
are mainly focused on multi-scale feature aggregation, _e.g_., [11] and
representative context feature learning, _e.g_., [12]. Different with the
existing methods to improve the SOD performance on remote sensing images, this
paper is focused on the adversarial attack and defense of the deep learning
based SOD models.

###  2.2 Adversarial Attack

There are two types of adversarial attacks: white-box attacks, where the
adversary has full access to the target model, including its parameters,
_i.e_., the model is transparent to the adversary, and black-box attacks,
where the adversary has little knowledge of the target model. As the white-box
attacks are usually more destructive than black-box ones in practice, the
literature more focuses on the white-box attacks. Among these white-box
attacks, Szegedy _et al_. [32] used a box-constrained L-BFGS method to
generate effective adversarial attacks for the first time. After that, the
fast gradient sign method (FGSM) [14] used the sign of the gradient to
generate attacks, with $\ell_{\infty}$-norm bound. As a multi-step attack
method, the projected gradient descent (PGD) was proposed in [33]. Carlini and
Wagner [34] proposed the so-called CW attack which is a margin-based attack.
More recently, Croce  _et al_. introduced a parameter-free attack named
AutoAttack [35], which is an ensemble of four diverse attacks, including two
proposed variants of PGD attacks and two existing complementary attacks,
_i.e_., FAB [36] and Square Attack [37]. Besides the perturbation ones, the
attacks could also be the small geometric transformations [38, 39] or designed
adversarial patches [40, 41].

Figure 2: Structure of the proposed Adversarial Cloud (AdvCloud) based attack
and the proposed DefenseNet as the defense against the AdvCloud for the remote
sensing Salient Object Detection (SOD).
$\mathcal{N}_{\text{E}},\mathcal{N}_{\text{M}}$ are Gaussian noises for the
Attack Generalizion Module (AGM). Given a clean image $\mathbf{I}$ multipled
by Exposure matrix $\mathbf{E}$ and summation of cloud mask $\mathbf{M}$, the
synthesized cloudy image $\hat{\mathbf{I}}$ could be obtained. The DefenseNet
is a learn-able pre-processing for the SOD network.

###  2.3 Adversarial Defense

With the development of adversarial examples, studies on how to defend against
those attacks and improve the robustness of the neural networks emerge. Among
them, the most effective and widely used defense model is adversarial training
(AT), although the most straightforward way is simply by attaching a detection
network to detect and reject adversarial examples [42]. AT based models, which
aim to minimize the loss function to the strongest adversarial attacks within
a constraint, were first proposed by [14]. After that, a number of defending
methods [33, 43, 44, 45, 46, 47, 17, 48] based on adversarial training were
proposed. For example, [43] and [44] built a triplet loss to enforce a clean
image and its corresponding adversarial example has a short distance in
feature space. TRADES [46] optimized the trade-off between robustness and
accuracy. In addition to focusing on the on-training model that utilizes
adversarial examples, [48] proposed to explore the information from the model
trained on clean images by using an attention guided knowledge distillation.
Besides the adversarial training, there are also a number of other defense
models have been designed. For example, Xie _et al_. [49] proposed feature
denoising models by adding denoise blocks into the architecture to defend the
adversarial attacks, while Cohen _et al_. [50] proposed to use randomized
smoothing to improve adversarial robustness. Several methods aimed to
reconstruct the clean image by using a generative model [51, 52, 53].

##  3 METHODOLOGY

###  3.1 Cloud Synthesizing for Remote Sensing

Given a clean remote sensing color image $\mathbf{I}\in\mathds{R}^{H\times
W\times 3}$, we aim to simulate a cloudy image via
$\hat{\mathbf{I}}=\text{Cloud}(\mathbf{I},\mathbf{E},\mathbf{M})$, where
$\mathbf{E}\in\mathds{R}^{H\times W\times 1}$ is an exposure matrix to define
exposure degree, $\mathbf{M}\in\mathds{R}^{H\times W\times 1}$ is a cloud mask
to simulate clouds, and $\text{Cloud}(\cdot)$ represents the cloudy image
synthesis function. The cloud mask $\mathbf{M}$ can be synthesized via a
summation of multi-scale random noises, and is defined as

$\displaystyle\mathbf{M}=\sum_{s}\mathbf{R}\left(\mathbf{f}(2^{s})\right)/2^{s},$
(1)

where $\mathbf{f}$ represents a randomizing function, $\mathbf{R}$ denotes a
resize process and $s$ is a scale factor. $\mathbf{f}$ produces random noises
with the image size $2^{s}$ followed by being resized by $\mathbf{R}$. $s$ is
a natural number with range $\in$ [1, $\text{log}_{2}N$], where $N=H\times W$
is the image size. Given a clean image $\mathbf{I}$, exposure matrix
$\mathbf{E}$, and cloud mask $\mathbf{M}$, we could synthesize a cloudy image
$\hat{\mathbf{I}}$ via

$\displaystyle\hat{\mathbf{I}}=\text{Cloud}(\mathbf{I},\mathbf{E},\mathbf{M})$
$\displaystyle=\mathbf{I}\odot\mathbf{E}\odot(1-\mathbf{M})+\mathbf{M},$ (2)

where $\odot$ denotes pixel-wise multiplication.

With this cloudy image synthesis, we could study the effects of cloud from the
viewpoint of adversarial attack by tuning the exposure matrix $\mathbf{E}$ and
cloud mask $\mathbf{M}$ to render the synthesized cloudy images to fool the
deep learning based SOD models. Later, we also employ these adversarial
examples, obtained by the proposed attack method, to study the defense
performance.

###  3.2 Network Architecture

In this section, we show the whole pipeline of adversarial cloud attack
(AdvCloud), and DefenseNet as attack and defense stages to fully explore the
cloud effects to a deployed deep SOD model in Fig. 2. In the attack stage,
given a clean image $\mathbf{I}$, an exposure matrix $\mathbf{E}$, a cloud
mask $\mathbf{M}$, a pre-trained deep remote sensing SOD model $\phi(\cdot)$,
and a well-trained discriminator $\mathcal{D}(\cdot)$, we aim to generate
adversarial cloudy image examples via the proposed AdvCloud. Then, we analyze
how the synthetic adversarial cloudy images hurt the SOD performance. As the
other main step of the pipeline, we perform defense process, _i.e_.,
DefenseNet, as a pre-processing stage for the adversarial images to generate
cloud-removed images as defense for the SOD model. The proposed DefenseNet can
avoid retraining the deep SOD model and make the salient object detector
process adaptive to cloudy images. For optimization, the proposed pipeline
aims to maximize the detection loss of SOD model and minimize the adversarial
loss of the discriminator in the attack stage to generate adversarial cloudy
images which are close to normal cloudy images, while minimizing the detection
loss of salient object detector by predicting a clean image in the defense
stage to maintain the accuracy of the SOD model.

###  3.3 Adversarial Cloud based Attack

In general, adversarial attack fails a deep model by adding an imperceptible
noise-like perturbation to an image under the guidance of the deep model. In
this work, we propose a novel adversarial attack method, _i.e_., AdvCloud, to
generate adversarial cloudy remote sensing images that can fool the SOD model
to verify the robustness of the SOD model.

By intuition, we can tune $\mathbf{E}$ and $\mathbf{M}$ to generate
adversarial cloudy images. Specifically, given $\mathbf{I}$, $\mathbf{E}$,
$\mathbf{M}$, and a pre-trained SOD detector $\phi(\cdot)$, we aim to tune the
$\mathbf{E}$ and $\mathbf{M}$ under a norm ball constraint by

$\displaystyle\operatornamewithlimits{arg\,max}_{\mathbf{E},\mathbf{M}}\mathcal{J}(\phi(\text{Cloud}(\mathbf{I},\mathbf{E},\mathbf{M})),y),$
$\displaystyle\text{subject~{}to~{}~{}}\|\mathbf{M}-\mathbf{M}_{0}\|_{\text{p}}\leq\epsilon_{\text{M}},\|\mathbf{E}-\mathbf{E}_{0}\|_{\text{p}}\leq\epsilon_{\text{E}},$
(3)

where $\mathcal{J}(\cdot)$ is the loss function of the SOD model $\phi(\cdot)$
under the supervision of the annotation label $y$. We set
$\epsilon_{\text{E}}$ and $\epsilon_{\text{M}}$ as the ball bound under
$L_{\text{p}}$ around their initialization (_i.e_., $\mathbf{E}_{0}$ and
$\mathbf{M}_{0}$) for the parameters $\mathbf{E}$ and $\mathbf{M}$ to avoid
the clean image $\mathbf{I}$ being changed significantly.

Similar to existing perturbation based adversarial attacks (_e.g_., [33]), the
object function, _i.e_., Eq. (3.3), can be optimized by gradient descent-based
methods. In specific: ❶ We initialize $\mathbf{E}_{0}$ as a mask with all
elements as 1 and set $\mathbf{M}_{0}$ via Eq. (1). Then, we get the initial
synthesized cloudy image by Eq. (2). ❷ We feed the synthesized image to the
SOD model $\phi(\cdot)$ and calculate the SOD loss $\ell$. ❸ We perform back-
propagation to obtain the gradient of $\mathbf{E}$ and $\mathbf{M}$ with
respective to the loss function. ❹ We calculate the sign of the gradient to
update the variables $\mathbf{E}$ and $\mathbf{M}$ by multiplying the sign of
their gradients with the corresponding step sizes for the next iteration,
which is formulated to

$\displaystyle\mathbf{\ell}$
$\displaystyle=\mathcal{J}(\phi(\text{Cloud}(\mathbf{I},\mathbf{E}_{i},\mathbf{M}_{i})),y),$
$\displaystyle\mathbf{M}_{i+1}$
$\displaystyle=\mathbf{M}_{i}+\alpha_{\text{M}}\cdot\mathbf{sign}(\nabla_{\mathbf{M}_{i}}(\mathbf{\ell})),$
$\displaystyle\mathbf{E}_{i+1}$
$\displaystyle=\mathbf{E}_{i}+\alpha_{\text{E}}\cdot\mathbf{sign}(\nabla_{\mathbf{E}_{i}}(\mathbf{\ell}))),$
(4)

where $\alpha_{\text{M}}$ and $\alpha_{\text{E}}$ represents the step sizes,
and $i\in\\{0,1,\ldots,K-1\\}$ is the iteration number. ❺ We generate a new
adversarial cloudy image and loop from ❷ to ❹ for $K$ iterations.

Figure 3: Structure of the proposed DefenseNet. Algorithm 1 Defense algorithm
against the Adversarial Cloud based attack for remote sensing SOD.

Input: Clean images from the training set of EORSSD,
$\epsilon_{\text{M}}=0.03$, $\epsilon_{\text{E}}=0.06$, iteration $K=10$,
$\alpha_{\text{M}}=0.003$, $\alpha_{\text{E}}=0.015$, a pre-trained remote
sensing SOD detector $\phi(\cdot)$ [12], and a pre-trained discriminator
$\mathcal{D}(\cdot)$ obtained by pre-processing on training set. Output:
Adversarial Cloudy Images, parameter $\theta$ for DefenseNet.

1: repeat

2: Attack Step:

3:

  * •

Initial cloudy image synthesizing by Eq. (2) with $\mathbf{E_{0}}$ and
$\mathbf{M_{0}}$.

4:

  * •

Solve Eq. (3.3) via Eq. (3.3) to obtain optimal $\mathbf{E}$ and $\mathbf{M}$
with $K$ iterations for each image to learn the corresponding adversarial
cloudy image $\hat{\mathbf{I}}$.

5: Defense Step:

6:

  * •

Obtain the generalized adversarial cloudy image $\hat{\mathbf{I}}_{g}$ via Eq.
(8).

7:

  * •

Solve Eq. (9) via AdamW optimizer [54] to obtain optimal $\theta$ by fixed
$\mathbf{E}$ and $\mathbf{M}$ (_i.e_., an adversarial cloudy image
$\hat{\mathbf{I}}$, the generalized adversarial cloudy image
$\hat{\mathbf{I}}_{g}$).

8: until convergence or maximum epochs reached.

Figure 4: Defense against the remote sensing salient object detection attacks.
From top to bottom: normal cloudy image, attacked cloudy images by FGSM [14],
PGD [33], and the proposed AdvCloud. From left to right: cloudy images,
defense images by JPG Compression [55], FFA-Net [56], proposed DefenseNet,
each of which is followed by its corresponding SOD result.

To make the adversarial cloudy image $\hat{\mathbf{I}}$ have close visualized
perception to the normal cloudy image, we also incorporate a discriminator
$\mathcal{D}$ to align the distribution of normal cloudy images and
adversarial cloudy images to avoid artifacts which might be introduced by Eq.
(3.3). The inputs of the discriminator are an adversarial cloudy image
$\hat{\mathbf{I}}$ and a normal cloudy image $\mathbf{I}_{c}$, obtained by
$\mathbf{I}_{c}=\text{Cloud}(\mathbf{I},\mathbf{M})=\mathbf{I}\odot(1-\mathbf{M})+\mathbf{M}$,
then the adversarial training loss of the discriminator $\mathcal{D}$ is

$\displaystyle\mathcal{L}_{\mathcal{D}}(\hat{\mathbf{I}},\mathbf{I}_{c})$
$\displaystyle=\mathds{E}_{\mathbf{I}_{c}\sim\mathbf{X}_{c}}[\log(\mathcal{D}(\mathbf{I}_{c}))]$
$\displaystyle+\mathds{E}_{\hat{\mathbf{I}}\sim\hat{\mathbf{X}}}[\log(1-\mathcal{D}(\hat{\mathbf{I}}))],$
(5)

where $\mathbf{I}_{c}$ and $\hat{\mathbf{I}}$ are instances from normal cloudy
images set $\mathbf{X}_{c}$ and adversarial cloudy images set
$\hat{\mathbf{X}}$, respectively.

The whole attack pipeline, incorporating AdvCloud and discriminator
$\mathcal{D}$, is trained on the training set of the remote sensing SOD
dataset EORSSD [12]. The above setting has an assumption for a reliable
discriminator $\mathcal{D}$ ahead for the following inference stage.
Specifically, we alternatively freeze adversarial parameters $\mathbf{E}$,
$\mathbf{M}$ and the discriminator $\mathcal{D}$ to optimize the other one to
get the reliable discriminator $\mathcal{D}$ in the training set of EORSSDc
before the following inference stage.

For the inference stage of the proposed AdvCloud attack, we attack the testing
set of EORSSD guided by the pre-trained discriminator $\mathcal{D}(\cdot)$ and
the SOD detector $\phi(\cdot)$. Given a clean image $\mathbf{I}$ from the
testing set of EORSSD, exposure matrix $\mathbf{E}$ and cloud mask
$\mathbf{M}$, a well-trained discriminator $\mathcal{D}(\cdot)$, and a SOD
detector $\phi(\cdot)$, we tune $\mathbf{E}$ and $\mathbf{M}$ for $K$
iterations based on back-propagation, while the optimization function Eq.
(3.3) is reformulated to

$\displaystyle\operatornamewithlimits{arg\,max}_{\mathbf{E},\mathbf{M}}(\mathcal{J}(\phi(\text{Cloud}(\mathbf{I},\mathbf{E},\mathbf{M})),y)-\mathcal{L}_{\mathcal{D}}(\hat{\mathbf{I}},\mathbf{I}_{c})),$
$\displaystyle\text{subject~{}to~{}~{}}\|\mathbf{M}-\mathbf{M}_{0}\|_{\text{p}}\leq\epsilon_{\text{M}},\|\mathbf{E}-\mathbf{E}_{0}\|_{\text{p}}\leq\epsilon_{\text{E}},$
(6)

which means the adversarial cloudy image $\hat{\mathbf{I}}$ could fail the SOD
detector and have the realistic cloud appearance and pattern close to normal
cloudy images. Then, the updating process of variables $\mathbf{E}$ and
$\mathbf{M}$, in Eq. (3.3), is reformulated to

$\displaystyle\mathbf{\ell}$
$\displaystyle=\mathcal{J}(\phi(\text{Cloud}(\mathbf{I},\mathbf{E}_{i},\mathbf{M}_{i})),y),$
$\displaystyle\mathbf{M}_{i+1}$
$\displaystyle=\mathbf{M}_{i}+\alpha_{\text{M}}\cdot\mathbf{sign}(\nabla_{\mathbf{M}_{i}}(\mathbf{\ell}-\mathcal{L}_{\mathcal{D}}(\hat{\mathbf{I}},\mathbf{I}_{c}))),$
$\displaystyle\mathbf{E}_{i+1}$
$\displaystyle=\mathbf{E}_{i}+\alpha_{\text{E}}\cdot\mathbf{sign}(\nabla_{\mathbf{E}_{i}}(\mathbf{\ell}-\mathcal{L}_{\mathcal{D}}(\hat{\mathbf{I}},\mathbf{I}_{c})))).$
(7)

After obtaining the updated $\mathbf{E}$ and $\mathbf{M}$ for each image from
the testing set of EORSSD, we can get the corresponding adversarial cloudy
images via Eq. (2).

###  3.4 Defense against Adversarial Cloud

The proposed AdvCloud attack can easily hurt the SOD performance, while
performing defense against adversarial attack is an effective way to alleviate
such performance drop. In this section, we propose a DefenseNet as a learn-
able pre-processing for adversarial cloudy images to acquire cloud-removed
images for SOD models to improve the robustness. The proposed DefenseNet
contains the two following branches as the inputs.

Vanilla AdvCloud Branch. Given the updated adversarial attack variables
$\mathbf{E}$ and $\mathbf{M}$, we can obtain an adversarial cloudy image
$\hat{\mathbf{I}}$. Then, it is the first-branch input to the DefenseNet to
perform the reconstruction for adversarial cloud removal. This is a simple
white-box defense setting to make DefenseNet see the proposed AdvCloud attack
so as to defend it.

Generalized AdvCloud Branch. To benefit a black-box defense making DefenseNet
robust to other cloud based adversarial examples generated by different attack
methods which are never seen before, we design an Attack Generalization Module
(AGM) to include the generalized AdvCloud images. We use two different levels
of Gaussian noise to simulate the changes produced by the gradient-based
learned exposure matrix ($\mathbf{E}$) and cloud mask ($\mathbf{M}$) under a
specified budget. Specifically, we add Gaussian noise $\mathcal{N}_{\text{E}}$
= $\omega_{\text{E}}\cdot\mathcal{N}(\cdot)$ and $\mathcal{N}_{\text{M}}$ =
$\omega_{\text{M}}\cdot\mathcal{N}(\cdot)$ to $\mathbf{E}$ and $\mathbf{M}$
respectively to obtain $\mathbf{E}_{g}$ and $\mathbf{M}_{g}$ so as to extend
the distribution space of parameters around the gradient direction, where
$\mathcal{N}(\cdot)$ is a standard Gaussian random noise generation function
in the range of [-1, 1]. Then, we could acquire a generalized adversarial
cloudy image $\hat{\mathbf{I}}_{g}$ with the generalized $\mathbf{E}_{g}$ and
$\mathbf{M}_{g}$ via Eq. (2), _i.e_.,

$\displaystyle\hat{\mathbf{I}}_{g}=\text{Cloud}(\hat{\mathbf{I}},\mathbf{E}_{g},\mathbf{M}_{g}),$
(8)

as the second-branch input to the DefenseNet.

Figure 5: Example images of remote sensing datasets EORSSD, EORSSDc,
EORSSDadv. (a) clean image of EORSSD, (b) synthesized normal cloud, (c) clean
image with normal cloud leading to EORSSDc, (d) proposed adversarial cloud,
and (e) clean image with proposed adversarial cloud leading to EORSSDadv.

DefenseNet Loss. We feed adversarial cloudy images $\hat{\mathbf{I}}$ and
$\hat{\mathbf{I}}_{g}$ to the DefenseNet to output the cloud-removed images
$\mathbf{I}^{\prime}=\text{DefenseNet}(\hat{\mathbf{I}};\theta)$ and
$\mathbf{I}^{\prime}_{g}=\text{DefenseNet}(\hat{\mathbf{I}}_{g};\theta)$,
respectively. $\theta$ means the parameters of DefenseNet. In the defense
stage, the output cloud-removed images are optimized by the image
reconstruction loss function $L_{r}$ and regularization loss item $L_{reg}$.
The object function is shown below:

$\displaystyle\mathcal{L}=L_{r}(\mathbf{I}^{\prime},\mathbf{I})+L_{r}(\mathbf{I}^{\prime}_{g},\mathbf{I})+wL_{reg}(\mathbf{I}^{\prime},\mathbf{I}^{\prime}_{g}),$
(9)

where $\mathbf{I}$ is the clean image for $\hat{\mathbf{I}}$ and
$\hat{\mathbf{I}}_{g}$, and $w$ is the balance weight which is set to 0.1.
$L_{r}$ and $L_{reg}$ loss functions are both implemented as $L_{1}$ loss.

The whole algorithm flow for the defense against the Adversarial Cloud based
attack for remote sensing salient object detection is summarized in Algorithm
1.

###  3.5 Structure of Proposed DefenseNet

For implementation, we design the proposed DefenseNet shown in Fig. 3.
DefenseNet consists of 6 basic residual blocks, where each block includes 2
convolution layers, one ReLu layer, and one Batch Normalization layer. The
first four stages are adopted from ResNet, but the first convolution layer has
64 filters with a size of $3\times 3$ and stride of 1. This makes that the
early feature map has the same resolution as the input image, which can lead
to a bigger receptive field. There is also a bottleneck stage after the
encoder part, and it consists of three convolutions layers with 512 dilated
$3\times 3$ filters, and all these convolutions layers are also followed by a
batch normalization and a ReLu activation function. There is a residual block
from the input to the output, making the network to focus on the residual
learning.

##  4 Experiments

###  4.1 Experimental Setting

Benchmark Datasets: To evaluate the salient object detection in remote sensing
images, we use the public EORSSD dataset [12] to perform experiments. It has
2,000 remote sensing satellite images and corresponding pixel-level labeled
salient object detection ground truth, which includes 1,400 images for
training and 600 images for testing. The EORSSD dataset includes the objects
of Aircraft, Building, Car, Island, Road, Ship, Water, None, and Other in the
satellite images. This dataset is quite challenging with complicated scene
types, complex object attributes, comprehensive real-world satellite
circumstances, and some small-size objects, therefore it is more difficult
than the normal salient object detection datasets with natural images. Using
each clean image in EORSSD dataset, we generate its corresponding image with
the normal cloud, leading to a new synthetic dataset named EORSSDc. Similarly,
adding the proposed Adversarial Cloud (AdvCloud) to each clean image of EORSSD
dataset, we could generate a new synthetic dataset named EORSSDadv. Figure 5
shows some example images of the datasets EORSSD, EORSSDc, and EORSSDadv.

Figure 6: Visualization of normal cloudy image and attacked cloudy examples by
different attack methods.

Evaluation Metrics: We evaluate the remote sensing salient object detection
performance using F-measure ($\text{F}_{\beta}$), Mean Absolute Error (MAE)
score and S-measure ($\text{S}_{m}$), same as those in [12]. The larger
F-measure, S-measure values and lower MAE score mean the better remote sensing
SOD performance. Based on these metrics, we could also compare the performance
of attack and defense for the remote sensing SOD task.

Comparison Methods: For the attack experiment, we compare the proposed
AdvCloud method with five additive perturbation based white-box attack methods
on the EORSSDc dataset, _i.e_., FGSM [14], MIFGSM [57], PGD [33], VMIFGSM
[58], and NIFGSM [59]. The maximum perturbation for these comparison methods
is set to be 8 with pixel values in [0, 255]. These comparison attack methods
are applied on the testing images of EORSSDc.

Table 1: Baseline remote sensing SOD performance before and after the proposed
adversarial cloud (AdvCloud) attack. The budget for the perturbation
cloud/noise is 8 pixels. We mark white-box attacks with * and highlight the
best performance in red. The gray part means the black-box attacking.

| Attack Performance |  DAFNet [12] |  BasNet [60] |  U2Net [61] |  RRNet [31]
---|---|---|---|---|---
MAE $\uparrow$ |  Fβ $\downarrow$ |  Sm $\downarrow$ | MAE $\uparrow$ | Fβ $\downarrow$ | Sm $\downarrow$ | MAE $\uparrow$ | Fβ $\downarrow$ | Sm $\downarrow$ | MAE $\uparrow$ | Fβ $\downarrow$ | Sm $\downarrow$
| Clean Image | 0.0060 | 0.9049 | 0.9058 | 0.0162 | 0.8071 | 0.8871 | 0.0157 | 0.7890 | 0.8516 | 0.0077 | 0.9086 | 0.925
| Normal cloud | 0.0126 | 0.8253 | 0.8540 | 0.0295 | 0.7270 | 0.8352 | 0.0359 | 0.6170 | 0.7410 | 0.0100 | 0.8345 | 0.8917
ATTACK | FGSM | 0.0432 * | 0.2880 * | 0.5773 * | 0.0381 | 0.5974 | 0.7488 | 0.0441 | 0.5027 | 0.6743 | 0.0202 | 0.6815 | 0.7937
MIFGSM | 0.0497 * |  0.1292 * |  0.5247 * | 0.0452 | 0.5176 | 0.7063 | 0.0461 | 0.4666 | 0.6611 | 0.0208 | 0.6344 | 0.7695
PGD |  0.0680 * | 0.1376 * |  0.5166 * | 0.0401 | 0.5860 | 0.7478 | 0.0426 | 0.5142 | 0.6869 | 0.0169 | 0.7026 | 0.8060
VMIFGSM | 0.0497 * |  0.1326 * | 0.5267 * | 0.0463 | 0.4924 | 0.6952 | 0.0463 | 0.4564 | 0.6561 | 0.0245 | 0.5807 | 0.7416
NIFGSM | 0.0472 * | 0.1519 * | 0.5360 * | 0.0439 | 0.5176 | 0.7108 | 0.0456 | 0.4698 | 0.6623 | 0.0213 | 0.6354 | 0.7735
AdvCloud w/o Noise | 0.0256 * | 0.6583 * | 0.7556 * | 0.0311 | 0.7080 | 0.8198 | 0.0373 | 0.5930 | 0.7286 | 0.0120 | 0.8018 | 0.8671
AdvCloud w/o Exposure Matrix | 0.0484 * | 0.4265 * | 0.6435 * | 0.0317 | 0.7026 | 0.8145 | 0.0379 | 0.5953 | 0.7265 | 0.0116 | 0.8103 | 0.8765
AdvCloud |  0.0714 * | 0.2572 * | 0.5609 * | 0.0361 | 0.6396 | 0.7771 | 0.0404 | 0.5504 | 0.7072 | 0.0143 | 0.7484 | 0.8370

For the defense experiment, we compare our proposed DefenseNet with JPEG
Compression [55], FFA-Net [56], and DefenseFFA (using FFA-Net as the
backbone). The defense methods are all trained on EORSSDadv generated by
attacking DAFNet which aims to remove the adversarial attack to obtain the
clean image.

For evaluating the generalization ability of the proposed attack and defense
methods, we additionally employ three SOD detectors, _i.e_., BasNet [60],
U2Net [61], and RRNet [31]. All SOD models are trained on EORSSD dataset until
convergence.

Since the proposed AdvCloud are generated based on cloud, to ensure fairness
in evaluating the effectiveness of different SOD (Salient Object Detection)
models in attacking and defending against these adversarial examples, the
performance of 4 different SOD models should treat the EORSSDc as the starting
point for attacking rather than EORSSD.

Figure 7:  Visualization of defense performance across various SOD detection
methods, including DAFNet, BasNet, U2Net, and RRNet, in which each column
represents the mean testing performance under different attack methods on
EORSSDc and defense scenarios. The EORSSD and EORSSDc represent each
detector’s performance under clean image and cloudy image (both without
attack); the Attack column shows the mean performance under FGSM, MIFGSM, PGD,
VMIFGSM, NIFGSM, and AdvCloud attacks (Generated on DAFNet); and the
subsequent columns show the mean defense results when applying JPEG, FFA-Net,
DefenseNet, and DefenseFFA methods, respectively. The gray stripes indicate
the black-box defenses directly applied on attacked images without training.

Implementation Details: The SOD Network to be attacked is the deep learning
based remote sensing salient object detection network DAFNet [12] pre-trained
on the clean training images of EORSSD dataset. For the proposed AdvCloud
attack, we set $\epsilon_{\text{M}}=0.03$, $\epsilon_{\text{E}}=0.06$, and the
generalization random noise range of $\omega_{\text{M}}$, $\omega_{\text{E}}$
are 0.05 and 0.1, respectively. The input image is resized to $256\times 256$.
We use the AdamW optimization algorithm [54] for the network training with the
following hyper parameters: learning rate as 0.0001, batch size as 8, and
training epoch as 80. All the experiments were run on a single NVIDIA RTX 3090
GPU card (24G). We use PyTorch to implement the proposed method.

###  4.2 Experimental Results

Attack Result. Table 1 shows the quantitative SOD performance for the baseline
attack. When the dataset is clean, _i.e_., no cloud is added, the target SOD
network, DAFNet [12], achieves 0.9049 overall F-measure on EORSSD dataset.
After normal clouds are added to the EORSSD dataset, the F-measure decreases
to 0.8253. When the proposed AdvCloud is added to the EORSSD dataset, the SOD
network is misled by the adversarial examples and the F-measure is 0.2572.
This demonstrates that the proposed AdvCloud severely reduces the performance
of the SOD network. Furthermore, we compare the proposed AdvCloud with other
attack methods, as shown in Table 1. It shows that each attack method could
effectively reduce the SOD performance Moreover, the white-box attacks on
DAFNet can be effective to other SOD detectors with varying degrees of
decline.

Fig. 4 shows the qualitative comparisons among different attack methods and
their corresponding SOD map. Due to the attack, some objects predicted by the
SOD model are ignored (a, b, d) and misidentified (c) in Fig. 4. As we can
observe, the proposed attacked image is very similar to the normal cloud in
human perception compared to that from other attack methods. We can see
visible defect and moire on the attacked images by other attack methods in
Fig. 6. Therefore, the proposed AdvCloud is more visually close to normal
cloud but with very competitive attack performance.

Table 2: Ablation study for the defense SOD performance of proposed DefenseNet under different attack methods. DefenseNet‡: DefenseNet w/o Generalized AdvCloud, DefenseNet†: DefenseNet w/o Vanilla AdvCloud. The white-box defense is highlighted in red color.  Attack Methods | DefenseNet‡ | DefenseNet† | DefenseNet
---|---|---|---
MAE $\downarrow$ | Fβ $\uparrow$ | Sm $\uparrow$ | MAE $\downarrow$ | Fβ $\uparrow$ | Sm $\uparrow$ | MAE $\downarrow$ | Fβ $\uparrow$ | Sm $\uparrow$
FGSM [14] | 0.0373 | 0.4734 | 0.6652 | 0.0279 | 0.6161 | 0.7395 | 0.0260 | 0.6468 | 0.7548
MIFGSM [57] | 0.0554 | 0.3144 | 0.5966 | 0.0600 | 0.4010 | 0.6399 | 0.0569 | 0.4534 | 0.6651
PGD [33] | 0.0400 | 0.5256 | 0.6986 | 0.0267 | 0.6770 | 0.7783 | 0.0213 | 0.7244 | 0.8039
VMIFGSM [58] | 0.0659 | 0.2271 | 0.5535 | 0.0754 | 0.2844 | 0.5760 | 0.0762 | 0.3268 | 0.5917
NIFGSM [59] | 0.0517 | 0.3187 | 0.6004 | 0.0553 | 0.4027 | 0.6386 | 0.0516 | 0.4698 | 0.6689
Proposed AdvCloud | 0.0249 | 0.7033 | 0.8011 | 0.0182 | 0.7477 | 0.8227 | 0.0128 | 0.8226 | 0.8572
Mean | 0.0459 | 0.4271 | 0.6526 | 0.0439 | 0.5215 | 0.6992 | 0.0408 | 0.5740 | 0.7236
Table 3: Defense remote sensing SOD performance of normal cloudy images of
EORSSDc with SOD detector DAFNet.

Methods | MAE $\downarrow$ | Fβ $\uparrow$ | Sm $\uparrow$
---|---|---|---
Clean Image | 0.0060 | 0.9049 | 0.9058
Normal Cloud | 0.0126 | 0.8253 | 0.8540
JEPG Compression [55] | 0.0139 | 0.7913 | 0.8367
DefenseNet | 0.0171 | 0.7747 | 0.8315
FFA-Net [56] | 0.0144 | 0.8079 | 0.8492
DefenseNetFFA | 0.0126 | 0.8320 | 0.8620

Defense Result. Table 4 shows the defense remote sensing SOD performance under
different attack methods. It shows the defense methods effectively improve the
SOD performance after applying defense methods to adversarial examples
generated by attack strategies in Table 1. Fig. 7 shows the comprehensive
defense results on all of the attack strategies. We can clearly see that the
proposed defense method, _i.e_., as a pre-processing step, achieves better Fβ
and Sm gains comparing with FFA-Net. The proposed DefenseNet could not only
predominantly defend the proposed AdvCloud attack (_i.e_., white-box defense)
but also effectively defend other attack methods (_i.e_., black-box defense).
As shown in Table 4, the Fβ performance gain by the proposed DefenseNet and
DefenseNetFFA can be generalization to other defense methods under each attack
method. Despite of the proposed Defense method never seen other adversarial
attack images created by other attack methods during training, the proposed
Defense method trained on AdvCloud can still achieve better generalization
performance to defend against other attack methods, with the help of the
proposed Attack Generalization Module (AGM) shown in Table 2.

Table 4:  Defense performance on EORSSDc dataset.
DefenseNet${}_{\textit{FFA}}$ means the proposed DefenseNet using FFA-Net as
backbone. The gray part means the black-box defensing.

| Defense Performance |  DAFNet [12] |  BasNet [60] |  U2Net [61] |  RRNet [31]
---|---|---|---|---|---
MAE $\uparrow$ |  Fβ $\downarrow$ |  Sm $\downarrow$ | MAE $\uparrow$ | Fβ $\downarrow$ | Sm $\downarrow$ | MAE $\uparrow$ | Fβ $\downarrow$ | Sm $\downarrow$ | MAE $\uparrow$ | Fβ $\downarrow$ | Sm $\downarrow$
JPEG | FGSM | 0.0332 | 0.5084 | 0.6756 | 0.0389 | 0.5848 | 0.741 | 0.0427 | 0.5112 | 0.679 | 0.0204 | 0.6780 | 0.7900
MIFGSM | 0.0421 | 0.3409 | 0.6117 | 0.0434 | 0.5147 | 0.7082 | 0.0451 | 0.4625 | 0.6599 | 0.0205 | 0.6278 | 0.7681
PGD | 0.0323 | 0.5205 | 0.6939 | 0.0396 | 0.5851 | 0.7479 | 0.0418 | 0.5136 | 0.6874 | 0.017 | 0.6874 | 0.8049
VMIFGSM | 0.0485 | 0.2710 | 0.5745 | 0.0464 | 0.4868 | 0.6926 | 0.0459 | 0.4518 | 0.6531 | 0.0242 | 0.5818 | 0.7413
NIFGSM | 0.0422 | 0.3575 | 0.6139 | 0.0433 | 0.5211 | 0.7088 | 0.0447 | 0.4671 | 0.6628 | 0.0215 | 0.6262 | 0.7685
AdvCloud | 0.0242 | 0.6228 | 0.7486 | 0.0363 | 0.6334 | 0.7756 | 0.0401 | 0.5524 | 0.706 | 0.0144 | 0.7353 | 0.8312
DefenseNet | FGSM | 0.0260 | 0.6468 | 0.7548 | 0.025 | 0.7182 | 0.8247 | 0.0237 | 0.6776 | 0.7865 | 0.0159 | 0.7668 | 0.8414
MIFGSM | 0.0569 | 0.4534 | 0.6651 | 0.0265 | 0.7017 | 0.8154 | 0.0237 | 0.6776 | 0.7865 | 0.016 | 0.7476 | 0.8352
PGD | 0.0213 | 0.7244 | 0.8039 | 0.0265 | 0.7017 | 0.8154 | 0.0209 | 0.7210 | 0.8106 | 0.0126 | 0.7963 | 0.8656
VMIFGSM | 0.0762 | 0.3268 | 0.5917 | 0.0302 | 0.6689 | 0.7954 | 0.026 | 0.6523 | 0.7704 | 0.0194 | 0.6912 | 0.8040
NIFGSM | 0.0516 | 0.4698 | 0.6689 | 0.0165 | 0.7508 | 0.8345 | 0.0241 | 0.6762 | 0.7844 | 0.0165 | 0.7508 | 0.8345
AdvCloud | 0.0128 | 0.8226 | 0.8572 | 0.0193 | 0.7496 | 0.8549 | 0.0173 | 0.7644 | 0.8368 | 0.0111 | 0.8365 | 0.8952
FFANet | FGSM | 0.0292 | 0.5993 | 0.7309 | 0.0363 | 0.6260 | 0.7725 | 0.0369 | 0.5846 | 0.7264 | 0.0190 | 0.7015 | 0.8092
MIFGSM | 0.0535 | 0.4077 | 0.6427 | 0.0331 | 0.6607 | 0.7907 | 0.0322 | 0.6168 | 0.7485 | 0.0174 | 0.7185 | 0.8170
PGD | 0.0244 | 0.6861 | 0.7799 | 0.0332 | 0.6557 | 0.7873 | 0.0306 | 0.6439 | 0.7653 | 0.0139 | 0.7722 | 0.8529
VMIFGSM | 0.0692 | 0.3017 | 0.5838 | 0.0354 | 0.6280 | 0.7711 | 0.0334 | 0.5972 | 0.7367 | 0.0205 | 0.6722 | 0.7909
NIFGSM | 0.0484 | 0.4318 | 0.6518 | 0.0332 | 0.6557 | 0.7873 | 0.0330 | 0.6112 | 0.7441 | 0.0180 | 0.7210 | 0.8182
AdvCloud | 0.0145 | 0.7965 | 0.8443 | 0.0180 | 0.7826 | 0.8710 | 0.0165 | 0.7768 | 0.8462 | 0.0102 | 0.8423 | 0.8971
Defense${}_{\textit{FFA}}$ | FGSM | 0.0224 | 0.6995 | 0.7821 | 0.0316 | 0.6891 | 0.8095 | 0.0354 | 0.6072 | 0.7363 | 0.0136 | 0.7901 | 0.8561
MIFGSM | 0.0255 | 0.6488 | 0.7618 | 0.0279 | 0.7145 | 0.8244 | 0.0301 | 0.6479 | 0.7637 | 0.0138 | 0.7788 | 0.8551
PGD | 0.0149 | 0.7778 | 0.8313 | 0.0260 | 0.7294 | 0.8357 | 0.0288 | 0.6689 | 0.7781 | 0.0122 | 0.7971 | 0.8692
VMIFGSM | 0.0393 | 0.5338 | 0.6962 | 0.0291 | 0.6894 | 0.8113 | 0.0313 | 0.6278 | 0.7528 | 0.0159 | 0.7419 | 0.8306
NIFGSM | 0.0259 | 0.6512 | 0.7586 | 0.0276 | 0.7097 | 0.8227 | 0.0313 | 0.6378 | 0.7580 | 0.0139 | 0.7860 | 0.8573
AdvCloud | 0.0130 | 0.8178 | 0.8592 | 0.0171 | 0.7924 | 0.8761 | 0.0169 | 0.7834 | 0.8486 | 0.0097 | 0.8586 | 0.9031

Ablation Study for Proposed DefenseNet. The proposed DefenseNet has two input
branches, _i.e_., regular attack image branch and generalized attack image
branch. Table 2 shows both the regular attack branch and the generalized
attack branch contribute to the final defense SOD performance, where the best
defense performance is obtained when combining the two branches. If the branch
of generalized attack is removed, it will lead to more significant defense
performance drop. The DefenseNet contain AGM module cane provide a promising
and effective solution for generative defense on different adversarial
attacks.

Table 5: Image quality comparison of different cloudy attack methods with
DAFNet as the SOD detector. EORSSDC: normal cloudy images, EORSSD: original
clean images.

Methods | Compare with EORSSDC | Compare with EORSSD
---|---|---
SSIM$\uparrow$ | PSNR$\uparrow$ | L2$\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | L2$\downarrow$
Normal Cloud | 1 | - | 0.00 | 0.64 | 10.01 | 331.85
FGSM | 0.63 | 30.25 | 181.57 | 0.44 | 9.96 | 330.46
MIFGSM | 0.70 | 31.45 | 137.95 | 0.47 | 9.99 | 330.87
PGD | 0.79 | 33.54 | 85.49 | 0.53 | 9.99 | 331.15
VMIFGSM | 0.70 | 31.37 | 121.55 | 0.47 | 9.99 | 330.79
NIFGSM | 0.69 | 31.24 | 137.26 | 0.47 | 9.99 | 330.78
ADvCloud | 0.88 | 36.24 | 46.91 | 0.58 | 10.00 | 331.32

Discussion about Defense on Normal Cloudy Images.

The DefenseNetFFA’s performance in defense remote sensing SOD was assessed
using normal cloudy images of EORSSDc. The results in Table 3 indicate that
the proposed defense mechanism is capable of effectively defending against
anonymous types of attacks, while maintaining strong performance on normal
images. This suggests that our defense method is reliable and effective in
both attack and non-attack scenarios.

Discussion about Visual quality. The image quality comparison results are
shown in Table 5. It turns out that the proposed AdvCloud has better image
quality after attack. We use 8-pixel as the budget for the perturbation attack
noise $M$, same as all of the comparison methods. Combing with the observation
in Fig. 4, although our proposed attack method can not achieve the best attack
performance, our AdvCloud attack is more imperceptible comparing with other
attack methods.

##  5 Conclusion

In this paper, we proposed a new Adversarial Cloud to attack the deep learning
based remote sensing salient object detection model, meanwhile a new
DefenseNet as pre-processing defense is proposed to purify the input image
without tuning the deployed remote sensing deep SOD model. To study this
research problem, we synthesized new benchmarks EORSSDc with normal cloud and
EORSSDadv with the proposed adversarial cloud from the existing remote sensing
SOD dataset EORSSD. The extensive experiment on 4 SOD networks shows that the
proposed DefenseNet could well pre-process the attacked cloudy images as
defense against different adversarial attack methods without changing the
deployed remote sensing deep SOD model, while the SOD performance on the
remote sensing normal cloudy images without attack is still promising.

## References

  * [1] Q. Wang, J. Lin, and Y. Yuan, “Salient band selection for hyperspectral image classification via manifold ranking,” IEEE Transactions on Neural Networks and Learning Systems, vol. 27, no. 6, pp. 1279–1289, 2016.
  * [2] M. Zhang, W. Li, and Q. Du, “Diverse region-based cnn for hyperspectral image classification,” IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 2623–2634, 2018.
  * [3] Y. Tian, C. Chen, and M. Shah, “Cross-view image matching for geo-localization in urban environments,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3608–3616, 2017.
  * [4] S. Zhu, T. Yang, and C. Chen, “Revisiting street-to-aerial view image geo-localization and orientation estimation,” in IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 756–765, 2021.
  * [5] Q. Zou, L. Ni, T. Zhang, and Q. Wang, “Deep learning based feature selection for remote sensing scene classification,” IEEE Geoscience and Remote Sensing Letters, vol. 12, no. 11, pp. 2321–2325, 2015.
  * [6] S. Song, H. Yu, Z. Miao, Q. Zhang, Y. Lin, and S. Wang, “Domain adaptation for convolutional neural networks-based remote sensing scene classification,” IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 8, pp. 1324–1328, 2019.
  * [7] L. Bruzzone and D. F. Prieto, “An adaptive semiparametric and context-based approach to unsupervised change detection in multitemporal remote-sensing images,” IEEE Transactions on Image Processing, vol. 11, no. 4, pp. 452–466, 2002.
  * [8] B. Du, L. Ru, C. Wu, and L. Zhang, “Unsupervised deep slow feature analysis for change detection in multi-temporal remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 12, pp. 9976–9992, 2019.
  * [9] G.-S. Xia, X. Bai, J. Ding, Z. Zhu, S. Belongie, J. Luo, M. Datcu, M. Pelillo, and L. Zhang, “Dota: A large-scale dataset for object detection in aerial images,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
  * [10] J. Ding, N. Xue, Y. Long, G.-S. Xia, and Q. Lu, “Learning roi transformer for oriented object detection in aerial images,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2849–2858, 2019.
  * [11] C. Li, R. Cong, J. Hou, S. Zhang, Y. Qian, and S. Kwong, “Nested network with two-stream pyramid for salient object detection in optical remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 11, pp. 9156–9166, 2019.
  * [12] Q. Zhang, R. Cong, C. Li, M.-M. Cheng, Y. Fang, X. Cao, Y. Zhao, and S. Kwong, “Dense attention fluid network for salient object detection in optical remote sensing images,” IEEE Transactions on Image Processing, vol. 30, pp. 1305–1317, 2020.
  * [13] R. Gao, Q. Guo, F. Juefei-Xu, H. Yu, H. Fu, W. Feng, Y. Liu, and S. Wang, “Can you spot the chameleon? adversarially camouflaging images from co-salient object detection,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [14] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
  * [15] R. Gao, Q. Guo, F. Juefei-Xu, H. Yu, and W. Feng, “Advhaze: Adversarial haze attack,” arXiv preprint arXiv:2104.13673, 2021.
  * [16] F. Liao, M. Liang, Y. Dong, T. Pang, X. Hu, and J. Zhu, “Defense against adversarial attacks using high-level representation guided denoiser,” in IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778–1787, 2018.
  * [17] H. Zhang and J. Wang, “Defense against adversarial attacks using feature scattering-based adversarial training,” Advances in Neural Information Processing Systems, vol. 32, 2019.
  * [18] Y. Xu, B. Du, and L. Zhang, “Assessing the threat of adversarial examples on deep neural networks for remote sensing scene classification: Attacks and defenses,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 2, pp. 1604–1617, 2020.
  * [19] H. Sun, Y. Lin, Q. Zou, S. Song, J. Fang, and H. Yu, “Convolutional neural networks based remote sensing scene classification under clear and cloudy environments,” in IEEE/CVF International Conference on Computer Vision Workshop, pp. 713–720, 2021.
  * [20] E. Li, S. Xu, W. Meng, and X. Zhang, “Building extraction from remotely sensed images by integrating saliency cue,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 10, no. 3, pp. 906–919, 2016\.
  * [21] L. Ma, B. Du, H. Chen, and N. Q. Soomro, “Region-of-interest detection via superpixel-to-pixel saliency analysis for remote sensing image,” IEEE Geoscience and Remote Sensing Letters, vol. 13, no. 12, pp. 1752–1756, 2016\.
  * [22] Q. Zhang, L. Zhang, W. Shi, and Y. Liu, “Airport extraction via complementary saliency analysis and saliency-oriented active contour model,” IEEE Geoscience and Remote Sensing Letters, vol. 15, no. 7, pp. 1085–1089, 2018.
  * [23] Z. Liu, D. Zhao, Z. Shi, and Z. Jiang, “Unsupervised saliency model with color markov chain for oil tank detection,” Remote Sensing, vol. 11, no. 9, p. 1089, 2019.
  * [24] C. Dong, J. Liu, F. Xu, and C. Liu, “Ship detection from optical remote sensing images using multi-scale analysis and fourier hog descriptor,” Remote Sensing, vol. 11, no. 13, p. 1529, 2019.
  * [25] D. Zhao, J. Wang, J. Shi, and Z. Jiang, “Sparsity-guided saliency detection for remote sensing images,” Journal of Applied Remote Sensing, vol. 9, no. 1, p. 095055, 2015.
  * [26] L. Zhang, Y. Liu, and J. Zhang, “Saliency detection based on self-adaptive multiple feature fusion for remote sensing images,” International Journal of Remote Sensing, vol. 40, no. 22, pp. 8270–8297, 2019.
  * [27] L. Zhang and K. Yang, “Region-of-interest extraction based on frequency domain analysis and salient region detection for remote sensing image,” IEEE Geoscience and Remote Sensing Letters, vol. 11, no. 5, pp. 916–920, 2013.
  * [28] Y. Zhang, L. Zhang, and X. Yu, “Region of interest extraction based on multiscale visual saliency analysis for remote sensing images,” Journal of Applied Remote Sensing, vol. 9, no. 1, p. 095050, 2015.
  * [29] C. Li, R. Cong, C. Guo, H. Li, C. Zhang, F. Zheng, and Y. Zhao, “A parallel down-up fusion network for salient object detection in optical remote sensing images,” Neurocomputing, vol. 415, pp. 411–420, 2020.
  * [30] L. Zhang and J. Ma, “Salient object detection based on progressively supervised learning for remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, 2021.
  * [31] R. Cong, Y. Zhang, L. Fang, J. Li, Y. Zhao, and S. Kwong, “Rrnet: Relational reasoning network with parallel multiscale attention for salient object detection in optical remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–11, 2021.
  * [32] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in International Conference on Learning Representations, 2014.
  * [33] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in International Conference on Learning Representations, 2018.
  * [34] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in 2017 IEEE symposium on security and privacy (SP), 2017.
  * [35] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in International Conference on Machine Learning, 2020.
  * [36] F. Croce and M. Hein, “Minimally distorted adversarial examples with a fast adaptive boundary attack,” in International Conference on Machine Learning, 2020.
  * [37] M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein, “Square attack: a query-efficient black-box adversarial attack via random search,” in European Conference on Computer Vision, Springer, 2020.
  * [38] I. Goodfellow, H. Lee, Q. Le, A. Saxe, and A. Ng, “Measuring invariances in deep networks,” in Conference on Neural Information Processing Systems, 2009.
  * [39] C. Kanbak, S.-M. Moosavi-Dezfooli, and P. Frossard, “Geometric robustness of deep networks: analysis and improvement,” in IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [40] L. Huang, C. Gao, Y. Zhou, C. Xie, A. L. Yuille, C. Zou, and N. Liu, “Universal physical camouflage attacks on object detectors,” in IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [41] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” International Conference on Learning Representations Workshop, 2017.
  * [42] J. Lu, T. Issaranon, and D. Forsyth, “Safetynet: Detecting and rejecting adversarial examples robustly,” in International Conference on Conference on Computer Vision, 2017.
  * [43] C. Mao, Z. Zhong, J. Yang, C. Vondrick, and B. Ray, “Metric learning for adversarial robustness,” in Conference on Neural Information Processing Systems, 2019.
  * [44] Y. Zhong and W. Deng, “Adversarial learning with margin-based triplet embedding regularization,” in International Conference on Conference on Computer Vision, 2019.
  * [45] A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free!,” in Conference on Neural Information Processing Systems, 2019.
  * [46] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan, “Theoretically principled trade-off between robustness and accuracy,” in International Conference on Machine Learning, 2019.
  * [47] Y. Wang, D. Zou, J. Yi, J. Bailey, X. Ma, and Q. Gu, “Improving adversarial robustness requires revisiting misclassified examples,” in International Conference on Learning Representations, 2019.
  * [48] H. Wang, Y. Deng, S. Yoo, H. Ling, and Y. Lin, “Agkd-bml: Defense against adversarial attack by attention guided knowledge distillation and bi-directional metric learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7658–7667, 2021.
  * [49] C. Xie, Y. Wu, L. v. d. Maaten, A. L. Yuille, and K. He, “Feature denoising for improving adversarial robustness,” in IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [50] J. Cohen, E. Rosenfeld, and Z. Kolter, “Certified adversarial robustness via randomized smoothing,” in International Conference on Machine Learning, 2019.
  * [51] Y. Song, T. Kim, S. Nowozin, S. Ermon, and N. Kushman, “Pixeldefend: Leveraging generative models to understand and defend against adversarial examples,” in International Conference on Learning Representations, 2018\.
  * [52] B. Sun, N.-h. Tsai, F. Liu, R. Yu, and H. Su, “Adversarial defense by stratified convolutional sparse coding,” in IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [53] J. Yuan and Z. He, “Ensemble generative cleaning with feedback loops for defending adversarial attacks,” in IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [54] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.
  * [55] N. Das, M. Shanbhogue, S.-T. Chen, F. Hohman, S. Li, L. Chen, M. E. Kounavis, and D. H. Chau, “Shield: Fast, practical defense and vaccination for deep learning using jpeg compression,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 196–204, 2018.
  * [56] X. Qin, Z. Wang, Y. Bai, X. Xie, and H. Jia, “Ffa-net: Feature fusion attention network for single image dehazing,” in AAAI Conference on Artificial Intelligence, pp. 11908–11915, 2020.
  * [57] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [58] X. Wang and K. He, “Enhancing the transferability of adversarial attacks through variance tuning,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1924–1933, 2021.
  * [59] J. Lin, C. Song, K. He, L. Wang, and J. E. Hopcroft, “Nesterov accelerated gradient and scale invariance for adversarial attacks,” arXiv preprint arXiv:1908.06281, 2019.
  * [60] X. Qin, Z. Zhang, C. Huang, C. Gao, M. Dehghan, and M. Jagersand, “Basnet: Boundary-aware salient object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7479–7489, 2019.
  * [61] X. Qin, Z. Zhang, C. Huang, M. Dehghan, O. R. Zaiane, and M. Jagersand, “U2-net: Going deeper with nested u-structure for salient object detection,” Pattern recognition, vol. 106, p. 107404, 2020.


# Observational constraints on the fractal cosmology

Mahnaz Asghari<EMAIL_ADDRESS>Department of Physics, College of
Sciences, Shiraz University, Shiraz 71454, Iran Biruni Observatory, College of
Sciences, Shiraz University, Shiraz 71454, Iran Ahmad Sheykhi
<EMAIL_ADDRESS>Department of Physics, College of Sciences, Shiraz
University, Shiraz 71454, Iran Biruni Observatory, College of Sciences, Shiraz
University, Shiraz 71454, Iran

###### Abstract

In this paper, we explore a fractal model of the universe proposed by Calcagni
[JHEP03(2010)120] for a power-counting renormalizable field theory living in a
fractal spacetime. Considering a timelike fractal profile, we derived field
equations in fractal cosmology, in order to explore the structure formation
and the expansion history in fractal universe. Numerical investigations based
on matter power spectra diagrams report higher structure growth in fractal
cosmology, being in contrast to local galaxy surveys. Additionally, according
to the evolution of Hubble parameter diagrams, it can be understood that
Hubble constant would decrease in fractal cosmology, which is also
incompatible with low redshift estimations of $H_{0}$. So, concerning primary
numerical studies, it seems that fractal cosmology is not capable to alleviate
the tensions between local and global observational probes. Then, in pursuance
of more accurate results, we constrain the fractal cosmology by observational
data, including Planck cosmic microwave background (CMB), weak lensing,
supernovae, baryon acoustic oscillations (BAO), and redshift-space distortions
(RSD) data. The derived constraints on fractal dimension $\beta$ indicate that
there is no considerable deviation from standard model of cosmology.

##  1 Introduction

Recent observational data from type Ia supernovae (SNeIa) [1, 2], the cosmic
microwave background (CMB) anisotropies [3, 4, 5], large scale structures [6,
7, 8], and baryon acoustic oscillations (BAO) [9, 10, 11], confirm
$\mathrm{\Lambda}$CDM as the concordance model of cosmology. On the other
hand, $\mathrm{\Lambda}$CDM model which is based on the theory of general
relativity (GR), encounters some observational discrepancies, principally
$\sigma_{8}$ and $H_{0}$ tensions. To be more specific, there is a
disagreement between low-redshift determinations and Planck CMB measurements
of matter perturbation amplitude, $\sigma_{8}$ [12, 5]. Moreover, direct
measurements of Hubble constant are in significant tension with Planck
observations [13, 14, 15, 16, 5]. So, inconsistencies between local and global
data, motivate cosmologists to prob beyond $\mathrm{\Lambda}$CDM model.
Accordingly, it is suggested to consider some corrections on GR, describing as
modified theory of gravity [17, 18, 19, 20].

On the other hand, Calcagni proposed an effective quantum field theory which
is power counting renormalizable and Lorentz invariant, living in a fractal
universe [21, 22]. The fractal nature firstly introduced by Mandelbrot in 1983
[23] suggests conditional cosmological principle in fractal universe, where
the universe appears the same from every galaxy. Thereafter, in 1986, Linde
[24] propounded a model of an eternally existing chaotic inflationary
universe, explaining a fractal cosmology. Accordingly, there are several
investigations on the theory of fractal cosmology in literatures. Rassem and
Ahmed [25] in 1996 considered a nonhomogeneous cosmological model with a
fractal distribution of matter which evolves to a homogeneous universe as time
passes. The conditional cosmological principle in fractal cosmology is
discussed in [26, 27]. [28] studies Multi-fractal geometry. Thermodynamics of
the apparent horizon in a fractal universe is explored in [29]. In order to
find more theoretical studies on fractal cosmology refer to e.g. [30, 31, 32,
33, 34]. Furthermore, it is interesting to explore fractal models with
cosmological data as discussed in [35, 36, 37, 38, 39, 40] (also see [41] for
a review). Correspondingly, in the present work we are going to investigate
the fractal universe in background and perturbation levels, as well as
studying observational constraints on parameters of fractal cosmology.

The paper is organized as follows. Section 2 is dedicated to field equations
in a fractal universe. In section 3, we study the fractal model numerically,
and further we constrain the model with current observational data in section
4. We summarize our results in section 5.

##  2 Field equations in a fractal universe

The total action in a fractal spacetime is given by [21]

$\displaystyle S=\frac{1}{16\pi
G}\int{\mathrm{d}\mathcal{\varrho}(x)\,\sqrt{-g}\big{(}R-2\mathrm{\Lambda}-\omega\partial_{\mu}v\partial^{\mu}v\big{)}}+S_{m}\,,$
(1)

where $\omega$ is the fractal parameter, $v$ is the fractional function, and
$\mathrm{d}{\varrho}(x)$ is a Lebesgue-Stieltjes measure. It is possible to
derive field equations from action (1) similar to scalar-tensor theories.
Thus, in a fractal universe we obtain [21]

$\displaystyle
R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}(R-2\mathrm{\Lambda})+g_{\mu\nu}\frac{\Box{v}}{v}-\frac{\nabla_{\mu}\nabla_{\nu}v}{v}$
$\displaystyle+\omega\Big{(}\frac{1}{2}g_{\mu\nu}\partial_{\sigma}v\partial^{\sigma}v-\partial_{\mu}v\partial_{\nu}v\Big{)}=8\pi
GT_{\mu\nu}\,.$ (2)

Furthermore, continuity equation in a fractal spacetime takes the form [21]

$\displaystyle\nabla_{\mu}\big{(}vT^{\mu}_{\nu}\big{)}-\partial_{\nu}v\mathcal{L}_{m}=0\,.$
(3)

It should be noted that for $v=1$, standard equations in GR will be recovered.
Here, we focus on a timelike fractal, then $v$ is only time dependent given by

$v=H_{0}^{\beta}\Big{(}\frac{a}{a_{0}}\Big{)}^{\beta}\,,$ (4)

in which $\beta=4(1-\alpha)$ is the fractal dimension, and the parameter
$\alpha$ ranges as $0<\alpha\leq 1$.

We consider a fractal universe with the following flat Friedmann-Lemaître-
Robertson-Walker (FLRW) metric in the synchronous gauge

$\mathrm{d}s^{2}=a^{2}(\tau)\Big{(}-\mathrm{d}\tau^{2}+\big{(}\delta_{ij}+h_{ij}\big{)}\mathrm{d}x^{i}\mathrm{d}x^{j}\Big{)}\;,$
(5)

in which

$h_{ij}(\vec{x},\tau)=\int\mathrm{d}^{3}k\,e^{i\vec{k}.\vec{x}}\bigg{(}\hat{k}_{i}\hat{k}_{j}h(\vec{k},\tau)+\Big{(}\hat{k}_{i}\hat{k}_{j}-\frac{1}{3}\delta_{ij}\Big{)}6\eta(\vec{k},\tau)\bigg{)},$

with scalar perturbations $h$ and $\eta$, and $\vec{k}=k\hat{k}$ [42]. Then,
considering the energy content of the universe as a perfect fluid with
$T_{\mu\nu}=\big{(}\rho+p\big{)}u_{\mu}u_{\nu}+g_{\mu\nu}p$, field equations
in background level take the form

$\displaystyle H^{2}\bigg{(}1+\beta-\frac{1}{6}\omega
H_{0}^{2\beta}\beta^{2}\Big{(}\frac{a}{a_{0}}\Big{)}^{2\beta}\bigg{)}=\frac{8\pi
G}{3}\sum_{i}\bar{\rho}_{i}\,,$ (6)
$\displaystyle(\beta+2)\frac{H^{\prime}}{a}+H^{2}\bigg{(}3+2\beta+\beta^{2}+\frac{1}{2}\omega
H_{0}^{2\beta}\beta^{2}\Big{(}\frac{a}{a_{0}}\Big{)}^{2\beta}\bigg{)}$
$\displaystyle=-8\pi G\sum_{i}\bar{p}_{i}\,,$ (7)

where a prime indicates a deviation with respect to the conformal time. It can
be easily seen that, $\beta=0$ restores field equations in standard cosmology.
According to equation (6), total density parameter can be find as

$\mathrm{\Omega}_{\mathrm{tot}}=1+\beta-\frac{1}{6}\omega
H_{0}^{2\beta}\beta^{2}\Big{(}\frac{a}{a_{0}}\Big{)}^{2\beta}\,,$ (8)

where we have considered a universe filled with radiation (R), baryons (B),
dark matter (DM) and cosmological constant ($\mathrm{\Lambda}$). Also field
equations to linear order of perturbations can be written as

$\displaystyle\frac{a^{\prime}}{a}\Big{(}1+\frac{1}{2}\beta\Big{)}h^{\prime}-2k^{2}\eta=8\pi
Ga^{2}\sum_{i}\delta\rho_{i}\,,$ (9) $\displaystyle k^{2}\eta^{\prime}=4\pi
Ga^{2}\sum_{i}\big{(}\bar{\rho}_{i}+\bar{p}_{i}\big{)}\theta_{i}\,,$ (10)
$\displaystyle\frac{1}{2}h^{\prime\prime}+3\eta^{\prime\prime}+\frac{a^{\prime}}{a}\big{(}h^{\prime}+6\eta^{\prime}\big{)}\Big{(}1+\frac{1}{2}\beta\Big{)}-k^{2}\eta=0\,,$
(11)
$\displaystyle\frac{a^{\prime}}{a}\Big{(}2+\beta\Big{)}h^{\prime}+h^{\prime\prime}-2k^{2}\eta=-24\pi
Ga^{2}\sum_{i}\delta p_{i}\,.$ (12)

In addition, regarding equation (3), conservation equations of fractal
cosmology for $i$th component of the universe in background and perturbation
levels become

$\displaystyle\bar{\rho}^{\prime}_{i}+\big{(}3+\beta\big{)}\frac{a^{\prime}}{a}\big{(}\bar{\rho}_{i}+\bar{p}_{i}\big{)}=0\,,$
(13) $\displaystyle\delta^{\prime}_{i}=$
$\displaystyle-\frac{a^{\prime}}{a}\big{(}3+\beta\big{)}\bigg{[}\delta_{i}\big{(}c^{2}_{si}-w_{i}\big{)}$
$\displaystyle+\big{(}c^{2}_{si}-c^{2}_{ai}\big{)}\frac{a^{\prime}}{a}\big{(}3+\beta\big{)}\big{(}1+w_{i}\big{)}\frac{\theta_{i}}{k^{2}}\bigg{]}$
$\displaystyle-\big{(}1+w_{i}\big{)}\theta_{i}-\frac{1}{2}\big{(}1+w_{i}\big{)}h^{\prime}\,,$
(14) $\displaystyle\theta^{\prime}_{i}=$
$\displaystyle\theta_{i}\bigg{[}-\frac{a^{\prime}}{a}\big{(}4+\beta\big{)}+\frac{a^{\prime}}{a}\big{(}3+\beta\big{)}\big{(}1+w_{i}+c^{2}_{si}-c^{2}_{ai}\big{)}\bigg{]}$
$\displaystyle+\frac{k^{2}c^{2}_{si}}{1+w_{i}}\delta_{i}\,.$ (15)

Then, choosing $\beta=0$ would recover equations in concordance
$\mathrm{\Lambda}$CDM model.

Now that we have described main equations in fractal cosmology, it is possible
to derive observational constraints on fractal model, using a modified version
of the CLASS111Cosmic Linear Anisotropy Solving System code [43], and also
applying an MCMC222Markov Chain Monte Carlo approach via the Monte Python code
[44, 45].

##  3 Numerical results

In this section, we modify the CLASS cosmological Boltzmann code according to
the field equations of fractal model outlined in section 2. Correspondingly,
we take into account the Planck 2018 data [5] (i.e.
$\mathrm{\Omega}_{\mathrm{B},0}h^{2}=0.02242$,
$\mathrm{\Omega}_{\mathrm{DM},0}h^{2}=0.11933$,
$H_{0}=67.66\,\mathrm{km\,s^{-1}\,Mpc^{-1}}$, $A_{s}=2.105\times 10^{-9}$, and
$\tau_{\mathrm{reio}}=0.0561$) in our numerical investigation.

In Fig. 1 we depict the CMB temperature power spectra in fractal cosmology
compared to $\mathrm{\Lambda}$CDM model. Upper panels display the CMB power
spectra for different values of $\beta$, where we have considered $\omega=0$.
In lower panels the CMB temperature anisotropy diagrams are illustrated for
different values of $\omega$, while regarding $\beta=0.04$. Furthermore, for a
better comparison we have shown the relative ratio with respect to $\omega=0$
in the bottom-right panel, which indicates that the effect of $\omega$ is not
significant in CMB power spectra.

Figure 1:  Upper panels show the CMB power spectra diagrams (left) and their
relative ratio with respect to standard cosmological model (right) for
different values of $\beta$, considering $\omega=0$. Lower panels show the the
CMB power spectra diagrams for different values of $\omega$, considering
$\beta=0.04$, compared to $\mathrm{\Lambda}$CDM model (left) and their
relative ratio with respect to the case $\omega=0$ (right).

Figure 2 demonstrates the matter power spectra of fractal model in comparison
with standard cosmological model. In left panel we see matter power spectra
for different values of $\beta$, where $\omega$ is considered to be zero.
Right panel shows the power spectra for different values of $\omega$, in which
we have fixed $\beta$ to be $0.04$. Considering this figure, fractal cosmology
predicts a higher growth of structure which is in conflict with local probes
of large scale structures. In addition, this plot explains that the fractal
parameter $\omega$ has no remarkable influence on structure formation.

Figure 2:  Matter power spectra diagrams for different values of $\beta$
compared to $\mathrm{\Lambda}$CDM model, considering $\omega=0$ (left), and
analogous diagrams for different values of $\omega$, where $\beta=0.04$
(right).

Moreover, the evolution of Hubble parameter in fractal cosmology is depicted
in Fig. 3. Accordingly, we can conclude that Hubble tension is aggravated in
fractal model, and also the expansion history would not be affected by the
fractal parameter $\omega$.

Figure 3:  Hubble parameter in term of conformal time for different values of
$\beta$ compared to $\mathrm{\Lambda}$CDM model, where $\omega=0$ (left), and
analogous diagrams for different values of $\omega$, regarding $\beta=0.04$
(right).

##  4 Constraints from observational data

In this part, we employ the MCMC sampling package Monte Python to investigate
observational constraints on fractal model. In this respect, we use the
following set of cosmological parameters: {
$100\,\mathrm{\Omega}_{\mathrm{B},0}h^{2}$,
$\mathrm{\Omega}_{\mathrm{DM},0}h^{2}$, $100\,\theta_{s}$,
$\ln(10^{10}A_{s})$, $n_{s}$, $\tau_{\mathrm{reio}}$, $\beta$ },
where $\mathrm{\Omega}_{\mathrm{B},0}h^{2}$ and
$\mathrm{\Omega}_{\mathrm{DM},0}h^{2}$ stand for the baryon and cold dark
matter densities respectively, $\theta_{s}$ represents the ratio of the sound
horizon to the angular diameter distance at decoupling, $A_{s}$ indicates the
amplitude of the primordial scalar perturbation spectrum, $n_{s}$ stands for
the scalar spectral index, $\tau_{\mathrm{reio}}$ is the optical depth to
reionization, and $\beta$ is the fractal dimension. We have also put
constraints on four derived parameters which are reionization redshift
($z_{\mathrm{reio}}$), the matter density parameter
($\mathrm{\Omega}_{\mathrm{M},0}$), the Hubble constant ($H_{0}$), and the
root-mean-square mass fluctuations on scales of 8 $h^{-1}$ Mpc ($\sigma_{8}$).
Additionally, preliminary numerical explorations consider the prior range
[$0$, $0.04$] for the fractal dimension. It is worth mentioning that, since
the fractal parameter $\omega$ is irrelevant to cosmological observable (based
on numerical results in section 3), we fix $\omega$ to be zero in MCMC
analysis.

The dataset we use to constrain fractal model includes the Planck likelihood
with Planck 2018 data (containing high-$l$ TT,TE,EE, low-$l$ EE, low-$l$ TT,
and lensing) [5], the Planck-SZ likelihood for the Sunyaev-Zeldovich effect
measured by Planck [46, 47], the CFHTLenS likelihood with the weak lensing
data [48, 49], the Pantheon likelihood with the supernovae data [50], the BAO
likelihood with the baryon acoustic oscillations data [51, 52], and the BAORSD
likelihood for BAO and redshift-space distortions (RSD) measurements [53, 54].

The best fit and $1\sigma$ and $2\sigma$ confidence intervals for cosmological
parameters of fractal model and also $\mathrm{\Lambda}$CDM as the reference
model, from "Planck + Planck-SZ + CFHTLenS + Pantheon + BAO + BAORSD" dataset
are summarized in Table 1.

Table 1:  Best fit values and 68% and 95% confidence limits for cosmological
parameters from "Planck + Planck-SZ + CFHTLenS + Pantheon + BAO + BAORSD" data
set for $\mathrm{\Lambda}$CDM and fractal model.

|  $\mathrm{\Lambda}$CDM | fractal model
---|---|---
parameter | best fit | 68% & 95% limits | best fit | 68% & 95% limits
$100\,\mathrm{\Omega}_{\mathrm{B},0}h^{2}$ | $2.261$ | $2.263^{+0.012+0.026}_{-0.013-0.025}$ | $2.274$ | $2.265^{+0.014+0.028}_{-0.015-0.027}$
$\mathrm{\Omega}_{\mathrm{DM},0}h^{2}$ | $0.1163$ | $0.1164^{+0.00078+0.0015}_{-0.00079-0.0015}$ | $0.1164$ | $0.1168^{+0.00086+0.0017}_{-0.00093-0.0018}$
$100\,\theta_{s}$ | $1.042$ | $1.042^{+0.00029+0.00055}_{-0.00026-0.00053}$ | $1.042$ | $1.042^{+0.00030+0.00055}_{-0.00028-0.00054}$
$\ln(10^{10}A_{s})$ | $3.034$ | $3.024^{+0.010+0.023}_{-0.014-0.021}$ | $3.023$ | $3.025^{+0.0091+0.022}_{-0.014-0.021}$
$n_{s}$ | $0.9712$ | $0.9719^{+0.0036+0.0072}_{-0.0039-0.0074}$ | $0.9724$ | $0.9720^{+0.0036+0.0074}_{-0.0038-0.0072}$
$\tau_{\mathrm{reio}}$ | $0.05358$ | $0.04963^{+0.0041+0.010}_{-0.0074-0.0096}$ | $0.04706$ | $0.04919^{+0.0039+0.010}_{-0.0077-0.0092}$
$\beta$ | — | — | $0.0003854$ | $0.0005421^{+0.00012+0.00092}_{-0.00054-0.00054}$
$z_{\mathrm{reio}}$ | $7.502$ | $7.084^{+0.50+1.0}_{-0.69-1.0}$ | $6.811$ | $7.046^{+0.49+1.0}_{-0.70-1.0}$
$\mathrm{\Omega}_{\mathrm{M},0}$ | $0.2871$ | $0.2876^{+0.0043+0.0086}_{-0.0044-0.0086}$ | $0.2828$ | $0.2841^{+0.0056+0.010}_{-0.0050-0.012}$
$H_{0}\;[\mathrm{km\,s^{-1}\,Mpc^{-1}}]$ | $69.56$ | $69.54^{+0.37+0.73}_{-0.36-0.71}$ | $70.16$ | $70.08^{+0.46+1.3}_{-0.68-1.1}$
$\sigma_{8}$ | $0.8079$ | $0.8044^{+0.0045+0.0096}_{-0.0051-0.0091}$ | $0.8063$ | $0.8092^{+0.0049+0.012}_{-0.0067-0.011}$

Correspondingly, contour plots for some selected parameters of fractal model
compared to $\mathrm{\Lambda}$CDM are displayed in Fig. 4.

Figure 4:  The one-dimensional posterior distribution and two-dimensional
posterior contours with 68% and 95% confidence limits for the selected
cosmological parameters of fractal model (orange) compared to
$\mathrm{\Lambda}$CDM (blue).

Considering the obtained constraints on fractal dimension, observational data
detect no significant deviation from $\mathrm{\Lambda}$CDM model.

Moreover, the Akaike information criterion (AIC) defined as
$\mathrm{AIC}=-2\ln{\mathcal{L}_{\mathrm{max}}}+2K$ where
$\mathcal{L}_{\mathrm{max}}$ is the maximum likelihood function and $K$
indicates the number of free parameters [55, 56], results in
$\mathrm{AIC_{(\mathrm{\Lambda}CDM)}}=3847.12$ and
$\mathrm{AIC_{(fractal)}}=3849.70$, then consequently $\mathrm{\Delta
AIC}=2.58$. Thus, $\mathrm{\Lambda}$CDM provides a better fit to observations,
while fractal model is still allowed.

##  5 Conclusions

We have considered a fractal model of the universe introduced by Calcagni [21,
22], which is a power counting renormalizable and also Lorentz invariant
model. Taking into account a timelike fractal, we have investigated the
influence of fractal model on observables, mainly power spectra and Hubble
constant, by using a modified version of the publicly available CLASS code.
Considering matter power spectra diagrams in Fig. 2, there is an enhancement
in structure growth for non-zero values of fractal dimension $\beta$, being
incompatible with low redshift structure formation. Moreover, according to
Fig. 3, the current value of Hubble parameter decreases in fractal cosmology,
which is inconsistent with local determinations of $H_{0}$. Consequently,
primary numerical results indicate that fractal cosmology has no positive
impact on relieving cosmological tensions.

Furthermore, we put constraints on the parameters of the fractal universe by
utilizing current observations, chiefly Planck CMB, weak lensing, supernovae,
BAO, and RSD data. Numerical results based on MCMC analysis, detect no
significant departure from standard cosmological model. On the other hand, the
model selection criterion AIC affirms that $\mathrm{\Lambda}$CDM model is more
favored by observations, however, the fractal model can not be ruled out.

## Acknowledgements We thank Shiraz University Research Council. We are also
grateful to the referee for valuable comments which helped us improve the
paper significantly.

## Data Availability Statement This manuscript has no associated data or the
data will not be deposited.[Authors’ comment: Observational data we have used
in this paper are publicly available in Refs. [5, 46, 47, 48, 49, 50, 51, 52,
53, 54]. The modified version of the CLASS code is available under reasonable
request.]

## References

  * [1] Adam G. Riess et al.  Observational Evidence from Supernovae for an Accelerating Universe and a Cosmological Constant.  The Astronomical Journal, 116(3):1009–1038, September 1998.
  * [2] S. Perlmutter et al.  Measurements of $\Omega$ and $\Lambda$ from 42 High-Redshift Supernovae.  The Astrophysical Journal, 517(2):565–586, June 1999.
  * [3] P. A. R. Ade et al.  Planck 2013 results. XVI. Cosmological parameters.  Astronomy & Astrophysics, 571:A16, November 2014.
  * [4] P. A. R. Ade et al.  Planck 2015 results. XIII. Cosmological parameters.  Astronomy & Astrophysics, 594:A13, September 2016.
  * [5] N. Aghanim et al.  Planck 2018 results. VI. Cosmological parameters.  Astronomy & Astrophysics, 641:A6, September 2020.
  * [6] Max Tegmark et al.  Cosmological parameters from SDSS and WMAP.  Phys. Rev. D, 69:103501, May 2004.
  * [7] Kevork Abazajian et al.  The Second Data Release of the Sloan Digital Sky Survey.  The Astronomical Journal, 128(1):502–512, July 2004.
  * [8] Kevork Abazajian et al.  The Third Data Release of the Sloan Digital Sky Survey.  The Astronomical Journal, 129(3):1755–1759, March 2005.
  * [9] H. Lampeitl et al.  First-year Sloan Digital Sky Survey-II supernova results: consistency and constraints with other intermediate-redshift data sets.  Monthly Notices of the Royal Astronomical Society, 401(4):2331–2342, 2010.
  * [10] V. Sahni, A. Shafieloo, and A. A. Starobinsky.  Model-independent Evidence for Dark Energy Evolution from Baryon Acoustic Oscillations.  The Astrophysical Journal Letters, 793(2):L40, October 2014.
  * [11] Will J. Percival et al.  Baryon acoustic oscillations in the Sloan Digital Sky Survey Data Release 7 galaxy sample.  Monthly Notices of the Royal Astronomical Society, 401(4):2148–2168, February 2010.
  * [12] S. W. Allen, R. W. Schmidt, A. C. Fabian, and H. Ebeling.  Cosmological constraints from the local X-ray luminosity function of the most X-ray-luminous galaxy clusters.  Monthly Notices of the Royal Astronomical Society, 342(1):287, 2003\.
  * [13] Adam G. Riess et al.  A 2.4% DETERMINATION OF THE LOCAL VALUE OF THE HUBBLE CONSTANT.  The Astrophysical Journal, 826(1):56, 2016.
  * [14] Adam G. Riess et al.  New parallaxes of galactic cepheids from spatially scanning the Hubble space telescope: Implications for the Hubble constant.  The Astrophysical Journal, 855(2):136, 2018.
  * [15] Adam G. Riess, Stefano Casertano, Wenlong Yuan, Lucas M. Macri, and Dan Scolnic.  Large magellanic cloud cepheid standards provide a 1% foundation for the determination of the Hubble constant and stronger evidence for physics beyond $\Lambda$CDM.  The Astrophysical Journal, 876(1):85, 2019.
  * [16] Adam G. Riess et al.  Cosmic distances calibrated to 1% precision with Gaia EDR3 parallaxes and Hubble space telescope photometry of 75 milky way cepheids confirm tension with $\Lambda$CDM.  The Astrophysical Journal, 908(1):L6, 2021.
  * [17] S. Tsujikawa.  Modified Gravity Models of Dark Energy, pages 99–145.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2010.
  * [18] Shin’ichi Nojiri and Sergei D. Odintsov.  Unified cosmic history in modified gravity: From F(R) theory to Lorentz non-invariant models.  Physics Reports, 505(2):59–144, 2011.
  * [19] Timothy Clifton, Pedro G. Ferreira, Antonio Padilla, and Constantinos Skordis.  Modified gravity and cosmology.  Physics Reports, 513(1):1–189, 2012.  Modified Gravity and Cosmology.
  * [20] S. Nojiri, S. D. Odintsov, and V. K. Oikonomou.  Modified gravity theories on a nutshell: Inflation, bounce and late-time evolution.  Physics Reports, 692:1–104, 2017.
  * [21] Gianluca Calcagni.  Quantum field theory, gravity and cosmology in a fractal universe.  Journal of High Energy Physics, 03:120, 2010.
  * [22] Gianluca Calcagni.  Fractal universe and quantum gravity.  Phys. Rev. Lett., 104:251301, Jun 2010.
  * [23] B. B. Mandelbrot.  The fractal geometry of nature.  W. H. Freeman and Comp., New York, 3 edition, 1983.
  * [24] A.D. Linde.  Eternally existing self-reproducing chaotic inflanationary universe.  Physics Letters B, 175(4):395–400, 1986.
  * [25] M. Rassem and E. Ahmed.  On Fractal Cosmology.  Astrophysical Letters and Communications, 35:311, January 1996.
  * [26] A. K. Mittal and T. R. Seshadri.  Fractals and the large-scale structure in the universe.  Resonance, 7:39, Apr 2002.
  * [27] A. K. Mittal and D. Lohiya.  Fractal dust model of the universe based on Mandelbrot’s conditional cosmological principle and general theory of relativity.  arXiv:astro-ph/0206300, 2002.
  * [28] Gianluca Calcagni.  Geometry and field theory in multi-fractional spacetime.  Journal of High Energy Physics, 2012:65, Jan 2012.
  * [29] Ahmad Sheykhi, Zeinab Teimoori, and Bin Wang.  Thermodynamics of fractal universe.  Physics Letters B, 718(4):1203–1207, Jan 2013.
  * [30] K. Karami, Mubasher Jamil, S. Ghaffari, and K. Fahimi.  Holographic, new agegraphic, and ghost dark energy models in fractal cosmology.  Canadian Journal of Physics, 91(10):770–776, Oct 2013.
  * [31] Surajit Chattopadhyay, Antonio Pasqua, and Sudipto Roy.  A study on some special forms of Holographic Ricci Dark Energy in fractal universe.  ISRN High Energy Physics, 2013:251498, Oct 2013.
  * [32] Gianluca Calcagni.  Multi-scale gravity and cosmology.  Journal of Cosmology and Astroparticle Physics, 2013(12):041–041, Dec 2013.
  * [33] Sayani Maity and Ujjal Debnath.  Co-existence of modified Chaplygin gas and other dark energies in the framework of fractal universe.  International Journal of Theoretical Physics, 55:2668, May 2016\.
  * [34] Gianluca Calcagni.  Multiscale spacetimes from first principles.  Phys. Rev. D, 95:064057, Mar 2017.
  * [35] Gianluca Calcagni, Sachiko Kuroyanagi, and Shinji Tsujikawa.  Cosmic microwave background and inflation in multi-fractional spacetimes.  Journal of Cosmology and Astroparticle Physics, 2016(08):039–039, Aug 2016.
  * [36] Dipanjana Das, Sourav Dutta, Abdulla Al Mamon, and Subenoy Chakraborty.  Does fractal universe describe a complete cosmic scenario ?  The European Physical Journal C, 78:849, Oct 2018.
  * [37] Ehsan Sadri, Martiros Khurshudyan, and Surajit Chattopadhyay.  An interacting new holographic dark energy in the framework of fractal cosmology.  Astrophysics and Space Science, 363:230, Oct 2018.
  * [38] Ujjal Debnath and Kazuharu Bamba.  Parametrizations of dark energy models in the background of general non-canonical scalar field in D-dimensional fractal universe.  The European Physical Journal C, 79:722, Aug 2019.
  * [39] S. Ghaffari, E. Sadri, and A. H. Ziaie.  Tsallis holographic dark energy in fractal universe.  Modern Physics Letters A, 35(14):2050107, Mar 2020.
  * [40] Gianluca Calcagni and Antonio De Felice.  Dark energy in multifractional spacetimes.  Phys. Rev. D, 102:103529, Nov 2020.
  * [41] Gianluca Calcagni.  Multifractional theories: An updated review.  Modern Physics Letters A, 36(14):2140006, 2021.
  * [42] Chung-Pei Ma and Edmund Bertschinger.  Cosmological perturbation theory in the synchronous and conformal Newtonian gauges.  The Astrophysical Journal, 455:7, 1995.
  * [43] Diego Blas, Julien Lesgourgues, and Thomas Tram.  The Cosmic Linear Anisotropy Solving System (CLASS). Part II: Approximation schemes.  Journal of Cosmology and Astroparticle Physics, 2011(07):034, 2011\.
  * [44] Benjamin Audren, Julien Lesgourgues, Karim Benabed, and Simon Prunet.  Conservative Constraints on Early Cosmology: an illustration of the Monte Python cosmological parameter inference code.  JCAP, 1302:001, 2013.
  * [45] Thejs Brinckmann and Julien Lesgourgues.  MontePython 3: boosted MCMC sampler and other features.  arXiv:1804.07261, 2018.
  * [46] P. A. R. Ade et al.  Planck 2015 results. XXIV. Cosmology from Sunyaev-Zeldovich cluster counts.  Astron. Astrophys., 594:A24, 2016.
  * [47] P. A. R. Ade et al.  Planck 2013 results. XX. Cosmology from Sunyaev-Zeldovich cluster counts.  Astron. Astrophys., 571:A20, 2014.
  * [48] Martin Kilbinger et al.  CFHTLenS: Combined probe cosmological model comparison using 2D weak gravitational lensing.  Monthly Notices of the Royal Astronomical Society, 430(3):2200, 2013\.
  * [49] Catherine Heymans et al.  CFHTLenS tomographic weak lensing cosmological parameter constraints: Mitigating the impact of intrinsic galaxy alignments.  Monthly Notices of the Royal Astronomical Society, 432(3):2433, 2013\.
  * [50] D. M. Scolnic et al.  The complete light-curve sample of spectroscopically confirmed SNe Ia from pan-STARRS1 and cosmological constraints from the combined pantheon sample.  The Astrophysical Journal, 859(2):101, 2018.
  * [51] Florian Beutler et al.  The 6DF Galaxy Survey: Baryon Acoustic Oscillations and the Local Hubble Constant.  Monthly Notices of the Royal Astronomical Society, 416(4):3017–3032, 2011.
  * [52] Lauren Anderson et al.  The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Baryon Acoustic Oscillations in the Data Releases 10 and 11 Galaxy Samples.  Monthly Notices of the Royal Astronomical Society, 441(1):24–62, 2014.
  * [53] Shadab Alam et al.  The clustering of galaxies in the completed SDSS-III baryon oscillation spectroscopic survey: cosmological analysis of the DR12 galaxy sample.  Monthly Notices of the Royal Astronomical Society, 470(3):2617–2652, 2017.
  * [54] Manuel A. Buen-Abad, Martin Schmaltz, Julien Lesgourgues, and Thejs Brinckmann.  Interacting Dark Sector and Precision Cosmology.  JCAP, 1801(01):008, 2018.
  * [55] Hirotugu Akaike.  A new look at the statistical model identification.  IEEE Transactions on Automatic Control, 19(6):716–723, 1974.
  * [56] K.P. Burnham and D.R. Anderson.  Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach.  Springer Verlag, 2002.