* Szegedy et al. [2017] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _Thirty-first AAAI Conference on Artificial Intelligence_ , 2017.
  * Tao et al. [2018] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia.  Scale-recurrent network for deep image deblurring.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8174–8182, 2018.
  * Tsai et al. [2021] Fu-Jen Tsai, Yan-Tsung Peng, Yen-Yu Lin, Chung-Chi Tsai, and Chia-Wen Lin.  Banet: Blur-aware attention networks for dynamic scene deblurring.  _arXiv preprint arXiv:2101.07518_ , 2021.
  * Tu [2007] Zhuowen Tu.  Learning generative models via discriminative approaches.  In _2007 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1–8. IEEE, 2007.
  * Ulyanov et al. [2016] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  _arXiv preprint arXiv:1607.08022_ , 2016.
  * Ulyanov et al. [2018] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Deep image prior.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 9446–9454, 2018.
  * Vincent et al. [2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In _Proceedings of the 25th International Conference on Machine Learning_ , pages 1096–1103, 2008.
  * Wang et al. [2017] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang.  Residual attention network for image classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3156–3164, 2017.
  * Wang et al. [2018] Guotai Wang, Wenqi Li, Maria A Zuluaga, Rosalind Pratt, Premal A Patel, Michael Aertsen, Tom Doel, Anna L David, Jan Deprest, Sébastien Ourselin, et al.  Interactive medical image segmentation using deep learning with image-specific fine tuning.  _IEEE Transactions on Medical Imaging_ , 37(7):1562–1573, 2018.
  * Wang and Bovik [2002] Zhou Wang and Alan C Bovik.  A universal image quality index.  _IEEE Signal Processing Letters_ , 9(3):81–84, 2002.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, 2004.
  * Woo et al. [2018] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 3–19, 2018.
  * Wu et al. [2021] Yanqiu Wu, Chaoqun Hong, Xuebai Zhang, and Yifan He.  Stack-based scale-recurrent network for face image deblurring.  _Neural Processing Letters_ , pages 1–18, 2021.
  * Xingjian et al. [2015] SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  In _Advances in Neural Information Processing Systems_ , pages 802–810, 2015.
  * Xu and Jia [2010] Li Xu and Jiaya Jia.  Two-phase kernel estimation for robust motion deblurring.  In _European Conference on Computer Vision_ , pages 157–170. Springer, 2010.
  * Xu et al. [2013] Li Xu, Shicheng Zheng, and Jiaya Jia.  Unnatural l0 sparse representation for natural image deblurring.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1107–1114, 2013.
  * Xu et al. [2014] Li Xu, Jimmy S Ren, Ce Liu, and Jiaya Jia.  Deep convolutional neural network for image deconvolution.  _Advances in Neural Information Processing Systems_ , 27:1790–1798, 2014.
  * Xu et al. [2017] Xiangyu Xu, Jinshan Pan, Yu-Jin Zhang, and Ming-Hsuan Yang.  Motion blur kernel estimation via deep learning.  _IEEE Transactions on Image Processing_ , 27(1):194–205, 2017.
  * Xu et al. [2021] Yong Xu, Ye Zhu, Yuhui Quan, and Hui Ji.  Attentive deep network for blind motion deblurring on dynamic scenes.  _Computer Vision and Image Understanding_ , 205:103169, 2021.
  * Yan and Shao [2016] Ruomei Yan and Ling Shao.  Blind image blur estimation via deep learning.  _IEEE Transactions on Image Processing_ , 25(4):1910–1921, 2016.
  * Yan et al. [2017] Yanyang Yan, Wenqi Ren, Yuanfang Guo, Rui Wang, and Xiaochun Cao.  Image deblurring via extreme channels prior.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4003–4011, 2017.
  * Yeh et al. [2016] Raymond Yeh, Chen Chen, Teck Yian Lim, Mark Hasegawa-Johnson, and Minh N Do.  Semantic image inpainting with perceptual and contextual losses.  _arXiv preprint arXiv:1607.07539_ , 2(3), 2016.
  * Yu and Koltun [2015] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  _arXiv preprint arXiv:1511.07122_ , 2015.
  * Zagoruyko and Komodakis [2015] Sergey Zagoruyko and Nikos Komodakis.  Learning to compare image patches via convolutional neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4353–4361, 2015.
  * Zhang et al. [2019a] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In _International Conference on Machine Learning_ , pages 7354–7363. PMLR, 2019a.
  * Zhang et al. [2019b] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz.  Deep stacked hierarchical multi-patch network for image deblurring.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5978–5986, 2019b.
  * Zhang et al. [2018] Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau, and Ming-Hsuan Yang.  Dynamic scene deblurring using spatially variant recurrent neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2521–2529, 2018.
  * Zhang et al. [2017] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang.  Learning deep cnn denoiser prior for image restoration.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3929–3938, 2017.
  * Zhang et al. [2020] Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjorn Stenger, Wei Liu, and Hongdong Li.  Deblurring by realistic blurring.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2737–2746, 2020.
  * Zhang et al. [2022] Kaihao Zhang, Wenqi Ren, Wenhan Luo, Wei-Sheng Lai, Bjorn Stenger, Ming-Hsuan Yang, and Hongdong Li.  Deep image deblurring: A survey.  _arXiv preprint arXiv:2201.10700_ , 2022.
  * Zhang et al. [2011] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang.  Fsim: A feature similarity index for image quality assessment.  _IEEE Transactions on Image Processing_ , 20(8):2378–2386, 2011.
  * Zhang et al. [2019c] Wenlong Zhang, Yihao Liu, Chao Dong, and Yu Qiao.  Ranksrgan: Generative adversarial networks with ranker for image super-resolution.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3096–3105, 2019c.
  * Zhao et al. [2016] Hang Zhao, Orazio Gallo, Iuri Frosio, and Jan Kautz.  Loss functions for image restoration with neural networks.  _IEEE Transactions on Computational Imaging_ , 3(1):47–57, 2016.
  * Zhao et al. [2021] Hongtian Zhao, Di Wu, Hang Su, Shibao Zheng, and Jie Chen.  Gradient-based conditional generative adversarial network for non-uniform blind deblurring via denseresnet.  _Journal of Visual Communication and Image Representation_ , 74:102921, 2021.
  * Zhou et al. [2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2921–2929, 2016.
  * Zhu et al. [2019] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9308–9316, 2019.


# Discovering Dynamic Functional Brain Networks via Spatial and Channel-wise
Attention

Yiheng Liu
School of Physics & Information Technology
Shaanxi Normal University
Shaanxi Normal University, Xi’an, China 710119
<EMAIL_ADDRESS>
Enjie Ge
School of Physics & Information Technology
Shaanxi Normal University
<EMAIL_ADDRESS>
Mengshen He
School of Physics & Information Technology
Shaanxi Normal University
<EMAIL_ADDRESS>
Zhengliang Liu
University of Georgia
<EMAIL_ADDRESS>
Shijie Zhao
Northwestern Polytechnical University
<EMAIL_ADDRESS>
Xintao Hu
Northwestern Polytechnical University
<EMAIL_ADDRESS>
Dajiang Zhu
University of Texas at Arlington
<EMAIL_ADDRESS>
Tianming Liu
University of Georgia
<EMAIL_ADDRESS>Bao Ge
School of Physics & Information Technology
Shaanxi Normal University
<EMAIL_ADDRESS>

###### Abstract

Using deep learning models to recognize functional brain networks (FBNs) in
functional magnetic resonance imaging (fMRI) has been attracting increasing
interest recently. However, most existing work focuses on detecting static
FBNs from entire fMRI signals, such as correlation-based functional
connectivity. Sliding-window is a widely used strategy to capture the dynamics
of FBNs, but it is still limited in representing intrinsic functional
interactive dynamics at each time step. And the number of FBNs usually need to
be set manually. More over, due to the complexity of dynamic interactions in
brain, traditional linear and shallow models are insufficient in identifying
complex and spatially overlapped FBNs across each time step. In this paper, we
propose a novel Spatial and Channel-wise Attention Autoencoder (SCAAE) for
discovering FBNs dynamically. The core idea of SCAAE is to apply attention
mechanism to FBNs construction. Specifically, we designed two attention
modules: 1) spatial-wise attention (SA) module to discover FBNs in the spatial
domain and 2) a channel-wise attention (CA) module to weigh the channels for
selecting the FBNs automatically. We evaluated our approach on ADHD200 dataset
and our results indicate that the proposed SCAAE method can effectively
recover the dynamic changes of the FBNs at each fMRI time step, without using
sliding windows. More importantly, our proposed hybrid attention modules (SA
and CA) do not enforce assumptions of linearity and independence as previous
methods, and thus provide a novel approach to better understanding dynamic
functional brain networks.

##  1 Introduction

Brain activity is measured by capturing signals of neural activities.
Functional magnetic resonance imaging (fMRI) [39] is a non-invasive imaging
method with no side effects that measures the blood-oxygen level dependence
(BOLD). It is a common approach used to indirectly observe neural activities
of the brain over time [28]. FMRI is a very useful source of information for
understanding the brain’s intrinsic functional network structure and human
brain functions [49, 45]. Functional brain networks (FBNs) can be defined as
collections of regions displaying functional connectivities [3, 45, 41].
Because these FBNs will change over time [2, 42, 29, 36], analyzing the
patterns of functional networks over time is critical to understand how the
brain works.

Most studies extract FBNs from entire fMRI signals, assuming that FBNs remain
constant throughout the fMRI scanning process. However, the FBNs are in fact
dynamic, fading in and out over time, because the brain is naturally dynamic,
characterized by continuous discharge activities at the neuron level. The
existing methods mainly adopt a sliding window scheme to detect time varying
FBNs [42, 19, 45, 55, 9, 34, 36]. They usually carefully set a window size and
a sliding step in order to detect time varying patterns, but FBNs exhibit
transient states of brain activity[10], which makes it difficult for these
methods to identify the transient FBNs with sliding windows. Furthermore, FBNs
can’t be selected automatically in previous methods, and the number of FBNs
needs to be set manually in a heuristic or experiential manner. Another
limitation is the assumption of linearity or independence in most of existing
methods [17, 4, 40]. For example, independent component analysis (ICA) [4, 43,
7] regards the identification of functional brain networks as a problem of
blind source separation, that is, it is assumed that the observed signals are
composed of the linear superposition of independent blind source signals.
Spatial ICA [8] assumes that each blind source component is spatially
independent. The generalized Linear model (GLM) [17] assumes that the observed
signal is a linear superposition of the task design signal, and considers each
voxel as an independent variable. Recently, some deep learning methods can
learn the better nonlinear features of fMRI signals, but they still recover
FBNs with linear LASSO regression [14, 27, 57, 15, 52, 38, 16, 47, 56, 48].
Overall, these methods oversimplify the relationship between BOLD signal
changes and the heterogeneity of neuronal activity within voxels, and cannot
reflect the complex functional interactions among brain regions.

To overcome these problems, we propose a novel method to discover dynamic
FBNs, which is named spatial and channel-wise attention autoencoder (SCAAE),
inspired by the visual attention mechanisms[25, 53] in the field of image
recognition. The method is based on the fact that fMRI signal in a neuron or
brain region is not only induced by its internal neural activities, but also
shaped by the interactions with other neurons or brain regions [26].
Therefore, it is necessary to increase the receptive field of the model. We
use 3D convolution neural networks with large kernel size to encode and
reconstruct the fMRI signals, spatial-wise attention (SA) to directly detect
the FBNs in the whole brain without linear and independent assumptions, and
channel-wise attention (CA) to weigh the important channels to select FBNs
automatically. Because neural activities lead to differences in fMRI signals
at each time step, the attention mechanism will focus on regions of neural
activities.

In short, our contributions are summarized as follows:

1\. The proposed SCAAE method can discover dynamic transient FBNs at each time
step without using sliding windows, thus providing a new perspective for
functional network analysis.

2\. SCAAE can directly recover spatially interactive FBNs via a spatial-wise
attention (SA) module, thus being free from linear and independent
assumptions, and can select FBNs automatically via a channel-wise attention
(CA) module, thus avoiding manual interventions.

3\. The decoder and partial encoder components are not needed in forward
inference. SCAAE only needs the output of the attention modules. This process
of constructing the FBNs is much faster than previous work because it
dispenses with learning the coefficient matrix iteratively.

##  2 Related Works

###  2.1 Functional Brain Network

At present, the commonly used methods to construct functional networks based
on fMRI data include: general linear model (GLM) [17], independent component
analysis (ICA) [4, 43, 8, 7] and sparse dictionary learning (SDL) [40, 31, 20,
21]. Because these methods require less data and are more stable than deep
learning methods, they are widely used in clinical practices. Among them, GLM
is used to model task-based fMRI (tfMRI), ICA is used to model resting state
fMRI (rsfMRI), and SDL can model both tfMRI and rsfMRI. To discover dynamic
FBNs, some works introduced sliding windows [42, 19, 2, 55, 9, 34, 36].
However, limited by the shallow nature of these traditional machine learning
methods and the sliding windows, they cannot extract fMRI intrinsic features
and are unable to discover FBNs at each time step. In addition, These methods
have their own assumptions about the fMRI data. For example, GLM relies on
task design, ICA assumes an independent relationship between the different
components, and SDL relies on sparsity assumption.

###  2.2 Autoencoder

Deep learning has powerful representation capabilities, therefore it is often
used to extract high-level features from low-level raw data. The output of
each layer in deep learning model can be viewed as a new representation of the
raw data and it has been proven in multiple tasks that deep learning methods
are superior at learning high-level and mid-level features from low-level raw
data. [6, 35]. Due to the lack of labeled data, many methods based on deep
learning use autoencoders (AE) to extract high-level features. Some examples
include Deep Belief Nets (DBN) [14], Convolutional Autoencoder (CAE) [27, 57,
15, 56], Recurrent Autoencoder (RAE) [52, 38], Transformer Attention
Autoencoder [16] and Variational Autoencoder (VAE) [47, 48]. Although deep
models can extract better feature representations than traditional methods,
constructing FBNs with lasso regression does not stay away from the
constraints of linearity assumption.

Previous work using autoencoders to construct FBNs is based on correlation.
Prior methods use the encoders to extract the global features $Z$ of the raw
fMRI signals $X$:

$Z_{T\times K}=encoder(X_{T\times M})$ (1) $min||X-ZW_{K\times
M}||^{2}_{2}+\lambda||W||_{1}$ (2)

Where $T$ refers to the time series length of the fMRI signals, $K$ refers to
the dimension of the extracted features, $M$ refers to the number of voxels of
the raw fMRI signals. The extracted features $Z$ are treated as source
components of the raw fMRI signals. $Z$ is used to predict the raw fMRI
signals. $W$ refers to the coefficient matrix of the lasso regression. The
matrix has $K$ rows, and the coefficients of each row represent the linear
correlation of the source components $Z$ and the corresponding fMRI voxels.
For voxels that are uncorrelated with the components, their coefficients are
set to 0 by L1 regularization. Finally, each row of the coefficient matrix is
inversely transformed to the 3D brain image space. This is done with masking
in data preprocessing [1]. Thus, FBNs has a neuroanatomically meaningful
context.

###  2.3 Attention Mechanisms

Attention Mechanisms have been applied to multiple tasks such as machine
translation [22, 51], image classification [25, 53], object detection [13],
semantic segmentation [18], medical image processing [44] and other tasks with
impressive performance.

The voxels of fMRI images can capture the intensity of neural activities.
Since attention mechanisms have great ability to select features, capturing
brain functional activities with attention mechanisms is very suitable. We
adopt a spatial-wise attention (SA) module to capture the FBNs in the whole
brain and a channel-wise attention (CA) module to weigh the importance of the
FBNs for selecting FBNs automatically.

##  3 Methods

The goal of our method is to construct FBNs via attention mechanism. The whole
framework is shown in Figure 1. We use the 3D convolutional neural network to
encode and decode the raw fMRI signals. In our experiments, the network
structure of down-sampling encoding and up-sampling decoding to reconstruct
the raw fMRI signals is not critical, but inserting the attention modules into
the specific layers can significantly affect the construction of FBNs. We
adopt a spatial-wise attention module and a channel-wise attention module in
the encoder part for discovering the functional brain networks. In section
3.1, we will explain why the FBNs can be discovered by the attention
mechanism. In section 3.2, we will illustrate the details of constructing FBNs
via attention mechanism. Our experimental results and details will be shown in
the section 4.

Figure 1: The key contribution of SCAAE is to construct FBNs via attention
mechanism. A Channel-wise Attention (CA) module is used to weight the
importance of FBNs for selecting FBNs automatically and a Spatial-wise
attention (SA) module is used to capture the neural activities regions in the
whole brain.

###  3.1 Attention

The spatial attention is promising since it models the hidden structure of
latent signals sources. We attempt to explain why attention mechanisms can
achieve great performance in constructing FBNs from the perspectives of
cognitive neuroscience and statistical correlations.

From a cognitive neuroscience perspective, neural activities can be directly
measured by BOLD signals. When neurons are excited, the BOLD signals will
rise, otherwise the BOLD signals will remain at around the baseline level.
Neural activities lead to differences in fMRI signals at each time step. To
achieve a good feature representation, the encoder must take into account
neural activity, which is the key factor that affects fMRI signal changes.
Thus, the attention mechanism will focus on the regions of neural activities.

From a statistical correlation perspective, activated voxels derived by
correlation-based functional connectivity methods are concentrated in the
regions near themselves as shown in Figure 2. The activated regions show
strong local correlations. Because locality is an inherent inductive bias for
convolution neural networks, using convolution neural networks to model the
fMRI signals is a natural choice.

Figure 2: The FBN templates constructed by ICA [49]. The activated voxels are
concentrated in the regions nearby themselves. The strong spatial correlation
enables the attention mechanism to capture the features of neural activities.

####  3.1.1 Spatial-wise Attention

The spatial-wise attention (SA) can be represented as:

$SA=sigmoid(conv3D(GELU(conv3D(X_{n}))))$ (3) $X_{n+1}=X_{n}\circ SA$ (4)

where $X_{n}$ refers to the output features of the number of convolutional
layers, $conv3D$ refers to the 3D convolutional layers, $GELU$ [24] and
$sigmoid$ refer to the activation functions. The $sigmoid$ function can map
the features to [0, 1]. We can obtain the weighted features map through
spatial-wise attention. However, fMRI signals may be from a mixture of two or
more functional networks [37, 2]. We use two branch networks to discover
different FBNs. Specifically, we insert the SA into each branch and then
concatenate the output together:

$X_{n+1}=concatenate(SA_{1}(X_{n}),SA_{2}(X_{n}))\circ X_{n}$ (5)

####  3.1.2 Channel-wise Attention

We weigh the importance of each channel by channel-wise attention (CA) which
can be represented as:

$CA=sigmoid(\sigma(\sigma(GAP(X_{n}){W_{1}}){W_{2}}){W_{3}})$ (6)

where $W$ refers to the fully connected layer, $\sigma$ refers to the
activation function GELU and $GAP$ refers to global average pooling.

###  3.2 FBN

Each output of the two SA modules is consist of 16 channels of 3D feature
maps. Because the sigmoid function maps larger outputs to 1 and smaller
outputs to 0, the activated weight map is included in the feature map before
it is activated by the sigmoid function. The feature map is defined as
$S^{C\times D\times H\times W}$, where $C$ refers to the number of channels,
$D$ refers to the depth of the feature map, $H$ refers to the height of the
feature map, and $W$ refers to the width of the feature map. Then the
$CA^{C\times 1}$ is used to weigh each channel:

$S_{weighted}=CA\circ S$ (7) $FBN=\sum_{n=1}^{C}S_{weighted}^{n}$ (8)

where $S_{weighted}$ refers to the weighted channels by $CA$, $n$ refers to
the number of channels. In order to display the FBNs more clearly, we
normalize it with Min-Max normalization and square the result to make the
activated regions more visible.

$FBN=(\frac{FBN-min(FBN)}{max(FBN)-min(FBN)})^{2}$ (9)

##  4 Experiments

###  4.1 Dataset and Preprocessing dataset

Attention Deficit Hyperactivity Disorder (ADHD) is a mental health disorder
that affects at least 5-10% of school-age children. The ADHD200111ADHD200
dataset http://fcon_1000.projects.nitrc.org/indi/adhd200/ released 904
resting-state fMRI (rsfMRI) and anatomical datasets aggregated across 8
independent imaging sites. The datasets of these 8 sites are released by
Kennedy Krieger Institute (KKI), Peking University (PU), New York University
Medical Center (NYU), Brown University (BU), Oregon Health & Science
University (OHSU), University of Pittsburgh (UP), Washington University (WU),
and NeuroImage (NI) respectively. Each site has different numbers of subjects,
as shown in Table 1. The entire ADHD200 dataset includes 554 typically
developing individuals and 350 ADHD patients between the ages of 7 and 12
years. We used preprocessed data publicly available from Preprocessed
Connectivity Project [5] for all experiments. The Athena preprocessing
pipeline, which is based on AFNI and FSL, was adopted to process the dataset
[30, 50, 54, 11, 12]. It includes skull stripping, slice time correction,
motion correction, detrending, band filtering, normalization and masking. All
subjects are registered to the MNI152 $4\times 4\times 4$ $mm^{3}$ standard
template space.

Table 1: Summary of ADHD200 dataset Imaging Site | Total Subject | Control Subject | ADHD Subject
---|---|---|---
KKI | 83 | 61 | 22
PU | 259 | 146 | 113
NYU | 222 | 99 | 123
NI | 48 | 23 | 25
BU | 51 | 27 | 24
OHSU | 79 | 42 | 37
UP | 101 | 95 | 6
WU | 61 | 61 | 0
Total Sites | 904 | 554 | 350

In accordance with HIPAA guidelines and 1000 Functional Connectomes Project
protocols, all datasets are anonymous, without any protected health
information.

###  4.2 Implementation Details

The experiments were performed on a workstation with 2 NVIDIA GeForce GTX 1080
Ti GPUs. We preprocess the 3D fMRI images with a mask and resample the images
to $49\times 58\times 47$. Before inputting the data to the attention modules,
we do not perform any downsample operations and only use 3D convolutional
layers for feature extraction. In our experiments, construction of FBNs can be
affected by the number of 3D convolutional layer used prior to the attention
mechanism. Specifically, the fewer the convolutional layers are, the closer
the FBN construction result is to the linear model. We think this is caused by
too many nonlinear transformations associated with the convolutional layers.
Therefore, to make the results comparable to the results of other models, we
use two convolutional layers, but increase the width of the convolutional
layers. Each convolutional layer consists of three convolutional kernels with
kernel size 5, 3 and 1 respectively. We also add batch normalization layers
(BN) and residual connections after the convolutional kernels.

When training SCAAE, we use the Adam [33] optimizer with an initial learning
rate of 0.001, and we decrease the learning rate with the step decay strategy.
The step size is set to 2, and the decay coefficient gamma is set to 0.9. We
use mean squared error (MSE) as the loss function. Here $y$ refers to the raw
fMRI signals, $\hat{y}$ refers to the reconstructed fMRI signals.

$MSE=\frac{1}{N}\sum_{n=1}^{N}(y-\hat{y})^{2}$ (10)

The batch size is set as 12, and epochs is set as 20. We found that the FBNs
constructed by the model saved at each epoch were nearly identical. The
activation function in each layer is $GELU$.

###  4.3 FBNs from SCAAE

In our experiments, we observe that the brain is constantly active even in the
resting state. The FBNs derived from SCAAE are shown in Figure 3. Since
previous work used linear statistical models, the constructed FBNs usually
contain a lot of noise and require thresholding with methods such as ICA, SDL
and GLM. We use the attention modules to output the FBNs directly. Here, we
show the weight map of the attention modules directly without using any
thresholding. It is observed that the resting state networks (RSNs) are active
even when there are no extra ongoing tasks, which provides evidence supporting
the conclusion in [23, 32, 46, 45]. By visual inspection, these FBNs can be
well interpreted, and they agree with domain knowledge of functional network
atlases in the literature. Also, we find out that the attention mechanism
captures the regions of neural activities very well.

Figure 3: Overview of the FBNs derived from SCAAE. These are the outputs we
obtain from the attention modules, and we can observe that the attention
mechanism finds the activation regions perfectly.

We can also observe the transferring process of FBNs in different states. We
select two representative transfer processes from individual data and show
them in Figure 4.

Figure 4: The method can observe changes in FBNs at each time step. These two
figures show the state transition process of FBNs and can observe a gradual
change process. This is what previous methods cannot provide.

We measure the similarity of two spatial maps to evaluate the performance of
our method. The spatial similarity is defined by the intersection over union
rate (IoU) between two FBNs $N^{(1)}$ and $N^{(2)}$, where $n$ refers to the
volume size:

$IoU(N^{(1)},N^{(2)})=\frac{\sum_{i=1}^{n}|N_{i}^{(1)}\cap
N_{i}^{(2)}|}{\sum_{i=1}^{n}|N_{i}^{(1)}\cup N_{i}^{(2)}|}$ (11)

In addition to subjectively observing the gradual change process of FBNs, we
also made a quantitative evaluation of the change of FBNs. We calculated the
IoU of the FBN of each individual at each time step versus the previous time
step, and averaged all the IoUs to evaluate the change at each time step. The
gradual change process of FBNs has been evaluated on 4 subsets (from 4 sites)
of the ADHD200 dataset, as shown in Table 2. The higher the average IoU is,
the more it can indicate that the change of the functional networks is
gradual. Our results indicates that the change of FBNs is indeed gradual.

Table 2: The average IoU of the FBN of each individual at each time step versus the previous time step. Site | KKI | PU | NYU | NI
---|---|---|---|---
Average IoU | 0.4071 | 0.4955 | 0.5024 | 0.5103

Because our method is the first work in the field that can derive the FBNs for
each time step, we use the standard templates derived by ICA [49] on a large
amount of data as the benchmark. We select the FBNs matching the standard
templates from all time steps of an individual and calculate the spatial
similarity between them. All time-steps of the individual were then group-
analyzed with ICA and SDL to calculate the spatial similarity between them and
the standard templates. The similarity metrics $IoU(N_{SCAAE},N_{template})$,
$IoU(N_{ICA},N_{template})$ and $IoU(N_{SDL},N_{template})$ are used to
evaluate the performance of SCAAE.

The results of the evaluation are shown in Figure 5. The results indicate that
SCAAE can identify FBNs very well, suggesting the effectiveness and
meaningfulness of our proposed model, even though this comparison is actually
negatively unfair to our method, because our method can only see a single fMRI
image at a time, while SDL and ICA are able to do group analysis of data from
all time steps of a single individual. Nevertheless, our method still achieves
excellent performance.

Figure 5: Comparison with SDL and ICA. The IoU refers to the spatial
similarity of the corresponding methods and templates. The templates are based
on ICA [49]. Figure 6: Overview of the functional network state transfer.

###  4.4 Functional Network State Transfer

The proposed model SCAAE provides a new perspective to observe the dynamic
changes of brain function. We visualize the state transfer of the functional
networks. Due to the complexity of human brain, there is no easy way to
determine the state of the functional networks. To solve this problem, we use
IoU to evaluate the similarity between the functional networks and the
templates at each time step, and classify the functional network into the
class of its most similar template. The functional networks at each time step
can be mapped to one of these ten templates [49]. We filter out the functional
networks with IoU less than 0.3. The state transfer diagram is shown in Figure
6. Thicker lines indicate more frequent transitions between the two states.
The number in the figure corresponds to the state of the template. We can
observe that the state of template 0 occurs the most and is most closely
connected to other states. Template 8 has not appeared yet, which means that
it is probably just a theoretical template estimated from a large amount of
data, and does not actually exist in individual brain’s activities.

##  5 Discussion and Conclusion

In this paper, we proposed a novel method named SCAAE to discover dynamic
functional brain networks via visual attention. This work is the earliest
study in the field to explore dynamic FBNs with attention mechanism, to the
best of our knowledge. Compared with the statistical machine learning models
used in previous studies, our method is free from the time-invariant, linear
and independent constraints. In addition, since the proposed model only needs
to perform forward inference once in the stage of constructing FBNs, our model
can construct FBNs faster than the previous methods. The greatest contribution
is that we provide a dynamic perspective to observe the changing patterns of
FBNs, which is a novel method to understand the human brain.

One limitation of our work is that we have only performed experiments on
resting-state fMRI (rsfMRI) but not on task-based fMRI (tfMRI) yet. Although
the model can be directly applied to task-based fMRI data, we believe that it
is more suitable to develop a new network structure based on the attention
mechanism that adapts to the task design requirements of task-based fMRI. In
addition, our work is based on autoencoders, and therefore we can also apply
the encoder representation as biomarkers to brain disorder characterization,
such as ADHD, Alzheimer’s disease, etc. Finally, we believe studying the
transition of functional networks states is an important direction for future
dynamic FBNs research, which provides a new perspective for understanding the
human brain.

## References

  * [1] Abraham, A., Pedregosa, F., Eickenberg, M., Gervais, P., Mueller, A., Kossaifi, J., Gramfort, A., Thirion, B., Varoquaux, G.: Machine learning for neuroimaging with scikit-learn. Frontiers in neuroinformatics 8, 14 (2014)
  * [2] Allen, E.A., Damaraju, E., Plis, S.M., Erhardt, E.B., Eichele, T., Calhoun, V.D.: Tracking whole-brain connectivity dynamics in the resting state. Cerebral cortex 24(3), 663–676 (2014)
  * [3] Archbold, K.H., Borghesani, P.R., Mahurin, R.K., Kapur, V.K., Landis, C.A.: Neural activation patterns during working memory tasks and osa disease severity: preliminary findings. Journal of Clinical Sleep Medicine 5(1), 21–27 (2009)
  * [4] Beckmann, C.F., Deluca, M., Devlin, J.T., Smith, S.M.: Investigations into resting-state connectivity using independent component analysis. Philosophical Transactions of The Royal Society B Biological Sciences 360(1457), 1001–1013 (2005)
  * [5] Bellec, P., Chu, C., Chouinard-Decorte, F., Benhajali, Y., Margulies, D.S., Craddock, R.C.: The neuro bureau adhd-200 preprocessed repository. Neuroimage 144, 275–286 (2017)
  * [6] Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35(8), 1798–1828 (2013)
  * [7] Calhoun, V.D., Adali, T.: Multisubject independent component analysis of fmri: A decade of intrinsic networks, default mode, and neurodiagnostic discovery. IEEE Reviews in Biomedical Engineering 5, 60–73 (2012)
  * [8] Calhoun, V.D., Adali, T., Pearlson, G.D., Pekar, J.J.: A method for making group inferences from functional mri data using independent component analysis. Human Brain Mapping 16(2) (2002)
  * [9] Calhoun, V.D., Adali, T.: Time-varying brain connectivity in fmri data: whole-brain data-driven approaches for capturing and characterizing dynamic states. IEEE Signal Processing Magazine 33(3), 52–66 (2016)
  * [10] Calhoun, V.D., Miller, R., Pearlson, G., Adalı, T.: The chronnectome: time-varying connectivity networks as the next frontier in fmri data discovery. Neuron 84(2), 262–274 (2014)
  * [11] Cox, R.W.: Afni: software for analysis and visualization of functional magnetic resonance neuroimages. Computers and Biomedical research 29(3), 162–173 (1996)
  * [12] Cox, R.W., Hyde, J.S.: Software tools for analysis and visualization of fmri data. NMR in Biomedicine: An International Journal Devoted to the Development and Application of Magnetic Resonance In Vivo 10(4-5), 171–178 (1997)
  * [13] Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y.: Deformable convolutional networks. In: Proceedings of the IEEE international conference on computer vision. pp. 764–773 (2017)
  * [14] Dong, Q., Ge, F., Ning, Q., Zhao, Y., Lv, J., Huang, H., Yuan, J., Jiang, X., Shen, D., Liu, T.: Modeling hierarchical brain networks via volumetric sparse deep belief network. IEEE transactions on biomedical engineering 67(6), 1739–1748 (2019)
  * [15] Dong, Q., Qiang, N., Lv, J., Li, X., Liu, T., Li, Q.: Discovering functional brain networks with 3d residual autoencoder (resae). In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 498–507. Springer (2020)
  * [16] Dong, Q., Qiang, N., Lv, J., Li, X., Liu, T., Li, Q.: Spatiotemporal attention autoencoder (staae) for adhd classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 508–517. Springer (2020)
  * [17] Friston, K.S.: Statistical parametric maps in functional imaging : A general linear approach. Hum Brain Map 2 (1995)
  * [18] Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., Lu, H.: Dual attention network for scene segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3146–3154 (2019)
  * [19] Ge, B., Wang, H., Wang, P., Tian, Y., Zhang, X., Liu, T.: Discovering and characterizing dynamic functional brain networks in task fmri. Brain Imaging and Behavior 14(5), 1660–1673 (2020)
  * [20] Ge, F., Lv, J., Hu, X., Ge, B., Guo, L., Han, J., Liu, T.: Deriving adhd biomarkers with sparse coding based network analysis. In: 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI). pp. 22–25. IEEE (2015)
  * [21] Ge, F., Lv, J., Hu, X., Guo, L., Han, J., Zhao, S., Liu, T.: Exploring intrinsic networks and their interactions using group wise temporal sparse coding. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). pp. 74–77. IEEE (2018)
  * [22] Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: Convolutional sequence to sequence learning. In: International Conference on Machine Learning. pp. 1243–1252. PMLR (2017)
  * [23] Harris, K.D., Mrsic-Flogel, T.D.: Cortical connectivity and sensory coding. Nature 503(7474), 51–58 (2013)
  * [24] Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)
  * [25] Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7132–7141 (2018)
  * [26] Huang, H., Hu, X., Han, J., Lv, J., Liu, N., Guo, L., Liu, T.: Latent source mining in fmri data via deep neural network. In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI). pp. 638–641. IEEE (2016)
  * [27] Huang, H., Hu, X., Zhao, Y., Makkie, M., Dong, Q., Zhao, S., Guo, L., Liu, T.: Modeling task fmri data via deep convolutional autoencoder. IEEE transactions on medical imaging 37(7), 1551–1561 (2017)
  * [28] Huettel, S.A., Song, A.W., McCarthy, G., et al.: Functional magnetic resonance imaging, vol. 1. Sinauer Associates Sunderland, MA (2004)
  * [29] Hutchison, R.M., Womelsdorf, T., Allen, E.A., Bandettini, P.A., Calhoun, V.D., Corbetta, M., Della Penna, S., Duyn, J.H., Glover, G.H., Gonzalez-Castillo, J., et al.: Dynamic functional connectivity: promise, issues, and interpretations. Neuroimage 80, 360–378 (2013)
  * [30] Jenkinson, M., Beckmann, C.F., Behrens, T.E., Woolrich, M.W., Smith, S.M.: Fsl. Neuroimage 62(2), 782–790 (2012)
  * [31] Jiang, X., Li, X., Lv, J., Zhang, T., Zhang, S., Guo, L., Liu, T.: Sparse representation of hcp grayordinate data reveals novel functional architecture of cerebral cortex. Human Brain Mapping 36(12), 5301–5319 (2016)
  * [32] Kanwisher, N.: Functional specificity in the human brain: a window into the functional architecture of the mind. Proceedings of the National Academy of Sciences 107(25), 11163–11170 (2010)
  * [33] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [34] Kiviniemi, V., Vire, T., Remes, J., Elseoud, A.A., Starck, T., Tervonen, O., Nikkinen, J.: A sliding time-window ica reveals spatial variability of the default mode network in time. Brain connectivity 1(4), 339–347 (2011)
  * [35] LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436–444 (2015)
  * [36] Leonardi, N., Van De Ville, D.: On spurious and real fluctuations of dynamic functional connectivity during rest. Neuroimage 104, 430–436 (2015)
  * [37] Li, M., Liu, Y., Chen, F., Hu, D.: Including signal intensity increases the performance of blind source separation on brain imaging data. IEEE transactions on medical imaging 34(2), 551–563 (2014)
  * [38] Li, Q., Dong, Q., Ge, F., Qiang, N., Wu, X., Liu, T.: Simultaneous spatial-temporal decomposition for connectome-scale brain networks by deep sparse recurrent auto-encoder. Brain Imaging and Behavior 15(5), 2646–2660 (2021)
  * [39] Logothetis, N.K.: What we can do and what we cannot do with fmri. Nature 453(7197), 869–878 (2008)
  * [40] Lv, J., Jiang, X., Li, X., Zhu, D., Chen, H., Zhang, T., Zhang, S., Hu, X., Han, J., Huang, H., et al.: Sparse representation of whole-brain fmri signals for identification of functional networks. Medical Image Analysis 20(1), 112–134 (2015)
  * [41] Lv, J., Lin, B., Li, Q., Zhang, W., Zhao, Y., Jiang, X., Guo, L., Han, J., Hu, X., Guo, C., et al.: Task fmri data analysis based on supervised stochastic coordinate coding. Medical image analysis 38, 1–16 (2017)
  * [42] Ma, S., Calhoun, V.D., Phlypo, R., Adalı, T.: Dynamic changes of spatial functional network connectivity in healthy individuals and schizophrenia patients using independent vector analysis. NeuroImage 90, 196–206 (2014)
  * [43] Mckeown, M.J.: Detection of consistently task-related activations in fmri data with hybrid independent component analysis. Neuroimage 11(1), 24–35 (2000)
  * [44] Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., Kainz, B., et al.: Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 (2018)
  * [45] Pessoa, L.: Understanding brain networks and brain organization. Physics of Life Reviews 11(3), 400–435 (2014)
  * [46] Pessoa, L., Lindquist, K.A., Wager, T.D., Kober, H., Bliss-Moreau, E., Barrett, L.F.: Beyond brain regions: Network perspective of cognition–emotion interactions. Behavioral and Brain Sciences 35(3), 158 (2012)
  * [47] Qiang, N., Dong, Q., Ge, F., Liang, H., Ge, B., Zhang, S., Sun, Y., Gao, J., Liu, T.: Deep variational autoencoder for mapping functional brain networks. IEEE Transactions on Cognitive and Developmental Systems 13(4), 841–852 (2020)
  * [48] Qiang, N., Dong, Q., Sun, Y., Ge, B., Liu, T.: Deep variational autoencoder for modeling functional brain networks and adhd identification. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). pp. 554–557. IEEE (2020)
  * [49] Smith, S.M., Fox, P.T., Miller, K.L., Glahn, D.C., Fox, P.M., Mackay, C.E., Filippini, N., Watkins, K.E., Toro, R., Laird, A.R.: Correspondence of the brain’s functional architecture during activation and rest. Proceedings of the National Academy of Sciences 106(31), 13040–13045 (2009)
  * [50] Smith, S.M., Jenkinson, M., Woolrich, M.W., Beckmann, C.F., Behrens, T.E., Johansen-Berg, H., Bannister, P.R., De Luca, M., Drobnjak, I., Flitney, D.E., et al.: Advances in functional and structural mr image analysis and implementation as fsl. Neuroimage 23, S208–S219 (2004)
  * [51] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [52] Wang, H., Zhao, S., Dong, Q., Cui, Y., Chen, Y., Han, J., Xie, L., Liu, T.: Recognizing brain states using deep sparse recurrent neural network. IEEE transactions on medical imaging 38(4), 1058–1068 (2018)
  * [53] Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)
  * [54] Woolrich, M.W., Jbabdi, S., Patenaude, B., Chappell, M., Makni, S., Behrens, T., Beckmann, C., Jenkinson, M., Smith, S.M.: Bayesian analysis of neuroimaging data in fsl. Neuroimage 45(1), S173–S186 (2009)
  * [55] Yuan, J., Li, X., Zhang, J., Luo, L., Dong, Q., Lv, J., Zhao, Y., Jiang, X., Zhang, S., Zhang, W., et al.: Spatio-temporal modeling of connectome-scale brain network interactions via time-evolving graphs. Neuroimage 180, 350–369 (2018)
  * [56] Zhao, Y., Dai, H., Zhang, W., Ge, F., Liu, T.: Two-stage spatial temporal deep learning framework for functional brain network modeling. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). pp. 1576–1580. IEEE (2019)
  * [57] Zhao, Y., Li, X., Zhang, W., Zhao, S., Makkie, M., Zhang, M., Li, Q., Liu, T.: Modeling 4d fmri data via spatio-temporal convolutional neural networks (st-cnn). In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 181–189. Springer (2018)

##  6 Appendix


# Blockchain-aided Secure Semantic Communication for AI-Generated Content in
Metaverse

Yijing Lin, Hongyang Du, Dusit Niyato, ,
Jiangtian Nie, Jiayi Zhang, Yanyu Cheng, and Zhaohui Yang Corresponding
author: Hongyang Du.Yijing Lin is with the State Key Laboratory of Networking
and Switching Technology, Beijing University of Posts and Telecommunications,
100876, Beijing, China (e-mail: yjlin@bupt.edu.cn).Hongyang Du, Dusit Niyato,
Jiangtian Nie, and Yanyu Cheng are with Nanyang Technological University,
639798, Singapore (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>yanyu.cheng@ntu.edu.sg).Jiayi Zhang is with School of
Electronic and Information Engineering, Beijing Jiaotong University, Beijing
100044, China (jiayizhangg@bjtu.edu.cn).Zhaohui Yang is with the College of
Information Science and Electronic Engineering, Zhejiang University, China
(e-mail: yang zhaohui@zju.edu.cn).

###### Abstract

The construction of virtual transportation networks requires massive data to
be transmitted from edge devices to Virtual Service Providers (VSP) to
facilitate circulations between the physical and virtual domains in Metaverse.
Leveraging semantic communication for reducing information redundancy, VSPs
can receive semantic data from edge devices to provide varied services through
advanced techniques, e.g., AI-Generated Content (AIGC), for users to explore
digital worlds. But the use of semantic communication raises a security issue
because attackers could send malicious semantic data with similar semantic
information but different desired content to break Metaverse services and
cause wrong output of AIGC. Therefore, in this paper, we first propose a
blockchain-aided semantic communication framework for AIGC services in virtual
transportation networks to facilitate interactions of the physical and virtual
domains among VSPs and edge devices. We illustrate a training-based targeted
semantic attack scheme to generate adversarial semantic data by various loss
functions. We also design a semantic defense scheme that uses the blockchain
and zero-knowledge proofs to tell the difference between the semantic
similarities of adversarial and authentic semantic data and to check the
authenticity of semantic data transformations. Simulation results show that
the proposed defense method can reduce the semantic similarity of the
adversarial semantic data and the authentic ones by up to 30% compared with
the attack scheme.

###### Index Terms:

Metaverse, Blockchain, Semantic Communication, Semantic Attacks, Semantic
Defenses

##  I INTRODUCTION

The word Metaverse was coined in the science-fiction novel Snow Crash [1] to
describe a virtual reality society in which people utilize digital avatars to
symbolize themselves and experience the world. In recent years, Metaverse has
received attention from academia and industries as a novel Internet
application due to the advancement of Augmented Reality (AR), Virtual Reality
(VR), Artificial Intelligence (AI), and Blockchain. Specifically, extending
reality (AR/VR) and AI technologies can provide users with immersive
experiences and facilitate continuous data synchronization from the physical
domain to the virtual domain, which is supported by data perceived by edge
devices and services driven in Metaverse. Blockchain can help the physical and
virtual domains to share information and construct economic systems in a
decentralized manner. Thus, the Metaverse can be viewed asthe integration of
multiple technologies supported by massive data interactions between the
physical and virtual domains.

One of the significant advantages of Metaverse is that people can conduct
experiments in the virtual world that cannot be conducted in the real world.
Because the virtual world can be created based on real-world data, e.g.,
images, sensing data, and text, the experimental result in Metaverse can be
used to guide the real world. One example is the virtual transportation
networks [2], i.e., virtual environments in which users can safely train
automatic driving algorithms and test vehicles. To build such virtual
environments, virtual service providers (VSPs) can leverage network edge
devices to capture images from the real world and then render the virtual
objects. However, this process entails three challenges as follows:

  1. D1)

How to use the data collected from the real world, e.g., images, to achieve
fast virtual world building?

  2. D2)

How to improve the efficiency of data transmission to facilitate interactions
of the physical and virtual domains?

  3. D3)

How to ensure the security of the data received by VSP to ensure that the
virtual world can be accurately synchronized with the real world?

Since virtual transportation networks require extensive interactions between
the physical domain and Metaverse, edge devices can be utilized to capture
images of geographical landmarks. However, converting these captured images
into a consistent style for the virtual world is complicated. In several
virtual service designs, VSPs need to hire digital painters to pre-process
images [3], which is time-consuming and costly. The boom in AI has brought
alternative solutions. AI-generated content (AIGC) technology [4] allows VSP
to process quickly images collected from the real world using well trained AI
models (For D1). However, the collected data could still burden the network.
For example, the authors in [5] state that a pair of sensing devices can
generate 3.072 megabytes of data per second, which challenges conventional
communication systems. Fortunately, semantic communication [6] is introduced
to filter out irrelevant information from edge devices to reduce information
redundancy of VSPs by extracting semantic data from raw data and expressing
desired meanings (For D2). With the help of semantic communications, massive
amounts of data can be circulated in the virtual transportation networks to
empower Metaverse services.

However, the introduction of semantic communication brings about higher
requirements of the data security. Efficient semantic data sharing should be
achieved between unknown VSPs and edge devices deployed in untrusted
environments. However, the extracted semantic data can be tampered to almost
the same descriptors (semantic similarities) but different desired meanings.
The attacker (edge device) can modify the pixels of a sunflower to make it
similar to the extracted semantic data (snowy mountain) in terms of semantic
similarities but visually dissimilar [7], which affects the output of the AIGC
models and is difficult for VSPs to detect the difference between the
adversarial and authentic semantic data. Moreover, it is hard to detect and
prevent semantic data mutations for virtual transportation networks in
Metaverse. Since VSPs and edge devices are distributed, it is difficult to
record, trace, and verify data transformations. To solve the aforementioned
problems, the blockchain and Zero-Knowledge Proof techniques can be used (For
D3). Thus, we present a blockchain-aided semantic communication framework with
AIGC services to achieve virtual transportation networks in Metaverse. Here,
blockchain-aided semantic communication can facilitate data circulation and
economic activities between VSPs and edge devices in a decentralized manner
[8]. Targeted semantic attacks [7] are utilized to generate adversarial
semantic data to improve their semantic similarities almost up to that of the
extracted semantic data by training various loss functions. Zero-Knowledge
Proof (ZKP) [9] is integrated into the proposed framework to process semantic
data and securely guarantee correct transformations. The contributions of this
paper are summarized as follows:

  * •

We propose a blockchain-aided semantic communication framework for AIGC in
virtual transportation networks that ensures the authenticity of semantic data
transmitted from edge devices to VSPs to facilitate interactions of the
physical and virtual domains.

  * •

We illustrate how a training-based targeted semantic attack scheme generates
adversarial semantic data (images) without revealing the authentic semantic
data. The attack semantic data has almost the same semantic similarities as
the authentic ones but is visually dissimilar to them.

  * •

We, for the first time, design a blockchain and zero-knowledge proof-based
semantic defense scheme to assure the authentication of semantic data. The
scheme can utilize zero-knowledge proof to record the transformations of
semantic data, and use blockchain to track and verify semantic data mutations.

The remainder of the paper is described as follows. Section II reviews
previous works on secure semantic communication. Section III demonstrates a
blockchain-aided semantic communication framework for AIGC in Metaverse.
Section IV illustrates a training-based targeted semantic attack scheme.
Section V designs a blockchain and zero-knowledge proof-based semantic defense
scheme. The proposed mechanisms are evaluated in Section VI. Section VII
concludes the paper and elaborates the future work.

##  II Related Work

In this paper, we consider the integration of blockchain-aided semantic
communications for Metaverse, which involves multiple emerging technologies.
Therefore, we divide the related work into three parts: Blockchain-aided
Semantic Communications, Semantic Attacks, and Semantic Defenses.

###  II-A Blockchain-aided Semantic Communications

Semantic communication [6] can lighten virtual transportation network burdens
by transmitting relevant semantic data to VSPs after processing original data
by AI technologies in edge devices. Z. Weng et al. [10] utilized deep learning
(DL) to identify the essential speech information with higher weights for
semantic communication in dynamic channel conditions. To enable edge devices
to perform DL-based semantic communication tasks, H. Xie et al. [11] proposed
a lite-distributed semantic communication framework for edge devices to
transmit low-complexity texts by optimizing training processes.

Although semantic communication can help Metaverse reduce information
redundancy, it can not handle the challenge that how to construct trust among
unknown edge devices and VSPs to facilitate data sharing and economic
activities. Blockchain [12] is a peer-to-peer network that can construct
decentralized ledgers for participants to share data. The integration of
blockchain and AI-based semantic communication can empower Metaverse
ecosystems to carry out rich activities between the physical and virtual
domains [13]. Y. Lin et al. [8] proposed a unified blockchain-semantic
framework to enable Web 3.0 services to implement on-chain and off-chain
interactions. A proof of semantic mechanism is proposed to verify semantic
data before adding it to blockchain. However, they do not mention the
performance indicators of semantic data. Y. Lin et al. [14] proposed a
blockchain-based semantic exchange framework that can mint Non-Fungible Tokens
(NFT) for semantic data, utilize the game theory to facilitate exchange, and
introduce ZKP to enable privacy-preserving. However, they do not consider
semantic attacks that may reduce exchange efficiency.

###  II-B Semantic Attacks and Defenses

The introduction of semantic communication brings about security issues for
Metaverse. However, current research on the security of semantic communication
is still in its infancy. Q. Hu et al. [15] analyzed semantic noise that causes
semantic data to express misleading meanings. To reduce the effects caused by
semantic noise, they added weight perturbation to adversarial training
processes, suppressed noise-related and task-unrelated features, and designed
semantic similarity-based loss functions to reduce transmitting overheads. X.
Luo et al. [16] focused on privacy leakages when sharing background knowledge.
They introduced symmetric encryption to adversarial training processes to
encrypt and decrypt semantic data to ensure confidentiality. H. Du et al. [7]
focused on the semantic Internet-of Things (SIoT) and proposed new performance
indicators for SIoT to quantify security issues, including semantic secrecy
outage probability and detection failure probability. They focused on image
transmission-oriented semantic communication and divided semantic attacks into
targeted and untargeted semantic attacks. The targeted attack can generate
adversarial semantic data (images) that can be recovered to a given target
requested by receivers. The adversarial semantic data has almost the same
descriptors (semantic similarities), but is visually dissimilar [17]. The
untargeted semantic attack can generate adversarial semantic data that
minimizes semantic similarities. They do not pursue to be recovered to any
target by receivers.

In this paper, we study the targeted semantic attacks and introduce ZKP to
differ in semantic similarities between the adversarial and target images to
protect semantic data. ZKP has been widely used in blockchain to enable
privacy-preserving and authenticity. R. Song et al. [9] utilized NFT and ZKP
to construct a traceable data exchange scheme in blockchain, which can protect
data privacy and exchange fairness. Z. Wang et al. [18] designed a ZKP-based
off-chain data feed scheme to take in off-chain sensitive data to execute the
business logic of smart contract-based applications. H. Galal et al. [19]
leveraged ZKP and smart contracts to hide details of NFTs and swap NFTs in a
fair manner. Y. Fang et al. [20] utilized ZKP to verify the authenticity of
model prediction processes without leasing private parameters of deep learning
models. However, the above methods do not consider how to use ZKP to prevent
targeted semantic attacks to detect adversarial semantic data.

###  II-C Artificial Intelligence Generated Content

AIGC refers to the use of AI algorithms, natural language processing (NLP),
and computer version (CV) methods to produce a variety of media forms,
including text, images, and audio. In terms of text content generation, an
epoch-making AIGC application is the ChatGPT, a conversational language model
developed by OpenAI [21]. ChatGPT is a type of the Generative Pre-trained
Transformer (GPT) model that is trained on a huge amount of conversational
data. ChatGPT is able to generate text that sounds as if it were written by a
human. This makes it helpful for a variety of purposes, including chatbots,
virtual assistants, and language translation. For image generation, diffusion
model [22], a new class of state-of-the-art generative models, have
demonstrated exceptional performance in Image Generation tasks and have
surpassed the performance of generative adversarial networks (GANs) [23] in
numerous tasks. Stable Diffusion, a AIGC model that is released by Stability
AI, is an open-source text-to-image generator that creates amazing artwork in
a matter of seconds. It operates with unprecedented speed and quality on
consumer-grade GPUs. Furthermore, AIGC techniques continues to advance in the
field of audio generation. The authors in [24] propose a deep learning-based
method that employs contrastive representation learning and clustering to
automatically derive thematic information from music pieces in the training
data. The simulation results show that the proposed model is capable of
generating polyphonic pop piano music with repetition and plausible
variations.

One of the primary benefits of AIGC is that it can be produced at a number and
speed that would be difficult for human beings to do alone. It also permits a
great degree of consistency and precision. Therefore, the development of AIGC
technologies can provide a strong boost to the evolution of the Internet.
Notably, despite the potential benefits, there are also some worries about the
ramifications of AIGC, including the possibility for prejudice, the semantic
errors in the generated content, and the blurring of the boundary between
human- and machine-generated content. Therefore, it is significant to ensure
the correctness of the AIGC, avoiding attacks from malicious parties that
causes resource waste to affect the quality of AIGC services.

##  III Blockchain-aided Semantic Communication Framework for AIGC in
Metaverse

The implementation of virtual transportation networks requires the following
main steps: 1) Semantic extraction and transmission for data interactions
between physical and virtual domains, 2) Semantic transformation and
verification, and 3) AIGC for Metaverse, as shown in Fig. 1.

Figure 1: Blockchain-aided Semantic Communication Framework for AIGC in
Metaverse

###  III-A Semantic Extraction and Transmission

Pedestrians are in danger when auto-driving models in vehicles are not trained
well enough, which motivates the development of virtual transportation
networks in the Metaverse. Therefore, it is necessary to digitize physical
domains using data produced in the real world to simulate environments for
training vehicles and drivers. VSPs can take pictures using edge devices like
smartphones, cameras, and sensors in physical domains to obtain information
about the weather, traffic, and geographical landmarks that can be used to
train and test the detection systems of vehicles in virtual domains. Virtual
domain output can be fed back into physical domains to configure vehicles for
better and safer performance. Based on the simulated environment,
inexperienced drivers and vehicles can practice their reactions under
unfamiliar weather or traffic conditions in Metaverse in a safe way.
Therefore, VSPs need to frequently interact with edge devices supported by a
tremendous amount of data to construct virtual transportation networks in
Metaverse to provide services, which challenges the transmission capabilities
of edge devices and VSPs.

Unfortunately, conventional communication systems cannot afford such frequent
interactions between the physical and virtual domains, which will cause
virtual transportation networks in Metaverse to lack enough data to simulate
virtual environments. Semantic communication, a completely new paradigm that
extracts semantic meaning from raw data for transmission, can be utilized to
resolve this challenge. Instead of transmitting original data, edge devices
can extract the semantic data and transmit to VSPs. The VSPs can then use the
received semantic data to generate simulation environments for drivers and
autonomous vehicle training on the virtual road. For instance, edge devices,
e.g., smartphones, positioned at various locations along the same road may
gather photographs of traffic conditions from their perspective. To reduce
information redundancy, they can utilize semantic segmentation modules to crop
key components of images as semantic data [2]. Then edge devices only transmit
semantic data to VSPs to report traffic conditions on roads.

However, since VSPs have to collect semantic data from multiple edge devices
to train virtual transportation networks in different situations, malicious
edge devices could falsify semantic data (images) and corrupt the training
process, which is dangerous for pedestrians and drivers. In this paper, we
study the targeted semantic attack [7, 17] in that the adversarial and
authentic semantic data have almost the same semantic similarities but are
totally irrelevant. The semantic similarities are calculated with the help of
high-dimensional descriptors extracted by a convolutional neural network
(CNN). However, malicious edge devices could train a corresponding neural
network to modify some pixels in attack images to achieve similar descriptors
as the authentic images to corrupt the construction of virtual transportation
networks. The training-based targeted semantic attack scheme is illustrated in
Section IV.

###  III-B AIGC in Metaverse

After receiving semantic data (images) from edge devices deployed in different
locations, VSPs can perceive conditions or views of landmarks, and render
images by AIGC services in Metaverse. For example, semantic data of landmarks
from different perspectives can be utilized by VSPs to render 3D scenes to
provide users with seamless experiences. VSPs can also utilize views of
landmarks to generate artworks or avatars in Metaverse. Therefore, AIGC
services play an important role in virtual transportation networks to
facilitate the use of data resources and the applications of Metaverse.
Besides, semantic data is important for subsequent AIGC services since its
quality may affect the content generated by AIGC.

However, since semantic data circulated in the Metaverse may be corrupted by
the aforementioned targeted semantic attacks produced by malicious edge
devices, the AIGC services may do useless work and provide users with hateful
content. For example, VSPs want to collect images of famous landmarks (i.e.,
Eiffel Tower) while malicious edge devices may extract unrelated images of
flowers to corrupt the subsequent AIGC services that renders 3D scenes with
different perspectives of the landmark. VSPs also want to perceive images of
landmarks to generate artworks or avatars while attackers transmit irrelative
semantic data of animals to obstacle AIGC services. Since the adversarial
semantic data (images) modified by attackers have almost the same semantic
similarities, it is difficult and time-consuming for VSPs to verify the
authenticity of images. Therefore, it is necessary to design a mechanism to
ensure the security of data transmission to protect the security of AIGC
services.

###  III-C Semantic Transformation and Verification

Malicious semantic data can affect the security of virtual transportation
services in Metaverse. Modifying pixels in unrelated semantic data increases
the semantic similarity score, causing the VSPs to use the wrong semantic data
as input to AIGC and corrupt the virtual environment. Inspired by [17], image
transformations can be performed to distinguish semantic similarities between
the adversarial and authentic images. However, malicious edge devices can
continue adjusting pixels in irrelative images to make them similar to
transformed semantic data. Moreover, it is impossible for edge devices to
transform semantic data unlimited times. A possible and practical solution is
to record and verify transformations performed on semantic data.

Therefore, we propose a blockchain and zero-knowledge proof-based semantic
defense scheme in Section V. The logic of transformations is recorded on the
circuit produced by the zero-knowledge algorithm. Edge devices utilize
extracted semantic data as inputs to generate proof of transformations and
output transformed semantic data. They send the proof and semantic data to
VSPs for verification. Since VSPs and edge devices are distributed in unknown
Metaverse environments represented by avatars, the blockchain should be used
to record and verify transformations to prevent data mutations.

##  IV Training-based Targeted Semantic Attack Scheme

Although the blockchain-aided semantic communication framework for AIGC in
Metaverse can reduce information redundancy and establish decentralized trust
between unknown edge devices and VSPs, malicious edge devices may conduct
training-based targeted semantic attacks to corrupt semantic data (images)
circulated in the Metaverse services. The targeted semantic attacks refer to
transmitting adversarial semantic data with almost the same semantic
descriptors but visually dissimilar to the authentic one [7], which is
difficult for VSPs to utilize semantic similarities (inner product of
descriptors among images) evaluation to distinguish them. In this section, we
illustrate the workflow of the training-based targeted semantic attack
targeted semantic attacks [7][17].

Descriptor Extraction. Since extracted semantic data (images) is difficult to
evaluate, edge devices can utilize a CNN to map images to high dimensional
descriptors. Then VSPs can use the descriptors to distinguish semantic
similarities of images. The process of descriptor extraction is illustrated as
follows.

Let us denote three types of semantic data produced by malicious edge devices,
including the adversarial semantic data $\mathbf{x_{a}}$, the authentic one
$\mathbf{x_{t}}$, and the carrier one $\mathbf{x_{c}}$. The authentic semantic
data $\mathbf{x_{t}}$ is extracted from original images by semantic extraction
in edge devices according to requirements and associated with label
$y_{t}=f_{\mathbf{c}}(x_{t})$, which has semantic similarities with
$\mathbf{x_{a}}$. $f_{\mathbf{c}}(\cdot)$ is utilized to classify semantic
data. The carrier semantic data $\mathbf{x_{c}}$ is an auxiliary image to help
malicious edge devices generate $\mathbf{x_{a}}$, which is classified as
$y_{c}=f_{\mathbf{c}}(x_{c})\neq y_{t}$ and has visual similarities with
$\mathbf{x_{a}}$. The adversarial semantic data $\mathbf{x_{a}}$ is produced
by training networks to learn how to modify pixels, which can achieve that
$\mathbf{x_{a}}$ has almost the same descriptors but is visually dissimilar
from the authentic one $\mathbf{x_{t}}$ generated by semantic extraction, as
shown in Fig. 2. Besides, $\mathbf{x_{a}}$ is visually similar to
$\mathbf{x_{c}}$ but classified incorrectly as $y_{t}$. Therefore, the
descriptor extraction process is vital for malicious edge devices to produce
adversarial images $\mathbf{x_{a}}$.

Figure 2: Relationship among Images

The input image $\mathbf{x}$ should be re-sampled to $\mathbf{x}^{s}$ with the
same dimension $s$ to make $\mathbf{x_{t}}$ and $\mathbf{x_{c}}$ with the same
resolution. The image $\mathbf{x}^{s}$ is utilized as an input to train a
Fully CNN given by $\mathbf{g_{x^{s}}}=g(\mathbf{x}^{s}):\mathbb{R}^{W\times
H\times 3}\rightarrow\mathbb{R}^{w\times h\times d}$ to implement feature
extraction where $W\times H\times 3$ and $w\times h\times d$ are weight,
height, and channel of images. A pooling layer is utilized to map the input
tensor $\mathbf{g_{x^{s}}}$ and the descriptor
$\mathbf{h_{x^{s}}}=h(\mathbf{g_{x^{s}}})$ by the CNN with network parameters
$\mathbf{\theta}$, which is mapped by $h:\mathbb{R}^{w\times h\times
d}\rightarrow\mathbb{R}^{d}$. The descriptor is the output of the pooling
layer, which can be utilized to compare with other descriptors to calculate
semantic similarities. The $l_{2}$ normalization is utilized to easily compare
semantic similarities.

Loss Function. Considering the goal of malicious edge devices is to make
$\mathbf{x_{a}}$ almost the same as $\mathbf{x_{t}}$ in terms of semantics but
visually similar to $\mathbf{x_{c}}$, the loss function consists of the
performance loss $l_{\mbox{ts}}(\mathbf{x},\mathbf{x}_{t})$ and the distortion
loss between $\mathbf{x}$ and $\mathbf{x}_{c}$, which can be defined as

$L_{\mbox{ts}}(\mathbf{x}_{c},\mathbf{x}_{t};\mathbf{x})=l_{\mbox{ts}}(\mathbf{x},\mathbf{x}_{t})+\lambda\|\mathbf{x}-\mathbf{x}_{c}\|^{2},$
(1)

where $\lambda$ is a hyper-parameter to show the impact of the distortion
loss.

Performance Loss. Let us assume that malicious edge devices can access to the
network structure of the descriptor extraction [17] since it is necessary for
edge devices to know the evaluation standard of VSPs. Considering different
scenarios where targeted semantic attacks generate, referring to [17], we
introduce the three empirical forms of the performance loss as follows.

Global Descriptor Loss is suitable that malicious edge devices even know all
parameters of the network of the descriptor extraction. Thus, the performance
loss function $l_{\mbox{ts}}$ can be given by calculating the inner product of
$\mathbf{x_{a}}$ and $\mathbf{x_{t}}$ as follows:

$l_{\mbox{global}}(\mathbf{x},\mathbf{x}_{t})=1-\mathbf{h}^{\top}_{\mathbf{x}}\mathbf{h}_{\mathbf{x}_{t}}.$
(2)

Activation Tensor Loss is adapted to the scenario where the outputs of the
network of the descriptor extraction are the same for $\mathbf{x_{a}}$ and
$\mathbf{x_{t}}$ before down-sampling to resolution $s$, which can be denoted
by the mean squared difference of $\mathbf{g}_{\mathbf{x}}$ and
$\mathbf{g}_{\mathbf{x}_{t}}$ as follows:

$l_{\mbox{tensor}}(\mathbf{x},\mathbf{x}_{t})=\frac{\|\mathbf{g}_{\mathbf{x}}-\mathbf{g}_{\mathbf{x}_{t}}\|^{2}}{w\cdot
h\cdot d}.$ (3)

Activation Histogram Loss is utilized to preserve first-order statistics of
activations $\|u(\mathbf{g_{x}},\mathbf{b})_{i}$ per channel $i$ to achieve
identical extracted descriptors regardless of spatial information in images,
which can be denoted as follows:

$l_{\mbox{hist}}(\mathbf{x},\mathbf{x}_{t})=\frac{1}{d}\sum_{i=1}^{d}\|u(\mathbf{g_{x}},\mathbf{b})_{i}-u(\mathbf{g_{x_{t}}},\mathbf{b})_{i}\|,$
(4)

where $\mathbf{b}$ is the histogram bin centers.

Optimization. Our goal is to find the optimal loss function that minimizes the
semantic similarities of $\mathbf{x_{t}}$ and $\mathbf{x}$, which equivalently
optimizes network parameters $\mathbf{\theta}$. We can use the Adam algorithm
to update $\mathbf{\theta}$ as

$\theta_{t+1}=\theta_{t}-\eta\frac{\rho_{t}}{\sqrt{\upsilon_{t}+\epsilon}},$
(5)

where $\eta$ is the learning rate, $\rho_{t}$ and $\upsilon_{t}$ are the
first-order and second-order momenta of gradients, and $\epsilon$ is utilized
to prevent the $\sqrt{\upsilon_{t}+\epsilon}$ from being zero. Therefore, the
adversarial semantic data $\mathbf{x}_{a}$ can be expressed by

$\mathbf{x}_{a}=\mathop{\arg\min}_{\mathbf{x}}L_{\mbox{ts}}(\mathbf{x}_{c},\mathbf{x}_{t};\mathbf{x}),$
(6)

where $L_{\mbox{ts}}$ can be replaced by $l_{\mbox{global}}$,
$l_{\mbox{tensor}}$, or $l_{\mbox{hist}}$ according to different scenarios.

Figure 3: Blockchain and Zero-Knowledge Proof-based Semantic Defense Scheme

##  V Blockchain and Zero-Knowledge Proof-based Semantic Defense Scheme

Since the adversarial semantic data generated by malicious edge devices has
almost the same descriptors (semantic similarity) but different desired
meanings from the authentic ones produced by honest edge devices, inspired by
[7][17], we utilize blockchain and zero-knowledge proof-based semantic defense
scheme to help VSPs to identify attack images transmitted in the Metaverse.
Instead of submitting extracted semantic data directly, edge devices should
transform or process semantic data by the bilinear interpolation algorithm
[25], and utilize Zero-Knowledge Proof to record and verify transformations.
The details of the proposed scheme are elaborated as follows, as shown in Fig.
3.

Transformation. The transformation of semantic data is a training-free defense
method that uses visual invariance to distinguish adversarial and authentic
semantic extraction. The reason is that attackers adjust some pixels to make
descriptors of adversarial images similar to the authentic ones [7]. As a
result, we attempt to increase visual invariance by blurring extracted images
using the spatial transformation of the bilinear interpolation algorithm.

Let us assume that $(x_{1},y_{1})$, $(x_{1},y_{2})$, $(x_{2},y_{1})$, and
$(x_{2},y_{2})$ are four points in the extracted images. Then the targeted
points $(x,y)$ can be obtained by the spatial transformation, i.e., bilinear
interpolation in the $x$ and $y$ directions. The spatial transformation can be
considered a mapping function $f(\cdot,\cdot)$. Thus, the linear interpolation
in the $x$ direction can be derived as follows:

$f(x,y_{1})\approx\frac{x_{2}-x}{x_{2}-x_{1}}f(x_{1},y_{1})+\frac{x-x_{1}}{x_{2}-x_{1}}f(x_{2},y_{1})$
(7)

and

$f(x,y_{2})\approx\frac{x_{2}-x}{x_{2}-x_{1}}f(x_{1},y_{2})+\frac{x-x_{1}}{x_{2}-x_{1}}f(x_{2},y_{2}).$
(8)

The targeted points are transformed by the linear interpolation in the $y$
direction as follows:

$\begin{split}&f(x,y)\approx\frac{y_{2}-y}{y_{2}-y_{1}}f(x,y_{1})+\frac{y-y_{1}}{y_{2}-y_{1}}f(x,y_{2})\\\
&\approx\frac{(x_{2}-x)(y_{2}-y)}{(x_{2}-x_{1})(y_{2}-y_{1})}f(x_{1},y_{1})+\frac{(x-x_{1})(y_{2}-y)}{(x_{2}-x_{1})(y_{2}-y_{1}))}f(x_{2},y_{1})\\\
&+\frac{(x_{2}-x)(y-y_{1})}{(x_{2}-x_{1})(y_{2}-y_{1})}f(x_{1},y_{2})+\frac{(x-x_{1})(y-y_{1})}{(x_{2}-x_{1})(y_{2}-y_{1})}f(x_{2},y_{2}).\end{split}$
(9)

Although edge devices can perform transformations to obtain the blurred
semantic data that can distinguish from the adversarial one, it is difficult
for VSPs to verify whether the blurred semantic data is derived from authentic
transformations. Malicious edge devices may modify some pixels to make
descriptors of their semantic data close to that of honest edge devices after
transformations. Therefore, to assure the authenticity of the blur
transformation for semantic data, we utilize ZKP to record transformations and
blockchain to verify them. The proposed mechanism is a tuple of 3 polynomial-
time schemes after extracting a circuit mapping the transformation, including
Key Generation, Proof, and Verification, which works as follows:

Extraction. The mechanism initiates the logic in a computation circuit
$\mathsf{C}$ using the simple arithmetic expression mapping the transformation
$f$ to construct the public statement $\mathsf{s}$ and the private witness
$\mathbf{w}$ [20]. The circuit implements the logic of bilinear interpolation
via circom [26], a ZKP circuit compiler. The circuit can provide a relation
between the inputs and outputs semantic data which can be used for verifiable
computation on transformations without disclosing the inputs. The process of
extraction can be illustrated as follows:

$\mathsf{Extract}(f)\rightarrow(\mathsf{s},\mathsf{w}).$ (10)

Key Generation. Edge devices take a security parameter $1^{\lambda}$ and the
transformation circuit $\mathsf{C}$ as inputs to generate a common reference
string $\mathsf{crs}$. The common reference string $\mathsf{crs}$ includes an
evaluation key $\mathsf{crs.ek}$ and a verification key $\mathsf{crs.vk}$ for
proof and verification. The process of key generation can be expressed as
follows:

$\mathsf{KeyGen}(1^{\lambda},\mathsf{C})\xrightarrow{\mathsf{C}}\mathsf{crs(ek,vk)}.$
(11)

Proof. Edge devices take the evaluation key $\mathsf{crs.ek}$, the statement
$\mathsf{s}$ and the witness $\mathsf{w}$ related to the transformed and
original semantic data, which satisfies $\mathsf{C(s,w)}=1$. The statement
$\mathsf{s}$ and the witness $\mathsf{w}$ stand for the public and private
information corresponding to the transformation relation. Thus, a zero-
knowledge proof $\pi$ is generated to reflect and verify the relation. The
process of proof generation can be denoted as follows:

$\mathsf{Prove}(\mathsf{crs.ek,s,w})\xrightarrow{\mathsf{C}}\pi.$ (12)

Verification. VSPs utilize smart contracts deployed on blockchain to verify
the authenticity of transformations and implement the business logic of
blockchain-aided semantic communication for AIGC in Metaverse. Since the
verification process is implemented in the blockchain, edge devices and VSPs
can query verification results to construct trust in a decentralized manner.
The verification key $\mathsf{crs.vk}$, the statement $\mathsf{s}$, and the
proof $\pi$ are the inputs written into smart contracts to determine whether
to accept or reject the proof according to outputs. When the status of the
output is 1, the verification succeeds and VSPs accept the semantic data
provided by edge devices; otherwise, VSPs refuse to accept the proof
corresponding to semantic data produced by malicious edge devices. The process
of proof verification can be given as follows:

$\mathsf{Verify}(\mathsf{crs.vk,s},\pi)\rightarrow\\{0,1\\}.$ (13)

The semantic defense scheme should satisfy the following properties including
completeness, soundness, and zero-knowledge [9][18][19][20].

Completeness means that an honest edge device with a valid witness
$\mathsf{w}$ can convince an honest VSP for the authenticity of transformed
semantic data generated from the circuit $\mathsf{C}$. If the transformed
semantic data extracted by bilinear interpolation is correct and edge devices
construct the proof $\pi$ correctly through the proof circuit $\mathsf{C}$ and
the evaluation key $\mathsf{crs.ek}$, then the VSPs can use the verification
key $\mathsf{crs.vk}$ generated by the proof circuit $\mathsf{C}$, the proof
$\pi$, and the public information (statement) $\mathsf{s}$ to obtain a
verification result. For each $\mathsf{C(s,w)}=1$, the proof $\pi$ generated
from an honest edge device will be accepted with probability $1$, which can be
denoted as follows:

$\Pr\left[\begin{array}[]{c}\mathsf{KeyGen}(1^{\lambda},\mathsf{C})\rightarrow\mathsf{crs(ek,vk)}\\\
\mathsf{Prove}(\mathsf{crs.ek,s,w})\rightarrow\pi\\\
\mathsf{Verify}(\mathsf{crs.vk,s},\pi)=1\end{array}\right]=1.$ (14)

Figure 4: Performance Loss of Attacks

(a) Semantic similarity between the adversarial image and the authentic (b)
Semantic similarity between the adversarial image and the carrier

Figure 5: Semantic Similarity Performance of Attacks

Soundness denotes that malicious edge devices cannot provide VSPs with
authentic transformed semantic data, which means that edge devices can provide
valid witnesses if they can produce valid proofs. If $(\mathsf{s,w})$ is not a
valid input to the proof circuit $\mathsf{C(s,w)}=0$, there is a polynomial
time extractor $\mathsf{Ext}$ for a malicious edge device $\mathcal{A}$ that
makes the probabilities of $\mathsf{Verify}(\mathsf{crs.vk,s},\pi^{*})=1$ are
negligible. The adversarial advantage of soundness
$\mathsf{Adv}_{\mathcal{A}}^{sd}(\lambda)$ can be represented as follows:

$\Pr\left[\begin{array}[]{c}\mathsf{KeyGen}(1^{\lambda},\mathsf{C})\rightarrow\mathsf{crs(ek,vk)}\\\
\mathcal{A}(\mathsf{crs.ek,s})\rightarrow\pi^{*}\\\
\mathsf{Ext(crs.ek,s,\pi^{*})}\rightarrow\mathsf{w}\\\
\mathsf{Verify}(\mathsf{crs.vk,s},\pi^{*})=1\end{array}\right]\leq\mathsf{negl}(\lambda).$
(15)

Malicious edge devices $\mathcal{A}$ can hardly falsify semantic data or make
honest VSPs $\mathcal{V}$ accept invalid proofs about transformations $f$.
According to (15), VSPs cannot accept false proof $\pi$ from $\mathcal{A}$
except for a negligible probability $\mathsf{negl}(\lambda)$.

Zero-knowledge represents that VSPs can only verify the authenticity of
transformed semantic data while they cannot obtain the original semantic data
unless edge devices send it to them. Let us assume that there are
probabilistic polynomial time simulators $\mathsf{S_{1}}$, $\mathsf{S_{2}}$,
and malicious edge devices $\mathcal{A}_{1}$, $\mathcal{A}_{2}$. The simulator
$\mathsf{S_{1}}$ can produce a common reference string that is used by the
simulator $\mathsf{S_{2}}$ to generate a simulated proof. Then the proposed
mechanism has the zero-knowledge property if the adversarial advantage of
zero-knowledge $\mathsf{Adv}_{\mathcal{A}}^{zk}(\lambda)$ satisfies:

$|\Pr(\mathsf{real})-\Pr(\mathsf{sim})|\leq\mathsf{negl}(\lambda),$ (16)

where $\Pr(\mathsf{real})$ and $\Pr(\mathsf{sim})$ can be denoted as

$\Pr(\mathsf{real})=\Pr\left[\begin{array}[]{c}\mathsf{KeyGen}(1^{\lambda},\mathsf{C})\rightarrow\mathsf{crs}\\\
\mathcal{A}_{1}(f)\rightarrow(\mathsf{s,w})\\\
\mathsf{Prove(crs,s,w)}\rightarrow\pi\\\
\mathcal{A}_{2}(\mathsf{crs,s,w,\pi})=1\end{array}\right]$ (17)

and

$\Pr(\mathsf{sim})=\Pr\left[\begin{array}[]{c}\mathsf{S_{1}}(1^{\lambda},\mathsf{C})\rightarrow\mathsf{crs}\\\
\mathcal{A}_{1}(f)\rightarrow(\mathsf{s,w})\\\
\mathsf{S_{2}(crs,s)}\rightarrow\pi\\\
\mathcal{A}_{2}(\mathsf{crs,s,w,\pi})=1\end{array}\right].$ (18)

Figure 6: Semantic Similarity Comparison Between Attack and Defense Schemes

(a) Semantic similarity between the adversarial image and the authentic (b)
Semantic similarity between the adversarial image and the carrier

Figure 7: Semantic Similarity Performance of Defenses

VSPs who know public statements mapping with semantic data can hardly learn
any knowledge about semantic data before and after transformation. According
to (16), VSPs cannot know anything about semantic data other than what can be
inferred from transformation.

##  VI Experimental Evaluations

We simulate the proposed mechanism on Ubuntu 20.04LTS, Intel Xeon, 8 core, 64G
memory, and 25000Mb/s with 2 Tesla V100, Go 1.19.4, Node v14.17.0, PyTorch
1.12.1, Torchvision 0.13.1, and CUDA 10.2. We perform experiments on a
standard image benchmark, Revisited Paris [27], to measure the attack and
defense performance among edge devices and VSPs. We exploit pre-trained
networks on ImageNet [28] and AlexNet [29] to execute the attacks and defenses
with 100 iterations. Unless otherwise mentioned, we use these parameters
referring to [17]. The learning rate $\eta$ is set to 0.01. The hyper-
parameter $\lambda$ that adjusts the impact of distortion loss is 0. The
training process performs 100 iterations for our experiments.

###  VI-A Performance of Semantic Attack Scheme

Fig. 4 is the performance loss of the proposed semantic attack scheme with
different loss functions [Global, Hist, Tensor] corresponding to the global
descriptor, activation tensor, and activation histogram as the number of
iterations increases. Tensor converges much slower than Global and Hist, which
requires more iterations to reach its plateau.

Fig. 5 is the semantic similarity attack performance between the adversarial
semantic data and the authentic or the carrier one with three loss functions
[Global, Hist, Tensor]. As shown in Fig. 5 (a) and Fig. 5 (b), Global and Hist
converge around the 40th iteration which is much faster than Tensor. The
semantic similarity performance of attacks is consistent with Fig. 4 which
shows the same trend in three loss functions.

###  VI-B Performance of Semantic Defense Scheme

Fig. 6 is the semantic similarity comparison between attack and defense
schemes by displaying example images. Fig. 7 is the semantic similarity
(descriptors) defense performance between the adversarial semantic data and
the authentic one or the carrier one with three loss functions [Global, Hist,
Tensor]. Compared with Fig. 5, the adversarial image (semantic data) can
differ in semantic similarities when applying the proposed semantic defense
scheme. Since the extracted semantic data is required to execute the defense
scheme and the execution can be assured by zero-knowledge proof, the images
can differ in descriptors. As shown in Fig. 5 (a) and Fig. 7 (a), the semantic
similarity can reduce by up to 35% between the adversarial and the authentic
images. Fig. 5 (b) and Fig. 7 (b) show that the semantic similarity can
decrease 10% between the adversarial and the carrier images.

Fig. 8 (a) is the blockchain consensus overhead for [Attack, Authentic,
Defense] types of semantic data (image) with [275 KB,467 KB,10 KB] data sizes
when the number of nodes is [5,50,5]. As shown in Fig. 8 (a), the proposed
scheme can reduce the time consumed in blockchain to improve consensus
efficiency due to the attack and defense schemes cropping authentic semantic
data while differing in semantic similarities according to Fig. 7. Fig. 8 (b)
is the ZKP computation overhead for [GenWitness, GenProof, VerifyProof]
operations with [10KB,100KB,1MB] data sizes. We can see from Fig. 8 (b) that
the ZKP computation overhead depends on the GenProof operation, while the
sizes of semantic data affect little on the proposed scheme. Besides, the
computation overhead for verifying proofs is less than for generating proofs,
which can be written into smart contracts [9][18][19].

(a) Consensus (b) ZKP Computation

Figure 8: Consensus and Computation Overheads

##  VII Conclusion and Future Work

In this paper, we first integrated blockchain-aided semantic communication
into Metaverse to support AIGC services in virtual transportation networks. We
also presented a training-based targeted semantic attack scheme to illustrate
potential attacks by generating adversarial semantic data with the same
descriptors but different desired meanings. To prevent the above attack, we
designed a blockchain and zero-knowledge proof-based semantic defense scheme
that records transformations of semantic data and verifies mutations in a
decentralized manner. Simulation results show that the proposed mechanism can
differ in the descriptors between the adversarial semantic data and the
authentic one. The defense scheme can identify malicious edge devices
falsifying semantic data to corrupt AIGC services in virtual transportation
networks. In future research work, we will study how to mitigate malicious
edge devices and facilitate resource allocation with economic incentive
mechanisms in the proposed framework.

## References

  * [1] N. Stephenson, _Snow Crash: A Novel_. Spectra, 2003.
  * [2] W. C. Ng, H. Du, W. Y. B. Lim, Z. Xiong, D. Niyato, and C. Miao, “Stochastic Resource Allocation for Semantic Communication-aided Virtual Transportation Networks in the Metaverse,” _arXiv preprint arXiv:2208.14661_ , 2022.
  * [3] V. Serkova, “The digital reality: Artistic choice,” in _IOP Conf. Series Materials Sci. Engin._ , vol. 940, no. 1, Jan. 2020, p. 012154.
  * [4] H. Du, Z. Li, D. Niyato, J. Kang, Z. Xiong, X. Shen, and I. K. Dong, “Enabling AI-generated content (AIGC) services in wireless edge networks,” _arXiv preprint arXiv:2301.03220_ , 2022.
  * [5] J. Wang, H. Du, Z. Tian, D. Niyato, J. Kang _et al._ , “Semantic-Aware Sensing Information Transmission for Metaverse: A Contest Theoretic Approach,” _arXiv preprint arXiv:2211.12783_ , 2022.
  * [6] W. Yang, H. Du, Z. Q. Liew, W. Y. B. Lim, Z. Xiong, D. Niyato, X. Chi, X. S. Shen, and C. Miao, “Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges,” _IEEE Communications Surveys & Tutorials_, 2022.
  * [7] H. Du, J. Wang, D. Niyato, J. Kang, Z. Xiong, M. Guizani, and D. I. Kim, “Rethinking Wireless Communication Security in Semantic Internet of Things,” _arXiv preprint arXiv:2210.04474_ , 2022.
  * [8] Y. Lin, Z. Gao, H. Du, D. Niyato, J. Kang, R. Deng, and X. S. Shen, “A Unified Blockchain-Semantic Framework for Wireless Edge Intelligence Enabled Web 3.0,” _arXiv preprint arXiv:2210.15130_ , 2022.
  * [9] R. Song, S. Gao, Y. Song, and B. Xiao, “ZKDET: A Traceable and Privacy-Preserving Data Exchange Scheme based on Non-Fungible Token and Zero-Knowledge,” in _2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS)_. IEEE, 2022, pp. 224–234.
  * [10] Z. Weng and Z. Qin, “Semantic Communication Systems for Speech Transmission ,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 8, pp. 2434–2444, 2021.
  * [11] H. Xie and Z. Qin, “A Lite Distributed Semantic Communication System for Internet of Things,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 1, pp. 142–153, 2020.
  * [12] “Bitcoin: A Peer-to-Peer Electronic Cash System,” _Decentralized Business Review_ , p. 21260, 2008.
  * [13] Q. Yang, Y. Zhao, H. Huang, Z. Xiong, J. Kang, and Z. Zheng, “Fusing Blockchain and AI With Metaverse: A Survey ,” _IEEE Open Journal of the Computer Society_ , vol. 3, pp. 122–136, 2022.
  * [14] Y. Lin, Z. Gao, Y. Tu, H. Du, D. Niyato, J. Kang, and H. Yang, “A Blockchain-based Semantic Exchange Framework for Web 3.0 toward Participatory Economy,” _arXiv preprint arXiv:2211.16662_ , 2022.
  * [15] Q. Hu, G. Zhang, Z. Qin, Y. Cai, G. Yu, and G. Y. Li, “Robust Semantic Communications with Masked VQ-VAE Enabled Codebook,” _arXiv preprint arXiv:2206.04011_ , 2022.
  * [16] X. Luo, Z. Chen, M. Tao, and F. Yang, “Encrypted Semantic Communication Using Adversarial Training for Privacy Preserving,” _arXiv preprint arXiv:2209.09008_ , 2022.
  * [17] G. Tolias, F. Radenovic, and O. Chum, “Targeted Mismatch Adversarial Attack: Query With a Flower to Retrieve the Tower,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5037–5046.
  * [18] Z. Wan, Y. Zhou, and K. Ren, “zk-AuthFeed: Protecting Data Feed to Smart Contracts with Authenticated Zero Knowledge Proof,” _IEEE Transactions on Dependable and Secure Computing_ , 2022.
  * [19] H. S. Galal and A. M. Youssef, “Aegis: Privacy-Preserving Market for Non-Fungible Tokens,” _IEEE Transactions on Network Science and Engineering_ , 2022.
  * [20] Y. Fan, B. Xu, L. Zhang, J. Song, A. Zomaya, and K.-C. Li, “Validating the integrity of Convolutional Neural Network predictions based on Zero-Knowledge Proof,” _Information Sciences_ , 2023.
  * [21] J. V. Pavlik, “Collaborating with ChatGPT: Considering the implications of generative artificial intelligence for journalism and media education,” _Journal. Mass Commun. Educ._ , 2023.
  * [22] F.-A. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” _arXiv preprint arXiv:2209.04747_ , 2022.
  * [23] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” _IEEE Signal Processing Mag._ , vol. 35, no. 1, pp. 53–65, Jan. 2018.
  * [24] Y.-J. Shih, S.-L. Wu, F. Zalkow, M. Muller, and Y.-H. Yang, “Theme transformer: Symbolic music generation with theme-conditioned transformer,” _IEEE Trans. Multimedia_ , to appear, 2022.
  * [25] K. R. Castleman, _Digital Image Processing_. Prentice Hall Press, 1996.
  * [26] Iden3, “zkSnark circuit compiler,” https://github.com/iden3/circom, 2022\.
  * [27] F. Radenović, A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, “Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5706–5715.
  * [28] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “ImageNet Large Scale Visual Recognition Challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [29] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2017.