# Diverse Embedding Expansion Network and Low-Light Cross-Modality
Benchmark for Visible-Infrared Person Re-identification

Yukang Zhang1,2, Hanzi Wang1,2,3
1Fujian Key Laboratory of Sensing and Computing for Smart City,
School of Informatics, Xiamen University, 361005, P.R. China.
2Key Laboratory of Multimedia Trusted Perception and Efficient Computing,
Ministry of Education of China, Xiamen University, 361005, P.R. China.
3Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author.

###### Abstract

For the visible-infrared person re-identification (VIReID) task, one of the
major challenges is the modality gaps between visible (VIS) and infrared (IR)
images. However, the training samples are usually limited, while the modality
gaps are too large, which leads that the existing methods cannot effectively
mine diverse cross-modality clues. To handle this limitation, we propose a
novel augmentation network in the embedding space, called diverse embedding
expansion network (DEEN). The proposed DEEN can effectively generate diverse
embeddings to learn the informative feature representations and reduce the
modality discrepancy between the VIS and IR images. Moreover, the VIReID model
may be seriously affected by drastic illumination changes, while all the
existing VIReID datasets are captured under sufficient illumination without
significant light changes. Thus, we provide a low-light cross-modality (LLCM)
dataset, which contains 46,767 bounding boxes of 1,064 identities captured by
9 RGB/IR cameras. Extensive experiments on the SYSU-MM01, RegDB and LLCM
datasets show the superiority of the proposed DEEN over several other state-
of-the-art methods. The code and dataset are released at:
https://github.com/ZYK100/LLCM

##  1 Introduction

Figure 1: Motivation of the proposed DEEN, which aims to generate diverse
embeddings to make the network focus on learning with the informative feature
representations to reduce the modality gaps between the VIS and IR images.

Person re-identification (ReID) aims to match a given person with gallery
images captured by different cameras [3, 52, 9]. Most existing ReID methods
[50, 22, 38, 30, 24] only focus on matching RGB images captured by visible
cameras at daytime. However, these methods may fail to achieve encouraging
results when visible cameras cannot effectively capture person’s information
under complex conditions, such as at night or low-light environments. To solve
this problem, some visible (VIS)-infrared (IR) person re-identification
(VIReID) methods [15, 41, 48, 39] have been proposed to retrieve the VIS (IR)
images according to the corresponding IR (VIS) images.

Compared with the widely studied person ReID task, the VIReID task is much
more challenging due to the additional cross-modality discrepancy between the
VIS and IR images [45, 49, 33, 51]. Typically, there are two popular types of
methods to reduce this modality discrepancy. One type is the feature-level
methods [40, 42, 5, 11, 35, 16], which try to project the VIS and IR features
into a common embedding space, where the modality discrepancy can be
minimized. However, the large modality discrepancy makes these methods
difficult to project the cross-modality images into a common feature space
directly. The other type is the image-level methods [32, 29, 28, 4], which aim
to reduce the modality discrepancy by translating an IR (or VIS) image into
its VIS (or IR) counterpart by using the GANs [8]. Despite their success in
reducing the modality gaps, the generated cross-modality images are usually
accompanied by some noises due to the lack of the VIS-IR image pairs.

Figure 2: Comparison of person images on the SYSU-MM01 (1st row), RegDB (2nd
row), and LLCM (3rd-5th rows) datasets. Each row shows four VIS images and
four IR images of two identities. It is obvious that our LLCM contains a more
challenging and realistic VIReID environment.

In this paper, we propose a novel augmentation network in the embedding space
for the VIReID task, called diverse embedding expansion network (DEEN), which
consists of a diverse embedding expansion (DEE) module and a multistage
feature aggregation (MFA) block. The proposed DEE module can generate more
embeddings followed by a novel center-guided pair mining (CPM) loss to drive
the DEE module to focus on learning with the diverse feature representations.
As illustrated in Fig. 1, by exploiting the generated embeddings with diverse
information, the proposed DEE module can achieve the performance improvement
by using more diverse embeddings. The proposed MFA block can aggregate the
features from different stages for mining potential channel-wise and spatial
feature representations, which increases the network’s capacity for mining
different-level diverse embeddings.

Moreover, we observe that the existing VIReID datasets are captured under the
environments with sufficient illumination. However, the performance of the
VIReID methods may be seriously affected by drastic illumination changes or
low illuminations. Therefore, we collect a challenging low-light cross-
modality dataset, called LLCM dataset, which is shown in Fig. 2. Compared with
the other VIReID datasets, the LLCM dataset contains a larger number of
identities and images captured under low-light scenes, which introduces more
challenges to the real-world VIReID task.

In summary, the main contributions are as follows:

$\bullet$ We propose a novel diverse embedding expansion (DEE) module with a
center-guided pair mining (CPM) loss to generate more embeddings for learning
the diverse feature representations. We are the first to augment the
embeddings in the embedding space in VIReID. Besides, we also propose an
effective multistage feature aggregation (MFA) block to mine potential
channel-wise and spatial feature representations.

$\bullet$ With the incorporation of DEE, CPM loss and MFA into an end-to-end
learning framework, we propose an effective diverse embedding expansion
network (DEEN), which can effectively reduce the modality discrepancy between
the VIS and IR images.

$\bullet$ We collect a low-light cross-modality (LLCM) dataset, which contains
46,767 images of 1,064 identities captured under the environments with
illumination changes and low illuminations. The LLCM dataset has more new and
important features, which can facilitate the research of VIReID towards
practical applications.

$\bullet$ Extensive experiments show that the proposed DEEN outperforms the
other state-of-the-art methods for the VIReID task on three challenging
datasets.

##  2 Related Work

Generally speaking, there are two main categories of methods in VIReID: the
image-level methods and the feature-level methods.

The image-level VIReID methods try to transform one modality into the other
for reducing the modality discrepancy between the VIS and IR images in the
image space. For this purpose, some GANs-based [32, 4, 28, 29] methods are
proposed to perform identity-preserving person image style transformation for
aligning cross-modality images and alleviating the problem of limited data.
These methods often design complex generative models to align cross-modality
images. However, due to the lack of VIS-IR image pairs, the generated images
are unavoidably accompanied by some noises. X-modality [14] and its variations
[49, 34] apply a lightweight network to introduce an auxiliary middle modality
to assist the cross-modality search task. However, there is still a modality
gap between this middle modality and the VIS / IR modality.

Figure 3: The pipeline of the proposed network, which includes a DEE module
and a MFA block. The DEE module can generate more embeddings with a novel CPM
loss to learn diverse feature representations. The MFA block can aggregate the
embeddings from different stages for mining diverse channel-wise and spatial
feature representations.

The feature-level methods aim to find a modality-shared and modality-specific
feature space, where the modality gaps can be minimized. For this purpose,
MAUM [15] tries to learn cross-modality metrics in two uni-directions to
further enhance them with memory-based augmentation. RFM [25] introduces a
cross-center loss to explore a more compact intra-class distribution. DCLNet
[23] encourages the positive pixels with the same semantic information to be
close, while it simultaneously pushes the negative pixels away. cmGAN [5]
designs a cutting-edge discriminator to learn discriminative representations
from different modalities. However, the large modality gaps between the VIS
and IR images make it difficult to project the cross-modality images into a
common space directly [7, 21, 26, 18].

##  3 Method

###  3.1 Model Architecture

Fig. 3 provides an overview of the proposed diverse embedding expansion
network (DEEN), which utilizes a two-stream ResNet-50 network [12, 44] as the
backbone. The VIS-IR features are fed into the proposed diverse embedding
expansion (DEE) module to generate more embeddings. Then, a center-guided pair
mining (CPM) loss is proposed to make the generated embeddings as diverse as
possible for learning informative feature representations. Besides, we
incorporate an effective MFA block to aggregate the features from different
stages for mining diverse channel-wise and spatial feature representations.
During the training stage, all the features before and after the batch
normalization (BN) layer are fed into different losses to jointly optimize
DEEN.

###  3.2 Diverse Embedding Expansion Module

The proposed DEE module is used to generate more embeddings to alleviate the
problem of insufficient training data by using a multi-branch convolutional
generation structure. Specifically, for each branch of DEE, we firstly use
three $3\times 3$ dilated convolutional layers $\varphi^{1}_{3\times 3}$,
$\varphi^{2}_{3\times 3}$, $\varphi^{3}_{3\times 3}$ with different dilation
ratios $(1,2,3)$ to reduce the number of feature maps f to 1 / 4 of its own
size, and then we obtain the feature maps by combining them into one feature
map, followed by a ReLU activation layer $\textbf{F}_{ReLU}$ to improve the
non-linear representation capability of the DEE. Then, another convolutional
layer $\theta_{1\times 1}$ with a kernel in size of $1\times 1$ is applied to
the obtained feature map to change its dimension as same as f. Thus, the
generated embeddings $\textbf{f}^{i}_{+}$ of the $i$-th branch can be written
as follows:

$\textbf{f}^{i}_{+}=\theta_{1\times 1}(\textbf{F}_{ReLU}(\varphi^{1}_{3\times
3}(\textbf{f})+\varphi^{2}_{3\times 3}(\textbf{f})+\varphi^{3}_{3\times
3}(\textbf{f}))).$ (1)

Then, all the generated embeddings are concatenated together and used as the
input to the next stage of the backbone network.

###  3.3 Center-Guided Pair Mining Loss

As we can see from the above operation, the DEE module can only generate more
embeddings using a multi-branch convolutional block. However, this operation
cannot effectively obtain diverse embeddings. Thus, we apply the following
three properties to constrain the generated embeddings as diverse as possible
to effectively reduce the modality discrepancy between the VIS and IR images:

Property 1: The generated embeddings should be as diverse as possible to
effectively learn the informative feature representations. This means that we
need to push away the distances between the generated embeddings and the
original embeddings to learn diverse features and mine diverse cross-modality
clues.

Property 2: The generated embeddings should facilitate reducing the modality
discrepancy between the VIS and IR images. This means that we need to pull
close the distances between the embeddings generated from the VIS modality and
the original IR embeddings. Similarly, we also need to pull close the
distances between the embeddings generated from the IR modality and the
original VIS embeddings.

Property 3: The intra-class distance should be less than the inter-class one.
By Property 2, it pushes close the distance between the generated embeddings
and the original ones, which may cause the embeddings of different classes to
become close. Thus, it is necessary to keep the intra-class distance less than
the inter-class distance.

Figure 4: Illustration of the proposed CPM loss for DEE.

As shown in Fig. 4, for embeddings generated from the VIS modality, the CPM
loss can be formulated as follows:

$\begin{split}{\mathcal{L}(\textbf{f}_{v},\textbf{f}_{n},\textbf{f}_{v+}^{i})}=[\textbf{{D}}(\textbf{f}^{j}_{n},\textbf{f}^{i,j}_{v+})-\textbf{{D}}(\textbf{f}^{j}_{v},\textbf{f}^{i,j}_{v+})-\textbf{{D}}(\textbf{f}^{j}_{v},\textbf{f}^{k}_{v})]_{+},\end{split}$
(2)

where ${\textbf{{D}}}(\cdot,\cdot)$ is the Euclidean distance between two
embeddings. $\textbf{f}_{v}$ and $\textbf{f}_{n}$ are the original embeddings
from the VIS and IR modalities, and $\textbf{f}^{i}_{v+}$ is the embeddings
generated from the $i$-th branch of the VIS modality. $j,k$ are different
identities in a minibatch and $[z]_{+}=max(z,0)$. In Eq. (2), the first term
can pull the generated embeddings $\textbf{f}^{i}_{v+}$ towards the original
IR’s embeddings $\textbf{f}_{n}$ to reduce the modality discrepancy between
$\textbf{f}^{i,j}_{v+}$ and $\textbf{f}^{j}_{n}$. The second term can push the
generated embeddings $\textbf{f}_{v+}$ away from the VIS’s embeddings
$\textbf{f}_{v}$ to enable $\textbf{f}_{v+}$ to learn informative feature
representations. The third term can make the intra-class distance less than
the inter-class distance.

Then, we use the embedding centers $\textbf{c}_{v}$ and $\textbf{c}_{n}$ of
each class to make the centers of generated embeddings $\textbf{c}^{i}_{v+}$
and $\textbf{c}^{i}_{n+}$ more discriminative, and introduce a margin term
$\alpha$ to balance the three terms in Eq. (2). Thus, for the embeddings from
VIS, the CPM loss is formulated as follows:

$\begin{split}{\mathcal{L}(\textbf{c}_{v},\textbf{c}_{n},\textbf{c}_{v+}^{i})}=[\textbf{{D}}(\textbf{c}^{j}_{n},\textbf{c}^{i,j}_{v+})-\textbf{{D}}(\textbf{c}^{j}_{v},\textbf{c}^{i,j}_{v+})-\textbf{{D}}(\textbf{c}^{j}_{v},\textbf{c}^{k}_{v})+\alpha]_{+}.\end{split}$
(3)

Similarly, for the class centers $\textbf{c}^{i}_{n+}$ of embeddings generated
from IR, we have:

$\begin{split}{\mathcal{L}(\textbf{c}_{v},\textbf{c}_{n},\textbf{c}_{n+}^{i})}=[\textbf{{D}}(\textbf{c}^{j}_{v},\textbf{c}^{i,j}_{n+})-\textbf{{D}}(\textbf{c}^{j}_{n},\textbf{c}^{i,j}_{n+})-\textbf{{D}}(\textbf{c}^{j}_{n},\textbf{c}^{k}_{n})+\alpha]_{+}.\end{split}$
(4)

Thus, the final CPM loss can be formulated as follows:

$\mathcal{L}_{cpm}={\mathcal{L}(\textbf{c}_{v},\textbf{c}_{n},\textbf{c}_{v+}^{i})}+{\mathcal{L}(\textbf{c}_{v},\textbf{c}_{n},\textbf{c}_{n+}^{i})}.$
(5)

Besides, to ensure that the generated embeddings from different branches can
capture different informative feature representations, we force these
different embeddings generated by different branches orthogonal to minimize
the overlapping elements. Therefore, the orthogonal loss can be formulated as
follows:

$\mathcal{L}_{ort}=\sum\limits_{m=1}^{i-1}\sum\limits_{n=m+1}^{i}({\textbf{f}^{m}_{+}}^{T}\textbf{f}^{n}_{+}),$
(6)

where $m$ and $n$ are the $m$-th and $n$-th generated embeddings from the
original embeddings, respectively. The orthogonal loss can enforce the
generated embeddings to learn more informative feature representations.

###  3.4 Multistage Feature Aggregation Block

Features aggregation of different levels has been demonstrated to be helpful
to semantic segmentation, classification and detection task[1, 55, 54]. To
aggregate the features from different stages for mining diverse channel-wise
and spatial feature representations, we incorporate an effective channel-
spatial multistage feature aggregation (MFA) block to aggregate multi-stage
features inspired by [31].

Next, we elaborate on the detail of the MFA block, which is shown in Fig. 3.
Specifically, we consider two types of source features for the channel-spatial
aggregation block in each stage of the backbone network: low-level feature
maps $\textbf{f}_{l}\in\mathbb{R}^{C_{l}\times H_{l}\times W_{l}}$ before the
stage and high-level feature maps $\textbf{f}_{h}\in\mathbb{R}^{C_{h}\times
H_{h}\times W_{h}}$ after the stage, where C, W and H denote the number of the
channel, width and height of features, respectively. First, we employ three
1×1 convolutional layers $\psi^{1}_{q},\psi^{1}_{v},\psi^{1}_{k}$ to transform
f into three compact embeddings: $\psi_{q}^{1}(\textbf{f}_{h})$ ,
$\psi_{v}^{1}(\textbf{f}_{l})$ and $\psi_{k}^{1}(\textbf{f}_{l})$ . Then, we
compute the channel-wise similarity matrix
$\textbf{M}^{c}\in\mathbb{R}^{C^{{}^{\prime}}\times C^{{}^{\prime}}}$ by
matrix multiplication followed by softmax:

$\textbf{M}^{c}=\textbf{F}_{softmax}(\psi_{q}^{1}(\textbf{f}_{h})\times\psi_{k}^{1}(\textbf{f}_{l})).$
(7)

Consequently, we implement the channel-wise multistage feature aggregation by
restoring the channel dimension by the matrix multiplication of
$\psi_{v}^{1}(\textbf{f}_{l})$ and $\textbf{M}^{c}$. After that, another
$1\times 1$ convolutional layer $\omega^{c}$ is applied to transform the size
of the above feature maps to that of $\textbf{f}_{h}$. Finally, we get the
output by adding $\textbf{f}_{h}$ to it by matrix addition:

$\textbf{f}_{h}^{c}={{\omega^{c}}(\psi_{v}^{1}(\textbf{f}_{l})\times\textbf{M}^{c})+\textbf{f}_{h}}.$
(8)

After that, $\textbf{f}_{h}^{c}$ obtained from the above operations and the
low-level feature map $\textbf{f}_{l}$ are used to perform the spatial feature
aggregation operation, which is similar to the channel-wise multistage feature
aggregation operation. Finally, we get the MFA’s output as follows:

$\textbf{f}_{h}^{s}={{\omega^{s}}(\psi_{v}^{2}(\textbf{f}_{l})\times\textbf{M}^{s})+\textbf{f}_{h}^{c}},$
(9)

where $\omega^{s}$ and $\psi_{v}^{2}$ are two $1\times 1$ convolutional
layers, and $\textbf{M}^{s}$ is the spatial similarity matrix.

###  3.5 Multi-Loss Optimization

Besides the proposed ${\mathcal{L}_{cpm}}$ and ${\mathcal{L}_{ort}}$, we also
combine the cross-entropy loss ${\mathcal{L}_{ce}}$ [17] and the triplet loss
${\mathcal{L}_{tri}}$ [13] to jointly optimize the network in an end-to-end
manner by minimizing the sum of these four losses ${\mathcal{L}_{total}}$,
which can be formulated as follows:

${\mathcal{L}_{total}}={\mathcal{L}_{ce}}+{\mathcal{L}_{tri}}+\lambda_{1}{\mathcal{L}_{cpm}}+\lambda_{2}{\mathcal{L}_{ort}},$
(10)

where $\lambda_{1}$ and $\lambda_{2}$ are the coefficients to control the
relative importance of the loss terms.

##  4 LLCM Dataset

###  4.1 Dataset Description

Figure 5: The distribution of the LLCM’s images in the 2D space. It can be
seen that the images under different light conditions present different
styles, which further increases the modality discrepancy between the VIS and
IR images.

In this paper, we collect a new challenging low-light cross-modality dataset,
called LLCM dataset. The LLCM dataset utilizes a 9-camera network deployed in
low-light environments, which can capture the VIS images in daytime and
capture the IR images at night. For protecting the personal privacy
information, we utilize MTCNN [47] to get the bounding boxes of persons’ faces
and blur those regions. We make sure that each annotated identity is captured
by both the VIS and IR cameras. Some examples from the LLCM dataset are shown
in Fig. 2. As shown in Tab. 1, compared with the existing VIReID datasets, the
LLCM dataset has the following new and important features:

First, the images in the LLCM dataset are captured under complex low-light
environment for both the VIS and IR modalities, which contains severe
illumination changes and is a common problem in the real scenes. As Fig. 2 and
Fig. 5 shown, the severe light conditions can change the color of persons’
clothes and cause the loss of texture information of the clothes, which
introduces great challenges to VIReID. Second, the LLCM dataset has a larger
number of identities and bounding boxes. This dataset contains 46,767 bounding
boxes of 1,064 identities, making it the largest VIReID dataset at present
(see Tab 1). Third, the LLCM dataset is collected in over 100 days from
January to April, and different climate conditions and cloth styles are
considered. Long-term data collection helps to study the VIReID task in
different climates and clothing styles, which increases the generalization of
the VIReID model.

Besides, considering the real-world applications, the LLCM dataset also
contains many images that suffer from various challenges, such as motion
bluring, pose variation, camera view changes, occlusion, low resolution and
others. All in all, the LLCM dataset is a challenging dataset for the VIReID
task, which can further facilitate the research of VIReID towards practical
applications.

###  4.2 Evaluation Protocol

Datasets | IDs | Images | VIS / IR cam. | low-light
---|---|---|---|---
RegDB [19] | 412 | 8,240 | 1 / 1 | ✗
SYSU-MM01 [36] | 491 | 38,271 | 4 / 2 | ✗
LLCM | 1,064 | 46,767 | 9 / 9 | ✔
Table 1: Comparison between the LLCM and other two popular VIReID datasets.

We divide the LLCM dataset into a training set and a testing set at a ratio
about 2:1. The training set contains 30,921 bounding boxes of 713 identities
(16,946 bounding boxes are from the VIS modality and 13,975 bounding boxes are
from the IR modality), and the testing set contains 13,909 bounding boxes of
351 identities (8,680 bounding boxes are from the VIS modality and 7,166
bounding boxes are from the IR modality). Similar to the RegDB [19] dataset,
both the VIS to IR mode and the IR to VIS mode are used to evaluate the
performance of the VIReID models. During the testing stage, for each camera,
we randomly choose one image from the images of each identity to form the
gallery set for evaluation the performance of the models. We repeat the above
evaluation 10 times with random split of the gallery set and report the
average performance.

Methods | SYSU-MM01 | RegDB
---|---|---
All Search | Indoor Search | VIS to IR | IR to VIS
R-1 | R-10 | R-20 | mAP | R-1 | R-10 | R-20 | mAP | R-1 | R-10 | R-20 | mAP | R-1 | R-10 | R-20 | mAP
BDTR[46] | 17.0 | 55.4 | 72.0 | 19.7 | - | - | - | - | 33.6 | 58.6 | 67.4 | 32.8 | 32.9 | 58.5 | 68.4 | 32.0
D2RL[32] | 28.9 | 70.6 | 82.4 | 29.2 | - | - | - | - | 43.4 | 66.1 | 76.3 | 44.1 | - | - | - | -
Hi-CMD[4] | 34.9 | 77.6 | - | 35.9 | - | - | - | - | 70.9 | 86.4 | - | 66.0 | - | - | - | -
JSIA-ReID[29] | 38.1 | 80.7 | 89.9 | 36.9 | 43.8 | 86.2 | 94.2 | 52.9 | 48.1 | - | - | 48.9 | 48.5 | - | - | 49.3
AlignGAN[28] | 42.4 | 85.0 | 93.7 | 40.7 | 45.9 | 87.6 | 94.4 | 54.3 | 57.9 | - | - | 53.6 | 56.3 | - | - | 53.4
X-Modality[14] | 49.9 | 89.8 | 96.0 | 50.7 | - | - | - | - | 62.2 | 83.1 | 91.7 | 60.2 | - | - | - | -
DDAG[44] | 54.8 | 90.4 | 95.8 | 53.0 | 61.0 | 94.1 | 98.4 | 68.0 | 69.3 | 86.2 | 91.5 | 63.5 | 68.1 | 85.2 | 90.3 | 61.8
LbA[20] | 55.4 | - | - | 54.1 | 58.5 | - | - | 66.3 | 74.2 | - | - | 67.6 | 67.5 | - | - | 72.4
NFS[2] | 56.9 | 91.3 | 96.5 | 55.5 | 62.8 | 96.5 | 99.1 | 69.8 | 80.5 | 91.6 | 95.1 | 72.1 | 78.0 | 90.5 | 93.6 | 69.8
CM-NAS[6] | 60.8 | 92.1 | 96.8 | 58.9 | 68.0 | 94.8 | 97.9 | 52.4 | 82.8 | 95.1 | 97.7 | 79.3 | 81.7 | 94.1 | 96.9 | 77.6
MCLNet[10] | 65.4 | 93.3 | 97.1 | 62.0 | 72.6 | 97.0 | 99.2 | 76.6 | 80.3 | 92.7 | 96.0 | 73.1 | 75.9 | 90.9 | 94.6 | 69.5
FMCNet[48] | 66.3 | - | - | 62.5 | 68.2 | - | - | 74.1 | 89.1 | - | - | 84.4 | 88.4 | - | - | 83.9
SMCL[34] | 67.4 | 92.9 | 96.8 | 61.8 | 68.8 | 96.6 | 98.8 | 75.6 | 83.9 | - | - | 79.8 | 83.1 | - | - | 78.6
DART[41] | 68.7 | 96.4 | 99.0 | 66.3 | 72.5 | 97.8 | 99.5 | 78.2 | 83.6 | - | - | 75.7 | 82.0 | - | - | 73.8
CAJ[43] | 69.9 | 95.7 | 98.5 | 66.9 | 76.3 | 97.9 | 99.5 | 80.4 | 85.0 | 95.5 | 97.5 | 79.1 | 84.8 | 95.3 | 97.5 | 77.8
MPANet[37] | 70.6 | 96.2 | 98.8 | 68.2 | 76.7 | 98.2 | 99.6 | 81.0 | 82.8 | - | - | 80.7 | 83.7 | - | - | 80.9
MMN [49] | 70.6 | 96.2 | 99.0 | 66.9 | 76.2 | 97.2 | 99.3 | 79.6 | 91.6 | 97.7 | 98.9 | 84.1 | 87.5 | 96.0 | 98.1 | 80.5
DCLNet[23] | 70.8 | - | - | 65.3 | 73.5 | - | - | 76.8 | 81.2 | - | - | 74.3 | 78.0 | - | - | 70.6
MAUM[15] | 71.7 | - | - | 68.8 | 77.0 | - | - | 81.9 | 87.9 | - | - | 85.1 | 87.0 | - | - | 84.3
DEEN (ours) | 74.7 | 97.6 | 99.2 | 71.8 | 80.3 | 99.0 | 99.8 | 83.3 | 91.1 | 97.8 | 98.9 | 85.1 | 89.5 | 96.8 | 98.4 | 83.4

Table 2: Comparisons between the proposed DEEN and some state-of-the-art
methods on the SYSU-MM01 and RegDB datasets.

##  5 Experiments

###  5.1 Datasets

The SYSU-MM01 dataset [36] contains 491 identities captured by 4 VIS cameras
and 2 IR cameras, including the All-Search and Indoor-Search modes. For the
All-Search mode, all the images captured by all the VIS cameras are used as
the gallery set. For the Indoor-Search mode, only the images captured by two
indoor VIS cameras are used as the gallery set. The RegDB dataset [19]
consists of 412 identities, and each identity has 10 VIS images and 10 IR
images captured by a pair of overlapping cameras.

###  5.2 Implementation Details

All the input images are firstly resized to $3\times 384\times 144$, and the
random horizontal flip and random erasing [53] techniques are adopted during
the training phase. The initial learning rate is set to $1\times 10^{-2}$ and
then it increases to $1\times 10^{-1}$ after 10 epochs with a warm-up
strategy. After that, we decay the learning rate to $1\times 10^{-2}$ at 20
epoch, and further decay to $1\times 10^{-3}$ and $1\times 10^{-4}$ at epoch
60 and epoch 120, respectively, until a total of 150 epochs. In each mini-
batch, we randomly select 4 VIS images and 4 IR images of 6 identities for
training. The SGD optimizer is adopted for training, where the momentum is set
to 0.9. For the RegDB dataset, we remove stage-4 and plug the proposed DEE
module into the DEEN after stage-2.

###  5.3 Comparison with State-of-the-art Methods

We firstly compare the proposed DEEN with several state-of-the-art methods to
demonstrate the superiority of our method. The experimental results on the
SYSU-MM01 and RegDB datasets are reported in Tab. 2, and the results on our
LLCM dataset are reported in Tab. 3.

SYSU-MM01 and RegDB: From Tab. 2, we can see that the results on the two
datasets show that the proposed DEEN achieves the best performance against all
other state-of-the-art methods. Specifically, for the All-Search mode on SYSU-
MM01, DEEN achieves 74.7% Rank-1 accuracy and 71.8% mAP. For the Indoor-Search
mode, DEEN achieves 80.3% Rank-1 accuracy and 83.3% mAP. For the VIS to IR
mode on RegDB, DEEN achieves 91.1% Rank-1 accuracy and 85.1% mAP. For the IR
to VIS mode, the proposed DEEN also obtains 89.5% Rank-1 accuracy and 83.4%
mAP. The results validate the effectiveness of our method. Moreover, the
results also demonstrate that the proposed DEEN can effectively reduce the
modality discrepancy between the VIS and IR modalities.

Model | LLCM
---|---
IR to VIS | VIS to IR
R-1 | R-10 | R-20 | mAP | R-1 | R-10 | R-20 | mAP
DDAG[44] | 40.3 | 71.4 | 79.6 | 48.4 | 48.0 | 79.2 | 86.1 | 52.3
DDAG*[44] | 41.0 | 73.4 | 81.9 | 49.6 | 48.5 | 81.0 | 87.8 | 53.0
AGW[45] | 43.6 | 74.6 | 82.4 | 51.8 | 51.5 | 81.5 | 87.9 | 55.3
LbA[20] | 43.8 | 78.2 | 86.6 | 53.1 | 50.8 | 84.3 | 91.1 | 55.6
LbA*[20] | 44.6 | 78.2 | 86.8 | 53.8 | 50.8 | 84.6 | 91.1 | 55.9
AGW*[45] | 46.4 | 77.8 | 85.2 | 54.8 | 56.0 | 84.9 | 90.6 | 59.1
CAJ[43] | 48.8 | 79.5 | 85.3 | 56.6 | 56.5 | 85.3 | 90.9 | 59.8
DART[41] | 52.2 | 80.7 | 87.0 | 59.8 | 60.4 | 87.1 | 91.9 | 63.2
MMN[49] | 52.5 | 81.6 | 88.4 | 58.9 | 59.9 | 88.5 | 93.6 | 62.7
DEEN (ours) | 54.9 | 84.9 | 90.9 | 62.9 | 62.5 | 90.3 | 94.7 | 65.8
Table 3: Performance obtained by the competing methods on our LLCM dataset.
The symbol of “*” represents the methods that we reproduced with the random
erasing technique.

LLCM: Tab. 3 shows the results on our LLCM dataset. Here, we use several
representative open-source methods to evaluate our LLCM dataset and compare
them with our method. From Tab. 3 we can draw the following conclusions: the
best method only obtains 54.9% Rank-1 accuracy and 62.9% mAP under the IR to
VIS mode. The results of the existing methods on our LLCM dataset are
generally unsatisfactory. This shows that, on one hand, our LLCM dataset is a
very challenging dataset. On the other hand, the change of light has serious
influence on the VIReID model. Besides, the proposed DEEN achieves the best
performance under both the VIS to IR mode and the IR to VIS mode, which
demonstrates the effectiveness of the proposed DEEN to reduce the modality
gaps between the VIS and IR images.

###  5.4 Ablation Studies

Effectiveness of each component: To evaluate the contribution of each
component to DEEN, we conduct some ablation studies on the LLCM and SYSU-MM01
datasets by removing certain modules from DEEN and evaluate the influence on
the performance. The overall settings remain the same, while only the module
under evaluation is used in or removed from DEEN. As shown in Tab. 4, although
the DEE module can generate more embeddings using a multi-branch convolutional
block, which slightly improves the performance of the baseline, the results
are not satisfactory. After being constrained by the proposed CPM loss to
generate diverse embeddings, DEE can greatly improve the performance of the
model and effectively reduce the modality discrepancy between the VIS and IR
images. Besides, the proposed MFA block can improve the performance of the
baseline by aggregating the features from different stages for mining diverse
channel-wise and spatial feature representations. With the incorporation of
DEE, CPM and MFA into an end-to-end learning framework, DEEN achieves an
impressive performance improvement on two challenging VIReID datasets, which
shows DEE and MFA can benefit from each other for generating diverse
embeddings.

| Settings | LLCM | SYSU-MM01
---|---|---|---
| DEE | $\mathcal{L}_{cpm}$ | $\mathcal{L}_{ort}$ | MFA | R-1 | mAP | R-1 | mAP
|  |  |  |  | 45.4 | 53.6 | 60.7 | 57.7
| ✓ |  |  |  | 50.5 | 59.0 | 64.7 | 62.0
| ✓ | ✓ |  |  | 53.1 | 61.1 | 69.2 | 66.2
| ✓ |  | ✓ |  | 51.5 | 60.1 | 65.3 | 63.2
| ✓ | ✓ | ✓ |  | 53.9 | 62.3 | 69.8 | 66.7
|  |  |  | ✓ | 51.2 | 59.6 | 64.7 | 62.0
| ✓ | ✓ | ✓ | ✓ | 54.9 | 62.9 | 74.7 | 71.8
Table 4: The influence of each component on the performance of the proposed DEEN. Methods | LLCM | SYSU-MM01
---|---|---
R-1 | mAP | R-1 | mAP
DEE after stage-0 | 48.5 | 57.1 | 63.4 | 59.4
DEE after stage-1 | 49.4 | 57.8 | 63.7 | 60.8
DEE after stage-2 | 49.6 | 57.9 | 65.3 | 61.7
DEE after stage-3 | 53.9 | 62.3 | 69.8 | 66.7
DEE after stage-4 | 50.9 | 59.6 | 60.0 | 58.0
Table 5: The influence of which stage of ResNet-50 to plug the DEE module. Methods | LLCM | SYSU-MM01
---|---|---
R-1 | mAP | R-1 | mAP
Two branches | 52.6 | 60.9 | 67.5 | 64.6
Three branches | 53.9 | 62.3 | 69.2 | 66.2
Four branches | 52.4 | 60.7 | 67.6 | 64.6
Table 6: Study about how many branches are suitable for DEE. Methods | LLCM | SYSU-MM01
---|---|---
R-1 | mAP | R-1 | mAP
NL | 50.1 | 57.4 | 63.8 | 60.7
MFA | 51.2 | 59.6 | 64.7 | 62.0
NL + DEE | 54.2 | 62.4 | 73.4 | 70.3
MFA+ DEE | 54.9 | 62.9 | 74.7 | 71.8
Table 7: Comparison with the Non-Local (NL) block.

Figure 6: Influence of different $\lambda_{1}$, $\lambda_{2}$ and $\alpha$
values on our LLCM. Figure 7: (a-e) show the intra-class and inter-class
distances of cross-modality features. The intra-class and inter-class
distances are indicated in blue and green colors, respectively. (f-j) show the
distribution of feature embeddings in the 2D feature space, where circles and
triangles in different colors denote visible and infrared modalities. A total
of 20 persons are selected from the test set. The samples with the same color
are from the same person. The “dot” and “cross” markers denote the images from
the VIS and IR modalities, respectively.

The influence of which stage of ResNet-50 to plug the DEE module. The proposed
DEE can be plugged after any stage of the backbone network. In our
experiments, we use ResNet-50 as the backbone, which has five stages: stage-0
to stage-4. We plug DEE after different stages of the ResNet-50 to study how
it will affect the performance of the DEEN. As shown in Tab. 5, when DEE is
plugged after stage-0 to stage-3, the performance gradually increases, which
shows the modality gaps become smaller and the generative ability of DEE
becomes stronger at deeper layers of the network. When DEE is plugged after
stage-3, it can achieve the best results on both LLCM and SYSU-MM01. However,
when DEE is plugged after stage-4, the performance drops rapidly because the
CPM loss works directly on the embeddings, enlarging the distances between the
generated embeddings and the original embeddings, which increases the
difficulty of model optimization. Based on the above analysis, we plug DEE
after stage-3 of the backbone if not specified.

Effectiveness on how many branches are more suitable for DEE. The proposed DEE
module utilizes a multi-branch convolutional block to generate diverse
embeddings. Here, we study how many branches are suitable for DEE. As shown in
Tab. 6, with the increase of the number of DEE’s branches from 2 to 3, more
embeddings are generated to reduce the modality gaps, so the performance
gradually increases. However, the increase of performance has an upper limit
when the number of branches is more than 3, because DEE generates too many
redundant features, which leads to the drop of performance. As a result, DEE
with three branches can achieve the best performances both on the LLCM and
SYSU-MM01 datasets. It indicates that DEE with 3 branches is more suitable for
generating diverse embeddings. Thus, we use 3 branches for DEE if not
specified.

Figure 8: Some Rank-8 retrieval results obtained by the baseline and the
proposed DEEN on our LLCM dataset.

Comparison with the Non-Local block. In this paper, we propose a MFA block to
mine diverse channel-wise and spatial feature representations inspired by the
Non-local (NL) block in [31]. Thus, we compare these two blocks to investigate
which block is more effective. As shown in Tab. 7, the MFA block outperforms
the NL block by 1.1% Rank-1 accuracy and 2.2% mAP, respectively. The results
validate the effectiveness of our MFA block. Moreover, the results also show
that the MFA block and the DEE module are complementary for generating diverse
embeddings to reduce the modality gaps between the VIS and IR images.

The influence of the hyperparameters $\lambda_{1}$, $\lambda_{2}$ and
$\alpha$. To evaluate the influence of the three hyperparameters, we give
quantitative comparisons and report the results in Fig. 6. As we can see, the
best performance is achieved when $\lambda_{1}$ is set to 0.8, $\lambda_{2}$
is set to 0.1 and $\alpha$ is set to 0.2, respectively.

###  5.5 Visualization

Feature distribution. To investigate the reason why DEEN is effective, we
visualize the inter-class and intra-class distances on our LLCM dataset as
shown in Fig. 7 (a-e). Comparing Fig. 7 (c-e) with Fig. 7 (a-b), the means
(i.e., the vertical lines) of inter-class and intra-class distances are pushed
away by MFA, DEE and DEEN, where $\delta_{1}$ <$\delta_{2}$ <$\delta_{3}$ and
$\delta_{1}$ <$\delta_{2}$ <$\delta_{4}$ <$\delta_{5}$. This shows that the
intra-class distance of DEEN is significantly reduced compared with the intra-
class distance of the initial features (Fig. 7 (a)) and the baseline features
(Fig. 7 (b)). Thus, DEEN can effectively reduce the modality discrepancy
between the VIS and the IR images. Meanwhile, we also visualize the feature
distribution with t-SNE [27] in the 2D feature space in Fig. 7 (f-j), which
shows that MFA, DEE and DEEN can effectively discriminate and aggregate
feature embeddings of the same person, and reduce the modality discrepancy.

Retrieval result. To further show the effectiveness of DEEN, we also show some
retrieval results of DEEN on our LLCM dataset in Fig. 8. For each retrieval
case, the retrieved images with green boxes mean the correct matches
corresponding the given query, while the red ones mean the incorrect matches.
In general, DEEN can effectively improve the ranking results with more
correctly matched images ranked in the top positions than the baseline.

##  6 CONCLUSION

In this paper, we propose a novel diverse embedding expansion network (DEEN)
in the embedding space for the VIReID task. The proposed DEEN can generate
diverse embeddings and mine diverse channel-wise and spatial embeddings to
learn the informative feature representations for reducing the modality
discrepancy between the VIS and IR images. Moreover, we also provide a
challenging low-light cross-modality (LLCM) dataset, which has more new and
important features and can further facilitate the research of VIReID towards
practical applications. Extensive experiments on the SYSU-MM01, RegDB and LLCM
datasets show the superiority of the proposed DEEN over several other state-
of-the-art methods.

##  7 Acknowledgments

This work was supported by the National Key Research and Development Program
of China under Grant 2022ZD0160402, by the National Natural Science Foundation
of China under Grant U21A20514, and by the FuXiaQuan National Independent
Innovation Demonstration Zone Collaborative Innovation Platform Project under
Grant 3502ZCQXT2022008.

## References

  * [1] Xuesong Chen, Canmiao Fu, Yong Zhao, Feng Zheng, Jingkuan Song, Rongrong Ji, and Yi Yang.  Salience-guided cascaded suppression network for person re-identification.  In Proceedings of the CVPR, pages 3300–3310, 2020.
  * [2] Yehansen Chen, Lin Wan, Zhihang Li, Qianyan Jing, and Zongyuan Sun.  Neural feature search for rgb-infrared person re-identification.  In Proceedings of the CVPR, pages 587–597, 2021.
  * [3] Yoonki Cho, Woo Jae Kim, Seunghoon Hong, and Sung-Eui Yoon.  Part-based pseudo label refinement for unsupervised person re-identification.  In Proceedings of the CVPR, pages 7308–7318, 2022.
  * [4] Seokeon Choi, Sumin Lee, Youngeun Kim, Taekyung Kim, and Changick Kim.  Hi-cmd: Hierarchical cross-modality disentanglement for visible-infrared person re-identification.  In Proceedings of the CVPR, pages 10257–10266, 2020.
  * [5] Pingyang Dai, Rongrong Ji, Haibin Wang, Qiong Wu, and Yuyu Huang.  Cross-modality person re-identification with generative adversarial training.  In Proceedings of the IJCAI, pages 677–683, 2018.
  * [6] Chaoyou Fu, Yibo Hu, Xiang Wu, Hailin Shi, Tao Mei, and Ran He.  Cm-nas: Cross-modality neural architecture search for visible-infrared person re-identification.  In Proceedings of the ICCV, pages 11823–11832, 2021.
  * [7] Yajun Gao, Tengfei Liang, Yi Jin, Xiaoyan Gu, Wu Liu, Yidong Li, and Congyan Lang.  Mso: Multi-feature space joint optimization network for rgb-infrared person re-identification.  In Proceedings of the ACM MM, pages 5257–5265, 2021.
  * [8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Proceedings of the NeurIPS, pages 2672–2680, 2014.
  * [9] Hongyang Gu, Jianmin Li, Guangyuan Fu, Chifong Wong, Xinghao Chen, and Jun Zhu.  Autoloss-gms: Searching generalized margin-based softmax loss function for person re-identification.  In Proceedings of the CVPR, pages 4744–4753, 2022.
  * [10] Xin Hao, Sanyuan Zhao, Mang Ye, and Jianbing Shen.  Cross-modality person re-identification via modality confusion and center aggregation.  In Proceedings of the CVPR, pages 16403–16412, 2021.
  * [11] Yi Hao, Nannan Wang, Jie Li, and Xinbo Gao.  Hsme: Hypersphere manifold embedding for visible thermal person re-identification.  In Proceedings of the AAAI, pages 8385–8392, 2019.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the CVPR, pages 770–778, 2016.
  * [13] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  ArXiv, 2017.
  * [14] Diangang Li, Xing Wei, Xiaopeng Hong, and Yihong Gong.  Infrared-visible cross-modal person re-identification with an x modality.  In Proceedings of the AAAI, pages 4610–4617, 2020.
  * [15] Jialun Liu, Yifan Sun, Feng Zhu, Hongbin Pei, Yi Yang, and Wenhui Li.  Learning memory-augmented unidirectional metrics for cross-modality person re-identification.  In Proceedings of the CVPR, pages 19366–19375, 2022.
  * [16] Yan Lu, Yue Wu, Bin Liu, Tianzhu Zhang, Baopu Li, Qi Chu, and Nenghai Yu.  Cross-modality person re-identification with shared-specific feature transfer.  In Proceedings of the CVPR, pages 13379–13389, 2020.
  * [17] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang.  Bag of tricks and a strong baseline for deep person re-identification.  In Proceedings of the CVPR Workshops, pages 1487–1495, 2019.
  * [18] Ziling Miao, Hong Liu, Wei Shi, Wanlu Xu, and Hanrong Ye.  Modality-aware style adaptation for rgb-infrared person re-identification.  In Proceedings of the IJCAI, pages 19–27, 2021.
  * [19] Dat Tien Nguyen, Hyung Gil Hong, Ki Wan Kim, and Kang Ryoung Park.  Person recognition system based on a combination of body images from visible light and thermal cameras.  Sensors, 17(3):605, 2017.
  * [20] Hyunjong Park, Sanghoon Lee, Junghyup Lee, and Bumsub Ham.  Learning by aligning: Visible-infrared person re-identification using cross-modal correspondences.  In Proceedings of the ICCV, pages 12046–12055, 2021.
  * [21] Nan Pu, Wei Chen, Yu Liu, Erwin M Bakker, and Michael S Lew.  Dual gaussian-based variational subspace disentanglement for visible-infrared person re-identification.  In Proceedings of the ACM MM, pages 2149–2158, 2020.
  * [22] Nan Pu, Wei Chen, Yu Liu, Erwin M. Bakker, and Michael S. Lew.  Lifelong person re-identification via adaptive knowledge accumulation.  In Proceedings of the CVPR, pages 7901–7910, 2021.
  * [23] Hanzhe Sun, Jun Liu, Zhizhong Zhang, Chengjie Wang, Yanyun Qu, Yuan Xie, and Lizhuang Ma.  Not all pixels are matched: Dense contrastive learning for cross-modality person re-identification.  In Proceedings of the ACM MM, page 5333–5341, 2022.
  * [24] Lei Tan, Pingyang Dai, Rongrong Ji, and Yongjian Wu.  Dynamic prototype mask for occluded person re-identification.  In Proceedings of the ACM MM, page 531–540, 2022.
  * [25] Lei Tan, Yukang Zhang, Shengmei Shen, Yan Wang, Pingyang Dai, Xianming Lin, Yongjian Wu, and Rongrong Ji.  Exploring invariant representation for visible-infrared person re-identification.  ArXiv, 2023.
  * [26] Xudong Tian, Zhizhong Zhang, Shaohui Lin, Yanyun Qu, Yuan Xie, and Lizhuang Ma.  Farewell to mutual information: Variational distillation for cross-modal person re-identification.  In Proceedings of the CVPR, pages 1522–1531, 2021.
  * [27] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  JMLR, 9:2579–2605, 2008.
  * [28] Guan’an Wang, Tianzhu Zhang, Jian Cheng, Si Liu, Yang Yang, and Zengguang Hou.  Rgb-infrared cross-modality person re-identification via joint pixel and feature alignment.  In Proceedings of the ICCV, pages 3623–3632, 2019.
  * [29] Guan-An Wang, Tianzhu Zhang Yang, Jian Cheng, Jianlong Chang, Xu Liang, Zengguang Hou, et al.  Cross-modality paired-images generation for rgb-infrared person re-identification.  In Proceedings of the AAAI, pages 12144–12151, 2020.
  * [30] Haochen Wang, Jiayi Shen, Yongtuo Liu, Yan Gao, and Efstratios Gavves.  Nformer: Robust person re-identification with neighbor transformer.  In Proceedings of the CVPR, pages 7297–7307, 2022.
  * [31] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the CVPR, pages 7794–7803, 2018.
  * [32] Zhixiang Wang, Zheng Wang, Yinqiang Zheng, Yung-Yu Chuang, and Shin’ichi Satoh.  Learning to reduce dual-level discrepancy for infrared-visible person re-identification.  In Proceedings of the CVPR, pages 618–626, 2019.
  * [33] Xing Wei, Diangang Li, Xiaopeng Hong, Wei Ke, and Yihong Gong.  Co-attentive lifting for infrared-visible person re-identification.  In Proceedings of the ACM MM, pages 1028–1037, 2020.
  * [34] Ziyu Wei, Xi Yang, Nannan Wang, and Xinbo Gao.  Syncretic modality collaborative learning for visible infrared person re-identification.  In Proceedings of the ICCV, pages 225–234, 2021.
  * [35] Ancong Wu, Wei-Shi Zheng, Shaogang Gong, and Jianhuang Lai.  Rgb-ir person re-identification by cross-modality similarity preservation.  IJCV, pages 1–21, 2020.
  * [36] Ancong Wu, Wei-Shi Zheng, Hong-Xing Yu, Shaogang Gong, and Jianhuang Lai.  Rgb-infrared cross-modality person re-identification.  In Proceedings of the ICCV, pages 5380–5389, 2017.
  * [37] Qiong Wu, Pingyang Dai, Jie Chen, Chia-Wen Lin, Yongjian Wu, Feiyue Huang, Bineng Zhong, and Rongrong Ji.  Discover cross-modality nuances for visible-infrared person re-identification.  In Proceedings of the CVPR, pages 4330–4339, 2021.
  * [38] Cheng Yan, Guansong Pang, Lei Wang, Jile Jiao, Xuetao Feng, Chunhua Shen, and Jingjing Li.  Bv-person: A large-scale dataset for bird-view person re-identification.  In Proceedings of the ICCV, pages 10943–10952, 2021.
  * [39] Bin Yang, Mang Ye, Jun Chen, and Zesen Wu.  Augmented dual-contrastive aggregation learning for unsupervised visible-infrared person re-identification.  In Proceedings of the ACM MM, page 2843–2851, 2022.
  * [40] Fan Yang, Zheng Wang, Jing Xiao, and Shin’ichi Satoh.  Mining on heterogeneous manifolds for zero-shot cross-modal image retrieval.  In Proceedings of the AAAI, pages 12589–12596, 2020.
  * [41] Mouxing Yang, Zhenyu Huang, Peng Hu, Taihao Li, Jiancheng Lv, and Xi Peng.  Learning with twin noisy labels for visible-infrared person re-identification.  In Proceedings of the CVPR, pages 14308–14317, 2022.
  * [42] Mang Ye, Xiangyuan Lan, Jiawei Li, and Pong C Yuen.  Hierarchical discriminative learning for visible thermal person re-identification.  In Proceedings of the AAAI, pages 7501–7508, 2018.
  * [43] Mang Ye, Weijian Ruan, Bo Du, and Mike Zheng Shou.  Channel augmented joint learning for visible-infrared recognition.  In Proceedings of the ICCV, pages 13567–13576, 2021.
  * [44] Mang Ye, Jianbing Shen, David J Crandall, Ling Shao, and Jiebo Luo.  Dynamic dual-attentive aggregation learning for visible-infrared person re-identification.  In Proceedings of the ECCV, pages 229–247, 2020.
  * [45] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi.  Deep learning for person re-identification: A survey and outlook.  ArXiv, 2020.
  * [46] Mang Ye, Zheng Wang, Xiangyuan Lan, and Pong C Yuen.  Visible thermal person re-identification via dual-constrained top-ranking.  In Proceedings of the IJCAI, pages 1092–1099, 2018.
  * [47] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  IEEE SPL, 23(10):1499–1503, 2016.
  * [48] Qiang Zhang, Changzhou Lai, Jianan Liu, Nianchang Huang, and Jungong Han.  Fmcnet: Feature-level modality compensation for visible-infrared person re-identification.  In Proceedings of the CVPR, pages 7349–7358, 2022.
  * [49] Yukang Zhang, Yan Yan, Yang Lu, and Hanzi Wang.  Towards a unified middle modality learning for visible-infrared person re-identification.  In Proceedings of the ACM MM, pages 788–796, 2021.
  * [50] Zhong Zhang, Haijia Zhang, and Shuang Liu.  Person re-identification using heterogeneous local graph attention networks.  In Proceedings of the CVPR, pages 12136–12145, 2021.
  * [51] Zhiwei Zhao, Bin Liu, Qi Chu, Yan Lu, and Nenghai Yu.  Joint color-irrelevant consistency learning and identity-aware modality adaptation for visible-infrared cross modality person re-identification.  In Proceedings of the AAAI, pages 3520–3528, 2021.
  * [52] Yi Zheng, Shixiang Tang, Guolong Teng, Yixiao Ge, Kaijian Liu, Jing Qin, Donglian Qi, and Dapeng Chen.  Online pseudo label generation by hierarchical cluster dynamics for adaptive person re-identification.  In Proceedings of the CVPR, pages 8371–8381, 2021.
  * [53] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In Proceedings of the AAAI, pages 13001–13008, 2020.
  * [54] Sanping Zhou, Fei Wang, Zeyi Huang, and Jinjun Wang.  Discriminative feature learning with consistent attention regularization for person re-identification.  In Proceedings of the ICCV, pages 8040–8049, 2019.
  * [55] Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai.  Asymmetric non-local neural networks for semantic segmentation.  In Proceedings of the ICCV, pages 593–602, 2019.


# SplitAMC: Split Learning for
Robust Automatic Modulation Classification

Jihoon Park  Seungeun Oh  Seong-Lyun Kim<EMAIL_ADDRESS>
###### Abstract

Automatic modulation classification (AMC) is a technology that identifies a
modulation scheme without prior signal information and plays a vital role in
various applications, including cognitive radio and link adaptation. With the
development of deep learning (DL), DL-based AMC methods have emerged, while
most of them focus on reducing computational complexity in a centralized
structure. This centralized learning-based AMC (CentAMC) violates data privacy
in the aspect of direct transmission of client-side raw data. Federated
learning-based AMC (FedeAMC) can bypass this issue by exchanging model
parameters, but causes large resultant latency and client-side computational
load. Moreover, both CentAMC and FedeAMC are vulnerable to large-scale noise
occured in the wireless channel between the client and the server. To this
end, we develop a novel AMC method based on a split learning (SL) framework,
coined SplitAMC, that can achieve high accuracy even in poor channel
conditions, while guaranteeing data privacy and low latency. In SplitAMC, each
client can benefit from data privacy leakage by exchanging smashed data and
its gradient instead of raw data, and has robustness to noise with the help of
high scale of smashed data. Numerical evaluations validate that SplitAMC
outperforms CentAMC and FedeAMC in terms of accuracy for all SNRs as well as
latency.

###### Index Terms:

Automatic modulation classification (AMC), split learning, federated learning,
noise robustness, latency

##  I Introduction

With the remarkable development of wireless communication, understanding the
radio spectrum plays an essential role in various applications, such as
cognitive radio and link adaptation [1, 2]. In this respect, automatic
modulation classification (AMC) is emerging as a promising technology in a way
that the receiver (Rx) identifies the modulation scheme of the corresponding
transmitter (Tx) signal without prior information [3]. As the first of its
kind, a traditional AMC method aims to achieve high detection probability
through the likelihood function design [4]. However, due to its sensitivity to
signal errors, it suffers from performance degradation and computational
complexity, especially in the wireless environment where channel state
information (CSI) is not given and channel gain fluctuates.

Figure 1: An overview of communication system composed of Tx, Rx with AMC, and
server.

The feature-based AMC method can detour these problems by extracting hand-
crafted features from the received signal and performing classification tasks.
Going further, DL-based AMC can solve the computational complexity problem of
likelihood-based AMC under a time-varying channel with deep learning (DL)
framework [5]. The existing DL-based AMC method is often rooted in a
centralized architecture that enjoys dispersed data among multiple clients by
direct local data aggregation on the server [6]. However, such centralized
learning-based AMC (CentAMC) violates data privacy leakage while causing
significant communication bottlenecks on the server-side. In order to
guarantee data privacy, several AMC works apply a distributed learning
framework to the AMC method. As its representative, FedeAMC [7], which
combines federated learning (FL) with AMC, performs modulation classification
by updating the local model through aggregation and redistribution of the
model parameters between the server and clients. However, this model exchange
incurs a huge communication overhead, so it is ill-suited for the large-sized
model. _In addition, CentAMC and FedeAMC tend to be vulnerable to large-scale
noise, leading to accuracy drop at low signal-to-noise ratio (SNR),
highlighting the need for alternatives (see Table II)._

Towards a noise-robust, communication-efficient, and data-private AMC, this
paper proposes a novel AMC method based on a split learning (SL) framework
called SplitAMC. The SplitAMC divides the entire deep neural network (DNN)
into two partitions depending on cut-layer, an upper model segment and a lower
model segment, each stored by the server and clients, respectively. Under this
model-split architecture, clients and the server communicate the cut-layer
representations, so-called smashed data, and its corresponding gradients. In
doing so, the SplitAMC can benefit from high accuracy over large-scale noise,
thanks to the scale of smashed data shown in Fig. 3. Also, exchanging smashed
data instead of model parameters or raw data results in improved communication
efficiency as well as data privacy guarantee of SplitAMC.

The contributions of this paper are summarized below:

  * •

By revisiting the SL framework [8], we propose SplitAMC, an AMC method based
on SL, that enables smashed data exchange instead of model parameters or raw
data exchange.

  * •

SplitAMC’s smashed data exchange solves data privacy leakage that occurs in
CentAMC, while reducing latency at the same time. Latency analysis for
SplitAMC is available in Sec. IV.

  * •

Thanks to the large scale of the smashed data, SplitAMC has robust accuracy
even in a large-noise environment, which is proven by experiments.

The rest of the paper is organized as follows: Sec. II introduces a system
model, including a single-carrier system-based signal model and a wireless
channel model. Next, Sec. III describes the operation of our proposed SplitAMC
in detail. In Sec. IV, the performance of SplitAMC is validated and analyzed
through extensive simulations, compared to CentAMC as well as FedeAMC.

##  II System Model

In this section, we describe the network topology in which the DL-based
approaches including the proposed SplitAMC run, and then sequentially describe
the signal model and channel model of communication links included in the
network.

###  II-A Network Topology

As described in Fig. 1, we consider a communication system for AMC consisting
of Tx-Rx pairs and server.

1) Tx-Rx Communication Link: Each Tx passes the source signal through the
sampler and quantizer to perform analog-to-digital conversion (ADC) and
encodes it, followed by selecting the modulation type, controlling the
transmit power, and transmitting it to the corresponding Rx. Then, each Rx
infers the modulation type of Tx through a modulation classifier and
demodulates it based on it. Consequently, the goal of this paper is to train a
modulation classifier with high accuracy, the key to the aforementioned
communication system, via DL-based approach.

2) Rx-Server Communication Link: To achieve this goal, DL-based approaches
perform additional communication between Rx and a server to train the model.
At this time, an analog-modulated signal is assumed to be used. Although
digital communication is widely used, it has notable drawbacks such as
significant performance drop when deviating from target channel conditions
(i.e., outage due to frequent channel fluctuations). On the other hand, analog
communication can avoid this and even improve performance thanks to the
regularizer effect of the noise [9]. Furthermore, analog-modulated signals can
enjoy enhanced data privacy as well as latency gains by enabling over-the-air
computation (AirComp) [10].

###  II-B Communication Model

####  II-B1 Signal Model

For Tx-Rx communication link, we use a regular signal model of the unknown
single-input single-output single-carrier systems as in [11, 12]. Let $r(n)$
represent the unknown modulated signals at the Rx, denoted by:

$\begin{split}&r(n)=A_{n}e^{j(2\pi f_{0}nT+\theta_{n})}s(n)+\sigma(n),\\\
&n\in\\{0,1,\cdots,N-1\\},\end{split}$ (1)

where $s(n)$ and $\sigma(n)$ are the transmitted modulation signal and
additive white Gaussian noise (AWGN), respectively. In addition, $A_{n}$ is
the signal amplitude of symbol $n$, following rayleigh channel fading, $f_{0}$
is the carrier frequency offset, $\theta_{n}$ is the time-varying carrier
phase offset of symbol $n$, $T$ is the symbol interval, and $N$ is the number
of symbols of the signals and also the number of samples to be used for
training of the DL model.

The IQ sample, which is the training and test sample of DL, consists of in-
phase ($I$) and quadrature ($Q$) parts of $r(t)$. Based on the received signal
model, the IQ sample can be expressed as:

$\begin{split}&I=\\{Real(r(0)),Real(r(1)),\cdots\,Real(r(N-1))\\},\\\
&Q=\\{Imag(r(0)),Imag(r(1)),\cdots\,Imag(r(N-1))\\},\end{split}$ (2)

where $Real(\cdot)$ and $Imag(\cdot)$ are functions for extracting values
corresponding to the real and imaginary parts of the received signal $r(t)$.

Also, we can define the SNR of the received signal as follows:

$\begin{split}\gamma_{data}=10\cdot\log_{10}(\frac{\sum_{n=0}^{N-1}{|A_{n}s(n)|}^{2}}{\sum_{n=0}^{N-1}{|\sigma(n)|}^{2}})[dB].\end{split}$
(3)

Figure 2: Graphical illustrations of a) centralized learning, b) federated
learning (FL), and c) split learning (SL).

####  II-B2 Channel Model

For Rx-server communication link, we consider the path-loss attenuation,
channel fading, and noise, and its SNR is expressed by the following equation:

$\gamma=10\cdot\log_{10}(\frac{h\cdot P\cdot d^{-\alpha}}{\sigma^{2}})[dB],$
(4)

where $P$ is the transmit power, $d$ is the distance between Tx and Rx,
$\alpha$ is the path loss attenuation exponent, which usually corresponds to a
value greater than or equal to 2, and $\sigma^{2}$ is the variance value of
the AWGN in the wireless channel environment. Moreover, $h$ indicates channel
fluctuations and follows an exponential distribution with a mean of 1
($h\sim{\exp{(1)}}$).

##  III Split learning-based AMC

This section introduces the operation of SplitAMC method along with two
existing AMC methods, CentAMC and FedeAMC. Let $m$ be the subscript for Rx.
Then, $m$-th Rx of set $\mathbb{M}=\\{1,2,\cdots,M\\}$ produces the IQ data
samples via (2). For all $m\in\mathbb{M}$, the local dataset
$D_{m}=(x_{m},y_{m})$ consists of IQ samples $x_{m}$ and its corresponding
ground-truth labels $y_{m}$. As depicted in Fig. 2c, to enable model-split
architecture shown in [8], the entire model weight $w_{m}$ is divided into the
upper model segment $w_{s}$ and the lower model segment $w_{c,m}$ based on the
cut-layer.

###  III-A Training Phase: The Operation of SplitAMC

Client-side Forward Propagation (FP). The $m$-th Rx randomly selects $B$ data-
label tuples from the local dataset $D_{m}$ to compose a batch, and passes it
through the lower model segment $w_{c,m}$ to generate smashed data $s_{m}$ as
follows:

$s_{m}=f(x_{m};w_{c,m}),$ (5)

where $f$($\cdot$) denotes a function that maps $x_{m}$ to $s_{m}$, determined
by $w_{c,m}$. Then, the $m$-th Rx sends $s_{m}$ to the server through an
analog-modulated signal. $\bar{s}_{m}$ reflecting the fluctuation and noise of
the uplink (UL) wireless channel between Rx and the server in $s_{m}$ is
expressed as belows:

$\bar{s}_{m}=h\cdot s_{m}+\mathbb{N}=h\cdot f(x;w_{c,m})+\mathbb{N},$ (6)

where $\mathbb{N}$ follows a zero-mean complex Gaussian distribution with
variance $\sigma^{2}$.

Server-side FP & Backpropagation (BP). To update the server-side model, the
smashed data with Gaussian noise $\bar{s}_{m}$ becomes the input of the upper
model segment located on the server, yielding a softmax output $\hat{y}_{m}$
as follows:

$\hat{y}_{m}=g(\bar{s}_{m};w_{s}),$ (7)

where $g$($\cdot$) is a function that maps $\bar{s}_{m}$ to $\hat{y}_{m}$ due
to the upper model segment $w_{s}$.

Then, by using the cross-entropy function, the loss $L_{CE}$ can calculated
by:

$L_{CE}=-\frac{1}{d_{y}}\sum_{i=1}^{d_{y}}y_{m,i}\log(\hat{y}_{m,i}),$ (8)

where $i$ is the subscript for element, so that $y_{m,i}$ and $\hat{y}_{m,i}$
are $i$-th elements of ground-truth label $y_{m}$ and prediction
$\hat{y}_{m}$, respectively, while $d_{y}$ denotes the dimension of $y_{m,i}$.

With the aid of downlink (DL) communication in the cut-layer, BP is available
allowing model update of Rx and server as follows:

$\left[\begin{matrix}w_{c,m}\\\
w_{s}\end{matrix}\right]\leftarrow\left[\begin{matrix}w_{c,m}\\\
w_{s}\end{matrix}\right]-\eta\left[\begin{matrix}\nabla_{w_{c,m}}L_{CE}\\\
\nabla_{w_{s}}L_{CE}\end{matrix}\right],$ (9)

where $\eta$ and $\nabla_{w_{c,m}(w_{s})}$ is the learning rate and the
partial derivative with respect to $w_{c,m}(w_{s})$, respectively.

After that, the $m$-th Rx transmits its lower model segment to the $(m+1)$-th
Rx as follows:

$w_{c,m+1}\leftarrow{w_{c,m}},$ (10)

completing the single communication round of SplitAMC. The training operation
of SplitAMC is detailed in the pseudocode of Algorithm 1.

###  III-B Inference Phase: Performance Metrics

When the model $w_{m}=[w_{c,m},w_{s}]$ converges after the $K$ communication
rounds, it enters the inference phase. To this end, two types of inference
method can be considered. In the first case, all clients download the shared
upper model segment of the server, then use it as a classifier for the
modulated signal transmitted from the paired Tx, followed by demodulation. In
other cases, we can consider an inference method in which the client and
server communicate the smashed data of test data and prediction through UL and
DL communication, respectively, while retaining the model. In both inference
methods, it is possible to measure the performance of the test model $w_{m}$
via the following performance metrics:

####  III-B1 Classification Performance

The correct classification probability $P_{cc}$ is employed to evaluate how
accurately the test model $w_{m}$ classifies the modulation scheme. When the
total number of test samples is $N_{test}$, $P_{cc}$ is defined by :

$P_{cc}=\frac{N_{correct}}{N_{test}}\times 100[\%],$ (11)

where $N_{correct}$ is the number of samples that successfully classified the
modulation scheme among all $N_{test}$ test samples.

Algorithm 1 Model training method of SplitAMC framework.

1:requirements: $k=0$, $w=[w_{c,m},w_{s}]$, $D_{m}$ $\forall m$

2:Hyperparameters:

  * •

number of clients: $|\mathbb{M}|=M$,

  * •

total number of communication rounds: $K$,

  * •

learning rate: $\eta$

3:while $k<K$ do

4: Client $m\in\mathbb{M}$:

5: produces $s_{m}$ via (5) _$\triangleright$ Client-side FP_

6: unicasts $(s_{m},y_{m})$ to the server _$\triangleright$ UL Communication_

7: Server:

8: produces $\hat{y}_{m}$ via (7) _$\triangleright$ Server-side FP_

9: produces $L_{CE}$ via (8)

10: updates $w_{s}$ via (9) _$\triangleright$ Server-side BP_

11: unicasts $\nabla_{\bar{s}_{m}}L_{CE}$ _$\triangleright$ DL Communication_

12: Client $m\in\mathbb{M}$:

13: updates $w_{c,m}$ via (9) _$\triangleright$ Client-side BP_

14: unicasts $w_{c,m}$ to $(m+1)$-th Rx as in (10) _$\triangleright$ Model
Transfer_

15: $k\leftarrow{k+1}$

16:end while

####  III-B2 Latency Model

Meanwhile, we can model the latency that occurs in the second inference method
mentioned above or training phase. We divide the overall latency into
communication latency $T_{comm}$, consisting of UL latency $T_{UL}$ & DL
latency $T_{DL}$, and computation latency $T_{comp}$. Here, we assume a static
channel condition, i.e., $h=1$ in (4) for convenience.

In terms of communication latency, it is proportional to the number of
transmitted bits and inversely proportional to the channel capacity, under the
assumption of a static channel. Let $L_{a}^{b}$ and $\beta_{a}^{b}$
($a\in{\\{SL,CL,FL}\\}$, $b\in{\\{UL,DL}\\}$) denote the number of parameters
exchanged and the number of bits per parameter during $b$ communication
between Rx and server in $a$-based AMC method, respectively. Then, for the UL
(DL) transmission rate $R_{UL(DL)}=BW\cdot log_{2}({1+\gamma})$ with $\gamma$
in (4) where $h=1$ and channel bandwidth $BW$, the latency for UL (DL)
communication is $\frac{L_{a}^{UL(DL)}\cdot{\beta_{a}^{UL(DL)}}}{R_{UL(DL)}}$
for all $a$ method.

Computational latency $T_{comp}$ is classified into client-side latency
$(T_{client})$ and server-side latency $(T_{server})$. Both latencies are
expressed as the ratio of $C$ to $f$, where $C$ is the number of CPU cycles
for processing FP as well as BP, and $f$ is for client-side or server-side
computational capacity (in CPU cycles$/$s). Assuming that the server-side
computational capacity is infinite, $T_{client}$ and $T_{server}$ become unit
computing time $\tau_{comp}$ and 0, respectively, during 1 communication
round.

Table I summarizes $T_{comm}$ and $T_{comp}$ for 1 communication round of
SplitAMC as well as FedeAMC and CentAMC. Note that the proposed SplitAMC can
benefit in terms of computational latency, multiplied by the
$\lambda\in[0,1]$, depending on where the cut-layer is located.

###  III-C Other AMC Methods

####  III-C1 CentAMC [6]

As described in Fig. 2a, in CentAMC, all local datasets $D_{m}$ are aggregated
on the server through a wireless channel to form a single global dataset,
becoming an input for a global model training of the server. By doing this,
CentAMC easily obtains data diversity gain, but it causes a communication
bottleneck when sending raw samples and violates data privacy guarantees.

####  III-C2 FedeAMC [7]

In FedeAMC, the server distributes the global model to each local client
$m\in\mathbb{M}$. Then, all local clients train the model using local dataset
$D_{m}$. After that, as shown in Fig. 2b, local model parameters $w_{m}$ are
transmitted to the server through the wireless channel link. The server takes
weighted averaging of aggregated local parameters, yielding a global model
$w$, i.e., $w=(\sum_{m=1}^{M}{|D_{m}|\cdot{w_{m}}})/\sum_{m=1}^{M}{|D_{m}|}$.
This approach can take data diversity gain without direct data exchange, but
large communication overhead occurs for large-sized models.

TABLE I: A summary of communication and computation latency for a single communication round of AMC methods. Latency | $T_{comm}$ | $T_{comp}$
---|---|---
$T_{UL}$ | $T_{DL}$ | $T_{client}$
SplitAMC | $\frac{L_{SL}^{UL}\cdot\beta_{SL}^{UL}}{R_{UL}}$ | $\frac{L_{SL}^{DL}\cdot\beta_{SL}^{DL}}{R_{DL}}$ | $\lambda\cdot\tau_{comp}$
FedeAMC | $\frac{L_{FL}^{UL}\cdot\beta_{FL}^{UL}}{R_{UL}}$ | $\frac{L_{FL}^{DL}\cdot\beta_{FL}^{DL}}{R_{DL}}$ | $\tau_{comp}$
CentAMC | $\frac{L_{CL}^{UL}\cdot\beta_{CL}^{UL}}{R_{UL}}$ | $\frac{L_{CL}^{DL}\cdot\beta_{CL}^{DL}}{R_{DL}}$ | -
TABLE II: Classification performance $P_{cc}$ of SplitAMC and comparison groups on datasets with different SNRs $\gamma_{data}$. Channel SNR |  $\gamma_{data}=$ 5dB | 10dB | 15dB
---|---|---|---
Avg. SNR ($=\gamma_{avg}$ ) | Fixed SNR ($=\gamma_{fix}$) | Avg. SNR | Fixed SNR | Avg. SNR | Fixed SNR
-10dB | +10dB | -10dB | +10dB | -10dB | +10dB | -10dB | +10dB | -10dB | +10dB | -10dB | +10dB
SplitAMC(1,3) | 81.2 % | 82.3 % | 81.7 % | 82.0 % | 97.8 % | 98.6 % | 98.6 % | 99.0 % | 99.6 % | 99.9 % | 99.7 % | 99.9 %
SplitAMC(2,2) | 81.1 % | 81.1 % | 80.8 % | 82.5 % | 97.7 % | 98.9 % | 98.6 % | 99.0 % | 99.6 % | 99.9 % | 99.8 % | 99.9 %
CentAMC | 66.6 % | 67.5 % | 68.4 % | 68.7 % | 77.3 % | 78.4 % | 77.9 % | 78.7 % | 97.4 % | 97.5 % | 97.7 % | 97.8 %
FedeAMC | 64.7 % | 64.4 % | 64.4 % | 65.1 % | 79.4 % | 79.2 % | 79.1 % | 78.6 % | 93.0 % | 93.5 % | 92.9 % | 93.4 %

##  IV Experimental Results

This section evaluates SplitAMC’s performance in terms of accuracy,
convergence speed, and latency. For comparison, we use the aforementioned
CentAMC and FedeAMC, and also SplitAMC with different cut-layer location. For
the ResNet-18 model [13] with 4 residual blocks, $\text{SplitAMC}_{(1,3)}$ and
$\text{SplitAMC}_{(2,2)}$ represent the SplitAMC framework when the cut-layers
are located after the 1st and 2nd blocks, respectively.

For signal generation between Tx and Rx, Communications Toolbox in MATLAB is
adopted. Considering the modulation schemes of QPSK, 16-QAM, and 64-QAM when
SNR $\gamma_{data}$ is 5, 10, and 15 dB, each modulation scheme includes 5000
data symbols, each composed of 1000 IQ samples. Prior to passing the CNN
model, each IQ sample is transformed into a constellation image and resized to
64$\times$64 dimensions as in [14, 13]. For experiment, we employ Intel
i7-10700 CPU and GTX 3080Ti GPU, where the software environment is based on
PyTorch v1.10.0 and CUDA v11.3 with Python v3.9.4. Other simulation parameters
are given as: $M=2$, batch size $B=64$, total communication rounds $K=50000$
steps, learning rate $\eta=0.004$, transmit power $P=100$ mW, distance between
Rx-server $d=100$ m, and path-loss exponent $\alpha=2$.

###  IV-A Accuracy & Convergence Speed

Table II shows the classification accuracy of AMC methods according to
different $\gamma_{data}$ and channel environments. We consider two wireless
communication environments depending on whether there is channel fluctuation,
where the averaged SNR with channel fluctuation $\gamma_{avg}$ and the fixed
SNR without channel fluctuation $\gamma_{fix}$ are the same as $\gamma$ with
$h=1$ in (4). First, in all cases except for a few cases when $\gamma_{data}$
is 10dB, the classification accuracy is high in the order of SplitAMC,
CentAMC, and FedeAMC, showing the noise-robustness of SplitAMC. This is rooted
in the scale difference of parameters exchanged through the Rx-server link in
SplitAMC and FedeAMC shown in Fig. 3, respectively. Considering also CentAMC,
which uploads raw data with mean and variance normalized to 0.5, when noise of
the same size is injected, the relatively large parameter scale of SplitAMC
leads to small performance degradation. These parameter scale gaps can be
compensated by increasing the transmit power (e.g., 200 times the transmit
power [mW] for 200 times scale difference) which is unachievable given the
client’s power constraints. Fig. 3 also implies a faster convergence speed of
SplitAMC compared to FedeAMC through the parameter distribution change
according to the number of steps, which is confirmed from the learning curve
of Fig. 4.

Returning to Table II, the classification performance tends to deteriorate as
$\gamma_{data}$ or $\gamma_{avg}(\gamma_{fix})$ decreases and when channel
fluctuations exist. $\gamma_{data}$ and $\gamma_{avg}(\gamma_{fix})$ are
respectively related to the noise inherent in the IQ data itself and the noise
applied to the resized constellation image, and it is confirmed that the
performance reduction for $\gamma_{data}$ is noticeable among them, indicating
the importance of the Tx-Rx communication link state. The existence of channel
fluctuations reduces the accuracy since it is difficult to learn a generalized
model in the training phase. Focusing on the gap in classification accuracy
between AMC methods, the worse the communication channel conditions, the
greater the difference, proving the effectiveness of SplitAMC. In addition, in
Table II as well as in Fig. 4, there is no significant difference in
performance in terms of accuracy or convergence speed for cut-layer in
SplitAMC.

(a) The CDF of $s_{m}$.

(b) The CDF of $w_{m}$.

Figure 3: Cumulative distributions of parameter values in SplitAMC and FedeAMC
for different steps.

(a) Learning curves when $\gamma_{data}=5$ dB.

(b) Learning curves when $\gamma_{data}=15$ dB.

Figure 4: Learning curves of AMC methods w.r.t different $\gamma_{data}$.

###  IV-B Latency Measurement for Training

Fig. 5 compares latency between AMC methods according to the ratio of
$\tau_{comp}$ and $\tau_{comm}$, which is the reciprocal of uplink rate
$R_{UL}$. Here, to reflect the UL-DL asymmetric capacity of the cellular
network, it is assumed that $R_{DL}=10R_{UL}$. Other parameters for
calculating latency are as follows: $\beta_{a}^{b}=32$ bits for all $a$ & $b$,
$\gamma=10$ dB for UL communication, $BW=10$ MHz, and $50$ total communication
rounds.

The first thing to note is that overall latency performance improves when the
ratio is 10:1, i.e., when $\tau_{comm}$ is small. In other words, this means
that the communication payload size of UL, which determines UL latency, is the
main factor influencing latency. Thanks to the low dimension of smashed data,
SplitAMC outperforms the baselines, except for CentAMC at 10:1. In the case of
CentAMC, $T_{comp}$ converges to 0 under our premise that the server-side
computational capacity is infinite, and this dramatically reduces overall
latency when $\tau_{comp}:\tau_{comm}$ is 10:1. Regarding the cut-layer, the
closer the cut-layer is to the input layer, the better in terms of latency.
This is because the client-side computational latency varies according to
$\lambda$ as the cut-layer location changes, although there is no huge gap in
the dimension of activation. When considering its accuracy and convergence
speed together, it is optimal to place the cut-layer close to the input layer
in SplitAMC, even beneficial in terms of client memory. Note that the above
method is also applicable for the inference phase, by replacing the payload
size.

##  V Conclusion

In this paper, we revisited the SL framework and applied it to AMC to design
SplitAMC framework. Unlike the existing AMC methods, in SplitAMC, the client
and server each hold a fraction of the entire model and exchange the output
and gradient of the cut-layer, thereby resulting in a small memory size,
computational cost, and communication payload size. Moreover, with the help of
large scale inherent in smashed data, SplitAMC guarantees noise-robustness
that can achieve high accuracy even when noise with large variance is ejected.
Numerical evaluations validate the accuracy, convergence speed, and latency of
SplitAMC. It is worth noting that this work focused on connecting parametric
properties in distributed learning (i.e., scale and dimension) to metrics in
wireless communication (i.e., classification accuracy and latency). As a
future work, we will exploit the classification performance as the number of
clients increases as in [15]. Also, investigating the accuracy-privacy
tradeoff for noise variance [16] in SplitAMC could be an interesting topic,
deferred as future work.

## Acknowledgement

This work was supported by Institute of Information & communications
Technology Planning & Evaluation(IITP) grant funded by the Korea
government(MSIT) (No.2022-0-00420,Development of Core Technologies enabling 6G
End-to-End On-Time Networking & No.2021-0-00270, Development of 5G MEC
framework to improve food factory productivity, automate and optimize flexible
packaging)

Figure 5: Latency according to the ratio of $\tau_{comp}$ to $\tau_{comm}$.

## References

  * [1] L. Liu, Y. Zhou, W. Zhuang, J. Yuan, and L. Tian, “Tractable coverage analysis for hexagonal macrocell-based heterogeneous udns with adaptive interference-aware comp,” IEEE transactions on wireless communications, vol. 18, no. 1, pp. 503–517, 2018.
  * [2] S. Luan, Y. Gao, T. Liu, J. Li, and Z. Zhang, “Automatic modulation classification: Cauchy-score-function-based cyclic correlation spectrum and fc-mlp under mixed noise and fading channels,” Digital Signal Processing, vol. 126, p. 103476, 2022.
  * [3] L. Liu, Y. Zhou, J. Yuan, W. Zhuang, and Y. Wang, “Economically optimal ms association for multimedia content delivery in cache-enabled heterogeneous cloud radio access networks,” IEEE Journal on Selected Areas in Communications, vol. 37, no. 7, pp. 1584–1593, 2019.
  * [4] F. Hameed, O. A. Dobre, and D. C. Popescu, “On the likelihood-based approach to modulation classification,” IEEE transactions on wireless communications, vol. 8, no. 12, pp. 5884–5892, 2009.
  * [5] T. Huynh-The et al., “Automatic modulation classification: A deep architecture survey,” IEEE Access, vol. 9, pp. 142950–142971, 2021.
  * [6] Y. Wang, J. Yang, M. Liu, and G. Gui, “Lightamc: Lightweight automatic modulation classification via deep learning and compressive sensing,” IEEE Transactions on Vehicular Technology, vol. 69, no. 3, pp. 3491–3495, 2020\.
  * [7] Y. Wang et al., “Federated learning for automatic modulation classification under class imbalance and varying noise condition,” IEEE Transactions on Cognitive Communications and Networking, vol. 8, no. 1, pp. 86–96, 2021.
  * [8] O. Gupta and R. Raskar, “Secure training of multi-party deep neural network,” Aug. 25 2020.  US Patent 10,755,172.
  * [9] X. Zhou et al., “Learning with noisy labels via sparse regularization,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 72–81, 2021.
  * [10] Y. Koda, J. Park, M. Bennis, P. Vepakomma, and R. Raskar, “Airmixml: Over-the-air data mixup for inherently privacy-preserving edge machine learning,” in 2021 IEEE Global Communications Conference (GLOBECOM), pp. 01–06, IEEE, 2021.
  * [11] B. Dong et al., “A lightweight decentralized learning-based automatic modulation classification method for resource-constrained edge devices,” IEEE Internet of Things Journal, 2022.
  * [12] S. Zheng, P. Qi, S. Chen, and X. Yang, “Fusion methods for cnn-based automatic modulation classification,” IEEE Access, vol. 7, 2019.
  * [13] Y. Lin, Y. Tu, Z. Dou, and Z. Wu, “The application of deep learning in communication signal modulation recognition,” in IEEE/CIC International Conference on Communications in China (ICCC), pp. 1–5, 2017.
  * [14] S. Peng et al., “Modulation classification based on signal constellation diagrams and deep learning,” IEEE transactions on neural networks and learning systems, vol. 30, no. 3, pp. 718–727, 2018.
  * [15] S. Oh et al., “Locfedmix-sl: Localize, federate, and mix for improved scalability, convergence, and latency in split learning,” in Proceedings of the ACM Web Conference 2022, pp. 3347–3357, 2022.
  * [16] S. Oh et al., “Differentially private cutmix for split learning with vision transformer,” in First Workshop on Interpolation Regularizers and Beyond at NeurIPS 2022, 2022.