# Collaborative Deep Learning at the Wireless Edge

Wing Fei Lo1, Nitish Mital2, Deniz Gunduz3

###### Abstract

This project studies the problem of multi-source collaborative person re-
identification at the wireless edge, where multiple edge devices capture
images of the same identity from different angles and locations, which are
then used jointly to retrieve similar images in the edge server. Based on the
previous success of the autoencoder-based joint source and channel coding
(JSCC) model, this project extends the work done by Jankowkski, Gündüz and
Mikolajczyk (2021) [jscc_ae] to a two-source collaborative scenario. Here we
first propose a pre-processed dataset specifically for the two-source person
re-identification problem. Then, we propose two novel deep learning-based JSCC
models for the task of two-source person re-identification at the wireless
edge on both additive white Gaussian noise (AWGN) and Rayleigh slow fading
channels, with the aim of maximizing the retrieval accuracy under given total
bandwidth constraints. The proposed two-source models are evaluated
extensively on a wide range of channel signal-to-noise ratios (SNR) and total
channel bandwidth constraints, and they are shown to both outperform the
single-source model [jscc_ae] significantly under different channel
conditions. We also propose two novel SNR-aware JSCC models with attention
modules to minimize the degradation of performance during channel mismatch
between training and testing, which are shown experimentally to outperform the
original (non-SNR-aware) two-source models during channel mismatch over a wide
range of testing SNRs.

###### Index Terms:

Deep learning, Internet of Things, person re-identification, joint source and
channel coding, image retrieval

##  I Introduction

In recent years, deep learning tasks at the wireless edge have been studied
extensively in the literature, in particular distributed inference problems
over a wireless channel where Internet-of-Things (IoT) devices are trained
centrally to perform inference over-the-air. Out of the deep learning tasks in
which remote inference is essential, image retrieval is useful in many
practical applications like surveillance and autonomous driving. This project
is going to investigate the problem of multi-source collaborative image
retrieval at the wireless edge, where multiple edge devices (e.g. surveillance
cameras in lamp posts) capture images for the same object from different
angles at different locations, which are then used to identify similar images
in a large gallery database from an edge server. In particular, the task of
person re-identification is studied. The problem of two-source collaborative
person re-identification at the wireless edge is illustrated in Figure 1. It
is emphasized that remote inference is necessary and the edge devices cannot
perform the retrieval task locally even if they have advanced computational
capabilities, as the gallery database is only available in the edge server.
Also, due to latency constraints, sending the whole image via the wireless
channel is impossible. It is assumed that learning-based feature extraction is
done at the edge and features of the input images are sent to the edge server
through a wireless channel.

Figure 1: Illustration of two-source collaborative person re-identification at
the wireless edge

Two approaches are possible for the specified task, namely the digital
separation-based scheme and the analog joint source and channel coding (JSCC)
scheme. While Shannon’s separation theorem [shannon] states that separating
source and channel coding in the digital scheme can achieve asymptotic
optimality, this theorem breaks down for finite block lengths in real-life
applications. Moreover, an autoencoder-based JSCC scheme proposed by Jankowski
et al. [jscc_ae] is found to outperform the corresponding digital scheme under
all channel conditions. Hence, in this project, we will extend the work done
by Jankowski et al. [jscc_ae] to a two-source collaborative person re-
identification problem at the wireless edge where two edge devices are present
to capture images of the same person from two different views. With one more
view of the same person from the extra edge device, it is expected that the
retrieval accuracy can be further improved given the same amount of
communication resources through incorporating the information from the two
different views to make a joint decision at the edge server.

###  I-A Motivation and Objectives

While the previous study [jscc_ae] shows that the autoencoder-based JSCC
scheme has achieved higher accuracy than conventional digital schemes, and the
performance of the model can be greatly improved by jointly designing the
communication scheme and inference algorithms through end-to-end training, it
has only considered the single source scenario. However, in many real-life
applications like surveillance camera systems, rather than having only one
edge device, multiple edge devices (e.g. surveillance cameras) are present in
the same scene which can capture the same person from different views and
angles at similar timestamps. This shows the higher practical applicability of
the multi-source image retrieval problem. However, currently the collaboration
between these edge devices to make joint inference decisions over-the-air have
not been studied much in the literature. As different camera views can capture
different characteristics of a person (for example, the appearances of the
front and the back of the same person can be very different), more information
about the identity-of-interest can be extracted through the collaboration of
multiple edge devices. Hence, with more information available, the features
extracted from the inputs will be more robust to different camera views and
this may potentially increase the retrieval accuracy of the model. Due to the
potential benefits from incorporating more views from different edge devices
and its high applicability in real-life scenarios, the aim of this project is
set to extend the existing autoencoder-based JSCC scheme [jscc_ae] for the
specified person re-identification problem to a two-source collaborative
scenario to further improve the performance of the model.

In this project, the major objective is to develop a deep learning-based JSCC
model for the two-source collaborative wireless person re-identification
problem, which maximizes the retrieval accuracy under a given constraint on
the communication resources over the wireless link. The inputs to the model
will be two images containing the same identity from two different camera
views. Note that it is assumed that there is no communication between the two
edge devices before the transmission of the channel symbols via the wireless
channel, which means that two transmitters, each corresponding to one edge
device, must exist. The total channel bandwidth available is constrained to be
$B$ complex symbols (i.e. $2B$ real symbols), but how the communication
resources are allocated among the two transmitters is not fixed. For a more
comprehensive evaluation, the proposed models will be tested on various
channel conditions, including a wide range of channel signal-to-noise ratios
(SNR) and total channel bandwidth constraints. They are also tested on both
the AWGN channel and Rayleigh slow fading channel. In addition, as the channel
conditions during training and testing may not be the same in real-life
applications, the performances of the models are also evaluated when the
training and testing channel SNRs do not match.

###  I-B Contributions

Numerous novel contributions to the field of multi-source collaborative person
re-identification at the wireless edge have been made during this project. The
technical contributions of this project are summarized as follows:

  1. 1.

We propose a pre-processed dataset specifically for the problem of two-source
collaborative person re-identification at the wireless edge.

  2. 2.

We propose two novel deep learning-based JSCC models for the problem of two-
source collaborative person re-identification at the wireless edge, on both
the AWGN channel and Rayleigh slow fading channel with and without channel
state information (CSI). The two models have different allocation schemes of
the available communication resources between the two sources.

  3. 3.

We perform extensive evaluation of the two proposed two-source models on a
wide range of channel SNRs and total bandwidth constraints, and show that both
of them outperforms the JSCC AE model for the single-source scenario in
[jscc_ae] on both AWGN channel and slow fading channel under a wide range of
channel conditions. The relative performances of the two proposed models are
analyzed with further experimental validation using the notion of squared
cosine similarity.

  4. 4.

We evaluate the two proposed two-source models when the training and testing
channel SNRs do not match and show that both models exhibit graceful
degradation during channel mismatch. To further minimize the degradation in
model performance during channel mismatch, we propose two novel SNR-aware deep
learning-based JSCC models with the use of soft-attention mechanism, which are
the modified versions of the two-source models above. The two proposed SNR-
aware models are evaluated extensively on a wide range of testing channel SNRs
and they are shown to outperform the proposed non-SNR-aware two-source models
mentioned above during channel mismatch, on both AWGN and slow fading
channels, making the SNR-aware models more robust towards channel mismatch
between training and testing.

##  II Related work

###  II-A Person re-identification

The goal of person re-identification is to improve the quality of identity
recognition, particularly in surveillance applications. During evaluation
stage, given a query image, its similarities with images from different
viewpoints in the gallery database are assessed through a given metric.
Through performing the nearest neighbours search, a rank list which sorts the
gallery images according to descending similarity with the given query image
will be generated. By checking if the gallery images on the rank list contain
the same identity as of the query image, the performance of the network can be
evaluated through the top-1 retrieval accuracy [cmc]. The algorithm for
retrieval tasks should work on any person, without limiting only on the
identities present in the training set. Hence, the training and testing sets
for the problem should ideally be disjoint. The testing set is further divided
into the query set and the gallery set.

The task of person re-identification has been studied extensively in recent
years [personreid]. In particular, the deep learning methods based on
convolutional neural networks (CNN) have been the most successful. Recent
development of the person re-identification algorithms includes the use of
attention mechanisms [attention], and a soft-max variant to the identity loss
function [softmax]. While improving the performance of the model, these
advanced improvements also add complexity to the architecture. Despite
successful implementation of the algorithms which achieve promising results,
the above methods have not considered the remote inference problem where the
retrieval task cannot be performed on the edge device locally, but information
has to be sent through a wireless communication channel to perform inference
in the edge server.

###  II-B Deep learning at the wireless edge

To address the remote inference problem in this project and apply the relevant
knowledge to this project, this section reviews the existing algorithms for
different deep learning tasks (e.g. image classification, transmission and
retrieval tasks using deep neural networks (DNNs)) at the wireless edge.

The conventional digital coding scheme is a separation-based approach which
separates the source coding and channel coding processes. The input image is
first fed into a source encoder for compression to reduce its redundancy.
Compression can be classified into two classes, namely lossless and lossy
compression. For lossless compression, the input data can be perfectly
reconstructed without the presence of noise, but the compression rate will be
limited. Common lossless compression schemes include entropy coding like
arithmetic coding [arithmetic_coding]. On the contrary, lossy compression can
achieve higher compression rates, but only an approximation of the input can
be reconstructed. After obtaining the compressed bit streams from the source
encoder, channel coding is performed. Redundant bits are selectively added to
allow detection and correction of bit errors at the receiver for a more
reliable transmission. Common channel coding schemes include turbo code
[turbo] and low-density parity-check code (LDPC) [ldpc]. Also, under a given
constraint on the channel bandwidth, there is a common trade-off between the
compression rate (for source coding) and the channel coding rate.

Multiple models [digital_2] [digital_1] have utilized the digital separation-
based coding scheme to perform distributed inference for various tasks at the
wireless edge. In particular, Jankowski et al. proposed a retrieval-oriented
image compression scheme [jscc_ae] specifically for the person re-
identification task. The model incorporates the digital coding scheme into the
retrieval baseline by firstly including a feature encoder to extract the
feature vector from the input image, followed by lossy compression, scalar
quantization and arithmetic coding to compress the feature vectors. The
compressed bits are then undergone channel coding and transmitted through the
channel.

Theoretically, asymptotic optimality can be achieved by separating the
processes of source coding and channel coding in the digital coding scheme,
according to Shannon’s separation theorem [shannon]. However, the above
theorem only holds for channels with infinite block lengths and breaks as the
block length decreases. In practice, due to latency and computation
constraints, we are only limited to have finite block length channels. Hence,
with short block lengths, channel codes can introduce significant error
particularly with inaccurate channel state information, which is the case in
many practical IoT applications. Another drawback of the digital coding
approaches is that they suffer from the ”cliff effect”, which refers to the
significant drop in model performance when the actual channel condition is
different from the target channel state where the design of the channel codes
is based upon.

In light of the disadvantages of the digital coding scheme, another analog
approach called JSCC has been developed. Source coding and channel coding is
performed jointly in JSCC, where the feature vectors from the input image are
directly mapped into the channel input symbols. Existing JSCC schemes are
mostly implemented using DNNs. Recently, JSCC schemes based on autoencoders
are introduced for image transmission [jscc_ae_1] [jscc_ae_2] [jscc_ae_3],
image classification [jscc_ae_4] and image retrieval [jscc_ae] tasks at the
wireless edge. After extensive evaluations, these models are found to have
better performance than the corresponding state-of-the-art digital schemes. In
particular, the JSCC AE model proposed by Jankowski et al. [jscc_ae] optimized
for the retrieval task at the wireless edge will be extended to apply onto the
two-source case.

To achieve optimal accuracy and minimize the degradation of model performance
during channel mismatch between training and testing, an Attention Deep
Learning based JSCC model (ADJSCC) for image transmission with the use of
attention mechanism was proposed [jscc_attention] for a single network to be
able to operate on multiple channel SNRs during testing. In the model, a
channel-wise soft-attention network is used to scale the features dynamically
according to the channel SNR. As compared its counterpart without attention
mechanism [jscc_ae_1], ADJSCC is more robust towards channel mismatch and has
a smaller extent of degradation in performance when there is channel mismatch
during training and testing.

###  II-C Multi-source collaborative deep learning

All of the algorithms and models covered so far in the previous sections
focused on the single-source scenario where only one image is taken as the
input and there is only one edge device involved. However, in many real-life
scenarios like surveillance applications, multiple devices will be present in
the same scene to capture the same object from different angles. The major
focus of this project is to investigate the effect of the collaboration
between multiple edge devices in joint remote inference tasks on the
performance of the model. This section reviews the existing models for multi-
source deep learning tasks at the wireless edge, as well as techniques related
to multi-view representation learning and a metric to find the correlation
between multi-view features.

####  II-C1 Multi-source collaborative deep learning at the wireless edge

As compared to the single-source scenario, multi-source deep learning tasks at
the wireless edge are much less studied in the literature. Existing multi-
source algorithms mainly focus on image transmission and classification tasks.
An autoencoder-based JSCC scheme was proposed by Yan, et al. for image
transmission and estimation in a multi-user wireless sensor network
[multi_imagetran]. In the model, the edge sensors observe the same image and
encode the input image independently via distributed JSCC encoders. The noisy
output vectors from each channel are first concatenated and then decoded with
a joint JSCC decoder to estimate the original image. Experimental results show
that this proposed JSCC scheme outperforms the conventional digital scheme in
both AWGN and slow fading channels, especially in harsh channel conditions
with low channel SNR or low channel bandwidth available. This finding aligns
with previous findings in the single-source scenario and further justifies the
superiority of the JSCC scheme over the digital scheme in deep learning tasks
at the wireless edge. However, one major difference in the experimental
setting of this model compared to our project is that the same image is fed
into all channels in this case, but different images from different camera
views containing the same identity should be fed into each separate channel in
our project. Also, as this is an image transmission task, no prior feature
extraction is needed before feeding the input image into the JSCC autoencoder.
This makes the proposed JSCC scheme not exactly suited for our application.
Nevertheless, the distributed encoders and joint decoder structure of the JSCC
autoencoder adopted in this model can still be referenced and tested in the
design process of our two-source model.

For distributed inference tasks which are more suited for our application,
Shao et al. proposed a task-oriented digital communication scheme for multi-
device collaborative edge inference [multidevice], which utilizes the
information bottleneck (IB) principle for feature extraction and the
deterministic distributed information bottleneck (DDIB) principle for
distributed feature encoding. The trade-off between the communication overhead
and inference accuracy is also addressed in the model. Even though the model
is tested to outperform other feature encoding algorithms for collaborative
image classification and shape recognition problems at the edge, there is no
comprehensive analysis on the performances of the proposed algorithm under
different channel conditions, e.g. different channel SNRs. As a result, the
cliff effect problem commonly suffered by digital coding scheme is also not
addressed. It is not justified that this digital approach will outperform the
JSCC approach especially under low channel SNRs. Together with the difference
between the image classification task where the proposed model is trained on,
and the retrieval task in which this project focuses on, this algorithm may
not be suited for our application.

####  II-C2 Multi-view representation fusion

Apart from existing models which deal with multi-source deep learning tasks at
the wireless edge with consideration of the communication channel directly,
multi-view feature representation fusion techniques [multiview], which aim to
integrate multi-view inputs into one single representation, are also useful to
our project. This is because in the testing stage of the models for person re-
identification tasks, the distances between the feature vectors of the query
image and all the gallery images are computed. Hence, we will need a single
compact representation of feature vector from all the feature vectors
extracted from each of the edge devices independently, and this is where this
set of techniques is needed.

In recent years, neural network-based multi-view representation fusion methods
have gained popularity, in which multi-view convolutional neural networks
(MVCNN) have been applied on various applications successfully. Unlike single-
view convolutional neural networks, MVCNN learns features (i.e.
representations) when there are multiple views of data available. The goal of
MVCNN is to extract useful information from each of the different views so
that they combine to form one single representation for the given tasks which
is more holistic and comprehensive. This is applicable to our project as the
inputs to our two-source model will be two images from two different camera
views of the same identity, which correspond to multi-view data. For instance,
Su et al. proposed a MVCNN model for 3D shape classification and retrieval
[viewpool], where the feature vectors are extracted from each input image
separately via CNNs with a common architecture, which are then fused in a view
pooling layer and lastly passed to the remaining part of the network. In the
view-pooling layer, element-wise maximum is computed amongst all feature
vectors. From the results, the proposed model outperforms all other state-of-
the-art 3D descriptors for both the classification and retrieval tasks.
Moreover, the concept of view-pooling layer can be applied to our model, which
is a simple yet proven effective way to combine different features into a
single representation for our two-source person re-identification task at the
wireless edge.

One important point to note regarding the use of the aforementioned feature
fusion techniques is that they can only be applied after the communication
channel. In our scenario, as mentioned, it is assumed there is no
communication between the edge devices themselves. Only after the feature
vectors have been sent to the edge server via the channel, the reconstructed
feature vectors can be combined together to form a single compact
representation for inference. Nevertheless, the view-pooling layer is still an
important tool for our two-source person re-identification model.

###  II-D Metric for measuring correlation between multi-source features

Apart from combining multiple feature vectors from the different sources into
a single compact representation for the evaluation stage of the model, it
might also be helpful to compute the correlation between the input feature
vectors or the input symbols (which are the input to the channel) from each of
the sources. In the two-source collaborative scenario, the correlation between
the two sets of input symbols may have an effect on the maximum retrieval
accuracy that can be achieved by the proposed models.

One metric that can be used to quantify the similarity between two vectors of
the same length is the squared cosine similarity. Given two vectors
$\mathbf{x_{1}}$ and $\mathbf{x_{2}}$, the squared cosine similarity between
$\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ is given by Equation 1 [cos].

$\cos^{2}(\mathbf{x_{1}},\mathbf{x_{2}})=\frac{\langle\mathbf{x_{1}},\mathbf{x_{2}}\rangle^{2}}{\|\mathbf{x_{1}}\|^{2}\|\mathbf{x_{2}}\|^{2}}$
(1)

where $\langle\cdot\rangle$ represents the inner product operator and
$\|\cdot\|$ represents the Euclidean norm operator. Note that the square is
adopted here (but not cosine similarity itself) so as to restrict the output
range to be $[0,1]$. This means that the value of the squared cosine
similarity between any two vectors will always be non-negative and thus can be
fit into the loss function during the training of the proposed models, if
necessary.

In conclusion, the above models and techniques discussed in this chapter serve
a good starting point for extending the existing JSCC AE model for the problem
of single-source person re-identification at the wireless edge into the
desired two-source collaborative scenario in this project.

##  III Deep learning-based JSCC models for two-source collaborative person
re-identification at the wireless edge

###  III-A Orthogonal model

####  III-A1 Channel model

In the orthogonal model, the encoded symbols from the two sources are
transmitted through two mutually orthogonal channels independently. It is
assumed that there is no interference between the two channels. As each set of
input symbols from each source is transmitted separately, the resources
allocation scheme employed in this model is that each source (i.e.
transmitter) gets allocated half of the available channel bandwidth. If the
total available bandwidth is $B$ complex input symbols (i.e. $2B$ real input
symbols), then the input vector to each channel will be of $\frac{B}{2}$
complex symbols.

In this section, both the AWGN and the Rayleigh slow fading channel will be
considered. The two channel input vectors are denoted as
$\mathbf{x_{1}},\mathbf{x_{2}}\in\mathbb{C}^{\frac{B}{2}}$ with $\frac{B}{2}$
complex symbols. For the AWGN channel, the output vectors
$\mathbf{\hat{x}_{1}},\mathbf{\hat{x}_{2}}\in\mathbb{C}^{\frac{B}{2}}$ of the
two channels are given in Equation 2.

$\begin{split}\mathbf{\hat{x}_{1}}&=\mathbf{x_{1}}+\mathbf{z_{1}}\\\
\mathbf{\hat{x}_{2}}&=\mathbf{x_{2}}+\mathbf{z_{2}}\end{split}$ (2)

where $\mathbf{z_{1}}$ and $\mathbf{z_{2}}$ are the complex noise vectors of
channel 1 and 2 respectively. The elements of the noise vectors are i.i.d. and
sampled from a complex Gaussian distribution of zero mean and variance of
$\sigma^{2}$. Same as the single-source case, for each channel, there is an
average input power constraint applied onto the set of input symbols. The
input vector to each channel is normalized individually before transmitting
through the corresponding channel so that the average input power per
(complex) symbol for each transmitter is not larger than 1, i.e.
$\frac{1}{\frac{B}{2}}\sum_{j=1}^{\frac{B}{2}}|x_{i_{j}}|^{2}\leq
P_{{\text{avg}}_{\text{max}}}=1$ where $i=1$ or 2 depending on which channel
it is. The channel SNR can be computed through
$\text{SNR}=10\log_{10}(\frac{1}{\sigma^{2}})$ dB, which can be varied by
varying the noise variance $\sigma^{2}$.

As for the Rayleigh slow fading channel, the output vectors
$\mathbf{\hat{x}_{1}},\mathbf{\hat{x}_{2}}\in\mathbb{C}^{\frac{B}{2}}$ from
the two channels are given by Equation 3.

$\begin{split}\mathbf{\hat{x}_{1}}&=h_{1}\mathbf{x_{1}}+\mathbf{z_{1}}\\\
\mathbf{\hat{x}_{2}}&=h_{2}\mathbf{x_{2}}+\mathbf{z_{2}}\end{split}$ (3)

where the channel gains
$h_{1},h_{2}\sim\mathbb{C}\mathcal{N}(0,\sigma_{h}^{2})$ are drawn from zero-
mean complex Gaussian distributions of variance $\sigma_{h_{1}}^{2}$ and
$\sigma_{h_{2}}^{2}$. $\mathbf{z_{1}}$ and $\mathbf{z_{2}}$ are as defined
above. The same average power constraint of $P_{{\text{avg}}_{\text{max}}}=1$
is applied and the channel SNR is evaluated as
$\text{SNR}=10\log_{10}(\frac{\sigma_{h}^{2}}{\sigma^{2}})$ dB. Note that we
set $\sigma_{h_{1}}^{2}=\sigma_{h_{2}}^{2}=1$ in all experiments so that it
corresponds to the same received power as in the AWGN channel case for fair
comparison.

####  III-A2 Network Architecture

Figure 2 shows the block diagram of the network architecture of the orthogonal
model for the two-source collaborative person re-identification task at the
wireless edge.

Figure 2: Network architecture of the orthogonal model

In the first half of the orthogonal model before transmission through the
channels, there are two parallel and independent branches with the same
structure, each corresponding to one of the sources. At each branch, the input
image is first fed into the feature encoder, a ResNet40 network pretrained on
ImageNet, to extract its corresponding feature vector $\mathbf{v_{1}}$ or
$\mathbf{v_{2}}$. Next, the feature vectors $\mathbf{v_{1}}$ and
$\mathbf{v_{2}}$ are directly mapped into the vectors of input symbols
$\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ through the JSCC encoder, which are then
transmitted through the noisy channel. With a total available channel
bandwidth of $B$ complex symbols, $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ are
each of length $\frac{B}{2}$. Before inputting to the channel,
$\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ are first normalized individually such
that they both have an average power per symbol of 1 to satisfy the average
input power constraint of $P_{{\text{avg}}_{\text{max}}}=1$.

Then, the noisy symbols from the other channel is also fed into each JSCC
decoder as side information. That is, the noisy symbols $\mathbf{\hat{x}_{1}}$
and $\mathbf{\hat{x}_{2}}$ from the two channels are combined through length-
wise concatenation, which then acts as the input to both JSCC decoders. The
two distributed JSCC decoders will output two sets of reconstructed feature
vectors $\mathbf{\hat{v}_{1}}$ and $\mathbf{\hat{v}_{2}}$ in the higher-
dimensional space, each corresponding to one source. To form a single compact
representation for the two sources for testing, an extra view-pooling layer
with length-wise concatenation is added. This compact feature vector will be
used for computing the Euclidean distances between the query image pair and
the gallery images for the nearest neighbours search during testing. The final
component of the model is the classifier, which is only used during training.
Auxiliary classifiers are introduced for each source, and a total of three
classifiers is used. The main classifier is connected to the view-pooling
layer so as to output ID predictions from the single compact representation,
and the two auxiliary classifiers are directly connected to the two outputs of
the two JSCC decoders $\mathbf{\hat{x}_{1}}$ and $\mathbf{\hat{x}_{2}}$ to
produce two more sets of ID predictions from the reconstructed feature vectors
directly. Each set of the ID predictions will be compared with the ground
truth and the cross-entropy loss will be computed to be part of the loss
function for training.

##### Structure of the JSCC autoencoder

While the overall network architecture is the same under the all channel
types, the structure of the JSCC autoencoder is adapted according to different
channel types to optimize performance. For the AWGN channel and slow fading
channel without CSI, the JSCC autoencoder has the same structure as the
single-source model, as in Figure 3. It consists of FC layers (size $2B$ is
changed to $B$ real symbols ($\frac{B}{2}$ complex symbols)), followed by BN
layers and LeakyReLU activation functions. Number of FC layers is unchanged as
it is optimal for the two-source case.

(a) JSCC Encoder

(b) JSCC Decoder

Figure 3: Architecture of the JSCC autoencoder in the JSCC AE model [jscc_ae]

As for the slow fading channel with CSI, it is assumed that perfect CSI is
available at the receiver but not at the transmitter. Therefore, the structure
of the JSCC encoder is the same as the previous cases as in Figure 3(a), while
the structure of the JSCC decoder is modified to incorporate the information
of channel gains $h_{1}$ and $h_{2}$ into the corresponding network. Soft-
attention mechanism has been applied at the JSCC decoder. Figure 4 shows the
architecture of the two JSCC decoders, consisting of attention feature (AF)
modules and FC layers, BN layers and LeakyReLU activation functions. Channel
gain $h_{1}$ of channel 1 is fed into the AF modules in JSCC decoder 1, while
the channel gain $h_{2}$ of channel 2 is input to the AF modules in JSCC
decoder 2. Figure 5 shows the detailed flow of the AF module.

Figure 4: Structure of the JSCC decoders in the orthogonal model for slow
fading channel with CSI

There are 3 major components in the AF module used in the JSCC decoders of the
orthogonal model for slow fading channel with CSI. In the context extraction
step, the input vector $\mathbf{f}$ is concatenated with the corresponding
channel gain $h_{i}$ to give the concatenated vector, where $i$ depends on
which decoder it is. Next, in the factor prediction step, an attention mask
$\mathbf{s}$ is predicted through a factor prediction neural network at which
its input is the concatenated vector from the last step. This neural network
is a simple network which consists of only two FC layers so as not to
significantly increase the computational complexity of the overall network.
Finally, at the feature recalibration step, the attention mask $\mathbf{s}$ is
multiplied element-wise with the input vector $\mathbf{f}$ to give the scaled
output vector $\mathbf{\hat{f}}$ which has the same dimension as the input.
Through the above steps, each element of the input vector is scaled
individually according to the corresponding channel gain.

Figure 5: Structure of the AF module used in the JSCC decoders of the
orthogonal model for slow fading channel with CSI

####  III-A3 Training strategy

A three-step training strategy similar to that of the single-source JSCC AE
model has been adopted, which consists of feature encoders pre-training
($T_{1}$), JSCC autoencoders pre-training ($T_{2}$) and end-to-end training
($T_{3}$). In $T_{1}$, the two feature encoders, together with the view
pooling layer and the main classifiers with the two auxiliary classifiers, are
pre-trained like an image classification problem for 30 epochs with a batch
size of 16. To optimize the model performance in this two-source collaborative
scenario, the loss function is slightly modified from the single-source
scenario. Instead of only including the cross-entropy loss from the main
classifier, the loss function used is the average of the 3 cross-entropy
losses computed from the 3 classifiers, as in Equation 4.

$l=\frac{1}{3}(l_{ce_{aux1}}+l_{ce_{main}}+l_{ce_{aux2}})$ (4)

where $l_{ce_{aux1}}$ is the cross-entropy loss between ID predictions from
auxiliary classifier 1 and the ground truth, $l_{ce_{main}}$ is the cross-
entropy loss from main classifier, and $l_{ce_{aux2}}$ is the cross-entropy
loss from auxiliary classifier 2.

In $T_{2}$, the pre-trained feature encoders are frozen and used to extract
the feature vectors from all image pairs in the training set. For each image
pair, there will be two feature vectors $\mathbf{v_{1}}$ and $\mathbf{v_{2}}$,
each corresponding to one source. These extracted feature vectors are then
used as inputs for the pre-training of the JSCC autoencoders. The loss
function used is the average of the MSE losses between the feature vector
$\mathbf{v_{i}}$ and the corresponding reconstructed feature vector
$\mathbf{\hat{v}_{i}}$ by the JSCC decoder, as in Equation 5.

$l=\frac{1}{2}(l_{MSE_{1}}+l_{MSE_{2}})$ (5)

where $l_{MSE_{1}}$ is the MSE loss between feature vector $\mathbf{v_{1}}$
and reconstructed feature vector $\mathbf{\hat{v}_{1}}$, while $l_{MSE_{2}}$
is the MSE loss between feature vector $\mathbf{v_{2}}$ and reconstructed
feature vector $\mathbf{\hat{v}_{2}}$. The network is trained for 200 epochs
with an initial learning rate of 0.1, which is reduced to 0.01 after 150
epochs. SGD with a momentum of 0.9 is applied as the optimizer, and L2
regularizer weighted by $5\cdot 10^{-4}$ is also applied.

In $T_{3}$, the whole network as in Figure 2 is trained jointly, with the loss
function equal to that of $T_{1}$ (i.e. average cross-entropy loss as in
Equation 4). The whole model is trained for 30 epochs in total, with different
learning rates for different components. For the feature encoder and the
classifiers, the initial learning rate is 0.01, and it is reduced to 0.001
after 20 epochs. As for the JSCC autoencoder, the learning rate is 0.001 for
the first 20 epochs and 0.0001 for the remaining 10 epochs. The same optimizer
and L2 regularization as the previous steps are used.

###  III-B Over-the-air model

####  III-B1 Channel model

In the over-the-air model, the encoded symbols from the two sources are
transmitted through the channel in a different way as the orthogonal model. In
this channel model, the two sets of input symbols from the two sources are
superimposed on each other and transmitted simultaneously through the channel
(i.e. over-the-air transmission). This way, each of the transmitter will be
allocated the full total available bandwidth of $B$ complex symbols. It is
worth noting that as the two input vectors are superimposed on each other in
the transmission, the interference between them has to be considered. One
major difference of this channel model from the orthogonal channel model is
that there is only one output from the channel instead of two.

Similar to that of the orthogonal model, both the AWGN channel and Rayleigh
slow-fading channel will be considered in this section. Denoting the two
channel input vectors as $\mathbf{x_{1}},\mathbf{x_{2}}\in\mathbb{C}^{B}$, the
output vector $\mathbf{\hat{x}}\in\mathbb{C}^{B}$ of the AWGN channel is given
by the sum of the two input vectors and the channel additive noise as in
Equation 6.

$\mathbf{\hat{x}}=\mathbf{x_{1}}+\mathbf{x_{2}}+\mathbf{z}$ (6)

where $\mathbf{z}$ is the complex noise vector of the channel and its elements
are i.i.d. and sampled from a complex Gaussian distribution of zero mean and
variance of $\sigma^{2}$. An average input power constraint is also applied
onto the input symbols, with a slight modification compared to the previous
models. In this case, before transmitting through the channel, the input
symbol vectors $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ are normalized
individually such that their input power per complex symbol is kept not larger
than 0.5, i.e. $\frac{1}{B}\sum_{j=1}^{B}|x_{i_{j}}|^{2}\leq
P_{{\text{avg}}_{\text{max}}}=0.5$ where $i=1$ or 2 depending on which channel
it is. The computation of channel SNR is the same as before:
$\text{SNR}=10\log_{10}(\frac{1}{\sigma^{2}})$ dB.

As for the slow fading channel, the output $\mathbf{\hat{x}}\in\mathbb{C}^{B}$
of the channel is given by Equation 7.

$\mathbf{\hat{x}}=h_{1}\mathbf{x_{1}}+h_{2}\mathbf{x_{2}}+\mathbf{z}$ (7)

where the channel gains
$h_{1},h_{2}\sim\mathbb{C}\mathcal{N}(0,\sigma_{h}^{2})$ are drawn from zero-
mean complex Gaussian distributions of variance $\sigma_{h_{1}}^{2}$ and
$\sigma_{h_{2}}^{2}$. $\mathbf{z}$ is the complex additive noise vector as
defined above. Similar to the AWGN channel, an average input power per symbol
constraint of $P_{{\text{avg}}_{\text{max}}}=0.5$ is imposed onto the each of
the input vector (i.e. each transmitter). The channel SNR is evaluated as
$\text{SNR}=10\log_{10}(\frac{\sigma_{h}^{2}}{\sigma^{2}})$ dB, at which
$\sigma_{h_{1}}^{2}=\sigma_{h_{2}}^{2}=1$ so that the average received signal
power is the same as the AWGN channel case for fair comparison.

####  III-B2 Network architecture

Figure 6 shows the block diagram of the overall network architecture of the
over-the-air model.

Figure 6: Network architecture of the over-the-air model

Most of the components in the over-the-air model are similar to that of the
orthogonal model, except for the JSCC autoencoder whose structure is dependent
on the channel model used. As the channel model of the over-the-air model is
different from the orthogonal model, it is expected that the structure of the
JSCC autoencoder is adapted to optimize performance in this particular
scenario. Before transmitting through the channel, the two input vectors
$\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ are normalized individually to each have
an average input power per symbol of 0.5, but not 1, as the average input
power constraint per symbol is $P_{{\text{avg}}_{\text{max}}}=0.5$ in this
model.

##### Structure of JSCC autoencoder

For the AWGN channel and the slow fading channel without CSI, the JSCC
autoencoder has the same structure as of the single-source model in Figure 3.
As for the slow fading channel with CSI scenario where perfect CSI is
available at the receiver only, the JSCC encoders and decoders have different
structures. The JSCC encoders share the same structure as the other types
(i.e. Figure 3(a)) since CSI is not available at the transmitter, while soft-
attention mechanism is applied in the JSCC decoder to incorporate the
information of the channel gains $h_{1}$ and $h_{2}$ into the network. Figure
7 shows the structure of the 2 JSCC decoders used in the over-the-air model.
It has the same structure as that of the orthogonal model, except that the
channel gains fed into the AF modules of the two decoders are different from
that in the orthogonal model. In this case, as the noisy output vector
$\mathbf{\hat{x}}$ from the channel is the sum of the two sets of input
symbols from the two sources, both of the channel gains $h_{1}$ and $h_{2}$
are fed into both of the decoders. The flow of the AF modules used in the JSCC
decoders of the over-the-air model is shown in Figure 8.

The AF modules used in the over-the-air model for slow fading channel with CSI
have the same flow as the orthogonal model. The only difference from the
orthogonal model is at the context extraction step. Instead of concatenating
only one channel gain of the corresponding channel, both $h_{1}$ and $h_{2}$
are concatenated with the input vector $\mathbf{f}$ to give the concatenated
vector.

Figure 7: Structure of the JSCC decoders in the over-the-air model for slow
fading channel with CSI Figure 8: Structure of the AF module used in the JSCC
decoders of the over-the-air model for slow fading channel with CSI

####  III-B3 Training strategy

Same as the orthogonal model, the three-step training strategy has been
adopted for the over-the-air model. In each of the training steps, the loss
function, number of epochs, learning rate, optimizer and the regularization
technique are the same as that of the orthogonal model.

###  III-C Evaluation and Analysis

####  III-C1 Evaluation of model performances on a wide range of channel SNRs

The two proposed two-source models, orthogonal model and over-the-air model,
are trained and tested on our pre-processed Market-1501 dataset over a wide
range of channel SNRs from -12dB to 15dB. For all evaluations of the proposed
models in this chapter, the testing channel SNR is kept the same as that
during training. The results of the single-source JSCC AE model in [jscc_ae]
are also included as a benchmark for comparison to check if our proposed
models outperform the original single-source model.

(a) AWGN channel

(b) Slow fading channel without CSI

(c) Slow fading channel with CSI

Figure 9: Top-1 retrieval accuracies of the proposed two-source models and the
single-source model under different channel SNRs with a total channel
bandwidth of $B=64$

####  III-C2 Comparison with the single-source model

The trends in the results of the three channel types in Figures 9(a), 9(b) and
9(c) are similar: both of our proposed two-source models outperform the
original single-source model for a wide range of channel SNRs. It is also
noted that the performance of the over-the-air model is even better than the
orthogonal model, in which the over-the-air model outperforms the single-
source model greatly even at very low channel SNRs. It is noted that the
accuracies achieved by all the models in slow fading channel with CSI are
higher than that in slow fading channel without CSI under most channel SNRs.
This is as expected as the availability of the information of channel gains in
the JSCC decoders for the slow fading channel with CSI allows the network to
reconstruct the feature vectors more effectively.

It is worth noting that even when the orthogonal model is not as good as the
over-the-air model, it still outperforms the single-source model for quite
some margin in a number of channel SNRs. The fact that both of our proposed
two-source models outperform the single-source models in all the channel types
proves the claim that incorporating one more view of the same identity to make
a collaborative decision at the edge server improves the retrieval performance
of the network.

####  III-C3 Relative performances between the proposed two-source models

##### Resources utilization

In terms of resources utilization, the channel model in the over-the-air model
allows a more efficient and effective use of communication resources. It is
first noted that as compared to the orthogonal channel, the maximum allowable
average input transmitted power per symbol for each of the transmitter of the
over-the-air model is halved. This is equivalent to an effective received
signal SNR drop of 3dB. Nevertheless, the output of the over-the-air channel
model in the AWGN case is given by the sum of the two individual inputs plus
the additive noise of the channel:
$\mathbf{\hat{x}}=\mathbf{x_{1}}+\mathbf{x_{2}}+\mathbf{z}$. If
$\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ are identical, the output will then be
$\mathbf{\hat{x}}=\mathbf{2x_{1}}+\mathbf{z}$, which means that the signal
amplitude is doubled and that the received signal power is quadrupled. This
means that the received signal SNR will be increased by 6dB. The same
conclusion can be drawn for the slow fading channel case both without and with
CSI. Combining the effect of halving the maximum allowable average input power
per symbol, having identical inputs leads to an overall net increase of the
received signal SNR by 3dB. In our two-source scenario where the input image
pair is two different images containing the same identity, the two inputs to
the channel will not be exactly identical. Moreover, summing up the two inputs
will not lead to as much as doubling of the signal amplitude, and thus the
effective channel SNR will not increase by as much as 3dB as in the ideal
case. However, there will still be a certain extent of increase in the
effective channel SNR. To maximize the increase in channel SNR and to optimize
its performance, the network should ideally learn to extract and encode the
two sets of input symbols to be as similar as possible to amplify the input
signal for better retrieval performance. Hence, given the same communication
resources (both the total transmitted power over all symbols per transmitter
and total available bandwidth), the over-the-air model is able to utilize the
channel resources more effectively and can be able to achieve the highest
retrieval accuracy.

##### Channel coding

Apart from a more effective utilization of resources, the better performance
of the over-the-air model can also be explained from the channel coding
perspective. Note that even though JSCC which is an ”analog” communication
scheme is used in our models, the following discussion based on the
conventional digital communication scheme will still be valid in our scenario
as the rationale behind and their relative performances are the same. For both
of the channel models in our two proposed two-source models, the channel can
be viewed as a multiple access channel, but with different schemes used. In
the orthogonal model, the channel model can be viewed as a time-division
multiple access channel (TDMA). As for the over-the-air model, superposition
between the two input vectors is allowed. Moreover, its channel model
essentially implements superposition coding, i.e. a code division multiple
access channel (CDMA). It is apparent that the achievable capacity region of
CDMA is larger than that of TDMA. For TDMA, the capacity region is a linear
region which is bounded by a straight line. As for CDMA, due to the larger
achievable capacity region, rate pairs that are originally not possible in
TDMA are now achievable. This shows that allowing the superposition between
the two input vectors from the two sources allow a more effective channel
coding scheme and thus boosts the retrieval performance of the model. Even
though source coding and channel coding are combined and are not separate
processes in JSCC which is employed in our proposed two-source models, it can
still be concluded that allowing superposition during transmission through the
channel in the over-the-air model allows more effective coding and decoding at
the JSCC autencoder. Therefore, this is another factor which contributes to
the significant increase in the top-1 retrieval accuracy of the over-the-air
model as compared to the orthogonal model.

####  III-C4 Further analysis of relative performances between the proposed
two-source models in terms of correlation between input symbols

STo study the effect of the correlation between the two sources on the
performance of the models, we first compute the squared cosine similarity of
the two sets of input symbols $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ which are
the outputs of the two JSCC encoders, for both the proposed orthogonal and
over-the-air model on both the AWGN channel and slow fading channel without
CSI. Table I shows the computed squared cosine similarity of $\mathbf{x_{1}}$
and $\mathbf{x_{2}}$ extracted from the query set of the pre-processed dataset
during the testing stage of both the orthogonal model and over-the-air model,
on both AWGN channel and slow fading channel without CSI.

Model | Squared cosine similarity between $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$
---|---
Orthogonal model on AWGN channel | 0.0151
Orthogonal model on slow fading channel | 0.0165
Over-the-air model on AWGN channel | 0.7523
Over-the-air model on slow fading channel | 0.8234
TABLE I: Squared cosine similarity between $\mathbf{x_{1}}$ and
$\mathbf{x_{2}}$ of the orthogonal and over-the-air models evaluated on AWGN
and slow fading channel without CSI

From Table I, it is apparent that the squared cosine similarity between
$\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ obtained for the over-the-air model is
much higher than that of the orthogonal model on both AWGN channel and slow
fading channel without CSI. For the orthogonal model, the values of squared
cosine similarities computed are close to 0. The JSCC encoders learn to encode
the feature vectors in a way such that the two sets of input symbols to the
wireless channel are almost independent. This is as expected as the two
channels are independent and there is no interference between the two channels
in the orthogonal model. By sending independent symbols with as low
correlation as possible, the vector of input symbols from each source will be
able to capture different information of the particular identity, so as to
maximize the use of the given limited communication resources. As for the
over-the-air model, the squared cosine similarity between $\mathbf{x_{1}}$ and
$\mathbf{x_{2}}$ can reach as high as around 0.8 for both channel types, which
means the JSCC encoders in the over-the-air model learn to send highly
correlated symbols for the two sources through the channel. From the above
validation, it can be seen that only the over-the-air model learns to send
correlated symbols, while the orthogonal model learns to send independent
symbols. Moreover, the claim at which the use of correlated sources increases
the retrieval accuracy of the model is only applicable to the over-the-air
model, which shows that the use of correlated sources can be a potential major
contribution behind the better performance of the over-the-air model than the
orthogonal model.

To prove that having a high correlation between the two sets of input symbols
$\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ improves the performance of the model,
we need to come up with a way to alter the value of the squared cosine
similarity between $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ during the training
of the over-the-air model and investigate the effect of decreasing the squared
cosine similarity between $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ on the
performance of the over-the-air model. To achieve this, we introduce a squared
cosine similarity regularization term in the loss function during the training
of the over-the-air model. Using the Lagrangian formulation and denoting the
squared cosine similarity between $\mathbf{x_{1}}$ and $\mathbf{x_{2}}$ as
$\cos^{2}{(\mathbf{x_{1}},\mathbf{x_{2}})}$, the modified loss function used
(which are modified from the original average cross-entropy loss as in
Equation 4) for training of the over-the-air model to investigate the effect
of correlation between the input symbols is given by Equation 8.

$l=\frac{1}{3}(l_{ce_{aux1}}+l_{ce_{main}}+l_{ce_{aux2}})+\lambda\cos^{2}{(\mathbf{x_{1}},\mathbf{x_{2}})}$
(8)

where $\lambda$ is the Lagrangian multiplier and is a hyperparameter to be
changed so as to change the value of squared cosine similarity between
$\mathbf{x_{1}}$ and $\mathbf{x_{2}}$. To reduce their squared cosine
similarity, we can increase $\lambda$ so that more emphasis is given to
minimizing the squared cosine similarity between the two sets of input symbols
in the loss function.

Figure 10 shows the top-1 retrieval accuracies achieved by the over-the-air
model on both the AWGN channel and slow fading channel without CSI at
different squared cosine similarities between $\mathbf{x_{1}}$ and
$\mathbf{x_{2}}$ by changing $\lambda$. The channel SNR is kept at 0dB and the
total available channel bandwidth is kept as $B=64$ complex symbols in all of
the cases for fair comparison. From the two curves in the figure, it is
apparent that increasing the squared cosine similarity between the two sets of
input symbols increases the retrieval accuracy of the over-the-air model, for
both the AWGN channel and the slow fading channel without CSI. The increase in
accuracy with the increase of correlation between $\mathbf{x_{1}}$ and
$\mathbf{x_{2}}$ is more obvious in the slow fading channel than the AWGN
channel. Hence, it can then be concluded that the high correlation between the
two sets of input symbols encoded in the over-the-air model improves the
performance of the over-the-air model, as the two sources will be able to
enforce each other.

Figure 10: Top-1 retrieval accuracies of the over-the-air model on both AWGN
channel and slow fading channel without CSI against different squared cosine
similarity between the two sets of input symbols $\mathbf{x_{1}}$ and
$\mathbf{x_{2}}$, with a channel SNR of 0dB and a total channel bandwidth of
$B=64$

####  III-C5 Effect of total available channel bandwidth on model performances

Another factor that significantly affect the performances of the proposed
models is the total available channel bandwidth $B$. The two proposed two-
source models (orthogonal model and over-the-air model), as well as the
single-source model, are trained and tested on the pre-processed Market-1501
dataset with total available channel bandwidths of $B=16$, 32, 64, 128, 256
complex symbols.

The top-1 retrieval accuracies achieved by the orthogonal model, over-the-air
model and the single-source model on the AWGN channel and the slow fading
channel without CSI scenarios are shown in Figures 11(a) and 11(b). From the
results obtained, it is apparent that the relative performances of the three
models are the same for different total available channel bandwidth $B$ for
the two channel types: both of our proposed two-source models outperform the
single-source model, with the over-the-air model having even better
performance than the orthogonal model. It is further noted that the lower the
total bandwidth available, the bigger the gap in accuracies between the two
proposed two-source models and the single-source model. This means that a
lower total available channel bandwidth $B$ will lead to a larger increase of
the accuracy of the model when image from one more camera view of the same
identity is introduced by the extra source of the over-the-air and orthogonal
models. This is because with a lower total bandwidth, one set of input symbols
from only one source is even less representative of the original image and
less information of the original image and the identity can be represented by
the set of input symbols, so the improvement of model performance by
introducing an extra set of input symbols for the inference problem becomes
even more obvious. The higher the total bandwidth available, the more
information can be included in a single set of input symbols, and thus the
improvement in accuracy after adding an extra view will be smaller for a
higher total bandwidth.

(a) AWGN channel

(b) Slow fading channel without CSI

Figure 11: Top-1 retrieval accuracies of the proposed two-source models and
the single-source model over different total channel bandwidths with a channel
SNR of 0dB

##  IV SNR-aware JSCC models with attention modules for two-source person re-
identification at the wireless edge

###  IV-A Motivation

In previous section, all of the proposed model design and evaluation processes
are done based on the assumption that the channel conditions during the
training and testing (i.e. deployment) stages of the model are the same. The
top-1 retrieval accuracies obtained through the previous evaluation are thus
the best possible accuracies that can be achieved by the model under each
channel condition. However, this is often not the case in practical
applications. If there is a mismatch between the training and testing channel
conditions, due to the fact that the previously proposed models are trained
only on a single specific channel SNR, their testing performances will
degrade. If we would like to achieve optimal accuracy, multiple models have to
be trained to cover a wide range of testing SNR values, which significantly
increases the training time and storage required and is undesirable.

To tackle this issue of increased computational complexity and storage, this
section aims to enhance the models in a way such that the models can cater the
situations when there is a mismatch between the training and testing channel
conditions and minimize the degradation of model performances at such
situations. In particular, we will focus on the mismatch of training and
testing channel SNRs, $\text{SNR}_{train}$ and $\text{SNR}_{test}$. Based on
the previous success of the Attention Deep Learning based JSCC (ADJSCC) for
wireless image transmission [jscc_attention], one direction of enhancement to
make the previous proposed models SNR-aware is to make use of the soft-
attention mechanism at which the sets of symbols from each source will be
scaled dynamically according to the $\text{SNR}_{test}$. This way, the channel
SNR will be fed into the network as one of the inputs and the model will
become adaptive to the channel SNR. It is hoped that through incorporating the
information of the channel SNR to the model design, the degradation of the
model performance due to channel mismatch will be of a smaller extent.

###  IV-B Network Architecture

Firstly, the channel models of the two SNR-aware models remain the same as
that of their corresponding original models. Next, the overall network
architecture of the two SNR-aware models is the same as their corresponding
non-SNR-aware models, with the same overall flow and the same components used.
For the SNR-aware over-the-air model, the overall network architecture is as
shown in Figure 2. The only difference between the two SNR-aware models and
their corresponding non-SNR-aware two-source models is the structure of the
JSCC autoencoder. It is noted that the structures of the JSCC encoders and
decoders used in the SNR-aware orthogonal model are the same as the SNR-aware
over-the-air model, as the way the channel SNR is incorporated into the
network is the same. Figures 12 and 13 show the structures of the JSCC
encoders and decoders used in both the proposed SNR-aware orthogonal and SNR-
aware over-the-air model. Both of the JSCC encoders and decoders consists of
the attention feature (AF) modules, FC layers, BN layers and LeakyReLU
activation functions. Note that the two JSCC encoders have the identical
structure, and the two decoders have the same structure as well, with the same
channel SNR being fed into the network. The numbers of FC layers in the
encoders and the decoders are slightly different, at which each encoder has 3
FC layers and each decoder has 4 FC layers. Another point to note is that the
AF modules and the FC layers are interleaved, with the AF modules at the outer
layers. Such an ordering of the modules are found to be able to achieve the
highest top-1 accuracies through extensive architectural search.

The structure of each AF module used in both the JSCC encoders and decoders in
the proposed SNR-aware models is the same and is shown in Figure 14. The
working principle of the AF module is that it generates an attention mask
$\mathbf{s}$ which scales the input vector dynamically according to the
channel SNR. Similar to the AF modules in the original two-source models on
slow fading channel with CSI, there are 3 major components in the AF module:
context extraction, factor prediction and feature recalibration. During
context extraction, the channel SNR, which is a scalar, will be concatenated
with the input vector. The length of the resulting concatenated vector will
then be increased by 1 as compared to the input vector. Depending if the model
is in the training or testing stage, the channel SNR fed into the network will
be either $\text{SNR}_{train}$ or $\text{SNR}_{test}$. As the input vectors
within the JSCC encoders or the JSCC decoders are already flattened
themselves, no pooling is needed before the concatenation. Then, the
concatenated vector which has incorporated the channel SNR information is used
as the input for the factor prediction stage, in which a factor prediction
neural network is utilized to generate an attention mask $\mathbf{s}$ of the
same length as the input vector. With reference to the ADJSCC model
[jscc_attention], the factor prediction neural network is a simple network
which consists of 2 FC layers that are each followed by a ReLU or a sigmoid
function respectively. The sigmoid function is used to restrict the output
range to (0,1). With the attention mask $\mathbf{s}$ generated using the
channel SNR information, it is then multiplied element-wise with the input
vector at the last feature recalibration step.

Figure 12: Architecture of the JSCC encoders in the SNR-aware models Figure
13: Architecture of the JSCC decoders in the SNR-aware models Figure 14:
Structure of the AF module used in the JSCC autoencoder of the SNR-aware
models

###  IV-C Evaluation and Analysis

Regarding the training strategy of the SNR-aware models, the three-step
training strategy is adopted. The number of epochs, batch size, learning
rates, optimizer and regularization technique used for both of the SNR-aware
models are the same as that in the proposed training strategy of the original
two-source models. Nevertheless, one major difference in the training of the
SNR-aware models is that instead of training the model with a single fixed
channel SNR as in the original two-source models, the training channel SNR
($\text{SNR}_{train}$) of the SNR-aware models is sampled from a uniform
distribution between $[-6,15)$ dB for every iteration (per batch). This range
of training channel SNR is used as they are the channel SNRs that may be found
during the deployment stage at the practical applications. This training
strategy allows the network to be trained with a wide range of channel SNRs
instead of only a single channel SNR, so that the ability for the model to
generalize to a mismatched testing channel SNR ($\text{SNR}_{test}$) will be
higher, which in turn boosts the performance of the model.

After training the model with the given range of $\text{SNR}_{train}$, the
proposed SNR-aware models are then tested with a wide range of
$\text{SNR}_{test}$ from -6 to 18dB. To evaluate the performances of the two
proposed SNR-aware models, the top-1 retrieval accuracies obtained for the
SNR-aware models are compared with the original non-SNR-aware two-source
models trained with a single $\text{SNR}_{train}$.

(a) Orthogonal model - AWGN channel

(b) Orthogonal model - Slow fading channel without CSI

(c) Over-the-air model - AWGN channel

(d) Over-the-air model - Slow fading channel without CSI

Figure 15: Top-1 retrieval accuracies of the SNR-aware model and the original
models trained with $\text{SNR}_{train}=-6$, 0, 6, 12dB against different
$\text{SNR}_{test}$

####  IV-C1 Graceful degradation of original non-SNR-aware two-source models
during channel mismatch

The performances of the original two-source models trained on a single
$\text{SNR}_{train}$ when there is channel SNR mismatch between training and
testing are first evaluated. From the curves labelled $\text{SNR}_{train}=-6$,
0, 6, 12dB obtained in Figures 15(a), 15(b), 15(c) and 15(d), both the
orthogonal and over-the-air models trained with different $\text{SNR}_{train}$
exhibit graceful degradation when the testing channel conditions are worse
than that of the training channel conditions, on both the AWGN channel and
slow fading channel without CSI. This is evident from the fact that as
$\text{SNR}_{test}$ decreases (i.e. when the testing channel conditions become
worse), the retrieval accuracies of our original proposed two-source models
decrease smoothly. This shows that the use of JSCC in our proposed models in
the two-source scenario can still be able to eliminate the cliff effect which
conventional digital communication schemes suffer from, where the performances
of the digital models drop sharply when the channel conditions worsen.

For models trained with moderate $\text{SNR}_{train}$, the reduction of
accuracy when there is a mismatch between the $\text{SNR}_{train}$ and
$\text{SNR}_{test}$ (i.e. at extreme $\text{SNR}_{test}$ values) is within the
acceptable range, for the AWGN channel case. It becomes possible to train the
original proposed non-SNR-aware orthogonal and over-the-air models on the AWGN
channel with a moderate $\text{SNR}_{train}$ and apply it on a wide range of
$\text{SNR}_{test}$ in the testing stage, with a moderate compromise in the
retrieval accuracy. However, in the slow fading channel without CSI case, for
both the orthogonal and over-the-air model trained with
$\text{SNR}_{train}=0$dB, the drop in accuracy at $\text{SNR}_{test}$ higher
than 0dB becomes significant and not negligible. Given the significant drop of
accuracy at channel mismatch even for the two-source models trained with a
moderate $\text{SNR}_{train}$ on the slow fading channel, we cannot just train
the models with a moderate $\text{SNR}_{train}$ and apply it on a wide range
of $\text{SNR}_{test}$ during the deployment stage. Therefore, the original
proposed non-SNR-aware two-source models are not optimal during channel
mismatch under all channel conditions.

####  IV-C2 Performances of the SNR-aware models

In this section, the performances of the SNR-aware orthogonal and over-the-air
models are evaluated and compared with the original orthogonal and over-the-
air models trained with single $\text{SNR}_{train}$. From the SNR-aware model
curves in Figures 15(a) and 15(c), when tested on the AWGN channel, both of
the SNR-aware models either have similar or even higher retrieval accuracies
over the whole range of $\text{SNR}_{test}$ used, than all of the non-SNR-
aware original models trained with a single $\text{SNR}_{train}$. In other
words, the SNR-aware orthogonal and over-the-air models have provided an upper
bound for the two-source model performance, which are the best achievable
accuracies when there is channel SNR mismatch.

When the models are tested on the slow fading channel without CSI, from the
SNR-aware model curves in Figures 15(b) and 15(d), the accuracies achieved by
both of the SNR-aware models outperform the corresponding original models
trained with a single $\text{SNR}_{train}$ in most of the $\text{SNR}_{test}$
tested, with certain exceptions as noted below. For the SNR-aware orthogonal
model, its top-1 accuracies achieved are around $1-2\%$ lower than the
original orthogonal model trained with $\text{SNR}_{train}=12$dB for
$\text{SNR}_{test}\geq 12$dB (i.e. almost noiseless channels during testing).
This may be because at very high $\text{SNR}_{test}$ (i.e. with very low
channel additive noise), the performance of the SNR-aware orthogonal model may
have saturated and cannot reach the optimal accuracy. Nevertheless, the gap
between the accuracy achieved by the SNR-aware orthogonal model and the best
possible accuracy from the model trained with $\text{SNR}_{train}=12$dB is
very small and can be neglected. As for the SNR-aware over-the-air model, its
accuracy achieved is around $1.5\%$ lower than the original over-the-air model
trained with $\text{SNR}_{train}=0$dB at the testing channel SNR of
$\text{SNR}_{test}=-3$dB. As there are no other testing channel SNRs at which
the original model trained with $\text{SNR}_{train}=0$dB outperforms the SNR-
aware over-the-air model where no general trend can be found, the
aforementioned observation may just be an exceptional case at which the
original learns better as the $\text{SNR}_{test}$ is very similar to the
training $\text{SNR}_{train}$. Also, as the gap in performance of the two
models is very small, it is negligible and will not affect the overall
performance of the SNR-aware over-the-air model.

Therefore, even with the exceptions in the slow fading channel without CSI,
when we look at the whole $\text{SNR}_{test}$ range, the two SNR-aware models
still have much better performance than the original two-source models trained
with single $\text{SNR}_{train}$, as there is no model trained with
$\text{SNR}_{train}$ that can achieve high accuracy in a wide range of
$\text{SNR}_{test}$ (not even the model trained with moderate
$\text{SNR}_{train}$ like at 0dB). Hence, it can be concluded that SNR-aware
orthogonal and over-the-air models with attention modules outperform the
original orthogonal and over-the-air model over a wide range of
$\text{SNR}_{test}$, in both the AWGN channel and slow fading channel without
CSI.

##  V Conclusion

In this project, the deep learning-based JSCC AE model for single-source
person re-identification at the wireless edge proposed by Jankowski et al.
[jscc_ae] has been extended and further developed to the two-source
collaborative scenario successfully. A number of novel technical contributions
regarding the field of multi-source collaborative person re-identification at
the wireless edge have been made in this project, which could serve as a great
foundation of future research in the related area.

First of all, to ensure the compatibility of the dataset used in the two-
source collaborative person re-identification task at the wireless edge, the
suitability of the existing datasets in the literature is first determined and
Market-1501 is chosen to be the dataset for our application. Then, the entire
process of pre-processing on the existing Market-1501 dataset has been
outlined so that the pre-processed dataset contains image pairs with the same
identity from two different camera views which can be directly fed into the
two-source models. The resulting pre-processed Market-1501 dataset is thus
used in the testing and evaluation of the proposed models for our two-source
application in the rest of the project.

The next important technical contribution of this project is that we have
proposed two novel deep learning-based JSCC models for the task of two-source
collaborative person re-identification at the wireless edge, namely the
orthogonal model and the over-the-air model. For a more comprehensive
evaluation, the two models are evaluated extensively under a wide range of
channel SNRs and total available channel bandwidths, and on both the AWGN
channel and Rayleigh slow fading channel with and without CSI. Both models are
shown to outperform the single-source JSCC AE model in [jscc_ae] under a wide
range of channel conditions and on all channel types, with the over-the-air
model having even better performance than the orthogonal model. The reasons
behind the relative performances of the two models are analyzed, in particular
the high correlation between sources of the over-the-air model have been
validated experimentally to be one of the major reasons for the improvement of
its performance, through the notion of squared cosine similarity between the
input symbols. Other observations regarding the performances of the two models
under different channel types have also been listed and explained in detail.

Last but not least, we have proposed two novel SNR-aware deep-learning based
JSCC models with attention modules for our application to minimize the
degradation of model performance when the training and testing SNRs do not
match, which are very common in real-life applications. The two SNR-aware
models are evaluated extensively on a wide range of testing SNRs on both AWGN
and slow fading channels, and their performances are compared with the
orthogonal and over-the-air models described in Chapter 4 which are trained on
a single training SNR. It is first shown that the orthogonal and over-the-air
models proposed in Chapter 4 exhibit graceful degradation during training and
testing SNRs mismatch, which proves the superiority of the JSCC scheme over
conventional digital schemes even in the two-source collaborative scenario.
Then, it is further shown that the two SNR-aware models outperform their non-
SNR-aware counterparts, and provides an upper bound for the best possible
retrieval accuracies achievable over a wide range of testing channel SNRs.
This illustrates the high robustness towards channel mismatch between training
and testing, as well as the high practical applicability of the two proposed
SNR-aware models.

##  Appendix A Proof of the First Zonklar Equation

Appendix one text goes here.

## Appendix B

Appendix two text goes here.

## Acknowledgment

The authors would like to thank…

|  Michael Shell Biography text here.
---|---
John Doe Biography text here.
---
Jane Doe Biography text here.
---