# YOLO-based Semantic Communication with Generative AI-aided Resource
Allocation for Digital Twins Construction

Baoxia Du, Hongyang Du, Haifeng Liu, Dusit Niyato, Peng Xin, Jun Yu, Mingyang
Qi, You Tang This work was supported by Jilin Scientific and Technological
Development Program (YDZJ202201ZYTS692).B. Du, Y. Tang are with School of
Electrical and Information Engineering, JiLin Agricultural Science and
Technology University, Jilin 132101, China, and also with School of
Information and Control Engineering, Jilin Institute of Chemical Technology,
Jilin 132022 (e-mail<EMAIL_ADDRESS>and tangyou9000@163.com).H. Du,
and D. Niyato are with School of Computer Science and Engineering, Nanyang
Technological University, Singapore (e-mail<EMAIL_ADDRESS>and
dniyato@ntu.edu.sg).H. Liu is with College of Agriculture, Yanbian University,
Yanji 133002 (e-mail: liufeng_1989@163.com).P. Xin and J. Yu are with School
of Information and Control Engineering, Jilin Institute of Chemical
Technology, Jilin 132022 (e-mail<EMAIL_ADDRESS>and
yujun@jlict.edu.cn).M. Qi is with School of Electrical and Information
Engineering, JiLin Agricultural Science and Technology University, Jilin
132101, China (e-mail: qimingyang0912@126.com).Corresponding author: M. Qi and
Y. Tang.

###### Abstract

Digital Twins play a crucial role in bridging the physical and virtual worlds.
Given the dynamic and evolving characteristics of the physical world, a huge
volume of data transmission and exchange is necessary to attain synchronized
updates in the virtual world. In this paper, we propose a semantic
communication framework based on You Only Look Once (YOLO) to construct a
virtual apple orchard with the aim of mitigating the costs associated with
data transmission. Specifically, we first employ the YOLOv7-X object detector
to extract semantic information from captured images of edge devices, thereby
reducing the volume of transmitted data and saving transmission costs.
Afterwards, we quantify the importance of each semantic information by the
confidence generated through the object detector. Based on this, we propose
two resource allocation schemes, i.e., the confidence-based scheme and the
artificial intelligence-generated scheme, aimed at enhancing the transmission
quality of important semantic information. The proposed diffusion model
generates an optimal allocation scheme that outperforms both the average
allocation scheme and the confidence-based allocation scheme. Moreover, to
obtain semantic information more effectively, we enhance the detection
capability of the YOLOv7-X object detector by introducing new Efficient Layer
Aggregation Network-HorNet (ELAN-H) and SimAM attention modules, while
reducing the model parameters and computational complexity, making it easier
to run on edge devices with limited performance. The numerical results
indicate that our proposed semantic communication framework and resource
allocation schemes significantly reduce transmission costs while enhancing the
transmission quality of important information in communication services.

###### Index Terms:

Semantic communication, resource allocation, object detection, digital twins.

##  I Introduction

In recent years, the advances in technologies such as
Augmented/Extended/Virtual Reality (AR/XR/VR), blockchain, Sixth-Generation
(6G) network, artificial intelligence (AI) and edge computing have led to an
increasing demand for virtual reality and digital worlds. The metaverse, as a
virtual reality concept, is considered an integration of multiple virtual
worlds that can provide people with a more immersive and realistic digital
space [1, 2], allowing them to engage in various activities such as virtual
conferences, remote collaboration, online learning, digital exhibitions, etc.
The emergence of these virtual reality activities has not only alleviated
social isolation and transportation restrictions, but also saved time and
costs, gradually becoming essential tools for people’s lives and work.
Moreover, the popularity of these activities in various social domains has
also accelerated the development of digital economy and digital transformation
[3, 4].

In agriculture, novel paradigms such as digital farms, smart agriculture, and
agricultural metaverse, which are combined with metaverse technology, are
emerging and flourishing [4, 5]. In terms of agricultural production, users,
such as farmers, can establish virtual farms in a virtual environment,
simulate the complete growth process of crops and livestock, and obtain real-
time growth data to achieve intelligent and refined agriculture. For instance,
the XR Lab of Alibaba DAMO Academy presented a case study of an autonomous
agricultural picking robot. The proposed approach entails the development of a
high-precision Three-Dimensional (3-D) model of the entire orchard via 3-D
modeling techniques of both the orchard and fruit trees. Subsequently, a
motion planning scheme can be established in the virtual environment, which
can facilitate the robot’s autonomous picking process in the real world. This
innovative approach can potentially minimize the costs associated with orchard
management [6]. Furthermore, virtual agriculture can be combined with other
fields such as agricultural leisure and agricultural education. For example,
the Faculty of Agriculture has developed an agricultural metaverse teaching
system for an egg chicken farm at the National University of Laos. Through VR
technology, the faculty members provide agricultural education to learners,
including knowledge related to technology, farm location, and other relevant
aspects. The system has been reported to have yielded positive results [7].

Digital Twins (DTs), namely digital replications of physical objects, have
emerged as a pivotal technology for creating virtual environments [8, 9]. In
agriculture, the physical realm is characterized by its intricate and
constantly-changing nature, necessitating DTs synchronizing with the physical
world to ensure their accuracy in virtual operations. This process requires
that edge devices persistently gather the most recent data from the physical
world, enabling real-time DTs updates. The acquisition and transmission of the
data from the physical world often rely on various advanced edge devices and
wireless communication technologies [10, 11]. Various fixed or mobile devices
(e.g., sensors and cameras) are deployed to collect status data of physical
objects, which, in real-time, update and interact with the virtual world
through wireless communication. However, continuous data transmission poses
stringent requirements on wireless communication systems, especially when
dealing with extensive data such as high-definition images, which can be both
expensive and challenging when the physical world is vast.

Fortunately, semantic communication has been introduced as a novel avenue for
tackling the aforementioned challenge [12, 13]. In contrast to conventional
communication technologies, semantic communication systems regard transmission
effective if the meaning of the received information maintains the original
meaning of the transmitted information [14]. For example, in the context of
image transmission, a semantic-based communication system can reduce the
amount of data that needs to be transmitted by only transmitting the semantic
information behind the image, while achieving the same effect [15].

In this paper, we present a case study focusing on the development of a
virtual apple orchard using a real apple dataset. In the virtual orchard,
users, i.e., fruit growers, can easily access various information such as the
quantity and location of fruit on each apple tree, as well as growth status
and view real images of individual apples. The virtual orchard can help users
manage their orchard more efficiently. In this case, the implementation of DTs
requires edge devices, such as Unmanned Aerial Vehicle (UAV) to capture the
status information of fruit trees by taking photographs, and then transmitting
the collected data to users via wireless communication technology. To reduce
costs and enhance communication quality during this process, we propose a
semantic communication and resource allocation framework based on You Only
Look Once (YOLO). Our main contributions are summarized as follows:

  * •

Unlike traditional communication methods that necessitate transmitting all
acquired images, we propose a YOLO-based semantic communication framework.
Specifically, the proposed framework discards irrelevant interference
information after image data acquisition, retaining only the critical semantic
information for transmission. This significantly reduces the data volume
needed for transmission and lowers resource costs while achieving the same
outcome.

  * •

We employ the YOLOv7-X object detector to extract semantic information from
images and enhance its performance on a real-world apple dataset. Considering
the limitations of existing object detectors in detecting small objects such
as small apples, and the constraints of processing power and memory in edge
devices, larger models necessitate increased computational resources and
memory for operation, which may result in performance degradation or
inoperability. Consequently, we improve the YOLOv7-X algorithm by introducing
the Efficient Layer Aggregation Network-HorNet (ELAN-H) and the SimAM
attention modules. These modifications elevate the detector’s performance and
reduce the parameters and computational requirements, facilitating deployment
on edge devices with greater ease.

  * •

In the pursuit of enhancing transmission quality during the wireless
transmission of a huge volume of images, we propose a resource allocation
scheme based on the significance of semantic information. The scheme allocates
transmission power following the relative importance of the semantic
information, with the aim of enhancing the overall communication quality of
image transmission systems by minimizing important information loss and
improving the reliability of transmitted information. This approach ensures
that critical information is transmitted with high quality to users, even in
challenging wireless communication environments.

  * •

Furthermore, we utilize the AI-generated resource allocation scheme algorithm
as an alternative allocation scheme, which facilitates more efficient
processing power distribution. Specifically, by using the denoising technique,
the AI-generated algorithm generates a design for the allocation scheme and
subsequently adds exploration noise to execute it, thereby gaining experience
in exploration. The numerical results clearly demonstrate that this method
achieves the highest score in terms of semantic information transmission
quality.

The remainder of this paper is organized as follows: In Section II, we
initially summarize the related work about DTs, semantic communication and
apple detection. Section III introduces the overall system design, semantic
communication approach, and the metric used for evaluating the system’s
communication quality. In Section IV, the YOLOv7-X object detector and its
enhancement methods are described in detail, followed by an explanation of two
distinct resource allocation methods for data transmission. Subsequently, we
analyze the numerical results in Section V. Lastly, Section VI concludes the
paper.

##  II Related Work

In this section, we briefly introduce three related techniques, i.e., DTs,
semantic communication and apple detection.

###  II-A Digital Twins

The physical system and physical world in agriculture are complex and dynamic
environments that include basic information and characteristics of physical
objects. DTs require continuous updating from the physical to the virtual
space as the state of physical objects changes over time [16, 17]. Li et al.
[18] proposed a deep learning-based single-view leaf reconstruction method for
a plant growth DT system, improving leaf reconstruction’s accuracy and speed.
Angin et al. [19] introduced a DT framework for agriculture called AgriLoRa,
which detects plant diseases and weeds using computer vision algorithms after
uploading data from UAV images and field sensor data to cloud servers. Awais
et al. [20] used the multispectral UAV and DTs model to achieve intelligent
irrigation in the field. However, these works have focused only on the use of
collected data and have not considered the impact of data transmission. DTs
require a significant amount of computing power to render 3-D objects, which
is achieved through collecting large amounts of data from perception networks
and ultra-low latency communication to maintain a seamless user experience
[1]. When physical objects are large enough, the massive data streams can
burden communication systems and cause excessive latency, or even transmission
failure. Therefore, in this paper, we use a semantic-aware communication
method to reduce the amount of data that needs to be transmitted.

###  II-B Semantic Communication

In classical communication theory, the semantic content and meaning of the
message are largely considered irrelevant to communication. However, in the
age of rapidly increasing data volume, the limitations of classical
communication theory have begun to be revealed [21]. Semantic communication
constitutes an innovative paradigm wherein message transmission is not
confined solely to the message content, but rather entails direct extraction
of pertinent semantic information, thereby eliminating redundant data and
mitigating associated costs. Xie et al. [22] proposed a text transfer
framework called DeepSC based on the Transformer [23], which can recover the
meaning of sentences through semantic information, thus minimizing semantic
errors during transmission. Zhou et al. [24] proposed a cognitive semantic
communication framework that utilizes knowledge graphs, which has good data
compression rates and communication reliability. In addition to text-based
semantic communication, some literature also proposes semantic communication
methods applied to images. [25] obtained semantic information through semantic
segmentation at the transmitter and used GAN networks to reconstruct the image
at the receiver, greatly saving bandwidth resources, but the reconstructed
image is slightly different from reality. Zhang et al. [26] proposed a neural
network-based image transfer semantic communication system, where the
transmitter can extract and transmit the required semantic information in a
dynamic environment through a receiver-leading training process without
knowing the task. Kang et al. [27] proposed a task-oriented semantic
communication framework, where users can match the semantic information of
images by querying text, and also consider the resource allocation problem
when there are multiple users. Although the above literature reduce
communication overhead through semantic communication, they have not
considered the varying importance of semantic information itself, which may
result in the loss of significant semantic information in the competition for
channel resources. Therefore, we assign different levels of importance to the
semantic information extracted from images to ensure the transmission quality
of critical semantic information by rationally allocating transmission power.

###  II-C Object Detection

In the communication framework proposed in this paper, the UAV needs to
extract the semantic information of the acquired images, i.e., to achieve the
separation of apples and backgrounds, and the core of achieving this is the
object detection technique.

Apple Detection: In recent years, deep learning-based object detection
techniques have achieved remarkable success. In contrast to traditional
algorithms that rely on appearance features such as shape and color [28, 29],
deep learning-based techniques demonstrate strong adaptability to different
scenarios and achieve higher accuracy. Chen et al. [30] utilized the DenseNet
network structure to optimize the YOLOv4 model, proposing a Des-YOLOv4
algorithm for detecting apples. However, the performance of the algorithm
significantly deteriorates under low-light conditions. Yan et al. [31]
proposed an improved YOLOv5 [32] algorithm for real-time apple recognition by
incorporating Squeeze-and-Excitation (SE) modules and modifying the fusion
mode of feature maps. Despite the improved performance, the algorithm’s
effectiveness in detecting small apples is suboptimal. Sun et al. [33]
proposed a novel Balanced Feature Pyramid Network (BFP Net) that enhances the
accuracy of small apple detection. Nevertheless, the BFP Net has a slower
detection speed. The above work indicates that it is difficult to balance the
detection speed and accuracy of the model, and detecting small apples in
complex environments remains a challenge. Therefore, we select the YOLOv7-X
object detector for this study due to its outstanding detection speed and
accuracy [34]. However, its performance on dataset with a considerable number
of small apples is slightly limited. Thus, we employed it as the baseline
model for improving detection performance.

Data Augmentation: In addition to modifying the model structure to improve
detection capability, data augmentation is another straightforward and
effective approach. Data augmentation allows generating additional equally
effective data based on limited data without altering the essential
information of the images. This significantly enhances the diversity of the
training data, thereby enhancing the model with stronger generalization
capabilities. Changing the color and shape of images is the fundamental and
common approach in data augmentation. In this study, we use fundamental data
augmentation methods such as randomly altering the hue, saturation, and
brightness of images, as well as performing random scaling and translation. In
addition to these basic data augmentation methods, some studies have proposed
more efficient approaches, such as Mosaic [35] and Mixup [36]. Mosaic involves
randomly cropping and scaling four images, then combining them into a single
image for training data. Mixup randomly selects two samples from the training
data and constructs new training samples and labels through linear
interpolation. In this work, we use a combination of Mosaic, Mixup and
fundamental methods.

In summary, we propose a YOLO-based semantic communication and resource
allocation framework to address the above problems in this paper. First, we
select the advanced YOLOv7-X object detector as the core of the entire system
and optimize its performance on real dataset. Following that, the UAV extracts
semantic information from collected images using the optimized YOLOv7-X object
detector. Then, the UAV allocates transmission power based on the importance
of semantic information before transmitting, ultimately achieving high-quality
transmission of critical information.

##  III System Model

In this section, we present the proposed data transmission framework and the
semantic-based communication approach, followed by an explanation of the
metric methodology for assessing image semantic information transmission
quality.

###  III-A Design of semantic communication and resource allocation framework

In data collection in DTs, a UAV takes images along a specified trajectory and
transmit the data to users. However, in wireless communication environments
with fading channels, the cost of transmitting all data to users at high
quality is prohibitive. To address this issue, we propose a semantic
communication framework based on YOLO, as shown in Fig. 1. We first simulate
the images captured by the UAV using real apple dataset. After the collection
task is completed, the UAV uses the trained YOLOv7-X object detector to
extract semantic information behind the image that people need and transmit
them. At the same time, we quantify the importance of each semantic
information based on the confidence generated by the object detector and
allocate transmission power accordingly to ensure the transmission quality of
important information. The problem of optimal power allocation schemes is
introduced in Section IV. It is worth noting that the YOLOv7-X object detector
is a flexible module that can be applied to semantic communication systems in
various scenarios by training on different datasets. For instance, use
pretrained models to extract semantic information from images containing other
single or multiple different classes of objects.

Figure 1: An illustration of the YOLO-based semantic communication system
model.

###  III-B Semantic Communication Solution

In conventional communication paradigms, edge devices transmit the entirety of
the acquired image data to facilitate continuous data synchronization for DTs,
resulting in voluminous data traffic. This imposes substantial burdens on both
edge devices and the communication infrastructure. To illustrate, in smart
agriculture, users deploy UAVs for image capture [37]. However, oftentimes,
only a portion of the captured content is relevant to users, e.g., ripening
fruits. Utilizing such an inefficient communication approach for transmitting
all images leads to the excessive consumption of communication resources and
energy for UAVs, thereby exacerbating resource wastage.

The development of semantic communication provides a solution to the
aforementioned problems. Upon completing the designated data acquisition task,
the UAV transmits only the pertinent semantic features extracted from the
captured images, rather than the entire dataset. The transmission of these
semantic features requires minimal channel resources and facilitates efficient
data storage for users. In the virtual orchard, users, e.g., fruit growers,
are primarily more concerned about the status of the fruits. As a result, we
discard irrelevant background and interference factors before transmitting the
images, ensuring that users only receive the semantic information they are
interested in, which helps reduce transmission cost.

###  III-C Semantic Communication Quality Analysis

To evaluate the quality of semantic communication, we propose a Metric for
Image Semantic Transmission (MIST) in this work, which involves combining the
importance weights of each semantic information with their respective
transmission quality to obtain the final evaluation result. Considering that a
UAV needs to send an image to the user after capturing it, semantic
information is first extracted by the object detector. Specifically, a total
of $U$ apple objects are detected, with $i$ denoting the $i_{\rm th}$ object
and ${c_{i}}$ denoting its corresponding confidence. The relationship between
the importance score ${W_{i}}$ and the confidence ${c_{i}}$ of the object $i$
can be represented as ${W_{i}}={c_{i}}^{\sigma},$ where $\sigma$ is a variable
that adjusts the importances among different semantic information. The final
semantic transmission quality score can be represented as follows:

$E(A,{W_{i}},Q({p_{\rm{i}}}))=A\sum\limits_{i=1}^{U}{({W_{i}}\times
Q({p_{\rm{i}}}))},$ (1)

where $A$ represents the accuracy of semantic information extraction, i.e.,
the performance evaluation metric Average Precision at 0.5 Intersection over
Union<EMAIL_ADDRESS>of the object detector, and $Q({p_{\rm{i}}})$ represents the
Structure Similarity Index Measure (SSIM) [38] value of object $i$ before and
after transmission, which is a function that is positively correlated with the
transmission power ${p_{i}}$ according to [27]. Therefore, considering the
definition of the communication system in this paper, our goal is to maximize
the MIST while satisfying the transmission power constraint, which can be
expressed as follows:

$\displaystyle\mathop{\max}\limits_{A,{W_{i}},{p_{i}}}\sum\limits_{i=1}^{U}$
$\displaystyle{E(A,{W_{i}},Q({p_{\rm{i}}})))},$ (2)
$\displaystyle\sum\limits_{{\rm{i}}=1}^{U}$ $\displaystyle{{p_{i}}}\leq P,$
(2a) $\displaystyle{{\rm{c}}_{i}}\in$ $\displaystyle[{{\rm{c}}_{\min}},1],$
(2b)

where the constraint in (2a) is the total transmitted power, and ${c_{\min}}$
in (2b) is the confidence threshold, the objects with confidence below this
value are not detected by the detector. In this study, we set the variable
${\sigma}=1$ and the confidence threshold ${c_{\min}}=0.25$ by default. The
proposed MIST considers not only the transmission quality of each semantic
information but also their respective significance, which can provide a more
comprehensive and accurate assessment of the performance of these methods.

##  IV YOLO-based Semantic Communication System Design

The YOLOv7-X object detector is a critical component of the overall
communication system. It aids UAVs in extracting semantic information from
images and subsequently quantifying the semantic information’s importance to
facilitate optimal transmission power allocation. In this section, we provide
a detailed exposition of the YOLOv7-X, along with the improvements made to it.
Subsequently, we discuss two distinct resource allocation schemes in data
transmission.

###  IV-A Overview of YOLOv7-X

The YOLOv7 model is a highly advanced and efficient end-to-end object
detector. It employs a state-of-the-art methodology for detecting objects in
an image, with exceptional accuracy and real-time performance. YOLOv7 has
seven different models of varying sizes, including YOLOv7-tiny, YOLOv7,
YOLOv7-X, and YOLOv7-W6, among others, which are suitable for different
application environments. Considering both model complexity and detection
performance, we select YOLOv7-X as the base model for improvement. The
YOLOv7-X model can be divided into three parts: Input, Backbone, and Head.
Specifically, the Input resizes the input image to the required training size.
The Backbone component includes multiple CBS convolutions, Max Pooling
Convolution (MPConv), Efficient Layer Aggregation Network in YOLOv7-X (ELAN-X)
modules and an SPPCSPC module. The CBS convolution consists of a convolutional
layer, a Batch Normalization layer (BN), and a Sigmoid Linear Unit (SiLU)
activation function. ELAN-X extends the ELAN module by increasing its depth
and width, and enhances the learning capability of the network by guiding the
computation blocks to learn more diverse features of different feature groups.
The MPConv module adds a Maxpool layer on top of CBS and strengthens the
feature extraction ability by merging features from the top and bottom
branches. The SPPCSPC module is similar to the Spatial Pyramid Pooling-Fast
(SPPF) used by YOLOv5 [32], which increases a network’s receptive field. The
Head component employs the same Path Aggregation Feature Pyramid Network
(PAFPN) [39] architecture as YOLOv5 to efficiently fuse features from multiple
levels. Finally, the fused and enhanced feature map is fed to three detection
heads to generate predictions for confidence, object category, and anchor
boxes.

###  IV-B Model Enhancement Methods

ELAN-H: The ELAN-X module in YOLOv7-X is an efficient network structure that
enables the network to learn more features and have stronger robustness by
controlling the shortest and longest gradient paths. The structure of ELAN-X
is shown in Fig. 4. The shortest branch passes through only one CBS
convolution to change the number of channels, while the longest branch
extracts features through seven CBS convolutions. The feature maps extracted
by each branch are concatenated through the Concatenation (Concat) operation
as the final result of feature-enhanced fusion. ELAN-X is repeatedly used in
the Neck to improve the model’s learning capability. However, too many
branches and convolution operations also increase model complexity and
parameter size, leading to increased processing time and consumption of
computing resources. Therefore, we reduce the depth and width of this module
without breaking its original architecture, changing the number of CBS in the
longest branch to three and correspondingly decreasing the number of output
feature maps to four. In addition, to compensate for the decrease in detection
performance caused by simplifying this module, we replace one CBS convolution
in the long branch with Hornet Block [40] to enhance the module’s ability to
learn important features. Finally, this module, named ELAN-H, has a structure
shown in Fig. 2(a).

(a) ELAN-H model structure, which stacks feature maps from different levels by
Concatenation (Concat) operation as the final output.

(b) Overview of the basic building block in HorNet with Recursive Gated
Convolution (${g^{n}}{\rm{Conv}}$).

Figure 2: ELAN-H model structure with HorNet.

Here, the Hornet block is a design based on Recursive Gated Convolution
(${{g}^{n}}\text{Conv}$). The output of the gated convolution
${\bf{y}}=g{\rm{Conv(}}{\bf{x}}{\rm{)}}$ can be represented as follows:

$\left[{{{\bf{p}}_{0}}^{HW\times C},{\bf{q}}_{0}^{HW\times
C}}\right]={\phi_{in}}({\bf{x}})\in{{\rm{R}}^{HW\times 2C}},$ (3)
${{\bf{p}}_{1}}=f({{\bf{q}}_{0}})\odot{{\bf{p}}_{0}}\in{R^{HW\times
C}},y={\phi_{out}}({{\bf{p}}_{1}})\in{R^{HW\times C}},$ (4)

where ${\bf{x}}\in{{\rm{R}}^{HW\times C}}$ represents the input features which
are linearly projected to obtain ${{\bf{p}}_{0}}$ and ${{\bf{q}}_{0}}$. Then,
${{\bf{q}}_{0}}$ is subjected to depth-wise convolution and multiplied with
${{\bf{p}}_{0}}$ to obtain ${{\bf{p}}_{1}}$. Lastly, ${{\bf{p}}_{1}}$ is
projected linearly again to yield the output ${\bf{y}}$.

High-order spatial interaction requires the implementation of gated
convolutions with recursive designs. Initially, a higher-order linear
projection is applied to ${\bf{x}}$ resulting in ${{\bf{p}}_{0}}$ and
${{\bf{q}}_{k}}(k=0,1,...,n-1)$. Subsequently, recursive gated convolutions
are executed to generate ${{\bf{p}}_{k+1}}$. The output of
${\bf{y}}={g^{n}}{\rm{Conv(}}{\bf{x}}{\rm{)}}$ can be mathematically expressed
as follows:

$\begin{split}\left[{{{\bf{p}}_{0}}^{HW\times{C_{0}}},{\bf{q}}_{0}^{HW\times{C_{0}}},...,{\bf{q}}_{n-1}^{HW\times{C_{n-1}}}}\right]\\\
={\phi_{in}}({\bf{x}})\in{{\rm{R}}^{HW\times({C_{0}}+\sum\nolimits_{0\leq
k\leq n-1}{{C_{k}}})}},\end{split}$ (5)
${{\bf{p}}_{k+1}}={f_{k}}({{\bf{q}}_{k}})\odot{g_{k}}({{\bf{p}}_{k}})/\alpha,\qquad
k=0,1,...,n-1,$ (6)
${g_{k}}=\left\\{\begin{array}[]{l}{\rm{Identity,}}\>\qquad\qquad\qquad
k{\rm{=0,}}\\\ {\rm{Linear}}({C_{k-1}},{C_{k}}),\quad\quad 1\leq k\leq
n-1.\end{array}\right.,$ (7)
${C_{k}}=\frac{C}{{{2^{n-k-1}}}},\quad\qquad\qquad 0\leq k\leq n-1,$ (8)

where $\\{{g_{k}}\\}$ are utilized to match the dimension in various orders
and $\\{{f_{k}}\\}$ are depth-wise convolution layers. As depicted in Fig.
2(b), the HorNet block employs a block-wise design inspired by Transformer
[23] and replace the self-attention sub-layer with ${{g}^{n}}\text{Conv}$ that
have high-order spatial modeling capability. We replace all ELAN-X modules in
the Neck with ELAN-H modules, which can better fuse and enhance the image
features extracted by the Backbone, leading to improved detection performance
while reducing model complexity.

SimAM [41]: The attention module can assign different weights to different
channels or regions in space, thereby helping the model to focus on extracting
more important information. Existing attention mechanisms typically generate
corresponding 1-D or 2-D weights in the channel or spatial dimension, as shown
in Fig. 3(a) and Fig. 3(b), such as BAM [42] , which parallelly connects two
kinds of attention, and CBAM [43], which serially connects them. However, they
treat each neuron in every channel or spatial position equally during the
generation process. This limitation restricts their ability to learn more
discriminative cues, while in the human brain, these two types of attention
often occur simultaneously [41].

(a) Channel-wise attention.

(b) Spatial-wise attention.

(c) Full 3-D weights for attention.

Figure 3: Comparisons of different attention steps.

As shown in Fig.3(c), SimAM is a unified weight attention module that can
derive 3-D attention weights for feature maps without requiring additional
parameters. In visual neuroscience, the most informative neurons typically
exhibit discharge patterns that differ from those of surrounding neurons, and
active neurons tend to inhibit surrounding neurons [44]. Drawing inspiration
from this, the SimAM module designs an energy function to measure the linear
separability between neurons, thereby identifying important neurons. The
energy function is defined as follows:

$\begin{split}e_{t}(w_{t},b_{t},y,x_{i})&=\frac{1}{{M-1}}\sum\limits_{i=1}^{M-1}{{{(-1-(w_{t}x_{i}+b_{t}))}^{2}}}\\\
&+{(1-(w_{t}t+b_{t}))^{2}}+\lambda w_{t}^{2},\end{split}$ (9)

where $t$ denotes the target neuron and ${{x}_{i}}$ denotes other neurons in a
single channel of the input feature $\mathbf{x}\in{{\text{R}}^{HW\times C}}$,
$i$ is index over spatial dimension and $M$ is the number of neurons. $w_{t}$
and ${b_{t}}$ are weight and bias the transform. Subsequently, by computing
the closed-form solutions of variables ${{w}_{t}}$ and ${{b}_{t}}$, and
substituting them into (9). The minimum energy can be obtained as follows:

$e_{t}^{*}=\frac{{4({{\hat{\sigma}}^{2}}+\lambda)}}{{{{(t-\hat{\mu})}^{2}}+2{{\hat{\mu}}^{2}}+2\lambda}}.$
(10)

The above formula indicates that a smaller energy value corresponds to a
greater linear separability between neuron $t$ and other neurons, which
signifies higher importance. Following the definition of attention mechanisms,
the SimAM module can be expressed as follows:

$\tilde{X}={\rm{sigmoid(}}\frac{1}{{\mathop{\rm
E}\nolimits}}{\rm{)}}\odot{\rm{X}},$ (11)

where, $\operatorname{E}$ is a classification function for all energy
functions $e_{t}^{*}$ in both channel and spatial dimensions. The sigmoid
function ensures that the larger values in $\operatorname{E}$ are constrained,
thereby not affecting the relative importance of each neuron. We integrate
SimAM modules into the Neck section of the YOLOv7-X model, which helps the
model to better focus on targets without introducing additional parameters.

YOLOv7-HS: The improved architecture of the YOLOv7-X model is depicted in Fig.
4. We enhance the YOLOv7-X object detector using the above methods to make it
more suitable for the Minneapple dataset, and named it YOLOv7-HS.

Figure 4: The network architecture of YOLOv7-HS contains general modules:
Input, Backbone, and Head, and basic components: CBS, Max Pooling Convolution
(MPConv), UPSampling (UP), SPPCSPC, SimAM, and two Efficient Layer Aggregation
Networks (ELAN) with different structures, namely ELAN-X and ELAN-H.

###  IV-C Resource Allocation in Semantic Communication

After the UAV obtains the semantic information of images, a straightforward
approach is to transmit it equally to users. However, due to the limited
bandwidth resources in wireless transmission, semantic information is
susceptible to signal attenuation during transmission, consequently affecting
communication quality. Simultaneously, as nearly every original image is
cropped into numerous apple images, and considering the highly complex orchard
environment and the inherent limitations of the detection model, the
importance of each cropped image (e.g., apple completeness, cropping accuracy)
varies. The average allocation method (named Avg-SemCom) faces intense
competition for scarce wireless channel resources among multiple images,
resulting in the discarding of crucial images. Therefore, we propose a
Confidence-based Semantic Communication (Conf-SemCom) method, which allocates
transmission power by quantifying the importance of semantic information to
ensure the transmission quality of important data.

The correlation between an object detector’s confidence and the object it
identifies is significant. Generally, the confidence of a detected object is
positively correlated with the amount of semantic features it contains. As
illustrated in Fig. 5, apples $f$, $g$, and $h$ exhibit relatively complete
and rich characteristic information of apples, making them easily detectable
with a high confidence. While apples $a$ and $c$ are heavily obscured by
leaves, resulting in low confidence scores for all of them. The semantic
information of these low-confidence objects, after being cropped, is far less
important than the other objects, and they should not occupy too much power
during wireless transmission. In summary, the Conf-SemCom method can ensure
that more important information is successfully transmitted to users.

Figure 5: The results of apple detection comprise anchor boxes and
corresponding confidence values for each object, apples a to f are sorted in
ascending order of confidence scores. Furthermore, the cropped images are
resized to the same height for ease of viewing.

Specifically, we prioritize sorting based on the confidence ${c_{i}}$ of each
object ${i}$ and allocate more transmission power with a higher priority
weight ${\bm{w}_{i}}$. The definition of ${\bm{w}_{i}}$ is as follows:

${\bm{w}_{i}}={c_{i}}^{\eta},$ (12)

we use the variable $\eta$ to adjust the relative difference in power
allocation between different semantic information. The proposed Conf-SemCom is
summarized in Algorithm 1 with the corresponding pseudo code.

Algorithm 1 Resource allocation in Conf-SemCom

Input: Captured image $X$ on a UAV
Output: Users receive semantic information $x$ of the image $X$

1:procedure UAV-SEND($x$)

2: YOLOv7-HS detects apple images ${x_{1}}$,${x_{2}}$,…,${x_{i}}$,…,${x_{U}}$
and their confidence ${c_{1}}$,${c_{2}}$,…,${c_{i}}$,…,${c_{U}}$

3: Calculate the priority weight ${\bm{w}_{i}}$ of ${x_{i}}$ by confidence
${c_{i}}$ according to (12)

4: for $i=1$ to $U$ do

5: Allocate transmission power to ${x_{i}}$ according to their priority weight
${\bm{w}_{i}}$ using PA [27]

6: end for

7: The UAV sends semantic information $x$ to users

8:end procedure

###  IV-D Diffusion-Based Resource Allocation

Diffusion model: The diffusion model has emerged as a new state-of-the-art
deep generative model [45]. The fundamental concept of the diffusion model
entails systematically perturbing the distribution of data during the forward
diffusion process by introducing Gaussian noise. Subsequently, the data
distribution is recovered through the reverse diffusion process, which can be
viewed as a denoising procedure. Specifically, within the forward diffusion
process, by iteratively adding Gaussian noise $T$ times to any initial sample
${{\mathbf{x}}_{0}}$, we can obtain
${{\mathbf{x}}_{1}},{{\mathbf{x}}_{2}},...,{{\mathbf{x}}_{T}}$. As $T$
approaches infinity, the original features of sample ${{\mathbf{x}}_{0}}$
completely vanish and become pure Gaussian noise. This process can be
represented as follows:

$q\left({{{\mathbf{x}}_{1}},...,{{\bf{x}}_{T}}\mid{{\bf{x}}_{0}}}\right)=\mathop{\prod}\limits_{t=1}^{T}q\left({{{\bf{x}}_{t}}\mid{{\bf{x}}_{t-1}}}\right),$
(13) $q\left({{{\bf{x}}_{t}}\mid{{\bf{x}}_{t-1}}}\right):={\cal
N}\left({{{\bf{x}}_{t}};\sqrt{1-{\beta_{t}}}{{\bf{x}}_{t-1}},{\beta_{t}}{\bf{I}}}\right),$
(14)

where, $\beta_{t}$ is a parameter that controls the progress of noise. From
(14), it can be inferred that given the sample ${\bf{x}}_{t-1}$, the sample
${\bf{x}}_{t}$ at time $t$ follows a Gaussian distribution with a mean of
$\sqrt{1-{\beta_{t}}}{{\bf{x}}_{t-1}}$ and a variance of
${\beta_{t}}{\bf{I}}$. The parameters under this condition only depend on the
${\bf{x}}_{t{\rm{-}}1}$ at the previous time step. Therefore, the diffusion
process is a Markov process.

When $\beta_{t}$ is sufficiently small, the reverse diffusion process
$q\left({{{\bf{x}}_{t{\rm{-}}1}}\mid{{\bf{x}}_{t}},{{\bf{x}}_{0}}}\right)$ is
the posterior probability distribution of the forward diffusion process
$q\left({{{\bf{x}}_{t}}\mid{{\bf{x}}_{t-1}}}\right)$. In order to achieve
incremental sampling from Gaussian noise ${{\bf{x}}_{T}}$ to obtain real
samples, it is necessary for the generative model
${p_{\theta}}\left({{{\bf{x}}_{0:T}}}\right)$ to learn sufficiently good
parameters $\theta$ from the training samples. This process can be represented
as follows:

${p_{\theta}}\left({{{\bf{x}}_{0:T}}}\right)=p\left({{{\bf{x}}_{T}}}\right)\prod\limits_{t=1}^{T}{{p_{\theta}}}\left({{{\bf{x}}_{t-1}}\mid{{\bf{x}}_{t}}}\right),$
(15) ${p_{\theta}}\left({{{\bf{x}}_{t-1}}\mid{{\bf{x}}_{t}}}\right)={\cal
N}\left({{{\bf{x}}_{t-1}};{\mu_{\theta}}\left({{{\bf{x}}_{t}},t}\right),{\Sigma_{\theta}}\left({{{\bf{x}}_{t}},t}\right)}\right),$
(16)

where, $p\left({{{\bf{x}}_{T}}}\right){\rm{=}}{\cal
N}\left({{{\bf{x}}_{T}};0,{\bf{I}}}\right)$. Finally, the reverse diffusion
process can be achieved by utilizing a well-trained
${p_{\theta}}\left({{{\bf{x}}_{t-1}}\mid{{\bf{x}}_{t}}}\right)$ to approximate
$q\left({{{\bf{x}}_{t{\rm{-}}1}}\mid{{\bf{x}}_{t}},{{\bf{x}}_{0}}}\right)$.

AI-Generated Power Allocation Scheme: Motivated by diffusion model-based AI-
generated contract [46], we propose an AI-generated power allocation scheme.
In Deep Reinforcement Learning (DRL), the intelligent agent learns the optimal
policy through interaction with the environment to maximize cumulative
rewards. As shown in Fig. 6, the AI-generated algorithm is capable of
addressing the challenges posed by high-dimensional state spaces and complex
action spaces. Compared to the aforementioned two methods, i.e., Avg-SemCom
and Conf-SemCom, this method exhibits superior performance in generating
resource allocation schemes. Specifically, we represent the environment using
the vector $\bm{e}$, which encompasses various factors such as the wireless
channel model, the transmission power $P$, and the number of objects $U$
involved in the semantic communication. In this given environment, our
objective is to maximize the expected cumulative reward across a series of
time steps, aiming to determine the transmission power weights ${\bm{w}_{i}}$
for each object.

Figure 6: The design principles of diffusion model. The diffusion model
performs multi-step denoising on noise and generates an optimal allocation
scheme.

We first construct a generative model
${\pi_{\theta}}\left({\bm{w}\mid\bm{e}}\right)$ that can map the environmental
state $\bm{e}$. The reverse process of the conditional diffusion model can be
represented as follows:

$\begin{split}{\pi_{\theta}}\left({\bm{w}\mid\bm{e}}\right)&={p_{\theta}}\left({{\bm{w}^{0:N}}\mid\bm{e}}\right)\\\
&={\cal
N}\left({{\bm{w}^{N}};0,{\bf{I}}}\right)\mathop{\prod}\limits_{i=1}^{N}{p_{\theta}}\left({{\bm{w}^{j-1}}\mid{\bm{w}^{j}}\bm{,e}}\right),\end{split}$
(17)

where, ${p_{\theta}}\left({{\bm{w}^{j-1}}\mid{\bm{w}^{j}}\bm{,e}}\right)$ can
be modeled as a Gaussian distribution ${\cal
N}\left({{\bm{w}^{{}_{j-1}}};{\mu_{\theta}}\left({{\bm{w}^{j}},\bm{e},j}\right),{\Sigma_{\theta}}\left({{\bm{w}^{j}},\bm{e},j}\right)}\right)$.
According to Denoising Diffusion Probabilistic Models (DDPM) [47], the
covariance matrix ${\Sigma_{\theta}}\left({{\bm{w}^{j}},\bm{e},j}\right)$ of
this Gaussian distribution is ${\beta_{j}}{\bf{I}}$, and the mean
${\bm{\mu}_{\theta}}\left({{\bm{w}^{j}},\bm{e},j}\right)$ can be represented
as
$\frac{1}{{\sqrt{{\alpha_{j}}}}}\left({{\bm{w}^{j}}-\frac{{{\beta_{j}}}}{{\sqrt{1-{{\bar{\alpha}}_{j}}}}}{\varepsilon_{\theta}}\left({{\bm{w}^{j}},\bm{e},j}\right)}\right)$.
Initially, we sample ${\bm{w}^{N}}\sim{\cal N}\left({0,{\bf{I}}}\right)$, and
then from the reverse diffusion chain parameterized by $\theta$ as:

${\bm{w}^{j-1}}|{\bm{w}^{j}}=\frac{1}{{\sqrt{{\alpha_{j}}}}}\left({{\bm{w}^{j}}-\frac{{{\beta_{j}}}}{{\sqrt{1-{{\bar{\alpha}}_{j}}}}}{\varepsilon_{\theta}}\left({{\bm{w}^{j}},\bm{e},j}\right)}\right)+\sqrt{{\beta_{j}}}\varepsilon.$
(18)

From (18), as can be seen that the result is only related to ${\bm{w}^{j}}$
and the added noise $\varepsilon$. Therefore, the training of the denoising
process ${\pi_{\theta}}$ can be achieved by training ${\varepsilon_{\theta}}$.
Subsequently, we use the quality network ${Q_{v}}$ to train the
${\varepsilon_{\theta}}$, which represents the expected cumulative reward that
an agent takes an allocation scheme in the current state and executes
accordingly. The objective function that needs to be optimized becomes:

$\pi=\underset{\pi_{\theta}}{\arg\min}\mathcal{L}(\theta)=-\mathbb{E}_{\bm{w}^{0}\sim\pi_{\theta}}\left[Q_{v}\left(\bm{e},\bm{w}^{0}\right)\right].$
(19)

Utilizing the double Q-learning method [48], the network ${Q_{v}}$ is learned
by minimizing the Bellman operator. Consequently, we construct two networks,
specifically ${Q_{v_{1}}}$ and ${Q_{v_{2}}}$, along with corresponding target
networks ${Q_{v_{1}^{{}^{\prime}}}}$, ${Q_{v_{2}^{{}^{\prime}}}}$, and
${\pi_{{\theta^{{}^{\prime}}}}}$. The optimization of ${v_{1}}$ and ${v_{2}}$
is achieved by minimizing the objective

$\mathbb{E}_{\bm{w}_{t+1}^{0}\sim\pi_{{\theta}^{\prime}}}\left[{{{\left\|\begin{array}[]{l}\left({r\left({\bm{e},{\bm{w}_{t}}}\right)+\gamma\mathop{\min}\limits_{i=1,2}{Q_{v_{i}^{\prime}}}\left({\bm{e},\bm{w}_{t+1}^{0}}\right)}\right)\\\
-{Q_{{v_{i}}}}\left({\bm{e},{\bm{w}_{t}}}\right)\end{array}\right\|}^{2}}}\right].$
(20)

In DRL, we set the training parameters: batch size ${N_{b}}$, discount factor
$\gamma$, diffusion step $N$, soft target update parameter $\tau$, and
exploration noise $\epsilon$. The loss function can be represented as

$\mathcal{L}\text{=
}\frac{1}{{{N}_{b}}}{{\sum\nolimits_{j}{\left({{r}_{j}}+\gamma{{Q}^{{}^{\prime}}}_{{{v}^{{}^{\prime}}}}\left({{\bm{e}}_{j}},{{\bm{w}}^{{}^{\prime}}}_{t}^{o}\right)-{{Q}_{v}}\left({{\bm{e}}_{j}},{{\bm{w}}_{j}}\right)\right)}}^{2}}.$
(21)

Then, we can obtain the optimal allocation scheme based on the wireless
communication environment. The detail of the AI-generated scheme is shown in
Algorithm 2.

Algorithm 2 Diffusion Model-based AI-Generated Scheme.

Training:

1:Initial:

2: Initialize replay buffer R and the weights of models, i.e., $\theta$,
${\theta^{{}^{\prime}}}$, $v$, ${v^{{}^{\prime}}}$

3:for Episode = 1 to Max_episode do

4: Initialize a random process N

5: for Step = 1 to Max_step do

6: Observe the existing environment ${\bm{e}_{t}}$

7: According to (18), set $\bm{w}_{t}^{N}$ as Gaussian noise and generate
allocate scheme $\bm{w}_{t}^{0}$ by denoising $\bm{w}_{t}^{N}$ using
${\varepsilon_{\theta}}$

8: Combine $\bm{w}_{t}^{0}$ with the exploration noise $\epsilon$

9: Execute scheme $\bm{w}_{t}^{0}$ and observe reward score according to (1)

10: Save the record (${{\bm{e}}_{t}}$, $\bm{w}_{t}^{0}$, ${{\tau}_{t}}$) in R

11: Randomly sample ${{N}_{b}}$ records (${{\bm{e}}_{j}}$, ${\bm{w}_{j}}$,
${{\tau}_{j}}$) from R as a minibatch

12: Minimize the loss to update ${{Q}_{v}}$ according to (21)

13: Update ${{\varepsilon}_{\theta}}$ by taking gradient descent step on

14:
${{\nabla}_{\theta}}{{\varepsilon}_{\theta}}\approx\frac{1}{{{N}_{b}}}\sum\nolimits_{j}{{{\nabla}_{{\bm{w}^{0}}}}}{{Q}_{v}}\left(\bm{e,}{{\bm{w}}^{0}}\right){{|}_{\bm{e=}{{\bm{e}}_{j}}}}{{\nabla}_{\theta}}{{\varepsilon}_{\theta}}|{{\bm{e}}_{j}}$

15: ${{\theta}^{\prime}}\leftarrow\tau\theta+(1-\tau){{\theta}^{\prime}}$

16: ${{v}^{\prime}}\leftarrow\tau v+(1-\tau){{v}^{\prime}}$

17: end for

18:end for

19:return ${{\varepsilon}_{\theta}}$

Inference:

1:Input $\bm{e}$

2:According to (18), denoise Gaussian noise using ${{\varepsilon}_{\theta}}$
to generate the optimal allocation scheme $\bm{w}^{0}$

3:return The optimal resource allocate scheme $\bm{w}^{0}$

##  V Numerical Results

This section primarily presents the experimental settings, materials, and
results in this study. Initially, we evaluate the effectiveness of the
improvements made to the object detector YOLOv7-X and compare its performance
with other state-of-the-art models. Subsequently, we assess the performance of
the proposed YOLO-based semantic communication system, verify the cost savings
of semantic communication, and examine the impact of two proposed power
allocation schemes on the transmission quality for critical information.

###  V-A Environment Setup

The experimental platform is built on a generic Ubuntu 20.04 system with 2
Intel(R) Xeon(R) Silver 4110 CPUs and GeForce RTX 3090 GPU. The parameters of
the object detection model training process used are shown in TABLE I. The
MinneApple dataset [49] is the apple image dataset used in this experiment. It
is a publicly available dataset utilized for apple detection and segmentation,
containing images of multiple apple varieties at different stages of growth,
with a large number of densely packed small apples. The MinneApple dataset
contains a total of 670 labeled images and 331 unlabeled images. Fig. 7 shows
example images from the MinneApple dataset.

TABLE I: Object Detection Models Training Parameters Parameter | Value | Parameter | Value
---|---|---|---
Learning Rate | 0.01 | Epochs | 300
Batch Size | 16 | Momentum | 0.937
Image Size | 640$\times$640 | Weight Decay | 0.0005
Figure 7: A partial overview of MinneApple dataset.

We use the Fisher-Snedecor ${\cal F}$ channel model [27] in wireless semantic
communication to analyze the performance of our model. The small-scale fading
between the UAV and users is represented by the Fisher-Snedecor ${\cal F}$
fading distribution, while small-scale variations follow the Nakagami$-m$
distribution and shadowing follows the inverse Nakagami$-m$ distribution [27].
We set the fading parameter ${m_{f}}$ = 6, the shadowing parameter ${m_{s}}$ =
6 and the transmit power $P$ = 3000 W by default. In addition, the parameters
of the resource allocation scheme generated by the AI-generated algorithm
during the training process are shown in TABLE II.

TABLE II: Diffusion Model-Based AI-Generated Algorithm
Training Parameters Parameter | Value
---|---
Diffusion Step $N$ | 50
Batch Size $N_{b}$ | 512
Discount Factor $\gamma$ | 0.95
Soft Target Update Parameter $\tau$ | 0.005
Exploration Noise $\epsilon$ | 0.05
The Learning Rate of Network ${\varepsilon_{\theta}}$ | ${10^{-5}}$
The Learning Rate of Network $Q_{v}$ | ${10^{-4}}$

###  V-B Results and Analysis

Results of ablation experiment. To evaluate the effectiveness of the ELAN-H
and SimAM attention modules, we utilize the amount of parameters and
computational complexity, i.e., Floating Point Operations (FLOPs), as well as
<EMAIL_ADDRESS>and AP@0.5:0.95 as indicators to measure the performance of the models.
Where<EMAIL_ADDRESS>and AP@0.5:0.95 are commonly-used evaluation standards in object
detection, with higher values indicating better model performance. From the
results in Table III, the utilization of the ELAN-H module leads to 1.3% and
1.7% increases in<EMAIL_ADDRESS>and AP@0.5:0.95, respectively, while reducing the
amount of parameters by 24% and FLOPs by 19%. The incorporation of the SimAM
attention module enhances the value of<EMAIL_ADDRESS>by 0.8%, with no changes to the
amount of parameters and FLOPs.

TABLE III: Results of Ablation Experiment. Model | Parameters | FLOPs |<EMAIL_ADDRESS>| AP@0.5:0.95
---|---|---|---|---
YOLOv7-X | 70.7M | 188.0G | 87.8% | 43.7%
YOLOv7-X+ELAN-H | 53.5M | 152.6G | 89.1% | 45.4%
YOLOv7-X+ELAN-H+SimAM | 53.5M | 152.6G | 89.8% | 45.4%

Comparison with other state-of-the-art object detectors. We conduct a
performance comparison of our enhanced YOLOv7-X model with other advanced
object detection models. In addition to the comparison items presented in
Table III, we also compare the detection speed.

TABLE IV: Comparison of State-of-the-art Object Detectors. Model | Parameters | FLOPs |<EMAIL_ADDRESS>| AP@0.5:0.95 | FPS
---|---|---|---|---|---
Faster R-CNN [50] | 41.7M | 59.4G | 74.1% | 33.1% | 34
RetinaNet [51] | 56.9M | 74.4G | 62.4% | 24.8% | 32
FCOS [52] | 51.2M | 66.3G | 66.6% | 27.8% | 25
Scaled-YOLOv4-p5 [53] | 70.2M | 165.1G | 88.3% | 45.9% | 22
YOLOX-X [39] | 104.5M | 312.0G | 88.4% | 47.0% | 27
YOLOv5-X | 86.2M | 203.8G | 87.5% | 44.7% | 32
YOLOR-CSP-X [54]] | 96.4M | 225.5G | 81.3% | 42.0% | 22
PPYOLOE-X [55] | 95.3M | 204.9G | 88.6% | 45.4% | 19
Ours | 53.5M | 152.6G | 89.8% | 45.4% | 34

As shown in Table IV, our proposed model achieves the best performance in
terms of<EMAIL_ADDRESS>compared to other models. Although the performance difference
between our proposed model and other advanced YOLO series models is not
significant, our model greatly reduces the amount of parameters and FLOPs
while achieving the fastest detection speed. However, Faster R-CNN, RetinaNet,
and FCOS models, although having smaller parameter size, perform poorly on the
MinneApple dataset, which contains many small objects, and their detection
performance fails to meet the requirements of our proposed scenarios.

The effects of reducing communication overhead. We use 331 unannotated test
images from the MinneApple dataset as the images to be sent by the UAV. Fig. 8
illustrates a comparison between the data size of images transmitted through
conventional communication methods and the data size resulting from semantic
communication after the implementation of semantic feature extraction. The
aggregate size of the original images amounts to 595.2MB. However, following
semantic feature extraction, the volume of data required for transmission by
edge devices is considerably reduced to 55.4MB, encompassing 54.8MB of image
format data and 0.6MB of text format data. This reduction corresponds to a 91%
decrease in communication costs, thereby substantially minimizing power
consumption during transmission.

Figure 8: Comparison of transferred bytes for each image in the two
communication methods.

The effect of $\eta$ on Conf-SemCom. Given that the textual data required for
transmission by the UAV is considerably smaller than image data, we focus
solely on the impact of wireless transmission environments on the semantic
information transfer of image formats. Taking Fig. 5 as an example, the UAV
first detects that the image contains 30 objects and then allocates power for
transmission according to the confidence of each object. We first show the
effectiveness of the proposed Conf-SemCom method. Additionally, to investigate
the impact of variable $\eta$ on the transmission performance, we increase
$\eta$ from 0.25 to 1.5 and the transmission distance from 10m to 30m. Each
experiment is repeated 100 times, and the average results are shown. The
curves of transmission quality values (i.e., MIST scores) is shown in Fig. 9.

(a) Transmission Distance $D=10m.$

(b) Transmission Distance $D=20m.$

(c) Transmission Distance $D=30m.$

Figure 9: The curves of transmission quality scores with different weight
adjustment variables $\eta$ and transmission distance $D$.

It can be observed that due to allocating more resources to important
information, Conf-SmeCom outperforms the Avg-SemCom method in most cases, with
its effectiveness becoming more evident as the transmission distance
increases. Furthermore, the optimal value of variable $\eta$ varies across
different transmission distances. At transmission distances of 10m, 20m, and
30m, the highest MIST scores are achieved when $\eta$ equals 0.5, 0.75, and 1,
respectively. This indicates that appropriately increasing the value of
variable $\eta$ as the transmission distance increases, while keeping the
total power $P$ unchanged, allows for a better enhancement of overall
communication quality by increasing the transmission power $p_{i}$ for more
significant objects $i$.

The effects of power allocation. We investigate the impact of two power
allocation methods, i.e., Avg-SemCom and Conf-SemCom, on the transmission
quality of images with different levels of confidence, over different
transmission distances (i.e., 10m, 20m, and 30m). As shown in Fig. 10, the
horizontal axis represents the 30 detected images, sorted in ascending order
of confidence levels, and the vertical axis represents the Bit Error Rate
(BER) values derived from the channel model according to the allocated power.
It is evident that, as the distance increases, the image transmission quality
declines for both communication methods. However, Conf-SemCom opts to allocate
more power to semantically important information, resulting in reduced error
rates for crucial semantic information even under poor channel conditions.

Figure 10: Bit error rate of images at different transmission distances under
two transmission methods.

To facilitate a comprehensive comparison between the two power allocation
methods, Fig. 11 illustrates the transmission performance of select images at
various communication distances. We use SSIM to evaluate the transmission
quality of each object. From the Fig. 11, the transmitted image quality for
both methods degrades significantly as the transmission distance increases.
Moreover, due to the uniformed distribution of transmission power, the SSIM
values transmitted by Avg-SemCom for each image exhibit a relatively uniform
and irregular pattern, resulting in certain critical images possessing
inferior transmission quality compared to original images. For instance,
apples $a$ and $c$ are heavily occluded and contain very little usable
information. The significance of semantic features for these images is less
than that for other images. However, their transmission quality surpasses that
of other images, which is unreasonable. In contrast, Conf-SemCom allocates
increased power to salient images, enabling the high-quality transmission of
these images even in poor channel conditions.

Figure 11: The transmission effects and corresponding SSIM values of partial
images at different transmission distances. The odd-numbered rows are Conf-
SemCom and the even-numbered rows are Avg-SemCom.

The effects of AI-Generated scheme. We compare the diffusion model-based AI-
generated scheme with two other transmission power allocation methods for
transmitting semantic information at the transmission distance $D$ = 20 m and
the transmission power $P$ = 4 kW. As illustrated in Fig. 12, the diffusion
model-based AI-generated algorithm exhibits rapid training speed during the
optimization of power allocation schemes, surpassing the confidence-based
allocation scheme at approximately 500 iterations. The superiority of the AI-
generated approach primarily stems from the exploration conducted through
diffusion method, which enhances the flexibility of strategies and prevents
the model from getting trapped in suboptimal solutions. Furthermore, it is
evident that the Avg-SemCom method significantly underperforms in terms of the
MIST score compared to the other two schemes, indicating the necessity of
considering the importance of semantic information during the communication
process.

Figure 12: Comparison of the training process of the diffusion model-based AI-
generated algorithm and the results of two other methods (i.e., Avg-SemCom and
Conf-SemCom), with transmission distance $D$ = 20 m and transmission power $P$
= 4 kW.

##  VI Conclusion

In this paper, we have proposed a YOLO-based semantic communication framework
for developing a virtual apple orchard case, focusing on optimizing semantic
information transmission and resource allocation for images collected by edge
devices. Initially, we have enhanced the performance of the object detector
YOLOv7-X on a real apple dataset and have employed the optimized object
detector to extract semantic information from images captured by edge devices,
aiming to reduce transmission costs. Furthermore, to ensure the high-quality
transmission of essential semantic information, we have allocated resource
based on the significance of their semantic content. Specifically, we have
allocated the transmission power of semantic information based on the
confidence generated by the object detection algorithm and the scheme
generated by the diffusion model-based AI-generated algorithm, respectively.
Numerical results have demonstrated that the proposed framework and strategy
have considerably reduced communication costs and have markedly improved the
transmission quality of important information during communication.

## References

  * [1] N. H. Chu, D. T. Hoang, D. N. Nguyen, K. T. Phan, and E. Dutkiewicz, “MetaSlicing: A Novel Resource Allocation Framework for Metaverse,” _ArXiv_ , vol. abs/2205.11087, 2022.
  * [2] D. Wu, Z. Yang, P. Zhang, R. Wang, B. Yang, and X. Ma, “Virtual-Reality Inter-Promotion Technology for Metaverse: A Survey,” _IEEE Internet Things J._ , pp. 1–1, 2023.
  * [3] Y. Wu, K. Zhang, and Y. Zhang, “Digital Twin Networks: A Survey,” _IEEE Internet Things J._ , vol. 8, pp. 13 789–13 804, 2021.
  * [4] C. Feng, X. Bin, L. Na, L. Haishen, and S. Chuanheng, “the Metaverse for Agriculture,” _Smart Agriculture_ , vol. 4, pp. 126–137, 2022.
  * [5] S. Neethirajan and B. Kemp, “Digital Twins in Livestock Farming,” _Animals (Basel)_ , vol. 11, 2021.
  * [6] P. Tan, “Tech Forum: Accelerating Digitalization with New Technology.” [Online]. Available: https://www.alibabacloud.com/zh/apsara-conference-2021
  * [7] C. Khansulivong, S. Wicha, and P. Temdee, “Adaptive of New Technology for Agriculture Online Learning by Metaverse: A Case Study in Faculty of Agriculture, National University of Laos,” _2022 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON)_, pp. 428–432, 2022.
  * [8] Y. Han, D. T. Niyato, C. Leung, D. I. Kim, K. Zhu, S. Feng, X. S. Shen, and C. Miao, “A Dynamic Hierarchical Framework for IoT-Assisted Digital Twin Synchronization in the Metaverse,” _IEEE Internet Things J._ , vol. 10, pp. 268–284, 2023.
  * [9] L. U. Khan, W. Saad, D. T. Niyato, Z. Han, and C. S. Hong, “Digital-Twin-Enabled 6G: Vision, Architectural Trends, and Future Directions,” _IEEE Commun Mag_ , vol. 60, pp. 74–80, 2021.
  * [10] M. S. Farooq, S. Riaz, A. Abid, K. Abid, and M. A. Naeem, “A Survey on the Role of IoT in Agriculture for the Implementation of Smart Farming,” _IEEE Access_ , vol. 7, pp. 156 237–156 271, 2019.
  * [11] D. S. Paraforos and H. W. Griepentrog, “Digital Farming and Field Robotics: Internet of Things, Cloud Computing, and Big Data,” _Agriculture Automation and Control_ , 2021.
  * [12] E. C. Strinati and S. Barbarossa, “6G Networks: Beyond Shannon Towards Semantic and Goal-Oriented Communications,” _Comput. Networks_ , vol. 190, p. 107930, 2020.
  * [13] P. Zhang, W. Xu, H. Gao, K. Niu, X. Xu, X. Qin, C. Yuan, Z. Qin, H. Zhao, J. Wei, and F. Zhang, “Toward Wisdom-Evolutionary and Primitive-Concise 6G:A New Paradigm of Semantic Communication Networks,” _Engineering_ , 2021\.
  * [14] W. C. Ng, H. Du, W. Y. B. Lim, Z. Xiong, D. T. Niyato, and C. Miao, “Stochastic Resource Allocation for Semantic Communication-aided Virtual Transportation Networks in the Metaverse,” _ArXiv_ , vol. abs/2208.14661, 2022.
  * [15] C. Dong, H. Liang, X. Xu, S. Han, B. Wang, and P. Zhang, “Semantic Communication System Based on Semantic Slice Models Propagation,” _IEEE J Sel Areas Commun_ , vol. 41, no. 1, pp. 202–213, 2023.
  * [16] Y. Han, D. T. Niyato, C. Leung, D. I. Kim, K. Zhu, S. Feng, X. S. Shen, and C. Miao, “A Dynamic Hierarchical Framework for IoT-assisted Metaverse Synchronization,” _ArXiv_ , vol. abs/2203.03969, 2022.
  * [17] K. Li, Y. Cui, W. Li, T. Lv, X. Yuan, S. Li, W. Ni, M. Simsek, and F. Dressler, “When Internet of Things Meets Metaverse: Convergence of Physical and Cyber Worlds,” _IEEE Internet Things J._ , vol. 10, no. 5, pp. 4148–4173, 2023\.
  * [18] W. Li, D. Zhu, and Q. Wang, “A single view leaf reconstruction method based on the fusion of ResNet and differentiable render in plant growth digital twin system,” _Comput. Electron. Agric._ , vol. 193, p. 106712, 2022.
  * [19] P. Angin, M. H. Anisi, F. Göksel, C. Gürsoy, and A. Büyükgülcü, “AgriLoRa: A Digital Twin Framework for Smart Agriculture,” _J. Wirel. Mob. Networks Ubiquitous Comput. Dependable Appl._ , vol. 11, pp. 77–96, 2020.
  * [20] M. Awais, W. Li, H. Li, M. J. M. Cheema, S. Hussain, and C. Liu, “Optimization of Intelligent Irrigation Systems for Smart Farming Using Multi-Spectral Unmanned Aerial Vehicle and Digital Twins Modeling,” _The 1st International Precision Agriculture Pakistan Conference 2022 (PAPC 2022) —Change the Culture of Agriculture_, 2022.
  * [21] Q. Lan, D. Wen, Z. Zhang, Q. Zeng, X. Chen, P. Popovski, and K. Huang, “What is Semantic Communication? A View on Conveying Meaning in the Era of Machine Intelligence,” _ArXiv_ , vol. abs/2110.00196, 2021.
  * [22] H. Xie, Z. Qin, G. Y. Li, and B.-H. Juang, “Deep Learning Enabled Semantic Communication Systems,” _IEEE Trans. Signal Process._ , vol. 69, pp. 2663–2675, 2020.
  * [23] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is All you Need,” _ArXiv_ , vol. abs/1706.03762, 2017.
  * [24] F. Zhou, Y. Li, X. Zhang, Q. Wu, X. Lei, and R. Q. Hu, “Cognitive Semantic Communication Systems Driven by Knowledge Graph,” _ICC 2022 - IEEE International Conference on Communications_ , pp. 4860–4865, 2022.
  * [25] M. Lokumarambage, V. S. S. Gowrisetty, H. Rezaei, T. Sivalingam, R. M. A. P. Rajatheva, and W. Fernando, “Wireless End-to-End Image Transmission System using Semantic Communications,” _ArXiv_ , vol. abs/2302.13721, 2023.
  * [26] H. Zhang, S. Shao, M. Tao, X. L. Bi, and K. B. Letaief, “Deep learning-enabled semantic communication systems with task-unaware transmitter and dynamic data,” _IEEE J. Sel. Areas Commun._ , vol. 41, pp. 170–185, 2022.
  * [27] J. Kang, H. Du, Z. Li, Z. Xiong, S. Ma, D. T. Niyato, and Y.-F. Li, “Personalized Saliency in Task-Oriented Semantic Communications: Image Transmission and Performance Analysis,” _IEEE J. Sel. Areas Commun._ , vol. 41, pp. 186–201, 2022.
  * [28] J. Lv, “Recognition of Overlapping and Occluded Fruits in Natural Environment,” _J. Comput. Theor. Nanosci._ , vol. 13, pp. 2475–2484, 2016\.
  * [29] X. Liu, D. Zhao, W. Jia, W. Ji, and Y. Sun, “A Detection Method for Apple Fruits Based on Color and Shape Features,” _IEEE Access_ , vol. 7, pp. 67 923–67 933, 2019.
  * [30] W. Chen, J. Zhang, B. Guo, Q. T. Wei, and Z. Zhu, “An Apple Detection Method Based on Des-YOLO v4 Algorithm for Harvesting Robots in Complex Environment,” _Math. Probl. Eng._ , 2021.
  * [31] B. Yan, P. Fan, X. Lei, Z. Liu, and F. Yang, “A Real-Time Apple Targets Detection Method for Picking Robot Based on Improved YOLOv5,” _Remote. Sens._ , vol. 13, p. 1619, 2021.
  * [32] Ultralytics, “YOLOv5.” [Online]. Available: https://github.com/ultralytics/yolov5
  * [33] M. Sun, L. Xu, X. Chen, Z. Ji, Y. Zheng, and W. Jia, “BFP Net: Balanced Feature Pyramid Network for Small Apple Detection in Complex Orchard Environment,” _Plant Phenomics_ , vol. 2022, 2022.
  * [34] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors,” _ArXiv_ , vol. abs/2207.02696, 2022.
  * [35] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “YOLOv4: Optimal Speed and Accuracy of Object Detection,” _ArXiv_ , vol. abs/2004.10934, 2020.
  * [36] H. Zhang, M. Cissé, Y. Dauphin, and D. Lopez-Paz, “mixup: Beyond Empirical Risk Minimization,” _ArXiv_ , vol. abs/1710.09412, 2017.
  * [37] F. K. Shaikh, S. Karim, S. Zeadally, and J. Nebhen, “Recent Trends in Internet-of-Things-Enabled Sensor Technologies for Smart Agriculture,” _IEEE Internet Things J._ , vol. 9, no. 23, pp. 23 583–23 598, 2022.
  * [38] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Trans Image Process._ , vol. 13, pp. 600–612, 2004.
  * [39] Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, “YOLOX: Exceeding YOLO Series in 2021,” _ArXiv_ , vol. abs/2107.08430, 2021.
  * [40] Y. Rao, W. Zhao, Y. Tang, J. Zhou, S. N. Lim, and J. Lu, “HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions,” _ArXiv_ , vol. abs/2207.14284, 2022.
  * [41] L. Yang, R.-Y. Zhang, L. Li, and X. Xie, “SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks,” in _Proc. Int. Conf. Mach. Learn._ , vol. 139, 18–24 Jul 2021, pp. 11 863–11 874.
  * [42] J. Park, S. Woo, J.-Y. Lee, and I.-S. Kweon, “Bam: Bottleneck attention module,” in _Proc. Brit. Mach. Vis. Conf._ , 2018.
  * [43] S. Woo, J. Park, J.-Y. Lee, and I.-S. Kweon, “CBAM: Convolutional Block Attention Module,” in _Proc. Eur. Conf. Comput. Vis._ , 2018.
  * [44] B. S. Webb, N. T. Dhruv, S. G. Solomon, C. Tailby, and P. Lennie, “Early and Late Mechanisms of Surround Suppression in Striate Cortex of Macaque,” _J Neurosci._ , vol. 25, pp. 11 666 – 11 675, 2005.
  * [45] L. Yang, Z. Zhang, S. Hong, R. Xu, Y. Zhao, Y. Shao, W. Zhang, M.-H. Yang, and B. Cui, “Diffusion Models: A Comprehensive Survey of Methods and Applications,” _ArXiv_ , vol. abs/2209.00796, 2022.
  * [46] H. Du, J. Wang, D. Niyato, J. Kang, Z. Xiong, and D. In Kim, “AI-Generated Incentive Mechanism and Full-Duplex Semantic Communications for Information Sharing,” _arXiv e-prints_ , pp. arXiv–2303, 2023.
  * [47] J. Ho, A. Jain, and P. Abbeel, “Denoising Diffusion Probabilistic Models,” _ArXiv_ , vol. abs/2006.11239, 2020.
  * [48] H. V. Hasselt, “Double Q-learning,” in _NIPS_ , 2010.
  * [49] N. Häni, P. Roy, and V. Isler, “MinneApple: A Benchmark Dataset for Apple Detection and Segmentation,” _IEEE Robot. Autom. Lett._ , vol. 5, pp. 852–858, 2019.
  * [50] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 39, pp. 1137–1149, 2015.
  * [51] T.-Y. Lin, P. Goyal, R. B. Girshick, K. He, and P. Dollár, “Focal Loss for Dense Object Detection,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 42, pp. 318–327, 2017.
  * [52] Z. Tian, C. Shen, H. Chen, and T. He, “FCOS: Fully Convolutional One-Stage Object Detection,” _Proc. IEEE Int. Conf. Comput. Vis._ , pp. 9626–9635, 2019.
  * [53] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, “Scaled-YOLOv4: Scaling Cross Stage Partial Network,” _Proc. IEEE Conf. Comput. Vis. Pattern Recog._ , pp. 13 024–13 033, 2020.
  * [54] C.-Y. Wang, I.-H. Yeh, and H. Liao, “You Only Learn One Representation: Unified Network for Multiple Tasks,” _J. Inf. Sci. Eng._ , vol. 39, pp. 691–709, 2021.
  * [55] S. Xu, X. Wang, W. Lv, Q. Chang, C. Cui, K. Deng, G. Wang, Q. Dang, S. Wei, Y. Du, and B. Lai, “PP-YOLOE: An evolved version of YOLO,” _ArXiv_ , vol. abs/2203.16250, 2022.