# DT/MARS-CycleGAN: Improved Object Detection for MARS Phenotyping Robot

David Liu*1, Zhengkun Li*2, Zihao Wu*3, Changying Li2,3 *Equal
Contribution.1Athens Academy, Athens, GA 30606, USA2University of Florida,
Gainesville, FL 32611, USA3University of Georgia, Athens, GA 30602, USA
_Corresponding author: Changying Li (email: cli2@ufl.edu)._

###### Abstract

Robotic crop phenotyping has emerged as a key technology to assess crops’
morphological and physiological traits at scale. These phenotypical
measurements are essential for developing new crop varieties with the aim of
increasing productivity and dealing with environmental challenges such as
climate change. However, developing and deploying crop phenotyping robots face
many challenges such as complex and variable crop shapes that complicate
robotic object detection, dynamic and unstructured environments that baffle
robotic control, and real-time computing and managing big data that challenge
robotic hardware/software. This work specifically tackles the first challenge
by proposing a novel Digital-Twin(DT)MARS-CycleGAN model for image
augmentation to improve our Modular Agricultural Robotic System (MARS)’s crop
object detection from complex and variable backgrounds. Our core idea is that
in addition to the cycle consistency losses in the CycleGAN model, we designed
and enforced a new DT-MARS loss in the deep learning model to penalize the
inconsistency between real crop images captured by MARS and synthesized images
sensed by DT MARS. Therefore, the generated synthesized crop images closely
mimic real images in terms of realism, and they are employed to fine-tune
object detectors such as YOLOv8. Extensive experiments demonstrated that our
new DT/MARS-CycleGAN framework significantly boosts our MARS’ crop object/row
detector’s performance, contributing to the field of robotic crop phenotyping.

##  I Introduction

To sustain the growing world population, crop breeders strive to develop high-
yielding and stress-tolerant cultivars that are more resilient to the changing
climate, pests, and diseases [1, 2, 3, 4]. Although advances in plant
biotechnology, such as genome sequencing and genotyping technologies, have
significantly accelerated the pace of crop breeding process [5, 6],
phenotyping has been a bottleneck for crop improvement [1, 7]. To address this
challenge, automated robotic crop phenotyping technologies have been developed
to assess and quantify phenotypic traits that are related to crop growth,
yield, and adaption to environmental stresses [7, 4, 8]. Important crop
phenotypic traits include morphological, structural, physiological, and
biochemical properties at multiple organizational scales. It is crucial to
phenotype a diverse range of crop genotypes across multiple environments to
enable crop breeders to identify and select genotypes with desirable
phenotypes best suited for specific environments. As a result, in-field high-
throughput robotic phenotyping is indispensable for the effective selection of
high-yielding and stress-tolerant varieties [1, 3, 4].

However, developing field crop phenotyping robots faces multiple challenges.
First, to address specific phenotyping needs, in-field crop phenotyping robots
need highly specific platforms such as tower-based, gantry-based, ground
mobile, low- and high-altitude aerial, and satellite-based systems. These crop
phenotyping platforms also need highly specific computing devices and sensors
such as edge devices, RGB color cameras, depth cameras, thermal imaging, and
spectral imaging, and specific software tools such as ROS (robotic operating
system) and deep learning acceleration packages [3, 9, 10], to reduce the
costs and allow for nondestructive characterization of the traits of interest.
Second, field-based crop phenotyping robots typically need to build maps of
the surrounding environment, plan their optimal paths, follow correct
trajectories, navigate between crop rows, avoid obstacles, and collect
phenotypical data in a breeding field. Although GPS is commonly used for robot
navigation, the real-time-kinematic GPS is costly and high-quality signal
reception is not always guaranteed. Therefore, the information provided by
cameras along with GPS is typically necessary to precisely guide the
phenotyping robot and avoid damage to crops and potential humans. Third, once
large amounts of crop trait sensing datasets are acquired, data science and AI
models are then needed to make necessary phenotypic inference and prediction
[11]. Specific image-based phenotyping may include plant stress, plant
development such as plant shoot morphology and growth, plant and plant organ
counting, root system architecture, and crop postharvest quality assessment,
among others.

To deal with the above-mentioned challenges, a variety of field-based high-
throughput crop phenotyping robots have been reported in the literature [1, 3,
4, 7, 11, 12, 13] to handle various types of crops, including fruits such as
grapes, apples, citrus, and tomatoes, and vegetables such as sugarcane, corn,
soybean, cucumber, maize, and wheat. Robotic crop phenotyping has been
particularly enhanced by deep learning models and vision sensor technologies,
as recently reviewed in [14]. However, developing and deploying robotic crop
phenotyping systems using computer vision and deep learning still face a few
key challenges, particularly in dynamic, complex, and unstructured field
environments [1]. One extensively studied while still challenging problem in
the field of robotic phenotyping domain is accurate crop detection. Accurately
identifying the crop target is a prerequisite for the robot to perform
multiple key downstream tasks such as crop row detection, path planning,
navigation, and extraction of phenotypic traits [1, 3, 4].

Many deep learning based crop object detection methods have been proposed in
the literature for various purposes such as crop seeding, yield estimation,
phenotyping, and robotic harvesting, as surveyed in [3, 15, 16]. In terms of
deep learning models, convolutional neural networks (CNNs) are the most widely
employed methods in crop object detection, as extensively reviewed in [3].
Briefly, object detection CNN models were specifically designed to identify
interested targets from images, such as the RCNN [17] and its variants based
on two stages of region proposals and classifications, and the YOLO series
models based on regressions of bounding boxes [18]. Further, these generic
RCNN/YOLO models were fine-tuned on crop images with object annotations to be
transferred to the agriculture domain. For instance, the RCNN model was
transferred for fruit detection by fine-turning a Faster RCNN model with 100
annotated images [19], and a comprehensive survey of this category of
approaches is available in [3]. Similarly, the YOLO model has been adopted and
customized for crop detection, such as apple and pear detection in [20]. With
the popularity of few-shot learning, many studies have explored the fine-
tuning of CNN models by using several examples for crop object detection, as
surveyed in [21]. However, a generic concern in few-shot object detection is
that the fine-tuned model might be subject to overfitting and its
generalization capability could be a challenge, as pointed out in [21].

In parallel, image data augmentation strategy has been extensively explored to
boost the training of deep learning models in agriculture [22]. The generative
adversarial network (GAN) model [23] has been particularly popular in
generating synthesized image data for model training, e.g., those studies in
[24, 25]. The GAN model has also been widely used in the general robotics
field, e.g., for synthesizing large-scale realistic images for modeling
training to deal with the reality gap problem in sim-to-real transfer [26, 27,
28, 29]. For instances, the authors in [30] developed a novel GAN model to
effectively adapt synthesized images to realistic ones by enforcing an object-
detection consistency for visual grasping, and the authors in [31] developed a
RL-CycleGAN model by applying the similarity of total expected future reward
(Q-value) of Q-learning as an additional reinforcement learning (RL)-scene
consistency loss, thus enabling effective sim-to-real transfer for
reinforcement learning.

Along this promising research direction, and inspired by the recent research
in [32], this paper specifically proposes a novel Digital-Twin(DT)/MARS-
CycleGAN model for image augmentation to improve crop object detection from
complex and variable farm background in the context of our MARS robotic
phenotyping [13, 33]. Our methodological innovation is that real and virtual
DT MARS robots are forced to mimic each other such that the gaps between
simulated and realistic robotic crop phenotyping, e.g., crop/row detection,
are minimized, thus the reality gap during zero-shot sim-to-real transfer of
trained crop detection models is minimized at a fundamental level. Our novel
algorithmic contribution here is that in addition to the classic cycle
consistency losses in the CycleGAN model [34], we designed and integrated a
novel digital-twin (DT)-MARS loss in the deep learning network to penalize the
inconsistency between synthesized images generated by DT MARS and real crop
images sensed by MARS. As a result, the generated synthetic crop images are
significantly more realistic to real images, and thus they can be used to
significantly better fine-tune the object detectors such as YOLOv8 [35]. The
improved crop object detector was then employed for crop row detection in
MARS’ real-world robotic phenotyping environment, which is a key step for
robotic path planning and navigation. Extensive experimental results have
demonstrated that our DT/MARS-CycleGAN framework significantly enhanced our
MARS’ crop object/row detector’s performance, thus contributing to the fields
of crop phenotyping robotics and precision agriculture.

##  II Related Works

###  II-A Crop Phenotyping Robot

There have been many innovative crop phenotyping robots reported in the
literature, as surveyed in [3, 4]. As discussed in [4], these phenotyping
robots can be categorized in different ways, e.g., autonomous vs non-
autonomous systems such as tractors, manually pushed or motorized carts,
gantries, and cable-driven systems, and wheeled vs tracked vs wheel-legged
systems. Here, we briefly introduce a few crop phenotyping robot examples that
are most relevant to our work in this paper. The Thorvald II system [36] was
developed based on the modular hardware and software concept, that is, the
robot can be easily reconfigured to obtain the necessary physical properties
to operate in multiple production contexts (e.g., tunnels, greenhouses, and
open fields) and the mechanical properties can be adjusted for different track
widths, power requirements, and load capacities. The modular agricultural
robotic system (MARS) [13] is an autonomous and multi-purpose phenotyping
robot with five essential hardware modules (wheel module, connection module,
robot controller, robot frame, and power module) and three optional hardware
modules (actuation module, sensing module, and smart attachment). An important
feature of MARS [13] is that different combinations of the hardware modules
can make the needed configurations for specific agricultural tasks, such as
crop phenotyping. The Field Scanalyzer [37] robot platform, developed by
LemnaTec, is a high-throughput phenotyping system that includes a sensor box
for phenotyping sensors and environmental sensors such as cameras and LiDAR,
to capture detailed phenotypic data on crops in large field plots. The Field
Scanalyzer has environmental sensors on board and thus it can record climatic
data during all phenotypic measurements to enable the meaningful link between
phenotype and environment data. A comprehensive review of crop phenotyping
robots is referred to [4].

###  II-B Digital Twin for Agriculture Robotics

Digital twin (DT) has made transformative impacts on many research and
industry domains [38], including agriculture [39] and robotics [40, 32]. The
idea of real-time virtual representation via DT technologies significantly
enable and facilitate the digitization of agriculture, in which data,
modeling, and what-if simulations are integrated to provide a promising
framework to address constraints in decision-making support and automation for
various agricultural applications, including crop phenotyping, as reviewed in
[39]. In parallel, DT technologies have been recently explored in robotics
[38, 40]. For instances, the authors in [40] reported a DT-enabled approach
for achieving the effective transfer of deep reinforcement learning (DRL)
algorithms to a physical robot. In [32], the authors acquired large-scale
visual grasping datasets with ground truth annotations in DT environment that
mimic the real world to train their DT-CycleGAN model and reported promising
results in zero-shot sim-to-real transfer of grasping models. In this paper,
our DT/MARS-CycleGAN model takes the advantages of using DT technologies in
both agriculture and robotics by maximizing the similarity between the
physical and virtual DT MARS robots so that their behaviors tend to be
identical, and thus the reality gap in zero-shot sim-to-real transfer is
minimized.

###  II-C Object Detection in Precision Agriculture

Since the early work of DeepFruit [19] for crop object detection using the
faster RCNN model [17], the precision agriculture field has devoted much
effort to improve deep learning based crop detection. For instance, in early
studies, the scheme of splitting a high-resolution agriculture image into
multiple small patches was proposed to deal with the CNN’s small input size
issue [3], and features from shallow layers of RCNN model were used for
regional proposals to deal with the small fruit object detection problem [41].
Later, few-shot learning has been employed to fine-tune the RCNN and YOLO
based models by using several examples for crop object detection [21].
Recently, vision transformer (ViT) [42] was fine-tuned for the detection of
plant infections and diseases, named GreenViT [43], and it was reported that
the GreenViT achieved remarkable accuracy and effectively reduced the
occurrence of false alarms [43]. The authors in [44] proposed a mask-guided
ViT model for fruit object detection based on few-shot learning and reported
promising results for apple detection. The interesting idea in [44] is that a
mask is applied on image patches to screen out the task-irrelevant ones and to
guide the ViT to focus on task-relevant and discriminative patches (e.g.,
apples) during few-shot learning. Particularly, the mask-guided ViT only
introduces an additional mask operation and a residual connection, which
enables the inheritance of parameters from pre-trained ViT without any
additional cost. The authors in [44] also optimally selected representative
few-shot samples (e.g., apples), and included an active learning based sample
selection method to further improve the generalizability of the mask-guided
ViT based few-shot learning. In terms of crop varieties, there are many types
of deep learning models that have been developed for detecting grapes [45],
apples [46], and citrus [47], among many others, as systematically surveyed in
[3].

However, applying deep learning techniques to agriculture applications poses
unique challenges too, particularly when it comes to data collection and
variability. Agricultural settings are complex and unpredictable, with factors
like plant development, weather fluctuations, and variable lighting
conditions, making it difficult to amass the large, diverse datasets that deep
learning models require. These uncontrollable elements complicate the task of
capturing a comprehensive range of data variations, limiting the deep learning
technology’s applicability in variable agricultural contexts. While there has
been an increase in agricultural datasets recently [48][49], there is still a
pressing need for specialized, publicly available datasets tailored
specifically for agricultural applications.

###  II-D Image Augmentation in Precision Agriculture

Image data augmentation has been a mainstream approach for improving the
generalization capability of deep learning models, and this effective strategy
has been extensively explored to boost the training of deep learning models in
agriculture [22, 50, 51]. Two general approaches have been proposed for image
data augmentation, that is, generative deep learning models and computational
simulations. The generative adversarial network (GAN) model [23] and its many
variants [34] have been particularly widely used in generating synthesized
image data for model training, e.g., those studies in precision agriculture
[24, 25, 52, 53]. Early work by Giuffrida et al [54] utilized DCGAN for plant
image synthesis, while Zhu et al [55] leveraged Pix2Pix for improved leaf
counting. Madsen et al’s models [56, 57] catered to multiple species, thus
achieving a notable 64.3% classification accuracy. Zhu et al [58] implemented
cDCGAN for plant vigor rating, achieving a 23% increase in testing accuracy,
and Drees et al [59] used CGAN for Brassica growth predictions, revealing
challenges in field applications. To address dataset scarcity, Shete et al
[60] introduced TasselGAN for maize images, though its realism needs
enhancement. The Global Wheat-Head Dataset (GWHD) [61] marked an advancement
for wheat phenotyping. However, domain shift challenges persisted, prompting
solutions like CycleGAN by Hartley and French [62] which, when combined with
real data, improved detection accuracy, underscoring the importance of real
labeled images in agriculture.

Computational simulations have also been widely employed in agriculture
robotics to generate training data. In an early study, Rahnemoonfar and
Sheppard [63] utilized a simplistic approach for tomato counting, that is,
overlaying red dots on a background. A more intricate method involves the
”Cut, Paste, and Learn” strategy [64], by merging portions of distinct RGB
images [65, 66]. However, this approach is limited in representing different
3D orientations and lighting nuances. For enhanced realism, 3D plant models in
virtual environments [67] and techniques like photogrammetry [68, 69] and
L-systems[70] have been employed. Alternatively, the authors in [71]
introduced a robotic crop phenotyping simulation system that simultaneously
navigates through occluded crop rows and performs phenotyping tasks, including
measuring plant volume and canopy height using a 2D LiDAR in a nodding
configuration.

However, neither GAN-based generation of synthesized images or simulation-
generated images are realistic enough, which is typically called reality gap,
for practical precision agriculture applications. Therefore, GAN model has
been adopted for synthesizing realistic images to deal with the reality gap
problem in sim-to-real transfer in the general robotics domain [26, 27, 28,
29]. For instances, a novel RetinaGAN model [30] was developed to effectively
adapt synthesized images to realistic ones by adding an object-detection
consistency for visual grasping, and a RL-CycleGAN model [31] was developed by
enforcing the similarity of total expected future reward (Q-value) of
Q-learning as an additional reinforcement learning (RL)-scene consistency loss
for visual grasping. Recently, a novel DT-CycleGAN [32] was proposed to
further bridge the reality gap in zero-shot sim-to-real transfer of visual
grasping models. In DT-CycleGAN, in addition to the traditional cycle
consistency losses, a grasping agent’s action consistency loss was defined and
penalized to minimize the inconsistency of the grasping agent’s actions
between the virtual states generated by the DT-CycleGAN generator and the real
visual states. A key innovation of DT-CycleGAN is that it effectively
integrates two mainstream image data augmentation methods of generative
CycleGAN model and simulations in the digital twin space. This DT-CycleGAN
model inspired us to perform simulations to generate synthesized crop images
in the virtual space and employ a novel DT-MARS consistency loss to make the
synthesized crop images more similar to real-world ones sensed by our MARS
phenotyping robot.

Figure 1: Illustration of the physical/DT robots (left panel), the DT/MARS-
CycleGAN model (middle panel), and the crop object/row detection network
(right panel). Additional annotation and description are referred to the main
text.

##  III Methods

As illustrated in Figure 1, our framework is comprised of three main
components: the robots operational in both real-world and simulated
environments, the crop object/row detection network; and the DT/MARS-CycleGAN
model that integrates the robot and the object detection network.

###  III-A Physical and Digital-Twin (DT) Robots

Figure 1 (left panel) showcases the MARS phenotyping robot (PhenoBot) [33]
that is employed in this work. The MARS PhenoBot is a solar-powered modular
platform with a four-wheel steering and four-wheel driving configuration. Each
wheel module of MARS PhenoBot is equipped with an independent suspension
mechanism, which makes it adaptable to uneven field terrain. The MARS PhenoBot
robot is designed for streamlined modular phenotyping and specifically
tailored for crop phenotyping [13, 33]. The robot is outfitted with three
cameras, capturing views from the front, rear, and bottom, respectively. This
setup allows for effective monitoring of crop growth and offers crucial row-
specific data to enhance farm field navigation and to prevent crop damage. The
bottom camera is specifically pointed at the ground to observe more details of
the farm field and to detect crops and rows. More details of the MARS PhenoBot
are referred to [33]. The virtual DT robot of MARS PhenoBot was developed
using the Solidworks tool. Special attention was given to simulating the
camera perspectives (e.g., bottom camera for crop/row detection) to reduce
discrepancies in visual sensing between the real robot and the DT version,
thereby narrowing the reality gap during zero-shot sim-to-real transfer of
crop object detection models.

###  III-B Crop Object/Row Detection Network

The crop object/row detection network includes two components: initially, crop
detection is facilitated by the YOLOv8 object detection model [35], which is
then followed by the application of the RANSAC (RANdom SAmple Consensus)
algorithm [72] for delineating crop rows.

As depicted in Figure 1, crop field images coupled with corresponding crop
object annotations are initially fed into the YOLOv8 model. The input image is
represented as $\mathbf{x}\in\mathbb{R}^{B2CHW}$, where $BCHW$ denotes the
batch size (B), channel (C), height (H), and width (W) of the images. For
expedited training, images are resized to 224×224. The bounding box annotation
is formatted with one row per object represented as
$\mathbf{y}\in\mathbb{R}^{Bn4}$, where $B$ is the batch size, $n$ represents
the number of objects in an image, each described in
$[x_{center},y_{center},width,height]$ format. By optimizing the IoU loss
between the predicted bounding boxes and the ground truth, the YOLOv8 model
can accurately locate the crop objects in the images.

With the crop bounding boxes predicted by YOLOv8, the centers of all bounding
boxes are extracted together to form the points data essential for crop row
detection. Subsequently, the RANSAC algorithm is employed to discern a line
representing the crop row that best fits the center points of the bounding
boxes of crop objects. Then the detected crop row will be used as the input
for visual serving control of the MARS PhenoBot. That is, the error offset of
the detected crop row from the robot’s reference is used as the signal input
to a PID (proportion, integration, and differentiation) controller to control
the robot’s velocity and movement [33]. It is apparent that accurate detection
of crops and crop rows is a fundamentally important task for crop phenotyping
robot’s path planning and navigation.

For zero-shot sim-to-real transfer of the crop detection model trained in a
simulated environment, the object detection model along with both the images
and the bounding box predictions are holistically modeled by the following
DT/MARS-CycleGAN model.

###  III-C DT/MARS-CycleGAN: Cycle Consistency Loss

The CycleGAN introduced by Zhu et al. [34] was originally developed for
unpaired image style transfer. In our work, it is adapted and integrated into
the DT/MARS-CycleGAN model to address the discrepancies in visual appearance
between simulated and real-world images, as demonstrated by the real-to-sim
and sim-to-real transitions in the middle panel of Figure 1. Here,
$\mathbf{x_{r}}\in\mathbf{R}$ represents real visual images, and
$\mathbf{x_{s}}\in\mathbf{S}$ denotes simulated visual images, as illustrated
in Figure 1. The simulated images are acquired from the simulation
environment, created by utilizing farm field images collected from the
Internet as background and rendering 3D models of the crops as target objects
at random positions. A virtual camera mounted on the DT robot captures these
images within the simulated environment, and an analytical virtual model
extracts the ground truth bounding boxes for each crop in the image based on
predefined position information. Meanwhile, real images were obtained from
actual farm environments by capturing video streams as the real robot
traversed along the crop rows in the farm field.

Specifically, as delineated in the DT/MARS-CycleGAN panel in Figure 1’s center
panel, the generator, $G_{r}(\cdot)$, transforms images to a real style, and
$G_{s}(\cdot)$ modifies the input images to a simulation style. These are
correspondent to the generators $F(*)$ and $G(*)$ in Zhu et al’s original
CycleGAN model [34]. The discriminator, $D_{r}(\cdot)$, evaluates whether the
input images are in real style, while $D_{s}(\cdot)$ discerns if the images
exhibit a simulation style. The cycle loss function of DT/MARS-CycleGAN is
mathematically described as:

$\displaystyle L(G_{s},G_{r},D_{s},D_{r})=$ $\displaystyle
L_{GAN}(G_{s},D_{s},\mathbf{R},\mathbf{S})+L_{GAN}(G_{r},D_{r},\mathbf{S},\mathbf{R})$
$\displaystyle+$ $\displaystyle
L_{cyc}(G_{r},G_{s})+L_{identity}(G_{r},G_{s})$ (1)

Here, $L_{GAN}(G_{s},D_{s},\mathbf{R},\mathbf{S})$ and
$L_{GAN}(G_{r},D_{r},\mathbf{S},\mathbf{R})$ denote the adversarial losses;
$L_{cyc}(G_{r},G_{s})$ signifies the cycle consistency loss, and
$L_{identity}(G_{r},G_{s})$ represents the identity mapping loss.

Adversarial losses aim to align the style of the generated crop image with the
style of the target domain, such as in a real-world farm environment. They are
expressed as:

$\displaystyle L_{GAN}(G_{s},D_{s},\mathbf{R},\mathbf{S})=$
$\displaystyle\mathbb{E}{\mathbf{x_{s}}\sim
p{data}(\mathbf{S})}[logD_{s}(\mathbf{x_{s}})]$ (2) $\displaystyle+$
$\displaystyle\mathbb{E}{\mathbf{x_{r}}\sim
p{data}(\mathbf{R})}[log(1-D_{s}(G_{s}(\mathbf{x_{r}})))]$ $\displaystyle
L_{GAN}(G_{r},D_{r},\mathbf{S},\mathbf{R})=$
$\displaystyle\mathbb{E}{\mathbf{x_{r}}\sim
p{data}(\mathbf{R})}[logD_{r}(\mathbf{x_{r}})]$ (3) $\displaystyle+$
$\displaystyle\mathbb{E}{\mathbf{x_{s}}\sim
p{data}(\mathbf{S})}[log(1-D_{r}(G_{r}(\mathbf{x_{s}})))]$

To inhibit the mapping of the input image to any arbitrary permutation of
images in the target domain style, the cycle consistency loss is utilized,
ensuring that the images can revert to their original form within, for
instance, a virtual environment. This is defined as:

$\displaystyle L_{cyc}(G_{r},G_{s})=$
$\displaystyle\mathbb{E}{\mathbf{x_{s}}\sim
p{data}(\mathbf{S})}[\|G_{s}(G_{r}(\mathbf{x_{s}}))-\mathbf{x_{s}}\|_{1}]$ (4)
$\displaystyle+$ $\displaystyle\mathbb{E}{\mathbf{x_{r}}\sim
p{data}(\mathbf{R})}[\|G_{r}(G_{s}(\mathbf{x_{r}}))-\mathbf{x_{r}}\|_{1}]$

Lastly, the identity mapping loss preserves content consistency within the
crop image and is formulated as:

$\displaystyle L_{identity}(G_{r},G_{s})=$
$\displaystyle\mathbb{E}{\mathbf{x_{s}}\sim
p{data}(\mathbf{S})}[\|G_{s}(\mathbf{x_{s}})-\mathbf{x_{s}}\|_{1}]$ (5)
$\displaystyle+$ $\displaystyle\mathbb{E}{\mathbf{x_{r}}\sim
p{data}(\mathbf{R})}[\|G_{r}(\mathbf{x_{r}})-\mathbf{x_{r}}\|_{1}]$

###  III-D DT/MARS-CycleGAN: DT-MARS Consistency loss

Inspired by the DT-CycleGAN model [32] that was originally designed for visual
grasping, a novel DT-MARS consistency loss is introduced here to facilitate
and enforce more coherent sim-to-real transfer of crop images. This additional
loss aims to penalize any disparities in the positions and sizes of the DT
crops represented by the bounding boxes on the image between the original and
the style-transformed image, which is the output of the generator.

Given an image, denoted as $\boldsymbol{x}\in\boldsymbol{S}$, the position and
size of the crops, represented as
$\boldsymbol{a}_{1}=\text{Detector}(\boldsymbol{x})$, should ideally
correspond with $\boldsymbol{a}_{2}=\text{Detector}(G_{r}(\boldsymbol{x}))$.
Therefore, the losses associated with DT-MARS consistency, which
quantitatively assess the discrepancies in the predicted bounding boxes as
depicted in the middle panel of Figure 1, are formulated as:

$\displaystyle L_{DT-MARS}(G_{r},G_{s},\text{Detector})$
$\displaystyle=\mathbb{E}{\boldsymbol{x_{s}}\sim
p{\text{data}}(\boldsymbol{S})}[\|\text{Detector}(G_{r}(\boldsymbol{x_{s}}))-\text{Detector}(\boldsymbol{x_{s}})\|_{1}]$
$\displaystyle+\mathbb{E}{\boldsymbol{x_{r}}\sim
p{\text{data}}(\boldsymbol{R})}[\|\text{Detector}(G_{s}(\boldsymbol{x_{r}}))-\text{Detector}(\boldsymbol{x_{r}})\|_{1}]$
(6)

Thus, the total loss function for DT/MARS-CycleGAN model is represented as:

$\displaystyle L(G_{s},G_{r},D_{s},D_{r},\text{Detector})$
$\displaystyle=\lambda_{\text{gan}}*L_{\text{GAN}}(G_{s},D_{s},\boldsymbol{R},\boldsymbol{S})+\lambda_{\text{gan}}*L_{\text{GAN}}(G_{r},D_{r},\boldsymbol{S},\boldsymbol{R})$
$\displaystyle+\lambda_{\text{cyc}}*L_{\text{cyc}}(G_{r},G_{s})+\lambda_{\text{identity}}*L_{\text{identity}}(G_{r},G_{s})$
$\displaystyle+\lambda_{\text{detector}}*L_{\text{DT-
MARS}}(G_{r},G_{s},\text{Detector})$ (7)

In this expression, $\lambda_{\text{gan}}$, $\lambda_{\text{cyc}}$,
$\lambda_{\text{identity}}$, and $\lambda_{\text{detector}}$ are the
hyperparameters governing the impact of the respective loss components in the
overall loss function. In our model, the values experimentally assigned to
these hyperparameters are $\lambda_{\text{gan}}=1$, $\lambda_{\text{cyc}}=5$,
$\lambda_{\text{identity}}=2$, and $\lambda_{\text{detector}}=10$.

The introduction of the DT-MARS consistency loss ensures that the bounding
boxes, representing the DT crops in both of the real and simulated domains,
are coherent in terms of their positions and dimensions, leading to more
robust and accurate sim-to-real transfer by mitigating the discrepancies
between these two domains.

###  III-E Training Strategies

The training process of the complete framework is divided into two stages: the
first step involves the training of the crop detection model, and the second
engages the DT/MARS-CycleGAN for zero-shot sim-to-real transfer. Both training
stages are conducted exclusively within simulated environments, without the
need for human annotation on the real world farm images.

Training Crop Detection Model: Initially, we constructed the simulation
environment by utilizing the Pybullet package, employing real farm field
images as the background for the simulation environment, and arbitrarily
placing 3D models of crops as target objects at random positions. To enrich
the diversity of the generated simulation images, we collected 500 distinct
publicly available images of farm fields and created 3D models of three
different crop species: sweet beet, polygonum and cirsium at varying growth
stages. Subsequently, images are captured by utilizing a virtual camera
positioned on the DT robot (bottom view for crop/row sensing) within the
simulation environment, and an analytical virtual model extracts the ground
truth bounding boxes for each crop in the image based on the predefined
position information.

The generated simulation crop detection dataset with mimicked appearances and
accurate bounding box annotations is then employed for crop detection model
training. Given the hardware constraints of real crop robot and the necessity
for real-time processing in robotic phenotyping tasks, we employ the YOLOv8n
detection model for lightweight deployment and swift inference. Specifically,
the YOLOv8n weights pre-trained on the COCO dataset serve as the
initialization, followed by a fine-tuning step on our generated simulation
crop detection dataset for domain adaptation.

Training DT/MARS-CycleGAN: By incorporating unlabeled real image frames
extracted from video streams captured by the real robot that navigated along
crop rows in actual farm environments, the reality gap in zero-shot sim-to-
real transfer is notably reduced during the training of DT/MARS-CycleGAN
within the simulation environment. Specifically, 2,400 simulation images with
automatically annotated bounding boxes, and 2,400 real frames extracted from
the video stream are employed. The training configurations align with those
described in [34]. In contrast to RetinaGAN [30], where the sim-to-real
transfer process solely functions as a method to augment data for training the
detection model, our framework facilitates the simultaneous training of the
crop detection model and the DT/MARS-CycleGAN. Subsequent experiments reveal
that this novel methodology achieves superior results when compared to the
RetinaGAN model [30].

##  IV Evaluation

###  IV-A Dataset

The training set is comprised of images from two domains: 2,400 auto-annotated
simulation crop detection data generated from the PyBullet simulation
environment, and 2,400 real frames extracted from the video stream as
previously detailed. The simulation data comprises three crop species: sweet
beet, polygonum, and cirsium at different growth stages, while the real data
includes two crop species: collard and kale. To assess the performance of the
trained model, we assembled a high-quality test set including 408 unseen image
frames gathered by the real robot in farm fields. Each image is manually
labeled by utilizing the RainbowFlow labeling tool, without data augmentation,
and is formatted in the YOLO object detection dataset format.

###  IV-B Zero-shot Sim-to-Real Transfer

We assess four distinct zero-shot sim-to-real transfer methods on the created
dataset:

  * •

Sim-Only: Training solely on simulation data, without any sim-to-real
transfer.

  * •

CycleGAN: Training on sim-to-real transferred images generated by CycleGAN.

  * •

RetinaGAN: Training on sim-to-real transferred images with additional
perception consistency, generated by RetinaGAN.

  * •

The proposed DT/MARS-CycleGAN.

Figure 2: Simulation-to-Real synthetic images generated by different models.

The crop detection model we utilize is YOLOv8n, which is chosen for
lightweight deployment and the necessity of real-time inference in real-world
agricultural applications. In all four sim-to-real transfer methods, the
YOLOv8n detection model utilizes the pretrained weights on the COCO dataset
for initialization and are trained for 100 epochs with a batch size of 64,
learning rate of 1e-2, weight decay of 5e-4, a cosine learning rate scheduler,
and the IoU loss function. All four models are trained exclusively in
simulation environments and are never fine-tuned on labeled real-world data to
ensure zero-shot sim-to-real transfer.

Figure 2 displays the visualizations of generated sim-to-real images by
different methods, indicating that although CycleGAN may effectively transfer
background information within an image, information pertinent to the crop
target is often distorted or even lost. Conversely, with the additional
consistency loss of target detection, RetinaGAN can somewhat adequately
preserve target information; however, the background transformed by this model
appears uniform and repetitive, lacking in variety. This could be due to the
fixed detection model parameters in RetinaGAN, which constrain the model
within the simulation domain and hamper its adaptability in the real domain,
leading to an incomplete transfer of background style. In contrast, DT/MARS-
CycleGAN showcases superior performance by enabling the concurrent training of
the crop detection model and the DT/MARS-CycleGAN model, thereby emphasizing
the effectiveness of our framework in addressing reality gaps in zero-shot
sim-to-real transfer scenarios.

Therefore, as shown in Table I, the proposed DT/MARS-CycleGAN achieves
superior performance on the test dataset collected in the real-world
environment, demonstrating promising zero-shot sim-to-real transfer
capability. In contrast, due to the target position shift and monotonous
background appearance, both the CycleGAN and RetinaGAN methods yield even
poorer performance than Sim-Only. Figure 3 provides qualitative visualizations
of crop detection for these methods. The proposed DT/MARS-CycleGAN model
effectively reduces both false positives and false negatives in comparison to
other methods.

Figure 3: Visualization of crop detection across different methods. TABLE I: The crop detection results of YOLOv8 with different sim-real transfer methods. Methods | P | R | mAP50 | mAP50-95
---|---|---|---|---
Sim-Only | 0.925 | 0.827 | 0.917 | 0.656
CycleGAN | 0.500 | 0.422 | 0.411 | 0.156
RetinaGAN | 0.741 | 0.573 | 0.650 | 0.221
DT/MARS-CycleGAN | 0.942 | 0.895 | 0.964 | 0.674
TABLE II: The crop row detection results with different sim-to-real transfer methods. Method | MAE with RANSAC | MAE with Linefit
---|---|---
Angle(deg) | Dist(px) | Angle(deg) | Dist(px)
Sim-Only | 2.47 | 4.34 | 1.23 | 2.55
CycleGAN | 9.4 | 32.18 | 8.83 | 27.84
RetinaGAN | 8.05 | 22.56 | 7.20 | 27.49
DT/MARS | 1.67 | 3.33 | 1.00 | 2.44

###  IV-C Crop Row Detection

Crop row detection is crucial for determining the direction of movement in the
robot’s visual navigation, as well as providing supplemental visual
constraints for a multi-sensor fusion navigation approach, as shown in prior
studies [33]. In the context of multiple visual servo controllers, the crop
row detection process can be viewed as a sensor. This visual sensor
consistently generates signals depicting the error offset from a reference, as
illustrated in Figure 4:

Angular Offset $\theta$: represents the angle at which the detected crop line
that deviates from the vertical orientation.

Horizontal Offset L: denotes the discrepancy between the image’s central point
and the detected line’s center point.

Figure 4: Illustration of the crop row detection. (a) shows the two primary
parameters to evaluate the row detection: angular offset and distance offset;
(b) is an example of row detection in the real farm field.

In the crop row detection evaluation, the four trained models with synthesized
data (Sim-only, CycleGAN, RetinaGAN, and DT/MARS-CycleGAN) were first used for
crop detection and to calculate the central points of each crop. Then two
common line fitting algorithms including RANSAC and basic line fitting
algorithm with the least squares method were used to calculate the two offset
parameters $\theta$ and L. Finally, we calculated the offset difference
between the ground truth and the predicted one with the trained models.

The mean absolute error (MAE) of offset difference was used to evaluate the
crop row detection. As shown in Table II, our proposed DT/MARS-CycleGAN
exhibits less crop row detection error in both angular offset error and
horizontal offset error. The main reason is that the crop detector trained
with DT/MARS-CycleGAN generated images possesses better zero-shot transfer
ability in real-world farm field situations. In addition, the overall
performances of the basic line fitting algorithm with the least squares method
are better than RANSAC, and the main reason is that the number of crops in
testing images are mainly in the range of 3 to 6 and the RANSAC-based
algorithm sometimes could filter abnormal values that actually belong to the
real crops. The improved crop row detection method will be deployed for
automated navigation of the MARS PhenoBot in real-world crop farm field in the
future.

TABLE III: The crop detection results of YOLOv8 with different numbers of training samples in simulation. # of training samples | P | R | mAP50 | mAP50-95
---|---|---|---|---
100 | 0.969 | 0.739 | 0.903 | 0.607
300 | 0.933 | 0.724 | 0.899 | 0.606
600 | 0.937 | 0.803 | 0.899 | 0.641
1200 | 0.923 | 0.808 | 0.902 | 0.649
2400 | 0.925 | 0.827 | 0.917 | 0.656
TABLE IV: The crop detection results of YOLOv8 trained on simulation images of different crop species. Crop | P | R | mAP50 | mAP50-95
---|---|---|---|---
Sugar beet | 0.909 | 0.802 | 0.889 | 0.612
Polygonum | 0.846 | 0.696 | 0.807 | 0.559
Cirsium | 0.966 | 0.616 | 0.885 | 0.585
Mixed | 0.925 | 0.827 | 0.917 | 0.656
TABLE V: The crop detection results of YOLOv8 trained on simulation images of sugar beet at different growth stages. Crop | P | R | mAP50 | mAP50-95
---|---|---|---|---
Seedling | 0.886 | 0.552 | 0.748 | 0.452
Well-grown | 0.907 | 0.802 | 0.878 | 0.599
Mixed | 0.909 | 0.802 | 0.889 | 0.612
TABLE VI: The crop detection results of YOLOv8 trained on simulation images
of different backgrounds and numbers of objects per image. Setting | P | R | mAP50 | mAP50-95
---|---|---|---|---
single object + single background | 0.512 | 0.466 | 0.516 | 0.339
single object + multi-background | 0.238 | 0.905 | 0.849 | 0.550
multi-object + single background | 0.607 | 0.927 | 0.910 | 0.613
multi-object + multi-background | 0.925 | 0.827 | 0.917 | 0.656

###  IV-D Ablation Studies

Evaluation on Different Training Data Sizes: Our proposed DT/MARS-CycleGAN
model significantly diminishes the human effort required to accumulate labeled
real-world training data of annotated crop object bounding boxes. By
eliminating this laborious endeavor, we can harness substantial amounts of
simulation data to enhance zero-shot sim-to-real transfer further. To quantify
the influence of the size of simulation data, we trained the crop detection
model with varying sample sizes in the simulation environment. As depicted in
Table III, the mAP50 elevates from 0.903 to 0.917 by merely augmenting
simulation data from 100 to 2,400, and the mAP50-90 ascends from 0.607 to
0.656. These findings underscore that ample simulation data can notably boost
detection performance and consequently aid in achieving high-quality sim-to-
real transfer by contributing to the DT-MARS consistency loss in the proposed
DT/MARS-CycleGAN.

Evaluation on Different Crop Species and Growth Stages: Crop species and
growth stages can significantly influence a model’s generalizability, as
appearances across different species and growth stages can be either highly
similar or distinctly different. Selecting an optimal set of crop species and
growth stages for generating simulation images can improve the performance of
crop detection models pre-trained on such simulation data. In Table IV, we
trained the detection model on simulation images with various species. Results
indicate that sugar beet achieves better detection performance compared to
other species. This is likely because its appearance closely resembles that of
collard and kale in the test images. Additionally, training data that combines
various species outperforms data from any single species, suggesting that
diverse species combinations can enrich training data and thus enhance model
generalizability. Table V presents the detection performance based on training
with different growth stages. Similarly, mixed data covering all growth stages
surpasses training that focuses solely on one stage.

Evaluation on Different Backgrounds and Number of Objects per Image: The
images generated in the simulation environment are pivotal to the proposed
framework as they compose the annotated crop images for both stages of the
training. To evaluate the effectiveness of varying generation settings in the
simulation environment, we conducted object detection model training using
different combinations presented in Table VI. As depicted in Table VI, the
object detection model achieves enhanced performance with a substantial number
of field images used as environment backgrounds and multiple 3D crop models,
underscoring the high diversity and quality of the simulation images generated
by our proposed framework.

##  V Conclusions and Discussions

This paper introduces a novel DT/MARS-CycleGAN framework, effectively bridging
the reality gap between simulated and real-world environments and facilitating
effective zero-shot sim-to-real transfer in robotic crop detection. By
imposing both cycle and DT-MARS consistency losses, which penalizes
discrepancies in visual appearance and crop target between synthesized and
real-world crop images, the proposed method achieves highly effective sim-to-
real transfer. The fine-tuned object detectors on this diverse and high-
quality synthetic data substantially elevates detection performance. Extensive
experiments demonstrate that the proposed DT/MARS-CycleGAN framework enables
more robust crop perception from complex backgrounds, advancing the field of
robotic crop phenotyping. This work provides an effective solution to a
critical robotic vision challenge in unstructured and complex agricultural
environments, moving towards more productive and resilient crop phenotyping
and crop breeding process in the future.

This work can be further extended along the general direction of foundation
models and large vision models [73, 74] in the future. Though our DT/MARS-
CycleGAN framework demonstrated a reasonably good zero-shot transfer
capability, its performance in dealing with the variation of real-world crop
images can be further improved using the new methodology of foundation model
[73]. To realize such a goal, the backbones of our DT/MARS-CycleGAN and object
detection networks would be upgraded by the more powerful ViT model and its
variants [42, 44, 75], and should be trained and fine-tuned on much larger
scale of datasets, towards a foundation model of crop image understanding and
other tasks in the future.

## References

  * [1] A. Atefi, Y. Ge, S. Pitla, and J. Schnable, “Robotic technologies for high-throughput plant phenotyping: Contemporary reviews and future perspectives,” _Frontiers in Plant Science_ , vol. 12, 2021.
  * [2] M. M. Rahaman, D. Chen, Z. Gillani, C. Klukas, and M. Chen, “Advanced phenotyping and phenotype data analysis for the study of plant growth and development,” _Frontiers in Plant Science_ , vol. 6, 2015.
  * [3] Y. Jiang and C. Li, “Convolutional neural networks for image-based high-throughput plant phenotyping: A review,” _Plant Phenomics_ , vol. 2020, 2020.
  * [4] R. Xu and C. Li, “A review of high-throughput field phenotyping systems: Focusing on ground robots,” _Plant Phenomics_ , vol. 2022, 2022.
  * [5] S. Moeinizade, G. Hu, L. Wang, and P. S. Schnable, “Optimizing Selection and Mating in Genomic Selection with a Look-Ahead Approach: An Operations Research Framework,” _G3 Genes—Genomes—Genetics_ , vol. 9, no. 7, pp. 2123–2133, 07 2019.
  * [6] L. Wang, G. Zhu, W. Johnson, and M. Kher, “Three new approaches to genomic selection,” _Plant Breeding_ , vol. 137, no. 5, pp. 673–681, 2018.
  * [7] J. Iqbal, R. Xu, H. Halloran, and C. Li, “Development of a multi-purpose autonomous differential drive mobile robot for plant phenotyping and soil sensing,” _Electronics_ , vol. 9, no. 9, 2020.
  * [8] A. Chawade, J. van Ham, H. Blomquist, O. Bagge, E. Alexandersson, and R. Ortiz, “High-throughput field-phenotyping tools for plant breeding and precision agriculture,” _Agronomy_ , vol. 9, no. 5, 2019.
  * [9] A. Mazis, S. D. Choudhury, P. B. Morgan, V. Stoerger, J. Hiller, Y. Ge, and T. Awada, “Application of high-throughput plant phenotyping for assessing biophysical traits and drought response in two oak species under controlled environment,” _Forest Ecology and Management_ , vol. 465, p. 118101, 2020\.
  * [10] F. Solimani, A. Cardellicchio, M. Nitti, A. Lako, G. Dimauro, and V. RenÃ², “A systematic review of effective hardware and software factors affecting high-throughput plant phenotyping,” _Information_ , vol. 14, no. 4, 2023\.
  * [11] T. Gao, H. Emadi, H. Saha, J. Zhang, A. Lofquist, A. Singh, B. Ganapathysubramanian, S. Sarkar, A. K. Singh, and S. Bhattacharya, “A novel multirobot system for plant phenotyping,” _Robotics_ , vol. 7, no. 4, 2018.
  * [12] M. Jenkins and G. Kantor, “Online detection of occluded plant stalks for manipulation,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2017, pp. 5162–5167.
  * [13] R. Xu and C. Li, “A modular agricultural robotic system (mars) for precision farming: Concept and implementation,” _Journal of Field Robotics_ , vol. 39, no. 4, pp. 387–409, 2022.
  * [14] Y. Qiao, J. Valente, D. Su, Z. Zhang, and D. He, “Editorial: Ai, sensors and robotics in plant phenotyping and precision agriculture,” _Frontiers in Plant Science_ , vol. 13, 2022.
  * [15] B. Darwin, P. Dharmaraj, S. Prince, D. E. Popescu, and D. J. Hemanth, “Recognition of bloom/yield in crop images using deep learning models for smart agriculture: A review,” _Agronomy_ , vol. 11, no. 4, p. 646, mar 2021\.
  * [16] G. Farjon, H. Liu, and Y. Edan, “Deep-learning-based counting methods, datasets, and applications in agriculture: a review,” _Precision Agriculture_ , vol. 24, 2023.
  * [17] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014, pp. 580–587.
  * [18] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 779–788.
  * [19] I. Sa, Z. Ge, F. Dayoub, B. Upcroft, T. Perez, and C. McCool, “Deepfruits: A fruit detection system using deep neural networks,” _Sensors_ , vol. 16, no. 8, 2016.
  * [20] K. Bresilla, G. D. Perulli, A. Boini, B. Morandi, L. Corelli Grappadelli, and L. Manfrini, “Single-shot convolution neural networks for real-time fruit detection within the tree,” _Frontiers in Plant Science_ , vol. 10, 2019\.
  * [21] N. Ragu and J. Teo, “Object detection and classification using few-shot learning in smart agriculture: A scoping mini review,” _Frontiers in Sustainable Food Systems_ , vol. 6, 2023.
  * [22] N. Giakoumoglou, E. Pechlivani, and D. Tzovaras, “Generate-paste-blend-detect: Synthetic dataset for object detection in the agriculture domain,” _Smart Agricultural Technology_ , vol. 5, p. 100258, 05 2023.
  * [23] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _Communications of the ACM_ , vol. 63, pp. 139–144, 2020.
  * [24] C. Karam, M. Awad, Y. Abou Jawdah, N. Ezzeddine, and A. Fardoun, “Gan-based semi-automated augmentation online tool for agricultural pest detection: A case study on whiteflies,” _Frontiers in Plant Science_ , vol. 13, 2022.
  * [25] L. Bi and G. Hu, “Improving image-based plant disease classification with generative adversarial network under limited training set,” _Frontiers in Plant Science_ , vol. 11, 2020.
  * [26] K. Kleeberger, R. Bormann, W. Kraus, and M. F. Huber, “A survey on learning-based robotic grasping,” _Current Robotics Reports_ , vol. 1, no. 4, pp. 239–249, 2020.
  * [27] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan, “Unsupervised pixel-level domain adaptation with generative adversarial networks,” in _CVPR_ , 2017, pp. 3722–3731.
  * [28] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in _2018 ICRA_. IEEE, 2018, pp. 3803–3810.
  * [29] V. M. Patel, R. Gopalan, R. Li, and R. Chellappa, “Visual domain adaptation: A survey of recent advances,” _IEEE signal processing magazine_ , vol. 32, no. 3, pp. 53–69, 2015.
  * [30] D. Ho, K. Rao, Z. Xu, E. Jang, M. Khansari, and Y. Bai, “Retinagan: An object-aware approach to sim-to-real transfer,” in _2021 ICRA_. IEEE, 2021, pp. 10 920–10 926.
  * [31] K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari, “Rl-cyclegan: Reinforcement learning aware simulation-to-real,” in _CVPR_ , 2020, pp. 11 157–11 166.
  * [32] D. Liu, Y. Chen, and Z. Wu, “Digital twin (dt)-cyclegan: Enabling zero-shot sim-to-real transfer of visual grasping models,” _IEEE Robotics and Automation Letters_ , vol. 8, no. 5, pp. 2421–2428, 2023.
  * [33] Z. Li, R. Xu, C. Li, and L. Fu, “Simulation of an in-field phenotyping robot: System design, vision-based navigation and field mapping,” in _2022 ASABE Annual International Meeting_. American Society of Agricultural and Biological Engineers, 2022, p. 1.
  * [34] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _ICCV_ , 2017, pp. 2223–2232.
  * [35] J. Solawetz, “What is yolov8? the ultimate guide,” _https://blog.roboflow.com/whats-new-in-yolov8/_ , 2023.
  * [36] L. Grimstad and P. J. From, “The thorvald ii agricultural robotic system,” _Robotics_ , vol. 6, no. 4, 2017.
  * [37] N. Virlet, K. Sabermanesh, P. Sadeghi-Tehran, and M. Hawkesford, “Field scanalyzer: An automated robotic field phenotyping platform for detailed crop monitoring,” _Functional Plant Biology_ , vol. 44, pp. 143–153, 11 2016\.
  * [38] M. Liu, S. Fang, H. Dong, and C. Xu, “Review of digital twin about concepts, technologies, and industrial applications,” _Journal of Manufacturing Systems_ , vol. 58, pp. 346–361, 2021.
  * [39] W. Purcell and T. Neubauer, “Digital twins in agriculture: A state-of-the-art review,” _Smart Agricultural Technology_ , vol. 3, p. 100094, 2023.
  * [40] Y. Liu and et al., “A digital twin-based sim-to-real transfer for deep reinforcement learning-enabled industrial robot grasping,” _Robotics and Computer-Integrated Manufacturing_ , vol. 78, p. 102365, 2022.
  * [41] X. Mai, H. Zhang, and M. Q.-H. Meng, “Faster r-cnn with classifier fusion for small fruit detection,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 7166–7172.
  * [42] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” _CoRR_ , vol. abs/2010.11929, 2020.
  * [43] S. Parez, N. Dilshad, N. S. Alghamdi, T. M. Alanazi, and J. W. Lee, “Visual Intelligence in Precision Agriculture: Exploring Plant Disease Detection via Efficient Vision Transformers,” _Sensors_ , vol. 23, no. 15, p. 6949, Aug. 2023.
  * [44] Y. Chen, Z. Xiao, L. Zhao, L. Zhang, H. Dai, D. W. Liu, Z. Wu, C. Li, T. Zhang, C. Li, D. Zhu, T. Liu, and X. Jiang, “Mask-guided vision transformer (mg-vit) for few-shot learning,” _arxiv: https://arxiv.org/abs/2205.09995_ , 2022.
  * [45] T. T. Santos, L. L. de Souza, A. A. dos Santos, and S. Avila, “Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association,” _Computers and Electronics in Agriculture_ , vol. 170, p. 105247, 2020.
  * [46] P. K. Sekharamantry, F. Melgani, and J. Malacarne, “Deep learning-based apple detection with attention module and improved loss function in yolo,” _Remote Sensing_ , vol. 15, no. 6, 2023.
  * [47] Y. Chen, X. An, S. Gao, S. Li, and H. Kang, “A deep learning-based vision system combining detection and tracking for fast on-line citrus sorting,” _Frontiers in Plant Science_ , vol. 12, 2021.
  * [48] Y. Lu and S. Young, “A survey of public datasets for computer vision tasks in precision agriculture,” _Computers and Electronics in Agriculture_ , vol. 178, p. 105760, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0168169920312709
  * [49] R. Polvara, S. Molina, I. Hroob, A. Papadimitriou, K. Tsiolis, D. Giakoumis, S. Likothanassis, D. Tzovaras, G. Cielniak, and M. Hanheide, “Bacchus long-term (blt) data set: Acquisition of the agricultural multimodal blt data set with automated robot deployment,” _Journal of Field Robotics_ , 2023\.
  * [50] C. M. de Melo, A. Torralba, L. Guibas, J. DiCarlo, R. Chellappa, and J. Hodgins, “Next-generation deep learning based on simulators and synthetic data,” _Trends in cognitive sciences_ , 2022.
  * [51] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IROS_. IEEE, 2017, pp. 23–30.
  * [52] N. E. Khalifa, M. Loey, and S. Mirjalili, “A comprehensive survey of recent trends in deep learning for digital images augmentation,” _Artificial Intelligence Review_ , pp. 1–27, 2022.
  * [53] Y. Lu, D. Chen, E. Olaniyi, and Y. Huang, “Generative adversarial networks (gans) for image augmentation in agriculture: A systematic review,” _Computers and Electronics in Agriculture_ , vol. 200, p. 107208, 2022.
  * [54] M. Valerio Giuffrida, H. Scharr, and S. A. Tsaftaris, “Arigan: Synthetic arabidopsis plants using generative adversarial network,” in _Proceedings of the IEEE international conference on computer vision workshops_ , 2017, pp. 2064–2071.
  * [55] Y. Zhu, M. Aoun, M. Krijn, J. Vanschoren, and H. T. Campus, “Data augmentation using conditional generative adversarial networks for leaf counting in arabidopsis plants.” in _BMVC_ , 2018, p. 324.
  * [56] S. L. Madsen, M. Dyrmann, R. N. Jørgensen, and H. Karstoft, “Generating artificial images of plant seedlings using generative adversarial networks,” _Biosystems Engineering_ , vol. 187, pp. 147–159, 2019.
  * [57] S. L. Madsen, A. K. Mortensen, R. N. Jørgensen, and H. Karstoft, “Disentangling information in artificial images of plant seedlings using semi-supervised gan,” _Remote Sensing_ , vol. 11, no. 22, p. 2671, 2019.
  * [58] F. Zhu, M. He, and Z. Zheng, “Data augmentation using improved cdcgan for plant vigor rating,” _Computers and Electronics in Agriculture_ , vol. 175, p. 105603, 2020.
  * [59] L. Drees, L. V. Junker-Frohn, J. Kierdorf, and R. Roscher, “Temporal prediction and evaluation of brassica growth in the field using conditional generative adversarial networks,” _Computers and Electronics in Agriculture_ , vol. 190, p. 106415, 2021.
  * [60] S. Shete, S. Srinivasan, and T. A. Gonsalves, “Tasselgan: An application of the generative adversarial model for creating field-based maize tassel data,” _Plant Phenomics_ , vol. 2020, 2020.
  * [61] E. David, S. Madec, P. Sadeghi-Tehran, H. Aasen, B. Zheng, S. Liu, N. Kirchgessner, G. Ishikawa, K. Nagasawa, M. A. Badhon _et al._ , “Global wheat head detection (gwhd) dataset: a large and diverse dataset of high-resolution rgb-labelled images to develop and benchmark wheat head detection methods,” _Plant Phenomics_ , 2020.
  * [62] Z. K. Hartley and A. P. French, “Domain adaptation of synthetic images for wheat head detection,” _Plants_ , vol. 10, no. 12, p. 2633, 2021.
  * [63] M. Rahnemoonfar and C. Sheppard, “Deep count: fruit counting based on deep simulated learning,” _Sensors_ , vol. 17, no. 4, p. 905, 2017.
  * [64] D. Dwibedi, I. Misra, and M. Hebert, “Cut, paste and learn: Surprisingly easy synthesis for instance detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1301–1310.
  * [65] A. Picon, M. G. San-Emeterio, A. Bereciartua-Perez, C. Klukas, T. Eggers, and R. Navarra-Mestre, “Deep learning-based segmentation of multiple species of weeds and corn crop using synthetic and real image datasets,” _Computers and Electronics in Agriculture_ , vol. 194, p. 106719, 2022.
  * [66] C. Wang, Y. Wang, S. Liu, G. Lin, P. He, Z. Zhang, and Y. Zhou, “Study on pear flowers detection performance of yolo-pefl model trained with synthetic target images,” _Frontiers in Plant Science_ , vol. 13, p. 911473, 2022.
  * [67] W. Qiu and A. Yuille, “Unrealcv: Connecting computer vision to unreal engine,” in _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14_. Springer, 2016, pp. 909–916.
  * [68] D. Andújar, M. Calle, C. Fernández-Quintanilla, Á. Ribeiro, and J. Dorado, “Three-dimensional modeling of weed plants using low-cost photogrammetry,” _Sensors_ , vol. 18, no. 4, p. 1077, 2018.
  * [69] W. Vierbergen, A. Willekens, D. Dekeyser, S. Cool _et al._ , “Sim2real flower detection towards automated calendula harvesting,” _Biosystems Engineering_ , vol. 234, pp. 125–139, 2023.
  * [70] M. Cieslak, N. Khan, P. Ferraro, R. Soolanayakanahally, S. J. Robinson, I. Parkin, I. McQuillan, and P. Prusinkiewicz, “L-system models for image-based phenomics: case studies of maize and canola,” _in silico Plants_ , vol. 4, no. 1, p. diab039, 2022.
  * [71] J. Iqbal, R. Xu, S. Sun, and C. Li, “Simulation of an autonomous mobile robot for LiDAR-based in-field phenotyping and navigation,” _Robotics_ , vol. 9, no. 2, p. 46, jun 2020.
  * [72] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” _Commun. ACM_ , vol. 24, no. 6, p. 381–395, jun 1981.
  * [73] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang, “On the opportunities and risks of foundation models,” 2021.
  * [74] C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He _et al._ , “A comprehensive survey on pretrained foundation models: A history from bert to chatgpt,” _https://arxiv.org/abs/2108.07258_ , 2023\.
  * [75] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo _et al._ , “Segment anything,” _arXiv preprint arXiv:2304.02643_ , 2023.


# Pose-aware 3D Beamwidth Adaptation for Mobile Extended Reality

Alperen Duru1, Mohammad Mozaffari2, Mehrnaz Afshang2, Ticao Zhang2, Talha
Khan2,
Todd E. Humphreys1, and Jeffrey G. Andrews1  1University of Texas at Austin,
Austin, TX, USA
Emails: {aduru, todd.humphreys<EMAIL_ADDRESS>2Ericsson Research,
Silicon Valley, Santa Clara, CA, USA.
Emails: {mohammad.mozaffari, mehrnaz.afshang, ticao.zhang,
<EMAIL_ADDRESS>
###### Abstract

This paper presents a sensor-aided pose-aware beamwidth adaptation design for
a conceptual extended reality (XR) Head-Mounted Display (HMD) equipped with a
2D planar array. The beam is tracked and adapted on the user side by
leveraging HMD orientation estimates. The beamwidth adaptation scheme is
effected by selective deactivation of elements in the 2D antenna array,
employing the angular estimation covariance matrix to overlap the beam with
the estimation confidence interval. The proposed method utilizes the
estimation correlations to adapt the beamwidth along the confidence interval
of these estimates. Compared to a beamwidth adaptation without leveraging
estimation correlations, the proposed method demonstrates the gain of
leveraging estimation correlations by improving the coverage area for a given
outage probability threshold by approximately 16%, or equivalently increasing
the power efficiency up to 18%.

###### Index Terms:

Extended Reality, XR, Pose-aware, Sensor-aided, Beamwidth Adaptation,
Confidence Interval.

##  I Introduction

Extended reality, which encompasses virtual, augmented, and mixed realities,
demands data rates up to Gbps levels and latency as low as few tens of
milliseconds. Advances towards meeting these demands in communications for XR
are being pursued as a part of 5G-Advanced and are a key part of 5G-Advanced
standardization. Within an XR communication framework integrated with edge
computing, prolonged battery life is also a key requirement of user equipment
[1]. The high data rate demands of XR require increased bandwidth, and
consequently, higher carrier frequencies. Higher frequency signals face higher
penetration losses and blockage [2], motivating the adoption of multiple
antenna communication frameworks with beamforming capabilities.

Although beamforming can counteract path loss at high frequencies, it
introduces a connectivity challenge for XR applications. Rapid changes in XR
HMD’s position and orientation complicate the precise tracking of the
beamforming direction. Such inaccuracies can lead to beam misalignment,
resulting in communication outages, particularly with narrow beamwidths. A
narrow beamwidth makes the system susceptible to communication outages arising
from headset position and orientation uncertainty, highlighting the need for a
robust beam adaptation algorithm.

User-side beamwidth adaptation can mitigate beam misalignment issues stemming
from high mobility and headset rotations, thereby saving energy and improving
coverage. At pedestrian speeds, the primary obstacle to tracking the
beamforming direction arises from inaccuracy in headset orientation
estimation. The proposed user-side beamwidth adaptation scheme is based on
selective deactivation of elements in the 2D antenna array to maximize overlap
between the beam and the uncertainty ellipse of the desired beamforming
direction.

###  I-A Related Work and Its Limitations

Prior literature on beamwidth adaptation extensively focuses on vehicular
channels where the beam is adapted at the Base Station (BS), particularly for
high vehicle position uncertainty or high vehicle speed. Studies specific to
user-side beamwidth adaptation, especially concerning HMDs, are scarce. This
paper aims to address this gap, which is pertinent to the XR communication
landscape.

Authors of [3] explore beamwidth adaptation for high-speed train
infrastructures, addressing location and velocity uncertainties. Similarly,
[4] and [5] use shared user locations with BS for adapting beamwidth at the BS
side. These works largely rely on 1D antenna configurations, in contrast to
the 2D antenna approach adopted in this paper. A user-side beam adaptation for
a 2D antenna with three degrees of freedom (3DoF) is proposed in [6], where
the focus is indoor environments. Authors of [7] consider outdoor path loss
effects by adjusting the beam according to UAV’s distance to BS, but the study
overlooks user rotations. Many studies such as [8, 9, 10, 11], where [8] uses
antenna deactivation, either address mmWave or terahertz frequency beamwidth
adaptations with a focus on 1D antenna arrays or explore multi-level 3D
beamforming techniques without considering sensor estimation effects on the
communication outages. In [12], the authors discuss the utilization of
confidence intervals, with beamwidth adaptation being restricted solely to
either azimuth or elevation dimensions. Authors of [13, 14] leveraged the
covariance matrices to their full extent, however, they do not address the
power saving concerns of XR.

The existing body of work either gravitates towards indoor settings,
permitting broader beamwidths at the cost of gain, or leans on vehicular
channel assumptions, not incorporating the 3DoF capabilities or low-power
requirements of XR HMDs. This underscores a need for a versatile beamwidth
adaptation approach suitable for diverse environments and universally
applicable to 6DoF terminals, be it a BS or User Equipment (UE).

###  I-B Contributions

This paper offers an analysis of XR communication from the user’s perspective,
emphasizing path loss effects and beamwidth adaptation for 2D antenna arrays
with 6DoF. The paper’s primary contributions are (1) a user-side 3D beamwidth
adaptation technique leveraging the 2D antenna array characteristics for
outdoor scenarios and (2) analysis and simulations showing coverage distance
and power efficiency improvements.

The proposed beamwidth adaptation improves the communication coverage region
by up to 16% and the power efficiency of the receive antennas by up to 18%
compared to a beamwidth adaptation without leveraging estimation correlations.
Increased power efficiency prolongs the battery life of the XR HMD, which is a
key requirement for an immersive XR. This paper extends the current literature
on beamwidth adaptation to the outdoor scenario for a 2D antenna array with
6DoF characteristics. The analysis considers the required array gain combined
with the beam coverage area by incorporating path loss effects. The generated
beamwidth adaptation algorithm is applicable to any 6DoF terminal with a 2D
antenna array structure.

##  II System Model

An XR headset communicating with the BS in downlink is considered. The headset
is assumed to be equipped with dual Global Navigation Satellite Systems (GNSS)
antennas and an Inertial Measurement Unit (IMU) sensor for precise position or
orientation estimation. These estimates undergo filtration via an Unscented
Kalman Filter (UKF), producing data on location, orientation, translational
and rotational velocities, and their associated covariance matrices. The XR
HMD is equipped with a 2D antenna array operating at 28 GHz, featuring
constant power per antenna and the capability to deactivate individual
elements, as illustrated in Fig. 1. Operating at pedestrian speeds of 5 km/h
on a horizontal plane, the headset supports 3DoF rotation. The UE to BS
distance spans 10 m to 4 km, with orientation accuracy becoming crucial for
beam alignment and minimizing communication outages at greater distances.

It is assumed that the initial alignment or realignment of beamforming is
established in a Line-of-Sight (LOS) link between a singular user and a BS.
The XR device conveys its location to the BS via a reliable communication link
different than the 28 GHz link, enabling the BS to direct its beam accurately
toward the user. Given the precise shared location and limited user speed, the
user’s location changes only cause small angular errors for the beamforming
from the BS. Thus, as a minor simplification, the beamforming from the BS is
assumed to be flawless. Therefore, this paper exclusively focuses on alignment
errors stemming from user orientation estimation errors.

Figure 1: Conceptual XR design with onboard GNSS and IMU sensors.

The analysis depends on distance-dependent path loss and neglects fast and
slow fading effects. Fast fading is neglected due to the assumed strong LOS
link being dominant compared to low-power reflections. Shadowing is also
neglected, as the angular changes due to HMD rotations are much faster than
the changes due to shadowing. With these assumptions, the received signal
power is modeled as

$P_{\mathrm{rx}}=P_{\mathrm{tx}}G_{\mathrm{tx}}G_{\mathrm{rx}}\left(\frac{\lambda}{4\pi
d}\right)^{\alpha},$ (1)

where $P_{\mathrm{rx}}$ and $P_{\mathrm{tx}}$ are receive and transmit powers,
$G_{\mathrm{tx}}$ and $G_{\mathrm{rx}}$ are the beamforming gains at the
transmitter and receiver, $\alpha$ is the path loss exponent, $\lambda$ is the
carrier wavelength, and $d$ is the distance between transmitter and receiver.
With the system assumptions, $G_{\mathrm{tx}}$ and
$G_{\mathrm{rx}}(\theta,\psi)$ are defined as

$G_{\mathrm{tx}}=N_{\mathrm{tx}}\quad
G_{\mathrm{rx}}(\theta,\psi)=\mathrm{AF}(\theta,\psi),$ (2)

where $\theta$ and $\psi$ are the elevation and azimuth angles,
$\operatorname{AF}(\theta,\psi)$ is the receive beamforming array factor (AF),
and $N_{\mathrm{tx}}$ is the number of transmit antennas. Due to the low
mobility of the user, beamforming from the BS is assumed to be flawless within
the signal reception time, and the transmit beamforming gain is thus set as
the number of antennas.

The 3GPP antenna pattern is adopted in this study, where the effective area
gain of a single antenna consists of vertical and horizontal, or elevation and
azimuth angles, respectively. The vertical and horizontal radiation patterns
are given as follows [15]:

$\displaystyle\mathrm{A}_{\mathrm{E},\mathrm{V}}(\theta)=-\min\left\\{12\left(\frac{\theta}{\theta_{3\mathrm{~{}dB}}}\right)^{2},\mathrm{SL}_{\mathrm{V}}\right\\}$
(3)
$\displaystyle\mathrm{A}_{\mathrm{E},\mathrm{H}}(\psi)=-\min\left\\{12\left(\frac{\psi}{\psi_{3\mathrm{~{}dB}}}\right)^{2},\mathrm{A}_{\mathrm{m}}\right\\},$

where $\mathrm{SL}_{\mathrm{V}}$ is the side-lobe level limit and
$\mathrm{A}_{\mathrm{m}}$ is the front-back ratio. The overall 3GPP antenna
gain pattern for an individual antenna in terms of $\mathrm{dB}$ is given
below, where $G_{\max}$ is the maximum gain in the main lobe center.

$\mathrm{AP}_{3\mathrm{GPP}}(\theta,\psi)=G_{\mathrm{max}}-\min\left\\{-\mathrm{A}_{\mathrm{E},\mathrm{V}}(\theta)-\mathrm{A}_{\mathrm{E},\mathrm{H}}(\psi),\mathrm{A_{m}}\right\\}$
(4)

The 3GPP antenna pattern is used to adjust the AF. Since the user orientation
estimation errors are assumed to be the main challenge for beam alignment, the
AF at the receiver should be adapted.

##  III Outage Probability

A low outage probability is pivotal for latency, as every outage triggers a
beam realignment, thereby increasing latency and reducing energy efficiency.
In the XR context, where latency and energy are constrained, a beamforming
algorithm resilient to estimation uncertainties is essential to preserve the
user quality of experience.

An outage is declared if the received SNR falls below a certain threshold.
From the array gain perspective, the SNR threshold transforms into a
beamforming gain threshold $G_{0}$ as

$\mathrm{AF}(\theta,\psi)<G_{0}.$ (5)

The UKF models the source direction of arrival (DoA) estimation as a jointly
Gaussian random variable across elevation and azimuth angles, which is
characterized entirely by its covariance matrix $\Sigma_{\theta,\psi}$. The
outage probability due to misalignment can be expressed as an integral over a
specific region determined by the gain threshold and AF. Let
$\boldsymbol{x}=[\theta,\psi]^{\mathrm{T}}$, let
$\boldsymbol{\mu}=[\bar{\theta},\bar{\psi}]^{\mathrm{T}}$ be the mean source
DoA, and define the set
$\mathcal{X}=\\{\boldsymbol{x}:|\mathrm{AF}(\theta,\psi)|\geq G_{0}\\}$. Then

$\displaystyle P_{\text{out}}=$ $\displaystyle
1-\underset{\boldsymbol{x}\in\mathcal{X}}{\iint}\frac{1}{2\pi\sqrt{|\Sigma_{\theta,\psi}|}}e^{-\frac{1}{2}(\boldsymbol{x}-\boldsymbol{\mu})^{\mathrm{T}}\Sigma_{\theta,\psi}^{-1}(\boldsymbol{x}-\boldsymbol{\mu})}~{}d\boldsymbol{x}$
$\displaystyle=$ $\displaystyle
1-\underset{\boldsymbol{x}\in\mathcal{X}}{\iint}\mathcal{N}(\boldsymbol{x};\boldsymbol{\mu},\Sigma_{\theta,\psi})~{}d\boldsymbol{x}.$
(6)

The integral limits, defined by $|\mathrm{AF}(\theta,\psi)|\geq G_{0}$, may be
incorporated via an indicator function to simplify the integral in (6). Since
the focus of this paper is to analyze the effects of source DoA estimation
inaccuracy on the outage probability, the required gain $G_{0}$ for a given
communication distance is set to be a constant. The indicator function is
defined as

$\mathrm{I}_{\mathrm{AF}}(\theta,\psi)=\left\\{\begin{array}[]{lr}1,&\text{if
}|\mathrm{AF}(\theta,\psi)|\in\mathcal{X}\\\ 0,&\text{if
}|\mathrm{AF}(\theta,\psi)|\notin\mathcal{X}\end{array}\right..$ (7)

The integral effectively spans regions where the beam gain surpasses the
threshold. Factors like changing distance, source DoA estimation error
covariance matrix, SNR threshold, and the receive beam array factor impact the
integral and, by extension, the outage probability.

##  IV Beamwidth Adaptation Approach

###  IV-A General Setup and Problem Formulation

The outage probability, once characterized by misalignment, can be managed by
modifying the beam shape, which depends on the array factor and, consequently,
on individual array weights. By altering these weights, the outage probability
can be minimized. For a total of $N$ antennas with element weights
$\boldsymbol{w}_{n}\in\mathbb{C}^{N}$, the array factor is

$\mathrm{AF}(\theta,\psi)=\sum_{n=0}^{N-1}\boldsymbol{w}_{n}e^{-j\boldsymbol{k}^{\mathrm{T}}\boldsymbol{d}_{n}},$
(8)

where $\boldsymbol{d}_{n}$ defines the element 3D location of array element n,
and
$\boldsymbol{k}=\frac{2\pi}{\lambda}[\sin\theta\cos\psi,\sin\theta\sin\psi,\cos\theta]^{\mathrm{T}}$
represents the wave vector with $\lambda$ as the carrier wavelength.
Traditionally, the beam direction can be controlled by adjusting the phase of
each element with the element weights $\boldsymbol{w}_{n}$.

Optimizing the array weights maximizes the integral in the outage probability
definition. The optimal array weights $\boldsymbol{W}_{n}$ for the
optimization of (6) through antenna activation becomes

$\displaystyle\boldsymbol{W}_{n}=\arg\max_{\boldsymbol{w}_{n}}\int_{0}^{2\pi}\int_{-\frac{\pi}{2}}^{\frac{\pi}{2}}\mathcal{N}\left(\boldsymbol{x};\boldsymbol{\mu},\Sigma_{\theta,\psi}\right)\mathrm{I}_{\mathrm{{AF}}}(\theta,\psi)~{}d\boldsymbol{x}.$
(9)

One practical approach to minimizing outage probability is activating a subset
of antennas after steering the beam to the target to adapt beamwidth. Antenna
deactivation is a straightforward method, particularly appealing from the user
side due to its simplicity and energy-saving potential.

While other beamwidth adaptation techniques might need complex antenna
structures, specific antenna couplings, and/or intricate codebooks, antenna
deactivation offers a simple solution by avoiding the complexities of
individually adjusting gains and phases. Given its benefits, this study adopts
antenna deactivation to examine the nuances of beamwidth adaptation for XR
HMDs.

###  IV-B Confidence Interval Estimation

The covariance matrix $\Sigma_{\theta,\psi}$, derived from angular errors in
elevation and azimuth domains, facilitates the generation of the confidence
interval. The eigendecomposition of this matrix characterizes its confidence
interval orientation, width, and height, which is tied to specific
probabilistic coverage requirements. Using the covariance matrix, obtained
through sensor estimations and UKF filtering, the confidence interval size and
orientation are characterized by its eigenvalues and corresponding
eigenvectors.

The covariance matrix, acting as a linear operator on white noise, outputs
colored noise, resulting in an ellipse influenced by the matrix’s
eigenvectors. For angular estimations, E, when projected onto a unit norm
vector $\hat{a}$, the variance of this projected data, $\sigma_{a}^{2}$, is
defined as $\hat{a}^{\mathrm{T}}\Sigma_{\theta,\psi}\hat{a}$

To pinpoint $\hat{a}$, $\sigma_{a}^{2}$ is optimized towards the direction
with the greatest variance. This process is like maximizing the Rayleigh
Quotient associated with the dominant eigenvector of $\Sigma$, revealing the
confidence interval’s orientation. The subsequent leading eigenvalues further
detail the distribution’s spread and orientation.

###  IV-C Outage Probability Optimization

For optimal coverage, the beam should span the entire uncertainty region which
has an elliptical confidence interval due to the joint Gaussian behavior of
angular estimations. By orienting the beam with the confidence interval,
outage probability is optimized.

To prove this, the beam pattern is approximated for uniform gain within its 3
dB beamwidth, and zero elsewhere. This mirrors the gain threshold and path
loss considerations in the outage probability’s integration region. The
effective array gain distributed over an area $\mathcal{S}$ can be
approximated as

$|\mathrm{AF}(\theta,\psi)|\approx\frac{\pi^{2}}{|\mathcal{S}|}.$ (10)

This approximation applies to the ideal scenario, where the beam gain is
equally distributed along the region of interest to maximize the angular
coverage for a fixed power.

Given the elliptical nature of the confidence interval and the Gaussian PDF’s
exponential decay, an offset in the beam orientation from the interval center
leads to reduced probability coverage. Differing beam shapes, especially those
not aligned with the confidence interval, yield increased outage
probabilities. Then, a locally optimal beam shape aligns closely with the
confidence interval, as depicted in Fig. 2.

Figure 2: Beam perfectly overlapping with the confidence interval (left), beam
eccentricity is the same as the confidence interval, but orientation is not
aligned (middle), beam eccentricity is not the same as the confidence interval
(right).

Similarly, when the beam shape and thus the coverage area is different, the
probability coverage will be smaller, increasing the outage probability. As a
result, the optimum beam shape should be the one overlapping the confidence
interval. When the beam is adapted by antenna deactivation, higher power
requirement translates to more antenna activation. Optimizing coverage,
therefore, involves aligning the beam with the confidence interval and
adjusting the number of active antennas.

Let
$\mathrm{I}_{\mathrm{AF}}\left(\phi_{\mathrm{\Sigma}},N_{\mathrm{A}},\theta,\psi\right)$
denote an indicator function of the beam shape aligned with the confidence
interval, where $\phi_{\mathrm{\Sigma}}$ is the confidence interval
orientation angle with respect to its azimuth estimation axis and
$N_{\mathrm{A}}$ is the number of active antennas. Since optimizing the outage
probability involves aligning the beam with the confidence interval and
changing the number of active antennas to cover the most area possible,
optimization in (9) can be rewritten with
$\mathrm{I}_{\mathrm{AF}}\left(\phi_{\mathrm{\Sigma}},N_{\mathrm{A}},\theta,\psi\right)$
as

$\displaystyle\boldsymbol{W}_{n}=\arg\max_{\boldsymbol{w}_{n}}\left[\int_{0}^{2\pi}\int_{-\frac{\pi}{2}}^{\frac{\pi}{2}}\mathrm{I}_{\mathrm{AF}}(\phi_{\mathrm{\Sigma}},N_{\mathrm{A}},\theta,\psi)\,d\theta
d\psi\right].$ (11)

Once the beam is steered and aligned with the confidence interval, optimizing
the outage probability becomes selecting the appropriate number of active
antennas.

###  IV-D Generalized Beamwidth Adaptation

Conventionally, the beamwidth is adapted along only azimuth and elevation
directions, however, utilizing the 2D antenna array structure of the XR HMD
helps create a beam that aligns with the shape of the confidence interval by
activating a subset of the antennas. To produce a beam matching the confidence
interval’s elliptical shape, the relationship between the 3 dB beamwidth and
the antenna count is vital and given in (12). Exploiting the covariance
matrix’s eigenvectors and eigenvalues facilitates the generation of the
desired beam shape. The required beamwidth within the 2D azimuth-elevation
domain can then be mapped to the necessary antenna count in specific
directions:

$\displaystyle b_{\beta}=\frac{0.886\lambda}{Nd\cos\beta}.$ (12)

Here, $b_{\beta}$ is the beamwidth, $N$ is the number of antennas, $d$ is the
antenna spacing, and $\beta$ is the beam pointing direction. This equation
converts the confidence interval’s widths defined by its covariance matrix
eigenvalues $a$ and $b$ to the required antenna numbers. The confidence
interval defines the required beamwidths and these beamwidths can be converted
into required antenna counts $N_{\mathrm{a}}$ and $M_{\mathrm{a}}$ along their
eigenvectors, with the inequalities representing antenna count boundaries:

$\displaystyle 2M_{\mathrm{a}}$
$\displaystyle\leq\frac{0.886}{\left(\frac{b\pi}{180}\right)\cos\beta}$ (13)
$\displaystyle 2N_{\mathrm{a}}$
$\displaystyle\leq\frac{0.886}{\left(\frac{a\pi}{180}\right)\cos\beta}.$ (14)

The identified beamwidths across the confidence interval’s dimensions
construct an ellipse in the number of antennas domain. A specific antenna
subset guaranteeing a minimum of 3 dB beamwidth aligned with the confidence
interval is chosen. The antenna ellipse is defined as

$\frac{\left(m\cos\left(\phi_{\mathrm{\Sigma}}\right)+n\sin\left(\phi_{\mathrm{\Sigma}}\right)\right)^{2}}{M_{\mathrm{a}}^{2}}+\frac{\left(m\sin\left(\phi_{\mathrm{\Sigma}}\right)-n\cos\left(\phi_{\mathrm{\Sigma}}\right)\right)^{2}}{N_{\mathrm{a}}^{2}}\leq
1.$ (15)

In (15), $m$ and $n$ denote the antenna indices and $\phi_{\mathrm{\Sigma}}$
signifies the confidence interval’s orientation relative to the azimuth axis.
The ellipse in (15) is placed at the center of the 2D antenna array and
antennas with indices inside the ellipse are activated to achieve the beam.
Fig. 3 depicts the transformation of a sample confidence interval into a beam
aligned with that confidence interval.

Figure 3: Transformation of confidence interval (left) to the resultant beam
shape (right).

##  V Results and Discussion

The system was simulated in MATLAB for varying link distances and for the 3GPP
antenna pattern with a 16 by 16 receive antenna at the XR HMD, comparing the
proposed algorithm with the beamwidth adaptation with antenna deactivation
without utilizing the estimation correlation. The 3GPP antenna pattern
features 3 dB beamwidths of 65 degrees in both vertical and horizontal planes
and a peak gain of 8 dBi at the main lobe center. Estimation correlation was
assumed to study the impact of the covariance matrix on outage probability
improvement. Orientation standard deviation was set at 2 degrees for both
azimuth and elevation, with a correlation metric of 0.5. The SNR threshold was
established at 3 dB, necessitating varied gain thresholds for different
communication distances. The communication parameters are given in Table I.

TABLE I: Communication Parameters for Simulations Parameter | Value
---|---
Transmit Power (dBm/MHz) | 3
Bandwidth (MHz) | 400
Carrier Frequency (GHz) | 28
Number of BS Antennas | 256
Noise PSD (dBm/Hz) | -174
Path Loss Exponent ($\alpha$) | 2.5
SNR Threshold (dB) | 3

###  V-A Optimal Number of Active Antennas

The number of antennas, subject to an application-specific outage probability
threshold and latency constraint, has both upper and lower bounds. The minimum
number ensures the required gain, while the maximum is constrained by the
inverse relationship between antenna count and beamwidth; as the number of
antennas increase, beamwidth decreases, subsequently increasing the outage
probability. The allowable number of antennas for a given outage probability
threshold is given as

$\displaystyle N_{\text{min}}\leq N\leq N_{\text{max}}.$ (16)

Figure 4: Number of allowable antennas versus distance for the 3GPP antenna
model.

Fig. 4 depicts the range of antennas permissible for a given outage
probability. Beyond a certain link distance, the maximum and minimum antenna
counts align, indicating the system’s effort to maintain the desired outage
probability, even when it may no longer meet the outage threshold. The
proposed adaptation algorithm extends coverage distance up to 8% by utilizing
the covariance matrix rather than simply adapting along azimuth and elevation.
The 8% increment in coverage distance translates into approximately 16%
coverage area increase. Variations in the correlation metric could modify this
improvement, with gains approaching conventional performance in the absence of
correlation.

The proposed algorithm reduces antenna usage by up to 18% at certain distances
to maintain the same outage probability, leading to equivalent power savings.
Fig. 5 shows that the optimal number of antennas for minimizing outage
probability lies between the maximum and minimum limits specified in Fig. 4,
according to a set outage probability threshold.

Figure 5: Optimal number of antennas versus distance for the 3GPP antenna
model.

###  V-B Changing Correlation’s Effect on Outage Probability

As the correlation metric increases, the generalized beamwidth adaptation uses
the correlation information between the azimuth and elevation to its
advantage. Therefore, it is expected and can be observed in Fig. 6 that the
generalized beamwidth adaptation performs better than the azimuth-elevation
only beamwidth adaptation as the correlation metric increases.

On the other hand, increasing the correlation metric decreases the adapted
beam’s coverage over the estimation confidence interval achieved by the
conventional beamwidth adaptation technique and increases the outage
probability, which can be observed in Fig. 6.

Figure 6: Outage probability versus correlation metric of the covariance
matrix for the 3GPP antenna model.

###  V-C Confidence Interval Orientation’s Effect on Outage Probability

When the confidence interval orientation is 0 degrees, the generalized
beamwidth adaptation is essentially the same as the conventional beamwidth
adaptation where only the azimuth and elevation beamwidths are adapted. As the
confidence interval orientation gets closer to 45 degrees, the generalized
beamwidth adaptation performs better by utilizing the covariance of the
estimations. Hence, the generalized beamwidth adaptation covers the
conventional beamwidth adaptation and supplies extra gain in outage
probability. When the confidence interval is oriented along 45 degrees with
respect to the horizontal axis azimuth, then the generalized beamwidth
adaptation was observed to supply the most gain in outage probability, as
depicted in Fig. 7.

Figure 7: Outage probability versus confidence interval orientation (in
degrees) for the 3GPP antenna model.

##  VI Conclusions

In this paper, a 6DoF XR HMD communication link has been modeled to analyze
the outage probability under different scenarios and link distances. Utilizing
sensor covariance data enabled more effective antenna use, enhancing coverage
distance by 8% and area by over 15%, while maintaining consistent outage
probabilities and therefore, achievable latency. This increase in coverage
also translates into energy efficiency, achieving comparable outage
probabilities with reduced power consumption. The proposed algorithm was
observed to achieve up to 18% power efficiency improvement as a result of
leveraging the DoA estimation correlations. A power savings of 18% can
substantially increase the battery life of XR HMD. Moreover, system parameter
variations that affect received power levels could alter the improvement
magnitude and distance at which improvements are realized.

The minimum number of allowable antennas satisfying the outage probability
constraint has been observed to increase with distance to supply the required
gain. On the other hand, the maximum allowable number of antennas needed to
meet the outage probability threshold decreases with distance to improve the
beamwidth. These two limitations meet at a critical point after which the
outage probability threshold cannot be satisfied. The proposed beamwidth
adaptation has pushed this critical point forward so that the coverage
distance of the system has been increased. As an extension, one can define a
latency metric for XR and analyze how the beamwidth adaptation can affect the
upper-layer communication delays.

## References

  * [1] P. Hande, P. Tinnakornsrisuphap, J. Damnjanovic, H. Xu, M. Mondet, H. Y. Lee, and I. Sakhnini, “Extended reality over 5g–standards evolution,” _IEEE Journal on Selected Areas in Communications_ , 2023.
  * [2] I. A. Hemadeh, K. Satyanarayana, M. El-Hajjar, and L. Hanzo, “Millimeter-wave communications: Physical channel models, design considerations, antenna constructions, and link-budget,” _IEEE Communications Surveys & Tutorials_, vol. 20, no. 2, pp. 870–913, 2017.
  * [3] V. Va, X. Zhang, and R. W. Heath, “Beam switching for millimeter wave communication to support high speed trains,” in _2015 IEEE 82nd Vehicular Technology Conference (VTC2015-Fall)_ , 2015.
  * [4] H. Zhu, X. Zhang, H. Qian, and X. Luo, “Joint adaptation of rate and beamwidth for large-scale antenna systems,” in _2019 IEEE Global Communications Conference (GLOBECOM)_ , 2019.
  * [5] V. Va, T. Shimizu, G. Bansal, and R. W. Heath, “Beam design for beam switching based millimeter wave vehicle-to-infrastructure communications,” in _2016 IEEE International Conference on Communications (ICC)_ , 2016.
  * [6] J. Struye, F. Lemic, and J. Famaey, “Covrage: Millimeter-wave beamforming for mobile interactive virtual reality,” _IEEE Transactions on Wireless Communications_ , vol. 22, no. 7, pp. 4828–4842, 2023.
  * [7] L. Yang and W. Zhang, “Beam tracking and optimization for UAV communications,” _IEEE Transactions on Wireless Communications_ , vol. 18, no. 11, pp. 5367–5379, 2019.
  * [8] H. Chung, J. Kang, H. Kim, Y. M. Park, and S. Kim, “Adaptive beamwidth control for mmwave beam tracking,” _IEEE Communications Letters_ , vol. 25, no. 1, pp. 137–141, 2021.
  * [9] Y. Karaçora, C. Chaccour, A. Sezgin, and W. Saad, “Event-based beam tracking with dynamic beamwidth adaptation in terahertz (thz) communications,” _IEEE Transactions on Communications_ , pp. 1–1, 2023.
  * [10] Z. Peng and W. Li, “Two-stage 3d codebook design and fast beam search algorithm for millimeter-wave massive MIMO systems,” _Electronics_ , vol. 9, no. 2, p. 302, 2020.
  * [11] Z. Yang, P. H. Pathak, Y. Zeng, and P. Mohapatra, “Sensor-assisted codebook-based beamforming for mobility management in 60 Ghz WLANS,” in _2015 IEEE 12th International Conference on Mobile Ad Hoc and Sensor Systems_ , 2015.
  * [12] D. Tagliaferri, M. Brambilla, M. Nicoli, and U. Spagnolini, “Sensor-aided beamwidth and power control for next generation vehicular communications,” _IEEE Access_ , vol. 9, pp. 56 301–56 317, 2021.
  * [13] K. L. Bell, Y. Ephraim, and H. L. Van Trees, “A bayesian approach to robust adaptive beamforming,” _IEEE Transactions on Signal Processing_ , vol. 48, no. 2, pp. 386–398, 2000.
  * [14] S. A. Vorobyov, A. B. Gershman, and Z.-Q. Luo, “Robust adaptive beamforming using worst-case performance optimization: A solution to the signal mismatch problem,” _IEEE transactions on signal processing_ , vol. 51, no. 2, pp. 313–324, 2003.
  * [15] M. Rebato, L. Resteghini, C. Mazzucco, and M. Zorzi, “Study of realistic antenna patterns in 5G mmwave cellular scenarios,” in _2018 IEEE International Conference on Communications (ICC)_ , 2018.