# Semantics-Aware Next-best-view Planning for Efficient Search and Detection
of Task-relevant Plant Parts

Akshay K. Burusa<EMAIL_ADDRESS>Joost Scholten  Xin Wang  David
Rapado Rincon  Eldert J. van Henten  Gert Kootstra

###### Abstract

To automate harvesting and de-leafing of tomato plants using robots, it is
important to search and detect the task-relevant plant parts, namely tomatoes,
peduncles, and petioles. This is challenging due to high levels of occlusion
present in tomato greenhouses. Active vision is a promising approach to
viewpoint planning, which helps robots to deliberately plan camera viewpoints
to overcome occlusion and improve perception accuracy. However, current
active-vision algorithms cannot differentiate between relevant and irrelevant
plant parts and spend time on perceiving irrelevant plant parts, making them
inefficient for targeted perception. We propose a semantics-aware active-
vision strategy that uses semantic information to identify the relevant plant
parts and prioritises them during view planning using an attention mechanism.
We evaluated our strategy on the task of searching and detecting the relevant
plant parts using simulation and real-world experiments. In simulation, using
3D models of tomato plants with varying growth stages and structural
complexity, our semantics-aware strategy could search and detect $81.8\%$ out
of the total relevant plant parts using nine viewpoints. It was significantly
faster and detected more plant parts than predefined, random, and volumetric
active-vision strategies that do not use semantic information. Our strategy
was also robust to uncertainty in plant and plant-part position, plant
complexity, and different viewpoint-sampling strategies. Further, in real-
world experiments, our semantics-aware strategy could search and detect
$82.7\%$ of the relevant plant parts using seven viewpoints, under complex
greenhouse conditions with natural variation and occlusion, natural
illumination, sensor noise, and uncertainty in camera poses. Our results
clearly indicate the advantage of using semantics-aware active vision for
targeted perception of plant parts and its applicability in real-world setups.
We believe that it can significantly improve the speed and robustness of
automated harvesting and de-leafing in tomato crop production.

###### keywords:

Active vision , Next-best-view planning , Semantics , Attention , Object
detection , Greenhouse robotics.

[inst1]organization=Agricultural Biosystems Engineering, Wageningen University
& Research, city=Wageningen, country=The Netherlands

##  1 Introduction

Tomatoes are among the most consumed vegetables globally (Beed et al., 2021),
with demand growing rapidly due to increasing population. However, the current
rate of tomato production cannot satisfy the future demand and efforts to
improve production are limited by labour shortages. The current labour force
is aging, as younger generations are not interested in the physically
demanding and monotonous work in the tomato industry (Rigg et al., 2020). To
improve production and meet the rising demand, there is a strong need for
automation, where robots can fill-in for humans and carry out the labour-
intensive tasks in tomato greenhouses, such as selective harvesting and de-
leafing (Van Henten, 2004).

Figure 1: Efficiently searching and finding the relevant plant-parts, such as
tomatoes, peduncles, and petioles, is essential for automated harvesting and
de-leafing in tomato greenhouses.

To automate harvesting and de-leafing of tomato plants, robots must accurately
perceive the relevant plant parts. For harvesting, it is relevant to perceive
ripe tomatoes and peduncles (part that connects tomatoes to the stem).
Similarly, for de-leafing, it is relevant to perceive the petioles (part that
connects leafs to the stem). These plant parts are shown in Figure 1. With
advancements in computer vision, the 2D detection of tomatoes, peduncles, and
petioles can be achieved using cameras and artificial neural networks (Koirala
et al., 2019). However, to estimate the cutting point for harvesting and de-
leafing, detection on a 2D image alone is not sufficient. We need to gather
more information about the plant parts and estimate their cutting point in 3D.
Hence, searching and sufficiently perceiving all the relevant plant parts of a
tomato plant remains to be a challenging problem, primarily due to occlusion
by other parts (Kootstra et al., 2021), leading to incomplete or ambiguous
information.

Many approaches to occlusion handling are proposed in robotics literature and
applied in agro-food environments. With advancements in deep learning, a
popular approach is to train a neural network to detect objects despite
occlusions or complete the shape of the occluded parts. However, such
approaches rely on training data and might not extend well to situations that
are out-of-distribution. Also, these cannot be applied to greenhouse tasks
that require careful inspection of the plant parts, such as disease detection.
In such situations, the best way to guarantee certainty of perception is to
physically move the camera and directly observe the occluded plant part.
Hence, one way to overcome occlusion is to explore the plant using multiple
viewpoints, as acquiring multiple measurements can help fill any missing
information or resolve ambiguities (Van Henten et al., 2002; Hemming et al.,
2014). However, these multiple viewpoints must be chosen efficiently. In
current practice, passive multi-view approaches are used, in which a camera is
moved in predefined patterns (e.g. zig-zag motions). Such approaches often
suffer from sub-optimal view selection, i.e., they are either time consuming
due to too many viewpoints or fail to detect all plant parts due to too few
viewpoints.

Active vision (Isler et al., 2016; Bircher et al., 2016; Daudelin and
Campbell, 2017; Schmid et al., 2020; Zeng et al., 2020) offers a better
approach for viewpoint planning, in which the camera viewpoints are
deliberately planned to maximise novel information about the plant based on
the information perceived so far. By maximising novel information, active
vision prioritises viewpoints that observe new or occluded parts of the plant.
Hence, it can explore the plant faster and more efficiently compared to
passive multi-view approaches (Gibbs et al., 2019; Zaenker et al., 2021;
Marangoz et al., 2022; Burusa et al., 2022). However, most active-vision
algorithms are designed to explore unknown space using occupancy information,
which is ideal for exploring the whole plant. The occupancy information cannot
distinguish between different plant parts and hence not suitable for targeted
perception of specific objects-of-interest (OOIs), such as the relevant plant
parts. To address this problem, active-vision algorithms must be made aware of
the OOIs. Adding semantic information about objects, such as class labels, has
been shown to improve targeted perception (Kay et al., 2021) and scene
understanding (Zheng et al., 2019) in other applications. This has not been
studied in complex agro-food environments. Zaenker et al. (2021) and Marangoz
et al. (2022) made a binary distinction between fruits and the rest of the
plant, but the semantic information was not directly used for planning
viewpoints. Also, these works were only tested in simulation. Hence, their
performance under real-world conditions is unknown.

In this paper, we proposed a semantics-aware active-vision strategy that
considers the class labels and confidence scores of the OOIs when planning
viewpoints. We hypothesised that using such semantic information will lead to
faster perception of all the relevant plant parts. We tested this hypothesis
using simulation and real-world experiments. In simulation, we used 3D mesh
models of tomato plants with varying growth stages and structural complexity,
and uncertainty in plant and OOI positions. With real-world experiments, we
introduced the complexities of the real world, such as natural variation and
occlusion in tomato plants, natural illumination, sensor noise, and
uncertainty in camera poses. Using these setups, we performed systematic and
repeatable experiments to study the advantage of adding semantic information
to active vision. We compared the performance of our semantics-aware active
vision method with an active vision strategy without semantic information,
predefined strategies, and a random strategy. Also, to the best of our
knowledge, our work was the first to test the performance of a semantics-aware
active vision approach using a robotic setup in a real-world greenhouse
environment.

##  2 Problem description

Given a tomato plant, placed within a bounded 3D space
$V\subset\mathbb{R}^{3}$, the task is to explore the plant using a robot with
an RGB-D camera and detect all the OOIs. The robot does not know the exact
position of the plant, except that the plant is somewhere in front of it.
Hence, the position of the plant base $p^{\text{pb}}\in V$ contains some
uncertainty. The robot starts from an initial camera viewpoint $\xi_{0}$ and
needs to find a set of sequential viewpoints that will lead to the detection
of the OOIs. Here, a camera viewpoint $\xi$ can have $6$ degrees-of-freedom
(DoF), with a 3D position and 3D orientation. Since the structure of the plant
is not known _a priori_ , the robot needs to simultaneously explore the plant
to find new OOIs and improve the perception of previously detected OOIs. We
formulate this task of visual search for OOIs as an online next-best-view
(NBV) planning problem within the active-vision paradigm. The goal of NBV
planning is to determine the next-best camera viewpoint $\xi_{\text{best}}$
that will help perceive novel information about the OOIs. We assume that by
maximising the novel information about the OOIs per viewpoint, we can find the
shortest sequence of viewpoints that can detect most of the OOIs with high
accuracy. The problem is terminated when all the OOIs are detected or when a
maximum number of viewing actions $a_{\text{max}}$ is reached.

##  3 Semantics-aware next-best-view planning

Figure 2: The pipeline of the Semantic NBV planning algorithm consists of
three major modules – (i) a sensing module to detect OOIs (Sec. 3.1), (ii) a
3D scene representation module to analyse and track the OOIs over multiple
viewpoints (Sec. 3.2 to 3.5), and (iii) a view-planning module to determine
the next-best camera viewpoint (Sec. 3.6, 3.7).

We proposed a semantics-aware NBV planning algorithm to address the problem
described above. The main loop of the planner is illustrated in Figure 2. The
algorithm consisted of three major modules – (i) a sensing module to detect
the OOIs, (ii) a 3D scene representation module to merge information about the
OOIs across multiple viewpoints, and (iii) a view-planning module to determine
the next-best camera viewpoint.

###  3.1 Sensing

The first step of the algorithm was to accurately detect the OOIs. For this,
we used a convolutional neural network, Mask R-CNN (He et al., 2017). For
simulation experiments, we fine-tuned the network to detect tomatoes,
peduncles, and petioles using a custom dataset of $90$ RGB images, of
dimension $960\times 540$ pixels, collected from the simulated environment.
The dataset was split into $72$ images for training and $18$ images for
validation. For real-world experiments, we focused only on the detection of
peduncles and petioles, as these parts are crucial for harvesting and de-
leafing and are more challenging to perceive than tomatoes due to occlusion.
We fine-tuned Mask R-CNN to detect peduncles and petioles using a custom
dataset of $746$ RGB images collected in a tomato greenhouse on seven
different occasions. A total of $658$ peduncles and $1478$ petioles were
annotated. $526$ images were used for training and $220$ images for
validation. The object detection module took an RGB image as input and
generated a segmented image with masks that separated the OOIs from the
background (Fig. 3(a)).

(a) (b) (c) Figure 3: Generating the semantic OctoMap had the following steps:
(a) The objects-of-interest (OOIs), were detected using Mask R-CNN to form a
segmented image (Sec. 3.1). (b) The segmented image was combined with the
corresponding depth image to produce a semantic point cloud (Sec. 3.2). (c)
The semantic point cloud was then inserted into the OctoMap, where each voxel
stores the class label and confidence score, along with the occupancy
probability (Sec. 3.3). Also, a clustering operation was performed on the
semantic OctoMap to identify the individual OOIs at an object level in 3D
space (Sec. 3.4).

$\xrightarrow{}$ $\xrightarrow{}$

###  3.2 Semantic Point Cloud

The segmented image was combined with the corresponding depth image to
generate a semantic point cloud (Fig. 3(b)). Each point $i$ in the point cloud
contained the spatial position of the OOIs in Cartesian space and also the
semantic class label $c_{s}(i)$ and confidence score $p_{s}(i)$ about what
object the point belonged to. The class label and confidence score were
defined as,

$\displaystyle\begin{split}c_{s}(i)&\in\\{-1=\text{background},0=\text{peduncle},\\\
&\quad\quad\;1=\text{petiole},2=\text{tomato}\\},\end{split}$ (1)
$\displaystyle p_{s}(i)$ $\displaystyle\in[0,1].$ (2)

The confidence score $p_{s}(i)$ was interpreted as the probability of the
point $i$ belonging to the detected class label $c_{s}(i)$. Hence,
$p_{s}(i)=0$ meant that the point certainly did not belong to $c_{s}(i)$,
whereas $p_{s}(i)=1$ meant that the point certainly belonged to $c_{s}(i)$.
The values in between implied uncertainty regarding the class label, with
maximum uncertainty at $p_{s}(i)=0.5$. If a point did not contain any semantic
information, a default value of $c_{s}(i)=-1$ and $p_{s}(i)=0.5$ were
assigned, which meant that the point belonged to the background and had
maximum uncertainty regarding its class label.

###  3.3 Semantic OctoMap

The semantic point cloud captured the spatial and semantic information of the
plant only in the current viewpoint. To keep track of the detected OOIs across
multiple viewpoints, we needed a 3D scene representation of the plant which
could be updated over time. So, we represented the observed plant and its
surrounding scene as a probabilistic occupancy map called an OctoMap (Hornung
et al., 2013). The OctoMap divided the 3D space into grid cells called voxels,
which were efficiently stored as hierarchical nodes in an Octree. We used a
resolution of $0.003$m for a single voxel. Each voxel was marked as empty or
occupied based on whether the voxel was observed by the camera. Voxels with an
occupancy probability of zero ($p_{o}=0$) were considered _empty_ and those
with occupancy probability of one ($p_{o}=1$) were considered _occupied_. A
value between $0$ and $1$ implied uncertainty regarding the occupancy, with a
maximum uncertainty at $p_{o}=0.5$. The regions that were not observed by the
camera yet did not contain any occupancy information and were considered
_unknown_ with maximum uncertainty ($p_{o}=0.5$). Such information about the
occupancy probabilities of the voxels is commonly referred to as _volumetric
information_. The volumetric information is useful in identifying the
uncertain and unexplored regions of the plant, and can help guide an NBV
planner towards them. However, it cannot distinguish between different plant
parts and hence not suitable for targeted perception of the OOIs.

To be able to explore the OOIs, we extended the OctoMap to include _semantic
information_ in addition to the volumetric information (Fig. 3(c)), similar to
Xuan and David (2018). That is, each voxel $x$ in the OctoMap consisted of a
semantic class label $c_{s}(x)$ and a confidence score $p_{s}(x)$, which were
obtained and inserted from the semantic point cloud in the previous step (Sec.
3.2). The class labels identified the OOIs and distinguished them from the
rest of the plant, while the confidence scores indicated how certain we were
about the OOI identities. Hence, the semantic OctoMap could help guide an NBV
planner to search for OOIs and further explore them until a degree of
certainty was achieved.

The semantic information in the OctoMap was updated iteratively when the
camera was moved to a new viewpoint and new information was observed. If the
voxel had no prior semantic information, the observed values were directly
assigned to it. Otherwise, the semantic information was updated using the
method of max-fusion from Xuan and David (2018). The new class label and
confidence score from the semantic point cloud were merged with the previous
values in the voxel as follows: (i) if the new class label was the same as the
previous, then the class label remained unchanged and the confidence score was
computed as the average of the new and previous scores; or (ii) if the new
class label was different from the previous, the semantic label with the
higher confidence score was chosen and that confidence score was reduced by
$10\%$ as a penalty for label mismatch. This small penalty increased the
uncertainty of the voxel, which in turn could guide the viewpoint planner to
take another look at the voxel and resolve the label mismatch. The max-fusion
method was used as it was simple, fast, and memory-efficient as it only stored
one class label and confidence score per voxel. The pseudo code for the max-
fusion method is shown in Algorithm 1.

Algorithm 1 Max fusion for merging semantic information

$\text{sem}_{1},\text{sem}_{2}$ $\triangleright$ Previous and new semantics

$\text{sem}_{f}$ $\triangleright$ Updated semantics

$c_{s},p_{s}$ $\triangleright$ Class label and confidence score

function Max Fusion($\text{sem}_{1},\text{sem}_{2}$)

if $\text{sem}_{1}.c_{s}==\text{sem}_{2}.c_{s}$ then

$\text{sem}_{f}.c_{s}\leftarrow\text{sem}_{2}.c_{s}$

$\text{sem}_{f}.p_{s}\leftarrow(\text{sem}_{1}.p_{s}+\text{sem}_{2}.p_{s})/2$

else

$\text{sem}_{f}.c_{s}\leftarrow\arg\max_{c_{s}}(\text{sem}_{1}.p_{s},\text{sem}_{2}.p_{s})$

$\text{sem}_{f}.p_{s}\leftarrow
0.9\times\max(\text{sem}_{1}.p_{s},\text{sem}_{2}.p_{s})$

end if

return $\text{sem}_{f}$

end function

###  3.4 Clustering

The semantic OctoMap represented the OOIs in a discretised form at the voxel
level. However, an object-level representation was required to locate the OOIs
in 3D space. We achieved this by clustering the voxels that belonged to the
same object. We used an algorithm proposed by Ankerst et al. (1999) called
_Ordering Points To Identify the Clustering Structure_ (OPTICS). Each cluster
was identified by three criteria – (i) all voxels in the cluster must belong
to an OOI, (ii) the total number of voxels in a cluster must be at least $20$,
which was empirically chosen based on the clustering performance, and (iii)
the maximum distance between any two voxels within the cluster is $0.03$m for
simulation experiments and $0.06$m for real-world experiments. These distances
were based on the size of the OOIs. With this clustering step, we got an
object-level representation of the OOIs. The number of detected OOIs was given
by the number of clusters, and the positions of the OOIs $p^{\text{ooi}}\in V$
were given by the center of the clusters. The clustering was repeated at every
iteration of the Semantic NBV planning pipeline. Hence, as the semantic
OctoMap got updated over multiple viewpoints (Sec. 3.3), the OOIs were also
updated.

###  3.5 Attention mechanism

An object-level representation of OOIs allowed the NBV planner to attend to
regions where the OOIs were located and improve their accuracy. So, similar to
our previous work (Burusa et al., 2022), we used an attention mechanism to
guide the NBV planner to explore and perceive the OOIs. We marked the
attention regions by defining bounding boxes around the estimated position of
the main stem and the detected positions of the OOIs, as shown in Figure 4.
The main-stem bounding box guided the planner to explore and identify new OOIs
and the OOI bounding boxes guided the planner to perceive the previously
detected OOIs more accurately. While Zaenker et al. (2021) had to define two
separate strategies to obtain these behaviours, we were able to obtain them
with a single strategy. The bounding boxes were defined using the center
position of the box, the dimensions of the sides, and the orientation with
respect to the global frame. The regions of attention for the main stem and
the OOIs were defined as follows:

Figure 4: An attention mechanism was used to guide the Semantic NBV planner. A
region-of-interest was defined along the estimated position of the main stem
(blue bounding box) to guide the planner to explore new OOIs. Multiple
regions-of-interest were defined around the detected OOIs (red bounding boxes)
to guide the planner to perceive them more accurately. The axes represent the
3D positions of the OOIs.

####  3.5.1 Attention along the estimated main stem

The main stem was not detected, but its position was estimated based on the
volumetric and semantic information gathered so far, as explained below.
Bounding boxes were defined around this estimated position of the main stem.
The position estimation of the main stem depended on how much information we
had about the plant. In particular, there were four different cases. (i) When
there was no information about the plant, attention along the main stem was
not defined. In this case, the NBV planner planned viewpoints without an
attention mechanism. (ii) Once the plant was viewed at least from one
viewpoint, the position of the main-stem center was estimated as the center of
the visible region of the plant. An axis-aligned bounding box of height $0.7$m
in simulation and $1.2$m in the real-world experiments was defined at this
estimated position to guide the NBV planner to explore the plant further.
Here, axis-aligned bounding box means that the edges of the box were parallel
to the global coordinate frame. (iii) Once some tomato clusters were detected,
the main-stem center was estimated as the center of the tomato clusters
instead, as this was a better estimate of the stem location than the center of
all visible regions of the plant. (iv) Finally, when at least one petiole or
peduncle cluster was detected, a set of oriented bounding boxes were defined
between consecutive petioles or peduncles, as shown in Figure 4. Two boxes at
the top and bottom of the bounded space $V$ (which is expected to contain the
plant) were also included, to ensure that the NBV planner also explored the
top and bottom regions of the stem. Hence, the attention on the main stem
improved iteratively as more information was obtained from multiple
viewpoints. The breadth and width of the bounding boxes were always set to
$0.05$m.

####  3.5.2 Attention on the detected OOIs

An OOI bounding box was defined only when an OOI cluster was detected (Sec.
3.4). The position of the OOI $p^{\text{ooi}}$ was used to define the center
of the OOI bounding box. The size of the bounding box was predefined as a cube
with size $0.03$m in simulation and $0.06$m in the real-world experiments,
according to the size of the objects. The bounding boxes were axis-aligned
(Fig. 4). These bounding boxes automatically adapted to any changes in the OOI
clusters such as a shift in position or were completely removed if the cluster
was not detected anymore.

###  3.6 Viewpoint Utility

The selection of the next-best viewpoint was guided by a metric called
_expected information gain_ , which estimated the amount of novel information
that could be obtained when a camera was moved to a given viewpoint. For the
Semantic NBV planner, we defined the information gain using the semantic
information in the OctoMap.

####  3.6.1 Expected semantic information gain

The semantic information gain is a measure of how much novel semantic
information can be gained by exploring a 3D volume. The semantic information
$I_{\text{sem}}$ expected to be gained by observing a voxel $x$ was defined as
its entropy,

$I_{\text{sem}}(x)=-p_{s}(x)\log_{2}(p_{s}(x))-(1-p_{s}(x))\log_{2}(1-p_{s}(x)),$
(3)

where $p_{s}(x)$ is the confidence score that the voxel $x$ belonged to a
certain object. The confidence score was obtained from the semantic OctoMap as
discussed in Section 3.3. The information gain has a range between $0$ and
$1$. Intuitively, it is high for voxels with high uncertainty (i.e. $p_{s}(x)$
close to $0.5$) and low for voxels with low uncertainty (i.e. $p_{s}(x)$ close
to $0$ or $1$). Hence, it is a measure of uncertainty.

To guide the Semantic NBV planner to search for OOIs, we designed it to
prioritise viewpoints that maximised the expected semantic information gain.
The expected semantic information gain $G_{\text{sem}}(\xi)$ for a viewpoint
$\xi$ was computed as the sum of the semantic information of all voxels that
were expected to be visible from $\xi$, that is,

$G_{\text{sem}}(\xi)=\sum_{x\in(\mathcal{X}_{\xi}\cap\mathcal{B})}I_{\text{sem}}(x),$
(4)

where $\mathcal{X}_{\xi}$ was the set of all voxels that were expected to be
visible from viewpoint $\xi$ and $\mathcal{B}$ was the set of all voxels that
were within the attention regions (defined in Sec. 3.5). $\mathcal{X}_{\xi}$
was determined by the process of ray-tracing, in which a set of rays were cast
from viewpoint $\xi$ evenly across the camera’s view frustum. Each ray
traversed until it hit an occupied voxel or reached a maximum distance. All
voxels that the rays intersected, were expected to be visible if the camera
was moved to viewpoint $\xi$ and hence were added to the set
$\mathcal{X}_{\xi}$.

The Semantic NBV planner used only the semantic information and discarded the
volumetric information. Such a formulation however did not limit the planner
from understanding occupied and unoccupied regions of space, since the
semantic information inherently contains occupancy information to some extent.
For example, any voxel that contains a semantic class label is, by definition,
occupied. However, the Semantic NBV planner gave no importance to how certain
we were about the occupancy and focused primarily on how certain we were about
the semantic labels.

####  3.6.2 Total viewpoint utility

In addition to the expected semantic information gain, the distance from the
current viewpoint was also considered when selecting the next viewpoint since
there was a time cost for executing the motion. We approximated the time cost
as the Euclidean distance between the current viewpoint and the next potential
viewpoint. So, the final view utility $U$ was the expected semantic
information gain $G_{\text{sem}}(\xi)$ scaled by the Euclidean distance cost
$d$,

$U_{\text{sem}}=G_{\text{sem}}(\xi)\times e^{-d},$ (5)

where the exponential to the negative distance cost prioritized the viewpoints
that were very close to the current viewpoint.

Now, the objective of NBV planner was to find a viewpoint that maximised the
viewpoint utility, which would lead to efficient exploration and detection of
the OOIs.

###  3.7 Viewpoint selection

To determine the next-best viewpoint, we sampled a set of candidates
$\mathcal{V}$ at each step within a planar-surface constraint, defined in
Section 4.2.1. These candidates were potential viewpoints where the camera
could be moved next. For each candidate, the view utility was computed using
Equation 5, and the viewpoint with the maximum utility was chosen as the next-
best viewpoint,

$\xi_{\text{best}}=\underset{\xi\in\mathcal{V}}{\arg\max}\;U_{\text{sem}}(\xi).$
(6)

Sampling the candidates is an important step for view selection. If all the
candidates were poor choices, we would end up with an inefficient sequence of
viewpoints despite maximising the information gain at each step. Hence, we
used a pseudo-random sampling strategy, similar to our previous work (Burusa
et al., 2022), which evenly distributed the candidates across the planar
surface. This strategy tried to ensure that there were at least a few
candidates, choosing whom, would lead to efficient exploration of OOIs.
Although, this was not guaranteed. The number of sampled candidates were $27$
for the simulation experiments and $60$ for the real-world experiments.

##  4 Experiments

###  4.1 Existing methods for comparison

We compared the performance of the Semantic NBV planner with a Volumetric NBV
planner (Burusa et al., 2022), two predefined planners and a random planner as
baseline.

####  4.1.1 Volumetric NBV planner

The Volumetric NBV planner is a common approach in NBV planning, which uses
the expected gain in volumetric information to determine the next-best
viewpoint. Many state-of-the-art approaches in view planning are built upon
volumetric or occupancy information (Zeng et al., 2020). The Volumetric NBV
planner tends to explore all uncertain regions of the plant, which is only
suitable for whole-plant perception. Similar to semantic information gain, the
volumetric information gain is a measure of how much novel volumetric
information can be gained by exploring a 3D volume. The volumetric information
$I_{\text{vol}}$ expected to be gained by observing a single voxel $x$ was
defined as,

$I_{\text{vol}}(x)=-p_{o}(x)\log_{2}(p_{o}(x))-(1-p_{o}(x))\log_{2}(1-p_{o}(x)),$
(7)

where $p_{o}(x)$ is the probability of voxel $x$ being occupied. The expected
volumetric information gain $G_{\text{vol}}(\xi)$, the viewpoint utility
$U_{\text{vol}}$, and the next-best viewpoint $\xi_{\text{best}}$ were
computed similar to Equations 4, 5, 6 respectively. We also provided an
adaptive attention mechanism for the Volumetric NBV planner to focus on the
main stem (similar to Sec. 3.5.1). The Volumetric NBV planner was similar to
the Semantic NBV planner, expect that it used $G_{\text{vol}}(\xi)$ instead of
$G_{\text{sem}}(\xi)$ and did not use OOI bounding boxes.

####  4.1.2 Pre-defiend planners

We used two predefined planners with a zig-zag pattern, which is a common
practice in greenhouse robotics to detect the OOIs. Since there was a large
uncertainty in plant position along the y-axis in the simulation experiments,
we could not design a single predefined planner that could perceive the plant
well across the whole uncertainty range ($y=\pm 0.3$m). So, we used two
planners with different ranges of motion: (i) A Predefined Narrow planner with
a narrow range making an angle of $18^{\circ}$ to the center of the plant, and
(ii) a Predefined Wide planner with a wider range making an angle of
$54^{\circ}$ (Fig. 5). The orientation of the viewpoints were fixed as
xyzw:$(0.0,0.0,0.0,1.0)^{T}$ defined as a quaternion, such that camera was
facing the plant. The predefined planners scanned the plants from the bottom
to top.

####  4.1.3 Random planner

We used a random planner that chose one viewpoint at random from the set of
candidate viewpoints $\mathcal{V}$ that were sampled at each step of the
Semantic NBV planner (Sec. 3.7). This planner was used a baseline to confirm
that the Semantic NBV planner was more efficient than choosing a viewpoint at
random.

(a)

(b)

Figure 5: The performance of the Semantic NBV planner was compared with a
Volumetric NBV planner, two predefined planners and a random planner. The
predefined planners had ten viewpoints and differed in their range of motion:
(a) Predefined Narrow planner and (b) Predefined Wide planner.

###  4.2 Simulation experiments

We studied how efficiently could the Semantic NBV planner detect the OOIs
compared to the Volumetric NBV, Predefined Wide, Predefined Narrow, and Random
planners. The simulation setup (Sec. 4.2.1) contained uncertainty in plant and
OOI positions, high level of occlusions from plant parts, and a planar-surface
constraint for sampling the candidate viewpoints. The evaluation criteria for
the comparison is discussed in Section 4.2.3.

####  4.2.1 Simulation setup

Figure 6: Schematic diagram of the experimental setup in simulation, with an
ABB IRB 1200 robotic arm and Realsense L515 camera. The plant was positioned
in front of the robot with uncertainty (shown in pink) to test the robustness
of the viewpoint planners. The robot was allowed to choose viewpoints on a
planar-surface constraint (shown in green) with $4$ DoF.

We used a simulated environment with _Gazebo_ 111http://gazebosim.org/, as
illustrated in Figure 6. The robotic setup consisted of a $6$ DoF robotic arm
(ABB’s IRB 1200) with an RGB-D camera (Intel Realsense L515) attached to its
end-effector. We used eight 3D mesh
models222https://www.cgtrader.com/3d-models/plant/other/xfrogplants-tomato of
tomato plants of varying growth stages and structural complexity, which the
robot explored to detect the OOIs (illustrated in Figure 7). A tomato plant
was placed in front of the robot at $p^{\text{pb}}=(0.7\pm 0.1\text{m},0.0\pm
0.3\text{m},0.8\text{m})$. Uncertainty was introduced in the plant-base
position by uniformly sampling the x and y positions in the range $0.7\pm
0.1$m and $0.0\pm 0.3$m respectively. The experiments were repeated for $12$
different rotations of the plant along the z-axis at intervals of $30^{\circ}$
each, leading to a total of $96$ experiments.

A planar-surface constraint was used for view planning, i.e., the robot could
choose viewpoints only within the bounds of the plane. The planar surface was
of size $0.7\times 0.7$ m2, with its center at a distance of $0.35$m from the
plant and at a height of $0.8$m. In addition to positioning the camera on the
planar surface, the viewpoints could also make a pan and tilt of $0\pm
15^{\circ}$, which were uniformly sampled. Hence, the viewpoints had $4$ DoF.
All the viewpoint planners started from the same initial viewpoint $\xi_{0}$
with position xyz:$(0.35,0.22,0.90)^{T}$ and orientation
xyzw:$(0.00,0.00,-0.26,0.97)^{T}$. The robot was allowed to choose a maximum
of $a_{\text{max}}=10$ consecutive viewing actions to explore the plant and
identify as many OOIs as possible.

(a)

(b)

Figure 7: Eight mesh models of tomato plants were used for the experiments.
The plants varied in terms of growth stage, number of OOIs, height and
complexity. Some examples of (a) the original plant models and (b) the same
plant models with around $50$-$60\%$ leaflets removed are shown.

####  4.2.2 Analysis of experimental and model parameters

We analysed the effect of the following experimental and model parameters on
the planner performance:

Impact of removing uncertainty in plant position In practice, the exact
position of the tomato plants will not be known. Hence, the Semantic NBV
planner should be able to handle such uncertainty. In the simulation
experiments, the plant positions had an uncertainty of $\pm 0.1$m in the
x-axis and $\pm 0.3$m in the y-axis. We analysed the impact of this
uncertainty by comparing the performance of the planners to a case where the
exact plant positions were assumed to be known.

Impact of removing uncertainty in OOI positions The Semantic NBV planner has
to deal with uncertainty in the OOI positions too, which could be caused due
to errors in detection. The Semantic NBV planner should still be able to
estimate the OOI positions accurately and define appropriate bounding boxes
for attention. We analysed the impact of OOI-position uncertainty by comparing
the performance of the planners to a case where the exact OOI-positions were
known.

Impact of removing attention Our proposed attention mechanism is crucial in
improving the efficiency of OOI detection. Without an attention mechanism, the
planner will consider semantic information from all voxels expected to be
visible from a viewpoint. Hence, it might prioritise every part of the plant
instead of the OOIs alone, which could make it inefficient. We tested this
hypothesis by comparing the performance of the planners to a case where the
the main-stem bounding boxes and the OOI bounding boxes were completely
removed.

Impact of reducing plant complexity The Semantic NBV planner needs to detect
the OOIs even when they are heavily occluded from the camera’s view. We
analysed the impact of occlusion by comparing the performance of the planners
to a case where the plant complexity was reduced by removing around
$50$-$60\%$ of the leaflets, as shown in Figure 7(b).

Sensitivity to sampling constraint The planar-surface constraint (Sec. 4.2.1)
defined a region in which the planners could sample candidate viewpoints. We
analysed the sensitivity of the planners to different sampling constraints to
ensure that our results would still hold under different sampling strategies.
We compared the performance of the planners under a planar-surface constraint
($4$ DoF) and a cylindrical-sector constraint ($2$ DoF), as used in Burusa et
al. (2022). Under the cylindrical-sector constraint, a viewpoint could be
positioned anywhere on the cylinder surface but was always oriented towards
the center of the cylinder. The cylinder had a height of $0.7$m, radius of
$0.4$m and a sector angle of $90^{\circ}$. The plant positions were subjected
to an uncertainty of $\pm 0.1$m in the x-axis and $\pm 0.3$m in the y-axis.

####  4.2.3 Evaluation

We evaluated the performance of the planners by calculating the percentage of
OOIs that were correctly detected out of all the OOIs in each plant. We call
this metric the _percentage of correctly-detected objects_ (PCO),

$\text{PCO}=\frac{\text{Correctly detected objects}}{\text{Total number of
objects}}\times 100.$ (8)

An object was considered correctly detected only when its semantic class was
correctly identified and at least $50\%$ of the object was accurately
reconstructed by the planner. This was quantified using the metric of
F1-score, similar to Burusa et al. (2022). The F1-score compares the ground-
truth and reconstructed OOIs and estimates both the correctness and
completeness of the reconstruction. It takes a value between $0$ and $1$,
where a value of $1$ implies a complete and correct reconstruction. The PCO
metric is more rigorous than simply evaluating the 2D detection performance,
as it requires the semantic class label, the position, and the shape of the
object to be correctly perceived. Also, to accurately reconstruct $50\%$ of
the object despite occlusions, the object needs to be perceived correctly from
multiple viewpoints.

Figure 8: The ground truth for evaluation was obtained by processing the
original 3D mesh of tomato plants in Blender. The ground-truth positions of
the objects-of-interest (OOIs) were estimated by averaging the positions of
vertices that belonged to the object. Examples of a tomato, petiole, and
peduncle with their vertices highlighted (in orange) is shown.

We followed a series of steps to enable the computation of the F1-score for
all the OOIs. First, the original 3D mesh of the tomato plants were converted
to a point cloud and down-sampled to the same resolution as the semantic
OctoMap (i.e. $0.003$m) using voxel-grid filtering. The semantic OctoMap was
also converted to a point cloud, to make it comparable to the ground-truth.
Only the occupied voxels were converted, i.e. voxels with $p_{o}(x)>0.5$.
Second, the true position of the OOIs were obtained manually by processing the
3D mesh of the plants in Blender333https://www.blender.org/. For each object,
the true position was calculated by averaging the positions of mesh vertices
that belonged to the object, as shown in Figure 8. Finally, from both the
ground-truth and reconstructed point clouds, the points belonging to each OOI
was extracted by using a bounding-box of size $0.03$m and placing it at the
true position of the OOI. The points within the bounding box were considered
to belong to the OOI. These steps gave us the ground-truth and reconstructed
point clouds for each OOI, which were then used to compute the F1-scores.

###  4.3 Real-world experiments

We performed real-world experiments with pre-recorded data collected from a
tomato greenhouse at Unifarm, Wageningen University and Research, The
Netherlands, to test if the Semantic NBV planner can perform accurately and
efficiently under real-world complexities such as natural variation and
occlusion, natural and varying illumination, sensor noise, and uncertainty in
camera poses. The objective of the viewpoint planner was to search and detect
the petiole and peduncles, as these are the plant parts that are relevant for
de-leafing and harvesting tasks respectively. Hence, they were defined as the
OOIs. We compared the performance of the Semantic NBV planner with Volumetric
NBV, predefined, and random planners. All the viewpoint planners started from
the same initial viewpoint $\xi_{0}$ for each plant. The robot was allowed to
choose a maximum of $a_{\text{max}}=10$ consecutive viewing actions to explore
the plant and identify as many OOIs as possible.

(a)

(b)

(c)

Figure 9: (a) The real-world setup with a $6$ DoF ABB IRB 1200 robotic arm and
a Realsense L515 RGB-D camera. (b) Front and top views of the $750$ viewpoints
on a planar surface used for collecting data of tomato plants. (c) Examples of
images of a tomato plants collected by the robot, showing the level of
occlusion.

####  4.3.1 Real-world setup and data collection

The real-world robot was similar to the simulation robot, consisting of a $6$
DoF robotic arm (ABB’s IRB 1200) with an RGB-D camera (Intel Realsense L515)
attached to its end-effector. The position of the camera with respect to the
robotic arm was calibrated using eye-in-hand calibration. Robotic Operating
System (ROS) was used as the middleware and a motion planning package for ROS,
MoveIt!, was used for moving the camera to a desired viewpoint. The robotic
setup was mounted on a cart that could move on the rails between rows of
tomato plants in a greenhouse.

A dataset of ten tomato plants was collected, in which the color and depth
images with a resolution of $960\times 540$ pixels, the corresponding
structured point cloud, and the 6D camera pose with respect to the robot’s
frame were recorded. The viewpoints were predefined along a grid of $15$
equally-spaced rows and $10$ equally-spaced columns on a planar surface of
height $0.6$m and width $0.4$m, with the center of the plane at a distance of
$0.5$m from the plant and at a height of $0.75$m from the robot’s base frame.
At each point on the grid, data was collected from five viewpoints with pan
and tilt angles of
$[(-15^{\circ},0^{\circ}),(15^{\circ},0^{\circ}),(0^{\circ},0^{\circ}),(0^{\circ},15^{\circ}),(0^{\circ},-15^{\circ})]$.
Hence, the viewpoints had $4$ DoF and made a total of $750$ viewpoints for
each plant, as shown in Figure 9(b).

This pre-recorded dataset allowed us to repeat experiments and compare the
performances of different planners. The objective of the planners was to
determine the shortest sequence of viewpoints from the $750$ candidates that
could efficiently find and perceive the petioles and peduncles. This was a
discretised case of viewpoint planning in which the planners only had a subset
of viewpoint to choose from. It was done for the sake of feasibility and
repeatability of the experiments. When applied in practice, the planners can
sample viewpoints uniformly from a continuous space that is reachable by the
robot, as done in the simulation experiments. For each plant, the experiment
was repeated $12$ times, leading to a total of $120$ experiments with ten
real-world plants.

####  4.3.2 Ground truth and evaluation

Figure 10: A plant reconstructed using COLMAP, which was used as the ground
truth for real-world experiments. The 3D positions of the objects-of-interest
were annotated manually (marked in yellow).

The true mesh models of real-world plants were not available, unlike the
simulation experiments. Hence, we compared the reconstruction from the
planners with COLMAP (Schonberger and Frahm, 2016), a structure-from-motion-
based reconstruction algorithm. The color images from all the $750$ viewpoints
for each plant, along with the camera poses, were provided to COLMAP to
generate a 3D point cloud of the plant. On this point cloud, the 3D position
of the plant nodes were manually annotated by selecting all the points that
belonged to each node and taking an average of their 3D positions. This COLMAP
point cloud and the annotated node positions (as shown in Figure 10) were used
as ground truth in our evaluation.

We evaluated the performance of the planners, similar to the simulation
experiments, by computing the percentage of correctly-detected objects (PCO).
An object was considered as correctly detected when it had an F1-score of at
least $62.5\%$. This F1-score threshold was increased compared to the
simulation experiments because the ground-truth point cloud was incomplete.
The ground-truth point cloud was generated from a set of $750$ predefined
viewpoints which observed the plant only from one side. We assumed that the
ground-truth point cloud represented only $80\%$ of the true OOI. So, in order
to evaluate if $50\%$ of the true OOI was observed, we had to increase the
F1-score threshold to $62.5\%$ of the ground-truth point cloud. The F1-score
compared the OOI points in the reconstructed point cloud from the semantic
OctoMap and the ground-truth point cloud from COLMAP. We followed a similar
procedure described in Sec. 4.2.3, except for two details. First, the point
clouds were down sampled to a resolution of two-voxels distance (i.e.
$0.006$m) instead of one-voxel distance, to account for the fact that the
COLMAP point cloud was not a perfect ground truth and had some reconstruction
error. Second, the size of the bounding box used to extract points belonging
to the OOIs was increased to $0.04$m, since the OOIs in the real-world were
larger.

(a)

(b)

Figure 11: Performance of the planners on the detection of all objects-of-
interest (OOIs) when a tomato plant was placed $0.7$m in front of the robot
with an uncertainty of $\pm 0.1$m in the x-axis and $\pm 0.3$m in the y-axis.
The percentage of correctly-detected objects (PCO) was determined for each
viewing action. A detection was considered correct when at least $50\%$ of the
object was perceived (F1-score $>=0.5$). (a) The mean PCO of the planners at
each viewing action over $96$ experiments. The error bands show the $95\%$
confidence interval of the mean over all experiments. (b) The distribution
density of PCO for the planners at the end of nine viewing actions over $96$
experiments. The median PCO is shown by the white dot in the plot.

##  5 Results

###  5.1 Simulation results

Figure 11(a) shows the performance of all planners on the detection of OOIs
when the plants where roughly positioned in front of the robot. We observed
that the Semantic NBV planner significantly outperformed the other planners in
terms of PCO at each viewing action and was able to detect $80\%$ of the OOIs
within nine viewing actions. After nine viewing actions, the Semantic NBV
planner was able to efficiently search and detect $81.8\%$ of the OOIs on
average over 96 experiments, despite occlusions and uncertainty in plant
positions. It perceived $13.3\%$ more OOIs than the Volumetric NBV planner,
$13.3\%$ more than the Predefined Wide planner, $27.6\%$ more than the
Predefined Narrow planner, and $34.7\%$ more than the Random planner. For
targeted perception of plant parts, these results clearly show that viewing
only the unexplored regions of the plant, as the Volumetric NBV planner, is
not efficient enough. It does not perform any better than a well-designed
predefined strategy, such as the Predefined Wide planner. However, by adding
knowledge of semantics, the performance improved significantly since the
planner was now able to identify the relevant plant parts and prioritise them.

The distribution density of the PCO after nine viewing actions averaged over
all $96$ experiments, shown in Figure 11(b), offers further insights on the
performance of the planners. We observe that the Semantic NBV planner’s
performance was more repeatable as indicated by the low spread of the density
function and had a high accuracy with a median PCO of $87.8\%$, i.e. half the
experiments had a PCO greater than this. The Volumetric NBV and Predefined
Wide planners had a median PCO of $75.5\%$ and $79.2\%$ respectively, and
their performances were less repeatable compared to the Semantic NBV planner
as indicated by the wider spread of their PCO densities. The Predefined Narrow
and Random planners were least repeatable as their PCO densities were spread
over the entire range of PCO and their median PCOs were much lower at $70.6\%$
and $50.0\%$ respectively.

####  5.1.1 Analysis of experimental and model parameters

For brevity, we only analysed the three best performing planners from the
previous experiment, namely Semantic NBV, Volumetric NBV and Predefined Wide
planners. In the following analysis, we refer to the experiment presented in
the previous section as the _original experiment_ and its results as the
_original result_.

(a)

(b)

(c)

(d)

(e)

Figure 12: Analysis of the performance of the planners based on various
experimental and model parameters. The plots show the impact of (a) removing
uncertainty in plant position, (b) removing uncertainty in OOI position, (c)
removing attention, (d) reducing plant complexity or occlusion, and (e)
changing the sampling constraint. The solid lines show the updated results
based on the modified parameter and the dotted lines show the original
results. The error bands show the $95\%$ confidence interval of the mean over
all 96 experiments. Please note: The solid and dotted lines of the Predefined
Wide planner overlap each other in plots (b) and (c) since the performance of
the planner was not influenced by these parameters.

Impact of removing uncertainty in plant position: Figure 12(a) shows the
performance of the planners when the plants were placed in a fixed and known
position in front of the robot. The Predefined Narrow planner is also shown in
the plot because it detected the most OOIs after ten viewpoints. Compared to
the original experiment with uncertainty in plant position (Fig. 11(a)), the
Semantic NBV planner detected $8.7\%$ more OOIs after nine viewpoints in this
case, since it did not have to search for the plant before detecting the OOIs.
The Predefined Narrow planner performed the best by detecting $96.0\%$ of the
OOIs after nine viewpoints. This indicates that the predefined planners in
practice can be effective when the positions of the plants are known
accurately. However, they are extremely sensitive to uncertainty in plant
position and their performance can be unreliable, as seen from Figure 11(b).
The Semantic and Volumetric NBV planners, on the other hand, are able to
handle the uncertainty better by using the adaptive attention mechanism.

Impact of removing uncertainty in OOI positions: Figure 12(b) shows the
performance of the planners when the true OOI positions were known, i.e., all
the OOI bounding boxes were provided at the beginning of the experiments. The
Semantic NBV planner showed a small improvement and could detect $6.1\%$ more
OOIs after nine viewing actions compared to the original experiment. This
implies that the Semantic NBV planner in the original experiment was able to
explore and detect the OOIs quite reliably. Hence, its performance was close
to the ideal case. On the other hand, the Volumetric NBV planner showed a
larger improvement and could detect $11.7\%$ more OOIs after nine viewing
actions, indicating that it could not detect the OOIs reliably in the original
experiment. The poor performance of the Volumetric NBV planner was likely due
to the lack of an attention mechanism to focus on the OOIs. The performance of
the Predefined Wide planner was independent of the OOI positions.

Impact of removing attention: Figure 12(c) shows the performance of the
planners in the absence of an attention mechanism. The impact of removing
attention was drastic on the Semantic and Volumetric NBV planners. Their
performance dropped $44.3\%$ and $42.2\%$ respectively and they were unable to
detect even half the OOIs after nine viewing actions. The drop in performance
was because the planners had less guidance to perceive the relevant plant
parts and spent time exploring irrelevant regions. This shows the significance
of an attention mechanism in making NBV planning more efficient when
performing a targeted search. Interestingly, the Semantic NBV planner
outperforms the Volumetric NBV planner in this case too. This shows that the
addition of semantic information, by itself, is helpful in guiding the planner
to detect OOIs. When combined with an attention mechanism, the performance is
further enhanced.

Impact of reducing plant complexity: Figure 12(d) shows that the performance
of all the planners improved when the plant complexity was reduced. The
Semantic NBV planner could detect $5.8\%$ more OOIs after nine viewing
actions, while the Volumetric NBV planner could detect $4.2\%$ more, and the
Predefined Wide planner could detect $5.3\%$ more. By removing the leaflets,
the OOIs were less occluded and easier to detect from many viewpoints. The
fact that only a small improvement in performance was observed indicates that
all the three planners were able to handle the plant complexity and occlusions
in the original experiment well. The overall results were still consistent
with the original results, with the Semantic NBV planner outperforming the
other planners by a significant margin.

(a)

(b)

Figure 13: Performance of the planners on the detection of petioles and
peduncles of real tomato plants in a greenhouse. The percentage of correctly-
detected objects (PCO) was determined for each viewing action. A detection was
considered correct when at least 62.5$\%$ of the object was perceived
(F1-score $>=0.625$). (a) The mean PCO of the planners at each viewing action
over $120$ experiments. The error bands show the $95\%$ confidence interval of
the mean over all experiments. (b) The distribution density of PCO for the
planners at the end of seven viewing actions over $120$ experiments. The
median PCO is shown by the white dot.

Sensitivity to sampling constraint: Figure 12(e) shows the performance of the
planners when a cylindrical-sector constraint was used for view sampling (Sec.
4.2.2) instead of a planar-surface constraint. All the planners performed
poorer in this case. The Semantic NBV planner detected $9.8\%$ less OOIs after
nine viewing actions, while the Volumetric NBV planner detected $11.3\%$ less
and the Predefined Wide planner detected $10.6\%$ less. Under the cylindrical-
sector constraint, the viewpoints were forced to look towards the center of
the cylinder, which reduced the DoF from four to two. This potentially reduced
the chances of finding good viewpoints and hence led to a drop in performance
for all planners. The overall results were still consistent with the original
results, with the Semantic NBV planner outperforming the other planners.

###  5.2 Real-world results

Figure 13(a) shows the performance of the planners on the detection of
petioles and peduncles of real-world tomato plants in a greenhouse. We
observed that the Semantic NBV planner outperformed the other planners in
terms of PCO after five viewing actions and was able to detect $80\%$ of the
OOIs within seven viewing actions. After seven viewing actions, the Semantic
NBV planner was able to search and detect $82.7\%$ of the OOIs on average over
$120$ experiments, despite the real-world complexities such as natural
variation and occlusion, natural illumination, sensor noise, and uncertainty
in camera poses. The Semantic NBV planner detected $6.9\%$ more OOIs than the
Volumetric NBV planner, $15.8\%$ more than the Predefined Wide planner, and
$7.6\%$ more than the Random planner. The Volumetric NBV planner did not
perform any better than the Random planner in terms of the PCO, as it was
attending to all unexplored regions of the plant and not attending
specifically to the relevant plant parts. The Predefined Wide planner was the
least efficient planner, which clearly indicates that using fixed or
predefined viewpoints cannot handle occlusions well.

We observed that the PCO of the Semantic NBV planner is similar to or slightly
less than that of the Random planner in the initial viewpoints (Figure 13(a)).
This could potentially be due to false positives in the instance segmentation
step (Section 3.1), in which Mask R-CNN sometimes mistook other plant parts as
petioles or peduncles, as shown in Figure 14(b). False positives will cause
the Semantic NBV planner to define a bounding box at a region that did not
contain any OOI. Hence, the planner will be inefficient as it will
unnecessarily spend viewpoints exploring an irrelevant region. However, in
later viewpoints, the PCO of the Semantic NBV planner improved faster compared
to the Volumetric NBV and Random planners, potentially because the falsely
detected OOIs were sufficient explored and the planner returned its focus to
the true OOIs. The Semantic NBV planner had an average of one or two false
positives per plant across all experiments (Figure 14(a)). By reducing or
handling these false positives, the performance of the Semantic NBV planner
can potentially be improved further. Although the Volumetric NBV and the
Random planners had a similar or higher number of false positives, these
algorithms were not influenced by the false positives because they did not
define bounding boxes on the OOIs.

(a)

(b)

Figure 14: False positives from the instance segmentation algorithm influenced
the performance of the Semantic NBV planner. (a) The mean number of false
positives of the planners at each viewing action over $120$ experiments. The
error bands show the $95\%$ confidence interval of the mean over all
experiments. (b) Examples of two false positives in the tomato greenhouse. The
top detection mistook a branch from a stem behind as a petiole, whereas the
bottom detection mistook a cable passing across a stem in the front as a
peduncle.

The distribution density of the PCO after seven viewing actions averaged over
all $120$ experiments, shown in Figure 13(b), shows that the Semantic NBV
planner’s performance was highly repeatable as indicated by the low spread of
the density function and had a high accuracy with a median PCO of $85.7\%$.
The density functions of the Volumetric NBV and Random planners had a wider
spread compared to the Semantic NBV planner and hence they were less
repeatable. They also had a lower accuracy with a median PCO of $80.0\%$ and
$75.0\%$ respectively. The Predefined Wide planner’s performance was highly
repeatable, but also the least accurate with a median PCO of $66.7\%$.

These results show that using semantic information about the relevant plant
parts, we can reliably and efficiently detect the OOIs using fewer viewpoints.
The results of the real-world experiments strongly support the results of the
simulation experiments, indicating the advantage of using semantic information
for NBV planning. Furthermore, our results are valid when using a real robot
and camera setup under the complexities of real-world greenhouse environments.

##  6 Discussion

The simulation and real-world results show that the Semantic NBV planner is
more efficient in detecting the OOIs compared the volumetric and predefined
planners. By adding semantic information to an OctoMap and using it to define
regions-of-interests, the planner chose viewpoints that paid attention to the
OOIs and hence was able to detect them effectively. There are no prior works
in agro-food robotics that used a similar approach, so we checked our
conclusion with literature in robotics. Our results are in line with Kay et
al. (2021), who showed that distinguishing target objects from non-target
objects using semantic information in NBV planning leads to faster
reconstruction performance. The results are also in line with Zheng et al.
(2019), who showed that the combined use of volumetric and semantic
information leads to faster scene reconstruction and improved segmentation of
objects. The main difference of our work from Kay et al. (2021) and Zheng et
al. (2019) is the use of OOI clustering and an attention mechanism in
combination with semantic information, which provided an object-level
representation of the scene. This allowed our NBV planner to be aware of
unexplored regions of each object. Moreover, these works were tested only in
simulation and were not exposed to real-world complexities.

While the performance of the Semantic NBV planner was promising, there are
certain aspects worth considering:

###  6.1 Influence of object-detection errors

The object-detection algorithm, Mask R-CNN, was able to detect most of the
OOIs, but had some false positives and false negatives. False positives will
cause the algorithm to explore irrelevant regions and make it inefficient.
False negatives, on the other hand, will cause the algorithm to not pay
attention to an OOI, leading to a poor perception performance. The impact of
false positives and false negatives on our results was less severe since we
used multiple viewpoints. Nevertheless, to improve the efficiency and accuracy
of the planner, it is important to analyse the impact of these object-
detection errors on the Semantic NBV planner and take measures to minimise
them. Reducing false positives in semantic segmentation is difficult due to
the complexity in greenhouses. There is always a chance for the instance
segmentation network to observe something out-of-distribution and make a false
detection. Future work could instead focus on effective strategies to remove
false positives, for example, by verifying a detection from multiple
viewpoints or estimating its uncertainty through repeated inferences.

###  6.2 Completeness of object perception

We used a threshold of $50\%$ and $62.5\%$ for the F1-score to consider an
object as detected. For different applications, this threshold can vary
depending on the desired perception accuracy. Although the sensitivity of the
algorithm to the F1-score threshold was not studied in this paper, we believe
that the Semantic NBV planner can still significantly outperform the other
planners when the threshold is varied.

In our implementation, the F1-score threshold was only applied during
evaluation. During execution, the Semantic NBV planner did not stop perceiving
an object when the threshold was met, but continued to perceive the object
completely. Hence, the planner might have taken several extra viewpoints than
necessary. If we already know the level of object completeness required for
our application, we can modify the planner to plan viewpoints around an object
only until the desired completeness is achieved.

###  6.3 Applicability in real-world conditions

We evaluated our method using a dataset collected in a tomato greenhouse,
which included real-world complexities such as natural variation and occlusion
in tomato plants, natural illumination, sensor noise, and uncertainty in
camera poses. The Semantic NBV planner was able to handle these conditions
using a probabilistic 3D representation that merged semantic information from
multiple viewpoint and kept track of the OOIs. Also, by using an adaptive
attention mechanism, the planner was able to select the most informative
viewpoints to perceive the OOIs.

The real world has a few other complexities that were not captured by our
dataset. In the dataset, we experimented with only one plant at a time.
However, in a tomato greenhouse, multiple plants can be close together and
their stems can overlap, leading to higher levels of occlusions. Also, we
assumed the scene in the greenhouse to be static, but a tomato plant can move
due to wind or contact by a human or the robot. The additional complexity
introduced by such dynamics were not included in our experiments. These
complexities lead to more uncertainty in perception of tomato plants, which
only supports the need for active vision even more. Future work could develop
methods to handle multiple plants and dynamics in tomato greenhouses.

##  7 Conclusion

In this paper, we presented a semantics-aware active-vision strategy for
efficiently searching and detecting targeted plant parts, such as tomatoes,
petioles, and peduncles. We developed a Semantic NBV planner, which added
semantic information and an attention mechanism to conventional next-best-view
planning. We performed simulation and real-world experiments to evaluate and
gain insights into the behaviour of the planner, while ensuring repeatability
and statistical significance. In simulation, the Semantic NBV planner detected
$81.8\%$ of the plant parts, about $13.3\%$ more plant parts per plant
compared to a Volumetric NBV planner, a commonly used active-vision strategy.
Also, it detected about $13.3\%$ and $27.6\%$ plant parts more compared to two
predefined planners and $34.7\%$ plant parts more compared to a random
planner. The Semantic NBV planner was able to search for the plant parts
reliably with a median of $87.8\%$ correctly-detected objects in $96$
experiments. We also showed that our Semantic NBV planner was robust to
uncertainty in plant and plant-part position, plant complexity, and different
sampling constraints.

To the best of our knowledge, our work was the first to test the performance
of a semantics-aware active vision strategy using a robotic setup in a real-
world greenhouse environment. In real-world experiments, the Semantic NBV
planner detected $82.7\%$ of the plant parts, about $6.9\%$ more plant parts
per plant compared to a Volumetric NBV planer, $15.8\%$ more than a predefined
planner, and $7.6\%$ more than a random planner. The real-world results
strongly support the simulation results, despite the complexities of the real
world, such as natural variation and occlusion, natural illumination, sensor
noise, and uncertainty in camera poses.

Our results indicate that using an active-vision strategy with semantic
information and an attention mechanism can effectively address the major
challenge of occlusion in agro-food environments, thereby improving the
accuracy and efficiency of perception. We hope it leads to significant
improvements in robotic perception and greenhouse crop production in the
future.

## CRediT author statement

Akshay K. Burusa: Conceptualization, Methodology, Software, Investigation,
Data Curation, MSc thesis - Supervision, Writing - Original draft & Editing;
Joost Scholten: MSc thesis - Investigation, Software; Xin Wang: Data
Collection, MSc thesis - Supervision, Writing - Review; David Rapado Rincon:
MSc thesis - Supervision, Writing - Review; Eldert J. van Henten:
Conceptualization, Writing - Review, Supervision, Funding acquisition; Gert
Kootstra: Conceptualization, Writing - Review, Supervision, Funding
acquisition.

## Funding

This research is funded by the Netherlands Organization for Scientific
Research (NWO) project Cognitive Robots for Flexible Agro-Food Technology
(FlexCRAFT), grant P17-01.

## Acknowledgements

We thank Thomas A.W. Versmissen for his help with annotating the real-world
data and training the object-detection network. We thank the members of the
FlexCRAFT project for engaging in fruitful discussions and providing valuable
feedback to this work.

## References

  * Ankerst et al. (1999) Ankerst, M., Breunig, M.M., Kriegel, H.P., Sander, J., 1999.  Optics: Ordering points to identify the clustering structure.  ACM Sigmod record 28, 49–60.
  * Beed et al. (2021) Beed, F., Taguchi, M., Telemans, B., Kahane, R., Le Bellec, F., Sourisseau, J.M., Malézieux, E., Lesueur-Jannoyer, M., Deberdt, P., Deguine, J.P., Faye, E., 2021.  Fruit and vegetables: Opportunities and challenges for small-scale sustainable farming.  FAO, CIRAD.
  * Bircher et al. (2016) Bircher, A., Kamel, M., Alexis, K., Oleynikova, H., Siegwart, R., 2016.  Receding horizon” next-best-view” planner for 3d exploration, in: 2016 IEEE international conference on robotics and automation (ICRA), IEEE. pp. 1462–1468.
  * Burusa et al. (2022) Burusa, A.K., van Henten, E.J., Kootstra, G., 2022.  Attention-driven active vision for efficient reconstruction of plants and targeted plant parts.  arXiv preprint arXiv:2206.10274 .
  * Daudelin and Campbell (2017) Daudelin, J., Campbell, M., 2017.  An adaptable, probabilistic, next-best view algorithm for reconstruction of unknown 3-d objects.  IEEE Robotics and Automation Letters 2, 1540–1547.
  * Gibbs et al. (2019) Gibbs, J.A., Pound, M., French, A., Wells, D., Murchie, E., Pridmore, T., 2019.  Active vision and surface reconstruction for 3d plant shoot modelling.  IEEE/ACM transactions on computational biology and bioinformatics .
  * He et al. (2017) He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017.  Mask r-cnn, in: Proceedings of the IEEE international conference on computer vision, pp. 2961–2969.
  * Hemming et al. (2014) Hemming, J., Ruizendaal, J., Hofstee, J.W., Van Henten, E.J., 2014.  Fruit detectability analysis for different camera positions in sweet-pepper.  Sensors 14, 6032–6044.
  * Hornung et al. (2013) Hornung, A., Wurm, K.M., Bennewitz, M., Stachniss, C., Burgard, W., 2013.  Octomap: An efficient probabilistic 3d mapping framework based on octrees.  Autonomous robots 34, 189–206.
  * Isler et al. (2016) Isler, S., Sabzevari, R., Delmerico, J., Scaramuzza, D., 2016.  An information gain formulation for active volumetric 3d reconstruction, in: 2016 IEEE International Conference on Robotics and Automation (ICRA), IEEE. pp. 3477–3484.
  * Kay et al. (2021) Kay, S.A., Julier, S., Pawar, V.M., 2021.  Semantically informed next best view planning for autonomous aerial 3d reconstruction, in: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE. pp. 3125–3130.
  * Koirala et al. (2019) Koirala, A., Walsh, K.B., Wang, Z., McCarthy, C., 2019.  Deep learning–method overview and review of use for fruit detection and yield estimation.  Computers and electronics in agriculture 162, 219–234.
  * Kootstra et al. (2021) Kootstra, G., Wang, X., Blok, P.M., Hemming, J., Van Henten, E., 2021.  Selective harvesting robotics: current research, trends, and future directions.  Current Robotics Reports 2, 95–104.
  * Marangoz et al. (2022) Marangoz, S., Zaenker, T., Menon, R., Bennewitz, M., 2022.  Fruit mapping with shape completion for autonomous crop monitoring, in: 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE), pp. 471–476.
  * Rigg et al. (2020) Rigg, J., Phongsiri, M., Promphakping, B., Salamanca, A., Sripun, M., 2020.  Who will tend the farm? interrogating the ageing asian farmer.  The Journal of Peasant Studies 47, 306–325.
  * Schmid et al. (2020) Schmid, L., Pantic, M., Khanna, R., Ott, L., Siegwart, R., Nieto, J., 2020.  An efficient sampling-based method for online informative path planning in unknown environments.  IEEE Robotics and Automation Letters 5, 1500–1507.
  * Schonberger and Frahm (2016) Schonberger, J.L., Frahm, J.M., 2016.  Structure-from-motion revisited, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104–4113.  doi:10.1109/CVPR.2016.445.
  * Van Henten (2004) Van Henten, E., 2004.  Greenhouse mechanization: state of the art and future perspective, in: International Symposium on Greenhouses, Environmental Controls and In-house Mechanization for Crop Production in the Tropics 710, pp. 55–70.
  * Van Henten et al. (2002) Van Henten, E.J., Hemming, J., Van Tuijl, B., Kornet, J., Meuleman, J., Bontsema, J., Van Os, E., 2002.  An autonomous robot for harvesting cucumbers in greenhouses.  Autonomous robots 13, 241–258.
  * Xuan and David (2018) Xuan, Z., David, F., 2018.  Real-time voxel based 3d semantic mapping with a hand held rgb-d camera.  URL: https://github.com/floatlazer/semantic_slam.
  * Zaenker et al. (2021) Zaenker, T., Smitt, C., McCool, C., Bennewitz, M., 2021.  Viewpoint planning for fruit size and position estimation, in: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE. pp. 3271–3277.
  * Zeng et al. (2020) Zeng, R., Wen, Y., Zhao, W., Liu, Y.J., 2020.  View planning in robot active vision: A survey of systems, algorithms, and applications.  Computational Visual Media 6, 225–245.
  * Zheng et al. (2019) Zheng, L., Zhu, C., Zhang, J., Zhao, H., Huang, H., Niessner, M., Xu, K., 2019.  Active scene understanding via online semantic reconstruction, in: Computer Graphics Forum, Wiley Online Library. pp. 103–114.


Results on Simulated Noisy Speech

Table : WER (%) result on LibriSpeech (test clean / test other) simulated
noisy datasets. The second line indicates the enhancement front-end applied
during inference.

max width=  2*Method 0 $\sim$ 5dB test clean / test other -5 $\sim$ 0dB test
clean / test other ConvTasNet DCUNet SkiM BLSTM NoEnh ConvTasNet DCUNet SkiM
BLSTM NoEnh 1\. Baseline 19.1/37.1 16.4/33.3 20.8/41.5 27.0/49.0 38.7/60.2
34.6/57.8 31.9/53.1 34.5/64.5 53.4/71.8 70.3/84.2 2\. $+$ IMST 16.7/32.9
15.2/29.8 17.1/33.3 21.6/38.6 18.2/34.4 29.0/48.1 26.8/43.8 29.6/48.3
41.5/57.6 34.4/51.7 3a. $++$ OA_all 14.0/28.9 12.8/27.4 15.1/29.2 16.4/32.6
17.1/34.0 26.1/44.4 23.9/41.4 27.6/44.7 34.7/51.5 35.6/52.3 3b. $++$ OA_first
13.2/28.9 12.4/26.6 13.9/29.2 17.8/35.0 16.6/33.9 24.2/43.7 22.8/40.5
25.0/43.9 38.0/54.0 35.5/53.1 3c. $++$ OA_last 14.8/29.8 13.1/28.0 15.5/30.6
17.1/34.1 16.4/33.1 28.2/46.3 25.6/43.4 29.3/46.7 36.8/53.9 34.3/50.9 4a. $++$
SF_all 40.5/57.8 40.6/57.7 40.8/57.6 40.9/57.3 40.5/57.6 66.0/76.0 65.8/75.4
65.8/75.8 66.1/75.9 66.1/75.9 4b. $++$ SF_first 14.0/29.3 12.6/26.7 15.0/29.7
18.0/34.6 18.7/35.5 25.5/43.7 23.3/40.5 26.3/44.1 37.8/54.0 37.9/54.4 4c. $++$
SF_last 13.6/29.3 12.8/27.4 14.4/29.6 16.2/32.8 16.3/32.7 26.6/45.7 25.0/43.4
28.3/46.1 34.9/52.3 33.5/51.0 5a. $++$ DA_all 46.4/64.1 46.3/64.0 46.3/64.1
46.4/64.3 46.4/64.0 69.8/80.1 69.6/80.2 69.5/80.1 69.8/80.0 69.9/80.1 5b. $++$
DA_first 14.4/29.4 12.8/26.7 15.1/30.6 18.7/35.3 20.3/36.4 26.2/44.5 24.1/40.9
27.5/45.1 38.8/55.1 40.2/55.6 5c. $++$ DA_last 14.8/30.6 18.5/28.5 15.9/31.3
17.5/34.5 17.3/33.8 29.6/47.7 27.9/45.6 31.4/48.2 38.8/55.0 36.9/53.0

Table  presents the performance of the fine-tuned SSL model on simulated test
sets derived from the LibriSpeech dataset, enhanced by various SE front-ends.
The table includes SE front-ends used for pretraining, as well as front-ends
unseen during pretraining, encompassing both time-domain and TF-domain models.
The baseline model is a HuBERT model finetuned on the LibriSpeech ls-clean-100
partition. Since the FAT-HuBERT model takes two branches as input, both
branches are fed with the same noisy waveform in the NoEnh columns. Comparing
the rows denoted as *.a with those denoted as *.b and *.c, applying fusion on
all layers generally leads to inferior performance compared to restricting
fusion to either the first or final layer. This is especially noticeable with
the DA module, possibly due to the extra parameters it introduces. Despite
residual connections are applied, these extra parameters still have an impact
on the initial parameters of the pretrained HuBERT model during continual
pretraining. Fusion at either the initial or final layer consistently
outperforms the IMST approach, highlighting the effectiveness of the fusion
module in mitigating distortions introduced by the SE front-ends. Notably, the
first-layer for OA fusion and the final-layer for DA and SF fusion yield the
best results. This distinction can be attributed to the minimal parameter
introduction of OA ($\alpha$), which has a smaller impact on the performance
of upper layers during pretraining. The benefits of the fusion module extend
beyond the training front-ends, as it demonstrates generalization capability
to unseen front-ends. Furthermore, the fusion module exhibits robust
performance even under lower SNRs than those encountered during front-end
training, indicating its ability to handle challenging conditions effectively.

[width=]imgs/wer.png

Figure : WER(%) on original test data (test clean / other) and simulated noisy
data under different SNR conditions.

##  Performance at different SNR conditions

Figure  presents the WER results of the proposed method under different SNR
conditions. The results are obtained by averaging the WER across all seven
different front-ends used for enhancement. Notably, in Figure  (a), the
original waveform is enhanced without being mixed with noise, demonstrating
the robustness of the fusion module under clean conditions. As depicted in
Figures  (c) and (d), the IMST strategy effectively enhances the model’s
performance in low SNR conditions. However, under high SNR conditions
illustrated in Figures  (a) and (b), a slight degradation in performance is
observed when IMST is applied, suggesting its limited utility in less noisy
conditions. On the other hand, the fusion module is beneficial in mitigating
the performance degradation associated with IMST under high SNR conditions.
Furthermore, it brings about additional WER reduction in low SNR conditions.

##  Results on Real-World Noisy Speech

In this section, our proposed methods are evaluated on the CHiME-4 1-channel
real noisy test sets as shown in Table . We include recently published SSL
results [wav2vec-switch, wav2vec-recons, wang2023hubert], which are also SSL
based robust ASR models.

Table : WER(%) of different systems on CHiME-4 1-channel real test sets  2*System |  2*front-end | CHiME-4 REAL
---|---|---
|  | dev | eval
Yang et al. [Yang2022ACB] |  3*N/A | 3.4 | 6.3
wav2vec-switch [wav2vec-switch] |  | 3.5 | 6.6
wav2vec (recons) [wav2vec-recons] |  | 5.0 | 9.0
5*HuBERT-AGG [wang2023hubert] |  |  |
(50k steps) | N/A | 3.3 | 6.1
| ConvTasNet | 3.4 | 6.2
| DCUNet | 3.5 | 6.4
| SKiM | 3.3 | 6.0
| BLSTM | 3.7 | 6.8
5*HuBERT | N/A | 4.4 | 8.6
| ConvTasNet | 4.6 | 8.7
| DCUNet | 4.1 | 8.3
| SKiM | 3.9 | 7.7
| BLSTM | 4.3 | 8.4
4*$+$IMST | ConvTasNet | 4.1 | 8.2
| DCUNet | 4.0 | 7.7
| SKiM | 3.8 | 7.4
| BLSTM | 4.0 | 8.1
$++$ OA_first |  3*SKiM | 3.3 | 5.9
$++$ SF_first |  | 3.1 | 5.7
$++$ DA_first |  | 3.5 | 6.3
|  |  |

Results on the CHiME-4 1-channel real-word test sets are presented in Table .
Results for HuBERT and HuBERT-AGG indicates directly prepending a SE front-end
does not necessarily improves ASR performance. The application of IMST
exhibits a consistent enhancement over the baseline. It is worth noting that
this improvement is achieved without joint tuning of the SSL backend and the
independently retrained front-ends using CHiME-4 data. Moreover, the
integration of the fusion module contributes additional enhancements, with the
SF fusion module demonstrating notable benefits.