# Some issues in robust clustering

Christian Hennig
Dipartimento di Scienze Statistiche “Paolo Fortunati”,
University of Bologna,
Via delle Belle Arti 41, 40126 Bologna, Italy,
<EMAIL_ADDRESS>
###### Abstract

Some key issues in robust clustering are discussed with focus on Gaussian
mixture model based clustering, namely the formal definition of outliers,
ambiguity between groups of outliers and clusters, the interaction between
robust clustering and the estimation of the number of clusters, the essential
dependence of (not only) robust clustering on tuning decisions, and
shortcomings of existing measurements of cluster stability when it comes to
outliers.

Keywords: Gaussian mixture model, trimming, noise component, number of
clusters, user tuning, cluster stability
MSC2010 classification: 62H30

##  1 Introduction

This is accepted for publication in P. Brito, J. G. Dias, B. Lausen, A.
Montanari, R. Nugent (eds.) Classification and Data Science in the Digital Age
(Proeedings of IFCS-2022 Porto), Springer (2023).

Cluster analysis is about finding groups in data. Robust statistics is about
methods that are not affected strongly by deviations from the statistical
model assumptions or moderate changes in a data set. Particular attention has
been paid in the robustness literature to the effect of outliers. Outliers and
other model deviations can have a strong effect on cluster analysis methods as
well. There is now much work on robust cluster analysis, see [1, 19, 9] for
overviews.

There are standard techniques of assessing robustness such as the influence
function and the breakdown point [15] as well as simulations involving
outliers, and these have been applied to robust clustering as well [19, 9].

Here I will argue that due to the nature of the cluster analysis problem,
there are issues with the standard reasoning regarding robustness and
outliers.

The starting point will be clustering based on the Gaussian mixture model, for
details see [3]. For this approach, $n$ observations are assumed i.i.d. with
density

$f_{\eta}(x)=\sum_{k=1}^{K}\pi_{k}\varphi_{\mu_{k},\Sigma_{k}}(x),$

$x\in\mathbb{R}^{p}$, with $K$ mixture components with proportions $\pi_{k}$,
$\varphi_{\mu_{k},\Sigma_{k}}$ being the Gaussian density with mean vectors
$\mu_{k}$, covariance matrices $\Sigma_{k},\ k=1,\ldots,K$, $\eta$ being a
vector of all parameters. For given $K$, $\eta$ can be estimated by maximum
likelihood (ML) using the EM-algorithm, as implemented for example in the
R-package “mclust”. A standard approach to estimate $K$ is the optimisation of
the Bayesian Information Criterion (BIC). Normally, mixture components are
interpreted as clusters, and observations $x_{i},\ i=1,\ldots,n$, can be
assigned to clusters using the estimated posterior probability that $x_{i}$
was generated by mixture component $k$. A problem with ML estimation is that
the likelihood degenerates if all observations assigned to a mixture component
lie on a lower dimensional hyperplane, i.e, a $\Sigma_{k}$ has an eigenvalue
of zero. This can be avoided by placing constraints on the eigenvalues of the
covariance matrices [8]. Alternatively, a non-degenerate local optimum of the
likelihood can be used, and if this cannot be found, constrained covariance
matrix models (such as $\Sigma_{1}=\ldots=\Sigma_{K}$) can be fitted instead,
as is the default of mclust. Several issues with robustness that occur here
are also relevant for other clustering approaches.

##  2 Outliers vs. clusters

It is well known that the sample mean and sample covariance matrix as
estimators of the parameters of a single Gaussian distribution can be driven
to breakdown by a single outlier [15]. Under a Gaussian mixture model with
fixed $K$, an outlier must be assigned to a mixture component $k$ and will
break down the estimators of $\mu_{k},\Sigma_{k}$ (which are weighted sample
means and covariance matrices) for that component in the same manner; the same
holds for a cluster mean in $k$-means clustering.

Addressing this issue, and dealing with more outliers in order to achieve a
high breakdown point, is a starting point for robust clustering. Central ideas
are trimming a proportion of observations [7], adding a “noise component” with
constant density to catch the outliers [4, 3], mixtures with more robust
component-wise estimators such as mixtures of heavy-tailed distributions (Sec.
7 of [18]).

But cluster analysis is essentially different from estimating a homogeneous
population. Given a data set with $K$ clear Gaussian clusters and standard ML-
clustering, consider adding a single outlier that is far enough away from the
clusters. Assuming a lower bound on covariance matrix eigenvalues, the outlier
will form a one-point cluster, the mean of which will diverge with the added
outlier, and the original clusters will be merged to form $K-1$ clusters [10].

The same will happen with a group of several outliers being close together,
once more added far enough away from the Gaussian clusters. “Breakdown” of an
estimator it is usually understood as the estimator becoming useless. It is
questionable that this is the case here. In fact, the “group of outliers” can
well be interpreted as a cluster in its own right, and putting all these
points together in a cluster could be seen as desirable behaviour of the ML
estimator, at least if two of the original $K$ clusters are close enough to
each other that merging them will produce a cluster that is fairly well fitted
by a single Gaussian distribution; note that the Gaussian mixture model does
not assume strong separation between components, and a mixture of two
Gaussians may be unimodal and in fact very similar to a single Gaussian. A
breakdown point larger than a given $\alpha,\ 0<\alpha<\frac{1}{2}$ may not be
seen as desirable in cluster analysis if there can be clusters containing a
proportion of less than $\alpha$ of the data, as a larger breakdown point will
stop a method from taking such clusters (when added in large distance from the
rest of the data) appropriately into account.

The core problem is that it is not clear what distinguishes a group of
outliers from a legitimate cluster. I am not aware of any formal definition of
outliers and clusters in the literature that allows this distinction. Even a
one-point cluster is not necessarily invalid. Here are some possible and
potentially conflicting aspects of such a distinction.

  * •

A certain minimum size may be required for a cluster; smaller groups of points
may be called outliers.

  * •

Groups of points in low density areas of the data may be called outliers. Note
that this particularly means that very widely spread Gaussian mixture
components would also be defined as outliers, deviating from the standard
interpretation of Gaussian mixture components as clusters.

  * •

Members of non-Gaussian mixture components may be called outliers. This does
not seem to be a good idea, because Gaussianity cannot be assessed for too
small groups of observations, and furthermore in practice model assumptions
are never perfectly fulfilled, and it may be desirable to interpret
homogeneous or unimodal non-Gaussian parts of the data as “cluster” and fit
them by a Gaussian component.

  * •

The term “outlier” suggests that outliers lie far away from most other
observations, so it may be required that outliers are farther away from the
clusters than the clusters are from each other. But this would be in conflict
with the intuition that strong separation is usually seen as a desirable
feature for well interpretable clusters. It may only be reasonable in
applications in which there is prior information that there is limited
variation even between clusters, as is implied by certain Bayesian approaches
to clustering [17].

  * •

The term “cluster” may be seen as flexible enough that a definition of an
outlier is not required. Clustering should accommodate whatever is “outlying”
by fitting it by one or more further clusters, if necessary of size one
(single linkage clustering can be useful for outlier detection, even though it
is inappropriate for most clustering problems).

Most of these items require specific decisions that cannot be made in any
objective and general manner, but only taking into account subject matter
information, such as the minimum size of valid clusters or the density level
below which observations are seen as outliers (potentially compared to density
peaks in the distribution). This implies that an appropriate treatment of
outliers in cluster analysis cannot be expected to be possible without user
tuning.

##  3 Robustness and the number of clusters

The last item suggests that there is an interplay between outlier
identification and the number of clusters, and that adding clusters might be a
way of dealing with outliers; as long as clusters are assumed to be Gaussian,
a single additional component may not be enough. More generally, concentrating
robustness research on the case of fixed $K$ may be seen as unrealistic,
because $K$ is rarely known, although estimating $K$ is a notoriously
difficult problem even without worrying about outliers [13].

The classical robustness concepts, breakdown point and influence function,
assume parameters from $\mathbb{R}^{q}$ with fixed $q$. If $K$ is not fixed,
the number of parameters is not fixed either, and the classical concepts do
not apply.

As an alternative to the breakdown point, [11] defined a “dissolution point”.
Dissolution is measured in terms of cluster memberships of points rather than
in terms of parameters, and is therefore also applicable to nonparametric
clustering methods. Furthermore, dissolution applies to individual clusters in
a clustering; certain clusters may dissolve, i.e., there may be no
sufficiently similar cluster in a new clustering computed after, e.g., adding
an outlier; and others may not dissolve. This does not require $K$ to be
fixed; the definition is chosen so that if a clustering changes from $K$ to
$L<K$ clusters, at least $K-L$ clusters dissolve.

[10, 11] showed that when estimating $K$ using the BIC and standard ML
estimation, reasonably well separated clusters do not dissolve when adding
possibly even a large percentage of outliers (this does not hold for every
method to estimate the number of clusters, see [11]). Furthermore, [11] showed
that no method with fixed $K$ can be robust for data in which $K$ is
misspecified - already [7] had found that robustness features in clustering
generally depend on the data.

An implication of these results is that even in the fixed $K$ problem, the
standard ML method can be a valid competitor regarding robustness if it comes
with a rule that allows to add one or possibly more clusters that can then be
used to fit the outliers (this is rarely explored in the literature, but [18],
Sec. 7.7, show an example in which adding a single component does not work
very well).

An issue with adding clusters to accommodate outliers is that in many
applications it is appropriate to distinguish between meaningful clusters, and
observations that cannot be assigned to such clusters (often referred to as
“noise”). Even though adding clusters of outliers can formally prevent the
dissolution of existing clusters, it may be misleading to interpret the
resulting clusters as meaningful, and a classification as outliers or noise
can be more useful. This is provided by the trimming and noise component
approaches to robust clustering. Also some other clustering methods such as
the density-based DBSCAN [5] provide such a distinction. On the other hand,
modelling clusters by heavy-tailed distributions such as in mixtures of
t-distributions will implicitly assign outlying observations to clusters that
potentially are quite far away. For this reason, [18], Sec, 7.7, provide an
additional outlier identification rule on top of the mixture fit. [6] even
distinguish between “mild” outliers that are modelled as having a larger
variance around the same mean, and “gross” outliers to be trimmed. The variety
of approaches can be connected to the different meanings that outliers can
have in applications. They can be erroneous, they can be irrelevant noise, but
they can also be caused by unobserved but relevant special conditions (and
would as such qualify as meaningful clusters), or they could be valid
observations legitimately belonging to a meaningful cluster that regularly
produces observations further away from the centre than modelled by a Gaussian
distribution.

Even though currently there is no formal robustness property that requires
both the estimation of $K$ and an identification or downweighting of outliers,
there is demand for a method that can do both.

Estimating $K$ comes with an additional difficulty that is relevant in
connection with robustness. As mentioned before, in clustering based on the
Gaussian mixture model normally every mixture component will be interpreted as
a cluster. In reality, however, meaningful clusters are not perfectly
Gaussian. Gaussian mixtures are very flexible for approximating non-Gaussian
distributions. Using a consistent method for estimating $K$ means that for
large enough $n$ a non-Gaussian cluster will be approximated by several
Gaussian mixture components. The estimated $K$ will be fine for producing a
Gaussian mixture density that fits the data well, but it will overestimate the
number of interpretable clusters. The estimation of $K$, if interpreted as the
number of clusters, relies on precise Gaussianity of the clusters, and is as
such itself riddled with a robustness problem; in fact slightly non-Gaussian
clusters may even drive the estimated $K\to\infty$ if $n\to\infty$ [12, 14].

This is connected with the more fundamental problem that there is no unique
definition of a cluster either. The cluster analysis user needs to specify the
cluster concept of interest even before robustness considerations, and
arguably different clustering methods imply different cluster concepts [13]. A
Gaussian mixture model defines clusters by the Gaussian distributional shape
(unless mixture components are merged to form clusters [12]). Although this
can be motivated in some real situations, robustness considerations require
that distributional shapes fairly close to the Gaussian should be accepted as
clusters as well, but this requires another specification, namely how far from
a Gaussian a cluster is allowed to be, or alternatively how separated Gaussian
components have to be in order to count as separated clusters. A similar
problem can also occur in nonparametric clustering; if clusters are associated
with density modes or level sets, the cluster concept depends on how weak a
mode or gap between high level density sets is allowed to be to be treated as
meaningful.

[14] propose a parametric bootstrap approach to simultaneously estimate $K$
and assign outliers to a noise component. This requires two basic tuning
decisions. The first one regards the minimum percentage of observations so
that a researcher is willing to add another cluster if the noise component can
be reduced by this amount. The second one specifies a tolerance that allows a
data subset to count as a cluster even though it deviates to some extent from
what is expected under a perfectly Gaussian distribution. There is a third
tuning parameter that is in effect for fixed $K$ and tunes how much of the
tails of a non-Gaussian cluster can be assigned to the noise in order to
improve the Gaussian appearance of the cluster. One could even see the
required constraints on covariance matrix eigenvalues as a further tuning
decision. Default values can be provided, but situations in which matters can
be improved deviating from default values are easy to construct.

##  4 More on user tuning

User tuning is not popular, as it is often difficult to make appropriate
tuning decisions. Many scientists believe that subjective user decisions
threaten scientific objectivity, and also background knowledge dependent
choices cannot be made when investigating a method’s performance by theory and
simulations. The reason why user tuning is indispensable in robust cluster
analysis is that it is required in order to make the problem well defined. The
distinction between clusters and outliers is an interpretative one that no
automatic method can make based on the data alone. Regarding the number of
clusters, imagine two well separated clusters (according to whatever cluster
concept of interest), and then imagine them to be moved closer and closer
together. Below what distance are they to be considered a single cluster? This
is essentially a tuning decision that the data cannot make on their own.

There are methods that do not require user tuning. Consider the mclust
implementation of Gaussian mixture model based clustering. The number of
clusters is by default estimated by the BIC. As seen above, this is not really
appropriate for large data sets, but its derivation is essentially asymptotic,
so that there is no theoretical justification for it for small data sets
either. Empirically it often but not always works well, and there is little
investigation of whether it tends to make the “right” decision in ambiguous
situations where it is not clear without user tuning what it even means to be
“right”. Covariance matrix constraints in mclust are not governed by a tuning
of eigenvalues or their ratios to be specified by the user. Rather the BIC
decides between different covariance matrix models, but this can be erratic
and unstable, as it depends on whether the EM-algorithm gets caught in a
degenerate likelihood maximum or not, and in situations where two or more
covariance matrix models have similar BIC values (which happens quite often),
a tiny change in the data can result in a different covariance matrix model
being selected, and substantial changes in the clustering. A tunable
eigenvalue condition can result in much smoother behaviour. When it comes to
outlier identification, mclust offers the addition of a uniform “noise”
mixture component governed by the range of the data, again supposedly without
user tuning. This starts from an initial noise estimation that requires tuning
(Sec. 3.1.2 of [3]) and is less robust in terms of breakdown and dissolution
than trimming and the improper noise component, both of which require tuning
[10, 11]. The ICL, an alternative to the BIC (Sec. 2.6 of [3]), on the other
hand, is known to merge different Gaussian mixture components already at a
distance at which they intuitively still seem to be separated clusters.
Similar comments apply to the mixture of t-distributions; it requires user
tuning for identifying outliers, scatter matrix constraints, and it has the
same issues with BIC and ICL as the Gaussian mixture.

Summarising, both the identification of and robustness against outliers and
the estimation of the number of clusters require tuning in order to be well
defined problems; user tuning can only be avoided by taking tuning decisions
out of the user’s hands and making them internally, which will work in some
situations and fail in others, and the impression of automatic data driven
decision making that a user may have is rather an illusion. This, however,
does not free method designers from the necessity to provide default tunings
for experimentation and cases in which the users do not feel able to make the
decisions themselves, and tuning guidance for situations in which more
information is available. A decision regarding the smallest valid size of a
cluster is rather well interpretable; a decision regarding admissible
covariance matrix eigenvalues is rather difficult and abstract.

##  5 Stability measurement

Robustness is closely connected to stability. Both experimental and
theoretical investigation of the stability of clusterings require formal
stability measurements, usually comparing two clusterings on the same data
(potentially modified by replacing or adding observations). Not assuming any
parametric model, proximity measures such as the Adjusted Rand Index (ARI;
[16]), the Hamming distance (HD; [2]), or the Jaccard distance between
individual clusters [11] can be used. Note that [2], standard reference on
cluster stability in the machine learning community, state that stability and
instability are caused in the first place by ambiguities in the cluster
structure of the data, rather than by a method’s robustness or lack of it.
Although the outlier problem is ignored in that paper, it is true that cluster
analysis can have other stability issues that are as serious as or worse than
gross outliers.

To my knowledge, none of the measures currently in use allow for a special
treatment of a set of outliers or noise; either these have to be ignored, or
treated just as any other cluster. Both ARI and HD, comparing clusterings
$\mathcal{C}_{1}$ and $\mathcal{C}_{2}$, consider pairs of observations
$x_{i},x_{j}$ and check whether those that are in the same cluster in
$\mathcal{C}_{1}$ are also in the same cluster in $\mathcal{C}_{2}$. An
appropriate treatment of noise sets
$N_{1}\in\mathcal{C}_{1},N_{2}\in\mathcal{C}_{2}$ would require that
$x_{i},x_{j}\in N_{1}$ are not just in the same cluster in $\mathcal{C}_{2}$
but rather in $N_{2}$, i.e., whereas the numberings of the regular clusters do
not have to be matched (which is appropriate because cluster numbering is
meaningless), $N_{1}$ has to be matched to $N_{2}$. Corresponding re-
definitions of these proximities will be useful to robustness studies.

##  6 Conclusion

Key practical implications of the above discussions are:

  * •

Outliers can be treated as forming their own clusters, or be collected in
outlier/noise or trimmed sets, or be integrated in clusters of non-outliers.
Which of these is appropriate depends on the nature of outliers in a given
application.

  * •

Methods that do not identify outliers but add clusters in order to accommodate
them are valid competitors of robust clustering methods, as are nonparametric
density-based methods.

  * •

Cluster analysis involving estimating the number of clusters and robustness
require tuning in order to define the problem they are meant to solve well.
Method developers need to provide sensible defaults, but also to guide the
users regarding a meaningful interpretation of the tuning decisions.

## References

  * [1] Banerjee, A., Davé, R. N.: Robust clustering. WIREs Data Mining Knowl. Discov. 2, 29–59 (2012)
  * [2] Ben-David, S., von Luxburg, U., Pál, D.: A sober look at clustering stability. In: Proceedings of the 19th annual conference on Learning Theory (COLT’06), pp. 5-19, Springer, Berlin (2006)
  * [3] Bouveyron, C., Celeux, G., Murphy, T. B., Raftery, A. E.: Model-based Clustering and Classification for Data Science. Cambridge University Press, Cambridge MA (2019)
  * [4] Coretto, P., Hennig, C.: Consistency, breakdown robustness, and algorithms for robust improper maximum likelihood clustering. J. Mach. Learn. Res.18, 1–39.
  * [5] Ester, M., Kriegel, H. P., Sander, J., Xu, X.: A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In: Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, pp. 226-231, AAAI Press, Portland OR (1996)
  * [6] Farcomeni, A., Punzo, A.: Robust model-based clustering with mild and gross outliers. TEST 29, 989–1007 (2020)
  * [7] García-Escudero, L. A., Gordaliza, A.: Robustness Properties of k-Means and Trimmed k-Means. J. Am. Stat. Assoc. 94, 956–969 (1999)
  * [8] García-Escudero, L. A., Gordaliza, A., Greselin, F., Ingrassia, S., Mayo-Iscar, A.: Eigenvalues and constraints in mixture modeling: geometric and computational issues. Adv. Data Anal. Classi. 12, 203–233 (2018)
  * [9] García-Escudero, L. A., Gordaliza, A., Matrán, C., Mayo-Iscar, A., Hennig, C.: Robustness and Outliers. In: Hennig, C., Meila, M., Murtagh, F., Rocci, R. (eds.) Handbook of Cluster Analysis, pp. 653-678. Chapman & Hall/CRC, Boca Raton FL (2016)
  * [10] Hennig, C.: Breakdown points for maximum likelihood estimators of location-scale mixtures. Ann. Stat. 32, 1313–1340 (2004)
  * [11] Hennig, C.: Dissolution point and isolation robustness: robustness criteria for general cluster analysis methods. J. Multivariate Anal. 99, 1154–1176 (2008)
  * [12] Hennig, C.: Methods for merging Gaussian mixture components. Adv. Data Anal. Classi. 4, 3–34 (2010)
  * [13] Hennig, C.: Clustering Strategy and Method Selection. In: Hennig, C., Meila, M., Murtagh, F., Rocci, R. (eds.) Handbook of Cluster Analysis, pp. 703-730. Chapman & Hall/CRC, Boca Raton FL (2016)
  * [14] Hennig, C., Coretto, P.: An adequacy approach for deciding the number of clusters for OTRIMLE robust Gaussian mixture-based clustering. Aust. N. Z. J. Stat. (2021) doi: 10.1111/anzs.12338
  * [15] Huber, P. J., Ronchetti, E. M.: Robust Statistics (2nd ed.). Wiley, Hoboken NJ (2009)
  * [16] Hubert, L., Arabie, P.: Comparing partitions. J. Classif. 2, 193–218 (1985)
  * [17] Malsiner-Walli, G., Frühwirth-Schnatter, S., Grün, B.: Identifying mixtures of mixtures using Bayesian estimation. J. Comput. Graph. Stat. 26, 285–295 (2017)
  * [18] McLachlan, G. J., Peel, D.: Finite Mixture Models. Wiley, New York (2000)
  * [19] Ritter, G.: Robust Cluster Analysis and Variable Selection. Chapman & Hall/CRC, Boca Raton FL (2015)


# Virtual Guidance as a Mid-level Representation for Navigation

Hsuan-Kung Yang1, Tsung-Chih Chiang1*, Ting-Ru Liu1*, Chun-Wei Huang1*, Jou-
Min Liu1*, and Chun-Yi Lee1 * indicates equal contribution.1 Elsa Lab,
Department of Computer Science, National Tsing Hua University, Hsinchu City,
Taiwan.This work has been submitted to the IEEE for possible publication.
Copyright may be transferred without notice, after which this version may no
longer be accessible.

###### Abstract

In the context of autonomous navigation, effectively conveying abstract
navigational cues to agents in dynamic environments poses challenges,
particularly when the navigation information is multimodal. To address this
issue, the paper introduces a novel technique termed “Virtual Guidance,” which
is designed to visually represent non-visual instructional signals. These
visual cues, rendered as colored paths or spheres, are overlaid onto the
agent’s camera view, serving as easily comprehensible navigational
instructions. We evaluate our proposed method through experiments in both
simulated and real-world settings. In the simulated environments, our virtual
guidance outperforms baseline hybrid approaches in several metrics, including
adherence to planned routes and obstacle avoidance. Furthermore, we extend the
concept of virtual guidance to transform text-prompt-based instructions into a
visually intuitive format for real-world experiments. Our results validate the
adaptability of virtual guidance and its efficacy in enabling policy transfer
from simulated scenarios to real-world ones.

##  I INTRODUCTION

Biological instincts are highly attuned to visual cues, a capability that is
of paramount importance for executing navigation tasks effectively. For
instance, Google Maps employs a combination of arrows and augmented reality to
guide users to their desired destination. In contrast, navigational guidelines
presented in textual or numerical formats, such as ‘turn 15 degrees at the
next building’ or ‘proceed for 500 meters,’ are less straightforward and
necessitate the high-level abstract information to be encoded into a form that
can more easily be comprehended. Although certain studies have endeavored to
transform these abstract navigational instructions into intermediate guidance
signals (e.g., locational waypoints) that provide agents with spatial or
directional cues [1, 2, 3], these often remain counterintuitive for model-free
agents to easily grasp the environmental dynamics for effective navigation.
This inadequacy becomes particularly pronounced when considering environments
characterized by dynamic variables, such as moving objects, uncertainties, or
other evolving conditions, where mere reliance on spatial or directional cues
is inadequate. Moreover, the integration of disparate modalities (e.g., image
observations and instructions) remains non-trivial due to the inherent
challenges in reconciling the information derived from different modalities.

Figure 1: Demonstration of (a) the concept of virtual guidance, and (b) the
agent’s behaviors in real-world scenarios in
$\\{\scalerel*{\includegraphics{emojis/xxx.jpeg}}{X},\scalerel*{\includegraphics{emojis/green_box.png}}{X},\scalerel*{\includegraphics{emojis/blue_box.png}}{X}\\}$
and
$\\{\scalerel*{\includegraphics{emojis/blue_umbrella.png}}{X},\scalerel*{\includegraphics{emojis/purple_umbrella.png}}{X}\\}$
scenarios.

Despite the aforementioned challenges, there is still often a necessity for
autonomous agents in visual navigation tasks to manage multiple modalities.
These modalities are essential for both local control mechanisms and long-term
planning strategies. Many previous navigation tasks have employed hierarchical
frameworks to handle these diverse modalities. For example, the planning
component may rely on non-visual cues such as LiDARr, GPS, or text
instructions, while the local controller utilizes visual observations. Recent
years have seen considerable success in integrating instructions into robot
navigation tasks through such hierarchical methods [4, 5, 6, 7, 8, 9, 10, 11,
12, 13]. However, such approaches necessitate that the agent first learn a
mapping mechanism to correlate navigation information with observed images. In
light of the extensive research on convolutional neural networks (CNNs) and
their demonstrated efficacy in identifying complex interrelationships within
an image, a potential promising avenue in the realm of visual navigation could
lie in the direct representation of navigation instructions on visual
perceptions. This could alleviate the cognitive burden on agents by rendering
non-visual, abstract modalities more easily comprehensible.

Figure 2: Left: The overview of the simulation framework. Right: The hardware
configuration used in our real-world experiments.

Inspired by [14, 15, 16, 17, 18], we propose a method of representation for
non-visual instructional or modal signals as visual format, which we term
’virtual guidance.’ The virtual guidance signals, rendered as either colored
paths or spheres, are superimposed on the semantic segmentation derived from
the agent’s camera view, with the goal of guiding the agent toward a specific
direction, as illustrated in Fig. 1 (a). This concept bears similarities to
technologies such as Google Maps’ Line View, which incorporates Augmented
Reality guidance based on Global Positioning Systems (GPS) or Visual
Positioning Systems (VPS). To evaluate the effectiveness of the virtual
guidance schemes, we developed a framework that comprises a planning module
and a virtual guidance representation module. Our objective is to assess the
efficacy of different kinds of vision-based virtual guidance schemes in
comparison to the hybrid approach that concatenate segmentation with vector-
based, non-visual guidance signals.

To validate our method, we provide simulated and real-world evaluations. For
the former, we developed a configurable a virtual city and multiple evaluation
scenarios using Unity [19]. Such virtual environments enable the rendering of
virtual guidance signals that are absent in existing off-the-shelf simulation
platforms [20]. Our environments comprise dynamic objects moving at
configurable speeds, and the agent is required to navigate while avoiding
obstacles. We evaluated diverse experimental settings and observed that
rendering guidance signals in the agent’s observations leads to superior
performance in both adhering to designated routes and reaching the
destination, compared to the hybrid guidance approach. Specifically, we
compared the capabilities of different guidance schemes in seen and unseen
conditions. Such examination offered empirical substantiation that that our
proposed method excels in generalizability and obstacle avoidance
proficiencies. Through impact, failure case, and trajectory analyses, we
established that the agents trained under this virtual guidance paradigm
outperformed those trained under the hybrid approach. For real-world
evaluations, we extend the concept of virtual guidance to transform
instructions from another modality into a more comprehensible form.
Specifically, we employ text prompts to facilitate object detection through
GroundingDINO [21], an open-vocabulary object detection model capable of zero-
shot object detection. This process is illustrated in Fig. 1. Using the
objects identified from the text prompts, we then generate virtual guidance
and overlay it onto the agent’s observations. Our experiments validate the
adaptability of the virtual guidance concept and the feasibility of
transferring policies pre-trained in virtual scenarios to real-world ones.

##  II Related Work

###  II-A Navigation and Guidance Mechanisms

Several navigation frameworks have been developed to direct an agent toward
specified destinations. One common branch involves leveraging hierarchical
frameworks to transmit navigational instructions or information to the agent
[4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. Such non-vision-based guidance mechanisms
can broadly be categorized as (1) implicit guidance and (2) vector-based
guidance. The former offers the agent spatial knowledge through environmental
cues, such as a localized map [22, 13, 23] or a set of waypoints [24, 8]. In
contrast, the latter instructs the agent on specific actions or orientations
to adopt [12]. To elaborate on this latter category, previous research
endeavors have experimented with giving agents directional guidance toward
targets or waypoints using non-visual mechanisms [25, 12, 26, 11]. Beyond
these two categories, recent literature has also delved into object-based
navigation, where the agent is guided by images of the target object or area
[27]. Some other research efforts represent waypoints as a sequence and
instruct the agent to reach them in an ordered fashion [22, 25, 26, 28, 24,
29, 23, 30, 13], or through text-based instructions in vision-language
frameworks. In this paper, the focus lies on explicit guidance realized
through visual rendering of guidance signals. Unlike previous methodologies
that concatenate observations with various modalities, our approach
incorporates path or waypoint guidance directly into the agent’s observations.

###  II-B Mid-Level Representation based Navigation

To implement virtual guidance, this paper explores approaches for the
representation of virtual guidance signals based on mid-level representations.
Mid-level representations are abstract concepts that capture physical or
semantic meanings, and are typically domain-invariant properties extracted
from visual scenes. Such representations have found applications in robotics
for conveying information from perception modules to control modules [18, 31].
These mid-level representations can assume various forms, such as depth maps,
optical flow, and semantic segmentation, each possessing unique strengths and
weaknesses in different scenarios [17]. A comprehensive, expressive, and
interpretable mid-level representation is essential for the success of
modular, learning-based frameworks. Existing research on navigation based on
mid-level representations has primarily focused on facilitating obstacle
avoidance or random path following for robotic agents, often in the absence of
explicit instructional or guidance signals on direction or path [18, 17]. As a
result, this study aims to extend this domain by introducing virtual guidance
as a new form of mid-level representation.

##  III Virtual Guidance in Simulated Environments

Figure 3: The overview of different types of virtual guidance schemes compared
to the vector based one. Figure 4: The overview of the framework for deploying
and validating the concept of virtual guidance in real-world scenarios.

###  III-A An Overview of the Simulation Framework

To investigate the feasibility of virtual guidance as a form of mid-level
representation, we have developed a flexible framework using the Unity engine
[19] and the Unity ML-Agents Toolkit [32]. This framework, illustrated in Fig.
2, is designed to be fully configurable and enables the generation of guidance
signals. The inputs of the agent are rendered in the form of semantic
segmentation, an effective mid-level representation that can serve as an input
observation for a Deep Reinforcement Learning (DRL) agent. The employment of
semantic segmentation also facilitates transfer from virtual to real-world
scenarios [18, 31]. The framework permits customization of various
experimental settings, including different navigation paths and a range of
training and evaluation environments featuring both static and dynamic
objects. This design philosophy enables the exploration of diverse ways of
presenting guidance signals to the agent, as well as various virtual guidance
and vector-based approaches. The agent receives the guidance signals along
with stacked semantic segmentation maps, and is tasked with processing these
inputs to learn a policy to reach its intended destination.

###  III-B Virtual Guidance Generation Workflow

This section presents the workflow for generating virtual guidance, which
consists of two primary components: (a) a planning module responsible for
determining the navigation trajectory, and (b) a virtual guidance
representation module tasked with rendering the virtual guidance for the DRL
agent.

####  III-B1 Planning Module

The function of the planning module is to facilitate the validation of the
effectiveness of different guidance schemes. In the simulated environments, we
assume that the planning module possesses awareness of the key locational
parameters, i.e., the starting point $S$, the destination $D$, as well as the
agent’s position $P_{t}$ at timestep $t$. This is to mitigate any potential
errors in the localization process. It should be noted that in real-world
scenarios, the planned trajectory can be generated in various ways, which is
elaborated upon in the subsequent section. Under this assumption, a navigation
trajectory can be planned between any two points, whether it be from $S$ to
$D$ or $P_{t}$ to $D$. The framework allows for configurability in the choice
of planning algorithm, and the A* algorithm [33] is employed in the
experiments of this study. The planned trajectory can either be rendered as a
vision-based virtual guidance signal or transformed into vectors for the agent
to interpret. The navigation trajectory can be generated in real-time between
the agent’s current position $P_{t}$ and its destination $D$. While real-time
planning offers advantages such as preventing the agent from losing track of
its current position, it might not be feasible and practical under
computational constraints [12]. In the simulated environments, we investigate
two distinct configurations for the planning module to generate the navigation
trajectory, including: (a) real-time planning between the DRL agent’s current
position $P_{t}$ and the destination $D$ at every timestep $t$, and (b) one-
time planning from the starting point $S$ to the destination $D$ at the
beginning of navigation.

####  III-B2 Virtual Guidance Representation Module

Once the navigation trajectory is obtained from the planning module, virtual
guidance can be generated through various approaches and rendered on semantic
segmentation to enable the agent to recognize and interpret its semantic
meaning. Different types of virtual guidance representations on semantic
segmentation carry distinct visual meanings and may influence the agent’s
learning and evaluation behaviors in different ways. The proposed virtual
guidance representation schemes are depicted in Fig. 3 and are elaborated as
follows.

##### Navigation Path

In the first scheme, the navigation line obtained from the planning module is
represented as a colored path on the semantic segmentation map. An example
visualization is illustrated in Fig. 3 (c). Specifically, the navigation path
is implemented as a 3D mesh in the simulated environments and projected onto
the camera view plane. This rendered navigation path can be considered as a
rich and informative signal that carries both semantic and guidance
information. It highlights the permissible regions for the DRL agent and the
route leading to the target location.

##### Waypoint

The second scheme generates a set of waypoints $\mathcal{W}$ by segmenting the
planned navigation trajectory, where different waypoints are spaced with a
regular distance from $S$ to $D$. The waypoints serve as hints to instruct the
agent to the destination. These waypoints are visualized as 3D virtual balls
in the virtual environments and are projected onto the camera image plane. The
visualization is presented in Fig. 3 (d). Unlike the first scheme, which
utilizes a navigation path to provide dense and informative signals, the
second one provides the waypoints as 3D virtual balls, which are sparse
signals for the DRL agent to locate.

##  IV Virtual Guidance in Real-World Scenarios

In this section, we describe the methodology for representing virtual guidance
in real-world settings, specifically through the use of text prompts. This
demonstrates the flexible nature of our proposed approach, which can
accommodate different modalities for generating virtual guidance.

###  IV-A An Overview of the Real-World Framework

To further validate the applicability of the proposed virtual guidance scheme
in real-world scenarios, we have designed a specific task. The objectives of
this task are twofold. First, the task seeks to verify the effective
transferability of the pre-trained DRL agent’s policy to real-world settings,
where it follows the virtual guidance. Second, it aims to demonstrate that the
virtual guidance scheme possesses the flexibility to adapt not only to
trajectories generated by specific planning algorithms but also to
instructions from diverse methods, provided these instructions can be
translated into visual representations. This adaptability highlights the
advantage of the virtual guidance scheme in eliminating the necessity for the
agent to interpret inputs from multiple modalities. In the experiment, we
provided a text prompt describing the appearance of the object to be
navigated. We then assessed the agent’s performance based on its ability to
reach the correct target object as described in the text prompt. The framework
for this experiment is illustrated in Fig. 4. Within this framework, we
employed GroundingDINO [21], an open-vocabulary object detection model capable
of zero-shot object detection, to generate detection results based on the
given text prompt. The advantage of GroundingDINO, which employs a text
encoder based on Bert-base [34], resides in its generalizability to detect a
wide array of objects, a benefit derived from leveraging large-scale language
models. The detection results are subsequently fed into the virtual guidance
representation module described in Section III-B, where the objects detected
according to the text prompt serve as guidance signals for the agent. These
signals can be rendered in two different formats, similar to those used in
simulated environments: (a) waypoints, or (b) a navigation path directed
toward these waypoints The DRL agent, which is pre-trained in our simulated
environments, employs its learned policy to follow these guidance signals and
control the robotic agent. In our experiments, the robot platform used is a
ClearPath Husky Unmanned Ground Vehicle (UGV), and the semantic segmentation
model employed is DACS [35].

##  V Experimental Results

In this section, we present the experimental results and discuss their
implications. We first describe the experimental setups, followed by a
comparison of various guidance schemes and a discussion of their impact. We
then investigate the causes of failure cases and present the trajectories
followed by the agents. Finally, we showcase a real-world experiment to
demonstrate the innovative capability of generating virtual guidance from text
prompts, as well as the successful transfer of the DRL agent’s policy from
virtual to real environments.

Figure 5: An overview of the designed simulated environment, where the green
and red lines represent the walking paths of the pedestrians.

###  V-A Experimental Setup

####  V-A1 Environment Setup

To evaluate the efficacy of virtual guidance, we developed a virtual
environment using Unity [19] and ML-Agents [32]. This environment was designed
to emulate an urban landscape with eight intersections, as depicted in Fig. 5.
The environment offers configurable starting and ending locations to create
diverse routes for both training and evaluation of the agents. In addition, it
incorporates dynamic objects with adjustable speeds, posing as obstacles that
the agent needs to avoid while adhering to the provided virtual guidance. For
the training phase, we employed a set of 89 routes. During the evaluation
phase, we evaluated the agents under two separate scenarios: (a) seen routes
and (b) unseen routes. The seen scenario focuses on the agent’s capacity to
follow virtual guidance and reach the destination without colliding with any
obstacles. This set includes all 89 routes used during training. On the other
hand, the unseen scenario aims to assess the agent’s ability to adapt while
navigating unfamiliar, novel routes with the support of virtual guidance. This
scenario incorporates combinations of starting and ending points that were not
included in the training phase, featuring four routes specifically selected
for this purpose. This is essential for validating the applicability of the
proposed method, as real-world scenarios could involve arbitrary planned
routes, and it is not feasible to cover all possible routes during training.

TABLE I: An impact analysis of the different navigation guidance schemes
discussed in this study.

| Guidance Scheme | Representation Form | Performance | Failure Cases
---|---|---|---|---
|  SPL ($\uparrow$) |  Success Rate ($\uparrow$) |  Line Following Rate ($\uparrow$) |  Waypoint Collecting Rate ($\uparrow$) |  Collision Rate ($\downarrow$) |  Out-of-Bound ($\downarrow$)
Seen |  $Hybrid$ (one-time) | {RGB, $(r,\theta)$} | $44.88\text{\,}\mathrm{\char 37\relax}$ | $46.93\text{\,}\mathrm{\char 37\relax}$ | $36.35\text{\,}\mathrm{\char 37\relax}$ | $26.36\text{\,}\mathrm{\char 37\relax}$ | $26.65\text{\,}\mathrm{\char 37\relax}$ | $26.43\text{\,}\mathrm{\char 37\relax}$
$VG_{waypoint}$ (one-time) | RGB | $73.16\text{\,}\mathrm{\char 37\relax}$ | $73.51\text{\,}\mathrm{\char 37\relax}$ | $69.54\text{\,}\mathrm{\char 37\relax}$ | $73.53\text{\,}\mathrm{\char 37\relax}$ | $19.41\text{\,}\mathrm{\char 37\relax}$ | $6.95\text{\,}\mathrm{\char 37\relax}$
$VG_{path}$ (one-time) | RGB | $72.68\text{\,}\mathrm{\char 37\relax}$ | $72.75\text{\,}\mathrm{\char 37\relax}$ | $89.46\text{\,}\mathrm{\char 37\relax}$ | ✗ | $26.26\text{\,}\mathrm{\char 37\relax}$ | $0.96\text{\,}\mathrm{\char 37\relax}$
$VG_{path}$ (real-time) | RGB | $88.85\text{\,}\mathrm{\char 37\relax}$ | $89.54\text{\,}\mathrm{\char 37\relax}$ | ✗ | ✗ | $9.02\text{\,}\mathrm{\char 37\relax}$ | $1.44\text{\,}\mathrm{\char 37\relax}$
Unseen |  $Hybrid$ (one-time) | {RGB, $(r,\theta)$} | $18.96\text{\,}\mathrm{\char 37\relax}$ | $20.86\text{\,}\mathrm{\char 37\relax}$ | $35.33\text{\,}\mathrm{\char 37\relax}$ | $20.33\text{\,}\mathrm{\char 37\relax}$ | $37.02\text{\,}\mathrm{\char 37\relax}$ | $42.12\text{\,}\mathrm{\char 37\relax}$
$VG_{waypoint}$ (one-time) | RGB | $57.69\text{\,}\mathrm{\char 37\relax}$ | $58.18\text{\,}\mathrm{\char 37\relax}$ | $67.77\text{\,}\mathrm{\char 37\relax}$ | $66.06\text{\,}\mathrm{\char 37\relax}$ | $30.11\text{\,}\mathrm{\char 37\relax}$ | $11.13\text{\,}\mathrm{\char 37\relax}$
$VG_{path}$ (one-time) | RGB | $57.46\text{\,}\mathrm{\char 37\relax}$ | $57.54\text{\,}\mathrm{\char 37\relax}$ | $89.19\text{\,}\mathrm{\char 37\relax}$ | ✗ | $36.94\text{\,}\mathrm{\char 37\relax}$ | $5.45\text{\,}\mathrm{\char 37\relax}$
$VG_{path}$ (real-time) | RGB | $81.48\text{\,}\mathrm{\char 37\relax}$ | $82.74\text{\,}\mathrm{\char 37\relax}$ | ✗ | ✗ | $15.08\text{\,}\mathrm{\char 37\relax}$ | $1.99\text{\,}\mathrm{\char 37\relax}$

####  V-A2 Agent Setup

In our experiments, we implemented the DRL agent using a Deep Neural Network
(DNN) and trained it with the Soft Actor-Critic (SAC) algorithm [36, 37]. The
agent’s observation space consists of three stacked semantic segmentation
frames, each having dimensions of $84\times 180$. These frames can be rendered
either with or without virtual guidance. The frames are resized using bilinear
downscaling, based on the outputs from the semantic segmentation model. The
agent’s action space is defined by a set $\mathcal{A}$, which includes two
primary actions: NOOP and $\textbf{TURN}(\alpha)$. The NOOP action maintains
the agent’s current directional orientation and advances it along a straight
trajectory. On the other hand, the $\textbf{TURN}(\alpha)$ action allows for
incremental orientation adjustments, where the angle $\alpha$ dictates the
degree of the turn. The sign of $\alpha$ is essential: negative values result
in a leftward adjustment, while positive values prompt a rightward shift. This
mechanism endows the agent with the ability to navigate and execute turns in a
non-binary manner. The angular velocity $\omega$, influenced by these
continuous adjustments to $\alpha$, is formulated as the following:

$\omega\mathrel{+}=\alpha\times\kappa\times\Delta t,$ (1)

where $\Delta t$ represents the time interval, and $\kappa$ is the steering
sensitivity. We set $\alpha$ to a standard value of
$35\text{\,}\mathrm{\SIUnitSymbolDegree}\mathrm{/}\mathrm{s}^{2}$ and $\kappa$
to two. Rather than relying on abrupt, binary changes in direction, the
agent’s trajectory evolves based on the cumulative effects of successive
adjustments to $\alpha$. Although the agent maintains a consistent velocity
$v$ of $6\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$, its orientation
continuously experiences refined and gradual modifications.

####  V-A3 Reward Function

In the training phase of our experiments, we use the reward function that
contains two terms: navigation following reward $R_{nav}$ and goal reward
$R_{goal}$. The navigation following reward $R_{nav}$ is designed for
encouraging the agent to follow the virtual guidance. For the waypoints
scheme, the agent receives $5.0$ when reaching any waypoint. While in the
navigation line scheme, the reward is calculated based on the shortest
distance between the agent’s position and the navigation line, and is
formulated as follows:

$\footnotesize
R_{nav}=\begin{cases}\min(\max(6-dist(P_{t},d^{\prime}),0.4)&\text{if on the
navigation line,}\\\ -0.2&\text{otherwise},\end{cases}$ (2)

where $d^{\prime}$ represents the shorted distance between the agent’s
position $P_{t}$ with the navigation line at timestep $t$. For the goal reward
$R_{goal}$, the agents receives $10.0$ when reaching the destination, and
$-10.0$ if the agent collides with any obstacle, moves out the boundaries, or
exceeds the time horizon. The final reward function $R$ is calculated as
$R=R_{nav}+R_{goal}$.

####  V-A4 Baseline

To evaluate the effectiveness of the vision-based virtual guidance
representation schemes described in Section III-B, we introduce a hybrid
approach as a comparative baseline (denoted as ‘Hybrid’). This approach
combines stacked segmentation observations with vector-based guidance to serve
as the agent’s inputs. In the vector provided to the agent, two critical
parameters are encapsulated: the distance $r$ to the closest forthcoming
waypoint $\omega^{*}$ and the orientation $\Theta$, which characterizes the
agent’s heading with respect to $\omega^{*}$. These are presented in polar
coordinates as $(r,\Theta)$. The orientation $\Theta$ falls within the range
of -180 to 180 degrees, i.e., $\Theta\in[-180^{\circ},180^{\circ}]$. It should
be noted that $\Theta$ is normalized to a range of $[-1,1]$ before being
offered to the agent. An example of this is visualized in Fig. 3 (e).

####  V-A5 Evaluation Metrics

In our experiments, four metrics are utilized to evaluate the agent’s
performance from different perspectives. These four metrics are described as
follows.

##### Success rate

In our evaluations, the success rate is utilized as a metric to assess the
proficiency of the agent in reaching the designated destination.

##### Success rate weighted by path length (SPL)

The SPL metric evaluates the agent’s navigational performance by accounting
for both the success in reaching the destination and the efficiency of the
selected trajectory [38]. This comparison ensures a consideration of both
navigational success and efficiency. SPL is mathematically represented as
follows:

$\frac{1}{N}\sum_{i=1}^{N}\frac{l_{i}}{\max(l_{i},p_{i})},$ (3)

where $l_{i}$ represents the shortest path distance from the agent’s starting
position to the goal for episode $i$, and $p_{i}$ denotes the length of the
path actually taken by the agent in that episode.

##### Line following rate

This metric evaluates the agent’s proficiency in adhering to the virtual
guidance path by calculating the overlapping ratio between the actual
trajectory of the agent and the planned virtual guidance path.

##### Waypoints collecting rate

This metric calculates the ratio between the number of encountered waypoints
and the total number of existing waypoints along the planned path.

###  V-B Impact Analysis of Virtual Guidance

This section presents a comparison of different guidance schemes. The virtual
guidance schemes were previously discussed in Section III-B, while the
baseline guidance scheme was discussed in Section V-A4. The evaluation
results, which include both the seen and unseen scenarios, are presented in
Table I. It is worth noting that agents trained with waypoints are granted
more flexibility in their line following rate, as they can take any trajectory
between two different waypoints. We evaluate the real-time and one-time
planning settings described in Section III-B1. The evaluation results
presented in Table I indicate that the schemes incorporating virtual guidance
(denoted as ‘$VG$’) consistently outperform the baseline scheme (denoted as
‘$Hybrid$’) in terms of SPL, success rate, line following rate, and waypoint
collection rate, in both seen and unseen scenarios. This suggests that our
vision-based guidance strategies can effectively offer informative
navigational cues. Therefore, it is able to alleviate the agent’s burden to
learn the correlation between visual observations and navigational
instructions derived from different non-visual modalities.

Figure 6: Trajectory analysis comparing the navigation paths planned by the
planning module with the actual paths navigated by the agent. The yellow balls
denote the waypoints, the blue lines represent the one-time planned navigation
paths, and the pink lines correspond to the actual trajectories. TABLE II:
Compilation of text prompts employed in the real-world navigation experiments
for validating virtual guidance.

Object Compositions | Text Prompts | Approaches | Success Rate
---|---|---|---
{*X, *X} | {‘purple umbrella’ & ‘blue umbrella’} | $VG_{waypoints}$ | $80.00\text{\,}\mathrm{\char 37\relax}$
$VG_{path}+VG_{waypoints}$ | $80.00\text{\,}\mathrm{\char 37\relax}$
{*X, *X, *X} | {‘purple umbrella’ $|$ ‘blue umbrella’ $|$ ‘green box’} | $VG_{waypoints}$ | $80.00\text{\,}\mathrm{\char 37\relax}$
$VG_{path}+VG_{waypoints}$ | $60.00\text{\,}\mathrm{\char 37\relax}$
{*X, *X, *X} | {‘purple umbrella’ $|$ ‘blue box’ $|$ ‘green box’} | $VG_{waypoints}$ | $60.00\text{\,}\mathrm{\char 37\relax}$
$VG_{path}+VG_{waypoints}$ | $80.00\text{\,}\mathrm{\char 37\relax}$
{*X, *X, *X} | {‘traffic cone’ $|$ ‘blue box’ $|$ ‘green box’} | $VG_{waypoints}$ | $60.00\text{\,}\mathrm{\char 37\relax}$
$VG_{path}+VG_{waypoints}$ | $60.00\text{\,}\mathrm{\char 37\relax}$

###  V-C Failure Case Analysis

To further examine the impact of different guidance schemes, we conducted an
analysis of failure cases to identify their root causes. These cases are
categorized into two types: (a) out-of-bound (OOB), where the agent traverses
into prohibited areas such as sidewalks, and (b) collision, which refers to
instances where the agent collides with either static or dynamic obstacles.
The results of this analysis are also presented in Table I. An observation
from this analysis is that the agents trained with $VG_{\text{path}}$ exhibit
a lower OOB compared to those trained with $VG_{\text{waypoint}}$ and
$Hybrid$. A lower OOB rate implies that the agent is more likely to stay on
the correct path while also avoiding prohibited areas. In contrast,
$VG_{\text{waypoint}}$ shows a lower collision rate than $VG_{\text{path}}$.
This is because the agent has more flexibility when navigating between
waypoints, which reduces the likelihood of collisions. $VG_{\text{path}}$, due
to its reward structure that encourages path following, tends to stick more
closely to the navigation path to earn rewards, which increases its chances of
colliding with pedestrians if they are on the virtual navigation path.

###  V-D Trajectory Analysis

To further investigate the rationale behind the observations mentioned above,
we present our qualitative results in Fig. 6, where the one-time planned
navigation paths are depicted as blue lines, the waypoints are depicted as
yellow balls, and the actual trajectories followed by the agent are depicted
in pink. We plotted the trajectories for successful cases from the agents
trained using three configurations: $VG_{path}$ (one-time), $VG_{waypoints}$
(one-time), and $Hybrid$. It can be observed that the agents using $VG_{path}$
closely adhere to the planned path, whereas the $VG_{waypoints}$ agents gather
the balls while maintaining some flexibility between waypoints. In contrast,
the baseline $Hybrid$ agents frequently deviates from the waypoints, resulting
in erratic, circuitous trajectories. This may be due to the difficulties the
agents face in reconciling different modalities, specifically visual
observations and vector-based instructions, for this baseline.

###  V-E Real-World Validation of Virtual Guidance

As described in Section IV, we validated the proposed virtual guidance concept
in real-world scenarios, using text prompts to derive virtual guidance paths.
In our experimental configurations, four different compositions of objects
were arranged, as summarized in Table II. Each composition contained multiple
objects, with at least two objects from the same category but with distinct
appearances. For example, the second composition featured two differently
designed umbrellas along with a green box. The agent received a text prompt
describing a particular composition of objects. Each experiment was performed
over five independent runs. Owing to the capabilities of GroundingDINO, all
objects described in the text prompts were correctly detected. The results are
presented in Table II. The symbols $\&$ and $|$ indicate that $\&$ requires
the agent to reach all the objects described by the text prompt, while $|$
requires the agent to reach just one of them. For example, in Fig. 1 (b), the
agent first moves toward the purple umbrella and then proceeds to navigate to
the blue umbrella. The agent’s task was to identify and approach the correct
object(s) based on the provided text prompt, using the transformed virtual
guidance. The results indicate that the policies trained in virtual
environments can be effectively transferred to real-world settings, and that
the agent can follow virtual guidance paths generated via text-prompt-based
navigation objectives.

##  VI Conclusion

In this paper, we introduced the concept of virtual guidance and explored
different potential methods of implementing it, including virtual navigation
paths and virtual waypoints. A virtual city was created to train and evaluate
the proposed virtual guidance scheme, where a planning module generated a path
from randomly selected starting and ending points. The planned path was then
rendered as virtual guidance to direct the agent. In our experiments, multiple
evaluation metrics were employed to assess the effectiveness of the proposed
virtual guidance approaches. We compared the performance of the agents trained
with virtual guidance to those trained with a baseline guidance scheme. Our
results demonstrated that the agents trained with virtual guidance
outperformed those trained with the baseline. This observation was also
supported by our qualitative results. Finally, in our real-world evaluations,
we extended virtual guidance by converting text prompts into visual cues. The
experiments on a Husky AGV confirmed that the policies trained in virtual
settings can effectively be transferred to real-world scenarios.

## References

  * [1] M. Mueller, A. Dosovitskiy, B. Ghanem, and V. Koltun, “Driving policy transfer via modularity and abstraction,” in _Proc. Conf. on Robot Learning (CoRL)_ , 2018, pp. 1–15.
  * [2] S. Bansal, V. Tolani, S. Gupta, J. Malik, and C. Tomlin, “Combining optimal control and learning for visual navigation in novel environments,” 2019.
  * [3] J. Krantz, A. Gokaslan, D. Batra, S. Lee, and O. Maksymets, “Waypoint models for instruction-guided navigation in continuous environments,” in _Proc. IEEE/CVF Int. Conf. on Computer Vision (ICCV)_ , October 2021, pp. 15 162–15 171.
  * [4] R. Guldenring, M. Görner, N. Hendrich, N. J. Jacobsen, and J. Zhang, “Learning local planners for human-aware navigation in indoor environments,” in _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2020, pp. 6053–6060.
  * [5] A. Pokle _et al._ , “Deep local trajectory replanning and control for robot navigation,” in _Proc. Int. Conf. on Robotics and Automation (ICRA)_ , 2019, pp. 5815–5822.
  * [6] C. Li, F. Xia, R. Martín-Martín, and S. Savarese, “HRL4IN: Hierarchical reinforcement learning for interactive navigation with mobile manipulators,” in _Proc. Conf. on Robot Learning (CoRL)_ , 2019, pp. 603–616.
  * [7] A. Faust _et al._ , “PRM-RL: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning,” in _Proc. IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2018, pp. 5113–5120.
  * [8] L. Kästner _et al._ , “Connecting deep-reinforcement-learning-based obstacle avoidance with conventional global planners using waypoint generators,” in _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2021, pp. 1213–1220.
  * [9] L. Kastner _et al._ , “Arena-Rosnav: Towards deployment of deep-reinforcement-learning-based obstacle avoidance into conventional autonomous navigation systems,” in _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2021, pp. 6456–6463.
  * [10] J. Wöhlke, F. Schmitt, and H. van Hoof, “Hierarchies of planning and reinforcement learning for robot navigation,” in _Prof. IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2021, pp. 10 682–10 688.
  * [11] B. Brito, M. Everett, J. P. How, and J. Alonso-Mora, “Where to go Next: Learning a subgoal recommendation policy for navigation in dynamic environments,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4616–4623, 2021.
  * [12] L. Kästner, X. Zhao, Z. Shen, and J. Lambrecht, “A hybrid hierarchical navigation architecture for highly dynamic environments using time-space optimization,” in _Proc. IEEE/SICE Int. Symp. on System Integration (SII)_ , 2023, pp. 1–8.
  * [13] D. S. Chaplot, R. Salakhutdinov, A. Gupta, and S. Gupta, “Neural topological SLAM for visual navigation,” in _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 12 872–12 881.
  * [14] M. Mueller, A. Dosovitskiy, B. Ghanem, and V. Koltun, “Driving policy transfer via modularity and abstraction,” in _Proc. Conf. on Robot Learning (CoRL)_ , 2018, pp. 1–15.
  * [15] A. Sax _et al._ , “Learning to navigate using mid-level visual priors,” in _Proc. Conf. on Robot Learning (CoRL)_ , 2020, pp. 791–812.
  * [16] B. Chen _et al._ , “Robust policies via mid-level visual representations: An experimental study in manipulation and navigation,” in _Proc. Conf. on Robot Learning_ , 2021, pp. 2328–2346.
  * [17] H.-K. Yang _et al._ , “Investigation of factorized optical flows as mid-level representations,” in _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2022, pp. 746–753.
  * [18] Z.-W. Hong _et al._ , “Virtual-to-Real: Learning to control in visual semantic segmentation,” in _Proc. Int. Joint Conf. on Artificial Intelligence IJCAI_ , 2018, pp. 4912–4920.
  * [19] Unity Technologies, “Unity engine,” https://unity.com.
  * [20] K. Yadav, J. Krantz, R. Ramrakhya, S. K. Ramakrishnan, J. Yang, A. Wang, J. Turner, A. Gokaslan, V.-P. Berges, R. Mootaghi, O. Maksymets, A. X. Chang, M. Savva, A. Clegg, D. S. Chaplot, and D. Batra, “Habitat challenge,” 2023.
  * [21] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, _et al._ , “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,” _arXiv preprint arXiv:2303.05499_ , 2023.
  * [22] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik, “Cognitive mapping and planning for visual navigation,” in _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 7272–7281.
  * [23] Y. Liang, B. Chen, and S. Song, “SSCNav: Confidence-aware semantic scene completion for visual semantic navigation,” in _Proc. IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2021, pp. 13 194–13 200.
  * [24] S. Bansal, V. Tolani, S. Gupta, J. Malik, and C. Tomlin, “Combining optimal control and learning for visual navigation in novel environments,” in _Proc. Conf. on Robot Learning (CoRL)_ , 2020, pp. 420–429.
  * [25] W. Gao, D. Hsu, W. S. Lee, S. Shen, and K. Subramanian, “Intention-Net: Integrating planning and deep learning for goal-directed autonomous navigation,” in _Proc. Conf. on Robot Learning_ , 2017, pp. 185–194.
  * [26] A. Mousavian _et al._ , “Visual representations for semantic target driven navigation,” in _Proc. Int. Conf. on Robotics and Automation (ICRA)_ , 2019, pp. 8846–8852.
  * [27] Y. Zhu _et al._ , “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in _Proc. IEEE Int. Conf. on Robotics and Automation (ICRA)_ , 2017, pp. 3357–3364.
  * [28] K. Fang, A. Toshev, L. Fei-Fei, and S. Savarese, “Scene memory transformer for embodied agents in long-horizon tasks,” in _Proc. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 538–547.
  * [29] D. S. Chaplot, D. Gandhi, A. Gupta, and R. Salakhutdinov, “Object goal navigation using goal-oriented semantic exploration,” in _Proc. Int. Conf. on Neural Information Processing Systems (NeurIPS)_ , 2020.
  * [30] S. K. Ramakrishnan, D. S. Chaplot, Z. Al-Halah, J. Malik, and K. Grauman, “PONI: Potential functions for objectgoal navigation with interaction-free learning,” in _Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 18 868–18 878.
  * [31] W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: a survey,” in _SSCI_ , 2020.
  * [32] A. Juliani, V.-P. Berges, E. Teng, _et al._ , “Unity: A general platform for intelligent agents,” _arXiv preprint arXiv:1809.02627_ , 2020.
  * [33] P. E. Hart, N. J. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,” _IEEE Trans. on Systems Science and Cybernetics_ , vol. 4, no. 2, pp. 100–107, 1968.
  * [34] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [35] W. Tranheden, V. Olsson, J. Pinto, and L. Svensson, “Dacs: Domain adaptation via cross-domain mixed sampling,” in _Proc. IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV)_ , January 2021, pp. 1379–1389.
  * [36] T. Haarnoja _et al._ , “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _ICML_ , 2018.
  * [37] P. Christodoulou, “Soft actor-critic for discrete action settings,” _arXiv preprint arXiv:1910.07207_ , 2019.
  * [38] P. Anderson, A. X. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. R. Zamir, “On evaluation of embodied navigation agents,” _arXiv preprint arXiv:1807.06757_.