# Storm Surges as Seen by Coastal and Spaceborne Radars:
Case Studies in British Columbia

Baptiste Domps and Charles-Antoine Guérin B. Domps is with Degreane Horizon,
83390 Cuers, France (e-mail: baptiste.domps@degreane-horizon.fr). C.-A. Guérin
is with the Mediterranean Institute of Oceanography (MIO), Université de
Toulon, Aix-Marseille Université, CNRS, IRD, Toulon, France (e-mail:
guerin@univ-tln.fr).

###### Abstract

Short-term sea level fluctuations prompted by abrupt atmospheric changes can
be hazardous phenomena for coastal regions. We report on two such recent storm
surges that occurred in 2020 on the shores of British Columbia, Canada. A rare
concordance of ground-based and spaceborne sensors made it possible to observe
these events with a variety of instruments : (1) a coastal oceanographic
radar; (2) the synthetic aperture radar onboard satellite Sentinel-1B; and (3)
a network of shoreside tide gauges. In the light of these case studies we show
how satellite-based radar data can be used to complement the observation and
interpretation of ground-based measurements in the context of “tsunami-like”
sea level oscillations.

##  1 Introduction

Coastal High-Frequency Radars (HFR) are today widely used for the real-time
monitoring of sea surface currents (e.g. [13]). Ocean Networks Canada (ONC)
has been operating one such instrument since 2015 in Tofino, on the West Coast
of Vancouver Island, British Columbia (BC), Canada (Fig. 1). This HFR is
equipped with a tsunami warning software based on the detection of abnormal
surface currents patterns [5]. The first real-time warning, on October 14,
2016, occurred in the absence of any reported seismic activity and was
therefore related to the family of atmospheric-driven phenomena ([6, 7]).
Unlike seismic tsunamis, which are generated by earthquakes or landslides,
such events are induced by strong atmospheric disturbances (e.g. pressure
jumps, atmospheric fronts, etc. [10]). The atmospheric forcing can trigger two
kinds of “tsunami-like” sea-level oscillations: long ocean waves (known as
“meteorological tsunamis”) or storm surges. The former yet requires some form
of resonance between the atmospheric propagation speed and the oceanic long
wave phase speed. Both can produce damages along the seashore, demonstrating
the need for early warning systems in the risk zones [17] such as the shore of
British Columbia [16]. In this paper we will only consider the detection of
storm surges, which are more common than meteo-tsunamis.

Figure 1: Map of the study area along the west coast of BC. Selected CHS tide
gauge stations are represented as $\displaystyle\blacktriangle$ (HB: Hartley
Bay; BB: Bella Bella; PH: Port Hardy; WH: Winter Harbour; TO: Tofino; BF:
Bamfield). Superimposed is the footprint of Sentinel-1B observations used in
this study. Typical coverage of the HFR of Tofino is also shown as —.

The main point is that the coastal HFR systems that are used for tsunami
warning can also be utilized for the detection and warning of storm surges.
While the first observation of a storm surge using HFR occurred in the
eighties [9], few observations were reported until the past few years (e.g.
[8, 2]). In particular, the October 2016 event was recently reanalyzed by the
authors, who shown that it was more likely a storm surge than a meteorological
tsunami [4]. It now seems clear that conventional tsunami detection algorithms
can be triggered by abrupt changes in surface currents propagating towards the
coast, should they be of seismic or atmospheric origin. Such ability makes HFR
irreplaceable for the short-term storm surge coastal warning. Thus, it was
recently proposed by [8] to use the wind direction measured by HFR to
discriminate between tsunami waves and storm surges in real-time. However,
this emerging application still requires cautious assessment in the light of
synthetic and real test cases. In this paper, we analyze two tsunami-like
events that caused a warning to be issued by the HFR of Tofino in 2020, on
January 5 and November 23. Both were fortuitously observed by the synthetic
aperture radar (SAR) onboard satellite Sentinel-1B during its flyby of BC,
just before the warnings (Section 3). To our best knowledge, this would be the
first reported observation of a storm surge in open ocean using spaceborne
SAR. We compare the event arrival times derived by the HFR of Tofino and a
network of coastal tide gauges (Section 2) and show that, similarly to the
October 2016 event, both warnings were more likely caused by a storm surge
rather than a meteorological tsunami (Section 4).

##  2 Tide Gauges and HF Radar Measurements

The shores of BC are equipped with tide gauges from the Permanent Water Level
Network (PWLN), operated by the Canadian Hydrographic Service (CHS) [12].
Predicted tides were removed from the measurements and the resulting sea level
anomalies within the study area are shown in Figs. 3a and 3b for the Jan. and
Nov. 2020 events, respectively. In both cases the records are very similar and
show sudden rises in sea level, up to $\displaystyle 20\text{\,}\mathrm{cm}$.
The time shift between the sea level maxima suggests that both phenomena were
induced by an atmospheric disturbance propagating from the northwest to the
southeast at an average speed of 15 to $\displaystyle
20\text{\,}\mathrm{m}\text{.}{\mathrm{s}}^{-1}$, possibly being a maritime
polar (mP) air mass originating from the Gulf of Alaska.

Figure 2: Sea level anomalies at 6 CHS tide gauges
($\displaystyle\mathrm{cm}$; see Fig. 1) from: (—) Jan. 4 to 6, 2020; and (—)
Nov. 23 to 25, 2020. The times of the Sentinel-1B flybys (at 14:43 UTC on Jan.
5 and 14:45 UTC on Nov. 24, — ; see Fig. 4) and of the tsunami alerts issued
by the HFR of Tofino (at $\displaystyle\cdot\cdot\cdot\cdot$ 15:22 UTC on Jan.
5 and \- - - 16:20 UTC on Nov. 24) are also shown.

Figure 3: Time series of radial surface current
($\displaystyle\mathrm{cm}\text{.}{\mathrm{s}}^{-1}$) inverted in the middle
of the beam of the HFR of Tofino (Fig. 1) for the Jan. 2020 event and computed
using the TVAR-MEM approach [3, 4] every $\displaystyle 4\text{\,}\mathrm{s}$
from overlapping signals of $\displaystyle 33\text{\,}\mathrm{s}$. Negatives
values represent surface currents directed toward the radar.

In coastal regions the main amplification mechanism of atmospheric waves at
the sea surface is the so-called Proudman resonance [10], occurring when the
atmospheric propagation speed equals the ocean waves speed $\displaystyle
c=\sqrt{gh}$, where $\displaystyle h$ is the local depth and $\displaystyle g$
is the gravitational acceleration. Thus, the observed atmospheric speeds would
correspond to resonant depths of 25 to $\displaystyle 40\text{\,}\mathrm{m}$,
which seems to be way too confined to set off a meteorological tsunami,
considering the bathymetry off of BC [4, Fig. 1]. This supports the hypothesis
that the observed events were rather related to a storm surge. Propagation of
the gusty polar air mass over the ocean induced a sudden shift in sea surface
currents velocity and direction (of the order of a few tenth of
$\displaystyle\mathrm{cm}\text{.}{\mathrm{s}}^{-1}$, Fig. 3), triggering the
HFR automatic tsunami warning in Tofino (at 15:22 and 16:20 UTC in Jan. and
Nov., resp.). This is consistent with the warnings being issued just after the
sea level maxima, as already noticed by [8] for the January event.

Figure 4: SAR data retrieved from satellite Sentinel-1B flying over British
Columbia (see Fig. 1) in interferometric wide (IW) swath mode at: (top) 14:43
UTC on Jan. 5, 2020; (bottom) 14:45 UTC on Nov. 24, 2020. Left: VV-NRCS map
($\displaystyle\mathrm{dB}$); middle: ocean surface wind speed $\displaystyle
u_{10}$ ($\displaystyle\mathrm{m}\text{.}{\mathrm{s}}^{-1}$); right: ocean
surface wind direction $\displaystyle\theta_{10}$ (meteorological convention,
$\displaystyle\mathrm{\SIUnitSymbolDegree}$).

##  3 Sea Surface as Seen by Sentinel-1B SAR

Sentinel-1 is a constellation of two SAR satellites operating in C-band, as
part of the European Copernicus earth observation program. Over the ocean,
they provide all-weather, day-and-night images of the surface normalized radar
cross section (NRCS) at the meter-scale resolution. In C-band, this NRCS
corresponds to the sea surface roughness being mostly dominated by the local
wind. Thus, numerous air-sea interaction processes can be observed in these
images (e.g., [18]). However, while the first observation of a storm surge in
open ocean using satellite altimeters goes back to 2005 [14], there has been
no such detection with SAR images. Fortuitously, the Jan. and Nov. 2020 events
were observed by satellite Sentinel-1B just before the HFR warnings. The
flybys occurred at 14:43 and 14:45 UTC, respectively.

The atmospheric front can be observed in the sea surface roughness images,
where a high electromagnetic return indicates a rough surface associated to
gusty winds. This is particularly visible for the Jan. 2020 event (Fig. 4a)
where the two air masses are clearly distinct. We used the optimal inversion
method [11] with the CMOD7 geophysical model function (GMF) from the KNMI [15]
to invert the $\displaystyle 10\text{\,}\mathrm{m}$-wind speed ($\displaystyle
u_{10}$) and direction ($\displaystyle\theta_{10}$) from the SAR images. The
resulting maps (Figs. 4b, c, e and f) show the atmospheric process at the
origin of the oceanic event. A sudden rise in $\displaystyle u_{10}$ (from 5
to $\displaystyle 15\text{\,}\mathrm{m}\text{.}{\mathrm{s}}^{-1}$) and shift
in $\displaystyle\theta_{10}$ (from 270 to $\displaystyle
320\text{\,}\mathrm{\SIUnitSymbolDegree}$, that is from west to north-west) is
seen at the passing of both fronts. Considering that the average velocity of
the wind-driven surface current is of the order of $\displaystyle
2\text{\,}\%$ of $\displaystyle u_{10}$ [1], this is consistent with the
amplitude of the jump in radial surface current observed by HFR.

##  4 Discussion and Conclusion

Coincident data from Sentinel-1B, the HFR of Tofino and tide gauges have been
synthesized to observe two storm surge events in January and November 2020. At
the times of the flybys, the atmospheric fronts were located in the open sea
according to the SAR images while the storm surges had already hit the shores
of BC, according to the tides gauges and the HFR warnings. This indicates that
the wind-induced bulges in sea level were located ahead of the atmospheric
fronts. We also investigated the speed of the atmospheric fronts between the
Sentinel-1B flybys and the HFR warnings. The distances between the fronts as
seen in the SAR images and the western outer extent of the HFR coverage are of
the order of $\displaystyle 50\text{\,}\mathrm{km}$ (Jan. event) and
$\displaystyle 100\text{\,}\mathrm{km}$ (Nov. event) while the delay between
the Sentinel-1B flybys and the HFR warnings are of $\displaystyle
39\text{\,}\mathrm{min}$ and $\displaystyle 1\text{\,}\mathrm{h}$
$\displaystyle 35\text{\,}\mathrm{min}$. This leads to an average speed of the
order of $\displaystyle 75\text{\,}\mathrm{km}\text{.}{\mathrm{h}}^{-1}$
($\displaystyle 21\text{\,}\mathrm{m}\text{.}{\mathrm{s}}^{-1}$) and
$\displaystyle 65\text{\,}\mathrm{km}\text{.}{\mathrm{h}}^{-1}$
($\displaystyle 18\text{\,}\mathrm{m}\text{.}{\mathrm{s}}^{-1}$),
respectively, which is consistent with the typical speed of cold fronts.
Further case studies will definitely be needed to assess a systematic
detection of storm surges from spaceborne SAR images and HFR tsunami-warnings.

## Acknowledgment

First author was supported by the Direction Générale de l’Armement (DGA) via
the Agence pour l’Innovation de Défense (AID). Sentinel-1 data was provided by
the European Space Agency (ESA); water levels by the Canadian Hydrographic
Service (CHS); and HFR data by Ocean Networks Canada (ONC). Many thanks go to
Dr Caroline Paugam for her advices about storm surges.

## References

  * [1] M. Berta _et al._ , “Wind-induced variability in the Northern Current (northwestern Mediterranean Sea) as depicted by a multi-platform observing system,” _Ocean Sci._ , vol. 14, no. 4, pp. 689–710, 2018.
  * [2] B. Domps, “Identification et détection de phénomènes transitoires contenus dans des mesures radar à faible rapport signal à bruit : applications conjointes aux problématiques océanographique et atmosphérique,” Ph.D. dissertation, Univ. Toulon, Dec. 2021.
  * [3] B. Domps, D. Dumas, C.-A. Guérin, and J. Marmain, “High-Frequency Radar Ocean Current Mapping at Rapid Scale With Autoregressive Modeling,” _IEEE J. Ocean. Eng._ , vol. 46, no. 3, pp. 891–899, 2021.
  * [4] B. Domps, J. Marmain, and C.-A. Guérin, “A Reanalysis of the October 2016 “Meteotsunami” in British Columbia With Help of High-Frequency Radars and Autoregressive Modeling,” _IEEE Geosci. Remote Sens. Lett._ , vol. 19, 2022\.
  * [5] A. Dzvonkovskaya, L. Petersen, T. Helzel, and M. Kniephoff, “High-frequency ocean radar support for Tsunami Early Warning Systems,” _Geosci. Lett._ , vol. 5, Dec. 2018.
  * [6] A. Dzvonkovskaya, L. Petersen, and T. L. Insua, “Real-time capability of meteotsunami detection by WERA ocean radar system,” in _International Radar Symposium_ , Prague, Czech Republic, 2017.
  * [7] C.-A. Guérin, S. Grilli, P. Moran, A. Grilli, and T. L. Insua, “Tsunami detection by high-frequency radar in British Columbia: performance assessment of the time-correlation algorithm for synthetic and real events,” _Ocean Dyn._ , vol. 68, pp. 423–438, 2018.
  * [8] M. L. Heron, B. Domps, C.-A. Guérin, M. Wang, and L. Petersen, “HF Radar Real-Time Alert to a Tsunami-Like Disturbance at Tofino, 5 January 2020: Surge or Tsunami?” _IEEE J. Ocean. Eng._ , 2022, submitted.
  * [9] B. J. Lipa and D. E. Barrick, “Tidal and Storm-Surge Measurements with Single-Site CODAR,” _IEEE J. Ocean. Eng._ , vol. OE-11, no. 2, pp. 241–245, Apr. 1986.
  * [10] S. Monserrat, I. Vilibić, and A. B. Rabinovich, “Meteotsunamis: Atmospherically Induced Destructive Ocean Waves in the Tsunami Frequency Band,” _Nat. Hazards Earth Syst. Sci._ , vol. 6, pp. 1035–1051, Mar. 2006\.
  * [11] M. Portabella, A. Stoffelen, and J. A. Johannessen, “Toward an optimal inversion method for synthetic aperture radar wind retrieval,” _J. Geophys. Res. Oceans_ , vol. 107, no. C8, 2002.
  * [12] A. B. Rabinovich and F. Stephenson, “Longwave Measurements for the Coast of British Columbia and Improvements to the Tsunami Warning Capability,” _Nat. Hazards_ , vol. 32, pp. 313–343, Jul. 2004.
  * [13] H. Roarty _et al._ , “The Global High Frequency Radar Network,” _Frontiers Mar. Sci._ , vol. 6, p. 164, 2019.
  * [14] R. Scharroo, W. H. F. Smith, and J. L. Lillibridge, “Satellite altimetry and the intensification of Hurricane Katrina,” _Eos Trans. AGU_ , vol. 86, no. 40, p. 366, 2005.
  * [15] A. Stoffelen, J. Verspeek, J. Vogelzang, and A. Verhoef, “The CMOD7 Geophysical Model Function for ASCAT and ERS Wind Retrievals,” _IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens._ , vol. 10, no. 5, pp. 2123–2134, May 2017.
  * [16] R. Thomson, A. B. Rabinovich, I. Fine, D. Sinnott, A. McCarthy, N. Sutherland, and L. Neil, “Meteorological tsunamis on the coasts of British Columbia and Washington,” _Phys. Chem. Earth_ , vol. 34, pp. 971–988, Dec. 2009.
  * [17] I. Vilibić, J. Šepić, A. B. Rabinovich, and S. Monserrat, “Modern Approaches in Meteotsunami Research and Early Warning,” _Front. Mar. Sci._ , vol. 3, 2016.
  * [18] C. Wang, A. Mouche, P. Tandeo, J. Stopa, N. Longépé, G. Erhard, R. C. Foster, D. Vandemark, and B. Chapron, “A labelled ocean SAR imagery dataset of ten geophysical phenomena from Sentinel-1 wave mode,” _Geoscience Data Journal_ , vol. 6, no. 2, pp. 105–115, 2019.


# Visual Semantic Navigation with Real Robots

Carlos Gutiérrez-Álvarez111Indicates equal contribution carlos.gutierrezalva,
pablo.rios, javier.acevedo<EMAIL_ADDRESS><EMAIL_ADDRESS>Pablo Ríos-Navarro222Indicates equal contribution Rafael Flor-Rodríguez-
Rabadán  Francisco Javier Acevedo-Rodríguez  Roberto Javier López-Sastre

###### Abstract

Visual Semantic Navigation (VSN) is the ability of a robot to learn visual
semantic information for navigating in unseen environments. These VSN models
are typically tested in those virtual environments where they are trained,
mainly using reinforcement learning based approaches. Therefore, we do not yet
have an in-depth analysis of how these models would behave in the real world.
In this work, we propose a new solution to integrate VSN models into real
robots, so that we have true embodied agents. We also release a novel ROS-
based framework for VSN, ROS4VSN, so that any VSN-model can be easily deployed
in any ROS-compatible robot and tested in a real setting. Our experiments with
two different robots, where we have embedded two state-of-the-art VSN agents,
confirm that there is a noticeable performance difference of these VSN
solutions when tested in real-world and simulation environments. We hope that
this research will endeavor to provide a foundation for addressing this
consequential issue, with the ultimate aim of advancing the performance and
efficiency of embodied agents within authentic real-world scenarios. Code to
reproduce all our experiments can be found at
https://github.com/gramuah/ros4vsn.

###### keywords:

robotics , embodied agents , vision-based navigation , artificial intelligence
, reinforcement learning

††journal: ArXiv

[uah]organization=University of Alcala, Department of Signal Theory and
Communications, city=Alcala de Henares, postcode=28801, country=Spain

##  1 Introduction

Is it possible for a robotic agent to navigate and interact in a real world
environment the same way we humans do? This is the question that the embodied
AI research community is pursuing to answer. The problem is formally known as
Visual Semantic Navigation (VSN). The mimic of human navigation is a
challenging task for robots, specially in unseen scenarios, since it requires
an efficient exploration of the environment and an understanding of the
objects and structures present in it. For unknown scenarios, humans can
leverage prior semantic information achieved from previous scenes to navigate
in new environments, but it is still a difficult task to incorporate that
knowledge into embodied agents, especially in real robotic platforms. The
possibilities of autonomous robots are endless, from assistive robots that can
accompany a person with reduced mobility to a specific location, to platforms
that can help in complex work scenarios, like search and rescue or logistic
centers. Although there are several VSN works that achieve near-human
navigation performance, _e.g_. (Ramrakhya et al., 2023), these systems are
trained and tested in virtual environments, so how these models would behave
in the real world is still a question to answer.

Figure 1:  ObjectNav task is a complex navigation problem. An agent needs to
employ vision-based sensors to navigate from a random starting point to
specific object goals within the scene. Many different hardware and software
components need to be fully integrated to solve it, making it difficult to
deploy and test these visual semantic navigation (VSN) models in real robots.
Therefore, the best current solutions are trained and tested in virtual
environments. Our goal is to bridge the gap between virtual and physical
environments by providing a ROS-based framework that simplifies testing and
comparing various VSN models on real robotic platforms.

In this work, we focus on embedding VSN models in real robotic platforms. We
address the Object-Goal Navigation (ObjectNav) (Batra et al., 2020) problem.
As it is seen in figure 1, in ObjectNav task the agent has to navigate from a
random spawn point to certain object goals present in the scene, mainly using
vision-based sensors. In contrast with geometric navigation approaches, where
the navigation problem is solved typically by using a map or generating it on
the fly, VSN models are learning-based approaches that use _no_ metric map.
Therefore, VSN solutions must learn visual representations of the environment
to reduce the exploration time and better generalize to unseen scenes and
object categories. It is interesting to note that most of these solutions
combine reinforcement learning (RL) strategies with recent advancements in
deep learning models for visual perception so as to tackle the navigation
problem in virtual environments where the embodied AI agents are trained and
tested.

In this work, our objective is to build true embodied agents proposing a new
solution for the integration of VSN models into real robots. The main
contributions of our work are as follows:

  1. 1.

We first provide to the embodied agents community the ROS4VSN development, a
novel Robot Operating System (ROS) (Quigley et al., 2009) based architecture
that allows to test and compare different VSN models in real robots. Our
ROS4VSN development is model agnostic, thus, any VSN solution can be
integrated into it with ease. Section 3.1 includes all the technical details.

  2. 2.

We also have embedded two state-of-the-art VSN models in two different robotic
platforms. The selected models have been PIRLNav (Ramrakhya et al., 2023) and
VLV (Chang et al., 2020). To achieve this integration, it is necessary to make
technical adaptations to the models so that they transition from interacting
with observations provided by simulation environments to observations from the
real world. We detail all these technical modifications to the models in
Section 3.2.

  3. 3.

Finally, we provide a detailed experimental evaluation of the adapted VSN
models, using our ROS4VSN, with two different robots, in a real world scenario
(Section 4). The main question we want to address with the designed
experiments is: Are the state-of-the-art VSN models able to successfully
operate with real robots? We have measured their success rate in real world
navigation experiments, which has allowed us to analyze the difference in
performance compared to those tests in simulation environments, where embodied
AI agents are typically tested. Our work shows that these VSN solutions
perform noticeably differently when evaluated in real-world and simulated
situations. With the ultimate goal of improving the performance and efficiency
of VSN systems in real robots, we hope that our work will help to provide the
groundwork for tackling this significant challenge.

##  2 Related Work

Visual Semantic Navigation. To navigate in unfamiliar environments,
traditional methods use depth sensors (Newcombe et al., 2011; Thrun et al.,
2001) and RGB cameras (Jones and Soatto, 2011; Sattler et al., 2018) to build
geometric maps and simultaneously determine the robot’s position in relation
to the map. This is known as Simultaneous Localization and Mapping (SLAM)
(Abaspur Kazerouni et al., 2022; Campos et al., 2021; Labbé and Michaud,
2022)). Typically, these SLAM models use heuristic algorithms to create graph-
based representations of the environment, allowing the robot to visit the
different nodes of the graph when navigating to specific points. Semantic SLAM
(_e.g_. (Zhang et al., 2018; Rosinol et al., 2020; Jin et al., 2023)) expands
upon SLAM by integrating semantic data from the environment, allowing the
robot to identify and store objects in memory.

A recent approach, made possible by advances in machine learning and computer
vision, involves designing navigation policies that directly train deep neural
networks to learn semantic information from visual observations in an end-to-
end fashion (_e.g_. (Ramrakhya et al., 2022; Yadav et al., 2023b; Gutiérrez-
Maestro et al., 2019; Khandelwal et al., 2022; Chaplot et al., 2020; Chang et
al., 2020)). This approach is termed visual semantic navigation (VSN). These
models often rely on the use of CNNs as visual encoders followed by RNNs; that
are in charge of predicting an action distribution directly from raw input
observations. The neural networks are trained using imitation learning (IL) or
reinforcement learning (RL) approaches.

When IL is applied to the visual navigation problem, navigation policies are
learnt from expert demonstrations (_e.g_. (Ramrakhya et al., 2022; Yadav et
al., 2023b)). It can also be used combined with a RL finetuning phase to
achieve better performance (Ramrakhya et al., 2023).

Other works focus on the use of an end-to-end RL approach to solve ObjectNav
navigation (Zhu et al., 2017; Gutiérrez-Maestro et al., 2019; Wijmans et al.,
2020; Khandelwal et al., 2022; Liu et al., 2022; Yadav et al., 2023a). Some
authors have proposed to combine the RL training with different strategies,
like auxiliary tasks (Ye et al., 2021), improved visual representations via
object relation graphs (Yang et al., 2018), semantic segmentations (Mousavian
et al., 2018) or combining audio feedback with the visual inputs (Wang et al.,
2023; Kondoh and Kanezaki, 2023).

Modular-learning based approaches (Chaplot et al., 2020; Chang et al., 2020)
decompose the navigation process in separate modules that execute different
tasks. It is common for these methods to be composed of a high-level semantic
exploration module trained by RL that indicates the agent sub-goals that have
to be reached by a low-level navigation policy. Modular learning can be also
combined with offline RL (Shah et al., 2022) techniques to leverage navigation
behaviors from fixed datasets, without any additional online data collection
or fine-tuning.

Finally, there are different approaches that try to tackle the problem of
rapidly adapting to unseen environments in visual navigation via meta-learning
(Wortsman et al., 2019; Luo et al., 2021; Zhang et al., 2022). These methods
are trained on a variety of different environments (usually designated as
tasks) and are able to generalize to unseen environments by learning a policy
that can be quickly adapted to new environments. And the recent progress in
large language models (LLMs) has led to the possibility of using them to solve
the visual navigation problem (Huang et al., 2023; Zhou et al., 2023) as well.
In this case, the LLMs are used as a reasoning module in charge of
understanding the semantic information present on the environment. They then
share this information with different modules in charge of navigating to the
specified goal.

Our goal in this work consists in the technical embedding of different state-
of-the-art VSN models in several real robots. We also provide the novel
ROS4VSN package that allows to easily test and compare different VSN methods
in the real world. To the best of our knowledge, our work is the first to
develop a model agnostic ROS package for visual semantic navigation.

Simulation-to-reality transfer. Deploying a model trained in simulation to a
real robot is a challenging task. Due to logistical limitations, it is often
not possible to train a model in the real world, especially when using RL
techniques. For example Kim et al. (2022), propose a monocular vision-based
time-to-collision estimation for small drones by domain adaptation of
simulated images. Their method converts simulated images into real-like
synthetic images using sim-to-real method. This is done with the aim of
minimizing efforts and time invested in the collection of training datasets
within real-world scenarios, while simultaneously maximizing the advantages
inherent in simulated environments. Overall, it is necessary to develop
methods that allow to efficiently transfer the knowledge learnt in simulation
to the real world (Kadian et al., 2020). Different approaches have been
proposed to solve this problem. For instance, CAD2RL (Sadeghi and Levine,
2017) system achieved remarkable success in training a collision avoidance
policy entirely within a simulated environment. This breakthrough was
subsequently tested on real aerial drones, with promising results.

By focusing on simulation refinement (Son et al., 2020), the accuracy of
simulations can be improved by exploiting the disparities between simulated
and real-world observations. In the field of locomotion, training legged
robotic systems in a simulated environment and subsequently transferring the
acquired policies to real-world applications (Hwangbo et al., 2019; Agarwal et
al., 2022) has always been a challenging task.

Finally, the excellent study by Gervet et al. (2022) shows how different
visual navigation methods perform in the real-world. Indeed, this work is
closely related to our research. However, we would like to highlight the novel
contributions that our work offers. While (Gervet et al., 2022) focuses mainly
on the comparison of navigation methods, we here, along with a similar study,
release to the robotics community the modular ROS4VSN architecture. Our main
goal is to facilitate the prototyping of new VSN solutions on real robots. So,
we offer a ROS compatible software architecture, model agnostic, that allows a
simple integration of different VSN approaches in ROS robots. Furthermore, we
include in our study more recent VSN solutions than the ones reported in
(Gervet et al., 2022), as the PIRLNav model (Ramrakhya et al., 2023), which
defines the state-of-the-art for the ObjectNav problem. We also provide, for
the first time, a detailed analysis on how a model directly trained with real
videos, such as the VLV (Chang et al., 2020), performs in real robots. This
allows us to compare, as in (Gervet et al., 2022), how a modular-learning
model (_i.e_. VLV (Chang et al., 2020)), compares with a typical end-to-end
learning approach (_i.e_. PIRLNav). Interestingly, our study also concludes,
like in (Gervet et al., 2022), that modular-learning approaches perform better
in the real world. Our results, with two different robots confirm that.
Overall, we hope that our ROS library will help to further advance the field
of visual semantic navigation in real robots.

##  3 Methodology

In this research work, our main objective has been to efficiently integrate
various state-of-the-art VSN models on multiple robotic platforms. To fulfill
our objectives, the first step of the proposed methodology has been to develop
a ROS-based solution to ease the integration of VSN solutions into real
robots. It is crucial that the approach is model-agnostic, allowing the
integration of any VSN-designed model. We have named this development ROS4VSN,
and it is detailed in Section 3.1. Once the ROS4VSN system is available, it is
time to select state-of-the-art VSN models that will allow us to
experimentally evaluate them with real robots. In Section 3.2, we justify the
selected VSN models and detail the technical modifications made to integrate
them into ROS4VSN.

###  3.1 ROS4VSN: ROS for Visual Semantic Navigation

We have designed ROS4VSN library to be modular and flexible, so that it can be
easily adapted to different robots and VSN models. It is built on top of ROS
Noetic (Quigley et al., 2009) open-source robotic middleware, because its
flexibility, support, compatibility and popularity among the robotics
community. ROS provides a collection of useful tools, libraries, and
conventions to simplify the task of creating complex and robust robot
behaviors across a wide variety of robotic platforms, which makes it perfect
for our framework.

We design the architecture of the framework, so it has three main
capabilities: it can receive and process information from the environment,
infer actions using an AI VSN model, and control the actuators of a platform
to reach a specific navigation goal. It makes it easy to integrate different
VSN models, since it only needs to replace the model with which the
experiments are to be carried out.

The architecture is divided into the following main packages, each of which
plays a specific role: robot_api, camera_api, discrete_move and
visual_semantic_navigation. These packages are connected to each other through
ROS topics and services, and also to external hardware devices: a camera (RGB
+ Depth) and a differential drive robotic platform.

Figure 2 shows a visual representation of the global architecture scheme,
illustrating the connections between the different developed ROS packages and
the hardware devices.

Figure 2: Architecture scheme of the ROS44VSN framework. It shows the
different packages, their topics and the connections within them and the
hardware devices.

####  3.1.1 Robot API

This package is responsible for controlling the actuators of the robot and
sending odometry information to the discrete_move and
visual_semantic_navigation packages (depicted in figure 2). It is typically
designed by the manufacturer of the robot, so it can be different depending on
the employed platform. In our particular case, the development of the
framework and experiments were done using two robots (see Figure 8). First, a
Turtlebot 2 robot, so the standard turtlebot2 (Ltd., 2023a) package is
integrated as robot_api. Since the Turtlebot 2 expects the velocity commands
in the /mobile_base/commands/velocity topic, we perform a remap to the
/cmd_vel topic from our discrete_move package. Our second robot is know as
LOLA2 (Nasri et al., 2022), for which we have developed the complete robot_api
package, so as to guarantee the compatibility with the rest of ROS4VSN
architecture.

In charge of the communication with the platform, this package is also
responsible for publishing the robot’s odometry information through the topic
/odom (odometry topic). This information is crucial for the
visual_semantic_navigation and discrete_move packages. On one hand, the
package discrete_move uses this information to adjust the velocity commands
sent to the robot_api package, achieving precise and controlled movements. On
the other hand, the visual_semantic_navigation package can use the odometry
information to help infer the action to be executed by the robot or to help a
planner reach its destination.

####  3.1.2 Camera API

This package is responsible for capturing RGB and depth images from the
camera. It publishes them through the /camera/color and /camera/depth topics,
respectively. The camera used in our robots is the Orbecc Astra S, a RGB plus
depth camera, based on structured light technology. We have adapted the
official ros_astra_camera package (Ltd., 2023b) to be integrated in our
ROS4VSN architecture. The modular design of ROS4VSN allows it to be used with
any other type of camera on the robots, simply by adapting this Camera API
package.

####  3.1.3 Discrete Move package

This package has been developed with the purpose of providing a precise and
customizable control of the robotic platform through discrete navigation
commands. Note that this is the way most VSN models interact with their agents
(_e.g_. (Ramrakhya et al., 2023; Chang et al., 2020)). In embodied AI,
navigation in simulated environments is performed using discrete action
commands that agents execute to reach the specific goals: move forward (25
cm), turn left or right (30 degrees), or stop. Our ROS4VSN package acts as a
server to which clients can request a set of discrete movements and configure
the forward distance or turning angles. The package communications scheme is
shown in detail figure 3. It is in charge of executing the actions requested
by the visual_semantic_navigation package and sending a response when the
action is completed.

Figure 3: Communications between visual_semantic_navigation and discrete_move
packages.

##### Set of navigation movements

The set of movements allowed by the package consists of the following actions:
turn_left, turn_right, move_forward, move_backward and stop. All the actions
are fully customizable in terms of distance and angle, except for the stop
action, which does not require any additional parameters since it just stops
the robot. This package has been designed as a ROS service, so the
communication between the visual_semantic_navigation package and the
discrete_move package is done synchronously and bidirectionally. That way, the
visual_semantic_navigation package can wait for the response of the
discrete_move package when the action has been completed before sending any
new action request. The discrete_move server is in charge of sending the right
/cmd_vel commands to the robot_api package, so the robot can execute the
requested action and to receive the /odometry topic information from the
robot_api package, so it can calculate the movement done by the robot, and
stop the action when the requested action has been finished.

Embodied AI navigation environments, such as Habitat (Szot et al., 2021), are
simulation environments where there are no movement errors in the agents.
However, our scenario is the real world, with real robots, therefore ROS4VSN
must integrate error control strategies. To achieve this, the discrete_move
package includes two error correction strategies: one for the turn error
$\epsilon_{turn}$ and one for the move straight error $\epsilon_{straight}$.

The turn error is calculated in degrees as follows,

$\epsilon_{turn}=(\alpha_{target}-\alpha_{current})\bmod(360)\;,$ (1)

where $\alpha_{target}$ is the target orientation and $\alpha_{current}$ is
the current orientation of the robot.

The move straight error is computed as:

$\epsilon_{straight}={d-\sqrt{(x-x_{init})^{2}+(y-y_{init})^{2}}}\;,$ (2)

where $d$ is the displacement distance requested by the system, $(x,y)$
encodes the current position of the robot and $(x_{init},y_{init})$ defines
the initial position of the robot. We consider a successful turn when
$\epsilon_{turn}$ is less than 0.1 degrees and a successful displacement when
$\epsilon_{straight}$ is less than 5 millimeters. Our ROS4VSN architecture
continuously measures these errors to adapt the rotation and displacement
movements of the robots, ensuring that they occur with the highest possible
precision.

##### Acceleration and braking control

When developing a navigation system based on discrete commands, it is crucial
to implement appropriate braking and acceleration mechanisms to achieve smooth
and efficient robot navigation.

The package includes an implementation that combines a constant acceleration
until the desired maximum speed is reached, with a deceleration phase to stop
the robot.

The smoothness of the movement and the time needed to complete it depend on
the percentage of the path in which the robot is accelerating and
decelerating, as well as on the initial speed. By properly adjusting these
parameters, a smoother movement and a more efficient navigation time can be
achieved. Figure 4 illustrates the total distance that the robot must travel
for a move_forward (or move_backward) command. This distance is divided so
that the robot performs the following phases: acceleration, displacement at
constant speed and deceleration. The instantaneous speed of the robot is
controlled so that during the first phase, there is a uniformly accelerated
motion, according to the following equation:
$v=\sqrt{v_{init}^{2}+2a\epsilon_{straight}}$, where $a$ is the desired
acceleration, $v_{init}$ represents the initial speed of the robot, and
$\epsilon_{straight}$ encodes the distance covered by the robot. Note that we
must continuously read $\epsilon_{straight}$ using our ROBOT API, in order to
dynamically adjust the speed. For the deceleration stage, we employ an
equivalent negative acceleration in the previous equation. With these
equations we can progressively adjust the linear speed ($v$) that is sent to
the platform in order to obtain an smooth navigation.

Figure 4: Acceleration and braking control scheme.

##### Configuration parameters

The discrete move package includes the configuration file discrete_move.yaml,
where the different parameters used for the execution can be modified. Table 1
shows the configuration parameters and its default value.

Parameter | Default Value
---|---
Linear Velocity | 0.3 m/s
Angular Velocity | 0.5 rad/s
Acceleration and deceleration distances | (move_forward distance) / 3
Table 1: Configuration parameters of discrete_move package.

####  3.1.4 Visual Semantic Navigation package

One of the main features of our ROS4VSN software is that it allows to easily
integrate different VSN models independently of the robotic platform used. To
achieve this, it is essential to be agnostic with respect to the software
environment needed by the particular VSN model, as each model may require
different dependencies.

The goal of this package is to simplify the deployment of VSN models on real
robots, providing an efficient software structure for the execution of these
methods. In other words, it aims to facilitate the inference tasks of discrete
movement actions that these systems produce, using the state of the robotic
platform. The state is defined by the robot’s position in the real world, the
information provided by its sensors, and the action that was previously taken.

VSN models make decisions using mainly RGB images of the environment. However,
some of them can also use additional information, such as the position and
orientation of the robot, or even depth images. Therefore, this package must
be responsible for: a) capturing all the information from the sensors and
processing the obtained data; b) inferring with the VSN models the next
navigation action; and c) communicating with the robotic platform to request
the corresponding discrete motion. This process is repeated iteratively,
collecting new data, making inferences and executing actions, until a stop
condition is reached or an error occurs.

In our architecture, the package connects to the camera through the /camera
topic published by the camera_api module, to receive the necessary RGB and
depth images. It also collects information from the robot’s odometry through
the /odometry topic published by the robot_api package. Furthermore, it acts
as a client of the discrete_move package to send the actions determined by the
VSN model and receive confirmations about their execution. This package
contains two additional submodules: image preprocessing and odom processing.

##### Image Preprocessing submodule

This submodule is in charge of collecting and preprocessing the images from
the camera of the robot. These images are necessary for the VSN model to infer
the appropriate action. The package must be able to communicate with the
camera in real time and receive its information. This communication is done
through the /camera/color and /camera/depth ROS standard topics. Typically,
depth images are taken using a time-of-flight camera. This type of camera can
lead to noise problems, including incomplete data in certain areas of the
image, noise on metallic surfaces, and the impact of scene lighting on
distance measurements. To address these problems, a temporal median filter is
implemented, so for a series of $N$ depth images, its noise can be reduced by
discarding outliers.

##### Odom Processing submodule

This submodule is in charge of collecting odometry information by subscribing
to the /odom topic (odometry topic), published by the robot_api package. This
odometry information consists of two main variables: 1) robot position, that
indicates the current location of the robot with respect to its initial
position; and 2) robot orientation, that defines the current direction in
which the robot is oriented in relation to its initial orientation. Some
state-of-the-art VSN models (_e.g_. (Ramrakhya et al., 2023)) need to input
these two sources of information.

##### Module workflow

Our VSN package uses the image submodule and the odometry submodule to capture
the camera images and the robot odometry data. This information can then be
passed as input to a particular VSN model. Using the VSN model integrated in
the package, the next navigation action that the robot must execute is
inferred. Once the action has been determined, the package sends a message
(request) to the discrete_move server to request the execution of the movement
by the robot. After sending the request to the discrete_move server, the
package waits to receive a confirmation message. This message indicates
whether the requested action has been performed correctly or if some problem
has occurred. If for some reason an action has not been completed
successfully, the server has been programmed to return $False$, which
completely stops the execution of the workflow. It is important to highlight
that the workflow is repeated until whether the stop action is inferred by the
model, the time limit for the episode is reached or the server responds with a
message indicating some problem during the execution.

A configuration file is provided (vsn.yaml) containing default values for the
parameters of our VSN package. By modifying this file one can easily change
the navigation target, parameters associated with the median filter, or even
the maximum number of steps allowed to be executed during a navigation
exercise.

###  3.2 VSN models

For this research work, we have decided to adapt and integrate into our robots
two state-of-the-art VSN models: PIRLNav (Ramrakhya et al., 2023) and VLV
(Chang et al., 2020). The first model, known as PIRLNav (Ramrakhya et al.,
2023), is a VSN approach that has been trained with a combination of imitation
learning and a RL fine-tuning. As of today, this model reports the best
results in the ObjectNav (Batra et al., 2020) task in Habitat (Szot et al.,
2021). The second model is the VLV approach (Chang et al., 2020), which is a
VSN model directly trained from YouTube videos. The VLV model makes use of
such videos to learn semantic cues for an effective navigation to semantic
targets in indoor scenarios. VLV is a modular learning solution, that combines
low-level and high-level navigation policies.

These models are complementary in the sense that they are based on two
paradigms: a) imitation learning plus RL; and b) modular learning. This aspect
will allow us to study, in the experimental evaluation, which type of approach
yields better results in the real world. Note that we do not intend to retrain
these models but rather subject them to evaluation in the real world. We aim
to analyze their generalization ability for navigation outside simulation
environments. It is precisely thanks to our ROS4VSN system that this can be
done, as the technical modifications made to the models will be oriented
towards embedding them in a ROS-based system. Next, we provide a detailed
description of the modifications and adaptations made to these models so that
they can be integrated into ROS4VSN and tested on real robotic platforms.

####  3.2.1 VLV

The first approach is known as Value Learning from Videos (VLV), developed by
Chang et al. (2020). VLV is a modular-learning based VSN model directly
trained from videos of real state agencies, taken from YouTube. In this type
of video, a human records, camera in hand, the properties for sale, showcasing
all the rooms they have, to generate a sort of virtual tour of the houses.
Note that the videos used do not have any type of information about the
navigation actions that take place during the recording, nor on what kind of
rooms or what type of objects appear.

The VLV model leverages such YouTube videos to learn semantic cues for an
effective navigation to semantic targets in indoor home environments. This
way, the VLV model is trained to find in these videos a set of object
categories. Technically, the model uses pseudo action labels obtained by
running an inverse model on the navigation sequences. This inverse model is
able to recognize the discrete movements that each of the transitions of the
different video frames involve. Then, the navigation policies are learned
following a reinforcement learning (RL) approach. VLV employs Q-learning to
learn from the video sequences that have been pseudo-labeled with the actions.
The learned Q-function, and the associated value function, implicitly learn
semantic cues for navigation. In other words, the model learns what images
lead to the desired category, and what do not.

For our experiments, we had to embed the VLV model in our ROS4VSN
architecture. See Figure 5(a). Technically, we integrated the two navigation
policies detailed in the experiments in (Chang et al., 2020) that were tested
in the virtual environment Habitat (Szot et al., 2021). However, now, our
objective is to implement them in a real robot in the real world.

This integration into our robots, using our ROS4VSN architecture, has
consisted of the following steps. First, a high-level policy that stores 12
images for each node in a topological graph (obtained by rotating 12 times by
30 degrees each) is employed. This high-level policy uses the learned value
function score over these 12 images, and samples the most promising direction
for seeking objects of a particular object category. VLV needs an object
detector output to produce the final score for these images. This way, the
high-level policy is equipped with a mechanism to seek the object once it has
been detected. Specifically, the detector we employ is the Mask R-CNN (He et
al., 2017), which we had to embed in our architecture as well. To navigate,
this policy will use the following discrete movements/actions: move_forward
25cm, turn_right $30^{\circ}$ or turn_left $30^{\circ}$.

These movements are compatible with the developed discrete_move package of our
ROS4VSN architecture. Once a main direction has been chosen, our approach
converts it into a short-term goal by sampling a location at an offset of 1.5
meters from the chosen node’s location, in the chosen view’s direction. This
is done using the depth-camera and a low-level navigation policy that uses
occupancy maps with a fast marching planning (Sethian, 1996) to execute robot
actions to reach the short-term goal. These two policies have been integrated
into our ROS4VSN nodes, and with them we get the robot to explore the
environment.

(a) VLV model.

(b) PIRLNav model.

Figure 5: VSN models integrated into our VSN-ROS.

####  3.2.2 PIRLNav

The second model selected for our experimental evaluation is known as PIRLNav
(Ramrakhya et al., 2023). As of today, this model reports the best performance
in the ObjectNav (Batra et al., 2020) task in Habitat (Szot et al., 2021).

Figure 6: One of the 3D reconstructions of the HM3D-Semantics v0.1 dataset
(Ramakrishnan et al., 2021), used to train PILRNav.

PILRNav is a VSN approach that has been trained with a combination of
imitation learning and a RL fine-tuning. The model uses behavior cloning (BC)
to pre-train the ObjectNav policy on a dataset of $77k$ human demonstrations,
amounting more than $2370$ human annotation hours, in the HM3D-Semantics v0.1
dataset (Ramakrishnan et al., 2021). This dataset provides up to 120 different
3D reconstructions of houses all around the world (see Figure 6 for an example
of one of them). Once this BC is finished, a RL finetuning is used following
the DD-PPO approach (Wijmans et al., 2019). The policy architecture used is a
simple CNN plus a RNN from (Yadav et al., 2023b).

In order to integrate PIRLNav into our ROS4VSN architecture, we had to perform
the following actions. See figure 5(b). The original PIRLNav needs to receive
as inputs the RGB image that the agent observes, as well as the noiseless GPS
and compass information offered by Habitat simulator. GPS and compass Habitat
sensor provides the agent’s current location and orientation information
relative to the start of the episode. In our case, because PIRLNav has to be
integrated in a real robot, navigating in the real world, we proceed to feed
the model with the RGB images that are acquired by the cameras in our robotic
platforms. GPS information is obtained through the odometry information
provided by the robot. For the compass, we recover the relative orientation
analyzing all the robot’s turning movements. Note that these are not anymore
noiseless sensors, as the ones used in the simulated world in which the
PIRLNav model was trained. Fortunately, we did not observe any important
impact on the performance of the model, due to this loss of precision for
these sensors.

PIRLNav is therefore integrated in the ROS4VSN architecture to control de
navigation of the robot as it has been detailed. For every captured image, as
well as the GPS+compass data, the model is able to determine the next discrete
movement action to be executed by our robotic platforms. The action space used
for our experiments with this model is: move_forward 25 cm, move_backward 25
cm, turn_right $30^{\circ}$, turn_left $30^{\circ}$ and stop. The original
PIRLNav was trained to produce also the discrete action look_up and look_down,
since the simulated agent could tilt its camera. However, as it is observed in
Figure 8, in our platforms these actions are not possible. We decided to
replace look_up with a move_backward action, and look_down movement with the
move_forward action. This choice is based on the reasoning that, by raising
the camera, more of the scene is captured; moving the robot backward serves
this purpose. Also, since lowering the camera provides a greater level of
scene detail, moving forward is considered the most appropriate choice to
replace the look_down action. Finally, in order to prevent collisions between
the robot and objects in the scene, a procedure was developed that uses
information from the depth image to detect obstacles at a given distance. Note
that PIRLNav does not need any low-level policy as in the VLV model. The robot
is controlled and navigates using only the set of discrete actions provided by
the ROS4VSN model.

##  4 Experiments

This section describes the experimental evaluation designed for testing our
developments in the real world. The goal of our experimental evaluation is to
answer the following question: Are the state-of-the-art VSN models able to
successfully operate with real robots? We start with a detailed description of
the experimental setup, where the experimental conditions and the evaluation
metric are explained. We then follow with an analysis of the results obtained
in the real world.

###  4.1 Experimental setup

One of the main objectives of our work is to provide a thorough evaluation of
state-of-the-art VSN models in a real scenario with real robots. For doing so,
we propose the following experimental setup.

Figure 7: Floor plan where the experiments were performed, indicating the 15
starting positions used.

We have defined 15 starting random positions in an apartment of 75 m2, fig 7.
When analyzing the apartment, it can be divided into three parts: bedroom,
bathroom and a larger room where the kitchen, the living room and the study
area are located. This house contains all the categories of objects employed
to train the VSN models used in the experiments: chair, bed, plant, bathroom,
monitor, table and sofa.

From these positions, the robot is asked to navigate towards the different
object categories. Therefore, we perform 15 navigation experiments for every
target and measure the success of the episodes if the robot reaches the
selected object category in less than 150 discrete actions, and without
collisions. This limit of 150 steps was chosen to establish a balance between
the average size of the houses typically used in Habitat (Szot et al., 2021)
and the apartment used in our experiments. We report the success rate (SR) of
the VSN models as the percentage of episodes in which navigation was
successful. An experiment is considered successful if the robot stops (the VSN
model samples the action stop) and the Euclidean distance to the target object
is less than one meter. Note that our navigation experiment mimics the
evaluation performed in the ObjectNav (Batra et al., 2020) task, with the same
metric. This is the standard experiment on which most VSN models are compared
and which currently defines the state of the art.

We have used two different robots for our experiments: a Turtlebot 2, and the
LOLA2 (Nasri et al., 2022) platform. For doing so, we had to embed our ROS4VSN
in both of these platforms. This can be easily done adapting the robot_api
module described in Section 3.1.1. Figure 8 shows the family picture of the
robots invited to our experiments. We mainly used the Turtlebot 2 for the
navigation experiments in the apartment described. The robot LOLA2 was also
used in navigation experiments, but in a different location, in order to test
the stability of the developed system, and to provide a study that contains
more hours of navigation, and on different platforms. Our intention in using
two different platforms has been to provide evidence of the generalizability
of our architecture, demonstrating that it can be tested on different robots.

(a) Turtlebot 2.

(b) LOLA2 platform.

Figure 8: Pictures of the robots used in our experiments.

When collecting information during the experiments, we developed a procedure
to record relevant data for each trial. This procedure stores the unique
identifier for each episode, the sequence of actions performed and the
category of object searched. In addition, during the tests, qualitative
information about the trajectory followed by the robot was recorded. In
particular, all the images observed by the robot during its trajectories have
been saved.

Figure 9: Hardware-software architecture for the development of experiments.

For the experiments, the following hardware-software setup has been employed,
as it is shown in Figure 9. The modular architecture of the developed ROS4VSN
system was used to deploy it in a distributed manner. This architecture allows
separating the execution of ROS packages on different devices, as long as they
are connected to the same network. The robotic platforms were equipped with a
laptop. This device was used to establish the communication with the robotic
platform and the camera by executing the robot_api and camera_api packages. At
the same time, the discrete_move package was executed to receive the actions
to be executed by the robot. On the other hand, we used a workstation to run
the VSN nodes, which was equipped with an i7-1165G7 processor and an NVIDIA
GeForce RTX 2060 graphics card.

###  4.2 VSN navigation results

We provide in this section the main results obtained during the navigation
experiments with our robots, including both quantitative and qualitative
results.

Following the experimental setup detailed in Section 4.1, we have obtained the
following results. Remember that we provide in this study an analysis of the
SR for two state-of-the-art VSN models, running in a Turtlebot 2 platform. For
every VSN model, _i.e_. VLV and PIRLNav, we have measured their SR for the
different object categories they can navigate to.

Table 2 compares the performance obtained by VLV and PIRLNav approaches when
they are tested in the real world (_i.e_. our experiments) and in a virtual
environment (_i.e_. the experiments reported in their respective papers). The
first thing we observe is the difference in terms of SR. The SR for the
PIRLNav model drops from $65\%$ to $21\%$, while the VLV model loses $\sim 10$
percentage points in this metric. One of the conclusions of our study is that
there is a considerable gap between the behavior of these models in the real
world and in the simulation environments in which they are trained. This
indicates that further research in this direction is needed. Interestingly,
the results obtained in our real-world experiments are not consistent with the
performance difference that already existed between the models in the
simulation environments: VLV is the winner in the real world! As we analyze in
the discussion section below (See section 4.4), we believe that this behavior
is due to the impact of the object detector that VLV integrates, but PIRLNav
does not. While the difference between VLV and PIRLNav SR in the virtual
environments is of $26$ percentage points, in the real world this gap becomes
of just $8$ percentage points.

Models | SR (Real World) | SR (Virtual Environment)
---|---|---
VLV (Chang et al., 2020) | $29.33\%$ | $39\%$
PIRLNav (Ramrakhya et al., 2023) | $21.11\%$ | $65\%$
Table 2: Real world success rate against simulation.

In the following, we analyze in detail the results reported by each of the
models. We start with VLV model. Table 3 reports the SR for every of the
target categories used in our experiments. Chairs, tables and sofas are the
categories that are easiest to navigate to. In analyzing various trials with
the robot using the VLV model with ROS4VSN, there were no successful outcomes
from starting positions 10 and 12 (see Figure 7). Additionally, only one
success was observed from positions 1, 2, and 13. Notably, our VLV
implementation can reach most targets in under 60 steps.

Object Goal | Successful episodes | SR | Avg. number of actions
---|---|---|---
Chair | 6/15 | 40% | 30
Sofa | 6/15 | 40% | 65
Table | 6/15 | 40% | 42
Bed | 3/15 | 20% | 39
Toilet | 1/15 | 6,67% | 42
Table 3: VLV VSN experiment. We report the number or successful episodes over
15, the corresponding SR per object goal, and the average number of actions
taken to reach the target.

Figure 10 shows qualitative results for four navigation experiments with VLV
model. We provide five representative images of the navigation experiments.
Two successful and two unsuccessful cases are presented. In the first
experiment (first row), the robot quickly reached the Table, as the detector
easily identified it in the images. In the second experiment, the robot took a
detour to the Chair because the detector failed to detect it initially. The
model predicted a point near the chair but out of view, causing the robot to
move closer only after it became visible. In the third and fourth experiments,
the robot started in a challenging position in front of a refrigerator. Due to
noise in the depth image, the target point calculation was inaccurate, leading
to collisions with the wall in both cases.

Figure 10: VLV qualitative navigation results. The first two rows show two
successful cases, where the robot reached the target, while the last two rows
show two situations where the navigation experiment failed.

We analyze now in detail the results reported by PIRLNav model. Table 4 shows
the SR obtained for the PIRLNav model integrated into our ROS4VSN
architecture. We detail the SR reported for every object category. The agent
was able to more easily locate the most common objects in the house, such as
the chair and the monitor. The abundant and well-distributed presence of these
objects facilitated the agent’s work. The model inferred the action stop on
both objects a total of five times. Large objects, such as the sofa and bed,
showed slightly lower results. Although easily visible from multiple locations
in the dwelling, the presence of only one of these objects made it difficult
for the robotic agent to spot them. The number of times the stop action was
sampled for these categories was substantially reduced. With the toilet we
have one of the most complex challenges in navigating the robot in this home.
The robot did not have full visibility of the toilet until it managed to fully
enter the bathroom, having to pass through the narrow door without colliding.
Finally, for the category plant, the robot was not able to locate this
category in any of the 15 attempts. Even though the plant was visible on
multiple occasions during navigation, the agent did not manage to head towards
this object. Overall, considering all the categories, the SR for the PIRLNav
model is of $21.11\%$.

Object Goal | Successful episodes | SR | Avg. number of actions
---|---|---|---
Chair | 5/15 | 33,33% | 49
Monitor | 5/15 | 33,33% | 91
Sofa | 5/15 | 33,33% | 70
Bed | 3/15 | 20,00% | 97
Toilet | 1/15 | 6,67% | 61
Plant | 0/15 | 0,00% | 82
Table 4: PIRLNav VSN experiment. We report the number or successful episodes
over 15, the corresponding SR per object goal, and the average number of
actions taken to reach the target.

To conclude our analysis of the PIRLNav model, we provide some qualitative
results. Figure 11shows the navigation trajectories for four different
experiments. We provide five representative images of the navigation
experiments. Two successful and two unsuccessful cases are presented.

In the first experiment, the robot starts from the refrigerator and navigates
through the house until it reaches the sofa. This episode is carried out in 69
actions and ends when the model infers the action stop in front of the sofa.
The second experiment is also a success story, but this time the robot starts
navigating from the kitchen. The robot leaves the kitchen and navigates to the
nearest chair. This episode is performed in 36 actions and ends with the stop
action determined by the model. The third experiment shows a case of
navigation failure, where the robot targeting the plant hits an obstacle. In
this episode, the robot navigates for 61 actions until it hits the couch.
Despite visualizing the plant from far away, when trying to approach it, the
robot ends up crashing. In the last episode, another case of navigation
failure is shown while the robot was trying to make its way to the bed. As it
can be seen in this episode, the model had difficulty getting through the
bathroom door without hitting itself.

Figure 11: PIRLNav qualitative navigation results. The first two rows show two
successful cases, where the robot reached the target, while the last two rows
show two situations where the navigation experiment failed.

We provide a video (see https://youtu.be/nD0JBWNCMGg) with more qualitative
results for both of the VSN models used in our experiments.

###  4.3 Stability analysis

In this section, we analyze the stability of the navigation solution proposed,
showing how it can robustly navigate a considerable distance, in two different
robots and over two different scenarios. The robots successfully navigated
over more than 5 kilometers in less than 38 hours in two dynamic environments.
The robots operated without direct assistance throughout the experiments,
being automatically operated by the VSN models integrated into our ROS4VSN
software architecture.

Robot | Time (Hours) | Distance (km)
---|---|---
LOLA2 | 8 | 1.12
Turtlebot 2 | 30 | 4.10
Total | 38 | 5.22
Table 5: Time spent and traveled distance for both robots during the
experiments.

###  4.4 Discussion

The main question we wanted to address with the designed experiment has been:
are the state-of-the-art VSN models able to successfully operate with real
robots? This implies knowing the SR that these models are capable of
delivering when tested in the real world, and not in the virtual environments
where they were trained. Note that we selected two VSN models that were
originally trained with images of the real world. Our intention was to reduce
as much as possible the influence of domain shift, which we know affects
artificial intelligence systems. Our study confirms that there is still room
for improvement so that these models can achieve the same SR in real robots.
We expect, therefore, that our ROS4VSN library plays a fundamental role in
this line of research.

Analyzing the particular behavior of the models, we can provide the following
interesting discussion. The integration of the VLV and PIRLNav models within
our ROS4VSN architecture has proven to be successful. It has resulted in a
mobile agent capable of navigating in closed environments autonomously,
obtaining numerous experiments where the robotic platforms reach the target
class without complications and following logical and direct trajectories.
This navigation is comparable to that observed within simulated environments.
For the VLV model that integrates an object detector, we have observed that
this fact has a significant impact on the agent’s navigation, especially when
it is close to the target class. Although the object detector does not
significantly affect the general exploration, its impact becomes crucial when
the robot is in the vicinity of the target. At this crucial stage of
navigation, the object detector provides a significant advantage by guiding
the robot to the target more effectively. This explains the difference of
performance we have observed between VLV and PIRLNav in the real world.

Figure 12: Histogram of navigation actions sampled by PIRLNav model.

In terms of qualitative aspects of navigation, we believe that the PIRLNav
model is better than the VLV model. Note that VLV, every time the high-level
policy has to make a decision, needs the robot to turn completely on itself,
taking 12 captures on which it will decide which direction to move forward.
This can be observed in the provided video. This feature slows down
navigation, although it could be solved with some specific hardware. In
contrast, PIRLNav offers a more direct navigation experience. It is
interesting to observe the type of action sampling that PIRLNav performs while
being executed on our robots. Figure 12 shows a histogram corresponding to the
distribution of navigation actions performed by PIRLNav. First, one can
observe that the look_up and look_down actions have hardly been selected. This
allows us to affirm that the impact of the adaptations we have made to replace
these actions by backward and forward movements, respectively, could hardly
have had a considerable impact on the final results. Second, the most popular
actions, as they are the ones that motivate the exploration of the
environment, are those of advances and turns. The stop action was sampled 31
times, that is, a $0.2\%$ as it reflects the histogram provided. We believe
that work should be done on solutions to increase the number of times the stop
action is selected, but to do so reliably.

Finally, our study confirms some of the conclusions reported in recent works,
_e.g_. (Gervet et al., 2022). Modular-learning models, such as VLV, perform
better than end-to-end learning approaches, _e.g_. PIRLNav, when tested in the
real world.

##  5 Conclusions

To conclude, we have presented a ROS-based framework for visual semantic
navigation named ROS4VSN that allows to easily test and compare different VSN
models in real robots. Using ROS4VSN, we have been able to embed two cutting-
edge VSN models into two distinct real robotic platforms. The chosen models
are PIRLNav (Ramrakhya et al., 2023) and VLV (Chang et al., 2020). To
seamlessly integrate these models, technical modifications have been needed.
These adaptations ensure a smooth transition for the models, enabling them to
shift from interacting with observations generated in simulation environments
to those obtained from the real world. We have also offered a thorough
experimental evaluation to showcase how these VSN approaches behave when
navigating in the real world. Our novel framework shows a robust stability,
being able to run for a considerable distance, in two different robots,
without any human intervention. Our study and results show that the
performance of state-of-the-art VSN models is significantly lower in the real
world than in the virtual environments where they were trained. We anticipate
that our efforts will lay the foundation for addressing this substantial
challenge.

## Acknowledgements

This research was funded by project AIRPLANE, with reference
PID2019-104323RB-C3 from the Ministry of Science and Innovation of Spain.

## References

  * Abaspur Kazerouni et al. (2022) Abaspur Kazerouni, I., Fitzgerald, L., Dooly, G., Toal, D., 2022\.  A survey of state-of-the-art on visual slam.  Expert Systems with Applications 205, 117734.
  * Agarwal et al. (2022) Agarwal, A., Kumar, A., Malik, J., Pathak, D., 2022\.  Legged Locomotion in Challenging Terrains using Egocentric Vision, in: CoRL.
  * Batra et al. (2020) Batra, D., Gokaslan, A., Kembhavi, A., Maksymets, O., Mottaghi, R., Savva, M., Toshev, A., Wijmans, E., 2020\.  ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects, in: arXiv.  doi:https://arxiv.org/abs/2006.13171, arXiv:2006.13171.
  * Campos et al. (2021) Campos, C., Elvira, R., Rodríguez, J.J.G., M. Montiel, J.M., D. Tardós, J., 2021.  Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam.  IEEE Transactions on Robotics .
  * Chang et al. (2020) Chang, M., Gupta, A., Gupta, S., 2020.  Semantic Visual Navigation by Watching Youtube Videos, in: NeurIPS.
  * Chaplot et al. (2020) Chaplot, D.S., Gandhi, D., Gupta, A., Salakhutdinov, R., 2020\.  Object Goal Navigation using Goal-Oriented Semantic Exploration, in: NeurIPS.
  * Gervet et al. (2022) Gervet, T., Chintala, S., Batra, D., Malik, J., Chaplot, D.S., 2022.  Navigating to Objects in the Real World.  Science Robotics .
  * Gutiérrez-Maestro et al. (2019) Gutiérrez-Maestro, E., López-Sastre, R.J., Maldonado-Bascón, S., 2019.  Collision anticipation via deep reinforcement learning for visual navigation, in: IbPRIA.
  * He et al. (2017) He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017\.  Mask R-CNN, in: ICCV.
  * Huang et al. (2023) Huang, W., Xia, F., Shah, D., Driess, D., Zeng, A., Lu, Y., Florence, P.R., Mordatch, I., Levine, S., Hausman, K., Ichter, B., 2023\.  Grounded decoding: Guiding text generation with grounded models for robot control.  ArXiv .
  * Hwangbo et al. (2019) Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., Hutter, M., 2019.  Learning agile and dynamic motor skills for legged robots.  Science Robotics .
  * Jin et al. (2023) Jin, S., Dai, X., Meng, Q., 2023.  “focusing on the right regions” — guided saliency prediction for visual slam.  Expert Systems with Applications 213, 119068.
  * Jones and Soatto (2011) Jones, E.S., Soatto, S., 2011\.  Visual-inertial navigation, mapping and localization: A scalable real-time causal approach.  The International Journal of Robotics Research 30, 407–430.  URL: https://doi.org/10.1177/0278364910388963, doi:10.1177/0278364910388963, arXiv:https://doi.org/10.1177/0278364910388963.
  * Kadian et al. (2020) Kadian, A., Truong, J., Gokaslan, A., Clegg, A., Wijmans, E., Lee, S., Savva, M., Chernova, S., Batra, D., 2020.  Sim2Real predictivity: Does evaluation in simulation predict real-world performance?  IEEE Robotics and Automation Letters .
  * Khandelwal et al. (2022) Khandelwal, A., Weihs, L., Mottaghi, R., Kembhavi, A., 2022\.  Simple but Effective: CLIP Embeddings for Embodied AI, in: CVPR.
  * Kim et al. (2022) Kim, M., Ladosz, P., Oh, H., 2022.  Monocular vision-based time-to-collision estimation for small drones by domain adaptation of simulated images.  Expert Systems with Applications 199, 116973.
  * Kondoh and Kanezaki (2023) Kondoh, H., Kanezaki, A., 2023\.  Multi-goal audio-visual navigation using sound direction map.  ArXiv .
  * Labbé and Michaud (2022) Labbé, M., Michaud, F., 2022\.  Multi-session visual slam for illumination-invariant re-localization in indoor environments.  Frontiers in Robotics and AI .
  * Liu et al. (2022) Liu, X., Guo, D., Liu, H., Sun, F., 2022.  Multi-agent embodied visual semantic navigation with scene prior knowledge.  IEEE Robotics and Automation Letters 7, 3154–3161.  doi:10.1109/LRA.2022.3145964.
  * Ltd. (2023a) Ltd., K., 2023a.  ROS wrapper for Kobuki base Turtlebot 2.  URL: https://github.com/yujinrobot/kobuki.git.
  * Ltd. (2023b) Ltd., O., 2023b.  ROS wrapper for Astra camera.  URL: https://github.com/orbbec/ros_astra_camera.
  * Luo et al. (2021) Luo, Q., Sorokin, M., Ha, S., 2021.  A few shot adaptation of visual navigation skills to new observations using meta-learning, in: ICRA, pp. 13231–13237.
  * Mousavian et al. (2018) Mousavian, A., Toshev, A., Fiser, M., Kosecka, J., Davidson, J., 2018.  Visual representations for semantic target driven navigation.  ICRA .
  * Nasri et al. (2022) Nasri, N., López-Sastre, R.J., da Costa, S.P., Fernández-Munilla, I., Gutiérrez-Álvarez, C., Pousada-García, T., Acevedo-Rodríguez, F.J., Maldonado-Bascón, S., 2022.  Assistive robot with an ai-based application for the reinforcement of activities of daily living: Technical validation with users affected by neurodevelopmental disorders.  Applied Sciences .
  * Newcombe et al. (2011) Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A., 2011\.  KinectFusion: Real-time dense surface mapping and tracking, in: International Symposium on Mixed and Augmented Reality.
  * Quigley et al. (2009) Quigley, M., Gerkey, B., Conley, K., Faust, J., Foote, T., Leibs, J., Berger, E., Wheeler, R., Ng, A., 2009.  ROS: an open-source robot operating system, in: ICRA, Workshop on Open Source Robotics.
  * Ramakrishnan et al. (2021) Ramakrishnan, S.K., Gokaslan, A., Wijmans, E., Maksymets, O., Clegg, A., Turner, J., Undersander, E., Galuba, W., Westbury, A., Chang, A.X., Savva, M., Zhao, Y., Batra, D., 2021.  Habitat-Matterport 3D Dataset (HM3D): 1000 large-scale 3D environments for embodied AI, in: NeurIPS.
  * Ramrakhya et al. (2023) Ramrakhya, R., Batra, D., Wijmans, E., Das, A., 2023\.  PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav, in: CVPR.
  * Ramrakhya et al. (2022) Ramrakhya, R., Undersander, E., Batra, D., Das, A., 2022.  Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale, in: CVPR.
  * Rosinol et al. (2020) Rosinol, A., Abate, M., Chang, Y., Carlone, L., 2020\.  Kimera: an open-source library for real-time metric-semantic localization and mapping.  ICRA .
  * Sadeghi and Levine (2017) Sadeghi, F., Levine, S., 2017\.  CAD2RL: Real Single-Image Flight Without a Single Real Image, in: Robotics: Science and Systems.
  * Sattler et al. (2018) Sattler, T., Maddern, W., Toft, C., Torii, A., Hammarstrand, L., Stenborg, E., Safari, D., Okutomi, M., Pollefeys, M., Sivic, J., Kahl, F., Pajdla, T., 2018\.  Benchmarking 6dof outdoor visual localization in changing conditions, in: CVPR.
  * Sethian (1996) Sethian, J.A., 1996.  A fast marching level set method for monotonically advancing fronts, in: Proceedings of the National Academy of Sciences.
  * Shah et al. (2022) Shah, D., Bhorkar, A., Leen, H., Kostrikov, I., Rhinehart, N., Levine, S., 2022\.  Offline reinforcement learning for visual navigation, in: CoRL.
  * Son et al. (2020) Son, D., Yang, H., Lee, D., 2020.  Sim-to-real transfer of bolting tasks with tight tolerance, in: IROS.
  * Szot et al. (2021) Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., Maestre, N., Mukadam, M., Chaplot, D.S., Maksymets, O., Gokaslan, A., Vondruš, V., Dharur, S., Meier, F., Galuba, W., Chang, A., Kira, Z., Koltun, V., Malik, J., Savva, M., Batra, D., 2021.  Habitat 2.0: Training home assistants to rearrange their habitat, in: NeurIPS.
  * Thrun et al. (2001) Thrun, S., Fox, D., Burgard, W., Dellaert, F., 2001\.  Robust Monte Carlo localization for mobile robots.  Artificial Intelligence 128, 99–141.  doi:10.1016/S0004-3702(01)00069-8.
  * Wang et al. (2023) Wang, H., Wang, Y., Zhong, F., Wu, M., Zhang, J., Wang, Y., Dong, H., 2023.  Learning semantic-agnostic and spatial-aware representation for generalizable visual-audio navigation.  IEEE Robotics and Automation Letters .
  * Wijmans et al. (2020) Wijmans, E., Kadian, A., Morcos, A., Lee, S., Essa, I., Parikh, D., Savva, M., Batra, D., 2020\.  DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames, in: ICLR.
  * Wijmans et al. (2019) Wijmans, E., Kadian, A., Morcos, A.S., Lee, S., Essa, I., Parikh, D., Savva, M., Batra, D., 2019\.  DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames, in: ICLR.
  * Wortsman et al. (2019) Wortsman, M., Ehsani, K., Rastegari, M., Farhadi, A., Mottaghi, R., 2019.  Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning.  CVPR , 6743–6752doi:10.1109/cvpr.2019.00691.
  * Yadav et al. (2023a) Yadav, K., Majumdar, A., Ramrakhya, R., Yokoyama, N., Baevski, A., Kira, Z., Maksymets, O., Batra, D., 2023a.  Ovrl-v2: A simple state-of-art baseline for imagenav and objectnav.  ArXiv .
  * Yadav et al. (2023b) Yadav, K., Ramrakhya, R., Majumdar, A., Berges, V.P., Kuhar, S., Batra, D., Baevski, A., Maksymets, O., 2023b.  Offline visual representation learning for embodied navigation, in: ICLR.
  * Yang et al. (2018) Yang, W., Wang, X., Farhadi, A., Gupta, A.K., Mottaghi, R., 2018.  Visual semantic navigation using scene priors.  ICLR .
  * Ye et al. (2021) Ye, J., Batra, D., Das, A., Wijmans, E., 2021.  Auxiliary tasks and exploration enable ObjectGoal navigation, in: ICCV.
  * Zhang et al. (2018) Zhang, L., Wei, L.Y., Shen, P., Wei, W., Zhu, G., Song, J., 2018.  Semantic slam based on object detection and improved octomap.  IEEE Access .
  * Zhang et al. (2022) Zhang, S., Li, W., Song, X., Bai, Y., Jiang, S., 2022\.  Generative meta-adversarial network for unseen object navigation, in: ECCV.
  * Zhou et al. (2023) Zhou, K.Q., Zheng, K., Pryor, C., Shen, Y., Jin, H., Getoor, L., Wang, X.E., 2023.  ESC: Exploration with soft commonsense constraints for zero-shot object navigation.  ArXiv .
  * Zhu et al. (2017) Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., Farhadi, A., 2017.  Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning, in: ICLR.