Further author information: (Send correspondence to N.S)
.: E-mail<EMAIL_ADDRESS>
# High contrast imaging wavefront sensor referencing from coronagraphic images

Nour Skaf  LESIA, Observatoire de Paris, Univ.PSL, CNRS, Sorbonne Univ.,
Univ.de Paris, 5 pl. Jules Janssen, 92195 Meudon, France  National
Astronomical Observatory of Japan, Subaru Telescope, 650 North A’ohōkū Place,
Hilo, HI 96720, U.S.A.  Department of Physics and Astronomy, University
College London, London, United Kingdom  Olivier Guyon  National Astronomical
Observatory of Japan, Subaru Telescope, 650 North A’ohōkū Place, Hilo, HI
96720, U.S.A.  Astrobiology Center of NINS, 2-21-1 Osawa, Mitaka, Tokyo
181-8588, Japan  Steward Observatory, University of Arizona, Tucson, AZ 85721,
USA  College of Optical Sciences, University of Arizona, Tucson, AZ 85721, USA
Anthony Boccaletti  LESIA, Observatoire de Paris, Univ.PSL, CNRS, Sorbonne
Univ., Univ.de Paris, 5 pl. Jules Janssen, 92195 Meudon, France  Vincent Deo
National Astronomical Observatory of Japan, Subaru Telescope, 650 North
A’ohōkū Place, Hilo, HI 96720, U.S.A.  Sébastien Vievard  National
Astronomical Observatory of Japan, Subaru Telescope, 650 North A’ohōkū Place,
Hilo, HI 96720, U.S.A.  Astrobiology Center of NINS, 2-21-1 Osawa, Mitaka,
Tokyo 181-8588, Japan  Julien Lozi  National Astronomical Observatory of
Japan, Subaru Telescope, 650 North A’ohōkū Place, Hilo, HI 96720, U.S.A.
Kyohoon Ahn  National Astronomical Observatory of Japan, Subaru Telescope, 650
North A’ohōkū Place, Hilo, HI 96720, U.S.A.  Barnaby Norris  Sydney Institute
for Astronomy, School of Physics, University of Sydney, NSW 200  Thayne Currie
NASA-Ames Research Center, Moffett Field, California, USA  Eric Gendron
LESIA, Observatoire de Paris, Univ.PSL, CNRS, Sorbonne Univ., Univ.de Paris, 5
pl. Jules Janssen, 92195 Meudon, France  Arielle Bertrou-Cantou  LESIA,
Observatoire de Paris, Univ.PSL, CNRS, Sorbonne Univ., Univ.de Paris, 5 pl.
Jules Janssen, 92195 Meudon, France  Florian Ferreira  LESIA, Observatoire de
Paris, Univ.PSL, CNRS, Sorbonne Univ., Univ.de Paris, 5 pl. Jules Janssen,
92195 Meudon, France  Arnaud Sevin  LESIA, Observatoire de Paris, Univ.PSL,
CNRS, Sorbonne Univ., Univ.de Paris, 5 pl. Jules Janssen, 92195 Meudon, France
Fabrice Vidal  LESIA, Observatoire de Paris, Univ.PSL, CNRS, Sorbonne Univ.,
Univ.de Paris, 5 pl. Jules Janssen, 92195 Meudon, France

###### Abstract

A key challenge of high contrast imaging (HCI) is to differentiate a speckle
from an exoplanet signal. The sources of speckles are a combination of
atmospheric residuals and aberrations in the non-common path. Those non-common
path aberrations (NCPA) are particularly challenging to compensate for as they
are not directly measured, and because they include static, quasi-static and
dynamic components. The proposed method directly addresses the challenge of
compensating the NCPA. The algorithm DrWHO - Direct Reinforcement Wavefront
Heuristic Optimisation - is a quasi-real-time compensation of static and
dynamic NCPA for boosting image contrast. It is an image-based lucky imaging
approach, aimed at finding and continuously updating the ideal reference of
the wavefront sensor (WFS) that includes the NCPA, and updating this new
reference to the WFS. Doing so changes the point of convergence of the AO
loop. We show here the first results of a post-coronagraphic application of
DrWHO. DrWHO does not rely on any model nor requires accurate wavefront sensor
calibration, and is applicable to non-linear wavefront sensing situations. We
present on-sky performances using a pyramid WFS sensor with the Subaru
coronagraph extreme AO (SCExAO) instrument.

###### keywords:

Adaptive Optics, Direct Imaging, Exoplanets

##  1 INTRODUCTION

Adaptive Optics (AO) instrumentation has undergone extensive growth in
sophistication and scientific capabilities over the last 30 years, enabling
high-contrast imaging technologies, with extreme AO (ExAO) systems consisting
of $\approx$2000-actuator DMs driven at over 1 kHz speeds [1, 2]. This has
lead to key scientific breakthroughs, including the first direct image of a
planet-mass companion [3] and the first imaged system of jovian exoplanets,
HR8799 [4].

Some of the current generation of ExAO systems, e.g. the Subaru Coronagraphic
Extreme Adaptive Optics (SCExAO) at the Subaru Telescope [5] and MagAO-X on
the Magellan Clay telescope [6] are pushing further achievable performance –
e.g. Strehl ratio (SR), contrast and sensitivity [7, 8, 9, 10, 11]– and serve
as technology prototypes for ExAO instruments on 25-40 m extremely large
telescopes [12, 13, 14, 15].

The ExAO loop corrects wavefront aberrations measured by the WFS, ideally
approaching an aberration-free image. However, the ExAO loop convergence
point, defined by the WFS reference, may not correspond to the optimal image
quality. Reasons for such a discrepancy include:

  * •

Non-common path aberrations (NCPA) due to optics located after the beam
splitting between WFS and science paths, inducing differential aberrations
between the science camera and the WFS, as seen on Figure 1. In a high
contrast imaging system, these aberrations may include coronagraph optics
defects. These aberrations can also vary with temperature and mechanical
deformation at timescales from minutes to hours, or with the positioning
errors of moving optics, making them particularly challenging to calibrate.
They are typically of the order of tens of nanometers, enough to lead to
static and quasi static speckles in coronagraphic images [16, 17]. These
speckles are also present in non coronagraphic images but less concerning
because they do not contribute significantly to the SR, as usually beneath the
camera contrast limit for unsaturated images.

  * •

WFS calibration errors due for example to a non-flat wavefront used to acquire
a calibration.

  * •

Chromaticity for systems where the wavelength of the WFS and the scientific
camera are different.

  * •

Choice of optimal wavefront state. The goal of the AO system may not be to
produce a flat wavefront. For example, in a HCI, cancellation of static
diffraction features (Airy diffraction rings, telescope spiders), such as dark
hole techniques [18], may seek to drive the AO loop to a non-flat wavefront
state.

Figure 1: Schematic view an AO system, presenting the problem of NCPA,
especially in the case of exoplanet imaging with a coronagraphic image.

The phenomena responsible for the effects described above have timescales
slower than the millisecond-level atmospheric turbulence timescale, and can be
(nearly) static. In this paper, we refer to all discrepancies between WFS
reference and optimal image wavefront as NCPA, noting that in previous
publications it usually only refers to differential optical defects.

NCPA affect all AO systems, but are particularly acute in high contrast
imaging where small wavefront errors can easily mask faint exoplanet images
(cf Figure 1). Furthermore, quasi-static aberrations do not average out to a
spatially smooth halo as fast as the atmospheric speckles, leaving structures
in the coronagraphic image which resemble planets, unlike speckles generated
by atmospheric turbulence alone.

The most reliable approach to measure and correct optical misalignments is to
use focal plane images [19]. In the particular case of NCPA, these images are
ideally generated by the science detector. Several solutions have been
suggested for static NCPA [20, 21, 22, 23, 24, 18], the quasi static part
being particularly challenging to measure and correct. The evolutionary
timescale of the aberrations with respect to the frequency of the correction
is a main difficulty.

In a closed loop AO system, NCPA are usually compensated by subtracting a
biasing reference signal in the WFS reference corresponding to the aberration
to correct. Doing so requires good WFS response knowledge and stability, so
that the adequate offset can be applied and maintained. Ideally, this offset
should be updated in nearly real-time due to optical and mechanical variations
in the instrument, however in practice it is typically updated on a daily or
weekly basis. Finding the adequate offset requires focal plane images to
update the reference, and can furthermore be particularly challenging if the
WFS exhibits non-linear response that must be accounted for, as is the case in
some high-performance WFSs such as the Pyramid WFS (PyWFS) [25, 26, 27, 28].

In this paper we will present the DrWHO algorithm, a model-free focal plane
wavefront sensing approach which is aimed at finding the offset mentioned
above to correct slow and static wavefront aberrations, including the NCPA, on
a few seconds timescale. After stating the issues to be solved in Section 2,
we describe the algorithm in Section 3. The application of the algorithm to
non-coronagraphic PSF, with numerical simulations and on-sky tests, is
presented in Section 4. Section 5 presents the main results of the on-sky
validation for coronagraphic images. In Section 6, we review the
characteristics of the algorithm and mention further implications for use in
HCI.

##  2 Problem statement: defining a good WFS reference

In a closed loop AO system, the WFS signal corresponds to the WFS measurement
subtracted by the WFS reference, to isolate residual wavefront errors that
must be corrected by the DM. This reference defines the convergence point of
the AO control loop, which brings the WFS signal to where it gives a flat
wavefront after the real-time controller performs a wavefront reconstruction.

This WFS reference is usually measured with an internal light source, before
on-sky observations. In reality, however, the ideal WFS reference is different
from the internal source WFS reference due to optical illumination
discrepancies. Furthermore, the WFS reference contains a dynamic part,
constantly evolving due to quasi-static aberrations and wavefront
chromaticity. The accurate calibration of the WFS, through the reference, is
therefore a significant challenge for any AO system, on top of the fact that
the static part cannot be accurately measured.

Therefore, a continuous way to measure the ideal reference is required for
high accuracy wavefront correction.

Finding this ideal reference is especially critical to ExAO systems, where a
slight deviation can introduce slow/static speckles that appear similar to a
planet image.

##  3 Algorithm description

Because the ideal reference changes continuously, the role of DrWHO is to
update frequently, on seconds timescales, the actual reference, and ensure the
AO loop converges towards a correction of NCPA. In the context of high
contrast imaging, the algorithm does so by wisely selecting the WFS images
corresponding to the best lucky imaging of coronagraphic or non-coronagraphic
images, in terms of contrast or intensity for the former, and Strehl ratio or
other metric for the latter. Hence, the WFS and coronagraphic images first
need to be synchronized in time.

The algorithm proceeds as follows: on a defined timescale $T_{DrWHO}$ (of the
order of tens of seconds for example), the algorithm first selects the best
frames according to a score metric, and a predefined selection fraction $P$,
in percent. Then, out of those $P$% frames, the corresponding WFS raw frames
are extracted from the AO telemetry and averaged; the resulting WFS frame
replaces the WFS actual reference, and the algorithm is iterated to
continuously optimize the WFS reference.
Hence, as DrWHO proceeds, the point of convergence of the AO loop includes
NCPA and other aberrations of a timescale longer than $T_{DrWHO}$, which are
thus compensated for. The image quality then improves and presents less
speckles features. In fact, the frequency of the aberrations that DrWHO can
correct depends on the timescale of one iteration of the algorithm, which
depends whether there is a coronagraph in place or not, and how good the
seeing conditions are.

We present the procedure step by step of DrWHO with the description in
Algorithm 1, and a schematic view of the algorithm is in Figure 2.

Data:

  * •

WFS frames

  * •

fast science camera frames

Result: Update the reference of the WFS

Initialization: take the reference of the WFS with the internal light source;

while  _AO loop is running_ do

  * •

acquire WFS and fast science focal plane coronagraphic images;

* •

synchronize data cubes;

while  _DrWHO iteration ( $T_{DrWHO}$ (seconds)) is running_ do

  * –

select the $P$% best coronagraphic science data;

* –

select the corresponding WFS images;

* –

average selected WFS images;

* –

the resulting WFS frame replaces the WFS reference;

* –

WFS reference updated.

end while

* •

restart DrWHO iteration

Change of AO point of convergence;

end while

Algorithm 1 DrWHO algorithm Figure 2: Schematic view of DrWHO. From left to
right, as the AO loop is running, the coronagraphic images, or focal plane
images, and WFS telemetry are acquired. These data streams are then
synchronised and re-sampled to the same timescale, here spanning the length of
the DrWHO iteration, $T_{DrWHO}$ (s). On the PSF data cube, the second step of
DrWHO is to compute the score for each frame. The following step is to select
a fraction ($P$%) of the best PSF according to the quality criteria. The 4th
step is to take the corresponding WFS images, then (5th) to average them to
make one single PyWFS image : this corresponds to the new reference for the
iteration. Finally, the 6th and final step is to apply this new reference to
the PyWFS, and thus update it. This last step is the only one not transparent
to the AO loop, as it changes its point of convergence.

##  4 Non-coronagraphic results

Most of those results have been extensively presented in 29, we present here a
summary of those results.

###  4.1 Validation via numerical simulation

####  4.1.1 Simulation setup

Numerical simulations have been performed with the AO simulator COMPASS [30].
Compass is a versatile AO simulator, designed to meet the need of high
performance for the simulation of AO systems, modeling several kinds of AO
features, including several types of WFS, atmospheric simulations, DMs,
telescopes, and RTCs. DrWHO was first implemented on the COMPASS software, to
explore the algorithm feasibility and potential. Table 1 synthetizes the
parameters used in the COMPASS simulations.

Table 1: ExAO numerical simulations parameters.  Numerical simulation
configuration
---
Telescope |  $D$ = 8. m diameter
No support spiders
Central Obstruction = 0.12
Turbulence layer | von Kármán, ground layer only
$r_{0}$ = 0.16 at 500 nm
L0 = 20 m
(to simulate post-woofer residuals)
$||\overrightarrow{\mathbf{v}}||$ = 10 m.s-1
PyWFS |
Subapertures | 64$\times$64
Wavelength | Monochromatic, 750 nm
Guide star magnitude | 7
Modulation | Circular, 3 $\frac{\lambda}{D}$ radius, 24 points
Photon noise only |
Deformable mirror |
| 48 actuators across the diameter
Focal Plane Camera |
Wavelength | Monochromatic, 1.65 $\mu$m
Photon noise only |
RTC controller |
Loop rate | 2 kHz
Method | Linear modal integrator
Basis | DM Karhunen-Loève (KL) basis
Loop gain | 0.3
Controlled modes | 1505
Modes filtered | 300

The simulation is idealized on several aspects. First, it does not take into
consideration the telescope spiders. Second, there is no source of noise other
than only the photon noise on both the WFS and the science detector (observing
mR=7 with 50% efficiency). Furthermore, we considered a PyWFS with a higher
sampling than on the SCExAO instrument (64 pixels over the PyWFS pupil
diameter, versus 50 pixels), hence higher performance than achieved on-sky.
The PyWFS is working in the visible at 750 nm, and the detector in the
infrared at 1650 nm (cf Table 1). Because COMPASS does not simulate
coronagraphic images, DrWHO was first tested on non-coronagraphic PSF, with
the Strehl ration as quality criteria for the image selection.

####  4.1.2 Simulation results

We performed a test with 15 DrWHO iterations of 10 000 iteration each
(corresponding to 5 seconds at 2kHz). The number of AO loop per DrWHO
iteration was constrained by simulation time. NCPA were applied on the science
path, thus not visible by the WFS. Those NCPA correspond to a linear
combination of the first 12 Karhunen-Loève modes (which is the linear and
orthogonal modal basis used for computing the response matrix), with a
randomized amplitude, and a total amplitude of 30 nm RMS, which is slightly
greater to what is expected in reality (approximately 20 nm RMS on SCExAO).

Before presenting the results in terms of SR, it is important to emphasize
that on most ExAO systems, the impact of NCPA over the SR is relatively small.
Hence, it is not the best metric to quantify the efficiency of the algorithm,
because the AO is dominated by dynamical wavefront residuals. In fact, NCPA
are usually small enough that they do not affect the SR significantly relative
to dynamical wavefront residuals. Their main contribution is to add quasi-
static speckles in the focal plane, which mimic exoplanets signal. This is
even more true when a coronagraph is used. However, the SR is a good metric to
make sure the algorithm does not diverge, in particular through the high-order
modes as they could adopt a random walk behavior. Furthermore, the simulation
presents a simplified system where a coronagraph is not simulated, hence the
SR remains useful.

Table 2 and Figure 3 present the result of the SR after the DrWHO run: the
final SR after the 15-iteration DrWHO run reaches 87.7%, with a maximum of
87.9% when there are no NCPA. As shown on Figure 3, nearly half of the SR
improvement is obtained within the first iteration of DrWHO. Then, the SR
keeps slowly improving, until it reaches a plateau. After 15 iterations, the
SR reached is almost the same as the SR measured without NCPA.

Table 2: Comparison of the long exposure (LE) Strehl ratio between the following cases of the AO loop: no NCPA, NCPA alone, and post DrWHO with the NCPA. The second line corresponds to the RMS of the wavefront for each of those cases, calculated from the SR of the line above with the Maréchal approximation. | No NCPA | with NCPA | with DrWHO
---|---|---|---
SR LE (%) | 87.9 | 86.8 | 87.7
$\sigma_{OPD}$ (nm RMS) | 94.3 | 98.8 | 95.1
Figure 3: Evolution of the long exposure (LE) Strehl ratio during the DrWHO
run on COMPASS, corresponding to 15 DrWHO iterations of 10 000 AO loop
iterations each.

In order to have a better understanding of the correction in terms of modes,
the 15 references for each iteration of the DrWHO run were projected over the
modal basis with which the response matrix was acquired. Figure 4 shows this
modal decomposition along with the ideal NCPA correction, where the opposite
of the NCPA was added, to better compare the correction. It appears from the
top plot of this figure that, as we previously concluded, the algorithm
converges from the first iteration towards a compensation of the NCPA. The
average of all those references (minus the reference 0, which is the initial
reference) is neighbouring the NCPA aberrations. This proves that DrWHO
measured them, with the correct amplitude, and compensated for them. The
middle and bottom plots of this figure prove that the algorithm does not
diverge at the higher order modes, as confirmed by the increase of SR. The
quadratic sum of 12 modes from the mean of all the references from the
algorithm corresponds to 36 nm RMS, which is roughly consistent with the
amplitude of the NCPA.

Figure 4: Top: projection of the references provided by the 15-iteration DrWHO
run over the modal basis, with the projection of the opposite of the NCPA, in
order to better visualise the correction. Bottom: projection over the first
100 modes. The trend for the higher order modes is the same.

To conclude the simulation results, DrWHO on COMPASS improved the image
quality in terms of SR, and the NCPA applied over the PSF, were almost totally
corrected. Half of the correction has occurred at the first iteration of the
algorithm. The projection of the references from DrWHO is showing that the
NCPA are nearly entirely compensated for, and the higher orders, where we did
not apply NCPA, are not diverging.

###  4.2 Validation with on-sky tests

DrWHO was then adapted and deployed on the SCExAO instrument. This instrument
is equipped with a Pyramid WFS operating in the 600-950 nm wavelength range
[31]. The real time control (RTC) of the AO system is managed by the CACAO
software - Compute And Control for Adaptive Optics - [32]. The DM has 45
actuators across the pupil, giving a control radius of 22.5$\lambda$/D. We use
the CACAO software to interact with the system and implement the algorithm, to
communicate between the PyWFS and the DM. Several scientific modules are
downstream SCExAO. For testing DrWHO on non-coronagraphic images, the two
VAMPIRES camera were used, operating in the visible [33]. The PyWFS and the
VAMPIRES camera run at different frequencies, with typical frequencies of 2kHz
for the former, and between 200 Hz to 2kHz for the latter, hence there is the
need to synchronize the data streams to run DrWHO.

####  4.2.1 PSF quality criteria

The Flux Concentration (FC) has been introduced in 29 as the quality criteria
for quantifying the results of the DrWHO run. It is defined as

$\centering\text{FC}=\frac{\sum_{i}(x_{i}^{\alpha})}{(\sum_{i}x_{i})^{\alpha}},\@add@centering$
(1)

with $x_{i}$ being the value of the pixel i, and $\alpha>1$ \- we will use
$\alpha$ = 2. If some pixels have a negative value following a dark
subtraction due to camera readout noise, then the pixel value is set to zero.
This quantity is independent of absolute flux, since it is normalized by the
total flux as seen in the denominator of equation 1.

The FC can be normalised by the FC of a simulated perfect PSF—which we will
call FC0 \- computed for the pupil of the telescope (cf Figure 5), as the FC
is a relative number. This new quantity is referred to as the normalized FC
(nFC): nFC = FC / FC0.

Figure 5: Left: simulated pupil plane, including the spiders and a mask for
two dead actuators. Right: simulated perfect SCExAO PSF

####  4.2.2 Observation setup

DrWHO was tested on-sky on the engineering night of December 8 UT, 2020. 20
iterations of the algorithm were performed, each of 30 seconds. Each iteration
entails synchronized data cubes resampled to a common 1 kHz frame rate for
both VAMPIRES and PyWFS images, running at 500Hz and 2kHz respectively. 1200
modes were corrected. The observation conditions at that time were a seeing of
0.5”, and a wind speed of 4 m/s. The observed target was the star $\xi$ Leo,
at an airmass of 1.03. The window size of the image for the selection of the
quality criteria was of 70x70 pixels, corresponding to 0.43x0.43 arcsec field
of view.

####  4.2.3 PSF quality evolution

The comparison of the PSFs before, after the fist iteration, and after the
last iteration of the DrWHO run is shown in Figure 6. The evolution of the nFC
over the 20 DrWHO iterations is shown in Figure 8. We plotted the evolution of
the nFC for the 10% best PSFs that were selected by the algorithm for the new
reference computation, as well as the nFC for the average of all the PSFs for
each iterations. The difference of those nFC is shown in Figure 8.
Furthermore, the Modulation Transfer Function (MTF) has been calculated for
the PSFs before and after the DrWHO run, and is shown in Figure 9. The MTF of
the ideal simulated PSF has been added to the figure. This figure shows as
well that most of the correction is done at the first iteration.

Here are the points highlighted in 29:

  * •

After 20 iterations, the PSF visually looks better and more circular,
attesting a partial correction of the low orders;

  * •

The major correction is applied at the first iteration, with the nFC jumping
from 24.3% to 35%, comparably to simulations;

  * •

The FC keeps increasing throughout the run, as shown by the best linear fit of
the two curves, which is something that was not observed in simulations, where
the evolution was smoother and reached a plateau ;

  * •

Considering PSFs averaged over each iteration, nFC increases from 28%, to 35%
at the first iteration, to then 40%: the overall improvement is 15.7% ;

  * •

The best DrWHO images present a sharper increase than all the images together
: Figure 8 presents the difference of these two nFC over the algorithm run.
Despite selecting only the best 10% frames, DrWHO seems to be improving both
the selected and unselected frames. We note that the algorithm has not fully
converged yet and would have performed even better on a longer period of time
;

  * •

Finally, Figures 8 and 8 show that even if the PSFs are subject to the
variations in seeing over a few minutes, they overall get better in terms of
the concentration of flux. Hence, DrWHO corrects the NCPA even if they are not
as dominant as the dynamical wavefront residuals from atmospheric turbulence.

Figure 6: Evolution of the on-sky PSF before running the algorithm, after the
first iteration, and the after last iteration. Each image is 0.25 arcsec
(40x40 pixels) across, acquired at $\lambda=$ 750 nm, 30 sec exposure time
(computed by co-addition of 15,000 frames acquired at 500 Hz). Those PSFs
should be compared to the ideal PSF in Figure 5, with matching wavelength,
field of view and orientation of Figure 6 shown in logarithmic brightness
scale.

Figure 7: Evolution of the FC over 20 iterations of DrWHO, corresponding to a
period of 10 minutes, including the nFC of the PSF preceding the run. The blue
curve corresponds to the evolution of the nFC of all the PSF averaged over the
DrWHO iteration, while the green one corresponds to the evolution of the nFC
of the 10% PSF chosen by DrWHO. The best linear fit is presented.

Figure 8: Difference between the nFC of the mean of the best 10% images, and
the nFC of the mean of all the PSFs, over 20 iterations of DrWHO. We
calculated the best linear fit to better understand the increase of the
difference. Figure 9: MTF of the PSF before running DrWHO, of the 1st
iteration, and the last iteration (cf Figure 6), compared to the ideal
simulated Vampires PSF.

####  4.2.4 WFS reference evolution

Figure 10 compares the WFS reference prior to the DrWHO run, Ref 0, which is
simply the WFS mask showing which pixels are active (i.e. respond to DM pokes)
in the WFS image, and the average of the last 3 WFS references applied by
DrWHO, during the on-sky 10-minute run. The averaging minimises the noise
contribution. The difference between the two references is also shown in
Figure 10. This difference was then converted to wavefront mode coefficients
by multiplication by the control matrix. Modal coefficients are shown in
Figure 12 where the first three modes are tip, tilt, and focus and the
following modes correspond to orthogonal modes, optimized for SCExAO’s AO
control law.

The total contribution of all modes is measured to be 116 nm RMS. However, as
seen on Figure 12, the high order modes, corresponding to the higher spatial
frequencies, seem to be noise more than actual correction, which will be
further explored thereafter.

When calculating the total contribution of the modes by removing the modes
beyond 900 (roughly where the modes amplitude starts increasing), we compute a
correction of 70 nm RMS.

Figure 10: Left: images of the PyWFS reference before the first iteration,
referred to as Ref0, set equal to the WFS mask. Middle: average of the last 3
iterations of DrWHO WFS references. The contrast scale for this image has been
modified for better visualisation. Right: difference between the two images in
the left and in the middle.

Figure 12 (top) shows the first 20 wavefront mode values, excluding tip-tilt.
Focus is the dominant contribution to the aberrations, with an amplitude of
nearly 14 nm, while the other modes have values below 5 nm. We furthermore
note that the higher modes from approximately the mode 900 to 1217,
corresponding to higher frequencies, seem to have a considerable contribution
in the overall correction. One would wonder whether those higher order
contributions are just random noise, or meaningful behavior. We plotted the
evolution of one example of a high mode (arbitrarily chosen for this paper to
be the mode 1120), in the bottom of Figure 12. After the first iteration, the
algorithm converges instantly towards a value of about -9 nm, with a standard
deviation of approximately 2 nm RMS, which probably corresponds to a noise
contribution. This observation is valid for the entirety of the high order
modes. Table 3 presents the contribution of the modes in nm RMS, in the
following cases: in considering all the modes except tip-tilt (from 2 to 1217)
; then in removing the higher order modes (from 2 to 900 only) to show what
the contribution would be without the higher spatial frequencies ; then the
low order modes (from 2 to 20), and finally the focus contribution, which is
13.9 nm.

Figure 11: Modal decomposition of DrWHO correction, computed from the
difference between the pre-DrWHO WFS reference and the average of the WFS
references for the last 3 DrWHO iterations.

Figure 12: Top: modal decomposition of DrWHO correction for the first 20 modes, excluding tip-tilt. Focus is the major contributor with an amplitude of 13.9 nm RMS. Middle: evolution of the focus WFS reference correction during the DrWHO run. Bottom: evolution of high order mode 1120 WFS reference correction during the DrWHO run.  Modes | 2 to 1217 | 2 to 900 | 2 to 20 | Focus
---|---|---|---|---
| Contribution
---
(nm RMS)
116.3 | 70.3 | 17.2 | 13.9
Table 3: Contribution of the modes of the difference of Figure 8 in nm RMS

##  5 On-sky preliminary coronagraphic tests

After successful tests on non-coronagraphic images, DrWHO has been adjusted to
fit coronagraphic images requirements. The only difference of the algorithm
lies in the selection criteria for the images, from the maximization of the FC
on PSF, to the minimization of the intensity on coronagraphic images, in order
to optimize the contrast. The fast focal plane camera was the science viewing
camera, running on the IR on H band, which is an InGaAs array.
We present here some preliminary results, one should however keep in mind that
tests are still ongoing.

###  5.1 Observation setup

DrWHO was tested on-sky on coronagraphic images during the engineering night
of July 14 UT, 2021. The successful test was of 5 iterations, each of 30
seconds. 1200 modes were corrected. The observation conditions at that time
were a seeing of 1.2”, and a wind speed of 40 m/s. The observed target was the
star Vega. The window size of the image for the selection of the quality
criteria was of 50x50 pixels, corresponding to 0.81x0.81 arcsec field of view.

####  5.1.1 Image quality evolution

The goal is here to optimize the contrast of the image. Hence, the selection
criteria for DrWHO becomes the minimization of the intensity, which is the
metric we are focusing on.

The comparison of the coronagraphic images before and after the DrWHO run is
shown in Figure 13, left and middle. The right part presents the difference of
the images before and after the run, normalized by the pre-DrWHO image. This
shows that the center of the image is where most of the correction is made, in
other words the essential of the correction is done over the lower order
modes, which is the main contribution of the NCPA. The edges of this figure
correspond to the higher order modes, highlighting the high seeing during this
test. This is indicative of a gain, however further tests are needed to
quantify it. The evolution of the intensity over the 5 DrWHO iterations is
shown in Figure 14. We plotted the evolution of the intensity for the 10% best
coronagraphic images that were selected by the algorithm for the new reference
computation, as well as the intensity for the average of all the coronagraphic
images for each iterations.

Figure 13: Left and middle: comparison of the coronagraphic image before and
after DrWHO, in logarithmic scale. Right: difference of those two images,
normalized by the pre-DrWHO image. The dark values are negative and indicate
improvement. Figure 14: Evolution of the intensity for each iteration of the
on-sky test on coronagraphic images.

A few points are worth noting:

  * •

First, the observations conditions were considerably worse compared to the
DrWHO test on non-coronagraphic images;

  * •

Figure 13 shows that the coronagraphic images after the 5 iterations, the
contrast is better. This is confirmed by the right image of the Figure,
highlighting the correction of the low orders;

  * •

There is an improvement of the image contrast, as shown in Figure 14, from the
first iteration of the algorithm. This improvement is however relatively
minor, which is, at least partially, the result of the high seeing during the
observations;

##  6 Discussion

Below are some noticeable characteristics of the DrWHO algorithm that have
been demonstrated in this proceeding:

  1. 1.

DrWHO is robust, for both non-coronagraphic and coronagraphic use: the
algorithm slowly but surely converges towards a better compensation of static
and slow aberrations. As it is based on lucky imaging, it relies on some
realisations of residual atmospheric turbulence to reduce wavefront
aberrations. DrWHO contributes to getting closer to an absolute sensor,
driving the WFS reference to optimize images in the focal plane.

  2. 2.

It does not rely on any model, and does not rely on using the DM for probing,
making it compatible with other wavefront control techniques: DrWHO’s approach
is different from active cancellation speckle, because it simply does a
statistical selection using the natural dynamical atmospheric residuals for
probing, instead of adding artificial probes.

  3. 3.

It does not require modification of the AO loop control scheme other than WFS
reference updates.

  4. 4.

It does not make any linear assumption, so it is not concerned with non-
linearities of the system, such as WFS non-linearity.

  5. 5.

It is flexible in the parameters, whether it is the score (Strehl, contrast,
intensity, etc.), or the selection fraction and the time of the iteration,
making it adaptable to different weather conditions and AO systems with
different characteristics and scientific goals.

  6. 6.

DrWHO requires decent seeing for being efficient, otherwise the atmospheric
residuals will simply overcome the NCPA. However, tests on seeing at 1.2” show
DrWHO does not worsen the image quality, and still improves it.

  7. 7.

However, DrWHO needs a fast focal plane camera as close as possible to the
science camera, to more accurately compensate the NCPA and low aberrations.

There are numerous ways to improve the algorithm efficiency and convergence
speed. For example, a possible extension is to parallelize DrWHO in terms of
spatial frequencies. If there is a good understanding of the relationship
between the input wavefront and the focal plane, as for example a Fourier
relationship, DrWHO can optimize separate spatial frequencies independently.

##  7 Conclusion

The DrWHO algorithm has first been tested on non-coronagraphic images in the
visible, by being initially implemented on an ExAO simulator, then on the
SCExAO instrument and tested on-sky, at the Subaru Telescope. Secondly, DrWHO
was validated on coronagraphic images in the IR.

Results presented here demonstrate its ability to measure and compensate for
NCPA, considered to be a limiting factor in the detection and characterisation
of exoplanets in high-contrast imaging observations. To quantify the quality
of the PSF, we used the Strehl ratio for the simulations, and the Flux
Concentration for the SCExAO run. For coronagraphic images, we used the total
intensity of the image, which is correlated with the image contrast. For the
simulation, we observed a nearly perfect compensation of the added NCPA, at
the first iteration.

For the on-sky test, we combined the PyWFS measurements and the focal plane
images in visible from the VAMPIRES module for non-coronagraphic tests, and
from the fast focal plane IR camera for coronagraphic tests. For non-
coronagraphic tests, when DrWHO was running, the PSF significantly improved
(15.7% relative improvement in flux concentration) over 20 iterations of 30
seconds, for a total of 10 minutes. The best PSF have a sharper increase over
the run of the algorithm, reaching 48% of the simulated ideal PSF flux
concentration. The visible improvement in the image quality was confirmed with
the calculation of the MTF before and after the DrWHO run.

Regarding coronagraphic tests, the preliminary results in relatively bad
seeing indicate that DrWHO does improve the image contrast, and the corrected
modes are mainly the low orders. Further tests are ongoing to fully quantify
the extend of the potential of the algorithm.

Overall, we show that DrWHO is able to improve the wavefront quality arriving
on the science camera, and partially correct for the static and quasi-static
NCPA, over a relatively short period of time of a few seconds. We show that
the algorithm converges rapidly, with about half of the correction achieved by
the first iteration. This has been observed in simulation and on-sky.
Furthermore, the correction of the NCPA is effective even if it is
considerably smaller than the dynamic of the atmosphere.

The characteristics of DrWHO are enumerated in section 6. The main strong
points are the robustness of the algorithm, its independence from any model,
making it compatible with other wavefront control methods, and that it does
not rely on any linear assumption, thus particularly fit for a PyWFS. DrWHO
combines focal plane wavefront sensing and the WFS.

A possible extension of DrWHO is to parallelize it in terms of spatial
frequencies, which would allow to correct speckles individually and to observe
fainter targets.

This paper is the first proof of concept of the algorithm on coronagraphic
images, following [29], and presents some first results of its image quality
and stability enhancement. DrWHO shows a wide potential of improvement, which
will be presented in future work.

###### Acknowledgements.

Based [in part] on data collected at Subaru Telescope, which is operated by
the National Astronomical Observatory of Japan. The development of SCExAO was
supported by the Japan Society for the Promotion of Science (Grant-in-Aid for
Research #23340051, #26220704, #23103002, #19H00703 & #19H00695), the
Astrobiology Center of the National Institutes of Natural Sciences, Japan, the
Mt Cuba Foundation and the director’s contingency fund at Subaru Telescope.
The authors wish to recognize and acknowledge the very significant cultural
role and reverence that the summit of Maunakea has always had within the
Hawaiian community. We are most fortunate to have the opportunity to conduct
observations from this mountain. NS acknowledges support from the PSL Iris-
OCAV project. NS and VD acknowledge support from NASA (Grant #80NSSC19K0336).
KA acknowledges support from the Heising-Simons foundation.

## References

  * [1] Macintosh, B., Graham, J. R., Ingraham, P., et al., “First light of the Gemini Planet Imager,” Proceedings of the National Academy of Science 111, 12661–12666 (2014).
  * [2] Beuzit, J. L., Vigan, A., Mouillet, D., et al., “SPHERE: the exoplanet imager for the Very Large Telescope,” arXiv e-prints , arXiv:1902.04080 (2019).
  * [3] Chauvin, G., Lagrange, A. M., Dumas, C., et al., “A giant planet candidate near a young brown dwarf. Direct VLT/NACO observations using IR wavefront sensing,” Astronomy & Astrophysics 425, L29–L32 (2004).
  * [4] Marois, C., Macintosh, B., Barman, T., et al., “Direct Imaging of Multiple Planets Orbiting the Star HR 8799,” Science 322, 1348 (2008).
  * [5] Jovanovic, N., Martinache, F., Guyon, O., et al., “The Subaru Coronagraphic Extreme Adaptive Optics System: Enabling High-Contrast Imaging on Solar-System Scales,” Publications of the Astronomical Society of the Pacific 127, 890 (2015).
  * [6] Males, J. R., Close, L. M., Miller, K., et al., “MagAO-X: project status and first laboratory results,” in [Proc. SPIE. ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10703, 1070309 (2018).
  * [7] Vigan, A., Gry, C., Salter, G., et al., “High-contrast imaging of Sirius A with VLT/SPHERE: looking for giant planets down to one astronomical unit,” Monthly Notices of the Royal Astronomical Society 454, 129–143 (2015).
  * [8] Currie, T., Guyon, O., Lozi, J., et al., “On-sky performance and recent results from the Subaru coronagraphic extreme adaptive optics system,” in [Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 11448, 114487H (2020).
  * [9] Vievard, S., Ahn, K., Arriola, A., et al., “Very high resolution spectro-interferometry with wavefront sensing capabilities on scexao/subaru using photonics,” Proceeding SPIE, Optical Engineering + Applications , Paper 11823–11 (2021).
  * [10] Deo, V., Vievard, S., Cvetojevic, N., et al., “Wavefront sensing using non-redundant aperture masking interferometry: tests and validation on subaru/scexao,” Proceeding SPIE, Optical Engineering + Applications , Paper 11823–39 (2021).
  * [11] Guyon, O., Norris, B., Males, J., et al., “High contrast imaging at the photon noise limit with self-calibrating wfs/c systems,” Proc. SPIE 11823, 11823–37 (2021).
  * [12] Kasper, M., Cerpa Urra, N., Pathak, P., et al., “PCS — A Roadmap for Exoearth Imaging with the ELT,” The Messenger 182, 38–43 (2021).
  * [13] Fitzgerald, M., Bailey, V., Baranec, C., et al., “The Planetary Systems Imager for TMT,” in [Bulletin of the American Astronomical Society ], 51, 251 (2019).
  * [14] Close, L. M., Males, J. R., Hedglen, A., Bouchez, A., & Guyon, O., “Concept for the GMT High-Contrast Exoplanet Instrument GMagAO-X and the GMT High-Contrast Phasing Testbed with MagAO-X,” arXiv e-prints , arXiv:2004.06808 (2020).
  * [15] Ahn, K., Guyon, O., Lozi, J., et al., “Scexao: a testbed for developing high-contrast imaging technologies for elts,” Proceeding SPIE, Optical Engineering + Applications , Paper 11823–2 (2021).
  * [16] Sauvage, J.-F., Fusco, T., Rousset, G., & Petit, C., “Calibration and precompensation of noncommon path aberrations for extreme adaptive optics,” Journal of the Optical Society of America A 24, 2334–2346 (2007).
  * [17] Vigan, A., N’Diaye, M., Dohlen, K., et al., “Calibration of quasi-static aberrations in exoplanet direct-imaging instruments with a Zernike phase-mask sensor. III. On-sky validation in VLT/SPHERE,” Astronomy & Astrophysics 629, A11 (2019).
  * [18] Potier, A., Galicher, R., Baudoz, P., et al., “Increasing the raw contrast of VLT/SPHERE with the dark hole technique. I. Simulations and validation on the internal source,” Astronomy & Astrophysics 638, A117 (2020).
  * [19] Gonsalves, R. A., “Phase retrieval and diversity in adaptive optics,” Optical Engineering 21(5), 215829–215829– (1982).
  * [20] Paxman, R. G. & Fienup, J. R., “Optical misalignment sensing and image reconstruction using phase diversity.,” Journal of the Optical Society of America A 5, 914–923 (1988).
  * [21] Frazin, R. A., “Treating Wavefront Measurement Error in Estimation of Non-Common Path Aberration for Direct Imaging of Exoplanets,” arXiv e-prints , arXiv:1811.05096 (2018).
  * [22] Vigan, A., N’Diaye, M., Dohlen, K., et al., “On-sky compensation of non-common path aberrations with the ZELDA wavefront sensor in VLT/SPHERE,” in [Proc. SPIE. ], Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10703, 107035O (2018).
  * [23] Vievard, S., Bos, S. P., Cassaing, F., et al., “Focal Plane Wavefront Sensing on SUBARU/SCExAO,” arXiv e-prints , arXiv:2012.12417 (2020).
  * [24] Bos, S. P., Vievard, S., Wilby, M. J., et al., “On-sky verification of Fast and Furious focal-plane wavefront sensing: Moving forward toward controlling the island effect at Subaru/SCExAO,” Astronomy & Astrophysics 639, A52 (2020).
  * [25] Esposito, S., Puglisi, A., Pinna, E., et al., “On-sky correction of non-common path aberration with the pyramid wavefront sensor,” A&A 636, A88 (2020).
  * [26] Deo, V., Gendron, É., Rousset, G., et al., “A telescope-ready approach for modal compensation of pyramid wavefront sensor optical gain,” Astronomy & Astrophysics 629, A107 (2019).
  * [27] Chambouleyron, V., Fauvarque, O., Janin-Potiron, P., et al., “Pyramid wavefront sensor optical gains compensation using a convolutional model,” arXiv e-prints , arXiv:2006.08294 (2020).
  * [28] Chambouleyron, V., Fauvarque, O., Sauvage, J.-F., Neichel, B., & Fusco, T., “The focal-plane assisted pyramid wavefront sensor: enabling frame-by-frame optical gains tracking,” arXiv e-prints , arXiv:2103.02297 (2021).
  * [29] Skaf, N., Guyon, O., Gendron, E., et al., “On-sky validation of image-based adaptive optics wavefront sensor referencing,” Astronomy & Astrophysics (2021).
  * [30] Ferreira, F., Gratadour, D., Sevin, A., et al., “Real-time end-to-end AO simulations at ELT scale on multiple GPUs with the COMPASS platform ,” in [Adaptive Optics Systems VI ], Close, L. M., Schreiber, L., & Schmidt, D., eds., 10703, 1155 – 1166, International Society for Optics and Photonics, SPIE (2018).
  * [31] Lozi, J., Jovanovic, N., Guyon, O., et al., “Visible and Near Infrared Laboratory Demonstration of a Simplified Pyramid Wavefront Sensor,” Publications of the Astronomical Society of the Pacific 131, 044503 (2019).
  * [32] Guyon, O., Sevin, A., Gratadour, D., et al., “The compute and control for adaptive optics (CACAO) real-time control software package,” in [Adaptive Optics Systems VI ], Close, L. M., Schreiber, L., & Schmidt, D., eds., 10703, 469 – 480, International Society for Optics and Photonics, SPIE (2018).
  * [33] Norris, B., Schworer, G., Tuthill, P., et al., “The VAMPIRES instrument: imaging the innermost regions of protoplanetary discs with polarimetric interferometry,” Monthly Notices of the Royal Astronomical Society 447, 2894–2906 (2015).


# Ego-to-Exo: Interfacing Third Person Visuals from Egocentric Views in Real-
time for Improved ROV Teleoperation

Adnan Abdullah⋆, Ruo Chen⋆, Ioannis Rekleitis⋄, and Md Jahidul Islam⋆ ⋆RoboPI
Laboratory, Dept. of ECE, University of Florida, USA. Email:
{adnanabdullah@,chenruo<EMAIL_ADDRESS>Department of CSE,
University of South Carolina, Columbia, USA. Email<EMAIL_ADDRESS>pre-print is currently under review.

###### Abstract

Underwater ROVs (Remotely Operated Vehicles) are unmanned submersible vehicles
designed for exploring and operating in the depths of the ocean. Despite using
high-end cameras, typical teleoperation engines based on first-person
(egocentric) views limit a surface operator’s ability to maneuver the ROV in
complex deep-water missions. In this paper, we present an interactive
teleoperation interface that enhances the operational capabilities via
increased situational awareness. This is accomplished by (i) offering on-
demand “third”-person (exocentric) visuals from past egocentric views, and
(ii) facilitating enhanced peripheral information with augmented ROV pose
information in real-time. We achieve this by integrating a 3D geometry-based
Ego-to-Exo view synthesis algorithm into a monocular SLAM system for accurate
trajectory estimation. The proposed closed-form solution only uses past
egocentric views from the ROV and a SLAM backbone for pose estimation, which
makes it portable to existing ROV platforms. Unlike data-driven solutions, it
is invariant to applications and waterbody-specific scenes. We validate the
geometric accuracy of the proposed framework through extensive experiments of
2-DOF indoor navigation and 6-DOF underwater cave exploration in challenging
low-light conditions. A subjective evaluation on $15$ human teleoperators
further confirms the effectiveness of the integrated features for improved
teleoperation. We demonstrate the benefits of Ego-to-Exo view generation and
real-time pose rendering for remote ROV teleoperation by following navigation
guides such as cavelines inside underwater caves. This new way of interactive
ROV teleoperation opens up promising opportunities for future research in
subsea telerobotics.

##  I Introduction

Unmanned submersible vehicles such as ROVs (Remotely Operated Vehicles) play a
crucial role in subsea inspection, remote surveillance, and underwater cave
exploration [1, 2, 3]. They are particularly useful in inspecting deep-water
structures and surveying confined spaces that are beyond the reach of human
scuba divers [4, 5]. In a typical mission, ROVs are controlled by human
operators from a surface vessel, who are responsible for the safe and
efficient maneuvering of the vehicle [6, 7]. The control consoles for
teleoperation typically offer real-time data such as the egocentric video
feed, pose, velocity, depth, etc. State-of-the-art ROVs can also include
autonomous features for atomic tasks such as hovering [8], following
navigation guidelines (e.g., cavelines inside underwater caves [9, 10]),
gripping objects [11, 12], trajectory estimation, etc.

While the subsea industries and agencies such as NOAA and naval defense teams
deploy underwater ROVs with high-end cameras, sonars, and IMUs [2, 13] – safe
and efficient teleoperation remains a challenge in adverse visibility
conditions and around complex or sensitive structures. The typical first-
person feeds from an ROV camera provide very limited information in landmark-
deprived underwater scenes. The operators on the surface can only see the
egocentric view, often without global or peripheral semantic information
around the ROV. Although ROVs can use artificial lights to enhance visibility
in low-light scenes, their bright light get reflected and back-scattered by
suspended particles directly at the front camera [9], creating glare and large
blind spots for the operator. Additionally, the autonomous and semi-autonomous
features of ROVs also become erroneous without peripheral positioning in such
noisy sensing conditions.

Figure 1: The proposed Ego-to-Exo teleoperation interface is demonstrated for
an underwater cave exploration scenario with an ROV. The traditional console
interfaces are based on egocentric views (top left), which is limiting and
disorienting to a surface operator in noisy low-light conditions. Our method
generates on-demand exocentric views from a given EOB (Eye On the Back)
viewpoint, i.e., third-person views from behind the ROV. This interactive EOB
viewpoint-selection option is integrated into a standard BlueROV2 console for
a significantly improved teleoperation experience.

In this paper, we address these issues by introducing an AR (augmented
reality) inspired ROV teleoperation interface that generates third-person
(exocentric) perspectives as well as provides interactive control choices for
viewpoint selection. As shown in Fig. 1, the proposed console can generate
multiple exocentric views from past egocentric images, with a virtual ROV
projected on the images as if it were taken by a third person following the
robot. Our earlier work introduced the idea of EOB (Eye On the Back) visuals
[14], which produces a single third-person view from immediately behind the
ROV to facilitate better teleoperation. This work develops the AR-based end-
to-end pipeline for generating on-demand exocentric perspectives given any EOB
viewpoints. Additionally, we integrate the feature for geometrically accurate
ROV positioning into those views for an interactive ROV teleoperation
experience in challenging underwater applications. With these features, an
operator can slide across multiple EOB viewpoints for the best view, see where
the ROV is, and get comprehensive peripheral information for safe and
efficient teleoperation.

Specifically, we design an efficient Ego-to-Exo (egocentric to exocentric)
view generation framework integrated into a monocular visual SLAM system for
underwater ROV teleoperation. The proposed Ego-to-Exo algorithm keeps track of
the ROV camera poses and exploits a buffer of egocentric views for exocentric
view synthesis. We then transform and project a pre-sampled 3D model of the
ROV, in the form of a point cloud, into those views to generate realistic
augmented visuals with more peripheral information. As seen in Fig. 1, such
views offer comprehensive information of the surrounding scene with global
semantics. In addition to the views, the integrated SLAM system provides real-
time pose and map updates for atomic tasks [15, 16] such as obstacle
avoidance, object following, next-best-view planning, etc.

The significance of this work resides in the simplicity of a 3D-geometric
formulation of the Ego-to-Exo problem, and its integration into a real-time
SLAM system. A monocular vision-only pipeline is chosen to ensure generalized
utility and computational efficiency. The proposed Ego-to-Exo solution, EOB
viewpoint-based parameterization, and ROV point cloud projection – are carried
out by closed-form solutions to ensure real-time performance. As opposed to
data-driven approaches for exocentric view synthesis, the proposed framework
is invariant to the changes in waterbody style, scene geometry, and
application scenario – making it transferable to any underwater ROV platforms.

We demonstrate the performance and utility of the proposed Ego-to-Exo
framework through a series of experiments. First, we prove the geometric
validity of our estimation with 2D indoor navigation scenarios utilizing a
TurtleBot4. We quantify the geometric validity based on ground plane
estimation errors and reprojection errors of known reference points in the
scene. Next, we conduct experiments for underwater cave exploration scenarios
where a BlueROV2 is teleoperated inside underwater caves and grotto systems at
various geographical locations. Remote teleoperation in such cave systems
includes unique challenges such as low visibility, turbid water conditions,
moving shadow effects, and hazy blind spots due to artificial lights. We
evaluate the usefulness of the proposed TeleOp console with $15$ human
subjects in such challenging scenarios. They use the interactive features such
as viewpoint selection to visualize augmented ROV poses on the map in real-
time. The subjects rank the complexity, efficiency, and other aspects of the
system in a questionnaire as well as provide open-ended feedback regarding
their cognitive workload.

We further discuss the observations for a variety of challenging cases of
caveline detection and following in noisy low-light conditions inside multiple
underwater caves. We demonstrate that the generated exocentric views, the
augmented robot pose, and the rendered map embed significantly more
information and global context about the scene to facilitate safe and
efficient robot control.

(a) We used two platforms: a 2D ground robot (TurtleBot4) and an underwater
ROV (BlueROV2); their 3D mesh models and sampled point clouds are shown.

(b) The proposed algorithm maintains a buffer of past egocentric views and
corresponding ROV poses from the monocular SLAM system. Given an EOB viewpoint
$r$, an exocentric view is generated by synthesizing the past egocentric
views; the 3D model of the ROV is then rendered onto the view.

Figure 2: The proposed Ego-to-Exo problem formulation is shown; the idea is to
generate EOB (Eye On the Back) views with augmented 3D robot’s pose for more
informative third-person perspectives to facilitate safer and more efficient
ROV teleoperation.

##  II Related Work & Applications

###  II-A ROV Deployment & Teleoperation

Safe and efficient robot teleoperation is crucial for marine biology,
oceanography, offshore energy, and infrastructure maintenance. Underwater ROVs
enable researchers to study deep-sea ecosystems and collect samples from
depths beyond the limits of human scuba divers. A few prominent applications
are as follows.

Subsea structure inspection. Underwater structures operate in challenging
remote environments and are susceptible to various hazards with significant
economic and environmental consequences. Corrosion/erosion combined with
tensile stresses caused by waves and seismic activities lead to cracking,
leakage, and malfunction which are regularly inspected by ROVs [1, 17]. ROVs
are also deployed for 3D mapping and acoustic profiling of marine structures
[18, 19].

Underwater infrastructure maintenance & security. The underwater energy sector
[20] and submerged data centers [21, 22] are experiencing rapid growth with
increasing demand for regular maintenance and surveillance operations. Once
ROVs are teleoperated to the site, they need to perform various autonomous and
semi-autonomous tasks to inspect, monitor, and ensure safety and integrity of
the infrastructures.

Marine archaeology and cave exploration. Exploring underwater heritage sites
[23] and caves [4] are important to unravel archaeological history as well as
for water resource management. Intelligent ROV capabilities enable effective
mapping, sampling, and exploration of remote sites and challenging overhead
environments. Moreover, seasonal monitoring of oceanographic features is
performed for photometric studies as well [24, 25, 26].

Beyond these sectors, improved ROV teleoperation interfaces based on our
proposed Ego-to-Exo augmented visuals will be useful in coastal habitat
restoration and surveying, aquaculture, and search-and-rescue missions as
well.

###  II-B Third-Person Views for ROV Teleoperation

A common issue reported by ROV operators is that using a remote vision
platform for teleoperation is like looking through a “ _soda straw_ ” [27].
This is because the typical ROV controller interfaces are based on egocentric
first person camera views – which provide no peripheral vision, resulting in
significantly reduced situational awareness [28, 29]. As such, there have been
multiple methods proposed to mitigate the issue and provide the operator with
more visual data.

Two types of commonly used approaches exist for generating an exocentric view
for unmanned ground and aerial vehicles. The first method uses an external
camera to provide the exocentric view such as using a fixed camera to record
the vehicle’s trajectory from a distance [30], mounting the camera on a UAV
(unmanned aerial vehicle) that flies above the target [31, 32, 33, 34],
attaching the camera to an elevated position on the robot [35], utilizing a
follower ROV with a camera to provide the external view [36], or using a fish-
eye camera to create a top-down perspective [37]. The second method utilizes
integrated sensors on the robot, such as LiDAR (Light Detection and Ranging)
to generate a point cloud of the surrounding environment [38, 39] and use that
toward creating an augmented/virtual environment for interfacing and
teleoperation [40, 41, 42].

Adapting the aforementioned methods from terrestrial or aerial domains to
underwater environments presents inherent challenges. Firstly, sending diver-
robot teams [43] is not always an option in complex deep-water missions –
which are the majority of use cases for ROVs. Secondly, UGVs that utilize past
egocentric views [44, 45] primarily rely on GPS-based localization that does
not apply to GPS-denied underwater environment. Unlike underwater ROVs, ground
vehicles generally operate on a 2D plane with limited pitch and roll
variations over rough terrain. Thirdly, installing an external visual system
requires significant hardware modifications, e.g., they need to be rugged and
pressure-sealed, re-calibrated for buoyancy and motion dynamics, and
additional tether integration for high-speed exocentric data transfer. Even
with all the structural modifications, an external camera will provide a
single additional third-person perspective. We attempt to address these issues
by synthesizing geometrically accurate exocentric views into a monocular SLAM
pipeline given a dynamic viewpoint.

##  III Problem Formulation

We formulate the Ego-to-Exo problem as a 3D geometric algorithm that involves
generating an on-demand EOB (Eye On the Back) view, and then projecting the
robot on it for an augmented rendering of the scene; see Fig. 2. The proposed
method has the following computational components.

###  III-A Curating Pose and Image Queue from Monocular SLAM

A monocular SLAM algorithm such as ORB-SLAM3 [46] provides a continuous
solution for estimating and tracking camera poses from a sequence of monocular
images. We use an ORB-SLAM3-based framework to obtain camera poses of each
keyframe location to eventually construct the trajectory map of the
teleoperated robot. Our adapter SLAM framework initiates the trajectory
estimation process by building a pose buffer of length $n$:
${}^{w}\mathbf{T}\triangleq[\,{}^{w}_{i}\mathbf{T},\,{}^{w}_{i-1}\mathbf{T},\,.\,.\,.,{}^{w}_{i-n+1}\mathbf{T}\,]$,
where, ${}^{w}_{i}\mathbf{T}=[{}^{w}_{i}\mathbf{R}_{3\times
3}\,|\,{}^{w}_{i}\mathbf{t}_{3\times 1}]$ denotes camera pose at instance $i$
in global ($world$) frame of reference. The corresponding raw egocentric views
$\mathbf{I}$ for each instance is also stored in a queue
$\mathbf{I}\triangleq[\,\mathbf{I}_{i},\,\mathbf{I}_{i-1},\,\cdots\,\mathbf{I}_{i-n+1}\,]$.
These memory buffers are updated instantaneously as the robot pose changes
during teleoperation. We use an empirically tuned threshold to trigger an
update only when the pose change is significant to avoid unnecessary updates
(when the robot is static).

###  III-B Generating Exocentric Perspective

Given the pose memory ${}^{w}\mathbf{T}$ and egocentric views $\mathbf{I}$, we
formulate the Ego-to-Exo problem of estimating an exocentric view from a
reference location $r$, looking toward the robots current location $c$, where
$r,c\in[i-n+1,\,i]$ and $r<c$. Typically, $c$ is set to $i$ (most recent
available frame), and $r$ remains a free variable with $n$ known samples in
memory – to mimic the EOB viewpoint generation.

We use the ROV point cloud model $\mathbf{P}$ of size $3\times m$ as prior.
These $m$ points are transformed from current camera pose
${}^{w}_{c}\mathbf{T}$ to reference camera pose ${}^{w}_{r}\mathbf{T}$ using:

$\mathbf{\tilde{P}}=({}^{w}_{r}\mathbf{R}^{-1}\,{}^{w}_{c}\mathbf{R})\cdot\mathbf{P}+({}^{w}_{c}\mathbf{t}-{}^{w}_{r}\mathbf{t}),$
(1)

where $[{}^{w}_{c}\mathbf{R}\,|\,{}^{w}_{c}\mathbf{t}]$ and
$[{}^{w}_{r}\mathbf{R}\,|\,{}^{w}_{r}\mathbf{t}]$ represent the pose for
current and reference (target) location in world coordinate, respectively. The
transformed point cloud $\mathbf{\tilde{P}}$ is then projected onto the target
image plane by using camera intrinsics $\mathbf{K}$ as:

$\begin{bmatrix}\mathbf{u}&\mathbf{v}&\mathbf{1_{m\times
1}}\end{bmatrix}^{T}=\lambda_{1}\,\mathbf{K}\cdot\mathbf{\tilde{P}}.$ (2)

Here, $\mathbf{u}$ and $\mathbf{v}$ vectors denote the pixel locations
($u$,$v$) on image $\mathbf{I}_{r}$ for projection; $\lambda_{1}$ is the
scale.

###  III-C 3D ROV Rendering and Map Update

While the SLAM system constructs a map of the surrounding, the proposed
algorithm simultaneously renders the 3D ROV model on the same spatial context.
As illustrated in Fig. 2(b), the ROV points $\mathbf{P}$ are transformed to
the current camera location and projected based on the relative pose
information ${}^{w}_{i}\mathbf{T}$ as follows:

$\mathbf{\tilde{P}}_{map}=\lambda_{2}\,{}^{w}_{i}\mathbf{R}\cdot\mathbf{P}+{}^{w}_{i}\mathbf{t}.$
(3)

Here, $\lambda_{2}$ is the scaling factor for the ROV model. Like all
monocular SLAM-based systems [47, 48], our proposed mapping and projection
method is up to scale. While the scale can be resolved with additional sensor
fusion, the augmented visuals of Eq. 3 are sufficient for teleoperation.

##  IV Implementation & Evaluation

###  IV-A Implementation Details

The proposed framework is implemented in ROS Noetic on an Ubuntu 20.04
environment. A ROS node for ORB-SLAM3 is integrated as the monocular SLAM
backbone. The buffer queue size $n$ is set to $100$ and the frame separation
threshold is set to $0.001$ unit. The ROV point clouds are generated by
sampling 3D mesh models of BlueROV2 and TurtleBot4; $10,000$ points are
sampled for each model. The same ROS packages are used on both robots with
domain-specific empirical tuning for the $\lambda$ parameters.

###  IV-B Proof of Concept: 2D Indoor Navigation

Experimental setup. The proof-of-concept experiments are conducted with
TurtleBot4, a 2D ground robot that can be teleoperated with egocentric views
from its front-facing monocular camera. It has only two degrees of freedom
(DOF) for linear and angular velocity - which simplifies the motion kinematics
for tracking its instantaneous position and orientation. We teleoperate it to
collect monocular visual data in office, laboratory, and hallway scenarios.
The experiments are designed to validate the proposed algorithm by evaluating
ground plane estimation and reprojection errors.

Geometric validation: reprojection error analysis. We first evaluate the
reprojection errors of known reference points from the generated Ego-to-Exo
views and the estimated ROV pose. As shown Fig. 3, we use standard
checkerboard corners as reference points from egocentric views and then
evaluate the reprojection errors for those points from exocentric views. This
test is iterated over different sets of past egocentric images, each
corresponding to a different EOB distance. As shown in Fig. 3(a), a
checkerboard is viewed from different EOB distances (further back into the
past) indicated by the parameter $f$. More specifically, $f$ is the number of
frames between the current egocentric view and the selected EOB view. The
corresponding reprojection error is plotted in Fig. 3(b), which shows how the
estimation is accurate for lower values of $f$, and gradually degenerates for
$f>100$. This is consistent with our visual observation of the projected ROV
point cloud, i.e., it is on the ground plane with accurate orientation based
on the SLAM trajectory estimates.

(a) The TurtleBot4 trajectory during teleoperation is shown; here, the $f$
numbers indicate the EOB distance from current to reference frame used for the
generated Ego-to-Exo views.

(b) Reprojection errors for reference points (checkerboard corners) are
evaluated for different EOB distances ($f$). The estimated ground plane is
shown with a cube placed on it for visual observation.

Figure 3: We conduct 2D indoor navigation experiments with a TurtleBot4 to
validate the geometric accuracy of our Ego-to-Exo estimation; here, results
are visualized for ground plane estimation and reprojection errors of known
reference points in the scene.

Geometric validation: ground plane estimation. We also visualize the accuracy
of the estimated ground plane in these experiments for qualitative
assessments. Fig. 3(b) illustrates two cases: one for $f=70$ with a low
reprojection error, and another for $f=260$ with a high error. As seen, the
estimated ground plane (and the drawn sample cube) validates the geometric
accuracy for the $f=70$ case. On the other hand, a misaligned ground plane for
the $f=260$ case demonstrates the underlying error in pose estimation as well
as in the reprojection process. Essentially, the geometric accuracy of our
proposed Ego-to-Exo algorithm depends on the trajectory estimation performance
of the SLAM system. More importantly, a teleoperator can make the right
viewpoint selection in real-time for optimal Ego-to-exo rendering.

Figure 4: The reconstructed 3D map of a cave segment in Devil’s Springs,
Florida is shown. The colored points represent the tracked ORB features,
whereas the ROV trajectory is shown in green circles. The popup frames show
samples of: (a) egocentric images; (b) synthesized exocentric images with
rendered ROV pose; (c) the underlying camera pose updates; and (d) an
exocentric view of the 3D map.

Computational efficiency. We analyze the computational complexity of the Ego-
to-Exo algorithm for different configurations to ensure real-time execution in
resource-constrained edge devices onboard standard ROV platforms. Table I
shows the memory requirement of our algorithm for different choices of buffer
size. The memory footprint is less than $300$ MB for a buffer size of up to
$180$ frames, making it highly efficient. Table II demonstrates that the end-
to-end framework maintains a consistent output rate of over $25$ FPS (frames
per second), making it suitable for integration in existing teleoperation
engines. In fact, our algorithm only adds $6.8\%$ overhead on the SLAM
backbone for the global map update.

TABLE I: The memory requirement of Ego-to-Exo algorithm is compared for different buffer length. Buffer size (# of frames) | $50$ | $100$ | $200$ | $300$ | $400$
---|---|---|---|---|---
Memory usage (MB) | $65$ | $142$ | $301$ | $455$ | $609$
|  |  |  |  |
TABLE II: The computational complexity of the proposed Ego-to-Exo framework integrated into a SLAM backbone is reported. Here, the metrics used are: (i) ROS node publish rate of the exocentric image; and (ii) the global map update rate. Method | SLAM Only | SLAM + Ego-to-Exo
---|---|---
Output image rate |  $26$ FPS |  $25.1$ FPS
Map update rate |  $26$ FPS |  $25.3$ FPS
|  |

###  IV-C Field Deployment: 3D Underwater Cave Exploration

Experimental setup. We extend our experiments to underwater cave exploration
scenarios, where the ROV performs full 6-DOF motions. While the $roll$ motion
is limited in the standard BlueROV2s, we consider all 6-DOF for teleoperation
with the buoyancy change and pressure imbalance caused by water flow at the
cave openings. For remote teleoperation, we consider the scenarios where human
operators maneuver an underwater ROV from the surface by following the
caveline and other navigation markers as a guide [49]. The mission objective
is to navigate the ROV $75$-$300$ feet deep inside the cave through its
complex structures, and then safely return it to the surface. In addition to
evaluating the geometric accuracy and robustness, we consider how informative
the generated Ego-to-Exo views are compared to traditional consoles for ROV
teleoperation.

Real-time map update and teleoperation. In addition to the Ego-to-Exo view
generation and ROV pose rendering, our framework simultaneously updates a 3D
map with extracted feature points from the SLAM system. Fig. 4 shows an ROV’s
trajectory mapped during an underwater cave mission in Devil’s Springs,
Florida. As seen, the generated Ego-to-Exo views embed significantly more
peripheral information about the scene. The exocentric view of the ROV pose
and its relative distance from cave walls or overhead obstacles are useful to
surface operators for obstacle avoidance and efficient decision-making.
Additionally, the 3D map shows the ROV’s past trajectory and its current pose
which are useful to analyze the mission progress, which is not possible in
traditional teleoperation consoles. Such a global view of the trajectory is
also useful during emergency evacuation and recovery. Beyond cave exploration,
these features will be crucial in ROV-based subsea surveillance and search-
and-rescue operations as well.

Figure 5: A snapshot from our cave exploration scenario is shown: (a)
Egocentric view with detected reference points; and (b) The synthesized Ego-
to-Exo view with projected ROV point cloud. We use a sample logo and perform
the homographic projection on the reference surface to demonstrate the
accuracy in pose estimation.

Accuracy and robustness. Due to the complex scene geometry and absence of a
smooth ground plane inside underwater caves, we adopt a homography estimation
approach for the performance validation. As shown in Fig. 5, April-Tag [50]
corners are used as reference points for reprojection. Specifically, we
compute the homography transformation between the egocentric and synthesized
exocentric views to visualize the reprojection errors. We use a sample 2D logo
and project it onto the reference April-Tag surface using the homographic
transform. The unskewed planar projection validates the accuracy of the Ego-
to-Exo pose estimation and point cloud rendering processes.

TABLE III: $15$ subjects provide their feedback to the following two sets of questions: Question $1-10$ from SUS [51], and the rest are custom designed. Response to each question is scaled from $1$ (strongly disagree) to $5$ (strongly agree). # |  Question |  mean, std
---|---|---
System Usability Scale (SUS)[51]
1 |  I think that I would like to use this system frequently. |  4.3$,~{}$0.6
2 |  I found the system unnecessarily complex. |  2.0$,~{}$0.7
3 |  I thought the system was easy to use. |  4.3$,~{}$0.4
4 |  I think that I would need the support of a technical person to be able to use this system. |  2.0$,~{}$0.6
5 |  I found the various functions in this system were well integrated. |  4.0$,~{}$0.8
6 |  I thought there was too much inconsistency in this system. |  2.3$,~{}$0.7
7 |  I would imagine that most people would learn to use this system very quickly. |  4.4$,~{}$0.5
8 |  I found the system very cumbersome to use. |  2.0$,~{}$0.6
9 |  I felt very confident using the system. |  3.7$,~{}$0.7
10 |  I needed to learn a lot of things before I could get going with this system. |  1.4$,~{}$0.5
Custom Questions
11 |  The proposed exocentric view is beneficial for ROV teleoperation. |  4.5$,~{}$0.5
12 |  I found the EOB distance tuning feature useful to get the best view. |  4.5$,~{}$0.5
13 |  The generated 3D map provides a better understanding of the ROV’s global location and its surroundings. |  4.6$,~{}$0.9

Observations: strengths and limitations. Our experiments reveal some key
strengths of the proposed teleoperation framework. First, the generated
exocentric views closely resemble the actual EOB views during a smooth
trajectory, which is usually the case for subsea exploration and surveying
tasks. Second, the buffer memory works as a backup during a temporary failure
of the SLAM system, typically observed at turning corners or due to abrupt
motion. In such cases, our algorithm retains historical poses from its buffer
memory; teleoperators can utilize this for situational awareness to safely
anchor or pause the mission until communication is restored. On the other
hand, its heavy dependency on the SLAM backbone leads to some inherent
limitations. Feature-based monocular SLAM systems often fail in feature-
deprived noisy underwater scenes, which leads to inaccurate pose tracking and
thus inaccurate Ego-to-Exo view synthesis. Tracking 6-DOF ROV motion from
monocular vision is particularly challenging with no additional sensor to
recover the scale information [52, 53]. We observe some instances where the
estimated ROV pose is incorrectly scaled in the rendering. To address this,
multi-sensor fusion-based underwater SLAM backbones [54, 55] can be utilized
in more critical applications.

###  IV-D Subjective User Study

The user study is conducted with multiple underwater cave exploration data. A
BlueROV2 recorded egocentric video feeds inside the caves at up to $100$-meter
penetrations. Later on, $15$ participants evaluate the ease of operation with
our developed console and compare it to traditional consoles. Their feedback
is recorded using the System Usability Scale (SUS) [51], with our interface
achieving an average SUS score of $77.5$. We also formulate an independent set
of questions that reflect the teleoperator’s preference for the novel features
of our method. The individual questions and corresponding scores are presented
in Table III. Some key observations from this study are listed below.

  1. 1.

The obtained SUS score is fairly above median (score: $68$) and is considered
Good for user experience; it is slightly below the Excellent (score: $80.3$)
category.

  2. 2.

Post-operation feedback from our ROV operators suggests that the exocentric
views are more useful for safe ROV maneuvers.

  3. 3.

The synthesized 3D map provides a better sense of the ROV’s global location
and improves spatial awareness of the teleoperators.

  4. 4.

The operators report a significantly less workload (perceived cognitive load)
in conducting complex tasks such as object following and structure mapping.

##  V Improved Underwater ROV Teleoperation

Multiple augmented viewpoints. We validate the utility of our proposed Ego-to-
Exo teleoperation interface through further experiments on underwater cave
exploration data. Our expedition in cave segments at Devil’s Springs, Florida
reveals that when ROVs move slowly against strong currents, extending the
exocentric viewpoint distance can significantly improve teleoperation. This is
achieved by tuning the queue parameters $r$, $c$, and $n$ in the proposed
TeleOp interface. We consistently find that exocentric views are more
informative, especially for about $5$-$10$ seconds preceding the ROV position
during navigation. The multiple preceding views offered by our interface are
particularly useful for mapping large structures such as newly discovered cave
segments or shipwrecks. As Fig. 6 shows, the synthesized viewpoints provide
more spatial context, enabling operators to control the ROV efficiently around
complex underwater structures.

Figure 6: A demonstration of our adjustable EOB viewpoint feature is shown.
The $f$ numbers indicate the EOB distance from current to the reference frame
used for the Ego-to-Exo view generation. Teleoperators can slide across the
distance ($f$) and find the best exocentric view, which is $f=200$ in this
example.

Efficient teleoperation in complex missions. We also conducted trials inside
an underwater cave system at Orange Grove, Florida, and a grotto system at
Hudson, Florida. We observe that maneuvering the robot by following the
caveline with egocentric views is challenging because little/no ambient light
penetrates inside underwater caves. Despite using powerful lights, problems
such as moving shadows and scattered waves create significant blind spots.
Consequently, tracking and following the caveline or any other navigation
markers [49] without any peripheral view is extremely disorienting to the
operator. In some cases, we observe that the cavelines get blended with the
texture and features of cave walls in noisy conditions; see Fig. 7. In such
scenarios, shifting the viewpoint to exocentric views allows easier
identification of cavelines against the surrounding and overhead cave walls.
Additionally, the augmented 3D map displays the robot’s pose, allowing much
safer maneuvering of the vehicle to its desired orientation.

Figure 7: Three challenging scenarios are shown for ROV teleoperation inside
underwater caves: (i) caveline is not visible, i.e., blended with the
background; (ii) caveline is not in the FOV; and (iii) front camera-light
interactions with suspended particles are causing hazy egocentric views. In
all cases, our Ego-to-Exo augmented visuals are clearer and more informative
to a surface operator.

Safer navigation in hazy low-light conditions. Underwater caves present a
unique formation of silt and sediment on their floor that results from erosion
over extended periods. The silt is susceptible to disturbance from external
factors, such as the motion of underwater ROVs or the turbulence generated by
their propellers. Although ROV operators pay close attention to avoid contact
with floor and cave-walls, it is often unavoidable due to buoyancy imbalance
and strong flow of water. Dislodging the sediments results in cloudy or hazy
conditions that obscure visibility. Bright lights from the ROV reflect from
these suspended particles and make it even more challenging to capture clear
imagery of the surroundings. In such cases, third-person EOB views from behind
the ROV offer a clearer and more informative perspective for navigation; see
Fig. 7. It improves spatial awareness and helps the operator to safely move
away from the sediment formations toward open, accessible areas and avoid
obstructing other scuba divers in the process.

##  VI Conclusion & Future Work

This work presents an AR-based framework to synthesize exocentric camera views
from egocentric feed in real-time for improved underwater ROV teleoperation. A
pose geometry-based closed-form solution is formulated for the proposed Ego-
to-Exo problem and then integrated into a SLAM backbone. The end-to-end
pipeline only requires a sequence of past egocentric views to generate an
exocentric view with the accurate ROV model projected onto it. The proof-of-
concept is validated by ground plane estimation and reprojection error
analyses in a series of 2D indoor navigation experiments. Subsequent field
experiments are conducted for various challenging ROV teleoperation scenarios
inside underwater caves. A subjective study proves the advantage of the
proposed TeleOp console over traditional systems and demonstrates that the
framework: (i) offers more informative peripheral views, (ii) provides better
situational awareness, and (iii) facilitates an interactive ROV teleoperation
experience. We are currently exploring more comprehensive multi-sensor fusion-
based underwater SLAM backbone such as the SVIn2 [54] for more accurate and
robust estimation. We further plan to develop and integrate more interactive
features that would serve as a simulation platform for ROV teleoperation and
navigation research.

## Acknowledgements

This work is supported in part by the NSF grants $2330416$, $1943205$, and
$2024741$. The authors would like to acknowledge the help from Woodville Karst
Plain Project (WKPP), El Centro Investigador del Sistema Acuífero de Quintana
Roo A.C. (CINDAQ), Global Underwater Explorers (GUE), Ricardo Constantino, and
Project Baseline in providing access to challenging underwater caves.

## References

  * [1] A. G. Rumson, “The Application of Fully Unmanned Robotic Systems for Inspection of Subsea Pipelines,” Ocean Engineering, vol. 235, p. 109214, 2021.
  * [2] S. Wishnak, “New Frontiers in Ocean exploration: the Ocean Exploration Trust,” NOAA Ocean Exploration, and Schmidt Ocean Institute 2021 Field Season, 2022.
  * [3] V. Siegel, W. Stone, and K. Richmond, “Robotic survey and 3-d mapping of underwater caves using a sunfish® autonomous underwater vehicle,” LPI Contributions, vol. 2697, p. 1037, 2023.
  * [4] B. Joshi, M. Xanthidis, M. Roznere, N. J. Burgdorfer, P. Mordohai, A. Q. Li, and I. Rekleitis, “Underwater exploration and mapping,” in IEEE OES AUV Symposium, (Singapore), pp. 1–7, Sept. 2022.
  * [5] P. L. Buzzacott, E. Zeigler, P. Denoble, and R. Vann, “American Cave Diving Fatalities 1969-2007,” International Journal of Aquatic Research and Education, vol. 3, no. 2, p. 7, 2009.
  * [6] A. Y. Konoplin, N. Y. Konoplin, and V. Filaretov, “Development of Intellectual Support System for ROV Operators,” in IOP Conference Series: Earth and Environmental Science, vol. 272, p. 032101, IOP Publishing, 2019.
  * [7] B. R. Kennedy, K. Cantwell, M. Malik, C. Kelley, J. Potter, K. Elliott, E. Lobecker, L. M. Gray, D. Sowers, M. P. White, et al., “The Unknown and the Unexplored: Insights into the Pacific Deep-sea Following NOAA CAPSTONE Expeditions,” Frontiers in Marine Science, vol. 6, p. 480, 2019.
  * [8] H.-S. Jin, H. Cho, H. Jiafeng, J.-H. Lee, M.-J. Kim, S.-K. Jeong, D.-H. Ji, K. Joo, D. Jung, and H.-S. Choi, “Hovering Control of UUV through Underwater Object Detection Based on Deep Learning,” Ocean Engineering, vol. 253, p. 111321, 2022.
  * [9] B. Yu, R. Tibbetts, T. Barna, A. Morales, I. Rekleitis, and M. J. Islam, “Weakly Supervised Caveline Detection For AUV Navigation Inside Underwater Caves,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9933–9940, IEEE, 2023.
  * [10] M. Mohammadi, S.-E. Huang, T. Barua, I. Rekleitis, M. J. Islam, and R. Zand, “Caveline detection at the edge for autonomous underwater cave exploration and mapping,” in IEEE International Conference on Machine Learning and Applications (ICMLA), (Jacksonville, FL, USA), Dec. 2023.
  * [11] M. Manjunatha, A. A. Selvakumar, V. P. Godeswar, and R. Manimaran, “A Low Cost Underwater Robot with Grippers for Visual Inspection of External Pipeline Surface,” Procedia computer science, vol. 133, pp. 108–115, 2018.
  * [12] S. Lensgraf, D. Balkcom, and A. Q. Li, “Buoyancy enabled autonomous underwater construction with cement blocks,” in IEEE International Conference on Robotics and Automation (ICRA), pp. 5207–5213, 2023.
  * [13] A. Elor, T. Thang, B. P. Hughes, A. Crosby, A. Phung, E. Gonzalez, K. Katija, S. H. Haddock, E. J. Martin, B. E. Erwin, et al., “Catching Jellies in Immersive Virtual Reality: A Comparative Teleoperation Study of ROVs in Underwater Capture Tasks,” in Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology, pp. 1–10, 2021.
  * [14] M. J. Islam, “Eye on the back: Augmented Visuals for Improved ROV Teleoperation in Deep Water Surveillance and Inspection,” in SPIE Defense and Commercial Sensing, (Maryland, USA), SPIE, 2024.
  * [15] W. Cai, Y. Wu, and M. Zhang, “Three-dimensional Obstacle Avoidance for Autonomous Underwater Robot,” IEEE Sensors Letters, vol. 4, no. 11, pp. 1–4, 2020.
  * [16] N. Palomeras, N. Hurtós, E. Vidal, and M. Carreras, “Autonomous Exploration of Complex Underwater Environments using a Probabilistic Next-best-view Planner,” IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 1619–1625, 2019.
  * [17] F. Yin, “Inspection Robot for Submarine Pipeline based on Machine Vision,” in Journal of Physics: Conference Series, vol. 1952, p. 022034, IOP Publishing, 2021.
  * [18] M. Xanthidis, M. Kalaitzakis, N. Karapetyan, J. Johnson, N. Vitzilaios, J. M. O’Kane, and I. Rekleitis, “AquaVis: A Perception-aware Autonomous Navigation Framework for Underwater Vehicles,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5410–5417, IEEE, 2021.
  * [19] M. Xanthidis, B. Joshi, M. Roznere, W. Wang, N. Burgdorfer, A. Quattrini Li, P. Mordohai, S. Nelakuditi, and I. Rekleitis, “Mapping of Underwater Structures by a Team of Autonomous Underwater Vehicles,” in International Symposium of Robotics Research, 2022.
  * [20] L. Vederhus and Y. Pan, “Surface-to-seabed Safety: Advantages of Simulator Practice for Subsea Installation,” International Journal of Safety and Security Engineering, vol. 6, no. 2, pp. 301–309, 2016.
  * [21] J. Sheldon, W. Zhu, A. Abdullah, K. Butler, M. J. Islam, and S. Rampazzi, “Deep note: Can acoustic interference damage the availability of hard disk storage in underwater data centers?,” in 15th ACM Workshop on Hot Topics in Storage and File Systems. Best Paper Award., pp. 51–57, 2023.  Best Paper Award.
  * [22] S. Curtis, “Subsea Fiber: Into the Deep,” Optics and Photonics News, vol. 34, no. 3, pp. 32–39, 2023.
  * [23] R. Eldred, J. Lussier, and A. Pollman, “Design and Testing of a Spherical Autonomous Underwater Vehicle for Shipwreck Interior Exploration,” Journal of Marine Science and Engineering, vol. 9, no. 3, p. 320, 2021.
  * [24] M. J. Islam, A. Q. Li, Y. A. Girdhar, and I. Rekleitis, “Computer Vision Applications in Underwater Robotics and Oceanography,” Computer Vision: Challenges, Trends, and Opportunities, pp. 173–196, 2024.
  * [25] B. Neuner, B. M. Pascoguin, A. Hening, and B. Dick, “Deployable Wavelength Optimization for Free-space Communication Undersea,” in OCEANS 2016 MTS/IEEE Monterey, pp. 1–5, IEEE, 2016.
  * [26] W. Wang, B. Joshi, N. Burgdorfer, K. Batsosc, A. Q. Lid, P. Mordohaia, and I. Rekleitisb, “Real-time Dense 3d Mapping of Underwater Environments,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5184–5191, IEEE, 2023.
  * [27] D. Woods, J. Tittle, M. Feil, and A. Roesler, “Envisioning Human-robot Coordination in Future Operations,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 34, no. 2, pp. 210–218, 2004.
  * [28] J. Casper and R. Murphy, “Human-robot Interactions during the Robot-assisted Urban Search and Rescue Response at the World Trade Center,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 33, no. 3, pp. 367–385, 2003.
  * [29] S. Zollmann, C. Hoppe, T. Langlotz, and G. Reitmayr, “Flyar: Augmented Reality Supported Micro Aerial Vehicle Navigation,” IEEE transactions on visualization and computer graphics, vol. 20, no. 4, pp. 560–568, 2014.
  * [30] R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang, “Look Closer: Bridging Egocentric and Third-Person Views With Transformers for Robotic Manipulation,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3046–3053, 2022.
  * [31] A. Gawel, Y. Lin, T. Koutros, R. Siegwart, and C. Cadena, “Aerial-Ground Collaborative Sensing: Third-Person View for Teleoperation,” in 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pp. 1–7, 2018.
  * [32] D. Saakes, V. Choudhary, D. Sakamoto, M. Inami, and T. Lgarashi, “A Teleoperating Interface for Ground Vehicles using Autonomous Flying Cameras,” in 2013 23rd International Conference on Artificial Reality and Telexistence (ICAT), pp. 13–19, 2013.
  * [33] M. Inoue, K. Takashima, K. Fujita, and Y. Kitamura, “BirdViewAR: Surroundings-aware Remote Drone Piloting Using an Augmented Third-person Perspective,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, (New York, NY, USA), Association for Computing Machinery, 2023.
  * [34] O. Erat, W. A. Isop, D. Kalkofen, and D. Schmalstieg, “Drone-augmented Human Vision: Exocentric Control for Drones Exploring Hidden Areas,” IEEE transactions on visualization and computer graphics, vol. 24, no. 4, pp. 1437–1446, 2018.
  * [35] N. Shiroma, N. Sato, Y.-h. Chiu, and F. Matsuno, “Study on Effective Camera Images for Mobile Robot Teleoperation,” in IEEE International Workshop on Robot and Human Interactive Communication (RO-MAN), pp. 107–112, 2004.
  * [36] K. Nagatani, S. Kiribayashi, Y. Okada, S. Tadokoro, T. Nishimura, T. Yoshida, E. Koyanagi, and Y. Hada, “Redesign of Rescue Mobile Robot Quince,” in 2011 IEEE International Symposium on Safety, Security, and Rescue Robotics, pp. 13–18, 2011.
  * [37] T. Sato, A. Moro, A. Sugahara, T. Tasaki, A. Yamashita, and H. Asama, “Spatio-temporal Bird’s-eye View Images using Multiple Fish-eye Cameras,” in Proceedings of the 2013 IEEE/SICE International Symposium on System Integration, pp. 753–758, 2013.
  * [38] F. Ferland, F. Pomerleau, C. T. Le Dinh, and F. Michaud, “Egocentric and exocentric teleoperation interface using real-time, 3d video projection,” in Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, HRI ’09, (New York, NY, USA), p. 37–44, Association for Computing Machinery, 2009.
  * [39] M. Lager, E. A. Topp, and J. Malec, “Remote Operation of Unmanned Surface Vessel through Virtual Reality -A Low Cognitive Load Approach,” 03 2018.
  * [40] S. Livatino, D. C. Guastella, G. Muscato, V. Rinaldi, L. Cantelli, C. D. Melita, A. Caniglia, R. Mazza, and G. Padula, “Intuitive Robot Teleoperation Through Multi-Sensor Informed Mixed Reality Visual Aids,” IEEE Access, vol. 9, pp. 25795–25808, 2021.
  * [41] J. Hing, K. Sevcik, and P. Oh, “Development and Evaluation of a Chase View for UAV Operations in Cluttered Environments,” Journal of Intelligent and Robotic Systems, vol. 57, pp. 485–503, 08 2010.
  * [42] J. Thomason, P. Ratsamee, J. Orlosky, K. Kiyokawa, T. Mashita, Y. Uranishi, and H. Takemura, “A comparison of adaptive view techniques for exploratory 3d drone teleoperation,” ACM Transactions on Interactive Intelligent Systems, vol. 9, pp. 1–19, 2019.
  * [43] M. J. Islam, J. Mo, and J. Sattar, “Robot-to-robot Relative Pose Estimation using Humans as Markers,” Autonomous Robots, vol. 45, no. 4, pp. 579–593, 2021.
  * [44] M. Ito, N. Sato, M. Sugimoto, N. Shiroma, M. Inami, and F. Matsuno, “A teleoperation interface using past images for outdoor environment,” in 2008 SICE Annual Conference, pp. 3372–3375, IEEE, 2008.
  * [45] R. Murata, S. Songtong, H. Mizumoto, K. Kon, and F. Matsuno, “Teleoperation System using Past Image Records for Mobile Manipulator,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014.
  * [46] C. Campos, R. Elvira, J. J. Gómez, J. M. M. Montiel, and J. D. Tardós, “ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM,” IEEE Transactions on Robotics, vol. 37, no. 6, pp. 1874–1890, 2021.
  * [47] I. A. Kazerouni, L. Fitzgerald, G. Dooly, and D. Toal, “A Survey of State-of-the-art on Visual SLAM,” Expert Systems with Applications, vol. 205, p. 117734, 2022.
  * [48] A. Macario Barros, M. Michel, Y. Moline, G. Corre, and F. Carrel, “A Comprehensive Survey of Visual SLAM Algorithms,” Robotics, vol. 11, no. 1, p. 24, 2022.
  * [49] A. Abdullah, T. Barua, R. Tibbetts, Z. Chen, M. J. Islam, and I. Rekleitis, “CaveSeg: Deep Semantic Segmentation and Scene Parsing for Autonomous Underwater Cave Exploration,” in IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024.
  * [50] E. Olson, “AprilTag: A Robust and Flexible Visual Fiducial System,” in 2011 IEEE international conference on robotics and automation, pp. 3400–3407, IEEE, 2011.
  * [51] J. Brooke et al., “SUS- A Quick and Dirty Usability Scale,” Usability Evaluation in Industry, vol. 189, no. 194, pp. 4–7, 1996.
  * [52] B. Joshi, S. Rahman, M. Kalaitzakis, B. Cain, J. Johnson, M. Xanthidis, N. Karapetyan, A. Hernandez, A. Quattrini Li, N. Vitzilaios, and I. Rekleitis, “Experimental Comparison of Open Source Visual-Inertial-Based State Estimation Algorithms in the Underwater Domain,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (Macau), pp. 7221–7227, Nov. 2019.
  * [53] J. Wu, B. Yu, and M. J. Islam, “3d reconstruction of underwater scenes using nonlinear domain projection,” in 2023 IEEE Conference on Artificial Intelligence (CAI), pp. 359–361, IEEE, 2023.  Best Paper Award.
  * [54] S. Rahman, A. Quattrini Li, and I. Rekleitis, “SVIn2: A Multi-sensor Fusion-based Underwater SLAM System,” International Journal of Robotics Research, vol. 41, pp. 1022–1042, July 2022.
  * [55] J. Mo, M. J. Islam, and J. Sattar, “Fast direct stereo visual slam,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 778–785, 2021.