# RestoreX-AI: A Contrastive Approach towards Guiding Image Restoration via
Explainable AI Systems

Aboli Marathe
Symbiosis Centre for Applied
Artificial Intelligence (SCAAI),
Symbiosis International Deemed
University (SIU), India
SCTR’s Pune Institute of
Computer Technology, India
<EMAIL_ADDRESS>Pushkar Jain
Symbiosis Centre for Applied
Artificial Intelligence (SCAAI),
Symbiosis International Deemed
University (SIU), India
SCTR’s Pune Institute of
Computer Technology, India
<EMAIL_ADDRESS>
Rahee Walambe
Symbiosis Centre for Applied
Artificial Intelligence (SCAAI),
Symbiosis International Deemed
University (SIU), India
Symbiosis Institute of Technology,
SIU, India

<EMAIL_ADDRESS>Ketan Kotecha
Symbiosis Centre for Applied
Artificial Intelligence (SCAAI),
Symbiosis International Deemed
University (SIU), India
Symbiosis Institute of Technology,
SIU, India

<EMAIL_ADDRESS>
###### Abstract

Modern applications such as self-driving cars and drones rely heavily upon
robust object detection techniques. However, weather corruptions can hinder
the object detectability and pose a serious threat to their navigation and
reliability. Thus, there is a need for efficient denoising, deraining, and
restoration techniques. Generative adversarial networks and transformers have
been widely adopted for image restoration. However, the training of these
methods is often unstable and time-consuming. Furthermore, when used for
object detection (OD), the output images generated by these methods may
provide unsatisfactory results despite image clarity. In this work, we propose
a contrastive approach towards mitigating this problem, by evaluating images
generated by restoration models during and post training. This approach
leverages OD scores combined with attention maps for predicting the usefulness
of restored images for the OD task. We conduct experiments using two novel
use-cases of conditional GANs and two transformer methods that probe the
robustness of the proposed approach on multi-weather corruptions in the OD
task. Our approach achieves an averaged 178 increase in mAP between the input
and restored images under adverse weather conditions like dust tornadoes and
snowfall. We report unique cases where greater denoising does not improve OD
performance and conversely where noisy generated images demonstrate good
results. We conclude the need for explainability frameworks to bridge the gap
between human and machine perception, especially in the context of robust
object detection for autonomous vehicles.

##  1 Introduction

Sensors are employed by unmanned autonomous vehicles to navigate through their
surroundings, with a substantial dependence on vision-based sensors like RGB
cameras. As these sensors are impacted by bad weather conditions, perception
pipelines require considerable training on diverse data to increase the
robustness on downstream tasks. One specific scenario which causes distortion
of images is adverse weather conditions, like heavy snowfall, haze and dust
tornados. In these critical situations, weather corruptions can hinder the
object detectability and pose a serious threat to navigation and reliability.
Thus, there is a need for efficient denoising, deraining and restoration
techniques.

However, denoising techniques are often evaluated using Image similarity
metrics like PSNR, SSIM [8] and not by their effectiveness in achieving
results for the targeted application. It is possible that the output image of
these methods has high image quality but is contextually irrelevant for the
object detection task. In this work we evaluate the effectiveness of
restoration techniques for denoising images with the intention of better
object detection. By introducing a contrastive approach towards restoration
evaluation, a method for guiding the training of GANs and restoration progress
is proposed. Additionally, attention maps are leveraged for understanding why
these techniques assist object detection and why certain classes are easily
recognized or ignored.

Primary contributions of this work are :

  1. 1.

Exploring two new data-based generative adversarial network techniques for
denoising weather corrupted images.

  2. 2.

Proposing a novel contrastive approach using a weighted loss for evaluating
the training progress and post-training performance of the highlighted
restoration techniques.

  3. 3.

Explaining the training progress of the proposed generative denoising methods
using attention maps and validating the results using object detection
evaluation metrics.

  4. 4.

Evaluating the optimal noise level of the trained Restormer denoising methods
(color and grayscale) using attention maps and validating the results using
object detection evaluation metrics.

Figure 1: RestoreX-AI: Proposed contrastive approach for evaluating
restoration models.

##  2 Related Work

The rapid development and deployment of autonomous vehicles have exposed
several critical challenges in computer vision, one of which being object
detection robust to weather corruptions. At the foundation of this challenge
is the classic vision task of object detection, the progress of which we
summarize in the following sub-section. Ultimately, when these modules are
integrated in autonomous driving systems, their suggestively “black-box”
nature, triggers the concerns of industry professionals, car-makers and users
alike. Without human-tailored explanations for the vehicle’s behavior, even
the most advanced systems fail to benefit from widespread adoption due to
concerns about safety and reliability. Especially in adverse weather, when
even human drivers exercise extra caution, these robust models need to provide
irrefutable explanations for the detections made at every state. We discuss
the progress of previous works in these 3 directions which set the background
of our work in restoration, detection and explainability.

###  2.1 Multi-weather corruption and restoration

The thermal variations accompanying weather change can adversely impact the
optical, electronic, and mechanical components used in capturing visual data,
thus harming the performance of visual recognition systems [6]. Frigid
temperatures, snowfall or dense fog, for example, can cause condensation on
the lens, further blurring the view and obscuring the object boundaries; rain
streaks on car windows can generate glares or act as a double lens [20]. For
an autonomous car, it is critical and essential to overcome the effects of
weather conditions to ensure reliability. Number of approaches for this have
been reported. For example, one study [4] looked at the performance of gated
cameras, while another [3] expanded the research to include stereo, gated, and
thermal cameras, as well as Radar and LiDAR scanners, and found considerable
increases in car recognition in varying levels of fog, and other adverse
weather conditions. The use of unique methodologies like domain adaptation to
transform the weather conditions while keeping objects of interest intact. For
example, [22] investigates the consequences of synthetic weather images on
road segmentation and traffic object detection, whereas [24] shows that using
synthetic time-of-day (night imagery) improves localization, and [23] proposes
a de-raining model to improve semantic segmentation. For their efficiency in
image restoration and effectiveness in removing weather corruptions, image
restoration employing restomers [34] and image denoising algorithms [5] are
becoming increasingly popular.

###  2.2 Object Detection

The introduction of fast detectors like SSD, Faster RCNN, and YOLO transformed
the face of object detection [16, 26, 25]. Taking 2D detectors forward, 3D
detection expanded with Stereo RCNN [15], AVOD [13], MVLidarNet [7] and MVF
algorithm [37] bringing new perspectives to the task of object detection.
Specific tasks like multiscale object detection [31], pedestrian detection in
crowds [17] and detection under adverse weather [32] have also been solved
using ensemble methods and data augmentation.

###  2.3 Explainability in Object Detection

The early rise of deployable machine learning technologies was accompanied by
criticism of the “black-box”-like nature of ML models. Particularly in
sensitive applications like medical diagnosis, self-driving cars and
algorithmic test checking, there is a need for thorough explainability in
models to ensure the trust and safety of users. To meet these requirements,
many techniques were proposed for explaining models [30, 36, 29, 1, 2]. In
[21], explainability is demonstrated by studying what each of its neurons has
learned to detect. [19] focused on individual predictions, using the technique
of heatmaps to highlight important pixels. Some works also interpret
classifiers by identifying representative training examples [11, 12]. [27]
introduced a new perspective to this challenge by making CNN-based models more
transparent by producing visual explanations. As newer machine learning
systems are rapidly adopted, the demand for explainable models which
incorporate diverse approaches is gaining attention of the community.

##  3 Methodology

This work proposes a contrastive approach (loss) for monitoring the training
progress of restoration models in the context of object detection. To put
forward a diverse range of training samples, new restoration techniques using
GANs have also been proposed. First, the restoration models were trained on
different tasks and tested on the DAWN dataset [10]. As the models trained,
their progress was monitored using the contrastive approach and simultaneously
attention maps were generations to support the detection task. Finally, the OD
performance of all the methods is compared using standard evaluation metrics
(class AP and mAP). The 4 restoration techniques experimented with in this
study are:

  1. 1.

Weather-NightGAN:Conditional GANs trained on night-to-day task for multi-
weather corruption restoration.

  2. 2.

Weather-RainGAN:Conditional GANs trained on rain-to-clear task for multi-
weather corruption restoration.

  3. 3.

Restormer (Gaussian color denoising) for multi-weather corruption restoration.

  4. 4.

Restormer (Gaussian grayscale denoising) for multi-weather corruption
restoration.

###  3.1 Conditional Generative Adversarial Networks

GANs or Generative Adversarial networks are generative models that learn
mapping between noisy z and output image y, G : z → y. Conditional GANs learn
a mapping from observed image x and random noise vector z, to y, G : x, z → y
[9]. The generator G is trained to produce images similar to the “real”
images, as compared by an adversarially trained discriminator, D, which is
used for detecting the “fakes”. The final objective of the conditional GAN can
be expressed as:

$G^{*}=argmin_{G}max_{D}\mathcal{L}_{cGAN}(G,D)+\lambda\mathcal{L}_{L1}(G)$

where G tries to minimize this objective against an adversarial D that tries
to maximize it, i.e. $G^{*}=argmin_{G}max_{D}L_{cGAN}(G,D).$

In this study, we propose 2 new use-cases of the conditional GAN for
restoration purposes. In the first case, the GAN is trained on night-to-day
images and tested for multi-weather corruption tasks. The intuition behind
this is the similarity in corruptions of night and bad weather images, like
poor lighting and condensation on lens. In the second use-case, the GAN is
trained on only rain-to-clear images (synthetically generated) and tested for
multi-weather corruption tasks. The intuition behind this is the similarity in
corruptions of rain and bad weather images, like snow and rain streaks. There
is an additional challenge which notes if the single-weather trained
conditional GAN can adapt to multi-weather corruptions.

###  3.2 Restormer

The restormer is a highly efficient transformer that was proposed for
denoising tasks in image restoration [35]. It consists of a multi-Dconv head
transposed attention (MDTA) and a gated-Dconv feed-forward network (GDFN).
These proposed architectural changes gave it the ability to capture long-range
pixel interactions, while still remaining applicable to large images. It is
both computationally efficient, and has the capacity to handle high-resolution
images, a feature critical for a task like adverse weather object detection.
In this work, we study the effects of trained noise levels (15, 25 and 50) on
the denoising performance of color and grayscale Restormers on the DAWN
dataset. The goal is to study how the noise level affects the model
performance in OD task and which level is optimal for generating explainable
detections.

###  3.3 Grad-CAM

Gradient-weighted Class Activation Mapping (Grad-CAM) is a technique that
produces visual explanations for the purpose of making CNN-based models more
transparent [28]. For getting the class discriminative localization map Grad-
CAM ${L_{Grad-CAM}^{c}}$ $\epsilon R^{u*v}$ of width u and height v for any
class c , first comes the computation of gradient of the score for class c,
$y^{c}$ (before the softmax), according to feature maps $A^{k}$ of a
convolutional layer, i.e. $\frac{\partial{\mathbf{y^{c}}}}{\partial A^{k}}$ .
The neuron importance weights $a_{c}^{k}$ are attained by global-average-
pooling these gradients flowing back. The ‘importance’ of feature map k for a
target class c is captured by this weight $a_{c}^{k}$. A weighted combination
of forward activation maps is performed, and followed by a ReLU to obtain,
This results in a coarse heat-map of the same size as the convolutional
feature maps. Grad-CAM is used for the purpose of explaining object detection
in the restored images of different techniques compared in this study. We
additionally use the Grad-CAM model’s detection probability in calculating the
contrastive metric for monitoring training progress.

###  3.4 Proposed Approach: RestoreX-AI

Due to the instability of training of GANs, over-or-under training does not
always lead to the perfect solution images for object detection. Parallelly,
tuning on high noise levels does not always provide the best images from the
Restormer model. Even after producing good images by standard metrics (PSNR,
SSIM), their applicability for the OD task remains uncertain, which brings the
need for a new evaluation standard. We propose using a weighted sum of the
explainability results (detection probability of class provided by the Grad-
CAM model) and the similarity of the predicted and actual label to define this
new standard. This weighted sum is calculated for every stage of training (one
stage can be a user-defined set of epochs), and then used to monitor the
progress of the model. We introduce this new parameter for assessing the
quality of restoration which is calculated using equation 1.

$\Delta\phi=\Delta(\Sigma(S(p,a)*d)/N)$ (1)

Here S is the similarity of labels that be measured either by grouping the
objects (cars, race cars and taxis have similarity 1, person, groom have 1 and
so on), or by strict parameters (cars and race cars have similarity 0). p and
a are the predicted and actual labels of the object under detection. N is the
number of training samples generated in that stage, which are used to evaluate
the current progress of that model. Refer to Appendix A Table 2 to view the
measure of similarity of objects grouped together for this study. The
explanation probability is d or the value returned by the Grad-CAM model,
which is its prediction of what is present in the image. The quality of
restoration between stages can be denoted as $\Delta\phi$, or the difference
between qualities at consecutive stages.

###  3.5 Datasets

For training the Weather-RainGAN and Weather-NightGAN, corresponding images of
the same scene in rain-clear and night-day conditions were required. For the
Weather-RainGAN, we used Rain 100L [33], which is a synthesized data of rain
streaks with corresponding rain-free images. For the Weather-NightGAN we used
Transient Attributes dataset [14], which used a high-level image editing
method which allows a user to adjust the attributes of a scene, e.g. change a
scene to be “night” or “day”. The final testing of all restoration methods
required a multi-weather dataset with high-resolution images, for which the
DAWN dataset was selected. The DAWN dataset is a large vehicle detection
dataset which has captured images of driving scenes in adverse weather
conditions [10]. It consists of 1000 images from real-traffic scenes as seen
in multiple adverse weather conditions including fog, snow, rain, and
sandstorms. The images have been annotated with 2D annotations(boxes) with 6
object classes namely car, bus, truck, motorcycle, person and bicycle.

Figure 2: Foggy weather condition: Grad-CAM Attention Maps for (a) Restormer Grayscale Denoising (b) Restormer Colour Denoising (c) Weather-RainGAN and (d) Weather-NightGAN. Figure 3: Snowfall weather condition: Grad-CAM Attention Maps for (a) Restormer Grayscale Denoising (b) Restormer Colour Denoising (c) Weather-RainGAN and (d) Weather-NightGAN. Image Restoration Technique |  | Class 1 AP
---
[car]
| Class 2 AP
---
[bus]
| Class 3 AP
---
[person]
| Class 4 AP
---
[motorcycle]
mAP
Gaussian Gray Denoising Restormer
Noise 15 | 1 | 5 | 22 | 0 | 6
Noise 25 | 1 | 9 | 23 | 0 | 6
Noise 50 | 1 | 0 | 29 | 0 | 6
Gaussian Color Denoising Restormer
Noise 15 | 1 | 3 | 21 | 0 | 5
Noise 25 | 1 | 5 | 22 | 0 | 6
Noise 50 | 1 | 11 | 24 | 0 | 7
Weather-RainGAN
Stage 1 | 1 | 0 | 0 | 0 | 0
Stage 2 | 1 | 17 | 0 | 0 | 4
Stage 3 | 1 | 0 | 0 | 0 | 0
Stage 4 | 1 | 0 | 0 | 76 | 15
Stage 5 | 1 | 0 | 0 | 0 | 0
Weather-NightGAN
Stage 1 | 11 | 0 | 0 | 0 | 2
Stage 2 | 1 | 0 | 0 | 0 | 0
Stage 3 | 1 | 0 | 0 | 0 | 0
Stage 4 | 17 | 0 | 0 | 0 | 3
Stage 5 | 48 | 0 | 0 | 0 | 10
Table 1: Object detection performance of all 4 restoration methods as measured
at the designed stages.

##  4 Experiments and Results

###  4.1 Restormer-based Grayscale Denoising

Section 1 of Table 1 shows the object detection scores of Gaussian grayscale
image denoising using the Restormer model. To compare the clarity and
detection scores, models trained on different noise levels 15, 25 and 50 are
included in testing. The purpose of this experimentation is to identify which
noise level is optimal for the object detection task, additionally verified by
the explainability setup shown in Figures 2 and 3. The overall mAP of object
detection remains constant on all noise levels, however individual class
scores are observed to fluctuate. The AP for bus and person class increases
from noise 15 to 25 by 4 AP and 1AP respectively. However the bus AP drops to
0 and person AP boosts to 29 when noise is set to 50. The increase in noise
level would correspond to smoother images produced by restormer, however, it
is observed that the bus AP drops when noise level is set to 50. This is very
interesting to note and also observable in Figure 2, where the attention maps
shift from the bus to the car as the noise level increases.

###  4.2 Restormer-based Colour Denoising

Section 2 of Table 1 shows the object detection scores of Gaussian color image
denoising using the Restormer model. To compare the clarity and detection
scores, models trained on different noise levels 15, 25 and 50 are included in
testing. The purpose of this experimentation is to identify which noise level
is optimal for the object detection task, additionally verified by the
explainability setup. The overall mAP of object detection increases steadily
by 1 with all increasing noise levels, and individual class scores are
observed to increase as well. The AP for bus and person class increases from
noise 15 to 25 by 2 AP and 1AP respectively. The bus AP further increases to
11 and person AP boosts to 24 when noise is set to 50. The increase in noise
level is improving the image quality and object detection. As visible in
Figure 3, the attention maps on the people are focusing in the region of
interest as the noise level increases.

###  4.3 Weather-RainGAN

The utilization of Pix2Pix GAN [9] for deraining purposes in mapped rain-clear
images is proposed, and validated using our proposed technique. The dataset
used for this purpose was Rain 100L [33], which is a synthesized data of rain
streaks with corresponding rain-free images. The images in Rain 100 L are
originally from BSD 200 dataset [18]. The GAN model is trained with the rainy
images as source and clear images as target, expecting this technique to
produce denoised images on test images of the DAWN dataset. The intuition
behind this idea was the inherent similarity between weather corruptions and
synthetic rain streaks, with the goal of the GAN learning to work similarly on
the 2 tasks. The training epochs are divided into stages(1,2,3,4,5) to monitor
the training progress of the GANs and produce results of OD and explainability
at each stage. As expected, the GANs are producing highly unstable behavior
with increasing and 0 AP at most of the stages. However, the sudden boost in
AP at specific epochs inspired the formulation of our proposed weighted
explainability measure. It can be seen in Section 3 of Table 1, that the bus
AP is 17 at Stage 4, but 0 at all other stages and the motorcycle AP is 76 at
Stage 8 and 0 at other Stages. The car AP however, remains constant at 1 and
overall mAP increases over time due to the individual class performance
boosts. As these results are vague, we take a closer look at Figures 2 and 3,
to determine possible causes for this object detection performance. It can be
observed that the GAN is actually performing quite well in denoising weather
conditions like fog and snow, and the attention maps are converging towards
the relevant objects as the training progresses. But while the object
detection results are very disjoint, the attention maps progress continuously
and show improvement. The effectiveness of this GAN can be observed as a
deraining solution as the image not only gets clearer, but the object
detection and attention maps get more precise over training as well.

###  4.4 Weather-NightGAN

A new use-case of the PixtoPix GAN [9] is proposed, trained on a different
use-case for deraining purposes in mapped rain-clear images, and validated our
proposed technique using the experimental procedure. The dataset used for this
purpose was Transient Attributes dataset [14], which used a high-level image
editing method which allows a user to adjust the attributes of a scene, e.g.
change a scene to be “night” or “day”. The GAN model is trained with the night
images as source and day images as target, expecting this technique to produce
denoised images on test images of the DAWN dataset. The intuition behind this
idea was the inherent similarity between weather corruptions and dark night
images. As expected, the GANs are producing highly unstable behavior with
increasing and 0 AP at most of the stages as shown in Table 1. This technique
actually worked out only for 1 class (car) and boosted its AP from 11 to 48
over the training period. The remaining classes had a constant 0 AP. By taking
a closer look at Figures 2 and 3, possible causes for this object detection
performance can be determined. The GAN is actually performing quite randomly
in denoising weather conditions like fog and snow using night training images,
but the attention maps are still converging towards the relevant objects as
the training progresses. This is exactly aligned with the initial analysis and
intuition for the discovery i.e. cases in which object detection is clear to a
CV detector, although imperceptible to the human eye. While the object
detection results are very disjoint, the attention maps progress continuously
and show improvement. The effectiveness of this GAN can be observed in
denoising weather conditions for cars, but not for other classes. The purpose
of our methodology is not just to observe the restoration progress in a
positive light, but to also stop training in case the GAN goes too far. As can
be seen in this particular case, the GAN distorts the images after Stage 2,
which may confuse detectors when tested against it. In Stage 4, the GAN
produces images which are heavily distorted but oddly easier for car detection
than even the original image.

The attention maps and results for all four restoration techniques and four
weather conditions: fog, rain, snow and dust tornado have been displayed in
Appendix A.

##  5 Conclusion

The goal of our work was to study the relationship between object detection
performance, image clarity, explainability and training time in the context of
novel versus established restoration techniques. The scope of this study
covered 4 different restoration techniques, aiming to denoise images corrupted
by over 6 different weather conditions presented in the DAWN dataset, namely
fog, snow storm, haze, dust tornadoes, rainfall and mist. All 4 techniques
worked differently, with the Restormer-based methods providing stable all-
rounded results, while the GANs provided class-specific boosts in performance.
The overall rise in mAP before and after applying the techniques was 0%, 40%,
275%, 400% respectively. Contrary to popular beliefs, greater denoising does
not always guarantee better object detection results as observed in this
paper. And conversely, poor denoising does not always guarantee worse object
detection results. Particularly for specific tasks like bus and car detection,
it can be seen that newer approaches like our proposed method (Weather-RainGAN
and Weather-NightGAN) can boost the detector’s performance with its resultant
images. We present conditional GANs cases that perform superbly on diverse
weather conditions ranging from dust tornadoes to snowfall, in spite of having
trained on limited single weather conditions.

We present 2 very interesting observations obtained through this study:

  1. 1.

GANs can generate images complex to the human eye, but comparatively
interpretable for vision models post processing. This opens the possibility of
exploring the capabilities of vision models beyond the scope of human vision
and also warding off potential attacks which can cripple modern detectors.

  2. 2.

Restoration and denoising methods which produce clearer images (as measured
using standard image quality metrics like PSNR), may in fact present a greater
challenge for machine perception in object detection.

Understanding how differently humans and detectors perceive information in the
same image demands greater exploration of explainability. We would like to
open the discussion for countless possibilities arising from these disparate
perspectives, E.g. if a model sees a pedestrian on a rainy road which a human
cannot see, or conversely a pedestrian visible to a passenger’s eye which a
model cannot capture. While using denoising techniques are a popular choice
for tackling corruptions, it must also be acknowledged that not all techniques
are suitable for all use-cases as seen for car detection. Certain classes may
respond better to denoising depending on their physical characteristics as
perceived by the detectors. Going ahead with building robust, explainable
models, this problem must be studied from multiple perspectives in the future.
We hope to inspire a new line of research in this direction which deals with
the complexity of the task of weather corruptions and can solve it using
diverse generalizable solutions.

## References

  * [1] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one, 10(7):e0130140, 2015.
  * [2] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller.  How to explain individual classification decisions.  The Journal of Machine Learning Research, 11:1803–1831, 2010.
  * [3] Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, and Felix Heide.  Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11682–11692, 2020.
  * [4] Mario Bijelic, Tobias Gruber, and Werner Ritter.  Benchmarking image sensors under adverse weather conditions for autonomous driving.  In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1773–1779. IEEE, 2018.
  * [5] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel.  A review of image denoising algorithms, with a new one.  Multiscale modeling & simulation, 4(2):490–530, 2005.
  * [6] Pak Hung Chan, Gunwant Dhadyalla, and Valentina Donzella.  A framework to analyze noise factors of automotive perception sensors.  IEEE Sensors Letters, 4(6):1–4, 2020.
  * [7] Ke Chen, Ryan Oldja, Nikolai Smolyanskiy, Stan Birchfield, Alexander Popov, David Wehr, Ibrahim Eden, and Joachim Pehserl.  Mvlidarnet: Real-time multi-class scene understanding for autonomous driving using multiple views.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2288–2294. IEEE, 2020.
  * [8] Alain Hore and Djemel Ziou.  Image quality metrics: Psnr vs. ssim.  In 2010 20th international conference on pattern recognition, pages 2366–2369. IEEE, 2010.
  * [9] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [10] Mourad A Kenk and Mahmoud Hassaballah.  Dawn: vehicle detection in adverse weather nature dataset.  arXiv preprint arXiv:2008.05402, 2020.
  * [11] Rajiv Khanna, Been Kim, Joydeep Ghosh, and Sanmi Koyejo.  Interpreting black box predictions using fisher kernels.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3382–3390. PMLR, 2019.
  * [12] Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In International conference on machine learning, pages 1885–1894. PMLR, 2017.
  * [13] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander.  Joint 3d proposal generation and object detection from view aggregation.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–8. IEEE, 2018.
  * [14] Pierre-Yves Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian, and James Hays.  Transient attributes for high-level understanding and editing of outdoor scenes.  ACM Transactions on Graphics (proceedings of SIGGRAPH), 33(4), 2014\.
  * [15] Peiliang Li, Xiaozhi Chen, and Shaojie Shen.  Stereo r-cnn based 3d object detection for autonomous driving.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [16] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer, 2016.
  * [17] Aboli Marathe, Rahee Walambe, and Ketan Kotecha.  Evaluating the performance of ensemble methods and voting strategies for dense 2d pedestrian detection in the wild.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3575–3584, 2021.
  * [18] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423. IEEE, 2001.
  * [19] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Methods for interpreting and understanding deep neural networks.  Digital Signal Processing, 73:1–15, 2018.
  * [20] Valentina Mușat, Ivan Fursa, Paul Newman, Fabio Cuzzolin, and Andrew Bradley.  Multi-weather city: Adverse weather stacking for autonomous driving.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2906–2915, 2021.
  * [21] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.  Advances in neural information processing systems, 29, 2016.
  * [22] Vladislav Ostankovich, Rauf Yagfarov, Maksim Rassabin, and Salimzhan Gafurov.  Application of cyclegan-based augmentation for autonomous driving at night.  In 2020 International Conference Nonlinearity, Information and Robotics (NIR), pages 1–5. IEEE, 2020.
  * [23] Horia Porav, Tom Bruls, and Paul Newman.  I can see clearly now: Image restoration via de-raining.  In 2019 International Conference on Robotics and Automation (ICRA), pages 7087–7093. IEEE, 2019.
  * [24] Horia Porav, Will Maddern, and Paul Newman.  Adversarial training for adverse conditions: Robust metric localisation using appearance transfer.  In 2018 IEEE international conference on robotics and automation (ICRA), pages 1011–1018. IEEE, 2018.
  * [25] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [26] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [27] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [28] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [29] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje.  Not just a black box: Learning important features through propagating activation differences.  arXiv preprint arXiv:1605.01713, 2016.
  * [30] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [31] Rahee Walambe, Aboli Marathe, and Ketan Kotecha.  Multiscale object detection from drone imagery using ensemble transfer learning.  Drones, 5(3):66, 2021.
  * [32] Rahee Walambe, Aboli Marathe, Ketan Kotecha, and George Ghinea.  Lightweight object detection ensemble framework for autonomous vehicles in challenging weather conditions.  Computational Intelligence and Neuroscience, 2021, 2021.
  * [33] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan.  Deep joint rain detection and removal from a single image.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1357–1366, 2017.
  * [34] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Restormer: Efficient transformer for high-resolution image restoration.  arXiv preprint arXiv:2111.09881, 2021.
  * [35] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Restormer: Efficient transformer for high-resolution image restoration.  arXiv preprint arXiv:2111.09881, 2021.
  * [36] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.
  * [37] Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan.  End-to-end multi-view fusion for 3d object detection in lidar point clouds.  In Conference on Robot Learning, pages 923–932. PMLR, 2020.

##  Appendix A Appendix

The table 2 contains the categories of labels grouped with a similarity
measure of 1 in the proposed approach.

Person | Car | Motorcycle | Bus | Bicycle
---|---|---|---|---
Groom | Cab | Moped | Trolley bus | Tandem bicycle
Bridegroom | Taxi |  | Mini bus | Tricycle
Baseball player | Race car |  | School bus | Unicycle
Scuba Diver | Jeep |  |  | Mountain bike
| Minivan |  |  | All-terrain bike
| Estate car |  |  | Off-roader
| Station wagon |  |  | Trike
Table 2: Categories of labels grouped with Similarity 1.

Given below are the extra figures to supplement the experimentation section of
the paper.

Figure 4: Restormer Grayscale Denoising: Grad-CAM Attention Maps for (a) Fog
(b) Dust tornado (c) Rain and (d) Snow weather conditions. Figure 5: Restormer
Color Denoising: Grad-CAM Attention Maps for (a) Fog (b) Dust tornado (c) Rain
and (d) Snow weather conditions. Figure 6: Weather-RainGAN: Grad-CAM Attention
Maps for (a) Fog (b) Dust tornado (c) Rain and (d) Snow weather conditions.
Figure 7: Weather-NightGAN: Grad-CAM Attention Maps for (a) Fog (b) Dust
tornado (c) Rain and (d) Snow weather conditions.


# Learned Contextual LiDAR Informed Visual Search in Unseen Environments

Ryan Gupta1, Kyle Morgenstein1, Steven Ortega2 and Luis Sentis1 1Department of
Aerospace Engineering and Engineering Mechanics, University of Texas at
Austin, Austin, TX 78712 USA<EMAIL_ADDRESS>of Mechanical
Engineering, University of Texas at Austin, Austin, TX 78712 USA

###### Abstract

This paper presents LIVES: LiDAR Informed Visual Search, an autonomous planner
for unknown environments. We consider the pixel-wise environment perception
problem where one is given 2D range data from LiDAR scans and must label
points contextually as map or non-map in the surroundings for visual planning.
LIVES classifies incoming 2D scans from the wide Field of View (FoV) LiDAR in
unseen environments without prior map information. The map-generalizable
classifier is trained from expert data collected using a simple cart platform
equipped with a map-based classifier in real environments. A visual planner
takes contextual data from scans and uses this information to plan viewpoints
more likely to yield detection of the search target. While conventional
frontier based methods for LiDAR and multi sensor exploration effectively map
environments, they are not tailored to search for people indoors, which we
investigate in this paper. LIVES is baselined against several existing
exploration methods in simulation to verify its performance. Finally, it is
validated in real-world experiments with a Spot robot in a 20x30m indoor
apartment setting. Videos of experimental validation can be found on our
project website at https://sites.google.com/view/lives-icra-2024/home.

##  I INTRODUCTION

Autonomous planning and real-world execution for active search and exploration
are receiving significant attention from the robotics community due to their
relevance in several scenarios including inspection, surveillance, and Search
and Rescue (SAR) [27]. Autonomous ground robots cannot assume reliable map
information a priori during active search missions, for example in a disaster
response scenario. This demands environment perception tailored for the task
of locating any persons in need quickly. Today’s robots are frequently
equipped with LiDAR sensors that cast a full view of the surroundings.
However, use of these scans is often limited to the task of localization and
mapping. This paper addresses the exploitation of wide Field of View (FoV)
LiDAR scans for visual planning. We propose to classify LiDAR scan points as
map or non-map in the robot’s surroundings for improved visual search
planning.

Several methods exist for pixel-wise classification of 3D LiDAR scans [22, 23,
34]. However, [15] notes that these methods are limited by the high
computational requirements to handle to large input size of 3D cloud data. In
contrast, real time projection-based methods aim to detect moving obstacles
[13] or objects to be avoided in 3D scans, often for autonomous driving
applications [18, 5, 20]. Despite parallel improvements in efficiency of
planning algorithms for both LiDAR and vision sensors [6, 14, 28, 32] it
remains an open problem to enhance performance in autonomous planning and
execution for the task of visual search. There exists a gap of real-time
pixel-wise classification of LiDAR scans for active visual search tasks. This
work addresses this limitation with an online 2D pixel-wise LiDAR scan
classifier and context informed visual planner. In particular, classified
LiDAR scans are used to generate highly informative plans quickly with a
multisensor Frontier Exploration [33] based planner.

Figure 1: Overview of LIVES. In (a) raw 2D LiDAR scans, pose information and
image data are received from the Spot robot. They are sent through a neural
network classifier III-C (b) to label each point as map or non-map (c).
Separately, LiDAR scans and pose data are used to maintain the global search
maps (d). Global search maps are used to generate frontier points (e), which
are used for finding candidate viewpoints. Finally, each frontier point is
sampled at four viewpoints (f) and scored III-E2. In this toy example the left
viewpoint in (f) will be scored highest and selected as the next waypoint due
to the high number of unknown cells and non-map points observed. Finally, the
selected viewpoint is sent back to the robot for execution.

Active planners in unknown spaces are required to make decisions based on
incomplete and noisy information about partial environments. State of the art
visual task planners like [6, 32] account for LiDAR scans without considering
contextual information. We provide a visual planner with map and non-map data
in the surroundings found in wide FoV LiDAR scans. This is accomplished by
learning to distinguish points unlikely to be permanent features in unknown
environments. The visual planner gives priority to features of interest under
the assumption that the search target belongs to the set of non-map points.
This information obviates the need to visually inspect parts of the
environment that are likely to be permanent. An overview of the proposed
method can be found in Fig. 1.

The contributions of this work can be summarized as follows:

  * •

A learning strategy for pixel-wise scan classification as map and non-map in
unseen, real-world environments

  * •

A formulation that incorporates classified scan information into a planner for
visual search

  * •

Demonstrate reduced mission completion time in the visual search task over
existing planners

  * •

Deploy and verify the proposed method on a real robotic system in an unseen
indoor environment

Sections II, III and IV describe related works, methods an results including
baseline comparisons.

##  II Related Works

###  II-A Lidar Classification

Indoor pointcloud classification is tackled in 3D [30] using the KPConv
architecture [29]. This method requires the offline annotation and map
building at instantiation, and is used for localization and navigation in
later work [31]. Neither work validates on a real robot platform. Ref. [21]
presents an in depth comparison and framework for Iterative Closest Point
(ICP) methods, similar to that of [30], focused on map building. Finally, [26]
present a map-free framework for segmenting 3D pointclouds into ground, edges
and planar points, which are used in a feature extraction process for mapping
environments. In contrast LIVES recognizes non-map features.

###  II-B Next-Best View and Informative Path Planning

Exploration is often solved using next-best viewpoint planners like Frontier
Exploration [33, 14]. A common approach uses an active perception formulation
to maximize entropy reduction [17, 12]. Ref. [11] learns a policy to generate
non-myopic plans online by estimating potential gains of exploring different
areas. Ref. [19] similarly learn an information aware policy offline that
proposes waypoints for the robot. A local planner then generates dynamically
feasible trajectories based on these suggestions. In contrast to the proposed
approach, [11, 19] only provide simulation results. Similar to our method,
[36] focuses on the problem of visual search. They propose an end-to-end
policy that learns to generate discrete actions that lead a robot to a
prescribed viewpoint in the given environment, however, the action space is
limited to simple discrete actions and they do not leverage LiDAR information.
Ref. [35] propose a POMDP planner for generalized local object search in
cluttered 3D environments and extend it with a 2D planner for larger areas. In
contrast to LIVES, this work is aimed at small areas and small objects, rather
than quickly searching buildings for people.

###  II-C World Prediction Based Planning

Often world prediction methods involve learning a model for relevant
environments and leveraging their structure for exploring high-value regions
first. Ref. [25] use topological features from a database of worlds to inform
a frontier-based exploration policy and increase the rate of area explored.
Similarly, [28] uses a prediction module that detects semantic objects,
classifies frontiers, and predicts information gain beyond those frontiers. A
planner uses this to explore rooms fully before continuing global exploration,
unlike LIVES that avoids complete exploration to reduce search completion
time. Ref. [24] instead consider visual navigation with occupancy anticipation
from egocentric RGB-D based visual context. Ref. [10] instead learn to predict
locations of exits in building with a convolutional neural network (CNN) using
a database of building blueprints. Prediction guide the robot towards likely
exit locations.

##  III Methods

The overall scan classifier and planning module are described in Alg. 1. The
key components are detailed in this section.

Algorithm 1 Map-Free LiDAR Informed Search()

$s_{i},x_{i}$

LiDAR Scan, Robot Pose

$x^{*}_{\text{next}}$ (Next viewpoint)

$s_{i}^{\text{class}}=\text{NeuralNetworkClassifier}(s_{i})$

$\mathcal{M}^{\text{LiDAR}}_{i}\leftarrow\text{LiDARMapUpdate}(s_{i},x_{i},\mathcal{M}^{\text{LiDAR}}_{i-1})$

$\mathcal{M}^{\text{Visual}}_{i}\leftarrow\text{VisualMapUpdate}(s_{i},x_{i},\mathcal{M}^{\text{Visual}}_{i-1})$

$\\{\mathbb{C}^{\text{LiDAR}}\\}=\text{GetLiDARFrontiers}(\mathcal{M}^{\text{LiDAR}}_{i})$

$\\{\mathbb{C}^{\text{Visual}}\\}=\text{GetVisualFrontiers}(\mathcal{M}^{\text{Visual}}_{i})$

for $n\leftarrow 1$ to $N$ do $\triangleright$ for each frontier $c_{n}$ in
$\mathbb{C}$

for $j\leftarrow 1$ to $4$ do $\triangleright$ for each viewpoint at $c_{n}$

$\text{util}_{n,j}=\text{ComputeSampleUtility}(c_{n},x_{i},s_{i}^{\text{class}})$

end for

end for

$x^{*}_{\text{next}}$ $\leftarrow$ $\arg\max_{n,j}(\text{util}_{n,j})$

return $x^{*}_{\text{next}}$

###  III-A Environment

The environment is described as a discrete set of points $\mathcal{E}$. This
can be split into two subsets $\mathcal{E}^{\text{free}}\subset\mathcal{E}$
and $\mathcal{E}^{\text{occ}}\subset\mathcal{E}$, which represent the free and
occupied cells in the environment. Both of these sets are initially unknown to
the robot, but are assumed to follow a structure inherent in indoor
environments (e.g. hallways, planar walls, open rooms and loops).

###  III-B Ground Truth LiDAR Scan Classification

The search target of interest belongs to
$\mathcal{E}^{\text{occ}}\subset\mathcal{E}$. In particular, scan
classification attempts to divide $\mathcal{E}^{\text{occ}}$ into two subsets,
namely $\mathcal{E}^{\text{non-map}}$ and $\mathcal{E}^{\text{map}}$. In this
work, $\mathcal{E}^{\text{map}}$ are points in the environment that are deemed
to be Long-Term Features (LTFs) during classification, while
$\mathcal{E}^{\text{non-map}}$ are points that are either Short-Term Features
(STFs) or Dynamic Features (DFs) [9]. LTFs represent the static map obstacles,
STFs represent static un-mapped obstacles and DFs represent moving points.
Given map information, scan points can be classified into these three
categories as follows.

Let $x_{i}$ denote the pose of the robot, and $s_{i}$ denote observation at
time step $t_{i}$. Each observation $s_{i}$ consists of $n_{i}$ 2D points,
$s_{i}=\\{p^{j}_{i}\\}_{j=1:n_{i}}$. Observations are transformed from robot
local frame into the global frame using an affine transformation $T_{i}\in
SE(3)$. Finally, let map $\mathcal{M}$ be represented as a set of lines
$\\{l_{i}\\}_{1:n}$.

####  III-B1 LTF

First, an analytic ray cast is performed [8] to determine expected laserscan
based on map $\mathcal{M}$ and current robot position $x_{i}$. Given
observations, the probability that points correspond to one of the lines of
that static map can be written

$P(p_{i}^{j}|x_{i},M)=\text{exp}\left(-\frac{\text{dist}(T_{i}p_{i}^{j},l_{j})^{2}}{\Sigma_{s}}\right)$
(1)

where $\Sigma_{s}$ is the scalar variance of observations, which comes from
sensor accuracy. If Eq. 1 is greater than a threshold, point $p_{i}^{j}$ is
classified as a LTF.

####  III-B2 STF

Remaining points will be classified as STF or DF. Observations at current time
$i$, $p_{i}^{j}$, are compared with prior observations at time $k$,
$p_{k}^{l}$ to determine correspondence between points in subsequent
observations. The likelihood of the remaining points corresponding to the same
point as in a previous laserscan is computed as

$P(p_{i}^{j},p^{l}_{k}|x_{i},x_{k})=\text{exp}\left(-\frac{||T_{i}p_{i}^{j}-T_{k}p^{l}_{k}||^{2}}{\Sigma_{s}}\right)$
(2)

where $p^{l}_{k}$ is the nearest point from $p_{i}^{j}$ among points which
does not belong to LTF at other timesteps, defined as

$p^{l}_{k}=\arg\text{min}||T_{i}p_{i}^{j}-T_{k}p^{l}_{k}||$ (3)

When Eq. 2 is greater than some threshold, point $p_{i}^{j}$ is classified as
an STF. Remaining points in $p_{i}^{j}$ are classified as DFs. The result is
classified scan $s^{\text{class}}_{i}$. In practice, there are a large number
of STFs and DFs due to the size of scans, therefore it is important to filter
them. First, the pooling operator is used to reduce duplicates within a
radius. Second, pooled STFs and DFs within some distance of LTFs are removed
to eliminate false positives caused by localization drift. Finally, points
inside visually observed regions of the map are removed.

###  III-C Map-free LiDAR Scan Classification

Figure 2: Pixel-wise LiDAR scan classification model architecture. $k$ is the
length of the history buffer. (a) $[3,k]$ pose history matrix containing
$[x,y,\theta]$. (b) $[n_{i},k]$ LiDAR range history matrix. (c)
$[n_{i},k-1]\cup[n_{i},1]$ estimated label history matrix concatenated with
its pixel-wise exponential weighted average. (d) The model consists of three
temporal-convolutional encoders (TCN), one for each input. The encoded poses,
scans, and labels are combined to produce a pixel-wise classification of the
LiDAR scan. In (e) a threshold is applied to the raw logits such that each
pixel is classified as either a map point or a non-map point.

The classification method defined in III-B is limited by the necessity of
static map information $\mathcal{M}$. Therefore, this work uses supervised
learning to train a model to reproduce classifications for
$\\{p^{j}_{i}\\}_{j=1:n_{i}}$ in scan $s_{i}$. Similarly to the ground-truth,
non-map points are filtered by pooling and then by removing those in
previously inspected regions.

####  III-C1 Dataset and Data Acquisition

The collected dataset $\mathcal{D}$ consists of tuples of data from the ground
truth classification model. The tuples are given as
$\mathcal{D}=\\{(x_{i},s_{i},s_{i}^{\text{class}})\\}^{N}_{i=1}$, where
$x_{i}$ represents robot pose, $s_{i}$ is the raw LiDAR scan data, and
$s_{i}^{\text{class}}$ is the classified scan. $N$ is the total number of data
points collected. Collecting a dataset using a real robot requires a human
expert to control the robot over long periods of time. This approach is
difficult to scale due to mechanical limits of the robot and difficulty of
control over large, potentially out-of-sight places. To ease data acquisition
we design a steerable cart platform (see fig. 3) to mimic the robot. Human
operators easily maneuver this cart to generate the dataset without strain to
the robot. We mount a Velodyne Puck LiDAR for recording scan information and a
front-mounted Intel RealSense T265 stereo camera to obtain the cart’s odometry
estimate, required for running the ground-truth classification. Finally, a
laptop is used to maintain the pose estimate [9], collecting the data and
generating online ground-truth labels for the dataset. This laptop records 1)
robot pose at each timestep, 2) raw scan data, and 3) classified scan from the
ground-truth module for dataset $\mathcal{D}$. The Puck has $n_{i}=897$ and
classified scans are recorded at 5Hz. A dataset of size $N\simeq 145,000$ is
collected in this study, representing roughly 8 hours of time. In practice,
the model only receives raw scan data, pose estimate and previously predicted
labels from the robot during operation and must classify pixel-wise the scan
online.

Figure 3: A figure depicting the cart used for labeled data acquisition and
the Spot robot used for deployment of the proposed method. The RealSense
provides odometry estimate to the cart platform, required for localization and
ground-truth estimation. The spot is equipped with an RGB-D Azure Kinect for
detection during the search task.

####  III-C2 Architecture and Training

We formulate pixel-wise LiDAR classification as a supervised learning problem
during training and an auto-regressive problem during inference. Given dataset
$\mathcal{D}$, we shuffle and batch the dataset between time steps and
locations to minimize location-based and temporal bias during training.
Because no map information is provided explicitly, we provide the model with a
history buffer of $k$ time steps for scan ranges and robot poses, and $k-1$
previously estimated ground truth labels. Because we do not have access to
ground truth labels at inference time, we corrupt the ground truth labels
during training by randomly bit-flipping the classification of 10% of the
labels, chosen by sampling indices from a uniform random distribution. We then
take the exponential weighted average of the last $k-1$ corrupted pixel-wise
classifications to estimate the current pixel-wise classification. This
estimate is concatenated with the $k-1$ corrupted ground truth labels to
produce a $k$-length matrix of estimated scan labels. During inference, the
label history buffer is populated by previous estimates from the policy,
cropped by a length $k$ sliding-window. The history buffer is then
concatenated with the updated pixel-wise exponential weighted average. At the
initialization of inference, the policy is bootstrapped with zero-value poses,
scans, and labels, and run for $k$ steps of inference until the history buffer
is initialized.

The policy consists of a temporal-convolutional encoder (TCN) for each input.
Each encoder contains a single convolutional layer with scan-wise circular
padding and a single linear layer. The scan and label encoders have kernel
size $[k,k]$, and the pose encoder has kernel size $[1,3]$. Hyperbolic tangent
activations are used for all layers. The output of each encoder is the same
size as the LiDAR scan to be classified. The output of the pose encoder is
summed with the output of the scan encoder as a pose correction, and then
normalized by applying a hyperbolic tangent function. The resulting LiDAR scan
encoding is multiplied element-wise with the label encoding. A threshold is
applied to the raw logits such that each pixel is classified as either a map
point or a non-map point. The policy is trained with mean squared error loss
against ground truth classifications. Training for 20 epochs on a workstation
Nvidia 3080 12Gb takes 3 minutes. The full model architecture is shown in Fig.
2.

###  III-D Map Updates

Two global occupancy maps (search maps) are maintained. The first is a LiDAR
global search map and the second is a visual global search map. Global search
maps are implemented in this work using ROS Costmap2D [2]. All cells in both
global maps are initialized to be unknown, $e=-1$ $\forall e\in\mathcal{E}$.
The robot is equipped with a visual sensor with a cone shaped FoV, implemented
as a triangular costmap. The LiDAR sensor has a 360 degree FoV, represented as
a square-type costmap. Observations for both sensors are a set of range
estimates to the nearest occupied point $e\in\mathcal{E}^{\text{occ}}$. Points
between the robot and the nearest occupied point along each ray of the LiDAR
scan are free points $e\in\mathcal{E}^{\text{free}}$. Given incoming
observations from the robot, each global costmap is updated as free space
$e=0$ or occupied $e=1$. This process is shown in Fig. 1(d).

###  III-E Multisensor Frontier Exploration

The planning module seeks to generate a viewpoint at each planning step that
results in the highest likelihood to find the target of interest. This is
achieved in several steps: candidate viewpoints are generated, viewpoints are
scored, and finally the point with the highest score is fed to the robot for
navigation to that point.

####  III-E1 Candidate Viewpoint Generation

Frontier points in each of the two global search maps are generated using a
process similar to [33], shown in Fig. 1(e). Any free cell ($e=0$) adjacent to
an unknown cell ($e=-1$) is a frontier edge, which are grouped together into
frontiers. Candidate viewpoints at frontiers are similar to those of [32].
First, a set of vision based viewpoints are generated by clustering frontiers
on the visual global search map. Next, a set of LiDAR based viewpoints are
generated by clustering frontiers on the LiDAR global search map. Each
frontier cluster becomes four candidate viewpoints, facing each direction, to
account for the limited FoV of the vision sensor.

####  III-E2 Viewpoint Selection

Candidate viewpoints are selected based on a computationally efficient
heuristic utility function. High value viewpoints are those that are 1) near
the robot, 2) allow inspection of many unknown map points and 3) result in
inspection of non-map points. This process is shown in Fig. 1(f). The utility
functions is a weighted sum of the following:

  * •

Subtract path distance from robot position to the viewpoint

  * •

Add a reward for expected number of unknown cells discovered at that viewpoint

  * •

Add a reward for viewpoints with frontiers near the path to the candidate
viewpoint

  * •

Add a reward for viewpoints that result in inspection of a scaled number non-
map points

Scaling parameters on each term are tuned during experiments. Weight are set
such that the agent is inclined to prioritize non-map points and those
viewpoints that result in high number of expected cells discovered. The next
waypoint is selected by sending the highest utility viewpoint,
$x^{*}_{\text{next}}$, to the robot.

##  IV Results

We test three hypotheses through our evaluation: (1) The proposed approach to
inform an exploring agent with non-map features is effective at finding search
targets, (2) LIVES reduces detection time compared against existing
exploration algorithms, and (3) The proposed method can be deployed on real
robot hardware and used in a human environment.

###  IV-A LiDAR Scan Classification

Figure 4: Ablation studies over policy architecture (top) and injected noise
during training (bottom). The inclusion of the label history buffer yields
11.63% higher test accuracy. The policy is robust up to 30% to bit-flipping
errors in the label history buffer. The mean accuracy depicts the 5-step
moving average.

First we compare the learned classification policy to the ground truth
classifier. The output of the policy is a vector of size $n_{i}$ normalized
between -1 and 1 for each scan point. If this value is non-negative, the point
is classified as a member of $\mathcal{E}^{\text{map}}$. Otherwise the point
is classified as a member of $\mathcal{E}^{\text{non-map}}$. We express
performance in terms of per-scan accuracy of the classified LiDAR scans from
the neural network estimator versus ground truth scan classification from map
information. This metric for a classified scan $s^{\text{class}}_{i}$ at time
$i$ compared to ground truth $s^{\text{class, true}}_{i}$ is given by

$\text{accuracy}=\frac{1}{{n_{i}}}\sum^{n_{i}}_{j=0}\\{+1\text{ if
}p^{j}_{i}==p^{j,\text{true}}_{i}\text{ else }0\\}$ (4)

The final test accuracy of the policy compared to ground truth is $86.19\%\pm
0.03\%$ with no a priori map requirement. We perform two ablation studies,
modulating the model architecture and injected noise during training, with
results in Fig. 4. The inclusion of the label encoder is responsible for a
11.63% increase in test accuracy. Because the robot moves relatively slowly
compared to the policy update frequency, recently predicted labels strongly
bias the current estimate toward the correct classification. We fix the
history buffer to 9 time steps, corresponding to 1.8 seconds. The history
buffer enables the policy to reason about its environment even though no
reference map is provided. During training, uniform random noise is injected
into the label history buffer to simulate inference-time auto-regressive
accumulated errors. The policy is robust up to 30% of injected noise before
the policy begins to over-fit and performance degrades. We select a policy
trained with 10% injected noise to deploy on the robot. All ablation studies
were performed with a fixed seed (0) and initialized with identical weights.
We do not tune the random seed.

###  IV-B Simulation and Baseline Comparison

Simulations are performed using Gazebo-ROS on the Turtlebot [4]. ROS move base
is used for navigation [1]. Baselines are: 1) a lightly modified version of
[32] (Multisensor Frontier Exploration), 2) the Next-Best View Planner (NBVP)
[7], 3) a Rapidly-Exploring Random Tree (RRT) [3], and 4) LIVES given ground-
truth scan classifications. The Multisensor baseline is equivalent to LIVES
and ground-truth without contextual LiDAR information.

Figure 5: The Apartment simulation environment (top row) in the Easy configuration (a) and Hard Configuration (b). The Office (c) shows the robot starting position as a red arrow. Easy setup involves the search target to the left of the cyan line and Hard involves the search target on the right of the cyan line. Map | Target | NBVP | RRT | Multisensor Frontier | Ground Truth | LIVES
---|---|---|---|---|---|---
Name | Difficulty | [7] | [3] | Exploration [32] | Classification | (Our Method)
Apartment | Easy | $112(70\%)$ | $72(80\%)$ | 36 (100%) |  34(100%) | 34(100%)
| Hard | $177(30\%)$ | $128(60\%)$ | 106 (100%) |  92(100%) | 90(100%)
Office | Easy | $70(80\%)$ | $54\pm(80\%)$ | 34 (100%) |  26(100%) | 24(100%)
| Hard | $300(0\%)$ | 111 (80%) | $141(90\%)$ |  99(100%) | 105(100%)
TABLE I: Average time (s) for each method to find the object of interest in
the given simulation environment.

Two Gazebo world environments are tested with easy and hard setups in each.
The first map is the apartment (20x30m) and examines the impact of non-target
objects on LIVES (see Fig. 5(a-b)). Here the easy setting incorporates fewer
objects in the environment and the search target being closer to the robot
while hard is defined by more objects in the environment and larger distance
to the target. The second map is the office (25x45m) and tests the method in a
large environment under significantly varied target position (see Fig. 5(c)).
The easy and hard settings are defined by the search target being in the same
or different half of the environment as the robot starting position. Both
difficulty levels in each map are repeated 10 times. Average detection times
and success rates are reported in Table I. The time limit given to planners is
2 and 3 minutes in the apartment easy and hard settings then 3 and 5 minutes
in the office easy and hard settings. Trials where the planner fails or the
time limit is reached are reported as a failure and failed trials are reported
as the maximum allotted time.

Results indicate that LIVES, the Ground-Truth variant and the Multisensor
Frontier Exploration method significantly outperform the RRT and NBVP
baselines in all settings except for the hard office setting, where the RRT
method performs similarly to LIVES and Ground-Truth methods. In the easy
apartment setting we observe similar performance among the three planners.
However in the hard setting, where LIVES and Ground-Truth perform nearly
identically, there is roughly a 15% improvement with the addition of the
knowledge of non-map points compared with the Multisensor method. We see a
large gap, however, in the easy office setting with a near 30% improvement
with the addition of contextual LiDAR information over the Multisensor
variant. In the hard office setting, LIVES and Ground-Truth variant outperform
the RRT method by 5-12%. Overall, the results indicate that the proposed
method to inform the frontier planner with non-map information outperforms the
baselines across the full spectrum of environments. We hypothesize this is due
to the requirement for the exploration planners to exhaustively explore the
environment, whereas the proposed method focuses on non-map points, allowing
the agent to move on when the target is unlikely to be nearby. These results
support hypotheses (1) and (2) that contextual LiDAR information significantly
improves performance in the search task.

Figure 6: A time lapse of one experiment scenario. In (a) the robot pose is a
purple arrow with the visual costmap, global search map, frontier points and
non-map points shown. The raw scan data is shown as small white squares. The
target object is behind the robot starting position. Figure (b) occurs
immediately before a planning step, wherein the marked frontier is selected
due to the high occurrence of non-map points nearby. (c) occurs after the
planning step and results in detection.

###  IV-C Hardware Experiments

Experiments are in a 20mx30m indoor apartment setting. The environment is out
of distribution as it is not included in dataset $\mathcal{D}$ used to train
the pixel-wise classification policy. The platform used is a Boston Dynamics
Spot robot with a Velodyne Puck LiDAR and Azure Kinect RGB-D camera. The
search target of interest is a small rolling suitcase and YOLOv5 [16] is used
for detection of the object. On detection, the exploration node is shut down
and the task is complete. We perform several trials over varying initial robot
position and target locations. A time lapse of one experiment is shown in Fig.
6 where the search target is situated behind the robot starting position. This
example demonstrates the benefit of non-map information. In (b), if selecting
simply based on number of unknown cells discovered, the agent may select
another frontier, for example the one on the left hand side. However, equipped
with this contextual information the agent selects the frontier that enables
inspection of many non-map features nearby. The success of LIVES onboard a
real robot verify hypothesis (3). Videos of experiments can be found on the
project website https://sites.google.com/view/lives-icra-2024/home.

##  V Conclusions

We present a method for visual search in unseen environments using contextual
LiDAR information to inform planning. Results show LIVES to outperform the
closest of several baseline methods in varied environments by 10-30%. Ablative
studies on the map-free scan classifier are performed to validate the choice
in architecture and training scheme. Furthermore, the proposed approach is
deployed on a real robot in an unseen environment. Experiments certify both
the visual search planner and the learning method for map-free scan
classification in real-world scenarios. Ongoing work aims to extend the LIVES
planner to multiple robots.

## References

  * [1] URL https://github.com/ros-planning/navigation.
  * [2] URL http://wiki.ros.org/costmap_2d.
  * [3] URL https://github.com/hasauino/rrt_exploration.
  * [4] URL https://www.turtlebot.com/turtlebot3/.
  * Baur et al. [2021] Stefan Andreas Baur, David Josef Emmerichs, Frank Moosmann, Peter Pinggera, Björn Ommer, and Andreas Geiger.  Slim: Self-supervised lidar scene flow and motion segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13126–13136, 2021.
  * Best et al. [2022] Graeme Best, Rohit Garg, John Keller, Geoffrey A Hollinger, and Sebastian Scherer.  Resilient multi-sensor exploration of multifarious environments with a team of aerial robots.  In _Robotics: Science and Systems (RSS)_ , 2022.
  * Bircher et al. [2016] Andreas Bircher, Mina Kamel, Kostas Alexis, Helen Oleynikova, and Roland Siegwart.  Receding horizon ”next-best-view” planner for 3d exploration.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1462–1468. IEEE, 2016.
  * Biswas and Veloso [2012] Joydeep Biswas and Manuela Veloso.  Depth camera based indoor mobile robot localization and navigation.  In _2012 IEEE International Conference on Robotics and Automation_ , pages 1697–1702. IEEE, 2012.
  * Biswas and Veloso [2017] Joydeep Biswas and Manuela M Veloso.  Episodic non-markov localization.  _Robotics and Autonomous Systems_ , 87:162–176, 2017.
  * Caley et al. [2019] Jeffrey A Caley, Nicholas RJ Lawrance, and Geoffrey A Hollinger.  Deep learning of structured environments for robot search.  _Autonomous Robots_ , 43:1695–1714, 2019.
  * Cao et al. [2023] Yuhong Cao, Tianxiang Hou, Yizhuo Wang, Xian Yi, and Guillaume Sartoretti.  Ariadne: A reinforcement learning approach using attention-based deep networks for exploration.  _arXiv preprint arXiv:2301.11575_ , 2023.
  * Charrow et al. [2015] Benjamin Charrow, Sikang Liu, Vijay Kumar, and Nathan Michael.  Information-theoretic mapping using cauchy-schwarz quadratic mutual information.  In _2015 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4791–4798. IEEE, 2015.
  * Chen et al. [2022] Xieyuanli Chen, Benedikt Mersch, Lucas Nunes, Rodrigo Marcuzzi, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss.  Automatic labeling to generate training data for online lidar-based moving object segmentation.  _IEEE Robotics and Automation Letters_ , 7(3):6107–6114, 2022.
  * Dai et al. [2020] Anna Dai, Sotiris Papatheodorou, Nils Funk, Dimos Tzoumanikas, and Stefan Leutenegger.  Fast frontier-based information-driven autonomous exploration with an mav.  In _2020 IEEE international conference on robotics and automation (ICRA)_ , pages 9570–9576. IEEE, 2020.
  * Jhaldiyal and Chaudhary [2023] Alok Jhaldiyal and Navendu Chaudhary.  Semantic segmentation of 3d lidar data using deep learning: a review of projection-based methods.  _Applied Intelligence_ , 53(6):6844–6855, 2023\.
  * Jocher [2020] Glenn Jocher.  YOLOv5 by Ultralytics, May 2020.  URL https://github.com/ultralytics/yolov5.
  * Julian et al. [2014] Brian J Julian, Sertac Karaman, and Daniela Rus.  On mutual information-based control of range sensing robots for mapping applications.  _The International Journal of Robotics Research_ , 33(10):1375–1392, 2014.
  * Liu et al. [2022] Minghua Liu, Yin Zhou, Charles R Qi, Boqing Gong, Hao Su, and Dragomir Anguelov.  Less: Label-efficient semantic segmentation for lidar point clouds.  In _European Conference on Computer Vision_ , pages 70–89. Springer, 2022.
  * Lodel et al. [2022] Max Lodel, Bruno Brito, Alvaro Serra-Gómez, Laura Ferranti, Robert Babuška, and Javier Alonso-Mora.  Where to look next: Learning viewpoint recommendations for informative trajectory planning.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 4466–4472. IEEE, 2022.
  * Mersch et al. [2022] Benedikt Mersch, Xieyuanli Chen, Ignacio Vizzo, Lucas Nunes, Jens Behley, and Cyrill Stachniss.  Receding moving object segmentation in 3d lidar data using sparse 4d convolutions.  _IEEE Robotics and Automation Letters_ , 7(3):7503–7510, 2022.
  * Pomerleau et al. [2013] François Pomerleau, Francis Colas, Roland Siegwart, and Stéphane Magnenat.  Comparing icp variants on real-world data sets: Open-source library and experimental protocol.  _Autonomous robots_ , 34:133–148, 2013.
  * Qi et al. [2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 652–660, 2017a.
  * Qi et al. [2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _Advances in neural information processing systems_ , 30, 2017b.
  * Ramakrishnan et al. [2020] Santhosh K Ramakrishnan, Ziad Al-Halah, and Kristen Grauman.  Occupancy anticipation for efficient exploration and navigation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 400–418. Springer, 2020.
  * Saroya et al. [2020] Manish Saroya, Graeme Best, and Geoffrey A Hollinger.  Online exploration of tunnel networks leveraging topological cnn-based world predictions.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6038–6045. IEEE, 2020.
  * Shan and Englot [2018] Tixiao Shan and Brendan Englot.  Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain.  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 4758–4765. IEEE, 2018.
  * Tan et al. [2021] Chee Sheng Tan, Rosmiwati Mohd-Mokhtar, and Mohd Rizal Arshad.  A comprehensive review of coverage path planning in robotics using classical and heuristic algorithms.  _IEEE Access_ , 2021.
  * Tao et al. [2023] Yuezhan Tao, Yuwei Wu, Beiming Li, Fernando Cladera, Alex Zhou, Dinesh Thakur, and Vijay Kumar.  Seer: Safe efficient exploration for aerial robots using learning to predict information gain.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1235–1241. IEEE, 2023.
  * Thomas et al. [2019] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 6411–6420, 2019.
  * Thomas et al. [2021] Hugues Thomas, Ben Agro, Mona Gridseth, Jian Zhang, and Timothy D Barfoot.  Self-supervised learning of lidar segmentation for autonomous indoor navigation.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 14047–14053. IEEE, 2021.
  * Thomas et al. [2022] Hugues Thomas, Matthieu Gallet de Saint Aurin, Jian Zhang, and Timothy D Barfoot.  Learning spatiotemporal occupancy grid maps for lifelong navigation in dynamic scenes.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 484–490. IEEE, 2022.
  * Vidal et al. [2020] Eduard Vidal, Narcís Palomeras, Klemen Istenič, Nuno Gracias, and Marc Carreras.  Multisensor online 3d view planning for autonomous underwater exploration.  _Journal of Field Robotics_ , 37(6):1123–1147, 2020.
  * Yamauchi [1997] Brian Yamauchi.  A frontier-based approach for autonomous exploration.  In _Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA’97.’Towards New Computational Principles for Robotics and Automation’_ , pages 146–151. IEEE, 1997\.
  * Zamorski et al. [2020] Maciej Zamorski, Maciej Zieba, Piotr Klukowski, Rafal Nowak, Karol Kurach, Wojciech Stokowiec, and Tomasz Trzciński.  Adversarial autoencoders for compact representations of 3d point clouds.  _Computer Vision and Image Understanding_ , 193:102921, 2020\.
  * Zheng et al. [2023] Kaiyu Zheng, Anirudha Paul, and Stefanie Tellex.  A system for generalized 3d multi-object search.  _arXiv preprint arXiv:2303.03178_ , 2023.
  * Zhu [2019] Yuke Zhu.  _Closing the Perception-action Loop: Towards General-purpose Robot Autonomy_.  Stanford University, 2019.