# Diffusion Based Augmentation for Captioning and Retrieval in Cultural
Heritage

Dario Cioni1 Lorenzo Berlincioni1 Federico Becattini2 Alberto del Bimbo1
1MICC, University of Florence 2University of Siena
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Cultural heritage applications and advanced machine learning models are
creating a fruitful synergy to provide effective and accessible ways of
interacting with artworks. Smart audio-guides, personalized art-related
content and gamification approaches are just a few examples of how technology
can be exploited to provide additional value to artists or exhibitions.
Nonetheless, from a machine learning point of view, the amount of available
artistic data is often not enough to train effective models. Off-the-shelf
computer vision modules can still be exploited to some extent, yet a severe
domain shift is present between art images and standard natural image datasets
used to train such models. As a result, this can lead to degraded performance.
This paper introduces a novel approach to address the challenges of limited
annotated data and domain shifts in the cultural heritage domain. By
leveraging generative vision-language models, we augment art datasets by
generating diverse variations of artworks conditioned on their captions. This
augmentation strategy enhances dataset diversity, bridging the gap between
natural images and artworks, and improving the alignment of visual cues with
knowledge from general-purpose datasets. The generated variations assist in
training vision and language models with a deeper understanding of artistic
characteristics and that are able to generate better captions with appropriate
jargon.

Figure 1: Schematic illustrating the data augmentation pipeline. The
conditional generative model allows for both and image and text input while in
its Image&Text$\longrightarrow$Image configuration. We provide the model both
the original artwork along with its detailed textual analysis from [47] and
use the diffusion model’s outputs as new datapoints for training other models
for downstream tasks.

##  1 Introduction

Deep learning applications on fine art suffer from an obvious scarcity of
data, due to the fact that an artwork is usually a unique piece. In addition,
tasks involving both vision and language require on the one hand the modeling
of technical language with domain-specific jargon and on the other hand the
understanding of difficult and underrepresented visual concepts, such as
abstract or stylized drawings. These difficulties entail a challenge for
learning algorithms, which would benefit from a large collection of annotated
data. A simple solution is to leverage models pre-trained on general-purpose
datasets to address relevant tasks for cultural heritage, such as retrieval,
visual question answering or captioning. However, the effect is that such
models tend to underperform when applied in the cultural heritage domain. In
fact, when the scope moves from real-world images to paintings and other more
abstract representations, there is a strong domain shift compared to standard
training data, being it composed of natural images. A standard approach to
deal with data scarcity is to leverage data augmentation, slightly perturbing
the training data to improve variability and let the trained model generalize
better. In the vision domain, perturbations usually include adding noise,
altering pixel values or changing the overall orientation or illumination of
the scene. We argue that these augmentations may indeed alter the semantics of
the artworks, where color and spatial distribution of objects can convey
significant meanings that are necessary to interpret the painting.

In this paper, we address the above-mentioned limitations by proposing a data
augmentation strategy for paintings that has the twofold advantage of
increasing the training data as well as preserving the content. In particular,
we explore the benefits of augmenting artwork datasets for image captioning.
To this end, we leverage both textual descriptions of the paintings and a
diffusion model to create several variations of the artworks. By conditioning
the diffusion model on the captions, we generate a variability in the visual
domain that aids the grounding of objects and entities expressed in artistic
form with the technical jargon that describes them. What we propose is
therefore an image augmentation at a semantic level, generating multiple
variations of artworks while retaining their content and style. By leveraging
the expert knowledge of art critics contained in painting descriptions and the
natural language understanding capabilities of state-of-the-art generative
models, we aim to provide a sophisticated augmentation pipeline capable of
generating a sufficient intra-class variability of depicted concepts to enable
an effective learning (Fig. 1).

Our main contributions presented in this paper are

  * •

We propose a data augmentation technique for low-data regime cultural heritage
tasks, that works at a semantic level rather than at a pixel-intensity level
as standard data augmentations in vision.

  * •

Thanks to our data augmentation strategy based on diffusion models we can
favor a visual grounding of linguistic concepts, which in the cultural
heritage domain are often expressed using technical and domain-specific
jargon.

  * •

We show the benefits of the proposed augmentation strategy in captioning tasks
as well as cross-domain retrieval tasks.

##  2 Related Works

##### Computer Vision for Cultural Heritage

In the domain of cultural heritage, several computer vision approaches have
been proposed in the literature. Artwork classification [34, 48, 15, 11, 35]
and recognition [16, 49, 28] have often been placed at the center of such
approaches, sometimes with the end-goal to develop user-engagement
applications [5, 33, 2, 8]. In this paper, we mostly deal with the task of
image captioning, which implies the automatic generation of a natural language
textual description of an image based only on the visual input. This has been
an extensively addressed research topic in recent years [46, 54, 29], but not
many contributions have been made in the domain of art historical data. In
this particular domain, which shifts from the one of natural images, the
complexity of the task increases due to a simultaneous lack of labeled data
and an increased abstraction.

Currently, available painting datasets with descriptions are constructed by
downloading descriptions from online museums or annotating descriptions by
crowdsourcing. The Artpedia [47] dataset is composed of paintings paired with
textual descriptions from WikiPedia. The dataset thus provides information
about artworks and their context and each sentence is categorized as either a
visual sentence or a contextual sentence. Visual sentences describe the visual
content of the painting, while contextual sentences provide information that
cannot be inferred from raw pixels alone. Such information includes, for
instance, the name of the painter, its artistic style, or the museum in which
it is kept. The dataset was originally introduced as a dataset for cross-modal
retrieval as well as captioning and it has been further annotated for visual
question answering purposes in [7]. Similarly, the AQUA dataset [21] has been
proposed to train visual question answering models in the cultural heritage
domain. More recently, ArtCap [32] provided an image captioning dataset
containing 3,606 paintings, each one associated to five textual descriptions,
with a mean length for each caption of 11 words.

A larger example of an artwork dataset is presented in [16] consisting of more
than 80K webly-supervised images from 3120 classes, and a subset of 200
classes with more than 1300 verified images. Text and metadata for each class
is also provided, to support zero-shot learning and other multi-modality
techniques in general. An ontological knowledge base has been exploited in [4]
to create a large-scale cultural heritage dataset, annotated with visual and
contextual data. The authors adopted ArCo [10], the Italian cultural heritage
knowledge graph, to extract information about approximately 500K cultural
assets and leveraged a semi-automatic annotation approach for generating 6.5M
question-answer pairs.

##### Generative Models for Data Augmentation

Data augmentation can be defined as the process through which data can be
transformed without changing its semantics. By using this definition we can
tie the efficacy of an augmentation method to a task and not to the type of
data alone. In most of the computer vision tasks that work with natural
images, the usual augmentation strategies involve flipping the image, adding
random noise, and changing its brightness and colors. When it comes to fine-
art tough, such changes might be detrimental due to the strict relation
between the semantics of the original art piece and its details (i.e. the
relative position of characters in religious art, the use of strong light
contrast in a Caravaggio painting, or the symbolic choice of a particular
color). An attempt to augment training data for object detection in artworks
has been recently proposed [27], where a style transfer model is applied to
natural images to generate images that resemble paintings. A possible approach
to obtain a larger, more diverse, dataset has been explored in many works
outside of the scope of cultural heritage. In these cases, the input data is
used to train a generative model, which in turn will produce new data coming
from the training domain. [44, 3, 26, 25, 6]. Also in [18] the authors used a
CycleGAN [56] for image-to-image translation of thermal to pseudo-RGB data.
The use of these frameworks to perform data augmentation in order to improve
the performance of a separate classifier has been studied in multiple previous
works such as [1] in which they focus on improving one-shot learning, and [9],
where segmentation of medical images is enhanced by GAN augmented data. In
[38] synthetic data coming from a simulator is adapted and used to train an RL
agent for autonomous driving.

Recently Diffusion Models (DM) [24, 41] reached new impressive levels,
compared to GANs, both in terms of output quality and fidelity to the
conditional inputs such as text or additional images and have been employed
for data augmentation objectives such in [51, 22]. Most of these applications
focused mostly on evaluating the ability of diffusion models to generate
synthetic data for classification problems, we instead are going to focus on
different downstream tasks. Latent Diffusion Models (LDM) [41] perform the
diffusion process in a latent space learned by a convolutional auto-encoder.
This allows to greatly reduce the training and inference cost of the model
compared to pixel-based DMs, while maintaining a high visual fidelity. By
introducing cross-attention layers in the diffusion model architecture, the
generation can be conditioned on a wide variety of sources, including text and
images. The popular Stable Diffusion model is based on LDM [41], bringing
further improvements thanks to an internet-scale training.

Motivated by the recent success of large generative models, we posed the
research question regarding whether diffusion models can be used to augment
visual recognition datasets with synthetic images, especially when working in
underrepresented domains such as cultural heritage. Our findings show that
using images generated by a diffusion model, conditioned by a textual
description, leads to improved performance compared to vanilla training as
well as training using standard computer vision data augmentation techniques.

(a)

(b)

Figure 2: Distribution of caption lengths in the Artpedia [47] and ArtCap [32]
datasets

##  3 Data

Experiments were performed on both the Artpedia [47] and the ArtCap [32]
datasets. Albeit similar in structure these two datasets differ in multiple
ways from one another. The [32] dataset contains a fixed set of 5 sentences
per artwork while [47] has on average, 3.1 visual sentences and 6.5 contextual
sentences per artwork. Artpedia contains a collection of 2,930 painting
images, each associated with a variable number of textual descriptions, which
are combined together into a single description. Overall, the dataset contains
a total of 28,212 sentences. Out of these, 9,173 are labeled as visual
sentences and the remaining 19,039 are categorized as contextual. In our work,
we only consider visual sentences since we focus on augmenting images.

The respective syntactic style is also quite different in the two datasets:
where Artpedia chooses paragraph-long academic descriptions, Artcap limits
itself to shorter and simpler captions.

The word count distribution of the captions for the two datasets is shown in
Fig. 2. Note that, on average, a single visual sentence of Artpedia is
composed by 22 words, and the caption is 70 words, which is considerably
longer than most common Image Captioning datasets [12, 43, 45]. We also
evaluated randomly sampling one of the visual sentences, but since each visual
sentence describes only a small portion of the image, it led to worse results.
Both Artpedia and ArtCap provide validation and test splits, composed of 10%
and 10% validation samples, and 10% and 50% test samples, respectively.
Samples from the two datasets are shown in Fig. 3.

The full-length painting depicts model Lise Tréhot posing in a forest. She
wears a white muslin dress and holds a black lace parasol to shade her from
the sunlight, which filters down through the leaves, contrasting her face in
the shadow and her body in the light, highlighting her dress rather than her
face. Lise is a full-length, almost life-size portrait of a young woman,
standing in a forest clearing. She wears a small, pork pie straw hat with red
ribbons, and a long white muslin dress with a long black sash; the dress is
modestly buttoned to the neck and has long sheer sleeves.

The woman is standing in the flower garden wearing a long skirt;
A woman is posing in front of the colorful garden;
A woman in a long skirt in a flower garden on a spring or summer day;
A woman is starring up in the sky in the garden;
A woman is standing in the garden in a white shirt and brown skirt.

Figure 3: Samples of images along with their textual descriptions from
Artpedia (top) and ArtCap (bottom) datasets.

Figure 4: Samples of the augmented images. Left: Original image and its
caption.; Right: Multiple samples of the augmented images using the
combination of the provided description and the original input image.

##  4 Method

In order to generate the augmented version of the datasets we employ a LDM
(Latent Diffusion Model), Stable Diffusion111https://stability.ai/blog/stable-
diffusion-public-release, to generate multiple version of each image belonging
to the original dataset as illustrated in Fig. 1.

In our work, we employed versions 1.4 and 1.5 of Stable Diffusion. In Stable
Diffusion 1.4 the checkpoint was initialized with the weights of the Stable-
Diffusion-v-1-2 checkpoint and subsequently fine-tuned on 225K steps at
resolution 512x512 on the ”laion-aesthetics v2 5+” subset of LAION dataset
[42] and 10% dropping of the text-conditioning to improve classifier-free
guidance sampling. In Stable Diffusion 1.5, the initialization checkpoint and
the finetuning procedure is the same as Stable-Diffusion-1-4, but the
finetuning is performed for more steps (595K). We employed Stable Diffusion
v1.4 to augment Artpedia [47] dataset and Stable Diffusion v1.5 for the ArtCap
[32] dataset.

Given a dataset $\mathcal{D}$ of $N$ samples $(x_{i},\mathbf{y}_{i})$ formed
by an image $x_{i}$ and a set of captions $\mathbf{y}_{i}$, we augment it by
generating a set
$S_{i}=\\{(\tilde{x}_{i1},\mathbf{y})\dots(\tilde{x}_{iM},\mathbf{y})\\}$ of
synthetic variations for each image $x_{i}$, obtaining a synthetic dataset
$\tilde{\mathcal{D}}$ of $N\times M$ samples. Each variation was generated
using both a textual prompt built by providing the caption and the original
image to guide the generation. To obtain different images, the generation seed
was changed for each variation (see Fig. 4).

To gain an intuition of the quality of the synthetic dataset, we calculated an
embedding of the text and images of each sample in $\mathcal{D}$ and
$\tilde{\mathcal{D}}$ using a CLIP-ViT/B-16 model [40]. We can see from Fig. 5
(a) and (b) that the average cosine similarity between images and their
captions maintains a similar value in the original dataset $\mathcal{D}$ and
the synthetic dataset $\tilde{\mathcal{D}}$, suggesting that synthetic images
preserve the relation with the caption. Moreover, in (see Fig. 5 (c)) we see
that variations maintain a high similarity with the original images. This can
also be seen in Fig. 4.

During training, we insert in each position of the training minibatch a sample
$(x_{i},\mathbf{y}_{i})\in\mathcal{D}$ with probability $\alpha$ or one of its
synthetic variations $(x_{ij},\mathbf{y}_{i})\in\tilde{\mathcal{D}}$ with
probability $(1-\alpha)$, where $x_{ij}$ is sampled uniformly from $S_{i}$. We
use a value of $\alpha=0.5$, to balance real and generated images during
training as suggested in [51].

(a)(b)(c)$0.2$$0.4$$0.6$$0.8$$0.33$$0.33$$0.87$$0.28$$0.3$$0.79$Cosine
similarityArtpediaArtCap

Figure 5: Average cosine similarity between CLIP embeddings of: (a) real
images and the associated captions; (b) synthetic images and the associated
captions; (c) real images and their synthetic variations.

##  5 Experiments

In order to test our augmentation technique we perform multiple experiments
over different tasks. As a first downstream task we train an image-captioning
model using both augmented and non-augmented versions of the dataset. For this
set of experiments, we selected medium-sized, Transformer-based Vision and
Language models which can be trained end-to-end and can be employed for a
variety of different tasks. In particular, we use the
$\text{GIT}_{\text{base}}$ [54] model and the $\text{BLIP}_{\text{base}}$ [29]
model.

GIT [54] (Generative Image-to-text Transformer) is a Transformer [52] model
which can be applied to many Vision and Language tasks. It leverages a CLIP
ViT image encoder [40] and a single Transformer text decoder, which are
jointly trained under a single language modeling task on large-scale pre-
training data. It is publicly available in two sizes, GIT-base (129 M
parameters) which employs a CLIP/ViT-B/16 encoder and GIT-large (347M
parameters), with a CLIP/ViT-L/14 encoder. BLIP [29] instead is a model that
effectively uses noisy web data for pre-training by bootstrapping the
captions, generating new synthetic captions and removing the noisy ones. It
employs a multimodal mixture of encoder-decoder which are jointly trained with
three vision-language objectives: image-text contrastive learning, image-text
matching, and image-conditioned language modeling. The architecture is
composed of a ViT [20] encoder to encode images and a BERT [19] to encode
text. Both the GIT and BLIP models were initialized with the available pre-
training weights222https://huggingface.co/microsoft/git-base and finetuned for
10 training epochs using the AdamW [31] optimizer with a $5e^{-05}$ learning
rate and 500 steps of warm-up using batches of 8 images.

The second task is adopt to prove the effectiveness of our proposed strategy
is cross-domain retrieval. Here, we perform retrieval both of images given
their textual description and vice versa. For this downstream application, we
use the CLIP model [40], using the
openCLIP333https://github.com/mlfoundations/open_clip implementation. To
finetune the CLIP model we used again the AdamW optimizer with a learning rate
of $5e^{-04}$.

Dataset | Model | B@1 | B@2 | B@3 | B@4 | METEOR | ROUGE | CIDEr | BERTScore
---|---|---|---|---|---|---|---|---|---
Artpedia |  $\text{GIT}_{b}$ (zero-shot) | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0144 | 0.0749 | 0.0144 | 0.6905
$\text{GIT}_{b}$ w/o DA | 0.0179 | 0.0088 | 0.0046 | 0.0026 | 0.0385 | 0.1433 | 0.0505 | 0.7291
$\text{GIT}_{b}$ w/ DA | 0.0184 | 0.0092 | 0.0048 | 0.0027 | 0.0390 | 0.1479 | 0.0673 | 0.7316
$\text{BLIP}_{b}$ (zero-shot) | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0161 | 0.0830 | 0.0043 | 0.7112
$\text{BLIP}_{b}$ w/o DA | 0.0050 | 0.0026 | 0.0014 | 0.0009 | 0.0331 | 0.1568 | 0.0766 | 0.7262
$\text{BLIP}_{b}$ w/ DA | 0.0118 | 0.0062 | 0.0035 | 0.0020 | 0.0369 | 0.1658 | 0.0906 | 0.7291
ArtCap |  $\text{GIT}_{b}$ (zero-shot) | 0.3993 | 0.2541 | 0.1548 | 0.0888 | 0.1237 | 0.3128 | 0.2114 | 0.7877
$\text{GIT}_{b}$ w/o DA | 0.7311 | 0.5675 | 0.4263 | 0.3196 | 0.2360 | 0.5148 | 0.6263 | 0.8752
$\text{GIT}_{b}$ w/ DA | 0.7475 | 0.5825 | 0.4407 | 0.3321 | 0.2376 | 0.5166 | 0.6445 | 0.8737
$\text{BLIP}_{b}$ (zero-shot) | 0.6224 | 0.4007 | 0.2487 | 0.1512 | 0.1606 | 0.3951 | 0.3467 | 0.8098
$\text{BLIP}_{b}$ w/o DA | 0.7710 | 0.5972 | 0.4515 | 0.3343 | 0.2442 | 0.5128 | 0.6851 | 0.8759
$\text{BLIP}_{b}$ w/ DA | 0.7654 | 0.5909 | 0.4541 | 0.3491 | 0.2466 | 0.5170 | 0.6862 | 0.8748
Table 1: Image Captioning results on Artpedia and ArtCap using GIT [54] model,
measuring BLEU ($n$-grams 1 to 4), METEOR, ROUGE, CIDEr and BERTScore metrics.

###  5.1 Quantitative Results

####  5.1.1 Metrics

To quantitatively assess the quality of the generated captions, standard
language evaluation metrics are used. Those include BLEU [39], ROUGE [30] and
METEOR [17], typically used for machine translation tasks, and CIDEr [53],
specifically developed for the image captioning task. In addition, a semantic
similarity between generated captions and references is measured with
BERTScore [55] metric. BLEU score calculates $n$-gram precisions between a
candidate sentence and a set of human-generated references, multiplied by a
brevity penalty. Single $n$-gram precisions are then combined following a
geometric mean to obtain a final score. It is common practice to report BLEU
scores with $n$-grams ranging from 1 to 4. ROUGE-L calculates a F-measure
using the Longest Common Subsequence (LCS) between a candidate sentence and a
set of references. METEOR computes a harmonic mean of precision and recall
between unigrams of aligned candidate and reference sentences, where the
mapping used for alignment follows various strategies, including exact match,
synonyms and paraphrases. CIDEr measures the consensus among a candidate
sentence and a set of references by computing the cosine similarity of TF-IDF
weighted $n$-gram vectors. BERTScore uses the word embeddings computed by a
pretrained Transformer model to measure the semantic similarity between a
candidate sentence and a reference.

####  5.1.2 Baselines

We compare our method against state-of-the-art augmentation techniques, such
as AutoAugment [13], AugMix [23], RandAugment [14] and TrivialAugment [36].
AutoAugment [13] is an augmentation framework for vision models that casts the
search of parameters for data augmentation as an optimization problem and
solves it using reinforcement learning. RandAugment [14] improves on
AutoAugment [13] by both considerably reducing the parameters search space
from $10^{32}$ to $10^{2}$ and matching or exceeding performances of [13].
AugMix [23] layers multiple randomly sampled augmentation operations in
concert with a consistency loss to improve model robustness. Finally [37]
improves on the previous strategies by further simplifying the search space.
All of the previous models are tailored to image classification tasks, more
recently a number of works focused on data augmentation specifically developed
for detection problems. In Fig. 6 we provide a comparison of the augmentation
operations performed by the aforementioned state of the art techniques and
ours applied to the same image.

No Augmentation |
---|---
AutoAugment |
RandAugment |
AugMix |
TrivialAugment |
Ours |
Figure 6: Sample of images produced by different augmentation methods: No augmentation, AutoAugment [13], RandAugment [14], AugMix [23], and TrivialAugment [36], Ours.  | No aug | AutoAugment | RandAugment | AugMix | TrivialAugment | Ours
---|---|---|---|---|---|---
Artpedia | 0.0505 | 0.0583 | 0.0533 | 0.0536 | 0.0510 | 0.0673
ArtCap | 0.6263 | 0.5829 | 0.6239 | 0.5717 | 0.5849 | 0.6445
Table 2: Comparison of CIDEr scores with GIT [54] model trained using our
proposed diffusion augmentation and other state-of-the-art augmentation
techniques for the image captioning task: AutoAugment [13], RandAugment [14],
AugMix [23], and TrivialAugment [36].

####  5.1.3 Results

##### Image Captioning

We present the results of the image captioning task using differently trained
GIT models in Tab. 1 over the two chosen datasets. For the Artpedia dataset,
the test results show a clear and consistent improvement using our
augmentation technique over all the aforementioned metrics. Similarly, on
ArtCap we report gains in most metrics, with only a slight decrease compared
to standard training with no data augmentation. It is also easy to notice how
the two datasets differ in terms of complexity. In fact, all the models
struggle more with Artpedia [47], obtaining results that are much lower in
absolute terms. This is due to the nature of the captions in Artpedia, which
are composed of long sentences, with lots of details, as can be seen in Fig. 3
and Fig. 4. Therefore, n-gram-based metrics fail to effectively convey the
quality of the captions. On the contrary, BERTSCore, which captures semantic
similarity between sentences rather than analyzing them from a structural
point of view, achieves much higher results and confirms the improved quality
of the captions generated with our data augmentation. On the contrary, ArtCap
has shorter sentences so metrics such as BLEU, METEOR, ROUGE and CIDEr manage
to obtain much higher results in absolute terms and can be used effectively in
this comparison.

In order to assess the quality of our data augmentation strategy we also
present a comparison between our method and different state of the art methods
for image data augmentation (from Section 5.1.2) in Tab. 2. It is important to
note that while our augmentation approach is beneficial to the model, other
augmentation techniques actually hurt performance. Intuitively we can infer
that data augmentation strategies such as the one we compare our method
against are engineered for classification tasks and might not might
semantically invariant with reference to image captioning.

##### Image Retrieval

For the retrieval task we test CLIP [40] using a similar setting to image
captioning. We test first in a zero-shot configuration and then with and
without data augmentation. The CLIP model is pre-trained on the YFCC dataset
[50] when performing zero-shot retrieval. As in the previous task, the results
shown in Tab. 3 present a clear indication of an improved performance in the
retrieval problem. Tab. 4 instead compares our results with the best ones
proposed in [47] using the same experimental protocol by the authors, i.e. by
fixing the maximum number of retrievable items to $N=100$.

Model | Task | R@1 | R@5 | R@10
---|---|---|---|---
CLIP - (zero-shot) | im2t | 0.0853 | 0.1557 | 0.2096
CLIP - w/o DA | im2t | 0.1048 | 0.2081 | 0.2665
CLIP - w/ DA | im2t | 0.1108 | 0.2096 | 0.2740
CLIP - (zero-shot) | t2im | 0.0644 | 0.1751 | 0.2290
CLIP - w/o DA | t2im | 0.0883 | 0.1751 | 0.2305
CLIP - w/ DA | t2im | 0.0868 | 0.1976 | 0.2470
Table 3: Test on Artpedia on the retrieval task with CLIP using a ResNet50 pretrained on YFCC [50]. We report Recall @1,@5,@10. We test both in the image-to-text (im2t) setting and in the text-to-image (t2im) setting. Model | Task | R@1 | R@5
---|---|---|---
X-Attn GloVe [47] | im2t | 0.086 | 0.227
CLIP - w/ DA | im2t | 0.090 | 0.230
X-Attn GloVe [47] | t2im | 0.041 | 0.136
CLIP - w/ DA | t2im | 0.090 | 0.250
Table 4: We compare our results using the same experimental protocol as in
[47] using $N=100$ retrievable items. We report Recall @1 and @5, testing both
in the image-to-text (im2t) setting and in the text-to-image (t2im) setting.

###  5.2 Qualitative Results

Due to the subjective nature of the task, it is necessary to perform a visual
inspection to better understand how are the models behaving under different
data regime conditions. In this section, we present a sample of qualitative
results to better appreciate the effect of our method. The example presented
in Fig. 7 shows the effectiveness of our model in terms of enriching the
quality of the description by comparing the output of the same captioning
method trained under different settings.

While the pre-trained model tends to offer vague but correct descriptions even
in a zero-shot setting, it is necessary to fine-tune the model on the target
dataset in order to match the language used in Artpedia dataset. Our data-
augmented finetuning helps the model to obtain a better representation of fine
visual details in the dataset, allowing to obtain richer captions using the
task-related technical knowledge that a large internet-wide trained model
might be missing.

Figure 7: Qualitative samples showing the original (GT) caption along with
different outputs from GIT [54] on Artpedia [47]

##  6 Conclusions

This paper presented technique for augmenting and better exploit fine art
datasets with the intent of making the fruition of semantically complex visual
art easier to digitalise, to access, and to retrieve for the general public.
In the field of cultural heritage a feature such as the uniqueness of the
artworks can become an obstacle for machine learning techniques that requires
large amount of data. At the same time the usual augmentation techniques such
as image flipping, random brightness change, random hue change do not suit the
task as they semantically change the original datapoint by changing small the
visual details that are actually meaningful. Therefore our contributions aims
at semantically enrich the popular pretrained LLMs models leveraging the
expert knowledge to create a more sophisticated image data augmentation
pipeline.

## Acknowledgements

This work is partially supported by the European Commission under European
Horizon 2020 Programme, Grant No. 101004545-ReInHerit.

## References

  * [1] Antreas Antoniou, Amos Storkey, and Harrison Edwards.  Augmenting image classifiers using data augmentation generative adversarial networks.  In Proc. of Artificial Neural Networks and Machine Learning (ICANN), pages 594–603, 2018.
  * [2] Beatrice Aruanno, Sandro Barone, Francesco Tamburrino, and Mario Covarrubias Rodriguez.  Tintoretto unveiled: Interactive virtual experience for artworks.  In International Joint Conference on Mechanics, Design Engineering & Advanced Manufacturing, pages 1352–1363. Springer, 2022.
  * [3] R. Barth, J. Hemming, and E.J. Van Henten.  Optimising realism of synthetic images using cycle generative adversarial networks for improved part segmentation.  Computers and Electronics in Agriculture, 173:105378, 2020.
  * [4] Federico Becattini, Pietro Bongini, Luana Bulla, Alberto Del Bimbo, Ludovica Marinucci, Misael Mongiovì, and Valentina Presutti.  Viscounth: A large-scale multilingual visual question answering dataset for cultural heritage.  ACM Transactions on Multimedia Computing, Communications and Applications, 2023.
  * [5] Federico Becattini, Andrea Ferracani, Lea Landucci, Daniele Pezzatini, Tiberio Uricchio, and Alberto Del Bimbo.  Imaging novecento. a mobile app for automatic recognition of artworks and transfer of artistic styles.  In Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection: 6th International Conference, EuroMed 2016, Nicosia, Cyprus, October 31–November 5, 2016, Proceedings, Part I 6, pages 781–791. Springer, 2016.
  * [6] Lorenzo Berlincioni, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo.  Multiple future prediction leveraging synthetic trajectories.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6081–6088. IEEE, 2021.
  * [7] Pietro Bongini, Federico Becattini, Andrew D Bagdanov, and Alberto Del Bimbo.  Visual question answering for cultural heritage.  In IOP Conference Series: Materials Science and Engineering, volume 949, page 012074. IOP Publishing, 2020.
  * [8] Pietro Bongini, Federico Becattini, and Alberto Del Bimbo.  Is gpt-3 all you need for visual question answering in cultural heritage?  In European Conference on Computer Vision, pages 268–281. Springer, 2022.
  * [9] Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger N. Gunn, Alexander Hammers, David Alexander Dickie, Maria Valdés Hernández, Joanna Marguerite Wardlaw, and Daniel Rueckert.  Gan augmentation: Augmenting training data using generative adversarial networks.  ArXiv, abs/1810.10863, 2018.
  * [10] Valentina Anita Carriero, Aldo Gangemi, Maria Letizia Mancinelli, Ludovica Marinucci, Andrea Giovanni Nuzzolese, Valentina Presutti, and Chiara Veninata.  Arco: The italian cultural heritage knowledge graph.  In The Semantic Web–ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II 18, pages 36–52. Springer, 2019.
  * [11] Eva Cetinic, Tomislav Lipic, and Sonja Grgic.  Fine-tuning convolutional neural networks for fine art classification.  Expert Systems with Applications, 114:107–118, 2018.
  * [12] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [13] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 113–123, 2019.
  * [14] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020.
  * [15] Riccardo Del Chiaro and et al.  Noisyart: A dataset for webly-supervised artwork recognition.  In VISIGRAPP (4: VISAPP), pages 467–475, 2019.
  * [16] Riccardo Del Chiaro and et al.  Webly-supervised zero-shot learning for artwork instance recognition.  Pattern Recognition Letters, 128:420–426, 2019.
  * [17] Michael Denkowski and Alon Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation, 2014.
  * [18] Chaitanya Devaguptapu, Ninad Akolekar, Manuj M Sharma, and Vineeth N Balasubramanian.  Borrow from anywhere: Pseudo multi-modal object detection in thermal imagery.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W), 2019.
  * [19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [21] Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mitamura.  A dataset and baselines for visual question answering on art.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 92–108. Springer, 2020.
  * [22] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi.  Is synthetic data from generative models ready for image recognition?  arXiv preprint arXiv:2210.07574, 2022.
  * [23] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple data processing method to improve robustness and uncertainty.  arXiv preprint arXiv:1912.02781, 2019.
  * [24] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.
  * [25] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  CyCADA: Cycle-consistent adversarial domain adaptation.  In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1989–1998. PMLR, 10–15 Jul 2018\.
  * [26] Sheng-Wei Huang, Che-Tsung Lin, Shu-Ping Chen, Yen-Yi Wu, Po-Hao Hsu, and Shang-Hong Lai.  Auggan: Cross domain adaptation with gan-based data augmentation.  In Proc. of European Conference on Computer Vision (ECCV), pages 718–731, 2018.
  * [27] Hyeong-Ju Jeon, Soonchul Jung, Yoon-Seok Choi, Jae Woo Kim, and Jin Seo Kim.  Object detection in artworks using data augmentation.  In 2020 International Conference on Information and Communication Technology Convergence (ICTC), pages 1312–1314. IEEE, 2020.
  * [28] Xun Jin and Jongweon Kim.  Artwork identification for 360-degree panoramic images using polyhedron-based rectilinear projection and keypoint shapes.  Applied Sciences, 7(5):528, 2017.
  * [29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * [30] Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  * [31] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2018.
  * [32] Yue Lu, Chao Guo, Xingyuan Dai, and Fei-Yue Wang.  Artcap: A dataset for image captioning of fine art paintings.  IEEE Transactions on Computational Social Systems, pages 1–12, 2022\.
  * [33] Mahshid Majd and Reza Safabakhsh.  Impact of machine learning on improvement of user experience in museums.  In 2017 Artificial Intelligence and Signal Processing Conference (AISP), pages 195–200. IEEE, 2017.
  * [34] Thomas Mensink and Jan Van Gemert.  The rijksmuseum challenge: Museum-centered visual recognition.  In Proceedings of International Conference on Multimedia Retrieval, pages 451–454, 2014.
  * [35] Federico Milani and Piero Fraternali.  A dataset and a convolutional model for iconography classification in paintings.  Journal on Computing and Cultural Heritage (JOCCH), 14(4):1–18, 2021.
  * [36] Samuel G Müller and Frank Hutter.  Trivialaugment: Tuning-free yet state-of-the-art data augmentation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 774–782, 2021.
  * [37] Samuel G. Müller and Frank Hutter.  Trivialaugment: Tuning-free yet state-of-the-art data augmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 774–782, October 2021.
  * [38] Xinlei Pan, Yurong You, Ziyan Wang, and Cewu Lu.  Virtual to real reinforcement learning for autonomous driving.  arXiv preprint arXiv:1704.03952, 2017.
  * [39] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
  * [40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [42] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [43] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [44] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb.  Learning from simulated and unsupervised images through adversarial training.  In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2107–2116, 2017.
  * [45] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.  Textcaps: a dataset for image captioning with reading comprehension.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020.
  * [46] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara.  From show to tell: A survey on deep learning-based image captioning.  IEEE transactions on pattern analysis and machine intelligence, 45(1):539–559, 2022.
  * [47] Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Massimiliano Corsini, and Rita Cucchiara.  Artpedia: A New Visual-Semantic Dataset with Visual and Contextual Sentences.  In Proceedings of the International Conference on Image Analysis and Processing, 2019.
  * [48] Wei Ren Tan, Chee Seng Chan, Hernán E Aguirre, and Kiyoshi Tanaka.  Ceci n’est pas une pipe: A deep convolutional network for fine-art paintings classification.  In 2016 IEEE international conference on image processing (ICIP), pages 3703–3707. IEEE, 2016.
  * [49] Frederik Temmermans, Bart Jansen, Rudi Deklerck, Peter Schelkens, and Jan Cornelis.  The mobile museum guide: artwork recognition with eigenpaintings and surf.  In Proceedings of the 12th International Workshop on Image Analysis for Multimedia Interactive Services, 2011.
  * [50] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 59(2):64–73, 2016.
  * [51] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov.  Effective data augmentation with diffusion models, 2023.
  * [52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [53] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [54] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  arXiv preprint arXiv:2205.14100, 2022.
  * [55] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert.  arXiv preprint arXiv:1904.09675, 2019.
  * [56] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.


# Simple But Effective Redundant Odometry for Autonomous Vehicles

Andrzej Reinke  Xieyuanli Chen  Cyrill Stachniss All authors are with the
University of Bonn, Germany. This work has partially been funded by the
Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under
Germany’s Excellence Strategy, EXC-2070 - 390732324 (PhenoRob) and by the
European Union’s Horizon 2020 research and innovation programme under grant
agreement No 101017008 (Harmony).

###### Abstract

Robust and reliable ego-motion is a key component of most autonomous mobile
systems. Many odometry estimation methods have been developed using different
sensors such as cameras or LiDARs. In this work, we present a resilient
approach that exploits the redundancy of multiple odometry algorithms using a
3D LiDAR scanner and a monocular camera to provide reliable state estimation
for autonomous vehicles. Our system utilizes a stack of odometry algorithms
that run in parallel. It chooses from them the most promising pose estimation
considering sanity checks using dynamic and kinematic constraints of the
vehicle as well as a score computed between the current LiDAR scan and a
locally built point cloud map. In this way, our method can exploit the
advantages of different existing ego-motion estimating approaches. We evaluate
our method on the KITTI Odometry dataset. The experimental results suggest
that our approach is resilient to failure cases and achieves an overall better
performance than individual odometry methods employed by our system.

##  I Introduction

The ability to estimate the ego-motion, also called odometry, is a vital part
of most autonomous mobile systems. Nowadays, autonomous vehicles are typically
equipped with multiple sensors, such as cameras or LiDARs, as different
sensing modalities have individual strengths. Therefore, a large number of
algorithms [35, 14, 15] have been proposed exploiting both, visual and LiDAR
information to explore the advantages of both and compensate for the drawbacks
of the other. In most cases, researchers focus on achieving better odometry
results by designing a potentially complex multi-sensor fusion algorithm. Even
if different pose estimation systems’ performances are remarkable within
specific datasets, such algorithms may not always operate reliably under all
conditions, setups, and application domains, as we will illustrate below in
Sec. IV-C. Thus, failure detection and recovery steps are useful in real-world
applications. We investigate in this paper a strategy that runs multiple
odometry approaches in parallel and then selects the method that appears to be
the best one. Thus, newly proposed individual methods can easily be added to
our approach.

Few studies are investigating redundant odometry methods running in parallel
on autonomous mobile systems. One example is the work by Luo et al. [18], who
propose to estimate odometry as an average of parallel odometry pipelines.
This approach, however, is vulnerable since already one gross estimation error
of one method will affect the overall performance. Santamaria-Navarro et al.
[27] propose another solution, which runs several methods in parallel. It
first sticks to one odometry method and only falls back to an alternative
odometry pipeline if the initial method’s divergence is detected. While this
method is straightforward to realize, it does not utilize to its full extent
the potential of all algorithms that run in parallel.

Figure 1: Our redundant odometry system employs multiple odometry algorithms
to provide proposals shown as differently colored dots for the each timestamp,
exploiting LiDAR and camera data. Our approach uses sanity checks and a
Chamfer distance-based score to choose the most promising pose estimation from
all candidates.

In this work, we investigate the possibility of building a redundant odometry
system with sanity checks, and that can deal with failure cases of different
odometry methods. As sketched in Fig. 1, instead of sticking to a specific
odometry method, the idea is to run various odometry algorithms in parallel
and choose the most promising one. We target to avoid explicit recovery
behaviors for individual odometry pipelines in case of failures but propose to
switch dynamically between estimation algorithms and independently of previous
decisions. Furthermore, we target a computationally light-weight and flexible
system.

The main contribution of this paper is a redundant odometry system to provide
a more resilient ego-motion estimation than the individual odometry methods
that the system consist of. Instead of sticking to one specific odometry
method, the proposed system runs multiple odometry algorithms in parallel and
exploits LiDAR and camera data. It combines sanity checks exploiting dynamic
and kinematic constraints of the vehicle and Chamfer distance-based scores
between sensor data and a local map to select the most promising pose
estimation among all the odometry candidates. We make three key claims: Our
approach is able to (i) use redundant odometry pipelines to provide better
odometry results, (ii) generally yields the best results compared to the
individual methods used, and, (iii) avoids several failure cases in different
situations. The paper and our experimental evaluation back up these claims.
Furthermore, the source code of our approach is available at:
https://github.com/PRBonn/MutiverseOdometry

##  II Related Work

Odometry estimation is a widely investigated topic in robotics. It can be
achieved using different types of sensors, including wheels [32], inertial
sensing [13], cameras [28], thermal cameras [9], radars [7], LiDARs [26],
sonars [1] as well as various combinations [11, 34, 35, 8, 3, 20]. Although
much work has been done, only a small amount of research has been devoted to
making a robust and redundant system to deal with failures of individual ego-
motion estimation approaches.

Works show that redundancy can make a robotic system more robust. Rollinson et
al. [25] show a robust estimation of the configuration of an articulated robot
that uses a large number of redundant proprioceptive sensors (encoders, gyros,
accelerometers) embedded in an unscented Kalman filter. They formulate the
state estimation problem to leverages the redundancy in the proprioceptive
information provided by the robot’s joint angle encoders and inertial sensors.
Additionally, they introduce a novel outlier detector that can identify
corrupted measurements by utilizing the Mahalanobis distance matrix. Kim et
al. [16] point out that redundant systems are important due to adversary
attacks that corrupt measurement. They propose an observer for a nonlinear
system to detect sensor attacks.

The importance of a redundant odometry system has also been shown on Mars
Exploration Rovers [19]. It successfully demonstrated a switching behavior
between the wheel, inertia, and visual odometry, where visual odometry was
used as compensation for any unforeseen slip encountered during a drive.
Another redundant odometry system was proposed by Luo et al. [18] taking an
average estimate over all odometry estimates that are currently in the active
mode. Others use fallback solutions and execute homing behaviors [24],
sampling-based motion proposals in a particle filter [30], or combinations of
dense and feature-based methods [33]. Carnevale et al. [6] propose a way to
fuse the information coming from two typologies of redundant radio-frequency
identification sensors with complementary characteristics using a Kalman
filter framework. A switching criterion is embedded in the filtering approach
where the observer uses uncertainties of the measurements and their average
values to provide a switching behavior. Recently, Ebadi et al. [10] created an
autonomous mapping and positioning system for the DARPA Subterranean
Challenge. This work uses visual-inertial, LiDAR, and wheeled odometry as a
front-end for a multi-robot mapping and positioning system to explore
perceptually-degraded environments. They use the generalized iterative closest
point (GICP) algorithm [29] to estimate scan-to-scan and scan-to-map matching
that provides the relative transformation between the consecutive timestamps.

The related approach to our work has recently been proposed by Santamaria-
Navarro et al. [27] in the context of UAVs. Their system called heterogeneous
redundant odometry (HeRO) evaluates the pose estimations from each pipeline by
checking gaps and jumps within the transforms sequence. In this method, the
different odometries run in parallel, and the potentially best estimation is
used as the output of the system until the uncertainty of the estimation
exceeds certain quality criteria. Then, the system falls back to another
method while at the same time trying to reinitialize the former method. Unlike
HeRO, which treats each odometry pipeline as a separate entity with a
separated history of transformations and transformed point clouds, our method
does not need any reinitialization procedure. It switches to an alternative
method or fallbacks to a constant velocity model if all other methods fail. We
focus on picking the best transform independently from the previous transforms
and use a chain of transformation and transformed point clouds as a common
history. HeRO assesses the best odometry based on the internal state of the
odometry algorithms, while our method aims at finding a more objective metric
that can choose fairly the best odometry at every timestamp. Based on HeRO,
Palieri et al. [22] propose a health monitor system for the odometry
estimation, which exploits LiDAR-based scan-to-scan and scan-to-map methods.
While this method is similar to our approach, we do not need to manually
specify the main odometry or submental ones. Our method chooses the most
promising odometry by each candidate’s performance scores at every timestamp
without relying on any prior assumptions or knowledge about the odometry
performance.

Different from all the methods mentioned above, we propose an integrated
visual-LiDAR odometry system that uses multiple odometry algorithms in
parallel, a set of dynamics and kinematic sanity checks that filter out
potential estimation failures, and a point cloud Chamfer distance-based
criterion, which chooses the most promising pose estimation at each timestamp,
independently from the history.

##  III Our Approach

Figure 2: System overview. Both 3D LiDAR and camera sensor data are used as
the input. We first exploit multiple existing odometry algorithms to process
the input data and provide transformation proposals. Those proposals will then
be checked by sanity checks based on dynamics and kinematics capabilities.
Only those transformations which are labeled as success cases will be
evaluated in the next step. The point-to-point Chamfer distance is then
calculated to score those success cases. As the result, the best
transformation in terms of the calculated scores will be chosen as the final
estimation. The final transformation will be saved to the odometry history and
employed as an initial guess for the next iteration. Our system is flexible,
every submodule could be extended by extra data, methods, or other criteria.

We propose a redundant odometry system enhanced with sanity checks. These
steps are visualized in Fig. 2. We use the modular design to keep our system
flexible. There are four modules in our system. First, the input preprocessing
module exploiting multiple sensing modalities as the input to the system (see
Sec. III-A). Second, the odometry candidates module employing multiple
different odometry algorithms to estimate the candidate transformations at
each timestamp as odometry proposals (see Sec. III-B). Third, the sanity-check
module using vehicle dynamics and kinematics properties to check for failure
cases in each odometry proposal (see Sec. III-C). Fourth, the scoring module
exploiting the Chamfer distance to score the odometry proposals and selects
the final transformation from the most recent vehicle motion (see Sec. III-D).

###  III-A Input Preprocessing Module

In this work, we do not focus on choosing the best sensor or combination to
improve odometry but provide a flexible system that allows us to use multiple
sensing modalities and ego-motion algorithms in parallel to improve the
robustness of the odometry. To this end, our system uses the data from a 3D
LiDAR scanner and a monocular camera as an example to show that our system can
exploit the redundancy of multiple sensing modalities to improve its
robustness. Specifically, we use SIFT features [31] extract from images
obtained by a monocular camera and voxelized point clouds from LiDAR scans as
the input to different odometry methods. Such a redundancy design at the input
level has the advantage that our approach is still functional even in case of
sensor failures.

###  III-B Stack of Odometry Pipelines

Our approach combines multiple odometry methods into a redundant system, which
is robust to failure cases and provides better odometry results. The odometry
algorithms used in this paper are generalized iterative closest point (GICP)
[29], point-to-plane iterative closest point (P2P-ICP) [26], normal
distributions transform (NDT) [4], Color iterative closest point (ColorICP)
[23], and Huang’s method [14] for combining LiDAR and camera data. Moreover,
we also employ the common constant velocity model (CVM), which directly uses
the last timestamp estimation without using the sensor data. Our approach is
not limited only to those methods and can be easily extended.

P2P-ICP [26, 36] minimizes the sum of the squared distance between the point
from the source points and the tangent plane at its corresponding point in the
target point cloud for every point.

GICP [29, 17] combines the Iterative Closest Point (ICP) and point-to-plane
ICP into a single probabilistic framework. It models the locally planar
surface structure from both scans and minimizes the objective function in a
typical ICP-based framework. This procedure may be interpreted as a plane-to-
plane approach embedded in the ICP framework.

ColorICP [23, 36] optimizes jointly geometrical information (in the sense of
point-to-plane ICP) together with RGB color information that is assigned to
the LiDAR points which are inside the field of view of the camera(s).

NDT [4, 17] uses a representation of the target point cloud to obtain a smooth
surface described as a set of local probability density functions in the
subdivided space as a grid of cells. A density function is computed for each
cell based on the point distribution within the cell. To estimate the pose,
NDT maximizes the likelihood that the current scan points lay on the reference
scan surface.

Huang’s method [14] employs the 5-point algorithm [21] within a RANSAC loop
that finds 5 of the 6 transformation parameters (rotation and translation
vector up to scale) between the previous camera image and the current image
with the SIFT descriptor. It then uses a grid search-based 1D ICP to estimate
the scale.

Besides the odometry information provided by different algorithms based on
different sensing modalities, our system also employs the so-called constant
velocity model (CVM), which takes the same transformation from the previous
timestamp as an additional odometry estimation with the assumption that the
acceleration of the car is zero. Although the constant velocity model is
simple, it will always pass the sanity checks and make the system robust.

Those odometry methods are run in parallel using the same input data and share
the same odometry history. This means only the transformation that is finally
selected by our approach at each timestamp will be stored and serves as the
initial guess for all the odometry methods in the next timestamp.

In this paper, we exploit six different odometry methods to exploit the
redundancy of multiple odometries to improve overall performance and
robustness. Note that we can easily add more methods if desired. Since our
method can switch dynamically for each timestamp independently from previous
decisions, multiple odometry methods can be easily integrated into our system,
even during online processing.

###  III-C Sanity Checks

In robotics, we typically know the used hardware and the physical model of our
robot or a typical car. Therefore, we use the dynamic and kinematic
constraints of a car to check the odometry proposals provided by different
algorithms, which we refer to as sanity checks. In this paper, we exploit two
types of sanity checks, one based on the dynamic constraint, and the other
based on the kinematic constraint.

We refer to the maximum acceleration $a_{max}$ a typical car can reach for
formulating the dynamic constraint. Our system first checks the acceleration
resulting from all odometry proposals. If the estimated transformation exceeds
the bound of the maximum possible acceleration, it will be rejected from
further considerations.

For the kinematic constraint, we use the Ackermann vehicle model [37], which
puts constraints on the kinematic possibilities of the car movement. If using
a differential drive model, one obviously must use the corresponding model. As
shown in Fig. 3, given the forward velocity $v$ and the steering angle rate
$\delta$, the car’s trajectory follows the Ackermann model if there is no slip
or any other highly dynamic conditions such as drifting or similar. It means
that in real operations, the car’s sideward movements are limited to a certain
range.

Applying this fact, our system uses the last pose estimation to calculate the
forward velocity. Based on the forward velocity and the Ackermann model, the
side velocity $v_{\text{s}}$ is calculated based on the derivation [5],

$\displaystyle v_{\text{s}}(\dot{\beta},v)=$ $\displaystyle
f\frac{\frac{v}{f}+l(1-\cos\frac{\dot{\beta}}{f})}{\sin\frac{\dot{\beta}}{f}}(1-\cos\frac{\dot{\beta}}{f})+l\sin\frac{\dot{\beta}}{f},$
(1)

where $v$ is a forward velocity of the vehicle, and $\dot{\beta}$ is the
turning rate, $f=\frac{1}{\Delta\tau}$, where $\Delta\tau$ is the difference
between timestamps of the current frame and previous frame, $l$ is the
distance from the rear axis to the placement of the sensor. A proposal will
then be rejected for future considerations if

$\mathinner{\\!\left\lvert v_{s}-v^{*}\right\rvert}>v_{th},$ (2)

where $v^{*}$ is taken from the odometry proposals and $v_{th}$ is a maximum
deviation from the side velocity that is acceptable.

Note that even if all sensors or odometry algorithms based on sensor data fail
to pass the sanity checks, the constant velocity model will always pass the
sanity checks and makes the proposed method fairly robust to failures.

Figure 3: Ackermann vehicle model. The variable $l$ is the distance between
the front and rear axis, $v$ is the velocity of the rear axis center, $\delta$
is the steering angle, $P$ is the instantaneous center of rotation, and $r$ is
the radius around $P$. See [37] for details.

###  III-D Scoring Candidate Transformations

All the odometry proposals, which pass the sanity checks, will be evaluated
based on the Chamfer distance metric [2] computed on the point cloud. Thus we
need at least one sensor that generates point cloud data. We use a 3D LiDAR in
our system. The metric on the point cloud is defined as follow,

$D_{\text{chamfer}}(M,N)=\frac{1}{M}\sum_{m\in M}d_{N}(m),$ (3)

where $M$ is a set of points in the source point cloud, $N$ is a set of points
in the target point cloud, and $d_{N}$ is the minimum distance between point
$m$ and its nearest neighbor point in the target point cloud $N$ given the
transformation between point cloud $M$ and $N$. In addition to the fact that
point clouds after transformation should close to each other, we apply a small
search radius $r_{s}$ for correspondence matching and delete the points for
which no correspondence could be found within the radius. The reason to use
the Chamfer distance as the criterion is that it is fast to calculate and
independent from all the baseline odometry methods.

Since each odometry candidate in our system estimates the transforms from the
same source point cloud to the same target point cloud, this metric allows us
to assess the candidate transformations estimated by different odometry
methods. Our system applies this metric to all the success odometry proposals,
and the lowest score is chosen as the final odometry result.

$T^{*}=\underset{T\in{T_{1},T_{2},...}}{\arg\min}\frac{1}{M}\sum_{m\in
M}d_{N}(m),$ (4)

The chosen transformation is stored in the odometry history and employed as
the initial guess for the next iteration. We use the last $N_{\text{map}}$
scans with the stored odometry to build a local point cloud map.

|  |  |  |
---|---|---|---|---
00 | 01 | 02 | 03 | 04
|  |  |  |
05 | 06 | 07 | 08 | 09
Figure 4: Trajectories of different methods evaluated on the KITTI Dataset.
For the sake of clarity, we only show the results of our method together with
the best and the worst results of the baseline methods employed by our system.

##  IV Experimental Evaluation

We present our experiments to showcase the capabilities of our method and to
support our key claims that our approach is able to (i) use redundant odometry
pipelines to provide better odometry results, (ii) generally yields the best
results compared to the individual methods used, and, (iii) avoids several
failure cases in different situations. For evaluation purposes, we use the
KITTI Odometry Benchmark [12] which provides point clouds generated by a
Velodyne HDL-64E 3D laser scanner ($10$ Hz, 64 laser beams, range: 100m) and
color images from PointGrey Flea2 video cameras ($10$ Hz, resolution:
$1382\times 512$ pixels). We tested our approach on an Intel Core i9-9900K CPU
with 16 cores @ 3.60GHz with 32 GB RAM. For all the experiments, we use
$l=1.0\,m$, $a_{\text{max}}=6.0\,m/s^{2}$ and $v_{th}=0.8\,m/s$, the search
radius $r_{s}=0.5\,m$, and the number of scans used to build the local map is
$N_{\text{map}}=10$.

###  IV-A Odometry Performance

Figure 5: Frequency of choosing different baseline methods in KITTI sequence
00. The upper figure shows the picking times of different methods, and the
lower figure shows the choice of each frame.

The experiments presented in this section are designed to show our approach’s
odometry performance and support the claim that using our redundant system
yields better overall odometry results.

To evaluate the odometry results, we use the KITTI odometry dataset sequences
00-10 and the KITTI odometry metric [12], which considers relative errors
concerning translation and rotation averaged over different distances.

For the qualitative results, we show the estimated trajectories of our method
compared to those generated by the individual odometry algorithms employed by
our redundant system, see in Fig. 4. For the sake of clarity, we only show the
trajectories generated by our method as well as those of the best and the
worst individual odometry. We can see that, in all sequences, our method
always tracks the ground truth poses well no matter how bad the worst
individual odometry candidate is, and there is no individual odometry baseline
that can always generate good results.

For the quantitative results, we show all evaluation results on all sequences
in Tab. I. In terms of translation error, our method is clearly superior in 10
out of 11 sequences to all other methods individually and in the other one
sequence achieves a comparable performance to the best individual odometry
algorithm. Overall, our method outperforms all baselines with the smallest
average translation error of $1.11\,\%$. In terms of rotation error, our
method achieves the second-best average rotation error of
$0.0045\,\text{deg}/\text{m}$.

To further insight into how our system chooses different proposals, we
illustrate when and how often different methods have been selected on sequence
00 in Fig. 5. We can see that our method switches between different proposals
frequently according to the sanity checks and scoring results.

TABLE I: Results on KITTI Odometry

| Sequence |
---|---|---
Approach | 00 | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10 | Average
GICP | 1.28/0.61 | 3.22/0.91 | 1.78/0.63 | 1.18/0.77 | 1.51/0.93 | 1.18/0.60 | 0.87/0.53 | 0.87/0.57 | 1.32/0.56 | 1.55/0.65 | 1.65/0.69 | 1.49/0.62
Huang’s | 2.40/0.33 | 6.09/0.21 | 1.50/0.25 | 1.38/0.18 | 0.98/0.22 | 2.06/0.42 | 1.43/0.25 | 6.86/4.04 | 2.71/0.29 | 1.37/0.22 | 2.37/0.24 | 2.35/0.38
P2P-ICP | 1.57/0.77 | 5.22/1.56 | 1.58/0.58 | 1.98/1.13 | 1.05/0.56 | 1.32/0.76 | 1.12/0.76 | 0.95/0.62 | 1.37/0.76 | 1.48/0.73 | 1.48/0.70 | 1.64/0.75
ColorICP | 3.09/1.60 | 26.08/1.68 | 3.17/1.45 | 2.93/1.40 | 3.11/2.00 | 1.74/0.99 | 0.99/0.63 | 1.60/1.51 | 3.14/1.65 | 2.44/1.33 | 4.04/1.66 | 3.89/1.44
NDT | 2.88/1.66 |  2.42/0.67 | 3.51/1.58 | 2.89/1.89 | 1.63/1.39 | 2.58/1.33 | 1.27/0.67 | 3.38/2.52 | 2.76/1.43 | 2.64/1.16 | 2.91/1.60 | 2.88/1.46
Our method |  0.96/0.43 | 2.53/0.67 |  1.40/0.50 |  1.05/0.67 |  0.97/0.61 |  0.72/0.35 |  0.66/0.36 |  0.62/0.45 |  0.99/0.39 |  0.99/0.44 |  1.30/0.54 |  1.11/0.45
Relative errors averaged over trajectories of $100$ to $800$ṁ length: relative
translational error in $\%$ / relative rotational error in $\mathrm{degrees}$
per $100$ṁ.
Bold numbers indicate best performance in terms of translational error.

###  IV-B Ablation Study on Odometry Candidates

TABLE II: Ablation study on input odometry methods.

P2P-ICP | NDT | ColorICP | Huang’s | GICP | Average
---|---|---|---|---|---
✓ | ✓ |  |  |  | 1.23/0.54
✓ | ✓ | ✓ |  |  | 1.21/0.52
✓ | ✓ | ✓ | ✓ |  | 1.17/0.50
✓ | ✓ | ✓ | ✓ | ✓ | 1.11/0.45
Here we use the same KITTI metric as used in Tab. I.
We show the average results of different setups evaluated with all sequences.

The second experiment shows the ablation study on different odometry
combinations. We use the same KITTI odometry dataset sequence 00-10 and the
same metric as in the first experiment to evaluate the odometry results of
different odometry input combinations. As shown in Tab. II, we can see that no
matter what combination of the odometry candidate methods is provided, our
system can always outperforms the individual odometry methods (see results in
Tab. I for the individual numbers per sequence).

###  IV-C Robustness

The third experiment illustrates our system’s robustness and supports the
claim that the proposed redundant odometry system is robust to different
failure cases. We use four different cases to show that our system can
successfully cover the failure cases where the individual candidate odometry
methods fail, see Fig. 6. The failure cases refer to the situations where the
odometry yields a clearly wrong estimate with respect to the ground truth
poses. The failure cases can be caused by the dynamic objects, the lack of
features of the environments, or the wrong data associations between different
observations. In Fig. 6, we can see that in all cases when the individual
entities fail, our system switches in between the odometry candidates at the
right time, choosing the correct estimate and provide proper odometry results.

(a) Huang’s method failure.

(b) NDT failure.

(c) ColorICP failure.

(d) Point-to-point ICP failure.

Figure 6: Robustness evaluation of our system. Here we show failure cases of
different methods employed by our system (Huang’s method, NDT, ColorICP and
Point-to-plane ICP respectively). We see that in all failure cases our system
work properly.

(a) Acc without sanity checks

(b) Acc with sanity checks

(c) Forward velocity comparison.

(d) Trajectory comparison.

Figure 7: Analysis of sanity checks for our system on dataset 01. (a) shows
the noise of acceleration estimation. (b) shows the results after our approach
filter out the noise using sanity checks. (c) shows that with sanity checks
our method can generate smoother velocity and provide also a better trajectory
shown in (d).

To gain further insight, we provide an analysis of the sanity checks. If we do
not apply sanity checks, our KITTI 00-10 overall score drops from $1.11\,\%$
to $1.37\,\%$ in terms of the average translation error. The results of the
acceleration sanity checks on KITTI sequence 01 is shown in Fig. 7. We can see
in Fig. 7(a) that there are some large accelerations that indicate the
failures of the odometry estimations. After using sanity checks, see Fig.
7(b), our system provides smoother velocity estimates, see Fig. 7(c), and
provide better overall results, see Fig. 7(d).

##  V Conclusion

In this paper, we presented a robust and resilient odometry system. Our method
is simple, effective, easy to implement, and we provide the source code for
further use. Our system exploits the redundancy of running multiple odometry
pipelines, including multiple streams, in parallel. It uses the dynamic and
kinematic constraints-based sanity checks and the Chamfer distance-based
criteria to avoid failure cases and select the most promising odometry
estimation among all proposals. For that, at least one sensor that generates
point clouds, such as a LiDAR, is required. We evaluated our approach on the
KITTI dataset and supported all claims made in this paper. The experiments
suggest that the proposed system is resilient and robust to odometry failures
and achieves better performance than all the baselines in terms of odometry
accuracy. Despite these encouraging results, there is space for further
improvements. In future work, we plan to use learning-based approaches to
design better selection criteria for choosing the most promising odometry
estimate.

## References

  * [1] A. Bahr, J.J. Leonard, and M.F. Fallon.  Cooperative localization for autonomous underwater vehicles.  Intl. Journal of Robotics Research (IJRR), 28(6):714–728, 2009\.
  * [2] H.G. Barrow, J.M. Tenenbaum, R.C. Bolles, and H.C. Wolf.  Parametric correspondence and chamfer matching: Two new techniques for image matching.  In Proc. of the Intl. Conf. on Artificial Intelligence (IJCAI), pages 659–663, 1977.
  * [3] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall.  SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences.  In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV), 2019.
  * [4] P. Bieber and W. Strasser.  The normal distributions transform: A new approach to laser scan matching.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), volume 3, pages 2743 – 2748, 11 2003.
  * [5] M. Buczko, V. Willert, J. Schwehr, and J. Adamy.  Self-validation for automotive visual odometry.  In Proc. of the IEEE Intelligent Vehicles Symposium (IV), pages 1–6, 2018.
  * [6] D. Carnevale and F. Martinelli.  State estimation for robots with complementary redundant sensors.  Intl. Journal of Advanced Robotic Systems, 12(10):138, 2015.
  * [7] S.H. Cen and P. Newman.  Precise ego-motion estimation with millimeter-wave radar under diverse and challenging conditions.  In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), 2018.
  * [8] X. Chen, A. Milioto, E. Palazzolo, P. Giguère, J. Behley, and C. Stachniss.  SuMa++: Efficient LiDAR-based Semantic SLAM.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2019.
  * [9] J. Delaune, R. Hewitt, L. Lytle, C. Sorice, R. Thakker, and L. Matthies.  Thermal-inertial odometry for autonomous flight throughout the night.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pages 1122–1128, 2019.
  * [10] K. Ebadi, Y. Chang, M. , A. Stephens, A. Hatteland, E. Heiden, A. Thakur, N. Funabiki, B. Morrell, S. Wood, L. Carlone, and A.A. Agha-mohammadi.  Lamp: Large-scale autonomous mapping and positioning for exploration of perceptually-degraded subterranean environments.  In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), pages 80–86, 2020.
  * [11] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun.  Vision meets robotics: The kitti dataset.  Intl. Journal of Robotics Research (IJRR), 32(11):1231–1237, 2013\.
  * [12] A. Geiger, P. Lenz, and R. Urtasun.  Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite.  In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3354–3361, 2012.
  * [13] P. Groves.  Navigation using inertial sensors [tutorial].  IEEE Aerospace and Electronic Systems Magazine, 30:42–69, 02 2015\.
  * [14] K. Huang and C. Stachniss.  Joint Ego-motion Estimation Using a Laser Scanner and a Monocular Camera Through Relative Orientation Estimation and 1-DoF ICP.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2018.
  * [15] K. Huang and C. Stachniss.  Accurate Direct Visual-Laser Odometry with Explicit Occlusion Handling and Plane Detection.  In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), 2019.
  * [16] J. Kim, C. Lee, H. Shim, Y. Eun, and J.H. Seo.  Detection of sensor attack and resilient state estimation for uniformly observable nonlinear systems having redundant sensors.  IEEE Trans. on Automatic Control, 64(3):1162–1169, 2019.
  * [17] K. Koide, M. Yokozuka, S. Oishi, and A. Banno.  Voxelized gicp for fast and accurate 3d point cloud registration.  EasyChair Preprint no. 2703, EasyChair, 2020.
  * [18] R.C. Luo, C.C. Yih, and K.L. Su.  Multisensor fusion and integration: approaches, applications, and future research directions.  IEEE Sensors Journal, 2(2):107–119, 2002.
  * [19] M. Maimone, Y. Cheng, and L. Matthies.  Two years of visual odometry on the mars exploration rovers.  Journal of Field Robotics (JFR), 24(3):169–186, 2007.
  * [20] A. Milioto, I. Vizzo, J. Behley, and C. Stachniss.  RangeNet++: Fast and Accurate LiDAR Semantic Segmentation.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), 2019.
  * [21] D. Nistér.  An efficient solution to the five-point relative pose problem.  IEEE Trans. on Pattern Analalysis and Machine Intelligence (TPAMI), 26(6):756–770, 2004.
  * [22] M. Palieri, B. Morrell, A. Thakur, K. Ebadi, J. Nash, A. Chatterjee, C. Kanellakis, L. Carlone, C. Guaragnella, and A. Agha-mohammadi.  Locus: A multi-sensor lidar-centric solution for high-precision odometry and 3d mapping in real-time.  IEEE Robotics and Automation Letters (RA-L), 2020.
  * [23] J. Park, Q. Zhou, and V. Koltun.  Colored Point Cloud Registration Revisited.  In Proc. of the IEEE Intl. Conf. on Computer Vision (ICCV), 2017\.
  * [24] D. Perea-Ström, I. Bogoslavskyi, and C. Stachniss.  Robust Exploration and Homing for Autonomous Robots.  In Journal on Robotics and Autonomous Systems (RAS), 2016.
  * [25] D. Rollinson, H. Choset, and S. Tully.  Robust state estimation with redundant proprioceptive sensors.  In Proc. of ASME Dynamic Systems and Control Conference (DSCC), October 2013.
  * [26] S. Rusinkiewicz and M. Levoy.  Efficient variants of the ICP algorithm.  In Proc. of the Intl. Conf. on 3-D Digital Imaging and Modeling, pages 145–152, 2001.
  * [27] A. Santamaria-Navarro, R. Thakker, D.D. Fan, B. Morrell, and A. Agha-Mohammadi.  Towards resilient autonomous navigation of drones.  arXiv preprint, abs/2008.09679, 2020.
  * [28] D. Scaramuzza and F. Fraundorfer.  Visual odometry [tutorial].  IEEE Robotics and Automation Magazine (RAM), 18(4):80–92, 2011\.
  * [29] A. Segal, D. Haehnel, and S. Thrun.  Generalized-ICP.  In Proc. of Robotics: Science and Systems (RSS), 2009.
  * [30] C. Stachniss, G. Grisetti, N. Roy, and W. Burgard.  Evaluation of Gaussian Proposal Distributions for Mapping with Rao-Blackwellized Particle Filters.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), San Diego, CA, USA, 2007.
  * [31] J. Tardif, Y. Pavlidis, and K. Daniilidis.  Monocular visual odometry in urban environments using an omnidirectional camera.  In Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pages 2531–2538, 2008.
  * [32] S. Thrun, W. Burgard, and D. Fox.  Probabilistic Robotics.  MIT Press, 2005.
  * [33] K.M. Wurm, C. Stachniss, and G. Grisetti.  Bridging the Gap Between Feature- and Grid-based SLAM.  Journal on Robotics and Autonomous Systems (RAS), 58(2):140 – 148, 2010.
  * [34] J. Zhang and S. Singh.  LOAM: Lidar Odometry and Mapping in Real-time.  In Proc. of Robotics: Science and Systems (RSS), 2014.
  * [35] J. Zhang and S. Singh.  Visual-lidar odometry and mapping: Low-drift, robust, and fast.  In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), pages 2174–2181, 2015.
  * [36] Q.Y. Zhou, J. Park, and V. Koltun.  Open3D: A modern library for 3D data processing.  arXiv:1801.09847, 2018.
  * [37] W. Zong and Q. Chen.  A robust method for detecting parking areas in both indoor and outdoor environments.  IEEE Sensors Journal, 18(6):1903, 2018.


the conformal partial wave literature. For example, $f_{t}(z)$ is decomposed
along the $\Psi^{(0)}$ conformal partial waves.

$\Psi_{\Delta,J}^{(0)}(z)=(-1)^{J}K_{1-\Delta,J}\,k_{\Delta}(z)+(-1)^{J}K_{\Delta,J}\,k_{1-\Delta}(z)\,,\qquad
z\in(0,1)\,,$ (157)

and for the principal discrete series

$\Psi_{\Delta,J}^{(0)}(z)=(-1)^{J}K_{1-\Delta,J}\,k_{\Delta}(z)\,,\qquad
z\in(0,1)\,.$ (158)

We can write the conformal partial wave in the other two regions in terms of
$\Psi_{\Delta,J}^{(0)}$ as

$\displaystyle\Psi_{\Delta,J}^{(-)}(z)$
$\displaystyle=(-1)^{J}\Psi_{\Delta,J}^{(0)}\left(\frac{z}{z-1}\right)\,,\qquad$
$\displaystyle z\in(-\infty,0)\,,$ (159)
$\displaystyle\Psi_{\Delta,J}^{(+)}(z)$
$\displaystyle=\frac{1}{2}\left[\Psi_{\Delta,J}^{(0)}(z+i\epsilon)+\Psi_{\Delta,J}^{(0)}(z-i\epsilon)\right]\,,\qquad$
$\displaystyle z\in(1,\infty)\,.$ (160)

It can be helpful to know that this last piece evaluates to

$\Psi_{\Delta,J}^{(+)}(z)=\frac{2^{J+1}}{\sqrt{\pi}}\left(\frac{z-2}{z}\right)^{\delta_{1J}}\Gamma\left(\tfrac{1-\Delta+J}{2}\right)\Gamma\left(\tfrac{\Delta+J}{2}\right)\,{}_{2}F_{1}\left(\tfrac{1-\Delta+J}{2},\tfrac{\Delta+J}{2};\tfrac{1}{2}+J;\tfrac{(2-z)^{2}}{z^{2}}\right)\,.$
(161)

For further technical details see results in Maldacena_2016 ; Bulycheva_2017 ;
Simmons_Duffin_2018 ; Mazac:2018qmi ; Hogervorst:2021uvp .

## References

  * (1) S. Pasterski, S.-H. Shao, and A. Strominger, Flat space amplitudes and conformal symmetry of the celestial sphere, Physical Review D 96 (9, 2017) [arXiv:1701.00049].
  * (2) S. Pasterski and S.-H. Shao, Conformal basis for flat space amplitudes, Phys. Rev. D96 (2017), no. 6 065022, [arXiv:1705.01027].
  * (3) C. Cheung, A. de la Fuente, and R. Sundrum, 4d scattering amplitudes and asymptotic symmetries from 2d cft, Journal of High Energy Physics 2017 (1, 2017) [arXiv:1609.00732].
  * (4) A. Strominger, Lectures on the Infrared Structure of Gravity and Gauge Theory, arXiv:1703.05448.
  * (5) A.-M. Raclariu, Lectures on Celestial Holography, arXiv:2107.02075.
  * (6) S. Pasterski, Lectures on Celestial Amplitudes, arXiv:2108.04801.
  * (7) S. Pasterski, M. Pate, and A.-M. Raclariu, Celestial Holography, in 2022 Snowmass Summer Study, 11, 2021.  arXiv:2111.11392.
  * (8) H. T. Lam and S.-H. Shao, Conformal Basis, Optical Theorem, and the Bulk Point Singularity, Phys. Rev. D98 (2018), no. 2 025020, [arXiv:1711.06138].
  * (9) D. Nandan, A. Schreiber, A. Volovich, and M. Zlotnikov, Celestial Amplitudes: Conformal Partial Waves and Soft Limits, JHEP 10 (2019) 018, [arXiv:1904.10940].
  * (10) C.-M. Chang, Y.-t. Huang, Z.-X. Huang, and W. Li, Bulk locality from the celestial amplitude, arXiv:2106.11948.
  * (11) W. Melton, Celestial Feynman Rules for Scalars, arXiv:2109.07462.
  * (12) W. Fan, A. Fotopoulos, S. Stieberger, T. R. Taylor, and B. Zhu, Conformal blocks from celestial gluon amplitudes, JHEP 05 (2021) 170, [arXiv:2103.04420].
  * (13) W. Fan, A. Fotopoulos, S. Stieberger, T. R. Taylor, and B. Zhu, Conformal Blocks from Celestial Gluon Amplitudes II: Single-valued Correlators, arXiv:2108.10337.
  * (14) A. Atanasov, W. Melton, A.-M. Raclariu, and A. Strominger, Conformal Block Expansion in Celestial CFT, arXiv:2104.13432.
  * (15) A. Guevara, Celestial OPE blocks, arXiv:2108.12706.
  * (16) S. Stieberger and T. R. Taylor, Symmetries of celestial amplitudes, Physics Letters B 793 (6, 2019) 141–143, [arXiv:1812.01080].
  * (17) Y. T. A. Law and M. Zlotnikov, Poincaré constraints on celestial amplitudes, JHEP 03 (2020) 085, [arXiv:1910.04356]. [Erratum: JHEP 04, 202 (2020)].
  * (18) M. Pate, A.-M. Raclariu, A. Strominger, and E. Y. Yuan, Celestial Operator Products of Gluons and Gravitons, arXiv:1910.07424.
  * (19) A. Fotopoulos and T. R. Taylor, Primary Fields in Celestial CFT, JHEP 10 (2019) 167, [arXiv:1906.10149].
  * (20) A. Fotopoulos, S. Stieberger, T. R. Taylor, and B. Zhu, Extended BMS Algebra of Celestial CFT, JHEP 03 (2020) 130, [arXiv:1912.10973].
  * (21) A. Fotopoulos, S. Stieberger, T. R. Taylor, and B. Zhu, Extended Super BMS Algebra of Celestial CFT, JHEP 09 (2020) 198, [arXiv:2007.03785].
  * (22) W. Fan, A. Fotopoulos, and T. R. Taylor, Soft Limits of Yang-Mills Amplitudes and Conformal Correlators, JHEP 05 (2019) 121, [arXiv:1903.01676].
  * (23) N. Arkani-Hamed, M. Pate, A.-M. Raclariu, and A. Strominger, Celestial Amplitudes from UV to IR, arXiv:2012.04208.
  * (24) S.-H. Shao, Celestial Free Fields, . https://www.youtube.com/watch?v=IlmiP8rsS2A&t=750s.
  * (25) M. Veltman, Unitarity and causality in a renormalizable field theory with unstable particles, Physica 29 (1963), no. 3 186–207.
  * (26) A. Denner and J.-N. Lang, The Complex-Mass Scheme and Unitarity in perturbative Quantum Field Theory, Eur. Phys. J. C 75 (2015), no. 8 377, [arXiv:1406.6280].
  * (27) J. F. Donoghue and G. Menezes, Unitarity, stability and loops of unstable ghosts, Phys. Rev. D 100 (2019), no. 10 105006, [arXiv:1908.02416].
  * (28) G. Menezes, Generalized unitarity method for unstable particles, arXiv:2111.11570.
  * (29) H. S. Hannesdottir and S. Mizera, What is the $i\varepsilon$ for the S-matrix?, arXiv:2204.02988.
  * (30) S. R. Coleman, R. Jackiw, and H. Politzer, Spontaneous Symmetry Breaking in the O(N) Model for Large N*, Phys. Rev. D 10 (1974) 2491.
  * (31) M. Moshe and J. Zinn-Justin, Quantum Field Theory in the Large N Limit: a review, Physics Reports 385 (Oct., 2003) 69–228. arXiv: hep-th/0306133.
  * (32) D. Carmi, L. Di Pietro, and S. Komatsu, A Study of Quantum Field Theories in AdS at Finite Coupling, JHEP 01 (2019) 200, [arXiv:1810.04185].
  * (33) S. L. Adler, Consistency conditions on the strong interactions implied by a partially conserved axial vector current, Phys. Rev. 137 (1965) B1022–B1033. [140(1964)].
  * (34) J. de Boer and S. N. Solodukhin, A Holographic reduction of Minkowski space-time, Nucl. Phys. B 665 (2003) 545–593, [hep-th/0303006].
  * (35) P. A. M. Dirac, Wave equations in conformal space, Annals Math. 37 (1936) 429–442.
  * (36) G. Mack and A. Salam, Finite component field representations of the conformal group, Annals Phys. 53 (1969) 174–202.
  * (37) S. Weinberg, Six-dimensional Methods for Four-dimensional Conformal Field Theories, Phys. Rev. D 82 (2010) 045031, [arXiv:1006.3480].
  * (38) M. S. Costa, J. Penedones, D. Poland, and S. Rychkov, Spinning Conformal Correlators, JHEP 11 (2011) 071, [arXiv:1107.3554].
  * (39) M. S. Costa, J. Penedones, D. Poland, and S. Rychkov, Spinning Conformal Blocks, JHEP 11 (2011) 154, [arXiv:1109.6321].
  * (40) D. Simmons-Duffin, Projectors, shadows, and conformal blocks, Journal of High Energy Physics 2014 (4, 2014) [arXiv:1204.3894].
  * (41) Y. T. A. Law and M. Zlotnikov, Massive Spinning Bosons on the Celestial Sphere, JHEP 06 (2020) 079, [arXiv:2004.04309].
  * (42) Y. T. A. Law and M. Zlotnikov, Relativistic partial waves for celestial amplitudes, JHEP 11 (2020) 149, [arXiv:2008.02331].
  * (43) S. Mizera and S. Pasterski, Celestial Geometry, arXiv:2204.02505.
  * (44) H. Bateman, Tables of integral transforms [volumes I & II], vol. 1.  McGraw-Hill Book Company, 1954.  https://resolver.caltech.edu/CaltechAUTHORS:20140123-101456353.
  * (45) D. Mazáč, A Crossing-Symmetric OPE Inversion Formula, JHEP 06 (2019) 082, [arXiv:1812.02254].
  * (46) S. Pasterski, A. Puhm, and E. Trevisani, Revisiting the Conformally Soft Sector with Celestial Diamonds, arXiv:2105.09792.
  * (47) L. Donnay, S. Pasterski, and A. Puhm, Goldilocks Modes and the Three Scattering Bases, arXiv:2202.11127.
  * (48) H. Elvang and Y.-t. Huang, Scattering Amplitudes, arXiv:1308.1697.
  * (49) D. Nandan, A. Schreiber, A. Volovich, and M. Zlotnikov, Celestial amplitudes: conformal partial waves and soft limits, Journal of High Energy Physics 2019 (10, 2019) [arXiv:1904.10940].
  * (50) M. Hogervorst and B. C. van Rees, Crossing symmetry in alpha space, JHEP 11 (2017) 193, [arXiv:1702.08471].
  * (51) W. Bailey, Generalized Hypergeometric Series.  Cambridge tracts in mathematics and mathematical physics. The University Press, 1935.
  * (52) J. Maldacena and D. Stanford, Remarks on the sachdev-ye-kitaev model, Physical Review D 94 (Nov, 2016) [arXiv:1604.07818].
  * (53) D. J. Gross and A. Neveu, Dynamical Symmetry Breaking in Asymptotically Free Field Theories, Phys. Rev. D 10 (1974) 3235.
  * (54) S. Pasterski and H. Verlinde, Mapping SYK to the Sky, arXiv:2201.05054.
  * (55) A. Kar, L. Lamprou, C. Marteau, and F. Rosso, A Celestial Matrix Model, arXiv:2205.02240.
  * (56) S. Jain, M. Mandlik, S. Minwalla, T. Takimi, S. R. Wadia, and S. Yokoyama, Unitarity, Crossing Symmetry and Duality of the S-matrix in large N Chern-Simons theories with fundamental matter, JHEP 04 (2015) 129, [arXiv:1404.6373].
  * (57) S. Caron-Huot, Analyticity in spin in conformal theories, Journal of High Energy Physics 2017 (Sep, 2017) [arXiv:1703.00278].
  * (58) D. Simmons-Duffin, D. Stanford, and E. Witten, A spacetime derivation of the lorentzian ope inversion formula, Journal of High Energy Physics 2018 (Jul, 2018) [arXiv:1711.03816].
  * (59) S. Rychkov, EPFL Lectures on Conformal Field Theory in D$>$= 3 Dimensions.  SpringerBriefs in Physics. Springer, 1, 2016.
  * (60) D. Simmons-Duffin, Tasi lectures on the conformal bootstrap, arXiv:1602.07982.
  * (61) J. Qiao and S. Rychkov, A tauberian theorem for the conformal bootstrap, JHEP 12 (2017) 119, [arXiv:1709.00008].
  * (62) A. Gadde, In search of conformal theories, arXiv:1702.07362.
  * (63) M. Hogervorst, J. a. Penedones, and K. S. Vaziri, Towards the non-perturbative cosmological bootstrap, arXiv:2107.13871.
  * (64) K. Bulycheva, A note on the syk model with complex fermions, Journal of High Energy Physics 2017 (Dec, 2017) [arXiv:1706.07411].
  * (65) R. Rattazzi, V. S. Rychkov, E. Tonni, and A. Vichi, Bounding scalar operator dimensions in 4D CFT, JHEP 12 (2008) 031, [arXiv:0807.0004].
  * (66) S. El-Showk, M. F. Paulos, D. Poland, S. Rychkov, D. Simmons-Duffin, and A. Vichi, Solving the 3D Ising Model with the Conformal Bootstrap, Phys. Rev. D 86 (2012) 025022, [arXiv:1203.6064].
  * (67) F. Kos, D. Poland, and D. Simmons-Duffin, Bootstrapping the $O(N)$ vector models, JHEP 06 (2014) 091, [arXiv:1307.6856].
  * (68) D. Poland, S. Rychkov, and A. Vichi, The Conformal Bootstrap: Theory, Numerical Techniques, and Applications, Rev. Mod. Phys. 91 (2019) 015002, [arXiv:1805.04405].
  * (69) P. Ferrero, K. Ghosh, A. Sinha, and A. Zahed, Crossing symmetry, transcendentality and the Regge behaviour of 1d CFTs, JHEP 07 (2020) 170, [arXiv:1911.12388].
  * (70) Y.-C. He, J. Rong, and N. Su, Non-Wilson-Fisher kinks of $O(N)$ numerical bootstrap: from the deconfined phase transition to a putative new family of CFTs, SciPost Phys. 10 (2021) 115, [arXiv:2005.04250].
  * (71) F. Dolan and H. Osborn, Conformal partial waves and the operator product expansion, Nucl. Phys. B 678 (2004) 491–507, [hep-th/0309180].
  * (72) F. Dolan and H. Osborn, Conformal Partial Waves: Further Mathematical Results, arXiv:1108.6194.