# Off the Radar: Uncertainty-Aware Radar Place Recognition with Introspective
Querying and Map Maintenance

Jianhao Yuan, Paul Newman, Matthew Gadd
Mobile Robotics Group, Department of Engineering Science, University of
Oxford, UK
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Localisation with Frequency-Modulated Continuous-Wave (FMCW) radar has gained
increasing interest due to its inherent resistance to challenging
environments. However, complex artefacts of the radar measurement process
require appropriate uncertainty estimation – to ensure the safe and reliable
application of this promising sensor modality. In this work, we propose a
multi-session map management system which constructs the “best” maps for
further localisation based on learned variance properties in an embedding
space. Using the same variance properties, we also propose a new way to
introspectively reject localisation queries that are likely to be incorrect.
For this, we apply robust noise-aware metric learning, which both leverages
the short-timescale variability of radar data along a driven path (for data
augmentation) and predicts the downstream uncertainty in metric-space-based
place recognition. We prove the effectiveness of our method over extensive
cross-validated tests of the Oxford Radar RobotCar and MulRan dataset. In
this, we outperform the current state-of-the-art in radar place recognition
and other uncertainty-aware methods when using only single nearest-neighbour
queries. We also show consistent performance increases when rejecting queries
based on uncertainty over a difficult test environment, which we did not
observe for a competing uncertainty-aware place recognition system.

###### Index Terms:

Radar, Place Recognition, Deep Learning, Uncertainty Estimation, Autonomous
Vehicles, Robotics

Figure 1:  Overview of system pipeline. In the offline stage, we use a
variational contrastive learning framework (see Sec. III-A for full details)
to learn a hidden embedding space with estimated uncertainty such that radar
scans from similar topological locations are close to each other, and vice
versa. In the online stage, we develop two uncertainty-based mechanisms to
handle sequentially collected radar scans for inference and map construction.
For repetitive traversal of the same routes, we actively maintain an
integrated map dictionary by replacing highly uncertain scans with more
certain ones. For query scans with low uncertainty, we retrieve the matching
map scans from the dictionary based on metric space distance. In contrast, we
reject prediction for high-uncertainty scans.

##  I Introduction

Place recognition and localisation are important tasks in the field of
robotics and autonomous systems, as they enable a system to understand and
navigate its environment. Traditional vision-based methods for place
recognition are often vulnerable to changes in environmental conditions such
as lighting, weather, and occlusion, leading to performance degradation [1].
To address this issue, there has been increasing interest in the use of FMCW
radar as a robust sensor substitute for such adversarial environments.
Existing works have demonstrated the effectiveness of FMCW radar place
recognition with hand-crafted [2, 3, 4, 5, 6] and learning-based feature
extraction approaches [7, 8, 9, 10, 11, 12, 13, 14, 15]. Despite the success
of existing works, the deployment of these methods in safety-critical
applications such as autonomous driving is still limited by the lack of
calibrated uncertainty estimation. While there have been works [16, 17, 18,
19] that address the problem in similar areas such as image retrieval or
visual place recognition, which leverage both Bayesian and learning-based
approaches, there is no previous work specifically targeting uncertainty
estimation for radar place recognition.

In this area, it is important to consider: (1) safety requires uncertainty
estimation to be well-calibrated to false positive rate in order to enable
introspective rejection; (2) real-time deployment requires fast single-scan
uncertainty-based inference capability; (3) repetitive route traversal in
long-term autonomy requires online continual map maintenance.

While the Variational Autoencoder (VAE) [20] is usually used for generative
tasks, its probabilistic latent space can serve as an effective metric space
representation for place recognition [21, 22, 23] and allow prior assumption
on the data noise distribution, which also gives a normalized aleatoric
uncertainty estimation.

Thus, in this paper, to achieve reliable and safe deployment of FMCW radar in
autonomous driving, we leverage a variational contrastive learning framework
and propose a unified uncertainty-aware radar place recognition method as
shown in Fig. 1. Our key contributions are as follows:

  1. 1.

Uncertainty-aware metric learning framework for radar place recognition.

  2. 2.

Introspective query mechanism based on false positive calibrated uncertainty
estimation for real-time autonomous driving deployment.

  3. 3.

Online recursive map maintenance and improvement mechanism for repeated
traversals of changing environments in long-term autonomy.

In doing so we outperform the previous radar place recognition state-of-the-
art [8] and show that our learned uncertainty is more suitable for query
rejection than a previous approach [17] developed for vision but tested here
in radar.

##  II Related Work

###  II-A Radar Place Recognition

Place recognition can be viewed as a query and retrieval process, where a map
is constructed with a dictionary of radar scans. For every new query scan, the
algorithm needs to retrieve a scan from the dictionary such that samples from
a similar topological location are close to each other, and vice versa.
Traditional hand-crafted feature extraction methods, such as correlative scan
matching [2], graph matching [3, 4], and scan context descriptor [6], have
demonstrated the effectiveness of using radar perception for place recognition
and localisation. Recently, learning-based methods have shown impressive
performance. Supervised metric learning methods [7, 9, 12, 10, 11, 13], which
exploit rotational invariance and spatial-temporal consistency in radar scans,
have demonstrated remarkable performance. Then, Gadd et al [8] achieved
comparable performance to supervised methods using contrastive learning.
Moreover, multi-modal methods incorporating additional modalities, such as
LIDAR points, through direct joint learning [15] or point-cloud registration
[24], are also explored to aid radar-based recognition. Among numerous metric
learning methods, the VAE structure is extensively explored. It uses
variational inference to approximate the posterior distribution of latent
representation and thus gives a probabilistic estimation on embedding with
consideration of data noise. Lin et al [25] use VAE in metric learning for the
visual classification task. Burnett et al [26] use a VAE-like structure with
factor graph optimisation for radar odometry.

Inspired by previous works of the variational metric learning approach of Lin
et al [25] and the contrastive learning approach of Gadd et al [8], we form
the basis of our representation learning framework in a contrastive manner
with a VAE-like structure.

###  II-B Uncertainty Estimation

Uncertainty estimation aims to quantify the model prediction confidence, which
is a crucial component in the safety-critical application for the model to
introspectively reject low confidence predictions. One line of work in similar
areas, such as image retrieval and place recognition, uses the Bayesian
approach such as Monte Carlo Dropout [16] to estimate the epistemic
uncertainty, which, however, usually is very computationally expensive and
hinders the deployment in real-time autonomous driving. Another line of work
directly learns the uncertainty from data to estimate the aleatoric
uncertainty. However, previous approaches such as PTL [19], BPE [18], and TAH
[27] are restricted to specific pairwise or triplet loss, which lacks
flexibility in offline system design. Later STUN [17] uses a knowledge
distillation framework to break such constraints that adapt to arbitrary
metric learning loss functions. Moreover, STUN [17] use a dynamic binning
strategy that requires a large batch of test samples to determine a meaningful
uncertainty rejection threshold at inference time. Here, only the relative
uncertainty within a batch can be exploited. However, this is not suitable for
real-time place recognition deployment as the entire test sequence is not
known a priori. Thus, to tackle this problem, we develop a static binning
strategy by leveraging the VAE variance prediction, such that the uncertainty
estimated is compared against a pre-defined prior distribution.

###  II-C Uncertainty-aware Radar Localisation

Although uncertainty estimation in radar place recognition tasks is still
largely under-explored, there have been attempts to develop introspective
radar systems for odometry. Adolfsson et al [28] present a radar system that
verifies loop closure candidates introspectively by combining multiple place-
recognition techniques, including tightly-coupled place similarity and
odometry uncertainty search, creating loop descriptors from origin-shifted
scans, and delaying loop selection until after verification. In contrast, our
system does not rely on a secondary odometry estimation system, as uncertainty
is inherent to the representation we learn for place recognition. Aldera et al
[29] focus only on odometry, and use inertial sensors to learn an uncertainty
classifier, which we do not.

##  III Method

In this section, we present the proposed modules for the variational
uncertainty-aware radar place recognition method including metric space
learning (Sec. III-A), mapping (Sec. III-B), and querying (Sec. III-C). We
explain how these modules are integrated to form a unified system.

Figure 2:  Overview of Variational Contrastive Learning Framework, based on
[25]. A metric space is learnt through an encoder-decoder structure with two
reparameterised parts: a deterministic embedding for recognition and a set of
parameter modelling multivariate Gaussian distribution with its variance
serving as the uncertainty measure. The overall learning is jointly driven by
both reconstruction and contrastive loss to ensure an informative and
discriminative hidden representation for radar scans.

###  III-A Variational Contrastive Learning

The work in this section is both a key enabler of our core contributions, map
maintenance and introspective querying – see below, but is also a novel
integration of deep variational metric learning (DVML) [25] with radar place
recognition [8, 14] and a new way to characterise uncertainty in place
recognition. Indeed, we show in Sec. V that learning uncertainty in this way
yields more calibrated introspection than in self-teaching uncertainty
estimation (STUN) [17].

As illustrated in Fig. 2, we adopt a VAE structure to disentangle the radar
scan embedding into a noise-induced variant part $Z_{V}$, which captures the
variance of prediction-irrelevant uncertainty sources, and a semantic
invariant part $Z_{I}$, for the essential features of the scene
representation. The variant part is later sampled from a prior multivariate
isotropic Gaussian distribution and added to the invariance part to form the
overall representation $Z=Z_{I}+Z_{V}$. The variant output is directly used as
the uncertainty measure.

The assumption is we only consider the aleatoric uncertainty in model
prediction caused by the inherent ambiguity and randomness in data as the main
source of uncertainty [30]. In particular for radar scanning, this is likely
due to speckle noise, saturation and temporary occlusion. The standard metric
learning approaches, regardless of the loss function chosen, tend to enforce
identical embeddings between positive sample pairs while ignoring the
underlying variances between them. However, this can lead to the model being
insensitive to minor features and overfitting to the training distribution.
Thus, to model the noise variance, we use the extra probabilistic variance
output in the VAE structure to estimate the aleatoric uncertainty.

To establish such a noise-aware representation for radar perception, we use
four loss functions to guide the overall training. We choose to demonstrate
the benefit of our specific method of learning uncertainty in the setting of
losses which feature increased numbers of negative examples – this having been
proven in STUN [17] to perform best (i.e. quadruplet, with $2$ negatives). In
this area, the state-of-the-art in radar place recognition [8] already uses a
loss with many (i.e. more than $2$) negative samples – so, we extend upon this
one. For a fair comparison, we compare the uncertainty mechanism of STUN with
ours under the same loss regime – see further experimental details in Sec.
IV-B.

(1) Invariant contrastive loss on the deterministic representation $Z_{I}$ to
disentangle the task-independent noise from radar semantics such that the
invariant embedding contains sufficient causal information; and

(2) Variant contrastive loss between on the overall representation $Z$ to
establish a meaningful metric space. Both contrastive losses take the
following form in Eq. 1.

$L_{Con}=-\sum_{i}^{m}\log
P\left(i\mid\hat{\mathbf{x}}_{i}\right)-\sum_{i}^{m}\sum_{j\neq
i}^{m}\log\left(1-P\left(i\mid\mathbf{x}_{j}\right)\right)$ (1)

where a batch consists of $m$ samples
$\left\\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{m}\right\\}$ and
synthetically rotated temporal proximal frames augmentations
$\left\\{\hat{\mathbf{x}}_{1},\hat{\mathbf{x}}_{2},\ldots,\hat{\mathbf{x}}_{m}\right\\}$
using a “spinning” strategy [8], which is simply rotation augmentation, for
rotation invariance. We aim to maximise the probability that an augmented
sample $\hat{x}_{i}$ is recognised as the original instance $x_{i}$ while
minimising the probability of the reversed case. The probability of
recognition is approximated as Eq. 2.

$P\left(i\mid\mathbf{x}_{j}\right)=\frac{\exp\left(\frac{\mathbf{Z}_{i}^{T}\mathbf{Z}_{j}}{\tau}\right)}{\sum_{k=1}^{m}\exp\left(\frac{\mathbf{Z}_{k}^{T}\mathbf{Z}_{j}}{\tau}\right)},j\neq
i$ (2)

where the embeddings $\mathbf{z}$ are either $Z_{I}$ or $Z$ as in 1) and 2).

(3) Kullback–Leibler (KL) divergence between the learned Gaussian distribution
and a standard isotropic multivariate Gaussian, which is our prior assumption
on the data noise. This ensures an identical distribution across all the
sample noises and provides a static reference for the absolute value of
variant output.

$D_{\mathrm{KL}}=\sum_{z\in\mathcal{Z_{V}}}\mathcal{N}\left(\boldsymbol{\mu},\boldsymbol{\Sigma}\right)\log\left(\frac{\mathcal{N}\left(\boldsymbol{\mu},\boldsymbol{\Sigma}\right)}{\mathcal{N}\left(\boldsymbol{0},\boldsymbol{I}\right)}\right)$
(3)

(4) Reconstruction loss between the extract feature map $M$ and decoder output
$M_{R}$, which forces the overall representation $Z$ to contain sufficient
information in the original radar scans for reconstruction. However, instead
of pixel-level radar scan reconstruction, we only reconstruct a lower
dimensional feature map to reduce the computational cost in the decoding
process.

$L_{Rec}=\left\lVert M_{R}-M\right\rVert_{2}$ (4)

While the vanilla VAE structure driven by only KL divergence and
reconstruction loss also provides latent variance, it is considered unreliable
for uncertainty estimation [31] due to its well-known problem of posterior
collapse [32] and vanishing variance [33]. We confirm this experimentally in
Sec. V. Such ineffectiveness is mainly due to the imbalance of two losses
during the training process: when the KL divergence dominates, the latent
space posterior is forced to equal the prior whereas when the reconstruction
loss dominates, the latent variance is pushed to zero. In our method, however,
we achieve more stable training by introducing the variant contrastive losses
as an extra regulariser, where the variance is driven to keep a robust
boundary among clustering centres in metric space. Hence, we obtain a more
reliable latent space variance reflecting the underlying aleatoric uncertainty
of radar perception. We show that this VAE-like uncertainty mechanism
alongside a metric learning objective is more well-calibrated to place
recognition than STUN [17], vanilla VAEs [20], MC Dropout [16], etc, which we
prove experimentally in Sec. V.

###  III-B Continual Map Maintenance

Continual Map maintenance is an important function of the online system since
we aim to fully exploit the scanning data obtained during a lifetime of
autonomous vehicle operation and improve the map in a recursive manner. The
process of merging a new radar scan into the parent map, consisting of scans
from previous traversals, is illustrated in Algorithm 1. Each radar scan is
represented by both a hidden representation $\boldsymbol{\mu}$ and an
uncertainty measure $U$. In the merging process, we search for matched
positive scans for each new scan with topological distance under a threshold
$D_{t}$. If the new scan has a lower uncertainty then it will be integrated
into the parent map and replace the matched scan, otherwise, it will be
discarded.

Algorithm 1 Map Maintenance

0: Parent Map:
$M=\\{\left(\boldsymbol{\mu}_{i},\boldsymbol{U}_{i}\right)|i\in[0,n]\\}$, Map
GPS $\\{D_{i}\\}|i\in[0,N]$ , New Scan:
$\left(\boldsymbol{\mu}_{j},\boldsymbol{U}_{j}\right)$, Scan GPS $D_{j}$.

0: Updated Parent Map:
$\underline{M}=\\{\left(\boldsymbol{\mu}_{i},\boldsymbol{U}_{i}\right)|i\in[0,n]\\}$
Find Matched Scan Index:

1: $\\{\Delta D_{i}|i\in[0,n]\\}$ $\leftarrow$ Sort$(\lVert
D_{i}-D_{j}\rVert)$

2: if $\Delta D_{i}<D_{t}$ then

3: $\\{U_{1},U_{2},...U_{m}\\}$ $\leftarrow$ GetUncertainty$(M,\Delta D_{i})$

4: end ifUncertainty Comparison and Replacement:

5: for $i=0$ to $m$ do

6: if ($U_{j}>U_{i}$) then

7:
$(\boldsymbol{\mu}_{i},\boldsymbol{U}_{i})\leftarrow(\boldsymbol{\mu}_{j},\boldsymbol{U}_{j})$

8: end if

9: end for

10: return $\underline{M}$

As shown in Fig. 3, The replacement of higher uncertainty scans will remove
noisy scans, which potentially lead to incorrect prediction, such as
saturation and range uncertainty.

Figure 3: Illustration of Map Maintenance. The red and green nodes each
represent radar scans with higher and lower uncertainty respectively. We
always maintain a parent map as the localisation reference consisting of only
scans with the lowest uncertainty at each location.

By iteratively performing the maintenance process for all obtained scans, we
can gradually enhance the quality of the integrated parent map. Thus, the
maintenance algorithm can serve as an effective online deployment strategy as
it continually exploits multiple experiences of the same route traversal to
boost recognition performance and, in the meanwhile, preserve a constant
parent map size resulting in budgeted computation and storage cost.

In order to facilitate this map maintenance, we currently use GPS, but it is
important to note that this step is purely offline (place recognition is
performed entirely online, with radar alone) and that this offline step may be
readily be replaced by whatever underlying map representation the place
recognition queries are issued against, i.e. one built by radar-only mapping
and motion estimation [3, 12, 28, 34]. It is also important to note that our
results in Sec. V show that even without this map maintenance, our
uncertainty-aware representation of Sec. III-A outperforms previous methods,
but that this method is capable of significantly boosting localisation
performance by being careful about the learned uncertainty of map contents
over many repeat traversals.

###  III-C Introspective Query Rejection

During the inference, we use the Euclidean distance between the query and
scanned map in a metric space as a measure of similarity. The top $k$ pairs
with the highest similarity serve as the prediction result. To further enhance
recognition performance and achieve introspective prediction, we also develop
an uncertainty rejection mechanism.

Static Thresholding Since the aleatoric uncertainty is measured against a
standard Gaussian distribution
$\mathcal{N}\left(\boldsymbol{0},\boldsymbol{I}\right)$, the estimated
variances in all dimensions are close to 1. Thus, we can use two
hyperparameters $\Delta$ and $N$ to fully define the scale and resolution of
uncertainty rejection respectively. The resultant thresholds $T$ are defined
as follows:

$T=\\{(1-\Delta)+n\times\frac{2\Delta}{N}|n\in[0,N]\\}$ (5)

Given a scan with $m$ dimension latent variance $\Sigma$, we average across
all the dimensions and obtain a scalar uncertainty measure
$U=\frac{1}{m}\sum_{i}^{m}\Sigma_{i}$.

Prediction Rejection At inference time, we perform introspective query
rejection, where the query scans with a higher variance than a defined
threshold will be rejected from recognition. Existing methods, such as STUN
and MC Dropout, dynamically divide the uncertainty range of batch samples into
threshold levels. However, this imposes the requirement of multiple samples
during inference and can result in unstable rejection performance,
particularly when a small number of samples are available. In contrast, our
static thresholding strategy offers sample-independent threshold levels and
provides consistent single-scan uncertainty estimation and rejection. This
feature is crucial for the real-time deployment of the place recognition
system, as the radar scan is acquired on a frame-by-frame basis during the
driving process.

##  IV Experimental Setup

###  IV-A Datasets

We evaluate the performance using two standard benchmarks. (1) Oxford Radar
RobotCar [35]: It contains radar scans from 30 traversals of the same route
around Oxford city centre equivalent to approximately
$300\text{\,}\mathrm{km}$ driving. We follow the train-test split of Săftescu
et al [7]. An unseen route representing forward and backward traversal of both
urban and vegetation environments is used for testing, and all the rest of the
routes are used for training. We use five test
sequences1112019-01-18-14-46-59-radar-oxford-10k, 2019-01-18-15-20-12-radar-
oxford-10k, 2019-01-18-14-14-42-radar-oxford-10k, 2019-01-14-14-15-12-radar-
oxford-10k, 2019-01-16-14-15-33-radar-oxford-10k where two arbitrary sequences
can form a mapping and localisation pair. We report evaluation metric averages
over all possible combinations222there are in total $20$ pairs, $10$ unique
pairs of sequences, but the assignment of reference/query being important for
introspective query rejection. (2) MulRan [6]: It contains radar scans from
the traversal of different routes at four locations in Daejeon equivalent to
around $120\text{\,}\mathrm{km}$ driving. We train on all sequences from three
locations, DCC, KAIST, and Sejong, then test on the same route in Riverside of
two sequences, which alternate to serve for mapping and localisation in two
testing trails.

Both datasets encompass data collected from a CTS350-X Navtech FMCW scanning
radar. This particular radar system is devoid of Doppler information and is
mounted on a platform in such a manner that its axis of rotation is
perpendicular to the driving surface. The operating frequency of this radar
system is situated within the range of
$76\text{\,}\mathrm{GHz}77\text{\,}\mathrm{GHz}$, which enables it to generate
up to $3768\text{\,}$ range readings with a resolution of
$4.38\text{\,}\mathrm{cm}$. The total range of the radar system is
$165\text{\,}\mathrm{m}$, and each range reading corresponds to one of the
$400\text{\,}$ azimuth readings, which have a resolution of
$0.9\text{\,}\mathrm{\SIUnitSymbolDegree}$ degrees. The radar system is
characterized by a scan rotation rate of $4\text{\,}\mathrm{Hz}$.

###  IV-B Baselines

Benchmarking the recognition performance is done by comparison to several
existing methods, including the vanilla VAE [20], the state-of-the-art radar
place recognition method by Gadd et al [8] (referred as BCRadar), and the non-
learning-based method RingKey [5] (one part of ScanContext, without rotation
refinement). Additionally, the performance is compared against MC Dropout
[16], and STUN [17], which serve as uncertainty-aware place recognition
baselines. We note that our testing of BCRadar uses a different list of
dataset/sequence combinations than in [8].

Ablation study In order to evaluate the effectiveness of our proposed
Introspective Query (Q) and Map Maintenance (M) modules, we conduct an
ablation study by comparing different variants of our method, denoted as
OURS(O/M/Q/QM), which are as follows

  1. 1.

O: No map maintenance, no introspective querying

  2. 2.

M: Map maintenance only

  3. 3.

Q: Introspective querying only

  4. 4.

QM: Both map maintenance and introspective querying

Specifically, we compare the performance of O against M for recognition
performance and Q against QM for uncertainty estimation performance. In each
group of the variant comparison, we perform independent mapping inference over
multiple localisation sequences and report the average metric, or merge
multiple localisation sequences and report a single aggregate metric.

Common settings To ensure a fair comparison, we adopt a common batch
contrastive loss [36] as for all metric learning-based methods, thus enabling
a training regime with a consistent loss function across the benchmarking.

###  IV-C Implementation details

Scan settings For all methods, we transform polar radar scans of with $A=400$
azimuths and $B=3768$ bins of size $4.38\text{\,}\mathrm{cm}$ to Cartesian
scans with side-length $W=256$ and bin size $0.5\text{\,}\mathrm{m}$.

Training hyperparameters We use a VGG-16 [37] as the backbone feature
extractor with a linear layer to project the extracted feature to a lower
embedding dimension of $d=128$. We train all baselines for $10$ epochs in
Oxford Radar RobotCar and $15$ epochs in MulRan respectively333STUN is
finetuned for another $5$ epochs for knowledge distillation, with a learning
rate of $1\mathrm{e}{-5}$, batch size of $8$. In adapting the batch criterion
of [36], we use a negative margin in the embedding space of $0.1$ and
temperature $\tau$ of $1$.

###  IV-D Evaluation Metric

To assess the place recognition performance, we use Recall@N (R@N) metric,
which is the accuracy of the localisation by determining whether at least one
candidate amongst $N$ candidates is close to the ground truth, as indicated by
GPS/INS. This is particularly important to safety assurance in the autonomous
driving application because it reflects the system’s calibration to false
negatives rate. We also use Average Precision (AP) to measure the mean
precision across all recall levels. Finally, we employ F-scores with
$\beta=2/1/0.5$ to assign the increasing level of importance to recall over
precision as an aggregate metric assessing overall recognition performance.
For all metrics, we impose a boundary of $25\text{\,}\mathrm{m}$ such that all
radar scans within it are considered positive pairs to a mapping scan.
Similarly, a $50\text{\,}\mathrm{m}$ lower boundary defines the corresponding
true negative pairs.

Moreover, to assess the uncertainty estimation performance. We use Recall@RR,
where we perform introspective query rejection and evaluate Recall@N=1 over
different uncertainty threshold levels – rejecting all queries of scans with
uncertainty greater than a threshold. We thus reject between
$0\text{\,}\mathrm{\char 37\relax}100\text{\,}\mathrm{\char 37\relax}$ of
queries. A reliable uncertainty measure should reflect the confidence level of
the model’s predictions, and the rejection of low-confidence scans should
result in improved recognition performance (i.e. we should not reject certain,
good results). Our goal is to achieve higher improvement in recognition
performance with a lower rate of rejections.

##  V Results

###  V-A Place Recognition Performance

As demonstrated in Tab. I of the Oxford Radar RobotCar experiments, our
methods, utilizing only the metric-learning module, achieve the highest
performance across all the metrics. Specifically, in terms of Recall@1, our
approach OURS(O) showcases the efficacy of the variance-disentangled
representation learned through the variational contrastive learning framework,
resulting in superior 90.46% recognition performance. This is further
supported with the MulRan experiment results demonstrated in Tab. II, where
our method outperforms all the other methods on Recall@1 and aggregate
F-scores and AP. Although VAE outperforms our method in Recall@5/10 in the
Mulran experiment, the best F-1/0.5/2 and AP of our method in both settings
indicate an overall more accurate and robust recognition performance with high
precision and recall. Furthermore, VAE uncertainty is shown below (Sec. V-B)
to be poor for introspective query rejection, while our learned uncertainty is
useful in both settings.

Moreover, by further utilizing the Continual Map Maintenance in Oxford Radar
RobotCar, we are able to further improve the Recall@1 to 93.67%, outperforming
the current state-of-the-art method, STUN, by a margin of 4.18%. This further
proves the effectiveness of learnt variance as a valid uncertainty measure and
the strategy of uncertainty-based map integration in improving place
recognition performance.

TABLE I: Oxford Radar RobotCar Recognition Performance. The best and second-best results are bold and underlined, respectively. | R@N=1/5/10 | F-1/0.5/2 | AP
---|---|---|---
BCRadar | $85.39$ | $88.59$ | $90.52$ | $0.89$ | $0.83$ | $0.75$ | $0.95$
VAE | $48.35$ | $62.88$ | $69.81$ | $0.56$ | $0.48$ | $0.56$ | $0.70$
RingKey | $83.68$ | $86.49$ | $88.32$ | $0.85$ | $0.81$ | $0.71$ | $0.94$
STUN | $89.49$ | $92.23$ | $93.71$ | $0.92$ | $0.87$ | $0.76$ | $0.98$
MC Dropout | $88.51$ | $91.57$ | $93.24$ | $0.91$ | $0.85$ | $0.76$ | $0.97$
OURS(O) | $\underline{90.46}$ | $\underline{93.76}$ | $\underline{95.22}$ | $\underline{0.93}$ | $\underline{0.87}$ | $\underline{0.77}$ | $\underline{0.98}$
OURS(M) | 93.67 | 96.24 | 97.39 | 0.95 | 0.91 | 0.78 | 0.99
TABLE II: MulRan Recognition Performance. Results reported following the same practice as Tab. I. | R@N=1/5/10 | F-1/0.5/2 | AP
---|---|---|---
BCRadar | $32.23$ | $37.81$ | $40.70$ | $0.36$ | $0.31$ | $0.36$ | $0.43$
VAE | $\underline{36.47}$ | 46.54 | 52.58 | $\underline{0.41}$ | $\underline{0.35}$ | $\underline{0.42}$ | $\underline{0.50}$
RingKey | $27.53$ | $29.34$ | $30.17$ | $0.30$ | $0.27$ | $0.30$ | $0.36$
MC Dropout | $32.90$ | $37.50$ | $40.50$ | $0.35$ | $0.32$ | $0.34$ | $0.43$
STUN | $26.76$ | $31.15$ | $34.04$ | $0.31$ | $0.26$ | $0.31$ | $0.36$
OURS(O) | 37.50 | $\underline{43.54}$ | $\underline{47.57}$ | 0.44 | 0.37 | 0.43 | 0.51

###  V-B Uncertainty Estimation Performance

The change in recognition performance, specifically the Recall@1, with an
increasing percentage of rejected uncertain queries, is illustrated in Fig. 4
for the Oxford Radar RobotCar experiment and in Fig. 5 for the MulRan
experiment. Notably, our method is the only one to exhibit a consistent
improvement in recognition performance as the rejection rate of uncertain
queries increases in both experimental settings – Oxford Radar RobotCar and
MulRan – whereas STUN increases for Oxford Radar RobotCar but not MulRan and
MC Dropout increases for MulRan but not Oxford Radar RobotCar. For Oxford
Radar RobotCar in Fig. 4, it is interesting to note that Ours(QM) and Ours(Q)
but outperform the initial Recall@1 at a lower rejection rate than STUN does –
about $40\text{\,}\mathrm{\char 37\relax}$ for Ours(QM) and Ours(Q) but a much
more comprehensively rejection of $70\text{\,}\mathrm{\char 37\relax}$ than
for STUN. This result serves as evidence of the robustness and effectiveness
of our uncertainty measure.

Figure 4: Oxford Radar RobotCar introspective query rejection performance.
Recall@1 increases/decreases as the percentage of uncertain query rejected
increases. VAE’s performance is not visualised as it is was much lower in
comparison to other methods (specifically (48.42/48.08/18.48)% for
Recall@RR=0.1/0.5/1.0). Figure 5: Mulran introspective query rejection
performance reported the same format as in Fig. 4.

In the MulRan experiments, OURS(Q) is the only method that consistently and
smoothly improves the Recall@RR metric as the rejection rate increases.
Compared to VAE and STUN, which also estimate aleatoric uncertainty like our
method, OURS(Q) achieves an improvement of +(1.32/3.02/8.46)% on
<EMAIL_ADDRESS>while VAE and STUN show a decline of -(3.79/5.24/8.80)%
and -(2.97/4.16/6.30)%, respectively. Although in the Oxford Radar RobotCar
experiments, all methods initially experience fluctuations in performance as
the rejection rate increases, ultimately, OURS(Q/QM) outperforms both VAE and
STUN with a higher Recall@RR metric at all rejection rates. The superior
performance of OURS over VAE suggests the effectiveness of using a
regularizing invariant discriminative loss to address the issues of posterior
collapse and vanishing variance in the vanilla VAE structure. Furthermore, the
better performance of OURS compared to STUN indicates that our variational
contrastive learning framework can be more effective than knowledge
distillation in learning a reliable variance for uncertainty estimation.

On the other hand, compared with MC Dropout, which estimates the epistemic
uncertainty resultant from biased data and model misspecification [30],
although it has a higher increase in Recall@1 at the early stage of rejection
in Oxford Radar RobotCar experiment, its performance is generally lower than
ours and fails to achieve greater improvement as the rejection rate increases
further. Also, in MulRan experiments, it fails to produce reliable uncertainty
estimates, as indicated by the general decreasing trend in Recall@RR. These
results suggest that aleatoric uncertainty plays a more significant role in
causing mispredictions in place recognition than epistemic uncertainty.

Finally, comparing OURS(Q) and OURS(QM) in Oxford Radar RobotCar experiment,
we observe a similar change in Recall@RR pattern while a considerable gap
exists between them. This suggests that the Introspective Query and Map
Maintenance mechanisms independently contribute to the place recognition
system and that each mechanism exploits the uncertainty measure in an
indispensable way.

##  VI Discussion

###  VI-A Qualitative Analysis and Visualisation

To qualitatively assess the source of uncertainty in radar perception, we
provide a visual comparison of the high/low uncertainty samples from both
datasets estimated with our method. As shown in Fig. 6, the high-uncertainty
radar scans usually demonstrate heavy motion blur and sparse undetected
regions, while the low-uncertainty scans usually contain distinct features
with a stronger intensity across the histogram.

Figure 6: Visualisation of Radar Scans with Different Levels of Uncertainty.
The four examples on the left are from the Oxford Radar RobotCar Dataset while
the four examples on the right are from MulRan. We show samples with the
Top-10 highest (top) / lowest (bottom) uncertainty. The radar scan is
displayed in a Cartesian coordinate with enhanced contrast. The histogram
below each image shows the Ring Key descriptor [5] extracted feature of the
intensity across all the azimuths.

This further supports our hypothesis on the source of uncertainty in radar
perception and serves as qualitative evidence of the effectiveness of our
uncertainty measure that captures this data noise.

###  VI-B Dataset Difficulty: Uncertain Environments

In our benchmark experiments, we observed a considerable discrepancy in the
recognition performance between the two datasets. The difficulty of MulRan as
compared to Oxford Radar RobotCar for both place recognition and metric
localisation is already well-documented in [38, 8]. We posit that the scale of
available training data can be a plausible cause. The training set in Oxford
Radar RobotCar comprises over $300\text{\,}\mathrm{km}$ driving experiences,
while the MulRan dataset includes only around $120\text{\,}\mathrm{km}$.
However, also consider the drop in performance for the non-learned Ring Key
descriptor method. This suggests potential inherent indistinguishable features
in radar scene perception. For instance, we found that environments with
sparse open areas usually lead to identical scan and suboptimal recognition
performance. We include results on this dataset to exhibit what happens to our
system and the various baselines in these cases of high uncertainty. As shown
in Fig. 5, our learned uncertainty is the most useful in these difficult
scenarios.

##  VII Conclusion

We have presented a novel application of uncertainty estimation to learned
radar place recognition. We bolster the performance of an invariant instance
feature learning approach with a noise-aware latent representation with
variational learning. We apply this to (1) continual map maintenance for
online system deployment in long-term autonomy, where reference sequences are
constructed from low-uncertainty scans only; and (2) introspective query
rejection with static thresholding for sequential single-scan real-time
inference, where our uncertainty gives the system the ability to understand
when it is likely to be incorrect In tests over two sizeable public radar
datasets, we outperform the previous state-of-the-art in radar place
recognition, and other uncertainty estimation techniques.

## Acknowledgements

This work was supported by EPSRC Programme Grant “From Sensing to
Collaboration” (EP/V000748/1). We would also like to thank our partners at
Navtech Radar.

## References

  * [1] I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” arXiv preprint arXiv:2007.01434, 2020.
  * [2] P. Checchin, F. Gérossier, C. Blanc, R. Chapuis, and L. Trassoudaine, “Radar scan matching slam using the fourier-mellin transform,” pp. 151–161, 2009.
  * [3] S. H. Cen and P. Newman, “Radar-only ego-motion estimation in difficult settings via graph matching,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 298–304, IEEE, 2019.
  * [4] Z. Hong, Y. Petillot, and S. Wang, “Radarslam: Radar based large-scale slam in all weathers,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5164–5170, IEEE, 2020.
  * [5] G. Kim and A. Kim, “Scan context: Egocentric spatial descriptor for place recognition within 3d point cloud map,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4802–4809, 2018.
  * [6] G. Kim, Y. S. Park, Y. Cho, J. Jeong, and A. Kim, “Mulran: Multimodal range dataset for urban place recognition,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2020.
  * [7] Ş. Săftescu, M. Gadd, D. De Martini, D. Barnes, and P. Newman, “Kidnapped radar: Topological radar localisation using rotationally-invariant metric learning,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4358–4364, IEEE, 2020.
  * [8] M. Gadd, D. De Martini, and P. Newman, “Contrastive Learning for Unsupervised Radar Place Recognition,” in Proceedings of the IEEE International Conference on Advanced Robotics (ICAR), 2021.
  * [9] J. Komorowski, M. Wysoczanska, and T. Trzcinski, “Large-scale topological radar localization using learned descriptors,” in Neural Information Processing: 28th International Conference, ICONIP 2021, Sanur, Bali, Indonesia, December 8–12, 2021, Proceedings, Part II 28, pp. 451–462, Springer, 2021.
  * [10] M. Gadd, D. De Martini, and P. Newman, “Look around you: Sequence-based radar place recognition with learned rotational invariance,” in 2020 IEEE/ION Position, Location and Navigation Symposium (PLANS), pp. 270–276, IEEE, 2020\.
  * [11] W. Wang, P. P. de Gusmo, B. Yang, A. Markham, and N. Trigoni, “RadarLoc: Learning to Relocalize in FMCW Radar,” in IEEE International Conference on Robotics and Automation (ICRA), 2021.
  * [12] D. De Martini, M. Gadd, and P. Newman, “kRadar++: Coarse-to-fine FMCW Scanning Radar Localisation,” Sensors, vol. 20, no. 21, p. 6002, 2020\.
  * [13] K. Cait, B. Wang, and C. X. Lu, “Autoplace: Robust place recognition with single-chip automotive radar,” in 2022 International Conference on Robotics and Automation (ICRA), pp. 2222–2228, IEEE, 2022.
  * [14] M. Gadd, D. De Martini, and P. Newman, “Unsupervised Place Recognition with Deep Embedding Learning over Radar Videos,” arXiv preprint arXiv:2106.06703, 2021.
  * [15] H. Yin, X. Xu, Y. Wang, and R. Xiong, “Radar-to-lidar: Heterogeneous place recognition via joint learning,” Frontiers in Robotics and AI, vol. 8, p. 661199, 2021.
  * [16] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in international conference on machine learning, pp. 1050–1059, PMLR, 2016.
  * [17] K. Cai, C. X. Lu, and X. Huang, “STUN: Self-teaching uncertainty estimation for place recognition,” pp. 6614–6621, 2022.
  * [18] F. Warburg, M. Jørgensen, J. Civera, and S. Hauberg, “Bayesian triplet loss: Uncertainty quantification in image retrieval,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12158–12168, 2021\.
  * [19] Y. Shi and A. K. Jain, “Probabilistic face embeddings,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6902–6911, 2019\.
  * [20] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
  * [21] J. Oh and G. Eoh, “Variational bayesian approach to condition-invariant feature extraction for visual place recognition,” Applied Sciences, vol. 11, 2021.
  * [22] H. Wu and M. Flierl, “Learning product codebooks using vector-quantized autoencoders for image retrieval,” in 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1–5, 2019.
  * [23] L. A. P. Rey, D. Jarnikov, and M. Holenderski, “Content-based image retrieval from weakly-supervised disentangled representations,” in NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
  * [24] Y. S. Park, J. Kim, and A. Kim, “Radar localization and mapping for indoor disaster environments via multi-modal registration to prior lidar map,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1307–1314, 2019.
  * [25] X. Lin, Y. Duan, Q. Dong, J. Lu, and J. Zhou, “Deep variational metric learning,” in European Conference on Computer Vision, 2018.
  * [26] K. Burnett, D. J. Yoon, A. P. Schoellig, and T. D. Barfoot, “Radar odometry combining probabilistic estimation and unsupervised feature learning,” arXiv preprint arXiv:2105.14152, 2021.
  * [27] A. Taha, Y.-T. Chen, T. Misu, A. Shrivastava, and L. Davis, “Unsupervised data uncertainty learning in visual retrieval systems,” arXiv preprint arXiv:1902.02586, 2019.
  * [28] D. Adolfsson, M. Karlsson, V. Kubelka, M. Magnusson, and H. Andreasson, “Tbv radar slam – trust but verify loop candidates,” arXiv preprint arXiv:2301.04397, 2023.
  * [29] R. Aldera, D. De Martini, M. Gadd, and P. Newman, “What could go wrong? introspective radar odometry in challenging environments,” in 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 2835–2842, IEEE, 2019\.
  * [30] Y. Gal, Uncertainty in deep learning.  PhD thesis, University of Cambridge.
  * [31] J. Bütepage, L. Maystre, and M. Lalmas, “Gaussian process encoders: Vaes with reliable latent-space uncertainty,” in Machine Learning and Knowledge Discovery in Databases. Research Track, pp. 84–99, Springer International Publishing, 2021.
  * [32] J. He, D. Spokoyny, G. Neubig, and T. Berg-Kirkpatrick, “Lagging inference networks and posterior collapse in variational autoencoders,” arXiv preprint arXiv:1901.05534, 2019.
  * [33] H. Akrami, A. Joshi, S. Aydore, and R. Leahy, “Quantile regression for uncertainty estimation in vaes with applications to brain lesion detection,” in Information Processing in Medical Imaging (A. Feragen, S. Sommer, J. Schnabel, and M. Nielsen, eds.), pp. 689–700, 2021.
  * [34] R. Weston, M. Gadd, D. De Martini, P. Newman, and I. Posner, “Fast-MbyM: Leveraging Translational Invariance of the Fourier Transform for Efficient and Accurate Radar Odometry,” in International Conference on Robotics and Automation (ICRA), pp. 2186––2192, May 2022.
  * [35] D. Barnes, M. Gadd, P. Murcutt, P. Newman, and I. Posner, “The oxford radar robotcar dataset: A radar extension to the oxford robotcar dataset,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), (Paris), 2020.
  * [36] M. Ye, X. Zhang, P. C. Yuen, and S.-F. Chang, “Unsupervised embedding learning via invariant and spreading instance feature,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6210–6219, 2019.
  * [37] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  * [38] T. Y. Tang, D. De Martini, S. Wu, and P. Newman, “Self-supervised learning for using overhead imagery as maps in outdoor range sensor localization,” The International Journal of Robotics Research, vol. 40, no. 12-14, pp. 1488–1509, 2021.


# Handwritten Text Recognition from Crowdsourced Annotations

Solène Tarride  TEKLIAParisFrance<EMAIL_ADDRESS>,  Tristan Faine
TEKLIA, Nantes UniversitéNantesFrance ,  Mélodie Boillet  TEKLIAParisFrance ,
Harold Mouchère  Nantes UniversitéNantesFrance and  Christopher Kermorvant
TEKLIAParisFrance

(2023; 28 April 2023)

###### Abstract.

In this paper, we explore different ways of training a model for handwritten
text recognition when multiple imperfect or noisy transcriptions are
available. We consider various training configurations, such as selecting a
single transcription, retaining all transcriptions, or computing an aggregated
transcription from all available annotations. In addition, we evaluate the
impact of quality-based data selection, where samples with low agreement are
removed from the training set. Our experiments are carried out on municipal
registers of the city of Belfort (France) written between 1790 and 1946. The
results show that computing a consensus transcription or training on multiple
transcriptions are good alternatives. However, selecting training samples
based on the degree of agreement between annotators introduces a bias in the
training data and does not improve the results. Our dataset is publicly
available on Zenodo.

Handwritten Text Recognition, Crowdsourcing, Text Aggregation, Historical
Documents, Neural Networks

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: 7th International Workshop on Historical Document
Imaging and Processing (HIP’23); August 25, 2023; San José, California††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Applied computing Optical character
recognition††ccs: Information systems Crowdsourcing††ccs: Computing
methodologies Neural networks Figure 1. Samples from the minutes of Belfort
municipal council drawn up between 1790 and 1946. These documents include
deliberations, lists of councillors, convocations, and agendas.

##  1\. Introduction

The Municipal Archives of Belfort have launched a pilot project for the
automatic transcription of all council minutes. The aim of the project is to
automatically process 18,500 pages of various documents written between 1790
and 1946, as presented in Figure 1. Images and transcriptions will then be
made freely available so that citizens can not only browse the pages but also
search for specific information. This will give citizens the opportunity to
participate in local life by discovering how the City Council works and how
the city has developed over the centuries.

To collect training data for automatic Handwritten Text Recognition (HTR)
models, we set up an open collaborative annotation campaign using the
Callico111https://callico.teklia.com annotation tool, which allows any user to
request tasks. The annotation interface is presented in Figure 2. To annotate
a page, the user must enter the transcription of the line highlighted in blue
in the corresponding text box, also highlighted in blue. All lines highlighted
in green must be transcribed by the user to complete the task.

Figure 2. Screenshot of Callico, the annotation tool used for collaborative
annotation.

Since the model will learn from the collaborative transcriptions, their
quality is crucial as it will affect its performance. An easy way to estimate
the quality of the collected data is to have two annotators transcribe each
line. For this reason, the transcription campaign is configured so that each
document can be annotated by two different annotators. As there is no clear
way to identify potential transcription issues for pages transcribed by a
single annotator, we also rely on pre-trained automatic HTR models to generate
two automatic transcriptions for each line. Ultimately, each line will have
either one or two human transcriptions and two automatic transcriptions.

This paper explores various approaches to train a model when multiple
uncertain transcriptions are available for each line. We investigate three
main issues: transcription selection, sample selection based on transcription
quality, and data splitting. This paper aims to answer the following
questions.

  1. (1)

How to train an HTR model when multiple annotations are available for each
image? Three main training strategies are explored: merging multiple
transcriptions, selecting only one transcription, or using all available
transcriptions.

  2. (2)

How to identify low-quality annotations, and what is their impact on model
performance? We measure the degree of agreement between multiple
transcriptions. Then, we use this metric to discard unreliable samples and
measure the impact on model performance.

  3. (3)

How to partition the data into training, validation and test sets when
transcription quality is unknown? We address the issue of data partitioning in
scenarios where the transcriptions are uncertain. Two distinct splits are
defined: the first ensures the reliability of the validation and test sets,
while the second is randomized.

This paper is organized as follows. The next Section 2 presents related works.
Section 3 introduces the dataset and Section 4 presents our experiments and
results. Finally, we discuss the results and conclude in Section 5.

##  2\. Related Works

In this paper, we focus on two main points. First, we explore the different
methods for quality assessment in the context of crowdsourcing. Second, we
investigate potential training strategies when multiple noisy labels are
available for each sample. The literature on each of these points is detailed
in the following paragraphs.

###  2.1. Label quality assessment

Crowdsourcing platforms are a relatively new way for task providers, platform
managers and workers to interact together in order to meet the task provider’s
expectations. The survey proposed by Daniel et al. (Daniel et al., 2018)
provides comprehensive insights into quality assessment in the context of
crowdsourcing.

####  2.1.1. Group assessment

The simplest method of ensuring high quality annotations is to allow humans to
agree on the best label (Daniel et al., 2018) by allowing voting strategies,
peer review mechanisms, or output agreement. Collaborative annotation has been
shown to be more efficient than aggregating individual annotations, as
demonstrated by Blickhan et al. (Blickhan et al., 2019), which suggests that
annotators should be allowed to see previous annotations made by others.
Another approach consists in relying on moderation to correct or discard
unreliable transcriptions.

Human-based approaches to quality control are extremely effective, but require
considerable time and effort to implement. In addition, additional features
need to be incorporated into the annotation interface to facilitate human-
based quality control.

####  2.1.2. Computational assessment

Computational methods can also be used for quality assessment (Daniel et al.,
2018). A simple method is to annotate a random subset of the dataset with gold
truth annotations and compute metrics for quality control. Other approaches
rely on content analysis to estimate task difficulty and worker reliability.
Zhu et al. (Zhu et al., 2021) propose an outlier detection method for
classification tasks to detect where annotators disagree. Based on the
assumption that similar samples are more likely to share the same label, their
method compares each noisy label with the labels of its nearest neighbours in
the feature space.

###  2.2. Learning with multiple labels

There are several methods for training a model when multiple transcriptions
are available. It is possible to select a single label using text aggregation
methods, or to adapt training strategies to allow multiple labels for each
sample.

####  2.2.1. Label selection

The task of text summarization can provide some insight into how consensus
between different texts can be calculated, using abstractive or extractive
methods (El-Kassas et al., 2021).

Abstractive aggregation techniques produce a transcription on which most
annotators agree. The consensus text combines the features of each
transcription and can be seen as a weighted average of the different
transcriptions. Recognizer Output Voting Error Reduction (ROVER) (Fiscus,
1997), is a system that uses dynamic programming to align word sequences then
applies a voting strategy on every edge of the resulting Word Transition
Network. Although ROVER was originally designed to align words, it can also be
used at character or sub-word level. The ROVER algorithm is included in the
crowd-kit Python package (Ustalov et al., 2023). Collatex (Group, 2019) is
another tool designed for collating textual sources, it provides different
alignment algorithms in order to align tokens, then a voting technique can be
applied to the resulting alignment table.

Extractive aggregation techniques select the transcription that is most
similar to each other. While this doesn’t really produce a consensus, it could
be used to discard unlikely annotations. The Reliability Aware Sequence
Aggregation (RASA) (Li and Fukumoto, 2019) and the Hybrid Reliability and
Representation Aware Sequence Aware Aggregation (HRRASA) (Li, 2020) algorithms
have also been proposed to determine which of the annotators’ answers is
closest to the estimated aggregated answer. Both methods are able to take into
account the reliability of each annotator, which is a representation of how
close they have been to previous aggregations over time. HRRASA differs from
RASA in that it considers local reliability within predefined subsets of
transcriptions. Both algorithms are implemented in the crowd-kit Python
package (Ustalov et al., 2023).

####  2.2.2. Training strategies that are able to deal with multiple noisy
labels

While the literature on crowdsourced text transcription is sparse, there are
many articles focusing on crowdsourced labels for classification tasks.

Training with all available labels is a straightforward and workable solution,
as shown by Wei et al. (Wei et al., 2022b). The authors assume that instances
can be associated with different noisy labels due to different
interpretations. According to their results on several datasets, the use of
multiple individual labels leads to an increase in accuracy when the number of
annotators is insufficient or when the noise rate is high (low-quality
annotations).

Adapting models to handle multiple noisy labels is also possible. CrowdLayer
(Rodrigues and Pereira, 2018) relies on a crowd layer that takes the softmax
outputs of a standard deep neural network as the input, and learns multiple
parametric annotator-specific transition matrices to fit the given label of
each annotator independently. UnionNet (Wei et al., 2022a) is a deep neural
network that can be trained directly by maximizing the likelihood of a
combined set of labels using a parametric transition matrix, thus lowering the
computational cost and possibly learn inter-annotator relationships. In
addition to annotator-based noise, researchers also highlight the impact of
sample-based noise (Gao et al., 2022; Wei et al., 2021). Gao et al. (Gao et
al., 2022) propose to add regularization terms in the loss function to make
weight vectors and confusion matrices sample-dependent.

###  2.3. Discussion

The literature review identifies two main issues that are relevant to our
application.

####  2.3.1. Quality assessment

As only two human annotations are available for each line, we also use two
pre-trained models to generate automatic transcriptions. As a result, four
annotations are available for each sample. This makes it possible to compute
an agreement score between individual annotations, which can help to estimate
the task difficulty and label noise.

####  2.3.2. Learning from multiple transcriptions

Existing methods are designed for classification tasks and cannot be directly
applied to handwritten text recognition. In addition, many of these methods
require the identification of the annotator, which is not currently possible
with our annotation campaign. However, label selection techniques are
interesting because they can potentially correct local errors made by
individual annotators.

##  3\. Collaborative annotation of the Belfort dataset

In this section, we describe how the transcriptions were collected and present
the annotated Belfort dataset. We first randomly selected 616 images and used
a pre-trained Doc-UFCN (Boillet et al., 2021) model to automatically detect
text lines in the pages. We then created a collaborative annotation campaign
to collect training data for the text recognition models.

###  3.1. Annotation guidelines

The following annotation guidelines have been established in order to make the
collected transcriptions as consistent as possible.

Only highlighted lines should be transcribed. If there is a line segmentation
error where the highlighting covers several lines, the transcription should be
left blank. If several words are illegible, the transcription should be left
blank. Illegible signatures should not be annotated. Spelling and conjugation
errors should be corrected, punctuation and capitalization should be restored,
and abbreviations should be expanded. If there is any uncertainty about the
transcription, it should be entered as accurately as possible, without adding
any characters, and the line should be marked as ”uncertain”. If the quality
of the page is poor and illegible, it should be skipped.

At the time we extracted the data from the annotation platform, 64% of the
pages were transcribed by two independent annotators, 27% by a single
annotator, and 9% were skipped. On average, an annotator took 21 minutes to
annotate a full page.

###  3.2. Description of the annotated dataset

From these annotations, we build a line-level dataset for handwritten text
recognition. This dataset is publicly available on
Zenodo222https://zenodo.org/record/8041668.

####  3.2.1. Dataset

24,105 text lines were transcribed, including 37% transcribed by two different
human annotators. A significant gap can be observed between the number of
pages with two annotations (64%) and lines with two annotations (37%). This is
because many lines were difficult to read, and were therefore skipped, left
blank or ignored by at least one annotator. Lines with no transcription are
excluded from the dataset.

####  3.2.2. Automatic transcriptions

We want to investigate training strategies when multiple annotations are
available for the same image. However, some lines in our dataset were
transcribed by a single annotator. For this reason, we also use two
handwriting recognition models (PyLaia (Puigcerver, 2017) and DAN (Coquenet et
al., 2023)) trained on an earlier version of this dataset to generate
automatic annotations.

As a result, each line now have three or four transcriptions. These
transcriptions often include small errors, as illustrated in Figure 3.

Figure 3. Different transcriptions associated to a single text line. None of
these transcriptions are exactly correct. The gold transcription should be
”Jeaugeage public, déclare que les lois des 7 Brumaire au 9”.

####  3.2.3. Splitting strategy

Since our goal is to train and compare different models, we need a reliable
test set to evaluate the models as accurately as possible. However, we have no
prior information about the quality of the transcriptions, so there is no
guarantee that the annotations are correct. When two manual annotations are
available, one way to ensure the quality of the collected transcriptions is to
look at the agreement level between the two annotators.

Therefore, the test set was designed to contain only lines where the two
annotators agree exactly (Character Error Rate = 0%). As the validation set is
also important when training machine learning models, lines with good
agreement (0% ¡ Character Error Rate ¡ 5%) between the two human annotations
are selected to be in the validation set. Finally, all the other lines are
used for training.

The detailed split is described in Table 1. We discuss the potential bias
introduced by this splitting strategy in Section 4.4.

Table 1. Details of the data splitting at line level Split | Number | Percentage (%)
---|---|---
Train | 19,013 | 78.9
Validation | 2,262 | 9.4
Test | 2,830 | 11.7
Total | 24,105 | 100.0

####  3.2.4. Level of agreement between annotators

The agreement score between multiple transcriptions can be computed using the
following approach. First, a consensus transcription is obtained by taking a
majority vote at each character position among the N transcriptions, using the
ROVER algorithm from the crowd-kit Python
package333https://toloka.ai/docs/crowd-
kit/reference/crowdkit.aggregation.texts.rover.ROVER/ at the character-level.
Next, the character distance between each transcription and the ROVER
consensus is calculated, to measure the dissimilarity between them. Finally,
the computed character distances are averaged across all transcriptions to
obtain an agreement score which reflects the overall level of agreement
between the transcriptions.

The main factors contributing to low agreement include poor-quality images,
line segmentation errors (mostly merged lines), challenging handwriting, and
non-compliance with annotation guidelines (incomplete transcription,
punctuation errors, casing errors, additional symbols (such as ”?”) for
uncertain words).

##  4\. Experiments

Our experiments focus on three research questions. First, we explore different
strategies for training a handwriting recognition model when multiple
transcriptions are available for each image. We also measure the impact of
data quality on model performance by filtering out unreliable samples.
Finally, we consider the potential bias introduced by our data splitting
strategy.

###  4.1. Experimental setup

For all the following experiments, we train a
PyLaia444https://github.com/jpuigcerver/PyLaia (Puigcerver, 2017) model
consisting of four convolutional blocks and three recurrent blocks, followed
by a linear layer. Each convolution block is composed of a convolution layer,
batch normalization, the LeakyReLU activation function and max pooling. Each
recurrent block includes a Bi-LSTM layer. All images are resized to a fixed
size of 128 pixels, while preserving the original aspect ratio. Training is
performed using a batch size of 8. Early stopping is applied after 50 epochs
with no improvement on the validation set.

The model is evaluated on the test set using human transcriptions.

###  4.2. Learning from multiple transcriptions

We want to compare different ways of integrating multiple transcriptions into
the training of a model. We investigate and compare three methods: selecting a
single annotation for each line or keeping multiple annotations for each line.

####  4.2.1. Selection of a single annotation

The first approach consists in selecting a single annotation for each line. We
compare three ways of doing this:

  * •

Randomly selecting a transcription;

  * •

Selecting the best transcription using the RASA algorithm;

  * •

Selecting the aggregated transcription obtained using the ROVER algorithm.

####  4.2.2. Retention of multiple annotations

The second approach consists in duplicating images with multiple
transcriptions, so that a single line appears $n$ times in the training set if
it has $n$ different transcriptions. We investigate the following three
strategies:

  * •

Use all human transcriptions;

  * •

Use all human and automatic transcriptions;

  * •

Use all human, automatic, and aggregated transcriptions (RASA and ROVER).

####  4.2.3. Results

Table 2. Evaluation results for experiments on transcription selection. The same PyLaia architecture is trained using different transcriptions Transcription selection strategy | Validation11footnotemark: 1 | Test
---|---|---
CER (%) | WER (%) | CER (%) | WER (%)
Random selection | 6.04 | 21.53 | 5.44 | 18.37
RASA selection | 4.80 | 17.15 | 4.95 | 17.17
ROVER consensus | 4.55 | 16.73 | 4.95 | 17.08
All human transcriptions | 6.05 | 22.18 | 5.57 | 19.12
All human and automatic transcriptions | 6.04 | 20.29 | 5.53 | 16.50
All transcriptions (human, automatic, RASA, ROVER) | 5.10 | 19.03 | 4.34 | 15.14
11footnotemark: 1 Validation sets include different transcriptions

The results of our experiments are presented in Table 2.

For label selection strategies, RASA and ROVER are better than a random
selection on the test set, with the WER reduced by 1 point. Although there is
no clear winner between RASA and ROVER, ROVER achieves the best WER.

For label retention strategies, we observe that including automatic
transcriptions helps to reduce the WER. Using all available transcriptions has
an even more significant effect, resulting in a decrease of 1 percentage point
on the CER and almost 4 points on the WER.

The best overall performance on the test set is achieved by using all
available transcriptions, resulting in a 4.34% CER and a 15.14% WER. One
possible reason for this improvement is that training with a larger number of
words in the training set improves PyLaia’s implicit language modelling
mechanism. Despite the fact that each image appears with different labels five
or six times in the training set, we observe that the model does not overfit
earlier than other training configurations.

###  4.3. Quality-driven training sample selection

In this section, we investigate how transcription quality affects model
performance. To achieve this, we use the agreement score to remove unreliable
samples from the training set.

####  4.3.1. Selection strategy

Using the ROVER consensus transcription seems to be the best option for
training a model. However, if the available transcriptions are very different,
the ROVER consensus transcription is likely to contain errors. This can lead
to the model being trained with unreliable labels. To avoid this, we rely on
the agreement score to filter out unreliable training samples (e.g. with low
agreement scores) during PyLaia’s training, allowing us to evaluate the
resulting impact of on performance.

In this study, we experiment with three different thresholds: 90%, 97%, and
99% of agreement score. These thresholds are chosen because they correspond to
retaining approximately 75%, 50%, and 30% of the training samples
respectively.

####  4.3.2. Results

Results of our experiments are presented in Table 3.

Table 3. Evaluation results for experiments on agreement-based quality selection. The same PyLaia architecture is trained using the ROVER consensus and by filtering out samples with agreement below a certain threshold Filtering strategy | Threshold | Training samples | Validation | Test
---|---|---|---|---
CER (%) | WER (%) | CER (%) | WER (%)
Retain all training samples | 0% | 19,013 (100.0%) | 4.55 | 16.73 | 4.95 | 17.08
Discard samples with low agreement | 90% | 14,384 (75.7%) | 5.26 | 17.87 | 5.53 | 17.38
Discard samples with low agreement | 97% | 9,561 (50.3%) | 6.63 | 20.66 | 7.13 | 21.13
Discard samples with low agreement | 99% | 5,578 (29.3%) | 7.98 | 22.76 | 8.51 | 23.48

The results indicate that filtering out low agreement samples degrades
performance. The CER increases from 4.95% to 5.53% when filtering out samples
with less than 90% agreement, reaches 6.63% for a 97% threshold, and 7% for a
99% threshold. Although this approach ensures more reliable labels, it also
reduces the size of the training dataset, which usually leads to poorer
performance. Another possible explanation is that removing low-agreement
samples may lead to the removal of the most challenging samples.

It should also be noted that PyLaia automatically discards invalid lines for
CTC computation, which includes images with abnormally long transcriptions. As
a result, a few unreliable samples are discarded before any agreement-based
filtering.

###  4.4. Impact of splitting strategy

In this section, we discuss the limitations of our splitting strategy
presented in Section 3.2.3.

####  4.4.1. Agreement based splitting

We identify a bias introduced by our splitting strategy. The test set consists
of lines with perfect agreement between annotators, which favors lines with
little text and lines with legible handwriting. The same observation applies
to a lesser extent to the validation set, which consists of lines with good
agreement between annotators.

####  4.4.2. Random splitting

In order to measure the impact of this bias, we train and evaluate another
model on a random split, using only human annotations to allow comparison
between the different sets. The random split is designed to have the same
number of samples as the agreement-based split to measure the difference in
terms of CER/WER. This corresponds to 19,013 images in the training set, 2,262
in the validation set, and 2,830 in the test set.

color=yellow]Solène: oui je suis d’accord, mais on n’a pas le temps de ré-
entraîner plusieurs modèles (un entraînement = 24h)

####  4.4.3. Results

Results of our experiments are presented in Table 4.

Table 4. Evaluation results using different splitting strategies Spliting strategy | Train | Validation | Test
---|---|---|---
CER (%) | WER (%) | CER (%) | WER (%) | CER (%) | WER (%)
Agreement-based split | 8.74 | 23.85 | 6.05 | 22.18 | 5.57 | 19.12
Random split | 5.89 | 16.98 | 10.87 | 27.84 | 10.54 | 28.11

Using the agreement-based split, the test labels can be considered as gold
transcriptions, as both annotators agree, which allows a fair assessment of
performance. However, the results on this split clearly show a bias, as the
performance on the training set is lower than on the test set. This suggests
that test samples are ”easier” than training samples. As a result, the model’s
performance on the test set is over-estimated.

On the random split, the CER and WER are much higher than on the agreement-
based split. However, the test set may contain uncertain and noisy
transcriptions, making it difficult to fairly assess the performance of the
model.

The actual performance of the model cannot be precisely assessed, it would be
necessary to do cross-validation to estimate more precisely the performance of
the model trained on the random split.

##  5\. Conclusion

In this paper, we have investigated different ways of training HTR models from
crowdsourced annotations that may contain noise. We explored different
training strategies to benefit from multiple uncertain annotations. The
experimental results indicate that training a model with multiple noisy
transcriptions or computing a consensus are two viable approaches. However, we
observe that filtering training samples based on agreement between annotators
introduces a bias in the training dataset and does not improve model
performance. This observation suggests that data quantity may be more
important than data quality, but it would be interesting to explore other
quality metrics that do not introduce such a bias.

This project provides insights for improving annotation interfaces and for
conducting future collaborative annotation campaigns. First, computing
annotator-based confidence scores would allow campaign administrators to
provide feedback to underperforming annotators. This information could also be
used to train HTR models. Secondly, we believe that collaboration between
annotators should be encouraged. For example, the integration of features that
allow users to agree, vote, validate or correct existing transcriptions would
be very useful. Thirdly, the use of image quality metrics would help to remove
blurred images and poorly segmented lines from the annotation campaign, as
these tasks confused most annotators. Finally, the guidelines were not always
followed by the annotators, so the addition of automatic validation checks
would be useful. For example, suspicious words could be underlined, a warning
could be issued if the text length does not match the line polygon area, or a
Named Entity Recognition (NER) model could be used to flag non-capitalised
names.

For future work, we plan to explore Large Language Models (LLMs) to compute a
clean consensus with corrected typing, grammar, and spelling errors. We would
also like to explore training strategies for combining multiple
transcriptions, such as using a custom annotator-weighted loss function or
mechanisms similar to label smoothing.

###### Acknowledgements.

We would like to thank the Municipal Archives of Belfort for giving us access
to these documents, as well as the 87 annotators who made this project
possible.

## References

  * (1)
  * Blickhan et al. (2019) Samantha Blickhan, Coleman Krawczyk, Daniel Hanson, Amy Boyer, Andrea Simenstad, and Victoria van Hyning. 2019.  Individual vs. Collaborative Methods of Crowdsourced Transcription. In _Journal of Data Mining and Digital Humanities_.  https://doi.org/10.46298/jdmdh.5759
  * Boillet et al. (2021) Mélodie Boillet, Christopher Kermorvant, and Thierry Paquet. 2021.  Multiple Document Datasets Pre-training Improves Text Line Detection With Deep Neural Networks. In _25th International Conference on Pattern Recognition_. 2134–2141.  https://doi.org/10.1109/ICPR48806.2021.9412447
  * Coquenet et al. (2023) Denis Coquenet, Clément Chatelain, and Thierry Paquet. 2023\.  DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition. In _IEEE Transactions on Pattern Analysis and Machine Intelligence_.  https://doi.org/10.1109/TPAMI.2023.3235826
  * Daniel et al. (2018) Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018\.  Quality Control in Crowdsourcing. In _ACM Computing Surveys_. 1–40.  https://doi.org/10.1145/3148148
  * El-Kassas et al. (2021) Wafaa S. El-Kassas, Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed. 2021.  Automatic text summarization: A comprehensive survey. In _Expert Systems with Applications_. 113679\.  https://doi.org/10.1016/j.eswa.2020.113679
  * Fiscus (1997) J.G. Fiscus. 1997\.  A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction (ROVER). In _IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings_. 347–354.  https://doi.org/10.1109/ASRU.1997.659110
  * Gao et al. (2022) Zhengqi Gao, Fan-Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Marc Engeler, Antonio Burazer, Linda Wildling, Luca Daniel, and Duane S. Boning. 2022\.  Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusion. In _Computer Vision – ECCV 2022_. 407–422.  https://doi.org/10.1007/978-3-031-20053-3_24
  * Group (2019) The Interedition Development Group. 2010-2019.  https://collatex.net/ Collatex software home page.  Accessed: 2023-04-14.
  * Li (2020) Jiyi Li. 2020.  Crowdsourced Text Sequence Aggregation Based on Hybrid Reliability and Representation. In _43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1761–1764.  https://doi.org/10.1145/3397271.3401239
  * Li and Fukumoto (2019) Jiyi Li and Fumiyo Fukumoto. 2019.  A Dataset of Crowdsourced Word Sequences: Collections and Answer Aggregation for Ground Truth Creation. In _Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP_. 24–28.  https://doi.org/10.18653/v1/D19-5904
  * Puigcerver (2017) Joan Puigcerver. 2017\.  Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition?. In _14th International Conference on Document Analysis and Recognition_. 67–72.  https://doi.org/10.1109/ICDAR.2017.20
  * Rodrigues and Pereira (2018) Filipe Rodrigues and Francisco Pereira. 2018.  Deep learning from crowds. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_. 1611–1618.  https://doi.org/10.5555/3504035.3504232
  * Ustalov et al. (2023) Dmitry Ustalov, Nikita Pavlichenko, and Boris Tseitlin. 2023\.  Learning from Crowds with Crowd-Kit.  https://arxiv.org/abs/2109.08584
  * Wei et al. (2022a) Hongxin Wei, Renchunzi Xie, Lei Feng, Bo Han, and Bo An. 2022a.  Deep Learning From Multiple Noisy Annotators as A Union. In _IEEE transactions on neural networks and learning systems_.
  * Wei et al. (2021) Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. 2021\.  Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations.  _ArXiv_ abs/2110.12088 (2021).
  * Wei et al. (2022b) Jiaheng Wei, Zhaowei Zhu, Tianyi Luo, Ehsan Amid, Abhishek Kumar, and Yang Liu. 2022b.  To Aggregate or Not? Learning with Separate Noisy Labels. In _CSW@WSDM_.
  * Zhu et al. (2021) Zhaowei Zhu, Zihao Dong, and Yang Liu. 2021.  Detecting Corrupted Labels Without Training a Model to Predict. In _International Conference on Machine Learning_.