# On the Calibration of Massively Multilingual Language Models

Kabir Ahuja1 Sunayana Sitaram1 Sandipan Dandapat2 Monojit Choudhury2
1 Microsoft Research, India
2 Microsoft R&D, India
<EMAIL_ADDRESS>
###### Abstract

Massively Multilingual Language Models (MMLMs) have recently gained popularity
due to their surprising effectiveness in cross-lingual transfer. While there
has been much work in evaluating these models for their performance on a
variety of tasks and languages, little attention has been paid on how well
calibrated these models are with respect to the confidence in their
predictions. We first investigate the calibration of MMLMs in the zero-shot
setting and observe a clear case of miscalibration in low-resource languages
or those which are typologically diverse from English. Next, we empirically
show that calibration methods like temperature scaling and label smoothing do
reasonably well towards improving calibration in the zero-shot scenario. We
also find that few-shot examples in the language can further help reduce the
calibration errors, often substantially. Overall, our work contributes towards
building more reliable multilingual models by highlighting the issue of their
miscalibration, understanding what language and model specific factors
influence it, and pointing out the strategies to improve the same.

##  1 Introduction

((a))

((b))

((c))

((d))

Figure 1: Reliability Diagrams for XLMR fine-tuned on XNLI using training data
in English. 1(a) and 1(b) provides the calibration of XLMR fine-tuned without
any calibration technique on English and Swahilli. 1(c) and 1(d) are obtained
by calibrating the model using TS + LS and Self-TS + LS techniques
respectively as described in Section 2.1

Massively Multilingual Language Models (MMLMs) like mBERT Devlin et al.
(2019), XLMR Conneau et al. (2020), mT5 Xue et al. (2021) and mBART Liu et al.
(2020) have been surprisingly effective at zero-shot cross lingual transfer
i.e. when fine-tuned on an NLP task in one language, they often tend to
generalize reasonably well in languages unseen during fine-tuning.

These models have been evaluated for their performance across a range of
multilingual tasks Pan et al. (2017); Nivre et al. (2018); Conneau et al.
(2018) and numerous methods like adapters Pfeiffer et al. (2020), sparse fine-
tuning Ansell et al. (2022) and few-shot learning Lauscher et al. (2020) have
been proposed to further improve performance of cross lingual transfer.

Despite these developments, there has been little to no attention paid to the
calibration of these models across languages i.e. how reliable the confidence
predictions of these models are. As these models find their way more and more
into the real word applications with safety implications, like Hate Speech
Detection Davidson et al. (2017); Deshpande et al. (2022) it becomes important
to only take extreme actions for high confidence predictions by the model
Sarkar and KhudaBukhsh (2021). Hence, calibrated confidences are desirable to
have when deploying such systems in practice.

Guo et al. (2017) showed that modern neural networks used for Image
Recognition He et al. (2016) perform much better than the ones introduced
decades ago Lecun et al. (1998), but are significantly worse calibrated and
often over-estimate their confidence on incorrect predictions. For NLP tasks
specifically, Desai and Durrett (2020) showed that classifiers trained using
pre-trained transformer based models Devlin et al. (2019) are well calibrated
both in-domain and out-of-domain settings compared to non-pre-trained model
baselines Chen et al. (2017). Notably, Ponti et al. (2021) highlights, since
zero-shot cross lingual transfer represents shifts in the data distribution
the point estimates are likely to be miscalibrated, which forms the core
setting of our work.

In light of this, our work has three main contributions. First, we investigate
the calibration of two commonly used MMLMs: mBERT and XLM-R on four NLU tasks
under zero-shot setting where the models are fine-tuned in English and
calibration errors are computed on unseen languages. We find a clear increase
in calibration errors compared to English as can be seen in Figures 1(a) and
1(b), with calibration being significantly worse for Swahili compared to
English.

Second, we look for factors that might affect the zero-shot calibration of
MMLMs and find in most cases that the calibration error is strongly correlated
with pre-training data size, syntactic similarity and sub-word overlap between
the unseen language and English. This reveals that MMLMs are mis-calibrated in
the zero-shot setting for low-resource languages and the languages that are
typologically distant from English.

Finally, we show that model calibration across different languages can be
substantially improved by utilizing standard calibration techniques like
Temperature Scaling Guo et al. (2017) and Label Smoothing Pereyra et al.
(2017) without collecting any data in the language (see Figure 1(c)). Using a
few examples in a language (the few-shot setting), we see even more
significant drops in the calibration errors as can be seen in Figure 1(d).

To the best of our knowledge, ours is the first work to investigate and
improve the calibration of MMLMs. We expect this study to be a significant
contribution towards building reliable and linguistically fair multilingual
models. To encourage future research in the area we make our code publicly
available111https://github.com/microsoft/MMLMCalibration.

##  2 Calibration of Pre-trained MMLMs

Consider a classifier $h:\mathcal{X}\rightarrow[K]$ obtained by fine-tuning an
MMLM for some task with training data in a pivot language $p$, where $[K]$
denotes the set of labels $\\{1,2,\cdots,K\\}$. We assume $h$ can predict
confidence of each of the $[K]$ labels and is given by $h_{k}(x)\in[0,1]$ for
the $k^{th}$ label. $h$ is said to be calibrated if the following equality
holds:

$p(y=k|h_{k}(x)=q)=q$

In other words, for a perfectly calibrated classifier, if the predicted
confidence for a label $k$ on an input $x$ is $q$, then with a probability $q$
the input should actually be labelled $k$. Naturally, in practical settings
the equality does not hold, and neural network based classifiers are often
miscalibrated Guo et al. (2017). One way of quantifying this notion of
miscalibration is through the Expected Calibration Error (ECE) which is
defined as the difference in expectation between the confidence in
classifier’s predictions and their accuracies (Refer Appendix A.1 for
details). In our experiments we compute ECE on each language $l$’s test data
and denote their corresponding calibration errors as $\texttt{ECE}(l)$.

###  2.1 Calibration Methods

We briefly review some commonly used methods for calibrating neural network
based classifiers.

1\. Temperature Scaling (TS and Self-TS)Guo et al. (2017) is applied by
scaling the output logits using a temperature parameter $T$ before applying
softmax i.e. :

$h_{k}(x)=\frac{\exp{o_{k}(x)/T}}{\sum_{k^{\prime}\in
K}\exp{o_{k^{\prime}}(x)/T}}$

, where $o_{k}$ denotes the logits corresponding to the $k^{th}$ class. $T$ is
a learnable parameter obtained post-training by maximizing the log-likelihood
on the dev set while keeping other network parameters fixed. We experiment
with two settings for improving calibration on a target language: using dev
data in English to perform temperature scaling (TS) and using the target
language’s dev data (Self-TS).

2\. Label Smoothing (LS) Pereyra et al. (2017) is a regularization technique
that penalizes low entropy distributions by using soft labels that are
obtained by assigning a fixed probability $q=1-\alpha$ to the true label
($0<\alpha<1$), and distributing the remaining probability mass uniformly
across the remaining classes. Label smoothing has been empirically shown to be
competitive with temperature scaling for calibration Müller et al. (2019)
especially in out of domain settings Desai and Durrett (2020)

3\. Few-Shot Learning (FSL) We also investigate if fine-tuning the MMLM on a
few examples in a target language in addition to the data in English, leads to
any improvement in calibration as it does in the performance Lauscher et al.
(2020). Since these models are expected to be calibrated worse for out-of-
domain data compared to in-domain data Desai and Durrett (2020), we try to
improve calibration by reducing the domain shift through few-shot learning.

Apart from these, we also consider combinations of different calibration
methods in our experiments, including Label Smoothing with Temperature Scaling
(TS + LS or Self-TS + LS) and Few-Shot Learning with Label Smoothing (FSL +
LS).

##  3 Experiments

We seek to answer the following research questions: a) How well calibrated are
fine-tuned MMLMs in the zero-shot cross lingual setting? b) What linguistic
and model-specific factors influence calibration errors across languages? c)
Can we improve the calibration of fine-tuned models across languages?

Dataset |  MMLM |  $\texttt{ECE}(en)$ |  $\displaystyle\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$ |  $\displaystyle\max_{l\in\mathcal{L}^{\prime}}\texttt{ECE}(l)$
---|---|---|---|---
XNLI |  XLM-R |  7.32 |  13.34 |  19.07 (sw)
mBERT |  5.44 |  12.34 |  45.15 (th)
XCOPA |  XLM-R |  14.54 |  20.07 |  29.33 (sw)
mBERT |  23.4 |  23.51 |  29.02 (sw)
MARC |  XLM-R |  7.15 |  9.65 |  13.45 (zh)
mBERT |  9.38 |  11.11 |  17.33 (ja)
PAWS-X |  XLM-R |  1.93 |  4.28 |  5.88 (ja)
mBERT |  3.57 |  10.32 |  15.65 (ko)
Table 1: Calibration Errors across tasks for XLM-R and mBERT. $\mathcal{L}^{\prime}$ in the fourth column denotes the set of supported languages in a task other than English. The language in parenthesis in the column 5 denotes the language with maximum calibration error. Dataset |  SIZE |  SYN |  SWO
---|---|---|---
XNLI |  -0.8 |  -0.88 |  -0.85
XCOPA |  -0.85 |  -0.73 |  -0.62
MARC |  -0.41 |  -0.46 |  -0.27
PAWS-X |  -0.48 |  -0.93 |  -0.92
Table 2: Pearson correlation coefficient of ECE with SIZE, SYN, and SWO features of different languages in the test set for XLMR. Dataset |  MMLM | Zero-Shot Calibration | Few-Shot Calibration
---|---|---|---
OOB |  TS |  LS |  TS + LS |  Self-TS |  Self-TS + LS |  FSL |  FSL + LS
XNLI |  XLM-R |  13.34 |  6.74 |  6.93 |  4.89† |  5.41 |  4.05‡ |  7.67 |  4.36
mBERT |  12.34 |  6.29† |  10.42 |  6.70 |  4.77 |  4.69 |  3.14 |  2.55‡
XCOPA |  XLM-R |  20.07 |  15.95 |  5.47 |  4.52† |  16.02 |  4.06‡ |  8.94 |  4.39
mBERT |  23.51 |  20.02 |  12.41 |  6.77† |  20.09 |  6.89 |  3.75 |  3.54‡
Table 3: Calibration Errors
($\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$) for XLM-R
and mBERT on using different methods for calibration. We categorize the
methods into zero-shot i.e. the methods that do not use any target language
data to calibrate and few-shot for the methods that require some examples in
target language. Detailed results are in Table 9 of Appendix

###  3.1 Experimental Setup

Datasets We consider 4 multilingual classification datasets to study
calibration of MMLMs which include: i) The Cross-Lingual NLI Corpus (XNLI)
Conneau et al. (2018), ii) Multilingual Dataset for Causal Commonsense
Reasoning (XCOPA) Ponti et al. (2020), iii) Multilingual Amazon Reviews Corpus
(MARC) Keung et al. (2020) and, iv) Cross-lingual Adversarial Dataset for
Paraphrase Identification (PAWS-X) Yang et al. (2019). Statistics of these
datasets can be found in Table 5.

Training setup We consider two commonly used MMLMs in our experiments i.e.
Multilingual BERT (mBERT) Devlin et al. (2019), and XLM-RoBERTa (XLMR) Conneau
et al. (2020). mBERT is only available in the base variant with 12 layers and
for XLMR we use the large variant with 24 layers. We use English training data
to fine-tune the two MMLMs on all the tasks and evaluate the accuracies and
ECEs on the test data for different languages. For the few-shot case we use
the validation data in target languages to do continued fine-tuning (FSL) and
temperature scaling (Self-TS). Refer to Section A.3 in the Appendix for more
details.

###  3.2 Results

#### Out of Box Zero-Shot Calibration (OOB)

We first investigate how well calibrated MMLMs are on the languages unseen
during fine-tuning without applying any calibration techniques. As can be seen
in the Table 1, the average calibration error on languages other than English
(column 4) is almost always significantly worse than the errors on English
test data (column 3) for both mBERT and XLMR across the 4 tasks. Along with
the expected calibration errors across unseen languages we also report the
worst case ECE (in column 5), where we see 2$\times$ to 5$\times$ increase in
errors compared to English. The worst case performance is commonly observed
for low resource languages like Swahili or the languages that are
typologically diverse from English like Japanese. Consequently, the overall
calibration is worse on tasks like XCOPA and XNLI compared to PAWS-X and MARC,
as the former two have more diverse set of languages while the latter consists
of high resource languages only.

#### Factors Affecting Calibration

Next, we analyze which model-specific and typological features might influence
the out-of-box calibration across languages. For a given task and MMLM, we
compute Pearson correlation coefficients between the calibration errors and
three factors studied extensively in zero shot cross lingual transfer
literature which are:

i) SIZE: Logarithm of the pre-training data size (number of tokens) of a
language i.e. how well a language is represented in the pre-training corpus of
an MMLM Lauscher et al. (2020); Wu and Dredze (2020),

ii) SYN: We utilize the syntactic features provided by the URIEL project
Littell et al. (2017) to compute the syntactic similarity between the pivot
and target language as done in Lin et al. (2019).

iii) SWO: Finally we consider the sub-word overlap between the pivot and
target language as defined in Srinivasan et al. (2021). To compute SWO, first
vocabularies $V_{p}$ and $V_{t}$ are identified for the pivot and target
langauge respectively by tokenizing the wikipedias in the two languages and
getting rid of the tokens that appear less than 10 times in the corpora. The
subword overlap is then computed as :

$\text{SWO}=\frac{|V_{p}\cap V_{t}|}{|V_{p}\cup V_{t}|}$

.

In the case of XLMR, for all the tasks except MARC, we observe strong negative
correlations between ECE and the three factors mentioned above (Table 2),
meaning lower the amount of pre-training data present for a language or its
relatedness with English, the higher is the calibration error. Out of the
three, the correlations are more consistently (negatively) high with SYN. We
observe similar correlations albeit to a slightly lower extent for mBERT as
well (Table 6 in Appendix).

#### Improving Calibration

Now that we have identified miscalibration as an issue in MMLMs and factors
influencing the same, we seek to improve their calibration across languages.
We utilize the calibration methods described in Section 2.1, and report the
average calibration errors across the unseen languages in Table 3 for XNLI and
XCOPA datasets 222Refer to Table 7 in Appendix for results on MARC and PAWS-X.
Both zero-shot calibration methods (TS and LS) that only make use of English
data to calibrate can be seen to obtain substantially lower calibration errors
compared to out of box calibration (OOB) across all the tasks and MMLMs. Out
of the two, temperature scaling often results in bigger drops in ECE compared
to label smoothing with an exception of XCOPA dataset where label smoothing
performs much better. In majority of the cases the errors can be reduced even
further by considering the combination of the two techniques i.e. TS + LS.
Temperature scaling by design does not affect the accuracy of uncalibrated
models. In all our experiments we see that models trained with label smoothing
also obtain accuracies very close to their uncalibrated counterparts. Refer to
Appendix Table 8 for the exact accuracy numbers.

Next, we investigate if it is possible to reduce the calibration errors on a
language even further if we are allowed to collect a few-samples in that
language. We observe that when combined with label smoothing, both using few-
shot data to do temperature scaling (Self-TS + LS) or fine-tuning (FSL + LS),
often results in significant drops over the errors corresponding to the best
performing zero-shot calibration method. We do not use more than 2500 examples
in the target language in any of the experiments, and the number of examples
can be as low as 100 for XCOPA. For XNLI and XCOPA datasets, we observe that
Self-TS + LS performs better than FSL+LS for XLMR and the reverse is true for
mBERT. One advantage of using FSL over TS is that it can often result in
increase in accuracy in addition to the reducing the calibration errors.
However, we do observe an exception to this phenomenon for XCOPA where fine-
tuning on the 100 validation examples hurts the overall test accuracy of the
models. Hence, our general recommendation is to use FSL + LS for calibrating
the models if the amount of data that can be collected is not too low,
otherwise it might be more appropriate to use Self-TS + LS as learning just
one parameter ($T$) should be less prone to overfitting compared to the
weights of the entire network.

##  4 Conclusion

In this work we showed that MMLMs like mBERT and XLMR are miscalibrated in a
zero-shot cross lingual setting, with the calibration errors being even worse
on low resource languages and languages that are typologically distant from
the pivot language (often English). We then demonstrated the effectiveness of
standard calibration techniques for improving calibration across languages
both with and without collecting any new language-specific labelled data. We
recommend that researchers and practitioners consider, measure and report the
calibration of multilingual models while using them for scientific studies and
building systems. In future work, we aim to bridge the gap between zero-shot
and few-shot calibration methods by exploring unsupervised calibration methods
under domain shift Pampari and Ermon (2020); Park et al. (2020) that utilizes
unlabelled data in new domains to improve calibration. Investigating the cross
lingual calibration of MMLMs for tasks other than sentence classification like
Sequence Labelling Pan et al. (2017); Nivre et al. (2018) and Question
Answering Artetxe et al. (2020) is another natural extension of our work.

## Limitations

Our work focused on measuring and improving calibration of MMLMs across
different languages and tasks. The languages that we considered in our
experiments were the ones for which labelled test sets were available in the 4
multilingual benchmarks that we considered. The number of languages in these
benchmarks ranged from 6 in case of MARC to 15 in XNLI, covering mostly high
resource languages 333class 3 or above according to the hierarchy defined by
Joshi et al. (2020) with Swahili being the lowest resource language studied.
However, the MMLMs considered in this work supports around 100 languages many
of which are arguably even lower resource compared to Swahili. Hence, how well
the methods discussed in the paper work towards improving the calibration for
such languages needs to be explored but is limited by the current state of
multilingual benchmarking Ahuja et al. (2022).

Additionally, investigating the state of calibration across the languages for
the 4 tasks and 2 MMLMs for different hyper-parameters and random seeds
required a reasonably large amount of GPU resources (we used NVIDIA V100 and
P100 GPUs). However, the calibration methods that we describe in the paper can
work with little (for temperature scaling and few-shot learning) to no (for
label smoothing) additional compute over the standard model training.

## Ethics Statement

Our work deals with calibration of confidence predictions of classifiers
trained on top of pre-trained multilingual models. Having well calibrated
predictions is imperative for building robust NLP systems especially when
using them for security sensitive applications like Hate Speech Detection to
flag social media accounts Cuthbertson (2021), decision making in law
enforcement and fraud detection Metz and Satariano (2020), where extreme
actions should only be taken when we are confident about the system’s
prediction. However, the predicted confidences mean essentially nothing if the
model is miscalibrated, posing major risks in using such models. Through our
work we highlight that the commonly used multilingual models are highly
miscalibrated when used in a zero-shot setting for low resource and
typologically diverse languages. Additionally, we manage to substantially
improve the calibration of these models across languages, addressing this
linguistic disparity and boosting the reliability of such models.

## Acknowledgements

We sincerely thank the the reviewers for their valuable feedback about our
work. We would also like to thank Abhinav Kumar for brain-storming about the
work and his helpful suggestions.

## References

  * Ahuja et al. (2022) Kabir Ahuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2022.  Beyond static models and test sets: Benchmarking the potential of pre-trained models across tasks and languages.  In _Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP_ , pages 64–74, Dublin, Ireland. Association for Computational Linguistics.
  * Ansell et al. (2022) Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. 2022.  Composable sparse fine-tuning for cross-lingual transfer.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1778–1796, Dublin, Ireland. Association for Computational Linguistics.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the Cross-lingual Transferability of Monolingual Representations.  In _Proceedings of ACL 2020_.
  * Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017\.  Enhanced LSTM for natural language inference.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1657–1668, Vancouver, Canada. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of EMNLP 2018_ , pages 2475–2485.
  * Cuthbertson (2021) Anthony Cuthbertson. 2021.  Ai mistakes ‘black and white’ chess chat for racism.  _The Independent_.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _ICWSM_ , pages 512–515.
  * Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020.  Calibration of pre-trained transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 295–302, Online. Association for Computational Linguistics.
  * Deshpande et al. (2022) Neha Deshpande, Nicholas Farris, and Vidhur Kumar. 2022.  Highly generalizable models for multilingual hate speech detection.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017.  On calibration of modern neural networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1321–1330. PMLR.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Keung et al. (2020) Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020.  The multilingual amazon reviews corpus.  _CoRR_ , abs/2010.02573.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR (Poster)_.
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020.  From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4483–4499, Online. Association for Computational Linguistics.
  * Lecun et al. (1998) Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324.
  * Lin et al. (2019) Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019.  Choosing transfer languages for cross-lingual learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3125–3135, Florence, Italy. Association for Computational Linguistics.
  * Littell et al. (2017) Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017.  URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 8–14, Valencia, Spain. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Metz and Satariano (2020) Cade Metz and Adam Satariano. 2020.  An algorithm that grants freedom, or takes it away. _The New York Times_.
  * Müller et al. (2019) Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019.  When does label smoothing help? In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Nivre et al. (2018) Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018.  Universal dependencies 2.2.
  * Pampari and Ermon (2020) Anusri Pampari and Stefano Ermon. 2020.  Unsupervised calibration under covariate shift.  _CoRR_ , abs/2006.16405.
  * Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.  Cross-lingual name tagging and linking for 282 languages.  In _Proceedings of ACL 2017_ , pages 1946–1958.
  * Park et al. (2020) Sangdon Park, Osbert Bastani, James Weimer, and Insup Lee. 2020.  Calibrated prediction with covariate shift via unsupervised domain adaptation.  In _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , volume 108 of _Proceedings of Machine Learning Research_ , pages 3219–3229. PMLR.
  * Pereyra et al. (2017) Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. 2017.  Regularizing neural networks by penalizing confident output distributions.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, Online. Association for Computational Linguistics.
  * Ponti et al. (2021) Edoardo M. Ponti, Ivan Vulić, Ryan Cotterell, Marinela Parovic, Roi Reichart, and Anna Korhonen. 2021.  Parameter space factorization for zero-shot learning across tasks and languages.  _Transactions of the Association for Computational Linguistics_ , 9:410–428.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  XCOPA: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376, Online. Association for Computational Linguistics.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _AAAI spring symposium: logical formalizations of commonsense reasoning_ , pages 90–95.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019.  Social IQa: Commonsense reasoning about social interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.
  * Sarkar and KhudaBukhsh (2021) Rupak Sarkar and Ashiqur R. KhudaBukhsh. 2021.  Are chess discussions racist? an adversarial hate speech data set (student abstract).  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(18):15881–15882.
  * Srinivasan et al. (2021) Anirudh Srinivasan, Sunayana Sitaram, Tanuja Ganu, Sandipan Dandapat, Kalika Bali, and Monojit Choudhury. 2021.  Predicting the performance of multilingual nlp models.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu and Dredze (2020) Shijie Wu and Mark Dredze. 2020.  Are all languages created equal in multilingual BERT? In _Proceedings of the 5th Workshop on Representation Learning for NLP_ , pages 120–130, Online. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of EMNLP 2019_ , pages 3685–3690.
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019.  PAWS: Paraphrase adversaries from word scrambling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Measuring Calibration

#### Reliability Diagrams

are an effective way of visualizing the calibration of a classifier. To plot
these, we first predict the (maximum) confidence for each example in the test
set and group the data points into $M$ equally sized bins based on the
predicted confidence. For each bin, accuracy is computed (by considering label
with maximum confidence as prediction) and plotted on the y axis with
confidence being on the x-axis, as can be seen in Figure 1 (blue bars). For a
perfectly calibrated classifier the confidence of a bin should match with the
classifier’s accuracy on that bin i.e. the accuracy vs confidence curve should
lie on the line $y=x$ (red-dotted line in Figure 1). The gap between the two
is plotted (as red bars in Figure 1) to represent the calibration error of the
classifier.

#### Expected Calibration Error (ECE)

is defined as expected value of the difference between the confidence and
accuracy of the classifier’s predictions. In practice ECE of a classifier is
computed using a binning strategy as described in Guo et al. (2017) where the
confidence predictions (corresponding to the maximum confidence class) on the
$n$ test examples are grouped into $M$ uniform-sized bins, such that set of
examples belonging to the $m^{th}$ bin are denoted by $B_{m}$. Accuracy
($\text{acc}(B_{m})$) and average confidence ($\text{conf}(B_{m})$) for each
bin is computed and a weighted average of the differences between the two is
taken to obtain ECE.

$\texttt{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{n}|\text{acc}(B_{m})-\text{conf}(B_{m})|$

###  A.2 Datasets Description

1\. XNLI444https://github.com/facebookresearch/XNLI Conneau et al. (2018) is a
Natural Langauge Inference task where a premise and hypothesis are given and
the task is to predict if the hypothesis is entailed in premise, contradicts
the premise, or is neutral towards it, hence being a three way classification
problem. MultiNLI Williams et al. (2018) corpus which is available in English
is used as training set, and dev and validation sets are obtained by manually
translating crowd sourced English sentences for the task into 14 other
languages (Arabic, Bulgarian, German, Greek, Spanish, French, Hindi, Russian,
Swahili, Thai, Turkish, Urdu, Vietnamese and Chinese.)

2\. XCOPA555https://huggingface.co/datasets/xcopa Ponti et al. (2020) is a
multilingual benchmark for causal commonsense reasoning. It was obtained by
extending the dev and test sets of Choice of Plausible Alternatives
(COPA)666https://huggingface.co/datasets/super_glue Roemmele et al. (2011)
dataset to 11 typologically diverse languages. The task here is, given a
premise and two alternatives, predict which alternative has a causal
relationship with the premise. The original COPA dataset has only 400 training
examples in English, hence it is common to first train the model on Social-IQA
(SIQA)777https://huggingface.co/datasets/social_i_qa Sap et al. (2019) dataset
(which has around 33k training examples), and then fine-tune it on COPA, and
that’s the strategy we adopt in our experiments. SIQA is similar to COPA but
the questions or premise are defined in such a way that the possible answers
have social implications, and instead of two alternatives it has three. Out of
the 11 supported languages we experiment with 9 languages, ignoring Quechua
and Haitian Creole as both mBERT and XLM-R were not pre-trained on these
languages, leaving us with Greek, Indonesian, Italian, Swahili, Tamil, Thai,
Turkish, Vietnamese and Chinese.

3\. MARC888https://registry.opendata.aws/amazon-reviews-ml/ Keung et al.
(2020) is the multilingual Amazon product reviews corpus. We are given title,
body and category of the review and the task is to predict the corresponding
rating from 1 to 5. The corpus contains train, dev and test sets in six high
resource languages: English, German, Spanish, French, Japanese and Chinese. In
our experiments we only fine-tune using title and body text and ignore the
category information.

4\. PAWS-X999https://huggingface.co/datasets/paws-x Yang et al. (2019) is an
adversarial dataset for multilingual paraphrase detection. It was adapted from
PAWS dataset Zhang et al. (2019) by manually translating the dev and test sets
in English to six high resource languages (French, German, Spanish, Chinese,
Japanese and Korean). The task is, given a pair of sentences predict whether
the two are paraphrases of each other. For training the original English PAWS
dataset is used.

The statistics of all these datasets are provided in Table 5

###  A.3 Detailed Experimental Setup

For fine-tuning the model we do a grid search over the learning
rate($[1\text{e-}5,3\text{e-}5,5\text{e-}6,7\text{e-}6]$) and the number of
epochs ($[1,3,4,5]$), run each setting for 3 random seeds (1, 11 and 22) and
select the best hyper-parameter set corresponding to the average dev accuracy
on English data 101010For XCOPA we did use dev data in all languages for model
selection as we saw performance in English data to not necessarily correlate
well with performance on other languages.. The final set of hyperparameters
used for each dataset and MMLM are provided in Table 4. Apart from these we
use a batch size of 8 in all experiments. We use Adam optimizer Kingma and Ba
(2015) to train all of our models and LBFGS to learn the temperature parameter
while performing temperature scaling.

For computing ECE values for different tasks and languages we use $M=10$ i.e.
10 buckets. For label smoothing we set the smoothing parameter $\alpha=0.1$ in
all experiments and initialize the temperature $T=1.5$ while doing temperature
scaling. For few-shot cases we use
$\min(2500,|\mathcal{D}_{val}(\mathfrak{t},l)|)$ examples, where
$|\mathcal{D}_{val}(\mathfrak{t},l)|$ denotes the number of dev examples
available in task $\mathfrak{t}$ for language $l$, and use that to perform
continued fine-tuning or temperature scaling.

All the experiments were run on NVIDIA V100 and P100 GPUs with 32GB and 16GB
memory respectively. We use pre-trained models available in Hugging Face’s
Transformers library Wolf et al. (2020). For computing the calibration errors
as well as plotting the reliability diagrams we use the open source tool
Calibration Framework111111https://github.com/fabiankueppers/calibration-
framework. To encourage the research in this area we will make our code
public.

Dataset |  MMLM |  Learning Rate |  Epochs |  Few-Shot Learning Rate |  Few-Shot Epochs
---|---|---|---|---|---
XNLI |  XLM-R |  7e-6 |  3 |  5e-06 |  2
mBERT |  3e-5 |  3 |  3e-5 |  1
XCOPA |  XLM-R |  5e-6 (for SIQA) & 5e-6 (for COPA) |  4 (for S-IQA) & 10 (for COPA) |  1e-5 |  1
mBERT |  1e-5 (for S-IQA) & 3e-5 (for COPA) |  3 (for SIQA) & 10 (for COPA) |  3e-5 |  10
MARC |  XLM-R |  5e-6 |  3 |  5e-6 |  1
mBERT |  5e-6 |  3 |  5e-6 |  1
PAWS-X |  XLM-R |  7e-6 |  3 |  3e-5 |  1
mBERT |  1e-5 |  3 |  3e-5 |  1
Table 4: Final list of hyperparameters used for reporting results. Dataset |  Number of Languages |  Number of Labels |  Training Size |  Dev Size |  Test Size
---|---|---|---|---|---
XNLI (sub-sampled) |  15 |  3 |  40000 |  2500 |  2500
XCOPA |  9 |  2 |  33410 (S-IQA) + 500 (COPA) |  100 |  400
MARC (sub-sampled) |  6 |  5 |  40000 |  2500 |  5000
PAWS-X |  7 |  2 |  50000 |  2000 |  2000
Table 5: Dataset statistics for the 4 multilingual classification tasks that we study in our experiments. We use sub-sampled versions of XNLI and MARC and only use the first 40k examples in their training sets to reduce the compute overhead and making the scale of training data is consistent across all the tasks. For completeness we also run some preliminary experiments with using the entire data for fine-tuning XNLI and present the calibration errors in Figure 2. Dataset |  SIZE |  SYN |  SWO
---|---|---|---
XNLI |  -0.4 |  -0.38 |  -0.41
XNLI (wo th) |  -0.86 |  -0.88 |  -0.78
XCOPA |  -0.14 |  -0.22 |  0.07
MARC |  -0.5 |  -0.69 |  -0.86
PAWS-X |  -0.91 |  -0.91 |  -0.98
Table 6: Pearson correlation coefficient between the Expected Calibration Error (ECE) and SIZE, SYN, and SWO features of different languages in the test set for mBERT Dataset |  MMLM | Zero-Shot Calibration | Few-Shot Calibration
---|---|---|---
OOB |  TS |  LS |  TS + LS |  Self-TS |  Self-TS + LS |  FSL |  FSL + LS
MARC |  XLM-R |  9.65 |  4.22† |  7.93 |  4.45 |  3.55 |  3.36 |  2.51‡ |  2.58
mBERT |  11.11 |  6.14 |  5.96 |  4.46† |  4.62 |  4.71 |  2.12‡ |  3.01
PAWS-X |  XLM-R |  4.28 |  2.29† |  3.37 |  4.44 |  - |  - |  - |  -
mBERT |  10.33 |  5.64 |  5.58 |  5.36† |  - |  - |  - |  -
Table 7: Calibration Errors across tasks for XLM-R and mBERT on using different methods for calibration. For PAWS-X we only report numbers for Zero-Shot methods as the dev data and test data of the benchmark have sentences in common (even though the pairs are unique), hence we avoid using dev examples as few-shot in this case. Dataset |  MMLM | Zero-Shot Calibration | Few-Shot Calibration
---|---|---|---
OOB |  TS |  LS |  TS + LS |  Self-TS |  Self-TS + LS |  FSL |  FSL + LS
XNLI |  XLM-R |  74.9 |  74.9 |  74.6 |  74.6 |  74.9 |  74.6 |  79.4 |  79.3
mBERT |  56.7 |  56.7 |  57.5 |  57.5 |  56.7 |  57.5 |  60.6 |  61.2
XCOPA |  XLM-R |  74.3 |  74.3 |  75.1 |  75.1 |  74.3 |  75.1 |  67.2 |  69.5
mBERT |  54.5 |  54.5 |  54.4 |  54.4 |  54.5 |  54.4 |  53.0 |  52.9
MARC |  XLM-R |  57.7 |  57.7 |  57.6 |  57.6 |  57.7 |  57.6 |  59.4 |  59
mBERT |  42.9 |  42.9 |  42.6 |  42.6 |  42.9 |  42.6 |  49.6 |  49.2
PAWS-X |  XLM-R |  76.1 |  76.1 |  75.6 |  75.6 |  - |  - |  - |  -
mBERT |  80.4 |  80.4 |  81.1 |  81.1 |  - |  - |  - |  -
Table 8: Accuracy
($\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{Accuracy}(l)]$) for
XLM-R and mBERT on using different methods for calibration. Similar to Table
7, here again we skip few-shot calibration for PAWS-X due to the possible data
leakage.

Method |  $\texttt{ECE}(en)$ |  $\displaystyle\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$ |  $\displaystyle\max_{l\in\mathcal{L}^{\prime}}\texttt{ECE}(l)$
---|---|---|---
Zero-Shot Calibration
OOB |  7.32 |  13.34 |  19.07
TS |  2.02 |  6.74 |  11.81
LS |  3.2 |  6.93 |  12.1
TS + LS |  4.1 |  4.9 |  9.35
Few-Shot Calibration
Self-TS |  2.02 |  5.41 |  9.7
Self-TS + LS |  4.1 |  4.05 |  4.64
FSL |  7.32 |  7.67 |  9.23
FSL + LS |  3.2 |  4.37 |  5.73
((a)) Detailed results on XNLI with XLMR

Method |  $\texttt{ECE}(en)$ |  $\displaystyle\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$ |  $\displaystyle\max_{l\in\mathcal{L}^{\prime}}\texttt{ECE}(l)$
---|---|---|---
Zero-Shot Calibration
OOB |  5.44 |  12.34 |  45.15
TS |  2.51 |  6.29 |  37.25
LS |  4.51 |  10.42 |  38.36
TS + LS |  2.61 |  6.71 |  30.51
Few-Shot Calibration
Self-TS |  2.51 |  4.77 |  29.82
Self-TS + LS |  2.61 |  4.7 |  23.23
FSL |  5.44 |  3.14 |  4.28
FSL + LS |  2.61 |  2.55 |  4.26
((b)) Detailed results on XNLI with mBERT

Method |  $\texttt{ECE}(en)$ |  $\displaystyle\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$ |  $\displaystyle\max_{l\in\mathcal{L}^{\prime}}\texttt{ECE}(l)$
---|---|---|---
Zero-Shot Calibration
OOB |  14.54 |  20.01 |  29.33
TS |  12.31 |  15.95 |  24.04
LS |  9.66 |  5.47 |  9.42
TS + LS |  8.98 |  4.52 |  7.2
Few-Shot Calibration
Self-TS |  12.31 |  16.02 |  24.02
Self-TS + LS |  8.98 |  4.06 |  5.36
FSL |  14.54 |  8.93 |  14.92
FSL + LS |  9.66 |  4.4 |  5.74
((c)) Detailed results on XCOPA with XLMR

Method |  $\texttt{ECE}(en)$ |  $\displaystyle\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$ |  $\displaystyle\max_{l\in\mathcal{L}^{\prime}}\texttt{ECE}(l)$
---|---|---|---
Zero-Shot Calibration
OOB |  23.4 |  23.51 |  29.02
TS |  21.76 |  20.02 |  23.99
LS |  15.85 |  12.41 |  15.87
TS + LS |  11.28 |  6.77 |  8.93
Few-Shot Calibration
Self-TS |  21.76 |  20.1 |  24.12
Self-TS + LS |  11.28 |  6.89 |  9.01
FSL |  23.4 |  3.75 |  10.5
FSL + LS |  15.85 |  3.54 |  6.65
((d)) Detailed results on XCOPA with mBERT

Method |  $\texttt{ECE}(en)$ |  $\displaystyle\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$ |  $\displaystyle\max_{l\in\mathcal{L}^{\prime}}\texttt{ECE}(l)$
---|---|---|---
Zero-Shot Calibration
OOB |  7.15 |  9.65 |  13.45
TS |  2.75 |  4.22 |  5.8
LS |  5.36 |  7.93 |  10.66
TS + LS |  3.39 |  4.55 |  6.33
Few-Shot Calibration
Self-TS |  2.75 |  3.56 |  4.1
Self-TS + LS |  3.39 |  3.36 |  3.64
FSL |  7.15 |  2.51 |  3.51
FSL + LS |  5.36 |  2.59 |  3.28
((e)) Detailed results on MARC with XLMR

Method |  $\texttt{ECE}(en)$ |  $\displaystyle\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$ |  $\displaystyle\max_{l\in\mathcal{L}^{\prime}}\texttt{ECE}(l)$
---|---|---|---
Zero-Shot Calibration
OOB |  9.38 |  11.1 |  17.3
TS |  3.56 |  6.14 |  10.9
LS |  5.19 |  5.96 |  12.9
TS + LS |  3.70 |  4.47 |  10.1
Few-Shot Calibration
Self-TS |  3.56 |  4.62 |  8.34
Self-TS + LS |  3.70 |  4.71 |  7.50
FSL |  9.38 |  2.12 |  3.45
FSL + LS |  5.19 |  3.02 |  4.24
((f)) Detailed results on MARC with mBERT

Method |  $\texttt{ECE}(en)$ |  $\displaystyle\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$ |  $\displaystyle\max_{l\in\mathcal{L}^{\prime}}\texttt{ECE}(l)$
---|---|---|---
Zero-Shot Calibration
OOB |  1.93 |  4.28 |  5.88
TS |  0.81 |  2.29 |  3.39
LS |  4.83 |  3.37 |  3.85
TS + LS |  6.30 |  4.44 |  5.07
((g)) Detailed results on PAWS-X with XLMR

Method |  $\texttt{ECE}(en)$ |  $\displaystyle\mathop{\mathbb{E}}_{l\in\mathcal{L}^{\prime}}[\texttt{ECE}(l)]$ |  $\displaystyle\max_{l\in\mathcal{L}^{\prime}}\texttt{ECE}(l)$
---|---|---|---
Zero-Shot Calibration
OOB |  3.57 |  10.3 |  15.6
TS |  0.99 |  5.64 |  9.80
LS |  3.35 |  5.57 |  8.94
TS + LS |  5.27 |  5.36 |  7.40
((h)) Detailed results on PAWS-X with mBERT

Table 9: Detailed results on improving calibration for the 4 datasets and 2
MMLMs considered in our experiments Figure 2: Out-of-Box Calibration Errors
(ECE) for XLMR and mBERT trained on XNLI with 40k samples (sub-sampled) and
392k samples(full data). Even though the calibration is better with using the
entire data for training, the observed patterns about the models being mis-
calibrated on languages other than English, especially low resource languages
like Swahili, Thai and Urdu still hold true.

((a))

((b))

Figure 3: Variation in ECE as we use more and more data for calibrating using
Self-TS method across languages for XNLI dataset. As can be seen 500 samples
are sufficient to obtain low calibration errors.

((a))

((b))

((c))

Figure 4: Visualizing the correlations of ECE with SIZE, SYN and SWO for XLMR
fine-tuned on XNLI.


# Zero-Shot Point Cloud Registration

Weijie Wang1,2, Guofeng Mei2, Bin Ren1,3, Xiaoshui Huang4
Fabio Poiesi2, Luc Van Gool3, Nicu Sebe1, Bruno Lepri2,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
1University of Trento, Italy 2Fondazione Bruno Kessler, Italy
3 Computer Vision Laboratory, ETH Zurich 4Shanghai AI Laboratory, China

###### Abstract

Learning-based point cloud registration approaches have significantly
outperformed their traditional counterparts. However, they typically require
extensive training on specific datasets. In this paper, we propose ZeroReg,
the first zero-shot point cloud registration approach that eliminates the need
for training on point cloud datasets. The cornerstone of ZeroReg is the novel
transfer of image features from keypoints to the point cloud, enriched by
aggregating information from 3D geometric neighborhoods. Specifically, we
extract keypoints and features from 2D image pairs using a frozen pretrained
2D backbone. These features are then projected in 3D, and patches are
constructed by searching for neighboring points. We integrate the geometric
and visual features of each point using our novel parameter-free geometric
decoder. Subsequently, the task of determining correspondences between point
clouds is formulated as an optimal transport problem. Extensive evaluations of
ZeroReg demonstrate its competitive performance against both traditional and
learning-based methods. On benchmarks such as 3DMatch, 3DLoMatch, and ScanNet,
ZeroReg achieves impressive Recall Ratios (RR) of over 84%, 46%, and 75%,
respectively.

##  1 Introduction

Figure 1:  Generally, traditional methods use handcrafted features with high
generalization but low performance; learning-based methods achieve high-
performance training with the requirements for extensive 3D data. ZeroReg, in
contrast, requires no training in 3D data (compared to the learning-based),
which utilizes a 2D frozen backbone to achieve higher performance and robust
generalization in PCR (compared to the traditional approaches).

Point cloud registration (PCR) aims to estimate the rigid transformation
between a source and a target point cloud [2]. With the overwhelming
development of deep learning, the ability of deep features for point cloud
registration outperforms traditional hand-crafted counterparts [19, 52, 33].
As a result, PCR has also been effective in challenging scenarios such as low-
overlap [29]. However, these scenarios typically require extensive training on
specific datasets.

Learning-based methods can be divided into two categories: supervised and
unsupervised methods. Supervised methods utilize the transformation or
correspondence annotations as supervision to train neural networks for PCR
[12, 18, 56, 26, 39, 40]. Unsupervised methods leverage reconstruction and
contrastive learning to train neural networks [47, 22, 32, 16, 41]. Recent
unsupervised methods [59] can achieve performance comparable to supervised
ones, showing that excellent results can _also_ be attained without
annotations. Hence, we argue that point cloud can be aligned with high
generalization and accuracy without the requirement for training on point
cloud datasets.

Because RGB and depth information typically come paired, we can use these
pieces of information together to assist in PCR based on geometry, which is
proving promising [61, 24]. Recent studies have started to explore the image
information to establish cross-modal correspondences for PCR tasks [61, 22].
However, most of them primarily incorporate 2D semantic features, while
ignoring the geometric features between points. Moreover, all of them require
extra training, which is costly to deploy in novel scenarios.

In this paper, we propose a simple yet effective zero-shot approach for point
cloud registration, which we name ZeroReg. ZeroReg can identify keypoints
within 3D point cloud pairs and capture the overlapping regions between them
by matching keypoints in 3D space. Given the inherent unordered and irregular
nature of point clouds, relying solely on visual features for PCR is
inadequate. Thus, the comprehensive integration of geometric information into
2D visual information is non-trivial but promising. Our method combines the
visual features from a 2D frozen model with the geometric attributes of point
clouds to achieve PCR in a zero-shot manner (Fig. 1).

To achieve this, we first establish the coarse-level correspondences between
point cloud pairs from their posed RGB pairs. We adopt SuperPoint [18] to
detect the keypoints in images and map these keypoints onto the 3D space.
Second, we utilize the 2D frozen model to extract features for the keypoints
and utilize optimal transport to establish coarse-level correspondences among
the keypoints on the point cloud. Due to the sparsity in the point cloud data,
not each pixel can be associated with a corresponding point in 3D space. Thus,
to maximize the preservation of keypoints from the images onto the point
cloud, we use point clouds reconstructed from depth images with identical
poses to the original ones. After obtaining coarse-level keypoints, following
[57], we adopt the coarse-to-fine strategy. Then, we apply the K-Nearest-
Neighbours algorithm ($k$NN) [46] to associate keypoints with the neighbouring
points, constructing patches. These patches, together with their point-wise
visual features extracted from the 2D frozen pretrained backbone, fed into our
custom-designed parameter-free decoder to infuse geometric information into
visual features, thereby enhancing distinctiveness at the point level.
Finally, the task of establishing correspondences between point clouds is
formulated as an optimal transport problem, and we use the RANSAC [52] to
estimate the transformation. We evaluate ZeroReg on three benchmarks 3DMatch
(1.6K pairs) [60], 3DLoMatch (1.7K pairs) [29], ScanNet (26K pairs) [15].

Since our approach is zero-shot, it does not perform on par with the SOTA
learning-based approaches with sufficient training data. However, the
performance of ZeroReg overall surpasses that of traditional approaches and
even competes with some learning-based methods, indicating its potential for
pairwise registration tasks. To summarize, our contributions are as follows:

  * •

We propose ZeroReg that, to the best of our knowledge, is the first method
that conducts pairwise registration in a zero-shot manner, which eliminates
the need for training on the point cloud dataset.

  * •

We design a novel parameter-free geometry decoder that integrates geometric
information with visual features for refinement in order to enhance
performance. We also demonstrate that the extracted visual features can be
utilized for pairwise registration, resulting in improved performance.

  * •

We conduct extensive experiments on three benchmarks such as 3DMatch,
3DLoMatch, and ScanNet, and achieve competitive results, with a Recall Ratio
of 84% (3DMatch), 46% (3DLoMatch), and 75% (ScanNet).

##  2 Related Work

###  2.1 Learning-based point cloud registration

Based on the input source, current 3D point cloud registration methods can be
classified into single-modal and multi-modal. The former solely relies on the
geometry descriptors of the point cloud as input, while the latter utilizes
RGB-D images as visual descriptors, or both geometry and visual descriptors
complement each other to further enhance point cloud registration performance.

##### Single-modal.

Early point cloud registration methods utilized handcrafted feature extraction
e.g., surface normals [19, 52, 33], point coordinates [13], on point cloud
geometry for registration [1, 52]. For example, MCOV [13] is a covariance
descriptor to directly fuse the texture and shape features in point clouds.
PPF [1] and FFPH [52] adopt the geometric features calculated from the
distances and relative angles between points and surface normals. With the
advancement of deep learning, deep neural networks achieve impressive
improvements over handcrafted approaches with training. Most learning-based
methods are two-stage methods, which extract rotation-invariant features [16,
17] from point clouds, enabling to estimate feature matching between a pair of
sparse points by the Nearest Neighbour Matcher [57, 29, 50, 58], and then
adopting the robust transformation estimators, e.g., RANSAC [23], which is
still widely applied for registration task. Predator [29] samples keypoints to
predict the overlapping regions by incorporating the global context into the
local descriptors. Cofinet [57] is a keypoint-free method, which extracts
hierarchical correspondences to finish point matching in a coarse-to-fine
manner. GeoTrans [50] and RoITr [58] propose the rotation-invariant descriptor
using transformers [54]. GeDi [47] proposes a general and distinctive point
cloud local descriptor obtained from different domains to conduct
registration. Finally, PointNetLK [3] and its variant [30] are single-stage
methods, which modify the classic Lucas & Kanade (LK) algorithm and apply it
to registration tasks by directly regressing the exact position of points.
However, all of the above methods are solely based on the single-source point
cloud.

##### Multi-modal.

Compared to single-modal methods, incorporating the visual features from RGB
images into geometric features exhibits higher discriminative power [31, 27,
36, 37]. For instance, UR&R [22] employs a differentiable renderer to create
projections of point clouds. It then calculates the geometric consistency
between these projections and the raw RGB-D video frames for unsupervised
point cloud registration. BYOC [20] proposes learning visual and geometric
features from RGB-D videos without the need for ground truth pose or
correspondence supervision. Based on them, LLT [21] introduces a multi-scale
local linear transformation method that integrates visual and geometric
features from both RGB and depth images, which mitigate the visual
discrepancies resulting from alterations in geometry. PCR-CG [61] designs a
2D-3D module, projecting 2D deep color features into 3D geometry
representation. PointMBF [59] implements a multi-scale bidirectional fusion
network between RGB images and point clouds, producing more distinctive deep
features for correspondence estimation.

However, all existing learning-based methods currently require labor-intensive
training, which limits their practical applicability. Handcrafted methods,
though not requiring extensive training like deep learning, often struggle
with weak performance on challenging cases. In this paper, we present a novel
approach that eliminates the requirements for laborious training to conduct
point cloud registration.

###  2.2 Zero-shot learning for 3D point cloud

Zero-shot learning (ZSL) focuses on transferring knowledge from seen
categories to unseen categories. Unlike its widespread use in 2D images, the
application of ZSL in the 3D domain is limited. Some approaches have employed
ZSL to explore classification [9, 11, 8, 10] of 3D point clouds. Cheraghian et
al. [9] adapt PointNet [48] for object feature extraction while incorporating
additional semantic features from W2V [43] or GloVe [45] embeddings to reason
the category of unseen objects.

Subsequently, emerging techniques utilize zero-shot learning techniques for
the semantic segmentation of 3D point clouds [35, 7, 42, 38]. Some of them
[35, 42] utilize a general generative model to produce fake semantic features
to achieve the ZSL, with label supervision of the seen class to train.
Different from the generative way, TGP [38] learns geometric primitives to
transfer knowledge from seen to unseen categories, leveraging visual feature
extraction to align with the transductive context. SMKM [38] utilizes both
point clouds and images to align the visual and semantic spaces, enhancing the
comprehensive understanding of unseen objects in zero-shot learning.

To our knowledge, none of the ZSL methods is related to point cloud
registration. In this paper, we utilize SuperPoint [18] to extract 2D
keypoints from images and project them into their 3D frames. This facilitates
coarse-level matching in 3D space, followed by keypoint-to-patch expansion
using the K-Nearest-Neighbours ($k$NN). Lastly, we perform point-level
refinement with associated features.

##  3 Method

Figure 2:  _The pipeline of ZeroReg._ First, extract keypoints from 2D images
and project them onto 3D point clouds. Use a 2D image backbone to extract
pixel-level features and project them onto point-wise features. Perform
coarse-level matching using keypoint features and coordinates. Apply $k$NN to
group keypoints to patches for refinement. Employ a parameter-free decoder to
integrate geometric features for refinement in the fine-level matching
process. Finally, RANSAC is used to calculate the transformation between the
point cloud pairs.

In this section, we organize our methodology as follows: Sec. 3.1 outlines the
problem formulation. Sec. 3.2 encompasses keypoint detection and feature
extraction in the image, where feature extraction is further divided into
keypoint descriptors and normal point descriptors. Also, the 2D-3D mapping
module and our custom-designed parameter-free decoder are described in this
section. Finally, Sec. 3.3 details the correspondence matching part.

###  3.1 Problem formulation

The pipeline of ZeroReg is illustrated in Fig. 2. Given the point cloud pairs
$\bm{\mathcal{P}}=\\{\bm{p}_{i}\in\mathbb{R}^{3}|i=1,2,...,N\\}$ and
$\bm{\mathcal{Q}}=\\{\bm{q}_{j}\in\mathbb{R}^{3}|j=1,2,...,M\\}$, point cloud
registration (PCR) aims to calculate the transformation $\mathcal{T}\in SE(3)$
that align the point cloud pairs. Our approach bypasses the direct calculation
of $\mathcal{T}$ from point cloud pairs. It utilizes their posed RGB images
$\bm{\mathcal{I}_{P}}\in\mathbb{R}^{3\times H\times W}$ and
$\bm{\mathcal{I}_{Q}}\in\mathbb{R}^{3\times H\times W}$ to establish
correspondences between point cloud pairs in a zero-shot manner.

###  3.2 Keypoint detection and feature extraction

##### Keypoint detection.

As illustrated in Fig. 2, we utilize Superpoint [18] for keypoint detection of
RGB pairs in our default implementation. This part is agnostic and other
keypoint algorithms can be implemented using different formulations. For image
feature extraction, we chose the ResUNet-50 encoder [28] as the 2D image
backbone, which is initialized by a pre-trained model on ImageNet. We extract
keypoints and image descriptors for $\bm{\mathcal{I}_{p}}$ and
$\bm{\mathcal{I}_{q}}$ to obtain keypoint descriptors (denoted as
($\mathcal{F}_{\eta}^{\mathcal{\mathcal{I}_{P}}}$,
$\mathcal{F}_{\eta}^{\mathcal{\mathcal{I}_{Q}}}$) and image descriptors
($\mathcal{F}^{\mathcal{\mathcal{I}_{P}}}$,
$\mathcal{F}^{\mathcal{\mathcal{I}_{Q}}}$)

2D-to-3D mapping. To utilize 2D information in the 3D space, it is crucial to
establish a mapping bridge between 2D pixels and 3D coordinates. However, due
to the sparse nature of point clouds, we chose to utilize the point cloud
recovered from the depth image. In theory, point clouds and pixels have a one-
to-one correspondence $\mathcal{M}$, (e.g. $N=H\times W$). Thus, the final
projecting process is as Eq. 1:

$\small\begin{bmatrix}x_{i}\\\ y_{i}\\\
z_{i}\end{bmatrix}=[\mathbf{R}|\mathbf{T}]\cdot\left(Z(u,v)\cdot\mathbf{K}^{-1}\cdot\begin{bmatrix}v\\\
u\\\ 1\end{bmatrix}\right)$ (1)

where $u,v$ is the depth coordinate corresponding to the i-th point with
$(x_{i},y_{i},z_{i})$ 3D coordinate in the point cloud. $K$ is the intrinsic
of the camera, $Z(u,v)$ is the value of depth in the position of $(u,v)$,
$\mathbf{R}$ and $\mathbf{T}$ are the transformation of the pose. In practice,
various factors, such as inconsistent depths and variations in depth map
resolution, contribute to the loss of depth information, resulting in
$N<H\times W$. Finally, we obtain the keypoint descriptors (denoted as
$\mathcal{F}_{\eta}^{\mathcal{\mathcal{P}}}$,
$\mathcal{F}_{\eta}^{\mathcal{\mathcal{Q}}}$), and point features
($\mathcal{F}^{\mathcal{P}}$ $\mathcal{F}^{\mathcal{Q}}$).

##### Parameter-free decoder for refinement.

Depending solely on keypoints, a limited subset of the original point cloud,
may not provide enough correspondences for effective registration,
particularly in point clouds with low overlap. Additionally, while features
transferred from 2D images may miss key geometric details, the inherent
geometric structure of point clouds is rich in information.

Motivated by this insight, we develop a parameter-free refinement decoder,
which is based on the distance-based propagation strategy [49]. Its primary
function is to expands keypoints
($\mathcal{F}_{\eta}^{\mathcal{\mathcal{P}}}$,
$\mathcal{F}_{\eta}^{\mathcal{\mathcal{Q}}}$) in coarse correspondences into
patches comprising point groups ($\mathcal{P}_{\mathcal{G}}$,
$\mathcal{Q}_{\mathcal{G}}$), and their associated descriptors
($\mathcal{F}_{\mathcal{G}}^{\mathcal{\mathcal{P}}}$,
$\mathcal{F}_{\mathcal{G}}^{\mathcal{\mathcal{Q}}}$). Our decoder utilizes not
only the coordinate information of the points but also the eigenvalue
information induced by their surrounding neighbours.

Specifically, we first select $L$ neighbours in Euclidean space for each
point, using the $k$NN algorithm. For each point $\bm{p}_{i}$, we calculate
eigenvalue tuples
$\bm{\lambda}_{i}=(\lambda_{i}^{1},\lambda^{2}_{i},\lambda^{3}_{i})$ based on
the coordinates of $L$ neighbouring points. We then select the $K$-nearest
neighbours from the keypoints in both Euclidean and eigenvalue spaces for each
point $\bm{p}_{i}$. Let $\\{\bm{p}_{i_{1}},\cdots,\bm{p}_{i_{K}}\\}$ and
$\\{\bm{f}_{i_{1}},\cdots,\bm{f}_{i_{K}}\\}$ represent the K-nearest
neighbours and their corresponding features in Euclidean space, respectively.
Similarly, $\\{\bm{f}_{\bar{i}_{1}},\cdots,\bm{f}_{\bar{i}_{K}}\\}$ and
$\\{\bm{\lambda}_{\bar{i}_{1}},\cdots,\bm{\lambda}_{\bar{i}_{K}}\\}$ denote
the features and eigenvalues of the $K$-nearest neighbors in eigenvalue space.
After that, we implement a reverse distance-weighted average interpolation
method, leveraging these neighbours to recover missing point features, denoted
as $\bm{f}_{i}^{c}$ and $\bm{f}_{i}^{e}$ in Euclidean and eigenvalue spaces,
respectively. The interpolation is governed by the following:

$\displaystyle\bm{f}_{i}^{c}$
$\displaystyle{=}\frac{\sum_{k=1}^{K}w^{c}_{i_{k}}(\bm{p}_{i_{k}})\bm{f}_{i_{k}}}{\sum_{k=1}^{k}w^{c}_{i_{k}}(x)},w^{c}_{i_{k}}(\bm{p}_{i_{k}}){=}\frac{1}{\|\left(\bm{p}_{i},\bm{p}_{i_{k}}\right)\|},$
(2) $\displaystyle\bm{f}_{i}^{e}$
$\displaystyle{=}\frac{\sum_{k=1}^{K}w^{e}_{\bar{i}_{k}}(\bm{\lambda}_{\bar{i}_{k}})\bm{f}_{\bar{i}_{k}}}{\sum_{k=1}^{k}w^{c}_{\bar{i}_{k}}(x)},w^{e}_{\bar{i}_{k}}(\bm{\lambda}_{\bar{i}_{k}}){=}\frac{1}{\|\left(\bm{\lambda}_{i},\bm{\lambda}_{\bar{i}_{k}}\right)\|},$

where $\|\cdot\|$ is the $L_{2}$ norm. Finally, the refined feature
$\bm{f}_{i}$ for $\bm{p}_{i}$ is the concatenation of $\bm{f}_{i}^{c}$ and
$\bm{f}_{i}^{e}$. This enables us to acquire geometrically enhanced point-
level features, which are instrumental in identifying more accurate
correspondences for registration purposes.

###  3.3 Correspondence Prediction

ZeroReg adopts a coarse-to-fine strategy to compute correspondences between
points. The rationale is that the global ambiguity is resolved through
keypoint matching in the coarse-level stage. Subsequently, in the fine-level
stage, the integration of geometric features into the distance calculation
between point pairs enhances point-to-point matching.

##### Coarse-level correspondence.

After obtaining keypoint ($\mathcal{P}_{\eta}$, $\mathcal{Q}_{\eta}$) and
corresponding descriptors ($\mathcal{F}_{\eta}^{\mathcal{\mathcal{Q}}}$,
$\mathcal{F}_{\eta}^{\mathcal{\mathcal{Q}}}$). we employ a simple yet
effective method to compute coarse-level correspondences between them. To
enhance optimization stability, we conduct a similarity matrix between
$\mathcal{S}_{\eta}$ = $\Omega$($\mathcal{F}_{\eta}^{\mathcal{\mathcal{P}}}$,
$\mathcal{F}_{\eta}^{\mathcal{\mathcal{Q}}}$), where $\Omega$ is the log-space
optimal transport [53]. Then, we expand $\mathcal{S}_{\eta}$ with a new row
and a new column as $\mathcal{S^{\prime}}_{\eta}$ in [6] and calculate it as
Eq. 3:

$\mathcal{S}^{\prime}_{\eta}=\left[\begin{array}[]{cc}{\mathcal{F}_{\eta}^{\mathcal{P}}(\mathcal{F}_{\eta}^{\mathcal{Q}})^{T}}&\mathbf{z}\\\
\mathbf{z}^{T}&z\end{array}\right],\quad\mathbf{S}^{\prime}\in\mathbb{R}^{\left(n^{\prime}+1\right)\times\left(m^{\prime}+1\right)}$
(3)

where $z$ is the same learnable parameter. Next, we apply the Sinkhorn
Algorithm [14] to $\mathcal{S^{\prime}}$, aiming to find an optimal solution
for the optimal transport problem. Dropping the last row and last column, the
keypoint correspondence $\mathcal{C}_{\eta}$ is finally obtained from the
optimal process.

##### Fine-level correspondences.

After obtaining the coarse-level correspondences, we apply group-$k$NN to each
matched keypoints to extend groups of points
($\mathcal{P}_{\mathcal{G}}$,($\mathcal{P}_{\mathcal{G}}$) and associated
grouped descriptors ($\mathcal{F}_{\mathcal{G}}^{\mathcal{\mathcal{P}}}$,
$\mathcal{F}_{\mathcal{G}}^{\mathcal{\mathcal{Q}}}$). This process identifies
their nearest points, subsequently forming patches. Based on patches from
coarse correspondences, following [50], we calculate the fine-level
correspondences as $C=\bigcup_{i=1}^{N_{\mathcal{G}}}C_{\eta_{i}}$.
$N_{\mathcal{G}}$ is the number of patches. Finally, RANSAC [52] can directly
compute the transformation with fine-level correspondences for registration.

##  4 Experiments

In this section, our experiments are organized as follows. First, we detail
the experimental setup, including datasets, implementation, and evaluation
metrics in Sec. 4.1. To evaluate the effectiveness of different methods, we
utilize three benchmarks in our experiments, _3DMatch_ and _3DLoMatch_
benchmark in Sec. 4.2, _ScanNet_ benchmark in Sec. 4.3. Finally, the ablation
studies in Sec. 4.4.

###  4.1 Experiments Setup

Dataset. (1) 3DMatch and 3DLoMatch datasets [60], an indoor dataset containing
62 scenes among which 46/8/8 scenes are used for training/validation/testing,
respectively. We focus solely on the test data, which comprises 1623 point
cloud fragments (from 3DMatch) and 1781 point cloud fragments (from 3DLoMatch)
and their transformation matrix, preprocessed by [29]. Their overlapping
regions exceed 30% and range from 10% to 30%, respectively. (2) ScanNet-v1
[15] comprises 1045/156/312 scenes for training, validation, and testing,
respectively. We generate view pairs in the test set by selecting image pairs
with a 20-frame interval, resulting in 26K pairs.

Implementation. ZeroReg is implemented in PyTorch [44] and Open3D [63]. For
testing, we use RANSAC with Open3D implementation to estimate transformation
between point clouds. All the experiments are conducted on a single Tesla V100
GPU. We resize RGB images at 320$\times$240\. _Coarse-Matching:_ For the
SuperPoint [18], we employ the Keypoint Threshold $\eta$=0.005, and the Max
Keypoints $\alpha=1024$. _Fine-Matching:_ We set the point limit $\beta$=64
for the limit of neighbouring points. In the optimal transport part, we
utilize the Sinkhorn algorithm [14], iterating 20 times for matching.

Evaluation metrics. Following the [57], we evaluate the results on several
metrics. (1) _Registration Recall_ (RR): Measures the percentage of successful
pairwise registrations whose error transformation estimated by RANSAC is
smaller than a certain threshold, e.g., RMSE $<$ 20cm, compared to ground
truth. (2) _Inlier Ratio (IR)_ is the ratio of correspondences whose residual
error in the geometric space is below a defined threshold of $\sigma=10cm$,
based on the ground truth transformation. (3) _Rotation Error (RE)_ , and (4)
_Translation Error (TE)_ , the computation of average TE and RE is limited to
these successfully registered pairs, as failed registrations can yield poses
that deviate significantly from the ground truth, rendering the error metrics
unreliable. See more details in the supplementary material.

###  4.2 Performance on 3DMatch and 3DLoMatch

Our competitors on 3DMatch & 3DLoMatch can be categorized into two categories.
Traditional: FGR [62], SM [34], RANSAC [52]. We employ FPFH [52] descriptors
of point cloud pairs. PointDSC [5] is an outlier removal method, we put it
into the traditional methods part for comparative analysis. Learning-Based:
3DSN [25], FCGF [12], D3Feat [4], SpinNet [2], Predator [29], CoFiNet [57],
YOHO [55].

####  4.2.1 Traditional methods comparision

Since traditional methods also do not require training, to ensure that ZeroReg
is meaningful, it must outperform traditional handcrafted descriptors (e.g.,
FPFH) in terms of performance. Thus, we first compare ZeroReg with traditional
methods. Following [5], we align 5cm-voxel-downsampled and extract their FPFH
descriptor with Open3D. As indicated in Tab. 1, ZeroReg attains the highest
_RR_ on 3DMatch and 3DLoMatch, as well as competitive _RE_ and _TE_. ZeroReg
exhibits stronger generalization than traditional methods on novel scenes,
especially on 3DLoMatch.

Quantative and quality analysis. On 3DMatch, PointDSC achieves the best _RE_
and _TE_ , demonstrating its better registration precision. ZeroReg attains
competitive results in both _RE_ and _TE_ when compared with “RANSAC-100k ICP”
since it also incorporates RANSAC to compute the final transformation.
Nonetheless, concerning _Registration Recall_ , ZeroReg significantly
outperforms both PointDSC and “RANSAC-100k ICP” by 8.91% and 10.18%
respectively on 3DMatch. We also provide the visualization as Fig. 3 shows. On
3DLoMatch, all the methods exhibit a decrease in three metrics. Both “RANSAC-
Based” and PointDSC experience significant performance drops, which proves
unsuitable for extremely low-overlap scenarios. The main reason is these
methods of sampling interest points within FPFH descriptors, which are
scattered globally throughout the entire point cloud. In low-overlap cases,
there is a limited number of interest points sampled, resulting in substantial
outliers and a drop in performance.

It is noteworthy that the reduction in _Resigitration Recall_ for PointDSC, at
63.25%, is more substantial than that of RANSAC, which stands at 61.20%. It
seems like PointDSC is overfitting on 3DMatch. In contrast, ZeroReg
demonstrates a 40.27% decrease from 3DMatch to 3DLoMatch, surpassing the
second position (RANSAC ICP) by 16.21%. This reduction of 44.99% indicates its
superior robustness in low-overlap scenarios. In such cases, RANSAC exhibits
inferior performance, as shown in Fig. 4. Quantitative and qualitative
analyses reveal that ZeroReg exhibits better robust generalization to novel
scenes compared to traditional methods.

Table 1: Quantitative results on 3DMatch and 3DLoMatch, compared with methods
using handcrafted FPFH descriptors: best performances are in bold, second-best
underlined.

Method(FPFH) | _3DMatch_ | _3DLoMatch_
---|---|---
RE$\downarrow$ | TE $\downarrow$ | RR(%)$\uparrow$ | RE$\downarrow$ | TE$\downarrow$ | RR(%)$\uparrow$
FGR [62] | 4.08 | 9.83 | 39.32 | 5.43 | 10.44 | 23.42
SM [34] | 2.94 | 8.15 | 52.92 | 6.10 | 13.00 | 6.79
RANSAC-1k [52] | 5.16 | 13.65 | 38.25 | 4.20 | 16.17 | 4.20
RANSAC-10k [52] | 4.35 | 11.79 | 57.43 | 7.13 | 15.04 | 11.50
RANSAC-100k [52] | 3.55 | 10.04 | 71.57 | 6.10 | 13.94 | 21.48
RANSAC-100k+ICP[51] | 2.62 | 7.42 | 74.23 | 3.54 | 10.07 | 28.80
PointDSC [5] | 2.07 | 6.57 | 75.50 | 4.37 | 10.61 | 27.74
ZeroReg | 2.19 | 7.75 | 84.41 | 3.31 | 10.78 | 46.43

Figure 3: Registration visualization results on 3DMatch.

Figure 4: Registration visualization results on 3DLoMatch.

Why ZeroReg achieves high _RR_? As noted by [57], directly detecting keypoints
on the point cloud is suboptimal, the strategy of ZeroReg avoid directly doing
it. In contrast, ZeroReg detects keypoints across the 2D image pairs and
projects them on 3D point cloud pairs as 3D keypoints. One immediate benefit
is the utilization of more robust visual features to distinguish, thereby
obtaining more reliable keypoints on point cloud pairs. Moreover, the number
of keypoints, a pivotal factor in successful pairwise registration, is
positively correlated with image size (See Tab. 5). As Fig. 5 shows, even in
3DLoMatch, there is a substantial number of keypoints persist in the source
and target pairs (_row 1-3_). In low-overlap cases, an increased number of
keypoints enhances the probability of successful matches, thereby enhancing
the _Registration Rate_. Nevertheless, in such circumstances, relying solely
on a limited number of key points for matching is unreliable. As shown red box
in Fig. 5, there are an exceedingly limited number of keypoints. And these
keypoints are mismatched, leading to failure cases (_row 4)_. Also, in
extremely low-overlap cases, registration can fail when no keypoints are
present on the point cloud, i.e., _row 5_.

####  4.2.2 Learning-Based Methods Comparision

As ZeroReg does not need additional training, it is not necessary to perform
on par with the SOTA learning-based approaches. However, given the premise of
known ZeroReg’s generalization, it is necessary to evaluate the upper limit of
its performance compared to learning-based methods. We assess the performance
ceiling of ZeroReg with learning-based methods. For a more comprehensive
evaluation, we follow [57] and report performance with different numbers of
sampled interest points in terms of _RR_ and _IR_. ZeroReg overall exceeds
3DSN, FCGF, and D3Feat by a large margin on 3DMatch and 3DLoMatch, which
indicates the potential to PCR in a zero-shot manner.

Table 2: Quantitative results on 3DMatch & 3DLoMatch with a different number
of sampling points. Best performance is highlighted in bold while the second
best is marked with an underline.

| 3DMatch | 3DLoMatch
---|---|---
# Samples | 5000 | 2500 | 1000 | 500 | 250 | 5000 | 2500 | 1000 | 500 | 250
|  Registration Recall (%) $\uparrow$
3DSN [25] | 78.4 | 76.2 | 71.4 | 67.6 | 50.8 | 33.0 | 29.0 | 23.3 | 17.0 | 11.0
FCGF [12] | 85.1 | 84.7 | 83.3 | 81.6 | 71.4 | 40.1 | 41.7 | 38.2 | 35.4 | 26.8
D3Feat [4] | 81.6 | 84.5 | 83.4 | 82.4 | 77.9 | 37.2 | 42.7 | 46.9 | 43.8 | 39.1
SpinNet [2] | 88.8 | 88.0 | 84.5 | 79.0 | 69.2 | 58.2 | 56.7 | 49.8 | 41.0 | 26.7
Predator [29] | 89.0 | 89.9 | 90.6 | 88.5 | 86.6 | 59.8 | 61.2 | 62.4 | 60.8 | 58.1
CoFiNet [57] | 89.3 | 88.9 | 88.4 | 87.4 | 87.0 | 67.5 | 66.2 | 64.2 | 63.1 | 61.0
YOHO [55] | 90.8 | 90.3 | 89.1 | 88.6 | 84.5 | 65.2 | 65.5 | 63.2 | 56.5 | 48.0
ZeroReg | 84.2 | 83.6 | 84.2 | 84.4 | 83.8 | 46.0 | 46.3 | 46.2 | 46.4 | 46.3
|  Inlier Ratio (%)
3DSN [25] | 36.0 | 32.5 | 26.4 | 21.5 | 16.4 | 11.4 | 10.1 | 8.0 | 6.4 | 4.8
FCGF [12] | 56.8 | 54.1 | 48.7 | 42.5 | 34.1 | 21.4 | 20.0 | 17.2 | 14.8 | 11.6
D3Feat [4] | 39.0 | 38.8 | 40.4 | 41.5 | 41.8 | 13.2 | 13.1 | 14.0 | 14.6 | 15.0
SpinNet [2] | 48.5 | 46.2 | 40.8 | 35.1 | 29.0 | 25.7 | 23.7 | 20.6 | 18.2 | 13.1
Predator [29] | 58.0 | 58.4 | 57.1 | 54.1 | 49.3 | 26.7 | 28.1 | 28.3 | 27.5 | 25.8
CoFiNet [57] | 49.8 | 51.2 | 51.9 | 52.2 | 52.2 | 24.4 | 25.9 | 26.7 | 26.8 | 26.9
YOHO [55] | 64.4 | 60.7 | 55.7 | 46.4 | 41.2 | 25.9 | 23.3 | 22.6 | 18.2 | 15.0
ZeroReg | 52.5 | 52.6 | 52.7 | 53.1 | 52.9 | 24.6 | 24.9 | 25.0 | 24.9 | 25.1

Figure 5: Keypoints visualization and registration results. The red box shows
the failure cases.

Registraition recall. It is worth mentioning that, in addition to FPFH, our
method consistently outperforms learning-based descriptors trained on 3DMatch,
such as FCGF, in most settings. Exceptionally good results are achieved in all
scenarios except when the sample point is set to _5000_ and _2500_. ZeroReg
achieves performance situated between that of SpinNet and Predator in terms of
_RR_ on 3DMatch and 3DLoMatch. In the case of “ _Samples=250 & =500_”, we
surpass SpinNet on both benchmarks. Moreover, we even compete with YOHO when “
_Samples=250_ ”. As the decrease of “ _Sample_ ”, the performance of all
learning-based methods exhibits a downward trend, while ZeroReg’s performance
remains stable overall, displaying minimal performance fluctuation. It
indicates that ZeroReg is not sensitive to the sampling point number.
Moreover, keypoints exhibit strong discriminativeness in contrast to other
ordinary points, irrespective of the quantity of sampling points. Thus, in the
case of a low sampling point, ZeroReg is more promising.

Inlier ratio. ZeroReg achieves significant performance in terms of selecting
inlier. As Tab. 2, most learning-based methods decrease performance with fewer
sampled points. They require substantial sampling points for optimal results,
while ZeroReg holds the most stable performance in terms of _IR_ as well as
_RR_. The variance of _RR_ /_IR_ is only 0.08/0.04 on 3DMatch and 0.01/0.02 on
3DLoMatch. Whereas, the most stable learning-based method, CoFiNet’s variance
is 0.80/5.38 on 3DMatch and 0.82/0.88 on 3DLoMatch. The capability of ZeroReg
to occupy inliers can be attributed to the high confidence in the selection of
keypoints. Even with an abundance of sampled points, key points consistently
remain included. Similar to the _RR_ aspect, YoLo demonstrates sensitivity to
sampled points, suffering from a drastic decline, especially on 3DLoMatch.

Table 3: Quantitative results on ScanNet with traditional methods with FPFH
descriptors. Best performance is highlighted in bold while the second best is
marked with an underline.

Method |  RE $\downarrow$ |  TE $\downarrow$ |  RR [%] $\uparrow$
---|---|---|---
FGR [62] | 5.95 | 11.78 | 41.42
SM [34] | 3.94 | 11.14 | 57.06
RANSAC-1k [52] | 5.11 | 12.85 | 55.91
RANSAC-10k [52] | 4.60 | 12.09 | 58.65
RANSAC-100k [52] | 4.42 | 11.89 | 59.06
RANSAC-100k ICP [51] | 3.22 | 10.66 | 59.90
PointDSC [5] | 4.14 | 11.51 | 59.39
ZeroReg | 4.17 | 11.65 | 75.13

###  4.3 Performance on ScanNet

For a comprehensive evaluation, we evaluate ZeroReg on ScanNet, which is a
high-overlap dataset with 20-frame intervals but, in terms of quantity, has
26K pairs that are sufficient to evaluate the performance of algorithms
compared to 3DMatch and 3DLoMatch. Our competitors focus on traditional
methods. As indicated in Tab. 3, the ZeroReg achieves the highest
_registration recall_ with a score of 75.13%, surpassing the second-best
“RANSAC-100k ICP” of 59.90%. This illustrates the strong generalization and
robustness of ZeroReg, indicating that it can effectively perform registration
in most cases, even in novel scenes. Fig. 6 is visualization results. In terms
of _RE_ and _TE_ , “RANSAC-100k ICP” achieves the best, attributed to the
refinement by ICP, resulting in higher registration accuracy. PointDSC suffers
from significant drops on ScanNet compared to 3DMatch. The reason for the
decline is that PointDSC conducted training on 3DMatch, the low generalization
performance on ScanNet, similar to 3DLoMatch.

###  4.4 Ablation studies

In this section, we conduct ablation studies on the main components of
ZeroReg, i.e., keypoint detection and feature refinement decoder, as well as
the impact of image size on 3DMatch and 3DLoMatch.

Figure 6: Registration visualization results on ScanNet.

Keypoint detection and decoder for refinement. Tab. 4 demonstrates the impact
of these two main modules on the registration performance. As can be seen,
using either of these two modules (_KP and Decoder_) leads to substantial _RR_
and _IR_ , and using both yields the best results. In summary, omitting the
_KP_ part would significantly decrease the model’s performance, as evidenced
by decreased _RR_ on 3DMatch and 3DLoMatch. _KP_ is crucial for successful
pairwise registration. Utilizing the _Parameter-free Decoder_ module for
feature refinement would further boost performance, owing to the integration
of geometric features on the basis of visual features. Overall, ZeroReg
achieves its best results when both keypoint detection and feature refinement
are utilized, demonstrating their synergistic effect in enhancing pairwise
registration.

Image size. The image size impacts the number of key points, thereby affecting
the number of matched pairs in both the coarse and refined stages as Tab. 5
shows. We remove the decoder part to verify the impact of image size on the
keypoint number. We compared two settings: 320$\times$240 and 640$\times$480\.
It demonstrates that changing the image size notably improves _RR_ and the
quantity of keypoints. As the image size increases, the number of pixels
increases accordingly. Consequently, the number of keypoints also increases.
This results in a significant number of potential correspondences that can be
successfully matched within the point cloud pairs. However, the larger
size(640$\times$480) results in longer processing times, which is
approximately 3.2 times longer to process than the smaller
size(320$\times$240). It is a trade-off between efficiency and performance.

Table 4: The impact of the keypoint detection and feature refinement decoder
on different metrics of ZeroReg (“Samples=500”).

# Samples=500 |  | _3DMatch_ |
---|---|---|---
Module |  RE $\downarrow$ |  TE $\downarrow$ |  _RR_ (%) $\uparrow$
$\mathbf{w/o}$ KP | 2.53 | 8.23 | 75.23
$\mathbf{w/o}$ Decoder | 2.18 | 7.80 | 82.99
ZeroReg | 2.19 | 7.75 | 84.40
# Samples=500 |  | _3DLoMatch_ |
Module |  RE $\downarrow$ |  TE $\downarrow$ |  _RR_ [%] $\uparrow$
$\mathbf{w/o}$ KP | 5.98 | 12.32 | 29.23
$\mathbf{w/o}$ Decoder | 3.44 | 11.12 | 44.24
ZeroReg | 3.37 | 10.78 | 46.40

Table 5: Ablation studies on Image Size, _2D Keypoint_ denotes the quantity of
keypoint in image.

| _3DMatch_ |  |
---|---|---|---
ImageSize |  RE $\downarrow$ |  TE $\downarrow$ |  _RR_ [%] $\uparrow$ | _2D Keypoint_ |  Time(s) $\downarrow$
Smaller | 2.18 | 7.80 | 82.99 | $200\pm 50$ | 3.1
Larger | 2.18 | 7.78 | 84.66 | $400\pm 50$ | 11.2
| _3DLoMatch_ |  |
ImageSize |  TE $\downarrow$ |  RE  $\downarrow$ |  _RR_ [%] $\uparrow$ | _Keypoint_ |  Time(s) $\downarrow$
Smaller | 11.31 | 3.31 | 44.58 | $200\pm 50$ | 3.1
Larger | 11.04 | 3.32 | 49.07 | $400\pm 50$ | 11.2

##  5 Conclusion

In this paper, we propose ZeroReg a novel zero-shot pairwise registration
method, which does not require training on point cloud datasets. Our approach
utilizes an existing 2D visual backbone to directly search for reliable
keypoints on 3D point cloud pairs and effectively capture overlapping regions.
Additionally, we custom-designed a parameter-free decoder for feature
refinement to integrate 2D visual features and geometric information. The
fusion of features would exhibit distinctiveness during matching, further
enhancing performance in pairwise registration. Extensive experiments on
different benchmarks show that our method not only surpasses traditional
approaches but also competes favorably with learning-based methods. Our method
expands upon existing traditional and learning-based approaches, which
suggests a new potential research direction for pairwise direction.

## References

  * Aiger et al. [2008] Dror Aiger, Niloy J Mitra, and Daniel Cohen-Or.  4-points congruent sets for robust pairwise surface registration.  In _ACM SIGGRAPH 2008 papers_ , pages 1–10. 2008.
  * Ao et al. [2021] Sheng Ao, Qingyong Hu, Bo Yang, Andrew Markham, and Yulan Guo.  Spinnet: Learning a general surface descriptor for 3d point cloud registration.  In _CVPR_ , 2021.
  * Aoki et al. [2019] Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey.  Pointnetlk: Robust & efficient point cloud registration using pointnet.  In _CVPR_ , pages 7163–7172, 2019.
  * Bai et al. [2020] Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai.  D3feat: Joint learning of dense detection and description of 3d local features.  In _CVPR_ , pages 6359–6367, 2020.
  * Bai et al. [2021] Xuyang Bai, Zixin Luo, Lei Zhou, Hongkai Chen, Lei Li, Zeyu Hu, Hongbo Fu, and Chiew-Lan Tai.  Pointdsc: Robust point cloud registration using deep spatial consistency.  In _CVPR_ , pages 15859–15869, 2021.
  * Busam et al. [2015] Benjamin Busam, Marco Esposito, Simon Che’Rose, Nassir Navab, and Benjamin Frisch.  A stereo vision approach for cooperative robotic movement therapy.  In _ICCV workshop_ , pages 519–527, 2015.
  * Chen et al. [2023] Runnan Chen, Xinge Zhu, Nenglun Chen, Wei Li, Yuexin Ma, Ruigang Yang, and Wenping Wang.  Bridging language and geometric primitives for zero-shot point cloud segmentation.  In _ACMMM_ , pages 5380–5388, 2023.
  * Cheraghian et al. [2019a] Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson.  Mitigating the hubness problem for zero-shot learning of 3d objects.  _BMVC_ , 2019a.
  * Cheraghian et al. [2019b] Ali Cheraghian, Shafin Rahman, and Lars Petersson.  Zero-shot learning of 3d point cloud objects.  In _MVA_ , pages 1–6. IEEE, 2019b.
  * Cheraghian et al. [2020] Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson.  Transductive zero-shot learning for 3d point cloud classification.  In _WACV_ , pages 923–933, 2020.
  * Cheraghian et al. [2022] Ali Cheraghian, Shafin Rahman, Townim F Chowdhury, Dylan Campbell, and Lars Petersson.  Zero-shot learning on 3d point cloud objects and beyond.  _IJCV_ , 130(10):2364–2384, 2022.
  * Choy et al. [2019] Christopher Choy, Jaesik Park, and Vladlen Koltun.  Fully convolutional geometric features.  In _ICCV_ , pages 8958–8966, 2019.
  * Cirujeda et al. [2014] Pol Cirujeda, Xavier Mateo, Yashin Dicente, and Xavier Binefa.  Mcov: a covariance descriptor for fusion of texture and shape features in 3d point clouds.  In _3DV_ , pages 551–558. IEEE, 2014.
  * Cuturi [2013] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  _Advances in neural information processing systems_ , 26, 2013.
  * Dai et al. [2017] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _CVPR_ , 2017.
  * Deng et al. [2018a] Haowen Deng, Tolga Birdal, and Slobodan Ilic.  Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors.  In _CVPR_ , 2018a.
  * Deng et al. [2018b] Haowen Deng, Tolga Birdal, and Slobodan Ilic.  Ppfnet: Global context aware local features for robust 3d point matching.  In _CVPR_ , pages 195–205, 2018b.
  * DeTone et al. [2018] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In _CVPR_ , pages 224–236, 2018.
  * Drost et al. [2010] Bertram Drost, Markus Ulrich, Nassir Navab, and Slobodan Ilic.  Model globally, match locally: Efficient and robust 3d object recognition.  In _CVPR_ , pages 998–1005. Ieee, 2010.
  * El Banani and Johnson [2021] Mohamed El Banani and Justin Johnson.  Bootstrap your own correspondences.  In _ICCV_ , pages 6433–6442, 2021.
  * El Banani and Johnson [2022] Mohamed El Banani and Justin Johnson.  Improving rgb-d point cloud registration by learning multi-scale local linear transformation.  In _ECCV_ , 2022.
  * El Banani et al. [2021] Mohamed El Banani, Luya Gao, and Justin Johnson.  Unsupervisedr&r: Unsupervised point cloud registration via differentiable rendering.  In _CVPR_ , pages 7129–7139, 2021.
  * Fischler and Bolles [1981] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  _Communications of the ACM_ , 24(6):381–395, 1981.
  * Giuliari et al. [2022] Francesco Giuliari, Geri Skenderi, Marco Cristani, Yiming Wang, and Alessio Del Bue.  Spatial commonsense graph for object localisation in partial scenes.  In _CVPR_ , pages 19518–19527, 2022.
  * Gojcic et al. [2019] Zan Gojcic, Caifa Zhou, Jan D Wegner, and Andreas Wieser.  The perfect match: 3d point cloud matching with smoothed densities.  In _CVPR_ , pages 5545–5554, 2019.
  * Gojcic et al. [2020] Zan Gojcic, Caifa Zhou, Jan D Wegner, Leonidas J Guibas, and Tolga Birdal.  Learning multiview 3d point cloud registration.  In _CVPR_ , pages 1759–1769, 2020.
  * Hatem et al. [2023] Ahmed Hatem, Yiming Qian, and Yang Wang.  Point-tta: Test-time adaptation for point cloud registration using multitask meta-auxiliary learning.  In _ICCV_ , pages 16494–16504, 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Huang et al. [2021] Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler.  Predator: Registration of 3d point clouds with low overlap.  In _CVPR_ , pages 4267–4276, 2021.
  * Huang et al. [2020] Xiaoshui Huang, Guofeng Mei, and Jian Zhang.  Feature-metric registration: A fast semi-supervised approach for robust point cloud registration without correspondences.  In _CVPR_ , pages 11366–11374, 2020.
  * Huang et al. [2022] Xiaoshui Huang, Wentao Qu, Yifan Zuo, Yuming Fang, and Xiaowei Zhao.  Imfnet: Interpretable multimodal fusion for point cloud registration.  _IEEE Robotics and Automation Letters_ , 7(4):12323–12330, 2022.
  * Insafutdinov and Dosovitskiy [2018] Eldar Insafutdinov and Alexey Dosovitskiy.  Unsupervised learning of shape and pose with differentiable point clouds.  _Advances in neural information processing systems_ , 31, 2018.
  * Johnson and Hebert [1999] Andrew E Johnson and Martial Hebert.  Using spin images for efficient object recognition in cluttered 3d scenes.  _IEEE TPAMI_ , 21(5):433–449, 1999.
  * Leordeanu and Hebert [2005] Marius Leordeanu and Martial Hebert.  A spectral technique for correspondence problems using pairwise constraints.  In _ICCV_ , pages 1482–1489, 2005.
  * Liu et al. [2021] Bo Liu, Shuang Deng, Qiulei Dong, and Zhanyi Hu.  Language-level semantics conditioned 3d point cloud segmentation.  _arXiv preprint arXiv:2107.00430_ , 2021.
  * Liu et al. [2023a] Jiuming Liu, Guangming Wang, Zhe Liu, Chaokang Jiang, Marc Pollefeys, and Hesheng Wang.  Regformer: An efficient projection-aware transformer network for large-scale point cloud registration.  In _ICCV_ , pages 8451–8460, 2023a.
  * Liu et al. [2023b] Quan Liu, Hongzi Zhu, Yunsong Zhou, Hongyang Li, Shan Chang, and Minyi Guo.  Density-invariant features for distant point cloud registration.  In _ICCV_ , pages 18215–18225, 2023b.
  * Lu et al. [2023] Yuhang Lu, Qi Jiang, Runnan Chen, Yuenan Hou, Xinge Zhu, and Yuexin Ma.  See more and know more: Zero-shot point cloud segmentation via multi-modal visual data.  In _ICCV_ , 2023.
  * Mei et al. [2022] Guofeng Mei, Xiaoshui Huang, Jian Zhang, and Qiang Wu.  Overlap-guided coarse-to-fine correspondence prediction for point cloud registration.  In _ICME_ , pages 1–6, 2022.
  * Mei et al. [2023a] Guofeng Mei, Fabio Poiesi, Cristiano Saltori, Jian Zhang, Elisa Ricci, and Nicu Sebe.  Overlap-guided gaussian mixture models for point cloud registration.  In _WACV_ , pages 4511–4520, 2023a.
  * Mei et al. [2023b] Guofeng Mei, Hao Tang, Xiaoshui Huang, Weijie Wang, Juan Liu, Jian Zhang, Luc Van Gool, and Qiang Wu.  Unsupervised deep probabilistic approach for partial point cloud registration.  In _CVPR_ , pages 13611–13620, 2023b.
  * Michele et al. [2021] Björn Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet.  Generative zero-shot learning for semantic segmentation of 3d point clouds.  In _3DV_ , pages 992–1002. IEEE, 2021.
  * Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ , 26, 2013.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In _EMNLP_ , pages 1532–1543, 2014.
  * Peterson [2009] Leif E Peterson.  K-nearest neighbor.  _Scholarpedia_ , 4(2):1883, 2009.
  * Poiesi and Boscaini [(early access] Fabio Poiesi and Davide Boscaini.  Learning general and distinctive 3d local deep descriptors for point cloud registration.  In _IEEE TPAMI_ , (early access) 2022.
  * Qi et al. [2017a] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _CVPR_ , pages 652–660, 2017a.
  * Qi et al. [2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _Advances in neural information processing systems_ , 30, 2017b.
  * Qin et al. [2023] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, Slobodan Ilic, Dewen Hu, and Kai Xu.  Geotransformer: Fast and robust point cloud registration with geometric transformer.  _IEEE TPAMI_ , 2023.
  * Rusinkiewicz and Levoy [2001] Szymon Rusinkiewicz and Marc Levoy.  Efficient variants of the icp algorithm.  In _Proceedings third international conference on 3-D digital imaging and modeling_ , pages 145–152. IEEE, 2001.
  * Rusu et al. [2009] Radu Bogdan Rusu, Nico Blodow, and Michael Beetz.  Fast point feature histograms (fpfh) for 3d registration.  In _ICRA_ , pages 3212–3217. IEEE, 2009.
  * Sarlin et al. [2020] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In _CVPR_ , pages 4938–4947, 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2022] Haiping Wang, Yuan Liu, Zhen Dong, and Wenping Wang.  You only hypothesize once: Point cloud registration with rotation-equivariant descriptors.  In _ACM MM_ , 2022.
  * Yi et al. [2016] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua.  Lift: Learned invariant feature transform.  In _ECCV_ , pages 467–483. Springer, 2016.
  * Yu et al. [2021] Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam, and Slobodan Ilic.  Cofinet: Reliable coarse-to-fine correspondences for robust point cloud registration.  In _Neurips_ , 2021.
  * Yu et al. [2023] Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, and Slobodan Ilic.  Rotation-invariant transformer for point cloud matching.  In _CVPR_ , 2023.
  * Yuan et al. [2023] Mingzhi Yuan, Kexue Fu, Zhihao Li, Yucong Meng, and Manning Wang.  Pointmbf: A multi-scale bidirectional fusion network for unsupervised rgb-d point cloud registration.  In _ICCV_ , 2023.
  * Zeng et al. [2017] Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser.  3dmatch: Learning local geometric descriptors from rgb-d reconstructions.  In _CVPR_ , pages 1802–1811, 2017.
  * Zhang et al. [2022] Yu Zhang, Junle Yu, Xiaolin Huang, Wenhui Zhou, and Ji Hou.  Pcr-cg: Point cloud registration via deep explicit color and geometry.  In _European Conference on Computer Vision_ , pages 443–459. Springer, 2022.
  * Zhou et al. [2016] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.  Fast global registration.  In _ECCV_ , pages 766–782. Springer, 2016.
  * Zhou et al. [2018] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.  Open3d: A modern library for 3d data processing.  _arXiv preprint arXiv:1801.09847_ , 2018.