# Why Does Zero-Shot Cross-Lingual Generation Fail?
An Explanation and a Solution

Tianjian Li1 Kenton Murray1,2
1Center for Language and Speech Processing
2Human Language Technology Center of Excellence
Johns Hopkins University
{tli104<EMAIL_ADDRESS>
###### Abstract

Zero-shot cross-lingual transfer is when a multilingual model is trained to
perform a task in one language and then is applied to another language.
Although the zero-shot cross-lingual transfer approach has achieved success in
various classification tasks (Wu and Dredze, 2019), its performance on natural
language generation tasks falls short in quality (Rönnqvist et al., 2019; Vu
et al., 2022) and sometimes outputs an incorrect language (Xue et al., 2021).
In our study, we show that the fine-tuning process learns language invariant
representations, which is beneficial for classification tasks but harmful for
generation tasks. Motivated by this, we propose a simple method to regularize
the model from learning language invariant representations and a method to
select model checkpoints without a development set in the target language,
both resulting in better generation quality. Experiments on three semantically
diverse generation tasks show that our method reduces the accidental
translation problem by 68% and improves the ROUGE-L score (Lin, 2004) by 1.5
on average.

##  1 Introduction

Language Models (LMs) pre-trained on multilingual corpora (Devlin et al.,
2019a; Conneau et al., 2020a; Liu et al., 2020; Xue et al., 2021) exhibit
zero-shot cross-lingual transfer ability (Wu and Dredze, 2019). Given only
annotated data in one language for a task, multilingual LMs are able to
perform this task in languages seen only during the pre-training stage. The
cross-lingual transferability of multilingual LMs reduces the need for
annotated data in low-resource languages, which is valuable for building
practical multilingual NLP systems.

Existing studies on cross-lingual transfer select tasks such as word alignment
(Artetxe et al., 2020b), POS tagging (Pires et al., 2019), dependency parsing
and sentence classification (Wu and Dredze, 2019) to investigate cross-lingual
transferability of multilingual LMs (Hu et al., 2020), and few works focus on
cross-lingual transfer in generation tasks (Maurya et al., 2021; Maurya and
Desarkar, 2022). Cross-lingual transfer approach in generation tasks are known
to produce incoherent text (Rönnqvist et al., 2021), generate in a wrong
language (Xue et al., 2021), and suffer from catastrophic forgetting (Vu et
al., 2022). Table 1 illustrates a common problem where the multilingual LM
generates text in an incorrect language. Moreover, such a problem becomes more
severe when under a true zero-shot setting (Zhao et al., 2021; Schmidt et al.,
2022), where we do not have annotated data in the target language to guide
model selection.

We show that the reason why zero-shot cross-lingual transfer in text
generation fails is because the fine-tuning process learns language invariant
representations, which is beneficial for classification tasks, but detrimental
to generation tasks. In our paper, we use the cosine similarity between
parallel sentence representations in different languages to measure the Cross-
Lingual Representation Similarity (XLRS). We use a range of tasks from
classification to extractive question answering, then to abstractive
generation to show that in the best performing model, the XLRS after fine-
tuning decreases as we move from classification to generation.

Prediction | Speak to your doctor, understand the dangers of alcohol consumption..
---|---
Target | 与您的医生交谈：改变您对戒酒的想法。
| (Speak to your doctor: change your thinking towards quitting alcohol)
Prediction | Review accounting books periodically.
Target | mj 기간을 결정한다. 회계장부를 모두 검토한다. 누락된 정보를 취합한다.
| (Determine the period. Review all accounting books. Gather missing
information.)
Table 1: Example predictions of mT5 model fine-tuned on English WikiHow
instructions and evaluated on Chinese and Korean input. The model outputs
relevant text in an incorrect language.

The fact that language invariant representations causes the degradation in
generation tasks challenges the common belief that invariant representations
generally enhance cross-lingual transfer on all downstream tasks (Cao et al.,
2020; Conneau et al., 2020b; Yang et al., 2022; Xian et al., 2022). To the
best of our knowledge, our work is the first to provide an analysis of how
XLRS affects cross-lingual transfer in language generation tasks.

Motivated by our findings, we propose to use an auxiliary source language that
implicitly regularizes the XLRS being too large and results in better
generation performance over three complex natural language generation tasks
(Summarization, Story Completion, and Title Generation). Under a true zero-
shot setting, choosing the model checkpoint with the lowest XLRS results in an
average of 4.1 point increase in ROUGE-L over using a source development set
in two generation datasets.

To sum up, our contributions are threefold:

  * •

We show that fine-tuning on a single source language increases the cosine
similarity between sentence representations of different languages (XLRS).

  * •

We show that the increase in XLRS causes degradation of cross-lingual transfer
in generation tasks, and argue that the prevalent understanding of the benefit
of similar representations does not apply to generation tasks.

  * •

We empirically show that using two gold-annotated source languages instead of
one regularizes the XLRS, resulting in an average increase of 1.5 in ROUGE-L.

##  2 Related Works

Multilingual Language Models. One line of work is to train multilingual
versions of modern Language Models. mBERT  (Devlin et al., 2019b) is the
multilingual version of BERT (Devlin et al., 2019a), which uses the same
encoder-only model architecture but is only trained on multilingual corpora.
XLM-R (Conneau et al., 2020a) is the multilingual version of RoBERTa (Liu et
al., 2019), which implements multiple optimization tricks and is larger in
scale, resulting in better performance than BERT. mBART (Liu et al., 2020) is
the multilingual version of BART (Lewis et al., 2020), an encoder-decoder
model trained to reconstruct the original text through various types of
artificially introduced noises. mT5 (Xue et al., 2021) is the multilingual
version of T5 (Raffel et al., 2020), an encoder-decoder model trained on a
span denoising objective.

Cross-lingual Transfer. Multilingual models are able to be fine-tuned on
annotated data of a task in only one source language and transfer the
knowledge to other target languages to perform the same task without any
supervision. While Pires et al. (2019) states that sub-word overlap between
source and target facilitates cross-lingual transfer, K et al. (2020) shows
that cross-lingual transfer manifests in pairs of source and target with zero
sub-word overlap and word order is instead the most crucial ingredient. The
performance of cross-lingual transfer between languages with a different order
severely drops. Although the importance of word order is echoed by later
studies (Artetxe et al., 2020b; Dufter and Schütze, 2020), recent studies have
also debated in favor of the importance of matching script also contributing
to cross-lingual transfer (Lauscher et al., 2020; Fujinuma et al., 2022). Wu
et al. (2022) points out that the optimal set of parameters that generalizes
well to all languages is a subset of parameters that achieves good performance
on the source language. Therefore it is hard to find the optimal zero-shot
cross-lingual transfer parameters by only optimizing source language
performance. Chen and Ritter (2021) train a scoring model with the input
features being the model’s hidden representations and the output score being
how well it generalizes to a given target language. However, previous studies
focus on lower-level NLP tasks, which include text classification, dependency
parsing, and extractive question answering (Hu et al., 2020) and rarely touch
on language generation.
Another line of work focuses on applying cross-lingual transfer to a wide
range of multilingual NLP applications, which include sequence tagging (Yang
et al., 2016), Named Entity Recognition (Xie et al., 2018), dependency parsing
(Ahmad et al., 2019), sentence classification (Conneau et al., 2018; Yang et
al., 2019), and information retrieval (Izacard et al., 2022). Empirical
studies also train ranking models (Lin et al., 2019), use meta-learning
(Nooralahzadeh et al., 2020), or use Shapley Value (Parvez and Chang, 2021) to
predict which sources perform the best for a given target language.

Natural Language Generation. Multilingual LMs are prone to produce text that
is repetitive (Xu et al., 2022), contains hallucinations (Raunak et al.,
2021), or is in the wrong language (Zhang et al., 2020; Xue et al., 2021; Vu
et al., 2022). Vu et al., 2022 proposed to use parameter efficient fine-tuning
methods (Lester et al., 2021; Qin and Eisner, 2021; Li and Liang, 2021) to
regularize the model to generate in a desired language. Other ways to improve
generation quality include using back translation (Gu et al., 2019; Zhang et
al., 2020), and transliteration (Sun et al., 2022) as data augmentation
techniques, mixing in the pretrain objective during fine-tuning (Xue et al.,
2021) and using an auxiliary source language in machine translation (Xu et
al., 2021). Two concurrent efforts are close to our work: Xu and Murray (2022)
and Schmidt et al. (2022) both empirically show that using multiple languages
during fine-tuning in few-shot cross-lingual transfer improves performance in
text classification. Our work differs in that we evaluated text generation
under a true zero-shot setting, where we have access to neither a few examples
to train on nor an annotated development set to guide model checkpoint
selection.

##  3 Setup

The consensus of the literature (Cao et al., 2020; Conneau et al., 2020b;
Tiyajamorn et al., 2021; Yang et al., 2022; Xian et al., 2022) is that if a
model can produce similar representations for parallel sentences, the model
would be able to achieve good cross-lingual transfer performance. Intuitively,
if a model maps parallel sentences in English and French into nearly identical
representations, and is able to predict the sentiment of the English sentence,
it will also be able to predict the sentiment of the French sentence.

We hypothesize that the fine-tuning process increases the similarity between
sentence representations of different languages. We use the following setups
and tasks to verify our hypothesis.

###  3.1 Models and Datasets

Models We select the state-of-the-art multilingual language model: mT5-base
(Xue et al., 2021). We use the Huggingface (Wolf et al., 2020) implementation.
We use a uniform learning rate of 7e-5, a batch size of 32 for 10 epochs for
all tasks described below.

Name | Task | Metric
---|---|---
UDPOS | Part-of-speech tagging | Acc.
PAWS-X | Paraphrase Identification | F1
TyDiQA | Question Answering | F1/EM
WikiLingua | Summarization | ROUGE
Table 2: Summary of tasks used in §5.

Datasets Table 2 describes the tasks we used in the following section to show
the transition from classification to generation111We follow (Vu et al., 2022)
and report the SP-ROUGE score.. We use the UDPOS (Nivre et al., 2018) dataset
containing sentences and the part-of-speech tag of each word. For sentence-
level classification, we use the PAWS-X (Yang et al., 2019) dataset containing
pairs of sentences and a binary tag on whether the second sentence entails the
first sentence. For extractive generation, we use the TyDiQA-GoldP (Clark et
al., 2020) dataset which contains paragraphs and questions whose answers are
spans extracted from the paragraphs. For abstractive generation, we use the
WikiLingua (Ladhak et al., 2020) dataset, which contains WikiHow instructions
and their summaries in 17 languages.

We use the story completion (SG) and title generation (TG) task in the MTG
benchmark (Chen et al., 2022), a recently introduced benchmark to evaluate
multilingual text generation. We follow Vu et al., 2022, which uses the
WikiLingua dataset to construct WikiLingua-0, where the model is only fine-
tuned on English and evaluated on other languages. We extend WikiLingua-0 and
use languages besides English as the source and evaluate the zero-shot
directions.

In all of our experiments, we report the results averaged across three runs
with different random seeds. For each source language, we only use the top 10k
training examples to train our model to ablate the effect of training data
size on cross-lingual transfer. Unless specified otherwise, we evaluate under
a true zero-shot setting, where we select the model checkpoint based on its
performance on a dev set of the source language.

###  3.2 Sequence to Sequence Learning

We cast sequence labeling and sentence classification tasks into a text-to-
text format using templates described in Table 10 in the Appendix. We follow
Raman et al. (2022) and cast sequence labeling tasks into the sentinel + tag
format. We follow Schick and Schütze (2021) and cast the sentence entailment
task into a cloze question, supervising the model to predict the word "yes"
for entailment and the word "no" for non-entailment.

##  4 Learning Dynamics of Cross-lingual Transfer

We plot the average cosine similarity between representations of parallel
sentences (XLRS)222We use the mean-pooled token encoder hidden states as the
sentence representation. We randomly sample 512 sentences from the test set of
the MTG story completion task. for each training iteration in two
classification tasks: POS tagging and paraphrase identification (PAWS-X) at
Figure 2 and Figure 2, respectively.

Figure 1: Average cosine similarity between parallel sentence representations
(XLRS) in pretrained mT5-base model fine-tuned on English POS tagging data.

Figure 2: Average cosine similarity between parallel sentence representations
(XLRS) in pretrained mT5-base model fine-tuned on English PAWS-X data.

In both tasks, the plot displays an increasing trend of XLRS between parallel
sentences between English and all the other languages. Notably, languages that
have the same script have a higher similarity. Our findings show that the
fine-tuning process on classification tasks does make the sentence
representations of different languages more similar.

We then plotted the XLRS between representations of parallel sentences of a
model when fine-tuned on WikiLingua: a summarization dataset in Figure 3. The
average similarity gradually increases as we progress further into the
training iterations, confirming our hypothesis that fine-tuning on a single
source language increases the XLRS between the source and other languages.

Based on our findings, we conjecture that the model jointly minimizes two
metrics, resulting in cross-lingual transfer:

  * •

The Cross-Entropy loss between the predicted labels and the ground-truth
labels, given an input in the source language (the standard training
objective).

  * •

The distance between parallel sentences of the source and target languages
(increase in XLRS).

And as a result, the cross-entropy loss between the predicted and ground-truth
labels, given a context in the target language, is minimized, enabling cross-
lingual transfer.

Figure 3: Average cosine similarity between parallel sentences representations
(XLRS) of pretrained mT5-base model fine-tuned on English WikiLingua data.

##  5 Unified View of Tasks

Figure 4: Illustration of the difference between classification tasks (top),
where the model needs to map parallel sentences to the same label ($d=100\%$),
and generation tasks (bottom), where the model needs to map parallel sentences
into different labels ($d=0\%$). We use label overlap, $d$, to denote the
fraction of parallel sentences that map to the same label.

With the intuition of how the model does cross-lingual transfer in
classification tasks, we note that language generation is a classification on
a large _vocabulary set_ , rather than a small label set. Thus, we point out
that the reason why good performance with cross-lingual transfer on generation
tasks is harder to achieve is actually caused by increasing XLRS.

Figure 4 illustrates our intuition. In classification tasks, the model needs
to map parallel sentences to the same label. Ideally, the model produces
identical representations for parallel sentences, resulting in the highest
possible XLRS of 1. This is why model cross-lingual transfers better with a
high XLRS. However, in generation tasks, if we view it as classifying over the
entire vocabulary set, we are mapping parallel sentences to different labels.
In an extreme case when XLRS is 1, the model fails to identify the source
language, resulting in the common problem of the model producing an incorrect
language (Xue et al., 2021). We introduce the notion of label overlap, $d$, to
measure the percentage of examples in a dataset where the model needs to map
parallel sentences into the same label.

We use $\mathcal{C}$ to denote the set of all discrete contexts and
$\mathcal{C}_{s}$ to denote the set of all discrete contexts in language $s$.
In classification tasks, the model learns to predict the ground-truth label
$\hat{y}=l(c)$ over a set of candidate labels $\mathcal{Y}$ for context
$c\in\mathcal{C}$. Similarly, in a simplified view of generation, the model
learns to predict the next word $v$ given context $c$. Therefore, we can
essentially view language generation as classification where the label set is
the entire vocabulary. In both cases, given context $c$, the model learns a
probability distribution $p_{\cdot|c}$. The only difference is between their
classification label set cardinality.

We define cross-lingual label overlap as an indicator of difficulty to cross-
lingual transfer for a given task at §5.1. We then use a range of tasks: word-
level classification (POS tagging §5.2) - Sentence level classification
(Entailment classification §5.3) - Span Extraction (Extractive Generation
§5.4) - Summarization (Abstractive generation §5.5) to show an increasing
level of difficulty to perform cross-lingual transfer.

###  5.1 Cross-lingual Label Overlap

We denote a task’s difficulty in transferring knowledge from one language to
another by the percentage of overlap of their label set for parallel
sentences. Given $n$ parallel sentences
$\\{c_{s}^{1},c_{s}^{2},...,c_{s}^{n}\\}$ and
$\\{c_{t}^{1},c_{t}^{2},...,c_{t}^{n}\\}$ in source language $s$ and target
language $t$, the cross-lingual overlap $d$ for task $\alpha$ is defined as:

$d_{\alpha}(s,t)=\frac{\sum_{i=1}^{n}\mathds{1}(l_{\alpha}(c_{s}^{i})=l_{\alpha}(c_{t}^{i}))}{n}$

In our analysis, we use English as the source language and evaluate the
difficulty of performing cross-lingual transfer on other languages. The total
label overlap for each task is the average label overlap for each target
language.

$d_{\alpha}=\sum_{j=1}^{m}d_{\alpha}(\text{english},t_{j})$

Where $t_{j}$ is the $j$th target language.

A higher $d_{\alpha}$ indicates an easier task to transfer knowledge from one
language to another, whereas a lower $d_{\alpha}$ indicates a more difficult
task to cross-lingual transfer.

###  5.2 POS tagging

The word-level label overlap for part-of-speech tagging should be close to
100%. With such a high percentage of label overlap, the model benefits from
producing identical representations for parallel sentences to predict the same
labels for different languages without supervision of the target language.

For example, if the model maps the English sentence the dog ran and the french
sentence le chien courir into nearly identical representations and
simultaneously learns a function to map the English words to their respective
POS tags "DET NOUN VERB", the model would also be able to predict the correct
label for french even without supervision. We denote the amount of "label
overlap" as a metric defining the difficulty for a model to perform cross-
lingual transfer on it.

###  5.3 Sentence Classification

The classification task discussed in this section (PAWS-X) includes sentiment
classification of a single sentence and entailment classification between two
sentences. For semantically equivalent parallel sentences, their sentiment or
entailment labels are always the same. Therefore, $d=100\%$.

Ideally, in sentence classification tasks, parallel sentences in different
languages should map to the same probability distribution. For example, if the
English sentence I am happy and the french sentence Je suis content maps to
nearly identical representations and the model learns to predict the sentiment
in English, the model would be able to cross-lingual transfer the ability to
predict sentiment in English to French without any supervision.

###  5.4 Span Extraction

Span extraction requires a model to select a correct answer span from a
passage given a question. Even though the data in TyDiQA is in different
languages, and not parallel, 16% of the answer spans are pure numbers, and
50.6% of answer spans are mainly composed of numbers (Time and Dates).

This indicates that span extraction is a harder task than sentence
classification, but with such a high amount of label overlap, the task is
solvable through cross-lingual transfer.

###  5.5 Generation

The amount of label overlap in abstractive generation tasks (e.g.
summarization, story completion, title generation) is close to zero as the
model needs to predict words in completely different languages. The amount of
label overlap for a subset of five languages (En, De, Fr, Es, Zh) of the
WikiLingua (Ladhak et al., 2020) dataset is $d=0.13\%$333This small percentage
represents mainly numbers and named entities (i.e. cities) that are the same
across languages..

In a generation task, if the model maps the source and the target into
identical representations, the model predicts the next word to be the same.
Even if this is correct in semantics and possibly results in the code-switched
results as shown in Figure 1, the model fails to generate in the correct
language.

| AR | ZH | CS | NL | EN | FR | HI | ID | IT | JA | KO | PT | RU | ES | TH | TR | VI |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
EN* | 17.4 | 15.1 | 17.8 | 20.1 | 39.6 | 22.4 | 9.1 | 23.0 | 20.3 | 14.6 | 17.3 | 23.8 | 15.3 | 23.3 | 17.9 | 17.5 | 21.9 |
EN | 24.1 | 22.4 | 18.6 | 20.0 | 31.7 | 22.4 | 18.2 | 19.4 | 20.6 | 21.0 | 23.1 | 23.7 | 17.6 | 23.5 | 20.9 | 17.8 | 21.5 |
EN+ZH | 24.3 | 27.5 | 20.4 | 22.6 | 33.2 | 23.8 | 18.8 | 21.6 | 22.1 | 18.4 | 21.2 | 27.2 | 20.2 | 24.9 | 21.8 | 18.2 | 24.3 |
DE | 23.9 | 22.5 | 20.4 | 23.2 | 24.1 | 24.1 | 19.2 | 23.2 | 22.5 | 18.3 | 23.1 | 26.1 | 19.7 | 25.2 | 19.5 | 17.9 | 27.0 |
DE+ZH | 24.8 | 27.9 | 21.2 | 24.2 | 26.0 | 25.2 | 19.5 | 24.2 | 24.1 | 20.3 | 23.7 | 26.9 | 21.6 | 25.9 | 21.7 | 19.1 | 27.1 |
EN+DE | 24.9 | 25.2 | 21.4 | 24.0 | 22.3 | 25.3 | 20.1 | 23.7 | 22.8 | 19.3 | 24.1 | 27.2 | 20.3 | 25.8 | 20.0 | 19.3 | 27.4 |
Table 3: ROUGE-L results on the WikiLingua (Ladhak et al., 2020) dataset.
Left: Source languages that we fine-tune on. Top: Target language that we
evaluate on. The bolded numbers refer to the highest zero-shot performance.
Note that some of the directions are not zero-shot. The amount of training
instances used in each row is the same. * indicate results reported in Vu et
al. (2022).

###  5.6 Analysis

We plotted the XLRS between English and four different languages in the best-
performing English supervised models for the four tasks and the pretrained
model at Figure 5.

Figure 5: Average cosine similarity between representations of parallel
sentences in English and 4 languages for the best performing model in 4
different tasks.

The plot confirms our belief that for tasks (POS tagging, PAWS-X, TyDiQA) with
large label overlap, the model cross-lingual transfers from increasing XLRS,
whereas in generation tasks with label overlap close to zero (title
generation), the best-performing model has a lower XLRS.

Task | POS | PAWS-X | TG | SG | WikiLingua
---|---|---|---|---|---
$\rho$ | 0.89* | 0.91* | -0.37* | -0.39* | -0.33*
Table 4: Spearman’s rank correlation $\rho$ between the average cosine
similarity between parallel sentences in source and 4 target languages (De,
Es, Fr, Zh) and the average zero-shot cross-lingual transfer performance (F1
for POS tagging, Acc. for PAWS-X and ROUGE-L for generation) on two
classification tasks and three generation tasks, * indicates that the p-value
is less than 0.05.

Following Yang et al. (2022), we calculate the Spearman’s rank correlation
score between (a) XLRS between English and 4 target languages (German, French,
Chinese, Spanish), and (b) The averaged zero-shot cross-lingual transfer
performance in each task. The results are reported at Table 4. In both
classification tasks, XLRS positively correlates with cross-lingual
performance. In contrast, in our three generation tasks, XLRS negatively
correlates with cross-lingual performance444We observed a stronger negative
correlation between cosine similarity and ROUGE-2 in generation tasks but
opted to report the ROUGE-L results to be consistent with our main results.,
indicating that XLRS is strongly correlated to transfer performance in
classification tasks but is detrimental to cross-lingual transfer in
generation tasks.

##  6 Text Generation Experiments

Figure 6: Average cosine similarity between representations of parallel
sentences (XLRS) in English and French for model trained on one and two source
languages.

Now that we know XLRS is negatively correlated with cross-lingual transfer in
text generation, since calculating XLRS during every iteration is
computationally expensive, we wonder if we can regularize XLRS implicitly.
Motivated by using auxiliary source languages improves machine translation (Xu
et al., 2021) and few-shot cross-lingual transfer (Xu and Murray, 2022;
Schmidt et al., 2022), we propose to use an additional source language to
regularize XLRS.

To verify our hypothesis, we plot the XLRS between English and French during
training on two different sources (En, De)in the story completion task,
compared to training only on one source (En) in Figure 6. We observe that when
the model is only given one language as the source, the XLRS increases,
whereas using two source languages allows the model to learn to control the
XLRS from being too high, resulting in fewer accidental translations and
better quality.

To show that regularizing XLRS does result in better generation quality, we
experiment with three semantically diverse generation tasks: Summarization
(WikiLingua), Title Generation (TG), and Story Completion (SG).

###  6.1 Results

Table 3 shows the results of fine-tuning with multiple languages. We observe
that adding Chinese to English data improves the performance in 13 out of 15
zero-shot directions555The results when the target language are Chinese (ZH)
or English (EN) is not zero-shot. compared to only using English. We point out
that our improvement is not due to an increase in the amount of training data
since we used the same amount of training data for all experiments. We further
observe that adding Chinese as an additional language to German also improves
the performance in all 14 zero-shot directions, which often results in the
best zero-shot performance.

Table 5 and 6 show the ROUGE-L results in the title generation (TG) and story
completion (SG) in the MTG (Chen et al., 2022) benchmark, respectively. Again,
we are able to observe that using two source languages almost always improves
the ROUGE-L score. Notably, using two related languages often results in
degraded performance than using two unrelated languages with different
scripts. We hypothesize that a language with a different script and order
provides a more substantial regularization effect, preventing the cosine
similarity between the source and target sentence representations from being
too high.

| EN | ES | DE | FR | ZH | Avg.
---|---|---|---|---|---|---
EN | 32.3 | 26.0 | 24.4 | 25.3 | 19.6 | 25.5
DE | 30.2 | 24.7 | 22.5 | 23.9 | 18.7 | 24.0
ZH | 25.1 | 24.5 | 21.4 | 23.7 | 26.0 | 24.1
DE+ZH | 29.2 | 24.8 | 23.6 | 24.9 | 22.7 | 25.0
DE+EN | 27.8 | 24.6 | 23.6 | 22.3 | 20.8 | 23.8
EN+ZH | 33.3 | 28.4 | 26.8 | 27.4 | 22.4 | 27.7
Table 5: ROUGE-L results of title generation task in MTG benchmark. All experiments used same amount of data. The best zero-shot performance on each target language is bold. | EN | ES | DE | FR | ZH | Avg.
---|---|---|---|---|---|---
EN | 29.1 | 28.9 | 27.8 | 28.9 | 20.3 | 27.0
DE | 29.3 | 28.7 | 29.9 | 32.0 | 20.4 | 28.0
ZH | 22.3 | 21.2 | 22.3 | 28.6 | 26.6 | 24.2
DE+ZH | 31.5 | 29.7 | 28.6 | 31.8 | 22.5 | 28.8
DE+EN | 30.4 | 27.3 | 26.4 | 28.2 | 19.8 | 26.4
EN+ZH | 31.8 | 28.5 | 29.3 | 29.1 | 28.6 | 29.5
Table 6: ROUGE-L results of story completion task in MTG benchmark. All
experiments used the same amount of data. The best zero-shot performance on
each target language is bold.

To verify that our method helps against the accidental translation problem, we
follow previous work (Vu et al., 2022) and calculate the language id
confidence score on the source language and target language on the title
generation task. The results are shown at Table 9 in Appendix A. Fine-tuning
with multiple source languages helps the model learn which language it should
produce.

###  6.2 Model Selection Using Parallel Sentences

Since XLRS negatively correlates with the performance of cross-lingual
generation, we use it as a criterion for model selection in the absence of an
annotated dev set. We report the performance on the WikiLingua dataset and the
story completion task in MTG benchmark at Figure 8 and 8, when selecting the
model using English dev set performance (en-dev), selecting the model with the
lowest XLRS between English and the target language (cos-sim), and selecting
the model using an annotated dev set on each target language (tgt-dev), which
serves as an upper bound for true zero-shot cross-lingual transfer.

In both tasks, selecting the model checkpoint with the lowest XLRS results in
better performance than using an English development set on all target
languages. The performance is on average less than one ROUGE-L point less on
Spanish and German in both datasets, compared to using an annotated dev set.
Our method results in an average increase of 5 ROUGE-L points in a distant
language (Chinese).

| ES | DE | FR | ZH | $\Delta$
---|---|---|---|---|---
en-dev | 23.5 | 19.8 | 22.4 | 22.4 | -3.23
cos-sim | 25.3 | 21.2 | 24.6 | 26.5 | -0.85
tgt-dev | 25.4 | 21.9 | 25.4 | 28.3 | 0

Table 7: ROUGE-L results by selecting model based on English development set
(en-dev), similarity of representations between English and target language
(cos-sim) and using target language development set (tgt-dev) on WikiLingua
(Ladhak et al., 2020).

| ES | DE | FR | ZH | $\Delta$
---|---|---|---|---|---
en-dev | 28.9 | 27.8 | 28.9 | 20.3 | -4.88
cos-sim | 30.8 | 30.3 | 35.6 | 26.4 | -0.58
tgt-dev | 31.2 | 30.5 | 35.6 | 28.1 | 0

Table 8: ROUGE-L results by selecting model checkpoints in Story Completion
(SG) task in MTG benchmark (Chen et al., 2022).

##  7 Conclusion

We show that multilingual LMs transfer supervision from one language to
another by increasing Cross-Lingual Representation Similarity (XLRS). Such a
learning process results in decent zero-shot cross-lingual transfer
performance in classification tasks but is harmful to text generation
performance. We demonstrate that regularizing XLRS improves text generation
quality and use parallel sentences to guide model selection without annotated
data in the target languages. We believe that this is valuable under a
practical setting (Artetxe et al., 2020c) where we have access to parallel
data between the source and target languages, but not task-specific data in
the target language.

## Limitations

Our work sheds light on understanding the training dynamics of cross-lingual
transfer learning of multilingual LMs. In our work, we selected to use English
as the source of cross-lingual transfer following previous work (Vu et al.,
2022). We acknowledge that using other languages as the source language can
provide benefits depending on the task (Lin et al., 2019; Turc et al., 2021).
Our work does not focus on choosing source language to maximize downstream
performance but instead focuses on the difference between classification tasks
and generation tasks in cross-lingual transfer.

Secondly, we acknowledge that some of the datasets (Yang et al., 2019; Chen et
al., 2022) used in our work is created by machine translation and human
annotation. Previous studies have pointed out that translationese in datasets
affects cross-lingual transfer performance (Artetxe et al., 2020a; Artetxe et
al., 2020c). We believe that translationese in datasets also has an impact on
XLRS. We leave the study of how dataset features (size, quality,
translationese) affect cross-lingual transfer for future work.

## Acknowledgements

We sincerely thank Haoran Xu, Kelly Marchisio and Daniel Khashabi for their
helpful suggestions.

## References

  * Ahmad et al. (2019) Wasi Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard Hovy, Kai-Wei Chang, and Nanyun Peng. 2019.  On difficulties of cross-lingual transfer with order differences: A case study on dependency parsing.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2440–2452, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Artetxe et al. (2020a) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020a.  Translation artifacts in cross-lingual transfer learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7674–7684, Online. Association for Computational Linguistics.
  * Artetxe et al. (2020b) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Artetxe et al. (2020c) Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020c.  A call for more rigor in unsupervised cross-lingual learning.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7375–7388, Online. Association for Computational Linguistics.
  * Cao et al. (2020) Steven Cao, Nikita Kitaev, and Dan Klein. 2020.  Multilingual alignment of contextual word representations.  In _International Conference on Learning Representations_.
  * Chen and Ritter (2021) Yang Chen and Alan Ritter. 2021.  Model selection for cross-lingual transfer.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5675–5687, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chen et al. (2022) Yiran Chen, Zhenqiao Song, Xianze Wu, Danqing Wang, Jingjing Xu, Jiaze Chen, Hao Zhou, and Lei Li. 2022.  MTG: A benchmark suite for multilingual text generation.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2508–2527, Seattle, United States. Association for Computational Linguistics.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages.  In _Transactions of the Association of Computational Linguistics_.
  * Conneau et al. (2020a) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Conneau et al. (2020b) Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b.  Emerging cross-lingual structure in pretrained language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6022–6034, Online. Association for Computational Linguistics.
  * Devlin et al. (2019a) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Devlin et al. (2019b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b.  mbert blog post.  https://github.com/google-research/bert/blob/master/multilingual.md.  Accessed: 2022-11-05.
  * Dufter and Schütze (2020) Philipp Dufter and Hinrich Schütze. 2020.  Identifying elements essential for BERT’s multilinguality.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4423–4437, Online. Association for Computational Linguistics.
  * Fujinuma et al. (2022) Yoshinari Fujinuma, Jordan Boyd-Graber, and Katharina Kann. 2022.  Match the script, adapt if multilingual: Analyzing the effect of multilingual pretraining on cross-lingual transferability.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1500–1512, Dublin, Ireland. Association for Computational Linguistics.
  * Gu et al. (2019) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O.K. Li. 2019.  Improved zero-shot neural machine translation via ignoring spurious correlations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1258–1268, Florence, Italy. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR.
  * Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022.  Unsupervised dense information retrieval with contrastive learning.  _Transactions on Machine Learning Research_.
  * K et al. (2020) Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020.  Cross-lingual ability of multilingual bert: An empirical study.  In _International Conference on Learning Representations_.
  * Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020.  WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4034–4048, Online. Association for Computational Linguistics.
  * Lauscher et al. (2020) Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020.  From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4483–4499, Online. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2019) Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019.  Choosing transfer languages for cross-lingual learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3125–3135, Florence, Italy. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Maurya and Desarkar (2022) Kaushal Maurya and Maunendra Desarkar. 2022.  Meta-xNLG: A meta-learning approach based on language clustering for zero-shot cross-lingual transfer and generation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 269–284, Dublin, Ireland. Association for Computational Linguistics.
  * Maurya et al. (2021) Kaushal Kumar Maurya, Maunendra Sankar Desarkar, Yoshinobu Kano, and Kumari Deepshikha. 2021.  ZmBART: An unsupervised cross-lingual transfer framework for language generation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2804–2818, Online. Association for Computational Linguistics.
  * Nivre et al. (2018) Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018.  Universal dependencies 2.2.
  * Nooralahzadeh et al. (2020) Farhad Nooralahzadeh, Giannis Bekoulis, Johannes Bjerva, and Isabelle Augenstein. 2020.  Zero-shot cross-lingual transfer with meta learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4547–4562, Online. Association for Computational Linguistics.
  * Parvez and Chang (2021) Md Rizwan Parvez and Kai-Wei Chang. 2021.  Evaluating the values of sources in transfer learning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5084–5116, Online. Association for Computational Linguistics.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Qin and Eisner (2021) Guanghui Qin and Jason Eisner. 2021.  Learning how to ask: Querying LMs with mixtures of soft prompts.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5203–5212, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Raman et al. (2022) Karthik Raman, Iftekhar Naim, Jiecao Chen, Kazuma Hashimoto, Kiran Yalasangi, and Krishna Srinivasan. 2022.  Transforming sequence tagging into a seq2seq task.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021.  The curious case of hallucinations in neural machine translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1172–1183, Online. Association for Computational Linguistics.
  * Rönnqvist et al. (2019) Samuel Rönnqvist, Jenna Kanerva, Tapio Salakoski, and Filip Ginter. 2019.  Is multilingual BERT fluent in language generation? In _Proceedings of the First NLPL Workshop on Deep Learning for Natural Language Processing_ , pages 29–36, Turku, Finland. Linköping University Electronic Press.
  * Rönnqvist et al. (2021) Samuel Rönnqvist, Valtteri Skantsi, Miika Oinonen, and Veronika Laippala. 2021\.  Multilingual and zero-shot is closing in on monolingual web register classification.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 157–165, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online. Association for Computational Linguistics.
  * Schmidt et al. (2022) Fabian David Schmidt, Ivan Vulić, and Goran Glavaš. 2022.  Don’t stop fine-tuning: On training regimes for few-shot cross-lingual transfer with multilingual language models.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Sun et al. (2022) Simeng Sun, Angela Fan, James Cross, Vishrav Chaudhary, Chau Tran, Philipp Koehn, and Francisco Guzmán. 2022.  Alternative input signals ease transfer in multilingual machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5291–5305, Dublin, Ireland. Association for Computational Linguistics.
  * Tiyajamorn et al. (2021) Nattapong Tiyajamorn, Tomoyuki Kajiwara, Yuki Arase, and Makoto Onizuka. 2021.  Language-agnostic representation from multilingual sentence encoders for cross-lingual similarity estimation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7764–7774, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Turc et al. (2021) Iulia Turc, Kenton Lee, Jacob Eisenstein, Ming-Wei Chang, and Kristina Toutanova. 2021.  Revisiting the primacy of english in zero-shot cross-lingual transfer.
  * Vu et al. (2022) Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant. 2022\.  Overcoming catastrophic forgetting in zero-shot cross-lingual generation.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 833–844, Hong Kong, China. Association for Computational Linguistics.
  * Wu et al. (2022) Shijie Wu, Benjamin Van Durme, and Mark Dredze. 2022.  Zero-shot cross-lingual transfer is under-specified optimization.  In _Proceedings of the 7th Workshop on Representation Learning for NLP_ , pages 236–248, Dublin, Ireland. Association for Computational Linguistics.
  * Xian et al. (2022) Ruicheng Xian, Heng Ji, and Han Zhao. 2022.  Cross-lingual transfer with class-weighted language-invariant representations.  In _International Conference on Learning Representations_.
  * Xie et al. (2018) Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018\.  Neural cross-lingual named entity recognition with minimal resources.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 369–379, Brussels, Belgium. Association for Computational Linguistics.
  * Xu and Murray (2022) Haoran Xu and Kenton Murray. 2022.  Por qué não utiliser alla språk? mixed training with gradient optimization in few-shot cross-lingual transfer.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2043–2059, Seattle, United States. Association for Computational Linguistics.
  * Xu et al. (2022) Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022.  Learning to break the loop: Analyzing and mitigating repetitions for neural text generation.  In _Advances in Neural Information Processing Systems_.
  * Xu et al. (2021) Weijia Xu, Yuwei Yin, Shuming Ma, Dongdong Zhang, and Haoyang Huang. 2021.  Improving multilingual neural machine translation with auxiliary source languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3029–3041, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yang et al. (2022) Huiyun Yang, Huadong Chen, Hao Zhou, and Lei Li. 2022.  Enhancing cross-lingual transfer by manifold mixup.  In _International Conference on Learning Representations_.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
  * Yang et al. (2016) Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2016.  Multi-task cross-lingual sequence tagging from scratch.  _CoRR_ , abs/1603.06270.
  * Zhang et al. (2020) Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020.  Improving massively multilingual neural machine translation and zero-shot translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1628–1639, Online. Association for Computational Linguistics.
  * Zhao et al. (2021) Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulić, Roi Reichart, Anna Korhonen, and Hinrich Schütze. 2021.  A closer look at few-shot crosslingual transfer: The choice of shots matters.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5751–5767, Online. Association for Computational Linguistics.

##  Appendix A Language Identification Scores

| FR | ES
---|---|---
| $\textrm{LID}_{DE}$ | $\textrm{LID}_{FR}$ | $\textrm{LID}_{DE}$ | $\textrm{LID}_{ES}$
DE | 67.7 | 20.5 | 73.6 | 18.8
DE+ZH | 9.8 | 88.2 | 11.2 | 86.4
DE+EN | 15.2 | 76.2 | 14.9 | 78.4
Table 9: Language identification confidence scores on the title generation task fine-tuned on single and multiple source languages. Task | Template
---|---
| Seq Tagging
---
(UDPOS)
| Input: <extra_id_0>In <extra_id_2>my <extra_id_3>view <extra_id_4>it
<extra_id_5>is <extra_id_6>significant
---
Output: <extra_id_0>ADP <extra_id_2>PRON <extra_id_3>NOUN <extra_id_4>PRON
<extra_id_5>AUX <extra_id_6>ADJ
| Classification
---
(PAWS-X)
| Input: The original version was skipped in favor of the mild edition.
<extra_id_0>The mild version was skipped in favor of the original version.
---
Output: <extra_id_0>No.
| QA
---
(TyDiQA)
| Input: What is the surface area of the human cortex? <extra_id_0>
---
Output: <extra_id_0>1.3 square feet
| Generation
---
(ByteCup)
| Input: story: {News article on Philadelphia Flower Show} title: <extra_id_0>
---
Output: <extra_id_0>philly flower show will treat visitors to sights, sounds
and scents of rainforest
Table 10: Templates for casting tasks into a text-to-text format.


††thanks: Corresponding author

# Nonsingular black holes from conformal symmetries

M. Cadoni<EMAIL_ADDRESS>Dipartimento di Fisica, Università di
Cagliari, Cittadella Universitaria, 09042 Monserrato, Italy  INFN, Sezione di
Cagliari, Cittadella Universitaria, 09042 Monserrato, Italy  A. P. Sanna
<EMAIL_ADDRESS>Dipartimento di Fisica, Università di Cagliari, Cittadella
Universitaria, 09042 Monserrato, Italy  INFN, Sezione di Cagliari, Cittadella
Universitaria, 09042 Monserrato, Italy

###### Abstract

We derive the form of the metric for static, nonsingular black holes with a de
Sitter core, representing a deformation of the Schwarzschild solution, by
assuming that the gravitational sources describe a flow between two conformal
points, at small and great distances. The resulting black-hole metric turns
out to be a particular case of the Fan $\&$ Wang metric, whose parameters have
been recently constrained by using the data of the S$2$ star orbits around the
galactic centre SgrA∗.

##  I Introduction

In recent times there has been renewed interest for asymptotically flat,
nonsingular black-hole solutions, which deform the Schwarzschild solution at
subleading order [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16].
Among them, the most interesting class of solutions is represented by
nonsingular black holes with a de Sitter (dS) core [13]. These black-hole
solutions are of interest for several reasons. Firstly, they allow to
circumvent Penrose’s theorem [17] by removing the classical singularity at
$r=0$. Secondly, they are solutions of Einstein’s field equations sourced by
an anisotropic fluid, effectively encoding the deviations responsible for the
smearing of the singularity. These deviations are described by an external
length scale $\ell$, which represents an additional “hair” of the black hole.
An intriguing possibility is that it could be also of superplanckian origin
[13]. Thirdly, they can play the role of black-hole “mimickers”, i.e., they
are indistinguishable from the Schwarzschild solution at great distances, but
could nonetheless lead to observable deviations from the latter, for instance
in the orbits of massive particles and photons and in the gravitational-wave
spectrum (see Refs. [13, 15] and references therein). Last but not least, they
could be very useful in solving the information puzzle arising during black-
hole evaporation [18].

On the other hand, such models suffer from a strong limitation, which is
purely theoretical. We can obtain them using general relativity (GR) with
anisotropic fluids as sources, but the underlying microscopic physics is
mostly unknown. This difficulty becomes particularly severe in those cases in
which the deformations from the usual Schwarzschild solution have
superplanckian origin [13]. The consequence is that we have a huge degeneracy,
giving rise to a broad class of metric solutions, which all describe
nonsingular black holes with a dS core. The coarse-grained description in term
of the anisotropic fluid is not stringent. The equation of state (EOS)
relating the radial pressure with the energy density,
$p_{\parallel}=p_{\parallel}(\rho)$, and the density profile $\rho(r)$,
interpolating between small and large $r$, are very weakly constrained.

Probably these difficulties are hinting at the fact that the microscopic
explanation of this kind of solutions cannot be found by merely looking at GR,
which only allows for an effective description of the sources in terms of
anisotropic fluids. What is needed, then, is a general guiding principle to
select the physically relevant solutions.

In this paper, we will use conformal symmetries, including Lorentz boosts, as
the guiding principle to remove the above mentioned degeneracy of solutions.
There is striking evidence that conformal symmetry could be a crucial feature
of any quantum theory of gravity. It is the pillar of the AdS/CFT
correspondence [19, 20] and is also crucial for most microscopic derivations
of the Bekenstein-Hawking black-hole entropy [21, 22, 23, 24, 25].

We will select the EOS for the anisotropic fluid using covariance under
rotations and radial Lorentz boosts. In order to fix the density profile
$\rho(r)$, we will use conformal symmetries, motivated by the role played by
the latter in black-hole physics. In particular, we will assume that $\rho(r)$
describes the flow of matter fields between two conformal points where the
local matter contribution is negligible, near $r=0$ and at $r=\infty$. These
are described, respectively, by the dS spacetime [26, 27] and a CFT in
Minwkoski spacetime. The existence of the latter, in particular, represents a
strong assumption, which is, however, motivated by the AdS/CFT correspondence.
We will show that these requirements select a specific spacetime metric, i.e.,
a particular case of the Fan $\&$ Wang metric [8], which represents the one
having the strongest subleading deviations from Schwarzschild at infinity and
which was recently constrained by S$2$ observational data [15].

The present paper is organized as follows.

In Section II, we will briefly review some basic features of nonsingular black
holes with a dS core and we will fix the EOS using Lorentz symmetries. In
Section III, we discuss the conformal symmetries we use to constrain the
density profile and derive the form of the metric. In Section IV, we give a
simple example for the source in terms of nonlinear electrodynamics. Finally,
in Section V we summarize our results.

##  II Nonsingular black holes with a de Sitter core

Due to Birkhoff’s theorem, any nonstandard GR black-hole solution has to be
obtained from Einstein’s equations sourced by a nonzero stress-energy tensor.
The most general one is that of an anisotropic fluid, which has been widely
adopted to effectively parametrize several different effects and deviations
from GR phenomenology, both at small and cosmological scales (for an
incomplete list, see, e.g., Refs. [28, 29, 30, 31, 32, 33, 34, 13, 10, 11, 16]
and references therein). This fluid is described by the stress-energy
tensor111Throughout the entire paper, we will use natural units in which
$\hbar=c=1$ and we will use $G=\ell_{\text{P}}^{2}$ interchangeably.

$T_{\mu\nu}=\left(\rho+p_{\perp}\right)u_{\mu}u_{\nu}+p_{\perp}g_{\mu\nu}+\left(p_{\parallel}-p_{\perp}\right)w_{\mu}w_{\nu}\,,$
(1)

where $\rho$, $p_{\parallel}$ and $p_{\perp}$ are the energy density, the
radial and perpendicular components of the pressure, respectively, while
$u_{\mu}$ and $w_{\mu}$ are a time-like and space-like 4-vectors,
respectively, satisfying the relations $u^{\mu}u_{\mu}=-w^{\mu}w_{\mu}=-1$. A
particular choice of the EOS $p_{\parallel}=p_{\parallel}(\rho)$ determines
and characterizes the solutions, whereas $p_{\perp}$ is determined by the
covariant conservation of the stress-energy tensor (see, e.g., Refs. [1, 5, 2,
35, 36, 10, 8, 13, 16, 6, 37] and references therein).

Requiring symmetry properties of the fluid constrains the free functions
$\rho$ and $p_{\parallel}$ in Eq. 1. In the following, we focus on
spherically-symmetric models. Given this, we consider fluids whose dynamic
equations are covariant under rotations in the $[\theta,\phi]$ plane and under
Lorentz boosts in the $[t,r]$ directions. Even if every stress-energy tensor
is covariant under boosts and rotation in general frames, these choices select
invariance for the particular class of radially-moving observers. The physical
consequence of this choice is that a stress-energy tensor satisfying these
properties is identified as describing a well-defined spherically symmetric
and Lorentz invariant vacuum [38, 2, 4]. Its structure reads

$\displaystyle T^{\theta}_{\theta}=T^{\phi}_{\phi}\,;$ (2a) $\displaystyle
T^{t}_{t}=T^{r}_{r}\,.$ (2b)

Equation 2b, in particular, fixes the EOS to be

$p_{\parallel}=-\rho\,.$ (3)

Notice that, apart from being dictated by symmetry arguments, this EOS is
quite natural, as well as simple, in an emergent gravity framework (see, e.g.,
Refs. [13, 39]). Additionally, it appears in several physical contexts, such
as the simplest form of dark energy (the cosmological constant, in the
isotropic case), exotic compact objects [40, 36] or solutions of GR coupled
with nonlinear electrodynamics [41, 42].

$p_{\perp}$, instead, is entirely determined by the covariant conservation of
the stress-energy tensor.

$p_{\perp}=-\rho-\frac{r}{2}\rho^{\prime}\,.$ (4)

With Eq. 3, the general solution of Einstein’s equations, sourced by Eq. 1 and
written in Schwarzschild coordinates $(t,r,\theta,\varphi)$, reads (see, e.g.,
Ref. [13])

$\displaystyle\text{d}s^{2}=-f(r)\text{d}t^{2}+\frac{\text{d}r^{2}}{f(r)}+r^{2}\text{d}\Omega^{2}\,;$
(5a) $\displaystyle f(r)=1-\frac{2Gm(r)}{r}\,,\quad
m(r)=4\pi\int_{0}^{r}\text{d}\tilde{r}\,\tilde{r}^{2}\,\rho(\tilde{r})\,,$
(5b)

where $m(r)$ the Misner-Sharp mass of the system.

Note that we could have adopted a different parametrization of the metric
(5a), resulting therefore in a different definition of the radial coordinate.
For example, the line element written in isotropic coordinates

$\text{d}s^{2}=-A(r^{\prime})\text{d}t^{2}+B(r^{\prime})\left(\text{d}r^{\prime
2}+r^{\prime 2}\text{d}\Omega^{2}\right)$ (6)

is related to Eq. 5a by the coordinate transformation
$\text{d}r^{\prime}/r^{\prime}=\text{d}r/\left(r\sqrt{f}\right)$, relating the
two radial coordinates. All the relevant physical results of the present paper
are essentially independent of the particular parametrization of the radial
coordinate. The freedom in the choice of the latter would simply amount to a
different realization of the same symmetries analyzed below, which, depending
on the chosen coordinates, could have an intricate form. In what follows,
therefore, we will limit ourselves to considering the parametrization (5a),
which allows for a simple realization of such symmetries and is the one most
widely used in the literature.

Considering, thus, the system (5a)-(5b), Eq. 3 fixes $p_{\parallel}$ and
$p_{\perp}$, but leaves the density profile and, hence, the form of $m(r)$,
completely unconstrained.

On the other hand, the behavior of $\rho$ at $r=0$ and $r\to\infty$ can be
determined by stringent physical considerations.

In light of the particular form of the EOS (3) and requiring the absence of
spacetime singularities, we expect that, whenever matter contribution is
negligible (at $r\sim 0$ and $r\to\infty$), the source of gravity is given by
an approximately isotropic fluid, which gives $\rho\sim\text{constant}$ using
Eqs. 3 and 4. Assuming the validity of the weak energy condition we have
$\rho\geq 0$. From Eq. 3, it follows now that, in the core, at $r\sim 0$, the
spacetime behaves as a dS spacetime 222See, however, Ref. [10] for a model
with an asymptotic Minkowski core.. The density reads

$\rho\sim\frac{1}{4\pi\ell_{\text{P}}^{2}\,L^{2}}\,,$ (7)

where $L$ represents the dS length in the core. This behavior at the center
breaks the strong energy condition, allowing to circumvent Penrose’s theorem
and to replace the classical singularity region with a completely regular
spacetime [1, 5, 2, 8, 13, 6]. Equation 7 constrains the Misner-Sharp mass
(5b) to behave extensively, as $m(r)\sim r^{3}/\ell_{\text{P}}^{2}L^{2}$, near
$r=0$.

If one considers the cosmological regime, dominated by a cosmological constant
at large $r$, one can still have $\rho\sim\text{constant}\neq 0$, and hence a
dS behavior. Since we are considering isolated bodies, we discard a dS
asymptotics, assuming that the density profile decays sufficiently rapidly to
zero at $r\to\infty$, so that we have asymptotically flat solutions. We also
note that, at infinity, $p_{\parallel}=p_{\perp}\to 0$ according to Eq. 4.

Moreover, the asymptotic value $M$ of $m(r)$ appears as an integration
constant in Eq. 5b, so that imposing a Schwarzschild behavior for the solution
at $r\to\infty$ implies

$\rho\sim r^{c}\,,\quad\text{with}\quad c<-3\,.$ (8)

Thus, our physically motivated “boundary conditions” at $r=0$ and $r\to\infty$
imply that the function $\rho$ interpolates between the constant value (7)
near $r=0$ to Eq. 8 at $r\to\infty$.

A major drawback of this construction, however, is that, as a consequence of
the freedom in choosing both the exponent $c$ in Eq. 8 and the interpolating
density profile $\rho(r)$, the model is not unique, but it exists an infinite
class of models which realize the same flow [13]. Particularly relevant
examples, discussed in the literature, are the Fan $\&$ Wang [8], Bardeen [1],
Hayward [5] nonsingular black holes, and black holes with Gaussian cores [3,
6] (see also Ref. [13] and references therein).

In the following, we will see that requiring some conformal symmetries and
scaling properties for the density profile and for the field generating this
energy density will select a particular metric belonging to this general
class.

##  III Conformal symmetries and scalar field description

Although GR is not a conformal field theory (see, however, Ref. [27]), it is
known that these symmetries could play an important role for particular
spacetime backgrounds, like, e.g., the anti de Sitter (or also the dS)
spacetime, for which they take the form of holographic correspondences [19,
20, 43]. Moreover, there is some evidence that conformal symmetry could
regularize the short-distance behavior of gravity, by generating an UV fixed
point, which is at the base of the asymptotic safety scenario [44].

Conformal symmetry plays also an important role for black holes, in particular
in the description of their near-horizon physics. It has been widely used to
give a microscopic derivation of the Bekenstein-Hawking entropy [21, 22, 23,
25]. Moreover, extremal black-hole background geometries (e.g., BPS states)
typically describe the flow between different conformal points, or a conformal
point and a flat spacetime [45, 46].

Finally, conformal symmetries are very important also for nonsingular black
holes with a dS core. In fact, the $4$D dS spacetime is endowed with a scale
invariance [27] and, in particular, invariance under transformations induced
by the conformal group SO$(2,4)$ [26] 333This becomes evident when embedding
dS spacetime in $\mathds{R}^{1,4}$ and writing it in the flat slicing.. For
the nonsingular black holes under consideration here, this scale symmetry
holds only in the dS core and it is broken at greater distances, when
localized matter begins to dominate [13]. On the other hand, the presence of
the dS core implies that, for some values of the hair $\ell$, the black hole
has necessarily two horizons, which, for a critical value of $\ell$, merge
into a single one. This produces an extremal configuration, whose near-horizon
geometry has an AdS2 factor, with an associated dual, near-horizon conformal
symmetry [13].

These considerations strongly suggest that the density $\rho(r)$ sourcing our
black hole could generate a flow between a conformal point near $r=0$,
described by the dS spacetime, and some conformal field theory in the
$r=\infty$ region. The scale invariance is broken during the flow by the
nucleation of a local mass $M$, with a related generation of an intermediate
scale $\ell$ [13]. The latter represents an additional “hair” of these models,
and allows to realize the interpolation between the small $r\sim 0$ and the
large scales $r\to\infty$.

Lacking a fundamental microscopic description of our nonsingular black holes,
we are unable to exactly identify the field content of the conformal matter
sourcing the black hole in the $r\to\infty$ region. However, scale symmetry
strongly constraints the form of $\rho$ in this regime. It must transform with
definite weight $\Delta$ under dilatations $r\to\omega r$: $\rho(\omega
r)=\omega^{\Delta}\rho(r)$. For conformal field theories, the scaling
dimension $\Delta$ must be equal to the engineering dimensions, $\Delta=-4$,
in such a way that the theory does not contain dimensional constants. This
fixes the exponent $c$ in the asymptotic behavior (8), so that we have

$\rho(r)=\frac{\alpha}{4\pi}\frac{1}{r^{4}}\,,$ (9)

where $\alpha$ is a dimensionless constant. This scaling is typical of the
energy density of conformal matter fields in four dimensional Minkowski
spacetime, like, for instance, a set of free masseless scalar fields. We are
therefore assuming that, in the asymptotic $r=\infty$ region, if we neglect
the contribution of the localized matter $M$, our system is well described by
a solution of GR given by a CFT in Minkowski spacetime, whose energy density
corresponds to Eq. 9. Naïvely, this energy density characterizes a system of
$N$ quanta inside a sphere of radius $r$. Each mode has a typical Compton
energy $E\sim r^{-1}$, so that the total energy density is $\rho\sim N/(r\cdot
r^{3})=N/r^{4}$.

The density profile (9) diverges at $r=0$. This is due to modes with
arbitrarily short wavelength contributing to the spectrum. According to our
assumption on the presence of a dS behavior at $r=0$, this singularity is,
however, not physical, because the density $\rho$ must interpolate between the
constant value (7) at $r=0$ and Eq. 9 at great distances.

The simplest way to regularize this divergent behavior is through a
translation of the radial coordinate $r\to r+\ell$, which moves the
singularity to nonphysical negative values of the radial coordinate $r$

$\rho(r)=\frac{\alpha}{4\pi}\frac{1}{(r+\ell)^{4}}\,.$ (10)

This introduces a length scale related to the local mass $M$ and to the dS
length $L$, which is the physical source of the breaking of the scale
symmetry. Evaluating Eq. 10 in $r=0$, comparing it with Eq. 7 and considering
the Schwarzschild limit $m(r)\to M$ as $r\to\infty$, we can easily identify
the dimensionless constant $\alpha$ and write the hair $\ell$ in terms of the
Schwarzschild radius $R_{\text{S}}=2\ell_{\text{P}}^{2}M$ and of $L$

$\alpha=\frac{\ell^{4}}{\ell_{\text{P}}^{2}\,L^{2}}\,,\quad\ell\sim
R_{\text{S}}^{1/3}\,L^{2/3}\,.$ (11)

The second equation, in particular, gives a universal scaling for every
geometry interpolating between the dS at $r=0$ and the Schwarzschild
spacetimes at great distances (see Ref. [13]). Specifically, Eq. 11 represents
a universal relation between $\ell$ and the black-hole mass $M$.

One can now easily find the mass function, using Eqs. 10 and 5b

$m(r)=\frac{Mr^{3}}{(r+\ell)^{3}}\,,$ (12)

which gives a particular case of the Fan $\&$ Wang model, investigated in
details in Ref. [8]. This model is characterized by strong, order $1/r^{2}$
corrections to the Schwarzschild solution at infinity. Additionally, the
parameter $\ell$ in this model was recently constrained by S$2$ observational
data [15].

In the next subsection, we will show that the result (10), which is dictated
by an Occam razor argument, can be derived using, again, conformal symmetry
arguments.

###  III.1 Scalar field description

The regularization proposed above is obviously not unique. A different choice
corresponds to different flows between the $r=0$ and $r=\infty$ points, and to
different patterns of the symmetry breaking. The most general solution
compatible with the boundary conditions (7) and (9), and with an analytic
behavior at $r\to\infty$ is $\rho\propto(P_{n})^{\gamma}/(P_{m})^{\delta}$,
with $P_{n}$, $P_{m}$ polynomials of degrees $n$ and $m$, respectively, and
$n\gamma-m\delta=-4$ (to guarantee the scaling (9) at great distances). This
is a remarkable particular case of the more general formula sourcing regular
fractional models

$\rho(r)=\frac{\alpha}{4\pi}\frac{\ell^{m-1}}{\left(r^{m}+\ell^{m}\right)^{\frac{3}{m}+1}}\,.$
(13)

Indeed, once used in Eqs. 5a and 5b, it gives for $m=1,\,2,\,3$, the Fan $\&$
Wang, Bardeen and Hayward black holes, respectively.

To gain further insights into the details of the two CFTs living at $r=0$ and
$r=\infty$, let us assume, for simplicity, that there is a regime in which the
flow between these two conformal points can be described by a static scalar
field $\Phi$, which is sourced by the density $\rho$. In a static and
spherically symmetric background, it will satisfy the Poisson equation

$\nabla^{2}\Phi=4\pi\rho\,.$ (14)

One can now easily find, using Eqs. 7 and 9, the asymptotic solutions of Eq.
14 near $r=0$ and $r\to\infty$,

$\Phi(r)\sim\,\begin{cases}&r^{2}/\ell^{2}\,,\,\quad\,\,\text{for}\,\quad
r\sim 0\\\ &r^{-2}\,,\qquad\,\text{for}\,\quad r\to\infty\end{cases}\,,$ (15)

where we have neglected the constant and $1/r$ terms in the $r\to\infty$
behavior, which are related to the presence of the mass $M$. As expected in
the flow from $r=\infty$ to $r=0$, the scaling dimension of $\Phi$ changes
from its engineering one $\Delta=-2$ to $\Delta=2$, which is associated with a
constant $\rho$.

An important feature of the two conformal points, which is immediately evident
in Eq. 15, is that they are mapped one into the other by the inversion

$r\to\frac{\ell^{2}}{r}\,.$ (16)

Discrete symmetries, changing small with large radii, are common in string
theory, where they are called $T$-dualities. They have been already used in
the past to investigate nonsingular black holes [47, 48].

The inversion can be used in combination with translations to produce special
conformal transformations, which, together with dilatations and translations,
generate the conformal group (isomorphic to the $\text{SL}(2,\mathds{R})$
group) realized here in one dimension as

$r\to\omega\,r,\quad r\to\frac{r}{1-\nu r}\,\quad r\to r+\sigma$ (17)

where $\omega$, $\nu$, $\sigma$ are the group parameters 444We stress again
that a different parametrization of the radial coordinate would simply imply a
different realization of the transformations (17), with $r$ replaced by a
particular function determined by the coordinate transformation (see the
discussion in Section II)..

A generic flow, for instance the one described by Eq. 13, will preserve
neither the scaling behavior for $\Phi$ nor the symmetry under inversion.
However, we can select a particularly symmetric profile for $\rho$, such that
the solution for $\Phi$ preserves at least part of the conformal symmetries,
in particular the scaling with $\Delta=2$ attained in the $r=0$ conformal
point. One can show that this happens if we choose the simple profile for
$\rho$ given by Eq. 10. Integrating the Poisson equation (14), we get

$\Phi(r)=\frac{\alpha\,}{6\,\ell^{2}}\frac{r^{2}}{(r+\ell)^{2}}\,.$ (18)

One can now check that the field $\Phi$ (18) transforms as

$\Phi\to\omega^{2}\,\Phi$ (19)

under a conformal transformation of the form

$r\to\omega\frac{r}{1-\nu r}\,,$ (20)

with $\omega\equiv 1+\nu\ell$, which represents the composition of a
dilatation and a special conformal transformation.

It is important to notice that Eq. 18 does not arise as the Newtonian limit of
the full GR solution with the mass function (12). The EOS (3), indeed,
prevents the weak-field limit from being performed together with the usual
nonrelativistic limit, and a Newtonian fluid, with $\rho\gg p_{\parallel}$,
$p_{\perp}$, from being considered. We can still perform a weak field limit,
which gives the Poisson equation, sourced however by the _active mass_
$\rho+p_{\parallel}+2p_{\perp}$. Using Eqs. 3 and 4, together with the profile
(13) (with $m=1$) yields the potential $\tilde{\Phi}=-GMr^{2}/(r+\ell)^{3}$.

##  IV Nonlinear Electrodynamics

It is interesting to note that the large scale behavior $r^{-4}$ of Eq. 13,
and the related scale invariance, could be explained in terms of the embedding
of these regular models as solutions of GR coupled with nonlinear
electrodynamics [41]. The action for such theory is

$\mathcal{S}=\int\text{d}^{4}x\,\sqrt{-g}\,\left[\frac{R}{16\pi
G}-\mathscr{L}(\mathcal{F})\right]\,,$ (21)

where $R$ is the Ricci scalar, while
$\mathcal{F}=\frac{1}{4}F^{\mu\nu}F_{\mu\nu}$ is the trace of the
electromagnetic potential. $\mathscr{L}$ is, in general, a nonlinear function
of $\mathcal{F}$. Maxwell’s theory is of course recovered in the linear case
$\mathscr{L}\propto\mathcal{F}$. If we compute the stress-energy tensor
related to $\mathscr{L}$, we see that it naturally satisfies the EOS (3) and
that $\rho(r)=\mathscr{L}(\mathcal{F})$. We can now combine this with Eq. 13
and the magnetic monopole solution of Maxwell’s equations

$\mathcal{F}=\frac{q_{\text{m}}^{2}}{2r^{4}}\,,$ (22)

which gives the lagrangian

$\mathscr{L}\left(\mathcal{F}\right)=\frac{\alpha}{4\pi}\frac{\mathcal{F}}{\left[\ell^{\beta}\mathcal{F}^{\beta/4}+2^{-\beta/4}q_{\text{m}}^{\beta/2}\right]^{4/\beta}}\,.$
(23)

We see now that the particular large-scale conformal scaling $r^{-4}$ can be
explained by the fact that the subclass of models described by Eq. 23 reduces
to the standard Maxwell theory in the weak field limit $\mathcal{F}\to 0$,
which is also conformally invariant.

##  V Summary and Outlook

One of the most unsatisfactory aspect of nonsingular black-hole solutions is
that, although we can generate them using anisotropic fluids as sources, their
physical origin in terms of elementary fields is mostly unknown. This is
particularly true if one considers nonsingular black holes in which the
deformations from the usual Schwarzschild solution have superplanckian origin
[13, 15]. An unpleasant consequence of this lack of knowledge is the existence
of a large number of solutions. Although it is possible that, in the near
future, astrophysical and gravitational waves data may be used to
select/exclude models [13, 15], some theoretical guiding principle is more
than welcome.

It is likely that these difficulties are indicating that the microscopic
origin of this kind of solutions cannot be found in a GR framework or its
extensions (see Ref. [49]), which allows only for a coarse-grained description
in terms of anisotropic fluids. For this reason, it is important to look at
general guiding principles, like symmetries, which are expected to underpin
the classical GR description.

In this paper, we have adopted this philosophy to constrain the broad class of
nonsingular black-hole models with a dS core. We have used conformal
symmetries, which are believed to be a crucial ingredient of any quantum
theory of gravity, as a selecting principle to single out the physically
relevant nonsingular black-hole solution.

We have found that the conformal symmetry selects a particular case of the Fan
$\&$ Wang metric, which has been recently investigated and constrained using
data of the orbits of the S$2$ star around the SgrA∗ black hole.

Obviously, the use of conformal symmetry to select solutions is only a first
step. Understanding the microphysics from which these symmetries originate is
the next important task.

## References

  * Bardeen [1968] J. M. Bardeen, Proceedings of the International Conference GR5, Tbilisi University Press (1968).
  * Dymnikova [1992] I. Dymnikova, Gen. Rel. Grav. 24, 235 (1992).
  * Bonanno and Reuter [2000] A. Bonanno and M. Reuter, Phys. Rev. D 62, 043008 (2000), arXiv:hep-th/0002196 .
  * Dymnikova and Galaktionov [2005] I. Dymnikova and E. Galaktionov, Class. Quant. Grav. 22, 2331 (2005), arXiv:gr-qc/0409049 .
  * Hayward [2006] S. A. Hayward, Phys. Rev. Lett. 96, 031103 (2006), arXiv:gr-qc/0506126 .
  * Nicolini  _et al._ [2006] P. Nicolini, A. Smailagic, and E. Spallucci, Phys. Lett. B 632, 547 (2006), arXiv:gr-qc/0510112 .
  * Ansoldi [2008] S. Ansoldi,  _Conference on Black Holes and Naked Singularities_ , (2008), arXiv:0802.0330 [gr-qc] .
  * Fan and Wang [2016] Z.-Y. Fan and X. Wang, Phys. Rev. D 94, 124027 (2016), arXiv:1610.02636 [gr-qc] .
  * Frolov [2016] V. P. Frolov, Phys. Rev. D 94, 104056 (2016), arXiv:1609.01758 [gr-qc] .
  * Simpson and Visser [2019a] A. Simpson and M. Visser, Universe 6, 8 (2019a), arXiv:1911.01020 [gr-qc] .
  * Simpson and Visser [2019b] A. Simpson and M. Visser, JCAP 02, 042 (2019b), arXiv:1812.07114 [gr-qc] .
  * Carballo-Rubio  _et al._ [2020] R. Carballo-Rubio, F. Di Filippo, S. Liberati, and M. Visser, Phys. Rev. D 101, 084047 (2020), arXiv:1911.11200 [gr-qc] .
  * Cadoni  _et al._ [2022a] M. Cadoni, M. Oi, and A. P. Sanna, Phys. Rev. D 106, 024030 (2022a), arXiv:2204.09444 [gr-qc] .
  * Sebastiani and Zerbini [2022] L. Sebastiani and S. Zerbini, (2022), arXiv:2206.03814 [gr-qc] .
  * Cadoni  _et al._ [2022b] M. Cadoni, M. De Laurentis, I. De Martino, R. Della Monica, M. Oi, and A. P. Sanna, (2022b), arXiv:2211.11585 [gr-qc] .
  * Akil  _et al._ [2022] A. Akil, M. Cadoni, L. Modesto, M. Oi, and A. P. Sanna, (2022), arXiv:2211.01657 [gr-qc] .
  * [17] R. Penrose, Phys. Rev. Lett. 14 (1965), 57-59 10.1103/PhysRevLett.14.57.
  * Cadoni  _et al._ [2023] M. Cadoni, M. Oi, and A. P. Sanna, (2023), arXiv:2303.05557 [hep-th] .
  * Maldacena [1998] J. M. Maldacena, Adv. Theor. Math. Phys. 2, 231 (1998), arXiv:hep-th/9711200 .
  * Witten [1998] E. Witten, Adv. Theor. Math. Phys. 2, 253 (1998), arXiv:hep-th/9802150 .
  * [21] A. Strominger and C. Vafa, Phys. Lett. B 379 (1996), 99-104 10.1016/0370-2693(96)00345-0, arXiv:hep-th/9601029 .
  * Strominger [1998] A. Strominger, JHEP 02, 009 (1998), arXiv:hep-th/9712251 .
  * Cadoni and Mignemi [1999a] M. Cadoni and S. Mignemi, Phys. Rev. D 59, 081501 (1999a), arXiv:hep-th/9810251 .
  * Cadoni and Mignemi [1999b] M. Cadoni and S. Mignemi, Nucl. Phys. B 557, 165 (1999b), arXiv:hep-th/9902040 .
  * [25] S. Carlip, Phys. Rev. Lett. 88 (2002), 241301 10.1103/PhysRevLett.88.241301, arXiv:gr-qc/0203001 .
  * Keane and Barrett [2000] A. J. Keane and R. K. Barrett, Class. Quant. Grav. 17, 201 (2000), arXiv:gr-qc/9907002 .
  * Cadoni [2006] M. Cadoni, Phys. Lett. B 642, 525 (2006), arXiv:hep-th/0606274 .
  * Bayin [1986] S. S. Bayin, Astrophys. J. 303, 101 (1986).
  * Herrera and Santos [1997] L. Herrera and N. O. Santos, Phys. Rept. 286, 53 (1997).
  * DeBenedictis  _et al._ [2006] A. DeBenedictis, D. Horvat, S. Ilijic, S. Kloster, and K. S. Viswanathan, Class. Quant. Grav. 23, 2303 (2006), arXiv:gr-qc/0511097 .
  * Aluri  _et al._ [2013] P. Aluri, S. Panda, M. Sharma, and S. Thakur, JCAP 12, 003 (2013), arXiv:1210.3159 [gr-qc] .
  * Raposo  _et al._ [2019] G. Raposo, P. Pani, M. Bezares, C. Palenzuela, and V. Cardoso, Phys. Rev. D 99, 104072 (2019), arXiv:1811.07917 [gr-qc] .
  * Cadoni  _et al._ [2018] M. Cadoni, R. Casadio, A. Giusti, W. Mück, and M. Tuveri, Phys. Lett. B 776, 242 (2018), arXiv:1707.09945 [gr-qc] .
  * Cadoni  _et al._ [2020] M. Cadoni, A. P. Sanna, and M. Tuveri, Phys. Rev. D 102, 023514 (2020), arXiv:2002.06988 [gr-qc] .
  * Chan  _et al._ [2011] R. Chan, M. F. A. da Silva, and P. Rocha, Gen. Rel. Grav. 43, 2223 (2011), arXiv:1009.4403 [gr-qc] .
  * Beltracchi and Gondolo [2019] P. Beltracchi and P. Gondolo, Phys. Rev. D 99, 044037 (2019), arXiv:1810.12400 [gr-qc] .
  * Kumar and Bharti [2021] J. Kumar and P. Bharti, (2021), arXiv:2112.12518 [gr-qc] .
  * Gliner [1966] E. B. Gliner, Soviet Journal of Experimental and Theoretical Physics 22, 378 (1966).
  * Jusufi [2022] K. Jusufi, (2022), arXiv:2208.12979 [gr-qc] .
  * [40] P. Martin Moruno, N. Montelongo Garcia, F. S. N. Lobo, and M. Visser, JCAP 03 (2012), 034 10.1088/1475-7516/2012/03/034, arXiv:1112.5253 [gr-qc] .
  * Bronnikov [2001] K. A. Bronnikov, Phys. Rev. D 63, 044005 (2001), arXiv:gr-qc/0006014 .
  * Dymnikova [2004] I. Dymnikova, Class. Quant. Grav. 21, 4417 (2004), arXiv:gr-qc/0407072 .
  * Strominger [2001] A. Strominger, JHEP 10, 034 (2001), arXiv:hep-th/0106113 .
  * Bonanno  _et al._ [2020] A. Bonanno, A. Eichhorn, H. Gies, J. M. Pawlowski, R. Percacci, M. Reuter, F. Saueressig, and G. P. Vacca, Front. in Phys. 8, 269 (2020), arXiv:2004.06810 [gr-qc] .
  * Ferrara and Kallosh [1996] S. Ferrara and R. Kallosh, Phys. Rev. D 54, 1514 (1996), arXiv:hep-th/9602136 .
  * Gubser and Rocha [2009] S. S. Gubser and F. D. Rocha, Phys. Rev. Lett. 102, 061601 (2009), arXiv:0807.1737 [hep-th] .
  * Smailagic  _et al._ [2003] A. Smailagic, E. Spallucci, and T. Padmanabhan, (2003), arXiv:hep-th/0308122 .
  * Nicolini  _et al._ [2019] P. Nicolini, E. Spallucci, and M. F. Wondrak, Phys. Lett. B 797, 134888 (2019), arXiv:1902.11242 [gr-qc] .
  * Knorr and Platania [2022] B. Knorr and A. Platania, Phys. Rev. D 106, L021901 (2022), arXiv:2202.01216 [hep-th] .


Experimental Resultssec:exp

Implementation Details Our training process includes two stages: 1) Networks
$\mathcal{F}$, $\mathcal{C}$, $\mathcal{T}$, $\mathcal{W}$ are pre-trained on
$3$D data. 2) $\mathcal{E}$ and $\mathcal{T}$ are trained or fine-tuned with
real images. The encoder $\mathcal{E}$ is implemented as a ResNet-$50$.
Networks $\mathcal{F}$, $\mathcal{C}$, $\mathcal{T}$ and $\mathcal{W}$ are
MLPs. In experiments, we set $L_{cloth}=L_{tex}=512$, $L_{id}=4,096$,
$L_{f}=L_{h}=256$, $K=24$, $n=200,000$. We implement in Pytorch and use Adam
optimizer in both stages.

Person ReID

Metric For person ReID, we follow the standard retrieval accuracy metrics,
namely the Cumulative Matching Characteristics (CMC) and mean average
precision (mAP).

Baseline We compare our method with eight SoTA person ReID methods: Two-Stream
zheng2017discriminatively, MLFN chang2018multi, HACNN li2018harmonious, Part-
Aligned suh2018part, PCB sun2018beyond, TriNet hermans2017defense, MGN
wang2018learning, DG-Net zheng2019joint, and five SoTA cloth-changing re-ID
methods: ReIDCaps huang2019beyond, $3$DSL chen2021learning, RCSAnet
huang2021clothing, FSAM hong2021fine and CAL gu2022clothes.

Results on Cloth-changing Person ReID datasets

Datasets We test on four popular cloth-changing ReID datasets: Celeb-
reID/Celeb-reID-light huang2019beyond,huang2019celebrities, PRCC
yang2019person, LTCC shu2021large and the recent CCVID dataset
gu2022clothes,zhang2019gait.

1!  2*Method 2* Backbone Celeb-reID Celeb-reID-light mAP Rank$1$ mAP Rank$1$
Two-Stream [zheng2017discriminatively] ResNet-$50$ $7.8$ $36.3$ - - MLFN
[chang2018multi] * $6.0$ $41.4$ $6.3$ $10.6$ HACNN [li2018harmonious] * $9.5$
$47.6$ $11.5$ $16.2$ Part-Aligned [suh2018part] GoogLeNetred$\times 2$ $6.4$
$19.4$ - - PCB [sun2018beyond] ResNet-$50$ $8.2$ $37.1$ - - MGN
[wang2018learning] ResNet-$50$ $10.8$ $49.0$ $13.9$ $21.5$ DG-Net
[zheng2019joint] ResNet-$50$ $10.6$ $50.1$ $12.6$ $23.5$ _cloth-changing
person ReID methods_ ReIDCaps- [huang2019beyond] DenseNet-$121$ $9.8$ $51.2$
$11.2$ $20.3$ ReIDCaps [huang2019beyond] DenseNet-$121$red$\times 6$ $15.8$
$63.0$ $19.0$ $33.5$ RCSAnet [huang2021clothing] DenseNet-$121$red$\times 2$
$11.9$ $55.6$ $16.7$ $29.5$ CAL [gu2022clothes] ResNet-$50$ $13.7$ $59.2$
$18.5$ $33.6$ LightCyan 3DInvarReID ResNet-$50$ $11.8$ $55.2$ $15.0$ $30.1$
LightCyan 3DInvarReID# ResNet-$50$ $15.2$ $61.2$ $21.8$ $37.0$ rowred
2*ReIDCaps+ 2*- 2*18.4 2*65.5 2*25.7 2*42.2 rowred 3DInvarReID#

Table : Comparison with SoTA on Celeb-reID and Celeb-reID-light datasets (%).
‘*’ indicates that the backbone is designed by the authors. The red number
means the total number of models. 3DInvarReID#’s weights are initialized using
the CAL model.

Results on Celeb-reID and Celeb-reID-light As reported in Tab. , our
3DInvarReID (Ours) outperforms all the general person re-ID baselines on both
datasets. Considering methods using a single network, we also outperform the
cloth-changing baseline, ReIDCaps- [huang2019beyond]. More importantly, to
investigate the complementarity between our learned $3$D shape features and
existing $2$D features, we fuse our method with cloth-changing baselines by
simple summation at the score level. By fusing with ReIDCaps
[huang2019beyond], our method improves the Rank$1$ accuracy on Celeb-reID from
$63.0\%$ to $65.5\%$ and from $33.5\%$ to $42.2\%$ on Celeb-reID-light. These
results clearly demonstrate that the $3$D shape features learned from our
method are both discriminative and complementary to the $2$D features,
indicating the effectiveness of our proposed approach for person ReID,
particularly under cloth-changing scenarios.

0.95!  2*Method 2* Backbone LTCC PRCC mAP Rank$1$ mAP Rank$1$ HACNN
[li2018harmonious] * $9.3$ $21.6$ $-$ $21.8$ PCB [sun2018beyond] ResNet-$50$
$10.0$ $23.5$ $38.7$ $41.8$ $3$DSL [chen2021learning] ResNet-$50$red$\times 2$
$14.8$ $31.2$ $-$ $51.3$ FSAM [hong2021fine] ResNet-$50$ $16.2$ $38.5$ $-$
$54.5$ CAL [gu2022clothes] ResNet-$50$ $18.0$ $40.1$ $55.8$ $55.2$ LightCyan
3DInvarReID ResNet-$50$ $16.7$ $37.8$ $52.5$ $51.6$ rowred 2*CAL+ 2*- 2*18.9
2*40.9 2*57.2 2*56.5 rowred 3DInvarReID rowlightblue _CAL_ ResNet-$50$ $2.8$
$3.8$ $20.3$ $31.6$ rowblue _3DInvarReID_ ResNet-$50$ $2.8$ $5.1$ $20.1$
$34.6$ rowblue _3DInvarReID #_ ResNet-$50$ 2.8 5.6 21.4 40.7

Table : Comparison with SoTA cloth-changing person ReID methods on the LTCC
and PRCC datasets (%). Models highlighted in pink are trained on the Celeb-
reID dataset. 3DInvarReID#’s weights are initialized using the CAL model.

0.95!  2*Method 2* Backbone General Cloth-changing mAP Rank$1$ mAP Rank$1$
TriNet [hermans2017defense] ResNet-$50$ $78.1$ $81.5$ $77.0$ $81.1$ CAL
[gu2022clothes] ResNet-$50$ $81.3$ $82.6$ $79.6$ $81.7$ LightCyan 3DInvarReID
ResNet-$50$ $66.1$ $70.8$ $65.4$ $70.2$ rowred 2*CAL+ 2*- 2*82.6 2*83.9 2*81.3
2*84.3 rowred 3DInvarReID

Table : Comparison with SoTA methods on CCVID (%).

0.9!  Method Backbone mAP Rank$1$ Rank$5$ ReIDCaps [huang2019beyond]
DenseNet-$121$red$\times 6$ $10.9$ $6.5$ $20.2$ CAL [gu2022clothes]
ResNet-$50$ $19.3$ $10.0$ $26.7$ rowred 3DInvarReID ResNet-$50$ 21.7 11.1 30.5

Table : Comparison with SoTA methods on CCDA dataset (%).

Results on LTCC, PRCC and CCVID The comparison of the LTCC, PRCC and CCVID
datasets is shown in Tabs.  and . Similarly, by fusing the best baseline CAL
[gu2022clothes], our method achieves additional improvements. For instance,
when evaluated in the cloth-changing setting, our method achieves a
significant improvement of $2.6\%$ in Rank$1$ accuracy by fusing with CAL on
the CCVID dataset. These findings highlight the effectiveness of 3DInvarReID,
with the $3$D shape features being shown to be both discriminative and
complementary to $2$D features. Our approach stands out for its superior
performance compared to other $3$D feature extraction methods for person ReID.
This is particularly noteworthy in comparison to the 3DSL [chen2021learning]
(Tab. ). Additionally, we assess our model under a _cross-domain setting_
—that is, the model is trained with the Celeb-reID dataset and tested using
the LTCC and PRCC datasets. Table  shows that our models outperform the
baseline CAL, indicating a superior discriminative feature representation.

###  Results on LT-ReID dataset (CCDA)

Given that both our CCDA and Celeb-reID datasets are obtained from the
Internet and share a similar image style, we choose trained models on Celeb-
reID and evaluate them on CCDA. We choose the SoTA cloth-changing methods,
ReIDCaps- [huang2019beyond] and CAL [gu2022clothes] as baselines. The results
in Tab.  demonstrate that our 3DInvarReID outperforms the baselines, providing
strong evidence of its effectiveness in handling person ReID with challenging
body poses and cloth-changing variations.

###  Results on Short-term Person ReID datasets

Despite our method is designed for LT-ReID, we additionally compare with SoTA
methods on two conventional short-term ReID datasets: Market-1501
[zheng2015scalable] and MSMT17 [wei2018person], in Tab. . By fusing with CAL
[gu2022clothes], we observe an average improvement of $2.7\%$ in Rank$1$
accuracy on both datasets, demonstrating the complementary nature of our 3D
shape feature, even on short-term datasets.

0.95!  2*Method 2* Backbone Market-1501 MSMT17 mAP Rank$1$ mAP Rank$1$ PCB
[sun2018beyond] ResNet-$50$ $81.6$ $93.8$ $40.4$ $68.2$ $3$DSL
[chen2021learning] ResNet-$50$red$\times 2$ $87.3$ $95.0$ $-$ $-$ FSAM
[hong2021fine] ResNet-$50$ $85.6$ $94.6$ $-$ $-$ CAL [gu2022clothes]
ResNet-$50$ $87.5$ $94.7$ $57.3$ $79.7$ 3DInvarReID ResNet-$50$ $85.5$ $94.2$
$55.1$ $76.3$ rowred 2*CAL+ 2*- 2*87.9 2*95.1 2*59.1 2*80.8 rowred 3DInvarReID

Table : Comparison on short-term ReID datasets (%).

##  3D Reconstruction

Most $3$D body reconstruction methods focus more on pose estimation than shape
estimation. Recently, SHAPY [choutas2022accurate] releases a dataset (HBW)
that contains ground-truth 3D body scans and the corresponding in-the-wild
images, which enables us to test the accuracy of our reconstructed 3D naked
body shapes. We thus evaluate our methods on the validation set of HBW, which
contains $237$ in-the-wild images of $10$ subjects. Our baseline includes LVD
[corona2022learned] and SHAPY [choutas2022accurate], which are recent
pipelines for discriminative identity shape fitting. Following
[corona2022learned], we evaluate the reconstruction accuracy with Chamfer
distance (CD-$L_{2}$), by uniformly sampling $10,000$ points on both ground-
truth and predicted meshes in the canonical space. As shown in Tab. , our
method outperforms the two baselines. We also visualize $3$D reconstructions
in Fig. . Our reconstructions resemble the ground truth better than the
baselines. These results demonstrate the superiority of the proposed method in
reconstructing naked $3$D body shapes. Fig.  shows qualitative comparisons
with ICON [xiu2022icon] and ClothWild [moon20223d]. Our approach achieves
comparable clothed $3$D body reconstructions.

0.6!  LVD [corona2022learned] SHAPY [choutas2022accurate] Ours CD-$L_{2}$
$0.654$ $0.632$ 0.610

Table : Comparison of $3$D boy reconstruction on HBW. Input | LVD [corona2022learned] | SHAPY [choutas2022accurate] | Ours | GT
---|---|---|---|---
[height=10cm]fig/recon_comp1.png |  [height=10cm]fig/recon_comp2.png |  [height=10cm]fig/recon_comp3.png |  [height=10cm]fig/recon_comp4.png |  [height=10cm]fig/recon_comp5.png
Figure : Qualitative comparisons with LVD [corona2022learned] and SHAPY
[choutas2022accurate] on $3$D naked body reconstruction. Our approach recovers
more accurate $3$D body shapes from the images.

0.9!  Model Type mAP Rank$1$ Rank$5$ LightCyan 3DInvarReID 17.5 36.3 56.4
3DInvarReID-w/o $3$D $14.6$ $28.4$ $48.3$ 3DInvarReID-w/o Pre-training $15.1$
$29.3$ $50.7$ 3DInvarReID-w/o $3$D clothing $16.0$ $32.1$ $52.8$

Table : Ablaton studies on CCDA dataset (%).

##  Ablation Study

In this section, all models are trained on Celeb-reID and tested on the CCDA
dataset. Effect of the $3$D module We compare our full model with an ablated
version that only incorporates the $\mathcal{L}_{cla}$ loss in its training,
disregarding $3$D modules. The results in Tab.  show that our 3DInvarReID
significantly improves the recognition accuracy, leading to a Rank$1$ accuracy
increase from $28.4$ to $36.3$. Effect of Our Two-layer Implicit Model Our
primary goal is to disentangle the identity feature from non-identity features
in $3$D shape space. To evaluate the effectiveness of our $3$D disentanglement
module, we train a model (Ours-w/o $3$D clothing) by replacing our $3$D body
model with SMPL shape bases and omitting the modeling of the $3$D clothing
component and rendering layer. The results in Tab.  demonstrate the advantages
of modeling clothing shape and texture for person ReID (Rank$1$:
$32.1$$\xrightarrow{}$$36.3$). Effect of Pre-training Tab.  shows that
removing the pre-training stage results in a lower Rank$1$ accuracy, with a
score of $29.3$ compared to our $36.3$. This finding highlights the
effectiveness of pre-training in addressing the inherent ambiguity of
disentanglement.

1!  Input ICON [xiu2022icon] ClothWild [moon20223d] Ours
[height=8cm]fig/cloth_recon1.png [height=8cm]fig/cloth_recon2.png
[height=8cm]fig/cloth_recon3.png [height=8cm]fig/cloth_recon4.png

Figure : Qualitative comparisons with ICON [xiu2022icon] and ClothWild
[moon20223d] on $3$D clothed body reconstruction.