# Multilingual Text Representation

Fahim Faisal
Department of Computer Science
George Mason University
<EMAIL_ADDRESS>
###### Abstract

Modern NLP breakthrough includes large multilingual models capable of
performing tasks across more than 100 languages. State-of-the-art language
models came a long way, starting from the simple one-hot representation of
words capable of performing tasks like natural language understanding, common-
sense reasoning, or question-answering, thus capturing both the syntax and
semantics of texts. At the same time, language models are expanding beyond our
known language boundary, even competitively performing over very low-resource
dialects of endangered languages. However, there are still problems to solve
to ensure an equitable representation of texts through a unified modeling
space across language and speakers. In this survey, we shed light on this
iterative progression of multilingual text representation and discuss the
driving factors that ultimately led to the current state-of-the-art.
Subsequently, we discuss how the full potential of language democratization
could be obtained, reaching beyond the known limits and what is the scope of
improvement in that space.

###### Index Terms:

NLP, Multilinguality, Language Model

##  I Introduction

Natural language processing (NLP) primarily involves making linguistic-
specific applications for machines to understand language. Earlier days of NLP
development mainly focused on the idea of distributional hypothesis [1] that
is, "words occurring in the same context tend to have a similar meaning or
closely related meaning". From there, NLP has come a long way in the modeling
language. The tasks NLP tries to solve are complex and multidimensional if we
put them into the perspective of the machine and numerical mapping. For
example, there are multiple dynamics to deal with here, like various languages
and dialects to consider and tasks to solve with different levels of
granularity. Combining all these dynamics in a unified representation space is
a complex problem. The base starting point could be just words, as the word is
one such unit of language that is quite universal. Following this thought, the
most straightforward idea could be to compute the word frequency, thus
constructing a count-based numerical mapping that we can think of as a
computationally feasible language representation. N-gram language models are
representations that have led the domain of NLP for a substantial time. Later
on, advancing over the distributional scheme of language representation
coupled with the availability of huge computation resources, neural models
became the go-to approach for all kinds of NLP tasks. Researchers investigated
different theories and directions in this domain of neural language modeling
before transformer-based neural networks revolutionized NLP. The transformer-
based model provides exceptional text representation utilizing the multi-
layered encoder blocks [2]. This is useful as text gets different meanings
based on how it is used in a context. In addition, multiple languages can
share a single representation space using transformers. This effectively led
to the path for multilingual text representation, where data collected from
languages existing all over the world can be accumulated in a single setting,
and models can learn and perform actionable inference on a wide variety of
tasks comprising language and dialectal varieties. Though a monolingual or
region-focused transformer still vastly outperforms a more generalized
multilingual model on most tasks, it is not always feasible to train multiple
versions of the domain-focused model. The idea is to make a shared
representation space that effectively works for many languages, while the
resource scarcity of specific languages should benefit from other high-
resource languages. mBERT [3] and xlm-r [4] trained with multiple objective
functions on more than hundreds of languages came a long way in achieving this
vision. However, the full potential of a unified multilingual text
representation is still a significant research problem to solve. Because,
quite regularly, the inclusion of new tasks and languages in the modeling
paradigm points out the fact that, when these models move beyond the
monolingual scheme, the total capacity of the model gets distributed across
languages, thus often resulting in capacity dilution [5]. An ideal scenario
would be to have no amount of negative interference, such that we get an
equitable performance across languages. Another important direction for
multilingual models is to ensure the easy-expand-ability to new languages and
adaptability to new tasks.

Keeping all these advanced development of multilingual text representation in
context, in this survey, we provide insight into the open problems and
questions to look for. In addition, we discuss how the text representation
model starting from the count-based vector representation of words, came to
the point of a multilingual text representation model capable of performing
across more than 100 languages. We structure the contents based on the
following contexts: (1) How did the text representation model make the
iterative progress, and which were the driving factors in each step? (2) What
are the primary building blocks of a unified multilingual text representation
model, and how do they vary given the difference in scenarios? (3) What are
the current barriers that limit the progression of full-scale multilingual
text representation? (4) The fairness and interpretability of currently
available models and how equitable they serve the intended user’s utility.

##  II Preliminaries

###  II-A Terminologies

##### Tokenization

: A process of transforming the text into tokens of words is generally known
as tokenization [6]. Not necessarily; it needs to be in words, but it can be
divided into words, symbols, phrases, or sub-word tokens.

##### Word segmentation

Separating the phrase, content, and keywords. Other steps include stemming,
lemmatizing, handling negation, and separating punctuation.

##### Embedding

Embedding is a relatively low-dimensional space where we can place the
transformed high-dimensional vectors. Word embedding generally means a type of
embedding space where the close-meaning words would be grouped, maintaining a
close distance.

##### Attention

We can interpret this mechanism as the vector of important weights [7]. An
attention vector provides insight into how words in a sentence might be
correlated with other words in that sentence. In addition, it determines the
most useful blocks of the input in terms of transferring the contextual
insights to actionable information for the target output. The attention
mechanism mitigates the long-lasting disadvantage of fixed-length vectors in
sequence-to-sequence models: the incapability of remembering long sentences.

###  II-B Early days of Multilingual Word Representation

Multilingual Word representation involves representing words from different
languages through a shared space. The bilingual word embedding model [8] is
one of the earlier works that uses machine translation word alignment and
embeds words from the source and target language into a single vector space.
During training, it imposes constraints over the distance based on the
translation pairs of two languages. Later on, [9] proposed an approach that
can learn bilingual word embedding given the lexicon of bilingual word pairs.
It performs a linear transformation to transform the source language vectors
to the target language vector space. However, when provided a lexicon of small
size, this model performs poorly. To tackle this issue, [10] introduced a
matching score mechanism along with the original bilingual lexicon pairs to
bring the embeddings closer.

###  II-C Neural Language Model Basics

In neural language modeling (NLM), we train a probabilistic classifier to
predict a probability distribution where the conditional probability of
selecting word $w_{i}$ is learned using various kinds of neural networks
(e.g., feed-forward, recurrent, etc.):

$\displaystyle P(s)=\prod_{i=1}^{l}P(w_{i}|w_{1}^{i-1})$

##### Feed-forward NLM:

A feed-forward NLM (FFN) [11] adopts the notion of an n-gram language model by
assuming each word in a sequence depends on the words closer to it
statistically though it fails to consider the long-term dependency. Instead of
considering the dependence of the whole previous sequence, a context window
($i-n+1\,to\,i-1$) is used for better approximation:
$P(w_{i}|W_{1}^{i-1}=P(w_{i}|W_{i-n+1}^{i-1})$. The context word sequence
$x=[w_{i-n+1}...,w_{i-1}]$ is fed into a FFN, and later, a softmax layer over
the final output matrix is used to get the output probability of
$P(w_{i}|W_{i-n}^{i-1})$.

##### Convolutional NLM:

This one enhances the ffn by injecting a CNN layer on top of the input
representation [12], which involves a sliding window of the input vectors
centered on each word vector and later on, performing max-pooling on it.

##### Recurrent NLM

Recurrent Neural Network (RNN) based LM [13] addresses the issue of long-term
dependency problem. At every time step of RNN, the input is just the previous
word vector instead of the concatenated vectors of n previous words.
Meanwhile, the information of all previous words is preserved by the internal
state of RNN. The most common RNN types are LSTM [14] and GRU [15]. The key
problem with RNN-based language models is falling into the vanishing gradient
(taking the multiplication of a large number of derivatives eventually results
in a value close to zero, which can further not be used in error function
calculation). This failure leads to the problem of not capturing the
dependencies among words in a long sentence as the amount of computation using
RNN increases when the distance among words increase in a sentence.

##### Transformer Language Models

This is a non-recurrent encoder-decoder architecture with a series of
attention-based blocks. An encoder prepares a contextual representation of the
given input, whereas the decoder can generate output based on the output
segments that are already generated. The attention mechanism is at the core of
the Transformer architecture [2]. An encoder-decoder-based transformer
contains three types of attention units facilitated with the help of queries,
keys, and values.

Key $K$: This is a label of a word and is used to distinguish between
different words.

Query $Q$: It represents an active request that checks all the keys and
selects the one that matches the request.

Value $V$: A value is always paired with a key, and when the query selects a
key, its value is the one that propagates. A value is an information a word
contains.

When $k,Q,\&V$ all get generated from the same source sequence, it is self-
attention (exist one in the encoder, one in the decoder). When they come from
different sequences, they form a cross-attention mechanism (happens in between
encoder-decoder interactions). The other type is the scaled-dot product
attention mechanism that performs dot product calculation between $Q$ and $K$
matrices. Before that, the $k,Q,\&V$ are calculated by the matrix
multiplication with learned word vectors. The scaled-dot product attention
then ensures that we select more information from the values where the key and
the query are more similar. On top of these mechanisms, we have multi-head
attention where multiple scaled dot product attentions run in parallel, thus
helping the network to attend to multiple pieces of information
simultaneously. In addition, there is no recurrent element in transformers.
So, to learn the position information of each word, a combination of sine and
cosine waves of different frequencies is used, as each position would have a
unique combination of values. There are other essential segments like residual
connections and layer normalization. Residual or skip connection adds the
input to the output after a layer, allowing the gradients to flow directly
through the network. Whereas the layer normalization keeps the mean of each
training sample close to 0 and the standard deviation close to 1, thus
stabilizing the training as well as reducing the training time.

In practice, a layer from the encoder contains multi-head attention sub-layers
and a position-wise feed-forward sub-layer. The sub-layers are connected using
the residual connections. The multi-head attention sub-layer contains several
attention heads. A head is a scaled dot product attention structure taking the
query matrix $Q$, the key matrix $k$, and the value matrix $V$. So the output,

$\displaystyle Attention(Q,K,V)$
$\displaystyle=Softmax(\frac{QK_{T}}{\sqrt{d_{k}}})V$ $\displaystyle
Multilhead(Q,K,V)$ $\displaystyle=[head_{1},head_{2},..head_{h}]W^{0}$
$\displaystyle\text{where, }head_{i}$
$\displaystyle=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})$
$\displaystyle\text{and }W_{i}^{Q}$ $\displaystyle=\text{linear projections},$
$\displaystyle\text{and }d_{k}$ $\displaystyle=\text{is the dimension of the
query matrix.}$

The fully connected position-wise feed-forward sub-layers contain two linear
transformations with RELU:

$\displaystyle FFN(x)$ $\displaystyle=W_{2}max(0,W_{1}x+b_{1})+b_{2}$

Transformers are better for dealing with long-term dependency than RNNs.
Originally it was proposed to solve the problem of machine translation, but
later on, it became the backbone of all kinds of NLP applications.

###  II-D Multilingual Language Modeling

A general Multilingual Language Model (MLLM) architecture contains an input
layer, transformer layer, and output layer.

  * •

input layer: sequence of tokens from a representation of one-hot subword token
vocabulary (concatenated from all languages).

  * •

transformer layer: each layer contains k attention heads, followed by a feed-
forward neural network.

  * •

output layer: contextual representation for each token. It contains a simple
linear transformation followed by a Softmax that takes the last layer token
representation and produces the probability distribution.

##  III Area Taxonomy

for tree= grow=east, growth parent anchor=west, parent anchor=east, child
anchor=west, edge path=[edge,->, >=latex] (!u.parent anchor) – +(10pt,0pt) |-
(.child anchor) edge label; [Taxonomy, basic, l sep=10mm, [Evaluation, xnode,
l sep=10mm, [xtreme [16], cnode] ] [Tokenization, xnode, l sep=10mm, [Subword
Segmentation, tnode , l sep=10mm, [wordpiece [17]
bpe [18]
unigram [19]
sentencepiece [20], cnode] ] [Character-
Based, tnode, l sep=10mm, [ [21, 22, 23], cnode] ] ] [Tasks, xnode, l
sep=10mm, [Semantic, tnode , l sep=10mm, [multi-nli [24] xnli [25], cnode] ]
[Syntactic, tnode, l sep=10mm, [pos [26] dep [26] ner [27], cnode] ] ] [Data
and Language, xnode, l sep=10mm, [Pretraining Data, tnode, l sep=10mm,
[Parallel, onode, l sep=10mm, [[4], cnode] ] [Monolingual, onode , l sep=10mm,
[ [3, 28], cnode] ] ] [Language, tnode] ] [LM Objectives, xnode, l sep=10mm,
[Monolingual, tnode , l sep=10mm, [mlm [3]
clm [28]
nsp [3], cnode] ] [Parallel, tnode, l sep=10mm, [tlm [29], cnode] ] ]
[Training, xnode, l sep=10mm, [Fine-tuning, tnode, l sep=10mm, [Multi-Task,
onode, l sep=10mm, [ [30, 28], cnode] ] [Parameter-Effieient, onode, l
sep=10mm, [adapters [31], cnode] ] [Gradient-Based, onode, l sep=10mm, [bert
[3], cnode] ] ] [Pre-training, tnode, l sep=10mm, [Encoder-Decoder, onode, l
sep=10mm, [T5 [30], cnode] ] [Decoder Only, onode, l sep=10mm, [gpt [28] bloom
[32] mgpt [33], cnode] ] [Encoder Only, onode, l sep=10mm, [bert [3]
xlm-r [4], cnode] ] ] ] [Representation Type, xnode, l sep=10mm, [Contextual,
tnode, l sep=10mm, [elmo [34] mbert [3] gpt [28], cnode] ] [Continuous, tnode,
l sep=10mm, [word2vec [35] fasttext [36] glove [37], cnode] ] [Classical,
tnode, l sep=10mm, [one-hot
tf-idf [38], cnode] ] ] ]

Figure 1: Area Taxonomy of Multilingual Text Representation

The multilingual text representation area taxonomy is presented in Figure 1.
Here we divide the taxonomy into certain primary parts, thus representing the
building blocks of text representation from different perspectives. For
example, Representation Type defines ways of representing text starting from
the classical models to recent Transformer breakthroughs. Furthermore, we can
divide a standard NLP training paradigm into parts like pre-training and fine-
tuning. We believe tokenization, being a step from text preprocessing,
deserves a separate branch of discussion because of its impact on multilingual
settings. In the next section, we report a detailed discussion of the existing
research in every direction of this taxonomy.

##  IV Taxonomy-Based Survey

###  IV-A Representation Type

The simplest form of text representation would be the word-count-based one
that fully relies on statistical information. From there, text representation
has advanced to a label, where we can represent texts from different languages
in a shared space. The progression is described in detail here.

##### Classical Models

Word frequency is the basis of the classical text representation model used in
earlier days. We can divide these models into two parts: (1) Categorical and
(2) Weighted. One-hot-encoding and Bag-of-Words (BoW) are the categorical
models. In one-hot encoding, the dimension to represent texts is equal to the
terms present in the vocabulary, where binary values are used to define the
presence or absence of a particular term. Whereas, BoW is just an updated one-
hot-encoding where we sum all the one-hot-representation of words in a
sentence. However, these categorical models fail to capture semantic relations
and the order of the words. A weighted text representation model known as Tf-
Idf was introduced to solve this problem. A Tf (i.e., term-frequency) matrix
just divides the word count by the length of a document, thus identifying how
often a word occurs in a document. Whereas, Idf (i.e., inverse document
frequency) matrix tries to reduce the effect of common words by putting more
weights on the critical words (words that are not equally frequent in all
documents, like stop words).

##### Continuous Representation

One popular approach to represent text is to present it as vectors where each
dimension corresponds to the frequency of words, thus resulting in a word
vector. The pitfall is that the vector length might be substantial depending
on the vocabulary size. In that case, adopting a dimensionality reduction
procedure becomes a default choice. Though these reduced vectors might be
compact and efficient to compute, they contain less of the original
information. Moreover, the individual dimensions no longer preserve the
interpretable features that could be mapped back to the original textual
building blocks. In one way, the context gets distributed throughout the
vector length, thus making it a distributed representation of continuous
values. In this distributed/continuous representation, each dimension of a
word type vector becomes a parameter to be learned and optimized based on the
observable patterns in the data. We can see these parameters as continuous
values that can be learned using a continuous objective function using
iterative algorithms like gradient descent. Word2vec [35] is one such
distributed vector representation of text. Word2vec considers similar meaning
words like "small" and "smaller" comparatively closer in the vector space.
There are two types of word2vec algorithms in practice: (1) Continuous bag of
words (CBOW) (2) Skip-gram. In CBOW, context is considered the input. The
neural network tries to correlate the weight matrix with each word, thus
improving the representation of words through backpropagation of the error
gradient. In skip-gram, the context is estimated based on the given the word.
However, both CBOW and Skip-gram were very time-consuming in practice. To
solve this issue, Hierarchical softmax and negative sampling approaches were
introduced. Negative sampling restricts the output sum so that only a subset
of the vectors get updated in each step, whereas, in hierarchical softmax,
words are chosen based on their count-based conditional probabilities. Later
on, GLOVE [37] improves significantly on word2vec by using the global
contextual information by constructing the global co-occurrence matrix and
factorizing it later on. However, these distributed representation methods
failed to consider the out-of-vocabulary (OOV) words. This is when Fasttext
[36] was proposed. Fasttext breaks the words to n-gram instead of using the
full-word representation at once, thus solving the OOV issue. These models can
extract syntactic and semantic information while dealing with specific corner
cases. However, there is this existing issue of keeping the full context-
specific representation of a document because understanding the actual context
is required for most downstream tasks in NLP.

##### Contextual Word Vectors

The weakness of the continuous word representation model was to failing to
capture the global context information on a low-dimensional scale. To solve
this issue, contextual word representation models like Context2vec [39], Cove
[40] and ELMO [41] were proposed later on. These advanced models solved many
of the existing issues but still face catastrophic forgetting. Context2vec
model is based on Word2vec’s CBOW model but replaces its average word
representation within a fixed window with a better and more powerful Bi-
directional LSTM neural network. Whereas Cove uses machine translation instead
of the approach used in Word2vec (skip-gram or CBOW) or GloVe (Matrix
factorization). They pre-train a two-layer BiLSTM for attending sequence-to-
sequence translation, starting from GloVe word vectors. Then they took the
output of the sequence encoder and called it a CoVe, followed by combining it
with GloVe vectors, and used it in a downstream task-specific mode using
transfer learning. On top of these models, ELMO (embeddings from language
model) [34] was the one to successfully advance the contextualization of word
vectors. The key idea was: while a word token will have its vector, this
vector needs to depend on the nearby word contexts. This is similar to the
distributional hypothesis [1]. However, unlike word-type vectors, these word
token vectors combine the word-level vectors with neural network parameters
going beyond the lookup table of word-type vectors. ELMO contains two neural
networks: One for the left context (start of the sentence to the token) and
another for the right context (from the token to the end of the sentence). It
uses a recurrent neural network that was the most advanced neural network at
that time and was a novel thing to introduce in language modelling. Later on,
ULMFiT [42] successfully adopt the concept of transfer-learning in the
contextual text representation. It segments the end-to-end process into three
steps then: (i) General LM pre-training, (ii) Target task LM fine-tuning, and
(iii) Target task classifier fine-tuning. Then finally, the transformer-based
fine-tuning appeared in the picture which proved to be more efficient and
faster than LSTM or CNN for language modelling.

##### Transformer Based Language Models

Transformer-based language models are more efficient than the ones with LSTM
or CNN for language modeling. The first model to efficiently use the
transformer architecture was GPT [28] which is a decoder-based model. GPT was
trained with Causal Language Modeling. The next one to efficiently represent
the semantics and context was BERT [3]: Bidirectional Encoder Representations
from Transformers. BERT uses a parallel attention layer instead of using
sequential recurrent layers. It is an encoder model that uses masked language
modelling (MLM) and next-sentence prediction (nsp) objectives in a self-
supervised manner for being trained on a large pool of textual data. Here, the
next sentence prediction is used to collect long-term or pragmatic
information.

###  IV-B Training

We can largely divide the current practice of transformer-based model training
into pre-training and fine-tuning. Pre-training is the training part of
preparing a base language model that can further be transferred and adapted to
any downstream task through fine-tuning.

####  IV-B1 Pre-training

Pre-training is a common step in making large language models in the current
NLP paradigm. Pre-training means training a language model on extensive
textual data in a self-supervised manner. Self-supervised because the pre-
training objective looks for the data labels to predict, which are
automatically contracted from the data itself (e.g., masked language modeling,
next sentence prediction). Here we discuss some common types of pretrained
transformer-based models and their specific approaches to performing the
training.

##### Decoder only

GPT [28] is the most common example of the decoder-only model. It uses the
causal language modeling objective. Other advanced decoder only models are OPT
[43], BLOOM [32], mGPT [33]. Recently, these decoder-only models are also
going multilingual (e.g., mGPT, BLOOM) and increasing the span of their
parameters.

##### Encoder Only

The most common example of encoder only models are bert [3]. It uses masked
language modeling (MLM) and next sentence prediction (NSP) objectives to train
on large monolingual texts. Several interesting works investigate improving
the self-supervised pre-training objective of encoder-only models. For
example, in [44], the authors find out that using dynamic masking during
training(randomly masking 15% of token each step) time instead of static
masking improves the performance. In addition, they show that NSP can be
dropped completely from the training objective to get better performance.

##### Encoder-Decoder

T5 [30] is a encoder-decoder model. The objective of T5 is closely related to
the MLM and word dropout techniques. The difference with the original MLM is
the consecutive span of corrupted or dropped tokens is replaced with one
single sentinel token. The output sequence contains these dropped-out spans
delimited by the sentinel tokens that replace the original text. Using this
technique of denoising sequence-to-sequence objective, the decoder can predict
the span of tokens in the masked position instead of just a single token.

####  IV-B2 Fine-tuning

Fine-tuning means adopting a pre-trained model for any downstream task.
Previously, fine-tuning was only used to indicate the gradient-based training
of the complete model. Here, we will consider any updates that make a base
model further usable for any downstream task as fine-tuning.

##### Gradient-based

The gradient-based finetuning means producing a whole set of new parameters by
training the entire model and updating all parts for a downstream task. In the
early days of transformer and monolingual language modeling, this was the go-
to approach to try any model on a new task or language [3].

##### Parameter efficient

The gradient-based finetuning is a massive bottleneck if we consider the
number of downstream tasks. Then there is parameter-efficient finetuning named
as adapters which include updating only parts of model parameters [31].
Adapters proved to help perform effective cross-lingual transfer while
reducing the amount of negative interference [45].

##### Multitask and zero-shot learning

Multitask learning includes training the model on a wide variety of tasks
simultaneously instead of just one task [30]. Following this scheme, there
would be less necessity for further finetuning. However, till now, the
utilization of this scheme is still confined to only computationally intensive
models and not in a general-purpose setting.

###  IV-C Language Modeling Objectives

We can think of the learning objectives of language modeling in a data-driven
manner. One is monolingual objectives which generally work on monolingual data
trying to figure out the representation of a missing token. Whereas in
bilingual objectives, data comes from a parallel corpus. The aim is to
represent similar-meaning words as close as possible.

##### Monolingual Objectives

Masked language modeling(MLM) is monolingual but surprisingly helps learn
multilingual models. Consider, a sentence $X=(x_{1},x_{2},..x_{s})$. In masked
language modeling, token $x_{i}$ is replaced with $\bar{x_{i}}$. So the input
to the model becomes $X=(x_{1},...\bar{x_{i}},..x_{s})$. Now the prediction
task ins $Y=(x_{i})$ where $x_{i}$ can be predicted from the final
representation of $\bar{x_{i}}$ [3]. Previously it was thought that MLM works
well because of its ability to discover syntactic and semantic mechanisms in
the pre-training stage. However, recent findings suggest MLM learns the higher
order distributional statistics, thus making it a very useful prior for
further fine-tuning [46]. Another common one is Causal Language Modeling
(CLM). CLM is the traditional learning objective in a language model that
includes estimating the probability of a word given the previous sequence of
words in a sentence, thus $P(x_{t}|x_{1},x_{2}..,x_{t-1},\theta)$. In other
words, a model can be trained with inputs $X=(x_{1},,,x_{t-1})$ and outputs
$Y=(x_{t})$. Here $x_{t}$ is the output label which can be predicted from the
final layer representation of token $x_{t-1}$.

##### Parallel-corpora based Objectives

This is mainly a bilingual objective, where the model learns to reduce the
distance of similar meaning text given a parallel corpus. As the amount of
parallel corpus is much less than monolingual one, a joint objective utilizing
both monolingual and parallel corpus is used. Translation language modeling
(TLM) [47] is the most common one, where $x_{1}^{A},x_{2}^{A},...x_{n}^{A}$ is
a sentence in language $A$ and $x_{1}^{B},x_{2}^{B},...x_{m}^{B}$ is a
sentence in language $B$. Now the MLM masks tokens from both sentences $A$ and
$B$. It tries to predict the missing token by utilizing either the surrounding
tokens in $A$ or the translation in $B$. Another useful objective is to use
the contextual representation from the base language model given word
alignment [48, 49, 50] between translations and ask the model explicitly to
reduce the distance of similar meaning word representation [51].

###  IV-D Data and Language

##### Pre-training Data

Multilingual Language Models explore two data types primarily for pre-
training: (1) Large Monolingual Data and (2) Parallel Corpus. Bert [3] uses
monolingual data from Wikipedia, whereas XLM-R [4] uses a much larger common-
crawl corpus to train the model. Language family or region-specific models
[52] have their focused source of data to explore.

##### Languages

BERT [3] and XLM-R [4] are two multilingual models having trained in more than
100 languages. One problem is that the data availability is not evenly
distributed across languages. Usually, models use exponential smoothing to
make the data ratio fair across high-resource and low-resource languages. The
main idea is if a language $i$ contains $m\%$ of total pre-training data, then
the probability of that language is $p_{i}=k/100$ where $p_{i}$ is
exponentiated by a factor $\alpha$. Then the resulting values are normalized
to give a probability value to all the languages. $\alpha<1$ means the high-
resourced ones will be under-sampled, whereas the low-resourced ones will be
over-sampled.

###  IV-E Tasks

The range of mainstream NLP tasks requires models to perform transfer learning
at different difficulty levels, thus requiring word, phrase, or sentence-level
understanding. Essentially, we can frame the tasks into two main groups:

##### Syntactic Task

These tasks focus on the sentence-level or word-level structure of the
languages. The most common example of this type of task includes dependency
parsing (DEP) [53], named entity recognition (NER) [27] and parts of speech
tagging (POS) [26]. Universal dependency project [26] contains a dataset of
DEP and POS tasks covering more than 100 languages as part of the evaluation.

##### Semantic Task

Some tasks require models to perform language-level understanding or inference
that can not be answered by just following the sentence structure. Natural
language inference [24, 25] that tries to predict whether a premise sentence
entails, contradicts, or is neutral toward a hypothesis sentence is one such
task.

###  IV-F Tokenization

Tokenization is splitting a sequence of characters given a document unit into
a piece of singular units (e.g., tokens) based on some level of heuristics.
The simplest form of heuristics would be to cut into words based on white
space, thus preserving the meaning at the unit level. However, this comes with
issues like phrase-level segmentation and different characterization of the
similar-meaning word because of minor spelling variations. Moreover, words
form differently in different languages. For example, in french, the use of
apostrophe sometimes works like the mention of a definite article
(l’ensemble). In German, compound words are written without white-space
(computational modeling $\rightarrow$ Computermodellierung). Thus, researchers
focus on sub-word and character-based splitting instead of white-space
tokenization to perform a more usable segmentation. Tokenization does the
text-to-numerical mapping of input, making it one of the primary steps before
feeding a text distribution to a computation model. As languages can be of
different scripts with different vocabulary, multilingual models accumulate
all happening subwords as part of the vocabulary for the supporting languages.
This becomes a limiting factor when we want to extend the model capacity to a
thousand more languages. Training monolingual tokenizers each time is not a
viable option, and the original multilingual tokenizer does not have
vocabulary distributional knowledge about the unseen languages.

##### Subword Segmentation

Subword-based tokenizations are the most widely used tokenization for
multilingual transformer-based models. Because simple whitespace-based
tokenization suffers from the dimensionality bottleneck problem, simple
character-based splitting results in losing all context signals. The main
difference in common sub-word-based tokenizers lies in the choice of character
pairs to merge. For example, BPE [18] makes a frequency-based merging, whereas
the Unigram [19] model uses a probability based merging (computing the
likelihood of each subword instead of using the most frequent ones). Word-
Piece [17] tries to utilize the advantage of both unigram and BPE. It merges
based on likelihood instead of frequency, but the choice of words to join is
based on frequency. SentencePiece [20] is another subword-based tokenizer that
rules out the initial whitespace-based splitting (useful for languages like
Chinese and Japanese) by considering space as just another character and then
employing BPE or Ingram on top of that. In the multilingual scenario,
languages can be of different scripts with different vocabulary, and
multilingual models accumulate all happening subwords as part of the
vocabulary from the supporting languages. Now subword-based tokenizer with a
fixed vocabulary size performs unfairly for low-resource languages due to the
data imbalance among languages resulting in excessive fragmentation of
subwords [54]. In addition, the fixed vocabulary becomes a limiting factor
when we want to extend the model capacity to a thousand more languages.

##### Character-based Model

Subword tokenizers eliminate the out-of-vocabulary problem to a large extent,
though the reliance on static vocabulary prevents end-to-end learning across
all languages. One alternative would be to use a character-based approach [21,
22]. Though this is more adaptable to the code-switched language and noisy
text, they may not capture the token-level representation and convert the
longer sequence to character-level representation [22], thus increasing the
task complexity [23].

###  IV-G Evaluation

Language models like mBERT [3] and XLM-R [4] helped in a way to shift the
overall focus towards a unified general multilingual framework of language
modeling. With this advancement, researchers identified the need for a unified
evaluation framework that can provide insight into the overall transfer-
learning capability of a multilingual model. With this aim, in [16], the
authors designed an evaluation setting that ranges from syntactic to semantic
tasks and structural prediction tasks. In each task evaluation, the training
is performed in English, whereas the evaluation comprises 40 languages from 12
families, thus ensuring the typological diversity in language selection.
Later, this practice of expanding over far-reach languages continued through
subsequent studies [55]. However, these unified evaluation frameworks still do
not entirely comprise languages from highly low-resource languages and
dialects.

##  V Open Problems

##### Tokenization free approach

In short, a silver-lined multilingual tokenizer is yet to be found as a
monolingual subword tokenizer still outperforms a multilingual tokenizer in
almost every task [56, 57]. Now monolingual Training tokenizer every time is
not a feasible option and the original multilingual tokenizer does not have
vocabulary distributional knowledge about the unseen languages. One option
could be the effort to rule out the tokenization step completely. Several
recent studies are exploring this direction. Models like CAINNE [23] are
tokenization-free modules that encode texts differently. Instead of using a
vocabulary and tokenization step, CANINE encodes texts at the character level,
produces character-level output, and uses a soft-max layer-based smoothing for
subword projection. Recently another model PIXEL [58], treats text as an image
and by doing that, bypasses the text encoding step. These methods are getting
inspiring results but the full potentiality of this direction is yet to
unfold. For example, pixel receives impressive results for syntactic tasks,
but this is not true for semantic tasks. Further studies need to be done on
the applicability of tokenization-free models in the case of zero-shot and
few-shot transfers.

##### Extending to new languages

In the current paradigm of MLLM, the curse-of-multilinguality [45] is an issue
where the per-language performance drops unequally when the model is trained
on multiple languages. In a general setting, the usual scenario is: (1) A
monolingual model performs better than a multilingual model on specific tasks
and language (2) The low-resourced ones perform poorly compared to the high-
resourced ones in MLLM (3) Languages unseen during the pre-training stage
perform the worst. To tackle this issue, the straightforward approach would be
to finetune the model to the specific target language. However, then, the
model becomes specific to only one language though it does not increase the
catastrophic forgetting in general. The authors in [59] have identified then,
while dealing with unseen language and scripts, having pretrained on any
closely related language usually helps. Another approach to improve the
capacity of MLLM would be to augment the vocabulary with new tokens, which
works better for unseen languages and improve performance for the languages
already seen during pre-training [60]. Another simple approach includes
mitigating the cross-lingual transfer gap by just performing training on a bit
of target data amount[61]. Another helpful approach to improve the model
capacity is to use adapters, modular parameter units that can be injected in
every layer of a base language model. These adapter layers can then be trained
on language or task-specific data, thus enabling the cross-lingual transfer
without changing the base model parameters[31]. This approach can be further
used for unseen language and optimized zero-shot transfer by using these
adapters at the pre-training stage, which lifts the curse-of-multilingualism
to a greater extent[62].

##### Model Interpretability and Shared Representation

It is hypothesized that most of the large neural models are over-parameterized
[63], thus, resulting in unnecessary computation and storage costs. Effective
model pruning could be a feasible approach to try in this regard. [64] is one
of the earlier studies on pretrained bert with model pruning. The results
demonstrate that the idea of lottery ticket observations [63] (i.e., if we
select multiple small networks, we will end up getting the similar performance
of the larger model at one time) remain relevant in this context of language
modeling also. We find matching subnetworks for a range of downstream tasks at
40% to 90% sparsity. Subsequently, [65] has done LTH based empirical study on
English bert. The findings suggest “random” subnetworks are still almost as
good as the “good” ones, and even the “worst”(sampled elements which didn’t
survive the pruning by LTH [63]) ones perform on par with a substantial
baseline. [66] is one of the studies which explore LTH-based pruning in
multilingual scenarios. The authors use the lottery ticket hypothesis in mBERT
and conclude that the sub-networks found for different languages are similar.
In addition, mBERT comprises a language-neutral sub-network shared among many
languages, which is the most useful one while performing cross-lingual
transfer. Another finding is for MLM tasks, the similar language & task
specific sub-network are primarily identical in the lower to the middle layer.
In contrast, network similarity is visible mainly in the higher layer for NER
& XNLI tasks. The prospect of different pruning-based methods is further
explored in [67]. The authors study two types of model pruning strategies:
regularization-based and gradient-based pruning. They also propose a method
called Dynamic Sparsification to allow training the model once and then
adapting to different model sizes at inference. They also used a diversity
loss to prune language-specific subnetworks. Results show that subnetworks of
different languages are indeed different. The most straightforward pruning
algorithm performs the best, and A fast model does not mean it should be
small. Another finding is for large multilingual models like XLM-R sharing the
subnetwork for a universal representation is preferable; the language-neutral
part contributes the most in cross-lingual transfer, which also aligns with
the finding of [66]. [68] did one study on the Effect of Dropping Layers of
Pre-trained Transformer Models where they pruned BERT, Roberta, and XLNet
models up to 40% while maintaining up to 98% of their actual performance. The
findings are: (i) the lower layers are most critical to maintaining downstream
task performance, (ii) some tasks, such as paraphrase detection and sentence
similarity, are more robust to the dropping of layers, and (iii) models
trained using a different objective function exhibit different learning
patterns and w.r.t the layer dropping.

Another question is whether MLLM learns universal patterns or not. Learning
universal patterns is essential for effective cross-lingual transfer. From the
discussion above, it can be inferred that MLLM contains particular language-
specific and shared representation space. The shared representation space
helps in probing-based tasks like POS tagging. However, performing complex
tasks like MT is still beyond scope just by using this shared representation
space.

Analysis of traditional classifier-based probing methods is another heavily
investigated direction that is still being explored to ensure model
interpretability. For example, [69] examines whether good networks include any
superior linguistic knowledge or not. It results in not finding any
interpretable patterns. [70] investigated Language Relationships in
Multilingual Sentence Encoders Through the Lens of Linguistic Typology; This
study looks into how languages are placed in multilingual subspace in mBERT
and XLM-R. At the same time, how language specific sub-spaces within
multilingual sentence encoders (LASER [71], m-BERT [3], XLM [72], and XLM-R
[4]) concerning a range of typological properties pertaining to the lexical,
morphological, and syntactic structure can be separated. Their results show
interesting differences in encoding linguistic variation associated with
different pre-training strategies. [69] is another recent study on linguistic
probing using universal dependency tasks and data points.

##### Fairness in Language Models

A substantial amount of work has investigated existing social bias (e.g.,
gender, racial, ethnic, occupational) identification and mitigation approach
in PLMs, including reducing token sensitivity during text generation [73],
investigating model sensitivity [74], prompting using natural sentences [75]
and probing via embedding lookup [76]. However, the state-of-the-art NLP
models and datasets are still biased toward certain attributes [77], and the
overall utility provided by the models is still skewed [78].

##  VI Conclusion

This survey provides insights into the current state of the art of
multilingual text representation and the extent of language modeling. While
many languages are already being covered under the current paradigm of
language modeling, the full potential is not explored yet. There are languages
left to cover. At the same time, the currently covered low-resourced ones are
still not free from the impact of capacity dilution. Keeping this in focus, we
did a review on the potential scope of improvement and research directions.

## References

  * [1] Z. Harris, “Distributional structure,” _Word_ , vol. 10, no. 2-3, pp. 146–162, 1954. [Online]. Available: https://link.springer.com/chapter/10.1007/978-94-009-8467-7_1
  * [2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017. [Online]. Available: https://arxiv.org/abs/1706.03762
  * [3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://www.aclweb.org/anthology/N19-1423
  * [4] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 8440–8451. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.747
  * [5] Z. Wang, Z. C. Lipton, and Y. Tsvetkov, “On negative interference in multilingual models: Findings and a meta-learning treatment,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 4438–4450. [Online]. Available: https://aclanthology.org/2020.emnlp-main.359
  * [6] J. A. Balazs and J. D. Velásquez, “Opinion mining and information fusion: A survey,” _Information Fusion_ , vol. 27, pp. 95–110, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1566253515000536
  * [7] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” 2014. [Online]. Available: https://arxiv.org/abs/1409.0473
  * [8] W. Y. Zou, R. Socher, D. Cer, and C. D. Manning, “Bilingual word embeddings for phrase-based machine translation,” in _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Seattle, Washington, USA: Association for Computational Linguistics, Oct. 2013, pp. 1393–1398. [Online]. Available: https://aclanthology.org/D13-1141
  * [9] T. Mikolov, Q. V. Le, and I. Sutskever, “Exploiting similarities among languages for machine translation,” 2013. [Online]. Available: https://arxiv.org/abs/1309.4168
  * [10] M. Zhang, H. Peng, Y. Liu, H. Luan, and M. Sun, “Bilingual lexicon induction from non-parallel data with minimal supervision,” in _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , ser. AAAI’17. AAAI Press, 2017, p. 3379–3385.
  * [11] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin, “A neural probabilistic language model,” _J. Mach. Learn. Res._ , vol. 3, no. null, p. 1137–1155, mar 2003.
  * [12] N.-Q. Pham, G. Kruszewski, and G. Boleda, “Convolutional neural network language models,” in _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 1153–1162. [Online]. Available: https://aclanthology.org/D16-1123
  * [13] T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur, “Recurrent neural network based language model.” in _INTERSPEECH_ , T. Kobayashi, K. Hirose, and S. Nakamura, Eds. ISCA, 2010, pp. 1045–1048. [Online]. Available: http://dblp.uni-trier.de/db/conf/interspeech/interspeech2010.html#MikolovKBCK10
  * [14] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Comput._ , vol. 9, no. 8, p. 1735–1780, nov 1997. [Online]. Available: https://doi.org/10.1162/neco.1997.9.8.1735
  * [15] K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder–decoder approaches,” in _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_. Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 103–111. [Online]. Available: https://aclanthology.org/W14-4012
  * [16] J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat, and M. Johnson, “Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization,” _CoRR_ , vol. abs/2003.11080, 2020.
  * [17] M. Schuster and K. Nakajima, “Japanese and korean voice search,” in _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2012, pp. 5149–5152.
  * [18] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 1715–1725. [Online]. Available: https://aclanthology.org/P16-1162
  * [19] T. Kudo, “Subword regularization: Improving neural network translation models with multiple subword candidates,” 2018. [Online]. Available: https://arxiv.org/abs/1804.10959
  * [20] T. Kudo and J. Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 66–71. [Online]. Available: https://www.aclweb.org/anthology/D18-2012
  * [21] P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck, “Learning deep structured semantic models for web search using clickthrough data.” ACM International Conference on Information and Knowledge Management (CIKM), October 2013. [Online]. Available: https://www.microsoft.com/en-us/research/publication/learning-deep-structured-semantic-models-for-web-search-using-clickthrough-data/
  * [22] K. Hwang and W. Sung, “Character-level language modeling with hierarchical recurrent neural networks,” _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 5720–5724, 2017.
  * [23] J. H. Clark, D. Garrette, I. Turc, and J. Wieting, “Canine: Pre-training an efficient tokenization-free encoder for language representation,” _Transactions of the Association for Computational Linguistics_ , vol. 10, pp. 73–91, 2022. [Online]. Available: https://aclanthology.org/2022.tacl-1.5
  * [24] A. Williams, N. Nangia, and S. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistics, 2018, pp. 1112–1122. [Online]. Available: http://aclweb.org/anthology/N18-1101
  * [25] A. Conneau, R. Rinott, G. Lample, A. Williams, S. Bowman, H. Schwenk, and V. Stoyanov, “XNLI: Evaluating cross-lingual sentence representations,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 2475–2485. [Online]. Available: https://aclanthology.org/D18-1269
  * [26] D. Zeman, J. Nivre, M. Abrams, E. Ackermann, N. Aepli, H. Aghaei, Ž. Agić, A. Ahmadi, L. Ahrenberg, Ajede, and et al., “Universal dependencies 2.9,” 2021, LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. [Online]. Available: http://hdl.handle.net/11234/1-4611
  * [27] A. Rahimi, Y. Li, and T. Cohn, “Massively multilingual transfer for NER,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 151–164. [Online]. Available: https://www.aclweb.org/anthology/P19-1015
  * [28] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” 2020, arXiv:2005.14165.
  * [29] D. Zeman and J. Hajič, Eds., _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_. Brussels, Belgium: Association for Computational Linguistics, Oct. 2018. [Online]. Available: https://www.aclweb.org/anthology/K18-2000
  * [30] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _J. Mach. Learn. Res._ , vol. 21, no. 1, jun 2022.
  * [31] J. Pfeiffer, I. Vulić, I. Gurevych, and S. Ruder, “MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 7654–7673. [Online]. Available: https://aclanthology.org/2020.emnlp-main.617
  * [32] “Bigscience large open-science open-access multilingual language model,” 2022, online resource. [Online]. Available: https://huggingface.co/bigscience/bloom-560m
  * [33] O. Shliazhko, A. Fenogenova, M. Tikhonova, V. Mikhailov, A. Kozlova, and T. Shavrina, “mgpt: Few-shot learners go multilingual,” 2022. [Online]. Available: https://arxiv.org/abs/2204.07580
  * [34] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 2227–2237. [Online]. Available: https://aclanthology.org/N18-1202
  * [35] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” 2013. [Online]. Available: https://arxiv.org/abs/1301.3781
  * [36] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” _Transactions of the Association for Computational Linguistics_ , vol. 5, pp. 135–146, 2017. [Online]. Available: https://aclanthology.org/Q17-1010
  * [37] J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1532–1543. [Online]. Available: https://aclanthology.org/D14-1162
  * [38] C. Sammut and G. I. Webb, Eds., _TF–IDF_. Boston, MA: Springer US, 2010, pp. 986–987. [Online]. Available: https://doi.org/10.1007/978-0-387-30164-8_832
  * [39] O. Melamud, J. Goldberger, and I. Dagan, “context2vec: Learning generic context embedding with bidirectional LSTM,” in _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 51–61. [Online]. Available: https://aclanthology.org/K16-1006
  * [40] B. McCann, J. Bradbury, C. Xiong, and R. Socher, “Learned in translation: Contextualized word vectors,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6297–6308.
  * [41] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in _Proc. of NAACL_ , 2018.
  * [42] J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, Jul. 2018, pp. 328–339. [Online]. Available: https://aclanthology.org/P18-1031
  * [43] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,” 2022.
  * [44] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692
  * [45] J. Pfeiffer, N. Goyal, X. V. Lin, X. Li, J. Cross, S. Riedel, and M. Artetxe, “Lifting the curse of multilinguality by pre-training modular transformers,” 2022, arXiv:2205.06266.
  * [46] K. Sinha, R. Jia, D. Hupkes, J. Pineau, A. Williams, and D. Kiela, “Masked language modeling and the distributional hypothesis: Order word matters pre-training for little,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 2888–2913. [Online]. Available: https://aclanthology.org/2021.emnlp-main.230
  * [47] A. CONNEAU and G. Lample, “Cross-lingual language model pretraining,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf
  * [48] Z.-Y. Dou and G. Neubig, “Word alignment by fine-tuning embeddings on parallel corpora,” in _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. Online: Association for Computational Linguistics, Apr. 2021, pp. 2112–2128. [Online]. Available: https://aclanthology.org/2021.eacl-main.181
  * [49] D. Yarowsky, G. Ngai, and R. Wicentowski, “Inducing multilingual text analysis tools via robust projection across aligned corpora,” in _Proceedings of the First International Conference on Human Language Technology Research_ , ser. HLT ’01. USA: Association for Computational Linguistics, 2001, p. 1–8. [Online]. Available: https://doi.org/10.3115/1072133.1072187
  * [50] C. Dyer, V. Chahuneau, and N. A. Smith, “A simple, fast, and effective reparameterization of IBM model 2,” in _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Atlanta, Georgia: Association for Computational Linguistics, Jun. 2013, pp. 644–648. [Online]. Available: https://aclanthology.org/N13-1073
  * [51] S. Cao, N. Kitaev, and D. Klein, “Multilingual alignment of contextual word representations,” 2020. [Online]. Available: https://arxiv.org/abs/2002.03518
  * [52] D. Kakwani, A. Kunchukuttan, S. Golla, G. N.C., A. Bhattacharyya, M. M. Khapra, and P. Kumar, “IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages,” in _Findings of EMNLP_ , 2020.
  * [53] T. Dozat and C. D. Manning, “Deep biaffine attention for neural dependency parsing,” 2016. [Online]. Available: https://arxiv.org/abs/1611.01734
  * [54] Y. Tay, V. Q. Tran, S. Ruder, J. Gupta, H. W. Chung, D. Bahri, Z. Qin, S. Baumgartner, C. Yu, and D. Metzler, “Charformer: Fast character transformers via gradient-based subword tokenization,” 2021. [Online]. Available: https://arxiv.org/abs/2106.12672
  * [55] S. Ruder, N. Constant, J. Botha, A. Siddhant, O. Firat, J. Fu, P. Liu, J. Hu, D. Garrette, G. Neubig, and M. Johnson, “XTREME-R: Towards more challenging and nuanced multilingual evaluation,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 10 215–10 245. [Online]. Available: https://aclanthology.org/2021.emnlp-main.802
  * [56] P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, and I. Gurevych, “How good is your tokenizer? on the monolingual performance of multilingual language models,” in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Online: Association for Computational Linguistics, Aug. 2021, pp. 3118–3135. [Online]. Available: https://aclanthology.org/2021.acl-long.243
  * [57] S. J. Mielke, Z. Alyafeai, E. Salesky, C. Raffel, M. Dey, M. Gallé, A. Raja, C. Si, W. Y. Lee, B. Sagot, and S. Tan, “Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp,” 2021. [Online]. Available: https://arxiv.org/abs/2112.10508
  * [58] P. Rust, J. F. Lotz, E. Bugliarello, E. Salesky, M. de Lhoneux, and D. Elliott, “Language modelling with pixels,” 2022. [Online]. Available: https://arxiv.org/abs/2207.06991
  * [59] B. Muller, A. Anastasopoulos, B. Sagot, and D. Seddah, “When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Online: Association for Computational Linguistics, Jun. 2021, pp. 448–462. [Online]. Available: https://aclanthology.org/2021.naacl-main.38
  * [60] Z. Wang, K. K, S. Mayhew, and D. Roth, “Extending multilingual BERT to low-resource languages,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_. Online: Association for Computational Linguistics, Nov. 2020, pp. 2649–2656. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.240
  * [61] A. Lauscher, V. Ravishankar, I. Vulić, and G. Glavaš, “From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 4483–4499. [Online]. Available: https://aclanthology.org/2020.emnlp-main.363
  * [62] J. Pfeiffer, N. Goyal, X. Lin, X. Li, J. Cross, S. Riedel, and M. Artetxe, “Lifting the curse of multilinguality by pre-training modular transformers,” in _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Seattle, United States: Association for Computational Linguistics, Jul. 2022, pp. 3479–3495. [Online]. Available: https://aclanthology.org/2022.naacl-main.255
  * [63] J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” 2018. [Online]. Available: https://arxiv.org/abs/1803.03635
  * [64] T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, Z. Wang, and M. Carbin, “The lottery ticket hypothesis for pre-trained bert networks,” 2020. [Online]. Available: https://arxiv.org/abs/2007.12223
  * [65] S. Prasanna, A. Rogers, and A. Rumshisky, “When BERT Plays the Lottery, All Tickets Are Winning,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 3208–3229. [Online]. Available: https://aclanthology.org/2020.emnlp-main.259
  * [66] N. Foroutan, M. Banaei, R. Lebret, A. Bosselut, and K. Aberer, “Discovering language-neutral sub-networks in multilingual language models,” 2022. [Online]. Available: https://arxiv.org/abs/2205.12672
  * [67] Y. Li, F. Luo, R. Xu, S. Huang, F. Huang, and L. Wang, “Probing structured pruning on multilingual pre-trained models: Settings, algorithms, and efficiency,” 2022. [Online]. Available: https://arxiv.org/abs/2204.02601
  * [68] H. Sajjad, F. Dalvi, N. Durrani, and P. Nakov, “On the effect of dropping layers of pre-trained transformer models,” 2021.
  * [69] K. Stańczak, E. Ponti, L. T. Hennigen, R. Cotterell, and I. Augenstein, “Same neurons, different languages: Probing morphosyntax in multilingual pre-trained models,” 2022. [Online]. Available: https://arxiv.org/abs/2205.02023
  * [70] R. Choenni and E. Shutova, “Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology,” _Computational Linguistics_ , pp. 1–38, 07 2022. [Online]. Available: https://doi.org/10.1162/coli\\_a\\_00444
  * [71] M. Artetxe and H. Schwenk, “Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond,” _Transactions of the Association for Computational Linguistics_ , vol. 7, pp. 597–610, nov 2019. [Online]. Available: https://doi.org/10.1162%2Ftacl_a_00288
  * [72] G. Lample and A. Conneau, “Cross-lingual language model pretraining,” 2019. [Online]. Available: https://arxiv.org/abs/1901.07291
  * [73] P. P. Liang, C. Wu, L.-P. Morency, and R. Salakhutdinov, “Towards understanding and mitigating social biases in language models,” 2021. [Online]. Available: https://arxiv.org/abs/2106.13219
  * [74] A. Immer, L. Torroba Hennigen, V. Fortuin, and R. Cotterell, “Probing as quantifying inductive bias,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 1839–1851. [Online]. Available: https://aclanthology.org/2022.acl-long.129
  * [75] S. Alnegheimish, A. Guo, and Y. Sun, “Using natural sentence prompts for understanding biases in language models,” in _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Seattle, United States: Association for Computational Linguistics, Jul. 2022, pp. 2824–2830. [Online]. Available: https://aclanthology.org/2022.naacl-main.203
  * [76] J. Ahn and A. Oh, “Mitigating language-dependent ethnic bias in BERT,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 533–549. [Online]. Available: https://aclanthology.org/2021.emnlp-main.42
  * [77] P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The state and fate of linguistic diversity and inclusion in the NLP world,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 6282–6293. [Online]. Available: https://aclanthology.org/2020.acl-main.560
  * [78] D. Blasi, A. Anastasopoulos, and G. Neubig, “Systematic inequalities in language technology performance across the world’s languages,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 5486–5505. [Online]. Available: https://aclanthology.org/2022.acl-long.376


WGMwgm QEqe

[ orcid=0000-0001-5138-230X, ]

1]organization=Université de Paris, Institut de Physique du Globe de Paris,
CNRS, city=Paris, postcode=F-75005, country=France

[1]Corresponding author

# Revisiting Dimorphos formation: A pyramidal regime perspective and
application to Dinkinesh’s satellite

Gustavo Madeira<EMAIL_ADDRESS>[  Sebastién Charnoz

###### Abstract

Dimorphos’ oblate shape challenges formation models. Landslides on Didymos,
induced by YORP effect, probably created a debris ring from which Dimorphos
would have formed. Nonetheless, ring-derived satellites typically form with a
prolate lemon shape. In light of the newest Dimorphos shape model, we revisit
our previous work, Madeira et al. (2023a), and conducted new 1D simulations
with material deposition in an extended region from 1.0 to 1.5 Didymos radii.
An instantaneous landslide leads to a fast formation of a prolate Dimorphos
directly from the ring. Now, if Didymos progressively deposits mass, Dimorphos
grows through low-velocity impacts with large impactor-to-target mass ratio
(pyramidal regime growth). Even during rapid, day-scale depositions, Dimorphos
experiences one of these impacts, while for slower depositions, the satellite
formation is primarily via pyramidal impacts. This process might reshape the
satellite into an oblate shape (Leleu et al., 2018) or even in a contact-
binary shape, a scenario worthy of investigation that should be studied in the
future with more suitable tools. Our conclusions can be applied to Dinkinesh’s
satellite, recently discovered by NASA’s Lucy mission.

###### keywords:

Asteroids, dynamics Debris disks Planetary rings Satellites, formation
Satellites, shapes

Didymos deposits material into orbit via landslides.

An instantaneous single landslide typically results in a prolate lemon-shaped
Dimorphos.

For slower depositions, Dimorphos grows via low-velocity impacts with a large
impactor-to-target mass ratio.

For slower depositions, Dimorphos is expected to have a non-prolate shape.

Low-velocity impacts with a large impactor-to-target mass ratio might also
explain the shape of Dinkinesh’s satellite

##  1 Introduction

Didymos’ secondary, Dimorphos, was impacted in September 2022 by NASA’s DART
mission probe, responsible for the first astronomical-scale test of the
kinetic deflection technology (Thomas et al., 2023). Located before the impact
at $a_{d}=1183$ m from Didymos center, Dimorphos has an estimated mass and
density of $M_{d}=4.3\times 10^{9}$ kg and $\rho_{d}=2400$ kg/m3, respectively
(Naidu et al., 2020; Daly et al., 2022, 2023; Thomas et al., 2023). As
revealed by the data from DART mission, the object has a remarkably oblate
shape, with an almost axisymmetric equatorial region (Daly et al., 2023),
which proves to be an obstacle for the current models of Dimorphos formation.
Didymos rotates near its spin limit and is believed to have experienced
landslides and shape deformations in the past (Yu et al., 2018; Zhang et al.,
2021; Trógolo et al., 2023). These events are believed to have given rise to a
ring of debris from which Dimorphos is thought to have formed. Nevertheless,
satellites formed directly from ring accretion are expected to exhibit a
prolate shape (Porco et al., 2007; Charnoz et al., 2010; Madeira et al.,
2023c, a), as opposed to the oblate shape observed in Dimorphos (Daly et al.,
2023). Here, we revisit the work of Madeira et al. (2023a, hereafter M23) and
explore, through 1D simulations, the conditions under which the satellite
could have formed with a oblate (or even a contact-binary) shape. In Section
2, we briefly describe the work of M23 and our new set of numerical
simulations. Our results are addressed in Section 3 and our conclusions drawn
in Section 4.

##  2 Revisiting the work of Madeira et al. (2023a)

M23 propose Dimorphos’ origin from a Didymos debris ring, originated from
landslides due to Didymos spin-up driven by YORP effect (Yu et al., 2018;
Zhang et al., 2021; Trógolo et al., 2023). Viscous ring spreading leads to the
formation of proto-satellites at Didymos’ Fluid Roche limit
(FRL=$1.6926R_{D}$) and, ultimately, to the formation of Dimorphos (Sugiura et
al., 2021; Hyodo & Sugiura, 2022; Madeira et al., 2023a). M23 focuses on the
viscous evolution of the ring and the formation of Dimorphos using the 1D
hybrid code HYDRORINGS (Salmon et al., 2010; Charnoz et al., 2010, 2011). Two
distinct scenarios are considered: one scenario with an initial ring of debris
formed right at Didymos surface in a single instantaneous event, and another
scenario in which the material is progressively deposited at Didymos’ surface
following the function for deposition of material:

$\dot{M}(t)=\frac{M_{T}}{\tau}{\rm exp}\left(-\frac{t}{\tau}\right),$ (1)

where ${\rm M_{T}}$ is the total mass deposited in the ring, $t$ is the
simulation time, and $\tau$ is the deposition timescale.

The mass of the largest satellite’ formed primarily depends on ${\rm M_{T}}$
and 25% of Didymos’ mass (${\rm M_{D}=5.6\times 10^{11}}$ kg, Daly et al.,
2023) must be deposited in the disk to form a satellite with Dimorphos’
estimated mass when the article was written. For $\tau\leq 1$ year, material
spreading beyond the FRL coagulates into a proto-Dimorphos that grows in the
”continuous regime” (Crida & Charnoz, 2012). In this regime, the newly-formed
satellite grows by accumulating material directly from the ring. In other
words, the ring’s edge is inside the satellite’s Hill sphere, so any material
leaving the ring is accreted by the massive nearby satellite (for a detailed
description, see Crida & Charnoz, 2012; Hyodo et al., 2015). Due to ring
torques, Dimorphos migrates outwards, while accreting moonlets formed at the
FRL that are in its gravitational reach (”discrete regime” growth, Crida &
Charnoz, 2012). In this scenario, Dimorphos reaches its final mass in a few
decades and is expected to be a prolate aggregate (Porco et al., 2007; Charnoz
et al., 2010).

For $\tau\geq 10^{2}$ yr, the picture is different: the satellite only reaches
a few percent of Dimorphos’ mass during the continuous regime. As the
satellite moves away, the system evolves to a collection of similarly sized
moonlets that eventually collide to form Dimorphos. This growth regime is
defined as the ”pyramidal regime” (Crida & Charnoz, 2012), and it takes
thousands of years for Dimorphos to form. Leleu et al. (2018) shows that low-
velocity impacts during the pyramidal regime may lead to the formation of
oblate-shaped objects, similar to what is observed for Saturn moons, which
could also be relevant for the oblate Dimorphos (Daly et al., 2023).

However, there are some caveats regarding a slow formation of Dimorphos in the
pyramidal regime. Numerical simulations indicate that landslides typically
last only a few hours (Sugiura et al., 2021; Hyodo & Sugiura, 2022), a
timescale considerably shorter than ours. Moreover, Didymos is subject to
strong solar influence, which reduces the lifetime of material around the
object (see Rossi et al., 2022; Madeira et al., 2023b). Thus, a faster
formation of Dimorphos is preferable.

Here, we present new simulations to explore Dimorphos’ pyramidal regime growth
more accurately. Our numerical model was improved using the secular
perturbation theory, calculating first-order satellite interactions (Brouwer &
Clemence, 1961; Dermott & Nicholson, 1986). Secular interactions induce
oscillations of orbital elements, and in each simulation step, we derive the
secular terms of the satellites’ disturbing function, solving them to
determine the forced elements (for details, see Chapter 7 of Murray & Dermott,
1999). Additionally, we calculate the eccentricity kicks on a satellite due to
the crossing of a first-order mean motion resonance (Murray & Dermott, 1999;
Madeira & Giuliatti Winter, 2020). With this, we can calculate the
eccentricity of the satellites (at least in order of magnitude), which will
allow us to crudely estimate the impact velocities.

To speed up simulations, we now start with a debris ring that occupies a
broader region, rather than being confined to the primary’s surface. Indeed
Hyodo & Sugiura (2022); Trógolo et al. (2023) have demonstrated that material
is ejected over a much wider region than what was assumed in M23 and, based on
their findings, we assume a uniform deposition of material from Didymos’
surface ($R_{D}=382$ m, Daly et al., 2023) to $1.5R_{D}$. Didymos’ tidal
parameter is set as $k_{2}/Q=10^{-5}$.

##  3 Extended debris ring simulations and Dimorphos formation

(a)

(b)

(c)

(d)

Figure 1: Summary of Dimorphos formation as a function of the deposition time
scale $\tau$ with panels showing: a) total mass $M_{T}$ for Dimorphos
formation. b) number of impacts in the pyramidal regime (left y-scale, dashed
line) and the average impactor-to-target mass ratio mimp/mtar (right y-scale,
solid line) with error bars indicating the range of mimp/mtar. c) time and d)
mass of Dimorphos at the end of the continuous regime, discrete regime, and
last impact with Dimorphos.

We first consider the instantaneous formed ring case ($\tau=0$) and then
explore scenarios where Didymos progressively supplies the ring over a
deposition timescale $\tau$111This deposition timescale $\tau$ should not be
confused with the disk lifetime $\tau_{\rm disk}$ in Crida & Charnoz (2012).
($\tau=10^{-3}-10^{2}$ years). Our simulations are summarized in Figure 1. In
the case of instantaneous deposition, Dimorphos can form from a ring with only
$0.04~{}{\rm M_{D}}$ (Fig. 1a), a much lower value than that reported in M23,
which assumes with a narrow, confined ring. This discrepancy is primarily due
to the comparable angular momentum of the ring in both cases. For M23, the
total angular momentum brought into the system is $\sim 2\times
10^{-12}~{}m^{2}/s$, while for the instantaneous formed extended ring the
angular momentum is $3\times 10^{-12}~{}m^{2}/s$.

In the instantaneous case, proto-Dimorphos reaches $\sim 0.74~{}{\rm M_{d}}$
in 21 hours ($\sim 0.04$ year) in the continuous and discrete regimes (Fig.
1c,1d). The satellite migrates outward until it reaches the 2:1 MMR position
with the Roche limit (a$\sim$1027 m), which is the farthest distance reachable
due to ring torques. At this location, Dimorphos achieves its final mass ${\rm
M_{d}}$ in $\sim 6$ months through three impacts with impactor-to-target mass
ratio of mimp/m${}_{\rm tar}\sim 5-15\%$. This differs from the typical
pyramidal regime, as the impactors here are much smaller than proto-Dimorphos.
In view of this, we define the limit of mimp/mtar=0.25 for the impact to be
considered in the pyramidal regime. No impact meets this condition for
$\tau=0$ year and therefore, a prolate Dimorphos is expected to form.

Now, let’s explore scenarios in which Didymos progressively supplies the ring.
As $\tau$ increases, the mass required for Dimorphos formation decreases. With
higher $\tau$, the ring’s viscous spreading rate slows down, causing less
material to fall onto Didymos, thus increasing the efficiency of converting
ring mass into satellites. However, this also results in the formation of
smaller satellites at the Roche limit. Consequently, as $\tau$ increases,
Dimorphos experiences reduced growth in the continuous and discrete regimes,
while we observe an increase in the number of impacts in the pyramidal regime.

For $\tau=0.001$ and $0.01$ years, Dimorphos acquires $\sim 0.4{\rm M_{d}}$ of
mass in the continuous and discrete regimes, then experiencing just one
pyramidal regime impact. This, however, might reshape it. With increasing
$\tau$, Dimorphos mostly forms in the pyramidal regime, being obtained more
collisions with higher mass ratio. These collisions occur at velocities of
$\sim 2-3$ mutual escape velocity and can lead to partial accretion or hit-
and-run events, potentially leading to a non-prolate satellite (Leleu et al.,
2018). This is particularly pronounced for $\tau=100$ yr, for which the
maximum mimp/m${}_{\rm tar}=0.94$ (red circle) and so, could give rise to a
satellite with contact-binary shape. Impacts involving similar-sized objects
at such velocities resemble the cases in which Leleu et al. (2018) obtain an
oblate shape.

##  4 Conclusion

We revisit Madeira et al. (2023a) and conduct new simulations with an extended
disk and shorter mass-deposition timescales for Didymos. An instantaneous
landslide on Didymos, distributing 4% of its mass uniformly, would create
Dimorphos in just a few hours, albeit likely with a prolate shape. However, if
we assume a deposition event with a finite duration, the growth of Dimorphos
directly from the ring is inhibited, and the satellite accumulates mass
through low-velocity, high-mass-ratio collisions. Even in a deposition lasting
a few days, Dimorphos undergoes one such impact, potentially altering its
initial prolate shape. Longer duration events lead to almost complete growth
in the pyramidal regime (Crida & Charnoz, 2012). The impacts associated to
this regime may result in an oblate object, as observed by the DART mission.
These conclusions certainly also apply to the recently discovered satellite of
asteroid (152830) Dinkinesh by the Lucy mission. This moon is a contact
binary, the result of a low-velocity encounter between two equally sized
satellites, which defines the pyramidal regime.

## Acknowledgments

G.M. thanks to the Centre National d’Études Spatiales (CNES) and European
Research Council (ERC). Numerical computations were performed on the
S-CAPAD/DANTE platform, IPGP, France. We thank Aurélien Crida for the really
nice comments that helped us to significantly improve the article.

## References

  * Brouwer & Clemence (1961) Brouwer, D., Clemence, G.M., 1961. Methods of celestial mechanics.
  * Charnoz et al. (2011) Charnoz, S., Crida, A., Castillo-Rogez, J.C., Lainey, V., Dones, L., Karatekin, Ö., Tobie, G., Mathis, S., Le Poncin-Lafitte, C., Salmon, J., 2011. Accretion of Saturn’s mid-sized moons during the viscous spreading of young massive rings: Solving the paradox of silicate-poor rings versus silicate-rich moons. Icarus 216, 535–550. doi:10.1016/j.icarus.2011.09.017, arXiv:1109.3360.
  * Charnoz et al. (2010) Charnoz, S., Salmon, J., Crida, A., 2010. The recent formation of Saturn’s moonlets from viscous spreading of the main rings. Nature 465, 752–754.doi:10.1038/nature09096.
  * Crida & Charnoz (2012) Crida, A., Charnoz, S., 2012. Formation of Regular Satellites from Ancient Massive Rings in the Solar System. Science 338, 1196. doi:10.1126/science.1226477, arXiv:1301.3808.
  * Daly et al. (2022) Daly, R., Ernst, C.M., Barnouin, O.S., Gaskell, R.W., Palmer, E.E., Nair, H., Espiritu, R.C., Hasnain, S., Waller, D., Stickle, A.M., Nolan, M.C., Trigo-Rodríguez, J.M., Dotto, E., Lucchetti, A., Pajola, M., Ieva, S., Michel, P., 2022. Shape Modeling of Dimorphos for the Double Asteroid Redirection Test (DART). The Planetary Science Journal 3, 207. doi:10.3847/PSJ/ac7523.
  * Daly et al. (2023) Daly, R.T., Ernst, C.M., Barnouin, O.S., Chabot, N.L., Rivkin, A.S., Cheng, A.F., Adams, E.Y., Agrusa, H.F., Abel, E.D., Alford, A.L., et al., 2023. Successful kinetic impact into an asteroid for planetary defense. Nature , 1–3.
  * Dermott & Nicholson (1986) Dermott, S.F., Nicholson, P.D., 1986. Masses of the satellites of Uranus. Nauret 319, 115–120.doi:10.1038/319115a0.
  * Hyodo et al. (2015) Hyodo, R., Ohtsuki, K., Takeda, T., 2015. Formation of Multiple-satellite Systems From Low-mass Circumplanetary Particle Disks. ApJ 799, 40. doi:10.1088/0004-637X/799/1/40, arXiv:1411.4336.
  * Hyodo & Sugiura (2022) Hyodo, R., Sugiura, K., 2022. Formation of Moons and Equatorial Ridge around Top-shaped Asteroids after Surface Landslide. ApJL 937, L36. doi:10.3847/2041-8213/ac922d, arXiv:2209.07045.
  * Leleu et al. (2018) Leleu, A., Jutzi, M., Rubin, M., 2018. The peculiar shapes of Saturn’s small inner moons as evidence of mergers of similar-sized moonlets. Nature Astronomy 2, 555–561. doi:10.1038/s41550-018-0471-7, arXiv:1805.08682.
  * Madeira et al. (2023a) Madeira, G., Charnoz, S., Hyodo, R., 2023a. Dynamical origin of Dimorphos from fast spinning Didymos. Icarus 394, 115428. doi:10.1016/j.icarus.2023.115428, arXiv:2301.02121.
  * Madeira et al. (2023b) Madeira, G., Charnoz, S., Rambaux, N., Robutel, P., 2023b. Could Didymos harbor material? A study of long-term dynamics in Didymos and Dimorphos system. Icarus in submission.
  * Madeira et al. (2023c) Madeira, G., Charnoz, S., Zhang, Y., Hyodo, R., Michel, P., Genda, H., Giuliatti Winter, S., 2023c. Exploring the Recycling Model of Phobos Formation: Rubble-pile Satellites. The Astronomical Journal 165, 161. doi:10.3847/1538-3881/acbf53, arXiv:2302.12556.
  * Madeira & Giuliatti Winter (2020) Madeira, G., Giuliatti Winter, S.M., 2020. Effects of immersed moonlets in the ring arc particles of Saturn. European Physical Journal Special Topics 229, 1527–1543. doi:10.1140/epjst/e2020-900129-5, arXiv:2004.14972.
  * Murray & Dermott (1999) Murray, C.D., Dermott, S.F., 1999. Solar system dynamics. Cambridge university press.
  * Naidu et al. (2020) Naidu, S.P., Benner, L.A.M., Brozovic, M.,Nolan, M.C., Ostro, S.J., Margot, J.L., Giorgini, J.D., Hirabayashi, T., Scheeres, D.J., Pravec, P., Scheirich, P., Magri, C., Jao, J.S., 2020. Radar observations and a physical model of binary near-Earth asteroid 65803 Didymos, target of the DART mission. Icarus 348, 113777. doi:10.1016/j.icarus.2020.113777.
  * Porco et al. (2007) Porco, C.C., Thomas, P.C., Weiss, J.W., Richardson, D.C., 2007. Saturn’s Small Inner Satellites: Clues to Their Origins. Science 318, 1602. doi:10.1126/science.1143977.
  * Rossi et al. (2022) Rossi, A., Marzari, F., Brucato, J.R., Della Corte, V., Dotto, E., Ieva, S., Ivanovski, S.L., Lucchetti, A., Epifani, E.M., Pajola, M., et al., 2022. Dynamical evolution of ejecta from the dart impact on dimorphos. The Planetary Science Journal 3, 118.
  * Salmon et al. (2010) Salmon, J., Charnoz, S., Crida, A., Brahic, A., 2010. Long-term and large-scale viscous evolution of dense planetary rings. Icarus 209, 771–785.doi:10.1016/j.icarus.2010.05.030, arXiv:1006.0633.
  * Sugiura et al. (2021) Sugiura, K., Kobayashi, H., Watanabe, S.i., Genda, H., Hyodo, R., Inutsuka, S.i., 2021. SPH simulations for shape deformation of rubble-pile asteroids through spinup: The challenge for making top-shaped asteroids Ryugu and Bennu. Icarus 365, 114505. doi:10.1016/j.icarus. 2021.114505, arXiv:2104.13516.
  * Thomas et al. (2023) Thomas, C.A., Naidu, S.P., Scheirich, P., Moskovitz, N.A., Pravec, P., Chesley, S.R., Rivkin, A.S., Osip, D.J., Lister, T.A., Benner, L.A., et al., 2023. Orbital period change of dimorphos due to the dart kinetic impact. Nature , 1–3.
  * Trógolo et al. (2023) Trógolo, N., Bagatin, A.C., Moreno, F., Benavidez, P.G., 2023. Lifted particles from the fast spinning primary of the near-earth asteroid (65803) didymos. Icarus 397, 115521.
  * Yu et al. (2018) Yu, Y., Michel, P., Hirabayashi, M., Schwartz, S.R., Zhang, Y., Richardson, D.C., Liu, X., 2018. The Dynamical Complexity of Surface Mass Shedding from a Top-shaped Asteroid Near the Critical Spin Limit. AJ 156, 59. doi:10.3847/1538-3881/aaccf7.
  * Zhang et al. (2021) Zhang,Y., Michel, P., Richardson, D.C., Barnouin, O.S., Agrusa, H.F., Tsiganis, K., Manzoni, C., May, B.H., 2021. Creep stability of the DART/Hera mission target 65803 Didymos: II. The role of cohesion. Icarus 362, 114433. doi:10.1016/j.icarus.2021.114433.