# Unraveling Code-Mixing Patterns in Migration Discourse:
Automated Detection and Analysis of Online Conversations on Reddit

Fedor Vitiugin, Sunok Lee, Henna Paakki, Anastasiia Chizhikova, Nitin Sawhney

###### Abstract

The surge in global migration patterns underscores the imperative of
integrating migrants seamlessly into host communities, necessitating inclusive
and trustworthy public services. Despite the Nordic countries’ robust public
sector infrastructure, recent immigrants often encounter barriers to accessing
these services, exacerbating social disparities and eroding trust. Addressing
digital inequalities and linguistic diversity is paramount in this endeavor.
This paper explores the utilization of code-mixing, a communication strategy
prevalent among multilingual speakers, in migration-related discourse on
social media platforms such as Reddit.

We present Ensemble Learning for Multilingual Identification of Code-mixed
Texts (ELMICT), a novel approach designed to automatically detect code-mixed
messages in migration-related discussions. Leveraging ensemble learning
techniques for combining multiple tokenizers’ outputs and pre-trained language
models, ELMICT demonstrates high performance (with F1 more than 0.95) in
identifying code-mixing across various languages and contexts, particularly in
cross-lingual zero-shot conditions (with avg. F1 more than 0.70). Moreover,
the utilization of ELMICT helps to analyze the prevalence of code-mixing in
migration-related threads compared to other thematic categories on Reddit,
shedding light on the topics of concern to migrant communities.

Our findings reveal insights into the communicative strategies employed by
migrants on social media platforms, offering implications for the development
of inclusive digital public services and conversational systems. By addressing
the research questions posed in this study, we contribute to the understanding
of linguistic diversity in migration discourse and pave the way for more
effective tools for building trust in multicultural societies.

## Introduction

Between 2000 and 2020, global migration patterns witnessed significant shifts,
with a 74% growth, equivalent to approximately 37 million people. Europe
experienced an increase of 30 million migrants, closely followed by North
America with 18 million, and Africa with 10 million migrants (McAuliffe et al.
2022). The escalating diversity emphasizes the importance of seamlessly
integrating migrants into local processes and requires public services to
facilitate smooth adaptation.

|  language |  message
---|---|---
M1 |  English |  Moved here from France because my wife is Finnish. I enjoy it enough to want to stay here.
M2 |  code-mixed |  WOW, sounds great to me, man. OK, kiitos paljon for your answer! [EN] WOW, sounds great to me, man. OK, thanks a lot for your answer!
M3 |  code-switched |  Sun tarinas on yksi hyvä syy miksi oppia Englanti. And of course… you know what happen next. [EN] Her/his story is one good reason to learn English. And of course… you know what happen next. .
M4 |  Finnish |  Moikka! En oo sujuva suomen kielessä ja juuri nyt opiskelen suomen lukiossa. [EN] Hello! I’m not fluent in Finnish, and right now, I’m studying at a Finnish high school.
Table 1: Examples of code-mixed and non-mixed messages for English-Finnish
pair. (Note: messages were paraphrased for anonymity.)

The Nordic countries have well-functioning and mostly equitable public sector
services, earning the trust of a majority of citizens. However, this trust and
efficiency do not always extend to recent immigrants and migrant communities
residing in the region. For many migrants, these public services may seem
inaccessible, lacking inclusivity or trustworthiness, which significantly
undermines their integration (Yeasmin et al. 2020; Intke-Hernández et al.
2015).

Digital inequalities among specific groups can exacerbate social disparities,
further marginalizing them and potentially undermining trust (Madianou 2015).
Therefore, it’s vital for local municipalities to prioritize objectives like
enhancing integration, promoting inclusion, and supporting migrant
communities. Many cities are actively working to create innovative digital
public services, such as chatbots, especially in areas like healthcare,
employment, and social services. However, it’s essential to ensure that these
services are accessible to users with diverse linguistic backgrounds and
varying levels of digital literacy.

In the public sector and non-profit organizations assisting migrants, language
is not restricted to a single mode of expression. Multilingual speakers tend
to interleave two or more languages when communicating, a phenomenon known as
code-mixing. This strategy has become increasingly prevalent in today’s
diverse linguistic and cultural landscape (Gumperz 1982). Due to this
communication style, migrants naturally lean towards code-mixing to more
effectively convey their circumstances and context.

A recent study highlights the complex linguistic practices employed by
migrants in computer-mediated communication (McEntee-Atalianis, Ateek, and
Gardner-Chloros 2023; Ikeh 2023). Social media platforms provide bilingual
users with a dynamic space to navigate their multiple identities online post-
resettlement (Harwood et al. 2019; Gardner-Chloros 2020). Our research focuses
on the Reddit platform, selected not only for the availability of data
collection but also for its community-based structure and user-generated
thread labels, simplifying content analysis.

Table 1 demonstrates examples of various code-mixed, code-switched, and non-
mixed messages. M1 and M4 are prototypical single-language messages in English
and Finnish, respectively. M2 is an example of a code-mixed message, where the
user included the Finnish phrase “kiitos paljon” instead of “thanks a lot” in
his English message. Finally, M3 is an example of a code-switched message,
where the user starts their text with a Finnish sentence to explain the
situation and continues with an English sentence. We will explain the
difference between code-mixing and code-switching in the next section.

The emergence of Multilingual Large Language Models (LLMs) has demonstrated
exceptional capabilities across various tasks (Chang et al. 2023; Zahera et
al. 2023), showcasing state-of-the-art performance through zero-shot or few-
shot methods. While extensive research has explored their monolingual
capabilities, their potential in cross-lingual communication remains
relatively unexplored (Zhang et al. 2023). However, current intelligence-based
conversational systems often fail to meet the communicative expectations of
multilingual migrant users, resulting in linguistic and cultural barriers.
Consequently, there is an urgent need for the public sector to evolve these
systems, considering the communication needs of migrants.

We explore migrants’ information requests shared on social media. Our paper
addresses the following research questions:

RQ1: Can we automatically identify code-mixed social media messages from
Reddit related to migration to Finland?

RQ2: How proficiently can the proposed approach identify instances of code-
mixing in social media conversations in cross-lingual zero-shot conditions

RQ3: Which content topics exhibit a high proportion of code-mixed messages,
and what are the differences in code-mixing usage between migration-related
threads and threads in other user-defined categories?

To tackle the first two questions, we introduced a flexible approach named
Ensemble Learning for Multilingual Identification of Code-mixed Texts
(ELMICT), which relies on ensemble learning techniques (Abimannan et al.
2023). This method effectively detects code-mixed social media messages. Our
model integrates outputs from multiple tokenizers and fine-tuned pre-trained
language models to identify texts containing code-mixing. We illustrate that
using tokenizers or fine-tuned models separately yields lower performance and
is less robust, particularly for texts containing out-of-vocabulary tokens. In
addressing the third question, we calculate the proportion of code-mixing
messages across various topics, including migration, tourism, politics, and
general discussions.

The subsequent section of this paper will first present related research,
followed by an explanation of our proposed method for detecting code-mixed
social media messages and the setup for topic modeling of detected messages.
Following this, we will detail our experimental setup and analyze the results.
Finally, we will offer our conclusions and outline potential future work.

## Related Work

In this section, we will discuss relevant works about code-mixing, and
research on its usage in migrant communication. We also discuss methods for
code-mixed text identification and the application of these methods for
different tasks.

### Code-Mixing and Code-Switching

Central to this work is the linguistic concept called “code-mixing”, how it
differs from “code-switching” and other language alternating techniques. Both
are commonly used throughout the world, and are especially crucial for
communities of migrants, expats, bilinguals, etc (Gardner-Chloros 2020;
McEntee-Atalianis, Ateek, and Gardner-Chloros 2023). These occur when two
languages are used spontaneously in one sentence or expression.

Although the main purpose of our work is to research code-mixing in migration-
related communication, we also wish to provide key definitions and discuss the
differences between code-mixing and code-switching, as these are crucial for
this study. More detailed information on these phenomena provided in related
linguistics research cited in this subsection.

Many scholars have attempted to define code-switching and code-mixing.
Weinreich, a leading researcher on bilingualism, has claimed that “the ideal
bilingual is someone who is able to switch between languages when required to
do so by changes in the situation but does not switch when the speech
situation is unchanged and certainly not within a single sentence” (Weinreich
1953). Specialists in code switching, however, recognize code switching as a
functional practice and as a sign of bilingual competence (Toribio 2001).
Competence includes two aspects: fluency in speaking two or more languages and
comprehensive understanding of them, even if speaking fluently is not
necessary. It’s evident that code-switching requires a high level of
proficiency in multiple languages, rather than being a consequence of
insufficient knowledge in one or the other language (Poplack et al. 2000).

Code-switching refers to the “use of two or more languages in the same
conversation, usually within the same conversational turn, or even within the
same sentence of that turn” (Myers-Scotton 1993). Code-switching is the
shifting by a speaker from language A to language B.

There are varying definitions of code-mixing. It’s described as instances
where a mix between the grammar of one language and another language is
employed without changing the grammar of the initial language used (Mabule
2015). On the other hand, “Conversational code-mixing involves the deliberate
mixing of two languages without an associated topic change” (Wardhaugh and
Fuller 2021). The definition indicates that code-mixing is typically used as a
solidarity marker in multilingual communities. Similarly, according to other
views, in code-mixing speakers switch between languages even within words
(e.g. Spanglish or Finglish as a mixture of the English and Spanish or English
and Finnish languages relatively) and/or phrases (Milroy and Muysken 1995;
Auer 2013)

In this paper, the term “code-mixing” is used to indicate a switch between
languages, in which a single word or phrase from one language (here: Finnish,
Spanish, or Korean) is integrated into another language (here: English).

### The Role of Code-Mixing in Migrant Communication

Code-mixing among multilingual speakers commonly observed in close
relationships, particularly when speaking with friends and family who share
similar linguistic and cultural backgrounds (Wulandari 2021). However,
speakers tend to avoid code-mixing if they’re unsure how their interlocutors
will react. Moreover, even when speakers are aware of their conversational
partner’s language proficiency, they may adjust their language usage to match
the partner’s code-mixing style and frequency, especially if trust is
perceived to be lacking (Kusumawati and Prihadi 2023). This highlights how
code-mixing serves as an indicator of trust and intimacy levels among
multilingual speakers (Choi, Lee, and Lee 2023).

From the civil service practitioners’ side, it is critical to make sure that
services are accessible at the user experience level and linguistically,
rather than broader aspects of its design and impact. Practitioners cited the
lack of staff diversity and linguistic exclusion as the main challenges for
better inclusion of citizens in such services (Drobotowicz et al. 2023). On
the other hand, migrants may encounter challenges, particularly in critical
contexts such as local government offices and hospitals, which place greater
demands on language proficiency (St John 2023). Moreover, personification of
the conversational agent could increase engagement (Ostrowski et al. 2021),
increase trust and relationships (Schaefer et al. 2016; Luria et al. 2019).

Conversational agents fail to understand users for many reasons, multilingual
users often blame their unique speech behavior—code-mixing and drop the
conversation or think they have lost control of the device because they do not
understand the reason for the failure (Ponnusamy et al. 2022). Experiences
like this could greatly diminish the users’ well-established trust and
intimacy with the conversational agent. For this reason, a code-mixing
conversational agent should be designed to make clear statements and detailed
explanations of their failure to prevent the multilingual users from getting
frustrated by unnecessary misunderstanding (Yap, Lee, and Roto 2021).

Recent study participants prefer their agent to avoid unnecessary code-mixing
but understand its usage in certain contexts. This preference originates from
experiences where they were perceived as code-mixing due to language
limitations and the importance of trust for acceptance in relationships.
Additionally, designers could enhance trust and intimacy with code-mixing
users by giving the agent a persona with diverse cultural or language
backgrounds and similar code-mixing skills. This would enable users to contact
the agent, similar to how they interact with other multilingual individuals
(Choi, Lee, and Lee 2023).

### Code-Mixed Data Processing

Recently, there has been a growing interest in the development of language
models and technologies tailored for handling code-mixed content. Researchers
have delved into exploring joint models capable of simultaneously performing
language identification and part-of-speech tagging (Barman, Wagner, and Foster
2016). This dual-level language identification spans both word and sentence
levels (Rani, McCrae, and Fransen 2022). A method, based on the UDLDI model,
employs a CNN architecture that incorporates enriched sentence and word
embeddings (Goswami et al. 2020).

Addressing the complexities of code-mixed content, certain studies have
simplified texts by transforming them into a monolingual form through back-
transliteration (Dowlagar and Mamidi 2021; Gautam et al. 2021). However, the
efficacy of these techniques heavily relies on the accuracy of transliteration
and translation methods employed.

Transfer Learning approaches have gained widespread attention in leveraging
pre-trained language models for analyzing code-mixed data (Aguilar and Solorio
2019; Krishnan et al. 2021). Yet, the substitution of tokens in cross-lingual
transfer learning can introduce grammatical inconsistencies in the resultant
sentences, potentially impairing performance on token-sensitive tasks. To
overcome this challenge, token-alignment techniques have emerged, facilitating
not only token replacement but also considering contextual similarity to
ensure grammatical coherence in both training and inference stages (Feng, Li,
and Koehn 2022). The word segmentation method has shown promising results in
code-mixed data processing. Utilization of a linguistics-based toolkit is
maintaining the quality of monolingual translation with Hokkien-Mandarin code-
mixed texts, widespread among Chinese immigrants (Lu et al. 2023).

A review of recent literature underscores a pronounced emphasis on the
tokenization issue. Indeed, accurate tokenization and word segmentation
significantly enhance performance in code-mixing-related tasks. Furthermore,
many studies have used synthetic training data, posing challenges for further
analysis of real-world scenarios where users employ code-mixing in their
communication.

## Method

This study aims to explore the usage of code-mixing by migrants in social
media. In this section, we present a supervised-learning classification model
for detecting code-mixing in social media and describe methods used for
analyzing code-mixed texts.

### Text Classification

Recent work in text classification analysis clearly demonstrates the necessity
of precise word segmentation and tokenization. Compound words, which are quite
rare in the majority of languages, play a significant role in Finnish. Figure
1 demonstrates the difference in English and Finnish pre-trained tokenizer
outputs for the word “terveyskeskus” which means “public health center”. This
word is widely used not only by locals but also migrants, and plays an
important role in their daily vocabulary.

Figure 1: Example of differences in single-language pre-trained model
tokenizer outputs.

The Ensemble Learning for Multilingual Identification of Code-mixed Texts
(ELMICT) model aims to merge pre-trained language models with features
generated by tokenizers through ensemble modeling. To ensure the
classification model receives comprehensive information, we experimented with
various combinations of tokenizer outputs, ultimately retaining four of them:

  * •

English BPE-tokenizer – English language is used as the basic language in our
datasets, so we choose the tokenizer from the most popular 111based on
HuggingFace.com model popularity statistics English transformer model;

  * •

local language BPE-tokenizer – Finnish, Korean, or Spanish BPE-tokenizer for
related datasets;

  * •

multilingual BPE-tokenizer – we found that for some cases, multilingual
tokenizers are also providing correct outputs and include multilingual BERT in
our model;

  * •

whitespace tokenizer – NLTK whitespace tokenizer provides additional
information as the most naive method.

Two other components of the proposed model are a language detection tool and a
fine-tuned pre-trained transformer model for code-mixing detection. For
language detection, lingua Python library
222https://github.com/pemistahl/lingua-py was used in mixed-language mode. The
model received information about the existence of English and local
wordsphrases in the target text. XLM-RoBERTa was used for contextual detection
of code-mixing. We fine-tuned a pre-trained model for the sequence
classification task on English-Finnish texts (for both monolingual and cross-
lingual tasks).

The architecture of ELMICT model presented in Figure 2 has combined two
approaches: contextual and feature-based. For the contextual approach, we
fine-tune the multilingual pre-trained large language model. Soft labels
output from the fine-tuned model were used as features for the ensemble model.
As features, we used information extracted by 4 tokenizers.

Our approach is based on the intuition that specialized tokenizers (Finnish
tokenizer for a word in Finnish) will split relevant text into tokens more
accurately, while unspecialized tokenizers (like English applied for word in
Finnish) will generate more tokens (parts of word). It means that when an
ensemble learning model receives the result of tokenization from different
models, it can track which tokenization split of out-of-vocabulary tokens was
done wrong.

Figure 2: ELMICT model architecture.

### Topic Modeling

To analyze the difference in migrant-related and general Reddit posts, we
applied BERTopic technique (Grootendorst 2022) for topic modeling. The method
utilizes BERT embeddings (Devlin et al. 2018) to cluster the texts and
leverages c-TF-IDF algorithm to further generate topic representations.

First, we utilized sentence embeddings to convert input documents into
numerical representations, enabling the capture of semantic similarity between
documents. Second, we employed the UMAP dimensionality reduction algorithm
(McInnes et al. 2018) to address the high dimensionality of embeddings, which
can make clustering challenging due to the curse of dimensionality. The
clustering algorithm was used HDBSCAN (McInnes, Healy, and Astels 2017), the
default for BERTopic. Third, we experimented with various topic representation
parameters and decided to use uni- and bi-grams only. Finally, we tested
different minimal topic sizes and determined a threshold of 0.3% of the
original dataset size. These steps resulted in not only a high coherence score
of 0.8 but also in topics that are interpretable by humans, which we further
analyze in-depth.

### Model Implementation

We maintain the same number of layers as the original pre-trained model – 24
layers for XML-RoBERTa (Conneau et al. 2019). For the model’s fine-tuning, we
used $0.5*10^{-5}$ learning rate, 10 epochs. The number of frozen layers for
each model was detected by grid search. The model was trained on NVIDIA
A100-SXM4 with 40Gb GPU RAM.

## Experiment Setup

### Data collection and annotation

We collected posts and comments through the official Reddit API
333https://www.reddit.com/dev/api/ from three country-related communities
(subreddits): r/Finland, r/korea, and r/GoingToSpain. All three communities
primarily use English, making them more accessible for migrants. Each
community has user-generated topic-related labels known as “flair”, including
migration-related flair.

All messages collected from location related communities were manually
annotated by one human assessor with living experience in the corresponding
area and language proficiency both in English and the code-mixed language.
Additionally, we enlisted two individuals residing in each location to label
100 random messages from their respective communities to calculate assessor
agreement. The Krippendorff’s alpha for r/GoingToSpain is 0.87, for r/Finland
is 0.75, and for r/korea – 0.92.

The labeling task was to assign one of the two classes for determining whether
a given message is code-mixed or not for a target language pair. For uncertain
words, the authors consulted with individuals, who are both proficient or
native in target languages and currently living in the target country.

Before dataset annotation, we conducted a simple preprocessing step to filter
out all uninformative tweets (based on manual analysis of a random sample,
more than 92% of messages with length $\leq$ 4 tokens are uninformative).
Table 2 presents the quantity of train and test instances for each category,
as well as unlabeled text entities.

Community |  Unlabelled |  Migration |  Non-mixed |  Code-mixed
---|---|---|---|---
r/GoingToSpain |  39871 |  3514 |  878 |  122
r/Finland |  174212 |  12632 |  2055 |  249
r/korea |  130388 |  3337 |  973 |  27
Table 2: Dataset statistics.

During the process of labelling data related to Finland, several types of
Finnish concepts were detected. The first group consists of cultural concepts
and includes words like “sisu” (strength of will), “handknit villasukat” (hand
knitted wool socks as marker of coziness), and “mummola” (grandmother’s
house). The second group contains words related to civil organizations and
public services, like “tilastokeskus” (national statistical institution in
Finland), “terveyskeskus” (public health center). The third group contains
figurative compound words: “piruntorjuntabunkkeri” (church), “betonihelvetti”
(concrete buildings). The final group is obscene language and slang: “mamu”;
“ryssä”. Code-mixing generally occurs without special marking within the
sentence, or sometimes marked by quotation marks.

In the context of texts related to migration in Korea, code-mixing has been
observed predominantly with the use of Korean terms that reflect specific
cultural and social contexts: “mukbanger”, “닭발”, “hagwon”, and “chaebol”, etc.
In code-switching texts, Korean words are romanized, meaning they are
transcribed phonetically into English, rather than being written directly in
Hangul, the Korean alphabet. For example, due to Korean culture’s unique
practice of using specific titles instead of names to address someone, terms
like “unnie”, which means older sister, or referring to a child’s father by
combining “Papa” with the child’s name, are used. Additionally, “mukbanger”,
which refers to a YouTuber who broadcasts their eating, and “닭발” (translated
as “chicken feet”), a word included to represent a facet of Korean food
culture, illustrate the expression of cultural phenomena related to food that
originated in Korea. Similarly, although “hagwon”, denoting a private tutoring
academy, can be translated into English, its use more precisely reflects
Korea’s unique educational culture. Moreover, “다문화” (damunhwa) is used to
refer to people from diverse cultural backgrounds within Korean society;
although ’migrant’ exists as an English equivalent, “damunhwa” is used to
convey the societal context more accurately. Notably, terms like “chaebol”
(representing rich people or conglomerates) and “JY Lee”, a quintessential
figure in Korean chaebol culture, are utilized to denote Korea’s distinctive
corporate culture.

In the Spanish migration context, the majority of cases in which there was
code-switching occurred are specific bureaucratic terms like “extranjeria”
(foreigner), “empadronamiento” (census), “pareja de hecho” (domestic
partnership) etc. These words and phrases do not have a direct English
translation and in the context of conversations related to migration, it is
important to be precise with the terms, so the users use the right Spanish
terminology. Interestingly, sometimes they do that with terms that could be
easily translated to English: “Generally you should be fine with the seguridad
social (social security)…” Other, much less frequent cases include the
insertion of Spanish slang “guirris” (tourists from Northern Europe or UK) and
the usage of greetings (Hola (Hi)! at the beginning of a message in English).

### Schemes

To evaluate the proposed method, we compared it with several state-of-the-art
models. The full list of proposed modeling schemes for evaluation is the
following (* denotes our proposed models and others are the baselines):

  * •

lingua – library for language identification based on model and data provided
by (Biemann et al. 2004);

  * •

Random Forest – classification model outputs of ensemble of tokenizers;

  * •

Adaptive Boosting – classification model outputs of ensemble of tokenizers;

  * •

Gradient Boosting – classification model outputs of ensemble of tokenizers;

  * •

XLM-RoBERTa – multilingual XLM-RoBERTa fine-tuned for code-mixed texts
identification;

  * •

ChatGPT-3.5 – zero-shot setting for detecting code-mixed texts with use of
OpenAI’s ChatGPT-3.5;

  * •

* ELMICT – the model based on Ensemble Learning for Multilingual Identification of Code-mixed Texts.

We utilize three metrics to assess the effectiveness of classification models
for detection of code-mixing, which are Accuracy (ACC), Area Under the
Receiver Operating Characteristic Curve (AUC), and macro F-measure (F1), in
alignment with practices of evaluation of binary text classification.

## Result Analysis and Discussion

We begin by presenting the performance results of the proposed ELMICT model
compared to baseline schemes for research question RQ1. This is followed by an
analysis of the proposed model’s performance for cross-lingual classification
for RQ2, and a detailed examination of social media content featuring code-
mixing for RQ3.

Model Scheme |  ACC |  F1 |  AUC
---|---|---|---
lingua |  78.70$\pm$2.18 |  62.39$\pm$3.16 |  73.20$\pm$5.28
Random Forest |  71.74$\pm$0.08 |  71.55$\pm$0.27 |  71.85$\pm$0.64
Adaptive Boosting |  69.57$\pm$5.75 |  69.17$\pm$6.21 |  69.77$\pm$6.55
Gradient Boosting |  68.70$\pm$6.81 |  68.04$\pm$7.65 |  68.86$\pm$6.33
XLM-RoBERTa |  97.05$\pm$0.62 |  92.08$\pm$1.74 |  91.60$\pm$1.23
ChatGPT-3.5 |  77.45$\pm$1.62 |  54.82$\pm$3.28 |  67.28$\pm$4.26
* ELMICT |  97.55$\pm$0.13 |  92.84$\pm$4.22 |  94.90$\pm$2.01
Table 3: Comparison with baselines. Results of binary classification. The best
performances are in bold. Training and developing data is code-mixing messages
in English-Finnish. 5-fold CV. $*$ denotes the proposed models.

### English-Finnish Code-Mixing Detection

Initially, we assess our model’s performance on English-Finnish code-mixed
messages. We employ 5-fold cross-validation to randomly split the data into
train-dev chunks in a 90-10 proportion. Table 3 illustrates the performance
evaluation of the ELMICT model compared to other schemes, addressing RQ1. The
results indicate that our proposed model consistently outperforms the
baselines across all metrics. Particularly noteworthy is the superior
performance of the ELMICT model compared to the fine-tuned XLM-RoBERTa model.
This underscores the significance of leveraging features generated by multiple
tokenizers and a language detector module for code-mixing text detection
tasks. Additionally, the performance of Random Forest and lingua models
demonstrates that utilizing features without soft labels generated by fine-
tuned pre-trained language models underperform. Moreover, the classification
results highlight the limitations of ChatGPT-3.5 in zero-shot learning
settings. While we don’t explore fine-tuning or prompt-engineering approaches,
it’s plausible that further enhancements could improve the performance of
LLMs. Lastly, the experiment reveals that Random Forest outperforms other
ensemble models like Adaptive Boosting and Gradient Boosting. We exclusively
utilize Random Forest for ensemble modeling in further experiments.

In addition to cross-validation, we test the higher performing models on a
test batch, which contains data from different threads and is excluded from
train and development batches used for models’ training and fine-tuning.
Additional experiments demonstrate the robustness of our model. The test batch
includes 297 texts (131 code-mixed and 166 non-code-mixed texts). Table 4
demonstrates comparable performance for the majority of schemes, and
significant improvement in the performance of lingua detector. Furthermore,
the experiment on test data batch helps to prove the usage of ELMICT for the
classification English-Finnish dataset to answer RQ3. While we expected a drop
in model performance because of possible overfitting, the performance is even
higher.

Model Scheme |  ACC |  F1 |  AUC
---|---|---|---
lingua |  81.82 |  81.64 |  81.55
XLM-RoBERTa |  93.94 |  93.93 |  93.93
ChatGPT-3.5 |  74.75 |  74.58 |  74.54
* ELMICT |  97.64 |  97.64 |  97.62
Table 4: Comparison with baselines. Results of binary classification. The best
performances are in bold. Testing data is code-mixing messages in English-
Finnish. $*$ denotes the proposed models.

### Cross-lingual Code-Mixing Detection

In addition to monolingual classification tasks, there are also cross-lingual
classification settings where the languages in the training and testing data
are different. To assess the proposed framework’s cross-lingual capability, we
utilize a zero-shot setting, where we train and validate classification
schemes on the data of English-Finnish dataset and test the model on the data
from the other dataset (English-Korean or English-Spanish). For test data
classification, we use the same models from the previous experiment. The
complete findings of the cross-lingual classification are outlined in Table 5.

Model Scheme |  ACC |  F1 |  AUC
---|---|---|---
English-Korean
lingua |  97.10 |  49.26 |  50.00
XLM-RoBERTa |  93.00 |  63.84 |  74.65
ChatGPT-3.5 |  92.40 |  57.58 |  63.14
* ELMICT |  96.70 |  66.80 |  64.85
English-Spanish
lingua |  87.80 |  46.75 |  50.00
XLM-RoBERTa |  88.60 |  76.88 |  81.16
ChatGPT-3.5 |  48.80 |  44.38 |  64.14
* ELMICT |  90.40 |  73.57 |  70.18
Table 5: Comparison with baselines. Results of cross-lingual binary
classification. The best performances are in bold. Testing data is code-mixing
messages in English-Korean and English-Spanish. $*$ denotes the proposed
models.

In comparison to the other schemes, ELMICT exhibits comparable performance
with the fine-tuned XLM-RoBERTa model. ELMICT demonstrates higher ACC for both
datasets, while because of strong imbalance in both datasets, the other two
metrics are more relevant. While for English-Korean messages ELMICT has higher
F1, for English-Spanish fine-tuned XLM-RoBERTa has higher F1. Moreover, XLM-
RoBERTa demonstrates higher AUC for both datasets. While at the same time, two
other schemes demonstrate random results with AUC equals 50%.

### Topic Analysis

Figure 3 presents the top-10 most popular topics with use of code-mixing in
threads with “Immigration” flair. The highest level of code-mixing usage in
migration-related posts was detected in the topic related to guns
(patruunatehdas – cartridge factory; tarkkuuskivääri – sniper rifle). The
second most topic with high code-mixing usage is about employment and bank
accounts in Finland (työ- ja elinkeinotoimisto (TE) – Employment and Economic
Development Office; pankki – bank) because they are widely used not only in
relation to financial services, but also for digital authentication in various
services. The third topic with a high level of code-mixed messages is about
the Russo-Ukrainian war and Finland’s membership in NATO (siviilipalvelus –
civil service; taisteli puolella – fought on the side). The other seven topics
could be divided into two groups. The first one includes everyday life
questions that could be addressed during migration: sauna (löyly – steam),
shopping (kierrätyskeskus – recycling center), apartment renting (asunto –
apartment), healthcare (hoito – care, therapy), and public utilities
(pörssisähkö – exchange electricity, also known as spot electricity).

The second group is about popular cultural media content: local music and
movie subtitles. These topics include many words and phrases in Finnish
related to song and movie titles and artists’ names. The latter group should
not be classified as code-mixing because all these words and phrases are
proper names. However, to avoid the classification of these messages as code-
mixing is a separate challenging NLP-task. It could be tackled by applying a
multilingual named entity recognition model in the future. This would have
required additional experiments, though, which were beyond the scope of this
paper.

Figure 3: Proportion of code-mixing messages per topic per flair in English-
Finnish dataset.

## Conclusions and Future Work

This paper explores code-mixing patterns in migration-related online
conversations. Our proposed Ensemble Learning for Multilingual Identification
of Code-mixed Text (ELMICT) method allows for detection of messages with code-
mixing in predominantly English-based datasets. The core idea of ELMICT is its
use in combination with multiple tokenizers outputs and soft labels generated
by a pre-trained language model. The utilization of context-based soft labels
allows us to predict code-mixing usage in migration-related contexts (everyday
life challenges and cultural nuances), while tokenizer’s outputs made models
more robust in the new linguistic context. Experiments on multiple English-
based datasets that included code-mixing with words from Finnish, Korean, or
Spanish show that the proposed model outperforms several baselines in the
classification task. Utilization of ELMICT allows us to analyse the usage of
code-mixing in migration-related threads on r/Finland subreddit. The results
of our analysis highlights a list of topics where code-mixing is highly
predictable (housing market, shopping, public utilities, and healthcare),
while also bringing to light particular topics (guns and hunting) and temporal
discourses (Russo-Ukrainian war and NATO membership of Finland) where code-
mixing was seen be be more widely used.

The ELMICT model holds promising potential for application in public services
that can utilize code-mixing in conversational agents and enhance trusting
relations with migrants by appropriate usage of specific vocabularies. This
proposed model could be a part of the pipeline of model training in the
Retrieval-Augmented Generation (RAG) module or database refinement. By
harnessing the capabilities of the ELMICT model, organizations can strengthen
their customer relationships by building trust based on communication and
potentially innovate new solutions, grounded in the vocabulary that unites
locals and migrants. The versatility and adaptability of ELMICT with the use
of different language-related tokenizers beyond its initial scope could be one
of the exploratory directions for future work.

There are certain limitations to our study that future work could address.
First, the dataset we used for experimentation only contains messages posted
during a limited time and contains information from only 3 subreddits. To
improve the model’s performance across various language pairs of code-mixing,
it would be valuable to extend this dataset to include data collected over a
longer period, more diverse topics, and additional languages. Second, the
topic analysis highlights the necessity of applying additional preprocessing
steps, such as named entities recognition for proper names related to popular
culture (titles, artists, etc.) Third, our proposed model only identifies text
contained in code-mixing, while for building conversational agents or any
other application of code-mixed vocabulary, it’s necessary to extract these
tokens. Usage of ELMICT will help to increase the efficiency of data
annotation for the token classification task because of the automated
filtering of monolingual texts.

### Reproducibility

Datasets and code for the experiments described in this paper will be
available for research purposes at the public repository
https://github.com/vitiugin/elmict.

## Broader Impact and Ethics Statement

For multilingual speakers, code-mixing is a communication method typically
used when they are in a relaxed state and with people with whom they share
close relationships. This linguistic strategy is employed specifically when
the multilingual speaker has a trustful and intimate relationship with another
person who shares similar linguistic and cultural backgrounds (Wulandari 2021;
Kusumawati and Prihadi 2023). In this context, to effectively build a trusting
relationship between multilingual migrants, counselors, and conversational
agents, incorporating the feature of code-mixing into the system is necessary.
This adaptation would help migrants perceive that public services using human
and conversational agents share similar linguistic and cultural backgrounds,
fostering a sense of trust. However, we must ensure that such perceptions of
trust induced by conversational agents using code-mixing in conversations do
not make users believe such systems to be infallible or anthropomorphised;
hence designing such systems to incorporate explainable outputs and accurate
content is crucial.

Furthermore, previous research has shown that multilingual users experience
similar feelings of pressure when conversing with monolingual conversational
agents as when conversing with strangers (Choi, Lee, and Lee 2023). This has
led to the recognition that code-mixing conversational agents can provide
multilingual users with a feeling of inclusion and acceptance in society. In
recognizing the deep connection between social integration and trust formation
during the migration process (Dinesen and Hooghe 2010), it becomes evident
that there is a significant opportunity for conversational agents to aid in
this process. By providing a window of opportunity for migrants to build trust
with public services, conversational agents can play a role in supporting
migrants (and their human counselors) as they adapt to and integrate into
their new environment. Aligning with this, it is of utmost importance to build
relationships between migrants, human counselors, and conversational agents as
part of a system of public services that can promote social integration and
acceptance while migrants adapt to their new environment. As a result, these
code-mixed digital offerings can be leveraged well to support the social
integration of migrants, providing a pivotal step in promoting more inclusive
public sector services.

We recognize that there are many ethical implications of this work related to
discrimination, misuse and privacy of end-users. Furthermore, we assume that
language identity of users such as code-mixing level could be used for
profiling, may result in discriminatory practices. Since we demonstrate the
potential to identify such attributes on social media, we are aware of how our
research could be misused and abused, discriminating migrants (Yim and Clément
2019; Faingold 2022; Ekwere 2022).

To protect privacy, we refrain from disclosing sensitive personal information.
Given Reddit’s anonymous nature and the absence of mandatory personal data
sharing, we commit to not sharing collected data that could be used to
identify individuals. For reproducibility, we only provide comment IDs and
code-mixing binary labels, keeping users’ right to delete their data in the
future if they choose.

We also strongly encourage future studies to consider the ethical dimensions
of detecting language-related characteristics in social media texts, from
study inception to final research dissemination.

## Acknowledgments

This work is supported by the Trust-M research project, a partnership between
Aalto University, University of Helsinki, Tampere University, and the City of
Espoo, funded in-part by a grant from the Strategic Research Council (SRC) in
Finland. The authors also express their deep gratitude to CRAI-CIS research
group members at Aalto University who helped in experimental setup and
provided valuable insights.

## References

  * Abimannan et al. (2023) Abimannan, S.; El-Alfy, E.-S. M.; Chang, Y.-S.; Hussain, S.; Shukla, S.; and Satheesh, D. 2023.  Ensemble multifeatured deep learning models and applications: A survey.  _IEEE Access_.
  * Aguilar and Solorio (2019) Aguilar, G.; and Solorio, T. 2019.  From English to code-switching: Transfer learning with strong morphological clues.  _arXiv preprint arXiv:1909.05158_.
  * Auer (2013) Auer, P. 2013.  _Code-switching in conversation: Language, interaction and identity_.  Routledge.
  * Barman, Wagner, and Foster (2016) Barman, U.; Wagner, J.; and Foster, J. 2016.  Part-of-speech tagging of code-mixed social media content: Pipeline, stacking and joint modelling.  In _Proceedings of the second workshop on computational approaches to code switching_ , 30–39.
  * Biemann et al. (2004) Biemann, C.; Bordag, S.; Heyer, G.; Quasthoff, U.; and Wolff, C. 2004.  Language-independent methods for compiling monolingual lexical data.  In _Computational Linguistics and Intelligent Text Processing: 5th International Conference, CICLing 2004 Seoul, Korea, February 15-21, 2004 Proceedings 5_ , 217–228. Springer.
  * Chang et al. (2023) Chang, Y.; Wang, X.; Wang, J.; Wu, Y.; Yang, L.; Zhu, K.; Chen, H.; Yi, X.; Wang, C.; Wang, Y.; et al. 2023.  A survey on evaluation of large language models.  _ACM Transactions on Intelligent Systems and Technology_.
  * Choi, Lee, and Lee (2023) Choi, Y. J.; Lee, M.; and Lee, S. 2023.  Toward a Multilingual Conversational Agent: Challenges and Expectations of Code-mixing Multilingual Users.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , 1–17.
  * Conneau et al. (2019) Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Unsupervised cross-lingual representation learning at scale.  _arXiv preprint arXiv:1911.02116_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dinesen and Hooghe (2010) Dinesen, P. T.; and Hooghe, M. 2010.  When in Rome, do as the Romans do: The acculturation of generalized trust among immigrants in Western Europe.  _International Migration Review_ , 44(3): 697–727.
  * Dowlagar and Mamidi (2021) Dowlagar, S.; and Mamidi, R. 2021.  A pre-trained transformer and CNN model with joint language ID and part-of-speech tagging for code-mixed social-media text.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_ , 367–374.
  * Drobotowicz et al. (2023) Drobotowicz, K.; Truong, N. L.; Ylipulli, J.; Gonzalez Torres, A. P.; and Sawhney, N. 2023.  Practitioners’ Perspectives on Inclusion and Civic Empowerment in Finnish Public Sector AI.  In _Proceedings of the 11th International Conference on Communities and Technologies_ , 108–118.
  * Ekwere (2022) Ekwere, E. 2022.  Language identity and discrimination in a multicultural society.  _European Journal of Linguistics_ , 1(2): 1–12.
  * Faingold (2022) Faingold, E. D. 2022.  Language Rights for Minorities and the Right to Code-Switch in the United States Workplace.  _A Critical Examination of Language and Community_ , 1.
  * Feng, Li, and Koehn (2022) Feng, Y.; Li, F.; and Koehn, P. 2022.  Toward the limitation of code-switching in cross-lingual transfer.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 5966–5971.
  * Gardner-Chloros (2020) Gardner-Chloros, P. 2020.  Contact and code-switching.  _The handbook of language contact_ , 181–199.
  * Gautam et al. (2021) Gautam, D.; Kodali, P.; Gupta, K.; Goel, A.; Shrivastava, M.; and Kumaraguru, P. 2021.  Comet: Towards code-mixed translation using parallel monolingual sentences.  In _Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching_ , 47–55.
  * Goswami et al. (2020) Goswami, K.; Sarkar, R.; Chakravarthi, B. R.; Fransen, T.; and McCrae, J. P. 2020.  Unsupervised deep language and dialect identification for short texts.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , 1606–1617.
  * Grootendorst (2022) Grootendorst, M. 2022.  BERTopic: Neural topic modeling with a class-based TF-IDF procedure.  _arXiv preprint arXiv:2203.05794_.
  * Gumperz (1982) Gumperz, J. J. 1982.  _Discourse strategies_.  1\. Cambridge University Press.
  * Harwood et al. (2019) Harwood, A.; Karunasekera, S.; Vanni, M.; Falzon, L.; Padia, P.; and Silva, A. 2019.  Understanding multilingual communities through analysis of code-switching behaviors in social media discussions.  In _2019 IEEE International Conference on Big Data (Big Data)_ , 2274–2283. IEEE.
  * Ikeh (2023) Ikeh, M. C. 2023.  Social Media as Tools for Social and Cultural Integration of Nigerian Migrants in Sweden.
  * Intke-Hernández et al. (2015) Intke-Hernández, M.; et al. 2015.  Migrant stay-at-home mothers learning to eat and live the Finnish way.  _Nordic Journal of Migration Research_ , 5(2): 75–82.
  * Krishnan et al. (2021) Krishnan, J.; Anastasopoulos, A.; Purohit, H.; and Rangwala, H. 2021.  Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Filling.  In _Proceedings of the 1st Workshop on Multilingual Representation Learning_ , 211–223.
  * Kusumawati and Prihadi (2023) Kusumawati, N.; and Prihadi, P. 2023.  Conversation Implicature and Code-Mixing in Mata Najwa’s Talk Show Exclusive Episode: Ganjar Pranowo and the World Cup.  _International Journal of Multicultural and Multireligious Understanding_ , 10(5): 419–430.
  * Lu et al. (2023) Lu, S.-E.; Lu, B.-H.; Lu, C.-Y.; and Tsai, R. T.-H. 2023.  Exploring methods for building dialects-Mandarin code-mixing corpora: A case study in Taiwanese Hokkien.  _arXiv preprint arXiv:2301.08937_.
  * Luria et al. (2019) Luria, M.; Reig, S.; Tan, X. Z.; Steinfeld, A.; Forlizzi, J.; and Zimmerman, J. 2019.  Re-Embodiment and Co-Embodiment: Exploration of social presence for robots and conversational agents.  In _Proceedings of the 2019 on Designing Interactive Systems Conference_ , 633–644.
  * Mabule (2015) Mabule, D. R. 2015.  What is this? Is it code switching, code mixing or language alternating.  _Journal of Educational and Social Research_ , 5(1): 339–350.
  * Madianou (2015) Madianou, M. 2015.  Digital inequality and second-order disasters: Social media in the Typhoon Haiyan recovery.  _Social Media+ Society_ , 1(2): 2056305115603386.
  * McAuliffe et al. (2022) McAuliffe, M.; et al. 2022.  World Migration Report 2022.
  * McEntee-Atalianis, Ateek, and Gardner-Chloros (2023) McEntee-Atalianis, L.; Ateek, M.; and Gardner-Chloros, P. 2023.  Multilingual repertoires and identity in social media: Syrian refugees on Facebook.  _International Journal of Bilingualism_ , 27(5): 731–748.
  * McInnes, Healy, and Astels (2017) McInnes, L.; Healy, J.; and Astels, S. 2017.  hdbscan: Hierarchical density based clustering.  _The Journal of Open Source Software_ , 2(11): 205.
  * McInnes et al. (2018) McInnes, L.; Healy, J.; Saul, N.; and Großberger, L. 2018.  UMAP: Uniform Manifold Approximation and Projection.  _Journal of Open Source Software_ , 3(29).
  * Milroy and Muysken (1995) Milroy, L.; and Muysken, P. 1995.  _One speaker, two languages: Cross-disciplinary perspectives on code-switching_ , volume 10.  Cambridge University Press.
  * Myers-Scotton (1993) Myers-Scotton, C. 1993.  _Social motivations for codeswitching: Evidence from Africa_.  Oxford University Press.
  * Ostrowski et al. (2021) Ostrowski, A. K.; Zygouras, V.; Park, H. W.; and Breazeal, C. 2021.  Small group interactions with voice-user interfaces: exploring social embodiment, rapport, and engagement.  In _Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_ , 322–331.
  * Ponnusamy et al. (2022) Ponnusamy, P.; Ghias, A.; Yi, Y.; Yao, B.; Guo, C.; and Sarikaya, R. 2022.  Feedback-based self-learning in large-scale conversational ai agents.  _AI magazine_ , 42(4): 43–56.
  * Poplack et al. (2000) Poplack, S.; et al. 2000.  _The English History of African American English_.  Blackwell Oxford.
  * Rani, McCrae, and Fransen (2022) Rani, P.; McCrae, J. P.; and Fransen, T. 2022.  MHE: Code-Mixed Corpora for Similar Language Identification.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , 3425–3433.
  * Schaefer et al. (2016) Schaefer, K. E.; Chen, J. Y.; Szalma, J. L.; and Hancock, P. A. 2016.  A meta-analysis of factors influencing the development of trust in automation: Implications for understanding autonomy in future systems.  _Human factors_ , 58(3): 377–400.
  * St John (2023) St John, O. 2023.  Social Inclusion Through Multilingual Assistants in Additional Language Learning.  _Social Inclusion_ , 11(4): 145–155.
  * Toribio (2001) Toribio, A. J. 2001.  On the emergence of bilingual code-switching competence.  _Bilingualism: language and cognition_ , 4(3): 203–231.
  * Wardhaugh and Fuller (2021) Wardhaugh, R.; and Fuller, J. M. 2021.  _An introduction to sociolinguistics_.  John Wiley & Sons.
  * Weinreich (1953) Weinreich, U. 1953.  Language in contact. the Hague, the Netherlands: Mouton.
  * Wulandari (2021) Wulandari, A. 2021.  Code Switching and Code Mixing Study in â€ œHitam Putihâ€ Talk Show Program.  _Vivid: Journal of Language and Literature_ , 10(1): 1–5.
  * Yap, Lee, and Roto (2021) Yap, C. E. L.; Lee, J.-J.; and Roto, V. 2021.  How HCI interprets service design: a systematic literature review.  In _Human-Computer Interaction–INTERACT 2021: 18th IFIP TC 13 International Conference, Bari, Italy, August 30–September 3, 2021, Proceedings, Part II 18_ , 259–280. Springer.
  * Yeasmin et al. (2020) Yeasmin, N.; et al. 2020.  _Immigration in the circumpolar north: Integration and resilience_.  Routledge.
  * Yim and Clément (2019) Yim, O.; and Clément, R. 2019.  “You’re a Juksing”: Examining Cantonese–English Code-Switching as an Index of Identity.  _Journal of Language and Social Psychology_ , 38(4): 479–495.
  * Zahera et al. (2023) Zahera, H. M.; Vitiugin, F.; Sherif, M. A.; Castillo, C.; and Ngonga Ngomo, A.-C. 2023.  Using Pre-Trained Language Models for Abstractive DBPEDIA Summarization: A Comparative Study.  In _Knowledge Graphs: Semantics, Machine Learning, and Languages_ , 19–37. IOS Press.
  * Zhang et al. (2023) Zhang, R.; et al. 2023.  Multilingual Large Language Models Are Not (Yet) Code-Switchers.  _arXiv preprint arXiv:2305.14235_.


Recently, various studies have been directed towards exploring dense passage retrieval techniques employing pre-trained language models, among which the masked auto-encoder (MAE) pre-training architecture has emerged as the most promising. The conventional MAE framework relies on leveraging the passage reconstruction of decoder to bolster the text representation ability of encoder, thereby enhancing the performance of resulting dense retrieval systems. Within the context of building the representation ability of the encoder through passage reconstruction of decoder, it is reasonable to postulate that a “more demanding” decoder will necessitate a corresponding increase in the encoder's ability. To this end, we propose a novel token importance aware masking strategy based on pointwise mutual information to intensify the challenge of the decoder. Importantly, our approach can be implemented in an unsupervised manner, without adding additional expenses to the pre-training phase. Our experiments verify that the proposed method is both effective and robust on large-scale supervised passage retrieval datasets and out-of-domain zero-shot retrieval benchmarks.

§ INTRODUCTION

Passage retrieval is a core sub-task in various downstream applications, such as open-domain question answering <cit.>, conversational systems <cit.> and web search <cit.>. Recently, a number of studies have demonstrated that dense passage retrieval systems based on pre-trained language models (PLMs) are significantly more effective compared to traditional sparse retrieval methods such as BM25 <cit.>. To balance efficiency and effectiveness, existing dense passage retrieval methods usually leverage a dual-encoder architecture, where query and passage are encoded into continuous vector representations by PLMs respectively, and then a lightweight score function such as dot product or cosine similarity between two vectors is used to estimate the semantic similarity between the query-passage pair.[We use the term “passage” and “document” interchangeably throughout this paper.]

Illustration of the Masked Auto-Encoder (MAE) pre-training framework.

In the dual-encoder architecture, the text representation capability of the PLMs plays a crucial role as it shall encode all essential information into the low-dimensional dense vector. However, it has been observed that the progress of PLMs in general language understanding benchmarks does not necessarily lead to an improvement in text representation ability <cit.> as the widely used masked language modeling (MLM) pre-training objective focuses more on representing individual tokens rather than the entire sentence. As a result, numerous recent studies have explored to enhance the base model's sentence representation ability via incorporating supplementary pre-training tasks or designing new pre-training architectures <cit.>.

Currently, the Masked Auto-Encoder (MAE) is arguably the most effective pre-training framework in retrieval tasks. As illustrated in Figure <ref>, MAE utilizes the encoder-decoder architecture in which the sentence is randomly masked twice as the input to the encoder and decoder, respectively, and the sentence embedding pooled from the encoder is concatenated with the masked input of the decoder to reconstruct the original input. In this framework, the encoding quality is critical to the success of the reconstruction task of decoder, which in turn, is the key to improving the representation ability of the encoder. To increase the difficulty of the decoder, previous works usually utilized a shallower decoder and a higher decoder mask ratio <cit.>. We argue that optimizing the training task of MLM can further enhance the challenge of the decoder, leading to an even greater improvement in the representation ability of the encoder.

Top 10 masked term frequency distribution using random masking with uniform sampling. Statistics are measured on the MS MARCO passage corpus.

During the MAE pre-training process, the MLM training objective is used at both the encoder and decoder ends, with masked tokens randomly sampled from a uniform distribution over the input sentence. However, such a random sampling strategy is sub-optimal since it is biased towards sampling uninformative high-frequency tokens (e.g., stop-words and punctuations as shown in Figure <ref>).[Uniform sampling from each sentence is statistically equivalent to sampling from the term frequency distribution over the corpus' vocabulary, which is a power-law distribution.]
Previously, many research works have attempted to address this issue by detecting and masking spans <cit.> or determining which tokens should be selected through the PLMs <cit.>.
In the information retrieval (IR) community, early lexical retrieval systems <cit.> also used inverse document frequency for term re-weighting in their score functions.
Some recent works have also suggested that incorporating the term importance information and token-level matching signal with deep contextualized representations <cit.> is beneficial.

In this paper, we follow similar principles to optimize the MAE pre-training framework for dense retrieval.
Specifically, we aim to increase the proportion of masked tokens that are informative (or meaningful for passage retrieval tasks) during MLM training of the decoder.
To achieve this goal, we introduce pointwise mutual information (PMI) between $n$-grams to estimate the importance of each token, which is subsequently used to regularize the sampling distribution of MLM, increasing the probability of important tokens being masked. More importantly, PMI can be calculated in an unsupervised manner and can model the corpus-wise keywords and co-occurrence features which are valuable for IR tasks.

Compared to previous research which relies on another PLM to determine the importance of tokens <cit.> or sample replacements <cit.>, our method is more computationally efficient because it only requires computing statistical information from the corpus.
We apply this masking strategy asymmetrically to the encoder and decoder sides, enforcing the dense bottleneck between them to encode more information about salient co-occurring tokens captured by decoder masking.
We evaluate the effectiveness of our approach on several supervised passage retrieval datasets and zero-shot retrieval benchmarks.

To summarize, our contributions are as follows:

* We reveal the problem of uniform sampling in random masking, which biases the learning process of language models toward high-frequency tokens that are uninformative in passage retrieval scenarios.

* To mitigate the above issue, we introduce an efficient approach for term importance estimation based on pointwise mutual information.
We further propose a novel importance-aware masking strategy and integrate this masking strategy with the bottlenecked masked autoencoder framework for better sentence representation learning.

* Experiments results demonstrate the effectiveness of our approach. Our model outperforms state-of-the-art dense retrievers in both supervised settings and zero-shot settings.

§ RELATED WORK

Pre-training for Dense Retrieval
Dense retrieval usually involves learning a dual-encoder that encodes query and document into dense vectors and takes their inner product (or cosine similarity) as their relevance score.
It's usually trained with the contrastive objective to distinguish positive pairs from negative ones <cit.>.
To reduce the gap between MLM and downstream retrieval tasks, plenty of works have designed various self-supervised contrastive pre-training tasks by automatically constructing positive and negative pairs <cit.>.

Different from the idea of designing pretext contrastive tasks, another line of research has focused on improving the global text representation learning ability in the masked auto-encoding framework.
For example, <cit.> proposed adding a shallow auto-regressive decoder on top of the encoder for causal language modeling conditioned on the sentence representation.
<cit.> changed the decoder to a bi-directional Transformer head trained with MLM task.
<cit.> proposed an enhanced-decoding mechanism with token-specific masking design and an aggressive masking ratio.
<cit.> introduced ELECTRA-style replaced language modeling to reduce the gap between pre-training and fine-tuning.
In addition to predicting the sentence itself based on its contextualized representation, <cit.> proposed to additionally predict its surrounding context.
<cit.> further used docT5query <cit.> and GPT-2 <cit.> for data augmentation and combined all these tasks with multiple decoders.
It should be noticed that our contributions are orthogonal to theirs since we only use the self-prediction task and the vanilla MAE architecture.

Pointwise Mutual Information
Using PMI to model word association in NLP was first introduced by  <cit.>.
<cit.> found that the skip-gram algorithm is implicitly factorizing the (shifted) word-context PMI matrix for word embedding learning.
More recently,  <cit.> extended PMI to detect more correlated spans for masking.
<cit.> identified more informative masking patterns by finding a maximum cut in the PMI matrix.
Different from previous work, we use the PMI to estimate how much a token contributes to the sentence and use the term importance information to regularize the sampling distribution.

§ METHODOLOGY

In this section, we describe our proposed pre-training method for the dense passage retrieval task. We first give a brief overview of the conventional MAE pre-training architecture. Then we introduce how to extend it to our model transforming the decoder to be more challenging. Finally, we detail the multi-stage fine-tuning and inference designs.

§.§ Bottlenecked Masked Autoencoders

In the bottlenecked masked autoencoder (BMAE) architecture, a deep encoder is used to get a compressed vector as sentence embedding, and a shallow decoder is used to recover masked tokens based on the sentence embedding offered by the encoder.
An information bottleneck (e.g., the representation at  position) is imposed as the communication budge between the encoder and decoder.
To make good predictions, the encoder must compress as much information to the bottleneck, whereas the decoder has to attend to the bottleneck representation due to its weakened capacity.

Formally, given a piece of text $\mathbf{x}$, two masking operations are independently applied to the input sequence $\mathbf{x}$ with different masking ratios $p_\text{enc}$ and $p_\text{dec}$. Among the masked tokens, $80\%$ are replaced with a special  token, $10\%$ are replaced by a random token in the vocabulary, and the remaining tokens are kept unchanged <cit.>,

\begin{equation}
\begin{aligned}
\hat{\mathbf{x}}_\text{enc} = \text{Mask}(\mathbf{x}, p_\text{enc}), \\
\hat{\mathbf{x}}_\text{dec} = \text{Mask}(\mathbf{x}, p_\text{dec}).
\end{aligned}
\end{equation}

Deep Encoder
We use a multi-layer Transformer network <cit.> as the deep encoder which can be initialized from PLMs such as BERT <cit.>.
We feed $\hat{\mathbf{x}}_\text{enc}$ through the deep language model encoder to get its contextualized representations $\mathbf{h}_{\hat{\mathbf{x}}_\text{enc}}$ for masked token reconstruction,

\begin{equation}
\mathbf{h}_{\hat{\mathbf{x}}_\text{enc}} = \text{Encoder}(\hat{\mathbf{x}}_\text{enc})
\end{equation}

The encoder is trained via the MLM objective where the language modeling head predicts masked tokens based on contextualized representations,

\begin{equation}
L_\text{enc} = - \sum_{x\in M_\text{enc}} \text{log} P_\text{lm}(x|\mathbf{h}_{\hat{\mathbf{x}}_\text{enc}})
\end{equation}

where $M_\text{enc}$ denotes the set of masked tokens at the encoder side.

Bottleneck Representation
To ensure consistency with downstream fine-tuning, we use the  token representation $\mathbf{h}_c \in \mathbb{R}^{d_\text{enc}}$ at the last layer of the deep encoder as the text representation.
We use a linear projection head to construct the bottleneck for decoder recovery,

\begin{equation}
\mathbf{h}_b = \mathbf{W} \mathbf{h}_c + \mathbf{b}
\end{equation}

where $\mathbf{W}\in \mathbb{R}^{d_\text{dec} \times d_\text{enc}} $ and $\mathbf{b} \in \mathbb{R}^{d_\text{dec}}$ are trainable parameters.

Weak Decoder
The decoder is a shallow neural network consisting of two randomly initialized Transformer layers.
Its objective is to recover the aggressively masked inputs conditioned on the bottleneck representation $\mathbf{h}_b\in \mathbb{R}^{d_\text{dec}}$ provided by the deep encoder,

\begin{equation}
\mathbf{h}_{\hat{\mathbf{x}}_\text{dec}} = \text{Decoder}(\mathbf{h}_b,\hat{\mathbf{x}}_\text{dec}).
\end{equation}

We use the same language model head for masked language modeling,

\begin{equation}
L_\text{dec} = - \sum_{x\in M_\text{dec}} \text{log} P_\text{lm}(x|\mathbf{h}_{\hat{\mathbf{x}}_\text{dec}})
\end{equation}

where $M_\text{dec}$ denotes the set of masked tokens at the decoder side.

§.§ Token Importance Aware Masking

Sampling distribution comparison.

Model architecture.

Overview of our pre-training approach. We first estimate term importance in an unsupervised manner for a piece of text: . Then we incorporate the term importance awareness into decoder masking to mask more salient co-occurring tokens.

The random masking strategy treats all tokens equally. Statistical analysis reveals that $40\%$ of the masked tokens produced by the $15\%$ ratio random masking method in BERT pre-training are stop-words or punctuation. However, prior research has shown that the contribution of these tokens to the dense passage retrieval task is limited. Furthermore, these frequently occurring tokens serve to ease the burden on the decoder in the MAE pre-training process. To address the limitation above, we propose a new masking strategy aiming to mask tokens that are more important thus posing a greater challenge for the decoder in the reconstruction task.

Token Importance Estimation
Mutual Information is widely used to estimate the correlation of two random variables. Inspired by previous work, we consider linking the importance of tokens to statistical pointwise mutual information. The pointwise mutual information between two words $w_1$ and $w_2$ represents the correlation between them. Mathematically, the pointwise mutual information of the bi-gram combination $w_1w_2$ can be formulated as:

\begin{equation}
\text{PMI}(w_1, w_2) = log\frac{p(w_1,w_2)}{p(w_1)p(w_2)}
\end{equation}

where the probability of any $n$-gram is defined as the number of its occurrences in the corpus divided by the number of all the n-grams in the corpus.
For each word in a sentence, if the word is more informative (namely more important), then statistically speaking, the PMI between this word and the remaining parts of the sentence should be larger. In practice, calculating the PMI between each word and the remaining parts of the sentence can be computationally expensive and may result in bias. As such, a more suitable approach is to determine the importance of a word by analyzing its PMI with adjacent fragments.

To simplify the calculation without sacrificing performance, we only need to compute the average mutual information (AMI) of fixed window size. In the determined window size, we use the average PMI value of all possible $n$-gram combinations delimited by the token (starting and ending with the token, and with a minimum $n$ value of $2$) as the ultimate reference for token importance estimation. For each word $w_i$ in sentence $\mathcal{S}$, if the window size is set to $L$, the final average mutual information of word $w_i$ can be represented as:

\begin{equation}
\begin{aligned}\label{eq:ami}
\text{AMI}(w_i) & = \frac{1}{L-1}\sum_{j=1}^{L-1}\text{PMI}(w_{i-j},...,w_i) \\
& + \frac{1}{L-1}\sum_{j=1}^{L-1}\text{PMI}(w_i,...,w_{i+j}).
\end{aligned}
\end{equation}

To clarify, we calculate the pointwise mutual information of bigrams when $L=2$. When $L \geq 2$, we will expand to the computation of all $n$-grams where $2\leq n \leq L$. In order to balance computational efficiency and accuracy, we set $L=4$ in this work.

Importance Aware Masking.
Input: $\mathbf{x}=(x_1,\dots,x_n)$, $\mathbf{t}=(t_1,\dots,t_n)$, $p$;

Output: $\mathbf{M}=(m_1,\dots,m_n)$;
[1]
$\mathbf{M} \gets []$
$i \textbf{ in } 1,\dots,n$
$t \sim \mathcal{N}(\mathbf{t}[i], \sigma)$
$\mathbf{t}[i] \gets t$
$k = \lfloor \Call{len}{\mathbf{x}} \times p \rfloor$
$\mathbf{I} = \Call{argsort}{\mathbf{t}}$
$i \textbf{ in } 1,\dots,n$
$i \textbf{ in } \Call{topk}{\mathbf{I},k}$
$\Call{append}{\mathbf{M}, 1}$
$\Call{append}{\mathbf{M}, 0}$

Importance Aware Sampling

Given a sentence $\mathbf{x} = (x_1, x_2, ..., x_n)$ that consists of $n$ tokens equipped with the estimated token importance $\mathbf{t} = (t_1, t_2, ..., t_n)$ where $t_i=\text{AMI}(x_i)$.
Instead of random sampling from a uniform distribution over the sequence of tokens, we propose a novel sampling strategy that takes the term importance into account.
Specifically, we sort tokens based on their perturbed importance and mask tokens with higher importance.
Each token importance is perturbed with a Gaussian noise before being sorted to preserve some randomness during masking.[We simply set the variance $\sigma=1$.]
Details can be found in Algorithm <ref>.
The output $\mathbf{M}=(m_1,\dots,m_n)$ is a binary-valued array in which 0 denotes unmasked and 1 for masked.
We apply the proposed masking strategy asymmetrically to the encoder and decoder sides.
The encoder side masking is unchanged while for the decoder, we use the importance aware masking to mask more salient tokens.
Figure <ref> compares the sampling distribution induced by the proposed importance-aware masking algorithm and the uniform sampling distribution of random masking.
More informative words have a higher probability of being masked.

§.§ Pre-training

The pre-training is conducted on the unlabeled corpus $\mathcal{C}=\{d_1, d_2, ..., d_N\}$.
The masked auto-encoder is pre-trained with the masked language modeling (MLM) objective from both the encoder and decoder,

\begin{equation}
L = L_{enc} + L_{dec},
\end{equation}

where each loss is normalized to stabilize training.

§.§ Fine-tuning

Given a labeled dataset $\mathcal{D}=\{(q_1,d^+_1), (q_2,d^+_2), ..., (q_n,d^+_n)\}$ consisting of query and supporting document pairs, we adopt the fine-tuning pipeline from prior work <cit.>. The fine-tuning consists of three stages: the first retriever is trained with the official BM25 hard negatives; the second retriever is trained with the hard negatives mined using the first retriever; the final stage retriever is trained with the hard negatives mined via the second retriever and knowledge distillation from a cross-encoder reranker.
At each of the three training stages, retrievers are initialized with the same pre-trained checkpoint.

Stage 1: BM25 Negatives.
For each query $q$,  we sample hard negatives $\mathbb{D}^-_\text{BM25}$ from top $K_1$ ranked documents excluding the positive ones returned by BM25.
In-batch negatives are also included to improve training efficiency.
The retriever is trained as a dual encoder with a contrastive loss,

\begin{equation} \label{equ:infonce}
L_{\text{cont}} = -\log \frac{\text{exp}(s(q, d^+))}{\sum_{d\in \{d^+\}\cup \mathbb{D}^-_\text{BM25}}\text{exp}(s(q, d))}
\end{equation}

where $s(q, d)$ measures the similarity score between query $q$ and document $d$.
In this paper, we use the dot product between query and document representations as their similarity score $s_\text{de}(q,d) = \mathbf{q} \cdot \mathbf{d}$, where $\mathbf{q} = \text{DE}(q)$ and $\mathbf{d} = \text{DE}(d)$.

Stage 2: Hard Negatives.
In the second stage, we substitute the BM25 lexical retriever with the dense retriever trained in the first stage for mining more challenging hard negatives <cit.>.
Top $K_2$ documents excluding positives are collected to construct $\mathbb{D}^-_\text{hard}$.
The retriever is trained with the same objective as Equation <ref> by substituting $\mathbb{D}^-_\text{BM25}$ with $\mathbb{D}^-_\text{hard}$.

Stage 3: Reranker-Distilled.
Hard negatives mined by retrievers suffer from the problem of false negatives due to the incomplete data annotation, which can be relieved by pseudo-labeling from a more powerful yet costly cross-encoder reranker <cit.>.
The cross-encoder takes the concatenation of query and document as input and models their relevance through multi-layer token-level interactions, making it a better architecture for fine-grained relevance estimation.
It is trained via the list-wise contrastive loss similar to the dual-encoder retriever except that the score function becomes a parametric form $s_\text{ce}(q,d)=\text{CE}(\text{concat}(q,d))$ and in-batch negatives are not involved in loss computation.
The hard negatives $\mathbb{D}^-_\text{hard$^2$}$ are mined from the top $K_3$ documents returned by the previous stage retriever.

We use the KL (Kullback-Leibler) divergence between the relevance distribution estimated by the dual-encoder retriever and the cross-encoder reranker for knowledge distillation:

\begin{equation}
\begin{aligned}
L_\text{kl} = \text{KL}(P_\text{ce}(D|q)||P_\text{de}(D|q)), \\
= \sum_{d\in D} p_\text{ce}(d|q) \log \frac{p_\text{ce}(d|q)}{p_\text{de}(d|q)}.
\end{aligned}
\end{equation}

where $p(d|q) \propto \text{exp}(s(q,d))$, $D$ consists of the positive document $d^+$ and a set of negative documents sampled from $\mathbb{D}^-_\text{hard$^2$}$.
The original contrastive loss is also added following <cit.>.
The final objective during knowledge distillation stage is $L = L_\text{kl} + \alpha L_\text{cont}$.

§.§ Inference

The inference for a dense retriever is conducted in two stages: index and retrieval.
The document encoder first encodes each document in the corpus into a fix-dimensional dense vector over which a dense index is built.
Given a query $q$, query encoder is employed to get the query representation, and retrieval is done by performing a maximum inner product search over the dense index to find relevant documents from the corpus.

§ EXPERIMENTS

§.§ Setup

The pre-training consists of two stages: generic pre-training followed by unsupervised corpus-aware pre-training.
Due to resource limitation, we omit the pre-training on general corpus by initializing from retromae checkpoint pre-trained on the combination of Wikipedia and BookCorpus and continually pre-train it on the MS MARCO passage corpus consisting of 8.8M passages.
We set encoder mask ratio $p_\text{enc}=0.3$ and decoder mask ratio $p_\text{dec}=0.5$.
After pre-training, the decoder is discarded and only the encoder is used for retriever initialization.
During fine-tuning, we use a siamese architecture that shares the parameters of query encoder and document encoder.
We set hard negative mining depths $K_1=1000$, $K_2=K_3=200$, and contrastive loss ratio $\alpha=0.2$ following <cit.>.
Other hyperparameters are adopted from prior work and detailed in Appendix <ref>.
During inference, we use the  library <cit.> to build the index and perform the exact search.

We evaluate our approach in two settings: supervised in-domain passage retrieval and zero-shot out-of-domain retrieval.
For supervised in-domain evaluation, we choose MS MARCO passage retrieval task <cit.>, TREC 2019 and 2020 Deep Learning Tracks <cit.>.
For zero-shot evaluation, we report results on 14 open available datasets of BEIR benchmark <cit.>.

We mainly compare our approach to other pre-training approaches tailored for dense retrieval, including Condenser <cit.>, coCondenser <cit.>, RetroMAE <cit.>, SimLM <cit.>, CoT-MAE <cit.>, MASTER <cit.>. We also list the performance of other retrieval systems for reference.

§.§ Main Results

Method     BM25     DPR     ColBERT     Contriever     Condenser     RetroMAE     SimLM     MASTER

TREC-COVID     65.6     33.2     67.7     59.6     75.0     77.2     63.7     62.0     79.0

NFCorpus     32.5     18.9     30.5     32.8     27.7     30.8     32.3     33.0     33.4

NQ     32.9     47.4     52.4     49.8     48.6     51.8     47.7     51.6     55.9

HotpotQA     60.3     39.1     59.3     63.8     53.8     63.5     58.1     58.9     65.5

FiQA-2018     23.6     11.2     31.7     32.9     25.9     31.6     29.2     32.8     33.6

ArguAna     31.5     17.5     23.3     44.6     29.8     43.3     42.1     39.5     49.1

Touche-2020     36.7     13.1     20.2     23.0     24.8     23.7     29.2     32.0     21.1

CQADupStack     29.9     15.3     35.0     34.5     34.7     31.7     33.2     32.7     36.1

Quora     78.9     24.8     85.4     86.5     85.3     84.7     77.3     79.1     86.0

DBPedia     31.3     26.3     39.2     41.3     33.9     39.0     34.5     39.9     42.0

SCIDOCS     15.8     7.7     14.5     16.5     13.3     15.0     14.5     14.1     17.0

FEVER     75.3     56.2     77.1     75.8     69.1     77.4     65.7     69.2     76.1

Climate-FEVER     21.3     14.8     18.4     23.7     21.1     23.2     16.3     21.5     23.6

SciFact     66.5     31.8     67.1     67.7     59.3     65.3     58.8     63.7     67.1

Best on     1     0     0     3     0     1     0     0     9

Average     43.0     25.5     44.4     46.6     43.0     47.0     43.0     45.0     49.0

Zero-shot transfer performance (nDCG@10) on BEIR benchmark. The best score on a given dataset is marked in bold, and the second best is underlined.
Baseline results are taken from the corresponding papers.

In-domain Evaluation.

Results on MS MARCO passage retrieval dataset and TREC DL benchmarks are shown in Table <ref>.
In line with previous study, we find that corpus-aware pre-training brings substantial improvements over sophisticated fine-tuning techniques, such as multi-vector encoding and data augmentations.
Among recently proposed dense retrieval pre-training approaches, our model achieves the best results on MS MARCO and TREC DL 2019, even outperforming MASTER, a semi-supervised pre-training approach that uses DocT5Query <cit.> generated queries for additional data augmentation.

Table <ref> illustrates the retrieval performance at different fine-tuning stages.
When fine-tuning with only BM25 negatives, our model achieves the best MRR@10 over all pre-training baselines, outperforming RetroMAE by 1.1 points.
When training with self-mined hard negatives, our model outperforms previous unsupervised baselines by a large margin and is competitive to MASTER which pre-trained longer using more supervision signals.
When using knowledge distillation from a cross-encoder reranker, our model achieves the best overall results.
We also observe that the improvement diminishes with the incorporation of knowledge distillation due to more accurate labeling.
However, running through the cross-encoder reranker for each training instance is costly.
Our model is more robust to label noise presented in the training data given its strong performance in the first two stages.

In terms of retrieval efficiency, our model uses a dual-encoder architecture which brings no extra inference overhead.

Zero-shot Evaluation.

Next, we evaluate the out-of-domain robustness of our pre-training approach on the BEIR benchmark.
We apply the same pre-training procedure on the unlabeled corpus and use the in-domain supervised training data from the BEIR benchmark for fine-tuning.
Table <ref> shows the zero-shot retrieval performance of the retriever on 14 out-of-domain datasets when finetuned with only in-domain data from MS MARCO.
We observe that the contrastively pre-trained Contriever is a strong baseline, which outperforms SimLM and MASTER, but underperforms RetroMAE by a small margin.
Our model shows the best overall performance, outperforming MASTER by 4 points of nDCG@10 and RetroMAE by 2 points, performing best on 9 out of 14 datasets.
This verifies the strong generalization ability of our pre-training approach.

§.§ Ablations and Analysis

In this section, we ablate and analyze several design choices adopted in model training.
By default, we report models' downstream retrieval performance when fine-tuned using only BM25 hard negatives.[
MRR@10 on MS MARCO dev (MS dev) and nDCG@10 on TREC DL tracks are used for ablation evaluation.

Projection Head.
Adding a projection head to transform the representation space is commonly used in contrastive representation learning <cit.>.
We investigate whether this similar approach can contribute to the representation learning in the MAE framework.
Table <ref> compares the downstream fine-tuning retrieval performance with and without the linear projection header between the encoder and decoder during pre-training.
We observe that adding a linear projection head consistently improves the downstream retrieval performance.
In the MAE framework, the projection head acts as a task adapter that can extract task-specific information from hidden representations.
Therefore, the representation bottleneck before the projection head can focus more on learning task-agnostic information which can generalize better to downstream tasks.

Setting     MS dev     DL 19     DL 20

w/o projection     38.4     66.3     65.9

w/ projection     38.8     67.0     67.9

Ablation of the projection head.

Pre-training Corpus.
Pre-training language models on the general corpus or continuous pre-training on domain-specific unlabeled corpus are beneficial for downstream domain-specific tasks <cit.>.
An unanswered question is whether pre-training on the general corpus is still beneficial in the presence of domain-specific corpus.
To validate this, we ablate the MAE-style pre-training on the general corpus by changing the initialization checkpoint before pre-training and fine-tuning it on the MS MARCO corpus.
Table <ref> illustrates the downstream fine-tuning retrieval performance when initialized from different pre-trained checkpoints.
We find that adding a warmup pre-training phase on the general corpus before pre-training on domain-specific corpus is still beneficial to downstream tasks.

Corpus     MS dev     DL 19     DL 20

MS     37.9     64.4     67.6

General $\rightarrow$ MS     38.8     67.0     67.9

Ablation of pre-training corpus.

Scaling Hard Negatives.
More and harder negatives help in contrastive learning by providing a better gradient estimation but suffer from the problem of false negatives <cit.>.
We empirically study the effect of the number of hard negatives used in retriever training.
In a training configuration of batch size $B$ and group size $N$, the total number of negatives for a given query is $B\cdot N - 1$, where $N-1$ are hard negatives and the remaining $(B-1)\cdot N$ are in-batch random negatives.
We keep the total number of negatives $B \cdot N - 1$ fixed and scale the proportion of hard negatives by changing $N$ and $B$ accordingly.
Results are shown in Figure <ref>.
We find that even without denoising false negatives, keep increasing the number of hard negatives during fine-tuning generally leads to better performance.

log basis x=2,
ymin = 30, ymax = 40,
minor tick num = 1,
width = 0.4,
[blue, mark=square] coordinates

log basis x=2,
ymin = 62, ymax = 72,
axis y line*=right,
ylabel near ticks,
minor tick num = 1,
width = 0.4,
legend cell align = left,
legend pos = south east,
/pgfplots/refstyle=plot_oneMS MARCO
[green, mark=o] coordinates
TREC DL

Analysis on the number of hard negatives used in fine-tuning on MS MARCO dev set and TREC DL tracks (averaged).

Masking Latency.
We list the theoretical complexity and empirical time latency of different masking operations adopted by various models in Table <ref>.
Empirical time consumptions are measured at max sequence length $n=150$ and batch size of 128 on one V100 GPU card.
One advantage of our masking strategy is that it is more efficient compared to previous approaches.
In contrast, RetroMAE <cit.> utilizes token-specific masks which involve sampling a mask for each token; SimLM <cit.> needs two extra forward computations of an ELECTRA generator for sampling masked token replacements.
This poses no problem for short sequences but is potentially problematic for longer sequences.

Model     Complexity     CPU     GPU

MAE     $O(n)$     0.4ms     n.a

RetroMAE     $O(n^2)$     11.6ms     n.a

SimLM     $O(n^2)$     0.4ms     2.8ms

$O(n\cdot logn)$     1.1ms     n.a

Time complexity comparison of different masking strategies adopted in various pre-training approaches. $n$ is the max sequence length.

§ CONCLUSION

In this paper, we propose a novel token importance-aware masking strategy to sample more salient co-occurring tokens based on pointwise mutual information, which can be implemented efficiently in an unsupervised manner.
This masking strategy is applied asymmetrically to the encoder and decoder side for better bottleneck representation learning.
Experiments on both in-domain and zero-shot retrieval benchmarks demonstrate the effectiveness of our method, outperforming recently proposed dense retrieval pre-training approaches.

§ IMPLEMENTATION DETAILS

For the pre-training stage, we use a learning rate of $3\times10^{-4}$ with linear scheduling, total batch size of 2048, 0.1 warmup ratio, and AdamW optimizer with 0.01 weight decay.
We run the pre-training for 20 epochs and take the checkpoint at $80k$ steps for retriever initialization.
For supervised fine-tuning, we adopt the three-stage fine-tuning procedure.
The first two-stage retrievers are trained for 3 epochs with a peak learning rate of $2\times 10^{-5}$ with linear scheduling, global batch size of 64, and train group size of 32 in which one is positive and the others are negatives.
We swap the batch size and group size in the third stage.
Reranker is initialized from ELECTRA-base following <cit.>.
We run all experiments on 8 NVIDIA Tesla V100 GPUs with 32GB memory.
We fix the random seed as 42 for reproducibility.