# News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings
for Cross-lingual News Recommendation

Andreea Iana1, Fabian David Schmidt2, Goran Glavaš2, Heiko Paulheim1,
1Data and Web Science Group, University of Mannheim, Germany
2Center For Artificial Intelligence and Data Science, University of Würzburg,
Germany
{andreea.iana<EMAIL_ADDRESS>{fabian.schmidt,
<EMAIL_ADDRESS>
###### Abstract

Rapidly growing numbers of multilingual news consumers pose an increasing
challenge to news recommender systems in terms of providing customized
recommendations. First, existing neural news recommenders, even when powered
by multilingual language models (LMs), suffer substantial performance losses
in zero-shot cross-lingual transfer (ZS-XLT). Second, the current paradigm of
fine-tuning the backbone LM of a neural recommender on task-specific data is
computationally expensive and infeasible in few-shot recommendation and cold-
start setups, where data is scarce or completely unavailable. In this work, we
propose a news-adapted sentence encoder (NaSE), domain-specialized from a
pretrained massively multilingual sentence encoder (SE). To this end, we
construct and leverage PolyNews and PolyNewsParallel, two multilingual news-
specific corpora. With the news-adapted multilingual SE in place, we test the
effectiveness of (i.e., question the need for) supervised fine-tuning for news
recommendation, and propose a simple and strong baseline based on (i) frozen
NaSE embeddings and (ii) late click-behavior fusion. We show that NaSE
achieves state-of-the-art performance in ZS-XLT in true cold-start and few-
shot news recommendation.

News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings
for Cross-lingual News Recommendation

Andreea Iana1, Fabian David Schmidt2, Goran Glavaš2, Heiko Paulheim1, 1Data
and Web Science Group, University of Mannheim, Germany 2Center For Artificial
Intelligence and Data Science, University of Würzburg, Germany {andreea.iana,
<EMAIL_ADDRESS>{fabian.schmidt, goran.glavas}@uni-
wuerzburg.de

##  1 Introduction

News recommender systems constitute the primary instrument used by digital
news platforms to cater to readers’ individual information needs. The ever
increasing language diversity of online users has given rise to new challenges
for news recommenders. Recommender systems must not only produce suitable,
balanced, and diverse recommendations for multilingual news consumers from a
variety of language backgrounds, but also accurately perform in cold-start
scenarios, where news data, user click logs, or both are missing.

On the one hand, recent advancements in pretrained (multilingual) language
models (LMs), used as the backbone of neural news recommenders (NNRs), has
allowed to extend NNRs beyond monolingual recommendation Wu et al. (2021); Guo
et al. (2023). In cross-lingual transfer (XLT), however, even NNRs based on
multilingual LMs display drastic performance loss in target-language
recommendation compared to their source-language (usually English)
recommendation performance Iana et al. (2024). Although accurate XLT is
critical especially for resource-lean(er) languages with limited-to-no click
behavior data, strong multilingual sentence encoders (SEs) – trained exactly
to align sentence semantics across a large number of languages (including many
low-resource ones) – have largely been left unexplored as news encoding
backbones in NNRs.

On the other hand, current NNRs typically fine-tune the underlying LM on task-
specific data. Not only is fine-tuning a time and resource-intensive process,
it is, more critically, infeasible in many real-world scenarios, with: little-
to-no news-click data (i) for news in the target language (for particularly
low-resource ones) or (ii) for the specific user (the so-called cold-start
problem) Wu et al. (2024). Even when using frozen embeddings, most NNRs
strictly require in-domain data to learn meaningful user representations for
prediction (i.e., to train their parameterized user encoders to aggregate the
embeddings of consumed news).

In this context, multilingual sentence encoders, which align sentence-level
semantics across a broad range of languages Yang et al. (2019); Artetxe and
Schwenk (2019); Yang et al. (2020); Feng et al. (2022), hold the promise of
reducing the performance gap in XLT for NNRs. Multilingual SEs, however, have
not been trained for news encoding, i.e., they lack the news domain-specific
knowledge, which is crucial for performance in XLT news recommendation and
cold-start scenarios. Moreover, even if equipped with robustly domain-adapted
LMs, the majority of NNRs would still require fine-tuning on click-behavior
data to learn the weights of their trainable user encoder modules.

Contributions. We address the above limitations and advance cross-lingual and
cold-start news recommendation with the following contributions: (1) We
construct PolyNews111https://huggingface.co/datasets/aiana94/polynews and
PolyNewsParallel222https://huggingface.co/datasets/aiana94/polynews-parallel,
two multilingual news-specific corpora which can be used not only for domain-
adaptation of existing LMs, but also for machine translation (MT). (2) We
train a news-adapted multilingual sentence encoder (dubbed NaSE) by domain-
specializing a general-purpose multilingual SE on PolyNews and
PolyNewsParallel with denoising auto-encoding and MT
objectives.333https://huggingface.co/aiana94/NaSE (3) Leveraging frozen NaSE
embeddings and resorting to non-parameterized late click-behavior fusion Iana
et al. (2023b), we introduce a simple and strong recommendation technique that
yields state-of-the-art performance in zero-shot cross-lingual transfer (ZS-
XLT) recommendation in cold-start setups, as well as in few-shot
recommendation. This challenges the established paradigm of fine-tuning LMs
for news recommendation on click-behavior data.

##  2 Related Work

News Recommendation. Personalized news recommenders attenuate the information
overload for readers by generating suggestions customized to their preferences
Li and Wang (2019); Wu et al. (2023). Most NNRs comprise a dedicated (i) news
encoder (NE), (ii) user encoder (UE), and (iii) click predictor. The NE learns
news embeddings from various input features Wu et al. (2019b, a, 2023) and the
UEs aggregate the embeddings of the users’ clicked news into user-level
representations Okura et al. (2017); An et al. (2019); Wu et al. (2022b).
Lastly, the recommendation score is computed by comparing the candidate’s
embedding against the user representation Wang et al. (2018); Wu et al.
(2019a). NNRs are trained via standard classification objectives Huang et al.
(2013); Wu et al. (2022a) and, more recently, contrastive objectives Iana et
al. (2023c); Liu et al. (2023).

The existing NNRs have two drawbacks. First, the quality of embeddings
produced by NEs with vanilla multilingual LMs seems inadequate for XLT, with
substantial performance losses for target languages Iana et al. (2024). Yet
the usage of multilingual SEs – precisely trained for cross-lingual semantic
alignment across languages on parallel data – as backbone for NEs is
unexplored. Second, fine-tuning of the NE (i.e., its LM) is required, which is
both (i) computationally expensive, as it updates the LM’s hundreds of
millions of parameters on large-scale click-behavior datasets, and, more
critically, (ii) infeasible in setups with limited or no click-behavior data
(e.g., with news in resource-lean languages or in cold-start setups, with no
prior user behavior). In this work, we address these limitations by adapting a
general-purpose multilingual SE for the news domain. As the NE’s backbone,
this enables robust XLT for news recommendation and supports setups where
task-specific fine-tuning is not possible.

Domain-Adaptation of Language Models. The most common approach for injecting
domain knowledge into LMs is pretraining on in-domain data with self-
supervised objectives, such as Masked Language Modeling Devlin et al. (2019),
or SimCSE Gao et al. (2021); Liu et al. (2021); Wang et al. (2022d).
Specializing to a particular domain is done either from scratch Beltagy et al.
(2019); Wu et al. (2020a); Lee et al. (2020) or by adapting already pretrained
LMs Gururangan et al. (2020); Hung et al. (2022). Training from scratch is
computationally intensive and requires large-scale domain-specific corpora,
often difficult to obtain Wang et al. (2022b); Hung et al. (2023). Adaptation
instead is a lighter-weight alternative, as it starts from the already
pretrained LM, requiring more moderate amounts of in-domain data.

Figure 1: Number of texts (log-scale) for the 77 languages of PolyNews and the
five provenance datasets.

For SEs specifically, domain specialization is achieved by training for
similarity or relevance estimation tasks with various self-supervised
objectives Wang et al. (2022a); Liu and Yang (2022) or via in-domain data
generation Wang et al. (2022b). Current work mainly derives domain-specific
SEs from general-purpose (i.e., not sentence-specialized) pretrained LMs. In
this work, in contrast, we adapt an existing multilingual SE on in-domain data
using denoising auto-encoding and machine translation objectives.

##  3 Multilingual News Corpora

A critical aspect of successful domain adaptation of multilingual LMs is the
availability of high-quality training data. To this end, we first compile two
large-scale multilingual corpora by leveraging existing news data collections.

PolyNews. We compile the multilingual non-parallel corpus PolyNews by
combining news from the following five sources: (i) WikiNews (latest dump from
May 2024)444https://www.wikinews.org/, (ii) the train split of MasakhaNews
Adelani et al. (2023), (iii) the train split of MAFAND Adelani et al. (2022),
(iv) WMT-News (v2019), and (v) GlobalVoices (v2018q4) Tiedemann (2012). We
group together the articles from all five sources according to language-script
combinations (some languages, e.g., Serbian, have multiple scripts). We
preprocess the resulting corpus by removing exact duplicates, as well as news
written in scripts not corresponding to the language of the sub-collection
(e.g., Arabic texts in a French or English collection). In addition, we remove
the $K\%$ shortest texts per language (in terms of character length).555We do
not use a fixed length, accounting for the varying expressivity of characters
in different languages. We determine $K$ separately for each of the five news
collections, as the texts’ quality heavily depends on their provenance.
WikiNews, for example, contains many short and uninformative texts, which we
consider noise.666We set $K=15\%$ for WikiNews and $K=3\%$ for the other four
sources. Lastly, given the importance of avoiding text duplication in LM
training Lee et al. (2022), we perform MinHash Broder (1997) near de-
duplication over all sentences of a language.777Following preliminary manual
inspection, we set the de-duplication similarity threshold to 0.9. Finally,
PolyNews contains 3,913,873 news texts, covering 77 languages and 19 scripts.
Fig. 1 shows its distribution across languages and provenance. We further
profile PolyNews in Appendix A.

PolyNewsParallel. We compile the translation corpus PolyNewsParallel from the
parallel news collections MAFAND Adelani et al. (2022), WMT-News and
GlobalVoices Tiedemann (2012). We use the same preprocessing pipeline as for
PolyNews, and remove near-duplicated texts from the source language. Our final
parallel corpus contains 5,386,846 texts over 833 language pairs, spanning 64
languages and 17 scripts.888Fig. 3 shows the texts’ distribution across
language pairs.

##  4 News-Adapted Sentence Encoder

We obtain the news-adapted sentence encoder NaSE via sequence-to-sequence
training on PolyNews and PolyNewsParallel. NaSE can be either (i) further
fine-tuned downstream for news recommendation (i.e., on click-behavior data)
or (ii) directly used as a strong NE in cross-lingual news recommendation,
without any fine-tuning (see §6).

###  4.1 Domain Adaptation

Our first training objective (DAE) is an adaption of the transformer-based
sequential denoising auto-encoding (TSDAE) approach from Wang et al. (2021),
which we use to specialize a pretrained multilingual SE for the news domain.
TSDAE encodes a corrupt version of the input sentence, obtained by adding
discrete noise (e.g., token deletion), and then learns to reconstruct the
original sentence from the encoding of the noisy input. TSDAE can be
formalized as follows:

$\begin{split}\mathcal{J}_{SDAE}(\Theta)&=\mathds{E}_{x\sim D}\left[\log
P_{\Theta}(x|\tilde{x})\right]\\\ &=\mathds{E}_{x\sim
D}\left[\sum_{t=1}^{l}\log P_{\Theta}(x|\tilde{x})\right]\\\
&=\mathds{E}_{x\sim
D}\left[\sum_{t=1}^{l}\log\frac{exp(h_{t}^{T}e_{t})}{\sum_{i=1}^{|V|}exp(h_{t}^{T}e_{i})}\right]\end{split}$
(1)

where $D$ is the training corpus, $x=x_{1}x_{2}...x_{l}$ the input sentence
with $l$ tokens, $\tilde{x}$ the corresponding corruption, $e_{t}$ the
sequence embedding of $x_{t}$, $|V|$ the vocabulary size, and $h_{t}$ the
hidden state at decoding step $t$. At inference, only the encoder is used to
produce the embedding for the input text. We train NaSE as the TSDAE encoder,
with the following adjustment. We initialize NaSE with the pretrained weights
of the popular, widely used multilingual SE LaBSE Feng et al.
(2022).999Because we start from an SE rather than vanilla LM, we remove the
layer from the TSDAE architecture that pools token-level representation into a
sentence embedding.

Our second training objective is domain-specific sequence-to-sequence machine
translation (MT), for which we leverage the parallel data from
PolyNewsParallel. For a given translation pair, we treat the input in the
source language to be the ‘corruption’ $\tilde{x}$ of the target sentence $x$
in the target language, which is to be ‘reconstructed’. Using these two
objective, DAE and MT, we train four different variants of NaSE (as the
resulting encoder of the corresponding sequence-to-sequence model):

1\. NaSEDAE reconstructs the original input sentence from its corruption;

2\. NaSEMT generates the target-language translation of the source-language
input sentence;

3\. NaSEDAE + MT combines the two objectives, by randomly choosing either
reconstruction or translation for each batch; for parallel data, the
reconstruction objective is applied independently on both source- and target-
language sentences;

4\. NaSESEQ is trained sequentially, first on reconstruction, and then on
translation, i.e., we continue training the NaSEDAE encoder for translation on
parallel data. If not specified otherwise, with just NaSE we refer to the
NaSESEQ variant.

We train NaSEDAE on PolyNews, and NaSEMT and NaSEDAE + MT on PolyNewsParallel.
NaSE (i.e., NaSESEQ), as a a sequential combination of both training
procedures, is trained first on PolyNews, and then on PolyNewsParallel.

###  4.2 Training Details

Training Data. The distributions of both PolyNews and PolyNewsParallel are
heavily skewed across languages, and the number of news texts for some low-
resource languages is particularly very limited.101010E.g., only 100 texts in
PolyNews for Tigrinya or Kurdish. We thus follow common practice and smoothen
the per-language distribution when sampling for model training Arivazhagan et
al. (2019); Conneau et al. (2020); Xue et al. (2021). We first sample only
languages and language-pairs that contain at least 100 texts in PolyNews and
PolyNewsParallel, respectively. We then sample texts from language $L$ by
sampling from the modified distribution $p(L)\propto|L|^{\alpha}$, with $|L|$
as the number of examples in $L$ and $\alpha$ as the smoothing rate
hyperparameter ($\alpha<1$ upsamples low-resource and downsamples high-
resource languages). We set $\alpha$ to $0.3$, the value found to balance the
performance between high- and low-resource languages well Conneau et al.
(2020); Xue et al. (2021).

Training Settings. We follow Wang et al. (2021) and use token deletion with a
ratio of 0.6 as the discrete corruption for the DAE variants of NaSE. In all
training setups, we tie the encoder and decoder parameters, and initialize
them with LaBSE weights. We train each model variant for 50K steps with a
learning rate of $3e^{-5}$ using AdamW Loshchilov and Hutter (2018) as the
optimizer. We checkpoint the model every 5K steps.

Validation Setup. We validate NaSE during training on cross-lingual news
recommendation. Concretely, for each model checkpoint, we encode the clicked
and candidate news using the frozen encoder. We adopt the late fusion approach
Iana et al. (2023b), and replace the parameterized UE with mean-pooling of
dot-product scores between the candidate’s and clicked news’ embeddings.
Eliminating a parameterized UE has two benefits: (1) increases computational
efficiency of NaSE validation, as we do not have to train the NNR (i.e., its
UE) and (2) isolates NaSE (i.e., the NE), as the only component that affects
recommendation performance, eliminating UE as a confounder.

Validation Data. A robust multilingual SE should produce good embeddings
across many languages. We thus validate the quality of the NaSE embeddings
during training—for the purposes of model (i.e., optimal checkpoint)
selection—on the small variants of the English MIND Wu et al. (2020b) and its
multilingual xMIND Iana et al. (2024) counterpart. xMIND comprises news
articles from MIND machine-translated into 14 languages. To measure
recommendation performance, we combine multilingual news articles from xMIND
with the user click-behavior data from MIND; our final validation set covers
user behavior from the last day of the MIND training set. Our resulting
validation corpus covers 15 linguistically diverse languages, offering a more
realistic estimate of the quality of multilingual sentence embeddings produced
by NaSE for XLT in news recommendation.

##  5 Experimental Setup

Zero-shot cross-lingual (ZS-XLT) news recommendation is the task for which we
primarily develop NaSE, and thus, the downstream task on which we evaluate it.
In all experiments, we assume only monolingual news consumption, i.e., that
each user reads news only in one language and, accordingly, also receives
recommendations in one language.

Neural News Recommenders. We evaluate four diverse NNRs, mutually differing in
the input fed to their NEs, as well as their UE architectures: (1) NAML Wu et
al. (2019a), (2) MINS Wang et al. (2022c), (3) CAUM Qi et al. (2022), and (4)
MANNeR Iana et al. (2023c).111111Only the base version with the CR-module
without any A-module for aspect-based diversification. Additionally, we
consider three simpler yet competitive baselines: (5) LFRec-CE and (6) LFRec-
SCL, and the text-agnostic (7) NAMLCAT. Both LFRec-CE and LFRec-SCL use a
pretrained LM as the NE, and resort to the non-parameterized late fusion
approach of Iana et al. (2023b) for aggregating the click behaviors. LFRec-CE
is optimized by minimizing the cross-entropy (CE) loss, whereas LFRec-SCL
minimizes the supervised contrastive loss (SCL) Khosla et al. (2020), similar
to MANNeR. Lastly, NAMLCAT is a text-agnostic variant of NAML which learns
news embeddings purely as randomly-initialized and fine-tuned category
vectors. Following Wu et al. (2021), we replace the original NEs of NAML,
MINS, and CAUM – consisting of word embeddings contextualized with convolution
or attention layers – with a pretrained LM. Concretely, we experiment with (1)
the pretrained multilingual LM XLM-RoBERTalargeConneau et al.
(2020),121212NAML, MINS, and CAUM pool token embeddings with an attention
layer to obtain the sentence embedding from the LM, whereas MANNeR uses the
vector of the CLS token. (2) the BERT-based multilingual SE LaBSE Feng et al.
(2022), and (3) our proposed NaSE. For further details about the recommenders’
architecture, we refer the reader to Appendix B.

| Train | Validation | Test
---|---|---|---
# News | 51,282 | 51,282 | 42,416
# Users | 45,214 | 19,703 | 48,593
# Impressions | 124,229 | 29,498 | 70,938

Table 1: MIND (small) and xMIND (small) statistics. Note that for xMIND we
report the statistics per language, i.e., in total xMIND contains 14
languages.

Model | # Parameters (M) | AUC | MRR | nDCG@5 | nDCG@10
---|---|---|---|---|---
Trainable | Total | ENG | AVG | $\%\Delta$ | ENG | AVG | $\%\Delta$ | ENG | AVG | $\%\Delta$ | ENG | AVG | $\%\Delta$
Text & language-agnostic |  |
NAMLCAT | 0.387 | 0.387 | 55.46$\pm$0.18 | 0.00 | 31.12$\pm$0.56 | 0.00 | 29.44$\pm$0.67 | 0.00 | 35.81$\pm$0.59 | 0.00
Fine-tuned news encoder (LM/SE) |  |
LFRec-CEXLM-RoBERTalarge | 307 | 559 | 50.00$\pm$0.00 | 50.00$\pm$0.00 | 0.00 | 35.66$\pm$0.82 | 33.05$\pm$0.42 | -2.61 | 33.68$\pm$0.91 | 31.23$\pm$0.52 | -7.27 | 39.95$\pm$0.94 | 37.53$\pm$0.49 | -6.06
LFRec-CELaBSE | 414 | 470 | 62.89$\pm$1.50 | 62.20$\pm$1.08 | -1.10 | 36.71$\pm$0.53 | 35.65$\pm$0.50 | -2.90 | 35.04$\pm$0.67 | 33.88$\pm$0.62 | -3.30 | 41.30$\pm$0.60 | 40.15$\pm$0.56 | -2.77
LFRec-CENaSE | 414 | 470 | 65.19$\pm$0.23 | 63.00$\pm$1.71 | -3.35 | 36.24$\pm$0.47 | 33.39$\pm$1.07 | -7.85 | 34.31$\pm$0.48 | 31.54$\pm$1.13 | -8.07 | 40.74$\pm$0.44 | 37.92$\pm$1.13 | -6.93
LFRec-SCLXLM-RoBERTalarge | 307 | 559 | 50.00$\pm$0.00 | 50.00$\pm$0.00 | 0.00 | 29.73$\pm$2.17 | 26.43$\pm$2.72 | -11.09 | 27.51$\pm$2.47 | 24.15$\pm$2.99 | -12.21 | 33.70$\pm$2.72 | 30.44$\pm$3.02 | -9.67
LFRec-SCLLaBSE | 414 | 470 | 68.94$\pm$0.87 | 67.29$\pm$0.75 | -2.39 | 36.45$\pm$0.65 | 35.04$\pm$0.53 | -3.87 | 34.93$\pm$0.62 | 33.42$\pm$0.49 | -4.32 | 41.14$\pm$0.60 | 39.65$\pm$0.47 | -3.62
LFRec-SCLNaSE | 414 | 470 | 68.63$\pm$1.23 | 66.73$\pm$0.91 | -2.76 | 36.59$\pm$0.87 | 34.78$\pm$0.61 | -4.95 | 34.80$\pm$0.07 | 33.09$\pm$0.57 | -4.93 | 41.07$\pm$0.82 | 39.35$\pm$0.52 | -4.17
NAMLXLM-RoBERTalarge | 312 | 564 | 52.85$\pm$2.27 | 52.49$\pm$2.60 | -0.68 | 35.98$\pm$0.44 | 33.98$\pm$0.95 | -5.56 | 34.11$\pm$0.46 | 32.13$\pm$1.08 | -5.80 | 40.43$\pm$0.39 | 38.38$\pm$1.02 | 5.06
NAMLLaBSE | 414 | 471 | 51.59$\pm$0.71 | 51.93$\pm$0.65 | +0.66 | 36.63$\pm$0.68 | 35.91$\pm$0.51 | -1.98 | 34.93$\pm$0.80 | 34.14$\pm$0.53 | -2.27 | 41.25$\pm$0.71 | 40.42$\pm$0.49 | -1.99
NAMLNaSE | 414 | 471 | 53.65$\pm$0.90 | 51.06$\pm$0.54 | -4.83 | 36.97$\pm$0.68 | 34.38$\pm$0.95 | -7.00 | 35.24$\pm$0.74 | 32.61$\pm$0.98 | -7.46 | 41.62$\pm$0.66 | 38.94$\pm$0.93 | -6.43
MINSXLM-RoBERTalarge | 316 | 568 | 59.89$\pm$0.29 | 56.94$\pm$1.40 | -4.93 | 34.75$\pm$0.24 | 33.11$\pm$0.51 | -4.70 | 32.54$\pm$0.23 | 31.32$\pm$0.52 | -3.75 | 39.35$\pm$0.20 | 37.64$\pm$0.50 | -4.35
MINSLaBSE | 416 | 473 | 63.70$\pm$1.02 | 62.31$\pm$0.77 | -2.19 | 34.31$\pm$0.80 | 33.58$\pm$0.58 | -2.11 | 32.52$\pm$0.80 | 31.72$\pm$0.54 | -2.46 | 39.06$\pm$0.69 | 38.22$\pm$0.46 | -2.17
MINSNaSE | 416 | 473 | 62.33$\pm$0.95 | 60.81$\pm$1.23 | -2.44 | 34.54$\pm$0.89 | 33.53$\pm$0.67 | -2.92 | 32.79$\pm$1.06 | 31.83$\pm$0.73 | -2.92 | 39.36$\pm$0.86 | 38.35$\pm$0.62 | -2.58
CAUMXLM-RoBERTalarge | 316 | 568 | 57.82$\pm$3.01 | 55.90$\pm$2.23 | -3.32 | 32.92$\pm$1.68 | 31.38$\pm$1.62 | -4.68 | 31.09$\pm$1.88 | 29.60$\pm$1.76 | -6.50 | 37.49$\pm$1.71 | 35.96$\pm$1.58 | -4.08
CAUMLaBSE | 417 | 474 | 64.92$\pm$0.83 | 63.52$\pm$0.80 | -2.16 | 34.36$\pm$0.40 | 33.53$\pm$0.69 | -2.69 | 32.82$\pm$0.19 | 31.73$\pm$0.55 | -3.33 | 39.40$\pm$0.21 | 38.34$\pm$0.54 | -2.70
CAUMNaSE | 417 | 474 | 64.54$\pm$0.53 | 62.54$\pm$0.37 | -3.10 | 35.24$\pm$0.40 | 33.97$\pm$0.27 | -3.61 | 33.42$\pm$0.38 | 32.03$\pm$0.30 | -4.16 | 40.09$\pm$0.35 | 38.59$\pm$0.29 | -3.74
MANNeRXLM-RoBERTalarge | 310 | 562 | 50.00$\pm$0.00 | 50.00$\pm$0.00 | 0.00 | 35.58$\pm$0.31 | 33.03$\pm$0.54 | -7.15 | 33.26$\pm$0.22 | 31.34$\pm$0.48 | -5.77 | 39.17$\pm$0.21 | 37.64$\pm$0.44 | -6.28
MANNeRLaBSE | 415 | 472 | 68.31$\pm$1.26 | 66.89$\pm$1.21 | -2.09 | 36.52$\pm$0.65 | 35.24$\pm$0.52 | -3.50 | 34.91$\pm$0.70 | 33.59$\pm$0.53 | -3.78 | 41.11$\pm$0.65 | 39.81$\pm$0.48 | -3.16
MANNeRNaSE | 415 | 472 | 67.22$\pm$0.40 | 65.29$\pm$0.68 | -2.87 | 35.41$\pm$0.61 | 33.86$\pm$0.44 | -4.36 | 33.49$\pm$0.38 | 32.12$\pm$0.39 | -4.09 | 39.88$\pm$0.41 | 38.44$\pm$0.36 | -3.59
Frozen news encoder (LM/SE) |  |
LFRecXLM-RoBERTalarge | 0 | 559 | 50.00$\pm$0.00 | 50.00$\pm$0.00 | -0.00 | 24.82$\pm$0.00 | 24.64$\pm$0.00 | -0.72 | 22.17$\pm$0.00 | 22.27$\pm$0.00 | +0.75 | 28.44$\pm$0.00 | 28.71$\pm$0.00 | +0.92
LFRecLaBSE | 0 | 471 | 50.47$\pm$0.00 | 50.98$\pm$0.00 | +1.00 | 30.96$\pm$0.00 | 28.99$\pm$0.00 | -6.35 | 28.62$\pm$0.00 | 26.64$\pm$0.00 | -6.92 | 34.83$\pm$0.00 | 32.95$\pm$0.00 | -5.41
LFRecNaSE | 0 | 471 | 64.19$\pm$0.00 | 63.79$\pm$0.00 | -0.62 | 34.74$\pm$0.00 | 33.96$\pm$0.00 | -2.26 | 33.02$\pm$0.00 | 32.14$\pm$0.00 | -2.66 | 39.01$\pm$0.00 | 38.23$\pm$0.00 | -2.00
NAMLXLM-RoBERTalarge | 4.9 | 564 | 50.01$\pm$0.01 | 50.01$\pm$0.01 | 0.00 | 26.75$\pm$1.11 | 26.59$\pm$0.92 | -0.62 | 24.58$\pm$1.18 | 24.29$\pm$0.92 | -1.21 | 31.06$\pm$1.26 | 30.93$\pm$0.89 | -0.42
NAMLLaBSE | 0.156 | 471 | 61.17$\pm$0.01 | 59.15$\pm$0.36 | -3.31 | 34.57$\pm$0.09 | 33.83$\pm$0.38 | -2.15 | 32.90$\pm$0.02 | 32.01$\pm$0.38 | -2.69 | 39.21$\pm$0.05 | 38.13$\pm$0.44 | -2.75
NAMLNaSE | 0.156 | 471 | 63.30$\pm$0.14 | 61.05$\pm$0.33 | -3.55 | 33.89$\pm$0.16 | 32.73$\pm$0.21 | -3.43 | 31.98$\pm$0.12 | 31.04$\pm$0.21 | -2.94 | 38.47$\pm$0.10 | 37.47$\pm$0.18 | -2.60
MINSXLM-RoBERTalarge | 8.8 | 568 | 49.93$\pm$0.28 | 50.030$\pm$0.22 | +0.74 | 29.77$\pm$0.42 | 28.13$\pm$1.49 | -5.83 | 27.66$\pm$0.51 | 26.33$\pm$0.51 | -5.04 | 33.96$\pm$0.52 | 32.74$\pm$1.47 | -3.70
MINSLaBSE | 2.7 | 473 | 61.25$\pm$1.19 | 60.37$\pm$1.38 | -1.43 | 32.69$\pm$0.35 | 32.00$\pm$0.51 | -2.09 | 30.99$\pm$0.35 | 30.21$\pm$0.55 | -2.51 | 37.37$\pm$0.39 | 36.50$\pm$0.55 | -2.34
MINSNaSE | 2.7 | 473 | 60.19$\pm$0.64 | 58.62$\pm$0.81 | -2.61 | 32.23$\pm$0.54 | 31.65$\pm$0.68 | -1.80 | 30.52$\pm$0.62 | 29.91$\pm$0.75 | -2.00 | 37.08$\pm$0.55 | 36.44$\pm$0.70 | -1.73
CAUMXLM-RoBERTalarge | 8.2 | 559 | 55.14$\pm$0.61 | 55.00$\pm$0.67 | -0.25 | 28.55$\pm$0.71 | 28.40$\pm$0.76 | -0.52 | 26.61$\pm$0.69 | 26.48$\pm$0.77 | -0.49 | 33.26$\pm$0.59 | 33.14$\pm$0.69 | -0.36
CAUMLaBSE | 3.7 | 474 | 62.82$\pm$0.77 | 61.35$\pm$0.74 | -2.34 | 33.79$\pm$0.59 | 33.00$\pm$0.48 | -2.33 | 31.92$\pm$0.65 | 31.16$\pm$0.54 | -2.40 | 38.40$\pm$0.56 | 37.61$\pm$0.46 | -2.05
CAUMNaSE | 3.7 | 474 | 64.40$\pm$0.61 | 62.69$\pm$0.64 | -2.65 | 34.42$\pm$0.44 | 33.49$\pm$0.37 | -2.72 | 32.73$\pm$0.40 | 31.63$\pm$0.34 | -3.36 | 39.15$\pm$0.43 | 37.98$\pm$0.34 | -2.98
MANNeRXLM-RoBERTalarge | 2.1 | 562 | 62.11$\pm$1.17 | 50.80$\pm$0.30 | -18.21 | 30.88$\pm$1.42 | 23.86$\pm$0.27 | -22.73 | 29.26$\pm$1.31 | 21.22$\pm$0.29 | 27.46 | 35.56$\pm$1.18 | 27.78$\pm$0.25 | -21.88
MANNeRLaBSE | 1.7 | 472 | 64.54$\pm$0.97 | 61.97$\pm$0.85 | -3.98 | 33.96$\pm$0.87 | 31.45$\pm$0.76 | -7.37 | 32.25$\pm$0.82 | 29.62$\pm$0.71 | -8.15 | 38.51$\pm$0.77 | 35.82$\pm$0.67 | -6.98
MANNeRNaSE | 1.7 | 472 | 65.51$\pm$1.07 | 64.14$\pm$0.78 | -2.08 | 34.68$\pm$0.89 | 33.45$\pm$0.72 | -3.54 | 32.94$\pm$0.92 | 31.78$\pm$0.71 | -3.53 | 39.13$\pm$0.77 | 37.94$\pm$0.62 | -3.05

Table 2: ZS-XLT recommendation performance. For each model, we report the size
in terms of millions of trainable and total parameters, and the performance
(i) on the English MIND dataset (denoted ENG), (ii) averaged across all 14
target languages in xMIND (denoted AVG), and (iii) the relative percentage
difference between average ZS-XLT and ENG performance ($\%\Delta$). We report
averages and standard deviations across three runs.

Data. We conduct experiments on the small variants of the English MIND Wu et
al. (2020b) and the multilingual xMIND Iana et al. (2024) datasets. As
mentioned in §4.2, we couple the news texts from xMIND with the click-behavior
data from MIND, via news IDs. Wu et al. (2020b) do not release test labels for
MIND. Hence, we use the validation set for testing, and split the training set
into temporarily disjoint portions for training (first four days) and
validation (last day), as per Table 1.

Fine-Tuning Details. In all experiments that require LM fine-tuning, we update
only the LM’s last four layers.131313In early experiments with XLM-
RoBERTabase, fine-tuning the whole LM did not bring gains compared to updating
only the last four layers. For computational efficiency, we thus keep the
bottom eight layers of LaBSE and NaSE, and the bottom 20 layers of XLM-
RoBERTalarge, frozen. We set the maximum length of click history to 50, and
sample four negatives per positive in training, as per Wu et al. (2022a). We
tune the main hyperparameters of all NNRs. We train the models for 10 epochs,
with a batch size of 8 for the SE-based variants, and 4 for the XLM-
RoBERTalarge-equipped NNRs. We train using mixed precision, the Adam optimizer
Kingma and Ba (2014), and a learning rate of 1e-5. We repeat runs three times
(with different seeds) and report mean and standard deviations for common
metrics: AUC, MRR, nDCG@5, and<EMAIL_ADDRESS>refer the reader to Appendix C
for further details about hyperparameter settings and computing
infrastructure.

##  6 Results and Discussion

We compare the NNRs’ performance for NEs with different LM/SE backbones in
cross-lingual transfer for news recommendation. In this setup, the user
history and candidates during training are monolingual and in English only. At
inference, both the user history and the candidate news are solely in one of
the 14 target languages of xMIND.

Fine-Tuning NEs. We first investigate the standard recommendation, in which
task-specific data (i.e., click behavior information and news impressions) are
used to train the NNR. The upper half of Table 2 displays the performance of
the NNRs when the underlying NEs are fine-tuned (i.e., the weights of the
backbone LM or SE are updated) for news recommendation, on English data.

While SE-powered NNRs outperform the text-agnostic baseline NAMLCAT, the NNRs
relying on XLM-RoBERTalarge sometimes underperform this simple baseline.
Similarly, XLM-RoBERTalarge underperforms LaBSE and NaSE on both English
recommendation and in ZS-XLT, regardless of the NNR in which it is used. We
believe this is because the XLM-RoBERTalarge-based NE first needs to learn how
to aggregate token representations into sentence embeddings. These results
clearly render (multilingual) SEs, pretrained to produce robust sentence
embeddings, beneficial for news recommendation. SE-based NNRs achieve similar
performance with LaBSE and NaSE, both in English, and in ZS-XLT on xMIND,
i.e., NaSE does not bring gains over LaBSE from which we derived
it.151515Appendix D.1 provides detailed per-language results. We argue that
this is due to the fact that fine-tuning on news recommendation also leads to
domain adaptation: LaBSE itself becomes sufficiently specialized for the news
domain through large-scale recommendation fine-tuning, compensating for NaSE’s
task-agnostic domain adaptation.

Lastly, we note that the simple baselines LFRec-CE, and in particular LFRec-
SCL, exhibit strong recommendation performance, often surpassing more complex
models like CAUM or MINS, which have richer input (e.g., topical categories,
named entities) and parameterized UEs. This is in line with the findings of
Iana et al. (2023b) and questions the need for complex parameterized UEs.

Model | AUC | MRR | nDCG@5 | nDCG@10
---|---|---|---|---
ENG | AVG | $\%\Delta$ | ENG | AVG | $\%\Delta$ | ENG | AVG | $\%\Delta$ | ENG | AVG | $\%\Delta$
10-shots
LFRec-SCLLaBSE | 51.02$\pm$0.06 | 51.17$\pm$0.02 | +0.30 | 31.17$\pm$0.16 | 29.02$\pm$0.08 | -6.90 | 28.88$\pm$0.18 | 26.70$\pm$0.11 | -7.57 | 35.11$\pm$0.18 | 33.01$\pm$0.10 | -5.96
LFRec-SCLNaSE | 61.62$\pm$0.58 | 61.40$\pm$0.50 | -0.35 | 33.54$\pm$0.30 | 32.98$\pm$0.26 | -1.69 | 31.66$\pm$0.33 | 31.02$\pm$0.27 | -2.03 | 37.70$\pm$0.28 | 37.15$\pm$0.25 | -1.44
NAMLLaBSE | 56.19$\pm$0.41 | 55.13$\pm$0.95 | -1.89 | 33.14$\pm$0.18 | 31.56$\pm$0.96 | -4.75 | 31.18$\pm$0.14 | 29.34$\pm$1.02 | -5.89 | 37.23$\pm$0.18 | 35.63$\pm$1.01 | -4.31
NAMLNaSE | 54.50$\pm$0.52 | 54.51$\pm$0.37 | +0.01 | 33.21$\pm$0.37 | 32.76$\pm$0.35 | -1.34 | 31.26$\pm$0.56 | 30.94$\pm$0.43 | -1.01 | 37.31$\pm$0.55 | 37.04$\pm$0.41 | -0.70
50-shots
LFRec-SCLLaBSE | 61.07$\pm$1.19 | 59.04$\pm$0.59 | -3.32 | 32.36$\pm$0.51 | 30.94$\pm$0.29 | -4.37 | 30.52$\pm$0.62 | 29.03$\pm$0.33 | -4.90 | 36.69$\pm$0.63 | 35.26$\pm$0.30 | -3.87
LFRec-SCLNaSE | 61.55$\pm$1.05 | 59.24$\pm$0.89 | -3.75 | 33.17$\pm$0.39 | 31.77$\pm$0.39 | -4.22 | 31.37$\pm$0.49 | 29.84$\pm$0.41 | -4.87 | 37.43$\pm$0.47 | 35.98$\pm$0.40 | -3.88
NAMLLaBSE | 55.50$\pm$1.13 | 54.57$\pm$0.67 | -1.67 | 32.93$\pm$0.85 | 31.54$\pm$0.56 | -4.23 | 31.15$\pm$0.91 | 29.26$\pm$0.55 | -5.75 | 37.21$\pm$0.87 | 35.60$\pm$0.52 | -4.32
NAMLNaSE | 53.32$\pm$0.21 | 53.47$\pm$0.38 | +0.28 | 33.31$\pm$0.39 | 33.10$\pm$0.42 | -0.64 | 31.42$\pm$0.32 | 31.24$\pm$0.36 | -0.59 | 37.49$\pm$0.32 | 37.33$\pm$0.38 | -0.41
100-shots
LFRec-SCLLaBSE | 61.20$\pm$0.72 | 59.54$\pm$0.43 | -2.72 | 33.19$\pm$0.72 | 31.10$\pm$0.58 | -6.27 | 31.25$\pm$0.80 | 29.07$\pm$0.60 | -6.97 | 37.34$\pm$0.73 | 35.32$\pm$0.53 | -5.42
LFRec-SCLNaSE | 62.29$\pm$1.03 | 59.85$\pm$0.72 | -3.92 | 33.40$\pm$0.42 | 32.07$\pm$0.42 | -3.97 | 31.36$\pm$0.60 | 29.99$\pm$0.45 | -4.36 | 37.52$\pm$0.64 | 36.17$\pm$0.45 | -3.61
NAMLLaBSE | 55.67$\pm$0.94 | 54.15$\pm$0.81 | -2.72 | 34.24$\pm$0.52 | 31.41$\pm$0.86 | -8.26 | 32.45$\pm$0.61 | 29.18$\pm$0.90 | -10.06 | 38.33$\pm$0.61 | 35.40$\pm$0.88 | -7.65
NAMLNaSE | 52.98$\pm$0.34 | 53.38$\pm$0.39 | +0.75 | 33.92$\pm$0.20 | 34.04$\pm$0.22 | +0.37 | 31.90$\pm$0.18 | 32.06$\pm$0.22 | +0.50 | 37.96$\pm$0.23 | 38.05$\pm$0.27 | +0.23

Table 3: ZS-XLT performance in few-shot recommendation (i.e., for different
number of impressions used in training). For each model, we report performance
(i) on the English MIND dataset (denoted ENG), (ii) averaged across all 14
target languages in xMIND (denoted AVG), and (iii) the relative percentage
difference between average ZS-XLT and ENG performance ($\%\Delta$). We report
averages and standard deviations across three runs.

Frozen NEs. Fine-tuning NE backbones (LMs with hundreds of millions of
parameters) on large-scale recommendation data can be prohibitively expensive
for many practitioners. We thus next analyze how NNRs perform with frozen NEs
(i.e., no updates to LM/SE), allowing updates only to other (fewer) trainable
parameters of NNRs. Specifically, the model can now only learn how to encode
other input features, such as categories, or how to aggregate the news
embeddings into user-level encoding, if equipped with parameterized UEs. For
most models, freezing the NE reduces the number of trainable parameters by two
orders of magnitude.

The lower half of Table 2 summarizes the recommendation performance of the
NNRs with frozen NEs. Unsuprisingly, XLM-RoBERTalarge-based NNRs yield the
weakest performance across all languages, as the LM itself cannot be tuned to
encode token sequences. Out-of-the-box NaSE embeddings substantially
outperform LaBSE in most cases cases : e.g., for nDCG@10 by 2.30% on English,
and 3.69% cross-lingually (averaged over all 14 xMIND languages), averaged
across all NNRs. The performance gap between LaBSE and NaSE becomes smaller
when the NNR uses a more complex trainable UE (e.g., MINS): the UE parameters
take over the domain-specialization task in large-scale fine-tuning.

Two key aspects point to successful (task-agnostic) domain-specialization of
NaSE. 1) We observe that for models without trainable UEs the relative
performance loss in ZS-XLT compared to English performance is less pronounced
with NaSE as NE than with LaBSE. As target-language data is often scarce in
many real-world applications, closing the gap to performance on the source
language on which the recommender is trained is crucial for multilingual news
recommendation. 2) Unlike XLM-RoBERTalarge or LaBSE, who both benefit from
large-scale recommendation fine-tuning, NaSE’s performance when frozen is much
closer to its fine-tuned performance. This calls into question the current
paradigm of performing expensive supervised fine-tuning of the NNR’s NE.

Figure 2: Ranking performance w.r.t. nDCG@10 of LFRec with frozen NaSE
embeddings for different training strategies of NaSE vs. LFRec with frozen
LaBSE embeddings, over English and the 14 languages of xMIND.

Most importantly, coupled with a frozen NaSE NE, LFRec – a model with no other
trainable parameters – achieves state-of-the-art performance over more
complex, trainable NNRs. On the one hand, this demonstrates the effectiveness
of the news-specialized NaSE encoder (trained in a task-agnostic manner) over
a general-purpose SE like LaBSE in news recommendation. On the other hand, it
proves that LFRec can produce good recommendations in true cold-start setup,
where no news or user data is available (for training a parameterized NNR).
This is in contrast to all the other models that require task-specific data to
learn meaningful user representations, used to compute the final
recommendation scores. Notably, this means that with LFRec with (frozen) NaSE
we obtain a state-of-the-art news recommendation performance without the need
for any task-specific training for news recommendation. LFRec with (frozen)
NaSE outperforms more complex NNRs (NAML, MINS, CAUM), with UEs trained for
recommendation, in terms of ranking (i.e., MRR, nDCG@$k$), and performs on-par
or better in terms of classification (i.e., AUC). These results suggest that
domain-specialization of a multilingual SE (i.e., NaSE) removes the need for
supervised training of NNRs.

Few-Shot Recommendation. To corroborate the previous findings, we also
investigate the performance of the NNRs when the underlying NE is fine-tuned
on just a few task-specific examples, namely in few-shot recommendation. Here,
we assume only a handful of task-specific examples for training the NNRs. In
terms of XLT, we stick to zero-shot XLT for recommendation, i.e., we assume
that the few training instances that we have are all in the source language,
i.e., English.161616This should not be confused with few-shot XLT for
recommendation, which assumes (few) target-language instances in the user
histories during training, but overall relies on large amounts of task-
specific training data Iana et al. (2024).

Table 3 shows the results for 10, 50, and 100 shots, i.e., the number of
impressions used to train the NNR.171717Appendix D.1 provides detailed per-
language results. We again observe that, when fine-tuned on small amounts of
data, NaSE as the NE generally outperforms LaBSE, especially for fewer shots
(10 and 50). In 10-shot recommendation, NaSE’s relative gain in terms of
nDCG@10 over LaBSE ranges from 3.81% (averaged over 14 target languages) and
0.21% on English (for NAML) to 11.14% and 6.87% (for LFRec-SCL), respectively.
The differences between the two decrease with more training data (100 shots).
Besides cold starts, these results render NaSE effective in ZS-XLT
recommendation in realistic low-data setups.

Impact of Training Strategy. Lastly, we ablate NaSE’s performance for
different training strategies. Fig. 2 shows the ranking performance (i.e.,
nDCG@10) of LFRec with frozen NaSE embeddings. We find that the denoising
auto-encoding pretraining objective (DAE) underperforms all other NaSE
configurations. Nonetheless, even LFRec with frozen NaSEDAE embeddings
achieves between 0.19% (on THA) and 11.43% (on FIN) relative improvements over
the model using frozen LaBSE embeddings, outperforming it on 12 out of 15
languages. The translation objective seems highly effective for all languages:
NaSEMT brings up to 16.66% and 16.82% improvements relative to NaSEDAE, and
LaBSE, respectively. Interestingly, the gains seem largest for languages not
present in PolyNewsParallel on which we train NaSE (e.g., THA) and those not
seen by LaBSE in pretraining (e.g., GRN): this hints at positive XLT of
domain-specialization via the MT training.

Training on both DAE and MT slightly improves DAE-only specialization for most
languages. However, sequentially combining the two (NaSESEQ; referred to with
just NaSE) brings further gains and overall best performance. We believe that
this is due to the sequential training being exposed to more data, i.e., both
PolyNews (in the DAE stage) and PolyNewsParallel (in the MT stage). Compared
to LaBSE, NaSE in this setup yields relative improvements ranging from 6.59%
on ZHO, to 10.71% on ENG, and 17.58% on THA.

It is worth emphasizing that news-specialization of NaSE, with a translation-
based objective, benefits also languages not included in PolyNewsParallel
(e.g., THA, HAT, KAT), but also even languages not present in LaBSE’s large-
scale parallel pretraining corpus (e.g., GRN). This points to the language-
agnostic nature of news-specific domain specialization. This finding is in
line with results from prior work in machine translation Tran et al. (2020);
Duquenne et al. (2023), which showed that modest amounts of parallel data (as
in our PolyNewsParallel) improve the cross-lingual semantic alignment in
multilingually trained encoder-decoder models. This then results in
performance gains even for languages not present in the parallel corpora.

##  7 Conclusion

Current NNRs based on multilingual LMs (i) suffer from substantial performance
losses in ZS-XLT recommendation, and (ii) usually require expensive fine-
tuning of the LM used as the news encoder backbone: such fine-tuning is often
infeasible, e.g., in cold-start setups without click-behavior data. In this
work, we proposed NaSE, a news-adapted sentence encoder obtained through
domain specialization of a pretrained multilingual SE. To this end, we
compiled and leveraged two multilingual news-specific corpora, PolyNews and
PolyNewsParallel. Our findings question the effectiveness of supervised fine-
tuning for news recommendation. As an efficient solution, we proposed a simple
and strong baseline based on frozen NaSE embeddings and late click behavior
fusion that achieves state-of-the-art performance in ZS-XLT in true cold start
and few-shot news recommendation.

## Limitations

As our model is initialized from LaBSE and pretrained on PolyNews, it inherits
some of their limitations. For instance, it supports out of the box a limited
set of 128 languages, i.e., the set of 109 languages included in LaBSE’s
pretraining Feng et al. (2022), and the 19 new languages included in PolyNews
which are out-of-sample for LaBSE (cf. Table 4). Consequently, NaSE could be
less effective as a sentence encoder for other languages.181818This would not
necessarily hold true for out-of-sample languages that are closely related to
the (high-resource) languages seen in LaBSE’s pretraining or NaSE’s domain-
adaptive pretraining, as we have seen evidence in $6. This additionally means
that NaSE should be in principle useless for languages written in scripts that
it has not seen seen in LaBSE’s or its own domain-adaptive pretraining, as the
corresponding encoder will output a sequence of unknown tokens ([UNK]). Hence,
NaSE supports only a tiny fraction of the world’s 7000+ languages Joshi et al.
(2020), similar to the majority of multilingual SEs.

Another limitation is that NaSE is evaluated on the xMIND dataset, and hence,
its ZS-XLT performance estimation is bounded to English and the 14 languages
of xMIND. Moreover, most of the languages included in xMIND are covered in the
set of 138 languages supported by NaSE. This means that the average
performance reported is highly likely a gross overestimate for languages
unseen during pretraining. Our results from $6, however, show that NaSE also
generalizes well to languages not seen during its pretraining (e.g., GRN).
Overall, evaluating NaSE on a wider range of languages would provide a better
estimate of its global ZS-XLT recommendation performance.

## Ethical Considerations

We use publicly available datasets and benchmarks for training and evaluation,
which are commonly used in the fields of NLP and recommender systems. As such,
any biases or misinformation that might exist in the news and user data
provided in the recommendation datasets could be propagated through the
different recommender systems. Furthermore, the pretrained LMs and SEs used as
the recommenders’ backbone could produce representations that reflect negative
societal biases and stereotypes. Although our current work does not focus on
mitigating them, many existing techniques for debiasing LMs Qian et al.
(2019); Kaneko and Bollegala (2021); Guo et al. (2022) can be directly applied
to NaSE without special modifications.

## Acknowledgments

The authors acknowledge support by the state of Baden-Württemberg through
bwHPC and the German Research Foundation (DFG) through grant INST 35/1597-1
FUGG.

## References

  * Adelani et al. (2022) David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, and Sam Manthalu. 2022.  A few thousand translations go a long way! leveraging pre-trained models for African news translation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3053–3070, Seattle, United States. Association for Computational Linguistics.
  * Adelani et al. (2023) David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abraham Owodunni, Nnaemeka Obiefuna, Muhidin Mohamed, Shamsuddeen Hassan Muhammad, Teshome Mulugeta Ababu, Saheed Abdullahi Salahudeen, Mesay Gemeda Yigezu, Tajuddeen Gwadabe, Idris Abdulmumin, Mahlet Taye, Oluwabusayo Awoyomi, Iyanuoluwa Shode, Tolulope Adelani, Habiba Abdulganiyu, Abdul-Hakeem Omotayo, Adetola Adeeko, Abeeb Afolabi, Anuoluwapo Aremu, Olanrewaju Samuel, Clemencia Siro, Wangari Kimotho, Onyekachi Ogbu, Chinedu Mbonu, Chiamaka Chukwuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola Awosan, Tadesse Kebede, Toadoum Sari Sakayo, Pamela Nyatsine, Freedmore Sidume, Oreen Yousuf, Mardiyyah Oduwole, Kanda Tshinu, Ussen Kimanuka, Thina Diko, Siyanda Nxakama, Sinodos Nigusse, Abdulmejid Johar, Shafie Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard Ngabire, Jules Jules, Ivan Ssenkungu, and Pontus Stenetorp. 2023.  MasakhaNEWS: News topic classification for African languages.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 144–159, Nusa Dua, Bali. Association for Computational Linguistics.
  * An et al. (2019) Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019.  Neural news recommendation with long- and short-term user representations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 336–345, Florence, Italy. Association for Computational Linguistics.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019.  Massively multilingual neural machine translation in the wild: Findings and challenges.  _arXiv preprint arXiv:1907.05019_.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.  _Transactions of the association for computational linguistics_ , 7:597–610.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _ICLR_.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Broder (1997) Andrei Z Broder. 1997.  On the resemblance and containment of documents.  In _Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171)_ , pages 21–29. IEEE.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Çağlar Gulçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using rnn encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dryer and Haspelmath (2013) Matthew S. Dryer and Martin Haspelmath, editors. 2013.  _WALS Online (v2020.3)_.  Zenodo.
  * Duquenne et al. (2023) Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023.  Sonar: sentence-level multimodal and language-agnostic representations.  _arXiv e-prints_ , pages arXiv–2308.
  * Falcon and The PyTorch Lightning team (2019) William Falcon and The PyTorch Lightning team. 2019.  PyTorch Lightning.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.  Language-agnostic BERT sentence embedding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 878–891, Dublin, Ireland. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Guo et al. (2023) Taicheng Guo, Lu Yu, Basem Shihada, and Xiangliang Zhang. 2023.  Few-shot news recommendation via cross-lingual transfer.  In _Proceedings of the ACM Web Conference 2023_ , pages 1130–1140.
  * Guo et al. (2022) Yue Guo, Yi Yang, and Ahmed Abbasi. 2022.  Auto-debias: Debiasing masked language models with automated biased prompts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1012–1023, Dublin, Ireland. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Hammarström et al. (2021) Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2021.  Glottolog 4.5.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013.  Learning deep structured semantic models for web search using clickthrough data.  In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_, pages 2333–2338.
  * Hung et al. (2023) Chia-Chien Hung, Lukas Lange, and Jannik Strötgen. 2023.  TADA: Efficient task-agnostic domain adaptation for transformers.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 487–503, Toronto, Canada. Association for Computational Linguistics.
  * Hung et al. (2022) Chia-Chien Hung, Anne Lauscher, Simone Ponzetto, and Goran Glavaš. 2022.  DS-TOD: Efficient domain specialization for task-oriented dialog.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 891–904, Dublin, Ireland. Association for Computational Linguistics.
  * Iana et al. (2023a) Andreea Iana, Goran Glavaš, and Heiko Paulheim. 2023a.  NewsRecLib: A PyTorch-lightning library for neural news recommendation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 296–310, Singapore. Association for Computational Linguistics.
  * Iana et al. (2023b) Andreea Iana, Goran Glavas, and Heiko Paulheim. 2023b.  Simplifying content-based neural news recommendation: On user modeling and training objectives.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2384–2388.
  * Iana et al. (2023c) Andreea Iana, Goran Glavaš, and Heiko Paulheim. 2023c.  Train once, use flexibly: A modular framework for multi-aspect neural news recommendation.  _arXiv preprint arXiv:2307.16089_.
  * Iana et al. (2024) Andreea Iana, Goran Glavaš, and Heiko Paulheim. 2024.  Mind your language: A multilingual dataset for cross-lingual news recommendation.  _arXiv preprint arXiv:2403.17876_.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kaneko and Bollegala (2021) Masahiro Kaneko and Danushka Bollegala. 2021.  Debiasing pre-trained contextualised embeddings.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1256–1266, Online. Association for Computational Linguistics.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , pages 18661–18673.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1746–1751, Doha, Qatar. Association for Computational Linguistics.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _ICLR_.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Lee et al. (2022) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022.  Deduplicating training data makes language models better.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8424–8445, Dublin, Ireland. Association for Computational Linguistics.
  * Lewis et al. (2009) M Paul Lewis, Gary F Simons, and Charles D Fennig. 2009.  Ethnologue: languages of the world, dallas, texas: Sil international.  _Online version: http://www. ethnologue. com_ , 12(12):2010.
  * Li and Wang (2019) Miaomiao Li and Licheng Wang. 2019.  A survey on personalized news recommendation technology.  _IEEE Access_ , 7:145861–145879.
  * Liu and Yang (2022) Alexander Liu and Samuel Yang. 2022.  Masked autoencoders as the unified learners for pre-trained sentence representation.  _arXiv preprint arXiv:2208.00231_.
  * Liu et al. (2021) Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021.  Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1442–1459, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liu et al. (2023) Rui Liu, Bin Yin, Ziyi Cao, Qianchen Xia, Yong Chen, and Dell Zhang. 2023.  Perconet: News recommendation with explicit persona and contrastive learning.  _arXiv preprint arXiv:2304.07923_.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017.  Embedding-based news recommendation for millions of users.  In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1933–1942.
  * Qi et al. (2022) Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2022.  News recommendation with candidate-aware user modeling.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1917–1921.
  * Qian et al. (2019) Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. 2019.  Reducing gender bias in word-level language models with a gender-equalizing loss function.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 223–228, Florence, Italy. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218.
  * Tran et al. (2020) Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. 2020.  Cross-lingual retrieval for iterative self-supervised training.  _Advances in Neural Information Processing Systems_ , 33:2207–2219.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pages 6000–6010.
  * Wang et al. (2022a) Chieh-Jen Wang, Yung-Ping Tien, and Yun-Wei Hung. 2022a.  Language model based Chinese handwriting address recognition.  In _Proceedings of the 34th Conference on Computational Linguistics and Speech Processing (ROCLING 2022)_ , pages 1–6, Taipei, Taiwan. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP).
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018.  DKN: Deep knowledge-aware network for news recommendation.  In _Proceedings of the 2018 world wide web conference_ , pages 1835–1844.
  * Wang et al. (2021) Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021.  TSDAE: Using transformer-based sequential denoising auto-encoderfor unsupervised sentence embedding learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 671–688, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wang et al. (2022b) Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022b.  GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2345–2360, Seattle, United States. Association for Computational Linguistics.
  * Wang et al. (2022c) Rongyao Wang, Shoujin Wang, Wenpeng Lu, and Xueping Peng. 2022c.  News recommendation via multi-interest news sequence modelling.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7942–7946. IEEE.
  * Wang et al. (2022d) Yaushian Wang, Ashley Wu, and Graham Neubig. 2022d.  English contrastive learning can learn universal cross-lingual sentence embeddings.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9122–9133, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wu et al. (2020a) Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020a.  TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 917–929, Online. Association for Computational Linguistics.
  * Wu et al. (2019a) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a.  Neural news recommendation with attentive multi-view learning.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ , pages 3863–3869.
  * Wu et al. (2019b) Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019b.  Neural news recommendation with multi-head self-attention.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6389–6394, Hong Kong, China. Association for Computational Linguistics.
  * Wu et al. (2022a) Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2022a.  Rethinking InfoNCE: How many negative samples do you need? In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , pages 2509–2515. International Joint Conferences on Artificial Intelligence Organization.
  * Wu et al. (2023) Chuhan Wu, Fangzhao Wu, Yongfeng Huang, and Xing Xie. 2023.  Personalized news recommendation: Methods and challenges.  _ACM Transactions on Information Systems_ , 41(1):1–50.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021.  Empowering news recommendation with pre-trained language models.  In _Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval_ , pages 1652–1656.
  * Wu et al. (2022b) Chuhan Wu, Fangzhao Wu, Tao Qi, Chenliang Li, and Yongfeng Huang. 2022b.  Is news recommendation a sequential recommendation task? In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2382–2386.
  * Wu et al. (2020b) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020b.  MIND: A large-scale dataset for news recommendation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3597–3606, Online. Association for Computational Linguistics.
  * Wu et al. (2024) Xuansheng Wu, Huachi Zhou, Yucheng Shi, Wenlin Yao, Xiao Huang, and Ninghao Liu. 2024.  Could small language models serve as recommenders? towards data-centric cold-start recommendation.  In _Proceedings of the ACM on Web Conference 2024_ , pages 3566–3575.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yang et al. (2019) Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , pages 5370–5378.
  * Yang et al. (2020) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020.  Multilingual universal sentence encoder for semantic retrieval.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 87–94, Online. Association for Computational Linguistics.

Code | Language | Script | Macro-area | Family | Subgrouping | # Speakers (M) | LaBSE | Avg. # bytes | Avg. # chars.
---|---|---|---|---|---|---|---|---|---
amh_Ethi | Amharic | Ethiopic | Africa | Afro-Asiatic | Semitic | 59.69 | ✓ | 190.42 | 80.10
arb_Arab | Modern Standard Arabic | Arabic | Eurasia | Afro-Asiatic | Semitic | 332.46 | ✓ | 225.86 | 130.60
ayr_Latn | Central Aymara | Latin | South America | Aymaran | Central Southern Aymara | 1.46 | ✗ | 122.33 | 118.48
bam_Latn | Bambara | Latin | Africa | Mande | Western Mande | 14.19 | ✗ | 135.94 | 122.92
bbj_Latn | Ghomálá’ | Latin | Africa | Niger-Congo | Atlantic-Congo | 0.35 | ✗ | 86.77 | 64.87
ben_Beng | Bengali | Bengali | Eurasia | Indo-European | Indo-Aryan | 278.18 | ✓ | 300.72 | 128.01
bos_Latn | Bosnian | Latin | Eurasia | Indo-European | Balto-Slavic | 2.68 | ✓ | 52.08 | 50.76
bul_Cyrl | Bulgarian | Cyrillic | Eurasia | Indo-European | Balto-Slavic | 7.88 | ✓ | 256.64 | 148.65
cat_Latn | Catalan | Latin | Eurasia | Indo-European | Italic | 9.29 | ✓ | 132.07 | 129.25
ces_Latn | Czech | Latin | Eurasia | Indo-European | Balto-Slavic | 12.32 | ✓ | 127.28 | 114.53
ckb_Arab | Central Kurdish | Arabic | Eurasia | Indo-European | Indo-Iranian | 5.35 | ✓ | 350.93 | 185.93
dan_Latn | Danish | Latin | Eurasia | Indo-European | Germanic | 5.81 | ✓ | 133.51 | 130.71
deu_Latn | German | Latin | Eurasia | Indo-European | Germanic | 133.91 | ✓ | 136.53 | 134.47
ell_Grek | Greek | Greek | Eurasia | Indo-European | Greek | 13.23 | ✓ | 273.36 | 159.73
eng_Latn | English | Latin | Eurasia | Indo-European | Germanic | 1,515.23 | ✓ | 123.89 | 123.28
est_Latn | Estonian | Latin | Eurasia | Uralic | Finnic | 1.09 (L1 only) | ✓ | 116.43 | 113.24
ewe_Latn | Éwé | Latin | Africa | Niger-Congo | Atlantic-Congo | 5.53 | ✗ | 78.44 | 72.39
fil_Latn | Filipino | Latin | Papunesia | Austronesian | Malayo-Polynesian | – | ✗ | 135.54 | 135.24
fin_Latn | Finnish | Latin | Eurasia | Uralic | Finnic | 5.61 | ✓ | 117.52 | 112.76
fon_Latn | Fon | Latin | Africa | Niger-Congo | Atlantic-Congo | 2.28 | ✗ | 179.91 | 136.43
fra_Latn | French | Latin | Eurasia | Indo-European | Italic | 311.58 | ✓ | 143.82 | 139.25
guj_Gujr | Gujarati | Gujarati | Eurasia | Indo-European | Indo-Iranian | 62.67 | ✓ | 301.16 | 118.81
guw_Latn | Gun | Latin | Africa | Niger-Congo | Atlantic-Congo | 1.54 | ✗ | 93.61 | 76.95
hau_Latn | Hausa | Latin | Africa | Afro-Asiatic | Chadic | 88.24 | ✓ | 138.36 | 137.40
heb_Hebr | Hebrew | Hebrew | Eurasia | Afro-Asiatic | Semitic | 9.36 | ✓ | 132.72 | 77.33
hin_Deva | Hindi | Devanagari | Eurasia | Indo-European | Indo-Iranian | 608.85 | ✓ | 321.32 | 129.44
hun_Latn | Hungarian | Latin | Eurasia | Uralic | – | 12.43 | ✓ | 141.26 | 128.87
ibo_Latn | Igbo | Latin | Eurasia | Niger-Congo | Attlantic-Congo | 30.91 | ✓ | 123.64 | 104.73
ind_Latn | Indonesian | Latin | Papunesia | Austronesian | Malayo-Polynesian | 199.00 | ✓ | 131.97 | 131.56
ita_Latn | Italian | Latin | Eurasia | Indo-European | Italic | 66.79 | ✓ | 138.79 | 137.64
jpn_Jpan | Japanese | Japanese | Eurasia | Japonic | Japanesic | 123.47 | ✓ | 160.54 | 64.63
kaz_Cyrl | Kazakh | Cyrillic | Eurasia | Altaic | Turkic | 16.58 | ✓ | 279.77 | 155.47
khm_Khmr | Khmer | Khmer | Eurasia | Austro-Asiatic | Khmeric | 17.60 | ✓ | 368.65 | 139.12
kor_Hang | Korean | Hangul | Eurasia | Koreanic | Korean | 81.13 | ✓ | 156.69 | 72.81
lav_Latn | Latvian | Latin | Eurasia | Indo-European | Baltic | 1.76 | ✓ | 135.63 | 124.39
lin_Latn | Lingala | Latin | Africa | Niger-Congo | Atlantic-Congo | 40.54 | ✗ | 66.14 | 65.62
lit_Latn | Lithuanian | Latin | Eurasia | Indo-European | Balto-Slavic | 2.78 | ✓ | 135.97 | 126.83
lug_Latn | Ganda | Latin | Africa | Niger-Congo | Atlantic-Congo | 11.14 | ✗ | 111.21 | 110.90
luo_Latn | Luo | Latin | Africa | Nilotic | Western Nilotic | 5.26 | ✗ | 158.18 | 157.76
mkd_Cyrl | Macedonian | Cyrillic | Eurasia | Indo-European | Balto-Slavic | 1.65 | ✓ | 261.49 | 150.68
mos_Latn | Mossi | Latin | Africa | Niger-Congo | Atlantic-Congo | 11.87 | ✗ | 140.87 | 127.62
mya_Mymr | Burmese | Myanmar | Eurasia | Sino-Tibetan | Tibeto-Burman | 43.17 | ✓ | 383.44 | 148.70
nld_Latn | Dutch | Latin | Eurasia | Indo-European | Germanic | 25.31 | ✓ | 124.46 | 123.98
nor_Latn | Norwegian | Latin | Eurasia | Indo-European | Germanic | 5.42 | ✓ | 44.14 | 43.20
npi_Deva | Nepali | Devanagari | Eurasia | Indo-European | Indo-Iranian | 31.98 | ✓ | 441.96 | 178.46
orm_Latn | Oromo | Latin | Africa | Afro-Asiatic | Cushitic | 45.53 (L1 only) | ✗ | 79.86 | 79.57
ory_Orya | Odia | Oriya | Eurasia | Indo-European | Indo-Iranian | 39.76 | ✓ | 268.32 | 107.68
pan_Guru | Eastern Panjabi | Gurmukhi | Eurasia | Indo-European | Indo-Iranian | 31.17 | ✓ | 685.89 | 275.91
pcm_Latn | Nigerian Pidgin | Latin | Africa | Creole | – | 120.65 | ✗ | 118.72 | 118.69
pes_Arab | Western Persian | Arabic | Eurasia | Indo-European | Indo-Iranian | 78.83 | ✓ | 218.93 | 128.05
plt_Latn | Malagasy | Latin | Africa | Austronesia | Malayo-Polinesian | 7.55 | ✓ | 152.57 | 151.77
pol_Latn | Polish | Latin | Eurasia | Indo-European | Balto-Slavic | 41.80 | ✓ | 120.85 | 114.56
por_Latn | Portuguese | Latin | Eurasia | Indo-European | Italic | 263.84 | ✓ | 124.35 | 120.73
ron_Latn | Romanian | Latin | Eurasia | Indo-European | Italic | 25.23 | ✓ | 129.89 | 123.63
run_Latn | Rundi | Latin | Africa | Niger-Congo | Atlantic-Congo | 13.19 | ✗ | 75.73 | 75.20
rus_Cyrl | Russian | Cyrillic | Eurasia | Indo-European | Balto-Slavic | 255.39 | ✓ | 151.75 | 88.92
sna_Latn | Shona | Latin | Africa | Niger-Congo | Atlantic-Congo | 10.88 | ✓ | 67.94 | 67.77
som_Latn | Somali | Latin | Africa | Afro-Asiatic | Cushitic | 23.88 | ✓ | 73.09 | 72.98
spa_Latn | Spanish | Latin | Eurasia | Indo-European | Italic | 559.52 | ✓ | 136.71 | 134.00
sqi_Latn | Albanian | Latin | Eurasia | Indo-European | Albanian | 6.39 | ✓ | 128.83 | 119.94
srp_Cyrl | Serbian | Cyrillic | Eurasia | Indo-European | Balto-Slavic | 10.12 | ✓ | 88.89 | 51.97
srp_Latn | Serbian | Latin | Eurasia | Indo-European | Balto-Slavic | 10.12 | ✓ | 78.22 | 76.17
swe_Latn | Swedish | Latin | Eurasia | Indo-European | Germanic | 13.25 | ✓ | 112.77 | 108.45
swh_Latn | Swahili | Latin | Africa | Niger-Congo | Atlantic-Congo | 86.52 | ✓ | 115.32 | 115.00
tam_Taml | Tamil | Tamil | Eurasia | Dravidian | Southern Dravidian | 86.72 | ✓ | 170.92 | 64.02
tet_Latn | Tetun | Latin | Africa | Austronesian | Malayo-Polynesian | 0.49 | ✗ | 130.19 | 128.88
tha_Thai | Thai | Thai | Eurasia | Kra-Dai | Kam-Tai | 61.10 | ✓ | 144.90 | 56.57
tir_Ethi | Tigrinya | Ethiopic | Africa | Afro-Asiatic | Semitic | 9.87 | ✗ | 116.47 | 47.10
tsn_Latn | Tswana | Latin | Africa | Niger-Congo | Atlantic-Congo | 13.75 | ✗ | 142.02 | 141.69
tur_Latn | Turkish | Latin | Eurasia | Altaic | Turkic | 90.09 | ✓ | 122.87 | 112.45
twi_Latn | Twi | Latin | Africa | Niger-Congo | Atlantic-Congo | 0.024 | ✗ | 119.62 | 109.66
ukr_Cyrl | Ukrainian | Cyrillic | Eurasia | Indo-European | Balto-Slavic | 38.92 | ✓ | 107.62 | 60.99
urd_Arab | Urdu | Arabic | Eurasia | Indo-European | Indo-Iranian | 169.70 | ✓ | 193.15 | 115.04
wol_Latn | Wolof | Latin | Africa | Niger-Congo | Atlantic-Congo | 22.65 | ✓ | 128.05 | 122.11
xho_Latn | Xhosa | Latin | Africa | Niger-Congo | Atlantic-Congo | 19.22 | ✓ | 47.79 | 47.37
yor_Latn | Yorùbá | Latin | Africa | Niger-Congo | Atlantic-Congo | 47.19 | ✓ | 174.46 | 124.68

Table 4: The 77 languages of PolyNews. We display the language Code (ISO
693-3), language name, Script, Macro-area, Family and Genus, and report the
total number of L1-level and L2-level speakers of the language according to
Ethnologue Lewis et al. (2009). The eighth column indicates whether the
language was included in the pretraining corpora of LaBSE Feng et al. (2022).
The last two columns specify the average byte length, and respectively,
character length of the texts for each language. The languages highlighted in
gray were not included in the adaptive pretraining of NaSE.

Code | Language | Script | Macro-area | Family | Subgrouping | # Speakers (M) | LaBSE | Avg. # bytes | Avg. # chars.
---|---|---|---|---|---|---|---|---|---
zho_Hans | Chinese | Han (Simplified) | Eurasia | Sino-Tibetan | Sinitic | 1,352.67 (L1 only) | ✓ | 78.06 | 36.56
zho_Hant | Chinese | Han (Traditional) | Eurasia | Sino-Tibetan | Sinitic | 1,352.67 (L1 only) | ✗ | 76.86 | 36.64
zul_Latn | Zulu | Latin | Africa | Niger-Congo | Atlantic-Congo | 27.80 | ✓ | 165.13 | 164.64

Table 4 (continued): The 77 languages of PolyNews. We display the language
Code (ISO 693-3), language name, Script, Macro-area, Family and Genus, and
report the total number of L1-level and L2-level speakers of the language
according to Ethnologue Lewis et al. (2009). The eighth column indicates
whether the language was included in the pretraining corpora of LaBSE Feng et
al. (2022). The last two columns specify the average byte length, and
respectively, character length of the texts for each language. The languages
highlighted in gray were not included in the adaptive pretraining of NaSE.

##  Appendix A PolyNews Corpora

###  A.1 Language Characteristics

Table 4 lists the languages included in the PolyNews dataset, summarizing the
following information, according to the #BenderRule:

  * •

Code: We denote each language using a BCP 47 tag sequence that combines a base
subtag indicating the three-letter ISO 693-3 code with the ISO 15924 script
subtag. We use the script code to differentiate between languages collected in
multiple scripts.

  * •

Language: In case of multiple denominations, we use the language name from
Ethnologue Lewis et al. (2009), cross-referenced against other linguistic
resources, namely Glottolog Hammarström et al. (2021) and World Atlas of
Structures Dryer and Haspelmath (2013).

  * •

Script: We provide the English name of the script.

  * •

Macro-area: We indicate the macro-area as per WALS Dryer and Haspelmath
(2013).

  * •

Family and Subgrouping: We specify the language family and subgrouping from
WALS Dryer and Haspelmath (2013) and Glottolog Hammarström et al. (2021).

  * •

Total Speakers: We report the total number of speakers of the language,
considering both L1-level (first-language) and L2-level (second-language)
speakers, according to Ethnologue.191919We use the statistics available in May
2024 at https://www.ethnologue.com/.

  * •

LaBSE support: We indicate whether the language was included in the
pretraining corpora of LaBSE Feng et al. (2022).

  * •

Average byte length: We report the average number of bytes per text for each
language.

  * •

Average character length: We report the average number of characters per text
for each language.

###  A.2 Parallel Corpora Statistics

Fig. 3 illustrates the number of texts for each language pair available in the
PolyNewsParallel dataset. Note that PolyNewsParallel consists of only 64 out
of the 77 languages listed in Table 4.

Figure 3: Distribution of texts across the 833 language pairs in
PolyNewsParallel. The gray cells indicate that no texts exist for the given
language pair.

##  Appendix B Neural News Recommenders

We provide further details about the NNRs architectures. Note that we replace
the original NEs of all NNRs not using a LM (i.e., contextualization of
pretrained word embeddings over convolutional neural networks (CNNs) Kim
(2014) or attention networks Bahdanau et al. (2014); Vaswani et al. (2017))
with a pretrained LM, as detailed in $5:

  1. 1.

NAML Wu et al. (2019a) encodes the title, abstract, and categories of news.
Its UE consists of an additive attention network which contextualizes the
embeddings of the users’ clicked news into a user-level representation.

  2. 2.

MINS Wang et al. (2022c) also leverages both textual features and categories
as input to its NE. However, it learns user representations via a combination
of multi-head self-attention Vaswani et al. (2017), multi-channel GRU-based
recurrent network Cho et al. (2014), and additive attention Bahdanau et al.
(2014).

  3. 3.

CAUM Qi et al. (2022) learns news embeddings from the titles, abstracts,
categories, and named entities of news. Its UE combines a candidate-aware
self-attention network with a candidate-aware CNN to learn long-term and
short-term user representations respectively. The intermediary embeddings are
combined in a final candidate-aware user representation by means of an
attention network.

  4. 4.

MANNeR Iana et al. (2023c) represents a modular framework for multi-aspect
neural news recommendation, comprising two kinds of modules, responsible for
content-based (CR-Module) and aspect-based customization (A-Module). In this
work, we experiment only with the base CR-Module. Similar to CAUM, it
leverages named entities in addition to the title and abstract of news as
input features to the NE. In contrast to the majority of NNRs, however, MANNeR
is trained with a contrastive metric objective Khosla et al. (2020). Moreover,
it employs a late click behavior fusion approach Iana et al. (2023b) instead
of parameterized UEs.

  5. 5.

LFRec-CE embeds the title and abstract of news using a pretrained LM and late
click behavior fusion Iana et al. (2023b). It is optimized by minimizing the
cross-entropy loss.

  6. 6.

LFRec-SCL has an analogous architecture to LFRec-CE, but it is trained with a
supervised contrastive objective Khosla et al. (2020).

  7. 7.

NAMLCAT is a variant of NAML Wu et al. (2019a) that computes recommendations
using only the news categories of news, and an additive attention network as
the UE. Hence, it is a text- and language-agnostic recommender.

##  Appendix C Reproducibility Details

Hyperparameter Optimization. We perform hyperparameter optimization on the
main hyperparameters of all NNRs using grid search. More specifically, we
search for the optimal learning rate in $\\{1e^{-3},1^{e-4},1^{e-5}\\}$,
finding $1e^{-5}$ to be the most suitable value for all NNRs. We optimize the
number of heads in the multi-head self-attention networks of NAML, MINS, CAUM
in $[8,12,16,24,32]$, and the query vector dimensionality in the additive
attention network of NAML and MINS in $[50,200]$, with a step of 50. We use
the following best performing settings: 32 attention heads for NAML and MINS
and 8 for CAUM when coupled with XLM-RoBERTalarge, and 24, 16, and 8 attention
heads for NAML, MINS, and CAUM, respectively, when using LaBSE or NaSE as the
NE’s backbone. Moreover, we use a query vector of dimension 200 for NAML and
50 for MINS in combination with XLM-RoBERTalarge, whereas for NEs based on
LaBSE or NaSE we use a dimensionality of 50 for NAML and 100 for MINS. We find
the optimal temperature of 0.38 for the supervised contrastive loss in MANNeR
and LFRec-SCL by sweeping the interval $[0.1,0.5]$, with a step of 0.02. We
set all the remaining model-specific hyperparameters to the optimal values
reported in the respective papers. We repeat each experiment three times with
the seed ($\\{42,43,44\\}$) set with PyTorch Lightning’s seed_everything.

Code. We train NaSE 202020https://github.com/andreeaiana/nase using Sentence
Transformers Reimers and Gurevych (2019) and PyTorch LightningFalcon and The
PyTorch Lightning team (2019).212121https://lightning.ai/docs/pytorch/stable/
We conduct all news recommendation experiments with the NewsRecLib library
Iana et al. (2023a).

Infrastructure and Compute. We train NaSE on a cluster with virtual machines,
on single NVIDIA A100 40GB GPUs. We conduct the news recommendation
experiments on the same cluster, on single NVIDIA A40 48GB GPUs.

##  Appendix D Additional Results

###  D.1 Per-language ZS-XLT performance

Fine-tuned embeddings. Figs. 4 shows the ranking performance of LFRec-SCL with
fine-tuned SE embeddings on MIND and xMIND.

Figure 4: Ranking performance w.r.t. nDCG@10 of LFRec-SCL with fine-tuned NaSE
and LaBSE embeddings, over English and the 14 languages of xMIND.

Few-shot recommendation. Fig. 5 shows the performance of LFRec-SCL in ZS-XLT
few-shot recommendation for different numbers of shots.

Figure 5: Ranking performance w.r.t. nDCG@10 of LFRec-SCL with NaSE and LaBSE
embeddings for different numbers of shots (i.e., impressions during training),
over English and the 14 languages of xMIND.

###  D.2 Influence of Training Strategy.

Figure 6: Recommendation performance w.r.t. AUC of LFRec with frozen NaSE
embeddings for different training strategies of NaSE vs. LFRec with frozen
LaBSE embeddings, over English and the 14 languages of xMIND.

Fig. 6 illustrates the performance in terms of AUC of LFRec with frozen NaSE
and LaBSE embeddings on the MIND and xMIND datasets.