# Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into
Multicultural Proverbs and Sayings

Chen Cecilia Liu1, Fajri Koto2, Timothy Baldwin2, Iryna Gurevych1
1Ubiquitous Knowledge Processing Lab,
Department of Computer Science and Hessian Center for AI (hessian.AI),
Technical University of Darmstadt
2Natural Language Processing Department, MBZUAI
www.ukp.tu-darmstadt.de

###### Abstract

Large language models (LLMs) are highly adept at question answering and
reasoning tasks, but when reasoning in situational context, human expectations
vary depending on the relevant cultural common ground. As human languages are
associated with diverse cultures, LLMs should also be culturally-diverse
reasoners. In this paper, we study the ability of a wide range of state-of-
the-art multilingual LLMs (mLLMs) to reason with proverbs and sayings in a
conversational context. Our experiments reveal that: (1) mLLMs ‘knows’ limited
proverbs and memorizing proverbs does not mean understanding them within a
conversational context; (2) mLLMs struggle to reason with figurative proverbs
and sayings, and when asked to select the wrong answer (instead of asking it
to select the correct answer); and (3) there is a “culture gap” in mLLMs when
reasoning about proverbs and sayings translated from other languages. We
construct and release our evaluation dataset MAPS (MulticultrAl Proverbs and
Sayings) for proverb understanding with conversational context for six
different languages.111Available at https://github.com/UKPLab/maps.

##  1 Introduction

Large language models (LLMs) have achieved impressive results for question
answering and reasoning tasks (Radford et al., 2019; Brown et al., 2020;
Ouyang et al., 2022a, inter alia). However, when reasoning in situational
context, human expectations may vary cross-culturally Thomas (1983) and depend
on the knowledge of the relevant cultural common ground (which cultural common
ground is the shared knowledge based on which people within a culture reason
and communicate, including concepts and common sense, Hershcovich et al.,
2022). The ability of pre-trained LLMs to understand such common ground in a
cross-lingual setting is specifically neglected in NLP Hershcovich et al.
(2022). As languages and cultures are intertwined Kramsch (2014); Hovy and
Yang (2021), it is crucial for models to serve all communities and be able to
reason and communicate in a relevant way.

Figure 1: Proverbs are fixed expressions used by different cultures. We
collect proverbs from six languages (top) and their usage within
conversational contexts. We evaluate mLLMs with a binary-choice inference task
in the conversational context that contains proverbs (bottom).

Focusing on the multilingual LLMs, several questions arise: (1) Do mLLMs embed
knowledge of cultural common ground, and does this knowledge affect their
reasoning performance? (2) Can mLLMs reason in contexts that require an
understanding of cultural common ground? and (3) Can mLLMs reason cross-
culturally (i.e., about another culture’s cultural common ground, after
translating into the same language) and are there gaps in the cultural
knowledge (a “cultural gap”)?222Reasoning with the cultural common ground
could be independent of language. For example, communications among different
cultural groups within a multi-cultural country or for international trade.

In order to answer the above questions, we need to assess LLMs using fixed,
culturally-diverse expressions in multiple languages, that are used flexibly
in situational contexts that require reasoning. Fixed expressions are
particularly important for evaluating the memorization of cultural common
ground knowledge of LLMs. However, prior work such as MaRVL (Liu et al., 2021,
which is in multimodal) or MABL Kabra et al. (2023) do not contain fixed
expressions.

Proverbs and sayings (such as the ones illustrated in Figure 1) are fixed
expressions that convey traditional wisdom, sometimes viewed as a form of folk
literature, and grounded in living experience and social-cultural context
White (1987); Mieder (2004); Honeck (2013). While different proverbs may
emerge for different cultures, the underlying meaning of proverbs usually
expresses universal human experiences. Yet, their literal expression and
interpretation can vary from culture to culture Honeck (2013).

For example, the English proverb The apple doesn’t fall far from the tree —
means a child grows up to resemble his/her parents. While a plain version like
father like son exists in many cultures, this proverb has a similar variant
Rebung tidak jauh dari rumpunnya “Bamboo shoots are not far from the clump” in
Indonesian, and 龙生龙，凤生凤，老鼠的儿子会打洞 “the dragon begets the dragon, the phoenix
begets the phoenix, the son of a rat can make a hole”/虎父无犬子 “tiger father no
dog son” in Chinese. Of course, not all proverbs have parallels in different
languages as they are often culturally dependent.

Furthermore, proverbs are used in writing or conversational settings to offer
advice, make arguments, or to console others. A proverb’s interpretation
depends on the context Mieder (2004) it is used in and is figurative where the
interpreted meaning does not entail the literal meaning. This makes them the
ideal devices for studying the ability to reason in situational contexts of
LLMs.

Hence, in this paper, we propose to use proverbs and sayings as a specific
proxy for studying culturally diverse reasoning. In particular, we study (1)
Do mLLMs know the proverbs and how well mLLMs memorize them? (2) Can mLLMs
choose the correct interpretation of a proverb given a conversational context?
and (3) Can mLLMs reason cross-culturally and are there cultural gaps in the
interpretation of proverbs cross-culturally?

We first present a dataset, MAPS (MulticultrAl Proverbs and Sayings). The
dataset consists of a collection of proverbs and sayings, a inference task for
interpreting the meaning of proverbs in conversational context, and binary
labels indicating if the proverb is figurative. The dataset covers six
languages with geographical diversity: English, German, Russian, Bengali,
Chinese, and Indonesian.

We design a suite of experiments with MAPS on a wide range of open source
state-of-the-art mLLMs. We find that mLLMs do possess knowledge of proverbs
and sayings to varying degrees (significantly biased toward English and
Chinese), and the amount of knowledge scales with model size. Through our
inference task, we also find that the memorization of proverbs does not
indicate better reasoning ability with proverbs, and figurative proverbs are
more difficult for mLLMs to reason about in many languages. On the ability of
mLLMs to reason cross-culturally with cultural common ground, we find that
significant cultural gaps exist when reasoning with translations. Our results
indicate that despite the apparent multilingual reasoning abilities of mLLMs,
further research to improve the cultural-diversity (in terms of cultural
common ground) of mLLMs is needed.

Our contributions are: 1) we provide an analysis of the ability of a wide
range of state-of-the-art open-source mLLMs to reason with cultural common
ground through the lens of proverbs and sayings; 2) We disentangle the effects
of memorization versus reasoning with proverbs and sayings, and reveal culture
gaps in mLLMs; and 3) We construct a multicultural dataset of proverbs and
sayings for six different languages with multiple levels of annotations.

##  2 Related Work

Prior work has evaluated LLMs ability for abstract reasoning (Ghosh and
Srivastava, 2022, recognize proverbs from short stories) in English or asses
models’ ability for matching proverbs across three languages (bench authors,
2023, with a small evalutation set). To the best of our knowledge, MAPS is the
largest multilingual dataset that focuses on proverbs and sayings, with
conversational contexts and a inference task.

MABL Kabra et al. (2023) is a task that’s close to ours, but focuses on novel
metaphors understanding in different cultures and on cross-lingual transfers.
It is less suitable to study memorization vs reasoning, nor studies reasoning
with a context. Ruis et al. (2022) and Hu et al. (2023) use conversational
context to study pragmatic reasoning in English LLMs and identification of
parallels between human and models, which provide limit insights beyond
English. While we also use conversational context, our work focuses on
cultural common ground and multilingual reasoning aspects of LLMs (we also
have a larger dataset).

Recent work by Haviv et al. (2023) aims to understand the memory retrieval
mechanism in LLMs with English idioms. Here, our goal is not to study memory
retrievals in LLMs, but rather focus on the multi-cultural and cross-cultural
reasoning of mLLMs.

##  3 MAPS - MulticultrAl Proverbs and Sayings

To help investigate our proposed research questions, we first present MAPS — a
dataset of proverbs across six geographically and topologically-diverse
languages. MAPS consists of: (1) proverbs, (2) conversational usages as
context, (3) interpretations of proverbs (one correct, one wrong), and (4)
labelling of whether the usage of the proverb is figurative or not (data
examples in Table 2, Figure 6, Appendix A.5 illustrates the annotation
process).

###  3.1 Dataset Creation

#### Language Choices.

We chose six languages for this dataset: English, German, Russian, Bengali,
Chinese, and Indonesian. Several factors where included when choosing the
languages, including geographical diversity such as Eastern vs. Western (to
increase the potential concept diversity), topological diversity, and resource
availability (high-resource vs. lower-resource).

#### Proverbs and Sayings.

We collect all proverbs and sayings (along with explanations) from
Wikiquote333https://en.wikiquote.org/ and
Wiktionary.444https://www.wiktionary.org/. Bengali has a significantly higher
quantity of proverbs compared to other languages, thus, we perform a random
sub-sampling of the proverbs for annotation to keep the final data roughly
balanced.

#### Conversational Context.

While proverbs and sayings are self-contained, they are typically used in
conversations and writing to offer advice or console others. In order to
investigate the ability of mLLMs to reason with proverbs, next we created
short conversations that use proverbs (i.e., conversational context for the
inference task).

To aid the data creation process, we use a human-model hybrid approach (i.e.
model-in-the-loop), inspired by the recent work Chakrabarty et al. (2022); Liu
et al. (2023). We first use GPT3.5 (gpt-3.5-turbo-0301; a sibling model of
Ouyang et al., 2022b) by prompting it with fixed templates to generate the
seed conversational context (see Appendix B for the model templates).555The
conversational contexts are in each perspective language, except for Russian
and Bengali where the contexts are in English due to quality issues. For
Russian and Bengali, the contexts are written in English first, then machine
translated and fixed by native speakers for two rounds. Next, we ask two or
more native speakers (experts or crowd, with least one expert per language) to
either accept the model-created conversation, or write a new conversation if
the human thinks the usage of the proverb is flawed.

In the final dataset, the conversational contexts for English, Chinese,
Russian, and Bengali were completely re-written,666The model has significant
trouble in creating relevant context when the proverb is figurative.
Anecdotally, human annotators found that machine-generated context is helpful
as a ‘prompt’ to human and helped to speed up the re-writes. whereas for
Indonesian and German, 22% and 20.5% of the original model-generated contexts
were retained (the difference is probably due to variations in individual
annotator preferences).

#### Interpretation of Proverbs in Context.

We formulate this part as an inference task (following Liu et al., 2022). We
ask annotators to create one correct answer and one wrong answer to the
following question based on the conversational context:

What does the person mean by {proverb}?

Additionally, we also label the proverb if the interpretation is figurative
(i.e. the interpreted meaning of the proverb is different than the expressed
literal meaning).777“An apple a day keeps the doctor away” is a literal
proverb that is advocating for apple consumption. “The apple doesn’t fall far
from the tree” is a figurative proverb where the literal meaning is about
apples and a natural phenomenon, where as the actual meaning of the proverb is
about a child grows up to resemble his/her parents.

#### Quality Control.

Finally, we sampled 100 conversational contexts with their answers from each
language. Then, we asked a separate set of native speakers to ensure the data
quality for (1) correct usage of proverb (i.e. the context is correct), and
(2) correct answers for interpreting the meaning. Sometimes, it is possible to
have more than one interpretation of a proverb given the context. We asked the
native speakers to score the answers as correct as long as the answers aligned
with one possible interpretation.

The final dataset consists of 2313 proverbs with conversational context. The
statistics for each language are in Table 1 (with additional data statistics,
in Table 6 in Appendix A). We further split the data for each language into a
test set and a few-shot train-dev set (30 randomly-selected examples each).
Table 2 shows examples from our dataset.

Lang | Code | #Data (Test Size) | Class†
---|---|---|---
English | En | 424 (394) | 5
Chinese | Zh | 364 (334) | 5
German | De | 364 (334) | 5
Russian | Ru | 420 (390) | 4
Bengali | Bn | 370 (340) | 3
Indonesian | Id | 371 (341) | 3

Table 1: Dataset statistics. †: language class identified in Joshi et al.
(2020), where 5 means the language is resource-rich.

###  3.2 Analysis of MAPS

Proverbs and sayings are cultural artifacts and reflect embodied experiences,
which contain diverse concepts often grounded with respect to real-world
objects and experiences.

For instance, dairy products (milk, cheese, yogurt etc.) exist in different
languages but not in Chinese, whereas concepts that are symbolically
meaningful in Chinese culture like dragons or phoenixes exist in the dataset.
To illustrate this, we selected interesting food items and animals from the
final dataset (details in Table 5, Appendix A.2). From the data, we see, for
example, that the tiger is a relatively important concept for Eastern
cultures, whereas the lion is more important for Western cultures.

We further encode the proverbs (without contexts) using a multilingual
sentence embedding LaBSE Feng et al. (2022) and plot the embeddings with
Kernel Density Estimate (KDE) (with dimensionality reduction to two components
using tSNE, van der Maaten and Hinton, 2012) to show the distinctiveness and
connections between proverbs across different languages and cultures (Figure
2), which further illustrates that proverbs and sayings are culturally-
diverse.

From Figure 2, we see that German, English, and Russian share similar proverbs
(parts of their embedding densities overlap). Indonesian proverbs have some
overlap with English, probably due to the use of Latin script. Chinese and
Bengali proverbs are relatively distinct from the Western languages.
Additional details for the KDE plot interpretation are in Appendix A.4.

Lang | Proverb | Context | Choices & Answer
---|---|---|---
Zh |  授人以鱼不如授人以渔 (figurative) |  A: 你可以帮我做这个项目吗？ B: 当然可以，但是我觉得“授人以鱼不如授人以渔”。 (A: Can you help me with this project? B: Of course, but I think "it is better to teach a man fishing than to give him fish".) |  A: B想帮A做项目而不是教A做项目。 (B wants to help A with the project instead of teaching A to do the project.) B: B想教A做项目而不是帮A做项目。 (B wants to teach A to do the project instead of helping A to do the project.) Answer: B
Id |  Nasi sudah menjadi bubur (figurative) |  Orang 1: Bagaimana reaksi bos-mu setelah kamu mengakui kesalahanmu? Orang 2: Kurang baik. Saya sudah mencoba menjelaskan alasan saya berbuat begitu, tetapi saya tetap diberi sangsi. Nasi sudah menjadi bubur. (Person 1: How did your boss react after you admitted your mistake? Person 2: Not good. I’ve tried to explain why I did this, but I’m still being penalized. The rice has become porridge.) |  A: Orang 2 tidak dapat melakukan apapun untuk mengubah reaksi bos. (Person 2 can do nothing to change the boss’s reaction.) B: Orang 2 masih bisa mengubah reaksi atasan. (Person 2 can still change the boss’s reaction.) Answer: A

Table 2: Examples from selected languages (examples for all languages in Table
7, Appendix A.5). Figure 2: Visualizing proverbs embeddings using kernel
density estimation (KDE).

##  4 Experimental Setup

We perform zero-shot evaluations and keep all prompt templates in English, as
previous studies on mLLMs have achieved better performance with English
prompts Lin et al. (2022); Scao et al. (2022); Muennighoff et al.
(2023).888For completeness, we also provide additional baselines using MAPS
for cross-lingual transfer and few-shot evaluation in Appendix C and Appendix
D.5.

#### Models.

We experiment with the following open source state-of-the-art multilingual
models: (1) masked LMs: XLM-R (355m, 3.5B, Conneau et al., 2020); (2)
encoder–decoder LMs: mT0 (580m, 3.7B, 13B, Muennighoff et al., 2023); and (3)
Causal LMs: BLOOMZ (560m, 3B, 7.1B, Muennighoff et al., 2023), and XGLM (564m,
2.9B, 7.5B, Lin et al. (2022)). Most of the models cover all 6 languages in
MAPS except BLOOMZ, which is derived from BLOOM Scao et al. (2022) and covers
only English, Chinese, and Bengali. In addition, although an English model,
LLaMA-2 Touvron et al. (2023) has some multilingual capabilities, and hence we
included two LLaMA-2 models (7B, 13B) in our studies.999While larger models
exist, we chose these models due to computational constraints. We can already
see differences in performance at these model sizes.

#### Memorization Evaluation.

Following previous work in assessing data memorization Magar and Schwartz
(2022); Haviv et al. (2023); Carlini et al. (2023, 2021), we mask out the last
word of each proverb and prompt the mLLMs to complete the proverb with
templates.

For the memorization task, let $t_{i}\in T$ be a prompt template, and let
$q_{j}$ be a proverb with n words where $q_{j}\triangleq\\{w_{1},w_{2}\cdots
w_{n}\\}$. We remove the last word $w_{n}$ for non-MLM models, if the LM
generates (greedily) a string that starts with the missing token, or the
entire proverb is a sub-string of generated string, then we count the model as
having memorized the proverb. For the MLM model, we mask out the last word
with ‘<mask>’ and do predictions (i.e. $w=\arg\max_{w_{n}\in
V}P(w_{n}|T_{i};\hat{q}_{j})$, where $\hat{q}_{j}$ is a proverb with mask
token, and $V$ is the vocabulary).

As the zero-shot prompting results are highly sensitive to the input patterns,
we create 5 different prompt patterns (Table 8, Appendix B), and take the
union of memorized examples among 5 patterns as the memorization accuracy.

#### Reasoning Evaluation.

For the reasoning task, we compute the correct answer by comparing logits of
the two answer candidates (‘A’ or ‘B’) as in Lin et al. (2022). In particular,
we use the prompt template $t^{r}$ for this task (as in Table 3) and compute
$P(t^{r};q_{i};\mlq A\mrq)$ and $P(t^{r};q_{i};\mlq B\mrq)$ and pick the
larger one as the correct answer. For the MLM model, we compare the prediction
logits of the answer candidates.

Both memorization and reasoning experiments are based on the test set.

Template
---
Question: What does the person mean by the proverb? Proverb: <proverb>
Context: <context> Choices: A: <answer 1> B: <answer 2> Answer:

Table 3: Zero-shot testing template, where the coloured part is the template.

(a) Memorization of proverbs in different languages.

(b) Zero-shot results of proverbs understanding with context.

Figure 3: Performance of mLLMs on the proposed MAPS dataset. The number of
parameters is in millions.

##  5 Results and Discussion

###  5.1 Knowledge of Proverbs

— A little knowledge is a dangerous thing.

Since proverbs are fixed expressions, successfully completing a proverb with
greedy decoding means that the model has seen the proverb during pre-training.
While it is possible that the proverbs in training data appears alone without
any contextual usage or explanation, we consider such an occurrence to be
unlikely.101010Webpages such as this
https://en.wiktionary.org/wiki/no_pain,_no_gain exist in the training data for
LLMs. Hence, we make the assumption that memorization of the fixed expression
also correlates with LLMs having embedded knowledge of the usage or meaning.

Figure 3(a) shows the results of proverb memorization, which unsurprisingly as
model size increases, memorization increases. Overall, LLaMA-2 completes the
most proverbs in English and BLOOMZ completes the most proverbs in Chinese.
While XLM-R, XLGM and mT0 cover all the languages in our dataset, they don’t
score particularly well on memorization of proverbs in a single language. All
models exhibit disparities among the memorization across all languages, and
are particularly bad for Indonesian, Bengali, and Russian (i.e. lower-resource
languages). These disparities are probably due to data exposures, as we don’t
find any significant attribution such as well known versus less known, long
versus short, or figurative versus not figurative proverbs by analyzing what’s
memorized.

###  5.2 Reasoning of Proverbs with Conversational Context

— All that glitters is not gold.

While many models embed knowledge about proverbs, it is unclear if
memorization translates to better reasoning with proverbs given context. Next,
we assess model performance on the inference task.

#### Memorization doesn’t indicate the ability to reason with proverbs.

We prompt models with the pattern in Table 3, and plot the accuracy (described
in §4) across languages in Figure 3(b). In general, the bigger the model is,
the better it performs on the inference task (i.e. the ability emerges with
scale).

Overall, the mT0 13B model scores the highest on this task across all
languages, with only a small performance gap across languages (except
Bengali), despite the fact that it memorized very few proverbs for languages
other than English and Chinese. Interestingly, while XGLM and XLM-R scored
well across many languages in the previous experiment, these models only have
chance-level performance on the reasoning task. Similarly, while having
memorized a lot of proverbs in Chinese, BLOOMZ’s performance in reasoning with
proverbs lags behind English and Indonesian.

Since we know which proverbs are memorized from the previous experiments, we
further break down the results into memorized vs. not memorized proverbs for
the 3 best-performing models for English and Chinese (in Table 10, Appendix
D.1). The benefit of memorization shows through for English, but is
inconsistent for Chinese (aligned with observations for other languages from
Figure 3(b)).

A possible explanation of the task not heavily depending on memorization is
that contextual information helps with inference, and the model may learn
other relevant cultural information implicitly from training data.
Consequently, this hints that LLMs may use contextual information rather than
memory retrieval when both are available. However, such hypothesis requires
rigorous study, which we will leave as the future work.

#### Figurative proverbs are difficult to understand in general.

Many proverbs are figurative, hence, we further divide the results of the
model based on this property (described in §3). From Table 4, we observe that
all models perform worse on the inference task when the interpretation is
figurative across English, German and Russian, versus the consistent opposite
observation for Chinese. Bigger models seem to understand Indonesian and
Bengali figurative proverbs better. One conjecture we have is while abstract
reasoning (the kind we have with figurative proverb understanding) can rely on
memorization, but less memorization may lead to better abstract reasoning in
LLMs.

Model | Non-Figurative / Figurative
---|---
| En | Zh | Id | De | Ru | Bn
BLOOMZ 3B | 58.76/57.60 | 53.12/61.97 | 53.33/60.52 | 51.66/47.54 | 52.43/45.13 | 55.88/49.26
BLOOMZ 7.1B | 79.66/68.20 | 66.66/68.30 | 72.00/75.18 | 54.30/53.55 | 52.43/49.55 | 67.64/53.30
mT0 XL (3.7B) | 75.14/62.21 | 62.50/64.08 | 74.67/69.54 | 74.17/61.74 | 73.78/61.94 | 69.12/52.94
mT0 XXL (13B) | 87.01/82.95 | 81.77/83.09 | 84.00/84.96 | 88.74/83.61 | 87.80/76.99 | 63.23/69.85
Llama-2 13B | 81.36/76.50 | 53.12/54.23 | 54.66/58.27 | 72.19/65.03 | 67.07/59.73 | 47.05/49.63

Table 4: Zero-shot accuracy of non-figurative and figurative proverbs (Non-
Fig./Fig.). The gray colour results indicate that the language is not
officially supported by the model. Figure 4: Performance of mLLMs on the
proposed MAPS \- Inference task when asking the ‘negative’ question.

#### Bias towards the correct answer amplify performance gaps across
languages.

Ideally, if the model truly understands what a proverb means in a situational
context, it should be able to pick out the correct answer as well as the wrong
answer, especially for a task with only two choices. Although not the focus of
our work, since this is a fundamental aspect of reasoning Blanco and Moldovan
(2011), we perform the experiments. Several prior work show that negation in
the natural language inference task weakens model performance Hartmann et al.
(2021); Truong et al. (2023); She et al. (2023), here we want to ask a
‘negative’ question rather than provide negative answers. Hence, we change the
question in the prompt template to What does the person not mean by the
proverb?, and keep everything else the same.

The results are in Figure 4. By simply asking the model to pick the wrong
answer, all previous well-performing models flipped their predictions except
mT0. The ‘negative’ question enlarged performance gaps across languages as the
model size increases. Additional results on asking the model to pick the wrong
answer without using the word not are in the Appendix D.2, where we observe
consistent trends of model failures and inverse scaling in many cases. While
we focus on the culture aspect of mLLMs, these results show fundamental work
is needed to improve the ability to handle ‘negative’ questions in current
mLLMs.

###  5.3 Culture Gaps in mLLMs - A Case Study

— When in Rome, do as the Romans do.

An ideal mLLM should perform on texts from all languages and translations in
any directions equally well. However, in our experiments, the performance on
English data is still stronger than other languages for most of the models we
studied. Recently, several works have shown that good performance can be
achieved by translating non-English text data in languages into English
(Conneau and Lample, 2019; Yang et al., 2019, inter alia). Here we demonstrate
that when a task relies on cultural context, there are two distinct
performance gaps in order to achieve true multilingual ability: one is the
language gap (due to mistakes by the translation system, may be fixed by a
perfect translation system), another is the culture gap. To demonstrate this,
we use English and Chinese as the focus of a case study.

#### Machine Translation (MT).

We translate Chinese data to English using Google translate (Zh-En). By
closely examining the translated data, current translation systems do not
handle cultural context (i.e. proverbs) well, such as incomplete translations
or wrong translations. For example, a polysemous phrase 大三 was translated to
“junior” (third year university student), but in a specific proverbial
context, it means someone is “three years older”.

#### Human-Adapted Translation (HT).

Next, we perform several adaptations to the machine translated context: 1)
manually fix the literal translation of the proverb if there are any mistakes,
fix the grammatical mistakes in the contexts and answers; 2) a light
adaptation of the translated data inspired by Majewska et al. (2023), by
replacing the names and locations in the dataset to align with the culture
(e.g. XiaoMing to Michael etc.) in case models are confused about whether an
entity is a person or a place. This is our best effort-adaptation to remove
the language gap.

Next, we perform zero-shot evaluation with the best-performing multilingual
models (mT0-XXL, 13B) and English model (Llama-2 13B) for Zh-En (in Figure 5).
In fact, both models show a performance gap in the translated data in
comparison to the target language. Interestingly, mT0 also show a performance
degradation comparing to the inference results in the original language
(Llama-2 is near chance level for Zh, improvement is not surprising). In all
cases, HT improves over MT, which the gain can be considered as the language
gap. More interestingly we demonstrate the gap between HT and the $\max$ of
source and target language is the culture gap in mLLMs, i.e. culture gap =
$|\textrm{Acc}^{HT}-\max(\textrm{Acc}^{Src},\,\textrm{Acc}^{Tgt})|$. The
culture gap for Zh-En is 5.73 for mT0 and 19.40 for Llama-2.111111We also
perform the same experiment in reverse direction En-Zh with mT0 (Appendix
D.3), similar results were observed. Other evaluation results on machine
translated data for other languages are in Appendix D.3. In an ideal
situation, these gaps should be 0, indicating that the model is culturally
aware and able to understand a language if speakers are from diverse cultural
backgrounds. These results suggest that additional research is needed to
improve cultural-awareness and the inclusion of cultural priors in MT models
and mLLMs Yao et al. (2023); Shaikh et al. (2023).

Figure 5: Performance gap between machine translated, human translated data
and results in the original source language (Zh), and target language (En).

##  6 Conclusion

In this work, we present an investigation of mLLMs’ ability to reason with
cultural common ground. We use proverbs and sayings from different languages
and cultures as an investigative tool. Specifically, we study a variety of
mLLMs on their ability to memorize proverbs, reason with proverbs and sayings
in a situational setting, and understand cross-cultural communications with
proverbs.

To aid the investigation, we created a multicultural proverbs and sayings
dataset MAPS. Our analysis shows that many models possess knowledge of
proverbs and sayings, however, knowing proverbs does not mean the model is
able to reason with proverbs in contextual settings. Indeed, we found that
some models did show culturally-diverse reasoning ability, it is only to a
very limited degree. We also found that the ability to reason in zero-shot
emerges with model scale, but the ability to understand a ‘negative’ question
is inverse with the model scale. The disparities in culturally-diverse
reasoning ability between languages grows with the model size, which raises
concerns in terms of multilingual availability and points to the need for more
robust mLLMs. Finally, we defined and observed several culture gaps in cross-
lingual communications. We hope to explore different aspects of cultural
common ground in the future.

##  7 Limitations

Our work uses proverbs and sayings as a proxy for cultural common ground, and
we explore mLLMs’ ability in understanding cultural common grounds in a
limited setting. One potential limitation with our experiments is the
evaluation data is relatively small compared to many automatically generated
benchmarks and may introduce lexical biases. However, these are not major
concerns as 1) we want to focus on cultural common ground, which automatically
limit us to a subset of lexical items; 2) to our best knowledge, this is the
largest proverbs dataset for reasoning, and there is enough signal to
distinguish between the tested models and uncover insights on current mLLMs
ability and limitations in understanding proverbs and sayings. We hope to
explore aspect of culture beyond proverbs and sayings, and with a more diverse
set of languages in the future.

In this work, we evaluate models of size up to 13B parameters (the biggest
available size of mT0) due to computational constraints. However, full
evaluation of larger models or task-specific models is necessary, especially
when asking ‘negative’ questions and assessing the culture gaps in the future.
Moreover, we focus on studying open-source LLMs in this paper for scientific
reproducibility, and closed-source LLM evaluations are beyond our scope. As
our dataset is open source, they can be used to evaluate closed-source LLMs in
the future.

##  8 Acknowledgement

This work was funded by the German Federal Ministry of Education and Research
(BMBF) under the promotional reference 13N15897 (MISRIK), and the LOEWE
initiative (Hesse, Germany) within the emergenCITY center. We would like to
thank Sukannya Purkayastha, Aniket Pramanick, Ilia Kuznetsov, Kexin Wang, Luke
Bates for their constructive feedback and discussions of the work. We thank
Sukannya Purkayastha, Jonathan Tonglet, and Yongxin Huang for their
suggestions on a draft of the paper.

## References

  * bench authors (2023) BIG bench authors. 2023.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_.
  * Blanco and Moldovan (2011) Eduardo Blanco and Dan Moldovan. 2011.  Semantic representation of negation using focus detection.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 581–589, Portland, Oregon, USA. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Carlini et al. (2023) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023.  Quantifying memorization across neural language models.  In _The Eleventh International Conference on Learning Representations_.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021.  Extracting training data from large language models.  In _30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021_ , pages 2633–2650. USENIX Association.
  * Chakrabarty et al. (2022) Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022.  FLUTE: Figurative language understanding through textual explanations.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 7139–7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  _Advances in neural information processing systems_ , 32.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022\.  Language-agnostic BERT sentence embedding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 878–891, Dublin, Ireland. Association for Computational Linguistics.
  * Ghosh and Srivastava (2022) Sayan Ghosh and Shashank Srivastava. 2022.  ePiC: Employing proverbs in context as a benchmark for abstract language understanding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3989–4004, Dublin, Ireland. Association for Computational Linguistics.
  * Hartmann et al. (2021) Mareike Hartmann, Miryam de Lhoneux, Daniel Hershcovich, Yova Kementchedjhieva, Lukas Nielsen, Chen Qiu, and Anders Søgaard. 2021.  A multilingual benchmark for probing negation-awareness with minimal pairs.  In _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 244–257, Online. Association for Computational Linguistics.
  * Haviv et al. (2023) Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2023\.  Understanding transformer memorization recall through idioms.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 248–264, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022.  Challenges and strategies in cross-cultural NLP.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics.
  * Honeck (2013) Richard P Honeck. 2013.  _A proverb in mind: The cognitive science of proverbial wit and wisdom_.  Psychology Press.
  * Hovy and Yang (2021) Dirk Hovy and Diyi Yang. 2021.  The importance of modeling social factors of language: Theory and practice.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 588–602, Online. Association for Computational Linguistics.
  * Hu et al. (2023) Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023.  A fine-grained comparison of pragmatic language understanding in humans and language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4194–4213, Toronto, Canada. Association for Computational Linguistics.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kabra et al. (2023) Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, and Graham Neubig. 2023.  Multi-lingual and multi-cultural figurative language understanding.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8269–8284, Toronto, Canada. Association for Computational Linguistics.
  * Kramsch (2014) Claire Kramsch. 2014.  Language and culture.  _AILA review_ , 27(1):30–55.
  * Li et al. (2023a) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023a.  Bactrian-x : A multilingual replicable instruction-following model with low-rank adaptation.  _CoRR_ , abs/2305.15011.
  * Li et al. (2023b) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023b.  CMMLU: measuring massive multitask language understanding in chinese.  _CoRR_ , abs/2306.09212.
  * Lin et al. (2022) Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. 2022.  Few-shot learning with multilingual generative language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , pages 9019–9052. Association for Computational Linguistics.
  * Liu et al. (2023) Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2023.  We’re afraid language models aren’t modeling ambiguity.  _CoRR_ , abs/2304.14399.
  * Liu et al. (2022) Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022.  Testing the ability of language models to interpret figurative language.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4437–4452, Seattle, United States. Association for Computational Linguistics.
  * Liu et al. (2021) Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021.  Visually grounded reasoning across languages and cultures.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Magar and Schwartz (2022) Inbal Magar and Roy Schwartz. 2022.  Data contamination: From memorization to exploitation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 157–165, Dublin, Ireland. Association for Computational Linguistics.
  * Majewska et al. (2023) Olga Majewska, Evgeniia Razumovskaia, Edoardo M. Ponti, Ivan Vulić, and Anna Korhonen. 2023.  Cross-lingual dialogue dataset creation via outline-based generation.  _Transactions of the Association for Computational Linguistics_ , 11:139–156.
  * Mieder (2004) Wolfgang Mieder. 2004.  _Proverbs: A handbook_.  Greenwood Publishing Group.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023\.  Crosslingual generalization through multitask finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  * Ouyang et al. (2022a) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022a.  Training language models to follow instructions with human feedback.  In _NeurIPS_.
  * Ouyang et al. (2022b) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022b.  Training language models to follow instructions with human feedback.  In _NeurIPS_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI_.
  * Ruis et al. (2022) Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2022.  Large language models are not zero-shot communicators.  _CoRR_ , abs/2210.14986.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. 2022.  BLOOM: A 176b-parameter open-access multilingual language model.  _CoRR_ , abs/2211.05100.
  * Shaikh et al. (2023) Omar Shaikh, Caleb Ziems, William Held, Aryan Pariani, Fred Morstatter, and Diyi Yang. 2023.  Modeling cross-cultural pragmatic inference with codenames duet.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 6550–6569, Toronto, Canada. Association for Computational Linguistics.
  * She et al. (2023) Jingyuan S. She, Christopher Potts, Samuel R. Bowman, and Atticus Geiger. 2023.  ScoNe: Benchmarking negation reasoning in language models with fine-tuning and in-context learning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1803–1821, Toronto, Canada. Association for Computational Linguistics.
  * Thomas (1983) Jenny Thomas. 1983.  Cross-cultural pragmatic failure.  _Applied linguistics_ , 4(2):91–112.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288.
  * Truong et al. (2023) Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023.  Language models are not naysayers: an analysis of language models on negation benchmarks.  In _Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)_ , pages 101–114, Toronto, Canada. Association for Computational Linguistics.
  * van der Maaten and Hinton (2012) Laurens van der Maaten and Geoffrey E. Hinton. 2012.  Visualizing non-metric similarities in multiple maps.  _Mach. Learn._ , 87(1):33–55.
  * White (1987) Geoffrey M White. 1987.  _Proverbs and cultural models: An American psychology of problem solving._ Cambridge University Press.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
  * Yao et al. (2023) Binwei Yao, Ming Jiang, Diyi Yang, and Junjie Hu. 2023.  Empowering llm-based machine translation with cultural awareness.  _CoRR_ , abs/2305.14328.

##  Appendix A Dataset

###  A.1 Annotations

We recruit crowd annotators through Prolific121212http://prolific.com/ with
the requirement of corresponding language as their first language, and fluent
in English. Expert annotators are Master’s, PhD and Post-doc researchers,
including authors of this paper. The annotation process is illustrated in
Figure 6.

Instructions to create the conversational context:

Step 1: Check if the proverb is used correctly in the conversation. Note:
Sometimes, the proverb is figurative, meaning that the underlying meaning and
the literal meaning of the proverb are different! The conversation should fits
the figurative usage/meaning of the proverb.
Example: Person 1: "I’m scared of my boss." Person 2: "Well, barking dogs
seldom bite." "Barking dogs seldom bite" -It has a literal meaning of dogs
that bark rarely taking actions and bite you, so you don’t need to be afraid
of getting hurt. The proverb metaphorically describes people that threaten you
a lot rarely take actions and harm you. Although this conversation maybe
missing some contexts, it should be labelled as correct.
Example: Person 1: "My dog is barking." Person 2: "Well, barking dogs seldom
bite." The proverb is used in a literal way, when it has a figurative meaning.
This should be labelled as wrong. Step 2: Re-write the conversation if the
proverb is not used correctly from step 1. The conversation should be 1-turn
(1 round between 2 people), and maximum 2-turn (2 rounds between 2 people).
Note: Please do not produce a conversation where one person is asking about
the meaning of the proverb.

Instructions to create the answers:

What does the person mean? • Identify the person that used the proverb in the
conversation. • Write down a short sentence in the OPT1 column, state what the
person mean by the proverb in this conversation. • Write down a negative of
OPT1 in the OPT2 column.

Figure 6: The data annotation process of MAPS.

###  A.2 Animal and Food Terms in the Dataset

Table 5 shows selected animal and food concepts across different languages.
From the data, we could see proverbs naturally contain culturally important
concepts. For examples, we could see the tiger is a relatively important
concept for Eastern cultures, whereas the lion is more important for Western
cultures; while bread is enjoyed by many people around the world, rice is
culturally more important in the East, etc.

---

Table 5: Selected food and animal concepts from the proverbs.

###  A.3 Additional Data Statistics

We include additional dataset statistics in Table 6. To calculate the average
tokens in the context for Chinese, we take each character as a word.

Lang | Avg Tok in Context | Avg Turns
---|---|---
English | 28.41 | 1.18
Chinese | 31.30 | 1.14
German | 27.91 | 1.12
Indonesian | 25.35 | 1.15
Russian | 31.25 | 1.47
Bengali | 35.16 | 1.63

Table 6: Additional dataset statistics: average number of tokens in the
context, and average turns in the context.

###  A.4 Interpreting the KDE Plot

For better comparison, we produce the Kernel Density Estimate (KDE) plot of
400 randomly sampled sentences each language (2400 sentences in total), from a
parallel multilingual dataset Li et al. (2023a) in Figure 7. As the original
data is much larger (67k sentences per language), sub-sampled sentences are
likely not translations of each other, but rather topic coherent.

When sentences are topic coherent, their embeddings overlapping on top of each
others and inseparable (Figure 7). In comparison with the KDE plot of proverb
embeddings (Figure 2), we can clearly see the difference of proverbs across
languages.

Figure 7: Visualizing embeddings with Kernel Density Estimate (KDE) when the
sentences are sampled from a parallel dataset (topic coherent across
languages).

###  A.5 Data Examples

We balance the labels in MAPS and we show example data for all languages in
Table 7.

---

Table 7: Examples for all six languages from MAPS.

##  Appendix B Templates

We use Generate a very short 1-turn dialogue ends with “proverb” in language
as the template to query GPT3.5 (gpt-3.5-turbo-0301) for the seed
conversational data. The model does not strictly generate seed conversation
with 1-turn. We also experimented with translated template and did not observe
quality improvements for our task.

Table 8 contains all the templates we used in our memorization experiments. As
the prompting results are highly variable based on the input patterns, we
created five different prompt patterns. We take the union of memorized
examples among 5 patterns as the memorization accuracy.

Templates
---
1\. Proverb: no pain, no
2\. Complete this proverb: no pain, no
3\. Finish the proverb: no pain, no
4\. What’s the last word of this proverb: no pain, no
5\. What’s missing at the end of this proverb:
no pain, no

Table 8: Memorization templates, and the coloured part is the template.

##  Appendix C Cross-lingual Transfers Baselines

For completeness, we provide cross-lingual transfer baselines on MAPS. For
cross-lingual transfer baselines, we re-split the English dataset into train
and test set (274/150 data point each), and evaluate on the test set for other
languages (i.e. zero-shot). We randomly sampled 20 data points from the
training set as validation. We formulate the task as binary classification and
experimented with XLM-R-Base (125m)/XLM-R-Large (355m)/XLM-R-XL (3.5B) and
mT0-Base (580m)/mT0-Large (1.2B)/mT0-XL (3.7B).

The data input format is: Context: {context} Choices: A: {answer 1} B: {answer
2}.

We use AdamW optimizer Loshchilov and Hutter (2019) and conduct hyperparameter
search of learning rate of [5e-5, 1e-4, 1e-5] and batch size of [8, 10, 16],
trained for 30 epochs with bfloat16 precision, on a single A100 GPU.

The zero-shot transfer results are in Table 9 and averaged over 4 random
seeds. The final hyperparameters for all models are [lr=1e-4, batch size=10],
except for mT0-Large, which is [lr=1e-4, batch size=8]. Following previous
work, we also include results for the translate-test baselines Conneau et al.
(2018) in Table 9.

Similar to our findings in the main paper, the model does not perform well on
the task with models under billion parameters. The performance gap between
English and other languages remains significant.

Model | En | De | Zh | Ru | Id | Bn | Cross-lingual Avg
---|---|---|---|---|---|---|---
XLM-R-Base (125m) | 52.06 | 50.00 | 50.07 | 50.19 | 50.37 | 50.22 | 50.17
XLM-R-Large (355m) | 49.85 | 50.00 | 50.07 | 50.00 | 49.93 | 50.00 | 50.00
XLM-R-XL (3.5B) | 58.38 | 53.67 | 52.25 | 53.65 | 52.79 | 53.01 | 53.07
mT0-Base (580m) | 60.74 | 55.01 | 52.02 | 50.77 | 50.29 | 53.75 | 52.37
mT0-Large (1.2B) | 65.00 | 56.89 | 56.59 | 53.53 | 50.44 | 55.59 | 54.61
mT0-XL (3.7B) | 72.65 | 67.51 | 60.63 | 61.54 | 60.26 | 53.82 | 60.75
Translate-test |  |  |  |  |  |  |
XLM-R-Base (125m) | - | 50.60 | 50.75 | 49.23 | 51.47 | 49.85 | 50.38
XLM-R-Large (355m) | - | 50.00 | 50.00 | 50.00 | 49.85 | 50.00 | 49.97
XLM-R-XL (3.5B) | - | 50.90 | 51.20 | 52.31 | 49.85 | 51.47 | 51.15
mT0-Base (580m) | - | 51.80 | 51.05 | 51.15 | 49.56 | 54.26 | 51.56
mT0-Large (1.2B) | - | 54.04 | 55.09 | 54.62 | 53.67 | 57.21 | 54.93
mT0-XL (3.7B) | - | 67.96 | 62.72 | 63.46 | 57.92 | 58.68 | 62.15

Table 9: Zero-shot cross-lingual transfers and translate-test baselines.
Cross-lingual averages are calculated over all languages except English.

##  Appendix D Additional Results

###  D.1 Memorized versus Not Memorized

We break down the results into memorized group versus not memorized group for
the three best performing models. We only show results when there are more
than 50 proverbs in a group in Table 10 (which left us with English and
Chinese). The benefit of memorization only shows for English, but not for
Chinese.

| En | Zh
---|---|---
Model |  $\in$Mem. |  $\notin$Mem. |  $\in$Mem. |  $\notin$Mem.
BLOOMZ 7.1B | 77.23 | 65.07 | - | -
mT0-XXL (13B) | 86.17 | 84.33 | 81.48 | 82.50
Llama-2 13B | 80.30 | 75.38 | 54.65 | 53.22

Table 10: Result on memorized versus not memorized proverbs on 3 best
performing models for English and Chinese. Results omitted due to less than 50
proverbs in the not memorized group.

###  D.2 ‘Negative’ Questions.

We experimented with 4 additional versions of ‘negative’ questions /
instructions (randomly created), without the use of the word ‘not’, they are:

  * •

Which answer is contrary to what the person means by the proverb?

  * •

Which answer is impossible as the interpretation of what the person means by
the proverb?

  * •

Pick the opposite answer to what the person means by the proverb.

  * •

Pick the wrong answer to what the person means by the proverb.

We use the same prompt template to evaluate the models. The results are in
Figure 8. While our work focus on reasoning with cultural common grounds, this
shows the importance and urgent need to improve model’s ability in answering
‘negative’ questions.

(a) Results using Which answer is contrary to what the person means by the
proverb?.

(b) Results using Which answer is impossible as the interpretation of what the
person means by the proverb?.

(c) Results using Pick the opposite answer to what the person means by the
proverb.

(d) Results using Pick the wrong answer to what the person means by the
proverb.

Figure 8: Performance of mLLMs on the proposed MAPS dataset when asking the
model a ‘negative’ question.

###  D.3 Culture Gaps

In addition to the results in §5.3, we follow the same procedure and perform
the experiment with mT0 for En-Zh translated data. We observe similar results
in Figure 9, and the culture gap for En-Zh is 5.33.

Figure 9: Performance gap between machine translated, human translated English
data and results in the original source language (En), and target language
(Zh).

###  D.4 More Results on Llama-2 with Translations

Since Llama-2 13B is one of the recent state-of-the-art (English officially)
models, we further conducted an zero-shot experiment by translating all date
from other languages into English. We use Google Translate for translation and
reported the results in Table 11. From the Table, we can see significant
performance gaps (to English).

Lang | Ori. Lang | MT | $\Delta_{En}$
---|---|---|---
En | 78.68 | - | -
De | 68.26 | 73.35 | 5.33
Ru | 62.82 | 71.02 | 7.66
Id | 57.47 | 69.79 | 8.89
Bn | 49.11 | 61.76 | 16.92
Zh | 53.59 | 54.19 | 24.49

Table 11: Results of machine translated data with Llama-2 13B. $\Delta_{En}$
is the result gap to model’s performance on English data.

###  D.5 Few-shot (In-context) Evaluation

For completeness we also provide evaluation results with few-shot
demonstrations. We perform 2-shot and 5-shot experiments by randomly sample 5
sets of n-shot demonstrations from the few-shot training set (using the same
template as zero-shot evaluation by concatenation). We evaluate on BLOOMZ
7.1B, mT0-XXL 13B and Llama-2 13B models, and Table 12 shows the results.

From Table 12, we do not observe any improvements with few-shot demonstrations
comparing to zero-shot. In fact, model performances consistently degrade with
more demonstrations. Since our task has very long context that may affects the
n-shot performance. Nonetheless, this degradation has been observed recently
in other work such as in Li et al. (2023b) with few-shot evaluations.

Model | En | De | Zh | Ru | Id | Bn | Cross-lingual Avg
---|---|---|---|---|---|---|---
BLOOMZ 7.1B 2-shot | 59.49 | 61.55 | 56.59 | 53.77 | 51.53 | 50.00 | 52.65
BLOOMZ 7.1B 5-shot | 51.57 | 52.39 | 50.85 | 50.35 | 50.25 | 50.52 | 50.30
mT0-XXL 13B 2-shot | 78.37 | 72.63 | 76.95 | 78.74 | 74.87 | 63.82 | 76.81
mT0-XXL 13B 5-shot | 68.48 | 67.90 | 70.38 | 71.50 | 67.64 | 60.00 | 69.57
Llama-2 13B 2-shot | 74.87 | 56.52 | 55.42 | 60.77 | 56.76 | 51.00 | 58.77
Llama-2 13B 5-shot | 64.16 | 52.69 | 54.89 | 55.56 | 52.71 | 50.17 | 54.14

Table 12: Few-shot evaluation results from MAPS. Cross-lingual averages are
calculated over all languages except English.


#  Historia Magistra Vitae: Dynamic Topic Modeling of Roman Literature using
Neural Embeddings

###### Abstract

Dynamic topic models have been proposed as a tool for historical analysis, but
traditional approaches have had limited usefulness, being difficult to
configure, interpret, and evaluate. In this work, we experiment with a recent
approach for dynamic topic modeling using BERT embeddings. We compare topic
models built using traditional statistical models (LDA and NMF) and the BERT-
based model, modeling topics over the entire surviving corpus of Roman
literature. We find that while quantitative metrics prefer statistical models,
qualitative evaluation finds better insights from the neural model.
Furthermore, the neural topic model is less sensitive to hyperparameter
configuration and thus may make dynamic topic modeling more viable for
historical researchers.

Keywords: dynamic topic modeling, Latin

Historia Magistra Vitae: Dynamic Topic Modeling of Roman Literature using
Neural Embeddings

Michael Ginn and Mans Hulden
---
University of Colorado
{michael.ginn<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Figure 1: LDA, NMF, and BERTopic topic clusters

Topic models, which model the distribution of how topics appear within
documents, have long been proposed as tools for computational social science
and history (Blei et al., 2003; Blei and Lafferty, 2006a). In particular,
dynamic topic models (Blei and Lafferty, 2006b) can be used to represent a
distribution of topics that changes over time. However, experts have been
critical of the practicality of topic models for research, noting the
difficulty in quantitative evaluation (Chang et al., 2009), handling documents
with figurative language (Rhody, 2012), and the difficulty in distinguishing
novel, meaningful insights from noise (Goldstone and Underwood, 2014).

Traditionally, topic models have used a Latent Dirichlet Allocation (LDA), a
Bayesian statistical model that explains observed text using unseen groups of
documents (Blei et al., 2003). Recent research has proposed an alternate
strategy for topic modeling, where neural embeddings are used to create topic
clusters (Grootendorst, 2022). In this work, we compare the effectiveness of
this neural approach with traditional statistical techniques, hypothesizing
that this approach will be more robust to noise and reduce the effort needed
to create useful topic models.

Due to the cultural and political dominance of the Ancient Romans, there
exists an extensive surviving corpus of Roman literature, with roughly 220,000
surviving texts and inscriptions (Clackson, 2011). Many of these texts are
individually well-studied, but holistic analysis of the corpus remains
difficult. We experiment with various techniques for dynamic topic modeling
over this entire corpus, which is noisy and broad in scope, to determine which
approach performs best at producing useful results, while requiring minimal
hyperparameter tuning.

We make the following contributions:

  1. 1.

We produce the first (to our knowledge) dynamic topic model spanning all
surviving Roman texts, and create visualizations demonstrating clear
historical trends.

  2. 2.

We find that the neural embedding approach produces a more readily
interpretable topic distribution compared to traditional LDA approaches.

  3. 3.

We demonstrate that quantitative metrics for evaluating topic models do not
necessarily align with human judgments, and better strategies for evaluating
topic models are needed.

Our code is available on GitHub.111link omitted for anonymity

## 2\. Related Work

Dynamic topic modeling has been used across disciplines to explore the
evolution of bodies of literature such as nineteenth-century British
Parliament debates (Guldi, 2019), medical publications about sepsis (Bostwick
et al., 2018), and Twitter data during the COVID-19 pandemic (Bogdanowicz and
Guan, 2022). However, such works (and the majority of similar ones) use the
traditional LDA model for dynamic topic modeling, while we utilize a modern
neural clustering technique. Additionally, to our knowledge, this is the first
work to apply dynamic topic models across the entire corpus of a historical
state.

There has been growing interest in NLP approaches for ancient languages
including Latin, evidenced by workshops such as LT4HALA (Sprugnoli and
Passarotti, 2022). Previous work in Latin NLP has included language modeling
(Bamman and Burns, 2020); lemmatization, part-of-speech tagging, and feature
identification (Sprugnoli et al., 2022); and automatic dating of Latin texts
(Allen, 2023).

## 3\. Data

We use the Latin Library corpus as our source for Latin texts (Carey, ). We
date documents according to the information gathered in (Allen, 2023),
although many of these dates are estimated years based on historical
knowledge. After removing documents that did not have known dates, the corpus
consisted of 1,350 plain-text documents, including histories, poems, plays,
inscriptions, and speeches. Documents spanned from roughly 449 BC (The Twelve
Tables) through AD 600 (The Epistles of Gregory the Great), which roughly
covers the time span from the founding of Rome to its fall.222We chose not to
include the majority of post-Roman literature, which includes many Latin texts
but is often considered as a distinct corpus by classical scholars. For
authors with many small texts (poems and inscriptions), multiple texts were
combined in a single document, whereas very long texts (such as the Aeneid)
were split into several documents.

As Latin is a highly inflectional language, preprocessing is essential to
extract meaningful topic words. We tokenized, lemmatized, and removed stop
words using the Classical Language Toolkit (CLTK) (Johnson et al., 2021).

## 4\. Methodology

We trained and compared three different models for dynamic topic modeling: a
Latent Dirichlet Allocation (LDA) model, as in Blei and Lafferty (2006b); a
model based on Non-negative Matrix Factorization (NMF), and a neural,
transformer-based model using BERTopic (Grootendorst, 2022).333Implementation
details and hyperparameters will be provided in an appendix in the final
version.

### 4.1. LDA Model

The LDA model follows the approach used in Blei and Lafferty (2006b) closely,
and uses Gensim444https://radimrehurek.com/gensim/ (Řehůřek and Sojka, 2010)
for implementation.

The LDA dynamic topic model extends a static topic model, modeling each time
slice as a static topic model. The parameters for the distribution at each
time step are modeled with a state space distribution, capturing how the
distribution of topics evolves over time. We separated documents into ten
timesteps, finding through qualitative observation that this was the most
effective setting across models.

### 4.2. NMF Model

Non-negative Matrix Factorization (NMF) is a linear algebra technique often
used for extracting latent information. We reimplement the approach used in
Greene and Cross (2017) for dynamic topic modeling. In this approach, we run
NMF on each timestep, factoring the matrix of documents and words into a
matrix of documents to topics and topics to terms. Then, we run NMF again,
using the terms for the topics at each timestep as the "documents".

### 4.3. BERTopic Model

BERTopic (Grootendorst, 2022) uses neural embeddings of documents to create
clusters of related documents, from which topics are extracted. For dynamic
topic modeling, BERTopic uses the static topic model as a global distribution.
We compute a topic distribution using the same topics for each time step and
fine-tune this representation by averaging each time step globally and
evolutionarily (with the previous time step).

### 4.4. Evaluation

Following the approach used in O’Callaghan et al. (2015), we use several
quantitative metrics to evaluate topic models. The TC-Embed metric scores
topic coherence by taking the average cosine similarity between embeddings for
the topic ten words in a topic. The Mean Pairwise Jaccard similarity measure
scores topic generality by taking the average ratio of shared terms to total
terms for a pair of topics. A good topic model is thought to have high
coherence and low generality.

## 5\. Results

Model | TC-Embed | MPJ
---|---|---
LDA | 0.209 | 0.016
NMF | 0.307 | 0.369
BERTopic | 0.189 | 0.065
Table 1: Topic Coherence (TC-Embed) and Generality (MPJ) Measures, averaged
over top-5 topics

The coherence and generality scores are given in Table 1, showing that the LDA
model minimalizes generality best, while the NMF model has the highest topic
coherence, while the BERTopic model underperforms in both. However, prior
research has suggested that these quantitative metrics often do not line up
with human intuition about the quality of the model, particularly for dynamic
topic models. Thus, we also evaluate the usefulness of the topic distributions
from each model. 555The full list of terms for each topic will be provided in
an appendix in the final version

### 5.1. Intertopic Distance Map

First, we analyzed the distribution of topics regardless of time. We use
LDAvis and principal component analysis (PCA) to create a graph showing the
distribution of topics for the two static topic models (Sievert and Shirley,
2014), presented in Figure 1.

### 5.2. Topics over time

We calculate the frequency of topics at each time step by calculating the
number of documents with each topic as its most likely topic and visualize the
popularity of topics over time. These results are presented in Figure 2.

Figure 2: LDA, NMF, and BERTopic topic frequency (# documents) over time

## 6\. Discussion

Though the LDA and NMF models outperform based on quantitative metrics, we can
see in Figure 1 and Figure 2 that their insights were limited, particularly
for the LDA model.

The LDA intertopic map shows that Topic 1 and its cluster of topics were very
general and dominant, and as a result Figure 2 does not reveal much of
interest about the topics’ frequency over time. The LDA model had difficulty
distinguishing meaningful topic words from noise. We experimented with
modifying hyperparameters (prior probabilities, chain variance) but were
unable to make improvements.

The NMF model produced a wider range of topics, spread out and changing over
time. However, most of the topic terms are difficult to interpret as
meaningful, and the generality score indicates that the topics are mainly
composed of common Latin terms. One promising topic is Topic 5, with terms
such as ignis ("fire") and magnus ("great"), which peaks around AD 100 and
could be related to the Great Fire of Rome in AD 64.

The results for the BERTopic model are insightful and useful, despite the
quantitative measures, and the model requires minimal configuration. The
intertopic map (Figure 1) shows a spread of evenly distributed topics, and the
topics over time graph (Figure 2) demonstrates clear trends in topic
popularity, described below. BERTopic benefits from utilizing embedded
document representations to create meaningful clusters of any shape, while the
LDA model considers all documents equally.

### 6.1. Topic 2: Christianity in Rome

One particularly clear and insightful topic is topic 1, which includes terms
such as deus ("god"), homo ("man"), christus ("Christ"), and spiritus
("breath, spirit"), and is clearly associated with Christian religious
writings. The topic peaks twice: once around A.D. 200, after Christianity was
first established and introduced in Rome, and again around A.D. 400, after the
Edict of Milan and the establishment of Christianity as the official religion
of the Roman Empire.

### 6.2. Topic 3: The fall of the Republic

Another interesting topic is topic 3, including terms such as res publicus
("republic"), populus ("the people"), and romanus ("Roman"). This topic seems
to be associated with writings and speeches about Republican values such as
Cicero’s orations. We can see that the topic is most popular around 100 B.C.
and decreases in popularity through A.D. 300, likely a reflection of the fall
of the Roman Republic (44 B.C.) and its associated republican ideals.

### 6.3. Comparison

The results from the LDA and NMF models were consistent with many of the
criticisms from humanities and social science experts: the models did not
extract particularly useful information, and the quantitative metrics did not
align with human judgments. We believe that with extensive hyperparameter
search, we could achieve comparable results to the BERTopic model, but this
effort may not be worth the resulting value to a historian.

In contrast, the BERTopic model mitigates many of these issues, with a more
robust method for clustering documents that produces insights with minimal
effort. However, it does require an existing model for creating document
embeddings, which may not always be available for many low-resource languages.
Future work could explore using a state space to model distributional
parameters over time while incorporating neural embeddings to guide topic
extraction.

## 7\. Conclusion

In this work, we trained three different dynamic topic models over documents
spanning the lifetime of the Roman Republic and Roman Empire. We find that the
traditional LDA and NMF models suffer from common issues with topic modeling:
they may be difficult to interpret in a useful way, they are sensitive to
noisy data, and they often require parameter search to find optimal
configurations. In contrast, the BERTopic model is able to achieve insightful
results with minimal effort, producing trends that line up with historical
intuitions. We believe this topic model could be useful in certain scenarios;
for example, a Roman historian might be interested in how a particular
author’s distribution of topics relates to the overall historical trends.

While this approach shows clear potential in the use of topic models for
historical analysis, it does not solve all of the issues. Interpretation is
still critical and difficult, and there is something of a paradox: it is
difficult to tell if a trend in a topic model is meaningful and accurate,
without already knowing that the trend exists. Nevertheless, we hope that
newer approaches can make topic modeling more robust and performant as one of
many tools for discovering insights and performing research.

## 8\. Ethics

We hope that through more sophisticated methods for topic modeling, we can aid
historians and social scientists in their research. However, topic modeling
and other computational approaches for humanities should not supplant human
experts, who are able to investigate research questions with more care and
nuance than an automated method. Topic models, and other forms of statistical
text summarization, tend to extract majority ideas and topics. Topic models
should be used to gain holistic information about trends and topics, but
should not be used to overlook minority opinions.

This work uses Latin text which is now in the public domain, but any work
utilizing data for living languages should ensure they have proper permission
to use the data, especially with indigenous and endangered languages.
Communities should always have control over the data they produce, and
language should be seen as a cultural and personal artifact, not merely data
for conducting research.

Finally, models that require high computational costs, particularly neural
models, use large amounts of energy and carry an environmental cost (Strubell
et al., 2020).

## 9\. Bibliographical References

##

  * Allen (2023) Mitchell Allen. 2023.  Dlt2.  https://github.com/Sittch/DLT2.
  * Bamman and Burns (2020) David Bamman and Patrick J. Burns. 2020.  Latin BERT: A contextual language model for classical philology.
  * Blei and Lafferty (2006a) David Blei and John Lafferty. 2006a.  Correlated topic models.  _Advances in neural information processing systems_ , 18:147.
  * Blei and Lafferty (2006b) David M. Blei and John D. Lafferty. 2006b.  Dynamic topic models.  In _Proceedings of the 23rd international conference on Machine learning_ , pages 113–120.
  * Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003.  Latent Dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Bogdanowicz and Guan (2022) Alexander Bogdanowicz and Chenghe Guan. 2022.  Dynamic topic modeling of twitter data during the COVID-19 pandemic.  _PLoS ONE_ , 17.
  * Bostwick et al. (2018) A Doran Bostwick, Kelly S. Peterson, Barbara E. Jones, Robert Paine, Matthew C. Samore, and Makoto M. Jones. 2018.  2148\. 100 years of sepsis: Using topic modeling to understand historical themes surrounding sepsis.  _Open Forum Infectious Diseases_ , 5:S632 – S632.
  * Chang et al. (2009) Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. 2009\.  Reading Tea Leaves: How Humans Interpret Topic Models.  In _Advances in Neural Information Processing Systems_ , volume 22. Curran Associates, Inc.
  * Clackson (2011) James Clackson. 2011.  Classical Latin.  _A companion to the Latin language_ , pages 236–256.
  * Goldstone and Underwood (2014) Andrew Goldstone and Ted Underwood. 2014.  The quiet transformations of literary studies: What thirteen thousand scholars could tell us.  _New Literary History_ , 45(3):359–384.
  * Greene and Cross (2017) Derek Greene and James P. Cross. 2017.  Exploring the Political Agenda of the European Parliament Using a Dynamic Topic Modeling Approach.  _Political Analysis_ , 25(1):77–94.
  * Grootendorst (2022) Maarten Grootendorst. 2022.  BERTopic: Neural topic modeling with a class-based TF-IDF procedure.
  * Guldi (2019) Jo Guldi. 2019.  Parliament’s debates about infrastructure: An exercise in using dynamic topic models to synthesize historical change.  _Technology and Culture_ , 60:1 – 33.
  * Johnson et al. (2021) Kyle P. Johnson, Patrick J. Burns, John Stewart, Todd Cook, Clément Besnier, and William J. B. Mattingly. 2021.  The Classical Language Toolkit: An NLP framework for pre-modern languages.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 20–29, Online. Association for Computational Linguistics.
  * O’Callaghan et al. (2015) Derek O’Callaghan, Derek Greene, Joe Carthy, and Pádraig Cunningham. 2015.  An analysis of the coherence of descriptors in topic modeling.  _Expert Systems with Applications_ , 42(13):5645–5657.
  * Řehůřek and Sojka (2010) Radim Řehůřek and Petr Sojka. 2010.  Software Framework for Topic Modelling with Large Corpora.  In _Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks_ , pages 45–50, Valletta, Malta. ELRA.  http://is.muni.cz/publication/884893/en.
  * Rhody (2012) Lisa M Rhody. 2012.  Topic modeling and figurative language.
  * Sievert and Shirley (2014) Carson Sievert and Kenneth E. Shirley. 2014.  Ldavis: A method for visualizing and interpreting topics.
  * Sprugnoli and Passarotti (2022) Rachele Sprugnoli and Marco Passarotti, editors. 2022.  _Proceedings of the Second Workshop on Language Technologies for Historical and Ancient Languages_. European Language Resources Association, Marseille, France.
  * Sprugnoli et al. (2022) Rachele Sprugnoli, Marco Passarotti, Flavio Massimiliano Cecchini, Margherita Fantoli, and Giovanni Moretti. 2022.  Overview of the EvaLatin 2022 evaluation campaign.  In _Proceedings of the Second Workshop on Language Technologies for Historical and Ancient Languages_ , pages 183–188, Marseille, France. European Language Resources Association.
  * Strubell et al. (2020) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2020.  Energy and Policy Considerations for Modern Deep Learning Research.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(09):13693–13696.  Number: 09.

## 10\. Language Resource References

##

  * (1) Carey, William L.  _The Latin Library_.  PID http://www.thelatinlibrary.com/.