# MAiDE-up: Multilingual Deception Detection of
GPT-generated Hotel Reviews

Oana Ignat Xiaomeng Xu∗ Rada Mihalcea
University of Michigan - Ann Arbor, USA
{oignat, phoebexu<EMAIL_ADDRESS>
Oana Ignat and Xiaomeng Xu contributed equally to the manuscript

###### Abstract

Deceptive reviews are becoming increasingly common, especially given the
increase in performance and the prevalence of LLMs. While work to date has
addressed the development of models to differentiate between truthful and
deceptive human reviews, much less is known about the distinction between real
reviews and AI-authored fake reviews. Moreover, most of the research so far
has focused primarily on English, with very little work dedicated to other
languages. In this paper, we compile and make publicly available the MAiDE-up
dataset, consisting of 10,000 real and 10,000 AI-generated fake hotel reviews,
balanced across ten languages. Using this dataset, we conduct extensive
linguistic analyses to (1) compare the AI fake hotel reviews to real hotel
reviews, and (2) identify the factors that influence the deception detection
model performance. We explore the effectiveness of several models for
deception detection in hotel reviews across three main dimensions: sentiment,
location, and language. We find that these dimensions influence how well we
can detect AI-generated fake reviews.

MAiDE-up: Multilingual Deception Detection of
GPT-generated Hotel Reviews

Oana Ignat††thanks: Oana Ignat and Xiaomeng Xu contributed equally to the
manuscript Xiaomeng Xu∗ Rada Mihalcea University of Michigan - Ann Arbor, USA
{oignat, phoebexu<EMAIL_ADDRESS>

##  1 Introduction

Recent advancements in Natural Language Generation (NLG) technology have
greatly improved the quality of LLM-generated text. One notable example is
OpenAI’s ChatGPT, which has demonstrated exceptional performance in tasks such
as story generation (Yuan et al., 2022), question answering (Bahak et al.,
2023), essay writing (Stokel-Walker, 2022), and coding (Becker et al., 2023).
However, this newfound ability to produce highly efficient, human-like texts
also raises concerns about detecting and preventing the misuse of LLMs
(Pagnoni et al., 2022; Mirsky et al., 2023).

One particular problem is the prevalence of AI-generated reviews, and while
tools and datasets have been proposed, none have solved the problem completely
(Wu et al., 2023b), which in turn, leads to the erosion of trust in online
opinions. Furthermore, most of the research so far has focused primarily on
English, with very little work dedicated to other languages. Our study aims to
fill in these gaps and provide novel insights into multilingual LLM-generated
hotel reviews. We analyze different types of human-interpretable features,
such as linguistic style, writing structure, topics, and psycholinguistic
markers, along with baselines across multiple models. We hope our research
will help organizations leverage NLP to combat the use of LLMs in scenarios
where genuine, human-generated content is highly valued, such as customer
reviews on platforms like Booking, Yelp, and Amazon.

Specifically, our paper aims to answer two main research questions.

RQ1:

How LLM-generated multilingual hotel reviews compare to real reviews across
syntactic and lexical elements.

RQ2:

Which factors influence multilingual deception detection performance?

The paper makes the following contributions. First, we compile and share a
multilingual dataset of 10,000 real and 10,000 AI-generated fake hotel
reviews, balanced across ten languages: Chinese, English, French, German,
Italian, Korean, Romanian, Russian, Spanish, and Turkish, as well as across
ten locations and different sentiment polarities. To the best of our
knowledge, this is the first dataset of multilingual reviews at this scale.
Second, using this dataset, we conduct extensive linguistic style and lexical
analyses to compare the AI-generated deceptive hotel reviews with the real
human-written hotel reviews. Finally, we explore the effectiveness of
different models for multilingual deception detection in hotel reviews across
three dimensions: sentiment, location, and language.

##  2 Related Work

##### LLM-generated Text Detection.

The powerful generation capabilities of LLMs has made it challenging for
humans to differentiate between LLM-generated and human-written texts Jakesch
et al. (2023). This led to extensive research being conducted on developing
models for detecting LLM-generated text, including fine-tuned LMs (Jawahar et
al., 2020; Guo et al., 2023), zero-shot methods (Solaiman et al., 2019;
Ippolito et al., 2019), watermarking techniques (Kirchenbauer et al., 2023a,
b), adversarial learning methods (Hu et al., 2023; Koike et al., 2024), LLMs
as detectors (Bhattacharjee and Liu, 2024; Yu et al., 2023), and human-
assisted methods (Dou et al., 2021).

Fine-tuning LMs, in particular Roberta Liu et al. (2019), has had great
success in binary classification settings (Fagni et al., 2021; Radford et al.,
2019), and therefore, we also adopt it in our work. On average, these models
yielded a 95% accuracy, outperforming zero-shot, and watermarking methods and
showing resilience to various attacks within in-domain settings. However, just
like other encoder-based fine-tuning approaches, these models lack robustness
when dealing with cross-domain or unforeseen data Bakhtin et al. (2019);
Antoun et al. (2023).

Our work falls under the category of black-box modeling, as described in a
recent survey of AI language detectors by Tang et al. (2023). We are
interested in the outputs of LLMs, rather than the specific details of a
model’s contents or design. This approach allows us to focus on the
differences between human and AI-generated texts, instead of the particular
implementation details of models or detectors.

##### Human vs. LLM-generated Text.

There have been several efforts to study the differences between AI and human-
generated text (Giorgi et al., 2023; Sadasivan et al., 2023; Jakesch et al.,
2023; Markowitz et al., 2024; Markowitz and Hancock, 2024)

In the context of deception, Giorgi et al. (2023) and Markowitz and Hancock
(2024) argue that AI-generated text is inherently deceptive when describing
human experiences like writing reviews because the system is not grounded in
material world experiences. At the same time, Giorgi et al. (2023) and
Markowitz and Hancock (2024) are the closest to our work. They use ChatGPT to
generate hotel reviews and compare them to human-written deceptive and
truthful hotel reviews from TripAdvisor. This data is collected by Ott et al.
(2011) from 20 hotels in Chicago, IL, USA. Markowitz and Hancock (2024) find
that AI-generated text has a more analytic style and is more affective, more
descriptive, and less readable than human-written text, while Giorgi et al.
(2023) find that human-written text is more diverse in its expressions of
personality than AI-generated text. We replicate their style analysis findings
and extend the data collection and analysis to nine other languages and ten
global hotel locations. Unlike prior work, we do not analyze deceptive human
reviews. In addition, we also analyze the role of sentiment, language, and
location in deception performance and implicitly in the quality of the AI-
generated reviews.

##### Multilingual LLM-generated Text Detection.

While several multilingual models Tulchinskii et al. (2024); Mitchell et al.
(2023); Antoun et al. (2023); Guo et al. (2023) and datasets (Wang et al.,
2023; Guo et al., 2020) have been proposed, Wu et al. (2023a), in their
comprehensive survey on LLM-generated text, address the need for multilingual
datasets and models to facilitate the evaluation of text detectors generated
by LLMs across different languages. Addressing them is essential for the
usability and fairness of detectors for LLM-generated text. In our work, we
address this gap, by providing a dataset, analysis, and classification models
for 10 languages.

##  3 The MAiDE-up Dataset

To answer our research questions, we compile a novel dataset, which we refer
to as MAiDE-up \- Multilingual Ai-generateD fakE reviews. MAiDE-up contains a
total of 20,000 hotel reviews: 10,000 are real, human-written, and 10,000 are
fake, LLM-generated. The reviews are balanced across language, location, and
sentiment. We outline our process for collecting both real and fake reviews
below.

###  3.1 Real Hotel Reviews

We collect 10,000 hotel reviews from Booking.com, 111booking.com which is one
of the largest marketplaces for online travel bookings. The data is balanced
per language, location, and sentiment. Finally, we ensure data quality through
automatic and manual assessments.

##### Languages.

The dataset is balanced across ten languages: Chinese, English, French,
German, Italian, Korean, Romanian, Russian, Spanish, and Turkish. We
automatically web-crawl Booking.com for each of the ten languages. Additional
details on how we automate this process can be found in Section A.1.

##### Locations.

We collect reviews from hotels located in popular capital cities: Ankara,
Beijing, Berlin, Bucharest, Madrid, New Delhi, Paris, Rome, Seoul, and
Washington. Most of the cities are selected to be the capitals of the
countries with the official language as one of the 10 languages. The only
exception is the New Delhi - Russian pair, as hotel reviews in Moscow are
restricted, and most hotel reviews in Hindi are written in English. To ensure
the collection of an equal number of reviews for each language, particularly
for Chinese, Korean, Romanian, and Turkish, where the number of reviews per
hotel is often limited, we identify 250 hotels in each city and collect up to
50 reviews per hotel.

Figure 1: An example of an English positive review, rated with a score of 10,
that contains both “upside” and “downside” sections. The reviewer can choose
to write in just one or both sections.

##### Sentiment.

The Booking.com platform provides review scores from 1 to 10, where a score of
1-6 represents a negative review and 7-10 is a positive review. We collect a
balanced number of positive and negative reviews for each language, i.e., 500
positive and 500 negative for each language. The platform provides a specific
review format consisting of two parts, upside and downside, to allow users to
separate their positive and negative feedback. The “upside” and “downside”
sections are optional, as the reviewer can write in just one or both sections.
Figure 1, shows a review example of a positive review, rated with a score of
10, that contains both “upside” and “downside” sections.

##### Quality Assurance.

We automatically verify each review’s language and filter out reviews in a
different language than the one crawled for (more information in the
Appendix). To further ensure the data quality, ten native speakers manually
verified 50 random reviews each. Specifically, they verified the review’s
syntax and semantics and ensure the sentiment aligns with the content for the
“upside” and “downside” parts of the review, as well as for the review
overall. One potential concern is the possibility of fake human-written
reviews. Booking.com addresses this by combining specialized personnel and
automated systems to detect and remove fake reviews(more information in
Section A.1).

###  3.2 LLM-generated Hotel Reviews

We generate 10,000 hotel reviews with
GPT-4.222https://platform.openai.com/docs/guides/text-generation/chat-
completions-api The generated reviews follow the same distribution across
language, location, and sentiment, as the real hotel reviews, as described in
section 3.1. We use GPT-4 because it is one of the largest LLMs available and
has been demonstrated to effectively emulate human texts (Achiam et al.,
2023).

####  3.2.1 Prompt Design and Robustness

GPT-4 takes a prompt as input, which is comprised of a list of message
objects, and returns one generated hotel review as output. We use messages,
which are more interactive and dynamic compared to the classical prompt-style.
Specifically, we use messages with two properties: role and content: the role
takes one of three values: “system”, “user” or “assistant”, while the content
contains the text of the
message.333https://help.openai.com/en/articles/7042661-chatgpt-api-transition-
guide

The prompt is first formatted with a “system message” role, which sets the
behavior of the model. This is followed by two rounds of conversations between
the roles of “assistant” and “user” in a few-shot prompting technique.
Finally, we use a “user message” to prompt the model to generate a hotel
review in a specified language, location (hotel name and capital city), and
sentiment.

##### System Prompt.

We find that we can obtain high-quality responses with additional context in
our prompts. Therefore, we instruct the model to be a well-traveled native
{language} tourist in the ”system message”. The {language} placeholder is
replaced by the language name of the hotel reviews we aim to collect. The
instructions also contain information about the language, location (hotel name
and city) and the sentiment of the review, as well as the output format, as
illustrated below:

> You are a well-traveled native {language} tourist, working on writing hotel
> reviews of hotels you have stayed in. Given hotel name, city name, language,
> and sentiment, you write a hotel review comprised of upside and downside.
> Then you give an overall review score, an integer ranging from 1 to 10 where
> the score larger than 6 indicates positive experience, otherwise negative
> experience. You always output a JSON containing the following keys:
> “Upside_Review”, “Downside_Review”, “Review_Score”. Reviews you write are
> always in consistent styles, tone, sentence structure.

##### User Prompt.

To increase the diversity and robustness of the generated data, we collect 36
multilingual “user messages”. Specifically, we ask ten native speakers to each
write four “user messages”: two in their own language and two in the
corresponding English translation. The guidelines given to the annotators are
shown in Section A.2.

We use the “user messages” to prompt GPT-4 to generate a hotel review with a
specified language, hotel location, and sentiment. Below is an example of a
“user message” in Spanish and its English translation. We show all the “user
messages” in the Appendix Table 5.

> ¿Podés escribir un comentario positivo en L sobre el hotel H de C?
>
> Can you please write a positive review in L for the hotel H located in C?

##### Few-shot prompting.

We use the conversations between “user” and “assistant” when generating data.
The “user messages” are randomly selected from our multilingual “user
messages” and the “assistant messages” are randomly extracted from our
collected real hotel reviews. The answer to the last “user message” is
automatically generated by the “assistant message” and used to collect the
GPT-4 generated hotel reviews.

##### Quality Assurance.

To ensure the quality of our generated data, we conduct sanity checks by
asking native speakers to review approximately 100 hotel reviews in their
respective languages. The reviews are checked for readability, syntax errors,
and style. Based on the feedback, we find that some of the generated Chinese
reviews contain nonsensical phrases. Additionally, reviews in other languages
such as Romanian, Korean, and Spanish have a formal tone. Table 1 displays
random reviews in English, both real and generated, with positive and negative
sentiments. Reviews sampled from all ten languages are shown in Appendix
Tables 6 and 7.

Real Hotel Reviews
---
+ Nicely furnished room and nicely decorated lobby. Room service is affordable and the receptionists, especially Mr. Omar, are usually eager to help with any queries; N/A
- The location was very central and the staff was nice and helped us.; The room smelled like cigarettes and there was mold in the bathroom
LLM-generated Hotel Reviews
+ The staff was extremely helpful and accommodating. Clean and well-furnished rooms. Central location with easy access to public transport.; The breakfast offerings could be more varied.
- Nothing. Horrible experience.; Bad customer service, rooms were not clean and the food was below average.
Table 1: Random review samples with positive (+) and negative (-) sentiments
in English. Each review has two parts: “upside” and “downside”, which are
separated by “;”. We manually find that it is very challenging for humans to
distinguish between real and generated reviews.

##  4 Multilingual Analyses of Real and LLM-generated Hotel Reviews

Using our dataset, we conduct extensive linguistic analysis to compare the AI-
generated fake hotel reviews with the real human-written hotel reviews.

##### Analytic writing

index measures the complexity and sophistication of the writing, which can be
an indicator of advanced thinking. This technique has been used in various
fields, including persuasion (Markowitz, 2020), analysis of political speeches
(Jordan et al., 2019), and gender studies (Meier et al., 2020), among others.
The formula for analytic writing is $[articles+prepositions-pronouns-
auxiliaryverbs-adverb-conjunctions-negations]$ from LIWC scores (Jordan et
al., 2019; Pennebaker et al., 2014). We use Linguistic Inquiry and Word Count
(LIWC) (Pennebaker et al., 2007, 2015), a gold-standard text analysis tool, to
obtain the categories in the index formula for the following languages:
Chinese, English, French, and Spanish.444While versions for Korean and Turkish
are listed as available upon request, we were unable to obtain them. Since
they are translations for different versions of the LIWC lexicon, e.g., 2001,
2007, and 2015, we first align them using the intersection with the 2015
English version.

In Table 2, we find that AI-generated texts in English are more complex than
real texts, which aligns with our initial observations from data quality
checks. The results for Chinese, French, and Spanish are not statistically
significant.

##### Descriptiveness

can be measured by the frequency of adjectives used in language patterns.
Texts with high rates of adjectives tend to be more elaborate and narrative-
like compared to texts with low rates of adjectives. (Chung and Pennebaker,
2008) Additionally, adjectives are often used in false speech, making them a
key marker of deceitful language (Johnson and Raye, 1981). We measure the
ratio of adjectives using the multilingual library textdescriptives from
Hansen et al. (2023).555https://github.com/HLasse/TextDescriptives For Turkish
we use HuggingFace from Altinok (2023).666https://huggingface.co/turkish-nlp-
suite

We show the results in Table 2. We find that AI-generated text is usually more
descriptive than real text. The only exceptions are German reviews, where the
real text is more descriptive, and Korean reviews, where the difference is not
significant.

##### Readability

considers not only word count, but also word complexity. For instance, longer
words are more complex than shorter ones. We use the Flesch Reading Ease
metric (Flesch, 1948), which counts the number of words per sentence and
syllables per word. This metric is used to assess the structural complexity of
language patterns in various texts, such as scientific articles (Markowitz and
Hancock, 2016), online petitions (Markowitz, 2023), and social media data
(Hubner and Bond, 2022). We use the multilingual library textdescriptives to
compute the Flesch Reading Ease metric and the word count.

Results are shown in Table 2. We find that AI-generated text is usually less
readable and more wordy than real text. The only exception is Russian data,
where the difference is not significant.

| Analytic writing | Descriptiveness | Readability | Word Count
---|---|---|---|---
Lang | Real | Gen | Real | Gen | Real | Gen | Real | Gen
[length=0.5cm, width=0.3cm]CN | 6.2 $\pm$ 7.0 | 6.5 $\pm$ 3.6 | 2.6 $\pm$ 3.9∗ | 4.2 $\pm$ 3.5∗ | - | - | 62.1 $\pm$ 92.9∗ | 79.4 $\pm$ 41.2∗
[length=0.5cm, width=0.3cm]US | 11.9 $\pm$ 6.8∗ | 18.6 $\pm$ 5.2∗ | 13.7 $\pm$ 8.6∗ | 15.5 $\pm$ 5.5∗ | 66.3 $\pm$ 29.0∗ | 54.6 $\pm$ 17.1∗ | 49.2 $\pm$ 54.7∗ | 55.7 $\pm$ 28.5∗
[length=0.5cm, width=0.3cm]FR | 21.0 $\pm$ 10.1 | 20.8 $\pm$ 6.2 | 13.9 $\pm$ 9.3∗ | 16.2 $\pm$ 6.7∗ | 51.3 $\pm$ 29.9∗ | 43.9 $\pm$ 15.8∗ | 39.6 $\pm$ 43.5∗ | 47.5 $\pm$24.2∗
[length=0.5cm, width=0.3cm]DE | - | - | 7.0 $\pm$ 7.4∗ | 5.4 $\pm$ 5.5∗ | 28.2 $\pm$ 32.3 | 27.2 $\pm$ 19.3 | 41.2 $\pm$ 43.0∗ | 47.0 $\pm$ 22.4∗
[length=0.5cm, width=0.3cm]IT | - | - | 15.4 $\pm$ 10.2∗ | 19.3 $\pm$ 7.6∗ | -9.4 $\pm$ 32.0∗ | -13.7 $\pm$ 17.9∗ | 37.9 $\pm$ 41.7∗ | 45.0 $\pm$ 23.2∗
[length=0.5cm, width=0.3cm]RO | - | - | 11.3 $\pm$ 9.1∗ | 15.5 $\pm$ 6.3∗ | 8.4 $\pm$ 36.7∗ | -0.6 $\pm$ 19.3∗ | 39.6 $\pm$ 46.3∗ | 44.1 $\pm$ 22.1∗
[length=0.5cm, width=0.3cm]KR | - | - | 6.0 $\pm$ 6.4 | 5.7 $\pm$ 4.3 | - | - | 30.1 $\pm$ 31.7∗ | 32.6 $\pm$ 14.1∗
[length=0.5cm, width=0.3cm]RU | - | - | 13.7 $\pm$ 8.7∗ | 18.9 $\pm$ 6.0∗ | -8.2 $\pm$ 35.1∗ | -16.8 $\pm$ 20.0∗ | 42.8 $\pm$ 45.5 | 40.7 $\pm$ 20.5
[length=0.5cm, width=0.3cm]ES | 12.9 $\pm$ 7.0 | 15.9 $\pm$ 4.1 | 11.5 $\pm$ 8.6∗ | 14.9 $\pm$ 5.4∗ | 19.9 $\pm$ 27.6∗ | 11.0 $\pm$ 17.1∗ | 39.0 $\pm$ 40.5∗ | 48.0 $\pm$ 23.1∗
[length=0.5cm, width=0.3cm]TR | - | - | 13.6 $\pm$ 9.5∗ | 14.4 $\pm$ 6.0∗ | - | - | 26.4 $\pm$ 25.3∗ | 32.8 $\pm$ 14.7∗

Table 2: To what degree is AI-generated text different from real text in terms
of analytic writing, descriptiveness, readability, and word count? We compute
the mean and standard deviation for all the reviews, across each language. We
mark (∗) when the difference between real and generated data is statistically
significant, based on the Student t-test (Student, 1908) with p-value ¡ 0.05.

##### Topic Modeling.

We compute the most prevalent topics and their keywords with a multilingual
pipeline from Scattertext (Kessler, 2017). Each review is processed to obtain
the most important words with high TF-IDF scores Ramos et al. (2003). We pre-
process the text with spaCy multilingual pipelines to tokenize hotel reviews
and lowercase tokens, remove stop words, and lemmatize
tokens.777https://spacy.io/models We display the topic distribution across
real and AI-generated hotel reviews in all languages in Figure 2 and in
English in Appendix Figure 9.

Comparing real to generated reviews, we find that in multilingual data (Figure
2), the word “Booking” is most frequent in real hotel reviews, along with
words like “reception (ro: receptie)”, “checking”, “bathroom (ro: baie, es:
bagno)”, “shower”. In contrast, generated hotel reviews contain more words
about “service”, “comfort (de: komfortabel, ch: 舒适)” and “room (ro:
camerele)”. In English hotel reviews (Figure 9), AI-generated reviews contain
more mentions of city names: “Bucharest”, “Washington”, “Ankara”, while real
hotel reviews contain more words about “cleaning” and “time”.

Figure 2: Visualization of topics used in the real and LLM-generated reviews
across all languages. Points are colored red or blue based on the association
of their corresponding topics with AI-generated or real hotel reviews. The
most associated topics are listed under Top AI-Generated and Top Real
headings. Interactive version:
https://anonymous.4open.science/r/hotel_reviews_deception.

##  5 Multilingual Deception Detection

We explore the effectiveness of different models for multilingual deception
detection in hotel reviews. As baselines, we test a random classifier, and, as
an interpretable model, we train and test a Naive Bayes (Lewis, 1998)
classifier. As our main model, we fine-tune an XLM-RoBERTa (Conneau et al.,
2019) classifier, which lacks interpretability but is highly performant.

###  5.1 Baselines

##### Random.

The reviews are split equally between real and generated. Therefore, a random
baseline would have an accuracy score of 50%.

##### Naive Bayes.

As a simple and interpretable baseline, we train a Naive Bayes classifier
(Lewis, 1998) to distinguish between real and AI-generated reviews. To
tokenize, we use a count vectorizer with a minimum frequency cutoff of $5$. We
use the Multinomial Naive Bayes model from the sklearn (Pedregosa et al.,
2011) library with the default settings. Interpretability results are shown in
Appendix B.

###  5.2 Main Model

##### XLM-RoBERTa.

We fine-tune XLM-RoBERTa base Conneau et al. (2019) model from
HuggingFace888https://huggingface.co/FacebookAI/xlm-roberta-base. The model is
a multilingual version of RoBERTa (Liu et al., 2019), and is pre-trained on
2.5TB of filtered CommonCrawl (Wenzek et al., 2020) data containing 100
languages.

We use a learning rate of $2e-5$ and a batch size of $8$. We train for $5$
epochs and take the best epoch based on validation accuracy. The validation
data represents 10% of the train data set.

##### Model Training Setup.

We use one model for all the multilingual data and also experiment with
different training and test data splits. Specifically, we use a default train-
test split of 80-20% and a few-shot train-test split of 1-99%, as shown in
Table 3.

| default | few-shot
---|---|---
| real | gen | real | gen
Train | 8,000 | 8,000 | 100 | 100
Test | 2,000 | 2,000 | 9,900 | 9,900

Table 3: Number of reviews for the experimental data split The default split corresponds to an 80-20 % train-test data split. The few-shot split corresponds to a 1-99 % train-test data split. Setup | Accuracy | F1
---|---|---
Interpretable Baseline: Naive Bayes
default | 85.9 | 87.1
few-shot | 54.4 | 54.9
zero-shot | 50.0 | 0.0
Main Model: XLM-RoBERTa
default | 94.8 | 94.9
few-shot | 76.6 | 80.1
zero-shot | 50.0 | 0.0
Table 4: Classification test results with the zero-shot, few-shot and default
setups.

##  6 Evaluation

We show the Accuracy and F1 results for all the models in Table 4. The model
with the best detection performance is XLM-RoBERTa, with an accuracy of
$94.9\%$ on the default train-test data split and $76.6\%$ accuracy on the
few-shot train-test data split. A fairly high few-shot performance shows that
the model is capable of learning this task from very few examples, i.e., 200
reviews balanced across 10 languages. We emphasize the few-shot performance
because it closely resembles the real-life scenario when a user wants to
generate fake multilingual reviews and has access to very few examples of fake
reviews. A zero-shot performance of 50% accuracy indicates that the model had
no previous knowledge about this task, and the only knowledge it receives is
from our fine-tuning. Finally, we show the accuracy of the main model across
multiple test-train splits in Appendix Figure 8. This shows that the XLM-
RoBERTa deception classification model learns considerably from just 2,000
reviews (train data ratio of 10%), and increasing the training data does not
lead to high-performance gains.

###  6.1 Ablation Studies

We also compute the XLM-RoBERTa main model few-shot performance across
sentiment, review language, prompt language, and hotel location and observe
differences in deception performance between different languages and hotel
locations, as shown in Figure 3. We use 10-fold cross-validation to compute
confidence intervals. We compute the significance using the Student t-test
(Student, 1908), with p-value ¡0.05.

##### Review Language.

As shown in Figure 3 (a), deception performance is lowest for Korean and
English reviews. This implies that GPT4 is better at generating deceptive,
“human-like” English and Korean reviews. On the other hand, deception
performance is highest for German and Romanian reviews, which implies that
GPT4 is worse at generating deceptive, “human-like” German and Romanian
reviews. In addition, we find that the GPT4 training language data amount
moderately correlates with the GPT4 performance per language. We measure the
Pearson correlation coefficient at 0.53. We approximate the amount of training
data per language by what is available in CommonCrawl Wenzek et al. (2020).

##### Prompt Language.

From the perspective of a user interested in generating hotel reviews with
GPT4, we measure how the language of the user prompt impacts the quality of
the generated reviews. As shown in Figure 3 (b), deception performance is
lowest for Turkish and Korean, meaning that multilingual reviews generated
with prompts in these languages are harder to distinguish from real reviews.
Thus, GPT4 is better at generating deceptive “human-like” multilingual reviews
when prompted in Turkish and Korean. On the other hand, deception performance
is highest for English and French prompts, thus GPT4 is worse at generating
deceptive “human-like” multilingual reviews when prompted in English and
French.

##### Hotel Location.

When generating multilingual hotel reviews using multilingual prompts, we also
measure how the location of the hotel impacts the quality of the generated
reviews. As shown in Figure 3 (c), deception performance is lowest for Seoul,
Rome, and Beijing, meaning that multilingual generated reviews for these hotel
locations are harder to distinguish from real reviews. Thus, GPT4 is better at
generating deceptive “human-like” multilingual reviews for hotels in Seoul,
Rome, and Beijing. On the other hand, deception performance is highest for
Bucharest, Washington, Ankara, and Berlin, thus GPT4 is worse at generating
deceptive “human-like” multilingual reviews for hotels in Bucharest,
Washington, Ankara, and Berlin.

##### Review Sentiment.

Across sentiment polarities, our main deception classification model obtains
an accuracy of $79.9$ for positive reviews and $82.6$ for negative reviews.
The performance difference between the two types of reviews is statistically
significant. A lower deception classification performance on positive reviews
indicates that GPT4 is better at generating “human-like” multilingual positive
reviews than “human-like” multilingual negative reviews.

##### Review and Prompt Language.

We also compute deception detection accuracy per prompt and review language
pair as shown in Figure 4. For each review language, we measure which prompt
languages achieve lower deception accuracy, which is equivalent to better,
more “human-like” generation performance. For example, we find that better
Chinese reviews are generated using Turkish prompts. Additionally, Korean
reviews have the lowest deception performance, independent of language
prompts. Therefore, GPT4 is best at generating deceptive, “human-like” Korean
reviews, independent of the prompt language. In contrast, GPT4 is worst at
generating deceptive, “human-like” Turkish reviews, independent of the prompt
language.

Figure 3: Accuracy with XLM-RoBERTa model per (a) review language, (b) prompt
language, and (c) hotel location on the few-shot train-test split.  Figure 4:
Accuracy with XLM-RoBERTa model per prompt and review language on few-shot
train-test split.

##  7 Conclusion

In this paper we focused on the identification of AI-written fake hotel
reviews in multiple languages. To facilitate research in this domain, we
released MAiDE-up, a dataset of 10,000 real and 10,000 AI-generated fake hotel
reviews balanced across ten languages. Using this data, we performed extensive
linguistic data analyses to gain insight into how AI fake hotel reviews differ
from real hotel reviews. Finally, we explored the effectiveness of several
models for deception detection in hotel reviews across three main dimensions:
sentiment, location, and language. Despite the difficulty humans have in
distinguishing between real hotel reviews and those generated by LLMs, we
discovered that these posts have noticeable differences in style, structure,
and semantics and that, even with little data, fine-tuned models accurately
detect deceptive reviews across different languages. Our dataset can be used
to train and analyze other multilingual models, and it is publicly available
at https://github.com/MichiganNLP/multilingual_reviews_deception.

## Limitations

### 1\. Multilingual models limitations

When analyzing data across ten different languages, we encounter significant
challenges in identifying computational models and tools that can be
universally applied. In Table 2, we cannot find the LIWC categories required
for analytic writing formula for the following languages: German, Italian,
Romanian, Korean, Russian, Turkish. Additionally, the textdescriptives library
does not currently support readability metrics for Chinese, Korean, and
Turkish. This highlights the limitations of computational linguistic methods,
which are currently predominantly English-focused. This, in turn, restricts
the potential for research on multilingual data.

### 2\. Temporal limitations of LLM-generated data

Although LLMs are incredibly powerful in terms of understanding and generating
text, they may not fully capture the current cultural and societal context in
their outputs. This is because they are trained on data up to a specific
cutoff date. For example, GPT-4 was trained on data only up to September 2021,
which means that it may not accurately reflect events or changes that have
occurred after that date.

## Ethics Concerns

We strongly oppose using our research findings and data to generate
multilingual reviews to deceive consumers into believing they are human-
authored. These unethical practices compromise the credibility of online
review platforms and erode consumer trust. Instead, we advocate for
transparency and authenticity in the digital marketplace. We have developed a
multilingual deception detection model to combat the proliferation of fake
reviews generated by bots. This model employs advanced algorithms to
meticulously analyze linguistic nuances and syntactic structures, enabling the
accurate differentiation between multilingual reviews created by language
models and those written by human users. By providing this tool, we aim to
empower businesses and online platforms to maintain ethical standards, protect
consumers from deceptive practices, and foster a more trustworthy and reliable
digital environment.

##

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Altinok (2023) Duygu Altinok. 2023.  A diverse set of freely available linguistic resources for Turkish.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13739–13750, Toronto, Canada. Association for Computational Linguistics.
  * Antoun et al. (2023) Wissam Antoun, Virginie Mouilleron, Benoît Sagot, and Djamé Seddah. 2023\.  Towards a robust detection of language model generated text: Is chatgpt that easy to detect?  _arXiv preprint arXiv:2306.05871_.
  * Bahak et al. (2023) Hossein Bahak, Farzaneh Taheri, Zahra Zojaji, and Arefeh Kazemi. 2023.  Evaluating chatgpt as a question answering system: A comprehensive analysis and comparison with existing models.  _arXiv preprint arXiv:2312.07592_.
  * Bakhtin et al. (2019) Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. 2019.  Real or fake? learning to discriminate machine from human generated text.  _arXiv preprint arXiv:1906.03351_.
  * Becker et al. (2023) Brett A Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023.  Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation.  In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1_ , pages 500–506.
  * Bhattacharjee and Liu (2024) Amrita Bhattacharjee and Huan Liu. 2024.  Fighting fire with fire: can chatgpt detect ai-generated text?  _ACM SIGKDD Explorations Newsletter_ , 25(2):14–21.
  * Chung and Pennebaker (2008) Cindy K Chung and James W Pennebaker. 2008.  Revealing dimensions of thinking in open-ended self-descriptions: An automated meaning extraction method for natural language.  _Journal of research in personality_ , 42(1):96–132.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _CoRR_ , abs/1911.02116.
  * Dou et al. (2021) Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. 2021\.  Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Fagni et al. (2021) Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. 2021.  Tweepfake: About detecting deepfake tweets.  _Plos one_ , 16(5):e0251415.
  * Flesch (1948) Rudolph Flesch. 1948.  A new readability yardstick.  _Journal of applied psychology_ , 32(3):221.
  * Giorgi et al. (2023) Salvatore Giorgi, David M. Markowitz, Nikita Soni, Vasudha Varadarajan, Siddharth Mangalik, and H. A. Schwartz. 2023.  ”i slept like a baby”: Using human traits to characterize deceptive chatgpt and human text.  In _IACT@SIGIR_.
  * Guo et al. (2023) Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection.  _arXiv preprint arXiv:2301.07597_.
  * Guo et al. (2020) Mandy Guo, Zihang Dai, Denny Vrandecic, and Rami Al-Rfou. 2020.  Wiki-40b: Multilingual language model dataset.  In _LREC 2020_.
  * Hansen et al. (2023) Lasse Hansen, Ludvig Renbo Olsen, and Kenneth Enevoldsen. 2023.  Textdescriptives: A python package for calculating a large variety of metrics from text.  _arXiv preprint arXiv:2301.02057_.
  * Hu et al. (2023) Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023.  Radar: Robust ai-text detection via adversarial learning.  _Advances in Neural Information Processing Systems_ , 36:15077–15095.
  * Hubner and Bond (2022) Austin Y Hubner and Robert Bond. 2022.  I am a scientist… ask me anything: Examining differences between male and female scientists participating in a reddit ama session.  _Public Understanding of Science_ , 31(4):458–472.
  * Ippolito et al. (2019) Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2019.  Automatic detection of generated text is easiest when humans are fooled.  _arXiv preprint arXiv:1911.00650_.
  * Jakesch et al. (2023) Maurice Jakesch, Jeffrey T Hancock, and Mor Naaman. 2023.  Human heuristics for ai-generated language are flawed.  _Proceedings of the National Academy of Sciences_ , 120(11):e2208839120.
  * Jawahar et al. (2020) Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. 2020.  Automatic detection of machine generated text: A critical survey.  _arXiv preprint arXiv:2011.01314_.
  * Johnson and Raye (1981) Marcia K Johnson and Carol L Raye. 1981.  Reality monitoring.  _Psychological review_ , 88(1):67.
  * Jordan et al. (2019) Kayla N Jordan, Joanna Sterling, James W Pennebaker, and Ryan L Boyd. 2019.  Examining long-term trends in politics and culture through language of political leaders and cultural institutions.  _Proceedings of the National Academy of Sciences_ , 116(9):3476–3481.
  * Kessler (2017) Jason Kessler. 2017.  Scattertext: a browser-based tool for visualizing how corpora differ.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 85–90, Vancouver, Canada. Association for Computational Linguistics.
  * Kirchenbauer et al. (2023a) John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023a.  A watermark for large language models.  In _International Conference on Machine Learning_ , pages 17061–17084. PMLR.
  * Kirchenbauer et al. (2023b) John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. 2023b.  On the reliability of watermarks for large language models.  _arXiv preprint arXiv:2306.04634_.
  * Koike et al. (2024) Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. 2024.  Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 21258–21266.
  * Lewis (1998) David D. Lewis. 1998.  Naive (bayes) at forty: The independence assumption in information retrieval.  In _European Conference on Machine Learning_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Markowitz (2020) David M Markowitz. 2020.  Putting your best pet forward: Language patterns of persuasion in online pet advertisements.  _Journal of Applied Social Psychology_ , 50(3):160–173.
  * Markowitz (2023) David M Markowitz. 2023.  Instrumental goal activation increases online petition support across languages.  _Journal of Personality and Social Psychology_ , 124(6):1133.
  * Markowitz and Hancock (2016) David M Markowitz and Jeffrey T Hancock. 2016.  Linguistic obfuscation in fraudulent science.  _Journal of Language and Social Psychology_ , 35(4):435–445.
  * Markowitz and Hancock (2024) David M Markowitz and Jeffrey T Hancock. 2024.  Generative ai are more truth-biased than humans: A replication and extension of core truth-default theory principles.  _Journal of Language and Social Psychology_ , 43(2):261–267.
  * Markowitz et al. (2024) David M Markowitz, Jeffrey T Hancock, and Jeremy N Bailenson. 2024.  Linguistic markers of inherently false ai communication and intentionally false human communication: Evidence from hotel reviews.  _Journal of Language and Social Psychology_ , 43(1):63–82.
  * Meier et al. (2020) Tabea Meier, Ryan L Boyd, Matthias R Mehl, Anne Milek, James W Pennebaker, Mike Martin, Markus Wolf, and Andrea B Horn. 2020.  Stereotyping in the digital age: Male language is “ingenious”, female language is “beautiful”–and popular.  _PloS one_ , 15(12):e0243637.
  * Mirsky et al. (2023) Yisroel Mirsky, Ambra Demontis, Jaidip Kotak, Ram Shankar, Deng Gelei, Liu Yang, Xiangyu Zhang, Maura Pintor, Wenke Lee, Yuval Elovici, et al. 2023.  The threat of offensive ai to organizations.  _Computers & Security_, 124:103006.
  * Mitchell et al. (2023) Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023.  Detectgpt: Zero-shot machine-generated text detection using probability curvature.  In _International Conference on Machine Learning_ , pages 24950–24962. PMLR.
  * Ott et al. (2011) Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T Hancock. 2011.  Finding deceptive opinion spam by any stretch of the imagination.  _arXiv preprint arXiv:1107.4557_.
  * Pagnoni et al. (2022) Artidoro Pagnoni, Martin Graciarena, and Yulia Tsvetkov. 2022.  Threat scenarios and best practices to detect neural fake news.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1233–1249.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pennebaker et al. (2007) James W. Pennebaker, Roger John Booth, and Martha E. Francis. 2007.  Linguistic inquiry and word count (LIWC2007).
  * Pennebaker et al. (2015) James W. Pennebaker, Ryan L. Boyd, Kayla Jordan, and Kate G. Blackburn. 2015.  The development and psychometric properties of LIWC2015.
  * Pennebaker et al. (2014) James W Pennebaker, Cindy K Chung, Joey Frazee, Gary M Lavergne, and David I Beaver. 2014.  When small words foretell academic success: The case of college admissions essays.  _PloS one_ , 9(12):e115844.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Ramos et al. (2003) Juan Ramos et al. 2003.  Using tf-idf to determine word relevance in document queries.  In _Proceedings of the first instructional conference on machine learning_ , volume 242, pages 29–48. Citeseer.
  * Sadasivan et al. (2023) Vinu Sankar Sadasivan, Aounon Kumar, S. Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023.  Can ai-generated text be reliably detected? _ArXiv_ , abs/2303.11156.
  * Solaiman et al. (2019) Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019\.  Release strategies and the social impacts of language models.  _arXiv preprint arXiv:1908.09203_.
  * Stokel-Walker (2022) Chris Stokel-Walker. 2022.  Ai bot chatgpt writes smart essays-should academics worry?  _Nature_.
  * Student (1908) Student. 1908.  The probable error of a mean.  _Biometrika_ , pages 1–25.
  * Tang et al. (2023) Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2023.  The science of detecting llm-generated texts.  _arXiv preprint arXiv:2303.07205_.
  * Tulchinskii et al. (2024) Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. 2024.  Intrinsic dimension estimation for robust detection of ai-generated texts.  _Advances in Neural Information Processing Systems_ , 36.
  * Wang et al. (2023) Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Alham Fikri Aji, et al. 2023.  M4: Multi-generator, multi-domain, and multi-lingual black-box machine-generated text detection.  _arXiv preprint arXiv:2305.14902_.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4003–4012, Marseille, France. European Language Resources Association.
  * Wu et al. (2023a) Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F. Wong, and Lidia S. Chao. 2023a.  A survey on llm-generated text detection: Necessity, methods, and future directions.  _ArXiv_ , abs/2310.14724.
  * Wu et al. (2023b) Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F Wong, and Lidia S Chao. 2023b.  A survey on llm-gernerated text detection: Necessity, methods, and future directions.  _arXiv preprint arXiv:2310.14724_.
  * Yu et al. (2023) Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Weiming Zhang, and Nenghai Yu. 2023.  Gpt paternity test: Gpt generated text detection with gpt genetic inheritance.  _arXiv preprint arXiv:2305.12519_.
  * Yuan et al. (2022) Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022.  Wordcraft: story writing with large language models.  In _27th International Conference on Intelligent User Interfaces_ , pages 841–852.

## Appendix

##  Appendix A Dataset

###  A.1 Real Hotel Reviews

##### Language

We automatically web-crawl Booking.com for each of the ten languages by
replacing the placeholder with the corresponding language abbreviation in the
base URL: https://www.booking.com/index.{language}.html. For example, data in
Turkish can be accessed via https://www.booking.com/index.tr.html.

To automate the web browsing process and make the data collection process more
efficient, we use Selenium999https://www.selenium.dev/. However, we observe
that even if we browse in specific language settings, hotel reviews may still
be in a mix of different languages. Therefore, we use the language filter bar
to select the language we specify, which is also automated by using Selenium
in the data collection process (Figure 5).

####  A.1.1 Data Quality

##### Automatic and manual language filtering.

First, we filter out reviews based on length and language, that contain less
than three tokens and are of a different language than the one crawled for. We
use the spaCy library101010https://spacy.io/models to tokenize Chinese and
Korean reviews, and nltk111111https://www.nltk.org/ for the rest of the
languages. Next, we automatically verify the language of each review using the
langdetect library.121212https://pypi.org/project/langdetect/ We find several
Chinese reviews were written in a combination of Chinese and another language,
mostly English. Specifically, the “upside” or “downside” review may be in a
different language, most commonly English, as seen in Appendix Figure 7. We
choose to keep these reviews as they reflect the most realistic Chinese
reviews written by people. Additionally, after a few manual language checks,
we find that some Chinese reviews are incorrectly classified as Korean, and
therefore chose to check all of them manually.

##### Detect fake human reviews.

According to Booking.com, only tourists who have stayed in the hotel they
booked or have visited the hotel but did not stay there can leave a review of
the accommodation within three months of checking out. Additionally,
Booking.com uses a combination of specialized personnel and an automated
system to detect and remove fake reviews.

Figure 5: We use the language filter bar to select the language we specify,
which is also automated by using Selenium in the data collection process.
Figure 6: An example of a Chinese review where the “downside” part of the
review is in English. Figure 7: Specifications from Booking.com regarding
checking for fake reviews

###  A.2 LLM-generated Hotel Reviews

##### User Prompt Guidelines

> Please write 4 prompts, 2 in English and 2 in {language}, asking GPT to
> write a hotel review. Each prompt should contain the following:
>
>   * •
>
> sentiment of the review: one positive and one negative for each: English and
> {language})
>
>   * •
>
> language of the review, translated in {language} for the {language} prompts
>
>   * •
>
> city of the hotel, translated in {language} for the {language} prompts
>
>   * •
>
> hotel name; just include it in the prompt as a placeholder: “hotel X”, so
> you just translate the word hotel for the {language} prompts
>
>

##### Challenges solved with few-shot prompting.

The prompt design process is complex because we aim to generate reviews that
capture the format of the real hotel reviews. A real review contains two
sections: the “upside” and the “downside”, each with corresponding sentiment.
At the same time, the review’s sentiment is given from the combined assessment
of both the “upside” and the “downside”. In addition, either the “upside” or
“downside” could be null. When we include this specification in the
instruction, we find that the reviews generated with a positive sentiment
predominantly contain content in the “upside” section, and no content in the
“downside”, and conversely for reviews with negative sentiment. We solve this
issue by using a few-shot prompting approach, as mentioned above.

Lang |  Prompt/ User Message
---|---
Ch |  + 请模仿人类用L为C的H酒店写一条正面评价
$T:$ please write a positive review using L for hotel H in C that would mimic
a human writing a hotel review
|  - 请模仿人类用L为C的H酒店写一条负面评价
|  $T:$ please write a negative review using L for hotel H in C that would
mimic a human writing a hotel review
Fr |  + Écrivez un avis positif sur l’hôtel H en C. Veuillez écrire en L e parler des différents aspects de votre séjour.
$T:$ Write a positive hotel review for hotel H in C. Please write it in L and
talk about different aspects of your stay.
|  - Écrivez un avis négatif sur l’hôtel H en C. Veuillez écrire en L e parler
des différents aspects de votre séjour.
|  $T:$ Write a negative hotel review for hotel H in C. Please write it in L
and talk about different aspects of your stay.
Ge |  + Du bist ein Tourist, und du hast in hotel H in der Stadt C übernachtet. Das hotel hat dich richtig gefallen. Schreib ein Positives Review auf L über dein Stay, was fand besonderes gut and ob du das empfehlen würdest.
$T:$ You are smart and helpful assistant. Your goal is to write a positive and
realistic review for the hotel H in the language L, where you stayed in the
city C. Make sure to mention why you enjoyed your stay and list all the
positive features of the hotel.
|  - Du bist ein Tourist, und du hast in hotel H in der Stadt C übernachtet.
Das hotel hat dich gar nicht gefallen. Schreib ein Negatives Review auf L über
dein Stay, was fandest du besonderes schlecht and warum du das Hotel nicht
empfehlen würdest.
|  $T:$ You are smart and helpful assistant. Your goal is to write a negative
and realistic review for the hotel H in the language L, where you stayed in
the city C. Make sure to mention why you disliked your stay and list all the
negative features of the hotel.
It |  + Scrivi una recensione positiva in L per l’albergo H di C. Parla di almeno tre aspetti diversi del tuo soggiorno.
$T:$ Write a positive review in L for hotel H in C. Talk about at least three
different aspects of your stay.
|  - Scrivi una recensione negativa in L per l’albergo H di C. Parla di almeno
tre aspetti diversi del tuo soggiorno.
|  $T:$ Write a negative review in L for hotel H in C. Talk about at least
three different aspects of your stay.
Ko |  + 지난주에 갔던 C의 호텔 H이 너무 맘에 들었다고 L로 리뷰좀 남겨줄래?
$T:$ I really enjoyed my stay at Hotel H in C last week. Can you write a
review for me in L?
|  - L로 C에 있는 호텔 H이 너무 별로였다고 평점 좀 남겨줘.
|  $T:$ Can you write a review that the hotel H that we stayed at in C last
week was terrible? Can you write it in L?
Ro |  + Scrie un comentariu pozitiv in limba L pentru hotelul H din orașul C. Scrie la fel ca un român care a vizitat hotelul și a lăsat un comentariu.
$T:$ Write a negative sentiment review in L language for the hotel H from C.
|  - Scrie un comentariu negativ in limba L pentru hotelul H din orașul C.
Scrie la fel ca un român care a vizitat hotelul și a lăsat un comentariu.
|  $T:$ Write a positive sentiment review in L language for the hotel H from
C.
Ru |  + Представь что ты турист, и тебе очень понравилось твое пребывание в отеле H в городе C. Напиши реалистичный отзыв на L языке об этом отеле. Упомяни все черты которые тебе понравились и почему ты порекомендовала бы этот отель другим туристам.
$T:$ You are a tourist and you really enjoyed staying in the hotel H in the
city C. Write a simple hotel review in language L, where you mention all the
positive features of the hotel and how much you liked them.
|  - Представь что ты турист, и тебе очень не понравился отель H в городе C.
Напиши реалистичный отзыв на L языке об этом отеле. Упомяни все черты которые
тебе не понравились и почему бы ты не порекомендовала этот отель другим
туристам.
|  $T:$ You are a tourist and you seriously dislike your stay in the hotel H
in the city C. Write a simple hotel review in language L, where you mention
all the things you disliked, and why you wouldn’t recommend this hotel to the
other tourists.
Sp |  + ¿Podés escribir un comentario positivo en L sobre el hotel H de C?
$T:$ Can you please write a positive review in L for the hotel H located in C?
|  - Escribí un comentario negativo en L sobre un hotel llamado H que está en
C.
|  $T:$ Write a review that’s negative in sentiment in L language for a hotel
named H that’s located in C.
Tu |  + C H için olumlu yorumu L olarak yaz.
$T:$ Write a review in L about a negative experience staying in the H hotel in
C.
|  - C H de kötü bir zaman geçirdiğini duşun. L olarak H hakkında olumsuz
yorum yaz.
|  $T:$ Imagine that you had an enjoyable stay at the hotel H while visiting
C. Write a review in L to describe your experience to others who are
considering booking a stay at H.
Table 5: Prompts/ System messages in each language, together with their English translation marked with $T$. Language |  Real Review |  Generated Review
---|---|---
Chinese |  + 早餐丰富，服务态度很好，房间很整洁，离地铁站很近。优秀 |  + 这家酒店的位置极佳，位于市中心，靠近主要的旅游景点。酒店的装修新颖，服务人员非常专业和友好。早餐也很美味，种类丰富。
- 交通方便 价格性价比不错; 设施有些陈旧 可能因为疫情的缘故 卫生各个地方显得很陈旧的污渍 打扫看着不是那么上心 早上在酒店大堂看见任何一个服务员均很冷漠 没有打招呼的习惯希望能改进 |  - 该酒店地理位置优越，交通便利。我订的房间缺乏维护，房间内设施感觉比较陈旧，尤其是地毯，看上去非常脏旧，且有种异味。另外，酒店早餐品种不多，食品口味偏弱。最不满的是酒店的服务态度，服务员的服务意识强度不足，没有解决问题的积极性和主动性，完全不符合四星级酒店的标准。
English |  + Nicely furnished room and nicely decorated lobby. Room service is affordable and the receptionists, especially Mr. Omar, are usually eager to help with any queries. |  + The staff was extremely helpful and accommodating. Clean and well-furnished rooms. Central location with easy access to public transport.; The breakfast offerings could be more varied.
- The location was very central and the staff was nice and helped us.; The room smelled like cigarettes and there was mold in the bathroom |  - Nothing. Horrible experience.; Bad customer service, rooms were not clean and the food was below average.
French |  + Pas besoin de tourner pendant un moment pour trouver une place de Parking, un membre de l’hôtel se charge de prendre votre voiture; En soirée la rue n’est pas très familiale |  + L’hôtel est juste à côte de l’aéroport, parfait pour les vols tôt le matin. Les chambres étaient propres et confortables.; La diversité de la nourriture pourrait être améliorée.
- Bon emplacement dans Ankara. Personnel très agréable.; Petit déjeuner moyen. Fuites dans la salle de bain. N’est pas du niveau d’un hôtel 5 étoiles. |  - L’emplacement est bien, situé à proximité du centre-ville d’Ankara; Le manque de propreté est notable. La nourriture n’est pas de bonne qualité et le service à la clientèle laisse à désirer. Les chambres sont bruyantes et mal isolées, et les meubles sont vieux et usés.
German |  + Die Lage ist super zentral! Zu Fuß nur 10 Minuten vom Kızılay Platz entfernt. Die Zimmer waren sehr sauber und sehr geschmackvoll eingerichtet. Das Frühstück war sehr reichhaltig und lecker. Das Personal ist sehr freundlich und hilfsbereit.; Keine |  + Das Personal war sehr freundlich und das Zimmer war sehr sauber. Die Lage ist fantastisch, nahe an vielen Sehenswürdigkeiten. Das Frühstück hatte eine gute Auswahl an Speisen.
- Das hotel ist einfach viel zu alt. Ehrlich gesagt ist es 3 sterne; Nicht sauber habe nicht im hotel duschen können. Wir hatten gesamt zwei zimmer beide schlecht sauber gemacht worden |  - Die Lage des Hotels ist sehr zentral; Der Service war sehr schlecht und die Sauberkeit ließ zu wünschen übrig.
Italian |  + Struttura bella e pulita. Vicinissimo alla metro e centro città raggiungibile a piedi. Letto e cuscini comodissimi.; Il bagno turco aveva un odore strano.. muffa? |  + Camera molto pulita e silenziosa, personale professionale e disponibile. La colazione era abbondante e deliziosa. Posizione eccellente nel centro della città.
- Staff disponibile e cordiale; La camera doveva essere doppia matrimoniale, ma il letto effettivamente è una piazza e mezzo, la moquet era macchiata. |  - La posizione è centralissima, vicino a molti negozi e ristoranti.; Stanze poco pulite e personale maleducato. Colazione molto scarsa e Wi-Fi praticamente inesistente.
Korean |  + 직원들이 자기일처럼 성실히 도와주었으며 거실,방마다있는 에어컨도 방성능이 좋았으며,조명도 밝고 시설이 깨끗해서 편안했습니다; 세탁기는 있는데 드라이어나 건조대가 없어 옷 말리기가 불편했습니다 |  + 뉴 파크 호텔’은 매우 좋았습니다. 직원들이 친절하였고, 호텔 시설도 깨끗했습니다. 특히 아침식사가 훌륭했어요.; 방음이 잘 되지 않아 약간 소음이 있었습니다. 하지만 많이 거슬리지는 않았어요.
- 영등포역에서 가까워요. 침구가 깨끗합니다. 방은 깨끗; 전철이 있어서 시끄럽다는 후기가 있어 씨티뷰로 신청했지만 마찬가지로 소음이 심했습니다. 실내가 좁아 불편. 주차요금을 내야합니다. 화장실이 문이 없습니다 |  - 파리 중심부에 위치해 있어서 접근성이 좋았습니다.;객실이 너무 좁아서 실망했고, 침구가 깨끗하지 않았습니다. 또한, 냉난방 시스템이 제대로 동작하지 않아서 매우 불편했습니다.
Table 6: Random review samples with positive (+) and negative (-) sentiments across the 10 languages. Language |  Real Review |  Generated Review
---|---|---
Romanian |  + Gazde foarte amabile, primitoare, camera mare, curată, paturi confortabile, parcare proprie în fața hotelului, micul dejun, bufet, diversificat. |  + Servicii excelente, camere curate și spațioase. Spa-ul și piscina sunt minunate pentru relaxare.”; Nu am avut nicio problemă sau dezamăgire.
- E ffff cald in camere! Și aparatele stricate, camerele mici!! |  - Amplasarea hotelului este convenabilă, relativ aproape de majoritatea obiectivelor turistice.; Confortul camerei a fost sub orice critica, patul a fost foarte inconfortabil, iar curăţenia a fost precară. De asemenea, personalul a fost nepoliticos si neajutorător.
Russian |  + Шикарный завтрак, удобное расположение, доброжелательный персонал, согласовали ранний заезд; Немного грязное окно и тусклое освещение, но это мелочи |  + Отель с удобным расположением, близко к основным достопримечательностям. Очень чистые и комфортные номера. Отзывчивый и вежливый персонал.; Сложно было найти русскоговорящий персонал. Завтраки могли бы быть более разнообразными.
- метро рядом , санлитунь рядом.; персонал за стойками регистрации, это просто треш, мало того, что цены в чеке не совпадали с ценой на букинге, так еще и чекаут оформляли 50 минут! это дно товарищи! |  - Нет положительных сторон.; Большая часть персонала не говорит по-английски, что особенно усложнило процесс общения. На завтраке был ограниченный выбор блюд, а в номерах отличался сильный запах курения.
Spanish |  + Me gustó todo, super bonito, limpio, cómodo y en una zona muy animada, con un desayuno buffet genial; Nada |  + Un hotel muy bueno. Estaba muy limpio y moderno. En el corazón de la ciudad, cerca de las tiendas y restaurantes. El personal fue amigable y la comida era buena.; La única queja que tenía era que el wifi en mi habitación era un poco lento.
- Hotel con bien trato, y muy céntrico, limpio y comodo; Discoteca justo debajo de la habitación muchísimo ruido y música disco mucho volumen hasta altas horas. No se puede descansar |  - Buena ubicación central y la habitación estaba limpia; El desayuno era muy básico y la presión del agua en la ducha era bastante baja
Turkish |  + Kahvaltısı çok güzeldi. Tam bir Fransız kahvaltısıydı. Personel çok güler yüzlüydü. Otel çok merkezi bir konumdaydı. Tekrar Paris’e gelirsem tercihim yine buradan yana olur. |  + Otelin konumu ve erişilebilirliği mükemmel. Odalar temiz, konforlu ve fonksiyonel.; Oda sıcaklık ayarları biraz daha iyi olabilirdi.
- Bookingde sigarasız yazmasına rağmen ortak havalandırmadan sigara dumanı geliyordu wi-fi çalışmıyordu |  - Otelin konumu iyi.; Oda temizliği yetersizdi, yemekler çok tuzluydu ve personel pek yardımcı olmadı. Bu nedenlerden dolayı diğer turistlere bu oteli önermiyorum.
Table 7: Random review samples with positive (+) and negative (-) sentiments
across the 10 languages.

| uni-gram | bi-gram | tri-gram
---|---|---|---
Lang | Real | Gen | Real | Gen | Real | Gen
[length=0.5cm, width=0.3cm]CN |  酒店 房间 位置 |  酒店 位置 服务 |  (地理, 位置) (工作, 人员) (位置, 不错) |  (地理, 位置) (酒店, 位置) (服务, 人员) |  (地理, 位置, 不错) (地理, 位置, 优越) (员工, 服务, 态度) |  (酒店, 地理, 位置) (服务, 人员, 态度) (地理, 位置, 优越)
[length=0.5cm, width=0.3cm]US |  room location hotel |  room hotel location |  (great, location) (room, clean) (location, good) |  (room, clean) (staff, friendly) (customer, service) |  (staff, friendly, helpful) (staff, nice, helpful) (staff, speak, english) |  (room, clean, comfortable) (leave, lot, desire) (staff, friendly, helpful)
[length=0.5cm, width=0.3cm]FR |  chambre petit hôtel |  chambre hôtel personnel |  (petit, déjeuner) (salle, bain) (bien, situer) |  (petit, déjeuner) (chambre, propre) (chambre, spacieux) |  (rapport, qualité, prix) (hôtel, bien, situer) (bon, petit, déjeuner) |  (petit, déjeuner, varier) (chambre, propre, confortable) (chambre, spacieux, propre)
[length=0.5cm, width=0.3cm]DE |  Zimmer Lage Hotel |  Hotel Zimmer Lage |  (freundlich, Personal) (Personal, freundlich) (zentral, Lage) |  (Lage, Hotel) (freundlich, hilfsbereit) (Personal, freundlich) |  (Personal, freundlich, hilfsbereit) (Lage, freundlich, Personal) (Personal, super, freundlich) |  (Personal, freundlich, hilfsbereit) (lassen, wünschen, übrig) (Lage, Hotel, zentral)
[length=0.5cm, width=0.3cm]IT |  posizione camera colazione |  posizione hotel camera |  (ottimo, posizione) (posizione, ottimo) (personale, gentile) |  (camera, pulito) (posizione, centrale) (posizione, hotel) |  (rapporto, qualità, prezzo) (personale, gentile, disponibile) (ottimo, rapporto, qualità) |  (personale, cordiale, disponibile) (camera, pulito, confortevole) (personale, gentile, disponibile)
[length=0.5cm, width=0.3cm]KR |  너무 좋았습니다 위치가 |  매우 또한 호텔 |  (바로, 앞에) (너무, 좋았어요) (위치가, 너무) |  (호텔의, 위치는) (방음이, 되지) (또한, 직원들의) |  (위치가, 너무, 좋았고) (바로, 앞에, 있어서) (좋음, 직원, 친절함) |  (방음이, 되지, 않아) (방음이, 되지, 않아서) (친절하고, 도움이, 되었습니다)
[length=0.5cm, width=0.3cm]RO |  cameră mic hotel |  cameră personal hotel |  (mic, dejun) (personal, amabil) (cameră, mic) |  (mic, dejun) (personal, amabil) (personal, prietenos) |  (mic, dejun, bun) (mic, dejun, bogat) (mic, dejun, ok) |  (cameră, curat, confortabil) (aproape, centru, oraș) (mic, dejun, putea)
[length=0.5cm, width=0.3cm]RU |  номер отель завтрак |  отель номер персонал |  (постельный, бельё) (горячий, вода) (приветливый, персонал) |  (центр, город) (желать, хороший) (wi, fi) |  (соотношение, цена, качество) (постельный, бельё, полотенце) (персонал, говорить, английский) |  (оставлять, желать, хороший) (дружелюбный, готовый, помочь) (бесплатный, wi, fi)
[length=0.5cm, width=0.3cm]ES |  habitación ubicación hotel |  habitación hotel ubicación |  (personal, amable) (habitación, pequeño) (aire, acondicionado) |  (personal, amable) (dispuesto, ayudar) (ubicación, hotel) |  (relación, calidad, precio) (personal, amable, habitación) (cerca, torre, eiffel) |  (amable, dispuesto, ayudar) (personal, amable, dispuesto) (personal, amable, servicial)
[length=0.5cm, width=0.3cm]TR |  bir konum od |  oda otel olmak |  (konum, iyi) (gülmek, yüz) (od, Küçük) |  (otel, konum) (oda, temiz) (yardımcı, olmak) |  (çalışmak, gülmek, yüz) (personel, gülmek, yüz) (yer, yürümek, mesafe) |  (personel, son, derece) (oda, te, konfor) (oda, temiz, yeter)

Table 8: Most frequent n-grams for real and generated data, for each review
language. Country flag according to review location.

##  Appendix B Deception Detection

####  B.0.1 Naive Bayes.

##### Interpretability

As seen in LABEL:tab:eval_all2, the Naive Bayes model benefits more than the
XLM-RoBERTa model from having more training data. The Naive Bayes performance
on the default train-test data split is 85.91%, and the few-shot performance
drops dramatically close to the Random baseline, with a 54.42% accuracy.

Table 9 shows the most salient features of the Naive Bayes classifier, in both
default and few-shot training setups. We find that Spanish, Romanian, and
Russian words are the most frequent. Additionally, for both real and generated
reviews, words related to the hotel topic are most salient: room, breakfast,
service, affordable, hotel, location, stay. Compared to generated reviews,
real reviews tend to have more numerals, prepositions, and pronouns as salient
words: 20, two, we, for, on, in, with, very, while generated reviews have more
nouns: connection, downside, options, room, hotel, location stay, verbs:
leaving, want, listen, and adjectives: affordable, friendly, best.

Train-Test Setup |  Most salient words for the real reviews |  Most salient words for the generated reviews
---|---|---
default |  20, noi (we), min, euro, habitacion (room), alors (then), personne (person), two, covid, pt (for) |  lasciava (leaving), bağlantısı (connection), дружелюбным (friendly), лучшего (the best), желать (want), serviciul (the service), convenabilă (affordable), downside, seçenekleri (options)
few-shot |  si (yes), am (have), на (on), in, desayuno (breakfast), muy (very), cu (with), было (was), room, camera (room) |  ve, oda (listen), chambres (rooms), est (east), отеля (hotel), avec (with), une (a), sont (are), emplacement (location), séjour (stay)

Table 9: Top 10 most salient features, for real and generated hotel reviews
from Naive Bayes model trained on multilingual data, for each training setup:
default and few-shot.

####  B.0.2 XLM-RoBERTa Main Model.

Figure 8: Accuracy with XLM-RoBERTa model on different ratios of training
data. The deception classification model learns considerably from just 2,000
reviews (training data ratio of 10%), and increasing the training data does
not lead to high performance gains.

##  Appendix C Data Analysis

##### Word frequency.

We compute the most frequent n-grams across each language after pre-processing
the text, which is shown in Table 8.

##### Topic Modeling

Pre-processing. We preprocess the text with spaCy multilingual pipelines to
tokenize hotel reviews and lowercase tokens, remove stop words, and lemmatize
tokens.131313https://spacy.io/models

Figure 9: Visualization of topics used in the real and LLM-generated English
reviews. Points are colored red or blue based on the association of their
corresponding topics with AI-generated or real hotel reviews. The most
associated topics are listed under Top AI-Generated and Top Real headings.
Interactive version:
https://anonymous.4open.science/r/hotel_reviews_deception.

| uni-gram | bi-gram | tri-gram
---|---|---|---
Lang | Real | Gen | Real | Gen | Real | Gen
[length=0.5cm, width=0.3cm]US |  room location hotel |  room hotel location |  (great, location) (room, clean) (location, good) |  (room, clean) (staff, friendly) (customer, service) |  (staff, friendly, helpful) (staff, nice, helpful) (staff, speak, english) |  (room, clean, comfortable) (leave, lot, desire) (staff, friendly, helpful)
[length=0.5cm, width=0.3cm]RO |  cameră mic hotel |  cameră personal hotel |  (mic, dejun) (personal, amabil) (cameră, mic) |  (mic, dejun) (personal, amabil) (personal, prietenos) |  (mic, dejun, bun) (mic, dejun, bogat) (mic, dejun, ok) |  (cameră, curat, confortabil) (aproape, centru, oraș) (mic, dejun, putea)

Table 10: Most frequent n-grams for real and generated data, for two
languages: English and Romanian. Country flag according to review location. We
highlight in bold the words that are different between real and AI-generated
reviews. The rest of the languages are shown in Appendix Table 8. We find that
most topics and their corresponding keywords overlap between real and AI-
generated reviews and repeat across languages: staff, location, food, noise,
hospitality, cleanliness, budget, amenities


# Annotating Norwegian Language Varieties on Twitter for Part-of-Speech

Petter Mæhlum1, Andre Kåsen2, Samia Touileb3 and Jeremy Barnes4

1University of Oslo
2National Library of Norway
3University of Bergen
4University of the Basque Country

<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Norwegian Twitter data poses an interesting challenge for Natural Language
Processing (NLP) tasks. These texts are difficult for models trained on
standardized text in one of the two Norwegian written forms (Bokmål and
Nynorsk), as they contain both the typical variation of social media text, as
well as a large amount of dialectal variety. In this paper we present a novel
Norwegian Twitter dataset annotated with POS-tags. We show that models trained
on Universal Dependency (UD) data perform worse when evaluated against this
dataset, and that models trained on Bokmål generally perform better than those
trained on Nynorsk. We also see that performance on dialectal tweets is
comparable to the written standards for some models. Finally we perform a
detailed analysis of the errors that models commonly make on this data.

Annotating Norwegian Language Varieties on Twitter for Part-of-Speech

Petter Mæhlum1, Andre Kåsen2, Samia Touileb3 and Jeremy Barnes4 1University of
Oslo 2National Library of Norway 3University of Bergen 4University of the
Basque Country<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Norwegian Twitter data poses an interesting challenge for Natural Language
Processing (NLP) tasks. Not only do these data represent a set of noisy, user-
generated texts with the kinds of orthographic variation common on social
media, but also because there is a considerable number of tweets written in
dialectal Norwegian. These dialectal variants are quite common and add another
level of difficulty for NLP models trained on clean data in one of the two
Norwegian written forms (Bokmål or Nynorsk).

Barnes et al. (2021) compiled a dataset of tweets classified according to
whether they are written in primarily Bokmål, Nynorsk, or a dialect of
Norwegian. We build upon this work by annotating a subset for Part-of-Speech
(POS). We investigate to what extent available Norwegian POS tagging models,
that were trained on Bokmål and Nynorsk Universal Dependency data Nivre et al.
(2020), perform on this Twitter dataset.

To this end, we use five POS models: three off-the-shelf models, and two
developed for the purpose of this work. Each of these models was trained on
either a dataset of Bokmål or Nynorsk texts. We explore the performance of
each model in terms of accuracy, and investigate which standardized written
form can be used as training data and yield good results for non-standardized
dialectal texts.

The main contributions of this work are:

  * •

we annotate a moderately sized Twitter dataset with POS labels and include
metadata related to which language variety it belongs (Bokmål, Nynorsk,
Dialect, or Mixed),

  * •

we perform a detailed error analysis of common model errors specific to our
Twitter data,

  * •

we include our insights into the annotation process for POS tagging of non-
standardized written forms,

  * •

we release two spaCy models built on top of a Norwegian BERT model.

##  2 Background

Johannessen (1990) outlined a system for automatic morphosyntactic analysis of
Norwegian nouns in the framework of Koskenniemi (1983). This was among the
first systems, if not even the very first, that automatically assigned
Norwegian texts any morphological information. The first widely used tagger,
however, was developed within the Taggerprosjektet111The project ran from
April 1996 to December 1998. and came to be known as the Oslo-Bergen
Tagger222https://github.com/noklesta/The-Oslo-Bergen-Tagger (OBT). Rather than
continuing and expanding the system of Johannessen (1990), OBT was implemented
in the framework of Karlsson (1990). OBT was initially a rule-based Constraint
Grammar tagger for Norwegian Bokmål. Later, both support for Norwegian Nynorsk
and a statistical disambiguation component were added Johannessen et al.
(2012). But one drawback of OBT is that it is made for written, edited text,
and therefore might not scale well to sources that are not standardised.

Extending tagger coverage to spoken Norwegian dialect transcription, on the
other hand, was the objective of both Nøklestad and Søfteland (2007) and Kåsen
et al. (2019). Both sampled data either from the Norwegian part of the Nordic
Dialect Corpus (NDC, Johannessen et al. (2009)) or the Language Infrastructure
made Accessible (LIA) Corpus.333https://tekstlab.uio.no/LIA/korpus.html
Annotations are found in the respective treebanks of the corpora and are
accounted for in Øvrelid et al. (2018) and Kåsen et al. (2022).

Besides Norwegian, there is a large amount of work on the difficulty of
processing noisy data from social media Xu et al. (2015), including the
difficulty of POS tagging on social media Albogamy and Ramasy (2015), with
dialectal variation Jørgensen et al. (2015), or whether lexical normalization
is helpful van der Goot et al. (2017). However, Norwegian currently lacks any
of these studies.

##  3 Data

Resources for evaluating NLP pipeline tasks for Norwegian are scarce. The only
dataset available for standard NLP tasks such as POS tagging, lemmatization,
and parsing is the Norwegian Dependency Treebank (NDT, Solberg (2013), Solberg
et al. (2014)) that has been converted to the Universal Dependencies standard
(Øvrelid and Hohle, 2016). There is, however, a notable exception when it
comes to transcribed spoken dialectal data, where the LIA and NDC treebanks as
mentioned above are available with annotations for POS tags, morphological
features, lemmas, and dependency-style syntax. Despite this, the transcribed
texts in the LIA and NDC corpora do not share the same characteristics as the
Twitter data. Twitter contains spelling errors and emoji,444Emoji has recently
gained some interest in the linguistic literature (see
https://ling.auf.net/lingbuzz/005981) along with mentions and hashtags. We
observe that although our Twitter data contains some characteristics of spoken
Norwegian, such as subjectless sentences as in 3, which is otherwise within
the spelling norms, the spelling conventions differ from those of LIA and NDC,
making it difficult to directly compare the data.

Kommer nok hjem snart . Comes probably home soon . ‘(Unspecified) probably
comes home soon .’

In LIA and NDC, all transcriptions are done according to a Norwegian-based
semi-phonetic standard Hagen et al. (2015), with strict marking of vowel
quantity, palatalization, retroflexion, and more. We see that writers on
Twitter do not conform to any specific spelling norm when writing in their own
or another dialect. This means that although not all dialectal traits from a
dialect are faithfully preserved, this still leads to much dialectal variation
in the Twitter data, as things that could have had a common spelling is
spelled according to the author’s own preference. Especially phonetic
differences are often not indicated on Twitter. Because of this, we needed a
separate dataset that could be used to evaluate how various systems for
Norwegian POS-tagging work on dialectal text as it is found on real data from
social media platforms.

We sampled a balanced subset of the dataset introduced by Barnes et al.
(2021), who developed to develop a dialect classifier for Norwegian tweets,
with the aim to be able to further investigate issues related to dealing with
dialectal data on Twitter. This subset includes a selection of 38 tweets in
Bokmål, 31 tweets in Nynorsk, and 35 in dialects, which comprises their full
test set. We acknowledge that the size of the dataset is small. The POS-tagged
dataset is subject to restrictions due to it containing personal information,
but is available upon request.

| Bokmål | Nynorsk | Dialectal | Mixed | All
---|---|---|---|---|---
PUNCT | 211 | 15.72% | 151 | 13.92% | 123 | 11.27% | 35 | 13.46% | 520 | 13.76%
NOUN | 168 | 12.52% | 150 | 13.82% | 116 | 10.63% | 37 | 14.23% | 471 | 12.47%
VERB | 157 | 11.7% | 130 | 11.98% | 129 | 11.82% | 24 | 9.23% | 440 | 11.65%
PRON | 134 | 9.99% | 97 | 8.94% | 140 | 12.83% | 29 | 11.15% | 400 | 10.59%
ADP | 120 | 8.94% | 89 | 8.2% | 85 | 7.79% | 21 | 8.08% | 315 | 8.34%
AUX | 78 | 5.81% | 84 | 7.74% | 93 | 8.52% | 19 | 7.31% | 274 | 7.25%
ADJ | 103 | 7.68% | 67 | 6.18% | 88 | 8.07% | 13 | 5.0% | 271 | 7.17%
PROPN | 92 | 6.86% | 74 | 6.82% | 50 | 4.58% | 18 | 6.92% | 234 | 6.19%
ADV | 77 | 5.74% | 68 | 6.27% | 74 | 6.78% | 11 | 4.23% | 230 | 6.09%
SCONJ | 46 | 3.43% | 42 | 3.87% | 43 | 3.94% | 5 | 1.92% | 136 | 3.6%
DET | 49 | 3.65% | 38 | 3.5% | 33 | 3.02% | 14 | 5.38% | 134 | 3.55%
CCONJ | 34 | 2.53% | 38 | 3.5% | 44 | 4.03% | 11 | 4.23% | 127 | 3.36%
PART | 36 | 2.68% | 22 | 2.03% | 29 | 2.66% | 9 | 3.46% | 96 | 2.54%
X | 16 | 1.19% | 15 | 1.38% | 9 | 0.82% | 10 | 3.85% | 50 | 1.32%
NUM | 11 | 0.82% | 8 | 0.74% | 14 | 1.28% | 2 | 0.77% | 35 | 0.93%
INTJ | 7 | 0.52% | 6 | 0.55% | 14 | 1.28% | 1 | 0.38% | 28 | 0.74%
SYM | 3 | 0.22% | 6 | 0.55% | 7 | 0.64% | 1 | 0.38% | 17 | 0.45%
Table 1: Distribution of each POS-tag in the Twitter test set, along with the
total number of occurrences for each tag and their corresponding percent-wise
distribution.

###  3.1 Norwegian Dialects

Norwegian is considered to have four main dialect groups based on four
different traits. This has been a controversial matter and the four-way divide
essentially follows Christiansen (1954). There are also recent proponents of a
two-way divide Skjekkeland (1997). The four-way distinctions have a Northern,
Middle, Western, and Eastern group, whereas the two-way divide only operates
with a Western and Eastern group. But these distinctions are made with traits
from the spoken language. And, as Mæhlum and Røyneland (2012, p. 29) point
out, there is a discrepancy between how dialectologists and lay people
classify dialects. What sort of dialectal traits Twitter users choose to
include may therefore lead to a different kind of divide than one can find in
the dialectology literature. That being said, Venås (1990) shows that there
has been a long tradition of writing in dialect, where the oldest text in
Venås (1990) dates back to 1525.

###  3.2 POS Annotations

The texts from the test set were annotated using the Universal Dependencies
POS tagset.555https://universaldependencies.org/u/pos/ The tweets were
tokenized with NLTK’s tokenizer (Bird et al., 2009) and split into sentences
manually. The NLTK tokenizer was chosen over other tokenizers as our
preliminary testing on our Twitter dataset shows that it performs better on
noisy Norwegian data. The tokenized data was then pre-annotated with Stanza’s
Bokmål tokenizer to alleviate the annotation task. The remaining task was to
correct each POS-tag for these pre-annotated sentences. One annotator
annotated the whole test set, while two other annotators annotated two
separate subsets of the dataset to give an indication of how robust the
annotations were. All three annotators were trained in linguistics and
language technology, and are native Norwegian speakers. An overview of the
distribution of each POS-tag for each written form is reported in table 1. We
see that the percent-wise distribution of POS-tags is similar in Bokmål,
Nynorsk and All, but that the PRON tag is somewhat more frequent than the VERB
tag in the Dialect tweets. This could be due to the fact that some dialectal
tweets only appear as dialectal due to specific dialectal pronouns.

###  3.3 Inter-Annotator Agreement

The inter-annotator score for the full doubly-annotated test set, using
Cohen’s $\kappa$, was 0.87, indicating quite high agreement. Looking at the
specific categories, we see that the agreement was 0.92 for Bokmål, 0.83 for
Nynorsk, and 0.88 for dialectal tweets. No specific error patterns are
observed that would account for the difference in scores, but all annotators
have more familiarity with the Bokmål variant. One common point of
disagreement across all is the copula verb å være ‘to be’, which according to
the UD guidelines should be tagged as AUX. This was commonly tagged as VERB by
one of the annotators. There is also some disagreement when it comes to words
such as opp ‘up’, and ned ‘down’, which can be tagged both as adverbs (ADV),
adpositions (ADP), and verbal particles. Since there is no tag for verbal
particles in UD, the annotators had to chose between the other two. Cases of
disagreement were solved by discussing tags where one or more annotators
disagreed.

##  4 Experiments

We test several models trained on available Norwegian UD datasets on our
Twitter data. Specifically, we compare OBT, Stanza, UDPipe 2.0, a simple
BiLSTM model, as well as training our own spaCy models.

Both Stanza (Qi et al., 2020) and UDPipe 2.0 (Straka, 2018) use a BiLSTM which
takes features from 1) pre-trained word embeddings, 2) a trainable frequent
word embedding that is randomly initialized before training, and 3) character-
level LSTM features. While UDPipe only uses a softmax layer for
classification, Stanza instead uses a biaffine classifier to ensure
consistency between the UPOS and XPOS predictions.

The BiLSTM model we use is a simplified version of the models used in UDPipe
and Stanza. The model does not take any pre-trained word embeddings as
features, but rather uses the vocabulary of the dataset it is trained on to
create the embeddings. The model uses a linear layer for classification.

The spaCy models are newly trained during the present work, and will be
released publicly in the near future. Since spaCy is a fully configurable and
trainable pipeline, we used the Norwegian BERT model described in Kummervold
et al. (2021) with a shared embedding layer for a tagger, morphologizer, and
trainable lemmatizer in an effort to optimize the tagger task.

##  5 Results and Discussion

| Bokmål | Nynorsk | Dialect | All
---|---|---|---|---
OBT Bokmål | 77.8 | - | 62.3 | -
OBT Nynorsk | - | 73.1 | 57.3 | -
BiLSTM_UD Bokmål | 80.5 | 63.8 | 63.4 | 70.2
BiLSTM_UD Nynorsk | 62.3 | 76.2 | 56.7 | 64.6
BiLSTM_UD Nynorsk_LIA | 47.6 | 56.2 | 43.9 | 48.9
Stanza Bokmål | 86.6 | 67.5 | 69.5 | 75.4
Stanza Nynorsk | 45.8 | 82.8 | 52.0 | 58.1
UDPipe Bokmål | 89.6 | 76.1 | 72.9 | 80.4
UDPipe Nynorsk | 74.4 | 82.9 | 63.2 | 73.2
spaCy Bokmål | 87.9 | 85.7 | 83.3 | 85.8
spacy Nynorsk | 62.5 | 83.2 | 65.0 | 69.6
Table 2: Accuracy on our Twitter test set using five different models trained
on either Bokmål or Nynorsk datasets.

Table 2 gives an overview of the accuracy on the Twitter test set using our
five models trained on either Bokmål or Nynorsk data. Note that due to their
small number, we do not include the mixed category by itself, but these tweets
are included in the ALL column. On our twitter Bokmål test set, the best model
is the UDPipe Bokmål model, which achieves 89.6 accuracy. Generally, the
models trained on the UD Bokmål data are consistently better than the Nynorsk
versions on this data (an average of 26.5 percentage points (pp)).
Interestingly, the same is not true for the Twitter Nynorsk data. One may
assume that models trained on the Nynorsk UD data would always perform better,
but in fact, the best performing model is the spaCy model trained on Bokmål
(85.7 acc) and on average, the models trained on UD Bokmål perform 4.9 pp
worse.

Finally, on the dialectal Twitter data, the spaCy Bokmål model once again
performs best (83.3). Again training on the Bokmål data generally performs
12.8 pp better than training on Nynorsk data. This may be due to the subset of
dialectal tweets, as a manual inspection showed a large number of tweets from
Central and Northern dialects, which share more features with Bokmål. A larger
number of tweets from Western and Southern dialects could potentially change
this. At the same time, however, it seems clear that the spaCy Bokmål model
performs quite well on all the Twitter test data (85.8 acc), so it may simply
be a stronger model.

###  5.1 Error Analysis

We note that the models struggle with features that are typical of the noisy
Twitter data containing several misspellings. One concrete example is å, which
in normative writing most likely refers to the identically spelled infinitive
marker å ‘to’. However, as dialectal writing is much more relaxed, alternative
spellings create new homographs that need to be dealt with. We see that some
cases of ‘å’ refer to the conjunction og ‘and’, which in many dialects is
homophonous with å. We also note that many of the errors come from erroneously
tagging pronouns as other word classes, such as INTJ, PART, or NOUN. One
reason why there are many errors of this type might simply be because these
are frequent indicators of dialect. Barnes et al. (2021) show that certain
pronouns such as æ and mæ (both ‘I’) are highly correlated with dialectal
tweets. They are in some cases the only dialectal indicator in a tweet.
Finally, we observe that there are problems with annotating enclitic elements
and words that should have been written separately, or conversely, with
compound words that have been split. The two latter problems are not exclusive
to dialects, but are common in informal writing. Enclitic elements, such as
the enclitic negation (’kke, ’kje, ’che, etc.) and enclitic pronouns such as
’n ‘he, him’ and ’a ‘she, her’ are sometimes added after words, and sometimes
without any punctuation, and there are no tokenizers that the authors are
aware of that can correctly separate out these enclitic elements. For example,
a spelling like ekkje, ‘is not’, which is the copula e with the enclitic
negation adverb kkje ‘not’ written as one word, has this issue. The same
happens with other words that according to the norm should be written as two
words, such as i dag ‘today’, being written as idag. This leads to tokens with
multiple possible POS-tags. In these cases the annotators would consider what
would be the best functional fit. For example, the resulting adverbial phrase
idag can be annotated as an adverb, and verbs negated by enclitics are tagged
as verbs. However, these are the annotators’ judgements, and their proper
treatment is not clear from the UD guidelines.

##  6 Conclusion

In this paper, we have introduced the first dataset of Norwegian tweets
annotated for Part-of-Speech, that also include the metadata for the language
variety of each tweet (Bokmål, Nynorsk, Dialect, or Mixed). We tested several
POS taggers trained on UD data and show that, for our Twitter data, it is
generally better to train on the UD Bokmål data, even if testing on Nynorsk or
Dialect. Our detailed error analysis showed that the models generally have
problems with dialectal pronouns and unfamiliar compounds. Finally, we release
the newly trained spaCy models, and make our annotated data available on
request, in order to enable the reproduction of our results.

## Acknowledgements

We thank Alexandra Wittemann and Annika Willoch Olstad for their valuable
contributions as annotators.

Parts of this work were supported by industry partners and the Research
Council of Norway with funding to MediaFutures: Research Centre for
Responsible Media Technology and Innovation, through the centers for Research-
based Innovation scheme, project number 309339.

## References

  * Albogamy and Ramasy (2015) Fahad Albogamy and Allan Ramasy. 2015.  Towards POS tagging for Arabic tweets.  In _Proceedings of the Workshop on Noisy User-generated Text_ , pages 167–171, Beijing, China. Association for Computational Linguistics.
  * Barnes et al. (2021) Jeremy Barnes, Petter Mæhlum, and Samia Touileb. 2021.  NorDial: A preliminary corpus of written Norwegian dialect use.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 445–451, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  "O’Reilly Media, Inc.".
  * Christiansen (1954) Hallfrid Christiansen. 1954.  _Hovedinndelingen av norske dialekter_ , volume 1954.  Bymålslaget Oslo.
  * Hagen et al. (2015) Kristin Hagen, Live Håberg, Eirik Olsen, and Ashild Søfteland. 2015.  Transkripsjonsrettleiing for LIA.  Technical report, Technical report.
  * Johannessen (1990) Janne B. Johannessen. 1990.  _Automatisk morfologisk analyse og syntese_.  Novus forlag, Oslo.
  * Johannessen et al. (2012) Janne B. Johannessen, Kristin Hagen, André Lynum, and Anders Nøklestad. 2012\.  Exploring Newspaper Language: Using the web to create and investigate a large corpus of modern Norwegian.  _Studies in Corpus Linguistics_ , 49:51.
  * Johannessen et al. (2009) Janne B. Johannessen, Joel Priestley, Kristin Hagen, Tor Anders Åfarli, and Øystein Alexander Vangsnes. 2009.  The Nordic Dialect Corpus–an advanced research tool.  In _Proceedings of the 17th nordic conference of computational linguistics (nodalida 2009)_ , pages 73–80.
  * Jørgensen et al. (2015) Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2015.  Challenges of studying and processing dialects in social media.  In _Proceedings of the Workshop on Noisy User-generated Text_ , pages 9–18, Beijing, China. Association for Computational Linguistics.
  * Karlsson (1990) Fred Karlsson. 1990.  Constraint grammar as a framework for parsing running text.  In _COLING 1990 Volume 3: Papers presented to the 13th International Conference on Computational Linguistics_.
  * Kåsen et al. (2019) Andre Kåsen, Kristin Hagen, Anders Nøklestad, and Joel Priestley. 2019.  Tagging a Norwegian dialect corpus.  In _Linköping Electronic Conference Proceedings_ , pages 350–355. Linköping University Electronic Press.
  * Kåsen et al. (2022) Andre Kåsen, Kristin Hagen, Anders Nøklestad, Joel Priestly, Per Erik Solberg, and Dag Trygve Truslew Haug. 2022.  The Norwegian Dialect Corpus Treebank.  In _Proceedings of the Language Resources and Evaluation Conference_ , pages 4827–4832, Marseille, France. European Language Resources Association.
  * Koskenniemi (1983) Kimmo Koskenniemi. 1983.  _Two-level morphology: A general computational model for word-form recognition and production_ , volume 11.  University of Helsinki, Department of General Linguistics Helsinki, Finland.
  * Kummervold et al. (2021) Per E Kummervold, Javier De la Rosa, Freddy Wetjen, and Svein Arne Brygfjeld. 2021\.  Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 20–29.
  * Mæhlum and Røyneland (2012) Brit Mæhlum and Unn Røyneland. 2012.  _Det norske dialektlandskapet: innføring i studiet av dialekter_.  Cappelen Damm akademisk.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal Dependencies v2: An evergrowing multilingual treebank collection.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4034–4043, Marseille, France. European Language Resources Association.
  * Nøklestad and Søfteland (2007) Anders Nøklestad and Åshild Søfteland. 2007.  Tagging a Norwegian speech corpus.  In _Proceedings of the 16th Nordic Conference of Computational Linguistics (NODALIDA 2007)_ , pages 245–248.
  * Øvrelid and Hohle (2016) Lilja Øvrelid and Petter Hohle. 2016.  Universal dependencies for Norwegian.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1579–1585.
  * Øvrelid et al. (2018) Lilja Øvrelid, Andre Kåsen, Kristin Hagen, Anders Nøklestad, Per Erik Solberg, and Janne B. Johannessen. 2018.  The LIA treebank of spoken Norwegian dialects.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 101–108, Online. Association for Computational Linguistics.
  * Skjekkeland (1997) Martin Skjekkeland. 1997.  _Dei norske dialektane: tradisjonelle særdrag i jamføring med skriftmåla_.  Høyskoleforl.
  * Solberg (2013) Per Erik Solberg. 2013.  Building gold-standard treebanks for Norwegian.  In _Proceedings of the 19th Nordic Conference of Computational Linguistics (NODALIDA 2013)_ , pages 459–464.
  * Solberg et al. (2014) Per Erik Solberg, Arne Skjærholt, Lilja Øvrelid, Kristin Hagen, and Janne Bondi Johannessen. 2014.  The Norwegian dependency treebank.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 789–795, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Straka (2018) Milan Straka. 2018.  UDPipe 2.0 prototype at CoNLL 2018 UD shared task.  In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ , pages 197–207, Brussels, Belgium. Association for Computational Linguistics.
  * van der Goot et al. (2017) Rob van der Goot, Barbara Plank, and Malvina Nissim. 2017.  To normalize, or not to normalize: The impact of normalization on part-of-speech tagging.  In _Proceedings of the 3rd Workshop on Noisy User-generated Text_ , pages 31–39, Copenhagen, Denmark. Association for Computational Linguistics.
  * Venås (1990) Kjell Venås. 1990.  _Den fyrste morgonblånen: tekster på norsk frå dansketida_.  Novus forlag.
  * Xu et al. (2015) Wei Xu, Bo Han, and Alan Ritter, editors. 2015.  _Proceedings of the Workshop on Noisy User-generated Text_. Association for Computational Linguistics, Beijing, China.