0 Research

# ConVIScope: Visual Analytics for Exploring Patient Conversations

Raymond Li11footnotemark: 1 Enamul Hoque22footnotemark: 2 Giuseppe
Carenini11footnotemark: 1 Richard Lester33footnotemark: 3 Raymond
Chau33footnotemark: 3
Department of Computer Science, University of British Columbia
School of Information Technology, York University
Department of Medicine, University of British Columbia e-mail: {raymondl,
<EMAIL_ADDRESS><EMAIL_ADDRESS>{rlester,
<EMAIL_ADDRESS>
###### Abstract

The proliferation of text messaging for mobile health is generating a large
amount of patient-doctor conversations that can be extremely valuable to
health care professionals. We present ConVIScope, a visual text analytic
system that tightly integrates interactive visualization with natural language
processing in analyzing patient-doctor conversations. ConVIScope was developed
in collaboration with healthcare professionals following a user-centered
iterative design. Case studies with six domain experts suggest the potential
utility of ConVIScope and reveal lessons for further developments.

Human-centered computingVisualizationVisualization design and evaluation
methods  ConVIScope Views (see §3): Metadata (A), Topics (B), Analysis (C) and
Conversations (D). Trend View shown in Figure 1.

Introduction

Text messaging has become an important component of mobile health (mHealth)
due to its wide adoption and affordable cost [17]. As seen during the COVID-19
pandemic, physicians and health systems worldwide are shifting towards
virtualized treatment programs to minimize physical visits and reduce hospital
transmissions [41]. In addition, studies involving texting-based mHealth
programs have demonstrated their success at lifestyle modification [16],
treatment adherence [28, 31], as well as disease monitoring [24].

The conversations generated from texting-based mHealth services can reveal
valuable information about the patients and the healthcare quality. For
example, at the micro-level health professionals may be interested to quickly
get an overview of what patients are discussing about, such as a concern with
medications or logistical problems. At the macro-level, they may be interested
to inform policy-making by knowing how conversations vary across different
patient groups or to monitor what patients are complaining about in different
regions. Visual text analytic solutions could be effective in supporting the
above tasks by helping clinicians to discover interesting trends and patterns
among the patient population, and assist the health care administrators in
optimizing the delivery of services. However, most existing solutions applied
Natural Language Processing (NLP) techniques on conversations extracted from
online sources such as blogs [19, 20], forums [12, 11], and social media [44,
43, 36]. In contrast, visualizations that focus specifically to the healthcare
domain primarily targeted healthcare records (e.g. [33, 10, 25]) or online
forums of health communities [26] as opposed to text messages between patients
and doctors. Finally, a few commercial mhealth platforms provide static charts
for report generation but they do not support interactive visual exploration
of a large set of messages [22].

In this paper, we conducted a design study with healthcare professionals to
understand how to effectively combine NLP with visualization to support them
in discovering useful trends and patterns from patient-doctor conversations
and monitoring the delivery of health care services. By following
visualization design study methodologies [30, 35], we first characterized the
domain by interviewing experts to elicit their requirements. We then performed
data abstractions on real datasets obtained from one of our collaborators who
is a healthcare researcher. After an iterative design process, we developed
ConVIScope: a visual text analytic system that supports the exploration and
analysis of patient-doctor conversations. Our contributions include: (1) A
requirement analysis from interviews with health professionals, which revealed
the unique challenges in understanding patient-doctor conversations and guided
us to identify user tasks and key design needs. (2) The design and
implementation of ConVIScope to meet the identified user requirements. To the
best of our knowledge, ConVIScope is the first visual text analytics framework
designed specifically for monitoring and exploring a large collection of
patient-doctor conversations. (3) An evaluation with six healthcare
professionals which provides initial evidence of the potential utility of
ConVIScope and reveals lessons for further developing visual analytics systems
for patient-doctor conversations.

##  1 Related Work

Visualization for Health Care: Several early works on visualizing healthcare
data mainly focused on electronic health records (EHR) data (e.g. [33, 40, 42,
15, 6, 14]). For example, Lifelines [33] organized individual patient records
in expandable facets, while Lifelines2 [40] provided a temporal summary of
combined records. Lifeflow [42] used icicle trees to organize records into
hierarchical structures to reveal general patterns and trends. Others have
extracted and visualized text features from clinical data. For example,
AdaptEHR [21] extracts medical concepts and maps them onto biomedical
ontologies, while HARVEST [18] displays extracted concepts using a tag cloud
and a timeline. RetainVis helps users interpret the prediction of neural
models on medical records [25]. Unlike all the above works, which focus on
summarizing and interpreting structured medical records, we focus on patient-
doctor conversations.

Visual Text Analytics for Conversations:  With the exponential growth of
online conversations, there has been increasing interest in text analytics for
such data. Several works focused on visualizing automatically extracted topics
and sentiments from conversations [19, 20, 9, 8]. For example, ConVis [19]
shows the conversation structure and supports multi-faceted exploration using
topics and authors. ConToVi helps to explore speaker behavior patterns in a
multi-party conversation using animations [9], while others have focused on
organizing a large number of topics into hierarchical tree structures [8, 20].
Overall, although the above body of work targets multi-party conversations,
they are not designed to deal with specific tasks of health professionals in
exploring patient-doctor conversations, as we do in this paper.

In this regard, work studying the visualization of medical conversations is
very limited [28, 2, 3]. For example, Discursis [1] has been applied to
provide an overview of a patient-doctor conversation by visualizing the
thematic contents as well as dynamics of turn-takings across speakers [2, 3].
However, these techniques mainly focus on the conversational structure while
ignoring content facets, that we found to be critical in our study. In
contrast, VisOHC [26] does visualize sentiments and topics of conversation
threads, but from health forums, not from patient-doctor conversations. Also,
it only focuses on supporting online health community administrators. Finally,
MedStory [38] aims to convey psychological and social aspects of a medical
condition, by extracting and visualizing medical concepts and emotions. But
again, its target are social media conversations and not patient-doctor
messages.

##  2 Tasks and Data Abstractions

###  2.1 Requirement Analysis

In order to gather an initial set of requirements, we performed semi-
structured interviews with eight healthcare professionals including 3
practicing physicians, 4 health researchers, and 1 healthcare administrator.
The interviews were open-ended in nature, where participants were asked about
their needs for analysis and what they need to achieve with the insights
derived from conversations with patients. In particular, the primary
objectives of the healthcare professionals can be categorized into two scales.
On a macro-scale, users require high-level insights across regions and
demographics with the goal of designing more effective policies or supporting
research objectives. On a micro-scale, users are interested in analyzing
conversations from a particular patient group or clinic for identifying
specific patient needs to improve the efficiency of the healthcare delivery.
Critically, the two scales are not orthogonal, as micro-level analysis can
provide context for the macro-level and vice versa. In summary, we organized
these goals as three generic requirements on both scales. First, the insights
should allow the users to understand and compare the needs of the patients
across different regions and demographics (UR-1). Second, our interface should
support evaluating the efficiency and quality of the healthcare services
(UR-2). Lastly, it is necessary for users to explore, identify and explain
emerging issues and problems faced by the patients (UR-3).

###  2.2 Task Model

Based on the identified user requirements, we derived a set of analytical
tasks that helps the domain experts in understanding and extracting insights
from the patient-doctor conversations.

T1. What is being discussed? Participants confirmed the importance of
understanding the underlying topics being discussed in order to identify the
patient needs (UR-1) and the emerging issues (UR-3) from the conversations.
Often, they are interested in analyzing conversations with respect to
predefined topics of medical relevance (e.g. treatment, prescriptions). Other
times, they are interested in exploring more specific and/or unforeseeable
emerging topics beyond the pre-defined ones (e.g. mental problems from COVID-
induced lockdown). To support these analytical needs, we extract three types
of topics (pre-defined, discovered, and user-defined).

T2. How are the attitudes being expressed? In addition to understanding what
was discussed, users are also interested in how sentiments are expressed in
conversations, which can be a very useful indicator of patient satisfaction
with the quality and efficiency of their healthcare service (UR-2). For
example, the attitudes expressed by patients while discussing logistic issues
(e.g., outpatient, hospitalization) can provide insights on the efficiency of
the healthcare service delivery. Moreover, visualizing the sentiment
associated with a topic can help users to make policy decisions; for instance,
addressing concerns about side-effects of a particular treatment.

T3. What are the trends for topics and opinions? Understanding the trend
associated with topics such as diseases and symptoms can help the users
determine whether there are certain problems or events causing the prevalence
of such topics (UR-3). For instance, the user might find an increasing number
of conversations mentioning financial troubles due to the pandemics. Also,
opinion trends can assist the user in assessing whether some aspects of the
healthcare quality or patient satisfaction are changing on an aggregated level
as well as deriving possible explanations for such changes (UR-2).

T4. How do the conversations differ across different demographics and
locations/clinics? Age plays an important role when considering phenotypic
changes in health and disease [13], and different patient groups (ex. cancer
patients) may have different needs or face different problems (UR-1).
Additionally, users are also interested in differences between conversations
across clinics and regions, because they can reveal and help explain emerging
trends occurring on a macro scale (UR-1, UR-3), providing insights on how to
improve health policies that are tailored to those differences.

###  2.3 Data Model

We pre-process the dataset by filtering out conversations containing less than
3 messages to focus on those that are long enough to perform meaningful
studies. Each conversation is tagged with patient features such as location,
patient group, age group, and gender. Additional metadata, including topics
and sentiment, are derived from text content using the following NLP
techniques.

Pre-defined Topics: To satisfy the need of organizing conversations based on
general medical knowledge (T1), in close collaboration with health
professionals, we define a set of hierarchically organized topic labels (e.g.
logistics, treatment, social - see ConVIScope: Visual Analytics for Exploring
Patient Conversations (B)). We refer to these labels as pre-defined topics.
These topics are assigned to conversations by supervised classifiers trained
on human-annotated data. We used logistic regression with bag-of-word features
for this task as it is a simple and interpretable classifier. Moreover, it has
shown promising results in recent work on predicting patient needs from social
media conversations, with limited training data [23, 27]. We involved
healthcare professionals and medical students for annotating 5K conversations
to train the classifiers. To measure the inter-rater reliability, we asked
four annotators to label the same set of $200$ conversations. The Cohen’s
Kappa coefficient was $0.64$, indicating a moderate level of agreement [29].

Machine-discovered Topics: In order to discover emerging topics beyond pre-
defined topics (T1), we used the standard Latent Dirichlet allocation from the
Gensim library [34] for unsupervised topic model [4] with $k=3$ as the number
of topics. The label for each topic comprises the five most likely words given
that topic.

User-defined Topics: Predefined topics cover generic medical terms like
diagnosis, treatment, exercise etc. However, during an analysis, users may
also become interested in more specific topics. ConVIScope allows the user to
specify such topics on the fly enabling them to find conversations that
contain a user-provided phrase. Such a mechanism also finds conversations with
semantically similar phrases to the given one, based on cosine similarity
between pre-trained embeddings for their words [32].

Sentiment Analysis: To determine whether each message expresses a positive or
negative attitude (T2), we use a lexicon-based method [39], which has been
shown to deliver robust performance on conversational data [5]. We predict the
sentiment value for each message which ranges from $-2$ to $+2$.

##  3 Visualization Design

In order to support the user requirements and tasks identified above, we
iteratively designed ConVIScope over a 9 month period based on biweekly
consultations with a team of health care researchers led by an medical
specialist. These meetings provided invaluable feedback on refining and
revising design choices. The resulting final prototype (see ConVIScope: Visual
Analytics for Exploring Patient Conversations) consists of multiple linked
views including an overview summarizing the conversations with sentiment and
topics using focus+context, and a detailed view showing the textual
conversations. The system also supports filtering by multiple facets namely
topics, time range, patient demographics, and locations.

###  3.1 Visual Encodings

The Conversation Analysis View (C) represents a visual summary of the entire
set of conversations based on topics and sentiments (T1, T2). It allows the
user to explore through the corpus on a conversational level as well as
identifying temporal trends and emerging patterns (T3). Unlike ConVis [19]
which visualizes a conversation as an indented tree and connects the topics
with tree nodes, we use a compact heatmap design to encode the topic
distributions, where each column represents a conversation, each row
represents a topic (linked to the corresponding topic node in the topic
hierarchy), and each cell indicates the presence of a topic using dark grey
color. At the top of each column, we use a vertical stacked bar to encode the
sentiment distribution of the corresponding conversation where green indicates
positive and red indicates negative sentiment.

Overall, the Conversation Analysis View helps users identify how topics and
sentiments are distributed across conversations and how they evolved over time
in a space-efficient layout. With respect to MultiConVis [20] and VisOHC [26],
which can only show a small number of conversations at a time, we improve
scalability by designing a novel more compact representation that enables a
focus+context approach. We opt for the focus+context technique to avoid the
temporal (e.g. zoom) and spatial (e.g. overview+detail) separation by
displaying the focus within the context in a single continuous display [7].
The focused window visualizes a constant number of conversations within a
fixed window, while the surrounding context (at the left and right side of
this window) summarizes the entire set of conversations by shrinking the
resolution to fit all the columns inside the view. Finally, we include a
separate Trend View that can be toggled to replace the Analysis View. This
Trend View allows the user to explore how the parent (more general) topics
evolve over time, by showing a histogram of the weekly volume of conversations
for each topic (Figure 1 E).

Figure 1: The Trend View (E) visualizes the number of conversations for each
parent topic over each week.

In the Metadata (A) and Topic View (B) we support the visualization of patient
features and topics (T1, T4) by using frequency charts to convey the
aggregated count for all the conversations containing the attribute. For
patient features, we used a bar chart to visualize the number of conversations
for each attribute (e.g. location, age, gender), where the attributes are the
metadata tags from our data provider. Similarly, the Topic View (B) presents
the list of topics where each topic is associated with a bar indicating the
frequency of that topic. We also visualize the aggregate sentiment
distribution of each topic by placing a horizontal stacked bar to the left
side of a topic node to provide a summary of the attitudes from patients
encountering specific problems (T2). Topics are organized in a hierarchy using
indentation and are connected to the corresponding row in the heatmap via
subtle curved links [37]. In this way, the user can perceive which topics are
being highly discussed (T1) and follow the associated rows in the Analysis
View via the curved links.

The Conversation View (D) is a scrollable list that displays the actual text
of the conversation. The sentiment distribution, patient features as well as
the timestamp for the first message in the conversation are displayed along
with the text.

###  3.2 User Interactions

Browsing through multiple granularity: The Analysis View initially provides an
overview of all the conversations in the current dataset. Then, the user can
create a focused window containing a constant number of conversations by
selecting a region in the scrollbar below the heatmap (C). The user can move
through the selection of a region in the scrollbar to change the focus which
changes the context prior to and after the focused window. Clicking on any
column representing a conversation causes scrolling to the relevant
conversation in the Conversation View via a smooth animation. This idiom
allows the visualization to provide information at multiple granularities,
where the context represents the low-resolution summary, the focused window
represents the high-resolution summary, and the most detailed view represents
the actual conversation.

Faceted exploration: The user can perform faceted exploration by selecting
attributes from the Metadata View and topic nodes in the Topic view. Hovering
on any topic node results in highlighting the corresponding rows in the
heatmap and the curved links that connect those rows with the hovered topic.
Similarly, selecting a column in the heatmap results in highlighting relevant
topic nodes in the Topic View for the corresponding conversation. Furthermore,
features associated with that conversation are highlighted in the Metadata
View by drawing a border around the corresponding rectangle in the bar charts.
An example of this interaction is illustrated in ConVIScope: Visual Analytics
for Exploring Patient Conversations, where the Metadata and Topic View
respectively shows the corresponding patient features (‘Clinic B’, ‘CHF’, ‘Age
70-80’, ‘Female’) and the topics (‘Physical’, ‘Social Services’, ‘Outpatient’,
etc.).

Figure 2: An example of cross-filtering.

Filtering via Selection: To support the user in selecting the conversations
pertaining to specific patient features and topics, we employed a cross-filter
approach. Specifically, whenever the user clicks on any item in the Metadata
and Topic View (e.g. a location, an age group) the interface shows the
proportion of conversations containing that selected item across other patient
features and topics. For example, when the user selects ‘Female’ under the
‘Gender’ attribute, the proportion of conversations containing ‘Female’
patients are highlighted using the blue colored component of the associated
bar of each attribute value. Additionally, the columns representing
conversations that do not match the selected attributes are filtered-out (de-
emphasized) in the Analysis View. Figure 2 demonstrates an example of cross-
filtering after the user selects three different criteria (“Diabetes patients
with Physical Symptoms from Clinic B”).

Interactive Labeling of Topics: Since the predefined topics are classified by
the supervised model, sometimes the predictions maybe inaccurate. To build
more accurate models for predicting predefined topics (T1, T2), we include a
feature that allows the user to interactively revise a pre-defined topic
assignment based on their domain expertise. By enabling the ‘Validate’ mode in
the Conversation View (top (D)), the user can select whether they agree or
disagree with the model prediction for a topic assignment based on the text
content. At any time the user can export the revised model in a CSV file which
can be used in future iterations of model training.

##  4 Expert Case studies

To assess the efficacy of ConVIScope, we performed case studies with 6 domain
experts. The goal was to understand: i) whether ConVIScope help users to
perform the tasks identified in Section 3; ii) which visualization features
worked and did not work and iii) how to improve the system given the experts’
feedback.

Participants: We ran the study with six domain experts (2 male, 4 female, age
range 27-50 years). All participants were health professionals: $2$ practicing
clinicians (C1, C2) and $4$ health researchers (R1-R4). Among them, C1, R1-R4
also participated in the requirement analysis interviews. The participants
held expertise in a variety of areas including infectious diseases, clinical
psychiatry, public health, and physiology. The clinicians (C1, C2) were
interested to use ConVIScope in improving the quality of practice as well as
finding issues that patients are facing. Researchers (R1-R4) were more
interested in various macro-aspects of patient conversations, such as studying
the reasons for hospital admission and evaluating mHealth solutions for
COVID-19 monitoring.

Procedure: The participants first went through short interviews, where we
asked questions about their goals and the type of insights they hoped to get
using ConVIScope. Then, they were provided with a 10-minute tutorial about how
ConVIScope works. Participants then accessed ConVIScope to explore a dataset
consisting of 5775 conversations between 03/2017 to 04/2021 collected from a
real-world healthcare facility by our collaborators. On average, each
conversation had $4.5$ messages. The task was open-ended in nature where
participants explored the dataset according to their own interests and they
were free to use as much time as they needed. During the studies, we followed
the think aloud protocol where we recorded all interactions and responses
suggesting potential use cases and design feedback. The studies ended with
semi-structured interviews, where participants answered questions about the
usefulness of individual visualization components and the extent to which
their goals were satisfied. All studies were conducted online, where
participants shared their screens. Each session lasted about an hour.

Interaction patterns: Reflecting the diversity of their goals and by
leveraging the flexible interactions provided by ConVIScope, participants
displayed a very rich and diverse set of exploration strategies. Half of them
began their exploration using the Analysis View, while the other half began by
interacting with the Metadata and Topics Views. In the Analysis View,
participants mainly focused on the temporal trend of topics and sentiment by
looking at the shrunken column in the surrounding context summaries, while
using the focused window to select specific conversations for accessing their
content in the Conversations View. In the Metadata and Topic Views,
participants often used the cross-filter function, with researchers (e.g. R3
and R1) seemingly more interested in the aggregated counts of the selected
conversations and correlations between topics and patient features (e.g.
social issues for female cancer patients), while the two clinicians more
frequently drilling down on the selected conversations in the Analysis View to
determine whether particular topics were relevant for a patient group and then
carefully reading the associated conversations.

Subjective feedback: Our analysis of the post-study interviews reveals that
participants were impressed with ConVIScope. In particular, they highly
appreciate the interface features for filtering conversations by topics and
other facets (e.g. demographics and locations) and enabling them to verify
what proportion of other facets belong to the filtered set of conversations
through blue highlighting in the corresponding bar charts. For instance, R4
said, “The number of (selected) conversations in the cross-filter can be used
to supplement my research“. Several participants also praised the heatmap
component for providing an effective overview of the dataset: “The (Analysis)
View provides a good summary of the dataset, I can get a good idea of what’s
being talked about without reading the conversations“ (C1). The ability to
drill down to the raw messages was also critical: “Conversation View is very
important, where I verify whether the topics are actually being mentioned“
(R1).

Admittedly, participants also raised some questions and suggested
improvements. Multiple participants expressed uncertainty regarding the
meaning of each sentiment bin, “It will be nice to have the criteria for each
sentiment color to see how the machine is thinking.” (R1), “What exactly is a
neutral conversation?” (R3). Questions were also raised about the support for
comparison. For instance, R4 said: “Although it’s nice to compare
conversations from two clinics by individually selecting them, it will be nice
to distinguish between the two pooled results (compare the two selected
clinics)”.

##  5 Conclusion and Future Work

We present ConVIScope, a visual analytic system that supports the exploration
and analysis of patient-doctor conversations. Our system enables users to get
an overview of a large set of conversations through multiple linked views,
then filter through the set using topics and various metadata before narrow
down to detailed messages. Results from our case study are encouraging, but
multiple participants expressed the need for more interpretable sentiment
analysis and for better support of comparison. These are the two main venues
for future work. Further testing of the current prototype is ongoing at four
different sites, where health care professionals can analyze their own data in
a more ecologically sound setting.

###### Acknowledgements.

We thank Edward Chiu, Will Choi, Jodi Gunawan, Chris Lee, and Abhishek Singh
for their efforts in developing the interface. This work was supported by
Michael Smith Foundation for Health Research Award #17273, with match-funding
from WelTel Inc.

## References

  * [1] D. Angus, A. Smith, and J. Wiles.  Conceptual recurrence plots: Revealing patterns in human discourse.  IEEE TVCG, 18(6):988–997, 2011.
  * [2] D. Angus, B. Watson, A. Smith, C. Gallois, and J. Wiles.  Visualising conversation structure across time: Insights into effective doctor-patient consultations.  PloS One, 7(6):e38014, 2012.
  * [3] R. Baker, D. Angus, E. R. Smith-Conway, K. S. Baker, C. Gallois, A. Smith, J. Wiles, and H. J. Chenery.  Visualising conversations between care home staff and residents with dementia.  Ageing and Society, 35(2):270, 2015.
  * [4] D. M. Blei, A. Y. Ng, and M. I. Jordan.  Latent dirichlet allocation.  the Journal of machine Learning research, 3:993–1022, 2003.
  * [5] G. Carenini, G. Murray, and R. Ng.  Methods for mining and summarizing text conversations.  Synthesis Lectures on Data Management, 3(3):1–130, 2011.
  * [6] A. Chetta, J. M. Carrington, and A. G. Forbes.  Augmenting ehr interfaces for enhanced nurse communication and decision making.  In Proc. Workshop on Visual Analytics in Healthcare, pp. 1–6, 2015\.
  * [7] A. Cockburn, A. Karlson, and B. B. Bederson.  A review of overview+ detail, zooming, and focus+ context interfaces.  ACM Computing Surveys (CSUR), 41(1):1–31, 2009.
  * [8] W. Dou, L. Yu, X. Wang, Z. Ma, and W. Ribarsky.  Hierarchicaltopics: Visually exploring large text collections using topic hierarchies.  IEEE TVCG, 19(12):2002–2011, 2013.
  * [9] M. El-Assady, V. Gold, C. Acevedo, C. Collins, and D. Keim.  Contovi: Multi‐party conversation exploration using topic‐space views.  Computer Graphics Forum, 35, 2016.
  * [10] A. Faiola, P. Srinivas, and J. Duke.  Supporting clinical cognition: a human-centered approach to a novel icu information visualization dashboard.  In AMIA Annual Symposium Proc., vol. 2015, p. 560. American Medical Informatics Association, 2015.
  * [11] S. Fu, Y. Wang, Y. Yang, Q. Bi, F. Guo, and H. Qu.  Visforum: A visual analysis system for exploring user groups in online forums.  ACM Transactions on Interactive Intelligent Systems (TiiS), 8(1):1–21, 2018.
  * [12] S. Fu, J. Zhao, W. Cui, and H. Qu.  Visual analysis of mooc forums with iforum.  IEEE TVCG, 23(1):201–210, 2016.
  * [13] N. Geifman, R. Cohen, and E. Rubin.  Redefining meaningful age groups in the context of disease.  Age, 35(6):2357–2366, 2013.
  * [14] M. Glueck, M. P. Naeini, F. Doshi-Velez, F. Chevalier, A. Khan, D. Wigdor, and M. Brudno.  Phenolines: phenotype comparison visualizations for disease subtyping via topic models.  IEEE TVCG, 24(1):371–381, 2017.
  * [15] D. Gotz, F. Wang, and A. Perer.  A methodology for interactive mining and visual analysis of clinical event patterns using electronic health record data.  Journal of biomedical informatics, 48:148–159, 2014.
  * [16] R. Haider, L. Sudini, C. K. Chow, and N. W. Cheung.  Mobile phone text messaging in improving glycaemic control for patients with type 2 diabetes mellitus: a systematic review and meta-analysis.  Diabetes Research and Clinical Practice, 150:27–37, 2019.
  * [17] A. K. Hall, H. Cole-Lewis, and J. M. Bernhardt.  Mobile text messaging for health: a systematic review of reviews.  Annual Review of Public Health, 36:393–415, 2015.
  * [18] J. S. Hirsch, J. S. Tanenbaum, S. Lipsky Gorman, C. Liu, E. Schmitz, D. Hashorva, A. Ervits, D. Vawdrey, M. Sturm, and N. Elhadad.  Harvest, a longitudinal patient record summarizer.  Journal of the American Medical Informatics Association, 22(2):263–274, 2015.
  * [19] E. Hoque and G. Carenini.  Convis: a visual text analytic system for exploring blog conversations.  In Proc. EuroVis, pp. 221–230, 2014.
  * [20] E. Hoque and G. Carenini.  Multiconvis: A visual text analytics system for exploring a collection of online conversations.  In Proc. IUI, pp. 96–107, 2016.
  * [21] W. Hsu, R. K. Taira, S. El-Saden, H. Kangarloo, and A. A. Bui.  Context-based electronic health record: toward patient specific healthcare.  IEEE Trans. Information Technology in Biomedicine, 16(2):228–234, 2012.
  * [22] S. J. Iribarren, W. Brown III, R. Giguere, P. Stone, R. Schnall, N. Staggers, and A. Carballo-Dieguez.  Scoping review and evaluation of sms/text messaging platforms for mhealth projects or clinical interventions.  Journal of Medical Informatics, 101:28–40, 2017.
  * [23] H. Jang, Y. J. Lee, G. Carenini, R. Ng, G. Campbell, and K. Ho.  Neural prediction of patient needs in an ovarian cancer online discussion forum.  In Proc. Canadian AI, pp. 492–497, 2019.
  * [24] A. Kuehne, E. Lynch, E. Marshall, A. Tiffany, I. Alley, L. Bawo, M. Massaquoi, C. Lodesani, P. Le Vaillant, K. Porten, et al.  Mortality, morbidity and health-seeking behaviour during the ebola epidemic 2014–2015 in monrovia results from a mobile phone survey.  PLoS Neglected Tropical Diseases, 10(8):e0004899, 2016.
  * [25] B. C. Kwon, M.-J. Choi, J. T. Kim, E. Choi, Y. B. Kim, S. Kwon, J. Sun, and J. Choo.  Retainvis: Visual analytics with interpretable and interactive recurrent neural networks on electronic medical records.  IEEE TVCG, 25(1):299–309, 2018.
  * [26] B. C. Kwon, S.-H. Kim, S. Lee, J. Choo, J. Huh, and J. S. Yi.  Visohc: Designing visual analytics for online health communities.  IEEE TVCG, 22(1):71–80, 2015.
  * [27] Y. J. Lee, H. Jang, G. Campbell, G. Carenini, T. Thomas, and H. Donovan.  Identifying language features associated with needs of ovarian cancer patients and caregivers using social media.  Cancer Nursing, 2021.
  * [28] R. T. Lester, P. Ritvo, E. J. Mills, A. Kariri, S. Karanja, M. H. Chung, W. Jack, J. Habyarimana, M. Sadatsafavi, M. Najafzadeh, et al.  Effects of a mobile phone short message service on antiretroviral treatment adherence in kenya (weltel kenya1): a randomised trial.  The Lancet, 376(9755):1838–1845, 2010.
  * [29] M. L. McHugh.  Interrater reliability: the kappa statistic.  Biochemia medica, 22(3):276–282, 2012.
  * [30] T. Munzner.  A nested model for visualization design and validation.  IEEE TVCG, 15(6):921–928, 2009.
  * [31] B. K. Ngwatu, N. P. Nsengiyumva, O. Oxlade, B. Mappin-Kasirer, N. L. Nguyen, E. Jaramillo, D. Falzon, and K. Schwartzman.  The impact of digital health technologies on tuberculosis treatment: a systematic review.  European Respiratory Journal, 51(1), 2018.
  * [32] J. Pennington, R. Socher, and C. D. Manning.  Glove: Global vectors for word representation.  In In EMNLP, pp. 1532–1543, 2014.
  * [33] C. Plaisant, R. Mushlin, A. Snyder, J. Li, D. Heller, and B. Shneiderman.  Lifelines: Using visualization to enhance navigation and analysis of patient records.  In Proc. AMIA Symposium, pp. 76–80, 1998.
  * [34] R. Rehurek and P. Sojka.  Software framework for topic modelling with large corpora.  In Proc. of the LREC Workshop on New Challenges for NLP Frameworks. Citeseer, 2010.
  * [35] M. Sedlmair, M. Meyer, and T. Munzner.  Design study methodology: Reflections from the trenches and the stacks.  IEEE TVCG, 18(12):2431–2440, 2012.
  * [36] W. So, E. P. Bogucka, S. Scepanovic, S. Joglekar, K. Zhou, and D. Quercia.  Humane visual ai: Telling the stories behind a medical condition.  IEEE TVCG, 2020.
  * [37] M. Steinberger, M. Waldner, M. Streit, A. Lex, and D. Schmalstieg.  Context-preserving visual links.  IEEE TVCG, 17(12):2249–2258, 2011.
  * [38] N. Sultanum, M. Brudno, D. Wigdor, and F. Chevalier.  More text please! understanding and supporting the use of visualization for clinical text overview.  In Proc. CHI, pp. 1–13, 2018.
  * [39] M. Taboada, J. Brooke, M. Tofiloski, K. Voll, and M. Stede.  Lexicon-based methods for sentiment analysis.  Computational Linguistics, 37(2):267–307, 2011.
  * [40] T. D. Wang, C. Plaisant, A. J. Quinn, R. Stanchak, S. Murphy, and B. Shneiderman.  Aligning temporal data by sentinel events: discovering patterns in electronic health records.  In Proc. CHI, pp. 457–466, 2008.
  * [41] P. Webster.  Virtual health care in the era of covid-19.  The Lancet, 395(10231):1180–1181, 2020.
  * [42] K. Wongsuphasawat, J. A. Guerra Gómez, C. Plaisant, T. D. Wang, M. Taieb-Maimon, and B. Shneiderman.  Lifeflow: visualizing an overview of event sequences.  In Proc. CHI, pp. 1747–1756, 2011.
  * [43] Y. Wu, Z. Chen, G. Sun, X. Xie, N. Cao, S. Liu, and W. Cui.  Streamexplorer: a multi-stage system for visually exploring events in social streams.  IEEE TVCG, 24(10):2758–2772, 2017.
  * [44] Y. Wu, S. Liu, K. Yan, M. Liu, and F. Wu.  Opinionflow: Visual analysis of opinion diffusion on social media.  IEEE TVCG, 20(12):1763–1772, 2014.


# Forget NLI, Use a Dictionary: Zero-Shot Topic Classification for Low-
Resource Languages with Application to Luxembourgish

###### Abstract

In NLP, zero-shot classification (ZSC) is the task of assigning labels to
textual data without any labeled examples for the target classes. A common
method for ZSC is to fine-tune a language model on a Natural Language
Inference (NLI) dataset and then use it to infer the entailment between the
input document and the target labels. However, this approach faces certain
challenges, particularly for languages with limited resources. In this paper,
we propose an alternative solution that leverages dictionaries as a source of
data for ZSC. We focus on Luxembourgish, a low-resource language spoken in
Luxembourg, and construct two new topic relevance classification datasets
based on a dictionary that provides various synonyms, word translations and
example sentences. We evaluate the usability of our dataset and compare it
with the NLI-based approach on two topic classification tasks in a zero-shot
manner. Our results show that by using the dictionary-based dataset, the
trained models outperform the ones following the NLI-based approach for ZSC.
While we focus on a single low-resource language in this study, we believe
that the efficacy of our approach can also transfer to other languages where
such a dictionary is available.

Keywords: Less-Resourced/Endangered Languages, Document Classification, Corpus

###### Abstract

In NLP, zero-shot classification (ZSC) is the task of assigning labels to
textual data without any labeled examples for the target classes. A common
method for ZSC is to fine-tune a language model on a Natural Language
Inference (NLI) dataset and then use it to infer the entailment between the
input document and the target labels. However, this approach faces certain
challenges, particularly for languages with limited resources. In this paper,
we propose an alternative solution that leverages dictionaries as a source of
data for ZSC. We focus on Luxembourgish, a low-resource language spoken in
Luxembourg, and construct two new topic relevance classification datasets
based on a dictionary that provides various synonyms, word translations and
example sentences. We evaluate the usability of our dataset and compare it
with the NLI-based approach on two topic classification tasks in a zero-shot
manner. Our results show that by using the dictionary-based dataset, the
trained models outperform the ones following the NLI-based approach for ZSC.
While we focus on a single low-resource language in this study, we believe
that the efficacy of our approach can also transfer to other languages where
such a dictionary is available.

Keywords: Less-Resourced/Endangered Languages, Document Classification, Corpus

Forget NLI, Use a Dictionary: Zero-Shot Topic Classification for Low-Resource
Languages with Application to Luxembourgish

Fred Philippy1,2, Shohreh Haddadan1, Siwen Guo1
---
1Zortify S.A., Luxembourg 2University of Luxembourg, Luxembourg
{fred<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Zero-shot classification (ZSC) allows to classify a text document into a
category for which no labeled examples are available. A common technique for
ZSC is to leverage pre-trained language models that have learned general
semantic representations from large corpora. These models can be fine-tuned on
a natural language inference (NLI) dataset and then be used to infer the
entailment between the document and the labels (Yin et al., 2019). In this
approach, each potential target label is considered as a hypothesis in natural
language, and the NLI model is used to evaluate the level of entailment
between the input document and potential labels. For example, given a document
"I always eat my soup with a spoon" and the labels "food" and "animals", the
model can predict a score of how likely the document entails each label. The
label with the highest entailment score can be selected as the predicted
class.

Directly adopting NLI datasets for ZSC poses several challenges and
limitations in real-world scenarios. We identify and highlight three main
limitations of such an approach. First, there is a mismatch between the NLI
and ZSC tasks. Second, the performance of this approach depends on the
availability and quality of NLI datasets, which are challenging and costly to
obtain. Third, for many low-resource languages, the lack of pre-training data
hinders the model’s ability to solve complex reasoning tasks such as NLI. In
this work, we discuss the case of Luxembourgish, a West Germanic language
spoken by around 400,000 people in Luxembourg. There is no large NLI dataset
for the language, and only a small amount of unlabeled pre-training data is
available. Therefore, using NLI datasets for ZSC in Luxembourgish results in
poor performance.

In this work, we propose an alternative solution that provides sufficient data
for low-resource languages in the context of ZSC. The proposed approach
exploits dictionaries as a source of data for ZSC. More specifically, this
dictionary-based approach offers two main advantages: 1) it provides data that
is more relevant to the task of ZSC, and 2) it leverages resources that are
more readily available in many low-resource languages. We demonstrate our
approach on the Luxembourgish language, for which we construct two new topic
relevance classification datasets based on a dictionary.111Our code and
datasets are accessible via
https://github.com/fredxlpy/LETZ/ In short, our main contributions are as
follows:

  1. 1.

We introduce a new approach for creating datasets that allow to adapt models
to ZSC for low-resource languages where a dictionary is available.

  2. 2.

Using this approach, we construct and release two new datasets for
Luxembourgish that are more suitable for ZSC tasks than existing NLI datasets.

  3. 3.

We evaluate our datasets on the task of zero-shot topic classification by
comparing the performance of models trained on our datasets and NLI datasets

## 2\. Motivation

Our work aims to address the following limitations and challenges that hinder
the effectiveness of zero-shot classification for low-resource languages such
as Luxembourgish:

  1. 1.

The mismatch between the fine-tuning task, NLI, and the inference task, topic
classification, as the former requires reasoning about logical relations
between sentences (entailment, contradiction, neutral), while the latter
evaluates the relevance of labels to a sentence (relevant, irrelevant) (Ma et
al., 2021).

  2. 2.

The difficulty and the expense of creating NLI data, especially for low-
resource languages. NLI data requires high-quality annotations that capture
the subtle nuances of entailment and contradiction between sentence pairs.
Moreover, such annotations are often prone to inter-annotator disagreement,
which undermines the validity and reliability of NLI datasets (Pavlick and
Kwiatkowski, 2019; Kalouli et al., 2023).

  3. 3.

The poor performance of language models on high-level tasks such as NLI for
low-resource languages (Ebrahimi et al., 2022). Low-resource language models
suffer from insufficient training data and vocabulary coverage, which affects
their ability to encode rich semantic representations and handle complex
reasoning tasks such as NLI.

## 3\. Related work

A common method for ZSC is the entailment approach (Yin et al., 2019), which
uses NLI datasets to fine-tune pre-trained language models and then apply them
to ZSC tasks. However, this approach has several drawbacks, as discussed by Ma
et al. (2021). They identify issues such as label mismatch, data imbalance,
and semantic ambiguity that affect the performance and generalization of the
entailment approach. Moreover, Ebrahimi et al. (2022) show that NLI models
perform cross-lingual transfer poorly for low-resource languages, which in
turn affects their ZSC capability. Therefore, they argue for the need of
creating annotated datasets for semantic tasks in low-resource languages.

#### Luxembourgish Language

Luxembourgish is one of the three national languages of Luxembourg and is
spoken by roughly 400,000 people ($\approx$ 70% of the population). According
to UNESCO World Atlas of Languages222https://en.wal.unesco.org, Luxembourgish
belongs to the world’s potentially vulnerable languages.

However, Luxembourgish has seen significant transformations over the past
century, including its development into a national language, expansion into
written and digital media, and its role as a symbol of national identity.

The sociolinguistic landscape of Luxembourg, with its unique multilingual
setup (Purschke and Gilles, 2023) and the dynamic evolution of Luxembourgish
from a dialect to a national language with increasing digital presence,
provides a fertile ground for NLP research. Researching Luxembourgish through
the lens of NLP contributes to the field of lesser-studied languages by
developing methodologies that can be applied to other multilingual and
language variation contexts.

## 4\. Our Dataset

Based on a publicly available online dictionary, we create two new topic
relevance classification datasets that allow to adapt pre-trained language
models to zero-shot topic classification in Luxembourgish.

### 4.1. Data Collection

Luxembourg Online Dictionary333https://lod.lu (LOD) is a publicly available
platform hosting a multilingual dictionary with the aim of promoting
Luxembourgish as the language of communication, integration and literature. In
the following, we present some statistics relevant to our work about the data
provided by the Center for the Luxembourgish Language (ZLS444Zenter fir
d’Lëtzebuerger Sprooch) in a
report555https://gouvernement.lu/fr/actualites/toutes_actualites/communiques/2022/06-juin/21-lod-
neie-look.html in 2022.

The dictionary contains around 10,000 synonyms and 48,000 example sentences on
approximately 31,000 entries. Words with multiple meanings are treated
separately for each of their distinct meanings, with corresponding synonyms
and example sentences. For most entries, the dictionary provides translations
from/to 5 languages: German, French, English, Portuguese and Sign Language. In
addition, it features 20,000 phonetic transcriptions, 30,000 audio recordings,
9,300 conjugation and declension tables as well as 5,000 proverbs and idiom
explanations.

ZLS released all of their data on the Luxembourgish Open Data
platform666https://data.public.lu/en/organizations/zenter-fir-dletzebuerger-
sprooch/ under a Creative Commons Zero (CC0) license. In this work, we use the
dataset version released on June 5, 2023.

### 4.2. From Dictionary to Dataset

We first extract the part-of-speech tag, synonyms, and example sentences for
each meaning of every word in the raw LOD data, and filter out the non-nouns.

Next, we assign all the synonyms of a word meaning as labels to its example
sentences. To prevent the model from exploiting the shortcut of matching the
label with the word occurrence in the sentence, we exclude the word itself
from the label set .

Moreover, since many Luxembourgish words are orthographic variants of French
or German words777 Examples: “alerte” → “Alert”, “Million” → “Millioun”., we
discard noun-synonym pairs that have a low Levenshtein distance.

Finally, we generate “non-entailment” samples by randomly selecting a word
from the entire noun vocabulary as a label for each example sentence. However,
we exclude any words that are similar to any of the words in the sentence
based on the Levenshtein distance.

Following the exact same approach, we additionally create a separate dataset
based on the word translations available in the dictionary instead of
synonyms.

This new type of dataset is termed Luxembourgish Entailment-based Topic
classification via Zero-shot learning (LETZ), with the synonym-based dataset
being referred to as LETZ-SYN and the one derived from word translations as
LETZ-WoT.

The number of "entailment"/"relevant" ("1") and "non-entailment"/"irrelevant"
("0") samples is balanced for all sets. The dataset split sizes are provided
in Table 1. We provide examples and more details of our data sets in Appendix
A.

Dataset | |Train| | |Dev| | |Test|
---|---|---|---
LETZ-SYN | 11,822 | 1,478 | 1,478
LETZ-WoT | 39,132 | 4,892 | 4,892
Table 1: Dataset statistics. |  | n = 568 | n = 11.822
---|---|---|---
Model | Train data | SIB-200 | LuxNews | SIB-200 | LuxNews
mBERT | NLI-lb | 17.52 (16.56) | 15.87 (12.51) | \ | \
NLI-de | 25.61 (24.69) | 30.22 (25.88) | 48.04 (43.76) | 43.06 (35.18)
NLI-en | 22.67 (22.38) | 28.55 (23.20) | 49.51 (44.34) | 50.73 (38.18)
NLI-fr | 22.30 (21.30) | 25.02 (20.01) | 49.75 (45.77) | 46.30 (37.65)
LETZ-WoT | 49.39 (49.50) | 59.81 (43.18) | 53.55 (52.46) | 59.96 (52.13)
LETZ-SYN | 52.08 (51.45) | 65.08 (49.20) | 53.80 (54.13) | 66.07 (47.73)
LuxemBERT | NLI-lb | 14.58 (12.91) | 24.69 (16.53) | \ | \
LETZ-SYN | 18.50 (15.86) | 30.63 (19.48) | 65.07 (64.07) | 51.81 (38.27)
Table 2: Results of our experiments on two topic classification datasets.
Experiments are conducted for different number of training samples n from the
different training sets. The performance metrics are reported as "Accuracy (F1
score)" for each task.

## 5\. Implementation

### 5.1. Training

We conduct experiments using two different models that have been pre-trained
on Luxembourgish data: LuxemBERT (Lothritz et al., 2022), a monolingual
Luxembourgish model, and mBERT (Devlin et al., 2019), a multilingual BERT
model that has been pre-trained on 102 languages, including Luxembourgish.

In order to perform the classification task, we append an additional layer to
the pre-trained model that consists of a linear layer and a tanh activation
function. The classification layer has two output nodes which are used to
determine whether a given document contains a topic or not (Figure 2(a)).
Considering the limited amount of fine-tuning data, which could lead to
variability in performance outcomes, we conduct each experiment four times
using distinct random seeds. We then report the average results to account for
any inconsistencies.

Besides fine-tuning both models on our new datasets, we use additional
training datasets for comparison:

  * •

NLI-lb (Lothritz et al., 2022), a Luxembourgish NLI dataset consisting of 568
train and 63 validation samples. The dataset only contains entailment ("1")
and contradiction samples ("0").

  * •

XNLI-de, XNLI-en & XNLI-fr, German, English and French subsets of the XNLI
(Conneau et al., 2018) dataset respectively.

In addition, we perform experiments in "high-resource" (11,822 train and 1,478
validation samples)888Number of samples in LETZ-SYN. and "low-resource" (568
train and 63 validation samples)999Number of samples in the Luxembourgish NLI
dataset (Lothritz et al., 2022). settings.

### 5.2. Evaluation

Due to the inherent limitations associated with Luxembourgish being a low-
resource language, there is a conspicuous lack of labeled datasets available.
Within the context of topic classification, we could only identify two
evaluation datasets that were suitable for our study:

  * •

The Luxembourgish subset of SIB-200 (Adelani et al., 2024), a multilingual
topic classification dataset, containing seven categories, namely:
science/technology, travel, politics, sports, health, entertainment, and
geography.

  * •

A Luxembourgish News Classification dataset introduced by Lothritz et al.
(2022), consisting of news articles from a Luxembourg-based news platform. For
our experiments we restrict it to the following 5 (out of 8) categories:
Sports, Culture, Gaming, Technology, Cooking recipes. We exclude National
news, International news and European news to avoid overlap with other
categories. In what follows we will refer to this dataset as LuxNews.

Following Yin et al. (2019), we use an entailment approach (Figure 2(b) in
Appendix B) to evaluate the models on these datasets, instead of a traditional
supervised classification approach, where the number of output nodes
corresponds to the number of categories. To be more exact, for a given sample
x and potential topics/categories $T=\\{T_{1},\ldots,T_{n}\\}$, we compute the
entailment probability for each pair
$(\textbf{x},T_{i})_{i\in\\{1,\ldots,n\\}}$ denoted by $\textbf{P}_{i,1}$ and
select $T_{i^{*}}$ where

$i^{*}=\operatorname*{argmax}_{i\in\\{1,\ldots,n\\}}\textbf{P}_{i,1}$

The details of the training and evaluation methodology and the datasets
employed are presented in Appendix B.

## 6\. Results

Table 2 shows that models fine-tuned on our datasets exceed the performance of
those trained on NLI data, especially in the "low-resource" setting. More
exactly, mBERT, with only 568 samples from our dictionary-based datasets,
exceeds the results achieved with 20x more NLI samples in French, German, or
English.

However, fine-tuning on German, French, or English NLI datasets markedly
improves results over Luxembourgish data for which the performance is
comparable to that of the random baseline. This suggests that the limited size
of the Luxembourgish pre-training corpus may hinder the model’s ability to
acquire a sufficient level of semantic and pragmatic understanding to solve
complex reasoning tasks such as NLI.

In the "low-resource" setting, LuxemBERT underperforms mBERT, suggesting it
needs more data for task-specific knowledge compared to mBERT’s general cross-
lingual knowledge acquired during pre-training from high-resource languages.
Nonetheless, in the "high-resource" setting, LuxemBERT outperforms mBERT on
SIB-200 but underperforms on LuxNews, possibly due to its inability to
interpret multilingual speech excerpts or quotes.

## 7\. Discussion

While we focus on Luxembourgish as an example of low-resource languages in
this paper, we believe that this approach can be generalized to other
languages where such dictionaries are available as well.

While we acknowledge that our method depends on the availability of
dictionaries for low-resource languages, it is crucial to note that
dictionaries often receive priority due to their fundamental role in
educational and cultural preservation efforts. They are typically more
prevalent because they form the bedrock for literacy and basic education,
which are more fundamental needs than specialized datasets like those required
for NLI. The creation of NLI datasets demands advanced linguistic knowledge
and resources, making it a less immediate concern compared to building basic
language tools. Initiatives, such as the
Dictionaria101010https://dictionaria.clld.org journal, the Living
Dictionaries111111https://livingdictionaries.app or the
Webonary121212https://www.webonary.org platform, support the development of
dictionaries for low-resource and even indigenous languages. So, while both
dictionaries and NLI datasets may not be universally available, there is a
stronger, more widespread motivation behind the creation of dictionaries,
rendering them relatively more accessible and likely to exist for low-resource
languages.

Additionally, our experiments suggest that these dictionaries would not
require tens of thousand of entries to be effective, as it appears that a
multilingual language model can attain satisfactory performance with just a
few hundred sentence-synonym or sentence-word translation pairs.

## 8\. Conclusion

This paper presents a new but simple approach to construct datasets that
enable a language model to perform zero-shot topic classification in a low-
resource language, such as Luxembourgish. We argue that the conventional
approach of transferring from NLI to ZSC is ineffective for such languages,
due to the semantic complexity of NLI and the scarcity of linguistic
resources. We propose an alternative approach that leverages a dictionary to
create a dataset that is more aligned with the ZSC task. We demonstrate that
our dataset enables the model to outperform the ones that employ cross-lingual
NLI transfer or in-language NLI fine-tuning on Luxembourgish ZSC, using over
20 times fewer training samples. In future work, we intend to explore the
effectiveness of our approach when applied to other low-resource languages, as
well as to high-resource ones.

## Limitations

One of the limitations of our study is that we only focus on a single low-
resource language, Luxembourgish, and we do not test our approach on other
languages. Therefore, the generalizability of our method may be limited by the
availability and quality of dictionaries for different languages. Another
limitation is that we rely on a single source of data, namely a dictionary,
which may not capture all the nuances and variations of natural language.

## Ethics Statement

Our study aims to provide a novel solution for zero-shot classification in
low-resource languages, which can potentially benefit various applications and
users who need to classify textual data without labeled examples. While our
method could potentially benefit any language, we specifically emphasize its
usefulness for low-resource languages that suffer from data scarcity and lack
of adequate tools. We believe that our method can contribute to the promotion
of linguistic diversity, as well as to the empowerment and inclusion of
speakers of low-resource languages.

However, we also acknowledge that some dictionaries may contain outdated,
inaccurate, or offensive information that could harm certain groups or
individuals. Therefore, we urge future researchers and practitioners to
carefully select and evaluate the dictionaries they use and to adhere to the
ethical principles and guidelines of their respective fields and communities.

## 9\. Bibliographical References

##

  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ebrahimi et al. (2022) Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022.  AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6279–6299, Dublin, Ireland. Association for Computational Linguistics.
  * Kalouli et al. (2023) Aikaterini-Lida Kalouli, Hai Hu, Alexander F. Webb, Lawrence S. Moss, and Valeria De Paiva. 2023.  Curing the SICK and Other NLI Maladies.  _Computational Linguistics_ , 49(1):199–243.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization.
  * Lothritz et al. (2022) Cedric Lothritz, Bertrand Lebichot, Kevin Allix, Lisa Veiber, Tegawende Bissyande, Jacques Klein, Andrey Boytsov, Clément Lefebvre, and Anne Goujon. 2022\.  LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 5080–5089, Marseille, France. European Language Resources Association.
  * Ma et al. (2021) Tingting Ma, Jin-Ge Yao, Chin-Yew Lin, and Tiejun Zhao. 2021.  Issues with Entailment-based Zero-shot Text Classification.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 786–796, Online. Association for Computational Linguistics.
  * Pavlick and Kwiatkowski (2019) Ellie Pavlick and Tom Kwiatkowski. 2019.  Inherent Disagreements in Human Textual Inferences.  _Transactions of the Association for Computational Linguistics_ , 7:677–694.
  * Purschke and Gilles (2023) Christoph Purschke and Peter Gilles. 2023.  Sociolinguistics in Luxembourg.  In _The Routledge Handbook of Sociolinguistics Around the World_ , 2 edition. Routledge.
  * Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.  Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3914–3923, Hong Kong, China. Association for Computational Linguistics.

## 10\. Language Resource References

##

  * Adelani et al. (2024) Adelani, David and Liu, Hannah and Shen, Xiaoyu and Vassilyev, Nikita and Alabi, Jesujoba and Mao, Yanke and Gao, Haonan and Lee, En-Shiun. 2024.  _SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects_.  Association for Computational Linguistics.
  * Conneau et al. (2018) Conneau, Alexis and Rinott, Ruty and Lample, Guillaume and Williams, Adina and Bowman, Samuel and Schwenk, Holger and Stoyanov, Veselin. 2018.  _XNLI: Evaluating Cross-lingual Sentence Representations_.  Association for Computational Linguistics.
  * Lothritz et al. (2022) Lothritz, Cedric and Lebichot, Bertrand and Allix, Kevin and Veiber, Lisa and Bissyande, Tegawende and Klein, Jacques and Boytsov, Andrey and Lefebvre, Clément and Goujon, Anne. 2022.  _LuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish_.  European Language Resources Association.

##  Appendix A Our Dataset

Figure 1 shows the distribution of the sample length of LETZ-SYN, expressed as
word count, and Table 3 shows a small example subset of LETZ-SYN. Both
datasets, LETZ-SYN and LETZ-WoT, are publicly available under a Creative
Commons Attribution 4.0 International (CC BY 4.0) license.

Figure 1: Distribution of text sample length, expressed in terms of word count, for the training, validation and test sets of LETZ-SYN. Text | Label | Class
---|---|---
Gedëlleg dech a waart op de richtegen Abléck! | Moment | 1
(Be patient and wait for the right point in time!) | (moment) |
Däin Auto huet hannen um Parechoc eng Téitsch. | Libell | 0
(Your car has a dent on the rear bumper.) | (dragon-fly) |
Bei esou vill Kandidate muss eng Auswiel gemaach ginn. | Selektioun | 1
(With so many candidates, a choice must be made.) | (selection) |
Ech schécken der d’Adress vun engem lëschtege Site. | Schrauwenzéier | 0
(I am sending you the link to a funny website.) | (screwdriver) |
Table 3: Examples from our dataset (with English translations).

##  Appendix B Implementation Details

### B.1. Methodology

We provide a visual illustration of the entailment approach (Yin et al., 2019)
that we use in our experiments in Figure 2. The natural language label
description words and number of samples per class during evaluation are
provided in Table 4.

(a) The model is fine-tuned on detecting whether a topic is present in a
sample x or not (= binary classifier). Translation: This example is about…

(b) The model estimates the likelihood of each candidate topic independently
at the inference stage and then the topic with the maximum probability is
chosen.

Figure 2: Illustration of the entailment approach (Yin et al., 2019) for ZSC.

### B.2. Models

We conduct our experiments on the base multilingual BERT (cased) (Devlin et
al., 2019) and LuxemBERT (Lothritz et al., 2022) models. Both models are based
on the same architecture and have 12 attention heads and 12 transformer blocks
with a hidden size of 768. mBERT and LuxemBERT have a vocabulary size of
30,000 and 119,547 respectively. Both models have 110 million parameters.

### B.3. Reproducibility

To reduce the computational expenses, we refrain from conducting hyper-
parameter tuning and employ the configurations that yielded satisfactory
results in our initial experiments. We conduct all the experiments using the
AdamW optimizer (Loshchilov and Hutter, 2019) with a learning rate of 2e-5
with 10% warm-sup steps and linear decay and a batch size of 32. We fine-tune,
with 10 warm-up steps, over 5 epochs. We perform validation after each epoch
and select the optimal checkpoint based on the lowest validation loss. The
maximum sequence length, during training, is set to 128 tokens. During
evaluation, we set the maximum length to 128 tokens for SIB-200, and to 512
for the LuxNews dataset. For each evaluation dataset, we output the accuracy
and macro-averaged F1 score.

### B.4. Computational Resources

All experiments were run within a few hours on 4 A100 40GB GPUs in parallel,
using 4 different random seeds (one per GPU).

Dataset |  Class |  Class Label |  n
---|---|---|---
LuxNews |  Sports |  Sport |  567
Culture |  Konscht |  266
Technology |  Technologie |  199
Gaming |  Videospiller |  82
Cooking recipes |  Rezept |  20
National news |  / |
International news |  / |
European news |  / |
SIB-200 |  Science/Technology |  Technologie |  51
Travel |  Rees |  40
Politics |  Politik |  30
Sports |  Sport |  25
Health |  Gesondheet |  22
Entertainment |  Entertainment |  19
Geography |  Geografie |  17
Table 4: The original classes and their corresponding translated Luxembourgish
class labels that were used our experimental setup. We used the classes marked
in bold for evaluation, and discarded the rest from the evaluation set. n is
the number of samples used for evaluation.