# Addressing Topic Granularity and Hallucination in Large Language Models for
Topic Modelling

Yida Mu, Peizhen Bai, Kalina Bontcheva, Xingyi Song
Department of Computer Science, The University of Sheffield
{y.mu, p.bai, k.bontcheva<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) with their strong zero-shot topic extraction
capabilities offer an alternative to probabilistic topic modelling and closed-
set topic classification approaches. As zero-shot topic extractors, LLMs are
expected to understand human instructions to generate relevant and non-
hallucinated topics based on the given documents. However, LLM-based topic
modelling approaches often face difficulties in generating topics with
adherence to granularity as specified in human instructions, often resulting
in many near-duplicate topics. Furthermore, methods for addressing
hallucinated topics generated by LLMs have not yet been investigated. In this
paper, we focus on addressing the issues of topic granularity and
hallucinations for better LLM-based topic modelling. To this end, we introduce
a novel approach that leverages Direct Preference Optimisation (DPO) to fine-
tune open-source LLMs, such as Mistral-7B. Our approach does not rely on
traditional human annotation to rank preferred answers but employs a
reconstruction pipeline to modify raw topics generated by LLMs, thus enabling
a fast and efficient training and inference framework. Comparative experiments
show that our fine-tuning approach not only significantly improves the LLM’s
capability to produce more coherent, relevant, and precise topics, but also
reduces the number of hallucinated topics.

Addressing Topic Granularity and Hallucination in Large Language Models for
Topic Modelling

Yida Mu, Peizhen Bai, Kalina Bontcheva, Xingyi Song Department of Computer
Science, The University of Sheffield {y.mu, p.bai, k.bontcheva,
<EMAIL_ADDRESS>

##  1 Introduction

Topic modelling is a widely-used unsupervised approach that automatically
summarises the underlying topics within a large, unlabelled collection of
documents (Blei et al., 2003; Grootendorst, 2022). It enables the discovery of
hidden topics in text data, facilitating easier summarisation and
understanding of large datasets (Wallach, 2006; Vayansky and Kumar, 2020;
Churchill and Singh, 2022).

Large pre-trained language models (LLMs) fine-tuned with feedback on human
preferences (Touvron et al., 2023b; OpenAI, 2024) provide an entirely new
pipeline of topic modelling. For example, LLMs are able to directly produce
human-interpretable topics (Pham et al., 2023; Mu et al., 2024), and offer
interactive options in topic modelling through user instructions (Ouyang et
al., 2022; Touvron et al., 2023b).

However, employing LLMs in topic modelling introduces new challenges in real-
world scenarios. Our experiments reveal that (i) Due to input length
limitations, using LLMs for topic extraction is often limited to a single
document, and the outputs for different documents display a degree of
randomness, generating topics that are near-duplicate (please refer examples
(a) & (b) displayed in Figure 1), yet convey highly similar meaning; (ii) LLMs
may not always generate topics with the desired granularity, even when
granularity instructions are provided Pham et al. (2023); Mu et al. (2024) in
the prompt (see examples (d) in Figure 1). This suggests that current LLMs
cannot always generate topics aligned with human instructions, which leads to
additional challenges when counting topic statistics and conducting analysis.
In addition, our experiments (see § 6 and Table 2) also reveal the risk of
hallucination when provided with more complex prompts such as those
incorporating a topic granularity description and seed topics (see samples (c)
in Figure 1).

Figure 1: Four real-world examples consist of the given document (grey), user
prompt (blue), and issues associated with LLM-generated topics (see legends
with different colours). Examples (a) and (b) demonstrate issues with
inconsistent naming; i.e., LLMs tend to generate topics with different
formats. Moreover, when prompting LLMs to generate topics related to ‘hard
disk’ topics given an unrelated document (examples (c), we observe that LLMs
might generate either hallucinated (i.e., ‘Harddisks’) or unwanted topics
(i.e., ‘electronics’). Note that we prompt LLMs to return ‘No related topics’
if there are no related topics in the given text.

The issues of topic granularity, topic naming adherence and risks of
hallucinations are vital to the LLM based topic modelling, as they impact the
quality of topics prior to any downstream applications (such as temporal topic
distribution analysis). In this work, we fine-tune an LLM to directly address
these issues making it more suitable for topic modelling tasks. The main
contributions are:

  * •

To the best of our knowledge, this is the first paper to explore the risks of
hallucination across various API-based and open source LLMs within the context
of LLM-driven topic modelling. It empirically demonstrates the experimental
conditions that can cause LLMs to output hallucinated topics (§ 6).

  * •

We propose an effective reconstruction pipeline for generating Direct
Preference Optimisation (DPO) (Rafailov et al., 2023) training samples that do
not require human annotations (§ 3.3).111Our code and data:
https://github.com/GateNLP/TopicLLM_Granularity_Hallucination

  * •

Experimental results demonstrate that our fine-tuned LLM (TopicMistral)
significantly reduces unlreated topics and hallucinated topics observed in
off-the-shelf LLM outputs. Meanwhile, our experiment shows TopicMistral is
better at generating topics that are more closely aligned with human-annotated
labels (§ 5).

  * •

Finally, we present plug-and-play evaluation protocols designed to measure the
quality of topics extracted by LLMs (§ 4.2).

##  2 Related Work

###  2.1 LLM-Assisted Topic Modelling

LLM-Assisted topic modelling refers to the out-of-box application of LLMs,
such ChatGPT (OpenAI, 2024) or LLaMA (Touvron et al., 2023b), to assist or
enhance the process of traditional probabilistic topic modelling approaches. A
typical output format produced by probabilistic topic modelling algorithms is
a list of words to represent the topic (Blei et al., 2003; Grootendorst,
2022). However, it is very challenging to assign descriptive and meaningful
names to the topic identified based on the list (Mei et al., 2007; Mao et al.,
2012). The process of naming a topic, based on its top words, can be
undertaken through either manual examination or automated techniques,
facilitating a deeper understanding of the underlying thematic content
(Magatti et al., 2009; Aletras and Stevenson, 2014; Wan and Wang, 2016).

Recent work has shown the application of improving BERTopic and LDA, which
leverages the text summarising capabilities of LLMs to automatically assign
descriptive names to each cluster of words (Grootendorst, 2022; Rijcken et
al., 2023; Reuter et al., 2024). As shown in Rijcken et al. (2023), domain
experts named topics derived from LDA and found that half of the human-
readable names produced by ChatGPT (OpenAI, 2024) were deemed useful.

###  2.2 LLM-based Topic Modelling

Previous work on topic modelling with an LLM-based framework is limited. Pham
et al. (2023) introduce a prompt-based topic modelling framework by leveraging
the strong capabilities of GPT-4. Similarly, Mu et al. (2024) conduct a
battery of experiments on topic modelling with rule-based topic clean
approaches.

To address the topic granularity and naming issues, some rule-based strategies
are straightforward to implement but may require domain knowledge; for
instance, crafting manual rules to match and merge these topics or employing
pre-trained word embeddings (e.g., Sentence Transformer (Reimers and Gurevych,
2019)) for clustering near-duplicate topics based on the semantic similarity
(Mu et al., 2024). As an alternative, one might use additional prompt
strategies on top of LLM outputs to merge near-duplicate topics under a single
topic name (Pham et al., 2023). However, post-processing could significantly
increase the computational complexity and require domain knowledge to craft
the post-process rules.

Moreover, previous studies have neglected the risk of hallucinated topics
generated by LLMs; as a result, strategies for mitigating hallucinations in
LLM-based topic modelling remain unexplored.

##  3 Methodology

This section describes our methodology for LLM-based topic modelling. $\S$ 3.1
introduces the prompting strategies to control topic granularity and adherence
to topic naming requirements. $\S$ 3.3 describes our fine-tuning strategy to
enhance LLM-based topic modelling and hallucination mitigation.

###  3.1 Controlling Topic Granularity Though Prompts

We experimented with two prompting strategies (Granularity Description and
Seed Topics) to control topic granularity and adherence to topic naming. Our
experiments ($\S$ 5) demonstrate that the proposed prompts consistently
produce better topics than the baseline prompt, which does not include topic
granularity requirements. The description of each prompting strategy is
detailed below, and the example prompts are shown in Figure 2. Note that we
empirically add tokens ‘Topic:’ at the end of the prompt to guide the LLM to
generate topics directly, rather than produce extra explanations.

  * •

Baseline Prompt (Baseline): We start with a baseline prompt without any
granularity description and seed topics, which directly instructs LLMs to
extract topics from a given text (i.e., the example prompt without the texts
in red and blue).

  * •

Granularity Description (Gran. Desc.): An enhanced prompt with topic
granularity requirements, which is aimed at guiding the LLMs to output topics
that are at a certain granularity or belong to a given domain (e.g., only
output topics related to hard disks).

  * •

Seed Topics (Seed Topics): Incorporating seed (i.e. example) topics into the
prompt provides the LLM with additional guidance on the required topic
granularity and naming conventions. For example, we introduce seed topics such
as ‘Hard Disk’ to prevent LLMs from generating near-duplicate topics such as
‘harddisk’.

Figure 2: An example prompt used in our work: Text enclosed by the special
tokens ‘[/INST]’ denotes the user instruction; red and blue colours denote the
Granularity Description and Seed Topics, respectively.

As detailed below ($\S$ 3.1), we demonstrate that prompt engineering is only
partially effective. This motivated our work on uniform naming and topic
granularity issues in LLM-based topic modelling.

In addition, we show that complex prompting strategies also increase the risk
of model hallucination. Therefore, we propose a fine-tuning pipeline next.
Further details on the hallucination experiments are provided in Section 6.

###  3.2 Fine-tuning LLMs Towards to Topic Modelling

In order to address the limitations of applying off-the-shelf LLMs for topic
modelling, we design an improved fine-tuning pipeline for topic modelling
through Direct Preference Optimization (DPO) (Rafailov et al., 2023). In
comparison to standard reinforcement learning from human feedback (RLHF
(Ouyang et al., 2022)), DPO refactors the fine-tuning process with a simple
classification loss, thereby eliminating the need for explicit reward.

#### DPO Fine-tuning Details

We fine-tune the off-the-shelf LLM (Mistral-7B (Jiang et al., 2023)) following
the DPO pipeline proposed by Rafailov et al. (2023). With the introduction of
the reference frozen LLM $\pi_{\text{ref}}$, the reference probabilities for
accepted topics $y_{a}$ and rejected topics $y_{r}$ can be calculated. Then we
formulate the DPO maximum likelihood objective for optimizing the LLM fine-
tuning $\pi_{\theta}$ as follows:

$\begin{split}\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=&-\mathbb{E}_{(x,y_{a},y_{r})\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{a}\mid
x)}{\pi_{\text{ref}}(y_{a}\mid x)}\right.\right.\\\
&\left.\left.-\beta\log\frac{\pi_{\theta}(y_{r}\mid
x)}{\pi_{\text{ref}}(y_{r}\mid x)}\right)\right],\end{split}$ (1)

where $\beta$ is a hyperparameter that controls the deviation from
$\pi_{\text{ref}}$ to $\pi_{\theta}$ and is set to $0.1$ in our task.
$\sigma(\cdot)$ is a sigmoid function. By using the alternative
parameterisation mechanism, DPO can fit an implicit reward in the objective,
analogous to the reward model in RLHF.

To understand the update mechanism of $\mathcal{L}_{\text{DPO}}$, the gradient
regarding the learnable parameters $\theta$ in $\pi_{\theta}$ is presented as:

$\nabla_{\theta}\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=\\\
-\beta\mathbb{E}_{(x,y_{a},y_{r})\sim\mathcal{D}}\bigg{[}\underbrace{\sigma(\hat{r}_{\theta}(x,y_{r})-\hat{r}_{\theta}(x,y_{a}))}_{\text{higher
weight if reward estimate is wrong}}\\\
\bigg{[}\underbrace{\nabla_{\theta}\log\pi(y_{a}\mid x)}_{\text{increase
likelihood of $y_{a}$}}-\underbrace{\nabla_{\theta}\log\pi(y_{r}\mid
x)}_{\text{decrease likelihood of $y_{r}$}}\bigg{]}\bigg{]},$

where $\hat{r}_{\theta}(x,y)=\beta\log\frac{\pi_{\theta}(y\mid
x)}{\pi_{\text{ref}}(y\mid x)}$ denotes the implicit reward defined by
$\pi_{\theta}$ and $\pi_{\text{ref}}$. This gradient intuitively increases the
likelihood of the accepted topics $y_{a}$ while decreasing the likelihood of
rejected topics $y_{r}$.

Figure 3: Our DPO fine-tuning framework. The ‘reconstruction pipeline’ (in
purple) denotes the approaches we used to modify the original output from
LLMs. We introduce the details of the reconstruction pipeline in Section 3.3.

#### DPO Framework

The standard pipeline for DPO involves training the LLM to optimise directly
for the preference of which ‘Answer’ is most relevant, given two candidate
answers. In our case, the format of training samples for DPO fine-tuning
comprises three parts, with examples of accepted and rejected topics provided
in Figure 3:

  * •

Instruction ($x$): User prompt including granularity description and seed
topics that specify the task requirements.

  * •

Accepted Topics ($y_{a}$): Topics that meet the requirements of the task,
including consistent topic names with expected granularity and non-
hallucinatory topics.

  * •

Rejected Topics ($y_{r}$): Represents the topics that fail to adhere to the
provided instructions.

###  3.3 Developing Accepted ($y_{a}$) and Rejected Topics ($y_{r}$)

In classic RLHF scenarios, human annotations are crucial, especially for
training the reward model, which serves as the foundation for aligning the
model behaviour with human preferences (i.e., ‘Accepted Topics ($y_{a}$)’)
(Ouyang et al., 2022).

In this study, we introduce a novel data reconstruction pipeline without the
need for human annotation for answer ranking purposes. Our approach involves
refining the raw topics generated by out-of-shelf LLMs to modify them to meet
our instructions.

Given a set of documents, by summarising the raw topics generated by off-the-
shelf LLM, we identify $T$ unique topics, denoted as
$T=\\{t_{1},t_{2},t_{3},\ldots,t_{n}\\}$. We consider the top 30 most frequent
topics ($T_{\text{Sub}}$) from $T$ as candidates.

For each of the topics $T_{\text{Sub},i}$ from $T_{\text{Sub}}$, we then use a
sentence-transformer222We use a compact, high-inference-speed base model,
namely ‘all-MiniLM-L6-v2’, to map each topic into a 384-dimensional vector
space for cosine similarity experiments. This model is available through
HuggingFace at: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2.
(Reimers and Gurevych, 2019) to identify the topic clusters, which contain
near-duplicate topics from $T$, by computing cosine similarity333Based on
exploratory tests, we empirically set the threshold at 0.55 to determine
whether a pair of topics are similar, consistent with (Pham et al., 2023)..
This process results in 30 clusters, which is used to create a matrix for
topic matching and replacement. We limit our selection to 30 candidate topics
because matching all topics from the extensive list $T$ would be overly time-
consuming. This limitation aligns with our goal of developing efficient
approaches.

For the LLM-generated raw topics of each given document, we proceed to match
and replace the raw topics using the matrix we obtained above to generate the
‘Accepted Topics’ (see Figure 3). For example, we replace ‘baseballs’ with
‘Baseball’.

In order to minimise hallucinations in the LLM outputs, we also experiment
with creating DPO samples for hallucination mitigation. Given documents from
the 20NG dataset, covering topics such as Computers and Religion, we prompt
the LLM to generate topics related to out-of-distribution domains (e.g.,
COVID-19). In this case, we expect the LLM will output ‘’ rather than
hallucinated topics. Therefore, we are able to create accepted topics
($y_{a}$) by replacing all hallucinated topics with a‘No Relevant Topics’
answer.

#### Rejected Topics ($y_{r}$)

We use the raw output from the off-the-shelf LLM as rejected topics. Further
data cleaning is conducted by filtering out instances where the raw output
requires no modification.

Finally, we obtain 2,500 and 900 valid samples for granularity and
hallucination fine-tuning, respectively, which are further divided into a DPO
training set of 2,800 samples and a DPO validation set of 600 samples.

##  4 Experimental Settings

###  4.1 Datasets and Splits

Following previous work on LLM-based topic modelling (Pham et al., 2023; Mu et
al., 2024), we use three publicly available datasets that include annotated
topic labels.

  * •

20 News Groups (20NG) dataset444http://qwone.com/~jason/20Newsgroups/ is an
open-domain dataset that contains documents spanning 20 categories, which is
widely recognised as a standard benchmark for various NLP downstream tasks.

  * •

20NG Specific We also use a subset of the 20NG dataset containing documents
specifically related to the ‘Recreation’ news.

  * •

Wiki dataset (Merity et al., 2018) includes 14k Wikipedia articles that have
been annotated into one of 15 general categories.

  * •

Bills dataset (Adler and Wilkerson, 2018) comprises summaries of 33k bills
from the 110th to the 114th U.S. Congresses, annotated with 21 general topics
by humans. 555We use the pre-processed Wiki and Bills datasets provided by
Pham et al. (2023).

We create two training sets for DPO fine-tuning and four different test sets
for model evaluation, covering both in-distribution and out-of-distribution
settings:

#### Fine-tuning Sets

We use 5,000 and 1,000 documents from Bills (10%) and 20NG (5%) to develop DPO
training samples for addressing topic granularity and hallucinations,
respectively.

#### Test Sets for Topic Granularity

We use 1,000 documents from each of the 20NG, Wiki, and Bills datasets as
hold-out sets. Additionally, we extract another 1,000 documents from the 20NG
dataset, specifically from the ‘Sports and Recreation’ category (namely 20NG
Specific), to serve as a specific domain dataset for testing LLMs on their
ability to generate topics with varying levels of granularity. Note that we do
not use documents from Wiki for the DPO fine-tuning.

#### Tests Sets for Hallucinations

We use three smaller test sets, each containing 100 documents from the Sports,
Technology, and Politics categories of the 20NG, respectively. These test sets
are strictly excluded from the DPO training sets. (See Table 3)

###  4.2 Evaluation Protocols

Compared to traditional, probabilistic topic models, LLM-based topic modelling
directly generates human-interpretable topic names rather than a list of topic
words. Therefore, traditional probabilistic topic modelling evaluation metrics
such as topic coherence Chang et al. (2009); Newman et al. (2010) are not
valid for evaluating LLM-based topic modelling. In this section, we propose a
set of automatic evaluation metrics to assess the generated topics in the
dimensions of naming adherence, human expectation alignment, and risk of
hallucination.

#### Metric 1: Number of Unique Topics (# of Unique)

is an indicator of the topic naming adherence rate. A better model should
produce a smaller number of unique topics because similar yet duplicated
topics are merged. However, reducing the number of topics should not diminish
the informativeness of the generated topics. Therefore, we also introduce the
following metric to assess informativeness.

#### Metric 2: Similarity Among Top N Topics (Similar N)

is used to evaluate the informativeness of the generated topics. We argue that
the generated topics should maximise the semantic differences among each other
to convey the most topical information in the documents. The metric measures
the average semantic similarity of the top N frequent topics, and is
calculated using the following formula ($Topic_{x}$ is sum of the word
embeddings of the topic $x$, we use SentenceBERT (Reimers and Gurevych, 2019)
to calculate the word embedding):

$\text{Similar
$N$}=\frac{\sum_{i=1}^{N-1}\sum_{j=i+1}^{N}\text{Cosine}(Topic_{i},Topic_{j})}{\frac{N\cdot(N-1)}{2}}$
(2)

#### Metric 3: Mutual Information (MI)

measures the alignment between model-generated topics and human expectations.
Specifically, we use MI for quantifying the similarity between topics across
two lists: the topics generated by the model and the list of human labelled
topics from the datasets.666Note that we reconstruct the names of categories
in 20NG to make them more interpretable. For example, we convert the original
label ‘comp.graphics’ into ‘Computer Graphics’. The higher Mutual Information
score indicates that the topics are better aligned with human expectations. We
compute the MI metric using the formula below: (as described in Metric 2,
$Topic_{x}$ is the sum of the word embeddings for topic $x$.)

Let $L\\_1$ and $L\\_2$ be two lists of LLM generated topics and human-
annotated labels, respectively. For each one-to-one mapping pair
($Topic\\_{i}$, $Topic\\_{j}$) where $Topic\\_{i}$ $\in$ $L\\_1$ and
$Topic\\_{j}$ $\in$ $L\\_2$, we compute the cosine_pairwise($Topic\\_{i}$,
$Topic\\_{j}$).

$\begin{split}\text{Cosine\\_pair}(x)&=\text{Cosine}(Topic_{i},Topic_{j})\\\
&\noindent\forall\,(Topic_{i},Topic_{j})\,|\,Topic_{i}\in L_{1}\&Topic_{j}\in
L_{2}\end{split}$ (3)

Finally, the MI score over $n$ pairs is calculated as follows::

$\text{MI}=\frac{1}{n}\sum_{x=1}^{n}\text{Cosine\\_Pair}(x),$ (4)

where $n$ is the total number of item pairs from $L_{1}$ and $L_{2}$, and
Cosine_Pair(x) is the obtained similarity value for the $x$-th item pair.

|  | 20NG Specific | 20NG | Bills | Wikipedia
---|---|---|---|---|---
Models | Prompts | # |  | Similar
---
N%
MI | # |  | Similar
---
N%
MI | # |  | Similar
---
N%
MI | # |  | Similar
---
N%
MI
LDA | - | - | 0.136 | 0.298 | - | 0.160 | 0.202 | - | 0.156 | 0.254 | - | 0.125 | 0.284
BERTopic | - | - | 0.170 | 0.217 | - | 0.171 | 0.286 | - | 0.171 | 0.296 | - | 0.135 | 0.288
Zero-shot LLM-based Topic Modelling
GPT-3.5 | Baseline | 2,625 | 0.191 | 0.284 | 3,058 | 0.178 | 0.210 | 2,426 | 0.210 | 0.292 | 2,367 | 0.134 | 0.313
GPT-3.5 | Gran. Desc. | 2,414 | 0.197 | 0.316 | 3,024 | 0.171 | 0.221 | - | - | - | - | - | -
GPT-3.5 | Seed Topics | 2,515 | 0.181 | 0.315 | 3,026 | 0.174 | 0.224 | 1,838 | 0.193 | 0.333 | 2,017 | 0.163 | 0.339
LLaMA-7B | Baseline | 2,185 | 0.136 | 0.256 | 2,703 | 0.105 | 0.283 | 2,358 | 0.115 | 0.305 | 2,360 | 0.113 | 0.273
LLaMA-7B | Gran. Desc. | 1,965 | 0.159 | 0.275 | 2,646 | 0.168 | 0.290 | - | - | - | - | - | -
LLaMA-7B | Seed Topics | 1,440 | 0.166 | 0.328 | 1,930 | 0.137 | 0.357 | 1,816 | 0.205 | 0.364 | 2,244 | 0.131 | 0.311
LLaMA-13B | Baseline | 2,421 | 0.198 | 0.302 | 2,909 | 0.175 | 0.266 | 2,261 | 0.137 | 0.300 | 2,098 | 0.143 | 0.331
LLaMA-13B | Gran. Desc. | 1,863 | 0.180 | 0.395 | 2,498 | 0.177 | 0.316 | - | - | - | - | - | -
LLaMA-13B | Seed Topics | 2,003 | 0.182 | 0.379 | 2,406 | 0.176 | 0.314 | 2,096 | 0.189 | 0.328 | 1,669 | 0.107 | 0.380
Mistral-7B | Baseline | 1,949 | 0.212 | 0.321 | 2,268 | 0.181 | 0.282 | 2,250 | 0.186 | 0.291 | 2,186 | 0.139 | 0.312
Mistral-7B | Gran. Desc. | 1,759 | 0.201 | 0.363 | 2,318 | 0.180 | 0.298 | - | - | - | - | - | -
Mistral-7B | Seed Topics | 2,098 | 0.200 | 0.360 | 2,646 | 0.190 | 0.287 | 2,177 | 0.203 | 0.311 | 1,939 | 0.147 | 0.342
Zero-shot LLM-based Topic Modelling + Post-processing
TopicGPT | - | - | - | 0.396 | - | - | 0.286 | - | - | 0.315 | - | - | 0.388
Mu et al. (2024) | - | 1,865 | 0.132 | 0.355 | 2,605 | 0.143 | 0.272 | 2,139 | 0.203 | 0.308 | 1,890 | 0.139 | 0.338
Zero-shot LLM-based Topic Extraction + DPO Fine-tuning
| Topic
---
Mistral
Seed Topics | 116 | 0.156 | 0.577 | 433 | 0.112 | 0.443 | 192 | 0.104 | 0.485 | 289 | 0.103 | 0.517
| Topic
---
Mistral
| Seed Topics
---
Dynamic
85 | 0.147 | 0.531 | 373 | 0.112 | 0.449 | 192 | 0.103 | 0.488 | 219 | 0.101 | 0.534

Table 1: Topic granularity evaluation results across all metrics, LLMs, and
prompt strategies. ‘#’ denotes ‘# of Unique’. TopicMistral’ denotes the new
fine-tuned Mistral-7B model. For MI, higher values $\uparrow$ indicate better
performance, while lower values $\downarrow$ are better for # of Unique Topics
and Similar $N$. Due to different experimental settings, we are unable to
directly compare the number of unique topics and Similar $N$’ between TopicGPT
and TopicMistral.

###  4.3 Implementation Details

#### Fine-Tuning

We use Mistral-7B (Jiang et al., 2023) as the base model for the DPO fine-
tuning experiments. Hence we name our fine-tuned model as TopicMistral. The
choice of Mistral-7B is driven by practical considerations, such as the
availability of computational resources. Furthermore, we focus exclusively on
open-source models, as fine-tuning and sharing models based on API-dependent
LLMs, such as ChatGPT and Claude, is not feasible.

We conduct DPO fine-tuning following the methodology introduced in Rafailov et
al. (2023).777Our approach adheres to the standard DPO training pipeline via
Hugging Face Transformer Reinforcement Learning (TRL) library:
https://huggingface.co/docs/trl/main/en/dpo_trainer. Considering the
substantial GPU memory requirement of up to 70 GB for fully fine-tuning LLMs
with the 7B size in the full precision mode, which is beyond our computational
resources; we opt for a Parameter-Efficient Fine-Tuning (PEFT) method,
specifically QLoRA (Dettmers et al., 2023), available through the Hugging Face
PEFT library.888https://huggingface.co/docs/peft/en/index.

We set the deviation controlling hyperparameter $\beta$ to 0.1, learning rate
to 4e-5, with a batch size of 2 and 200 warm-up steps. All experiments are
executed on a single Nvidia A100 GPU equipped with 40GB of memory. However,
given the efficiency of the PEFT approach, our experimental setup is
applicable on any commercial GPU with at least 12GB memory.

Algorithm 1 Dynamic Seed Topics

1:Input: Docs, N,

2:Input: Seed_Topics $\leftarrow$ [initial topics],

3:Function: Get_Top10(topic_list)

4:Function: TopicMistral(document, Seed_Topics)

5:$\texttt{Topic\\_List}\leftarrow[]$

6:for $i\leftarrow 0$ to $\text{len}(\texttt{Docs})-1$ do

7: if $i>\texttt{N}$ then

8:
$\texttt{Seed\\_Topics}\leftarrow\texttt{Get\\_Top10}(\texttt{Topic\\_List})$

9: end if

10:
$T\\_i\leftarrow\texttt{TopicMistral}(\texttt{Docs}[i],\texttt{Seed\\_Topics})$

11: Topic_List.append($T\\_i$)

12:end for

13:return Topic_List

#### Dynamic Seed Topics:

As mentioned in Section 3.1, one of the primary functions of using seed topics
is to control topic granularity. However, manually setting an optimal set of
seed topics for an unseen corpus is non-trivial. Hence, we introduce Dynamic
Seed Topics, where the list of seed topics is updated dynamically based on the
statistics of the topics generated by TopicMistral. The pseudo-code for
Dynamic Seed Topics is presented in Algorithm 1. The pipeline first
initialises with a set of initial seed topics (Seed_Topics). After generating
topics for N documents (Docs) using an TopicMistral (TopicMistral), the seed
topics are replaced with the top 10 most frequent topics (Get_Top10) from the
generated topics (Topic_List). In our experiment, N is set to 20.

###  4.4 Baselines

This paper evaluates our fine-tuned model against 8 baselines.

  * •

GPT-3.5 (GPT)999https://platform.openai.com/docs/models/gpt-3-5 is one of the
most popular API-based LLM developed by OpenAI, renowned for its strong
ability to generate human-like text.

  * •

LLaMA-2-7B & LLaMA-2-13B (Touvron et al., 2023b) represent advanced versions
of LLaMA-7B (Touvron et al., 2023a), featuring more training resources for
enhanced performance.

  * •

TopicGPT: Pham et al. (2023) uses a post-processing approach by adding
additional prompts on the top of the LLM’s outputs to merge near-duplicate
topics. We adapt their prompts to fit the Mistral-7B model for a fair
comparison.

  * •

Mu et al. (2024) use manually generated rules to merge near-duplicate topics
to solve topic naming issues. The final topics list is obtained by adding an
additional prompt on the list of deduplicated topics.

  * •

Mistral-7B (Jiang et al., 2023) For reference, we also evaluate the off-the-
shelf Mistral-7B, which uses grouped-query attention for quick inference and
incorporates sliding window attention to efficiently process sequences of
mixed lengths at a faster inference speed

  * •

LDA and BERTopic For reference, we also compare against two widely-used topic
modelling approaches: LDA (Blei et al., 2003) and BERTopic (Grootendorst,
2022). Since the outputs of LDA and BERTopic are unnamed, we employ
LLM101010Specifically, we use the GPT-3.5 Turbo model via the OpenAI API. to
assign names to topics based on a list of representative words for each topic.
For example, we prompt the LLM to assign a name for a topic given a list of
words, which represents the typical output format of LDA and BERTopic. For
direct comparison with LLMs, we also transfer our automatic evaluation
protocols to LDA and BERTopic. We directly calculate the ‘Similar N’ metric
using the names of the top 10 topics generated by ChatGPT. Similarly, we use
names generated by ChatGPT to represent each document in order to assess the
‘Mutual Information’ metric.

##  5 Results and Analysis

Models | L-7B | L-13B | M-7B | GPT | L-7B | L-13B | M-7B | GPT | L-7B | L-13B | M-7B | GPT | L-7B | L-13B | M-7B | GPT
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Docs. | Sports News | Sports News | Sports News | Sports News
Gran. Desc. | COVID-19 | Vehicles | COVID-19 | Vehicles
Seed Topics | Side effects & Effective | Autos & Motorcycles | No seed | No seed
Adherence% | 0% | 32% | 50% | 100% | 0% | 7% | 1% | 100% |  0% | 18% | 29% | 100% | 0% | 3% | 0% | 99%
Hallu.% | 14% | 29% | 5% | 0% | 3% | 1% | 9% | 0% | 49% | 68% | 13% |  0% | 11% | 2% | 2% | 1%
Aligned% | 86% | 39% | 45% | 0% | 97% | 92% | 90% | 0% | 51% | 14% | 58% | 0% | 89% | 95% | 98% | 0%

Table 2:  LLM Hallucination Evaluation Results across different combinations
of granularity descriptions and seed topics. L-7B and M-7B denote LLaMA-7B and
Mistral 7B respectively. For Adherence%, higher values $\uparrow$ indicate
better performance, while lower values $\downarrow$ are better for
Hallucination% and Aligned%. The sum of the three metrics should be 100%.

#### Off-The-Shelf LLMs

Similar to Pham et al. (2023) and Mu et al. (2024), we observe that all off-
the-shelf LLMs prompted without seed topics, tend to produce a large number of
near-duplicate topics (See Table 1). When including either granularity
descriptions or seed topics, or both, most LLMs demonstrate a slight
improvement in their ability to generate fewer near-duplicate topics. However,
challenges with inconsistent topic naming persist (e.g., Music, Song, and
Music Production). A higher number of inconsistently named topics results in
higher # of Unique and Similar $N$ scores across all off-the-shelf LLMs. This
indicates that current LLMs are not well suited to zero-shot topic modelling,
underscoring the significance of our research.

#### LDA and BERTopic

On the other hand, existing topic modelling approaches such as LDA and
BERTopic, with assistance from GPT for topic labelling and summarisation, can
achieve similar performance to most off-the-shelf LLMs. This finding is
aligned with prior work on LLM-assisted topic modelling (Rijcken et al., 2023;
Reuter et al., 2024; Chang et al., 2024).

#### Post-processing Approaches

Our findings on off-the-shelf LLMs highlight the motivation behind prior work,
such as that of Pham et al. (2023) and Mu et al. (2024), which leverage post-
processing methods (such as rule-based topic cleaning and additional prompting
to mitigate the generation of inconsistent naming and near-duplicate topics by
off-the-shelf LLMs. Note that while post-processing can be also applied to the
output of TopicMistral, we believe that this process will be considerably
simpler because our LLM produces significantly fewer unique topics (e.g., 90%
less unique topics generated by TopicMistral on Bills).

#### TopicMistral v.s. Off-The-Shelf Mistral

A notable observation is that our fine-tuned TopicMistral model reliably
yields significantly fewer inconsistent naming and near-duplicate topics
across all datasets compared to off-the-shelf LLMs (e.g., $289$ < $1,939$ on
Bills). Note that the lower Similar $N$ values indicate that our model can
generate more diverse top $N$ topics (e.g., $0.104$ < $0.203$ on Bills).
Besides, we observe significant improvement in the MI (taking Wiki as an
example, the MI value has improved from $0.342$ to $0.534$), which suggests
that the fine-tuned TopicMistral model excels in generating topics that align
more closely with the original labels. Considering that the Wiki dataset is
not included in the DPO training samples, we argue that our LLM fine-tuning
strategies can be generalised to other out-of-distribution datasets.

Meanwhile, our comparative experiments (See Table 3) reveal that our fine-
tuning approach has significantly reduced the number of hallucinated topics
across different adversarial prompt settings. Detailed discussion provided in
§ 6.

#### Dynamic List of Seed Topics

For all datasets, the use of dynamic seed topics can further reduce the number
of unique topics with less similar $N$ values, i.e., generating more diverse
top $N$ topics. However, this approach may yield slightly worse MI metrics on
domain-specific datasets, as seen in 20N Specific ($0.147$ < $0.156$).
Furthermore, our findings highlight the advantages of using a dynamic list of
seed topics when the user is not familiar with the domain of the given set of
documents.

##  6 Topic Hallucinations

Models | M-7B | TM-7B | M-7B | TM-7B | M-7B | TM-7B | M-7B | TM-7B | M-7B | TM-7B | M-7B | TM-7B
---|---|---|---|---|---|---|---|---|---|---|---|---
Docs. | Sports News | Computer News | Politics News
Gran. Desc. | COVID-19 | Vehicles | COVID-19 | Vehicles | COVID-19 | Vehicles
Seed Topics |  | Side effects
---
Ineffective
| Autos
---
Motorcycles
| Side effects
---
Ineffective
| Autos
---
Motorcycles
| Side effects
---
Ineffective
| Autos
---
Motorcycles
Adherence% | 50% | 100% | 1% | 99% | 35% | 100% | 0% | 94% | 10% | 97% | 1% | 94%
Hallucination % | 5% | 0% | 9% | 0% | 0% | 0% | 2% | 0% |  14% | 3% | 10% | 3%
Aligned% | 45% | 0% | 90% | 1% | 65% | 0% | 98% | 6% | 76% | 0% | 89% | 3%
Seed Topics | No Seeds | No Seeds | No Seeds | No Seeds | No Seeds | No Seeds
Adherence% | 29% | 97% | 0% | 60% | 28% | 99% | 0% | 50% | 7% | 85% | 0% | 59%
Hallucination % | 13% | 3% | 2% | 8% | 36% | 1% | 2% | 7% | 46% | 15% | 5% | 26%
Aligned% | 58% | 0% | 98% | 32% | 36% | 0% | 98% | 43% | 47% | 0% | 95% | 15%

Table 3:  Direct comparison between TopicMistral and off-the-shelf Mistral
across different combinations of document domains, granularity descriptions,
and seed topics. Note that the testing samples used are strictly excluded from
the DPO training samples. TM-7B denotes TopicMistral.

Models | M-7B | TM-7B | M-7B | TM-7B | M-7B | TM-7B | M-7B | TM-7B | M-7B | TM-7B | M-7B | TM-7B
---|---|---|---|---|---|---|---|---|---|---|---|---
Docs. / Gran. Desc. | Sports News / Sports | Computer News / Computer | Politics News / Politics
Seed Topics |  | Baseball
---
Hocky
No Seeds |  | Hardware
---
Software
No Seeds |  | Government
---
Politician
No Seeds
True Positive % | 68% | 70% | 100% | 81% | 100% | 93% | 100% | 100% | 28% | 65% | 100% | 70%

Table 4:  Comparative experimental results on True Positive% across three test
sets. TM-7B denotes TopicMistral.

This section investigates the hallucination effect in LLM-based topic
modelling. We first introduce the hallucination evaluation metrics (Section
6.1), followed by results and discussion in Section 6.2.

###  6.1 Hallucinated Topics Evaluation Metrics

To investigate the degree of LLM hallucination, we manually conduct
evaluations on the different prompt settings as shown in Table 2 and Table 3.
111111The manual evaluations were conducted by the authors of this paper.
Since each of our hallucination test sets contains 100 samples, we report
three metrics expressed as percentages.

#### Instruction Adherence Rate (Adherence%)

: This metric demonstrates that the topics generated are aligned with human
instructions. For example, given a sports news article, the LLM should return
answers like ‘No related topics’ if we instruct it to generate COVID-19
related topics.

#### Hallucination Rate%

Measures the cases where topics unrelated to the given text are hallucinated
topics. An example is when we instruct LLMs to generate topics related to
COVID-19 given a sports news article; however, the LLMs return topics related
to COVID-19 instead of ‘No related topics’.

#### Task-Aligned Rate (Aligned%)

Similarly, Aligned% shows the number of cases where the model generates topics
related to the given text but does not align with the instruction provided.

#### True Positive%

displays the ratio at which LLMs correctly return topics aligned with
instructions and the given document. We use this metric to evaluate whether
our fine-tuned model can accurately determine if there are no related topics
in the given text, rather than defaulting to the answer ‘No related topics’.
For instance, we assess our new model by providing sports-related news
articles and guiding it to generate topics associated with sports.

###  6.2 Results and Discussion

#### OpenAI’s GPT generates significantly fewer hallucinated topics than those
open source LLMs

We uncover that GPT closely adheres to our instructions regarding the domain
of the desired topics, with a mean adherence rate of 99.3%, whereas most
vanilla open-source LLMs tend to overlook human instructions (achieving a mean
adherence rate of only 8.6% across all settings). For example, GPT can usually
return ‘No related topics’ if there are no relevant topics in the given text.
This suggests that most open source LLMs are at risk of generating
hallucinated topics, especially when provided with adversarial prompts.
Meanwhile, this finding highlights the importance of our research.121212Top 10
frequent topics across all LLMs and prompt strategies are provided in the
appendix.

#### Seed topics play an important role in guiding LLMs to generate the
expected topics.

For all off-the-shelf LLMs (see Table 2), we observe that using an adversarial
Baseline prompt (w/o Seed Topics) can result in a higher number of
hallucinated topics (average hallucination rates of 8% vs. 18% across 4 off-
the-shelf LLMs). Even GPT can occasionally generate hallucinated topics when
given an adversarial Baseline prompt without seed topics (1%).

#### In most cases, unwanted topics are related to the given text rather than
the granularity of the description.

LLMs tend to neglect the human-designed granularity descriptions and seed
topics, instead focusing on the task of topic modelling based on the given
document. (see Aligned% scores in Table 3

#### TopicMistral demonstrates significantly better capability in generating
fewer hallucinated topics when provided with an adversarial prompt.

As shown in Table 3, TopicMistral achieves a low average hallucination rate of
$1\%$ versus $7\%$ and a higher adherence rate of $97\%$ versus 16%, compared
to off-the-shelf Mistral. However, when given a granularity description prompt
without seed topics, our TopicMistral can sometimes produce hallucinated
topics, with an average rate of $10\%$. This further reinforces the finding
that using seed topics is an effective hallucination mitigation approach.

#### TopicMistral does not default to producing ‘No related topics’ when
provided with a non-adversarial prompt.

In Table 4, we display the results of comparative experiments by using related
granularity descriptions and seed topics. We find that the fine-tuned
TopicMistral is sensitive to documents lacking content related to the
granularity description and seed topics, which results in the generation of
‘No related topics’. However, it still achieves better performance than the
off-the-shelf Mistral (the average True Positive % of $76\%$ v.s. $68\%$).
Note that off-the-shelf Mistral always generates topics that exist in the
given text when no seed topic is provided; therefore, we assume that the True
Positive value will be 100%.

#### TopicMistral aggregates less hallucinated topic information during the
attention propagation within the deep layers.

To further investigate the freasons behind TopicMistral generating fewer
hallucinations, we conduct an analysis of the attention weight shift (Wang et
al., 2023) from the topic instruction (i.e., seed topic tokens) to the next
prediction token within the hidden layers of TopicMistral and off-the-shelf
Mistral. Figure 4 shows the average attention weight shift in instances of
topic hallucinations using Mistral-7B. We observe that the attention weights
on TopicMistral significantly decrease in the deep layers, indicating a
reduced aggregation of hallucinated topic information for the next prediction
token.

Figure 4: Average attention weight shift on Mistral-7B and TopicMistral. The
x-axis represents the number of hidden layers and y-axis indicates the average
attention weight from topic instruction to the next prediction token.

##  7 Conclusion

This work focused on improving LLM performance on topic modelling. The novel
contributions of the paper lie in introducing novel approaches to address
topic granularity and hallucination issues that occur when off-the-shelf API-
based and open-source LLMs are used for LLM-driven topic modelling. We also
demonstrate an efficient LLM fine-tuning pipeline through the implementation
of the DPO fine-tuning framework. Our method, which does not require human
annotation and favours an automated pipeline, has proven effective in fine-
tuning LLM outputs to produce a higher number of coherent and relevant topics
while simultaneously reducing the number of hallucinated topics.

## Ethics Statement

Our work has been approved by the Research Ethics Committee of our institute.
All datasets are publicly available via the links or the respective cited
papers.

## References

  * Adler and Wilkerson (2018) E Scott Adler and John Wilkerson. 2018.  Congressional bills project: 1995-2018.
  * Aletras and Stevenson (2014) Nikolaos Aletras and Mark Stevenson. 2014.  Labelling topics using unsupervised graph-based methods.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 631–636.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Chang et al. (2009) Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. 2009.  Reading tea leaves: How humans interpret topic models.  _Advances in neural information processing systems_ , 22.
  * Chang et al. (2024) Shuyu Chang, Rui Wang, Peng Ren, and Haiping Huang. 2024.  Enhanced short text modeling: Leveraging large language models for topic refinement.  _arXiv preprint arXiv:2403.17706_.
  * Churchill and Singh (2022) Rob Churchill and Lisa Singh. 2022.  The evolution of topic modeling.  _ACM Computing Surveys_ , 54(10s):1–35.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Grootendorst (2022) Maarten Grootendorst. 2022.  Bertopic: Neural topic modeling with a class-based tf-idf procedure.  _arXiv preprint arXiv:2203.05794_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Magatti et al. (2009) Davide Magatti, Silvia Calegari, Davide Ciucci, and Fabio Stella. 2009.  Automatic labeling of topics.  In _2009 Ninth International Conference on Intelligent Systems Design and Applications_ , pages 1227–1232. IEEE.
  * Mao et al. (2012) Xian-Ling Mao, Zhao-Yan Ming, Zheng-Jun Zha, Tat-Seng Chua, Hongfei Yan, and Xiaoming Li. 2012.  Automatic labeling hierarchical topics.  In _Proceedings of the 21st ACM international conference on Information and knowledge management_ , pages 2383–2386.
  * Mei et al. (2007) Qiaozhu Mei, Xuehua Shen, and ChengXiang Zhai. 2007.  Automatic labeling of multinomial topic models.  In _Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 490–499.
  * Merity et al. (2018) Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018.  Regularizing and optimizing lstm language models.  In _International Conference on Learning Representations_.
  * Mu et al. (2024) Yida Mu, Chun Dong, Kalina Bontcheva, and Xingyi Song. 2024.  Large language models offer an alternative to the traditional approach of topic modelling.  _arXiv preprint arXiv:2403.16248_.
  * Newman et al. (2010) David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010.  Automatic evaluation of topic coherence.  In _Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics_ , pages 100–108.
  * OpenAI (2024) OpenAI. 2024.  Chatgpt [large language model].  _https://chat.openai.com_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pham et al. (2023) Chau Minh Pham, Alexander Hoyle, Simeng Sun, and Mohit Iyyer. 2023.  Topicgpt: A prompt-based topic modeling framework.  _arXiv preprint arXiv:2311.01449_.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992.
  * Reuter et al. (2024) Arik Reuter, Anton Thielmann, Christoph Weisser, Sebastian Fischer, and Benjamin Säfken. 2024.  Gptopic: Dynamic and interactive topic representations.  _arXiv preprint arXiv:2403.03628_.
  * Rijcken et al. (2023) Emil Rijcken, Floortje Scheepers, Kalliopi Zervanou, Marco Spruit, Pablo Mosteiro, and Uzay Kaymak. 2023.  Towards interpreting topic models with chatgpt.  In _The 20th World Congress of the International Fuzzy Systems Association_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vayansky and Kumar (2020) Ike Vayansky and Sathish AP Kumar. 2020.  A review of topic modeling methods.  _Information Systems_ , 94:101582.
  * Wallach (2006) Hanna M Wallach. 2006.  Topic modeling: beyond bag-of-words.  In _Proceedings of the 23rd international conference on Machine learning_ , pages 977–984.
  * Wan and Wang (2016) Xiaojun Wan and Tianming Wang. 2016.  Automatic labeling of topic models using text summaries.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2297–2305.
  * Wang et al. (2023) Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023.  Label words are anchors: An information flow perspective for understanding in-context learning.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 9840–9855.

## Appendix

# | 20NG Top 10 Topics
---|---
| Original
---
Categories
| Computer [graphics, os.ms-windows.misc, sys.ibm.pc.hardware,
sys.mac.hardware, windows.x, misc.forsale],
---
Recreation. [autos, motorcycles, sport.baseball, sport.hockey], Science
[electronics, medical, space, crypt],
Social [religion.christian], Talk. [politics.guns, politics.mideast,
politics.misc, religion.misc], alt.atheism
TopicMistral | Computer, Sports, Religion, Politics, Technology, Medicine, Security, Car, Software, Space
| TopicMistral
---
Dynamic
Technology, Computer, Religion, Sports, Politics, Medicine, Security, Car,
Software, Space
| Mistral 7B
---
Baseline
Baseball, Homosexuality, Hockey, Email, Sale, Religion, Windows, Car,
Christianity, Faith
| Mistral 7B
---
Gran. Desc.
Sports, Religion, Baseball, Politics, Hockey, Business, Homosexuality,
Technology, Space, Software, Cars, Genocide
| Mistral 7B
---
Seed Topics
Baseball, Hardware, Religion, Hockey, News, Sports, Politics, Software,
Genocide, Cars, Bible, Encryption
| LLaMA 7B
---
Baseline
Book, God, Encryption, Religion, Government, Technology, Faith, Genocide,
Window, Do
| LLaMA 7B
---
Gran. Desc.
Religion, Baseball, Political scandal, Technology, Economic downturn, Hockey,
News article, Sport, Encryption, Sale
| LLaMA 7B
---
Seed Topics
Hardware, Technology, Sport, Religion, Baseball, Car, Software, Hockey,
Windows, Politics
| LLaMA 13B
---
Baseline
Baseball, Homosexuality, Space exploration, Hockey, Fire, Religion, Morality,
Genocide, Christianity, Encryption
| LLaMA 13B
---
Gran. Desc.
Technology, Computer hardware, Hockey, Religion, Baseball, Space exploration,
Hardware, Sports, Sales, Political scandals
| LLaMA 13B
---
Seed Topics
Technology, Baseball, Hockey, Hardware, Sports, Space exploration, Religion,
Computer hardware, Motorcycles, Sales
| GPT
---
Baseline
Encryption, Homosexuality, Christianity, Expertise, Expertise in, Price,
Genocide, Original sin, Motorcycle, Modem
| GPT
---
Gran. Desc.
Sport, Politics, Entertainment, Encryption, Technology, Current event, Law
enforcement, Homosexuality, Morality, Hockey
| GPT
---
Seed Topics
| Technology, Politics, Entertainment, Encryption, Computer hardware,
Environment, Baseball players, Baseball, Hockey,
---
Armenian genocide
# | 20NG Specific Domain Top 10 Topics
| Original
---
Categories
Recreation. [autos, motorcycles, sport.baseball, sport.hockey]
TopicMistral | Baseball, Hockey, Car, Sports, Cars, Football, Motor, Bike, Insurance, Engine
| TopicMistral
---
Dynamic
Sports, Baseball, Recreation, Cycling, Ice, Hockey, Football, Car, Bicycle
| Mistral 7B
---
Baseline
Baseball, Hockey, Nhl, Game, Espn, Bike, Engine, Teams, Car
| Mistral 7B
---
Gran. Desc.
Baseball, Sports, Hockey, Motorcycles, Nhl, Politics, Ice hockey, Cars, Car,
Sports news
| Mistral 7B
---
Seed Topics
Baseball, Hockey, Sports, Nhl, Politics, Motorcycles, Ice hockey, Bike, Espn,
Football
| LLaMA 7B
---
Baseline
Hockey, Baseball, Bruins, Espn, Bike, Bmw, Sabres, Playoffs, Players, Time
| LLaMA 7B
---
Gran. Desc.
Hockey, Baseball, Sports news, Motorcycles, Injury, Racing, Playoffs, Car
maintenance, Cars, Sabres
| LLaMA 7B
---
Seed Topics
Hockey, Baseball, Sports, Cars, Motorcycles, Safety, Pitching, Playoffs, Car
maintenance
| LLaMA 13B
---
Baseline
Baseball, Hockey, Bike, Motorcycles, Injury, Espn, Waving, Price, Bruins,
Space exploration
| LLaMA 13B
---
Gran. Desc.
| Baseball, Hockey, Sports, Motorcycles, Player performance, Car maintenance,
Celebrity gossip, Political scandals,
---
Cars, Automotive
| LLaMA 13B
---
Seed Topics
| Baseball, Hockey, Sports, Motorcycles, Car maintenance, Cars, Player
performance, Pitching, Hockey playoffs,
---
Motorcycle safety
| GPT
---
Baseline
Baseball, Hockey, Grant fuhr, Expertise, Baseball game, Countersteering, Bike,
Hockey coverage, Expertise in, Motorcycles
| GPT
---
Gran. Desc.
Sports news, Sports events, Baseball, Hockey, Basketball, Football, Tennis,
Sports, Sports updates, News
| GPT
---
Seed Topics
Baseball, Hockey, Entertainment, Sports, Politics, Technology, Car
maintenance, Grant fuhr, Pittsburgh penguins, Playoffs

Table 5: Top 10 most frequent topics across all prompt strategies and LLMs on
the 20NG and 20NG Specific datasets.

# | Wiki Top 10 Topics
---|---
| Original
---
Categories
| Media and drama, Agriculture, food, and drink, Philosophy and religion,
Warfare, Engineering and technology,
---
Art and architecture, Mathematics, Geography and places, History, Social
sciences and society, Video games,
Music, Language and literature, Sports and recreation, Natural sciences
TopicMistral | History, Music, Sports, Technology, Television, Politics, Military, Football, Transport, Geography
| TopicMistral
---
Dynamic
History, Music, Technology, Television, Sports, Military, Politics, Football,
Transport, Film
| Mistral 7B
---
Baseline
History, Music, Football, Politics, Film, Early life, Meteorological history,
World War II, Gameplay, Architecture
| Mistral 7B
---
Seed Topics
History, Music, Football, Politics, Meteorological history, Early life,
Architecture, , Film, World War II
| LLaMA 7B
---
Baseline
History, Early life, Route description, Gameplay, Design, Music,
Meteorological history, World War II, Geography
| LLaMA 7B
---
Seed Topics
History, Music, Football, Architecture, Politics, World War II, Early life,
Boat Race, Geography
| LLaMA 13B
---
Baseline
History, Music, Geography, Football, Architecture, Transportation, World War
II, World War I, Sports
| LLaMA 13B
---
Seed Topics
History, Music, Transportation, Sports, Geography, Military, Football,
Architecture, Education
| GPT
---
Baseline
| History, Music, World War II, Meteorological history, Early life, Geography,
Football career, Music Production,
---
Personal life, Military Service
| GPT
---
Seed Topics
| History, Music, Transportation, Naval History, World War II, Geography,
Meteorological history, Education,
---
Performance, Relationships
# | Bills Top 10 Topics
| Original
---
Categories
| Defense, Agriculture, Public Lands, Culture, Macroeconomics, Technology, Law
and Crime, Health, Domestic
---
Commerce, Transportation, Energy, Social Welfare, Immigration, Foreign Trade,
Civil Rights, Government
Operations, Education, Housing, Environment, International Affairs, Labor
TopicMistral | Government, Security, Education, Finance, Regulation, Health, Tax, Economy, Energy, Environment
| TopicMistral
---
Dynamic
Government, Security, Regulation, Tax, Economy, Medicare, Finance, Education,
Energy, Medicine
| Mistral 7B
---
Baseline
| Tariff Schedule, Act, Tax Credit, Medicare, Duty Suspension, Education,
Immigration, Veterans, Social Security,
---
Amendment
| Mistral 7B
---
Seed Topics
Education, Tariffs, Medicare, Tax Credit, Health, Immigration, Agriculture,
Social Security, Veterans, Funding
| LLaMA 7B
---
Baseline
Medicare, Tariff, Tax Credit, Tariff Schedule, Education, Suspension, Duty
Suspension, Immigration, Veterans
| LLaMA 7B
---
Seed Topics
Education, Healthcare, Trade, Immigration, Taxation, Medicare, Health, Tariff,
Tax Credit
| LLaMA 13B
---
Baseline
| Tariffs, Medicare, Education, Tax Credits, Suspension, Tax Credit, Renewable
Energy, Land Management,
---
Social Security
| LLaMA 13B
---
Seed Topics
| Tariffs, Education, Medicare, Tax Credits, Trade, Environmental Protection,
Immigration, Transportation,
---
Land Management
| GPT
---
Baseline
| Tariff Schedule, Duty Suspension, Tax Credit, Veterans Affairs, Education,
Environmental Protection, Immigration,
---
Taxation, Homeland Security, Tax Credits
| GPT
---
Seed Topics
| Trade, Education, Taxation, Tax Credits, Environmental Protection,
Immigration, Land Management, National
---
Security, Healthcare, Financial Regulation

Table 6: Top 10 most frequent topics across all prompt strategies and LLMs on
Bills and Wiki datasets.