# Interactive Topic Models with Optimal Transport

Garima Dhanania* Sheshera Mysore*† Chau Minh Pham
Mohit Iyyer Hamed Zamani Andrew McCallum
University of Massachusetts Amherst, MA, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>*
Equal contribution.† Currently a Postdoctoral Researcher at Microsoft.

###### Abstract

Topic models are widely used to analyze document collections. While they are
valuable for discovering latent topics in a corpus when analysts are
unfamiliar with the corpus, analysts also commonly start with an understanding
of the content present in a corpus. This may be through categories obtained
from an initial pass over the corpus or a desire to analyze the corpus through
a predefined set of categories derived from a high level theoretical framework
(e.g. political ideology). In these scenarios analysts desire a topic modeling
approach which incorporates their understanding of the corpus while supporting
various forms of interaction with the model. In this work, we present EdTM, as
an approach for _label name supervised topic modeling_. EdTM models topic
modeling as an assignment problem while leveraging LM/LLM based document-topic
affinities and using optimal transport for making globally coherent topic-
assignments. In experiments, we show the efficacy of our framework compared to
few-shot LLM classifiers, and topic models based on clustering and LDA.
Further, we show EdTM’s ability to incorporate various forms of analyst
feedback and while remaining robust to noisy analyst inputs.

##  1 Introduction

Topic models have had a long history of development and use for exploring
document collections and representing documents and collections for downstream
tasks Boyd-Graber et al. (2017). While topic models are most commonly
associated with probabilistic generative models Blei et al. (2003), other
approaches leveraging matrix factorization Lund (2019) and clustering
embeddings from pre-trained language models Thompson and Mimno (2020) have
also been explored for topic modeling. Across these models, a valuable feature
for practitioners is their ability to represent latent topics with
interpretable descriptors such as word, sentence, or documents assigned to
latent topics.

While latent topics are valuable for several forms of analysis Roberts et al.
(2013); Hoyle et al. (2019), topic models fall short of practitioners’
expectations when analysts don’t wish to be biased by machine generated
topics, wish to have topics capture their own understanding of the corpus, or
analyze a corpus in a top-down manner through categories defined by an
analytical framework Hong et al. (2022); Carlsen and Ralund (2022); Jasim et
al. (2021). One approach to address such concerns may involve labeling a
training set of documents with categories, followed by classification or
supervised clustering to label the whole collection Finley and Joachims
(2005). However, these prove to be time consuming due to the need to label
data and brittle when the set of categories evolve as practitioners’
understanding of the corpus changes.

An alternative explored in a large body of work on probabilistic topic models
involves supervising latent topics. This work has explored use of document
metadata Card et al. (2018), seed words per topic Jagarlamudi et al. (2012),
and constraints to supervise and refine latent topics Hu et al. (2011).
However, generative topic models prove challenging to scale to large corpora
Lund et al. (2017), are influenced by document lengths Hong and Davison
(2010), and are limited in their ability to leverage performant pre-trained
language models Hoyle et al. (2022). While an emerging body of work has
effectively leveraged pre-trained language models for unsupervised topic
modeling Pham et al. (2024); Wang et al. (2023); Thompson and Mimno (2020)
exploration of contemporary pre-trained models for interactive topic modeling
has been limited. Our work fills this gap.

In this work, we propose a framework for _label name supervised topic
modeling_. We show label names as a flexible form of interaction with topic
models – allowing analysts to specify topics as short label names akin to
class labels, longer label descriptions, as well as seed documents. We model
topic modeling as an _assignment problem_ , requiring a globally coherent
assignment of documents to topics, and leverage the well explored machinery of
optimal transport Peyré et al. (2019) for interactive topic modeling. Use of
optimal transport offers several advantages: assignment algorithms capable of
leveraging GPU computation, an ability to leverage document-topic similarities
obtained from pre-trained language models, and a mature body of work on
assignment algorithms which may be applied for various topic modeling
applications. We refer to our approach to interactive topic modeling as an
_editable topic model_ , EdTM (Figure 1). In experiments, we show EdTM to
induce high quality topics compared to a range of baselines, support various
forms of interaction from analysts, and induce robust topics even when
presented with noisy analyst supervision. Code and datasets for our work will
be released upon acceptance.

Figure 1: Interactive topic modeling with EdTM consists of two steps,
document-topic scoring for analyst provided topic names using LM/LLM bi-
encoders and cross-encoders followed by computation of partial or complete
topic assignments using optimal transport. Analyst topic names may take on
various forms such as label names, descriptions, or documents, to support rich
forms of interaction.

##  2 Problem Definition

We consider a _label supervised topic modeling_ problem where we are presented
with a collection of documents $D$ that an analyst is interested in studying
and a set of topic labels $l\in L$ that the analyst would like to organize $D$
by or believes is contained in $D$. We are interested in predicting a
partitioning of the documents in $D$ into the labels $L$ where each document
could be assigned to multiple topics in $L$ or no topics at all. The latter
case represents a selective topic assignment intended to capture the scenario
where analysts intuitions of the topics $L$ are incorrect or incomplete,
requiring that some documents or topics remain unassigned. We assume documents
$d\in D$ to be short sentences or multi-sentence paragraphs.

Further, we are interested in supporting different forms of interactive
analysis and exploration of $D$ \- this is done through different forms of
topic labels $l$. Topic labels may be represented as natural language _names_
for topics, example _words or longer descriptions_ of topics, or example
_documents_ representing a topic. The various forms of topic labels combined
with selective topic assignment allows for rich ways of interaction with a
topic model.

##  3 Proposed Approach

We propose a flexible framework for interactive topic modeling, EdTM,
consisting of two components: a document-topic scoring leveraging expressive
LM/LLMs, and document-topic assignment leveraging optimal transport assignment
algorithms. We explore scores computed using pretrained LM bi-encoder models
as well as expressive LLM crossencoders allowing analysts to specify their
understanding of topics in a variety of intuitive natural language forms. For
document-topic assignment, optimal transport algorithms allow assignment
decisions to be made in a globally coherent manner and naturally incorporate
document-topic affinities from expressive LLM scoring functions. Further, we
leverage partial assignment algorithms Benamou et al. (2015) that selectively
exclude high cost document-topic assignments, to ensure that topic-assignments
remain robust to potentially noisy or incomplete labels provided by users – a
likely occurrence in interactive topic modeling. Finally, since computation of
the $D\times L$ assignment remain expensive for large corpora with optimal
transport, we compute approximate assignments through batched computation of
assignments Fatras et al. (2020). We discuss the assignment algorithms and
scoring functions used in EdTM next.

###  3.1 Optimal Transport Topic Assignment

Optimal transport may be seen as a way to compute a global minimum cost
alignment between sets of points given the cost of aligning individual pairs
of points. Additionally, OT problems associate the sets with probability
distributions where valid alignments satisfying capacity constraints specified
over these distributions. Solutions to the OT problem result in sparse and
soft alignments between the two sets when given meaningful pairwise costs from
most scoring models, this makes them well suited to topic modeling while
leveraging expressive LM/LLM models.

Specifically, we assume the source and target points to be document and label
sets, $D$ and $L$, of size $n$ and $m$ respectively. We assume them to be
distributed according to distributions $\mathbf{x}_{D}$ and $\mathbf{x}_{L}$,
with pairwise costs $\mathbf{C}\in\mathbb{R}^{n\times m}_{+}$. Since the
distributions are unknown, we treat $\mathbf{x}_{D}$ and $\mathbf{x}_{L}$ to
be uniform distributions. The solution to the OT problem is a soft assignment,
the _transport plan_ $\mathbf{Q}^{*}$, which converts $\mathbf{x}_{D}$ into
$\mathbf{x}_{L}$ by transporting probability mass from $\mathbf{x}_{D}$ to
$\mathbf{x}_{L}$ while minimizing an aggregate cost $\mathcal{W}$, referred to
as the Wasserstein Distance. Capacity constraints on $\mathbf{Q}$ result in a
constrained linear optimization problem:

${\mathcal{W}}=\underset{\mathbf{Q}\in\mathcal{S}}{\texttt{min}}\langle\mathbf{C},\mathbf{Q}\rangle$
(1)

We examine two kinds of capacity constraints on $\mathbf{Q}$ resulting in
different feasible sets $\mathcal{S}$ – a _complete assignment_ which assigns
all source points to target points and a _partial assignment_ which does not
require all source and target points to be assigned. This allows potentially
noisy topic assignments to be ignored. Complete assignment ensures that the
columns and rows of $\mathbf{Q}$ marginalize respectively to $\mathbf{x}_{D}$
and $\mathbf{x}_{L}$, resulting in
$\mathcal{S}_{c}=\\{\mathbf{Q}\in\mathbb{R}^{n\times
m}_{+}|\mathbf{Q}\mathbf{1}_{m}=\mathbf{x}_{D},\mathbf{Q}^{T}\mathbf{1}_{n}=\mathbf{x}_{L}\\}$.
Partial assignment requires only a fraction $p<1$ of the source or target
points to be assigned resulting in:
$\mathcal{S}_{p}=\\{\mathbf{Q}\in\mathbb{R}^{n\times
m}_{+}|\mathbf{Q}\mathbf{1}_{m}\leq\mathbf{x}_{D},\mathbf{Q}^{T}\mathbf{1}_{n}\leq\mathbf{x}_{L},\mathbf{1}^{T}\mathbf{Q}^{T}\mathbf{1}=p\\}$.
The mass $p$, to preserve in $\mathbf{Q}^{*}$ may be specified by analysts and
captures the fraction of high quality topic assignments that can be made for
$D$ and $L$.

In practice, for complete as well as partial assignments we solve an entropy
regularized variant of Eq (1):
$\mathcal{W}=\texttt{min}\langle\mathbf{C},\mathbf{Q}\rangle-1/\lambda
H(\mathbf{Q})$. While exact solutions to Eq (1) require $O(n^{3})$
computations, entropy regularization allows both problems to be solved using
iterative methods with an empirical complexity of $O(n^{2})$ and solvers
capable of leveraging GPU computation. This allows scaling to larger datasets.
To further speed up topic assignments for large $D$, we compute $\mathbf{Q}$
in batches of source points, with batches sized $<|D|$. To solve complete and
partial batch assignment problems we leverage algorithms of Cuturi (2013) and
Benamou et al. (2015) implemented in off the shelf solvers.111OT solvers:
https://pythonot.github.io Topic assignments are described further in §3.3 and
Algorithm 1.

Algorithm 1 Complete topic assignment in EdTM

1:Input: $D$, $L$, $f_{\text{dist}}$

2:$\mathbf{C}\leftarrow f_{\text{dist}}(D,L)$ $\triangleright$ Compute
pairwise costs

3:$\mathbf{Q}^{*}\leftarrow\text{zeros}(|D|,|L|)$ $\triangleright$ Initialize
plan with zeros

4:for epoch $e$ of $E$ do

5: for batch $b$ of $B$ do

6: $\mathbf{x}^{b}_{D}$,
$\mathbf{x}_{L}\leftarrow\text{uniform}(b),\text{uniform}(L)$

7:
$\mathbf{Q}^{*b}\leftarrow\underset{\mathbf{Q}^{b}\in\mathcal{S}_{c}}{\texttt{argmin}}\langle\mathbf{C}^{b},\mathbf{Q}^{b}\rangle-1/\lambda
H(\mathbf{Q}^{b})$

8: $\mathbf{Q}^{*}[b,:]\leftarrow\mathbf{Q}^{*b}$ $\triangleright$ Update plan
for batch

9: end for

10:end for

11:$\mathbf{Q}^{*}\leftarrow\mathbf{Q}^{*}/B\cdot E$ $\triangleright$
Normalize plan

12:$\\{l^{*}\\}_{i=1}^{|D|}\leftarrow\mathtt{argmax}_{l}\mathbf{Q}^{*}$

###  3.2 Document Topic Scoring

To allow users to provide rich natural language topic targets $L$, we leverage
LM/LLM models to compute document-topic costs $\textbf{C}[d,l]$ in Eq (1).
Here, we leverage off-the-shelf BERT based bi-encoders and expressive T5-XL
crossencoders trained for predicting query-document relevance in IR tasks
Nogueira et al. (2020); Lin et al. (2020). With a bi-encoder we compute
pairwise costs as L2 distances between document and label embeddings:
$\text{L2}(f_{\text{BE}}(d),f_{\text{BE}}(l))$. From crossencoders,
$f_{\text{CE}}$, we first obtain the probability of relevance
$P_{\text{CE}}(\text{rel}=1|d,l)$. Then we obtain costs as:
$1-{P_{\text{CE}}}/{max_{l}P_{\text{CE}}}$. The normalization per input $d$
ensures that costs remain comparable across all inputs with a minimum value of
0 across all documents. We note that while computation of costs with
crossencoders is an expensive operation, recent work Yadav et al. (2022)
leveraging matrix factorization to efficiently compute crossencoder based
scores for large corpora promise a ready pathway toward scaling our approach
to larger corpora – we leave exploration of this to future work. We refer to
bi-encoder and crossencoder variants as $\textsc{EdTM}_{\text{BE}}$ and
$\textsc{EdTM}_{\text{CE}}$. We provide further implementation details in
§4.1.

###  3.3 Hardening Topic Assignments

Here we outline the final procedure for obtaining hard topic assignments for
documents from $\mathbf{Q}^{*}$. Both partial and complete topic assignments
in $\mathbf{Q}^{*}$ may assign documents to a variable number of topics,
however we only retain the top topic assignment per document for to make
comparisons to baseline methods that make single topic assignments,
representing the majority of recent baselines that we compare to. We leave
exploration of multi-topic assignments to future work. Complete assignments
are made as: $\mathtt{argmax}_{l}\mathbf{Q}^{*}$. For partial assignment, we
first marginalize $\mathbf{Q}^{*}$ over the topic labels as:
$\mathbf{x}^{*}_{D}=\sum_{l}\mathbf{Q}[\cdot,l]$. Then, we only make
assignments for the fraction $p$ of documents which have the highest values in
$\mathbf{x}^{*}_{D}$ (adding a step after Line 11 in Algorithm 1). Recall that
$p$ represents the fraction of mass conserved in making partial assignments by
the optimal transport solution of Eq 1. In practice, points which don’t
receive assignments represent high-cost topic assignments which are unlikely
to receive accurate assignments.

##  4 Experiments

We experiment with EdTM in a variety of English datasets commonly used to
evaluate topic-models. We compare EdTM against methods for few-shot
classification leveraging LM/LLMs, clustering methods, and LDA based topic
models. We evaluate EdTM using extrinsic metrics commonly used to evaluate
clustering since we propose EdTM as a model for data exploration. Further, we
evaluate EdTM in various interactive setups leveraging varying human
interaction and noise. We outline our primary experimental setup next, and
detail interaction setups in the respective sections.

###  4.1 Experimental Setup

Dataset | $|D|$ | $|L|$ | $|d|$ | Label name and frequency
---|---|---|---|---
Twitter | $4373$ | $6$ | $28$ |  pop culture, $1705$; sports and gaming, $1528$; daily life, $647$
Wiki | $8024$ | $15$ | $2888$ |  Media and drama, $1118$; Warfare, $1112$; Music, $1007$
Bills | $15242$ | $21$ | $215$ |  Health, $1755$; Public Lands, $1355$; Domestic Commerce, $1295$
Bookgenome | $9177$ | $226$ | $170$ |  fiction, $548$; adventure, $352$; suspense, $301$

Table 1: Summary of the datasets used for experiments: number of documents
($|D|$), number of labels ($|L|$), and average length of documents ($|d|$). We
also present the top three labels and their frequencies, illustrating the
topic names and label skews present in our datasets.

Datasets. We use four datasets of short- to medium-length texts, each
accompanied by a corresponding gold label. These datasets feature a diverse
range of domains, including Wikipedia articles, Congressional Bills summaries,
Twitter posts, and Goodreads book descriptions. As a result, the four datasets
exhibit varying types and numbers of labels ($|L|$). Our datasets and their
topic labels are summarized in Table 1.

Twitter contains short tweets from Sep 2019 to Aug 2021 paired with 1 of 6
high level labels assigned by crowd workers Antypas et al. (2022).222HF
Datasets: cardiffnlp/tweet_topic_single

Wiki contains Wikipedia articles designated to be “Good articles” based on
adherence to editorial standards by Wikipedia editors. The articles are paired
with 1 of 15 high-level and 114 finer-grained labels (Merity et al., 2018).

Bills contains US congress bill summaries from January 2009 to January 2017
manually paired with 1 of 21 high-level and 45 finer-grained labels (Adler and
Wilkerson, 2018; Hoyle et al., 2022).

Bookgenome contains book descriptions from Goodreads scored by crowd workers
against 727 user-generated tags (Kotkov et al., 2022). Of these we retain only
the max scoring tag per book, and exclude tags which are used fewer than 5
times. This results in 226 tags which contain a mix of high level and finer
grained tags, which serve as the gold labels in our evaluations of EdTM (§5).

Evaluation metrics. We evaluate EdTM using extrinsic clustering evaluation
metrics by comparing predicted clusters $\mathcal{C}^{\prime}$, to gold
clusters $\mathcal{C}$ induced by predicted and gold label assignments
respectively. We report the set overlap based metric $P_{1}$ Zhao (2005);
Amigó et al. (2009) and the mutual information (MI),
$I(\mathcal{C},\mathcal{C}^{\prime})$ Meilă (2007). $P_{1}$ is the harmonic
mean of cluster purity and inverse purity, and is bounded between 0 and 1.
While purity and inverse purity may be trivially maximized by over or under-
segmenting $D$ into $1$ or $|D|$ clusters respectively, the harmonic mean
trades these quantities off. Notably, $P_{1}$ captures the user experience in
data exploration setups – the coherence of clusters experienced by a user
while accounting for over-segmentation. While $P_{1}$ represents a cluster
level metric, MI represents a instance level metric – telling us the reduction
in uncertainty of the label for a point according to $\mathcal{C}$ if we know
its labeling according to $\mathcal{C}^{\prime}$.

In our evaluations we don’t employ the Normalized Mutual Information, NMI:
$I(\mathcal{C},\mathcal{C}^{\prime})/(H(\mathcal{C})+H(\mathcal{C}^{\prime}))$
commonly used in prior work since NMI results in higher metrics for imbalanced
clusterings i.e clusterings with low entropy ($H(C)$ or $H(C^{\prime})$), even
if lower in mutual information $I(\mathcal{C},\mathcal{C}^{\prime})$. Notably,
all our datasets represent a realistic imbalanced labeling raising the chances
of inflated NMI. Further, we don’t rely on the Adjusted Rand Index (ARI) since
it evaluates if pairs of points belong to the same cluster in $\mathcal{C}$
and $\mathcal{C}^{\prime}$ – while meaningful, pairwise relationships are less
aligned with analyst workflows of cluster exploration where topic models are
commonly employed.

Baselines. We compare EdTM against various few-shot classification and
clustering/topic modeling approaches. Clustering approaches range: LDA:
Represents a widely used approach to topic modeling representing documents as
mixtures of latent topics, in turn represented as mixtures of the corpus
vocabulary. Document-topic distributions are used for topic assignment. We use
the MALLET implementation of LDA with Gibbs sampling McCallum (2002). We set
$|V|=15,000$, $\alpha=1.0$, $\beta=0.1$, and run LDA for 2,000 iterations with
optimization at every 10 intervals. BertTopic: Represents an approach to topic
modeling implemented in the widely used BERTTopic package Grootendorst (2022).
Topic modeling is performed by embedding input texts with a pre-trained
BiEncoder333HF Model: all-MiniLM-L6-v2, reducing dimensionality with UMAP, and
clustering resulting embeddings using HDBSCAN. KMeans: A standard clustering-
based approach (MacQueen et al., 1967; Lloyd, 1982) to topic modeling,
embedding inputs using the pretrained bi-encoder $f_{\text{BE}}$ used in EdTM
then performing KMeans clustering. This has shown to be an effective approach
to topic modeling Thompson and Mimno (2020) – notably however, we cluster
document embeddings rather than contextualized word embeddings. TopicGPT: A
LLM based topic modeling approach, which involves prompting GPT-4 to generate
topics based on a small subset of $D$, and then using GPT-3.5-Turbo for
assigning the generated topics to all documents in $D$ Pham et al. (2024). For
LDA, BertTopic, and KMeans we set the number of clusters to be $|L|$. The
number of clusters in TopicGPT vary across datasets444$k=15$ for Twitter, $31$
for Wiki, $79$ for Bills, and $482$ for Bookgenome. since we cannot fix the
number of topics generated by GPT-4. Note here, that while we report the
performance of TopicGPT we preclude extensive comparison to it given its use
of two highly performant commercial LLMs, instead we treat it as an upper
bound in performance of existing approaches.

| Bookgenome | Bills | Wiki | Twitter
---|---|---|---|---
|  $|L|$: 226 |  $|L|$: 21 |  $|L|$: 15 |  $|L|$: 6
Method | P1 | MI | P1 | MI | P1 | MI | P1 | MI
TopicGPT | _0.18_ | _1.97_ | _0.57_ | _1.66_ | _0.73_ | _1.79_ | _0.75_ | _0.70_
LDA | 0.17 | 1.37 | 0.56 | 1.30 | 0.73 | 1.62 | 0.49 | 0.29
BertTopic | 0.15 | 1.04 | 0.39 | 0.93 | 0.52 | 1.17 | 0.53 | 0.31
KMeans | 0.16 | 1.77 | 0.46 | 1.27 | 0.55 | 1.37 | 0.55 | 0.58
GPT-3.5-Turbo | 0.25 | 1.92 | 0.51 | 1.21 | 0.71 | 1.59 | 0.55 | 0.21
NN${}_{\text{BE}}$ | 0.17 | 1.74 | 0.52 | 1.19 | 0.57 | 1.10 | 0.64 | 0.57
$\textsc{EdTM}_{\text{BE}}$ | 0.17 | 1.84 | 0.54 | 1.22 | 0.57 | 1.11 | 0.62 | 0.55
NN${}_{\text{CE}}$ | 0.20 | 1.76 | 0.58 | 1.34 | 0.61 | 1.25 | 0.71 | 0.68
$\textsc{EdTM}_{\text{CE}}$ | 0.20 | 1.77 | 0.58 | 1.35 | 0.65 | 1.37 | 0.72 | 0.70

Table 2: Cluster quality with EdTM. Underline for a baseline represents the
best evaluation metric, and underline for NN and EdTM indicate better or
matched performance to the best baseline. Bold indicates the better performing
model between NN and EdTM. We compare most closely to NN given that it
represents the most similar approach to EdTM as we note in §4.1.

While clustering approaches remain agnostic to analyst provided label names,
zero- and few-shot classification approaches leverage analyst provided label
names, these baselines range: GPT3.5-Turbo: This approach uses few-shot
prompting to assign one of the $L$ labels to each input document with
GPT3.5-Turbo. Nearest Neighbor (NN): This nearest neighbor model predicts the
label most similar to the input text using a similarity metric identical to
EdTM (§3.2), i.e labels are predicted as $\mathtt{argmin}_{l}\mathbf{C}$. We
differentiate Bi-Encoder and crossencoder approaches as NN${}_{\text{BE}}$ and
NN${}_{\text{CE}}$. Note that NN may seen as the greedy version of EdTM,
making label assignments greedily for each $d\in D$ and may be seen as most
similar to EdTM. For label assignment with TopicGPT and GPT-3.5-Turbo, we
truncate the input document and our topic list if their combination exceed the
LLM context length of 4,096 tokens.555We truncate either the topic list or the
document if either component exceeds approximately half of the input context
length (around 1,700 tokens). Given the large label space in Bookgenome,
truncation of the topic list is necessary. In such cases, we include only a
top set of candidate labels, which are selected based on their cosine
similarity with the input text as computed by a pretrained Bi-Encoder.666HF
Model: all-MiniLM-L6-v2

Implementation Details. For computing document-topic costs $\mathbf{C}$ in
EdTM, for $f_{\text{BE}}$ we use a 110M parameter BERT-like Bi-Encoder pre-
trained for dense retrieval on weakly supervised query-document pairs
constructed from web-forums. For $f_{\text{CE}}$, we leverage the MonoT5
crossencoder based on T5-XL and trained for query-document relevance on the
MS-MARCO dataset Nogueira et al. (2020). Owing to the size of Bookgenome, we
leverage a T5-Large crossencoder to keep experiments feasible.777HF Models;
$f_{\text{BE}}$: multi-qa-mpnet-base-cos-v1, $f_{\text{CE}}$:
monot5-3b-msmarco-10k While, the average number of tokens in input texts
across our datasets of Table 1 are $28$, $2888$, $215$, and $170$ – we retain
the first $450$ tokens to meet length limitations of LM/LLMs. Further, in
inputing label text to $f_{\text{BE}}$ we format labels as questions, e.g. for
Wiki, “Is this a Wikipedia article about LABEL?” to mimic the structure of
training data for $f_{\text{BE}}$. Next, for computing complete assignments
with optimal transport, we set the entropy regularizer $\lambda=1$, compute
$\mathbf{Q}$ in batches $b$ of size $500$ and averaged over 3 epochs. In
experiments we distinguish bi-encoder and crossencoder versions as
$\textsc{EdTM}_{\text{BE}}$ and $\textsc{EdTM}_{\text{CE}}$.

###  4.2 Main Results

Table 2 compares EdTM against baseline clustering as well as zero- and few-
shot prediction approaches on 4 datasets varying in characteristics.

Baseline performance. We begin by examining the performance of baseline
models. First, we note that LDA shows strong performance compared to other
clustering models, KMeans and BertTopic. Next, LDA sees significantly lower
performance on shorter Twitter texts and sees performance nearing that of
TopicGPT on the significantly longer texts in Wiki – this trend mirrors prior
results of poor performance on short texts Hong and Davison (2010). Next, we
consider the performance of GPT-3.5-Turbo. First, we note that GPT-3.5-Turbo
sees consistently lower performance compared to TopicGPT in 3 of 4 datasets –
indicating an inability of GPT-3.5-Turbo to make high quality assignments with
analyst provided labels $L$. However, it sees stronger performance in
Bookgenome. As we note in §4.1, due to its limited context length,
GPT-3.5-Turbo _re-ranks_ labels, this differs from the three other datasets –
relying on a first stage retrieval of labels using a bi-encoder followed by
LLM based assignment and likely explains its strong performance. This may also
represent a meaningful strategy for label supervised topic modeling with large
$L$ that may be explored in future work.

EdTM performance. We begin by examining EdTM compared with NN. Here, note that
$\textsc{EdTM}_{\text{BE}}$ matches or outperforms NN${}_{\text{BE}}$ in three
of four datasets and $\textsc{EdTM}_{\text{CE}}$ matches or outperforms
NN${}_{\text{CE}}$ in all four datasets – indicating the value of joint
assignment of texts to labels over the greedy assignment of NN. Next, we note
that use of a crossencoder improves upon the results of a bi-encoder in both
NN and EdTM indicating the value of more expressive text similarity models.
Finally, we compare the performance of $\textsc{EdTM}_{\text{BE}}$ and
$\textsc{EdTM}_{\text{CE}}$ with the best baseline approaches. Here we see
that EdTM results in improvements compared to clustering approaches in
Bookgenome, Bills, and Twitter. This may be attributed to the effective use of
label name supervision absent in clustering approaches. Further, EdTM methods
also outperform GPT-3.5-Turbo based assignment in Bills and Twitter data
indicating their value in domains likely to be missing from LLM pretraining
data. Finally, $\textsc{EdTM}_{\text{CE}}$ also approaches the performance of
TopicGPT in Bills, Bookgenome, and Twitter indicating its ability to induce
high quality clusters at par with large scale LLMs while adhering to analyst
provided topic labels.

##  5 Interaction Experiments

In Tables 3, 4, and 5 we present results in various interactive scenarios
demonstrating respectively, the ability of EdTM to incorporate finer grained
label descriptions, seed documents, and make high quality topic assignments in
the presence of incomplete topic names. We make complete assignments for label
descriptions and seed documents and partial assignments for incomplete topic
names. For each interactive evaluation we first describe the experimental
setup and follow with a discussion of experimental results.

| Bills | Wiki
---|---|---
|  $|L|$: 21 |  $|L|$: 15
Method | P1 | MI | P1 | MI
GPT-3.5-Turbo | 0.51 | 1.21 | 0.71 | 1.59
SeededLDA | 0.48 | 1.22 | 0.65 | 1.71
NN${}_{\text{BE}}$ | 0.55 | 1.27 | 0.65 | 1.32
$\textsc{EdTM}_{\text{BE}}$ | 0.58 | 1.31 | 0.67 | 1.36
NN${}_{\text{CE}}$ | 0.63 | 1.44 | 0.72 | 1.56
$\textsc{EdTM}_{\text{CE}}$ | 0.65 | 1.48 | 0.74 | 1.62

Table 3: Cluster quality with finer grained supervision provided per label in
the form of seed words. Bold indicates better performance between NN and EdTM,
and underline indicates better performance compared to a baseline.

###  5.1 Seed words as topic labels

Setup. In this experiment we simulate a scenario where an analyst authors
longer form topic descriptions instead of topic names alone. We limit
experiments to the Bills and Wiki dataset, and use their finer grained topic
labels to generate descriptions for each topic. We format these finer grained
topics into a natural language description for the target label set $L$, for
example: “Is this a wikipedia article about Media and drama or Television or
Film or Actors?”. For NN and EdTM, these richer labels are used to compute
document topic costs $\mathbf{C}$ that are used for topic assignment with bi-
encoders or crossencoders. Here, we also compare to SeededLDA Jagarlamudi et
al. (2012), an LDA topic model incorporating user provided seed words into
induced topics.

Results. In Table 3, first we note that both NN and EdTM outperform
GPT-3.5-Turbo and SeededLDA in Bills, and NN and EdTM outperform baselines in
Wiki with crossencoders. Further, comparing to Table 2, we note that addition
of label descriptions consistently improved performance for NN and EdTM. This
indicates NN and EdTM’s ability to incorporate rich natural language topic
labels from analysts. Finally, we note that across Bills and Wiki, EdTM
consistently outperforms NN with bi-encoder and crossencoder text similarities
– indicating joint assignments to benefit from improved similarities/cost
estimates.

###  5.2 Seed documents as topic labels

| Bookgenome | Bills | Wiki | Twitter
---|---|---|---|---
|  $|L|$: 226 |  $|L|$: 21 |  $|L|$: 15 |  $|L|$: 6
Method | P1 | MI | P1 | MI | P1 | MI | P1 | MI
NN${}_{\text{BE}}$ | 0.23 | 1.91 | 0.54 | 1.25 | 0.53 | 0.97 | 0.64 | 0.46
$\textsc{EdTM}_{\text{BE}}$ | 0.19 | 1.94 | 0.54 | 1.26 | 0.56 | 1.08 | 0.56 | 0.41

Table 4: Cluster quality with topic labels represented with averaged
embeddings of high precision seed documents retrieved using the target label
with a retrieval model.

Setup. Here, we simulate a scenario where analysts use topic names in $L$ to
perform a search over the corpus $D$, verifies their correctness, and uses the
verified sample documents as topic targets. This setup also mirrors a common
scenario where seed documents serve as queries for corpus exploration Wang and
Diakopoulos (2021). Here, we compare NN and EdTM alone given that few-shot
classification with GPT-3.5-Turbo runs into context length limitations in
using document examples for our datasets. Further, we only experiment with bi-
encoder variants given that crossencoders remain limited by context length
limitations for larger number of seed documents. For NN${}_{\text{BE}}$ and
$\textsc{EdTM}_{\text{BE}}$ we compute costs $\mathbf{C}$ between documents
$d$ and topic labels $l$, using the top five verified retrievals per label as:
$\text{L2}[f_{\text{BE}}(d),\mathtt{mean}_{k=1\dots
5}f_{\text{BE}}(d_{l}^{k})]$. We choose $k=5$ to represent a reasonable effort
to verify label-document correctness by an analyst.

Results. In Table 4, we note that $\textsc{EdTM}_{\text{BE}}$ matches or
outperforms NN${}_{\text{BE}}$ in Bills, Wiki, and Bookgenome (MI) indicating
its ability to incorporate seed document supervision. We also note, comparing
to Table 2 that while NN${}_{\text{BE}}$ and $\textsc{EdTM}_{\text{BE}}$ sees
improvement from using seed documents in Bookgenome and Bills, they see drops
in performance in datasets with fewer labels, Wiki and Twitter. This follows
from the finer grained labels of Bookgenome and Bills being better represented
using high precision seed documents. However, in Wiki and Twitter, seed
documents are only likely to represent certain aspects of the higher level
labels. The resulting lower quality document-topic similarities result in
especially degraded assignments in $\textsc{EdTM}_{\text{BE}}$ on Twitter.
However, in the presence of finer grained labels in the remaining three
datasets EdTM results in high quality topic assignments.

###  5.3 Partial assignment of topics

| Bookgenome | Bills | Wiki | Twitter
---|---|---|---|---
|  $|L|$: 226 |  $|L|$: 21 |  $|L|$: 15 |  $|L|$: 6
Method | P1 | MI | P1 | MI | P1 | MI | P1 | MI
NN${}_{\text{BE}}$ | 0.18 | 1.76 | 0.51 | 1.19 | 0.58 | 1.13 | 0.65 | 0.54
$\textsc{EdTM}_{\text{BE}}$ | 0.17 | 1.76 | 0.51 | 1.17 | 0.56 | 1.09 | 0.64 | 0.52
NN${}_{\text{CE}}$ | 0.21 | 1.79 | 0.55 | 1.27 | 0.62 | 1.29 | 0.71 | 0.65
$\textsc{EdTM}_{\text{CE}}$ | 0.20 | 1.76 | 0.55 | 1.31 | 0.64 | 1.33 | 0.73 | 0.65

Table 5: Cluster quality with a topic label omitted from the label set $L$,
following which NN and EdTM make partial assignments i.e witholding
predictions for certain inputs. This setup simulating a scenario where
analysts may not list all topic labels due to insufficient knowledge of a
corpus. Reported numbers are averaged over metrics obtained from excluding 3
high frequency topic labels one at a time.

Setup. To demonstrate the value of making partial assignments we simulate a
scenario where a label is missing from the target label set $L$, e.g. due to
an analyst having insufficient knowledge of the corpus topics. This creates a
scenario where a performant model should not make topic assignments for some
inputs. Specifically, we exclude one label, $l_{e}$ from $L$ selected at
random from the most frequent 5 labels and make a partial assignment with NN
or EdTM. We repeat this procedure with 3 different $l_{e}$ labels and report
averaged performance over 3 different clusterings. In each case, $p$ is set to
the proportion of documents with $l_{e}$ in the gold labeling. For making
partial assignments we follow the procedure outlined in §3.3. In computing
evaluation metrics we exclude input texts which don’t receive a cluster
assignment from NN and EdTM.

Results. In Table 5 we note that with bi-encoder costs, NN slightly
outperforms or matches the performance of EdTM. However, with crossencoder
costs, $\textsc{EdTM}_{\text{CE}}$ sees stronger performance than NN. Note
here that in the presence of missing labels in $L$, both NN and EdTM could
make alternative topic assignments for documents which could have received
$l_{e}$ while leaving a fraction $p$ of documents unassigned. For example,
when Language and literature topic name is excluded from $L$ in Wiki,
$\textsc{EdTM}_{\text{CE}}$ most frequently makes assignments to: Media and
drama, History, Philosophy and religion, and others. Manual examination
revealed these to often be reasonable. However, since we only consider single
topic assignments in our evaluations evaluating alternative topic assignments
or multi-topic assignments, more generally, remains future work. Nevertheless,
these results indicate the ability of EdTM to make high quality topic
assignments despite receiving incomplete topic sets $L$ indicating an ability
to handle the errors likely in interacting with users.

##  6 Related Work

We begin by discussing prior work on interactive and supervised topic
modeling. Then we discus prior work leveraging optimal transport and large
language models for topic modeling.

Interactive topic modeling. Topic modeling has focused on learning human
interpretable topics given only a corpus of documents. A vast body of work has
explored probabilistic generative models for topic modeling, representing
topics as distributions over word types and documents as mixtures of latent
topics Boyd-Graber et al. (2017); Lund (2019). We first examine the line of
work which has sought to incorporate supervision from users into these models.
Supervision from users has been of three broad types: (1) document labels
(e.g. sentiment) and metadata (eg. dates) paired with each document, (2) user
specified seed-words available at the corpus level, and (3) through
constraints (e.g. “must-link”) over latent topics. While work on (1) and (2)
have not historically been considered interactive topic modeling, we consider
them as such.

Incorporation of document level labels and metadata has been explored through
generation of labels conditioned on latent topics (Mcauliffe and Blei, 2007,
STM), conditioning document generation on observed metadata Card et al.
(2018), or as priors influencing document-topic distributions(Mimno and
McCallum, 2008, DMR). While these approaches aim to influence latent topics
with labels/metadata, Ramage et al. (2009, LabeledLDA) sought to ensure one-
to-one correspondence between latent topics and document multi-labels,
learning a word-label assignment. In ensuring a correspondence between topics
and labels LabeledLDA bears resemblance to EdTM. However, in contrast with
this line of work, EdTM does not assume knowledge of documents paired with
labels or metadata.

In assuming corpus level topic labels our work resembles prior work that seeks
to incorporate user provided seed words for topics. Here, work of Jagarlamudi
et al. (2012); Churchill et al. (2022) incorporates topic-seed words into the
generation of document words with a mixture of seeded and latent topic-word
distributions or through modified sampling schemes. On the other hand,
Harandizadeh et al. (2021) incorporate seed information into an embedded topic
model by regularizing the topic-word matrix. Relatedly, Akash et al. (2022)
incorporate weak supervision into topic-word and document-topic matrices from
a LabeledLDA and zero-shot classification model respectively. EdTM extends
topic models based on seed words to more verbose forms of topics supervision
such as descriptions and documents through its use of LM/LLM similarity
functions, while also showing partial assignments to result in accurate topic
assignments in the presence of noise in user provided topics.

Finally, a third line of work prior work has explored interaction with topic
models through “must-link” and “cannot-link” constraints available a-priori
Andrzejewski et al. (2009) or supplied interactively once a model of topics
has been learned Hu et al. (2011) with tree structured priors for word-topic
distributions. Interaction through such constraints provides a complementary
form of interaction than ours – we leave exploration of such constraints into
EdTM to future work.

While a large body of work has explored incorporation of various forms of
interaction into generative topic models such interactions have also been
explored for topic modeling in other frameworks. The Anchor algorithm for
topic modeling based on non-negative matrix factorization Lund (2019) was
extended to incorporate document metadata as well as seed word supervision
from users Nguyen et al. (2015); Lund et al. (2017). Pacheco and Goldwasser
(2021) introduce a probabilistic programming framework for relational learning
and demonstrate its use for interactive topic modeling via first order logic
rules Pacheco et al. (2023). In contrast with these approaches EdTM offers a
more natural form of interaction with topic models through long form natural
language interactions by leveraging LM/LLMs for modeling text similarities.
Work of Meng et al. (2020) learns a discriminative model for retrieving words
representative of a topic given only a topic name, bootstrapped from pre-
trained word-embeddings. While mirroring EdTM in its use of topic names it
remains limited to exploring greedy assignments similar to our nearest
neighbor (NN) baseline. Finally, while a large body of work, including EdTM,
has explored use of supervision to guide latent topics toward user provided
topics, Thompson and Mimno (2018) highlight the value of biasing topics away
from user labels and discovering more novel topical structure Ramage et al.
(2011) – we leave exploration of such considerations in EdTM to future work.

LLM topic models. The advent of highly performant LM/LLMs has lead recent work
to explore use of these models for topic modeling and text clustering. The
dominant line of work here has explored unsupervised topic modeling. These
approaches commonly consist of two stages, topic generation with an LLM
followed by a topic assignment to the generated topics. A considerable design
space has been explored for both stages. Pham et al. (2024, TopicGPT)
leverages GPT-4 for topic generation and GPT-3.5-Turbo for topic assignment,
we include TopicGPT in our experiments. Other two stage approaches have also
explored iterative generation of topic taxonomies rather than flat topic lists
Lam et al. (2024); Wan et al. (2024). For topic assignment, Lam et al. (2024)
explore multi-choice assignment with LLMs while Wan et al. (2024) train small
classifiers on LLM assignments to speed up test time topic assignment.
Notably, these approaches omit joint assignment of documents to topics as in
EdTM. In this regard, Wang et al. (2023, GoalEx) bears resemblance to EdTM,
leveraging integer linear programs for global document-topic assignment
following LLM based topic generation and initial greedy assignment. Differing
from GoalEx, EdTM’s use of optimal transport for assignment allows richer
document-topic costs to be used and fractional assignments to be made rather
than single topic assignments of GoalEx – however, both forms of joint
assignment may be valuable in various applications. Also similar in its use of
joint assignment to topic names is the work of Fei et al. (2022) – leveraging
greedy classification with pretrained embedding models followed by joint
document-topic assignment with Gaussian Mixture Models. However use of GMM’s
precludes use of black-box similarity functions such as crossencoders possible
to use in EdTM.

Besides leveraging the zero/few shot classification ability of LLMs for topic
modeling as in EdTM, a smaller body of work has also leveraged LLMs for
refining text clustering. Viswanathan et al. (2023) explore using LLMs to
augment input texts to clustering algorithms, generate oracle constraints for
constrained clustering methods, and iteratively re-assign low confidence
cluster assignments. On the other hand, Zhang et al. (2023) explore actively
training text similarity models based on LLM similarities and determining
cluster granularities with LLMs. Leveraging LLMs for actively refining
clustering presents an underexplored line of future work.

Optimal transport for topic models. A small body of work has leveraged optimal
transport for topic modeling. Zhao et al. (2020) learn encoders for
transforming bag of word document representations to topic distributions by
minimizing OT distances between the two distributions – building on embedded
topic models Dieng et al. (2020). On the other hand, Huynh et al. (2020) model
topic modeling as a dictionary learning problem, representing documents as
mixtures of latent topics which are learned by minimizing the Wasserstain
distance to bag of word document representations. While these approaches must
learn latent topics, we explore optimal transport for supervised or
interactive topic modeling. While this simplifies the problem to an assignment
problem, it introduces complexities such as dealing with noise in labels – to
the best of our knowledege no prior works have explored optimal transport in
this setup and EdTM represents initial work in this space.

Finally, Mysore et al. (2023) leverage the label-name interactions of EdTM for
developing a controllable recommendation model that represents historical user
documents using a model similar to $\textsc{EdTM}_{\text{BE}}$– in contrast,
this work explores various other forms of interaction, partial assignments,
and crossencoder similarities for interactive topic modeling. However, future
work may explore other downstream applications while leveraging a EdTM to
represent a collection of texts.

##  7 Conclusion

In this paper in introduce EdTM, a label supervised topic model. EdTM
leverages optimal transport based assignment algorithms to make globally
coherent topic assignments for documents based on pre-trained LM/LLM based
document-topic affinities. The proposed method results in high quality topics
compared to a range of baseline methods based on pretrained LM/LLMs, LDA topic
models, and clustering methods. EdTM is also shown to incorporate numerous
different forms of interaction from analysts while remaining robust to noise
in analysts’ input.

##  8 Acknowledgments

This work was supported in part by awards IIS-2202506 and IIS-2312949 from the
National Science Foundation (NSF), the Center for Intelligent Information
Retrieval, Lowe’s, IBM Research AI through the AI Horizons Network, and Chan
Zuckerberg Initiative under the project Scientific Knowledge Base
Construction. Any opinions, findings and conclusions or recommendations
expressed in this material are those of the authors and do not necessarily
reflect those of the sponsor.

## References

  * Adler and Wilkerson (2018) E Scott Adler and John Wilkerson. 2018.  Congressional bills project: 1995-2018.
  * Akash et al. (2022) Pritom Saha Akash, Jie Huang, and Kevin Chen-Chuan Chang. 2022.  Coordinated topic modeling.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9831–9843, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Amigó et al. (2009) Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo. 2009.  A comparison of extrinsic clustering evaluation metrics based on formal constraints.  _Information retrieval_ , 12:461–486.
  * Andrzejewski et al. (2009) David Andrzejewski, Xiaojin Zhu, and Mark Craven. 2009.  Incorporating domain knowledge into topic modeling via dirichlet forest priors.  In _Proceedings of the 26th annual international conference on machine learning_ , pages 25–32.
  * Antypas et al. (2022) Dimosthenis Antypas, Asahi Ushio, Jose Camacho-Collados, Vitor Silva, Leonardo Neves, and Francesco Barbieri. 2022.  Twitter topic classification.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3386–3400, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Benamou et al. (2015) Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré. 2015.  Iterative bregman projections for regularized transportation problems.  _SIAM Journal on Scientific Computing_ , 37(2):A1111–A1138.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Boyd-Graber et al. (2017) Jordan Boyd-Graber, Yuening Hu, and David Mimno. 2017.  Applications of topic models.  _Foundations and Trends® in Information Retrieval_ , 11(2-3):143–296.
  * Card et al. (2018) Dallas Card, Chenhao Tan, and Noah A. Smith. 2018.  Neural models for documents with metadata.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2031–2040, Melbourne, Australia. Association for Computational Linguistics.
  * Carlsen and Ralund (2022) Hjalmar Bang Carlsen and Snorre Ralund. 2022.  Computational grounded theory revisited: From computer-led to computer-assisted text analysis.  _Big Data & Society_, 9(1):20539517221080146.
  * Churchill et al. (2022) Robert Churchill, Lisa Singh, Rebecca Ryan, and Pamela Davis-Kean. 2022.  A guided topic-noise model for short texts.  In _Proceedings of the ACM Web Conference 2022_ , WWW ’22, page 2870–2878, New York, NY, USA. Association for Computing Machinery.
  * Cuturi (2013) Marco Cuturi. 2013.  Sinkhorn distances: Lightspeed computation of optimal transport.  _Advances in neural information processing systems_ , 26:2292–2300.
  * Dieng et al. (2020) Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020.  Topic modeling in embedding spaces.  _Transactions of the Association for Computational Linguistics_ , 8:439–453.
  * Fatras et al. (2020) Kilian Fatras, Younes Zine, Rémi Flamary, Remi Gribonval, and Nicolas Courty. 2020\.  Learning with minibatch wasserstein : asymptotic and gradient properties.  In _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , volume 108 of _Proceedings of Machine Learning Research_ , pages 2131–2141. PMLR.
  * Fei et al. (2022) Yu Fei, Zhao Meng, Ping Nie, Roger Wattenhofer, and Mrinmaya Sachan. 2022.  Beyond prompting: Making pre-trained language models better zero-shot learners by clustering representations.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8560–8579, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Finley and Joachims (2005) Thomas Finley and Thorsten Joachims. 2005.  Supervised clustering with support vector machines.  In _Proceedings of the 22nd International Conference on Machine Learning_ , ICML ’05, page 217–224, New York, NY, USA. Association for Computing Machinery.
  * Grootendorst (2022) Maarten Grootendorst. 2022.  Bertopic: Neural topic modeling with a class-based tf-idf procedure.  _arXiv preprint arXiv:2203.05794_.
  * Harandizadeh et al. (2021) Bahareh Harandizadeh, J. Hunter Priniski, and Fred Morstatter. 2021.  Keyword assisted embedded topic model.  _CoRR_ , abs/2112.03101.
  * Hong and Davison (2010) Liangjie Hong and Brian D. Davison. 2010.  Empirical study of topic modeling in twitter.  In _Proceedings of the First Workshop on Social Media Analytics_ , SOMA ’10, page 80–88, New York, NY, USA. Association for Computing Machinery.
  * Hong et al. (2022) Matt-Heun Hong, Lauren A. Marsh, Jessica L. Feuston, Janet Ruppert, Jed R. Brubaker, and Danielle Albers Szafir. 2022.  Scholastic: Graphical human-ai collaboration for inductive and interpretive text analysis.  In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ , UIST ’22, New York, NY, USA. Association for Computing Machinery.
  * Hoyle et al. (2022) Alexander Miserlis Hoyle, Pranav Goel, Rupak Sarkar, and Philip Resnik. 2022.  Are neural topic models broken? In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 5321–5344, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Hoyle et al. (2019) Alexander Miserlis Hoyle, Lawrence Wolf-Sonkin, Hanna Wallach, Isabelle Augenstein, and Ryan Cotterell. 2019.  Unsupervised discovery of gendered language through latent-variable modeling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1706–1716, Florence, Italy. Association for Computational Linguistics.
  * Hu et al. (2011) Yuening Hu, Jordan Boyd-Graber, and Brianna Satinoff. 2011.  Interactive topic modeling.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 248–257, Portland, Oregon, USA. Association for Computational Linguistics.
  * Huynh et al. (2020) Viet Huynh, He Zhao, and Dinh Phung. 2020.  Otlda: A geometry-aware optimal transport approach for topic modeling.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 18573–18582. Curran Associates, Inc.
  * Jagarlamudi et al. (2012) Jagadeesh Jagarlamudi, Hal Daumé III, and Raghavendra Udupa. 2012.  Incorporating lexical priors into topic models.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 204–213, Avignon, France. Association for Computational Linguistics.
  * Jasim et al. (2021) Mahmood Jasim, Enamul Hoque, Ali Sarvghad, and Narges Mahyar. 2021.  Communitypulse: Facilitating community input analysis by surfacing hidden insights, reflections, and priorities.  In _Proceedings of the 2021 ACM Designing Interactive Systems Conference_ , DIS ’21, page 846–863, New York, NY, USA. Association for Computing Machinery.
  * Kotkov et al. (2022) Denis Kotkov, Alan Medlar, Alexandr Maslov, Umesh Raj Satyal, Mats Neovius, and Dorota Glowacka. 2022.  The tag genome dataset for books.  In _Proceedings of the 2022 Conference on Human Information Interaction and Retrieval_ , CHIIR ’22, page 353–357, New York, NY, USA. Association for Computing Machinery.
  * Lam et al. (2024) Michelle S Lam, Janice Teoh, James Landay, Jeffrey Heer, and Michael S Bernstein. 2024.  Concept induction: Analyzing unstructured text with high-level concepts using lloom.  _arXiv preprint arXiv:2404.12259_.
  * Lin et al. (2020) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020.  Pretrained transformers for text ranking: Bert and beyond. corr abs/2010.06467 (2020).  _arXiv preprint arXiv:2010.06467_.
  * Lloyd (1982) Stuart Lloyd. 1982.  Least squares quantization in pcm.  _IEEE transactions on information theory_ , 28(2):129–137.
  * Lund (2019) Jeffrey Lund. 2019.  _Fine-grained Topic Models Using Anchor Words_.  Brigham Young University.
  * Lund et al. (2017) Jeffrey Lund, Connor Cook, Kevin Seppi, and Jordan Boyd-Graber. 2017.  Tandem anchoring: a multiword anchor approach for interactive topic modeling.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 896–905, Vancouver, Canada. Association for Computational Linguistics.
  * MacQueen et al. (1967) James MacQueen et al. 1967.  Some methods for classification and analysis of multivariate observations.  In _Proceedings of the fifth Berkeley symposium on mathematical statistics and probability_ , volume 1, pages 281–297. Oakland, CA, USA.
  * Mcauliffe and Blei (2007) Jon Mcauliffe and David Blei. 2007.  Supervised topic models.  _Advances in neural information processing systems_ , 20.
  * McCallum (2002) Andrew Kachites McCallum. 2002.  Mallet: A machine learning for languagetoolkit.  _http://mallet. cs. umass. edu_.
  * Meilă (2007) Marina Meilă. 2007.  Comparing clusterings—an information based distance.  _Journal of Multivariate Analysis_ , 98(5):873–895.
  * Meng et al. (2020) Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. 2020.  Discriminative topic mining via category-name guided text embedding.  In _Proceedings of The Web Conference 2020_ , pages 2121–2132.
  * Merity et al. (2018) Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018.  Regularizing and optimizing LSTM language models.  In _International Conference on Learning Representations_.
  * Mimno and McCallum (2008) David M Mimno and Andrew McCallum. 2008.  Topic models conditioned on arbitrary features with dirichlet-multinomial regression.  In _UAI_ , volume 24, pages 411–418. Citeseer.
  * Mysore et al. (2023) Sheshera Mysore, Mahmood Jasim, Andrew Mccallum, and Hamed Zamani. 2023.  Editable user profiles for controllable text recommendations.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_.
  * Nguyen et al. (2015) Thang Nguyen, Jordan Boyd-Graber, Jeffrey Lund, Kevin Seppi, and Eric Ringger. 2015\.  Is your anchor going up or down? fast and accurate supervised topic models.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 746–755, Denver, Colorado. Association for Computational Linguistics.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 708–718, Online. Association for Computational Linguistics.
  * Pacheco and Goldwasser (2021) Maria Leonor Pacheco and Dan Goldwasser. 2021.  Modeling Content and Context with Deep Relational Learning.  _Transactions of the Association for Computational Linguistics_ , 9:100–119.
  * Pacheco et al. (2023) Maria Leonor Pacheco, Tunazzina Islam, Lyle Ungar, Ming Yin, and Dan Goldwasser. 2023.  Interactive concept learning for uncovering latent themes in large text collections.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 5059–5080, Toronto, Canada. Association for Computational Linguistics.
  * Peyré et al. (2019) Gabriel Peyré, Marco Cuturi, et al. 2019.  Computational optimal transport: With applications to data science.  _Foundations and Trends® in Machine Learning_ , 11(5-6):355–607.
  * Pham et al. (2024) Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit Iyyer. 2024\.  Topicgpt: A prompt-based topic modeling framework.
  * Ramage et al. (2009) Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D. Manning. 2009.  Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora.  In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_ , pages 248–256, Singapore. Association for Computational Linguistics.
  * Ramage et al. (2011) Daniel Ramage, Christopher D. Manning, and Susan Dumais. 2011.  Partially labeled topic models for interpretable text mining.  In _Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’11, page 457–465, New York, NY, USA. Association for Computing Machinery.
  * Roberts et al. (2013) Margaret E Roberts, Brandon M Stewart, Dustin Tingley, Edoardo M Airoldi, et al. 2013.  The structural topic model and applied social science.  In _Advances in neural information processing systems workshop on topic models: computation, application, and evaluation_. Harrahs and Harveys, Lake Tahoe.
  * Thompson and Mimno (2018) Laure Thompson and David Mimno. 2018.  Authorless topic models: Biasing models away from known structure.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3903–3914, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Thompson and Mimno (2020) Laure Thompson and David Mimno. 2020.  Topic modeling with contextualized word representation clusters.  _arXiv preprint arXiv:2010.12626_.
  * Viswanathan et al. (2023) Vijay Viswanathan, Kiril Gashteovski, Carolin Lawrence, Tongshuang Wu, and Graham Neubig. 2023.  Large language models enable few-shot clustering.  _arXiv preprint arXiv:2307.00524_.
  * Wan et al. (2024) Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, Chirag Shah, Ryen W White, Longqi Yang, et al. 2024.  Tnt-llm: Text mining at scale with large language models.  _arXiv preprint arXiv:2403.12173_.
  * Wang and Diakopoulos (2021) Yixue Wang and Nicholas Diakopoulos. 2021.  Journalistic source discovery: Supporting the identification of news sources in user generated content.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , CHI ’21, New York, NY, USA. Association for Computing Machinery.
  * Wang et al. (2023) Zihan Wang, Jingbo Shang, and Ruiqi Zhong. 2023.  Goal-driven explainable clustering via language descriptions.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 10626–10649, Singapore. Association for Computational Linguistics.
  * Yadav et al. (2022) Nishant Yadav, Nicholas Monath, Rico Angell, Manzil Zaheer, and Andrew McCallum. 2022.  Efficient nearest neighbor search for cross-encoder models using matrix factorization.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2171–2194, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhang et al. (2023) Yuwei Zhang, Zihan Wang, and Jingbo Shang. 2023.  Clusterllm: Large language models as a guide for text clustering.  _arXiv preprint arXiv:2305.14871_.
  * Zhao et al. (2020) He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. 2020.  Neural topic model via optimal transport.  In _International Conference on Learning Representations_.
  * Zhao (2005) Ying Zhao. 2005.  _Criterion Functions for Document Clustering_.  Ph.D. thesis, University of Minnesota, USA.  AAI3180039.