# More Than Words: Collocation Tokenization for Latent Dirichlet Allocation
Models

Jin Cheevaprawatdomrong
Chulalongkorn University
<EMAIL_ADDRESS>
Alexandra Schofield
Harvey Mudd College
<EMAIL_ADDRESS>
Attapol T. Rutherford
Chulalongkorn University
<EMAIL_ADDRESS>

###### Abstract

Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a collection
of documents to discover their latent topics using word-document co-
occurrences. However, it is unclear how to achieve the best results for
languages without marked word boundaries such as Chinese and Thai. Here, we
explore the use of Pearson’s chi-squared ($\chi^{2}$) test, t-statistics, and
Word Pair Encoding (WPE) to produce tokens as input to the LDA model. The
$\chi^{2}$, t and WPE tokenizers are trained on Wikipedia text to look for
words that should be grouped together, such as compound nouns, proper nouns,
and complex event verbs. We propose a new metric for measuring the clustering
quality in settings where the vocabularies of the models differ. Based on this
metric and other established metrics, we show that topics trained with merged
tokens result in topic keys that are clearer, more coherent, and more
effective at distinguishing topics than those unmerged models.

##  1 Introduction

Latent Dirichlet allocation (LDA) models (Blei et al., 2003) provide useful
insights into themes and trends in a large text collection through the
unsupervised inference of topics, or probability distributions over unigram
word types in the corpus. In this model, a topic is often interpreted based on
its highest-probability words, with documents expressed in terms of
proportions of each topic. Unfortunately, the context in which these tokens
arise can be obscured in the bag-of-words rendering of text as unigram counts
in documents. For instance, a topic with high probabilities of both “coffee”
and “table” is tempting to interpret as focusing on the furniture item “coffee
table”, but both words could be frequent in a discussion of cafes containing
no coffee tables. This problem is amplified in languages without marked word
boundaries, such as Chinese and Thai: tokenizers for these languages may split
conceptual units into segments that, while functional as standalone words, do
not express the concept of the original text. Meaningful interpretation of
topics can be lost without careful recombination of these words.

In this paper, we evaluate three techniques to merge multiple adjacent words
into conceptually-unified phrasal tokens prior to LDA model inference: Pearson
Chi-square test, t-statistic, and word pair encoding (WPE). We apply merging
strategies to different language families including Indo-European language
(English), Kra-Dai language (Thai) and Sinitic language (Chinese). Inspired by
silhouette coefficients, we also introduce a new method to assess the
coherence of topics in a setting with variable vocabularies caused by
different pre-processing treatments, which was not possible with previously
proposed methods. Using this new metric and existing topic model evaluations,
we find that all three approaches to merging adjacent words can improve the
likelihood, coherence, and topic distinctiveness of LDA models.

##  2 Related Work

Despite their popularity in analyzing large amounts of text data, LDA models
are notoriously complex to evaluate. One must evaluate both the statistical
fit of a model and the human-registered thematic coherence of the words found
to arise in the high-probability words, or keys, of a topic, which may not
correlate Chang et al. (2009). Analyses often combine evaluations of fit
Wallach et al. (2009) and automated approximations of human judgments of
coherence Bouma (2009); Mimno et al. (2011) based on mutual information, even
with the expectation these may only somewhat correlate with true human
judgments Lau et al. (2014). A limitation of these existing approaches,
however, is that they expect the vocabulary and tokenization to remain
constant between two models. For our evaluation, we use a normalized log
likelihood approach to capture fit while accounting for changes in vocabulary
Schofield and Mimno (2016).

Pre-processing steps can meaningfully alter the results of the LDA models even
in languages with good tokenization heuristics such as English Schofield and
Mimno (2016); Schofield et al. (2017). We believe that languages that do not
have clear tokenization standards deserve investigation into what kind of
processing is appropriate. Many works recognize that LDA results can be
improved when input are including phrases (Lindsey et al., 2012; Yu et al.,
2013; El-Kishky et al., 2014; Wang et al., 2016; Bin et al., 2018; Li et al.,
2018). We consider it valuable to specifically assess approaches to
determining these phrases.

##  3 Collocations and Word Pair Encoding

_Collocations_ consist of two or more words that can express conventional
meaning. Since collocations can convey information about multi-word entities,
context, and word usage, we hypothesize that the introduction of multi-word
tokens, which capture collocations as unigrams through concatenation, can help
achieve more useful and coherent topic models. For languages that do not have
clear word boundaries, there is a possible additional benefit to multi-word
tokens: it can be hard to intuit whether inferred word boundaries will have a
large impact on the final results. Merging adjacent words into ’multi-word’
tokens may help remedy the potential problem of a segmentation that is not
optimal for topic modeling purposes.

Many methods are possible to select collocations from tokenized text, such as
frequency, mean and variance, and statistical hypothesis testing. In this
paper, we evaluate Pearson’s chi-squared test ($\chi^{2}$) and the t-statistic
for word co-occurrence, two hypothesis tests to determine if two words are
collocated significantly more than would occur randomly. To implement these
tests, we use the NLTK package to compute Bird (2006). We impose a minimum
frequency in the corpus for each selected bigram: otherwise, top bigrams from
the $\chi^{2}$ test will contain only exceptional rare words, as these are
expected to co-occur so rarely that even a few co-occurrences can trigger
significance.

Taking inspiration from byte-pair encoding, or BPE, we propose an alternative
to obtain _word-pair encoding_ (WPE) tokens. To do this, we first tokenize a
large corpus and then collect bigram counts for all bigrams found in the
corpus. Second, we merge the most frequent bigram to form a new WPE token.
This new bigram is then treated as a word in all occurrences. Next, we
continue to repeat the counting and merging process with one extra word type.
Finally, we obtain a vocabulary list of both unigram and WPE tokens.

##  4 Evaluation Metrics

Held-Out Likelihood. When multi-word phrases are converted to individual
tokens, the number of tokens in the document decreases while the size of the
corpus vocabulary increases. It is therefore illogical to compare the
likelihoods of the word-token model and WPE-token model directly.

In order to normalize the scores between the two models that do not have the
exact same vocabulary and tokens, we use the log-likelihood ratio between the
LDA model likelihood and the null (unigram) likelihood for each model. In
other words, we normalize the LDA model likelihood
($\mathcal{L}_{\textrm{model}}$) by dividing it with the unigram likelihood
($\mathcal{L}_{\textrm{unigram}}$) as introduced by Schofield and Mimno
(2016):

$\textrm{PTLL}_{\textrm{norm}}=\frac{\log\mathcal{L}_{\textrm{model}}-\log\mathcal{L}_{\textrm{unigram}}}{N}$
(1)

New Metric: Concatenation-based Embedding Silhouette (CBES) Previous measures
of topic coherence rely on statistics from the training data and assume that
the vocabularies are identical for both models, which is not the case for our
settings. To address this, we propose a new metric called a _concatenation-
based embedding silhouette_ (CBES), which measures the coherence within the
same topic and also the distinguishability of different topics in the LDA
results. CBES extends silhouette coefficients (Rousseeuw, 1987), a common
clustering evaluation metric, by projecting tokens and multiword tokens into
the same space and computing the silhouette coefficients in this vector space
in the usual way.

A good topic should have all of their topic keys close to each other and away
from other words that do not belong in the same topic. Silhouette coefficients
computed in this vector space capture exactly this. It is critical that
embeddings from the two models that we want to compare must be from the same
embedding space. We achieve this by concatenating each document with the
versions containing $\chi^{2}$, t, and WPE collocations before training the
embeddings. We use the gensim (Řehůřek and Sojka, 2010) implementation of with
the Continuous Bag-of-Word (CBOW) algorithm (Mikolov et al., 2013) to obtain
word embeddings.

##  5 Experiments

We test our methods on various corpora in English, Thai, and Chinese (Table
1). The English corpora are drawn from The New York Times (Sandhaus, 2008),
the Yelp Dataset 111www.yelp.com/dataset, and United States State of the Union
addresses (1790 to 2018) divided into paragraphs
222www.kaggle.com/rtatman/state-of-the-union-corpus-1989-2017. The Thai data
come from the news articles in Prachathai
333github.com/PyThaiNLP/prachathai-67k, the restaurant reviews from Wongnai
444www.kaggle.com/c/wongnai-challenge-review-rating-prediction, the BEST
corpus 555thailang.nectec.or.th/downloadcenter, and the Thai National Corpus
(Aroonmanakun, 2007). The Chinese data come from three corpora: the news
articles from Chinanews, restaurant reviews from Dianping,
666github.com/zhangxiangxiao/glyph and the movie reviews from Douban
777www.kaggle.com/utmhikari/doubanmovieshortcomments. Each corpus is separated
into 75% training documents and 25% test documents.

|  |  | %merged
---|---|---|---
Corpus |  Docs |  Tokens | $\chi^{2}$ | t | WPE
NYTimes | 80K | 2M | 15.43 | 15.99 | 16.55
SOTU | 21K | 1M | 12.21 | 12.90 | 13.53
Yelp | 200K | 13M | 9.47 | 10.33 | 12.16
TNC | 2K | 4M | 11.43 | 11.50 | 8.40
BEST | 4K | 6M | 12.43 | 12.50 | 9.53
Wongnai | 40K | 8M | 5.68 | 5.71 | 4.48
Prachathai | 68K | 119M | 13.41 | 13.45 | 10.36
Chinanews | 100K | 3M | 12.62 | 14.35 | 10.62
Dianping | 100K | 4M | 3.22 | 3.84 | 2.59
Douban | 200K | 2M | 4.61 | 5.18 | 3.99
Table 1:  A survey of corpora providing the number of documents and tokens, as
well as the percentage of unigram tokens merged using each approach.

We train the $\chi^{2}$, t and WPE-based tokenizers for each language on
Wikipedia articles for that language. For Thai and Chinese, we use the entire
Wikipedia database, but for English we use the filtered Wiki103 dataset
(Merity et al., 2016). English, Thai, and Chinese documents are tokenized with
NLTK (Bird, 2006), Attacut (Chormai et al., 2020), and Stanford Word Segmenter
(Tseng et al., 2005) respectively. We follow the same pre-precessing steps for
the training and the test documents: lemmatize and lowercase in English, and
remove stopwords, symbols and digits for all languages. We limit the
$\chi^{2}$, t and WPE approaches to 100,000 types. Note that the top
$\chi^{2}$ collocations are full of specific names and rare words from
Wikipedia because they appear together more than they would do randomly
(Figure 1). We use MALLET (McCallum, 2002) with the default hyperparameters to
train and evaluate topic models in both word and multi-word documents with 10,
50, 100 topics. We run the experiment 10 times for each combination of corpus,
type of model (word, Chi, t or WPE) and number of topics to compute the means
of the normalized held-out likelihood and CBES explained in section 4.

$\chi^{2}$: debes jugar, euskaltel euskadi, taare zameen, chetro ketl, hetch
hetchy, ngwat mahop, mullum malarum, pazz jop, phnom penh, eisernen kreuzes,
sirimalle chettu, kasa vubu, moondram pirai, gjems onstad, lettow vorbeck,
pather panchali, ioann zlatoust, kud wafter, poquita ropa, viribus unitis
---
t: united states, new york, world war, km h, take place, miles km, los
angeles, united kingdom, first time, high school, tropical storm, new zealand,
war ii, video game, mph km, h mph, north america, air force, two years, peak
number
WPE: unite state, new york, take place, first time, unite kingdom, follow
year, world war ii, also know, next day, new york city, high school, los
angeles, north america, even though, new zealand, follow day, become first,
also use, year old, take part
Figure 1:  Different collocation scoring methods result in different top 20
English collocations. Word: court federal judge charge case former trial say
rule today state sentence supreme prison justice accuse order law file jury
---
$\chi^{2}$: today rule washington say judge state law court federal case legal
supreme_court may ban order abortion lawyers allow violate laws
t: washington today state say rule judge federal law case ban may court
supreme_court violate seek right settlement abortion july allow
WPE: judge federal today charge court case trial washington former say lawyer
lawyers accuse supreme_court rule state hear order file federal_judge
Figure 2:  Topic keys about judges in State of the Union | 10 topics | 50 topics | 100 topics
---|---|---|---
| Word | $\chi^{2}$ | t | WPE | Word | $\chi^{2}$ | t | WPE | Word | $\chi^{2}$ | t | WPE
NYTimes | .3781 | .4263 | .6223 | .4460 | .5595 | .6105 | .6237 | .6439 | .6091 | .6501 | .6651 | .6867
SOTU | .2711 | .3032 | .3148 | .3273 | .3867 | .4240 | .4375 | .4573 | .4153 | .4444 | .4584 | .4821
Yelp | .1717 | .1974 | .2028 | .2149 | .2846 | .3226 | .3303 | .3506 | .3201 | .3586 | .3672 | .3883
TNC | .7614 | .7512 | .7468 | .7578 | 1.0214 | 1.0459 | 1.0484 | 1.0405 | 1.0735 | 1.1027 | 1.1073 | 1.0972
BEST | .7029 | .6742 | .6773 | .6969 | .9210 | .9293 | .9306 | .9358 | .9899 | 1.0053 | 1.0067 | 1.0097
Wongnai | .2014 | .2125 | .2141 | .2102 | .3191 | .3379 | .3378 | .3338 | .3492 | .3663 | .3675 | .3443
Prachathai | .4366 | .4723 | .4730 | .4659 | .7139 | .7761 | .7783 | .7599 | .8036 | .8736 | .8761 | .8565
Chinanews | .5161 | .5444 | .6560 | .5492 | .8114 | .8392 | .9548 | .8544 | .9186 | .9353 | .9758 | .9620
Dianping | .2571 | .2629 | .2649 | .2617 | .4087 | .4144 | .4179 | .4536 | .4538 | .4594 | .4631 | .4969
Douban | .2974 | .3027 | .3079 | .3046 | .4136 | .4139 | .4211 | .4168 | .4464 | .4417 | .4492 | .4451
Table 2:  Normalized unigram log-likelihood per token improvement in collocation models.  | 10 topics | 50 topics | 100 topics
---|---|---|---
| Word | $\chi^{2}$ | t | WPE | Word | $\chi^{2}$ | t | WPE | Word | $\chi^{2}$ | t | WPE
NYTimes | .0111 | .0293 | .0374 | .0484 | -.0509 | -.0451 | -.0415 | -.0305 | -.0820 | -.0768 | -.0748 | -.0690
SOTU | -.0052 | .0014 | .0081 | .0131 | -.0603 | -.0589 | -.0573 | -.0541 | -.0802 | -.0784 | -.0773 | -.0731
Yelp | -.0617 | -.0524 | -.0463 | -.0374 | -.1103 | -.1028 | -.0981 | -.0912 | -.1291 | -.1249 | -.1211 | -.1141
TNC | -.0223 | -.0125 | -.0161 | -.0196 | -.0935 | -.0810 | -.0837 | -.0862 | -.1134 | -.1031 | -.1041 | -.1072
BEST | -.0323 | -.0171 | -.0145 | -.0261 | -.0984 | -.0863 | -.0851 | -.0831 | -.1143 | -.0987 | -.1019 | -.0986
Wongnai | -.0618 | -.0658 | -.0676 | -.0655 | -.1417 | -.1406 | -.1404 | -.1406 | -.1631 | -.1591 | -.1602 | -.1616
Prachathai | -.0153 | .0094 | .0073 | .0025 | -.0814 | -.0632 | -.0600 | -.0692 | -.1124 | -.0922 | -.0900 | -.0995
Chinanews | -.0003 | .0091 | .0207 | .0222 | -.0536 | -.0530 | -.0459 | -.0406 | -.0694 | -.0679 | -.0582 | -.0545
Dianping | -.0614 | -.0539 | -.0518 | -.0516 | -.0977 | -.0940 | -.0908 | -.0933 | -.1143 | -.1124 | -.1126 | -.1137
Douban | .0072 | .0106 | .0208 | .0149 | -.0763 | -.0747 | -.0734 | -.0721 | -.0994 | -.1010 | -.1002 | -.0991
Table 3:  Silhouette improvement in collocation models

##  6 Results and Discussion

In general, corpora containing news have higher percentages of merged words,
while those containing restaurant and movie reviews tend to see lower
percentages (Table 1). This could be because the news corpora are in a similar
domain to that of the Wikipedia which we use to build the list of co-occurring
words. In $\chi^{2}$, t and WPE models, where the input contains multi-word
tokens, the results usually improve over the word-token model. Most of the
exceptions are from corpora containing restaurant and movie reviews, in which
the percentages of merged words are lower.

The normalized log-likelihood per token of the multi-word models is generally
higher than the word model across languages and corpora (Table 2). This means
the multi-word models are better than their corresponding word models in
reproducing the statistics of the held-out data. We also see a general
improvement in coherence in multi-word models (Table 3). Further, the higher
CBES score indicates that topic-keys are more semantically coherent and topics
are more distinct. The topic keys from multi-word models form a coherent
conceptual unit (Table 2). We can see that supreme_court in the multi-word
models is more meaningful than supreme or court in the word model. Similarly,
meaning of federal_judge is more precise than just federal and judge.

If we compare by looking at the topic-keys of the word and multi-word models,
we can come up with similar topics because we as a human who understands
English and have general knowledge of the world can make the connection based
on surrounding topic-keys that soviet and union, or super and bowl, or new and
york are part of connected words even though they are not explicitly merged.
However, if we want to use these topic-keys as input to other tasks such as
query search or neural network modeling, it is useful to feed the merged
tokens to be explicit that the bowl here doesn’t refer to the deep dish used
for food, and that the union here doesn’t refer to the worker association.

##  7 Conclusion

In this work, we improve the quality of LDA models by better processing the
input text before training the model. We found that all three approaches to
select candidate multi-word tokens—Pearson’s chi-squared test, t-statistics,
and word-pair encoding—improve the results of trained topic models across
numerous metrics. We also propose a new evaluation metric necessary for
evaluating LDA topic models in scenarios where pre-processing changes the
corpus vocabulary. As a future direction, we would like to explore other
collocation measures with applications to other language families that are
morphologically rich.

## References

  * Aroonmanakun (2007) Wirote Aroonmanakun. 2007.  Creating the thai national corpus.  _MANUSYA: Journal of Humanities_ , 10(3):4–17.
  * Bin et al. (2018) GE Bin, Chun-hui HE, Sheng-ze HU, and GUO Cheng. 2018.  Chinese news hot subtopic discovery and recommendation method based on key phrase and the lda model.  _DEStech Transactions on Engineering and Technology Research_ , (ecar).
  * Bird (2006) Steven Bird. 2006.  Nltk: the natural language toolkit.  In _Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions_ , pages 69–72.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Bouma (2009) Gerlof Bouma. 2009.  Normalized (pointwise) mutual information in collocation extraction.  _Proceedings of GSCL_ , pages 31–40.
  * Chang et al. (2009) Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-graber, and David Blei. 2009\.  Reading tea leaves: How humans interpret topic models.  In _Advances in Neural Information Processing Systems_ , volume 22. Curran Associates, Inc.
  * Chormai et al. (2020) Pattarawat Chormai, Ponrawee Prasertsom, Jin Cheevaprawatdomrong, and Attapol Rutherford. 2020.  Syllable-based neural thai word segmentation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4619–4637.
  * El-Kishky et al. (2014) Ahmed El-Kishky, Yanglei Song, Chi Wang, Clare Voss, and Jiawei Han. 2014.  Scalable topical phrase mining from text corpora.  _arXiv preprint arXiv:1406.6312_.
  * Lau et al. (2014) Jey Han Lau, David Newman, and Timothy Baldwin. 2014.  Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 530–539.
  * Li et al. (2018) Bing Li, Xiaochun Yang, Rui Zhou, Bin Wang, Chengfei Liu, and Yanchun Zhang. 2018\.  An efficient method for high quality and cohesive topical phrase mining.  _IEEE Transactions on Knowledge and Data Engineering_ , 31(1):120–137.
  * Lindsey et al. (2012) Robert Lindsey, William Headden, and Michael Stipicevic. 2012.  A phrase-discovering topic model using hierarchical pitman-yor processes.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 214–222.
  * McCallum (2002) Andrew Kachites McCallum. 2002.  Mallet: A machine learning for language toolkit.  Http://mallet.cs.umass.edu.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.  Pointer sentinel mixture models.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mimno et al. (2011) David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011.  Optimizing semantic coherence in topic models.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 262–272.
  * Řehůřek and Sojka (2010) Radim Řehůřek and Petr Sojka. 2010.  Software Framework for Topic Modelling with Large Corpora.  In _Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks_ , pages 45–50, Valletta, Malta. ELRA.
  * Rousseeuw (1987) Peter J Rousseeuw. 1987.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.  _Journal of computational and applied mathematics_ , 20:53–65.
  * Sandhaus (2008) Evan Sandhaus. 2008.  The new york times annotated corpus.  _Linguistic Data Consortium, Philadelphia_ , 6(12):e26752.
  * Schofield et al. (2017) Alexandra Schofield, Måns Magnusson, and David Mimno. 2017.  Pulling out the stops: Rethinking stopword removal for topic models.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 432–436.
  * Schofield and Mimno (2016) Alexandra Schofield and David Mimno. 2016.  Comparing apples to apple: The effects of stemmers on topic models.  _Transactions of the Association for Computational Linguistics_ , 4:287–300.
  * Tseng et al. (2005) Huihsin Tseng, Pi-Chuan Chang, Galen Andrew, Dan Jurafsky, and Christopher D Manning. 2005.  A conditional random field word segmenter for sighan bakeoff 2005.  In _Proceedings of the fourth SIGHAN workshop on Chinese language Processing_.
  * Wallach et al. (2009) Hanna M. Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. 2009.  Evaluation methods for topic models.  In _Proceedings of the 26th Annual International Conference on Machine Learning_ , ICML ’09, page 1105–1112, New York, NY, USA. Association for Computing Machinery.
  * Wang et al. (2016) Minmei Wang, Bo Zhao, and Yihua Huang. 2016.  Ptr: phrase-based topical ranking for automatic keyphrase extraction in scientific publications.  In _International Conference on Neural Information Processing_ , pages 120–128. Springer.
  * Yu et al. (2013) Zhiguo Yu, Todd R Johnson, and Ramakanth Kavuluru. 2013.  Phrase based topic modeling for semantic information processing in biomedicine.  In _2013 12th International Conference on Machine Learning and Applications_ , volume 1, pages 440–445. IEEE.