# Coordinated Topic Modeling

Pritom Saha Akash Jie Huang Kevin Chen-Chuan Chang
University of Illinois at Urbana-Champaign, USA
{pakash2, jeffhj<EMAIL_ADDRESS>
###### Abstract

We propose a new problem called coordinated topic modeling that imitates human
behavior while describing a text corpus. It considers a set of well-defined
topics like the axes of a semantic space with a reference representation. It
then uses the axes to model a corpus for easily understandable representation.
This new task helps represent a corpus more interpretably by reusing existing
knowledge and benefits the corpora comparison task. We design ECTM, an
embedding-based coordinated topic model that effectively uses the reference
representation to capture the target corpus-specific aspects while maintaining
each topic’s global semantics. In ECTM, we introduce the topic- and document-
level supervision with a self-training mechanism to solve the problem.
Finally, extensive experiments on multiple domains show the superiority of our
model over other baselines.111Code and data are available at
https://github.com/ pritomsaha/Coordinated-Topic-Modeling

##  1 Introduction

We often ask questions about well-defined topics when we read articles (e.g.,
news/research articles). E.g., in news domain, such a question can be like
“what is in entertainment today? (it’s about oscar 2022)” or in academic
domain, it can be like “what is trending in machine learning in 2016? (it’s
about deep learning)”. This is a practical human trait while understanding
information in a text corpus. Rather than finding arbitrary topics, people
often want to explore the text based on some well-defined topics. E.g., in
news domain, such topics are business, politics, sports, etc.

A well-defined topic is not merely a name (i.e., surface name); it generally
also has a representation (e.g., word distribution) that can be obtained from
a large text corpus, which we may call the reference representation. E.g., in
such representation of the topic sports, words like play, game, and player are
mainly dominated. Similarly, for politics, these are vote, election, party,
etc. However, a topic’s representation generally deviates from its reference
to other corpora. E.g., while describing a news corpus specific to the USA, it
is likely to use words such as nfl, baseball or football to represent sports,
while words like cricket, ipl or run are more relevant for India.

Figure 1: Coordinated interpretation of Topic model

The above scenario can be explained by the concept of the coordinate system in
geometry. Specifically, we can consider a set of well-defined topics as the
axes or basis of semantic space with their reference representation
$\beta^{r}$. E.g., in Figure 1, two topics $\mathcal{C}_{1}$:sports and
$\mathcal{C}_{2}$:politics represents two axes (i.e., reference axes) defined
by their reference representations $\beta^{r}_{1}$ and $\beta^{r}_{2}$
respectively. Defining a target corpus by these two topics is analogous to
finding a representation in the space defined by the reference axes. Figure 1
shows such target representation $\beta^{USA}$ and $\beta^{INDIA}$ as
coordinates by reference axes for corpora specific to USA and INDIA,
respectively.

In practice, the above interpretation of defining a corpus can be helpful from
several perspectives. First, it has better interpretability than traditional
document modeling approaches like topic models Blei et al. (2003); Jordan et
al. (1999) that discover some unknown topics. It defines a corpus over well-
defined topics by finding their corpus-specific representations. Thus, users
do not need much effort to understand unknown topics; instead, they easily
grasp the corpus-specific aspects of given well-known topics. Second, it
facilitates the comparability between two corpora. It uses a predefined set of
topics with their reference representation as the axes to describe a corpus in
a semantic space. Thus, two corpora represented by the same axes can be easily
comparable by studying their relative position in the coordinates. Third, it
allows reusability by utilizing existing information to explore new knowledge.
As the reference representation is already known information about some
topics, resuing it to model a new corpus in a similar domain is intuitive and
helpful. Specifically, this is useful for small target corpora as traditional
models are not typically effective in such cases.

Considering the above motivations, we thus formalize a new problem,
Coordinated Topic Modeling (CTM). It takes a target corpus $\mathcal{D}$, a
reference representation $\beta^{r}$ for $k$ well-defined topics with their
surface names $\mathcal{C}$. As output, it aims to learn a target
representation $\beta$ to best define $\mathcal{D}$. In many cases, we can
obtain a set of well-defined topics with their representation. E.g., there may
exist some large corpora in a particular domain annotated with topics, and
there are effective existing methods Ramage et al. (2009, 2011) to obtain the
reference representations. Specifically, we use labeled LDA Ramage et al.
(2009) to get reference representation (more details on Section 3.1)

Figure 2: Proposed Architecture

Some previous attempts incorporate prior knowledge into topic models to impose
more interpretability. One such work takes document-level supervision by
providing topic labels of all or a subset of documents Mcauliffe and Blei
(2007); Ramage et al. (2009). While they improve the predictive ability of
unsupervised topic models, they require a massive set of annotated documents
to be effective. An alternative way is called topic-level supervision by
providing seed words for each topic Eshima et al. (2020); Harandizadeh et al.
(2022). Although they make the topics converge toward the user’s interest, the
seed words should be only from the target corpus vocabulary, which may be
impractical in many cases. Finally, the category-guided topic mining Meng et
al. (2020a) considers the topic names as the only supervision for mining user-
desired discriminative topics. However, it also assumes that the name of each
topic appears in the corpus. Moreover, none of the above works does impose
requirements of making topics comparable over multiple corpora (see Appendix A
for a detailed discussion on related work).

From the above discussion, we identify the following two requirements that the
solution of CTM needs to meet. (1) The target representation should be learned
based on reference representation by capturing the target corpus-specific
aspects; (2) It also needs to relate the documents in the corpus with the
global semantics of each topic represented by its surface name so that it
maintains general interpretability and comparability. Although previous work
considers mining topics from user guidance, none of those fulfill these two
requirements.

There are several challenges to fulfilling the above requirements. The first
challenge is handling the vocabulary mismatch problem. As the vocabulary of
$\beta^{r}$ and $\mathcal{D}$ can be the different, $\beta^{r}$ cannot be
directly calibrated into the target $\beta$. To solve this problem, we propose
a method called reference projection. Here, the main idea is first to generate
a proxy representation $\tilde{\beta}^{r}$ of the target $\beta$ having the
same vocabulary with reference $\beta^{r}$, making it comparable for
supervision. It then enforces $\tilde{\beta}^{r}$ and $\beta^{r}$ to be as
close as possible, which indirectly allows the target $\beta$ to be guided by
given $\beta^{r}$ even if they have different vocabularies. Moreover, another
benefit of $\tilde{\beta}^{r}$ is that it enables directly comparing two
corpora given the same topics set.

The second challenge is providing surface names guidance. We only know the
surface name of each given topic without further knowledge of how each
document corresponds to them. Therefore, we generate document-level
supervision from the surface names using the textual entailment approach Yin
et al. (2019). It imitates how humans determine the topic(s) of a document by
filling in a template (e.g., “this document is about <topic name>”). In this
paper, we utilize a pre-trained textual entailment model Liu et al. (2019)
with given surface names to generate document-level distribution matrix
$\theta^{t}$. This $\theta^{t}$ later guides CTM by relating the global
semantics of surface names with given documents.

The final challenge is combining two supervisions. Now, we have topic-level
supervision from prior projection and document-level supervision from the
textual entailment model. To combine these two supervisions, we propose a
method called embedding based coordinated topic model, ECTM. It exploits the
architecture of an embedded topic model (ETM) Dieng et al. (2020). The main
idea is to regularize the objective of ETM using our two proposed
supervisions. To further generalize ECTM, we employ a mechanism similar to
self-training Meng et al. (2020b) where we iteratively use the model’s current
output to update $\theta^{t}$.

Our contributions can be summarized as follows. First, we propose coordinated
topic modeling, a new problem for modeling a text corpus using well-defined
topics as reference. Second, we develop an embedding-based framework for
solving the problem. It uses given reference representation to effectively
model a corpus while maintaining the semantics of given surface names. Third,
We propose methods for generating topic-level and document-level supervisions
using an introduced projection mechanism and the textual entailment approach,
respectively. We then combine these two supervisions into a unified model to
solve the CTM problem. Finally, we conduct a comprehensive set of experiments
on multiple domains for different tasks, demonstrating our framework’s
superiority against strongly designed baselines.

##  2 Proposed Methodology

Overview.

To solve the CTM problem, we design a method ECTM (Sec. 2.2) based on an
embedded topic model (ETM) Dieng et al. (2020) by imposing our requirements.
First, we incorporate a topic-level supervision (Sec. 2.2.1) from given
reference representation $\beta^{r}$ by introducing a mechanism called
reference projection. It generates a proxy representation of intended $\beta$
with the same vocabulary dimension with $\beta^{r}$, thus enabling
supervision. Second, we include a document-level supervision (Sec. 2.2.2) from
the given surface name of each topic by employing the textual entailment
approach Yin et al. (2019). Finally, we combine two supervisions (Sec. 2.2.3)
by regularizing the ETM’s objective, To further generalize the model, we
employ a self-training Meng et al. (2020b) by iteratively using the model’s
current output to update the supervision. The rest of the section first
reviews ETM and then presents our ECTM.

###  2.1 Embedded Topic Model

ETM is a neural topic model that uses vector representation of both words and
topics to improve topic quality and predictive accuracy of LDA Blei et al.
(2003). Let $\rho\in\mathbb{R}^{L\times V}$ and $\alpha\in\mathbb{R}^{L\times
K}$ be $L$-dimensional embeddings of $V$ vocabulary words and $K$ latent
topics respectively, ETM defines $k^{th}$ topic
$\beta_{k}=f(\rho^{\top}\alpha_{k})$ where $f(\cdot)$ is the softmax function.
It uses $\alpha$ in its generative process of $d^{th}$ document of corpus
$\mathcal{D}$ as follows:

  1. 1.

Draw topic proportion $\theta_{d}\sim\mathcal{LM}(0,I)$ where
$\mathcal{LM}(\cdot)$ is a logistic normal distribution Atchison and Shen
(1980) that transforms a standard Gaussian random variable to the simplex. In
other word, $\theta_{d}=f(\delta_{d})$ where $\delta_{d}=\mathcal{N}(0,I)$.

  2. 2.

For each word $n$ in the document:

    1. (a)

Draw topic assignment $z_{dn}\sim Cat(\theta_{d})$.

    2. (b)

Draw the word $w_{dn}\sim$ $f(\rho^{\top}$ $\alpha_{z_{dn}})$.

Here, $Cat(\cdot)$ denotes the categorical distribution. ETM employs variation
inference Jordan et al. (1999); Blei et al. (2017) that uses the evidence
lower bound (ELBO) as a function of the model parameters ($\alpha,\rho$) and
the variational parameters $v$:

$\displaystyle\mathcal{L}(\Theta)=\sum_{d\in\mathcal{D}}\sum_{n=1}^{N_{d}}\mathbb{E}_{q}[\log
p(w_{dn}\mid\delta_{d},\rho,\alpha)]-$
$\displaystyle\sum_{d\in\mathcal{D}}KL(q(\delta_{d};w_{d},v)\mid\mid
p(\delta_{d})),$ (1)

where $\Theta$ represents the model and variational parameters. $q(\cdot)$ is
a Gaussian whose mean, and variance are estimated from a neural network.

As a function of model parameters, ELBO in Eq. (1) is equivalent to the
expected complete log-likelihood maximization. On the other hand, the first
term in Eq. (1) encourages variational parameters to place mass on topic
proportions $\delta_{d}$ that explains the observed words, and the second term
forces them to be as close as possible to $p(\delta_{d})$.

###  2.2 ECTM

The proposed ECTM uses ETM as the base model to solve the CTM problem. There
are several reasons for this. Firstly, ETM combines the strength of the neural
topic model and word embedding for modeling corpus more effectively. Secondly,
it lets us use pre-trained word embedding to map words in a common vector
space even if the words do not appear in the target corpus vocabulary.
Thirdly, we can regularize the objectives of ETM by imposing our problem-
specific requirements.

Figure 2 shows the base model, excluding the parts involving dashed lines.
Similar to ETM, we use a two-layer perceptron to encode the bag of words (BOW)
representation $X_{d}$ of a document $d$ into document-topic distribution
$\theta_{d}$. At the same time, vector representation of words $\rho$ and
topics $\alpha$ generate topic-word distribution $\beta$. Finally, from
$\theta_{d}$ and $\beta$, $X_{d}$ is reconstructed as
$\hat{X}_{d}=\log(\theta_{d}^{\top}\beta+b)$. where $b$ is the background bias
representing the relative frequency of the vocabulary words in $\mathcal{D}$.
Unlike ETM, we include $b$ to account for words with approximately the same
frequency across documents. It helps produce coherent topics by weighing down
common words. Based upon this base model, ECTM consists of the following three
components for incorporating the CTM’s requirements.

####  2.2.1 Topic-level Supervision

Our first requirement stems from the fact that, in many cases, describing a
corpus with some arbitrary topics incurs a burden for users to understand them
first. Instead, it is more convenient to summarize the corpus over some well-
defined topics. Thus, in CTM, we consider a set of well-defined topics with
reference representation $\beta^{r}$ (i.e., word distribution) like the axes
in semantic space by employing supervision to generate a target representation
$\beta$ that best describes the given corpus $\mathcal{D}$. Here, $\beta^{r}$
may come from existing sources. E.g., a large corpus in a similar domain
annotated with topics can be used by supervised topic models as they are
effective for finding high-quality topics from a large annotated corpus.

Now, the question is, how can we use $\beta^{r}$ as reference axes in the
semantic space to guide generating $\beta$? One possible solution can be
constraining the ETM so that the generated $\beta$ comes close to $\beta^{r}$
while also maximizing ELBO. However, the problem is that we cannot assume that
$\beta^{r}$ and $\beta$ share the same vocabulary. Hence, $\beta^{r}$ cannot
be used directly as guidance. To solve this problem, we take an indirect way
of providing supervision which we call reference projection.

Reference Projection.

As shown in Figure 2, alongside $\beta$, we also generate a representation
$\tilde{\beta}^{r}=f(\tilde{\rho}^{\top}\alpha)$. Here, $\tilde{\rho}$ is the
embedding matrix of the vocabulary that the given reference $\beta^{r}$ is
based on. Now, we aim to enforce $\tilde{\beta}^{r}$ and $\beta^{r}$ to be as
close as possible by minimizing the following:

$\displaystyle
R_{\beta}=\frac{1}{k}\sum_{j=1}^{k}KL(\beta^{r}_{j},\tilde{\beta}^{r}_{j}).$
(2)

where $k$ is the number of topics. Here, minimizing $R_{\beta}$ makes the
model update the topic embedding matrix $\alpha$, which involves generating
$\beta$. Thus, it indirectly guides $\beta$ even if there is a vocabulary
mismatch between $\beta$ and $\beta^{r}$. In other words, $\beta$ is a kind of
projection of $\beta^{r}$ on the target corpus vocabulary dimension.

####  2.2.2 Document-level Supervision

Figure 3: Document-level Supervision Generation

According to the second requirement, we want to maintain a global semantic of
the given topic represented by its surface name. The reason is that if the
target representation of each topic deviates much from its well-defined
meaning, it may compromise the interpretability.

For this, we first try to understand the semantics of a document by relating
it with given surface names. Mainly, we use a textual entailment approach Yin
et al. (2019) to calculate the probability of a document belonging to a topic
(i.e., surface name). As humans, based on a document’s content, we can
determine the topic(s) by asking to what extent that document belongs to a
topic out of some options. Similarly, using the textual entailment approach,
we imitate humans to determine the probability that a document is about a
topic by creating a hypothesis by filling in the surface name in a template
(e.g., “this document is about <surface name>”), given the context document.

Given the surface names $\mathcal{C}$, we generate $\theta^{t}$ for document-
level supervision by using a pre-trained textual entailment model Liu et al.
(2019) fine-tuned on Yahoo Answers topic
classification222https://huggingface.co/joeddav/bart-large-mnli-yahoo-answers.
It considers an input document $d$ as the “premise”, creates a “hypothesis”
made of a template filled with a $c_{k}\in\mathcal{C}$, and generates a
probability $p_{dk}$ denoting to what extent the premise entails the
hypothesis. We have two major choices of $\theta^{t}$ from these generated
probabilities. Firstly, we can use hard labeling Lee et al. (2013) which
converts high-probability over a threshold $\tau$ to one-hot labels, i.e.,
$\theta_{dk}^{t}=1(p_{dk}>\tau)$ where $1(\cdot)$ is the indicator function.
Secondly, we can use the generated $p_{dk}$ as it is for $\theta_{dk}^{t}$ as
a soft label. In soft labeling, we can further use an approach Bhatia et al.
(2016) which elevates the high-probability label while demoting the low-
probability ones. More specifically, it squares and normalizes the $p_{dk}$ as
follows:

$\displaystyle\theta^{t}_{dk}=\frac{p_{dk}^{2}/f_{k}}{\sum_{k^{\prime}}p_{d{k^{\prime}}}^{2}/f_{k^{\prime}}},\quad
f_{k}=\sum_{d\in\mathcal{D}}p_{dk}.$ (3)

We observe that the soft labeling strategy consistently performs better and
provides more stable results than the hard labeling. The likely reason is that
as hard labeling considers high-probability topics as direct labels, it is
more susceptible to error propagation. Moreover, another drawback of hard
labeling is that we need to set a threshold explicitly, while soft labeling
does not require that. Figure 3 shows the overview of generating $\theta^{t}$.

The $\theta^{t}$ is then used to provide document-level supervision by
minimizing the following :

$\displaystyle
R_{\theta}=\frac{1}{\mid\mathcal{D}\mid}\sum_{d\in\mathcal{D}}KL(\theta^{t}_{d},\theta_{d}).$
(4)

As we use an existing knowledge source pre-trained on a massive volume of
documents over various domains, this supervision helps maintain the global
semantics of each topic in its target representation. Moreover, it also
improves the model’s predictive power to identify its topic.

####  2.2.3 Unification and Self-training

Now, we have topic-level and document-level supervision $R_{\beta}$ and
$R_{\theta}$ respectively by imposing CTM’s requirements. We unify them into
one model by constraining the objective of our base model as follows:

$\displaystyle\mathcal{L}(\Theta)=ELBO-\lambda_{\beta}R_{\beta}-\lambda_{\theta}R_{\theta},$
(5)

where $\lambda_{\beta}$ and $\lambda_{\theta}$ are the regularization weights
for $R_{\beta}$ and $R_{\theta}$ respectively. Maximizing Eq. (5) jointly
ensures the following objectives: (1) The ELBO part enforces the model to
explain $\mathcal{D}$ by reducing the reconstruction error; (2) $R_{\beta}$
encourages the model to converge $\beta$ in the direction of $\beta^{r}$; and
(3) $R_{\theta}$ encourages the model to maintain the global semantics of
given coordinates in $\beta$ by enforcing $\theta$ and $\theta^{t}$ as close
as possible.

Finally, to further generalize the model’s predictive strength in $\theta$, we
use a self-training Meng et al. (2020b) like mechanism. Here, the main idea is
to iteratively use the model’s current $\theta$ to update $\theta^{t}$ for
supervision. More specifically, we update $\theta^{t}$ after every $50$
iteration as follows:

$\displaystyle\theta^{t}=0.5*\theta^{t}+0.5*\theta.$ (6)

It also reduces the chance of error propagation from the textual entailment
model. We summarize our ECTM framework in Algorithm 1.

Input: A target corpus $\mathcal{D}$; a set of topics with surface names
$\mathcal{C}$ and reference representation $\beta^{r}$; a pre-trained word
embedding $\mathcal{W}$; an entailment model $\mathcal{E}$.

Output: Trained Model $\mathcal{M}$ with $\beta$.

1: Use $\mathcal{E}$ to initialize $\theta^{t}$ $\leftarrow$ Section 2.2.2;

2: From $\mathcal{W}$, obtain $\rho$ and $\tilde{\rho}$;

3: $B$ $\leftarrow$ Total number of batches;

4: for $i\leftarrow$ 0 to $B-1$ do

5: Train model $\mathcal{M}$ on $\mathcal{D}$ with Eq. (5);

6: if $i$ mod $50=0$ then

7: Update $\theta^{t}$ with Eq. (6);

8: end if

9: end for

10: return $\mathcal{M}$;

Algorithm 1 ECTM

##  3 Experiments

In this section, we employ empirical evaluations, which are designed mainly to
answer the following research questions (RQs): RQ1. How effective is the ECTM
quantitatively in terms of the quality of topics for the CTM problem and text
classification performance? RQ2. How does the ECTM qualitatively perform in
terms of interpretability and distinctiveness of generated topic words? RQ3.
How does ECTM benefit the task of comparing multiple corpora? RQ4. How does
each part of ECTM contribute to its performance?

###  3.1 Experiment Setup

Datasets.

We use datasets from three different domains news articles, review sentiment
and academic articles domains. For news articles, we obtain three datasets:
(1) 20 Newsgroup corpus333http://qwone.com/~jason/20Newsgroups/ (20Newsg); (2)
New York Times annotated corpus (NYT) Sandhaus (2008); (3) AG’s News dataset
(AGNews) from Yang et al. (2016). For review sentiment domain, we use: (1)
Yelp restaurant review dataset used in Meng et al. (2020a) (Yelp-Sent); (2)
IMDB Movie Review dataset used in Hoang et al. (2019) (IMDB-Sent). Finally for
academic articles: (1) Arxiv Artificial Intelligence (AI) article abstracts
spanning 2020-2022 444https://www.kaggle.com/Cornell-University/arxiv (Arxiv-
AI); (2) Microsoft Academic Graph AI article abstracts Sinha et al. (2015)
(MAG-AI).

Baselines.

We compare our model with the following baselines.

  * •

GLDA: Guided LDA Jagarlamudi et al. (2012) biases LDA’s generative process
using topic-level priors over vocabulary from given seed words.

  * •

Sup+LLDA: Supervised Labeled LDA is based on Labeled-LDA Ramage et al. (2009)
where a label for each document is predicted from a supervised BERT classifier
555https://huggingface.co/docs/transformers/tasks/sequence_classification
learned on annotated reference corpus.

  * •

ZS+LLDA: Zero-Shot Labeled LDA is also based on Labeled-LDA Ramage et al.
(2009) where a label for each document is inferred from given surface names
using a Zero-Shot classification Yin et al. (2019).

  * •

ACorEx: Unlike LDA, Anchored CorEx Gallagher et al. (2017) is not based on
generative assumptions and uses topic correlation to learn maximally
informative topics. Moreover, it uses user-provided seed words as anchors to
bias compression of the original corpus.

  * •

AVIAD: AVIAD Hoang et al. (2019) extends the Autoencoding Variational
Inference for Topic Models (AVITM) Srivastava and Sutton (2017) approach to
incorporate prior knowledge from seed words by modifying the loss function to
infer desired topics. It uses a variational autoencoder like ETM but does not
employ word embedding in the modeling.

  * •

KeyETM: Keyword Assisted ETM Harandizadeh et al. (2022) modifies ETM’s
objective to include prior knowledge from given seed words.

Reference Topic Representation.

As an input, CTM requires prior knowledge $\beta^{r}$. For each domain, we use
a large annotated corpus as a reference to get $\beta^{r}$ using labeled LDA
(LLDA) Ramage et al. (2009). Because LLDA is effective in producing high-
quality topics when there is a large annotated corpus. E.g., in news domain,
we use a large annotated corpus AGNews as reference with topic names business,
politics, sports and technology. In the sentiment domain, the reference corpus
we used is IMDB with positive (good) and negative (bad) sentiments. Finally,
we use the MAG-AI as reference corpus for the AI domain to generate prior,
annotated with four topics: computer vision (CV), information retrieval (IR),
machine learning (ML), and natural language processing (NLP).

Seed words.

For the baselines that require seed words, we collect them from given surface
names $\mathcal{C}$ and top topic words based on given $\beta^{r}$. If a word
does not appear in the corresponding target corpus, we replace that with the
most similar word using cosine similarity. As suggested in the baselines, we
provide 10 seed words for each topic.

The implementation details of our model and baselines are specified in
Appendix B.

Methods | 20Newsg | NYT | Yelp-Senti | Arxiv-AI
---|---|---|---|---
TC | TD | TQ | TC | TD | TQ | TC | TD | TQ | TC | TD | TQ
GLDA | 0.25 | 0.87 | 0.22 | 0.26 | 0.85 | 0.22 | 0.08 | 0.80 | 0.06 | 0.09 | 0.93 | 0.09
Sup+LLDA | 0.23 | 0.79 | 0.18 | 0.20 | 0.63 | 0.12 | 0.06 | 0.70 | 0.04 | 0.04 | 0.46 | 0.02
ZS+LLDA | 0.23 | 0.80 | 0.18 | 0.17 | 0.65 | 0.11 | 0.06 | 0.76 | 0.05 | 0.14 | 0.80 | 0.11
ACorEX | 0.25 | 1.00 | 0.25 | 0.27 | 1.00 | 0.27 | 0.07 | 1.00 | 0.07 | -0.03 | 0.96 | -0.03
AVIAD | 0.13 | 1.00 | 0.13 | -0.26 | 1.00 | -0.26 | -0.01 | 1.00 | -0.01 | -0.34 | 1.00 | -0.34
KeyETM | 0.26 | 1.00 | 0.26 | 0.19 | 0.89 | 0.17 | 0.07 | 0.92 | 0.07 | 0.04 | 1.00 | 0.04
ECTM | 0.30 | 1.00 | 0.30 | 0.28 | 0.97 | 0.27 | 0.09 | 1.00 | 0.09 | 0.15 | 0.97 | 0.15

Table 1: Quality Measures of Topic

| 20Newsg | NYT | Yelp-Senti | Arxiv-AI
---|---|---|---|---
sports | politics | business | technology | good | bad | ML | IR
| Reference
---
Topic
Words
| night
---
play
sport
player
beat
| leader
---
election
attack
afp
iraqi
| stock
---
sale
share
billion
fall
| software
---
technology
service
internet
launch
| song
---
music
musical
wonderful
dance
| waste
---
awful
terrible
boring
poor
| machine
---
learning
algorithm
optimization
problem
| retrieval
---
document
query
search
base
GLDA |  | game
---
team
year $(\times)$
play
player
| people
---
time $(\times)$
government
gun
year $(\times)$
| company
---
percent
year $(\times)$
bank
market
| president $(\times)$
---
bush $(\times)$
official $(\times)$
united $(\times)$
house $(\times)$
| good
---
place $(\times)$
food $(\times)$
great
order $(\times)$
| order $(\times)$
---
food $(\times)$
time $(\times)$
place $(\times)$
service $(\times)$
| adversarial
---
distribution $(\times)$
class $(\times)$
function $(\times)$
attack $(\times)$
| graph
---
search
user
class $(\times)$
recommendation
Sup+LLDA |  | game
---
team
year $(\times)$
play
time $(\times)$
| people
---
government
kill
time $(\times)$
year $(\times)$
| year $(\times)$
---
percent
company
market
government $(\times)$
| year $(\times)$
---
time $(\times)$
people $(\times)$
president $(\times)$
official $(\times)$
| place $(\times)$
---
food $(\times)$
good
great
service $(\times)$
| food $(\times)$
---
order $(\times)$
place $(\times)$
service $(\times)$
time $(\times)$
| demonstrate $(\times)$
---
problem $(\times)$
feature
training
neural
| retrieval
---
exist $(\times)$
demonstrate $(\times)$
representation
feature $(\times)$
ZS+LLDA |  | game
---
team
year $(\times)$
play
player
| people
---
time $(\times)$
government
year $(\times)$
point $(\times)$
| year $(\times)$
---
percent
company
market
lead $(\times)$
| year $(\times)$
---
time $(\times)$
american $(\times)$
official $(\times)$
today $(\times)$
| food $(\times)$
---
place $(\times)$
good
great
service $(\times)$
| food $(\times)$
---
order $(\times)$
place $(\times)$
service $(\times)$
time $(\times)$
| efficient $(\times)$
---
reduce $(\times)$
number $(\times)$
leverage $(\times)$
module $(\times)$
| retrieval
---
search
user
document
query
ACorEX |  | point
---
play
player
league
beat
| force
---
country
attack
military
political
| billion
---
business
buy
stock
profit
| release $(\times)$
---
technology
phone
time $(\times)$
space
| good
---
hear $(\times)$
beautiful
music $(\times)$
sound $(\times)$
| bad
---
money $(\times)$
terrible
poor
waste
| optimization
---
gradient
convergence
stochastic
print $(\times)$
| search
---
document
query
retrieval
semantics
AVIAD |  | robitaille $(\times)$
---
probert $(\times)$
howe $(\times)$
player
nhl
| tragedy $(\times)$
---
policy
serbian
freedom
unite $(\times)$
| sanwa $(\times)$
---
zoete $(\times)$
earning
overprice
acquirer
| genscher $(\times)$
---
enlargement $(\times)$
abm $(\times)$
teng $(\times)$
chechnya $(\times)$
| traditional $(\times)$
---
snow $(\times)$
filling
bisque $(\times)$
seaweed $(\times)$
| email $(\times)$
---
upset
management $(\times)$
yell
acknowledge $(\times)$
| bind
---
analytically $(\times)$
certify $(\times)$
arm $(\times)$
pruning
| ehr
---
healthy $(\times)$
progression $(\times)$
patient $(\times)$
ehrs
KeyETM |  | game
---
team
season
play
win
| people
---
government
person $(\times)$
armenian
law
| year $(\times)$
---
percent
market
time $(\times)$
month $(\times)$
| company $(\times)$
---
bank $(\times)$
japan $(\times)$
china $(\times)$
russia $(\times)$
| good
---
place $(\times)$
great
time $(\times)$
love
| food $(\times)$
---
order $(\times)$
service $(\times)$
eat $(\times)$
restaurant $(\times)$
| function
---
estimation $(\times)$
distribution $(\times)$
parameter $(\times)$
efficient $(\times)$
| translation $(\times)$
---
user
search
annotation
point $(\times)$
ECTM |  | game
---
team
win
season
league
| government
---
war
military
armenian
attack
| company
---
bank
percent
market
price
| space
---
site
technology
station
network
| great
---
music $(\times)$
love
wonderful
amazing
| waste
---
awful
terrible
bad
horrible
| optimization
---
convergence
stochastic
gradient
function
| retrieval
---
document
query
search
user

Table 2: Qualitative Evaluation

###  3.2 Topic Quality

Evaluation Metrics.

We use the following three well-defined quantitative measurements to evaluate
the inferred topics’ quality:

  * •

Topic coherence (TC) is a standard measure of interpretability based on the
average point-wise mutual information between randomly drawn two words from
the same document Lau et al. (2014).

  * •

Topic diversity (TD) measures the percentage of unique words in the top 25
words of all topics Dieng et al. (2020). It captures the semantical
diverseness of the inferred topics.

  * •

Topic quality (TQ) Dieng et al. (2020) is the overall metric for measuring the
quality of topics as the product of topic coherence and diversity.

Results and Discussions.

We first show the quantitative results of topic quality in Table 1. The
results suggest that, in general, ECTM generates more coherent and
interpretable topics than other baselines. In some cases, other methods
produce more diverse topics than ours. E.g., ECTM’s generated topics’
diversity scores are slightly lower than the best one in NYT and Arxiv-AI
datasets. However, our method significantly outperforms others in quality
scores in those cases by producing more coherent topics. Thus, ECTM produces
more interpretable topics while also maintaining diversity.

In Table 2, we show randomly selected two topics from each dataset and show
the top-5 words under each topic. We also show the top-5 words from the given
$\beta^{r}$. Words that authors determined to be irrelevant to the
corresponding topic are marked with ($\times$). Overall, our method-generated
topic words are relevant and easily interpretable in nearly all cases compared
to baselines. We also observe that the topics produced by AcorEx are
reasonably good in terms of interpretability. However, AcorEx’s produced
topics strictly converge toward the prior representation rather than adapting
according to the target corpus. E.g., in sports topic, 3 out of 5 topic words
overlap with priors topic words. In contrast, our method tends to capture the
target corpus-specific aspects of the given topics.

On the other hand, AVIAD suffers from the opposite issue. It adapts too much
that the topics become very difficult to understand. The KeyETM with a similar
base model (ETM) to ours performs better when the target corpus is balanced.
E.g., as 20Newsg is a comparatively balanced dataset, the keyETM performs
well. In contrast, our proposed ECTM consistently performs better as it enjoys
the benefit of both topic-level supervision from $\beta^{r}$ and document-
level supervision from existing knowledge sources to make the topics adjusted
to the target corpus as well as not going away from well-known semantics of
given topics’ names.

Methods | 20Newsg | NYT | Yelp-Senti | Arxiv-AI
---|---|---|---|---
Acc. | F1-Mac | F1-Mic | Acc. | F1-Mac | F1-Mic | Acc. | F1-Mac | F1-Mic | Acc. | F1-Mac | F1-Mic
GLDA | 0.70 | 0.71 | 0.73 | 0.70 | 0.59 | 0.75 | 0.75 | 0.75 | 0.75 | 0.67 | 0.59 | 0.71
Sup+LLDA | 0.68 | 0.53 | 0.62 | 0.77 | 0.70 | 0.80 | 0.94 | 0.94 | 0.94 | 0.38 | 0.37 | 0.38
ZS+LLDA | 0.89 | 0.80 | 0.88 | 0.91 | 0.79 | 0.90 | 0.87 | 0.87 | 0.87 | 0.34 | 0.34 | 0.33
ACorEX | 0.38 | 0.43 | 0.44 | 0.72 | 0.63 | 0.73 | 0.60 | 0.59 | 0.58 | 0.66 | 0.51 | 0.66
AVIAD | 0.72 | 0.71 | 0.74 | 0.69 | 0.52 | 0.72 | 0.75 | 0.74 | 0.75 | 0.64 | 0.57 | 0.68
KeyETM | 0.66 | 0.62 | 0.69 | 0.39 | 0.31 | 0.40 | 0.49 | 0.34 | 0.32 | 0.30 | 0.24 | 0.23
SupBERTCLs | 0.68 | 0.53 | 0.62 | 0.77 | 0.70 | 0.80 | 0.94 | 0.94 | 0.94 | 0.38 | 0.37 | 0.38
Zero-ShotCLs | 0.89 | 0.80 | 0.88 | 0.91 | 0.79 | 0.90 | 0.87 | 0.87 | 0.87 | 0.34 | 0.34 | 0.33
WesSTCLs (Names) | 0.55 | 0.47 | 0.57 | 0.72 | 0.65 | 0.76 | 0.80 | 0.79 | 0.79 | 0.44 | 0.44 | 0.48
WesSTCLs (Seeds) | 0.73 | 0.65 | 0.76 | 0.79 | 0.70 | 0.83 | 0.64 | 0.64 | 0.64 | 0.79 | 0.68 | 0.81
LOTCLs | 0.84 | 0.68 | 0.81 | 0.84 | 0.75 | 0.86 | 0.76 | 0.75 | 0.75 | 0.07 | 0.09 | 0.06
ECTM | 0.91 | 0.85 | 0.91 | 0.91 | 0.83 | 0.92 | 0.90 | 0.90 | 0.90 | 0.83 | 0.76 | 0.83

Table 3: Text Classification Performance

###  3.3 Text Classification

Although the primary purpose of our model is not document categorization, we
can use learned $\theta$ to analyze the document representation. More
specifically, we can consider the topic with maximum probability $\theta_{d}$
as the document’s label of $d$. Therefore, we can evaluate how learned topics
are distinctive and informative enough to represent a document to categorize
it correctly. Alongside the topic models, to compare text classification
performance, we also include several supervised and semi-supervised baselines:
SupBERTCLs, Zero-ShotCLs Yin et al. (2019), WeSTCLs Meng et al. (2018), and
LOTCls Meng et al. (2020b), with details described in Appendix C.

Evaluation Metrics.

We use accuracy (Acc.), macro-F1 (F1-Mac) and micro-F1 (F1-Mic) scores that
are commonly used in classification evaluations Meng et al. (2018, 2020b) as
the metrics.

Results and Discussions.

The text classification results on 20Newsg, NYT, Yelp-Senti and Arxiv-AI
datasets are shown in Table 3. We can see that ECTM mostly performs better by
all three evaluation metrics than other models with large margins. One reason
for this better result is the document-level supervision we employ from the
textual entailment model. It enforces the model to generate $\theta$ with
better predictive power. It can also be explained by the comparable
performance with the baselines ZS+LLDA and Zero-ShotCLs models, as they also
use the textual entailment approach. Here, we can see Sup+LLDA and SupBERTCLs
perform slightly better than ours in the Yelp-Sent dataset because it is
rather an easy task to learn a classifier of binary sentiment classes from
supervision. However, it performs very poorly for other more hard
classification datasets.

Moreover, our model performs better even when the textual entailment model
fails to perform well. E.g., in Arxiv-AI dataset, while the ZS+LLDA and Zero-
ShotCLs have less than 0.40 for all three measures, ECTM performs
significantly better by attaining close or more than 0.80 scores in the
evaluation measures. This is because our model does not rely solely on the
textual entailment model. It self-trains itself by updating supervision
iteratively by reducing the chance of error propagation from the entailment
model.

Figure 4: Corpus Comparison

Dec. 2000 | court, election, vote, bush, president, florida, judge, supreme
---|---
Sep. 2001 | attack, war, president, military terrorism, terrorist, afghanistan, muslim
USA | republican, president, governor, senator, clinton, bush, giuliani, senate
Britain | prime, minister, ireland, irish, northern, blair, union, thatcher

Table 4: Context defining words in politics topic

###  3.4 Corpus Comparison

This section explores how ECTM facilitates the quantitative comparison of any
two corpora given the same topics. In ECTM, as described before, while
incorporating topic-level supervision, we generate a proxy topic-word
distribution $\tilde{\beta}^{r}$ of the target $\beta$ to make it comparable
with $\beta^{r}$. As both $\beta^{r}$ and $\tilde{\beta}^{r}$ share the same
vocabulary, any two corpora with the same sets of topics can be thus
quantitatively compared using their generated $\tilde{\beta}^{r}$s. To
investigate the comparison, we apply our ECTM to multiple corpora over
different contexts using the same topics. More specifically, we use two
contexts: time and location. For time context, we use NYT articles from
December 2000 and September 2001 as two target corpora. For location context,
we use NYT articles on USA and Britain as two target corpora. In both cases,
we use the same set of topics, and the entire NYT corpus is used to obtain the
reference.

Methods | Quality | Classification | Top 5 Words of Topic
---|---|---|---
TC |  TD |  TQ |  Acc. |  F1 |  F1 | Business
ECTM - $R_{\beta}$ \- $R_{\theta}$ \- $b$ |  0.24 |  0.99 |  0.24 |  – |  – |  – | time, work, year, problem, people
ECTM - $b$ |  0.28 |  0.98 |  0.27 |  0.90 |  0.83 |  0.90 | year, president, price, month, week
ECTM - $R_{\beta}$ |  0.20 |  0.99 |  0.19 |  0.92 |  0.87 |  0.92 | fan, sale, la, great, pay
ECTM - $R_{\theta}$ |  0.30 |  0.99 |  0.30 |  0.83 |  0.71 |  0.82 | president, price, buy, stock, oil
ECTM |  0.30 |  1.0 |  0.30 |  0.91 |  0.85 |  0.91 |  price, stock, sale, buy, oil

Table 5: Ablation Study

Let $\tilde{\beta}^{r(1)}$ and $\tilde{\beta}^{r(2)}$ are generated by two
corpora $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$ respectively, we compare
$\mathcal{D}_{1}$ and $\mathcal{D}_{2}$ by calculating
$KL(\tilde{\beta}^{r(1)}_{j}||\tilde{\beta}^{r(2)}_{j})$ for each topic
$j\in\\{1,\cdots,k\\}$. Figure 4 illustrates such results for our two
contexts. Here, we see in both contexts, the topic politics has made the
highest difference between the two corresponding corpora. We also show some
context defining words out of top 20-words obtained from ECTM generated
$\beta$ for politics in Table 4 (full list is in Appendix D). As December 2000
was the month when the Bush vs. Gore 2000 presidential race was settled, the
words like election, vote or judge make sense. And words like attack,
terrorism or war are dominated in politics topic on September 2001 as there
was a terror attack that month. Similarly, the words that mainly make the
difference in politics for USA and Britain are self-explanatory.

Figure 5: Effect of (a) Document-level supervision and (b) Topic-level
supervision on topic’s adaptability to the target corpus

###  3.5 Ablation Study

In this section, we investigate the role of different parts of our proposed
model in 20Newsg dataset (shown in Table 5). First, when we exclude the
background bias $b$ from the model, we see the common words (i.e., time, year)
are being dominated in the top-5 topic words, thus making it less
interpretable and distinctive. It explains the effectiveness of background
bias in the model. Second, when only the document-level supervision (ECTM
-$R_{\beta}$) is employed, the model’s predictive power gets improved.
However, the topic quality scores are significantly downgraded, which is also
reflected in irrelevant words (i.e., general words like great, la) in the
top-5 topic words. It justifies the intuition of our topic-level supervision
to make topics interpretable by calibrating them from well-defined topics. On
the other hand, only using topic-level supervision degrades the model’s
predictive power. Finally, the complete model balances them by generating
interpretable and discriminatory topics, which is reflected in the
quantitative measures.

Moreover, in Figure 5, we show the effect of two supervisions on the topic’s
adaptability in the target corpus. The vertical axis shows the divergence of
learned topics with the given reference topics. If the divergence score
increases, it means topics are getting more adapted to the corpus than being
similar to the reference one. Figure 5 (a) shows that if we increase the
regularization weight parameter $\lambda_{\theta}$ for document-level
supervision by fixing $\lambda_{\beta}$, the topics get deviated from
reference topic focusing more on the target corpus content. On the other hand,
while increasing topic-level regularization weight $\lambda_{\beta}$ by fixing
$\lambda_{\theta}$, the model tends to converge towards the reference topics.
It is intuitive and reasonable according to our discussion of model
requirements.

##  4 Conclusion

In this paper, we propose a new problem called coordinated topic modeling that
uses a set of well-defined topics to describe a target corpus. It takes a
reference representation of topics with their surface names and calibrates
those representing the target corpus. The proposed problem describes corpora
in a more interpretable and comparable way. To solve the problem, we design an
embedding-based coordinated topic model that leverages topic-level and
document-level supervision with a self-training mechanism. A set of empirical
evaluations demonstrate the superiority of our approach over other strong
baselines in multiple datasets.

## Acknowledgements

We thank the reviewers for their constructive feedback. This material is based
upon work supported by the National Science Foundation IIS 16-19302 and IIS
16-33755, Zhejiang University ZJU Research 083650, IBM-Illinois Center for
Cognitive Computing Systems Research (C3SR)– a research collaboration as part
of the IBM Cognitive Horizon Network, grants from eBay and Microsoft Azure,
UIUC OVCR CCIL Planning Grant 434S34, UIUC CSBS Small Grant 434C8U, and UIUC
New Frontiers Initiative. Any opinions, findings, and conclusions or
recommendations expressed in this publication are those of the author(s) and
do not necessarily reflect the views of the funding agencies.

## Limitations

The proposed model can be applied only to those domains where we can find a
reference representation of intended topics. However, we aim for the topic’s
interpretability and comparability with other corpora. Thus, in this work, we
are mainly interested in defining a corpus with well-defined topics as input
because they are easily understandable to represent a corpus. Moreover, it is
very reasonable to think that we can usually obtain reference representation
for well-defined topics.

In this paper, the model assumes that the reference and target corpus are from
a common domain. However, it is worth exploring how our model works in cross-
domain problems. On the other hand, exploring the proposed model in cross-
domain scenarios is also an excellent future direction. We can apply our model
in cross-language applications if word embeddings in the vocabulary are
available for multiple corpora from different languages.

## References

  * Atchison and Shen (1980) J Atchison and Sheng M Shen. 1980.  Logistic-normal distributions: Some properties and uses.  _Biometrika_ , 67(2):261–272.
  * Bhatia et al. (2016) Shraey Bhatia, Jey Han Lau, and Timothy Baldwin. 2016.  Automatic labelling of topics with neural embeddings.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 953–963.
  * Blei and Lafferty (2006) David Blei and John Lafferty. 2006.  Correlated topic models.  _Advances in neural information processing systems_ , 18:147.
  * Blei et al. (2017) David M Blei, Alp Kucukelbir, and Jon D McAuliffe. 2017.  Variational inference: A review for statisticians.  _Journal of the American statistical Association_ , 112(518):859–877.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Dieng et al. (2020) Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020.  Topic modeling in embedding spaces.  _Transactions of the Association for Computational Linguistics_ , 8:439–453.
  * Eshima et al. (2020) Shusei Eshima, Kosuke Imai, and Tomoya Sasaki. 2020.  Keyword assisted topic models.  _arXiv preprint arXiv:2004.05964_.
  * Gallagher et al. (2017) Ryan J Gallagher, Kyle Reing, David Kale, and Greg Ver Steeg. 2017.  Anchored correlation explanation: Topic modeling with minimal domain knowledge.  _Transactions of the Association for Computational Linguistics_ , 5:529–542.
  * Griffiths et al. (2003) Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. 2003.  Hierarchical topic models and the nested chinese restaurant process.  _Advances in neural information processing systems_ , 16.
  * Harandizadeh et al. (2022) Bahareh Harandizadeh, J. Hunter Priniski, and Fred Morstatter. 2022.  Keyword assisted embedded topic model.  In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , WSDM ’22, page 372–380, New York, NY, USA. Association for Computing Machinery.
  * Hoang et al. (2019) Tai Hoang, Huy Le, and Tho Quan. 2019.  Towards autoencoding variational inference for aspect-based opinion summary.  _Applied Artificial Intelligence_ , 33(9):796–816.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Jagarlamudi et al. (2012) Jagadeesh Jagarlamudi, Hal Daumé III, and Raghavendra Udupa. 2012.  Incorporating lexical priors into topic models.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 204–213.
  * Jordan et al. (1999) Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. 1999\.  An introduction to variational methods for graphical models.  _Machine learning_ , 37(2):183–233.
  * Lacoste-Julien et al. (2008) Simon Lacoste-Julien, Fei Sha, and Michael Jordan. 2008.  Disclda: Discriminative learning for dimensionality reduction and classification.  _Advances in neural information processing systems_ , 21.
  * Lau et al. (2014) Jey Han Lau, David Newman, and Timothy Baldwin. 2014.  Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 530–539.
  * Lee et al. (2013) Dong-Hyun Lee et al. 2013.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _Workshop on challenges in representation learning, ICML_ , volume 3, page 896.
  * Li and McCallum (2006) Wei Li and Andrew McCallum. 2006.  Pachinko allocation: Dag-structured mixture models of topic correlations.  In _Proceedings of the 23rd international conference on Machine learning_ , pages 577–584.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mcauliffe and Blei (2007) Jon Mcauliffe and David Blei. 2007.  Supervised topic models.  _Advances in neural information processing systems_ , 20.
  * Meng et al. (2020a) Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. 2020a.  Discriminative topic mining via category-name guided text embedding.  In _Proceedings of The Web Conference 2020_ , pages 2121–2132.
  * Meng et al. (2018) Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018.  Weakly-supervised neural text classification.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , pages 983–992. ACM.
  * Meng et al. (2020b) Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020b.  Text classification using label names only: A language model self-training approach.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9006–9017.
  * Ramage et al. (2009) Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D Manning. 2009.  Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora.  In _Proceedings of the 2009 conference on empirical methods in natural language processing_ , pages 248–256.
  * Ramage et al. (2011) Daniel Ramage, Christopher D Manning, and Susan Dumais. 2011.  Partially labeled topic models for interpretable text mining.  In _Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 457–465.
  * Sandhaus (2008) Evan Sandhaus. 2008.  The new york times annotated corpus.  _Linguistic Data Consortium, Philadelphia_ , 6(12):e26752.
  * Sinha et al. (2015) Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. 2015.  An overview of microsoft academic service (mas) and applications.  In _Proceedings of the 24th international conference on world wide web_ , pages 243–246.
  * Srivastava and Sutton (2017) Akash Srivastava and Charles Sutton. 2017.  Autoencoding variational inference for topic models.  In _5th International Conference on Learning Representations_.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_ , pages 1480–1489.
  * Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.  Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3914–3923.

Dec. 2000 |  | court, election, vote, bush, president, florida, law, judge, justice, government,
---
supreme, party, clinton, official, case, presidential, decision, ballot,
republican, democratic
Sep. 2001 |  | united, attack, war, president, government, official, american, bush, terrorist, military,
---
country, terrorism, national, afghanistan, vote, election, administration,
leader, political, muslim
USA |  | republican, vote, campaign, election, president, party, governor, senator, candidate, political,
---
mayor, democratic, clinton, bush, voter, democrat, democrats, giuliani,
senate, race
Britain |  | minister, president, ireland, party, government, political, official, prime, irish,
---
northern, clinton, leader, peace, blair, union, thatcher, republican, war,
election, sinn

Table 6: Top 20 words in politics topic on different contexts

##  Appendix A Related Work

It is of great interest to automatically mine a set of meaningful and coherent
topics for effectively and efficiently understanding and navigating a large
text corpus. Topic models Jordan et al. (1999); Blei et al. (2003) are such
statistical tools that can discover latent semantic themes from a text
collection. The main idea is to represent each document in a corpus as a
mixture of hidden topics, each represented by word distribution. While most of
the early attempts of topic models Griffiths et al. (2003); Blei and Lafferty
(2006); Li and McCallum (2006) are probabilistic, with the advent of the deep
neural network, neural topic models are also proposed. One such example is the
Autoencoded Variational Inference For Topic Model (AVITM) Srivastava and
Sutton (2017). Recently, Embedded Topic Model (ETM) Dieng et al. (2020) blends
the strengths of the neural topic model and word embedding.

Despite their effectiveness and efficiency, traditional topic models have
several limitations. The standard topic models whether it is probabilistic
Blei et al. (2003) or neural Srivastava and Sutton (2017); Dieng et al. (2020)
has the inability to incorporate user guidance. Existing topic models are
typically learned in a purely unsupervised manner and, thus, tend to discover
the most general and major topics ignoring user interests. There have been
several modifications in the traditional topic models to incorporate user
interests or existing knowledge about documents in the corpus. More
specifically, there are generally three forms of supervision in existing
supervised topic models.

The first type of supervision is in the form of labeled corpus where all or a
subset of documents are annotated with some predefined topic, or class labels
Ramage et al. (2009, 2011); Mcauliffe and Blei (2007); Lacoste-Julien et al.
(2008). E.g., Labeled LDA Ramage et al. (2009) constrains Latent Dirichlet
Allocation (LDA) Blei et al. (2003) model through one-to-one correspondence
between LDA’s latent topics and document labels. While they improve the
predictive ability of unsupervised topic models, they require a massive set of
annotated documents to be effective. The second type of supervision comes in
the form of seed words where for each user-interested topic, a set of
represented keywords are provided to guide the topic generation process
Jagarlamudi et al. (2012); Eshima et al. (2020); Harandizadeh et al. (2022);
Gallagher et al. (2017). More specifically, the seed guided topic models make
the topics converge in the direction of given seed words. Although they make
the topics converge in the direction of user-desired seed words, they require
the seed words to appear in the corpus vocabulary, which may be impractical in
many cases. Finally, there is an approach called category-guided topic mining
Meng et al. (2020a) (CatE), which considers the topics’ surface names as the
only supervision for mining user-interested discriminative topics. However, it
also assumes that the surface name of each topic appears in the corpus.
Moreover, none of the above work does impose requirements of making topics
comparable over multiple corpora. Moreover, because of strict discrimination
assumptions in CatE, they often suffer from mining too specific words to
represent topics that are very hard to interpret.

##  Appendix B Implementation Details

There are some parameters our model we have to set. We use the 300-dimensional
pre-trained word embedding from Spacy Honnibal and Montani (2017). The
dimension of hidden layers in MLP is set to 300. The learning rate is set to
0.005. We use different epochs based on different datasets. We mainly use 150
for news domain datasets and 100 for other cases. The regularization
parameters $\lambda_{\beta}$ and $\lambda_{\theta}$ are set to 20 and 35,
respectively, for all datasets. We follow the standard procedure for
preprocessing text before applying all the models. For example, we remove
stopwords; the words appear more than 0.70 of the whole corpus, and infrequent
words appear less than ten times. For baselines, we used their default
settings.

##  Appendix C Additional Text Classification Baselines

  * •

SupBERTCLs: It trains a BERT-based supervised text classifier
666https://huggingface.co/docs/transformers/tasks/sequence_classification on
labeled reference corpus and uses that to predict labels for documents in
target corpus.

  * •

Zero-ShotCLs: It predicts the topic of each document out of given surface
names $\mathcal{C}$ using a Zero-Shot classification Yin et al. (2019) method
based on a pre-trained textual entailment
model777https://huggingface.co/joeddav/bart-large-mnli-yahoo-answers

  * •

WeSTCLs: This is a weakly-supervised neural text classification method Meng et
al. (2018) that can classify a text document based on given class names or
seed words. We use the two variants: (1) WeSTCLs (Names) uses the topic names
$\mathcal{C}$; (2) WeSTCLs (Seeds) uses seed words.

  * •

LOTCls: This is a language model-based text classification method Meng et al.
(2020b) that uses label names as supervision. As supervision, we provide the
given topic surface names $\mathcal{C}$.

##  Appendix D Additional Result on Corpus Comparison

Table 6 shows qualitative results of corpus comparison. It shows the top 20
words for each corpus generated by our ECTM model, each representing a context
for the topic of politics.


# Structural Bioinformatics

K. Anton Feenstra  Sanne Abeln
Centre for Integrative Bioinformatics (IBIVU), and
Department of Computer Science,
Vrije Universiteit, De Boelelaan 1081A, 1081 HV Amsterdam, Netherlands

### Abstract

This chapter gives a graceful introduction to problem of protein three-
dimensional structure prediction, and focuses on how to make structural sense
out of a single input sequence with unknown structure, the ‘query’ or ‘target’
sequence. We give an overview of the different classes of modelling
techniques, notably template-based and template free. We also discuss the way
in which structural predictions are validated within the global community, and
elaborate on the extent to which predicted structures may be trusted and used
in practice. Finally we discuss whether the concept of a single fold
pertaining to a protein structure is sustainable given recent insights. In
short, we conclude that the general protein three-dimensional structure
prediction problem remains unsolved, especially if we desire quantitative
predictions. However, if a homologous structural template is available in the
PDB model or reasonable to high accuracy may be generated.

###### Contents

  1. 7 Introduction to Protein Structure Prediction
    1. 7.1 What is the protein structure prediction problem?
      1. 7.1.1 Predicting the structure for a protein sequence
      2. 7.1.2 Structure is more conserved than sequence
      3. 7.1.3 Terminology in structure prediction
      4. 7.1.4 Different classes of structure prediction methods
      5. 7.1.5 Domains
    2. 7.2 Assessing the quality of structure prediction methods
      1. 7.2.1 Critical Assessment of protein Structure Prediction
      2. 7.2.2 Root-Mean-Square Deviation (RMSD):
      3. 7.2.3 GDT – Global Distance Test
      4. 7.2.4 How difficult is it to predict?
      5. 7.2.5 For which gene sequences can we predict a three-dimensional structure?
      6. 7.2.6 How accurate do we need to be?
    3. 7.3 Is there such a concept as a single native fold?
      1. 7.3.1 Disordered proteins
      2. 7.3.2 Allostery and functional structural ensembles
      3. 7.3.3 Amyloid fibrils
    4. 7.4 Acknowledgements

Sanne Abeln Jaap Heringa K. Anton Feenstra
Centre for Integrative Bioinformatics (IBIVU) and
Department of Computer Science,
Vrije Universiteit, De Boelelaan 1081A, 1081 HV Amsterdam, Netherlands

##  Chapter 7 Introduction to Protein Structure Prediction

###  7.1 What is the protein structure prediction problem?

####  7.1.1 Predicting the structure for a protein sequence

This chapter revolves around a simple question: “given an amino acid sequence,
what is the folded structure of the protein?” (Figure 7.1) Even though this
seems like a simple question, the answer is far from straightforward. In fact,
whether we can give an answer at all depends heavily on the sequence in
question and available protein structures that can be used as modelling
templates. While the number of structures deposited in the Protein Data Bank
(PDB) Berman et al.,  (2000) continues to rise rapidly
111https://www.rcsb.org/pdb/statistics/contentGrowthChart.do?content=total&seqid=100,
the number of sequenced genes rises much faster. The large and widening gap
between protein structures and sequences makes structure prediction an
important problem to solve. Fortunately, recently developed methods can use
these large resources of sequence data to increase the quality of some
predictions. Here, we will give an overview of current structure prediction
methods, and describe some tools that provide insight into how reliable the
structure predicted will be.

Figure 7.1: Structure prediction methods try to answer the question: given an
amino acid sequence, what is the folded protein structure?

The typical problem is that we want to generate a structural model for a
protein with a sequence, but without an experimentally determined structure.
In this chapter, we will build up a workflow for tackling this problem,
starting from the easy options that, if applicable, are likely to generate a
good structural model, and gradually working up to the more hypothetical
options whose results are much more uncertain.

Another very important remark is in place here: the modelling strategy should
depend heavily on what we want to do with the structure. Do we want to predict
where the functional site of the protein is, whether a specific substrate
binds, or if a certain residue may be exposed to the surface? These different
questions imply a different degree of accuracy in the answer, and may lead to
choices regarding technology and methods to carry out these predictions. It is
important to keep in mind that one of the most important aspects of any
scientific model is whether a research question may be answered with the model
produced or not. Even if we do have an experimental structure available, some
of these questions may not be straightforward to answer; we will come back to
this issue later in the chapter.

Figure 7.2: Protein structure more conserved than sequence. Here the output of
a structural alignment is shown on the left, created using Chimera
333Molecular graphics and analyses were performed with the UCSF Chimera
package. Chimera is developed by the Resource for Biocomputing, Visualization,
and Informatics at the University of California, San Francisco (supported by
NIGMS P41-GM103311). Pettersen et al.,  (2004). The structural alignment shows
both proteins are highly similar; the RMSD is 2.3 Å over 144 aligned residues.
Furthermore, the function of the two proteins, one from cattle (PDB:1L9H,
light brown) and one from a archaeon (PDB:1GUE, light blue), is similar: both
are light sensitive rhodopsins, used for vision and phototaxis, respectively.
However, as can be seen in the sequence alignment on the right, the sequence
identity is only 7%. This is lower than would be expected for any two random
sequences. The alignment shown is based on the structural alignment on the
left, and visualised using JalView Waterhouse et al.,  (2009).

####  7.1.2 Structure is more conserved than sequence

Almost all structure prediction relies on the fact that, for two homologous
proteins, structure is more conserved than sequence (see Figure 3). The real
power of this observation manifests itself when we turn this statement around:
if two protein sequences are similar, these two proteins are likely to have a
very similar structure. The latter statement has very important consequences.
It means that if our sequence of interest is similar to a protein sequence
with a known structure, we have a good starting point for a structural model.
In such a scenario we use sequence similarity, suggesting an homologous
relation between the proteins, to predict the structure. The vast majority of
accurate structure prediction methods use structure conservation as an
underlying principle; while methods that have been developed to deal with the
more difficult modelling questions, exploit the sequence-structure-
conservation relation in an advanced manner, as discussed towards the end of
this chapter.

Figure 7.3: Terminology used in protein structure prediction. We start from
our protein of interest (with no known structure): the target sequence. First
step is find a matching protein: a template sequence with known structure; the
template structure. We then create a template-target sequence alignment, and
from this alignment create the structural model which is the solution
structure for our target protein.

####  7.1.3 Terminology in structure prediction

Firstly, we should take care to lay down a good problem definition. Here we
will generously borrow the nomenclature from the Critical Assessment of
Protein Structures (CASP). CASP is a scientific competition, in which
structure prediction groups and structure prediction servers compete to
predict the structure for an unknown sequence, that has been running since
1995 Moult et al.,  (1995). The sequence for which we will predict a structure
is called the _target_ sequence. If there is a suitable structure to build a
model for our query sequence we call this structure a _template_ , see also
Figure 7.3. Using the structure of the template and using the _sequence
alignment_ between the template and the target sequence, we can create one or
more structural _models_ : the predicted structure, for a target sequence. In
CASP structural models from different prediction methods, are compared to the
experimentally determined solution or target structure.

Figure 7.4: Overview of Structure Prediction. Template-based modelling: a
template is found on the basis of homology between the template and the
target. Fold recognition: no obvious homologous structure can be found in the
PDB, we need fold recognition methods to find a suitable template. Template-
free modelling: no suitable template for protein domains can be found. Without
template, we need to use a combination of coarse constraints from experiment
or co-evolution analysis, and ab initio prediction. Ab initio methods
typically work with taking fragment templates from various proteins, and
assemble these into a model or decoy structure. Expected model accuracy
declines from left to right: good accuracy is expected if based on homology;
in contrast, ab initio modelling should only be considered if no other options
remain.

####  7.1.4 Different classes of structure prediction methods

We can classify structure prediction strategies into two categories of
difficulty: template-based modelling, and template-free modelling (see Figure
7.4). In the first case, it is possible to find a suitable template for the
target sequence in the PDB, as a basis for the model, whereas for template-
free modelling no such experimental structure is available. Note that it may
not be trivial at all to find out in which of these two categories a structure
prediction problem falls. Only if we can find a close homolog – based on
sequence similarity – in the PDB we can be sure that a template based
modelling strategy will suffice; this is also referred to as homology
modelling. With a template, the constraints from the alignment between the
model and the template sequence, in addition to the template structure, will
give sufficient constraints to build a structural model for the target
sequence. Even in this case, small missing substructures in the alignment,
e.g. loops, may require a template-free modelling strategy.

If no close homologs are available in the PDB, we may need to use more
advanced template finding strategies, such as remote homology detection or
fold recognition methods.

If no suitable template is available, we will need to resort to a template-
free modelling strategy. In the “ab initio” approach knowledge-based energy
terms are used to generate structural models based on the sequence of the
template alone. Small, suitable fragments, from various PDB structures are
assembled to generate possible structural models. In some cases, we can find
additional constraints, for example from experiments, such as NMR or cryoEM,
or from contact prediction methods; in that case we have a much better chance
of building a suitable model Moult et al.,  (2016). In fact, we could consider
such constraints an alternative for the constraints provided by homology.

Lastly, several steps may be taken to refine the model, and to select the most
likely model, from several model building attempts. Note that some structure
prediction methods, may also include variations of model refinement and model
selection steps higher up in the modelling workflow.

####  7.1.5 Domains

So far we have implied that we may follow the above strategy for an entire
protein, however, this generally is not the case. In fact many proteins
consist of multiple domains. If this is the case, it is wise to also run one
or multiple disorder prediction methods on the target sequence; Any large
regions ($>25$ residues) predicted to be disordered should be left out for
further structure prediction and template finding.

Most structure prediction methods only work well at the domain level. This
means that a sequence first needs to be split in multiple domains, before we
can start to make models. However, domain splitting is often ambiguous both
given the sequence and the structure, while combining models built from
various domains is far from trivial. In practice, this means that multiple
templates might be necessary for a single target sequence and that it is
difficult to resolve the orientation of the modelled domains with respect to
each other.

Predicting the orientation for several domains is currently an unsolved
problem, unless there is a suitable, homologous, template available – with the
domains in the same orientation. In some cases, coarse constraints on the
domain orientations such as data from small-angle scattering experiments, or
distance restraints from NMR, chemical cross-links or co-evolution may help to
put different homology models in the correct orientation.

Typically, it only makes sense to generate a model, be it template-based or
template-free, for a single domain. In fact, in CASP model predictions are
assessed per structural domain, separately. Therefore, it is essential to
split the target sequence into its constituent domains – which is a non-
trivial task, particularly if no homologous templates are available for each
of the domains.

###  7.2 Assessing the quality of structure prediction methods

Generally, as with any prediction problem, we can assess the quality of a
prediction if we have a true answer to the question. Here, truth will be
represented by an experimentally determined protein structure (of high
quality). Fortunately, there are now (November 2017) over 120,000 deposited
protein structures in the PDB444structures in the PDB:
https://www.rcsb.org/pdb/statistics/holdings.do. However, simply assessing how
well a method performs over this set is problematic. The methods have been
trained on this data set; that means that there may be a strong bias in these
methods, to predict good models for sequences that are within their dataset,
and therefore homologs of those. In order to truly assess a method, a
completely independent data set is required.

####  7.2.1 Critical Assessment of protein Structure Prediction

Every other year CASP, a Critical Assessment of Protein Structure Prediction,
provides such an independent validation benchmark. CASP is a blind test or
competition: experimentalists provide sequences for which they know the
structures will be solved imminently; modelling groups and servers try to
predict the structure Moult et al.,  (1995). Once the structure is solved, the
models can be evaluated using the solution structure of the target (see also
Figure 7.5).

CASP was started because the protein structure prediction problem was claimed
to have been solved several times. The problem was, that algorithms were
trained on databases that contained the structures that were evaluated in
benchmarking tests. CASP overcomes this problem.

Note that the very first step in any practical structure prediction approach,
should be to inspect the results from the latest CASP round Moult et al.,
(2016) via the CASP website555CASP website: http://predictioncenter.org/ to
see what the state of the art methods are, and what their expected performance
is.

####  7.2.2 Root-Mean-Square Deviation (RMSD):

If we want to asses the quality of a method, we need to measure the quality of
the predictions made by the method. Hence, one would like to structurally
compare atomic coordinates of the model and of the solution structure, and
quantify the (dis)similarity.

The problem of comparing a model to to a solution structure, is less difficult
than the comparison between two homologous protein structures. This is because
the alignment is trivial: the model has the same sequence as the solution
structure; we know which residues, and atoms should correspond in the two
structures.

The easiest way to compare structures, is the calculate the Root-Mean-Square
Deviation (RMSD) after a structural superpositioning Marti-Renom et al.,
(2009). The superpositioning is required, because two arbitrary structures
will typically not be positioned at coordinates suitable for comparison; first
a translation and rotation needs to be applied to one of the two structures,
to minimise the RMSD; the resulting RMSD after superpositioning can be used as
a dissimilarity measure.

The Root-Mean-Square Deviation (RMSD) calculates the squared difference
between two sets of atoms, and can be defined as follows:

$\displaystyle RMSD(v,w)$
$\displaystyle=\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left\|v_{i}-w_{i}\right\|^{2}}$
$\displaystyle=\sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(v_{ix}-w_{ix}\right)^{2}+\left(v_{iz}-w_{iz}\right)^{2}+\left(v_{iz}-w_{iz}\right)^{2}}$

Here, $v_{i}$ is the position vector of $i^{th}$ atom of structure $v$;
$w_{i}$ is the position vector of $i^{th}$ atom of structure $w$; and $n$ is
the total number of aligned atoms.

The RMSD takes the average over all aligned pairs. In protein structures
typically one representative atom per residue is chosen, such as C$\alpha$ or
C$\beta$.

Figure 7.5: Example of structural comparison for the target T0886-D2 and two
models submitted to CASP12. The top panel shows individual traces for all
models generated for this target; the distance cutoff (vertical axis, in Å) is
plotted against the fraction of residues (horizontal axis, in %) that can be
aligned within this cutoff. The traces were obtained from
predictioncenter.org/casp12. The dotted lines indicate the thresholds used in
the GDT_TS (1, 2, 4, 8 Å) and GDT_HA (0.5, 1, 2, 4 Å) scores. Two models are
highlighted in blue: a bad model (TS236, GDT_TS=18.90) on the left, and a good
model (TS173; GDT_TS=51.97) on the right. Both model structures are also shown
in the panels below in red, superposed onto the solution crystal structure in
blue (PDB:5FHY). Structural superposition created using LGA at
proteinmodel.org/AS2TS/LGA/ Zemla,  (2003), 3D visualisation using Chimera
1.11.2 Pettersen et al.,  (2004).

####  7.2.3 GDT – Global Distance Test

If a model gets a loop very wrong, it tends to stick out and can be positioned
very distant from the true structure, even though the remaining structure may
be reasonably accurate. This partial outlier weighs heavily on the average
distance calculated. Hence the RMSD is over sensitive to such outliers.

The global distance test total score (GDT_TS) is a more robust structural
similarity measure that is well defined given an alignment between two
structures. The key idea is to count the number of residues that can maximally
be fitted within a certain distance cutoff, see also Figure 7.5. The GDT score
will therefore produce a percentage. In the formula below, the final score is
the average over four different distance cutoffs (1, 2, 4, 8 Å).

$GDT\\_TS=\frac{1}{4}\sum_{v=1,2,4,8\text{\AA}}\frac{G(v)}{t}$ (7.1)

Here, $G(v)$ is the number of aligned residues within given RMSD cutoff $v$
(in Ångstrom – $10^{-10}m$) and $t$ is the total number of aligned residues. A
related score called GDT_HA was introduced in CASP some time ago Read and
Chavali,  (2007) using stricter distance cutoffs (0.5, 1, 2, 4 Å), to cater
for targets in the template-based modelling category where very high
accuracies can be realized:

For a typical “difficult” CASP target no model even comes close to the
experimentally solved structure; typical results would be similar to the left-
most model in Figure 7.5. If we have a look at the latest CASP results one
will see that a performance of GDT_TS$<20\%$ is not an exception. In other
words, the protein structure prediction problem has NOT yet been solved,
especially not if one considers targets without a good template structure.

Figure 7.6: Distribution of GDT_TS scores for the different model categories
in CASP11 for template-based Modi et al.,  (2016), template-free with contact
information Kinch et al., 2016a and template-free Kinch et al., 2016b . The
legend coloring corresponds to the GDT_TS scores, the bars indicate the
fraction of models in each GDT_TS range for the six categories (GDT_TS scores
for Modi et al.,  (2016) were estimated from the reported GDT_HA scores using
their Figure 4A). “Outliers” targets have unusually high GDT_TS due to being
very short ($\sim 50$ residue) with extended structures. Targets selected for
server prediction (top bar) were considered easier than those for human
prediction (second from top), average sequence identity was 26% vs. 20%,
respectively. It is clear that overall prediction accuracy sharply declines
going down this list of categories. For template-free modelling, the quality
of contact information used is crucial. Experimental information (from
chemical cross linking or simulated NMR) can give reasonable models. Predicted
contacts do not guarantee that an acceptable model can be obtained, but
without even predicted contacts, more than two-thirds of models are at most
20% correct.

####  7.2.4 How difficult is it to predict?

Overall, if one can find a good template, the quality of the predicted model
will be relatively good. CASP results show that for homology modelling based
on close homologues, it is possible to obtain models similar to the
experimentally determined structure Moult et al.,  (2016). The modelled
structure will typically have a good accuracy for the regions that can be well
aligned between the target and template (using the sequences). The top two
bars in Figure 7.6 shows that one may expect the majority of such models to be
accurate for $>50\%$ of their residues. Gaps in an alignment will typically
lie in loop regions of a structure and are more difficult to model. So, if we
are interested in a large loop region that is not present in our template, we
still may not be able to answer our scientific question with the resulting
model structure Moult et al.,  (2016).

If no acceptable template can be found, the chances of successfully answering
our scientific question will become very low. As a last resort, ab initio
modelling can provide us with structural models. Typically, ab initio methods
use very small templates from various proteins (see Figure 7.4). The state of
the art is that on average one may expect to find one structure that looks
somewhat like the solution structure for the target among the top five or ten
models Moult et al.,  (2016). However, be aware that the best model is
typically not recognised as being the best through the scores of the
prediction program. In Figure 7.6 one sees that very clearly in the bottom few
bars: without template, even with predicted contacts, one may have less than
20% of the structure correct in the majority of models; even in the best cases
at most 40% of the residues are modelled accurately.

####  7.2.5 For which gene sequences can we predict a three-dimensional
structure?

If and only if there is a structure of a homologous protein present in the
PDB, it is possible to generate a structural model of reasonable accuracy.
Based on this notion, we can estimate for which (fraction of) gene sequences
it is possible to predict a structure. This way it has been estimated that for
a about 44% of residues in Eukaryotic gene sequences, we cannot yet make a
homology model, and 15% of these residues lie within a gene for which we can
not make a homology model for a single domain Perdigão et al.,  (2015).
Especially membrane proteins are underrepresented in the PDB, due to the
experimental difficulty of determining these structures. Note that these
residues, may also lie in natively disordered regions (see also Section 7.3).

Similarly, it is possible to predict the range of protein structures present
in an organism, based on the gene sequences their completed genome. This
reveals that there is a subset of protein structures, that is present in
nearly all organisms, for example TIM-barrels or Rossmann-folds Abeln and
Deane,  (2005); Edwards et al.,  (2013). Nevertheless, there is also a group
of structures that is extremely lineage specific. It is to be expected that
for this type of protein structures, many new structures remain to be
discovered. This also implies that it will remain difficult to find suitable
templates for homology modelling for these lineage specific protein families.

####  7.2.6 How accurate do we need to be?

We already mentioned that we may approach the modelling of a protein structure
of interest differently, depending on the biological question we want to
ask,e.g. which residues are likely to be crucial for the functioning of the
protein. Sometimes an answer to the research question may be possible in a
simpler way, without full-scale prediction of the protein structure, e.g. by
direct prediction of the impact of certain mutations or of protein-protein
interaction sites. Examples of fully-automated webservers that do just that,
are HOPE – Venselaar et al.,  (2010) and SeRenDIP Hou et al.,  (2017). In some
cases, a rough homology model inspires the understanding of experimental
results, spurring forward the project and eventually ending with crystal
structures highlighting the protein function (in this case, protein-protein
interactions) of interest (e.g., De Vries-van Leeuwen et al., , 2013). Also,
specifically for enzymes, such as for example cytochromes P450, modelling of
the protein structure should be done combination with that of the ligand de
Graaf et al.,  (2005).

In CASP11, three functional aspects were explictly scored, selected on being
able to qualitatively evaluate them: multimeric state, (small) ligand binding,
and mutation impact. Targets were selected that in solved crystal structure
were dimeric, or had a ligand bound, or where from the crystallographers or in
literature interest was expressed for evaluating mutants Huwe et al.,  (2016).

For prediction of dimer structures, only in two cases out of ten a dimer model
with reasonable accuracy could be generated for the majority of monomer model
structures Huwe et al.,  (2016). In the critical assessment of prediction of
protein interaction (CAPRI) between 30-80% of models were of ‘acceptable’ or
‘medium’ quality for easy dimer targets, while for harder targets (difficult
dimers, multimers and heteromers), this fraction dropped to below 10% Lensink
et al.,  (2016). Encouragingly, it was seen that also structure models of
lower quality could sometimes lead to acceptable or even medium quality models
of the bound proteins Lensink et al.,  (2016).

For ligand binding, it was found that the accuracy of even the best models
($\sim 2$Å) are not good enough for accurate ligand docking; the best ligands
were around $5$Å RMSD Huwe et al.,  (2016). Something similar was found for
mutation impact prediction; for most targets, model accuracy did not correlate
with accuracy of impact prediction Huwe et al.,  (2016). Apparently, either
homology models are not yet accurate enough for this purpose, or methods are
tuned to particular characteristics of crystal structures.

###  7.3 Is there such a concept as a single native fold?

Before we conclude, we should consider a more physical description of protein
structure. In fact, protein folding from a physical point of view is a very
interesting process: given a sequence, a protein tends to fold always, and
exactly into the same functional structure. In material design, it is
extremely difficult to mimic such high specificity. The apparent observation
of folding specificity also leads to the question, is there such a concept as
a single native fold? Or, more pragmatically, is sequence-to-structure truly a
one-to-one relation?

In fact, if one wants to start making quantitative predictions, such as the
stability of a protein fold, or the binding strength between two proteins in
terms of free energy, it is much more helpful to think in ensembles of
structural configurations for a protein sequence (e.g. May et al., , 2014;
Pucci et al., , 2017). The probability to find a protein in a specific
ensemble of structural configurations will depend on conditions such as the
presence or absence of binding partners, the pressure, the pH or the
temperature (e.g. van Dijk et al., , 2015, 2016). There are a few specific
cases, common cases, for which even the functional or biologically relevant
structural ensembles do not resemble a single globular folded structure.

####  7.3.1 Disordered proteins

Not all proteins fold into single configurations, some proteins stay natively
unfolded, i.e. they can take up a a large variety of more extended, and very
different configurations Uversky et al.,  (2000); Mészáros et al.,  (2007).
Some disordered regions contain elements that do form stable structures upon
binding. The regions that remain disordered are thought to be important to
prevent aggregation within the cell Abeln and Frenkel,  (2008). Missing
residues in X-ray structures are typically removed for crystallization; for
this reason disorder prediction methods have been developed. Disordered
regions are relatively easy to predict in protein sequences just like
secondary structures; broadly speaking, prediction can be based on the large
amount of charged/polar (hydrophilic) amino acids in combination with the
presence of amino acids that disrupt the secondary structure (proline and
glycine) in these regions Oldfield et al.,  (2005); Wang et al.,  (2016). We
know sequences of many proteins contain large disordered segments (33% of
eukaryotic, 2% archaeal, and 4% bacterial proteins).

####  7.3.2 Allostery and functional structural ensembles

It is important to realize that one protein, typically, does not correspond to
one defined three-dimensional structure. Disordered regions or proteins are
one particularly salient case, but also proteins which fold into specific
three-dimensional configurations, may exist in multiple functional states each
with a specific structure. The biological question of interest dictates which
state is relevant. Most proteins have only been crystallized in one particular
state, and often it is not known to which biological condition this crystal
structure may correspond. One may have cases where a homology model of the
relevant state may be preferred over a crystal structure of a different or
unknown state (e.g., de Graaf et al., , 2005).

####  7.3.3 Amyloid fibrils

Lastly, we should consider a competing state of folded proteins: the
aggregated state, where multiple peptide chains clog together in fibrillar
structures or amorphous aggregates. Amyloid fibres are formed by
$\beta$-strands formed between different protein or peptide (small protein)
chains. Fibril formation is associated with various neurodegenerative
diseases, such as Alzheimer’s, Creutzfeldt-Jakob and Parkinson’s Chiti and
Dobson,  (2006). In fact, the fibrillar state is more favorable than the state
of separately folded structures for several protein types. The general
cellular toxicity of such aggregates, puts evolutionary pressure on avoiding
structural characteristics on the surface of proteins; hence it is extremely
rare to observe solvent accessible $\beta$-strand edges or large hydrophobic
surface patches (Richardson and Richardson, , 2002; Abeln and Frenkel, ,
2011). The propensity proteins have to form Amyloid fibrils is relatively easy
to predict Graña-Montes et al.,  (2017). However, reference databases are
still small so it is difficult to verify such methods.

###  7.4 Acknowledgements

We thank Nicola Bonzanni, Kamil K. Belau, Ashley Gallagher, Jochem Bijlard for
insightful discussions and critical proofreading of early versions.

## References

  * Abeln and Deane, (2005) Abeln, S. and Deane, C. M. (2005).  Fold usage on genomes and protein fold evolution.  Proteins, 60(4):690–700.
  * Abeln and Frenkel, (2008) Abeln, S. and Frenkel, D. (2008).  Disordered flanks prevent peptide aggregation.  PLoS Comput. Biol., 4(12):e1000241.
  * Abeln and Frenkel, (2011) Abeln, S. and Frenkel, D. (2011).  Accounting for protein-solvent contacts facilitates design of nonaggregating lattice proteins.  Biophys. J., 100(3):693–700.
  * Berman et al., (2000) Berman, H. M., Westbrook, J., Feng, Z., Gilliland, G., Bhat, T. N., Weissig, H., Shindyalov, I. N., and Bourne, P. E. (2000).  The Protein Data Bank.  Nucleic Acids Res, 28(1):235–242.
  * Chiti and Dobson, (2006) Chiti, F. and Dobson, C. M. (2006).  Protein misfolding, functional amyloid, and human disease.  Annu. Rev. Biochem., 75:333–366.
  * de Graaf et al., (2005) de Graaf, C., Vermeulen, N. P. E., and Feenstra, K. A. (2005).  Cytochrome P450 in Silico: An Integrative Modeling Approach.  Journal of Medicinal Chemistry, 48(8):2725–2755.
  * De Vries-van Leeuwen et al., (2013) De Vries-van Leeuwen, I. J., da Costa Pereira, D., Flach, K. D., Piersma, S. R., Haase, C., Bier, D., Yalcin, Z., Michalides, R., Feenstra, K. A., Jiménez, C. R., de Greef, T. F. A., Brunsveld, L., Ottmann, C., Zwart, W., and de Boer, A. H. (2013).  Interaction of 14-3-3 proteins with the estrogen receptor alpha F domain provides a drug target interface.  Proceedings of the National Academy of Sciences of the United States of America, 110(22):8894–9.
  * Edwards et al., (2013) Edwards, H., Abeln, S., and Deane, C. M. (2013).  Exploring Fold Space Preferences of New-born and Ancient Protein Superfamilies.  PLoS computational biology, 9(11):e1003325.
  * Graña-Montes et al., (2017) Graña-Montes, R., Pujols-Pujol, J., Gómez-Picanyol, C., and Ventura, S. (2017).  Prediction of Protein Aggregation and Amyloid Formation.  In From Protein Structure to Function with Bioinformatics, pages 205–263. Springer Netherlands, Dordrecht.
  * Hou et al., (2017) Hou, Q., De Geest, P., Vranken, W., Heringa, J., and Feenstra, K. (2017).  Seeing the trees through the forest: Sequencebased homo- and heteromeric protein-protein interaction sites prediction using random forest.  Bioinformatics, 33(10).
  * Huwe et al., (2016) Huwe, P. J., Xu, Q., Shapovalov, M. V., Modi, V., Andrake, M. D., and Dunbrack, R. L. (2016).  Biological function derived from predicted structures in CASP11.  Proteins: Structure, Function, and Bioinformatics, 84(S1):370–391.
  * (12) Kinch, L. N., Li, W., Monastyrskyy, B., Kryshtafovych, A., and Grishin, N. V. (2016a).  Assessment of CASP11 contact-assisted predictions.  Proteins: Structure, Function, and Bioinformatics, 84(S1):164–180.
  * (13) Kinch, L. N., Li, W., Monastyrskyy, B., Kryshtafovych, A., and Grishin, N. V. (2016b).  Evaluation of free modeling targets in CASP11 and ROLL.  Proteins: Structure, Function, and Bioinformatics, 84(S1):51–66.
  * Lensink et al., (2016) Lensink, M. F., Velankar, S., Kryshtafovych, A., Huang, S.-Y., Schneidman-Duhovny, D., Sali, A., Segura, J., Fernandez-Fuentes, N., Viswanath, S., Elber, R., Grudinin, S., Popov, P., Neveu, E., Lee, H., Baek, M., Park, S., Heo, L., Rie Lee, G., Seok, C., Qin, S., Zhou, H.-X., Ritchie, D. W., Maigret, B., Devignes, M.-D., Ghoorah, A., Torchala, M., Chaleil, R. A., Bates, P. A., Ben-Zeev, E., Eisenstein, M., Negi, S. S., Weng, Z., Vreven, T., Pierce, B. G., Borrman, T. M., Yu, J., Ochsenbein, F., Guerois, R., Vangone, A., Rodrigues, J. P., van Zundert, G., Nellen, M., Xue, L., Karaca, E., Melquiond, A. S., Visscher, K., Kastritis, P. L., Bonvin, A. M., Xu, X., Qiu, L., Yan, C., Li, J., Ma, Z., Cheng, J., Zou, X., Shen, Y., Peterson, L. X., Kim, H.-R., Roy, A., Han, X., Esquivel-Rodriguez, J., Kihara, D., Yu, X., Bruce, N. J., Fuller, J. C., Wade, R. C., Anishchenko, I., Kundrotas, P. J., Vakser, I. A., Imai, K., Yamada, K., Oda, T., Nakamura, T., Tomii, K., Pallara, C., Romero-Durana, M., Jiménez-García, B., Moal, I. H., Férnandez-Recio, J., Joung, J. Y., Kim, J. Y., Joo, K., Lee, J., Kozakov, D., Vajda, S., Mottarella, S., Hall, D. R., Beglov, D., Mamonov, A., Xia, B., Bohnuud, T., Del Carpio, C. A., Ichiishi, E., Marze, N., Kuroda, D., Roy Burman, S. S., Gray, J. J., Chermak, E., Cavallo, L., Oliva, R., Tovchigrechko, A., and Wodak, S. J. (2016).  Prediction of homoprotein and heteroprotein complexes by protein docking and template-based modeling: A CASP-CAPRI experiment.  Proteins: Structure, Function, and Bioinformatics, 84(S1):323–348.
  * Marti-Renom et al., (2009) Marti-Renom, M. A., Capriotti, E., Shindyalov, I. N., and Bourne, P. E. (2009).  Structure Comparison and Alignment.  In Gu, J. and Bourne, P. E., editors, Structural Bioinformatics, 2nd Edition, pages 397–418. John Wiley & Sons, Inc.
  * May et al., (2014) May, A., Pool, R., van Dijk, E., Bijlard, J., Abeln, S., Heringa, J., and Feenstra, K. A. (2014).  Coarse-grained versus atomistic simulations: realistic interaction free energies for real proteins.  Bioinformatics (Oxford, England), 30(3):326–334.
  * Mészáros et al., (2007) Mészáros, B., Tompa, P., Simon, I., and Dosztányi, Z. (2007).  Molecular principles of the interactions of disordered proteins.  J Mol Biol, 372(2):549–561.
  * Modi et al., (2016) Modi, V., Xu, Q., Adhikari, S., and Dunbrack, R. L. (2016).  Assessment of template-based modeling of protein structure in CASP11.  Proteins: Structure, Function, and Bioinformatics, 84(S1):200–220.
  * Moult et al., (2016) Moult, J., Fidelis, K., Kryshtafovych, A., Schwede, T., and Tramontano, A. (2016).  Critical assessment of methods of protein structure prediction: Progress and new directions in round XI.  Proteins: Structure, Function and Bioinformatics, 84(S1):4–14.
  * Moult et al., (1995) Moult, J., Pedersen, J. T., Judson, R., and Fidelis, K. (1995).  A large-scale experiment to assess protein structure prediction methods.  Proteins: Structure, Function, and Genetics, 23(3):ii–iv.
  * Oldfield et al., (2005) Oldfield, C. J., Cheng, Y., Cortese, M. S., Brown, C. J., Uversky, V. N., and Dunker, A. K. (2005).  Comparing and combining predictors of mostly disordered proteins.  Biochemistry, 44(6):1989–2000.
  * Perdigão et al., (2015) Perdigão, N., Heinrich, J., Stolte, C., Sabir, K. S., Buckley, M. J., Tabor, B., Signal, B., Gloss, B. S., Hammang, C. J., Rost, B., Schafferhans, A., and O’Donoghue, S. I. (2015).  Unexpected features of the dark proteome.  Proceedings of the National Academy of Sciences, 112(52):15898–15903.
  * Pettersen et al., (2004) Pettersen, E. F., Goddard, T. D., Huang, C. C., Couch, G. S., Greenblatt, D. M., Meng, E. C., and Ferrin, T. E. (2004).  UCSF Chimera – A visualization system for exploratory research and analysis.  Journal of Computational Chemistry, 25(13):1605–1612.
  * Pucci et al., (2017) Pucci, F., Kwasigroch, J. M., and Rooman, M. (2017).  SCooP: an accurate and fast predictor of protein stability curves as a function of temperature.  Bioinformatics.
  * Read and Chavali, (2007) Read, R. J. and Chavali, G. (2007).  Assessment of CASP7 predictions in the high accuracy template-based modeling category.  Proteins: Structure, Function, and Bioinformatics, 69(S8):27–37.
  * Richardson and Richardson, (2002) Richardson, J. S. and Richardson, D. C. (2002).  Natural beta-sheet proteins use negative design to avoid edge-to-edge aggregation.  Proc Natl Acad Sci U S A, 99(5):2754–2759.
  * Uversky et al., (2000) Uversky, V. N., Gillespie, J. R., and Fink, A. L. (2000).  Why are ”natively unfolded” proteins unstructured under physiologic conditions?  Proteins, 41(3):415–427.
  * van Dijk et al., (2015) van Dijk, E., Hoogeveen, A., and Abeln, S. (2015).  The Hydrophobic Temperature Dependence of Amino Acids Directly Calculated from Protein Structures.  PLOS Computational Biology, 11(5):e1004277.
  * van Dijk et al., (2016) van Dijk, E., Varilly, P., Knowles, T. P. J., Frenkel, D., and Abeln, S. (2016).  Consistent Treatment of Hydrophobicity in Protein Lattice Models Accounts for Cold Denaturation.  Physical Review Letters, 116(7):078101.
  * Venselaar et al., (2010) Venselaar, H., te Beek, T. A., Kuipers, R. K., Hekkelman, M. L., and Vriend, G. (2010).  Protein structure analysis of mutations causing inheritable diseases. An e-Science approach with life scientist friendly interfaces.  BMC Bioinformatics, 11(1):548.
  * Wang et al., (2016) Wang, S., Ma, J., and Xu, J. (2016).  AUCpreD: proteome-level protein disorder prediction by AUC-maximized deep convolutional neural fields.  Bioinformatics (Oxford, England), 32(17):i672–i679.
  * Waterhouse et al., (2009) Waterhouse, A. M., Procter, J. B., Martin, D. M. A., Clamp, M., and Barton, G. J. (2009).  Jalview Version 2–a multiple sequence alignment editor and analysis workbench.  Bioinformatics, 25(9):1189–1191.
  * Zemla, (2003) Zemla, A. (2003).  LGA: A method for finding 3D similarities in protein structures.  Nucleic acids research, 31(13):3370–4.

## Index