#  TopicAdapt- An Inter-Corpora Topics Adaptation Approach

Pritom Saha Akash Trisha Das Kevin Chen-Chuan Chang
University of Illinois at Urbana-Champaign, USA
{pakash2, trishad2<EMAIL_ADDRESS>
###### Abstract

Topic models are popular statistical tools for detecting latent semantic
topics in a text corpus. They have been utilized in various applications
across different fields. However, traditional topic models have some
limitations, including insensitivity to user guidance, sensitivity to the
amount and quality of data, and the inability to adapt learned topics from one
corpus to another. To address these challenges, this paper proposes a neural
topic model, TopicAdapt, that can adapt relevant topics from a related source
corpus and also discover new topics in a target corpus that are absent in the
source corpus. The proposed model offers a promising approach to improve topic
modeling performance in practical scenarios. Experiments over multiple
datasets from diverse domains show the superiority of the proposed model
against the state-of-the-art topic models.

##  1 Introduction

To effectively and quickly comprehend and navigate a big text corpus, it is
important to mine a set of diverse and cohesive topics automatically. Topic
models Jordan et al. (1999); Blei et al. (2003) are statistical tools for
detecting latent semantic themes in a text collection. These approaches have
gained popularity for text mining Foster and Kuhn (2007); Mei et al. (2007)
and information retrieval tasks Dou et al. (2007); Wei and Croft (2006)
spanning a wide range of applications in fields such as science, humanities,
business, and other related areas Boyd-Graber et al. (2017).

Despite the effectiveness of standard topic models for understanding latent
topics in a large corpus, they suffer from several drawbacks. Firstly, the
traditional topic Jordan et al. (1999); Blei et al. (2003) models do not
consider user guidance in learning the topics. For example, users may already
know the name of topics but want to know the corpus-specific representation of
that topic.

Secondly, the performance of topic models is often sensitive to the amount of
data and the quality of the data, and a small corpus may not provide enough
information to identify the underlying topics accurately. One possible way to
handle this is to adapt a pre-trained topic model from a related corpus to the
target corpus. It leverages the knowledge learned from a large source corpus
to improve the topic modeling performance on the small target corpus. However,
in traditional topic models, there is no specific way to adapt learned topics
from one corpus to another.

Moreover, not all the topics of the related source domain are actual topics of
the target domain, and there may also exist new topics in the target corpus
different from the source corpus. For example, the source domain may cover
topics such as “politics” and “sports” where the target may have a new topic,
“entertainment”, different from the source domain. Therefore, we develop a
model named TopicAdapt that can dynamically adapt relevant topics from the
source domain by transfer learning and also can discover new topics available
in the target domain but absent in the source domain. To evaluate the
performance of the proposed model, we conduct both quantitive and qualitative
evaluations over multiple datasets from diverse domains. The experimental
results show the superiority of the proposed model against the state-of-the-
art topic models.

##  2 Methodology

###  2.1 Problem Statement

We proposed a problem of adapting topics from one corpus (i.e., domain) to
another. As input, it takes a target corpus $\mathcal{D}$, the topic-word
distribution from a source reference corpus or alternatively named
representation $\beta^{r}$ for $k$ well-defined topics with their surface
names $\mathcal{C}$. As output, we want to learn topic-word distribution
$\beta$ for the target corpus that best represents the corpus by given well-
known topics. It also aims to generate new topics from the target corpus
without any supervision or minimal supervision, such as using only the topic
surface names. Similar to an existing topic model named Coordinated Topic
Model (CTM) Akash et al. (2022), we can get a set of well-defined topics with
their representation. More specifically, we use labeled LDA Ramage et al.
(2009) to get reference representation (more details on Akash et al. (2022)).

Figure 1: Proposed Architecture

To solve our problem, we have chosen to use the Embedded Topic Model (ETM)
Dieng et al. (2020) as the foundation of our proposed model and extension of a
recent topic model named Coordinated Topic Modeling (CTM) Akash et al. (2022).
We have several compelling reasons for this choice. Firstly, ETM is an
excellent choice because it effectively combines the strengths of neural topic
modeling and word embedding when modeling a corpus. Secondly, using pre-
trained word embeddings enables us to map words in a common vector space, even
if those words are not present in the target corpus vocabulary. Finally, we
can impose our problem-specific requirements by applying regularization
techniques to the objectives of ETM. Similar to CTM, the proposed framework
uses topic-level and document-level supervision. Moreover, our model also
incorporates word-level supervision for having topics comprising semantically
similar words.

As part of our problem, we are given a topic-word distribution $\beta^{r}$ for
some known topics, along with their surface names $\mathcal{C}$. We aim to
adapt these topics for a target corpus $\mathcal{D}$ and discover new topics.
To achieve this, we have modified the ETM model to incorporate supervision
from $\mathcal{C}$ and $\beta^{r}$ as guidance. However, we cannot directly
use $\beta^{r}$ in the ETM model for the target corpus due to the vocabulary
mismatch problem with the reference corpus. Therefore, we have modified the
original ETM model structure, similar to CTM, to learn a topic-word
distribution with vocabulary dimensions comparable to $\beta^{r}$.
Additionally, we have generated pseudo-labeled documents in the target corpus
using $\mathcal{C}$ to enhance document modeling in ETM. Lastly, we have also
used $\mathcal{C}$ to bias the topic distribution and create topics consisting
of semantically closer words. The overview of our model is shown in Figure 1.

###  2.2 TopicAdapt

Topic-level Supervision: A set of topics with a reference representation
$\beta^{r}$ is employed as source topics to guide the generation of a target
representation $\beta$ that best captures the characteristics of the given
$\mathcal{D}$. The reference representation may be obtained from sources such
as a large annotated corpus in a similar domain. However, a key issue arises
in using $\beta^{r}$ directly as guidance, as it cannot be assumed that
$\beta^{r}$ and $\beta$ share the same vocabulary. To solve this, following
CTM, Akash et al. (2022), an indirect method of supervision called "reference
projection" is employed. To elaborate further, in conjunction with the
parameter $\beta$, the projected representation
$\tilde{\beta}^{r}=f(\tilde{\rho}^{\top}\alpha)$ is computed where
$\tilde{\rho}$ denotes the embedding matrix associated with the lexicon upon
which the reference $\beta^{r}$ is constructed. Finally, $\tilde{\beta}^{r}$
is used to indirectly guide $\beta$ by minimizing the following:

$\displaystyle R_{\beta}$
$\displaystyle=\frac{1}{k}\sum^{k}_{j=1}KL(\beta^{r}_{j},\tilde{\beta}^{r}_{j})\vspace{-5mm}$

Document-level Supervision: Similar to CTM Akash et al. (2022), in this study,
we utilize $\mathcal{C}$ to obtain $\theta^{t}$ for document-level
supervision. To achieve this, we employ a pre-trained textual entailment model
Liu et al. (2019). The model takes an input document $d$ as the “premise”
creates a “hypothesis” by filling a template with a surface name
$c_{k}\in\mathcal{C}$, and produces a probability $p_{dk}$ representing the
extent to which the premise entails the hypothesis. This distribution is then
used to guide document topic distribution. We directly utilize the generated
probabilities $p_{dk}$ as a soft label for $\theta^{t}_{dk}$. Soft labeling
offers the opportunity to implement a technique proposed by Bhatia et al.
(2016), which emphasizes the high-probability label while diminishing the low-
probability ones. To accomplish this, the method squares and normalizes the
$p_{dk}$ values in the following manner:

$\displaystyle\theta^{t}_{dk}$
$\displaystyle=\frac{p^{2}_{dk}/f_{k}}{\sum_{k^{\prime}}p^{2}_{dk^{\prime}}/f_{k^{\prime}}},f_{k}=\sum_{d\in
D}p_{dk}$

The $\theta^{t}$ value is employed to offer supervision at the document level
by reducing the following:

$\displaystyle R_{\theta}$ $\displaystyle=\frac{1}{|D|}\sum_{d\in
D}KL(\theta^{t}_{d},\theta_{d}).$

Word-level Supervision: The distribution of topics over vocabulary words is
such that the most relevant words in a given topic are semantically related to
the topic’s name. To leverage this observation, pretrained word embeddings are
employed to obtain embeddings for all vocabulary words. Subsequently, cosine
similarity between the surface name of a topic and vocabulary words is used to
generate a topic conditional probability distribution over all the vocabulary
words $\gamma$. $\gamma$ serves as a guide for constructing the topic-word
distribution.

$\displaystyle R_{\gamma}$
$\displaystyle=\frac{1}{k}\sum^{k}_{j=1}KL(\gamma_{j},\beta_{j})$

####  2.2.1 Training

We unify topic-level, document-level, and word-level supervision into one
model by constraining the objective of our base model as follows:

$\displaystyle\mathcal{L}(\theta)$
$\displaystyle=ELBO-\gamma_{\beta}R_{\beta}-\gamma_{\theta}R_{\theta}-\gamma_{\gamma}R_{\gamma},$
(1)

where $\gamma_{\beta}$, $\gamma_{\theta}$ and $\gamma_{\gamma}$ are the
regularization weights for $R_{\beta}$, $R_{\theta}$ and $R_{\gamma}$
respectively. Maximizing Eq. 1 ensures the following objectives: (1) The ELBO
part enforces the model to explain $D$ by reducing the reconstruction error;
(2) $R_{\beta}$ enforces the model to move $\beta$ in the direction of
$\beta^{r}$; (3) $R_{\theta}$ encourages the model to maintain the global
semantics of given topics in $\beta$ by enforcing $\theta$ and $\theta^{t}$ as
similar as possible; and (4) $R_{\gamma}$ enforces topic words to be similar
to relevant words in the vocabulary.

##  3 Experiments

###  3.1 Data

We use three datasets from news articles: 20 Newsgroup corpus
111http://qwone.com/ jason/20Newsgroups/, New York Times annotated corpus
Sandhaus (2008), AG’s News dataset Yang et al. (2016). For the review
sentiment domain, we use the Yelp restaurant review dataset and IMDB Movie
Review dataset. For academic articles, we use: Arxiv abstracts
222https://www.kaggle.com/Cornell-University/arxiv, Microsoft Academic Graph
AI article abstracts Sinha et al. (2015). See Appendix A.2 for more details.

###  3.2 Baselines

We compare our model with the following baselines. GLDA Jagarlamudi et al.
(2012), Sup+LLDA Ramage et al. (2009), ZS+LLDA Ramage et al. (2009), ACorEx,
AVIAD Hoang et al. (2019), KeyETM Harandizadeh et al. (2022), ECTM Saha Akash
et al. (2022). The details of the baselines can be found in Appendix A.3.

###  3.3 Topic Quality Evaluation

We use the following three quantitative measurements to evaluate the quality
of inferred topics: Topic coherence (TC), Topic diversity (TD), and Topic
Quality (TQ). Details about these metrics can be found in the Appendix A.4.

We first show the quantitative results of topic quality in Table 1. The
results suggest that, for news and sentiment domains, TopicAdapt generates
more coherent and interpretable topics than other baselines.

Methods | 20Newsg | NYT | Yelp-Senti | Arxiv-AI
---|---|---|---|---
TC | TD | TQ | TC | TD | TQ | TC | TD | TQ | TC | TD | TQ
GLDA | 0.25 | 0.87 | 0.22 | 0.26 | 0.85 | 0.22 | 0.08 | 0.80 | 0.06 | 0.09 | 0.93 | 0.09
Sup+LLDA | 0.23 | 0.79 | 0.18 | 0.20 | 0.63 | 0.12 | 0.06 | 0.70 | 0.04 | 0.04 | 0.46 | 0.02
ZS+LLDA | 0.23 | 0.80 | 0.18 | 0.17 | 0.65 | 0.11 | 0.06 | 0.76 | 0.05 | 0.14 | 0.80 | 0.11
ACorEX | 0.25 | 1.00 | 0.25 | 0.27 | 1.00 | 0.27 | 0.07 | 1.00 | 0.07 | -0.03 | 0.96 | -0.03
AVIAD | 0.13 | 1.00 | 0.13 | -0.26 | 1.00 | -0.26 | -0.01 | 1.00 | -0.01 | -0.34 | 1.00 | -0.34
KeyETM | 0.26 | 1.00 | 0.26 | 0.19 | 0.89 | 0.17 | 0.07 | 0.92 | 0.07 | 0.04 | 1.00 | 0.04
ECTM | 0.30 | 1.00 | 0.30 | 0.28 | 0.97 | 0.27 | 0.09 | 1.00 | 0.09 | 0.15 | 0.97 | 0.15
TopicAdapt | 0.30 | 1.00 | 0.30 | 0.28 | 0.97 | 0.27 | 0.11 | 1.00 | 0.11 | 0.12 | 0.93 | 0.12

Table 1: Quality Measures of Topic

In Table 2, we show randomly selected two topics from each dataset and top-5
words under each topic from reference topic words, ECTM and TopicAdapt. Words
that we found to be irrelevant to the corresponding topic are marked with
($\times$) in Table 2. The table consisting of results from all baselines can
be found in A.5.

| 20Newsg | NYT | Yelp-Senti | Arxiv-AI
---|---|---|---|---
sports | politics | business | technology | good | bad | ML | IR
| Reference
---
Topic
Words
| night
---
play
sport
player
beat
| leader
---
election
attack
afp
iraqi
| stock
---
sale
share
billion
fall
| software
---
technology
service
internet
launch
| song
---
music
musical
wonderful
dance
| waste
---
awful
terrible
boring
poor
| machine
---
learning
algorithm
optimization
problem
| retrieval
---
document
query
search
base
ACorEX |  | point
---
play
player
league
beat
| force
---
country
attack
military
political
| billion
---
business
buy
stock
profit
| release $(\times)$
---
technology
phone
time $(\times)$
space
| good
---
hear $(\times)$
beautiful
music $(\times)$
sound $(\times)$
| bad
---
money $(\times)$
terrible
poor
waste
| optimization
---
gradient
convergence
stochastic
print $(\times)$
| search
---
document
query
retrieval
semantics
AVIAD |  | robitaille $(\times)$
---
probert $(\times)$
howe $(\times)$
player
nhl
| tragedy $(\times)$
---
policy
serbian
freedom
unite $(\times)$
| sanwa $(\times)$
---
zoete $(\times)$
earning
overprice
acquirer
| genscher $(\times)$
---
enlargement $(\times)$
abm $(\times)$
teng $(\times)$
chechnya $(\times)$
| traditional $(\times)$
---
snow $(\times)$
filling
bisque $(\times)$
seaweed $(\times)$
| email $(\times)$
---
upset
management $(\times)$
yell
acknowledge $(\times)$
| bind
---
analytically $(\times)$
certify $(\times)$
arm $(\times)$
pruning
| ehr
---
healthy $(\times)$
progression $(\times)$
patient $(\times)$
ehrs
KeyETM |  | game
---
team
season
play
win
| people
---
government
person $(\times)$
armenian
law
| year $(\times)$
---
percent
market
time $(\times)$
month $(\times)$
| company $(\times)$
---
bank $(\times)$
japan $(\times)$
china $(\times)$
russia $(\times)$
| good
---
place $(\times)$
great
time $(\times)$
love
| food $(\times)$
---
order $(\times)$
service $(\times)$
eat $(\times)$
restaurant $(\times)$
| function
---
estimation $(\times)$
distribution $(\times)$
parameter $(\times)$
efficient $(\times)$
| translation $(\times)$
---
user
search
annotation
point $(\times)$
ECTM |  | game
---
team
win
season
league
| government
---
war
military
armenian
attack
| company
---
bank
percent
market
price
| space
---
site
technology
station
network
| great
---
music $(\times)$
love
wonderful
amazing
| waste
---
awful
terrible
bad
horrible
| optimization
---
convergence
stochastic
gradient
function
| retrieval
---
document
query
search
user
TopicAdapt |  | game
---
team
win
season
play
| government
---
war
military
president
political
| percent
---
company
bank
year
market
| company
---
technology
space
site
station
| excellent
---
great
good
superb
perfect
| waste
---
bad
horrible
crap
garbage
| machine
---
problem
algorithm
convergence
optimization
| retrieval
---
document
search
query
semantic

Table 2: Qualitative Evaluation

In comparison to baselines, our method’s generated topic terms are generally
pertinent and simple to understand. We also note that the topics created by
AcorEx have respectable interpretability (See Appendix A.5). However, rather
than adapting to the target corpus, AcorEx’s produced topics strictly converge
toward the prior representation. Our approach, in contrast, tends to capture
the elements of the given themes that are unique to the target corpus. AVIAD,
on the other hand, has the opposite problem. It varies so widely that the
subjects are incredibly challenging to comprehend. When the target corpus is
balanced, the KeyETM with a similar base model (ETM) to ours performs better.
For instance, the keyETM works well since the dataset 20Newsg is relatively
balanced. Our model consistently outperforms the competition because it
benefits from both topic-level supervision and document-level supervision from
existing knowledge sources to make the topics adjusted to the target corpus
while also maintaining the semantics of the given topic names. Moreover, the
words from each inferred topic are more semantically related to each other
than other baselines, thanks to our word-level supervision.

###  3.4 Case studies

Case study 1: From Table 3, we can see our model can generate new topics from
the target corpus without supervision from the source corpus. We infer the
topic names by observing the top 5 words of each topic.

Topic Name | Top 5 words
---|---
gun violence | gun, law, president, firearm, crime
sales | price, sale, buy, sell, work
Table 3: Case study 1- No supervision for target corpus-specific topic. We
infer the topic names by observing the top 5 words of each topic

Case study 2: For this experiment, we use the AG News dataset as the source
corpus and NYT corpus as the target domain. Particularly, we selected the
period of attack at the Twin Towers from the New York Times corpus to
investigate if the model can adapt given topics from the source corpus as well
as find new topics from the target corpus. From Table 4, we can see that the
model is able to adapt relevant words for each topic related to both cases.
For the topics of 911 and 9/11, we used minimal supervision by providing the
topic surface names to the model. 911 is mostly related to medical emergency
cases, whereas 9/11 refers to the terrorist attack. The model is able to
identify the most relevant words from the target documents (NYT corpus on
9/11/2001).

Topic Name | Top 5 Words
---|---
9/11 | terrorist, terror, terrorism, militant, terrorists
911 | doctor, hospital, medical, physician, nurse

Table 4: Case study 2- Topic name as minimal supervision for target corpus
specific topic

##  4 Conclusion

In this paper, we propose a problem of adapting topics from a source corpus to
a target corpus, also identifying new topics for the target corpus. Different
from a recent work called coordinated topic modeling which only uses well-
defined topics to describe a new corpus, we also mine new topics that
represent the target corpus. For this purpose, we design a method named
TopicAdapt which is based on an embedded topic model Dieng et al. (2020) that
uses three levels of supervision namely word-level supervision, topic-level
supervision, and document-level supervision. An extensive experiment over a
set of datasets from different domains demonstrates the superiority of the
proposed model over multiple strong baselines.

##  5 Limitation

The proposed model depends on two pretrained models- pretrained word-embedding
for vocabulary words and pretrained language model for textual entailment
during generating document-level supervision. However, for a very specific
target domain, the pretarined knowledge might be appropriate. In such a case,
finetuning those models on the target corpus is worth exploring for better
performance. Moreover, similar to CTM Akash et al. (2022), in this paper, we
assume that the reference and target corpora are from common or very similar
domains. However, practically, it is very probable that we may need to
transfer topic knowledge from one domain to another. Therefore, extending our
model for cross-domain scenarios is also an interesting future direction.

## References

  * Akash et al. (2022) Pritom Saha Akash, Jie Huang, and Kevin Chen-Chuan Chang. 2022.  Coordinated topic modeling.  _arXiv preprint arXiv:2210.08559_.
  * Andrzejewski and Zhu (2009) David Andrzejewski and Xiaojin Zhu. 2009.  Latent dirichlet allocation with topic-in-set knowledge.  In _Proceedings of the NAACL HLT 2009 Workshop on Semi-Supervised Learning for Natural Language Processing_ , pages 43–48.
  * Bhatia et al. (2016) Shraey Bhatia, Jey Han Lau, and Timothy Baldwin. 2016.  Automatic labelling of topics with neural embeddings.  _arXiv preprint arXiv:1612.05340_.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Boyd-Graber et al. (2017) Jordan L Boyd-Graber, Yuening Hu, David Mimno, et al. 2017.  _Applications of topic models_ , volume 11.  now Publishers Incorporated.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dieng et al. (2020) Adji B Dieng, Francisco JR Ruiz, and David M Blei. 2020.  Topic modeling in embedding spaces.  _Transactions of the Association for Computational Linguistics_ , 8:439–453.
  * Dou et al. (2007) Zhicheng Dou, Ruihua Song, and Ji-Rong Wen. 2007.  A large-scale evaluation and analysis of personalized search strategies.  In _Proceedings of the 16th international conference on World Wide Web_ , pages 581–590.
  * Foster and Kuhn (2007) George Foster and Roland Kuhn. 2007.  Mixture-model adaptation for smt.  In _Proceedings of the Second Workshop on Statistical Machine Translation_ , pages 128–135.
  * Harandizadeh et al. (2022) Bahareh Harandizadeh, J Hunter Priniski, and Fred Morstatter. 2022.  Keyword assisted embedded topic model.  In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , pages 372–380.
  * Hoang et al. (2019) Tai Hoang, Huy Le, and Tho Quan. 2019.  Towards autoencoding variational inference for aspect-based opinion summary.  _Applied Artificial Intelligence_ , 33(9):796–816.
  * Hofmann (1999) Thomas Hofmann. 1999.  Probabilistic latent semantic indexing.  In _Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 50–57.
  * Jagarlamudi et al. (2012) Jagadeesh Jagarlamudi, Hal Daumé III, and Raghavendra Udupa. 2012.  Incorporating lexical priors into topic models.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 204–213.
  * Jordan et al. (1999) Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. 1999\.  An introduction to variational methods for graphical models.  _Machine learning_ , 37(2):183–233.
  * Lacoste-Julien et al. (2008) Simon Lacoste-Julien, Fei Sha, and Michael Jordan. 2008.  Disclda: Discriminative learning for dimensionality reduction and classification.  _Advances in neural information processing systems_ , 21.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mcauliffe and Blei (2007) Jon Mcauliffe and David Blei. 2007.  Supervised topic models.  _Advances in neural information processing systems_ , 20.
  * Mei et al. (2007) Qiaozhu Mei, Xuehua Shen, and ChengXiang Zhai. 2007.  Automatic labeling of multinomial topic models.  In _Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 490–499.
  * Meng et al. (2020) Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. 2020.  Discriminative topic mining via category-name guided text embedding.  In _Proceedings of The Web Conference 2020_ , pages 2121–2132.
  * Miao et al. (2016) Yishu Miao, Lei Yu, and Phil Blunsom. 2016.  Neural variational inference for text processing.  In _International conference on machine learning_ , pages 1727–1736. PMLR.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Ramage et al. (2009) Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D Manning. 2009.  Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora.  In _Proceedings of the 2009 conference on empirical methods in natural language processing_ , pages 248–256.
  * Saha Akash et al. (2022) Pritom Saha Akash, Jie Huang, and Kevin Chen-Chuan Chang. 2022.  Coordinated topic modeling.  _arXiv e-prints_ , pages arXiv–2210.
  * Sandhaus (2008) Evan Sandhaus. 2008.  The new york times annotated corpus.  _Linguistic Data Consortium, Philadelphia_ , 6(12):e26752.
  * Sinha et al. (2015) Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. 2015.  An overview of microsoft academic service (mas) and applications.  In _Proceedings of the 24th international conference on world wide web_ , pages 243–246.
  * Srivastava and Sutton (2016) Akash Srivastava and Charles Sutton. 2016.  Neural variational inference for topic models.  _ArXiv Preprint_ , 1(1):1–12.
  * Wei and Croft (2006) Xing Wei and W Bruce Croft. 2006.  Lda-based document models for ad-hoc retrieval.  In _Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 178–185.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_ , pages 1480–1489.

##  Appendix A Appendix

###  A.1 Related Works

### Transfer Learning

Transfer learning is a machine learning technique where a model trained on one
task is used as a starting point for a model on a related but different task.
This allows the model to take advantage of previously learned representations,
improving the training speed and performance compared to starting from
scratch. Transfer learning has become a popular technique in deep learning,
where pre-trained models on large datasets can be fine-tuned for specific
tasks with much smaller datasets.

Transfer learning has been used in image classification, natural language
processing (NLP), speech recognition, etc. In NLP, transfer learning refers to
pre-training an NLP model on a large corpus of text, such as a general-purpose
language modeling dataset, and then fine-tuning the model on a smaller,
specific NLP task, such as sentiment analysis or question answering. Some
examples of pre-trained models used for transfer learning in NLP include BERT
Devlin et al. (2019), GPT Radford et al. (2018), ELMo Peters et al. (2018),
etc.

### Topic Modeling

Topic modeling is a machine learning technique used to identify patterns in
large collections of text data. The goal of topic modeling is to discover
topics or themes that occur in a set of documents and quantify each document’s
relevance to these topics. There are mainly three types of topic modeling
approaches:

  * •

Probabilistic topic modeling: Probabilistic topic modeling uses probabilistic
methods to identify topics in a collection of text data. This type of topic
modeling views each document as a mixture of topics and each topic as a
distribution over words. Some common probabilistic topic modeling algorithms
are LDA Blei et al. (2003), PLSA Hofmann (1999), etc.

  * •

Neural topic modeling: Neural topic modeling is a type of topic modeling that
uses deep learning techniques to model the relationships between topics and
words in a collection of text data. In neural topic modeling, the model
typically consists of two components: an encoder that transforms the text data
into a low-dimensional representation and a decoder that maps the low-
dimensional representation back to the original space of words. The model is
trained to minimize the reconstruction error between the input and the
reconstructed text while also encouraging the low-dimensional representation
to capture the underlying topics in the data. Some well-known neural topic
models are ProdLDA Miao et al. (2016), NVDM Srivastava and Sutton (2016), etc.

  * •

User-guided topic modeling: User-guided topic modeling is a type of topic
modeling that allows the user to provide additional information or guidance to
the topic modeling process. When training a model using predicted document
category labels, Supervised LDA Mcauliffe and Blei (2007) and DiscLDA Lacoste-
Julien et al. (2008) make the assumption that each document has a label
associated with it. To create topic models, several research uses word-level
supervision. For instance, Dirichlet Forest Andrzejewski and Zhu (2009) priors
have been used to include constraints on must-link and cannot-link
relationships between seed words. A seed topic distribution is used by Seeded
LDA Jagarlamudi et al. (2012) to learn seed-related topics under the
supervision of user-supplied seed words. Finally, there is an approach called
category-guided topic mining Meng et al. (2020) (CatE), which considers the
topics’ surface names as the only supervision for mining user-interested
discriminative topics.

###  A.2 Data

We use three datasets from news articles:

  * •

20 Newsgroup corpus 333http://qwone.com/ jason/20Newsgroups/: It consists of
approximately 20,000 documents, each belonging to one of 20 different
newsgroups.

  * •

New York Times annotated corpus Sandhaus (2008): It contains over 1.8 million
articles, each annotated with a rich set of metadata including headline,
byline, date, section, and article abstract.

  * •

AG’s News dataset Yang et al. (2016):It contains over 1 million news articles,
with approximately 30,000 articles per category.

For review sentiment domain, we use:

  * •

Yelp restaurant review dataset: The dataset includes over 8 million reviews,
200,000 businesses, and 6 million users from various locations around the
world.

  * •

IMDB Movie Review dataset: The dataset includes 50,000 movie reviews, with
25,000 labeled as positive and 25,000 labeled as negative.

For academic articles we use:

  * •

Arxiv Artificial Intelligence (AI) article abstracts spanning 2020-2022
444https://www.kaggle.com/Cornell-University/arxiv;

  * •

Microsoft Academic Graph AI article abstracts Sinha et al. (2015).

###  A.3 Baselines

We compare our model with the following baselines.

  * •

GLDA: Guided LDA Jagarlamudi et al. (2012) introduces bias into the generative
process of LDA by utilizing topic-level priors over vocabulary based on
designated seed words.

  * •

Sup+LLDA: Supervised Labeled LDA is an extension of Labeled-LDA Ramage et al.
(2009) where a label for each document is predicted from a supervised BERT
learned on annotated reference corpus.

  * •

ZS+LLDA: Zero-Shot Labeled LDA is also an extension of Labeled-LDA Ramage et
al. (2009) where a label for each document is inferred from given surface
names using a Zero-Shot classification.

  * •

ACorEx: Anchored CorEx uses topic correlation to learn topics with maximal
information. It also uses user-provided seed words as anchors to bias
compression of the original corpus.

  * •

AVIAD: AVIAD Hoang et al. (2019) aims to incorporate prior knowledge obtained
from seed words into the model by altering the loss function to infer the
desired topics.

  * •

KeyETM: Keyword Assisted ETM Harandizadeh et al. (2022) integrates prior
knowledge obtained from designated seed words.

  * •

ECTM: ECTM Saha Akash et al. (2022) uses topic- and document-level supervision
for topic modeling.

###  A.4 Evaluation Metrics

  * •

Topic coherence (TC): TC is a standard measure of interpretability based on
the average point-wise mutual information between randomly drawn two words
from a document.

  * •

Topic diversity (TD): TD measures the percentage of unique words in the top 25
words from all topics.

  * •

Topic Quality (TQ): TQ is the product of topic coherence and topic diversity.

###  A.5 Qualitative Evaluation

The complete qualitative result is shown in Table 5.

| 20Newsg | NYT | Yelp-Senti | Arxiv-AI
---|---|---|---|---
sports | politics | business | technology | good | bad | ML | IR
| Reference
---
Topic
Words
| night
---
play
sport
player
beat
| leader
---
election
attack
afp
iraqi
| stock
---
sale
share
billion
fall
| software
---
technology
service
internet
launch
| song
---
music
musical
wonderful
dance
| waste
---
awful
terrible
boring
poor
| machine
---
learning
algorithm
optimization
problem
| retrieval
---
document
query
search
base
GLDA |  | game
---
team
year $(\times)$
play
player
| people
---
time $(\times)$
government
gun
year $(\times)$
| company
---
percent
year $(\times)$
bank
market
| president $(\times)$
---
bush $(\times)$
official $(\times)$
united $(\times)$
house $(\times)$
| good
---
place $(\times)$
food $(\times)$
great
order $(\times)$
| order $(\times)$
---
food $(\times)$
time $(\times)$
place $(\times)$
service $(\times)$
| adversarial
---
distribution $(\times)$
class $(\times)$
function $(\times)$
attack $(\times)$
| graph
---
search
user
class $(\times)$
recommendation
Sup+LLDA |  | game
---
team
year $(\times)$
play
time $(\times)$
| people
---
government
kill
time $(\times)$
year $(\times)$
| year $(\times)$
---
percent
company
market
government $(\times)$
| year $(\times)$
---
time $(\times)$
people $(\times)$
president $(\times)$
official $(\times)$
| place $(\times)$
---
food $(\times)$
good
great
service $(\times)$
| food $(\times)$
---
order $(\times)$
place $(\times)$
service $(\times)$
time $(\times)$
| demonstrate $(\times)$
---
problem $(\times)$
feature
training
neural
| retrieval
---
exist $(\times)$
demonstrate $(\times)$
representation
feature $(\times)$
ZS+LLDA |  | game
---
team
year $(\times)$
play
player
| people
---
time $(\times)$
government
year $(\times)$
point $(\times)$
| year $(\times)$
---
percent
company
market
lead $(\times)$
| year $(\times)$
---
time $(\times)$
american $(\times)$
official $(\times)$
today $(\times)$
| food $(\times)$
---
place $(\times)$
good
great
service $(\times)$
| food $(\times)$
---
order $(\times)$
place $(\times)$
service $(\times)$
time $(\times)$
| efficient $(\times)$
---
reduce $(\times)$
number $(\times)$
leverage $(\times)$
module $(\times)$
| retrieval
---
search
user
document
query
ACorEX |  | point
---
play
player
league
beat
| force
---
country
attack
military
political
| billion
---
business
buy
stock
profit
| release $(\times)$
---
technology
phone
time $(\times)$
space
| good
---
hear $(\times)$
beautiful
music $(\times)$
sound $(\times)$
| bad
---
money $(\times)$
terrible
poor
waste
| optimization
---
gradient
convergence
stochastic
print $(\times)$
| search
---
document
query
retrieval
semantics
AVIAD |  | robitaille $(\times)$
---
probert $(\times)$
howe $(\times)$
player
nhl
| tragedy $(\times)$
---
policy
serbian
freedom
unite $(\times)$
| sanwa $(\times)$
---
zoete $(\times)$
earning
overprice
acquirer
| genscher $(\times)$
---
enlargement $(\times)$
abm $(\times)$
teng $(\times)$
chechnya $(\times)$
| traditional $(\times)$
---
snow $(\times)$
filling
bisque $(\times)$
seaweed $(\times)$
| email $(\times)$
---
upset
management $(\times)$
yell
acknowledge $(\times)$
| bind
---
analytically $(\times)$
certify $(\times)$
arm $(\times)$
pruning
| ehr
---
healthy $(\times)$
progression $(\times)$
patient $(\times)$
ehrs
KeyETM |  | game
---
team
season
play
win
| people
---
government
person $(\times)$
armenian
law
| year $(\times)$
---
percent
market
time $(\times)$
month $(\times)$
| company $(\times)$
---
bank $(\times)$
japan $(\times)$
china $(\times)$
russia $(\times)$
| good
---
place $(\times)$
great
time $(\times)$
love
| food $(\times)$
---
order $(\times)$
service $(\times)$
eat $(\times)$
restaurant $(\times)$
| function
---
estimation $(\times)$
distribution $(\times)$
parameter $(\times)$
efficient $(\times)$
| translation $(\times)$
---
user
search
annotation
point $(\times)$
ECTM |  | game
---
team
win
season
league
| government
---
war
military
armenian
attack
| company
---
bank
percent
market
price
| space
---
site
technology
station
network
| great
---
music $(\times)$
love
wonderful
amazing
| waste
---
awful
terrible
bad
horrible
| optimization
---
convergence
stochastic
gradient
function
| retrieval
---
document
query
search
user
TopicAdapt |  | game
---
team
win
season
play
| government
---
war
military
president
political
| percent
---
company
bank
year
market
| company
---
technology
space
site
station
| excellent
---
great
good
superb
perfect
| waste
---
bad
horrible
crap
garbage
| machine
---
problem
algorithm
convergence
optimization
| retrieval
---
document
search
query
semantic

Table 5: Qualitative Evaluation


# Probing Language Models’ Gesture Understanding
for Enhanced Human-AI Interaction

Philipp Wicke
Center for Information and Language Processing, LMU Munich
Munich Center for Machine Learning (MCML)
<EMAIL_ADDRESS>
###### Abstract

The rise of Large Language Models (LLMs) has affected various disciplines that
got beyond mere text generation. Going beyond their textual nature, this
project proposal aims to investigate the interaction between LLMs and non-
verbal communication, specifically focusing on gestures. The proposal sets out
a plan to examine the proficiency of LLMs in deciphering both explicit and
implicit non-verbal cues within textual prompts and their ability to associate
these gestures with various contextual factors. The research proposes to test
established psycholinguistic study designs to construct a comprehensive
dataset that pairs textual prompts with detailed gesture descriptions,
encompassing diverse regional variations, and semantic labels. To assess LLMs’
comprehension of gestures, experiments are planned, evaluating their ability
to simulate human behaviour in order to replicate psycholinguistic
experiments. These experiments consider cultural dimensions and measure the
agreement between LLM-identified gestures and the dataset, shedding light on
the models’ contextual interpretation of non-verbal cues (e.g. gestures).

Probing Language Models’ Gesture Understanding
for Enhanced Human-AI Interaction

Philipp Wicke Center for Information and Language Processing, LMU Munich
Munich Center for Machine Learning (MCML<EMAIL_ADDRESS>

##  1 Introduction

The successful launch of OpenAI’s ChatGPT in November 2022, with one million
users within five days Hu (2023), sparked considerable interest in
conversational AI. Built on the Generative Pretrained Transformer (GPT)
architecture Radford et al. (2019); Brown et al. (2020), ChatGPT employs
attention mechanisms for text generation in a dialogue-format. Trained
extensively, underlying Large Language Models (LLMs) demonstrate rudimentary
forms of creativity and comprehension, prompting speculation about “sparks of
general artificial intelligence” Bubeck et al. (2023). This raises questions
about whether LLMs simulate language and exhibit cognitive elements akin to
machine or human cognition. Distinguishing human cognition from machines
connects to the philosophical concept of Multiple Realizability Putnam et al.
(1967), suggesting that different physical systems can produce similar
cognitive processes.

Embodied cognition, emphasizing the role of an organism’s body and sensory
experiences Varela et al. (2017), adds to this perspective. Exploring how
embodied experiences influence LLMs’ cognitive attributes becomes crucial as
scientific and non-scientific domains increasingly leverage LLM capabilities
Biswas (2023); Van Dis et al. (2023); Wu et al. (2023). Simultaneously,
addressing bias within language models is essential due to concerns about
perpetuating societal biases Tamkin et al. (2021). Given the absence of a
physical body in LLMs, questions arise about the presence or absence of
fundamental principles governing human-AI interactions in their
representations. These inquiries necessitate multifaceted exploration.

Figure 1: Probing a Large Language Model (LLM) through the input of gesture
descriptions can serve as a valuable means to evaluate its understanding of
gestures, contributing to the refinement of human-AI interaction.

The proposed research complements ongoing efforts exploring these questions.
Related research investigates embodying LLMs in robots for enhanced human-
robot interactions Wicke et al. (2023). Additionally, an examination of LLMs’
interpretation of figurative language, especially metaphors, reveals perceived
embodiment significantly aids their interpretative capacity Wicke (2023).
Adding a third perspective, the proposed project focuses on the spatial
dynamics of human non-verbal communication, namely gestures. Gestures, crucial
for structuring communication McNeill (1992), bridge the gap between
linguistic concepts and bodily expressions Mittelberg (2006). Currently, no
interdisciplinary exploration exists into the role of gestures in the LLM
context. Important questions remain:

> How do LLMs conceptualize gestures, and can they accurately interpret
> gestural cues translated into text?

This research aims to fill this gap, offering insights into the comprehension,
interpretation, and potential utilization of non-verbal communication cues by
LLMs.

##  2 Related Works

### Computational Representations of Gestures

Historically, gesture representation has been a fundamental aspect of
understanding human communication and interaction with computers. Early works,
such as those by McNeill (1992) laid the foundation for analysing the
linguistic and semiotic aspects of gestures. More recently, scholars such as
Cienki and Müller (2008) are at the forefront of human gesture studies. Cienki
and Müller (2008) emphasise the embodied and dynamic nature of gestures,
viewing them as integral components of language that convey meaning through
their interaction with speech and the surrounding context. This perspective
aligns with the idea that gestures are not mere embellishments but constitute
an essential part of the communication process. Regarding mental
representations, investigates the endurance of iconic origins in emblematic
gestures. Bergen’s studies suggest that emblematic gestures, despite their
historical associations, undergo cognitive changes over time, highlighting the
dynamic, cultural nature of the relationship between gestures and mental
representations Bergen (2019). Computational representations on gestures can
be found in work by Wicke and Veale (2020). The work provides a taxonomy of
schematic movements and gestures, which can be used to implement a variety of
creative performance types with robots with an emphasis on an apt use of
spatial movement.

### LLMs and Robotic Bodies

Large Language Models are opaque statistical systems, which do not allow us,
as opposed to word embeddings, to simply look up how certain concepts are
defined. Moreover, their high-dimensional latent space may define “gestures”
as multimodal as we humans conceptualise them. Hence, we can probe LLMs with
datasets of tests, which require a certain concept to be present. For example,
in research by Wicke (2023) tests LLMs ability to interpret figurative
language with the FigQA dataset Liu et al. (2022).

Moreover, to test the effect of embodiment on LLMs’ ability to understand
metaphors, which are assumed to be derived from bodily experiences Lakoff
(2008), the study correlates the performance of metaphor interpretation with
the perceived embodiment of the action words Sidhu et al. (2014) in each
metaphor. The results suggest that the degree of embodiment has a positive
impact on LLMs ability to do the correct interpretation. Similarly, the
proposed project follows a comparable approach by testing LLMs’ aptitude to
accurately comprehend and attribute gestures to text, shedding light on the
underlying conceptualisation within these text-based models.

Figure 2: Suggested Turing Experiment (TE) based on the VLM list. The item
from the VLM list (e.g. Stop) is turned into an appropriate prompt for the TE,
which is then fed to the language model (e.g. Llama-2) for evaluation.

##  3 Methodology

### Research Questions

Motivated by the need for a systematic and comprehensive exploration, the
research plan for this study is designed to shed light on the relationship
between LLMs and non-verbal communication (e.g. gestures). This investigation
has a bipartite structure: First, the most powerful types of models will be
investigated, namely LLMs of different sizes (e.g. Llama-2, GPT-3). Even
though these models are monomodal (i.e. textual), they provide the basis for
multimodal models, which can be investigated in a second step, extending the
project. Related works have already used multimodal models to create systems
for gesture classification and generation Ao et al. (2023); Gao et al. (2023).
It is important to limit the initial study to textual models, because current
multimodal models are using smaller LLMs as a backbone structure, which
inevitably guides any further multimodal conceptualisation. Hence, the
following first set of pivotal questions can be derived:

  * •

Accuracy in Interpretation: How effectively can LLMs decipher explicit and
implicit non-verbal cues (i.e. gestures) embedded within textual prompts?

  * •

Coherence in Cultural Contexts: How adept are LLMs at associating these
gestures with different contextual backdrops, revealing the depth of their
comprehension?

### Model selection

It is crucial to note that important LLMs (e.g. GPT-3) have been developed by
corporations, exemplified by entities like OpenAI. However, an inherent
limitation of these models is their unavailability with respect to model
weights and the exact training procedures employed. In the pursuit of
fostering transparency, reproducibility, and embracing the principles of open-
source research, we choose to abstain from relying on these proprietary
models. This approach not only facilitates transparency but also encourages
the broader scientific community to engage with and build upon our work. The
selection and comparison encompasses a diverse range of model parameter sizes,
including (currently) popular open-source options such as Llama-2 Touvron et
al. (2023), OPT Zhang et al. (2022), Alpaca Zhang et al. (2023), GPT-NeoX
Black et al. (2022), Bloom Workshop et al. (2022), as well as others that may
become available during the course of our research. For a potential multimodal
extension, the selection of VLMs includes the open-source models InstructBlip
Dai et al. (2023) and OpenFlamnigo Awadalla et al. (2023).

### Dataset Selection

A fundamental issue of gestural data is that there cannot be a gold standard
for the use of gestures, i.e. there is no one correct gesture accompanying a
specific sentence. To begin with, gestures are not needed for communication,
but can provide helpful additional semantic information. Yet, they show a
range of profound socio-cultural differences Matsumoto and Hwang (2013).
Fortunately, a plethora of research in gesture studies provides appropriate
psycholinguistic study designs that can aid the dataset construction.
Matsumoto and Hwang (2013) present a Verbal Message List (VML), which includes
96 items “deemed important for individuals interacting with people from a
different culture for the first time to know in order to highlight emblematic
differences” Matsumoto and Hwang (2013). These items include expressions such
as Catastrophe, Girlfriend, I’m very strong and their list is publicly
available. The list comes categories (e.g. insult, request), the verbal
message (e.g. Girlfriend), region (e.g. global, East Asia) and a description
of the gesture (e.g. Thumb of one hand out, other fingers curled; thumb
pointing in a desired direction). This data has several benefits over other
available datasets. Many recent text/gesture datasets rely on video
annotations using the ELAN (EUDICO Linguistic Annotator) system Zheng and Peng
(2022). These datasets Turchyn et al. (2018); Ienaga et al. (2022) rely on
annotations of videos at certain temporal intervals and are mostly confined to
the generation of new gestures without classification or explicit
conceptualisation of gesture types. The VML will be used to curate a diverse
dataset pairing textual prompts with detailed gesture descriptions, covering a
wide spectrum of non-verbal cues, regional labels and semantic labels (e.g.
request, insult etc.). Hence, the VML is chosen as the basis for the
evaluation.

### Turing Experiments

Aher et al. (2023) present the Turning Experiment (TE), which is a novel test
used to assess the ability of LLMs to simulate various aspects of human
behaviour, including replicating classic experiments in psycholinguistics. TEs
involve simulating a representative group of human research participants with
different cultural backgrounds within prompts given to the LLM.

These experiments aim to uncover how well LLMs can reproduce established
research findings. These TEs enable to assess multiple different answers for
gesture conceptualisation and can provide a cultural dimension. The
performance of the LLMs can thus be assessed on the VML and the metric will be
the amount of overlap between the LLMs identified gesture and the VML based on
its description and context (see Figure 2).

In Figure 2, the VML provides 96 items (abstracted with coloured cards on the
left) that are used to construct the individual TEs. Each experiment features
a scenario description (white box in the center) with a varying gender
(Mr./Mrs./Mx.) and cultural parameter. For example, Fig. 1 shows Ms. Huang as
a female, East Asian participant. The response of the LLM is compared with the
VLM label and a standard accuracy measure is conducted to assess the LLM
performance on the task. Lastly, all LLMs will be compared and correlations
between the cultural parameters will be assessed with respect to the
conceptualisation of the gestures provided by the VLM semantics.

##  4 Discussion

The proposed research represents a pioneering exploration into the interaction
between Large Language Models (LLMs), such as Llama-2 and GPT-3, and non-
verbal communication cues, particularly gestures. By aiming to unravel how
LLMs conceptualize and interpret gestures, the research delves into a novel
and interdisciplinary domain at the intersection of artificial intelligence
and human communication dynamics. The primary objective is to discern the
proficiency of LLMs in comprehending and accurately representing non-verbal
cues within textual prompts in order to lay the ground work for further
studies intersecting the field of gesture studies and statistical language
modelling.

The planned investigation acknowledges the importance of cultural context in
gesture comprehension. The findings should acknowledge that LLMs, when exposed
to diverse datasets, may exhibit a commendable variability to associate
gestures with different cultural backgrounds. This cultural sensitivity aligns
with the broader aim of developing more inclusive and contextually aware
conversational AI systems. Its insights may prove valuable for the future
design on human-robot interaction that utilises generative language models.

### Limitations

The reliance on open-source models, while promoting transparency, may
introduce limitations in terms of model complexity compared to proprietary
counterparts like GPT-3. This raises questions about the generalizability of
the findings to models with different architectures and scales. Moreover, the
initial focus on monomodal (textual) models limits the exploration of
multimodal capabilities, which could potentially enhance gesture
comprehension. Future research may need to address this limitation to provide
a more comprehensive understanding of non-verbal communication with artificial
agents using generative models.

Lastly, as a proposal, the research has not been conducted, and the strengths
and weaknesses outlined are based on anticipated outcomes. The actual
performance of LLMs in gesture comprehension remains to be empirically tested
and evaluated.

### Future Work

Future research directions could include extending the study to multimodal
models, exploring the integration of visual information to enhance gesture
comprehension. Additionally, investigating the impact of gesture-based
communication on user experience and engagement with conversational AI systems
could provide valuable insights for the design and development of more
effective and user-friendly interfaces. Moreover, all empirical evidence from
the dataset needs to be verified in real-world robotic experiments.

##  5 Conclusion

The proposed research aims to establish a pioneering investigation between the
domains of gesture studies, cognitive linguistics, and computational language
modeling. This approach not only identifies prospective models but also
delineates a pertinent dataset. The research design outlines the methodology
for an empirical investigation, promising insights into the depth of LLMs’
comprehension of gestures, with unclear outcomes.

The exploration of various perspectives within the plan underscores the
potential of this study to transcend its immediate context. The results of
this interdisciplinary inquiry may serve as a catalyst for nuanced and
multifaceted investigations. By interweaving insights from multiple
disciplines, this research seeks to contribute substantively to the evolving
discourse on human-AI interaction that rely on multimodal generative AI
systems.

## Acknowledgement

  * •

Figure 1 Image AI generated: Prompt = “Simplistic abstract drawing of a figure
((gesturing)) on white background. clean lines.”, Model = stable-diffusion
(sdxl), Guidance Scale = 7.5, Seed = 154221118039499.

  * •

Figure 1 Gesture Interpretation: Partial answer of Bing’s Copilot.

  * •

Main Text: Formatting, grammar, syntax and writing assistance with OpenAI’s
ChatGPT.

## References

  * Aher et al. (2023) Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. 2023.  Using large language models to simulate multiple humans and replicate human subject studies.  In _International Conference on Machine Learning_ , pages 337–371. PMLR.
  * Ao et al. (2023) Tenglong Ao, Zeyi Zhang, and Libin Liu. 2023.  Gesturediffuclip: Gesture diffusion model with clip latents.  _arXiv preprint arXiv:2303.14613_.
  * Awadalla et al. (2023) Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. 2023.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  _arXiv preprint arXiv:2308.01390_.
  * Bergen (2019) Benjamin K Bergen. 2019.  Do gestures retain mental associations with their iconic origins, even after they become emblematic? an analysis of the middle-finger gesture among american english speakers.  _Plos one_ , 14(4):e0215633.
  * Biswas (2023) Som Biswas. 2023.  Chatgpt and the future of medical writing.
  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022.  Gpt-neox-20b: An open-source autoregressive language model.  _arXiv preprint arXiv:2204.06745_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Cienki and Müller (2008) Alan Cienki and Cornelia Müller. 2008.  _Metaphor and gesture_ , volume 3.  John Benjamins Publishing.
  * Dai et al. (2023) W Dai, J Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi. 2023.  Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023.  _arXiv preprint arXiv:2305.06500_.
  * Gao et al. (2023) Nan Gao, Zeyu Zhao, Zhi Zeng, Shuwu Zhang, and Dongdong Weng. 2023.  Gesgpt: Speech gesture synthesis with text parsing from gpt.  _arXiv preprint arXiv:2303.13013_.
  * Hu (2023) Krystal Hu. 2023.  Chatgpt sets record for fastest-growing user base - analyst note.  _Reuters_.  Accessed on 17 August 2023.
  * Ienaga et al. (2022) Naoto Ienaga, Alice Cravotta, Kei Terayama, Bryan W Scotney, Hideo Saito, and M Grazia Busa. 2022.  Semi-automation of gesture annotation by machine learning and human collaboration.  _Language Resources and Evaluation_ , 56(3):673–700.
  * Lakoff (2008) George Lakoff. 2008.  _Women, fire, and dangerous things: What categories reveal about the mind_.  University of Chicago press.
  * Liu et al. (2022) Emmy Liu, Chen Cui, Kenneth Zheng, and Graham Neubig. 2022.  Testing the ability of language models to interpret figurative language.  _arXiv preprint arXiv:2204.12632_.
  * Matsumoto and Hwang (2013) David Matsumoto and Hyisung C Hwang. 2013.  Cultural similarities and differences in emblematic gestures.  _Journal of Nonverbal Behavior_ , 37:1–27.
  * McNeill (1992) David McNeill. 1992.  _Hand and mind: What gestures reveal about thought_.  University of Chicago press.
  * Mittelberg (2006) Irene Mittelberg. 2006.  _Metaphor and metonymy in language and gesture: Discourse evidence for multimodal models of grammar_.  Cornell University.
  * Putnam et al. (1967) Hilary Putnam et al. 1967.  Psychological predicates.  _Art, mind, and religion_ , 1:37–48.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sidhu et al. (2014) David M Sidhu, Rachel Kwan, Penny M Pexman, and Paul D Siakaluk. 2014.  Effects of relative embodiment in lexical and semantic processing of verbs.  _Acta psychologica_ , 149:32–39.
  * Tamkin et al. (2021) Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. 2021.  Understanding the capabilities, limitations, and societal impact of large language models.  _arXiv preprint arXiv:2102.02503_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Turchyn et al. (2018) Sergiy Turchyn, Inés Olza Moreno, Cristóbal Pagán Cánovas, Francis Steen, Mark Turner, Javier Valenzuela, and Soumya Ray. 2018.  Gesture annotation with a visual search engine for multimodal communication research.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Van Dis et al. (2023) Eva AM Van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi L Bockting. 2023.  Chatgpt: five priorities for research.  _Nature_ , 614(7947):224–226.
  * Varela et al. (2017) Francisco J Varela, Evan Thompson, and Eleanor Rosch. 2017.  _The embodied mind, revised edition: Cognitive science and human experience_.  MIT press.
  * Wicke (2023) Philipp Wicke. 2023.  Lms stand their ground: Investigating the effect of embodiment in figurative language interpretation by language models.  _arXiv preprint arXiv:2305.03445_.
  * Wicke et al. (2023) Philipp Wicke, Lütfi Kerem Şenel, Shengqiang Zhang, Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin, and Hinrich Schütze. 2023.  Towards language-based modulation of assistive robots through multimodal models.  _arXiv preprint arXiv:2306.14830_.
  * Wicke and Veale (2020) Philipp Wicke and Tony Veale. 2020.  The show must go on: On the use of embodiment, space and gesture in computational storytelling.  _New Generation Computing_ , 38(4):565–592.
  * Workshop et al. (2022) BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Wu et al. (2023) Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023.  Bloomberggpt: A large language model for finance.  _arXiv preprint arXiv:2303.17564_.
  * Zhang et al. (2023) Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Zheng and Peng (2022) Yuting Zheng and Jian-E Peng. 2022.  Elan (eudico linguistic annotator).  _RELC Journal_ , 53(2):469–474.