P}(V_{p}-\bar{V})(E_{p,o}-\bar{E}_{o})\right|$

where $O$ is the set of output units and $P$ is the training dataset. $V_{p}$
is the hidden unit’s value (its activation) when the training pattern $p$ was
passed through the network. $\bar{V}$ is the hidden unit’s value averaged over
all training patterns. $E_{p,o}$ is the error at the output unit $o$ on the
training pattern $p$ and $\bar{E}_{o}$ is the error at output unit $o$
averaged over all training patterns.

####  A.1.3 Accuracy Computation

In the experiments contained in this thesis, the primary performance metric is
the accuracy of the neural network’s predictions on a classification task. Let
$C$ be the set of $|C|=c$ classes. Let the output of the neural network be
given by $y$ where $y\in\mathbb{R}^{c}$. After passing the output of the
neural network through the softmax function $\sigma$, we obtain $z=\sigma(y)$
where $z\in\mathbb{R}^{c}$. The accuracy $\tau$ can be computed as follows:

$\tau=\text{argmax}_{i}\ z$

where $i\in\\{1,\ldots,c\\}$.

###  A.2 Activation functions

In my thesis, I am using three different activation functions, namely relu
(Rectified Linear Unit), tanh (hyperbolic tangent), and softmax.

The relu function is a function $\text{relu}:\mathbb{R}\rightarrow\mathbb{R}$:

$\text{relu}(x)=\begin{cases}0&x<0\\\ x&x\end{cases}$

The tanh function is a function $\text{tanh}:\mathbb{R}\rightarrow\mathbb{R}$:

$\text{tanh}(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}\in[-1,1]$

Both the relu and the tanh function can be applied to vectors of real numbers
by applying the function to each of its elements individually.

The softmax function $\sigma$ is defined on a vector of $K$ real numbers and
normalizes that vector into a probability vector,
$\sigma:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$:

$\sigma(z)_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{K}e^{z_{j}}}\in[0,1]$

where $z\in\mathbb{R}^{K}$ and $1\leq i\leq K$.

###  A.3 Neural Network Optimization Algorithms

This section closely follows and paraphrases the paper by Ruder, [2016] which
gives a good overview of different gradient descent optimization algorithms
commonly used for training neural networks.

####  A.3.1 Stochastic Gradient Descent

There are different variations of the standard gradient descent algorithm that
vary in the amount of data that they take in before updating the parameters.

Let $N$ be the number of patterns in the training data, $\eta$ be the learning
rate, $\theta$ be the parameter vector (the vector of all connection weights
in a neural network), $\mathcal{L}_{i}(\theta)$ be the loss for pattern $i$
(given parameter vector $\theta$), then the standard (”batch”) gradient
descent algorithm updates the weight vector in the following way:

$\theta_{t+1}=\theta_{t}-\eta\frac{1}{N}\sum_{i=1}^{N}\nabla\mathcal{L}_{i}(\theta_{t})$

where $t$ indicates the step of the gradient descent optimization.

This computation can be slow and for large datasets even intractable if they
do not fit into memory. We can break down the update rule and update the
parameter vector with every single pattern that we train on. This is called
stochastic gradient descent, which is applied for every pattern
$i\in\\{1,\ldots,N\\}$:

$\theta_{t+1}=\theta_{t}-\eta\nabla\mathcal{L}_{i}(\theta_{t})$

However, this is not very efficient either, because we update the parameter
vector for every single pattern in the dataset. In order to strike a
compromise between batch gradient descent and stochastic gradient descent, one
may use so-called ”mini-batches”, i.e. subsets of the total training data of
size $m$, after each of which the parameters are updated as follows:

$\theta_{t+1}=\theta_{t}-\eta\frac{1}{m}\sum_{i=1}^{m}\nabla\mathcal{L}_{i}(\theta_{t})$

This is called mini-batch gradient descent and it is the algorithm that I am
referring to as SGD (stochastic gradient descent) because this is what the
algorithm is called in Keras, the deep learning framework that I am using for
my code implementations. For all experiments found in this thesis, I used a
mini-batch size of 128.

####  A.3.2 RMS Prop

RMS Prop (Root Mean Square Propagation) is the optimization algorithm that I
used to train most neural networks in this thesis. It deals with some of the
challenges that vanilla gradient descent methods face. RMS Prop belongs to a
family of gradient descent optimization algorithms that use momentum and/or
adaptive learning rates. A more detailed discussion of these methods can be
found in Ruder, [2016]. Herein, I am using RMS Prop without further
discussion.

In RMS Prop, the learning rate is adapted for every single parameter in the
parameter vector $\theta$. The idea is to divide the learning rate for a
weight by a running average of the magnitudes of recent gradients for that
weight [Tieleman and Hinton,, 2012]. This running average is computed by:

$v_{t+1}(\theta)=\gamma\
v_{t}(\theta)+(1-\gamma)\nabla\mathcal{L}_{i}(\theta)^{2}$

where $v_{t}$ is the moving average at step $t$ and $\gamma$ is the momentum
rate, or forgetting factor. The parameter vector is then updated as follows:

$\theta_{t+1}=\theta_{t}-\frac{\eta}{\sqrt{v_{t+1}(\theta_{t})}}\nabla\mathcal{L}_{i}(\theta_{t})$

In my implementations, I am using the recommended values $\gamma=0.9$ and
$\eta=0.001$ [Ruder,, 2016] and [Tieleman and Hinton,, 2012].

###  A.4 Futher Results

Layers | Epochs | Time | Train acc | Test acc
---|---|---|---|---
5 x 750 | 50 / 27 |  266.9s / 493.1s |  100.00% / 97.58% |  97.67% / 96.38%
2 x 512 | 33 / 28 |  92.3s / 123.3s |  99.95% / 98.53% | 97.42% / 97.57%
3 x 850 | 24 / 21 |  133.1s / 284.16s |  99.78% / 98.36% |  97.36% / 96.97%
Table 10: Network performances when trained using forward thinking (left
values) and backpropagation (right values).

## References

  * Bartlett et al., [1999] Bartlett, P. L., Maiorov, V., and Meir, R. (1999).  Almost Linear VC Dimension Bounds for Piecewise Polynomial Networks.  In Kearns, M. J., Solla, S. A., and Cohn, D. A., editors, Advances in Neural Information Processing Systems 11, pages 190–196. MIT Press.
  * Bergstra and Bengio, [2012] Bergstra, J. and Bengio, Y. (2012).  Random Search for Hyper-Parameter Optimization.  The Journal of Machine Learning Research, 13(Feb):281–305.
  * Bianchini and Scarselli, [2014] Bianchini, M. and Scarselli, F. (2014).  On the Complexity of Shallow and Deep Neural Network Cassifiers.  In European Symposium on Artificial Neural Networks, volume 22, pages 371–376.
  * Choromanska et al., [2015] Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., and LeCun, Y. (2015).  The Loss Surfaces of Multilayer Networks.  In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, volume 38 of JMLR: W&CP, pages 192–204. JMLR.org.
  * Conneau et al., [2016] Conneau, A., Schwenk, H., Barrault, L., and Lecun, Y. (2016).  Very Deep Convolutional Networks for Text Classification.  arXiv preprint arXiv:1606.01781.
  * Cortes et al., [2017] Cortes, C., Gonzalvo, X., Kuznetsov, V., Mohri, M., and Yang, S. (2017).  Adanet: Adaptive Structural Learning of Artificial Neural Networks.  In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 874–883. JMLR.
  * Eldan and Shamir, [2016] Eldan, R. and Shamir, O. (2016).  The Power of Depth for Feedforward Neural Networks.  In Conference on Learning Theory, volume 49, pages 907–940.
  * Elsken et al., [2019] Elsken, T., Metzen, J. H., and Hutter, F. (2019).  Neural Architecture Search: A Survey.  The Journal of Machine Learning Research, 20(55):1–21.
  * Fahlman, [1991] Fahlman, S. E. (1991).  The Recurrent Cascade-Correlation Architecture.  In Lippmann, R. P., Moody, J. E., and Touretzky, D. S., editors, Advances in Neural Information Processing Systems 3, pages 190–196. Morgan-Kaufmann.
  * Fahlman and Lebiere, [1990] Fahlman, S. E. and Lebiere, C. (1990).  The Cascade-Correlation Learning Architecture.  In Touretzky, D. S., editor, Advances in Neural Information Processing Systems 2, pages 524–532. Morgan-Kaufmann.
  * Frean, [1990] Frean, M. (1990).  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks.  Neural Computation, 2(2):198–209.
  * Goodfellow et al., [2016] Goodfellow, I., Bengio, Y., and Courville, A. (2016).  Deep Learning.  MIT press.
  * Hanson, [1990] Hanson, S. J. (1990).  Meiosis Networks.  In Touretzky, D. S., editor, Advances in Neural Information Processing Systems 2, pages 533–541. Morgan-Kaufmann.
  * Harvey, [2017] Harvey, M. (2017).  Let’s evolve a neural network with a genetic algorithm — code included.
  * Hassibi et al., [1994] Hassibi, B., Stork, D. G., and Wolff, G. (1994).  Optimal Brain Surgeon: Extensions and Performance Comparisons.  In Cowan, J. D., Tesauro, G., and Alspector, J., editors, Advances in Neural Information Processing Systems 6, pages 263–270. Morgan-Kaufmann.
  * Hassibi et al., [1993] Hassibi, B., Stork, D. G., and Wolff, G. J. (1993).  Optimal brain surgeon and general network pruning.  In IEEE International Conference on Neural Networks, volume 1, pages 293–299. IEEE.
  * He et al., [2016] He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep Residual Learning for Image Recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 1, pages 770–778.
  * Hettinger et al., [2017] Hettinger, C., Christensen, T., Ehlert, B., Humpherys, J., Jarvis, T., and Wade, S. (2017).  Forward Thinking: Building and Training Neural Networks One Layer at a Time.  arXiv preprint arXiv:1706.02480.
  * Hinton, [2012] Hinton, G. E. (2012).  A Practical Guide to Training Restricted Boltzmann Machines.  In Neural Networks: Tricks of the Trade, pages 599–619. Springer.
  * Hornik et al., [1989] Hornik, K., Stinchcombe, M., and White, H. (1989).  Multilayer Feedforward Networks Are Universal Approximators.  Neural Networks, 2(5):359–366.
  * Keras, [2019] Keras (2019).  Simple Deep Neural Network on the MNIST Dataset.
  * Krizhevsky et al., [2017] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2017).  ImageNet Classification with Deep Convolutional Neural Networks.  Commununity of the Association of Computing Machinery, 60(6):84–90.
  * Krogh and Hertz, [1992] Krogh, A. and Hertz, J. A. (1992).  A Simple Weight Decay Can Improve Generalization.  In Moody, J. E., Hanson, S. J., and Lippmann, R. P., editors, Advances in Neural Information Processing Systems 4, pages 950–957. Morgan-Kaufmann.
  * Kumar et al., [2016] Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., and Socher, R. (2016).  Ask Me Anything: Dynamic Memory Networks for Natural Language Processing.  In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pages 1378–1387. JMLR.
  * Larochelle et al., [2007] Larochelle, H., Erhan, D., Courville, A., Bergstra, J., and Bengio, Y. (2007).  An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation.  In Proceedings of the 24th International Conference on Machine Learning, pages 473–480. ACM.
  * LeCun and Bengio, [1998] LeCun, Y. and Bengio, Y. (1998).  Convolutional Networks for Images, Speech, and Time Series.  In Arbib, M. A., editor, The Handbook of Brain Theory and Neural Networks, pages 255–258. MIT Press.
  * LeCun et al., [1998] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al. (1998).  Gradient-Based Learning Applied to Document Recognition.  Proceedings of the IEEE, 86(11):2278–2324.
  * LeCun et al., [1990] LeCun, Y., Denker, J. S., and Solla, S. A. (1990).  Optimal Brain Damage.  In Touretzky, D. S., editor, Advances in Neural Information Processing Systems 2, pages 598–605. Morgan-Kaufmann.
  * LeCun et al., [2012] LeCun, Y. A., Bottou, L., Orr, G. B., and Müller, K.-R. (2012).  Efficient Backprop.  In Neural Networks: Tricks of the Trade, pages 9–48. Springer.
  * Levin et al., [1994] Levin, A. U., Leen, T. K., and Moody, J. E. (1994).  Fast Pruning Using Principal Components.  In Cowan, J. D., Tesauro, G., and Alspector, J., editors, Advances in Neural Information Processing Systems 6, pages 35–42. Morgan-Kaufmann.
  * Levine et al., [2016] Levine, S., Finn, C., Darrell, T., and Abbeel, P. (2016).  End-To-End Training of Deep Visuomotor Policies.  The Journal of Machine Learning Research, 17(1):1334–1373.
  * Littmann and Ritter, [1992] Littmann, E. and Ritter, H. (1992).  Cascade network architectures.  In [Proceedings 1992] IJCNN International Joint Conference on Neural Networks, volume 2, pages 398–404. IEEE.
  * Littmann and Ritter, [1993] Littmann, E. and Ritter, H. (1993).  Generalization Abilities of Cascade Network Architecture.  In Hanson, S. J., Cowan, J. D., and Giles, C. L., editors, Advances in Neural Information Processing Systems 5, pages 188–195. Morgan-Kaufmann.
  * Maass et al., [1994] Maass, W., Schnitger, G., and Sontag, E. D. (1994).  A Comparison of the Computational Power of Sigmoid and Boolean Threshold Circuits.  In Theoretical Advances in Neural Computation and Learning, pages 127–151. Springer.
  * Mezard and Nadal, [1989] Mezard, M. and Nadal, J.-P. (1989).  Learning in Feedforward Layered Networks: The Tiling Algorithm.  Journal of Physics A: Mathematical and General, 22(12):2191.
  * Mozer and Smolensky, [1989] Mozer, M. C. and Smolensky, P. (1989).  Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment.  In Touretzky, D. S., editor, Advances in Neural Information Processing Systems 1, pages 107–115. Morgan-Kaufmann.
  * Nowlan and Hinton, [1992] Nowlan, S. J. and Hinton, G. E. (1992).  Simplifying Neural Networks by Soft Weight-Sharing.  Neural Computation, 4(4):473–493.
  * Poole et al., [2016] Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., and Ganguli, S. (2016).  Exponential Expressivity in Deep Neural Networks Through Transient Chaos.  In Lee, D. D., Sugiyama, M., Luxburg, U. V., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems 29, pages 3360–3368. Curran Associates, Inc.
  * Prechelt, [1997] Prechelt, L. (1997).  Investigation of the Cascor Family of Learning Algorithms.  Neural Networks, 10(5):885–896.
  * Raghu et al., [2017] Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Dickstein, J. S. (2017).  On the Expressive Power of Deep Neural Networks.  In Proceedings of the 34th International Conference on Machine Learning, volume 70 of JMLR: W&CP, pages 2847–2854. JMLR.org.
  * Real et al., [2018] Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. (2018).  Regularized evolution for image classifier architecture search.  arXiv preprint arXiv:1802.01548.
  * Reed, [1993] Reed, R. (1993).  Pruning algorithms - a survey.  IEEE Transactions on Neural Networks, 4(5):740–747.
  * Ruder, [2016] Ruder, S. (2016).  An Overview of Gradient Descent Optimization Algorithms.  arXiv preprint arXiv:1609.04747.
  * Sermanet et al., [2013] Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. (2013).  Overfeat: Integrated Recognition, Localization and Detection Using Convolutional Networks.  arXiv preprint arXiv:1312.6229.
  * Sietsma, [1988] Sietsma, J. (1988).  Neural Net Pruning - Why and How.  In Proceedings of International Conference on Neural Networks, volume 1, pages 325–333.
  * Silver et al., [2018] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018).  A General Reinforcement Learning Algorithm that Masters Chess, Shogi, and Go through Self-Play.  Science, 362(6419):1140–1144.
  * Sjogaard, [1991] Sjogaard, S. (1991).  A Conceptual Approach to Generalisation in Dynamic Neural Networks.  PhD thesis, Aarhus University.
  * Srivastava et al., [2014] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014).  Dropout: A Simple Way to Prevent Neural Networks from Overfitting.  The Journal of Machine Learning Research, 15(1):1929–1958.
  * Telgarsky, [2015] Telgarsky, M. (2015).  Representation Benefits of Deep Feedforward Networks.  arXiv preprint arXiv:1509.08101.
  * Tieleman and Hinton, [2012] Tieleman, T. and Hinton, G. (2012).  Lecture 6.5-rmsprop.
  * Vapnik and Chervonenkis, [2015] Vapnik, V. N. and Chervonenkis, A. Y. (2015).  On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities.  In Measures of Complexity, pages 11–30. Springer.
  * Vinyals et al., [2015] Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015).  Show and tell: A neural image caption generator.  In IEEE Conference on Computer Vision and Pattern Recognition, volume 28, pages 3156–3164.
  * Wang et al., [1994] Wang, Z., Di Massimo, C., Tham, M. T., and Morris, A. J. (1994).  A Procedure for Determining the Topology of Multilayer Feedforward Neural Networks.  Neural Networks, 7(2):291–300.
  * Waugh, [1994] Waugh, S. (1994).  Dynamic learning algorithms.  Department of Computer Science, University of Tasmania.
  * Wu et al., [2016] Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016).  Google’s Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation.  arXiv preprint arXiv:1609.08144.
  * Wynne-Jones, [1992] Wynne-Jones, M. (1992).  Node Splitting: A Constructive Algorithm for Feed-Forward Neural Networks.  In Moody, J. E., Hanson, S. J., and Lippmann, R. P., editors, Advances in Neural Information Processing Systems 4, pages 1072–1079. Morgan-Kaufmann.
  * Yang and Honavar, [1998] Yang, J. and Honavar, V. (1998).  Experiments with the Cascade-Correlation Algorithm.  Microcomputer Applications, 17(2):40–46.
  * Yao, [1999] Yao, X. (1999).  Evolving Artificial Neural Networks.  Proceedings of the IEEE, 87(9):1423–1447.
  * Zhang et al., [2016] Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2016).  Understanding Deep Learning Requires Rethinking Generalization.  arXiv preprint arXiv:1611.03530.
  * Zoeller and Huber, [2019] Zoeller, M. and Huber, M. (2019).  Survey on automated machine learning.  arXiv preprint arXiv:1904.12054.
  * Zoph et al., [2018] Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. (2018).  Learning Transferable Architectures for Scalable Image Recognition.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1.


BibTeX

# Large language models in textual analysis for gesture selection

Laura B. Hensel1,∗, Nutchanon Yongsatianchot2, Parisa Torshizi2, Elena
Minucci1, Stacy Marsella1,2 1University of Glasgow, 2Northeastern University,
∗Corresponding Author<EMAIL_ADDRESS>
(2023)

###### Abstract.

Gestures perform a variety of communicative functions that powerfully
influence human face-to-face interaction. How this communicative function is
achieved varies greatly between individuals and depends on the role of the
speaker and the context of the interaction. Approaches to automatic gesture
generation vary not only in the degree to which they rely on data-driven
techniques but also the degree to which they can produce context and speaker
specific gestures. However, these approaches face two major challenges: The
first is obtaining sufficient training data that is appropriate for the
context and the goal of the application. The second is related to designer
control to realize their specific intent for the application. Here, we
approach these challenges by using large language models (LLMs) to show that
these powerful models of large amounts of data can be adapted for gesture
analysis and generation. Specifically, we used ChatGPT as a tool for
suggesting context-specific gestures that can realize designer intent based on
minimal prompts. We also find that ChatGPT can suggests novel yet appropriate
gestures not present in the minimal training data. The use of LLMs is a
promising avenue for gesture generation that reduce the need for laborious
annotations and has the potential to flexibly and quickly adapt to different
designer intents.

gesture analysis; gesture selection; large language models

††journalyear: 2023††copyright: rightsretained††conference: INTERNATIONAL
CONFERENCE ON MULTIMODAL INTERACTION; October 9–13, 2023; Paris,
France††booktitle: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI
’23), October 9–13, 2023, Paris, France††doi: 10.1145/3577190.3614158††isbn:
979-8-4007-0055-2/23/10††ccs: Human-centered computing HCI theory, concepts
and models

##  1\. Introduction

Human co-speech gestures have been extensively studied. Research has
demonstrated that gestures convey a range of meaning and can have a powerful
impact on face to face interaction (Kendon, 1997; Calbris, 2011; Goldin-Meadow
and Alibali, 2013; McNeill, 1992). Gestures positively impact the speaker’s
persuasiveness as well as an addressee’s comprehension, recall, engagement,
and trust in the speaker (Tversky and Hard, 2009; Jamalian and Tversky, 2012;
Bavelas, 1994). However these impacts are dependent on the particular gestures
being used and the context in which they occur (e.g., Hostetter, 2011).
Broadly speaking, people differ in the degree they use gestures, what types of
gestures they use and the impact those gestures have (Kendon, 2004; McNeill,
2005; Chu et al., 2014; Özer and Göksun, 2020). This is especially clear when
one considers the situation and role of the gesturer. A clinician may use
gestures to establish a safe, trusted place for the interaction with a client
as they discuss difficult issues (Foley and Gentile, 2010). A politician may
seek to be dismissive of opponents (Hall et al., 2016), or arouse fear and
anger (Casasanto and Jasmin, 2010). A comedian may use entirely different
gestures to generate laughter at their own foibles or ridicule others (Seizer,
2011). A lack of gestures or use of inappropriate gestures can appear uncanny
(Thepsoonthorn et al., 2021; Ostermeier, 1997).

This potential power of gestures can be exploited in the creation of virtual
agents. Selecting gestures for virtual agents is, however, a complex problem
due to the multidimensionality of the gesture space. To solve this problem,
researchers have focused on two main approaches. The first approach is to
predict appropriate gestures based on analysis of the virtual human’s
utterances (e.g., Cassell et al., 2004; Lee and Marsella, 2006). Because of a
lack of data sets that include both language and co-speech gestures, work in
this area has sometimes exploited existing language tools (Cassell et al.,
2004; Ravenet et al., 2018; Saund et al., 2019) to analyze utterance content
and suggest suitable gestures. In particular, machine learning derived tools
have been used to analyze prosody and parse rhetorical structure of utterances
(Joty et al., 2015) while ontology tools like Wordnet and Trains (Allen et
al., 2020; Miller, 1995) have been used to analyze lexical and semantic
content. In essence, these approaches aim to generalize example gestures to
semantically related utterances. However, the emphasis is increasingly
shifting to use of state of the art deep learning approaches to generate
gestures directly from the utterance, in part driven by the fact that larger
language+gesture corpora have become available (see (Nyatsanga et al., 2023)
for a comprehensive review).

With any approach, the data that informs gesture selection is critical to the
appropriateness of the output gestures. For example, it may be undesirable for
a virtual human clinician to gesture like a politician, a comedian, or someone
who rarely gestures. Similarly, it may be counterproductive to base the
virtual human’s gestures on an average over these different people and roles.
Tailoring a virtual human’s gestures to the role the virtual human inhabits
and the designer’s intent for the gesture’s impact has potential to greatly
improve the virtual human’s impact (Salem et al., 2013, 2012). Given this
context-dependency of gestures, the central question in gesture generation
becomes what gestural data can realize designer intent.

Here, we explore this question from a large language model (LLM) perspective.
LLMs have the key advantage that they comprise enough data to perform textual
analysis. However, the issue remains how to map that analysis to gestures, in
a way consistent with design intentions. In this paper, we demonstrate that
LLMs, specifically OpenAI’s ChatGPT and GPT-4, are a powerful tool for
selecting appropriate gestures (Brown et al., 2020; AI, 2023). We address the
critical issue of design intention through the use of speaker annotations (see
(Kipp and Martin, 2009) for related work) consistent with design intentions to
achieve mappings from the the utterance to gesture.

We illustrate the use of LLMs for context-relevant gesture selection based on
several examples, starting with the popular but less recent model BERT
(Bidirectional Encoder Representations from Transformers (Devlin et al.,
2019)) before moving on to explore GPT in more detail. These examples
demonstrate the potential power of the approach to not only select individual
gestures, but also suggest novel gestures outside the data the system was
provided. With this, we aim to demonstrate the transformative potential of
LLM’s for gesture generation research. However, we also note that the
evaluation of these gestures is still a fundamental challenge in the field
(Wolfert et al., 2021; Saund and Marsella, 2021) and lay out a road map to
formally developing the use of LLM’s for gesture generation.

##  2\. Related Work

Generating co-speech gestures faces two distinct challenges: The specification
problem and the animation problem (Neff, 2016). While the specification
problem deals with what gesture should be generated, the animation problem is
concerned with how to achieve the animation of that gesture. In this paper, we
focus on the specification problem. For a review of the work on the animation
problem see (Nyatsanga et al., 2023). The specification problem can be
approached with rule-based or data-driven approaches which the following
sections discuss.

###  2.1. Rule-based gesture generation

As the name suggests, rule-based approaches rely on a knowledge base along
with inference procedures to determine gesture selection, where gestures are
often chosen from a gesture dictionary, i.e., a corpus of gestures (Cassell et
al., 2000). This approach frequently relies on manual annotations of gestures
and co-speech. These annotations are then used to curate a gesture lexicon as
well as synthesize rules for gesture selection. For example, the Behavior
Expression Animation Toolkit relies on heuristics derived from extensive
behavioral research to select pre-recorded gestures based on linguistic and
contextual information (Cassell et al., 2001). The system uses a bespoke
language tagging module and WordNet (Miller, 1995) to automatically extract
this information from text. Similarly, Marsella et al. (Marsella et al., 2013)
determined the communicative function of utterances using a collection of
machine learning based prosodic and syntactic analyses as well as semantic
database tools. The communicative functions were then mapped to gestures based
on rules codified in an indexed gesture dictionary.

In contrast, Ravenet et al. (Ravenet et al., 2018) based their heuristics on
image schemas – recurring cognitive processes that aid in establishing meaning
(Kendon et al., 1980; McNeill, 1985). Example image schemas include ’big’,
’full’, or ’object’. The system uses WordNet to identify synonyms for in-text
words that correspond to image schemas which they then map to gestures via a
gesture dictionary compiled from the literature (Ravenet et al., 2018).

While such rule-based gesture generation systems offer a high degree of
flexibility when it comes to adjusting rules for gesture-selection, and
thereby changing designer intent, they have several key limitations. Firstly,
their reliance on gesture dictionaries, i.e., pre-determined sets of gestures,
limits the range of gestures that can be produced. As a result, the gestures
lack diversity and, by extension, these systems typically only produce a
limited number of different gestures for the same speech input. Secondly,
generating such extensive heuristics and corresponding gesture dictionaries is
time consuming, relies on expert-knowledge and is therefore not scaleable.
Because of these limitations, a growing body of work is focusing instead on
data-driven gesture generation.

###  2.2. Data-driven gesture generation

Early approaches to data-driven gesture generation often relied on
probabilistic mapping from semantic content to gesture type (Neff et al.,
2008; Ishi et al., 2018). For example, several approaches used annotated
gesture data to produce gesture profiles to which they could apply statistical
analyses to ultimately predict plausible gestures for new, annotated input-
text (Kipp, 2003; Neff et al., 2008). Similarly, Bergmann and Kopp (Bergmann
and Kopp, 2009) built a Bayesian network to determine whether and what type of
gesture should be generated, based on variables such as the communicative
action the gesture aims to achieve. While these approaches are more flexible
than rule-based approaches, they still often rely on laboriously annotated
data and a relatively small set of variables to apply the statistical analyses
to.

Because of this, data-driven gesture generation increasingly uses machine
learning techniques, and specifically deep learning (Nyatsanga et al., 2023;
Ahuja et al., 2020). Many of these approaches use encode-decoder recurrent
neural networks that learn the mapping from utterance text to gestures (Yoon
et al., 2020, 2019; Ao et al., 2022). However, these approaches often still
rely on gesture lexica (Ao et al., 2022) and in all cases, these data-driven
approaches are dependent on the data they are trained on, with limited data
sets available. Additionally, much of the semantic richness is often lost
(Nyatsanga et al., 2023).

###  2.3. Large Language Models

In contrast, the extensive pre-training of LLMs lends itself as a natural next
step in automatic gesture generation. Currently, there are multiple available
LLMs, such as OpenAI’s GPTs, Meta’s LLaMA, Google’s Bard, and Anthropic’s
Claude, to name a few (for a recent review, see (Yang et al., 2023)) (Brown et
al., 2020; Touvron et al., 2023; Google, 2023; Anthropic, 2023; AI, 2023). One
LLM that has found much traction in the research community is BERT (Devlin et
al., 2019). BERT consists of transformer encoder layers and is designed for
language modeling and next-sentence prediction. It has been shown to
powerfully predict semantic similarities between word- and sentence-pairs
(Mutinda et al., 2021; Peinelt et al., 2020) and this makes it a potentially
valuable tool for gesture generation as mappings from words to gestures can be
generalized over synonyms (Fares et al., 2022; Kucherenko et al., 2020).
However, BERT does not have in-built explainable AI, limiting its application
to realizing designer intent. For this reason, we explore BERT only briefly
before we primarily focus on models from OpenAI.

We chose OpenAI’s models for two key reasons: First, they provide easy access
API and second, at the time of writing, GPT-4 is the most recent and powerful
model (Peng et al., 2023). Specifically, we use two of OpenAI’s Generative
Pre-trained Transformer models or GPT: ChatGPT (gpt-3.5-turbo) and GPT-4
(Brown et al., 2020; AI, 2023). GPTs are an autoregressive decoder-only
transformer-based large language models. They are pretrained on large data
sets to predict next words and fine-tuned using Reinforcement Learning with
Human Feedback to make the models align to human preferences and common tasks
(Ouyang et al., 2022). One of the key capabilities of these LLMs in relation
to this work is in-context learning – their ability to learn a new task given
only a few examples without having to fine-tune the model (Brown et al., 2020;
Xie et al., 2021).

Recent work has begun to demonstrate the ability of GPTs’ to approximate human
cognitive processes. For example, Binz and Schulz (Binz and Schulz, 2023)
studied GPT-3’s decision-making, information search, and causal reasoning
using common cognitive psychological tests such as the conjunction and base-
rate fallacies. They found that GPT-3 can solve these problems at a similar or
better than human level but fall for some common human biases. Similarly,
Kosinski (Kosinski, 2023) examined Theory of Mind in LLMs using standard
false-belief tasks and found that ChatGPT and GPT-4 can solve most of these
tasks. Similarly, Bubeck et al. (Bubeck et al., 2023) tested GPT-4 in various
domains including mathematics, coding, medicine, law, and psychology,
including Theory of Mind and emotion understanding. They showed that GPT-4 can
perform these tasks well and outperforms ChatGPT. Taken together, this body of
works shows GPTs’ promise for applications that normally require human
cognitive input.

One such application is gesture generation. GPTs’ extensive textual analysis
training has the potential to yield gestures that are semantically rich and
flexible while not relying on manual annotations. The first foray into this
approach used ChatGPT to identify the intent of an utterance, based on this
their system then selected a gesture based on a pre-compiled gesture
dictionary (Gao et al., 2023). The work demonstrated that ChatGPT can
successfully aid in gesture generation but did not yet fully realize its
potential which lies in ChatGPT’s ability to a) suggest new gestures with
little input and b) adapt gestures to designer intent.

###  2.4. Gesture generation driven by designer intent

Work that aimed to realize designer intent has often focused on producing
gestures for a specific novel target, i.e., a single gesturer. For example,
Ahuja et al. (Ahuja et al., 2022) were able to generate gestures for a novel
target by adapting a pre-trained source model based on just two minutes of
target data. Their model did this by learning the difference in the crossmodal
grounding, i.e., binding between language and physical gestures, between the
source model and the target data. In contrast, other work has focused on
learning different speaker styles that can then be used in a zero-shot style
transfer to generate gestures for targets not present in training data (Fares
et al., 2023; Ghorbani et al., 2023). These approaches are often very
efficient but are also tailored to generate gestures for specific targets.

##  3\. examples

One approach to applying modern LLMs to gesture generation is to use them to
replace the generalization from examples. Earlier rule-based work discussed in
previous sections relied on analyses using a range of ontology based tools
(Grady, 1997; Miller, 1995; Allen et al., 2020). Such tools can be used to
analyze text to find similarities with, and thereby generalize from, a set of
annotated text-gesture pairings. For example, consider the phrase ”important
concept”. A rule based ontological approach may codify the metaphoric mapping
from ”important” to big and the semantic mapping from ”concept” to idea which
in turn can be metaphorically mapped to a container gesture. As the following
example shows, language models have the potential to replace this
codification.

###  3.1. BERT

The pre-trained BERT model can realize a mapping from text to gesture directly
through its cosine similarity measures, even without re-training. Given some
phrase-gesture example mappings, derived for example from some annotations of
a speaker, and a novel utterance, BERT can measure the cosine similarity
between each example utterance and a novel utterance. Additionally it provides
a metric for this similarity which can be used to choose potential alternative
gestures. Consider the following example where BERT gives similarity measures
between important concept and five phrases of varying relatedness.

Seed Phrase: important concept

  * •

big idea correlation: 0.861

  * •

great thought correlation: 0.81

  * •

red napkin correlation: 0.239

  * •

weak idea correlation: 0.469

  * •

silly idea correlation: 0.674

We see a strong relation between the seed important concept and big idea as
well as great thought. As expected the relation to red napkin is the smallest
as they are semantically unrelated. The seed’s relation with weak idea as one
might expect is somewhat stronger but still weaker than big idea and great
thought. In contrast, the seed’s relation to silly idea is slightly stronger
than weak idea, arguably because silliness has both positive and negative
attributes. Critically, however, the exact reason for this lower correlation
is hidden from the user at this level of probing the model.

In short, BERT’s pre-trained model could substitute for the lexical, semantic,
metaphoric analyses discussed above. One would expect BERT to realize a much
broader generalization than an ontological approach, though this depends on
the boundary condition set on the cosine similarity. A key benefit of BERT is
that it provides a clear, overall metric on this mapping as the above examples
show. However, the derivation of this metric is opaque and outside the
designer’s control.

In contrast, the ontology based approach is more transparent as to how this
mapping of a novel utterance to gesture suggestion is realized. The semantic,
lexical and metaphoric elements are well-defined, which potentially supports
finer grain designer manipulations. For example, specific analysis elements
could be eliminated or discounted, such as mapping ”important” to size, or
specific analyses could be generalized, such as generalizing abstract concepts
to any concept so any object, abstract or physical could be conveyed by a
container.

Whereas BERT is arguably similar to prior approaches, ChatGPT, specifically
its prompting ability, provides another, novel approach to achieving
generalization from example annotations while allowing for designer control
through the prompting mechanism.

(a) Anything

(b) Besides

Figure 1. Ideational Unit showing a container gesture depicted a protected
space followed by a sweep gesture suggesting something discarded from the
container

###  3.2. ChatGPT and GPT-4

To illustrate the power of LLMs for gesture selection, we prompted ChatGPT and
GPT-4 with text-gesture pairings which in practice would be drawn from
annotations of an actual speaker or speakers. In the following example, we
intentionally used opaque terms for the gesture that ChatGPT could not
interpret, such as ”lcg” meaning ”large container gesture” and ”ng” for
”negation gesture”. Based on just three example prompts when given the phrase
”it is an excellent idea”, ChatGPT responded with ”lcg”, shown in bold
italics.

#### ChatGPT

  * •

Example prompt: It is an important idea. lcg

  * •

Example prompt: It is a silly idea. ng

  * •

Example prompt: It is a big idea. lcg

  * •

Prompt: It is a excellent idea.

  * •

Output: lcg

This method is very similar to the BERT approach – it identifies the
similarity between ”important”, ”big”, and ”excellent” and produces the
corresponding gesture. However, if we supply gesture descriptions that are
interpretable to ChatGPT, then it produces novel gestural suggestions based on
its interpretation of the gestural prompts. The following example demonstrate
this.

#### ChatGPT

  * •

Example prompt: It is a important idea. large container gesture

  * •

Example prompt: It is a bad idea. no gesture

  * •

Example prompt: It is a big idea. large container gesture

  * •

Prompt: It is an excellent thought.

  * •

Output: thumbs up gesture

ChatGPT suggests a gesture that is appropriate for illustrating the utterance
but was never prompted: ”thumbs-up gesture”. Such novel suggestions raise the
question as to the degree to which the prompting is reflecting an underlying
relation between utterance and gesture. To illustrate the depth of GPT’s
ability in this regard, consider an utterance sequence adapted from an
interaction between a clinician and woman suffering from PTSD. This utterance
and gestural sequence is adapted111We did not use the exact utterance on one
hand to make the context explicit, i.e., a clinician talking to a client, and
additionally to remove dysfluencies (Lhommet and Marsella, 2014). from an 90
minute corpus in which a real world clinician is going through a PTSD protocol
with a woman actor playing the role of a client suffering from PTSD. She is
talking to the clinician because of her husband who wants her to change
behavior. A key moment in he interaction is as follows:

The clinician is asking a client whether (1) there is anything the client
wants to work on, adding the proviso of (2) besides what her husband wants.
She uses the following gestures: As she says says (1) ”anything”, she performs
a container gesture suggesting a set of all the possible topics they could
discuss. Specifically, this container gesture depicts a domed region of space,
perhaps suggesting a protected space (see Figure 1). Next, she clarifies what
topic she does not want to discuss, (2; “besides what he wants”) using a
gesture that sweeps to the right, suggesting removing that topic from the
container formed by the previous gesture. The ”he” is a reference to the
client’s husband. See Figure 1. These gestures constitute what Calbris
(Calbris, 2011) calls an ideational unit, a sequence of related gestures. Such
gestures tend to set up gestural space in a consistent fashion, such as
depicting sets of abstract things and then removing or putting items into the
set.

To explore GPT’s ability to form ideational units, we first prompted ChatGPT
to provide an explanation for the use of the container gesture in the
clinician example. This yielded the following explanation that is consistent
with a prior analysis of the clinician’s gesture done by gesture researchers
(DeVault et al., 2014):

#### ChatGPT

  * •

Prompt: The clinician said we can talk about anything. She used a container
gesture when she said anything.

  * •

ChatGPT: The clinician’s use of a container gesture when saying ”anything” may
suggest that she wants you to feel safe and comfortable to discuss any topic
that comes to mind, but also to keep those topics contained within the
therapeutic space. The container gesture is a nonverbal cue that can signal a
sense of containment and boundaries, creating a safe space for exploration and
self-discovery.

Thus, ChatGPT captured the relation of the container to the possible topic.
The response also unexpectedly captured the notion of a protected space which
aligns with the specialized dome-shaped container gesture used by the
clinician which we did not include in the prompt. GPT-4 yielded similar
results but did not capture the subtlety of a ”safe space” as explicitly. It
instead suggested a container gesture to show inclusiveness and make the
patient comfortable.

Importantly, the following example shows that GPT-4 can provide a good
prediction of what gesture the clinician could perform next consistent with
the gesture sequence the clinician actually used (ChatGPT yielded similar
results) . This suggests an ability to construct an ideational unit.

#### GPT-4

  * •

Prompt: The clinician said ”We can talk about anything” and then she added
”besides what your husband wants us to talk about.” She used a container
gesture to illustrate ”talk about anything.” To illustrate ”except what your
husband wants us to talk about”, what gesture might she use after the
container gesture.

  * •

GPT-4: After using the container gesture to illustrate ”talk about anything”,
the clinician could use a swiping or brushing away gesture with one hand to
represent ”except what your husband wants us to talk about”. This gesture
would visually convey the idea of removing or excluding a specific topic from
the otherwise open and inclusive container of conversation topics.

Finally, we prompted GPT-4 to visualize the entire action:

  * •

Prompt: Can you visualize it in some way?

  * •

GPT-4: Sure! Here’s a step-by-step visualization of the gestures:

    1. (1)

The clinician starts by using a container gesture to illustrate ”talk about
anything.” She holds her hands in front of her, palms facing each other, and
forms an imaginary circle or container in the air.

    2. (2)

To illustrate ”besides what your husband wants us to talk about,” she then
moves one hand, palm facing down, from one side of the imaginary container to
the other, as if swiping or brushing away a specific topic.

    3. (3)

The clinician’s facial expression could also change to emphasize the
exclusion, such as raising an eyebrow or giving a slight smile to show that
she’s aware of the exception and is still open to discussing other topics.

  * •

This combination of gestures and facial expressions would help to visually
convey the message that all topics are open for discussion, except for the
specific topic the husband wants them to talk about.

Note GPT-4 has gone beyond the hand gestures to suggest facial actions. These
examples illustrate that there is important, and often surprising, potential
for gesture selection and analysis in LLMs. The next sections explore this
potential more systematically.

##  4\. Approach

In this section, we detail two approaches to investigating the use of LLMs to
a) predict gestures from annotated examples and b) suggest gestures without
example prompts. Note that these do not form an exhaustive exploration of
using LLMs for gesture selection but are aimed at giving initial insight into
the power of these tools. In the discussion, we set out a road map for
developing these approaches and the use of LLMs in gesture generation. Note
that all analyses in this paper are based on GPT versions from March 2023.

###  4.1. Data collection

To demonstrate GPT’s potential for gesture selection, two researchers jointly
annotated the first four minutes of Barack Obama’s 2020 speech to the
Democratic National Convention. Specifically, we focused on three gesture
categories that were easily identifiable and semantically meaningful: sweep,
span, and container. In total, we identified thirty-seven such gestures. We
further distinguished between six unique physical gesture types: span,
container, and four types of sweeps with different orientations of the palm
(up, down, in, and forward). We also classified these gestures in terms of the
semantic meaning conveyed, such as temporal span, negative sweep, and
enumeration sweep. The final dataset contained seventeen unique semantic
gestures (semantic gesture description + gesture category) and fifteen unique
semantic gesture descriptions. Code and data are available at:
https://osf.io/c82tq/?view_only=4e0c9c1923cc432abd2c82d0ed8f05c8

###  4.2. GPT gesture prompting

For this work, we used ChatGPT and GPT-4. In all of our experiments, we set
the temperature parameter to 0, making the outputs mostly deterministic to
maximize the replicability of our results. All other parameters were set to
default.

To elicit gesture suggestions, we prompted ChatGPT and GPT-4 in the following
ways. First, we provided each model with a number of examples (2, 4, 6, or 33,
i.e., leave-one-out) for each gesture category. For example, we provided two
container gestures, two sweep gestures, and two span gestures. Each prompt was
structured in three parts:

  1. (1)

The following contextual statement:

> Barrack Obama is giving a speech at the Democratic National Convention.

  2. (2)

Examples of gestures Obama used. Each example consisted of three elements: The
segment of the speech the utterance occurred in, the utterance associated with
the gesture, and the gesture. For example:

> He said I want to talk as plainly as I can about the stakes in this election
> because what we do these next 76 days will echo through generations to come.
> When he said these next 76 days, he used the following gesture: span.

  3. (3)

The target utterance excluding the gesture. The models filled in the gesture
at this point.

We investigate four gesture specifications: Gesture category (span, sweep,
container), physical gesture descriptions (e.g., sweep with palm facing
up/down), semantic gesture descriptions (e.g., negative sweep), and semantic
description only (e.g., negative).

The semantic specification in particular yielded a variety of gestures. We
summarise an illustrative selection of these in Table 1 for the reader’s
convenience.

Original gesture |  ChatGPT |  GPT-4
---|---|---
inclusive span |  inclusive span/sweep |  inclusive span/sweep, compare and contrast
temporal span |  temporal span, spatial span, forward span, positive span |  temporal span, negation span, counting on fingers or tapping on wrist, forward sweep, guiding motion, continuation, positive sweep, process sweep
positive sweep |  upward sweep, emphatic sweep, temporal span |  upward sweep
negative sweep |  negative/negation span/sweep, inclusive span, negative head shake |  negative sweep, downward sweep, negation span, negation or dismissive wave, shrug or open palm, uncertain span
Table 1. ChatGPT and GPT-4 gesture suggestions for semantic specifications

Finally, to explore GPT’s ability to suggest appropriate gestures without any
additional user input, we input each utterance prompt without gesture
examples. An example includes:

> Barrack Obama is giving a speech at the Democratic National Convention. He
> said I want to talk as plainly as I can about the stakes in this election
> because what we do these next 76 days will echo through generations to come.
> When he said these next 76 days, what gesture did he use?

##  5\. analysis

To explore LLMs’ performance at predicting context-specific gestures, that is,
gestures produced by Barack Obama during his speech, we first calculated
prediction accuracies for the four gesture specifications (gesture type,
physical gesture description, semantic gesture, semantic only) and for each
level of example numbers (2, 4, 6, and leave-one-out). Note, that for semantic
specifications, we based this analysis on the first part of the semantic
descriptor (e.g., ’negat’, ’inclu’) to avoid bias from slight deviations in
spelling. Next, we measured pairwise cosine similarity between each original
gesture and GPT-4’s suggestions based on word embeddings derived using BERT
(ChatGPT excluded for brevity, results are similar). This analysis accounted
for cases in which GPT’s description of the gesture differed from the
annotation, therefore appearing inaccurate, but nevertheless showed high
semantic similarity. Figure 2 shows the results.

Specifically, in the top row, each plot shows the prediction accuracy (y-axis)
for each level of examples (x-axis), where ’LOO’ denotes leave-one-out, for
ChatGPT (purple) and GPT-4 (yellow), separately. The dotted line indicates
chance performance (1/number of unique gesture types of that category in the
dataset). In the bottom row, each plot shows the distribution of pairwise
cosine similarities, ranging from 0 (very different) to 1 (very similar), for
each level of examples (x-axis) for GPT-4.

Figure 2. Prediction performance for four gesture specifications. Top:
Prediction accuracy (percent correct). Bottom: Cosine similarity between true
and predicted gesture (GPT-4).

Results show that both models predicted the correct gesture category and the
correct physical gesture description with above chance accuracy when provided
with four or more example prompts. Moreover, both models correctly predicted
gesture types with semantic description and on only semantic descriptions with
above chance accuracy from as little as two example prompts. Across all four
gesture specifications, increasing the number of example prompts improved
prediction performance and we observed the highest performance during gesture
category and semantic description prediction. However, two examples appear to
be too few to predict specific physical gestures. Instead, the models
frequently generated gesture descriptions that did not map directly onto the
gesture types we examined, thus causing the low performance. Notably, GPT-4
outperformed ChatGPT when predicting gesture categories and physical
descriptions but not when predicting specifications involving semantic
information such as ’negative’ or ’inclusive’.

Figure 3 shows that misclassifications were primarily driven by confusions
between span and sweep gestures. GPT-4 performed particularly poorly in this
regard (shown in Figure 3). This indicates that the models used these gestures
interchangeably.

Figure 3. Confusions of gesture category prediction by GPT-4.

Finally, to explore GPT-4’s ability to generate appropriate co-speech gestures
without any example input, we categorized the models’ gesture output when no
examples were provided based on its appropriateness. Specifically, we applied
one of the following labels to each output gesture: Similar (to the original
gesture), Different appropriate (the gesture was different but still
appropriate in the context of the speech), Different inappropriate (the
gesture was different and not appropriate in the context of the speech), No
gesture (the model did not supply a gesture). Two researchers categorised all
37 output gestures of GPT-4, for brevity, and discussed any ambiguous cases
until an agreement was reached. Figure 4 shows the resulting appropriateness
levels. Red denotes inappropriate suggestions.

Figure 4. Appropriateness of gestures suggested by GPT-4 without example
prompts.

Results showed that in 43.2% of cases GPT-4 suggested a gesture similar to the
true gesture. Importantly, when the produced gesture was not similar to the
true gesture, in most cases (43.2%), the novel gesture was nevertheless
appropriate in the context of Obama’s speech. We found GPT-4 only produced
inappropriate gestures in 13.5% of cases.

Notably, the model not only made appropriate suggestions for gestures but
frequently supplied relevant and appropriate justifications for these
suggestions. For example, GPT-4 described the gesture for ’unleashed’ as
”releasing something or setting free”. This description very closely mirrored
the actual gesture observed which was a quick motion of the right hand from
the palm facing toward the body to the palm facing upward and the hand moving
forward.

##  6\. Discussion

Here, we illustrated several key benefits of using LLMs for gesture selection.
We showed that LLMs (1) can predict gestures used in a real-world context with
above chance accuracy from as few as two example prompts and (2) can
generalize over annotations to suggest novel, previously unseen gestures that
are context-appropriate. Combined, these results demonstrate a key benefit of
using LLMs for gesture selection – designer control, which is enabled by the
selection of the speaker and context. For example, if the design goal for a
virtual human is a clinical application where nonverbal behavior can play a
critical role (Hall et al., 1995), the annotation of a trained clinician with
highly effective gesturing skills interacting in a clinical session provides
an ideal seed. Indeed, the clinician example highlighted the LLM’s powerful
ability to adapt to specific contexts as it appeared sensitive to the clinical
context when providing an interpretation of the clinician’s gestures.

Note that we provided the models with very few example prompts and required
outputs that matched relatively strict criteria (e.g., ’sweep’, ’palm down
sweep’, ’negat sweep’, ’negat’; depending on gesture specification).
Additionally, the differentiation between spans, sweeps, and containers is
nuanced because each suggests a region in space. This further adds to the
difficulty of the task. Finally, the models were not fine-tuned to achieve
gesture selection. Taken these challenges into account, the difficulty of the
task should not be underestimated and the observed prediction accuracies of
well above chance represent an impressive performance. Moreover, both models
not only predicted gesture types and physical gesture descriptions but also
semantic gesture specifications. This is especially impressive as we did not
provide the models with information about different gesture types or how they
relate semantically. Nevertheless, the models were capable of suggesting
plausible gestures.

However, a key challenge to applying LLMs to gesture generation is the
annotation process itself. For one, manual annotation is laborious.
Additionally, designing a gesture naming scheme can be difficult due to the
wide variety of real-world gestures and the lack of an agreed upon coding
system for gestures, unlike the facial action coding system used in facial
expression research(Ekman and Friesen, 1978). Any coding scheme used needs to
reliably distinguish the speaker’s gestures and there can be significant
individual and situational differences (Hostetter, 2011).

Additionally, coding schemes need to distinguish physical properties across
semantically related sequences of gestures, i.e., ideational units(Calbris,
2011). As discussed earlier, in an ideational unit, the physical properties of
a gesture can be carried over to subsequent gestures. For example, Obama uses
a beat gesture with a precision grip to emphasize a specific point and then
holds that prevision grip as he does a sweep gesture to illustrate a related
issue. The precision grip is not intrinsic to this sweeping gesture but rather
is carried over from the previous beat gesture. During annotation, the
annotator must ideally distinguish such carryovers of form.

These shortcomings can be circumvented by not using annotations but instead
prompting LLMs with just the context and utterance. However, the unconstrained
gestural suggestions provided by the models can be hard to parse by machine
making automated gesture generation on this basis difficult. Another approach
is automatic annotation which would minimize laborious user input. However,
this would also cede some degree of designer control. On the upside, we showed
that leveraging an LLM’s powerful generalization capability does not require a
large amount of annotations. The important requirement that remains is to have
sufficient annotations to span the speaker’s gesture repertoire.

Another important issue is timing, specifically what parts of an utterance are
accompanied by a co-speech gesture. Gestures are often associated with what
the speaker seeks to emphasize (Clough and Duff, 2020). As a result, altering
which part of an utterance is emphasized by a gesture can transform the
meaning conveyed. It is therefore critical for co-speech gesture generation to
derive emphasis information about the utterance. A common approach to this is
to use prosodic cues such as pitch and pauses (Guellaï et al., 2014; Fares et
al., 2023). However, this presumes that the spoken utterance that is driving
gesture generation includes prosodic cues appropriate for the context in which
the utterance occurs. A more flexible method is discourse analysis. For
example, discourse analysis can identify the rheme of the utterance. This is
the part of the utterance commenting on and providing new information about
the topic or background of the conversation, what is referred to as the theme.
The rheme tends to be more associated with gesturing (McNeill, 1992; Cassell
et al., 2004). It is likely that LLMs can also be leveraged to identify
rhemes.

However, in addition to temporal disconnects between gesture and speech, co-
speech gestures often convey additional information that is not transmitted by
the utterance at all. For example, in his speech, Obama says ”I have sat in
the Oval Office with both of the men who are running for president”,
accompanying the ’oval office’ with a sweep of his right hand. This may simply
function as an emphasis but, equally, alludes to the length of time that he
has spent in that office – 8 years – and the experience he has therefore
accumulated. It is a temporal sweep not conveyed by the semantic content of
the speech and thus difficult to identify not only for humans but also for
machine approaches.

Finally, the models used in this paper are proprietary and not open source. As
a result, any update to the models may change the results without recourse for
the researcher. This poses an issue for both reproducibility and reliability
of any applications based on these models.

It is clear then from the work in this paper that LLMs hold great potential
for gesture selection and, with that, generation. Yet, a number of challenges
remain to be addressed. In the following we set out our current research
efforts to formally and systematically explore the use of LLMs for gesture
selection and implementation for gesture generation.

  1. (1)

Expansion of analysis to several types of speakers, including, for example,
politicians, psychotherapists, and orators.

  2. (2)

Annotation of these videos, going beyond the restricted set of gestures used
in this paper and instead annotating all gestures.

  3. (3)

Expansion of the types of prompts used to explore a) which prompts most
effectively aid GPT in suggesting gestures and b) the minimal approach
necessary to elicit appropriate and interpretable gestures.

  4. (4)

Extension of analysis of the gestures produced to include rheme, theme, and
ideational units.

  5. (5)

Evaluation of the appropriateness of the suggested gestures using human
observer studies in addition to expert opinion.

  6. (6)

Implementation of gesture generation by extracting gestures selected by GPT,
matching to a gesture dictionary, and generating gestures for virtual human
avatars.

##  7\. Conclusion

In this paper we showcased the use of LLMs to realize a generative mapping
between verbal utterances and accompanying co-speech gestures. We explored
approaches to using LLMs for gesture selection based on minimal to no example
prompts. LLMs consistently suggested context-appropriate gestures. While this
approach largely relies on laborious manual annotation, GPT required very few
annotated examples to suggest suitable gestures. Nevertheless, it’s important
to note that the effectiveness of these gesture suggestions hinges on
annotations derived from performances by a skilled speaker in a context and
role that mirror the designer’s intent for a virtual agent. Throughout our
exploration of LLMs in this context, we not only highlighted their potent
ability to select gestures but also underscored the challenges that still need
to be surmounted for the complete implementation of LLMs for gesture
generation. We outlined our ongoing research endeavors aimed at overcoming
these obstacles.

In conclusion, we view this novel approach as a valuable and versatile tool
for gesture researchers to use. However, we recognize that its suitability and
alignment with goals may vary, as it either necessitates annotation (when
example prompts are used) or interpretation (when example prompts are absent).
We nevertheless believe that it will be a fruitful avenue for many gesture
researchers.

##  8\. Acknowledgements

The work in this article has been supported by EPSRC Grant Number EP/T021136/1
(LBH, SM) and UKRI CDT in Socially Intelligent Artificial Agents, Grant Number
EP/S02266X/1 (EM).

## References

  * (1)
  * Ahuja et al. (2020) Chaitanya Ahuja, Dong Won Lee, Ryo Ishii, and Louis-Philippe Morency. 2020.  No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 1884–1895.  https://doi.org/10.18653/v1/2020.findings-emnlp.170
  * Ahuja et al. (2022) Chaitanya Ahuja, Dong Won Lee, and Louis-Philippe Morency. 2022.  Low-Resource Adaptation for Personalized Co-Speech Gesture Generation. In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, New Orleans, LA, USA, 20534–20544.  https://doi.org/10.1109/CVPR52688.2022.01991
  * AI (2023) Open AI. 2023.  GPT-4 Technical Report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Allen et al. (2020) James Allen, Hannah An, Ritwik Bose, Will de Beaumont, and Choh Man Teng. 2020.  A broad-coverage deep semantic lexicon for verbs.  _arXiv preprint arXiv:2007.02670_ (2020).
  * Anthropic (2023) Anthropic. 2023.  Claude.  https://www.anthropic.com/product
  * Ao et al. (2022) Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. 2022.  Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings.  _ACM Transactions on Graphics_ 41, 6 (Nov. 2022), 209:1–209:19.  https://doi.org/10.1145/3550454.3555435
  * Bavelas (1994) Janet Beavin Bavelas. 1994.  Gestures as part of speech: Methodological implications.  _Research on language and social interaction_ 27, 3 (1994), 201–221.
  * Bergmann and Kopp (2009) Kirsten Bergmann and Stefan Kopp. 2009.  Increasing the expressiveness of virtual agents: autonomous generation of speech and gesture for spatial description tasks. In _Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 1_. International Foundation for Autonomous Agents and Multiagent Systems, 361–368.
  * Binz and Schulz (2023) Marcel Binz and Eric Schulz. 2023.  Using cognitive psychology to understand GPT-3.  _Proceedings of the National Academy of Sciences_ 120, 6 (2023), e2218523120.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023\.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ (2023).
  * Calbris (2011) Geneviève Calbris. 2011.  _Elements of meaning in gesture_. Vol. 5.  John Benjamins Publishing.
  * Casasanto and Jasmin (2010) Daniel Casasanto and Kyle Jasmin. 2010.  Good and Bad in the Hands of Politicians: Spontaneous Gestures during Positive and Negative Speech.  _PLOS ONE_ 5, 7 (July 2010), e11805.  https://doi.org/10.1371/journal.pone.0011805 Publisher: Public Library of Science.
  * Cassell et al. (2000) Justine Cassell, Matthew Stone, and Hao Yan. 2000.  Coordination and context-dependence in the generation of embodied conversation. In _Proceedings of the first international conference on Natural language generation - Volume 14_ _(INLG ’00)_. Association for Computational Linguistics, USA, 171–178.  https://doi.org/10.3115/1118253.1118277
  * Cassell et al. (2001) J Cassell, H. Vilhjalmsso, and T. Bickmore. 2001.  BEAT: The behavior expression animation toolkit. In _Proceedings of ACM SIGGRAPH_. 477–486.
  * Cassell et al. (2004) Justine Cassell, Hannes Hogni Vilhjalmsson, and Timothy Bickmore. 2004.  Beat: the behavior expression animation toolkit.  In _Life-Like Characters_. Springer, 163–185.
  * Chu et al. (2014) Mingyuan Chu, Antje Meyer, Lucy Foulkes, and Sotaro Kita. 2014.  Individual differences in frequency and saliency of speech-accompanying gestures: The role of cognitive abilities and empathy.  _Journal of Experimental Psychology: General_ 143, 2 (2014), 694.
  * Clough and Duff (2020) Sharice Clough and Melissa C. Duff. 2020.  The Role of Gesture in Communication and Cognition: Implications for Understanding and Treating Neurogenic Communication Disorders.  _Frontiers in Human Neuroscience_ 14 (2020).  https://www.frontiersin.org/articles/10.3389/fnhum.2020.00323
  * DeVault et al. (2014) David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, Gale Lucas, Stacy Marsella, Fabrizio Morbini, Angela Nazarian, Stefan Scherer, Giota Stratou, Apar Suri, David Traum, Rachel Wood, Yuyu Xu, Albert Rizzo, and Louis-Philippe Morency. 2014.  SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support.  (2014), 1061–1068.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Ekman and Friesen (1978) Paul Ekman and Wallace V Friesen. 1978.  Facial action coding system.  _Environmental Psychology & Nonverbal Behavior_ (1978).
  * Fares et al. (2023) Mireille Fares, Michele Grimaldi, Catherine Pelachaud, and Nicolas Obin. 2023.  Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding.  https://hal.science/hal-03972415
  * Fares et al. (2022) Mireille Fares, Catherine Pelachaud, and Nicolas Obin. 2022.  Transformer Network for Semantically-Aware and Speech-Driven Upper-Face Generation.  http://arxiv.org/abs/2110.04527 arXiv:2110.04527 [eess].
  * Foley and Gentile (2010) Gretchen N. Foley and Julie P. Gentile. 2010.  Nonverbal Communication in Psychotherapy.  _Psychiatry (Edgmont)_ 7, 6 (June 2010), 38–44.  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2898840/
  * Gao et al. (2023) Nan Gao, Zeyu Zhao, Zhi Zeng, Shuwu Zhang, and Dongdong Weng. 2023.  GesGPT: Speech Gesture Synthesis With Text Parsing from GPT.  http://arxiv.org/abs/2303.13013 arXiv:2303.13013 [cs].
  * Ghorbani et al. (2023) Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F. Troje, and Marc-André Carbonneau. 2023.  ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech.  _Computer Graphics Forum_ 42, 1 (2023), 206–216.  https://doi.org/10.1111/cgf.14734 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14734.
  * Goldin-Meadow and Alibali (2013) Susan Goldin-Meadow and Martha Wagner Alibali. 2013.  Gesture’s role in speaking, learning, and creating language.  _Annual review of psychology_ 64 (2013), 257–283.
  * Google (2023) Google. 2023.  Claude.  https://bard.google.com
  * Grady (1997) Joseph Grady. 1997.  Foundations of meaning: Primary metaphors and primary scenes.  (1997).
  * Guellaï et al. (2014) Bahia Guellaï, Alan Langus, and Marina Nespor. 2014.  Prosody in the hands of the speaker.  _Frontiers in Psychology_ 5 (2014).  https://www.frontiersin.org/articles/10.3389/fpsyg.2014.00700
  * Hall et al. (1995) Judith A. Hall, Jinni A. Harrigan, and Robert Rosenthal. 1995.  Nonverbal behavior in clinician—patient interaction.  _Applied and Preventive Psychology_ 4, 1 (1995), 21–37.  https://doi.org/10.1016/S0962-1849(05)80049-6
  * Hall et al. (2016) Kira Hall, Donna M. Goldstein, and Matthew Bruce Ingram. 2016.  The hands of Donald Trump: Entertainment, gesture, spectacle.  _HAU: Journal of Ethnographic Theory_ 6, 2 (Sept. 2016), 71–100.  https://doi.org/10.14318/hau6.2.009 Publisher: The University of Chicago Press.
  * Hostetter (2011) Autumn B. Hostetter. 2011.  When do gestures communicate? A meta-analysis.  _Psychological Bulletin_ 137, 2 (2011), 297.  https://doi.org/10.1037/a0022128 Publisher: US: American Psychological Association.
  * Ishi et al. (2018) Carlos T. Ishi, Daichi Machiyashiki, Ryusuke Mikata, and Hiroshi Ishiguro. 2018.  A Speech-Driven Hand Gesture Generation Method and Evaluation in Android Robots.  _IEEE Robotics and Automation Letters_ 3, 4 (Oct. 2018), 3757–3764.  https://doi.org/10.1109/LRA.2018.2856281 Conference Name: IEEE Robotics and Automation Letters.
  * Jamalian and Tversky (2012) Azadeh Jamalian and Barbara Tversky. 2012.  Gestures alter thinking about time. In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , Vol. 34. 503–508.
  * Joty et al. (2015) Shafiq Joty, Giuseppe Carenini, and Raymond T Ng. 2015.  Codra: A novel discriminative framework for rhetorical analysis.  _Computational Linguistics_ 41, 3 (2015), 385–435.
  * Kendon (1997) Adam Kendon. 1997.  Gesture.  _Annual review of anthropology_ 26, 1 (1997), 109–128.
  * Kendon (2004) Adam Kendon. 2004.  _Gesture: Visible action as utterance_.  Cambridge University Press.
  * Kendon et al. (1980) Adam Kendon et al. 1980\.  Gesticulation and speech: Two aspects of the process of utterance.  _The relationship of verbal and nonverbal communication_ 25, 1980 (1980), 207–227.
  * Kipp (2003) Michael Kipp. 2003.  Gesture generation by imitation : from human behavior to computer character animation. Universität des Saarlandes.  https://doi.org/10.22028/D291-25852
  * Kipp and Martin (2009) Michael Kipp and Jean-Claude Martin. 2009.  Gesture and emotion: Can basic gestural form features discriminate emotions?. In _2009 3rd international conference on affective computing and intelligent interaction and workshops_. IEEE, 1–8.
  * Kosinski (2023) Michal Kosinski. 2023.  Theory of mind may have spontaneously emerged in large language models.  _arXiv preprint arXiv:2302.02083_ (2023).
  * Kucherenko et al. (2020) Taras Kucherenko, Patrik Jonell, Sanne Van Waveren, Gustav Eje Henter, Simon Alexandersson, Iolanda Leite, and Hedvig Kjellström. 2020.  Gesticulator: A framework for semantically-aware speech-driven gesture generation. In _Proceedings of the 2020 International Conference on Multimodal Interaction_. ACM, Virtual Event Netherlands, 242–250.  https://doi.org/10.1145/3382507.3418815
  * Lee and Marsella (2006) Jina Lee and Stacy Marsella. 2006.  Nonverbal behavior generator for embodied conversational agents. In _International Conference on Intelligent Virtual Agents_. Springer, 243–255.
  * Lhommet and Marsella (2014) Margot Lhommet and Stacy Marsella. 2014.  Metaphoric gestures: towards grounded mental spaces. In _Intelligent Virtual Agents: 14th International Conference, IVA 2014, Boston, MA, USA, August 27-29, 2014. Proceedings 14_. Springer, 264–274.
  * Marsella et al. (2013) Stacy Marsella, Yuyu Xu, Margaux Lhommet, Andrew Feng, Stefan Scherer, and Ari Shapiro. 2013.  Virtual Character Performance from Speech. In _Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation_ (Anaheim, California) _(SCA ’13)_. ACM, New York, NY, USA, 25–35.  https://doi.org/10.1145/2485895.2485900
  * McNeill (1985) David McNeill. 1985.  So you think gestures are nonverbal?  _Psychological review_ 92, 3 (1985), 350.
  * McNeill (1992) David McNeill. 1992.  _Hand and mind: What gestures reveal about thought_.  University of Chicago press.
  * McNeill (2005) David McNeill. 2005.  Gesture, gaze, and ground. In _International workshop on machine learning for multimodal interaction_. Springer, 1–14.
  * Miller (1995) George A Miller. 1995.  WordNet: a lexical database for English.  _Commun. ACM_ 38, 11 (1995), 39–41.
  * Mutinda et al. (2021) Faith Wavinya Mutinda, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2021.  Semantic Textual Similarity in Japanese Clinical Domain Texts Using BERT.  _Methods of Information in Medicine_ 60, S 01 (June 2021), e56–e64.  https://doi.org/10.1055/s-0041-1731390
  * Neff (2016) Michael Neff. 2016.  Hand Gesture Synthesis for Conversational Characters.  https://doi.org/10.1007/978-3-319-30808-1_5-1
  * Neff et al. (2008) Michael Neff, Michael Kipp, Irene Albrecht, and Hans-Peter Seidel. 2008.  Gesture modeling and animation based on a probabilistic re-creation of speaker style.  _ACM Transactions on Graphics (TOG)_ 27, 1 (2008), 1–24.
  * Nyatsanga et al. (2023) Simbarashe Nyatsanga, Taras Kucherenko, Chaitanya Ahuja, Gustav Eje Henter, and Michael Neff. 2023.  A Comprehensive Review of Data-Driven Co-Speech Gesture Generation.  https://doi.org/10.1111/cgf.14776 arXiv:2301.05339 [cs].
  * Ostermeier (1997) Terry H. Ostermeier. 1997.  _Gender, Nonverbal Cues, and Intercultural Listening: Conversational Space and Hand Gestures_.  Technical Report.  https://eric.ed.gov/?id=ED416520 ERIC Number: ED416520.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.
  * Özer and Göksun (2020) Demet Özer and Tilbe Göksun. 2020.  Gesture use and processing: A review on individual differences in cognitive resources.  _Frontiers in Psychology_ 11 (2020), 573555.
  * Peinelt et al. (2020) Nicole Peinelt, Dong Nguyen, and Maria Liakata. 2020.  tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 7047–7055.  https://doi.org/10.18653/v1/2020.acl-main.630
  * Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023.  Instruction Tuning with GPT-4.  _arXiv preprint arXiv:2304.03277_ (2023).
  * Ravenet et al. (2018) Brian Ravenet, Catherine Pelachaud, Chloé Clavel, and Stacy Marsella. 2018.  Automating the production of communicative gestures in embodied characters.  _Frontiers in psychology_ 9 (2018).
  * Salem et al. (2013) Maha Salem, Friederike Eyssel, Katharina Rohlfing, Stefan Kopp, and Frank Joublin. 2013.  To err is human (-like): Effects of robot gesture on perceived anthropomorphism and likability.  _International Journal of Social Robotics_ 5, 3 (2013), 313–323.
  * Salem et al. (2012) Maha Salem, Stefan Kopp, Ipke Wachsmuth, Katharina Rohlfing, and Frank Joublin. 2012.  Generation and evaluation of communicative robot gesture.  _International Journal of Social Robotics_ 4, 2 (2012), 201–217.
  * Saund and Marsella (2021) Carolyn Saund and Stacy Marsella. 2021.  The Importance of Qualitative Elements in Subjective Evaluation of Semantic Gestures. In _2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)_. IEEE, 1–8.
  * Saund et al. (2019) Carolyn Saund, Marion Roth, Mathieu Chollet, and Stacy Marsella. 2019.  Multiple metaphors in metaphoric gesturing. In _2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 524–530.
  * Seizer (2011) Susan Seizer. 2011.  On the Uses of Obscenity in Live Stand-Up Comedy.  _Anthropological Quarterly_ 84, 1 (2011), 209–234.  https://www.jstor.org/stable/41237487 Publisher: The George Washington University Institute for Ethnographic Research.
  * Thepsoonthorn et al. (2021) Chidchanok Thepsoonthorn, Ken-ichiro Ogawa, and Yoshihiro Miyake. 2021.  The Exploration of the Uncanny Valley from the Viewpoint of the Robot’s Nonverbal Behaviour.  _International Journal of Social Robotics_ 13, 6 (Sept. 2021), 1443–1455.  https://doi.org/10.1007/s12369-020-00726-w
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023\.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * Tversky and Hard (2009) Barbara Tversky and Bridgette Martin Hard. 2009.  Embodied and disembodied cognition: Spatial perspective-taking.  _Cognition_ 110, 1 (2009), 124–129.
  * Wolfert et al. (2021) Pieter Wolfert, Jeffrey M Girard, Taras Kucherenko, and Tony Belpaeme. 2021.  To rate or not to rate: Investigating evaluation methods for generated co-speech gestures. In _Proceedings of the 2021 International Conference on Multimodal Interaction_. 494–502.
  * Xie et al. (2021) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021.  An explanation of in-context learning as implicit bayesian inference.  _arXiv preprint arXiv:2111.02080_ (2021).
  * Yang et al. (2023) Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023.  Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond.  _arXiv preprint arXiv:2304.13712_ (2023).
  * Yoon et al. (2020) Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. 2020.  Speech gesture generation from the trimodal context of text, audio, and speaker identity.  _ACM Transactions on Graphics_ 39, 6 (Dec. 2020), 1–16.  https://doi.org/10.1145/3414685.3417838
  * Yoon et al. (2019) Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. 2019.  Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots. In _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 4303–4309.