# Iconic Gesture Semantics

Andy Lücking
Department of Computer Science
Goethe University Frankfurt
Frankfurt a/M., D-60325, Germany
<EMAIL_ADDRESS>
&Alexander Henlein
Department of Computer Science
Goethe University Frankfurt
Frankfurt a/M, D-60325, Germany
<EMAIL_ADDRESS>
Alexander Mehler
Department of Computer Science
Goethe University Frankfurt
Frankfurt a/M, D-60325, Germany
<EMAIL_ADDRESS>

###### Abstract

The “meaning” of an iconic gesture is conditioned on its informational
evaluation. Only informational evaluation lifts a gesture to a quasi-
linguistic level that can interact with verbal content. Interaction is either
vacuous or regimented by usual lexicon-driven inferences. Informational
evaluation is spelled out as extended exemplification (extemplification) in
terms of perceptual classification of a gesture’s visual iconic model. The
iconic model is derived from Frege/Montague-like truth-functional evaluation
of a gesture’s form within spatially extended domains. We further argue that
the perceptual classification of instances of visual communication requires a
notion of meaning different from Frege/Montague frameworks. Therefore, a
heuristic for gesture interpretation is provided that can guide the working
semanticist. In sum, an iconic gesture semantics is introduced which covers
the full range from kinematic gesture representations over model-theoretic
evaluation to inferential interpretation in dynamic semantic frameworks.

_Keywords_ Iconic Gesture Semantics, Exemplification, Extemplification,
Perceptual Classification, Speech–Gesture Integration, Semantic Theory

###### Contents

  1. 1 Introduction
  2. 2 A brief gesture primer
  3. 3 Vector space semantics
    1. 3.1 Kinematic gesture representation
    2. 3.2 Spatial frames of reference
    3. 3.3 Applying projection semantics?
    4. 3.4 Basic vector space semantics
    5. 3.5 Iconic models in vectorial gesture space
  4. 4 Perceptual classification and extemplification
    1. 4.1 Excursus: Vectorial biological movement analysis and perceptual classification
    2. 4.2 Linguistic toolkit: Extended exemplification as gesture interpretation
    3. 4.3 Lexicon-driven speech–gesture integration
    4. 4.4 Applications
  5. 5 Discussion
    1. 5.1 “Energy spaces”
    2. 5.2 Non-lexicalized iconic models
    3. 5.3 Repercussions for semantic theories
  6. 6 Conclusions

##  1 Introduction

> “as we see _them_ , we see something _in them_.”
>
> Jürgen [131, p. 286 ]

“So You Think Gestures are Nonverbal?” was asked by [96] in McNeill:1985, and
the close interweaving of gesture and speech in utterance production has been
emphasized since then [111, see, e.g.,]. And indeed, speech and gesture also
seem to interact semantically, as has been pointed out frequently [97, 65,
see], and is, for instance, most clearly shown by examples where the gesture
can be construed in such a way that it adds to speech meaning, as in (), where
the gesture can be understood to provide shape information specifying the type
of the staircases talked about:111The example is taken from dialogue V10 from
the SaGA corpus [92], starting at minute 3:19. “Ich g[laube das  sollen
TREP]pen sein” (I think that should be staircases; capitalization indicates
main stress of the first syllable of the noun Treppen ‘staircases’, square
brackets indicate the temporal alignment of speech and gesture)

If the gesture is construed in such a way that it makes the concept wounded or
spiral salient, then there is a strong inclination to interpret the noun
staircases in terms of its hyponym spiral staircases. The modest conditional
formulation – _if the gesture is construed in such a way that …_ – is indeed
part of the iconic gesture semantics developed in the following. We argue that
the _informational evaluation_ of a gesture lifts its visual properties to
quasi-linguistic status, which can interact with natural language semantics
and reasoning in the first place (we argue further that this reasoning is
mainly driven by lexical knowledge). This is because gestures and words are
quite different things: It is widely assumed that the meaning of natural
languages are conventionalized, largely arbitrary systems, while iconic
gestures are driven by non-arbitrary, visuo-spatial properties.222See, for
instance, [80] respectively [2, 26, 7]. This ambivalence – the verbal and
nonverbal characteristics of iconic gestures – has also beset iconic gesture
semantics: On the one hand, there are gesture semantics that assume a shared
semantic representation of speech and gesture, namely the predicate constants
of the formal semantic representation language [79]. This is obviously a word-
like construal of iconic gesture meaning. On the other hand, [41] developed a
mereotopological model of the interpretation of iconic gestures, accounting
for the nonverbal, visual nature of iconic gestures.333The status of other
works is less clear. [122], for instance, pursue a labelling approach but
restrict the gestural predicates to spatial ones, which give rise to partial
ontology descriptions [49, see also]. Hence, they use labels to describe
spatial configurations. [83] employs a spatial, vector-based model, but only
to the effect of deriving verbal labels therefrom. This shows that both,
labelling and spatial, views should be kept apart and combined in a unified
approach, which is one of the goals of this paper.

The difference between both approaches can be illustrated by means of the
gesture from (): The first approach – _labelling theories_ – assigns semantic
predicates to the gesture which express its meaning(s)444Since gestures are
considered vague or underspecified, a single gesture usually is compatible to
several predicates [79]. (a), the second one – _spatial theories_ – models the
meaning of the gesture in terms of a spatial structure, or iconic model (b).
labelling theories:
$\textit{spiral}(x)\vee\textit{curled}(x)\vee\textit{twined}(x)\vee\ldots$
spatial theories:

We conjecture that this ambivalence is due to the “twofoldedness” of gestures
diagnosed in gesture research: “as we see _them_ , we see something _in
them_.” [131, p. 286; original emphsis]. We aim at an explanation and a
reconciliation of this ambiguity, since from a theoretical perspective, this
situation is unsatisfactory for several reasons:

Firstly, labelling theories lack a principled way of assigning predicates to
iconic gestures; semantic representations are brought about “by hand”, merely
following (not necessarily justified, see section 4.4) interpretive
assumptions.

Secondly, spatial theories fail to deliver a computational procedure for
deriving iconic models from kinematic gesture representations.555The gesture
interpretation function $\phi$ assumed by [41], for instance, is not spelled
out.

Thirdly, the meaning representations in (a) and (b) are obviously related in a
non-arbitrary way. However, the relation between labelling and spatial
theories has not been addressed in gesture semantics so far. We provide the
basis of a unified iconic gesture semantics that takes these desiderata into
account. Hence, gesture semantics is orthogonal (and intuitively and logically
prior, cf. sections 4.3 and 4.4) to recent investigations of the inferential
interplay of speech and gesture [29, 126, 32, e.g.,].666“It should be
emphasized that we will not seek to explain how a gesture […] comes to have
the content that it does” [125, p. 296].

Our starting point is a spatial theory, namely a vector space model, following
work by [140, 144, 142] (section 3). Our first contribution is a formal notion
of _gesture space_ modelled as an oriented vector space in terms of an
adaptation of vector space semantics. We then show how kinematic gesture
representations can be interpreted in terms of vector sequences in gesture
space (addressing the second issue raised above).

Our second contribution is a shift in semantic architecture: we propose to
employ vector spaces as models of _intensions_ of certain lexical items
(section 4). This move paves the way for explaining the relation between
labelling and spatial theories (third issue): iconic models and semantic
predicates are linked via a semiotic relation reminiscent of _exemplification_
[42]. We will argue that in order to maintain the semiotic nature of iconic
gestures (i.e., avoiding treating them as “mere” objects or events), a
modified notion of exemplification is needed. This is perhaps the most
consequential contribution (why we have said that it is a shift in semantic
architecture): The notion of meaning in mainstream possible worlds semantics,
$\llbracket\cdot\rrbracket$, is such that the meaning
$\llbracket\alpha\rrbracket$ of a word $\alpha$ when applied to a world $w$
returns $\alpha$’s extension in $w$.777We are simplifying somewhat, of course,
ignoring intensional expressions and contexts. This does not touch on the
general point we want to make, however. The notion of meaning emerging from
extended exemplification (“extemplification”, which is needed for assigning
“meaning” to gestures), to the contrary, has to be such that when applied to
an object (in some world) it returns a linguistic label for that object. A
consequence of this view is that iconic gesture semantics cannot be fully
spelled out within a Frege/Montague framework (and this is most likely the
reason that there is still no formal iconic gesture semantics). We therefore
provide a heuristic for interpreting gestures within possible worlds semantics
(section 4.2).

The general architecture of the iconic gesture semantics is shown in Figure 1,
extending an image used by [41]. Speech is interpreted as usual by a
denotation function within frames of reference. The kinematic representation
of a gesture, however, is translated via function $\phi$ into an iconic model.
This model intersectively constrains the spatial projection of the entities
talked about within the spatial frame of reference. Hence, iconicity is
semantically differentiated in two aspects: a direct interpretation of a
gesture’s form ($\phi$), and a spatial model interacting with verbal meaning.
This part of gestures semantics can be integrated into spatially extended but
otherwise traditional Frege/Montague models. The spatial iconic models are
then _informationally evaluated_ in terms of the predicate constants of the
semantic language (extended exemplification, or classification). This step
brings about quasi-linguistic gesture meaning that can interact with speech
meaning (e.g., lexical meaning; section 4.3). We discuss repercussions of
iconic gesture semantics with regard to semantic theorizing in general in
section 5.

_A word of warning._ We consider iconic gesture semantics a theoretical
contribution. As such, the following work is largely conceptual: the starting
point is a standard semantic framework in the tradition of Frege/Montague. It
is extended by a spatial framework to capture the semantic, that is, truth-
functional contribution of iconic co-speech gestures. Furthermore, and in line
with the above-said, we take seriously the spatial (as opposed to
linguistic888We use the term ‘linguistic’ and its morphosyntactic variants to
denote constituents of speech (i.e., words phrases, clauses) in contrast to
gestures. Given a bulk of gesture research that tries to collect evidence that
gestures are linguistic, too, this distinction can be questioned, and we won’t
argue that. It is nonetheless a handy distinction.) nature of iconic gestures.
As a consequence, many spatial, vectorial representations are used, which may
be unusual for textbook semantics. These are needed to spell out a fully
compositional procedure for iconic gesture semantics, and an algorithm for
deriving gesture interpretations. We only come back to more concrete examples
in section 4. The theory and framework we are proposing is nonetheless rooted
in much previous work and empirical findings, some of which are discussed in
the following section to derive some “guidelines” which a decent iconic
gesture semantics has to live up to.

GestureFrame of referenceVerbal constituentIconic modelSpatial frame of
reference$\llbracket\text{$\cdot$}\rrbracket$$\phi$$\llbracket\text{$\cdot$}\rrbracket$LociconicityVerbal
constituentLexicon (frames)classifiesexemplifies extendedlyconnected
toindirect interaction

Figure 1: Architecture of iconic gesture semantics. See main text for details
and pointers to pertinent sections.

##  2 A brief gesture primer

By ‘gesture’, we refer to hand and arm movements that accompany speech and are
related to the narrative. In early taxonomies, several kinds of manual
gestures have been distinguished, namely _metaphorics_ , _deictics_ ,
_iconics_ , _beat gestures_ [97].999The original taxonomy includes a fifth
class, namely _cohesives_ , which is usually not adopted in later taxonomies.
In any case, manual gestures are distinguished from signs of a sign language
and from emblems. These gesture classes are, however, not mutually exclusive:
iconic gestures may exhibit rhythmic patterns, and iconic features can overlay
deictic gestures. For instance, while pointing at a wooden disc, the index
finger can rotate, tracing the outline of the referent [71]. Hence, gesture
“classes” are better conceived as _dimensions_ of gesture meaning [98].

Deictic gestures can be subdivided according to targeting an object,
indicating a direction or region [118], or being abstract (e.g., pointing at a
location in gestures space; [99]). Based on gesture research, a more specific
classification of iconic gestures can be given, too: in iconic gesturing, the
hands virtually do what the hands do in everyday life and artwork, namely
_acting_ , _molding_ , _drawing_ , and _representing_ [131, 105, we adopt
Müller’s quadrinomial nomenclature; Streeck distinguishes even 12
_practices_]. _Acting_ classifies a miming action. _Molding_ describes three-
dimensional sculpturing, _drawing_ two-dimensional tracing. In _representing_
, the hand/arm is a proxy for an object. Such gestural modes of representation
will play a prominent role for capturing the meaning of iconic gestures, as
will become clear in the following.

As has been pointed out by [8], there are frequent gesture occurrences that
are functionally bound up with coordination or dialogue management, rather
than dialogue content. Such gestures have also been termed _interactive
gestures_ , emphasizing that they serve pragmatic functions in
dialogue.101010This includes some uses of pointing gestures, which, while
directed at the addressee, are embedded in information-state processing [38]:
they can be assigned to the object-deictic and dialogue-management dimensions.
A summary is provided in Figure 2. We are concerned with the iconic dimension
of co-speech gestures (and the corresponding modes of representation) in the
following.

[ manual gesture [formal [deictic, [object] [direction] [abstract] ] [iconic,
draw, l*=2 [acting, name=acting] [molding, name=molding] [drawing,
name=drawing] [representing, name=representing] ] [metaphoric, l*=2,
name=metaphor] [rhythmic
(beat), align=center] ] [functional [dialogue
content, align=center, l*=2] [dialogue
management
(interactive), align=center, l*=2] ] ]

Figure 2: Dimensions of classifying gestures. The focus is on iconic gestures.

The semantic problem here is that iconic gestures are not regimented by fixed
form–meaning associations (i.e., a lexicon; cf. [97]). Hence, iconic gestures
do not constitute a fixed class which can simply be interpreted by the
interpretation function, $\llbracket\text{$\cdot$}\rrbracket$, for language.
In fact, the very same gesture receives quite different interpretations when
produced in combination with different words and phrases, as exemplified in
(), varying a constructed example used by [32].111111The gesture is drawn by
using Ressler’s _Sketch_ system (http://www.frontiernet.net/~eugene.ressler/).
S. might bring her  bringboxdog. $\Rightarrow$“holding/touching dog” S. might
bring her  bringboxdog in a box. $\Rightarrow$ “holding/touching box” S. might
bring her  bringboxsmall dog. $\Rightarrow$ “sizing dog” S. might bring her
bringboxdog by carrying it. $\Rightarrow$ “carrying dog”

Gesture interpretation also varies with respect to temporal alignment with
speech. Consider again example (1), repeated in (). The part of speech which
roughly co-occurs with the gesture is indicated by brackets. Since the first
syllable from the noun _Treppen_ is not only part of the portion of speech
that co-occurs with gesture but also has primary stress (indicated by capital
letters), it is the first candidate for providing an integration point for
gesture information [82, 83, 1]. Hence, the noun is the integration locus of
the spiral-movement gesture. Ich g[laube das  sollen TREP]pen sein // I think
that should staircases be // ‘I think that should be staircases’ \+ gesture
overlapping with bracketed speech //

The multimodal expression composed of the noun staircases and the spiral
gesture _can_ (but need not) be interpreted in such a way that the gesture
depicts the shape of staircases, indicating the hyponym spiral staircases.

Changing the affiliated word also changes gesture interpretation. Suppose the
gesture is produced a bit earlier and the main stress is carried by the main
verb, as indicated in (), taken from [87, p. 1206 ]: I[ch  GLAUbe das] sollen
Treppen sein // I think that should staircases be // ‘I think that should be
staircases’ \+ gesture overlapping with bracketed speech // Now the gesture is
associated with think and interpreted as a metaphorical depiction of a
cognitive process. The gestures’ co-speech dependence in () is therefore bound
up with _affiliation_ : A gesture occurrence has a “docking point” in speech,
its _lexical affiliate_ [124]. The affiliate guides the interpretation of the
gesture [48]. Note that while the affiliate was initially assumed to be a
lexical item (i.e., a “single word”), it is lexical in only about 80
$\mathrm{\%}$ of occurrences, the remainder exhibit syntactically more complex
verbal attachment sites [101].

In many instances the gesture remains “informationally vacuous”, however,
depicting its affiliate, as in (a,b). Such types of examples have not received
much attention in the semantic literature, but need to be covered, too. Of
more interest are examples such as the spiral staircase one or (c), where
there is no direct depiction relation between the gesture and its affiliate.
We are concerned with a semantic framework that allows deriving the
interpretations given after the arrow (‘$\Rightarrow$’). These do not cover
the full range of iconic gestures, though. We will not discuss examples such
as (d), which arguably involve question-under-discussion management [75], but
we briefly come back to it in the discussion in section 5.

Stewart Robson on ESPN FC Extra Time:

https://www.youtube.com/watch?v=CiVS5_HKFY8&t=0h1m18s

You know when they go on that wheel [circle gesture] and throw [throwing
movement] the dagger would you ever like to see that go wrong?
$\Rightarrow$ circle gesture represents the wheel, and the throwing movement
pantomimically mimics throw

Toni Kroos (in Spanish)

https://www.youtube.com/watch?v=XwGkfrvXPpI&t=0h6m12s

dart? dart? Sabes que es dart? [throws virtual dart arrow]
$\Rightarrow$ the throwing movement pantomimically mimics darting

Daniel Levitin, Ted talk:

https://www.youtube.com/watch?v=8jPQjjsBbIc&t=0h1m30s

so I [balancing/weighing movement] figured, under the circumstances, I was
coming out even.
$\Rightarrow$ figure out by weighing

Daniel Levitin, Ted talk 2:

https://www.youtube.com/watch?v=8jPQjjsBbIc&t=0h3m55s

some of them are obvious [locate right], some of them are not so obvious
[locate left]

The co-text dependence of gestures obvious from () is, however, not arbitrary.
It is restricted by semantic speech–gesture congruence, or semantic
compatibility, as evinced by speech–gesture mismatches. An example from a
study of [14] is given in (), where the brackets indicate the segment of the
utterance that overlaps with the offering gesture (i.e., an action that
simulates a delivery event):121212The stimuli are two-time videotaped
retellings of Sylvester and Tweety Bird cartoons, where the second recording
involve a modified (mismatch) gesture. Granny sees him and says “oh what a
nice little monkey”. And then she [offers him a penny]. normal: left hand
proffers penny in the direction of listener. # mismatched: left hand offers
penny to self.

The speaker in () adopts the viewpoint of the granny, and offering is an
action that is directed towards an addressee, hence the incongruity. Notice
that incongruity is a stronger notion than denotational vacuity: the mismatch
in () seems to rest on a _conflict_ between speech meaning and iconic model,
not just in the fact that there is no model that satisfies both. A gesture
semantics should be part of the explanation of such mismatches.

To summarize, an iconic gestures semantics needs to address (at least) three
basic desiderata:

D1

Gesture interpretation is strongly dependent on the accompanying speech
(affiliate dependence).

D2

Speech–gesture affiliation is regimented by (in-)congruency.

D3

Gesture interpretation is an instance of visual communication, which rests on
a perceptual interpretation of gestural forms.

##  3 Vector space semantics

To capture the visual content of an iconic gesture in line with spatial
theories (briefly introduced in section 3.2), we adopt a vector space model
(section 3.4). One might also try to apply projective semantics to iconic
gestures, which is discussed in section 3.3).131313This has, to our knowledge,
not been seriously proposed so far (and we will see that it is a non-starter
for gestures), but is alluded to by [126, p. 744 ]. In any case, some kind of
“preprocessing” is required that works on the kinematic gesture representation
and brings about visuo-spatial constraints – this is the heart of iconicity.
To this end, a kinematic, form-based gesture representation is introduced in
section 3.1. Its interpretation in vector space is discussed in section 3.5.

###  3.1 Kinematic gesture representation

> “Bodily movements cannot be transcribed directly, they have to be described.
> Descriptions in their turn are to be read like stage directions that do not
> have to be pronounced but enacted.”
>
> [10, p. 1003 ]

The alphabet provides a ready-made transcription system for written text, and
phonetic transcription systems for spoken language. How to represent iconic
gestures? Given that manual gestures are visible, bodily movements, a form-
based, kinematic representation is appropriate. There are several such
transcription systems around, we illustrate gesture annotation according to
the format used by [69, 92]. A brief introduction to kinematic gesture
representation is needed since it is the basis for the construction of iconic
models in vector space (section 3.5). In fact, “annotation playing for
gestures the same role as syntax representation plays for linguistic
utterances” [121, p. 123].

The formal description of a gestural movement is given for each hand in terms
of the _handshape_ , the orientations of the _palm_ and the _back of the hand_
(boh), the movement trajectory (if any) of the _wrist_ , and the relation
between both hands (synchronicity, sync). The handshape can be transcribed
according to a handshape nomenclature, for instance, the fingerspelling
alphabet of American Sign Language, where “G” labels an extended index finger.
The orientations (orient) of the palm and back of the hand are specified with
reference to the speaker’s body (e.g., PAB encodes “palm away from body” and
BUP encodes “back of hand upwards”). Movement trajectories of the whole hand
are specified with respect to the wrist in terms of the described path, its
direction (dir), and the extent of the movement. Position and extent are given
with reference to the _gesture space_ [97, p. 86–89]. Originally, gesture
space was a two-dimensional descriptive “map” for kinematic gesture
descriptions, where the third dimension is given only as the annotation of the
distance of the hand from the speaker’s body (feature dist). Hence, it seems
reasonable to construe gesture space as a three-dimensional space that is
oriented along the speaker’s/gesturer’s anatomical planes, as displayed in
Figure 3. Since this representational system of gesture “phonology” is divided
according to the anatomical joints of the arm it arguably implements complete
coverage of gestural movements. Directional information is given in terms of
45° sections (e.g., MF ‘forward’, ML ‘left’, and MF/ML ‘between forward and
left’, i.e., “diagonally ahead”), which was sufficient for annotation projects
[93, see], but could be made more fine-grained.

CBLCLCULCBCCCUCBRCRCURbackrightupNMF CBL: center below left CL: center left
CUL: center upper left CB center below CC: center center … … N: near M: middle
F: far Figure 3: Gesture space model. Three-dimensional extension of the two-
dimensional model of [97] (taken from [85]).

As an example, consider the representation of the spiral gesture from our
first examples (1), (2) and (2). The right hand with the index finger extended
is moved upwards, whereby the back of the hand is rotated – a combined
movement that generates the visual impression of a spiral. The corresponding
annotation is shown in Figure 4. The feature value LHH (“left hand held”)
tells us that the left hand is not in rest position, but is actually not
performing a gesture. Such information is captured as synchronization
information (sync), which also hosts the starting and ending slot of a gesture
movement in gesture space (sloc respectively eloc).

$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
right hand\hfil\\\ handshape &1.99997pt$\mskip-3.0mu\left[\hskip
0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
shape &1.99997ptG \\\ path &1.99997pt0 \\\ dir &1.99997pt0 \crcr}\vskip
1.65761pt}\hskip 0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ palm
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
orient &1.99997ptPAB$>$PAB/PUP$>$PAB\\\ path &1.99997pt0 \\\ dir &1.99997pt0
\crcr}\vskip 1.65761pt}\hskip 0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ boh
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
orient &1.99997ptBUP$>$BTB/BUP$>$BUP \\\ path &1.99997ptarc$>$arc$>$arc \\\
dir &1.99997ptMR$>$MF$>$ML\crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ wrist
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
path &1.99997ptline \\\ dir &1.99997ptMU \\\ dist &1.99997ptD-EK\\\ extent
&1.99997ptsmall\crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ sync
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
config &1.99997ptBHA \\\ rel.mov &1.99997ptLHH \\\ s-loc &1.99997ptP-R\\\
e-loc &1.99997ptP-UR \crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$\ignorespaces\crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$

Figure 4: Kinematic representation of the spiral gesture from examples (1),
(2) and (2).

In the following, it is shown that kinematic gesture representations can be
mapped onto vector sequences within vector space models. Vector space models
are instances of spatial theories, that extended standard models with a
spatial frame of reference.

###  3.2 Spatial frames of reference

The basic idea of spatial theories is that linguistic expressions are
interpreted in terms of a spatial frame of reference in addition to the
standard entity frame of reference. The spatial frame of reference can be
understood as an abstract configuration of the space inhabited by the
discourse entities (the objects and situations talked about). The spatial
theory developed by [41], for instance, makes use of a rather powerful
mereotopological language for modeling the spatial frame. On this account, an
iconic gesture is mapped via a procedure $\phi$ onto an iconic space, that is,
a mereotopological rendering of the gesture’s kinematic representation. The
iconic space bears an equivalence relation (under some fixed perspective) to
the spatial frame of reference – capturing the iconic gestures’ visual
contribution. The integration of speech and gesture is intersective: a gesture
assigns the arguments of its verbal affiliate (which can be an $n$-ary
predicate or an $n$-ary predicate modifier; a grammar of affiliation is
presupposed on this account) a common subspace in the spatial frame of
reference and requires this subspace to be equivalent to the iconic space,
that is, the output of $\phi$. An illustration of the architecture of the
spatial model of [41] is given in Figure 5, which provides a useful summary
for spatial theories in general.

The approach conservatively extends the truth conditions of multimodal
utterances: a multimodal utterance (an utterance consisting of a sentence and
an accompanying gesture141414Since integration in Giorgolo’s model happens on
the level of predicates or modifiers, it seems to be possible that a sentence
may be coupled with several gestures.) is true just in case the sentence is
true in the Montagovian frame of reference, and the spatial configuration of
the referents is such that it complies to the iconic space (via iconic
equivalence between the spatial frame of reference and the output of $\phi$).
It is dynamic in the sense that a gesture is not lexicalized, in line with
insights from gesture studies (see section 1). A gesture is considered a
physical act. The biggest downside of [41]’s theory, however, is that a spell-
out for $\phi$ is not yet provided. It therefore provides an impressive
conceptual and analytical model but falls short computationally.

GestureFrame of referenceVerbal constituentIconic spaceSpatial frame of
reference$\llbracket\text{$\cdot$}\rrbracket$$\phi$$\llbracket\text{$\cdot$}\rrbracket$Lociconicity
Figure 5: The process of iconic gesture interpretation within a spatial-frame-
of-reference model (reproduced after [41, p. 53]).

###  3.3 Applying projection semantics?

A truth-functional approach to the meaning of figurative
drawings151515“Canonical examples of pictures include architectural and
engineering drawings, figurative paintings, functional illustrations sketches
and illustrations, photographs, as well as many kinds of maps.” [44, p. 849]
has been developed by [43, 44] in terms of projection semantics. A projection
is a geometric projection in the sense of the artist’s technique to construct
perspectival paintings.161616See, for instance, Albrecht Dürer’s Underweysung
der Messung, in particular illustrated in the final image of the fourth volume
(available at
https://de.wikisource.org/wiki/Underweysung_der_Messung,_mit_dem_Zirckel_und_Richtscheyt,_in_Linien,_Ebenen_unnd_gantzen_corporen/Viertes_Buch).
Projection semantics, however, uses geometric projections to _interpret_
pictures. A picture expresses a content (a pictorial space), if the picture is
a projection of that content (relative to a viewpoint, that is, a pair of
projection source and picture plane) – see Figure 6 for a simple
example.171717Formally: $\llbracket\text{P}\rrbracket_{S,c}\subseteq\\{\langle
w,v\rangle\mid\textit{proj}_{S}(w,v)=P\\}$ (the denotation of a picture $P$ in
context $c$ is a subset of the world–viewpoint pairs $\langle w,v\rangle$
(scenes) such that there is a projection from the world–viewpoint pairs onto
$P$ relative to a system of depiction $S$).  Projection semantics is semantic
because it offers a notion of accuracy: a picture is accurate iff (abbreviates
if and only if) there is a viewpoint that provides a geometric projection from
the pictorial space to the picture. Viewpoints can straightforwardly be
embedded in standard possible world semantics: they provide a geometric system
on top of the worlds already part of the model’s domains. Further refinements
can, or need to, be given in terms of different systems of depiction (in
particular “impurely projective” ones like caricature; Figure 6 only shows
simple line drawing) and depth constraints, among others [44, cf.].

sourceworldprojection linespicture plane Figure 6: Geometric projection: The
pair of projection source (determining perspective) and picture plane
(determining orientation) is a viewpoint. A viewpoint and a world define a
scene. The set of all such scenes is the pictorial space, the content of a
picture.

Does projection semantics also provide an adequate framework for analyzing
iconic gestures? A mapping between paintings and gestures is obtained
straightforwardly: the gesturer and his static pose or dynamic hand and arm
movement plays the role of the picture plane which displays the projection
source. The content of the gesture, as with pictures, depends on the depiction
system to be at work. Given the basic figurative projection function of
projection semantics, this means, that the content of a gesturer’s gesture is
the set of world–viewpoint pairs where a (possibly different, let us assume)
gesturer performs that gesture, seen from the viewpoint in question. Such a
verbatim interpretation obviously misses the semiotic point of a gesture: the
content of a gesture surely is not a look-alike gesturing situation. In other
words, projection semantics, when applied to iconic gestures, fails to
distinguish between the gesture and the content of the gesture.

Besides this semiotic issue, geometric projections of oriented worlds face
perceptual challenges. Consider an iconic gesture that can be interpreted as
rolling: the index finger rotates in circles while the wrist is moved
rightwards. The decisive feature is that there is a part of the rotation
movement of the index finger that runs backwards (i.e., in the opposing
direction of the wrist movement), as illustrated in Figure
7LABEL:sub@subfig:perzmov (see also [13, p. 1088]). This configuration,
however, is a purely perceptual one; it can never be projected onto a physical
movement (Figures 7LABEL:sub@subfig:rolling-motion and
LABEL:sub@subfig:pointb), where “going back” is simply impossible. Thus,
geometric projections fall short of capturing the semiotic potential of
gestures (they only account for the gesturer’s movement), and they make wrong
predictions with respect to gestures where the perceptual image diverges from
its physical origin.

AB

(a)

(b)

BA

(c)

Figure 7: Perceiving a rolling disc as simultaneous movements of points $A$
and $B$ LABEL:sub@subfig:rolling-motion. The movement trajectory of point $B$
in isolation corresponds to a cycloidal trace LABEL:sub@subfig:pointb. The
shared movement of $A$ and $B$ – a horizontal rightward movement – is the
perceptual reference frame for both individual movements. The resulting
_percept_ is that $B$ circles around $A$ LABEL:sub@subfig:perzmov. (Reproduced
after [60, p. 207])

###  3.4 Basic vector space semantics

If mereotopological models and geometric projections do not live up to the
representational and computational requirements for an iconic gesture
semantics (as discussed in sections 3 and 3.3), what kind of model can be
employed to aptly interpret visual communication? In various areas of
semantics and cognitive science, vector representations have been developed to
this end [132, 73, 59, 109, e.g.,]. Within formal semantics, a vector space
semantics has been introduced by Joost Zwaarts [140, 141, 144]. For an
overview of spatial semantics see [142].

Following [144], we assume that vectors are primitive spatial entities in
natural language models. Vectors are given within a vector space $V$ over the
real numbers $\mathbb{R}$.181818We assume a standard Euclidean space, but this
is not a necessity: spaces defined in terms of polar instead of Cartesian
coordinates are formally as well as conceptually potent alternatives [143].
Vector spaces are closed under vector addition and scalar multiplication. Two
domains are defined from this ontology:

  * •

The domain of points: $D_{p}=V$ (each point is defined by a vector’s endpoint)

  * •

The domain of vectors: $D_{v}=V\times V$ (the Cartesian product of $V$)

  * •

For each element (“point”) $w\in D_{p}$ there is a vector space $V_{w}\in
D_{v}$ ($w$ is the zero-vector, or centre of $V_{w}$).

$D_{p}$ and $D_{v}$ are added to the model along with the domain of entities
$E$. Elements from $D_{p}$ are referred to as points, elements from $D_{v}$ as
vectors. The former are noted as $\mathbf{p}$, $\mathbf{q}$, the latter as
$\mathbf{u}$, $\mathbf{v}$, $\mathbf{w}$.

$E$ (the domain of entities) and $D_{v}$ are related by a couple of functions,
some of which are briefly introduced in the following (we basically follow the
vector space model introduced by [140, 141]):

  * •

The vector space located at a concrete object denoted by an NP $\alpha$ is
given by
‘$\operatorname{space}(\llbracket\text{$\alpha$}\rrbracket_{M})$’.191919In
addition to located vector spaces, also the “bounding box” of objects will be
needed [135].

  * •

Place and axis vectors determine spatial relationships:

    * –

$\operatorname{place}(x,\mathbf{v})$: $x$ is placed at the end of
$\mathbf{v}$; $\operatorname{place}(\mathbf{v},x)$: the beginning point of
$\mathbf{v}$ is placed at $x$; $\operatorname{place}(\mathbf{u},\mathbf{v})$:
the beginning point of $\mathbf{u}$ is placed at the end of $\mathbf{v}$ – see
Figure 10.

    * –

$\operatorname{axis}(x,\mathbf{v})$ object $x$ has an axis $\mathbf{v}$ – see
Figure 10.

  * •

Paths: sequences of axis or place vectors, see Figure 10. Paths are defined as
a mapping from an interval $[0,1]$ to vectors. Paths are notated as
$\mathbf{v}[k]$. The beginning of a path is indexed as $\mathbf{v}[0]$, its
end point as $\mathbf{v}[1]$.202020Hence,
$\operatorname{place}(\mathbf{w},\mathbf{v})\Leftrightarrow\mathbf{w}[0]=\mathbf{v}[1]$.
Note that defining paths to the normalized interval $[0,1]$ diverges from the
original model of [141], where real or natural numbers have been used.
Normalization is a simplification that allows to address a path’s beginning
and end points without additional mapping. Note further that for the
vectorization of kinematic gesture representations introduced in section 3.5
vector sequences built by concatenating vectors head to tail are more useful.
However, we stipulate that any head-to-tail vector sequence approximates a
continuous vector sequence, or, conversely, that there is an interpolation for
a head-to-tail sequence that equals a continuous vector sequence. Note that
paths are non-temporal entities. They receive a temporal interpretation only
if index $k$ is mapped to points or intervals in time.

[3] []
$y$$\operatorname{place}(\mathbf{v},y)$$\mathbf{v}$$\operatorname{place}(\mathbf{w},\mathbf{v})$$\mathbf{w}$$x$$\operatorname{place}(x,\mathbf{w})$
[] $x$$\mathbf{v}$ [] $x$

Figure 8: Place vector Figure 9: Axis vector Figure 10: Place path

Vectors pointing in the same direction constitute an _axis_. The set of
vectors varying only in the direction of one of their three dimensions make up
a _plane_.

  * •

The _inverse_ $\mathbf{-v}$ of a vector $\mathbf{v}$ points in the opposite
direction of $\mathbf{v}$, see Figure 14. The inverse of an axis $A$ is the
set of vectors $-A$ pointing in the opposite direction of $A$.

  * •

The _orthogonal complement_ $A^{\perp}$ of an axis or plane $A$ is the set of
vectors orthogonal to those in $A$ ($A^{\perp}=\\{\mathbf{v}\in
D_{v}\mid\forall\mathbf{w}\in A:\mathbf{v}\perp\mathbf{w}\\}$), see Figure 14.

  * •

Each vector space $V$ provides three, mutually perpendicular _orienting half-
axes_. Intuitively, these axes correspond to the directions up (up), forward
or front (ft), and right (rt). These half-axes in addition to their
corresponding inverses, give rise to an _oriented vector space_ , see Figure
14. The orienting half-axes are determined by external reference frames, or by
intrinsic or functional properties of reference objects. Except for the
anatomical planes (see section 3.5 below), we will leave open further
specifications in this regard.

  * •

Given orienting axes $A$ and $A^{\prime}$, a vector $\mathbf{v}$ can be
decomposed into its _projections_ onto the axes, $\mathbf{v}_{A}$ and
$\mathbf{v}_{A}^{\prime}$. Figure 14 shows the orthogonal components of
$\mathbf{v}$ on the up and the rt axes.

[4] [] $\mathbf{v}$$\mathbf{-v}$ [] A [] up$-$uprt$-$rtft$-$ft []
uprt$\mathbf{v}$$\mathbf{v}_{\textsc{rt}}$$\mathbf{v}_{\textsc{up}}$

Figure 11: Inverse of a vector Figure 12: Orthogonal complements of an axis
Figure 13: Orienting half-axes and their inverses Figure 14: Projections of a
vector

Why do we need vector denotations? Vectors provide a mathematical model for
spatial prepositions such as above. The differently shaded areas in Figure 16
show three readings of above (of a glass), which can all be defined in terms
of sets of vectors from the vector space located at the object in question
[140].212121A more elegant solution is to define a probability distribution
over the space surrounding the object denoted by the NP [109].

[2] [+2cm]

*

Figure 15: above(NP) $=$
$\\{\mathbf{v}\in\text{space}(\llbracket\text{NP}\rrbracket)\mid\ldots\\}$.

[+2cm]

Figure 16: $\lambda x.\lambda
e[\text{walk}(e,x)\wedge\exists\mathbf{v}[\text{path}(e,\mathbf{v})]]$.

Vectors (that is, spatial entities) are also part of the truth conditions for
movements in space. For instance, a walking event is a walking event only if
there is a path of motion, be it undirected, as in Figure 16, or directed
[73]:222222Since we need some notion of events, we make use of a neo-
Davidsonian semantics in the style of [113]. On the integration of event
semantics and Montague/Frege ones see [16]. Mary walked from the university to
the capitol. $\lambda e\exists
x[\text{name}(x,\text{\leavevmode\ltxml@oqmark@open\textquotedblleft\penalty
10000\hskip-0.0002pt\hskip
0.0002ptMary\textquotedblright\ltxml@oqmark@close{}})\wedge\text{walk}(e)\wedge\text{agent}(e,x)\wedge\text{source}(e,u)\wedge\text{goal}(e,c)\wedge\exists\mathbf{v}[\operatorname{place}(\mathbf{v},u)\wedge\operatorname{place}(c,\mathbf{v})]]$

The sentence in (a) is true if the walking event in question took place, in
which case there is a path $\mathbf{v}$.

Oriented vector spaces have also been used to spell out a spatial semantics
for pointing gestures [88], in line with current (that is, post-Kaplanian)
pointing cone construals [4, 70, 91, 86]. An illustration is shown in Figure
17.

Figure 17: Illustrating a vector space semantics for pointing gestures. The
manikin points straight ahead, thereby excluding the bottom book from the set
of potential referents. The space “highlighted” by the pointing gesture is
defined in terms of vectors emanating from the index finger (the image is
adapted from [88, p. 112]; the book icon is taken from LaTeX’s beamer class,
[102]).

In sum, vector spaces have already proven to be adequate models for spatial
language as well as non-verbal demonstration acts. In the following, we show
how vector spaces can be extended to provide a formal model to interpret
iconic gestures, too.

###  3.5 Iconic models in vectorial gesture space

A main obstacle for spatial theories is to provide a mapping $\phi$ from
kinematic gesture representations to iconic models (cf. section 3) – that is
in our case to vector sequences in vector space. Such a mapping is needed for
an iconic gesture semantics to get off the ground in the first place. In the
following, we provide such a mapping, drawing on symbolic-computational work
by [85]. The basic idea is twofold but simple:

  * •

The notion of oriented vector space is used as a formal model of a speaker’s
gesture space.

  * •

The annotation predicates that represent gesture kinematics (section 3.1) are
re-interpreted in terms of vector sequences in vectorial gesture space.

This two-step process gives rise to a computational procedure to derive
vector-based iconic models from the form of a gesture – the first
characterizing feature of iconicity (the second one is the intersection of
these iconic models with the models of the spatial configurations of the
objects and events talked about, cf. section 1).

A vector space $\operatorname{space}(s)$ is anchored at each speaker $s$ (that
is, $s$ is placed at the origin of $V$, $\operatorname{place}(s,V)$). This
vector space is oriented along the anatomical planes which hypothetically
transect the speaker’s body, see Figure 18: Anatomical planes _Sagittal plane_
$\coloneqq\\{\mathbf{v}\in
D_{v}\mid\mathbf{v}\perp\textsc{up}\cup\textsc{rt}\vee\mathbf{v}\perp\textsc{-up}\cup\textsc{-rt}\\}$

_Transverse plane_ $\coloneqq\\{\mathbf{v}\in
D_{v}\mid\mathbf{v}\perp\textsc{ft}\cup\textsc{up}\vee\mathbf{v}\perp\textsc{-ft}\cup\textsc{-up}\\}$

_Coronal_ (or _Frontal_) _plane_ $\coloneqq\\{\mathbf{v}\in
D_{v}\mid\mathbf{v}\perp\textsc{rt}\cup\textsc{ft}\vee\mathbf{v}\perp\textsc{-rt}\cup\textsc{-ft}\\}$

Speaker-centered vector spaces provide a _semantic model for gesture spaces_ ,
that is, the space roughly in front of a speaker’s chest where hand and arm
movements are carried out.

up$-$uprt$-$rtft$-$ft (a)

up$-$uprt$-$rtft$-$ft (b)

up$-$uprt$-$rtft$-$ft (c)

Figure 18: Oriented planes in vector space.

With the formal notion of gesture space as vector space, a mapping from
kinematic gesture representation of simple, uni-directional movements to
vectors is straightforward: the directions encoded in gesture annotation
predicates are translated to orienting vectors, as spelled out in ().
annotation mnemonic vector towards TL towards left $\mathbf{-RT}$ TR towards
right $\mathbf{RT}$ UP upwards $\mathbf{UP}$ DN downwards $\mathbf{-UP}$ TB
towards body $\mathbf{-FT}$ AB away from body $\mathbf{FT}$

The mapping from () allows deriving an iconic model from a simple movement of
the hand, for example, from left to right: $\mathbf{RT}$. Moving on to vector
sequences, vectors are concatenated head-to-tail, while the concatenation
obeys the trajectory annotation _line_ vs. _arc_ , which distinguishes
roundish from angular paths (see section 3.1). A minimal example is shown in
(), where the iconic models emerging from vector sequence
$\mathbf{UP}>_{\text{line}}\mathbf{RT}$ respectively
$\mathbf{UP}>_{\text{arc}}\mathbf{RT}$ are given.
$\mathbf{UP}$$>$line$\mathbf{RT}$$\mathbf{UP}$$>$arc$\mathbf{RT}$

The resulting vector sequences,
$\mathbf{Z}=\mathbf{UP}>_{\text{line}}\mathbf{RT}$ and
$\mathbf{Z^{\prime}}=\mathbf{UP}>_{\text{arc}}\mathbf{RT}$, are both open,
that is $\mathbf{Z}[0]\neq\mathbf{Z}[1]$ and
$\mathbf{Z^{\prime}}[0]\neq\mathbf{Z^{\prime}}[1]$. Distinguishing open and
closed trajectories is important and is brought about by comparing the
starting and ending positions of vector trajectories in vectorial gesture
space: both are part of kinematic gesture representations (cf section 3.1).
This procedure – the annotation of gesture and form and its mapping onto
vector sequences – involves some degree of abstraction. For instance, the
sequence $\mathbf{Z}=\mathbf{UP}>_{\text{line}}\mathbf{RT}$ is compatible with
numerous iconic models and does not distinguish the following ones:
$\mathbf{UP}$$>$arc$\mathbf{RT}$$\mathbf{UP}$$>$arc$\mathbf{RT}$$\mathbf{UP}$$>$arc$\mathbf{RT}$

The procedure, however, constrains the iconic model along the lines of () to
be a bent trajectory (with a certain orientation, namely starting upwards,
going to the right). This degree of abstraction takes into account the
“sloppiness” of gesture performance [120, e.g.,].

We notate $>_{\text{line}}$ as $\perp$ and $>_{\text{arc}}$ as $\circ$.
Together with the vectorization of basic predicates in (18), there is a
complete vectorization of kinematic gesture representation, that is, a spell
out of function $\phi$ from spatial theories. The algebraic gesture derivation
‘$\operatorname{vec}(\gamma)$’ is given in (). Gesture vectorization function
$\operatorname{vec}(\gamma)$ as model for $\phi$.
$\operatorname{vec}(\mathbf{u}>_{\text{line}}\mathbf{v})=$
$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
traj &1.99997pt=&1.99997pt$\mathbf{u}\perp\mathbf{v}$\crcr}\vskip
1.65761pt}\hskip 0.0pt\right]\mskip-3.0mu$
$\operatorname{vec}(\mathbf{u}>_{\text{arc}}\mathbf{v})=$
$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
traj &1.99997pt=&1.99997pt$\mathbf{u}\circ\mathbf{v}$\crcr}\vskip
1.65761pt}\hskip 0.0pt\right]\mskip-3.0mu$
$\operatorname{vec}(\text{s-loc,e-loc})=\begin{cases}\text{sync :
traj}[0]=\text{traj}[1]&\text{if s-loc}=\text{e-loc}\\\ \text{sync:
traj}[0]\neq\text{traj}[1]&\text{else}\end{cases}$

The input of ‘$\operatorname{vec}$’ is a kinematic gesture representation
$\gamma$ as introduced in section 3.1.232323Obviously, ‘vec’ takes arguments
of different annotation types. This move is not uncommon in programming
languages, and we stick to it for brevity’s sake. Vectorization applies
progressively over movement annotations (a,b). Condition (c) is the closure
condition, which checks whether a given movement trajectory brings about a
closed or an open path. Such information is necessary, for instance, for shape
interpretations such as distinguishing a full circle from a four-fifths one.

Obviously, more interpretational complexity can be built on top of vector
interpretations. For current purposes, the system developed so far is
sufficient: we have introduced a basic computational procedure for spelling
out $\phi$, the so far missing function that produces iconic models from
kinematic gesture representations. Before detailing the semantics of gesture
in relation to speech, let us briefly exemplify the interpretation of the
spiral gesture shown in (1), (2) and (2). The new property here is that it
actually involves two motion aspects: an internal (rotation) and a
translational (going upwards) one. The corresponding annotation of the right
hand is shown in (). $\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
right hand\hfil\\\ handshape &1.99997pt$\mskip-3.0mu\left[\hskip
0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
shape &1.99997ptG \crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ boh
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
path &1.99997ptarc$>$arc$>$arc \\\ dir &1.99997ptMR$>$MF$>$ML\crcr}\vskip
1.65761pt}\hskip 0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ wrist
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
path &1.99997ptline \\\ dir &1.99997ptMU \\\ }\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ sync
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
s-loc &1.99997ptP-R \\\ e-loc &1.99997ptP-UR\crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$\ignorespaces\crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$

The two movements involved in the spiral gesture introduce two vector
sequences. The translational one determines an open path since the starting
and the end position differ. The output of the vectorization function (18) is
given in (): $\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
traj1 &1.99997pt=&1.99997pt$\mathbf{RT}\circ\mathbf{FT}\circ\mathbf{-RT}$ \\\
traj2 &1.99997pt=&1.99997pt$\mathbf{UP}$ \\\ sync
&1.99997pt:&1.99997pttraj2[0] $\neq$ traj2[1] \crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$

The “vectorization” of the back of the hand (BoH) rotation gives rise to the
iconic model to the left in (), the wrist movement introduces a straight
vector pointing upwards in gesture space. The _combined_ movement of BoH and
wrist results in a wound, spiral model.242424Since the iconic model, in this
case, is the result of a blend of two movements, one might think of such
gestures as portmanteau gestures [83, 193 sqq.]. BoH:

$\mathbf{RT}$$>$arc$\mathbf{FT}$$>$arc$\mathbf{-RT}$

Wrist: $\mathbf{UP}$line Combined:

Let us take stock: following spatial theories of iconic gesture meaning, we
formally modelled the visual content of an iconic gesture in terms of a
(possibly dynamic) spatial structure, the iconic model. A crucial contribution
is a spell-out for mapping $\phi$, a function that brings about a systematic
construction of iconic models from kinematic gesture representations. How are
iconic models related to speech (the second aspect of iconicity)? Following
ideas developed by [41], speech and gesture are basically related in terms of
intersectivity: the spatial configuration of the objects and events talked
about is such that it intersects with the iconic model derived from the
gesture. To complete our vector space semantics of iconic gestures, we assume
a linear map between iconic models and real-world spatial configuration. A
linear map is a mapping $V\mapsto W$ of vector spaces $V$ and $W$ that
preserves the operations of vector addition and scalar multiplication. Hence,
iconicity in vector spaces allows for some linear “distortion” between iconic
models and spatial configurations. The most obvious one is the linear
transformation of scaling: a gestural movement does not necessarily depict in
life-size, so the linear map allows scaled mappings, both larger and smaller.
But also rotation and reflection are possible, both are bound up with
perspective.

We are now in a position to spell out a compositional semantics for
speech–gesture integration. As the compositional “backbone” we assume a
multimodal grammar along the lines of [1, 83]. Such grammar extensions combine
an iconic gesture with its lexical or phrasal affiliate. Schematically, a
multimodal utterance $\alpha[\beta/\gamma]$ consisting of a sentence $\alpha$,
a co-speech gesture $\gamma$ and its affiliate $\beta$ is true, iff $\alpha$
is true and there is an intersection of the iconic model of $\gamma$ and the
spatial configuration ‘$\operatorname{space}(\beta)$’ projected from
$\beta$.252525For the sake of simplicity, we assume that a multimodal
utterance involves just one gesture, but the set-up scales up
straightforwardly. Given a standard Montagovian model, which is extended by
vector spaces as outlined in section 3.4, the semantics of multimodal
utterances is compositionally derived straightforwardly: $\alpha$, including
constituent $\beta$, is recursively interpreted as usual [52, 17, cf.].
Additionally, the vector space projected from the verbal affiliate is
evaluated against the linear maps from the vector space obtained from the
vectorization of the iconic gesture. The recursive, truth-functional
interpretation of a multimodal utterance within a simple quantifier-free,
first-order fragment involving transitive verbs is spelled out in (). The only
new, multimodal rule is given in (d). _Semantic fragment for multimodal
utterances._

$\llbracket\text{[\textsubscript{A}B]}\rrbracket=\llbracket\text{B}\rrbracket$,
for any lexical category A and lexical entry B.

$\llbracket\text{[NP VP]}\rrbracket=1$ iff
$\llbracket\text{NP}\rrbracket\in\llbracket\text{VP}\rrbracket$.

$\llbracket\text{[V NP]}\rrbracket=\\{x\mid\langle
x,\llbracket\text{NP}\rrbracket\rangle\in\llbracket\text{V}\rrbracket\\}$

$\llbracket\text{[\textsubscript{MM}A $\gamma$]}\rrbracket=\\{f\mid
f:\operatorname{vec}{\gamma}\mapsto\operatorname{space}(\llbracket\text{A}\rrbracket)\\}$,
for a multimodal category MM,262626From multimodal grammars, that is, a
situated word [1] or a multimodal ensemble [83]. an iconic gesture $\gamma$,
and a linear map $f$.

A multimodal utterance $\alpha[\beta/\gamma]$ including an iconic gesture
$\gamma$ and its affiliate $\beta$ is true iff
$\llbracket\text{[$\alpha$]}\rrbracket=1\wedge\exists
f.f:\operatorname{vec}(\gamma)\mapsto\operatorname{space}(\llbracket\text{[$\beta$]}\rrbracket)$.

Consider, for example, a sentence that includes the noun staircase and an
accompanying spiral gesture $\gamma$. The denotation of staircase in some
worlds is like in Figure 19a and like Figure 19b in other worlds.

(a) Grand Staircase (Robert N. Clinton at www.cybershutterbug.com, CC BY-NC-ND
4.0 DEED).

(b) Concrete Spiral Staircase (Pat Joyce at www.flickr.com, CC BY-NC 2.0
DEED).

Figure 19: Two kinds of staircase denotations.

(a) Axis path of Grand Staircase within its oriented vector space.

(b) Axis path of Concrete Spiral Staircase within its oriented vector space.

Figure 20: Axis vectors within
‘$\operatorname{space}(\llbracket\text{staircase}\rrbracket)$’.

The axis path of the staircase in Figure 19a within its vector space
‘$\operatorname{space}(\llbracket\text{staircase}\rrbracket)$’ is , shown in
Figure 20a; the one of the staircase in Figure 19b is

, shown in Figure 20b. A linear map with the domain of the spiral gesture,

$\operatorname{vec}$($\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
right hand\hfil\\\ handshape &1.99997pt$\mskip-3.0mu\left[\hskip
0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
shape &1.99997ptG \crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ boh
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
path &1.99997ptarc$>$arc$>$arc \\\ dir &1.99997ptMR$>$MF$>$ML\crcr}\vskip
1.65761pt}\hskip 0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ wrist
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
path &1.99997ptline \\\ dir &1.99997ptMU \\\ }\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$\ignorespaces\\\ sync
&1.99997pt$\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
s-loc &1.99997ptP-R \\\ e-loc &1.99997ptP-UR\crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$\ignorespaces\crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$ ) $=$ $\mskip-3.0mu\left[\hskip 0.0pt\vbox{\vskip
1.65761pt\halign{\avmjfont\strut#\unskip\hfil&&\hskip\avmjhskip\avmjvalfont#\unskip\hfil\cr
traj1 &1.99997pt=&1.99997pt$\mathbf{RT}\circ\mathbf{FT}\circ\mathbf{-RT}$ \\\
traj2 &1.99997pt=&1.99997pt$\mathbf{UP}$ \\\ sync
&1.99997pt:&1.99997pttraj2[0] $\neq$ traj2[1] \crcr}\vskip 1.65761pt}\hskip
0.0pt\right]\mskip-3.0mu$, only exists for the latter, but not the former
staircase’s axis path. Hence, the spiral gesture is compatible with worlds
where $\llbracket\text{staircase}\rrbracket$ is a singleton or a proper set of
objects of the kind displayed in Figures 19b and 20b, respectively, but not of
the kind shown in Figures 19a and 20a, respectively. Hence, spatial theories
give rise to a well-behaved, truth-functional semantics of iconic gestures as
a conservative extension of a Montagovian semantic framework. Note that (d)
provides us with a first formal notion of _multimodal well-formedness_ ,
requested by desideratum D2 in section 2: a speech–gesture mismatch occurs if
there is no linear map $f$ which embeds the iconic model into the spatial
configuration projected from the verbal constituents. The outline of a vector
space-based iconic gesture semantics is shown in Figure 21, where the
instances of the bottom row model components are given as boldface labels. The
semantics captures the visuo-spatial contribution of iconic gestures and is
semantic since it provides truth conditions. Given the spell-out of function
$\phi$, the semantics is also compositional. Having established a solid
semantic backbone, we turn to how it interacts with labelling theories.

GestureFrame of referenceVerbal constituentIconic modelVector sequenceSpatial
frame of referenceVector
space$\llbracket\text{$\cdot$}\rrbracket$$\phi$$\llbracket\text{$\cdot$}\rrbracket$Lociconicitylinear
map

Figure 21: Vector space semantics as spatial theory

##  4 Perceptual classification and extemplification

> “On the one hand, the Frege/Montague research program, based on the idea
> that truth-conditions are the core ingredient of clause meaning and that
> meanings of complex expressions are computed from the meanings of the parts,
> has been extremely successful. On the other, it did not really address the
> central question: What, precisely, are the meanings of the smallest parts,
> the meanings of words, or rather, lexemes?”
>
> Manfred [74, p. 223 ]

The spatial, vectorial theory of iconic gesture meaning harmonizes iconic
gestures with truth-conditional semantics. However, just as semantics in the
Frege/Montague tradition cannot provide an answer to the question “What _is_
the meaning of $\ulcorner\text{a certain word}\urcorner$?”, it also lacks an
answer to the question “What _is_ the meaning of $\ulcorner\text{a certain
gesture}\urcorner$?” We will see that an answer to the first question also
provides a solution to the second one.

But when is meaning at stake in communicating with gestures at all? There are
at least two kinds of situations: (i) gesture uptake [45] and clarification
interaction [39]; (ii) the analysis of gestures within gesture studies [47,
e.g.,]. Note that both kinds of situations involve quite different
perspectives: the first one is the perspective of the interlocutors in
interaction, and the second one is that of the researcher studying this
interaction. Let us briefly turn to each of them.

Somewhat surprisingly, addressees do not always informationally evaluate the
speaker’s gestures. This has been tested, for instance, by [45] in a drawing
response study, where participants had to draw a situation that they saw
described in a video of a speaker using speech and gesture. The speaker’s
gesture included a target gesture, that is, a gesture that displayed
information not verbalized in speech (e.g., the direction of a movement). The
authors found that the drawings included the information exclusively gestured
more often if the speaker gazed at the target gesture. Hence, interlocutors
themselves make a distinction between (mostly peripheral) _seeing_ a gesture
and _interpreting_ a gesture (gesture uptake, or informational evaluation, to
which we turn shortly).272727This is reminiscent of the twofoldedness of
gestures pointed out in section 1: seeing gestures (iconic model) and seeing
something in gesture (informational evaluation).

Iconic gestures can also be part of clarification interaction, which is of
interest from a dialogue semantics point of view [51]. Consider again the
spiral gesture example, re-given in its English translation in (). Following
A’s utterance, addressee B can inquire about the gesture by either reproducing
it (1), or by informationally evaluating it (2). A: I think that should be
staircases
B: (1)

? (2) Do you mean spiral staircases?

B’s two kinds of responses correspond to two different clarification
strategies: confirmation questions and intended meaning requests.282828On
different types of clarification requests see [36, §6.2 ]. In (1), B asks to
confirm the perceived gestural movement, or the intended iconic model,
respectively. Thus, (1) is a nonverbal variant of verbal “Have I heard
correctly? Did you say $u$”, or “Do you mean $u$?”, for some verbal
constituent $u$. This reading does not seem to be available for (2), however,
which addresses the intended meaning of the gesture: “Do you mean $z$ as the
content of $\gamma$?”292929This confirms, to our minds, the diagnosis of [41,
p. 91 sq. ], namely that the “meaning” (i.e., spatial model) of iconic
gestures has to be kept apart from communicating with these gestures.

In gesture studies and Conversational Analysis, _describing_ what the hands do
is a longstanding and useful way of characterizing a gesture’s meaning. For
instance, “Palm-Up-Open-Hand is recurrently used to communicatively present,
give, offer, show a discursive object” [108, p. 279]. A certain gesture form
is labelled to perform a certain (range of potential) action(s), and based on
this description, the informal analysis of the gesture and its relation to
speech can be given.

This points to an important, albeit trivial-sounding fact, namely that _the
interpretation of a multimodal utterance is conditioned on the informational
evaluation of the gesture involved._ If this is true, then a gesture fails to
contribute content, _unless_ interlocutors agree on an informational
evaluation of the gesture. Using the nondeniability test for gestures [27,
126], () and () show that this is indeed the case.303030The quoted authors
conclude from these tests that gestures contribute non-at-issue content.
However, this follows only if failing to contribute at-issue content is the
same as contributing non-at-issue content, as has been pointed out by [56, p.
326 sq. ]. We do not subscribe to this implicit assumption, since according to
spatial gesture semantics the gesture in this case simply remains sub-
linguistic, visual – semantically inert. A: Inside the hall was an imposing
staircase. # B: No, that’s not true. The staircase was actually straight.

A: Inside the hall was an imposing  staircase. B: Do you mean a spiral
staircase? A: Yes. B: No, that’s not true. The staircase was actually
straight.

Non-deniability is a feature of non-at-issue contributions. However, other
contexts induce non-deniable contents, too, in particular the antecedents of
indicative conditional sentences. Their consequences cannot be picked out by
negation: The negation of a sentence of the form “If $A$ then $C$” is either
the conjunction “$A$ and not $C$” or the conditional “If $A$ then not $C$”
[30, cf.].313131A weak version of denial of indicative conditionals has “[…]
then possibly not $C$” as consequence. A denying continuation of an indicative
conditional targets the asserted implication: A: If the staircase is spiral,
it is an imposing one. # B: No, that’s not true. The staircase is imposing. B:
No, that’s not true. The staircase is imposing even without being spiral.

Hence, we would expect contexts of conditioned, but not explicitly agreed,
gesture interpretation to involve nondeniable consequences. Accordingly, () is
in line with the predictions of our theory: If  is interpreted as “spiral”,
then in the hall was an imposing spiral staircase. # No, that’s not true. The
staircase was actually straight. No, that’s not true. The staircase was
actually straight, even if you interpret  as spiral.

Things are different if the antecedent condition is fulfilled, which happens
if the gesture is lifted to a quasi-linguistic status due to explicitly agreed
informational evaluation, as in (4). From “If $A$ then $C$” and “$A$”, “$C$”
follows and be negated.

[28] distinguishes two additional non-at-issue tests for co-speech gestures,
projection, and ellipsis. Let us turn to each in turn. The first one tests the
fact that non-at-issue contents project across sentential operators: It is not
the case that in the hall was an imposing  staircase. # No, the staircase was
actually straight.

Again, this behaviour is explained by conditioned interpretation: If  is
interpreted as “spiral”, then it is not the case that in the hall was an
imposing spiral staircase. # No, the staircase was actually straight.

The second test says that the co-speech gesture contribution is ignored in
ellipsis constructions: In the hall was an imposing  staircase, and a window,
too.

Ignoring for now that resolving elliptical constructions is a complex process
in itself [37, see, e.g., ], then a conditional context produces a
straightforward interpretation of the multimodal utterance, where spiral does
not need to take scope over window: If  is interpreted as “spiral”, then in
the hall was an imposing spiral staircase, and a window, too.

Hence, iconic gesture semantics can _explain_ observations concerning the
information status of iconic gestures, namely in terms of conditioned
interpretation.323232Some authors had the intuition that the ellipsis test
brings about a different result for pro-speech gestures [127]. As will be
discussed in section 5 by the example of non-lexicalized iconic models, we
show that there is no clear difference in the information status of co- and
pro-speech gestures. The latter, when taken to be produced in purpose for the
sake of communication (i.e., as _foreground_ not as _background_ gesture;
[24]) may invoke a stronger obligation for informational evaluation,
potentially explaining the gradience of information status [5].

Schematically, using again the template $\alpha[\gamma/\beta]$ for a
multimodal utterance $\alpha$ which includes an affiliate $\beta$ and the
affiliated gesture $\gamma$: _Information-evaluation conditioned
interpretation_ :

If a gesture is interpreted to mean $m$, then the utterance is interpreted as
$\alpha[R(m,\beta)]$.

‘$R(m,\beta)$’ in () means that $m$, the supposed gestural meaning, is applied
to the meaning of its affiliate $\beta$ via a relation $R$. In the simplest
case, $R$ is identity, namely when $m$ extendedly exemplifies $\beta$ (in a
sense explained shortly in section 4.2) and is informationally evaluated as
$\beta$. This is the standard case of affiliation, in which the gesture
intuitively “reduplicates” its affiliate, see examples (2a,b) in section 2 –
and this is the reason why gestures, even if informationally evaluated, mostly
remain semantically vacuous. Only if $m\neq\beta$, as in the spiral staircase
example, an inference step is needed to combine gesture interpretation and
affiliate meaning. Here, standard mechanisms from dynamic semantics resolving
$R$ apply, as is pointed out in section 4.3.

In the following, we develop a labelling approach based on visual perception
theories and perceptual classification. The objects of classification will be
iconic models, thereby reconciling spatial and labelling theories. To make
this work, we effect a non-trivial modification of the architecture of the
semantic framework: we employ vector models not only as visual denotations,
but also as representations of _the meaning of lexical items_. Having the same
kinds of representations available for denotations, iconic models, and
meanings paves the way for spelling out a computational approach to extended
exemplification/classification and the (in)congruence between words and
gesture interpretations beyond denotational vacuity (see section 3). The
architecture of the lexical, classifier-based extension of spatial theories is
shown in Figure 1, re-given as Figure 22.333333The iconic gesture semantics
theory is, of course, not a theory of speech–gesture production. However, it
should be emphasized that it is compatible with most multimodal production
models (apart from the fact that it incorporates empirical findings on (lack
of) informational evaluation, see [45] and the above discussion). For
instance, the _Sketch Model_ [123] assumes an abstract spatio-temporal
representation alongside verbal ones. This clearly corresponds to our
separation of iconic models and spatial frames of reference, respectively the
lexicon. The _Lexical Access Model_ [72] emphasizes a gesture’s facilitation
of word retrieval [46, see also]; the underlying connection between iconic
models and verbal items is precisely captured by the extemplification
relation. The _Interface Model_ rests on a continuous “negotiation process”
between verbal and gestural production channels [68]. The iconicity relation
mediates between these two generation streams, but as it stands is not a
temporal, incremental process. It is less clear to our minds, however, how
iconic gesture semantics relates to the heterogeneous “multimodal idea units”
postulated by the _Growth Point Model_ [100]. In any case, we believe that it
is an advantage that our iconic gesture semantics is compatible with a wide
range of empirical findings and cognitive, psycholinguistic models.

We motivate the use of vector intensions from biological motion perception in
section 4.1, which provides the conceptual interface for _computing_
exemplification relations.343434Hence, we relate semantics to non-linguistic
cognitive activities such as perception, but contra [81, p. 19 ], we do not
think that “confusion comes of mixing these two topics”. We then develop an
exemplification heuristic as a “linguistic toolkit” for semanticists working
on gestures (section 4.2). In section 4.3, we briefly show how gesture
integrates with speech if relation $R$ from () is not identity by re-using
standard methods from dynamic semantics. Extemplification is applied in
section 4.4 to showcase semantic analyses of some constructed, but widely
discussed examples.

GestureFrame of referenceVerbal constituentIconic modelSpatial frame of
reference$\llbracket\text{$\cdot$}\rrbracket$$\phi$$\llbracket\text{$\cdot$}\rrbracket$LociconicityVerbal
constituentLexicon (frames)connected toindirect
interactionclassifiesexemplifiesextendedly

Figure 22: Classifier-based iconic gesture semantics

###  4.1 Excursus: Vectorial biological movement analysis and perceptual
classification

Using geometric representations as representations for meanings is not new in
natural language semantics and the philosophy of language [134, 143,
e.g.,].353535[15] makes the case for _epistemic intensions_ , but does not
provide a formal (e.g., vector based) model. In cognitive science, it is
commonplace that lexical items have both symbolic and visual meaning
components [112]. A related dual coding approach has already been developed
for the synthesis of three-dimensional iconic gestures in terms of _imagistic
description trees_ [129]. Computational linguistics developed “words-as-
classifiers” approaches [67, 77], where the meaning of perception-related
words is perceptually grounded [50, 130]. Such perceptual groundings – like
speaker judgments [110] – can even be visualized (as illustrated in Figure
16). [6] developed a comparable approach in semantics and philosophy of
language. Following this direction of thrust, it seems appropriate to underpin
visual communication with insights from visual perception – ultimately,
gestures are physical actions. Let us consider the example of biological
motion and motion verbs. Motion verbs vary along two dimensions: manner and
path [31]. The eigenmovement distinguishes motion verbs according to manner,
regardless of the distance travelled: run walk stroll saunter
…$\left\\{\vbox{\hrule
height=36.87064pt,depth=36.87064pt,width=0.0pt}\right.$$\left.\vbox{\hrule
height=36.87064pt,depth=36.87064pt,width=0.0pt}\right\\}$

Translational movement gives rise to a path that distinguishes motion verbs
irrespective of the manner of motion: run detour circle criss-cross
…$\left\\{\vbox{\hrule
height=37.13454pt,depth=37.13454pt,width=0.0pt}\right.$$\left.\vbox{\hrule
height=37.13454pt,depth=37.13454pt,width=0.0pt}\right\\}$

The path component – the translational dimension of motions – is already
covered by the vector denotations within the spatial model (see section 3),
which truth-conditionally distinguish the verbs in (). But what about the
manner dimension? In an important series of experimental studies on the
perception of biological motion, [60, 61, 62] investigated, among others, the
difference between walking and running. How are we able to consistently tell
both motion manners apart across different human (and presumably some non-
human) individuals? Is there an abstract perceptual commonality between
running events on the one hand and walking events on the other hand? Johansson
and colleagues placed little lights at the anatomical joints that actually
bring about the movement (the so-called _motion carriers_) – see Figure 23a.
The recorded light pattern (Figure 23b) is then shown to participants, who
“saw” and correctly classified a walking event. “How can 10 points moving
simultaneously on a screen in a rather irregular way give such a vivid and
definite impression of human walking?” [60, p. 204]. An answer to this
question was found in geometric analyses of the temporal stimulus pattern.
Walking is characterized by two horizontal trajectories (due to hip and knee
carriers) and an up-and-down sequence (ankle) – see Figure 23c. Factoring out
common movement shares, the kernel percept of a walking event is the _abstract
vector model_ shown in Figure 23d. If we observe something that looks like
this vector model, we can classify it as walking.

(a) Stimulus for walking

(b) Carriers of walking movement

$A_{1}$$A_{2}$$A_{3}$$A_{4}$$A_{1}$$B_{1}$$B_{2}$$B_{3}$$B_{4}$$B_{1}$$C_{1}$$C_{2}$$C_{3}$hipkneeankle

(c) Model of carrier movement

$A$$B_{1}$$B_{4}$$A_{1}\rightarrow A_{4}$

(d) Abstract vector model of walking

Figure 23: Motion perception: walking. (Taken from, respectively reproduced
after [60, p. 202 and 208])

Now singling out walking events is exactly what the meaning of the verb walk
is supposed to achieve, and what is “pre-compiled” in model-theoretic
semantics.363636As is common practice in semantics, we do not distinguish
sharply between possible worlds semantics and model-theoretic semantics, since
it is usually of no practical significance and does not impact the present
discussion, but see [137, 138] for a more careful discussion. Accordingly, the
model in Figure 23d provides a representation of the intensional meaning of
walk. We, following [83], refer to this visuo-spatial representation of an
intension as _conceptual vector meaning_ , or cvm for short. Arguably, the
meaning of any visuo-spatial expression comes with a cvm (cf. _dual coding_ ;
[112]). As above indicated, cvms are studied in cognitive science, and
computational and lexical semantics as _perceptual classifiers_ , but they are
still alien to compositional semantics in the Frege/Montague tradition (we
briefly return to this issue in the conclusions in section 6).373737There is
some related work in the framework of conceptual semantics, however, which,
among others, addresses the problem of how we are able to talk about what we
see [58]. On this approach, a theory of language is connected to a theory of
vision, namely the 3-D models of [94]. Marr’s model is specialized for object
recognition and identification, vector models seem to be better suited for
capturing biological movement [95, but see]. A semantic rendering of Marr’s
model also underlies the logic of vision of [25], but it remains purely
extensional. However, we believe and have argued at some length, that
perceptual classification is key to understanding iconic meaning in the sense
of the informational evaluation of vector models acted out by iconic gestures.
Hence, to make iconic gesture semantics accessible to mainstream formal
semantics, we therefore propose an _extended exemplification heuristic_ for
gesture interpretation. The heuristic will be elaborated in the following
section, but the main idea is as follows. To get things started, the meaning
of the lexical entry in (a) is replaced by one involving a cvm, as illustrated
in (b). $\llbracket\text{walk}\rrbracket=\lambda x.\lambda
e[\text{walk}(e)\wedge\text{agent}(e,x)\wedge\exists\mathbf{v}[\text{place-
path}(e,\mathbf{v})]]$ $\llbracket\text{walk}\rrbracket=\lambda x.\lambda
e[\text{walk-
cvm}(e)=1\wedge\text{agent}(e,x)\wedge\exists\mathbf{v}[\text{place-
path}(e,\mathbf{v})]]$

(a) is standard: the meaning of walk, $\llbracket\text{walk}\rrbracket$, is
spelled out in terms of a function of type $\\{0,1\\}^{D_{s}^{D_{e}}}$. (b)
adds that the set of events $E$ characterized by the embedded function is such
that the classification of each event $e\in E$ according to the perceptual
walking classifier walk-cvm is successful.383838Computationally and
conceptually, a probabilistic version of the classifier is more attractive,
but for the sake of keeping things simple we assume a binary one, just
returning ‘true’ (1) or ‘false’ (0). The point of (b) is that it applies to
iconic gestures, too: the lexical meaning can be used to interpret a gestural
movement that looks like walking as walking, as exemplified in ().

$x$$\text{walk-cvm}(\gamma)=1$$\mathbf{v}$

The gesture in () is compatible with walk, but fails for, e.g., stagger,
crawl, give, ride, etc. because of different, incompatible cvms.393939Of
course, distinguishing gestures from real-world events follows from a double
classification of the gesture according to a perceptual cvm and a hand
classifier (a hand-cvm returns false for real-world events). We simply
presuppose this marginal detail in the following. Given the successful
informational evaluation of the moving fingers in terms of walking, this
predicate figures as input to the gesture-conditioned affiliation principle
(4) and provides an interpretation of the multimodal utterance of which the
walking gesture is part of.

###  4.2 Linguistic toolkit: Extended exemplification as gesture
interpretation

As a simple example, consider the toy model in (a). The denotation of green is
the set of three green objects. Given this, any object within the denotation
can be used to _exemplify_ , $\models$ex, green, for instance, the green
circle as in (b). This is the influential reversed denotation relation (i.e.,
exemplification) introduced by [42].
$\llbracket\text{green}\rrbracket=\\{\leavevmode\hbox to9.96pt{\vbox
to9.96pt{\pgfpicture\makeatletter\hbox{\hskip 4.97923pt\lower-4.97923pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\hbox to0.0pt{\hbox to0.0pt{\hbox
to0.0pt{ {}{{}}{}
{}{{}}{}{}{}{}{{}}{}{}\pgfsys@moveto{-20.12631pt}{-34.71272pt}\pgfsys@moveto{-20.12631pt}{-34.71272pt}\pgfsys@lineto{-20.12631pt}{43.24026pt}\pgfsys@lineto{73.68887pt}{43.24026pt}\pgfsys@lineto{73.68887pt}{-34.71272pt}\pgfsys@closepath\pgfsys@moveto{73.68887pt}{43.24026pt}\pgfsys@stroke\pgfsys@invoke{
} \hss} {}{{}}{}
{}{{}}{}{}{}{}{{}}{}{}\pgfsys@moveto{-26.72713pt}{-4.50554pt}\pgfsys@moveto{-26.72713pt}{-4.50554pt}\pgfsys@lineto{-26.72713pt}{61.41106pt}\pgfsys@lineto{25.47713pt}{61.41106pt}\pgfsys@lineto{25.47713pt}{-4.50554pt}\pgfsys@closepath\pgfsys@moveto{25.47713pt}{61.41106pt}\pgfsys@stroke\pgfsys@invoke{
} \hss} {}{{}}{}
{}{{}}{}{}{}{}{{}}{}{}\pgfsys@moveto{-33.5583pt}{-90.46382pt}\pgfsys@moveto{-33.5583pt}{-90.46382pt}\pgfsys@lineto{-33.5583pt}{5.10555pt}\pgfsys@lineto{33.5583pt}{5.10555pt}\pgfsys@lineto{33.5583pt}{-90.46382pt}\pgfsys@closepath\pgfsys@moveto{33.5583pt}{5.10555pt}\pgfsys@stroke\pgfsys@invoke{
} \hss}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0,0.8,0}\pgfsys@color@rgb@fill{0}{0.8}{0}\pgfsys@invoke{
}{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}
{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0,0.8,0}\pgfsys@color@rgb@fill{0}{0.8}{0}\pgfsys@invoke{
}\pgfsys@rect{-4.97923pt}{-4.97923pt}{9.95847pt}{9.95847pt}\pgfsys@fill\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}},\leavevmode\hbox
to9.96pt{\vbox to9.96pt{\pgfpicture\makeatletter\hbox{\hskip
4.97923pt\lower-4.97923pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\hbox to0.0pt{\hbox to0.0pt{\hbox
to0.0pt{ {}{{}}{}
{}{{}}{}{}{}{}{{}}{}{}\pgfsys@moveto{-20.12631pt}{-34.71272pt}\pgfsys@moveto{-20.12631pt}{-34.71272pt}\pgfsys@lineto{-20.12631pt}{43.24026pt}\pgfsys@lineto{73.68887pt}{43.24026pt}\pgfsys@lineto{73.68887pt}{-34.71272pt}\pgfsys@closepath\pgfsys@moveto{73.68887pt}{43.24026pt}\pgfsys@stroke\pgfsys@invoke{
} \hss} {}{{}}{}
{}{{}}{}{}{}{}{{}}{}{}\pgfsys@moveto{-26.72713pt}{-4.50554pt}\pgfsys@moveto{-26.72713pt}{-4.50554pt}\pgfsys@lineto{-26.72713pt}{61.41106pt}\pgfsys@lineto{25.47713pt}{61.41106pt}\pgfsys@lineto{25.47713pt}{-4.50554pt}\pgfsys@closepath\pgfsys@moveto{25.47713pt}{61.41106pt}\pgfsys@stroke\pgfsys@invoke{
} \hss} {}{{}}{}
{}{{}}{}{}{}{}{{}}{}{}\pgfsys@moveto{-33.5583pt}{-90.46382pt}\pgfsys@moveto{-33.5583pt}{-90.46382pt}\pgfsys@lineto{-33.5583pt}{5.10555pt}\pgfsys@lineto{33.5583pt}{5.10555pt}\pgfsys@lineto{33.5583pt}{-90.46382pt}\pgfsys@closepath\pgfsys@moveto{33.5583pt}{5.10555pt}\pgfsys@stroke\pgfsys@invoke{
} \hss}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0,0.7,0}\pgfsys@color@rgb@fill{0}{0.7}{0}\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0,0.7,0}\pgfsys@color@rgb@fill{0}{0.7}{0}\pgfsys@invoke{
}\pgfsys@moveto{4.97923pt}{0.0pt}\pgfsys@curveto{4.97923pt}{2.74998pt}{2.74998pt}{4.97923pt}{0.0pt}{4.97923pt}\pgfsys@curveto{-2.74998pt}{4.97923pt}{-4.97923pt}{2.74998pt}{-4.97923pt}{0.0pt}\pgfsys@curveto{-4.97923pt}{-2.74998pt}{-2.74998pt}{-4.97923pt}{0.0pt}{-4.97923pt}\pgfsys@curveto{2.74998pt}{-4.97923pt}{4.97923pt}{-2.74998pt}{4.97923pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@fill\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}},\leavevmode\hbox
to14.71pt{\vbox to12.19pt{\pgfpicture\makeatletter\hbox{\hskip
3.33305pt\lower-6.09416pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\hbox to0.0pt{\hbox to0.0pt{\hbox
to0.0pt{ {}{{}}{}
{}{{}}{}{}{}{}{{}}{}{}\pgfsys@moveto{-20.12631pt}{-34.71272pt}\pgfsys@moveto{-20.12631pt}{-34.71272pt}\pgfsys@lineto{-20.12631pt}{43.24026pt}\pgfsys@lineto{73.68887pt}{43.24026pt}\pgfsys@lineto{73.68887pt}{-34.71272pt}\pgfsys@closepath\pgfsys@moveto{73.68887pt}{43.24026pt}\pgfsys@stroke\pgfsys@invoke{
} \hss} {}{{}}{}
{}{{}}{}{}{}{}{{}}{}{}\pgfsys@moveto{-26.72713pt}{-4.50554pt}\pgfsys@moveto{-26.72713pt}{-4.50554pt}\pgfsys@lineto{-26.72713pt}{61.41106pt}\pgfsys@lineto{25.47713pt}{61.41106pt}\pgfsys@lineto{25.47713pt}{-4.50554pt}\pgfsys@closepath\pgfsys@moveto{25.47713pt}{61.41106pt}\pgfsys@stroke\pgfsys@invoke{
} \hss} {}{{}}{}
{}{{}}{}{}{}{}{{}}{}{}\pgfsys@moveto{-33.5583pt}{-90.46382pt}\pgfsys@moveto{-33.5583pt}{-90.46382pt}\pgfsys@lineto{-33.5583pt}{5.10555pt}\pgfsys@lineto{33.5583pt}{5.10555pt}\pgfsys@lineto{33.5583pt}{-90.46382pt}\pgfsys@closepath\pgfsys@moveto{33.5583pt}{5.10555pt}\pgfsys@stroke\pgfsys@invoke{
} \hss}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0,0.75,0}\pgfsys@color@rgb@fill{0}{0.75}{0}\pgfsys@invoke{
}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{{}}{}{{{}}{{}}{}{}{}}{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgffillcolor}{rgb}{0,0.75,0}\pgfsys@color@rgb@fill{0}{0.75}{0}\pgfsys@invoke{
}\pgfsys@moveto{11.37955pt}{0.0pt}\pgfsys@lineto{-3.33305pt}{6.09416pt}\pgfsys@lineto{-3.33305pt}{-6.09416pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{0.0pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\}$ $\models$ex green

Conversely, we can say that the predicate green can be used to _label_ or
_classify_ the green circle. The exemplification of transitive and $n$-place
predicates in general is brought about by collections of objects. Consider
throwing, as in the throwing-a-dagger example (2) in section 2. The lexical
semantics of throw is given in (). $\llbracket\text{throw}\rrbracket=\lambda
y.\lambda x.\lambda e[\text{throw-
cvm}(e)=1\wedge\text{agent}(e,x)\wedge\text{theme}(e,y)\wedge\exists\mathbf{v}[\operatorname{place-
path}(y,\mathbf{v})]]$

() gives rise to the following exemplification conditions: if there is a body
movement which looks like throwing (‘$\text{throw-cvm}(e)=1$’), performed by
$x$, and if there is something acted upon (‘theme($y$)’) and that something is
dislocated (‘$\operatorname{place-path}(y,\mathbf{v})$’; we abstract over
time), we can classify this event $e$ as a throwing event. Hence, a real-world
event $e$ exemplifies a predicate constant $P$ if the event provides a witness
for each of $P$’s arguments, and only for the arguments. We call this _minimal
exemplification_.404040Truthmaker accounts would say that $e$ is a truthmaker
for $P$ [106]. Any extended event $e^{\prime}$ which includes $e$ will also
exemplify that predicate, but not minimally.414141Being confined to minimal
exemplification exempts from considerations of situational upwards persistence
[19]. Note that more specific predicates can be exemplified, such as being
thrown by Mike, in which case Mike needs to be the agent of $e$, and so on.

This semiotic idea, we argue, also underlies the interpretation of iconic
gestures, but with a twist: exemplification can only be applied to iconic
gestures straightforwardly, if iconic gestures are taken to be objects or
events in $D_{e}$ and $D_{s}$, respectively. But a throwing event is different
from a throwing gesture: the gesture _simulates_ the real-world event. Being
an event simulation rather than a real-world event itself, a gesture abstracts
away over at least one object that would have been involved in the real-world
event.424242Gesture studies speak of “ad hoc abstraction” here and provide an
interpretation drawing on metonymy [103, p. 1747]. A throwing gesture (as in
example 2 in section 2), in contrast to a throwing event, does not involve a
theme (the object thrown) and therefore no dislocation path. In this case, the
interpretation of the gesture as throwing involves a “virtual object”, which
can be thought of as a visual presupposition, since it is a precondition for
interpreting the movement as throwing, regardless of linguistic context (e.g.,
negation). It still does involve the gesturer as an agent and a body movement
which looks like throwing.

Explicit representations of intensions in terms of cvms provide the conceptual
tools for expressing the required distinctions and abstractions to move from
exemplification to _extended exemplification_ , $\models$ext, or
_extemplification_ as a short coinage.434343This is a significant improvement
over [83], who uses Goodmanian exemplification, inheriting the above-discussed
problems. Schematically: _Extended exemplification as gesture interpretation_
A gesture $\gamma$ extemplifies a linguistic constituent $p$, $\gamma$
$\models$ext $p$, if $\text{$p$-cvm}(\gamma)=1$ and $\gamma$ is minimal wrt.
$p$. $\gamma$ is minimal wrt. $p$ iff there is a bijective mapping between (i)
form features of $\gamma$, or (ii) visual, presupposed features of $\gamma$
and the arguments of $p$. If a. and b., that is, if $\gamma$ exemplifies $p$,
we can use $p$ to informationally evaluate $\gamma$.

Step (b) is to be brought about by the working semanticist, unless a
computational classifier system is available. This is part of what makes () a
heuristic.444444In human vision, this is brought about by aligning a
perceptual image with stored visual models [133, in particular chapters 6 and
7]. Step (c) delivers the input for the information-evaluation conditioned
interpretation of multimodal utterances expressed in template (4) in the
introduction to section 4.

throw-cvm$(\gamma)=1$agent $=$ speakeryvirtualthemevirtual trajectory
$\mathbf{v}$ Figure 24: Exemplifying throw.

Let us look at an example of how () works and offers a toolkit for semantic
gesture studies, namely the throwing gesture from example (2) in section 2.
The relevant extract is given in (): […] throw [throwing movement] the dagger

Let Figure 24 be an illustration of the gesture involved in () – it
extemplifies throw via the lexical meaning in (4.2). The movement “looks like”
throwing (captured by the cvm), it is a pantomimic action (the speaker mimics
the agent of the simulated action), and the imagined continuation of the
stopped gestural movement evokes a virtual trajectory which triggers a virtual
theme. Both the virtual trajectory and the theme are presupposed by evaluating
the gesture as throwing. If this presupposition is not fulfilled, the gesture
cannot “mean” throwing (it could exemplify some intransitive predicate
instead, for instance, some direction instruction of a flight attendant, “Exit
on the right”). Extended exemplification – in addition to a gesture’s
preference to directly depict its affiliate –, thus, provides some
justification to interpret the gesture in () as throw. Template (4) then gives
‘If the gesture is interpreted as throw, then […] $R$(throw, throw) the
dagger’. Obviously, $R$ can be resolved to $=$, hence the multimodal utterance
means ‘[…] throw the dagger’.

There is another interpretation available, namely the VP figuring as
affiliate, that is $\lambda x.\lambda e[\text{throw-
cvm}(e)=1\wedge\text{agent}(e,x)\wedge\text{theme}(e,\text{dagger})\wedge\exists\mathbf{v}[\operatorname{place-
path}(y,\mathbf{v})]]$. In this case, the gesture extemplifies throw the
dagger, which is brought about by the straightforward additional interpretive
assumption that the virtual object $y$ is identified with the dagger. Gesture
interpretation then delivers “If the gesture is interpreted as throw the
dagger, then […] $R$(throw the dagger, throw the dagger)”, which amounts to
the same interpretation as the V-affiliated one. If also a subject were
involved, a sentence-based extemplification can be derived, too. But recall
from section 2 that the affiliate of a gesture is a lexical item in the
majority of cases. Thus, the interplay of extemplification and affiliation
offers some harmless leeway, which corresponds to the openness of gesture
interpretation.

Throwing is an action, ‘ ’ is a shape which has been drawn by manual movement.
Both, acting and drawing, correspond to two of the four modes of
representation distinguished in section 2. Bridging to gesture studies,
extemplification can be related to the modes of representation in the
following way, where $\beta$ stands for the extemplified predicate: _acting_ :
$\beta$ is usually a transitive predicate whose agent is the speaker/gesturer
(e.g., throw) or the gesturer’s hand or arm (e.g., walk), and the theme is a
virtual object _molding_ : $\beta$ is a predicate denoting a volume _drawing_
: $\beta$ is a predicate denoting a shape _representing_ : $\gamma$ is
identified with the discourse referent introduced by the affiliate (the
gesture being a representative of a discourse referent)

From the perspective of extemplification, it is not a coincidence that –
except for _representing_ , which works differently – all modes of
representation involve visuo-spatial predicates, featuring a cvm.

###  4.3 Lexicon-driven speech–gesture integration

Although the aim of this paper is a semantics of iconic gestures, this
semantics is closely tight to affiliated speech. Usually, the gesture just
extemplifies the affiliate, as seen in the previous section. But what has
extemplification to say about examples where the extemplified predicate does
not match the overtly uttered one, as in the spiral staircases example? We
argue that the same kind of implicit meanings triggered by minimized contexts
such as those in () arise if extemplified and affiliated predicate diverge. I
can’t ride my bike today. The back wheel’s tire is flat. The footage shows a
man running on stage and stabbing Adamowicz […]. The assailant paces back and
forth, arms aloft like a victorious boxer, still holding the 15cm (six-inch)
knife.454545Taken from BBC news, https://www.bbc.com/news/world-
europe-46878325, accessed 2024-01-10. Pawel Adamowicz was the mayor of Gdansk.

The tire in (a) is understood as the tire of the bike. The knife in (b) is
understood as the instrument of the stabbing event, and pacing back and forth
the stabbing action [18, bridging;]. Accordingly, the same mechanisms used to
resolve bridging anaphora apply in minimized speech–gesture affiliation pairs.
An approach particularly well-suited to iconic gesture semantics is lexicon-
driven and employs _frames_ [33, 34] to compute inferences on minimized
content.464646To extend lexical meaning to gesture integration has been
proposed early on by [119]. Corresponding lexical extensions have been
developed within dynamic semantics, namely within _Discourse Representation
Theory_ respectively _Segmented Discourse Representation Theory_ [11, 57] and
a _Type Theory with Records_ [20, 22]. Frames can be conceived as
stereotypical situation types which are connected to lexical items (“single
words”). A single word not only contributes its content, but it also _evokes_
the frame it is connected to. Frame semantics is organized in a frame-base
lexicon called FrameNet.474747For a short introduction to FrameNet see [35],
the FrameNet resource is available at https://framenet.icsi.berkeley.edu/. The
lexical entry for staircase.n, for instance, is linked to the
connecting_architecture frame. The connecting_architecture frame has a _Part_
element as its core element, which is the connecting_architecture in question,
and resolves to the staircase in our example. Additionally, there are nine
non-core elements, namely _Connected_locations_ , _Creator_ , _Descriptor_ ,
_Direction_ , _Goal_ , _Material_ , _Orientation_ , _Source_ , and _Whole_.
Via _frame evocation_ [57], the content of staircase in (a) is extended by
frame elements as in (b) (using DRT’s handy box notation; some frame elements
are omitted for reasons of space): a. $\lambda x.$ xstaircase(x) b. $\lambda
x.$ x ∣ e, y_1, y_2, y_3, y_4e : connecting_architecture
Part(e,x)
staircase(x)
Creator(e,y_1), y_1=?
Descriptor(e,y_2), y_2=?
Direction(e,y_3), y_3=?
Material(e,y_4), y_4=?

Some remarks are in order here. The first condition instantiates the
connecting_architecture frame as eventuality $e$. $e$ as well as the discourse
referents of the non-core arguments $y_{1\ldots 4}$ are merely implicit and
are separated from the regular discourse referents of the universe (vertical
bar in the top row). Implicit discourse referents were introduced by [64], and
a related distinction between discourse referents introduced by speech and
those introduced by gesture has been argued for by [79]. Frame evocation,
thus, can be seen as a computational implementation of lexical presupposition
triggering [63]. Since in this particular frame example, the part _is_ the
connecting_architecture talked about, all conditions that apply to $e$ also
apply to $x$.

If the gesture in example (4), re-given in (): Inside the hall was an imposing
staircase.

is interpreted as spiral, then information-evaluation conditioned utterance
interpretation from (4), re-given in (): _Information-evaluation conditioned
interpretation_ :

If a gesture is interpreted to mean $m$, then the utterance is interpreted as
$\alpha[R(m,\beta)]$. returns ‘$R(\text{spiral}(z),\text{staircase}(x))$’ as
the affiliation argument of $\alpha$. (Just as a reminder: $\alpha$ denotes a
sentence, which is accompanied by gesture, which extemplifies $m$ and is
affiliated to expression $\beta$ within $\alpha$.) This information adds to
the conditions of the frame-wise extended content of staircase: $\lambda x.$
x, z ∣ e, y_1, y_2, y_3, y_4e : connecting_architecture
Part(e,x)
staircase(x)
Creator(e,y_1), y_1=?
Descriptor(e,y_2), y_2=?
Direction(e,y_3), y_3=?
Material(e,y_4), y_4=?
spiral(z)
R(spiral(z),staircase(x)), R=?

The implicit discourse referents in () are supposed to be filled by content of
different kinds: $y_{1}$ is likely to be an individual, $y_{4}$ a substance,
$y_{3}$ a direction, and $y_{2}$ some property.484848Such constraints can
nicely be captured by type constraints in a type theory [21]. Since spiral has
been produced by a gestural _drawing_ mode of representation (or,
alternatively, since it is a shape predicate), the only plausible frame
element to resolve $R$ is $R=$ _Descriptor_. We arrive at the following
multimodal meaning, given that the gesture is informationally evaluated to
mean spiral and affiliated to staircase: $\lambda x.$ x, z ∣ e, y_1, y_2, y_3,
y_4e : connecting_architecture
Part(e,x)
staircase(x)
Creator(e,y_1), y_1=?
Descriptor(e,y_2), y_2= spiral(z), z=x
Direction(e,y_3), y_3=?
Material(e,y_4), y_4=?

The frame-extended predicate in () is processed as usual in further semantic
composition.494949On harmonizing DRT and Montague-style semantics, see [107,
136], although the natural framework of () is SDRT [3]. Much more could (and
should) be said of lexicon-based speech-gesture integration, but we believe
that the above-given example is intuitively clear enough.

The same mechanism derives examples discussed elsewhere, such as (), taken
from [125, p. 303 ]. John [slapping gesture] punished his son.

If we interpret the gesture as slapping and punished as the lexical affiliate,
then the multimodal information package ‘$R(\text{slapped},\text{punished})$’
is obtained. The lexical unit punish.v evokes the Rewards_and_punishment
frame. Thus, the lexical content in (a) is frame-wise extended to include the
implicit content in (b) (slightly abbreviated; the agent role maps to frame
element ‘Agent’, the patient role to Evaluee; cf. [57]): a. $\lambda y.\lambda
x.\lambda e.$ y,x,epunish(e)
agent(e,x)
patient(e,y) b. $\lambda y.\lambda x.\lambda e.$ y,x,e ∣ z_1, z_2, z_3, z_4,
z_5e : rewards_and_punishment
punish(e)
Agent(e,x)
Evaluee(e,y)
Reason(e,z_1), z_1=?
Degree(e,z_2), z_2=?
Instrument(e,z_3), z_3=?
Manner(e,z_4), z_4=?
Means(e,z_5), z_5=?

Being an action-simulating gesture, slapping instantiates the non-core Means
frame element: punish by slapping, see the resolved content in (). $\lambda
y.\lambda x.\lambda e.$ y,x,e,e’,x’,y’ ∣ z_1, z_2, z_3, z_4, z_5e :
rewards_and_punishment
punish(e)
Agent(e,x)
Evaluee(e,y)
Reason(e,z_1), z_1=?
Degree(e,z_2), z_2=?
Instrument(e,z_3), z_3=?
Manner(e,z_4), z_4=?
Means(e,z_5), z_5=slap(e’), e’=e
agent(e’,x’), x’=x
patient(e’,y’), y’=y

() reads as $x$ punished $y$ by slapping $y$. This captures Schlenker’s
intended reading for this example [125, p. 303], but is even stronger than the
local context actually induced in his approach. The local context one gets
without further stipulation is If John punished his son, then slapping would
be involved [125, p. 318]. But the slapping could be slapping someone else –
think of John punishing his son by slapping the son’s pet. In () this is
captured by a non-maximal interpretation and dispensing condition
‘$y^{\prime}=y$’. (The same kind of objection applies to Schlenker’s helping-
by-lifting example, which has an additional comitative reading.)

Furthermore, presuppositional accounts do not seem to prevent speech–gesture
mismatches. Take, for instance, (), where the spiral gesture is replaced by a
slapping one: Inside the hall was an imposing [slapping gesture] staircase.

This gesture obviously does not extemplify its affiliate staircase.
Furthermore, slapping, denoting an action, is not a good candidate to fill any
of the frame elements evoked by staircase in (). Hence, frame-based dynamic
semantics algorithms would fail to integrate speech and gesture in this case
and signal a mismatch. This does not seem to hold for other approaches such as
those resting on local contexts and assertion-dependent presuppositions, since
nothing prevents local contexts of the form “every world $w$ in which a
staircase is in the hall is one in which slapping is involved” from being
computed [125, cf.].505050Schlenker, by the way, puts a lot of weight on
computability. But we would argue that the alleged “algorithm” for generating
local contexts and iconic presuppositions is not computable at all. The main
reason is that it operates on possible worlds. Possible worlds, however, are
too much for human, cognitive processing [114], and since, according to
philosophical arguments, they lack criteria of individuation and thereby
countability [117], they presumably resist any computational access. (We owe
these arguments to [22, p. 238 sqq. ].) But this might be okay from a purely
technical point of view. The FrameNet approach at least _is_ computable (for
an approach close to the present formal model see [54]; for a recent overview
including state-of-the-art neural models see [116]).

Examples such as (2), where the verbal affiliate figuring out, is accompanied
by a balancing gesture, are covered by figure_out.v, which evokes the
Coming_to_believe frame. This frame introduces a _Means_ element hosting the
act performed by the agent, which enables them to figure something out. This
means is extemplified by a balance scale gesture.

###  4.4 Applications

In this section, we apply iconic gesture semantics to some examples in order
to illustrate how it is supposed to work and contribute to multimodal semantic
analyses. To start with, the gesture displayed in Figure 25 evokes the visual
image of a holding event. Holding is a two-place predicate with the meaning
represented in (a) and the extemplification mappings are explicated in (b).
Being a static action, no vector path is involved in this case.
$\llbracket\text{hold}\rrbracket=\lambda y.\lambda x.\lambda e[\text{hold-
cvm}(e)=1\wedge\textit{agent}(e,x)\wedge\textit{theme}(e,y)]$ Bijective iconic
mappings:
– hold-cvm($\gamma$) $\mapsto 1$
– speaker/gesturer $\mapsto\textit{agent}(e)$
– space between hands $\mapsto\textit{theme}(e)$ (i.e., the theme remains
virtual)

Every content part of hold can be mapped onto morphological or visual features
of the gesture. Hence, the gesture successfully extemplifies hold.

_Space between hands:_ virtual theme _Free ride:_ distance $d$_Visual image:_
hold-cvm$(\gamma)=1$_Agent:_ speaker

Figure 25: Visual image of holding or touching an item. A distance (red line)
comes as a _Free Ride_ as studied in diagrammatic reasoning [128, e.g.,].

Now, this type of gesture has been claimed to mean large ([125, p. 304], [32,
p. 118]). This interpretation does not seem to live up to iconic gesture
semantics, however. There is, apart from implementational details, consensus
that large is an adjective that is lexically associated with a measurement
scale and a standard of comparison, as is expressed in (a) [66, 104, see]. But
can this meaning be extemplified by the sample gesture? The gesture, in any
case, extemplifies the property of _being this large_ (d-large from (a)) by
means of the spatial distance $d$ between the hands (this from a _Free Ride_
type of inference from diagrammatic reasoning, [128]). Comparing () and the
visual image displayed in Figure 25, however, it is apparent that the gesture
does not extemplify the standard of comparison. The standard is not an
_intrinsic_ property of (virtual or real) sizing actions, hence large is not a
fully visual property (as is already indicated by a lack of a cvm).
$\llbracket\text{large}\rrbracket=\lambda
x[\text{standard}(\text{large})\leq\text{large}(x)]$,
where large is a measure function $\lambda
x.\rotatebox[origin={c}]{180.0}{$\iota$}d.[x\textit{ is }d\textit{-large}]$ of
type $\langle e,d\rangle$ Bijective iconic mappings:
– is there a large-cvm?
– (from _Free Ride_) distance $d\mapsto d$-large
– ? $\mapsto$ standard (and it is unclear what to do with the agent)

Extemplification of this large already involves a visual inference via a free
ride from holding something to sizing.515151The sizing mode of representation
has been observed by [65] in the Grappolo gesture family; it has also been
used in iconic gesture annotation [9]. Arriving at large – if possible at all
without further information in context525252The study described in [32], for
instance, introduces the verbal alternation small and large in experimental
instructions, which is exploited by participants to reason about the gesture
in terms of sizes. – needs further inferential processing.535353Note that the
verbal affiliates in such examples are concrete nouns like bottle or dog. The
lexical meaning of both expressions does not refer to tallness (or even size).
This analysis confirms the judgments reported by [56, p. 322 ], namely that a
large reading of the gesture is unavailable for such examples.

An additional trigger for pragmatic reasoning is the absence of a “lexical
affiliate” [124], which is usually associated with an iconic gesture and
guides its interpretation [48]. The imagined affiliate in case of (11) is the
word large and its absence already indicates that the example is somewhat
deviant: if large is the important property in question, then it is produced
in an information structurally distinguished way, that is, being the focus
expression. Omitting the focused predicate leads to a pragmatically
infelicitous utterance. Pragmatically infelicitous utterances in turn trigger
specific implicatures (e.g., evasion moves; [40]). Hence, such examples are
particularly ill-suited to tell apart asserted and implied contents.

Example () is taken from [126, p. 761].545454The original example has a nice
picture of the gesture labelled with unscrew-ceiling instead of a verbal
description, though. This light bulb, are you going to [speaker stretches arm
overhead and rotates hand]

Schlenker interprets the gesture as unscrewing at the ceiling. How can we
derive this interpretation? unscrew.v gives us the following lexical
information and evokes the _Closure_ frame:
$\llbracket\text{unscrew}\rrbracket=\lambda y.\lambda x.\lambda
e[\text{unscrew-cvm}(e)=1\wedge\text{agent}(e,x)\wedge\text{theme}(e,y)\wedge
e:\text{Closure}]$

The _Closure_ frame contributes a non-core _Place_ element, which the gesture
extemplifies to overhead. This is what we get from gesture semantics.
Schlenker claims further that the multimodal utterance involving the pro-
speech gesture in () triggers the presupposition that the light bulb is at the
ceiling. Since this is not part of the multimodal meaning, the location of the
lamp should not be a presupposition – speaking in terms of FrameNet: the
_Ground_ element of the _Location_of_light_ frame is not extemplified.
Accordingly, we expect to find a situation where it does not hold. And indeed,
stretching the arm over the head might also be necessary to unscrew light
bulbs from other kinds of tall lamps, as shown in Figures
26LABEL:sub@subfig:floorlamp and 26LABEL:sub@subfig:tablelamp. Hence, the
impression that the light bulb is at the ceiling cannot be a presupposition of
() – it is just an artifact of gesture interpretation (which in turn might be
influenced by a defeasible abductive inference to a common location of light
bulbs fixed overhead, though).

[regular] (a) Ceiling lamp

[regular] (b) Floor lamp

[regular] (c) Table Lamp

Figure 26: Different lamps whose lightbulbs have to be unscrewed overhead.

This shows, we believe, the importance of a semantic toolkit for interpreting
gestures in semantics, in particular, if different kinds of inferential
meanings are to be kept apart.

Above all, however, iconic gesture semantics elucidates that a “linguistic
lift” in terms of informational evaluation of a gesture is needed for the
gesture to be able to interact with linguistic meaning in the first place.
Being conditioned on a linguistic interpretation (_If the gesture is
interpreted to mean “m” …_) explains that gestures do not contribute content,
neither at-issue nor non-at-issue (cf. footnote 30); _if_ they are
informationally evaluated, however, they are just like additional words.

##  5 Discussion

###  5.1 “Energy spaces”

The iconic gesture semantics developed above is confined to representational
gestures, that is, gestures that exhibit a rather pictorial (aka iconic)
content. This is not surprising since such gestures catch the eye and have a
rather clear interpretation. However, a lot of gestures occurring in natural
language interactions have a strong impact on the beat dimension and/or are
bound up with functional information structuring interpretations (see example
2 in section 2). A mere spatial interpretation seems to miss the point of such
a gesture. Intuitively, they seem to be of a more “somatic” origin. The vector
space semantics, we argue, can be adapted to such gestures, too, following
ideas of [132]. He envisaged the use of _force vectors_ instead of spatial
place or path ones. One of the examples he discusses is the (intensional)
meaning of the verb climbing. According to the analysis of [132], the
semantics of climb is captured in terms of two forces: one pulling downwards,
one striving upwards. Mathematical vector spaces are ontologically neutral.
The same formal devices can be used to model “energy spaces” consisting of
force vectors. Following this idea, a domain extension is straightforward: in
addition to spatial vector spaces, each individual $d\in D_{e}$ is assigned
two force spaces, a pulling and an attracting one. Speakers occupy the
respective “center of gravity” – see Figures 28 and 28. Talmy’s analyses of
climbing can be made precise in terms of an energy space spanned by the
orthogonal projections of force vectors onto the downwards and upwards pulling
ones in repulsion space.

[2] []  []

Figure 27: Repulsion space Figure 28: Attractor space

Force vectors are arguably involved in verbal construction like on the one
hand … on the other hand: the two poles referred to are pulled apart by force
vectors drawing in opposing directions. Accordingly, this verbal construction
is often accompanied by two placing gestures that locate the two poles in two
different hemispheres of gesture space. Another point in this case could be
gestures of uncertainty: being unsure corresponds to a lack of direction in
energy space. Meandering through force vectors manifests itself in manual
wiggling movements. Of course, this is speculative to a great extent, but a
line of theoretical research worth to be explored in future work.

###  5.2 Non-lexicalized iconic models

Since a gesture’s meaning contribution is conditioned on its informational
evaluation, we expect no difference in at-issue contexts. The demonstrative
context like this in () is supposed to shift its referent to at-issue content
[126, p. 303]. The staircase looked like this:

While this works for speech, what is the at-issue contribution of the gesture
in ()? You have to informationally evaluate it first, as usual. That gestures
without informational evaluation fail to contribute at-issue (or non-at-issue,
for that matter) content can be seen by gestural movements that resist
perceptual classification because the trajectory has no lexicalized label. An
example is given in ().555555At least the authors do not know how to call this
shape; it might be possible that some readers think that it resembles
something they know and can name. We rely on these readers to appreciate the
point of the example nonetheless. The inscription looked like this:

We do think that the at-issue test still does work, and that it works for
gestures. A gesture that is such that it resists perceptual classification in
terms of single words just contributes its iconic model.565656This is not
puzzling from a dual coding [112] or multimodal propositions [84] point of
view. It is also part-and-parcel of grounding non-verbal behaviour such as
motor actions [55].

Other gesture uses that can be subsumed under non-lexical ones are metaphoric
ex(t)emplifications (such as the gesture from example 2 in section 2). We
ignored them so far, but just want to mention that a frame-based approach
provides a modelling clue in terms of the blending of frames [115, MetaNet, ].

###  5.3 Repercussions for semantic theories

Speech–gesture integration often leads to a reduplication in the sense that an
iconic gesture extemplifies its linguistic affiliate. In other cases (i.e., if
$R\neq=$), standard dynamic semantic resolutions apply. In this sense are
iconic gestures not very exciting content-wise. But they are intriguing
theory-wise: iconic gestures forces us to tell apart (perceptual) meaning and
truth conditions. The notion of meaning needed is such that, when applied to
an object, it returns a linguistic label (cf. section 1). We spell this out in
terms of extemplification and perceptual classification. The resulting outline
of an iconic gesture semantics according to Figure 22 involves both a standard
denotational frame of reference and a classification/extemplification
component. Arguably, these components cannot be reconciled with a textbook
possible worlds semantics. If this is right,575757And there is ample
supporting evidence. To name a few: the representability problem of possible
worlds clashes with cognitive tractability [76], which is a characterizing
feature of extemplification/classification (see also footnote 50); the
extensions of classifier-based semantics are indeterminate, which is not
compatible with fixed universes $D_{e}$ of quantification of traditional
models [78]; the learnability problem in Montagovian models [139] is at odds
with classifier learning. then formal semantic thinking about iconic meaning
ultimately requires looking for a different semantic theory. A suitable
candidate, to our minds, is a _Type Theory with Records_ [22, TTR,]. TTR
incorporates words-as-classifiers [77], and can be given a probabilistic
interpretation [23], which is needed for learning and graded judgments (cf.
above from Figure 16). TTR’s modal theory does not assume possible worlds and
hence fares better in terms of computability and cognitive
interpretability.585858This extends into the domain of plurality and
quantification, as _Referential Transparency Theory_ [89, RTT,], which is
formulated within TTR, avoids well-known processing obstacles of generalized
quantifier theory. TTR includes frames as both, situations and situation types
[20, 22]. Additional means of rhetorical coherence are expressed in terms of
enthymemes [12]. TTR has been applied to iconic gesture [85]. It also provides
the ontology for dialogue semantics [36] (recall the importance of, e.g.,
clarification interaction discussed in section 4). After all, as is widely
agreed, it is dialogue that is the “ecological niche” of multimodal
interaction [53, 90]. Thus, the contour not only of a compositional but also
of a _computational_ theory of iconic gesture semantics is emerging and will
be elaborated in future work.

##  6 Conclusions

The theory-strategic aim of this paper was to reconcile spatial and labelling
theories. This reconciliation amounts to a theory of iconic gesture semantics
and has been achieved by spelling out a spatial, truth-functional account of
iconic gestures, and by drawing on extended exemplification
(extemplification), respectively perceptual classification. Key to this
reconciliation was that gesture meaning depends on the informational
evaluation of the gesture; otherwise, the gesture remains semantically inert.
Iconic gesture semantics takes it seriously that gestures are not like words,
and pays due attention to the characteristics of visual communication.

All the desiderata D1, D2, D3 identified in section 2 are addressed:

D1

Gesture interpretation is strongly dependent on the accompanying speech
(affiliate dependence).

D2

Speech–gesture affiliation is regimented by (in-)congruency.

D3

Gesture interpretation is an instance of visual communication, which rests on
a perceptual interpretation of gestural forms.

D1 is captured by the affiliate being either the extemplified predicate, or by
the affiliate being the frame-evoking expression which introduces an implicit
discourse referent figuring as the extemplified meaning component.

D2 is captured within the spatial model in terms of an empty intersection of
the gesture’s iconic model and the spatial projection of its affiliate.
Incongruency between speech and gesture is signalled if the gesture does
neither extemplify its affiliate nor one of the frame elements evoked by the
affiliate.

D3 has been the biggest obstacle to formal semantic modelling; in fact, it
forces us to leave the well-trodden paths of Frege/Montague-style possible
worlds semantics and to adopt a procedural, classifier-based notion of
meaning. It is this repercussion of iconic gestures, we have argued, that
makes visual communication a theoretically interesting object of research in
formal semantics and the philosophy of language.

As a workaround for interpreting gestures in semantic analyses, an
extemplification heuristic has been developed. This heuristic paves the way
for systematic, empirically informed gesture studies within formal semantics
and related fields. It is computationally implementable, thus enabling cross-
talk between iconic gesture semantics and cognitive science.

## Acknowledgments

Support by the _Deutsche Forschungsgemeinschaft_ (DFG), grant number
502018965, is gratefully acknowledged.

## References

  * [1] Katya Alahverdzhieva, Alex Lascarides and Dan Flickinger  “Aligning speech and co-speech gesture in a constraint-based grammar”  In _Journal of Language Modelling_ 5.3, 2017, pp. 421–464
  * [2] Martha W. Alibali  “Gesture in Spatial Cognition: Expressing, Communicating, and Thinking About Spatial Information”  In _Spatial Cognition and Computation_ 5, 2005, pp. 307–331
  * [3] Nicholas Asher and Alex Lascarides  “Logics of Conversation”  Cambridge: Cambridge University Press, 2003
  * [4] Adrian Bangerter and Daniel M. Oppenheimer  “Accuracy in Detecting Referents of Pointing Gestures Unaccompanied by Language”  In _Gesture_ 6.1, 2006, pp. 85–102
  * [5] Kathryn Barnes and Cornelia Ebert  “The information status of iconic enrichments: modelling gradient at-issueness”  In _Theoretical Linguistics_ 49.3-4, 2023, pp. 167–223  DOI: 10.1515/tl-2023-2009
  * [6] Renate Bartsch  “Dynamic Conceptual Semantics. A Logico-Philosophical Investigation Into Concept Formation and Understanding”  Stanford, CA: CSLI Publications, 1998
  * [7] Janet B. Bavelas and Nicole Chovil  “Visible Acts of Meaning: An Integrated Message Model of Language in Face-to-Face Dialogue”  In _Journal of Language and Social Psychology_ 19.2, 2000, pp. 163–194  DOI: 10.1177/0261927X00019002001
  * [8] Janet B. Bavelas, Nicole Chovil, Douglas A. Lawrie and Allan Wade  “Interactive Gestures”  In _Discourse Processes_ 15.4, 1992, pp. 469–489  DOI: 10.1080/01638539209544823
  * [9] Kirsten Bergmann et al.  “Documentation – Sagaland” https://www.phonetik.uni-muenchen.de/Bas/BasSaGAeng.html, 2014  SFB 673, Project B1
  * [10] Ulrike Bohle  “Approaching notation, coding, and analysis from a conversational analysis point of view”  In _Body – Language – Communication. An International Handbook on Multimodality in Human Interaction_ 1.38, Handbooks of Linguistics and Communication Science  BerlinBoston: De Gruyter Mouton, 2013, pp. 992–1007  DOI: 10.1515/9783110261318.992
  * [11] Johan Bos and Malvina Nissim  “Combining Discourse Representation Theory with FrameNet”  In _Frames, Corpora, and Knowledge Representation_ Bologna: Bononia University Press, 2008, pp. 169–183
  * [12] Ellen Breitholtz  “Enthymemes and Topoi in Dialogue”, Current Research in the Semantics/Pragmatics Interface 41  LeidenBoston: Brill, 2020
  * [13] Jana Bressem  “A linguistic perspective on the notation of form features in gestures”  In _Body – Language – Communication. An International Handbook on Multimodality in Human Interaction_ 1.38, Handbooks of Linguistics and Communication Science  BerlinBoston: De Gruyter Mouton, 2013, pp. 1079–1098  DOI: 10.1515/9783110261318.1079
  * [14] Justine Cassell, David McNeill and Karl-Erik McCullough  “Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information”  In _Pragmatics & Cognition_ 7.1, 1999, pp. 1–34  DOI: 10.1075/pc.7.1.03cas
  * [15] David J. Chalmers  “On Sense and Intension”  In _Philosophical Perspectives_ 16, 2002, pp. 135–182
  * [16] Lucas Champollion  “The interaction of compositional semantics and event semantics”  In _Linguistics and Philosophy_ 38, 2015, pp. 31–66  DOI: 10.1007/s10988-014-9162-8
  * [17] Gennaro Chierchia and Sally McConnell-Ginet  “Meaning and Grammar – An Introduction to Semantics”  Cambridge: MIT Press, 2000
  * [18] Herbert H. Clark  “Bridging”  In _Proceedings of the 1975 Workshop on Theoretical Issues in Natural Language Processing_ , TINLAP ’75  Cambridge, Massachusetts: Association for Computational Linguistics, 1975, pp. 169–174  DOI: 10.3115/980190.980237
  * [19] Richard P. Cooper  “Persistence and Structural Determination”  In _Situation Theory and its Applications_ 2.26, CSLI Lecture Notes  Stanford, CA: CSLI, 1991, pp. 295–309
  * [20] Robin Cooper  “Frames in Formal Semantics”  In _Advances in Natural Language Processing_ , Lecture Notes in Computer Science 6233  BerlinHeidelberg: Springer, 2010, pp. 103–114  DOI: 10.1007/978-3-642-14770-8\\_13