Also at<EMAIL_ADDRESS>
# A String-Graph Approach to Molecular Geometry

Sebastián Alí Sacasa Céspedes1,2 [<EMAIL_ADDRESS>1Universidad de
Costa Rica (UCR), San Pedro de Montes de Oca, San José, 11501-2060, Costa Rica
2Plasma Laboratory for Fusion Energy and Applications, Instituto Tecnológico
de Costa Rica (ITCR), Cartago, 30109, Costa Rica

(July, 9, 2024)

###### Abstract

Introduction: molecular geometry, the three-dimensional arrangement of atoms
within a molecule, is fundamental to understanding chemical reactivity,
physical properties, and biological activity. The prevailing models used to
describe molecular geometry include the Valence Shell Electron Pair Repulsion
(VSEPR) theory, hybridization theory, and molecular orbital theory. While
these models provide significant insights, they also have inherent
limitations. Applying string theory and graph theory with topological and
macrotensorial methods could improve the understanding of molecular behavior.
Objective: explore the potential applications of string and graph theory to
material science, focusing on molecular geometry, electron domains, and phase
changes via symmetries. Molecular geometry: each molecule is associated with a
simple graph with an orthonormal representation inducing metrics via the usage
of macrotensor operators, allowing the calculation of angles between molecules
and following the equations of motion. Phase changes: a series of inequalities
are proposed depending on the energy-momentum densities of bonds and the edges
of the associated graph where electrons or atoms are located, its topology,
and isometries, exploring possible new states of matter. Conclusions:
application of macrotensors, graphs, string theory, partitions and
correlations functions of dimensions to material science, specifically to
molecular geometry and phase changes allows for a more dynamic and flexible
description of natural phenomena involving matter and the prediction of
possible new states of matter. This presents a different perspective, opening
possibilities for experimental confirmation and applications of the approach
presented here.

Keywords

macrotensor, partition function, string theory, graph theory, molecular
geometry, states of matter.

††preprint: APS/123-QED

##  I  Introduction

Molecular geometry, the three-dimensional arrangement of atoms within a
molecule, is fundamental to understanding chemical reactivity, physical
properties, and biological activity. The prevailing models used to describe
molecular geometry include the Valence Shell Electron Pair Repulsion (VSEPR)
theory, hybridization theory, and molecular orbital theory. While these models
provide significant insights, they also have inherent limitations. VSEPR, for
instance, assumes idealized shapes and does not fully account for the
complexities of electron-electron interactions [8]. Hybridization theory
simplifies orbitals into sp, sp², and sp³ types, often overlooking the nuances
of molecular environments. Molecular orbital theory, although powerful, can
become computationally intensive and less intuitive for large systems [19].

When string theory is applied to molecular systems, it provides a novel
framework for describing the modes of vibrations and interactions within
molecules, potentially leading to more accurate predictions of molecular
geometry and behavior [4]. Using the mathematical and physical tools developed
for macrotensors, partition functions and correlation functions. [1,3]

Topological methods can capture the intrinsic properties of molecular
structures that remain invariant under continuous deformations. This
perspective is particularly valuable in the study of large, complex molecules
and materials where traditional geometrical methods fall short [13]. Relating
these invariant structures to conserves quantities via Noether’s Theorem.

Graph theoretical approaches can model the connectivity of atoms within a
molecule, providing insights into the molecule’s stability, reactivity, and
electronic properties [6]. By extending graph theory with concepts from string
theory and topology, it is possible to develop more comprehensive models that
incorporate the dynamic and flexible nature of molecular structures.

This article explores the potential applications of string and graph theory to
material science, focusing on molecular geometry and electron domains. It will
develop how string theory and graph theory can offer more robust and
predictive frameworks. By bridging these mathematical concepts with practical
applications in material science, it aim to open new avenues for research and
development in the design and understanding of complex molecular systems and
their symmetries.

##  II Molecular Geometry

Let $(M,g[^{|\mu|}_{|\alpha|}])$ be an oriented, smooth, complete manifold
without boundaries, with variable dimension $D(d_{1},d_{2})=d_{1}\otimes
d_{2}$ with relative maximum number of dimensions $(n_{1},n_{2})$ via a
partition function

$Z(|d_{1}|,|d_{2}|)=\sum_{i=0}^{i=n_{1}}\sum_{j=0}^{j=n_{2}}\lambda\sqrt{\frac{\epsilon_{1}}{\epsilon_{2}\pi}}\exp[-\frac{\epsilon_{1}d_{i}}{\epsilon_{2}d_{j}}]$
(1)

with expectation value of $s=1,2$ with ${\mathbf{h}}$ auxiliary parameters
associated to dimensions.

$E[d_{s}]=\frac{\partial}{\partial
h_{s}}\ln[Z(|d_{1}|,|d_{2}|,{\mathbf{h})}]|_{\mathbf{h}=0}\\\
=\sum_{i=0}^{i=n_{1}}\sum_{j=0}^{j=n_{2}}\frac{d_{s}}{\lambda\sqrt{\frac{\epsilon_{1}}{\epsilon_{2}\pi}}\exp[-\frac{\epsilon_{1}d_{i}}{\epsilon_{2}d_{j}}]}$
(2)

Which has an Ursell Function according to Camia, Jiang and Newman show it in
[4] applicated to this case as

$u_{2}(d_{1},d_{2}):=\frac{\partial^{2}}{\partial h_{1}\ \partial
h_{2}}\ln[Z(|h_{1}|,|h_{2}|,{\mathbf{h})}]|_{\mathbf{h}=0}\\\
=\frac{1}{Z(|d_{1}|,|d_{2}|,{\mathbf{h})}}\\\ \frac{\partial^{2}}{\partial
h_{1}\ \partial
h_{2}}\ln\left\langle\lambda\sqrt{\frac{\epsilon_{1}}{\epsilon_{2}\pi}}\exp\left(-\frac{\epsilon_{1}d_{1}}{\epsilon_{2}d_{2}}+h_{1}d_{1}+h_{2}d_{2}\right)\right\rangle]|_{\mathbf{h}=0}\\\
=\sum_{\mathcal{Z}}(-1)^{|\mathcal{Z}|-1}(|\mathcal{Z}|-1)!\prod_{Z\in\mathcal{Z}}\langle
d_{Z}\rangle$ (3)

Where the sum is over all partitions $\mathcal{Z}$ in $\\{1,\ldots,n_{1}\\}$
and $\\{1,\ldots,n_{2}\\}$ referring to the probabilistic and dynamical
dimensions $d_{1},d_{2}$ and $|Z|$ is the number of blocks in $\mathcal{Z}$
(so $\mathcal{Z}=\\{Z_{1},\ldots,Z_{|\mathcal{Z}|}\\}$), and

$\langle d_{Z}\rangle:=\left\langle\prod_{f\in Z}d_{f}\right\rangle$ (4)

It has a probability measure

$P_{s}=\int_{-1}^{1}\frac{\exp[-\frac{\epsilon_{1}d_{1}}{\epsilon_{2}d_{2}}]}{Z(|d_{1}|,|d_{2}|)}d(d_{s})$
(5)

as described in [1] being $\lambda=\pm 1$ for the loss or the gain of
information and $\epsilon_{1},\epsilon_{2}$ parameters associated to
dimensions $d_{1},d_{2}$, respectively. Here, $|\mu|$ and $|\alpha|$ denote
the cardinalities of the indices $\mu$ and $\alpha$ associated to every output
vector space $V_{0}\otimes V_{1}\otimes\dots\otimes V_{d_{1}}$ from $0$ to
$d_{1}$ and for every input vector space $W_{0}\otimes
W_{1}\otimes\dots\otimes W_{d_{2}}$ from $0$ to $d_{2}$, respectively. The
metric macrotensor $g[^{|\mu|}_{|\alpha|}]$ acts as a multilinear application
between these collections of ordered vector spaces, containing each one an
ordered subcollection of vector subspaces, transforming each index by a tensor
in tensor multiplication and each tensor by matrix multiplication. The
macrotensor $g[^{|\mu|}_{|\alpha|}]$ is isomorphic to a ring
$(K^{D},\otimes,\oplus)$. The P-branes operators $\psi[^{|\mu|}_{|\alpha|}]$
will be considered as a macrotensor contained in $(M,g[^{|\mu|}_{|\alpha|}])$.

Each molecule exhibits a specific geometry influenced by its electronic links,
which can be associated with a macrotensorial P-brane operator. As open
Dirichlet1-branes are special cases of P-branes called strings, they must have
end fixed points attached to other Dirichlet1+z-branes [5]. This is useful for
modeling chemical bonds between molecules. Additionally, the end fixed points
are not strictly necessary, as it is possible to form nodes between strings.

The electron domain geometry and molecular geometry can be represented as
simple graphs without loops or multiple edges, not necessarily connected,
denoted as $G(V,E)$. Here, $V(G)=v$ and $E(G)=e=v-1$. Thus,
$\psi[^{|\mu|}_{|\alpha|}]$ is formed by a combination of $e=v-1$ external
vertices representing the external atoms, 1 vertex identifying the central
atom, and the corresponding edges that are the bonds between them.

As a graph holds for the Delta Conjecture:

$msr(G)\leqslant|G|-\delta(G)$ (6)

Where $\delta(G)$ is the minimum degree of the graph representing the
molecule, $|G|$ is the order of the graph, and $msr(G)$ is the minimum
semidefinite rank of the graph, as developed by Díaz-Navarro [2]. There exist
different representations of a graph via isomorphisms of two graphs $G(V,E)$
and $H(V^{\prime},E^{\prime})$ if there exist bijections $\theta:V\rightarrow
V^{\prime}$ and $\phi:E\rightarrow E^{\prime}$ such that
$\psi_{G}(e)=\\{v^{\prime},v\\}$ if and only if
$\psi_{H}(\phi(e))=\\{\theta(v^{\prime}),\theta(v)\\}$ as denoted in [2]. This
induces symmetries in the system, and for every symmetry, there exists a
conserved quantity between the initial and final points of motion as per
Noether’s theorem [16].

$\psi[^{|\mu|}_{|\alpha|}]=G(V,E)$ has an orthonormal representation such that
at the nodes of P-branes or vertices, if they are path-connected or not by the
bonds, the following condition holds:

$\langle\psi_{i}[^{|\mu|}_{|\alpha|}]|\psi_{j}[^{|\mu|}_{|\alpha|}]\rangle_{p(D)}=\begin{cases}0&\text{if
they are path-connected}\\\ \varsigma_{ij}\neq 0&\text{otherwise}\end{cases}$
(7)

The inner product for the continuous case is defined through the element of
manifold in [1] as

$s[^{|\mu|}_{|\alpha|}]=[\langle
x[^{|\mu|}]|x[^{|\alpha|}]\rangle]^{1/p(D)}=\int_{\tau^{|\mu|}}^{\tau^{\prime|\mu|}}\prod_{p(D)>1}^{p(D)=D}\
[g[^{|\mu|}_{|\alpha|}]_{[p(D)-1]}\
\frac{dx[^{|\mu|}]_{[p(D)-1]}}{d\tau^{|\mu|}}\
\frac{dx[^{|\alpha|}]_{[p(D)-1]}}{d\tau^{|\mu|}}]^{1/p(D)}d\tau^{|\mu|}$ (8)

For $p(D)=2$, this leads to:

$\cos(\theta)=\frac{\langle\psi_{i}[^{|\mu|}_{|\alpha|}]|\psi_{j}[^{|\mu|}_{|\alpha|}]\rangle_{2}}{\sqrt{\langle\psi_{i}[^{|\mu|}_{|\alpha|}]|\psi_{i}[^{|\mu|}_{|\alpha|}]\rangle_{2}}\sqrt{\langle\psi_{j}[^{|\mu|}_{|\alpha|}]|\psi_{j}[^{|\mu|}_{|\alpha|}]\rangle_{2}}}$
(9)

This allows for the calculation of the angle between atoms and their bonds.

The combination of P-branes operators leads to the formation of bonds and
molecules, sometimes resulting in crystal structures through superpositions,
unions, and joints of the graph, forming more complex molecules. To make graph
complements is useful for representing chains of biomolecules. These
structures are arrangements of P-branes associated with macrotensors.

The passage of electrons through a material can be conceptualized as
vibrations or derivatives of $G(V,E)=\psi[^{|\mu|}_{|\alpha|}]$, as solutions
to the motion equations $\delta S=0$ (proven in [5] and [9] for some action)
can be expanded via Fourier series and transformations using analytical
methods [21] solving the equations given by

$\frac{\partial L}{\partial\psi}-\frac{\partial}{\partial
x^{\mu}}\left[\frac{\partial L}{\partial\left(\frac{\partial\psi}{\partial
x^{\mu}}\right)}\right]-\left[\frac{\partial x^{\mu}}{\partial
x^{\prime\mu}}\right]^{-1}\frac{\partial}{\partial
x^{\mu}}\left[\frac{\partial L}{\partial\left(\frac{\partial x^{\mu}}{\partial
x^{\prime\mu}}\right)}\right]\\\
=\Gamma\left[\lambda_{\psi}+[\lambda_{\psi},\lambda_{x^{\mu}}]_{+}\delta
x^{\mu}\right]$ (10)

$\frac{\partial L}{\partial x^{\mu}}-\frac{\partial}{\partial
x^{\mu}}\left[\frac{\partial L}{\partial\left(\frac{\partial x^{\mu}}{\partial
x^{\prime\mu}}\right)}\right]=-\Gamma\left[m^{\mu}-\lambda_{x^{\mu}}\delta
x^{\mu}\right](\delta x^{\mu})^{-1}$ (11)

As shown in [1] for some Lagrangian density. $\Gamma$ is a Lagrange
multiplier, and $m^{\mu}$ can either be zero or lie between the initial and
final points of the motion:

$m^{\mu}=\begin{cases}0\\\ \in[x^{\mu},x^{\prime\mu}]\end{cases}$ (12)

$\lambda_{x^{\mu}}=\frac{\partial\Lambda}{\partial x^{\mu}}$ (13)

$\lambda_{\psi}=\frac{\partial\Lambda}{\partial\psi}$ (14)

All under the following condition:

$\frac{\partial\Lambda}{\partial\psi}\delta\psi+\frac{\partial\Lambda}{\partial
x^{\mu}}\delta
x^{\mu}+\left[\frac{\partial\Lambda}{\partial\psi},\frac{\partial\Lambda}{\partial
x^{\mu}}\right]_{+}\delta\psi\ \delta x^{\mu}=m^{\mu}$ (15)

Thus, the main equation describing the molecules is:

$\psi[^{|\mu|\ |\nu|}_{|\alpha|\
|\beta|}]=\psi_{i}[^{|\mu|}_{|\alpha|}]+\psi_{j}[^{|\nu|}_{|\beta|}]+[\psi_{i}[^{|\mu|}_{|\alpha|}],\psi_{j}[^{|\nu|}_{|\beta|}]]_{+}\\\
[\psi_{j}[^{|\nu|}_{|\beta|}],\psi_{i}[^{|\mu|}_{|\alpha|}]]_{+}+[s_{ik}[^{|\mu|}_{|\alpha|}],s_{jt}[^{|\nu|}_{|\beta|}]]_{+}[s_{jt}[^{|\nu|}_{|\beta|}],s_{ik}[^{|\mu|}_{|\alpha|}]]_{+}\\\
\exp\\{[F_{ik}[^{|\mu|}_{|\alpha|}],F_{jt}[^{|\nu|}_{|\beta|}]]_{+}[F_{jt}[^{|\nu|}_{|\beta|}],F_{ik}[^{|\mu|}_{|\alpha|}]]_{+}\\}$
(16)

Here, the Einstein Summation Convention is used and :the indexes go for

$i,j=0,...,v$ (17)

$\mu,\alpha=0,...,D$ (18)

$k,t=0,...,\infty$ (19)

The coefficients for Fourier expansions,
$[s_{ik}[^{|\mu|}_{|\alpha|}],s_{jt}[^{|\nu|}_{|\beta|}]]_{+}=s_{ik}[^{|\mu|}_{|\alpha|}]s_{jt}[^{|\nu|}_{|\beta|}]+s_{jt}[^{|\nu|}_{|\beta|}]s_{ik}[^{|\mu|}_{|\alpha|}]$,
do not commute. Similarly, the operators
$[F_{ik}[^{|\mu|}_{|\alpha|}],F_{jt}[^{|\nu|}_{|\beta|}]]_{+}$, which depend
on coordinates and frequencies and involve factors of $i$, also do not
commute. This non-commutativity is crucial as it allows branes to oscillate in
infinitely many ways across coordinates in D-dimensions, each associated with
vertices of graph $v$, depending on the choice of initial and final points and
their variations in motion. These oscillations can explain why crystal
structures and molecules emit energy without necessarily emitting radiation.
Equation (16) models the superpositions, unions, and joints of molecules,
analogous to a graph requiring at least two bodies for interactions.

Each mode of oscillation can be viewed as a linear combination of Quantum
Macrotensor Bits States, which are linearly independent:

$\psi[^{|\mu|}_{|\alpha|}]]\ket{\phi}=K[^{|\mu|}_{|\alpha|}]]\ket{\phi}=K[^{|\mu|}_{|\alpha|}]]\ket{S}\\\
=\sum_{i=0}^{i=v}K_{i}[{{}^{|\mu|}_{|\alpha|}]}\ket{s_{0}s_{1}\cdots
s_{v}}_{q}$ (20)

The low index $q$ represents the number of permutations of the elements
$\\{s_{0},s_{1},\cdots,s_{v}\\}$, making $\ket{S}$ the state formed by the set
of all possible permutations of its elements. $\ket{S}$ has $q=v!=\Gamma(v+1)$
permutations of zeros and ones, achieved by diagonalizing the macrotensors and
then expressing them as sets of zeros and ones in some basis. $\Gamma(v+1)$ is
the gamma function:

$\Gamma(v+1)=\int_{0}^{\infty}x^{v}e^{-x}dx$ (21)

$K[^{|\mu|}_{|\alpha|}]]$ is the macrotensor containing a collection of
tensors satisfying the following condition:

$\|K[^{|\mu|}_{|\alpha|}]\|_{2}^{2}=\lambda$ (22)

$\lambda$ is $\pm 1$ according to [1]. Geometrically, it spans a
$B_{v}(\lambda^{1/2})$, meaning a probabilistic superposition space of states
that is a ball of $dim[B_{v}(\lambda^{1/2})]=v$ with induced Euclidean
geometry that has by dimensions the number of electrons that conform the
molecule. This space is topologically invariant under spherical symmetries, so
the molecules remain invariant under global rotations for some fixed angle
$\theta\in[0,2\pi n];\ n\in\mathbf{Z}$. Conserving their angular momentum by
Noether’s Theorem being invariant under transformations
$\psi^{\prime}=e^{iJ^{\mu\alpha}_{(\theta)}}\psi$. The term
$J^{\mu\alpha}_{(\theta)}$ is a tensor that generates rotations in some
direction. It is also possible to define a product $\psi=\lambda\otimes\phi$
which leaves $\psi$ orthogonal to $\phi$ and invariant.

##  III Phase Changes

Phase changes in matter occur through the breaking of molecular bonds by
applying changes in temperature, which correspond to changes in the average
energy of molecules and on the rest of their themodynamic quantities. For a
molecule represented by graph $G(V,E)$ the following inequalities are
proposed:

$\|T[^{|\mu|\ |\nu|}_{|\alpha|\
|\beta|}](\psi_{i}[^{|\mu|}_{|\alpha|}],\psi_{j}[^{|\nu|}_{|\beta|}])\|_{2}+\|T[^{|\nu|\
|\mu|}_{|\beta|\
|\alpha|}](\psi_{j}[^{|\nu|}_{|\beta|}],\psi_{i}[^{|\mu|}_{|\alpha|}])\|_{2}\\\
\geqslant\|T[^{|\mu|}_{|\alpha|}](\psi_{i}[^{|\mu|}_{|\alpha|}])\|_{2}+\|T[^{|\nu|}_{|\beta|}](\psi_{j}[^{|\nu|}_{|\beta|}])\|_{2}$
(23)

$\\\ |T[^{|\mu|\ |\nu|}_{|\alpha|\
|\beta|}](\psi_{i}[^{|\mu|}_{|\alpha|}],\psi_{j}[^{|\nu|}_{|\beta|}])\|_{2}+\|T[^{|\nu|\
|\mu|}_{|\beta|\
|\alpha|}](\psi_{j}[^{|\nu|}_{|\beta|}],\psi_{i}[^{|\mu|}_{|\alpha|}])\|_{2}\\\
\leqslant\|T[^{|\mu|}_{|\alpha|}](\psi_{i}[^{|\mu|}_{|\alpha|}])\|_{2}+\|T[^{|\nu|}_{|\beta|}](\psi_{j}[^{|\nu|}_{|\beta|}])\|_{2}$
(24)

The first inequality holds for molecules where the energy-momentum macrotensor
evaluated for the molecular bonds is greater than the energy-momentum of each
vertex where atoms or electrons are located, thus conserving the molecular
structure for elements of manifold $p(D)=2$. Conversely, the second inequality
holds for molecules where the energy-momentum of each vertex where atoms or
electrons are located is greater than the energy-momentum of the bonds,
leading to bond breaking and deformation of the molecule, thus changing its
geometry. This leads to defining the following inequalities for states of
matter:

$(\|(T(\psi_{i},\psi_{j})+T(\psi_{j},\psi_{i})\|_{2})_{\text{Condensates}}\\\
<(\|T(\psi_{i},\psi_{j}))+T(\psi_{j},\psi_{i})\|_{2})_{\text{Solids}}\\\
<(\|T(\psi_{i},\psi_{j})+T(\psi_{j},\psi_{i})\|_{2})_{\text{Liquids}}\\\
<(\|T(\psi_{i},\psi_{j})+T(\psi_{j},\psi_{i})\|_{2})_{\text{Plasma}}\\\
<(\|T(\psi_{i},\psi_{j})+T(\psi_{j},\psi_{i})\|_{2})_{\text{Gases}}$ (25)

The indexes are omitted for simplicity. According to the triangle inequality
in Euclidean geometry, $p(D)=2$ as shown by Horn and Johnson [10]. The
inequalities can be computed as
$\|T(\psi_{i},\psi_{j})+T(\psi_{j},\psi_{i})\|_{2}\leqslant\|T(\psi_{i},\psi_{j})\|_{2}+\|T(\psi_{j},\psi_{i})\|_{2}$.

These inequalities ensure the known properties of matter and open
possibilities for new predictions. For $p(D)\neq 2$, the properties of matter
will be conserved if there exists an isometry connecting two parts
$(M_{1},g_{1}[^{|\mu|}_{|\alpha|}])$ and $(M_{2},g_{2}[^{|\mu|}_{|\alpha|}])$
of the manifold
$(M,g[^{|\mu|}_{|\alpha|}])=(M_{1},g_{1}[^{|\mu|}_{|\alpha|}])\cup(M_{2},g_{2}[^{|\mu^{\prime}|}_{|\alpha^{\prime}|}])$
such that
$\Omega:(M_{1},g_{1}[^{|\mu|}_{|\alpha|}])\rightarrow(M_{2},g_{2}[^{|\mu^{\prime}|}_{|\alpha^{\prime}|}])$
fulfilling
$s[^{|\mu|}_{|\alpha|}]=s[^{\Omega[|\mu^{\prime}|]}_{\Omega[|\alpha^{\prime}|]}]$.

If no isometry exists, there will be no orthonormal representation, causing
the locations of P-branes and their interactions to change, thus altering the
molecular structure and consequently its geometry beyond just rotations,
translations, and reflections. If the change in dimensions is positive from a
certain point making $\frac{\partial D}{\partial\gamma}>0$,
$\gamma=d_{1},d_{2}$, distances will increase, potentially inducing a phase
change as molecules and their bonds vanish in the dimensions $d_{1}$, $d_{2}$
and in the product of these two where emerges $D$ or in the three dimenions
such that $(\frac{\partial D}{\partial d_{1}},\frac{\partial D}{\partial
d_{2}})>(0,0)$. Conversely, if the change in dimensions is negative
$\frac{\partial D}{\partial\gamma}<0$, distances will decrease, causing atoms
and particles to come closer together and condense in some of the dimensions
or in all sets of dimensions. However, due to the Pauli Exclusion Principle,
they cannot occupy the same state. If the states are mixed across dimensions
and in superposition, this may induce a new state of matter by their geometry
up to a certain point, condensing the particles, making a geometrical
condensate. As denoted in [1], the absolute minimum that $D(d_{1},d_{2})$ can
have is $D_{0}=(3,3)$ being the minimum number of observed dimensions,
confirming

$\frac{\partial^{2}D}{\partial d_{1}^{2}}>0$ (26)

$\frac{\partial^{2}D}{\partial d_{1}^{2}}\frac{\partial^{2}D}{\partial
d_{2}^{2}}-\left[\frac{\partial}{\partial d_{1}}\frac{\partial D}{\partial
d_{2}}\right]^{2}>0$ (27)

When thermodynamic quantities are taken into account, the properties of matter
will change accordingly. Additionally, isomorphisms in the graph
representation may exist, conserving certain quantities and symmetries,
regardless of whether isometries are preserved. In case there are no
symmetries, the entire molecule will undergo changes.

##  IV Conclusion

This paper presents an application of macrotensors, graphs, string theory,
partitions and correlations functions of dimensions to material science,
specifically to molecular geometry and phase changes. This approach offers a
more dynamic and flexible description of natural phenomena involving matter.
It provides a new perspective on molecular bonds and their representation as
simple graphs with orthonormal representation in a space induced with
different geometries and dynamical dimensions. This framework facilitates the
study of molecular interactions under various circumstances and the
exploration of a possible new state of matter.

Experimental confirmation, applications, and further development of the
presented approach are anticipated.

###### Acknowledgements.

The author wishes to acknowledge the patience, support, and encouragement from
his family and friends, including Lic. Gisela Patricia Céspedes Barrantes,
Lic. Marco Vinicio Sacasa Soto, Kiara Grisel Sacasa Céspedes, Óscar R. Jiménez
Fernández, Lilliam Barrantes Rodriguez, Yalile Soto Alvarado, M.Sc. Irwin J.
Céspedes Barrantes and many others. Special thanks go to PhD Pedro Díaz
Navarro for developing a significant portion of the mathematics of graph
theory used in this work. The author also expresses deep gratitude to Octavio
Gerardo Rojas Quesada for his unwavering belief and support; PhD V. Iván
Vargas Blanco for his kindness, conversations and access to the Plasma
Laboratory for Fusion Energy and Applications; Julián Sánchez Castro, Kendy,
José María and Physicist Ricardo Solano Piedra for their kindness; PhD
Alejandro Jenkins Villalobos for his openness in discussions; and PhD Heidy
Gutierrez Gutiérrez Garro for her valuable notes in mathematics and physics.
The author is profoundly grateful to all those who contributed in various
ways, making this work possible.

## References

  * [1] S. A. Sacasa-Céspedes. A Thermodynamical Approach to Fundamental Interactions. To be published.
  * [2] P. A. Díaz-Navarro. On the Delta Conjecture and the Graph Complement Conjecture for Minimum Semidefinite Rank of a Graph. Doctoral dissertation, Central Michigan University, 2014.
  * [3] A. T. Balaban. Applications of Graph Theory in Chemistry. Journal of Chemical Information and Computer Sciences, 25(3), 334-343, 1985.
  * [4] F. Camia, J. Jiang, C. M. Newman. Monotonicity of Ursell Functions in the Ising Model. Communications in Mathematical Physics, 401(3), 2459–2482, 2023. DOI: 10.1007/s00220-023-04693-x.
  * [5] K. Becker, M. Becker, & J. H. Schwarz. String Theory and M-Theory: A Modern Introduction. Cambridge University Press, 2007.
  * [6] R. Diestel. Graph Theory. Springer, 2017.
  * [7] R. P. Feynman. Statistical Mechanics: A Set Of Lectures. Westview Press, 1998.
  * [8] R. J. Gillespie. The VSEPR model of molecular geometry. Journal of Chemical Education, 68(3), 178, 1991.
  * [9] M. B. Green, J. H. Schwarz, & E. Witten. Superstring Theory: Volume 1, Introduction. Cambridge University Press, 1987.
  * [10] R. A. Horn & C. R. Johnson. Matrix Analysis. Cambridge University Press, 2010.
  * [11] W. S. Massey. Algebraic Topology: An Introduction. Springer, 1967.
  * [12] A. Mucherino, C. Lavor, L. Liberti, & N. Maculan (Eds.). Distance Geometry: Theory, Methods, and Applications. Springer, 2012.
  * [13] J. R. Munkres. Topology. Prentice Hall, 2000.
  * [14] M. Nakahara. Geometry, Topology and Physics. CRC Press, 2003.
  * [15] L. Pauling. The Nature of the Chemical Bond. Cornell University Press, 1960.
  * [16] M. E. Peskin & D. V. Schroeder. An Introduction to Quantum Field Theory. Westview Press, 1995.
  * [17] J. Polchinski. String Theory. Cambridge University Press, 1998.
  * [18] C. Rovelli & F. Vidotto. Covariant Loop Quantum Gravity: An Elementary Introduction to Quantum Gravity and Spinfoam Theory. Cambridge University Press, 2014.
  * [19] R. H. Swendsen. An Introduction to Statistical Mechanics and Thermodynamics. Oxford Graduate Texts. Springer, 2012.
  * [20] A. Szabo & N. S. Ostlund. Modern Quantum Chemistry: Introduction to Advanced Electronic Structure Theory. Dover Publications, 1996.
  * [21] F. S. Woods. Advanced Calculus (New Edition). Ginn and Company, 1934.
  * [22] B. Zwiebach. A First Course in String Theory. Cambridge University Press, 2009.


# Asking the Right Question at the Right Time: Human and Model Uncertainty
Guidance to Ask Clarification Questions

Alberto Testoni  Raquel Fernández
Institute for Logic, Language and Computation
University of Amsterdam
{a.testoni<EMAIL_ADDRESS>

###### Abstract

Clarification questions are an essential dialogue tool to signal
misunderstanding, ambiguities, and under-specification in language use. While
humans are able to resolve uncertainty by asking questions since childhood,
modern dialogue systems struggle to generate effective questions. To make
progress in this direction, in this work we take a collaborative dialogue task
as a testbed and study how model uncertainty relates to human uncertainty—an
as yet under-explored problem. We show that model uncertainty does not mirror
human clarification-seeking behavior, which suggests that using human
clarification questions as supervision for deciding when to ask may not be the
most effective way to resolve model uncertainty. To address this issue, we
propose an approach to generating clarification questions based on model
uncertainty estimation, compare it to several alternatives, and show that it
leads to significant improvements in terms of task success. Our findings
highlight the importance of equipping dialogue systems with the ability to
assess their own uncertainty and exploit in interaction.

Asking the Right Question at the Right Time: Human and Model Uncertainty
Guidance to Ask Clarification Questions

Alberto Testoni and Raquel Fernández Institute for Logic, Language and
Computation University of Amsterdam {a.testoni<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: Overview of our experimental setup in CoDraw. After receiving an
instruction from the Teller, the Drawer agent selects the clipart(s) to draw,
together with their attributes. If the entropy over an attribute exceeds a
threshold $\theta$ (size in the figure), then the Drawer asks a clarification
question. The question-answer pair is added to the dialogue history before
performing the next drawing action. Different human players may react
differently (bottom-left box); the agent decides whether to ask for
clarification on the basis of its own uncertainty, independently from the
clarification decisions of human players.

The ability to ask effective and informative questions is a peculiar feature
of human intelligence. From an evolutionary perspective, it has been argued
that being able to ask questions to reliable informants gives humans a key
evolutionary advantage over other species (Tomasello, 2009). Questions are
also shown to play a fundamental role in cognitive development and language
learning during childhood (Demetras et al., 1986; Chouinard et al., 2007;
Leech et al., 2013; Ruggeri et al., 2016, 2017). Both children and adults
generally ask questions to resolve a state of uncertainty (Piaget, 1954; Rothe
et al., 2018). Deciding how to resolve uncertainties and maintaining mutual
understanding are essential steps to achieve conversational grounding (Clark
and Schaefer, 1987; Clark and Brennan, 1991), the process through which humans
negotiate the production and acceptance of their utterances. Clarification
questions are among the key strategies speakers use to achieve conversational
grounding. However, while humans handle and resolve uncertainties through
clarification questions almost effortlessly in everyday conversations, this is
particularly challenging for modern dialogue systems based on deep neural
networks (Chandu et al., 2021). Even the new generation of conversational
agents, such as ChatGPT,111https://chat.openai.com/ seem to struggle to
generate clarification questions when dealing with ambiguous instructions
(Deng et al., 2023).

In this paper, we investigate uncertainty-driven strategies for asking
clarification questions. Clarification requests are usually triggered when
ambiguity and/or underspecification make it difficult for the receiver to
follow up with enough confidence (Purver, 2004; Benotti and Blackburn, 2017;
Pezzelle and Fernández, 2023). It should come as no surprise then that there
is a long-standing tradition of work highlighting the importance of
incorporating clarification requests in dialogue systems (Paek and Horvitz,
2000; Gabsdil, 2003; Schlangen, 2004; Rieser and Moore, 2005; Rieser and
Lemon, 2006, among others). Recent years have witnessed a renewed interest in
this ambitious goal, especially in the context of collaborative dialogue tasks
(e.g., Nguyen and Daumé III, 2019; Chi et al., 2020; Zhu et al., 2021; Shi et
al., 2022; Shen and Lourentzou, 2023; Madureira and Schlangen, 2023b). Yet,
several key aspects remain under-explored.

For example, some existing approaches use a dedicated neural network model
trained on human clarification behaviour to decide when to ask (Shi et al.,
2022; Madureira and Schlangen, 2023b). However, it is well known that humans
may follow different confirmation-seeking strategies (Schegloff et al., 1977;
Schegloff, 1987; Purver et al., 2001). This potential human disagreement casts
doubt on the effectiveness of using human decisions as training signal for
supervised or reinforcement learning. As an alternative approach, model
uncertainty has often been employed to decide when and/or what to ask (Nguyen
and Daumé III, 2019; Zhu et al., 2021; Shen and Lourentzou, 2023; Chi et al.,
2020). Yet the two types of approaches have not been directly compared, nor is
there any study (to the best of our knowledge) on the extent to which model
uncertainty correlates with human uncertainty. In our work, we take a closer
look at these fundamental aspects.

We consider the CoDraw task and dataset (Kim et al., 2019)—where a Drawer
player is asked to recreate a clipart scene following instructions from a
Teller player—as a testbed to study uncertainty-driven clarification
strategies in collaborative agents. We propose different ways to represent
model uncertainty in CoDraw, and then carry out a study on the relation
between model uncertainty and human uncertainty, as measured by the decision
to ask clarification requests in human-human dialogues. This study reveals
that there is a poor relation between model and human uncertainty, which in
part appears to stem from our finding that in CoDraw human players do not
follow a consistent clarification strategy. This poor relation suggests that
using human clarification decisions as supervision for deciding when to ask
may not be the most effective way to resolve model uncertainty. We therefore
propose a new Drawer agent model equipped with an uncertainty-based
clarification module (illustrated in Figure 1), where we use a template-based
approach to generate clarification questions triggered by the uncertainty
state of the agent. This approach allows us to modulate the level of
uncertainty that leads to posing a question and thus control for the number of
questions asked per dialogue, which may be advantageous given the processing
cost of questions for humans as well as models (Chiyah-Garcia et al., 2023).

Our results and analyses show that our proposed model is significantly more
effective in terms of task success than several strong baselines, including a
version based on the human supervision model by Madureira and Schlangen
(2023b). Overall, our findings highlight the importance of equipping
conversational agents with the ability to assess and act upon their own
uncertainty by asking effective clarification questions.

##  2 Related Work

Clarification requests have long been studied as a key repair mechanism in
human dialogue (Gabsdil, 2003; Purver, 2004; Schlangen, 2004; Rieser and
Lemon, 2006; Schlöder and Fernández, 2014; Benotti and Blackburn, 2017, 2021).
Datasets of human conversations where spontaneous clarification questions are
annotated have been an important resource for data-driven research on
clarification strategies. For example, Rodríguez and Schlangen (2004) proposed
an annotation scheme based on Clark’s four levels of communication Clark
(1996) to identify the form and function of clarification requests. These and
similar annotation schemes have been applied to small datasets (Benotti, 2009;
Gervits et al., 2021; Benotti and Blackburn, 2021).

At a larger scale, the TEACh dataset (Padmakumar et al., 2022) has been
annotated by Gella et al. (2022) with dialogue acts, including acts related to
clarification, and the Minecraft Dialogue Corpus (Narayan-Chen et al., 2019)
and its extensions (Shi et al., 2022; Kiseleva et al., 2022; Mohanty et al.,
2022) have been annotated with different types of clarification requests.
However, in the latter case the dataset was augmented with further
clarification questions a-posteriori, introduced asynchronously by other
annotators who were asked to decide whether clarifications were needed.
Aliannejadi et al. (2021) and Gao et al. (2022) adopt similar techniques for
including clarification questions in dialogue data.

Recently, Madureira and Schlangen (2023b, a) released CoDraw-iCR (v1 and v2),
a fine-grained annotation of spontaneous clarification requests in the CoDraw
dataset (Kim et al., 2019). This constitutes a worthwhile resource for
analysing clarification behaviour in collaborative tasks. Madureira and
Schlangen (2023b) exploit these annotations to train a neural model for
deciding when to clarify. In contrast, we shall use them to study the relation
between human and model uncertainty in Section 5 and then, in Section 6,
propose a model for the CoDraw Drawer agent that asks clarification questions
on the basis of its own uncertainty.

Model uncertainty represents a valuable signal for Natural Language Generation
models to deal with the different sources of ambiguity in language use (Baan
et al., 2023). Earlier on, when the speech recogniser used to be a common
source of errors in spoken dialogue systems, ASR confidence scores were
exploited to decide on error handling strategies such as confirmation and
clarification (Skantze, 2008; Stoyanchev et al., 2014, among others). In more
recent systems, where the focus is on the next action to take in a
collaborative task, a common approach consists of using model uncertainty (as
measured by the entropy over the model prediction on a set of actions) to
signal ambiguity and trigger the generation of clarification questions (Nguyen
and Daumé III, 2019; Zhu et al., 2021; Shen and Lourentzou, 2023). Without
explicitly estimating uncertainty, Testoni and Bernardi (2021) propose a beam
search re-ranking algorithm guided by the model intermediate predictions about
the target in a referential task. Similar approaches use the probability
difference between the top two predicted actions to estimate uncertainty (Chi
et al., 2020). Recently, Naszadi et al. (2023) investigated the alignment
between predictive uncertainty and ambiguous instructions in visually-grounded
communication tasks. They show that well-calibrated prediction probabilities
benefit the detection of ambiguous instructions. Differently from our work,
the authors do not investigate the effectiveness of model uncertainty to
generate follow-up clarification questions.

Instead of relying on model uncertainty, other proposed methods involve
training external neural networks or sub-modules with the objective of
deciding when to ask for clarification Madureira and Schlangen (2023b); Shi et
al. (2022). Khalid et al. (2020) propose to incorporate cognitive modeling in
a Reinforcement Learning framework to generate context-sensitive clarification
strategies in a referential game. While a few approaches generate
clarification questions from scratch in an end-to-end fashion Khalid et al.
(2020); Zhu et al. (2021), heuristics are often applied at the generation
stage (Shen and Lourentzou, 2023; Zhang and Zhu, 2021; Sekulić et al., 2021).
In this paper, we propose an approach to generating clarification questions
based on model uncertainty estimation and compare it to alternative methods
for deciding when to clarify by training on human clarification decisions.

##  3 Task & Dataset

CoDraw (Kim et al., 2019) is a goal-driven instruction-following collaborative
task between two players. The Teller player has access to an abstract target
scene containing a variable number of cliparts (on average, target scenes
contain 6 cliparts), leveraging the Abstract Scene dataset (Zitnick and
Parikh, 2013; Zitnick et al., 2013). The Teller provides written instructions
in English to the Drawer in order to reconstruct the scene in a turn-based
fashion. The Drawer has access to a gallery of 58 cliparts that they can place
on a canvas. The Drawer can remove, resize (small, medium, large), or flip
(facing right or left) each object in the scene. After receiving an
instruction, the Drawer has to reply to the Teller before proceeding to the
following turn. Typically, the Drawer either replies with confirmation
feedback or asks follow-up questions. The CoDraw dataset consists of around
10k dialogues, with an average length of 7.7 turns per dialogue. Additional
details can be found in Appendix D.

In CoDraw, task success is measured by comparing the target image with the one
drawn by the Drawer at the end of the conversation. To this aim, Kim et al.
(2019) designed a similarity score metric, ranging from 0 to 5, tailored to
the CoDraw task. This score takes into account different cliparts (by checking
whether cliparts in the target scene appear in the reconstructed one) and
their attributes (size, position, flipping, plus some additional attributes
for cliparts representing people, i.e., facial expression and body pose) by
assigning different weights to each component. The scenes drawn by human
players reach a similarity score of 4.17 (sd=0.64). In our work, we apply some
changes to the similarity score metric to fix some flaws in its design and
better capture the properties under analysis in the experimental setup, as
described in Appendix A. In the following, we report the results using the new
version of the metric.

##  4 Computing Uncertainty in CoDraw

###  4.1 The Silent Drawer Model

Kim et al. (2019) proposed a neural Drawer model for the CoDraw task. At each
round of the dialogue, this model is conditioned on the Teller’s latest
message, which is encoded into a vector using a bidirectional LSTM module. The
Drawer also receives as input a vector representation of the current state of
the canvas that is being drawn (the cliparts added so far and their
attributes). These input representations are passed to a dense feed-forward
neural network. The resulting vector represents the Drawer’s follow-up action,
which determines the cliparts to be added to the canvas, together with their
attributes. Thus, Kim et al.’s Drawer model is silent, i.e., it can only draw
objects on the canvas but cannot contribute to the dialogue. The model is
trained in a Supervised Learning fashion on human data. At inference time, the
model assigns an unbounded score to each of the 58 available cliparts: each
clipart with a score above 0 is drawn on the canvas, together with its
predicted attributes (size, position, orientation, etc.). The model is tested
by providing turn-by-turn Teller’s instructions from the CoDraw test set,
reaching an average similarity score of 3.31 (sd=0.67).

###  4.2 Computing Model Uncertainty

We are interested in representing the uncertainty of the model regarding its
next action. After receiving an instruction, the Drawer model has to make
decisions about which clipart(s) to draw and their corresponding attributes.
For each dialogue turn and selected cliparts, we estimate model uncertainty
regarding these components as follows:

  * •

For clipart selection, we take the score assigned to each clipart as a proxy
of model uncertainty. The lower the value, the more uncertain the model is
about selecting the given clipart.

  * •

For the size (small, medium, large) and orientation (right, left) attributes,
we compute the entropy of the probability distribution over the possible
labels per attribute. The higher the entropy, the higher the uncertainty on
that attribute.

  * •

For the position attribute, the Drawer outputs two values per clipart,
corresponding to the $x$\- and $y$-axis coordinates of the canvas. Measuring
uncertainty from continuous values is inherently more challenging than
discrete variables. We take inspiration from uncertainty estimation using
Ensembles of different models (Lakshminarayanan et al., 2017) and consider the
variance among the prediction of five Drawer models trained with different
random initialization seeds as a representation of model uncertainty.

##  5 Model vs. Human Uncertainty

In this section, we investigate the relation between model and human
uncertainty. In the absence of clues such as eye gaze or reaction times (which
could be indicative of uncertainty, but are not available in CoDraw), we take
clarification questions as a signal of human uncertainty. We exploit the
annotation of clarification questions released by Madureira and Schlangen
(2023b, a), which consists of turns by the Drawer player annotated as
clarification requests plus the attributes mentioned in such requests (size,
position, orientation, etc.).222In particular, we take advantage of the
annotation in CoDraw-iCR (v2) by Madureira and Schlangen (2023a). Around 40%
of the dialogues in CoDraw contain at least one clarification question, with
an average number of 2.2 clarification requests per dialogue in this subset.

###  5.1 Setup

We explore whether we can use model uncertainty (as defined in the previous
section) to predict when human players ask clarification questions. To this
aim, we train a logistic regression model with human decisions about whether
to ask a clarification question in the follow-up turn as dependent variable
and each model uncertainty type (clipart selection, size, orientation,
position) as independent variables. We use the Drawer model described in
Section 4.1 to compute model uncertainty at each dialogue turn. Using the
methods described in Section 4.2, we compute uncertainty for each clipart with
a score above 0 and each of its attributes. If this includes more than one
clipart at a given turn, we take the highest uncertainty value per variable
(clipart selection, size, orientation, position).333Other approaches, such as
taking the average of the uncertainty values, do not lead to any significant
difference in the final results. We use the CoDraw test set (1002 dialogues,
with an average of 7.7 turns per dialogue) to extract model uncertainty. We
further split the CoDraw test set to train and evaluate the logistic
regression model (70% and 30%, respectively). We study the effect of different
initialization seeds and training epochs and compare the model performance to
a random baseline.

Figure 2:  Average Precision (AP, green) of a logistic regression model in
predicting when human players asked clarification questions based on the
uncertainty extracted from a Drawer trained for a variable number of epochs
($x$-axis). The plot also shows the similarity score (SS, blue) achieved by
the Drawer, and its uncertainty on clipart selection (normalized and reversed
clipart scores), and on the size and orientation attributes (as measured by
the entropy H). For each epoch, we report the average and variance of 5 Drawer
models trained with different seeds. Note that the $y$-axis has two scales: we
report SS on the left, while all the other metrics refer to the right side.

###  5.2 Results

The results are displayed in Figure 2. Let us consider the solid and dotted
green lines in the plot. The logistic regression model trained to predict when
human players ask clarification questions based on the uncertainty of the
Drawer model yields an Average Precision of 0.188 (epoch 15; average across 5
different initialization seeds). This result is above the random baseline
(0.188 vs. 0.142, $p<0.05$), indicating that model uncertainty has some degree
of predictive power. Nevertheless, performance is poor. Among the independent
variables analyzed, only the uncertainty about clipart selection has
statistically significant predictive power ($p<0.001$). Additional details
about the coefficients can be found in Appendix B.

Figure 2 provides additional interesting insights: we can see that the Drawer
model learns to perform the task over the training epochs, as evidenced by the
upward trend of the similarity score, which measures task success. At the same
time, the uncertainty of the model decreases (the plot shows uncertainty on
clipart selection, size, and orientation. Same pattern is observed for
position). Thus, the Drawer model is reasonably well calibrated: higher task
performance goes hand in hand with lower uncertainty. Yet, this trend does not
have an impact on the power of model uncertainty to predict human
clarification behaviour, as the Average Precision of the logistic regression
model remains constantly low over training epochs. We verified that other
methods to capture model uncertainty, such as extracting epistemic and
aleatoric uncertainty (Kendall and Gal, 2017) using an Ensemble of models,
lead to the same results and do not have significant predictive power.
Additional details can be found in Appendix B.

###  5.3 Analysis

There could be many reasons why model uncertainty and human uncertainty are
poorly related. Human players do not only have immensely more knowledge and
resources than the relatively simple Drawer model; different speakers may also
follow different conversation strategies and such variation may be difficult
to predict. It is well known that there is variability in human clarification-
seeking and repair strategies (Schegloff et al., 1977; Schegloff, 1987; Purver
et al., 2001). To check whether such variation is present in the CoDraw
dataset, we carry out an exploratory analysis. We use a keyword-matching
approach to cluster together instructions by Teller players with similar form
and content that lack some key information, and check how frequently they are
followed by clarification questions. We only consider Teller’s instructions
uttered in the first dialogue turn to exclude the possible confounding effect
of the previous dialogue history. Full details on the procedure and the
results can be found in Appendix B. Here we highlight the main takeaways. For
example, we observe that there are a total of 2274 Teller first instructions
that mention at least one clipart but do not specify its size (e.g., _‘a tree
to the left a few inches from the edge’_). Only 23% of follow-up turns by the
Drawer player include a clarification question, and among these, only 27% are
about the size of the clipart. Similarly, there are 240 Teller first
instructions that mention at least one clipart but do not specify its location
(e.g., _‘little girl running towards her friend that has a pie in his hand’_).
49% of follow-up turns include a clarification question, and among these, 48%
are about the location of the clipart.

Thus we observe that when analyzing utterances that do not mention a specific
necessary attribute, the human Drawer players may or may not ask about the
missing information. Humans may be more or less cautious, take more or fewer
risks, or guess/hope that more information will be provided later on in the
dialogue (Purver, 2004; Benotti and Blackburn, 2017, 2021). This high human
disagreement on deciding whether to clarify or not suggests that learning when
to ask clarification questions from human clarification behaviour may not be
optimal. Indeed the model proposed by Madureira and Schlangen (2023b) achieves
only 0.35 Average Precision. Instead, in the next section we investigate
whether model uncertainty represents an effective guidance to decide when to
ask clarification questions by studying their impact on task success.

##  6 Asking Clarification Questions

Figure 3: Example of clarification exchanges (in green) with questions
generated by QDrawer and answers by the Teller. The images on the right depict
the QDrawer’s canvas after each round of conversation.

We implement a Questioning Drawer model (QDrawer) by adapting the Silent
Drawer described in Section 4.1. We focus on the size attribute, because it
applies to all cliparts and it is often underspecified in the Teller’s
instructions (see Section 5). Our goal is to test whether asking clarification
questions triggered by size uncertainty results in (1) higher accuracy
regarding the size attribute, and (2) higher overall similarity score. To
isolate the effect of asking questions and avoid confounds related to language
generation, we use a template-based approach to generate clarification
questions and their answers.

###  6.1 The Uncertainty-based QDrawer

We adapt the Silent Drawer as follows: (a) the model receives as input the
immediately preceding turn exchange between the Drawer and Teller, instead of
just the Teller’s utterance; (b) we add a term in the cross-entropy loss to
include not only the cliparts that are added to the canvas but also the ones
that are updated by changing some attributes;444In this way, the model learns
to assign a high score to all cliparts mentioned in the instruction and not
just to the ones that do not appear yet on the canvas—an essential step to
successfully profit from clarification exchanges. See Appendix E for
additional details. (c) we include a clarification generation module that
works as follows: At any turn of the conversation, if the entropy over the
size attribute labels (small, medium, or large) exceeds a given
threshold,555We experiment with several thresholds, as it will become clear in
subsequent sections. QDrawer generates a clarification question targeting a
maximum number of 2 cliparts (i.e., the ones with the highest entropy).

We employ the following template, which corresponds to the most common
formulation of size questions in CoDraw according to Madureira and Schlangen
(2023a): “what size is the __?” or “what size are the __ and the __?”, where
the slots are filled with the name(s) of the clipart(s). We exploit the
ground-truth annotation in the CoDraw dataset to provide an answer to the
question on behalf of the Teller using these templates: “The __ is __” or “The
__ is __ and the __ is __”, where the slots are filled with the name(s) of the
clipart(s) under discussion and their size (small, medium, large). Before
performing a new drawing action, _Q_ Drawer is conditioned on the current
state of the canvas, as well as the clarification question asked and the
corresponding answer. Figure 3 shows an example of clarification questions
asked by QDrawer and the answers received, as well as the updated canvas after
each round of the conversation. Another example is reported in Appendix C.

We experimented with replacing the Bi-LSTM language encoder of the Drawer
agent (see Section 4.1) with more complex models (both fine-tuned on CoDraw
and trained from scratch), such as BERT and RoBERTa (Devlin et al., 2019; Liu
et al., 2019) and their corresponding distilled versions (Sanh et al., 2019).
We found that these BERT-based QDrawer variants give rise to the same patterns
on the role of clarification discussed in the results below, but lead to a
deterioration of model performance as measured by the similarity score (max.
3.07).666This result is in line with a strand of work highlighting that LSTM-
based models outperform BERT-like ones for small datasets and/or specific
domains (Zeyer et al., 2019; Ezen-Can, 2020; Le et al., 2021; Wei et al.,
2022). We thus decided to keep the original model architecture.777 Again, a
logistic regression model trained with the uncertain extracted from the Drawer
variants based on BERT and RoBERTa poorly correlates with human behaviour on
asking clarification questions, reaching an average precision (0.18 AP) in
line or slightly below the one discussed in Section 5.

###  6.2 Baselines

We compare _Q_ Drawer to the Silent Drawer model, as well as to different
configurations of _Q_ Drawer that differ with respect to the information used
to decide when to ask a question. In particular, we compare our proposed
uncertainty-based approach to the following alternatives:

  * •

The supervised learning approach by Madureira and Schlangen (2023b), where an
external neural model trained on human clarification behaviour is used to
decide when to ask (Decider). In this case, size uncertainty is still used to
select which clipart(s) to target in the question, and the clarification
question is generated and answered in the same way as in the uncertainty-based
_Q_ Drawer.

  * •

Asking questions at the same position (in terms of dialogue turn) where human
players ask questions about size in CoDraw.

  * •

Asking questions at random dialogue turns (discussed in Section 6.5).

|  | When
---
To Ask
| Size
---
Acc.
| SS
---
| CQs Size
---
Acc. boost
| CQs
---
SS boost
_S_ D | – | 76.9 | 3.31 | – | –
_Q_ D | Human | 77.3 |  3.31∘ |  +6.2% |  +.04∙
Decider | 80.1 | 3.34 |  +7.1% | +.06
_Q_ D | $\theta=0.3$ | 87.3 | 3.40 | +11.0% | +.09
$\theta=0.7$ | 86.0 | 3.39 | +11.0% | +.09
$\theta=1.1$ | 82.4 | 3.36 |  +9.8% | +.09

Table 1: Silent (_S_ D) and Questioning Drawer (_Q_ D) results using different
approaches to generate clarification questions. Top part: baselines. CQs:
clarification questions. Human: dialogue turns when human players ask CQs. SS:
similarity score on the whole test set. Size acc.: accuracy (%) of the size
attribute of all drawn cliparts on the whole test set. Boost columns: SS and
Size Acc. boost brought by CQs on the subset of dialogues containing at least
one CQ. All the differences are statistically significant except ∘ (non-
significant difference with SD) and ∙ (adding CQs does not significantly
increase SS on the subset).

###  6.3 Task Success Results

Table 1 summarizes the results of our experiments on the CoDraw test set. We
compare different models on their accuracy in correctly selecting the size of
the cliparts drawn on the canvas and the overall similarity score between the
drawn and the target scene at the end of the dialogue. The Silent Drawer (SD)
reaches a size accuracy of 76.9% and a similarity score of 3.31. The version
of _Q_ Drawer that generates size questions in the same position as human
players only marginally improves performance on size accuracy, while the
similarity score does not show any statistically significant increase (paired
t-test, $p>0.05$). Using an external neural Decider to choose when to ask
clarification questions improves the performance over SD (up to 80.1% accuracy
and 3.34 similarity score). Finally, the uncertainty-driven _Q_ Drawer
outperforms the baselines to a large extent: up to 87.3% size accuracy and
3.40 similarity score.888 An exploratory study on extending our approach to
other attributes reveals that using both size and orientation uncertainty
further increases the similarity score on the test set up to 3.46, indicating
that our proposed method can scale up. Regarding the comparison with the
Decider model, it is worth noting that this model is trained to ask questions
at each position where human players asked questions, not only when humans
asked size questions. While this method is somewhat coarse-grained (because
the Decider may ask size-CQ when they are not needed), it ensures full recall
(the Decider will ask a size question whenever humans asked a size question).
Hence, the Decider baseline is still meaningful with respect to the size
attribute.

The _Q_ Drawer model shows a consistent improvement as the entropy threshold
is lowered. This aspect is investigated in more detail in Section 6.5.

###  6.4 Clarification Questions’ Contribution

To further analyze how clarification exchanges contribute to task success, for
each model configuration, we consider the subset of dialogues ${CQ}$ where at
least one clarification question is asked. We then remove all clarification
questions from these dialogues, resulting in $!{CQ}$. We compute the
difference in size accuracy and similarity score between ${CQ}$ and $!{CQ}$ to
check the specific contribution of clarification questions (accuracy and SS
boost). As we can see on Table 1, the boost brought about by clarification
questions is much higher for the uncertainty-driven _Q_ Drawer than for the
baseline models. In particular, for the subset of dialogues in ${CQ}$,
clarification exchanges increase size accuracy by $+11\%$ in absolute terms
and the similarity score by $+0.09$. Asking questions in the same position as
human players does not significantly increase the similarity score when
comparing ${CQ}$ and $!{CQ}$ (paired t-test, $p>0.05$). To make sure that this
statically non-significant result does not stem from our generation approach,
we replace machine-generated clarification questions with human ones for the
dialogues in ${CQ}$. We find that in this case the increase in both size
accuracy and similarity score is even less pronounced than what we observe in
Table 1, resulting in non-significant results for both question-generation
methods.

We further investigate the impact of clarification exchanges by analyzing the
calibration of the model uncertainty estimates. We compare the probability
distribution over the size attribute against the ground-truth size of each
object by using two metrics: the Expected Calibration Error (ECE, Naeini et
al., 2015; Guo et al., 2017) and the Brier score (Brier, 1950). For both
metrics, a lower score indicates better calibration.999We refer to Ovadia et
al. (2019) for a broader discussion on calibration metrics. We compare the
Silent Drawer and _Q_ Drawer ($\theta=0.3$) and find that the Silent Drawer
reaches an ECE value of 0.10 while the ECE for _Q_ Drawer goes down to 0.05
(the difference is significant: t-test, $p<0.001$). Similarly, the Brier score
significantly improves (t-test, $p<0.001$) from 0.36 (Silent Drawer) to 0.28
(_Q_ Drawer). These results indicate that asking clarification questions
improves the calibration of the model uncertainty estimates.

$\theta$ |  | % dialogues with
---
at least 1 CQ
| avg. number of
---
CQ per dialogue
0.3 | 96.6 | 3.63
0.7 | 84.7 | 2.69
1.1 | 57.1 | 1.84

Table 2: The effect of different $\theta$ values on the number of questions
asked by _Q_ Drawer. We report the percentage of dialogues with at least one
clarification question and, within this subset, the average number of
clarification questions per dialogue. Figure 4: Effect of the average number
of questions per dialogue (considering all dialogues in the test set) on the
size accuracy. We compare the uncertainty-guided _Q_ Drawer with a version
that asks questions in random turns and one that asks questions in the turns
selected by an external Decider model.

###  6.5 How Many Questions to Ask

Asking clarification questions carries a cost, both from the perspective of
the agent asking the question and the agent processing it (Clark, 1996;
Purver, 2004). We take this partially into account by adopting a simple
approach: we evaluate _Q_ Drawer by using different entropy threshold values
$\theta$ to control for the number of questions generated by the model. As
reported in Table 2, in _Q_ Drawer the value of the entropy threshold $\theta$
has, by design, a direct impact on the number of questions asked: i.e., a
model that only asks for clarification in the face of high uncertainty
($\theta=1.1$) gives rise to fewer dialogues with questions and to a lower
average number of questions per dialogue.

We compare the uncertainty-driven _Q_ Drawer against (1) a version that asks
questions in random dialogue turns with different average number of questions
per dialogue and (2) the _Q_ Drawer paired with the Decider module as
described in Section 6.2. For the latter, the maximum possible number of
questions per dialogue (average of 1.5) is defined by the Decider model
output. Figure 4 shows the effect of the number of questions on size accuracy
for different model configurations. While the performance of the Decider model
almost overlaps with the _Q_ Drawer asking questions in random dialogue turns,
using model uncertainty significantly improves accuracy, even when a few
questions are asked. This result confirms that even when controlling for the
number of questions asked (which may be desirable in terms of cost control),
the uncertainty-driven _Q_ Drawer shows the most promising results.

##  7 Conclusions

Different kinds of uncertainties characterize language use, and asking
informative and effective questions is a crucial tool to address these
uncertainties. While humans are able to ask strategic questions since early
childhood to resolve a state of uncertainty, equipping modern dialogue systems
with the ability to ask effective questions has proved challenging. Some
existing approaches propose employing dedicated neural network models trained
on human behaviour to decide when to ask for clarification in a dialogue.
However, the high variability in human clarification strategies casts doubt on
the effectiveness of this approach. Other proposed methods involve using model
uncertainty to decide when and/or what to ask. Yet, estimating and exploiting
model uncertainty is not trivial, and the relationship between model and human
uncertainty is still unclear.

In our work, we considered the CoDraw collaborative dialogue task as a test
bed. We proposed different ways of extracting model uncertainty and
investigated its relationship to human uncertainty, as measured by the
clarification decisions of human players. We found that human and model
uncertainty are poorly correlated, possibly due to the wide diversity of human
clarification strategies. We then presented an approach that makes use of
model uncertainty to generate clarification questions, and compared it to
other methods for deciding when clarification is needed. Our approach allows
us to vary the level of uncertainty that elicits the generation of
clarification questions to control for the number of questions generated. The
results of our experiments show that our proposal outperforms a number of
baselines and other methods trained on human clarification decisions. More
generally, our findings pave the way for further research on integrating model
uncertainty into natural language generation approaches as an effective way to
resolve ambiguities and address underspecification in language use (Pezzelle,
2023).

## Limitations

Even though our experimental setup (which focuses on the size attribute using
a template-based generation approach) allows us to isolate the effect of
clarification exchanges in a controlled setting, we acknowledge that further
work is required to confirm our findings in more complex collaborative tasks
and model architectures. In our work, we focused on questions about the size
attribute, and we mentioned some results about expanding our approach to the
orientation attribute to assess the robustness of our approach. Assessing the
contribution of clarification exchanges is particularly challenging given the
wide range of factors involved, both on the appropriateness of the question
asked and the correctness of the answer received. Without some restrictions,
it is difficult to disentangle the role of these factors and this may threaten
a fair model evaluation. For this reason, we focus on the size attribute, as
it applies to all cliparts and it is often underspecified in the Teller’s
instructions (see Section 5). An additional advantage of focusing on the size
attribute is that we can ensure the correctness of the answer (see the
templates in Section 6.1), while for other attributes such as position there
is a less direct mapping between natural language expressions and space
coordinates.

Regarding the generalization of our setup to other settings, we would like to
highlight that, although simple, the CoDraw task shares many features with a
wide variety of different tasks and datasets. In particular, in Vision-and-
Language Navigation tasks an agent has to reach a goal by moving around an
environment and performing specific tasks. We believe this setting, where the
agent has a set of actions at their disposal, shares many similarities with
the CoDraw setup. On the other hand, in more complex tasks the role of the
visual input may play a greater role compared to CoDraw, leading to new
challenges in evaluating the grounding of the generated question-answer
exchanges. At the same time, we acknowledge the fact that a template-based
approach may very well not scale up to other settings. However, in our work,
this choice is motivated by the focus on the effect of clarifying rather than
the question generation problem.

Finally, we only used a simple approach to control for the “cost” of asking
clarification questions by limiting the overall number of possible questions.
We acknowledge that a more fine-grained approach is required to consider the
wide variety of different factors that influence the cost of generating
clarification requests. Given the simplicity of the CoDraw task and the
structure of the questions our method generates, we do not analyze how
clarification questions may influence the perceived naturalness of the
generated dialogues. We acknowledge that this aspect should be taken into
account when dealing with more complex tasks and generative strategies.

## Acknowledgments

We would like to thank Brielen Madureira for discussions about the CoDraw task
and dataset, as well as for sharing insights on CoDraw-iCR. We are grateful to
Patrick Kahardipraja for sharing an updated version of the original CoDraw
codebase. Finally, we thank the Dialogue Modelling Group (DMG) at the
University of Amsterdam for their feedback and support at the different stages
of this work. This project has received funding from the European Research
Council (ERC) under the European Union’s Horizon 2020 research and innovation
programme (grant agreement No. 819455).

## References

  * Aliannejadi et al. (2021) Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021.  Building and evaluating open-domain dialogue corpora with clarifying questions.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4473–4484, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Baan et al. (2023) Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023.  Uncertainty in natural language generation: From theory to applications.  _arXiv preprint arXiv:2307.15703_.
  * Benotti (2009) Luciana Benotti. 2009.  Clarification potential of instructions.  In _The 10th Annual SIGDIAL Meeting on Discourse and Dialogue_.
  * Benotti and Blackburn (2017) Luciana Benotti and Patrick Blackburn. 2017.  Modeling the clarification potential of instructions: Predicting clarification requests and other reactions.  _Computer Speech & Language_, 45:536–551.
  * Benotti and Blackburn (2021) Luciana Benotti and Patrick Blackburn. 2021.  A recipe for annotating grounded clarifications.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4065–4077, Online. Association for Computational Linguistics.
  * Brier (1950) Glenn W Brier. 1950.  Verification of forecasts expressed in terms of probability.  _Monthly weather review_ , 78(1):1–3.
  * Chandu et al. (2021) Khyathi Raghavi Chandu, Yonatan Bisk, and Alan W Black. 2021.  Grounding ‘grounding’ in NLP.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4283–4305, Online. Association for Computational Linguistics.
  * Chi et al. (2020) Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, and Dilek Hakkani-tur. 2020.  Just ask: An interactive learning framework for vision and language navigation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34-03, pages 2459–2466.
  * Chiyah-Garcia et al. (2023) Javier Chiyah-Garcia, Alessandro Suglia, Arash Eshghi, and Helen Hastie. 2023.  ’What are you referring to?’ Evaluating the ability of multi-modal dialogue models to process clarificational exchanges.  In _Proceedings of the 24rd Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL’23)_.
  * Chouinard et al. (2007) Michelle M Chouinard, Paul L Harris, and Michael P Maratsos. 2007.  Children’s questions: A mechanism for cognitive development.  _Monographs of the society for research in child development_ , pages i–129.
  * Clark (1996) Herbert H Clark. 1996.  _Using language_.  Cambridge university press.
  * Clark and Brennan (1991) Herbert H Clark and Susan E Brennan. 1991.  Grounding in communication.  In _Perspectives on socially shared cognition._ , pages 127–149. American Psychological Association.
  * Clark and Schaefer (1987) Herbert H Clark and Edward F Schaefer. 1987.  Collaborating on contributions to conversations.  _Language and cognitive processes_ , 2(1):19–41.
  * Demetras et al. (1986) Marty J Demetras, Kathryn Nolan Post, and Catherine E Snow. 1986.  Feedback to first language learners: The role of repetitions and clarification questions.  _Journal of child language_ , 13(2):275–292.
  * Deng et al. (2023) Yang Deng, Wenqiang Lei, Lizi Liao, and Tat-Seng Chua. 2023.  Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration.  _arXiv preprint arXiv:2305.13626_.
  * Depeweg et al. (2018) Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. 2018.  Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning.  In _International Conference on Machine Learning_ , pages 1184–1193. PMLR.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ezen-Can (2020) Aysu Ezen-Can. 2020.  A comparison of LSTM and BERT for small corpus.  _arXiv preprint arXiv:2009.05451_.
  * Gabsdil (2003) Malte Gabsdil. 2003.  Clarification in spoken dialogue systems.  In _Proceedings of the 2003 AAAI Spring Symposium. Workshop on Natural Language Generation in Spoken and Written Dialogue_ , pages 28–35.
  * Gao et al. (2022) Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme. 2022.  Dialfred: Dialogue-enabled agents for embodied instruction following.  _IEEE Robotics and Automation Letters_ , 7(4):10049–10056.
  * Gella et al. (2022) Spandana Gella, Aishwarya Padmakumar, Patrick Lange, and Dilek Hakkani-Tur. 2022.  Dialog acts for task driven embodied agents.  In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 111–123, Edinburgh, UK. Association for Computational Linguistics.
  * Gervits et al. (2021) Felix Gervits, Antonio Roque, Gordon Briggs, Matthias Scheutz, and Matthew Marge. 2021.  How should agents ask questions for situated learning? an annotated dialogue corpus.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 353–359, Singapore and Online. Association for Computational Linguistics.
  * Greco et al. (2022) Claudio Greco, Alberto Testoni, Raffaella Bernardi, and Stella Frank. 2022.  A small but informed and diverse model: The case of the multimodal GuessWhat!? guessing game.  In _Proceedings of the 2022 CLASP Conference on (Dis)embodiment_ , pages 1–10, Gothenburg, Sweden. Association for Computational Linguistics.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International conference on machine learning_ , pages 1321–1330. PMLR.
  * Hüllermeier and Waegeman (2021) Eyke Hüllermeier and Willem Waegeman. 2021.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  _Machine Learning_ , 110:457–506.
  * Kendall and Gal (2017) Alex Kendall and Yarin Gal. 2017.  What uncertainties do we need in bayesian deep learning for computer vision?  _Advances in neural information processing systems_ , 30.
  * Khalid et al. (2020) Baber Khalid, Malihe Alikhani, and Matthew Stone. 2020.  Combining cognitive modeling and reinforcement learning for clarification in dialogue.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4417–4428, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Kim et al. (2019) Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019.  CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6495–6513, Florence, Italy. Association for Computational Linguistics.
  * Kiseleva et al. (2022) Julia Kiseleva, Alexey Skrynnik, Artem Zholus, Shrestha Mohanty, Negar Arabzadeh, Marc-Alexandre Côté, Mohammad Aliannejadi, Milagro Teruel, Ziming Li, Mikhail Burtsev, et al. 2022.  Iglu 2022: Interactive grounded language understanding in a collaborative environment at neurips 2022.  _arXiv preprint arXiv:2205.13771_.
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ , 30.
  * Le et al. (2021) Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. 2021.  Hierarchical conditional relation networks for multimodal video question answering.  _International Journal of Computer Vision_ , 129(11):3027–3050.
  * Leech et al. (2013) Kathryn A Leech, Virginia C Salo, Meredith L Rowe, and Natasha J Cabrera. 2013.  Father input and child vocabulary development: The importance of wh questions and clarification requests.  In _Seminars in speech and language_ , volume 34-04, pages 249–259. Thieme Medical Publishers.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Madureira and Schlangen (2023a) Brielen Madureira and David Schlangen. 2023a.  "Are you telling me to put glasses on the dog?” Content-grounded annotation of instruction clarification requests in the codraw dataset.  _arXiv preprint arXiv:2306.02377_.
  * Madureira and Schlangen (2023b) Brielen Madureira and David Schlangen. 2023b.  Instruction clarification requests in multimodal collaborative dialogue games: Tasks, and an analysis of the CoDraw dataset.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2303–2319, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Mohanty et al. (2022) Shrestha Mohanty, Negar Arabzadeh, Milagro Teruel, Yuxuan Sun, Artem Zholus, Alexey Skrynnik, Mikhail Burtsev, Kavya Srinet, Aleksandr Panov, Arthur Szlam, et al. 2022.  Collecting interactive multi-modal datasets for grounded language understanding.  _arXiv preprint arXiv:2211.06552_.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015.  Obtaining well calibrated probabilities using bayesian binning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 29.
  * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019.  Collaborative dialogue in Minecraft.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5405–5415, Florence, Italy. Association for Computational Linguistics.
  * Naszadi et al. (2023) Kata Naszadi, Putra Manggala, and Christof Monz. 2023.  Aligning predictive uncertainty with clarification questions in grounded dialog.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 14988–14998, Singapore. Association for Computational Linguistics.
  * Nguyen and Daumé III (2019) Khanh Nguyen and Hal Daumé III. 2019.  Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 684–695, Hong Kong, China. Association for Computational Linguistics.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  _Advances in neural information processing systems_ , 32.
  * Padmakumar et al. (2022) Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2022.  Teach: Task-driven embodied agents that chat.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36-2, pages 2017–2025.
  * Paek and Horvitz (2000) Tim Paek and Eric Horvitz. 2000.  Conversation as action under uncertainty.  In _Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence_ , pages 455–464.
  * Pezzelle (2023) Sandro Pezzelle. 2023.  Dealing with semantic underspecification in multimodal NLP.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12098–12112, Toronto, Canada. Association for Computational Linguistics.
  * Pezzelle and Fernández (2023) Sandro Pezzelle and Raquel Fernández. 2023.  Semantic adaptation to the interpretation of gradable adjectives via active linguistic interaction.  _Cognitive Science_ , 47(2):e13248.
  * Piaget (1954) J Piaget. 1954.  The construction of reality in the child (vol. xiii).  _New York: Basic Books. https://doi. org/10_ , 1037:11168–000.
  * Purver et al. (2001) Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2001.  On the means for clarification in dialogue.  In _Proceedings of the Second SIGdial Workshop on Discourse and Dialogue_.
  * Purver (2004) Matthew Richard John Purver. 2004.  _The theory and use of clarification requests in dialogue_.  Ph.D. thesis, University of London.
  * Rieser and Lemon (2006) Verena Rieser and Oliver Lemon. 2006.  Using machine learning to explore human multimodal clarification strategies.  In _Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions_ , pages 659–666, Sydney, Australia. Association for Computational Linguistics.
  * Rieser and Moore (2005) Verena Rieser and Johanna D Moore. 2005.  Implications for generating clarification requests in task-oriented dialogues.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 239–246.
  * Rodríguez and Schlangen (2004) Kepa Joseba Rodríguez and David Schlangen. 2004.  Form, intonation and function of clarification requests in german task-oriented spoken dialogues.  In _Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04)_.
  * Rothe et al. (2018) Anselm Rothe, Brenden M Lake, and Todd M Gureckis. 2018.  Do people ask good questions?  _Computational Brain & Behavior_, 1:69–89.
  * Ruggeri et al. (2016) Azzurra Ruggeri, Tania Lombrozo, Thomas L Griffiths, and Fei Xu. 2016.  Sources of developmental change in the efficiency of information search.  _Developmental psychology_ , 52(12):2159.
  * Ruggeri et al. (2017) Azzurra Ruggeri, Zi Lin Sim, and Fei Xu. 2017.  “why is toma late to school again?” preschoolers identify the most informative questions.  _Developmental psychology_ , 53(9):1620.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Schegloff (1987) Emanuel A. Schegloff. 1987.  Some sources of misunderstanding in talk-in-interaction.  _Linguistics_ , 25(287):201–218.
  * Schegloff et al. (1977) Emanuel A Schegloff, Gail Jefferson, and Harvey Sacks. 1977.  The preference for self-correction in the organization of repair in conversation.  _Language_ , 53(2):361–382.
  * Schlangen (2004) David Schlangen. 2004.  Causes and strategies for requesting clarification in dialogue.  In _Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004_ , pages 136–143.
  * Schlöder and Fernández (2014) Julian J Schlöder and Raquel Fernández. 2014.  Clarification requests on the level of uptake.  In _Proceedings of the 18th Workshop on the Semantics and Pragmatics of Dialogue (SemDial 2014,‘Dial-Watt’)_ , pages 237–239. Citeseer.
  * Sekulić et al. (2021) Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2021.  Towards facet-driven generation of clarifying questions for conversational search.  In _Proceedings of the 2021 ACM SIGIR international conference on theory of information retrieval_ , pages 167–175.
  * Shen and Lourentzou (2023) Ying Shen and Ismini Lourentzou. 2023.  Learning by asking for embodied visual navigation and task completion.  _arXiv preprint arXiv:2302.04865_.
  * Shi et al. (2022) Zhengxiang Shi, Yue Feng, and Aldo Lipani. 2022.  Learning to execute actions or ask clarification questions.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2060–2070, Seattle, United States. Association for Computational Linguistics.
  * Skantze (2008) Gabriel Skantze. 2008.  Galatea: A discourse modeller supporting concept-level error handling in spoken dialogue systems.  _Recent Trends in Discourse and Dialogue_ , pages 155–189.
  * Stoyanchev et al. (2014) Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. 2014.  Towards natural clarification questions in dialogue systems.  In _AISB symposium on questions, discourse and dialogue_ , volume 20.
  * Testoni and Bernardi (2021) Alberto Testoni and Raffaella Bernardi. 2021.  Looking for confirmations: An effective and human-like visual dialogue strategy.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9330–9338, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tomasello (2009) Michael Tomasello. 2009.  _The cultural origins of human cognition_.  Harvard university press.
  * Wei et al. (2022) Xin Wei, Jian Wu, Kehinde Ajayi, and Diane Oyen. 2022.  Visual descriptor extraction from patent figure captions: A case study of data efficiency between bilstm and transformer.  In _Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries_ , pages 1–5.
  * Zeyer et al. (2019) Albert Zeyer, Parnia Bahar, Kazuki Irie, Ralf Schlüter, and Hermann Ney. 2019.  A comparison of Transformer and LSTM encoder decoder models for asr.  In _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pages 8–15. IEEE.
  * Zhang and Zhu (2021) Zhiling Zhang and Kenny Zhu. 2021.  Diverse and specific clarification question generation with keywords.  In _Proceedings of the Web Conference 2021_ , pages 3501–3511.
  * Zhu et al. (2021) Yi Zhu, Yue Weng, Fengda Zhu, Xiaodan Liang, Qixiang Ye, Yutong Lu, and Jianbin Jiao. 2021.  Self-motivated communication agent for real-world vision-dialog navigation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1594–1603.
  * Zitnick and Parikh (2013) C Lawrence Zitnick and Devi Parikh. 2013.  Bringing semantics into focus using visual abstraction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3009–3016.
  * Zitnick et al. (2013) C Lawrence Zitnick, Devi Parikh, and Lucy Vanderwende. 2013.  Learning the visual interpretation of sentences.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1681–1688.

## Appendix

##  Appendix A Similarity Score (v2)

In our work, we apply some changes to the similarity score metric proposed in
Kim et al. (2019) to fix some flaws in its design and better capture the
properties under analysis in the experimental setup. More specifically, (a) we
assign the same weights to all components of the similarity score; (b) we
consider the orientation attribute only for those cliparts that are not
perfectly symmetrical; (c) we change the denominator of the facial expression
and body pose attributes so that they apply only to cliparts representing
people. Using the new metric, scenes drawn by human players reach a similarity
score of 4.42 (sd=0.62).

##  Appendix B Model vs. Human Uncertainty

#### Logistic Regression results and analysis

Table 3 shows the Average Precision of the logistic regression model trained
to predict when human players ask clarification questions in CoDraw based on
model uncertainty. We extract uncertainty from Drawer models trained for
different training epochs and initialization seeds. We also report the
performance of a random baseline and the CoDraw similarity score for each
configuration. Similarly, Table 4 reports the F1 score of the same model.
Table 5 shows the correlation coefficients of the logistic regression model
for each attribute-specific uncertainty analyzed (clipart selection, size,
orientation, position). We report the coefficients for each model
configuration (initialization seed + training epoch). Finally, Table 6
illustrates some insights and examples of the “clusters” analyzed in Section 5
to investigate the variability of human confirmation-seeking strategies.

#### Epistemic and Aleatoric uncertainty

We also experimented with including Epistemic and Aleatoric uncertainty in the
Logistic Regression analysis. We obtain the same results as using the other
uncertainty measures described above, without any statistically significant
predictive power. Similarly to Greco et al. (2022), we use an Ensemble of
models trained with different initialization seeds to compute data (aleatoric)
and model (epistemic) uncertainty (Depeweg et al., 2018; Hüllermeier and
Waegeman, 2021) for the size and orientation attributes. The ensemble _total
uncertainty_ is computed as the entropy of its predictive distribution
$[p_{\theta_{n}}(y|x)]$, which averages the distributions of the single
ensemble components:

$H[p(y|x)]=H[1/N\sum_{n=1}^{N}p_{\theta_{n}}(y|x)].$ (1)

_Data uncertainty_ is measured as the average uncertainty of each ensemble
component: $1/N\sum_{n=1}^{N}H[p_{\theta_{n}}(y|x)]$. The difference between
total uncertainty and data uncertainty results in _model uncertainty_. As
mentioned above, including these uncertainties in the logistic regression
model described in Section 5 does not improve its Average Precision, which
remains significantly above chance level but low in absolute values (0.19 AP,
similar to the values reported in Figure 2).

##  Appendix C Asking Clarification Questions

Figure 5 shows some examples of clarification questions asked by different
models for the same dialogue.

##  Appendix D The CoDraw Dataset and Models

The Codraw dataset (Kim et al., 2019) is available at this URL:
https://github.com/facebookresearch/CoDraw. CoDraw is licensed under Creative
Commons Attribution-NonCommercial 4.0 International Public License. The CoDraw
dataset is fully anonymized. The Silent drawer model is available at this
address: https://github.com/facebookresearch/codraw-models. This repository is
licensed under Creative Commons Attribution-NonCommercial 4.0 International
Public License. We use the released data and models consistently with their
intended use. Codraw-iCR (v1/v2) is licensed under Creative Commons
Attribution-NonCommercial 4.0 International Public License. The CoDraw dataset
contains 9993 dialogues, with a total number of 138K utterances. The dataset
is randomly split into training, validation, and test set with the following
proportions: 80%, 10%, 10%, respectively.

##  Appendix E _Q_ Drawer Details

The _Q_ Drawer model has the same architecture as the Silent Drawer described
in Section 4.1. The model has 3.347.390 parameters. For training
configuration, we follow the original paper (Kim et al., 2019). We trained the
model on a NVIDIA RTX A5000 GPU for 15 epochs ($\approx$ 1 minute/epoch). In
_Q_ Drawer, we compute the entropy using the SciPy library
(https://docs.scipy.org/doc/scipy/index.html). As mentioned in Section 6, we
slightly modify the loss function of the Silent Drawer. More specifically, the
Silent Drawer is trained using a combination of losses, including a cross-
entropy loss for categorical decisions on which clip art pieces to add to the
canvas at a given dialogue turn. This loss considers which new cliparts human
players added to the scene at each turn of the dialogue. If the Drawer player
updates some attributes of a clipart that was added to the canvas in previous
turns (for instance, changing its size or location), this clipart is not
included in the loss function. This step is crucial because, at inference
time, only the cliparts whose score exceeds a threshold are added to and
updated in the canvas. To address this issue, at each dialogue turn, we
include in the loss function of _Q_ Drawer not only the new cliparts added to
the canvas but also the ones already in the canvas that are modified by the
Drawer at that turn. In this way, clarification exchanges on cliparts added in
previous dialogue turns can be properly addressed.

Average Precision
---
|  | traning epoch
|  | 1 | 5 | 10 | 15
seed | 0 | .176 (.134) | .186 (.150) | .181 (.143) | .205 (.143)
1 | .179 (.150) | .171 (.152) | .185 (.1.38) | .172 (.126)
2 | .168 (.142) | .181 (.139) | .176 (.137) | .196 (.1.48)
3 | .194 (.145) | .190 (.153) | .213 (.154) | .189 (.161)
4 | .202 (.156) | .179 (.145) | .176 (.125) | .180 (.135)
Similarity Score
---
|  | traning epoch
|  | 1 | 5 | 10 | 15
seed | 0 | 3.02 | 3.33 | 3.32 | 3.31
1 | 3.05 | 3.33 | 3.31 | 3.34
2 | 3.02 | 3.31 | 3.34 | 3.36
3 | 3.01 | 3.30 | 3.35 | 3.34
4 | 3.04 | 3.31 | 3.35 | 3.35

Table 3: On the left, Average Precision of the logistic regression model
trained to predict human decisions to ask clarification questions based on
model uncertainty. In parenthesis, we report the performance of a random
model, which is always significantly lower than its trained counterpart
($p<0.05$). We study the effect of different initialization seeds and training
epochs. On the right, we show the corresponding average similarity score
reached by the Drawer model at the end of the dialogue in the CoDraw task.

F1 Score - Logistic Regression Model
---
|  | traning epoch
|  | 1 | 5 | 10 | 15
seed | 0 | 0.479 | 0.482 | 0.477 | 0.468
1 | 0.493 | 0.499 | 0.485 | 0.463
2 | 0.475 | 0.487 | 0.472 | 0.481
3 | 0.469 | 0.481 | 0.496 | 0.486
4 | 0.483 | 0.479 | 0.474 | 0.474
F1 Score - Random baseline
---
|  | traning epoch
|  | 1 | 5 | 10 | 15
seed | 0 | 0.469 | 0.492 | 0.509 | 0.513
1 | 0.500 | 0.525 | 0.490 | 0.488
2 | 0.505 | 0.495 | 0.504 | 0.503
3 | 0.481 | 0.507 | 0.474 | 0.520
4 | 0.493 | 0.514 | 0.490 | 0.519

Table 4: F1 score of the logistic regression model (left) and the performance
of a random baseline (right).

seed | epoch | uncertainty(clipart selection) | uncertainty(size) | uncertainty(orientation) | uncertainty(position)
---|---|---|---|---|---
0 | 1 | -0.397 | 0.002 | -0.041 | -0.096
0 | 5 | -0.392 | 0.006 | -0.038 | -0.012
0 | 10 | -0.390 | 0.004 | -0.044 | -0.075
0 | 15 | -0.390 | 0.006 | -0.044 | 0.040
1 | 1 | -0.397 | 0.002 | -0.041 | -0.087
1 | 5 | -0.392 | 0.006 | -0.038 | 0.016
1 | 10 | -0.390 | 0.004 | -0.044 | -0.027
1 | 15 | -0.390 | 0.006 | -0.044 | 0.073
2 | 1 | -0.397 | 0.002 | -0.041 | -0.070
2 | 5 | -0.392 | 0.006 | -0.038 | 0.025
2 | 10 | -0.390 | 0.004 | -0.044 | -0.054
2 | 15 | -0.390 | 0.006 | -0.044 | 0.103
3 | 1 | -0.397 | 0.002 | -0.041 | -0.054
3 | 5 | -0.392 | 0.006 | -0.038 | -0.037
3 | 10 | -0.390 | 0.004 | -0.044 | 0.007
3 | 15 | -0.390 | 0.006 | -0.044 | -0.015
4 | 1 | -0.397 | 0.002 | -0.041 | -0.167
4 | 5 | -0.392 | 0.006 | -0.038 | -0.019
4 | 10 | -0.390 | 0.004 | -0.044 | -0.048
4 | 15 | -0.390 | 0.006 | -0.044 | -0.017

Table 5: Correlation coefficients of the logistic regression model for each
uncertainty type extracted from the Drawer model. We show the coefficients for
each model configuration, as defined by its initialization seed and number of
training epochs.

| # utts. | Clip. | Size | Loc. | Examples | % CQ | CQ type
---|---|---|---|---|---|---|---
A | 3805 | 1 | 1 | $\geq 1$ |  | \- large pine tree on right half of top
---
and right edge cut off
7.6% | other 48.3%
| \- middle scene swing, small size. facing right.
---
red part above horizon
loc. 43.8%
B | 240 | $\geq 1$ | - | 0 |  | \- little girl running towards her friend that has
---
a pie in his hand
48.8% | loc. 48.3%
\- girl and boy, boy is on swing set | other 35.6%
C | 2274 | $\geq 1$ | 0 | - | \- a tree to the left a few inches from the edge | 23.1% | other 40.6%
| \- left side is a bear, facing right, left arm cut
---
from scene. bear cut in half by horizon line
size 27.1%
D | 76 | 1 | - | 0 | \- it ’s at a park with a side and a sandbox | 46.1% | loc. 57.1%
\- up in the tree are 4 balloons . small size | other 22.9%
E | 1388 | 1 | 0 | - | \- boy far lower right , facing left with happy face | 19.5% | size 40.0%
| \- girl sitting on ground , fingers touching edge
---
of left screen , 1 2 ” from bottom screen
other 30.4%
F | 5328 | 1 | - | - |  | \- small maple tree on left . almost all leaves
---
above grass line
10.1% | other 40.1%
\- left top small sun left edge little bit cut | loc. 34.8%

Table 6: Cluster analysis by grouping together similar instructions. # utts.
refers to the number of utterances in the cluster; Clip., Size, and Loc. refer
to the occurrences of clipart, size, and location keywords, respectively. % CQ
refers to the percentage of clarification questions following the given set of
instructions. “-” means no restrictions on that attribute.

(a)

(b)

Figure 5: Examples of dialogues and clarification questions asked by different
models. T stands for Teller, SD for Silent Drawer, and QD for _Q_ Drawer.
Clarification exchanges are highlighted in green.