# It Couldn’t Help But Overhear: On the Limits of Modelling Meta-Communicative
Grounding Acts with Supervised Learning

Brielen Madureira1 David Schlangen1,2
1Computational Linguistics, Department of Linguistics
University of Potsdam, Germany
2German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
<EMAIL_ADDRESS>
###### Abstract

Active participation in a conversation is key to building common ground, since
understanding is jointly tailored by producers and recipients. Overhearers are
deprived of the privilege of performing grounding acts and can only conjecture
about intended meanings. Still, data generation and annotation, modelling,
training and evaluation of NLP dialogue models place reliance on the
overhearing paradigm. How much of the underlying grounding processes are
thereby forfeited? As we show, there is evidence pointing to the impossibility
of properly modelling human meta-communicative acts with data-driven learning
models. In this paper, we discuss this issue and provide a preliminary
analysis on the variability of human decisions for requesting clarification.
Most importantly, we wish to bring this topic back to the community’s table,
encouraging discussion on the consequences of having models designed to only
“listen in”.

It Couldn’t Help But Overhear: On the Limits of Modelling Meta-Communicative
Grounding Acts with Supervised Learning

Brielen Madureira1 David Schlangen1,2 1Computational Linguistics, Department
of Linguistics University of Potsdam, Germany 2German Research Center for
Artificial Intelligence (DFKI), Berlin, Germany
<EMAIL_ADDRESS>

##  1 Is Grounding “Supervisable”?

“What are you looking at?” asked Bob. “Magpies are building a nest outside!”
Alice replied. If you were Bob, how would you continue that conversation? He
could for instance say “Awesome!” or “I saw that”. Whatever you say, it will
probably differ from how he continued: “Building what?”. The decision to
request clarification depends on mutual understanding, which is contingent on
e.g. the current situation, the familiarity between interlocutors and the
previous utterances. Or, more formally, it depends on the clarification
potential of these utterances (Ginzburg, 2012) and how they are assimilated
into their _common ground_ (Clark, 1996).

Figure 1: Variability of clarification requests produced by three overhearers
in comparison to the original one.

The one-to-many property of dialogue continuations is well-known in NLP (Zhao
et al., 2017; Yeh et al., 2021; Towle and Zhou, 2022). There is a
combinatorial explosion of possibilities for any interaction (Bates and Ayuso,
1991; Dingemanse and Enfield, 2023), and individual human behaviour may vary
at each point. This variability is hard to measure, since arguably no two
people will ever be in the exact same situation with the same conversation
history to react to (Yeomans et al., 2023).

Still, the prevailing end-to-end deep learning methods commonly rely on
supervised learning (SL) from a sample of human behaviour instantiating the
reaction of _a single human_ at each observed context. Besides the issue of
multiplicity of valid continuations, this paradigm faces another conceptual
contention: Dialogue models are trained to react upon a conversational history
produced by someone else. In other words, they act as overhearers111We will
use this term to also mean reading or seeing signs. of a dialogue in which
they did not participate.

The suitability of data-driven methods and fixed corpora for modelling
strategies and _conversational_ grounding phenomena like Clarification
Requests (CR) has been questioned (Schatzmann et al., 2005; Benotti and
Blackburn, 2021b). Static datasets of human observations have empirically
failed to provide enough information to define a human-like CR policy (Testoni
and Fernández, 2024; Madureira and Schlangen, 2024). Moreover, chat-optimised
LLMs mostly do not engage in grounding acts and, when they do, it does not
align with human behaviour (Kuhn et al., 2022; Deng et al., 2023; Shaikh et
al., 2023). The latter is not necessarily a problem: one can use other
effective methods when it comes to building applications. But the first is:
grounding is essential for human communication, and lack of it can lead to
undesired breakdowns (Benotti and Blackburn, 2021a).

Since modelling human dialogue strategies remains unsolved, we hereby wish to
re-open the discussion on the consequences of overhearing, focusing on two
grounding devices: backchannels and interactive repair (Fusaroli et al.,
2017).

##  2 Overhearers in a Conversation

As Clark (1996) defined it, in addition to speakers and addressees, a
conversation can have _side-participants_ , who are part of it but at a given
moment are neither of the those two, and _overhearers_ , who are spectators
without any rights or responsibilities, e.g. a silent audience or a minute-
taker who lacks the opportunity to interfere (Peters, 2010). They are further
divided into _bystanders_ , if one is aware of their presence, or
_eavesdroppers_ , who listen secretly (or at a later time). There is evidence
that the very process of understanding differs between addressees and
overhearers: while interlocutors actively construct mutual understanding with
each other, overhearers only passively consume the product of that process
(Schober and Clark, 1989).

Speakers can design their utterances while taking different attitudes towards
overhearers when they are aware of their presence (Clark, 1992; Liu et al.,
2016), but covert overhearers are not acknowledged at all in the conversation,
and can only conjecture about the intended meanings (Clark, 1992). Although
the grounding acts they witness, like backchannels, and the availability of
multiple perspectives may indeed aid their comprehension (Tolins and Fox Tree,
2016; Tree and Mayer, 2008), the original interaction was opportunistically
produced to be understood against the original participants’ common ground
(Schober and Clark, 1989).

In their corpus analysis of common ground in multi-party interactions, Eshghi
and Healey (2007) showed evidence that overhearers reach lower levels of
understanding than ratified side participants, who in their turn are not very
different from direct addressees, in what they call _collective states of
understanding_. Related to that, Georgila et al. (2020) showed that observers
and participants perceive interactions differently and the experiments by Fox
Tree (1999) provided evidence that overhearers can more easily comprehend
instructions while listening to dialogues than to monologues. Clark (1992)
even argued that most psycholinguistic subjects are actually overhearers, so
theories of language processing may actually be theories of overhearing, due
to their lack of interactivity.

##  3 Are NLP Models Only Listening In?

More than a decade ago, Rieser and Lemon (2011) already discussed the
limitations of using supervised approaches for learning dialogue strategies.
They flagged up three concerns: textual data does not contain the underlying
uncertainty measures, instances are treated as local point-wise estimates
(instead of the sequences they really are) and exploration of novel strategies
is not possible, since the model has access only to the outcomes of the chosen
dialogue trajectory originally perpetrated by the humans. This reflects the
(offline) _overhearering paradigm_ : a person or agent interpreting a pre-
existing conversation and deciding what to do if they were in the original
participants’ shoes.

In NLP, this paradigm is widely used in various modelling steps. Let us look
closer at four main practices, which may have cascaded effects.

### Data Collection

Given the extra cost of coordinating the presence of more than one subject for
generating dialogical data, especially in crowdsourcing campaigns, many
strategies have been proposed to bypass that with overhearing. To name a few
related to grounding, we have Zhou et al. (2022) who extracted dialogue
contexts from existing datasets and presented them in a two-stage approach for
some workers to generate common ground inferences and, separately, others
generated a continuation as a response. Variations of overhearing manifest in
techniques to generate CRs or their responses (Aliannejadi et al., 2021; Gao
et al., 2022; Addlesee and Eshghi, 2024) and are even embedded in data
collection tools that allow dialogues to be constructed without persistent
workers (Cascante-Bonilla et al., 2019).

### Annotation and Analysis

Corpus studies of interactive linguistic use can only be performed from an
overhearer perspective, without full evidence of what participants intended
and understood or the reasons for their decisions (Brennan, 2000). This is
particularly challenging for research on common ground. For instance,
Rodríguez and Schlangen (2004) and Schlöder and Fernández (2015) were
confronted with the limitations of overhearers having only indirect access to
the intentions of interlocutors when annotating CRs, partly remediating that
by making a long dialogue context available. Niekrasz and Moore (2010)
annotated references to conversation participants, joint actions that also
serve to build common ground, emphasising that annotators were overhearers
instructed to judge the speaker’s intended purpose. Other annotations of
grounding acts and common ground states had to rely on overhearers (Markowska
et al., 2023; Zhang et al., 2023; Mohapatra et al., 2024).

### Modelling

Prototypical data-driven models trained with supervised learning to _process_
dialogue, and possibly continue it, can, by design, be regarded as
overhearers. This fact was made clear, for instance, in the CR model by
Schlangen (2004). Traum (2017) differentiated between the perspective of an
observer in _dialogue modelling_ and of a participant in _dialogue management_
, stating that the main difference lies in the decision-making process of the
latter, although some specific applications also exist for the first.

### Evaluation

In human evaluation, overhearer experiments (Whittaker and Walker, 2005) are
very common, even though it limits the judgements to user’s _perceptions_ of
the dialogue (Moore, 2011) and restricts assessment of metrics like
effectiveness and efficiency (Paksima et al., 2009). It has historically been
a ubiquitous approach due to advantages like having control on one aspect of
the evaluation while avoiding navigational and timing aspects of real
interactions (Villalba et al., 2017), avoiding interference from ASR and other
technical problems (Buß et al., 2010), allowing the collection of feedback
about alternative system responses (Walker et al., 2004) and avoiding natural
language interpretation problems (Wärnestal et al., 2007). Demberg et al.
(2011) contrasted text overhearers with speech overhearers, pointing out that
reading dialogues is artificially simplified, since participants can go back
to difficult portions and choose the pace, and the two setups may also impact
how evaluators rate the system. The available context may also have to be
adjusted (Spanger et al., 2010). Cercas Curry and Rieser (2019) explicitly
addressed the limitations of evaluation by overhearing and advocated for
interaction with users. For a recent overview of works that use similar forms
of _static_ evaluation, see (Finch and Choi, 2020).

As we have seen, the overhearing paradigm (fairly silently) permeates
fundamental phases of dialogue modelling. The choice of this paradigm used to
be a salient concept, with authors showing awareness of its limitations when
it was employed. Kousidis and Schlangen (2015) even modelled a ratified side
participant and had evaluators “overhear the overhearer”. In recent
publications, however, it is often taken for granted, as if it was the only
natural way to go. What can be the consequences when it comes to cognitive
models of conversational grounding?

##  4 Variability in Human Grounding Acts

As humans speak, they can provide positive and negative evidence of mutual
understanding (Clark and Brennan, 1991; Roque and Traum, 2008), but modelling
their timing and decision-making is challenging. Traum (2017) claimed that “it
can be very difficult to efficiently capture regularities in behavioral
patterns that lead to similar, but not identical structures”. In connection to
that, people may take various paths in similar conversational situations
(Bates and Ayuso, 1991). It is thus an open question how far data-driven
supervised learning can get given the inherent variability of explicit (not to
mention the latent) collateral signs of grounding.

Backchannels, a positive evidence of grounding, were demonstrated to involve
individual variability, and even idiosyncrasy, possibly due to personality,
gender or randomness (Huang and Gratch, 2012; Blomsma et al., 2024). Although
those works showed some regularity in their timing, the SotA for the
backchannel prediction task is not very high (.66 weighted F1) (Liermann et
al., 2023).

Findings on the variability of human decisions to initiate a CR, a negative
sign of grounding, are still sparse. Stoyanchev et al. (2013) measured an
absolute agreement of 39% among three annotators for _scripted_ dialogues with
missing ASR information. As another reference, Shaikh et al. (2023) reported a
Cohen’s $\kappa$ of 48.45 for clarification in emotional support
conversations, which, they claimed, may even be inflated. The task of deciding
when to request clarification in collaborative instruction following is under
active investigation, but models’ performance is still suboptimal (Shi et al.,
2022; Li et al., 2023; Madureira and Schlangen, 2023; Mohanty et al., 2023).
Recent works on the multimodal CoDraw dialogue game (Kim et al., 2019) argued
that this may be due to the variability in human decisions and the limitations
of using supervised learning (Testoni and Fernández, 2024; Madureira and
Schlangen, 2024).

##  5 A Brief Analysis of Regularity in CRs

In CoDraw, an instruction follower receives instructions to reconstruct a
scene using cliparts (as in Figure 1). Only the instruction giver sees the
target scene. Madureira and Schlangen (2023) identified all CRs (around 11% of
the instruction follower’s utterances) and defined the task of deciding when
to request clarification, where models reached only up to .41 binary F1. What
is missing as evidence for the claim that data-driven models cannot fully
succeed in learning a “when policy” from human data is the actual human
performance on this NLP task, i.e. what _overhearers_ predict.

For an initial analysis, we collected a convenience sample with three
annotators performing a similar task as the trained models: given a dialogue
history and the current state of the reconstructed scene, decide which actions
to take and, if needed, request clarification (details in Appendix). We
randomly selected a sample with 90 instances; in half of them, the original
player had produced a CR.

The average binary F1 of overhearers with respect to the original decision was
.51, not much above what SotA models achieve. But the proportion of CRs widely
ranged from 36 to 85%. Among the three annotators, the Krippendorff’s $\alpha$
was $0.10$ and the mean pairwise Cohen’s $\kappa$ was $0.18$. That is already
low, but if we consider the original decision as a fourth annotator, measures
are even lower: $\alpha$ was 0.02 and $\kappa$ was 0.06. This indicates that
there is slightly more agreement among overhearers than among addresses and
overhearers, but in general there is little agreement on deciding when a CR
should be realised. In the Appendix, we show an overview of all decisions. In
terms of surface forms, the average BLEU score is $0.11$ (std$=0.10$) using
the original CR as a source and the produced utterances as a reference. The
mean cosine similarity between the embedding of the produced and the original
CRs was $0.38$, $0.29$ and $0.36$ for the three overhearers (Figure 1 shows an
example).

These are preliminary insights from a pilot study. Further standardised
experiments with a larger sample must be conducted. Still, the results are
already useful to strengthen the argument that, like backchannelling, human CR
decisions lack regularity and overhearers have a much harder task trying to
interpret and uptake someone else’s grounding acts. Decisions depend on how
interlocutors distribute grounding costs, as per the principle of least
collaborative effort (Clark and Brennan, 1991). Besides, there might be
adaptive behaviours that models are not capturing (Dideriksen et al., 2023).

##  6 Discussion

Mutual understanding is crafted by “interacting minds” (Dingemanse et al.,
2023). In dialogue, “interlocutors share or synchronise aspects of their
private mental states and act together in the world” (Brennan et al., 2010).
On the other hand, we have shown that the current NLP methodology mostly
limits us to learning how _overhearers_ predict discourse representations
without the actual joint decision making facet, due to the way that data is
produced and annotated, the assumptions behind training mechanisms and the
evaluation protocols, each adding a layer of overhearing.

What can be a better setup to learn human dialogue behaviour, realising it as
a truly interactive process? One needs to move on from one-off supervised
learning to multi-step models that not only _understand_ dialogues but also
sequentially _participate_ in them. Reinforcement learning provides that
framing with a fully accessible and explorable environment (Rieser and Lemon,
2011), but somewhat circularly requires a good simulation of an user or
interlocutor (Schatzmann et al., 2005; Georgila et al., 2006; Li et al.,
2020). Another possibility are hybrid combinations of supervised and
reinforcement learning (Henderson et al., 2008). But independently of the
learning regime, data-driven approaches, which rely on extracting latent
patterns and regularities in a corpus, stumble upon the individual variability
of some dialogue phenomena, so that tasks may be ill-defined in datasets.

Indeed, interfaces do not necessarily have to conform to human behaviour, as
long as they can sustain graceful interaction (Hayes, 1980). But from a
cognitive perspective, the current NLP resources do not seem to satisfactorily
meet our needs for modelling grounding mechanisms. To study the human mind, do
we want cognitive models of how meaning and common ground are constructed or
only of how they can be reverse engineered from someone else’s interactions?

### To conclude

With this argumentative paper, we wish to encourage more studies on the
variability of human grounding acts and its impact in modelling human dialogue
strategies. Besides, we advocate making the overhearing paradigm explicit
whenever it is used in future publications and discussing how it can have
influenced reported findings.

## References

  * Addlesee and Eshghi (2024) Angus Addlesee and Arash Eshghi. 2024.  You have interrupted me again!: making voice assistants more dementia-friendly with incremental clarification.  _Frontiers in Dementia_ , 3:1343052.
  * Aliannejadi et al. (2021) Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2021.  Building and evaluating open-domain dialogue corpora with clarifying questions.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4473–4484, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Bates and Ayuso (1991) Madeleine Bates and Damaris Ayuso. 1991.  A proposal for incremental dialogue evaluation.  In _Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991_.
  * Benotti and Blackburn (2021a) Luciana Benotti and Patrick Blackburn. 2021a.  Grounding as a collaborative process.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 515–531, Online. Association for Computational Linguistics.
  * Benotti and Blackburn (2021b) Luciana Benotti and Patrick Blackburn. 2021b.  A recipe for annotating grounded clarifications.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4065–4077, Online. Association for Computational Linguistics.
  * Blomsma et al. (2024) Peter Blomsma, Julija Vaitonyté, Gabriel Skantze, and Marc Swerts. 2024.  Backchannel behavior is idiosyncratic.  _Language and Cognition_ , page 1–24.
  * Brennan (2000) Susan E. Brennan. 2000.  Invited talk: Processes that shape conversation and their implications for computational linguistics.  In _Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics_ , pages 1–11, Hong Kong. Association for Computational Linguistics.
  * Brennan et al. (2010) Susan E Brennan, Alexia Galati, and Anna K Kuhlen. 2010.  Two minds, one dialog: Coordinating speaking and understanding.  In _Psychology of learning and motivation_ , volume 53, pages 301–344. Elsevier.
  * Buß et al. (2010) Okko Buß, Timo Baumann, and David Schlangen. 2010.  Collaborating on utterances with a spoken dialogue system using an ISU-based approach to incremental dialogue management.  In _Proceedings of the SIGDIAL 2010 Conference_ , pages 233–236, Tokyo, Japan. Association for Computational Linguistics.
  * Cascante-Bonilla et al. (2019) Paola Cascante-Bonilla, Xuwang Yin, Vicente Ordonez, and Song Feng. 2019.  Chat-crowd: A dialog-based platform for visual layout composition.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 138–142, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Cercas Curry and Rieser (2019) Amanda Cercas Curry and Verena Rieser. 2019.  A crowd-based evaluation of abuse response strategies in conversational agents.  In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 361–366, Stockholm, Sweden. Association for Computational Linguistics.
  * Clark (1992) Herbert H Clark. 1992.  _Arenas of language use_.  University of Chicago Press.
  * Clark (1996) Herbert H. Clark. 1996.  _Common ground_, page 92–122.  “Using” Linguistic Books. Cambridge University Press.
  * Clark and Brennan (1991) Herbert H Clark and Susan E Brennan. 1991.  Grounding in communication.  In _Perspectives on socially shared cognition._ , pages 127–149. American Psychological Association.
  * Demberg et al. (2011) Vera Demberg, Andi Winterboer, and Johanna D. Moore. 2011.  A strategy for information presentation in spoken dialog systems.  _Computational Linguistics_ , 37(3):489–539.
  * Deng et al. (2023) Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua. 2023.  Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10602–10621, Singapore. Association for Computational Linguistics.
  * Dideriksen et al. (2023) Christina Dideriksen, Morten H Christiansen, Kristian Tylén, Mark Dingemanse, and Riccardo Fusaroli. 2023.  Quantifying the interplay of conversational devices in building mutual understanding. _Journal of Experimental Psychology: General_ , 152(3):864.
  * Dingemanse and Enfield (2023) Mark Dingemanse and NJ Enfield. 2023.  Interactive repair and the foundations of language.  _Trends in Cognitive Sciences_.
  * Dingemanse et al. (2023) Mark Dingemanse, Andreas Liesenfeld, Marlou Rasenberg, Saul Albert, Felix K Ameka, Abeba Birhane, Dimitris Bolis, Justine Cassell, Rebecca Clift, Elena Cuffari, et al. 2023.  Beyond single-mindedness: A figure-ground reversal for the cognitive sciences.  _Cognitive science_ , 47(1):e13230.
  * Eshghi and Healey (2007) Arash Eshghi and Patrick G.T. Healey. 2007.  Collective states of understanding.  In _Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue_ , pages 2–9, Antwerp, Belgium. Association for Computational Linguistics.
  * Finch and Choi (2020) Sarah E. Finch and Jinho D. Choi. 2020.  Towards unified dialogue system evaluation: A comprehensive analysis of current evaluation protocols.  In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 236–245, 1st virtual meeting. Association for Computational Linguistics.
  * Fox Tree (1999) Jean E Fox Tree. 1999.  Listening in on monologues and dialogues.  _Discourse processes_ , 27(1):35–53.
  * Fusaroli et al. (2017) Riccardo Fusaroli, Kristian Tylén, Katrine Garly, Jakob Steensig, Morten H Christiansen, and Mark Dingemanse. 2017.  Measures and mechanisms of common ground: Backchannels, conversational repair, and interactive alignment in free and task-oriented social interactions.  In _the 39th Annual Conference of the Cognitive Science Society (CogSci 2017)_ , pages 2055–2060. Cognitive Science Society.
  * Gao et al. (2022) Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme. 2022.  Dialfred: Dialogue-enabled agents for embodied instruction following.  _IEEE Robotics and Automation Letters_ , 7(4):10049–10056.
  * Georgila et al. (2020) Kallirroi Georgila, Carla Gordon, Volodymyr Yanov, and David Traum. 2020.  Predicting ratings of real dialogue participants from artificial data and ratings of human dialogue observers.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 726–734, Marseille, France. European Language Resources Association.
  * Georgila et al. (2006) Kallirroi Georgila, James Henderson, and Oliver Lemon. 2006.  User simulation for spoken dialogue systems: learning and evaluation.  In _Ninth International Conference on Spoken Language Processing_.
  * Ginzburg (2012) Jonathan Ginzburg. 2012.  _The interactive stance: Meaning for conversation_.  Oxford University Press.
  * Hayes (1980) Phil Hayes. 1980.  Expanding the horizons of natural language interfaces.  In _18th Annual Meeting of the Association for Computational Linguistics_ , pages 71–74, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Henderson et al. (2008) James Henderson, Oliver Lemon, and Kallirroi Georgila. 2008.  Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets.  _Computational Linguistics_ , 34(4):487–511.
  * Huang and Gratch (2012) Lixing Huang and Jonathan Gratch. 2012.  Crowdsourcing backchannel feedback: understanding the individual variability from the crowds.  In _Feedback behaviors in dialog_.
  * Kim et al. (2019) Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019.  CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6495–6513, Florence, Italy. Association for Computational Linguistics.
  * Kousidis and Schlangen (2015) Spyridon Kousidis and David Schlangen. 2015.  The power of a glance: Evaluating embodiment and turn-tracking strategies of an active robotic overhearer.  In _Proceedings of AAAI Spring Symposium on Turn-taking and Coordination in Human-Machine Interaction_.
  * Kuhn et al. (2022) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022.  Clam: Selective clarification for ambiguous questions with generative language models.  _arXiv preprint arXiv:2212.07769_.
  * Li et al. (2023) Haau-Sing (Xiaocheng) Li, Mohsen Mesgar, André Martins, and Iryna Gurevych. 2023.  Python code generation by asking clarification questions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14287–14306, Toronto, Canada. Association for Computational Linguistics.
  * Li et al. (2020) Ziming Li, Julia Kiseleva, and Maarten de Rijke. 2020.  Rethinking supervised learning and reinforcement learning in task-oriented dialogue systems.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3537–3546, Online. Association for Computational Linguistics.
  * Liermann et al. (2023) Wencke Liermann, Yo-Han Park, Yong-Seok Choi, and Kong Lee. 2023.  Dialogue act-aided backchannel prediction using multi-task learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 15073–15079, Singapore. Association for Computational Linguistics.
  * Liu et al. (2016) Kris Liu, Jean Fox Tree, and Marilyn Walker. 2016.  Coordinating communication in the wild: The artwalk dialogue corpus of pedestrian navigation and mobile referential communication.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 3159–3166, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Madureira and Schlangen (2023) Brielen Madureira and David Schlangen. 2023.  Instruction clarification requests in multimodal collaborative dialogue games: Tasks, and an analysis of the CoDraw dataset.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2303–2319, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Madureira and Schlangen (2024) Brielen Madureira and David Schlangen. 2024.  Taking action towards graceful interaction: The effects of performing actions on modelling policies for instruction clarification requests.  In _Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language_ , pages 1–21, Malta. Association for Computational Linguistics.
  * Markowska et al. (2023) Magdalena Markowska, Mohammad Taghizadeh, Adil Soubki, Seyed Mirroshandel, and Owen Rambow. 2023.  Finding common ground: Annotating and predicting common ground in spoken conversations.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 8221–8233, Singapore. Association for Computational Linguistics.
  * Mohanty et al. (2023) Shrestha Mohanty, Negar Arabzadeh, Julia Kiseleva, Artem Zholus, Milagro Teruel, Ahmed Awadallah, Yuxuan Sun, Kavya Srinet, and Arthur Szlam. 2023.  Transforming human-centered ai collaboration: Redefining embodied agents capabilities through interactive grounded language instructions.  _arXiv preprint arXiv:2305.10783_.
  * Mohapatra et al. (2024) Biswesh Mohapatra, Seemab Hassan, Laurent Romary, and Justine Cassell. 2024.  Conversational grounding: Annotation and analysis of grounding acts and grounding units.  In _Proceedings of LREC-COLING 2024_.
  * Moore (2011) Johanna D. Moore. 2011.  Language generation for spoken dialogue systems [invited talk].  In _Proceedings of the 13th European Workshop on Natural Language Generation_ , page 132, Nancy, France. Association for Computational Linguistics.
  * Niekrasz and Moore (2010) John Niekrasz and Johanna D. Moore. 2010.  Annotating participant reference in English spoken conversation.  In _Proceedings of the Fourth Linguistic Annotation Workshop_ , pages 256–264, Uppsala, Sweden. Association for Computational Linguistics.
  * Paksima et al. (2009) Taghi Paksima, Kallirroi Georgila, and Johanna Moore. 2009.  Evaluating the effectiveness of information presentation in a full end-to-end dialogue system.  In _Proceedings of the SIGDIAL 2009 Conference_ , pages 1–10, London, UK. Association for Computational Linguistics.
  * Peters (2010) Stanley Peters. 2010.  Listening in.  In _Proceedings of the 24th Pacific Asia Conference on Language, Information and Computation_ , pages 31–31, Tohoku University, Sendai, Japan. Institute of Digital Enhancement of Cognitive Processing, Waseda University.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Rieser and Lemon (2011) Verena Rieser and Oliver Lemon. 2011.  _Reinforcement learning for adaptive dialogue systems: a data-driven methodology for dialogue management and natural language generation_.  Springer Science & Business Media.
  * Rodríguez and Schlangen (2004) Kepa Joseba Rodríguez and David Schlangen. 2004.  Form, intonation and function of clarification requests in german task-oriented spoken dialogues.  In _Proceedings of Catalog (the 8th workshop on the semantics and pragmatics of dialogue; SemDial04)_.
  * Roque and Traum (2008) Antonio Roque and David Traum. 2008.  Degrees of grounding based on evidence of understanding.  In _Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue_ , pages 54–63, Columbus, Ohio. Association for Computational Linguistics.
  * Schatzmann et al. (2005) Jost Schatzmann, Kallirroi Georgila, and Steve Young. 2005.  Quantitative evaluation of user simulation techniques for spoken dialogue systems.  In _Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue_ , pages 45–54, Lisbon, Portugal. Special Interest Group on Discourse and Dialogue (SIGdial).
  * Schlangen (2004) David Schlangen. 2004.  Causes and strategies for requesting clarification in dialogue.  In _Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004_ , pages 136–143, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
  * Schlöder and Fernández (2015) Julian J. Schlöder and Raquel Fernández. 2015.  Clarifying intentions in dialogue: A corpus study.  In _Proceedings of the 11th International Conference on Computational Semantics_ , pages 46–51, London, UK. Association for Computational Linguistics.
  * Schober and Clark (1989) Michael F Schober and Herbert H Clark. 1989.  Understanding by addressees and overhearers.  _Cognitive psychology_ , 21(2):211–232.
  * Shaikh et al. (2023) Omar Shaikh, Kristina Gligorić, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. 2023.  Grounding or guesswork? large language models are presumptive grounders.  _arXiv preprint arXiv:2311.09144_.
  * Shi et al. (2022) Zhengxiang Shi, Yue Feng, and Aldo Lipani. 2022.  Learning to execute actions or ask clarification questions.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2060–2070, Seattle, United States. Association for Computational Linguistics.
  * Spanger et al. (2010) Philipp Spanger, Ryu Iida, Takenobu Tokunaga, Asuka Terai, and Naoko Kuriyama. 2010.  Towards an extrinsic evaluation of referring expressions in situated dialogs.  In _Proceedings of the 6th International Natural Language Generation Conference_. Association for Computational Linguistics.
  * Stoyanchev et al. (2013) Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. 2013.  Modelling human clarification strategies.  In _Proceedings of the SIGDIAL 2013 Conference_ , pages 137–141, Metz, France. Association for Computational Linguistics.
  * Testoni and Fernández (2024) Alberto Testoni and Raquel Fernández. 2024.  Asking the right question at the right time: Human and model uncertainty guidance to ask clarification questions.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 258–275, St. Julian’s, Malta. Association for Computational Linguistics.
  * Tolins and Fox Tree (2016) Jackson Tolins and Jean E Fox Tree. 2016.  Overhearers use addressee backchannels in dialog comprehension.  _Cognitive science_ , 40(6):1412–1434.
  * Towle and Zhou (2022) Benjamin Towle and Ke Zhou. 2022.  Learn what is possible, then choose what is best: Disentangling one-to-many relations in language through text-based games.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4955–4965, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Traum (2017) David Traum. 2017.  Computational approaches to dialogue.  _The Routledge Handbook of Language and Dialogue_ , 1:143–161.
  * Tree and Mayer (2008) Jean E Fox Tree and Sarah A Mayer. 2008.  Overhearing single and multiple perspectives.  _Discourse Processes_ , 45(2):160–179.
  * Villalba et al. (2017) Martín Villalba, Christoph Teichmann, and Alexander Koller. 2017.  Generating contrastive referring expressions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 678–687, Vancouver, Canada. Association for Computational Linguistics.
  * Walker et al. (2004) Marilyn A Walker, Stephen J Whittaker, Amanda Stent, Preetam Maloor, Johanna Moore, Michael Johnston, and Gunaranjan Vasireddy. 2004.  Generation and evaluation of user tailored responses in multimodal dialogue.  _Cognitive Science_ , 28(5):811–840.
  * Wärnestal et al. (2007) Pontus Wärnestal, Lars Degerstedt, and Arne Jönsson. 2007.  Emergent conversational recommendations: A dialogue behavior approach.  In _Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue_ , pages 63–66, Antwerp, Belgium. Association for Computational Linguistics.
  * Whittaker and Walker (2005) Steve Whittaker and Marilyn Walker. 2005.  Evaluating dialogue strategies in multimodal dialogue systems.  _Spoken Multimodal Human-Computer Dialogue in Mobile Environments_ , pages 247–268.
  * Yeh et al. (2021) Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021.  A comprehensive assessment of dialog evaluation metrics.  In _The First Workshop on Evaluations and Assessments of Neural Conversation Systems_ , pages 15–33, Online. Association for Computational Linguistics.
  * Yeomans et al. (2023) Michael Yeomans, F Katelynn Boland, Hanne K Collins, Nicole Abi-Esber, and Alison Wood Brooks. 2023.  A practical guide to conversation research: How to study what people say to each other.  _Advances in Methods and Practices in Psychological Science_ , 6(4):25152459231183919.
  * Zhang et al. (2023) Xuanming Zhang, Rahul Divekar, Rutuja Ubale, and Zhou Yu. 2023.  GrounDialog: A dataset for repair and grounding in task-oriented spoken dialogues for language learning.  In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_ , pages 300–314, Toronto, Canada. Association for Computational Linguistics.
  * Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017.  Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 654–664, Vancouver, Canada. Association for Computational Linguistics.
  * Zhou et al. (2022) Pei Zhou, Hyundong Cho, Pegah Jandaghi, Dong-Ho Lee, Bill Yuchen Lin, Jay Pujara, and Xiang Ren. 2022.  Reflect, not reflex: Inference-based common ground improves dialogue response quality.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10450–10468, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

##  Appendix A Appendix

(see next page)

Figure 2: Variability in the decision of when to request clarification,
comparing the original player with three overhearer annotators over 90
instances in the horizontal axis. Each cell is a data instance and columns
correspond to decisions on the same instance.

### Annotation Task

The decisions from the overhearer perspective were performed by 3 annotators.
Two of them are student assistants employed in our lab and one is a volunteer
acquainted with first author. A simple GUI interface showed the dialogue
history (from 1 to 3 turns) up to the last instruction giver instruction, the
current state of the reconstructed scene and the gallery of available
cliparts. They could select up to 4 high level, discrete actions (add, move,
resize, flip, delete) and the corresponding cliparts from dropdown lists.
Besides, they could type a clarification request to continue the dialogue if
they wished (otherwise, the next utterance field should be left blank). In
future studies, a full interface similar to the original game should be used,
i.e. giving the opportunity for cliparts to be moved around and edited in the
scene. Here, the selection of actions was just used to enforce that the
overhearers reflected on the pertinent actions while deciding whether to
request clarification. Note that the step of action taking makes annotators
more privileged than plain overhearers that just process the dialogue, but it
better approximates the decision of the iCR-Action-Taker models in Madureira
and Schlangen (2024). In this case, they are overhearers of the dialogue
context, but try to minimally act as a player doing the next step. The results
work as an upper bound for plain overhearers.

### Additional Details

The inter-annotator agreement metrics were computed with nltk using
chencherry.method3 for smoothing. The sentence embeddings for the CR
utterances were computed with model sentence-transformers/all-MiniLM-L6-v2
from SentenceTransformers (Reimers and Gurevych, 2019). Figure 2 presents the
main binary decision (whether to request clarification or not) for each of the
90 annotation instances.


* [79] LHCb collaboration, R. Aaij et al., _Measurements of the $\Lambda_{b}^{0}$, $\Xi_{b}^{-}$, and $\Omega_{b}^{-}$ baryon masses_, Phys. Rev. Lett. 110 (2013) 182001, arXiv:1302.1072
  * [80] LHCb collaboration, R. Aaij et al., _Precision measurement of $D$ meson mass differences_, JHEP 06 (2013) 065, arXiv:1304.6865
  * [81] LHCb collaboration, R. Aaij et al., _Study of beauty baryon decays to $D^{0}ph^{-}$ and $\Lambda_{c}^{+}h^{-}$ final states_, Phys. Rev. D89 (2014) 032001, arXiv:1311.4823
  * [82] LHCb collaboration, R. Aaij et al., _Observation of $B^{0}\rightarrow\bar{D}^{0}K^{+}K^{-}$ and evidence for $B^{0}_{s}\rightarrow\bar{D}^{0}K^{+}K^{-}$_, Phys. Rev. Lett. 109 (2012) 131801, arXiv:1207.5991
  * [83] LHCb collaboration, R. Aaij et al., _Observation of the decay $B^{0}_{s}\rightarrow\bar{D}^{0}\phi$_, Phys. Lett. B727 (2013) 403, arXiv:1308.4583
  * [84] T. Skwarnicki, A study of the radiative cascade transitions between the Upsilon-prime and Upsilon resonances, PhD thesis, Institute of Nuclear Physics, Krakow, 1986, DESY-F31-86-02
  * [85] G. N. Fleming, _Recoupling effects in the isobar model. 1. General formalism for three-pion scattering_ , Phys. Rev. 135 (1964) B551
  * [86] D. Morgan, _Phenomenological analysis of $I=\frac{1}{2}$ single-pion production processes in the energy range 500 to 700 MeV_, Phys. Rev. 166 (1968) 1731
  * [87] D. Herndon, P. Soding, and R. J. Cashmore, _A generalised isobar model formalism_ , Phys. Rev. D11 (1975) 3165
  * [88] BaBar collaboration, B. Aubert et al., _Dalitz-plot analysis of the decays $B^{\pm}\rightarrow K^{\pm}\pi^{\mp}\pi^{\pm}$_, Phys. Rev. D72 (2005) 072003, Erratum ibid. D74 (2006) 099903, arXiv:hep-ex/0507004
  * [89] C. Zemach, _Three pion decays of unstable particles_ , Phys. Rev. 133 (1964) B1201
  * [90] C. Zemach, _Use of angular-momentum tensors_ , Phys. Rev. 140 (1965) B97
  * [91] Belle collaboration, A. Garmash et al., _Dalitz analysis of the three-body charmless decays ${{{B}^{+}}}\rightarrow K^{+}{{\pi}^{+}}{{\pi}^{-}}$ and ${{{B}^{+}}}\rightarrow K^{+}K^{+}K^{-}$_, Phys. Rev. D71 (2005) 092003, arXiv:hep-ex/0412066
  * [92] B. Meadows, _Low mass S-wave $K\pi$ and $\pi\pi$ systems_, eConf C070805 (2007) 27, arXiv:0712.1605
  * [93] LASS collaboration, D. Aston et al., _A study of $K^{-}\pi^{+}$ scattering in the reaction $K^{-}p\rightarrow K^{-}\pi^{+}n$ at $11\,Ge\kern-1.00006ptV\\!/c$_, Nucl. Phys. B296 (1988) 493
  * [94] BaBar collaboration, B. Aubert et al., _Ambiguity-free measurement of $\cos(2\beta)$: time-integrated and time-dependent angular analyses of $B\rightarrow{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}}K\pi$_, Phys. Rev. D71 (2005) 032005, arXiv:hep-ex/0411016
  * [95] Laura++ Dalitz plot fitting package, http://laura.hepforge.org/
  * [96] M. Williams, _How good are your fits? Unbinned multivariate goodness-of-fit tests in high energy physics_ , JINST 5 (2010) P09004, arXiv:1006.3019
  * [97] S. M. Flatté, _Coupled-channel analysis of the $\pi\eta$ and $K{\kern 1.99997pt\overline{\kern-1.99997ptK}{}}$ systems near $K{\kern 1.99997pt\overline{\kern-1.99997ptK}{}}$ threshold_, Phys. Lett. B63 (1976) 224
  * [98] D. V. Bugg, _Comments on the $\sigma$ and $\kappa$_, Phys. Lett. B572 (2003) 1, Erratum ibid. B595 (2004) 556
  * [99] B. El-Bennich et al., _$CP$ violation and kaon-pion interactions in $B\rightarrow K\pi^{+}\pi^{-}$ decays_, Phys. Rev. D79 (2009) 094005, Erratum ibid. D83 (2011) 039903, arXiv:0902.3645
  * [100] FOCUS collaboration, J. M. Link et al., _Dalitz plot analysis of the $D^{+}\rightarrow K^{-}\pi^{+}\pi^{+}$ decay in the FOCUS experiment_, Phys. Lett. B653 (2007) 1, arXiv:0705.2248
  * [101] G. J. Gounaris and J. J. Sakurai, _Finite-width corrections to the vector-meson-dominance prediction for $\rho\rightarrow{e^{+}e^{-}}$_, Phys. Rev. Lett. 21 (1968) 244

##  Appendix A Results for interference fit fractions

The central values of the interference fit fractions are given in Table 17.
The statistical, experimental systematic and model uncertainties on these
quantities are given in Tables 18, 19 and 20, respectively.

Table 17: Interference fit fractions (%) from the nominal Dalitz plot fit. The
amplitudes are: ($A_{0}$) ${\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(892)^{0}$, ($A_{1}$) ${\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(1410)^{0}$, ($A_{2}$) ${{\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{0}}(1430)^{0}$, ($A_{3}$) LASS
nonresonant, ($A_{4}$) ${{\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{2}}(1430)^{0}$, ($A_{5}$)
${\kern 1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(1680)^{0}$, ($A_{6}$)
${{\kern 1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{0}}(1950)^{0}$,
($A_{7}$) $D^{*-}_{s\,v}$, ($A_{8}$) $D^{*}_{s0\,v}(2317)^{-}$, ($A_{9}$)
$D^{*}_{s2}(2573)^{-}$, ($A_{10}$) $D^{*}_{s1}(2700)^{-}$, ($A_{11}$)
$D^{*}_{s3}(2860)^{-}$, ($A_{12}$) $D^{*}_{s1}(2860)^{-}$, ($A_{13}$)
$B^{*+}_{v}$, ($A_{14}$) Nonresonant. The diagonal elements correspond to the
fit fractions shown in Table 5.

| $A_{0}$ | $A_{1}$ | $A_{2}$ | $A_{3}$ | $A_{4}$ | $A_{5}$ | $A_{6}$ | $A_{7}$ | $A_{8}$ | $A_{9}$ | $A_{10}$ | $A_{11}$ | $A_{12}$ | $A_{13}$ | $A_{14}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$A_{0}$ | $\phantom{-}28.6$ | $\phantom{-}2.2$ | $-0.0$ | $-0.0$ | $-0.0$ | $-0.6$ | $\phantom{-}0.0$ | $-0.4$ | $-0.3$ | $\phantom{-}0.5$ | $-0.3$ | $\phantom{-}0.3$ | $-1.2$ | $-0.8$ | $-3.5$
$A_{1}$ |  | $\phantom{-}1.7$ | $\phantom{-}0.0$ | $\phantom{-}0.0$ | $-0.0$ | $-0.6$ | $\phantom{-}0.0$ | $-0.0$ | $\phantom{-}0.2$ | $\phantom{-}0.3$ | $-0.3$ | $\phantom{-}0.1$ | $-0.6$ | $-0.5$ | $-2.2$
$A_{2}$ |  |  | $\phantom{-}20.0$ | $-12.3$ | $-0.0$ | $-0.0$ | $\phantom{-}2.1$ | $-2.9$ | $-2.2$ | $-1.4$ | $\phantom{-}0.7$ | $-0.4$ | $\phantom{-}0.6$ | $-3.2$ | $\phantom{-}0.0$
$A_{3}$ |  |  |  | $\phantom{-}13.7$ | $\phantom{-}0.0$ | $-0.0$ | $-1.5$ | $\phantom{-}6.1$ | $\phantom{-}1.8$ | $\phantom{-}2.1$ | $-1.5$ | $\phantom{-}0.0$ | $-0.5$ | $\phantom{-}2.5$ | $-2.5$
$A_{4}$ |  |  |  |  | $\phantom{-}3.7$ | $-0.0$ | $-0.0$ | $-0.6$ | $-0.6$ | $-0.3$ | $-0.1$ | $-0.1$ | $-0.5$ | $-0.1$ | $-0.3$
$A_{5}$ |  |  |  |  |  | $\phantom{-}0.5$ | $-0.0$ | $-0.9$ | $-0.5$ | $-0.5$ | $\phantom{-}0.1$ | $-0.1$ | $-0.2$ | $-0.2$ | $-0.3$
$A_{6}$ |  |  |  |  |  |  | $\phantom{-}0.3$ | $-0.6$ | $-0.2$ | $-0.3$ | $\phantom{-}0.1$ | $-0.1$ | $-0.1$ | $-0.1$ | $-0.7$
$A_{7}$ |  |  |  |  |  |  |  | $\phantom{-}4.7$ | $-0.0$ | $-0.0$ | $-0.6$ | $-0.0$ | $\phantom{-}3.0$ | $\phantom{-}0.3$ | $\phantom{-}0.0$
$A_{8}$ |  |  |  |  |  |  |  |  | $\phantom{-}2.3$ | $\phantom{-}0.0$ | $-0.0$ | $\phantom{-}0.0$ | $-0.0$ | $\phantom{-}0.1$ | $-0.7$
$A_{9}$ |  |  |  |  |  |  |  |  |  | $\phantom{-}25.7$ | $-0.0$ | $\phantom{-}0.0$ | $-0.0$ | $\phantom{-}0.4$ | $\phantom{-}0.0$
$A_{10}$ |  |  |  |  |  |  |  |  |  |  | $\phantom{-}1.6$ | $-0.0$ | $-0.9$ | $-0.6$ | $\phantom{-}0.0$
$A_{11}$ |  |  |  |  |  |  |  |  |  |  |  | $\phantom{-}2.2$ | $-0.0$ | $\phantom{-}0.0$ | $-0.0$
$A_{12}$ |  |  |  |  |  |  |  |  |  |  |  |  | $\phantom{-}5.0$ | $-1.6$ | $\phantom{-}0.0$
$A_{13}$ |  |  |  |  |  |  |  |  |  |  |  |  |  | $\phantom{-}1.9$ | $\phantom{-}3.7$
$A_{14}$ |  |  |  |  |  |  |  |  |  |  |  |  |  |  | $\phantom{-}12.4$

Table 18: Absolute statistical uncertainties on the interference fit fractions
(%) from the Dalitz plot fit. The amplitudes are: ($A_{0}$) ${\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(892)^{0}$, ($A_{1}$) ${\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(1410)^{0}$, ($A_{2}$) ${{\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{0}}(1430)^{0}$, ($A_{3}$) LASS
nonresonant, ($A_{4}$) ${{\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{2}}(1430)^{0}$, ($A_{5}$)
${\kern 1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(1680)^{0}$, ($A_{6}$)
${{\kern 1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{0}}(1950)^{0}$,
($A_{7}$) $D^{*-}_{s\,v}$, ($A_{8}$) $D^{*}_{s0\,v}(2317)^{-}$, ($A_{9}$)
$D^{*}_{s2}(2573)^{-}$, ($A_{10}$) $D^{*}_{s1}(2700)^{-}$, ($A_{11}$)
$D^{*}_{s3}(2860)^{-}$, ($A_{12}$) $D^{*}_{s1}(2860)^{-}$, ($A_{13}$)
$B^{*+}_{v}$, ($A_{14}$) Nonresonant. The diagonal elements correspond to the
statistical uncertainties on the fit fractions shown in Table 5.

| $A_{0}$ | $A_{1}$ | $A_{2}$ | $A_{3}$ | $A_{4}$ | $A_{5}$ | $A_{6}$ | $A_{7}$ | $A_{8}$ | $A_{9}$ | $A_{10}$ | $A_{11}$ | $A_{12}$ | $A_{13}$ | $A_{14}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$A_{0}$ | 0.6 | 0.4 | 0.0 | 0.0 | 0.0 | 0.3 | 0.0 | 0.5 | 0.3 | 0.2 | 0.1 | 0.1 | 0.1 | 0.3 | 0.4
$A_{1}$ |  | 0.5 | 0.0 | 0.0 | 0.0 | 0.4 | 0.0 | 0.4 | 0.2 | 0.1 | 0.1 | 0.0 | 0.1 | 0.2 | 0.4
$A_{2}$ |  |  | 1.6 | 1.2 | 0.0 | 0.0 | 0.6 | 0.5 | 0.6 | 0.2 | 0.1 | 0.1 | 0.2 | 1.3 | 0.8
$A_{3}$ |  |  |  | 2.5 | 0.0 | 0.0 | 0.6 | 1.0 | 0.6 | 0.3 | 0.3 | 0.0 | 0.5 | 1.5 | 0.7
$A_{4}$ |  |  |  |  | 0.6 | 0.0 | 0.0 | 0.1 | 0.1 | 0.1 | 0.1 | 0.0 | 0.1 | 0.1 | 0.3
$A_{5}$ |  |  |  |  |  | 0.4 | 0.0 | 0.3 | 0.1 | 0.2 | 0.1 | 0.1 | 0.2 | 0.2 | 0.3
$A_{6}$ |  |  |  |  |  |  | 0.2 | 0.3 | 0.1 | 0.1 | 0.0 | 0.0 | 0.1 | 0.2 | 0.3
$A_{7}$ |  |  |  |  |  |  |  | 1.4 | 0.0 | 0.0 | 0.6 | 0.0 | 0.8 | 1.5 | 0.0
$A_{8}$ |  |  |  |  |  |  |  |  | 1.1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.5 | 2.9
$A_{9}$ |  |  |  |  |  |  |  |  |  | 0.6 | 0.0 | 0.0 | 0.0 | 0.2 | 0.0
$A_{10}$ |  |  |  |  |  |  |  |  |  |  | 0.4 | 0.0 | 0.5 | 0.3 | 0.0
$A_{11}$ |  |  |  |  |  |  |  |  |  |  |  | 0.1 | 0.0 | 0.0 | 0.0
$A_{12}$ |  |  |  |  |  |  |  |  |  |  |  |  | 1.2 | 0.6 | 0.0
$A_{13}$ |  |  |  |  |  |  |  |  |  |  |  |  |  | 1.2 | 1.0
$A_{14}$ |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 2.7

Table 19: Absolute experimental systematic uncertainties on the interference
fit fractions (%). The amplitudes are: ($A_{0}$) ${\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(892)^{0}$, ($A_{1}$) ${\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(1410)^{0}$, ($A_{2}$) ${{\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{0}}(1430)^{0}$, ($A_{3}$) LASS
nonresonant, ($A_{4}$) ${{\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{2}}(1430)^{0}$, ($A_{5}$)
${\kern 1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(1680)^{0}$, ($A_{6}$)
${{\kern 1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{0}}(1950)^{0}$,
($A_{7}$) $D^{*-}_{s\,v}$, ($A_{8}$) $D^{*}_{s0\,v}(2317)^{-}$, ($A_{9}$)
$D^{*}_{s2}(2573)^{-}$, ($A_{10}$) $D^{*}_{s1}(2700)^{-}$, ($A_{11}$)
$D^{*}_{s3}(2860)^{-}$, ($A_{12}$) $D^{*}_{s1}(2860)^{-}$, ($A_{13}$)
$B^{*+}_{v}$, ($A_{14}$) Nonresonant. The diagonal elements correspond to the
experimental systematic uncertainties on the fit fractions shown in Table 8.

| $A_{0}$ | $A_{1}$ | $A_{2}$ | $A_{3}$ | $A_{4}$ | $A_{5}$ | $A_{6}$ | $A_{7}$ | $A_{8}$ | $A_{9}$ | $A_{10}$ | $A_{11}$ | $A_{12}$ | $A_{13}$ | $A_{14}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$A_{0}$ | 0.74 | 0.60 | 0.00 | 0.00 | 0.00 | 0.12 | 0.00 | 0.61 | 0.40 | 0.21 | 0.08 | 0.04 | 0.08 | 0.20 | 0.67
$A_{1}$ |  | 0.16 | 0.00 | 0.00 | 0.00 | 0.34 | 0.00 | 0.61 | 0.54 | 0.30 | 0.11 | 0.08 | 0.07 | 0.22 | 0.39
$A_{2}$ |  |  | 0.72 | 0.65 | 0.00 | 0.00 | 0.45 | 0.67 | 1.61 | 0.30 | 0.11 | 0.07 | 0.16 | 0.72 | 0.74
$A_{3}$ |  |  |  | 1.52 | 0.00 | 0.00 | 0.37 | 0.62 | 1.57 | 0.26 | 0.19 | 0.06 | 0.43 | 0.78 | 0.70
$A_{4}$ |  |  |  |  | 0.39 | 0.00 | 0.00 | 0.08 | 0.19 | 0.14 | 0.03 | 0.05 | 0.06 | 0.13 | 0.17
$A_{5}$ |  |  |  |  |  | 0.26 | 0.00 | 0.23 | 0.33 | 0.12 | 0.05 | 0.03 | 0.17 | 0.11 | 0.30
$A_{6}$ |  |  |  |  |  |  | 0.13 | 0.18 | 0.08 | 0.06 | 0.02 | 0.02 | 0.08 | 0.15 | 0.25
$A_{7}$ |  |  |  |  |  |  |  | 1.09 | 0.00 | 0.00 | 0.52 | 0.00 | 0.86 | 1.25 | 0.00
$A_{8}$ |  |  |  |  |  |  |  |  | 1.94 | 0.00 | 0.00 | 0.00 | 0.00 | 1.97 | 4.63
$A_{9}$ |  |  |  |  |  |  |  |  |  | 0.78 | 0.00 | 0.00 | 0.00 | 0.17 | 0.00
$A_{10}$ |  |  |  |  |  |  |  |  |  |  | 0.44 | 0.00 | 0.54 | 0.28 | 0.00
$A_{11}$ |  |  |  |  |  |  |  |  |  |  |  | 0.28 | 0.00 | 0.06 | 0.00
$A_{12}$ |  |  |  |  |  |  |  |  |  |  |  |  | 0.65 | 0.63 | 0.00
$A_{13}$ |  |  |  |  |  |  |  |  |  |  |  |  |  | 1.07 | 1.29
$A_{14}$ |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 4.30

Table 20: Absolute model uncertainties on the interference fit fractions (%).
The amplitudes are: ($A_{0}$) ${\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(892)^{0}$, ($A_{1}$) ${\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(1410)^{0}$, ($A_{2}$) ${{\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{0}}(1430)^{0}$, ($A_{3}$) LASS
nonresonant, ($A_{4}$) ${{\kern
1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{2}}(1430)^{0}$, ($A_{5}$)
${\kern 1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}(1680)^{0}$, ($A_{6}$)
${{\kern 1.79997pt\overline{\kern-1.79997ptK}{}}{}^{*}_{0}}(1950)^{0}$,
($A_{7}$) $D^{*-}_{s\,v}$, ($A_{8}$) $D^{*}_{s0\,v}(2317)^{-}$, ($A_{9}$)
$D^{*}_{s2}(2573)^{-}$, ($A_{10}$) $D^{*}_{s1}(2700)^{-}$, ($A_{11}$)
$D^{*}_{s3}(2860)^{-}$, ($A_{12}$) $D^{*}_{s1}(2860)^{-}$, ($A_{13}$)
$B^{*+}_{v}$, ($A_{14}$) Nonresonant. The diagonal elements correspond to the
model uncertainties on the fit fractions shown in Table 11.

| $A_{0}$ | $A_{1}$ | $A_{2}$ | $A_{3}$ | $A_{4}$ | $A_{5}$ | $A_{6}$ | $A_{7}$ | $A_{8}$ | $A_{9}$ | $A_{10}$ | $A_{11}$ | $A_{12}$ | $A_{13}$ | $A_{14}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$A_{0}$ | 0.88 | 1.58 | 0.00 | 0.00 | 0.00 | 1.18 | 0.00 | 1.67 | 0.96 | 0.88 | 0.35 | 0.36 | 0.66 | 0.81 | 2.66
$A_{1}$ |  | 1.37 | 0.00 | 0.00 | 0.00 | 1.11 | 0.00 | 0.79 | 0.68 | 0.62 | 0.29 | 0.24 | 0.45 | 0.48 | 1.58
$A_{2}$ |  |  | 3.32 | 3.63 | 0.00 | 0.00 | 1.33 | 1.21 | 0.85 | 0.42 | 0.17 | 0.11 | 0.43 | 1.22 | 2.16
$A_{3}$ |  |  |  | 4.09 | 0.00 | 0.00 | 0.95 | 1.96 | 1.28 | 0.77 | 0.54 | 0.11 | 0.99 | 3.28 | 2.82
$A_{4}$ |  |  |  |  | 1.06 | 0.00 | 0.00 | 0.20 | 0.28 | 0.46 | 0.16 | 0.22 | 0.26 | 0.43 | 1.84
$A_{5}$ |  |  |  |  |  | 0.80 | 0.00 | 1.50 | 0.76 | 0.78 | 0.27 | 0.23 | 0.40 | 0.43 | 1.15
$A_{6}$ |  |  |  |  |  |  | 2.42 | 1.45 | 0.68 | 0.60 | 0.20 | 0.16 | 0.30 | 1.29 | 2.49
$A_{7}$ |  |  |  |  |  |  |  | 4.02 | 0.00 | 0.00 | 1.24 | 0.00 | 1.47 | 4.30 | 0.00
$A_{8}$ |  |  |  |  |  |  |  |  | 2.30 | 0.00 | 0.00 | 0.00 | 0.00 | 1.69 | 7.43
$A_{9}$ |  |  |  |  |  |  |  |  |  | 1.05 | 0.00 | 0.00 | 0.00 | 0.35 | 0.00
$A_{10}$ |  |  |  |  |  |  |  |  |  |  | 0.54 | 0.00 | 0.75 | 1.12 | 0.00
$A_{11}$ |  |  |  |  |  |  |  |  |  |  |  | 0.42 | 0.00 | 0.03 | 0.00
$A_{12}$ |  |  |  |  |  |  |  |  |  |  |  |  | 3.28 | 1.27 | 0.00
$A_{13}$ |  |  |  |  |  |  |  |  |  |  |  |  |  | 1.83 | 4.68
$A_{14}$ |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 7.64

LHCb collaboration

R. Aaij41, B. Adeva37, M. Adinolfi46, A. Affolder52, Z. Ajaltouni5, S. Akar6,
J. Albrecht9, F. Alessio38, M. Alexander51, S. Ali41, G. Alkhazov30, P.
Alvarez Cartelle37, A.A. Alves Jr25,38, S. Amato2, S. Amerio22, Y. Amhis7, L.
An3, L. Anderlini17,g, J. Anderson40, R. Andreassen57, M. Andreotti16,f, J.E.
Andrews58, R.B. Appleby54, O. Aquines Gutierrez10, F. Archilli38, A.
Artamonov35, M. Artuso59, E. Aslanides6, G. Auriemma25,n, M. Baalouch5, S.
Bachmann11, J.J. Back48, A. Badalov36, C. Baesso60, W. Baldini16, R.J.
Barlow54, C. Barschel38, S. Barsuk7, W. Barter47, V. Batozskaya28, V.
Battista39, A. Bay39, L. Beaucourt4, J. Beddow51, F. Bedeschi23, I. Bediaga1,
S. Belogurov31, K. Belous35, I. Belyaev31, E. Ben-Haim8, G. Bencivenni18, S.
Benson38, J. Benton46, A. Berezhnoy32, R. Bernet40, M.-O. Bettler47, M. van
Beuzekom41, A. Bien11, S. Bifani45, T. Bird54, A. Bizzeti17,i, P.M.
Bjørnstad54, T. Blake48, F. Blanc39, J. Blouw10, S. Blusk59, V. Bocci25, A.
Bondar34, N. Bondar30,38, W. Bonivento15,38, S. Borghi54, A. Borgia59, M.
Borsato7, T.J.V. Bowcock52, E. Bowen40, C. Bozzi16, T. Brambach9, J. van den
Brand42, J. Bressieux39, D. Brett54, M. Britsch10, T. Britton59, J.
Brodzicka54, N.H. Brook46, H. Brown52, A. Bursche40, G. Busetto22,r, J.
Buytaert38, S. Cadeddu15, R. Calabrese16,f, M. Calvi20,k, M. Calvo Gomez36,p,
P. Campana18,38, D. Campora Perez38, A. Carbone14,d, G. Carboni24,l, R.
Cardinale19,38,j, A. Cardini15, L. Carson50, K. Carvalho Akiba2, G. Casse52,
L. Cassina20, L. Castillo Garcia38, M. Cattaneo38, Ch. Cauet9, R. Cenci58, M.
Charles8, Ph. Charpentier38, M. Chefdeville4, S. Chen54, S.-F. Cheung55, N.
Chiapolini40, M. Chrzaszcz40,26, K. Ciba38, X. Cid Vidal38, G. Ciezarek53,
P.E.L. Clarke50, M. Clemencic38, H.V. Cliff47, J. Closier38, V. Coco38, J.
Cogan6, E. Cogneras5, P. Collins38, A. Comerma-Montells11, A. Contu15, A.
Cook46, M. Coombes46, S. Coquereau8, G. Corti38, M. Corvo16,f, I. Counts56, B.
Couturier38, G.A. Cowan50, D.C. Craik48, M. Cruz Torres60, S. Cunliffe53, R.
Currie50, C. D’Ambrosio38, J. Dalseno46, P. David8, P.N.Y. David41, A.
Davis57, K. De Bruyn41, S. De Capua54, M. De Cian11, J.M. De Miranda1, L. De
Paula2, W. De Silva57, P. De Simone18, D. Decamp4, M. Deckenhoff9, L. Del
Buono8, N. Déléage4, D. Derkach55, O. Deschamps5, F. Dettori38, A. Di Canto38,
H. Dijkstra38, S. Donleavy52, F. Dordei11, M. Dorigo39, A. Dosil Suárez37, D.
Dossett48, A. Dovbnya43, K. Dreimanis52, G. Dujany54, F. Dupertuis39, P.
Durante38, R. Dzhelyadin35, A. Dziurda26, A. Dzyuba30, S. Easo49,38, U.
Egede53, V. Egorychev31, S. Eidelman34, S. Eisenhardt50, U. Eitschberger9, R.
Ekelhof9, L. Eklund51, I. El Rifai5, Ch. Elsasser40, S. Ely59, S. Esen11,
H.-M. Evans47, T. Evans55, A. Falabella14, C. Färber11, C. Farinelli41, N.
Farley45, S. Farry52, RF Fay52, D. Ferguson50, V. Fernandez Albor37, F.
Ferreira Rodrigues1, M. Ferro-Luzzi38, S. Filippov33, M. Fiore16,f, M.
Fiorini16,f, M. Firlej27, C. Fitzpatrick39, T. Fiutowski27, M. Fontana10, F.
Fontanelli19,j, R. Forty38, O. Francisco2, M. Frank38, C. Frei38, M.
Frosini17,38,g, J. Fu21,38, E. Furfaro24,l, A. Gallas Torreira37, D.
Galli14,d, S. Gallorini22, S. Gambetta19,j, M. Gandelman2, P. Gandini59, Y.
Gao3, J. García Pardiñas37, J. Garofoli59, J. Garra Tico47, L. Garrido36, C.
Gaspar38, R. Gauld55, L. Gavardi9, G. Gavrilov30, A. Geraci21,v, E.
Gersabeck11, M. Gersabeck54, T. Gershon48, Ph. Ghez4, A. Gianelle22, S.
Gianì39, V. Gibson47, L. Giubega29, V.V. Gligorov38, C. Göbel60, D.
Golubkov31, A. Golutvin53,31,38, A. Gomes1,a, C. Gotti20, M. Grabalosa
Gándara5, R. Graciani Diaz36, L.A. Granado Cardoso38, E. Graugés36, G.
Graziani17, A. Grecu29, E. Greening55, S. Gregson47, P. Griffith45, L.
Grillo11, O. Grünberg62, B. Gui59, E. Gushchin33, Yu. Guz35,38, T. Gys38, C.
Hadjivasiliou59, G. Haefeli39, C. Haen38, S.C. Haines47, S. Hall53, B.
Hamilton58, T. Hampson46, X. Han11, S. Hansmann-Menzemer11, N. Harnew55, S.T.
Harnew46, J. Harrison54, J. He38, T. Head38, V. Heijne41, K. Hennessy52, P.
Henrard5, L. Henry8, J.A. Hernando Morata37, E. van Herwijnen38, M. Heß62, A.
Hicheur1, D. Hill55, M. Hoballah5, C. Hombach54, W. Hulsbergen41, P. Hunt55,
N. Hussain55, D. Hutchcroft52, D. Hynds51, M. Idzik27, P. Ilten56, R.
Jacobsson38, A. Jaeger11, J. Jalocha55, E. Jans41, P. Jaton39, A. Jawahery58,
F. Jing3, M. John55, D. Johnson38, C.R. Jones47, C. Joram38, B. Jost38, N.
Jurik59, M. Kaballo9, S. Kandybei43, W. Kanso6, M. Karacson38, T.M. Karbach38,
S. Karodia51, M. Kelsey59, I.R. Kenyon45, T. Ketel42, B. Khanji20, C.
Khurewathanakul39, S. Klaver54, K. Klimaszewski28, O. Kochebina7, M. Kolpin11,
I. Komarov39, R.F. Koopman42, P. Koppenburg41,38, M. Korolev32, A.
Kozlinskiy41, L. Kravchuk33, K. Kreplin11, M. Kreps48, G. Krocker11, P.
Krokovny34, F. Kruse9, W. Kucewicz26,o, M. Kucharczyk20,26,38,k, V.
Kudryavtsev34, K. Kurek28, T. Kvaratskheliya31, V.N. La Thi39, D. Lacarrere38,
G. Lafferty54, A. Lai15, D. Lambert50, R.W. Lambert42, G. Lanfranchi18, C.
Langenbruch48, B. Langhans38, T. Latham48, C. Lazzeroni45, R. Le Gac6, J. van
Leerdam41, J.-P. Lees4, R. Lefèvre5, A. Leflat32, J. Lefrançois7, S. Leo23, O.
Leroy6, T. Lesiak26, B. Leverington11, Y. Li3, T. Likhomanenko63, M. Liles52,
R. Lindner38, C. Linn38, F. Lionetto40, B. Liu15, S. Lohn38, I. Longstaff51,
J.H. Lopes2, N. Lopez-March39, P. Lowdon40, H. Lu3, D. Lucchesi22,r, H. Luo50,
A. Lupato22, E. Luppi16,f, O. Lupton55, F. Machefert7, I.V. Machikhiliyan31,
F. Maciuc29, O. Maev30, S. Malde55, A. Malinin63, G. Manca15,e, G.
Mancinelli6, A. Mapelli38, J. Maratas5, J.F. Marchand4, U. Marconi14, C. Marin
Benito36, P. Marino23,t, R. Märki39, J. Marks11, G. Martellotti25, A.
Martens8, A. Martín Sánchez7, M. Martinelli39, D. Martinez Santos42, F.
Martinez Vidal64, D. Martins Tostes2, A. Massafferri1, R. Matev38, Z. Mathe38,
C. Matteuzzi20, A. Mazurov16,f, M. McCann53, J. McCarthy45, A. McNab54, R.
McNulty12, B. McSkelly52, B. Meadows57, F. Meier9, M. Meissner11, M. Merk41,
D.A. Milanes8, M.-N. Minard4, N. Moggi14, J. Molina Rodriguez60, S. Monteil5,
M. Morandin22, P. Morawski27, A. Mordà6, M.J. Morello23,t, J. Moron27, A.-B.
Morris50, R. Mountain59, F. Muheim50, K. Müller40, M. Mussini14, B. Muster39,
P. Naik46, T. Nakada39, R. Nandakumar49, I. Nasteva2, M. Needham50, N. Neri21,
S. Neubert38, N. Neufeld38, M. Neuner11, A.D. Nguyen39, T.D. Nguyen39, C.
Nguyen-Mau39,q, M. Nicol7, V. Niess5, R. Niet9, N. Nikitin32, T. Nikodem11, A.
Novoselov35, D.P. O’Hanlon48, A. Oblakowska-Mucha27, V. Obraztsov35, S.
Oggero41, S. Ogilvy51, O. Okhrimenko44, R. Oldeman15,e, G. Onderwater65, M.
Orlandea29, J.M. Otalora Goicochea2, P. Owen53, A. Oyanguren64, B.K. Pal59, A.
Palano13,c, F. Palombo21,u, M. Palutan18, J. Panman38, A. Papanestis49,38, M.
Pappagallo51, L.L. Pappalardo16,f, C. Parkes54, C.J. Parkinson9,45, G.
Passaleva17, G.D. Patel52, M. Patel53, C. Patrignani19,j, A. Pazos Alvarez37,
A. Pearce54, A. Pellegrino41, M. Pepe Altarelli38, S. Perazzini14,d, E. Perez
Trigo37, P. Perret5, M. Perrin-Terrin6, L. Pescatore45, E. Pesen66, K.
Petridis53, A. Petrolini19,j, E. Picatoste Olloqui36, B. Pietrzyk4, T.
Pilař48, D. Pinci25, A. Pistone19, S. Playfer50, M. Plo Casasus37, F. Polci8,
A. Poluektov48,34, E. Polycarpo2, A. Popov35, D. Popov10, B. Popovici29, C.
Potterat2, E. Price46, J. Prisciandaro39, A. Pritchard52, C. Prouve46, V.
Pugatch44, A. Puig Navarro39, G. Punzi23,s, W. Qian4, B. Rachwal26, J.H.
Rademacker46, B. Rakotomiaramanana39, M. Rama18, M.S. Rangel2, I. Raniuk43, N.
Rauschmayr38, G. Raven42, S. Reichert54, M.M. Reid48, A.C. dos Reis1, S.
Ricciardi49, S. Richards46, M. Rihl38, K. Rinnert52, V. Rives Molina36, D.A.
Roa Romero5, P. Robbe7, A.B. Rodrigues1, E. Rodrigues54, P. Rodriguez Perez54,
S. Roiser38, V. Romanovsky35, A. Romero Vidal37, M. Rotondo22, J. Rouvinet39,
T. Ruf38, F. Ruffini23, H. Ruiz36, P. Ruiz Valls64, J.J. Saborido Silva37, N.
Sagidova30, P. Sail51, B. Saitta15,e, V. Salustino Guimaraes2, C. Sanchez
Mayordomo64, B. Sanmartin Sedes37, R. Santacesaria25, C. Santamarina Rios37,
E. Santovetti24,l, A. Sarti18,m, C. Satriano25,n, A. Satta24, D.M. Saunders46,
M. Savrie16,f, D. Savrina31,32, M. Schiller42, H. Schindler38, M. Schlupp9, M.
Schmelling10, B. Schmidt38, O. Schneider39, A. Schopper38, M.-H. Schune7, R.
Schwemmer38, B. Sciascia18, A. Sciubba25, M. Seco37, A. Semennikov31, I.
Sepp53, N. Serra40, J. Serrano6, L. Sestini22, P. Seyfert11, M. Shapkin35, I.
Shapoval16,43,f, Y. Shcheglov30, T. Shears52, L. Shekhtman34, V. Shevchenko63,
A. Shires9, R. Silva Coutinho48, G. Simi22, M. Sirendi47, N. Skidmore46, T.
Skwarnicki59, N.A. Smith52, E. Smith55,49, E. Smith53, J. Smith47, M. Smith54,
H. Snoek41, M.D. Sokoloff57, F.J.P. Soler51, F. Soomro39, D. Souza46, B. Souza
De Paula2, B. Spaan9, A. Sparkes50, P. Spradlin51, S. Sridharan38, F.
Stagni38, M. Stahl11, S. Stahl11, O. Steinkamp40, O. Stenyakin35, S.
Stevenson55, S. Stoica29, S. Stone59, B. Storaci40, S. Stracka23,38, M.
Straticiuc29, U. Straumann40, R. Stroili22, V.K. Subbiah38, L. Sun57, W.
Sutcliffe53, K. Swientek27, S. Swientek9, V. Syropoulos42, M. Szczekowski28,
P. Szczypka39,38, D. Szilard2, T. Szumlak27, S. T’Jampens4, M. Teklishyn7, G.
Tellarini16,f, F. Teubert38, C. Thomas55, E. Thomas38, J. van Tilburg41, V.
Tisserand4, M. Tobin39, S. Tolk42, L. Tomassetti16,f, D. Tonelli38, S. Topp-
Joergensen55, N. Torr55, E. Tournefier4, S. Tourneur39, M.T. Tran39, M.
Tresch40, A. Tsaregorodtsev6, P. Tsopelas41, N. Tuning41, M. Ubeda Garcia38,
A. Ukleja28, A. Ustyuzhanin63, U. Uwer11, V. Vagnoni14, G. Valenti14, A.
Vallier7, R. Vazquez Gomez18, P. Vazquez Regueiro37, C. Vázquez Sierra37, S.
Vecchi16, J.J. Velthuis46, M. Veltri17,h, G. Veneziano39, M. Vesterinen11, B.
Viaud7, D. Vieira2, M. Vieites Diaz37, X. Vilasis-Cardona36,p, A. Vollhardt40,
D. Volyanskyy10, D. Voong46, A. Vorobyev30, V. Vorobyev34, C. Voß62, H.
Voss10, J.A. de Vries41, R. Waldi62, C. Wallace48, R. Wallace12, J. Walsh23,
S. Wandernoth11, J. Wang59, D.R. Ward47, N.K. Watson45, D. Websdale53, M.
Whitehead48, J. Wicht38, D. Wiedner11, G. Wilkinson55, M.P. Williams45, M.
Williams56, F.F. Wilson49, J. Wimberley58, J. Wishahi9, W. Wislicki28, M.
Witek26, G. Wormser7, S.A. Wotton47, S. Wright47, S. Wu3, K. Wyllie38, Y.
Xie61, Z. Xing59, Z. Xu39, Z. Yang3, X. Yuan3, O. Yushchenko35, M. Zangoli14,
M. Zavertyaev10,b, L. Zhang59, W.C. Zhang12, Y. Zhang3, A. Zhelezov11, A.
Zhokhov31, L. Zhong3, A. Zvyagin38.

1Centro Brasileiro de Pesquisas Físicas (CBPF), Rio de Janeiro, Brazil
2Universidade Federal do Rio de Janeiro (UFRJ), Rio de Janeiro, Brazil
3Center for High Energy Physics, Tsinghua University, Beijing, China
4LAPP, Université de Savoie, CNRS/IN2P3, Annecy-Le-Vieux, France
5Clermont Université, Université Blaise Pascal, CNRS/IN2P3, LPC, Clermont-
Ferrand, France
6CPPM, Aix-Marseille Université, CNRS/IN2P3, Marseille, France
7LAL, Université Paris-Sud, CNRS/IN2P3, Orsay, France
8LPNHE, Université Pierre et Marie Curie, Université Paris Diderot,
CNRS/IN2P3, Paris, France
9Fakultät Physik, Technische Universität Dortmund, Dortmund, Germany
10Max-Planck-Institut für Kernphysik (MPIK), Heidelberg, Germany
11Physikalisches Institut, Ruprecht-Karls-Universität Heidelberg, Heidelberg,
Germany
12School of Physics, University College Dublin, Dublin, Ireland
13Sezione INFN di Bari, Bari, Italy
14Sezione INFN di Bologna, Bologna, Italy
15Sezione INFN di Cagliari, Cagliari, Italy
16Sezione INFN di Ferrara, Ferrara, Italy
17Sezione INFN di Firenze, Firenze, Italy
18Laboratori Nazionali dell’INFN di Frascati, Frascati, Italy
19Sezione INFN di Genova, Genova, Italy
20Sezione INFN di Milano Bicocca, Milano, Italy
21Sezione INFN di Milano, Milano, Italy
22Sezione INFN di Padova, Padova, Italy
23Sezione INFN di Pisa, Pisa, Italy
24Sezione INFN di Roma Tor Vergata, Roma, Italy
25Sezione INFN di Roma La Sapienza, Roma, Italy
26Henryk Niewodniczanski Institute of Nuclear Physics Polish Academy of
Sciences, Kraków, Poland
27AGH - University of Science and Technology, Faculty of Physics and Applied
Computer Science, Kraków, Poland
28National Center for Nuclear Research (NCBJ), Warsaw, Poland
29Horia Hulubei National Institute of Physics and Nuclear Engineering,
Bucharest-Magurele, Romania
30Petersburg Nuclear Physics Institute (PNPI), Gatchina, Russia
31Institute of Theoretical and Experimental Physics (ITEP), Moscow, Russia
32Institute of Nuclear Physics, Moscow State University (SINP MSU), Moscow,
Russia
33Institute for Nuclear Research of the Russian Academy of Sciences (INR RAN),
Moscow, Russia
34Budker Institute of Nuclear Physics (SB RAS) and Novosibirsk State
University, Novosibirsk, Russia
35Institute for High Energy Physics (IHEP), Protvino, Russia
36Universitat de Barcelona, Barcelona, Spain
37Universidad de Santiago de Compostela, Santiago de Compostela, Spain
38European Organization for Nuclear Research (CERN), Geneva, Switzerland
39Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
40Physik-Institut, Universität Zürich, Zürich, Switzerland
41Nikhef National Institute for Subatomic Physics, Amsterdam, The Netherlands
42Nikhef National Institute for Subatomic Physics and VU University Amsterdam,
Amsterdam, The Netherlands
43NSC Kharkiv Institute of Physics and Technology (NSC KIPT), Kharkiv, Ukraine
44Institute for Nuclear Research of the National Academy of Sciences (KINR),
Kyiv, Ukraine
45University of Birmingham, Birmingham, United Kingdom
46H.H. Wills Physics Laboratory, University of Bristol, Bristol, United
Kingdom
47Cavendish Laboratory, University of Cambridge, Cambridge, United Kingdom
48Department of Physics, University of Warwick, Coventry, United Kingdom
49STFC Rutherford Appleton Laboratory, Didcot, United Kingdom
50School of Physics and Astronomy, University of Edinburgh, Edinburgh, United
Kingdom
51School of Physics and Astronomy, University of Glasgow, Glasgow, United
Kingdom
52Oliver Lodge Laboratory, University of Liverpool, Liverpool, United Kingdom
53Imperial College London, London, United Kingdom
54School of Physics and Astronomy, University of Manchester, Manchester,
United Kingdom
55Department of Physics, University of Oxford, Oxford, United Kingdom
56Massachusetts Institute of Technology, Cambridge, MA, United States
57University of Cincinnati, Cincinnati, OH, United States
58University of Maryland, College Park, MD, United States
59Syracuse University, Syracuse, NY, United States
60Pontifícia Universidade Católica do Rio de Janeiro (PUC-Rio), Rio de
Janeiro, Brazil, associated to 2
61Institute of Particle Physics, Central China Normal University, Wuhan,
Hubei, China, associated to 3
62Institut für Physik, Universität Rostock, Rostock, Germany, associated to 11
63National Research Centre Kurchatov Institute, Moscow, Russia, associated to
31
64Instituto de Fisica Corpuscular (IFIC), Universitat de Valencia-CSIC,
Valencia, Spain, associated to 36
65KVI - University of Groningen, Groningen, The Netherlands, associated to 41
66Celal Bayar University, Manisa, Turkey, associated to 38

aUniversidade Federal do Triângulo Mineiro (UFTM), Uberaba-MG, Brazil
bP.N. Lebedev Physical Institute, Russian Academy of Science (LPI RAS),
Moscow, Russia
cUniversità di Bari, Bari, Italy
dUniversità di Bologna, Bologna, Italy
eUniversità di Cagliari, Cagliari, Italy
fUniversità di Ferrara, Ferrara, Italy
gUniversità di Firenze, Firenze, Italy
hUniversità di Urbino, Urbino, Italy
iUniversità di Modena e Reggio Emilia, Modena, Italy
jUniversità di Genova, Genova, Italy
kUniversità di Milano Bicocca, Milano, Italy
lUniversità di Roma Tor Vergata, Roma, Italy
mUniversità di Roma La Sapienza, Roma, Italy
nUniversità della Basilicata, Potenza, Italy
oAGH - University of Science and Technology, Faculty of Computer Science,
Electronics and Telecommunications, Kraków, Poland
pLIFAELS, La Salle, Universitat Ramon Llull, Barcelona, Spain
qHanoi University of Science, Hanoi, Viet Nam
rUniversità di Padova, Padova, Italy
sUniversità di Pisa, Pisa, Italy
tScuola Normale Superiore, Pisa, Italy
uUniversità degli Studi di Milano, Milano, Italy
vPolitecnico di Milano, Milano, Italy