# Analyzing the Use of Camera Glasses in the Wild

Taryn Bipat<EMAIL_ADDRESS>University of Washington, Snap Inc. ,  Maarten
Willem Bos<EMAIL_ADDRESS>Snap Inc. ,  Rajan Vaish<EMAIL_ADDRESS>Snap Inc.
and  Andrés Monroy-Hernández<EMAIL_ADDRESS>Snap Inc.

(2019; 2019)

###### Abstract.

Camera glasses enable people to capture point-of-view videos using a common
accessory, hands-free. In this paper, we investigate how, when, and why people
used one such product: Spectacles. We conducted 39 semi-structured interviews
and surveys with 191 owners of Spectacles. We found that the form factor
elicits sustained usage behaviors, and opens opportunities for new use-cases
and types of content captured. We provide a usage typology, and highlight
societal and individual factors that influence the classification of
behaviors.

camera glasses, smart glasses, wearables, usability

††copyright: none††doi: 10.475/123_4††isbn: 123-4567-24-567/08/06††conference:
; 2019††journalyear: ;††ccs: Human-centered computing User
studies††journalyear: 2019††copyright: acmcopyright††conference: CHI
Conference on Human Factors in Computing Systems Proceedings; May 4–9, 2019;
Glasgow, Scotland UK††booktitle: CHI Conference on Human Factors in Computing
Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland UK††price:
15.00††doi: 10.1145/3290605.3300651††isbn: 978-1-4503-5970-2/19/05

##  1\. Introduction

The range of form factors for wearable devices has expanded dramatically in
recent years, from watches to clothing, helmets, and more. As people expand
their technology use from mobile phones to wearables, new research questions
emerge about how, why, and where these technologies are used. In this paper we
focus on studying camera glasses, a type of smart glasses that have the camera
as their core functionality.

Researchers have classified the functionality of, and tried to predict the
motivations for early adoption of, various types of wearables (Park et al.,
2015; Bhat et al., 2014). However, there is not much work on how these devices
are actually used once they are adopted. Furthermore, when it comes to camera
glasses, most of the existing research focuses on experimental applications
(McAtamney and Parker, 2006; Byrne et al., 2008; Muralidhar et al., 2016) and
not much on how everyday people use camera glasses in their life, outside
research settings.

In this work we seek to further understand the postadoption use and social
implications of smart glasses. We do this by studying the everyday use of
Spectacles, a set of commercially available camera glasses that its
manufacturer describes as “sunglasses that capture your world” and a “hands-
free camera” (Inc., 2018). Spectacles’ users capture 10-second videos by
pressing a button at the top left corner of the device. Pressing the button
again adds 10 seconds to the recording, for a maximum of 30 seconds. Pressing-
and-holding captures a still photo.

The content is automatically and wirelessly transferred from the device’s
storage to the user’s Snapchat application on their smartphone, which people
can use to view and share their content. The videos and photos are recorded in
a high-definition (HD) circular shape, but they are exportable in circular,
square, or wide-screen formats.

We see “camera glasses,” like Spectacles, as part of a “smart glasses”
category of wearables that include VR/AR headsets and glasses with speakers
(enjoyvue.com, 2018). Unlike other smart glasses, camera glasses consist only
of capturing technology. To make the distinction clear, we will refer to
Spectacles as camera glasses.

We conducted 39 semi-structured interviews and surveys with 191 owners of
Spectacles. We found that participants used their camera glasses primarily
during social interactions and while being physically active. Furthermore,
participants captured their own perspective for preserving their memories and
to share them with others. For them, camera glasses supplement or even replace
other capturing devices. Users’ individual lifestyles—from their job to the
events they attend—dictate how they use camera glasses. Lastly, we found that
individual and societal boundaries affect how people use camera glasses.
People take societal views of camera glasses into account when deciding in
what scenarios to use them.

##  2\. Literature Review

Some of the earliest descriptions of wearables focused on three key modes of
interactions they enable: constancy, always ready to be used; augmentation,
enhancing analog tasks; mediation, filtering incoming information and
protecting users’ privacy (Mann, 1998).

Recently this model was expanded (Wright and Keith, 2014), adding a focus on
form factors. The updated model describes wearables as electronics and
computers that are integrated into clothing and other accessories worn
comfortably on the body. This recent work argues that wearables have the
potential to improve people’s health by becoming integrated into people’s way
of life.

Despite this interest in how wearables fit in people’s everyday lives, a lot
of the research on smart glasses has focused on experimental hardware and
niche applications(Zhang et al., 2014; Rallapalli et al., 2014; Le et al.,
2013; Xiong et al., 2017). Even when investigating commercially-available
products like Google Glass, researchers focused on niche medical applications
(Malu and Findlater, 2014).

Another line of research on smart glasses has centered around identifying
potential early adopters, and their motivations (Rauschnabel et al., 2015;
Rauschnabel and Ro, 2016). This work is elucidating but it does not get at
what people actually do once they purchase these devices. For example,
Kalatari and colleagues (Kalantari, 2017) highlighted the lack of qualitative
studies on the types of usage people exhibit after acquiring smart glasses.

Recent research has also found that wearables’ adoption, although increasing,
has been slower and more varied than initially expected (Kalantari, 2017).
This also highlights the importance of understanding sustained use “in the
wild” when it does happen.

A study like ours is needed to delve deeper into sustained behaviors;
revealing the need to understand concerns around privacy, fashion trends,
technological challenges and more (Kalantari, 2017).

###  2.1. Social Implications of Wearables

Prior research has shown that wearables’ style and the fact that they are worn
close to the body contribute to the technology being perceived as a companion
and fashion item, rather than just as a tool (Kao et al., 2017). This has
perhaps influenced the design of camera glasses, many of which are built to
look like regular sunglasses.

Research has also shown that body-worn cameras can create tensions in certain
social settings (Koelle et al., 2017) due to privacy concerns. It is worth
noting that Spectacles, the camera glasses described in this paper, mitigate
these challenges through the use of LEDs that inform bystanders that
recordings are occurring. We identified the opportunity to contribute to the
literature by studying how camera glasses are used post-adoption. As part of
this, we frame our investigation around two dimensions: _activities_ and
_time_. We build on prior work on “context” by Dey and colleagues who
identified four contextual dimensions for technology: location, identity,
activities, and time (Abowd et al., 1999).

Although there is not much research specifically on Spectacles, it is worth
noting that prior work on smart glasses referred to Spectacles as a tool for
lifelogging (Chow and Rissman, 2017; Koelle et al., 2017; Fan et al., 2018)
and storytelling (Bossert, 2017), a point-of-view camera platform (Chong et
al., 2017), and as an extension of the Snapchat mobile app (Khan and Carroll,
2017; Lanigan, 2017).

##  3\. Methods

Since we aimed to understand not only how and where, but also why people use
camera glasses, we decided to use both surveys and interviews. The semi-
structured interviews allowed us to understand who the users are and their
lived experiences with camera glasses. For detailed information, both
protocols can be found in the supplementary documents.

We partnered with the Spectacles team at Snap Inc., the company that created
Spectacles, to reach out to people who had opted to be contacted and had used
their camera glasses for at least two weeks. The research protocol was
reviewed by an internal team of privacy experts at Snap Inc. before being run.

We do not differentiate between the two versions111one released in November,
2016, and the other in April, 2018 of Spectacles that were available at the
time of this study because their basic functionality—as it relates to this
paper—remained largely the same. That said, it is worth noting that because
the second version is water-resistant, some of the use cases we describe might
vary slightly across versions.

###  3.1. Surveys

The 191 participants who answered our survey were recruited via email
approximately two weeks after they had successfully paired their camera
glasses with a phone for the first time. We emailed participants during May
and June of 2018. The survey included multiple-choice and open-ended questions
about who participants were, how they used their device, and their reasons for
using it. Through these surveys, we were able to understand users’ initial
perceptions of the device, reasons for purchasing it, and how it could be
improved. The open-ended responses were hand-coded, and then bucketed into
unique categories or placed within similar multiple-choice buckets.

###  3.2. Semi-structured interviews

We used semi-structured interviews to obtain qualitative data of people’s
experiences. We recruited 39 participants who had owned their camera glasses
for three to twenty months. Participants stemmed from seven different
countries: USA, UK, The Netherlands, Sweden, Israel, Canada, and France.
Participants included students, scientists, marketing specialists,
entertainers, car mechanics, journalists, and IT professionals.

Each video interview lasted approximately 45 minutes and was conducted over
video conference, which allowed us to broaden our geographic reach. All
interviews were recorded with participants’ consent, and the interviewer took
detailed notes of the conversations. Participants received a USD $50 gift card
at the end of the session.

###  3.3. Data Analysis

To analyze the notes taken during the interviews, the first author developed a
common set of themes that emerged from our data through thematic analysis
(Braun and Clarke, 2006). Two researchers met on a weekly basis to extract and
refine the coding scheme. After all user interviews were conducted, the main
findings from each of the participants were listed and then reviewed by all
authors of the paper. Subsequently, we used the interview recordings to
identify additional content related to those themes.

##  4\. Results

Most participants were employed and highly educated, or in the process of
getting their education (see Figure 1). This might correspond to the state of
adoption for wearables in general, as early adopters tend to skew towards
higher socio-economic status.

Figure 1. Survey Participants’ Occupation and Education.

We structure the results in terms of when (time), how (scenarios), and why
(activities) people use their camera glasses. We intertwine the results from
the surveys and the interviews as they displayed the same patterns, with the
interviews providing more in-depth insights to the survey results.

###  4.1. Time: When are camera glasses used?

Figure 2.  Participants were asked to report how often they use their camera
glasses normally and indoors

About one in three participants reported using their camera glasses once or
twice a week, and one in four reported using them all the time. As expected
for a device with sunglasses-style lenses, usage is primarily outdoors (see
Figure 2).

###  4.2. Scenarios: How are camera glasses used?

Figure 3.  Most popular usage scenarios

Our surveys showed that camera glasses afford many different uses. We focus on
the top four usage scenarios from the survey (see Figure 3), which we also
corroborated in the semi-structured interviews.

Spectacles, the camera glasses investigated in our study, were created as a
sunglasses-only product. User scenarios such as outdoor activities, traveling,
and water activities are all situations reflecting this design choice.
Approximately 23% of survey respondents mentioned they would use Spectacles
more if they were in more situations where sunglasses are appropriate.
Furthermore, activities such as (indoor) concerts, shopping, and work-related
activities showed less usage because people are less prone to wear sunglasses
in these situations.

Additionally, activities such as hanging out with friends and family were
popular. This may be because users are more likely to use camera glasses in
social situations to capture their memories. Camera glasses were less popular
in more solitary activities.

Outdoor activities.:

People used their camera glasses for outdoor activities because it allowed
them to capture an immersive and personal view of their environment:

> “I took them hiking and you could hear my breathing getting heavier and
> heavier as we climbed. So when you look back at it, you can see what I was
> seeing and feeling.” —P24

Furthermore, having the camera glasses allowed participants to document
collective outdoor activities for others:

> “I took them skiing because I was with my family, girlfriend, younger
> brother. They were quite keen to get in the shots. They would all want some
> clips, while I was following them.” —P12

Traveling.:

The second most common use of the camera glasses was while traveling,
especially when being “out and about”:

> “I use it for beach, travel… when I am going away, I can take pictures and
> videos of my destination. Those are really the big ones, when I am out and
> about exploring.” —P11

Participants noted that camera glasses allowed them to capture their traveling
adventures and share those experiences with others:

> “I use it for travel and sightseeing. Since I live so close to Washington, I
> use it for travel. I get to bring people to other places.” —P2

Water activities.:

The release of the water-resistant feature in the second version of the
Spectacles allowed people to capture water activities. These activities
included trips to the beach and the pool, and having smooth transitions to and
from non-water-related activities:

> “We were playing around with a football, trying to throw a ball into the
> pool. We were recording with a camera, dunking into the pool.” —P27

Spending time with friends & family:

People often captured moments while hanging out with other people:

> “I use it mainly when I am with my friends. If I want to capture a moment,
> now I can get them together and take a picture.” —P13

Also, some people reporting letting others wear the glasses to allow them to
capture their own perspective. For instance, a parent might let their child
wear the glasses and capture their perspective.

###  4.3. Why are camera glasses used?

In our interviews, we were able to get a deeper understanding of why camera
glasses are particularly fit for these circumstances. To help explain why
camera glasses are used, we conducted a thematic analysis on the interview
data. Below, we report user quotes that describe the majority of reasons users
felt that camera glasses were the best way to capture that moment.

Our data show that capturing behavior using camera glasses is dependent on
lifestyle factors. These factors include other devices people have, such as
cellphones and action cameras, what types of careers they have, and what their
daily lives look like. Another factor is what users do with content after
capturing. In the discussion section, we extend these findings by postulating
some factors that influence the reasons camera glasses are used.

Capturing their own point of view.:

The development of new wearables has opened up the world of storytelling
(Rieger et al., 2018), where users can capture content to present their own
perspective on an event. As an example, participant P11 mentioned using the
content captured with their camera glasses to view their own first-hand
experiences:

> “My birthday, this time, last year, at a pool club around the city. We got a
> daybed with friends. Some of it I don’t remember taking. Now I look back and
> it’s fun to have the POV [point of view] shot.” –P11

Furthermore, camera glasses let users view their common activities from a
different perspective:

> “I use them on days off. For sports, hanging out with friends, walking
> around, getting glimpses of my perspective.” —P30

Capturing special events.:

By understanding users’ individual accounts of their usage, we learned about
special events that camera glasses are used for. Most participants noted that
they did not use their camera glasses for mundane activities, but rather for
special ones. Spectacles were created and designed to be integrated into
users’ lives as a pair of sunglasses. However, many users waited to capture
special events where sunglasses could be worn, such as visiting a theme park
or going to the vet:

> “Yesterday, we went to Six Flags and we were wearing them on the roller
> coasters that aren’t too extreme.” —P1

> “I adopted a cat. A little after I got the Spectacles, there was a yearly
> check up after I got her. This was the first time I was taking her out of my
> apartment. So I did some clips when I was driving, putting her in the
> carrier, taking her into the vet.” —P11

Capturing as an extension of daily life.:

Spectacles are not used all day, every day. However, at times they are used to
augment daily activities. Some participants did mention integrating their
camera glasses into their lives for more “mundane” activities, especially
around other people:

> “I usually use it when we go out with friends and are just walking around
> outside.” —P5

> “I use it when going to playground with my son, swimming, I have been to
> pool with them. One video was me taking it to the pool and you can see the
> water rushing towards you and it was great.” —P24

Capturing for work.:

Some people found professional uses for their camera glasses. For example, a
participant who organizes school trips found camera glasses useful for
documenting the trips:

> “I go on field trips with my students and I have to take content for the
> weekly newsletter so this is the easiest way to do it without getting
> distracted.” —P29

Similarly, a professional drone pilot used them to capture another perspective
of flying a drone:

> “They offer footage from multiple angles. I use them to show videos of me
> flying the drone, setting it up, etc… then upload videos and share them on
> Snapchat and Instagram.” —P1

Participants in the entertainment and social media industry were able to
capture scenes for their followers that provided novel, behind-the-scenes
perspectives:

> “It is a different way to show my followers what I am doing.” —P10

> “I capture behind the scenes things where I can capture the reaction of the
> thing then morph all the content into a video. —P9

Supplementing or replacing other capturing:

technology.:

In the interviews, many participants noted that their camera glasses were a
unique device that augmented or replaced other devices:

> “I was going to use them in conjunction with my GoPro. Primarily POV,
> especially when I don’t have my DSLR.”—P21

Additionally, users even replaced their other capturing devices with camera
glasses to capture their active lifestyles:

> “This was an interesting way to get GoPro content without having to strap it
> onto my body.”—P32

Moreover, users were able to capture moments that would have been difficult to
capture with other, more conspicuous technologies:

> “Playing with my niece, I never imagined I could get her to take a picture.
> She threw a water balloon at me and it smashed me in the face. It was a neat
> perspective. Kids kind of freeze up when they see a camera, so I get a
> natural moment.”—P29

Lastly, camera glasses allowed for hands-free capturing: over 25% of survey
participants mentioned purchasing Spectacles to capture hands-free content.
From our interviews, this was one of the major reasons for capturing. It made
capturing more accessible than other technologies in situations where people’s
hands and minds are occupied:

> “One time I used it to capture my driving experiences, test driving, press
> the 10 second record, which would be hard to do it with my phone while
> driving. I was going from 0-60 mph, people got to see what I was
> seeing.”—P15

Capturing to share.:

Participants were often using their camera glasses to share content with their
friends and family:

> “I brought them during a trip to Italy, it was really nice because I came
> back and showed the content to my grandparents and it was like they went on
> the trip with me.”—P2

Camera glasses also allowed people to share content from a specific event more
easily. A participant mentioned being able to use their camera glasses to hold
on to a memory and share it with other people attending the same event:

> “Independence day, I had a lot of friends coming around. We were having BBQ,
> drinking beer, and everyone had a great time, and making sure the kids were
> alright. When they left, I shared the videos with everyone and they were
> shocked. It was a short memorable video for them and they really loved
> it.”—P22

Sharing the device itself came up a few times. For instance one of the
participants mentioned passing around their camera glasses during a particular
event, so that at the end they could put together their own perspectives and
share the entirety of the event:

> “Well, I was hanging out with some of my friends, we started passing them
> around. People were sharing them, cool to see what people are looking at
> rather than a camera view.”—P4

Capturing to archive.:

Our findings show that users capture their own point of view with camera
glasses to retain memorable events and share them with others. Prior work has
shown that while social media are mostly about building strong social
networks, they also have the potential of becoming a personal archive over
time (Zhao and Lindley, 2014). We found in our interviews that camera glasses
are used to capture and preserve memories. They have become a tool that aids
in digitally archiving those memories for personal consumption:

> “I don’t post everything, most of the time I just post it on my Snapchat,
> Sometimes I just want to record stuff for me”—P14

##  5\. Discussion

Our results helped us understand three key phenomena. First, people use camera
glasses to capture, archive, and share with others. Second, these behaviors
are influenced by societal norms that are still evolving because camera
glasses adoption is still in its early stages. Third, people’s behaviors are
influenced by individual preferences such as lifestyle and aesthetics. These
findings can be extended to other research on camera glasses, and open a
research toolbox for designers to further study this space, which can
ultimately lead to better camera glasses.

###  5.1. Social interaction and privacy considerations

Camera glasses still make up a small fraction of the wearables market (Nguyen,
2016). Their newness leads to curious reactions from those in the immediate
environment of the user, because many people are not aware of the capabilities
of the device. One participant noted that people asked them about it:

> “When they talk to me, they often have this reaction of ‘Are you filming me
> now?”’ —P30

The increased popularity of wearables has heightened designers’ interest in
the technology’s privacy considerations. The designers of Spectacles
approached this by adding a ring of LEDs that light up while recording. In our
interviews, participants mentioned that people around them noticed the LEDs
and often asked what they meant:

> “I get a lot of ‘What is the light on the glasses?,’ [in the] majority of
> the clips of people [they] are saying ‘What’s that light?’ [I tell them,]
> ‘They’re Snapchat glasses’.” —P18

As more people gain awareness of devices like these, one might see changes in
people’s understanding of features like the LED ring and the sociocultural
norms around the technology. As with other types of cameras, users of camera
glasses consider the privacy for themselves and the people around them. Prior
research on lifeloggers has shown that these considerations occur at the
moment of capturing content, like the participant above (P18) described (Hoyle
et al., 2014). Additionally, some of our participants mentioned thinking about
privacy _before_ capturing too, and making a conscious decision about when it
was appropriate to record:

> “ …respect people’s privacy: the first thing I do is that. When I am taking
> a video, I don’t use this for a spy game.” —P23

All of this highlights the value for researchers to ask how the design and use
of camera glasses impact the privacy of both their users and the people around
them.

###  5.2. Individual preferences

Camera glasses have the ability to take both videos and photos. However, users
mentioned that the design of the device as glasses was better suited for
capturing videos in active scenarios. This is further supported by the high
number of users that use camera glasses for outdoor activities and traveling.
Participants noted that video recording with the camera glasses allowed them
to stay in the moment.

> “The quality of me just moving around with glasses, recording is a lot
> better effect, the fish eye effect is really something.” —P2

> “I think with photo, you try to take a perfect shot. I think in a short
> video, you actually manage to capture a life.” —P30

The hands-free nature of capturing and novel features such as the HD circular
video format influence what situations and types of content are better suited
for use of the camera glasses. This opens up the door for new research on how
features drive behavior and how we can continue to design to motivate
effective usage. Along with individual preferences in design and
functionality, it was noted that users’ lifestyle impacted their usage.
Although some participants have been able to integrate their camera glasses
into their daily activities, some consider their everyday activities “too
mundane” and therefore not worth capturing. In the surveys, over 20% of users
mentioned that they would use Spectacles more if they discovered more
opportunities for capturing. As two participants noted:

> “Maybe it is because it was a new experience [at first], like I did not want
> to keep posting the same type of videos of me riding my bike everyday.”—P23

> “I wish I could record more if I did more stuff.”—P5

Perhaps, like with the evolution from professional camera usage for special
events to the use of camera phones for capturing everyday life, the culture
around camera glasses may evolve to make it more common to use the technology
for mundane activities. It is important for researchers to continue to ask
what individual lifestyle differences impact the post-adoption usage of camera
glasses to further develop products that fit within these dependencies.

##  6\. Limitations

While this paper focuses on laying out a typology of user behavior around
camera glasses, we only interacted with people who used one of two versions of
one particular product. More work is needed to compare usage of these devices,
even across versions of the same product, and other products.

Furthermore, like in any study, the participants in our study may be different
from the general population because of self-selection bias. Future work could
focus on performing quantitative log data analyses on larger and more
representative samples.

##  7\. Conclusion and future work

We classified the scenarios in which people use camera glasses, and what
motivates them to capture content in these situations. Drawing from both
surveys and semi-structured interviews with Spectacles users, we showed how
societal and individual barriers influence usage.

Future research can extend this work by studying the content captured with
camera glasses, i.e., rather than asking people what they do, analyze the
traces of what they do. For instance, one could grab publicly available
content captured with camera glasses and perform manual or automated analyses
(e.g., using computer vision) to identify common scenes and scenarios where
these devices are used. For this study, we did not use actual content captured
by users, because it involves a trade-off between participant privacy and
potential useful insights that could be elicited from the data. Assuming only
a subset of participants would have given us approval to analyze their user
content, we would have had little data to draw upon. From our perspective,
this was not worth the privacy trade-off.

Lastly, as the technology evolves, researchers need to examine the social
implications of camera glasses, and how devices with different aesthetics and
features are adopted, and shape or get shaped by the culture around the people
who use them.

###### Acknowledgements.

We thank the Spectacles team at Snap Inc. for their assistance with this
project.

## References

  * (1)
  * Abowd et al. (1999) Gregory D Abowd, Anind K Dey, Peter J Brown, Nigel Davies, Mark Smith, and Pete Steggles. 1999\.  Towards a Better Understanding of Context and Context-Awareness.  In Handheld and Ubiquitous Computing. Springer Berlin Heidelberg, Karlsruhe, Germany, 304–307.  https://doi.org/10.1007/3-540-48157-5_29
  * Bhat et al. (2014) Avinash Bhat, Priya Badri, and Uday Shankar Reddi. 2014\.  Wearable devices: The next big thing in CRM.  (2014).
  * Bossert (2017) Julian Bossert. 2017\.  Innovative Storytelling with Snap Inc. Spectacles.  In # Innovationtelling, Harald Eichsteller, Marie Elisabeth Mueller, and Devadas Rajaram (Eds.). Nomos Verlagsgesellschaft mbH & Co. KG, Germany, 59–62.  https://doi.org/10.5771/9783845281117-1
  * Braun and Clarke (2006) Virginia Braun and Victoria Clarke. 2006.  Using thematic analysis in psychology.  Qualitative research in psychology 3, 2 (2006), 77–101.
  * Byrne et al. (2008) Daragh Byrne, Aiden R. Doherty, Gareth J. F. Jones, Alan F. Smeaton, Sanna Kumpulainen, and Kalervo Järvelin. 2008.  The SenseCam As a Tool for Task Observation. In Proceedings of the 22Nd British HCI Group Annual Conference on People and Computers: Culture, Creativity, Interaction - Volume 2 (BCS-HCI ’08). BCS Learning & Development Ltd., Swindon, UK, 19–22.  http://dl.acm.org/citation.cfm?id=1531826.1531832
  * Chong et al. (2017) Eunji Chong, Katha Chanda, Zhefan Ye, Audrey Southerland, Nataniel Ruiz, Rebecca M. Jones, Agata Rozga, and James M. Rehg. 2017\.  Detecting Gaze Towards Eyes in Natural Social Interactions and Its Use in Child Assessment.  Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 43 (Sept. 2017), 20 pages.  https://doi.org/10.1145/3131902
  * Chow and Rissman (2017) Tiffany E Chow and Jesse Rissman. 2017.  Neurocognitive mechanisms of real-world autobiographical memory retrieval: insights from studies using wearable camera technology.  Annals of the New York Academy of Sciences 1396, 1 (2017), 202–221.
  * enjoyvue.com (2018) enjoyvue.com. 2018\.  Meet Vue.  (2018).  Retrieved December 20, 2018 from https://www.enjoyvue.com/
  * Fan et al. (2018) Chenyou Fan, Zehua Zhang, and David J Crandall. 2018.  DeepDiary: Lifelogging Image Captioning and Summarization.  Journal of Visual Communication and Image Representation 55 (aug 2018), 40–55.  https://doi.org/10.1016/j.jvcir.2018.05.008
  * Hoyle et al. (2014) Roberto Hoyle, Robert Templeman, Steven Armes, Denise Anthony, David Crandall, and Apu Kapadia. 2014\.  Privacy Behaviors of Lifeloggers Using Wearable Cameras. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’14). ACM, New York, NY, USA, 571–582.  https://doi.org/10.1145/2632048.2632079
  * Inc. (2018) Snap Inc. 2018.  Spectacles: A New Way to Look.  (2018).  Retrieved September 2, 2018 from https://www.spectacles.com
  * Kalantari (2017) Mahdokht Kalantari. 2017\.  Consumers’ adoption of wearable technologies: literature review, synthesis, and future research agenda.  International Journal of Technology Marketing 12, 3 (2017), 274–307.
  * Kao et al. (2017) Hsin-Liu (Cindy) Kao, Deborah Ajilo, Oksana Anilionyte, Artem Dementyev, Inrak Choi, Sean Follmer, and Chris Schmandt. 2017\.  Exploring Interactions and Perceptions of Kinetic Wearables. In Proceedings of the 2017 Conference on Designing Interactive Systems (DIS ’17). ACM, New York, NY, USA, 391–396.  https://doi.org/10.1145/3064663.3064686
  * Khan and Carroll (2017) Roozehra Khan and Christopher Carroll. 2017.  snapchat as a Tool for Medical Education and Opportunity for Engagement.  Chest 152, 4 (2017), A544.
  * Koelle et al. (2017) Marion Koelle, Wilko Heuten, and Susanne Boll. 2017\.  Are You Hiding It?: Usage Habits of Lifelogging Camera Wearers. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’17). ACM, New York, NY, USA, Article 80, 8 pages.  https://doi.org/10.1145/3098279.3122123
  * Lanigan (2017) Patrick Lanigan. 2017\.  Snapchat Guide: Learn How to Use Snapchat Like a Tech-Savvy Teen.  Independently published,
  * Le et al. (2013) Hoang Le, Thanh Dang, and Feng Liu. 2013.  Eye blink detection for smart glasses. In Multimedia (ISM), 2013 IEEE International Symposium on. IEEE, IEEE, Anaheim, CA, USA, 305–308.  https://doi.org/10.1109/ISM.2013.59
  * Malu and Findlater (2014) Meethu Malu and Leah Findlater. 2014.  ”OK Glass?” A Preliminary Exploration of Google Glass for Persons with Upper Body Motor Impairments. In Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS ’14). ACM, New York, NY, USA, 267–268.  https://doi.org/10.1145/2661334.2661400
  * Mann (1998) Steve Mann. 1998\.  Wearable computing as means for personal empowerment. In Keynote Address for The First Int. Conf. on Wearable Computing (ICWC). ICWC-98, Fairfax VA, 51–59.
  * McAtamney and Parker (2006) Gerard McAtamney and Caroline Parker. 2006.  An Examination of the Effects of a Wearable Display on Informal Face-to-face Communication. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’06). ACM, New York, NY, USA, 45–54.  https://doi.org/10.1145/1124772.1124780
  * Muralidhar et al. (2016) Skanda Muralidhar, Jean M. R Costa, Laurent Son Nguyen, and Daniel Gatica-Perez. 2016.  Dites-moi: Wearable Feedback on Conversational Behavior. In Proceedings of the 15th International Conference on Mobile and Ubiquitous Multimedia (MUM ’16). ACM, New York, NY, USA, 261–265.  https://doi.org/10.1145/3012709.3012733
  * Nguyen (2016) My Nguyen. 2016\.  The Most Successful Wearables for Consumers.  https://www.wearable-technologies.com/2016/01/the-most-successful-wearables-for-consumers/. (2016).  Accessed: 2018-09-015.
  * Park et al. (2015) Sungmee Park, Kyunghee Chung, and Sundaresan Jayaraman. 2015\.  Wearables: Fundamentals, advancements, and a roadmap for the future.  In Wearable sensors. Elsevier, US, 1–23.  https://doi.org/10.1016/B978-0-12-418662-0.00001-5
  * Rallapalli et al. (2014) Swati Rallapalli, Aishwarya Ganesan, Krishna Chintalapudi, Venkat N. Padmanabhan, and Lili Qiu. 2014\.  Enabling Physical Analytics in Retail Stores Using Smart Glasses. In Proceedings of the 20th Annual International Conference on Mobile Computing and Networking (MobiCom ’14). ACM, New York, NY, USA, 115–126.  https://doi.org/10.1145/2639108.2639126
  * Rauschnabel et al. (2015) Philipp A. Rauschnabel, Alexander Brem, and Bjoern S. Ivens. 2015\.  Who Will Buy Smart Glasses? Empirical Results of Two Pre-market-entry Studies on the Role of Personality in Individual Awareness and Intended Adoption of Google Glass Wearables.  Comput. Hum. Behav. 49, C (Aug. 2015), 635–647.  https://doi.org/10.1016/j.chb.2015.03.003
  * Rauschnabel and Ro (2016) Philipp A Rauschnabel and Young K Ro. 2016.  Augmented reality smart glasses: An investigation of technology acceptance drivers.  International Journal of Technology Marketing 11, 2 (2016), 123–148.
  * Rieger et al. (2018) Kendra L. Rieger, Christina H. West, Amanda Kenny, Lisa Demczuk, Rishma Chooniedass, Kim M. Mitchell, Joanne Chateau, and Shannon D. Scott. 2018.  Digital storytelling as a method in health research: a systematic review protocol.  Systematic Reviews 7, 1 (05 Mar 2018), 41\.  https://doi.org/10.1186/s13643-018-0704-y
  * Wright and Keith (2014) Robin Wright and Latrina Keith. 2014.  Wearable technology: If the tech fits, wear it.  Journal of Electronic Resources in Medical Libraries 11, 4 (2014), 204–216.
  * Xiong et al. (2017) Sijie Xiong, Sujie Zhu, Yisheng Ji, Binyao Jiang, Xiaohua Tian, Xuesheng Zheng, and Xinbing Wang. 2017.  iBlink: Smart Glasses for Facial Paralysis Patients. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys ’17). ACM, New York, NY, USA, 359–370.  https://doi.org/10.1145/3081333.3081343
  * Zhang et al. (2014) Lan Zhang, Xiang-Yang Li, Wenchao Huang, Kebin Liu, Shuwei Zong, Xuesi Jian, Puchun Feng, Taeho Jung, and Yunhao Liu. 2014.  It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses. In Proceedings of the 20th Annual International Conference on Mobile Computing and Networking (MobiCom ’14). ACM, New York, NY, USA, 91–102.  https://doi.org/10.1145/2639108.2639119
  * Zhao and Lindley (2014) Xuan Zhao and Siân E. Lindley. 2014.  Curation Through Use: Understanding the Personal Value of Social Media. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 2431–2440.  https://doi.org/10.1145/2556288.2557291


# How Many Answers Should I Give?
An Empirical Study of Multi-Answer Reading Comprehension

Chen Zhang1, Jiuheng Lin1, Xiao Liu1, Yuxuan Lai3,
Yansong Feng1,2, Dongyan Zhao1,4,5
1 Wangxuan Institute of Computer Technology, Peking University, China
2 The MOE Key Laboratory of Computational Linguistics, Peking University,
China
3 Department of Computer Science, The Open University of China
4 State Key Laboratory of Media Convergence Production Technology and Systems
5 Beijing Institute for General Artificial Intelligence
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding author.

###### Abstract

The multi-answer phenomenon, where a question may have multiple answers
scattered in the document, can be well handled by humans but is challenging
enough for machine reading comprehension (MRC) systems. Despite recent
progress in multi-answer MRC, there lacks a systematic analysis of how this
phenomenon arises and how to better address it. In this work, we design a
taxonomy to categorize commonly-seen multi-answer MRC instances, with which we
inspect three multi-answer datasets and analyze where the multi-answer
challenge comes from. We further analyze how well different paradigms of
current multi-answer MRC models deal with different types of multi-answer
instances. We find that some paradigms capture well the key information in the
questions while others better model the relationship between questions and
contexts. We thus explore strategies to make the best of the strengths of
different paradigms. Experiments show that generation models can be a
promising platform to incorporate different paradigms. Our annotations and
code are released for further research111https://github.com/luciusssss/how-
many-answers.

##  1 Introduction

In the typical setting of machine reading comprehension, such as SQuAD
Rajpurkar et al. (2016), the system is expected to extract a single answer
from the passage for a given question. However, in many scenarios, questions
may have multiple answers scattered in the passages, and all the answers
should be found to completely answer the questions, such as the examples
illustrated in Figure 1. Recently, a series of MRC benchmarks featuring multi-
answer instances have been constructed, including DROP Dua et al. (2019),
Quoref Dasigi et al. (2019) and MultiSpanQA Li et al. (2022). Most current
research efforts focus primarily on improving the overall QA performance on
these benchmarks Hu et al. (2019); Segal et al. (2020); Li et al. (2022). Yet,
as far as we know, there still lacks a systematic analysis of how the
phenomenon of multi-answer arises and how we can better tackle this challenge.

In this paper, we systematically analyze the categorization of multi-answer
MRC instances and investigate how to design a strong multi-answer MRC system.
We try to answer the following research questions: (1) Where does the multi-
answer challenge come from? (2) How do different MRC models specifically deal
with the multi-answer challenge? (3) How can we design better models by
combining different multi-answer MRC paradigms?

Figure 1: Two examples from existing multi-answer MRC datasets.

We first analyze existing multi-answer MRC datasets to track the origin of the
multi-answer challenge. Previous works have attempted to categorize multi-
answer instances primarily based on the distances or relationships between
multiple answers Li et al. (2022); Ju et al. (2022). Yet, they did not
holistically consider the interaction between questions and contexts. We
observe that in some cases the number of answers is indicated in the question
itself (_two players_ in Example A of Figure 1) while in others we have no
idea until we read the documents carefully (Example B of Figure 1).

To better understand this challenge, we develop a taxonomy for the multi-
answer phenomenon, based on how the number of answers is determined: the
question itself suffices, or both the question and the passage should be taken
into consideration. We annotate 6,857 instances from DROP, Quoref, and
MultiSpanQA based on our taxonomy and find that the procedure of dataset
construction has a large influence on the expressions in the questions. Most
questions in crowdsourced datasets contain certain clues indicating the number
of answers. By contrast, real-world information-seeking questions are less
likely to specify the number of answers, which is usually dependent on the
passages.

We further use our annotations to examine the performance of current MRC
solutions regarding the multi-answer challenge Hu et al. (2019); Segal et al.
(2020); Li et al. (2022), which can be categorized into 4 paradigms, i.e.,
Tagging, NumPred, Iterative and Generation. We analyze their strengths and
weaknesses and find that some efforts, e.g., NumPred, are good at capturing
the key information in the questions, while others, e.g., Iterative, can
better model the relation between questions and contexts. This motivates us to
investigate better ways to benefit from different paradigms.

Given the complementary nature of these paradigms, we wonder whether a
combination of paradigms improves performance on multi-answer MRC. We explore
two strategies, early fusion and late ensemble, to benefit from different
paradigms. With a generation model as the backbone, we attempt to integrate
the paradigms NumPred and Interative, in a lightweight Chain-of-Thought style
Wei et al. (2022). Experiments show that the integration remarkably improves
the performance of generation models, demonstrating that Generation is a
promising platform for paradigm fusion.

Our contributions are summarized as follows: (1) We design a taxonomy for
multi-answer MRC instances according to how the number of answers can be
determined. It considers both questions and contexts simultaneously,
enlightening where the multi-answer challenge comes from. (2) We annotate
6,857 instances from 3 datasets with our taxonomy, which enables us to examine
4 paradigms for multi-answer MRC in terms of their strengths and weaknesses.
(3) We explore various strategies to benefit from different paradigms.
Experiments show that generation models are promising to be backbones for
paradigm fusion.

##  2 Task Formulation

In multi-answer MRC, given a question $Q$ and a passage $P$, a model should
extract several spans, $A=\\{a_{1},a_{2},...,a_{n}\\}(n\geq 1)$, from $P$ to
answer $Q$. Each span, $a_{i}\in A$, corresponds to a partial answer to $Q$,
and the answer set $A$ as a whole answers $Q$ completely. These spans can be
contiguous or discontiguous in the passage.

We distinguish between two terms, multi-answer and multi-span, which are often
confused in previous works. Multi-answer indicates that a question should be
answered with the complete set of entities or utterances. Multi-span is a
definition from the perspective of answer annotations. In certain cases, the
answer annotation of a question can be either single-span or multi-span, as
explained in the next paragraph. Ideally, we expect that the answers to a
multi-answer question should be annotated as multi-span in the passage, where
each answer is grounded to a single span, although some of them can be
contiguous in the passage.

> Q0: What’s Canada’s official language?
> P: […] English and French, are the official languages of the Government of
> Canada. […]
>

For example, in Q0, there are two answers, English and French, to the given
question. According to the annotation guidelines of SQuAD, one might annotate
this instance with a single continuous span English and French. Yet, this form
of annotation is not preferred in the multi-answer MRC setting. It blurs the
boundary of different answers and fails to denote explicitly the number of
expected answers. Thus, it is suboptimal for a comprehensive model evaluation.
Instead, we suggest denoting each answer with distinct spans, say, annotating
this instance with two spans, English and French. With this criterion, we can
encourage models to disentangle different answers. With fine-grained answer
annotations, we can also assess how well a model answers a question
sufficiently and precisely.

This annotation criterion generally conforms to the annotation guidelines of
existing multi-answer datasets, e.g., DROP, Quoref and MultiSpanQA. A few
instances violating the criterion are considered as bad annotations, as
discussed in Section 4.2. See more remarks on the task formulation in Appendix
A.

##  3 Taxonomy of Multi-Answer MRC

To better understand the challenge of multi-answer, we first design a taxonomy
to categorize various multi-answer MRC instances. It assesses how the number
of answers relates to the question or passage provided. Different from the
previous works that classify questions according to the distances or relations
between multiple answers Li et al. (2022); Ju et al. (2022), our taxonomy,
taking both questions and passages into consideration, focuses on how the
number of answers is determined. This enables us to analyze multi-answer
questions and single-answer questions in a unified way. We illustrate our
taxonomy in Figure 2 and elaborate on each category as follows.

Figure 2: Illustration of our taxonomy for multi-answer MRC instances.  Type |  Question |  # Ans.
---|---|---
Cardinal |  Which two players completed 1-yard TD pass? |  2
Ordinal |  Who scored the first touchdown of the game? |  1
Comp./Super. |  What’s the largest pizza chain in America? |  1
Alternative |  Is San Juan Bautista incorporated or unincorporated? |  1
Other Semantics |  What are the first names of the trio who try to call 911? |  3
Table 1: Examples of various types of clue words. Comp./Super. denotes
comparatives and superlatives.

#### Question-Dependent

If one can infer the exact number of answers from the question without
referring to the passage, this instance belongs to the question-dependent
category. According to whether there are clue words that directly indicate the
number of answers, this type is further divided into two sub-categories:

(a) In a with-clue-words question, one can find a few words that indicate the
number of answers. In Q1, the word two in the question indicates that two
answers are expected.

> Q1: What are the two official languages of Puerto Rico?
> P: […] English is an official language of the Government of Puerto Rico. […]
> As another official language, Spanish is widely used in Puerto Rico. […]
>

We group the clue words into five types: cardinal, ordinal,
comparative/superlative, alternative, and other lexical semantics, as
illustrated in Table 1.

(b) In a without-clue-words question, although we can not locate obvious clue
words, we can infer the number of answers with sentence semantics or
commonsense knowledge. In Q2, we can determine that there is only one
conversion result for the question based on sentence semantics instead of any
single words.

> Q2: 1 light year equal to how many km?
> P: […] The light-year is a unit of length used to express astronomical
> distances. It is about 9.5 trillion kilometres or 5.9 trillion miles. […]
>

In Q3, we can infer that the following question has only one answer, based on
the commonsense that there is only one winner of a given Super Bowl.

> Q3: Who won Super Bowl XXXIX?
> P: […] The Eagles advanced to Super Bowl XXXIX, where they dueled the 2004
> New England Patriots season. […] The Patriots won 24-21. […]
>

#### Passage-Dependent

In a passage-dependent instance, the question itself is not adequate to infer
the number of answers. One needs to rely on the provided passage to decide how
many answers are needed to answer the question. In Q4, we have no idea of the
number of answers solely based on the question. If we refer to the passage, we
will find ten answers to the question.

> Q4: Which countries does the Danube River flow through?
> P: […] Originating in Germany, the Danube flows southeast for 2,850 km,
> passing through or bordering Austria, Slovakia, Hungary, Croatia, Serbia,
> Romania, Bulgaria, Moldova and Ukraine before draining into the Black Sea.
> […]
>

##  4 Analyses of Multi-Answer Datasets

We investigate existing multi-answer datasets based on our designed taxonomy
to analyze where the multi-answer challenge comes from.

###  4.1 Datasets

We annotate the validation sets of three widely-used multi-answer MRC
datasets, i.e., DROP Dua et al. (2019), Quoref Dasigi et al. (2019), and
MultiSpanQA Li et al. (2022). The number of annotated questions is listed in
Table 2 and more statistics are in Appendix B.

DROP is a crowdsourced MRC dataset for evaluating the discrete reasoning
ability. The annotators are encouraged to devise questions that require
discrete reasoning such as arithmetic. DROP has four answer types: numbers,
dates, single spans, and sets of spans. Since the previous two types of
answers are not always exact spans in the passages, we only consider the
instances whose answers are single spans or sets of spans.

Quoref focuses on the coreferential phenomena. The questions are designed to
require resolving coreference among entities. 10% of its instances require
multiple answer spans.

MultiSpanQA is a dataset specialized for multi-span reading comprehension. The
questions are extracted from NaturalQuestions Kwiatkowski et al. (2019), which
are real queries from the Google search engine.

Dataset | All | Single-Ans. | Multi-Ans.
---|---|---|---
DROP | 3,133 | 2,609 | 524
Quoref | 2,418 | 2,198 | 220
MultiSpanQA | 1,306 | 653 | 653
Total | 6,857 | 5,460 | 1,397
Table 2: The number of instances for human annotation in the validation set of
each dataset.

###  4.2 Annotation

#### Annotation Process

Our annotation process is two-staged: we first automatically identify some
question-dependent instances and then recruit annotators to classify the
remaining ones.

In the first stage, we automatically identify the questions containing certain
common clue words such as numerals (full list in Appendix B) to reduce the
workload of whole-process annotation. Afterward, the annotators manually check
whether each instance is question-dependent. Out of the 4,594 recalled
instances, 3,727 are identified as question-dependent.

In the second stage, we recruit annotators to annotate the remaining 3,130
instances. For each instance, given both the question and the answers, the
annotators should first check whether the form of answers is correct and mark
incorrect cases as bad-annotation222In the first stage, the annotators also
need to check whether an instance is bad-annotation. . We show examples of
common bad-annotation cases in Table 10. After filtering out the bad-
annotation ones, the annotators are presented with the question only and
should decide whether they could determine the number of answers solely based
on the question. If so, this instance is annotated as question-dependent;
otherwise passage-dependent. For a question-dependent instance, the annotators
are further asked to extract the clue words, if any, from the question, which
determines whether the instance is with-clue-words or without-clue-words.

#### Quality Control

Six annotators participated in the annotation after qualification. Each
instance is annotated by two annotators. In case of any conflict, a third
annotator resolves it. An instance is classified as bad-annotation if any
annotator labels it as bad-annotation. Cohen’s Kappa between two initial
annotators is 0.70, indicating substantial agreement. See more details in
Appendix B.

###  4.3 Analyses of Annotation Results

Dataset | passage-dependent | question-dependent | bad-annotation
---|---|---|---
|  | All | with-clue-word | no-clue-word |
DROP | 826 (26.4%) | 2,242 (71.6%) | 2,204 (70.3%) | 38 (1.2%) | 65 (2.1%)
Quoref | 711 (29.4%) | 1,704 (70.5%) | 1,639 (67.8%) | 65 (2.7%) | 3 (0.2%)
MultiSpanQA | 991 (75.9%) | 285 (21.8%) | 121 (9.3%) | 164 (12.6%) | 30 (2.3%)
Total | 2,528 (36.9%) | 4,231 (61.7%) | 3,964 (57.8%) | 267 (3.9%) | 98 (1.4%)
Table 3: Distribution of instance types in three datasets. Dataset | with-clue-word | Cardinal | Ordinal | Comp./Super. | Alternative | Other Semantics
---|---|---|---|---|---|---
DROP | 2,204 | 113 (5.1%) | 592 (26.9%) | 1,298 (58.9%) | 1,214 (55.1%) | 135 (6.1%)
Quoref | 1,639 | 83 (5.1%) | 35 (2.1%) | 25 (1.5%) | 0 (0.0%) | 1,501 (91.6%)
MultiSpanQA |  121 | 51 (41.8%) | 26 (21.3%) | 23 (19.0%) | 2 (1.6%) | 19 (15.6%)
Table 4: Distribution of clue word types in three datasets. A question may
contain multiple types of clue words.

With our annotated data, we study how the multi-answer instances differ across
different datasets under our designed taxonomy. We find that the distributions
of instance types are closely related to how the datasets are constructed.

#### Instance Types

The distributions of instance types in different datasets are shown in Table
3. Question-dependent prevails in DROP and Quoref, making up over 70% of the
two datasets. In contrast, most instances in MultiSpanQA are passage-
dependent. This difference stems from how the questions are collected.

DROP and Quoref use crowdsourcing to collect questions with specific
challenges. Given a passage, the annotators know the answers in advance and
produce questions that can only be answered through certain reasoning skills.
These artificial questions are more likely to contain clues to the number of
answers, such as the question with ordinal in Table 1. By contrast, the
questions in MultiSpanQA are collected from search engine queries. Users
generally have no idea of the answers to the queries. The number of answers,
as a result, is more often dependent on the provided passages, such as Q4 in
Section 3.

#### Clue Words

Since a large portion (57.8%) of the annotated instances belong to the with-
clue-word type, we further investigate the distribution of clue words in
different datasets, shown in Table 4. On the one hand, the questions contain a
large variety of clue words, demonstrating the complexity of multi-answer MRC.
On the other hand, the prevailing type of clue words is different in each
dataset, reflecting the preference in dataset construction. Specifically,
nearly 60% of the with-clue-word questions in DROP are alternative questions
with comparatives/superlatives, because DROP’s annotators are encouraged to
inject discrete reasoning challenges, e.g., comparison, when writing
questions. In Quoref, 91% of the clue words indicate the number of answers
through their lexical semantics. This unbalanced distribution results from the
emphasis on coreference resolution: most questions begin with what is the name
of the person who …, where name of the person is identified as clue words. In
MultiSpanQA, whose questions are search engine queries, 63% of the with-clue-
word questions contain numerals. If users already know the number of desired
answers, they tend to restrict it in the question, such as seven wonders of
the world.

We provide more analyses on of how the instance types are distributed with
respect to the specific number of answers in Appendix C.

##  5 Existing Multi-Answer MRC Models

Based on our categorization of the multi-answer instances, we continue to
investigate how existing multi-answer MRC models perform differently on
various types of multi-answer instances. We summarize current solutions into
four paradigms according to how they obtain multiple answers, as illustrated
in Figure 3.

Figure 3: An illustration of four paradigms for multi-answer MRC.

#### Tagging

Segal et al. (2020) cast the multi-answer MRC task as a sequence tagging
problem, similar to named entity recognition (NER), so that the model can
extract multiple non-contiguous spans from the context.

####  NumPred (Number Prediction)

Hu et al. (2019) first predict the number of answers $k$ as an auxiliary task
and then select the top $k$ non-overlapped ones from the output candidate
spans.

#### Iterative

Searching for evidence iteratively is widely adopted in many QA tasks Xu et
al. (2019); Zhao et al. (2021); Zhang et al. (2021), but it is not explored in
multi-answer MRC. We adapt this idea to extract multiple answers iteratively.
In each iteration, we append the previously extracted answers to the question,
with the word except in between, and then feed the updated question to a
single-answer MRC model. The iterative process terminates when the model
predicts no more answers.

#### Generation

Generation has been adopted as a uniform paradigm for many QA tasks Khashabi
et al. (2020, 2022), but it is less explored on multi-answer MRC. For
Generation, we concatenate all answers, with semicolons as separators, to form
an output sequence, and finetune the model to generate it conditioned on the
question and passage.

###  5.1 Experimental Setup

#### Implementation Details

We use RoBERTa-base Liu et al. (2019) for the three extractive paradigms and
BART-base Lewis et al. (2020) for Generation. We train models on the training
sets of each dataset and evaluate them on the corresponding validation sets
with our instance type annotations. See more details in Appendix D.1.

#### Metrics

We adopt the official metrics of MultiSpanQA Li et al. (2022), including the
precision (P), recall (R), and F1 in terms of exact match (EM) and partial
match (PM). See Appendix D.2 for details.

Model | EM | PM
---|---|---
| P | R | F1 | P | R | F1
DROP
Tagging | 61.86 | 63.91 | 62.87 | 77.53 | 77.39 | 77.46
NumPred | 61.59 | 56.77 | 59.09 | 76.71 | 74.86 | 75.77
Iterative | 60.66 | 60.07 | 60.36 | 76.19 | 76.04 | 76.11
Generation | 60.07 | 57.15 | 58.58 | 75.39 | 72.39 | 73.86
Quoref
Tagging | 71.00 | 72.21 | 71.60 | 80.44 | 79.74 | 80.09
NumPred | 65.61 | 63.57 | 64.57 | 77.30 | 78.20 | 77.75
Iterative | 67.28 | 66.35 | 66.81 | 78.57 | 78.58 | 78.57
Generation | 63.57 | 63.39 | 63.48 | 73.38 | 74.02 | 73.70
MultiSpanQA
Tagging | 61.31 | 68.84 | 64.85 | 80.45 | 83.08 | 81.75
NumPred | 55.03 | 46.06 | 50.15 | 80.16 | 75.26 | 77.63
Iterative | 66.32 | 67.98 | 67.14 | 84.39 | 80.96 | 82.64
Generation | 65.40 | 62.60 | 63.97 | 82.06 | 78.14 | 80.06
Table 5: Performance of four paradigms on three datasets. Model | p-dep. | q-dep.
---|---|---
|  |  All | w/-clue | w/o-clue
DROP
Tagging | 74.57 | 79.11 | 80.88 | 68.77
NumPred | 72.37 | 77.54 | 79.32 | 70.08
Iterative | 73.47 | 77.60 | 79.21 | 65.73
Generation | 72.18 | 74.77 | 76.19 | 72.62
Quoref
Tagging | 70.60 | 84.86 | 85.23 | 75.76
NumPred | 69.45 | 81.88 | 82.44 | 70.12
Iterative | 71.42 | 82.18 | 82.37 | 77.30
Genration | 66.31 | 77.41 | 78.38 | 52.63
MultiSpanQA
Tagging | 82.28 | 79.66 | 86.60 | 73.36
NumPred | 77.77 | 77.11 | 78.19 | 78.77
Iterative | 82.78 | 82.09 | 87.22 | 77.80
Generation | 80.57 | 78.05 | 81.73 | 75.85
Table 6: The performance (PM F1) of four paradigms on different types of
instances. p-dep. denotes passage-dependent. q-dep. denotes question-
dependent.

###  5.2 Results and Analyses

We report the overall performance in Table 5, and the performance on different
instance types in Table 6. We observe that each of these paradigms has its own
strengths and weaknesses.

Tagging outperforms other paradigms on DROP and Quoref, whose dominating
instance type is question-dependent. Although Tagging has no explicit answer
number prediction step, it can still exploit this information implicitly
because it takes the question into account during the sequential processing of
every token. Besides, Tagging, as a common practice for entity recognition, is
good at capturing the boundaries of entities. Thus, it is not surprising that
it performs the best on DROP and Quoref, most of whose answers are short
entities.

Iterative achieves the best overall performance on MultiSpanQA, whose
prevailing instance type is passage-depenent. This paradigm does not directly
exploit the information of the number of answers given in the question.
Rather, it encourages adequate interactions between questions and passages,
performing single-answer extraction at each step. As a result, Iterative does
well for the questions whose number of answers heavily depends on the given
context.

As for NumPred, although we expect high performance on question-dependent
instances, it lags behind Tagging by approximately 2% in PM F1 on DROP and
Quoref. This might result from the gap between training and inference. The
model treats the answer number prediction and answer span extraction as two
separate tasks during training, with limited interaction. Yet during
inference, the predicted number of answers is used as a hard restriction on
multi-span selection. Different from the decent performance on DROP and
Quoref, NumPred performs worst among the four paradigms on MultiSpanQA,
because it is difficult for models to accurately predict the number of answers
for a long input text that requires thorough understanding.

Among all paradigms, Generation generally performs the worst. Under the same
parameter scale, extractive models seem to be the better choice for tasks
whose outputs are exact entity spans from the input, while generation models
do well in slightly longer answers. This also explains the smaller gap between
Generation and extractive paradigms on MultiSpanQA compared to that on DROP
and Quoref: MultiSpanQA has many descriptive long answers instead of short
entities only.

Figure 4: An illustration of different strategies for early fusion of
paradigms.

##  6 Fusion of Different Paradigms

From the above analysis, we can see that extractive methods can better locate
exact short spans in the passage, and NumPred can provide potential guidance
on the number of answers. Meanwhile, the generation models can better handle
longer answers and are more adaptable to different forms of inputs and
outputs. Now an interesting question is how to combine different paradigms to
get the best of both worlds.

We explore two strategies for combining different paradigms: early fusion and
late ensemble. The former mixes multiple paradigms in terms of model
architectures while the latter ensembles the predictions of different models.
We discuss our exploration of late ensemble in Appendix E.1 since model
ensemble is a well-explored technique. Here we primarily elaborate on early
fusion. We carry out a series of pilot studies to demonstrate the potential of
paradigm fusion.

Previous works attempt to fuse two extractive paradigms, Tagging and NumPred
Segal et al. (2020); Li et al. (2022). However, they only lead to marginal
improvements, probably because Tagging can already implicitly determine answer
numbers well and the help of NumPred is thus limited.

Although the performance of base-size generation models on multi-answer MRC is
inferior to that of extractive ones, generation models of larger sizes show
great potential with more parameters and larger pre-training corpora Khashabi
et al. (2020, 2022). More importantly, Generation can easily adapt to various
forms of inputs and outputs. We carry out pilot studies using a generation
model as the backbone and benefiting from the ideas of other paradigms. We
propose several lightweight methods to combine Generation with NumPred and
Iterative, as illustrated in Figure 4.

####  Generation \+ NumPred

Inspired by recent works on Chain-of-Thought Wei et al. (2022), we guide the
model with prompts indicating the number of answers. We introduce a NumPred
prompt sentence (NPS) in the form of There are $\\{2,3,...\\}$ answers/There
is only one answer. We experiment with two variants, multitask and pipeline.
In the multitask variant, the model outputs an NPS before enumerating all the
answers. In the pipeline variant, we predict the number of answers with a
separate classifier and then append the NPS to the question as extra guidance.

####  Generation \+ Iterative

We substitute the original extractor of Iterative with a generator. The
iterative process terminates when the model outputs the string No answer.
Besides the normal setting, we experiment with another variant that
additionally outputs an NPS in the form of The number of remaining answers is
$\\{1,2,3,...\\}$.

#### Results

Our main experiments are conducted with BART-base and BART-large due to our
limited computational budget. For the pipeline variant of Generation \+
NumPred, we use RoBERTa-base as an answer number classifier. The overall
experiment results are reported in Table 7 and the results on different
question types are reported in Appendix E.2.

When Generation is multitasking with NumPred, it outperforms the vanilla one
consistently. The NPS in the output provides a soft but useful hint for the
succeeding answer generation, improving the accuracy of answer number
prediction by 1.7% on average for BART-base. The pipeline variant is often
inferior to the multitasking one due to error propagation. Especially, its
performance drops a lot on MultiSpanQA, whose instances are passage-dependent.
The accuracy of the answer number classifier on MultiSpanQA lags behind that
on the other two datasets by more than 12%. Thus the NPS in the input, with an
unreliably predicted answer number, is more likely to mislead the subsequent
answer span generation.

The combination of Generation and Iterative does not always lead to
improvement. This might be because the answer generation process of Generation
is already in an iterative style: in the output sequence, each answer is
generated conditioned on the previously-generated ones. The incorporation of
Iterative thus does not lead to further improvement. When we further introduce
an NPS with the number of remaining answers, the performance generally
outperforms the normal setting. This proves that Generation, as a backbone, is
easy to integrate with various hints.

Model | Base | Large
---|---|---
EM | PM | EM | PM
DROP
Vanilla Generation | 58.58 | 73.86 | 66.43 | 80.55
+NumPred (multitask) | 60.02 | 74.34 | 69.61 | 82.85
+NumPred (pipeline) | 59.19 | 73.94 | 66.45 | 80.63
+Iterative (normal) | 58.44 | 73.58 | 66.55 | 80.53
+Iterative (number) | 58.98 | 74.07 | 68.19 | 82.17
Quoref
Vanilla Generation | 63.48 | 73.70 | 76.57 | 84.47
+NumPred (multitask) | 66.25 | 75.43 | 77.04 | 84.45
+NumPred (pipeline) | 67.94 | 77.42 | 75.42 | 83.66
+Iterative (normal) | 68.81 | 78.23 | 74.72 | 82.60
+Iterative (number) | 63.33 | 73.34 | 76.67 | 84.57
MultiSpanQA
Vanilla Generation | 63.97 | 80.06 | 69.13 | 84.61
+NumPred (multitask) | 64.85 | 80.58 | 69.31 | 84.82
+NumPred (pipeline) | 39.71 | 60.94 | 45.34 | 68.09
+Iterative (normal) | 63.26 | 79.97 | 65.62 | 82.88
+Iterative (number) | 63.84 | 80.04 | 66.77 | 83.41
Table 7: The performance (EM F1 and PM F1) of different strategies for early fusion of paradigms.  Model | Setting | EM F1 | PM F1
---|---|---|---
Vanilla BART-base | Supervised | 66.77 | 81.24
Vanilla BART-large | Supervised | 71.93 | 85.83
Vanilla GPT-3.5 | One-Shot | 53.34 | 79.27
GPT-3.5 + NumPed | One-Shot | 63.45 | 82.38
Table 8: The performance of BART and GPT-3.5 on the multi-answer instances of
MultiSpanQA.

#### Pilot Study on GPT-3.5

To investigate whether these fusion strategies work on larger models, we
conduct a pilot study on GPT-3.5. We use the 653 multi-answer instances in the
validation set of MultiSpanQA for experiments. The prompts are listed in
Appendix E.2. The experiment results are shown in Table 8.

When given only one example for in-context learning, GPT-3.5 can already
achieve 79.27% PM F1 on the multi-answer instances, with only a small gap
between BART trained on full data. Its EM F1 score is low because GPT-3.5
cannot handle the boundaries of answer spans well. This is not unsurprising
since one example is not sufficient for GPT-3.5 to learn the annotation
preference of span boundaries in MultiSpanQA. If we ask GPT-3.5 to predict the
number of answers before giving all the answers, we observe an improvement of
10.1% EM F1 and 3.1% PM F1. This proves the effectiveness of fusing NumPed
with larger generation models

As evidenced by the above trials, it is promising to fusion different
paradigms. We hope that our exploration will inspire future works adopting
larger generation models for multi-answer MRC.

##  7 Related Works

Compared to the vast amount of single-answer MRC datasets, the resources for
multi-answer MRC are limited. Aside from the datasets in Section 4.1, MASH-QA
Zhu et al. (2020) focuses on the healthcare domain, with 27% of the questions
having multiple long answers, ranging from phrases to sentences. CMQA Ju et
al. (2022) is another multi-answer dataset in Chinese, featuring answers with
conditions or different granularities. For our analysis, we select two
commonly-used datasets, DROP and Quoref, as well as a newly-released dataset,
MultiSpanQA.

Current models addressing multi-answer MRC generally fall into two paradigms:
Tagging Segal et al. (2020) and NumPred Hu et al. (2019), as explained in
Section 5. Iterative Xu et al. (2019); Zhao et al. (2021); Zhang et al.
(2021); Gao et al. (2021) and Generation Khashabi et al. (2020, 2022) have
been adopted for many types of QA tasks including knowledge base QA, multiple-
choice QA, and open-domain QA. Nevertheless, their performance on multi-answer
MRC is less explored. In our paper, we also study how to adapt these paradigms
for multi-answer MRC. Apart from the exploration of model architectures for
multi-answer MRC, Lee et al. (2023) attempt to generate multi-answer questions
as data augmentation.

Previous works have made preliminary attempts in fusing two extractive
paradigms. Segal et al. (2020) adopt a single-span extraction model for
single-answer questions and Tagging for multi-answer questions; Li et al.
(2022) add a NumPred head to the Tagging framework. The predicted number of
answers is used to adjust the tagging results. Both strategies lead to
marginal improvement over the baselines. We instead resort to Generation for
paradigm fusion, considering its potential with larger sizes and its
flexibility in inputs and outputs.

##  8 Conclusion

In this paper, we conduct a systematic analysis for multi-answer MRC. We
design a new taxonomy for multi-answer instances based on how the number of
answers is determined. We annotate three datasets with the taxonomy and find
that multi-answer is not merely a linguistic phenomenon; rather, many factors
contribute to it, especially the process of data collection. With the
annotation, we further investigate the performance of four paradigms for
multi-answer MRC and find their strengths and weaknesses. This motivates us to
explore various strategies of paradigm fusion to boost performance. We believe
that our taxonomy can help determine what types of questions are desirable in
the annotation process and aid in designing more practical annotation
guidelines. We hope that our annotations can be used for more fine-grained
diagnoses of MRC systems and encourage more robust MRC models.

## Limitations

First, our taxonomy of multi-answer MRC instances only considers whether we
know the exact number of answers from the questions. In some cases, one might
have an imprecise estimate of answer numbers from the question. For example,
for the question Who are Barcelona’s active players?, one might estimate that
there are dozens of active players for this football club. Yet, these
estimations are sometimes subjective and difficult to quantify. Therefore,
this instance is classified as passage-dependent according to our current
taxonomy. We will consider refining our taxonomy to deal with these cases in
the future.

Second, we did not conduct many experiments with pre-trained models larger
than the large-size ones due to limited computational budgets. Generation
models of larger sizes show great potential with more parameters and larger
pre-training corpora. We encourage more efforts to deal with multi-answer MRC
with much larger models, such as GPT-3.5.

## Acknowledgments

This work is supported by NSFC (62161160339). We would like to thank the
anonymous reviewers for their valuable suggestions, and our great annotators
for their careful work, especially Zhenwei An, Nan Hu, and Hejing Cao. Also,
we would like to thank Quzhe Huang for his help in this work. For any
correspondence, please contact Yansong Feng.

## References

  * Dasigi et al. (2019) Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019.  Quoref: A reading comprehension dataset with questions requiring coreferential reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5925–5932, Hong Kong, China. Association for Computational Linguistics.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2368–2378, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2021) Yifan Gao, Henghui Zhu, Patrick Ng, Cicero Nogueira dos Santos, Zhiguo Wang, Feng Nan, Dejiao Zhang, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021\.  Answering ambiguous questions through generative evidence fusion and round-trip prediction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3263–3276, Online. Association for Computational Linguistics.
  * Hu et al. (2019) Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. 2019.  A multi-type multi-span network for reading comprehension that requires discrete reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1596–1606, Hong Kong, China. Association for Computational Linguistics.
  * Jacobs et al. (1991) Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991\.  Adaptive mixtures of local experts.  _Neural computation_ , 3(1):79–87.
  * Ju et al. (2022) Yiming Ju, Weikang Wang, Yuanzhe Zhang, Suncong Zheng, Kang Liu, and Jun Zhao. 2022\.  CMQA: A dataset of conditional question answering with multiple-span answers.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 1697–1707, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Khashabi et al. (2022) Daniel Khashabi, Yeganeh Kordi, and Hannaneh Hajishirzi. 2022.  Unifiedqa-v2: Stronger generalization via broader cross-format training.  _arXiv preprint arXiv:2202.12359_.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  UNIFIEDQA: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1896–1907, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lee et al. (2023) Seongyun Lee, Hyunjae Kim, and Jaewoo Kang. 2023.  Liquid: A framework for list question answering dataset generation.  _arXiv preprint arXiv:2302.01691_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2022) Haonan Li, Martin Tomko, Maria Vasardani, and Timothy Baldwin. 2022.  MultiSpanQA: A dataset for multi-span question answering.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1250–1260, Seattle, United States. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Segal et al. (2020) Elad Segal, Avia Efrat, Mor Shoham, Amir Globerson, and Jonathan Berant. 2020.  A simple and effective model for answering multi-span questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3074–3080, Online. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2019) Kun Xu, Yuxuan Lai, Yansong Feng, and Zhiguo Wang. 2019.  Enhancing key-value memory neural networks for knowledge based question answering.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2937–2947, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Zhang et al. (2021) Chen Zhang, Yuxuan Lai, Yansong Feng, and Dongyan Zhao. 2021.  Extract, integrate, compete: Towards verification style reading comprehension.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2976–2986, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhao et al. (2021) Chen Zhao, Chenyan Xiong, Jordan Boyd-Graber, and Hal Daumé III. 2021.  Multi-step reasoning over unstructured text with beam dense retrieval.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4635–4641, Online. Association for Computational Linguistics.
  * Zhu et al. (2020) Ming Zhu, Aman Ahuja, Da-Cheng Juan, Wei Wei, and Chandan K. Reddy. 2020.  Question answering with long multiple-span answers.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3840–3849, Online. Association for Computational Linguistics.

##  Appendix A Additional Remarks on Task Formulation

As discussed in Section 2, multi-answer and multi-span are two orthogonal
concepts. We have already shown an example (Q0 in Section 2) where a multi-
answer question can be annotated as single-span by certain annotation
guidelines. Here is another example to demonstrate the difference between
multi-answer and multi-span.

> Q: Which offer of Triangle-Transit is most used by students?
> P: […] Triangle-Transit offers scheduled, fixed-route regional and commuter
> bus service. The first is most used by students.
>

This is an example where a single-answer question can be annotated as multi-
span. A single answer, scheduled bus service, will be annotated as multiple-
span, i.e., scheduled and bus service in the passage.

Considering the differences between multi-answer and multi-span, we suggest
carefully distinguishing between these two terms in the future.

##  Appendix B Annotation Details

#### Dataset Statistics

We report more statistics of the annotated datasets in Table 9. MultiSpanQA
has the largest average number of answers since it is a dataset designed
especially for multi-answer questions. The answers in MultiSpanQA are
generally longer than those in DROP and Quoref because many of the answers in
MultiSpanQA are long descriptive phrases or clauses instead of short entities.
For all three datasets, the distances between answers are large. This
indicates that the answers to a large proportion of the questions are
discontiguous in the passages, demonstrating the difficulty of multi-answer
MRC.

Dataset | DROP | Quoref | MultiSpanQA
---|---|---|---
Length of Question | 9.4 | 15.5 | 9.0
Length of Context | 214.7 | 326.0 | 219.9
Length of Answer | 1.9 | 1.6 | 3.1
#Answers | 1.2 | 1.1 | 1.9
#Answers (Multi) | 2.5 | 2.4 | 2.9
Distance Between Ans. | 30.5 | 17.3 | 10.3

Table 9: Dataset Statistics, including the (a) average length (in words) of
questions, contexts, and answers, (b) the average number of answers for all
the instances and the multi-answer ones, (c) the average distances (in words)
between answers.

#### Pre-defined Clue Words

Here, we list the pre-defined clue words in the first stage of annotation:

  * •

Numerals, including cardinals and ordinals

  * •

Comparatives and superlatives

  * •

The word or, as an indicator of alternative questions.

  * •

Other words, including only, last, single, name of the person, and, top.

#### Selection of Annotators

A total of 10 graduates proficient in English participated in our annotation
task. We first provided training materials to the annotators and asked them to
annotate 100 sample instances. Based on their annotation accuracy on the
sample instances, six of them are qualified to continue annotating the
remaining instances. The annotators are paid $10 per hour, which is adequate
given the participants’ demographic. The annotators are informed of how the
data would be used.

#### Examples of Bad Annotations

In Table 10, we present several examples we marked as bad-annotation. Common
reasons for bad annotations including incorrect segmentation of answers,
irrelevant answers, and duplicate answers.

Type |  Example |  Explanation
---|---|---
Incorrect segmentation of answers |  Source:  DROP Question: Which event occurred first, Duke Magnus Birgersson started a war or Erik Klipping gathered a large army? Annotated Answers: Duke Magnus Birgersson; started a war |  The correct answer Duke Magnus Birgersson started a war is wrongly split into two spans, Duke Magnus Birgersson and started a war.
Irrelevant Answers |  Source:  DROP Question: Who scored first in the second half of the game, Cowboys or 49ers? Annotated Answers: end of the half; San Francisco scored; making the score 28-14 |  All three annotated answers are not related to the questions. A correct answer should be either Cowboys or 49ers.
Duplicate answers |  Source:  MultiSpanQA Question: who benefited by title ix of the education amendments Annotated Answers: women; women playing college sports |  One annotead answer, women is duplicated with the other, women playing college sports.
Table 10: Examples and explanations of bad-annotation cases.

##  Appendix C Additional Analyses on Annotation Results

We report more statistics of the annotation results in Table 11 and Table 12,
and conduct additional analyses from the perspective of the number of answers.

For multi-answer instances, passage-dependent questions account for the
largest proportion, followed by with-clue-word. As for the single-answer
instances in DROP and Quoref, they tend to be question-dependent, while in
MultiSpanQA most of them are passage-dependent. In terms of the clue words in
the with-clue-word questions, cardinal numbers are more common in multi-answer
questions while other types of clue words are more likely to appear in single-
answer questions.

#Ans | p-dep. |  q-dep.
---|---|---
|  | w/-clue | w/o-clue
DROP
1 | 480 | 2,085 | 37
2 | 209 | 105 | 1
3 | 74 | 11 | 0
>3 | 63 | 3 | 0
Quoref
1 | 582 | 1,548 | 65
2 | 82 | 62 | 0
3 | 28 | 23 | 0
>3 | 19 | 6 | 0
MultiSpanQA
1 | 448 | 56 | 140
2 | 300 | 32 | 22
3 | 131 | 14 | 2
>3 | 112 | 19 | 0

Table 11: Distribution of question types according to the number of answers.
p-dep. denotes passage-dependent. q-dep. denotes question-dependent.

#Ans | Alternative | Cardinal | Comp./Super. | Ordinal | Others
---|---|---|---|---|---
DROP
1 | 1,213 | 3 | 1,293 | 588 | 132
2 | 1 | 97 | 4 | 3 | 3
3 | 0 | 11 | 0 | 0 | 0
>3 | 0 | 2 | 1 | 1 | 0
Quoref
1 | 0 | 1 | 25 | 35 | 1,492
2 | 0 | 55 | 0 | 0 | 7
3 | 0 | 21 | 0 | 0 | 2
>3 | 0 | 6 | 0 | 0 | 0
MultiSpanQA
1 | 2 | 1 | 14 | 25 | 14
2 | 0 | 21 | 5 | 1 | 5
3 | 0 | 12 | 2 | 0 | 0
>3 | 0 | 17 | 2 | 0 | 0

Table 12: Distribution of clue word types in three datasets according to the
number of answers.

##  Appendix D Experimental Setup

###  D.1 Implementation Details

We use base-size models for our main experiments for sake of energy savings.
Since T5-base has twice as many parameters as RoBERTa-base and BART-base, we
did not use it to ensure fair comparisons. We carefully tune each model on the
training set and report its best performance on the validation set. We use an
NVIDIA A40 GPU for experiments. A training step takes approximately 0.5s for
RoBERTa-base and 0.2s for BART-base. We describe the implementation details of
different models here.

#### Tagging

We use the implementation by Segal et al.
(2020)333https://github.com/eladsegal/tag-based-multi-span-extraction. We use
the IO tagging variant, which achieves the best overall performance according
to the original paper. We adopt the best-performing hyperparameters provided
by the original paper.

#### NumPred

Because the implementation by the original paper Hu et al.
(2019)444https://github.com/huminghao16/MTMSN does not support RoBERTa, we re-
implement the model with Huggingface Transformers Wolf et al.
(2020)555https://github.com/huggingface/transformers. We use the
representation of the first token in the input sequence for answer number
classification. The maximum number of answers of the classifier is 8. The
batch size is 12. The number of training epochs is 10. The learning rate is
3e-5. The maximum sequence length is 512.

#### Iterative

Our implementation is based on the scripts of MRC implemented by Huggingface.
During training, the order of answers for each iteration is determined by
their order of position in the passage. The batch size is 8. The number of
training epochs is 8. The learning rate is 3e-5. The maximum sequence length
is 384. During inference, the beam size is set to 3 and the length penalty is
set to 0.7. The maximum length of answers is 10.

#### Generation

Our implementation is based on the scripts of sequence generation implemented
by Huggingface. The batch size is 12. The learning rate is 3e-5. The number of
training epochs is 10. The maximum input length is 384. The maximum output
length is 60.

###  D.2 Evaluation Metrics

Here, we describe the evaluation metrics used in our experiments, which are
the official ones used by MultiSpanQA Li et al. (2022). The metrics consist of
two part: exact match and partial match.

#### Exact Match

An exact match occurs when a prediction fully matches one of the ground-truth
answers. We use micro-averaged precision, recall, and F1 score for evaluation.

#### Partial Match

For each pair of prediction $p_{i}$ and ground truth answer $t_{j}$ , the
partial retrieved score $s_{ij}^{ret}$ and partial relevant score
$s_{ij}^{rel}$ are calculated as the length of the longest common substring
(LCS) between $p_{i}$ and $t_{j}$, divided by the length of $p_{i}$ and
$t_{j}$ respectively, as:

$s_{ij}^{ret}=\frac{\mathrm{len}(\mathrm{LCS}(p_{i},t_{j}))}{\mathrm{len}(p_{i})}$
$s_{ij}^{rel}=\frac{\mathrm{len}(\mathrm{LCS}(p_{i},t_{j}))}{\mathrm{len}(t_{j})}$

Suppose there are $n$ predictions and $m$ ground truth answers for a question.
We compute the partial retrieved score between a prediction and all answers
and keep the highest one as the retrieved score of that prediction. Similarly,
for each ground truth answer, the relevant score is the highest one between it
and all predictions. The precision, recall, and F1 are finally defined as
follows:

$\mathrm{Precision}=\frac{\sum^{n}_{i=1}\mathrm{max}_{j\in[1,m]}(s_{ij}^{ret})}{n}$
$\mathrm{Recall}=\frac{\sum^{m}_{j=1}\mathrm{max}_{i\in[1,n]}(s_{ij}^{rel})}{m}$
$\mathrm{F1}=\frac{\mathrm{2*Precision*Recall}}{\mathrm{Precision+Recall}}$

We use micro-averaged scores for these metrics.

##  Appendix E Additional Experiment Results

###  E.1 Late Ensemble

By late ensemble, we aggregate the outputs from models of different paradigms
to boost performance. We experiment with a simple voting strategy. If a span
is predicted as an answer by more than one model, we add it to the final
prediction set. If a span is part of another span, we consider them equivalent
and take the longer one. In rare cases where the four models predict totally
different answers, we add them all to the final prediction set.

Our voting strategy leads to improvements of 1.0%, 1.2%, and 1.3% in PM F1 on
DROP, Quoref, and MultiSpanQA, respectively, over the best-performing models
in Table 5. Yet, this strategy might discard many correct answers. In the
future, we can explore more sophisticated strategies. For example, similar to
the idea of Mixture of Experts Jacobs et al. (1991), the system can evaluate
the probability that the instance belongs to a certain category and then
adjust the weight of the model based on its capabilities in this category.

###  E.2 Early Fusion

In Table 13, we report the performance of different strategies for early
fusion on different types of instances. In Table 14, we list the prompts used
for our pilot study on GPT-3.5.

| BART-base | BART-large
---|---|---
Model | p-dep. | q-dep. | p-dep. | q-dep.
|  |  All | w/-clue | w/o-clue |  |  All | w/-clue | w/o-clue
DROP
Vanilla Generation | 72.18 | 74.77 | 76.19 | 72.62 | 78.57 | 81.65 | 83.42 | 77.31
+NumPred (multitask) | 72.45 | 75.37 | 76.80 | 70.58 | 80.35 | 84.24 | 86.06 | 77.88
+NumPred (pipeline) | 70.58 | 75.72 | 77.14 | 76.77 | 76.79 | 82.66 | 84.34 | 79.65
+Iterative (normal) | 71.82 | 74.55 | 75.97 | 68.26 | 78.07 | 81.90 | 83.56 | 74.03
+Iterative (number) | 71.90 | 75.27 | 76.66 | 72.93 | 80.58 | 83.05 | 84.91 | 72.66
Quoref
Vanilla Generation | 66.31 | 77.41 | 78.38 | 52.63 | 76.41 | 88.51 | 88.90 | 79.76
+NumPred (multitask) | 67.54 | 79.37 | 80.15 | 58.30 | 77.73 | 87.88 | 88.11 | 82.35
+NumPred (pipeline) | 66.26 | 82.88 | 83.55 | 65.20 | 75.37 | 87.71 | 88.22 | 77.36
+Iterative (normal) | 69.40 | 82.68 | 83.24 | 68.24 | 73.13 | 87.43 | 87.93 | 77.20
+Iterative (number) | 65.79 | 77.16 | 77.97 | 55.63 | 77.69 | 88.09 | 88.59 | 75.41
MultiSpanQA
Vanilla Generation | 80.57 | 78.05 | 81.73 | 75.85 | 84.52 | 84.96 | 88.78 | 81.65
+NumPred (multitask) | 81.08 | 78.65 | 81.09 | 77.73 | 84.83 | 84.80 | 89.66 | 81.06
+NumPred (pipeline) | 60.24 | 63.56 | 68.67 | 58.25 | 67.27 | 71.21 | 74.53 | 69.33
+Iterative (normal) | 80.46 | 78.06 | 81.81 | 74.78 | 83.16 | 81.78 | 84.84 | 80.87
+Iterative (number) | 80.15 | 79.63 | 83.47 | 76.17 | 83.49 | 83.06 | 86.08 | 80.44
Table 13: The performance (PM F1) of different strategies for early fusion on
different types of instances. p-dep. denotes passage-dependent. q-dep. denotes
question-dependent. Vanilla GPT-3.5
---
Answer the question based on the given context. Each question has more than
one answer. Please give all the answers and separate them with a semicolon.
Context: Laura Horton is a fictional character from the NBC soap opera , Days
of Our Lives , a long - running serial drama about working class life in the
fictional , United States town of Salem . Created by writer Peggy Phillips ,
the role was originated by actress Floy Dean on June 30 , 1966 till October 21
, 1966 . Susan Flannery stepped into the role from November 22 , 1966 to May
27 , 1975 . Susan Oliver briefly stepped into the role from October 10 , 1975
, to June 9 , 1976 , followed by Rosemary Forsyth from August 24 , 1976 , to
March 25 , 1980 .
Question: who played laura horton on days of our lives
Answers: Floy Dean; Susan Flannery; Susan Oliver; Rosemary Forsyth
Following the example above and answer the following multi-answer question.
Please give all the answers and separate them with a semicolon.
Context: {context}
Question: {question}
Answers:
GPT-3.5 + NumPed
Answer the question based on the given context. Each question has more than
one answer. Please predict the number of answers first, then give all the
answers and separate them with a semicolon.
Context: Laura Horton is a fictional character from the NBC soap opera , Days
of Our Lives , a long - running serial drama about working class life in the
fictional , United States town of Salem . Created by writer Peggy Phillips ,
the role was originated by actress Floy Dean on June 30 , 1966 till October 21
, 1966 . Susan Flannery stepped into the role from November 22 , 1966 to May
27 , 1975 . Susan Oliver briefly stepped into the role from October 10 , 1975
, to June 9 , 1976 , followed by Rosemary Forsyth from August 24 , 1976 , to
March 25 , 1980 .
Question: who played laura horton on days of our lives
Answers: The number of answers is 4: Floy Dean; Susan Flannery; Susan Oliver;
Rosemary Forsyth
Following the example above and answer the following multi-answer question.
Please predict the number of answers first, then give all the answers and
separate them with a semicolon.
Context: {context}
Question: {question}
Answers:
Table 14: The one-shot prompts for GPT-3.5 to answer multi-answer questions in
MultiSpanQA.

##  Appendix F Licenses of Scientific Artifacts

The license for Quoref and DROP is CC BY 4.0. The license for HuggingFace
Transformers is Apache License 2.0. Other datasets and models provide no
licenses.