# Building for Speech: Designing the Next Generation of Social Robots for
Audio Interaction

Angus Addlesee<EMAIL_ADDRESS>Heriot-Watt UniversityEdinburghScotland,
UK ,  Ioannis Papaioannou<EMAIL_ADDRESS>Alana AIEdinburghScotland, UK
and  Oliver Lemon<EMAIL_ADDRESS>Heriot-Watt UniversityEdinburghScotland,
UK

(5 June 2023; 13 June 2023; 5 July 2023)

###### Abstract.

There have been incredible advancements in robotics and spoken dialogue
systems (SDSs) over the past few years, yet we still don’t find social robots
in public spaces like train stations, shopping malls, or hospital waiting
rooms. In this paper, we argue that early-stage collaboration between robot
designers and SDS researchers is crucial to create social robots that can
legitimately be used in real-world environments. We draw from our experiences
running experiments with social robots, and the surrounding literature, to
highlight recurring issues. Robots need more speakers, more microphones,
quieter motors, and quieter fans to enable human-robot spoken interaction in
the wild and improve accessibility. More robust robot joints are also needed
to limit potential harm to older adults and other more vulnerable groups.

social robots, spoken dialogue, accessibility, robotics

††conference: make sure it’s right; July 19, 2023; Eindhoven, The
Netherlands††ccs: Hardware Sound-based input / output††ccs: Human-centered
computing Human computer interaction (HCI)††ccs: Human-centered computing
Accessibility technologies

##  1\. Introduction

Social robots are not yet found in our public spaces, despite this vision
being an imminent reality over 25 years ago (Thrun, 1998). They don’t roam our
shopping malls helping lost families find the bathrooms, we don’t bump into
them in museums and airports, and they are not helping patients in hospital
waiting rooms (see Figure 1).

Figure 1. A person asking for directions in a hospital.

Spoken dialogue systems (SDSs) have consistently improved over time (Glass,
1999; Williams, 2009; Lemon, 2022), with many years of peer-reviewed papers
containing remarkable results. These models, however, are often evaluated
automatically upon collected data, or with users in highly controlled lab
settings. Social robots work wonderfully in the lab, but fail when deployed in
the real world for experiments or demonstration. Some of these failures stem
from the embedded SDS (for example: multi-party interactions (Addlesee et al.,
2023), or voice accessibility (Addlesee, 2023)), but even interactions that
the SDS should be able to handle with ease go wrong. These failures are often
caused by the design of the social robot itself. The field of robotics has
also seen incredible advancements over the past years, today’s robots can
navigate obstacle courses (Xiao et al., 2022), manipulate objects in their
environment (Chai et al., 2022), generate human-like gestures (Tatarian et
al., 2022), and follow complex human instructions using large language models
(LLMs) (Ahn et al., 2022). Sadly, while collaboration between these two fields
is common, they often begin after the robot has been designed. In our
experience from multiple international robot dialogue projects, spoken
interaction is not considered during the initial design phase. This lack of
early-stage collaboration between robot designers and SDS researchers leaves
room for oversight of critical features for spoken interaction. In this paper,
we have combed the literature and drawn from our own experiences to highlight
the main problems that repeatedly surface when experimenting with social
robots and real users. We hope this paper sparks discussion between both
communities to create fully functioning social robots that do genuinely work
in public spaces in the future.

##  2\. People Struggle to Hear Robots

The first issue that crops up commonly in the literature is the limited volume
of the robot’s voice. Robot designers simply attach a speaker to the robot
without considering the fact that the world is noisy, and some users, such as
older adults, may have hearing loss.

In recent work, researchers deployed a robot to interact with real users in an
assisted living facility. The robot had to be fitted with an additional
speaker that had a louder maximum volume. The modification was necessary
because the users simply could not hear the robot’s voice, preventing any
basic interaction (Stegner et al., 2023).

This issue is not constrained to this particular setting, or to one particular
robot. Researchers had to repeat every sentence the robot said in the lobby of
a concert hall, as participants couldn’t hear it (Langedijk et al., 2020). In
various school environments, the robot’s volume was not loud enough to enable
effective interaction, so external speakers had to be fitted (Nikolopoulos et
al., 2011). When guiding people in an elder care facility, the robot’s single
speaker faced the wrong direction, so users could not hear it (Langedijk et
al., 2020). A robot was deployed in the homes of a few older adults, and they
noted that its volume was not loud enough. People couldn’t hear the social
robot in a gym (Sackl et al., 2022), and the list goes on.

Robots are expensive. The speakers that researchers had to retrofit to the
robots were inexpensive and readily available. This low-cost change was
simple, yet _crucial_ , to enable effective communication with a user in a
real-world setting. When designing robots for spoken interaction, we recommend
fitting multiple speakers (facing various directions) that have a loud maximum
volume. This will guarantee that the robot can be heard in public spaces, and
ensure its accessibility for people with limited hearing.

##  3\. Robots Struggle to Hear People

There is another conversation participant that cannot hear what their
interlocutor is saying – the robot. This problem is similar to the one in
Section 2, and is also frequently found in the literature. A social robot
struggled to hear users in a hotel lobby, for example (Hahkio, 2020). Many
researchers retrofit better microphones to the robot (Villalpando et al.,
2018), or next to the robot (Wagner et al., 2023), in order to hear the user
more clearly.

In an assisted living facility, researchers had to resort to listening to the
user through an ajar door to run their experiments. The microphone array could
not reliably pick up what users said (Stegner et al., 2023).

Home voice assistants do successfully hear people in noisy environments, like
family homes (Porcheron et al., 2018), however. They can pick up what the user
said when other conversations are happening in the room, and when the TV or
radio are on (we are finding this in ongoing work (Addlesee, 2022)). Today’s
social robots typically have four microphones111We checked the technical
specifications of several commonly used social robots, and robots that we have
deployed ourselves. We are refraining from naming specific robot creators, as
this paper aims to encourage collaboration, and not criticise specific
robots., but we argue that this is far too few. Apple’s Homepod originally had
six microphones (Calore, 2019) and Amazon’s Alexa Echo had seven (Spekking,
2021). The newest Homepod and Echo’s have reduced to four microphones for two
reasons: (1) These devices are incentivised to keep their device’s costs low
to encourage adoption by new users (Welch, 2023); and (2) The device’s shape
and internal component arrangements have been refined and optimised over many
years through experiments with millions of users (Wilson, 2020). Robots do not
share either of these arguments. Microphones are trivially inexpensive
relative to the price of a robot, and instead of helping microphones, the
robot’s shape actively hinders their performance. The body parts of a social
robot often sit between the user and the microphone (for example, when the
user is behind the robot, or in a wheelchair). Robots also create a lot of
noise themselves, called ego-noise. Related research required high-quality
audio input from a noisy propellered UAV, so they attached sixteen microphones
in various locations around the device (Nakadai et al., 2017), not just four.

Human-robot spoken communication can also be disrupted by societal or
linguistic phenomena, such as overlapping or poorly formed turn-taking
conditions. Such conditions include barging-in (Wagner et al., 2021) (where
the user interrupts the robot mid-sentence, but the robot fails to recognise
the user started speaking), and poor end-of-turn detection (due to long pauses
or intermittent speech from the user). In our experience, users sometimes
barge-in because of high latency caused by limited computational power onboard
the robot, or on-site connectivity issues, in addition to the latency of the
SDS.

Figure 2. A social robot providing stability to an older adult.

Potential approaches to this challenge include incremental dialogue processing
(Addlesee et al., 2020; Aylett et al., 2023), or explicit turn-taking signals
which enable the user to better understand when the robot is actually
listening to them. For instance, Foster et al. (2019) employed a tablet on the
robot’s torso that was showing ”I am listening” and ”I am speaking” text to
help guide the users in a noisy shopping mall setup.

We recommend fitting multiple high-quality microphones in various locations
around the robot’s body, as well as appropriate signal processing techniques,
such as beam-forming (Adel et al., 2012). Latency issues must be addressed
within the SDS, and by increasing the robot’s computational capabilities.
These changes will again ensure that spoken interactions can realistically
take place in public spaces. Designers must also consider microphone placement
lower down on the robot for shorter users, and for people in wheelchairs, as
they are commonly just placed on the top of the robot’s head.

##  4\. Ego-Noise

This issue of ego-noise, introduced in Section 3, is so problematic that an
entire field of research has grown to tackle it. Researchers find that ego-
noise doesn’t just negatively impact ASR performance, but ego-noise reduction
methods also suppress some of the user’s utterance (Ince et al., 2010; Schmidt
et al., 2018). To clarify, both the ego-noise reduction techniques, and the
ego-noise itself negatively impact ASR performance.

This issue would be helped by additional speakers and microphones, ideally not
placed next to noise sources, allowing both parties to hear each other. An
optimal social robot designed for spoken interaction would also have much
quieter joint motors and fans. These are more expensive than speakers and
microphones, but they would greatly improve the SDSs ability to understand the
user. This could be paired with research to repair and understand disrupted
sentences (Addlesee and Damonte, 2023b, a), while quieter motors are
developed.

##  5\. Joint Robustness

Ego noise obviously does not impact robots that do not have a body. In our
minds, though, social robots should be able to point and guide users. For
example, consider a hospital waiting room in a hospital memory clinic (Gunson
et al., 2022). Patients are typically older adults, and may use the robot’s
arm for stability, like they would with another human (see Figure 2). Current
social robots can generate social gestures like waving or holding its hand out
for a handshake. If you were to shake the robot’s hand, however, it would
likely break.

This fragility could potentially harm users if deployed in this setting.
People may assume that they can link arms with the robot while being guided, a
perfectly natural assumption. When an older adult puts their weight on the
robot’s joint, though, they would fall. This is clearly a potentially harmful
design flaw that must be resolved if we are ever going to find robot
assistants in the wild.

##  6\. Conclusion

Interacting with social robots in public spaces is currently still a sci-fi
fantasy. There are challenges that SDS researchers must tackle to reach this
goal, but that is not the only bottleneck. Even a perfect SDS would fail if it
was embedded in today’s robots. We have highlighted that robot’s need more
speakers, more microphones, quieter fans, and quieter motors to allow both
parties to hear each other. These are critical problems that completely block
spoken interactions outside a lab setting. We additionally highlighted that
robots also need to be more robust if they are to be safely applied in the
real world, particularly in settings with older adults.

Our suggestions are not an exhaustive list, but we hope that they spark
discussion and encourage collaboration between robot designers and SDS
researchers. This collaboration should take place in the initial-stages of a
robot’s design to avoid the retrofitting of hardware discussed in this paper.

###### Acknowledgements.

The images in this paper were generated with Hotpot.io.

## References

  * (1)
  * Addlesee (2022) Angus Addlesee. 2022\.  Securely Capturing People’s Interactions with Voice Assistants at Home: A Bespoke Tool for Ethical Data Collection. In _Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI)_.
  * Addlesee (2023) Angus Addlesee. 2023\.  Voice Assistant Accessibility. In _The International Workshop on Spoken Dialogue Systems Technology, IWSDS 2023_.
  * Addlesee and Damonte (2023a) Angus Addlesee and Marco Damonte. 2023a.  Understanding and Answering Incomplete Questions. In _Proceedings of the 5th Conference on Conversational User Interfaces_.
  * Addlesee and Damonte (2023b) Angus Addlesee and Marco Damonte. 2023b.  Understanding Disrupted Sentences Using Underspecified Abstract Meaning Representation. In _Interspeech_.
  * Addlesee et al. (2023) Angus Addlesee, Weronika Sieińska, Nancie Gunson, Daniel Hernández García, Christian Dondrup, and Oliver Lemon. 2023.  Data Collection for Multi-party Task-based Dialogue in Social Robotics. In _The International Workshop on Spoken Dialogue Systems Technology, IWSDS 2023_.
  * Addlesee et al. (2020) Angus Addlesee, Yanchao Yu, and Arash Eshghi. 2020.  A comprehensive evaluation of incremental speech recognition and diarization for conversational AI. In _Proceedings of the 28th International Conference on Computational Linguistics_.
  * Adel et al. (2012) Hidri Adel, Meddeb Souad, Abdulqadir Alaqeeli, and Amiri Hamid. 2012\.  Beamforming Techniques for Multichannel audio Signal Separation.  arXiv:1212.6080 [cs.OH]
  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022\.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ (2022).
  * Aylett et al. (2023) Matthew P Aylett, Andrea Carmantini, and David A Braude. 2023\.  Why is My Social Robot so Slow? How a Conversational Listener can Revolutionize Turn-Taking. In _Proceedings of the 5th Conference on Conversational User Interfaces_.
  * Calore (2019) Michael Calore. 2019\.  Review: Apple Homepod.  _Wired_ (2019).
  * Chai et al. (2022) Hui Chai, Yibin Li, Rui Song, Guoteng Zhang, Qin Zhang, Song Liu, Jinmian Hou, Yaxian Xin, Ming Yuan, Guoxuan Zhang, et al. 2022\.  A survey of the development of quadruped robots: Joint configuration, dynamic locomotion control method and mobile manipulation approach.  _Biomimetic Intelligence and Robotics_ 2, 1 (2022).
  * Foster et al. (2019) Mary Ellen Foster, Bart Craenen, Amol Deshmukh, Oliver Lemon, Emanuele Bastianelli, Christian Dondrup, Ioannis Papaioannou, Andrea Vanzo, Jean-Marc Odobez, Olivier Canévet, Yuanzhouhan Cao, Weipeng He, Angel Martínez-González, Petr Motlicek, Rémy Siegfried, Rachid Alami, Kathleen Belhassein, Guilhem Buisan, Aurélie Clodic, Amandine Mayima, Yoan Sallami, Guillaume Sarthou, Phani-Teja Singamaneni, Jules Waldhart, Alexandre Mazel, Maxime Caniot, Marketta Niemelä, Päivi Heikkilä, Hanna Lammi, and Antti Tammela. 2019\.  MuMMER: Socially Intelligent Human-Robot Interaction in Public Spaces.  arXiv:1909.06749 [cs.RO]
  * Glass (1999) James Glass. 1999\.  Challenges for spoken dialogue systems. In _Proceedings of the 1999 IEEE ASRU Workshop_ , Vol. 696. MIT Laboratory fot Computer Science Cambridge, MA, USA.
  * Gunson et al. (2022) Nancie Gunson, Daniel Hernández Garcia, Weronika Sieińska, Angus Addlesee, Christian Dondrup, Oliver Lemon, Jose L Part, and Yanchao Yu. 2022.  A Visually-Aware Conversational Robot Receptionist. In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue_.
  * Hahkio (2020) Linh Hahkio. 2020\.  Service robots’ feasibility in the hotel industry: A case study of Hotel Presidentti.  (2020).
  * Ince et al. (2010) Gökhan Ince, Kazuhiro Nakadai, Tobias Rodemann, Yuji Hasegawa, Hiroshi Tsujino, and Jun-ichi Imura. 2010\.  A hybrid framework for ego noise cancellation of a robot. In _2010 IEEE International Conference on Robotics and Automation_. IEEE.
  * Langedijk et al. (2020) Rosalyn Melissa Langedijk, Cagatay Odabasi, Kerstin Fischer, and Birgit Graf. 2020.  Studying drink-serving service robots in the real world. In _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE.
  * Lemon (2022) Oliver Lemon. 2022\.  Conversational AI for multi-agent communication in Natural Language.  _AI Communications_ Preprint (2022).
  * Nakadai et al. (2017) Kazuhiro Nakadai, Makoto Kumon, Hiroshi G Okuno, Kotaro Hoshiba, Mizuho Wakabayashi, Kai Washizaki, Takahiro Ishiki, Daniel Gabriel, Yoshiaki Bando, Takayuki Morito, et al. 2017\.  Development of microphone-array-embedded UAV for search and rescue task. In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE.
  * Nikolopoulos et al. (2011) Christos Nikolopoulos, Deitra Kuester, Mark Sheehan, Shashwati Ramteke, Aniket Karmarkar, Supriya Thota, Joseph Kearney, Curtis Boirum, Sunnihith Bojedla, and Angela Lee. 2011.  Robotic agents used to help teach social skills to children with autism: the third generation. In _2011 RO-MAN_. IEEE.
  * Porcheron et al. (2018) Martin Porcheron, Joel E Fischer, Stuart Reeves, and Sarah Sharples. 2018.  Voice interfaces in everyday life. In _proceedings of the 2018 CHI conference on human factors in computing systems_.
  * Sackl et al. (2022) Andreas Sackl, Daniele Pretolesi, Sophia Burger, Markus Ganglbauer, and Manfred Tscheligi. 2022\.  Social Robots as Coaches: How Human-Robot Interaction Positively Impacts Motivation in Sports Training Sessions. In _2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE.
  * Schmidt et al. (2018) Alexander Schmidt, Heinrich W Löllmann, and Walter Kellermann. 2018.  A novel ego-noise suppression algorithm for acoustic signal enhancement in autonomous systems. In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE.
  * Spekking (2021) Raimond Spekking. 2021\.  Amazon Echo Dot (RS03QR) - LED and microphone board.  _Wikimedia_ (2021).
  * Stegner et al. (2023) Laura Stegner, Emmanuel Senft, and Bilge Mutlu. 2023\.  Situated participatory design: A method for in situ design of robotic interaction with older adults. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_.
  * Tatarian et al. (2022) Karen Tatarian, Rebecca Stower, Damien Rudaz, Marine Chamoux, Arvid Kappas, and Mohamed Chetouani. 2022\.  How does modality matter? investigating the synthesis and effects of multi-modal robot behavior on social intelligence.  _International Journal of Social Robotics_ 14, 4 (2022).
  * Thrun (1998) Sebastian Thrun. 1998\.  When robots meet people.  _IEEE Intelligent Systems and their Applications_ 13, 3 (1998).
  * Villalpando et al. (2018) Antonio Pico Villalpando, Guido Schillaci, and Verena V Hafner. 2018\.  Predictive Models for Robot Ego-Noise Learning and Imitation. In _2018 Joint IEEE 8th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)_. IEEE.
  * Wagner et al. (2023) Nicolas Wagner, Matthias Kraus, Niklas Lindemann, and Wolfgang Minker. 2023.  Comparing Multi-User Interaction Strategies in Human-Robot Teamwork. In _The International Workshop on Spoken Dialogue Systems Technology, IWSDS 2023_.
  * Wagner et al. (2021) Nicolas Wagner, Matthias Kraus, Niklas Rach, and Wolfgang Minker. 2021.  _How to Address Humans: System Barge-In in Multi-user HRI_.  147–152.  https://doi.org/10.1007/978-981-15-9323-9_13
  * Welch (2023) Chris Welch. 2023\.  Apple’s new HomePod unsurprisingly sounds close to the original.  _The Verge_ (2023).
  * Williams (2009) Jason D Williams. 2009\.  Spoken dialogue systems: Challenges, and opportunities for research.. In _ASRU_.
  * Wilson (2020) Mark Wilson. 2020\.  Why Amazon radically redesigned the Echo.  _Fast Company_ (2020).
  * Xiao et al. (2022) Xuesu Xiao, Zifan Xu, Zizhao Wang, Yunlong Song, Garrett Warnell, Peter Stone, Tingnan Zhang, Shravan Ravi, Gary Wang, Haresh Karnan, et al. 2022\.  Autonomous Ground Navigation in Highly Constrained Spaces: Lessons Learned From the Benchmark Autonomous Robot Navigation Challenge at ICRA 2022 [Competitions].  _IEEE Robotics & Automation Magazine_ 29, 4 (2022).