# Exploring Large Language Models to Facilitate Variable Autonomy for Human-
Robot Teaming

Younes Lakhnati 1,∗, Max Pascher 1,2,∗, and Jens Gerken 1,∗

###### Abstract

## 1

In a rapidly evolving digital landscape autonomous tools and robots are
becoming commonplace. Recognizing the significance of this development, this
paper explores the integration of Large Language Models like  Generative pre-
trained transformer (GPT) into human-robot teaming environments to facilitate
variable autonomy through the means of verbal human-robot communication. In
this paper, we introduce a novel framework for such a GPT-powered multi-robot
testbed environment, based on a Unity Virtual Reality (VR) setting. This
system allows users to interact with robot agents through natural language,
each powered by individual GPT cores. By means of OpenAI’s function calling,
we bridge the gap between unstructured natural language input and structure
robot actions. A user study with 12 participants explores the effectiveness of
GPT-4 and, more importantly, user strategies when being given the opportunity
to converse in natural language within a multi-robot environment. Our findings
suggest that users may have preconceived expectations on how to converse with
robots and seldom try to explore the actual language and cognitive
capabilities of their robot collaborators. Still, those users who did explore
where able to benefit from a much more natural flow of communication and
human-like back-and-forth. We provide a set of lessons learned for future
research and technical implementations of similar systems.

##  2 Keywords:

assistive robots, virtual reality, evaluation, shared control, variable
autonomy

##  3 Introduction

Mankind’s success is predicated upon our ability to communicate and cooperate.
This fundamental principle, evident throughout the history of human
civilization, has now reached a pivotal point. The tools we have created can
think and act on their own. This requires us to rethink past approaches to
human-machine interaction, which had relied mainly on humans operating
machines, not collaborating with them (Grudin, 2022). However, for humans to
cooperate seamlessly with autonomous tools, such as robots, a delicate balance
must be struck between both a user’s need for autonomy Deci and Ryan (2012)
and the robot’s capabilities and level of autonomous behavior. In this
context, concepts such as variable autonomy and shared control emerge as a
vital research area. Shared control, in particular, aims to find optimal
configurations of how control can be shared between an autonomous machine such
as a robot and human so that the overall system is as effective and efficient
as possible (Erdogan and Argall, 2017). It requires, at its core, an effective
way for communication between parties to negotiate and adapt the sharing of
control (Flemisch et al., 2019; Abbink et al., 2018; Pascher et al., 2023a).
While such communication between human and machines can rely on a variety of
input and output modalities, spoken language, as a vehicle of thought, has
played an essential role for human-human communication and collaboration
(Clark, 1996, pp. 17–18). The exploration of natural language in fields like
human-machine cooperation and AI has a long and storied history. Natural
Language Processing (NLP) has been an incredibly significant branch of
robotics and Artificial Intelligence (AI) research since its infancy,
spearheaded by foundational works like Winograd’s _SHRDLU_ (Winograd, 1972)
and Nilsson et al.’s _Shakey the Robot_ (Nilsson et al., 1984).

Conceptually, however, for language to be an effective means for
collaboration, research has suggested the overarching concept of _Joint
Actions_ (Clark, 1996, p. 59), which has become a popular approach to
understanding human-human cooperation (Sebanz et al., 2006; Garrod and
Pickering, 2004; Vesper et al., 2010). This concept has the potential to be
extrapolated to the human-and-machine side of cooperation, as suggested by
Flemisch et al. (2019, p. 2). This conceptual perspective is joined by recent
advances in the domain of NLP, following Vaswani et al.’s groundbreaking
_Transformer_ architecture(Vaswani et al., 2017), which gave rise to so-called
LLMs, such as GPT and  Bidirectional Encoder Representations from Transformers
(BERT) (Radford et al., 2018, p. 6) (Devlin et al., 2018, p. 6). These have
demonstrated competences in fields such as translation (Brown et al., 2020, p.
6–7), question-answering (Nakano et al., 2021), creative writing (Swanson et
al., 2021) and even medical diagnostics (Hirosawa et al., 2023; Nori et al.,
2023).

With an LLM’s ability to parse unstructured natural language and to adapt and
learn on the fly based on _prompts_ , we see significant potential in their
application within the field of shared control and variable autonomy. Not only
do they present modes of interaction that align with our inherent tendencies
to use language (like we would in inter-human cooperation), these systems are
also very adept at producing believable output that can mirror a conversation.
LLMs could facilitate the bi-directional flow of information between humans
and autonomous agents in a shared control system by understanding nuances,
asking for clarifications and providing feedback. Additionally, the degree to
which an LLM is autonomous can be influenced by the prompts it is given. Of
course, there are certain inherent challenges in play as well. The use cases
where LLMs excel do not require to map spoken commands to a very specific
output that is bound by the physical capabilities of an autonomous agent.
Still, existing work has shown that available LLMs such as GPT might already
allow the communication and coordination based on natural language with a
robot (Koubaa, 2023b). This paper aims to explore this up and coming research
area both from a feasibility and a user perspective. To this end, we have
designed an LLM-powered multi-agent system as part of a VR-based framework. In
the virtual game world, a user can fulfill simple tasks solely by conversing
with three simulated robot agents, which are controlled by their own GPT
cores. Utilizing the prototype, a user study spanning 12 participants was
conducted to investigate the ways in which humans engaged with the robots, how
natural it felt to them and how well GPT was suited for this use case.
Consequently, the contribution of this paper consists of

  * •

The design and architecture of a Unity-based framework for a voice-controlled
multi-agent system in VR, with which the interaction and control dynamics
between a single user and a scalable amount of LLM-based robots can be
explored111We plan to make the framework available as Open-Source on GitHub
upon acceptance and are happy to provide it to reviewers on request in the
meantime.

  * •

Based on our study, an improved understanding of user’s strategies and
behaviors when being confronted with an LLM-based multi-robot envionrment.

  * •

A set of lessons learned on the feasibility and practicability of adapting
GPT-based LLMs to interact with autonomous multi-robot agents.

##  4 Background

###  4.1 Shared Control & Variable Autonomy

While research in autonomous robotics is trending (Canal et al., 2016;
Lauretti et al., 2017; Gallenberger et al., 2019; Rakhimkul et al., 2019),
there are numerous indications that an autonomous robot does not help to
satisfy basic psychological needs such as self-autonomy and competence. For
example, Kim et al. (2012) reported that the sporadic and supervisory nature
of working with an autonomous robot results in the users experiencing them as
“one more external agent that does their work for them”. These findings are
corroborated by Pollak et al. (2020) who reported that manual control compared
to autonomous behavior led to significantly reduced symptoms of stress and a
lower heart rate. Similarly, Złotowski et al. (2017) found that autonomous
robots generally evoke a more negative attitude than non-autonomous robots and
are experienced as more threatening. Latikka et al. (2021) reported, that in
particular in the workplace, users preferred to see robots as equipment and as
a result preferred non-autonomous robots. In addition Park et al. (2020)
showed that fully autonomous robots may discourage people with motor
impairments from using their remaining physical capabilities.

To counter that, shared-control (or traded-control) systems aim to strike a
balance between autonomous robot behavior and manual user control (Erdogan and
Argall, 2017; Pascher et al., 2023b, 2024). Due to the lack of clear
definitions of these terms, Abbink et al. (2018) introduced a topology of
shared control systems and axioms for the design and evaluation thereof,
unifying varying shared control concepts and definitions under one common
framework. Flemisch et al. (2019) later expanded upon Abbink et al.’s
framework and explored shared-control in the context of human-robot
cooperation, emphasizing the importance of cooperation on multiple layers of
interaction, such as a strategical, tactical and operational layer.

Abbink et al. (2018) argue, that robot functionality, and as a result users
preferences and abilities, are situational and depend on context. As a result,
concepts such as variable autonomy or adjustable autonomy have emerged, which
build upon the principle of shared control by adding a dynamic and situated
component (Chiou et al., 2023; Bustamante et al., 2021). They often provide
the user with some means to adjust the degree and level of involvement or
control in a task. While typically, such means are realized through user
interface elements, e.g. control elements such as buttons, research has long
aimed to establish language as means of collaborating with a machine.

###  4.2 Natural Language Interface systems and Large Language Models

The domain of Natural language interfaces represents an important reference
point for our research. Historical works such as SHRDLU (Winograd, 1972) and
LUNAR system (Woods, 1973) are early examples of natural language being used
to control complex systems. Both Woods and Winograd identified that the
biggest challenges stemmed from semantic barriers that would need to be
overcome by advancements in the field of NLI and NLP. While still battling
with the challenges of human linguistics, commercial product developments in
the form of Virtual Personal Assistants or Intelligent Personal Assistants,
such as Microsoft’s Cortana, Apple’s Siri, Amazon Alexa, Google Assistant, and
so on were able to create a public awareness, establish a user base and
provide a better understanding of the potential of such systems (Kepuska and
Bohouta, 2018). Related approaches were also introduced into the field of
robotics – here Liu and Zhang (2019) offer an extensive overview related to
NLI-controlled robots.

While these approaches were certainly impressive, they still lacked a
fundamental understanding that transcends their confined domains.

Following Vaswani et al. (2017)’s breakthrough work on self-attention, the
development of transformer models has reshaped the landscape of AI as well as
NLP and gave rise to LLMs: AI systems, which are trained on massive amounts of
textual data using deep learning techniques. Building upon this concept, BERT,
revolutionized many NLP tasks by training bidirectionally, meaning it
considers both the left and the right context in all layers, capturing
information from the entire passage (Devlin et al., 2018). This made BERT
particularly suited for tasks that require understanding context. On the other
hand, GPT is a unidirectional model (every token can only attend to previous
tokens) trained to predict the next word in a sequence (Radford et al., 2018).
Despite this, it excels in various NLP tasks by leveraging its transformer
architecture and a massive amount of data for training. GPT is especially
noteworthy for its capability to generate coherent and contextually relevant
text over long passages. Both, BERT and GPT have not only been released for
research collaboration, but have been integrated into commercial products and
provide Application Programming Interface (API) access for software
developers. Noteworthy, there is a sheer infinite number of further LLMs and
several significant extensions to these existing approaches (Yang et al.,
2019; Brown et al., 2020; Chen et al., 2021; Neelakantan et al., 2022;
Stiennon et al., 2020; Ouyang et al., 2022). Thanks to the generality of
language, these models have been applied in various applications, ranging from
search engines and chat bots (Kelly et al., 2023), general problem-solving and
knowledge extraction (Petroni et al., 2019), medical diagnosis (Hirosawa et
al., 2023; Nori et al., 2023; Waisberg et al., 2023; Shea et al., 2023; Rao et
al., 2023), education (Tack and Piech, 2022; Firat, 2023; Ausat et al., 2023),
law (Cyphert, 2021; Perlman, 2022; Trozze et al., 2023) and robotics, as we
will discuss in the next section.

###  4.3 LLMs and Robots

LLMs have opened up new possibilities in the field of robotics and human-robot
teaming, most apparently for social robots and Socially Assistive Robots (Lee
et al., 2023; Alessa and Al-Khalifa, 2023; Kahambing, 2023; Lekova et al.,
2023; Irfan et al., 2023). However, the ability of LLMs to converse is not
only beneficial in social settings but their ability to process all kinds of
textual input seems extremely promising in environments like speech-controlled
robotics, which have historically struggled with unstructured input. Recent
research indicates that LLMs work well in this environment and that users
generally prefer unstructured modes of communication in comparison to
structured ones (Kodur et al., 2023, p. 10).

The issue at hand is the mapping between unstructured voice input and
structured robot actions, sometimes referred to as language grounding or as
part of the alignment problem. Trott et al. (2015), while not yet being able
to rely on a LLM, presented a promising architectural layout for verbal
communication with multiple robots. They introduced a boss-agent, which routes
the communication between multiple robots and the user via N-tuple, which is
translated into a pattern response. While the approach was limited through the
need for an explicit grammar and lexicon, similar to the grammar based
approach by Misra et al. (2016), we adapted this approach for our work, as we
will discuss in the next chapter. Arumugam et al. (2019) analyzed the issue of
language grounding more closely and proposed an approach to achieve a high
“accuracy, efficiency and generalization through the construction of simple,
semantic goal representations within Markov decision processes”. Ahn et al.
(2022) propose a method called “SayCan”, that integrates the capabilities of
LLMs with robotic affordances (learned mappings that quantify the likelihood
of a robot successfully executing specific actions given its current state) to
better interpret and execute high-level textual instructions. Their approach
seeks to bridge the gap between the knowledge embedded within LLMs and the
practical constraints of physical environments. Similar to our work, Koubaa
(2023a) integrated a GPT model and a parser to operate a robot – in this case
a real robot through Robot Operating System (ROS). They employed an ontology
approach to map unstructured language input into structured robotic
instructions, but encountered issues with the model unexpectedly straying from
the ontology at times. We initially chose the same approach and encountered
similar issues of GPT not adhering to the ontology.

Also just recently, Händler introduced a multi-dimensional taxonomy,
specifically designed to evaluate and categorize the way autonomous LLM-
powered multi-agent systems manage the relationship between autonomy and
alignment Händler (2023). This analysis covers various architectural
perspectives, focusing on their intrinsic characteristics and operational
mechanisms. Händler’s approach is motivated by the observation that existing
taxonomies fall short to categorize and understand the complexity of these new
systems.

In 2023, OpenAI introduced function calls222OpenAI Function calling.
https://platform.openai.com/docs/guides/function-calling, last retrieved
August 31, 2024. as a way to bridge the gap between unstructured text input
and structured system operation that is much less prone to unexpected model
behaviors. We see this as promising alternative to the existing approaches
that have been discussed here. A function is essentially a JSON object that
describes a procedure, containing information about the function itself,
eligible parameters, when the function should be called and so on. Based on
this description, a GPT module, which would usually respond to a user in form
of a chat response, can decide to not answer at all but return a function call
instead. This decision is up to the LLM’s discretion and is heavily influenced
by previous messages and the descriptions of its available functions. For
example, if a GPT module is fed with function descriptions containing a “pick
up” function that defines “apple” as an eligible parameter, a user asking it
anything close enough to “pick up the apple” would cause it to return a “pick
up” function call with the parameter “apple”. Conversely, if a user asked the
LLM to pick up an object that is not listed as an eligible parameter in the
“pick up” function description, the LLM would not execute the function call
but ask the user for clarification instead. As a result, function calls also
provide an additional grounding of a GPT module. In addition, function calls
can be employed as GPT’s close connection to the program’s backend. Since
function calls (unlike an LLM’s regular answers) are clearly defined and
predictable, procedures in the code can be called as a direct consequence of
the LLM’s decisions. For instance, if the GPT module returns the example
function call from above, the extracted information can be used to start
whatever routine is defined as the “pick up” procedure, with “apple” as its
parameter.

Aiming to understand how users interact with machines using language, the work
of Porcheron and colleagues is notable. Starting with IPAs, Porcheron et al.
examined the integration of IPAs within human conversation, focusing on
linguistic aspects. They explore how IPAs such as Siri and Alexa influence
conversational dynamics and shed light on the linguistic interplay between
human users and AI agents (Porcheron et al., 2017). Expanding on this theme,
Fuentes et al. explore linguistic challenges faced in human-robot interactions
in low-resource environments (Fuentes et al., 2023). Their research emphasized
the importance of robots possessing a deep contextual understanding to
accurately interpret user instructions. They identified natural language
challenges such as referential expression resolution and the dynamic nature of
language, which can pose hurdles in human-robot communication. Their findings
accentuate the need for improved linguistic reasoning in robots, especially in
specialized environments where context plays a key role.

###  4.4 Take-away

In our work, we add to this emerging area of research by addressing two
perspectives in particular. First, we realized that while the utilization and
integration of LLMs has become much easier, prototyping LLM-powered robots is
not straightforward which limits our ability to explore the inherent trade-
offs and limitations. Therefore, we explored and developed the integration of
a GPT-4 model within a Unity-based simulation environment, relying on the just
introduced function calls capabilities to map unstructured speech input to
structured robot actions. Second, there is currently very little research
which helps to understand how the combination of speech input and robots
influences and changes the way users talk and interact with robots or virtual
agents. In particular, the non-deterministic nature of LLMs makes it essential
to explore this in a working implementation of LLM-powered robot agents.
Wizard-of-Oz like studies, such as Fuentes et al. (2023), mostly try to
simulate a perfectly functioning AI, while human-robot conversation may be
most interesting in situations where there is miscommunication.

##  5 LLM Framework Concept and Architecture

As discussed, the application of LLMs to the field of robotics is still in its
infancy, but could be in particular promising to utilize the inherent
capability of language-based communication to support variable autonomy in a
human-robot teaming environment. Still, a range of challenges must be
addressed to effectively develop a flexible LLM-based framework for the study
of human-robot collaboration, which is the focus of this chapter.

To give an overview, these are:

  * •

Decide and select between an existing generic LLM compared to either fine-
tuned or custom-trained models.

  * •

Mapping unstructured verbal input to actionable robot agent behavior.

  * •

Provide a flexible testbed-framework, which allows the technical exploration
as well as the conduction of user studies.

  * •

Provide a scalable architecture for the framework, which supports multiple
robot agents.

_LLM selection_ : The motivation for our research is grounded in the idea that
advanced AI technologies, such as LLMs, have reached a level of maturity that
they can be applied by non-AI experts and thereby be leveraged for a variety
of contexts, such as human-robot teaming. Therefore, we decided to select an
existing general purpose LLM for our framework and explore the capabilities
and limitations this may pose.

_Mapping problem_ : Based on the second challenge, mapping unstructured verbal
input to actionable robot agent behavior, we eventually decided to utilize
OpenAI’s GPT-4 model, as the already described function calls capability
provides a promising approach to overcome this problem. One additional reason
why function calls are worth exploring in this context is that they provide a
degree of agency to the LLM. Instead of just answering in chat messages, when
the LLM deems it probable enough that the user wants it to execute an action
which is within its abilities, it can decide to “seize control” and initiate a
function. However, when the LLM is unsure and requires further information, it
can “relinquish control” again. For example, if a user asked the LLM to pick
up an object that is not listed as an eligible parameter in a “pick up”
function description, the LLM would not execute the function call but ask the
user for clarification instead.

In this way, the function descriptions also provide a scaffolding to the model
of what it can and cannot do. The LLM can then share this information with the
user, contributing to a shared understanding of what the agents are capable of
and to which degree they can cooperate with the user. Function descriptions
are also highly extensible and modifiable, making them well-suited for an
experimental framework that might need to be adjusted to investigate specific
situations and contexts.

_Testbed framework_ : We decided for a VR-based testbed framework to both
explore technical exploration of LLM-based human-robot teaming and also study
human-robot interaction and communication in a flexible and adaptable
environment. VR was chosen over physical robots to simplify complexity and
increase flexibility, while also maintaining an immersive environment to
facilitate realistic interaction dynamics.

Figure 1: A mock-up image portraying a user in the gameworld, interacting with
the three robots, Neptune (in blue), Jupiter (in yellow) and Pluto (in red).

In our initial framework setting, we feature three heterogeneous robot agents
named Jupiter, Pluto, and Neptune (see Figure 1). Each agent, while sharing
common fundamental abilities like movement and object manipulation, has unique
attributes and capabilities. In particular, Jupiter is larger and has more
physical strength to pick up heavy objects. Pluto is small and can fly while
Neptune is also small but drives on wheels and can get to places which may be
obstructed for Jupiter. The physical environment is a abstract combination of
multiple rooms that are connected through doors. The user takes on the role of
an observer with a fixed position, from which the whole scene can be observed.
Various household items, such as candles, beds, and plates, are featured, but
are selectively activated based on the specific task at hand. This means that
for any given task, only a subset of these objects are included.

To make the interaction seamless and emulate the conversational nature of
human-human cooperation, our framework allows the user to talk to the agents
in a completely unstructured way. The framework includes a speech-to-text-
transcription of the microphone input to provide the LLM with text-input and
uses text-to-speech-synthesis as output to interact with the user.

_Scalable architecture_ : Figure 2 illustrates the overall architecture of our
framework, which we will explain here in more detail. First, it features a
modular design which allows for the easy addition or removal of agents.
Naturally, in a situation involving multiple agents, additional challenges
arise, such as deciding which agent the user is currently talking to, how to
deal with commands that are directed at multiple agents and how to handle
continuous conversations where no new recipient is decided. We bypassed
traditional methods (like specific buttons for each agent) in favor of a
central GPT controller. In addition, each agent is powered by it’s own GPT
instance. This controller interprets user inputs and distributes them to
relevant agents (0…n) based on conversation history and context. This
decentralized approach is scalable and computationally efficient, as it limits
the need for all agents to process every input.

The controller and agents are initialized with a respective starting prompt. A
typical starting prompt is constructed as follows:

  * •

Briefing: The model is told which role it embodies and what the current
situation is, e.g., “You are the yellow robot agent Jupiter…”

  * •

Restraints: The model is instructed to stay in character and not slip back
into unwanted GPT standard behaviors, e.g., “Don’t ever break character. Don’t
ever mention that you are an AI model”.

  * •

Few-shot examples: Providing the model with a few examples of how to
“correctly” behave as part of its prompt has been proven to be an effective
way to enhance its performance (Brown et al., 2020). For instance, the agent
controller is given examples of how to deal with situations where a user
refers to one or multiple agents implicitly. As an example, the controller is
instructed that, when continuing a conversation with a single agent, “you”
probably refers to that agent but when engaged in a conversation with multiple
agents, “you” can refer to all of them.

Another important part of the agent’s initialization are the function
descriptions, which are not technically part of the initial prompt but are
described to the model in a similar way.

Figure 2: An illustration of the architecture of the introduced framework. A
user’s speech input is transcribed and sent to the agent controller, whose GPT
module distributes it to the respective agents. The agent’s GPT module, based
on its function descriptions and the state of its robot representation,
interprets the command and decides to execute a function and then responds.
The response is synthesized into audio that is received by the user. Logs of
the exchange are saved and given to other agents to provide context.

###  5.1 Implementation

For our testbed framework we used a VR environment, which was created using
the _Unity 2022.3.4f1_ editor and optimized for usage with a _Meta Quest 2_ VR
Head-Mounted Display (HMD). Srcnalt’s OpenAI-Unity package 333OpenAI-Unity
Package. https://github.com/srcnalt/OpenAI-Unity, last retrieved August 31,
2024. was used to connect Unity to the OpenAI endpoints. During the study,
user behavior was recorded with appropriate software on a _Schenker XMG Key
17_ laptop with _Windows 10 64-bit_ and _Oculus Link_ connected to the VR
headset.

The user communicates with the robot agents via speech input, which is then
transcribed by OpenAI’s Whisper v2.444OpenAI Speech to text.
https://platform.openai.com/docs/guides/speech-to-text, last retrieved August
31, 2024. The transcription is subsequently sent to the agent controller,
whose GPT555OpenAI API Reference. https://platform.openai.com/docs/api-
reference/chat, last retrieved August 31, 2024. module decides which agent the
user is addressing, based on previous messages and context information.
Lastly, their output is transformed into sound via Amazon Polly.666Amazon
Polly. https://aws.amazon.com/polly/, last retrieved August 31, 2024.

All agents employ the gpt-4-0613 model777OpenAI GPT-4 and GPT-4 Turbo.
https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo, last retrieved
August 31, 2024, which was up until November 2023 the latest GPT model
optimized for function calling. Our framework exclusively uses chat
completions888OpenAI Chat Completions API.
https://platform.openai.com/docs/guides/text-generation/chat-completions-api,
last retrieved August 31, 2024, meaning that the GPT models receive a growing
list of messages from user interactions. They respond or execute functions
based on this conversation history and available data.

Function outputs are validated by the system, and the GPT modules are updated
with context, mimicking a robot’s sensory input (like cameras and sensors) in
a real-world scenario. For instance, if an agent attempts to pick up an
object, the system evaluates the distance between the agent and the object,
updating the model with textual feedback to facilitate the task execution.
This approach simplifies the simulation’s validation process.

##  6 Study Method and Materials

To observe interaction between participants and the multi-robot systems we
conducted an exploratory within-subjects study with 12 participants. There
were 14 participants in total but the system went through significant changes
after the first two sessions. Consequently, participant 1 (P1) and P2 were
excluded from further data analysis and the remaining participants were
labeled P3 through P14 to reflect this decision.

The participants’ ages varied between 20 and 68 years. The average age (mean,
M) was 36.45 years, with a standard deviation (SD) of 14.57 years. Out of the
12 participants, 9 of them self-identified as female and 3 of them self-
identified as male. All except one of them were students or employees of the
TU Dortmund University.

###  6.1 Procedure

The study was conducted face-to-face in a small room, with the participants
sitting on a chair at the far side of a table. Before starting, participants
were briefed on the study’s objectives and the mode by which they would
communicate with the application. They willingly gave their consent for
participation, as well as for audio and video recording of the session. During
the study, participants used a Quest 2 VR headset and controlled the
application with a Quest 2 controller in their right hand, while the left
controller was used by a researcher to toggle between the different tasks. The
Quest 2 footage was streamed to a laptop for recording purposes. The laptop’s
speakers were used to play back the sound of the application, like the
synthesized voice replies of the virtual robots. The participants were free to
ask questions during the experiment but were only given answers of a pre-
defined nature or encouraged to ask the virtual robots for help.

All participants underwent the same condition, with the tasks building on one
another. The experiment spanned seven tasks in total, all sharing the same
starting position. In order to avoid any bias due to verbal or written task
descriptions, participants were given a virtual task goal screen. By comparing
that to the starting position, participants were able to derive the necessary
task instructions they had to give the robots. During the task, participants
could check back with the task goal screen at all times, which was placed
virtually below the scene. All task goals are depicted in Section 6.1:

minipage0.24

(a) Start position

Task goal 1

Task goal 2

Task goal 3

Task goal 4

Task goal 5

Task goal 6

Task goal 7

The preview images of the goal conditions that the participants needed to
reach during the study. For all tasks, the starting position is the same.
Objects not relevant to the current task are hidden.

  * •

Task 1: Neptune needs to move to the candle.

  * •

Task 2: All agents need to move to the candle.

  * •

Task 3: Jupiter needs to move to the dumbbell and pick it up, Neptune and
Pluto have to move to the fridge.

  * •

Task 4: Pluto needs to fly over to the red key, pick it up, fly to the red
door and open it. With the door open, Neptune needs to move to the yellow key,
pick it up and bring it to the user.

  * •

Task 5: Pluto needs to move to the candle. Jupiter needs to move onto the
pressure plate, opening a glass door at the back of the room for a few
seconds. While the door is open, Neptune needs to move to the chair behind the
glass door.

  * •

Task 6: Three dinner plates have to be put into the trash and all agents need
to end up next to the garbage bin.

  * •

Task 7: The robots have to flip a bed with a vase on it. The vase needs to be
picked before the bed is flipped, so it does not break. Jupiter and Neptune
must flip the bed together.

The tasks were designed to get more complex as the experiment progressed to
elicit different interaction dynamics and to promote varied communication
patterns. For example, for the first task, the participant would likely only
speak to Neptune but for the second task, they might speak to all virtual
robots at once. Task 5, for instance, required simultaneous commanding of both
Jupiter and Neptune, as the timer on the door was purposefully low enough to
where sequential commands did not seem feasible due to latency. After the
fifth task, the participants were asked to take a five-minute break. After the
seventh task, the participants were asked to fill out a questionnaire and a
subsequent semi-structured interview would be conducted. The mean time of the
experiment was 27 minutes and 28 seconds (SD $=$ 05:14 minutes) with the
subsequent interview mean time 18 minutes and 51 seconds (SD $=$ 06:32
minutes). In total, five hours and 28 minutes of experiment data and three
hours and 46 minutes of interview data were captured. Participants were given
a 15€ voucher as compensation.

###  6.2 Materials & Data Analysis

To analyze the qualitative data from the interviews, the audio recordings were
transcribed using OpenAI’s Whisper transcription tool. Likewise, the
transcriptions created as part of the study and the corresponding agent dialog
were saved as text files and extracted after the studies were over. The
transcribed records were then manually checked and corrected. Subsequently,
the transcriptions were prepared for analysis. A thematic analysis approach
was employed to identify and analyze patterns and themes within the data Braun
and Clarke (2006). We employed an iterative process in which first one of the
researchers thoroughly read and analyzed the data separately, generating
initial codes that captured the general themes and experiences of the
participants. Subsequently, two additional researchers independently analyzed
the data under an analytical lens of variable autonomy. In this step,
additional codes were generated and grouped by all three researchers to refine
and investigate overarching themes that capture the general sentiment of the
participants’ experiences. Lastly, the researcher from the first step double-
checked the transcripts of all the participants according to the refined
codes. For the questionnaire, a German version of Kate Hone and Robert
Graham’s Subjective Assessment of Speech System Interfaces (SASSI)
questionnaire (Hone and Graham, 2000) was selected. Designed specifically for
speech input systems, the SASSI aims to measure usability and gauges the
user’s perception across six factors: Accuracy, Likeability, Cognitive Demand,
Annoyance, Habitability, and Speed (Hone and Graham, 2000, p.2). This
questionnaire was selected based on recommendations from Kocabalil et al.
(Kocabalil et al., 2018). In this overview, Kocabalil et al. compare six
questionnaires for conversational interfaces and describe in detail which user
experience (UX) dimensions they cover. As a standalone questionnaire for
testing a system with conversational agents, the authors recommend the SASSI
(Kocabalil et al., 2018, p. 8). We also deemed the high amount of questions
related to the pragmatic qualities of the UX compared to other questionnaires
(Kocabalil et al., 2018, p. 7) a good fit for the study, as we were interested
in how usable and controllable the LLM-powered architecture would end up
being.

##  7 Results

Our study provides numerous insights into understanding how human operators
interact with multiple LLM-based robots. In this section, we present our
results mainly through the lens of variable autonomy. This means, we paid
particular attention towards verbal interactions between participants and the
robots that can help to understand how and to what extend a voice user
interface based on a LLM supports or hinders variable autonomy. The footage of
the study, as well as the three hours and 45 minutes of interview data, were
thematically analyzed, as described in the previous section. In the following,
we will present the themes we have found, supported by quotes from the
interview or specific interactions between participants and robots that we
observed. In total, we found four themes, see an overview below. In addition,
we will report overall aspects of user experience:

  * •

Towards a shared world model between user and robot

  * •

Instruction-based dialog

  * •

Dialog engagement vs. Command optimization

  * •

Holistic coordination vs. task decomposition

  * •

Overall aspects of User Experience

###  7.1 Towards a shared world model between user and robot

Throughout the study, an overarching process that could be observed was an
exchange between the participants and the robots about how they see the world.
These exchanges would often arise in conflict situations where the perceptions
of the participants and robots diverged. Such conflicts were sometimes caused
by user error, as shown in the following conversation P7 had with Neptune,
where there are two keys present in the scene but P7 had only noticed one of
them:  P7: Neptune, please get the key and then come to me. Neptune: Could you
please specify which key you would like me to get? There is a yellow key and a
red key. P7: The yellow key, please. Another common case was participants
using words for certain objects that differed from the descriptions the LLM
was familiar with, such as P14 and P9 using the word “disc” and then being
informed by the LLM that the object is a “plate”. Interestingly, there were
also some conflicts in which the participants had to convince the robots that
their perception of the world was incorrect, as shown in the following
interaction between P12 and Neptune, in which Neptune assumes that the yellow
key is still unreachable until P12 convinces them otherwise:  P12: Neptune,
please get the yellow key. Neptune: I’m sorry, but I currently can’t reach the
yellow key. It’s behind a glass pane that I can’t cross. Is there another task
I can do for you? P12: ((Opens a door through the pane with the help of
Pluto)) P12: Neptune, please get the yellow key. Neptune: Sorry, but I have
already tried to reach the yellow key, and I can’t. The key is behind a glass
pane that I can’t cross. Is there another task you would like? P12: I have had
the glass wall opened. You can go through now. Neptune: ((Moves to the yellow
key and picks it up)) I have successfully picked up the yellow key. What
should I do next? In the interviews, many participants expressed that the
robots’ explanation of their world view was helpful and made their actions
more understandable, as shown by quotes like “the feedback was helpful. They
informed me about their understanding of commands and their perception of the
environment, allowing me to align it with my own” by P11. However, at times,
the LLM would hallucinate and convince the participants of things that were
not true about the world and engage in repeating or oversharing of information
that was not immediately relevant to the task, as P7 expresses here: “When the
robots couldn’t do something, they always explained, ’I can’t do that’, and
then they always added, ’I can only do this and that.’ And I mean, I know by
now what you can do, it’s enough if you just say, ’I can’t do that right now,’
then I’ll think of something else.”

###  7.2 Instruction-based dialog

While we did not instruct participants to engage with robots in a specific
way, all of them considered the robots as recipients of instructions, which
they in turn had to provide. Interestingly, however, some participants applied
this in a very command-like style. For example, P14 said “Pluto, Jupiter,
Neptune all of you move towards the candle”. Others instead opted for a more
humane and conversation-like language, e.g., P7 asking Neptune “can you move
next to the candle?” and further politely advising Jupiter to do the same
“Jupiter, please move next to the candle”. Interestingly, this also had an
impact on the way users engaged in the conversation. While the command-style
participants aimed to formulate the command in the best way straight away, the
conversation-like participants sometimes started talking and figuring out the
exact instruction on the fly, resulting also in situations where they needed
to correct them, e.g., P11, when asking Jupiter to pick up the plates
“Jupiter, please drive to the plates in the room and take, [thinking aloud] I
think there are three, [resumes instruction language] and take three plates
and throw them in the garbage bin.” This second group of participants in
general was more likely to ask the robots for advice, again P11 after a robot
had picked up a key “Can you do anything with the key? Can you place it on the
key sign?”. But overall, such questions remained the exception, as
participants did not seem to regard the robots as equal conversational
partners. This was also a point made in the interviews, where participants
mentioned that it felt like talking to young children and that they felt the
need to be very specific and precise in what they want from them, as evidenced
by statements like “it’s similar to interacting with a young child, where you
have to give very specific instructions and keep it concise” by P4 and “the
experience felt somewhat like talking to young children, where you try to keep
things simple in your communication” by P5. We believe this may have been
amplified by the fact that the LLM-based robots were happy to provide lengthy
explanations and asked for clarifications whenever they were not sure what to
do. This also happened in situations, where participants did not expect the
robots to have any problems understanding what they were told to do, e.g., P9
being surprised at the robot’s response when being asked to pick up an object
it was currently not next to: “When the system mentioned that an object was
out of its reach and asked for permission to approach it first, I found it
unintuitive. I expected it to act on the instruction directly.”

###  7.3 Dialog engagement vs. Command optimization

We observed two main ways in which the participants reacted to the robots’
answers: Often, the robots would offer a dialogue by ending their sentences
with questions like “[…] Should I do that?”. While some users accepted the
offer and replied, others simply scanned the robots’ answer for relevant
information to come up with a new disconnected and optimized command. For
instance, when P6 and P9 asked Jupiter to place the vase next to the bed and
Jupiter responded with “I can only place the vase on specific surfaces, like
the bed or the ground. Can you specify where I should place it?”, P6 responded
with “on the ground” while P9 responded with “Jupiter, please place the vase
on the ground”. While P6 engaged directly with the robot’s question, P9 issued
a standalone command, independent of the prior dialogue, effectively resetting
the interaction. Based on information from the interviews, a big reason as to
why participants did not respond to the robots’ questions was a lack of trust
in their ability to understand them. P14 alludes to this, while also
mentioning the aspect of delay as a possible cause: “I perceived that they
were offering a dialogue by asking, but I didn’t believe they would understand
me if I replied. I think it was also partly because, between their question
and my answer, time had passed, making me think ’their memory was wiped’”. P6
also explicitly mentioned that their initial idea was to always say the names
of the robots and the commands each time they spoke. This idea was reinforced,
as the system’s inability to decide the right recipient (which is very
unlikely if the names are mentioned), can be a costly mistake.

###  7.4 Holistic coordination vs. task decomposition

We observed significant differences in the complexity of the participants’
requests, especially when it comes to tasks that can be completed with
multiple robots. Some participants seemed to prefer breaking down tasks into
simple steps, and wait for the robots to complete one before moving on to the
next step. On the other hand, some participants issued strategic multi-robot
commands, giving each robot one or multiple steps to execute simultaneously.
Generally, the participants who used more complex language seemed more
inclined to try more elaborate instructions. P11, for instance, issued complex
commands like “first to you, Pluto. Put down the weight and fly to the
refrigerator. To you, Neptune. Move in front of the refrigerator. To you,
Jupiter. Pick up the weight after Pluto has put it down” and remarked in the
interview that their initial impression that each robot had to be addressed
individually had been subverted, and the instructions could be surprisingly
elaborate and target multiple robots at once. Conversely, other participants
broke down the tasks into the smallest possible steps and issued commands to
each robot separately, such as P6 and P13. One reason for this seemed to be a
lack of faith in the robots’ abilities to understand longer commands, or a
preconceived notion that the robots would perform better when given simple
orders. This coincides with the interviews, in which P6 likened their default
mode of communicating to how they would talk to a dog and P13 mentioned that
they tried to be “clear and concise” from the beginning. A similar
interpretation was expressed by P4, who stated that “I always had to think
beforehand about what to say, to say it briefly and precisely, and yes, very
clearly. I had to put myself in the system’s shoes, guessing what it probably
expected in the way I spoke.” P4 was one of the participants who used very
simple language from the beginning and intuitively broke down tasks into
simple steps, likely expecting that the system would not be able to handle
longer sequences directed at multiple robots. These patterns were exacerbated
by the LLM’s tendency to mirror the user’s way of talking, responding more
“robotically” when given “robotic” instructions. Additionally, in response to
failure situations, it is not unreasonable to assume that participants would
tone down the complexity of their commands. However, our data suggests that
the participants generally decided on the complexity of their requests from
the start and did not adjust its overall nature during the study, even in
response to robot failure.

###  7.5 Overall aspects of User Experience

In this section, we will present a collection of sub-themes describing the
user’s experiences with the system and their positive and negative impressions
of it according to the interviews. These sub-themes range from the degree to
which the system was lenient with input, to system feedback and
responsiveness. At the end, we will present an overview of the SASSI factors
to corroborate these findings.

####  7.5.1 Semantic flexibility

Participants appreciated that there was a high degree of flexibility to the
words they were able to use. e.g., P14 said “I found the system’s
understanding impressive. I used various words like ’move’, ’go there’, ’fly
there’, ’proceed there’, and I believe they were all understood. I didn’t feel
tied to a specific word, which made the interaction feel more natural.” P3
echoed this statement, while also appreciating that the system did not have
any issues with pauses during a request or exchanging pleasantries like
“please” and “thank you”. It was expected that the LLM, with its wealth of
linguistic knowledge, would be fairly flexible when it comes understanding
variations of words and commands. However, the results indicate that the LLM
was rather meticulous when it came to words that are part of its function
descriptions, sometimes refusing to do something if, for example, the name of
an object was slightly off, as shown in the following exchange between P11 and
Pluto:  P11: Pluto, please go to the blue trash can. Pluto: Sorry, I can only
go to locations that are defined in my functions. A blue trash can is
unfortunately not on my list. However, I can move to the trash bin, if that
helps. Should I do that? This is further substantiated by the interview data.
P9 states that “they did execute the commands, but sometimes they were very
meticulous”. While the example above is an outlier and the robots could
generally make that connection, it was apparent that they paid special
attention to terms outlined by their functions. This phenomenon and the fact
that participants were not provided with any guidelines as to which words to
choose is also reflected in the SASSI results. Question 29, which states “I
sometimes wondered if I was using the right word”, received an average score
of 5.92 (M = 5.92, SD = 1.73) out of 7, where 7 indicates ’strongly agree’.

####  7.5.2 Verbal robot feedback

The verbal feedback by the robots was generally regarded as helpful.
Participants mentioned that the clarifications provided by the robots made
tasks easier and the robots’ decisions more understandable. For example, P9
stated in their interview that “the robots behaved very understandably. Like
with the key handover, Neptune couldn’t directly take the key from Pluto but
could pick it up from the ground. He explained that, so it was clear what I
had to do.” In a similar vein, P6 called the feedback helpful and appreciated
that the robots corrected their nomenclature in cases where P6 could not
visually discern what type of object was in front of them. However, P6 also
remarked that the feedback was too long for their taste, which aligns with
statements made by P13 and P14.

####  7.5.3 Mismatch regarding expected robot autonomy

Participants reported that they sometimes expected more autonomous behavior by
the robots, especially in situations that required a sequence of tasks to be
executed. As it was up to the discretion of the non-deterministic LLM to
decide between calling a function or asking a user for clarification before
proceeding, there was a certain amount of variance to the autonomy with which
the robots executed given tasks. For example, P3 expressed: “In certain
situations, I didn’t realize I had to explicitly instruct the system to
approach an object before picking it up. With humans, if I say ’pick up the
key’, I’d expect them to walk over to the key and then pick it up.” This was
referring to a situation in which P3 had asked Jupiter to pick up the key, to
which Jupiter responded by asking for permission to approach the key first. P6
and P9 echoed this sentiment, expressing a similar surprise about the robots
asking before executing a pre-requisite task. However, there were also
occurrences where a robot would by itself decompose a task into necessary sub-
tasks and perform them autonomously. For example, when P8 asked Jupiter to
“take three plates and throw them in the trash bin”, the robot executed the
correct function-triplet (consisting of movement, pick up, and throw away)
three times in succession.

####  7.5.4 Inherent response-lag

A common complaint lodged by the participants was that the system took too
long to answer. In the application, there was a visual cue to indicate the
status of the transcription and which robots were currently processing it to
let the users know that they had been heard. However, there was an amount of
delay before the robots’ responses that was unlike an inter-human
conversation, where feedback is usually instant. This led to cases in which
participants started talking without waiting for the responses, as described
by P9: “In terms of naturalness, the robots’ feedback always came late. There
was a lot of narration, and by the time they responded, I had already
continued speaking”. This is reflected by the “Speed” dimension of the SASSI,
where the system received an average score of 5.25 (M = 5.25, SD = 1.57) (see
Table 1), with 7 being the slowest. The reason for this behavior are due to
the technical setup and we will discuss them in more detail in the discussion
section.

####  7.5.5 SASSI factors

For an overview, Table 1 shows the different dimensions of the SASSI
questionnaire, each on a scale from 1 to 7, averaged across the 12
participants with standard deviation.

Table 1: Descriptive statistics for individual SASSI dimensions.  | System Response Accuracy | Likeability | Cognitive Demand | Annoyance | Habitability | Speed
---|---|---|---|---|---|---
$M$ | 4.43 | 5.36 | 4.88 | 3.80 | 4.44 | 5.25
$SD$ | 0.95 | 1.01 | 1.02 | 1.44 | 1.22 | 1.57

##  8 Discussion

Based on our findings, it is apparent that users generally do not perceive
robots as equal conversation partners. Conversations between the participants
and the robots often boiled down to simple instructions and some users
effectively ignored the robot’s part in the conversation, despite the much
more elaborate conversational capabilities of GPT. Robot error and the overly
meticulous nature of the LLM in some cases could have been contributing
factors to a user’s low expectations of the robot’s abilities. We believe that
to allow for smooth cooperation, the LLM must be aligned in such a way that
its decisions are as understandable to the user as another human’s. While our
data shows attempts on the side of the LLM to stay as aligned as possible by
frequently communicating and asking questions, the LLM was almost “too
communicative”, leading to a mismatch regarding the expected robot autonomy.
It is clear that a deeper, intuitive understanding is required on the LLM’s
side so that the LLM’s and the human’s conceptual models can be matched
closely.

###  8.1 Lessons Learned

Function calls are a double-edged sword: Function calls, with their reliable
formatting and consistent nature, turned out to be very effective at
connecting the language-based GPT-model to coded robot processes. Functions
are highly modular and they are very adjustable, making them a good fit for an
experimental framework that can hold multiple agents with vastly different
abilities. This feature shows promise, even when applied to more elaborate
scenarios, as function descriptions can be complex with multiple nested
properties. The function feature also displayed the ability to confine the LLM
more effectively than a textual ontology approach. However, the feature
created other issues in the model’s behaviors. For instance, the cases in
which the model was overly meticulous illustrated in Section 7.5.1 were not
conducive to intuitive cooperation. Likewise, the model’s refusal to execute a
function and ask the user for permission needlessly, as shown in Section 7.5.3
could have been a result of the function feature. Due to the opaque nature of
LLMs it is difficult to say what exactly caused these particular problems.
Overall, we believe that functions have proven to be an effective way for
prototyping and testing GPT’s abilities to make decisions that can be
translated into robot actions. GPT displays flaws as a solitary controller:
Our results suggest that GPT as a solitary controller for an actual robot is
not yet feasible, especially when it comes to usability concerns. Perhaps the
most glaring issue of GPT as a solitary controller is its inherently non-
deterministic nature. Even when the model is given a low temperature (a
parameter which controls the randomness and creativity of the responses), the
outputs are too inconsistent to act as a governing element for a real robot.
Additional modules would have to be developed to support it, if consistency is
to be ensured. One of GPT’s greatest advantages, its massive parameter count
which endows it with a wealth of linguistic knowledge, also limits its
deployment to a physical robot. Being a cloud-based service, the GPT API
introduces latency which is exacerbated by problems with the internet
connection. The system’s slowness is reflected within the SASSI’s factor 7, as
well as the statements made by the participants during the interviews. Aside
from negatively impacting the user experience, instant feedback is essential
to language-based communication, especially during tasks where time-critical
cooperation is required. Calibration could be very beneficial: We believe that
a calibration process, in which a user and the LLM clarify what their
preferred way of communicating is, could be helpful. Our data suggests that,
intuitively, humans do not expect that they could influence a robot with this
kind of meta-communication. A calibration process could not only help to
improve the user’s initial expectations of the robot, but also prevent
unwanted behaviors like overly long explanations. A competent sensory system
for the robots is essential: During communication, certain assumptions arise
about what the conversation partner knows, sees and feels. What we have seen
in our study is the importance of the LLM’s conceptual model to be aligned to
the user’s. For one, this will necessitate the inclusion of non-verbal
communication, such as pointing. Additionally, an improved sensory system
would allow for more varied and flexible interpretation of user requests. The
more data the LLM has access to when deciding whether to execute an action,
the more informed its decision will be. Supplying the LLM with sensor and
camera information from a physical robot therefore shows a lot of potential
but will bring a new set of challenges. Additional avenues for control are
necessary: Intervention is very important when interacting with a robot that
can act autonomously and perhaps even more so when interacting with multiple.
Our study indicated an additional need for intervention as well. The issue
with intervention in this context is that, due to the processing time of the
transcription and GPT modules, the regular channels of communication between
the user and the robot are unfit for intervening. Therefore, additional
possibilities of intervention will have to be explored. An intermediary
solution would be the introduction of a button that immediately stops the
robot regardless of the LLM’s current state. Another addition to improve the
controllability of the system would be to make the processes of the LLM less
opaque and find ways to visualize its current state. This will be especially
important as task complexity and robot complexity grow. Inter-robot
communication is needed: While the framework shows promise in allowing a user
to interact with multiple robots simultaneously in an unstructured way, it is
clear now that additional communication between the robots themselves is
required. While the robots share a general sense of what their fellow robots
are doing, textual representation and a unified log of conversations is
insufficient, especially during tasks where the robots have to collaborate.
Their responses should also be clustered in a way that the response to the
user is streamlined and free of repetitions. The nature of our framework would
support such approaches, with the agent controller potentially taking a more
elaborate part in controlling agent-to-agent communication and interaction.
The agents themselves could potentially exchange function calls with each
other, allowing for synchronized and planned cooperation towards certain
goals.

###  8.2 Limitations

There are a few noteworthy implementations that pertain to the framework, as
well as the study the framework was used in. Utilizing an LLM comes with
certain LLM-inherent limitations. First off, due to their opaque nature,
programs utilizing LLMs are notoriously hard to debug. While there were some
unforeseen consequences caused by unexpected LLM behavior, resulting bugs
during the study only caused minor inconveniences and all participants were
able to solve the tasks successfully. The use of German language could have
been a possible cause for some of these unexpected behaviors, especially in
regards to functions - although the objects described in the function
descriptions were deliberately given German names to prevent the LLM from
disambiguating. Additionally, the framework inherits other LLM-related
limitations, such as possible biases in training data and errors being
propagated forward, as false output becomes part of future input. Another
limitation arose from the University-based internet connection used for the
study. While the cloud services employed in this application inherently
involve some latency, the sometimes unreliable internet connection further
compounded the issue. While there were no disconnects or complete outages, the
delays in transcriptions and model responses were at times noticeably longer,
in contrast to quicker performance observed with a stable internet connection.

##  9 Conclusion

This study makes a significant contribution to understanding the integration
of LLMs such as GPT in human-robot teaming and the potential of language as
means to facilitate variable autonomy. Our findings indicate that despite the
advanced capabilities of LLMs, user perception and interaction with robots
present challenges. The study highlights that users do not perceive robots as
equal conversational partners, leading to a communication limited to simple
instructions. Still, the study also showcased more elaborate conversation for
those users who did not expect preconceived limitations in the conversational
abilities of the robot counterparts. A critical aspect is the alignment of the
LLM to make decisions understandable and predictable for the user. The over-
communication by the LLM and the resulting ambiguity about robot autonomy
demonstrate the need for a more balanced approach. Furthermore, the study
underscores the importance of a calibration process to align communication
preferences between users and LLM. The study also reveals that using GPT as a
sole controller for robots has limitations, primarily due to its non-
deterministic nature and latency issues from cloud connectivity. This
underscores the necessity of considering LLMs as part of a broader system that
incorporates both user feedback and sensor information. In conclusion, our
research provides valuable insights into the dynamics of human-robot teaming,
indicating a significant need for further research and development to
effectively deploy LLMs in such systems. Our findings suggest that a
multidisciplinary approach, encompassing technology, user experience, and
psychological aspects of interaction, is required to fully realize the
potentials of LLMs for variable autonomy in human-robot teaming.

## References

  * Abbink et al. (2018) Abbink, D. A., Carlson, T., Mulder, M., de Winter, J. C. F., Aminravan, F., Gibo, T. L., et al. (2018).  A topology of shared control systems—finding common ground in diversity.  _IEEE Transactions on Human-Machine Systems_ 48, 509–525.  10.1109/thms.2018.2791570
  * Ahn et al. (2022) Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., et al. (2022).  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_
  * Alessa and Al-Khalifa (2023) Alessa, A. and Al-Khalifa, H. (2023).  Towards designing a chatgpt conversational companion for elderly people.  _arXiv preprint arXiv:2304.09866_
  * Arumugam et al. (2019) Arumugam, D., Karamcheti, S., Gopalan, N., Williams, E. C., Rhee, M., Wong, L. L. S., et al. (2019).  Grounding natural language instructions to semantic goal representations for abstraction and generalization.  _Autonomous Robots_ 43, 449–468.  10.1007/s10514-018-9792-8
  * Ausat et al. (2023) Ausat, A. M. A., Massang, B., Efendi, M., Nofirman, N., and Riady, Y. (2023).  Can chat gpt replace the role of the teacher in the classroom: A fundamental analysis.  _Journal on Education_ 5, 16100–16106
  * Braun and Clarke (2006) Braun, V. and Clarke, V. (2006).  Using thematic analysis in psychology.  _Qualitative research in psychology_ 3, 77–101
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., et al. (2020).  Language models are few-shot learners.  _Advances in neural information processing systems_ 33, 1877–1901
  * Bustamante et al. (2021) Bustamante, S., Quere, G., Hagmann, K., Wu, X., Schmaus, P., Vogel, J., et al. (2021).  Toward seamless transitions between shared control and supervised autonomy in robotic assistance.  _IEEE Robotics and Automation Letters_ 6, 3833–3840.  10.1109/LRA.2021.3064449
  * Canal et al. (2016) Canal, G., Alenyà, G., and Torras, C. (2016).  Personalization framework for adaptive robotic feeding assistance.  In _Social Robotics_ (Cham: Springer International Publishing), vol. 9979. 22–31.  10.1007/978-3-319-47437-3_3
  * Chen et al. (2021) Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., et al. (2021).  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_
  * Chiou et al. (2023) Chiou, M., Booth, S., Lacerda, B., Theodorou, A., and Rothfuß, S. (2023).  Variable autonomy for human-robot teaming (vat).  In _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ (New York, NY, USA: Association for Computing Machinery), HRI ’23, 932–934.  10.1145/3568294.3579957
  * Clark (1996) Clark, H. H. (1996).  _Using language_ (Cambridge university press)
  * Cyphert (2021) Cyphert, A. B. (2021).  A human being wrote this law review article: Gpt-3 and the practice of law.  _UC Davis L. Rev._ 55, 401
  * Deci and Ryan (2012) Deci, E. L. and Ryan, R. M. (2012).  Self-determination theory.  _Handbook of theories of social psychology_ 1, 416–436
  * Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_
  * Erdogan and Argall (2017) Erdogan, A. and Argall, B. (2017).  The effect of robotic wheelchair control paradigm and interface on user performance, effort and preference: An experimental assessment.  _Robotics and Autonomous Systems_ 94, 282–297.  https://doi.org/10.1016/j.robot.2017.04.013
  * Firat (2023) Firat, M. (2023).  How chat gpt can transform autodidactic experiences and open education.  _Department of Distance Education, Open Education Faculty, Anadolu Unive_
  * Flemisch et al. (2019) Flemisch, F., Abbink, D. A., Itoh, M., Pacaux-Lemoine, M.-P., and Wessel, G. (2019).  Joining the blunt and the pointy end of the spear: towards a common framework of joint action, human–machine cooperation, cooperative guidance and control, shared, traded and supervisory control.  _Cognition, Technology & Work_ 21, 555–568
  * Fuentes et al. (2023) Fuentes, C., Porcheron, M., and Fischer, J. E. (2023).  Roboclean: Contextual language grounding for human-robot interactions in specialised low-resource environments
  * Gallenberger et al. (2019) Gallenberger, D., Bhattacharjee, T., Kim, Y., and Srinivasa, S. S. (2019).  Transfer depends on acquisition: Analyzing manipulation strategies for robotic feeding.  In _2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. 267–276.  10.1109/HRI.2019.8673309
  * Garrod and Pickering (2004) Garrod, S. and Pickering, M. J. (2004).  Why is conversation so easy?  _Trends in cognitive sciences_ 8, 8–11
  * Grudin (2022) Grudin, J. (2022).  _From tool to partner: The evolution of human-computer interaction_ (Springer Nature)
  * Händler (2023) Händler, T. (2023).  Balancing autonomy and alignment: A multi-dimensional taxonomy for autonomous llm-powered multi-agent architectures.  _arXiv preprint arXiv:2310.03659_
  * Hirosawa et al. (2023) Hirosawa, T., Harada, Y., Yokose, M., Sakamoto, T., Kawamura, R., and Shimizu, T. (2023).  Diagnostic accuracy of differential-diagnosis lists generated by generative pretrained transformer 3 chatbot for clinical vignettes with common chief complaints: A pilot study.  _International journal of environmental research and public health_ 20, 3378
  * Hone and Graham (2000) Hone, K. S. and Graham, R. (2000).  Towards a tool for the subjective assessment of speech system interfaces (sassi).  _Natural Language Engineering_ 6, 287–303
  * Irfan et al. (2023) Irfan, B., Kuoppamäki, S.-M., and Skantze, G. (2023).  Between reality and delusion: Challenges of applying large language models to companion robots for open-domain dialogues with older adults
  * Kahambing (2023) Kahambing, J. G. (2023).  Chatgpt, public health communication and ‘intelligent patient companionship’.  _Journal of public health_ , fdad028
  * Kelly et al. (2023) Kelly, D., Chen, Y., Cornwell, S. E., Delellis, N. S., Mayhew, A., Onaolapo, S., et al. (2023).  Bing chat: The future of search engines?  _Proceedings of the Association for Information Science and Technology_ 60, 1007–1009
  * Kepuska and Bohouta (2018) Kepuska, V. and Bohouta, G. (2018).  Next-generation of virtual personal assistants (microsoft cortana, apple siri, amazon alexa and google home).  In _2018 IEEE 8th annual computing and communication workshop and conference (CCWC)_ (IEEE), 99–103
  * Kim et al. (2012) Kim, D.-J., Hazlett-Knudsen, R., Culver-Godfrey, H., Rucks, G., Cunningham, T., Portee, D., et al. (2012).  How autonomy impacts performance and satisfaction: Results from a study with spinal cord injured subjects using an assistive robot.  _IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans_ 42, 2–14.  10.1109/TSMCA.2011.2159589
  * Kocabalil et al. (2018) Kocabalil, A. B., Laranjo, L., and Coiera, E. (2018).  Measuring user experience in conversational interfaces: a comparison of six questionnaires.  In _Proceedings of the 32nd International BCS Human Computer Interaction Conference 32_. 1–12
  * Kodur et al. (2023) Kodur, K., Zand, M., Tognotti, M., Jauregui, C., and Kyrarini, M. (2023).  Structured and unstructured speech2action frameworks for human-robot collaboration: A user study
  * Koubaa (2023a) Koubaa, A. (2023a).  Gpt-4 vs. gpt-3.5: A concise showdown
  * Koubaa (2023b) Koubaa, A. (2023b).  Rosgpt: Next-generation human-robot interaction with chatgpt and ros.  _Preprints_ 10.20944/preprints202304.0827.v2
  * Latikka et al. (2021) Latikka, R., Savela, N., Koivula, A., and Oksanen, A. (2021).  Attitudes toward robots as equipment and coworkers and the impact of robot autonomy level.  _International Journal of Social Robotics_ 13, 1747–1759
  * Lauretti et al. (2017) Lauretti, C., Cordella, F., Guglielmelli, E., and Zollo, L. (2017).  Learning by demonstration for planning activities of daily living in rehabilitation and assistive robotics.  _IEEE Robotics and Automation Letters_ 2, 1375–1382
  * Lee et al. (2023) Lee, Y. K., Jung, Y., Kang, G., and Hahn, S. (2023).  Developing social robots with empathetic non-verbal cues using large language models.  _arXiv preprint arXiv:2308.16529_
  * Lekova et al. (2023) Lekova, A., Tsvetkova, P., and Andreeva, A. (2023).  System software architecture for enhancing human-robot interaction by conversational ai.  In _2023 International Conference on Information Technologies (InfoTech)_ (IEEE), 1–6
  * Liu and Zhang (2019) Liu, R. and Zhang, X. (2019).  A review of methodologies for natural-language-facilitated human–robot cooperation.  _International Journal of Advanced Robotic Systems_ 16, 1729881419851402
  * Misra et al. (2016) Misra, D. K., Sung, J., Lee, K., and Saxena, A. (2016).  Tell me dave: Context-sensitive grounding of natural language to manipulation instructions.  _The International Journal of Robotics Research_ 35, 281–300.  10.1177/0278364915602060
  * Nakano et al. (2021) Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., et al. (2021).  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_
  * Neelakantan et al. (2022) Neelakantan, A., Xu, T., Puri, R., Radford, A., Han, J. M., Tworek, J., et al. (2022).  Text and code embeddings by contrastive pre-training.  _arXiv preprint arXiv:2201.10005_
  * Nilsson et al. (1984) Nilsson, N. J. et al. (1984).  Shakey the robot
  * Nori et al. (2023) Nori, H., King, N., McKinney, S. M., Carignan, D., and Horvitz, E. (2023).  Capabilities of gpt-4 on medical challenge problems.  _arXiv preprint arXiv:2303.13375_
  * Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., et al. (2022).  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35, 27730–27744
  * Park et al. (2020) Park, D., Hoshi, Y., Mahajan, H. P., Kim, H. K., Erickson, Z., Rogers, W. A., et al. (2020).  Active robot-assisted feeding with a general-purpose mobile manipulator: Design, evaluation, and lessons learned.  _Robotics and Autonomous Systems_ 124, 103344.  10.1016/j.robot.2019.103344
  * Pascher et al. (2024) Pascher, M., Goldau, F. F., Kronhardt, K., Frese, U., and Gerken, J. (2024).  AdaptiX – A Transitional XR Framework for Development and Evaluation of Shared Control Applications in Assistive Robotics.  _Proc. ACM Hum.-Comput. Interact._ 8\.  Preprint on arXiv: https://arxiv.org/abs/2310.15887
  * Pascher et al. (2023a) Pascher, M., Gruenefeld, U., Schneegass, S., and Gerken, J. (2023a).  How to communicate robot motion intent: A scoping review.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems - CHI ’23_.  10.1145/3544548.3580857
  * Pascher et al. (2023b) Pascher, M., Kronhardt, K., Goldau, F. F., Frese, U., and Gerken, J. (2023b).  In Time and Space: Towards Usable Adaptive Control for Assistive Robotic Arms.  In _2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_ (IEEE), 2300–2307.  10.1109/RO-MAN571019.2023.10309381
  * Perlman (2022) Perlman, A. (2022).  The implications of chatgpt for legal services and society.  _Available at SSRN 4294197_
  * Petroni et al. (2019) Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., et al. (2019).  Language models as knowledge bases?  _arXiv preprint arXiv:1909.01066_
  * Pollak et al. (2020) Pollak, A., Paliga, M., Pulopulos, M. M., Kozusznik, B., and Kozusznik, M. W. (2020).  Stress in manual and autonomous modes of collaboration with a cobot.  _Computers in Human Behavior_ 112, 106469.  https://doi.org/10.1016/j.chb.2020.106469
  * Porcheron et al. (2017) Porcheron, M., Fischer, J. E., and Sharples, S. (2017).  ” do animals have accents?” talking with agents in multi-party conversation.  In _Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing_. 207–219
  * Radford et al. (2018) Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. (2018).  Improving language understanding by generative pre-training
  * Rakhimkul et al. (2019) Rakhimkul, S., Kim, A., Pazylbekov, A., and Shintemirov, A. (2019).  Autonomous object detection and grasping using deep learning for design of an intelligent assistive robot manipulation system.  In _2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)_ (IEEE), 3962–3968
  * Rao et al. (2023) Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., and Succi, M. D. (2023).  Evaluating chatgpt as an adjunct for radiologic decision-making.  _medRxiv_ , 2023–02
  * Sebanz et al. (2006) Sebanz, N., Bekkering, H., and Knoblich, G. (2006).  Joint action: bodies and minds moving together.  _Trends in cognitive sciences_ 10, 70–76
  * Shea et al. (2023) Shea, Y.-F., Lee, C. M. Y., Ip, W. C. T., Luk, D. W. A., and Wong, S. S. W. (2023).  Use of gpt-4 to analyze medical records of patients with extensive investigations and delayed diagnosis.  _JAMA Network Open_ 6, e2325000–e2325000
  * Stiennon et al. (2020) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., et al. (2020).  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ 33, 3008–3021
  * Swanson et al. (2021) Swanson, B., Mathewson, K., Pietrzak, B., Chen, S., and Dinalescu, M. (2021).  Story centaur: Large language model few shot learning as a creative writing tool.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , eds. D. Gkatzia and D. Seddah (Stroudsburg, PA, USA: Association for Computational Linguistics), 244–256.  10.18653/v1/2021.eacl-demos.29
  * Tack and Piech (2022) Tack, A. and Piech, C. (2022).  The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues.  _arXiv preprint arXiv:2205.07540_
  * Trott et al. (2015) Trott, S., Appriou, A., Feldman, J., and Janin, A. (2015).  Natural language understanding and communication for multi-agent systems.  In _2015 AAAI Fall Symposium Series_
  * Trozze et al. (2023) Trozze, A., Davies, T., and Kleinberg, B. (2023).  Large language models in cryptocurrency securities cases: Can chatgpt replace lawyers?  _arXiv preprint arXiv:2308.06032_
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., et al. (2017).  Attention is all you need.  _Advances in neural information processing systems_ 30
  * Vesper et al. (2010) Vesper, C., Butterfill, S., Knoblich, G., and Sebanz, N. (2010).  A minimal architecture for joint action.  _Neural Networks_ 23, 998–1003
  * Waisberg et al. (2023) Waisberg, E., Ong, J., Masalkhi, M., Kamran, S. A., Zaman, N., Sarker, P., et al. (2023).  Gpt-4: a new era of artificial intelligence in medicine.  _Irish Journal of Medical Science (1971-)_ , 1–4
  * Winograd (1972) Winograd, T. (1972).  Understanding natural language.  _Cognitive Psychology_ 3, 1–191.  10.1016/0010-0285(72)90002-3
  * Woods (1973) Woods, W. A. (1973).  Progress in natural language understanding.  In _Proceedings of the June 4-8, 1973, national computer conference and exposition on - AFIPS ’73_ (New York, New York, USA: ACM Press).  10.1145/1499586.1499695
  * Yang et al. (2019) Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., and Le, Q. V. (2019).  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ 32
  * Złotowski et al. (2017) Złotowski, J., Yogeeswaran, K., and Bartneck, C. (2017).  Can we control it? Autonomous robots threaten human identity, uniqueness, safety, and resources.  _International Journal of Human-Computer Studies_ 100, 48–54

  *[GPT]: Generative pre-trained transformer
  *[VR]: Virtual Reality
  *[NLP]: Natural Language Processing
  *[AI]: Artificial Intelligence
  *[LLMs]: Large Language Model
  *[BERT]: Bidirectional Encoder Representations from Transformers
  *[LLM]: Large Language Model
  *[NLI]: Natural language interface
  *[API]: Application Programming Interface
  *[ROS]: Robot Operating System
  *[IPAs]: Intelligent Personal Assistant
  *[HMD]: Head-Mounted Display
  *[SASSI]: Subjective Assessment of Speech System Interfaces
  *[UX]: user experience