# ON DESIGNING LIGHT-WEIGHT OBJECT TRACKERS THROUGH
NETWORK PRUNING: USE CNNS or TRANSFORMERS?

###### Abstract

Object trackers deployed on low-power devices need to be light-weight,
however, most of the current state-of-the-art (SOTA) methods rely on using
compute-heavy backbones built using CNNs or transformers. Large sizes of such
models do not allow their deployment in low-power conditions and designing
compressed variants of large tracking models is of great importance. This
paper demonstrates how highly compressed light-weight object trackers can be
designed using neural architectural pruning of large CNN and transformer based
trackers. Further, a comparative study on architectural choices best suited to
design light-weight trackers is provided. A comparison between SOTA trackers
using CNNs, transformers as well as the combination of the two is presented to
study their stability at various compression ratios. Finally results for
extreme pruning scenarios going as low as 1% in some cases are shown to study
the limits of network pruning in object tracking. This work provides deeper
insights into designing highly efficient trackers from existing SOTA methods.

Index Terms—  object tracking, light-weight, convolutional network,
transformers, network pruning

##  1 Introduction

A general consensus about object tracking is that those using deeper
convolutional neural network (CNN) architectures help in better discrimination
of the target from the background and this is verified in siamese tracking
networks. SiamFC [1] uses an AlexNet-variant backbone whereas SiamRPN++ [2]
and SiamRCNN [3] use ResNet50 and ResNet101 [4] backbones. As expected, the
performance of SiamRPN++ is significantly better than SiamFC and SiamRCNN
outperform the other two. Further in this landscape, we have deep transformer
models such as OSTrack [5] and STARK [6] that outperform all the CNN trackers
stated above. While deeper models, be it CNN-based or transformer-based [7,
8], are found to help deliver better performance, they are generally large in
size and often slow in terms of inference speed. This can be a big setback,
especially for applications where close to real-time processing of the video
frames is desired. There exist cases where beyond good accuracy, inference
time on deployment device is of more importance.

In this paper, we focus on building light-weight object trackers that can be
deployed on small devices and can operate in low-power conditions. Previous
work on similar lines includes [9], [10], and [11]. For fast and light-weight
trackers, the existing dense trackers need to be compressed, and it is of
interest to study how CNNs and transformers behave in such scenarios. To the
best of our knowledge, our work is the first to identify which tracker model
to choose for compression when designing a light-weight tracker. Beyond
accuracy and inference speed, stability in the performance of the model is
also important when building light trackers. It is of great interest to put
all the SOTA trackers under the common roof of model size and the associated
inference compute and study the drop in their performance for extreme model
compression.

Contributions (1) We present a comparative study on how the performance of
SOTA object tracking methods is affected by different compression ratios. For
this purpose, we choose three popular tracking methods: a fully CNN-based
(Super-DiMP [12]), a fully transformer-based (OSTrack [5]), and a hybrid CNN-
transformer model (STARK [6]). (2) We study the effect of model compression on
each tracker independently, as well as perform a comparative study across them
for different budgets of inference computing. (3) For the robustness study, we
analyze the inference of the compressed models on multiple tracking datasets
and study the precision and success curves. (4) For the trackers to be really
light-weight, the demanding level of compression is really high, and this
implies that we need to push for extremely low inference budgets. In this
regard, we also study the stability of the models at inference budgets of as
low as 1% of the original base model.

##  2 Pruning in Tracking

###  2.1 Background

Model pruning [13, 14, 15, 16, 17] refers to inducing sparsity into the
architecture of the model through removing the undesired or less important
weights or filters. We present here a $pruning$-$in$-$tracking$ approach that
can shrink their heavy CNN or transformer backbones to reduce the number of
channels in the network, thereby improving their inference speed. We compress
the model using a three-stage pipeline: 1) training the object tracking model
with sparsity parameters and an added sparsity penalty function, 2) pruning
the less important channels based on the learnt sparsity pattern, and 3) fine-
tuning the model to recover its performance.

It has been observed that the post-pruning results are better if the pruning-
related hyperparameters are embedded and the model is re-trained with them.
Due to this reason, we train the network from scratch using the sparsity
parameters. In general, the addition of these scaling parameters is not
expected to affect the model performance since during the first training
phase, it only serves to scale the weights. However, the added parameters need
to be pushed towards 0 or 1 as a part of the training process. This requires
adding an additional penalty on the intermediate values. For example, network
slimming [15] uses $L_{1}$ norm penalty on the weights, and ChipNet [13]
achieves it through Heaviside projections and added constraints.

Learning sparsity masks: The core of our pruning-in-tracking method lies in
how well the sparsity of masks is learnt. For this, we employ network
slimming[15] directly leveraging $\gamma$ parameters of the batch-
normalization layers as scaling factors. Since transformers do not contain
batchnorm layers, we prune MHSA and MLP modules by explicitly introducing
masks as done in [14]. The scaling factors and the weights of the original
backbone network are jointly trained with an additional sparsity
regularization introduced on the scaling factors. Based on this, the
optimization problem solved during training can be stated as

$\underset{\mathbf{W},\boldsymbol{\gamma}}{\text{min}}\enskip\mathcal{L}(f(\boldsymbol{\gamma}\odot\mathbf{H}(\mathbf{w});\mathbf{x}),\mathbf{y})+\lambda
g(\boldsymbol{\gamma})$ (1)

where $(\mathbf{x},\mathbf{y})$ denotes the training dataset, $\mathbf{H}$
denotes set of hidden channels, $\mathbf{w}$ denotes the weights of the
network, and $\gamma_{i}\in\boldsymbol{\gamma}$ denotes the scaling factor
associated with feature map/channel $\mathbf{c}_{i}\in\mathbf{H}$. Further,
$\mathcal{L}(\cdot)$ denotes the standard training loss of the object tracking
problem and $g(\cdot)$ denotes the sparsity-related penalty added to the
scaling terms, weighted by hyper-parameter $\lambda$. Similar to [15], we use
$L_{1}$-norm for achieving sparsity.

Channel pruning and fine-tuning: After learning the sparsity, the trained
values of $\boldsymbol{\gamma}$ are used to remove the undesired set of
channels from the network. There are many values in the distribution of
$\gamma$ which are close to zero. To prune, we remove all the incoming and
outgoing connections and the corresponding weights for a certain $\gamma_{i}$,
and this can be achieved in a budget-aware manner. In [15], channels are
pruned with a global threshold across all layers. However, we have observed
that a global threshold is generally not very efficient in extreme pruning
scenarios. It leads to excessive pruning in the early layers and introduces a
bottleneck on the information that can be propagated in the later layers. To
circumvent this issue, we impose channels budgets separately for each layer or
globally throughout the network depending upon the budget and empirical
performance, and ensure that the budget is satisfied at the global level
throughout.

###  2.2 Light-weight SiamFC: A basic example

We demonstrate the construction of light-weight object trackers through
network pruning on SiamFC, a simple yet very effective object tracking method
[1]. This method forms the foundation for most current SOTA trackers, and a
hypothesis experimented on this method should be transferable to the recent
methods with minimal modifications. SiamFC uses a variant of AlexNet [18] as
the backbone and the goal of the presented experiments is to perform network
pruning over it and measure the performance vs. compute operations/model
parameters tradeoff.

Implementation details. SiamFC uses AlexNet-type backbone with five
convolutional blocks. Since the last layer does not use batch-normalization,
we restrict the pruning of the model to only the first four convolutional
layers. A sparsity penalty term, as indicated in Eq. 1, is used on all batch-
normalization layers to learn the importance of the channels. The tracking
model is then trained from scratch on the GOT-10k dataset [19]. The
hyperparameter settings for this stage are similar to those reported in the
official implementation111Accessed at https://github.com/huanglianghua/siamfc-
pytorch.

Once the model is trained, channels corresponding to low values of the
sparsity parameter are pruned. We report scores for a variety of channel
budgets which denote the percentage of channels in the pruned network as
compared to the parent network. As motivated in Section 2.1, we do not impose
a global threshold across all layers of the network, rather a threshold is
separately chosen for each layer in such a manner that the overall channel
budget is satisfied. Unlike the results reported in [15], we have observed
that layerwise thresholding is more stable and better for pruning for this
particular task.

Table 1: Performance scores for various pruned variants of SiamFC tracking model at different channel budgets. Here, ‘AO’, ‘SR’ and ’Params’ stand for average overlap, success rate and parameters in MB, respectively.  Budget | AO | SR@50 | SR@75 | GFLOPs | Params
---|---|---|---|---|---
- | 0.343 | 0.377 | 0.107 | 2.812 | 8.93
0.75 | 0.346 | 0.376 | 0.105 | 1.75 | 5.45
0.625 | 0.347 | 0.375 | 0.109 | 1.24 | 3.73
0.50 | 0.344 | 0.374 | 0.107 | 0.86 | 2.6
0.375 | 0.322 | 0.347 | 0.100 | 0.56 | 1.64
0.25 | 0.316 | 0.345 | 0.099 | 0.31 | 0.93

Results. Table 1 presents the results for the different pruning budgets.
Across budgets, there is a small change in the observed tracking performance,
however, it is almost negligible. For 75% channel budget, there is no
performance drop, however, the compute operations and model size are already
reduced by around 40% each. For 50% budget, the gains are even higher (FLOPs
reduction by 60% and model size by 70%) with no performance drop. A similar
observation is made at higher compression levels, but with a minimal
performance drop. For the extreme case of 25% channel budget, the performance
drops by only 8% while compute FLOPs and model size reduce remarkably by 90%.
Depending upon the available hardware compute, the above experiments
demonstrate the available control over the performance vs. speed trade-off for
SiamFC.

(a) GOT-10k

(b) LaSOT

Fig. 1: Performance scores for for OSTrack [5], STARK [6] and SuperDiMP [12]
at different budgets of FLOPs and number of parameters in MB for two benchmark
datasets.

##  3 Light-weight deep trackers

We have demonstrated that current object tracking methods can be compressed
significantly to design light-weight trackers. Here, we investigate further
the effect of compressing the SOTA deep trackers, and for this purpose, we
choose CNN-based SuperDiMP [12], transformer-based OSTrack [5] and a hybrid
CNN-transformer architecture STARK [6]. The choice of trackers covers a wide
spectrum of well-performing architectural choices in computer vision, thus
making this study thorough.

SuperDiMP is a pure CNN-based tracker with bounding box regressor and
classification head as well as a Resnet50 backbone. We prune the backbone as
the structure of other components is not suitable for structured pruning. We
implement pruning on the block level for Resnet-based SuperDiMP backbone as it
gives superior results as compared to imposing a global budget. OSTrack uses
attention layers throughout the network allowing to implicitly reduce FLOPs
proportional to the pruning budget as shown in [14]. Since our focus is on
efficiency, we consider the smaller $256\times 256$ image size in all our
experiments for OSTrack. To prevent catastrophic pruning at extremely low
inference budgets, we impose layerwise pruning at 1% budget. Finally,
experiments on encoder-decoder based STARK are presented. STARK requires a
large number of epochs to train, and this makes it computationally infeasible
for pruning experiments. Along with this, the FLOPs and Parameters in STARK
are an order of magnitude higher than other trackers making a fairer
comparison difficult. Hence, we designed a lighter version of this model,
referred occasionally to as mini-STARK (4 stacks of encoder-decoder with 96
dimensions, 4 heads and a feedforward with 768 units), and this model provides
performance similar to STARK, while the FLOPs are reduced by almost 2$\times$
and parameters by $3\times$. Even for mini-STARK, the use of global budget
constraint leads to empty decoder layers and to circumvent this, we use
layerwise budgets for this method. We also decoupled training and pruning of
the encoder and decoder as coupling makes the decoder unstable. All the
trackers are trained on GOT-10k dataset from scratch and tested on various
datasets.

Fig. 2: Performance Plots. OSTrack ( base,  0.5,  0.25,  0.1); mini-STARK (
base,  0.5,  0.25,  0.1); SuperDiMP ( base,  0.5,  0.25,  0.1). Fig. 3: %
Active dimensions at different budgets for OSTrack (left) and SuperDiMP
(right); Budgets: 1.0 ( ), 0.75 ( ), 0.5 ( ), 0.25 ( ), 0.1 ( ), 0.01 ( )

###  3.1 Experimental insights

Our experiments are based on channel pruning with channel budgets ranging from
75% to even 1% in some cases.

Fig. 4: Active attention modules per layer of OSTrack for different
compression budgets. Initial attention modules are prioritized for removal for
early compression budgets. For high compression levels, deeper and and deeper
attention modules are removed, however, in general the network prefers to
retain the later ones.

Performance of compressed models. We first study the performance of the three
trackers at different levels of compression. Fig. 1 shows the performance
comparisons for GOT-10k [19] and LaSOT [20] datasets. For all the trackers,
the rightmost point on the curve corresponds to the full-scale models with no
compression. For STARK, the performance of the original fullscale model is
shown in red whereas mini-STARK corresponds to the second rightmost points in
the plots of Fig. 1. Although the three trackers differ significantly in terms
of the underlying architecture, the compute FLOPs are approximately the same
with those for DiMP being slightly lower. However, OStrack is heavily
overparameterized using more than 2$\times$ parameters than the other
trackers, thereby adding to the memory overhead.

A generic observation across the two datasets is that OSTrack outperforms the
other trackers for almost all scenarios of compression. For an extremely less
number of parameters (equivalent to approximately 1% of the channels of the
base model), OSTrack seems to be slightly unstable with STARK performing
marginally better. Below 10 GFLOPs computation, the performance of STARK and
SuperDiMP models significantly deteriorates. However, at this extreme
compression as well, OSTrack is very stable with its performance being only
marginally lower than the full-scale model. We see an interesting anomalous
behavior of SuperDiMP on GOT-10k, where its full-scale configuration seems to
perform better, however, it is only stable for this configuration and is
outperformed at all compression budgets. Moreover, no such anomaly exists for
LaSOT, which confirms SuperDiMP is not to be preferred.

We further perform a comparative study on OTB100 dataset [21] (see Fig. 2). It
is observed that OSTrack consistently outperforms the other two for all cases
of channel budget. Further, the full-scale version of mini-STARK seems to
perform better than SuperDiMP, however, this model is relatively less stable
in terms of compression and the performance of the pruned versions drops
significantly for the success as well as precision scores.

Early compression dip. A general observation across all cases of pruning is
that during the initial pruning, the performance of all models dips, however,
it is more prominent in SuperDiMP and mini-STARK. In Fig. 1, the two points
around the dip correspond to the channel budgets of 75% and 50%. This dip
occurs because the network finds it easier to remove the $1\times 1$ channels
from the network to achieve a certain budget. This is also confirmed by the
fact that for 75% budget, there occurs only a minimal drop in FLOPs. However,
when pruned beyond, the network starts removing $3\times 3$ filters as well
and this arrangement is more optimal for the performance of the network,
leading to improved generalization and an increase in its performance. This is
also reflected by the bigger drop in FLOPs at this stage. Overall this initial
dip is found to be common across all trackers that use CNNs, being more
prominent for deep trackers. Although very small, a similar trend can also be
found in the previous results shown for SiamFC.

Induced sparsity pattern. Next, we attempt to study the sparsity pattern
induced in different trackers due to pruning. Fig. 3 shows the number of
active dimensions per layer of OSTrack and SuperDiMP at different compression
budgets. Note that for STARK, we perform layerwise pruning due to its
instability issues, and the budgets are satisfied per layer. Clearly, a
constant pattern would be observed in such a scenario and is not of interest
to investigate further. For SuperDiMP, we do not perform any pruning at the
input and output of each convolutional block. This is done to avoid
inconsistency due to additional input head from the residual skip connections.

From Fig. 3, we see that the active dimensions across different MLP layers of
OSTrack are almost similar with higher pruning in the deeper layers of the
network. For SuperDiMP, the trend is completely different. For its active
dimensions, measured in terms of the fraction of active channels per layer, we
see that more units are active in the middle of the network, however, the
trend is not as smooth as OSTrack. For the attention modules of OSTrack as
shown in Fig. 4, we see that for initial levels of compression, the starting
layers of the network are preferred, however, the deeper and deeper layers get
chosen for pruning as the compression level increases. This implies that the
mixing of information through attention is not very relevant for the network
during the initial layers and an adaptive choice is to retain them more
towards the later parts, this is in contrast to classification models as shown
in [14]. For STARK, due to layerwise pruning, active units are the same across
all layers, hence not reported here.

Overall, based on the experiments above, OSTrack is consistently better than
the other trackers for almost all scenarios. It is stable at even extreme
pruning levels and the performance drop is not very large. Further, since
OSTrack is fully transformer-based, imposing budget constraint on channels
translates directly into FLOPs and memory budgets, and this provides a better
control on the compression levels.

##  4 Conclusion

In this paper, we studied how light-weight object trackers can be designed by
compressing the current SOTA deep trackers. First, we use a basic object
tracking model which can be pruned significantly with a very limited drop in
its performance. Further, we presented a comparative study on building light-
weight trackers from SOTA tracking methods. We compared trackers that use
CNNs, transformers as well as the combination of the two, and studied their
stability at various compression levels, including extreme compression levels.
From the presented experiments, we infer that a fully transformer-based model
is a better choice for designing light-weight trackers, both from the
perspective of performance as well as stability.

## References

  * [1] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr,  “Fully-convolutional siamese networks for object tracking,”  in European conference on computer vision. Springer, 2016, pp. 850–865.
  * [2] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan,  “Siamrpn++: Evolution of siamese visual tracking with very deep networks,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4282–4291.
  * [3] Paul Voigtlaender, Jonathon Luiten, Philip HS Torr, and Bastian Leibe,  “Siam r-cnn: Visual tracking by re-detection,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 6578–6588.
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [5] Botao Ye, Hong Chang, Bingpeng Ma, and Shiguang Shan,  “Joint feature learning and relation modeling for tracking: A one-stream framework,”  arXiv preprint arXiv:2203.11991, 2022.
  * [6] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu,  “Learning spatio-temporal transformer for visual tracking,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10448–10457.
  * [7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.
  * [9] Vasyl Borsuk, Roman Vei, Orest Kupyn, Tetiana Martyniuk, Igor Krashenyi, and Jiři Matas,  “Fear: Fast, efficient, accurate and robust visual tracker,”  arXiv preprint arXiv:2112.07957, 2021.
  * [10] Manqiang Che, Runling Wang, Yan Lu, Yan Li, Hui Zhi, and Changzhen Xiong,  “Channel pruning for visual tracking,”  in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0–0.
  * [11] Bin Yan, Houwen Peng, Kan Wu, Dong Wang, Jianlong Fu, and Huchuan Lu,  “Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecture search,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15180–15189.
  * [12] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte,  “Learning discriminative model prediction for tracking,”  in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6182–6191.
  * [13] Rishabh Tiwari, Udbhav Bamba, Arnav Chavan, and Deepak K Gupta,  “Chipnet: Budget-aware pruning with heaviside continuous approximations,”  arXiv preprint arXiv:2102.07156, 2021.
  * [14] Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric P Xing,  “Vision transformer slimming: Multi-dimension searching in continuous optimization space,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4931–4941.
  * [15] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang,  “Learning efficient convolutional networks through network slimming,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2736–2744.
  * [16] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang,  “Chasing sparsity in vision transformers: An end-to-end exploration,”  Advances in Neural Information Processing Systems, vol. 34, pp. 19974–19988, 2021.
  * [17] Jonathan Frankle and Michael Carbin,  “The lottery ticket hypothesis: Finding sparse, trainable neural networks,”  arXiv preprint arXiv:1803.03635, 2018.
  * [18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,  “Imagenet classification with deep convolutional neural networks,”  Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017.
  * [19] Lianghua Huang, Xin Zhao, and Kaiqi Huang,  “Got-10k: A large high-diversity benchmark for generic object tracking in the wild,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 5, pp. 1562–1577, 2019.
  * [20] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling,  “Lasot: A high-quality benchmark for large-scale single object tracking,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 5374–5383.
  * [21] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang,  “Object tracking benchmark,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1834–1848, 2015.


# “You Might Like It”: How People Respond to Small Talk in Human-Robot
Collaboration

Kaitlynn Taylor Pineda<EMAIL_ADDRESS>Johns Hopkins UniversityBaltimore,
MDUSA ,  Amama Mahmood<EMAIL_ADDRESS>Johns Hopkins
UniversityBaltimore, MDUSA and  Chien-Ming Huang<EMAIL_ADDRESS>Johns
Hopkins UniversityBaltimore, MDUSA

###### Abstract.

In this work, we investigate people’s engagement and attitudes towards a non-
anthropomorphic robot manipulator that initiates small talk with the user
during a collaborative assembly task, and explore how the presence of negative
team feedback may affect team dynamics and blame attribution. Through an
exploratory study with 20 participants, we found that 18 individuals
interacted socially with the robot, nine of which initiated questions back to
the robot. We report the frequency and length of users’ responses in task-
oriented and non-task-oriented dialogue, and further elaborate on people’s
reactions to the negative system feedback and robot-initiated small talk. We
discuss the potential for integrating small talk in non-social robots, and
propose three design guidelines to enhance human-robot small talk
interactions.

Human-Robot Collaboration, Social Robots, Small Talk, HRI

††ccs: Human-centered computing Interaction design process and methods††ccs:
Computer systems organization Robotics

Figure 1. We explore participants’ interactions with a manipulator robot that
engages in small-talk during a collaborative assembly task. This figure shows
the sequence of some spoken phrases shared by the participant (P1) and robot;
task-oriented dialogue is interchanged with small-talk. The task-oriented
requests and responses are highlighted.

This figure includes 6 photos, alternating between the participant themselves
and the participant and the robot interacting together. Below is text for each
image that depicts the conversation that is exchanged between the human and
the robot. The conversation is as follows: robot: on another note, do you like
to play any games?; participant: um, I like Tetris if you if you’ve ever
played that?; participant: Can I get another yellow pipe?; robot: Getting it
now.; robot: No, I have not; participant: you might like it; robot: Can you
tell me more about it?; participant: [chuckle] I think it’s a pretty classic
game you know you just like the pieces different shape pieces are dropping
down and you’re trying to get them to fill up a line across and then it’ll
like clear it. they even made a movie about it.

##  1\. Introduction

Human-robot collaboration research envisions robots as human collaborators, or
co-workers, to enhance human work in diverse domains. Take manufacturing as an
example: the manufacturing industry is currently undergoing a significant
transformation to become more flexible and agile, allowing it to accommodate
mass product customization even while product life cycles are continuously
decreasing. Moreover, small and medium-sized manufacturing companies face a
shortage of skilled production workers. Collaborative robots—robots capable of
working alongside people on joint tasks safely and efficiently—promise to fill
the needed jobs, enable flexible manufacturing, and increase productivity.

Though these collaborative robots are positioned as co-workers, they are
mostly non-anthropomorphic as in the form of robotic arm (manipulator) and are
designed to support functional, physical tasks. Human collaboration, on the
other hand, is more than just performing the joint task but involves off-task
social interactions, which are integral to establishing and maintaining user
trust and rapport, key ingredients for continued productive collaboration
(kopp2021success, 21). While prior research has found that people like to have
and expect social interactions with their robot co-workers in manufacturing
settings (Sauppé_Mutlu_2015, 41), the robots studied were the Baxter robot
with animated facial expressions and a humanlike form factor. It is unclear if
people would engage in meaningful social interactions with non-anthropomorphic
robot manipulators.

In this work, we explore the possibility of a non-anthropomorphic robot
manipulator not just working with people to complete physical tasks but also
engaging in social exchanges, particularly small talk—commonly seen in human
interactions for rapport and trust building (Laver1981LinguisticRA, 24)—as
human co-workers would. Specifically, we sought to study whether and how
people respond to small talk from a robot manipulator during a collaborative
task. Moreover, we were interested in understanding how negative performance
feedback to the human-robot team might shape the team dynamics and blame
attribution; this inquiry is relevant and of interest because of the rising
adoption of performance monitoring systems in industrial settings (_e.g.,_
warehouse). Through an exploratory study with 20 participants, we found that
most participants (18) interacted socially with the robot (Fig. 1), nine of
which even initiated questions back to the robot. Overall, the robot’s small
talk led to pleasant experiences and did not negatively impact task
efficiency. We discuss the potential for integrating small talk in
collaborative robots, and present design guidelines to enhance human-robot
small talk interactions.

##  2\. Background and Related Work

###  2.1. Social Human-Robot Interactions

There have been efforts to study how robots can interact socially with humans
in various domains such as for tutoring and educational contexts
(belpaeme2018education, 5) (salomons2022we, 40), encouraging exercise
(antony2023co, 2), providing companionship (kidd2006sociable, 18)
(tsoi2021challenges, 50) and emotional support (sabelli2011conversational,
39), and other therapeutic interventions (feil2009toward,
13)(scassellati2018improving, 42). Many of these engagements explore the
integration of various multi-modal capabilities in robots to promote their
engagements with humans (youssef2022survey, 53). These can consist of robot
eye gaze behaviors (mutlu2009footing, 32) (admoni2017social, 1), head
movements (sidner2006effect, 45) (kkedzierski2013emys, 17) and gestures
(meena2012integration, 30) alongside robot speech abilities
(striepe2021modeling, 48) (sinnema2019attitude, 46). While research has been
conducted to analyze the characteristics of conversational interactions with
robots (seok2022cultural, 44) (laban2020tell, 22), the majority of these works
utilize social robots as their robotic entity. However, the prevalence of non-
social robots such as cobots or other manipulator robots is much greater than
social robots themselves, and few efforts have been made to further explore
what will happen if we have conversations between humans and non-social robots
(chowdhury2021you, 8) (li2022tod, 26).

###  2.2. People Treat Machines Like People

The inclination of humans to treat machines socially can be contextualized
within broader theoretical frameworks. The Computers Are Social Actors (CASA)
paradigm (nass1994computers, 33) and the Media Equation Theory
(reeves1996media, 36) argue that humans often, even subconsciously, treat
machines, computers, and various forms of media as sentient entities, reacting
to them in inherently social manners. Such anthropomorphic tendencies
highlight the intersection of technology and human social instincts. Even in
industrial manufacturing environments, there remains an intrinsic human desire
for social interaction. As operators often partake in casual conversations or
“small talk” during shifts (cheon2022robots, 7), they have also found
themselves speaking to the robot they work with (Sauppé_Mutlu_2015, 41). This
suggests that individuals not only expect but actively desire social features
in manufacturing robots.

Some studies explored the potential of enhancing cobots primarily designed for
physical manipulation tasks with more human-like movements
(hostettler2023human, 16). One study specifically integrated principles from
character animations (terziouglu2020designing, 49) to enhance the perception
of cobots as social entities. Such amalgam of technology and socialization
emphasizes the importance of recognizing and harnessing the persistent human
drive for social connection, even with machines. Given the limited research on
designing social behaviors for industrial robots and manipulators with a non-
human form, despite its potential, a pressing question arises: How can we
further design human-like attributes for robots that are inherently devoid of
such characteristics? Building on this notion, our exploratory study
investigates the potential of using small talk to instill social
characteristics in a physical robot arm manipulator, specifically within the
context of a human-robot collaborative assembly task.

###  2.3. Small Talk

Social dialogue that consists of light conversation between new or unfamiliar
parties is often referred to as “small talk” (Laver1981LinguisticRA, 24).
Small talk is significant in the workplace as it is used by coworkers to
establish trust, build rapport and strengthen interpersonal relationships
(coupland2003small, 10, 35). It typically occurs at the start of conversations
with topics such as weather, sport, or other social engagements outside of the
workplace (holmes2005small, 15). While prevalent in American culture,
attitudes towards the value of small talk differ across cultures
(bickmore2005social, 6, 38, 52).

The integration of small talk to non-human agents engaging in conversations
with people has been explored to help promote trust and rapport between the
human and the agent. Small talk has mainly been applied to support additional
conversations in settings where people are interacting with conversational
agents (clark2019good, 9, 14, 20), voice assistants (mahmood2023llm, 28), and
virtual agents (zhou2019trusting, 54). Additionally, incorporating small talk
in more anthropomorphic social robots, combined with gaze (babel2021small, 4),
facial expressions (paradeda2016facial, 34), or emotional displays
(aroyo2018trust, 3), has been shown to elevate user trust levels. Furthermore,
researchers have investigated which behaviors in particular can build rapport
between humans and a collaborative robot (seo2018investigating, 43). However,
less explored is the domain of applying small talk to foster rapport between
humans and other agents with physical capabilities that are not typically
poised to engage in conversations with people, such as autonomous self driving
vehicles (large2019s, 23), or industrial robots (li2023irwoz, 25). Preliminary
studies hint at the benefits of integrating small talk with task-oriented
dialogue in industrial robot settings (li2022tod, 26). Yet, understanding
users’ interactions and experiences with such robots remains unexplored.
Investigating individuals’ responses and reactions to small talk initiated by
non-humanoid robots is essential for comprehending its influence on human-
robot collaborations and gathering insights on how to foster better human-
robot partnership.

##  3\. METHODS

We conducted an exploratory study to analyze people’s behaviors and attitudes
towards an industrial robot that exhibited social behavior (via initiated
small talk) while examining the change in human-robot team dynamics after the
team receives team feedback.

###  3.1. Collaborative Assembly Task

Participants were informed that the assembly task was intended to mimic a
real-world collaboration setup between a human-robot team where the robot
retrieves parts out of reach for the human and refrains from entering the
human’s designated workspace. They were given a fictional team assignment
(Team 3) and were told the collaborative task scenario was a prototype for a
larger-scale warehouse interaction. It was intended to simulate a modern
warehouse that may contain a computerized efficiency monitoring system to
oversee the team collaboration. The monitoring system would communicate with
them via speech but would be distinguishable from any speech communication
coming from the robot.

Each participant engaged in a collaborative PVC pipe structure building task
with a robot manipulator mounted on a table. They were given diagrams
corresponding to the structures they were required to assemble. The
participants read instructions for how to interact with the robot via speech
commands such as _“Bring me a [green or yellow] pipe”_ , in order for the
robot to retrieve the corresponding pipe. The robot would provide a verbal
confirmation message if a request was received correctly and an additional
instruction reminder message once per structure when the participant requested
their second to last pipe for that structure. The pipe would be placed on a
ramp that connected the robot’s workspace to the participant’s workspace so
that the pipe could be delivered to the participant without the robot entering
their workspace. Participants were provided with joints in containers that
corresponded to each structure to be built. To simulate a realistic industrial
collaborative environment that might involve the use of defective materials,
each participant was given one faulty joint for every pipe structure. They
were instructed to inspect each joint before using it in the structure. If
participants identified a faulty joint (marked with red sticker), they would
place the joint on a ramp that would deliver it to a container inside the
robot’s workspace. The robot would retrieve the joint when given a verbal
command such as _”Report a faulty joint”_. The experimental setup is shown in
Figure 2.

Figure 2. The experimental setup. The robot can grab a pipe from the pipe
dispenser and place it on a pipe ramp, which delivers it to the participant.
The participant can grab a joint from the joint containers on the left and
place a faulty joint on the joint ramp for it to be retrievable by the robot.

The experimental setup. The top image shows two tables next to one another.
The table on the left is the participant workspace and contains four boxes
with pipe connectors that the participant can grab joints from. The table on
the right is where the robot is mounted and serves as the robot’s workspace. A
ramp that the participant can use for the delivery of faulty pipes to the
robot and a ramp that the robot can use to deliver requested pipes to the
participant connect the two tables. The bottom image shows another view of the
two tables, with a closer view of the robot workspace that highlights a pipe
dispenser to the right of the robot from which the robot collects the pipe the
participant requested.

###  3.2. System Manipulation and Robot Behavior

We present the system and robot behavior manipulations implemented in our
exploratory study to understand how the social facets of a robot manipulator
influence users’ perceptions of the robot.

####  3.2.1. Negative Team Feedback

Midway through the first and second structures in the main session, the human-
robot team was provided with verbal negative feedback from the monitoring
system. Each message was prefaced by an audible beep:

  1. (1)

Feedback 1: _“System announcement. Team 3, please work on improving your
performance.”_

  2. (2)

Feedback 2: _“System announcement. Team 3, you are currently not meeting
expectations. Improve your performance.”_

####  3.2.2. Social Behavior Robot Small-Talk

The robot exhibited four types of social and task-oriented speech behaviors:
questions, statements, reminders and confirmations. During the main assembly
task, the robot would engage in small talk by initiating questions and/or
statements after it delivers a request confirmation message or an instruction
reminder message. The robot initiated a minimum of one question or statement
per request, regardless of the response rate of the participants. After a
break in conversation via the negative feedback announcement, the robot
initiated questions and statements on a different topic. The conversation
touched on a range of questions spanning three primary categories to
accommodate the diverse interests of individuals: travel; books, TV shows, and
movies; and sports and games. Some examples of robot small-talk phrases are
shown in Table 1.

Table 1. Examples of robot questions and statements

Topic | Phrase
---|---
Travel | By the way, do you like to travel?
I arrived to the U.S. from Germany in a large box.
Books, TV, Movies | What do you like about those books?
| That sounds like a great way to relax.
---
Sports, Games | Do you have a favorite sport to watch?
It is good to spend time outdoors exercising.

###  3.3. System Implementation

####  3.3.1. Robot System

We utilized a Panda robot from Franka Emika. It is an industrial robotic arm
with 7 degrees of freedom (7-DoF). The robot was restricted to its designated
workspace and never intruded into the participant’s allocated area. Its
movements were preprogrammed to execute specific sequences of actions. Among
these, two involved pick-and-place motions: one to retrieve and deliver either
a green or yellow PVC pipe to the participant, and the other to pick up a
container holding a PVC pipe joint and subsequently move the container over a
bin to discard the joint. The robot interacted with participants using text-
to-speech communication.

####  3.3.2. Wizard-of-Oz Control

The experimenter controlled the robot using a Wizard-of-Oz (WoZ) approach
(woz2012riek, 37). Through a remote graphical user interface (GUI), the
experimenter executed robot actions and selected from preprogrammed
conversational speech phrases. Robot actions and initial task messages were
triggered using one of five GUI buttons. For conversational speech, the
experimenter inputted specific alphanumeric codes into a textbox.
Additionally, the experimenter could manually type text into the textbox to
produce non-preprogrammed phrases via the text-to-speech feature using Google
Text-to-Speech (TTS) online.

####  3.3.3. Preprogrammed Speech Behavior

When a robot movement was triggered, a confirmation message was automatically
activated to notify the participant that the action would occur. Robot
confirmation messages were preprogrammed and selected randomly during each
execution. Task reminder messages were similarly preprogrammed and triggered
automatically when the participant had requested enough pipes to be one away
from completing a structure. Negative feedback directed towards the human-
robot team was triggered automatically when participants had requested half
the pipes needed for the first and second structure.

###  3.4. Study Procedure

Participants first completed the consent form and a demographics
questionnaire. Afterward, the experimenter explained the task scenario and
gave the participant instructions about the task and how to communicate with
the robot. After the experimenter left the room, the participant received an
introductory message from the monitoring system. The participant then engaged
in a practice session to construct one pipe structure, allowing them to become
acquainted with the robot’s task-oriented behaviors and the use of verbal
commands. The robot only exhibited task-oriented dialogue during the practice
task, such as a reminder message to move their pipe structure over to another
table when finished and a confirmation message signifying it comprehended the
request. When finished, the participant informed the experimenter that they
were done. Following this, they took part in a main session where they built
three additional pipe structures with the robot. After completing the
structures, the participant informed the experimenter that they were done. At
the end of the study, the experimenter conducted a semi-structured interview
with each participant to understand their experience collaborating with the
robot.

###  3.5. Participants

24 participants were recruited through convenience sampling, out of which 4
were excluded due to technical issues. The remaining 20 participants (11
female, 9 male) ranged in age from 18 to 38 ($M=24.35,SD=5.44$). Participants
had a variety of educational backgrounds, including computer science,
engineering, natural sciences, education, biotechnology, administration and
international studies. Twelve participants listed Asian as their ethnicity;
five as Caucasian; one as Hispanic, Latino, or Spanish origin of any race; one
as Asian and Caucasian; and one as Caucasian and Hispanic, Latino, or Spanish
origin of any race. Participants were familiar with technology
($M=4.15,SD=0.75$, 5-point scale with 5 being Very Familiar and 1 being Not At
All), had moderate experience with robots ($M=2.5,SD=1$), and had little
experience teaming with robots ($M=1.65,SD=0.93$). The study took about 45
minutes and was approved by the university’s institutional review board.
Participants were compensated with $15 US dollars.

###  3.6. Data Analysis

We collected audio-visual data of the participants’ collaboration with the
robot during the pipe assembly tasks, and of the semi-structured interview.
Each participant had an average of 12 minutes of interaction with the robot
over the main tasks. The experimenter transcribed the audio data and conducted
thematic analysis on the transcripts to identify central themes and attitudes
surrounding the participants’ interactions with the robot. The participants’
verbal phrases during the main session were coded by the context in which it
was made (_i.e.,_ request, response to robot task-oriented message, response
to robot small-talk, question initiated to the robot). The participant small
talk responses were further coded to identify the general length of the
response: vocable, short sentence (¡= 4 words), medium-length sentence (¿ 4
words), and multiple sentences. We deduced response rates based on code
frequencies from the interaction data. Response rates for the robot’s
questions, statements, reminders, and confirmations were calculated as the
percentage of those prompts that received a user response.

Given the robot’s non-human-like appearance, people might find its small talk
unexpected. Thus, to evaluate the impact of small talk in robots primarily
designed for physical collaboration, we analyzed the participants’ reactions
to the robot’s initial small talk conversational attempt via three different
aspects:

1\. Conversation reciprocation: We coded a person’s response to the robot’s
first conversational attempt as no response or response, and also looked at if
the response was an answer to a robot question.

2\. Impact on task: To observe whether the robot’s question disrupted the
participant’s ongoing task, we code people’s behavior in response to the
robot’s first small talk question into three actions: minimal (quick glance at
the robot but no task disruption), freeze (stops current action and looks at
the robot before continuing the task), and none (did not stop or look at the
robot).

3\. Affective reactions: To gather insights into the participants’ emotional
or mental state, we grouped people’s non-verbal reactions during the robot’s
first question into three categories: pleasant (smile), surprise (chin raise,
lower lip raise, brow lowering etc.), and indifference (no reaction).

##  4\. Results

In this exploratory study, first, we aim to understand whether, how, and to
what extent participants engage with robot-initiated small talk. We examine
the response rate, length of responses, and user-initiated questions. Second,
we assess how participants perceive the robot’s initial small talk question,
specifically its impact on their ongoing action and emotional state. Third, we
investigate whether a negative evaluation of the human-robot team influences
participants’ perceptions of the robot and their teamwork; this involves
analyzing their verbal and non-verbal reactions to negative feedback and
determining whom they blame for such feedback.

###  4.1. Participant Response Rates

Of the 20 participants, two participants did not engage in additional
conversational interaction with the robot apart from the verbal commands
needed to complete the collaborative task. We determined the question response
rate for all participants to be 86.67% (325 out of 375), the statement
response rate as $36.27\%$ (70 out of 193), and the task-oriented response
rate for confirmation and reminder messages as 1.68% (8 out of 475) and 16.67%
(10 out of 60), respectively. For the 18 participants who did engage in small
talk, we calculated the question response rate to be 91.55% (325 out of 355),
the statement response rate as $39.11\%$ (70 out of 179), and the task-
oriented response rate for confirmation and reminder messages as 1.87% (8 out
of 427) and 18.52% (10 out of 54), respectively.

###  4.2. How Participants Respond to Small Talk

####  4.2.1. Length of participant responses

There were a total of 395 verbal responses made to the robot for non-task-
oriented dialogue; examples are shown in Table 2. Participants’ responses were
mostly one sentence ($N=345$, $87.34\%$). The one-sentence responses were
further broken down as follows: vocable ($N=11$, $2.78\%$), short sentence (¡=
4 words; $N=168$, $42.53\%$), and medium-length sentence (¿ 4 words; $N=166$,
$42.03\%$). The remaining participant responses were comprised of multiple
sentences ($N=50$, $12.66\%$), reflecting higher and possibly meaningful
engagement.

Table 2. Examples of People’s Responses That Vary in Length

Category | Conversation
---|---
|  | R: In the US, I have traveled from the third floor
---
of this building to the second floor.
Vocable | P: Whaaat? (P20)
| R: Do you like to play sports?
Short Sentence | P: No, not really. (P6)
| R: How often do you watch movies?
Medium Sentence |  | P: Not that often, like twice three times
---
a month. (P20)
| R: Where have you been in the United States?
Multi-Sentence |  | P: Most of the East Coast, a lot of the West
---
Coast, the Midwest. I was born in Utah. Uh, I’ve
been to a lot of places in the United States. (P6)

####  4.2.2. Participant-initiated questions

Apart from answering the robot-initiated small talk questions, we observed
that participants also asked the robot questions. Nine participants ($45\%$)
asked a total of 46 questions to the robot. While some asked questions back as
a matter of courtesy, others asked out of genuine curiosity. P1 explained, ”I
wanted to know more about the robot, like once it started talking to me”.

Furthermore, the questions asked to the robot were often personal, asking it
about its preferences or capabilities. P7 remarked, ”Because he’s a robot and
I’m not familiar with a robot and …I …truly want to know …how long does it
take to get assemble …he said, …he’s from Germany by a group, but I’m more
…interested …I tried to ask …do you like your owner?” For the eleven
participants that did not initiate a question to the robot, many described
their perception of robot as non-human-like as a contributing factor. One
participant commented how the conversation, ”just doesn’t have as much value
as …if I was talking to an actual person and hearing where they want to
travel”(P13). For the two participants that did not engage with the robot at
all, one stated ”It’s not a human, you don’t have a conversation” (P17). The
other explained how ”from beginning to end, I always felt like it’s a robot”
(P3).

###  4.3. Participant Response to the First Robot Small Talk Question

In addition to understanding participants’ respond to robot small talk, we are
interested in their first reaction to the robot’s small talk question. We
present our findings on three different aspects of the their responses, see
Figure 3.

####  4.3.1. Conversation reciprocation

18 participants responded to the robot’s first question: “By the way, do you
like to travel?”. 17 participants responded with an answer similar to _e.g.,_
“Yeah, I like traveling”. However, one participant replied with a rhetorical
question: “Why are you talking to me robot? This has nothing to do with the
task.” (P12). Only two participants did not respond to the robot’s question;
the same two participants who did not engage in any non-task conversation with
the robot.

Figure 3. Different aspects of participants’ response to the first robot small
talk initiation: (1) conversational reciprocation, (2) impact on task, and (3)
affective reaction. For each aspect, the states are mutually exclusive. The
numbers denote participants who exhibited the corresponding responses.

Figure 3: A figure graphically displaying the participant’s response to the
first robot small task question (e.g., ”By the way, do you like to travel?”)
in terms of conversation reciprocation, impact on task, and affective
reaction, with numbers listed for each type of participant response
(conversation reciprocation: 17 answer, 2 no engagement, 1 refuse; impact on
task: 13 minimal [quick glance], 4 freeze, 3 none [no behavioral
acknowledgment]; affective reaction: 6 indifference, 6 surprise, 6 smile)

####  4.3.2. Impact on task

The majority of participants ($N=13$) merely exchanged a brief glance with the
robot when it posed the first question; they looked at it momentarily, without
letting it interrupt their ongoing task, suggesting that their attention was
momentarily captured by the robot’s ”unusual” behavior (initiating small
talk), but not to a degree that proved disruptive. In contrast, three
participants did not look at the robot at all. Interestingly, four
participants momentarily halted their task for at least two seconds, looked
intently at the robot (indicating a ”freeze” reaction), and then resumed their
activity. These individuals also engaged in a verbal response to the robot.

####  4.3.3. Affective reaction

Eight out of the 20 participants smiled in response to the robot’s first
question, indicating a pleasant emotional state. Six participants displayed
signs of surprise through various non-verbal cues such as chin raiser, lower
lip raiser, and eye-brow lowering. The remaining six participants exhibited
indifference, as evidenced by their lack of any non-verbal response.

###  4.4. Reactions to Negative System Feedback

####  4.4.1. Negative feedback reaction rate

We analyzed a time window starting from the system feedback announcement and
lasting up to 5 seconds post-announcement, yielding a duration of 10 seconds
for the first and 13 seconds for the second announcement. Of the 20
participants, 18 reacted to the negative feedback. Notably, the two non-
respondents did not engage with the robot throughout. Additionally, two
participants responded to only one of the two feedback statements. Out of 40
negative feedback messages, 8 prompted a verbal reaction and 34 elicited a
non-verbal response. Participants exhibited a range of non-verbal reactions;
while some smiled and laughed, others gestured in frustration or hesitated,
seemingly confused. Eye-rolling was also observed in response to the system
announcement. Verbal reactions further highlighted this confusion. One
participant remarked, ”I … okay” (P18), while others expressed frustration and
justified their own performance, such as ”I’m doing as much as I can do,
system announcer” (P12) or the team’s performance; ”I mean we’re fast enough”
(P7).

####  4.4.2. Blame attribution

Overall, 10 participants blamed no one for the system announcement, 4 blamed
the robot, 3 the team, and 3 themselves. Among those blaming the robot, only
one cited its small talk, while the rest attributed the announcements to its
slow speed. One participant who blamed the team stated, “…Maybe in terms of
like the team chemistry, that was probably where it was lacking”(P16). Another
explained how they thought the novelty of the conversation impacted the team’s
performance, “I also get a little more distracted, because …this is just …a
new thing …it’s something that I guess you would get used to if you’re working
with it every day”(P10).

Those who blamed themselves acknowledged possible mistakes. Meanwhile, 15
participants discussed they were confused at what prompted the system
announcements because they thought the team was performing efficiently. P1
explained how the announcement “felt like out of my control. So I was like, I
don’t think there was anything else that could have been done.”

Figure 4. Examples of positive and negative user interaction with the robot
small talk behaviors. The top row shows a positive example where the
participant expressed genuine interest in learning more about the robot (P7).
The bottom row is an example of a user who perceived the interaction
negatively and exhibited a potential mental model mismatch of expectations
(P12).

Figure 4: Examples of positive and negative user interaction with the robot
small talk behaviors. The figure has two rows. The top row contains seven
images alternating between images focusing on the participant and images
focusing on the robot, with snippets of the conversation shown below
indicating participant interest in conversation about the robot’s background
and specifications. The bottom row contains six images alternating between
images focusing on the participant and images focusing on the robot, with
snippets of the conversation shown below indicating participant indicating
surprise or confusion on why the robot is engaging in conversation unrelated
to the task at hand.

##  5\. Discussion

We explored the incorporation of conversational elements such as small talk
into industrial robots, especially those not inherently built for social
interactions.

While opinions on the robot’s conversational abilities vary, participants
showed a willingness to engage in off-topic conversations initiated by the
robot. In this section, we discuss our findings and offer design guidelines
for robot small talk.

###  5.1. Robot-Initiated Small Talk: First Impression

Participants’ immediate impressions and engagements in small talk with the
robot validate considering such dialogue for non-social robots. Our findings
(section 4.3) suggest the majority of participants ($N=17$) positively
received the robot’s conversation attempts, attesting to the potential of
robots to engage meaningfully in small talk right from the outset. Yet, not
every response was unequivocally positive. Some participants abstained from
engagement ($N=2$) or posed a rhetorical question ($N=1$, see Figure 4
bottom). Such user attitudes, potentially anchored in pre-established
expectations, might indicate their expectation for manipulator robots to
adhere strictly to task-related interactions (dennler2023design, 12).

Participants’ task engagement revealed insights into their receptivity to
unexpected robot behavior (small talk) for a otherwise non-social robot
(section 4.3). While most participants ($N=13$) briefly glanced at the robot,
suggesting a brief attention shift without disruption, the ”freeze” reactions
of some ($N=4$) signaled a task disruption. This echos prior work noting that
extended mutual eye-gaze in human-human exchanges often signals ”problematic
interruptions” (degutyte2021role, 11). The robot’s initial conversation also
evoked affective reactions. The smiles from many participants ($N=8$) suggest
robots can spark positive emotions early on. Although a smile might also
indicate pleasant ”surprise”, overt signs of surprise from others ($N=6$)
emphasize the unexpectedness of small talk from a primarily task-oriented
robot. Such shift in user attention and emotional state underscores the
unexpectedness of the first question, likely due to the robot’s unforeseen
conversational approach and a perceived mismatch with its physical embodiment.

In summary, the notable engagement and minimal task disruption despite
surprise indicate that even if participants are surprised by the robot’s first
question, it does not hinder their task. This emphasizes participants’
readiness to converse with robots, even during ongoing tasks and when the
unexpected conversation is most pronounced. As participants adapt to the
robot’s conversational style, diversions due to small talk might reduce
further. Future research should investigate optimal small talk design to
balance task efficiency and user enjoyment. Calibrating conversation frequency
is vital to avoid minimize excessive talkativeness that distracts, or causes
awkwardness due to prolonged silence.

###  5.2. Potential of Small Talk for Otherwise Non-Social Robots

####  5.2.1. User engagement: Small talk in purely functional tasks

While engaging in small talk with a robot arm manipulator during collaborative
tasks, we observed a noticeable willingness to not only respond to the robot’s
direct questions but also its comments that do not warrant a user response
(section 4.1). For instance, in Figure 1, the participant comments (“You might
like it.”) on the robot’s response (“No, I have not”) to her question (“if
you’ve ever played that?”). While most of the responses by the participants
were single sentenced, 12% of the responses were multiple sentences showing
higher and meaningful engagement (section 4.2.1). Such level of user
engagement is evident. In Figure 1, the participant’s response ”(chuckle) I
think it’s a pretty classic game …They even made a movie about it.” goes
beyond merely answering the robot’s query; it offers supplementary
information, perhaps in anticipation of the robot’s genuine interest in the
game. It is as if the participant was attempting to pique the robot’s
curiosity, subtly suggesting beforehand, “You might like it”.

Although the desire to ask robot questions in return varied among
participants, 45% of them posed personal questions to the robot, inquiring
about its preferences or capabilities (see section 4.2.2 and Figure 4).
Additionally, participants perceived the small talk positively; P20 mentioned,
“it was cool to have somebody to talk to”, while P16 characterized the
conversation with the robot as, “more personal and …less mechanical”. Thus,
our exploratory study indicates user engagement and responsiveness to robot-
initiated off-topic conversations, suggesting potential for integrating small
talk to enhance user engagement in human-robot collaborative tasks that
involve physical manipulation.

> Potential 1:  Small talk may boost meaningful engagements in human-robot
> interactions, even with non-anthropomorphic robots in purely functional
> scenarios.

####  5.2.2. Ice-breaking through small talk: building rapport and unveiling
robot capabilities

Participants demonstrated curiosity about the robot’s functions, often posing
small talk questions to probe its capabilities and background. For instance,
as illustrated in Figure 4 top, one participant inquired, ”Hey robot, tell me
more about yourself. I’m very interested. Like, you said you’re from Germany.”
Another, referencing Figure 4 bottom, sought clarity on the robot’s primary
function by asking, ”Aren’t you here just to help me with these pipes?” Such
questions highlight the potential of employing conversational frameworks, even
for non-social robots, to build common ground and mutual understanding in
human-robot teams. By structuring initial interactions around themes such as
”getting to know your teammate”, robots can ease introductions, reducing
potential barriers to seamless collaboration. Participant P7’s sentiment, “I’m
curious to learn more about it” encapsulates this curiosity. Thus, small talk
can serve as an initial icebreaker, fostering team unity and rapport. Over
time, or in subsequent sessions, these conversations can become instrumental
in communicating the robot’s capabilities, ensuring that users develops a
comprehensive understanding of its functions. Furthermore, by incorporating
task-related insights, tidbits, or safety tips into these dialogues, the robot
can harness users’ social attention to support task performance.

> Potential 2:  Small talk can be an effective ice-breaker, fostering rapport
> and understandings of robot capabilities in collaborative settings.

###  5.3. Challenges and Design Guidelines for Robot Small Talk Framework

####  5.3.1. Dynamic and user-aware conversational framework

The conversational framework we employed for robot small talk demonstrated
constraints. The design was rigid, causing interactions to feel scripted
rather than organic as pointed out by P16: “the robots question feel kind of
scripted or …inorganic so,…asking it questions, which is kind of like asking
questions to a voice assistant.” The robot’s knowledge bank was confined to a
limited scope of topics (by design). For instance, in the conversation with
P15, after the robot’s question “By the way, do you like to travel?” , the
participant responded in negation and requested next pipe: “No. Bring me a
yellow pipe”. The next robot question (“What is the coolest place you have
ever been to?”) was not well received by the participant. Despite their clear
negation to the previous question indicating disinterest, the robot asked a
follow-up question on the same topic, agitating (“Shut up!”) by the
participant. Similarly, P18 responded to the robot question about traveling:
“You know I don’t know and I don’t feel like talking about traveling right
now.” Thus, rigid conversational frameworks fail to adapt to the evolving
nature of human conversation, highlighting the future need for a more dynamic,
expansive and user-aware conversational approach.

To increase conversational flexibility, small talk should reflect user
interests, allowing them to initiate and engage in a broader range of topics.
Leveraging Large Language Models (LLMs) can help tailor small talk for robots
and expand a robot’s knowledge. However, they must be adapted to align with a
robot’s physical limitations, especially when addressing questions about their
origins and experiences such as, ”How were you created?”, or ”Have you
traveled?” Thus, LLMs can enhance the conversational capabilities but should
not be utilized without necessary modifications via prompt engineering or fine
tuning to avoid confusion about a robot’s physical capabilities and realistic
experiences.

> Design Guideline 1:  Robots should use dynamic, user-aware conversational
> frameworks supporting flexible small talk interactions beyond rigid topics
> or formats.

####  5.3.2. Balancing small talk with task performance

Engaging in small talk with people during physical tasks poses challenges,
such as with the frequency of questions posed. For instance, initiating small
talk during every user command or robot action might be excessive as expressed
by a participant (P15) through a rhetorical question to the robot: “Why do you
talk so much?” Over-engagement in non-task related conversations can be
distracting for users (veletsianos2012learners, 51, 31), potentially
jeopardizing the overall performance or efficiency of the collaboration.
Therefore, it is imperative to strike a balance between fostering rapport and
ensuring task performance. Small talk frequency and complexity should be
adapted based on task intricacy and observed team performance. During critical
tasks, or when user confusion or distraction arises, it might be wise to limit
or halt small talk. Using error and confusion detection frameworks (as
proposed in (stiber2023using, 47, 27)) can help robots gauge progress and
performance. Using these frameworks, robots can determine the appropriate
time, or whether at all, to engage in small talk.

> Design Guideline 2:  Robots should ensure small talk is context-sensitive,
> enhancing team rapport without impeding task completion.

####  5.3.3. Mental model mismatch: managing user expectations

Several participants were unaware of interactive possibilities with the robot
as evident in section 4.2.2. Participant P12 comments and inquires, “That’s
strange coming from a robot. Aren’t you here to just help me with these
pipes?” (see Figure 4 bottom). This points towards a potential misalignment in
their mental model of the robot’s capabilities. P3’s statement, ”From
beginning to end, I always felt like it’s a robot [non-human form]”, further
illustrates this misalignment. It raises a pivotal question: What
conversational depth is appropriate for a robot manipulator that lacks human-
like form and operates within constrained physical and cognitive boundaries?

Conversely, when robots participate in conversations, it might inflate
people’s expectations beyond the robot’s actual capabilities. For instance,
few participants expected the robot to be more capable, only to find the robot
lacking, as P7 pointed out: “I mean, because …right now, artificial
intelligence is so popular. I thought a robot [is] gonna be more intelligent.”

Such misalignment may be exacerbated by participants’ familiarity with
advanced conversational AI tools, such as Alexa and ChatGPT, leading them to
expect that the “social” robot possesses enhanced conversational abilities.
Therefore, when a physical robot starts chatting, it may lead to misalignment
in user expectations and robot capabilities, as P10 expressed: “Because once
[robot] starts asking me stuff, then I was …oh, maybe it’s …kind of like GPT-
esque, where …has further knowledge rather than just questions, but …then my
expectation went back a little bit.” If the robot’s dialogue suggests a high
degree of intelligence, any shortfall in meeting those expectations could lead
to increased user disappointment. Even slight mismatches between the voice and
appearance of robots have been shown to introduce artifacts in human-robot
interactions (mcginn2019can, 29). Establishing common ground and rapport
crucially relies on instilling an accurate mental model of the robot in
individuals’ minds (kiesler2005fostering, 19). Thus, it is essential to
consider how user expectations can be managed adequately while designing
social characteristics for robots. The robot’s social cues may shape
individuals’ perceptions; if misaligned initially, the robot’s early behaviors
can correct these views and recalibrate common ground (kiesler2005fostering,
19).

> Design Guideline 3:  Intentionally calibrate user expectations by matching
> robot conversational cues with its capabilities, considering users’
> familiarity with AI.

###  5.4. Limitations and Future Work

Despite its insights, our study’s short duration and single-session focus may
not fully capture the depth, intricacies and dynamics of extended human-robot
collaborations and how they are influenced by small talk. Additionally, the
study did not consider cultural nuances of participants, which are vital as
different cultures perceive and value small talk differently. While this work
serves as an exploratory examination of the use of small talk in human-robot
interactions, future studies should focus on testing hypotheses to quantify
the benefits of small talk for rapport building and to examine its role in
human-robot collaborations with negative feedback, team evaluations, or robot
errors. When designing a conversational framework for robot small talk, it is
essential to research and ensure the appropriateness of content, timing, and
context. For instance, the framework should be sensitive to user preferences
on topics, adapt according to the user’s state for ideal timing, and consider
team performance metrics to prevent disruptions during critical tasks. Lastly,
future research should explore methods to align a robot’s conversational
capabilities with its functional abilities. It is important to keep users
apprised of the robot’s true abilities, possibly using small talk as an
informative medium. Nevertheless, the strategic introduction of small talk can
be instrumental in building and strengthening rapport between users and task-
oriented robots in collaborative scenarios.

###### Acknowledgements.

This work was supported by National Science Foundation award #2141335.

## References

  * (1) Henny Admoni and Brian Scassellati  “Social eye gaze in human-robot interaction: a review”  In _Journal of Human-Robot Interaction_ 6.1 Journal of Human-Robot Interaction Steering Committee, 2017, pp. 25–63
  * (2) Victor Nikhil Antony, Sue Min Cho and Chien-Ming Huang  “Co-Designing with Older Adults, for Older Adults: Robots to Promote Physical Activity”  In _Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , 2023, pp. 506–515
  * (3) Alexander Mois Aroyo, Francesco Rea, Giulio Sandini and Alessandra Sciutti  “Trust and social engineering in human robot interaction: Will a robot make you disclose sensitive information, conform to its recommendations or gamble?”  In _IEEE Robotics and Automation Letters_ 3.4 IEEE, 2018, pp. 3701–3708
  * (4) Franziska Babel et al.  “Small talk with a robot? The impact of dialog content, talk initiative, and gaze behavior of a social robot on trust, acceptance, and proximity”  In _International Journal of Social Robotics_ Springer, 2021, pp. 1–14
  * (5) Tony Belpaeme et al.  “Social robots for education: A review”  In _Science Robotics_ 3.21, 2018, pp. eaat5954  DOI: 10.1126/scirobotics.aat5954
  * (6) Timothy Bickmore and Justine Cassell  “Social dialongue with embodied conversational agents”  In _Advances in natural multimodal dialogue systems_ Springer, 2005, pp. 23–54
  * (7) EunJeong Cheon, Eike Schneiders, Kristina Diekjobst and Mikael B Skov  “Robots as a Place for Socializing: Influences of Collaborative Robots on Social Dynamics In-and Outside the Production Cells”  In _Proceedings of the ACM on Human-Computer Interaction_ 6.CSCW2 ACM New York, NY, USA, 2022, pp. 1–26
  * (8) Aparajita Chowdhury, Aino Ahtinen, Roel Pieters and Kaisa Väänänen  “” How are you today, Panda the Robot?”–Affectiveness, Playfulness and Relatedness in Human-Robot Collaboration in the Factory Context”  In _2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)_, 2021, pp. 1089–1096  IEEE
  * (9) Leigh Clark et al.  “What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents”  In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , CHI ’19  New York, NY, USA: Association for Computing Machinery, 2019, pp. 1–12  DOI: 10.1145/3290605.3300705
  * (10) Justine Coupland  “Small talk: Social functions”  In _Research on language and social interaction_ 36.1 Taylor & Francis, 2003, pp. 1–6
  * (11) Ziedune Degutyte and Arlene Astell  “The role of eye gaze in regulating turn taking in conversations: a systematized review of methods and findings”  In _Frontiers in Psychology_ 12 Frontiers Media SA, 2021, pp. 616471
  * (12) Nathaniel Dennler et al.  “Design metaphors for understanding user expectations of socially interactive robot embodiments”  In _ACM Transactions on Human-Robot Interaction_ 12.2 ACM New York, NY, 2023, pp. 1–41
  * (13) David Feil-Seifer and Maja J Matarić  “Toward socially assistive robotics for augmenting interventions for children with autism spectrum disorders”  In _Experimental Robotics: The Eleventh International Symposium_ , 2009, pp. 201–210  Springer
  * (14) Jasper Feine, Ulrich Gnewuch, Stefan Morana and Alexander Maedche  “A taxonomy of social cues for conversational agents”  In _International Journal of Human-Computer Studies_ 132 Elsevier, 2019, pp. 138–161
  * (15) Janet Holmes  “When small talk is a big deal: Sociolinguistic challenges in the workplace”  In _Second language needs analysis_ 344, 2005, pp. 371
  * (16) Damian Hostettler, Simon Mayer and Christian Hildebrand  “Human-Like Movements of Industrial Robots Positively Impact Observer Perception”  In _International Journal of Social Robotics_ 15.8 Springer, 2023, pp. 1399–1417
  * (17) Jan Kędzierski et al.  “EMYS—emotive head of a social robot”  In _International Journal of Social Robotics_ 5 Springer, 2013, pp. 237–249
  * (18) Cory D Kidd, Will Taggart and Sherry Turkle  “A sociable robot to encourage social interaction among the elderly”  In _Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006._ , 2006, pp. 3972–3976  IEEE
  * (19) Sara Kiesler  “Fostering common ground in human-robot interaction”  In _ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005._ , 2005, pp. 729–734  IEEE
  * (20) Tina Klüwer  ““I Like Your Shirt”-dialogue acts for enabling social talk in conversational agents”  In _International Workshop on Intelligent Virtual Agents_ , 2011, pp. 14–27  Springer
  * (21) Tobias Kopp, Marco Baumgartner and Steffen Kinkel  “Success factors for introducing industrial human-robot interaction in practice: an empirically driven framework”  In _The International Journal of Advanced Manufacturing Technology_ 112 Springer, 2021, pp. 685–704
  * (22) Guy Laban, Jean-Noël George, Val Morrison and Emily S Cross  “Tell me more! Assessing interactions with social robots from speech”  In _Paladyn, Journal of Behavioral Robotics_ 12.1 De Gruyter, 2020, pp. 136–159
  * (23) David R Large et al.  “” It’s small talk, jim, but not as we know it.” engendering trust through human-agent conversation in an autonomous, self-driving car”  In _Proceedings of the 1st International Conference on Conversational User Interfaces_ , 2019, pp. 1–7
  * (24) John Laver and Florian Coulmas  “Linguistic Routines and Politeness in Greeting and Parting”, 1981  URL: https://api.semanticscholar.org/CorpusID:141308546
  * (25) Chen Li, Dimitrios Chrysostomou, Xiaochun Zhang and Hongji Yang  “IRWoZ: Constructing an Industrial Robot Wizard-of-OZ Dialoguing Dataset”  In _IEEE Access_ 11 IEEE, 2023, pp. 28236–28251
  * (26) Chen Li, Xiaochun Zhang, Dimitrios Chrysostomou and Hongji Yang  “Tod4ir: A humanised task-oriented dialogue system for industrial robots”  In _IEEE Access_ 10 IEEE, 2022, pp. 91631–91649
  * (27) Na Li and Robert Ross  “Hmm, you seem confused! tracking interlocutor confusion for situated task-oriented hri”  In _Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , 2023, pp. 142–151
  * (28) Amama Mahmood et al.  “LLM-Powered Conversational Voice Assistants: Interaction Patterns, Opportunities, Challenges, and Design Guidelines”  In _preprint arXiv:2309.13879_ , 2023
  * (29) Conor McGinn and Ilaria Torre  “Can you tell the robot by the voice? An exploratory study on the role of voice in the perception of robots”  In _2019 14th ACM/IEEE international Conference on human-robot interaction (HRI)_ , 2019, pp. 211–221  IEEE
  * (30) Raveesh Meena, Kristiina Jokinen and Graham Wilcock  “Integration of gestures and speech in human-robot interaction”  In _2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom)_ , 2012, pp. 673–678  IEEE
  * (31) Jessica R Methot, Emily H Rosado-Solomon, Patrick E Downes and Allison S Gabriel  “Office chitchat as a social ritual: The uplifting yet distracting effects of daily small talk at work”  In _Academy of Management Journal_ 64.5 Academy of Management Briarcliff Manor, NY, 2021, pp. 1445–1471
  * (32) Bilge Mutlu et al.  “Footing in human-robot conversations: how robots might shape participant roles using gaze cues”  In _Proceedings of the 4th ACM/IEEE international conference on Human robot interaction_ , 2009, pp. 61–68
  * (33) C Nass, J Steuer and ER Tauber  “Computers Are Social Actors: Conference Companion on Human Factors in Computing Systems-CHI’94”  In _Association for Computing Machinery: New York, NY, USA_ , 1994
  * (34) Raul Benites Paradeda, Mojgan Hashemian, Rafael Afonso Rodrigues and Ana Paiva  “How facial expressions and small talk may influence trust in a robot”  In _Social Robotics: 8th International Conference, ICSR 2016, Kansas City, MO, USA, November 1-3, 2016 Proceedings 8_ , 2016, pp. 169–178  Springer
  * (35) Patricia Pullin  “Small talk, rapport, and international communicative competence: Lessons to learn from BELF”  In _The Journal of Business Communication (1973)_ 47.4 SAGE Publications Sage CA: Los Angeles, CA, 2010, pp. 455–476
  * (36) Byron Reeves and Clifford Nass  “The media equation: How people treat computers, television, and new media like real people”  In _Cambridge, UK_ 10.10, 1996
  * (37) Laurel D Riek  “Wizard of oz studies in hri: a systematic review and new reporting guidelines”  In _Journal of Human-Robot Interaction_ 1.1 Journal of Human-Robot Interaction Steering Committee, 2012, pp. 119–136
  * (38) Lana Rings  “Beyond grammar and vocabulary: German and American differences in routine formulae and small talk”  In _Die Unterrichtspraxis/Teaching German_ JSTOR, 1994, pp. 23–28
  * (39) Alessandra Maria Sabelli, Takayuki Kanda and Norihiro Hagita  “A conversational robot in an elderly care center: an ethnographic study”  In _Proceedings of the 6th international conference on Human-robot interaction_ , 2011, pp. 37–44
  * (40) Nicole Salomons, Kaitlynn Taylor Pineda, Adérónké Adéjàre and Brian Scassellati  ““We Make a Great Team!”: Adults with Low Prior Domain Knowledge Learn more from a Peer Robot than a Tutor Robot”  In _2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2022, pp. 176–184  IEEE
  * (41) Allison Sauppé and Bilge Mutlu  “The Social Impact of a Robot Co-Worker in Industrial Settings”  In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ Seoul Republic of Korea: ACM, 2015, pp. 3613–3622  DOI: 10.1145/2702123.2702181
  * (42) Brian Scassellati et al.  “Improving social skills in children with ASD using a long-term, in-home social robot”  In _Science Robotics_ 3.21 American Association for the Advancement of Science, 2018, pp. eaat7544
  * (43) Stela H Seo et al.  “Investigating people’s rapport building and hindering behaviors when working with a collaborative robot”  In _International Journal of Social Robotics_ 10 Springer, 2018, pp. 147–161
  * (44) Sukyung Seok, Eunji Hwang, Jongsuk Choi and Yoonseob Lim  “Cultural differences in indirect speech act use and politeness in human-robot interaction”  In _2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2022, pp. 1–8  IEEE
  * (45) Candace L Sidner, Christopher Lee, Louis-Philippe Morency and Clifton Forlines  “The effect of head-nod recognition in human-robot conversation”  In _Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction_ , 2006, pp. 290–296
  * (46) Lizzy Sinnema and Maryam Alimardani  “The attitude of elderly and young adults towards a humanoid robot as a facilitator for social interaction”  In _Social Robotics: 11th International Conference, ICSR 2019, Madrid, Spain, November 26–29, 2019, Proceedings 11_ , 2019, pp. 24–33  Springer
  * (47) Maia Stiber, Russell H Taylor and Chien-Ming Huang  “On using social signals to enable flexible error-aware HRI”  In _Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , 2023, pp. 222–230
  * (48) Hendrik Striepe, Melissa Donnermann, Martina Lein and Birgit Lugrin  “Modeling and evaluating emotion, contextual head movement and voices for a social robot storyteller”  In _International Journal of Social Robotics_ 13 Springer, 2021, pp. 441–457
  * (49) Yunus Terzioğlu, Bilge Mutlu and Erol Şahin  “Designing social cues for collaborative robots: the role of gaze and breathing in human-robot collaboration”  In _Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction_ , 2020, pp. 343–357
  * (50) Nathan Tsoi et al.  “Challenges deploying robots during a pandemic: An effort to fight social isolation among children”  In _Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_ , 2021, pp. 234–242
  * (51) George Veletsianos  “How do learners respond to pedagogical agents that deliver social-oriented non-task messages? Impact on student learning, perceptions, and experiences”  In _Computers in Human Behavior_ 28.1 Elsevier, 2012, pp. 275–283
  * (52) Wenhui Yang  “Small talk: A strategic interaction in Chinese interpersonal business negotiations”  In _Discourse & Communication_ 6.1 Sage Publications Sage UK: London, England, 2012, pp. 101–124
  * (53) Karim Youssef, Sherif Said, Samer Alkork and Taha Beyrouthy  “A survey on recent advances in social robotics”  In _Robotics_ 11.4 MDPI, 2022, pp. 75
  * (54) Michelle X Zhou, Gloria Mark, Jingyi Li and Huahai Yang  “Trusting virtual agents: The effect of personality”  In _ACM Transactions on Interactive Intelligent Systems (TiiS)_ 9.2-3 ACM New York, NY, USA, 2019, pp. 1–36

## Appendix

Table 3. Overview of Codes and Their Definitions for the Data Analysis
Response to First Robot Question: Conversation Reciprocation
---
Name |  Definition
Answer |  Response (Yes) and answered robot’s question
Refuse |  Response (Yes) but did not answer the robot’s question
No engagement |  Participant did not answer the robot’s question
Response to First Robot Question: Impact on Task
Name |  Definition
Minimal (glance) |  Looks at the robot but no task disruption
Freeze |  Stops current action (minimum of 2 seconds) and looks at the robot before continuing the task
None |  Did not stop or look at the robot
Response to First Robot Question: Affective Reactions
Name |  Definition
Pleasant |  Individual exhibits a smile
Surprise |  Individual exhibits behaviors such as a chin raise, lower lip raise, brow lowering, etc.
Indifference |  No reaction
Non-Verbal Reactions to Negative Feedback
Name |  Definition
Verbal Response |  An audible or verbal phrase spoken by the participant
Non-Verbal Response |  Behaviors such as smiling, laughing, eye-rolls, head nods, arm movements, etc.
Blame Attribution
Name |  Definition
Self |  Selects themselves as a potential cause for the negative feedback announcement
Robot |  Selects the robot as a potential cause for the negative feedback announcement
Team |  Selects the team as a potential cause for the negative feedback announcement
Nobody |  Believes neither they nor the robot caused the negative feedback announcement
Motives for Initiating Questions
Name |  Definition
Curiosity |  Individual describes a level of curiosity
Felt Human-like |  Individual makes connections to human-likeness
Social Convention |  Individual describes factors relating to social conventions