# Crafting Better Contrastive Views for Siamese Representation Learning

Xiangyu Peng1 Kai Wang111footnotemark: 1 Zheng Zhu2 Mang Wang3 Yang You1
1National University of Singapore 2Tsinghua University 3Alibaba Group
{xiangyupeng, kai.wang<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Code: https://github.com/xyupeng/ContrastiveCrop Equal
contribution.Corresponding author.

###### Abstract

Recent self-supervised contrastive learning methods greatly benefit from the
Siamese structure that aims at minimizing distances between positive pairs.
For high performance Siamese representation learning, one of the keys is to
design good contrastive pairs. Most previous works simply apply random
sampling to make different crops of the same image, which overlooks the
semantic information that may degrade the quality of views. In this work, we
propose ContrastiveCrop, which could effectively generate better crops for
Siamese representation learning. Firstly, a semantic-aware object localization
strategy is proposed within the training process in a fully unsupervised
manner. This guides us to generate contrastive views which could avoid most
false positives (_i.e_., object _vs_. background). Moreover, we empirically
find that views with similar appearances are trivial for the Siamese model
training. Thus, a center-suppressed sampling is further designed to enlarge
the variance of crops. Remarkably, our method takes a careful consideration of
positive pairs for contrastive learning with negligible extra training
overhead. As a plug-and-play and framework-agnostic module, ContrastiveCrop
consistently improves SimCLR, MoCo, BYOL, SimSiam by $0.4\%\sim 2.0\%$
classification accuracy on CIFAR-10, CIFAR-100, Tiny ImageNet and STL-10.
Superior results are also achieved on downstream detection and segmentation
tasks when pre-trained on ImageNet-1K.

##  1 Introduction

(a)

(b)

(c)

(d)

Figure 1: The motivation of our proposed ContrastiveCrop. (a) and (c) are
generated by typical RandomCrop, while (b) and (d) are crops from our method.
We address the false positive problem (object _vs_. background) shown in (a)
by localizing the object and restricting the crop center within the bounding
box (the black dashed box) in (b). Moreover, we enlarge the variance of crops
in (d) by keeping them away from the center, which avoids the close appearance
as shown in (c).

Self-supervised learning (SSL) has attracted much attention in the computer
vision community due to its potential of exploiting large amount of unlabeled
data. As a mainstream approach in SSL, contrastive learning has achieved
higher performance on several downstream tasks (_e.g_., object detection,
segmentation and pose estimation [32, 21, 18, 16, 27, 50]) than its supervised
counterpart. Such promising results can be largely attributed to the Siamese
structure, which is commonly applied in state-of-the-art unsupervised methods,
including SimCLR [5], MoCo V1 & V2 [20, 7], BYOL [17] and SimSiam [8].
Typically, the Siamese structure takes two augmented views from an image as
input, and minimizes their distance in the embedding space. With proper views
selected, Siamese networks demonstrate a strong capability to learn generic
visual features [37].

One of the key issues of contrastive learning is to design positives
selection. Some works generate different positive views by strong data
augmentation, such as color distortion and jigsaw transformation [37, 4].
Another work [34] applies mixture [49, 48] in an unsupervised manner to
produce positive pairs that incorporate multiple samples. Additionally,
different from data augmentation, [51] creates hard positives with
transformation at the feature level. Despite different techniques, these works
commonly apply RandomCrop to sample multiple views of an image, and further
make the views more diverse.

As a basic sampling method, RandomCrop enables all individual crops to be
selected equiprobably. However, it fails to look at the semantic information
of paired views, which helps to learn better representations more efficiently
and accurately. As shown in Fig. 1(a), random crops are prone to miss the
object when no prior of object (_e.g_., scale and location) is given.
Optimizing the distance between object and background in the embedding space
would mislead the learning of representations. Besides, Fig. 1(c) indicates
that random crops cannot always carry sufficient variances of an object. Such
views with large similarity are trivial for learning discriminative models.

In this paper, we propose ContrastiveCrop, aiming to craft better contrastive
pairs for Siamese representation learning. False positives indicate that a
better sampling strategy for contrastive learning should consider the content
information of the image. Hereby, we propose a semantic-aware localization
scheme. The module serves as a guidance to select crops, avoiding most false
positives, as shown in Fig. 1(b). Moreover, we propose a center-suppressed
sampling strategy to tackle trivial positive pairs with large similarity. Fig.
1(d) shows that our crops are more likely to cover different parts of the
object. The semantic-aware localization and center-suppressed sampling scheme
can be gracefully combined to generate better crops for contrastive learning.

The proposed ContrastiveCrop considers both semantic information and
maintaining large variance when making pairs. As a plug-and-play method, it
can be easily applied into the Siamese structure. More importantly, our
approach is agnostic to contrastive frameworks, regardless using negative
samples or not. With negligible training overhead, our strategy consistently
improves SimCLR, MoCo, BYOL, SimSiam by $0.4\%\sim 2.0\%$ classification
accuracy on CIFAR-10, CIFAR-100, Tiny ImageNet and STL-10. Superior results
are also achieved on downstream detection and segmentation tasks when pre-
trained on ImageNet-1K.

The main contributions of this paper can be summarized as:

  * •

To the best of our knowledge, this is the first work to investigate the
problem of commonly used RandomCrop in contrastive learning. We propose
ContrastiveCrop that is customized to generate better views for this task.

  * •

In ContrastiveCrop, the semantic-aware localization is adopted to avoid most
false positives and the center-suppressed sampling strategy is applied to
reduce trivial positive pairs.

  * •

ContrastiveCrop consistently outperforms RandomCrop with popular contrastive
methods on a variety of datasets, showing its effectiveness and generality for
Siamese representation learning.

##  2 Related works

In this section, we introduce contrastive learning and positives selection
related to this work.

###  2.1 Contrastive Learning

The core idea of contrastive learning is to pull positive pairs closer while
pushing negatives apart in the embedding space. This methodology has shown
great promise in learning visual representations without annotation [2, 23,
43, 29, 30, 47, 36]. More recently, contrastive methods based on the Siamese
structure achieve remarkable performance on downstream tasks [5, 20, 7, 17, 8,
45, 40, 46, 15], some of which even surpass supervised models.

The milestone work is SimCLR [5], which presents a simple framework for
contrastive visual representation learning. It significantly improves the
quality of learned representations with a non-linear transformation head.
Another famous work is MoCo [20], which uses a memory bank to store large
number of negative samples and smoothly updates it with momentum for better
consistency. Methods that learn useful representations without negative
samples are also proposed. BYOL [17] trains an online network to predict the
output of the target network, with the latter slowly updated with momentum.
The authors hypothesize that the additional projector to the online network
and the momentum encoder are important to avoid collapsed solutions without
negative samples. SimSiam [8] further explores simple Siamese networks that
can learn meaningful representations without negative sample pairs, large
batches and momentum encoders. The role of stop-gradient is emphasized in
preventing collapsing. In addition to framework design, theoretical analyses
and empirical studies have also been proposed to better understand the
behavior and properties of contrastive learning [39, 39, 53, 44, 41, 1, 24,
31, 35, 6, 9, 3].

Figure 2: The training dynamic of localization is shown from left to right in
each subfigure. We initialize the localization box as the whole image, and
update it at a regular interval using the latest heatmap. Note that our goal
is not to derive precise localization, but to guide generation of crops by
finding the object of interest.

###  2.2 Positives Selection

One of the key issues in contrastive learning is the design of positives
selection. An intuitive approach to generating positive pairs is to create
different views of a sample using data augmentation. Most SSL works apply data
augmentation pipelines that are directly adapted from those in supervised
learning [13, 12, 26, 19, 49, 48]. Chen  _et al_. [5] comprehensively study
the effect of a range of data transformations, and find out the composition
made of random cropping and random color distortion can lead to better
performance. Tian  _et al_. [37] propose an InfoMin principle to catch a sweet
point of mutual information between views, and accordingly generate positive
pairs with its InfoMin Augmentation. A close work to this paper is [33], which
also uses unsupervised saliency maps as a constraint of crops, but crops are
still randomly sampled. All these works commonly apply RandomCrop as the basic
sampling method to generate input views, which we find may not be the optimal
solution for contrastive learning. [28] take object-scene relation into
account when making crops, but require additional object proposal algorithms.
In this work, we propose ContrastiveCrop that is tailored to create better
positives views for contrastive learning, without the need of external
functions.

##  3 Method

In this section, we introduce ContrastiveCrop for Siamese representation
learning. Firstly, we briefly review RandomCrop as the preliminary knowledge.
Then, we describe semantic-aware localization and center-suppressed sampling
as two submodules of our ContrastiveCrop. Finally, favorable properties of our
method are further discussed for better understanding.

###  3.1 Preliminary

RandomCrop, an efficient data augmentation method, has been widely used in
both supervised learning and self-supervised learning (SSL). Here, we briefly
review this technique, using API in
Pytorch111https://pytorch.org/vision/stable/transforms.html as an example.
Given an image $I$, we first determine the scale $s$ and aspect ratio $r$ of
the crop from a pre-defined range (_e.g_., $s\in[0.2,1.0]$ and
$r\in[3/4,4/3]$) Then, the height and width of the crop can be obtained with
$s$ and $r$. After that, the location of the crop is randomly selected on the
image plane, as long as the whole crop lies within the image. The procedure of
RandomCrop can be formulated as

$(x,y,h,w)=\mathbb{R}_{crop}(s,r,I),$ (1)

where $\mathbb{R}_{crop}(\cdot,\cdot,\cdot)$ is the random sampling function
that returns a quaternion $(x,y,h,w)$ representing the crop. We denote $I$ as
the input image, $(x,y)$ as the coordinate of the crop center, and ($h$, $w$)
as the height and width of crop. Usually, the scale $s$ and aspect ratio $r$
of crops are set flexibly, so that crops of variant sizes could be made.

In principle, RandomCrop enables all individual crops to be selected, thus
could provide diverse views of a sample. However, it performs sampling
equiprobably (_i.e_., each single view is sampled with the same probability),
which ignores the semantic information of images. As shown in Fig. 1(a),
RandomCrop is prone to generate false positives when the scale of object is
small. Given objects with variant scales in contrastive learning, RandomCrop
would inevitably generate false positives due to lack of the consideration of
semantic information. As a result, optimizing the false positives in Fig. 3
may mislead the learning of good representations. Therefore, designing a
semantic-aware sampling strategy for crops is crucial and vital for Siamese
representation learning.

Figure 3: Visualization of RandomCrop, RandomCrop + Semantic-aware
Localization and our ContrastiveCrop. We show the sampling distributions and
operable regions for three settings on the left, and correspondent sampled
pairs on the right. Pairs made by RandomCrop include several false positives
that totally miss the object (marked in yellow box). Using RandomCrop with
Semantic-aware Localization reduces false positives, but introduces easy
positive pairs that share large similarity. Last, our ContrastiveCrop could
reduce false positive pairs while increasing variance at the same time.

###  3.2 Semantic-aware Localization

To tackle the issue of poor content understanding in RandomCrop, we design a
semantic-aware localization module that can effectively reduce false positives
in an unsupervised manner. To better study the process of feature learning in
Siamese networks, we visualize the heatmaps generated at different training
stages (_e.g_., 0th, 20th, 40th, 60th, 80th epoch) in Fig. 2. Note that we
derive the heatmap by summing the features of last convolutional layer across
the channel dimension and normalizing it to [0, 1]. There are several
inspirations from visualization: 1) The Siamese representation learning
framework is capable of capturing the location of the object, which can be
leveraged to guide the generation of better crops; 2) Heatmaps can roughly
indicate the object, but may need some warm-up at early stages.

Based on above analyses, we propose to locate the object during the training
process using the information in heatmaps. Specifically, RandomCrop is applied
at early stage of training to collect semantic information of the whole image.
Then, we apply an indicator function to obtain the bounding box of object $B$
from heatmaps, which can be written as,

$B=L(\mathds{1}[M>k]),$ (2)

where $M$ represents heatmap, $k\in[0,1]$ is the threshold of activations,
$\mathds{1}$ is the indicator function and $L$ calculates the rectangular
closure of activated positions. After obtaining the bounding box $B$, the
semantic crops could be generated as follows,

$[\dot{x},\dot{y},\dot{h},\dot{w}]=\mathbb{R}_{crop}(s,r,B),$ (3)

where the definitions of $\dot{x}$, $\dot{y}$, $\dot{h}$, $\dot{w}$, $s$, $r$,
and $\mathbb{R}_{crop}$ are similar to Eq. 1. Considering the probable coarse
localization, we enlarge the operable region by only constraining center of
crops within $B$. This also reduces the potential negative effect of
resolution discrepancy at training and inference stages [38].

At the training stage, the bounding box is progressively updated at a regular
interval to leverage the latest features learned by the model. Note that our
goal is not to derive precise localization, but to guide generation of crops
by finding the object of interest. The scale of the bounding box is controlled
by the threshold parameter $k\in[0,1]$. Generally speaking, a large $k$ leads
to a small box and would limit the diversity of possible crops to be made. A
small $k$, however, may still include much unrelated background texture and is
not sufficient for finding the object. We study the effect of different
threshold $k$ in Sec. 4.4. We empirically find that the proposed localization
module is not sensitive to this parameter and could improve over baseline
within a wide range of $k$.

Finally, we show the sampling effect of semantic-aware localization in Fig. 3.
Compared with RandomCrop, one can find that the false positive pairs reduce
dramatically when the proposed module is applied. This provides evidence that
self-supervised neural networks trained without annotations are capable of
recognizing the object of interest as well as its location. In this way,
additional region proposals or ground truth bounding boxes are no longer
necessary for views generation [10, 52].

###  3.3 Center-suppressed Sampling

The semantic-aware localization scheme provides useful guidance to reduce
false positive cases, but increases the probability of close appearance pairs
due to the smaller operable region. In this subsection, we introduce the
center-suppressed sampling that aims to tackle this dilemma.

The main idea is to reduce the probability of crops gathering around center by
pushing them apart. Specifically, we adopt the beta distribution
$\beta(\alpha,\alpha)$ with two identical parameters $\alpha$, which shows a
symmetric function. In this way, we could easily control the shape of the
distribution with different $\alpha$. As the goal is to enlarge the variance
of crops, we set $\alpha<1$ which gives us a U-shaped distribution (_i.e_.,
with lower probability near center and greater one at other positions). In
this way, crops are more likely to be scattered to near border lines of the
operable region, and cases of much overlap could be largely avoided.

Combining center-suppressed sampling with semantic-aware localization, we can
finally formulate our ContrastiveCrop as

$(\dot{x},\dot{y},\dot{h},\dot{w})=\mathbb{C}_{crop}(s,r,B),$ (4)

where $\mathbb{C}_{crop}$ denotes sampling function that applies a center-
suppressed distribution, and $B$ is the same bounding box as in Eq. 3. Note
that the shape of beta distribution is determined by the parameter $\alpha$
and affects the variance of crops. We study the impact of different $\alpha$
in in Sec. 4.4, including $\alpha>1$ that gives a inverted U shape.

Algorithm 1 ContrastiveCrop for Siamese Representation Learning

Input: Image $I$, Crop Scale $s$, Crop Ratio $r$, Threshold of Activations
$k$, Parameter of $\beta$ Distribution $\alpha$.

$h=\sqrt{s\cdot r}$ $\triangleright$ Height of the crop

$w=\sqrt{s/r}$ $\triangleright$ Width of the crop

$F=Forward(I)$ $\triangleright$ Features of last layer

$M=Normalize(F)$ $\triangleright$ Heatmap after normalizing

$B=L(\mathds{1}[M>k])$ $\triangleright$ Bounding box by Eq. 2.

$x=B_{x0}+(B_{x1}-B_{x0})\cdot u,u\sim\beta(\alpha,\alpha)$

$y=B_{y0}+(B_{y1}-B_{y0})\cdot v,v\sim\beta(\alpha,\alpha)$

$\triangleright$ Sample crop center $x$ and $y$ from $\beta$ distribution

Output: Crop $C=(x,y,h,w)$

Figure 4: The relation between semantic richness and positives similarity.
Dots are obtained by varying $\alpha$ (fix $k=0.1$, delineated in Sec. 4.2),
and scores of each dot are calculated by averaging results of a large number
of cropping trials. Compared with RandomCrop, our ContrastiveCrop conveys more
semantic information at the same level of similarity (vertical), and yields
less similar positive pairs under equal semantic information (horizontal).
Figure 5: Similarity of positive pairs in training. Smaller positives
similarity indicates harder positive samples which may enhance representation
learning [51]. Taking RandomCrop as baseline, adding only localization results
in slightly larger similarity. Our ContrastiveCrop combines both semantic-
aware localization and center-suppressed sampling, which effectively reduces
similarity of positives.

The effect of our ContrastiveCrop is visualized in Fig. 3. Compared with
RandomCrop, our method could significantly reduce false positive pairs due to
the semantic-aware localization. Meanwhile, it introduces larger variance
within a positive pair by applying the center-suppressed distribution. We show
the pipeline for ContrastiveCrop in Algorithm 1. The whole module is agnostic
to other transformations and can be easily integrated into general contrastive
learning frameworks.

Method | CIFAR-10 | CIFAR-100 | Tiny ImageNet | STL-10
---|---|---|---|---
| R-Crop | C-Crop | R-Crop | C-Crop | R-Crop | C-Crop | R-Crop | C-Crop
SimCLR [5] | 89.63 | 90.08 | 60.30 | 61.91 | 45.19 | 46.21 | 88.95 | 89.53
MoCo [20] | 86.73 | 88.78 | 56.10 | 57.65 | 47.09 | 47.98 | 89.17 | 89.81
BYOL [17] | 91.96 | 92.54 | 63.75 | 64.62 | 46.08 | 47.23 | 91.84 | 92.42
SimSiam [8] | 90.96 | 91.48 | 64.79 | 65.82 | 43.03 | 44.54 | 89.39 | 89.83
Table 1: Linear classification results for different contrastive methods and
datasets. R-Crop and C-Crop mean RandomCrop and ContrastiveCrop, respectively.
We adopt ResNet-18 as the base model and reproduce all the methods with a
unified training setup as described in Sec. 4.2.

###  3.4 Discussion

To better understand the behavior of ContrativeCrop, we discuss several
properties that may contribute to its effectiveness. We first investigate the
relation between semantic information and positives similarity. We take the
class score of a crop as an indicator of richness of categorized semantic
information. The similarity of positive pairs is calculated in the latent
space as the cosine similarity between positive representations. Both the
class score and similarity are average results of a large number of cropping
trials from a standard ResNet-50 [22] trained with ImageNet [14] labels. Their
relation is shown in Fig. 4. One can find that ContrastiveCrop conveys more
semantic information than RandomCrop at the same level of variance, showing
the effectiveness of semantic-aware localization. Furthermore, with equal
semantic information, ContrastiveCrop achieves larger variance than
RandomCrop, which can be owed to center-suppressed sampling.

We further visualize the similarity of positive pairs in the training process
in Fig. 5. As shown in the figure, adding only semantic-aware localization to
RandomCrop slightly increases similarity, as localization restrains crops in a
smaller operable region. Our ContrastiveCrop further incorporates center-
suppressed sampling, showing smaller positives similarity than the other two.
This indicates positive pairs sampled by ContrastiveCrop are harder ones,
which are helpful in learning more view-invariant features as suggested in FT
[51]. However, different from FT that reduces positives similarity in the
feature space, we directly sample harder crops from raw data, while taking a
careful consideration of semantic information.

##  4 Experiments

In this section, we conduct extensive experiments with popular contrastive
methods on a variety of datasets, to demonstrate the effectiveness and
generality of our method. We first introduce the datasets and contrastive
methods in Sec. 4.1. Sec. 4.2 describes the implementation details. We then
evaluate our method with the common linear evaluation protocol in Sec. 4.3.
Results of ablation experiments are shown in Sec. 4.4. Finally, Sec. 4.5
presents transfer performance on downstream object detection and segmentation
tasks.

###  4.1 Datasets & Baseline Approaches

We perform evaluation of our method with state-of-the-art unsupervised
contrastive methods, on a wide range of datasets. The datasets include
CIFAR-10/CIAFR-100 [25], Tiny ImageNet, STL-10 [11] and ImageNet [14].
Generally, these datasets are built for object recognition and the images
contain iconic view of objects. The baseline contrastive methods include
SimCLR [5], MoCo V1 & V2 [20, 7], BYOL [17] and SimSiam [8].

###  4.2 Implementation Details

Our ContrastiveCrop aims to make better views for contrastive learning, which
is agnostic to self-supervised learning frameworks and their related training
components, such as backbone networks, losses, optimizers, _etc_. Thus, we
strictly keep the same training setting when making comparison. Larger gains
could be expected with further hyper-parameter tuning, which is not the focus
of this work.

For small datasets (_i.e_., CIFAR-10/100, Tiny ImageNet and STL-10), we use
the same training setup in all experiments. At the pre-train stage, we train
ResNet-18 [22] for 500 epochs with a batch size of 512 and a cosine-annealed
learning rate of 0.5. The linear classifier is trained for 100 epochs with
initial learning rate of 10.0 multiplied by 0.1 at 60th and 80th epochs.

For experiments on ImageNet, we adopt ResNet-50 as the base model. Pre-train
settings of MoCo V1, MoCo V2 and SimSiam exactly follow their original works.
We reproduce SimCLR with a smaller batch size of 512 and cosine-annealed
learning rate of 0.05. We adopt the same setting as in [20] for training the
linear classifier for all baseline methods.

(a) Accuracy v.s. $k$

(b) Accuracy v.s. $\alpha$

Figure 6: Ablation results on IN-200 w.r.t. $k$ and $\alpha$. Subfigure (a)
compares RandomCrop baseline (the black plot) with RandomCrop + Semantic-aware
Localization (the blue plot). In subfigure (b), we fix the best $k=0.1$ for
localization (the blue plot) and compare it with ContrastiveCrop to study the
influence of different $\alpha$.

For our method, we set $k=0.1$ for the threshold of activations and
$\alpha=0.6$ for sampling. Localization boxes are updated at a frequency of
20% (_i.e_., 4 updates in total, except the last epoch), which adds negligible
extra training overhead; RandomCrop is applied before the first update to
collect global information, as described in Sec. 3.2. All the experiments are
conducted on an 8-GPU server. We use SGD optimizer with momentum of 0.9,
weight decay of $10^{-4}$ and $0$ for pre-train and linear evaluation,
respectively.

###  4.3 Linear Classification

In this section, we verify our method with linear classification following the
common protocol. We freeze pre-trained weights of the encoder and train a
supervised linear classifier on top of it. Top-1 classification accuracy
results on the validation set are reported.

#### Results on CIFAR-10/100, Tiny ImageNet and STL-10.

Our results on these small datasets are shown in Tab. 1. With the same
training setup for all experiments, ContrastiveCrop consistently improves
baseline methods by at least $0.4\%$. Results show that the proposed method is
generic and does not require heavy parameter tuning. The localization boxes
are updated at a frequency of 20% in the training process (_i.e_., 4 times in
total, except the last epoch), adding negligible training overhead.

#### Results on ImageNet.

The results of ImageNet are two-part: 1) standard ImageNet-1K (IN-1K), which
is used for pre-training. 2) IN-200, which consists of 200 random classes of
IN-1K and is used for ablation experiments. As shown in Tab. 2, our method
outperforms RandomCrop with SimCLR, MoCo V1, MoCo V2, SimSiam on IN-1K by
$0.25\%$, $1.09\%$, $0.49\%$ and $0.33\%$, respectively. A larger improvement
is seen on IN-200. The consistent gain over baseline methods shows the
effectiveness and generality of ContrastiveCrop for contrastive methods.

Method | Arch. | Epoch | IN-200 | IN-1K
---|---|---|---|---
Top-1 | Top-1
SimCLR | R50 | 100 | 62.14 | 61.60
SimCLR + Ours | R50 | 100 | 63.08 | 61.85
MoCo V1 | R50 | 100 | 64.52 | 57.25
MoCo V1 + Ours | R50 | 100 | 65.80 | 58.34
MoCo V2 | R50 | 100 | 63.43 | 64.40
MoCo V2 + Ours | R50 | 100 | 64.61 | 64.89
SimSiam | R50 | 100 | 62.89 | 65.62
SimSiam + Ours | R50 | 100 | 63.54 | 65.95
Table 2: Comparison of RandomCrop and our ContrastiveCrop with linear
classification results on IN-200 and IN-1K. Models are pre-trained for 100
epochs, with the same training setup within a method for fair comparison.

###  4.4 Ablation Studies

In ablation studies, we investigate the semantic-aware localization module and
center-suppressed sampling respectively. We also study the effect of
ContrastiveCrop when it is combined with different transformations. We conduct
experiments with MoCo V2 and ResNet-50, and report the linear classification
results on IN-200.

#### Semantic-aware Localization.

In our method, the unsupervised semantic-aware localization serves as a
guidance to make crops. We study the influence of $k$ that determines the
scale of the localization box, with a larger $k$ leading to a smaller box. We
also make comparison with RandomCrop that does not use localization (_i.e_.,
$k=0$), and ground truth bounding boxes (the red plot). Experimental results
are shown in Fig. 6(a). One can find that using localization box outperforms
RandomCrop baseline (the black plot) within a range from 0.05 to 0.2. This
shows the effectiveness of largely removing false positives. Yet, as $k$
increases over $0.25$, the performance starts to fall quickly. We guess the
reason is smaller bounding boxes dramatically reduce variance of views, making
it trivial to learn discriminative features.

Freq. | 0% | 10% | 20% | 30% | 50%
---|---|---|---|---|---
Acc. (%) | 63.43 | 64.40 | 64.61 | 64.40 | 64.11
Table 3: Linear classification accuracy w.r.t different update frequencies of
localization boxes. $0\%$ means no update (i.e., RandomCrop baseline) and
$50\%$ means one update in the right middle of training. RandomCrop is applied
before the first update.

We also study the effect of update frequency of localization boxes in Tab. 3.
It shows that only one update in the middle of training (_i.e_., $50\%$) could
outperform RandomCrop baseline (_i.e_., $0\%$). A larger improvement is seen
in a range of $10\%\sim 30\%$ where there are more updates. These results show
that our method could work well for different update frequencies.

Pre-train | IN-1K | VOC detection | COCO instance seg. | COCO detection
---|---|---|---|---
Top-1 | AP | AP50 | AP75 | APmk | AP${}^{mk}_{50}$ | AP${}^{mk}_{75}$ | APbb | AP${}^{bb}_{50}$ | AP${}^{bb}_{75}$
Random init | - | 33.8 | 60.2 | 33.1 | 29.3 | 46.9 | 30.8 | 26.4 | 44.0 | 27.8
Supervised | 76.1 | 53.5 | 81.3 | 58.8 | 33.3 | 54.7 | 35.2 | 38.2 | 58.2 | 41.2
InfoMin [37] | 70.1 | 57.6 | 82.7 | 64.6 | 34.1 | 55.2 | 36.3 | 39.0 | 58.5 | 42.0
MoCoV1 [20] | 60.6 | 55.9 | 81.5 | 62.6 | 33.6 | 54.8 | 35.6 | 38.5 | 58.3 | 41.6
MoCoV1 + ContrastiveCrop | 63.0 | 56.1 | 81.7 | 63.0 | 33.9 | 55.2 | 36.1 | 38.8 | 58.5 | 41.9
MoCoV2 [7] | 67.5 | 57.0 | 82.4 | 63.6 | 34.2 | 55.4 | 36.2 | 39.0 | 58.6 | 41.9
MoCoV2 + ContrastiveCrop | 67.8 | 57.3 | 82.5 | 63.8 | 34.5 | 55.5 | 36.4 | 39.2 | 58.8 | 42.2

Table 4: Fine-tuning results on PASCAL VOC detection and COCO detection and
instance segmentation. All models are pre-trained for 200 epochs on
ImageNet-1K. On VOC, the training and evaluation sets are trainval2007+2012
and test2007, on COCO are the train2017 and val2017. All models are fine-tuned
for 24K iterations on VOC and 90K on COCO.

#### Center-suppressed Sampling.

In this work, we use $\beta$ distribution for the center-suppressed sampling,
which allows to control its variance with different $\alpha$. Here, we
investigate the impact of different variance by iterating over multiple
$\alpha$. Results are shown in Fig. 6(b) with $k=0.1$ for localization. When
$\alpha<1$, our ContrastiveCrop consistently outperforms RandomCrop with
localization, showing the effect of center-suppressed sampling. We also study
$\alpha>1$ that has a smaller variance than uniform distribution (_i.e_.,
$\alpha=1$). A drop in accuracy is observed with $\alpha>1$. This indicates
that larger variance of crops is required for better contrast.

####  ContrasitveCrop with Other Transformations.

To further compare the effect of ContrastiveCrop and RandomCrop, we study
their combinations with other image transformations. Here, we choose the
transformations used in MoCo V2 [7], including Flip, ColorJitter, Grayscale
and Blur. The ablation results are shown in Tab. 5. In case all other
transformations are removed, ContrastiveCrop is $0.4\%$ higher than
RandomCrop, which is a direct evidence of its superiority. Moreover, with only
one extra transformation, ContrastiveCrop outperforms RandomCrop by $0.3\%\sim
0.8\%$. The largest gap of $1.2\%$ is achieved when all of the transformations
are incorporated, which indicates that the potential of ContrastiveCrop can be
larger exploited with further color transformations. Additionally, these
results show that our ContrastiveCrop is compatible and orthogonal to other
transformations.

Flip |  ColorJitter \+ Grayscale | Blur | R-Crop | C-Crop
---|---|---|---|---
✓ | ✓ | ✓ | 63.4 | 64.6
✓ |  |  | 50.4 | 50.9
| ✓ |  | 60.6 | 61.4
|  | ✓ | 44.9 | 45.2
|  |  | 45.5 | 45.9
Table 5: Ablation of other transformations used in MoCo V2. We combine
ColorJitter and Grayscale as a single color transformation. R-Crop and C-Crop
denote RandomCrop and ContrastiveCrop, respectively. The results are from
ResNet-50 pre-trained on IN-200 for 100 epochs.

###  4.5 Downstream Tasks

In this section, we measure the transferability of our method on the object
detection and instance segmentation task. Following previous works [20, 51],
we pre-train ResNet-50 on IN-1K for 200 epochs. For downstream tasks, we use
PASCAL VOC [16] and COCO [27] as our benchmarks and we adopt the same setups
as in MoCo’s detectron2 codebase [42]. All layers of pre-trained models are
fine-tuned end-to-end at target datasets.

#### PASCAL VOC Object Detection.

Following [20], we use Faster R-CNN [32] with a backbone of R50-C4 [21] as the
detector. We fine-tune the model on the trainval2007+2012 split and evaluate
on the VOC test2007. The results are present in Tab. 4. Compared with MoCo V1
baseline, our method achieves a consistent improvement of +0.2AP, +0.2AP50 and
+0.4AP75.

#### COCO Object Detection/Instance Segmentation.

The model for both detection and segmentation is Mask R-CNN [21] with R50-C4
backbone. We fine-tune 90K iterations on the train2017 set and evaluate on
val2017. As shown in Tab. 4, the proposed ContrastiveCrop achieves superior
performance in all metrics.

##  5 Discussion and Conclusion

In this work, we propose ContrastiveCrop, that is tailored to make better
contrastive views for Siamese representation learning. ContrastiveCrop adopts
semantic-aware localization to avoid most false positives and applies the
center-suppressed sampling to reduce trivial positive pairs. We innovatively
take semantic information into account when transforming a sample, and
thoroughly investigate the suitable variance for contrastive learning. We have
shown the effectiveness and generality of our method through extensive
experiments with state-of-the-art contrastive methods including SimCLR, MoCo,
BYOL and SimSiam. Finally, we hope this work could inspire future research of
positives designing, considering its significant role in contrastive learning.

##  6 Acknowledge

This research is supported by the National Research Foundation, Singapore
under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-008). We
thank Google TFRC for supporting us to get access to the Cloud TPUs. We thank
CSCS (Swiss National Supercomputing Centre) for supporting us to get access to
the Piz Daint supercomputer. We thank TACC (Texas Advanced Computing Center)
for supporting us to get access to the Longhorn supercomputer and the Frontera
supercomputer. We thank LuxProvide (Luxembourg national supercomputer HPC
organization) for supporting us to get access to the MeluXina supercomputer.

## References

  * [1] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi.  A theoretical analysis of contrastive unsupervised representation learning.  arXiv preprint arXiv:1902.09229, 2019.
  * [2] Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  In NeurIPS, 2019.
  * [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  arXiv preprint arXiv:2104.14294, 2021.
  * [4] Pengguang Chen, Shu Liu, and Jiaya Jia.  Jigsaw clustering for unsupervised visual representation learning.  In CVPR, 2021.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [6] Ting Chen, Calvin Luo, and Lala Li.  Intriguing properties of contrastive losses.  arXiv preprint arXiv:2011.02803, 2020.
  * [7] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [8] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  CVPR, 2021.
  * [9] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  arXiv preprint arXiv:2104.02057, 2021.
  * [10] Ming-Ming Cheng, Ziming Zhang, Wen-Yan Lin, and Philip Torr.  Bing: Binarized normed gradients for objectness estimation at 300fps.  In CVPR, 2014.
  * [11] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In AISTATS, 2011.
  * [12] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation policies from data.  arXiv preprint arXiv:1805.09501, 2018.
  * [13] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In CVPR, 2020.
  * [14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [15] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.  arXiv preprint arXiv:2104.14548, 2021.
  * [16] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  IJCV, 2010.
  * [17] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  arXiv preprint arXiv:2006.07733, 2020.
  * [18] Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.  Densepose: Dense human pose estimation in the wild.  In CVPR, 2018.
  * [19] Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama.  Faster autoaugment: Learning augmentation strategies using backpropagation.  In ECCV, 2020.
  * [20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [23] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord.  Data-efficient image recognition with contrastive predictive coding.  arXiv preprint arXiv:1905.09272, 2019.
  * [24] Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao.  On feature decorrelation in self-supervised learning.  In ICCV, 2021.
  * [25] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  Technical report, Citeseer, 2009.
  * [26] Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim.  Fast autoaugment.  NeurIPS, 2019.
  * [27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [28] Shlok Mishra, Anshul Shah, Ankan Bansal, Abhyuday Jagannatha, Abhishek Sharma, David Jacobs, and Dilip Krishnan.  Object-aware cropping for self-supervised learning.  arXiv preprint arXiv:2112.00319, 2021.
  * [29] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In CVPR, 2020.
  * [30] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [31] Senthil Purushwalkam and Abhinav Gupta.  Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases.  arXiv preprint arXiv:2007.13916, 2020.
  * [32] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [33] Ramprasaath R. Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik.  Casting your model: Learning to localize improves self-supervised representations.  In CVPR, pages 11058–11067, June 2021.
  * [34] Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, Trevor Darrell, and Eric Xing.  Un-mix: Rethinking image mixtures for unsupervised visual representation learning.  arXiv preprint arXiv:2003.05438, 2020.
  * [35] Yuandong Tian, Xinlei Chen, and Surya Ganguli.  Understanding self-supervised learning dynamics without contrastive pairs.  arXiv preprint arXiv:2102.06810, 2021.
  * [36] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  ECCV, 2019.
  * [37] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning?  arXiv preprint arXiv:2005.10243, 2020.
  * [38] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou.  Fixing the train-test resolution discrepancy.  arXiv preprint arXiv:1906.06423, 2019.
  * [39] Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In ICML, 2020.
  * [40] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  CVPR, 2021.
  * [41] Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman.  On mutual information in contrastive learning for visual representations.  arXiv preprint arXiv:2005.13149, 2020.
  * [42] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [43] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In CVPR, 2018.
  * [44] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell.  What should not be contrastive in contrastive learning.  ICLR, 2021.
  * [45] Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Zhenguo Li, and Ping Luo.  Detco: Unsupervised contrastive learning for object detection.  arXiv preprint arXiv:2102.04803, 2021.
  * [46] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu.  Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.  CVPR, 2021.
  * [47] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang.  Unsupervised embedding learning via invariant and spreading instance feature.  In CVPR, 2019.
  * [48] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In ICCV, 2019.
  * [49] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  ICLR, 2018.
  * [50] Junhao Zhang, Yali Wang, Zhipeng Zhou, Tianyu Luan, Zhe Wang, and Yu Qiao.  Learning dynamical human-joint affinity for 3d pose estimation in videos.  IEEE Transactions on Image Processing, 30:7914–7925, 2021.
  * [51] Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen.  Improving contrastive learning by visualizing feature transformation.  arXiv preprint arXiv:2108.02982, 2021.
  * [52] C Lawrence Zitnick and Piotr Dollár.  Edge boxes: Locating object proposals from edges.  In ECCV, 2014.
  * [53] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le.  Rethinking pre-training and self-training.  arXiv preprint arXiv:2006.06882, 2020.


# No More Mumbles: Enhancing Robot Intelligibility through Speech Adaptation

Qiaoqiao Ren1, Yuanbo Hou2, Dick Botteldooren2, and Tony Belpaeme1 1Qiaoqiao
Ren and Tony Belpaeme are with the Faculty of Engineering and Architecture,
IDLab-AIRO, Ghent University – imec, Technologiepark 126, 9052 Gent, Belgium
(e-mail: {Qiaoqiao.Ren, Tony.Belpaeme}@UGent.be).2Yuanbo Hou and Dick
Botteldooren are with the Faculty of Engineering and Architecture, WAVES,
Ghent University, Belgium (e-mail: {Yuanbo.Hou,
Dick.Botteldooren}@UGent.be).The source code, pre-trained model and demos are
available at: https://github.com/qiaoqiao2323/robot-speech-intelligibility

###### Abstract

Spoken language interaction is at the heart of interpersonal communication,
and people flexibly adapt their speech to different individuals and
environments. It is surprising that robots, and by extension other digital
devices, are not equipped to adapt their speech and instead rely on fixed
speech parameters, which often hinder comprehension by the user. We conducted
a speech comprehension study involving 39 participants who were exposed to
different environmental and contextual conditions. During the experiment, the
robot articulated words using different vocal parameters, and the participants
were tasked with both recognising the spoken words and rating their subjective
impression of the robot’s speech. The experiment’s primary outcome shows that
spaces with good acoustic quality positively correlate with intelligibility
and user experience. However, increasing the distance between the user and the
robot exacerbated the user experience, while distracting background sounds
significantly reduced speech recognition accuracy and user satisfaction. We
next built an adaptive voice for the robot. For this, the robot needs to know
how difficult it is for a user to understand spoken language in a particular
setting. We present a prediction model that rates how annoying the ambient
acoustic environment is and, consequentially, how hard it is to understand
someone in this setting. Then, we develop a convolutional neural network model
to adapt the robot’s speech parameters to different users and spaces, while
taking into account the influence of ambient acoustics on intelligibility.
Finally, we present an evaluation with 27 users, demonstrating superior
intelligibility and user experience with adaptive voice parameters compared to
fixed voice.

###### Index Terms:

Human-Centered Robotics, Design and Human Factors, Social HRI

##  I Introduction

There is the expectation that social robots will one day be capable of
engaging in human-like spoken language interaction with people. However,
achieving effective spoken interaction with devices such as robots and digital
assistants presents a significant challenge, especially in environments with
background noise and suboptimal acoustic properties. The parameters that
govern a robot’s vocal characteristics—such as volume, speech rate, pitch, and
emphasis—can impact speech intelligibility and the overall user experience
[1].

Earlier research has underscored the detrimental effects of background noise
on speech intelligibility [2]. The reverberation time of a space, which
strongly correlates with its acoustic quality, has been found to negatively
affect speech intelligibility [3]. Additionally, the distance between the
listener and the sound source has a crucial impact on speech intelligibility
and on the user experience[4]. Earlier studies have identified pitch, pitch
range, volume, and speech rate as fundamental vocal characteristics conveying
personality [5]. For instance, voice pitch has been effectively employed to
model a robot’s personality and emotion [6]. Speech speed influences speech
intelligibility, and Jones _et al._ found that a faster speaking rate lowers
comprehension for listeners [7]. Except for the critical role of acoustic
factors, individual differences in listeners are also important. Different
listeners vary in their ability to understand speech in noisy environments
[8]. As such, it is crucial to consider the listener’s personal information
when designing the robot’s adaptivity.

This work advances the field of data-driven adaptive speech, building upon a
foundation of prior research that has explored various facets of the
challenge, which has distinct advantages, particularly when addressing
complex, variable environments and tailoring individual user needs. People
adapt their speech during spoken conversation. This is known as the Lombard
effect, which is an involuntary speech adaptation where speakers naturally
modify their speech in noisy environments to improve communication. In
response to loud environments, speakers will increase their speech volume,
adjust pitch, and enunciate more, thus optimising interaction between
interlocutors. However, robots lack this natural adaptability, highlighting
the need for advanced systems to mimic the Lombard effect to improve human-
robot interaction (HRI). Earlier studies have looked into adapting speech to
noisy environments and the distance between the user and the robot [9],
addressing the needs of individuals with hearing impairments [10], or scaling
the loudness of the voice dynamically when approaching a user [11]. Niculescu
_et al._ find that pitch adaption of a robot’s voice impacted the users’
rating of overall interaction quality [6]. However, most robots (and other
interactive devices for that matter) do not take the acoustic properties of
the environment, the ambient sound, or the user’s characteristics into account
during the conversation. The large majority of spoken language interactions
with robots use fixed voice parameters, often ignoring the dynamic nature of
environmental and user factors.

Our study addresses three research questions (RQs) - RQ1: Is there a
relationship between robot speech parameters, user characteristics, acoustic
quality of the environment and ambient sound, and the robot’s intelligibility
as well as user experience? RQ2: Can the robot use information about the
environment and the user to adapt its speech parameters? RQ3: Can the robot’s
speech parameters be dynamically adapted to optimise the user’s experience and
the robot’s intelligibility?

To address RQ1, we collect data to map how intelligibility suffers under
different conditions in Section II. From this, Section III responds to RQ2 by
building an automated system to adapt the robot’s voice to changing user and
environmental factors. Finally, we evaluate our system in a user study in
Section IV to address RQ3. Discussion is given in Section V.

##  II Intelligibility assessment

###  II-A Experimental design

####  II-A1 Materials

To reveal how ambient sound, the environment’s acoustic quality, the distance
between the user and the robot, and the user’s hearing can affect both the
robot’s intelligibility and the user’s experience, we first set up a data
collection campaign. We used a speech recognition task using a paradigm from
[12]. The task relies on open-set recognition of four lists of English
consonant-nucleus-consonant (CNC) monosyllabic words (e.g. _hash_ , _dodge_ ,
_should_ , …) from the Northwestern University Auditory Test 6 (NU-6) [13].
The words were spoken by an NAO V5 humanoid robot (United Robotics Group,
formerly known as Aldebaran Robotics). During the task, a recording of ambient
sound (ranging from soft chatter to the sound of power tools) was played
through a speaker placed near the robot. The voice parameters of the robot
(i.e., volume, speech rate, pitch, and emphasis) were drawn using a uniform
distribution from predetermined parameter ranges, thus sampling the voice
parameter space.

Figure 1: Experimental procedure.

####  II-A2 Procedure

Participants were recruited via a social media campaign in the local area,
excluding those with self-reported reading-related learning difficulties, to
ensure data quality. The data collection and study adhered to the ethics
procedures of the _Universiteit Gent_. Participants gave informed consent and
were told they could withdraw at any time without providing a reason. The
experiment process is shown in the Fig. 1. A pre-test questionnaire gathered
personal information, including gender and age, their self-evaluated hearing
difficulties in noisy environments, and their self-reported Common European
Framework of Reference for Languages (CEFR) levels as their English levels. 39
participants (17 identified as female, 20 as male, 2 as other; $28.0\pm 3.7$
years old) took part in the data collection exercise. Each participant was
randomly assigned to one of six different rooms, each having its own acoustic
quality. The participants were exposed to three different ambient sounds drawn
from a total of 11; complete silence was used as a baseline condition. After,
the participants were placed at a random interaction distance to the robot,
ranging from 60 to 500 cm. Each participant is exposed to 50 words per round;
During the task, the robot spoke a word picked randomly from the NU-6 list and
the participant was asked to type the word using a keyboard, along with
providing feedback on the user experience. Fig. 2 shows examples of different
settings in which data was collected.

Figure 2: Illustration of different environments in which data was collected.

Next, we build a model to examine how different factors influence the
intelligibility of the robot’s speech and the user experience. As independent
variables, we have two participant characteristics, four robot voice
parameters, and three environmental factors. As dependent variables, we
collected similarity scores and user experience from the participants per
word. Therefore, each word is a tuple consisting of the independent variables
and dependent variables. We collected 5442 tuples in total.

###  II-B Factors influencing intelligibility

####  II-B1 Participant characteristics

We collect data on participants’ hearing loss and English language level.
Individuals with normal hearing already experience difficulties in
comprehending speech in noisy environments, and this is exacerbated when
people experience hearing loss. We therefore ask participants to self-report
hearing loss. Moreover, listening in adverse conditions presents more
challenges for non-native listeners in a foreign language context [14]. Thus,
we also record their English proficiency level. Specifically, participants
rated their difficulty hearing in noisy environments, such as restaurants or
social gatherings, using a Likert scale (1: No, never, to 5: Yes, always),
which has been used in previous research [15] and rated their English
proficiency using the CEFR scale, which rates English proficiency from A1 (1:
beginner) to C2 (6: fluent). This yields distributions for hearing difficulty
($2.2\pm 1$) and English proficiency level ($4.8\pm 0.8$).

####  II-B2 Robot voice parameters

The second data category relates to four specific parameters associated with
the robot’s vocal characteristics: speed, pitch, emphasis, and volume. These
features play a crucial role in determining the clarity and intelligibility of
the robot’s speech. We study their impact on participants’ experiences and
speech intelligibility.

The words are produced using the built-in US English speech engine of the NAO
robot, which is supplied by Nuance. The designers of NAO state that the
default voice is the voice of a machine but with a fluidity that makes it very
comfortable to listen to [16], and the default voice has been widely used in
research studies [17]. We modulate four key parameters in the voice engine:
volume, speed, pitch, and emphasis (which is implemented by double voice
synthesis and controlled by the doubleVoiceLevel parameter, which adds a
second voice over the first voice). The volume ranges between [0, 2], and the
pitch ranges between [0.5, 2], with a default of 1. Speed is set as a multiple
of the default rate of 100%, ranging between [0.4, 4], and governs the pace at
which the speech is articulated. Lastly, the doubleVoiceLevel, ranging between
[0, 4], grants control over the gain of a second voice; 0 disables the effect.
In the word recognition task, all the voices are played with NAO’s
loudspeakers set to 100%. The default voice settings for the voice
parameters—volume, speed, and pitch are all set at 1, with the double voice
effect deactivated.

####  II-B3 Environmental factors

The third category of data comprises three environmental factors. The
reverberation time (T30), the participants’ annoyance rating (AR) of the
ambient noise, and the distance of the individuals to the robot.

Reverberation time is a key metric for assessing a room’s acoustic quality.
The T30 is a standard metric and reports the time it takes for the sound
pressure level to decrease by 60 decibels [18], as detailed in the ISO3382
standard. To measure the T30, an impulse noise (a popping balloon) was used,
and the reverberation time was measured using a Svantek SVAN 959 Sound and
Vibration Analyser. During the measurement, the impulse source and microphone
were 1.0m above floor level, and we measured at three positions for each room
(the middle of the room and two diagonal corners). The T30 measures were
averaged over a frequency range from 250Hz to 2kHz based on the three
measurements per room [19]. The measured results are given in Fig. 3. For the
data collection and evaluation, we used different spaces, each with varying
reverberation times (in seconds(s)). Data was collected in an office setting,
which includes an open-plan area (T30 $=0.78$s), meeting room (T30 $=0.56$s),
and a home setting (the imec Homelab) consisting of a living room (T30
$=0.43$s), bedroom (T30 $=0.19$s) and meeting room (T30 $=0.32$s), and an
anechoic chamber (T30 $=0.04$s) used for echo-free sound assessment. Those
environments offer a stable setting without disturbance from uncontrollable
variables like crowds.

Figure 3: Reverberation time (T30) of different rooms.

Next to the different rooms, we also make use of different ambient sounds. The
subjective perception of these sounds– and specifically the extent to which
these sounds are perceived as annoying, which is correlated with the volume of
the sounds– is expected to influence the participant’s performance on tasks
[20]. The ambient noise played in the experiment is from the DeLTA dataset,
which includes 2980 samples of 15-second binaural audio recordings with 24
classes of sound sources and human-annotated annoyance ratings [21]. The
annoyance rating ranges from 1 (not annoying at all) to 10 (very annoying).
All the ambient sounds are played at the same speaker volume, guaranteeing a
consistent playback level. For the robot to quantify the subjective experience
of ambient sounds, it needs a model that maps a sound recording into an
annoyance rating. Important here is that the model not only looks at the
amplitude of the ambient sound but also takes a holistic view of how annoying
the sound is.

Finally, the distance participants keep from the robot will impact the robot’s
intelligibility. During data collection, we sampled distances ranging from 60
to 500 cm.

###  II-C Intelligibility metrics

The intelligibility metrics encompass both user experience and speech
intelligibility.

####  II-C1 User experience

Participants’ user experience related to the robot’s voice is also collected.
This is rated from 1 (not pleasant at all) to 10 (very pleasant). These
measures help gauge the affective and subjective aspects of the interaction.

####  II-C2 Speech intelligibility

Participants were tasked with typing the words spoken by the robot. We
evaluated the robot’s speech intelligibility by measuring the phonetic
similarity score between the spoken words and the words typed by the
participants. This compensates for homophones, such as _flour_ and _flower_ ,
and gives us a scalar metric for how poorly the word was understood. We used a
phonetic embedding method to calculate the phonetic similarity of test and
input words from participants, as in [22]. This phonetic embedding represents
each phonetic component as a vector in a continuous vector space, thus
encoding phonetic information, considering phoneme alignment in the words and
capturing intricate phonetic characteristics and enhancing similarity
assessment.

###  II-D Generalised linear mixed-effect models

Based on the independent variables collected in Section II-B and dependent
variables from Section II-C, we build a generalised linear mixed-effect model
(GLMM) to study the contributions of various factors to spoken language
intelligibility and to the user experience, including fixed effects, interact
effects and random effects. We use the method by [23], which can handle non-
normal distributed response variables. A Variance Inflation Factors analysis
revealed an absence of multicollinearity among the independent variables - all
factors were below the threshold ($\leq 2$). In addition, the correlations
between the annoyance rating and acoustic quality, as well as the user
information, are all smaller than 0.1.

The gamma distribution is a continuous probability distribution to model data
that is skewed, continuous and non-negative [24], such as the residuals of the
phonetic scores and user experience that we observed. To choose the most
accurate model, we compare the prediction error using the Akaike Information
Criterion (AIC) of the reverse link function and the log link function. We
chose the log link function, as it obtained the lowest AIC.

The generalised model examines the relationship between predictor variables
and both the intelligibility and user experience, as represented by Eq II-D
and Eq II-D. To adequately account for the non-independence of observations
stemming from repeated measurements across subjects and the inherent
variability in word difficulty, we have integrated subject ID and word ID as
random effects in our models to control for intra-subject variability and
variable difficulty levels of the words tested.

$\displaystyle\text{SP}_{ij}\sim\text{Gamma}(\mu_{ij},\phi)$
$\displaystyle\log({\mu_{ij}+\mu})=\beta_{0}\cdot\text{v}_{ij}+\beta_{1}\cdot\text{p}_{ij}+\beta_{2}\cdot\text{e}_{ij}+\beta_{3}\cdot\text{s}_{ij}$
$\displaystyle\quad+\beta_{4}\cdot\text{A}_{ij}+\beta_{5}\cdot\text{T}_{ij}+\beta_{6}\cdot\text{D}_{ij}+\beta_{7}\cdot\text{E}_{ij}+\beta_{8}\cdot\text{H}_{ij}$
$\displaystyle\quad+\beta_{9}\cdot(\text{v}\cdot\text{T})+1|pn_{ij}+1|w_{ij}+\epsilon_{1}$
$\displaystyle\epsilon_{1}\sim\text{Normal}(0,\sigma^{2})$ (1)

$\displaystyle\text{UX}_{ij}\sim\text{Gamma}(\lambda_{ij},\phi)$
$\displaystyle\log(\lambda_{ij})=\lambda_{0}\cdot\text{v}_{ij}+\lambda_{1}\cdot\text{p}_{ij}+\lambda_{2}\cdot\text{e}_{ij}+\lambda_{3}\cdot\text{s}_{ij}$
$\displaystyle\quad+\lambda_{4}\cdot\text{A}_{ij}+\lambda_{5}\cdot\text{T}_{ij}+\lambda_{6}\cdot\text{D}_{ij}+\lambda_{7}\cdot\text{E}_{ij}+\lambda_{8}\cdot\text{H}_{ij}$
$\displaystyle\quad+\lambda_{9}\cdot(\text{v}\cdot\text{T})+1|pn_{ij}+1|w_{ij}+\epsilon_{2}$
$\displaystyle\epsilon_{2}\sim\text{Normal}(0,\sigma^{2})$ (2)

Eq II-D and Eq II-D use the coefficients $\beta_{0}$ to $\beta_{9}$ and
$\lambda_{0}$ to $\lambda_{9}$ respectively to represent the estimated impact
of predictor variables on the response variables (speech intelligibility, user
experience). They show how each predictor is expected to log-linearly impact
the response, and these effects are fixed across the dataset, not subject to
random variation within the model’s framework. Specifically, the coefficients
$\beta_{0}$ to $\beta_{3}\ ,\lambda_{0}$ to $\lambda_{3}$ correspond to the
voice attributes (volume, pitch, emphasis, and speed) of the robot on speech
intelligibility and user experience, respectively, while $\beta_{4}$ to
$\beta_{6}\ ,(\lambda_{4}$ to $\lambda_{6}$) represent factors such as room
characteristics (AL, T30) and distance to the robot. English proficiency of
participants and their auditory challenges in noisy environments are signified
by $\beta_{7}$ to $\beta_{8}$ and $\lambda_{7}$ to $\lambda_{8}$ for user’s
intelligibility and user experience, respectively. The coefficients
$\beta_{9}$ pertain to the interaction of volume and the room’s acoustic
quality and the influence on both speech intelligibility and user experience.
Similarly, $\lambda_{9}$ represents the interactions in the context of user
experience. Simultaneously, the unaccounted-for residuals that evade fixed and
random effects are encapsulated by the residual component $\epsilon$. The
subscript $ij$ indicates the $j$th observation within the $i$th group. The
response variable $SP_{ij}$, $UX_{ij}$ embodies speech intelligibility and
user experience in this context, respectively, and $\mu$ is $0.0001$.

We used the SIMR package in R for post-hoc power analysis, based on the 1000
Monte Carlo simulations [25]. As the log link function was used in GLMM, we
did the exponential transform of the coefficients ( $\beta_{0}$ to $\beta_{9}\
,\lambda_{0}$ to $\lambda_{9}$ ) to get the estimate of the fixed effect for
speech intelligibility and user experience.

###  II-E Intelligibility model results

####  II-E1 Speech intelligibility model analysis

The negative estimate of $-0.75$ associated with annoyance rating underscores
the adverse impact of heightened annoyance on speech intelligibility
($p<0.001$), the post-hoc power is 98.60%. Additionally, the acoustic quality
of the space (T30) has a significant negative estimate of $-0.77$ with speech
intelligibility ($p<0.01$), the post-hoc power is 66.90%; this finding
signifies that good acoustic conditions of a space are conducive to
comprehension of the robot’s speech. Additionally, the speed has a significant
negative estimate of $-0.21$ ($p<0.001$) of speech intelligibility, which
obtained 100% post-hoc power. Finally, pitch has a negative estimate of
$-0.71$ ($p<0.001$), which implies that the high pitch leads to decreased
intelligibility with 91.70% post-hoc power. We observed some weak estimates
without statistical significance ($p>0.05$). For instance, there is a negative
correlation between the distance, emphasis as well as hearing difficulty and
speech intelligibility. In terms of English proficiency level, which serves as
an indicator of language skills, our analysis revealed a marginally positive
coefficient.

We explored the interplay between the T30 of spaces and the volume of the
robot’s voice and its impact on speech intelligibility, the results suggest a
significant positive estimate of $1.44$ interaction effect between room
characteristics and the volume of the robot’s voice ($p<0.001$) with 76.20%
power, indicates that the relationship between volume and speech
intelligibility is influenced by the quality of room acoustics, and the
optimal volume for improve the robot speech intelligibility depending quality
of the spatial acoustic, which might explain why the post-hoc power for T30 is
relatively lower. This may indicate that the effect size of it is trivially
small compared with the interaction effect. The random effect attributed to
subjects and words ID accounts for smaller than 1% of the overall variance for
speech intelligibility. This indicates that the influence of the random
effect, including the subject and word ID, on the total variability of speech
intelligibility is modest.

####  II-E2 User experience model analysis

The results showed that the predictor annoyance rating exhibited a negative
estimate of $-0.74$ ($p<0.001$) and the post-hoc power is 83.00%, signifying
that heightened annoyance ratings due to ambient sounds correspond to a
decrease in user experience. Likewise, the predictor pitch displayed a
significant negative coefficient of $-0.89$ ($p<0.05$) with 65.6% post-hoc
power, indicating that elevated pitch levels are associated with lower user
experience scores. This suggests that an increased pitch might have an adverse
impact on user experience during such interactions. The predictor distance
also yielded a significant negative coefficient of $-0.74$ ($p<0.01$) with
95.7% post-hoc power, implying that greater distances from the sound source
are linked to lower user experience scores. Furthermore, the predictor T30 of
space revealed a negative coefficient of $-0.67$ with statistical significance
($p<0.01$) and the post-hoc power is 81.30%. This implies that the quality of
the room environment significantly influences user experience, with a decrease
in room quality corresponding to a decrease in user experience. Surprisingly,
the English proficiency level of users exhibited a significant negative
estimate of $-0.54$ with user experience ($p<0.01$) with 75.20% post-hoc
power. This implies that individuals with higher language proficiency
experienced the robot’s speech as less pleasant. This might be attributed to
their heightened expectations regarding word intelligibility. In addition, the
predictor speed ($p<0.001$) displayed a significant negative coefficient of
$-0.56$ with 99.6% post-hoc power, indicating that faster speech rates result
in a poorer user experience.

Interestingly, the predictor hearing difficulty in noisy environments did not
predict user experience ($p>0.05$). This suggests that participants who
reported higher hearing difficulty experienced a significantly better user
experience listening to robot speech. In contrast, the predictor emphasis
showed a negative estimate with user experience in this analysis ($p>0.05$),
although it was not statistically significant. In the context of user
experience, the direct effect of volume on user experience showed a
significant negative relationship (fixed estimate of -0.92, $p<0.01$),
suggesting that higher volume levels are generally perceived negatively.
However, this finding comes with an important caveat as the statistical power
for this effect was notably low (18.8%), indicating potential limitations in
the robustness of this result. Our finding indicated the significant
interaction between room acoustics and the robot’s volume (coefficient 1.32,
$p<0.001$), with a robust post-hoc power of 83.3%. The positive interaction
coefficient indicates that the optimal volume for enhancing user experience
shifts depending on the acoustic quality of the room. This interaction effect
potentially explains the low power observed in the direct effect of volume,
which shows that the robot voice should be adapted to the environment to get
better user experience and intelligibility. The random effect attributed to
individual subjects and word ID accounts for less than 5% of the overall user
experience variance. This suggests the user experience is similar for all
subjects and all words.

##  III Building an adaptive robot voice

One of the key takeaways from our analysis is the importance of adapting the
robot’s voice to the environment and user. To create an adaptive robot voice,
we required a model to assess ambient sound annoyance. Then, we constructed an
adaptive speech model that leverages the annoyance rating, and together with
other parameters– the distance to the robot, the T30 reverberation of the
room, the user’s English proficiency, and hearing difficulties– adjusts the
robot’s speech parameters.

###  III-A Ambient sounds’ annoyance rating prediction

As ambient sound is known to strongly impact the intelligibility of speech and
user experience, we require an automated method to rate the annoyance of
ambient sounds. This not only takes into account the amplitude of the sound,
but also takes into account the frequency spectrum of ambient sound to predict
the annoyance rating (AR). This is a scalar value used as an overall
evaluation metric to reflect the impact of ambient sounds on participants’
ability to understand speech. Fig. 4 shows the convolutional neural network
(CNN)–based annoyance rating prediction (ARP) model we use for this.

Figure 4: The CNN-based ARP model for overall evaluation of ambient sounds.

Inspired by the outstanding performance of the convolution-based sound source
classification model [26], the ARP model uses 6 convolutional blocks of the
same structure but with a sequentially increasing number of filters. Each
convolutional block consists of two convolutional layers equipped with ReLU
activation functions and a global average pooling layer [27]. To stabilise and
speed up the neural network training, batch normalisation (BN) [28] is
introduced. After the convolutional blocks, an embedding layer consisting of
fully connected layers with a dimension of 2048 is used to transform ambient
sound representations extracted by convolutional blocks into high-level
embeddings suitable, for the final prediction layer of the whole environmental
sound.

####  III-A1 Ambient sound ARP model training

To train the ARP model to successfully infer the impact of environmental
sounds on participants, i.e., the annoyance rating for participants, based on
acoustic features containing amplitude, frequency, and category information of
ambient sounds, we trained the model on the DeLTA real-life polyphonic audio
dataset [21]. The DeLTA contains 2980 samples of 15-second binaural audio
recordings with 24 classes of sound sources from European cities for noise
annoyance detection. Participants rated 15-second binaural recordings of urban
environments by providing an annoyance rating on a scale of 1 to 10.

As shown in Fig. 4, given the prediction output from the final prediction
layer is $\hat{y_{ar}}$, and its corresponding label is ${y_{ar}}\in[1,10]$,
the mean squared error (MSE) is used as the loss function for the ambient
sound ARP model, $Loss=MSE(\hat{y_{ar}},y_{ar})$. To comprehensively consider
polyphonic audio information such as the amplitude and frequency of ambient
sound sources, the acoustic feature log Mel that performs well in sound-
related tasks is adopted. Log Mel-filter 64-bank spectrogram is extracted by
the Short-Time Fourier Transform with a Hamming window length of 46ms and a
window overlap of $1/3$ [29], which means that the duration of each frame is
0.031s, and the acoustic features can capture the acoustic information of
sounds with a time resolution of 0.031s. In training, dropout and
normalisation are used to prevent over-fitting of the model. The Adam
optimiser [30], with an initial learning rate of 0.005, minimises the loss
function, with a batch size of 64. The model is trained for 100 epochs. In
inference, using a Tesla V100-SXM2-32GB GPU and Intel(R) Xeon(R) CPU E5-2698
<EMAIL_ADDRESS>as an example, the response time of the model from inputting
acoustic features to outputting prediction results is 1.87ms, which means the
model can process input sounds in real-time.

####  III-A2 Ambient sound ARP model performance

We used 2200 clips from the DeLTA dataset to train the model, retaining 245
clips for validation and 445 for testing. Table I shows the performance of the
proposed CNN-based ARP model on the test set, and also shows the performance
of several other typical neural network models. Among them, the deep neural
network (DNN) consists of four fully connected (FC) layers and the ARP layer.
Each FC layer’s number of units is 64, 128, 256, and 512, respectively. The
final FC layer’s output is flattened and input to the ARP layer. The Simple
CNN consists of 4 convolutional layers, each equipped with (3 × 3) kernels,
and the ARP layer. Each convolutional layer’s filter numbers are 64, 128, 256,
and 512, respectively. The final convolutional layer’s output is flattened and
input to the ARP layer. The CNN-Transformer consists of 3 convolutional layers
with (3 × 3) kernels, a Transformer encoder [31], and the ARP layer. In Table
I, the results of the simple CNN are slightly better than those of the DNN,
illustrating the effectiveness of the convolutional layer in extracting
information such as the amplitude and frequency of ambient sounds from the
input acoustic features. The CNN-Transformer, equipped with a Transformer
encoder suited for temporal modelling, performs better than a simple CNN.
Finally, the proposed CNN model achieved more competitive results with lower
MSE and MAE. This shows that the proposed CNN-based ARP model can effectively
predict the impact of ambient sound based on its comprehensive information on
the participants.

TABLE I: The ARP performance on the test set of the DeLTA dataset. Model |  DNN |  Simple CNN |  CNN-Transformer |  Proposed CNN
---|---|---|---|---
MSE |  1.73 |  1.67 |  1.45 |  1.10
MAE |  1.01 |  1.00 |  0.97 |  0.83

###  III-B Proposed adaptive speech model

Our findings emphasise that it is crucial to tailor the robot’s voice to the
specific environment to achieve both optimal speech intelligibility and a
positive interaction experience for different users. We trained a neural
network to predict optimal robotic voices based on user and environment
characteristics. The network is trained only on data for which participant
speech recognition was completely accurate and user satisfaction was greater
than 5. This excludes data of negative user experiences or where the robot’s
speech was poorly understood. We call this model the Environment-to-Voice
model (ETV). It takes environmental factors as input, including annoyance
ratings of ambient sound (reported by the ARP model), the distance to the
robot, the English proficiency of the user, the user’s difficulty hearing in
noisy environments, and the room’s acoustic quality. The model’s output
consists of robot voice parameters, which consist of volume, speed, pitch, and
emphasis. The network has 5 input nodes, 2 hidden layers with a ReLU
activation function (16 and 32 neurons, respectively) and a 4-neuron output
layer with a linear activation function.

Model training: Data augmentation is used to expand the dataset and introduce
noise. We use the Adam optimiser [30], with a default learning rate of 0.0001.
The model trains for 200 epochs, achieving a minimal MSE of 0.15.

We evaluated the adaptive robot voice using the method reported in section
II-A. The ARP model is used to predict environmental annoyance ratings via the
robot. Notably, due to the use of a lightweight model, the computation,
despite running on the CPU, only requires 2 seconds. Subsequently, the
predictions are communicated to a pre-trained ETV model. This predictive data,
together with user-provided information on hearing difficulties and English
proficiency and assessments of acoustic quality, serves as input for the ETV
model, which then generates adaptive voice parameters. The robot speech
adaptive system is visually depicted in Fig. 5.

##  IV Evaluation

###  IV-A Evaluation experiment design

To assess the generalisation of our proposed ETV model with unseen data, we
conducted an evaluation experiment involving 27 participants (14 identified as
female, 13 as male; $26.6\pm 2.4$ years old, Hearing difficulty: $2.4\pm 0.6$,
English proficiency level: $4.8\pm 0.92$). The experiment followed a within-
subject design, using two conditions - (1) Adaptive Speech: The robot’s speech
parameters were dynamically adjusted based on the environmental and spatial
settings. (2) Fixed Speech: The robot uses its default voice parameters.

Figure 5: Robot speech adaptive system.

Each participant experienced both conditions in a balanced random order. The
evaluation experiment took place in two rooms with T30 values of $0.04$ and
$0.56$, respectively. Participants were asked to type the words spoken by the
robot within one of these two rooms. Within each condition, there were two
sessions featuring distinct ambient sounds, which are high annoyance rating
(AR) ($AR\geq 5$) and low AR ($AR<5$). In each session, the robot spoke 15
words randomly drawn from the NU-6 list.

Figure 6: User experience ratings for default robot voice (left) and the
adaptive voice (right).

###  IV-B Fixed robot voice versus adaptive voice

In our within-subject design study, the phonetic similarity scores and user
experience data did not follow a normal distribution; hence we used the non-
parametric Wilcoxon Signed-Rank Test; this analysis yielded an effect size
(r), demonstrating that the users’ intelligibility of the robot’s speech is
significantly better ($Z=-3.79,p<0.001$) for the adaptive robot voice than for
the fixed robot voice. The adaptive robot voice also leads to a significantly
better user experience ($Z=-2.99,p=0.003$). This can be seen in Fig. 7, Fig. 6
and Tab II; The post-hoc power is calculated using G*Power [32] with 27
participants. The phonetic similarity scores post-hoc power is $97.5\%$ with
$\alpha=0.05$ and $r=0.73$, and user experience post-hoc power is $88.8\%$
with $\alpha=0.05$ and $r=0.58$. The horizontal line that splits the box in
two is the median, which in Fig. 7 coincides with the top line; the mean is
indicated by the red dot. The top and bottom boundaries of the box indicate
the 25th and 75th percentiles, respectively.

Figure 7: Intelligibility of the robot, expressed as phonetic similarity scores, when it used the default voice (left) or its adaptive voice (right). TABLE II: Means and Standard Deviations for Phonetic Similarity with fixed and adaptive robot voice parameters AR |  Voice parameters |  Scores |  User experience
---|---|---|---
(1-10) |  adaptive Vs fixed |  Mean $\pm$ Std |  Mean $\pm$ Std
All |  fixed |  0.73 $\pm$ 0.31 |  6.27 $\pm$ 2.42
|  adaptive |  0.79 $\pm$ 0.29 |  6.60 $\pm$ 2.43
Low AR |  fixed |  0.78 $\pm$ 0.28 |  6.60 $\pm$ 2.36
|  adaptive |  0.81 $\pm$ 0.28 |  6.86 $\pm$ 2.32
High AR |  fixed |  0.67 $\pm$ 0.33 |  5.93 $\pm$ 2.43
|  adaptive |  0.76 $\pm$ 0.30 |  6.34 $\pm$ 2.52
|  |  |

###  IV-C Fixed robot voice under different annoyance ratings environment
sound

As all the participants experienced the two different kinds of ambient sounds,
i.e. low annoyance and high annoyance sounds, the Wilcoxon Signed-Rank Test
was used to evaluate the differences between fixed robot voice and adaptive
voice under high AR and Low AR, respectively. The results can be seen in the
Tab. II. For high AR, we observed that the users rated significantly higher
pleasantness for adaptive voice than the fixed default voice setting
($Z=-4.01,p<0.001$) and significantly better scores are observed for the
adaptive voice than the default voice ($Z=-3.82,p=0.009$). However, there is
no significant difference between the adaptive voice and the default voice on
the scores under the low AR. Interestingly, the users rate the adaptive voice
to be significantly more pleasant than the default voice ($Z=-2.33,p=0.043$).
This indicates that in the case of low ambient noise, although the adaptive
sound does not improve the intelligibility, it does improve the user
experience.

##  V Discussion

###  V-A Recommendations

As expected, there is a relationship between robot speech parameters, user
characteristics, environment factors, and the robot’s intelligibility as well
as user experience. Specifically, we find that annoying sounds in the
environment make it harder to understand the robot’s speech, which in turn
hurts the user experience. This emphasises the need for adaptive robot speech
in noisy settings. In addition, the room’s acoustic quality influences the
intelligibility and the user experience, highlighting the need to take the
room into account. Moreover, increased distance will worsen the user
experience. Therefore, it might be interesting to further explore the proper
distance for spoken conversation between a robot and its user. The results
show that, as expected, the default and therefore nonadaptive robot voice will
lead to subpar speech intelligibility and user experience when there is
ambient sound in the environment. In brief, a robot voice that does not adapt
to the user and the environment is a missed opportunity.

Regarding the factors related to the user, unsurprisingly, English proficiency
matters. People understand the English words spoken by the robot better when
they are more proficient in English. Surprisingly, they report having a lower
interaction experience, which we hypothesise might be explained by their
having stricter pronunciation expectations for the robot. Similarly, the
user’s hearing difficulties will exacerbate their understanding of the robot.
These outcomes strongly suggest that user characteristics should be taken into
consideration when setting the robot’s voice and speech parameters.

As for robot speech parameters, a higher volume will help intelligibility, but
worsen the user experience. Unsurprising, as shouting makes you heard, but it
is not pleasant. Slow speech will improve both the user experience and the
intelligibility. Surprisingly, robots with higher pitch values lead to poorer
satisfaction and lower intelligibility. One possible explanation is that users
may find sharp voices unpleasant or may not appreciate the emotional
implications conveyed by higher pitch levels. For instance, earlier research
noted that menacing voices often exhibit an increase in both volume and pitch.
Additionally, increased pitch levels are commonly associated with expressions
of anger [33]. Surprisingly, better emphasis —implemented through adding a
double voice — had a small negative influence on both intelligibility and user
experience. This effect could be attributed to the anomaly of emphasising a
single word without a context.

Designing adaptive, user-centric robot systems that consider the individual
user and environmental conditions is essential for effective communication in
HRI. This requires the robot to be informed about the environment and the
user, ideally in an automated way. The distance between the user and the robot
was now manually entered into the model, but could, of course, be extracted
from an RGBD camera. T30 reverberation estimates could also be approximated by
the robot through speech [34] during the conversation or by playing an impulse
sound when entering a new environment and analysing the reverberation. The ARP
model predicts the impact of ambient noise. Other factors, such as the user’s
language proficiency or potential hearing difficulties are harder to
automatically extract but could be collected during the making of a user
profile. In addition, from the evaluation experiment, we can conclude that the
adaptive robot voice optimises the user’s experience and the robot’s
intelligibility than its default voice.

###  V-B Limitations

This study has certain limitations. First, the data we collected suffers from
a class imbalance across predictors, as people with hearing difficulties or
very low English proficiency levels are underrepresented. This imbalance can
introduce bias into the model’s outcomes. Secondly, the participants may not
be representative of the broader population, thus limiting the
generalizability of the findings. Future studies could draw from a more
diverse user population. The experimental design, involving both within-
subject and between-subject factors, introduces an additional complexity as
not all group conditions for environmental factors are fully covered within
the study. In addition, even though the self-reported hearing difficulties or
English levels has been used in previous research, it might still be
influenced by individuals’s perception.

The speech recognition task relies on recognising single words, which is a
difficult task. Embedding words in a sentence would improve intelligibility
due to the availability of context. Our data was collected under controlled
conditions, and a range of factors typical of real-world HRI scenarios, such
as the presence of other speakers or contextual cues, were not taken into
account. Our findings may not fully extend to more complex scenarios. However,
it serves as a foundation for optimizing robots for more complicated
scenarios.

One participant noted that it might be advantageous to inquire about
participants’ listening and English _writing_ abilities specifically, as the
task exclusively involves listening and writing rather than an assessment of
overall English proficiency; in addition, the different participant’s accent
or background could influence their perception of words, especially for non-
native speakers, who might struggle with understanding short phrases and
single words. Finally, the position of the robot, whether on the left or right
side of the participants, may influence their intelligibility of the robot’s
speech.

###  V-C Conclusion

Given our results, it is difficult to understand why virtually all current
robot voices do not adapt to the environmental context and the diversity in
users. Our research sheds light on how adaptive voices can have a positive
impact on both the user experience and speech intelligibility. Simply put,
shouting louder to be better understood is not the solution. Instead, a
judicious balance is needed between making the robot understood and not
irritating the user. In light of these, we presented the ETV model that seeks
to enhance robot speech intelligibility by predicting and setting user-adapted
and contextually suitable robot voice parameters, tailored to users, and to
spatial and environmental conditions. Important to note is that the use of a
data-driven model alleviates the need to use heuristics to change the robot’s
voice.

## References

  * [1] Robinson _et al._ , “Designing sound for social robots: Candidate design principles,” _International Journal of Social Robotics_ , vol. 14, no. 6, pp. 1507–1525, 2022.
  * [2] K. Ishikawa, S. Boyce, L. Kelchner, M. G. Powell, _et al._ , “The effect of background noise on intelligibility of dysphonic speech,” _Journal of Speech, Language, and Hearing Research_ , vol. 60, no. 7, pp. 1919–1929, 2017.
  * [3] M. Klatte _et al._ , “Effects of noise and reverberation on speech perception and listening comprehension of children and adults in a classroom-like setting,” _Noise and Health_ , vol. 12, no. 49, p. 270, 2010.
  * [4] Walters and o, “Human approach distances to a mechanical-looking robot with different robot voice styles,” in _Proc. of RO-MAN_ , 2008, pp. 707–712.
  * [5] W. Apple _et al._ , “Effects of pitch and speech rate on personal attributions.” _Journal of personality and social psychology_ , vol. 37, no. 5, p. 715, 1979.
  * [6] A. Niculescu, B. van Dijk, A. Nijholt, H. Li, and S. L. See, “Making social robots more attractive: the effects of voice pitch, humor and empathy,” _International Journal of Social Robotics_ , vol. 5, pp. 171–191, 2013.
  * [7] C. Jones, L. Berry, and C. Stevens, “Synthesized speech intelligibility and persuasion: Speech rate and non-native listeners,” _Computer Speech & Language_, vol. 21, no. 4, pp. 641–651, 2007.
  * [8] A. Heinrich, H. Henshaw, and M. A. Ferguson, “The relationship of speech intelligibility with hearing sensitivity, cognition, and perceived hearing difficulties varies for different speech perception tests,” _Frontiers in psychology_ , vol. 6, p. 782, 2015.
  * [9] E. Martinson and D. Brock, “Improving human-robot interaction through adaptation to the auditory scene,” in _Proceedings of ACM/IEEE HRI_ , 2007, pp. 113–120.
  * [10] G. Lindley, “Adaptation to loudness: Implications for hearing aid fittings.” _The Hearing Journal_ , vol. 52, no. 11, pp. 50–52, 1999.
  * [11] K. Fischer, L. Naik, _et al._ , “Initiating human-robot interactions using incremental speech adaptation,” in _Proceedings of ACM/IEEE HRI_ , 2021, pp. 421–425.
  * [12] J. R. Dubno, D. D. Dirks, and D. E. Morgan, “Effects of age and mild hearing loss on speech recognition in noise,” _The Journal of the Acoustical Society of America_ , vol. 76, no. 1, pp. 87–96, 1984.
  * [13] T. W. Tillman and R. Carhart, _An expanded test for speech discrimination utilizing CNC monosyllabic words: Northwestern University Auditory Test No. 6_. USAF School of Aerospace Medicine Brooks Air Force Base, TX, 1966.
  * [14] M. L. G. Lecumberri _et al._ , “Non-native speech perception in adverse conditions: A review,” _Speech communication_ , vol. 52, no. 11-12, pp. 864–886, 2010.
  * [15] T. M. Mikkola, H. Polku, E. Portegijs, M. Rantakokko, T. Rantanen, and A. Viljanen, “Self-reported hearing status is associated with lower limb physical performance, perceived mobility, and activities of daily living in older community-dwelling men and women,” _Journal of the American Geriatrics Society_ , vol. 63, no. 6, pp. 1164–1169, 2015.
  * [16] Y. Kali, M. Saad, J.-F. Boland, J. Fortin, and V. Girardeau, “Walking task space control using time delay estimation based sliding mode of position controlled nao biped robot,” _International Journal of Dynamics and Control_ , vol. 9, pp. 679–688, 2021.
  * [17] Tozadore _et al._ , “Wizard of oz vs autonomous: Children’s perception changes according to robot’s operation condition,” in _IEEE RO-MAN_ , 2017, pp. 664–669.
  * [18] Paini _et al._ , “Is reverberation time adequate for testing the acoustical quality of unroofed auditoriums?” _Proceedings of Ins. Ac._ , vol. 28, no. 2, pp. 66–73, 2006.
  * [19] M. S. Dobreva, W. E. O’Neill, and G. D. Paige, “Influence of aging on human sound localization,” _Journal of neurophysiology_ , vol. 105, no. 5, pp. 2471–2486, 2011.
  * [20] D. Västfjäll, “Influences of current mood and noise sensitivity on judgments of noise annoyance,” _The Journal of psychology_ , vol. 136, no. 4, pp. 357–370, 2002.
  * [21] A. Mitchell, M. Erfanian, C. Soelistyo, T. Oberman, _et al._ , “Effects of soundscape complexity on urban noise annoyance ratings: A large-scale online listening experiment,” _International Journal of Environmental Research and Public Health_ , vol. 19, no. 22, p. 14872, 2022.
  * [22] R. Sharma, K. Dhawan, and B. Pailla, “Phonetic word embeddings,” _arXiv preprint arXiv:2109.14796_ , 2021.
  * [23] J. A. Nelder _et al._ , “Generalized linear models,” _Journal of the Royal Statistical Society Series A: Statistics in Society_ , vol. 135, no. 3, pp. 370–384, 1972.
  * [24] Moran _et al._ , “New models for old questions: generalized linear models for cost prediction,” _Journal of evaluation in clinical practice_ , vol. 13, no. 3, pp. 381–389, 2007.
  * [25] P. Green and C. J. MacLeod, “Simr: An r package for power analysis of generalized linear mixed models by simulation,” _Methods in Ecology and Evolution_ , vol. 7, no. 4, pp. 493–498, 2016.
  * [26] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM TASLP_ , vol. 28, pp. 2880–2894, 2020.
  * [27] J. Schmidt-Hieber, “Nonparametric regression using deep neural networks with relu activation function,” _The Annals of Statistics_ , vol. 48, no. 4, pp. 1875–1897, 2020.
  * [28] J. Bjorck, C. Gomes, B. Selman, and K. Q. Weinberger, “Understanding batch normalization,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2018, pp. 7705–7716.
  * [29] Y. Hou, B. Kang, W. Van Hauwermeiren, and D. Botteldooren, “Relation-guided acoustic scene classification aided with event embeddings,” in _Proc. of International Joint Conference on Neural Networks_ , 2022, pp. 1–8.
  * [30] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2015.
  * [31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, _et al._ , “Attention is all you need,” in _Proc. of International Conference on Neural Information Processing Systems_ , 2017, pp. 5998–6008.
  * [32] M. E. Bartlett, C. Edmunds, T. Belpaeme, and S. Thill, “Have I got the power? analysing and reporting statistical power in HRI,” _ACM Transactions on Human-Robot Interaction (THRI)_ , vol. 11, no. 2, pp. 1–16, 2022.
  * [33] I. R. Murray and J. L. Arnott, “Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion,” _The Journal of the Acoustical Society of America_ , vol. 93, no. 2, pp. 1097–1108, 1993.
  * [34] R. Ratnam, D. L. Jones, B. C. Wheeler, W. D. O’Brien Jr, C. R. Lansing, and A. S. Feng, “Blind estimation of reverberation time,” _The Journal of the Acoustical Society of America_ , vol. 114, no. 5, pp. 2877–2892, 2003.