# A Near-Real-Time Processing Ego Speech Filtering Pipeline Designed for
Speech Interruption During Human-Robot Interaction

Yue Li1, Florian A. Kunneman2, and Koen V. Hindriks1 *This work was supported
by Chinese Scholarship Council1Yue Li and Koen V. Hindriks are with Department
of Computer Science, Vrije Universiteit Amsterdam, 1081 HV Amsterdam, The
Netherlands. y6.li<EMAIL_ADDRESS>A. Kunneman is with the
Department of Languages, Literature and Communication, Utrecht University,
3584 CS Utrecht, The Netherlands<EMAIL_ADDRESS>
###### Abstract

With current state-of-the-art automatic speech recognition (ASR) systems, it
is not possible to transcribe overlapping speech audio streams separately.
Consequently, when these ASR systems are used as part of a social robot like
Pepper for interaction with a human, it is common practice to close the
robot’s microphone while it is talking itself. This prevents the human users
to interrupt the robot, which limits speech-based human-robot interaction. To
enable a more natural interaction which allows for such interruptions, we
propose an audio processing pipeline for filtering out robot’s ego speech
using only a single-channel microphone. This pipeline takes advantage of the
possibility to feed the robot ego speech signal, generated by a text-to-speech
API, as training data into a machine learning model. The proposed pipeline
combines a convolutional neural network and spectral subtraction to extract
overlapping human speech from the audio recorded by the robot-embedded
microphone. When evaluating on a held-out test set, we find that this pipeline
outperforms our previous approach to this task, as well as state-of-the-art
target speech extraction systems that were retrained on the same dataset. We
have also integrated the proposed pipeline into a lightweight robot software
development framework to make it available for broader use. As a step towards
demonstrating the feasibility of deploying our pipeline, we use this framework
to evaluate the effectiveness of the pipeline in a small lab-based feasibility
pilot using the social robot Pepper. Our results show that when participants
interrupt the robot, the pipeline can extract the participant’s speech from
one-second streaming audio buffers received by the robot-embedded single-
channel microphone, hence in near-real time.

##  I Introduction

During a human-like multi-party conversational interaction, the ability to
handle overlapping speech signals, such as backchanneling and interruption,
ensures efficient turn-taking between all parties [1]. For the same reason, in
speech-based human-robot interaction (HRI) where a user and a robot try to
have a natural conversation, the robot should listen all the time, implying
that the microphone of the robot is always on, also when the robot is speaking
[2]. In practice, however, this is difficult, as current social robots are not
equipped to extract human speech while the robot is also talking. Current
state-of-the-art automatic speech recognition (ASR) systems are unable to
transcribe an overlapping speech audio stream and will instead most likely
return a transcript of what the robot is saying itself. What is lacking is the
ability of the robot to filter out its ego speech from the audio recorded by
its microphone when it overlaps with the speech of a human user.

Therefore, it is common to adopt a simplex channel approach in HRI, which
results in a rigid and unnatural turn-taking scheme in which the microphone is
switched on and off repeatedly [4]. As an alternative to this type of
interaction scheme, the adoption of a duplex channel has been explored to
filter ego noise (the robot’s own speech), for example, by using an additional
headset that only picks up the speech from the target human user [3], or
additional microphones that may be placed either on or next to the social
robot [5, 6] to enhance the target speech signal. In this work, we propose a
pipeline that directly filters the ego speech from the signal recorded by the
robot. The pipeline aims to extract the overlapping human speech directly from
the signal received by the embedded robot microphone. Furthermore, to make
this practical, it is important to develop a pipeline that can filter robot
ego speech in real-time from the recorded audio [2].

One idea to build such a pipeline is to use an existing Target Speech
Extraction (TSE) model. These models use auxiliary information about the
target speaker and extract their speech from the recorded mixture of different
speakers and perhaps additional noise. The results reported in the literature
are very promising [7]. There are, however, a few issues that prevent the use
of these models for social robots in practice. First, current TSE models
assume that the speech signal to be extracted has approximately the same or
slightly less power than the interference speech [9]. However, this is often
not the case for social robot platforms, where the distance between the
microphone and the loudspeaker is small. Because of this, the robot’s ego
speech has typically much more power than the target speech in the received
signal [3], as illustrated in Fig.1. This severely hinders most TSE systems in
extracting the target speech. Second, current TSE models expect a complete
audio fragment to be processed. They have not been designed to incrementally
extract the target speech from an ongoing audio stream from the robot’s
embedded microphone [7, 8, 9, 10], which is essential for real-time human-
robot interaction.

Figure 1: Illustration of audio typically used for target speech extraction
versus overlapping speech recordings of the Pepper robot (in time domain).

To address these issues, we propose a new pipeline in this paper that enables
a robot to keep its microphone open and extract the interruption speech from a
human user that overlaps with the robot ego speech in near-real time. Based on
our previous work [3], the basic idea of our pipeline is to take advantage of
the fact that we know what speech signal the robot is to produce, instead of
assuming that any characteristics about the target speech or the user
interacting with the robot are known. Our aim is to address the following
three research questions:

  1. 1.

Pipeline Design: How can we design an ego speech filtering pipeline that
filters out robot ego speech and preserves the overlapping human speech from
the audio recorded by the robot’s single-channel microphone?

  2. 2.

Performance:

    1. (a)

Offline: How does the proposed pipeline perform compared to two other state-
of-the-art TSE models re-trained on the same overlapping human-robot speech
dataset?

    2. (b)

Online: How does the pipeline perform in a small lab-based feasibility pilot?
Can it be used for real-time interaction?

Compared to our previous work [3], the main contribution of this work is that
we propose a convolutional neural network (CNN) that estimates the spectrogram
of robot ego speech in the received audio, and combine it with spectral
subtraction (SS) to obtain interrupted speech from an interacting human user.
We evaluated the performance of this pipeline and compared it with our
previous work and two other retrained models in the literature on the same
dataset [3]. To assess the feasibility of using our pipeline in practice, we
integrated it into the Social Interaction Cloud (SIC) software framework
[25]111The Social Interaction Cloud (SIC) is a lightweight software framework
developed with the aim of facilitating developers and researchers to (quickly)
prototype a social robot application. SIC provides an interface for the Pepper
robot used in this paper. Among other things, it provides several components
to facilitate speech-based interaction such as Google TTS APIs, which enables
the Pepper robot to use other voices than its built-in voice., which
facilitates the easy deployment of our pipeline on a social robot and makes
our work available to the broader community. We then used this framework to
evaluate the effectiveness of our pipeline in a small lab-based feasibility
pilot, where we also looked at whether our pipeline can be used in a near-
real-time setting on the popular humanoid robot Pepper.

The remainder of this paper is organized as follows. In Section II, we discuss
related work focusing on different approaches to TSE, which we use as
inspiration for our pipeline. In Section III, we introduce the proposed
pipeline to filter out robot ego speech. Section IV discusses the experimental
setup and metrics to evaluate the pipeline, while Section V presents and
analyzes the results. Section VI concludes and discusses several limitations
of the proposed pipeline in real-life HRI, as well as future work.

##  II Related Work

Fig. 2 illustrates the main problem that a TSE system aims to address:
extracting the target speech overlapping with robot ego speech from the audio
recorded by the robot single-channel microphone. Here, we make a distinction
between two main approaches: a signal processing-based (SP-based) approach and
a deep learning-based approach.

Figure 2: The problem of robot ego speech filtering

###  II-A Signal Processing-Based TSE

From an engineering point of view, the TSE problem in the single-channel
approach is directly related to noise reduction and blind source separation
[7]. SP-based TSE methods generally calculate and reduce audio noise in a
spectrum space. The basic idea is that if noise $N_{ft}$ can be estimated,
then an estimation of the target speech spectrogram $\hat{S}_{ft}$ can be
obtained by subtracting the noise estimate from the sampled noisy spectrogram
$X_{ft}$ [13]. A wide variety of approaches have been proposed to optimize the
estimation of $\hat{S}_{ft}$, including spectral oversubtraction, multiband
spectral subtraction, Wiener filtering, and iterative spectral subtraction
[13]. These methods have been commonly designed and implemented to estimate
$N_{ft}$ during pause or silence. They have shown their effectiveness in
attenuating stationary noise [14], and some have been tested to filter robotic
fan noise or joint noise [15]. Other SP-based TSE methods, such as minimum
mean-square estimation and the factorial hidden Markov model, are also
investigated [12]. However, these methods require the estimation of the number
of speakers and can lead to global permutation ambiguity [16]222An ambiguous
permutation is a permutation which cannot be distinguished from its inverse
permutation..

###  II-B Deep Learning-Based TSE

The success in solving global permutation by deep-clustering [17] and
permutation invariant training [18] leads to the emergence of proposed neural
network-based TSE systems. Wang et al. proposed a speaker-open network
VoiceFilter [8] and its following work, VoiceFilter-Lite [19] as plug-ins
before ASR, which extracted a representation of the target speaker from a
clean enrollment sequence and then isolated the speaker’s voice in a mix.
Similar ideas have been explored by Ge et al. [20]. Zmolikova et al. [9] later
introduced SpeakerBeam, which explored three different methods of informing
the network to modify the behavior of the acoustic model. To avoid the adverse
effect on performance from different window lengths when analyzing the
reference signal and the input mixture signal, Subakan et al. [10] and Luo et
al. [21], respectively, proposed two time-domain solutions for TSE, Sepformer
and ConvTasNet. Although the performance of these proposed networks is
promising in public datasets [22], they have not been tested or evaluated in
real recordings during HRI, where human speech overlaps with robot speech in
HRI.

In our previous work [3], we proposed two ego speech filtering methods, SP-
based and neural network-based (NN-based). The SP-based method calculated the
loudspeaker-microphone response function to map the speech spectrogram
generated by the text-to-speech (TTS) API to the spectrogram of the
microphone-picked signal and estimated the target speech by subtracting it
from the spectrogram of the overlapping speech. The NN-based method consisted
of a convolutional recurrent network that directly adopted the spectrograms of
the mixture audio and generated speech audio as input. The result was not
satisfactory. In the speech spectrogram extracted by the SP-based method,
there were residues of robot speech caused by reverberation. The NN-based
method was proved to be robust to reverberation, but there were frames not
predicted. Nevertheless, the improved WER showed that the design strategy was
feasible, which mapped the spectrogram of the robot-generated speech to that
of the robot-recorded speech and subtracted it from the spectrogram of the
recorded overlapping speech. In this work, we followed the same strategy and
aimed to improve performance.

##  III Method

###  III-A Problem Formulation

The spectrum of the received single-channel audio $X_{ft}$ can be modelled by
Equation 1:

$X(f,t)=S(f,t)+N(f,t)$ (1)

where $S_{ft}$ is the short-time Fourier transformed (STFT) spectrum of the
target speech signal, that, in our case, consists of speech that is
interrupting or overlapping with robot speech, and $N_{ft}$ is the noise
signal that in our case is composed of the overlapped robot speech,
reverberation of the robot speech, and fan noise; $f$ and $t$ respectively
denote the frequency and time indices. We denote the spectrogram of the speech
signal to be played by the robot as $R_{ft}$. In this paper, our objective is
to extract the target speech $S_{ft}$ from the recorded audio $X_{ft}$ with
the generated robot speech $R_{ft}$. However, $R_{ft}$ cannot be directly
subtracted from $X_{ft}$ due to the inconsistent and non-linear response of
the speaker and microphone at different frequency levels [3]. Furthermore, the
reverberation of the room will cause the value $R_{f_{0}t_{0}}$ to appear
again at time $t_{1}$ ($t_{1}>t_{0}$) [23]. Therefore, a function $G(\cdot)$
must be learned to map $R_{ft}$ to an estimate of the recorded ego speech
spectrogram $\hat{R}_{ft}$. The estimate $\hat{S}(ft)$ of the spectrogram of
the overlapping human speech can then be obtained by Eq. 2:

$\hat{S}(f,t)=X(f,t)-G(R(f,t))$ (2)

###  III-B Proposed CNN

In this work, we propose a CNN to learn $G(\cdot)$, to estimate the
spectrogram of robot ego speech, as shown in Fig. LABEL:fig:5.

This network predicts the magnitude spectrogram $\hat{R}_{ft}$ from the
magnitude spectrogram $R_{ft}$. It is trained to minimize the difference
between the prediction and the magnitude spectrogram of the recorded robot ego
speech. The network architecture consists of three blocks: an encoder, a T-F
Learning block, and a decoder. The encoder block is a $5\times 5$
2-dimensional (2D) convolutional layer with 128 channels and $2\times 2$
padding, attached with a ReLU activation layer. The decoder block is a
transposed $5\times 5$ 2D convolutional layer with 128 channels and $2\times
2$ padding, attached to a sigmoid activation layer.

Figure 4: The proposed robot ego filtering pipeline

The T-F Learning block is designed to learn relations between the frequency
axis and the time axis, respectively. It has two Dilation blocks and one Skip
Connection block. The Skip Connection block is designed to ensure feature
reusability from the last layer [24]. It is a $1\times 1$ 2D convolutional
layer with 128 channels followed by a ReLU activation layer. Each Dilation
block is designed to learn the information between the frequency axis and the
time axis separately. Each has one $5\times 5$ 2D convolutional layer with 128
channels and an incremental dilation along the last dimension, followed by
ReLU activation layer. The input is padded to make the output size the same as
the input. After each Dilation block, the output is transposed between the
last two axes. The dilation hypermeter is set to $[2,4,8,16]$ for each block
to increase the receptive field of each kernel. The number of parameters in
the network is 1.71MB.

###  III-C Pipeline Design for Real-time Processing

In the SIC framework, once the command is sent to the robot to make it speak,
the speech content is sent to either Google [26] or its embedded TTS API.
Afterwards, the entire corresponding speech audio is generated, sent to the
local computer, and then uploaded to the robot. The robot will then play this
audio through its loudspeaker, and the microphone will remain open and
recording throughout the speech. The proposed real-time processing robot ego
speech filtering pipeline is illustrated in Fig. 4.

We create two threads in the pipeline. In Threat I, we preload the proposed
and trained CNN and perform STFT on the generated speech audio. Its magnitude
spectrogram $R_{ft}$ is sent to the network. The trained CNN then predicts the
corresponding estimated magnitude spectrogram $\hat{R}_{ft}$ and we send it to
Thread II.

In Thread II, we create an audio buffer of one second to store the recorded
audio stream for further processing.We adopt the same voice activity detection
method proposed in our previous work [3] to align the time between $X_{ft}$
and $\hat{R}_{ft}$, empty the streamed data in the buffer before speech
starts, and pad with new streaming data. We then perform STFT on this audio
buffer and get the magnitude spectrogram $X_{ft}$ and the phase spectrogram
$\phi_{ft}$. After subtracting the corresponding frames in $\hat{R}_{ft}$ from
$X_{ft}$, the estimated speech signal can be obtained by performing inverse
STFT (iSTFT) on the combination of the estimated magnitude spectrogram
$\hat{S}_{ft}$ and the unprocessed $\phi_{ft}$. To avoid the truncation effect
caused by windowing, only the middle 0.8 seconds of the estimated speech
signal are kept and divided by its maximum value [30]. We then store it or
stream it to the ASR system for further analysis. To avoid repetition of
calculations or omission of information, we empty the first 0.8 seconds of the
data in the buffer, pad it with the new streaming data, and restart the
estimation again until the end of the speech.

##  IV Experiments

We evaluate the proposed pipeline offline by means of a test dataset and
online in a small lab-based pilot.

###  IV-A Dataset Generation and Offline Evaluation

As discussed in Section I, existing public overlapping speech datasets are not
valid for this study, where target speech possess the similar power as
interfering speech. So, for offline evaluation, we adopted the same dataset
from our previous work [3], which used the public datasets Robot
Voice333https://osf.io/v4y6h/ and Librispeech444https://www.openslr.org/12
[27]. We evaluate our proposed method in two different ways. First, we used
the entire duration of the mixture as input for TSE, the same as that of a
typical TSE system. Second, in order to evaluate the proposed near-real-time
processing pipeline, we cut the mixture into 1 second long blocks with a
200-millisecond stride, to simulate the input as the streaming audio buffer.
We then performed TSE on each block. After being divided by its maximum value,
the output of each block was concatenated as the human speech extracted. The
window length for STFT and iSTFT is set at 25 ms with a 10 ms stride.

###  IV-B Network Training

Based on the distribution in room size, 200 segments recorded in the large lab
and 600 in the small office are randomly selected as the test set to evaluate
different approaches, while the other segments are selected for the training
set. We adopt the power law compressed reconstruction error [8] as the loss
function to train our proposed network and VoiceFilter, the scale-invariant
signal-to-distortion ratio [28] as the loss function to train ConvTasNet, and
monitored all training processes with Tensorboard to avoid overfitting. We set
the learning rate at 0.001 and adopt Adam as the optimizer.

###  IV-C Baseline

As evaluated in [3], the pre-trained TSE system, VoiceFilter, could not filter
out the robot’s speech from the recorded mixture. To fairly compare the
performance of the proposed network with other state-of-the-art TSE systems,
we retrain VoiceFilter, provided by Seuong-won Park555https://github.com/maum-
ai/voicefilter with 18.88MB parameters, and ConvTasNet, provided by Kaituo
Xu666https://github.com/kaituoxu/Conv-TasNet/tree/master with 4.47MB
parameters, on the same training set and evaluate them on the same test set.
To train and evaluate VoiceFilter, we use another recorded segment from Robot
Voice as auxiliary information, which shares the same voice identification as
the robot speech in the mixture. For ConVasNet, which originally predicts
multiple estimations from different speakers, we change the last layer to
output only the overlapping target human speech. We also report on the
performance of public VoiceFilter without retraining, as well as the NN-based
and SP-based methods in our previous work for comparison.

###  IV-D Small Lab-Based Feasibility Pilot

To verify the effectiveness of the proposed pipeline in the real-life HRI
scenario, we integrate it with the SIC framework and conduct a pilot study of
a laboratory-conditioned HRI experiment based on robot Pepper. The aim is to
verify whether this pipeline can filter out Pepper’s ego speech from its
recordings and how well it can interpret the human speech contents.

####  IV-D1 Participants

We recruited 11 participants (6 males, 5 females) for our experiment. They are
all randomly chosen undergraduates and graduates from the school campus who
can fluently speak English. Consent forms for permission to record audio and
video for analysis and demonstration are signed prior to the experiment.

####  IV-D2 Procedure

Before the experiment started, we informed the participants that Pepper was
designed to and help people find the correct direction to a store in a
shopping center. We asked the participants to stand or sit in front of Pepper
at a distance where they felt comfortable interacting with it and instructed
them to ask for directions to a place they want to go. We also informed the
participants that Pepper would misinterpret their intentions and give the
wrong direction to a wrong store (in this setup, we chose Adidas or Etos).
Participants were asked to use speech to stop Pepper during its speech. We
recorded the entire interaction and collected the estimated audio for further
analysis. After the interaction, we asked the participants to watch the
recorded video and help transcribe the words they used during the
interruption. We then asked them to repeat these words as closely as they said
during the interruption and recorded the clean speech using one Pepper-
embedded microphone without overlapping robot speech. The generative voice
identity was ”en-US-Neural2-G”, a female voice from the Google Text-to-Speech
API. The entire incorrect instruction took approximately 8 seconds.

###  IV-E Evaluation Metrics

To evaluate the performance in interpreting the content of overlapping human
speech between the proposed pipeline and the retrained baseline networks, we
use the word error rate (WER) as the evaluation metric for the offline
evaluation. For calculation, we adopt
Whisper777https://huggingface.co/openai/whisper-large-v3 as the ASR system to
translate all the estimated and the original human speech and use the
translation of original human speech as the ground truth. We report on the
mean, median, and standard deviation (SD) values of the WER. We also report on
the percentage of files whose WER is lower than 10% and 50%.

For the small lab-based feasibility pilot, we use the WER and the computing
time for each thread as the evaluation metric. The WER is to evaluate the
ability of the proposed pipeline to restore human speech content. The ASR
result of clean speech will be taken as the ground truth. In addition,
computing time is used to evaluate the ability of the proposed method to
process in real time.

All calculations for evaluation are performed on a local desktop with an
Intel(R) Core(TM) i9-9900K CPU and a NVIDIA GeForce RTX 2070 SUPER GPU.

##  V Results and Analysis

###  V-A Offline Evaluation

In Table I, we present the results of the proposed method compared to the
baseline models, with the original unfiltered overlapping speech data as a
reference. The bold fonts in the table represent the best result.

TABLE I: Offline evaluation results Method | Input | Retrain | WER /% | Percentage of files /%
---|---|---|---|---
Mean | Median | SD | <10% | <50%
Unfilterd | - | X | 138.01 | 120.63 | 84.91 | 0 | 0
SS-based | Blocks | X | 57.82 | 60.00 | 82.58 | 8.61 | 40.00
NN-based | Entire | X | 67.90 | 75.38 | 31.60 | 6.07 | 29.02
ConvTasNet | Entire | ✓ | 23.53 | 17.86 | 21.57 | 32.07 | 88.49
VoiceFilter | Entire | X | 131.9 | 113.8 | 82.58 | 0 | 0
VoiceFilter | Entire | ✓ | 19.53 | 15.38 | 18.54 | 37.34 | 93.09
Proposed | Entire | ✓ | 14.43 | 7.69 | 17.82 | 55.56 | 94.78
Proposed | Blocks | ✓ | 17.41 | 10.53 | 19.76 | 48.82 | 92.26

We can observe that the VoiceFilter fails to extract the target speech from
the mixture without retraining. After retraining, its performance on the test
dataset is relatively stable with an average WER value of 19.53% (SD=18.54%).
Nevertheless, only 37.34% of its predictions result in less than 10% WER. The
retrained ConvTasNet has a mean WER of 23. 53% (SD=21.57%). However, only
approximately 32% of its predictions can achieve a WER less than 10%. In
comparison, when TSE is performed on the entire mixture, our proposed method
achieves the best result on all evaluation standards. Its extracted speech has
less than 15% mean WER (SD=17.82%). In more than half of its predictions, only
10% of the words are incorrect. And in almost 95% of its predictions, less
than half of the words are incorrectly translated. Furthermore, when human
speech is extracted from the blocks, our proposed pipeline also obtains the
second-best average and median WER values. In almost $50\%$ of its
predictions, the ASR interprets less than 10% words incorrectly, and 92.26% of
the predictions have more than half of the words correctly translated.

Figure 5: Results of agglomerative clustering.

To inquire into challenging conditions for TSE, we analyze what
characteristics the processed sound fragments are most related to a poor WER
performance by applying Agglomerative Hierarchical clustering. The following
features were included in this analysis: the number of words in the target
speech, the number of words in the estimated speech, the SNR of the target
speech during overlapping, and the alpha ratio (AR) of the target speech,
which we define as Eq. 3.

$AR=\frac{\sum(\sum_{i=1001}^{fs/2}(20\times
log_{10}||S(f_{i},t_{j})||))}{\sum(\sum_{i=0}^{1000}(20\times
log_{10}||S(f_{i},t_{j})||))}$ (3)

where $fs$ is the sampling rate, which is 16k Hz. The higher the value of AR
is, the more power the speech possess in high frequency bins. We found that
the point in the dendrogram with four clusters gives the most insightful
patterns, which are presented in Fig. 5. WER is included as information, with
blocks of mixture. We also add information on the percentage of female (0) and
male (1) speakers in each cluster. The average SNR of the target speech that
overlaps with the robot speech is -22.33 db (SD=4.09). In comparison, classic
TSE systems focus on extracting the target speech from the overlapping audio,
where the SNR of the target speech is no less than -5 dB.

A comparison of spectrograms is presented in Fig. LABEL:fig:7. The WER of the
extracted speech in each selected file is as follows. File index 3612: 8.82%
for the blocks as input, 58.82% for the entire mixture as input; File index
6070: 97.73% for the blocks as input, 11.36% for the entire mixture as input.

We will offer further interpretations of Fig. 5 and LABEL:fig:7 in Section
V-C1.

###  V-B Small Lab-based Feasibility Pilot

In Table II, we present the performance of the proposed pipeline during the
laboratory-conditioned HRI. The translation mean WER of the extracted target
speech is 8.39% (SD=8.25%). Regarding the computing time in each thread, we
find that the processing time in Thread I is 980 ms, less than 1 second, while
the computing time is as low as 10 ms in Thread II.

TABLE II: Performance lab-based pilot WER | Computing Time /ms
---|---
Mean | Median | SD |  Thread I |  Thread II
8.39% | 12.50% | 8.25% | 980 | 10

Among the 11 participants, the target speech of 5 male participants was
perfectly restored and the WER of their speech is 0%, and the WER of the other
speech was also less than 20%. In each misinterpreted human speech, there are
no more than 2 incorrectly predicted words. 4 out of 8 mistranslated words are
the names of a brand, which themselves are challenges for the ASR system.

TABLE III: Detailed information about the ASR result. Num | Gender | WER /% | Incorrectly Predicted Words
---|---|---|---
Ground Truth | Predicted
1 | F | 12.5 | ”Nike” | ”90”
3 | F | 18.18 | ”Hema”,”Hema” | ”Ema”,”Ema”
6 | F | 14.29 | ”Adidas” | ”anidots”
8 | F | 18.18 | ”bookstore” | ”box now”
10 | M | 16.67 | ”was talking you” | ”wasn’t talking”
12 | F | 12.5 | ”Eh” | ””

###  V-C Analysis and Discussion

####  V-C1 Offline Evaluation

From Table I, we can find that the TSE system with a raw audio wave as input
performs worse than the systems with a spectrogram as input. This is because
the first system extracts the target speech based on wave information, which
can be hidden by robot speech in the recorded mixture, as shown in Fig. 2. We
also find that the proposed system based on the generated speech signal as
auxiliary information performs better than that based on speaker
identification. This is because the performance of the speaker identification-
based system depends on the values in the spectrogram. Nevertheless, compared
to the robot interference speech, some values in the spectrogram of the target
speech are too small during normalization and may be discarded from
prediction. It will cause some of the frames to be incorrectly predicted in
the extracted speech and further results in recognition failure.

From Fig. 5, we can observe that the SNR of the target speech in the mixture
is the decisive factor for WER. Furthermore, when there are fewer words in the
target speech, the WER will increase significantly. We also find that the WER
is not obviously related to the speaker’s gender. However, if a female speaks
fewer words and her speech does not have a high AR value, it will be poorly
translated, even at the same volume. On the contrary, the speech will be
better translated when it is long and has a higher volume and AR value. This
finding aligns with the results in [29], where the authors found that the ASR
system would translate female speech relatively worse than male speech in a
noisy environment.

When comparing the speech spectrograms extracted from the proposed method with
different inputs in Fig. LABEL:fig:7, we find that the values between 0 and
500 Hz are rarely restored. This is because the values in this range are
shared by all speech, and spectral subtraction can easily cause
oversubtraction. However, as discussed in [3], the values in high-frequency
bins are essential for speech recognition. The proposed pipeline can preserve
and restore these values, and the WER is relatively low. We also find that in
Fig.LABEL:fig:7a, the speech spectrogram extracted from the entire recording
is cleaner than that extracted from the blocks. It shows that even though we
designed this pipeline to avoid the truncation effect caused by windowing, the
pipeline that adopts blocks of mixture as input inevitably introduces noises
into the extracted speech. Nevertheless, comparing the first spectrograms in
Fig. LABEL:fig:7a and Fig. LABEL:fig:7b, we also find that if the target
speech is stronger or has a higher AR value, these noises can be concealed.
From the second and third spectrograms in Fig. LABEL:fig:7b, we can hardly see
any value in the latter. This happens because there is one value in the
spectrogram that is large enough to conceal all the other values. After
listening to the corresponding original recorded Pepper speech, we find that
there is a very sharp babble noise at the end of the speech. This explains the
failure to recognize words from the speech after extraction. This sharp babble
noise is enhanced and shrinks the extracted speech signal during
normalization. The extracted speech content becomes unrecognizable for the ASR
system. In comparison, this can be alleviated by extracting target speech from
blocks, because we normalize the extracted speech and concatenate it by
blocks.

####  V-C2 Small Lab-based Feasibility Pilot

As we can observe from Table II, the proposed pipeline successfully filtered
out Pepper’s ego speech and restored the overlapping human speech from the
overlapping mixture recorded by its microphone with a slight distortion. We
also find that the computation time is less than 1 second in Thread I. This
means that before the first audio buffer is recorded and sent to Thread II,
the $\hat{A}_{ft}$ can be predicted and ready to be subtracted from $X_{ft}$.
The processing time for the streaming audio in Thread II is 10 ms, which means
that the time difference between the interruption of the user and the
detection of the interruption by the robot is less than 810 ms. This shows the
feasibility of the proposed pipeline to extract the target speech in near-real
time during HRI. Furthermore, the low mean and SD values demonstrate the
reliability of the proposed pipeline.

After further analysis, we may find a gender effect on the performance of our
proposed pipeline. 5 out of 6 misinterpreted speech comes from females. This
is because female participants tend to speak more softly than male
participants during interruptions. This also cross-validates the analysis in
Section V-C1. In addition, we find that incorrectly recognized words have
voiceless consonants whose frequency is in the lower range of the spectrogram.
This means that only restoring the information in the higher range of the
spectrogram is sufficient but not enough for the ASR system. With the restored
information in the higher frequency bins, a speech enhancement system should
be designed to recover the values in the lower frequency bins of the
spectrogram to ensure the correct interpretation of the extracted human
speech.

Compared to our previous SP-based and NN-based methods, there are a few
advantages of this proposed pipeline:

  1. 1.

The proposed CNN is robust to reverberation and will not cause too much
oversubtraction as the adaptive step-size method based on our test result.

  2. 2.

Compared to the end-to-end NN-based method from previous work, our method
focuses more on the spectrogram estimation of the recorded robot ego speech,
instead of that of the target speech. The reason is that the values in the
higher frequency range that distinguish the target speech from the robot
speech are too small in the spectrogram of the mixture. They may be discarded
during input normalization or any normalization layers in the network. As a
result, the optimization progress may just blur the entire input to achieve a
local minimum loss value during training. Thus, we replace $X_{ft}$ with
$R_{ft}$ as the input of the network, and it shows a great result in WER.

##  VI Conclusion and Future Work

In this paper, we propose an ego speech filtering pipeline for robot Pepper to
enable its embedded single-channel microphone to keep recording during their
own speech. The proposed pipeline can filter out robot ego speech and extract
human interruption speech from the recordings. We evaluated the proposed
pipeline and compared its performance with our previous work [3] and two other
state-of-the-art retrained TSE systems on the same dataset of overlapping
robot-human speech. Our proposed pipeline exceeded all three methods. The
small lab-based feasibility pilot also verified the effectiveness of our
proposed pipeline in extracting human interruption speech during Pepper speech
in real-time HRI. We also integrated the proposed pipeline into the SIC
framework for broader accessibility to the HRI community.

Based on our analysis, there are still several limitations of the proposed
pipeline’s application in broader and more general scenarios. First, the
values in the lower range of frequency bins are oversubtracted. This will
result in incorrect translation of words that have voiceless consonants.
Second, the proposed pipeline with the entire mixture as input is vulnerable
to sharp babble noises. Third, the proposed pipeline with blocks as input
requires the speaker to be relatively loud.

In terms of future work, we will look for possible solutions in speech
enhancement to enhance the overlapping human speech extracted from blocks of
mixture. Not only should this speech enhancement method be robust to sharp
babble noise and be able to recover the values in the lower frequency range
that have been oversubtracted by this approach, but also to process blocks as
input. We also plan to do an HRI experiment on how people will prefer to
interrupt a robot when they find the answers provided are wrong.

## References

  * [1] Gervits, Felix, and Matthias Scheutz. ”Pardon the interruption: Managing turn-taking through overlap resolution in embodied artificial agents.” Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue. 2018.
  * [2] Aylett, Matthew Peter, and Marta Romeo. ”You Don’t Need to Speak, You Need to Listen: Robot Interaction and Human-Like Turn-Taking.” Proceedings of the 5th International Conference on Conversational User Interfaces. 2023.
  * [3] Yue Li, Koen V Hindriks, and Florian Kunneman. ”Single-Channel Robot Ego-Speech Filtering during Human-Robot Interaction.” International Symposium on Technological Advances in Human-Robot Interaction, USA, 2024.
  * [4] Skantze, Gabriel. ”Turn-taking in conversational systems and human-robot interaction: a review.” Computer Speech & Language 67 (2021): 101178.
  * [5] Podpora, Michal, et al. ”Human interaction smart subsystem—extending speech-based human-robot interaction systems with an implementation of external smart sensors.” Sensors 20.8 (2020): 2376.
  * [6] Nakadai, Kazuhiro, Hiroshi G. Okuno, and Hiroaki Kitano. ”Humanoid active audition system improved by the cover acoustics.” PRICAI 2000 Topics in Artificial Intelligence: 6th Pacific Rim International Conference on Artificial Intelligence Melbourne, Australia, August 28–September 1, 2000 Proceedings 6. Springer Berlin Heidelberg, 2000.
  * [7] Zmolikova, Katerina, et al. ”Neural target speech extraction: An overview.” IEEE Signal Processing Magazine 40.3 (2023): 8-29.
  * [8] Wang, Quan, et al. ”Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking.” arXiv preprint arXiv:1810.04826 (2018).
  * [9] Žmolíková, Kateřina, et al. ”Speakerbeam: Speaker aware neural network for target speaker extraction in speech mixtures.” IEEE Journal of Selected Topics in Signal Processing 13.4 (2019): 800-814.
  * [10] Subakan, Cem, et al. ”Attention is all you need in speech separation.” ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.
  * [11] Oliveira, Joao Lobato, et al. ”An active audition framework for auditory-driven HRI: Application to interactive robot dancing.” 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 2012.
  * [12] Benesty, Jacob, et al. Noise reduction in speech processing. Vol. 2. Springer Science & Business Media, 2009.
  * [13] Upadhyay, Navneet, and Abhijit Karmakar. ”Speech enhancement using spectral subtraction-type algorithms: A comparison and simulation study.” Procedia Computer Science 54 (2015): 574-584.
  * [14] Wake, Naoki, et al. ”Enhancing listening capability of humanoid robot by reduction of stationary ego‐noise.” IEEJ Transactions on Electrical and Electronic Engineering 14.12 (2019): 1815-1822.
  * [15] Ince, Gökhan, et al. ”Ego noise suppression of a robot using template subtraction.” 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2009.
  * [16] Jain, Sanjeev N., and Chandrashekhar Rai. ”Blind source separation and ICA techniques: a review.” International Journal of Engineering Science and Technology 4.4 (2012): 1490-1503.
  * [17] Hershey, John R., et al. ”Deep clustering: Discriminative embeddings for segmentation and separation.” 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2016.
  * [18] Yu, Dong, et al. ”Permutation invariant training of deep models for speaker-independent multi-talker speech separation.” 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017.
  * [19] Wang, Quan, et al. ”VoiceFilter-Lite: Streaming targeted voice separation for on-device speech recognition.” arXiv preprint arXiv:2009.04323 (2020).
  * [20] Ge, Meng, et al. ”Spex+: A complete time domain speaker extraction network.” arXiv preprint arXiv:2005.04686 (2020).
  * [21] Luo, Yi, and Nima Mesgarani. ”Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation.” IEEE/ACM transactions on audio, speech, and language processing 27.8 (2019): 1256-1266.
  * [22] Maciejewski, Matthew, et al. ”WHAMR!: Noisy and reverberant single-channel speech separation.” ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.
  * [23] Borgström, Bengt J., and Michael S. Brandstein. ”Speech enhancement via attention masking network (SEAMNET): An end-to-end system for joint suppression of noise and reverberation.” IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2020): 515-526.
  * [24] Targ, Sasha, Diogo Almeida, and Kevin Lyman. ”Resnet in resnet: Generalizing residual architectures.” arXiv preprint arXiv:1603.08029 (2016).
  * [25] Thomas Orden, Mike EU Ligthart, Koen Hindriks, Thomas Wiggers, and Karen Chiang. [n. d.]. The Social Interaction Cloud (SIC) https://socialrobotics.atlassian.net/wiki/spaces/CBSR/overview
  * [26] Google. 2023. Google Cloud Text-to-Speech AI. https://cloud.google.com/text-to-speech
  * [27] Panayotov, Vassil, et al. ”Librispeech: an asr corpus based on public domain audio books.” 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015.
  * [28] S. Li, H. Liu, Y. Zhou and Z. Luo, ”A SI-SDR Loss Function based Monaural Source Separation,” 2020 15th IEEE International Conference on Signal Processing (ICSP), Beijing, China, 2020, pp. 356-360, doi: 10.1109/ICSP48669.2020.9321080.
  * [29] Rodrigues, Ana, et al. ”Analyzing the performance of ASR systems: The effects of noise, distance to the device, age and gender.” Proceedings of the XX International Conference on Human Computer Interaction. 2019.
  * [30] Zhivomirov, Hristo. ”On the development of STFT-analysis and ISTFT-synthesis routines and their practical implementation.” TEM Journal 8.1 (2019): 56-64.


11institutetext: TH Köln - University of Applied Sciences, Germany
11email<EMAIL_ADDRESS>22institutetext: University of Duisburg-Essen, Germany
22email<EMAIL_ADDRESS>
# Context-Driven Interactive Query Simulations Based on Generative Large
Language Models

Björn Engelmann 11 Timo Breuer 11 Jana Isabelle Friese 22
Philipp Schaer 11 Norbert Fuhr 22

###### Abstract

Simulating user interactions enables a more user-oriented evaluation of
information retrieval (IR) systems. While user simulations are cost-efficient
and reproducible, many approaches often lack fidelity regarding real user
behavior. Most notably, current user models neglect the user’s context, which
is the primary driver of perceived relevance and the interactions with the
search results. To this end, this work introduces the simulation of context-
driven query reformulations. The proposed query generation methods build upon
recent Large Language Model (LLM) approaches and consider the user’s context
throughout the simulation of a search session. Compared to simple context-free
query generation approaches, these methods show better effectiveness and allow
the simulation of more efficient IR sessions. Similarly, our evaluations
consider more interaction context than current session-based measures and
reveal interesting complementary insights in addition to the established
evaluation protocols. We conclude with directions for future work and provide
an entirely open experimental setup.

###### Keywords:

User Simulation Interactive Retrieval Query Generation.

##  1 Introduction

The Cranfield paradigm is the de facto standard approach for evaluating IR
methods, allowing a fair and reproducible comparison of different retrieval
systems. However, these merits come at the cost of a strong abstraction of the
user behavior. The underlying user model of Cranfield-style experiments
assumes that the user formulates a single query, scans the result list in its
entirety until a fixed rank, and judges the relevance irrespective of earlier
seen search results. In the real world, users behave differently.

Simulating user interactions offers a more cost-efficient and reproducible
alternative to real-world user experiments. It allows us to conclude the
generalizability of experimental results beyond the boundaries of Cranfield-
style experiments. However, current endeavors lack the inclusion of the user’s
context in the simulations. To this end, our work analyzes the contextual
influence on simulated users, focusing on query generation.

To address the changing knowledge state of the user, we propose two new query
generation methods based on generative LLMs that consider different types of
context information. The methods leverage this information by incorporating it
into the reformulations, thereby enhancing the fidelity of the generated
queries to real-world user interactions and improving the quality of the user
simulation in total.

In addition, we provide a first-of-its-kind comparison of lexical (sparse) and
Transformer-based (dense) retrieval methods for simulated interactive
retrieval. While dense retrieval methods typically show higher retrieval
effectiveness, they come at a significantly higher cost in terms of time and
resources compared to sparse methods. The simulated environment in this work
allows for a direct comparison of both approaches in a user-oriented setting.

To assess the results of the simulations, we conduct an in-depth evaluation of
the simulations. For a holistic understanding of the considered factors and
their effects on retrieval effectiveness, the evaluation includes different
perspectives of retrieval effectiveness in simulated interactive retrieval
sessions. While prevalent evaluation measures report the information gain
during a search session, this work also considers the effort required to
acquire that knowledge beyond the reformulation of queries.

Besides the reported findings, we contribute additions for the TREC test
collections in the form of new query datasets. Moreover, we provide an
entirely open and reusable experimental
setup.111https://github.com/irgroup/SUIR

##  2 Related Work

Early experiments with user simulations date back to the 1980s [27, 26], but
recently, the topic got more attention from the IR community [4, 5]. Like
earlier works, we implement the simulations with a user model covering 1)
query formulations, 2) scanning of the retrieved lists, 3) selecting and
clicking appealing items, 4) reading and judging documents for relevance, and
5) inspecting other items in the result list and making stop decisions either
leading to query reformulations or abandoning the search session (cf. Figure
1). The literature provides different simulation frameworks [9, 19, 23, 31].
However, this work mainly aligns with the Complex Searcher Model implemented
by the SimIIR toolkit [18, 30].

Generating high-fidelity queries is an important aspect of interactive user
simulations [9, 7, 10, 8]. Earlier works propose different query generation
approaches that either rely on principled rules or language models. Doc2Query
is a method that produces a set of questions that a given document may answer.
The first presented use of these questions was to enrich documents during
indexing [22]. This enrichment significantly increased the retrieval
effectiveness for term-based ranking methods like BM25. [3] have shown that
this method is also suitable for varying queries in a simulation environment.
Recently, instruction-tuned LLMs were used to generate queries based on a
given prompt providing context information about the information need [1].
This approach has not been used for simulating interactive IR sessions, which
is a contribution of this work.

A commonly used measure to simulate sessions with multiple query
reformulations is the Session-based DCG (sDCG) [14], which adds an additional
discount to the discounted cumulated gain (DCG) [13] by the query position in
the session. More recently, a session-based variant of rank-biased precision
(RBP) [20] that also builds upon a query-based discount was introduced as
Session RBP (sRBP) [16]. The experiments by Lipani et al. [16] suggest that
sDCG and sRBP lead to different outcomes and let the authors conclude that
both measures provide different evaluation perspectives. For a more in-depth
review of user simulations, we refer the reader to Balog and Zhai [6].

##  3 Methodology

Figure 1: Focus of this work in the context of the Complex Searcher Model
[19].

Similar to earlier works [9, 19, 23, 31], we ground our simulations on a
defined sequence of interactions with the retrieval system and its search
results, visualized in Figure 1. In this work, the particular focus is the
query generation and the interaction with search engine result pages in
different contexts. We evaluate simulated sessions under two different
retrieval paradigms: sparse, i.e., lexical-based ranking methods, and dense,
i.e., Transformer-based methods (cf. 3.1). Regarding the interaction with
SERPs, we analyze user-specific click decisions and context-dependent stopping
strategies (cf. 3.2). Additionally, we analyze the user’s scope of the
information need and the implications of interactive feedback for query
reformulations (cf. 3.3). The simulation runs we examine are analyzed with
three different measures (cf. 3.4).

###  3.1 Retrieval Models

Table 1: Retrieval effectiveness of BM25 (sparse) and MonoT5 (dense).

| Core17 | Core18
---|---|---
Model |  P@10 | nDCG@10 |  Bpref |  P@10 | nDCG@10 |  Bpref
BM25 | 0.458 | 0.372 | 0.216 | 0.404 | 0.371 | 0.222
MonoT5@50 | 0.562 | 0.451 | 0.139 | 0.480 | 0.454 | 0.195
MonoT5@100 | 0.558 | 0.454 | 0.185 | 0.482 | 0.449 | 0.232
MonoT5@200 | 0.574 | 0.468 | 0.232 | 0.476 | 0.447 | 0.261
MonoT5@500 | 0.566 | 0.460 | 0.283 | 0.460 | 0.435 | 0.292
MonoT5@1000 | 0.550 | 0.447 | 0.312 | 0.466 | 0.439 | 0.311

The retrieval model is a key element in our simulations. In the experiments,
we evaluate the benefits of using a more effective model than a baseline
ranking. To this end, we compare BM25 [24] against MonoT5 [21] that is known
to outperform the former method in a zero-shot setting. To better understand
the quality of search results shown to the simulated users, we evaluate the
retrieval effectiveness in a preliminary ad hoc retrieval experiment.

Table 1 compares BM25 to MonoT5 at different cutoffs, i.e., the cutoff level
determines how many documents are retrieved by the first-stage ranker and will
be re-ranked by MonoT5. In principle, the re-ranker can rely on a larger set
of potentially relevant documents and place them at higher ranks. However, an
increasing cutoff level does not substantially improve P@10 and nDCG@10, which
can be explained by unjudged documents placed at higher ranks, as also shown
by the increasing Bpref scores of MonoT5.

In our simulations, we want to compare BM25 against a strong competitor that
is also computationally efficient, as the user simulations imply several other
computations as well. For this reason, we consider MonoT5@100 to be sufficient
for our experiments. A lower cutoff level substantially reduces the
computation time and also helps to reduce the environmental impact of our
research [25]. Higher cutoff levels do not lead to remarkable improvements in
precision, which is of primary interest for our simulation experiments.

From a more general perspective, these evaluations further reveal the
limitations of using ad-hoc test collections for simulated interactive
retrieval experiments. When there are many unjudged documents in the rankings,
the simulated users are indecisive about clicks, and in this case, the true
potential of Transformer-based models cannot be fully analyzed in simulations
when the test collection is biased towards a particular kind of retrieval
method. We consider it a part of the future work to provide adequate
resources, data, and tools in this regard.

###  3.2 Click and Stop Decisions

Table 2: Overview of our user simulation configurations based on query
generation methods and click models.

(a) Query generation methods

Strategy | Topic | Feedback | Generation
---|---|---|---
GPT | - | - | Probabilistic
GPT+ | + | - | Probabilistic
GPT* | + | - | Rule-based
GPT** | - | - | Rule-based
D2Q | - | - | Rule-based
D2Q+ | - | + | Rule-based
D2Q++ | + | + | Rule-based

(b) Click models

Model |  rel |  nrel
---|---|---
Perfect | 1.0 | 0.0
Navigational | 0.9 | 0.1
Informational | 0.8 | 0.4
Almost random | 0.6 | 0.4

To be in line with earlier work, we compare four different types of click
behavior in our experiments. All of the methods are based on probabilistic
decisions biased towards the level of relevance and are implemented as shown
in Table 2. The perfect and almost random browsing behaviors serve as sky- and
baselines, respectively. The perfect user always clicks on relevant results
and never clicks non-relevant or unjudged documents, i.e., the user scents the
relevant information in a strongly idealistic way. In contrast, the almost
random behavior mainly neglects the relevance labels. In between, there are
the informational and navigational models with different degrees of
exploratory behavior. We model the stop decisions in two different ways.

Static The first stopping criterion is based on the naive assumption that the
user browses a fixed number of results per page (rpp) and then continues the
session by reformulating another query about the topic.

Dynamic In addition, we implement a time-based stopping criterion, first
applied by Maxwell [17] in simulated IR sessions, that is based on the give-up
heuristic by Krebs et al. [15] stemming from the foraging theory. More
precisely, the simulated user has a fixed time budget (tnr) that starts to
deplete after the last seen relevant search result. If another relevant
document is seen, the time budget resets. If the budget is depleted, the user
reformulates the query.

###  3.3 Query Generation

In the following, we describe our query generation methods, for which an
overview is given in Table 2. Generally, we employ two different strategies:
prompting GPT to output query strings and summarizing document contents with
the help of Doc2Query (D2Q). Depending on the simulated context, we provide
the simulated user with the contents of the topic file, i.e., the description
and narrative, and let the user consider the feedback of earlier seen search
results. On the one side, the generation methods can be classified as fully
probabilistic since they rely entirely on the language models’ outputs.
Conversely, there are rule-based methods that expand the query stem based on
the topic’s title with newly generated query terms in a principled way.

####  3.3.1 Prompting query reformulations

Given the information need described by the topic file, we construct a topic-
specific prompt for the LLM. More specifically, we let the LLM generate
outputs based on the prompt Please generate one-hundred keyword queries about
<title>. <description> <narrative>, whereas <title>, <description>, and
<narrative> are taken from the respective fields of the particular topic. To
better understand the effects of different prompting strategies, we consider
the first generation method GPT that omits the additional topic fields
<description> and <narrative> in the prompt to have context-free query
variants. The second generation method GPT+ uses the entire prompt. We
implement two additional methods to better understand the benefits of having
different query variants with semantics. GPT* uses the topic’s title as the
seed query for each query (re)formulation and expands it with a single term
out of the vocabulary generated for the topic by the GPT+ strategy. Like GPT*,
GPT** is also a rule-based variant based on the context-free vocabulary of
GPT. For each topic in the test collections (cf. 3.5), we adapt the prompting
strategy to query OpenAI’s API and parse the outputs of GPT-3.5 (more
specifically, gpt-3.5-turbo-0301222https://platform.openai.com/docs/model-
index-for-researchers). In total, we generated 400 queries for all 50 topics
in two test collections, which resulted in a total of 40,000 generated queries
that stem from the outputs of OpenAI’s LLM.

####  3.3.2 Doc2Query

Doc2Query relies on a set of documents to generate queries. To obtain the
first set of documents, a seed query is required, which equals the <title> of
the corresponding topic in this case. Our Doc2Query generation approach is
based on [3] and extends the methodology by integrating background
information. Unlike the GPT variants, this method allows to generate queries
dynamically at runtime, taking seen results and, therefore, the context of the
current session into account. To factor in the user’s changing knowledge
state, terms are extracted from seen documents and added to the current
knowledge state. This approach assumes that terms that come from documents
actually seen are more likely to satisfy the information need than random
terms from the corpus. For each query reformulation, a term is taken from the
knowledge state and added to the seed query. The knowledge state at the time
of the $i$-th query $KS_{i}$ is a set of terms defined according to:

$KS_{i}=\bigcup_{j\in\\{1,\dots,i\\}}\phi(\theta(Q_{j})).$ (1)

$\theta(Q_{j})$ is the set of seen documents for the $j$-th query. The terms
extracted for a given set of documents are defined by:

$\phi(D)=\bigcup_{d\in D}\\{t\in d2q(d)\,|\,idf(t)<0.5\land t\notin S\\}.$ (2)

All terms identified by the Doc2Query function $d2q$ are joined, except for
those with an inverse document frequency (IDF) below $0.5$ and stopwords.

To determine the effect of background information and relevance feedback on
cumulative retrieval effectiveness, we present three different Doc2Query
variants. D2Q serves as the baseline of the Doc2Query variants, which
generates queries without background information and feedback. Here, the
knowledge state is based solely on terms taken from seen documents. D2Q+
integrates the relevance feedback of the simulated user. In addition to
$KS_{i}$, a knowledge state $KS_{i}^{rel}$ is used, which only contains terms
from seen documents that have been marked as relevant by the simulated user.
As long as $KS_{i}^{rel}$ is not empty, D2Q+ uses these terms to generate
queries. Otherwise, terms from $KS_{i}$ are used:

$Q_{i+1}=\left\\{\begin{array}[]{llllll}Q_{0}\cup
t^{rel}&:&t^{rel}&\in&KS_{i}^{rel}&KS_{i}^{rel}\neq\emptyset\\\ Q_{0}\cup
t&:&t&\in&KS_{i}&else.\par\end{array}\right.$ (3)

D2Q++ extends the D2Q+ mechanism with background information. For this
purpose, a set of terms is generated from the description and narrative
fields. Because these terms are topic-specific, they add another form of
context to the query generation. The topic-specific terms are used until no
more new queries can be generated, and then the D2Q+ method takes over.

###  3.4 Evaluation Measures

####  3.4.1 Effort vs. Effect

At the most granular level, we evaluate the effect of a search session by
determining the cumulated information gain (IG) over the costs of all logged
interactions, i.e., the effort, by:

$\mathrm{Effect}=\sum_{s\in\mathcal{S}}\mathrm{IG}(s),\qquad\mathrm{IG}(s)=\left\\{\begin{array}[]{ll}\mathrm{rel}_{d},&\mathrm{if}\
s=s_{\mathrm{rel}}\\\ 0,&\mathrm{else}.\end{array}\right.$ (4)

$\mathrm{rel}_{d}$ is the relevance level of the document $d$, $\mathcal{S}$
is the set of all logged interactions, and $s_{rel}$ denotes the particular
session log considering the read document as relevant. We note that in this
case, the costs are modeled by several additional factors besides the query
formulations, including the click decision based on the snippet, the reading
of the corresponding document, and making a judgment of the document’s
relevance for a given topic, and there is no discount over the progress of the
session. For the costs of each action, we use the default values of SimIIR.

####  3.4.2 Session-based DCG

As proposed in earlier work, we evaluate the simulated sessions by the sDCG
[14], which shows high correlations with real user effectiveness [11]. sDCG
discounts the cumulated gain document- and also query-wise by the logarithm
and the corresponding base $bq$ as well as by the query position $i$ in a
session:

$\mathrm{sDCG}=\sum_{i\in\\{1,\dots,n_{j}\\}}\frac{\mathrm{DCG}_{q_{i}}}{1+\log_{bq}(i)},\qquad\mathrm{DCG}_{q_{i}}=\sum_{r\in\\{1,\dots,n_{d}\\}}\frac{2^{\mathrm{rel}_{r}}-1}{\log_{2}(r+1)},$
(5)

where $n_{j}$ denotes the number of available queries for a topic $j$ and $bq$
is a free parameter set to $4$. The DCG [13] is defined by the log-harmonic
discounted sum of relevance $\mathrm{rel}_{r}$ of $n_{d}$ documents in the
ranking corresponding to query $q_{i}$. Obviously, sDCG exclusively considers
query formulations as costs and models the user’s stopping behavior by a log-
harmonic probability distribution over the number of queries and documents.

####  3.4.3 Session RBP

Similar to sDCG, there is a session-based variant of RBP [20], which was
introduced by Lipani et al. [16] and is defined as follows:

$\mathrm{sRBP}=(1-p)\sum_{i\in\\{1,\dots,n_{j}\\}}\left(\frac{p-bp}{1-bp}\right)^{i-1}\sum_{r\in\\{1,\dots,n_{d}\\}}(bp)^{r-1}\cdot\mathrm{rel}_{r},$
(6)

where $p$ models the user’s persistence similar to RBP and $b$ is introduced
as a parameter that balances reformulating a query and continuing to browse
the result list. sRBP discounts later results more than sDCG. Because of the
considerable length of the simulated search sessions, we set $p$ to $0.99$ to
ensure that results found later in the sessions would still contribute to the
outcome. For a good trade-off between results later in the queries and results
from later queries, we set $b$ to $0.9$. Like the comparison between DCG and
RBP, evaluating sDCG and sRBP provides two different perspectives of retrieval
effectiveness that can be explained by the underlying user models based on
different probability distributions.

###  3.5 Implementation Details and Datasets

We implement the experimental setup with a rich set of state-of-the-art
software libraries, including Pyterrier, ir_datasets, SIMIIR v2.0, and
HuggingFace’s transformers. We ground the experiments on two TREC newswire
test collections, i.e., the New York Times Annotated
Corpus333https://catalog.ldc.upenn.edu/LDC2008T19 used as part of TREC Common
Core 2017 (Core17) [2], and the TREC Washington Post
Corpus444https://trec.nist.gov/data/wapost/ used as part of TREC Common Core
2018 (Core18) [29]. Here it is important to note that Core18 includes
$\sim$26k ($\sim$4k positive) relevance labels, while Core19 includes
$\sim$30k ($\sim$9k positive) labels. All of the experiments are run on a Dell
workstation with an Intel i9-12900K CPU, 64GB of RAM, and an NVIDIA GeForce
RTX 3070 GPU on Ubuntu 22.04 LTS.

##  4 Experimental Results

The following experiments investigate how the information gain develops for
increasing costs. In our case, the cost is either the number of queries or the
time spent, expressed in time units. The evaluations of the simulation runs
are based on different user behaviors regarding click and stop decisions (cf.
3.2), two retrieval systems (cf. 3.1), and seven different methods for varying
queries (cf. 3.3). Three measures (cf. 3.4) are evaluated, providing different
perspectives on cost-benefit trade-offs and using two different test
collections. To determine the effect of specific methods on the information
gain, a default configuration is set for each method type and varied only over
the aspect that is being evaluated. This default configuration is: BM25 \- GPT
\- informational \- 10rpp. In our study, we evaluate the differences between
the various configurations in an exploratory manner. To make statements about
the effectiveness of the different configurations, we compare both their order
and the trend of the information gain pairwise over the simulated runs.

###  4.1 Retrieval Models and Users

This section investigates the influence of the retrieval system and different
types of simulated users. Both user characteristics, i.e., the click behavior
(cf. 2) and the stop decisions, are evaluated with BM25 and MonoT5. In Figure
2, we see a clear order if we look at the sDCG values for the different click
profiles. The closer the user is to the perfect click behavior, the higher
their information gain. Comparing all pairs of retrieval models over the click
profiles (e.g., BM25-perfect vs. MonoT5-perfect) shows a slight dominance of
MonoT5. This effect corresponds to the results of the ad hoc retrieval from
Table 1. Compared to the sDCG plots, the effort-based evaluation shows a clear
difference in user behavior, while the retrieval model plays a lesser role.

Figure 2: Results of the simulated sessions for different click behaviors.
Core18 (top) and Core17 (bottom). BM25 in solid lines, MonoT5 dashed,
respectively.

We distinguish two types of stop decisions (cf. 3.2). Either a new query is
formulated when a fixed number of snippets has been considered (rpp) or when a
defined time has passed since the last relevant document was examined (tnr).
Each type is evaluated with two parameters: $tnr\in\\{50,110\\}$ and
$rpp\in\\{10,20\\}$. In Figure 3, it can be seen that the dynamic click
behavior positively affects the information gain. In addition, both an
increase in the number of documents examined and an increase in the dynamic
time budget have a positive effect. Again, the effort-based measure weights
the impact of user behavior more heavily than the effectiveness of the
retrieval model. Furthermore, sRBP is shown to punish dynamic stop behavior.

Figure 3: Results of the simulated sessions for different stop decisions.
Core18 (top) and Core17 (bottom). BM25 in solid lines, MonoT5 dashed,
respectively.

Figure 4: Distributions for the average number of snippets examined for $i$-th
query. Core18 (left) and Core17 (right).

To put the different stop profiles in an insightful context, Figure 4 shows
the distribution over the average number of snippets considered across the
issued queries. First, it is noticeable that both runs of rpp vary much less
than the tnr variants. This effect was expected since the dynamic criterion’s
stop decision depends on the relevant documents in the result list. In
contrast, the upper limit is strictly fixed for the static criterion, and
downward outliers only occur when the result list is smaller than the fixed
value or the global time limit is reached. Furthermore, the dynamic variants
examine more snippets on average at the beginning of a session and fewer
snippets during the session than the corresponding static variants. For 110nr,
a termination can be seen at about 80 queries. This is because the global time
budget is reached here.

###  4.2 Query Variation

Since it has been shown in Section 4.1 that MonoT5 does not cause any
substantial differences in the trends of the curves, we continue with only
BM25 due to the enormous computational effort. We examine the following
aspects: query generation type, topic background, and feedback (cf. Table 2).
The plots of the seven different variants are shown in Figure 5.

Query generation type To determine what effect the query variation procedure
has, we distinguish between probabilistic and rule-based query variations as
described in Section 3.3. In our case, GPT and GPT+ stand for probabilistic
variation, while the rest works ruled-based. In Figure 5, it can be clearly
seen that the probabilistic variations have a strong positive effect on the
information gain. All measures indicate a clear dominance of GPT and GPT+.

Topic background To examine the effect of using topic background, we compare
the following strategies in pairs. GPT vs. GPT+, GPT* vs. GPT** and D2Q+ vs.
D2Q++. Across all measures, the topic background has a strong positive effect
on the probabilistic methods. Considering the rule-based methods (GPT* vs.
GPT** and D2Q++ vs. D2Q+), the topic background does not substantially affect
the information gain.

Feedback The effect of feedback on information gain is evaluated in our
setting by comparing the mechanisms D2Q+ and D2Q. As described in Section 3.3,
D2Q+ integrates the relevance feedback of the simulated user and accordingly
favors terms that come from relevant documents. Comparing the plots displays
that feedback leads to a clear increase in information gain.

Figure 5: Results of the simulated retrieval sessions for different query
generation methods. Core18 (top) and Core17 (bottom).

##  5 Discussion

The results in Section 4.1 show that although the dense retrieval model gives
slightly better results, these differences stay the same throughout the
simulation. Minor differences in trends align with other studies [12, 28], and
in our perspective, user behavior leads to a much greater variance in
effectiveness. In particular, taking into account the context within the
search session — modeled by dynamic click behavior, background knowledge, and
the integration of feedback — substantially impacts information gain. From our
point of view, it is crucial for future simulation experiments to integrate
the user context instead of only investigating cost-intensive retrieval
models.

For both test collections, the trends were similar in all evaluations, even
though the absolute information gain differed. We suspect this is related to
the substantial difference in the number of documents rated as relevant (cf.
3.5).

In future evaluations, the costs for specific user actions used for the
effort-based measure should be specified and justified. Still, the order and
trends of the effort-based evaluations resemble the sDCG evaluations closely.
Furthermore, the effort-based measure can reveal effects that would remain
hidden if only the number of queries were considered. For example, Figure 3
shows that the difference between 50tnr and 20rpp is more pronounced for the
effort-based measure since the larger number of examined documents comes with
high costs for the user that sDCG does not consider.

Regarding sRBP, sDCG also mostly supports the produced orders of the different
configurations. However, the information gain stagnates more quickly, and a
difference in the orders can be seen in Figure 3, so dynamic click behavior is
penalized by sRBP. Both findings can be explained by the substantially
stronger discounting of late results by sRBP. Interestingly, dynamic stop
behavior is notably rewarded by the other measures. As Figure 4 indicates, the
larger number of examined documents can likely further be attributed to this
opposing evaluation behavior. Determining not only realistic durations for
user actions but also estimating suitable parameters for sRBP is vital for
future work to enable more accurate evaluations and could be done by analyzing
real user behavior.

The use of probabilistic query generation methods shows promising results. Not
only do they dominate rule-based approaches, but they are also more capable of
integrating background knowledge of the topic. Integrating feedback into
probabilistic approaches would be an exciting direction for future work, as
our rule-based approach has clearly shown that feedback is beneficial.

## References

  * [1] Alaofi, M., Gallagher, L., Sanderson, M., Scholer, F., Thomas, P.: Can generative llms create query variants for test collections? an exploratory study. In: SIGIR. pp. 1869–1873. ACM (2023)
  * [2] Allan, J., Harman, D., Kanoulas, E., Li, D., Gysel, C.V., Voorhees, E.M.: TREC 2017 common core track overview. In: TREC. NIST Special Publication, vol. 500–324. National Institute of Standards and Technology (NIST) (2017)
  * [3] Anonymous authors: Anonymized. In: Accepted for publication at CIKM’23 (2023)
  * [4] Azzopardi, L., Järvelin, K., Kamps, J., Smucker, M.D.: Report on the SIGIR 2010 workshop on the simulation of interaction. SIGIR Forum 44(2), 35–47 (2010)
  * [5] Balog, K., Maxwell, D., Thomas, P., Zhang, S.: Sim4ir: The SIGIR 2021 workshop on simulation for information retrieval evaluation. In: SIGIR. pp. 2697–2698. ACM (2021)
  * [6] Balog, K., Zhai, C.: User simulation for evaluating information access systems. CoRR abs/2306.08550 (2023)
  * [7] Baskaya, F., Keskustalo, H., Järvelin, K.: Modeling behavioral factors in interactive information retrieval. In: He, Q., Iyengar, A., Nejdl, W., Pei, J., Rastogi, R. (eds.) 22nd ACM International Conference on Information and Knowledge Management, CIKM’13, San Francisco, CA, USA, October 27 - November 1, 2013. pp. 2297–2302. ACM (2013). https://doi.org/10.1145/2505515.2505660, https://doi.org/10.1145/2505515.2505660
  * [8] Breuer, T., Fuhr, N., Schaer, P.: Validating simulations of user query variants. In: ECIR (1). Lecture Notes in Computer Science, vol. 13185, pp. 80–94. Springer (2022)
  * [9] Carterette, B., Bah, A., Zengin, M.: Dynamic test collections for retrieval evaluation. In: Allan, J., Croft, W.B., de Vries, A.P., Zhai, C. (eds.) Proceedings of the 2015 International Conference on the Theory of Information Retrieval, ICTIR 2015, Northampton, Massachusetts, USA, September 27-30, 2015. pp. 91–100. ACM (2015). https://doi.org/10.1145/2808194.2809470, https://doi.org/10.1145/2808194.2809470
  * [10] Günther, S., Hagen, M.: Assessing query suggestions for search session simulation. Sim4IR: The SIGIR 2021 Workshop on Simulation for Information Retrieval Evaluation (2021), http://ceur-ws.org/Vol-2911/paper6.pdf
  * [11] Hagen, M., Michel, M., Stein, B.: Simulating ideal and average users. In: AIRS. Lecture Notes in Computer Science, vol. 9994, pp. 138–154. Springer (2016)
  * [12] Hersh, W.R., Turpin, A., Price, S., Chan, B., Kraemer, D., Sacherek, L., Olson, D.: Do batch and user evaluation give the same results? In: Yannakoudakis, E.J., Belkin, N.J., Ingwersen, P., Leong, M.K. (eds.) SIGIR 2000: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 24-28, 2000, Athens, Greece. pp. 17–24. ACM (2000). https://doi.org/10.1145/345508.345539, https://doi.org/10.1145/345508.345539
  * [13] Järvelin, K., Kekäläinen, J.: IR evaluation methods for retrieving highly relevant documents. In: SIGIR. pp. 41–48. ACM (2000)
  * [14] Järvelin, K., Price, S.L., Delcambre, L.M.L., Nielsen, M.L.: Discounted cumulated gain based evaluation of multiple-query IR sessions. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) Advances in Information Retrieval , 30th European Conference on IR Research, ECIR 2008, Glasgow, UK, March 30-April 3, 2008. Proceedings. Lecture Notes in Computer Science, vol. 4956, pp. 4–15. Springer (2008). https://doi.org/10.1007/978-3-540-78646-7_4, https://doi.org/10.1007/978-3-540-78646-7_4
  * [15] Krebs, J.R., Ryan, J.C., Charnov, E.L.: Hunting by expectation or optimal foraging? a study of patch use by chickadees. Animal Behaviour 22, 953–IN3 (Nov 1974). https://doi.org/10.1016/0003-3472(74)90018-9, https://doi.org/10.1016/0003-3472(74)90018-9
  * [16] Lipani, A., Carterette, B., Yilmaz, E.: From a user model for query sessions to session rank biased precision (sRBP). In: Fang, Y., Zhang, Y., Allan, J., Balog, K., Carterette, B., Guo, J. (eds.) Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR 2019, Santa Clara, CA, USA, October 2-5, 2019. pp. 109–116. ACM (2019). https://doi.org/10.1145/3341981.3344216, https://doi.org/10.1145/3341981.3344216
  * [17] Maxwell, D.: Modelling search and stopping in interactive information retrieval. Ph.D. thesis, University of Glasgow, UK (2019)
  * [18] Maxwell, D., Azzopardi, L.: Simulating interactive information retrieval: SimIIR: A framework for the simulation of interaction. In: SIGIR. pp. 1141–1144. ACM (2016)
  * [19] Maxwell, D., Azzopardi, L., Järvelin, K., Keskustalo, H.: Searching and stopping: An analysis of stopping rules and strategies. In: CIKM. pp. 313–322. ACM (2015)
  * [20] Moffat, A., Zobel, J.: Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst. 27(1), 2:1–2:27 (2008)
  * [21] Nogueira, R.F., Jiang, Z., Pradeep, R., Lin, J.: Document ranking with a pretrained sequence-to-sequence model. In: EMNLP (Findings). Findings of ACL, vol. EMNLP 2020, pp. 708–718. Association for Computational Linguistics (2020)
  * [22] Nogueira, R.F., Yang, W., Lin, J., Cho, K.: Document expansion by query prediction. CoRR abs/1904.08375 (2019)
  * [23] Pääkkönen, T., Kekäläinen, J., Keskustalo, H., Azzopardi, L., Maxwell, D., Järvelin, K.: Validating simulated interaction for retrieval evaluation. Inf. Retr. J. 20(4), 338–362 (2017). https://doi.org/10.1007/s10791-017-9301-2, https://doi.org/10.1007/s10791-017-9301-2
  * [24] Robertson, S.E., Zaragoza, H.: The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4), 333–389 (2009)
  * [25] Scells, H., Zhuang, S., Zuccon, G.: Reduce, reuse, recycle: Green information retrieval research. In: SIGIR. pp. 2825–2837. ACM (2022)
  * [26] Tague, J., Nelson, M.J.: Simulation of user judgments in bibliographic retrieval systems. In: Crouch, C.J. (ed.) Theoretical Issues in Information Retrieval, Proceedings of the Fourth International Conference on Information Storage and Retrieval, Oakland, California, USA, May 31 - June 2, 1981. pp. 66–71. ACM (1981). https://doi.org/10.1145/511754.511764, https://doi.org/10.1145/511754.511764
  * [27] Tague, J., Nelson, M.J., Wu, H.: Problems in the simulation of bibliographic retrieval systems. In: Oddy, R.N., Robertson, S.E., van Rijsbergen, C.J., Williams, P.W. (eds.) Information Retrieval Research, Proc. Joint ACM/BCS Symposium in Information Storage and Retrieval, Cambridge, UK, June 1980. pp. 236–255. Butterworths (1980), http://dl.acm.org/citation.cfm?id=636684
  * [28] Turpin, A., Hersh, W.R.: Why batch and user evaluations do not give the same results. In: Croft, W.B., Harper, D.J., Kraft, D.H., Zobel, J. (eds.) SIGIR 2001: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, September 9-13, 2001, New Orleans, Louisiana, USA. pp. 225–231. ACM (2001). https://doi.org/10.1145/383952.383992, https://doi.org/10.1145/383952.383992
  * [29] Voorhees, E.M., Ellis, A. (eds.): Proceedings of the Twenty-Seventh Text REtrieval Conference, TREC 2018, Gaithersburg, Maryland, USA, November 14-16, 2018, NIST Special Publication, vol. 500–331. National Institute of Standards and Technology (NIST) (2018), https://trec.nist.gov/pubs/trec27/trec2018.html
  * [30] Zerhoudi, S., Günther, S., Plassmeier, K., Borst, T., Seifert, C., Hagen, M., Granitzer, M.: The SimIIR 2.0 framework: User types, markov model-based interaction simulation, and advanced query generation. In: CIKM. pp. 4661–4666. ACM (2022)
  * [31] Zhang, Y., Liu, X., Zhai, C.: Information retrieval evaluation as search simulation: A general formal framework for IR evaluation. In: ICTIR. pp. 193–200. ACM (2017)

  *[IR]: information retrieval
  *[LLM]: Large Language Model
  *[LLMs]: Large Language Model
  *[sDCG]: Session-based DCG
  *[DCG]: discounted cumulated gain
  *[RBP]: rank-biased precision
  *[sRBP]: Session RBP
  *[SERPs]: search engine result page
  *[IG]: information gain