# Speech Aware Dialog System Technology Challenge (DSTC11)

Hagen Soltau, Izhak Shafran, Mingqiu Wang, Abhinav Rastogi, Jeffrey Zhao, Ye
Jia
Wei Han, Yuan Cao, Aramys Miranda
Google Research
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Most research on task oriented dialog modeling is based on written text input.
However, users interact with practical dialog systems often using speech as
input. Typically, systems convert speech into text using an Automatic Speech
Recognition (ASR) system, introducing errors. Furthermore, these systems do
not address the differences in written and spoken language. The research on
this topic is stymied by the lack of a public corpus. Motivated by these
considerations, our goal in hosting the speech-aware dialog state tracking
challenge was to create a public corpus or task which can be used to
investigate the performance gap between the written and spoken forms of input,
develop models that could alleviate this gap, and establish whether Text-to-
Speech-based (TTS) systems is a reasonable surrogate to the more-labor
intensive human data collection. We created three spoken versions of the
popular written-domain MultiWoz task – (a) TTS-Verbatim: written user inputs
were converted into speech waveforms using a TTS system, (b) Human-Verbatim:
humans spoke the user inputs verbatim, and (c) Human-paraphrased: humans
paraphrased the user inputs. Additionally, we provided different forms of ASR
output to encourage wider participation from teams that may not have access to
state-of-the-art ASR systems. These included ASR transcripts, word time
stamps, and latent representations of the audio (audio encoder outputs). In
this paper, we describe the corpus, report results from participating teams,
provide preliminary analyses of their results, and summarize the current
state-of-the-art in this domain.

Speech Aware Dialog System Technology Challenge (DSTC11)

Hagen Soltau, Izhak Shafran, Mingqiu Wang, Abhinav Rastogi, Jeffrey Zhao, Ye
Jia Wei Han, Yuan Cao, Aramys Miranda Google Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

In recent years, Automatic Speech Recognition (ASR) and Natural Language
Processing (NLP) models have converged to utilize common components like
Transformers and encoder/decoder modules. They increasingly rely on large
amounts of data, large model sizes and large amounts of compute resources.
This is a substantial departure from a previous era when ASR and NLP utilized
different modeling architectures, chosen to inject domain-specific knowledge
and constraints. This shift to a common paradigm has stimulated research in
fusing audio and text modalities to substantially improve performance in tasks
such as audio intent classification and speech-to-speech translation. The
fusion of modalities in many cases allows the direct optimization of the end-
task, overcoming the hurdles of the older cascaded approaches that often led
to accumulation of errors.

Despite the general trend to develop end-to-end models for various tasks,
spoken dialog systems stick out as a sore thumb. Most practical systems
utilize a cascaded approach where the output of a general ASR system is fed
into a dialog model trained separately on written domain. This mismatch
between written and spoken inputs to the dialog models is not well-studied,
largely due to the lack of a public task with spoken user inputs.

Research into combined audio-text models is limited by the lack of paired
data. While the paired data requirement can be relaxed to some extent for
training data via un- or self-supervised training techniques, test sets with
paired data are crucial for model evaluation. In addition to an evaluation
task, a training set with spoken input would also be helpful in quantifying
the gains from supervised learning and recent advances in self-supervised
learning.

The focus of our effort was to bring most benefit to the community with the
limited resources available. While a Wizard-of-Oz style data collection in
spoken domain would have been ideal to fully investigate all the phenomena of
spoken domain, that would be extremely labor-intensive especially in
annotating the dialog states and was beyond the scope of our effort. Instead,
we chose to create a spoken version of a well-studied written-domain task, the
MultiWoz task. One advantage of this approach was that we could directly
compare the performance of the spoken version with the continuing advances in
the written domain. Knowing that the current Text-to-Speech (TTS) systems have
come a long way, we also chose to create a TTS-version of the MultiWoz
training corpus as a surrogate for the human spoken version.

In the Speech-Aware DSTC11 challenge, participants are asked to infer the
dialog states from the sequence of agent (text-input) and user (audio-input)
turns. We evaluated the performance on three versions of audio inputs – TTS-
Verbatim, Human-Verbatim (humans speaking the written user inputs), and Human-
Paraphrased (humans paraphrasing the written user inputs). Aside from the
audio-inputs, we provided transcripts from a state-of-the-art ASR system
trained on 33k hours of People’s Speech corpus to encourage participation from
teams that did not have an easy access to ASR systems. The transcripts were
accompanied with timestamps of the words and the latent representations of the
acoustic encoder, which could be used by participants to train joint audio-
text encoders.

In the course of developing this challenge, we developed a cascaded baseline
system with data augmentation and report performances on a few variants of
cascaded systems. In the process, we uncovered a bias in the MultiWoz
evaluation task, the slot values in the evaluation set have a substantial
overlap with those of the training set. To address this bias, we created a new
version of the MultiWoz evaluation set, the dev-dsct11 and eval-dstc11. We
observed that the new task surfaces many of the challenges in practical spoken
dialog systems associated with mismatch in modalities, inability to recover
from ASR errors, and more generally difficulty of extracting semantically
relevant information from audio signals.

##  2 Related Work

Motivated by similar consideration as ours, Kim et al. (2022) organized a DST
Challenge in 2021 where they created a task using spoken human-human dialogs
on tourist information in San Francisco for three target domains: hotel,
restaurant, and attraction. One of the serious limitations of the challenge
was that the audio data was not released, only the ASR transcripts. The
transcripts had an error rate of about 26.25% which is significantly higher
than the average performance of most state-of-the-art ASR systems. The larger
focus of their effort was on evaluating correctness of detecting knowledge-
seeking turns, identifying the knowledge snippets and knowledge supplied in
the generated responses. Prior to this effort, there have been much smaller
efforts with fewer domains and dialogs in DSTC2 and DSTC3 Henderson et al.
(2014b, a). Here again the organizers only provided ASR transcripts with error
rates in high 20s and low 30s, which limited the utility of the corpus as the
ASR systems improved over time.

Meanwhile considerable progress has been achieved in improving the naturalness
of dialog systems, for example, with chat-bots like Meena Adiwardana et al.
(2020), raising expectations of interacting with dialog systems using spoken
language. Similarly, the convergence of model architectures for ASR and NLP
has stimulated research in creating joint audio-text encoders that could
potentially compensate for ASR errors and other spoken language phenomena
Drexler and Glass (2019); Chen et al. (2021); Chung et al. (2020); Tang et al.
(2022); Bapna et al. (2021, 2022); Chen et al. (2022), where they propose
different approaches to align the speech input (frames, phones, utterances) to
the corresponding text units. However, none of them have been evaluated on
dialog models due to the lack of a dialog task and corpus with audio input. We
hope the task and corpus released in this work will bridge this gap along with
other recent speech understanding tasks such as superb and slurp Yang et al.
(2021); Bastianelli et al. (2020)

##  3 Data

We chose to create a spoken version of MultiWoz (2.1 version) so we could
directly compare the written and spoken versions and reuse the annotations of
dialog state labels, avoiding the labor-intensive process of annotating
reference labels.

###  3.1 Redesigned DSTC11 Evaluation Sets

Before launching into the data collection and in the process of developing
baseline systems, we noticed that there is a substantial overlap in slot
values between the training and evaluation sets, leading to overestimation of
performance of the models that memorize the slot values, as reported elsewhere
Song et al. (2020). To illustrate the issue, we probed an existing DST model
with input whose slot values were replaced with other viable slot values
(e.g., cambridge to new york, ely to dublin), 17:43 to 17:41 and 19:00 to
19:12). We chose a model that achieves close to the state-of-the-art
performance on the task with JGA of 55.4% Zhao et al. (2021). The model
ignores the new slot values and regurgitates the original slot values
memorized from the training data, as shown in the Table 1.

Text Input | Text Output
---|---
i want to go to new york | dest=cambridge
i want to go to dublin | dest=ely
train leaving at 17:41 | time=17:43
train leaving at 19:12 | time=19:00
Table 1: Illustration of a model memorizing slot values: the model
regurgitates memorized values and ignores the replaced slot values in the
input.

This overlap in slot values between the training and evaluation sets of
MultiWoz will mask the effect of misrecognition of the slot values by ASR
systems in a practical spoken dialog system. For a fair evaluation of the
models, we redesigned the MultiWoz evaluation sets to replace the slot values
in the evaluation sets with new slot values as described below. The
replacements were performed at the dialog level to maintain consistency across
turns.

  1. 1.

Location Names: The destination and departure cities for trains and buses were
replaced with randomly sampled city names from 12655 cities in the United
States.

  2. 2.

Hotel Names: Hotel names were replaced with random names sampled from 1562
hotels in the United States.

  3. 3.

Restaurant Names: Restaurant names were replaced by sampling from 214
restaurants in New York City.

  4. 4.

Time Slots: Timestamps were offset by a random value across all the times
mentioned in the dialog.

We measured the impact of the redesigned evaluation sets, henceforth referred
to as dev-dstc11 and test-dstc11, using two dialog models – a seq-to-seq model
Zhao et al. (2021) and a model that utilizes the power of large language model
by fine-tuning the prefix encoding with dialog-specific instructions Zhao et
al. (2022). The second model has three variants – D3ST-base, D3ST-large and
D3ST-XXL – related to the size of the underlying T5x large language model. The
results, shown in the Table 2, confirms our overestimation of the original
MultiWoz evaluation sets. The actual performance is almost 50% worse than
reported on the original evaluation sets for most models with the exception of
D3ST-XXL.

Model | org. Dev | Dev-DSTC11
---|---|---
seq2seq | 55.4 | 20.8
D3ST-base | 54.2 | 22.0
D3ST-large | 54.5 | 25.2
D3ST-xxl | 57.8 | 43.1
Table 2: Performance of dialog models, measured in JGA, on dev-dstc11, the
redesigned MultiWoz evaluation sets. The comparison with original dev set
illustrates the substantial overestimation due to the overlap in training and
dev slot values.

The memorization of the slot values completely distorts the effect of ASR
errors on the downstream dialog models and paints an overly rosy picture. We
illustrate this with a simple cascade baseline where an ASR system transcribes
the input speech into written form which is fed into a seq-to-seq dialog model
Zhao et al. (2021). In the Table 3 we compare the performance degradation from
switching from written to spoken input and then retraining the seq-to-seq
model on the ASR transcripts. According to the performance on the original dev
set, the degradation from written to ASR transcripts is 58.1% to 47.2%.
Furthermore, most of the degradation is recovered by retraining on the ASR
transcripts. Both these results are incorrect and misleading as will show in
the Section 5.4, training on ASR output is not nearly as effective as the
results in the table suggests.

Train Data | Test Data | JGA
---|---|---
Written | Written | 58.1
Written | Spoken/ASR | 47.2
Spoken/ASR | Spoken/ASR | 56.0
Table 3: Results of training and evaluating a seq-to-seq model on written and
ASR transcripts. The results are misleading as we will show in the Section
5.4.

###  3.2 Text-to-Speech Version

One of the questions we were interested in understanding was whether TTS is a
good substitute for speech collected from humans. For answering this question,
we generated TTS-Verbatim, a TTS version of the evaluation test and training
sets using the system described in Jia et al. (2021), a system that represents
phonemes and graphemes to represent input text. Additionally, for training
data, we generated four versions of the training data, each corresponding to a
different TTS speaker. There was no overlap between the speakers in the
training and evaluation sets.

###  3.3 Human Data Collection

We focused our data collection on the user turns since the agent turns are
already available to any practical dialog system. The data collection was
performed via Amazon Mechanical Turk and consisted of two versions – Human-
Verbatim and Human-Paraphrased. Crow-workers were presented a full dialog in
text including both user and agent turns. In the verbatim version, the workers
were instructed to utter the user turns verbatim as naturally as possible, one
at a time till the end of the dialog. In the paraphrased version, the workers
were instructed to utter a paraphrased version of the user turn preserving the
semantic meaning and the entities (e.g., names, times). After they finished
the dialog, they were asked to transcribe their own paraphrased user turns.

The data was collected in several iterations. After each batch was collected,
the quality of the recordings were measured and only those above certain
quality were retained. There were several factors that contributed to
failures: 1) missing audio, 2) incomplete dialog, 3) unintelligible speech, 4)
high background noise, 5) speakers uttering verbatim when they should have
paraphrased, and 6) speakers not transcribing their utterance in the
paraphrased version.

We developed objective measures to perform quality control in bulk. The
missing audio and incomplete dialogs were detected programmatically. To filter
out utterances with unintelligible speech and high background noise, we
transcribed the collected user utterances with an ASR system (for details of
the system see Section 5) and measured the accuracy with respect to the
corresponding transcripts. Since insertions may occur due to disfluencies, we
used deletions as a stronger quality indicator and used a threshold of 25%.

Measuring the quality of paraphrased version was challenging. We iterated with
objective metrics and listen to sampled utterances. After a few iterations, we
converged on the following criteria: 1) paraphrased versions had at least 60%
as many words as verbatim, 2) the paraphrased version differed from the
verbatim version by at least 45% as indicated by the WER between recognized
words and the written user prompt, and 3) less than 30% WER with respect to
the crowd-worker generated transcripts. Minimum word ratio is to ensure the
paraphrased speech carries enough information as in the original written user
prompt. The second criterion was to ensure that the paraphrased version was
sufficiently different from the written version. The last criterion was to
ensure audio quality and intelligibility.

Besides the automatic quality control, we also sampled and listened to a small
portion of the collected recordings. We noticed that even though the crowd-
workers were asked to speak naturally, the verbatim was stilted more towards a
read version. In a natural dialog, the speakers are likely to show more
emotional variations and disfluencies. In comparison, the paraphrased version
was more natural since they were using their own words.

In most natural conversations, speakers use 12 hour format for time (e.g.,
1315 hrs vs 1:15pm). Since we wanted the crowd-workers to speak as naturally
as possible, we replaced all the 24 hour time format in the original MultiWoz
set with their corresponding 12 hour format.

##  4 Challenge Task and Evaluation

The main task in the DSTC11 Challenge is to infer the dialog states correctly
from the audio inputs that were provided corresponding to the redesigned
MultiWoz test set, redesigned as described in Section 3.1. For ablation
studies, the participants were required to submit their results for three
conditions: TTS-Verbatim, Human-Verbatim and Human-paraphrased, which were
collected as described in Section 3.2 and 3.3. The participants were free to
use their own ASR system or the outputs provided from our baseline ASR system,
described in Section 5.1.

The performance in the challenge is measured using the standard Joint Goal
Accuracy (JGA) as a primary metric and Slot Error Rate (SER) as a secondary
metric for fine-grained comparisons. In the literature, results are often
difficult to compare since research groups apply their own output
normalization before scoring the results. To sidestep the resulting confusion,
this challenge is evaluated using standard evaluation tool published on GitHub
Zang et al. (2020). We created a patch to support the results from this
challenge and published the golden reference for our redesigned evaluation
sets, which can be downloaded from the DSTC11 challenge website
https://dstc11.dstc.community.

The participants were allowed to utilize any publicly available data or
checkpoints, but no private data to allow fair comparisons. They were allowed
to augment data and no restrictions were imposed on model sizes or
computational costs.

##  5 Baseline

###  5.1 ASR and Related Outputs

As mentioned earlier, we provided the output of a baseline ASR system to
reduce the barrier for participation by teams which didn’t have ASR systems
available to them.

The baseline ASR model is based an RNN-Transducer Graves (2012) which is
essentially an encoder-decoder model, as illustrated in Figure 1. Unlike
typical encoder-decoder models, the RNN-T uses hard attention and learns
alignments via Forward-Backward algorithm similar to HMM, albeit with a
different topology. These models have been extremely successful in ASR and
achieves state-of-the-art performance in several benchmarks.

Figure 1: RNN-Transducer

Our baseline models is trained on the PeopleSpeech corpus Galvez et al.
(2021), a supervised, publicly available data set of approx. 32,000 hours of
audio data. The encoder consists of 16 Transformer layers, and the language
model (decoder) is a single LSTM layer, altogether a 220m parameter model. The
performance of the model on the MultiWoz evaluation sets are reported in Table
4.

| TTS-Verbatim | Human-Verbatim
---|---|---
dev-dstc11 | 8.1(5.7/0.3/2.1) | 11.9(7.7/2.7/1.5)
test-dstc11 | 8.2(5.7/0.3/2.1) | 13.0(8.2/3.4/1.5)
Table 4: Performance of the baseline ASR model on the MultiWoz evaluation
sets, reported as WER(Sub/Del/Ins)

We provided the following outputs as part of the DSTC11 challenge corpus.

  1. 1.

Audio Waveforms (16Khz/16-bit PCM): This allows researchers to investigate DST
models that directly operate on audio such as end-to-end systems.

  2. 2.

Audio Encodings (75ms frame rate, 1024-dim): These are the output activation
from the last Transformer layer in the audio encoder. Since the input to the
model are log-mel features at 25ms frame rate, and the encoder reduces the
frame rate by 3x, the resulting output frame rate is 75ms, i.e. for an
utterance of 15 sec, the audio encodings would be of length 200. These
encodings are provided to support research in loosely-coupled ASR-DST cascaded
systems.

  3. 3.

ASR Hypotheses with Time Alignment: The ASR recognition outputs are provided
to support research in different cascaded ASR-DST systems. Time alignments
allow teams to pinpoint the location of recognized tokens in the audio
encodings. Note, due the nature of RNN-T, multiple output tokens (words) can
correspond to the same audio frame.

Typical recognition errors (ordered by frequency), illustrated in Table 5,
include time formatting issues, spoken single-digit numbers, split words, and
more general misspelled content words, where the last category is probably
very relevant for DST purposes.

Spoken vs Written Numbers
---
3 stars | three stars
031 | thirty-on
Split Words
i am | i’m
wifi | wi-fi
guesthouse | guest house
goodbye | good bye
double-checking | double checking
Misspelled content words
hennessey | hennessy
moxee | marcy
kinbrae | kinbray
Table 5: Typical ASR Recognition Errors

###  5.2 Dialog State Tracking Models

For our baseline DST experiments, we relied on two models.

  1. 1.

Vanilla seq-to-seq model: This seq-to-seq model is obtained by fine-tuning a
T5-base model on MultiWoz corpus Zhao et al. (2021).

  2. 2.

D3ST: This is also a seq-to-seq model trained on T5, but the input to this
model contains a prompt which describes slot names in short natural language
descriptions along with potential values Zhao et al. (2022). The ordering of
the slot names are randomizes to improve robustness.

###  5.3 Data Augmentation

Data augmentation is a common technique to improve accuracy and robustness.
Since our preliminary results in Table 2 unearthed problems related to
memorization, we felt the need to incorporate data augmentation into our
baseline systems. We created new versions of user responses for training data
by replacing slot values with randomly picked city names, time offsets and
restaurant names, as described in Section 3.1. The new slot names were drawn
from a different list than the ones used for generating the redesigned
evaluation sets. In all, we generated about 100x training data.

###  5.4 Cascaded ASR-DST System

For a baseline, we created a cascades ASR-DST system where the transcripts
from the ASR model in Section 5.1 were fed as input to the DST model in
Section 5.2. The performance of the models of various sizes are summarized in
Tables tables 6, 7, 8 and 9. The best baseline performance is obtained with
D3ST-xxl, obtaining 33.5% JGA on the human paraphrased test set.

Test Data | Training Data
---|---
| Text (Ref) | ASR Hyp
| 1x | 100x | 1x | 100x
Text | 20.1 | 41.7 | - | -
H-Paraphrased | 16.5 | 24.1 | 18.3 | 25.6
Table 6: ASR-Vanilla Seq-to-Seq DST base cascaded model (JGA) Test Data | Training Data
---|---
| Text (Ref) | ASR Hyp
| 1x | 100x | 1x | 100x
Text | 22.0 | 41.6 | 20.9 | 28.6
TTS-Verbatim | 19.2 | 27.7 | 19.9 | 27.1
H-Verbatim | 17.1 | 23.6 | 17.6 | 22.5
H-Paraphrased | 17.0 | 22.8 | 17.3 | 21.8
Table 7: ASR-D3ST base cascaded model (JGA) Test Data | Training Data
---|---
| Text (Ref) | ASR Hyp
| 1x | 100x | 1x | 100x
Text | 25.2 | 44.2 | 22.3 | 32.6
TTS-Verbatim | 20.5 | 29.1 | 20.7 | 29.5
H-Verbatim | 18.6 | 25.1 | 18.5 | 24.8
H-Paraphrased | 18.0 | 23.9 | 17.5 | 23.6
Table 8: ASR-D3ST large cascaded model (JGA) Test Data | Training Data
---|---
| Text (Ref) | ASR Hyp
| 1x | 100x | 1x | 100x
Text | 43.1 | 52.8 | 33.9 | 43.0
TTS-Verbatim | 27.3 | 32.1 | 26.8 | 38.4
H-Verbatim | 23.6 | 27.9 | 23.7 | 31.8
H-Paraphrased | 21.8 | 26.1 | 21.9 | 30.9
Table 9: ASR-D3ST xxl cascaded model (JGA)

We point a few pertinent observations on cascaded ASR-DST models and data
augmentation.

  1. 1.

TTS (TTS-Verbatim) shows a degradation with respect to written version.
However, the degradation is larger with Human-Verbatim (41.0% vs. 33.5% JGA),
confirming our suspicion that we cannot rely on TTS as a surrogate for human
speech.

  2. 2.

Surprisingly, the drop in performance from Human-Verbatim to Human-Paraphrased
is not very large. This is not because the two versions are similar since
quality checks described in Section 3.3 assures us that the paraphrased
version is sufficiently different from verbatim version.

  3. 3.

For models of smaller sizes, the differences in model type (Vanilla seq-to-seq
vs D3ST) seems less relevant. However, scaling the D3ST model from D3ST-Large
(3B params) to D3ST-XXL (11B params) has substantial impact on performance
(24.8% vs 33.5% JGA). This raises a question of fairness since teams may not
have resources to work with larger model sizes. Nonetheless, we did find
examples in results provided by participants were smaller models were able to
outperform larger models, see Section 6 for details.

  4. 4.

Training DST models with ASR hypotheses is simple and improves the performance
substantially from 28.2% to 33.5% even when the training data is based on TTS
and not human speech.

  5. 5.

Data augmentation gives consistent gains across all conditions.

##  6 Results

We received $11$ submissions from $6$ teams and the results are shown in
figures 2 and 3. The full set of numbers are also available in the link under
the DSTC11 Challenge website https://dstc11.dstc.community.

We provide a high level overview of the results, but refer to the team’s
system descriptions for details. While we provided data to build direct audio-
to-dst and tightly coupled models, all teams chose a cascaded approach with
separately trained ASR and DST models. Many teams employed an explicit ASR
error correction model and re-trained their DST models on ASR hypotheses
together with various forms of TTS-based data augmentation. Within this
general approach teams experimented with several variations and as a result
the performance across submissions vary substantially. The highest performing
submission obtains a JGA of 37.9% while the lowest performance is at 18.2%.

Figure 2: Joint Goal Accuracy (JGA) of team’s submissions Figure 3: Slot Error
Rate (SER) of team’s submissions

###  6.1 Alternative ASR models

While we provided ASR output with our baseline ASR system, described in
Section 5.1 based on PeopleSpeech corpus, three submissions used Whisper
instead Radford et al. (2022). We compared the two ASR models to tease apart
the differences. As reported in Table 10, not surprisingly, we found that
Whisper transcribes the evaluation sets more accurately than our baseline
model (see Table 4) since it is trained on a magnitude order more data.

| TTS-Verbatim | Human-Verbatim
---|---|---
dev-dstc11 | 4.8(3.8/0.6/0.4) | 8.5(5.8/1.4/1.4)
test-dstc11 | 4.6(3.7/0.6/0.3) | 8.9(6.1/1.5/1.3)
Table 10: Performance of Whisper on evaluation sets, reported as
WER(Sub/Del/Ins).

We evaluated Whisper in the cascaded ASR-DST system with two models, one
trained on 100x augmented written text and one trained on Whisper transcripts
of the TTS version of the same training data. The results shown in Table 11
clearly demonstrate that the improvements in transcription accuracy translates
to improvement in DST accuracy, whereby JGA scores improve from $30.9\%$ to
$34.3\%$ for the Human-Paraphrased test data.

Test Data | 100x Training Data
---|---
| Text (Ref) | ASR Hyp
Text | 52.7 | 45.3
TTS-Verbatim | 35.6 | 41.3
H-Verbatim | 32.3 | 35.5
H-Paraphrased | 30.9 | 34.3
Table 11: Performance in JGA of cascaded Whisper ASR-D3ST-XXL trained on text
and Whisper hypotheses, evaluated on Whisper hypotheses of evaluation sets.

##  7 Conclusions

In this paper, we describe a new corpus for stimulating research in modeling
spoken dialogs that builds on popular written dialog corpus, MultiWoz. Having
a spoken version of the same evaluation set allows researchers to study and
bridge the performance gap between written and spoken dialog models. In the
course of characterizing the task, we observed that there is substantial
overlap between the slot values in the training and evaluation sets of the
original MultiWoz corpus. We redesigned the evaluation sets (dev-dstc11 and
eval-dstc11) by sampling new non-overlapping slot values and show that the new
sets captures the weakness of the written dialog models better. We released
three versions of the task – TTS-Verbatim, Human-Verbatim and Human-
Paraphrased, where the last two were collected from crowd-workers. One caveat
of this collection is that disfluencies such as speech repair are not as well
represented as in natural conversations.

We have characterized this gap using a baseline cascaded ASR-DST system using
ASR and DST models. While the performance improves with model size and data
augmentation, even the best models (D3ST-XXL) show substantial drop in
performance when switching from written version to the spoken version (53.8%
to 26.1% JGA). Retraining the D3ST-XXL model on the ASR hypotheses, improves
the performance to 30.9% JGA but still leaves substantial ground to be
covered.

We report the results of 11 submissions from the DSTC11 Challenge. The
dominant paradigm across teams was to rely on large language models for DST.
Several submissions inserted ASR error correction modules of different
complexities. The performance of the teams varied from a high of 37.9% to a
low of 18.2% JGA.

One area of research that is not well explored is better utilization of the
latent representations of the audio encoder in the ASR in the downstream DST
models. Similarly, we hope the release of the audio and the audio encoders
outputs will allow researchers to evaluate the power of joint audio-text
encoders on dialog tasks.

## References

  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020.  Towards a human-like open-domain chatbot.  _CoRR_ , abs/2001.09977.
  * Bapna et al. (2022) Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. 2022.  mslam: Massively multilingual joint pre-training for speech and text.  _CoRR_ , abs/2202.01374.
  * Bapna et al. (2021) Ankur Bapna, Yu-An Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan H. Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang. 2021.  SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training.  _CoRR_.
  * Bastianelli et al. (2020) Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020\.  SLURP: A spoken language understanding resource package.  _CoRR_ , abs/2011.13205.
  * Chen et al. (2021) Jiawei Chen, Xu Tan, Yichong Leng, Jin Xu, Guihua Wen, Tao Qin, and Tie-Yan Liu. 2021.  Speech-t: Transducer for text to speech and beyond.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 6621–6633.
  * Chen et al. (2022) Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro J. Moreno, Ankur Bapna, and Heiga Zen. 2022.  MAESTRO: matched speech text representations through modality matching.  _CoRR_ , abs/2204.03409.
  * Chung et al. (2020) Yu-An Chung, Chenguang Zhu, and Michael Zeng. 2020.  Semi-supervised speech-language joint pre-training for spoken language understanding.  _CoRR_ , abs/2010.02295.
  * Drexler and Glass (2019) Jennifer Drexler and James R. Glass. 2019.  Explicit alignment of text and speech encodings for attention-based end-to-end speech recognition.  In _IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019, Singapore, December 14-18, 2019_ , pages 913–919. IEEE.
  * Galvez et al. (2021) Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. 2021.  The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage.  _CoRR_ , abs/2111.09344.
  * Graves (2012) Alex Graves. 2012.  Sequence transduction with recurrent neural networks.  _CoRR_.
  * Henderson et al. (2014a) Matthew Henderson, Blaise Thomson, and J. Williams. 2014a.  The third dialog state tracking challenge.  _2014 IEEE Spoken Language Technology Workshop (SLT)_ , pages 324–329.
  * Henderson et al. (2014b) Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014b.  The second dialog state tracking challenge.  In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_ , pages 263–272, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
  * Jia et al. (2021) Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, and Yonghui Wu. 2021.  Png bert: Augmented bert on phonemes and graphemes for neural tts.  In _Proc. Interspeech_. ISCA.
  * Kim et al. (2022) Seokhwan Kim, Yang Liu, Di Jin, Alexandros Papangelis, Behnam Hedayatnia, Karthik Gopalakrishnan, and Dilek Hakkani-Tur. 2022.  Knowledge-grounded task-oriented dialogue modeling on spoken conversations track at dstc10.  In _AAAI 2022 Workshop on Dialog System Technology Challenge_.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.  _OpenAI Blog_.
  * Song et al. (2020) Xiaohui Song, Liangjun Zang, Yipeng Su, Xing Wu, Jizhong Han, and Songlin Hu. 2020\.  Data augmentation for copy-mechanism in dialogue state tracking.  _CoRR_ , abs/2002.09634.
  * Tang et al. (2022) Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. 2022\.  Unified speech-text pre-training for speech translation and recognition.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1488–1499, Dublin, Ireland. Association for Computational Linguistics.
  * Yang et al. (2021) Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee. 2021.  SUPERB: speech processing universal performance benchmark.  _CoRR_ , abs/2105.01051.
  * Zang et al. (2020) Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020.  MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , Online. Association for Computational Linguistics.
  * Zhao et al. (2022) Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu, Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022.  Description-driven task-oriented dialog modeling.  _CoRR_ , abs/2201.08904.
  * Zhao et al. (2021) Jeffrey Zhao, Mahdis Mahdieh, Ye Zhang, Yuan Cao, and Yonghui Wu. 2021.  Effective sequence-to-sequence dialogue state tracking.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.


# Gaussian Process Self-triggered Policy Search
in Weakly Observable Environments

Hikaru Sasaki1, Terushi Hirabayashi2, Kaoru Kawabata2, and Takamitsu
Matsubara1 1H. Sasaki and T. Matsubara are with the Division of Information
Science, Graduate School of Science and Technology, Nara Institute of Science
and Technology (NAIST), Japan. (email: {sasaki.hikaru.rw3,
takam-m}@is.naist.jp)2T. Hirabayashi and K. Kawabata are with Hitachi Zosen
Corporation, Japan. (email: {hirabayashi, kawabata_k} @hitachizosen.co.jp)

###### Abstract

The environments of such large industrial machines as waste cranes in waste
incineration plants are often weakly observable, where little information
about the environmental state is contained in the observations due to
technical difficulty or maintenance cost (e.g., no sensors for observing the
state of the garbage to be handled). Based on the findings that skilled
operators in such environments choose predetermined control strategies (e.g.,
grasping and scattering) and their durations based on sensor values, we
propose a novel non-parametric policy search algorithm: Gaussian process self-
triggered policy search (GPSTPS). GPSTPS has two types of control policies:
action and duration. A gating mechanism either maintains the action selected
by the action policy for the duration specified by the duration policy or
updates the action and duration by passing new observations to the policy;
therefore, it is categorized as self-triggered. GPSTPS simultaneously learns
both policies by trial and error based on sparse GP priors and variational
learning to maximize the return. To verify the performance of our proposed
method, we conducted experiments on garbage-grasping-scattering task for a
waste crane with weak observations using a simulation and a robotic waste
crane system. As experimental results, the proposed method acquired suitable
policies to determine the action and duration based on the garbage’s
characteristics.

##  I INTRODUCTION

Policy search reinforcement learning has received much attention as a method
for learning control policies for robots [1]. In particular, policy search
using non-parametric policies such as Gaussian process (GP) regression is
effective for complex tasks due to its nonlinear and stochastic properties [2,
3]. Moreover, the complexity of its model can be automatically adjusted using
data [4]. Therefore, compared to parametric policies such as neural networks,
policy search using GP-policy reduces the effort of model design, resulting in
high sample efficiency [2, 3].

However, policy search is difficult for automating heavy industrial machinery
in actual workplaces. Their control systems still have to rely on complicated
system-specific models created by humans [5]. In particular, two major
challenges must be tackled in policy search for industrial machines:

  1. 1.

Obtaining trial and error data by a large and heavy industrial machine is very
costly.

  2. 2.

Since the environments of industrial machines are weakly observable, little
information about the state is contained in observations due to technical
difficulties or maintenance costs.

More specifically, a waste crane, remotely controlled by a skilled operator at
a waste incineration plant, does not have sensors that collect information
about the garbage’s state, although its characteristics have diversity and
changing trends that depend on the day of the week and the season (size and
hardness of each element, moisture content, stickiness, etc.) [6, 7]. The
crane is only equipped with a weight sensor to observe information for grasped
garbage by the bucket. Although the operators can roughly see the whole
garbage pit from the control room, due to severe occlusion they generally
cannot see the garbage around the bucket, especially when it lands the waste
and executes a grasping motion.

Figure 1: Garbage grasping by an actual waste crane by a human operator

Such a control problem can generally be viewed as a partially observed Markov
decision process (POMDP). POMDP solution methods aim to estimate beliefs
(probability distributions of states) from observed data to decide actions
[8]. However, such methods are intractable in such environments as garbage
cranes, where the observations contain very little information. skilled
operators can control such machines and achieve high performance. Their
behavior seems different from general sensory feedback control that selects an
action based on sensor values at regular time intervals (Fig. 1), which is a
typical policy model in policy search and RL (e.g., [9, 3]). This situation
can be attributed to the weakness of the observations; if the value of the
weight sensor does not change significantly, the operator will not be able to
select a different action. Instead, selecting a predetermined control strategy
(e.g., grasping or scattering) and a duration based on sensor values creates
robust behavior and stabilizes it even with weak observations. Such an
approach can reduce the dependency of the policy on sensor values and the
frequency of references. However, these policies need to be adjusted according
to the garbage’s characteristics. Such thoughts motivated us to explore a
novel policy search framework with a specific policy applicable in weakly
observable environments in a sample-efficient way.

In this paper, we propose GP self-triggered policy search (GPSTPS), which is a
novel non-parametric policy search algorithm. GPSTPS has two types of control
policies: action and duration. The gating mechanism either maintains the
action selected by the action policy for the duration specified by the
duration policy or updates the action and duration by passing new observations
to the policy; therefore, it is described as self-triggered. GPSTPS
simultaneously learns both policies by trial and error based on sparse GP
priors [10] and variational learning [3] to maximize the return. We
experimentally verified the performance of our proposed method on a garbage-
grasping-scattering task with a waste crane with weak observations using a
simulation and a robotic waste crane system. Our experimental results suggest
that our method acquired suitable policies to determine the action and
duration based on the garbage’s characteristics.

##  II RELATED WORK

###  II-A Crane and Bucket Automation

Crane automation is conventionally implemented with a complicated model of a
crane system [5]. Robust control over disturbances has also been explored by
modeling such as wind [11] and waves [12]. For such difficult-to-model objects
as soil, previous work designed robust movements for an excavator [13]. In a
data-driven approach, learning methods were proposed that are predictive
models for excavation performance [14] and imitation learning methods for
wheel loaders [15]. Such works designed elaborate models and learned models
with much sensor information; however, such approaches may not be suitable in
environments that have weak observations. In this paper, we propose a data-
driven policy search method for weakly observable environments.

###  II-B Aperiodic Action Update

In the control theory domain, event and self-triggered controls, which are
aperiodic action update methods, have been proposed to reduce communication
costs between the controller and the plant [16]. Event- and self-triggered
control consists of a controller and triggering mechanism. In event-triggered
control, an action update event is triggered when a current observation
violates the triggering condition. Thus, event-triggered control is robust to
disturbance, however, it assumes that the full state information is available.
In self-triggered control, the next action update time is precomputed when an
action is updated. Self-triggered control computes the next action update time
using the model.

In this paper, we employ self-triggered control-like duration determination by
GP-based duration policy since the proactive duration does not need to capture
small changes in observations and can determine a duration from observation.

###  II-C Reinforcement Learning with extended MDP

Complex tasks that cannot be solved by general RL are formulated with extended
MDP. One extension of MDP is the semi-Markov decision process (SMDP). The
option framework reinforcement learning is one of the methods for SMDP, that
temporally abstract task to sub-tasks related to option [17, 18, 19]. In these
learning methods, the task is segmented into sub-tasks and sub-policies are
learned to achieve related sub-tasks. Those methods execute complex tasks by
switching a suitable option aperiodically. The option switching is happened by
a termination model that is similar to a triggering condition in event-
triggered control.

Tasks that are difficult to make decisions based on current observations due
to insufficient information are formulated as partially observable Markov
decision processes (POMDP). For POMDP, RL methods have introduced a belief
that indicates a current state cannot be observed and is inferred from the
past state and action series to determine action [8].

In an environment with less information, observable information is inadequate
for inferring additional information like belief and option. Therefore, our
proposed method assumes designing predetermined control strategies instead of
inference and verifies the effectiveness of the application.

##  III POLICY SEARCH

In this section, we describe model-free policy search as preliminary to derive
our proposed method. Policy search is an algorithm that acquires a policy that
maximizes the expected return in the environment formulated by the Markov
decision process. The policy is learned by repeating the exploration and
improvement as episodic reinforcement learning. Episode
$d=\\{\mathbf{s}_{1},a_{1},\cdots,\mathbf{s}_{T},a_{T},\mathbf{s}_{T+1}\\}$,
which is a series of state $\mathbf{s}_{t}$ and action $a_{t}$ is collected in
a trial and error by the policy $\pi$. A reward $r(\mathbf{s}_{t},a_{t})$ is
calculated using a state and an action. Here we assume the following
derivation where the action is one-dimensional for clarity without a loss of
generality. Let the initial state probability and the state transition
probability be $p(\mathbf{s}_{1})$ and
$p(\mathbf{s}_{t+1}\mid\mathbf{s}_{t},a_{t})$, and express the probability of
episode $p(d\mid\pi)$ as:

$\displaystyle
p(d\mid\pi)=p(\mathbf{s}_{1})\prod_{t=1}^{T}\pi(a_{t}\mid\mathbf{s}_{t})p(\mathbf{s}_{t+1}\mid\mathbf{s}_{t},a_{t}).$
(1)

The policy is improved by calculating it to maximize the expected return
$J(\pi)=\int R(d)p(d\mid\pi)~{}\mathrm{d}d$ using return function
$R(d)=\sum_{t=1}^{T}r(\mathbf{s}_{t},a_{t})$ and probability $p(d\mid\pi)$:

$\displaystyle\pi^{*}$ $\displaystyle=\mathop{\rm
arg~{}max}\limits_{\pi}J(\pi).$ (2)

##  IV GAUSSIAN PROCESS SELF-TRIGGERED POLICY SEARCH

In this section, we propose a self-triggered policy search with GP as a policy
model. We introduce the self-triggered policy model to policy search and
derive the policy update low based on variational policy search [3] by
formulating the expected return as the return-weighted marginal likelihood.

###  IV-A Problem Formulation

Self-triggered policy search has an execution duration of $\tau_{t}$, which
indicates the time to continue action $a_{t}$, and binary gating variable
$o_{t}$, which indicates when the action and its duration time are updated by
the policy at time $t$. The action and duration at each time are modeled:

$\displaystyle p(a_{t}\mid\mathbf{s}_{t},o_{t},a_{t-1},\pi_{a})$
$\displaystyle=\begin{cases}\pi_{a}(a_{t}\mid\mathbf{s}_{t})&\mathrm{if}~{}o_{t}=1\\\
\delta(a_{t}-a_{t-1})&\mathrm{else}\end{cases},$ (3) $\displaystyle
p(\tau_{t}\mid\mathbf{s}_{t},o_{t},\tau_{t-1},\pi_{\tau})$
$\displaystyle=\begin{cases}\pi_{\tau}(\tau_{t}\mid\mathbf{s}_{t})&\mathrm{if}~{}o_{t}=1\\\
\delta(\tau_{t}-\tau_{t-1}+1)&\mathrm{else}\end{cases}.$ (4)

These two distributions include action policy $\pi_{a}$ and duration policy
$\pi_{\tau}$. Gating variable $o_{t}$ indicates action update or maintenance,
and when $o_{t}=1$, the action and the duration are updated by each policy.
When $o_{t}=0$, the previous action is continued and the duration is
decremented. Duration time $\tau_{t}$ indicates the remaining execution
duration of action $a_{t}$. Gating variable $o_{t}$ is determined by duration
$\tau_{t-1}$:

$\displaystyle
p(o_{t}\mid\tau_{t-1})=\begin{cases}\delta(o_{t}-1)&\mathrm{if}~{}~{}\tau_{t-1}=1\\\
\delta(o_{t})&\mathrm{else}\end{cases}.$ (5)

The gating variable becomes $o_{t}=1$ when the duration is $\tau_{t-1}=1$.

In self-triggered policy search, the episode is extended as
$d_{s}=\\{\mathbf{s}_{1},a_{1},\tau_{1},o_{1},\cdots,\mathbf{s}_{T},a_{T},\tau_{T},o_{T},\mathbf{s}_{T+1}\\}$.
The extended episode’s probability is described:

$\displaystyle
p(d_{s}\mid\pi_{a},\pi_{\tau})=p(\mathbf{s}_{1})\prod_{t=1}^{T}p(a_{t}\mid\mathbf{s}_{t},o_{t},a_{t-1},\pi_{a})\cdot$
$\displaystyle\hskip
14.22636ptp(\tau_{t}\mid\mathbf{s}_{t},o_{t},\tau_{t-1},\pi_{\tau})p(o_{t}\mid\tau_{t-1})p(\mathbf{s}_{t+1}\mid\mathbf{s}_{t},a_{t}).$
(6)

Self-triggered policy search uses the episode’s probability to calculate the
expected return $J(\pi_{a},\pi_{\tau})=\int
R(d_{s})p(d_{s}\mid\pi_{a},\pi_{\tau})\mathrm{d}d$ and learns action policy
$\pi_{a}$ and duration policy $\pi_{\tau}$:

$\displaystyle\pi_{a}^{*},\pi_{\tau}^{*}$ $\displaystyle=\mathop{\rm
arg~{}max}\limits_{\pi_{a},\pi_{\tau}}J(\pi_{a},\pi_{\tau}).$ (7)

###  IV-B Sparse Gaussian Process Policy Models

GPs are employed as a policy model in this paper. The nonlinear functions of
the action and duration policies are defined as $f$ and $g$, and GPs are set
as their priors:

$\displaystyle
f\sim\mathcal{GP}(m_{f},\mathrm{k}(\mathbf{s},\mathbf{s}^{\prime})),g\sim\mathcal{GP}(m_{g},\mathrm{k}(\mathbf{s},\mathbf{s}^{\prime})),$
(8)

where $\mathrm{k}(\cdot,\cdot)$ is a kernel function and $m_{f}$ and $m_{g}$
are mean of each GP. We assume a Gaussian distribution for the action and
duration:

$\displaystyle
p(a_{t}|f_{t})=\mathcal{N}(a_{t}|f_{t},\sigma^{2}_{f}),~{}~{}p(\tau_{t}|g_{t})=\mathcal{N}(\tau_{t}|g_{t},\sigma^{2}_{g}),$
(9)

where $f_{t}=f(\mathbf{s}_{t})$, $g_{t}=g(\mathbf{s}_{t})$, $\sigma^{2}_{f}$
and $\sigma^{2}_{g}$ are the variances of each Gaussian distribution.

To reduce the computational complexity of GPs used as the prior distribution
of the nonlinear function, pseudo outputs $\bar{\mathbf{f}}$ and
$\bar{\mathbf{g}}$ corresponding to pseudo input $\bar{\mathbf{s}}$ [10] are
introduced and the prior distributions are set:

$\displaystyle
p(\bar{\mathbf{f}}|\bar{\mathbf{s}})=\mathcal{N}(\bar{\mathbf{f}}|m_{f}\mathbf{1},\mathbf{K}_{\bar{\mathbf{s}}}),p(\bar{\mathbf{g}}|\bar{\mathbf{s}})=\mathcal{N}(\bar{\mathbf{g}}|m_{g}\mathbf{1},\mathbf{K}_{\bar{\mathbf{s}}}),$
(10)

where
$\mathbf{K}_{\bar{\mathbf{s}}}=\mathrm{k}(\bar{\mathbf{s}},\bar{\mathbf{s}})$
is the kernel gram matrix of pseudo input $\bar{\mathbf{s}}$. The
distributions of nonlinear function outputs $f_{t}$ and $g_{t}$ are
represented by the following Gaussian distributions as the GP regression of
the distribution of the pseudo outputs:

$\displaystyle p(f_{t}\mid\mathbf{s}_{t},\bar{\mathbf{f}})$
$\displaystyle=\mathcal{N}(f_{t}\mid\mathbf{k}_{\mathbf{s}_{t},\bar{\mathbf{s}}}\mathbf{K}_{\bar{\mathbf{s}}}^{-1}(\bar{\mathbf{f}}-m_{f})+m_{f},\lambda_{t}),$
(11) $\displaystyle p(g_{t}\mid\mathbf{s}_{t},\bar{\mathbf{g}})$
$\displaystyle=\mathcal{N}(g_{t}\mid\mathbf{k}_{\mathbf{s}_{t},\bar{\mathbf{s}}}\mathbf{K}_{\bar{\mathbf{s}}}^{-1}(\bar{\mathbf{g}}-m_{g})+m_{g},\lambda_{t}),$
(12)

where
$\lambda_{t}=k_{\mathbf{s}_{t}}-\mathbf{k}_{\mathbf{s}_{t},\bar{\mathbf{s}}}\mathbf{K}_{\bar{\mathbf{s}}}^{-1}\mathbf{k}_{\mathbf{s}_{t},\bar{\mathbf{s}}}^{T}$,
$k_{\mathbf{s}_{t}}=\mathrm{k}(\mathbf{s}_{t},\mathbf{s}_{t})$, and
$\mathbf{k}_{\mathbf{s}_{t},\bar{\mathbf{s}}}=\mathrm{k}(\mathbf{s}_{t},\bar{\mathbf{s}})$.

The probability of an episode of self-triggered policy search using GPs as a
policy model is calculated as follows:

$\displaystyle
p(d_{s},\mathbf{f},\mathbf{g}\mid\bar{\mathbf{f}},\bar{\mathbf{g}})=p(\mathbf{s}_{1})\prod_{t=1}^{T}p(a_{t}\mid
f_{t},o_{t},a_{t-1})p(f_{t}\mid\mathbf{s}_{t},\bar{\mathbf{f}})\cdot$
$\displaystyle
p(\tau_{t}|g_{t},o_{t},\tau_{t-1})p(g_{t}|\mathbf{s}_{t},\bar{\mathbf{g}})p(o_{t}|\tau_{t-1})p(\mathbf{s}_{t+1}|\mathbf{s}_{t},a_{t}),$
(13)

where $\mathbf{f}=\\{f_{t}\\}_{t=1}^{T}$ and
$\mathbf{g}=\\{g_{t}\\}_{t=1}^{T}$.

###  IV-C Variational Learning for Policy Improvement

We derive GPSTPS’s policy improvement as a return-weighted marginal likelihood
maximization problem using explored data. The posterior distribution of the
nonlinear functions $f$ and $g$, kernel parameters $\theta$, and pseudo inputs
$\bar{\mathbf{f}}$ and $\bar{\mathbf{g}}$ are optimized by variational
learning.

With the episode’s probability using the GP policy model in (13), the expected
return is calculated:

$\displaystyle J=\int
R(d_{s})p(d_{s},\mathbf{f},\mathbf{g}\mid\bar{\mathbf{f}},\bar{\mathbf{g}})p(\bar{\mathbf{f}})p(\bar{\mathbf{g}})\mathrm{d}d_{s}\mathrm{d}\mathbf{f}\mathrm{d}\bar{\mathbf{f}}\mathrm{d}\mathbf{g}\mathrm{d}\bar{\mathbf{g}}.$
(14)

This equation cannot be analytically solved due to the complexity of $d_{s}$,
$\bar{\mathbf{f}}$, and $\bar{\mathbf{g}}$. Therefore, by introducing the
distribution of the data sample $p_{\mathrm{old}}(d_{s})$, expected return
$J_{\mathrm{old}}$ by that distribution, and variation distribution
$q(d_{s},\mathbf{f},\bar{\mathbf{f}},\mathbf{g},\bar{\mathbf{g}})=p_{\mathrm{old}}(d_{s})p(\mathbf{f}\mid\bar{\mathbf{f}})q(\bar{\mathbf{f}})p(\mathbf{g}\mid\bar{\mathbf{g}})q(\bar{\mathbf{g}})$,
the lower bound of expected return $\log J_{L}$ is derived:

$\displaystyle\log\frac{J(\theta)}{J_{\mathrm{old}}}\geq\int\frac{R(d_{s})}{J_{\mathrm{old}}}q(d_{s},\mathbf{f},\bar{\mathbf{f}},\mathbf{g},\bar{\mathbf{g}})\cdot$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\log\frac{p(d_{s}\mid\bar{\mathbf{f}},\bar{\mathbf{g}})p(\bar{\mathbf{f}})p(\bar{\mathbf{g}})}{q(d_{s},\mathbf{f},\bar{\mathbf{f}},\mathbf{g},\bar{\mathbf{g}})}\mathrm{d}d_{s}\mathrm{d}\mathbf{f}\mathrm{d}\bar{\mathbf{f}}\mathrm{d}\mathbf{g}\mathrm{d}\bar{\mathbf{g}}$
$\displaystyle=\int\frac{R(d_{s})}{J_{\mathrm{old}}}p_{\mathrm{old}}(d_{s})p(\mathbf{f}\mid\bar{\mathbf{f}})q(\bar{\mathbf{f}})\log\frac{p(\mathbf{a}\mid\mathbf{f})p(\bar{\mathbf{f}})}{q(\bar{\mathbf{f}})}\mathrm{d}d_{s}\mathrm{d}\mathbf{f}\mathrm{d}\bar{\mathbf{f}}$
$\displaystyle+\int\frac{R(d_{s})}{J_{\mathrm{old}}}p_{\mathrm{old}}(d_{s})p(\mathbf{g}\mid\bar{\mathbf{g}})q(\bar{\mathbf{g}})\log\frac{p(\boldsymbol{\tau}\mid\mathbf{g})p(\bar{\mathbf{g}})}{q(\bar{\mathbf{g}})}\mathrm{d}d_{s}\mathrm{d}\mathbf{g}\mathrm{d}\bar{\mathbf{g}}$
$\displaystyle+C$ $\displaystyle=\log J_{L}(\theta,q),$ (15)

where $\mathbf{a}=\\{a_{t}\\}_{t=1}^{T}$ and
$\boldsymbol{\tau}=\\{\tau_{t}\\}_{t=1}^{T}$. Each policy model can be learned
using variational policy search [3] since the lower bound is a sum of the
marginal likelihood for each policy.

##  V EXPERIMENT

We applied GPSTPS to a garbage-grasping-scattering task by simulation and a
robotic waste crane system to investigate its effectiveness.

(a) Grasping

(b) Scattering

Figure 2: Garbage-grasping-scattering task

###  V-A Garbage-grasping-scattering Task

The garbage-grasping-scattering task aims to evenly and widely scatter a large
amount of garbage in a short time (Fig. 2). The grasping strategy is set as a
motion that lowers the bucket onto the garbage’s surface and closes the claws
while raising the bucket. The scattering strategy is set as a motion where
some of the garbage in the bucket falls by opening and closing it. To achieve
the aim of the task, we need to switch between grasping and scattering, as
well as their execution duration based on the garbage’s characteristics

In this task, state $s_{t}$ is defined as the weight of the grasped garbage.
The action is defined as binary $a_{t}=\\{0,1\\}$, which indicates either
grasping or scattering strategies. Execution duration $\tau_{t}$ indicates the
number of execution steps for each strategy. The reward function is defined:

$\displaystyle r_{t}=\begin{cases}0&(a_{t}=0)\\\ r_{a}\times
r_{\tau}&(a_{t}=1)\end{cases}.$ (16)

$r_{a}$ is the action reward for the scattering performance, and $r_{\tau}$ is
the reward related to a scattering’s duration. Each episode terminates when
the crane finishes scattering the garbage.

(a) Learning curve

(b) Learning curve

(c) Action policy

(d) Duration policy

(e) Learning curve

(f) Learning curve

(g) Action policy

(h) Duration policy

Figure 3: Result of simulation experiments: (a)-(d) show results in setting 1.
(e)-(h) show results in setting 2. (a) and (e) are the mean and standard
deviation of the learning curve of ten experiments by GPSTPS and GPPS with a
fixed duration. (b) and (f) mean and standard deviation of the learning curve
of ten experiments by GPSTPS and STPS with NN. (c) and (g) learned action
policies by GPSTPS. (d) and (h) learned duration policy by GPSTPS.

###  V-B Simulation

####  V-B1 Experimental settings

We simulated garbage grasping and scattering and verified the GPSTPS
performance. The amount of garbage grasped by the waste crane depends on its
characteristics. Thus, we prepared the two garbage characteristics shown in
Table I. Settings 1 and 2 indicate soft and hard garbage that requires
different execution durations. Hard garbage’s grasping strategy requires a
longer duration since it cannot be loosened by being grasped. We assumed that
for one execution duration step, a grasping strategy takes ten seconds and
that a scattering strategy takes five seconds.

We defined the action and time rewards:

$\displaystyle r_{a}$
$\displaystyle=\mathrm{min}(s_{t},\tau_{t})-\alpha_{\mathrm{sim}}\|s_{t}-\tau_{t}\|,$
(17) $\displaystyle r_{\tau}$
$\displaystyle=\mathrm{exp}\\{-\beta_{\mathrm{sim}}(u_{\mathrm{act}}-u_{\textrm{min}})^{2}\\},$
(18)

where $\alpha_{\mathrm{sim}}=1.5$, $\beta_{\mathrm{sim}}=0.004$, and
$u_{\mathrm{min}}=30$ are the parameters of the reward function and
$u_{\mathrm{act}}$ is the seconds required during garbage scattering. The
action reward is high when the execution durations are similar to a state and
a large amount of garbage is scattered. Time reward $r_{\tau}$ increases as
the execution time is shortened.

The task begins without no grasped garbage in the bucket, and an episode
terminates when the crane has scattered all of the grasped garbage. The action
policy is modeled by a binary sparse GP classification model with $m_{f}=0.5$.
Since the classification model directly regresses the probability of the
action selection, we ignore the uncertainty of the predictive model. The
duration policy is sparse GP regression model with $m_{g}=0$. The maximum
execution duration is set to six steps. For comparison, we employed GP policy
search (GPPS) with a fixed duration from one to six steps. Pseudo inputs of
sparse GP in each method are set $M=5$. Also, we implemented a neural network
(NN) based STPS, which employs NNs with three full-connect layers for both
action and duration policies. We set multiple numbers of units in a hidden
layer in NN as 10, 100, and 300. STPS with NN uses return weight likelihood
derived in [3] as the objective function and learns each NN policy by Adam
optimizer.

TABLE I: Amount of garbage grasped with respect to grasping durations in simulation experiment: $\epsilon$ is sampled by $\mathcal{N}(0,0.7)$. Grasping durations | 1 | 2 | 3 | later
---|---|---|---|---
Setting 1 (soft) | 3$+\epsilon$ | 3$+\epsilon$ | 3$+\epsilon$ | 3$+\epsilon$
Setting 2 (hard) | 2$+\epsilon$ | 3$+\epsilon$ | 5$+\epsilon$ | 5$+\epsilon$

####  V-B2 Result

Fig. 3 shows the experimental result. Fig. 3, 3, 3, and 3 compares learning
curves of GPSTPS, GPPS with a fixed duration, and GPSTPS with NN policy in
each setting. Fig. 3 shows that the action policy appropriately selected the
grasping and scattering strategies based on the state. In Fig. 3, the duration
policy selected a short duration (a one- or two-step grasping strategy) since
the amount of grasped garbage was not changed by the execution duration. The
duration policy selected a similar execution time step as a state for the
scattering strategy. The action policy learned by GPSTPS appropriately
selected the action based on the state (Fig. 3). The learned duration policy
selected a longer two- or three-step execution duration than the duration
policy in setting 1 as its grasping strategy (Fig. 3). For the scattering
strategy, this duration policy resembles the duration policy learned in
setting 1.

In the learning curve of GPPS with a fixed duration in both settings shown in
Figs. 3 and 3, we found different duration needs for each setting. Our GPSTPS
outperformed the fixed duration method by learning a suitable duration for
each setting. In comparison with the NN policy model, the performance of STPS
with the NN policy model largely depends on the number of units in hidden
layers. If we could appropriately set it (100 units), it would result in high
performance; if we set too many (300 units) or too few (10 units), the
performance will be severely degraded. On the other hand, our GPSTPS achieved
comparable performance to that by NN policy models with appropriate settings
without explicitly setting the number of units in hidden layers due to non-
parametric characteristics of GPs.

In summary, these simulation results indicated that GPSTPS can learn
appropriate action and execution duration policies depending on the garbage’s
characteristics.

###  V-C Robot Experiment

(a) Paper-based garbage

(b) Magnet-based garbage

Figure 4: The mock garbage with two different characteristics

(a) Learning curve

(b) Return by learned policies

(c) Action policy

(d) Duration policy

(e) Learning curve

(f) Return by learned policies

(g) Action policy

(h) Duration policy

Figure 5: Result of robot experiment: (a)-(d) show results with paper-based
garbage. (e)-(h) show results with magnet-based garbage. (a) and (e) mean and
standard deviation of return of three experiments. (b) and (f) test
performance of learned policies where * denotes $p<0.05$ on paired t-test.
(c), (d), (g), and (h) action and duration policies learned by each method.

(a) Paper-based garbage

(b) Magnet-based garbage

Figure 6: Robot trajectory by policy learned by GPSTPS

####  V-C1 Experimental settings

We conducted an experiment with a robotic waste crane to verify GPSTPS’s
effectiveness. The detail of the robotic waste crane is described in [7]. We
used two types of garbage (Figs. 4 and 4) to verify the performance with
different garbage characteristics. Fig. 4 shows paper-based garbage that
consists of shredded paper and rubber balls with 17 and 30 mm diameters. Since
this garbage is soft with a small particle size, it resembles dry and non-
sticky garbage like plastic. The bucket can grasp paper-based garbage with a
short duration and the grasped garbage falls from a small gap between the
claws. The magnet-based garbage in Fig. 4 is composed of 27-mm diameter
capsules containing magnets and iron balls. This garbage has a large particle
size, and the magnets attract each other. It resembles wet and easily
aggregated garbage that is collected on rainy days. The bucket needs a longer
grasp duration because the garbage falls from the bucket during the
aggregation. The initial positions of the robotic crane in the grasping and
scattering strategies are randomly selected in the garbage pit and
automatically moved to the initial position. The crane’s moving distance in
the scattering strategy is set to 30 cm.

In the robot experiment, action reward function $r_{a}$ and time reward
function $r_{\tau}$ are defined:

$\displaystyle r_{a}$
$\displaystyle=w_{\mathrm{max}}\mathrm{exp}\\{-\gamma\mathrm{RMS}(\mathbf{m}-\mathbf{m}_{\mathrm{I}})\\},$
(19) $\displaystyle r_{\tau}$
$\displaystyle=\mathrm{exp}\\{-\beta_{\mathrm{robot}}(u_{\mathrm{act}}-u_{\mathrm{min}})^{2}\\},$
(20)

where $\beta_{\mathrm{robot}}=2.5\times 10^{-4}$, $\gamma=7$,
$u_{\mathrm{min}}=30$, and $u_{\mathrm{act}}$ is the execution time of an
episode. $u_{\mathrm{min}}$ means minimum execution time of grasping and
scattering. The purpose of this task is to scatter a lot of garbage in a
shorter time evenly. The action reward function evaluates scattering
performance using RMS between sequence of grasped weight $\mathbf{m}$ and
ideal weight $\mathbf{m}_{\mathrm{I}}$ that decreases weight linearly. The
time reward function evaluates the shortness of scattering.

The policy is learned with the same experimental settings as in the simulation
experiment. The initial action policy selects the grasping strategy when the
grasped weight is 0 g and the scattering strategy at other times.

####  V-C2 Result

Fig. 5 shows the experimental result of the robotic experiment. Figs. 5 and 5
show that GPSTPS outperformed GPPS with each fixed duration. The paired t-test
result shows that GPSTPS’s return has a significant difference against all the
compared methods. Figs. 5 and 5 indicate the learned policies. Figs. 6 show
the robotic waste crane trajectory by the policies learned by GPSTPS. The
learned action policy appropriately selected a control strategy. The learned
duration policy selected two and five steps of execution duration for grasping
and scattering. This result indicates that the duration policy captured the
paper-based garbage’s easy-to-grasp and gradually falling characteristics from
a small gap between the claws.

Figs. 5 and 5 show that GPSTPS outperformed GPPS with each fixed duration. The
paired t-test result shows that GPSTPS’s return has a significant difference
against all the compared methods except GPPS with a fixed four duration.
GPSTPS and GPPS with such a duration do not have a significant difference,
although GPSTPS obtained higher returns than GPPS with the fixed four
duration. Figs. 5 and 5 show the learned policies. Fig. 6 and the robotic
waste crane trajectory by the policy learned by GPSTPS. The learned duration
policy selected three duration steps for grasping the garbage and four for
scattering it by capturing the characteristic of the magnet-based garbage of
the difficult-to-grasp and falling together. In the robotics experiment,
running each episode took about two to three minutes. One learning experiment
took three hours.

Table II shows the performance of initial and learned policies by GPSTPS in
terms of RMS weight sequence and task execution time. We confirmed that
learned policy by GPSTPS significantly improved the evenness of scattering and
execution time by t-test.

TABLE II: Comparison of the performance of initial and learned policy by GPSTPS. Each policy is compared in terms of RMS weight sequence and the execution time of an episode. Values indicate mean and standard deviation of reward by 20 times of execution of the garbage-grasping-scattering task with paper- and magnet-based garbage. * and ** denote $p<0.05$ and $p<0.01$ on paired t-test, respectively. | Policy | Paper garbage | Magnet garbage
---|---|---|---
RMS of | Initial |  $1.17\pm 0.992$* |  $1.74\pm 1.33$**
weight seq. | Learned |  $\mathbf{0.516\pm 0.354}$* |  $\mathbf{0.629\pm 0.694}$**
Execution | Initial |  $64.0\pm 19.5$* |  $59.4\pm 14.2$*
time | Learned |  $\mathbf{50.9\pm 18.4}$* |  $\mathbf{50.8\pm 8.91}$*

In summary, we conducted experiments on a garbage-grasping-scattering task
using two different kinds of mock garbage. GPSTPS properly learned the action
and duration policies based on the garbage’s characteristics.

##  VI DISCUSSION

A future task is indispensable that experimentally verifies the possibility of
applications with an actual waste crane. We must also verify the effectiveness
of cranes/buckets in other weakly observable environments. GPSTPS assumes a
specific factorized action and duration policy model as
$\pi_{a}(a_{t}\mid\mathbf{s}_{t})\pi_{\tau}(\tau_{t}\mid\mathbf{s}_{t})$;
however, there are other possibilities in factorization, such as
$\pi_{a}(a_{t}\mid\mathbf{s}_{t})\pi_{\tau}(\tau_{t}\mid\mathbf{s}_{t},a_{t})$
so that the duration depends on the action for more complicated tasks.

Combining a model-based policy learning approach [20, 21] with our method may
further improve the sample efficiency. Moreover, variational learning-based
policy search may converge to local optima. To alleviate such an issue, we
could alternatively use Markov chain Monte Carlo methods.

Since industrial machines are operated by humans, learning control strategies
by imitation learning [18] may be useful for a wide range of applications. If
the applied task has multimodal state transitions, its policy model can be
extended by multimodality or robustness [3]. In addition, deep kernel learning
[22] can be introduced if handling such high-dimensional data is needed.

##  VII CONCLUSIONS

This study aimed to automate machines in weakly observable environments in an
industrial work place and proposed GPSTPS that can learn both action and
duration policies to repeatedly perform the same action to overcome
uncertainty. Its effectiveness was experimentally verified with simulations
and a robotic waste crane system.

## References

  * [1] K. Chatzilygeroudis, V. Vassiliades, F. Stulp, S. Calinon, and J. Mouret, “A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials,” _IEEE Transactions on Robotics_ , vol. 36, no. 2, pp. 328–347, 2020.
  * [2] H. van Hoof, G. Neumann, and J. Peters, “Non-parametric Policy Search with Limited Information Loss,” _Journal of Machine Learning Research_ , vol. 18, no. 73, pp. 1–46, 2017.
  * [3] H. Sasaki and T. Matsubara, “Variational policy search using sparse Gaussian process priors for learning multimodal optimal actions,” _Neural Networks_ , vol. 143, pp. 291–302, 2021.
  * [4] C. E. Rasmussen and C. K. I. Williams, _Gaussian Processes for Machine Learning_ , ser. Adaptive Computation and Machine Learning. MIT Press, 2006.
  * [5] L. Ramli, Z. Mohamed, A. M. Abdullahi, H. I. Jaafar, and I. M. Lazim, “Control strategies for crane systems: A comprehensive review,” _Mechanical Systems and Signal Processing_ , vol. C, no. 95, pp. 1–23, 2017.
  * [6] R. el Asri and D. Baxter, “Process Control in Municipal Solid Waste Incinerators: Survey and Assessment,” _Waste Management & Research_, vol. 22, no. 3, pp. 177–185, 2004.
  * [7] H. Sasaki, T. Hirabayashi, K. Kawabata, Y. Onuki, and T. Matsubara, “Bayesian Policy Optimization for Waste Crane With Garbage Inhomogeneity,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 4533–4540, 2020\.
  * [8] G. Shani, J. Pineau, and R. Kaplow, “A survey of point-based POMDP solvers,” _Autonomous Agents and Multi-Agent Systems_ , vol. 27, no. 1, pp. 1–51, 2013.
  * [9] Y. Tsurumine, Y. Cui, K. Yamazaki, and T. Matsubara, “Generative Adversarial Imitation Learning with Deep P-Network for Robotic Cloth Manipulation,” in _IEEE-RAS International Conference on Humanoid Robots_ , 2019, pp. 274–280.
  * [10] M. Titsias, “Variational Learning of Inducing Variables in Sparse Gaussian Processes,” in _International Conference on Artificial Intelligence and Statistics_ , 2009, pp. 567–574.
  * [11] A. M. Abdullahi, Z. Mohamed, H. Selamat, H. R. Pota, M. S. Zainal Abidin, F. S. Ismail, and A. Haruna, “Adaptive output-based command shaping for sway control of a 3D overhead crane with payload hoisting and wind disturbance,” _Mechanical Systems and Signal Processing_ , vol. 98, pp. 157–172, 2018.
  * [12] Y. Qian, Y. Fang, and B. Lu, “Adaptive repetitive learning control for an offshore boom crane,” _Automatica_ , vol. 82, pp. 21–28, 2017.
  * [13] F. E. Sotiropoulos and H. H. Asada, “A Model-Free Extremum-Seeking Approach to Autonomous Excavator Control Based on Output Power Maximization,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1005–1012, 2019.
  * [14] Q. Lu and L. Zhang, “Excavation Learning for Rigid Objects in Clutter,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 7373–7380, 2021.
  * [15] S. Dadhich, F. Sandin, U. Bodin, U. Andersson, and T. Martinsson, “Field test of neural-network based automatic bucket-filling algorithm for wheel-loaders,” _Automation in Construction_ , vol. 97, pp. 1–12, 2019.
  * [16] W. P. M. H. Heemels, K. H. Johansson, and P. Tabuada, “An introduction to event-triggered and self-triggered control,” in _IEEE IEEE Conference on Decision and Control_ , 2012, pp. 3270–3285.
  * [17] R. S. Sutton, D. Precup, and S. Singh, “Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning,” _Artificial Intelligence_ , vol. 112, no. 1, pp. 181–211, 1999.
  * [18] C. Daniel, H. van Hoof, J. Peters, and G. Neumann, “Probabilistic inference for determining options in reinforcement learning,” _Machine Learning_ , vol. 104, no. 2, pp. 337–357, 2016.
  * [19] M. Wulfmeier, D. Rao, R. Hafner, T. Lampe, A. Abdolmaleki, T. Hertweck, M. Neunert, D. Tirumala, N. Siegel, N. Heess, and M. Riedmiller, “Data-efficient Hindsight Off-policy Option Learning,” in _International Conference on Machine Learning_ , 2021, pp. 11 340–11 350.
  * [20] M. P. Deisenroth, D. Fox, and C. E. Rasmussen, “Gaussian Processes for Data-Efficient Learning in Robotics and Control,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 37, no. 2, pp. 408–423, 2015.
  * [21] K. Hashimoto, Y. Yoshimura, and T. Ushio, “Learning Self-Triggered Controllers With Gaussian Processes,” _IEEE Transactions on Cybernetics_ , pp. 1–11, 2020.
  * [22] A. G. Wilson, Z. Hu, R. R. Salakhutdinov, and E. P. Xing, “Stochastic Variational Deep Kernel Learning,” in _Advances in Neural Information Processing Systems_ , vol. 29, 2016, pp. 2586–2594.