Carlos Carvalho1,2, Alberto Abad1,2

# Memory-augmented conformer for improved end-to-end long-form ASR††thanks:
This work was supported by Portuguese national funds through Fundação para a
Ciência e a Tecnologia (FCT), with references UIDB/50021/2020 and
2022/12328/BD, as well as by the Portuguese Recovery and Resilience Plan (RRP)
through project C644865762-00000008 (Accelerat.AI)

###### Abstract

Conformers have recently been proposed as a promising modelling approach for
automatic speech recognition (ASR), outperforming recurrent neural network-
based approaches and transformers. Nevertheless, in general, the performance
of these end-to-end models, especially attention-based models, is particularly
degraded in the case of long utterances. To address this limitation, we
propose adding a fully-differentiable memory-augmented neural network between
the encoder and decoder of a conformer. This external memory can enrich the
generalization for longer utterances since it allows the system to store and
retrieve more information recurrently. Notably, we explore the neural Turing
machine (NTM) that results in our proposed Conformer-NTM model architecture
for ASR. Experimental results using Librispeech train-clean-100 and train-960
sets show that the proposed system outperforms the baseline conformer without
memory for long utterances.

Index Terms: conformer, end-to-end speech recognition, neural Turing machine,
memory-augmented neural networks, long-form speech

##  1 Introduction

Traditional speech recognition systems rely on sophisticated and individual
components, including acoustic, pronunciation and language models (LMs) [1].
In contrast, end-to-end (E2E) speech recognition systems rely on a single deep
neural network that learns to map an input sequence of features or raw audio
to the corresponding labels; usually, characters or sub-words [2, 3]. Because
of this simplicity, and in some situations, superior performance over
traditional systems, E2E systems have become a favoured procedure for
automatic speech recognition (ASR) [4, 5]. Some widely used E2E approaches are
based on connectionist temporal classification (CTC) [6], recurrent neural
network transducers (RNN-T) [7] and attention-based encoder-decoders (AEDs)
[4].

The transformer [8] architecture is an AED-based system that uses self-
attention to capture long-range interactions. Nevertheless, a transformer has
more difficulty extracting fine-grained local feature patterns than
convolution neural networks (CNNs) [9]. For this reason, conformers [9] have
been proposed as an approach for E2E ASR, which outperform RNN-based
approaches and transformers since they can model the global and local
dependencies of an audio sequence by combining CNNs with transformers.
Nonetheless, E2E ASR methods, particularly AED-based procedures, are known to
degrade performance on long utterances when trained on short utterances [10,
11]. Besides, long-form transcription is crucial for creating continuous
transcriptions of real-world scenarios, like lectures, meetings, and video
captions (e.g. YouTube).

The problem of long-form speech has been addressed in some previous works by
simulating training on longer utterances [12, 13]. For example, [12] proposed
a method where the transformer or conformer accepts multiple consecutive
utterances simultaneously and predicts an output for the last utterance only.
This procedure is repeated with a sliding window using one-utterance shifts to
recognise the whole recording. Another solution is to segment the audio in
advance using a separate voice activity detector (VAD) based approach [14], or
an E2E model that learns to predict segment boundaries [15]. The E2E segmenter
proposed in [15] relies on human-created heuristics to insert end-of-segment
tokens in utterances at training time so that the model can learn to predict
those tokens.

Only few works try to improve the generalisation of E2E ASR systems to long
speech without requiring some pre-processing stage or changing how the model
trains and decodes compared to traditional E2E ASR. For instance, [16]
proposes the replacement of self-attention with fast attention, which improves
the model generalisation ability for long utterances.

In contrast to the works mentioned above, we hypothesise whether adding a
memory-augmented neural network (MANN) in between the encoder and decoder
module – like a neural Turing machine (NTM) [17] – may be a convenient method
to enrich the learning capacity of a conformer, contributing to increase the
network generalisation for longer utterances without the need for any _ad hoc_
pre-processing or optimisation in training or decoding. Thus, NTMs have
demonstrated superior performance over long short-term memory cells (LSTMs) in
several learning tasks. Moreover, to our knowledge, few works have
investigated the use of MANNs for the E2E ASR task. In particular, NTM has
been used to perform unsupervised speaker adaptation in [18] by storing
i-vectors [19] and then reading from the memory to combine the resulting read
vector with the hidden vectors of the encoder of the listen, attend and spell
(LAS) architecture [4]. However, in that work, the write operation of the NTM
was not explored, therefore not taking advantage of the full potential of the
external memory.

In this work, we propose incorporating a MANN based on NTM to improve the
generalisation of the offline E2E ASR system to long sentences. We refer to
this newly proposed ASR architecture as Conformer-
NTM111https://github.com/Miamoto/Conformer-NTM.git. This proposed model and
the state-of-the-art (SOTA) conformer baseline (without memory) are trained on
Librispeech [20] 100 hours clean and 960 hours. Then, we use the test clean
and other partitions to evaluate the overall performance of all models. We
follow this with an ablation study, testing the models with different
utterance lengths (long and very-long). Our results show that the E2E system
can generalise better with the external memory for longer utterances with the
Conformer-NTM model. Notice that while the focus of this work is on offline
ASR settings, the proposed MANN is also expected to complement streaming ASR
approaches that address the long-form ASR problem [10, 21, 22].

The rest of the paper is organised as follows. Section 2 summarises the MANN
system. Section 3 introduces the proposed approach. In Section 4, we describe
the experimental evaluation and obtained results, and in Section 5, we provide
some concluding remarks and possible directions for future work.

##  2 Memory-augmented neural networks

MANNs refer to a class of neural networks equipped with external memory that
can help improve the learning capability of the neural network [17, 23, 24].
Examples of MANNs are the NTM [17], described below and the differentiable
neural computer (DNC) [23].

###  2.1 NTM model

NTMs [17] can read and write arbitrary content to memory cells without
supervision by using a soft-attention mechanism. Moreover, they are fully
differentiable, which makes it possible to train them in an E2E fashion.

The overall architecture of the NTM is composed of a controller network, e.g.,
a deep neural network or RNN, that receives inputs, reads vectors and emits
outputs in response. The controller reads and writes from an external memory
matrix via a set of parallel read and write heads. Additionally, the
controller network emits a set of vectors and values for each individual read
(e.g., $\mathbf{k}_{t}$, $\beta_{t}$, $g_{t}$, $\mathbf{s}_{t}$ and
$\gamma_{t}$) and write head (e.g.,
$\mathbf{a}_{t},\mathbf{e}_{t},\mathbf{k}_{t}$, $\beta_{t}$, $g_{t}$,
$\mathbf{s}_{t}$ and $\gamma_{t}$), detailed below, to help in the reading and
writing operations.

The memory is a matrix $\mathbf{M}\in R^{N\times W}$, where $N$ is the number
of memory locations (rows) and $W$ is the vector size of each memory location
(columns). The read operation at time step $t$ is the weighted average sum of
all memory locations, i. e.,

$\mathbf{r}_{t}=\sum_{i}\mathbf{w}_{t}^{read}(i)\mathbf{M}_{t}(i),$ (1)

where $\mathbf{w}_{t}^{read}(i)$ is the weight associated to row $i$, and
$\mathbf{M}_{t}(i)$ is the memory vector from row $i$. Also, the sum of all
weights adds up to one. The write operation at time step $t$ contains two main
steps. The first step is an erase phase, i. e.,

$\mathbf{M}_{t}(i)^{{}^{\prime}}=\mathbf{M}_{t-1}(i)[\mathbf{1}-\mathbf{w}_{t}^{write}(i)\mathbf{e}_{t}],$
(2)

where $\mathbf{1}$ is a vector of ones and $\mathbf{e}_{t}\in R^{W}$ is an
erase vector. Last, the second step is an add phase:

$\mathbf{M}_{t}(i)=\mathbf{M}_{t}(i)^{{}^{\prime}}+\mathbf{w}_{t}^{write}(i)\mathbf{a}_{t},$
(3)

where $\mathbf{a}_{t}\in R^{W}$ is an add vector.

The weights mentioned above for reading and writing are computed using the
same addressing mechanism in parallel for each of the two heads. For this
reason, we explain the addressing process in general terms. Overall, the
addressing mechanism combines two main addressing mechanisms: content-based
addressing and location-based addressing. The first step to computing the
weights is to measure the similarity between $\mathbf{k}_{t}$, outputted by
the controller, and each entry of the memory, $\mathbf{M}_{t}(i)$, by using
cosine similarity:

$K[\mathbf{u},\mathbf{v}]=\frac{\mathbf{u}\cdot\mathbf{v}}{\left\|\mathbf{u}\right\|_{2}\left\|\mathbf{v}\right\|_{2}}.$
(4)

By applying cosine similarity and softmax over the rows, $\mathbf{M}_{t}(i)$,
the computation of the weights using the content-based addressing mechanism is
obtained following:

$\mathbf{w}^{c}_{t}(i)=softmax(\beta_{t}K[\mathbf{k}_{t},\mathbf{M}_{t}])_{i},$
(5)

where $\beta_{t}$, outputted by the controller, is a positive scalar parameter
that determines how concentrated the content weight vector should be. The
following three steps are focused on location-based addressing. The second
stage creates $\mathbf{w}^{g}_{t}$ by interpolating $\mathbf{w}^{c}_{t}$ with
the weight vector from last time step, $\mathbf{w}_{t-1}$, using
$g_{t}\in(0,1)$, also outputted by the controller. This interpolation
operation allows the system to learn when to use or ignore content-based
addressing. Next, for the focus of the weights to be shifted to different
rows, the controller emits a shift weighting vector, $\mathbf{s}_{t}$, that
defines a normalised distribution over the allowed integer shifts. Each
element in this vector gives the degree to which different integer shifts can
occur. The actual shift occurs with a circular convolution:

$\mathbf{w}^{*}_{t}(i)=\sum_{j=0}^{N-1}\mathbf{w}^{g}_{t}(j)\mathbf{s}_{t}(i-j).$
(6)

Next, there is a sharpening parameter $\gamma_{t}\geq 1$, outputted by the
controller, which controls the sharpness of the vector weights:

$\mathbf{w}_{t}(i)=softmax(\mathbf{w}_{t}^{*\gamma_{t}})_{i}.$ (7)

Finally, we have a weight vector, $\mathbf{w}_{t}$, that determines where to
read from and write to in memory, depending on the specific head.

Figure 1: Conformer E2E ASR system with an external fully-differentiable
memory.

##  3 E2E ASR with a MANN

The main structure of our memory-based E2E ASR network is depicted in Figure
1. In this architecture we add the proposed external memory in between the
encoder and decoder of the Conformer system.

At first, the input goes through the encoder module, which contains: a
convolution subsampling module, a linear projection layer, a relative
positional encoding module, N conformer blocks and, at last, a layer norm.
Then, the output of the layer norm, $\mathbf{h}$, goes trough the external
memory system. For each time step $t$, the vector $\mathbf{h_{t}}$ will be
transformed with a feedforward layer for each write and read head so that it
is possible to obtain all vectors and values required to help in the reading
and writing operations, as mentioned with more detail in Section 2.1.
Following, the attention weights for the read and write head are computed via
the addressing mechanism as described in Section 2.1.

Writing and reading occur at each time step, allowing the system to memorise
long-term acoustic dependencies recurrently. After the read operation, the
read vector is concatenated with the encoder output $\mathbf{h}_{t}$. Then,
this concatenated vector is transformed into a new output vector with the same
size of $\mathbf{h}_{t}$ by going through a feedforward layer. Next, the
sequence of outputs of the memory goes into the decoder. In addition, the
decoder also receives the transcription outputs shifted right as input, which
go through an embedding layer and a positional encoding module. The decoder
contains D transformer blocks.

At last, the E2E ASR system combined with the external memory learns in a
fully-differentiable way by utilising the joint CTC-attention objective [25].

##  4 Experiments

###  4.1 Data and Experimental Setup

Table 1: Mean, minimum (Min.) and maximum (Max.) duration in seconds of the training sets, the test clean, test other, and the subsets long - 100 and very long - 100. Set/Subset | Category | Mean | Min. | Max.
---|---|---|---|---
train-clean-100 | full set | 12.69 | 1.41 | 24.53
train-960 | full set | 12.30 | 0.83 | 29.74
test clean | full set | 7.42 | 1.29 | 34.96
test other | full set | 6.54 | 1.25 | 34.51
test clean | long - 100 | 23.59 | 19.86 | 34.96
test other | long - 100 | 21.11 | 17.13 | 34.51
concat-clean | very long - 100 | 33.41 | 25.39 | 68.14
concat-other | very long - 100 | 30.76 | 21.77 | 68.98

Our experiments use the Librispeech corpus exclusively. We train with the
train-clean-100 subset from Librispeech, with 28539 utterances and 585
speakers, and the train-960 set containing 281241 utterances and 5466
speakers. We use the dev clean and dev other, with 5567 utterances and 188
speakers. Finally, we report word-error-rate (WER) and character-error-rate
(CER) results for both test clean and test other, with 2620/2939 utterances
and 87/90 speakers, respectively.

We also perform an ablation study varying the distribution of the test data
conditions, especially considering the utterance length (long and very long).
For the long setting, we created subsets from the test clean and the test
other containing only the longest 100 utterances with the script
subset_data_dir.sh, from Kaldi [26]. For the very long set, we used the
original time segmentation information of the Librispeech corpus and
concatenated the continuous segments present in the original test clean and
the test other sets. Then, we selected the 100 longest concatenated segments
using the same Kaldi script mentioned above, resulting into two new subsets:
concat-clean and concat-other. Information about the average, minimum and
maximum length of the utterances of these subsets are present in Table 1.

ESPnet2 [27] is the toolkit we use to implement and investigate our proposed
methods. Also, we use a default ESPnet conformer recipe from Librispeech to
run all our setups. The conformer baseline and the Conformer-NTM models were
trained on 2 NVIDIA GeForce RTX 3080. For the 100 hours setup, all models were
trained for 80 epochs, while for the 960 hours setting, the models were
trained for 50 epochs. In both settings, an average of the ten best
checkpoints in the dev set was used as the final model. Notice that the
Conformer-NTM requires longer training time when compared to the baseline.

The conformer baseline model extracts 80-dimensional FBANK acoustic features
on the fly, followed by SpecAugment [28] and global mean and variance
normalization. The raw input data has speed perturbation factors of 0.9, 1.0
and 1.1. Additionally, the model uses as targets 5000-byte pair encoding (BPE)
[29] unigram units learned from the training data.

The encoder is composed of one Conv2D module followed by 12 conformer blocks.
The Conv2D includes two 2D-CNN layers with 256 channels, a kernel size of 3 x
3, a stride of size two and a ReLU activation. The decoder is composed of six
transformer blocks. Both the encoder and decoder have four attention heads of
dimension 256. The hidden dimension of the feedforward layer for the encoder
and decoder is 256, while the output dimensions are 1024 and 2048,
respectively. Also, the Adam optimizer was used with a learning rate of 0.002,
a weight decay of 0.000001 and 15000 warmup steps. The number of batch bins
for both training setups (train-clean-100 and train-960 hours) was 16 million.
Also, the CTC weight was set to 0.3 for training and testing time.

Table 2: (WERs [%] (CERs [%]) on the E2E ASR proposed models – trained with train-clean-100 and train-960 sets from Librispeech – for the test other and clean sets (full set, long - 100 and very long - 100). |  | Full Set | Long - 100 | Very Long - 100
---|---|---|---|---
Model | LM | test clean | test other | test clean | test other | concat-clean | concat-other
Conformer-100h | - | 6.5 (2.6) | 17.4 (8.5) | 7.5 (3.4) | 16.6 (8.0) | 12.2 (6.5) | 24.5 (14.3)
Conformer-NTM-100h | - | 6.4 (2.4) | 17.2 (8.4) | 6.8 (2.6) | 15.8 (7.3) | 9.2 (4.1) | 21.7 (12.2)
Conformer-960h | - | 2.8 (1.0) | 6.7 (2.9) | 3.3 (1.2) | 6.5 (2.7) | 10.0 (5.8) | 15.3 (9.8)
Conformer-NTM-960h | - | 2.7 (0.9) | 6.8 (2.8) | 3.0 (0.9) | 5.8 (2.1) | 4.3 (2.0) | 11.6 (6.9)
Conformer-100h | 6-gram | 5.3 (2.3) | 14.7 (7.6) | 6.3 (3.0) | 14.2 (7.1) | 10.0 (5.9) | 21.2 (12.9)
Conformer-NTM-100h | 6-gram | 5.2 (2.1) | 14.5 (7.4) | 5.5 (2.3) | 13.6 (6.5) | 7.4 (3.7) | 18.9 (11.2)
Conformer-960h | 6-gram | 2.4 (0.8) | 5.8 (2.5) | 2.7 (0.9) | 5.3 (2.4) | 9.3 (5.5) | 14.7 (9.7)
Conformer-NTM-960h | 6-gram | 2.4 (0.8) | 5.9 (2.5) | 2.7 (0.8) | 4.8 (1.9) | 3.9 (1.8) | 10.8 (6.7)

Regarding the external memories, we first experimented with a different number
of rows (128, 256, 512) and columns (5,8,10,40) for the NTM using the train-
clean-100 hours training setup. The best configuration parameters discovered
were 256 rows and ten columns. We also tried to run with a different MANN, the
DNC [23], which is the follow-up of the NTM system, but for the same chosen
parameters, the NTM gave the best results in preliminary experiments.

At last, we used the KenLM toolkit [30] with Kneser-Ney smoothing to train a
6-gram LM. For that, we used the Librispeech LM corpus from Kaldi and applied
the BPE model to transform all words into sub-word units. The beam size is 60,
and the weight for the 6-gram LM is 1. Furthermore, we report results with and
without LM.

###  4.2 Results

Table 2 compares the performance of our proposed architecture, Conformer-NTM,
versus the ESPnet conformer baseline, with and without an LM.

Regarding the "Full Set" column results, without an LM, the Conformer-NTM
slightly improves upon the Conformer baseline for test clean and test other in
both training settings (train-clean-100 and train-960), except for the test
other in the train-960 setting, where the Conformer-960h model achieves
6.7%/2.9% WER/CER and the Conformer-NTM-960h model achieves 6.8%/2.8% WER/CER.

Decoding with the LM, our proposed models still achieve the lowest WERs and
CERs compared to the baseline model, except for the train-960 setting, where
the results in terms of WER are the same for the Conformer and Conformer-NTM.

###  4.3 Analysis

To examine the behaviour of the Conformer-NTM model for longer sentences, we
decided to create subsets from the test clean and the test other sets
containing long and very long utterances as described in Section 4.1, and
evaluate its performance under these conditions.

####  4.3.1 Long Utterances

Regarding long utterances, without LM, we can observe from Table 2 ("Long -
100" column) that the Conformer-NTM model achieves the lowest WER and CER
results compared to the baseline for both test clean and test other. For the
train-clean-100 setting, the Conformer-NTM model improves from 7.5% to 6.8% in
WER for the test clean and improves from 16.6% to 15.8% WER for the test
other. For the train-960 setting, the Conformer-NTM model improves from 3.3%
to 3.0% in WER for the test clean and from 5.3% to 4.8% WER for the test
other.

With LM, the Conformer-NTM model still obtains better results for both test
clean and test other subsets, except for the train-960 setting in the test
clean subset where the result of the Conformer is equal to the Conformer-NTM
in WER.

####  4.3.2 Very Long Utterances

Concerning very long utterances, described in Section 4.1, we can observe from
Table 2 ("Very Long - 100" column) that the baseline conformer results start
to degrade more when compared to the baseline conformer results in column
"Long - 100", mainly because the distribution of lengths in concat-clean and
concat-other is more distant from the distribution of lengths present in the
training and development data.

Additionally, with and without LM, the Conformer-NTM improves by a significant
margin all the WER and CER scores when compared to the baseline conformer in
both training settings, i.e., train-clean-100 and train-960. For instance, in
the train-960 setting using LM, the Conformer-NTM achieves a relative WER
reduction up to 58.1% and 26.5% for the concat-clean and concat-other sets,
respectively.

These improvements demonstrate that the MANN based on the NTM memory helps the
E2E Conformer ASR system to generalise better for very long sentences not seen
during training without relying on any pre-processing of the data or changing
training and decoding strategies when compared to traditional E2E ASR.
Furthermore, the presence of the LM does not affect the improvements of the
Conformer-NTM when compared to the conformer baseline. We hypothesise that the
NTM memory learns to create more extended contexts at an acoustic level which
benefits the decoder of the conformer when making the inference step.

##  5 Conclusions and Future Work

In this work, we propose a new architecture, Conformer-NTM, that combines a
MANN (based on the NTM) with a conformer for E2E ASR. We demonstrated that
including the external memory is relevant to enhance the performance of the
E2E ASR system for long utterances. Also, we observed that the Conformer-NTM
starts to be more effective when the distribution length of the test data gets
further away from the distribution length of the training data. Furthermore,
in the presence of an LM and for very long utterances, the Conformer-NTM in
the train-960 setting achieves a 58.1% WER relative reduction for the concat-
clean set and a 26.5% WER relative reduction to the concat-other when compared
to the Conformer baseline. Our future work includes investigating the effect
of the NTM on other SOTA E2E ASR architectures.

## References

  * [1] M. Gales and S. Young, ``The application of hidden markov models in speech recognition,'' _Foundations and Trends® in Signal Processing_ , vol. 1, no. 3, pp. 195–304, 2008.
  * [2] A. Graves and N. Jaitly, ``Towards end-to-end speech recognition with recurrent neural networks,'' in _Proceedings of the 31st International Conference on Machine Learning (ICML)_ , ser. Proceedings of Machine Learning Research, vol. 32. PMLR, Jun 2014, pp. 1764–1772.
  * [3] Z. Xiao, Z. Ou, W. Chu, and H. Lin, ``Hybrid ctc-attention based end-to-end speech recognition using subword units,'' in _2018 11th International Symposium on Chinese Spoken Language Processing (ISCSLP)_. IEEE, May 2018, pp. 146–150.
  * [4] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, ``Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,'' in _2016 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, May 2016, pp. 4960–4964.
  * [5] C. Chiu, T. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. Weiss, K. Rao, E. Gonina _et al._ , ``State-of-the-art speech recognition with sequence-to-sequence models,'' in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, Apr 2018, pp. 4774–4778.
  * [6] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _Proceedings of the 23rd international conference on Machine learning (ICML)_ , Jun 2006, pp. 369–376.
  * [7] A. Graves, ``Sequence transduction with recurrent neural networks,'' _In Proceedings of the 29th International Conference on Machine Learning (ICML)_ , 2012\.
  * [8] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _Advances in Neural Information Processing Systems (NIPS)_ , vol. 30. Curran Associates, Inc., 2017.
  * [9] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, ``Conformer: Convolution-augmented Transformer for Speech Recognition,'' in _Proc. Interspeech_ , 2020, pp. 5036–5040.
  * [10] A. Narayanan, R. Prabhavalkar, C. Chiu, D. Rybach, T. Sainath, and T. Strohman, ``Recognizing long-form speech using streaming end-to-end models,'' in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2019, pp. 920–927.
  * [11] C. Chiu, W. Han, Y. Zhang, R. Pang, S. Kishchenko, P. Nguyen, A. Narayanan, H. Liao, S. Zhang, A. Kannan _et al._ , ``A comparison of end-to-end models for long-form speech recognition,'' in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2019, pp. 889–896.
  * [12] T. Hori, N. Moritz, C. Hori, and J. Le Roux, ``Transformer-based long-context end-to-end speech recognition.'' in _Proc. Interspeech_ , October 2020, pp. 5011–5015.
  * [13] T. Hori, N. Moritz, C. Hori, and J. L. Roux, ``Advanced long-context end-to-end speech recognition using context-expanded transformers,'' in _Proc. Interspeech_ , August 2021, pp. 2097–2101.
  * [14] M. Bain, J. Huh, T. Han, and A. Zisserman, ``Whisperx: Time-accurate speech transcription of long-form audio,'' _arXiv preprint arXiv:2303.00747_ , 2023\.
  * [15] W. R. Huang, S. yiin Chang, D. Rybach, T. N. Sainath, R. Prabhavalkar, C. Allauzen, C. Peyser, and Z. Lu, ``E2e segmenter: Joint segmenting and decoding for long-form asr,'' in _Proc. Interspeech_ , September 2022.
  * [16] B. Lyu, C. Fan, Y. Ming, P. Zhao, and N. Hu, ``En-hacn: Enhancing hybrid architecture with fast attention and capsule network for end-to-end speech recognition,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 31, pp. 1050–1062, 2023.
  * [17] A. Graves, G. Wayne, and I. Danihelka, ``Neural turing machines,'' _arXiv preprint arXiv:1410.5401_ , 2014.
  * [18] L. Sarı, N. Moritz, T. Hori, and J. Roux, ``Unsupervised speaker adaptation using attention-based speaker memory for end-to-end asr,'' in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7384–7388.
  * [19] N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, ``Front-end factor analysis for speaker verification,'' _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 4, pp. 788–798, 2010.
  * [20] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [21] C.Wu, Y.Wang, Y.Shi, C.Yeh, and F.Zhang, ``Streaming Transformer-Based Acoustic Models Using Self-Attention with Augmented Memory,'' in _Proc. Interspeech 2020_ , 2020, pp. 2132–2136.
  * [22] E.Tsunoo, Y.Kashiwagi, T.Kumakura, and S.Watanabe, ``Transformer asr with contextual block processing,'' in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2019, pp. 427–433.
  * [23] A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwińska, S. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou _et al._ , ``Hybrid computing using a neural network with dynamic external memory,'' _Nature_ , vol. 538, no. 7626, pp. 471–476, 2016\.
  * [24] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap, ``Meta-learning with memory-augmented neural networks,'' in _Proceedings of The 33rd International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, vol. 48. PMLR, 20–22 Jun 2016, pp. 1842–1850.
  * [25] S. Kim, T. Hori, and S. Watanabe, ``Joint ctc-attention based end-to-end speech recognition using multi-task learning,'' in _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2017, pp. 4835–4839.
  * [26] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, ``The kaldi speech recognition toolkit,'' in _IEEE 2011 Workshop on Automatic Speech Recognition and Understanding_ , Dec. 2011.
  * [27] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. Soplin, J. Heymann, M. Wiesner, N. Chen _et al._ , ``Espnet: End-to-end speech processing toolkit,'' _in Proc. ISCA Interspeech_ , Sep 2018.
  * [28] D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. Le, ``SpecAugment: A simple data augmentation method for automatic speech recognition,'' in _Interspeech 2019_. ISCA, Sep 2019.
  * [29] R. Sennrich, B. Haddow, and A. Birch, ``Neural machine translation of rare words with subword units,'' in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Aug 2016.
  * [30] K. Heafield, ``Kenlm: Faster and smaller language model queries,'' in _Proceedings of the sixth workshop on statistical machine translation_ , 2011, pp. 187–197.


# “I Don’t Think So”: Summarizing Policy Disagreements for Agent Comparison

Yotam Amitai, Ofra Amir

###### Abstract

With Artificial Intelligence on the rise, human interaction with autonomous
agents becomes more frequent. Effective human-agent collaboration requires
users to understand the agent’s behavior, as failing to do so may cause
reduced productivity, misuse or frustration. Agent strategy summarization
methods are used to describe the strategy of an agent to its destined user
through demonstration. A summary’s objective is to maximize the user’s
understanding of the agent’s aptitude by showcasing its behaviour in a
selected set of world states. While shown to be useful, we show that current
methods are limited when tasked with comparing between agents, as each summary
is independently generated for a specific agent. In this paper, we propose a
novel method for generating dependent and contrastive summaries that emphasize
the differences between agent policies by identifying states in which the
agents disagree on the best course of action. We conduct user studies to
assess the usefulness of disagreement-based summaries for identifying superior
agents and conveying agent differences. Results show disagreement-based
summaries lead to improved user performance compared to summaries generated
using HIGHLIGHTS, a strategy summarization algorithm which generates summaries
for each agent independently.

## Introduction

With the maturing of reinforcement learning (RL) methods, RL-based agents are
being trained to perform complex tasks in various domains, including robotics,
healthcare and transportation. Importantly, these agents do not operate in a
vaccum – people interact with agents in a wide range of settings. Effective
interaction between an agent and a user requires from the latter the ability
to anticipate and understand its behavior. For example, clinicians should
understand treatment regimes recommended by agents to determine their
viability.

To facilitate improved user understanding of agents’ behavior, a range of
explainable RL methods have been developed (Puiutta and Veith 2020; Heuillet,
Couthouis, and Díaz-Rodríguez 2021). These can be divided into two explanation
categories: (1) “local” explanation approaches explaining why an agent chose a
particular action in a given state, e.g., saliency maps highlighting the
information an agent attends to (Greydanus et al. 2017), and (2) “global”
explanation methods that describe the policy of an agent more generally, such
as strategy summaries that demonstrate the agents’ behavior in a selected set
of states (Amir, Doshi-Velez, and Sarne 2019). While these approaches have
been shown to improve people’s understanding of agent behavior, they are
typically not optimized for a particular user task.

In this work, we aim to enhance users’ ability to distinguish between the
behavior of different agents. Such scenarios arise when people are required to
select an agent from a set of alternatives. E.g., a user might need to choose
from a variety of smart home assistants or self-driving cars available on the
market. Importantly, there often isn’t a clear “ground-truth” for which agent
is superior, as agents may prioritize alternative outcomes and users may
occupy different preferences. For example, some users may prefer self-driving
cars that value safety very highly, while others might be willing to relax
safety considerations to a degree to allow for faster driving. The ability to
distinguish policies is also important for model developers, as different
configurations of reward functions and algorithm parameters can lead to
different behaviors in unexpected ways, especially in domains where the reward
function is not obvious, such as healthcare (Gottesman et al. 2019).

One possible approach for helping users distinguish between policies of agents
is to use strategy summarization methods (Amir, Doshi-Velez, and Sarne 2018,
2019). Using these methods, a summary is generated for each agent, allowing
the user to compare their behavior. However, these approaches are not
optimized for the task of agent comparison, as each summary is generated
_independently_. For instance, the HIGHLIGHTS algorithm (Amir and Amir 2018)
selects states based on their importance as determined by the differences in
Q-values for alternative actions in a given state. If two high-quality agents
are compared, it is possible that they will consider the same states as most
important, and will choose the same (optimal) action in those states,
resulting in similar strategy summaries. In such a case, even if the agents’
policies differ in numerous regions of the state-space, it will not be
apparent from the summaries.

This work presents the DISAGREEMENTS algorithm which is optimized for
comparing agent policies. Our algorithm compares agents by simulating them in
parallel and noting disagreements between them, i.e. situations where the
agents’ policies differ. These disagreement states constitute behavioral
differences between agents and are used to generate visual summaries optimized
towards showcasing the most prominent conflicts, thus providing contrastive
information. Our approach assumes access to the agent’s strategy, described
using a Markov Decision Process (MDP) policy, and quantifies the importance of
disagreements by making use of agents’ Q-values.

To evaluate DISAGREEMENTS, we conducted two human-subject experiment to test
the following properties of the algorithm: Firstly, whether it improves users’
ability to identify a superior agent when one exists, i.e. given ground-truth
performance measures. Secondly, do DISAGREEMENTS summaries outperform
HIGHLIGHTS summaries in conveying differences in behavior of agents to users.
Both experiments make use of HIGHLIGHTS summaries as a baseline for
comparison. Results indicate a significant improvement in user performance for
the agent superiority identification task, while not falling short in the
behaviour conveying task, as compared to HIGHLIGHTS.

Our contributions are threefold: i) We introduce and formalize the problem of
comparing agent policies; ii) we develop DISAGREEMENTS, an algorithm for
generating visual contrastive summaries of agents’ behavioral conflicts, and
iii) we conduct human-subject experiments, demonstrating that summaries
generated by DISAGREEMENTS lead to improved user performance compared to
HIGHLIGHTS summaries.

## Related Work

In recent years, explainable AI has regained interest, initially focusing
mainly on explaining supervised models. More recently, research has begun
exploring explanations of reinforcement learning agents (Puiutta and Veith
2020; Heuillet, Couthouis, and Díaz-Rodríguez 2021). In this work, we focus on
global explanations that aim to describe the policy of the agent rather than
explain a particular action. Specifically, we develop a new method for
strategy summarization. In this section we describe in more depth strategy
summary methods (Amir, Doshi-Velez, and Sarne 2019).

Strategy summarization techniques convey agent behavior by demonstrating the
actions taken by the agent in a selected set of world states. The key question
in this approach is then how to recognize meaningful agent situations.

One such approach called HIGHLIGHTS (Amir and Amir 2018), extracts _important_
states from execution traces of the agent. Intuitively, a state is considered
important if the decision made in that state has a substantial impact on the
agent’s utility. To illustrate, a car reaching an intended highway exit would
be an important state, as choosing a different action (continuing on the
highway) will cause a significant detour. HIGHLIGHTS has been shown to support
people’s ability to understand the capabilities of agents and develop mental
models of their behavior (Amir and Amir 2018; Huber et al. 2020).

Sequeira and Gervasio (2020) extended HIGHLIGHTS by suggesting additional
importance criteria for the summary state selection referred to as
_Interestingness Elements_. Huang et al. (2017); Lage et al. (2019) proposed a
different approach for generating summaries based on machine teaching methods.
The key idea underlying this approach is to select a set of states that is
optimized to allow the reconstruction of the agents’ policy using imitation
learning or inverse reinforcement learning methods.

Common to all previous policy summarization approaches is that each summary is
generated specifically for a single agent policy, independent of other agents.
This can hinder users’ ability to compare agents, as the summaries might show
regions of the state-space where the agents act similarly, failing to capture
useful information with respect to where the agent policies diverge. For
example, HIGHLIGHTS focuses on “important” states and it is likely that the
states found to be most the important to one agent will be considered
important by another agent as well. These could be inherently important stages
of the domain such as reaching the goal or evading a dangerous state. If the
agents act similarly in these important states, the HIGHLIGHTS summaries of
the agents might portray similar behavior, even for agents whose global
aptitude varies greatly. In contrast, if the summaries do differ from one
another and portray different regions of the state-space, they do not convey
how the alternative agent would have acted had it been tasked with the same
scenario.

To address these limitations, we propose a new approach that is specifically
optimized for supporting users’ ability to distinguish between policies.

## Background

For the purpose of this work, we assume a Markov Decision Process (MDP)
setting. Formally, an MDP is a tuple $\langle S,A,R,Tr\rangle$, where S is the
set of world states; A is the set of possible actions available to the agent;
$R:S\times A\rightarrow\mathbb{R}$ is a reward function mapping each state,
and $Tr(s,a,s^{\prime})\rightarrow[0,1]\;s.t.\;s,s^{\prime}\in S,\;a\in A$ is
the transition function. A solution to an MDP is a _policy_ denoted $\pi$.

### Summaries

A summary, denoted $\mathbb{S}$, is a set of trajectories $T=\langle
t_{1},\dots t_{n}\rangle$. Each trajectory $t$ being a sequence of $l$
consecutive states $t=\langle s_{i},\dots,s_{D},\dots,s_{i+l}\rangle$
surrounding the disagreement state $s_{D}$ and extracted from the agent’s
execution traces.

We formally define a summary extraction process of an agent’s policy given an
arbitrary importance function $Im$, mapping state-action pairs to numerical
scores.

######  Definition 1 (Summary Extraction).

Given an agent’s execution traces, a summary trajectory budget $k$, and an
importance function $Im$.
The agent’s summary $\mathbb{S}$ is then the set of trajectories
$T=\\{t_{1},...,t_{k}\\}$ that maximizes the importance function.

$\displaystyle\mathbb{S}=\max_{T}\;Im(T)$ (1)

In this paper, our baseline is the HIGHLIGHTS algorithm, which computes
importance as a function of the $Q$-values in a given state. Specifically, we
implement the importance function from Huber et al. (2020), an extension to
HIGHLIGHTS, which suggests determining the importance of a state based on the
difference between the maximal and second $Q$-values. Formally:

$\displaystyle
Im(s)=\max_{a}\;Q^{\pi}(a,s)-second\underset{a}{h}ighest\;Q^{\pi}(a,s)$ (2)

The trajectory is then the sequence of states preceding and succeeding the
important state.

## Disagreement-Based Summaries

We propose a new summary method which supports the comparison of alternative
agents by explicitly highlighting the _disagreements_ between them. Thus,
constructing _contrastive_ summaries that convey policy divergence between
agents exposed to the same world states. This approach is in line with the
literature on explanations from the social sciences, which shows that people
prefer contrastive explanations (Miller 2018). We note that while typically
contrastive explanations refer to “why not?” questions and consider
counterfactuals, in our case the contrast is between the decisions made by two
different policies.

We next describe our proposed disagreement-based summary method. Specifically,
we formalize the notion of agent disagreement, describe the “DISAGREEMENTS”
algorithm for generating a joint summary of the behavior of two agents, and
describe how to measure the importance of a disagreement state and
disagreement trajectory.

### Agent Disagreement

The two main dimensions on which agents can differ are their valuations of
states and their policies, i.e. their choice of action. These dimensions are
of course related, as different state valuations will naturally lead to
different policies. Our definition of a disagreement focuses on the policy. We
then utilize the value function for ranking purposes.

In other words, any state $s$ for which different agents choose different
actions is considered a disagreement state. We use these states to analyze and
portray how the agents differ from one another in their behavior. Formally:

######  Definition 2 (Disagreement State).

Given two agents ${Ag}_{1}$ and ${Ag}_{2}$ with policies $\pi_{1},\pi_{2}$,
respectively. Define a state $s$ as a disagreement state $s_{D}$ iff:

$\displaystyle\pi_{1}(s)\neq\pi_{2}(s)$ (3)

The set of all disagreement states $\mathbb{D}$ would then be:

$\displaystyle\forall s\in S\;|\;\pi_{1}(s)\neq\pi_{2}(s):s\in\mathbb{D}$ (4)

For a compact MDP where every state may be computed, this definition would
suffice. Alas, for more complex settings containing a continuous or vast state
space, it is not feasible to compare all states. The proposed method must be
able to overcome this difficulty.

### Identifying Agent Disagreements Through Parallel Online Execution

Given two alternative agents to compare, we initiate an online execution of
both agents simultaneously such that we follow the first (denoted as the
_Leader_ or $L$ for short, with policy $\pi_{L}$), while querying the second
(denoted as the _Disagreer_ or $D$ for short, with policy $\pi_{D}$) for the
action _it_ would have chosen in each state. Both agents are constrained to
act greedily and deterministically with respect to their $Q$-values. Upon
reaching a disagreement state, we allow the _Disagreer_ to “split-off” from
following the _Leader_ and continue independently for a limited number of
steps while recording the states it reaches for later analysis. Once the limit
is reached, we store the _disagreement trajectory_ , and revert the
_Disagreer_ back to the disagreement state, from which it continues to follow
the _Leader_ until the next disagreement state is reached and the process is
repeated.

### The DISAGREEMENTS Algorithm

The algorithm pseudo-code is supplied in Algorithm 1. It’s parameters are
summarized in Table 1.

Parameter |  Description |  F |  H
---|---|---|---
$k$ |  Summary budget, i.e. number of trajectories |  5 |  5
$l$ |  Length of each trajectory |  10 |  20
$h$ |  Number of states following $s$ to include in the trajectory |  5 |  10
$numSim$ |  The number of simulations (episodes) run by the DISAGREEMENTS algorithm |  10 |  10
$overlapLim$ |  Maximal number of shared states allowed between two trajectories in the summary |  3 |  5
$impMeth$ |  Importance method used for evaluating disagreements |  _Last State_ |  _Last State_

Table 1: Parameters for Frogger & Highway domains.

_The Algorithm._ First, three lists are initialized for the Leader traces,
disagreement states and Disagreer trajectories (lines 4–6). Then, simulation
of the agents are run (lines 7–27). Each simulation collects all states seen
by the Leader during the execution (line 24), disagreement states (line 13),
and the Disagreer trajectories (lines 14–19). Each step of the simulation,
both agents are queried for their choice of action (lines 10–11), if they
disagree on the action — a disagreement state is added (line 13) and a
disagreement trajectory is created (lines 14–19), after which the simulation
is reverted to the last disagreement state (line 21). Upon simulations
completion, all disagreement trajectories (coupled pairs of Leader and
Disagreer trajectories) are obtained (line 28) and the most important ones are
passed as output (line 29).

Algorithm 1 The DISAGREEMENTS algorithm.

1: Input: $\pi_{L},\pi_{D},k,l,h,$

2: $overlapLim,numSim,impMeth$

3: Output: $\mathbb{S}$

4: $L_{Tr}\leftarrow$ empty list  #Leader traces

5: $\mathbb{D}\leftarrow$ empty list  #Disagreement states

6: $D_{T}\leftarrow$ empty list  #Disagreer trajectories

7: for $i=1$ to $numSim$ do

8: $sim,s=InitializeSimulation()$

9: while $(!sim^{\pi_{L}}.ended())$ do

10: $a^{\pi_{L}}\leftarrow sim.getAction(\pi_{L}(s))$

11: $a^{\pi_{D}}\leftarrow sim.getAction(\pi_{D}(s))$

12: if $a^{\pi_{L}}!=a^{\pi_{D}}$ then

13: $\mathbb{D}.add(s)$

14: $d_{t}\leftarrow$ empty list  #Disagreer trajectory

15: for $i=1$ to $h$ do

16: $s^{\pi_{D}}\leftarrow sim.advanceState(\pi_{D})$

17: $a^{\pi_{D}}\leftarrow sim.getAction(\pi_{D}(s))$

18: $d_{t}.add(s)$

19: end for

20: $D_{T}.add(d_{t})$

21: $sim,s=reloadSimulation(D_{s}[-1])$

22: end if

23: $s\leftarrow sim.advanceState(\pi_{L})$

24: $L_{traces}.add(s)$

25: end while

26: $runs=runs+1$

27: end for

28: $DA_{T}\leftarrow disagreementTrajPairs(\mathbb{D},L_{Tr},D_{T},l,h)$

29: $\mathbb{S}\leftarrow topImpTraj(DA_{T},k,overlapLim,impMeth)$

Since the _Leader_ agent controls which areas of the domain state-space are
reached, we repeat the process again, reversing the agent’s roles. This is
important because the Leader determines which regions of the state-space are
reached and as such also conveys information about the agent’s preferences.
This process results in two sets of disagreements states (one for each agent
as the _Leader_). Typically, these sets can be very large, and it is
infeasible for a user to explore all of them. Therefore, a ranking procedure
is necessary for the disagreements found in order to generate a compact
summary. We next describe approaches for quantifying the importance of a
disagreement.

### Disagreement Importance

Various methods can be used for determining disagreement importance. We first
define the notion of a state’s value based on multiple agents.

State Value We assume the agents are $Q$-based, possessing a $Q$ function
($Q(s,a)\rightarrow\mathbb{R}$) which quantifies their valuation of state-
action pairs, denoted as $Q$-values. $Q$-values are calculated and adjusted
during the training phase of the agent and depend on the algorithm used as
well as on the specification of the reward function. Therefore, $Q$-values of
different agents may vary greatly. This renders each individual agent’s
assessment of a state-action pair as its own estimate rather than representing
a ground truth. The values themselves may not even be on the same scale. To
allow for comparison between values, we normalize each agent’s $Q$-values by
dividing by the maximum $Q$-value appearing in each, thus rendering each value
an indication of how good a state-action pair is compared to the best one
observed by the agent. Formally:

$\displaystyle
Q^{\prime}=\frac{Q-\min_{s,a}\big{(}Q(s,a)\big{)}}{\max_{s,a}\big{(}Q(s,a)\big{)}-\min_{s,a}\big{(}Q(s,a)\big{)}}$
(5)

Since our agents’ action selection is greedy and deterministic with respect to
their $Q$-values, we denote the value of state $s$ as the highest $Q$-value
associated with it.

######  Definition 3 (Agent State Value).

Given an agent $Ag$, its $Q$ function $Q^{Ag}$ and a state $s$, we define the
value of $s$ as:

$\displaystyle V^{Ag}(s)=\max_{a}{Q^{\prime}}^{Ag}(s,a)$ (6)

Alas, measuring only the importance of a single state has its limitations.
Without ground truth, we are left only with the agents’ estimations which may
be flawed. Suppose our agents reach a disagreement state where both are
convinced the other’s action is a poor choice. They each go their separate
ways only to reunite at a shared state after several steps with minimal to no
impact on the succeeding execution, e.g. overtaking a vehicle from the left or
from the right. This realization led us to formulate a trajectory-based
approach for determining the importance of a disagreement state.

### Disagreement Trajectory Importance

To determine the importance of a disagreement state, we compare the
trajectories that branch out of it by following each agent. We formulate
trajectory value metrics to evaluate these, while constraining ourselves to
observing only trajectories of similar length.

A trajectory ${t}_{h}^{\pi}(s)=\langle s_{+1},...,s_{+h}\rangle$ denotes the
sequence of states encountered when following state $s$ for $h$ steps
according to a policy $\pi$. Since each agent evaluates states differently, we
consider the value of a state as the sum of both agents’ valuations, i.e.
$V(s)=V^{L}(s)+V^{D}(s)$. There are numerous ways to quantify the importance
of a trajectory. A description of several methods tested in our work is
provided in the Appendix. The summaries generated for the user studies made
use of the _last-state_ importance method.

Last State Importance: We define the importance of a disagreement trajectory
as the difference between the values of the last states reached by each of the
agents. This reflects how “far off” from each other the disagreement has led
the agents. Formally:

$\displaystyle
Im({t}_{h}^{\pi_{L}}(s),{t}_{h}^{\pi_{D}}(s))=|V(s^{\pi_{L}}_{+h})-V(s^{\pi_{D}}_{+h})|$
(7)

This metric achieves a high importance score when one agent arrives at a state
which both agents agree is valuable, while the other agent arrives at a state
whose value both agents agree is poor.

### Trajectory Diversity

Using these methods we are able to acquire the set of disagreement states
$\mathbb{D}$ ordered by importance, and for each, their corresponding
trajectories. These shall be woven together to create a visual summary for
displaying to the user. To increase the coverage of the summary and avoid
showing redundant trajectories (in both methods), we restrict the summary
generated to not contain _i)_ multiple trajectories that end or begin at the
same state, _ii)_ trajectories where the Leader and Disagreer share the same
state before-last and _iii)_ overlapping trajectories which share more than a
predefined number of states.

## Empirical Methodology

To evaluate our method we conducted two user studies. The first was designed
to assess whether DISAGREEMENTS summaries help users identify the superior
between two alternative agents, while the second user study examines whether
such summaries are useful for conveying agent behavior differences. In both
studies we use the HIGHLIGHTS algorithm as a baseline for comparison. We note
here that there is a significant difference between the output summaries of
both methods rooted in the fact that DISAGREEMENTS was designed for presenting
two policies in a contrastive manner. This is achieved by portraying both
agents simultaneously on the screen. To our knowledge no other global
explanation methods exist that directly compare policies and visualizes their
differences to lay users.

_Empirical Domains._ To evaluate our algorithm we generated summaries of
agents playing the game of Frogger (Sequeira and Gubert 2020) and controlling
a vehicle in a highway environment (Leurent 2018).

Frogger The objective of the game is to guide a frog from the bottom of the
screen to an empty lily-pad at the top of the screen. The agent controls the
frog and can initiate the following four movement actions: up, down, left or
right, causing the frog to hop in that direction. To reach the goal the agent
must lead the frog across a road with moving cars while avoiding being run
over, then, the agent must pass the river by jumping on passing logs. This
domain allows us to compare different agents in a setting with ground truth
information about agents’ skill, i.e. game score.

Highway This domain consists of a busy highway with multiple lanes and
vehicles. The agent controls a vehicle driving through traffic with the intent
of avoiding collisions. The agent can choose to move right or left (changing
lanes), increase or decrease velocity or stay idle, i.e. make no change. There
is no defined target the agent is required to reach, instead the road goes on
continuously. This property allows us to observe the agent’s general behavior
and preferences instead of focusing on its progression towards reaching the
goal.Screenshots from the DISAGREEMENTS output summaries displayed in Figure
1.

Figure 1: Top: Frogger; Bottom: Highway.
Red & black rectangles represent different agents.

_Frogger Agents._ We made use of the framework developed by Sequeira and
Gervasio (2020) to test the DISAGREEMENTS algorithm on multiple configurable
agents of varying capabilities. Three different agents were trained using
standard $Q$-learning (watkins1992q), based on the configurations provided by
the framework.

  * •

Expert (E): 2000 training episodes. Default rewards. Average game score:
110,000.

  * •

Mid-range (M): 1000 training episodes. Default rewards. Average game score:
50,000.

  * •

LimitedVision (LV): 2000 training episodes. Default rewards. Lower perception
of incoming cars. Average game score: 55,000.

Agent performance was calculated by averaging the game score of ten
executions. Each agent’s unique configuration contributes to its performance,
providing us a ground truth for the assessment tasks we present to the
experiment participants. The agents’ skill hierarchy, based on their average
score, is as follows: $E>LV>M$. An important requirement for the experiment
was that all agents have a decent ability to play Frogger. Prior to this
study, we verified via an additional experiment that HIGHLIGHTS summaries are
indeed useful for comparing Frogger agents that differ _substantially_ in
their skills (see Appendix). All HIGHLIGHTS and DISAGREEMENTS summaries were
generated for fully trained agents, thus reflecting their final policies.

_Highway Agents._ Agents with varying behaviors were trained by altering their
reward functions. All highway agents were trained for 2000 episodes using
double DQN architecture (Hasselt 2010) and rewarded for avoiding collisions.

  * •

ClearLane (CL): Rewarded for high velocity while maximizing the distance
between itself and the nearest vehicle in front of it.

  * •

SocialDistance (SD): Rewarded for maximizing the distance between itself and
the closest $k$ vehicles.

  * •

FastRight(FR): Rewarded for high velocity and driving in the rightmost
(bottom) lane.

Henceforth, we will refer to all domain agents by their abbreviations.

_Summary Attributes_ All summaries were composed of five trajectories made up
of sequential states, ten for Frogger and twenty for Highway. These contained
the important state at the center of the trajectory, with half the states
preceding and the rest succeeding it. Video-clips of the summaries were
generated to present to the users and a fade-in and fade-out effect was added
to further call attention to the transition between trajectories. For more
details, sensitivity analysis and the complete surveys, see Appendix.

### Experiment 1 - Identifying Superiority

The objectives of the first experiment were twofold. Firstly, to support our
claims regarding the limitations of the HIGHLIGHTS algorithm for comparing
agents, and secondly, to compare the DISAGREEMENTS algorithm to HIGHLIGHTS and
show its added value.

_Hypotheses._ We hypothesized that summaries generated by the HIGHLIGHTS
algorithm are limited in their ability to help users distinguish between
agents and the DISAGREEMENTS algorithm is more suited for this task. More
specifically, we state the following hypotheses:
H1: Participants shown summaries generated by HIGHLIGHTS for agents of decent
skill will struggle to identify the better performing agent.
H2: Participants shown summaries generated by the DISAGREEMENTS algorithm will
exhibit a higher success rate for identifying the better performing agent,
compared to ones shown HIGHLIGHTS summaries.

_Experimental Conditions_ A between-subject experimental setup was designed
with two experimental conditions that varied in the summary generation method,
DISAGREEMENTS or HIGHLIGHTS (Amir and Amir 2018). Participants were randomly
assigned a condition.

_Participants._ 74 participants were recruited through Amazon Mechanical Turk
(27 female, mean age $=36.51$, STD $=9.86$), each receiving $\$3$ for their
completion of the Task. To incentivize participants to make an effort, they
were provided a bonus of 10 cents for each correct answer in the superiority
identification task. Participants who spent less than a threshold duration of
time on experiment tasks, based on the length of the task summary video, were
filtered out.

_Procedure._ Participants were first introduced to the game of Frogger and the
concept of AI agents. Each explanation was followed by a short quiz to ensure
understanding before advancing to the task. Next, participants were randomly
split into one of two conditions and were shown summary videos of pairs of
different agents generated using either DISAGREEMENTS or HIGHLIGHTS.

Participants in both groups were first introduced to the summary method they
would be shown and were required to pass a quiz to ensure their understanding.
Participants were then asked to choose the better performing agent based on
the summary videos. They were able to pause, play and repeat the summary
videos without restrictions, allowing freedom to fully inspect the summary
before deciding which agent they believe is more skillful. Participants were
also asked to provide a textual explanation for their selection and to rate
their decision confidence on a 7-point Likert scale (0 - not at all confident
to 6 - very confident). Overall, there were 3 pairs of agent comparisons
$\langle E,M\rangle\langle E,LV\rangle\langle LV,M\rangle$. The ordering of
the agent pairs was randomized to avoid learning effects, and participants
were also not told if the same agent appeared in multiple comparisons, that
is, they made each decision independently of other decisions.

Participants in the HIGHLIGHTS condition were shown a HIGHLIGHTS summary of
each agent (i.e. two separate videos, one for each agent.), while participants
in the DISAGREEMENTS group were supplied two configurations of the
DISAGREEMENTS summaries. One summary where the first agent is the Leader while
the second is the Disagreer, and the opposite summary, where the first agent
is the Disagreer and the second is the Leader. Upon conclusion, participants
answered a series of explanation satisfaction questions adapted from (Hoffman
et al. 2018).

_Evaluation Metrics and Analyses._ The main evaluation metric of interest was
the success rate of identifying the superior Frogger agent with each summary
method. We compare this metric across all the agent selection tasks given to
participants. We also compare participants’ confidence in their decision. To
compare the explanation satisfaction ratings given to the summaries, we
averaged the values of the different items normalizing in such a way that
higher values always mean that the summary is more helpful. In all analyses we
used the non-parametric Mann-Whitney $U$ test and computed effect sizes using
rank-biserial correlation. In all plots the error bars depict the bootstrapped
$95\%$ confidence intervals (Efron and Tibshirani 1994).

Figure 2: Participant Success Percentage per Summary Method. DA:
DISAGREEMENTS; HL: HIGHLIGHTS.

### Experiment 2 - Conveying Agent Differences

This experiment’s objective was to test the usefulness of DISAGREEMENTS
summaries for conveying differences in general agent behavior in comparison to
HIGHLIGHTS.

_Hypotheses._ The DISAGREEMENTS algorithm is designed to portray instances of
disagreement between agents. We hypothesized that this would provide a clear
and contrastive distinction between the alternative agents, thus emphasizing
behavioral differences and providing a more appealing visual experience. More
specifically,we state the following hypotheses:
H3: Success rate of participants shown DISAGREEMENTS summaries will surpass
that of participants shown HIGHLIGHTS summaries.
H4: participants will prefer summaries generated by the DISAGREEMENTS method.

_Experimental Conditions_ A within subject setup was chosen in order to allow
participants to provide a direct comparison between the methods and state
their preferences. As participants experience both HIGHLIGHTS and
DISAGREEMENTS summaries, to reduce cognitive overload, we chose to display
only two agent comparisons for each method. We chose the comparisons between
the most distinctly dissimilar agents, dropping $\langle SD,CL\rangle$ due to
similarity of reward associated with distance from neighboring vehicles.

_Participants._ 45 participants were recruited through Amazon Mechanical Turk
(13 female, mean age $=37.51$, STD $=10.35$), receiving similar pay and bonus
incentive as in the first experiment. As in Exp#1, participants were filtered
out if their task completion time was below a threshold.

_Procedure._ Participants followed a similar procedure as in the previous
experiment diverging solely in the domain introduction and the questions
asked. Instead of superiority between the agents, participants were queried
about which trait was more dominant in the compared agents. The ground truth
was established directly from the reward functions of the agents. Each
comparison included two such questions along with a mandatory confidence
rating and textual explanation of the answers. In addition, participants were
ultimately asked which method they preferred. Since this was a within-subject
design, participants saw both methods, in a random order. All questions are
provided in the Appendix.

_Evaluation Metrics and Analyses._ The main evaluation metric of interest was
the success rate of correctly assigning a more dominant trait to an agent. We
compare this metric across the different agent selection tasks and summary
methods. Similarly to experiment 1, we compare participants’ confidence in
their answers. For evaluating summary method satisfaction of participants we
used the non-parametric Wilcoxon signed-rank test (Wilcoxon 1947) for matched
pairs.

## Results

We now describe the results of our comparison between the HIGHLIGHTS strategy
summarization method and our novel DISAGREEMENTS approach. We report the main
experimental results with respect to the hypotheses raised in the previous
section.

Figure 2 shows the percentage of participants who were successful in answering
the experiment tasks in each of the experiment conditions, for each agent pair
combination.

_(H1) Participants in the HIGHLIGHTS condition struggled to successfully
identify the better performing agent in the comparison task._ When all agents
are of decent performance, we see the difficulty of distinguishing between
them manifest itself in a poor success rate. Based on participants’ textual
explanations of the choice of agent, it seems they were concerned that agent
$E$ was indecisive, e.g., “[Agent $E$] seems very indecisive while … [Agent
$M$] seems to have a plan and is going with it.”. We hypothesize that these
responses are a consequence of a single trajectory in agent $E$’s summary
where the frog is seen leaping between logs in a seemingly indecisive manner.
These results emphasize the limitations of independent comparisons.

_(H2) Participants in the DISAGREEMENTS condition were more successful in the
agent comparison task_. Participants in the DISAGREEMENTS group showed vast
improvement in the ability to identify the better performing Frogger agent
(see Figure 2). The differences in success rate between conditions were
statistically significant and substantial for all agent comparisons ($E$ vs.
$LV$: $p=1.6^{-5}$; $E$ vs. $M$: $p=1.7^{-5}$; $LV$ vs. $M$: $p=0.018$).
Textual explanations provide insights regarding how the contrastive nature of
the DISAGREEMENTS summaries helped participants decide which agent to choose,
e.g. “ I preferred the path that … [Agent $E$] was taking”; “I felt that …
[Agent $E$] was making slightly stronger moves, and pushing ahead further”.

_(H3) A significant difference was found between success rates of participants
in the conveying differences task_. Participants achieved a significantly
higher success rate with the DISAGREEMENTS method in the $FR$ vs. $SD$
comparison ($p=0.014$). Albeit, this was mostly a result of the low
performance of HIGHLIGHTS in $Q1$ due to the $SD$ summary containing,
coincidentally, only trajectories of the agent at the bottom lane. The
inferior performance of DISAGREEMENTS in $Q1$ of $CL$ vs. $FR$ ($p=0.187$) can
be explained by summary trajectories where no vehicles were present in $CL$’s
lane allowing it to drive faster than $FR$ and appear less considerate of
keeping distance. While not necessarily outperforming it, the DISAGREEMENTS is
at least equivalently useful as HIGHLIGHTS, which was shown to be better than
random (Huber et al. 2020).

_(H4) No clear participant preference towards one summary method was
observed._ Most participants answered that both methods were equally
beneficial. However, more participants found DISAGREEMENTS more _helpful_ and
containing less irrelevant information than HIGHLIGHTS, while finding the
latter more _pleasing_.

_Confidence and satisfaction_ In both experiments no statistically significant
differences were found between the confidence or satisfaction of participants
in different conditions (See Appendix).

## Discussion and Future Work

With the maturing of AI, circumstances which require people to choose between
alternative market-available solutions, are likely to arise. The necessity of
distinguishing between alternative agents becomes ever more clear. Moreover,
distinguishing between policies is key for developers when analyzing different
algorithms and configurations.

This paper presented a new approach for comparing RL agents by generating
policy disagreement summaries. Experimental results show summaries help convey
agent behaviour differences and improve users’ ability to identify superior
agents, when one exists.

As for future work, we note the following possible directions: i) expanding
DISAGREEMENTS to enable comparison of more than two agents; ii) testing
additional state and trajectory importance methods; iii) further enhancing the
diversity between trajectories in the summary, and iv) formulating and
defining disagreement “types” for generating further user-specific summaries.

## Acknowledgements

The research was partially funded by the Israeli Science Foundation grant
#2185/20

## References

  * Amir and Amir (2018) Amir, D.; and Amir, O. 2018.  HIGHLIGHTS: Summarizing Agent Behavior to People.  In _Proc. of the 17th International conference on Autonomous Agents and Multi-Agent Systems (AAMAS)_.
  * Amir, Doshi-Velez, and Sarne (2018) Amir, O.; Doshi-Velez, F.; and Sarne, D. 2018.  Agent Strategy Summarization.  In _Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , 1203–1207. International Foundation for Autonomous Agents and Multiagent Systems.
  * Amir, Doshi-Velez, and Sarne (2019) Amir, O.; Doshi-Velez, F.; and Sarne, D. 2019.  Summarizing agent strategies.  _Autonomous Agents and Multi-Agent Systems_ , 33(5): 628–644.
  * Efron and Tibshirani (1994) Efron, B.; and Tibshirani, R. J. 1994.  _An introduction to the bootstrap_.  CRC press.
  * Gottesman et al. (2019) Gottesman, O.; Johansson, F.; Komorowski, M.; Faisal, A.; Sontag, D.; Doshi-Velez, F.; and Celi, L. A. 2019.  Guidelines for reinforcement learning in healthcare.  _Nature medicine_ , 25(1): 16–18.
  * Greydanus et al. (2017) Greydanus, S.; Koul, A.; Dodge, J.; and Fern, A. 2017.  Visualizing and Understanding Atari Agents.  _arXiv preprint arXiv:1711.00138_.
  * Hasselt (2010) Hasselt, H. 2010.  Double Q-learning.  _Advances in neural information processing systems_ , 23: 2613–2621.
  * Heuillet, Couthouis, and Díaz-Rodríguez (2021) Heuillet, A.; Couthouis, F.; and Díaz-Rodríguez, N. 2021.  Explainability in deep reinforcement learning.  _Knowledge-Based Systems_ , 214: 106685.
  * Hoffman et al. (2018) Hoffman, R. R.; Mueller, S. T.; Klein, G.; and Litman, J. 2018.  Metrics for explainable AI: Challenges and prospects.  _arXiv preprint arXiv:1812.04608_.
  * Huang et al. (2017) Huang, S. H.; Held, D.; Abbeel, P.; and Dragan, A. D. 2017.  Enabling robots to communicate their objectives.  _Autonomous Robots_ , 1–18.
  * Huber et al. (2020) Huber, T.; Weitz, K.; André, E.; and Amir, O. 2020.  Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps.  _CoRR_ , abs/2005.08874.
  * Lage et al. (2019) Lage, I.; Lifschitz, D.; Doshi-Velez, F.; and Amir, O. 2019.  Exploring Computational User Models for Agent Policy Summarization.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , 1401–1407. International Joint Conferences on Artificial Intelligence Organization.
  * Leurent (2018) Leurent, E. 2018.  An Environment for Autonomous Driving Decision-Making.  https://github.com/eleurent/highway-env.
  * Miller (2018) Miller, T. 2018.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial Intelligence_.
  * Puiutta and Veith (2020) Puiutta, E.; and Veith, E. M. S. P. 2020.  Explainable Reinforcement Learning: A Survey.  _CoRR_ , abs/2005.06247.
  * Sequeira and Gervasio (2020) Sequeira, P.; and Gervasio, M. 2020.  Interestingness elements for explainable reinforcement learning: Understanding agents’ capabilities and limitations.  _Artificial Intelligence_ , 288: 103367.
  * Sequeira and Gubert (2020) Sequeira, P.; and Gubert, J. 2020.  Frogger python implementation.
  * Wilcoxon (1947) Wilcoxon, F. 1947.  Probability tables for individual comparisons by ranking methods.  _Biometrics_ , 3(3): 119–122.