Joonyong Park1, Shinnosuke Takamichi1, Tomohiko Nakamura1,
Kentaro Seki1, Detai Xin1, Hiroshi Saruwatari1 ††thanks: T. Nakamura is
currently with National Institute of Advanced Industrial Science and
Technology (AIST), Japan.

# How Generative Spoken Language Modeling Encodes Noisy Speech:
Investigation from Phonetics to Syntactics

###### Abstract

We examine the speech modeling potential of generative spoken language
modeling (GSLM), which involves using learned symbols derived from data rather
than phonemes for speech analysis and synthesis. Since GSLM facilitates
textless spoken language processing, exploring its effectiveness is critical
for paving the way for novel paradigms in spoken-language processing. This
paper presents the findings of GSLM's encoding and decoding effectiveness at
the spoken-language and speech levels. Through speech resynthesis experiments,
we revealed that resynthesis errors occur at the levels ranging from phonology
to syntactics and GSLM frequently resynthesizes natural but content-altered
speech.

Index Terms: speech synthesis, noise reduction, speech correction, self-
supervised learning

##  1 Introduction

The field of natural language processing has made significant advancements
[1], particularly in the area of language models. Studies have shown that
language models can naturally learn multiple tasks even without supervision,
which can enable the model to comprehend, answer questions, summarize, and
translate text [2, 3, 4]. Despite the success in text-level language modeling,
it does not handle spoken-language expressions, which is another essential
component of speech. To resolve this problem, investigations on speech-
language models through the use of extensive unlabeled speech data have been
conducted [5, 6, 7, 8, 9, 10].

The most promising approach to this problem is generative spoken language
modeling (GSLM) [11], which involves learning the acoustic and spoken
linguistic features only from speech audio. As shown in Fig. 1(a), GSLM
involves analyzing and synthesizing speech audio through a sequence of
discrete symbols. These symbols can be viewed as pseudo-phones, though they
are not exactingly equivalent to phonemes or phones as those defined in
phonetics [11, 12, 13]. Since these symbols can be determined through
training, GSLM does not require transcriptions associated with speech data,
i.e., it enables text-free spoken language processing [14, 15, 16]. It also
potentially expands the available speech data beyond carefully designed and
annotated laboratory data to real-world data.

However, real-world data usually contain noise and sometimes include speech of
non-target speakers. This may affect the encoding performance in GSLM [17],
posing a challenge for GSLM in being robust to noise contamination. This
motivated us to investigate what information GSLM encodes from noisy speech
and identify which levels of spoken language, ranging from phonetic to
syntactic levels, are affected by noise.

We investigated the encoding and decoding effectiveness of GSLM in a noisy
environment (see Fig. 1(b)). To do this, we conducted experiments on speech
resynthesis from noisy speech using the GSLM model trained with speech without
noise. The experimental results were analyzed at the spoken-language and
speech levels.

##  2 Generative spoken language modeling

Figure 1: GSLM-model architecture and our investigation.

This section outlines the architecture of the GSLM model [11]. Figure 1(a)
shows a schematic illustration of the GSLM-model architecture. The GSLM model
analyzes and synthesizes an audio waveform of speech through discrete symbols
(units) instead of phoneme symbols. The primary components of the GSLM model
are speech2unit and unit2speech modules. The speech2unit module first converts
the audio waveform into framewise features using the encoder of a self-
supervised speech representation model (e.g., contrastive predictive coding
[5], wav2vec2.0 [8], and HuBERT [9]). It then quantizes the features with a
predetermined codebook to obtain the discrete symbol sequence. The codebook is
obtained by applying $k$-means clustering to the framewise features of the
training data. The unit2speech module decodes the discrete symbol sequence
into an audio waveform. For this module, we can use the network architectures
used in conventional text-to-speech synthesis models and neural vocoder models
(e.g., Tacotron2 [18]).

Another important module of the GSLM model is a unit-based language model
(uLM), which is omitted in Figure 1. It serves as a generative language model
in the unit domain and enables speech generation and continuation without
transcription. However, the focus of this syudy was on speech resynthesis,
which does not require the uLM. Thus, we analyze the speech2unit and
unit2speech modules.

##  3 Experimental analysis on noisy speech

###  3.1 Experimental setup

We analyzed the robustness of GSLM to noise contamination through speech
resynthesis experiments. We chose HuBERT [19] with the codebook of 200 classes
as the speech2unit module and Tacotron2 [18] as the unit2speech module, which
achieved the highest performance in word error rate (WER) and mean opinion
score of synthesized speech in [11]. The GSLM model, in which the encoder is
trained on the LibriSpeech corpus [20] and decoder is trained on the LJSpeech
corpus [21] is available via the fairseq toolkit [22].

The noisy speech signals were created by adding noise signals to clean speech
signals with varying signal-to-noise ratios (SNRs) from $0$ to $15$ dB with an
interval of $5$ dB. We used the dev-clean set of the LibriSpeech corpus (2704
utterances) [20] as the clean-speech signals. The noise signals were drawn
from the diverse environments multichannel acoustic noise database (DEMAND)
[23], which contains noise signals recorded with a 16-channel microphone array
in 17 noisy environments. We randomly chose one of the channels for each noise
signal. The number of the test dataset was $183,872$ in total.

To examine the effect of background speech, we classified the noise types into
three categories along with the amount of speech contained in the noise. Table
1 shows the categorization of the noise types. See [23] for details of these
noise types. L-BAB is the collection of the noise types that contain little or
no speech, i.e., low babble noise. M-BAB contains the noise types including
speech in less than half the signal length. H-BAB is the collection of the
rest, i.e., high babble noise. Some of the background speech came from radio,
television, and telephone.

We investigated at spoken-language (Section 3.2) and at speech (Section 3.3).
The former was aimed at exploring which spoken-language content changes due to
noise contamination, while the latter was aimed at finding the relation
between changes in spoken-language levels and those in speech levels. Figure 2
summarizes the levels investigated.

Table 1: Categorization of noise types of DEMAND Category | Noise type of DEMAND
---|---
L-BAB | DKITCHIN, DWASHING, NFIELD,
NRIVER, OHALLWAY, OOFFICE, TCAR
M-BAB | DLIVING, NPARK, PSTATION,
SPSQUARE, STRAFFIC
H-BAB | OMEETING, PCAFETER, PRESTO,
TBUS, TMETRO

###  3.2 Analysis at spoken language levels

We examined the impact of noise on the spoken language level from phonetic to
syntactic levels.

Figure 2: Levels investigated in experiments.

Phone level: Since the discrete symbols of GSLM can be viewed as pseudo-
phones, we defined a counterpart of the WER for the discrete symbol sequences
which we call the unit error rate (UER). The UERs were computed in the same
manner as the WER but using the discrete symbol sequences obtained from the
clean and noisy speech as reference and hypothesis, respectively.

Table 2: UERs [%] of noisy speech | SNR
---|---
| 15 dB | 10 dB | 5 dB | 0 dB
L-BAB | 13.4 | 17.1 | 22.0 | 28.6
M-BAB | 20.2 | 25.9 | 34.3 | 47.1
H-BAB | 21.8 | 29.2 | 40.1 | 54.1

Table 2 shows the average UERs. The UER was above 10 % even under the mildest
condition (SNR of 15 dB and L-BAB) and increased significantly as the SNR
decreased. This indicates that the GSLM model is vulnerable to noisy inputs
when trained with speech without noise. We also observed that background
speech deteriorated encoding performance, which was further worsened by lower
SNR. This observation suggests that the encoder partially encodes background
speech.

Phoneme level: For phoneme-level analysis, we first trained unit-to-phoneme
mappings, such as allophone-to-phoneme many-to-one mappings. An allophone is a
set of phones mapped to a phoneme in a particular language, e.g.,
[$\textrm{p}^{h}$, p] (phones) $\rightarrow$ /p/ (phoneme). The IBM model 2
aligner with monotonic alignment constraints aligns unit sequences of clean
speech and phoneme sequences obtained from texts in the LibriSpeech corpus. We
calculated the probability $P\left(\mathrm{phoneme}|\mathrm{unit}\right)$ and
mapped each unit to its most probable phoneme, e.g., ["128", "52"] (units)
$\rightarrow$ /p/ (phoneme). The phonemizer and aligner used were espeak [24]
and fast_align library [25], respectively. We calculated the phoneme error
rate (PER), just as we calculated the UER in the phone-level evaluation. The
reference and hypothesis are phoneme sequences mapped from clean-speech units
and noisy-speech units, respectively. Repeated symbols were removed from the
sequence to prevent duplication of the same phoneme due to the consecutive
estimation of symbols. We used 40 phonemes after removing the stress marks
from the espeak notation.

Figure 3: $P\left(\mathrm{phoneme}|\mathrm{unit}\right)$. The darker the
color, the more uniquely the unit is mapped by a single phoneme.

Before discussing the PER, we show alignments from units to phonemes. Figure 3
shows the heatmap of the probability
$P\left(\mathrm{phoneme}|\mathrm{unit}\right)$. We can find three trends.
First, some phonemes, e.g., /g, h, j, u, $\int$/, have high probability to
their corresponding phonemes, meaning that these phonemes are more uniquely
mapped to the single phonemes. However, /n, r, s, @, I/ have many darker cells
and mapped from many units. Finally, /q, x, y, / have only brighter cells. One
individual unit is mapped to each of the phonemes, but the probability is low.

Table 3: PERs [%] of noisy speech | SNR
---|---
| 15 dB | 10 dB | 5 dB | 0 dB
L-BAB | 14.8 | 18.1 | 21.9 | 27.2
M-BAB | 20.3 | 25.3 | 32.5 | 43.9
H-BAB | 24.4 | 31.9 | 44.4 | 61.9

Table 3 lists the average PERs with varying SNR. Consistent with the trend
shown in UER, PER increased as SNR decreased and reached closer to H-BAB. In
M-BAB and L-BAB, lower PERs (Table 3) were observed compared with UERs (Table
2). However, the PERs in H-BAB increased regardless of the SNR. Therefore, we
can deduce the following two points: First, some unit errors in L-BAB and
M-BAB are within allophones and negligible at the phoneme level, and second,
unit errors in H-BAB change the phonemes. This is reasonable if the background
speech in H-BAB causes changes in the phoneme level.

Figure 4: Heatmap of confusion matrix at phoneme level. Values are on log10
scale. Only misrecognized phonemes are counted; correctly recognized ones are
not shown in this figure. Noise condition is mildest case: 15 dB SNR of L-BAB
noise.

To discuss the PER results, we computed the confusion matrix of the phonemes.
Specifically, we calculated the matrix by using fast_align to align the
groundtruth and hypothesis phonemes used in the PER calculation. The results
of the confusion matrix are shown in Figure 4. First, focusing on the
groundtruth phonemes (i.e., x-axis), we see that most of the errors are
concentrated in specific phonemes, e.g., /e, æ, r/, etc. Therefore, we can say
that units corresponding to these phonemes entangle the phonemes. Next,
focusing on the error pairs (i.e., the entire matrix), the matrix is very
sparse. Some errors are close in articulation, e.g., /r/ (x-axis)
$\rightarrow$ /l/ (y-axis); however, most errors are not, e.g., vowels
$\Leftrightarrow$ consonants. Therefore, it is likely that the GSLM encoder
captures features that are different from the human articulations.

Word level: For the word-level analysis, we compared the noisy and
resynthesized speech in terms of WER. The references are transcriptions
provided by the LibriSpeech corpus, while the hypotheses are automatic speech
recognition (ASR) results obtained from the noisy/resynthesized speech by
using Whisper-base [26].

Table 4: WERs [%] of raw input (before slash) and resynthesized (after slash) speech per SNR. Values of CLEAN are for clean speech | SNR
---|---
| 15 dB | 10 dB | 5 dB | 0 dB
CLEAN | 4.3/14.5
L-BAB | 5.4/15.3 | 5.5/16.5 | 5.4/20.3 | 7.9/27.1
M-BAB | 5.4/17.1 | 6.0/20.8 | 7.7/30.9 | 12.8/52.5
H-BAB | 5.8/19.8 | 8.5/28.6 | 11.4/49.0 | 25.5/75.8

Table 4 lists the average WERs of the noisy and resynthesized speech. Since
Whisper had recognition errors on some clean speech, we computed WERs for the
clean speech and their resynthesized versions for reference. Comparing the
input and resynthesized speech, the difference in WER was above ten points
($4.3\rightarrow 14.5$) for the reference. Compared with the reference, the
WERs for the noisy inputs increased only a few points in L-BAB and at the SNRs
of 10 and 15 dB. The results indicate that GSLM resynthesis dominates in word-
level degradation rather than mild non-speech noise contamination.

The WER scores increased significantly under noise conditions with more
background speech and lower SNRs, similar to the results in UER and PER. The
scores also apparently increased, along with the decrease in SNR and the
increase in background speech. These results indicate that noise contamination
(particularly, background speech) may change the spoken content in GSLM
resynthesis.

Figure 5: Scatterplots between word frequency and $\Delta$WER per SNR. Blue
lines are regression lines.

For a more detailed analysis, we investigated the relationship between word
usage frequency in the corpus and WER. To remove the effects of speech
recognition performance, we defined a normalized WER ($\Delta$WER) by
subtracting the WERs of the CLEAN noise category from those of L-BAB, M-BAB,
and H-BAB for the resynthesized speech.

Figure 5 shows scatterplots between word frequencies and $\Delta$WER per SNRs.
The $\Delta$WERs were negatively correlated with word frequency, which became
more pronounced as SNR decreased. This result indicate that word substitutions
are more likely to occur for less frequent words; for example, words such as
``the'' appeared the most in the corpus and showed a WER of 3.3 %, but words
such as ``ill'' and ``law'', which have the same number of phonemes appeared
about $100$ times in the entire corpus, and showed a WER of around 40 %. We
can also observe that both the slope and bias of the regression line increase
as the SNR decreases. This indicates that noise contamination not only
degrades the overall performance of word recognition but also worsens the
recognition of rare words.

Syntactic level: To quantify syntactic errors by resynthesis, we used a WER
counterpart for word class, i.e., part-of-speech, which we call the word class
error rate (WCER). The references and hypotheses are the word class sequences
extracted from the texts of the LibriSpeech corpus and the speech recognition
results, respectively. The word classes were estimated using the natural
language toolkit [27].

Table 5: WCERs [%] of raw input (before slash) and resynthesized (after slash) speech per SNR. Values of CLEAN are for clean speech | SNR
---|---
| 15 dB | 10 dB | 5 dB | 0 dB
CLEAN | 3.1/11.7
L-BAB | 4.0/12.3 | 4.0/13.1 | 4.5/16.1 | 5.7/21.6
M-BAB | 3.9/13.6 | 4.4/16.5 | 5.8/24.4 | 9.9/41.8
H-BAB | 4.3/15.6 | 6.5/23.1 | 8.8/39.6 | 20.6/61.8

Table 5 lists the average WCERs of the noisy and resynthesized speech per SNR.
We observe a similar trend in WCER as in UER, PER, and WER. Regardless of the
SNRs and noise category, the WCERs were slightly lower than the WERs but still
large. This result indicates that the word substitution caused by the
resynthesis from the noisy speech frequently changes the word class.

###  3.3 Analysis at speech level

We next evaluated GSLM at speech levels by quantifying the resynthesis quality
of GSLM under noisy environments. For this evaluation, metrics for measuring
codec distortion and resynthesized speech quality were used. The former metric
is within speech levels, while the latter is a comparison between speech and
spoken language levels.

Codec distortion: Since the discrete symbol sequence is a compressed speech
representation, GSLM can be viewed as a neural speech codec. Thus, we
investigated the distortion between an input speech and its resynthesized
version, which we call _codec distortion_. The codec distortion was measured
using WARP-Q, which was developed to predict the quality of neural speech
codecs [28]. It internally compensates for the temporal discrepancy between
the ground truth and resynthesized signal, and is robust to the changes in
pronunciation timing after resynthesis that often occur in neural speech
codecs. It ranges from one to five points and increases as the codec
distortion decreases. We used clean speech as the ground truths for computing
WARP-Q.

Figure 6 shows the distributions of the WARP-Q values for noisy and
resynthesized speech111We treated the additive noise as a form of codec
distortion and computed the WARP-Q values for the noisy speech.. The WARP-Q
improved from the resynthesis as the SNR decreased and the background speech
increased which is consistent with the results in Section 3.2. The WARP-Q
values of the noisy speech were weakly or not correlated with those of the
resynthesized speech under all noise conditions. These results indicate that
GSLM resynthesis on noisy speech decrease codec distortion when speech is
included in the noise, but does not mitigate distortion in general.

Naturalness: To evaluate the naturalness of the resynthesized speech, we used
a pre-trained UTMOS model [29] strong learner [30]. It predicts a five-scale
pseudo-mean opinion score (MOS) value from a synthesized speech222Although the
UTMOS was trained with English/Chinese synthesized speech of the Blizzard
Challenge or Voice Conversion Challenge, and works robustly for speech
synthesized in other languages and with speech synthesis systems [31].. Unlike
WARP-Q, UTMOS does not require clean speech for computation and is agnostic to
spoken-linguistic changes between clean and resynthesized speech. Therefore,
we investigated the relation between linguistic changes ($\Delta$WER) and
speech quality (UTMOS).

Figure 6: Scatterplots of noisy (raw) and resynthesized (resyn) speech in
terms of WARP-Q at various SNRs. Region above diagonal line indicates WARP-Q
improvement via GSLM resynthesis. Figure 7: Scatterplots of $\Delta$WERs and
MOS values by UTMOS for resynthesized speech. Blue and orange lines are
regression lines for L-BAB and H-BAB, respectively.

Figure 7 shows the sentence-by-sentence relationship between $\Delta$WER and
values by UTMOS. The values by UTMOS were distributed around 3.8 points, and
their average decreased as SNR decreased. This result is consistent with the
observation in the literature where the discrete speech representation is
applied to speech separation/enhancement [32]. There were either weak
correlations or none between $\Delta$WER and UTMOS values under all noise
conditions, showing that GSLM frequently converts noisy speech into natural
but content-altered speech. L-BAB does not degrade the UTMOS value but worsens
the WER.

##  4 Conclusion

We assessed the GSLM performance for noisy speech at the spoken-language and
speech levels through speech-resynthesis experiments. The analysis at the
spoken-language levels shows that the noise contamination strongly affects
resynthesis performance and promotes frequent changes in phones, phonemes,
words, and word classes. It also suggests that the GSLM encoder captures
different speech features from human articulations. The analysis at the speech
levels reveals that GSLM frequently resynthesizes natural but content-altered
speech and has the risk of falsification if GSLM is used in real (noisy)
situations without any special care.

Acknowledgments. This work is supported by JSPS KAKENHI 22H03639, 23H03418,
and Moonshot R&D Grant Number JPMJPS2011.

## References

  * [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, ``Attention is all you need,'' in _Proceedings of Advances in Neural Information Processing Systems_ , 2017, pp. 6000–6010.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``BERT: Pre-training of deep bidirectional Transformers for language understanding,'' in _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics_ , vol. 1, 2019, pp. 4171–4186.
  * [3] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, ``BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,'' in _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 7871–7880.
  * [4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, ``Language models are few-shot learners,'' in _Proceedings of Advances in Neural Information Processing Systems_ , vol. 33, 2020, pp. 1877–1901.
  * [5] A. van den Oord, Y. Li, and O. Vinyals, ``Representation learning with contrastive predictive coding,'' arXiv preprint, arXiv:1807.03748, 2018\.
  * [6] Y.-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, ``An unsupervised autoregressive model for speech representation learning,'' in _Proceedings of INTERSPEECH_ , 2019, pp. 146–150.
  * [7] A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee, ``Mockingjay: Unsupervised speech representation learning with deep bidirectional Transformer encoders,'' in _Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2020, pp. 6419–6423.
  * [8] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' in _Proceedings of Advances in Neural Information Processing Systems_ , vol. 33, 2020, pp. 12 449–12 460.
  * [9] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [10] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, ``WavLM: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1505–1518, 2022.
  * [11] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed, and E. Dupoux, ``On generative spoken language modeling from raw audio,'' _Transactions of the Association for Computational Linguistics_ , vol. 9, pp. 1336–1354, 2021.
  * [12] D. Wells, H. Tang, and K. Richmond, ``Phonetic Analysis of Self-supervised Representations of English Speech,'' in _Proceedings of INTERSPEECH_ , 2022, pp. 3583–3587.
  * [13] M. de Seyssel, M. Lavechin, Y. Adi, E. Dupoux, and G. Wisniewski, ``Probing phoneme, language and speaker information in unsupervised speech representations,'' in _Proceedings of INTERSPEECH_ , 2022, pp. 1402–1406.
  * [14] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, ``AudioLM: A language modeling approach to audio generation,'' arXiv preprint, arXiv:2209.03143, 2022.
  * [15] A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, ``Speech resynthesis from discrete disentangled self-supervised representations,'' arXiv preprint, arXiv:2104.00355, 2021\.
  * [16] J. Ao, R. Wang, L. Zhou, C. Wang, S. Ren, Y. Wu, S. Liu, T. Ko, Q. Li, Y. Zhang, Z. Wei, Y. Qian, J. Li, and F. Wei, ``SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing,'' arXiv preprint, arXiv:2110.07205, 2021.
  * [17] G. Itai, K. Felix, L. Ann, C. Jade, S. Gabriel, D. Emmanuel, and A. Yossi, ``On the robustness of self-supervised representations for spoken language modeling,'' arXiv preprint, arXiv:2209.15483, 2022.
  * [18] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu, ``Natural TTS synthesis by conditioning WaveNet on mel-spectrogram predictions,'' in _Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2018, pp. 4779–4783.
  * [19] W.-N. Hsu, Y.-H. H. Tsai, B. Bolte, R. Salakhutdinov, and A. Mohamed, ``Hubert: How much can a bad teacher benefit ASR pre-training?'' in _Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2021, pp. 6533–6537.
  * [20] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``LibriSpeech: An ASR corpus based on public domain audio books,'' in _Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2015, pp. 5206–5210.
  * [21] K. Ito and L. Johnson, ``The LJ speech dataset,'' https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [22] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, ``fairseq: A fast, extensible toolkit for sequence modeling,'' in _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics_ , 2019, pp. 48–53.
  * [23] J. Thiemann, N. Ito, and E. Vincent, ``DEMAND: A collection of multi-channel recordings of acoustic noise in diverse environments,'' https://doi.org/10.5281/zenodo.1227121, 2013.
  * [24] ``eSpeak NG Text-to-Speech,'' https://github.com/espeak-ng/espeak-ng, 2006.
  * [25] C. Dyer, V. Chahuneau, and N. A. Smith, ``A simple, fast, and effective reparameterization of IBM model 2,'' in _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics_ , 2013, pp. 644–648.
  * [26] A. Radford, K. Jong Wook, X. Tao, B. Greg, M. Christine, and S. Ilya, ``Robust speech recognition via large-scale weak supervision.'' arXiv preprint, arXiv:2212.04356, 2022.
  * [27] S. Bird, E. Klein, and E. Loper, _Natural language processing with Python: Analyzing text with the natural language toolkit_. O'Reilly Media, Inc., 2009.
  * [28] W. A. Jassim, J. Skoglund, M. Chinen, and A. Hines, ``Speech quality assessment with WARP-Q: From similarity to subsequence dynamic time warp cost,'' _IET Signal Processing_ , vol. 16, no. 9, pp. 1050–1070, 2022.
  * [29] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, ``UTMOS: UTokyo-SaruLab system for VoiceMOS challenge 2022,'' in _Proceedings of INTERSPEECH_ , 2022, pp. 4521–4525.
  * [30] ``UTMOS: UTokyo-SaruLab MOS prediction system,'' https://github.com/sarulab-speech/UTMOS22, 2022.
  * [31] K. Seki, T. Shinnosuke, S. Takaaki, and H. Saruwatari, ``Text-to-speech synthesis from dark data with evaluation-in-the-loop data selection,'' in _Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing_ , 2023, (to appear, arXiv:2210.14850).
  * [32] J. Shi, X. Chang, T. Hayashi, Y.-J. Lu, S. Watanabe, and B. Xu, ``Discretization and re-synthesis: an alternative method to solve the cocktail party problem,'' arXiv preprint, arXiv:2112.09382, 2021.


# Multi-State TD Target for Model-Free Reinforcement Learning

Wuhao Wang, Zhiyong Chen, and Lepeng Zhang W. Wang and Z. Chen are with the
School of Engineering, The University of Newcastle, Callaghan, NSW 2308,
Australia. L. Zhang is with the Department of Computer and Information
Science, The Linköping University, 581 83 Linköping, Sweden.

###### Abstract

Temporal difference (TD) learning is a fundamental technique in reinforcement
learning that updates value estimates for states or state-action pairs using a
TD target. This target represents an improved estimate of the true value by
incorporating both immediate rewards and the estimated value of subsequent
states. Traditionally, TD learning relies on the value of a single subsequent
state. We propose an enhanced multi-state TD (MSTD) target that utilizes the
estimated values of multiple subsequent states. Building on this new MSTD
concept, we develop complete actor-critic algorithms that include management
of replay buffers in two modes, and integrate with deep deterministic policy
optimization (DDPG) and soft actor-critic (SAC). Experimental results
demonstrate that algorithms employing the MSTD target significantly improve
learning performance compared to traditional methods.

###### Index Terms:

Reinforcement learning, temporal difference, actor-critic learning, state-
action value, Q-value

##  I Introduction

Deep reinforcement learning consistently grapples with various challenges,
notably inaccurate Q-function estimation and delayed rewards. These obstacles
often result in suboptimal policies and sluggish convergence rates, posing
significant hurdles to the efficacy of reinforcement learning algorithms. To
address the former challenge, researchers have proposed diverse techniques
within existing algorithmic frameworks. For instance, off-policy algorithms
like deep deterministic policy gradient (DDPG) employ target networks to
stabilize value estimations during training. Similarly, soft actor-critic
(SAC) integrates entropy regularization to foster exploration and prevent
premature convergence, while twin delayed DDPG (TD3) introduces twin critics
and policy smoothing to enhance stability and sample efficiency. As for the
latter challenge, researchers have advocated for multi-step reinforcement
learning, which accounts for returns over multiple time steps. This approach
better accommodates delayed rewards, thereby enhancing the agent’s performance
within the environment.

In model-based reinforcement learning, multi-step greedy reinforcement
learning is widely employed, leveraging the environment model to predict
future states and inform action selection [1]. Conversely, in model-free
reinforcement learning, numerous multi-step methods compute Q-values by
considering rewards and final actions across extended trajectories. These
approaches are applied in various domains, including image processing and
energy management, showcasing their versatility and effectiveness in improving
learning efficiency and performance [2], [3], [4], [5].

Existing multi-step reinforcement learning algorithms typically focus on
processing reward signals while considering the Q-function of a single
subsequent state. To overcome this limitation, we introduce a novel approach,
multi-state TD target (MSTD), which enables the agent to simultaneously
integrate multi-step reward signals and Q-values from various states, thereby
enhancing performance. To facilitate updates based on MSTD, we propose two
modes for managing the replay buffer: action-loaded and action-generated. The
former emphasizes past experiences, aiding Q-value accuracy in fitting the
data within the replay buffer, while the latter permits the agent to reselect
actions from past experiences, ensuring Q-value estimation aligns more closely
with the current policy.

We integrated MSTD into DDPG and SAC and conducted experiments in virtual
environments, including Walker-v4, HalfCheetah-v4, and Hopper-v4 [6]. The
results demonstrate a significant improvement in learning performance. Our
contributions can be summarized as follows:

  1. 1.

Introduction of the MSTD framework, incorporating the multi-state replay
buffer and multi-state objective functions.

  2. 2.

Theoretical and experimental analysis of alignment with actor-critic
algorithms and the maximum entropy framework.

  3. 3.

Introduction of two modes: action-loaded and action-generated, with
experimental comparisons between them.

##  II Related Work

Kristopher [7] introduced a method by incorporating $Q_{\phi}(\sigma)$ into
function approximation to sample varying degrees between Sarsa and Expected
Sarsa. Experimental results demonstrate that intermediate values of $\sigma$
yield better results compared to both algorithms. However, the experiments in
this study are limited to on-policy prediction and control problems.

Rainbow [8] integrates existing advancements in reinforcement learning,
including multi-step settings, arguing that multi-step learning not only
enhances final rewards but also accelerates learning in the early stages.
However, their research primarily focuses on discrete action spaces, whereas
our work investigates continuous control tasks.

Lingheng [4] explores multi-step reinforcement learning in continuous control
tasks by combining multi-step settings with DDPG. However, their algorithm
only aggregates rewards from multiple steps, neglecting intermediate
information from middle state-action pairs, thus adhering to conventional
multi-step reinforcement learning.

Yuan and Yu [9] propose averaging $n$ standard return functions, each
representing a sum of discounted reward signals from different time intervals.
Their method mitigates reward hacking by emphasizing further rewards to the
agent. Their experiments focus on grid worlds with simple discrete actions and
states, and their model’s horizon is constrained to a single state.

Multi-step settings are also applied to model-based algorithms. Model-based
value expansion (MVE) [1] is a multi-step method that applies multi-step
learning on a learned environment model. Although the algorithm is model-free,
the environment model requires additional calculations, and accurately
modeling a realistic environment can be challenging.

Enhanced algorithms like stochastic ensemble value expansion (STEVE) [10]
extend various multi-step Q-value functions to achieve better performance in
complex environments. However, using ensemble environment models, which
include transition dynamics and reward functions, involves extra computation.
Model-based planning policy learning with multi-step plan value estimation
(MPPVE) [11] collects k-step model rollouts, where the start state is real and
others are generated by the model. However, only the gradient on the start
state is computed, reducing computation pressure and alleviating model error.
This approach, however, cannot be applied to model-free algorithms as they
require an environment model.

##  III Preliminaries

We consider a standard RL setup according to [12]. At each timestep $t$, the
agent receives an observation $\mathbf{s}_{t}$ from the environment, takes an
action $\mathbf{a}_{t}$, and obtains a scalar reward $\mathbf{r}_{t+1}$
together with the next observation $\mathbf{s}_{t+1}$. For an infinite Markov
decision process (MDP) defined by
$(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$, $\mathcal{S}$,
$\mathcal{A}$, and $\mathcal{R}$ are the sets of $\mathbf{s}_{t}$,
$\mathbf{a}_{t}$, and $\mathbf{r}_{t+1}$ respectively.
$\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$
is the transition probability function. $\gamma\in(0,1]$ is the discount
factor that balances the future and current reward.

The accumulated expected reward $\mathbf{G}_{T}$:

$\displaystyle\mathbf{G}_{T}=\sum_{t=0}^{T}\gamma^{t}\mathbf{r}_{t},$ (1)

where $T$ is the final time step in an episodic task and $T=\infty$ when a
task is continuous. An agent’s behavior is defined by a policy $\pi$, which
maps the observation $\mathbf{s}_{t}$ to action $\mathbf{a}_{t}$. The state-
action value function $Q^{\pi}(s,a)$ defined below:

$\displaystyle Q^{\pi}(s,a)$
$\displaystyle=\underset{\mathbf{s}_{t}\sim\mathcal{P},\mathbf{a}_{t}\sim\pi}{\mathbb{E}}[\mathbf{r}_{t+1}+\gamma\mathbf{r}_{t+2}+\cdots\mid
s=\mathbf{s}_{t},a=\mathbf{a}_{t}].$ (2)

According to the algorithms, the behavior policy can be deterministic or
stochastic. An RL agent aims to find the optimal policy $\pi^{*}$ to maximize
$Q^{\pi}(s,a)$.

Under policy gradient and actor-critic style, the behavior policy is defined
as actor $\pi_{\theta}$, parameterized by $\theta$. Normally, a deterministic
actor will output a scalar for each dimension of action, whereas a stochastic
actor will generate the parameters of a certain distribution to sample an
action. The critic, parameterized by $\phi$, estimates the state-action value
function $Q^{\pi}_{\phi}(s,a)$.

In the following text, we will use
$Q^{\pi}_{\phi}(\mathbf{s}_{t},\mathbf{a}_{t})$ to indicate the state-action
value function at time $t$. The temporal difference (TD) method, which allows
the gradient update to happen when a task is not finished, is applied to
accelerate convergence and reduce variance, the corresponding TD target is:

$\displaystyle\operatorname{TD}$ $\displaystyle=\mathbf{r}_{t+1}+\gamma
Q^{\pi}_{\phi}(\mathbf{s}_{t+1},\mathbf{a}_{t+1}).$ (3)

This target determines the state-action function by considering its value at
the subsequent one-step and the intermediate reward. It implies a TD error,
$\operatorname{TD}-Q^{\pi}_{\phi}(\mathbf{s}_{t},\mathbf{a}_{t})$, to update
the value function as follows:

$\displaystyle
Q^{\pi}_{\phi}(\mathbf{s}_{t},\mathbf{a}_{t})\leftarrow(1-\alpha)Q^{\pi}_{\phi}(\mathbf{s}_{t},\mathbf{a}_{t})+\alpha\operatorname{TD}.$
(4)

for a learning rate $\alpha$.

Recent literature extends this one-step TD target into a multi-step rule, as
shown below:

$\displaystyle\operatorname{TD}$
$\displaystyle=\sum_{i=1}^{L}\gamma^{i-1}\mathbf{r}_{t+i}+\gamma^{L}Q^{\pi}_{\phi}(\mathbf{s}_{t+L},\mathbf{a}_{t+L}),$
(5)

where $L$ is the step size. When $L=1$, the multi-step method degrades to
single-step reinforcement learning. For an episodic task or task with limited
steps, where the maximum step $T$ exists, the multi-step update rule for time
interval $T-L<t\leq T$ is different. If $t>T-L$, a truncated return function
is applied:

$\displaystyle\operatorname{TD}$
$\displaystyle=\mathbf{r}_{t+1}+\gamma\mathbf{r}_{t+2}+\cdots+\gamma^{T-t-1}\mathbf{r}_{T-1}.$
(6)

This could be done by setting the element in (5) whose time index is greater
than $L$ to zero. Meanwhile, this update can not happen when the length of the
collected trajectory is shorter than $L$. It is noted that in some literature
such as [9], the multi-step intermediate reward
$\sum_{i=1}^{L}\gamma^{i-1}\mathbf{r}_{t+i}$ in the definition of (5) takes
different weighted summation form.

##  IV Multi-state Temporal Difference

In both the single-step TD (3) and the multi-step version (5), the TD target
is calculated based on the Q-value at one state, either $\mathbf{s}_{t+1}$ or
$\mathbf{s}_{t+L}$, and the corresponding action. Therefore, they are referred
to as single-state TD methods. The proposed TD target is based on the Q-value
at multiple states. The idea is motivated by averaging the TD targets defined
for various step sizes, that is,

$\displaystyle\operatorname{TD}$
$\displaystyle=\frac{1}{L}\sum_{l=1}^{L}\left\\{\sum_{i=1}^{l}\gamma^{i-1}\mathbf{r}_{t+i}+\gamma^{l}Q^{\pi}_{\phi}(\mathbf{s}_{t+l},\mathbf{a}_{t+l})\right\\}$
$\displaystyle=\frac{1}{L}\sum_{l=1}^{L}\sum_{i=1}^{l}\gamma^{i-1}\mathbf{r}_{t+i}+\frac{1}{L}\sum_{l=1}^{L}\gamma^{l}Q^{\pi}_{\phi}(\mathbf{s}_{t+l},\mathbf{a}_{t+l}).$
(7)

This multi-state TD (MSTD) target requires the $Q$-values at multiple states
$\mathbf{s}_{t+l}$, $l=1,\cdots,L$, and the corresponding actions. We aim to
study how this new MSTD can be used for value function update as in (4) and
its advantages compared with the conventional (3) or (5).

The objective for training the neural network is defined as:

$\displaystyle\min_{\phi}J(\phi)=\frac{1}{N}\sum_{i=1}^{N}\left[Q^{\pi}_{\phi}(\mathbf{s}^{i}_{t},\mathbf{a}^{i}_{t})-\operatorname{TD}^{i}\right]^{2},$
(8)

where each pair $(\mathbf{s}^{i}_{t},\mathbf{a}^{i}_{t})$ is a sample from the
replay buffer, $\operatorname{TD}^{i}$ is the corresponding TD target, $i$
indicates the sample index, and $N$ is the batch size. Since the convergence
of conventional n-step reinforcement learning is guaranteed, our method based
on the multi-state TD target can also converge to the optimal policy, noting
that the gradient in this case is the average of the gradients in conventional
multi-step TD, as derived from (IV).

Using the conventional multi-step (single-state) TD target defined in (5), we
require
$(\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{r}_{t+1},\cdots,\mathbf{r}_{t+L},\mathbf{s}_{t+L},\mathbf{a}_{t+L})$
for each sample $i$ to complete the calculation (8). Since $\mathbf{a}_{t+L}$
is generated by the actor according to the current policy, the sample data in
the replay buffer takes the tuple structure
$(\mathcal{S},\mathcal{A},\mathcal{R}_{1},\mathcal{R}_{2},\cdots,\mathcal{R}_{L},\mathcal{S}_{\operatorname{next}})$,
where $\mathcal{S}_{\operatorname{next}}$ indicates the next state that
involves in TD calculation (if $\mathcal{S}$ is recorded at time step $t$,
then $\mathcal{S}_{\operatorname{next}}$ will be recorded at time step $t+L$).
The design of a replay buffer and the corresponding training algorithms for
the new MSTD in (IV) becomes more complex as it requires more state/action
pairs to complete the calculation in (8). These aspects will be discussed in
the subsequent section.

##  V Algorithms

In this section, we investigate the design of a replay buffer and a training
scheme for MSTD within actor-critic settings. We introduce two types of modes:
action-loaded and action-generated. To illustrate these algorithms, we provide
pseudocode based on DDPG with MSTD as a case study. Additionally, we discuss
the integration of the soft actor-critic algorithm with MSTD.

###  V-A Maintenance of a Replay Buffer

The MSTD method, as described in (IV), takes into account the Q-values of
state-action pairs over an extended trajectory. To efficiently compute (8)
using MSTD, we need to store the sequences

$\displaystyle(\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{r}_{t+1},\cdots,\mathbf{s}_{t+L-1},\mathbf{a}_{t+L-1},\mathbf{r}_{t+L},\mathbf{s}_{t+L},\mathbf{a}_{t+L})$
(9)

for each sample $i$ in the replay buffer. This storage is necessary to capture
the detailed interaction information between the agent and the environment
over the trajectory.

Since $\mathbf{a}_{t+L}$ is generated by the actor according to the current
policy, the sample data in the replay buffer follows the structure
$(\mathcal{S}_{0},\mathcal{A}_{0},\mathcal{R}_{1},\cdots,\mathcal{S}_{L-1},\mathcal{A}_{L-1},\mathcal{R}_{L},\mathcal{S}_{L})$.
To maintain and update samples in this fixed-length structure, we continuously
collect tuples
$(\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{S}_{\operatorname{next}})$
during training. Each time a new tuple is collected, we delete the oldest
triplet $(\mathcal{S}_{0},\mathcal{A}_{0},\mathcal{R}_{1})$ from the sample
and append the new tuple to the end with $\mathcal{S}=\mathcal{S}_{L}$. This
process ensures that the sample is continuously updated. Figure 1 illustrates
this process.

It is noteworthy that when the agent’s episode terminates at time $T$, no more
tuples are collected. In this scenario, we repeatedly append the last tuple,
known as the termination tuple, to the sample using the same updating process.
This repetition continues for $L$ updates until the termination tuple fills
the sample. These repetitive tuples are marked with a termination flag,
indicating that they do not contribute to the gradient computation. This
operation helps the agent learn from the final states of the episode, as
mentioned in (6).

Figure 1: Management of sample structure in a replay buffer.

###  V-B Action-loaded and Action-generated Modes

In the sequences (9), it is observed that the action $\mathbf{a}_{t+L}$ is
generated by the actor according to the current policy, while the intermediate
actions $\mathbf{a}_{t+1},\cdots,\mathbf{a}_{t+L-1}$ are saved in the buffer
and loaded for the calculation of (8). This operation is called an action-
loaded mode. Therefore, the average of the Q-values in (IV) is specifically
expressed as:

$\displaystyle\frac{1}{L}\left\\{\sum_{l=1}^{L-1}\gamma^{l}Q^{\pi}_{\phi}(\mathbf{s}_{t+l},\mathbf{a}_{t+l})+\gamma^{L}Q^{\pi}_{\phi}(\mathbf{s}_{t+L},\pi_{A}(\mathbf{s}_{t+L}))\right\\}$
(10)

where $\pi_{A}(\mathbf{s}_{t+L})$ indicates that this action is generated by
the actor according to the policy $\pi_{A}$.

Alternatively, similar to the action $\mathbf{a}_{t+L}$, the other actions
$\mathbf{a}_{t+1},\cdots,\mathbf{a}_{t+L-1}$ can also be generated by the
actor, rather than being saved in the buffer. This operation is called an
action-generated mode, and the average of the Q-values in (IV) is specifically
expressed as:

$\displaystyle\frac{1}{L}\sum_{l=1}^{L}\gamma^{l}Q^{\pi}_{\phi}(\mathbf{s}_{t+l},\pi_{A}(\mathbf{s}_{t+l})).$
(11)

Figure 2 illustrates the structures of these two modes.

Figure 2: Two modes of sample structure in a replay buffer.

These two modes of operation exhibit different emphases on empiricism and
rationalism in multi-state reinforcement learning. The action-loaded mode
emphasizes the use of collected data, with the computed results fully
reflecting the accumulated experience. On the other hand, the action-generated
mode allows the most recent policies to participate in gradient updating. We
hypothesize that the policy network will make more accurate decisions as it is
trained further. Consequently, this mode leads to the acquisition of
Q-functions that are more closely aligned with the true values.

###  V-C Action-loaded DDPG with MSTD

In this section, we present a DDPG algorithm with action-loaded MSTD as a case
study. The algorithm is abbreviated as MSDDPG. The objective for the actor in
this algorithm is the same as DDPG:

$\displaystyle
J_{\pi}(\theta)=-Q^{\pi}_{\phi}(\mathbf{s}_{t},\pi(\mathbf{s}_{t})),$ (12)

where $Q$ and $\pi$ are the Q-network and policy-network. The details can be
found in Algorithm 1

Algorithm 1 Action-Loaded MSDDPG

0: Environment with state space $\mathcal{S}$ and action space $\mathcal{A}$

0: Learned policy $\pi$ and Q-value function $Q$

1: Initialize policy network $\pi(s|\theta)$ and Q networks $Q(s,a|\phi)$ with
random weights

2: Initialize target networks $\pi^{\prime}$ and $Q^{\prime}$ with weights
$\theta^{\prime}\leftarrow\theta$, $\phi^{\prime}\leftarrow\phi$

3: Initialize multi-state replay buffer $\mathcal{B}$

4: Set hyperparameters: discount factor $\gamma$, target smoothing parameter
$\tau$, exploration noise $\mathcal{N}$, step size $L$, number of total
training episodes $M$, batch size $B$

5: for $episode=1$ to $M$ do

6: Receive random initial observation state $\mathbf{s}_{0}$

7: while Not Terminated do

8: Select action $\mathbf{a}_{t}=\pi(\mathbf{s}_{t}|\theta)+\mathcal{N}_{t}$
according to the current policy and exploration noise;

9: Execute $\mathbf{a}_{t}$ in the environment and observe next state
$\mathbf{s}_{t+1}$ and reward $\mathbf{r}_{t+1}$;

10: Append $(\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{r}_{t},\mathbf{s}_{t+1})$
to $\mathcal{B}$ as illustrated in Figure 1;

11: if Len($\mathcal{B}$) $\geq$ $B$ then

12: Sample a minibatch of $N$ transitions from $\mathcal{B}$;

13: Calculate TD target and objective for Q network according to (IV) and (8);

14: Calculate the objective for the policy network according to (12);

15: update the policy network and the Q network;

16: Update the target networks: $\theta^{\prime};\leftarrow\theta$,
$\phi^{\prime}\leftarrow\phi$

17: end if

18: end while

19: Adding final-states-samples to replay buffer as illustrated in SectionV-A;

20: end for

###  V-D Integration with Maximum Entropy

The MSTD method can also be integrated with the soft actor-critic (SAC)
algorithm. In SAC, the Q-function includes an additional entropy term to
encourage exploration, defined as follows:

$Q_{\phi}^{s\pi}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)=Q^{\pi}_{\phi}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)-\alpha\log\pi\left(\mathbf{a}_{t}\mid\mathbf{s}_{t}\right).$
(13)

By incorporating this Q-function into MSTD (IV), we can derive a soft MSTD
target.

The SAC algorithm integrated with MSTD, abbreviated as MSSAC, can be
implemented in both action-loaded and action-generated modes. In the action-
loaded mode, to maintain the consistency of the replay buffer, we do not store
the action entropy in the replay buffer. Thus, in practice, action-loaded
MSSAC only introduces the entropy term in the last Q-value function.
Consequently, the average of the Q-values in (IV) is modified to:

$\displaystyle\frac{1}{L}\left\\{\sum_{l=1}^{L-1}\gamma^{l}Q^{\pi}_{\phi}(\mathbf{s}_{t+l},\mathbf{a}_{t+l})+\gamma^{L}Q^{s\pi}_{\phi}(\mathbf{s}_{t+L},\pi_{A}(\mathbf{s}_{t+L}))\right\\}.$
(14)

In the action-generated mode, the entropy term is included in all the Q-value
functions in (IV), and the average of the Q-values is modified to:

$\displaystyle\frac{1}{L}\sum_{l=1}^{L}\gamma^{l}Q^{\pi}_{s\phi}(\mathbf{s}_{t+l},\pi_{A}(\mathbf{s}_{t+l})).$
(15)

##  VI Experiments

We conducted experiments in the Gymnasium virtual environment [6]
HalfCheetah-v4 and Walker-v4, using vectorized observations as state inputs.
The environment HalfCheetah-v4 involves a two-legged robot moving forward in a
2D plane to maximize speed while keeping stable. The state space is a
17-dimensional vector including the robot’s position, velocity, and joint
angles. Walker-v4 features a bipedal robot tasked with walking forward in a 2D
plane to achieve maximum forward speed. The state space has a 17-dimensional
vector that captures the robot’s position, velocities, and joint angles. The
results are averaged over 15 random seeds. In this section, we aim to
illustrate three points:

  1. 1.

The effectiveness of the multi-state setting compared to original algorithms
and conventional multi-step algorithms (5).

  2. 2.

The comparison between action-loaded and action-generated settings.

  3. 3.

The comparison between different step sizes.

We employed the default hyperparameters outlined in the respective papers for
DDPG and SAC, maintaining consistency for MSDDPG and MSSAC across both action-
generated and action-loaded settings. Additionally, we compared the results
with conventional multi-step algorithms based on the TD target (5), denoted as
MPDDPG and MPSAC, respectively. To ensure a fair comparison, each model
underwent training on the same device for an equal number of training steps.

The empirical results demonstrate that both action-generated and action-loaded
MSDDPG, as well as MSSAC, exhibit superior performance compared to both
conventional multi-step and original algorithms in the HalfCheetah-v4
environment. Notably, in the Walker-v4 environment, MSDDPG achieved the
highest score, while MSSAC, particularly in the action-loaded setting,
surpassed other algorithms.

We also observed that the benefits of using action-loaded and action-generated
settings are environment-specific. Specifically, in the Walker-v4 environment,
the action-loaded setting consistently outperforms the action-generated
setting, whereas the opposite trend is observed in the HalfCheetah-v4
environment.

We briefly show our results in Table 1 and other detailed information can be
found in Appendix B.

| Walker-v4 | HalfCheetah-v4
---|---|---
SAC | $4476.2\pm 707.2$ | $9003.5\pm 2846.2$
MPSAC | $4371.1\pm 350.2$ | $4506.8\pm 564.3$ | $4227.1\pm 556.3$ | $8459.9\pm 856.7$ | $6478.6\pm 875.9$ | $5869.2\pm 942.7$
MSSAC(AL) | $\mathbf{4938.4\pm 341.5}$ | $\mathbf{4953.2\pm 503.4}$ | $4504.15\pm 1226.3$ | $11447.4\pm 1729.9$ | $10574.8\pm 1516.2$ | $9349.4\pm 2494.9$
MSSAC(AG) | $4456.6\pm 844.9$ | $4530.7\pm 826.5$ | $\mathbf{4778.4\pm 1098.2}$ | $\mathbf{13088.8\pm 1134.4}$ | $\mathbf{12168.1\pm 1166.2}$ | $\mathbf{11073.3\pm 1309.1}$
| 2 STEP | 3 STEP | 4 STEP | 2 STEP | 3 STEP | 4 STEP
DDPG | $406.0\pm 280.5$ | $5634.5\pm 919.0$
MPDDPG | $1971.0\pm 870.3$ | $2853.9\pm 908.2$ | $3611.1\pm 944.3$ | $5247.9\pm 1049.3$ | $4184.6\pm 802.4$ | $3759.4\pm 475.4$
MSDDPG(AL) | $\mathbf{2756.1\pm 1107.5}$ | $\mathbf{4089.2\pm 981.1}$ | $\mathbf{4726.0\pm 906.0}$ | $6623.2\pm 1476.4$ | $6085.0\pm 1316.1$ | $5338.3\pm 1285.6$
MSDDPG(AG) | $1473.8\pm 615.1$ | $2011.6\pm 797.6$ | $2424.7\pm 1153.3$ | $\mathbf{7717.8\pm 624.3}$ | $\mathbf{7069.2\pm 1402.1}$ | $\mathbf{7108.33\pm 548.0}$
TABLE 1: Experiment results. AL and AG indicate action-loaded and action-
generated methods respectively.

##  VII Conclusion

We propose a multi-state TD target and reinforcement learning algorithms based
on this target in two modes: action-loaded and action-generated. Empirical
results demonstrate that both modes offer distinct advantages. Future work
could explore how different environmental factors influence the performance of
these action-loaded and action-generated approaches.

## References

  * [1] V. Feinberg, A. Wan, I. Stoica, M. I. Jordan, J. E. Gonzalez, and S. Levine, “Model-based value estimation for efficient model-free reinforcement learning,” CoRR, vol. abs/1803.00101, 2018.
  * [2] T. Y. Ryosuke Furuta, Naoto Inoue, “Fully convolutional network with multi-step reinforcement learning for image processing,” in Proceedings of the 19th AAAI Conference on Artificial Intelligence, vol. 33(01), pp. 3598–3605, AAAI Press, 2019.
  * [3] Z. Quan, L. Ji, S. Bin, W. Huw, H. Yinglong, L. Ziyang, X. Hongming, and F. Yan, “Multi-step reinforcement learning for model-free predictive energy management of an electrified off-highway vehicle,” Applied Energy, vol. 255, no. 113755, 2019.
  * [4] L. Meng, R. Gorbet, and D. Kulić, “The effect of multi-step methods on overestimation in deep reinforcement learning,” in 2020 25th International Conference on Pattern Recognition (ICPR), pp. 347–353, 2021.
  * [5] S. Han, Y. Chen, G. Chen, J. Yin, H. Wang, and J. Cao, “Multi-step reinforcement learning-based offloading for vehicle edge computing,” in 2023 15th International Conference on Advanced Computational Intelligence (ICACI), pp. 1–8, 2023.
  * [6] M. Towers, J. K. Terry, A. Kwiatkowski, J. U. Balis, G. d. Cola, T. Deleu, M. Goulão, A. Kallinteris, A. KG, M. Krimmel, R. Perez-Vicente, A. Pierré, S. Schulhoff, J. J. Tai, A. T. J. Shen, and O. G. Younis, “Gymnasium,” Mar. 2023\.
  * [7] K. De Asis, J. F. Hernandez-Garcia, G. Z. Holland, and R. S. Sutton, “Multi-step reinforcement learning: a unifying algorithm,” in Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18, AAAI Press, 2018.
  * [8] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 10 2017.
  * [9] Y. Yuan, Z.-L. Yu, and Z. Gu, “A novel multi-step reinforcement learning method for solving reward hacking,” Applied Intelligence, vol. 49, no. 8, pp. 2874–2888, 2019.  Published online: 09 February 2019, Issue Date: 15 August 2019.
  * [10] J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee, “Sample-efficient reinforcement learning with stochastic ensemble value expansion,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, (Red Hook, NY, USA), p. 8234–8244, Curran Associates Inc., 2018.
  * [11] H. Lin, Y. Sun, J. Zhang, and Y. Yu, Model-Based Reinforcement Learning with Multi-Step Plan Value Estimation, vol. 372, pp. 1481–1488.  09 2023.
  * [12] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction.  MIT press, 2018.
  * [13] F. S. Melo, “Convergence of q-learning: A simple proof,” Institute Of Systems and Robotics, Tech. Rep, pp. 1–4, 2001.

## Appendix A: Convergence proof for Multi-state TD Target

In this part, we prove that the multi-state TD target can converge to the
optimal policy under some commonly used conditions. We drive the proof based
on the convergence of Q-learning mentioned in [12] and [13].

######  Lemma 1.

We define a contraction operator H, for a function
$q:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, we have

$\displaystyle(\mathbf{H}q)(s,a)=\sum_{y\in
S}P_{a}(s,y)\big{[}r+\gamma\max_{b\in\mathcal{A}}q(y,b)\big{]}.$ (16)

The optimal Q-function is a fixed point of the operator
H:$Q^{*}=\mathbf{H}Q^{*}$. Meanwhile, this operator is a contraction in the
up-norm:

$\displaystyle\|\mathbf{H}_{q1}-\mathbf{H}_{q2}\|_{\infty}\leq\gamma\|q_{1}-q_{2}\|_{\infty}$
(17)

######  Lemma 2.

The random process $\\{\Delta_{t}\\}$ taking values in $\mathbb{R}^{n}$ and
defined as

$\displaystyle\Delta_{t+1}(x)=\left(1-\alpha_{t}(x)\right)\Delta_{t}(x)+\alpha_{t}(x)F_{t}(x)$
(18)

converges to zero under the following assumptions:

  1. 1.

$0\leq\alpha_{t}\leq 1$,$\sum_{t}\alpha_{t}(x)=\infty$ and
$\sum_{t}\alpha_{t}^{2}(x)<\infty$

  2. 2.

$\left\|\mathbb{E}\left[F_{t}(x)\right]\right\|_{W}\leq\gamma\left\|\Delta_{t}\right\|_{W},\text{
with }\gamma<1$

  3. 3.

$\mathbf{var}[F_{t}(x)]\leq C(1+\|\Delta_{t}\|^{2}_{W})$, for $C>0$.

The proof of lemma 1 and 2 can be found in [13].

######  Theorem 1.

For a a finite MDP defined by
$(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$, given by the
multi-state TD update rule:

$\displaystyle Q_{t+1}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\leftarrow
Q_{t}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+\alpha_{t}TD,$ (19)

where TD is:

$\frac{1}{L}\sum_{l=1}^{L}\Bigg{(}\sum_{i=1}^{l}\gamma^{i-1}\mathbf{r}_{t+i}+\max_{b\in\mathcal{A}}\gamma^{l}Q_{t}(\mathbf{s}_{t+l},\mathbf{b})-Q_{t}(\mathbf{s}_{t},\mathbf{a}_{t})\Bigg{)},$

will converge to optimal Q-function as the following assumptions are held.

  1. 1.

$\mathcal{S}$ and $\mathcal{A}$ are finite.

  2. 2.

$\alpha_{t}\in[0,1]$, $\sum_{t}\alpha_{t}=\infty$,
$\sum_{t}\left(\alpha_{t}\right)^{2}<\infty$.

  3. 3.

The reward function $\mathcal{R}$ is bounded.

  4. 4.

$\gamma\in[0,1)$.

###### Proof.

We first rewrite (19):

$\displaystyle Q_{t+1}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\leftarrow$
$\displaystyle(1-\alpha_{t})Q_{t}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+\alpha_{t}\frac{1}{L}\sum_{l=1}^{L}$
$\displaystyle\sum_{i=1}^{l}\gamma^{i-1}\mathbf{r}_{t+i}+\max_{b\in\mathcal{A}}\gamma^{l}Q_{t}(\mathbf{s}_{t+l},\mathbf{b})$
(20)

According to
$\Delta_{t}(\mathbf{s}_{t})=Q_{t}(\mathbf{s}_{t},\mathbf{a}_{t})-Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t})$,

$\displaystyle\Delta_{t+1}(\mathbf{s}_{t})$ $\displaystyle=$
$\displaystyle\frac{1}{L}\sum_{l=1}^{L}\bigg{(}(1-\alpha_{t})(Q_{t}(\mathbf{s}_{t},\mathbf{a}_{t})-Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t}))$
$\displaystyle+\alpha_{t}\big{(}\sum_{i=1}^{l}\gamma^{i-1}\mathbf{r}_{t+i}+\gamma^{l}\max_{b\in\mathcal{A}}Q_{t}(\mathbf{s}_{t+l},\mathbf{b})$
$\displaystyle-Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t})\big{)}\bigg{)}$
$\displaystyle=$
$\displaystyle\frac{1}{L}\sum_{l=1}^{L}\bigg{(}(1-\alpha_{t})\Delta_{t}(\mathbf{s}_{t})+\alpha_{t}\big{(}\sum_{i=1}^{l}\gamma^{i-1}\mathbf{r}_{t+i}$
$\displaystyle+\gamma^{l}\max_{b\in\mathcal{A}}Q_{t}(\mathbf{s}_{t+l},\mathbf{b})-Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t})\big{)}\bigg{)}$
$\displaystyle=$
$\displaystyle\frac{1}{L}\sum_{l=1}^{L}\bigg{(}(1-\alpha_{t})\Delta_{t}(\mathbf{s}_{t})+\alpha_{t}F^{l}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})\bigg{)},$
(21)

where $F^{l}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})$ is defined as:

$\displaystyle F^{l}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})=$
$\displaystyle\sum_{i=1}^{l}\gamma^{i-1}\mathbf{r}_{t+i}+\gamma^{l}\max_{b\in\mathcal{A}}Q_{t}(\mathbf{s}_{t+l},\mathbf{b})$
$\displaystyle-Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t}).$ (22)

Notice that
$\sum_{i=1}^{l}\gamma^{i-1}\mathbf{r}_{t+i}+\gamma^{l}\max_{b\in\mathcal{A}}Q_{t}(y,b)$
will decrease while $l$ increase as the action taken each step might not be
the one that maximizes the reward. So, we can conclude:

$\displaystyle F^{l}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})\leq
F^{1}_{t}(\mathbf{s}_{t},\mathbf{a}_{t}).$ (23)

Therefore,

$\displaystyle\Delta_{t+1}(\mathbf{s}_{t})=$
$\displaystyle\frac{1}{L}\sum_{l=1}^{L}\bigg{(}(1-\alpha_{t})\Delta_{t}(\mathbf{s}_{t})+\alpha_{t}F^{l}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})\bigg{)}$
$\displaystyle\leq(1-\alpha_{t})\Delta_{t}(\mathbf{s}_{t})+\alpha_{t}F^{1}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})$
(24)

Then, we only need to prove that (Proof.) meets the conditions mentioned in
Lemma 2. It is clear that our $\alpha_{t}$ meets the requirement 1 as
mentioned in the assumption of Theorem1. Now consider
$F^{1}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})$, we have

$\displaystyle\mathbb{E}\left[F^{1}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})\right]=$
$\displaystyle\sum_{y\in\mathcal{X}}\mathrm{P}_{a}(\mathbf{s}_{t},y)\bigg{[}\mathbf{r}_{t}+\gamma\max_{b\in\mathcal{A}}Q_{t}(y,b)$
$\displaystyle-Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t})\bigg{]}$ (25)
$\displaystyle=$
$\displaystyle\left(\mathbf{H}Q_{t}\right)(\mathbf{s}_{t},\mathbf{a}_{t})-Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t}).$
(26)

We mentioned in Lemma 1 that $Q^{*}=\mathbf{H}Q^{*}$, so we have

$\displaystyle\mathbb{E}\big{[}F^{1}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})\big{]}=\left(\mathbf{H}Q_{t}\right)(\mathbf{s}_{t},\mathbf{a}_{t})-\left(\mathbf{H}Q^{*}\right)(\mathbf{s}_{t},\mathbf{a}_{t})$
(27)

According to Lemma 1, we have

$\displaystyle\|\mathbb{E}\big{[}F^{1}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})\big{]}\|_{\infty}\leq\gamma\|Q_{t}-Q^{*}\|_{\infty}=\gamma\|\Delta_{t}\|_{\infty}$
(28)

, which satisfy the condition 2 in Lemma2. The variance of
$F^{1}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})$ now can be calculated:

$\displaystyle\mathbf{var}$
$\displaystyle[F^{1}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})]$ $\displaystyle=$
$\displaystyle\mathbb{E}\Bigg{[}\bigg{(}\mathbf{r}_{t}+\gamma\max_{b\in\mathcal{A}}Q_{t}(\mathbf{s}_{t+1},\mathbf{b})-Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t})$
$\displaystyle-(\mathbf{H}Q_{t})(\mathbf{s}_{t},\mathbf{a}_{t})+Q^{*}(\mathbf{s}_{t},\mathbf{a}_{t})\bigg{)}^{2}\Bigg{]}$
$\displaystyle=$
$\displaystyle\mathbb{E}\Bigg{[}\bigg{(}\mathbf{r}_{t}+\gamma\max_{b\in\mathcal{A}}Q_{t}(\mathbf{s}_{t+1},\mathbf{b})-(\mathbf{H}Q_{t})(\mathbf{s}_{t},\mathbf{a}_{t})\bigg{)}^{2}\Bigg{]}$
$\displaystyle=$
$\displaystyle\mathbf{var}\Bigg{[}\mathbf{r}_{t}+\gamma\max_{b\in\mathcal{A}}Q_{t}(\mathbf{s}_{t+1},\mathbf{b})\Bigg{]}$
(29)

Since we have the assumption that the reward function is bounded, it is clear
that

$\displaystyle\mathbf{var}$
$\displaystyle[F^{1}_{t}(\mathbf{s}_{t},\mathbf{a}_{t})]\leq
C(1+\|\Delta_{t}\|^{2}_{W})$ (30)

for constant C, which satisfy the requirement of condition 3 in Lemma2. ∎

To this end, we prove that our multi-state TD algorithm can converge to the
optimal Q-function.

## Appendix B: Experiments Details

”Hidden width” indicates the number of neurons in the hidden layers. $\tau$ is
the proportion in the soft target policy update. ”lr” represents the learning
rates. $\alpha$ is the entropy regularizer for SAC algorithms while $\mu$,
$\theta$, and$\sigma$ is the hyperparameter controlling the random process in
DDPG.

hyperparameters | value
---|---
hidden width | 256
$\tau$ | 0.005
lr | 0.0004
$\gamma$ | 0.99
batch size | 128
$\alpha$ | 0.12
TABLE 2: Hyperparameters for SAC-related algorithms. hyperparameters | value
---|---
hidden width | 256
$\tau$ | 0.005
lr | 0.0003
$\gamma$ | 0.99
batch size | 128
$\mu$ | 0
$\theta$ | 0.2
$\sigma$ | 0.3
TABLE 3: Hyperparameters for DDPG-related algorithms.

Figure 3: Performance comparison of various DDPG algorithms in Walker-v4
environment.

Figure 4: Performance comparison of various SAC algorithms in Walker-v4
environment.

Figure 5: Performance comparison of various DDPG algorithms in HalfCheetah-v4
environment.

Figure 6: Performance comparison of various SAC algorithms in HalfCheetah-v4
environment.