Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview

Peter Bell, Member, IEEE,
Joachim Fainberg, Member, IEEE,
Ondrej Klejch,  Member, IEEE,

Jinyu Li,  Member, IEEE,
Steve Renals, Fellow, IEEE, and
Pawel Swietojanski, Member, IEEE

Authors have made equal contributions and are listed alphabetically.
Peter Bell, Ondrej Klejch, and Steve Renals are with the Centre for Speech Technology Research, University of Edinburgh, Edinburgh EH8 9AB, UK (email<EMAIL_ADDRESS><EMAIL_ADDRESS>s.renals@ed.ac.uk).
Joachim Fainberg did this work at the Centre for Speech Technology Research, University of Edinburgh, Edinburgh EH8 9AB, UK. He is now with JPMorgan Chase, New York, NY 10017, USA (email: j.fainberg@ed.ac.uk).
Jinyu Li is with the Microsoft Corporation, Redmond, WA 98052, USA (e-mail<EMAIL_ADDRESS>Pawel Swietojanski did this work at the School of Computer Science and Engineering, University of New South Wales, Sydney, NSW 2052, Australia. He is now with Apple, Cambridge, UK (email: p.swietojanski@unsw.edu.au).
This work was partially supported by the  EPSRC Project EP/R012180/1 (SpeechWave), a PhD studentship funded by Bloomberg, and the EU H2020 project ELG (grant agreement 825627).
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Air Force Research Laboratory (AFRL) contract #FA8650-17-C-9117. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, AFRL or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

We present a structured overview of adaptation algorithms for neural network-based speech recognition, considering both hybrid hidden Markov model / neural network systems and end-to-end neural network systems, with a focus on speaker adaptation, domain adaptation, and accent adaptation.  The overview characterizes adaptation algorithms as based on embeddings, model parameter adaptation, or data augmentation.  We present a meta-analysis of the performance of speech recognition adaptation algorithms, based on relative error rate reductions as reported in the literature.

Speech recognition, speaker adaptation, speaker embeddings, structured linear transforms, regularization, data augmentation, domain adaptation, accent adaptation, semi-supervised learning

§ INTRODUCTION

The performance of automatic speech recognition (ASR) systems has improved dramatically in recent years thanks to the availability of larger training datasets, the development of neural network based models, and the computational power to train such models on these datasets [1, 2, 3, 4].  However, the performance of ASR systems can still degrade rapidly  when their conditions of use (test conditions) differ from the training data.  There are several causes for this, including speaker differences, variability in the acoustic environment, and the domain of use.

Adaptation algorithms attempt to alleviate the mismatch between the test data and an ASR system's training data.  Adapting an ASR system is a challenging problem since it requires the modification of large and complex models, typically using only a small amount of target data and without explicit supervision.  Speaker adaptation – adapting the system to a target speaker – is the most common form of adaptation, but there are other important adaptation targets such as the domain of use, and the spoken accent.   Much of the work in the area has focused on speaker adaptation: it is the case that many approaches developed for speaker adaptation do not explicitly model speaker characteristics, and can be applied to other adaptation targets.  Thus our core treatment of adaptation algorithms is in the context of speaker adaptation, with a later discussion of particular approaches for domain adaptation and accent adaptation. Specifically, domain adaptation in this paper refers to the task of adapting the models to the target domain that has either acoustic or content mismatch from the source domain in which the models were trained.

This overview focuses on the adaptation of neural network (NN) based speech recognition systems, although we briefly discuss earlier approaches to speaker adaptation of hidden Markov model (HMM) based systems.  NN-based systems [5, 1, 6] have revolutionized the field of speech recognition, and there has been intense activity in the development of adaptation algorithms for such systems.  Adaptation of NN-based speech recognition is an exciting research area for at least two reasons: from a practical point of view, it is important to be able to adapt state-of-the-art systems;  and from a theoretical point of view the fact that NNs require fewer constraints on the input than a Gaussian-based system, along with the gradient-based discriminative training which is at the heart of most NN-based speech recognition systems, opens a range of possible adaptation algorithms.

§.§ NN/HMM Hybrid systems

Neural networks were first applied to speech recognition as so-called NN/HMM hybrid systems, in which the neural network is used to estimate (scaled) likelihoods that act as the HMM state observation probabilities [5] (fig:architectures_a). During the 1990s both feed-forward networks [5] and recurrent neural networks (RNNs) [7] were used in such hybrid systems and close to state-of-the-art results were obtained [8]. These systems were largely context-independent, although context-dependent NN-based acoustic models were also explored [9].

The modeling power of neural network systems at that time was computationally limited, and they were not able to achieve the precise levels of modeling obtained using context-dependent GMM-based HMM systems which became the dominant approach.  However, increases in computational power enabled deeper neural network models to be learned along with context-dependent modeling using the same number of context-dependent HMM tied states (senones) as GMM-based systems [1, 2].  This led to the development of systems surpassing the accuracy of GMM-based systems.  This increase in computational power also enabled more powerful neural network models to be employed, in particular time-delay neural networks (TDNNs) [10, 11], convolutional neural networks (CNNs) [12, 13],  long short-term memory (LSTM) RNNs [14, 15], and bidirectional LSTMs [16, 17].

§.§ End-to-end systems

Since 2015, there has been a significant trend in the field moving from hybrid HMM/NN systems to end-to-end (E2E) NN modeling [18, 19, 20, 6, 21, 22, 4, 23, 24] for ASR. E2E systems are characterized by the use of a single model transforming the input acoustic feature stream to  a target stream of output tokens, which might be constructed of characters, subwords, or even words.  E2E models are optimized using a single objective function, rather than comprising multiple components (acoustic model, language model, lexicon) that are optimized individually.   Currently, the most widely used E2E models are  connectionist temporal classification (CTC) [25, 26],  the RNN  Transducer (RNN-T) model [27, 21], and the  attention-based encoder-decoder (AED) model [18, 6].

CTC and the RNN-T both map an input speech feature sequence to an output label sequence, where the label sequence (typically characters) is considerably shorter than the input sequence.  Both of these architectures use an additional blank output token to deal with the sequence length differences, with
an objective function which sums over all possible alignments using the forward backward algorithm [28].  CTC is an earlier, and simpler, method which assumes frame independence and functions similarly to the acoustic model in hybrid systems without modeling the linguistic dependency across words;  its architecture is similar to that of the neural network in the hybrid system (fig:architectures_a).

An RNN-T (fig:architectures_b) combines an additional prediction network with the acoustic encoder.  The prediction network is an RNN modeling linguistic dependencies whose input is the previously  output symbol.  It is possible to initialize some of its layers from an external language model trained on additional text data.  The acoustic encoder and the prediction network are combined using a feed-forward joint network followed by a softmax to predict the next output token given the speech input and the linguistic context.

Together, the RNN-T's prediction and joint networks may be regarded as a decoder, and we can view the RNN-T as a form of encoder-decoder system.  The AED architecture (fig:architectures_c) enriches the encoder-decoder model with an additional attention network which interfaces the acoustic encoder with the decoder.  The attention network operates on the entire sequence of encoder representations for an utterance, offering the decoder considerably more flexibility.  A detailed comparison of popular E2E models in both streaming and non-streaming modes with large scale training data was conducted by Li et al. [29]. It is worth noting that with the recent success in machine translation, there is a trend of using the transformer model [30] to replace LSTM for both the AED [31, 32, 33] and RNN-T models [34, 35, 36].

§.§ Adaptation and transfer learning in related fields

Adaptation and transfer learning have become important and intensively researched topics in other areas related to machine learning, most notably computer vision and natural language processing (NLP).  In both these cases the motivation is to train powerful base models using large amounts of training data, then to adapt these to specific tasks or domains, for which considerably less training data is available.

In computer vision, the base model is typically a large convolutional network trained to perform image classification or object recognition using the ImageNet database [37, 38].  The ImageNet model is then adapted to a lower resource task, such as computer-aided detection in medical imaging [39].  Kornblith et al. [40] have investigated empirically how well ImageNet models transfer to different tasks and datasets.

Transfer learning in NLP differs from computer vision, and from the speech recognition approaches discussed in this paper, in that the base model is trained in an unsupervised fashion to perform language modeling or a related task, typically using web-crawled text data.  Base models used for NLP include the bidirectional LSTM [41] and Transformers which make use of self-attention [42, 43].  These models are then trained on specific NLP tasks, with supervised training data, which is specified in a common format (text-to-text transfer [43]), often trained in a multi-task setting. Earlier adaptation approaches in NLP focused on feature adaptation (e.g. [44]), but more recently better results have been obtained using model-based adaptation, for instance “adapter layers” [45, 43], in which trainable transform layers are inserted into the pretrained base model.

More broadly there has been extensive work on domain adaptation and transfer learning in machine learning, reviewed by  Kouw and Loog [46].  This includes work on few-shot learning [47, 48, 49] and normalizing flows [50, 51].  Normalizing flows  which provide a probabilistic framework for feature transformations, were first developed for speech recognition as Gaussianization [52], and more recently have been applied to speech synthesis [53] and voice transformation [54].

§.§ Structure of this review

We begin by considering the issues of identifying suitable data and target labels to adapt to in sec:adaptation_targets.
After discussing speaker adaptation of non NN-based HMM systems in Section <ref>, we present a general framework for adaptation of NN-based speech recognition systems (both hybrid and E2E) in sec:adaptNN, where we organize adaptation algorithms into three general categories:  embedding-based approaches (discussed in sec:embeddings), model-based approaches (discussed in Secs. <ref>–<ref>), and data augmentation approaches (discussed in sec:data_augmentation).

As mentioned above, most of our treatment of adaptation algorithms is in the context of speaker adaptation.  In Secs. <ref> and <ref> we discuss specific approaches to accent adaptation and domain adaptation respectively.

Our primary focus is on the adaptation of acoustic models and end-to-end models.  In sec:lmadapt we provide a summary of work in language model (LM) adaptation, mentioning both n-gram and neural network language models, and the use of LM adaptation in E2E systems.

Finally we provide a meta analysis of  experimental studies using the main adaptation algorithms that we have discussed (sec:meta_analysis).  The meta-analysis is based on experiments reported in  papers, carried out using  datasets, and is primarily based on the relative error rate reduction arising from adaptation approaches.  In this section we  analyze the performance of the main adaptation algorithms across a variety of adaptation target types (for instance speaker, domain, and accent), in supervised and unsupervised settings,  in six different languages, and  using six different NN model types in both hybrid and E2E settings. Raw data, aggregated results and the corresponding scripts are available at .

§ IDENTIFYING ADAPTATION TARGETS

Adaptation aims to reduce the mismatch between training and test conditions.  For an adaptation algorithm to be effective, the distribution of the adaptation data should be close to that  encountered in test conditions.  For this reason it is important to ensure that the target labels adapted to form coherent classes.  For the task of acoustic adaptation this requirement is typically satisfied by forming the adaptation data from one or more speech segments from known testing conditions (the same speaker, accent, domain, or acoustic environment).

While for some tasks labels ascribed to speech segments may exist, allowing segments to be grouped into larger adaptation clusters, it is unrealistic to assume the availability of such metadata in general. However, depending on the application and the operating regime of the ASR system, it may be possible to derive reasonable proxies.

Utterance-level adaptation derives adaptation statistics using a single speech segment [55]. This waives the requirement to carry information about speaker identity between utterances, which may simplify deployment of recognition systems – in terms of both engineering and privacy – as one does not need to estimate and store offline speaker-specific information. On the other hand, owing to the small amounts of data available for adaptation the gains are usually lower than one could obtain with speaker-level clusters. While many approaches use utterances to directly extract corresponding embeddings to use as an auxiliary input for the acoustic model [56, 57, 58, 59], one can also build a fixed inventory of speakers, domains, or topic codes [60] or embeddings [61, 62] when learning the acoustic model or acoustic encoder, and then use the test utterance to select a combination of these at test stage. The latter approach alleviates the necessity of estimating an accurate representation from small amounts of data.

It may be possible to relax the utterance-level constraint by iteratively re-estimating adaptation statistics using a number of preceding segment(s) [57]. Extra care usually needs to be taken to handle silence and speech uttered by different speakers, as failing to do so may deteriorate the overall ASR performance [63, 64, 62].

Speaker-level adaptation aggregates statistics across two or more segments uttered by the same talker, requiring a way to group adaptation utterances produced by different talkers.  In some cases – for example lecture recordings and telephony – speaker information may be available.  In other cases potentially inaccurate metadata is available, for instance in the transcription of television or online broadcasts.  In many cases (for instance, anonymous voice search) speaker metadata is not available.  The generic approach to this problem relies on a speaker diarization system [65], that can identify speakers and accordingly assign their identities to the corresponding segments in the recordings. This is often used in the offline transcription of meetings or broadcast media.  Alternative clustering approaches can be used to define the adaptation classes  [66, 67].

Domain-level adaptation broadens the speaker-level cluster by including speech produced by multiple talkers characterized by some common characteristic such as accent, age, medical condition, topic, . This typically results in more adaptation material and an easier annotation process (cluster labels need to be assigned at batch rather than segment level). As such, domain adaptation can usually leverage adaptation transforms with greater capacity, and thus offer better adaptation gains.

Depending on whether adaptation transforms are estimated on held out data, or adaptation is iteratively derived from test segments, we will refer to these as enrolment or online modes, respectively.  In enrolment mode, the adaptation data would be ideally labeled with a gold-standard transcription, to enable supervised learning algorithms to be used for adaptation.  However, supervised data is rarely available:  small amounts may be available for some domain adaptation tasks (for example, adapting a system trained on typical speech to disordered speech [68]).  In the usual case, where supervised adaptation data is not available, supervised training algorithms can still be used with “pseudo-labels” that are automatically obtained from a seed model, a process which is a type of semi-supervised training [69]. Alternatively,  unsupervised training can be applied to  learn embeddings for the different adaptation classes, such as i-vectors [56] or bottleneck features extracted from an auto-encoder neural network [70].
A two-pass system is a special case for which the statistics are estimated from test data using the first pass decoding with a speaker-independent model in order to obtain adaptation labels, followed by a second pass with the speaker-adapted model.

For semi-supervised approaches, it is possible to further filter out regions with low-confidence to avoid the reinforcement of potential errors [71, 72, 73]. There is some evidence in the literature that, for some limited-in-capacity transforms estimated in a semi-supervised manner, the first pass transcript quality has a small impact on the adapted accuracy as long as these are obtained with the corresponding speaker-independent model [74, 75]. In lattice supervision multiple possible transcriptions are used in a semi-supervised setting by generating a lattice, rather than the one-best transcription [76, 77, 78, 79].

§ ADAPTATION ALGORITHMS FOR HMM-BASED ASR

Speaker adaptation of speech recognition systems has been investigated since the 1960s [80, 81].  In the mid-1990s, the influential maximum likelihood linear regression (MLLR) [82] and maximum a posteriori (MAP) [83] approaches to speaker adaptation for HMM/GMM

systems were introduced.  These methods, described below, stimulated the field leading an intense activity in algorithms for the adaptation of HMM/GMM systems, reviewed by Woodland [84] and Shinoda [85], as well as in section 5 of Gales and Young's broader review of HMM-based speech recognition [86].  As we later discuss, some of the algorithms developed for HMM-based systems, in particular feature transformation approaches have been successfully applied to NN-based systems.  In this section we review  MAP,  MLLR, and related approaches to the adaptation of HMM/GMM systems, along with earlier approaches to speaker adaptation.

§.§ Speaker normalisation

Many of these early approaches were designed to normalize speaker-specific characteristics, such as vocal tract length, building on linguistic findings relating to speaker normalization in speech perception [87], often casting the problem as one of spectral normalization.  This work included formant-based frequency warping approaches  [80, 81, 88] and the estimation of linear projections to normalize the spectral representation to a speaker-independent form [89, 90].

Vocal tract length normalization (VTLN) was introduced by Wakita [91] (and again by Andreou [92]) as a form of frequency warping with the aim to compensate for vocal tract length differences across speakers. VTLN was extensively investigated for speech recognition in the 1990s and 2000s  [93, 94, 95, 96], and is discussed further in sec:embeddings.

§.§ Model based approaches

In model based adaptation, the speech recognition model is used to drive the adaptation.  In work prefiguring subspace models, Furui [97] showed how speaker specific models could be estimated from small amounts of target data in a dynamic time warping setting, learning linear transforms between preexisting speaker-dependent phonetic templates, and templates for a target speaker.  Similar techniques were developed in the 1980s by adapting the vector quantization (VQ) used in discrete HMM systems. Shikano, Nakamura, and Abe [98] showed that mappings between speaker dependent codebooks could be learned to model a target speaker (a technique widely used for voice conversion [99]); Feng et al. [100] developed a VQ-based approach in which speaker-specific mappings were learned between codewords in a speaker-independent codebook, in order to maximize the likelihood of the discrete HMM system. Rigoll  [101] introduced a related approach in which the speaker-specific transform took the form of a Markov model.  A continuous version of this approach, referred to as probabilistic spectrum fitting, which aimed to adjust the parameters of a Gaussian phonetic model was introduced by Hunt [102] and further developed by Cox and Bridle [103].

These probabilistic spectral modeling approaches can be viewed as precursors to maximum likelihood linear regression (MLLR) introduced by Leggetter and Woodland [82] and generalized by Gales [104].  MLLR applies to continuous probability density HMM systems, composed of Gaussian probability density functions.  In MLLR, linear transforms are estimated to adapt the mean vectors and – in [104] – covariance matrices of the Gaussian components.  If $\mu$ and $\Sigma$ are the mean vector and covariance matrix of a particular Gaussian, then MLLR adapts the parameters as follows:

\begin{align}
\hat{\mu}_s &= A_s \, \mu - b_s \\
\hat{\Sigma}_s &= H_s \, \Sigma \, H_{s}^\intercal \, .
\end{align}

The speaker-specific parameters $b_s, A_s$ and $H_s$ are estimated using maximum likelihood.

MLLR is a compact adaptation technique since the transforms are shared across Gaussians: for instance all Gaussians corresponding to the same monophone  might share mean and covariance transforms.  Very often, especially when target data is sparse, a greater degree of sharing is employed – for instance two shared adaptation transforms, one for Gaussians in  speech models and one for Gaussians in non-speech models.

Constrained MLLR [105, 104], is an important variant of MLLR, in which the same transform is used for both the mean and covariance:

\begin{align}
\hat{\mu}_s &= A_s \mu - b_s \\
\hat{\Sigma}_s &= A_s \, \Sigma \, A_s^\intercal
\end{align}

In this case, the log likelihood for a single Gaussian is given by

\begin{gather}
L^{\text{c}\textsc{MLLR}}(x; \hat{\mu}_s, \hat{\Sigma}_s)
\log \mathcal{N}(x; A_s \mu - b_s, A_s \, \Sigma \, A_s^\intercal)\\
= \log \mathcal{N}(A_s^{-1}x + A^{-1}_s b_s; \mu, \Sigma) - \log|A_s|
\end{gather}

It can be seen that this transform of the model parameters is equivalent to applying an affine transform to the data – hence constrained MLLR is often referred to as feature-space MLLR (fMLLR),

although it is not strictly feature-space adaptation unless a single transform is shared across all Gaussians in the system, in which case the Jacobian term $-\log|A_s|$ can be ignored.  MLLR and its variants have been used extensively in the adaptation of Gaussian mixture model (GMM)-based HMM speech recognition systems [84, 86].

§.§ Bayesian methods

The above model-based adaptation approaches have aimed to estimate transforms between a speaker independent model and a model adapted to a target speaker.  An alternative Bayesian approach attempts to perform the adaptation by using the speaker independent model to inform the prior of a speaker-adapted model.  If the set of parameters of a speech recognition model are denoted by $\theta$, then maximum likelihood estimation sets $\theta$ to maximize the likelihood $p(X \,|\, \theta)$.  In MAP training, the estimation procedure maximizes the posterior of the parameters given the data $X = \left \{ x_1 \dots x_T \right \}$:

\begin{equation}
P(\theta \mid X) \propto p(X \mid \theta) \, p(\theta)^r \, ,
\end{equation}

where $p(\theta)$ is the prior distribution of the parameters, which can be based on speaker independent models, and $r$ is an empirically determined weighting factor.  Gauvain and Lee [83] presented an approach using MAP estimation as an adaptation approach for HMM/GMM systems.  A convenient choice of function for $p(\theta)$ is the  conjugate  to the likelihood – the function which ensures the posterior has the same form as the prior.  For a GMM, if it is assumed that the mixture weights $c_i$ and the Gaussian parameters ($\mu_i$, $\Sigma_i$) are independent, then the conjugate prior may take the form of a mixture model $p_D(c_i) \prod_i p_W(\mu_i, \Sigma_i)$, where $p_D()$ is a Dirichlet distribution (conjugate to the multinomial) and $p_W()$ is the normal-Wishart density (conjugate to the Gaussian).  This results in the following intuitively understandable parameter estimate for the adapted mean of a Gaussian $\hat{\mu} \in \mathbb{R}^d$:

\begin{equation}
\hat{\mu} = \frac{\tau \mu_0 + \sum_t \gamma (t) x_t} {\tau + \sum_t \gamma (t)} \, ,
\end{equation}

where $\mu_0 \in \mathbb{R}^d$ is the unadapted (speaker-independent) mean, $x_t \in \mathbb{R}^d$ is the adaptation vector at time $t$, $\gamma (t) \in \mathbb{R}$ is the component occupation probability (responsibility) for the Gaussian component at time $t$ (estimated by the forward-backward algorithm), and $\tau$ is a positive scalar-valued parameter of the normal-Wishart density, which is typically set to a constant empirically (although Gauvain and Lee [83] also discuss an empirical Bayes estimation approach for this parameter). The re-estimated means of the Gaussian components take the form of a weighted interpolation between the speaker independent mean and data from the target speaker.  When there is no target speaker data for a Gaussian component, the parameters remain speaker-independent;  as the amount of target speaker data increases, so the Gaussian parameters approach the target speaker maximum likelihood estimate.

§.§ Speaker adaptive training

In the model-based approaches discussed above (MLLR and MAP), we have implicitly assumed that adaptation takes place at test time:  speaker independent models are trained using recordings of multiple speakers in the usual way, with only the test speakers used for adaptation.  In contrast to this, it is possible to employ a model-based adaptive training approach.
In speaker adaptive training [106], a transform is estimated for each speaker in the training set, as well as for each speaker in the test set.  During training, speaker-specific transforms and a speaker-independent canonical model are updated in an iterative fashion.

Speaker space approaches represent a speaker-adapted model as a weighted sum of a set of individual models which may represent individual speakers or, more commonly, speaker clusters.  In cluster-adaptive training (CAT)  [66], the mean for a Gaussian component for a specific speaker $s$ is given by:

\begin{equation}
\hat{\mu}_s = \sum_{c=1}^C w_c \mu_c \,
\end{equation}

where $\mu_c \in \mathbb{R}^d$ is the mean of the particular Gaussian component for speaker cluster $c$, and $w_c \in \mathbb{R}$ is the cluster weight. This expresses the speaker-adapted mean vector as a point in a speaker space.  Given a set of canonical speaker cluster models, CAT is efficient in terms of parameters, since only the set of cluster weights need to be estimated for a new speaker.  Eigenvoices [107] are alternative way of constructing speaker spaces, with a speaker model again represented as a weighted sum of canonical models.  In the Eigenvoices technique, principal component analysis of “supervectors” (concatenated mean vectors from the set of speaker-specific models) is used to create a basis of the speaker space.

A number of variants of cluster-adaptive training have been presented, including representing a speaker by combining MLLR transforms from the canonical models [66], and using sequence discriminative objective functions such as minimum phone error (MPE) [108].  Techniques closely related to CAT have been used for the adaptation of neural network based systems (sec:structured_xforms).

In contrast to model-based methods, in feature-based adaptation it is usual to adapt or normalize the acoustic features for each speaker in both the training and test sets– this may be viewed as a form of speaker adaptive training.  For example, in the case of cepstral mean and variance normalization (CMVN), statistics are computed for each speaker and the features normalized accordingly, during both training and test.  Likewise, VTLN is also carried out for all speakers, transforming the acoustic features to a canonical form, with the variation from changes in vocal tract length being normalized away.

§ ADAPTATION ALGORITHMS FOR NN-BASED ASR

The literature describing methods for adaptation of NNs has tended to inherit terminology from the algorithms used to adapt HMM-GMM systems, for which there is an important distinction between feature space and model space formulations of MLLR-type approaches [104], as discussed in the previous section.  In a 2017 review of NN adaptation, Sim et al. [109] divide adaptation algorithms into feature normalisation, feature augmentation and structured parameterization.  (They also use a further category termed constrained adaptation, discussed further below.)

The task of an ASR model is to map a sequence of acoustic feature vectors, $X=(x_1, \dotsc x_t, \dotsc,  x_T)$,  to a sequence of words $W$.
Although – as we discuss below – most techniques described in this paper apply equally to end-to-end models and hybrid HMM-NN models, we generally treat the model to be adapted as an acoustic model.  That is, we ignore aspects of adaptation that affect only $P(W)$, independently of the acoustics $X$ (LM adaptation is discussed in sec:lmadapt).
Further, with only a small loss of generality, in what follows we will assume that the model operates in a framewise manner, thus we can define the model as:

\begin{equation}
y_t = f(x_t; \theta)
\end{equation}

where $f(x;\theta)$ is the NN model with parameters $\theta$ and $y_t$ is the output label at frame $t$.  In a hybrid HMM-NN system, for example, $y_t$ is taken to be a vector of posterior probabilities over a senone set.  In a CTC model, $y_t$ would be a vector of posterior probabilities over the output symbol set, plus blank symbol.  Note that NN models often operate on a wider windowed set of input features, $x_t(w) = [ x_{t-c}, x_{t-c+1}, \dotsc, x_{t+c-1}, x_{t+c} ]$ with the total window size $w = 2c+1$.  For reasons of notational clarity, we generally ignore the distinction between $x_t$ and $x_t(w)$, unless it is specifically relevant to a particular topic.

In this framework, we can define feature normalisation approaches as acting to transform the features in a speaker-dependent manner, on which the speaker-independent model operates.  For each speaker $s$, a transformation function  computes:

\begin{equation}
x^\prime_t = g(x_t;\phi_s)  \label{eqn:feature_normalisation}
\end{equation}

where $\phi_s$ is a set of speaker-dependent parameters.  Commonly the dimension of the normalised features is the identical to the original ($d=d^\prime$) but this is not required.  This family is closely related to feature space methods used in GMM systems described above in sec:early, including fMLLR (when only a single affine transform is used), VTLN, and CMVN.

Structured parameterization approaches, in contrast, introduce a speaker-dependent transformation of the acoustic model parameters:

\begin{equation}
\theta^\prime_s = h(\theta; \varphi_s) \label{eqn:structured_parametrisation}
\end{equation}

In this case, the function $h$ would typically be structured so as to ensure that the number of speaker-dependent parameters $\varphi_s$ is sufficiently smaller than the number of parameters of the original model.   Such methods are closely related to model-based adaptation of GMMs such as MLLR.

Finally, feature-augmentation approaches extend the feature vector $x_t$ with a speaker-dependent embedding $\lambda_s$, which we can write as

\begin{equation}
x^\prime_t = \begin{pmatrix} x_t \\ \lambda_s  \end{pmatrix} \label{eqn:feature_augmentation}
\end{equation}

Close variants of this approach use the embedding to augment the input to higher layers of the network.  Note that the incorporation of an embedding requires the addition of further parameters to the acoustic model controlling the manner in which the embedding acts to adapt the model, which can be written $f(x_t;\theta, \theta^E)$.  The embedding parameters $\theta^E$ are themselves speaker-independent.

We suggest that the distinctions described above may not always be helpful when considering NN adaptation specifically, because all three approaches can be seen to be closely related or even special cases of each other.  As we saw in Section <ref> this is not the case in HMM-GMM systems, where the distinction between feature-space and model adaptation is important (as noted by Gales [104]) because in the former case, different feature space transformations can be carried out per senone class if the appropriate scaling by a Jacobian is performed; whilst in the latter case, it is necessary for the adapted probability density functions to be re-normalized.

As an example of the equivalence of the close relationship between the three approaches to NN adaptation, the normalisation function $g$ can generally be formulated as shallow NN, possibly without a non-linearity.  If there is a set of “identity transform” parameters $\phi^I$ such that

\begin{equation}
g(x_t; \phi^I) = x_t,  \; \forall x_t
\end{equation}

then we have

\begin{equation}
y_t = f(x_t;\theta) = f(g(x_t;\phi^I);\theta) = f^\prime(x_t; \theta, \phi^I)
\end{equation}

where $f^\prime$ is a new network comprising of a copy of the original network $f$ with the layers of $g$ prepended.  Applying  feature normalization (<ref>) leads to:

\begin{equation}
y_t = f(x_t^\prime;\theta) = f(g(x_t;\phi_s);\theta) = f^\prime(x_t; \theta, \phi_s)
\end{equation}

which we can write as a structured parameter transformation of $f^\prime$, as defined in (<ref>):

\begin{equation}
\theta^\prime_s = \{ \theta, \phi_s \} = h(\{\theta, \phi^I\}; \varphi_s)
\end{equation}

where the transformation $h(\,\cdot\,;\varphi_s)$ is simply set to replace the parameters pertaining to $g$ with the original normalisation parameters,  $\phi_s = \varphi_s$, leaving the other parameters unchanged.

Similarly, feature augmentation approaches may be readily seen to be a further special case of structured adaptation.  In the simple case of input feature augmentation (<ref>), we see that the output of the first layer, prior to the non-linearity, can be written as

\begin{equation}
z = Wx^\prime + b = W \begin{pmatrix}x \\ \lambda_s \end{pmatrix} + b
\end{equation}

where $W$ and $b$ are the weight and bias of the first layer respectively.  By introducing a decomposition of $W$, $W= \begin{pmatrix} U & V \end{pmatrix}$ we write this as

\begin{equation}
z = \begin{pmatrix} U & V \end{pmatrix} \begin{pmatrix}x \\ \lambda_s \end{pmatrix} + b = U x + b + V \lambda_s
\end{equation}

with $U \in \theta$ and $V \in \theta^E$ being weight matrices pertaining to the input features and speaker embedding, respectively.

This can be expressed as a structured transformation of the bias:

\begin{equation}
\theta^\prime_s = \{U^\prime,b^\prime\} = h(\{U,b\}; \varphi_s) = \{U, b+V\lambda_s\}
\end{equation}

with $\varphi_s = V\lambda_s$.  Similar arguments apply to embeddings used in other network layers.

Certain types of
feature normalisation approaches can be expressed as feature augmentation.  For example, cepstral mean normalisation given by

\begin{equation}
x_t^\prime = g(x_t;\phi_s) = x_t - \mu_s
\end{equation}

can be expressed as

\begin{equation}
z = W(x - \mu_s) + b = \begin{pmatrix} W & W \end{pmatrix} \begin{pmatrix}x \\ - \mu_s \end{pmatrix} + b
\end{equation}

with augmented features $\lambda_s = -\mu_s$.

As we have seen, approaches to NN adaptation under the traditional categorization of feature augmentation, structured parameterization and feature normalization can usually be seen as special cases of one another. Therefore, in the remainder of this paper, we adopt an alternative categorization:

* Embedding-based approaches in which any speaker-dependent parameters are estimated independently of the model, with the model $f(x_t; \theta)$ itself being unchanged between speakers, other than the possible need for additional embedding parameters $\theta^E$;

* Model-based approaches in which the model parameters $\theta$ are directly adapted to data from the target speaker according to the primary objective function;

* Data augmentation approaches which attempt to synthetically generate additional training data with a close match to the target speaker, by transforming the existing training data.

This distinction is, we believe, particularly important in  speaker adaptation of NNs because in ASR it has become standard to perform adaptation in a semi-supervised manner, with no transcribed adaptation data for the target speaker.  In this setting, as we will discuss, standard objective functions such as cross-entropy, which may be very effective in supervised training or adaptation, are particularly susceptible to transcription errors in semi-supervised settings.

We describe the model-independent approaches as embedding-based  because any set of speaker-dependent parameters can be viewed as an embedding.  Embedding-based approaches are discussed in sec:embeddings.  Well-known examples of speaker embeddings include i-vectors [110, 56], and x-vectors [111], but can also include parameter sets more classically viewed as normalizing transforms such as CMVN statistics and global fMLLR transforms (see sec:early above). However, for the reasons mentioned above,
we exclude from this category methods where the embedding is simply a subset of the primary model parameters and estimated according to the model's objective function.  Note that methods using a one-hot encoding for each speaker are also excluded, since it would be impossible to use these with a speaker-independent model, without each test speaker having been present in training data; such methods might however be useful for closely related tasks such as domain adaptation, discussed in sec:domain_adaptation.

The primary benefit of speaker adaptive approaches over simply using speaker-dependent models is the prevention of over-fitting to the adaptation data (and  its possibly errorful transcript). A large number of model-based adaptation techniques have been proposed to achieve this; in this paper, we sub-divide them into:

* Structured transforms: Methods in which a subset of the parameters are adapted, with many instances structuring the model so as to permit a reduced number of speaker-dependent parameters, as in the Learning Hidden Unit Contributions (LHUC) scheme [112, 75]. The can be viewed as an analogy to MLLR transforms for GMMs.  They are discussed in sec:structured_xforms.

* Regularization: Methods with explicit regularization of the objective function to prevent over-fitting to the adaptation data, examples including the use of L2 loss or KL divergence terms to penalize the divergence from the speaker-independent parameters [113, 114].  Such methods can be viewed as related to the MAP approach for GMM adaptation.  They are discussed in sec:regularization.

* Variant objective functions: Methods which  adopt variants of the primary objective function to overcome the problems of noise in the target labels, with examples including the use of lattice supervision [79] or multi-task learning [115].  They are discussed in sec:objective_functions.

The second two categories above are collectively termed constrained adaptation in the review by Sim et al. [109].  Within this, multi-task learning is labeled by Sim et al. as attribute aware training; however,
we do not believe that all multi-task learning approaches to adaptation can be labeled in this way.

Data augmentation methods have proved very successful in adaptation to other sources of variability, particularly those – such as background noise conditions – where the required model transformations are hard to explicitly estimate, but where it is easy to generate realistic data.  In the case of speaker adaptation, it is significantly harder to generate sufficiently good-quality synthetic data for a target speaker, given only limited data from the speaker in question.  However, there is a growing body of work in this area using, for example, techniques from the field of speech synthesis [116].  Approaches in this area are discussed in sec:data_augmentation.

Most works suitable for adapting hybrid acoustic models can be leveraged to adapt acoustic encoders in E2E models. Both Kullback-Leibler divergence (KLD) regularization (sec:regularization) and multi-task learning (MTL) methods (sec:objective_functions) have been used for speaker adaptation for CTC and AED models [117, 118].

Sim et al. [119] updated  the  acoustic encoder of RNN-T models using speaker-specific adaptation data. Furthermore, by generating text-to-speech (TTS) audio from the target speaker, more data can be used to adapt the acoustic encoder. Such data augmentation adaptation (discussed in sec:data_augmentation) was shown to be an effective way for the speaker adaptation of E2E models [120] even with very limited raw  data from the target speaker.  Embeddings have also been used to train a speaker-aware  AED model [121, 62, 122].

Because AED and RNN-T also have components corresponding to the language model, there are also techniques specific to adapting the language modeling aspect of  E2E models, for instance using a text embedding instead of an  acoustic embedding  to bias an E2E model in order to produce outputs relevant to the particular recognition context [123, 124, 125]. If the new domain differs from the source domain mainly in content instead of acoustics, domain adaptation on E2E models can be performed by either interpolating the E2E model with an external language model or updating language model related components inside the E2E model with the text-to-speech audio generated from the text in the new domain [126, 127], discussed in sec:lmadapt.

A key aspect of model-based adaptation is to prevent overfitting to the limited amount of adaptation data.  This can be done by restricting the size of the adaptation space, as done using various structured linear transformations, or through explicit regularisation [114, 154, 183].

There are several additional aspects of model-based adaptation that we shall cover in the review including semi-supervised training [76, 78],  adaptive training [106, 323, 67], relations to model-based normalisation approaches (batch normalisation <cit.>), multi-task learning [187, 115], adversarial learning [181], and meta-learning – learning to learn new tasks <cit.>.

We shall review adaptation algorithms for both hybrid HMM/NN systems and for end-to-end systems, using feed-forward, convolutional, and recurrent NNs.  In nearly all cases the algorithms considered are applicable to both hybrid and end-to-end systems, although it is the case that the literature has a lot more coverage of hybrid systems.

In addition to a review and analysis of the literature we shall also provide results on reference implementations of several of the main methods in both supervised and unsupervised settings, for hybrid and end-to-end systems.  These reference implementations will be made available open source via Github.

The default setting for adaptation algorithms is speaker adaptation.  However, we shall also cover work in domain adaptation <cit.> (although not focusing on language model issues) and on accent adaptation [228].  The proposed overview will not extend to adapting to noise or room acoustics; although adaptation algorithms are an important aspect of the approaches to robust speech recognition, it is a broader problem, and was reviewed recently by Li et al. [191].  Although we review accent and dialect adaptation, we do not extend to the broader problem of multilingual and cross-lingual speech recognition (reviewed by Besacier et al. <cit.>), although we note that adaptation algorithms and semi-supervised training are of particular importance for low-resource languages.

§.§ Feature-space normalization

First of the approaches is the feature-space normalisation that performs speaker adaptation by applying some transformation to the input acoustic features with the aim of normalising certain statistical properties of each speaker's data.

Probably the simplest method is mean and variance normalisation (MVN) of the input features, sometimes referred to as Cepstral Mean and Variance Normalisation (CMN, CVN or CMVN) due to its traditional use on Cepstal features specifically.  Features are normalised to zero mean and unit variance for each speaker as follows:

\begin{equation}
x' = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}},
\end{equation}

where $\mu$ is a mean, $\sigma^2$ is a cepstral variance and $\epsilon$ is a small constant used to ensure numerical stability.
The cepstral mean and variance statistics are computed on the corresponding speaker data,
which might be some enrolment utterance, a current utterance or a whole recording.
The effectiveness of MVN, and also all other methods,
depends on the amount and quality of the data that we use to compute these statistics.
For example, estimating them on short recordings or recordings with a lot of silence can be harmful for the performance of the acoustic model.

Another feature-space speaker adaptation method is Vocal Tract Length Normalisation (VTLN) [92, 94].
This technique is based on the observations that the vocal tract length,
which varies a lot in the population (children and females usually have shorter vocal tract than males [94]),
significantly affects formant frequencies of vowels causing the performance of the speaker independent model to degrade.
Therefore, a speaker specific warping factor of the frequency-axis is applied to minimise this mismatch between speakers.
The warping factor $\alpha$ can be estimated from formant positions which correlate with vocal tract length [91, 93] or it can be estimated with a line-search to maximise the likelihood of the adaptation data.

The most powerful adaptation method traditionally viewed as feature-space normalisation is an affine transformation with parameters $A \in \mathbb{R}^{n \times n}$ and $b \in \mathbb{R}^n$ that adapts the input features as

\begin{equation}
x' = A x + b.
\end{equation}

There are two major approaches for the estimation of the parameters $A$ and $b$.
The first, Constrained Maximum Likelihood Linear Regression (cMLLR) sometimes also called Feature-space Maximum Likelihood Linear Regression (fMLLR),
uses the EM algorithm <cit.> to estimate these parameters to maximise the likelihood of the adaptation data.
cMLLR belongs to a wider family of Maximum Likelihood Linear Regression (MLLR) speaker adaptation methods developed for GMM-HMM models [104].
The second, Linear Input Network (LIN) [155] estimates the parameters discriminatively with gradient descent.

§.§ Auxiliary features

Many different auxiliary features have been used to inform the acoustic model about target speaker characteristics in the past.
For example i-vectors [110, 56], d-vectors <cit.>,  x-vectors [111, 130],
speaker codes [159],
bottleneck speaker vectors [64],
features produced by summary networks [59, 128], LHUC features [175]
or pooled and transformed activations of another acoustic model [148, 130].
r-vectors [151] were used to inform the acoustic model about target room acoustics and factorised representations [234] were used to inform the acoustic model both about speakers and environments independently.

§.§ Model-based adaptation

Model-based speaker adaptation uses the adaptation data $\mathcal{D}^\text{adapt}$ to update the acoustic model $f(x; \theta)$, herein represented as a neural network, to allow the acoustic model to adapt to differences in the adaptation data [155].
The acoustic model is adapted by discriminatively adapting the parameters $\theta$ using gradient descent with a suitable objective function.
Model-based adaptation methods can be split into two groups.
The first group adapts the whole acoustic model or some of its layers [113, 114, 154, 290].
The second group employs linear transformations to transform input features $x$, hidden activations $h$ or outputs $y$ of the acoustic model.
These transformations are called Linear Input Network (LIN) [155], Linear Hidden Network (LHN) [156] and Linear Output Network (LON) [157] respectively.
These transforms can be parameterised with a transformation matrix $A \in \mathbb{R}^{n \times n}$ and a bias $b \in \mathbb{R}^n$.
The transformation matrix $A$ is initialised as an identity matrix and the bias $b$ is initialised as a zero vector prior to the speaker adaptation.
The adapted hidden activations then become

\begin{equation}
h' = A h + b.
\end{equation}

Neural network acoustic models have a large modelling capacity, but the adaptation of too many parameters might lead to overfitting to the adaptation data.
The overfitting can manifest itself in three ways.
First, if the amount of adaptation data is limited,
then speaker adaptation can overfit to classes seen in the adaptation data and forget about classes that are not present in the adaptation data.
Second, the model can also overfit to certain speaker characteristics that are present in the adaptation data but that are not fully representative of the speaker.
Third, in unsupervised speaker adaptation the labels are obtained from erroneous transcripts,
which makes the ASR model prone to overfitting to errors made in the first pass decoding.
Therefore, model-based adaptation typically needs to be strongly regularised.
Overfitting is traditionally prevented either by reducing the number of speaker dependent parameters,
or by using appropriate regularisation losses that prevent the adapted model in diverging too far from the original model.

§ SPEAKER EMBEDDINGS

Speaker embeddings map speakers to a continuous space. In this section we consider embeddings that may be extracted in a manner independent of the model, and which are also typically unsupervised with respect to the transcript. They can therefore also be useful in a standalone manner for other tasks such as speaker recognition. When used with an acoustic model, the model learns how to incorporate the embedding information by, in effect, speaker-aware training. Speaker embeddings may encode speaker-level variations that are otherwise difficult for the AM to learn from short-term features [64], and may be included as auxiliary features to the network. Specifically, let $x\in\mathbb{R}^d$ denote the acoustic features, and $\lambda_s\in\mathbb{R}^k$ a k-dimensional speaker embedding. The speaker embeddings may be  concatenated with the acoustic input features, as previously seen in (<ref>):

\begin{equation}
x^\prime_t = \begin{pmatrix} x_t \\ \lambda_s  \end{pmatrix} \label{eqn:feature_augmentation_2}
\end{equation}

Alternatively they may be concatenated with the activations of a hidden layer. In either case the result is bias adaptation of the next hidden layer as  discussed in sec:structured_xforms. As noted by Delcroix et al. [128] the auxiliary features may equivalently be added directly to the features using a learned projection matrix $P$, with the benefit that the downstream architecture can remain unchanged:

\begin{equation}
x^\prime_t = x_t + P\lambda_s
\end{equation}

There are many other ways to incorporate embeddings into the AM: for example, they may be used to scale neuron activations as in LHUC [75]. More generally we may consider embeddings applied to either biases or activations through context-adaptive [129] or control networks [130]. It is possible to limit connectivity from the auxiliary features to the rest of the network in order to improve robustness at test time or to better incorporate static features [131, 132, 133]. We will further consider transformations of the features as speaker embeddings, such as with fMLLR [105, 104], and they may also be used as label targets [134].

There is a considerable overlap between this paragraph and the penultimate paragraph of Sec. VI
I read both carefully and think it's ok

§.§ Feature transformations

We may consider speaker-level transformations of the acoustic features as speaker embeddings. These include methods traditionally viewed as normalisation, such as CMVN and fMLLR, which produce affine transformations of the features:

\begin{equation}
x_s' = A_s x + b_s
\end{equation}

CMVN derives its name from the application to cepstral features, but corresponds to the standardization of the features to zero mean and unit variance (z-score):

\begin{equation}
x_s' = \frac{x-\mu}{\sqrt{\sigma^2 + \epsilon}}
\end{equation}

where $\mu \in \mathbb{R}^d$ is the cepstral mean, $\sigma^2 \in \mathbb{R}^d$ is the cepstral variance, and $\epsilon$ is a small constant for numerical stability.

fMLLR [104] belongs to a family of speaker adaptation methods originally developed for HMM-GMM models, as discussed in Section <ref>. The technique has, however, later been used with success to transform features for hybrid models as well [135, 136]. While the fMLLR transforms were traditionally estimated using maximum likelihood and HMM-GMM models, the transforms may also be estimated using a neural network trained to estimate fMLLR features [137] (in sec:structured_xforms we will further discuss structurally similar transforms estimated using the main objective function). Instead of transforming the input features, some work has explored fMLLR features as an additional, auxiliary, feature stream to the standard features in order to improve robustness to mismatched transforms [133], or to obtain speaker-adapted features derived from GMM log-likelihoods [138], otherwise known as GMM-derived features.

Another technique with a long history is VTLN [91, 92, 94, 139], which was briefly introduced in Section <ref>. To control for varying vocal tract lengths between speakers, VTLN typically uses a piecewise linear warping function to adjust the filterbank in feature extraction. This requires only a single warping factor parameter that can be estimated using any AM with a line search. Alternatively, linear-VTLN ([95]) obtains a corresponding affine transform similar to fMLLR, but chooses from a fixed set of transforms at test time . A related idea is that of the exponential transform [140], which forgoes any notion of vocal tract length, but akin to VTLN is controlled by a single parameter. More recently, adaptation of learnable filterbanks, operating as the first layer in a deep network, has resulted in updates which compensate for vocal tract length differences between speakers [141].

§.§ i-vectors

Many types of embeddings stem from research in speaker verification and speaker recognition. One such approach is identity vectors, or i-vectors [110, 142, 56], which are estimated using means from GMMs trained on the acoustic features.
Specifically, the extraction of a speaker i-vector, $\lambda_s \in \mathbb{R}^k$, assumes a linear relationship between the global means from a background GMM (or universal background model, UBM), $m_{g}\in \mathbb{R}^m$, and the speaker-specific means, $m_s\in \mathbb{R}^m$

\begin{equation}
m_{s} = m_{g} + T\lambda_s
\end{equation}

where $T\in \mathbb{R}^{m\times k}$ is a matrix that is shared across all speakers which is sometimes called the total variability matrix from its relation to joint factor analysis [143]. An i-vector thus corresponds to coordinates in the column space of $T$. $T$ is estimated iteratively using the EM algorithm. It is possible to replace the GMM means with posteriors or alignments from the AM
[144, 145, 131]
although this is no longer independent of the AM and requires transcriptions. The i-vectors are usually concatenated with the acoustic features as discussed above, but have also been used in more elaborate architectures to produce a feature mapping of the input features themselves [146, 147].

§.§ Neural network embeddings

A number of works proposed to extract low-dimensional embeddings from bottleneck layers in neural network models trained to distinguish between speakers [64, 132] or across multiple layers followed by dimensionality reduction in a separate AM (CNN embeddings [148]). One such approach, using Bottleneck Speaker Vector (BSV) embeddings [64], trains a feed-forward network to predict speaker labels (and silence) from spliced MFCCs (diag:bottleneck). Tan et al. [132] proposed to add a second objective to predict monophones in a multi-task setup. The bottleneck layer dimension is typically set to values commonly used for i-vectors. In fact, Huang and Sim [64] note that if the speaker label targets are replaced with speaker deviations from a UBM, then the bottleneck-features may be considered frame-level i-vectors. The extracted features are averaged across all speech frames of a given speaker to produce speaker-level i-vectors.

There are several more recent approaches that we may collectively refer to as $\star$-vectors. Like bottleneck features, these approaches typically extract embeddings from neural networks trained to discriminate between speakers, but not necessarily using a low-dimensional layer. For instance, deep vectors, or d-vectors [149, 150], extract embeddings from feed-forward or LSTM networks  trained on filterbank features to predict speaker labels. The activations from the last hidden layer are averaged over time.  X-vectors [111, 130] use TDNNs  with a pooling layer that collects statistics over time and the embeddings are extracted following a subsequent affine layer. A related approach called r-vectors [151] uses the architecture of x-vectors, but predicts room impulse response (RIR) labels rather than speaker labels. In contrast to the above approaches, label embeddings, or l-vectors [134], are designed to be used as soft output targets for the training of an AM. Each label embedding represents the output distribution for a particular senone target. In this way they are, in effect, uncoupled from the individual data points and can be used for domain adaptation without a requirement of parallel data. We will discuss this idea further in sec:domain_adaptation. For completeness we also mention h-vectors [152] which use a hierarchical attention mechanism to produce utterance-level embeddings, but has only been applied to speaker recognition tasks.

X-vector embeddings are not widely used for adaptating ASR algorithms in practice – especially in comparison to commonly used i-vectors – as experiments have not shown consistent improvements in recognition accuracy.  One reason for this is related to the speaker identification training objective for the x-vector network which implicitly factors out channel information, which might be beneficial for adaptation. The optimal objective for speaker embeddings used in ASR differs from the objective used in speaker verification.

Summary networks [59, 128] produce sequence level summaries of the input features and are closely related to  (diag:summary_network).  Auxiliary features are produced by a neural network that takes as input the same features as the AM, and produces embeddings by taking the time-average of the output. By incorporating the averaging into the graph, the network can be trained jointly with the AM in an end-to-end fashion [128]. A related approach is to produce LHUC feature vectors (sec:structured_xforms) from an independent network with embedded averaging [153].

§.§ Embeddings for E2E systems

The embedding method is also helpful to the adaptation of E2E systems. Fan et al. [121] and Sari et al.  [62] generated a soft embedding vector by combining a set of i-vectors from multiple speakers with the combination weight calculated from the attention mechanism. The soft embedding vector is appended to the acoustic encoder output of the E2E model, helping the model to normalize speaker variations.  While the soft embedding vectors in [121, 62] are different at each frame, the speaker i-vectors are concatenated with the speech utterance as the input of every encoder layer in [122] to form  a persistent memory through the depth of encoder, hence learning utterance-level speaker knowledge.

In addition to acoustic embedding, E2E models can also leverage text embedding to improve their modeling accuracy. For example, E2E models can be optimized to produce outputs relevant to the particular recognition context, for instance user contacts or device location. One solution is to add a context bias encoder in addition to the original audio encoder into E2E models [123, 124, 125].  This bias encoder takes a list of biasing phrases as the input. The context vector of the biasing list is generated by using the attention mechanism, and is then  concatenated with the context vector of acoustic encoder and is fed into the decoder.

§ STRUCTURED TRANSFORMS

Methods to adapt the parameters $\theta$ of a neural network-based acoustic model $f(x; \theta)$ can be split into two groups.
The first group adapts the whole acoustic model or some of its layers [113, 114, 154].
The second group employs structured  transformations [109] to transform input features $x$, hidden activations $h$ or outputs $y$ of the acoustic model.
Such transformations include the linear input network (LIN) [155], linear hidden network (LHN) [156] and the linear output network (LON) [157].
These transforms are parameterized with a transformation matrix $A_s \in \mathbb{R}^{n \times n}$ and a bias $b_s \in \mathbb{R}^n$.
The transformation matrix $A_s$ is initialized as an identity matrix and the bias $b_s$ is initialized as a zero vector prior to speaker adaptation.
The adapted hidden activations then become

\begin{equation}
h' = A_s h + b_s.
\end{equation}

However, even a single transformation matrix $A_s$ can contain many speaker dependent parameters, making adaptation susceptible to overfitting to the adaptation data.
It also limits its practical usage in real world deployment because of memory requirements related to storing speaker dependent parameters for each speaker.
Therefore there has been considerable research into how to structure the matrix $A_s$ and the bias $b_s$ to reduce the number of speaker dependent parameters.

The first set of approaches restricts the adaptation matrix $A_s$ to be diagonal.  If we denote the diagonal elements as $r_s = diag(A_s)$, then
the adapted hidden activations become

\begin{equation}
h' = r_s \odot h + b_s.
\end{equation}

There are several methods that belong to this set of adaptation methods.
LHUC [112, 75]  adapts only the parameters $r_s$:

\begin{equation}
h' = r_s \odot h.
\label{eq:lhuc}
\end{equation}

Speaker Codes [158, 159] prepend an adaptation neural network to an existing SI model in place of the input features. The adaptation network – which operates somewhat similarly to control networks, described below – uses the acoustic features as inputs, as well as an auxiliary low-dimensional speaker code which essentially adapts speaker dependent biases within the adaptation network:

\begin{equation}
h' = h + b_s.
\end{equation}

The network and speaker codes are learned by back-propagating through the frozen SI network with transcribed training data. At test time the speaker codes are derived by freezing all but the speaker code parameters and back-propagating on a small amount of adaptation data.

Similarly, Wang and Wang [160] proposed a method that adapts both $r_s$ and $b_s$ as parameters $\beta_s \in \mathbb{R}^n$ and $\gamma_s \in \mathbb{R}^n$ of a batch normalization layer, adapting both the scale and the offset of the hidden layer activations with mean $\mu \in \mathbb{R}^n$ and variance $\sigma^2 \in \mathbb{R}^n$:

\begin{equation}
h' = \gamma_s \frac{h - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta_s.
\end{equation}

Mana et al. [161] showed that batch normalization layers can be also updated by recomputing the statistics $\mu$ and $\sigma^2$ in online fashion.

A similar approach with a low-memory footprint adapts the activation functions instead of the scale $r_s$ and offset $b_s$.
Zhang and Woodland [162] proposed the use of parameterised sigmoid and ReLU activation functions.
With the parameterised sigmoid function,
hidden activations $h$ are computed from hidden pre-activations $z$ as

\begin{equation}
h = \eta_s \frac{1}{1 + e^{-\gamma_s z + \zeta_s}},
\end{equation}

where $\eta_s \in \mathbb{R}^n$, $\gamma_s \in \mathbb{R}^n$ and $\zeta_s \in \mathbb{R}^n$ are speaker dependent parameters.
$|\eta_s|$ controls the scale of the hidden  activations,
$\gamma_s$ controls the slope of the sigmoid function
and $\zeta_s$ controls the midpoint of the sigmoid function.
Similarly, parameterised ReLU activations were defined as

\begin{equation}
h = \begin{cases}
\alpha_s z & \text{if } z > 0 \\
\beta_s z & \text{if } z \leq 0
\end{cases},
\end{equation}

where $\alpha_s \in \mathbb{R}^n$ and $\beta_s \in \mathbb{R}^n$ are speaker dependent parameters that correspond to slopes for positive and negative pre-activations, respectively.

Other approaches factorize the transformation matrix $A_s$ into a product of low-rank matrices to obtain a compact set of speaker dependent parameters.
Zhao et al. [163] proposed the Low-Rank Plus Diagonal (LRPD) method, which reduces the number of speaker dependent parameters by approximating the linear transformation matrix $A_s \in \mathbb{R}^{n \times n}$ as

\begin{equation}
A_s \approx D_s + P_s Q_s,
\end{equation}

where the $D_s \in \mathbb{R}^{n \times n}$, $P_s \in \mathbb{R}^{n \times k}$ and $Q_s \in \mathbb{R}^{k \times n}$ are treated as speaker dependent matrices ($k < n$) and $D_s$ is a diagonal matrix.
This approximation was motivated by the assumption that the adapted hidden activations should not be very different from the unadapted hidden activations  when only a limited amount of adaptation data is available; hence the adaptation linear transformation should be close to a diagonal matrix.
In fact, for $k = 0$ LRPD reduces to LHUC adaptation.
LRPD adaptation can be implemented by inserting two hidden linear layers and a skip connection as illustrated in diag:LRPD.

Zhao et al. [164] later presented an extension to LRPD called Extended LRPD (eLRPD),  which removed the dependency of the number of speaker dependent parameters on the hidden layer size by performing a different approximation of the linear transformation matrix $A_s$,

\begin{equation}
A_s \approx D_s + P T_s Q,
\end{equation}

where matrices $D_s \in \mathbb{R}^{n \times n}$ and $T_s \in \mathbb{R}^{k \times k}$ are treated as speaker dependent, and matrices $P \in \mathbb{R}^{n \times k}$ and $Q \in \mathbb{R}^{k \times n}$ are treated as speaker independent.
Thus the number of speaker dependent parameters is mostly dependent on $k$, which can be chosen arbitrarily.

Instead of factorizing the transformation matrix, a technique typically known as feature-space discriminative linear regression (fDLR) [165, 135, 166] imposes a block-diagonal structure such that each input frame shares the same linear transform. This is, in effect, a tied variation of LIN with a reduction in the number of speaker dependent parameters.

Another set of approaches uses the speaker dependent parameters as mixing coefficients $\theta_s = \left \{ \alpha_0 \dots \alpha_k \right \}$
for a set of $k$ speaker independent bases $\left \{ B_0 \dots B_k \right \}$
which factorize the transformation matrix $A_s$.
Samarakoon and Sim [167, 168] proposed to use factorized hidden layers (FHL) that allow both speaker-independent and speaker dependent modelling.
With this approach, activations of a hidden layer $h$ with an activation function $\sigma$ are computed as

\begin{equation}
h = \sigma\left( (W + \sum_{i=0}^{k} \alpha_i B_i) x + b_s + b \right ).
\label{eq:fhl1}
\end{equation}

Note, that  when $\alpha_s = 0$ and $b_s = 0$,
the activations correspond to a standard speaker independent model.
If the bases $B_i$ are rank-1 matrices, $B_i = \gamma_i \psi_i^T$, then this allows the reparameterization of (<ref>) as [168]:

\begin{equation}
% \begin{split}
%     h & = \sigma\left( (W + \sum_{i=0}^{k} [\alpha_s]_i B_i) x + b_s + b \right ) \\
%       & = \sigma\left( (W + \sum_{i=0}^{k} [\alpha_s]_i \gamma_i \psi_i^T) x + b_s + b \right ) \\
%       & = \sigma\left( (W + \Gamma D \Psi^T) x + b_s + b \right ),
% \end{split}
h = \sigma\left( (W + \Gamma D \Psi^T) x + b_s + b \right ),
\end{equation}

where vectors $\gamma_i$ and $\psi_i$ are $i$-th columns of matrices $\Gamma$ and $\Psi$, respectively, and the mixing coefficients $\alpha_s$ correspond to the diagonal of matrix $D$.
This approach is very similar to the factorization of hidden layers used for Cluster Adaptive Training of DNN networks (CAT-DNN) [67] that uses full rank bases instead of rank-1 bases.

Similarly, Delcroix et al. [129] proposed to adapt the activations of a hidden layer using a mixture of experts [169].
The adapted hidden unit activations are then

\begin{equation}
h' = \sum_{i = 0}^{k} \alpha_i B_i h.
\end{equation}

There have also been approaches, that further reduce the number of speaker dependent parameters by removing the dependency on the hidden layer width by using control networks that predict the speaker-dependent parameters

\begin{equation}
\theta_s = c(\lambda_s; \phi),
\end{equation}

In contrast to the adaptation network used in the Speaker Codes scheme, the control networks themselves are speaker-independent,
taking as input some lower dimensional speaker embedding $\lambda_s \in \mathbb{R}^k$.
As such, they form a link between structured transforms and the embedding-based approaches of Sec. <ref>.
The control networks $c(\lambda_s, \phi)$ can be implemented as a single linear transformation or as a multi-layer neural network.
These control networks are similar to the conditional affine transformations referred to as Feature-wise Linear Modulation (FiLM) [170].
For example, Subspace LHUC [171] uses a control network to predict LHUC parameters $r_s$ from i-vectors $\lambda_s$, resulting in a $94\%$ memory footprint reduction compared to standard LHUC adaptation.
Cui et al. [172] used auxiliary features to adapt both the scale $r_s$ and offset $b_s$.
Other approaches adapted the scale $r_s$ or the offset $b_s$ by leveraging the information extracted with summary networks instead of auxiliary features [173, 174, 175].

Finally, the number of speaker dependent parameters in all the aforementioned linear transformations can be reduced by applying them to bottleneck layers that have much lower dimensionality than the standard hidden layers.
These bottleneck layers can be obtained directly by training a neural network with bottleneck-layers or by applying Singular Value Decomposition (SVD) to the hidden layers [176, 177].

This is done by applying SVD to the weight matrix $W \in \mathbb{R}^{n \times n}$ which factorises it as:

\begin{equation}
A = U \Sigma V^T
\end{equation}

where the columns of the matrix $U \in \mathbb{R}^{n \times n}$ are called the left singular vectors of A, the columns of the matrix  $V \in \mathbb{R}^{n \times n}$ are the right singular vectors of A, and $\Sigma \in \mathbb{R}^{n \times n}$ is a diagonal matrix with singular values of A on the diagonal.
By setting the $(n-k)$ smallest singular values of A to 0, we can approximate A as

\begin{equation}
A \approx U' \Sigma' V'^T,
\end{equation}

where $U' \in \mathbb{R}^{n \times k}$ correspond to the first $k$ columns of $U$ and $V'^T \in \mathbb{R}^{k \times n}$ corresponds to the first $k$ rows of $V^T$.
By setting $W' = \Sigma' V'^T$,
we approximate A as a product of two low-rank matrices $U' \in \mathbb{R}^{n \times k}$ and $W' \in \mathbb{R}^{k \times n}$

\begin{equation}
A \approx U' W'.
\end{equation}

In [177] both matrices $U'$ and $W'$ are treated as speaker-independent and the smaller matrix $S_s$ is used for speaker adaptation

\begin{equation}
A \approx U' S_s W'.
\end{equation}

This results in a substantial reduction of speaker dependent parameters,
even for LHN adaptation.

§ REGULARIZATION METHODS

Even with the small number of speaker dependent parameters required by structured transformations, speaker adaptation can still overfit to the adaptation data.
One way to prevent this overfitting is through the use of regularization methods that prevent the adapted model from diverging too far from the original model.
This can be achieved by using early stopping and appropriate learning rates,
which can be obtained with a hyper-parameter grid-search or by meta-learning [178, 179].
Another way to prevent the adapted model from diverging too far from the original can be achieved by limiting the distance between the original and the adapted model.
Liao [113] proposed to use the L2 regularization loss of the distance between the original speaker dependent parameters $\theta_s$ and the adapted speaker dependent parameters $\theta_s'$

\begin{equation}
\mathcal{L}_{L2} = \lvert\theta_s - \theta_s'\rvert^2_2.
\end{equation}

Yu et al. [114] proposed to use Kullback-Leibler (KL) divergence to measure the distance between the senone distributions of the adapted model and the original model

\begin{equation}
\mathcal{L}_{KL} = D_{KL}(f(x; \theta) \,||\, f(x; \theta_s')).
\end{equation}

If we consider the overall adaptation loss using cross-entropy:

\begin{equation}
\mathcal{L} = (1 - \lambda) \mathcal{L}_{xent} + \lambda \mathcal{L}_{KL},
\end{equation}

we can show that this loss equals to cross-entropy with the target distribution for a label $y$ given the input frame $x_t$

\begin{equation}
(1 - \lambda) \hat{P}(y \mid x_t) + \lambda f(x_t; \theta),
\end{equation}

where $\hat{P}(y \,\vert\, x_t)$  is a distribution corresponding to the provided labels $y^\text{adapt}$.
Although initially proposed for adapting hybrid models, the KLD regularization method may also be used for speaker adaptation of E2E models [117, 118, 180].

Meng et al. [181] noted that KL divergence is not a

distance metric between distributions because it is asymmetric, and therefore proposed to use adversarial learning which guarantees that the local minimum of the regularization term is reached only
if the senone distributions of the speaker independent and the speaker dependent models are identical.
They achieve this by adversarially training a discriminator $d(x; \phi)$ whose task is to discriminate between the speaker dependent deep features $h'$ and speaker independent deep features $h$ that are obtained by passing the input adaptation frames through speaker dependent and speaker independent feature extractor respectively.
This process is illustrated in diag:adversarial_speaker_adaptation.
The regularization loss of the discriminator is

\begin{equation}
% \mathcal{L}_{disc} = -\frac{1}{T} \sum_{t = 1}^{T} \left \lbrace \log d(f_i(x_t; \theta); \phi) - \log \left [ 1 - d(f_i(x_t; \theta, \theta_s'); \phi)  \right ] \right \rbrace.
\mathcal{L}_{disc} = - \log d(h; \phi) - \log \left [ 1 - d(h'; \phi)  \right ],
\end{equation}

where $h$ are hidden layer activations of the speaker independent model and $h'$ are hidden layer activations of the adapted model.
The discriminator is trained in a minimax fashion during adaptation by minimizing $\mathcal{L}_{disc}$ with respect to $\phi$ and maximizing $\mathcal{L}_{disc}$ with respect to $\theta_s$.
Consequently, the distribution of activations of the i-th hidden layer of the speaker dependent model will be indistinguishable from the distribution of activations of the i-th hidden layer of the speaker independent model,
which ought to result in more robust performance of speaker adaptation.

Other approaches aim to prevent overfitting by leveraging the uncertainty of the speaker-dependent parameter space.
Huang et al. [182] proposed Maximum A Posteriori (MAP) adaptation of neural networks, inspired by MAP adaptation of GMM-HMM models [83] (sec:early).
MAP adaptation estimates speaker dependent parameters as a mode of the distribution

\begin{equation}
\hat{\theta}_s = \arg \max_{\theta_s} P(Y \mid X, \theta_s) p(\theta_s),
\end{equation}

where $p(\theta_s)$ is a prior density of the speaker dependent parameters.
In order to obtain this prior density, Huang et al. [182] employed an empirical Bayes approach (following Gauvain and Lee [83]) and treated each speaker in the training data as a data point.
They performed speaker adaptation for each speaker and observed that the speaker parameters across speakers resemble Gaussians.
Therefore they decided to parameterise the prior density $p(\theta_s)$ as

\begin{equation}
p(\theta_s) = \mathcal{N}(\theta_s ; \mu, \Sigma),
\end{equation}

where $\mu$ is the mean of adapted speaker dependent parameters across different speakers,
and $\Sigma$ is the corresponding diagonal covariance matrix.
With this parameterisation the regularization term of the prior density $p(\theta_s)$ is

\begin{equation}
\mathcal{L}_{MAP} = \frac{1}{2}(\theta_s - \mu)^T \Sigma^{-1} (\theta_s - \mu),
\end{equation}

which for the prior density $p(\theta_s) = \mathcal{N}(\theta_s ; 0, I)$ degenerates to the

regularization loss.  Huang et al. investigated their proposed MAP approach with LHN structured transforms, but noted that it may be
used in combination with other schemes.

Xie at al [183] proposed a fully Bayesian way of dealing with uncertainty inherent in speaker dependent parameters $\theta_s$, in the context of estimating the LHUC parameters $r_s$ (see Sec. <ref>).  In this method, known as BLHUC, the
posterior distribution of the adapted model is approximated as:

\begin{equation}
% \begin{split}
%         P(Y \mid X, \mathcal{D}^\text{adapt}) & = \int P(Y \mid X, r_s) p(r_s \mid \mathcal{D}^\text{adapt}) dr_s \\
%                           & \approx P(Y \mid X, \mathbb{E}[r_s \mid \mathcal{D}^\text{adapt}]),
% \end{split}
P(Y \mid X, \mathcal{D}^\text{adapt}) \approx P(Y \mid X, \mathbb{E}[r_s \mid \mathcal{D}^\text{adapt}]),
\label{eq:blhuc_posterior}
\end{equation}

Xie at al propose to use a distribution $q(r_s)$ as a variational approximation of the posterior distribution of the LHUC parameters, $p(r_s | \mathcal{D}^\text{adapt})$.

For simplicity, they assume that both $q(r_s)$ and $p(r_s)$ are normal, such that $q(r_s) = \mathcal{N}(r_s ; \mu_s, \gamma_s)$ and $p(r_s) = \mathcal{N}(r_s ; \mu_0, \gamma_0)$, which results in the expectation for the speaker dependent parameters in (<ref>) being given by :

\begin{equation}
\mathbb{E}[r_s \mid \mathcal{D}^\text{adapt} ] = \mu_s.
\end{equation}

The parameters are computed using gradient descent with a Monte Carlo approximation.

Similarly to MAP adaptation, the effect is to force the adaptation to stay close to the speaker independent model when we perform adaptation with a small amount of adaptation data.

§ VARIANT OBJECTIVE FUNCTIONS

Another challenge in speaker adaptation is overfitting to targets seen in the adaptation data and to errors in semi-supervised transcriptions.
This issue can be mitigated by an appropriate choice of objective function.

Gemello et al. [156] proposed Conservative Training,
which modifies the target distribution to ensure that labels not seen in the adaptation data will not be catastrophically forgotten.
Given a set of labels not seen in the adaptation data  $U$ and the reference label $\hat{y}_t$ at a time-step $t$ the adjusted target distribution $\hat{P}$ is defined as

\begin{equation}
\hat{P}(y \mid x_t) = \begin{cases}
P(y \mid x_t)                              & \text{if } y \in U \\
1 - \sum\limits_{y' \in U} P(y' | x_t)   & \text{if } y = \hat{y}_t \\
0                                           & \text{otherwise}.
\end{cases}
\end{equation}

To mitigate errors in semi-supervised transcriptions we can replace the transcriptions with a lattice of supervision,
which encodes the uncertainty arising from the first pass decoding.
Lattice supervision has previously been used in work on unsupervised adaptation [76] and training [77] of GMMs,
as well as discriminative [184] and semi-supervised training [78], and adaptation [79], of neural network models.
For instance, lattice supervision can be used with the MMI criterion where for a single utterance we have:

\begin{equation}
\mathcal{F}_{MMI}(\theta) = \log \frac{p (X \mid \mathbb{M}^{num}; \theta)}{p (X \mid \mathbb{M}^{den}; \theta)},
\end{equation}

where the $\mathbb{M}_r^{num}$ is a numerator lattice containing multiple hypotheses from a first pass decoding and $\mathbb{M}_r^{den}$ is a denominator lattice containing all possible sequences of words.

Another family of methods prevents overfitting to adaptation targets by performing adaptation through the use of a lower entropy task such as monophone or senone cluster targets.
This has the advantage that the unsupervised targets might be less noisy
and also that the targets have higher coverage even with small amounts of adaptation data.
Price et al. [185] proposed to append a new output layer predicting monophone targets on top of the original output layer predicting senones.
The layer can be either full rank or sparse – leveraging knowledge of relationships between monophones and senones.
Its parameters are trained on the training data with a fixed speaker independent model.
Only the monophone targets are used for the adaptation of the speaker dependent parameters.

Huang et al. [115] presented an approach that used  multi-task learning [186] to leverage both senone and monophone/senone clusters targets.
It worked by having multiple output layers, each on top of the last hidden layer, that predicted the corresponding targets.
These additional output layers were also trained after a complete training pass of the speaker independent model with its parameters fixed.
Thus, the adaptation loss was a weighted sum of individual losses,
for example monophone and senone losses (diag:multitask_adaptation).

Swietojanski et al. [187] combined these two approaches and used multi-task learning for speaker adaptation through a structured output layer, which predicts both monophone targets and senone targets.  Unlike the approach by Price et al. [185],  the monophone predictions are used for the prediction of senones.

Li et al. [117] and Meng et al. [118] applied multi-task learning to speaker adaptation of CTC and AED models. These E2E models typically use subword units, such as word piece units, as the output target in order to achieve high recognition accuracy. The number of subword units is usually at the scale of thousands or even more. Given very limited speaker-specific adaptation data, these units may not be fully covered.  Multi-task learning using both character and subword units can significantly alleviate such sparseness issues.

§ DATA AUGMENTATION

Data augmentation has been proven to be an effective way to decrease the acoustic mismatch between training and testing conditions.  Data augmentation approaches supplement the training data with distorted or synthetic variants of speech with characteristics resembling the target acoustic environment, for instance with reverberation  or interfering sound sources. Thanks to realistic room acoustic simulators [188] one can generate large numbers of room impulse responses and reuse clean corpora to create multiple copies of the same sentence under different acoustic conditions [189, 190, 191].

Similar approaches have been proposed for increasing robustness in speaker space by augmenting training data with, typically label-preserving, speaker-related distortions or transforms. Examples include creating multiple copies of clean utterances with perturbed VTL warp factors [192, 193], augmenting related properties such as volume or speaking rate [11, 194, 195], or voice-conversion [196] inspired transformations of speech uttered by one speaker into another speaker using stochastic feature mapping [197, 193, 198].

While voice conversion does not create any new data with respect to unseen acoustic / linguistic complexity (just replicas of the utterances with different voices, often from the same dataset), recent advances in text-to-speech (TTS) allows the rapid building of new multi-speaker TTS voices [199] from small amounts of data.  TTS may then be used to arbitrarily expand the adaptation set for a given speaker, possibly to cover unseen acoustic domains [116, 120]. If TTS is coupled with a related natural language generation module, it is possible to generate speech for domain-related texts. In this way, the speaker adaptation uses more data, not only from the speaker's original speech but also from the TTS speech.  Because the transcription used for TTS generation is also used for model adaptation, this approach also circumvents the obstacle of the hypothesis error in unsupervised adaptation. Moreover, TTS generated data can also help to adapt E2E models to a new domain which has more discrepancy in contents from the source domain, which will be discussed in sec:lmadapt.

Finally, for unbalanced data sets the acoustic models may under-perform for certain demographics that are not sufficiently represented in training data. There is an ongoing effort to address this using generative adversarial networks (GANs). For example, Hosseini-Asl et al. [200] used GANs with a cycle-consistency constraint [201] to balance the speaker ratios with respect to gender representation in training set.

§ ACCENT ADAPTATION

Although there is significant literature on automatic dialect identification from speech ([202]), there has been less work on accent and dialect adaptive speech recognition systems.  The MGB–3 [203] and MGB–5 [204] evaluation challenges have used dialectal Arabic test sets, with a modern standard Arabic (MSA) training set, using broadcast and internet video data.  The best results reported on these challenges have used a straightforward model-based transfer learning approach in an lattice-free maximum mutual information (LF-MMI) framework [205],  adapting MSA trained baseline systems to specific Arabic dialects [206, 207].

Much of the reported work on accent adaptation has taken approaches for speaker adaptation, and applied them using an adaptation set of utterances from the target accent.  For instance, Vergyri et al. [208] used MAP adaptation of a GMM/HMM system. Zheng et al. [209] used both MAP and MLLR adaptation, together with features selected to be discriminative towards accent, with the accent adaptation controlled using hard decisions made by an accent classifier.

Earlier work on accent adaptation focused on automatic adaptation of the pronunciation dictionary  [210, 211].  These approaches resemble approaches for acoustic adaptation of VQ codebooks (discussed in section <ref>), in that they learn an accent-specific transition matrix between the phonemic symbols in the dictionary.  Selection of utterances for accent adaptation has been explored, with Nallasamy et al. [212] proposing an active learning approach.

Approaches to accent adaptation of neural network-based systems have typically employed accent-dependent output layers and shared hidden layers [213, 214], based on a similar approach to the multilingual training of deep neural networks [215, 216, 217].  Huang et al. [213] combined this with KL regularization (sec:regularization), and Chen et al. [214] used accent-dependent i-vectors (sec:embeddings);  Yi et al. [218] used accent-dependent bottleneck features in place of i-vectors; and Turan et al. [219] used x-vector accent embeddings in a semi-supervised setting.

Multi-task learning approaches, where the secondary task is accent/dialect identification has been explored by a number of researchers [220, 221, 222, 223, 224] in the context of both hybrid and end-to-end models.   Improvements with multi-task training were observed in some instances, but the evidence indicates that it gives a small adaptation gain.  Sun et al. [225] replaced multi-task learning with domain adversarial learning (sec:objective_functions), in which the objective function treated accent identification as an adversarial task, finding that this improved accented speech recognition over multi-task learning.

More successfully, Li et al. [226] explored learning multi-dialect sequence-to-sequence models using one-hot dialect information as input. Grace et al. [227] also used one-hot dialect codes and also explored a family of cluster adaptive training and hidden layer factorization approaches. In both cases using one-hot dialect codes as an input augmentation (corresponding to bias adaptation) proved to be the best approach, and cluster-adaptive approaches did not result in a consistent gain.  These approaches were extended by Yoo et al. [228] and Viglino et al. [224] who both explored the use of dialect embeddings for multi-accent end-to-end speech recognition. Ghorbani et al. [229] used accent-specific teacher-student learning, and Jain et al. [230] explored a mixture of experts (MoE) approach, using mixtures of experts both at the phonetic and accent levels.

Yoo et al. [228] also applied a method of feature-wise affine transformations on the hidden layers (FiLM), that are dependent both on the network’s internal state and the dialect/accent code (discussed in sec:structured_xforms).  This approach, which can be viewed as a conditioned normalization, differs from the previous use of one-hot dialect codes and multi-task learning in that it has the goal of learning a single normalized model rather than an implicit combination of specialist models.  A related approach is gated accent adaptation [231], although this focused on a single transformation conditioned on accent.

Winata et al. [232] experimented with a meta-learning approach for few-shot adaptation to accented speech, where the meta-learning algorithm learns a good initialization and hyperparameters for the adaptation.

§ DOMAIN ADAPTATION

The performance of automatic speech recognition (ASR) always drops significantly when the recognition model is evaluated in a mismatched new domain. Domain adaptation is the technology used to adapt the well-trained source domain model to the new domain. The most straightforward way is to collect and label data in the new domain to fine-tune the model. Most adaptation technologies discussed in this paper can also be applied to domain adaptation [154, 233, 234, 235, 236]. When the amount of adaptation data is limited, a common practice is adapting only partial layers of the network [237]. To let the adapted model still perform well on the source domain, Moriya et al. [238] proposed progressive neural networks by adding an additional model column to the original model for each new domain and only update the new model column with the new domain data. In the following, we focus on technologies more specific to domain adaptation.

§.§ Teacher-student learning

While  conventional adaptation techniques require large amounts of labeled data in the target domain, the teacher-student (T/S) paradigm [239, 240] can better take advantage of large amounts of unlabeled data and has been widely used for industrial scale tasks [241, 242].

The most popular T/S learning strategy was proposed in 2014 by Li et al. [239] to minimize the KL divergence between the output posterior distributions of the teacher network and the student network. This can also be considered as learning soft targets generated by a teacher model instead of 1-hot hard targets

\begin{equation}
-\sum_{t=1}^T\sum_{y=1}^N P_T(y\mid x_t) \log P_S(y\mid x_t),
\end{equation}

where $P_T$ and $P_S$ are posteriors of teacher and student networks, $x_t$ and $y_t$ are the input speech and output senone at time $t$, respectively. $T$ is the number of speech frames in an utterance, and $N$ is the number of senones in the network output layer.

Later, Hinton et al. [240] proposed knowledge distillation by introducing a temperature parameter (like chemical distillation) to scale the posteriors. This has been applied to speech by Asami et al. [243]. There are also variations such as learning the interpolation of soft and hard targets [240] and conditional T/S learning [244]. Although initially proposed for model compression, T/S learning is also widely used for model adaptation if source and target signals are frame-synchronized, which can be realized by simulation. The loss function is   [245, 246]

\begin{equation}
-\sum_{t=1}^T\sum_{y=1}^N P_T(y\mid x_t) \log P_S(y\mid \hat{x}_t),
\end{equation}

where $x_t$ is the source speech signal while $\hat{x}_t$ is the frame-synchronized target signal. It can be further improved with sequence-level loss function as the speech signal is a sequence signal [247, 248].

The biggest advantage of T/S learning is that it can leverage large amounts of unlabeled data by using soft labels $P_T(y_t=y|x_t)$. This is particularly useful in industrial setups where effectively unlimited unlabeled data is available [241, 242]. Furthermore, soft labels produced by the teacher network carry knowledge learned by the teacher on the difficulty of classifying each sample, while the hard labels do not contain such information. Such knowledge helps the student to generalize better, especially when adaptation data size is small.

E2E models tend to memorize the training data well, and therefore may not generalize well to a new domain. Meng et al. [249] proposed T/S learning for the domain adaptation of E2E models. The loss function is

\begin{equation}
-\sum_{l=1}^L\sum_{y=1}^N P_T(y \mid Y_{1:u-1}, X) \log P_S(y \mid Y_{1:u-1}, \hat{X}),
\end{equation}

where $X$ and $\hat{X}$ are the source and target domain speech sequence, $Y$ is the label sequence of length $L$ which is either the ground truth in the supervised adaptation setup or the hypothesis generated by the decoding of the teacher model with $X$ in the unsupervised adaptation setup.  Note that in the unsupervised case, there are two levels of knowledge transfer: the teacher’s token posteriors (used as soft labels) and one-best predictions as decoder guidance.

One constraint to T/S adaptation is that it requires paired source and target domain data. While the paired data can be obtained with simulation in most cases, there are scenarios in which it is hard to simulate the target domain data from the source domain data. For example, simulation of children's speech  or accented speech remains challenging. In [134], a neural label embedding scheme was proposed for domain adaptation with unpaired data. A  label embedding, l-vector, represents the output distribution of the deep network trained in the source domain for each output token, , senone. To adapt the deep network model to the target domain,  the l-vectors learned from the source domain are used as the soft targets in the cross entropy criterion.

§.§ Adversarial learning

It is usually hard to obtain the transcription in the target domain, therefore unsupervised adaptation is critical. Although the transcription can be generated by decoding the target domain data using the source domain model, the generated hypothesis quality is often poor given the domain mismatch.
Recently, adversarial training was applied to the area of unsupervised domain adaptation in a form of multi-task learning [250] without the need for transcription in the target domain. Unsupervised adaptation is achieved by learning deep intermediate representations that are both discriminative for the main task on the source domain and invariant with respect to mismatch between source and target domains. Domain invariance is achieved by adversarial training of the domain classification objective functions using a gradient reversal layer (GRL) [250]. This GRL approach has been applied to acoustic models for unsupervised adaptation in [251, 252, 253]. Meng et al. [254] further combine adversarial learning and T/S learning as adversarial T/S learning to improve the robustness against condition variability during adaptation.

There is also increasing interest in the use of GANs with cycle consistency constraints for domain adaptation [255, 256, 257]. This enables the use of non-parallel data without labels in the target domain by learning to map the acoustic features into the style of the target domain for training. The cycle-consistency constraint also provides the possibility of mapping features from the target to the source style for, in effect, test-time adaptation or speech enhancement.

Unsupervised domain adaptation is more attractive than the supervised one because there is usually large amount of unlabeled data in the new domain while transcribing the new domain data usually is time consuming with large cost. T/S learning and adversarial learning both can utilize  unlabeled data well. Specifically, T/S learning has been very successful in industry-scale tasks. In contrast, adversarial learning was reported successful in relatively smaller tasks. Therefore, T/S learning is more promising if the parallel data is available. However, if there is no prior knowledge about the new domain, adversarial learning can be a good choice. There are also other works on unsupervised domain adaptation. For example, Hsu et al. [70] use a variational autoencoder instead of adversarial learning to obtain a latent representation robust to domains. However, similar to adversarial learning, such method is pending examination when large amount of unlabeled training data is available.

§ LANGUAGE MODEL ADAPTATION

LM adaptation typically involves updating an LM estimated from a large general corpus, with data from a target domain.  Many approaches to LM adaptation were developed in the context of n-gram models, and are reviewed by Bellegarda [258]. Hybrid NN/HMM speech recognition systems still make use of n-gram language models and a finite state structure, at least in the first pass; it is difficult to use neural network LMs (with infinite context) directly in  first pass decoding in such systems.  Neural network LMs are typically used to rescore lattices in hybrid systems, or may be combined (in a variety of ways) in end-to-end systems.

The main techniques for n-gram language model adaptation include interpolation of multiple language models [259, 260, 261], updating the model using a cache of recently observed (decoded) text  [262, 263, 264, 259], or merging or interpolating n-gram counts from decoded transcripts [265].  There is also a large body of work incorporating longer scale context, for instance modelling the topic and style of the recorded speech  [266, 267, 268, 269]. LM adaptation approaches making use of wider context have often built on approaches using unigram statistics or bag-of-words models, and a number of approaches for combination with n-gram models have been proposed, for example dynamic marginals [270].

Neural network language modelling [271] has become state-of-the-art, in particular recurrent neural network language models (RNNLMs) [272].

There has been a range of work on adaptation of RNNLMs, including the use of topic or genre information as auxiliary features  [273, 274] or combined as marginal distributions [275],  domain specific embeddings [276], and the use of curriculum learning and fine-tuning to take account of shifting contexts  [277, 278].  Approaches based on acoustic model adaptation, such as LHUC [278] and LHN [274], have also been explored.

There have a been a number of approaches to apply the ideas of cache language model adaptation to neural network language models [275, 279, 280], along with so-called dynamic evaluation approaches in which the recent context is used for fine tuning [275, 281].

E2E models are trained with paired speech and text data. The amount of text data in such a paired setup is much smaller than the amount of text data used in training a separate external LM. Therefore, it is popular to adjust E2E models by fusing the external LM trained with a large amount of text data. The simplest and most popular approach is shallow fusion [282, 283, 284, 285], in which the external LM is interpolated log-linearly with the E2E model at inference time only.

However, shallow fusion does not have  a clear probabilistic interpretation. McDermott et al. [286] proposed a density ratio approach based on Bayes' rule. An LM is built on text transcripts from  the training set which has paired speech and text data, and a second LM is built on the target domain. When decoding on the target domain, the output of the E2E model is modified by the ratio of target/training LMs. While it is well grounded with Bayes' rule, the density ratio method requires the training of two separate LMs, from the training and target data respectively. Variani et al.  [287] proposed a hybrid autoregressive transducer (HAT) model to improve the RNN-T model.  The HAT model builds a training set LM internally and the label distribution is derived by normalizing the score functions across all labels excluding blank. Therefore, it is mathematically justified to integrate the HAT model with an external or target LM using the density ratio formulation.

In [126, 127], RNN-T models were adapted to a new domain  with the TTS data generated from the domain-specific text.  Because the prediction network in RNN-T works similarly to a LM, adapting it without updating the acoustic encoder  is shown to be more effective than interpolating the RNN-T model with an external LM trained from the domain-specific text [127].

Adaptation studies used in the meta-analysis categorized on the level they operate at, and system architecture.

Level     System      References

21.3cmModel     Hybrid     [241, 214, 141, 213, 154, 195, 288, 289, 178, 179, 113, 290, 74, 168, 291, 292, 293, 75, 180, 153, 294, 231]

E2E     [229, 295, 117, 249, 118, 119, 180, 232]

21.3cmEmbedding        Hybrid     [159, 214, 296, 150, 161, 74, 61, 148, 130, 168, 56, 57, 132, 138, 153]

E2E     [128, 62, 218]

Feature         Hybrid     [289, 74, 168, 56, 135, 293, 75, 138, 297, 294, 231]

Data             Hybrid     [193, 116]

§ META ANALYSIS

In this section we present an aggregated review of published results in experiments applying adaptation algorithms to speech recognition.  This differs from typical experimental reporting that focuses on one-to-one system comparisons typically using a small fixed set of systems and benchmark tasks and data.  The proposed meta-analysis approach offers insights into the performance of adaptation algorithms that are difficult to capture from individual experiments.

We divide this section into four main parts. The first, ssec:init, explains the protocol and overall assumptions of the meta-analysis, followed by a top-level summary of findings in ssec:findings_generic, with a more detailed analysis  in ssec:findings_details.  The final part, ssec:findings_corpora, aims to quantify the adaptation performance across languages, speaking styles and datasets.

§.§ Protocol and Literature

The meta-analysis is based on  peer-reviewed studies selected such that they cover a wide range of systems, architectures, and adaptation tasks. Each study was required to compare adaptation results versus a baseline, enabling the configurations of interest to be compared quantitatively. There was no fixed target for the total number of papers included, due to our aim to cover as many different methods as possible. Note that the meta-analysis spans several model architectures, languages, and domains; although most studies use word error rate (WER) as the evaluation metric, some studies used character error rate (CER) or phone error rate (PER). Since we are interested in the relative improvement brought by adaptation, we  report Relative Error Rate Reductions (RERR).

The meta-analysis is based on the studies shown in Table <ref>, with additional splits into level of operation and top-level system architecture. The positions were selected such that they cover most of the topics mentioned in the review. For an adaptation of end-to-end systems we included all peer-reviewed works we could find (their number is relatively limited). For the hybrid approach, the studies are shortlisted such that they enable the quantification of the gains for the categories outlined in the preceding theoretical sections. As a generic rule, when choosing papers for the analysis we first included works that introduced a specific adaptation method in the context of neural models, or that offered some additional experiments allowing the comparison of different areas of interest - such as the impact of objective functions, the complementarity of adaptation transforms or that show behavior under different operating regimes.  In the case of certain more commonly-used techniques, due to the laborious nature of the analysis, it was not always possible to include an exhaustive set of somewhat similar papers.  In this situation, the papers selected were those with higher citation counts.

The analysis spans  datasets (more than 50 unique {train, test} pairings), 28 of which are public and 10 are proprietary. These cover different speaking styles, domains, acoustic conditions, applications and languages (though the study is strongly biased towards English resources). The public corpora used include the following:

AISHELL2 [298], AMI [299], APASCI [300], Aurora4 [301], CASIA [302], ChildIt [303], Chime4 [304], CSJ [305], ETAPE [306], HKUST [307], MGB[308], RASC863 [309], SWBD [310], TED [311], TED-LIUM [312], TED-LIUM2 [313], TIMIT [314], WSJ [315], PF-STAR [316], Librispeech [317], Intel Accented Mandarin Speech Recognition Corpus [214], UTCRSS-4EnglishAccent [295]. To save space we do not provide detailed corpora statistics in this paper, but make them available via a corresponding repository[.] alongside raw data and scripts used to perform the analysis.

Overall, the meta-analysis is based on ASR systems trained on datasets with a combined duration of over 30,000 hours, while the baseline acoustic models were estimated from as little as 5 hours to around 10,000 hours of speech. Adaptation data varies from a few seconds per speaker to over 25,000 hours of acoustic material used for domain adaptation.

§.§ Overall findings

Aggregated summary of adaptation RERR from all studies (top), considering single method only (middle) and two or more methods stacked (bottom). The top graph is annotated to explain the information presented in each of the boxplot graphs in this section.

Comparison of feature, embedding, and model-level adaptation approaches. Speaker (middle) and domain (bottom) adaptations are based on {utterance, speaker} and {accent, child, domain, disordered} clusters, respectively.

fig:ma_total (Top) presents the average adaptation gains for all considered systems, adaptation methods, and adaptation classes. The overall RERR is 9.72%
[We do not report exact numbers in tabular form due to space limitations, but they are available in the GitHub repository].
Since grouping data across attributes of interest may result in unbalanced (or very sparse) sample sizes, we also report additional statistics such as the number of samples, datasets and studies the given statistic is based on. As can be seen in the right part of  fig:ma_total (Top), the results in this review were derived from  samples produced using  datasets reported in  studies. A single sample is defined as a 1:1 system comparison for which one can unambiguously state the RERR. Likewise, a dataset refers to a particular training corpus configuration. Note that there may be some data-level overlap between different corpora originating from the same source (TED talks) and we make a distinction for the acoustic condition (AMI close-talking and distant channels are counted as two different datasets when they are used to estimate separate acoustic models). A study refers to a single peer-reviewed publication.

Depending on which property we want to measure the analysis set can be split into smaller subsets, as the ones shown in the lower part of fig:ma_total. The majority of analyses in this review are reported for models adapted using a single method

with some additional groupings used to better capture further details such as complementarity of adaptation methods or their performance in different operating regimes.

As mentioned in sec:adaptNN, adaptation methods were historically categorized based on the level they operated at in the speech processing pipeline. fig:ma_levels (top) quantifies the ASR performance along this attribute, showing that model-based adaptation obtains the best average improvements of 11.8%, followed by embedding and feature levels at 7.2% and 5.0% RERR, respectively. This is not surprising, as model level adaptation allows large amounts of adaptation data to be leveraged by allowing the update of large portions of the model (including re-training the whole model). In more data-constrained regimes, such as utterance or speaker-level adaptation, where only a limited amount of adaptation data is typically available, differences are less pronounced and model-based speaker adaptation obtains 8.9% RERR while adapting to domains gives 15.5% RERR (middle and bottom plots in Fig <ref>). Embedding approaches stay at a similar level for speaker adaptation,  improving to 9.2% RERR for domain adaptation (although based on only two studies). Feature-space domain adaptation was used in only one study, which reported a small deterioration of -0.3% RERR.

fig:ma_levels (middle) additionally shows results for speaker-oriented data augmentation as described in sec:data_augmentation. These were found to increase accuracy by 4.6% RERR on average, or by 3.3%, 3.6% and 8.2% RERR for VTL perturbations (VTLP) [192, 193], stochastic feature mapping (SFM) [197, 193] and when using synthetically generated TTS utterances [116], respectively. Note that the TTS method was used to augment the adaptation set to better estimate additional adaptation transforms while VTLP and SFM were used to directly expand the training data, and were found particularly effective for low-resource training conditions. Data augmentations are beyond the scope of this meta-analysis and will not be further investigated in this review.

The results for different adaptation clusters, introduced in sec:adaptation_targets,  are shown in fig:ma_clusters. Models benefit more when adapting to accent, from adult to child speech, to the domain, and to disordered speech conditions (such as arising from speech motor disorders), as opposed to speaker or utterance adaptation. This is expected, since domain adaptation usually has more adaptation data, and the acoustic mismatch introduced by unseen domains is greater than the mismatch caused by unseen speakers – unless these are substantially mismatched to the training data as is often the case for child or disordered speech recognition. But in the latter case the adaptation is typically not carried out at the speaker level, but at the domain level (tailoring the acoustic model to better handle dysarthric speech, not a single dysarthric speaker).

fig:ma_hyb_e2e aggregates the adaptation along the two main neural network-based ASR approaches - hybrid and E2E. It is interesting to observe that E2E systems gain more from adaptation (12.8% RERR) than hybrid systems (9.2% RERR) in both the overall and speaker-based regimes. This is somewhat expected, as hybrid systems benefit from strong inductive biases – such as access to pronunciation dictionaries and hand engineered modeling constraints – whereas E2E models must learn these from data. Given limited amounts of training data one may expect that E2E may struggle to learn these as well as hybrid models, as such adaptation may bring greater gains. This reverses for domain adaptation, with E2E and hybrid improving by 12.2 and 14.9% RERR, respectively. Note that for domain adaptation, the hybrid approach was studied more often for child and disordered speech applications, which makes adaptation gains bigger (see also fig:ma_clusters).

Table <ref> further reports average amounts of training data used to estimate hybrid and E2E models. It is interesting to notice that E2E systems on average leverage twice as much acoustic material when compared to hybrid setups but still seem to substantially benefit from adaptation.

These results suggest that adaptation for E2E is a promising direction for future investigations, that remains under-investigated as of now based on the relatively few works published to date.

Adaptation results for different adaptation clusters.

Comparison of adaptation results for hybrid and E2E systems.

Amounts of data used to estimate Hybrid and E2E models for  and  adaptation clusters.

System     Cluster     Average Hours

Hybrid     2*All     862

E2E            1747

Hybrid     2*Speaker     874

E2E            2640

Hybrid     2*Domain     824

E2E           1033

Amounts of data used to estimate Hybrid and E2E models for  and  adaptation clusters.

2*Cluster     2cAvg. Training Data [hours]
Hybrid     E2E

All     862     1747

Speaker     874     2640

Domain     824     1033

Comparison of adaptation results for FF and RNN architectures.

Next we compare feed-forward (FF) and recurrent neural network (RNN) architectures in both hybrid and E2E models. Hybrid models can leverage either FF or RNN architectures while most E2E systems use some form of RNN. (Note, transformer-based E2E models [30] are built from FF (CNN) modules, however, due to their relative novelty in ASR there is only one accent adaptation study included in our meta-analysis [232]). fig:ma_am_type reports similar adaptation gains of 9.8% RERR for both FF and RNN architectures. RNNs seem to benefit more when adapting to speakers (9.2% vs 7.4% RERR for RNN and FF, respectively), and less when adapting to domain (10.4% vs 17.0% RERR for RNN and FF, respectively). When controlling for the system paradigm (E2E vs. Hybrid), RNNs mostly benefit through adapting E2E models ( Fig <ref> 6.6% vs 15.7% RERR for Hybrid (RNN) and E2E (RNN), respectively). We observed a similar trend for speaker and domain clusters separately (figure not shown).

fig:ma_unsup_sup compares the RERR for unsupervised and supervised modes of adaptation. Overall, deriving the adaptation transform with manually annotated targets results in an average 12.8% RERR, whereas unsupervised methods result in 8% RERR. fig:ma_unsup_sup shows results specifically for semi-supervised adaptation, which are captured by the 2pass and enrol (Unsup.) conditions.

fig:ma_methods_sup also shows further analysis on the modes of deriving adaptation statistics (sec:adaptation_targets). Both online and two-pass adaptation are unsupervised, while the enrollment mode may be either supervised or unsupervised. The supervised approach offers most accurate adaptation, as expected.  Unsupervised enrollment  outperforms the other two unsupervised methods mainly due to the T/S domain adaptation study [249] (sec:domain_adaptation) that leverages large amounts of data. When considering speaker adaptation only, the two-pass approach obtains 8.2% RERR and is more effective than enrol (Unsup.) (7.3% RERR) and online adaptation (6.5% RERR).

Comparison of adaptation results for FF and RNN architectures split by hybrid and E2E systems.

Comparison of adaptation results for supervision modes.

Comparison of adaptation results for different adaptation targets: online adaptation, supervised and unsupervised enrollment, and two-pass decoding.

Comparison of adaptation results for different amount of adaptation data.

Comparison of adaptation results for acoustic models estimated from different amounts of training data.

[Adaptation Type]

[Hybrid vs. E2E]

[Adaptation Clusters]

Regression analysis for the three major control variables.

Finally, we consider the overall trends for the considered systems and their operating regions. fig:ma_adaptbins reports results obtained with different amounts of adaptation data. fig:ma_regressions

further shows regression trends when splitting by adaptation type, hybrid or E2E, and adaptation clusters. These are in line with the observations so far: i) more adaptation data brings (on average) larger improvements; ii) model-based adaptation is more powerful and gives better results than embedding or feature-based approaches; and iii) adaptation is particularly effective in scenarios with a large mismatch and where obtaining matched training data is difficult.

In fig:ma_trainbins we further report adaptability of acoustic models estimated from different amounts of training material. Interestingly, models trained on small amounts of data (up to 50 hours) benefit from adaptation to a similar degree as models estimated from several thousands of hours. This is somewhat an unexpected result - if test sets are kept fixed, increasing the training material typically results in a less mismatched model, thus lowering gains from adaptation (and most experiments evaluating adaptation performance as a function of data are carried out in this way). However, when training from more data one should proportionally increase the complexity of the testing conditions. We hypothesize that this is what implicitly occurs across different datasets in the meta-analysis - someone who has access to a large training set may also sample a more diverse testing set. Note that the acoustic models in this work were trained from relatively limited amounts of data (up to 10k hours), and adaptation protocols between studies may not be exactly comparable. However, this does not change the conclusion that some form of adaptation is beneficial for most considered systems, regardless of how many hours of acoustic data was used to train it.

Since this meta-analysis combines results across many different studies with many reference systems,  the results should not necessarily be compared at the sample level, but rather in an aggregated form to outline dominant trends and typical data regimes that each category was tried in. Data amounts for some systems for the purpose of plotting were assumed approximately to be at a given level: two-pass systems unless shown otherwise assumed 10 minutes per speaker, while embedding approaches 30 seconds.

§.§ Detailed findings

Comparison of results for different adaptation approaches.

[Adaptation Family]

[SAT vs. Test-only]

[Adaptation Losses]

Regression analysis for adaptation families, speaker-adaptive training and adaptation losses.

Comparison of adaptation results for SAT vs Test-only modes.

In this subsection we investigate the effect of the specific approach to adaptation, beyond the broad categories discussed above. fig:ma_families reorganizes the earlier split into feature, embedding, and model-level adaptation (fig:ma_levels) into embedding (sec:embeddings) and model-based transformations (sec:structured_xforms).

For the embeddings, we introduce three sub-categories referred to as GMMEmb, NNEmb and NNTransformEmb.

GMMEmb comprises GMM-related embedding extractors primarily based on i-vectors [110, 56, 57] but also include adaptation results for other GMM-derived (GMMD) features [138].

NNEmb are neural network-based embedding extractors that estimate speaker/utterance statistics from speaker-independent acoustic features. Examples of NNEmb approaches include $\star$-vector techniques, such as d-vectors [149] and x-vectors [111], discussed in sec:embeddings, sentence-level embeddings [128, 59], and other bottleneck approaches [132, 130].

NNTransformEmb are transformed embeddings which typically rely on i-vectors as input instead of acoustic features. These have been proposed to help alleviate issues related to inconsistent DNN adaptation performance when using raw i-vectors [57, 58, 318]. The NNTransformEmb group includes studies doing standard i-vector transformations with NNEmb [147, 74, 218] but also more recent memory-based approaches in which an embedding is selected via attention from a fixed training stage embedding inventory [61, 62]. As shown in fig:ma_families GMMEmb, NNEmb and NNTransformEmb obtain 8.1%, 5.2% and 9.2% average RERR, respectively.

The second group in fig:ma_families comprises model-based approaches split into Linear Transform (LT), Activation, and Finetuning–based methods. LT methods introduce new speaker dependent affine transformations in the model, either in the form of new LIN/LHN/LON layers ( [155, 157, 135, 289]) or transforms estimated using a GMM system such as fMLLR [319, 135, 56, 112].

Finetune refers to approaches which assume that the adaptation is carried out by altering a subset of the existing model parameters. This is often done in a similar manner to an LT approach by adapting an input, output and/or one or more hidden layers that are already present in the model [113, 213, 214, 141].

Finally, activation methods perform adaptation by introducing speaker-dependent parameters in the activation functions of the neural network [320, 321, 112, 322, 162]. Note that, as outlined in sec:structured_xforms, some of activation-based methods can be expressed as constrained LT methods. The results obtained by LT, Activation and Finetune–based methods score 6.7%, 9.0% and 13.9% average RERR, respectively.

fig:ma_regressions_details (a) shows the regression trends for amounts of adaptation data for each of the six considered categories.

The use of embeddings implies that the acoustic model is trained in a speaker adaptive manner, whereas the majority of model-based techniques are carried out in a test-only manner – meaning that  speaker-level information is not used during training –  though some methods offer SAT variants [323, 167]. fig:ma_sat_vs_test_spk shows that SAT trained systems offer a small advantage (8%  vs. 7.6% RERR) when adapted with limited amounts of data (up to around 10 minutes). When looking at the average performance across all data-points, however, test-only approaches obtain 10.8% RERR, primarily because of greater adaptation gains for larger amounts of data. See also fig:ma_regressions_details (b) for operating regions of SAT and non-SAT systems.

fig:ma_adapt_loss quantifies gains for different adaptation objectives and regularization approaches – results for  the online condition are given only for reference, as in this case adaptation information is obtained via an embedding extractor (which is usually not updated, although not always [218]). The second group depicts approaches where the adaptation information is derived by adapting a GMM in model-space using an MLE or MAP criterion when extracting speaker-adapted auxiliary features for NN training [138, 324] or by estimating fMLLR transforms with MLE under a GMM to obtain speaker adapted acoustic features [319, 135, 56].

The third group comprises methods which aim to explicitly match the model's output distribution to the one found in the adaptation data. CE is a non-regularized frame-level cross-entropy baseline obtaining 8.7% average RERR. This can be improved to 14.8% average RERR by penalizing the adapted model's predictions such that they do not deviate too much from the speaker independent variant by KL regularization (CE-KL) [114]. KL regularization can be applied to either CE or sequential objective functions [154], although most models estimated in a sequential discriminative manner can successfully be adapted with a CE (or CE-KL) criterion [75, 168, 297, 195] (see also fig:ma_am_loss). Teacher-student (T/S) [239] is a special case (see sec:domain_adaptation) where the adaptation is carried with the targets directly produced by a teacher model, rather than the ones obtained from first pass decodes (possibly KL-regularized with the SI model). T/S allows the use of large amounts of unsupervised data and in this analysis was found to offer an average 28.2% RERR when adapting to domains [241, 229, 249].

The final group in fig:ma_adapt_loss includes objectives that try to leverage auxiliary information at the objective function level.

Meta-learning [178, 179, 232] estimates the adaptation hyper-parameters jointly with the adaptation transform while multi-task learning [115, 187, 295, 118] leverages  additional phonetic priors to circumvent the (potential) sparsity of senones when adapting with small amounts of data. Meta-learning and multi-task adaptation obtain 6.8% and 7.6% average RERR, respectively. See also fig:ma_regressions_details (c).

Comparison of results for different adaptation loss functions.

fig:ma_am_loss further summarizes the adaptability of acoustic models trained in a frame-based (CE) or a sequential (Seq) manner.  The results indicate that sequential models benefit more from adaptation when compared to frame-based systems (11.6% vs. 9.8% average RERR). However, when controlling for the same dataset and baseline (reference systems were expected to exist for both CE and Seq) the difference decreases to around 0.6% RERR in favor of the frame-based systems.

Comparison of adaptation results for acoustic models trained with CE and Sequence-level objectives.

fig:ma_adapt_archs compares the adaptation gains  obtained using various model architectures. LSTM benefits the most (15.4% average RERR). The feed-forward TDNN, DNN, and ResNet architectures all improve by around 10.5% RERR. Smaller gains were observed for Transformer, CNN and BLSTM, improving by 7.6, 6.5 and 4.9% average RERR, respectively. This result is somewhat expected as the last three architectures either normalize some of the variability by design, or have access to a larger speech context during recognition.

Comparison of adaptation results for different architectures.

In fig:ma_complementarity we study the complementarity of the different adaptation techniques. These results are based on 22 samples and 6 studies for which there were a complete set of baseline experiments allowing improvements to be quantified when adapting an SI model with Method1, and then measuring further gains when adding Method2. fig:ma_complementarity shows that, on average, stacking adaptation techniques improved the adaptation performance by an additional 4%, from 8% to 12% RERR.