Controllable Confidence-Based Image Denoising

Haley Owsianko   Florian Cassayre   Qiyuan Liang

École Polytechnique Fédérale de Lausanne ‐ EPFL, Switzerland

{haley.owsianko, florian.cassayre<EMAIL_ADDRESS>
Image denoising is a classic restoration problem. Yet, current deep learning methods are subject to the problems of generalization and interpretability. To mitigate these problems, in this project, we present a framework that is capable of controllable, confidence-based noise removal. The framework is based on the fusion between two different denoised images, both derived from the same noisy input. One of the two is denoised using generic algorithms (e.g. Gaussian), which make few assumptions on the input images, therefore, generalize in all scenarios. The other is denoised using deep learning, performing well on seen datasets. We introduce a set of techniques to fuse the two components smoothly in the frequency domain. Beyond that, we estimate the confidence of a deep learning denoiser to allow users to interpret the output, and provide a fusion strategy that safeguards them against out-of-distribution inputs. Through experiments, we demonstrate the effectiveness of the proposed framework in different use cases.

§ INTRODUCTION

Image noise removal is one of the fundamental problems in image restoration. The degradation model can be expressed as

\begin{equation}\label{eqn:image_formation}
\end{equation}

where $y$ is the noisy image captured by the camera, $x$ is the clean scene, and $n$ is the additive noise. One common assumption is that the $n$ is the additive white gaussian noise (AWGN). The goal of the image denoising is to recover the clean image $x$ from a noisy input $y$. Yet, they are inherently ill-posed problems as the plausible solutions highly depend on prior assumptions and are rarely universal. In addition, the measurement of the quality of a restoration algorithm also depends on the chosen assessment method [26]. Until now, there is no single metric that concludes with certainty the quality of the denoising output.

Source code available under: <https://gitlab.epfl.ch/owsianko/controllable-denoising>

Traditional algorithms focus on developing image priors to add constraints to the noisy image [5, 10]. With the blooming of deep learning (DL) and the creation of large datasets, new solutions to these problems have been developed [30, 19, 18, 1], outperforming the traditional ones quantitatively and qualitatively. However, nothing is perfect and DL methods have their own drawbacks: poor generalization on unseen data, e.g. images from another domain, and images with different noise levels [13]. Also, deep networks are often treated as a black box for which it is difficult to evaluate the predictability and interpret the results. At times when deep learning makes mistakes that users are not able to identify, this motivates the need of having some control over the system and the ability to reason about the output [23].

Controllable confidence-based image denoising pipeline. The noisy image goes through both reliable filters and deep neural networks (DNN) to obtain two denoised images. The denoised image of DNN is further fed into the confidence prediction network to get a coarse-grained confidence map. Users can choose to either apply the confidence map to guide fusion or display it solely to interpret the output of the DNN method.

There are various approaches proposed to address the above issues. As for generalization, common approaches that applied to any DL methods are taken, including data augmentation, training on a larger dataset, dropout layer in DL model, and so on [2]. Specific to image denoising, in SFM, the authors alleviated the overfitting problems of denoising by masking the high-frequency component of the training images [13]. Kim et al. [16] proposed to use an adaptive instance normalization during training together with a transfer learning schema to improve the robustness of the network. Regarding the interpretability, users essentially require more information than single output values to understand the mechanism, in particular, how confident the network is about its prediction. Even when dealing with unseen types of data, DL methods work as usual and give a normal output value, without any alert for the users. This limits its usage in many safety-critical domains. Some literature has proposed ways to model the confidence of an output [11, 23, 14], either to regularize the prediction values or to detect the less confident samples directly. In [29, 28], the authors incorporated the idea of a confidence map in deraining tasks to guide the residual map generation. The BIGPrior decoupled the hallucination and data fidelity component of the denoised output, and the final clear image is based on the confidence of the prior assumption [12].

In this project we propose a new framework, controllable confidence-based image denoising (CCID), to address the issues with generalization and interpretability of the image denoising task, based on two observations. First, building a model that generalizes to all noise settings is now technically infeasible. However, differentiation between in-distribution data and out-of-distribution (OOD) data could be done. Second, convolution output on the original noisy images is reliable. Unlike DL methods that might make mistakes sometimes, low-pass filters (e.g., Gaussian filter) are always trustworthy. Although the filtered image does lose some details of the scene, there are never wrong hallucinations. As shown in Figure <ref>, CCID first processes the input noisy image to two intermediate denoised output. Next, the confidence prediction network evaluates the output of DNN to produce a coarse-grained confidence map. In the presence of three auxiliary images, namely the reliable denoised image, the DNN denoised image, and the confidence map, users control the fusion to produce the final denoised image. Our contributions can be summarized as follows. We propose and evaluate fusion methods in the frequency domain between the reliable denoised image and the DNN denoised image. We add the interpretability of DL methods through the confidence map. Confidence prediction identifies regions where the DL denoising method might make mistake. We give users the flexibility to smoothly vary from the reliable denoised image to the DNN denoised image. In the presence of mispredictions of DNN, users have the possibility to switch to the reliable denoised image. It gives CCID robustness against various kinds of data inputs. In addition, users can choose to use the confidence map as guidance to the fusion process and achieve better fusion quality. Finally we demonstrate the possibility of extending this approach to other restoration tasks such as super-resolution. The following sections are managed as follows. In Section <ref>, we present a brief survey of related topics. Section <ref> introduces our detailed implementations. Section <ref> evaluate our proposed methods from different perspectives. Finally, Section <ref> summarize the report and present future potentials.

§ LITERATURE REVIEW

§.§ Image denoising

Before the era of deep learning, image denoising was based on prior knowledge of noisy images. BM3D [5] made the assumption that there are similar patches inside the same image. Collaborative filters were then used on every group of similar patches to denoise the image. In WNNM [10], the researchers assumed that images could be represented as low-rank matrices. Taking the physical meanings of the singular value of the images, it achieved relatively good denoising results. Nevertheless, although these assumptions hold for common scenarios, the performance degrades when they do not. These methods require a known noise level, limiting their usage. Moreover, they generally attempt to solve complex optimization problems, which makes the denoising procedure computationally expensive.

With the increase of computational power, DL methods became dominant. With a carefully designed architecture and a relatively large dataset, DL achieves superior performance compared to traditional algorithms. In DnCNN [30], Zhang et al. observed that direct prediction of the denoised image from scratch is relatively hard for DL models. Instead, they proposed to predict the residual image, the noise $n$, from the input. Together with batch normalization, it turned out to vastly boost the performance in both accuracy and speed. The network also applies for super-resolution (SR) and JPEG blocking with a properly configured training set. MWCNN [19] introduced a new type of downsampling and upsampling layer in building DL models through the discrete wavelet domain. Feeding the network with distinguished low and high-frequency information helps to guide the learning process, therefore, improving the performance. To boost the denoising performance further, the DDFN [4] integrated a dilated convolutional network with the deep boosting framework. Also, the generative adversarial network (GAN) [27] is utilized to create virtual samples, augmenting the training set. FFDNet [31] used a tunable noise level map as the input to tackle blind noise settings. BUIFD [8] presented a blind and universal denoiser for AWGN removal to improve the model's generalization strength.

§.§ Image fusion

Image fusion is a classic problem in image processing. With the growth of current image processing applications, it is necessary to have a good fusion algorithm to integrate the complementary information from two or more images into one with improved quality and important features. In the literature, image fusion usually refers to integrate complementary information from multiple sensors, multiple time domains, and multiple views [15]. In this project, we focus specifically on fusion of the same scene but restored images using different methods. Classic image fusion techniques can be categorized into two groups, spatial and frequency domain. In general, direct fusion in spatial domain introduce some undesirable distortions in details of the image [25]. Therefore, producing inferior results compared to fusion in the frequency domain. There are two widely used fusion techniques in the frequency domain, namely Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT) [3, 24, 22]. A 2D DCT transform $F(u,v)$ of an image $f(i,j)$ of size $N \times M$ can be modeled as

\begin{align}
F(u,v) =&
\sum_{i=0}^{N-1}
\sum_{j=0}^{M-1}\nonumber\\
cos\left[ \frac{\pi u}{2N}(2i+1) \right]
cos\left[ \frac{\pi v}{2M}(2j+1) \right]. \label{eqn:dct}
\end{align}

Whereas for DWT, it is computed by applying a low pass filter $g$ and a high pass filter $h$ on the image pixels. The outputs contain the approximation coefficients (from the low-pass) and the detail coefficients (from the high-pass filter). The 2D DWT can be viewed as a combination of two 1D DWT. In other words, a horizontal 1D DWT on each row and a vertical 1D DWT on each column. The approximation coefficient $y_{low}$ can be computed as
\begin{equation}
y_{low}[n] = (\sum_{k=-\infty}^{\infty}x[k]g[2n-k])\downarrow_2,
\end{equation}
where $\downarrow_2$ refers to downsampling of the output coefficients by 2. The detail coefficients are computed by replacing the low pass filter $g$ with the high pass filter $h$. The process can be repeated further to decompose the approximation coefficients with an increased frequency resolution.

§.§ Confidence on DL

The overconfident prediction issues of DL have limited its practical values. The authors of [11] pointed out that, although modern neural networks achieve high accuracy, they also generate miscalibrated outputs. The problem can be viewed from two angles. On the one hand, DL is unlikely to make confident mispredictions when evaluating inputs that are similar to the training set. On the other hand, when encountering unseen data, previously learned parameters hardly make sense. Even though DNN outputs valid answers, they are just unreliable at all. This kind of problem refers to as OOD. To solve the problem, researchers introduced the idea of confidence. In other words, the models should know what it does not know. In [14], researchers introduced a baseline to evaluate the confidence of DNN, which was based on the insight that correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified OOD examples. [7] demonstrated that mispredicted in-distribution data can be used as a proxy for OOD samples to mitigate the cost of collecting new OOD datasets. [23] proposed the correctness ranking loss for train neural networks that are aware of the confidence. Confidence has been widely discussed in the field of classification. However, to the best of our knowledge, it has seen little use in the field of image restoration. Two recent works [29, 28] introduced architecture to incorporate confidence inside the neural network to guide the deraining process, and the confidence map was calculated at multiple levels to achieve higher accuracy.

§.§ Super resolution

Single image super-resolution (SISR) is a restoration task that consists of reconstructing the high-resolution (HR) image from the low-resolution (LR) observations by adding missing pixel information. Given an LR image $y$, the image formation process can be modeled as

\begin{equation}
y = (x \circledast k)\downarrow_s +\ n,
\end{equation}

where $x \circledast k$ denotes convolution between the HR image $x$ and unknown blurry kernel $k$, $\downarrow_s$ refers to the downsampling with a scale $s$, and $n$ is the independent noise. In reality, the $k$ is unknown, and one common yet unrealistic assumption is that the degradation kernel is bicubic. Depending on the source, the noise $n$ can be assumed to equal zero. SISR problem is similar to image denoising if we view $y$ in Equation <ref> as the bicubic upsampled low-resolution image and $x$ as the high-resolution image. The reconstruction involves the removal of noise in $y$ to recover $x$. The usage of DL for this task is even more prominent than for denoising, and most approaches rely on deep residual networks. In RCAN [34], the authors proposed a residual in residual structure to train a very deep network efficiently and a channel attention mechanism to improve the results. In SAN [6], Dai et al. proposed an architecture capable of extracting interdependent information as second-order features. Liu et al. proposed HBPN [20], a hierarchy of residual hourglass modules that aim to provide a better error estimation and achieve state-of-the-art performance. However, the generalization ability of these models is limited by the finite kernel set $k$ and the scaling factors $s$ [9]. Recent works incorporated the degradation kernel $k$ explicitly inside the networks to mitigate the problem [32, 33].

§ IMPLEMENTATION

As shown in Figure <ref>, our method consists of three major components: image denoising, confidence prediction, and image fusion. The noisy input image first goes through both a reliable filter and a deep denoising network, which results in two different denoised images. Next, the confidence prediction network takes the residual map by DNN method, the original noisy image, and reliable filtered denoised image as inputs, and outputs a coarse-grained confidence map which is eight times smaller in both width and height compared to the original input. With the presence of the confidence map, users have a better idea about in which areas that deep denoising network might go wrong, and control the weighted fusion between two different denoised images. If users observe that the DNN method is highly unconfident, they could put more weight on the reliable filters, resulting in a more pleasant denoised scene. We proposed two modes of fusion. One is unguided fusion, which does not utilize the information from the confidence map. The other is the confidence-aware fusion, i.e. using the confidence map as auxiliary information to refine fusion results. In the following subsections, we discuss each component in detail regarding the implementation. We also offer a graphical user interface (GUI) to simplify user interaction.  It comes as the last part of this section.

§.§ Image denoising

§.§.§ Reliable denoising

We define reliable denoising as a method that makes few prior assumptions to provide a highly predictable output improving the noise removal. The few assumptions on these methods give them high robustness against different noise settings. The most straightforward example of such a method is the gaussian filter, acting as a low-pass filter and therefore alleviating the noise through averaging. As a side-effect, the image loses some of its sharpness due to blurring. We also provide support for other filters such as bilateral filtering and non-local means (NLM). For the experiments described at the end, we resorted to using the gaussian filter, as it outputs consistent results and does not require tweaking parameters other than the kernel size.

§.§.§ DNN denoising

In general, DL methods for denoising show superior performances compared to traditional methods and, of course, reliable filters. However, it is in the settings that the DNN has seen similar data during the training phase. The selected training set serves as the prior knowledge that the DL methods have. Their major strength and weakness at the same time is their ability to produce highly plausible results, but for which the information in the original noisy image could be distorted. This effect is particularly noticeable on extreme super-resolution tasks, which is discussed later. We call it "hallucinatory" as it is very distinctive from averaging or interpolation, and often correlates to the learned prior information. In this project, our focus is not to develop a denoising architecture that out-performs the state-of-art. Instead, we would like to explore the generalization problem when the inputs are OOD. All current DL denoising methods suffer more or less from these issues. In the seek of simplicity, we adopted the well-known denoising model DnCNN [30] as the default model. To facilitate the exploration of the OOD phenomena, we had the DnCNN trained on $\sigma=25$ using 400 images from the Berkeley Segmentation Dataset (BSDS500) [21], where $\sigma$ is the noise level. The DnCNN outputs the residual map ($n$) from the original noisy image ($y$). The final clean image ($x$) is computed by $x = y - n$.

§.§ Confidence prediction

§.§.§ Architecture

The architecture of the confidence prediction network. The input consists of three images, the original noisy image, the reliable filtered denoised image, and the residual map from DNN. The output is a coarse-grained confidence map, eight times smaller in both width and height compared to the input image. The architecture consists of several layers of convolution and pooling.

The architecture of the confidence prediction network is shown in Figure <ref>. It takes the original noisy image, the reliable filtered denoised image, and the residual map from DNN as input. The intention is to evaluate whether the residual map is wrong or not. The original noisy image and the reliable filtered denoised image offer extra information and structural features to improve the convergence speed and accuracy. Our first attempt was predicting confidence value at the pixel level. However, there was too much fluctuation in the residual map. As a result, the prediction was far from accurate and unusable. The next attempt, which is the current model, is to predict a confidence value for every $8 \times 8$ region. In other words, the confidence map is $8$ times smaller in both width and height than the original image. The $8 \times 8$ region is a compromise between localizability and precision. The region-based approach produces fairly precise predictions and offers useful confidence information for users to identify regions that may contain incorrect hallucinations. The small-sized confidence map is achieved by multiple pooling layers. The accuracy relies on the convolution layers to extract useful features.

§.§.§ Data generation

As mentioned previously, the input would be the stacked array of the original noisy image, the reliable filtered denoised image, and the residual map, which is the difference between the original, noisy input, and the output of the DnCNN.

Since we only focused on gray scale image denoising, it results in a 3-channel input tensor. The intention is to evaluate the confidence of the DNN, namely how much error it makes. The ground truth $y_{confidence}$ is computed as

\begin{equation}
y_{confidence} = 1-
\frac{\left\lVert y_{clean\_image} - y_{dnn\_denoised} \right\lVert \downarrow_8}
\end{equation}

$\left\lVert y_{clean\_image} - y_{dnn\_denoised} \right\lVert$ computes the $L_1$ norm of the difference between the clean image and the DNN denoised image. It is average downsampled by $8$ to get the error that DNN has made in a $8 \times 8$ region. The error is limited to $\sigma_{max}$. In our experiments, we only consider the noise range between $0$ and $100$ because images with noise levels over $100$ are almost unrecognizable. Therefore, $\sigma_{max}$ is set to be $100$.

For each image, a set of sub-images is generated. Those sub-images are referred to as patches and are then used to create the actual data points used for training the network, and each patch goes through standard data augmentation procedures to be more robust against overfitting. For each patch, a value $\sigma_p$ is sampled from a uniform distribution over $[0, 100]$, and is then used to generate the noise. This ensures that each noise level appears equally often in the dataset.

The data generated in this way is managed by a custom subclass of torch's Dataset. This class, ImageDataset, takes care of abstracting away the data generation and writing its results to disk to save computation time earlier. Additionally, data generation was written in a way that allows for GPU acceleration.

§.§.§ Loss function

The confidence predictor intends to show users regions where DNN makes mistakes. Therefore, it is acceptable to have a lower predicted value compared to a higher one. The extreme case would be all regions have a high confidence value $1$. To suppress an overconfident confidence predictor, we adapted the loss function, the sum of the squared estimate of errors (SSE), to favor under-confident predictions by

\begin{align}
SSE_{confidence} =
& SSE[output < target]*p_{under} \nonumber\\
& + SSE[output \ge target]*p_{over},
\end{align}

where $p$ is the penalty coefficient. Through giving a higher $p_{over}$ than the $p_{under}$, we prevent it from being overconfident on evaluation to some extent.

§.§.§ Training

The network was trained using the standard procedure. Many values of hyper-parameters were tried. At first, the network was slightly overfitting. After we introduced a weight decay of $10^{-5}$, this problem was partially remedied. A value of $2 \cdot 10^{-4}$ produced better results. It did however also lead to a less stable validation loss curve. The final model was trained with a value of $10^{-4}$, which still produced a rather jittery validation loss. The plot of the training and validation loss can be seen in Figure.

The training and validation loss of the model. The x-axis stands for the number of training epoch.

§.§ Image fusion

In this part, we describe procedures to fuse a blurry but reliable signal with a sharp but hallucinatory one. We discuss the effectiveness of these fusion techniques. Two categories of tasks were considered. The first one is relatively general and consists of fusing the two images according to a single scalar parameter. The second relies on an additional confidence input.

§.§.§ Unguided fusion

An illustration of 2D multilevel DWT on an image. A, V, H, D stand for the approximation, vertical, horizontal and diagonal wavelet coefficients respectively.

For this approach, intuitively we wanted the two extreme values of this parameter to yield the two input images respectively, while the values in between would perform a combination of the two in a way that had to be defined. A scalar interpolation on the spatial domain provides an accurate fusion in terms of pixel-wise distance, but the resulting images are hardly plausible due to ghosting [22]. We adapted fusion techniques in the frequency domain to combine the structure of both images smoothly. Here we consider two strategies, one in the DWT domain and the other in the DCT.

As for DWT, to generate a smooth combination between the two input images, we considered a multilevel DWT. Figure <ref> shows the DWT on an image from level 1 to level 3. A higher level indicates a higher frequency resolution. Two images of the same size go through the same level of the DWT, and coefficients at each level are retrieved accordingly. The fusion is then computed as the weighted average between the two. Besides, we proposed an advanced version to incorporate the cross-correlation of two coefficients to perform the fusion. The idea is that, if the reliable denoised image and the DNN one have a similar coefficient, there is no reason that we give too much weight to the possible wrong estimations.

For DCT, over a 2D domain, the spectral power is usually modeled as a probability density function of a Laplacian distribution; however it has been shown that it follows more closely a half-Gaussian probability density function [17]. Given this observation and some assessments we derived a mask-based technique, in the DCT domain:

\begin{align}
& M_w(\omega_x,\omega_y) = \exp(-(\omega_x^2+\omega_y^2)/(2s)) \nonumber\\
& \text{with } s = a\left(\frac{1}{1-w+\epsilon}-1\right).
\end{align}

$w \in [0, 1]$ is the provided fusion weight. The value $a$ controls the scale of the weight and was determined empirically. The value $\epsilon$ should be a small positive number.

The mask is then used to perform a simple interpolation on the values of each frequency, namely when $M=1$ the output equals the hallucinatory input and when $M=0$ it equals the reliable one. The low frequencies of the hallucinatory image are the first to get fused, followed by higher frequencies as the weight increases. For most image restoration tasks, including denoising and super-resolution, the low frequencies are the easiest to predict with high confidence - the reason why we integrate them first. An issue that we encountered in our previous attempts was that even though the fusion was smooth, it would introduce small wave artifacts near edges which were undesired. We suppose that it was due to an incorrect estimation of the spectral power over the frequency space, as it is absent from the results of this method.

§.§.§ Confidence-aware fusion

In the previous part, we discussed our initial strategies to fuse a DNN denoised output with a reliable component. In this part, we describe strategies that build on top of the previously defined functions and rely on the auxiliary confidence map component. The confidence map contains spatial reliability information that allows us to perform a fine-grained fusion. The goal is then to preserve the details when the risk of error is low, and otherwise average them to mitigate the consequences of an error. Remark that the confidence map is only an estimation and can be wrong too. However, as shown during the evaluation part this happens sufficiently rarely and the reliability can be provided to the user.

For DWT, we introduced patch-wise fusion. Patch-wise fusion refers to performing the DWT on each small $8 \times 8$ region, and stitch all fusion results together to get the whole image. Each $8 \times 8$ region contains a confidence value, therefore, we could use the confidence value on each region to guide the fusion individually. More precisely, we updated fusion weight on each region through

\begin{equation}\label{eqn:updated_conf}
w_{region} = w \cdot (1 + c - t),
\end{equation}

where $w$ is the global fusion weight, $c$ is the confidence of that region, and $t$ is the threshold. The intuition is that if the confidence value is higher than the threshold, we give more weight to the DNN denoised image to obtain a better denoising result safely. In practice, the threshold is set to $0.8$ according to the data distribution of in-distribution data and OOD data, which is discussed later in Section <ref>.

Unlike DWT, which produces stable transformation even on an $8 \times 8$ region, DCT is incapable of adopting the same strategy as DWT.
As the normal DCT fusion already works with a single global weight for which the scale is evenly spread, we chose to do a pixel-wise application of this function. One caveat is that each value on the confidence map represents an $8\times8$ pixels patch on the images. Therefore we rescale the confidence map using bicubic interpolation to match the input size and to minimize the artifacts. Also, following Equation <ref>, in practice, it is not required to compute the full transform and inverse transform for every confidence value.

§.§ User interface

Screenshot of the user interface. It is possible to switch between denoising and super resolution, select the reliable filter and fusion strategy, modify the zoom level and change the visualization mode (eg. highlighting the predicted hallucinations, or the error with respect to the ground truth). The ground truth is not available in evaluation mode.

Our project is fundamentally centered around users. One of the aspects is the input and corresponds to a specific need. For instance, we could assume that people working in medical imagery would be interested in having the possibility to configure the system to work at the designated confidence level, based on their requirements. On the other hand, the users must be able to interpret and reason about the output. Taking the same example, not only should it ease their job but it must absolutely reduce the risk of misinterpretation; as it is the very problem we are trying to solve.

To make our research process more effective, we designed early on a user interface (GUI) that provides interactivity (Figure <ref>). We kept in mind the two aspects while building this tool and investigated different strategies to present the information in a meaningful way.

§ RESULTS

In this section, we evaluate our proposed architecture from multiple perspectives both qualitatively and quantitatively. We show the flexibility of our controllable fusion strategy. With the guidance of the confidence map, users are able to interpret the results of DL methods and to make further decisions based on the results. Also, we evaluate the confidence-guided fusion algorithm. Last, we access the effects of OOD and the same infrastructure on SISR.

§.§ Controllable fusion

Comparison of the fusion technique described above for different fusion weights $w$. The Gaussian filter used here has parameter $\sigma=4$ for exaggeration. The hallucinatory image is replaced by the ground truth for comparison purposes. First two rows represent the fusion results using DWT and last two rows using DCT. Observe that the edges gradually become less blurry.

Figure <ref> shows the qualitative results of the proposed fusion algorithms, DWT and DCT. For the illustration purpose, we substitute the DNN denoised images with the clean images, and the Gaussian filter is applied to the clean images directly. As we could observe, the images vary smoothly from the Gaussian blurred one to the clean image, with gradual increasing details with the $w$ increasing. With the zoom-in details, it is easy to find out the edges and textures monotonically appear in the second and fourth row of the Figure <ref>. With a controllable fusion weight $w$, users are free to choose a combination of the reliable filtered image and the DNN denoised images. For the in-distribution data, when DNN demonstrates predominant advantages, users can choose a higher $w$, up to $1$. When users are not satisfied with the DNN denoised images, they could switch to a lower $w$ to achieve the most pleasant results. Detailed use cases that DNN fails to produce good results are discussed in Section <ref>.

Quantitative evaluation of the DCT (first row) and DWT (second row) fusion performance with a gaussian filter $\sigma=4$, over the BSD Set68 images corpus.

To support our assertions, we also evaluate the results using quantitative measures, including the peak-signal-to-noise ratio (PSNR), the structural similarity index measure (SSIM), and the mean squared error (MSE). The corresponding quantitative results of the same experiment settings as the Figure <ref> are shown in Figure <ref>. The gradual increased PSNR, SSIM, and decreased MSE are in accordance with our qualitative observations. DWT and DCT fusion methods achieve similar results with small variations for the chosen metrics.

Application of controllable fusion to denoising. The two images use the DCT algorithm on noisy inputs $\sigma=25$. The weight $w$ controls the fusion between the reliable image and the DNN output, as in Figure <ref>.

§.§ Confidence prediction

After experimenting with the neural network, we ended with an SSE loss of around 0.0146 for the training set and 0.0142 for the validation set. The test set ended up at 0.0145. While it is odd that the training loss is higher than for the test and validation sets, the model performs rather well. All three sets contain an input with size $40 \times 40$, and an output with size $5 \times 5$. The SSE loss is indeed small and the prediction is quite close to the ground truth. Overfitting ended up minimal, as the two losses were very close together.

The boxplot of the difference between individual pixels of target confidence and network output. A positive value indicates that the target confidence is larger than what the network predicted, meaning the the network is under-confident. A negative value means the network is over-confident.

The absolute value of the data presented in Figure <ref>

However, an abstract number such as the SSE loss is rarely enough to assess how well a model is performing. It is worthwhile to look at the statistics corresponding to the loss. Figure <ref> and Figure <ref> show statistics for thirty different test images. The network still tends to be overconfident on this set. On both the training and the validation set, it has no problem being under-confident. This is perhaps due to the training and validation sets coming from a similar distribution. However, since the average loss is roughly the same, it is fair to assume the network trained well. Additionally, test samples with the largest losses tend to also have surprisingly little variance in absolute terms. In a way, this constitutes a success for our custom Loss Function, as defined in Section <ref>. When the network overestimates the confidence, at least there is some guarantee of consistency. Figure <ref> gives us insight in how effective the network is. For instance, we noticed that most of the items with low loss tend to not deviate much from the target, as expected. The vast majority of points have a loss of less than $0.05$, and thus Figure <ref> and Figure <ref> are fairly representative.

Visualization of the confidence maps for $\sigma=25$ noisy inputs (only the ground truth is shown). The threshold is set at $0.95$, all the values above are colored green while the ones below are purple. The color intensity represents the distance to the boundary. The confidence map is $8$ times smaller than the input stack.

Figure <ref> shows the predicted confidence map for each corresponding images. For textureless regions, e.g., sky, wall, and plain color T-shirts, the confidence is high. For regions with a lot of variations, e.g., grass ground, table cloth, and hairs, the confidence is relatively low. It follows the general assumption: the corruption of high-frequency components is significantly harder or impossible to recover. Therefore, high-frequency regions are likely to be places where DNN makes greater mistakes, which corresponds to a low confidence value.

§.§ Fusion with confidence

The distribution of the ground-truth confidence value using DNN and reliable filter respectively. The confidence value is negatively correlated to the mistakes that each method makes.

In this part, we evaluated the confidence-guided fusion method. First, to understand confidence. Figure <ref> plots the ground-truth confidence distribution using the DNN and the reliable filter respectively. Here, we use DnCNN trained on $\sigma=25$ and a Gaussian filter. As we could observe from the plot, the results of the Gaussian filter have an exponential distribution as expected when varying the noise level from $0$ to $100$, the distribution of the DNN method is kind of weird. The weirdness is also anticipated since the model is only trained on $\sigma=25$, and it does not generalize to other noise levels. We could observe a valley around confidence value equals to $0.8$. For the regions that have a confidence value below $0.8$, it is better to have more reliable components. This observation built up our updated confidence schema, as shown in Equation <ref>. The confidence threshold is then set up to $0.8$

Confidence aware DWT and DCT fusion for noise $\sigma=25$. On the magnified fragment of the image (bottom row), the left part was estimated to have higher confidence than the right one. For both techniques, the fused output is both sharp and reliable for the left part. The right part on the other hand was mostly contributed by the reliable input and is thus blurry. Note that the model was trained to be cautious rather than overconfident, which is the reason why there is bleeding on the transition.

PSNR and SSIM distribution comparison between confidence-guided fusion and normal fusion. Confidence-guided fusion has a slightly better statistical performance at every fusion weight.

Figure <ref> shows the fusion results using confidence. The magnified regions (bottom row) show two areas with disparate confidence values, the plain cloth (left part) has a higher confidence value compared to the threshold, the right part contains a lot of textures, therefore has a lower confidence value. For the confidence fusion, we intentionally put a relatively higher weight on the confident region through scaling the global fusion weight using Equation <ref>. Compared to the fusion method without confidence under the same fusion weight, we could observe that the confidence-aware methods perform slightly better by preserving the trustworthy component from the DL method. The observation is also backed up by the statistical evaluation, as shown in Figure <ref>. Although the proposed method is not capable of improving the overall performance of the in-distribution denoised results. It is still worthwhile for offering better results at every fusion weight and giving users a higher possibility to select the best candidate. For in-distribution data, we almost always achieve the best performance at $w=1$ (DNN denoised image), and all mentioned fusion methods seem meaningless. However, it is not the case. As we would discuss in the following section, their effectiveness gets larger when dealing with OOD data, and in a real-life deployment, it is very likely that we are dealing with data that are not strictly similar to the training set.

§.§ Out-of-distribution data

Sample zoomed-in outputs of fusions that optimize the three measures (as given by the extremum point shown in Figure <ref>) on OOD data. Respective fusion weights were $0.55$, $0.25$, $0.5$, $0.2$, $0.4$, and $0.15$. Rows 1 and 2 are from the FMD dataset [35] and have noise $\sigma=25$. Row 3 and 4 come from the BSD dataset with Gaussian noise $\sigma=35$. Row 5 and 6 come from the BSD dataset with Poisson noise having the same standard deviation as Gaussian noise $sigma=25$. Observe that in the fused output, the incorrect hallucinations introduced by DnCNN are mitigated in exchange for details.

Row 1 and 2 are for noise $\sigma=25$ using DWT and DCT respectively on the microscopy image dataset [35]. Row 3 and 4 use $\sigma=30$ (out of distribution noise). Row 5 and 6 use the Poisson noise, having same standard deviation as Gaussian noise $\sigma=25$. The red line highlights the fusion value that achieves the best performance with respect to the corresponding measure (PSNR, SSIM and MSE). The $\Delta$ value is the difference to the DNN denoised output ($w=1$).

In this section, we evaluate the phenomena of OOD. We consider three types of OOD: images having different noise levels from the training set; images from another domain; images having different noise type. Figure <ref> shows the quantitative results on the OOD dataset. As we could observe, the fusion no longer achieves the best performance at $w=1$, in terms of PSNR, SSIM, and MSE. Instead, in some cases, the results using the Gaussian filter are already better compared to the results using DL. This problem of DL is often referred to as poor generalizability, i.e., DL methods only perform well on seen data. Images denoised using Gaussian, on the other hand, do not have this issue. It is capable of any kind of denoising, through filtering the high-frequency components. For the microscopy data (first two rows in Figure <ref>), we could find the border of the cell (first row) of the DnCNN denoised output is wrong, and some weird line patterns on the background of the second row that do not exist in the original noisy image. Also, when we increased the noise levels (middle two rows in Figure <ref>), the DL methods are no longer capable of removing the noise entirely. The training set is only on $\sigma=25$ and the model is incapable of predicting a residual map that has a higher absolute value, say $\sigma=35$. When facing a different noise type (last two rows in Figure <ref>), there are some black pots using DnCNN (fifth row) and the background of the last row is noisy.

In a nutshell, when facing OOD data the change of data distribution nullifies the learned feature maps from the training process and DL models no longer produce desirable denoised outputs. In such a case, the proposed fusion algorithm takes advantage of both the reliable filter to maintain the basic structures and the DL methods to retrieve more sharp but possibly wrong details. Allowing users to change the fusion weight offers one more layer of protection against incorrect hallucinations.

§.§ Super resolution

Adapting the fusion methods to super-resolution tasks, without the confidence map. GT, LR stand for ground truth and low resolution respectively. The DL model used is SAN on 4x problem instances.

In this section, we evaluate the possibility of extending our architecture to super-resolution. In fact, while reliable denoising relies on smoothing filter, reliable increase of the resolution uses interpolation methods. Most notably, bilinear are bicubic interpolation are widely used to scale an image upwards. DL methods are commonly adopted but on the other hand they are also more susceptible to the hallucinating effect, and thus run the risk of mis-interpretation of their output. We focused on low-scale 4x SR and observed that our methods could be adapted for these tasks. In particular, the reliable component could be defined by an interpolation filter. The different fusion techniques presented previously are still applicable and do not require changes, as shown in Figure <ref>. The confidence model would have to be retrained accordingly with the new inputs, although it is possible to exploit the model trained on denoising with limited accuracy.

§ CONCLUSION

In this report, we have presented a novel framework CCID to address the interpretability and generalizability problem to the existing DL-based denoising methods. In our approach, we allow users to interpret what has been done and where the DL method might go wrong through explicit confidence map prediction. The confidence map gives users concrete feedback, helping users to understand the output of the DL model. Also, by introducing the concept of a reliable filter, we are free to vary the fusion weight between the two intermediate denoised results to select the preferred candidate. Through extensive experiments, we show that the fusion process offers both flexibility and robustness against heterogeneous inputs. Meanwhile, the same infrastructure is applicable to SISR with minor changes.

There are still a few limitations of the current approach. First, since the fusion happens in the frequency domain, it would be better to have confidence prediction in the frequency domain directly, to facilitate the confidence-guided fusion. However, due to the challenges it introduces, this idea is not implemented yet. Also, currently, the confidence prediction network is trained only on different noise levels. To have a robust prediction of the confidence in a general setting, more extensive training is required.

[1]
Saeed Anwar and Nick Barnes.
Real image denoising with feature attention.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 3155–3164. IEEE, 2019.

[2]
Jason Brownlee.
Better Deep Learning: Train Faster, Reduce Overfitting, and Make
Better Predictions.
Machine Learning Mastery, 2018.

[3]
M. Munawwar Iqbal Ch, Muhammad Mohsin Riaz, Naima Iltaf, Abdul Ghafoor, and
Syed Sohaib Ali.
A multifocus image fusion using highlevel DWT components and guided
Multimedia Tools and Applications, 79(19):12817–12828, 2020.

[4]
Chang Chen, Zhiwei Xiong, Xinmei Tian, and Feng Wu.
Deep boosting for image denoising.
In Proceedings of the European Conference on Computer Vision,
pages 3–19, 2018.

[5]
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen O. Egiazarian.
Image denoising by sparse 3-D transform-domain collaborative
IEEE Transactions on image processing, 16(8):2080–2095,

[6]
Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang.
Second-order attention network for single image super-resolution.
In IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pages 11057–11066, 2019.

[7]
Terrance DeVries and Graham W Taylor.
Learning confidence for out-of-distribution detection in neural
arXiv preprint arXiv:1802.04865, 2018.

[8]
Majed El Helou and Sabine Süsstrunk.
Blind universal Bayesian image denoising with Gaussian noise
level learning.
IEEE Transactions on Image Processing, 29:4885–4897, 2020.

[9]
Jinjin Gu, Hannan Lu, Wangmeng Zuo, and Chao Dong.
Blind super-resolution with iterative kernel correction.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2019.

[10]
Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng.
Weighted nuclear norm minimization with application to image
In 2014 IEEE Conference on Computer Vision and Pattern
Recognition, pages 2862–2869, 2014.

[11]
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.
On calibration of modern neural networks.
In Proceedings of the 34th International Conference on Machine
Learning, volume 70, pages 1321–1330. PMLR, 2017.

[12]
Majed El Helou and Sabine Süsstrunk.
Bigprior: Towards decoupling learned prior hallucination and data
fidelity in image restoration.
arXiv preprint arXiv:2011.01406, 2020.

[13]
Majed El Helou, Ruofan Zhou, and Sabine Süsstrunk.
Stochastic frequency masking to improve super-resolution and
denoising networks.
In Proceedings of the European Conference on Computer Vision,
Lecture Notes in Computer Science, pages 749–766, 2020.

[14]
Dan Hendrycks and Kevin Gimpel.
A baseline for detecting misclassified and out-of-distribution
examples in neural networks.
In 5th International Conference on Learning Representations,

[15]
Harpreet Kaur, Deepika Koundal, and Virender Kadyan.
Image fusion techniques: a survey.
Archives of Computational Methods in Engineering, pages 1–23,

[16]
Yoonsik Kim, Jae Woong Soh, Gu Yong Park, and Nam Ik Cho.
Transfer learning from synthetic to real-noise denoising with
adaptive instance normalization.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 3479–3489. IEEE, 2020.

[17]
E.Y. Lam and J.W. Goodman.
A mathematical analysis of the DCT coefficient distributions for
IEEE transactions on image processing, 9(10):1661–1666, 2000.

[18]
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras,
Miika Aittala, and Timo Aila.
Noise2Noise: Learning image restoration without clean data.
In Proceedings of the 35th International Conference on Machine
Learning, volume 80, pages 2971–2980. PMLR, 2018.

[19]
Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo.
Multi-level wavelet-CNN for image restoration.
In Proceedings of the IEEE conference on computer vision and
pattern recognition workshops, pages 773–782, 2018.

[20]
Zhi-Song Liu, Li-Wen Wang, Chu-Tak Li, and Wan-Chi Siu.
Hierarchical back projection network for image super-resolution,

[21]
David R. Martin, Charless C. Fowlkes, Doron Tal, and Jitendra Malik.
A database of human segmented natural images and its application to
evaluating segmentation algorithms and measuring ecological statistics.
In Proceedings of the Eighth International Conference On
Computer Vision (ICCV-01), pages 416–425. IEEE Computer Society, 2001.

[22]
Onofre Martorell, Catalina Sbert, and Antoni Buades.
Ghosting-free DCT based multi-exposure image fusion.
Signal Processing Image Communication, 78:409–425, 2019.

[23]
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang.
Confidence-aware learning for deep neural networks.
In Proceedings of the 37th International Conference on Machine
Learning, volume 119, pages 7034–7044. PMLR, 2020.

[24]
Rekha R. Nair and Tripty Singh.
Multi-sensor medical image fusion using pyramid-based DWT: a
multi-resolution approach.
IET Image Processing, 13(9):1447–1459, 2019.

[25]
Deepak Kumar Sahu and MP Parsai.
Different image fusion techniques–a critical review.
International Journal of Modern Engineering Research (IJMER),
2(5):4298–4301, 2012.

[26]
Umme Sara, Morium Akter, and Mohammad Shorif Uddin.
Image quality assessment through FSIM, SSIM, MSE and PSNR–a
comparative study.
Journal of Computer and Communications, 7(3):8–18, 2019.

[27]
Subarna Tripathi, Zachary C Lipton, and Truong Q Nguyen.
Correction by projection: Denoising images with generative
adversarial networks.
arXiv preprint arXiv:1803.04477, 2018.

[28]
Rajeev Yasarla and Vishal M. Patel.
Uncertainty guided multi-scale residual learning-using a cycle
spinning CNN for single image de-raining.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 8405–8414, 2019.

[29]
Rajeev Yasarla and Vishal M. Patel.
Confidence measure guided single image de-raining.
IEEE Transactions on image processing, 29:4544–4555, 2020.

[30]
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.
Beyond a gaussian denoiser: Residual learning of deep CNN for image
IEEE Transactions on image processing, 26(7):3142–3155,

[31]
Kai Zhang, Wangmeng Zuo, and Lei Zhang.
Ffdnet: Toward a fast and flexible solution for cnn-based image
IEEE Transactions on image processing, 27(9):4608–4622,

[32]
Kai Zhang, Wangmeng Zuo, and Lei Zhang.
Learning a single convolutional super-resolution network for multiple
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), June 2018.

[33]
Kai Zhang, Wangmeng Zuo, and Lei Zhang.
Deep plug-and-play super-resolution for arbitrary blur kernels.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2019.

[34]
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu.
Image super-resolution using very deep residual channel attention
In Proceedings of the European Conference on Computer Vision
(ECCV), September 2018.

[35]
Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith,
and Scott Howard.
A poisson-gaussian denoising dataset with real fluorescence
microscopy images.
In 2019 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pages 11702–11710, 2019.


Jinchuan Tian1,3, Jianwei Yu1,2, Hangting Chen1, Brian Yan3, Chao Weng1,2,
Dong Yu1, Shinji Watanabe3

# Bayes Risk Transducer: Transducer with Controllable Alignment Prediction

###### Abstract

Automatic speech recognition (ASR) based on transducers is widely used. In
training, a transducer maximizes the summed posteriors of all paths. The path
with the highest posterior is commonly defined as the predicted alignment
between the speech and the transcription. While the vanilla transducer does
not have a prior preference for any of the valid paths, this work intends to
enforce the preferred paths and achieve controllable alignment prediction.
Specifically, this work proposes Bayes Risk Transducer (BRT), which uses a
Bayes risk function to set lower risk values to the preferred paths so that
the predicted alignment is more likely to satisfy specific desired properties.
We further demonstrate that these predicted alignments with intentionally
designed properties can provide practical advantages over the vanilla
transducer. Experimentally, the proposed BRT saves inference cost by up to 46%
for non-streaming ASR and reduces overall system latency by 41% for streaming
ASR.111We gratefully acknowledge the support of NVIDIA Corporation with the
donation of the GPUs used for this research. 222Code available:
https://github.com/espnet/espnet

Index Terms: speech recognition, transducer, alignment

##  1 Introduction

Automatic speech recognition (ASR) based on transducers [1] is one of the most
popular frameworks [2, 3, 4]. In the past few years, a series of approaches
have been proposed as extensions of the transducer with the goals of
optimizing its recognition accuracy [5, 6], language model integration [7],
flexibility [8], decoding efficiency [9] and simplicity [10, 11], memory
efficiency during training [12, 13, 14] and overall system latency during
streaming decoding [15, 16, 17, 18, 19, 20]. During training, the vanilla
transducer, as well as its extensions [9, 12, 13, 15, 16, 17, 20], maximizes
the summed posterior of all potential aligning sequences (a.k.a, paths)
between the speech and the transcription. In particular, these extensions
achieve their goals by manipulating the transducer paths, such as allowing
multi-frame big skips [9], pruning paths with minor posteriors with [13, 17,
20] or without [12] reference alignment labels, discouraging blank emissions
[15] and encouraging non-blank emissions [16]. This work provides another
extension of the transducer, which also conducts manipulation over paths.
Specifically, as a follow-up of the previous work [21] which attempts to
achieve controllable alignment prediction in CTC criterion [22], this work
extends this controllability to the transducer model by taking its distinctive
forward-backward process into consideration.

The alignment prediction of the transducer is commonly defined as the path
with the highest posterior. In vanilla transducer formulation, there is no
prior preference among the paths since predicting each valid path will yield
the correct textual transcription. Currently, the alignment selection among
the paths (i.e., which path will become the predicted alignment) can hardly be
affected by human intervention during training. To achieve controllable
alignment prediction in the transducer is exactly to intentionally choose
paths with specific desired properties as the alignment prediction. With this
motivation, this work proposes an extension of the transducer called Bayes
Risk Transducer (BRT), which adopts a Bayes risk function to intentionally
enforce a preference for paths with the desired properties, so that the
predicted alignments are more likely to be characterized by these properties.
Particularly, the original forward-backward algorithm of the transducer is
revised into a divide-and-conquer approach: all paths are firstly divided into
multiple exclusive groups and the groups with more favored properties are
enforced by receiving lower risk values than the others.

This work further demonstrates that BRT with controllable alignment prediction
has practical advantages over vanilla transducers. By designing various Bayes
risk functions, we can obtain alignment predictions with desired properties
that are specific to different task setups, which subsequently helps to offer
novel solutions for two practical challenges in ASR: inference cost for non-
streaming ASR and overall system latency for streaming ASR. In the non-
streaming setup, a Bayes risk function is designed to enforce the paths that
emit the last non-blank predictions earlier. As a benefit, the last non-blank
prediction occurs at an early time stamp so the inference cost can be reduced
by terminating the decoding loop early without exploring all frames. In the
streaming setup, another Bayes risk function is designed to encourage early
emissions for all non-blank tokens. Thus, the model emits before waiting too
long context and the latency for each non-blank token is reduced.
Experimentally, the former case accelerates non-streaming inference by up to
46% and the latter case reduces the overall system latency of streaming ASR
system by 41%.

##  2 Bayes Risk Transducer

(a) Transducer Lattice

(b) $\log\mathcal{G}(t,u)$ \- Vanilla Transducer

(c) $\log\mathcal{G}(t,u)$ \- BRT

Figure 1: (a): Transducer lattice. $\mathcal{G}(4,2)$ is the summed posterior
of all paths that go through the vertical arrow (in green) from node $(4,1)$
to node $(4,2)$ and emit 2nd token at the 4th frame. The red path ends all
non-blank predictions at $\tau=3$ while the blue path ends at $\tau=6$. The
red path is preferred in Sec.3.2. (b) & (c): The heat maps for
$\log\mathcal{G}(t,u)$.

###  2.1 Vanilla Transducer

In training, vanilla transducer maximizes the posterior of the transcription
$\mathbf{l}=[l_{1},...,l_{U}]$ with the given acoustic feature sequence
$\mathbf{x}=[\mathbf{x}_{1},...,\mathbf{x}_{T}]$. Instead of maximizing
$P(\mathbf{l}|\mathbf{x})$ directly, the transducer maximizes the summed
posterior of all paths in the transducer lattice (see Fig.1.a). Note
$\varnothing$ as the blank symbol and extend the vocabulary
$\hat{\mathcal{V}}=\mathcal{V}\cup\\{\varnothing\\}$, each symbol sequence
$\mathbf{\pi}=[\pi_{1},...,\pi_{T+U}]$ is a valid path if all entries of
$\mathbf{\pi}$ are in $\hat{\mathcal{V}}$ and
$\mathcal{B}(\mathbf{\pi})=\mathbf{l}$. Here $\mathcal{B}$ is a mapping that
removes all $\varnothing$. So the vanilla transducer objective to minimize is
defined as:

$\mathbf{J}_{\text{transducer}}(\mathbf{l},\mathbf{x})\triangleq-\log
P(\mathbf{l}|\mathbf{x})=-\log\sum_{\mathbf{\pi}\in\mathcal{B}^{-1}(\mathbf{l})}P(\mathbf{\mathbf{\pi}}|\mathbf{x})$
(1)

where $\mathcal{B}^{-1}(\mathbf{l})$ is the set of all valid paths. Next, the
posterior of each path $P(\mathbf{\pi}|\mathbf{x})$ is computed as:

$P(\mathbf{\pi}|\mathbf{x})=\prod_{i=1}^{T+U}p(\pi_{i}|\mathbf{x}_{1:t},\mathbf{l}_{1:u})$
(2)

where the condition $(\mathbf{x}_{1:t},\mathbf{l}_{1:u})$ specifies the node
$(t,u)$ on the transducer lattice s.t.
$\mathcal{B}(\mathbf{\pi}_{1:i})=\mathbf{l}_{1:u}$ and $t+u=i-1$. Instead of
enumerating all paths and summing their posteriors, the transducer objective
is computed efficiently by forward-backward algorithm [23], which recursively
computes the forward-backward variables $\alpha(t,u)$ and $\beta(t,u)$ for
each node $(t,u)$ in the transducer lattice:

$\alpha(t,u)=\sum_{\mathbf{\pi}\in\mathcal{V^{\prime}}^{(T+U)},\mathcal{B}(\mathbf{\pi}_{1:t+u})=\mathbf{l}_{1:u}}P(\mathbf{\pi}|\mathbf{x})$
(3)
$\beta(t,u)=\sum_{\mathbf{\pi}\in\mathcal{V^{\prime}}^{(T+U)},\mathcal{B}(\mathbf{\pi}_{t+u+1:T+U})=\mathbf{l}_{u+1:U}}P(\mathbf{\pi}|\mathbf{x})$
(4)

Subsequently, by decomposing each path $\mathbf{\pi}$ into partial paths
$\mathbf{\pi}_{1:t+u}$ and $\mathbf{\pi}_{t+u+1:T+U}$ and using Eq.{3, 4}, the
transducer objective is derived as333Note, for a path that go through a known
node $(t,u)$ in transducer lattice, the partial path
$\mathbf{\pi}_{t+u+1:T+U}$ is independent to partial path
$\mathbf{\pi}_{1:t+u}$ so that the factorization of the path posterior is
$P(\mathbf{\pi}|\mathbf{x})=P(\mathbf{\pi}_{1:t+u}|\mathbf{x})\cdot
P(\mathbf{\pi}_{t+u+1:T+U}|{\cancel{\mathbf{\pi}_{1:t+u}}},\mathbf{x})$.:

$\displaystyle\mathbf{J}_{\text{transducer}}(\mathbf{l},\mathbf{x})$
$\displaystyle=-\log\sum_{\mathbf{\pi}\in\mathcal{B}^{-1}(\mathbf{l})}P(\mathbf{\mathbf{\pi}}|\mathbf{x})$
$\displaystyle=$
$\displaystyle-\log\sum_{(t,u):t+u=n}\sum_{\mathcal{B}(\mathbf{\pi}_{1:t+u})=\mathbf{l}_{1:u}\atop\mathcal{B}(\mathbf{\pi}_{t+u+1:T+U})=\mathbf{l}_{u+1:U}}P(\mathbf{\mathbf{\pi}}|\mathbf{x})$
$\displaystyle=$
$\displaystyle-\log\sum_{(t,u):t+u=n}\alpha(t,u)\cdot\beta(t,u)$ (5)

where $n$ is any known integer s.t. $n\in[0,T+U]$. Finally, the path with the
highest posterior is usually considered as the alignment prediction between
$\mathbf{x}$ and $\mathbf{l}$:
$\text{ali}(\textbf{x},\textbf{l})=\arg\max_{\mathbf{\pi\in\mathcal{B}^{-1}(\mathbf{l})}}P(\mathbf{\pi}|\mathbf{x})$.

(a) WER / CER%

(b) Decoding Frames (DF)

(c) Real Time Factor (RTF)

Figure 2: Results of non-streaming ASR. (a) Recognition accuracy in CER / WER
%; (b) Average decoding frames (DF); (c) Real time factor (RTF). With
comparable recognition accuracy, the proposed BRT achieve more efficient
decoding by decoding fewer frames.

###  2.2 Bayes Risk Transducer

As suggested in Eq.1, the formulation of the vanilla transducer has no prior
preference among paths. This work intentionally selects the predicted
alignment among the paths and thus attempts to achieve controllable alignment
prediction. With this motivation, this work proposes Bayes Risk Transducer
(BRT), which adopts a customizable Bayes risk function to express the
preference for specific paths with desired properties. To preserve a similar
format like Eq.1, we sets the risk function for each path as
$-r(\mathbf{\pi})$ so that minimizing the expected risk is equivalent to
minimizing the BRT objective 444In the remained part of this work,
$r(\mathbf{\pi})$ is termed as the Bayes risk function even though the real
Bayes risk function is $-r(\mathbf{\pi})$. Higher $r(\mathbf{\pi})$ value
represents a lower risk, a.k.a., preference.:

$\mathbf{J}_{\text{BRT}}(\mathbf{l},\mathbf{x})\triangleq-\log\sum_{\mathbf{\pi}\in\mathcal{B}^{-1}(\mathbf{l})}[P(\mathbf{\mathbf{\pi}}|\mathbf{x})\cdot
r(\mathbf{\pi})]$ (6)

The computation of the proposed BRT objective still adopts the forward-
backward variables and works in a divide-and-conquer approach. To express the
preference for specific property of the paths, the Bayes risk function
specifies 1) what property is concerned and 2) what values of the concerned
property are preferred. To answer these questions, the concerned property of
each path is defined by $f(\mathbf{\pi})$. Then, all paths are divided into
multiple exclusive groups s.t. paths with the identical concerned property
value $\tau$ are in the same groups. For paths in one identical group, it is
reasonable to assign the same risk value as the concerned property is the
same. Thus, the Bayes risk function $r(\mathbf{\pi})$ is replaced by a group-
level risk function $r_{g}(\tau)$, which only depends on the group-level
concerned property $\tau$ rather than the path $\mathbf{\pi}$. Formally, this
process is written as:

$\displaystyle\mathbf{J}_{\text{BRT}}(\mathbf{l},\mathbf{x})$
$\displaystyle=-\log\sum_{\tau}\sum_{\mathbf{\pi}\in\mathcal{B}^{-1}(\mathbf{l}),f(\mathbf{\pi})=\tau}[P(\mathbf{\mathbf{\pi}}|\mathbf{x})\cdot
r(\mathbf{\pi})]$
$\displaystyle=-\log\sum_{\tau}\sum_{\mathbf{\pi}\in\mathcal{B}^{-1}(\mathbf{l}),f(\mathbf{\pi})=\tau}[P(\mathbf{\mathbf{\pi}}|\mathbf{x})\cdot
r_{g}(\mathbf{\tau})]$
$\displaystyle=-\log\sum_{\tau}[r_{g}(\tau)\cdot\sum_{\mathbf{\pi}\in\mathcal{B}^{-1}(\mathbf{l}),f(\mathbf{\pi})=\tau}P(\mathbf{\mathbf{\pi}}|\mathbf{x})]$
(7)

Please be aware of: 1) when splitting the path into groups, the groups are
supposed to be mutually exclusive so that each path is considered once and
only once; 2) by adopting the group-level risk function, we can avoid the
complex weighted summation over all paths within each group. 3) the summed
posterior of each path group should be fully tractable by the forward-backward
variables so the computation remains efficient; 4) by pursuing the desired
properties in the predicted alignment between $\mathbf{x}$ and $\mathbf{l}$
during training, these properties are expected to be preserved in the
predicted alignments between the test speech and the textual hypotheses during
decoding.

Provided the general formulation of BRT in Eq.2.2, a naive example is in
Eq.2.1, where the concerned property $\tau$ represents the pair $(t,u)$ and
$r_{g}(\tau)=1$. Under this setting, the vanilla transducer is a special case
of the proposed BRT. Alternatively, given the $u$-th non-blank token $l_{u}$
in the transcription, another useful example is to set the concerned property
$\tau$ as the time stamp when $l_{u}$ is emitted, a.k.a, $\pi_{\tau+u}=l_{u}$.
With a similar factorization like Eq.2.1 and considering Eq.{2, 3, 4}, the BRT
objective is further revised as:

$\displaystyle\mathbf{J}_{\text{BRT}}(\mathbf{l},\mathbf{x},u)=-\log\sum_{\tau}[r_{g}(\tau)\cdot\sum_{\mathbf{\pi}\in\mathcal{B}^{-1}(\mathbf{l}),\pi_{\tau+u}=l_{u}}P(\mathbf{\mathbf{\pi}}|\mathbf{x})]$
$\displaystyle=-\log\sum_{\tau}[r_{g}(\tau)\cdot\sum_{\begin{subarray}{c}\mathcal{B}(\mathbf{\pi}_{1:\tau+u-1})=\mathbf{l}_{1:u-1}\\\
\pi_{\tau+u}=l_{u}\\\
\mathcal{B}(\mathbf{\pi}_{\tau+u+1:T+U})=\mathbf{l}_{u+1:U}\end{subarray}}P(\mathbf{\mathbf{\pi}}|\mathbf{x})]$
$\displaystyle=-\log\sum_{\tau}r_{g}(\tau)\cdot\underbrace{[\alpha(\tau,u-1)\cdot
p(l_{u}|\mathbf{x}_{1:\tau},\mathbf{l}_{1:u-1})\cdot\beta(\tau,u)]}_{\triangleq\
\ \mathcal{G}(\tau,u)}$ (8)

Here is $\mathcal{G}(\tau,u)$ the summed posterior of all paths that go
through the vertical arrow from node $(\tau,u-1)$ to node $(\tau,u)$. Fig.1.a
gives a demonstration of $\mathcal{G}(\tau,u)$ in the lattice and Fig.1.{b, c}
provides numerical examples of $\mathcal{G}(\tau,u)$. $\mathcal{G}(\tau,u)$
measures the summed probability of all valid paths that $l_{u}$ is emitted at
$\tau$-th frame, which indicates the alignment prediction. So far the group-
level risk function $r_{g}(\tau)$ is not defined. Below we show two
applications of Eq.2.2 with different $r_{g}(\tau)$ designs.

###  2.3 Non-streaming Application: Efficient Decoding

For frame-synchronized decoding algorithms of the transducer [1, 24], the
inference cost highly depends on $T$ as the decoding loop is conducted frame-
by-frame. In Fig.1.b, the whole sequence $\mathbf{l}$ cannot be predicted
until all frames are explored. By contrast, in Fig.1.c, all non-blank tokens
are emitted before reaching the red line, which allows us to stop decoding at
an early time stamp (e.g., the red line) to save computation.

To achieve the heat map like Fig.1.c, the concerned property is exactly the
time stamp $\tau$ when the last token $l_{U}$, as well as the whole sequence,
is emitted: $\pi_{\tau+U}=l_{U}$. In addition, paths with smaller $\tau$ are
preferred (see Fig.1.a) since fewer frames are consumed to predict all tokens.
Set $u=U$ in Eq.2.2, the objective to minimize is:

$\mathbf{J}_{\text{BRT}}(\mathbf{l},\mathbf{x},U)=-\log\sum_{\tau}\text{min}(e^{-\lambda\cdot(\tau-m\cdot
U)/T},1)\cdot\mathcal{G}(\tau,U)$ (9)

where the risk function $r_{g}(\tau)=\text{min}(e^{-\lambda\cdot(\tau-m\cdot
U)/T},1)$ expresses the preference for $\tau\in[1,m\cdot U]$ and shows
exponentially decayed interest in $\tau>m\cdot U$555We do not express an extra
preference for very small $\tau$ so it is less likely that multiple non-blank
tokens are emitted at a single frame.. $\lambda$ and $m$ are hyper-parameters.
$m$ is empirically set to 2 and $\lambda$ varies according to datasets.

This work further provides an early-stop mechanism to reduce the number of
decoding frames of BRT. First, assume we obtain a hypothesis
$\mathbf{\hat{l}}=[\hat{l}_{1},...,\hat{l}_{u}]$ at $\tau$-th frame during
decoding. The hypothesis is considered complete if no additional non-blank
tokens are expected to be emitted in the search process over the remaining
frames after $\tau$. In other words, for any possible path of the complete
$\mathbf{\hat{l}}$, its sub-path after the $\tau$-th frame only consists of
continuous $\varnothing$ (see the blue line in Fig.1.c). So $\mathbf{\hat{l}}$
is considered complete only if the accumulated probability of the continuous
$\varnothing$ since the $\tau$-th frame are with high confidence:
$\sum_{t=\tau}^{T}\log
p(\varnothing|\mathbf{x}_{1:t},\mathbf{\hat{l}}_{1:u})>D$, where $D=-10$ is a
threshold value666The computation for this condition cannot be considered as a
search over the frames after $\tau$ since the series
$\\{p(\varnothing|\mathbf{x}_{1:t},\mathbf{\hat{l}}_{1:u})\\}$ can be computed
in parallel fashion and requires no loop.. Secondly, for a search beam that
contains multiple hypotheses, we terminate the search when 1) the top $k=3$
blank-free hypotheses do not change for $f=5$ frames and 2) all top $k=3$
hypotheses are considered complete.

###  2.4 Streaming Application: Early Emission

A streaming ASR system is expected to emit each token accurately and timely.
The accuracy and latency, however, usually form a trade-off: better
recognition accuracy requires longer context, which results in higher latency.
For streaming ASR, BRT is designed to encourage all tokens to emit at early
time stamps, even at the cost of slight performance degradation. By doing so,
BRT achieves a better accuracy-latency trade-off than the vanilla transducer,
which is further demonstrated in Sec.3.3.

The vanilla transducer only attempts to transcribe the speech correctly but
poses no constraint on when tokens would be emitted. By contrast, the proposed
BRT can reduce the latency by enforcing the paths that emit each token at a
smaller time stamp. Formally, with any non-blank token $l_{u}$ and the
exponentially decayed risk function
$r_{g}(\tau)=e^{-\lambda\cdot(\tau-\tau^{\prime})/T}$, a BRT objective is
derived from Eq.2.2 with the goal of encouraging $l_{u}$ to be emitted
earlier:

$\mathbf{J}_{\text{BRT}}(\mathbf{l},\mathbf{x},u)=-\log\sum_{\tau}e^{-\lambda\cdot(\tau-\tau^{\prime})/T}\cdot\mathcal{G}(\tau,u)$
(10)

where $\tau$ is the concerned property that specifies the time-stamp when
$l_{u}$ is emitted and is enforced to be smaller;
$\tau^{\prime}=\arg\max_{\tau}\mathcal{G}(\tau,u)$ is a bias term to ensure
that the path group with the highest summed posterior $\mathcal{G}(\tau,u)$
would always receive the risk value of $r_{g}(\tau)=1$ so that the absolute
value of $\mathbf{J}_{\text{BRT}}(\mathbf{l},\mathbf{x},u)$ does not vary
along with $u$ significantly. Here $\lambda$ is still an adjustable hyper-
parameter varying with datasets. Subsequently, to guide every token $l_{u}$ to
be emitted earlier requires the consideration of all tokens. So we simply
attempts to minimize the mean of
$\mathbf{J}_{\text{BRT}}(\mathbf{l},\mathbf{x},u)$ in Eq.10 over every $u$:

$\mathbf{J}(\mathbf{l},\mathbf{x})=\frac{1}{U}\cdot\sum_{u=1}^{U}\mathbf{J}_{\text{BRT}}(\mathbf{l},\mathbf{x},u)$
(11)

##  3 Experiments

(a) Demonstration of drift latency

(b) Aishell-1

(c) Aishell-2

(d) Librispeech-100

Figure 3: Results of streaming ASR. (a) A demonstration of drift latency (DL).
Blue arrows stand for the reference duration of each token. The alignment of
the hypothesis is represented by the red path. Token $l_{2}$ starts at 1st
frame but is predicted at 4th frame, which is a 3-frame drift latency. (b) &
(c) & (d): the accuracy-latency trade-off achieved with varying data
collecting latency (DCL)

.

###  3.1 Experimental Setup

Datasets: Experiments are conducted on Aishell-1 [25], Aishell-2 [26] and
Librispeech-100 [27] datasets. The volumes of these datasets range from 100
hours to 1k hours. Librispeech-100 is in English and the others are in
Mandarin. All data is augmented by SpecAugment [28] and speed perturbation.
For English, tokens are 500 BPE units.
Evaluation Metrics: CER / WER% is adopted to show the recognition accuracy. To
compare the decoding efficiency of non-streaming ASR, the average number of
decoding frames (DF) before the decoding termination and the real-time factor
(RTF) over CPU777Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz, single thread.
are reported. For streaming ASR, the overall latency is defined as the sum of
data collecting latency (DCL) and drift latency (DL) [21]888The latency caused
by computation is not considered in this work: it is marginal for a light
model and without an external language model.. DCL is the time to wait before
the input speech forms a chunk (a.k.a., the latency caused by chunk size and
look-ahead length). DL is exemplified in Fig.3.a. and its reference999For an
English word that consists of multiple BPE tokens, we only count the last BPE
unit of that word. is obtained by standard GMM-HMM systems101010Models are
trained by Kaldi: https://github.com/kaldi-asr/kaldi.
Features & Models: 80-dim Fbank features with the window size of 10ms are
down-sampled by 4x using CNN before being fed into the encoder. The acoustic
encoder is Conformer [29] for non-streaming ASR and Emformer [30] for
streaming ASR. For streaming ASR, DCL is set to {160, 320, 480, 640}ms. The
prediction network is a standard LSTM and the joint network is linear. For
English tasks, an auxiliary CTC criterion is adopted on the top of the encoder
to stabilize the training. Model sizes for non-streaming and streaming
experiments are 95M and 57M respectively.
Training & Decoding: For {Aishell-1, Aishell-2, Librispeech-100}, models are
trained for {100, 100, 300} epochs with $\lambda$ of {5, 5, 20} and {10, 10,
50} for non-streaming and streaming ASR respectively. The original decoding
algorithm proposed in [1] is adopted with a beam size of 10. No language model
is adopted in decoding.

###  3.2 Results on non-streaming ASR

This part evaluates the effectiveness of the proposed BRT method on non-
streaming ASR. Our results are shown in Fig.2. Firstly, as shown in Fig.2.a,
with datasets in varying scales and languages, the recognition accuracy
achieved by the proposed BRT method and the vanilla transducer are comparable.
Secondly, Fig.2.b demonstrates the effectiveness of the proposed BRT in
reducing the decoding frames. E.g., by introducing the BRT criterion, the DF
for Aishell-2 dataset is reduced from 71 to 16, which is a 77% reduction.
Finally, for models trained by BRT, the overall inference cost (a.k.a., RTF)
is reduced since the proposed early-stop mechanism allows the model not to
explore all the frames. By adopting the BRT criterion and the early-stop
mechanism, the RTF of Aishell-1 is reduced from 0.25 to 0.14, which is a 46%
relative reduction. The reduction in Fig.2.c is not as considerable as that in
Fig.2.b since the encoder inference accounts for a large part of the
computation cost.

###  3.3 Results on streaming ASR

This part evaluates the effectiveness of the proposed BRT method on streaming
ASR. Our results are shown in Fig.3. As discussed in Sec.2.4, the streaming
ASR has a trade-off between the recognition accuracy and the overall system
latency. As shown in Fig.3.{b,c,d}, on the three datasets, the curve of the
proposed BRT (the red one) consistently lies in the lower-left direction of
its baseline (vanilla transducer, the blue one), which suggested that the
proposed BRT criterion achieves better accuracy-latency trade-off than vanilla
transducer. In addition, BRT can build systems with extremely low latency that
cannot be achieved by the vanilla transducer, even at the cost of recognition
performance degradation. E.g., on the Aishell-2 dataset, the lowest overall
latency achieved by the vanilla transducer and BRT is 430ms and 251ms
respectively, which is a 41% relative reduction in latency, even with accuracy
degradation.

Further ablation study is conducted on Aishell-1 dataset. As shown in Fig.4.a,
the transducer system with extremely low latency cannot be built by simply
reducing the chunk size (a.k.a., small DCL) since the model is allowed to wait
for very long context before emitting (a.k.a., larger DL). In addition, the
adoption of BRT can effectively reduce the DL, which is aligned with our
motivation in Sec.2.4111111The DL can be negative due to the look-ahead of the
model. The BRT model has this strength of early emission since the paths that
emit non-blank prediction earlier are enforced during training. Next, Fig.4.b
shows that the vanilla transducer outperforms the proposed BRT in accuracy
with all DCL settings, which is reasonable since the accessible right context
is reduced if the non-blank tokens are emitted earlier. Combining Fig.4.{a,b}
will reach Fig.3.b, which demonstrates that: BRT provides an alternative
solution for streaming transducer, i.e., increasing DCL with a larger chunk
size and reducing DL by using BRT to meet the latency budget so that a better
overall accuracy-latency trade-off is achieved.

(a) DCL vs. DL

(b) DCL vs. CER

Figure 4: Ablation study for streaming ASR (on Aishell-1)

##  4 Conclusion

To achieve controllable alignment prediction in the transducer, this work
proposes an extension of the transducer called Bayes Risk Transducer (BRT),
which adopts a Bayes risk function to enforce specific paths with the desired
properties. By designing different Bayes risk functions, the predicted
alignment is enriched with task-specific properties, which provides practical
benefits besides recognizing the speech accurately: efficient decoding for
non-streaming ASR and early emission for streaming ASR. The claimed two
applications are experimentally validated on multiple datasets and in multiple
languages.

## References

  * [1] A. Graves, ``Sequence transduction with recurrent neural networks,'' _arXiv preprint arXiv:1211.3711_ , 2012.
  * [2] R. Prabhavalkar, T. Hori, T. N. Sainath, R. Schlüter, and S. Watanabe, ``End-to-end speech recognition: A survey,'' _arXiv preprint arXiv:2303.03329_ , 2023.
  * [3] D. Wang, X. Wang, and S. Lv, ``An overview of end-to-end automatic speech recognition,'' _Symmetry_ , vol. 11, no. 8, 2019.
  * [4] J. Li, ``Recent advances in end-to-end automatic speech recognition,'' _arXiv preprint arXiv:2111.01690_ , 2021.
  * [5] H. Sak, M. Shannon, K. Rao, and F. Beaufays, ``Recurrent neural aligner: An encoder-decoder neural network model for sequence to sequence mapping,'' _Proc. Interspeech 2017_ , pp. 1298–1302, 2017.
  * [6] T. N. Sainath, R. Pang, D. Rybach, Y. He, R. Prabhavalkar, W. Li, M. Visontai, Q. Liang, T. Strohman, Y. Wu, I. McGraw, and C.-C. Chiu, ``Two-Pass End-to-End Speech Recognition,'' in _Proc. Interspeech 2019_ , 2019, pp. 2773–2777.
  * [7] E. Variani, D. Rybach, C. Allauzen, and M. Riley, ``Hybrid autoregressive transducer (hat),'' in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 6139–6143.
  * [8] N. Moritz, T. Hori, S. Watanabe, and J. L. Roux, ``Sequence transduction with graph-based supervision,'' in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 7212–7216.
  * [9] H. Xu, F. Jia, S. Majumdar, S. Watanabe, and B. Ginsburg, ``Multi-blank transducers for speech recognition,'' _arXiv preprint arXiv:2211.03541_ , 2022\.
  * [10] M. Ghodsi, X. Liu, J. Apfel, R. Cabrera, and E. Weinstein, ``Rnn-transducer with stateless prediction network,'' in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 7049–7053.
  * [11] A. Tripathi, H. Lu, H. Sak, and H. Soltau, ``Monotonic recurrent neural network transducer and decoding strategies,'' in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2019, pp. 944–948.
  * [12] F. Kuang, L. Guo, W. Kang, L. Lin, M. Luo, Z. Yao, and D. Povey, ``Pruned RNN-T for fast, memory-eﬀicient ASR training,'' in _Proc. Interspeech 2022_ , 2022, pp. 2068–2072.
  * [13] J. Li, R. Zhao, H. Hu, and Y. Gong, ``Improving rnn transducer modeling for end-to-end speech recognition,'' in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2019, pp. 114–121.
  * [14] A. Zeyer, A. Merboldt, R. Schlüter, and H. Ney, ``A New Training Pipeline for an Improved Neural Transducer,'' in _Proc. Interspeech 2020_ , 2020, pp. 2812–2816.
  * [15] J. Yu, C.-C. Chiu, B. Li, S.-y. Chang, T. N. Sainath, Y. He, A. Narayanan, W. Han, A. Gulati, Y. Wu, and R. Pang, ``Fastemit: Low-latency streaming asr with sequence-level emission regularization,'' in _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6004–6008.
  * [16] W. Kang, Z. Yao, F. Kuang, L. Guo, X. Yang, P. Żelasko, D. Povey _et al._ , ``Delay-penalized transducer for low-latency streaming asr,'' _arXiv preprint arXiv:2211.00490_ , 2022.
  * [17] J. Mahadeokar, Y. Shangguan, D. Le, G. Keren, H. Su, T. Le, C.-F. Yeh, C. Fuegen, and M. L. Seltzer, ``Alignment restricted streaming recurrent neural network transducer,'' in _2021 IEEE Spoken Language Technology Workshop (SLT)_ , 2021, pp. 52–59.
  * [18] Y. Shinohara and S. Watanabe, ``Minimum latency training of sequence transducers for streaming end-to-end speech recognition,'' in _Proc. Interspeech 2022_ , 2022, pp. 2098–2102.
  * [19] J. Kim, H. Lu, A. Tripathi, Q. Zhang, and H. Sak, ``Reducing Streaming ASR Model Delay with Self Alignment,'' in _Proc. Interspeech 2021_ , 2021, pp. 3440–3444.
  * [20] T. N. Sainath, R. Pang, D. Rybach, B. García, and T. Strohman, ``Emitting Word Timings with End-to-End Models,'' in _Proc. Interspeech 2020_ , 2020, pp. 3615–3619.
  * [21] J. Tian, B. Yan, J. Yu, C. WENG, D. Yu, and S. Watanabe, ``BAYES RISK CTC: CONTROLLABLE CTC ALIGNMENT IN SEQUENCE-TO-SEQUENCE TASKS,'' in _The Eleventh International Conference on Learning Representations_ , 2023.
  * [22] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 369–376.
  * [23] L. Rabiner and B. Juang, ``An introduction to hidden markov models,'' _IEEE ASSP Magazine_ , vol. 3, no. 1, pp. 4–16, 1986.
  * [24] G. Saon, Z. Tüske, and K. Audhkhasi, ``Alignment-length synchronous decoding for rnn transducer,'' in _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 7804–7808.
  * [25] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, ``Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline,'' in _2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA)_ , 2017, pp. 1–5.
  * [26] J. Du, X. Na, X. Liu, and H. Bu, ``Aishell-2: Transforming mandarin asr research into industrial scale,'' _arXiv preprint arXiv:1808.10583_ , 2018\.
  * [27] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: An asr corpus based on public domain audio books,'' in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5206–5210.
  * [28] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, ``SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,'' in _Proc. Interspeech 2019_ , 2019, pp. 2613–2617.
  * [29] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, ``Conformer: Convolution-augmented Transformer for Speech Recognition,'' in _Proc. Interspeech 2020_ , 2020, pp. 5036–5040.
  * [30] Y. Shi, Y. Wang, C. Wu, C.-F. Yeh, J. Chan, F. Zhang, D. Le, and M. Seltzer, ``Emformer: Efficient memory transformer based acoustic model for low latency streaming speech recognition,'' in _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6783–6787.