# USDnet: Unsupervised Speech Dereverberation via Neural Forward Filtering

Zhong-Qiu Wang Manuscript received on Jan. 30, 2023.Z.-Q. Wang is with
Carnegie Mellon University, Pittsburgh, PA, USA (e-mail:
wang.zhongqiu41@gmail.com).

###### Abstract

In reverberant conditions with a single speaker, each far-field microphone
records a reverberant version of the same speaker signal at a different
location. In over-determined conditions, where there are more microphones than
speakers, each recorded mixture signal can be leveraged as a constraint to
narrow down the solutions to target anechoic speech and thereby reduce
reverberation. Equipped with this insight, we propose USDnet, a novel deep
neural network (DNN) approach for unsupervised speech dereverberation (USD).
At each training step, we first feed an input mixture to USDnet to produce an
estimate for target speech, and then linearly filter the DNN estimate to
approximate the multi-microphone mixture so that the constraint can be
satisfied at each microphone, thereby regularizing the DNN estimate to
approximate target anechoic speech. The linear filter can be estimated based
on the mixture and DNN estimate via neural forward filtering algorithms such
as forward convolutive prediction. We show that this novel methodology can
promote unsupervised dereverberation of single-source reverberant speech.

###### Index Terms:

Unsupervised neural speech dereverberation

##  I Introduction

Room reverberation, caused by signal reflections inside reverberant
enclosures, is very detrimental to machine perception in tasks such as
automatic speech recognition (ASR) and speaker recognition [1]. It also
dramatically degrades human hearing, leading to much lower speech
intelligibility and quality. Rereverberation results from a linear convolution
between a room impulse response and the source signal, and speech
dereverberation is known as a classic blind deconvolution problem [2, 1, 3],
which is ill-posed in nature and difficult to be solved as only the convolved
signal is observed but the filter and source are both unknown and need to be
estimated. There are infinite solutions to the filter and source pair, where,
in each solution, their linear-convolution result can well-approximate the
observed reverberant speech. For example, even if the source estimate is a
reverberant version of the source, there still exists a filter such that
convolving the source estimate with the filter can roughly approximate the
mixture. The key to successful speech dereverberation, we believe, is to
cleverly narrow down the infinite solutions to the source and filter so that
the models or algorithms can have a clear idea about what target signals to
predict.

A popular approach for speech dereverberation is based on inverse filtering
[1, 4, 5, 6], mainly leveraging signal processing principals. Representative
algorithms include weighted prediction error (WPE) [7, 8, 9], where a linear
filter is computed to filter past observations with a prediction delay to
estimate late reverberation to remove. The rationale is that the speech
signals beyond the prediction delay in past frames is little linearly-
correlated with the target anechoic signal at the current frame. WPE leverages
this physical principal to narrow down the solutions to target speech in an
unsupervised way. It has gradually become the most popular and successful
dereverberation algorithm [6], and often produces consistent improvement for
modern ASR systems. Subsequent studies along this line of research combine WPE
with beamformers in a sequential [6] or in a jointly-optimal [10, 11] way.

Another popular approach is based on supervised learning, where anechoic and
reverberant speech pairs are synthesized via room simulation and a supervised
DNN is trained to predict the anechoic speech from reverberant speech [12, 13,
14, 15, 16, 17, 18, 19, 20, 21]. The target anechoic speech serves as the
training label, and the DNN can learn to find the true solution to target
speech through supervised learning. Recent studies along this line of research
leverage target speech estimated by supervised DNNs to improve signal
processing based dereverberation [22, 23, 24, 25, 26, 27], and leverage
generative modeling [28] to improve dereverberation. This approach is often
effective as the DNN has strong capabilities at modeling speech patterns that
can be very informative for dereverberation.

In this context, we propose to investigate unsupervised neural speech
dereverberation. Different from signal processing based approaches, which are
unsupervised in nature and usually not designed to learn speech patterns from
massive training data, we leverage unsupervised deep learning to model speech
patterns. Different from supervised learning based approaches, unsupervised
neural dereverberation models can be trained directly on existing reverberant
speech, avoiding using simulated training data which often incurs
generalization issues [29, 30, 31, 32, 33].

The key to unsupervised neural speech dereverberation, we believe, is to
cleverly narrow down the infinite solutions to the source and filter so that
DNNs can receive a clear supervision on what target signals to predict, or a
clear regularization on what estimates produced by the DNN are not good and
should be avoided. This way, the strong modeling capability of DNNs on speech
signals can be sufficiently utilized for dereverberation, rather than being
wasted due to obscure supervision. Motivated by the UNSSOR algorithm [34],
which deals with unsupervised neural speaker separation and does not perform
any dereverberation, we assume the availability of over-determined training
mixtures (i.e., multi-channel reverberant speech in single-speaker cases), and
leverage each mixture signal, during training, as a mixture constraint to
narrow down the solutions to the source and filter. This way, the DNN can (a)
receive a clear supervision about what the target speech is by checking
whether its prediction satisfies the mixture constraints during training; and
(b) be trained directly on a set of collected reverberant signals to realize
dereverberation. This paper makes three major contributions:

  * •

We, for the first time, formulate unsupervised neural speech dereverberation
as a blind deconvolution problem, which requires estimating both the source
and the filter.

  * •

We propose USDnet, which estimates target (source) speech by modeling speech
patterns via unsupervised deep learning. Based on the source estimate and
input mixture, filter estimation is formulated as a linear regression problem,
which can be solved via a neural forward filtering algorithm named forward
convolutive prediction (FCP) [27] in multiple ways.

  * •

Following the blind deconvolution problem, we propose novel mixture-constraint
loss functions, which leverage multi-microphone mixtures as constraints to
regularize the DNN prediction to have it approximate target speech.

Evaluation results on two speech dereverberation tasks show that USDnet can
effectively reduce reverberation. A sound demo page is provided in the link
below.111See https://zqwang7.github.io/demos/USDnet_demo/index.html.

##  II Related Work

Unsupervised neural speech separation, mainly being studied for speech de-
noising, separating mixture of multiple anechoic speakers, and separating
multi-speaker mixtures to reverberant speaker images without any
dereverberation, has attracted consistent research interests in recent years.
However, unsupervised neural dereverberation has been largely under-explored,
although dereverberation is an important task in speech separation and
enhancement.

MixIT [35, 32, 36], a representative algorithm, first synthesizes mixtures of
mixtures (MoM), each created by mixing existing recorded mixtures, and then
trains a DNN to separate each MoM to a number of sources such that the
separated sources can be partitioned into groups and, within each group, the
separated sources can sum up to one of the mixtures used for synthesizing the
MoM. Although MixIT is shown capable of separating mixtures of independent
reverberant sources, it cannot reduce the reverberation of each source as the
reflected signals of each source are dependent with each other. Noisy-target
training [37, 38] is an unsupervised approach for speech de-noising. Similarly
to MixIT, it synthetically mixes noisy speech with noise, and trains DNNs to
predict the noisy speech based on the synthetic mixtures. This approach is
found capable of reducing noise as speech and noise are independent. However,
whether it would be effective at dereverberation is unknown, especially when
dealing with reflection signals that are highly dependent with each other.

Another approach requires source prior models, which can be built based on
clean anechoic speech [39, 40, 41, 42], or anechoic and reverberant speech
with some annotations [43]. The prior models can, to some extent, narrow down
the solutions to target speech and provide a supervision on whether DNN
estimates are desired or not during training or inference.

There are approaches leveraging pseudo-labels produced by conventional
algorithms such as spatial clustering and independent vector analysis (IVA)
for training unsupervised neural source separation models [44, 45, 46, 47],
leveraging pseudo-labels provided by DNN-supported WPE for training
unsupervised dereverberation models [48], or training DNNs to optimize the
objective functions of conventional separation algorithms [49, 50, 51]. Their
systems heavily rely on conventional algorithms and their performance is often
limited by conventional algorithms.

There are studies realizing unsupervised speech de-noising via positive and
unlabeled learning [52], where noisy speech and noise signals are assumed
available so that a binary classifier can be trained to determine whether each
small patch of T-F units belongs to noisy speech that can be maintained or
noise that should be removed. However, it is unclear whether this approach can
be modified for dereverberation.

The proposed USDnet is motivated by UNSSOR [34], which formulates unsupervised
neural speaker separation as a blind deconvolution problem, but UNSSOR is
designed to maintain the reverberation of each speaker rather than reducing
it. As we will show in this paper, unsupervised dereverberation has unique
challenges and difficulties very different from the ones in unsupervised
separation of mixed reverberant speakers, and therefore calls for innovative
solutions.

During training, USDnet utilizes recorded mixtures as mixture constraints to
narrow down target speech and realizes unsupervised dereverberation by solely
leveraging recorded mixtures. It exploits implicit regularization afforded by
multi-microphone signals, and does not require using synthetic data (e.g.,
MoM, and paired clean and corrupted speech), source prior models, or pseudo-
labels provided by conventional algorithms. These differences distinguish
USDnet from existing unsupervised neural methods, and, meanwhile, indicate
that USDnet can complement many existing algorithms.

##  III Problem Formulation

For a $P$-microphone mixture with a single speaker in noisy-reverberant
conditions, the physical model can be formulated in the short-time Fourier
transform (STFT) domain as follows.

At a reference microphone $q\in\\{1,\dots,P\\}$, we have

$\displaystyle Y_{q}(t,f)$ $\displaystyle=X_{q}(t,f)+\varepsilon_{q}(t,f)$
$\displaystyle=S_{q}(t,f)+H_{q}(t,f)+\varepsilon_{q}(t,f)$
$\displaystyle=S_{q}(t,f)+\mathbf{g}_{q}(f)^{\mathsf{H}}\widetilde{\mathbf{S}}_{q}(t,f)+\varepsilon^{\prime}_{q}(t,f),$
(1)

where $t$ indexes $T$ frames and $f$ indexes $F$ frequencies. In the first
row, $Y_{q}(t,f)$, $X_{q}(t,f)$, and $\varepsilon_{q}(t,f)$ respectively
denote the STFT coefficients of the noisy-reverberant mixture, reverberant
speaker image, and a weak noise signal captured at time $t$, frequency $f$ and
microphone $q$. In the rest of this paper, when dropping indices $t$, $f$ and
$q$, we refer to the corresponding spectrograms. In the second row, we
decompose $X_{q}$ into direct-path signal $S_{q}$ and reverberation $H_{q}$
(i.e., $X_{q}=S_{q}+H_{q}$). In the third row, following narrowband
approximation [53, 5] we model reverberation $H_{q}$ as a linear convolution
between a filter $\mathbf{g}_{q}$ and the source $S_{q}$, i.e.,
$H_{q}(t,f)\approx\mathbf{g}_{q}(f)^{\mathsf{H}}\widetilde{\mathbf{S}}_{q}(t,f)$,
where
$\widetilde{S}_{q}(t,f)=\big{[}S_{q}(t-A+1,f),\dots,S_{q}(t-\Delta,f)\big{]}^{\mathsf{T}}\in{\mathbb{C}}^{A-\Delta}$
stacks a window of T-F units, $\Delta$ is a non-negative prediction delay,
$\mathbf{g}_{q}(f)\in{\mathbb{C}}^{A-\Delta}$ is a relative transfer function
relating direct-path $S_{q}(\cdot,f)$ to reverberation $H_{q}(\cdot,f)$,
$(\cdot)^{\mathsf{H}}$ computes Hermitian transpose, and
$\varepsilon^{\prime}_{q}$ absorbs the modeling error.

At any non-reference microphone $p\in\\{1,\dots,P\\}$, where $p\neq q$, we
formulate the physical model as

$\displaystyle Y_{p}(t,f)$ $\displaystyle=X_{p}(t,f)+\varepsilon_{p}(t,f)$
$\displaystyle=\mathbf{h}_{p}(f)^{\mathsf{H}}\overline{\mathbf{S}}_{q}(t,f)+\varepsilon^{\prime}_{p}(t,f),$
(2)

where
$\overline{\mathbf{S}}_{q}(t,f)=\big{[}S_{q}(t-B+1,f),\dots,S_{q}(t+C,f)\big{]}^{\mathsf{T}}\in{\mathbb{C}}^{B+C}$
stacks a window of T-F units, $\mathbf{h}_{p}(f)\in{\mathbb{C}}^{B+C}$ is the
relative transfer function relating $S_{q}$ at the reference microphone $q$ to
the reverberant image at non-reference microphone $p$ (i.e., $X_{p}$), and
$\varepsilon^{\prime}_{p}$ absorbs the modeling error.

We aim at reducing the reverberation of the target speaker in an unsupervised
way. With the physical models in (1) and (2) and assuming that $\varepsilon$
and $\varepsilon^{\prime}$ are weak, stationary and time-invariant,
unsupervised dereverberation can be realized by solving, e.g., the
minimization problem below:

$\displaystyle\underset{\mathbf{g}_{\cdot}(\cdot),\mathbf{h}_{\cdot}(\cdot),S_{q}(\cdot,\cdot)}{\text{argmin}}\Big{(}\sum_{t,f}\Big{|}Y_{q}(t,f)-S_{q}(t,f)-\mathbf{g}_{q}(f)^{\mathsf{H}}\widetilde{\mathbf{S}}_{q}(t,f)\Big{|}^{2}$
$\displaystyle\quad\quad\quad\quad+\sum_{p=1,p\neq
q}^{P}\sum_{t,f}\Big{|}Y_{p}(t,f)-\mathbf{h}_{p}(f)^{\mathsf{H}}\overline{\mathbf{S}}_{q}(t,f)\Big{|}^{2}\Big{)},$
(3)

which finds source and filters that are most consistent with the physical
models in (1) and (2) (i.e., most satisfied with the mixture constraints).
This is a blind deconvolution problem [2], which is non-convex in nature and
is not solvable if not assuming any prior knowledge about the filters or the
source, because all of them are unknown and need to be estimated.

In the next section, following UNSSOR [34] we propose USDnet, which tackles
this problem by modeling speech patterns (i.e., source prior) via unsupervised
deep learning. The high-level idea is to leverage unsupervised deep learning
to first produce a source estimate, and, with the source estimated in (3),
filter estimation then becomes a simple linear regression problem, which has a
closed-form solution.

##  IV USDnet

Fig. 1 illustrates the proposed USDnet. It takes as input the mixtures at all
the microphones or just at the reference microphone $q$, and produces an
estimate $\hat{S}_{q}$ at the reference microphone $q$. To encourage
$\hat{S}_{q}$ to approximate $S_{q}$, we regularize it during training by
linearly filtering it via FCP [27] so that the filtered estimates can
approximate the mixtures at different microphones. This section decribes the
DNN setup, loss functions, FCP filtering, an extension to monaural
unsupervised dereverberation, and the rationale of various design choices of
USDnet.

###  IV-A DNN Setup

The DNN estimate $\hat{S}_{q}$ can be produced via complex spectral mapping
[54, 55], where the real and imaginary (RI) parts of input mixtures are
stacked as input features for the DNN to predict the RI parts of
$\hat{S}_{q}$. Alternatively, it can be estimated via complex ratio masking
[56, 57], where the RI parts of a complex mask $\hat{M}_{q}$ is predicted by
the DNN and $\hat{S}_{q}$ is obtained by $\hat{S}_{q}=\hat{M}_{q}\odot Y_{q}$,
with $\odot$ denoting point-wise multiplication. In supervised speech
separation [58], both masking and mapping are popular, and mapping usually
produces better separation than masking [59]. In unsupervised dereverberation,
however, we find that, for non-trivial reasons, training USDnet with masking
is clearly better. We will discuss the reasons in Section IV-H. Other DNN
configurations such as network architectures are provided later in Section
V-C.

Figure 1: Illustration of USDnet (described in first paragraph of Section IV).

###  IV-B Mixture-Constraint Loss

Following UNSSOR [34], we design a mixture-constraint (MC) loss
$\mathcal{L}_{\text{MC}}$ to regularize the DNN estimate $\hat{S}_{q}$ so that
it can approximate $S_{q}$:

$\displaystyle\mathcal{L}_{\text{MC}}=\mathcal{L}_{\text{MC},q}+\alpha\times\sum_{p=1,p\neq
q}^{P}\mathcal{L}_{\text{MC},p},$ (4)

with $\alpha\in{\mathbb{R}}_{>0}$ denoting a microphone weighting term for
non-reference microphones, and $\mathcal{L}_{\text{MC},q}$ and
$\mathcal{L}_{\text{MC},p}$, defined by closely following the two terms in
(3), as

$\displaystyle\mathcal{L}_{\text{MC},q}$
$\displaystyle=\sum_{t,f}\mathcal{F}\Big{(}Y_{q}(t,f),\hat{Y}_{q}(t,f)\Big{)}$
$\displaystyle=\sum_{t,f}\mathcal{F}\Big{(}Y_{q}(t,f),\hat{S}_{q}(t,f)+\hat{H}_{q}(t,f)\Big{)}$
$\displaystyle=\sum_{t,f}\mathcal{F}\Big{(}Y_{q}(t,f),\hat{S}_{q}(t,f)+\hat{\mathbf{g}}_{q}(f)^{\mathsf{H}}\widetilde{\hat{\mathbf{S}}}_{q}(t,f)\Big{)},$
(5) $\displaystyle\mathcal{L}_{\text{MC},p}$
$\displaystyle=\sum_{t,f}\mathcal{F}\Big{(}Y_{p}(t,f),\hat{Y}_{p}(t,f)\Big{)}$
$\displaystyle=\sum_{t,f}\mathcal{F}\Big{(}Y_{p}(t,f),\hat{\mathbf{h}}_{p}(f)^{\mathsf{H}}\overline{\hat{\mathbf{S}}}_{q}(t,f)\Big{)},$
(6)

where
$\widetilde{\hat{\mathbf{S}}}_{q}(t,f)=\big{[}\hat{S}_{q}(t-K+1,f),\dots,\hat{S}_{q}(t-\Delta,f)\big{]}^{\mathsf{T}}\in{\mathbb{C}}^{K-\Delta}$
stacks a window of T-F units in $\hat{S}_{q}$ with a non-negative prediction
delay $\Delta$, and
$\overline{\hat{\mathbf{S}}}_{q}(t,f)=\big{[}\hat{S}_{q}(t-I+1,f),\dots,\hat{S}_{q}(t+J,f)\big{]}^{\mathsf{T}}\in{\mathbb{C}}^{I+J}$.
The number of filter taps, $K$, differs from $A$ defined in
$\widetilde{S}_{q}(t,f)$, and $I$ and $J$ differ from $B$ and $C$ defined in
$\overline{\mathbf{S}}_{q}(t,f)$, since the oracle taps $A$, $B$ and $C$ are
unknown in unsupervised setups. They are hyper-parameters to tune. Following
[60, 34], $\mathcal{F}(\cdot,\cdot)$ computes a loss on the estimated RI
components and their magnitude:

$\displaystyle\mathcal{F}\Big{(}Y_{r}(t,f),\hat{Y}_{r}(t,f)\Big{)}=$
$\displaystyle\frac{1}{\sum\nolimits_{t^{\prime},f^{\prime}}|Y_{r}(t^{\prime},f^{\prime})|}$
$\displaystyle\Big{(}\Big{|}\mathcal{R}(Y_{r}(t,f))-\mathcal{R}(\hat{Y}_{r}(t,f))\Big{|}$
$\displaystyle+\Big{|}\mathcal{I}(Y_{r}(t,f))-\mathcal{I}(\hat{Y}_{r}(t,f))\Big{|}$
$\displaystyle+\Big{|}|Y_{r}(t,f)|-|\hat{Y}_{r}(t,f)|\Big{|}\Big{)},$ (7)

where $r$ indexes all the microphones, $|\cdot|$ computes magnitude,
$\mathcal{R}(\cdot)$ and $\mathcal{I}(\cdot)$ respectively extract RI
components, and the normalization term balances the losses at different
microphones and across training mixtures.

Notice that, in (5), we regularize $\hat{S}_{q}$ to approximate the direct-
path signal $S_{q}$ and
$\hat{\mathbf{g}}_{q}(f)^{\mathsf{H}}\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ to
approximate reverberation $H_{q}(t,f)$, constraining their summation to
approximate the mixture (i.e.,
$\hat{Y}_{q}(t,f)=\hat{S}_{q}(t,f)+\hat{H}_{q}(t,f)=\hat{S}_{q}(t,f)+\hat{\mathbf{g}}_{q}(f)^{\mathsf{H}}\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$);
and in (6), we regularize $\hat{S}_{q}$ such that it can be linearly filtered
to approximate the mixture at another microphone (i.e.,
$\hat{Y}_{p}(t,f)=\hat{\mathbf{h}}_{p}(f)^{\mathsf{H}}\overline{\hat{\mathbf{S}}}_{q}(t,f)$).
In (5), we would expect
$\hat{\mathbf{g}}_{q}(f)^{\mathsf{H}}\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ to
approximate $H_{q}(t,f)$ because of the non-negative prediction $\Delta$ in
$\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$.

To compute $\mathcal{L}_{\text{MC}}$ and enforce the above-mentioned
regularizations to $\hat{S}_{q}$, we need to first estimate the filters,
$\hat{\mathbf{g}}_{q}(f)$ and $\hat{\mathbf{h}}_{p}(f)$. We describe the
method in the next subsection.

In our experiments, we observe that minimizing $\mathcal{L}_{\text{MC}}$ can
promote unsupervised dereverberation. A plot of its loss curve is described
later in Section VI-B and shown in Fig. 2.

###  IV-C FCP for Filter Estimation

Following UNSSOR [34], we compute the relative transfer functions
$\hat{\mathbf{g}}_{q}(f)$ and $\hat{\mathbf{h}}_{p}(f)$ via forward
convolutive prediction (FCP) [27, 61], a neural forward filtering algorithm
that linearly filters the DNN estimate $\hat{S}_{q}$ to approximate the
mixture:

$\displaystyle\hat{\mathbf{g}}_{q}(f)=\underset{\mathbf{g}_{q}(f)}{{\text{argmin}}}\sum\limits_{t}\frac{\Big{|}Y_{q}(t,f)-\hat{S}_{q}(t,f)-\mathbf{g}_{q}(f)^{{\mathsf{H}}}\
\widetilde{\hat{\mathbf{S}}}_{q}(t,f)\Big{|}^{2}}{\hat{\lambda}_{q}(t,f)},$
(8)
$\displaystyle\hat{\mathbf{h}}_{p}(f)=\underset{\mathbf{h}_{p}(f)}{{\text{argmin}}}\sum_{t}\frac{\Big{|}Y_{p}(t,f)-\mathbf{h}_{p}(f)^{{\mathsf{H}}}\
\overline{\hat{\mathbf{S}}}_{q}(t,f)\Big{|}^{2}}{\hat{\lambda}_{p}(t,f)},$ (9)

where $\hat{\lambda}_{r}$, with $r$ indexing all the microphones, is a
weighting term balancing the importance of various T-F units. Following [27,
61, 34], it is defined as

$\displaystyle\hat{\lambda}_{r}(t,f)=\frac{1}{P}\sum_{p=1}^{P}|Y_{p}(t,f)|^{2}+\xi\times\text{max}\big{(}\frac{1}{P}\sum_{p=1}^{P}|Y_{p}|^{2}\big{)},$
(10)

where $\xi$ (tuned to $10^{-4}$ in this paper) floors the weighting term to
avoid placing too much weights on low-energy T-F units, and
$\text{max}(\cdot)$ extracts the maximum value of a spectrogram. Both (8) and
(9) are weighted linear regression problems, where closed-form solutions can
be readily computed. We then plug the closed-form solutions to (5) and (6),
compute the $\mathcal{L}_{\text{MC}}$ loss in (4), and optimize the DNN.

###  IV-D Interpretation of MC Loss and FCP Filtering

To facilitate understanding, we give an interpretation of the proposed
$\mathcal{L}_{\text{MC}}$ loss.

Suppose that $\mathcal{F}(\cdot,\cdot)$ is defined as
$\mathcal{F}\big{(}Y_{r}(t,f),\hat{Y}_{r}(t,f)\big{)}=|Y_{r}(t,f)-\hat{Y}_{r}(t,f)|^{2}$
(rather than as the one in (IV-B), no weighting is used in FCP (i.e.,
$\hat{\lambda}_{r}(t,f)=1$), and $\alpha=1$, $\mathcal{L}_{\text{MC}}$ in (4)
can be alternatively formulated as

$\displaystyle\mathcal{L}_{\text{M}}{}_{\text{C,alt}}=$
$\displaystyle\sum_{f}\underset{\mathbf{g}_{q}(f)}{{\text{min}}}\sum_{t}\Big{|}Y_{q}(t,f)-\hat{S}_{q}(t,f)-\mathbf{g}_{q}(f)^{\mathsf{H}}\widetilde{\hat{\mathbf{S}}}_{q}(t,f)\Big{|}^{2}$
$\displaystyle+\sum_{p=1,p\neq
q}^{P}\sum_{f}\underset{\mathbf{h}_{p}(f)}{{\text{min}}}\sum_{t}\Big{|}Y_{p}(t,f)-\mathbf{h}_{p}(f)^{\mathsf{H}}\overline{\hat{\mathbf{S}}}_{q}(t,f)\Big{|}^{2},$
(11)

which, given a DNN estimate $\hat{S}_{q}$, first computes the relative
transfer functions $\hat{\mathbf{g}}_{q}(f)$ and $\hat{\mathbf{h}}_{p}(f)$
that best relate the DNN estimate $\hat{S}_{q}$ to the mixtures (i.e., best
satisfy the physical mixture constraints) by minimizing a quadratic objective
per frequency at each microphone, and then uses the minimum as the loss for
DNN training. The proposed approach can be considered as a minimizing the
minimum approach. This echoes the idea of permutation invariant training [62]
in supervised speech separation, where the minimum to minimize is first
computed by finding the best speaker permutation and then used as the loss to
train DNNs.

In our study, we use $\mathcal{L}_{\text{MC}}$ in (4) rather than
$\mathcal{L}_{\text{MC,alt}}$ for DNN training, as we find that using
$\mathcal{F}(\cdot,\cdot)$ defined in (IV-B) and the weighting term in (10)
produces better performance.

###  IV-E Alternative FCP Filtering at Reference Microphone

An alternative way to compute $\hat{\mathbf{g}}_{q}(f)$ is as follows:

$\displaystyle\hat{\mathbf{g}}_{q}(f)=\underset{\mathbf{g}_{q}(f)}{{\text{argmin}}}\sum\limits_{t}\frac{\Big{|}Y_{q}(t,f)-\mathbf{g}_{q}(f)^{{\mathsf{H}}}\
\widetilde{\hat{\mathbf{S}}}_{q}(t,f)\Big{|}^{2}}{\hat{\lambda}_{q}(t,f)},$
(12)

where the difference from (8) is that, in the numerator, $\hat{S}_{q}$ is not
subtracted from $Y_{q}$. Although this change appears minor, its benefits are
non-trivial. We find that it (a) helps USDnet avoid the trivial solution of
copying the input mixture as the output (i.e., $\hat{S}_{q}=Y_{q}$); (b)
enables USDnet to be trained on monaural mixtures; and (c) leads to an
$\hat{S}_{q}$ that better approximates $S_{q}$ and a
$\hat{\mathbf{g}}_{q}(f)^{{\mathsf{H}}}\
\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ that better approximates $H_{q}$,
rather than an $\hat{S}_{q}$ and a $\hat{\mathbf{g}}_{q}(f)^{{\mathsf{H}}}\
\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ whose summation in (5) can better
approximate the mixture $Y_{q}$ (and hence better minimize
$\mathcal{L}_{\text{MC},q}$) but not respectively better approximating $S_{q}$
and $H_{q}$. We will describe these benefits in more details later in Section
IV-G.

###  IV-F Monaural Unsupervised Dereverberation

USDnet can be trained to realize monaural unsupervised dereverberation by only
feeding in the mixture at the reference microphone (i.e., $Y_{q}$) to the DNN
while still optimizing $\mathcal{L}_{\text{MC}}$ computed on multi-channel
mixtures. Fig. 1 illustrates the idea. The loss computed on multiple
microphones could guide USDnet to model spectro-temporal patterns in speech
and realize monaural unsupervised dereverberation. Similarly, we can also use
a subset of microphone signals in the input or for loss computation.

If the training mixtures are all monaural (i.e., single-channel input and
loss), we find that we can still train USDnet if the FCP filtering in (12) is
used, but the training would fail if (8) is used for filter estimation. We
will describe the rationale in the second paragraph of Section IV-G.

###  IV-G Necessity of Over-Determined Training Mixtures

If $\hat{\mathbf{g}}_{q}(f)$ is computed using (8), the mixtures for training
USDnet need to be over-determined. In the case of single-speaker
dereverberation, the training mixtures just need to be multi-channel. If the
training mixtures are instead all monaural, $\mathcal{L}_{\text{MC}}$ would
only contain the first term $\mathcal{L}_{\text{MC},q}$, which can be
optimized to zero by the DNN by simply copying the input mixture as the output
(i.e., $\hat{S}_{q}=Y_{q}$ and, in this case, $\hat{\mathbf{g}}_{q}(f)$
computed via (8) would be all-zero). This trivial solution would not yield any
dereverberation. By leveraging extra microphone recordings as constraints, the
second term in $\mathcal{L}_{\text{MC}}$ defined in (4) could help avoid the
trivial solution. Note that over-determined mixtures are only needed for
training. At run time, the trained model can perform monaural dereverberation,
by using, e.g., the method described in the first paragraph of Section IV-F to
train USDnet.

If $\hat{\mathbf{g}}_{q}(f)$ is instead computed using (12), we find that the
mixtures for training USDnet can also be monaural. Different from the case in
the previous paragraph, in this case if the DNN simply copies the input as
output (i.e., $\hat{S}_{q}=Y_{q}$), (12) becomes identical to the optimization
problem in WPE [7], a classic dereverberation algorithm. Following WPE, the
filtering result $\hat{\mathbf{g}}_{q}(f)^{{\mathsf{H}}}\
\widetilde{\hat{\mathbf{S}}}_{q}(t,f)=\hat{\mathbf{g}}_{q}(f)^{{\mathsf{H}}}\
\widetilde{Y}_{q}(t,f)$ would inevitably, and largely only, approximate late
reverberation if the prediction delay $\Delta$ used in defining
$\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ is reasonably large222This is because
when the prediction delay $\Delta$ is set reasonably large, the target speech
in the current T-F unit becomes much less linearly-correlated with the T-F
units that are beyond $\Delta$ frames in the past [7].. As a result, the
reconstructed mixture used for computing $\mathcal{L}_{\text{MC},q}$ in (5),
$\hat{Y}_{q}(t,f)=\hat{S}_{q}(t,f)+\hat{\mathbf{g}}_{q}(f)^{{\mathsf{H}}}\
\widetilde{\hat{\mathbf{S}}}_{q}(t,f)=Y_{q}(t,f)+\hat{\mathbf{g}}_{q}(f)^{{\mathsf{H}}}\
\widetilde{Y}_{q}(t,f)$, would produce a large loss, due to the approximated
late reverberation. Therefore, using (12) and minimizing
$\mathcal{L}_{\text{MC},q}$ could help USDnet avoid the trivial solution. For
similar reasons, using (12) could yield better dereverberation than (8) when
multiple microphones are used in loss computation.

###  IV-H Difficulties of Time Alignment, and Masking vs. Mapping

In supervised speech separation, the availability of target speech signals can
provide a strong supervision at each sample for DNNs to learn to produce an
estimate that is strictly time-aligned with target speech. In WPE, a time-
aligned estimate can be obtained by only removing estimated late
reverberation. In unsupervised neural dereverberation, however, we find it
particularly difficult to produce an estimate time-aligned with target speech,
simply due to the lack of an explicit sample-level supervision (e.g., oracle
target speech). For $\mathcal{L}_{\text{MC},q}$ in (5), even if $\hat{S}_{q}$
produced by the DNN is not time-aligned with the oracle target speech $S_{q}$,
$\mathcal{L}_{\text{MC},q}$ can still be small because the filter
$\hat{\mathbf{g}}_{q}(f)$, which is multi-tap, could compensate the time-
misalignment through multi-frame filtering. This compensation can also happen
when we compute $\mathcal{L}_{\text{MC},p}$ in (6).

To partially address this issue, we propose to use complex masking instead of
mapping to obtain target estimate $\hat{S}_{q}$, and observe that masking
obtains better performance. This could be because, in masking, $\hat{S}_{q}$
is produced by masking the mixture $Y_{q}$, while, in mapping, $\hat{S}_{q}$
is produced directly by the DNN and could hence more likely be misaligned with
target speech. For example, when using complex mapping, at each T-F unit the
DNN could just output an estimated signal not time- and gain-aligned with the
mixture, e.g., $\hat{S}_{q}(t,f)=\beta\times Y_{q}(t+\Delta,f)$ where
$\beta\in{\mathbb{C}}$ denotes an arbitrary scaling factor with $|\beta|$
being small. In this case, in (12), $\hat{\mathbf{g}}_{q}(f)^{{\mathsf{H}}}\
\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ can perfectly approximate $Y_{q}(t,f)$
since now
$\widetilde{\hat{\mathbf{S}}}_{q}(t,f)=\big{[}\hat{S}_{q}(t-K,f),\dots,\hat{S}_{q}(t-\Delta,f)\big{]}^{\mathsf{T}}=\beta\times\big{[}Y_{q}(t+\Delta-K,f),\dots,Y_{q}(t,f)\big{]}^{\mathsf{T}}$
has $Y_{q}(t,f)$ in the last element; and in (5), the resulting reconstructed
mixture
$\hat{Y}_{q}(t,f)=\hat{S}_{q}(t,f)+\hat{\mathbf{g}}_{q}(f)^{\mathsf{H}}\widetilde{\hat{\mathbf{S}}}_{q}(t,f)=\beta\times
Y_{q}(t+\Delta,f)+Y_{q}(t,f)$ would be very close to the mixture $Y_{q}(t,f)$
if $|\beta|$ is small, yielding a very small $\mathcal{L}_{\text{MC},q}$ even
though the solution is trivial. By using masking, this trivial solution could
be avoided.

###  IV-I Input Microphone Dropout

When multiple microphone signals are used as the network input and in loss
computation, we find that USDnet often optimizes the loss well but the
dereverberation result is poor. This could be because USDnet sees, from the
input, the same microphone signals used for loss computation and, therefore,
could figure out a way to optimize the loss well.

To deal with this issue, we propose to apply dropout [63] to input non-
reference microphone signals, where, during training, each non-reference
microphone signal in the input is entirely dropped out with a probability of
$\theta$.

###  IV-J Introducing Garbage Sources

To model environmental noises and the modeling error caused by narrow-band
approximation (i.e., $\varepsilon^{\prime}$ in (1) and (2)), we propose to use
a garbage source to absorb them.

We modify $\mathcal{L}_{\text{MC},q}$ in (5) and $\mathcal{L}_{\text{MC},p}$
in (6) as follows:

$\displaystyle\mathcal{L}_{\text{MC},q}$
$\displaystyle=\sum_{t,f}\mathcal{F}\Big{(}Y_{q}(t,f),$
$\displaystyle\hat{S}_{q}(t,f)+\hat{\mathbf{g}}_{q}(f)^{\mathsf{H}}\widetilde{\hat{\mathbf{S}}}_{q}(t,f)+\hat{\mathbf{w}}_{q}(f)^{\mathsf{H}}\breve{\hat{\mathbf{V}}}_{q}(t,f)\Big{)},$
(13) $\displaystyle\mathcal{L}_{\text{MC},p}=\sum_{t,f}\mathcal{F}\Big{(}$
$\displaystyle Y_{p}(t,f),$
$\displaystyle\hat{\mathbf{h}}_{p}(f)^{\mathsf{H}}\overline{\hat{\mathbf{S}}}_{q}(t,f)+\hat{\mathbf{w}}_{p}(f)^{\mathsf{H}}\breve{\hat{\mathbf{V}}}_{q}(t,f)\Big{)},$
(14)

where the DNN is trained to output one more spectrogram $\hat{V}_{q}$ in
addition to $\hat{S}_{q}$,
$\breve{\hat{\mathbf{V}}}_{q}(t,f)=\big{[}\hat{V}_{q}(t-L,f),\dots,\hat{V}_{q}(t+L,f)\big{]}^{\mathsf{T}}\in{\mathbb{C}}^{L+1+L}$
stacks a window of $L+1+L$ T-F units in $\hat{V}_{q}$, and
$\hat{\mathbf{g}}_{q}(f)$ and $\hat{\mathbf{h}}_{p}(f)$ are respectively
computed in the same ways as in (8) or (12), and (9).
$\hat{\mathbf{w}}_{q}(f)$ and $\hat{\mathbf{w}}_{p}(f)$ are computed as
follows, similarly to (9):

$\displaystyle\hat{\mathbf{w}}_{r}(f)=\underset{\mathbf{w}_{r}(f)}{{\text{argmin}}}\sum_{t}\frac{\Big{|}Y_{r}(t,f)-\mathbf{w}_{r}(f)^{{\mathsf{H}}}\
\breve{\hat{\mathbf{V}}}_{q}(t,f)\Big{|}^{2}}{\hat{\lambda}_{r}(t,f)},$ (15)

where $r$ indexes all the microphones.

We point out that using garbage sources to absorb modeling errors is a widely-
adopted technique in conventional separation algorithms such as IVA [64], and
CAGMM- and MESSL-based spatial clustering [65, 66]. Our novelty here is
adapting this idea for DNN-based unsupervised dereverberation.

###  IV-K Run-Time Inference

At run time, we use $\hat{S}_{q}$ as the final prediction. We just need to run
feed-forwarding for interference, without needing to compute FCP filters.

In other words, USDnet is trained to optimize the $\mathcal{L}_{\text{MC}}$
loss in (4) so that it can learn to model signal patterns in reverberant
speech for dereverberation. At run time, it can be used to dereverberate
signals in the same way as models trained in a supervised way. This novel
design makes USDnet possible to be readily integrated with semi-supervised
learning, where, for example, (a) USDnet trained on massive unlabeled data can
be fine-tuned on labeled target-domain data via supervised learning; and (b)
pre-trained supervised models can be adapted in an unsupervised way on target-
domain unlabeled data by initializing USDnet with the pre-trained model.

Following WPE [7], we can alternatively use
$Y_{q}(t,f)-\hat{\mathbf{g}}_{q}(f)^{{\mathsf{H}}}\
\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ as the final prediction, where the
subtracted term is obtained by filtering past DNN estimates with a prediction
delay $\Delta$ (in $\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$) and hence can be
viewed as an estimate of reverberation to remove. However, in our experiments
the performance is worse than directly using $\hat{S}_{q}$.

##  V Experimental Setup

We train USDnet using an existing speech dereverberation dataset, and evaluate
it on its simulated test set and the REVERB dataset. This section describes
the datasets, miscellaneous setup, baseline systems, and evaluation metrics.

###  V-A WSJ0CAM-DEREVERB Dataset

WSJ0CAM-DEREVERB is simulated and has been used in several supervised
dereverberation studies [67, 27, 59], which reported strong results. The dry
source signals are from the WSJ0CAM corpus, which has $7,861$, $742$ and
$1,088$ utterances respectively in its training, validation and test sets.
Based on them, $39,293$ ($\sim$$77.7$ h), $2,968$ ($\sim$$5.6$ h) and $3,262$
($\sim$$6.4$ h) noisy-reverberant mixtures are respectively simulated as the
training, validation and test sets. For each utterance, a room with random
room characteristics and speaker and microphone locations is sampled. The
simulated microphone array has $8$ microphones uniformly placed on a circle
with a diameter of $20$ cm. The speaker-to-array distance is drawn from the
range $[0.75,2.5]$ m and the reverberation time (T60) from $[0.2,1.3]$ s. For
each utterance, an $8$-channel diffuse air-conditioning noise is sampled from
the REVERB dataset [68], and added to the reverberant speech at an SNR
(between the direct-path signal and the noise) sampled from the range $[5,25]$
dB. The sampling rate is $16$ kHz.

###  V-B REVERB Dataset

We evaluate USDnet, trained on WSJ0CAM-DEREVERB, directly on the REVERB
dataset [68], which has real near- and far-field reverberant speech recorded
in rooms with air-conditioning noises by an 8-microphone circular array with a
diameter of $20$ cm. The rooms have a T60 at around $0.7$ s and the speaker to
microphone distances are around $1$ m in the near-field case and $2.5$ m in
the far-field case. We feed dereverberated signals produced by USDnet to an
ASR backend for recognition. The ASR backend is trained by using the default
recipe in Kaldi333commit 61637e6c8ab01d3b4c54a50d9b20781a0aa12a59, which
trains a TDNN-based acoustic model based on the default multi-condition
training data consisting of the noisy-reverberant speech of REVERB. The
sampling rate is $16$ kHz.

We emphasize that the purpose of this evaluation is to show the effectiveness
of USDnet on real-recorded data and compare its performance with WPE, not to
obtain state-of-the-art ASR performance on REVERB.

###  V-C Miscellaneous System Setup

The STFT window size is $32$ ms, hop size $8$ ms, and the square root of Hann
window is used as the analysis window. We use $512$-point discrete Fourier
transform to extract $257$-dimensional complex spectra at each frame.

We employ TF-GridNet [59] as the DNN architecture. Using symbols defined in
Table I of [59], we set its hyper-parameters to $D=48$, $B=4$, $I=4$, $J=4$,
$H=192$, $L=4$ and $E=2$. Please do not confuse the symbols of TF-GridNet with
the ones in this paper. When the DNN is trained for complex ratio masking, we
truncate the RI parts of the estimated masks into the range $[-5,5]$ before
multiplying it with the mixture.

We consider $1$-, $2$-, $4$\- and $8$-microphone dereverberation. The first
microphone is always considered as the referenece microphone. For $1$-channel
dereverberation, we use the mixture signal captured by microphone $1$; for
$2$-channel dereverberation, we use microphone $1$ and $4$; for $4$-channel
dereverberation, we use microphone $1$, $3$, $5$ and $7$; and for $8$-channel
dereverberation, all the $8$ microphones are used.

###  V-D Baseline Systems

We consider WPE [7] as the baseline. It is unsupervised and is so far the most
popular and successful dereverberation algorithm. We use the implementation in
the nara_wpe toolkit [69]. The STFT configuration for WPE is the same as that
in USDnet. Following [69, 10], the filter tap is tuned to $37$ in monaural
cases, to $10$ in $4$-channel cases, to $5$ in $8$-channel cases, the
prediction delay is $3$, and $3$ iterations is performed.

We considered comparing USDnet with unsupervised neural dereverberation models
such as [40, 42, 43]. However, they only deal with monaural cases and require
separate source prior models trained on anechoic speech or DNN-based metric
models (such as DNSMOS) trained on annotated anechoic and reverberant speech,
and cannot be directly trained solely on a set of reverberant signals, while,
differently, USDnet exploits implicit regularizations afforded by multiple
microphones and can be trained directly on a set of reverberant signals. In
other words, they utilize very different signal principals, which can be
likely integrated with USDnet in future studies. We therefore do not consider
them as baselines for comparison, and mainly focus on showing the
effectiveness of leveraging mixture constraints in this paper.

###  V-E Evaluation Metrics

For evaluation on the test set of WSJ0CAM-DEREVERB, we use the direct-path
signal at the first microphone (designated as the reference microphone) as the
reference signal for metric computation. We report scores of perceptual
evaluation of speech quality (PESQ) [70] and extended short-time objective
intelligibility (eSTOI) [71]. They are widely-adopted objective metrics of
speech quality and intelligibility. We use the python-pesq toolkit to report
narrow-band MOS-LQO scores for PESQ, and the pystoi toolkit for eSTOI.
Additionally, we report scale-invariant signal-to-distortion ratio (SI-SDR)
[72] but do not consider it as a major evaluation metric due to the
difficulties, and in many applications unnecessities, of time alignment. For
the ASR evaluation on REVERB, we report word error rates (WER).

All of these metrics, except WER, favor estimated signals that are time-
aligned with reference signals.

##  VI Evaluation Results

This section validates USDnet and its design choices based on the WSJ0CAM-
DEREVERB and REVERB datasets. Table I lists major hyper-parameters of USDnet.
We start with $I=40$, $J=0$, $K=40$, $\Delta=3$, no garbage sources (i.e., the
filter tap $L$ is shown as “-” in the tables), no microphone weight (i.e.,
$\alpha=1$), and no dropout to input microphones (i.e., $\theta=0$).

TABLE I: Major Hyper-Parameters of USDnet

Symbols | Description
---|---
$K$, $\Delta$ | $\widetilde{\hat{\mathbf{S}}}_{q}(t,f)=\big{[}\hat{S}_{q}(t-K+1,f),\dots,\hat{S}_{q}(t-\Delta,f)\big{]}^{\mathsf{T}}\in{\mathbb{C}}^{K-\Delta}$
$I$, $J$ | $\overline{\hat{\mathbf{S}}}_{q}(t,f)=\big{[}\hat{S}_{q}(t-I+1,f),\dots,\hat{S}_{q}(t+J,f)\big{]}^{\mathsf{T}}\in{\mathbb{C}}^{I+J}$
$L$ |  $\breve{\hat{\mathbf{V}}}_{q}(t,f)=\big{[}\hat{V}_{q}(t-L,f),\dots,\hat{V}_{q}(t+L,f)\big{]}^{\mathsf{T}}\in{\mathbb{C}}^{L+1+L}$ for garbage source
$\alpha$ | Microphone weight for non-reference microphones
$\theta$ | Dropout ratio for input non-reference microphones

###  VI-A Effects of Using More Microphones in Loss and Input

In row 2a of Table II, the training mixtures are all monaural. The performance
is not good, since only one microphone signal can be used as the mixture
constraint and this is likely not enough for the DNN to solve the ill-posed
problem and learn to figure out what the target signal is. On the other hand,
through the formulation of $\mathcal{L}_{\text{MC},q}$ in (5) and using (12)
for filter estimation, USDnet does not fail completely and shows some
effectiveness.

In row 2b-2d of Table II, only one microphone signal is used in input but
multiple microphone signals are used for loss computation. We observe that the
dereverberation performance gets much better, compared with row 2a. This
supports our idea that using extra microphone signals as mixture constraints
in the loss can help USDnet better figure out the target signal.

In row 3b-3d, multiple microphone signals are used in input and all of them
are used for loss computation. We observe better performance than row 2a.

Next, we use the results in 2a-2d and 3b-3d as baselines, and validate the
effects of various design choices in USDnet.

TABLE II: Results on Test Set of WSJ0CAM-DEREVERB using Various Number of
Microphone Signals in Input and Loss

Row | Systems | #Mics in | $\hat{\mathbf{g}}_{q}(f)$ | $I/J/K/\Delta/L$ | Masking/ | $\alpha$ | $\theta$ | PESQ$\uparrow$ | eSTOI$\uparrow$ | SI-SDR
---|---|---|---|---|---|---|---|---|---|---
input/loss | Mapping |  (dB)$\uparrow$
1 | Mixture | 1 / - | - | - | - | - | - | $1.640\,610\,543\,333\,775\,2$ | $0.493\,757\,274\,092\,272$ | $-3.563\,633\,992\,835\,417$
2a | USDnet | 1 / 1 |  (12) | 40 / 0 / 40 / 3 / - | Masking | $1$ | - | $1.759\,163\,199\,430\,006$ | $0.561\,225\,480\,546\,709$ | $-2.147\,380\,687\,798\,935\,7$
2b | USDnet | 1 / 2 |  (12) | 40 / 0 / 40 / 3 / - | Masking | $1$ | - | $2.339\,252\,534\,378\,971$ | $0.735\,487\,625\,424\,018\,8$ | $1.617\,121\,472\,629\,272\,4$
2c | USDnet | 1 / 4 |  (12) | 40 / 0 / 40 / 3 / - | Masking | $1$ | - | $2.506\,682\,344\,077\,921\,3$ | $0.760\,103\,172\,876\,98$ | $2.508\,255\,508\,120\,865\,5$
2d | USDnet | 1 / 8 |  (12) | 40 / 0 / 40 / 3 / - | Masking | $1$ | - | $2.350\,717\,771\,170\,258$ | $0.742\,094\,320\,151\,134\,9$ | $2.230\,378\,166\,732\,061\,5$
3b | USDnet | 2 / 2 |  (12) | 40 / 0 / 40 / 3 / - | Masking | $1$ | $0$ | $2.169\,792\,887\,954\,022$ | $0.696\,095\,396\,611\,442\,4$ | $1.113\,166\,088\,267\,944\,2$
3c | USDnet | 4 / 4 |  (12) | 40 / 0 / 40 / 3 / - | Masking | $1$ | $0$ | $2.279\,547\,061\,019\,701$ | $0.715\,882\,871\,013\,462\,7$ | $1.598\,769\,641\,992\,428$
3d | USDnet | 8 / 8 |  (12) | 40 / 0 / 40 / 3 / - | Masking | $1$ | $0$ | $2.330\,097\,478\,675\,667$ | $0.729\,609\,516\,419\,143\,1$ | $2.046\,792\,088\,948\,371\,7$
4a | USDnet | 1 / 1 |  (12) | 40 / 0 / 40 / 3 / - | Mapping | $1$ | - | $1.612\,181\,271\,825\,518$ | $0.493\,112\,388\,789\,571$ | $-3.560\,001\,230\,252\,541\,2$
4b | USDnet | 1 / 8 |  (12) | 40 / 0 / 40 / 3 / - | Mapping | $1$ | - | $1.887\,490\,903\,103\,300\,5$ | $0.279\,390\,436\,173\,125\,37$ | $-35.958\,197\,095\,290\,345$
4c | USDnet | 8 / 8 |  (12) | 40 / 0 / 40 / 3 / - | Mapping | $1$ | $0$ | $1.868\,818\,414\,350\,143\,6$ | $0.507\,280\,871\,312\,554$ | $-28.515\,882\,272\,430\,9$
5a | USDnet | 1 / 1 |  (8) | 40 / 0 / 40 / 3 / - | Masking | $1$ | - | $1.640\,556\,440\,698\,376\,8$ | $0.493\,755\,682\,176\,754\,7$ | $-3.563\,789\,905\,204\,356\,2$
5b | USDnet | 1 / 8 |  (8) | 40 / 0 / 40 / 3 / - | Masking | $1$ | - | $2.076\,893\,406\,510\,718\,5$ | $0.671\,319\,253\,288\,188\,5$ | $-0.103\,743\,951\,606\,639\,45$
5c | USDnet | 8 / 8 |  (8) | 40 / 0 / 40 / 3 / - | Masking | $1$ | $0$ | $2.084\,950\,191\,560\,952$ | $0.677\,684\,802\,707\,013\,7$ | $-0.029\,409\,321\,889\,598\,77$

###  VI-B Illustration of Loss Curve of $\mathcal{L}_{\text{MC}}$

To show that minimizing $\mathcal{L}_{\text{MC}}$ can promote unsupervised
dereverberation, we plot the loss curve of $\mathcal{L}_{\text{MC}}$ in Fig.
2, by using a simulated noisy-reverberant speech signal sampled from WSJ0CAM-
DEREVERB. Given the simulated direct-path RIR $o_{d}\in{\mathbb{R}}^{M_{d}}$
and reverberant-speech RIR $o_{i}\in{\mathbb{R}}^{M_{i}}$, both at the
reference microphone $q$, we first compute the relative RIR $o_{r}$ relating
the direct-path signal to reverberant speech as follows:

$\displaystyle
o_{r}=\text{iFFT}\Big{(}\frac{\text{FFT}(o_{i},M_{r})}{\text{FFT}(o_{d},M_{r})},M_{r}\Big{)}\in{\mathbb{R}}^{M_{r}},$
(16)

where $M_{r}=M_{i}+M_{d}-1$, $\text{FFT}(\cdot,M_{r})$ computes an
$M_{r}$-point fast Fourier transform (FFT) of the RIR, and
$\text{iFFT}(\cdot,M_{r})$ computes an $M_{r}$-point inverse FFT (iFFT). We
then compute a hypothesized dereverberation result as

$\displaystyle\hat{S}_{q}=\text{STFT}\big{(}s_{q}*\operatorname{T}_{0}^{\tau}(o_{r})\big{)},$
(17)

where $s_{q}$ is the time-domain direct-path signal corresponding to $S_{q}$,
the operator $*$ denotes linear convolution, and
$\operatorname{T}_{0}^{\tau}(\cdot)$ truncates the RIR to length $\tau$ so
that we can control how reverberant $\hat{S}_{q}$ is. We then use
$\hat{S}_{q}$ to compute $\mathcal{L}_{\text{MC}}$ based on the hyper-
parameters shown in row 2d of Table II, and plot the loss curve against
$\tau$. From Fig. 2, we observe much smaller $\mathcal{L}_{\text{MC}}$ when
$\hat{S}_{q}$ is configured less reverberant. This supports our claim that
minimizing $\mathcal{L}_{\text{MC}}$ can promote dereverberation.

Figure 2:  Loss curve of $\mathcal{L}_{\text{MC}}$ against truncated length of
relative RIR, $\tau$. For this example utterance, the T60 is $1.204$ seconds.
The truncation length is enumerated every $10$ samples, starting from one
sample long all the way up to the length of T60. See Section VI-B for more
details.

###  VI-C Effects of Masking vs. Mapping

In row 4a-4c of Table II, we use complex spectral mapping instead of complex
masking. The results are clearly worse than those in 2a, 2d and 3d,
respectively, and, in particular, the SI-SDR scores are much lower. This is
because, in complex spectral mapping, the DNN has fewer restrictions in
producing target estimates, and this would result in trivial solutions where
reverberation is not reduced (see our discussions in Section IV-H). During
training, we indeed observe that, when using mapping, the training loss often
suddenly decreases to values very close to zero.

In the rest of this paper, we use masking in default.

###  VI-D Effects of (12) vs. (8) for FCP Filtering at Reference Mic

In row 5a-5c of Table II, we use (8) instead of (12) to compute the filter
$\hat{\mathbf{g}}_{q}(f)$ at the reference microphone $q$. The results are
respectively much worse than the ones in row 2a, 2d and 3d, indicating the
effectiveness of using (12).

In 5a, no improvement is observed over the unprocessed mixture. This is
because the network produces a trivial solution by copying the input mixture
as the output. See the discussions in the first paragraph of Section IV-G.
Differently, in 2a, we observe some improvement over the mixture by using (12)
instead of (8) for FCP filtering. The rationale is provided in the second
paragraph of Section IV-G.

In the rest of this paper, we use (12) in default.

###  VI-E Effects of Non-Negative Prediction Delay $\Delta$

The prediction delay $\Delta$ in defining
$\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ (see the text below (6)) is a hyper-
parameter to tune. Ideally, for $\mathcal{L}_{\text{MC},q}$ in (5), we want
$\hat{S}_{q}(t,f)$ to approximate the direct-path signal and
$\hat{\mathbf{g}}_{q}(f)^{\mathsf{H}}\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ to
approximate all the reflections so that their summation can add up to the
mixture $Y_{q}$. When using (12) to compute $\mathbf{g}_{q}(f)$, a large
$\Delta$ would make $\mathbf{g}_{q}(f)^{{\mathsf{H}}}\
\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ in the numerator not capable of
approximating early reflections, and a small $\Delta$ could result in a
$\mathbf{g}_{q}(f)^{{\mathsf{H}}}\ \widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ that
cancels out target direct-path speech. In Table IV, we sweep $\Delta$ based on
the set of $\\{1,2,3,4\\}$, and observe that setting $\Delta$ to $3$ or $2$
results in better performance. However, the performance differences are not
very large.

###  VI-F Effects of Filter Length $I$ and $K$

The filter length $K$ and $I$ in defining
$\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ and
$\overline{\hat{\mathbf{S}}}_{q}(t,f)$ (see the text below (6)) are hyper-
parameters to tune. We sweep them based on the set of $\\{20,40,60,80\\}$.
Given that the STFT window and hop sizes are respectively $32$ and $8$ ms,
this set of filter taps roughly samples filter lengths in the range of
$[0.2,0.7]$ seconds. From Table IV, we observe that using $I=60$ together with
$K=60$ produces the best performance, but the differences in performance are
not very large.

TABLE III: Results on Test Set of WSJ0CAM-DEREVERB using Various Prediction
Delays $\Delta$

Row | Systems | #CH in | $I/J/K/\Delta/L$ | $\alpha$ | $\theta$ | PESQ$\uparrow$ | eSTOI$\uparrow$ | SI-SDR (dB)$\uparrow$
---|---|---|---|---|---|---|---|---
input/loss
1 | USDnet | 8 / 8 | 40 / 0 / 40 / 1 / - | $1$ | $0$ | $2.344\,406\,643\,467\,838\,3$ | $0.716\,654\,399\,385\,269\,1$ | $2.344\,406\,643\,467\,838\,3$
2 | USDnet | 8 / 8 | 40 / 0 / 40 / 2 / - | $1$ | $0$ | $2.340\,049\,011\,951\,425\,7$ | $0.734\,337\,679\,016\,450\,8$ | $2.820\,294\,403\,629\,080\,5$
3 | USDnet | 8 / 8 | 40 / 0 / 40 / 3 / - | $1$ | $0$ | $2.330\,097\,478\,675\,667$ | $0.729\,609\,516\,419\,143\,1$ | $2.046\,792\,088\,948\,371\,7$
4 | USDnet | 8 / 8 | 40 / 0 / 40 / 4 / - | $1$ | $0$ | $2.324\,709\,243\,857\,736$ | $0.725\,862\,773\,088\,348\,5$ | $1.310\,272\,337\,295\,945$

TABLE IV: Results on Test Set of WSJ0CAM-DEREVERB using Various Filter Length
$K$ and $I$

Row | Systems | #CH in | $I/J/K/\Delta/L$ | $\alpha$ | $\theta$ | PESQ$\uparrow$ | eSTOI$\uparrow$ | SI-SDR (dB)$\uparrow$
---|---|---|---|---|---|---|---|---
input/loss
1 | USDnet | 8 / 8 | 20 / 0 / 20 / 3 / - | $1$ | $0$ | $2.302\,729\,584\,189\,882$ | $0.715\,440\,821\,757\,632$ | $1.322\,270\,285\,174\,525\,6$
2 | USDnet | 8 / 8 | 40 / 0 / 40 / 3 / - | $1$ | $0$ | $2.330\,097\,478\,675\,667$ | $0.729\,609\,516\,419\,143\,1$ | $2.046\,792\,088\,948\,371\,7$
3 | USDnet | 8 / 8 | 60 / 0 / 60 / 3 / - | $1$ | $0$ | $2.353\,980\,152\,794\,074$ | $0.739\,299\,959\,732\,241\,6$ | $2.284\,862\,076\,075\,505\,4$
4 | USDnet | 8 / 8 | 80 / 0 / 80 / 3 / - | $1$ | $0$ | $2.267\,627\,137\,522\,987$ | $0.727\,182\,419\,106\,583\,1$ | $2.766\,748\,311\,463\,708$

###  VI-G Effects of Filter Delay $J$ for Non-Reference Microphones

In the text below (6), $\widetilde{\hat{\mathbf{S}}}_{q}(t,f)$ is configured
to only contain past T-F units, while $\overline{\hat{\mathbf{S}}}_{q}(t,f)$
is configured to have $J$ future T-F units. In earlier experiments, $J$ is set
to $0$, while, in this subsection, we increase it to $1$ to account for the
fact that the reference microphone is not always the closest microphone to the
target speaker. Notice that we do not need more future taps, since the
microphone array is a compact array with an aperture size of only $20$ cm,
which is much small than the speed of sound. From Table VIII, we do not
observe consistent improvement.

###  VI-H Effects of Weighting Microphones

Table VIII presents the results of using a smaller microphone weight $\alpha$
in (4) for non-reference microphones. The motivation is that when setting
$\alpha=1.0$ and many microphones are used to compute
$\mathcal{L}_{\text{MC}}$ in (4), the loss on non-reference microphones would
dominate the overall loss, causing the loss on the reference microphone,
$\mathcal{L}_{\text{MC},q}$, which is more relevant to the target signal we
aim to estimate, not optimized well. In our experiments, if there are $P$
($>4$) microphones for loss computation, we set $\alpha=\frac{3}{P-1}$, and
still set $\alpha=1.0$ otherwise. In Table VIII, better performance is
observed.

TABLE V: Results on Test Set of WSJ0CAM-DEREVERB using Various Filter Delay
$J$

Row | Systems | #CH in | $I/J/K/\Delta/L$ | $\alpha$ | $\theta$ | PESQ$\uparrow$ | eSTOI$\uparrow$ | SI-SDR (dB)$\uparrow$
---|---|---|---|---|---|---|---|---
input/loss
1 | USDnet | 1 / 8 | 40 / 0 / 40 / 3 / - | $1$ | - | $2.350\,717\,771\,170\,258$ | $0.742\,094\,320\,151\,134\,9$ | $2.230\,378\,166\,732\,061\,5$
2 | USDnet | 1 / 8 | 39 / 1 / 40 / 3 / - | $1$ | - | $2.355\,559\,416\,119\,852$ | $0.729\,871\,118\,855\,590\,1$ | $1.639\,540\,109\,389\,126\,4$
3 | USDnet | 8 / 8 | 40 / 0 / 40 / 3 / - | $1$ | $0$ | $2.330\,097\,478\,675\,667$ | $0.729\,609\,516\,419\,143\,1$ | $2.046\,792\,088\,948\,371\,7$
4 | USDnet | 8 / 8 | 39 / 1 / 40 / 3 / - | $1$ | $0$ | $2.314\,370\,258\,098\,581\,4$ | $0.719\,414\,782\,665\,487$ | $1.564\,529\,232\,018\,841\,5$

TABLE VI: Results on Test Set of WSJ0CAM-DEREVERB using Various Microphone
Weights $\alpha$

Row | Systems | #CH in | $I/J/K/\Delta/L$ | $\alpha$ | $\theta$ | PESQ$\uparrow$ | eSTOI$\uparrow$ | SI-SDR (dB)$\uparrow$
---|---|---|---|---|---|---|---|---
input/loss
1 | USDnet | 1 / 8 | 40 / 0 / 40 / 3 / - | $1$ | - | $2.350\,717\,771\,170\,258$ | $0.742\,094\,320\,151\,134\,9$ | $2.230\,378\,166\,732\,061\,5$
2 | USDnet | 1 / 8 | 40 / 0 / 40 / 3 / - | $\frac{3}{8-1}$ | - | $2.480\,670\,537\,815\,585\,3$ | $0.748\,535\,241\,479\,823\,1$ | $2.224\,682\,997\,196\,528\,5$
3 | USDnet | 8 / 8 | 40 / 0 / 40 / 3 / - | $1$ | $0$ | $2.330\,097\,478\,675\,667$ | $0.729\,609\,516\,419\,143\,1$ | $2.046\,792\,088\,948\,371\,7$
4 | USDnet | 8 / 8 | 40 / 0 / 40 / 3 / - | $\frac{3}{8-1}$ | $0$ | $2.398\,516\,595\,802\,155$ | $0.744\,936\,838\,972\,866\,1$ | $2.245\,083\,764\,643\,317\,3$

TABLE VII: Results on Test Set of WSJ0CAM-DEREVERB using
Input Microphone Dropout

Row | Systems | #CH in | $I/J/K/\Delta/L$ | $\alpha$ | $\theta$ | PESQ$\uparrow$ | eSTOI$\uparrow$ | SI-SDR (dB)$\uparrow$
---|---|---|---|---|---|---|---|---
input/loss
1a | USDnet | 2 / 2 | 40 / 0 / 40 / 3 / - | $1$ | $0$ | $2.169\,792\,887\,954\,022$ | $0.696\,095\,396\,611\,442\,4$ | $1.113\,166\,088\,267\,944\,2$
1b | USDnet | 4 / 4 | 40 / 0 / 40 / 3 / - | $1$ | $0$ | $2.279\,547\,061\,019\,701$ | $0.715\,882\,871\,013\,462\,7$ | $1.598\,769\,641\,992\,428$
1c | USDnet | 8 / 8 | 40 / 0 / 40 / 3 / - | $\frac{3}{8-1}$ | $0$ | $2.398\,516\,595\,802\,155$ | $0.744\,936\,838\,972\,866\,1$ | $2.245\,083\,764\,643\,317\,3$
2a | USDnet | 2 / 2 | 40 / 0 / 40 / 3 / - | $1$ | $0.7$ | $2.371\,631\,669\,201\,491\,4$ | $0.745\,008\,337\,000\,469\,8$ | $2.139\,048\,303\,784\,847$
2b | USDnet | 4 / 4 | 40 / 0 / 40 / 3 / - | $1$ | $0.7$ | $2.533\,177\,250\,152\,274\,7$ | $0.750\,952\,036\,801\,077\,7$ | $2.296\,071\,436\,252\,039\,4$
2c | USDnet | 8 / 8 | 40 / 0 / 40 / 3 / - | $\frac{3}{8-1}$ | $0.7$ | $2.423\,170\,863\,851\,164\,6$ | $0.746\,278\,609\,142\,762\,7$ | $2.145\,577\,937\,170\,951\,4$

TABLE VIII: Results on Test Set of WSJ0CAM-DEREVERB using
Garbage Sources

Row | Systems | #CH in | $I/J/K/\Delta/L$ | $\alpha$ | $\theta$ | PESQ$\uparrow$ | eSTOI$\uparrow$ | SI-SDR (dB)$\uparrow$
---|---|---|---|---|---|---|---|---
input/loss
1 | Mixture | 1 / - | - | - | - | $1.640\,610\,543\,333\,775\,2$ | $0.493\,757\,274\,092\,272$ | $-3.563\,633\,992\,835\,417$
2a | USDnet | 1 / 8 | 60 / 0 / 60 / 3 / - | $\frac{3}{8-1}$ | - | $2.446\,380\,413\,068\,283\,6$ | $0.756\,921\,563\,443\,627\,8$ | $2.495\,406\,880\,906\,926$
2b | USDnet | 8 / 8 | 60 / 0 / 60 / 3 / - | $\frac{3}{8-1}$ | $0.7$ | $2.421\,752\,320\,009\,824\,5$ | $0.759\,528\,334\,315\,908\,6$ | $2.578\,174\,839\,644\,560\,7$
3a | USDnet | 1 / 8 | 60 / 0 / 60 / 3 / 1 | $\frac{3}{8-1}$ | - | $2.527\,006\,792\,700\,561\,3$ | $0.771\,691\,210\,136\,212\,6$ | $2.860\,394\,877\,577\,935\,5$
3b | USDnet | 8 / 8 | 60 / 0 / 60 / 3 / 1 | $\frac{3}{8-1}$ | $0.7$ | $2.451\,282\,974\,111\,018$ | $0.760\,557\,652\,709\,721\,1$ | $2.675\,615\,118\,266\,085$
4a |  WPE [7] | 1 / - | - | - | - | $1.777\,453\,876\,250\,657$ | $0.579\,248\,917\,141\,486\,2$ | $-1.699\,056\,528\,695\,723\,5$
4b |  WPE [7] | 8 / - | - | - | - | $2.022\,079\,673\,301\,654\,6$ | $0.689\,944\,716\,942\,314\,2$ | $1.998\,465\,577\,307\,669\,2$

###  VI-I Effects of Input Microphone Dropout

In Table VIII, we report the results of applying input microphone dropout to
input non-reference microphone signals during training. The dropout
probability $\theta$ of each non-reference microphone is $0.7$ in default.
Better performance is observed for all the $2$-, $4$\- and $8$-channel cases.

###  VI-J Effects of Using Garbage Sources

In Table VIII, we increase the filter taps $I$ and $K$ to $60$ and report the
results of using one garbage source to absorb environmental noises and
modeling errors. We tune the FCP filter for filtering the garbage source to
$3$-tap, by setting $L$ to $1$ (see the text below (14) for the definitions).
Comparing row 2a with 3a, and 2b with 3b, we observe noticeable improvements.

###  VI-K Comparison with WPE on WSJ0CAM-DEREVERB

Table VIII also reports the results of monaural and eight-channel WPE. We
observe that USDnet obtains clearly better performance than WPE, especially in
monaural input cases (e.g., $2.53$ vs. $1.78$ in PESQ).

Figure 3:  Illustration of (a) unprocessed mixture; (b) dereverberation
results of USDnet (configured in row 3a of Table VIII); (c) dereverberation
results of WPE (row 4a of Table VIII); and (d) clean anechoic speech. For this
example, the T60 is $1.204$ seconds and direct-to-reverberation energy ratio
is around $-5.9$ dB, the PESQ scores are respectively $1.44$, $2.41$, $1.51$
and $4.55$, and eSTOI scores are respectively $0.322$, $0.735$, $0.423$ and
$1.0$. Best in color.  TABLE IX: WER (%) on Real Data of REVERB (1-channel
Input)

|  |  |  |  |  |  |  |  |  |  WER (%)$\downarrow$ on val. set |  WER (%)$\downarrow$ on test set
---|---|---|---|---|---|---|---|---|---|---|---
Row | Systems | Cross-reference | #CH in input/loss | $I/J/K/\Delta/L$ | $\alpha$ | $\theta$ |  PESQ$\uparrow$ |  eSTOI$\uparrow$ |  SI-SDR (dB)$\uparrow$ | Near | Far | Avg. | Near | Far | Avg.
1 | Mixture | - | 1 / - | - | - | - | $1.640\,610\,543\,333\,775\,2$ | $0.493\,757\,274\,092\,272$ | $-3.563\,633\,992\,835\,417$ | $15.35$ | $16.88$ | $16.11$ | $17.09$ | $17.29$ | $17.19$
2a | USDnet |  2a of Table II | 1 / 1 | 40 / 0 / 40 / 3 / - | $1$ | - | $1.759\,163\,199\,430\,006$ | $0.561\,225\,480\,546\,709$ | $-2.147\,380\,687\,798\,935\,7$ | $21.09$ | $24.20$ | $22.64$ | $20.73$ | $22.08$ | $21.41$
2b | USDnet |  2d of Table II | 1 / 8 | 40 / 0 / 40 / 3 / - | $1$ | - | $2.350\,717\,771\,170\,258$ | $0.742\,094\,320\,151\,134\,9$ | $2.230\,378\,166\,732\,061\,5$ | $11.73$ | $15.04$ | $13.38$ | $11.56$ | $12.63$ | $12.10$
2c | USDnet |  2 of Table VIII | 1 / 8 | 40 / 0 / 40 / 3 / - | $\frac{3}{8-1}$ | - | $2.480\,670\,537\,815\,585\,3$ | $0.748\,535\,241\,479\,823\,1$ | $2.224\,682\,997\,196\,528\,5$ | $10.29$ | $12.44$ | $11.36$ | $10.60$ | $11.75$ | $11.18$
2d | USDnet |  2a of Table VIII | 1 / 8 | 60 / 0 / 60 / 3 / - | $\frac{3}{8-1}$ | - | $2.446\,380\,413\,068\,283\,6$ | $0.756\,921\,563\,443\,627\,8$ | $2.495\,406\,880\,906\,926$ | $10.29$ | $13.33$ | $11.81$ | $10.19$ | $11.51$ | $10.85$
2e | USDnet |  3a of Table VIII | 1 / 8 | 60 / 0 / 60 / 3 / 1 | $\frac{3}{8-1}$ | - | $2.527\,006\,792\,700\,561\,3$ | $0.771\,691\,210\,136\,212\,6$ | $2.860\,394\,877\,577\,935\,5$ | $10.36$ | $12.44$ | $11.40$ | $10.06$ | $10.70$ | $10.38$
3 |  WPE [7] |  4a of Table VIII | 1 / - | - | - | - | $1.777\,453\,876\,250\,657$ | $0.579\,248\,917\,141\,486\,2$ | $-1.699\,056\,528\,695\,723\,5$ | $15.60$ | $15.99$ | $15.79$ | $13.19$ | $15.12$ | $14.15$

TABLE X: WER (%) on Real Data of REVERB (8-Channel Input)

|  |  |  |  |  |  |  |  |  |  WER (%)$\downarrow$ on val. set |  WER (%)$\downarrow$ on test set
---|---|---|---|---|---|---|---|---|---|---|---
Row | Systems | Cross-reference | #CH in input/loss | $I/J/K/\Delta/L$ | $\alpha$ | $\theta$ |  PESQ$\uparrow$ |  eSTOI$\uparrow$ |  SI-SDR (dB)$\uparrow$ | Near | Far | Avg. | Near | Far | Avg.
1 | Mixture | - | 8 / - | - | - | - | $1.640\,610\,543\,333\,775\,2$ | $0.493\,757\,274\,092\,272$ | $-3.563\,633\,992\,835\,417$ | $15.35$ | $16.88$ | $16.11$ | $17.09$ | $17.29$ | $17.19$
2a | USDnet |  3d of Table II | 8 / 8 | 40 / 0 / 40 / 3 / - | $1$ | $0$ | $2.330\,097\,478\,675\,667$ | $0.729\,609\,516\,419\,143\,1$ | $2.046\,792\,088\,948\,371\,7$ | $8.98$ | $11.07$ | $10.03$ | $8.88$ | $9.52$ | $9.20$
2b | USDnet |  4 of Table VIII | 8 / 8 | 40 / 0 / 40 / 3 / - | $\frac{3}{8-1}$ | $0$ | $2.398\,516\,595\,802\,155$ | $0.744\,936\,838\,972\,866\,1$ | $2.245\,083\,764\,643\,317\,3$ | $9.17$ | $11.89$ | $10.53$ | $8.46$ | $8.78$ | $8.62$
2c | USDnet |  2c of Table VIII | 8 / 8 | 40 / 0 / 40 / 3 / - | $\frac{3}{8-1}$ | $0.7$ | $2.423\,170\,863\,851\,164\,6$ | $0.746\,278\,609\,142\,762\,7$ | $2.145\,577\,937\,170\,951\,4$ | $10.36$ | $12.03$ | $11.20$ | $10.28$ | $9.32$ | $9.80$
2d | USDnet |  2b of Table VIII | 8 / 8 | 60 / 0 / 60 / 3 / - | $\frac{3}{8-1}$ | $0.7$ | $2.421\,752\,320\,009\,824\,5$ | $0.759\,528\,334\,315\,908\,6$ | $2.578\,174\,839\,644\,560\,7$ | $9.86$ | $11.96$ | $10.91$ | $9.65$ | $9.79$ | $9.72$
2e | USDnet |  3b of Table VIII | 8 / 8 | 60 / 0 / 60 / 3 / 1 | $\frac{3}{8-1}$ | $0.7$ | $2.451\,282\,974\,111\,018$ | $0.760\,557\,652\,709\,721\,1$ | $2.675\,615\,118\,266\,085$ | $10.67$ | $11.35$ | $11.01$ | $9.45$ | $9.62$ | $9.54$
4 |  WPE [7] |  4b of Table VIII | 8 / - | - | - | - | $2.022\,079\,673\,301\,654\,6$ | $0.689\,944\,716\,942\,314\,2$ | $1.998\,465\,577\,307\,669\,2$ | $15.47$ | $19.34$ | $17.41$ | $11.82$ | $13.94$ | $12.88$

###  VI-L Illustration of Dereverberation Results

Fig. 3 illustrates and compares the dereverberation results of USDnet and WPE,
based on a simulated reverberant speech signal sampled from WSJ0CAM-DEREVERB.
We observe that USDnet produces clear suppression of reverberation and clear
reconstruction of target speech patterns, while WPE only reduces reverberation
slightly and the speech patterns are still very smeared. This comparison
suggests that USDnet can better reduce reverberation than WPE. A sound demo is
provided (see the end of Section I).

###  VI-M ASR Results on REVERB

In Table X and X, we evaluate USDnet trained on WSJ0CAM-DEREVERB directly on
the ASR tasks of REVERB. Although WSJ0CAM-DEREVERB is simulated and is
mismatched with the real-recorded REVERB dataset, we observe that USDnet can
significantly improve ASR performance in both near- and far-field cases over
the unprocessed mixtures. In both tables, USDnet outperforms WPE.

In Table X, we observe that applying input microphone dropout (see the
$\theta$ column), although producing better enhancement scores, is detrimental
to ASR.

Comparing the results in Table X and X, we observe that USDnet with eight-
channel input obtains better recognition performance but worse PESQ, eSTOI and
SI-SDR scores than the one with monaural input.

##  VII Conclusion

We have formulated unsupervised neural speech dereverberation as a blind
deconvolution problem, which requires estimating both the source and filter,
and proposed a novel algorithm named USDnet, which leverages over-determined
training mixtures as constraints to narrow down the solutions to target
speech. Evaluation results on two dereverberation tasks show that USDnet, even
if only trained on a set of reverberant mixtures in an unsupervised way, can
learn to effectively reduce reverberation. Future research will modify USDnet
and evaluate it on real-recorded conversational speech dereverberation and
separation tasks with multiple sound sources.

A major contribution of this paper, we emphasize, is our finding that DNNs can
be trained directly on a set of reverberant mixtures, in an unsupervised way,
to reduce reverberation, by solely exploiting mixture constraints afforded by
multi-microphone mixtures. The novel way of modeling reverberation and the
novel concept of leveraging mixture constraints exploit signal cues and
physical principals very different from the ones utilized by existing
algorithms. The proposed algorithms, hence, we believe, would in many ways
complement existing dereverberation and separation algorithms, and motivate
new ones.

In closing, another major contribution of this paper, we point out, is our
unsupervised deep learning based approach for solving blind deconvolution
problems, which exist not only in speech dereverberation but also in many
other engineering applications. By modeling source priors in an unsupervised
way, filter estimation becomes a differentiable operation given a source
estimate. This enables us to train DNNs on signal patterns in a discriminative
way to optimize mixture-constraint loss and realize unsupervised
deconvolution. This novel methodological contribution, we believe, would
generate broader impact beyond speech dereverberation.

## References

  * [1] E. A. P. Habets and P. A. Naylor, “Dereverberation,” in _Audio Source Separation and Speech Enhancement_ , E. Vincent, T. Virtanen, and S. Gannot, Eds. Wiley, 2018, pp. 317–343.
  * [2] A. Levin, Y. Weiss, F. Durand, and W. T. Freeman, “Understanding Blind Deconvolution Algorithms,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 33, no. 12, pp. 2354–2367, 2011.
  * [3] “Blind Deconvolution.” [Online]. Available: https://en.wikipedia.org/wiki/Blind_deconvolution
  * [4] T. Yoshioka, A. Sehr, M. Delcroix, K. Kinoshita, R. Maas, T. Nakatani, and W. Kellermann, “Making Machines Understand Us in Reverberant Rooms: Robustness against Reverberation for Automatic Speech Recognition,” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 114–126, 2012.
  * [5] S. Gannot, E. Vincent, S. Markovich-Golan, and A. Ozerov, “A Consolidated Perspective on Multi-Microphone Speech Enhancement and Source Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 25, pp. 692–730, 2017\.
  * [6] R. Haeb-Umbach, J. Heymann, L. Drude, S. Watanabe, M. Delcroix, and T. Nakatani, “Far-Field Automatic Speech Recognition,” _Proc. IEEE_ , 2020\.
  * [7] T. Nakatani, T. Yoshioka, K. Kinoshita, M. Miyoshi, and B.-H. Juang, “Speech Dereverberation Based on Variance-Normalized Delayed Linear Prediction,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 18, no. 7, pp. 1717–1731, 2010.
  * [8] A. Jukiać, T. Van Waterschoot, T. Gerkmann, and S. Doclo, “Multi-Channel Linear Prediction-Based Speech Dereverberation With Sparse Priors,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 23, no. 9, pp. 1509–1520, 2015.
  * [9] A. Jukić, T. Van Waterschoot, T. Gerkmann, and S. Doclo, “A General Framework for Incorporating Time-Frequency Domain Sparsity in Multichannel Speech Dereverberation,” _Journal of the Audio Engineering Society_ , vol. 65, no. 1-2, pp. 17–30, 2017.
  * [10] T. Nakatani and K. Kinoshita, “A Unified Convolutional Beamformer for Simultaneous Denoising and Dereverberation,” in _IEEE Signal Process. Lett._ , vol. 26, no. 6, 2019, pp. 903–907.
  * [11] C. Boeddeker, T. Nakatani, K. Kinoshita, and R. Haeb-Umbach, “Jointly Optimal Dereverberation and Beamforming,” in _Proc. ICASSP_ , 2020, pp. 216–220.
  * [12] K. Han, Y. Wang, D. Wang, W. S. Woods, I. Merks, and T. Zhang, “Learning Spectral Mapping for Speech Dereverberation and Denoising,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 23, no. 6, pp. 982–992, 2015.
  * [13] T. May, “Robust Speech Dereverberation with A Neural Network-Based Post-Filter that Exploits Multi-Conditional Training of Binaural Cues,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , 2017.
  * [14] O. Ernst, S. E. Chazan, S. Gannot, and J. Goldberger, “Speech Dereverberation using Fully Convolutional Networks,” in _Proc. EUSIPCO_ , 2018, pp. 390–394.
  * [15] Y. Zhao, D. Wang, B. Xu, and T. Zhang, “Late Reverberation Suppression using Recurrent Neural Networks with Long Short-Term Memory,” in _Proc. ICASSP_ , 2018, pp. 5434–5438.
  * [16] ——, “Monaural Speech Dereverberation using Temporal Convolutional Networks with Self Attention,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 28, pp. 1598–1607, 2020.
  * [17] Z.-Q. Wang and D. Wang, “Deep Learning Based Target Cancellation for Speech Dereverberation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 28, pp. 941–950, 2020.
  * [18] ——, “Multi-Microphone Complex Spectral Mapping for Speech Dereverberation,” in _Proc. ICASSP_ , 2020, pp. 486–490.
  * [19] V. Kothapally and J. H. Hansen, “SkipConvGAN: Monaural Speech Dereverberation using Generative Adversarial Networks via Complex Time-Frequency Masking,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 30, pp. 1600–1613, 2022.
  * [20] J. M. Lemercier, J. Thiemann, R. Koning, and T. Gerkmann, “A Neural Network-Supported Two-Stage Algorithm for Lightweight Dereverberation on Hearing Devices,” _Eurasip Journal on Audio, Speech, and Music Processing_ , no. 1, 2023.
  * [21] V. Kothapally and J. H.L. Hansen, “Monaural Speech Dereverberation using Deformable Convolutional Networks,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , 2024.
  * [22] K. Kinoshita, M. Delcroix, H. Kwon, T. Mori, and T. Nakatani, “Neural Network-Based Spectrum Estimation for Online WPE Dereverberation,” in _Proc. Interspeech_ , 2017, pp. 384–388.
  * [23] J. Heymann, L. Drude, R. Haeb-Umbach, K. Kinoshita, and T. Nakatani, “Frame-Online DNN-WPE Dereverberation,” in _Proc. IWAENC_ , 2018, pp. 466–470.
  * [24] ——, “Joint Optimization of Neural Network-Based WPE Dereverberation and Acoustic Model for Robust Online ASR,” in _Proc. ICASSP_ , 2019, pp. 6655–6659.
  * [25] L. Drude, J. Heymann, C. Boeddeker, and R. Haeb-Umbach, “NARA-WPE: A Python Package for Weighted Prediction Error Dereverberation in Numpy and Tensorflow for Online and Offline Processing,” in _ITG Speech Communication_ , 2020, pp. 216–220.
  * [26] W. Zhang, A. S. Subramanian, X. Chang, S. Watanabe, and Y. Qian, “End-to-End Far-Field Speech Recognition with Unified Dereverberation and Beamforming,” in _Proc. Interspeech_ , 2020, pp. 324–328.
  * [27] Z.-Q. Wang, G. Wichern, and J. Le Roux, “Convolutive Prediction for Monaural Speech Dereverberation and Noisy-Reverberant Speaker Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 29, pp. 3476–3490, 2021.
  * [28] J. Richter, S. Welker, J. Marie Lemercier, B. Lay, and T. Gerkmann, “Speech Enhancement and Dereverberation With Diffusion-Based Generative Models,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 31, pp. 2351–2364, 2023.
  * [29] J. Barker, S. Watanabe, E. Vincent, and J. Trmal, “The Fifth ’CHiME’ Speech Separation and Recognition Challenge: Dataset, Task and Baselines,” in _Proc. Interspeech_ , 2018, pp. 1561–1565.
  * [30] S. Watanabe, M. Mandel, J. Barker, E. Vincent, A. Arora, X. Chang _et al._ , “CHiME-6 Challenge: Tackling Multispeaker Speech Recognition for Unsegmented Recordings,” in _Proc. CHiME_ , 2020, pp. 1–7.
  * [31] W. Zhang, J. Shi, C. Li, S. Watanabe, and Y. Qian, “Closing The Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions,” in _Proc. WASPAA_ , 2021, pp. 146–150.
  * [32] E. Tzinis, Y. Adi, V. K. Ithapu, B. Xu, P. Smaragdis, and A. Kumar, “RemixIT: Continual Self-Training of Speech Enhancement Models via Bootstrapped Remixing,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1329–1341, 2022.
  * [33] S. Leglaive, L. Borne, E. Tzinis, M. Sadeghi, M. Fraticelli, S. Wisdom, M. Pariente, D. Pressnitzer, and J. R. Hershey, “The CHiME-7 UDASE Task: Unsupervised Domain Adaptation for Conversational Speech Enhancement,” in _arXiv preprint arXiv:2307.03533_ , 2023.
  * [34] Z.-Q. Wang and S. Watanabe, “UNSSOR: Unsupervised Neural Speech Separation by Leveraging Over-determined Training Mixtures,” in _Proc. NeurIPS_ , 2023\.
  * [35] S. Wisdom, E. Tzinis, H. Erdogan, R. J. Weiss, K. Wilson, and J. R. Hershey, “Unsupervised Sound Separation using Mixture Invariant Training,” in _Proc. NeurIPS_ , 2020.
  * [36] K. Saijo and T. Ogawa, “Self-Remixing: Unsupervised Speech Separation via Separation and Remixing,” in _Proc. ICASSP_ , 2023.
  * [37] T. Fujimura, Y. Koizumi, K. Yatabe, and R. Miyazaki, “Noisy-target Training: A Training Strategy for DNN-based Speech Enhancement without Clean Speech,” in _Proc. EUSIPCO_ , 2021, pp. 436–440.
  * [38] T. Fujimura and T. Toda, “Analysis of Noisy-Target Training For DNN-Based Speech Enhancement,” in _Proc. ICASSP_ , 2023.
  * [39] Y. Nakagome, M. Togami, T. Ogawa, and T. Kobayashi, “Efficient and Stable Adversarial Learning using Unpaired Data for Unsupervised Multichannel Speech Separation,” in _Proc. Interspeech_ , 2021, pp. 3051–3055.
  * [40] X. Bie, S. Leglaive, X. Alameda-Pineda, and L. Girin, “Unsupervised Speech Enhancement using Dynamical Variational Autoencoders,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 30, pp. 2993–3007, 2022.
  * [41] X. Hao, C. Xu, and L. Xie, “Neural Speech Enhancement with Unsupervised Pre-Training and Mixture Training,” _Neural Networks_ , vol. 158, pp. 216–227, 2023.
  * [42] K. Saito, N. Murata, T. Uesaka, C.-H. Lai, Y. Takida, T. Fukui, and Y. Mitsufuji, “Unsupervised Vocal Dereverberation with Diffusion-Based Generative Models,” in _Proc. ICASSP_ , 2023, pp. 1–5.
  * [43] S.-W. Fu, C. Yu, K.-H. Hung, M. Ravanelli, and Y. Tsao, “MetricGAN-U: Unsupervised Speech Enhancement / Dereverberation Based Only on Noisy / Reverberated Speech,” in _Proc. ICASSP_ , 2022, pp. 7412–7416.
  * [44] E. Tzinis, S. Venkataramani, and P. Smaragdis, “Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures Using Spatial Information,” in _Proc. ICASSP_ , 2019, pp. 81–85.
  * [45] L. Drude, D. Hasenklever, and R. Haeb-Umbach, “Unsupervised Training of a Deep Clustering Model for Multichannel Blind Source Separation,” in _Proc. ICASSP_ , 2019, pp. 695–699.
  * [46] P. Seetharaman, G. Wichern, J. Le Roux, and B. Pardo, “Bootstrapping Single-Channel Source Separation via Unsupervised Spatial Clustering on Stereo Mixtures,” in _Proc. ICASSP_ , 2019, pp. 356–360.
  * [47] M. Togami, Y. Masuyama, T. Komatsu, and Y. Nakagome, “Unsupervised Training for Deep Speech Source Separation with Kullback-Leibler Divergence Based Probabilistic Loss Function,” in _Proc. ICASSP_ , 2020, pp. 56–60.
  * [48] P. N. Petkov, V. Tsiaras, R. Doddipatla, and Y. Stylianou, “An Unsupervised Learning Approach to Neural-Net-Supported WPE Dereverberation,” in _Proc. ICASSP_ , 2019, pp. 5761–5765.
  * [49] L. Drude, J. Heymann, and R. Haeb-Umbach, “Unsupervised Training of Neural Mask-based Beamforming,” in _Proc. Interspeech_ , 2019, pp. 1253–1257.
  * [50] Y. Bando, K. Sekiguchi, Y. Masuyama, A. A. Nugraha, M. Fontaine, and K. Yoshii, “Neural Full-Rank Spatial Covariance Analysis for Blind Source Separation,” _IEEE Signal Process. Lett._ , vol. 28, pp. 1670–1674, 2021\.
  * [51] P. Wang and X. Li, “RVAE-EM: Generative Speech Dereverberation Based on Recurrent Variational Auto-Encoder and Convolutive Transfer Function,” in _arXiv preprint arXiv:2309.08157_ , 2023.
  * [52] N. Ito and M. Sugiyama, “Audio Signal Enhancement with Learning from Positive and Unlabeled Data,” in _Proc. ICASSP_ , 2023.
  * [53] R. Talmon, I. Cohen, and S. Gannot, “Relative Transfer Function Identification using Convolutive Transfer Function Approximation,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 17, no. 4, pp. 546–555, 2009.
  * [54] Z.-Q. Wang, P. Wang, and D. Wang, “Complex Spectral Mapping for Single-and Multi-Channel Speech Enhancement and Robust ASR,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 28, pp. 1778–1787, 2020.
  * [55] ——, “Multi-Microphone Complex Spectral Mapping for Utterance-Wise and Continuous Speech Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 29, pp. 2001–2014, 2021.
  * [56] D. S. Williamson, Y. Wang, and D. Wang, “Complex Ratio Masking for Monaural Speech Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , pp. 483–492, 2016.
  * [57] D. S. Williamson and D. Wang, “Speech Dereverberation and Denoising using Complex Ratio Masks,” in _Proc. ICASSP_ , 2017, pp. 5590–5594.
  * [58] D. Wang and J. Chen, “Supervised Speech Separation Based on Deep Learning: An Overview,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 26, pp. 1702–1726, 2018.
  * [59] Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, “TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 31, pp. 3221–3236, 2023.
  * [60] Z.-Q. Wang, G. Wichern, and J. Le Roux, “On The Compensation Between Magnitude and Phase in Speech Separation,” _IEEE Signal Process. Lett._ , vol. 28, pp. 2018–2022, 2021.
  * [61] ——, “Convolutive Prediction for Reverberant Speech Separation,” in _Proc. WASPAA_ , 2021, pp. 56–60.
  * [62] M. Kolbæk, D. Yu, Z.-H. Tan, and J. Jensen, “Multi-Talker Speech Separation with Utterance-Level Permutation Invariant Training of Deep Recurrent Neural Networks,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 25, no. 10, pp. 1901–1913, 2017.
  * [63] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler, “Efficient Object Localization using Convolutional Networks,” in _Proceedings of CVPR_ , 2015, pp. 648–656.
  * [64] H. Sawada, N. Ono, H. Kameoka, D. Kitamura, and H. Saruwatari, “A Review of Blind Source Separation Methods: Two Converging Routes to ILRMA Originating from ICA and NMF,” _APSIPA Transactions on Signal and Information Processing_ , vol. 8, pp. 1–14, 2019.
  * [65] M. I. Mandel, R. J. Weiss, and D. P. W. Ellis, “Model-Based Expectation-Maximization Source Separation and Localization,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 18, no. 2, pp. 382–394, 2010.
  * [66] C. Boeddeker, F. Rautenberg, and R. Haeb-Umbach, “A Comparison and Combination of Unsupervised Blind Source Separation Techniques,” in _ITG Conference on Speech Communication_ , 2021, pp. 129–133.
  * [67] Z.-Q. Wang, G. Wichern, and J. Le Roux, “Leveraging Low-Distortion Target Estimates for Improved Speech Enhancement,” _arXiv preprint arXiv:2110.00570_ , 2021.
  * [68] K. Kinoshita, M. Delcroix, S. Gannot, E. A. Emanuël, R. Haeb-Umbach, W. Kellermann, V. Leutnant, R. Maas, T. Nakatani, B. Raj, A. Sehr, and T. Yoshioka, “A Summary of The REVERB Challenge: State-of-The-Art and Remaining Challenges in Reverberant Speech Processing Research,” _Eurasip J. Adv. Signal Process._ , vol. 2016, no. 1, pp. 1–19, 2016.
  * [69] L. Drude, J. Heymann, C. Boeddeker, and R. Haeb-Umbach, “NARA-WPE: A Python Package for Weighted Prediction Error Dereverberation in Numpy and Tensorflow for Online and Offline Processing,” in _ITG-Fachtagung Sprachkommunikation_ , 2020, pp. 216–220.
  * [70] A. Rix, J. Beerends, M. Hollier, and A. Hekstra, “Perceptual Evaluation of Speech Quality (PESQ)-A New Method for Speech Quality Assessment of Telephone Networks and Codecs,” in _Proc. ICASSP_ , vol. 2, 2001, pp. 749–752.
  * [71] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An Algorithm for Intelligibility Prediction of Time–Frequency Weighted Noisy Speech,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 19, no. 7, pp. 2125–2136, 2011.
  * [72] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR – Half-Baked or Well Done?” in _Proc. ICASSP_ , 2019, pp. 626–630.