# Wideband Audio Waveform Evaluation Networks: Efficient, Accurate Estimation
of Speech Qualities

Andrew Catellier, and Stephen Voran A. Catellier and S. Voran are with the
Institute for Telecommunication Sciences, Boulder, CO, 80305 USA e-mail:
<EMAIL_ADDRESS>
###### Abstract

Wideband Audio Waveform Evaluation Networks (WAWEnets) are convolutional
neural networks that operate directly on wideband audio waveforms in order to
produce evaluations of those waveforms. In the present work these evaluations
give qualities of telecommunications speech (e.g., noisiness, intelligibility,
overall speech quality). WAWEnets are no-reference networks because they do
not require “reference” (original or undistorted) versions of the waveforms
they evaluate. Our initial WAWEnet publication introduced four WAWEnets and
each emulated the output of an established full-reference speech quality or
intelligibility estimation algorithm.

We have updated the WAWEnet architecture to be more efficient and effective.
Here we present a single WAWEnet that closely tracks seven different quality
and intelligibility values. We create a second network that additionally
tracks four subjective speech quality dimensions. We offer a third network
that focuses on just subjective quality scores and achieves very high levels
of agreement. This work has leveraged 334 hours of speech in 13 languages,
over two million full-reference target values and over 93,000 subjective mean
opinion scores.

We also interpret the operation of WAWEnets and identify the key to their
operation using the language of signal processing: ReLUs strategically move
spectral information from non-DC components into the DC component. The DC
values of 96 output signals define a vector in a 96-D latent space and this
vector is then mapped to a quality or intelligibility value for the input
waveform.

###### Index Terms:

convolutional neural network, no-reference objective estimator, speech
intelligibility, speech quality, subjective testing, wideband speech.

##  I Introduction

Wired and wireless telecommunications options continue to proliferate and
evolve. Speech quality and speech intelligibility can vary dramatically
between systems and devices, and further variation is driven by changes in
acoustic environment, network loading, and radio conditions. Efficient real-
time measurements of received speech quality and intelligibly are invaluable.
Researchers have been developing such measurement tools for decades but
telecommunications systems, devices, and use cases continue to evolve. This
can cause entirely new classes of speech impairments and existing measurement
tools may then fail to give meaningful results. This scenario motivates the
development of new tools that do give meaningful results in the current
environment.

Measurement tools fall into two major classifications—full-reference (FR)
tools require reference (transmitted) and impaired (received) speech and are
practical in off-line applications. No-reference (NR) tools operate on
impaired speech alone and are much more practical for real-time, in-service
monitoring. Some examples of FR tools developed over the years can be found in
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. The most effective FR tools apply
psycho-acoustic transformations to reference and impaired speech and then
compare these internal representations in ways that mimic key attributes of
human judgement.

NR tools eliminate the need for time alignment and the issue of comparisons,
but they require an embodied model for how speech should sound, independent of
what speech was sent. This is a significant challenge, but success allows for
practical real-time, in-service monitoring of speech quality or
intelligibility. Thus these tools have also been named “single-ended,” “in-
service,” “non-intrusive,” or “output only.” Some examples of early NR tools
are found in [14, 15, 16].

###  I-A Existing Machine Learning Approaches

As machine learning (ML) has become more powerful and accessible, numerous
research groups have sought to apply ML to develop NR tools [17, 18, 19, 20,
21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50]. Some of these NR tools produce
estimates of subjective test scores that report speech or sound quality mean
opinion score (MOS) [17, 18, 19, 24, 25, 26, 27, 30, 35, 38, 40, 41, 46, 47],
naturalness [28, 34, 36], listening effort [23], noise intrusiveness [47], and
speech intelligibility [20, 32]. The non-intrusive speech quality assessment
model called NISQA [50] produces estimates of subjective speech quality as
well as four constituent dimensions: noisiness, coloration, discontinuity, and
loudness. Other NR tools produce estimates of objective values including FR
speech quality values [22, 29, 31, 42, 48], FR speech intelligibility values
[29, 31, 42, 49], speech transmission index [21], codec bit-rate [43], and
detection of specific impairments, artifacts, or noise types [33, 37, 39, 49].
Some of these tools perform a single task and others perform multiple tasks.

The works cited here cover a wide variety of ML architectures. They address
application areas that include room acoustics (noise and reverberation),
speech enhancers, telecommunications systems, and hearing aids. Each addresses
one or more of narrowband (NB) (nominally 300 Hz–3.5 kHz), wideband (WB)
(nominally 50 Hz–7 kHz), super-wideband (SWB) (nominally 50 Hz–16 kHz), or
fullband (FB) (nominally 20 Hz–20 kHz) speech, except for [33, 43] which
address music. Recent work shows that NR-tools can measure the speech quality
at a system input in spite of the fact that such tools can only access the
system output. [44].

It is natural to consider agreement with subjective test results as the
ultimate goal for NR tools. For ML-based tools this requires large datasets of
subjective test results (most commonly speech quality MOS values) for training
and validation. Sufficient datasets are rare and expensive to generate through
laboratory testing, so crowd-sourced tests are becoming common. Joint training
or transfer learning can leverage objective FR quality values [25, 30, 47] or
impairment categories [46] alongside MOS values to maximize the benefit of
those MOS values. Semi-supervised learning [40] is also an effective way to
compensate for scarce MOS values.

It is important to recognize that the MOS scale is relative, not absolute, and
the use of the scale in any given subjective test will depend on the
conditions included in that test. For example, a given condition might be be
rated 4.0 when it appears in a test with lower quality conditions. But that
same condition might be rated 3.0 when it appears in a test with higher
quality conditions. Per-subject corrections are explored in [47]. A method to
learn per-test correction factors is given in [45] and this approach may be
viewed as the ML version of the linear algebra solution previously given in
[51].

Key considerations when using ML to develop NR tools are the total amount of
data available, how that data is used to both train and test a model, and the
homogeneity or diversity of the training and testing data. The number of
talkers, languages, and sources of impairment are also potentially important
factors, depending on the application. Because the speech quality measurement
community has not yet settled on standardized datasets, some published work is
backed by extensive data and rigorous testing while other work could be
described as an initial proof-of-concept for some specific architecture or
application area, backed by a domain-specific dataset. For this reason three
separate datasets with three different application areas are considered in
this work.

###  I-B The WAWEnet Machine Learning Approach

The most common ML approach applies ML to a set of features that form a
spectral description, often one that uses a perceptual frequency scale (e.g.
mel, gammatone, or third-octave bands) and that may use the log of the
spectral magnitudes since the log function can very roughly approximate the
human perception of loudness. Alternately, ML has been applied to features
such as mel-frequency cepstral coefficients, delta MFCCs, perceptual linear
prediction coefficients, pitch, voice activity, frame energy, and features
used in the legacy NR tools [15, 16].

One distinction of WAWEnets is that they apply ML directly to speech waveforms
instead of applying it to features extracted from speech waveforms. Extracting
features from waveforms is a data compression step—it reduces the number of
inputs per unit time that the ML algorithm must process. Good features
identify the most useful information in the speech waveform so that the ML
algorithm may more easily and efficiently map that information to the desired
target value.

But in general, feature extraction is not lossless data compression. The most
popular features apply the magnitude function to complex-valued time-to-
frequency transform results. This clearly removes information, and that
information cannot be restored. Feature extraction makes assumptions about
what information is important and what can be discarded. While many feature-
based approaches have been successful in this problem space, WAWEnets
demonstrate that using waveforms directly is both practical and effective.
Thanks to the convolutional neural network architecture, WAWEnets are not
excessively large, yet they can process waveform samples directly, thus
gaining access to all the information those samples carry and effectively
generating the best features for the task internally, rather than having
potentially sub-optimal features mandated externally.

Before our initial WAWEnet publication [31] we found just one other NR
research effort that considered speech waveforms for input to ML. In [21] ML
is applied to a spectrogram representation of audio in order to estimate
speech transmission index. But this spectrogram representation is also
learned—waveforms are processed with 128 filters, each with length 128, and
those filters are “initialized with a Fourier basis (sine waves at different
frequencies).” So in effect, ML is actually applied to the audio waveform, but
with a strong suggestion that the first ML step should be to calculate a
spectral representation. This spectral representation does not use the
magnitude function and thus it has the potential to preserve all of the
information in the audio waveform.

We are aware of three additional waveform-oriented research efforts that have
emerged since [31] was published. In [40] ML is applied to $\mu$-law
compressed speech in order to estimate speech quality MOS. But the value of
$\mu$ is learned (it is initialized to 256 as in G.711 [52]) so in effect, ML
is actually applied to the speech waveform, but with the strong suggestion
that the first ML step should be compression. No quantization is added, so
this approach can preserve all of the information in the speech waveform.

In [42] the MOS estimation task is learned in two different ways. Speech
waveforms are first processed by either a learnable 1-D convolution layer, or
by the “conventional STFT,” and then supplied to the main ML architecture.
Authors report that compared to the STFT, “the learnable 1-D convolution layer
leads to slight improvements for all targets in nearly all criteria.” This
result suggests that allowing ML to operate on waveforms is preferable to
providing it a spectral representation. And finally, in [53] a WAWEnet-style
architecture is applied to waveforms to generate quality information to guide
dialog remixing.

In [31] we introduced four WAWEnets. Three of these emulated FR speech quality
tools POLQA [10], WB-PESQ [8], and PEMO [7]. The fourth WAWEnet emulated the
FR speech intelligibility tool STOI [54].

Since that time we have updated the WAWEnet architecture to be more efficient
and effective and have leveraged large amounts of data to train three very
effective WAWEnets. Section II describes the new architecture. In Section III
we describe how we used 252 hours of speech from 13 languages to train and
test individual WAWEnets to very closely emulate seven FR tools (four speech-
quality and three speech-intelligibility). For comparison purposes we provide
results from FR versions of these WAWEnets as well. Next we present a single
WAWEnet that emulates all seven FR tools at once.

Section IV introduces 30 hours of subjectively-rated speech and we train
another WAWEnet to produce values for four subjective rating dimensions as
well as the seven FR quality and intelligibility values. In Section V we train
and evaluate a third WAWEnet using 52 hours of speech with MOS scores. This
WAWEnet tracks MOS scores quite closely, confirming that the single
architecture can be trained to produce very good results for both FR objective
targets and subjective targets.

Using ML to develop NR tools for evaluating speech and audio is presently a
rich and active field, with many successful results already published. Our
work is novel because we adopt a relatively simple convolutional architecture,
we apply it directly to speech waveforms, we successfully train to closely
emulate eleven different targets, and we use an unprecedented quantity and
diversity of speech data. Finally Section VII uses the language of signal
processing to explain how WAWEnets process speech signals to produce
evaluations of themselves. To our knowledge, interpretations of this sort have
never before been published.

Figure 1: Diagram depicting the overall shape of our WAWEnet model. Each
rectangle represents one section and the enclosed text describes the shape of
the section’s input vector. Starting with 48,000 speech samples, the model
coalesces information in the time domain until a 96-dimension feature vector
remains.

##  II WAWEnet Architecture

Similar to the model we described in [55], WAWEnets accept 3 seconds of speech
sampled at 16,000 smp/s as input. The architecture is composed of sections
that use one-dimensional, 96 channel convolutional layers to filter the signal
and pooling layers which effectively downsample the signal.

This model differs from the model described in [55] in a few key ways. We
added additional convolutional sections that downsample all the way to one
“sample” per channel for each of the 96 channels, thus yielding a 96-dimension
feature vector. A dense layer then maps the feature vector to an estimate.
This model therefore only uses convolutional layers and downsampling to
coalesce temporal information, whereas the previous model also used a dense
layer to coalesce information across a temporal and a feature dimension. All
PReLU activations have been replaced with ReLU, and average pooling is used
throughout. The first section downsamples the input signal by 4, thereby
significantly reducing memory and computation requirements.

In this work we allow the dense layer in $S_{14}$ to generate $N_{T}\geq 1$
estimates. This model is visualized in Fig. 1 and is fully described by Tables
I and II. With $N_{T}=1$, the new formulation has a total of 335,905
parameters, a roughly 50% increase over the 225,025 parameters of our previous
model. For brevity, we refer to specific WAWEnet implementations using a
subscript that can indicate the type and number ($N_{T}$) of estimates
produced. For example, $\text{WAWEnet}_{11}$ is a WAWEnet with one input
channel and 11 estimates. In addition, $\text{WAWEnet}_{\text{S}1}$ estimates
one subjective target and $\text{WAWEnet}_{\text{O}1}$ estimates one objective
FR target.

We also made a separate WAWEnet configuration that allows $S_{1}$ to accept
two 48,000 sample vectors. We used this configuration to create an FR WAWEnet
that uses reference and impaired speech ($\text{WAWEnet}_{1}^{\text{FR}}$)
and, for comparison purposes, a WAWEnet that uses two identical copies of the
impaired speech ($\text{WAWEnet}_{1}^{\text{2}}$).

TABLE I: WAWEnet architecture: sections $S_{1}$–$S_{13}$ are composed of one of the two section types listed in Table II. Number of input and output samples per channel are given by $l_{in}$ and $l_{out}$, effective sample rate by $\hat{f_{s}}$, and effective sample spacing by $s_{l}$. The dense layer $S_{14}$ maps $f_{n}=96$ scalar outputs from $S_{13}$ to the $N_{T}$ estimates. $S$ | section type |  $\hat{f_{s}}$ (Hz) | $l_{in}$ |  $s_{l}$ (ms) | $l_{out}$
---|---|---|---|---|---
$S_{1}$ | Conv A-4 | 16,000 | 48,000 | 0.0625 | 12,000
$S_{2}$ | Conv A-2 | 4,000 | 12,000 | 0.25 | 6,000
$S_{3}$ | Conv A-2 | 2,000 | 6,000 | 0.5 | 3,000
$S_{4}$ | Conv A-4 | 1,000 | 3,000 | 1 | 750
$S_{5}$ | Conv A-2 | 250 | 750 | 4 | 375
$S_{6}$ | P Conv A-2 | 125 | 376 | 8 | 188
$S_{7}$ | Conv A-2 | 62.50 | 188 | 16 | 94
$S_{8}$ | Conv A-2 | 31.25 | 94 | 32 | 47
$S_{9}$ | P Conv A-2 | 15.63 | 48 | 64 | 24
$S_{10}$ | Conv A-2 | 7.81 | 24 | 128 | 12
$S_{11}$ | Conv A-2 | 3.91 | 12 | 256 | 6
$S_{12}$ | Conv A-2 | 1.95 | 6 | 512 | 3
$S_{13}$ | Conv A-3 | 0.98 | 3 | 1024 | 1
$S_{14}$ | Dense | — | $f_{n}$ | — | $N_{T}$
TABLE II: WAWEnet section types. Each section contains a 1-D convolution layer C-$f_{n}$-$f_{l}$ with $f_{n}$$=96$ filters per channel and $f_{n}$ channels, filter length $f_{l}=3$, stride of 1, and zero padding $\lfloor\frac{f_{l}}{2}\rfloor=1$. Padding layer $\text{Pad}(a,b)$ prepends $a$ zeros and appends $b$ zeros to the input vector. ReLU indicates a ReLU activation. $k$ denotes average pooling layer kernel size. name | Conv A-$k$ | P Conv A-$k$
---|---|---
|  | $\text{Pad}(0,1)$
| C-$f_{n}$-$f_{l}$ | C-$f_{n}$-$f_{l}$
| BatchNorm | BatchNorm
| ReLU | ReLU
layers | AvgPool-$k$ | AvgPool-$k$

##  III WAWEnet for Seven Objective FR Targets

###  III-A Data

The Institute for Telecommunication Sciences (ITS) dataset is formed from
high-quality WB speech recordings. We carefully selected these from an array
of sources including [56, 57, 58, 59, 60, 61, 62, 63, 64, 65] and also from
recordings made in our lab. We extracted three-second “reference segments”
from these recordings such that each reference segment has a speech activity
factor (SAF) of 50% or greater (determined by the P.56 Speech Voltmeter found
in [66]) and any segment has at most 50% (1.5 sec) content in common with any
other segment. We then normalized each reference segment to have an active
speech level of $26\pm 0.2$ dB below the overload point, again using the P.56
Speech Voltmeter.

This process provided 84 hours of speech in the form of 100,681 reference
segments, representing 13 languages and 1230 different talkers. Spanish,
Mandarin, and North American English each account for 29% of the segments.
Korean contributes 6% of the segments, African-accented French 3%, Japanese
2%, while Italian, French, German, Portuguese, Hindi, British English,
Finnish, Dutch, and Polish combine to contribute the remaining 2%.

We strategically split the reference segments into training, testing, and
validation groups, as well as an unseen group. We generated the unseen group
by reserving 10% of the talkers in each language, to the extent possible. The
resulting 127 talkers associated with the unseen group do not contribute any
segments to the remaining data and are only used for evaluation. The unseen
dataset contains 10,391 segments (9 hours) of speech. We split the remaining
segments into training (50%), testing (40%), and validation groups (10%) with
approximate sizes of 38, 30, and 7 hours respectively.

We processed each reference segment with software to simulate the impairments
found in a wide range of current telecommunications environments. These
impairments cover three classes: background noise and suppression, speech
codecs, and packet loss and concealment. Background noises include coffee
shop, party, and street noise at SNRs between 5 and 25 dB. Noise suppression
ranged from mild (minimal artifacts but significant noise) to very aggressive
(significant artifacts and distortion). We applied 6 WB codec algorithms: EVS,
AMR-WB, Opus, G.711.1, G.722.1, and G.722. We selected bit-rates ranging from
8 to 64 kbps for a total of 49 WB codec modes. We used 13 different NB codecs
including EVS, AMR, Opus, G.711, G.729, G.723.1, G.726, MELP and others. Bit-
rates range from 1.2 to 64 kbps for a total of 40 NB codec modes. We applied
both independent and bursty packet losses at rates ranging from 5 to 40%
followed by packet loss concealment (PLC). Finally, we normalized each
impaired segment to have an active speech level of $26\pm 0.2$ dB below the
overload point.

Each reference segment was impaired in three different ways: a randomly
selected NB noise or codec impairment, a randomly selected WB noise or codec
impairment, and a randomly selected NB or WB impairment that combined a codec
with noise or PLC or both. This produced roughly 302,000 segments of impaired
speech. This is 252 hours (114 training, 90 testing, 21 validation, and 27
unseen). A high-level summary of the impairment distribution is given in Table
III. In each row 50% of the speech is NB and 50% is WB. A total of 321
distinct conditions are present in the ITS dataset.

TABLE III: Distribution of impairments in ITS dataset. Hours | Noise & | Speech Codecs | Packet Loss &
---|---|---|---
| Suppression |  | Concealment
$84$ | ✓ |  |
$84$ |  | ✓ |
$28$ | ✓ | ✓ |
$28$ |  | ✓ | ✓
$28$ | ✓ | ✓ | ✓

Each impaired segment in the dataset was then labeled with values from seven
established FR estimators: WB-PESQ [8], POLQA [10], ViSQOL (compliance version
c310) [12], and PEMO (software available via [67]) estimate the quality of the
impaired segment, while STOI [54], ESTOI [9], and SIIBGauss [68] give
estimates of the speech intelligibility. Each of these seven FR tools compared
every impaired segment with the corresponding WB reference segment. We then
trained WAWEnets to estimate these FR values using only the impaired segments.

###  III-B Training

To prepare each segment for the training process, we converted the speech data
to float and normalized it to the range [-1, 1]. We used affine
transformations to map values from PESQ ([1.02, 4.64]), POLQA ([1, 4.75]),
PEMO ([0, 1]), ViSQOL ([1, 5]), STOI ([0.45, 1]), ESTOI ([0.23, 1]), and
SIIBGauss ([0, 750]) to [-1, 1] before use as targets. As in [29], we
performed inverse phase augmentation (IPA) to augment all datasets in order to
train WAWEnet to learn invariance to waveform phase inversion. This
augmentation increased the amount of data available to just over 500 hours of
total speech data.

When training our model from scratch on the ITS dataset we used one set of
initial weights for each training process. This set of weights was initialized
using the Kaiming-Normal initialization method [69]. In the cases where
$N_{T}>1$, the weights in the last layer were duplicated $N_{T}$ times,
resulting in a shape of $N_{T}\times 96$.

We seeded all random number generators such that batch order and batch
contents were consistent for every training run. WAWEnets were trained using
mini-batches that were as large as GPU memory would allow; in this case, 60
segments per batch. We used the Adam optimizer [70] with $10^{-4}$ learning
rate, and $L_{2}$ regularization parameter set to $10^{-5}$. When the network
had trained for an entire epoch we evaluated the validation set and logged the
epoch RMSE (root mean-squared error) loss $E_{l}$ and per-segment correlation
between the target and the WAWEnet output, $\rho_{seg}$. In the case that
$E_{l}$ on the validation set had not decreased by at least $10^{-4}$ for 5
epochs, we multiplied the learning rate by $10^{-1}$. The network was trained
for 30 epochs. Training $\text{WAWEnet}_{\text{O}1}$ on one NVIDIA GTX 1070
took about 14 hours.111Certain products are mentioned in this paper to
describe the experiment design. The mention of such entities should not be
construed as any endorsement, approval, recommendation, prediction of success,
or that they are in any way superior to or more noteworthy than similar
entities not mentioned.

WAWEnets are NR tools, but for completeness of the research effort, we also
created some FR versions. An NR WAWEnet processes 48,000 samples
($3\text{s}\times 16,000$ smp/s). The FR version
($\text{WAWEnet}_{1}^{\text{FR}}$) processes reference speech and the
corresponding impaired speech. Thus the first convolutional layer of
$\text{WAWEnet}_{1}^{\text{FR}}$ has more parameters (input size $2\
\text{channels}\times 48,000\ \text{samples}\times 96\ \text{filters}$)
compared to the NR WAWEnet (input size $1\ \text{channel}\times 48,000\
\text{samples}\times 96\ \text{filters}$). During initialization, all weights
in the $S_{1}$ were duplicated resulting in a shape of $2\times 48,000\times
96$. In order to make a fair comparison, we also created a dual input NR
version called $\text{WAWEnet}_{1}^{\text{2}}$.
$\text{WAWEnet}_{1}^{\text{2}}$ and $\text{WAWEnet}_{1}^{\text{FR}}$ have
identical architecture and number of parameters.
$\text{WAWEnet}_{1}^{\text{2}}$ processes two identical copies of the impaired
speech. Results for all three versions are presented and compared below.

###  III-C Individual network for each of seven FR targets

We trained individual WAWEnets ($\text{WAWEnet}_{\text{O}1}$) for each of the
seven FR targets. Table IV gives the resulting per-segment Pearson
correlations and Table V shows the corresponding per-segment RMS errors. To
allow for direct comparisons, these errors are normalized and shown as a
percentage of the full scale for each target.

The network outputs are highly correlated to the FR target values across a
vast amount of data that spans a wide range of impairment types, talkers and
languages, in spite of the fact that the networks have no access to the
reference speech for comparison purposes. In effect, the networks embody very
effective generalized models for speech quality (or intelligibility). These
generalized models are invariant to speech content, talker, and language and
this allows them to operate without comparison to a reference speech signal.

The “Dual-NR” column of Table IV shows that the additional weights in
$\text{WAWEnet}_{1}^{\text{2}}$’s $S_{1}$ have minimal effect on correlation.
And the “FR” column ($\text{WAWEnet}_{1}^{\text{FR}}$) shows that access to
reference speech (and necessarily increasing the network size) does produce
some additional benefit, as expected. Table V tells the same story in the RMSE
domain. The networks produce impressive estimation error values that range
from 5 to 9% of full scale. These values barely change when network size is
increased, but are further reduced when reference speech is provided.

TABLE IV: Pearson correlations between predictions from three WAWEnets with $N_{T}=1$ and seven objective FR targets, unseen portion of ITS dataset, individual network for each target, correlations calculated per-segment. Target | NR | Dual-NR | FR
---|---|---|---
| $\text{WAWEnet}_{\text{O}1}$ | $\text{WAWEnet}_{1}^{\text{2}}$ | $\text{WAWEnet}_{1}^{\text{FR}}$
WB-PESQ | 0.94 | 0.95 | 0.98
POLQA | 0.93 | 0.93 | 0.97
ViSQOL | 0.95 | 0.95 | 0.98
PEMO | 0.95 | 0.95 | 0.97
STOI | 0.91 | 0.90 | 0.98
ESTOI | 0.95 | 0.95 | 0.99
SIIBGauss | 0.96 | 0.97 | 0.99
TABLE V: Normalized RMSE between predictions from three WAWEnets with $N_{T}=1$ and seven objective FR targets, unseen portion of ITS dataset, individual network for each target, errors calculated per-segment, values are percent of full scale. Target | NR | Dual-NR | FR
---|---|---|---
| $\text{WAWEnet}_{\text{O}1}$ | $\text{WAWEnet}_{1}^{\text{2}}$ | $\text{WAWEnet}_{1}^{\text{FR}}$
WB-PESQ | 9.2 | 9.1 | 4.9
POLQA | 9.4 | 9.4 | 6.6
ViSQOL | 7.2 | 7.0 | 4.9
PEMO | 6.1 | 6.2 | 5.1
STOI | 5.1 | 5.2 | 2.4
ESTOI | 6.1 | 6.1 | 3.2
SIIBGauss | 4.9 | 4.6 | 2.2

###  III-D Single network for all seven FR targets

The outstanding success of individual WAWEnets for each target suggests that
the WAWEnet architecture has plenty of capacity for this class of problems.
This then suggests the possibility of a single network that maps speech
signals to points in a single latent space such that those points can then be
mapped to all four quality and all three intelligibility targets. Speech
quality indicates how pleasing a speech signal is to the ear and speech
intelligibility measures the information transferred by the speech. These are
certainly different quantities, but they are also related, and this bodes well
for the possibility of a single network.

We trained a single WAWEnet to produce four, five, six, and all seven target
values and found that this co-training is both possible and beneficial. Seven
networks are replaced with a single network of nearly identical size without
compromise in performance. In fact, co-training with seven targets appears to
regularize the problem and results in improved performance. Table VI provides
correlation and RMSE values for the single network called
$\text{WAWEnet}_{7}$. The single network correlation values are better than
those for the individual networks (see Table IV) in four cases and they are
matched in the other three cases. The single network RMSE values are better
than those for the individual networks (see Table V) in six cases and they are
matched in the seventh case. Fig. 2 contains two-dimensional histograms that
show the joint distribution of $\text{WAWEnet}_{7}$ per-segment estimates and
actual target values for four of the seven targets.

TABLE VI:  Pearson correlation, RMSE, and normalized RMSE between seven estimates from a single network ($\text{WAWEnet}_{7}$) and seven objective FR targets, unseen portion of ITS dataset, correlation and error calculated per-segment, error shown as actual values and percent of full scale. | Correlation | RMSE | RMSE (%)
---|---|---|---
WB-PESQ | 0.95 | 0.31 | 7.8
POLQA | 0.94 | 0.33 | 8.3
ViSQOL | 0.95 | 0.28 | 6.9
PEMO | 0.96 | 0.06 | 6.1
STOI | 0.92 | 0.05 | 4.7
ESTOI | 0.95 | 0.06 | 5.8
SIIBGauss | 0.96 | 35.1 | 4.7

(a) WB-PESQ (b) POLQA (c) ViSQOL (d) STOI

Figure 2: Two-dimensional histogram showing joint distribution of
$\text{WAWEnet}_{7}$ per-segment estimates and actual target values with per-
segment Pearson correlation and unnormalized RMSE for four of the seven
targets on the unseen portion of the ITS dataset. Number of segments per bin
is given by the scale at the right.

##  IV WAWEnet for Seven Objective FR Targets and Four Subjective Targets

Having successfully developed a single WAWEnet that emulates seven FR tools,
we asked if that network might be further trained to also emulate subjective
scores.

###  IV-A Data

We added to our collection the dataset described in [50] and generously
provided by the Quality and Usability Lab of the Technische Universität
Berlin. We designate this the TUB dataset. It contains a variety of speech
sources, simulated impairments (added background noise, selected codecs,
packet loss, bandpass filtering, and clipping) and live impairments
(background noise, landline-to-mobile calls, and VoIP calls) Additional
details are given in [50]. We successfully computed seven FR target values for
14,220 of the TUB speech files.222The “TEST_LIVETALK” database had no
reference files so FR targets could not be calculated. FR estimators
occasionally fail to produce valid results so this reduces the usable number
of files as well. Each of these files also has crowd-sourced subjective
ratings of overall speech quality, noisiness, coloration, discontinuity, and
loudness.

We used the subjective ratings “overall speech quality,” “noisiness,”
“coloration,” and “discontinuity” (labeled as MOS, NOI, COL, and DIS,
respectively) as targets for WAWEnet training. The “loudness” rating is not a
practical target for WAWEnets because WAWEnets use normalized input speech
levels. This removes variation in overall signal level which is a dominating
factor behind the “loudness” ratings. This normalization could be removed if
training for “loudness” is desired.

We divided the dataset in two ways. The first was according to the labeling
that was provided with it. We used the 10,903 files (77%) from “TRAIN_LIVE”
and “TRAIN_SIM” for training, the 642 (4%) files from “TEST_FOR,” “TEST_NSC,”
and “TEST_P501” for testing, and the 2,675 files (19%) from “VAL_LIVE” and
“VAL_SIM” for validation. Note that “TEST_NSC” contains German language speech
and the remainder of the dataset is English language speech. The second way we
divided this dataset was through random sampling: 50% (7,110 files) were used
for training, 40% (5,688 files) for testing, and 10% (1,422 files) for
validation. The results presented below are based on the testing portion in
both cases.

File lengths range from 4.5 s to 14.6 s with a mean of 8.8 s and a median of
9.0 s. WAWEnets work on three-second segments where the SAF is at least 50
percent. For each file we find all such distinct segments in the file—98% of
the files produce two segments, 48% produce three and 10% produce four. The
result is approximately 28,200 training segments, 6,800 validation segments,
and 1400 testing segments—a total of 30 hours of speech. Using G.191 tools
[66], we converted the data from 48,000 smp/s to 16,000 smp/s and normalized
each segment to $26\pm 0.2$ dB below the overload point.

Each subjective rating is based on an entire file. For training we replicate
that file rating to create an identical target for each segment extracted from
that file. If the file shows little temporal variation in the rated attribute,
then this target replication incurs only a minor approximation. But if there
is major variation (e.g. localized packet loss or non-stationary background
noise) then replication of targets can be a significant approximation and
source of error. For testing, the correlations and RMSE values compare each
per-segment WAWEnet output with the subjective rating of the corresponding
file.

Signal bandwidths create an additional approximation in this work. The
subjective ratings in the TUB dataset are based on FB speech signals but
WAWEnets are WB and only analyze the lower 8 kHz of these signals.

(a) MOS (b) DIS (c) WB-PESQ (d) STOI

Figure 3: Two-dimensional histogram showing joint distribution of
$\text{WAWEnet}_{11}$ per-segment estimates and actual target values with per-
segment Pearson correlation and unnormalized RMSE for four of the eleven
targets on the test portion of the TUB dataset when using the 50/40/10 split.
Number of segments per bin is given by the scale at the right.

###  IV-B Training and Results

Starting with the weights from $\text{WAWEnet}_{7}$ we allowed the optimizer
to update the weights in each section. This strategy improved overall
performance compared to the random initialization strategy used in Section
III-B. We used affine transformations to map all subjective scores from [1, 5]
to [-1, 1]. Besides those exceptions, we followed the training process
described in Section III-B.

We trained a WAWEnet to emulate the four subjective ratings:
$\text{WAWEnet}_{\text{S}4}$. The per-segment correlation and normalized RMSE
values are shown in Table VII. We also trained WAWEnets to emulate the four
subjective ratings and three, four, five, six, or all seven of the FR values.
Table VII also shows the result for a single WAWEnet that emulates four
subjective and seven objective FR targets: $\text{WAWEnet}_{11}$. Fig. 3
contains two-dimensional histograms that show the joint distribution of
$\text{WAWEnet}_{11}$ per-segment estimates and actual target values for four
of the eleven targets using the 50/40/10 split.

Table VII makes clear that co-training with the objective FR targets improves
correlations and RMS errors for the four subjective targets. It appears that
the extra constraints regularize the problem and lead to a better solution.
Considering the test data prescribed by the TUB dataset, Table VII shows
dramatic correlation drops and RMS error increases for the FR targets. But the
correlation drop is smaller when evaluated on the random split. Table VI and
Table VII are based on different and dissimilar datasets so the comparison is
not exact.

Estimators are often judged by per-condition statistics. Target values are
averaged for all results from each condition (e.g., codec mode or noise
environment) and the same is done for the estimates. These per-condition
averages are then compared by correlation or error statistics. This averaging
reflects a common and relevant estimation situation—it removes variation due
to talkers, utterances, and other factors so we can draw clear conclusions
about the systems we are testing. Table VIII provides per-condition results
analogous to those in Table VII. Here we see that $\text{WAWEnet}_{11}$ gives
per-condition correlations of 0.85 or better for four subjective and three
objective FR targets.

TABLE VII: Per-segment Pearson correlation ($\rho_{seg}$) and normalized RMSE between single WAWEnet trained for 4 ($\text{WAWEnet}_{\text{S}4}$) or 11 ($\text{WAWEnet}_{11}$) targets.Testing portion of TUB dataset. Error values are percent of full scale. | 77/4/19 Split | 50/40/10 Split
---|---|---
| $\rho_{seg}$ | RMSE (%) | $\rho_{seg}$ | RMSE (%)
$N_{T}$ | 4 | 11 | 4 | 11 | 4 | 11 | 4 | 11
MOS | 0.80 | 0.82 | 16 | 15 | 0.84 | 0.85 | 15 | 14
NOI | 0.80 | 0.82 | 14 | 13 | 0.80 | 0.82 | 14 | 14
COL | 0.79 | 0.81 | 13 | 12 | 0.78 | 0.80 | 15 | 14
DIS | 0.75 | 0.78 | 17 | 16 | 0.71 | 0.73 | 17 | 16
WB-PESQ | - | 0.79 | - | 19 | - | 0.89 | - | 15
POLQA | - | 0.80 | - | 17 | - | 0.88 | - | 14
ViSQOL3 | - | 0.83 | - | 14 | - | 0.91 | - | 12
PEMO | - | 0.58 | - | 17 | - | 0.85 | - | 8
STOI | - | 0.52 | - | 16 | - | 0.85 | - | 8
ESTOI | - | 0.58 | - | 22 | - | 0.88 | - | 10
SIIBGauss | - | 0.75 | - | 16 | - | 0.82 | - | 19
TABLE VIII: Per-condition Pearson correlation ($\rho$) and normalized RMSE between single WAWEnet trained for 4 ($\text{WAWEnet}_{\text{S}4}$) or 11 ($\text{WAWEnet}_{11}$) targets. Testing portion of TUB dataset. Error values are percent of full scale. | 77/4/19 Split | 50/40/10 Split
---|---|---
| $\rho$ | RMSE (%) | $\rho$ | RMSE (%)
$N_{T}$ | 4 | 11 | 4 | 11 | 4 | 11 | 4 | 11
MOS | 0.91 | 0.91 | 12 | 12 | 0.96 | 0.96 | 9 | 9
NOI | 0.91 | 0.91 | 10 | 9 | 0.97 | 0.97 | 9 | 9
COL | 0.91 | 0.91 | 9 | 9 | 0.97 | 0.97 | 7 | 8
DIS | 0.90 | 0.90 | 12 | 12 | 0.97 | 0.96 | 10 | 10
WB-PESQ | - | 0.85 | - | 15 | - | 0.94 | - | 10
POLQA | - | 0.91 | - | 10 | - | 0.96 | - | 10
ViSQOL3 | - | 0.89 | - | 12 | - | 0.97 | - | 8
PEMO | - | 0.66 | - | 18 | - | 0.93 | - | 13
STOI | - | 0.66 | - | 19 | - | 0.95 | - | 11
ESTOI | - | 0.60 | - | 25 | - | 0.91 | - | 15
SIIBGauss | - | 0.80 | - | 13 | - | 0.88 | - | 10

##  V WAWEnet for Subjective Scores Only

We also trained a WAWEnet to closely emulate just subjective speech quality
scores.

###  V-A Data

We are very grateful that the Audio, Speech, and Information Retrieval Group
at Indiana University Bloomington provided us with the dataset described in
[71]. We call this the IUB dataset and it is better suited to WAWEnets than
the TUB dataset is. The IUB dataset is WB, so WAWEnets need not approximate FB
subjective scores from WB signals as was the case with the TUB dataset. In
addition, the IUB file lengths range from 2.0 to 7.8 s with a mean of 3.8 s
and a median of 3.7 s thus providing a much closer match to the WAWEnet three-
second window than was possible with the TUB data.

The IUB dataset includes high-quality speech from close-talking microphones
and lower quality speech from more distant microphones. The distant
microphones necessarily capture more natural and artificial environmental
noise (SNRs reported in the range -10 to +11 dB) and natural reverberation
(speech-to-reverberation ratios reported in the range -5 to +4 dB)[71]. In
addition, some recordings were subjected to 3.4 kHz low-pass filtering in
order to create anchor conditions for subjective testing. Subjective testing
was crowd-sourced. Scores were collected, filtered, and normalized [71] to
produce speech quality scores on a scale of zero to ten. The result is 36,000
speech files, each with a scaled subjective speech quality MOS value.

The IUB dataset contains 35,428 files that are six seconds or shorter. From
each of these files we selected all disjoint three-second segments with SAF at
or above 50%. Any file shorter than three seconds was zero-padded to create a
single three-second segment. By this process 27,004 files produced two
segments each and the remaining 8,424 files produced one segment each. This
gives over 62,000 segments or 52 hours of speech data. Each segment was
assigned the scaled MOS value of the file that it came from. Using only files
six seconds or shorter means that a three-second segment contains at least
half of the file. This minimizes error associated with assigning per-file MOS
values to individual segments when speech quality is not constant throughout
the file. Using G.191 tools [66], we normalized each file to $26\pm 0.2$ dB
below the overload point.

###  V-B Training and Results

We used affine transformations to map all subjective scores from [0, 10] to
[-1, 1]. Starting with the weights from $\text{WAWEnet}_{7}$ we allowed the
optimizer to update the weights in each section. However, with this dataset
equivalent results are achieved using the initialization method described in
III-B. We randomly selected 50% of the segments for training, 10% for
validation and 40% for testing. The rest of the training process was similar
to the process described in Section III-B except we allowed training for 60
epochs in this case. The results that follow are based on the approximately
25,000 testing segments, which comprise about 21 hours of speech.

$\text{WAWEnet}_{\text{S}1}$ achieves a per-segment correlation to MOS of 0.96
and normalized RMSE of 6 % of full scale. Fig. 4 shows this correlation
graphically. Six other recently proposed NR tools are applied to the IUB
dataset in [42] and the resulting Pearson correlations to MOS range from 0.93
to 0.96. The $\text{WAWEnet}_{\text{S}1}$ MOS correlation of 0.965 matches the
best of these. Mean absolute errors (MAEs) given in [42] range from 0.40 to
0.50 and the $\text{WAWEnet}_{\text{S}1}$ MAE is 0.41, which places
$\text{WAWEnet}_{\text{S}1}$ near the best previous result. Note that this is
not a complete comparison because the best tools in [42] produce MOS and three
additional estimates, while $\text{WAWEnet}_{\text{S}1}$ produces just MOS.
But there is no question that this WAWEnet architecture performs on par with
these other top-performers on this dataset.

Figure 4: Two-dimensional histogram showing joint distribution of
$\text{WAWEnet}_{\text{S}1}$ per-segment estimates and actual target values
with per-segment Pearson correlation and unnormalized RMSE for scaled MOS on
the test portion of the IUB dataset. Number of segments per bin is given by
the scale at the right.

##  VI Discussion and Extensions

We have shown that the NR WAWEnet architecture can evaluate wideband speech
signals in a manner consistent with a variety of objective and subject
evaluations without using any reference speech signal. $\text{WAWEnet}_{7}$
can simultaneously very closely emulate seven diverse objective targets.
$\text{WAWEnet}_{\text{S}1}$ closely emulates MOS values. And simultaneous
emulation of seven objective targets and four subjective targets with
$\text{WAWEnet}_{11}$ appears to improve performance on subjective targets
compared to emulating solely subjective targets.

The weights present in $\text{WAWEnet}_{7}$ appear to be relevant to both the
TUB and the IUB datasets. Training on the TUB dataset with the 50/40/10 split
with a random initialization and 11 targets resulted in MOS, NOI, COL, and DIS
per-segment correlations of 0.80, 0.80, 0.75, and 0.64 respectively—lower than
the correlations achieved when starting from $\text{WAWEnet}_{7}$. When
training on the IUB dataset, the weights present in $\text{WAWEnet}_{7}$ don’t
improve estimation performance, but the training process converged to the best
performance roughly 5 epochs sooner. With both the TUB and the IUB datasets,
solely allowing the weights in $\text{WAWEnet}_{7}$ $S_{14}$ to be trained
resulted in worse performance than allowing the weights in all sections to be
trained.

It is possible to extend the WAWEnet architecture to wider bandwidths (higher
sample rates) by adding one or more sections to the input of the network. For
example, adding a section with a $k=3$ would allow WAWEnets to process 3
seconds of speech with a sample rate 48 kHz.

Likewise, there are several strategies suitable for extending WAWEnets to
process longer signals. One strategy would be to insert a section with a $k=2$
between $S_{13}$ and $S_{14}$. This would allow WAWEnets to generate estimates
for six-second signals. Another strategy would be to use either the 96-D
feature vector or target estimates as an input to a recurrent neural network
of some kind, e.g. long short-term memory (LSTM), or gated recurrent unit
(GRU), thus allowing WAWEnets to process arbitrary-length speech. With
sufficient data either strategy might learn to properly account for the
various principles at work when speech quality varies. These principles
include (see [72]): long-term ratings are lower bounded by minima and upper-
bounded by averages, larger and more frequent quality variations reduce
quality, and recency. Alternately, longer signals can be accommodated by a
sliding three-second WAWEnet processing window followed by averaging or more
sophisticated processing of the multiple results produced.

##  VII Signal Processing Interpretation

We have established that the WAWEnet architecture offers an efficient and
effective tool for evaluating wideband speech waveforms. It does this by
mapping a wideband speech waveform to a 96-dimensional vector in a latent
space. Different projections in that space produce scalar values that can
track different objective or subjective values related to speech quality or
intelligibility. We also seek to understand, to the extent possible, _how_ the
WAWEnet architecture maps waveforms to this space. As is often the case with
algorithms developed through machine learning, a fully-satisfying
interpretation is elusive. But we can describe the signal processing that a
WAWEnet applies and give a high-level description of how this signal
processing converts a waveform into an evaluation of that waveform.

###  VII-A Functions

To map waveforms to the latent space a WAWEnet uses 13 sections and each
section consists of four layers. In the language of ML, these four layers are
convolution, batch normalization, ReLU, and average pooling. Table IX shows
how the four layers map to six signal-processing (SP) functions. Linear time-
invariant (LTI) systems are often relatively amenable to analysis. Table IX
shows that WAWEnets are linear except for the bias and half-wave rectification
(HWR). Of course these non-linear functions are what enable WAWEnets (and many
other ML based algorithms) to accomplish the assigned tasks and also prevent
the network from simplifying into a trivial and ineffective one. Time-
invariance is satisfied by all functions except the sub-sampling (which shows
time-invariance only for time shifts that are integral multiples of the output
sampling period).

TABLE IX:  WAWEnet ML layers expanded into signal processing functions ML Layer |  SP Function |  Equation |  Linear |  Time-Invar
---|---|---|---|---
Conv. |  FIR Filtering |  $y_{k}=\sum\limits_{i=0}^{2}h_{i}x_{k-i}$ |  ✓ |  ✓
Batch Norm. |  Gain |  $y_{k}=ax_{k}$ |  ✓ |  ✓
Bias |  $y_{k}=x_{k}+b$ |  |  ✓
ReLU |  Half-Wave Rect. |  $y_{k}=\text{max}(x_{k},0)$ |  |  ✓
Avg. Pool |  FIR Filtering |  $y_{k}=\frac{1}{m}\sum\limits_{i=0}^{m-1}x_{k-i}$ |  ✓ |  ✓
Sub-sampling |  $y_{k}=x_{mk}$ |  ✓ |

$S_{1}$ has one input channel and 96 output channels. $S_{1}$ begins by
splitting the input audio signal into 96 identical copies which then feed into
the 96 channels. These 96 channels are processed in parallel and independent
of each other. Each channel starts with FIR (finite impulse response)
filtering, followed by the application of gain and bias, then HWR, and finally
a low-pass FIR filter and sub-sampling with a factor of four.

$S_{2}$ through $S_{13}$ have 96 input channels and 96 output channels. The
filtering layer of these sections can be described as a full matrix of
filtering. That is, $96^{2}=9216$ filters are used to produce 96 filtered
versions of each input channel. Then each of the 96 output channels is formed
by summing one filtered version of each input channel (96 signals in each
sum). The remaining layers of $S_{2}$ through $S_{13}$ are the same as those
in $S_{1}$, although the sub-sampling may use a factor of two, three, or four.
In each of these layers the 96 channels are processed in parallel and
independent of each other. $S_{6}$ and $S_{9}$ start with zero padding
(appending one zero at the end of the signal) to allow sub-sampling by a
factor of two at the end of the section.

###  VII-B Per-Function Operation

A time-domain description of the operations says that WAWEnets replace samples
with weighted local averages, add up signals from 96 different channels, scale
and shift all samples in a channel uniformly, replace negative samples with
zero, and replace blocks of samples with their average value. A frequency-
domain description provides more insight, so we next describe how these
functions change the spectra of the signals as they move through the network.
We describe separately changes to the DC component of the signal spectra and
changes to the other components. This distinction is key to the description of
the overall operation that follows.

The second-order FIR filters have three unconstrained real coefficients,
resulting in either a pair of complex-conjugate zeros, or two zeros on the
real line. The result is gentle spectral peaks and gentle or deep spectral
nulls, depending on the location of the zeros relative to the unit circle.
Forty percent of these filters are lowpass, 29% are highpass, 19% bandstop,
and 12% bandpass. These proportions change by less than 1% between
$\text{WAWEnet}_{7}$, $\text{WAWEnet}_{11}$, and $\text{WAWEnet}_{\text{S}1}$.
This commonality is consistent with the fact that both $\text{WAWEnet}_{11}$
and $\text{WAWEnet}_{\text{S}1}$ use the weights from $\text{WAWEnet}_{7}$ as
their initial state during the training process.

The gain function simply scales all values of the spectrum (DC and non-DC) by
a single value. The bias function adjusts only the DC value of the spectrum
and leaves the rest unchanged. The effect of HWR is controlled by the bias. If
the bias forces all samples to be negative, HWR removes the signal. If the
bias forces all samples to be positive, HWR does nothing. If the bias is such
that the signal is bipolar, the most common and prominent spectral effect is
the creation of new spectral components, thus increasing the spectral density.
An example is given in the middle panel of Fig. 5. The effect of HWR on the DC
value of the signal depends strongly on the signal.

Figure 5: Top panel shows spectra of two tones (345 Hz, shown in light blue,
and 6789 Hz, shown in gold), middle panel shows the result of half-wave
rectification with zero bias, bottom panel shows the additional effect of
average pooling (pooling factor is 2). Colors emphasize non-linearity: light
blue shows spectral components that are produced by the 345 Hz tone alone,
gold shows components produced by the 6789 Hz tone alone, blue shows
components that only appear when both tones are present. A significant DC
component will be produced in any of these three cases and it is shown in
black.

The FIR filtering that precedes the subsampling is length $m$ with all
coefficients equal to $m^{-1}$ ($m=$ 2, 3, or 4). These are low-pass filters
and each one has a perfect null at any frequency that would alias to DC so the
subsequent subsampling cannot change the DC value of the spectrum. (This is
consistent with the fact that averaging cannot change the DC value of a
signal.) The subsampling will remove all spectral content above the new
Nyquist frequency and can produce aliasing at any non-DC frequency below the
new Nyquist frequency. The aliasing is significant because these FIR low-pass
filters are very short and have responses that are far from the near brick-
wall responses needed to achieve alias-free sub-sampling. For example, when
$m=$2 spectral components just above the new Nyquist frequency are aliased to
those just below the new Nyquist frequency with only 3 dB of attenuation.
Aliasing involves addition of complex values, so aliasing may reduce or
increase the original spectral magnitudes depending on the relative phases of
the two addends. The bottom panel of Fig. 5 shows an example.

Note that in conventional sample rate reduction, a filter calculates one
output sample for each input sample, then subsampling retains every $m^{th}$
sample. The avgpool function integrates filtering and subsampling so that only
every $m^{th}$ sample is calculated. Because the filtering is FIR and the
filter length matches the downsampling factor, this unconventional approach
produces the same results as the conventional approach.

###  VII-C Overall Operation

We can view the end-to-end mapping from audio signals to the 96-D latent space
as 96 individual (but coupled) signal processors. The job of these processors
is to shorten the signals and to strategically shape and move relevant
spectral information to DC. In $S_{13}$ the length-3 filtering and 3-to-1 sub-
sampling (the avgpool layer) serve to extract the DC value of a signal that
has 3 samples. The ensemble of the 96 DC values defines a vector in the latent
space which is then mapped to a final output value by $S_{14}$.

The particulars of shaping spectral information and moving it to DC are shown
graphically for one section in Fig. 6. In each section non-DC spectral values
of the signals are modified by FIR filtering, gain, HWR, the pooling FIR
filter, and sub-sampling. The DC spectral values of the signals are modified
by just four of the six functions. In the FIR filtering, gain, and bias
functions the modification of the DC value is determined solely by the
processor. These modifications are independent of the signal itself. But in
the HWR function the modification of the DC spectral component is driven by
the non-DC spectral components. The HWR is the stage where relevant non-DC
spectral information is strategically moved to DC.

Figure 6: In each section four functions change DC component and five change
non-DC components. Non-DC components affect the DC component _only_ in the
half-wave rectification function. Sections sequentially move non-DC
information to DC.

For example, consider a speech signal and noise signal each with a DC value of
zero. When added, the noisy speech signal still has a DC value of zero. But
after HWR, the original and noisy speech signal can have very different DC
values and this DC value can thus serve to indicate that noise was present in
the speech signal. This is a very simple example but by using many sections of
intricate spectral shaping and folding, this processor can also accurately
assess a broad array of much more nuanced perturbations to speech signals.

Fig. 6 emphasizes the fact that every function modifies either the DC
component or the non-DC component, or both, and that the HWR is the sole
function where non-DC information influences the DC value. The input audio
signal will typically have a DC value near zero, but as the processing
continues, the spectral shaping and the flow of information to DC at HWRs
results in DC values that describe important characteristics of the original
audio signal.

Fig. 7 gives a visual example of DC values in the first 13 sections of
$\text{WAWEnet}_{\text{S}1}$ (1 input signal plus the outputs of 6 SP
functions $\times$ 13 sections gives 79 rows) for each of the 96 channels.
Input speech is shown at the top of the figure and DC values there are all
zero as expected. As signals move through the network (down the figure) DC
values build in various channels for part or all of a section. No continuous
downward “flow” of DC appears because channels are fully connected with each
other in every section and channel numbers are arbitrary. The most dramatic
distribution of DC values is in the second half of the network (lower half of
the figure), and this distribution then moderates to form the output.

Figure 7: Example DC values (dark purple for large negative, bright yellow for
large positive) in the first 13 sections (shown as labeled macro-rows) and all
96 channels (shown as columns) of $\text{WAWEnet}_{\text{S}1}$.The six rows
within each section show the DC output value of six signal processing
functions (FIR, gain, bias, HWR, FIR, sub-sampling, in that order) in that
section. The DC value of the input signal is replicated 96 times above
$S_{1}$.

In effect, the wideband input speech signal passes through 96 parallel coupled
signal processors (composed of $S_{1}$–$S_{13}$) and the 96 DC values of the
96 output signals form a vector that is then mapped by $S_{14}$ to an estimate
of some quality of that speech signal. Fig. 8 shows examples of these 96-D
$\text{WAWEnet}_{7}$ outputs for 30 different conditions (1,320 segments
averaged for each condition). These examples show how different dimensions
respond to different attributes while they work together to produce an
estimate of speech quality or intelligibility.

Figure 8: 96 outputs of $S_{13}$ for 30 conditions. From left to right, six
increasing bit-rates for each of four codecs, then three noise suppression
thresholds for each of two SNRs. Example observations: channel 32 responds
strongly to WB vs NB and also to noise suppression thresholds, channel 17
responds to codec bit-rates and to noise levels.

##  VIII Conclusion

WAWEnets are no-reference wideband audio waveform evaluation networks that
process waveforms directly into evaluations of those waveforms. We have
trained and evaluated multiple WAWEnets and this work is based on 334 hours of
speech in 13 different languages produced by well over 1000 different talkers.
We have demonstrated that a $\text{WAWEnet}_{7}$ can produce speech quality
and intelligibility estimates that agree (correlations of 0.92 to 0.96) with
values from seven established FR objective estimators. $\text{WAWEnet}_{11}$
demonstrates that this architecture is useful for estimating both objective
and subjective speech qualities at the same time. $\text{WAWEnet}_{\text{S}1}$
agrees with the subjective MOS values of the IUB dataset with per-segment
correlation 0.965, commensurate with top results from other current
approaches. We expect that WAWEnets will generate accurate predictions of
speech qualities for applications where speech impairments resemble those
represented in our wide-ranging training dataset. In addition, we expect they
are useful for narrower applications (for example, measuring reverb present in
a room) if trained to do so. WAWEnets’ small size make them approachable for
applications where computing power is at a premium.

WAWEnets are distinct from the vast majority of alternatives because they
apply ML directly to speech waveforms instead of applying it to features
extracted from speech waveforms. This gives WAWEnets access to all the
information in the waveforms and allows them to effectively generate the best
features for the task internally, rather than having potentially sub-optimal
features mandated externally. Our work shows that this is indeed a viable
approach.

We have proposed several strategies to extend WAWEnets to wider bandwidths and
longer signals as data becomes available.

WAWEnets are composed of six common signal processing functions and, as
expected, the non-linear functions (bias and HWR) are critical. These
functions move spectral information (shaped by FIR filtering and gains) to DC
so that the DC values of the final, very short, signals provide a 96-D
description of the original, much longer signals. We encourage further
experimentation and development with WAWEnets and that is enabled by the
software repository at https://github.com/NTIA/WEnets.

## Acknowledgments

Sincere thanks to Gabriel Mittag, Sebastian Möller, and the members of the
Quality and Usability Lab of the Technische Universität Berlin for creating
and sharing the TUB dataset; to Zhuohuang Zhang, Donald Williamson, and the
members of the Audio, Speech, and Information Retrieval Group at Indiana
University for creating and providing the IUB dataset; and to Michael Chinen,
Andrew Hines, and Jan Skoglund for support with ViSQOL 3.0.

## References

  * [1] S. Quackenbush, T. Barnwell, and M. Clements, _Objective Measures of Speech Quality_. Englewood Cliffs, New Jersey: Prentice Hall, 1988.
  * [2] S. Wang, A. Sekey, and A. Gersho, “An objective measure for predicting subjective quality of speech coders,” _IEEE Journal on Selected Areas in Communications_ , vol. 10, no. 5, pp. 819–829, June 1992.
  * [3] J. G. Beerends and J. A. Stemerdink, “A perceptual speech-quality measure based on a psychoacoustic sound representation,” _J. Audio Eng. Soc_ , vol. 42, no. 3, pp. 115–123, 1994. [Online]. Available: http://www.aes.org/e-lib/browse.cfm?elib=6957
  * [4] S. Voran, “Objective estimation of perceived speech quality — Part I: Development of the measuring normalizing block technique,” _IEEE Transactions on Speech and Audio Processing_ , vol. 7, no. 4, pp. 371–382, July 1999.
  * [5] “Results of objective speech quality assessment including receiving terminals using the advanced TOSQA2001,” ITU-T, SG12, Q9, Contribution 20, Jan. 9, 2001\.
  * [6] J. Beerends, A. Hekstra, A. Rix, and M. Hollier, “Perceptual evaluation of speech quality (PESQ) — The new ITU standard for end-to-end speech quality assessment, Part II — Psychoacoustic model,” _J. Audio Engineering Society_ , vol. 50, pp. 765–778, Oct. 2002.
  * [7] R. Huber and B. Kollmeier, “PEMO-Q—A new method for objective audio quality assessment using a model of auditory perception,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 14, no. 6, pp. 1902–1911, Nov. 2006.
  * [8] _ITU-T Recommendation P.862.2, Wideband Extension to Recommendation P.862 for the Assessment of Wideband Telephone Networks and Speech Codecs_ , Geneva, 2007\.
  * [9] J. Jensen and C. H. Taal, “An algorithm for predicting the intelligibility of speech masked by modulated noise maskers,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 24, no. 11, pp. 2009–2022, 2016\.
  * [10] _ITU-T Recommendation P.863, Perceptual objective listening quality prediction_ , Geneva, 2018.
  * [11] S. Van Kuyk, W. B. Kleijn, and R. C. Hendriks, “An instrumental intelligibility metric based on information theory,” _IEEE Signal Processing Letters_ , vol. 25, no. 1, pp. 115–119, 2018.
  * [12] M. Chinen, F. S. C. Lim, J. Skoglund, N. Gureev, F. O’Gorman, and A. Hines, “Visqol v3: An open source production ready objective speech and audio metric,” in _Proc. Twelfth International Conference on Quality of Multimedia Experience_ , 2020.
  * [13] W. A. Jassim, J. Skoglund, M. Chinen, and A. Hines, “Warp-Q: Quality prediction for generative neural speech codecs,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2021, pp. 401–405.
  * [14] J. Liang and R. Kubichek, “Output-based objective speech quality,” in _Proc. IEEE Vehicular Technology Conference_ , vol. 3, June 1994, pp. 1719–1723.
  * [15] L. Malfait, J. Berger, and M. Kastner, “P.563—The ITU-T standard for single-ended speech quality assessment,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 14, no. 6, pp. 1924–1934, Nov. 2006\.
  * [16] D. Kim and A. Tarraf, “Anique+: A new American national standard for non-intrusive estimation of narrowband speech quality,” _Bell Labs Technical Journal_ , vol. 12, no. 1, pp. 221–236, Spring 2007.
  * [17] T. H. Falk and W. Chan, “Single-ended speech quality measurement using machine learning methods,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 14, no. 6, pp. 1935–1947, Nov. 2006.
  * [18] M. H. Soni and H. A. Patil, “Novel deep autoencoder features for non-intrusive speech quality assessment,” in _Proc. European Signal Processing Conference_ , Nov. 2016, pp. 2315–2319.
  * [19] M. Hakami and W. B. Kleijn, “Machine learning based non-intrusive quality estimation with an augmented feature set,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2017, pp. 5105–5109.
  * [20] C. Spille, S. D. Ewert, B. Kollmeier, and B. T. Meyer, “Predicting speech intelligibility with deep neural networks,” _Computer Speech & Language_, vol. 48, pp. 51 – 66, 2018. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0885230817300803
  * [21] P. Seetharaman, G. J. Mysore, P. Smaragdis, and B. Pardo, “Blind estimation of the speech transmission index for speech quality prediction,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , April 2018, pp. 591–595.
  * [22] S. Fu, Y. Tsao, H. Hwang, and H. Wang, “Quality-Net: An end-to-end non-intrusive speech quality assessment model based on BLSTM,” in _Proc. Interspeech_ , 2018, pp. 1873–1877. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2018-1802
  * [23] R. Huber, M. Krüger, and B. T. Meyer, “Single-ended prediction of listening effort using deep neural networks,” _Hearing Research_ , vol. 359, pp. 40 – 49, 2018. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0378595517304951
  * [24] H. Salehi, D. Suelzle, P. Folkeard, and V. Parsa, “Learning-based reference-free speech quality measures for hearing aid applications,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 12, pp. 2277–2288, Dec 2018.
  * [25] G. Mittag and S. Möller, “Non-intrusive speech quality assessment for super-wideband speech communication networks,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , May 2019, pp. 7125–7129.
  * [26] J. Ooster and B. T. Meyer, “Improving deep models of speech quality prediction through voice activity detection and entropy-based measures,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , May 2019, pp. 636–640.
  * [27] J. F. Santos and T. H. Falk, “Towards the development of a non-intrusive objective quality measure for DNN-enhanced speech,” in _Proc. Eleventh International Conference on Quality of Multimedia Experience_ , 2019.
  * [28] C.-C. Lo, S.-W. Fu, W.-C. Huang, X. Wang, J. Yamagishi, Y. Tsao, and H.-M. Wang, “MOSNet: Deep learning-based objective assessment for voice conversion,” in _Proc. Interspeech_ , Sep. 2019.
  * [29] A. A. Catellier and S. D. Voran, “WEnets: A convolutional framework for evaluating audio waveforms,” _arXiv e-prints 1909.09024_ , Sep. 2019.
  * [30] H. Gamper, C. K. A. Reddy, R. Cutler, I. J. Tashev, and J. Gehrke, “Intrusive and non-intrusive perceptual speech quality assessment using a convolutional neural network,” in _Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics_ , Oct. 2019, pp. 85–89.
  * [31] A. A. Catellier and S. D. Voran, “WAWEnets: A no-reference convolutional waveform-based approach to estimating narrowband and wideband speech quality,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020, pp. 331–335.
  * [32] M. B. Pedersen, A. Heidemann Andersen, S. H. Jensen, and J. Jensen, “A neural network for monaural intrusive speech intelligibility prediction,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020, pp. 336–340.
  * [33] N. Simou, Y. Mastorakis, and N. Stefanakis, “Towards blind quality assessment of concert audio recordings using deep neural networks,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020, pp. 3477–3481.
  * [34] Y. Choi, Y. Jung, and H. Kim, “Deep MOS predictor for synthetic speech using cluster-based modeling,” in _Proc. Interspeech_ , Oct. 2020.
  * [35] G. Mittag, R. Cutler, Y. Hosseinkashi, M. Revow, S. Srinivasan, N. Chande, and R. Aichner, “DNN no-reference PSTN speech quality prediction,” in _Proc. Interspeech_ , Oct. 2020.
  * [36] G. Mittag and S. Möller, “Deep learning based assessment of synthetic speech naturalness,” in _Proc. Interspeech_ , Oct. 2020.
  * [37] Y. Saishu, A. H. Poorjam, and M. G. Christensen, “A CNN-based approach to identification of degradations in speech signals,” _EURASIP Journal on Audio, Speech, and Music Processing_ , Feb. 2021.
  * [38] M. Liu, J. Wang, W. Yi, and F. Liu, “Neural network-based non-intrusive speech quality assessment using attention pooling function,” _EURASIP Journal on Audio, Speech, and Music Processing_ , May 2021.
  * [39] H. Nylén, S. Chatterjee, and S. Ternström, “Detecting signal corruptions in voice recordings for speech therapy,” in _Proc. International Conference on Acoustics, Speech and Signal Processing_ , 2021, pp. 386–390.
  * [40] J. Serrà, J. Pons, and S. Pascual, “SESQA: Semi-supervised learning for speech quality assessment,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2021, pp. 381–385.
  * [41] Y. Leng, X. Tan, S. Zhao, F. Soong, X.-Y. Li, and T. Qin, “MBNET: MOS prediction for synthesized speech with mean-bias network,” in _Proc. 2021 IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2021, pp. 391–395.
  * [42] Z. Zhang, P. Vyas, X. Dong, and D. S. Williamson, “An end-to-end non-intrusive model for subjective and objective real-world speech assessment using a multi-task framework,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2021, pp. 316–320.
  * [43] X. Zheng and C. Zhang, “Towards blind audio quality assessment using a convolutional-recurrent neural network,” in _Proc. Thirteenth International Conference on Quality of Multimedia Experience_ , 2021, pp. 91–96.
  * [44] S. Voran, “Measuring speech quality of system input while observing only system output,” in _Proc. Thirteenth International Conference on Quality of Multimedia Experience_ , 2021, pp. 125–128.
  * [45] G. Mittag, S. Zadtootaghaj, T. Michael, B. Naderi, and S. Möller, “Bias-aware loss for training image and speech quality prediction models from multiple datasets,” in _Proc. Thirteenth International Conference on Quality of Multimedia Experience_ , 2021, pp. 97–102.
  * [46] A. Ragano, E. Benetos, and A. Hines, “More for less: Non-intrusive speech quality assessment with limited annotations,” in _Proc. Thirteenth International Conference on Quality of Multimedia Experience_ , 2021, pp. 103–108.
  * [47] N. Nessler, M. Cernak, P. Prandoni, and P. Mainar, “Non-intrusive speech quality assessment with transfer learning and subject-specific scaling,” in _Proc. Interspeech_ , 2021, pp. 2406–2410.
  * [48] M. Yu, C. Zhang, Y. Xu, S.-X. Zhang, and D. Yu, “MetricNet: Towards Improved Modeling For Non-Intrusive Speech Quality Assessment,” in _Proc. Interspeech_ , 2021, pp. 2142–2146.
  * [49] Ĺuboš Marcinek, M. Stone, R. Millman, and P. Gaydecki, “N-MTTL SI model: Non-intrusive multi-task transfer learning-based speech intelligibility prediction model with scenery classification,” in _Proc. Interspeech_ , 2021, pp. 3365–3369.
  * [50] G. Mittag, B. Naderi, A. Chehadi, and S. Möller, “NISQA: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets,” in _Proc. Interspeech_ , 2021, pp. 2127–2131.
  * [51] S. Voran, “An iterated nested least-squares algorithm for fitting multiple data sets,” U.S. Department of Commerce, National Telecommunications and Information Administration, Institute for Telecommunication Sciences, Tech. Rep. TM-03-397, 2002.
  * [52] _ITU-T Recommendation G.711, Pulse code modulation (PCM) of voice frequencies_ , Geneva, 1988.
  * [53] M. Torcoli, J. Paulus, T. Kastner, and C. Uhle, “Controlling the remixing of separated dialogue with a non-intrusive quality estimate,” in _Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics_ , Oct. 2021, pp. 91–95.
  * [54] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time–frequency weighted noisy speech,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 7, pp. 2125–2136, 2011.
  * [55] A. A. Catellier and S. D. Voran, “WAWEnets: A no-reference convolutional waveform-based approach to estimating narrowband and wideband speech quality,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020, pp. 331–335.
  * [56] _ITU-T P Series Supplement 23 Speech Database_ , Geneva, 1998.
  * [57] _ITU-T Recommendation P.501, Test signals for use in telephonometry_ , Geneva, 2017.
  * [58] McGill University, Telecommunications and Signal Processing Laboratory, “Speech database,” available at http://www-mmsp.ece.mcgill.ca/Documents/Data/.
  * [59] J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren, “DARPA TIMIT acoustic phonetic continuous speech corpus CDROM,” 1993.
  * [60] A. Rousseau , P. Deléglise, and Y. Estève , “TED-LIUM: An automatic speech recognition dedicated corpus,” in _Proc. Eighth International Conference on Language Resources and Evaluation_ , May 2012.
  * [61] C. D. Hernandez-Mena, “TEDx Spanish corpus. audio and transcripts in spanish taken from the TEDx talks; shared under the cc by-nc-nd 4.0 license,” Web Download, 2019.
  * [62] D. Wang, X. Zhang, and Z. Zhang, “THCHS-30: A free Chinese speech corpus,” 2015. [Online]. Available: http://arxiv.org/abs/1512.01882
  * [63] Y. Choi and B. Lee, “Pansori: ASR corpus generation from open online video contents,” in _Proc. IEEE Seoul Section Student Paper Contest_ , Nov. 2018, pp. 117–121.
  * [64] Recordings of African Accented French speech, Available at http://openslr.org/57.
  * [65] Open Speech Repository, Available at https://www.voiptroubleshooter.com/.
  * [66] _ITU-T Recommendation P.191, Software tools for speech and audio coding standardization_ , Geneva, 2005.
  * [67] V. Emiya, E. Vincent, N. Harlander, and V. Hohmann, “Subjective and objective quality assessment of audio source separation,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 7, pp. 2046–2057, Sep. 2011.
  * [68] S. Van Kuyk, W. B. Kleijn, and R. C. Hendriks, “An instrumental intelligibility metric based on information theory,” _IEEE Signal Processing Letters_ , vol. 25, no. 1, pp. 115–119, 2018.
  * [69] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in _Proc. IEEE International Conference on Computer Vision_ , 2015, pp. 1026–1034.
  * [70] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. Third International Conference on Learning Representations_ , Y. Bengio and Y. LeCun, Eds., May 2015. [Online]. Available: http://arxiv.org/abs/1412.6980
  * [71] X. Dong and D. S. Williamson, “A pyramid recurrent network for predicting crowdsourced speech-quality ratings of real-world signals,” in _Proc. Interspeech_ , Oct. 2020.
  * [72] S. Voran, “A basic experiment on time-varying speech quality,” in _Proc. Fourth International MESAQIN (Measurement of Speech and Audio Quality in Networks) Conference_ , Prague, Czech Republic, Jun. 2005, pp. 51–64.

Andrew Catellier (Senior Member, IEEE) attended the University of Wyoming
where he earned a B.S. in Electrical Engineering in 2006 and an M.S. in
Electrical Engineering in 2007. Since then, Andrew conducted traditional and
crowd-sourced subjective experiments for audio and video signals and explored
objective speech quality assessment using ML techniques at the Institute for
Telecommunication Sciences (ITS) in Boulder, Colorado. He worked to facilitate
crop yield measurements, forecasts, and classification via remote sensing,
computer vision, and ML at GeoVisual Analytics from 2017 until 2022.
Currently, Andrew uses computer vision and ML to validate identification
documents and identity at Nametag, Inc.
---
Stephen Voran (Senior Member, IEEE) received the B.A. degree in Mathematics
from Carleton College, Northfield, MN, in 1985, and the M.S. degree in
Electrical Engineering from the University of Colorado, Boulder, in 1989.
Since 1990 he has been with the Institute for Telecommunication Sciences,
Boulder, Colorado and has been contributing to signal-processing based
advances in objective assessment of telecommunications speech quality and
intelligibility, as well as subjective audio testing, speech coding, and audio
quality enhancement.
---