# FG-NeRF: Flow-GAN based Probabilistic Neural Radiance Field for
Independence-Assumption-Free Uncertainty Estimation

Songlin Wei1,2 Jiazhao Zhang1,2 Yang Wang1 Fanbo Xiang3 Hao Su3 He Wang1,2

###### Abstract

Neural radiance fields with stochasticity have garnered significant interest
by enabling the sampling of plausible radiance fields and quantifying
uncertainty for downstream tasks. Existing works rely on the independence
assumption of points in the radiance field or the pixels in input views to
obtain tractable forms of the probability density function. However, this
assumption inadvertently impacts performance when dealing with intricate
geometry and texture. In this work, we propose an independence-assumption-free
probabilistic neural radiance field based on Flow-GAN. By combining the
generative capability of adversarial learning and the powerful expressivity of
normalizing flow, our method explicitly models the density-radiance
distribution of the whole scene. We represent our probabilistic NeRF as a
mean-shifted probabilistic residual neural model. Our model is trained without
an explicit likelihood function, thereby avoiding the independence assumption.
Specifically, We downsample the training images with different strides and
centers to form fixed-size patches which are used to train the generator with
patch-based adversarial learning. Through extensive experiments, our method
demonstrates state-of-the-art performance by predicting lower rendering errors
and more reliable uncertainty on both synthetic and real-world datasets.

## Introduction

As a popular approach for neural scene modeling, Neural Radiance Fields (NeRF)
(Mildenhall et al. 2021) have been widely studied in recent years due to their
ability to render photorealistic novel views. Additionally, NeRF has shown
effectiveness in various related fields, including 3D reconstruction (Wang et
al. 2021; Yariv et al. 2021; Müller et al. 2022), camera-pose recovery (Meng
et al. 2021; Schwarz et al. 2020), 3D semantic segmentation (Zhi et al. 2021;
Vora et al. 2021), and photo-realistic simulation (Li et al. 2023, 2021).
However, the majority of existing works (Gao et al. 2022) only output one
color for each pixel while falling short to capture the uncertainty in neural
scene modeling. This limitation hinders the applications of NeRFs in active
perception and interaction tasks, e.g., robotics (Ran et al. 2022; Pan et al.
2022), autonomous driving (Li et al. 2022b), and human-computer interaction
(Li et al. 2022a), where it is essential to incorporate uncertain information
(Cai et al. 2021).

Figure 1: We propose to use adversarial learning to model probabilistic neural
radiance field $\mathcal{F}$. Previous work models the whole density-radiance
probability distribution as $p(\mathcal{F})$. S-NeRF factorizes the
$p(\mathcal{F})$ as the product of the density distribution $p(\sigma_{ijk})$
and radiance distribution $p(c_{ijk})$ of all sampled points, while CF-NeRF
(Shen et al. 2022) uses latent variable $z$ to make the sampled points
conditional independent. On the contrary, our work FG-NeRF directly models the
distribution $P_{Real}$ of the rendered image $I$ through volume rendering of
$\mathcal{F}$.

To capture the uncertainty of NeRF, recent studies have proposed various types
of probabilistic neural radiance fields (Shen et al. 2021, 2022) that aim to
learn a distribution over the radiance field. However, learning a distribution
over such continuous fields presents significant challenges, due to the vast
probability space involved. Consequently, existing approaches usually rely on
strong assumptions. For example, Stochastic NeRF (S-NeRF) (Shen et al. 2021)
assumes the radiance and density values are independent Gaussian distributions
at different locations, which neglects the important spatial consistency in 3D
spaces. The state-of-the-art method, Conditional-Flow NeRF (CF-NeRF) (Shen et
al. 2022), leverages normalizing flow (Winkler et al. 2019) technique to
capture arbitrary radiance/density distributions. Furthermore, CF-NeRF
introduces a global latent variable and achieves conditional independence
among the values via De Finetti’s representation theorem (Kirsch 2019).
However, we contend that the conditional independence assumption still
restricts the capacity of probabilistic NeRF to represent complex and large
scenes accurately.

In this work, we propose a novel probabilistic neural radiance field, Flow-GAN
NeRF (FG-NeRF). Inspired by Flow-GAN (Grover, Dhar, and Ermon 2018a), our
method leverages both normalizing flow (Winkler et al. 2019) and GAN
(Goodfellow et al. 2020). We propose to use adversarial training to learn the
radiance/density distributions of the whole scene. In our FG-NeRF, the
normalizing flow module serves only as a generator and is trained along with
an additional discriminator, which together form a GAN framework. As a result,
our method exhibits two primary distinctions from S-NeRF or CF-Nerf (See Fig.
1: first, we leverage generative adversarial learning instead of variational
inference used in prior methods. Second, our method dispenses with the need
for scene-wide distribution factorization to compute the likelihood or self-
entropy term during variational inference (See Sec. 5), thereby achieving
independence-assumption free. Consequently, FG-NeRF demonstrates the
capability to produce high-quality uncertainty estimations while effectively
capturing intricate appearance and geometry in room-level scenes.

Through extensive experiments, FG-NeRF showcases new state-of-the-art
performance when compared to existing uncertainty estimation methods on LLFF
(Mildenhall et al. 2021), ScanNet (Dai et al. 2017), and Replica (Straub et
al. 2019) datasets. Our proposed method achieves improved uncertainty
estimation, accompanied by high-quality color and depth predictions. Moreover,
by integrating FG-NeRF with a multi-level hash encoding technique (Müller et
al. 2022), our approach achieves a remarkable four times faster compared to
CF-NeRF. With this significant improvement in computational efficiency, we
further build an autonomous NeRF reconstruction, showcasing the superior
performance of our predicted uncertainty.

## Related Work

NeRF (Mildenhall et al. 2021) has been widely studied due to its impressive
capability in reconstructing scenes for photorealistic novel views. By
utilizing a sparse collection of 2D views and employing a neural network to
encode a representation of the 3D scene. This neural network consists of a set
of fully-connected layers that output the radiance for any given input 3D
spatial location and 2D viewing direction within the scene. In recent years,
NeRF has been extensively researched to improve the quality (Zhang et al.
2020; Kosiorek et al. 2021; Barron et al. 2021; Wang et al. 2021) and
efficiency (Neff et al. 2021; Müller et al. 2022; Chen et al. 2022a). However,
despite the advancements made, NeRF is unable to quantify the uncertainty
associated with rendered views or estimated geometry, which limits its
applicability in active tasks. To address this limitation, we propose FG-NeRF,
a probabilistic neural radiance field that explicitly quantifies uncertainty.

#### Uncertainty estimation in NeRF.

The estimation of uncertainty has been a widely explored topic in deep
learning (Hendrycks et al. 2019; Brachmann et al. 2016; Malinin and Gales
2018; Teye, Azizpour, and Smith 2018; Geifman, Uziel, and El-Yaniv 2018).
Here, we only review the papers that are highly relevant to NeRF. Early
studies (Lee et al. 2022) regard uncertainty as the entropy of the density
among the sampled ray. Despite the simplicity, this approach tends to perform
poorly on complex geometry and lacks the capability to estimate uncertainty at
arbitrary positions. Another line of research (Ran et al. 2022; Pan et al.
2022; Martin-Brualla et al. 2021) attempted to implicitly predict the variance
associated with rendered pixels or positions. However, These methods apply the
volume rendering process over uncertainty values are not theoretically
grounded.

Recently, the most compelling approach is to learn a posterior distribution
over the model parameters. S-NeRF (Shen et al. 2021), for instance, models a
distribution encompassing all possible radiance fields that explain the scene.
To ensure tractability, S-NeRF makes a strong assumption by assuming that all
predicted distributions of each point are independent. In an effort to
alleviate this limitation, CF-NeRF (Shen et al. 2022) softens this assumption
by conditioning the predicted distributions on a shared latent variable (Yang
et al. 2019). This design enables spatially smooth distribution and improves
the quality of predictions. However, we argue that CF-NeRF still relies on a
conditional independence assumption through the use of a global latent
variable. Notably, our FG-NeRF stands out by sidestepping the independence
assumptions by employing a generator and a discriminator that are trained
adversarially.

#### Adversarial Learning for NeRF.

Adversarial learning (Goodfellow et al. 2020; Creswell et al. 2018; Grover,
Dhar, and Ermon 2018b) has recently emerged as a promising approach for
learning the radiance field for various challenging tasks. Previous relevant
studies (Meng et al. 2021; Schwarz et al. 2020; Chan et al. 2021) have
primarily focused on leveraging generative models to learn the radiance field
from unstructured 2D images. Recent research efforts (Deng et al. 2022; Chan
et al. 2022) have made significant advancements in enhancing synthesis quality
by achieving higher resolutions, improved 3D geometry, and faster rendering.
However, to the best of our knowledge, our work is the first to utilize
adversarial learning specifically for uncertainty estimation in NeRF.

## Background

#### Deterministic and probabilistic formulations of NeRF

Neural radiance field (NeRF) (Mildenhall et al. 2021) is an implicit scene
representation. It is modeled as a continuous function
$\mathcal{F}:(x,d)\rightarrow(\sigma,c)$, which maps the pair of 3D spatial
location $x\in\mathbb{R}^{3}$ and viewing direction $d\in\mathbb{S}^{2}$ into
density $\sigma\in\mathbb{R}^{+}$ and RGB radiance $c\in[0,1]^{3}$. Given a
camera ray $r(t)=x_{0}+td~{}$shooting from origin $x_{o}$ along the direction
$d$, the color $c$ of the ray is computed via the physics-based volume
rendering:

$\displaystyle c(r)=\int_{0}^{\infty}T(t)\sigma(r(t))c(r(t),d)dt,$ (1)
$\displaystyle~{}\textnormal{where}~{}T(t)=exp(-\int_{0}^{t}\sigma(r(t))ds),$

and the function $T(t)$ is the accumulated transmittance along the ray. It is
the probability that the ray travels from the origin to $x_{t}$ without
hitting any obstacles.

The radiance field $\mathcal{F}$ is deterministic in terms of having a single
density and RGB radiance for each point in the space. Deterministic NeRF is
unable to output the uncertainty of the rendering when the output confidence
is desired. One approach to remedy this problem is to formulate the neural
radiance construction as probabilistic. Probabilistic NeRF models the scene as
a joint probabilistic density and radiance distribution $p(\mathcal{F})$ for
all the spatial locations and view directions. In this manner, the
deterministic radiance field can be regarded as one realization over the
distribution $p(\mathcal{F})$.

#### Modeling density-radiance distributions with normalizing flow

To learn the probability distribution $\mathcal{F}$ of the whole scene, we
first discuss how to model the distribution of a single point through
normalizing flow. For any location $x$ viewed from direction $d$, the
conditional joint distribution of the density and radiance is
$p(\sigma,c|x,d)=p(\sigma|x)p(c|x,d)$ where density $\sigma$ is assumed to
depend only on location $x$. Approximating the radiance distribution
$p(c|x,d)$ with unimodal distributions like Gaussian is sub-optimal, as a
point in the real-world can have different colors viewed from different
directions due to reflection or iridescence. Inspired by CF-NeRF (Shen et al.
2022), we use conditional normalizing flow (CNF) (Winkler et al. 2019) which
transforms a simple distribution into arbitrary complex shape conditioning on
the location-view pair $(x,d)$. Specifically, conditional normalizing flow
maps the initial random variable $(\sigma_{0},c_{0})$ with the prior
distributions $q_{\sigma}(\sigma_{0})$ and $q_{c}(c_{0})$ into the target
$(\sigma,c)$ via $K$ steps of bijective mappings
$\sigma_{k}=f_{k}^{\sigma}(\sigma_{k-1},x)$ and $c_{k}=f_{k}^{c}(c_{k-1},x,d)$
respectively. The target distributions can be computed via the change-of-
variables formula:

$p(\sigma|x)=q_{\sigma}(\sigma_{0})\prod_{k=1}^{K-1}\left|\det{\frac{\partial
f_{k}^{\sigma}}{\partial\sigma_{k-1}}}\right|^{-1},$ (2)
$p(c|x,d)=q_{c}(c_{0})\prod_{k=1}^{K-1}\left|\det{\frac{\partial
f_{k}^{c}}{\partial c_{k-1}}}\right|^{-1}$ (3)

The initial distribution $q_{\sigma}$ and $q_{c}$ is usually set to standard
Gaussian distributions.

Because directly computing the density-radiance distribution $p(\mathcal{F})$
for the whole scene is intractable, previous work (Shen et al. 2021)(Shen et
al. 2022) approximate the probabilistic distribution using variational
inference (Blei, Kucukelbir, and McAuliffe 2017) via a parametric function
$q(\mathcal{F};\theta)$. The models boil down to the computation of a
likelihood term $p(\mathcal{T}|\mathcal{F})$ of the training sample and a
self-entropy term. They assume the distribution for each point in the scene is
independent or conditionally independent to constrain the
$q(\mathcal{F};\theta)$, which can then be factorized into the product of each
point distribution to make the computation tractable:

$q_{\vartheta}(\mathcal{F})=\prod_{x\in\mathbb{R}^{3}}\prod_{d\in\mathbb{S}^{2}}p(\sigma|x)p(c|x,d),~{}~{}\textnormal{or}$
(4)
$q_{\vartheta}(\mathcal{F}|z)=\int{p(z)\prod_{x\in\mathbb{R}^{3}}\prod_{d\in\mathbb{S}^{2}}p(\sigma|x,z)p(c|x,d,z)}.$
(5)

Nevertheless, neighboring 3D points within the scene are prone to exhibit
analogous density or color, thereby casting doubt on the plausibility of the
conditional independence assumption.

Although these works produce high-quality results of uncertainty estimation on
object-level scenes, experiments show that this assumption hampers the
performance when the scene becomes large and complex.

## Method

Figure 2: Overview of our pipeline. Our pipeline learns to generate fake image
patches given the training image $I$ and pose $T$. The generator $G_{\xi}$ and
discriminator $D_{\eta}$ jointly guide the normalizing flow (CNF) for modeling
the density/radiance distribution.

We study the probability distribution of the entire density-radiance field.
More concretely, given $\mathcal{T}=\\{(I^{i},T^{i},D^{i})\\}_{i=1}$ denotes
the triplets of image $I$, pose $T$ and optional depth $D$ indexed by $i$, we
goal is to learn the posterior distribution $p(\mathcal{F}|\mathcal{T})$.

Our whole pipeline is illustrated in Fig. 2. Given an image $I$, we
dynamically sample a batch of rays $\mathcal{R}$ to obtain an image patch $P$.
The location-view pairs $(x,d)$ which are sampled through the stratified
sampling of the rays originating from camera pose $T$ are fed into the
generator $G_{\theta}$. The generator consists of two branches: (1) the
deterministic branch learns the mean values for (2) the probabilistic branch
which, is implemented with the conditional normalizing flow (CNF), learns to
predict the mean-shifted probability distribution of the radiance field of the
scene. Finally, a synthesized patch $\hat{P}$ is obtained through volume
rendering of the combined branches and is sent to the discriminator $D_{\phi}$
to compare with the real training patch $P$.

### Decomposing probabilistic neural radiance field

Directly learning the probability distribution of the density-radiance field
presents additional challenges due to the significant divergences in
distribution between occupied and free regions. Consequently, the presence of
large density values (ranging from $0$ to $+\infty$) can lead to numerical
instability during the training of the normalizing flow. To tackle this
challenge, we decompose $\mathcal{F}$ into a deterministic branch and a
residual probabilistic branch:

$\mathcal{F}=\mathcal{F}^{\mu}+\mathcal{F}^{\epsilon}$ (6)

The deterministic branch $\mathcal{F}^{\mu}$ acting like the mean of the
probabilistic density-radiance field is estimated following the standard
volume rendering procedure. It provides the mean values of density and
radiance for each location-view pair (x,d) in the scene. Thus, the
probabilistic branch $\mathcal{F}^{\epsilon}$ is only needed to learn a mean-
shifted residual distribution. The distributions are learned via conditional
normalizing flow as described in Section Background. Given a batch of rays
$\mathcal{R}$ sampled from the training images $I$, the deterministic branch
$\mathcal{F}^{\mu}$ of the network is supervised with the $L2$ loss between
the rendered color $\hat{c}(r)$ and the ground truth color of each ray $r$.
The loss is denoted as

$\mathcal{L}_{det}=\sum_{r\in\mathcal{R}}\|I[r]-\hat{c}(r)\|^{2}_{2}~{}$ (7)

where $\hat{c}(r)$ represents the rendered color of the ray $r$. It is
obtained through alpha-composition:

$\displaystyle\hat{c}=\sum_{i=1}^{N}T^{i}\alpha^{i}c^{\mu,i},$ (8)
$\displaystyle T^{i}=\prod_{j=1}^{i-1}(1-\alpha^{j}),$
$\displaystyle\alpha^{i}=1-\exp(-\sigma^{\mu,i}\delta^{i})$

where superscript $i$ indicates the index of the discrete sampling points
along the ray. $\alpha$ denotes the opacity of the color at the sampling point
and $\delta$ is the distance between neighboring sampling points. With this
deterministic branch been regularized by the rendering loss, our network can
efficiently learn a rough geometry and appearance of the scene. Furthermore,
the residual mean-shifted probabilistic branch $\mathcal{F}^{\epsilon}$ is
eased to learn a zero-mean centered distribution of density
$\sigma^{\epsilon}$ and radiance $c^{\epsilon}$.

Thus, the combined density and radiance are

$\sigma=\sigma^{\mu}+\sigma^{\epsilon}~{}~{}~{}~{}~{}~{}c=c^{\mu}+c^{\epsilon}~{}.$
(9)

### Adversarial learning of probabilistic NeRF based on Flow-GAN

Instead of the factorizations of the probability distributions as done in
(Shen et al. 2021)(Shen et al. 2022), we propose to employ adversarial
learning to sidestep the direct computation of the likelihood functions as
illustrated in Fig. 3. Therefore, our method does not rely on any form of
independence assumptions. We first dynamically sample pixels (Isola et al.
2017) structured with $K\times K$ pixels from training image $I$. The image
patch follows the real data distribution $p_{data}$. Then, we train a
generator $G_{\xi}$ to synthesize the patch using known camera pose $T$ and a
discriminator $D_{\eta}$ to compare the real and fake patches.

Figure 3: Comparison of patch-based adversarial loss and pixel-level rendering
loss

More specifically, the $K^{2}$ rays in the patch are spaced evenly with random
intervals and the center is also randomly shifted within the image. This
sampling strategy preserves the image structure which allows the generator to
generate high-quality samples. We then employ stratified sampling following
standard NeRF to get $N$ points along each ray. We follow (Müller et al. 2022)
to use multi-resolution hash encoding to encode sampling points $x$ and use
the spherical harmonics to encode the view direction $d$.

The generator $G_{\xi}$ is implemented with CNF. For each sampling point $x$,
we first acquire its mean density $\sigma^{\mu}$ and color $c^{\mu}$ through
the deterministic branch $\mathcal{F}^{\mu}$ and then transform an initial
density $\sigma^{\epsilon}$ and color $c^{\epsilon}$ following standard
Guassians into the target distribution using CNF, which is conditional on the
location feature $h^{\sigma}$ and radiance feature $h^{c}$. Subsequently, we
synthesize the image patch via volume rendering of the combined densities and
colors as in Eq. (9) and Eq. (8). The synthesized image $\hat{P}$ will be fed
into a discriminator $D_{\eta}$ to be compared with the real image patches
$P$. Finally, we minimize a distribution distance between the generated image
$p(\mathcal{F})$ and the real image patch $p_{data}$ based on adversarial loss
$\mathcal{L}_{adv}$ as is done in classic GAN framework. The training
objective is:

$\displaystyle\min_{\xi\in\Xi}\max_{\eta\in
E}\mathcal{L}_{adv}=\mathbb{E}_{P\sim p_{data}}[\log(D(P;\eta))]+$ (10)
$\displaystyle~{}~{}~{}\mathbb{E}_{\hat{P}\sim
p(\mathcal{F})}[\log(1-D(G(\sigma_{0},c_{0};\xi);\eta))].$

The discriminator $D_{\eta}$ is trained to output $1$ for real patches and $0$
for fake patches. It is implemented as a convolutional neural network
following (Schwarz et al. 2020). The discriminator is conditioned on the scale
$s$ to perform well. The scale $s$ is the ratio of the size of the sampled
patch with respect to the input image. $s$ is set to 1 at the beginning of the
training to have a large perceptive field and gradually decreased to a small
value to capture fine-grained details of the scene. The generator $G_{\xi}$
and discriminator $D_{\eta}$ are trained adversarially to promote the
generator to synthesize more realistic images. As a result, the normalizing
flow in the generator is trained to produce high-quality density-radiance
distributions for synthesizing photo-realistic images. Thus, uncertainty
quantified by such distributions has better qualities compared with previous
works as demonstrated in the experiments.

### Optional extra depth supervision

Unlike object-level scenes where training rays are densely available for every
point in the scene. It is challenging for NeRF to model large scenes like
ScanNet (Dai et al. 2017) due to the limited number of training images. In
such cases, depth supervision further improves the learning of geometry and
radiance. The ray depth can be obtained by the integral:

$\hat{D}=\sum_{i=1}^{N}w^{i}R^{i}$ (11)

where $w^{i}=T^{i}\alpha^{i}$ is the weight of the ray depth $R^{i}$ of the
sampling point. However, straightforward $L2$ loss
$\mathcal{L}_{depth}=||D-\hat{D}||^{2}_{2}$ supervision performs poorly. We
empirically found that using the cross-entropy loss between the sample weights
$w(t)=T(t)\alpha(t)$ and an assumed ground truth depth centered Gaussian
distribution $\mathcal{N}(D,\sigma^{2})$ performs better than straightforward
$L2$ loss. The cross-entropy loss is defined as:

$\mathcal{L}_{depth}=\frac{1}{|\mathcal{R}|}\sum_{r\in\mathcal{R}}\log
w_{r}^{i}\exp(-\frac{(t_{i}-D)^{2}}{2\sigma^{2}})$ (12)

where $\Sigma$ is set to $0.1$ in the experiments. Intuitively, such cross
entropy constrains the weights to follow a unimodal distribution hence
enforcing stronger regularization when learning the density distributions.

### Quantifying uncertainty

After the distributions over the radiance field are learned, we are able to
quantify the uncertainty of density or radiance of the scene. For each point
$x$, we first sample $S$ values from the distribution learned with normalizing
flow and then use the sample variance to quantify the density uncertainty. The
density uncertainty is a measure of convergence of the learned density. A
similar procedure is employed to obtain the radiance uncertainty. We can also
quantify the uncertainty associated with the rendering image and depth map.
Given any camera pose, we run the generator $M$ times to sample $M$
trajectories of random variables $(x^{i},c^{i})_{1:M}$ along each ray.
Subsequently, we synthesize the images $\hat{I}_{1:M}$ and depth maps
$D_{1:M}$. The uncertainties can then be obtained through the color and depth
variance of each pixel. Object edges often have high-density uncertainties due
to the discontinuity of the depth whereas areas with high-frequency colors
often have high radiance uncertainties. This kind of uncertainty is shown to
be beneficial for active NeRF reconstruction in the experiments.

### Training protocol

In contrast to standard NeRF, which primarily focuses on minimizing rendering
loss, our GAN based on probabilistic NeRF incorporates a generator and
discriminator that are trained in a min-max game framework. The generator
comprises a deterministic branch and a probabilistic branch implemented with a
normalizing flow. To ensure training stability, we adopt instance
normalization (Ulyanov, Vedaldi, and Lempitsky 2016) and spectral
normalization (Miyato et al. 2018) in the discriminator. To enhance
convergence, we employ a small number of $M=8$ samples for each point in the
normalizing flow, gradually increasing it to 16 to generate $M$ fake image
patches. Simultaneously, we replicate the real patch $M$ times before feeding
it into the discriminator to maintain gradient balance during backpropagation.
More implementation details can be found in the supplemental material.

## Experiments

#### Datasets.

We conducted evaluations of our method on three diverse publicly available
datasets, namely LLFF (Mildenhall et al. 2021), Replica (Straub et al. 2019),
and ScanNet (Dai et al. 2017). The LLFF dataset contains complex geometry and
appearance, with camera views predominantly directed toward a single object.
Notably, the LLFF dataset has also been utilized in S-NeRF (Shen et al. 2021)
and CF-NeR (Shen et al. 2022). Additionally, we employed the Scannet and
Replica datasets to evaluate the performance of our methods in challenging
scenarios involving large scenes with diverse camera views. The Replica
dataset comprises high-precision reconstructed scenes with nearly perfect
rendering results, while the Scannet dataset consists of real-captured RGB-D
sequences obtained using consumer-level cameras. Following the same
experimental setting of (Yu et al. 2022), we utilized a subset of scenes in
accordance with the methodology.

Table 1: Comparison of uncertainty estimation and rendered color on the LLFF
dataset. The best results are indicated in bold, with the second-best results
underlined.

|  | Quality Metrics |  | Uncertainty Metrics |
---|---|---|---|---|---
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |  | AUSE RMSE$\downarrow$ | AUSE MAE$\downarrow$ |
D.E. (Lakshminarayanan, Pritzel, and Blundell 2017) |  | 22.32 | 0.788 | 0.236 |  | 0.0254 | 0.0122 |
Drop. (Gal and Ghahramani 2016) |  | 21.90 | 0.758 | 0.248 |  | 0.0316 | 0.0162 |
NeRF-W (Martin-Brualla et al. 2021) |  | 20.19 | 0.706 | 0.291 |  | 0.0268 | 0.0113 |
S-NeRF (Shen et al. 2021) |  | 20.27 | 0.738 | 0.229 |  | 0.0248 | 0.0101 |
CF-NeRF (Shen et al. 2022) |  | 21.96 | 0.790 | 0.201 |  | 0.0177 | 0.0078 |
FG-NeRF (Ours) |  | 24.41 | 0.816 | 0.170 |  | 0.0118 | 0.0028 |

#### Metrics.

To evaluate the predicted uncertainty and rendered quality, we employ standard
metrics commonly used in the literature (Mildenhall et al. 2021; Shen et al.
2021, 2022). For uncertainty evaluation, we follow previous works by utilizing
the AUSE (Area Under the Sparsification Error curve) metric proposed in (Bae,
Budvytis, and Cipolla 2021; Poggi et al. 2020). AUSE evaluates the correlation
between uncertainty and predicted error. Given an error metric (RMSE or MAE),
we can sort the pixels of the uncertainty map and the error map respectively.
Repeatedly removing the top $1\%$ of the sorted error pixels, leads to a
sparsification curve and an oracle curve. The AUSE is then computed as the
area between these two curves. For rendered quality evaluation, we employ
widely adopted metrics such as PSNR, SSIM (Structural Similarity Index), and
LIPIP (Learned Perceptual Image Patch Similarity). Detailed descriptions of
these metrics can be found in the supplementary material.

Figure 4: Quality comparison on the LLFF, ScanNet, and Replica datasets.

### Comparision on LLFF dataset

Following the same experimental setup in (Shen et al. 2022), we evaluate our
uncertainty estimation method on the LLFF dataset to compare with existing
methods (Lakshminarayanan, Pritzel, and Blundell 2017; Gal and Ghahramani
2016; Martin-Brualla et al. 2021; Shen et al. 2022, 2021). Note that, the
camera views from LLFF dataset are primarily forward-facing to a central
object within a small range of angles. Consequently, the coordinates can be
constrained within the range of $[0,1]$ using Normalized Device Coordinates
(NDC), thereby facilitating the learning of the radiance field. The average
results of eight scenes from LLFF dataset are presented in Tab. 1. Our method
FG-NeRF achieves significant improvements in uncertainty and demonstrates
better rendering quality as well (see Fig. 4 third rows for a visual
comparison). Compared to the previous SOTA method CF-NeRF, the AUSE of our
method decreased by $33.3\%$ and $64.1\%$ on RMSE and MAE, respectively,
clearly demonstrating the superiority of our methods. Additionally, the
results prove that the CF-NeRF outperforms the S-NeRF by relaxing the strong
independence assumption. Furthermore, our method achieves the best performance
by introducing adversarial learning (Grover, Dhar, and Ermon 2018a) to
sidestep any independence assumptions.

Table 2: Quantitatively comparison with baseline method on Scanet and Replica
datasets.

Scene Type | Method |  | Quality Metrics |  | Uncertainty Metrics
---|---|---|---|---|---
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |  | AUSE RMSE$\downarrow$ | AUSE MAE$\downarrow$
ScanNet | CF-NeRF (Shen et al. 2022) |  | 20.67 | 0.627 | 0.477 |  | 0.0293 | 0.0353
NeurAR (Ran et al. 2023) |  | 23.78 | 0.765 | 0.347 |  | 0.0265 | 0.0150
Ours |  | 24.76 | 0.793 | 0.388 |  | 0.0136 | 0.0043
Replica | CF-NeRF (Shen et al. 2022) |  | 20.35 | 0.621 | 0.418 |  | 0.0196 | 0.0058
NeurAR (Ran et al. 2023) |  | 23.31 | 0.715 | 0.395 |  | 0.0175 | 0.0050
Ours |  | 25.40 | 0.816 | 0.364 |  | 0.0136 | 0.0040

### Comparison on Replica and Scannet dataset

Compared to object-level scenes, room-scale scenes present additional
challenges due to their diverse camera views and complex geometry and
appearance. In this context, we compare our methods with the previous state-
of-the-art approach CF-NeRF (Shen et al. 2022) and NeurAR (Ran et al. 2022),
specifically designed for scene-level uncertainty estimation. The results can
be found in Table 2, and visual comparisons are illustrated in Figure 4.
Notably, our method consistently achieves the best performance on both
datasets. It is worth mentioning that CF-NeRF exhibits a performance drop when
transitioning from the LLFF dataset (as shown in Table 1) to the ScanNet and
Replica datasets (as shown in Table 2). In contrast, our method maintains a
substantial level of performance, as observed in Table 1 on the LLFF dataset.

Additionally, we conduct a more challenging experiment to evaluate uncertainty
estimation using a reduced number of frames ($10\%$ of the frames). The
visualizations can be found in Figure 5, where it is evident that our method
successfully predicts a clear boundary between the unobserved region and the
observed space. Moreover, our approach consistently maintains accurate
uncertainty estimation along the object edges.

Figure 5: Comparison on uncertainty estimation under limited observations on
ScanNet (the closer to blue, the smaller uncertainty). The unseen region is
indicated by a gray mask.

### Ablation study

We conduct an ablation study to verify the effectiveness of key techniques and
hyperparameter settings. As shown in Tab. 3, the baseline is our full
implementation with adversarial learning and has a deterministic branch. We
first compare our proposed adversarial learning with plain L2 loss supervision
as illustrated in Fig. 3. The L2 loss is the squared difference of the pixel
color between rendered image patch $\hat{P}$ and ground truth image patch $P$.
In the experiments, we observed that both the rendering quality and the
uncertainty metrics of adversarial learning are better than the case with
plain L2 loss supervision. We also ablation studies the influence of the
deterministic branch as proposed in the main pipeline. Experiments have shown
a slight performance drop when the deterministic branch is removed.
Additionally, we studied the different settings of the hyperparameters, eg.,
the batch and image patch size, the sample size along each ray, and the scale
annealing effect.

Table 3: Ablations on LLFF Trex

|  |  | Quality Metrics |  | Uncertainty Metrics
---|---|---|---|---|---
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |  | AUSE RMSE$\downarrow$ | AUSE MAE$\downarrow$
Baseline |  |  | 26.58 | 0.891 | 0.105 |  | 0.0070 | 0.0017
Color Supervision | L2 loss |  | 25.89 | 0.842 | 0.197 |  | 0.0093 | 0.0043
Deterministic branch | off |  | 24.96 | 0.723 | 0.285 |  | 0.0085 | 0.0049
Batch $B$ $\times$ Patch $K^{2}$ | $4\times 16^{2}$ |  | 25.43 | 0.817 | 0.091 |  | 0.0083 | 0.0024
Sample size $M$ | 24 |  | 25.44 | 0.854 | 0.146 |  | 0.0079 | 0.0019
| 16 |  | 24.64 | 0.825 | 0.184 |  | 0.0082 | 0.0019
| 8 |  | 23.58 | 0.777 | 0.248 |  | 0.0089 | 0.0022
Scale annealing | Off |  | 24.38 | 0.814 | 0.199 |  | 0.0087 | 0.0021

## Limitations

We acknowledge the presence of two main limitations in our approach. Firstly,
there is a significant computational cost requirement. FG-NeRG, similar to
S-NeRF and CF-NeRF, is trained to predict a radiance/density distribution,
which is more challenging to learn compared to a simplistic radiance/density
value. Consequently, these methods require more extensive computational
resources than the original NeRF framework. Despite adopting a multi-level
hashing coding method (Müller et al. 2022) for acceleration, our approach
still takes nearly 6 GPU hours (NVIDIA V100) to obtain fully trained results
on the LLFF dataset. This issue can potentially be alleviated by integrating
advanced NeRF acceleration techniques (Chen et al. 2022a, c). Secondly, the
rendering quality of our method is relatively low when compared to the latest
works in NeRF literature. We are keen on further enhancing the quality by
leveraging scene prior information (Peng et al. 2020), incorporating advanced
training strategies (Chen et al. 2022b), and employing novel backbone
architectures (Sun et al. 2022).

## Conclusions

In this paper, we propose FG-NeRF, a novel probabilistic neural radiance
field, for high-quality uncertainty estimation. Our approach utilizes
adversarial learning to model radiance/density distributions. Specifically,
FG-NeRF performs as a generator and is trained in conjunction with a
discriminator, forming an adversarial loss. By jointly supervising the colors
of all pixels within an image patch, our method avoids making the independence
assumption. Consequently, FG-NeRF effectively learns intricate radiance and
density distributions and demonstrates its capacity to predict high-quality
uncertainty, radiance, and density. Experimental evaluations on both synthetic
and real-world datasets showcase that our method outperforms existing methods,
exhibiting lower rendering errors and providing more reliable uncertainty
predictions. Moreover, we build and evaluate an active NeRF reconstruction
framework based on FG-NeRF, showcasing the potential applications of this
method.

## References

  * Bae, Budvytis, and Cipolla (2021) Bae, G.; Budvytis, I.; and Cipolla, R. 2021.  Estimating and exploiting the aleatoric uncertainty in surface normal estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 13137–13146.
  * Barron et al. (2021) Barron, J. T.; Mildenhall, B.; Tancik, M.; Hedman, P.; Martin-Brualla, R.; and Srinivasan, P. P. 2021.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 5855–5864.
  * Blei, Kucukelbir, and McAuliffe (2017) Blei, D. M.; Kucukelbir, A.; and McAuliffe, J. D. 2017.  Variational inference: A review for statisticians.  _Journal of the American statistical Association_ , 112(518): 859–877.
  * Brachmann et al. (2016) Brachmann, E.; Michel, F.; Krull, A.; Yang, M. Y.; Gumhold, S.; et al. 2016.  Uncertainty-driven 6d pose estimation of objects and scenes from a single rgb image.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3364–3372.
  * Cai et al. (2021) Cai, K.; Chen, W.; Wang, C.; Zhang, H.; and Meng, M. Q. 2021.  Curiosity-based Robot Navigation under Uncertainty in Crowded Environments.  _IEEE Robotics and Automation Letters_ , 8: 800–807.
  * Chan et al. (2022) Chan, E. R.; Lin, C. Z.; Chan, M. A.; Nagano, K.; Pan, B.; De Mello, S.; Gallo, O.; Guibas, L. J.; Tremblay, J.; Khamis, S.; et al. 2022.  Efficient geometry-aware 3D generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16123–16133.
  * Chan et al. (2021) Chan, E. R.; Monteiro, M.; Kellnhofer, P.; Wu, J.; and Wetzstein, G. 2021.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 5799–5809.
  * Chen et al. (2022a) Chen, A.; Xu, Z.; Geiger, A.; Yu, J.; and Su, H. 2022a.  Tensorf: Tensorial radiance fields.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_ , 333–350. Springer.
  * Chen et al. (2022b) Chen, T.; Wang, P.; Fan, Z.; and Wang, Z. 2022b.  Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 15170–15181.
  * Chen et al. (2022c) Chen, Z.; Funkhouser, T. A.; Hedman, P.; and Tagliasacchi, A. 2022c.  MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures.  _ArXiv_ , abs/2208.00277.
  * Creswell et al. (2018) Creswell, A.; White, T.; Dumoulin, V.; Arulkumaran, K.; Sengupta, B.; and Bharath, A. A. 2018.  Generative adversarial networks: An overview.  _IEEE signal processing magazine_ , 35(1): 53–65.
  * Dai et al. (2017) Dai, A.; Chang, A. X.; Savva, M.; Halber, M.; Funkhouser, T.; and Nießner, M. 2017.  ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes.  In _Proc. Computer Vision and Pattern Recognition (CVPR), IEEE_.
  * Deng et al. (2022) Deng, Y.; Yang, J.; Xiang, J.; and Tong, X. 2022.  Gram: Generative radiance manifolds for 3d-aware image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10673–10683.
  * Gal and Ghahramani (2016) Gal, Y.; and Ghahramani, Z. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , 1050–1059. PMLR.
  * Gao et al. (2022) Gao, K.; Gao, Y.; He, H.; Lu, D.; Xu, L.; and Li, J. 2022.  NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review.  _ArXiv_ , abs/2210.00379.
  * Geifman, Uziel, and El-Yaniv (2018) Geifman, Y.; Uziel, G.; and El-Yaniv, R. 2018.  Bias-reduced uncertainty estimation for deep neural classifiers.  _arXiv preprint arXiv:1805.08206_.
  * Goodfellow et al. (2020) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2020.  Generative adversarial networks.  _Communications of the ACM_ , 63(11): 139–144.
  * Grover, Dhar, and Ermon (2018a) Grover, A.; Dhar, M.; and Ermon, S. 2018a.  Flow-gan: Combining maximum likelihood and adversarial learning in generative models.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32.
  * Grover, Dhar, and Ermon (2018b) Grover, A.; Dhar, M.; and Ermon, S. 2018b.  Flow-gan: Combining maximum likelihood and adversarial learning in generative models.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32.
  * Hendrycks et al. (2019) Hendrycks, D.; Mazeika, M.; Kadavath, S.; and Song, D. 2019.  Using self-supervised learning can improve model robustness and uncertainty.  _Advances in neural information processing systems_ , 32.
  * Isola et al. (2017) Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A. 2017.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1125–1134.
  * Kirsch (2019) Kirsch, W. 2019.  An elementary proof of de Finetti’s theorem.  _Statistics & Probability Letters_, 151: 84–88.
  * Kosiorek et al. (2021) Kosiorek, A. R.; Strathmann, H.; Zoran, D.; Moreno, P.; Schneider, R.; Mokrá, S.; and Rezende, D. J. 2021.  Nerf-vae: A geometry aware 3d scene generative model.  In _International Conference on Machine Learning_ , 5742–5752. PMLR.
  * Lakshminarayanan, Pritzel, and Blundell (2017) Lakshminarayanan, B.; Pritzel, A.; and Blundell, C. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ , 30.
  * Lee et al. (2022) Lee, S.; Chen, L.; Wang, J.; Liniger, A.; Kumar, S.; and Yu, F. 2022.  Uncertainty Guided Policy for Active Robotic 3D Reconstruction Using Neural Radiance Fields.  _IEEE Robotics and Automation Letters_ , 7(4): 12070–12077.
  * Li et al. (2022a) Li, C.; Li, S.; Zhao, Y.; Zhu, W.; and Lin, Y. 2022a.  RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering.  _2022 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_ , 1–9.
  * Li et al. (2023) Li, X.; Qiao, Y.-L.; Chen, P. Y.; Jatavallabhula, K. M.; Lin, M.; Jiang, C.; and Gan, C. 2023.  PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification.  _ArXiv_ , abs/2303.05512.
  * Li et al. (2021) Li, Y.; Li, S.; Sitzmann, V.; Agrawal, P.; and Torralba, A. 2021.  3D Neural Scene Representations for Visuomotor Control.  _ArXiv_ , abs/2107.04004.
  * Li et al. (2022b) Li, Z.; Li, L.; Ma, Z.; Zhang, P.; Chen, J.; and Zhu, J.-Z. 2022b.  READ: Large-Scale Neural Scene Rendering for Autonomous Driving.  _ArXiv_ , abs/2205.05509.
  * Malinin and Gales (2018) Malinin, A.; and Gales, M. 2018.  Predictive uncertainty estimation via prior networks.  _Advances in neural information processing systems_ , 31.
  * Martin-Brualla et al. (2021) Martin-Brualla, R.; Radwan, N.; Sajjadi, M. S.; Barron, J. T.; Dosovitskiy, A.; and Duckworth, D. 2021.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7210–7219.
  * Meng et al. (2021) Meng, Q.; Chen, A.; Luo, H.; Wu, M.; Su, H.; Xu, L.; He, X.; and Yu, J. 2021.  Gnerf: Gan-based neural radiance field without posed camera.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 6351–6361.
  * Mildenhall et al. (2021) Mildenhall, B.; Srinivasan, P. P.; Tancik, M.; Barron, J. T.; Ramamoorthi, R.; and Ng, R. 2021.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1): 99–106.
  * Miyato et al. (2018) Miyato, T.; Kataoka, T.; Koyama, M.; and Yoshida, Y. 2018.  Spectral normalization for generative adversarial networks.  _arXiv preprint arXiv:1802.05957_.
  * Müller et al. (2022) Müller, T.; Evans, A.; Schied, C.; and Keller, A. 2022.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4): 1–15.
  * Neff et al. (2021) Neff, T.; Stadlbauer, P.; Parger, M.; Kurz, A.; Mueller, J. H.; Chaitanya, C. R. A.; Kaplanyan, A.; and Steinberger, M. 2021.  DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks.  In _Computer Graphics Forum_ , volume 40, 45–59. Wiley Online Library.
  * Pan et al. (2022) Pan, X.; Lai, Z.; Song, S.; and Huang, G. 2022.  ActiveNeRF: Learning Where to See with Uncertainty Estimation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII_ , 230–246. Springer.
  * Peng et al. (2020) Peng, S.; Niemeyer, M.; Mescheder, L. M.; Pollefeys, M.; and Geiger, A. 2020.  Convolutional Occupancy Networks.  _ArXiv_ , abs/2003.04618.
  * Poggi et al. (2020) Poggi, M.; Aleotti, F.; Tosi, F.; and Mattoccia, S. 2020.  On the uncertainty of self-supervised monocular depth estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 3227–3237.
  * Ran et al. (2023) Ran, Y.; Zeng, J.; He, S.; Chen, J.; Li, L.; Chen, Y.; Lee, G.; and Ye, Q. 2023\.  NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction with Implicit Neural Representations.  _IEEE Robotics and Automation Letters_.
  * Ran et al. (2022) Ran, Y.; Zeng, J.; He, S.; Li, L.; Chen, Y.; Lee, G.; Chen, J.; and Ye, Q. 2022\.  NeurAR: Neural Uncertainty for Autonomous 3D Reconstruction.  _arXiv preprint arXiv:2207.10985_.
  * Schwarz et al. (2020) Schwarz, K.; Liao, Y.; Niemeyer, M.; and Geiger, A. 2020.  Graf: Generative radiance fields for 3d-aware image synthesis.  _Advances in Neural Information Processing Systems_ , 33: 20154–20166.
  * Shen et al. (2022) Shen, J.; Agudo, A.; Moreno-Noguer, F.; and Ruiz, A. 2022.  Conditional-flow nerf: Accurate 3d modelling with reliable uncertainty quantification.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part III_ , 540–557. Springer.
  * Shen et al. (2021) Shen, J.; Ruiz, A.; Agudo, A.; and Moreno-Noguer, F. 2021.  Stochastic neural radiance fields: Quantifying uncertainty in implicit 3d representations.  In _2021 International Conference on 3D Vision (3DV)_ , 972–981. IEEE.
  * Straub et al. (2019) Straub, J.; Whelan, T.; Ma, L.; Chen, Y.; Wijmans, E.; Green, S.; Engel, J. J.; Mur-Artal, R.; Ren, C.; Verma, S.; Clarkson, A.; Yan, M.; Budge, B.; Yan, Y.; Pan, X.; Yon, J.; Zou, Y.; Leon, K.; Carter, N.; Briales, J.; Gillingham, T.; Mueggler, E.; Pesqueira, L.; Savva, M.; Batra, D.; Strasdat, H. M.; Nardi, R. D.; Goesele, M.; Lovegrove, S.; and Newcombe, R. 2019.  The Replica Dataset: A Digital Replica of Indoor Spaces.  _arXiv preprint arXiv:1906.05797_.
  * Sun et al. (2022) Sun, J.; Chen, X.; Wang, Q.; Li, Z.; Averbuch-Elor, H.; Zhou, X.; and Snavely, N. 2022.  Neural 3D Reconstruction in the Wild.  _ACM SIGGRAPH 2022 Conference Proceedings_.
  * Teye, Azizpour, and Smith (2018) Teye, M.; Azizpour, H.; and Smith, K. 2018.  Bayesian uncertainty estimation for batch normalized deep networks.  In _International Conference on Machine Learning_ , 4907–4916. PMLR.
  * Ulyanov, Vedaldi, and Lempitsky (2016) Ulyanov, D.; Vedaldi, A.; and Lempitsky, V. 2016.  Instance normalization: The missing ingredient for fast stylization.  _arXiv preprint arXiv:1607.08022_.
  * Vora et al. (2021) Vora, S.; Radwan, N.; Greff, K.; Meyer, H.; Genova, K.; Sajjadi, M. S. M.; Pot, E.; Tagliasacchi, A.; and Duckworth, D. 2021.  NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes.
  * Wang et al. (2021) Wang, P.; Liu, L.; Liu, Y.; Theobalt, C.; Komura, T.; and Wang, W. 2021.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_.
  * Winkler et al. (2019) Winkler, C.; Worrall, D.; Hoogeboom, E.; and Welling, M. 2019.  Learning likelihoods with conditional normalizing flows.  _arXiv preprint arXiv:1912.00042_.
  * Yang et al. (2019) Yang, G.; Huang, X.; Hao, Z.; Liu, M.-Y.; Belongie, S.; and Hariharan, B. 2019.  Pointflow: 3d point cloud generation with continuous normalizing flows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 4541–4550.
  * Yariv et al. (2021) Yariv, L.; Gu, J.; Kasten, Y.; and Lipman, Y. 2021.  Volume rendering of neural implicit surfaces.  _Advances in Neural Information Processing Systems_ , 34: 4805–4815.
  * Yu et al. (2022) Yu, Z.; Peng, S.; Niemeyer, M.; Sattler, T.; and Geiger, A. 2022.  MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction.  _ArXiv_ , abs/2206.00665.
  * Zhang et al. (2020) Zhang, K.; Riegler, G.; Snavely, N.; and Koltun, V. 2020.  Nerf++: Analyzing and improving neural radiance fields.  _arXiv preprint arXiv:2010.07492_.
  * Zhi et al. (2021) Zhi, S.; Laidlow, T.; Leutenegger, S.; and Davison, A. J. 2021.  In-place scene labelling and understanding with implicit scene representation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 15838–15847.


# A fully recurrent feature extraction for single channel speech enhancement

Muhammed PV Shifas, Santelli Claudio, Vassilis Tsiaras, Yannis Stylianou This
work was funded by the E.U. Horizon2020 Grant Agreement 675324, Marie
Sklodowska-Curie Innovative Training Network, ENRICH.Muhammed PV Shifas,
Vassilis Tsiaras and Yannis Stylianou are with the Speech Signal Processing
Laboratory, Department of Computer Science, University of Crete, Greece
(e-mail: shifaspv@csd.uoc.gr).Santelli Claudio is associated with the Sonova
AG, Staefa, Switzerland.

###### Abstract

Convolutional neural network (CNN) modules are widely being used to build
high-end speech enhancement neural models. However, the feature extraction
power of vanilla CNN modules has been limited by the dimensionality constraint
of the convolution kernels that are integrated – thereby, they have
limitations to adequately model the noise context information at the feature
extraction stage. To this end, adding recurrency factor into the feature
extracting CNN layers, we introduce a robust context-aware feature extraction
strategy for single-channel speech enhancement. As shown, adding recurrency
results in capturing the local statistics of noise attributes at the extracted
features level and thus, the suggested model is effective in differentiating
speech cues even at very noisy conditions. When evaluated against enhancement
models using vanilla CNN modules, in unseen noise conditions, the suggested
model with recurrency in the feature extraction layers has produced a
segmental SNR (SSNR) gain of up to 1.5 dB, an improvement of 0.4 in subjective
quality in the Mean Opinion Score scale, while the parameters to be optimized
are reduced by 25%.

###### Index Terms:

Speech enhancement, deep neural network, recurrent features extraction.

##  I Introduction

speech enhancement is concerned with improving the intelligibility and/or
overall perceptual quality of speech that has been degraded by additive noise.
With the increased use of communication devices in noisy environments, the
need for robust enhancement strategies is of paramount importance. Classical
speech enhancement techniques deal with the problem of enhancing speech
signals that have been degraded by background quasi-stationary noise. These
methods enhance speech by either subtracting the noise magnitude spectrum from
the noisy speech spectrum [1] or by modeling the noise distribution with first
and second-order statistics [2]. However, the spectral subtraction produces
distortions at very low SNRs [3]. In addition, the first and second order
statistics are not sufficient to separate non-stationary noise from the speech
and the intelligibility of the processed noisy speech tend to diminish.

Neural networks (NNs) have attracted attention for the speech enhancement task
because they are capable to learn high order statistical information and thus,
being able to represent the complex mapping function from noisy to clean
speech [4, 5]. The first applications of NNs in speech enhancements considered
fully connected neural networks (FNNs) [4, 5]. FNNs predict an output frame
from the corresponding input frame or from a small window of frames around it.
In tasks that require long receptive fields, such as separating a target
speaker from babble noise, their performance drop [6]. Alternative
architectures that are capable to model time dependencies efficiently include
convolutional neural network (CNN) layers and/or recurrent neural network
(RNN) layers. A CNN layer captures the local dependencies and a network of CNN
layers can capture longer dependencies. As a result CNNs perform better than
FNNs [7]. Also, CNNs are more memory efficient than FNNs due to their weight
sharing property. However, as the depth of the CNNs grows, the number of their
parameters also grows and this limits their applicability in low end and
embedded devices. On the other hand, RNNs are capable to model long
dependencies with only one or few stacking layers. Weninger et al. [8] used
LSTMs recurrent neural networks to pre-process and clean speech before using
it for noise-robust automatic speech recognition and they reported state of
the art word error rate. Networks that combine CNN and RNN layers were also
considered recently [9], [10]. In these networks the CNN layers specialize in
feature extraction and the RNN layers in modeling the longer dependencies.

In this work we employ a recurrent cell, called gruCNN, which combines the
feature extraction ability of CNN with the long-term memory of GRU cell [11].
The gruCNN cell was first proposed by Hartmann [12] in order to make machine
vision robust under imperfect lighting conditions and noisy environments. The
gruCNN use recurrent connections within the CNN’s convolution layers and can
learn to integrate information over time. This feature is particularly helpful
when there is low signal quality such as low-quality video, taken at night
with poor lighting conditions and with motion distortions and with occluded
objects [12]. The architecture of gruCNN cell, which was designed to integrate
images over time, fits well with the speech enhancement task since in the new
task the input is also a sequence of two dimensional spectrograms. This
architecture can be seen as an extension of the ConvLSTM-SE model presented in
[13], where the feature integration over time was done with a long short-term
memory cell type architecture with input, forget and output gates. Here, the
suggested enhancement model (gruCNN_FC-SE) utilizes gruCNN cells to learn to
extract features that are maximally relevant in every temporal context. The
suggested model is robust in modelling speech recursion with minimal
parameters. When trained and evaluated on a multi-speaker data set, under
different unseen noise conditions, gruCNN_FC–SE model provides promising
results over the traditional networks. The speech intelligibility is improved,
in the segmental SNR scale, up to 1.5 dB, across different SNR levels. At the
same time, the number of parameters is reduced by 25% compared to the
traditional recurrent model.

The rest of this paper is structured as follows. In Section II, we discuss the
suggested feature extraction strategy, and the gruCNN_FC–SE enhancement model.
The model evaluation procedure is in Section III. Section IV, includes the
results and discussion on the observations. The paper is concluded in Section
V.

##  II Recurrent Feature Extraction Technique

The problem of speech enhancement is framed on the manually extracted feature
(spectral) domain of speech, for the higher computational complexity of
temporal models. Since speech is highly auto-regressive in nature, the speech
samples generation should be modelled statistically. Let $X_{k}$ be the slice
of $k^{th}$ frequency bin values over time, from the noisy input spectrum $X$,
such that $X_{k}$ = [$x_{1},....,x_{T-1},x_{T}$]; where $T$ is the total
number of frames considered. Then, the probability of $X_{k}$ to happen can be
expressed as

$p(X_{k})=p(x_{1},....,x_{T-1},x_{T})$ (1)

with the product rule of probability, the joint distribution can be redefined
as the product of individual probabilities:

$p(X_{k})=\prod_{t=1}^{T}p(x_{t}/x_{t-1},...,x_{t-T})$ (2)

Preserving this statistical structure is essential when designing speech
enhancement models to ensure the auto-regressive nature of predictions.
Moreover, the quality of enhancement will be determined by how accurately this
dependency is being modelled. Though there may have been some inter-bin
dependencies between the spectral bins within a frame, as k varies from 1 to K
(the final bin), present modelling has not considered that for it may be
trivial compared to the temporal dependency. With this decomposition, only the
past dependencies are considered for the model to be causal.

In conventional speech enhancement neural models [9][10] the temporal
recurrency of speech was modelled by fully connected recurrent neural network
(FC-RNN) layers, like LSTM [14], employed towards the end of model
architecture. Therefore, the front-end feature extraction with CNN layers and
the recurrency modelling with FC-RNN operate independently. Such modeling,
without counting recurrency factor at the feature extraction level, leads to
the lack of qualitative features at front-end. Further, due to the inherent
fully connected nature of FC-RNN, the bin-wise recurrrency factor described in
(2) has been ignored.

In this paper, a new feature extraction strategy utilizing the local
recurrency of speech is suggested. The feature extraction layers are designed
to model the local recursion over time with kernels of fixed dimension that
trace the local statistics of previous frame to be integrated into the current
feature estimation. At a given frame index $t$, the new feature extraction
layer (gruCNN) has inputs the previous layer output $X_{t}$ – which is the
noisy speech spectrum at the beginning layer – and the feature status of the
previous frame ($H_{t-1}$). This is being processed through the nonlinear
transformations in (3) – (6) to get the feature representation of the current
frame ($H_{t}$). Whereby, the feature map $H_{t}$ encodes information from the
current frame together with the past context.

$Z_{t}=\sigma(W_{zh}*H_{t-1}+W_{zx}*X_{t})$ (3)
$R_{t}=\sigma(W_{rh}*H_{t-1}+W_{rx}*X_{t})$ (4)
$\hat{H_{t}}=\tanh(W_{hh}*(R_{t}\odot H_{t-1})+W_{hx}*X_{t})$ (5)
$H_{t}=Z_{t}\odot H_{t-1}+(1-Z_{t})\odot\hat{H_{t}}$ (6)

where the operations $*$ and $\odot$ indicate convolution and element-wise
matrix multiplication, respectively. The capitalized variables highlight the
fact that they are matrices of dimension $[K\times C]$ at every frame instant,
where $K$ and $C$ are the dimension of frequency and channel axis,
respectively. While training in this setting, the network will learn the
optimal kernels ($W_{zh}$, $W_{zx}$, $W_{rh}$, $W_{rx}$, $W_{hh}$ and
$W_{hx}$) that maximize the local bins recurrency, whereby ensure the best
features at the layers. It is worth to note that unlike fully connected RNN
cells [11], [14] that use matrix operation to model the long-term context, the
gruCNN has kernel coefficients that are shared, which in turn reduces the
parameter complexity.

Figure 1: The recurrent feature extraction of $gru$CNN_FC–SE model. Where
$H_{0}$ was set to zero valued tensor at each layer.

By layering a set of gruCNN modules one after another, the gruCNN_FC–SE
network has the final structure shown in Fig. 1. At the end of model
architecture, it is a time distributed fully connected layer which regresses
the recurrently extracted features into the enhanced spectral bins. These
predictions are combined with the noisy phase information to reconstruct back
the enhanced speech samples.

##  III Evaluation Procedure

As the primary focus is on evaluating the efficacy of suggested recurrent
feature extraction strategy over the conventional CNN architecture, the
comparing models should have had the same structural setting. To this purpose,
a model without any recurrent connections in the feature extracting CNN layers
is considered (CNN_FC–SE). Since it does not incorporate any form of temporal
recurrency at all in its modeling, the architecture is similar to Fig. 1, but
without the recurrent connections. Secondly, to quantify the benefits of
recurrency modelled precisely at the feature extraction stage, a model rather
having the front-end CNN layers followed by the standard fully connected LSTM
cell [15] (CNN_LSTM–SE) is implemented. The LSTM cell was selected instead of
GRU for they have shown better enhancement, as have been reported in the past
studies [9][10].

All the models considered have six convolutional layers (recurrent/casual)
followed by the final fully connected (recurrent/casual) layer. The
convolutional kernels of each layer are set to be of [3 $\times$ 3] size. The
filter size was selected to be of basic for swiftly isolate the performance
gain by different models. Each layer of the models has had channel depth of
256 with Parametric ReLU (PReLU) activation. Further details about the
individual layers are highlighted in TABLE I, for an input tensor of shape
$[1,161,128,1]$.

TABLE I: Layer-wise description of different models

Layer | CNN_FC-SE | CNN_LSTM-SE | gruCNN_FC-SE | Output shape
---|---|---|---|---
1 | [$3\times 3$] CNN | [$3\times 3$] CNN | [$3\times 3$] gruCNN | [1, 161, 128, 256]
2 | [$3\times 3$] CNN | [$3\times 3$] CNN | [$3\times 3$] gruCNN | [1, 161, 128, 256]
3 | [$2\times 1$] Maxpool | [$2\times 1$] Maxpool | [$2\times 1$] Maxpool | [1, 81, 128, 256]
4 | [$3\times 3$] CNN | [$3\times 3$] CNN | [$3\times 3$] gruCNN | [1, 81, 128, 256]
5 | [$3\times 3$] CNN | [$3\times 3$] CNN | [$3\times 3$] gruCNN | [1, 81, 128, 256]
6 | [$2\times 1$] Maxpool | [$2\times 1$] Maxpool | [$2\times 1$] Maxpool | [1, 41, 128, 256]
7 | [$3\times 3$] CNN | [$3\times 3$] CNN | [$3\times 3$] gruCNN | [1, 41, 128, 256]
8 | [$3\times 3$] CNN | [$3\times 3$] CNN | [$3\times 3$] gruCNN | [1, 41, 128, 256]
9 | FC | LSTM + FC | FC | [1, 161, 128, 1]

Data Set (Training and Testing): The speech set is a selection of ten British
English speakers – both male and female – from the Voice Bank speech corpus
[16], each of which has around 400 clean utterances. Eight speaker’s data were
used for training, and the remaining two (one male and one female) were
reserved for performance testing. The noisy mixtures were created manually.
The noises are from [17], which contains 20 different types of common
environmental noises. Fourteen of which were used for the training, and the
remaining six were used as the unseen noises, under which the models are
tested. For training set mixtures, each speech sample was masked by a random
training set noise at a random SNR point from [0, 5, 10, 15, 20] dB. A similar
process has been followed for the test set, but with the unseen noises at
unseen SNR points of [2.5, 12.5, 22.5] dB.

Although the original speech were sampled at 48kHz, it was down-sampled to 16
kHz for our experiment as in [18][19]. The 16kHz sampled signals were framed
into 20 ms frames with 10 ms overlap. The frames were Fourier transformed into
320 points. The log-power spectra feature is the domain on which the
enhancement task is modeled [20]. Therefore, the frequency dimension of input
spectrum is halved to 161 points, due to the spectral symmetry.

Model Training: All the comparing models are trained in an end-to-end mode,
where the losses are computed directly between the magnitudes of predicted
($\hat{Y}(k,t)$) and target ($Y(k,t)$) STFT components. For each noisy-clean
training set pair $(X,Y)$, the model parameters are optimized by minimizing
the mean square error (MSE) objective function.

$L_{X,Y}=\frac{1}{T\times
K}\sum_{t=1,f=1}^{t=T,f=K}(|Y(k,t)|-|\hat{Y}(k,t)|)^{2}$ (7)

where K denotes the dimension of frequency axis that is 161, and the variable
T is the number of time frames recurrently generated in the training process;
which has been set to T = 128. The T value for testing varies based on the
input signal duration for the recurrency is being modeled over the temporal
axis. The loss was minimized by the Adam optimizer [21] with an exponentially
decaying learning rate method with learning rate = 0.001, decay steps = 20,000
and decay rate = 0.99.

For objective evaluation of processed samples, the perceptual evaluation of
speech quality (PESQ) metric (ITU P.862.2) [22] that measures the quality, and
the short-time objective intelligibility (STOI) [23] that measures the
intelligibility, are considered. The composite quality of the model
predictions (COVL) has also been measured [17], which reports a compound count
of the noise reduction and speech restoration. Besides, the SNR
intelligibility gain through model processing is measured by the segmental SNR
(SSNR) score [17]. Subjectively, the quality of enhanced samples were measured
by the mean opinion score (MOS). In total, 20 participants (non-native English
speakers) listened to and assigned the individual perceptual score based on
the noise artifacts present, in a scale of 0 – 5 (0 – very annoying artifacts
, 5 – no artifacts at all).

(a)

(b)

Figure 2: Model enhancement under construction (upper panel) and street (lower
panel) noise

##  IV Results and Discussion

The objective scores averaged over the test samples at each noise condition
are displayed in TABLE II. Along with the processing types, the scores of
unprocessed noisy speech are also provided to better understand the relative
processing gain. Compared to CNN_FC–SE architecture, which does not
incorporate any form of recurrency described in (2), the suggested
gruCNN_FC–SE model with recurrency modelled in the feature extraction layers
has distinctly outperformed on all the metrics. This gain is almost consistent
across the noise conditions. With the inclusion of global recurrency by
CNN_LSTM–SE, the performance has improved over CNN_FC–SE. This broadly conveys
the benefits that can be achieved through temporal inclusive modeling of
speech.

TABLE II: Objective measures for unseen noise conditions

Noise level | Metric | Noisy | CNN_FC-SE | CNN_LSTM-SE | gruCNN_FC-SE
---|---|---|---|---|---
2.5 dB | PESQ | 1.20 | 1.41 | 1.51 | 1.57
STOI | 0.68 | 0.71 | 0.72 | 0.74
COVL | 1.58 | 1.96 | 2.15 | 2.22
SSNR | \- 3.63 | 2.39 | 3.20 | 3.94
12.5 dB | PESQ | 1.49 | 1.87 | 2.01 | 2.08
STOI | 0.77 | 0.78 | 0.79 | 0.80
COVL | 2.11 | 2.59 | 2.74 | 2.83
SSNR | 3.24 | 7.61 | 7.85 | 8.96
22.5 dB | PESQ | 2.27 | 2.47 | 2.58 | 2.66
STOI | 0.85 | 0.83 | 0.84 | 0.85
COVL | 3.05 | 3.20 | 3.30 | 3.41
SSNR | 12.26 | 11.21 | 11.14 | 12.83

When comparing the two recurrent models, the proposed gruCNN_FC–SE, that is
concerned of the bin-wise recurreny factor, has shown better enhancement over
CNN_LSTM–SE. Even at the higher SNR point of 22.5dB, where the noise
attributes are expected to be mild, gruCNN_FC–SE model elicited noticeable
enhancement, showing an SSNR intelligibility gain of up to 1.5 dB over the
other methods. This gain must be attributed to the qualitative restoration of
speech components with the suggested feature extraction strategy.

Regarding the consistency of model predictions in different noise types, the
model enhancements under the two unseen noise conditions are plotted in Fig.
2. The upper panel shows construction noise (type–1) while the lower panel
refers to street noise (type–2). Since type–1 noise is quite stationary and
has the spectral energy that is distributed uniformly in a very wide frequency
band (0 - 3 kHz), it is straightforward for a network to get a frequency
smoothed estimate of the noise statistics. While type–2 noise (street) are
highly localized at the lower band (0 - 0.5 kHz) of the spectrum (marked by a
straight line in Fig. 2). Unless the model looks into the local statistics of
the spectrum, these noise activities could easily be miss-classified as speech
events. We suggest that this explains the performance of CNN_FC–SE and
CNN_LSTM–SE, whereas gruCNN_FC–SE seems to be successful in disentangling out
the noise activities by exploiting the local patterns.

The subjective scores of different models are displayed in TABLE III. In line
with the objective measures, the suggested gruCNN_FC–SE model is ranked closer
to the clean speech with a score of 3.16 on the five point scale, while there
was not any statistically observable difference between the scores of the
other two methods.

Pragmatically, the performance gain of a neural model could be argued by the
additional parameters that is floated into the modeling. To this end, the
parameter counts of different models are shown in TABLE IV. CNN_FC–SE is the
least complex among the models and indeed its performance has been much lower
than the other two models. On the other hand, the suggested gruCNN_FC–SE
produces far better enhancement with only 75% parameters of the CNN_LSTM–SE.
This reduction in complexity is from the replacement of the fully-connected
LSTM cell with the fixed kernels of gruCNN to model the temporal flow. All of
which indicates the potentiality to have it implemented on computationally
constraint applications, like hearing aid. A Tensorflow implementation and
enhanced samples from the model are provided at
111https://www.csd.uoc.gr/~shifaspv/IEEE_Letter-demo
222https://github.com/shifaspv/gruCNN-speech-enhancement-tensorflow.

TABLE III: Mean opinion score (MOS) with standard error

Metric | Noisy | CNN_FC-SE | CNN_LSTM-SE | gruCNN_FC-SE | Clean
---|---|---|---|---|---
MOS | 2.01$\pm$0.97 | 2.75$\pm$0.92 | 2.77$\pm$0.89 | 3.16$\pm$0.92 | 4.86$\pm$0.42

TABLE IV: Model parameters count in Million (M)

Metric | CNN_FC-SE | CNN_LSTM-SE | gruCNN_FC-SE
---|---|---|---
Parameters | 11.13M | 36.10M | 27.22M

##  V Conclusion

In this letter, we presented a new concept of recurrent feature extraction
that is found to be beneficial for single-channel speech enhancement. In
contrast to the traditional independent modelling of feature extraction and
temporal recurrency, the suggested module with recurrent connections inside is
proved to be robust to unseen noise conditions and efficient. The subjective
and objective evaluations have confirmed the benefits that the suggested
recurrent feature extraction technique elicited. While at the same time, the
parameter complexity of the suggested model is reduced by 25% compared to the
traditional model.

## References

  * [1] S. Boll, “Suppression of acoustic noise in speech using spectral subtraction,” _IEEE Transactions on acoustics, speech, and signal processing_ , vol. 27, no. 2, pp. 113–120, 1979.
  * [2] Y. Ephraim, “Statistical-model-based speech enhancement systems,” _Proceedings of the IEEE_ , vol. 80, no. 10, pp. 1526–1555, 1992.
  * [3] N. W. Evans, J. S. Mason, W.-M. Liu, and B. Fauve, “An assessment on the fundamental limitations of spectral subtraction,” in _2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings_ , vol. 1. IEEE, 2006, pp. I–I.
  * [4] Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, “An experimental study on speech enhancement based on deep neural networks,” _IEEE Signal processing letters_ , vol. 21, no. 1, pp. 65–68, 2014.
  * [5] X. Lu, Y. Tsao, S. Matsuda, and C. Hori, “Speech enhancement based on deep denoising autoencoder.” in _Interspeech_ , 2013, pp. 436–440.
  * [6] K. Tan and D. Wang, “A convolutional recurrent neural network for real-time speech enhancement.” in _Interspeech_ , 2018, pp. 3229–3233.
  * [7] S. R. Park and J. Lee, “A fully convolutional neural network for speech enhancement,” _arXiv preprint arXiv:1609.07132_ , 2016.
  * [8] F. Weninger, H. Erdogan, S. Watanabe, E. Vincent, J. Le Roux, J. R. Hershey, and B. Schuller, “Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR,” in _International Conference on Latent Variable Analysis and Signal Separation_. Springer, 2015, pp. 91–99.
  * [9] H. Zhao, S. Zarar, I. Tashev, and C.-H. Lee, “Convolutional-recurrent neural networks for speech enhancement,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 2401–2405.
  * [10] G. Naithani, T. Barker, G. Parascandolo, L. Bramsl, N. H. Pontoppidan, T. Virtanen _et al._ , “Low latency sound source separation using convolutional recurrent neural networks,” in _2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_. IEEE, 2017, pp. 71–75.
  * [11] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _arXiv preprint arXiv:1406.1078_ , 2014.
  * [12] T. S. Hartmann, “Seeing in the dark with recurrent convolutional neural networks,” _arXiv preprint arXiv:1811.08537_ , 2018.
  * [13] M. S. PV, C. Santelli, and Y. Stylianou, “Towards neural-based single channel speech enhancement for hearing aids,” _In International Conference on Acoustics (ICA) Proceedings_ , pp. 5745–5748, 2019.
  * [14] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [15] F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with lstm,” 1999.
  * [16] C. Veaux, J. Yamagishi, and S. King, “The voice bank corpus: Design, collection and data analysis of a large regional accent speech database,” in _2013 International Conference Oriental COCOSDA held jointly with 2013 Conference on Asian Spoken Language Research and Evaluation (O-COCOSDA/CASLRE)_. IEEE, 2013, pp. 1–4.
  * [17] P. C. Loizou, _Speech enhancement: theory and practice_. CRC press, 2013.
  * [18] P. Muhammed Shifas, N. Adiga, V. Tsiaras, and Y. Stylianou, “A non-causal fftnet architecture for speech enhancement,” _Proc. Interspeech 2019_ , pp. 1826–1830, 2019.
  * [19] D. Rethage, J. Pons, and X. Serra, “A wavenet for speech denoising,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5069–5073.
  * [20] M. Portnoff, “Time-frequency representation of digital signals and systems based on short-time fourier analysis,” _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , vol. 28, no. 1, pp. 55–69, 1980.
  * [21] P. Kingma and J. Ba, “Adam: A method for stochastic optimization, arxiv (2014),” _arXiv preprint arXiv:1412.6980_ , vol. 106, 2015.
  * [22] Y. Hu and P. C. Loizou, “Evaluation of objective quality measures for speech enhancement,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 16, no. 1, pp. 229–238, 2007.
  * [23] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “An algorithm for intelligibility prediction of time–frequency weighted noisy speech,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 7, pp. 2125–2136, 2011.