††thanks: Contributed equally

# Emergence of Accurate Atomic Energies from Machine Learned Noble Gas
Potentials

Frank Uhlig  Institute for Computational Physics, University of Stuttgart,
Stuttgart, Germany  Samuel Tovey* Institute for Computational Physics,
University of Stuttgart, Stuttgart, Germany  Christian Holm<EMAIL_ADDRESS>stuttgart.de Institute for Computational Physics, University of Stuttgart,
Stuttgart, Germany

###### Abstract

The quantum theory of atoms in molecules (QTAIM) gives access to well-defined
local atomic energies. Due to their locality, these energies are potentially
interesting in fitting atomistic machine learning models as they inform about
physically relevant properties. However, computationally, quantum-mechanically
accurate local energies are notoriously difficult to obtain for large systems.
Here, we show that by employing semi-empirical correlations between different
components of the total energy, we can obtain well-defined local energies at a
moderate cost. We employ this methodology to investigate energetics in noble
liquids or argon, krypton, and their mixture. Instead of using these local
energies to fit atomistic models, we show how well these local energies are
reproduced by machine-learned models trained on the total energies. The
results of our investigation suggest that smaller neural networks, trained
only on the total energy of an atomistic system, are more likely to reproduce
the underlying local energy partitioning faithfully than larger networks.
Furthermore, we demonstrate that networks more capable of this energy
decomposition are, in turn, capable of transferring to previously unseen
systems. Our results are a step towards understanding how much physics can be
learned by neural networks and where this can be applied, particularly how a
better understanding of physics aids in the transferability of these neural
networks.

## Introduction

Machine learning (ML) has impacted many aspects of modern society. From
artificial intelligence, smart homes, and social media to an industrial
context, e.g., predictive maintenance or medical diagnosis, ML methods have
changed how we interact with computers and our environment. In recent years,
ML methods have been exploited in the development of inter-atomic potentials
for liquids [1, 2], solids [3, 4, 5] and interfaces [6, 7] with almost ab
initio accuracy but at a highly reduced cost for running associated molecular
dynamics (MD) simulations. Many reviews have been dedicated to ML and its
various applications to potential energy surfaces [8],
computational/theoretical chemistry [9, 10], and cheminformatics [11, 12]. ML
can interpolate between training data without knowledge of the differential
equations describing the underlying physics, thus offering considerable
flexibility at the cost of interpretability in the final model. The most
common approaches to developing machine-learned potentials include directly
fitting the total energy of a system [8], splitting the total energy into
local atomic energies [13, 14], or using ML to produce parameters for
physically motivated potentials from which the energy of a system can be
calculated [15, 16]. Splitting the total energy into a sum of local atomic
energies has important features: it offers accuracy, flexibility, and
scalability if models are fitted only for the involved atomic species. This is
achieved by representing atomic environments centred on individual atom sites
as feature vectors. Different variations of these feature vectors, commonly
referred to as _descriptors_ , have been developed with varying success and
accuracy [13, 14, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]. However, it is
unclear whether assuming that a system’s total energy can be split into a
simple sum of atomic energies is valid. It can be argued that, due to the
locality of quantum mechanics, the energy of an atom is mainly dependent on
its local environment [21]. Indeed, the _quantum theory of atoms in molecules_
(QTAIM) allows us to dissect the total quantum-mechanical energy of any system
into well-defined atomic energies at stationary states of the atomic
configuration [27, 28]. Further development has enabled the calculation of
atomic energies for non-stationary states by directly calculating them from
density matrices of the system. [29] This has been exploited to train machine
learning models to predict these local atomic energies. [30, 31] One can then
accurately predict atomic energies and effective atomic charges[15] that can
be used to describe the interaction of atoms and molecules. One of the most
challenging aspects of machine learning is to provide transferable models that
can be trained on a small set of reference data and applied to a more
extensive set with comparable accuracy [32]. In the field of computer vision,
some transferability is assumed in the context of transfer learning, i.e.,
refining pre-trained models for a specific task. Similarly, models for mixed
component systems have been systematically fitted on the pure components first
and only then refined for the full systems [33]. Recent efforts regarding the
prediction electron densities show that transferability can be achieved by
using atom-centered symmetry functions as descriptors [34]. To achieve
transferability and scalability, it is crucial that the employed descriptor
can suitably accommodate the inclusion of different atomic species without
growing in a computationally intractable manner. Models trained on large
subsets of with vast chemical diversity offer good performance for various
tasks, from predicting energetics to identifying chemical concepts. [35]
Current developments focus on generating transferable descriptors that offer a
reduction in scaling with respect to the number of involved chemical species,
and thus, training data [36, 37, 38, 39, 16, 40]. Another approach to enforce
transferability is to reproduce the underlying physics more accurately. This
could mean ensuring that the predicted local energies correspond to a physical
theory in the above context. Here, to the best of our knowledge, for the first
time, we report a direct comparison between physically meaningful local
energies and energies obtained with different machine-learning algorithms for
bulk liquids. This work investigates under which conditions machine learning
models reproduce the underlying energetics and how much data is needed to
learn the corresponding physics. We argue that smaller networks are forced to
learn more fundamental representations of the underlying data they fit,
sometimes leading to the emergence of learned physics.

## Theory

The quantum mechanical virial theorem establishes a connection between the
Laplacian of the electron density $\rho({\bf r})$, the kinetic energy density
$\mathcal{T}({\bf r})$, and the total virial $\mathcal{V({\bf r})}$ [27]:

$\left(\frac{\hbar^{2}}{4m}\right)\nabla^{2}\rho({\bf r})=2\mathcal{T}({\bf
r})+\mathcal{V}({\bf r}).$ (1)

If integrated over the whole system, e.g., an isolated molecule, the left-hand
side of eq. 1 disappears and relates the kinetic energy to the total virial.
The same holds true for zero-flux surfaces $S({\bf r})$ of the electron
density, defined by means of Gauss’ divergence theorem using the gradient of
the electron density and the normal vector, $\bf{n}(\bf r)$ as

$\nabla\rho({\bf r})\cdot\bf{n}({\bf r})=0$ (2)

which, within the framework of QTAIM, determines atomic regions, $\Omega$.
Hence, for QTAIM, we can also define a relationship between the local virial
and the kinetic energy. At a stationary state with respect to the nuclei, the
virial becomes the local potential energy of the $i^{\text{th}}$ atom:

$-2\mathcal{T}(\Omega_{i})=\mathcal{V}(\Omega_{i})\overset{\text{stat.
state}}{=}V(\Omega_{i})$ (3)

and the total energy, $E$ can be obtained as a sum of local energies of atoms
in molecules by:

$E=\sum_{i}V(\Omega_{i}).$ (4)

With eq. 4, or modified versions of it [29], one has access to well-defined
local atomic energies that can be used to fit machine-learned models. [30, 31]
A disadvantage of eq. 4 is that the connection between potential and kinetic
energy only exists as such for stationary states. Another caveat is that the
kinetic energy density is difficult to obtain at an adequate resolution for
large-scale analysis. Due to the involved gradient of the wavefunction, very
fine grids must be used, which quickly become impractical to handle. However,
one might expect that certain correlations between energy components still
exist, even for structures sampled at finite temperature. If so, it would
unlock new methods for validating and even training machine learned
interatomic potentials.

In typical machine-learning approaches for the fitting of machine-learned
inter-atomic potentials, a model is fit to reproduce the same total total
energy of the system by first computing the local energies of so-called atomic
environments, or descriptors, $\vec{g}$, and then summing over them by

$\centering E=\sum\limits_{i}f_{\theta}(\vec{g}_{i}),\@add@centering$ (5)

where $f_{\theta}$ is a neural network parameterised by the set $\theta$. The
parameters used in constructing these descriptors will define the environment
in which the local energy is computed. While many descriptors exist, this work
uses the invariant 4D-Bispectrum descriptor in all model training [20]. These
descriptors are fixed, meaning that they do not evolve during training, and
they include a species scaling to differentiate different atom types. The
network is updated via gradient descent using a loss function of the form

$\centering\mathcal{L}=\left(E-\sum\limits_{i}f_{\theta}(\vec{g}_{i})\right)^{2}.\@add@centering$
(6)

Under this construction, there is much ambiguity about how atomic energies are
assigned to specific atoms, i.e., which partitioning scheme is learned by the
network. We argue that the correct partitioning of energies into their atomic
contributions will result in better models as the underlying physics is better
represented. This can be considered a feature learning problem, as the
networks must learn how to correctly represent specific atomic environments in
the latent space to be transferable to unseen local environments, something
only possible if the partitioning is done physically.

Feature learning, or representation learning as it is often referred to,
describes the ability of a neural network to learn a fundamental
representation of a data manifold rather than simply fitting to training
points [41]. For neural networks, this means that features in the data, for
example certain shapes in the case of images, have been learned during
propagation through layers. In the context of fitting machine-learned inter-
atomic potentials, feature learning can refer to both the learning of a
descriptor internally within the neural network or, as is discussed here,
learning the most physical distribution of local energies, that is, the
distribution closest to the physically motivated QTAIM method. It has been
shown that feature learning is fundamental to the success of both pre-training
and transfer learning paradigms [42, 43]. However, not all neural networks are
capable of feature learning. In their 2021 report, Yang and Hu demonstrated
that large neural networks’ ability to perform feature learning strongly
depends on their initialisation. Often, they are incapable of feature
learning, becoming purely regressive machines; that is to say, they learn
purely the data they are trained on and not features connecting the points
[44].

It is not difficult to argue that learning a fundamental physical principle
would amount to feature learning for a neural network or, to say it
differently, a neural network with knowledge of physics has learned the
correct features to describe a problem. This point is often leveraged in
developing physics-informed networks, which have been shown to lead to better
transferrability [45, 46, 47]. However, it is not well studied whether a
neural network, constrained in its size and, therefore, in the complexity of
features that can be learned, will, in turn, develop knowledge of physics.
This work examines this question by determining under what conditions a neural
network can learn the correct energy decomposition in an atomistic system.

## Local energy computation

Before identifying the role of architecture and the emergence of correct
energy partitioning, we demonstrate that it is possible to apply the QTAIM
method to the extraction of local energies in the non-stationary states
produced in our noble-liquid DFT simulations. To do so, a correlation must be
identified between the classical electrostatic contribution to the system’s
total energy and the other Hamiltonian components. Such a correlation would
allow us extract local energy information on much larger grids, thereby making
the computation accessible to large, periodic systems. To investigate if this
correlation exists, we performed ab initio molecular dynamics simulations
(AIMD) based on density functional theory (DFT) for three different bulk
liquids, Ar(108), Kr(108), and Kr(54)Ar(54) at 120 K. AIMD simulations used
Kohn-Sham density functional theory (KS-DFT) and were carried out with CP2K
[48, 49]. The exchange-correlation energy was approximated using a van-der-
Waals density functional [50, 51]. The Kohn-Sham one-particle wavefunctions
were expanded into an atom-centered double-$\zeta$ basis set optimized for
dense liquids[52]. The electron density was expanded in an auxiliary plane-
wave basis set with a kinetic energy cutoff of 600 Ry. Initial configurations
were obtained from molecular dynamics simulations using suitable noble liquid
potentials. Newton’s equations of motion were integrated using a timestep of 2
fs, and the temperature of the system was kept constant at 120 K using a Nosé-
Hoover chain thermostat [53] (chain length: 3, time constant: 100 fs). This
work focuses on noble liquid mixtures as a proxy for more complicated liquids,
specifically liquid argon, krypton and their equimolar mixture.

Figure 1: Total energy minus classical electrostatic energy versus the
classical electrostatic energy in pure Ar.

Figure 1 outlines the results of this study for the liquid argon (see Figures
5 and 6 in SI for the mixture and pure Kr respectively.). In these systems, we
observe a strong correlation between the classical electrostatic energy and
the sum of all other contributions to the total energy, i.e., kinetic energy
and exchange-correlation energy. The correlation between these variables can
be fit linearly, and while the values for slope and y-intercept are system-
specific, they can be attributed to physical interpretations. For the pure
systems, the intercepts can be related to the classical electrostatic energy
of the isolated atoms in vacuo. This is not a general characteristic as it
does not apply to the mixed system. Nevertheless, this study demonstrates a
way to relate the electrostatic energy effectively to the remaining energy
components. The fit’s root mean squared error is approximately 0.3 meV/atom,
the same order as a typically achieved fit accuracy in machine learned
potentials. We can largely ignore this source of uncertainty as the error in
the machine learning model fit will outweigh it. Using these results, we can
dissect the classical electrostatic interaction between all charged particles
(electron density and protons) as it is readily available and accurately
reproduced on grids of moderate resolution and can therefore be used for
large, 3D-periodic systems. The dissection follows in accordance with eq. 3
and is considered a robust numerical procedure[54]. The electrostatic energy
of an atom $E_{\Omega}$ is calculated using the electron density $\rho$ and
the Hartree potential $V_{\mathrm{H}}$ as
$E_{\Omega}=\frac{1}{2}\int_{\Omega}\rho V_{\mathrm{H}}$. We will refer to
these energies as local DFT energies, $E_{\mathrm{loc}}^{\mathrm{DFT}}$.

## Local energy fitting

With the ability to produce reference data for the local energies, we can turn
our attention to understanding under which conditions neural networks, trained
purely on the total energy of ab initio simulations, can accurately reproduce
the atomic energies computed using QTAIM. To do so, neural networks of various
architectures were trained on pure argon and krypton systems and the ArKr
mixture using the total energy of each configuration in the loss function. Due
to the similarity of the results, only the results from a single architecture
are shown in the manuscript, namely, a single layer network with hyperbolic
tangent activation. Other architectures are displayed in the appendix. Neural
networks utilised hyperbolic-tangent and linear activation functions and were
trained using a mean-squared-error loss function. In all cases, the
correlation score for the total energy, the quantity on which the network was
trained, remains high across the architecture space (see appendix Figure 7).
For small models, the initialisation variance results in large error bars,
which are saturated for the wider and deeper models.

As all model architecture are capable of fitting the total energies, the
interest now lies in seeing how the change in architecture will impact the
ability of the models to reproduce the local QTAIM energies of the systems
correctly. To do so, Pearson correlation coefficients are computed on the
local energy test data produced from the DFT simulations. This is data that
the neural networks have never seen during their training Figure 2 displays
these correlation coefficients between the predicted and DFT-produced local
energies for the pure and mixed systems (see appendix Figure 8 for all
architectures).

0100200300400500Layer Width0.40.60.81.0R2ArKrMx Figure 2: Correlation score
for the local energy of each system, the argon (Ar), krypton (Kr), and the
50-50 mixture (Mx).

Studying these local correlation plots, it is clear that larger neural
networks assign the local energies more arbitrarily and show little
correlation with the underlying physics. On the other hand, smaller networks
maintain a reasonable correlation between the predicted and actual
partitioning of the local energy values. It is clear, however, that the models
trained on the mixtures struggle more to achieve this local energy
decomposition, with these networks achieving lower maximum scores and dropping
off faster than their pure system counterparts.

## Transferability

Given the emergence of correct local energies, it is of interest to see if
this emergent structure learning can be used in transfer studies. We
investigate this by testing if models more capable of local energy
decomposition are also capable of being transferred, without re-training, to
previously unseen configurations. The models trained on the ArKr mixture are
used to compute the total and local energies of the pure Ar and Kr systems
whereas the models trained on the pure systems are used to compute the
energies in the mixed systems. This is done on the same test data as in the
previous section using the pair style command in the LAMMPS simulation engine
where atomic environments centered on an Ar atom will use the pure Ar model
for the force computation and the pure Kr models handle the Kr centered
environments. Figure 3 outlines the results of this study on the total energy
computations for all architectures and systems.

0100200300400500Layer Width0.00.20.40.60.81.0R2ArKrMx Figure 3: Correlation
score for the total energy of each system computed using the transferred
models, the argon (Ar), krypton (Kr), and the 50-50 mixture (Mx).

As is to be expected, the models fail in their predictions early in all cases.
It is, however, interesting that the smaller models can retain a high
correlation score for the transferred systems. This suggests that these models
represent the underlying physics of the problem better. Another point of note
is the ability of the Tanh activation function models to achieve more
significant correlation scores for slightly longer than the linear models.
Moving away from the total energies, Figure 4 shows the results of the
transfer experiment on the local energy predictions.

0100200300400500Layer Width0.00.20.40.60.81.0R2ArKrMx Figure 4: Correlation
score for the local energy of each system computed using the transferred
models, the argon (Ar), krypton (Kr), and the 50-50 mixture (Mx).

These plots closely resemble those of the total energies in the gradual
degradation of performance with increasing network width. In typical machine
learning theory, the networks addressed here would not be considered purely
regressive as they are not approaching an infinite width limit. However, even
amongst the feature learning limit, there appears to be a discrepancy between
architectures. We argue that this comes down to the number of degrees of
freedom each of these neural networks has and how they can use these to
distribute information. Namely, the networks that are required to find
solutions with less degrees of freedom typically learn more physical
representations. Such an outcome is supported by the data and resembles the
principle of Occam’s razor: the simplest solution is often the correct one.

## Conclusion

The decomposition of total energy into atomic contributions has been
significantly utilised in machine-learned inter-atomic potentials. However,
whether these local energies can be considered physically meaningful requires
clarification. In this investigation, we have restructured this problem as one
of physics-aware feature learning in neural networks. By considering local
energies obtained from the QTAIM as ground truth, we have argued that correct
feature learning of a neural network involves learning these physically
derived local energies given the system’s total energy. To this end, studies
of pure liquid argon and krypton and their mixture have been performed wherein
DFT-MD was performed to generate ab initio data. We have shown that QTAIM can
be applied to bulk liquid systems and computed these local energies. Neural
network models were then fitted to the total energies of the ab initio
simulations for different architectures before the local decomposition was
compared with theoretical values. We find that neural networks of smaller
width (12-100 nodes) display improved decomposition over larger models. These
results align with those of feature learning in the field of learning theory,
wherein it is understood that smaller neural networks are capable of so-called
feature learning, i.e., extraction of features in data, while larger models
are purely regressive. We further highlight that even among feature learning
networks, those with fewer degrees of freedom learn more physically relevant
representations, thus resulting in more physically correct energies. We
conclude that smaller networks are faster to train and more easily deployed
whilst retaining the accuracy of larger networks. They also learn a more
physical representation of atomic decomposition and are thus favorable from a
physics perspective. Finally, we have shown that the models capable of
computing accurate local energies also showed the ability to accurately
transfer to previously unseen configurations. While it is expected that this
transferability is predominantly a geometric one, future work should identify
the limits to which this can be pushed in chemical space.

###### Acknowledgements.

The authors acknowledge financial support from the German Funding Agency
(Deutsche Forschungsgemeinschaft DFG) under Germany’s Excellence Strategy EXC
2075-390740016. This work was supported by SPP 2363- ”Utilization and
Development of Machine Learning for Molecular Applications – Molecular Machine
Learning.” Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation), Project-No 497249646. S.T would like to thank David Tovey for his
thorough review of and comments on the manuscript. The authors would like to
thank Anand Narayanan Krishnamoorthy and Matthias Bauer for their helpful
input on early versions of this study involving Gaussian process-based models.
All authors would like to acknowledge and thank David Beyer for his detailed
reading and editing of the manuscript.

## References

  * [1] Samuel Tovey, Anand Narayanan Krishnamoorthy, Ganesh Sivaraman, Jicheng Guo, Chris Benmore, Andreas Heuer, and Christian Holm.  Dft accurate interatomic potential for molten nacl from machine learning.  J Phys Chem C, 124:25760–25768, 2020.
  * [2] János Daru, Harald Forbert, Jörg Behler, and Dominik Marx.  Coupled cluster molecular dynamics of condensed phase systems enabled by machine learning potentials: Liquid water benchmark.  Phys. Rev. Lett., 129:226001, Nov 2022.
  * [3] Ganesh Sivaraman, Anand Narayanan Krishnamoorthy, Matthias Baur, Christian Holm, Gabor Csányi Marius Stan, Chris Benmore, and Álvaro Vázquez-Mayagoitia.  Machine-learned interatomic potentials by active learning: amorphous and liquid hafnium dioxide.  npj Computational Materials, 6:1–8, 2020.
  * [4] S.S. Shishvan, G. Csányi, and V.S. Deshpande.  Strain rate sensitivity of the hydrogen embrittlement of ferritic steels.  Acta Materialia, 257:119173, 2023.
  * [5] Xia Liang, Johan Klarbring, William J. Baldwin, Zhenzhu Li, Gábor Csányi, and Aron Walsh.  Structural dynamics descriptors for metal halide perovskites.  The Journal of Physical Chemistry C, 127(38):19141–19151, 2023\.
  * [6] Christoph Schran, Fabian L. Thiemann https, Patrick Rowe, and Angelos Michaelides.  Machine learning potentials for complex aqueous systems made simple.  PNAS, 118, September 2021.
  * [7] Molpeceres, G., Zaverkin, V., Furuya, K., Aikawa, Y., and Kästner, J.  Reaction dynamics on amorphous solid water surfaces using interatomic machine-learned potentials - microscopic energy partition revealed from the p \+ h → ph reaction.  A&A, 673:A51, 2023.
  * [8] Chris M. Handley and Paul L. A. Popelier.  Potential energy surfaces fitted by artificial neural networks.  The Journal of Physical Chemistry A, 114(10):3371–3383, 2010.  PMID: 20131763.
  * [9] Garrett B. Goh, Nathan O. Hodas, and Abhinav Vishnu.  Deep learning for computational chemistry.  Journal of Computational Chemistry, 38(16):1291–1307, 2017.
  * [10] Matthias Rupp, O. Anatole von Lilienfeld, and Kieron Burke.  Guest editorial: Special topic on data-enabled theoretical chemistry.  The Journal of Chemical Physics, 148(24):241401, 2018.
  * [11] John B. O. Mitchell.  Machine learning methods in chemoinformatics.  Wiley Interdisciplinary Reviews: Computational Molecular Science, 4(5):468–481, 2014.
  * [12] Yu-Chen Lo, Stefano E. Rensi, Wen Torng, and Russ B. Altman.  Machine learning in chemoinformatics and drug discovery.  Drug Discovery Today, 23(8):1538 – 1546, 2018.
  * [13] Jörg Behler and Michele Parrinello.  Generalized neural-network representation of high-dimensional potential-energy surfaces.  Phys. Rev. Lett., 98:146401, Apr 2007.
  * [14] Albert P. Bartók, Mike C. Payne, Risi Kondor, and Gábor Csányi.  Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons.  Phys. Rev. Lett., 104:136403, Apr 2010.
  * [15] Timothy L. Fletcher and Paul L. A. Popelier.  Fflux: Transferability of polarizable machine-learned electrostatics in peptide chains.  Journal of Computational Chemistry, 38(13):1005–1014, 2017.
  * [16] Tristan Bereau, Robert A. DiStasio, Alexandre Tkatchenko, and O. Anatole von Lilienfeld.  Non-covalent interactions across organic and biological subsets of chemical space: Physics-based potentials parametrized from machine learning.  The Journal of Chemical Physics, 148(24):241706, 2018.
  * [17] Jörg Behler.  Atom-centered symmetry functions for constructing high-dimensional neural network potentials.  The Journal of Chemical Physics, 134(7):074106, 2011.
  * [18] Nongnuch Artrith, Tobias Morawietz, and Jörg Behler.  High-dimensional neural-network potentials for multicomponent systems: Applications to zinc oxide.  Phys. Rev. B, 83:153101, Apr 2011.
  * [19] Matthias Rupp, Alexandre Tkatchenko, Klaus-Robert Müller, and O. Anatole von Lilienfeld.  Fast and accurate modeling of molecular atomization energies with machine learning.  Phys. Rev. Lett., 108:058301, Jan 2012.
  * [20] Albert P. Bartók, Risi Kondor, and Gábor Csányi.  On representing chemical environments.  Physical Review B, 87:184115, 2013.
  * [21] A.P. Thompson, L.P. Swiler, C.R. Trott, S.M. Foiles, and G.J. Tucker.  Spectral neighbor analysis method for automated generation of quantum-accurate interatomic potentials.  Journal of Computational Physics, 285:316 – 330, 2015.
  * [22] Katja Hansen, Franziska Biegler, Raghunathan Ramakrishnan, Wiktor Pronobis, O. Anatole von Lilienfeld, Klaus-Robert Müller, and Alexandre Tkatchenko.  Machine learning predictions of molecular properties: Accurate many-body potentials and nonlocality in chemical space.  The Journal of Physical Chemistry Letters, 6(12):2326–2331, 2015\.  PMID: 26113956.
  * [23] O. Anatole von Lilienfeld, Raghunathan Ramakrishnan, Matthias Rupp, and Aaron Knoll.  Fourier series of atomic radial distribution functions: A molecular fingerprint for machine learning models of quantum chemical properties.  International Journal of Quantum Chemistry, 115(16):1084–1093, 2015\.
  * [24] Jan Jenke, Aparna P. A. Subramanyam, Marc Densow, Thomas Hammerschmidt, David G. Pettifor, and Ralf Drautz.  Electronic structure based descriptor for characterizing local atomic environments.  Phys. Rev. B, 98:144102, Oct 2018.
  * [25] Behnam Parsaeifard and Stefan Goedecker.  Manifolds of quasi-constant soap and acsf fingerprints and the resulting failure to machine learn four-body interactions.  Journal of Chemical Physics, 156:034302, 2022.
  * [26] V. Zaverkin and J. Kästner.  Gaussian moments as physically inspired molecular descriptors for accurate and scalable machine learning potentials.  Journal of Chemical Theory and Computation, 16(8):5410–5421, 2020\.  PMID: 32672968.
  * [27] R. F. W. Bader.  Atoms in molecules.  Accounts of Chemical Research, 18(1):9–15, 1985.
  * [28] R.F.W. Bader.  Atoms in Molecules: A Quantum Theory.  International Ser. of Monogr. on Chem. Clarendon Press, 1994.
  * [29] P. L. A. Popelier and D. S. Kosov.  Atom–atom partitioning of intramolecular and intermolecular coulomb energy.  The Journal of Chemical Physics, 114(15):6539–6547, 2001.
  * [30] Paul L. A. Popelier.  Qctff: On the construction of a novel protein force field.  International Journal of Quantum Chemistry, 115(16):1005–1011, 2015\.
  * [31] Paul L A Popelier.  Molecular simulation by knowledgeable quantum atoms.  Physica Scripta, 91(3):033007, 2016.
  * [32] Ilyes Batatia, Philipp Benner, Yuan Chiang, Alin M. Elena, Dávid P. Kovács, Janosh Riebesell, Xavier R. Advincula, Mark Asta, William J. Baldwin, Noam Bernstein, Arghya Bhowmik, Samuel M. Blau, Vlad Cărare, James P. Darby, Sandip De, Flaviano Della Pia, Volker L. Deringer, Rokas Elijošius, Zakariya El-Machachi, Edvin Fako, Andrea C. Ferrari, Annalena Genreith-Schriever, Janine George, Rhys E. A. Goodall, Clare P. Grey, Shuang Han, Will Handley, Hendrik H. Heenen, Kersti Hermansson, Christian Holm, Jad Jaafar, Stephan Hofmann, Konstantin S. Jakob, Hyunwook Jung, Venkat Kapil, Aaron D. Kaplan, Nima Karimitari, Namu Kroupa, Jolla Kullgren, Matthew C. Kuner, Domantas Kuryla, Guoda Liepuoniute, Johannes T. Margraf, Ioan-Bogdan Magdău, Angelos Michaelides, J. Harry Moore, Aakash A. Naik, Samuel P. Niblett, Sam Walton Norwood, Niamh O’Neill, Christoph Ortner, Kristin A. Persson, Karsten Reuter, Andrew S. Rosen, Lars L. Schaaf, Christoph Schran, Eric Sivonxay, Tamás K. Stenczel, Viktor Svahn, Christopher Sutton, Cas van der Oord, Eszter Varga-Umbrich, Tejs Vegge, Martin Vondrák, Yangshuai Wang, William C. Witt, Fabian Zills, and Gábor Csányi.  A foundation model for atomistic materials chemistry, 2023.
  * [33] Samad Hajinazar, Junping Shao, and Aleksey N. Kolmogorov.  Stratified construction of neural network based interatomic models for multicomponent materials.  Phys. Rev. B, 95:014114, Jan 2017.
  * [34] Clemence Corminboeuf, Michele Certiotti, benjamin meyer, Alberto Fabrizio, Andrea Grisafi, and David M. Wilkins.  A transferable machine-learning model of the electron density, Sep 2018\.
  * [35] Kristof T. Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R. Müller, and Alexandre Tkatchenko.  Quantum-chemical insights from deep tensor neural networks.  Nature Communications, 8:13890 EP –, Jan 2017.  Article.
  * [36] M. Gastegger, L. Schwiedrzik, M. Bittermann, F. Berzsenyi, and P. Marquetand.  wacsf—weighted atom-centered symmetry functions as descriptors in machine learning potentials.  The Journal of Chemical Physics, 148(24):241709, 2018.
  * [37] Nongnuch Artrith, Alexander Urban, and Gerbrand Ceder.  Efficient and accurate machine-learning interpolation of atomic energies in compositions with many species.  Phys. Rev. B, 96:014112, Jul 2017.
  * [38] Oliver T. Unke and Markus Meuwly.  A reactive, scalable, and transferable model for molecular energies from a neural network approach based on local information.  The Journal of Chemical Physics, 148(24):241708, 2018.
  * [39] S. Jindal and S. S. Bulusu.  A transferable artificial neural network model for atomic forces in nanoparticles.  ArXiv e-prints, October 2018.
  * [40] Ryo Tamura, Jianbo Lin, and Tsuyoshi Miyazaki.  Machine learning forces trained by gaussian process in liquid states: Transferability to temperature and pressure.  arXiv preprint arXiv:1810.05586, 2018.
  * [41] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013.
  * [42] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio.  Why does unsupervised pre-training help deep learning?  Journal of Machine Learning Research, 11(19):625–660, 2010.
  * [43] Oscar Day and Taghi M. Khoshgoftaar.  A survey on heterogeneous transfer learning.  Journal of Big Data, 4(1):29, Sep 2017.
  * [44] Greg Yang and Edward J. Hu.  Tensor programs iv: Feature learning in infinite-width neural networks.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11727–11737. PMLR, 18–24 Jul 2021.
  * [45] Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli.  Scientific machine learning through physics–informed neural networks: Where we are and what’s next.  Journal of Scientific Computing, 92(3):88, Jul 2022.
  * [46] Anshul Choudhary, John F. Lindner, Elliott G. Holliday, Scott T. Miller, Sudeshna Sinha, and William L. Ditto.  Physics-enhanced neural networks learn order and chaos.  Phys. Rev. E, 101:062207, Jun 2020.
  * [47] Rahul S. Patel, Sharad Bhartiya, and Ravindra D. Gudi.  Physics constrained learning in neural network based modeling.  IFAC-PapersOnLine, 55(7):79–85, 2022.  13th IFAC Symposium on Dynamics and Control of Process Systems, including Biosystems DYCOPS 2022.
  * [48] Joost VandeVondele, Matthias Krack, Fawzi Mohamed, Michele Parrinello, Thomas Chassaing, and Jürg Hutter.  Quickstep: Fast and accurate density functional calculations using a mixed gaussian and plane waves approach.  Computer Physics Communications, 167(2):103–128, April 2005.
  * [49] Jürg Hutter, Marcella Iannuzzi, Florian Schiffmann, and Joost VandeVondele.  CP2K: atomistic simulations of condensed matter systems.  Wiley Interdisciplinary Reviews: Computational Molecular Science, 4(1):15–25, 2014.
  * [50] Oleg A. Vydrov and Troy Van Voorhis.  Nonlocal van der waals density functional: The simpler the better.  The Journal of Chemical Physics, 133(24):244103, 2010.
  * [51] Riccardo Sabatini, Tommaso Gorni, and Stefano de Gironcoli.  Nonlocal van der waals density functional made simple and efficient.  Phys. Rev. B, 87:041108, Jan 2013.
  * [52] Joost VandeVondele and Jürg Hutter.  Gaussian basis sets for accurate calculations on molecular systems in gas and condensed phases.  The Journal of Chemical Physics, 127(11):114105, 2007.
  * [53] Glenn J. Martyna, Michael L. Klein, and Mark Tuckerman.  Nosé–hoover chains: The canonical ensemble via continuous dynamics.  Journal of Chemical Physics, 97(4):2635–2643, 1992.
  * [54] Min Yu and Dallas R. Trinkle.  Accurate and efficient algorithm for bader charge integration.  The Journal of Chemical Physics, 134(6):064111, 2011.

##  Appendix A Appendix A: Energy Decomposition

Here we show the energy decomposition for the pure krypton and argon-krypton
mixture systems. These results highlight that using the electrostatic
component of the energy to infer local energies is valid across all studied
systems.

Figure 5: Total energy minus classical electrostatic energy versus the
classical electrostatic energy in the ArKr mixture.

.

Figure 6: Total energy minus classical electrostatic energy versus the
classical electrostatic energy in the pure Kr system.

.

##  Appendix B Appendix B: Architectures

Here we outline the results of local energy prediction investigations on
different neural network architectures.

0100200300400500Layer
Width0.900.951.001.05R2$Linear\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.850.900.951.001.051.10$Tanh\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.80.91.01.1R2$Linear\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.70.80.91.01.1$Tanh\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{N},\mathcal{D}^{1}\right)$ArKrMx
Figure 7: Correlation score for the total energy of each system. As the
networks are trained on the total energy, we expect the score to remain high
over the full space of architecture. 0100200300400500Layer
Width0.40.60.81.0R2$Linear\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.40.60.81.0$Tanh\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.20.40.60.81.0R2$Linear\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.20.40.60.81.0$Tanh\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{N},\mathcal{D}^{1}\right)$ArKrMx
Figure 8: Correlation score for the local energy of each system.
0100200300400500Layer
Width0.00.20.40.60.81.0R2$Linear\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.00.20.40.60.81.0$Tanh\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.00.20.40.60.81.0R2$Linear\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.00.20.40.60.81.0$Tanh\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{N},\mathcal{D}^{1}\right)$ArKrMx
Figure 9: Correlation score for the total energy of each system computed using
the transferred models. 0100200300400500Layer
Width0.00.20.40.60.81.0R2$Linear\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.00.20.40.60.81.0$Tanh\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.00.20.40.60.81.0R2$Linear\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{N},\mathcal{D}^{1}\right)$0100200300400500Layer
Width0.00.20.40.60.8$Tanh\left(\mathcal{D}^{d},\mathcal{D}^{N},\mathcal{D}^{N},\mathcal{D}^{1}\right)$ArKrMx
Figure 10: Correlation score for the local energy of each system computed
using the transferred models.


# Audio Denoising with Deep Network Priors

###### Abstract

We present a method for audio denoising that combines processing done in both
the time domain and the time-frequency domain. Given a noisy audio clip, the
method trains a deep neural network to fit this signal. Since the fitting is
only partly successful and is able to better capture the underlying clean
signal than the noise, the output of the network helps to disentangle the
clean audio from the rest of the signal. The method is completely unsupervised
and only trains on the specific audio clip that is being denoised. Our
experiments demonstrate favorable performance in comparison to the literature
methods, and our code and audio samples are available at
https://github.com/mosheman5/DNP.

Index Terms: Audio denoising; Unsupervised learning

##  1 Introduction

Many unsupervised signal denosing methods work in a similar way. First, a
spectral mask is estimated, which predicts for every frequency, whether it is
relevant to the clean signal or mostly influenced by the noise. Then, one of a
few classical methods, such as the Wiener filter [1] or MMSE-LSA [2] are used
to clean the audio.

The denoising methods differ in the way in which the mask is first estimated.
Each method is based on a different set of underlying assumptions on the
properties of the signal, the noise, or both. For example, some algorithms
assume that the change in the power spectrum of the noise is slower than the
change in that of the clean signal and, therefore, in order to estimate the
noise statistics, averaging of the power signal over multiple time points is
performed.

In this work, we investigate the use of deep network priors for the task of
unsupervised audio denoising. These priors are based on the assumption that
the clean signal, in the time domain, is well-captured by a deep convolutional
neural network. The method, therefore, trains a network to fit the input
signal, and observes the part of the signal that has the largest amount of
uncertainty, i.e., which was modeled most poorly. This part is then masked out
and one of the classical speech-enhancement methods is applied.

Similar priors have been recently used in computer vision, in order to
reconstruct noise free images [3]. However, we note that the cleaning of audio
signals is much more involved. We observe three major differences between the
usage of deep network priors in images vs. its use in audio:

  1. 1.

In computer vision, the clean image emerges from the learned network simply as
its output. While averaging over multiple training iterations does improve the
accuracy to some degree, its contribution is minor. In contrast, when a
similar method is applied to audio, the network produces an output that is
unacceptable in quality.

  2. 2.

In computer vision, if early stopping is not applied, the network fits the
noisy input image. In audio, this fitting does not occur nearly as quickly (if
at all) and instead of converging to a solution with a very small loss, the
network displays relatively large fluctuations.

  3. 3.

In vision, the networks train much faster on a clean image than on a mixed
signal that contains both image and noise. In audio, there is a difference in
the training progress between a clean and an extremely noisy signal, but
moderate amounts of noise do not significantly change the convergence speed.

Due to these differences, we cannot assume, as is done when applying deep
image priors in computer vision, that the output of the network can be used
directly. Instead, our method tracks the sequence of network outputs during
training and observes its behavior. A robust spectral mask is obtained, by
considering the relative stability of every point in the spectrogram of the
signal.

Our method achieves results that surpass all of the unsupervised literature
methods and approach those of the supervised methods. Note that similar to
most unsupervised methods in the literature, the method observes only the
input signal and does not benefit from observing (even in an unsupervised way)
other signals in the dataset.

##  2 Related work

Unsupervised Noise estimation algorithms  Noise spectrum estimation is a
crucial part of speech enhancement systems. Traditional noise estimation
algorithms are based on similar assumptions, namely that the speech signal
contains pauses and low-energy segments where statistics of the noise can be
measured, and that the noise is more stationary than the speech signal. Noise
estimation algorithms can be divided into three main categories: minimal-
tracking algorithms [4, 5], which find the minimum for each frequency bin
using a short time window; time-recursive averaging algorithms [6, 7, 8, 9],
which average over time in order to provide the noise estimation; and
histogram-based algorithms [10], in which the amplitude histogram is
calculated for different frequency bands and the noise power is assumed to be
the most occurring value.

The estimated a-priori SNR of the noise signal is then used as an input for
one of a few classical speech enhancement algorithms. These algorithms
multiply the original signal with a gain calculated from the a-priori SNR,
either by a direct element-wise application, as in the Wiener filter, or using
a regularization over the time-freqeuncy domain.

Supervised Noise estimation algorithms  Supervised speech denoising algorithms
observe, during training, both the noisy sample and the underlying clean
samples and learn to map from noisy samples to clean samples. The SEGAN method
[11] employs an encoder-decoder architecture, which is trained with an
additional GAN loss [12].

The current state-of-the-art method, called Deep Feature Loss [13] uses a
context aggregation network, and instead of using the MSE loss between the
output and the target, employs a perceptual loss function. The perceptual loss
is derived from the deep layer activations of a network that is pre-trained
for audio classification tasks.

Deep Image Priors The DIP method [3] can be viewed a regularized inverse-
problem method, in which the regularization is given implicitly, by training a
deep CNN to fit the data. Specifically, a CNN of a given architecture is
trained to produce the input image as its output, given a random tensor as the
network’s input. Assuming that the learning algorithm can fit a clean image
much faster than it fits a noisy signal, the algorithm is stopped after it
starts to fit the given image, but before it fits all of its details. In other
words, there is a “noise impedance” effect that arises from the challenge of
fitting the noisy signal, which is unpredictable in nature.

##  3 Method

The same phenomenon of noise impedance can be observed in audio, however, in a
markedly different way that necessitates a different algorithmic approach. In
our experiments, we employ the CNN architecture known as the WaveUnet [14],
which consists of an encoder-decoder architecture with skip-connections
between the two subnetworks. We create a random input signal $z$ of the same
dimension as the noisy signal $y=x+n$ (we assume an additive noise model, and
the clean signal $x$ and the noise $n$ are unknown) and train the network
$f=f_{\theta}$ to fit the noise, i.e., we solve the following minimization
problem:

$\min_{\theta}\|f_{\theta}(z)-y\|,$ (1)

where $\theta$ is the parameter vector of the function $f$, i.e., the weights
and biases of this network.

|  |
---|---|---
(a) | (b) | (c)
Figure 1: Typical loss profiles obtained during training for a signal that is
clean, noisy, or entirely noise. (a) first 2000 iterations. (b) zoom-in to the
first 50 iterations. (c) zoom in to iteration 250 onward.

As can be seen in the example given in Fig. 1, the network fits clean speech
or music signals much faster than it fits noise signals. However, unlike the
situation in computer vision, there is little difference in the coarse
behavior between the clean signal and the signal with the added noise.

While in images, the signal recovered by the network $f_{\theta}(z)$ during
training becomes very similar to $x$, before it starts to resemble $y$, in
audio the situation is different. As can be seen in Fig. 4, at every iteration
$i$ of training, the current network, which we denote as $f_{i}$, produces a
signal $f_{i}(z)$ that is only partly denoised. In addition, while in images
the method converges to a stable solution, i.e., $f_{i}(z)\sim f_{i+1}(z)$
after a few training iterations [3], this is not the case in audio. In the
case of audio, the network output rapidly changes between iterations.

Moreover, the noise-free signal $x$ was never reached in our experiments, even
after extremely long training sessions. This can also be seen in the baseline
experiment we perform (Sec. 4), in which we report the minimal error obtained
when training the network ($\min_{i}\|f_{i}(z)-x\|$). This hindsight
experiment, which would have produced a good result in computer vision,
produces poor outputs in audio.

The discussion above does not mean that $f$ does not evolve during training.
As can be seen in Fig. 4, as the iterations progress, the output of $f$
becomes more expressive, and the network models additional frequencies in the
signal.

---
(a) |  (b) |  (c) |  (d)
(e) |  (f) |  (g) |  (h)

Figure 2: Instability during training. (a-d) the network output $f_{i}(z)$ for
four consecutive iterations. (e-g) the difference between pairs of consecutive
iterations. (h) the spectrogram of the clean signal

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)

Figure 3: Progress during training. (a) iteration 25. (b) iteration 100. (c)
iteration 250. (d) iteration 1000. The low frequencies are learned first, and
the higher frequencies follow.

|  |  |
---|---|---|---
(a) | (b) | (c) | (d)

Figure 4: Sample results. (a) The spectrogram of the clean signal $x$. (b) The
spectrogram of the noisy signal $y$. (c) The a-priori mask of the signal $M$
returned by our method. (d) The mask obtained by the Connected Frequencies [8]
method. Algorithm 1 The denoising with network priors method

1:$n$: noisy input, $t$: number of iterations

2:$\theta_{0}\leftarrow\texttt{XavierInit}()$ $\triangleright$ Initialize the
weights of $f_{0}$

3:$z\sim N(0,1)$ $\triangleright$ Initialize the random vector $z$

4:${Y}_{0}\leftarrow STFT(f_{0}(z))$

5:$C=0$

6:for $i\leftarrow 1:t$ do

7: $\theta_{i}\leftarrow\texttt{semi}\arg\min_{\theta}\|f_{\theta}(z)-y\|$
$\triangleright$ One training iteration on $f_{i-1}$, starting with
$\theta=\theta_{i-1}$ obtaining $f_{i}$

8: ${Y}_{i}\leftarrow\texttt{STFT}(f_{i}(z))$

9: $H_{i}\leftarrow(|~{}|{Y}_{i}|-|{Y}_{i-1}|~{}|)/|{Y}_{i}|$ $\triangleright$
Absolute differences

10: $p_{1}\leftarrow\texttt{percentile}(H_{i},10)$

11: $p_{2}\leftarrow\texttt{percentile}(H_{i},90)$

12: $H_{i}\leftarrow\max(\min(H_{i},P_{2}),P_{1})$ $\triangleright$ Clip
values

13: $C\leftarrow C+H_{i}$ $\triangleright$ Accumulate the differences

14:end for

15:$M=(\max(C)-C)/(\max(C)-\min(C))$ $\triangleright$ Normalize

16:return $M$$\triangleright$ Estimated a-priori SNR of the signal

Based on these observations, we propose the method depicted in Alg. 1 for
estimating the a-priori SNR of the clean signal. The input to the method is
the signal $y$. Its output is a mask of the dimensions of the signal’s STFT,
with values in the range [0,1].

After computing a random $z$ vector in line 2, the method undergoes an
iterative process for $t$ iterations. Unlike the situation in computer vision,
in speech, and other audio signals that we tried, the network $f$ cannot
easily fit $y$. Early stopping is, therefore, not a major concern, and we can
choose any number of iterations $t$ that is large enough.

Each iteration consists of the following steps, where $i$ is the iteration
index. First, in line 6 of the algorithm, the network $f_{i-1}$ is trained for
one iteration, obtaining $f_{i}$. Then, in line 7, one computes $f_{i}(z)$ and
its STFT $Y_{i}$. We next compute $H_{i}$, which is the absolute difference
between $|Y_{i-1}|$ and $|Y_{i}|$ normalized by the latter.

In order to avoid extreme values, every value in $H_{i}$ that is above the
90th percentile or below the 10th percentile is clipped. An accumulator $C$
sums the resulting matrices (line 12). The accumulator would have high values
in the coordinates of the time-frequency domain, in which there is the least
stability in the reconstruction of $y$ by the network $f$.

Once the $t$ iterations are over, $C$ is normalized to be in the range of
$[0,1]$ (line 14). High accumulated variability implies noise and we,
therefore, flip the values, before returning the mask $M$.

With this estimation of the a-pirori SNR, a classical denoising method, such
as LSA [2] or the Weiner filter can be used to perform denoising. In our
experiments, we employ the former.

##  4 Experiments

Table 1: Quantitative evaluation denoising. A higher score means better performance. Approach | Supervised | CSIG | CBAK | COVL | PESQ | SSNR
---|---|---|---|---|---|---
SEGAN [11] | yes | 3.48 | 2.94 | 2.80 | 2.16 | 7.73
Deep Feature Loss [13] | yes | 3.86 | 3.33 | 3.22 | - | -
MCRA [6] | no | 2.23 | 2.36 | 1.91 | 1.80 | 5.17
IMCRA [7] | no | 2.49 | 2.53 | 2.13 | 1.92 | 5.89
MCRA2 [9] | no | 2.39 | 2.50 | 2.08 | 1.97 | 5.92
Martin [5] | no | 2.48 | 2.61 | 2.21 | 2.12 | 6.37
Doblinger [4] | no | 2.55 | 2.66 | 2.29 | 2.21 | 6.48
Hirsch [10] | no | 2.67 | 2.66 | 2.35 | 2.21 | 6.26
Connected Frequencies [8] | no | 2.73 | 2.66 | 2.38 | 2.21 | 6.18
Wiener [15] | no | 3.23 | 2.68 | 2.67 | 2.22 | 5.07
Ours | no | 3.08 | 2.84 | 2.67 | 2.39 | 7.27
Best $f_{i}(z)$ | hindsight | 3.04 | 2.36 | 2.39 | 1.81 | 1.59
Averaged $f_{i}(z)$ | no | 3.13 | 2.39 | 2.47 | 1.87 | 1.66
Noisy Samples | no | 3.35 | 2.44 | 2.63 | 1.97 | 1.68

When applying our method, we employ a WaveUnet with six layers and 60 filters
per layer. Each mask filter was produced after $t=5000$ iterations using the
Adam optimizer with a learning rate of 0.0005. The method seems insensitive to
either of these parameters.

Noisy speech samples were provided by the authors of [16]. For the purpose of
our work, only the test set has been used. This test set is composed by mixing
multiple speakers with 5 different noise types and 4 different SNR setting
(2.5, 7.5, 12.5 and 17.5 dB). The original 48 kHz files were downsampled to 16
kHz, the same as other baseline methods [13, 11]. The spectrograms are
obtained by using 32 ms Hann window and 8 ms hop length.

For the purpose of computing the results of the baseline unsupervised
denoising algorithms, the open source metrics evaluation and noise estimation
toolbox [17] was used. The statistics of the supervised baselines were taken
from the respective papers [11, 13].

Multiple quality scores are used to measure the success of the methods,
including (1) CSIG: Mean opinion score (MOS) predictor of signal distortion,
(2) CBAK: MOS predictor of background-noise intrusiveness, (3) COVL: MOS
predictor of overall signal quality, (4) PESQ: Perceptual evaluation of speech
quality, and (5) SSNR: Segmental SNR.

All unsupervised methods were post-processed by a highpass filter with a
cutoff frequency of 60 Hz, to remove noise below the human speech base
frequency. The noise cleaning method used is MMSE-LSA [2], and the various
measures were computed by using the open source toolbox mentioned above.

A sample result is give in Fig. 4, and the statistics are reported in Tab. 1.
As can be seen from the table, our method outperforms all unsupervised
literature methods in all metrics, with the exception of the CSIG metric, in
which it is the second highest method. The method is also largely comparable
to the SEGAN method [11], despite not being trained on any sample outside the
single sample $y$, while SEGAN is fully supervised. Our unsupervised method is
outperformed by the Deep Feature Loss method [13], which enjoys both a large
fully supervised training set and a strong pretrained perceptual loss.

We also present the results of a baseline, in which the best network output
obtained during training $f_{i}(z)$, $i=1..t$ is returned. This happens in
hindsight, by comparing it to the clean signal $x$. As can be seen, the
statistics for this result are very similar to those of the original noisy
signal. One can also observe that averaging multiple network reconstructions
$f_{i}(z)$, $i=250,500,750,\dots,5000$, does not lead to an acceptable result.
This is in contrast to the application of deep network priors in computer
vision [3], where the network produces clean outputs during its training.

The samples, up to the conference file size limit, are attached as
supplementary. More samples can be found next to our code at
https://github.com/mosheman5/DNP.

##  5 Conclusions

The advent of deep learning has led to effective supervised denoising
algorithms. However, as far as we know, little progress has been made in the
unsupervised domain. In this work, we explore the usage of deep network priors
for this task and observe that the approach used to employ these priors in
images is not suitable for audio signals. We, therefore, develop a new method,
which is shown to outperform the unsupervised methods and even approach the
quality of the supervised methods.

## References

  * [1] J. S. Lim and A. V. Oppenheim, “Enhancement and bandwidth compression of noisy speech,” _Proceedings of the IEEE_ , vol. 67, no. 12, pp. 1586–1604, Dec 1979.
  * [2] Y. Ephraim and D. Malah, “Speech enhancement using a minimum mean-square error log-spectral amplitude estimator,” _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , vol. 33, no. 2, pp. 443–445, April 1985.
  * [3] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9446–9454.
  * [4] G. Doblinger, “Computationally efficient speech enhancement by spectral minima tracking in subbands,” in _in Proc. Eurospeech_ , 1995, pp. 1513–1516.
  * [5] R. Martin, “Noise power spectral density estimation based on optimal smoothing and minimum statistics,” _IEEE Transactions on Speech and Audio Processing_ , vol. 9, no. 5, pp. 504–512, July 2001.
  * [6] I. Cohen and B. Berdugo, “Noise estimation by minima controlled recursive averaging for robust speech enhancement,” _IEEE Signal Processing Letters_ , vol. 9, no. 1, pp. 12–15, Jan 2002.
  * [7] I. Cohen, “Noise spectrum estimation in adverse environments: improved minima controlled recursive averaging,” _IEEE Transactions on Speech and Audio Processing_ , vol. 11, no. 5, pp. 466–475, Sep. 2003.
  * [8] K. V. Sørensen and S. V. Andersen, “Speech enhancement with natural sounding residual noise based on connected time-frequency speech presence regions,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2005, no. 18, p. 305909, Nov 2005. [Online]. Available: https://doi.org/10.1155/ASP.2005.2954
  * [9] S. Rangachari and P. C. Loizou, “A noise-estimation algorithm for highly non-stationary environments,” _Speech Communication_ , vol. 48, no. 2, pp. 220 – 231, 2006.
  * [10] H. G. Hirsch and C. Ehrlicher, “Noise estimation techniques for robust speech recognition,” in _1995 International Conference on Acoustics, Speech, and Signal Processing_ , vol. 1, May 1995, pp. 153–156 vol.1.
  * [11] S. Pascual, A. Bonafonte, and J. Serrà, “Segan: Speech enhancement generative adversarial network,” in _Interspeech_ , 2017, pp. 3642–3646.
  * [12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _NIPS_ , 2014, pp. 2672–2680.
  * [13] F. G. Germain, Q. Chen, and V. Koltun, “Speech denoising with deep feature losses,” _arXiv preprint arXiv:1806.10522_ , 2018.
  * [14] D. Stoller, S. Ewert, and S. Dixon, “Wave-u-net: A multi-scale neural network for end-to-end audio source separation,” _19th International Society for Music Information Retrieval Conference (ISMIR 2018)_ , 2018.
  * [15] P. Scalart and J. V. Filho, “Speech enhancement based on a priori signal to noise estimation,” in _1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings_ , vol. 2, May 1996, pp. 629–632 vol. 2.
  * [16] C. Valentini-Botinhao, “Noisy speech database for training speech enhancement algorithms and tts models,” 2017.
  * [17] P. C Loizou, _Speech Enhancement: Theory and Practice 2nd Edition_. CRC Press, 2013.