remark ∎ example ∎

# Deformation Decomposition versus Energy Decomposition for Chemo- and Poro-
Mechanics

Janel Chua<EMAIL_ADDRESS>Department of Civil and Environmental
Engineering, Carnegie Mellon University  Mina Karimi  Department of Mechanical
and Civil Engineering, California Institute of Technology  Patrick Kozlowski
Department of Civil and Environmental Engineering, Carnegie Mellon University
Mehrdad Massoudi  National Energy Technology Laboratory, 626 Cochran Mill Road
Pittsburgh, PA 15236  Santosh Narasimhachary  Siemens Corporation  Kai Kadau
Siemens Energy Inc  George Gazonas  DEVCOM Army Research Laboratory, Aberdeen
Proving Ground, 21005 MD, USA  Kaushik Dayal  Department of Civil and
Environmental Engineering, Carnegie Mellon University  Center for Nonlinear
Analysis, Department of Mathematical Sciences, Carnegie Mellon University
Department of Mechanical Engineering, Carnegie Mellon University

###### Abstract

We briefly compare the structure of two classes of popular models used to
describe poro- and chemo- mechanics wherein a fluid phase is transported
within a solid phase. The multiplicative deformation decomposition has been
successfully used to model permanent inelastic shape change in plasticity,
solid-solid phase transformation, and thermal expansion, which has motivated
its application to poro- and chemo- mechanics. However, the energetic
decomposition provides a more transparent structure and advantages, such as to
couple to phase-field fracture, for models of poro- and chemo- mechanics.

††preprint: Journal of Applied Mechanics, Vol. 91, 014501, 2024.
https://doi.org/10.1115/1.4062967

##  1 Introduction

There is significant current interest in modeling problems of fluid transport
in porous media as well as fluid phase transport in solid materials, i.e.,
poro- and chemo- mechanics. The motivations range from modeling hydrogels [1],
to transport in geological structures [2], to hydrogen embrittlement of metals
[3], among various other applications. An approach that has been proposed in
the literature to model poro- and chemo- mechanics is to decompose the
deformation gradient into an elastic part – that causes stress – and an
inelastic part – that accounts for the shape change due to fluid transport;
this is the “multiplicative decomposition”. The application of the
multiplicative decomposition to poro- and chemo- mechanics is motivated by the
success of this strategy in modeling thermoelasticity, plasticity, twinning,
solid-solid phase transformations, and related phenomena that involve
inelastic deformation, e.g. reviewed in [4, 5].

However, an important distinction between thermoelasticity, plasticity,
twinning on the one hand, and poro- and chemo- mechanics on the other hand, is
that the former class of phenomena do not involve the introduction of material
into the bulk of the solid, whereas the latter class do. The introduced
material has energy and stress that is distinct from the energy and stress of
the solid. This motivates an approach that is based on additively combining
the energies of the solid and the fluid, e.g. [1, 6, 7] and many others.

In this note, we briefly contrast the overall structure of these two
approaches, and argue that the additive decomposition of the energy is the
more appealing alternative. We also highlight [8], which critically examined a
model micromechanism for biological growth, and consequently argued against a
multiplicative decomposition in that context.

### Definitions and Notation.

We use $\displaystyle{\mathbold F}$ for the deformation gradient,
$\displaystyle{\mathbold T}$ for the 1st Piola-Kirchoff (P-K) stress, and
$\displaystyle\mu$ for the chemical potential. For simplicity, we assume a
single fluid phase that is defined by the densities in the deformed and the
reference configurations $\displaystyle\rho$ and $\displaystyle\rho_{0}$;
i.e., $\displaystyle\rho$ and $\displaystyle\rho_{0}$ are the mass of the
fluid phase per unit deformed and reference volumes. For simplicity, we follow
the affine deformation assumption that the volume of the fluid phase in the
deformed and reference configurations are related by $\displaystyle
J=\det{\mathbold F}$, implying that $\displaystyle\rho=J^{-1}\rho_{0}$.

The energy density is written in terms of $\displaystyle{\mathbold F}$ and
$\displaystyle\rho_{0}$, rather than $\displaystyle{\mathbold F}$ and
$\displaystyle\rho$, because the former pair of arguments can be independently
varied in a simple way that decouples deformation and transport.

##  2 Multiplicative Deformation Decomposition into Elastic and Inelastic
Parts

The central idea in the multiplicative deformation decomposition is to write
the deformation gradient $\displaystyle{\mathbold F}$ as the product of an
elastic part $\displaystyle{\mathbold F}_{e}$, that causes stress, and an
inelastic part $\displaystyle{\mathbold F}_{i}\left(\rho_{0}\right)$, that is
driven by the coupled field $\displaystyle\rho_{0}$. That is,
$\displaystyle{\mathbold F}={\mathbold F}_{e}{\mathbold F}_{i}$, and the free
energy density is typically of the general form given by:

$W({\mathbold F},\rho_{0})=W_{e}\left({\mathbold F}{\mathbold
F}_{i}^{-1}\left(\rho_{0}\right)\right)+W_{i}(\rho_{0})$ (1)

The elastic energy $\displaystyle W_{e}$ is minimized when
$\displaystyle{\mathbold F}={\mathbold F}_{i}\left(\rho_{0}\right)$ up to
rotations.

The resulting P-K stress has the form:

${\mathbold T}=\frac{\partial W}{\partial{\mathbold F}}=\frac{\partial
W_{e}}{\partial{\mathbold F}_{e}}{\mathbold F}_{i}^{-T}$ (2)

In general, $\displaystyle{\mathbold F}_{i}$ is invertible. Consequently,
$\displaystyle{\mathbold T}={\bf 0}\iff\frac{\partial
W_{e}}{\partial{\mathbold F}_{e}}={\bf 0}$.

The (referential) chemical potential is the key quantity that governs the
transport of the fluid phase. It is defined as the energy-conjugate to
$\displaystyle\rho_{0}$, e.g. [9], and has the form:

$\mu=\frac{\partial W}{\partial\rho_{0}}=\frac{\partial
W_{e}}{\partial{\mathbold F}_{e}}\frac{\ \mathrm{d}{\mathbold F}_{i}^{-T}}{\
\mathrm{d}\rho_{0}}:{\mathbold F}+\frac{\ \mathrm{d}W_{i}}{\
\mathrm{d}\rho_{0}}$ (3)

where $\displaystyle:$ represents a double contraction over 2-nd order
tensors.

We note the key undesirable features of this class of models. Consider a
homogeneous body described by such a model with zero applied traction on the
entire boundary and uniform $\displaystyle\rho_{0}$. A solution to this
boundary-value problem is $\displaystyle{\mathbold T}\equiv{\bf 0}$, implying
that $\displaystyle{\mathbold F}={\mathbold F}_{i}$ on the entire body, up to
a rigid rotation. Hence, even if the solid material is highly deformed due to
fluid infiltration, e.g. due to hydrogen in a metallic lattice with stretched
atomic bonds or due to fluid in a hydrogel with stretched polymer chains, the
elastic energy $\displaystyle W_{e}$ is minimized. While it is possible to
augment the inelastic energy $\displaystyle W_{i}$ to depend on the
deformation, this would not allow the interpretation of the decomposition of
$\displaystyle{\mathbold F}$ as an elastic and inelastic part.

##  3 Additive Energy Decomposition into Solid Strain Energy and Fluid Energy

The central idea in the additive energy decomposition is to additively combine
the energetic contributions of the solid and fluid phases to find the total
free energy. An example of such a form is:

$W({\mathbold F},\rho_{0})=\alpha W_{s}({\mathbold
F})+(1-\alpha)JW_{f}\left(J^{-1}\rho_{0}\right)$ (4)

where $\displaystyle J=\det{\mathbold F}$. The referential volume fraction of
the solid phase is $\displaystyle\alpha$, and we assume a single fluid phase;
for the case with multiple fluids with the possibility of evolving volume
fractions, we refer to [2] and references therein.

The form of the fluid contribution $\displaystyle
JW_{f}\left(J^{-1}\rho_{0}\right)$ is motivated by the requirement that the
energy density $\displaystyle W_{f}$ of a simple fluid depends only on the
density in the deformed state, i.e., $\displaystyle\rho=J^{-1}\rho_{0}$, when
we consider the isothermal setting. Further, the leading factor of
$\displaystyle J$ accounts for the fact that $\displaystyle W_{f}$ is the
energy per unit deformed volume, whereas the hyperelastic energy density
$\displaystyle W$ is per unit reference volume. An important assumption here
is that of affine deformation; i.e., both the solid skeleton and the fluid
volume deform under $\displaystyle{\mathbold F}$ affinely but this can be
relaxed [2].

The resulting P-K stress has the form:

$\begin{split}{\mathbold T}&=\frac{\partial W}{\partial{\mathbold
F}}=\alpha\frac{\partial W_{s}}{\partial{\mathbold
F}}+(1-\alpha)\frac{\partial J}{\partial{\mathbold
F}}\left(W_{f}\left(J^{-1}\rho_{0}\right)-J^{-1}\rho_{0}\frac{\partial
W_{f}}{\partial\rho}\right)=\alpha\frac{\partial W_{s}}{\partial{\mathbold
F}}+(1-\alpha)J{\mathbold
F}^{-T}\left(W_{f}\left(\rho\right)-\rho\frac{\partial
W_{f}}{\partial\rho}\right)\\\ &={\mathbold T}_{s}+(1-\alpha)J{\mathbold
F}^{-T}p\end{split}$ (5)

where we have defined the solid stress $\displaystyle{\mathbold
T}_{s}:=\alpha\frac{\partial W_{s}}{\partial{\mathbold F}}$; used the relation
$\displaystyle\frac{\partial J}{\partial{\mathbold F}}=J{\mathbold F}^{-T}$;
and used the relation that the fluid pressure111 The pressure $\displaystyle
p$ is the derivative of the Helmholtz free energy with respect to volume,
keeping temperature and mass fixed [10]. In terms of the density
$\displaystyle\rho$ – which is inversely proportional to the volume when the
mass is fixed – and in terms of the Helmholtz free energy _density_ , we have:
$p=\frac{\partial\
}{\partial\left(\frac{1}{\rho}\right)}\left(\frac{1}{\rho}W_{f}(\rho)\right)=W_{f}(\rho)-\rho\frac{\partial
W_{f}}{\partial\rho}$ (6) is given by $\displaystyle
p=\left(W_{f}\left(\rho\right)-\rho\frac{\partial
W_{f}}{\partial\rho}\right)$. We can then define the P-K fluid stress
$\displaystyle{\mathbold T}_{f}:=(1-\alpha)J{\mathbold F}^{-T}p$,
corresponding to a Cauchy stress
$\displaystyle\mathbold{\sigma}_{f}=(1-\alpha)p{\mathbold I}$.

The chemical potential for this model has the form:

$\mu=\frac{\partial W}{\partial\rho_{0}}=(1-\alpha)\frac{\partial
W_{f}}{\partial\rho}$ (7)

which corresponds to the standard thermodynamic expression for fluids.

We consider again a homogeneous body with zero applied traction on the entire
boundary and uniform $\displaystyle\rho_{0}$. In this energetic decomposition
model, a solution to this boundary-value problem is that the total stress
$\displaystyle{\mathbold T}\equiv{\bf 0}$, implying that the fluid and solid
stresses $\displaystyle{\mathbold T}_{s}$ and $\displaystyle{\mathbold T}_{f}$
balance each other but neither is necessarily zero. Given a fluid pressure
$\displaystyle p\neq 0$, there will generally be a fluid stress
$\displaystyle{\mathbold T}_{f}\neq{\bf 0}$ which in turn requires a solid
stress $\displaystyle{\mathbold T}_{s}\neq{\bf 0}$. With this state of fluid
and solid stress, $\displaystyle W_{s}$ will not reach its minimum, and the
body will deform. Hence, the deformation of the solid material due to fluid
infiltration, e.g. the stretching of atomic bonds or polymer chains, will be
reflected in the solid stress $\displaystyle{\mathbold T}_{s}$ and energy
$\displaystyle W_{s}$.

##  4 A Remark on Phase-field Fracture Modeling for Poro- and Chemo- Mechanics

The phase-field approach provides a powerful method for modeling fracture,
e.g. [11]. Briefly, a phase-field $\displaystyle\phi$ tracks the level of
damage, with $\displaystyle\phi=1$ denoting the intact undamaged material and
$\displaystyle\phi=0$ denoting the completely damaged or fractured material.
An energetic framework uses an energy density with contributions that include
$\displaystyle\phi^{2}W({\mathbold F})+G_{c}(1-\phi)^{2}$, where the first
term accounts for the elastic energy and the second for the work to fracture,
with $\displaystyle G_{c}$ being the Griffith parameter. This structure of the
energy density sets up a competition between elastic energy and the work to
fracture: minimizing over $\displaystyle\phi$ drives $\displaystyle\phi\to 0$
when the elastic energy becomes larger due to deformation than the work to
fracture. Given this reasoning, it is natural to develop poro- and chemo-
mechanical models of phase-field fracture wherein only the energy of the solid
phase $\displaystyle W_{s}$ contributes to the fracture energetic balance.
That is, in a simple model, we would replace (4) by the expression
$\displaystyle\phi^{2}W_{s}({\mathbold
F})+JW_{f}\left(J^{-1}\rho_{0}\right)+G_{c}(1-\phi)^{2}$ to model fracture
which releases the stress in the solid but does not directly affect the fluid.

### Competing Interest Statement.

The authors have no competing interests to declare.

### Acknowledgments.

We acknowledge financial support from ARO (MURI W911NF-19-1-0245) and NSF
(DMREF 2118945, DMS 2108784); NSF for XSEDE computing resources provided by
Pittsburgh Supercomputing Center; and Noel Walkington and Tony Rollett for
useful discussions. Kaushik Dayal acknowledges an appointment to the National
Energy Technology Laboratory sponsored by the U.S. Department of Energy.

## References

  * [1] Wei Hong, Xuanhe Zhao, Jinxiong Zhou, and Zhigang Suo.  A theory of coupled diffusion and large deformation in polymeric gels.  Journal of the Mechanics and Physics of Solids, 56(5):1779–1793, 2008.
  * [2] Mina Karimi, Mehrdad Massoudi, Noel Walkington, Matteo Pozzi, and Kaushik Dayal.  Energetic formulation of large-deformation poroelasticity.  International Journal for Numerical and Analytical Methods in Geomechanics, 46(5):910–932, 2022.
  * [3] Emilio Martínez-Pañeda, Alireza Golahmar, and Christian F Niordson.  A phase field formulation for hydrogen assisted cracking.  Computer Methods in Applied Mechanics and Engineering, 342:742–761, 2018.
  * [4] Vlado A Lubarda.  Constitutive theories based on the multiplicative decomposition of deformation gradient: Thermoelasticity, elastoplasticity, and biomechanics.  Appl. Mech. Rev., 57(2):95–108, 2004.
  * [5] Souhayl Sadik and Arash Yavari.  On the origins of the idea of the multiplicative decomposition of the deformation gradient.  Math. Mech. Solids, 22(4):771–772, 2017.
  * [6] Virginia von Streng, Rami Abi-Akl, Bianca Giovanardi, and Tal Cohen.  Morphogenesis and proportionate growth: A finite element investigation of surface growth with coupled diffusion.  Journal of the Mechanics and Physics of Solids, 146:104211, 2021\.
  * [7] Timothy J Truster and Arif Masud.  A unified mixture formulation for density and volumetric growth of multi-constituent solids in tissue engineering.  Computer Methods in Applied Mechanics and Engineering, 314:222–268, 2017.
  * [8] Isaac Vikram Chenchiah and Patrick D Shipman.  An energy-deformation decomposition for morphoelasticity.  Journal of the Mechanics and Physics of Solids, 67:15–39, 2014\.
  * [9] Olivier Coussy.  Poromechanics.  John Wiley & Sons, 2004.
  * [10] Morton E Gurtin.  An introduction to continuum mechanics.  Academic press, 1982.
  * [11] John D Clayton and Jarek Knap.  A geometrically nonlinear phase field theory of brittle fracture.  International Journal of Fracture, 189(2):139–148, 2014.


# Multimodal Speech Enhancement Using Burst Propagation

Leandro A. Passos, Ahmed Khubaib, Mohsin Raza, and Ahsan Adeel
University of Wolverhampton, Wolverhampton, England, UK
<EMAIL_ADDRESS>
###### Abstract

This paper proposes the MBURST, a novel multimodal solution for audio-visual
speech enhancements that consider the most recent neurological discoveries
regarding pyramidal cells of the prefrontal cortex and other brain regions.
The so-called burst propagation implements several criteria to address the
credit assignment problem in a more biologically plausible manner: steering
the sign and magnitude of plasticity through feedback, multiplexing the
feedback and feedforward information across layers through different weight
connections, approximating feedback and feedforward connections, and
linearizing the feedback signals. MBURST benefits from such capabilities to
learn correlations between the noisy signal and the visual stimuli, thus
attributing meaning to the speech by amplifying relevant information and
suppressing noise. Experiments conducted over a Grid Corpus and CHiME3-based
dataset show that MBURST can reproduce similar mask reconstructions to the
multimodal backpropagation-based baseline while demonstrating outstanding
energy efficiency management, reducing the neuron firing rates to values up to
$70\%$ lower. Such a feature implies more sustainable implementations,
suitable and desirable for hearing aids or any other similar embedded systems.

###### Index Terms:

Burstpropagation, Multimodal Learning, Audio-Visual Speech Enhancement

##  I Introduction

The World Health Organization states that $430$ million people suffer from
moderate to higher hearing loss nowadays and estimates that nearly $2.5$
billion will present hearing impairment to some degree by 2050 [1]. The
problem impacts the individual social relationships and the perception of
surrounding sounds [2], which may also lead to loneliness and psychological
distresses [3, 4]. Therefore, intelligent computer systems that boost and
clean the audio signal are highly desirable.

In this context, many efforts have been applied toward machine learning-based
approaches for speech enhancement [5, 6]. A particularly interesting approach
comprises multimodal approaches that combine correlated audio and visual (AV)
information to amplify relevant information and suppress noise [7]. The idea
can be illustrated by a dialogue in a pub, where people talk and live music is
played in the background, and the listener focuses on reading the speaker’s
lips and expressions to mind the context and infer meaning from the
conversation. The work of Adeel et al. [8] comprises related applications
regarding IoT and 5G for lip-reading hearing aids, encrypted audio speech
reconstruction [9], and speech enhancement in different conditions using deep
learning [10, 11]. Further, the recent work of Passos et al. [12] investigates
audio and visual information fusion using Graph Neural Networks and canonical
correlation analysis, as well as a cortical cell-inspired model that
approximates the computational model to a more biologically plausible approach
[13].

Despite such approximation, the methods mentioned above and most of the
machine and deep learning solutions rely on backpropagation, an algorithm
inspired by an antiquate modeling of neuronal information flow where inputs
are linearly combined and exposed to an activation function, whose output
feeds consecutive layers. Further, the outcome is compared to an expected
target, and the error is back-propagated, assigning the credit for any
mistakes or successes to neurons that are multiple synapses away from the
output and updating the weights associated with such neurons accordingly. In
this scenario, Payeur [14] proposed a novel approach inspired by more recent
studies on the physiological mechanism of pyramidal neurons, where the
learning is regulated by the frequency of bursts, the so-called
Burstpropagation.

Burstpropagation tackles the credit assignment problem in a more biologically
plausible manner by addressing the primary principles of pyramidal neurons
suggested by Körding and König [15] as follows. First, it employs a burst-
dependent learning rule to steer the sign and magnitude of plasticity through
feedback stimuli. Second, it multiplexes the feedback and feedforward
information across multiple layers using different connections, i.e., distinct
weights are used during the forward and the backward propagation processes.
Third, it performs the alignment between feedback and feedforward connections
by approximating the loss-function gradients through burst-dependent learning.
Finally, the fourth principle regards the linearization of the feedback
signals concerning the credit information, which can be performed using
recurrent connections.

Therefore, this paper proposes the so-called MBURST, a multimodal approach
that combines audio and visual information to enhance speech quality using
burst propagation. The model preserves biological properties of real neurons
like short-term synaptic plasticity [16], dendritic excitability [17],
synaptic transmission, inhibitory microcircuit, and burst-dependent synaptic
plasticity [14]. Experiments conducted over a Grid Corpus and CHiME3-based
dataset for clean audio mask reconstruction show that MBURST can generate
clean audio masks with similar quality to a multimodal backpropagation-based
baseline while providing a dramatic reduction in the energy consumption,
reaching values up to $70\%$ lower. Such an attribute is extremely desirable
for real-world applications like embedded systems on hearing aid devices.
Thus, this paper comprises two main contributions:

  * •

it provides the MBURST, a biologically plausible and energy-efficient method
that combines audio and visual information for speech enhancement; and

  * •

it introduces burst propagation, a state-of-the-art algorithm inspired by
pyramidal neurons, in the context of multimodal learning.

The remainder of this paper is presented as follows. Section II provides a
theoretical background regarding Burstpropagation and the proposed approach.
Further, Sections III and IV describe the methodology employed to conduct the
experiments and the results, respectively. Finally, Section V states the
conclusions and future work.

##  II Theoretical Background

This section briefly introduces the concepts behind burst propagation, as well
as the proposed burst propagation-based multimodal approach, namely MBURST.

###  II-A Burst Propagation

The burst-dependent algorithm, or burst propagation, defines the weighted sum
at the neuron’s input as “somatic potentials”. Similarly, the neuron’s output
is termed the event rate. The ’somatic potentials’ of a dense layer are
defined as:

$v_{l}=W_{l}e_{l-1},$ (1)

where $l\in\\{1,2,\cdots,L\\}$ denotes the network’s layers, $W_{l}$ is the
weight connecting layer $l-1$ to layer $l$ and $e_{l}$ is the $l$th layer’s
event rate, defined by:

$e_{l}=f(v_{l}),$ (2)

where $f_{l}$ stands for any activation function, e.g., the Rectifier Linear
Unit (ReLU) or Sigmoid, for layer $l$. Regarding convolution layers, the
multiplications are simply replaced by a convolution operator where $W_{l}$
represents layer $l$’s kernel.

Since the burst rate drives the model’s learning procedure, the first step in
the process comprises computing the output layer ($l=L$) bursting probability:

$p_{L}=\zeta(\bar{p}_{L}-h(e_{L})\odot\Delta_{e_{L}}\mathcal{L})$ (3)

where $\bar{p}_{L}$ is a hyperparameter that defines the reference bursting
probability, $\zeta$ is a squashing function that guarantees
$P_{L,i}\in\left[0,1\right]$, $\Delta_{e_{L}}\mathcal{L}$ stands dor the loss
function derivative, and $h(e_{l})$ is a vector-valued function defined by:

$h(e_{l})\equiv f^{{}^{\prime}}(v_{l})\odot e_{l}^{-1}.$ (4)

The bursting probability is used to calculate the bursting rate with and
without teaching, i.e., $b_{l}$ and $\bar{b}_{l}$, respectively, at any layer
$l$:

$\begin{array}[]{c}b_{l}=p_{l}\odot e_{l},\\\ \bar{b}_{l}=\bar{p}_{l}\odot
e_{l}.\end{array}$ (5)

Further, the bursting rates are propagated to the previous layers through
feedback weights, namely $Y$. The result is employed to compute the current
hidden layer’s “dendritic potential” with ($u_{l}$) and without
($\bar{u}_{l}$) teacher. Notice that, for convolutional layers, $Y$ stands for
a kernel, and a convolution operation replaces the multiplication.

$\begin{array}[]{c}u_{l}=h(e_{l})\odot(Y_{l}b_{l+1}),\\\
\bar{u}_{l}=h(e_{l})\odot(Y_{l}\bar{b}_{l+1}).\end{array}$ (6)

In the sequence, the somatic potentials are used to calculate the hidden
layer’s bursting probability and the reference bursting probability:

$\begin{array}[]{c}p_{l}=\sigma(\beta u_{l}+\alpha),\\\
\bar{p}_{l}=\sigma(\beta\bar{u}_{l}+\alpha),\end{array}$ (7)

where $\beta=1$ and $\alpha=0$ are are constants that control the dendritic
transfer function and $\sigma$ is a sigmoid function.

Finally, the change in the forward and feedback weights, $W$ and $Y$,
respectively, are computed as follows:

$\begin{array}[]{c}\Delta W_{l}=\Delta Y_{l}=-(b_{l}-\bar{b}_{l})\odot
e_{l-1}^{T}.\end{array}$ (8)

Notice that, once again, the multiplication is replaced by a backward
convolution in the context of convolutional layers.

###  II-B MBURST

The proposed multimodal burst propagation-based approach to speech enhancement
combines audio and visual information for clean audio estimation. The model
takes advantage of recent neurological research on pyramidal cells to learn
coherent relationships between noisy audio and visual information and extract
clean information by amplifying relevant information and suppressing noise.

The model comprises two burst-dependent convolutional neural networks that
work in parallel to extract features from noisy audio inputs and their
respective frames. The outcomes from both channels are flattened and exposed
to a burst-dependent fully-connected layer for embedding extraction. The
embeddings are concatenated into a single feature vector, which is used to
feed a similar burst-dependent dense layer for classification. Figure 1
depicts the process pipeline.

Figure 1: Proposed method. The left image depicts the multimodal network with
noisy audio and visual inputs. Both channels comprise two layers of burst-
based convolutions, followed by flattening and a burst-based dense layer for
feature embedding. Finally, such embeddings are concatenated and used to feed
another burst-based dense for classification. Right-top and right-bottom
frames illustrate the burst-based convolutional and dense layers,
respectively. Notice such layers comprise a forward weight matrix (kernel)
$\bm{W}$ and a backward weight matrix $\bm{Y}$.

##  III Methodology

This section describes the dataset and the environment configuration
considered in the experiments.

###  III-A Dataset

The experiments performed in this work were conducted over a dataset based on
roughly $1,000$ sentences composed of a six-word sequence extracted from
Speakers $1$ in Grid Corpus [18] dataset. The clear Grid utterances are
blended with non-stationary random noises, e.g., bus, cafe, street, and
pedestrian, using four distinct Signal-to-noise ratios (SNR), i.e.,
$\\{-12dB,-6dB,0dB,6dB\\}$, extracted from the $3$rd CHiME Challenge
(CHiME3)[19].

####  III-A1 Data pre-processing

The original audio was resampled at $16$ kHz. Each utterance was divided into
approximately $75$ frames with $1,244$ samples per frame and a $25\%$
increment rate. Further, a $622$-bin power spectrum was computed using STFTs
and a hamming window procedure. A Viola-Jones lip detector [20] and an object
tracker [21] extract the speakers’ lip images from the GRID Corpus films,
recorded at $25$ frames per second. A $92\times 50$-inch area around the lip’s
center was picked. The recovered lip sequences were up-sampled three times to
match the $75$ STFT frames of the audio signal.

####  III-A2 Ideal Binary Mask

Hu and Wang [22] proposed the so-called Ideal Binary Mask (IBM), a binary mask
with a positive value to a time-frequency (TF) unity whose goal energy is
greater than the interference energy and zero otherwise. Among a number of the
desired qualities in speech enhancement, the binary masks retain the speech-
dominated units while zeroing out the noise-dominated units of noisy speech
and providing high intelligibility scores on speech reconstruction, even
considering meager signal-to-noise ratio (SNR) settings [23].

IBM employs a premixing of target and interference signals, i.e., the speech
and noise signals, computed through a criterion function:

$\displaystyle IBM(t,f)$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{ll}1&\mbox{if
}10\log_{10}\left(\frac{X(t,f)^{2}}{N(t,f)^{2}}\right)\geq LC,\\\
0&\mbox{otherwise}\end{array}\right.$ (11)

where $X$ stands for the speech with no background noise, $N$ represents the
noise portion, and $LC$ is the local criterion used to differentiate between
speech and background noise, i.e., a threshold. Further, the frame of
reference $t$ and the frequency range $f$ denote the time and frequency
dimensions, respectively.

The method assigns a positive value to the more prevalent TF units if their
SNR is greater than or equal to the LC. Conversely, TF units dominated by
noise are presumed to have SNRs lower than LC. Thus they are given assigned a
value of 0 and muted. The proportion of the TF-positive speech versus noise-
dominant samplings relies on a proper LC selection. Since SNRs greater than
$0$ dB are speech dominating and those less than $0$ dB are noise dominant, an
LC of 0 dB is supposedly optimal. However, studies suggest that such selection
may imply disproportionate removal of TF units on mixed noisy and speech
instances due to the addition of noise known as artifacts, therefore
suggesting employing an LC of $5$ dB [24, 24] for optimum performance.

(a) (b) (c) (d)

Figure 2: Multimodal-based mask reconstructions: (a) Original mask, (b)
reconstruction using Unimodal, (c) Multimodal, and (d) the proposed MBURST.

###  III-B Experimental Setup

This work implements MBURST, a burst propagation-based multimodal architecture
for speech enhancement. The architecture comprises two parallel networks,
i.e., an audio and a visual channel, each containing two convolutional layers
with $32$ channels, followed by flattening and a dense layer with $256$ units
for embedding extraction. Finally, such embeddings are concatenated and used
to feed the top layer, which is encharged to estimate the clean audio masks.
It also implements two baselines for comparison purposes, i.e., a similar
network using the same configuration but the backpropagation algorithm for
learning, namely Multimodal and a unimodal version also trained using
backpropagation that comprises the same architecture except by the visual
channel, namely unimodal.

The convolutions consider kernels of size $3\times 3$, paddings equal to $1$,
and a stride of size $2$ for the visual and $1$ for the audio channels. This
difference in the stride concerns the shape of the audio input, i.e., since
the input audio comprises the current signal, composed of $500$ features, plus
the seven prior frames, forming an $8\times 500$ matrix, we opted to use
stride $1$ to maintain this shape due to the reduced size.

The experiments were conducted using the weighted binary cross entropy as the
loss function since the rate of white pixels in the masks (relevant
information) is several times smaller than black ones. The same justificative
is considered for employing the F1-score as the evaluation metric. Further,
the optimization of the parameters is conducted using Adam with a learning
rate of $10^{-3}$ and weight decay of $10^{-6}$. Regarding burst propagation,
the reference bursting probability follows the standard value adopted by the
original work [14], setting $\bar{P}_{L}=0.2$. The dataset was randomly split
into training ($80\%$), testing ($15\%$), and proxy ($5\%$) sets. The training
is conducted for $4$ runs, considering different dataset splits, during $40$
epochs.

Finally, MBURST, as well as the baseline architectures, were implemented in
Python using Pytorch framework [25], and the code is available on
GitHub111Available at:https://github.com/Leandropassosjr/MBURST..

##  IV Experimental Results

This section presents the results and discussion under the perspectives of
clean audio signal mask reconstruction and energy efficiency.

###  IV-A Clean Audio Signal Mask Reconstruction

Table I presents the average F1-score and accuracy, as well as their
respective standard deviations, over the training and test sets for the
proposed MBURST and the baseline architectures. In this scenario, one can
notice that the Unimodel was not capable of providing competitive results,
which is expected since the visual context is essential to introduce context
to the noisy signal and boost the reconstruction. On the other hand, both the
Multimodal and MBURST obtained similar results, with the Multimodal approach
achieving slightly better values (less than $2\%$ on average), which can be
possibly explained by activation sparsity induced by the burst rate mechanism.
Such results reinforce the relevance of visual context for clean speech
reconstruction.

TABLE I: F1-score and Accuracy concerning the Unimodal, Multimodal, and the
proposed MBURST for clean audio mask reconstruction over the train and test
sets.

Data set | Metric | Unimodal | Multimodal | MBURST
---|---|---|---|---
Train | F1 | $0.677\pm 0.000$ | $0.802\pm 0.005$ | $0.782\pm 0.001$
Acc. | $82.919\pm 0.043$ | $91.898\pm 0.220$ | $90.602\pm 0.067$
Test | F1 | $0.696\pm 0.000$ | $0.790\pm 0.003$ | $0.768\pm 0.002$
Acc. | $81.758\pm 0.047$ | $90.236\pm 0.164$ | $88.798\pm 0.233$

Figure 2 depict some examples of the reconstructions themselves. Notice that
such images were generated by gathering groups of $150$ randomly selected
sequential samples from the proxy data set. In this context, one can observe
that the masks reflect the results provided in Table I, i.e., Unimodal
presents a poor mask reconstruction, while both the Multimodal and the MBURST
provide more accurate and very similar results. Once again, the results
suggest that correlated visual context is essential to extract more relevant
information from noisy audio signals and improve the reconstruction quality.

###  IV-B Energy Analysis

This work considers the energy efficiency in terms of neurons’ activation
rate, which shows itself as MBURST’s major triumph. In this aspect, one can
observe in Figure 3 that the MBURST neurons’ firing rate decreases
dramatically to values below $0.2$ in the first iterations and strides towards
values close to $0.1$ in the subsequent iterations for both training and
testing sets, showing an energy gain around $70\%$ and $65\%$ over the
Multimodal and Unimodal approaches, respectively. Regarding the
backpropagation-based approaches, the Unimodel performed better than
Multimodel, suggesting that the backpropagation was not robust enough to
combine the noisy audio and visual information in an energy efficiency
fashion, thus implying the worst performance achieved by the Multimodel
approach in this context.

(a)

(b)

Figure 3: Energy efficiency rate over (a) train and (b) test sets.

Table II analytically reinforces MBURST’s robustness in the context of energy
efficiency by presenting the area under the curve over the average energy rate
measured during the training and testing procedures. In this context, MBURST
shows itself $48\%$ and $58\%$, on average over train and test sets, more
efficient than Unimodal and Multimodal, respectively.

TABLE II: Area under the curve considering the average energy rate during
training and testing for the Unimodal, Multimodal, and the proposed MBURST.

Data set | Unimodal | Multimodal | MBURST
---|---|---|---
Train | $11.12$ | $13.70$ | $5.63$
Test | $11.51$ | $14.14$ | $6.08$

##  V Conclusions

This paper proposed the MBURST, a burst propagation-based multimodal approach
that implements a more biologically plausible solution for the task of AV
speech enhancement. One can draw two main conclusions from experiments
conducted over a Grid Corpus and CHiME3-based dataset. First, multimodal AV
architectures are more efficient than a standard unimodal approach for clean
audio mask reconstruction since correlated visual information provides
contextual meaning to noisy audio. Second, MBURST can produce similar results
to a traditional backpropagation multimodal architecture in the context of
accuracy and reconstruction task metrics, with a massive advantage regarding
energy efficiency due to the burst-rate-based activation mechanism.

Regarding future work, we aim to extend the burst propagation concepts to
recent studies comprising context-sensitive neocortical neurons [26] and
canonical cortical networks [13].

## Acknowledgments

The authors are grateful to the Engineering and Physical Sciences Research
Council (EPSRC) grant EP/T021063/1.

## References

  * [1] W. H. Organization _et al._ , “Hearing screening: considerations for implementation,” 2021.
  * [2] W. Noble, _Self-assessment of hearing and related function_. Wiley-Blackwell, 1998.
  * [3] A. R. Huang, J. A. Deal, G. W. Rebok, J. M. Pinto, L. Waite, and F. R. Lin, “Hearing impairment and loneliness in older adults in the united states,” _Journal of Applied Gerontology_ , vol. 40, no. 10, pp. 1366–1371, 2021.
  * [4] A.-S. Helvik, G. Jacobsen, and L. R. Hallberg, “Psychological well-being of adults with acquired hearing impairment,” _Disability and rehabilitation_ , vol. 28, no. 9, pp. 535–545, 2006.
  * [5] N. Das, S. Chakraborty, J. Chaki, N. Padhy, and N. Dey, “Fundamentals, present and future perspectives of speech enhancement,” _International Journal of Speech Technology_ , vol. 24, no. 4, pp. 883–901, 2021.
  * [6] R. Rehr and T. Gerkmann, “On the importance of super-gaussian speech priors for machine-learning based speech enhancement,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 26, no. 2, pp. 357–366, 2017\.
  * [7] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal deep learning,” in _ICML_ , 2011.
  * [8] A. Adeel, J. Ahmad, and A. Hussain, “Real-time lightweight chaotic encryption for 5g iot enabled lip-reading driven secure hearing-aid,” _arXiv preprint arXiv:1809.04966_ , 2018.
  * [9] A. Adeel, J. Ahmad, H. Larijani, and A. Hussain, “A novel real-time, lightweight chaotic-encryption scheme for next-generation audio-visual hearing aids,” _Cognitive Computation_ , vol. 12, no. 3, pp. 589–601, 2020\.
  * [10] A. Adeel, M. Gogate, and A. Hussain, “Contextual deep learning-based audio-visual switching for speech enhancement in real-world environments,” _Information Fusion_ , vol. 59, pp. 163–170, 2020.
  * [11] M. Gogate, K. Dashtipour, A. Adeel, and A. Hussain, “Cochleanet: A robust language-independent audio-visual model for real-time speech enhancement,” _Information Fusion_ , vol. 63, pp. 273–285, 2020.
  * [12] L. A. Passos, J. P. Papa, J. Del Ser, A. Hussain, and A. Adeel, “Multimodal audio-visual information fusion using canonical-correlated graph neural network for energy-efficient speech enhancement,” 2022. [Online]. Available: https://arxiv.org/abs/2202.04528
  * [13] L. A. Passos, J. P. Papa, and A. Adeel, “Canonical cortical graph neural networks and its application for speech enhancement in future audio-visual hearing aids,” _arXiv preprint arXiv:2206.02671_ , 2022.
  * [14] A. Payeur, J. Guerguiev, F. Zenke, B. A. Richards, and R. Naud, “Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits,” _Nature neuroscience_ , vol. 24, no. 7, pp. 1010–1019, 2021.
  * [15] K. P. Körding and P. König, “Supervised and unsupervised learning with two sites of synaptic integration,” _Journal of computational neuroscience_ , vol. 11, no. 3, pp. 207–215, 2001.
  * [16] H. Markram, Y. Wang, and M. Tsodyks, “Differential signaling via the same axon of neocortical pyramidal neurons,” _Proceedings of the National Academy of Sciences_ , vol. 95, no. 9, pp. 5323–5328, 1998.
  * [17] M. E. Larkum, J. J. Zhu, and B. Sakmann, “A new cellular mechanism for coupling inputs arriving at different cortical layers,” _Nature_ , vol. 398, no. 6725, pp. 338–341, 1999.
  * [18] M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio-visual corpus for speech perception and automatic speech recognition,” _The Journal of the Acoustical Society of America_ , vol. 120, no. 5, pp. 2421–2424, 2006.
  * [19] J. Barker, R. Marxer, E. Vincent, and S. Watanabe, “The third ‘chime’speech separation and recognition challenge: Analysis and outcomes,” _Computer Speech & Language_, vol. 46, pp. 605–626, 2017.
  * [20] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in _Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001_ , vol. 1. Ieee, 2001, pp. I–I.
  * [21] D. A. Ross, J. Lim, R.-S. Lin, and M.-H. Yang, “Incremental learning for robust visual tracking,” _International journal of computer vision_ , vol. 77, no. 1, pp. 125–141, 2008.
  * [22] G. Hu and D. Wang, “Monaural speech segregation based on pitch tracking and amplitude modulation,” _IEEE Transactions on neural networks_ , vol. 15, no. 5, pp. 1135–1150, 2004.
  * [23] Y. Jiang, H. Zhou, and Z. Feng, “Performance analysis of ideal binary masks in speech enhancement,” in _2011 4th International Congress on Image and Signal Processing_ , vol. 5. IEEE, 2011, pp. 2422–2425.
  * [24] E. W. Healy, S. E. Yoho, J. Chen, Y. Wang, and D. Wang, “An algorithm to increase speech intelligibility for hearing-impaired listeners in novel segments of the same noise type,” _The Journal of the Acoustical Society of America_ , vol. 138, no. 3, pp. 1660–1669, 2015.
  * [25] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [26] A. Adeel, M. Franco, M. Raza, and K. Ahmed, “Context-sensitive neocortical neurons transform the effectiveness and efficiency of neural information processing,” _arXiv preprint arXiv:2207.07338_ , 2022.