# BirdSoundsDenoising: Deep Visual Audio Denoising for Bird Sounds

Youshan Zhang
Yeshiva University, NYC, NY
<EMAIL_ADDRESS>Jialu Li
Cornell University, Ithaca, NY
<EMAIL_ADDRESS>
###### Abstract

Audio denoising has been explored for decades using both traditional and deep
learning-based methods. However, these methods are still limited to either
manually added artificial noise or lower denoised audio quality. To overcome
these challenges, we collect a large-scale natural noise bird sound dataset.
We are the first to transfer the audio denoising problem into an image
segmentation problem and propose a deep visual audio denoising (DVAD) model.
With a total of 14,120 audio images, we develop an audio ImageMask tool and
propose to use a few-shot generalization strategy to label these images.
Extensive experimental results demonstrate that the proposed model achieves
state-of-the-art performance. We also show that our method can be easily
generalized to speech denoising, audio separation, audio enhancement, and
noise estimation.

Figure 1: The overall progress of our proposed deep visual audio denoising
model (DVAD).

##  1 Introduction

With the development of technology, audio signals have been increasingly used
as main sources of information transmission [17], such as teleconferences
[14], the speech-to-text function in social media [32], the lung [22] and
heart [15] sounds for disease diagnosis, instrument solo identification [11],
and hearing aid [37, 27, 33], etc. Therefore, it is important to maintain the
quality of signal transmission and retain as much useful information as
possible. However, due to existing noises in the actual environment, the
transmission of audio signals, including speech and other signals that we
intend to collect, are inevitably affected, resulting in the poor quality and
intelligibility of audio signals. Audio denoising can significantly increase
audio quality and contribute to a better outcome of information transition.

Audio denoising has been a popular research area in recent years and different
methods have been applied to reduce noise and separate audio, including
traditional statistics approaches [5, 28, 12, 19] and deep learning approaches
[27, 26, 2, 24, 16]. While there are several difficulties encountered across
these models. In this paper, we specifically use samples from the natural
environment, which presents more challenges to the proposed research models.

Why natural audio denoising is difficult?

Firstly, the most common difficulty encountered is the limited sources for
training. Deep learning-based models require both clean and noisy audio
samples for training. However, in reality, audio signals come with noises that
cannot be separated to produce desired training samples [14]. Secondly, most
noisy audio samples used for model training are artificially compiled, such as
white gaussian noise (WGN) [30, 39], which is composed differently from
natural noise. In addition, we could still observe the clean signal patterns
in the artificial noise audio, while it is difficult to observe the clean
signal patterns in real noise audio as shown in Fig. 1 (leftmost and rightmost
signal). Therefore, the denoising performance of the training models might not
perform as well in the real setting compared to experiments.

These two challenges are commonly encountered in the audio denoising field,
and we address them using a deep visual audio denoising model (DVAD). In this
paper, we first collect audio samples that are directly acquired from the
natural environment. The proposed model can process more complex and natural
noises compared to previous models. We offer three principal contributions:

  * •

We present a benchmark bird sounds denoising dataset with the goal of
advancing the state-of-the-art in audio denoising under natural noise
background.

  * •

To the best of our knowledge, we are the first to transfer audio denoising
into an image segmentation problem. By removing the noise area in the audio
image, we can realize the purpose of audio denoising.

  * •

We develop an audio ImageMask tool to label the collected dataset and apply a
few-shot generalization strategy to accelerate the data label process. We also
demonstrate that our model can be easily extended to speech denoising, audio
separation, audio enhancement, and noise estimation.

##  2 Related Work

Audio denoising has been widely explored, and many methods have evolved from
traditional methods of estimating the difference between noise and clean audio
statistics [37], to the adoption of deep learning methods [3].

Traditional methods for audio denoising can be dated back to the 1970s. Boll
[5] proposed a noise suppression algorithm for spectral subtraction using the
spectral noise bias calculated in a non-speech environment. Another
statistical method proposed in [28] is a more comprehensive algorithm,
combining the concept of A Priori Signal-to-Noise Ratio (SNR) with earlier
typical audio enhancement schemes such as Wiener filtering [6, 18], spectral
subtraction, or Maximum Likelihood estimates. In the realm of the frequency-
domain algorithm, minimum mean square error (MMSE) based approaches is a
mainstream approach besides Wiener filtering. Hansen et al. [12] proposed an
auditory masking threshold enhancement method by applying Generalized MMSE
estimator in an auditory enhancement scheme. In [19], the MMSE estimator is
used to enhance the performance of short-time spectral coefficients by
estimating discrete Fourier transform (DFT) coefficients of both noisy and
clean speech. One major problem is that the performance of traditional methods
for noise separation and reduction will be degraded with the presence of
natural noises, which are largely different from artificial noises applied in
the experiments.

Wavelet transformation methods are developed to overcome the difficulty of
studying signals with low SNR and are reported with better performance than
filtering methods. Zhao et al. [41] used an improved threshold denoising
method, overcame the discontinuity in hard-threshold denoising, and reduced
the permanent bias in soft-threshold denoising. Srivastava et al. [30]
developed a wavelet denoising approach based on wavelet shrinkage, allowing
for the analysis of low SNR signals. Pouyani et al. [22] proposed an adaptive
method based on discrete wavelet transform and artificial neural network to
filtrate lung sound signals in a noisy environment. Kui et al. [15] also
combined the wavelet algorithm with CNNs to classify the log mel-frequency
spectral coefficients features from the heart sound signal with higher
accuracy. These combined methods outperformed single wavelet transformation
methods.

Deep learning methods are later introduced to the audio denoising field,
complementing the disadvantages of traditional methods and demonstrating a
stronger ability to learn data and characteristics with a few samples [37].
The deep neural network (DNN)-based audio enhancement algorithms have shown
great potential in their ability to capture data features with complicated
nonlinear functions [16]. Xu et al. [38] introduced a deep learning model for
automatic speech denoising to detect silent intervals and better capture the
noise pattern with the time-varying feature. Saleem et al. [27] used the deep
learning-based approach for audio enhancement accompanying complex noises. An
ideal binary mask (IBM) is used during the training and testing, and the
trained DNNs are used to estimate IBM. Xu et al. [39] proposed a DNN-based
supervised method to enhance audio by finding a mapping function between noisy
and clean audio samples. A large mixture of noisy dataset is used during the
training and other techniques, including global variance equalization and the
dropout and noise-aware training strategies. Saleem et al. [26] also developed
a supervised DNN-based single channel audio enhancement algorithm and applied
less aggressive Wiener filtering as an additional DNN layer. Vuong et al. [34]
described a modulation-domain loss function for a deep learning-based audio
enhancement approach, applying additional Learnable spectro-temporal receptive
fields to enhance the objective prediction of audio quality and
intelligibility.

Yet a problem in speech denoising application of DNN is that sometimes, it is
difficult for models to track a target speaker in multiple training speakers,
which means that the DNNs are not easy to handle long-term contexts [33, 16].
Therefore, deep learning approaches, such as convolutional neural network
(CNN)-based and recurrent neural network (RNN)-based models, are explored.
Alamdari et al. [2] applied a fully convolutional neural network (FCN) for
audio denoising with only noisy samples, and the study displayed the
superiority of the new model compared to the traditional supervised
approaches. Germain et al. [10] trained an FCN using a deep feature loss,
trained for acoustic environment detection and domestic audio tagging. The
research showed that this new approach is particularly useful for audio with
the most intrusive background noise. Kong et al. [14] proposed an audio
enhancement method with pre-trained audio neural networks using weakly labeled
data with only audio tags of audio clips and applied a convolutional U-Net to
predict the waveform of individual anchor segments selected by PANNs. Raj et
al. [24] proposed a multilayered CNN-based auto-CODEC for audio signal
denoising, using the mel-frequency cepstral coefficients, providing good
encoding and high security. Abouzid et al. [1] combined the convolutional and
denoising autoencoders into convolutional denoising autoencoders for the
suppression of noise and compression of audio data.

On top of single-type deep learning methods, Tan et al. [33] proposed a
recurrent convolutional network by incorporating a convolutional encoder-
decoder and long short-term memory (LSTM) into the convolutional recurrent
neural network (CRN) architecture to address real-time audio enhancement. This
method outperformed an existing LSTM-based model with fewer trainable
parameters. Gao et al. [8], and [9] respectively applied a progressive
learning framework for DNN-based and LSTM-based audio enhancement to improve
model performance and reduce complexity. Li et al. [16] combined the
progressive learning framework with a causal CRN to further reduce the
trainable parameters and improve audio quality and intelligibility. This
proposed method produced a close performance to the CRN.

Many deep learning approaches are implemented in the time-frequency domain,
using short-time Fourier transform (STFT) and inverse short-time Fourier
transform (ISTFT) [36]. Some methods address audio enhancement via time-domain
algorithms, viewing audio enhancement as a filtering problem [40]. Yu et al.
[40] proposed a DNN-based Kalman filter algorithm for audio enhancement. The
DNN is used for estimating the linear prediction coefficients in the KF.
Sonning et al. [29] investigated the performance of a time-domain network for
speech denoising, addressing the original inability of STFT/ISTFT-based time-
frequency approaches to capture short-time changes and discovered its
usefulness in a real-time setting. Wang et al. [36] proposed a two-stage
transformer neural network for end-to-end audio denoising in the time domain,
including an encoder, a two-stage transformer module , a masking module and a
decoder. Their model outperformed many time- or frequency-domain models with
less complexity.

##  3 Methods

###  3.1 Problem

Given a noisy audio signal $\\{x_{t}\\}_{t=1}^{T}$, we aim to extract the
clean audio $\\{y_{t}\\}_{t=1}^{T}$ by learning a mapping $\mathcal{M}$. The
goal of audio denoising is to minimize the approximation error between the
denoised audio $\\{\mathcal{M}(x_{t})\\}_{t=1}^{T}$ and clean audio
$\\{y_{t}\\}_{t=1}^{T}$. In our DVAD model, we convert audio denosing to an
image segmentation problem. Given the audio images
$\mathcal{I}=\\{I^{i}\\}_{i=1}^{n}$ based on audio signals
$X=\\{x^{i}\\}_{i=1}^{n}$ and its ground truth labeled masks
$M=\\{m^{i}\\}_{i=1}^{n}$, we propose to minimize the error between prediction
of any image segmentation model $F(I)$ and $M$.

###  3.2 Motivation

Although some existing deep audio denoising models utilized magnitude images
of audio signals, they only filtered out some regions of the image to realize
the purpose of denoising. The details of these images are less explored. Our
DVAD model delves into the audio image to find different patterns between
noise and clean signal areas. As shown in Fig. 2(b), we can find that there
are significant differences between the noise and clean signal areas.
Therefore, we can achieve the purpose of audio denoising if we can segment the
clean signal areas. We further treat the audio denoising as an image
segmentation problem.

###  3.3 Preliminary

####  3.3.1 Short-Time Fourier Transform (STFT)

STFT is used to analyze how the frequency content of a nonstationary signal
changes over time.

$STFT_{x}(t,f)=\int_{-\infty}^{\infty}x(t)\omega(t-\tau)e^{-j2\pi ft}dt$ (1)

where $STFT_{x}(t,f)$ is the coefficient of STFT. STFT is a function of time
($t$) and frequency ($f$), and it shows how frequency $f$ of the signal $x(t)$
changes with time $t$. $\omega$ is a window function, $\tau$ is a short time
and $j$ is the square root of $-1$. In our model, we aim to convert the signal
to frequency domain and get the raw images for each bird sound.

####  3.3.2 Inverse Short-Time Fourier Transform (ISTFT)

The STFT is invertible, $i.e.$, the original signal can be reconstructed from
the transform by the inverse STFT. It is defined as:

$\tilde{x_{t}}=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}STFT_{x}(t^{\prime},f^{\prime})\omega(t-t^{\prime})e^{-j2\pi
f^{\prime}t^{\prime}}dt^{\prime}\ df^{\prime}.$ (2)

In our model, we aim to reconstruct the bird sound based on the segmented bird
sound image.

(a) Audio signal

(b) Audio image

Figure 2: The conversion from audio signal (a) to audio image (b) by taking
the absolute value of STFT. (b) is a symmetrical image, and most noise areas
are concentrated in the center of an image (the light white horizontal lines).
The informative and clean signal (bird sound) majorly lies in the bright white
patterns. Please refer to supplementary material for more noise signal areas.

###  3.4 Methodology

To form the audio denoising problem as an image segmentation problem, we first
need to represent the audio in image format. After performing STFT using Eq.
(1), let $S=STFT_{x}(t,f)$, we can define the audio image ($I$) in the
following equation,

$I=abs(S),$ (3)

where $abs$ takes the absolute value from the complex frequency domain $S$. As
shown in Fig. 2, we convert a one-second bird sound audio to its audio image.
We observe that the patterns of noise areas and clean sound areas are
distinguishable. If we segment the clean sound areas in the audio image, then
we could remove the noise from the frequency domain $S$. As shown in Fig.
3(b), we can apply ISTFT in Eq. (2) after getting the denoised audio image to
reconstruct the denoised audio signal. Therefore, we can convert the audio
denoising problem into an image segmentation problem.

(a) Mask image

(b) Segmented image

Figure 3: The mask (a) and segmented audio image (b) for the signal in Fig. 2.
(a) is the mask of clean signal areas in Fig. 2(b) and (b) is the segmented
audio image by removing noise areas.

To realize the image segmentation task, we need to train a segmentation model
$F$ to segment the clean audio signal area. In our DVAD model, we train the
segmentation model using dice loss as follows.

$Dice\ loss=1-2\times\frac{m\cap\tilde{m}}{m+\tilde{m}},$ (4)

where $m$ is the ground truth mask and $\tilde{m}=F(I)$ is the predicted mask
of the segmentation model given the input image $I$. In the mask, we denote
the clean audio areas using 1 and represent the noise areas using 0. After
training $F$, we can predict the segmented mask of any audio image. Next, we
aim to reconstruct the denoised audio.

In Eq. (2), we can recover the original $x(t)$ given the key input frequency
domain $S$. To remove the noise audio, we need to filter out noise areas in
$S$ given the predicted mask from the segmentation model. We define the new
frequent domain $S^{\prime}$ in the following equation:

$S^{\prime}=S,\ \ \ \text{and}\ \ \ S^{\prime}[\tilde{m}<1]=0,$ (5)

where $S^{\prime}[\tilde{m}<1]=0$ aims to replace all noise area with 0 to
realize the purpose of removing noise areas. Then, we can apply ISTFT to
reconstruct the denoised audio as follows.

$\tilde{x_{t}}=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}S^{\prime}\omega(t-t^{\prime})e^{-j2\pi
f^{\prime}t^{\prime}}dt^{\prime}\ df^{\prime}.$ (6)

As shown in Fig. 4(a), we can get the denoised audio after removing the noise
audio using Eq. (5). We also show the overlapping of original signal with
denoised signal in Fig. 4(b). The rest part of blue signal (noise areas) is
removed from the red denoised signal.

(a) Denosied audio

(b) Overlapping signals

Figure 4: The denoised audio using our DVAD model (a) and overlapping of
original signal (blue color) and denoised audio signal (red color).

###  3.5 DVAD overall algorithm

Considering all steps in Sec. 3.4, the scheme of our proposed DVAD model is
shown in Fig. 1 and the overall algorithm is presented in Alg. 1.

Algorithm 1 Deep Visual Audio Denoising (DVAD). $B(\cdot)$ denotes the mini-
batch training sets, $I$ is the number of iterations.

1: Input: Audio signals $X=\\{x^{i}\\}_{i=1}^{n}$ and labeled mask images
$M=\\{m^{i}\\}_{i=1}^{n}$, where $n$ is the total number of audios.

2: Output: Denoised audio signals

3: Generate audio images $\mathcal{I}=\\{I^{i}\\}_{i=1}^{n}$ using Eq. (3)

4: for $iter=1$ to $I$ do

5: Derive $B(\mathcal{I})$ and $B(M)$ sampled from $\mathcal{I}$ and $M$

6: Optimize any segmentation model $F$ using Eq. (4)

7: end for

8: Get the clean frequency domain using Eq. (5)

9: Output the denoised audio signals using Eq. (6)

Figure 5: Four sample images and masks. In each sample, the left is the audio
image, and the right is its labeled mask. The width of each sample depends on
the length of the audio. Longer audio will produce a wider audio image.

##  4 Datasets

###  4.1 Data Collection

Our data is collected from the xeno-canto website, which is a public website
to share bird sounds from around the world 111https://xeno-canto.org/explore.
We first collect 15,300 bird sounds from one second to fifteen seconds. Unlike
many audio denoising datasets, which have manually added artificial noise, our
collected bird sounds contain natural noises, including wind, waterfall, rain,
etc. Then, we apply STFT in Eq. (1) to convert bird sounds to the frequency
domain and get the audio images using Eq. (3). Some audios have two
soundtracks (left and right soundtracks). Hence we get more images than
collected audios. Since we have converted audio denoising to an image
segmentation problem, we need to provide masks for audio images to optimize
any image segmentation models. Therefore, our next task is to label these
audio images.

###  4.2 Data labeling

Image mask labeling is time-consuming, tedious, and expensive. However, to
train any machine learning algorithms, we have to provide enough labeled
datasets to achieve a good performance. Given that there is no specific
software for audio image labeling, we also developed an audio ImageMask tool.
The ImageMask software has three key functions. (1). It can open an audio
image and label it to create a mask and show the overlapping between the raw
audio image and labeled mask. (2). We can save created masks and denoised
audios. The software also supports human verification. All accepted denoised
audio will be saved in an ‘Accepted’ folder. The folder contains another four
sub-folders: original audio, denoised audio, audio images, and audio masks.
(3). We can also compare the ground truth masks with predicted masks from any
segmentation models to validate the performance of models. More details of our
developed audio ImageMask tool can be found in the supplementary material.

Few-shot generalization

Although we developed a specific audio denoising software, it takes around 5
minutes to label one audio image. We have more than 15,000 audio images, and
it is still time-consuming to label all images. To accelerate the labeling
process, we propose to utilize the few-shot generation to first predict coarse
masks for audio images. Then we can verify and update these coarse masks to
get better masks.

Few-shot learning aims to learn a robust model based on a few labeled samples,
then improve the performance of new datasets. To ease the process of image
labeling from scratch, we first manually labeled 100 audio images as training
and 40 images as test. We select DeepLabV3 [7] as the segmentation model and
train the DeepLabV3 model using these 140 labeled images to get a basic model
$F$. We could then predict the coarse mask via $F(I)$. Given any unlabeled
audio image $I^{i}$, we can get all predicted coarse masks as
$\\{F(I^{i})\\}_{i=1}^{n}$. Finally, these coarse masks can be further
modified using our developed audio ImageMask tool. After using the proposed
few-shot generalization strategy, the whole dataset is labeled by four experts
in one month.

###  4.3 BirdSoundsDenoising dataset

After finishing the data labeling process using Sec. 4.2, we can save all
accepted labeled audio to create the BirdSoundsDenoising dataset (note that
some low-quality audio will be removed during the labeling process). The
BirdSoundsDenoising dataset contains 14,120 audio images and has three
folders: training, validation, and test. In each folder, another four sub-
folders are included: raw audios, denoised audios, images, and masks. Tab. 1
shows the statistics of each folder. As shown in Fig. 5, we list four audio
images and their labeled masks. These audio images are varied, and clean
signal areas are also different. Longer audio can produce a longer image. In
the experiments, we will fine-tune models using training and validation
datasets and report results on the test dataset.

Table 1: Statistics on BirdSoundsDenoising dataset Datasets | Training | Validation | Test
---|---|---|---
Number of samples | 10,000 | 1,400 | 2,720

###  4.4 Dataset creation details

In STFT, converting audio signals to audio images, we first use 128-point
Hamming as the window function, the number of overlapped samples $=64$, the
number of DFT points $=1024$, to change bird sounds to frequency domain $S$,
and save all bird audio images using Eq. (3). In few-shot generalization, we
utilize DeepLabV3 as a basic segmentation model to generate coarse audio image
masks. During the training of 140 images, we set batch size = 16, training
iteration $I=100$, learning rate = 0.0001 with an Adam optimizer on a RTX
A6000 GPU. The input image size of the DeepLabV3 model is $[512\times
512\times 3]$. We excluded audio images with no bird sound or extreme noisy
background.

##  5 Experiments

To evaluate the performance of our proposed DVAD model, we test it on our
created BirdSoundsDenoising dataset. We first train six different state-of-
the-art segmentation models to demonstrate the effects of different segmented
masks on audio denoising performance. These six selected segmentation models
have an encoder-decoder architecture. The encoder aims to extract important
features from images (e.g., edge), and the decoder learns how to map these
low-resolution features to the prediction at the pixel level.

  1. 1.

SegNet [4]: the encoder network utilizes the layers from VGG16, and the
decoder network is followed by a pixel-wise classification layer. The decoder
of SegNet can upsample its lower-resolution input feature map(s) using pooling
indices computed in the maxpooling step of the corresponding encoder to
perform non-linear up-sampling.

  2. 2.

U-Net [25]: it has a structure called dilated convolutions and removes the
pooling layer structure. The U-Net architecture comprises a contracting path
to capture context, and a symmetric expanding path that enables precise
localization.

  3. 3.

DeepLabV3 [7]: it uses dilated convolutions and a fully connected conditional
random field to implement the atrous spatial pyramid pooling (ASPP), which is
an atrous version of SPP and can account for different object scales and
improve the accuracy.

  4. 4.

U2-Net [23]: it is able to capture more contextual information from different
scales with the mixture of receptive fields of different sizes in ReSidual
U-blocks.

  5. 5.

Segmenter [31]: it utilizes the Vision Transformer (ViT) as the encoder to
encode all image patches. A point-wise linear decoder is applied to patch
encodings. It also includes a decoder with a mask transformer to further
improves the performance.

  6. 6.

MTU-Net [35]: it proposes a novel transformer module named Mixed Transformer
Module (MTM). It calculates self affinities efficiently by a Local-Global
Gaussian-Weighted Self-Attention (LGG-SA). It also mines inter-connections
between data samples using External Attention (EA).

###  5.1 Implementation details

The training settings of six different segmentation methods are the same as
Sec. 4.4 except that we use batch size of 12 for MTUNet and 32 for
Segmenter222BirdSoundsDenoising dataset and code are available at
https://github.com/YoushanZhang/BirdSoundsDenoising. To show the superiority
of the DVAD model, we also compare the proposed model with three audio
denoising methods [21, 13, 20].

Figure 6: Segmentation results comparisons. Leftmost column is the original
audio image. Ground truth is the labeled mask.  Figure 7: Denoising results
comparisons. Raw audio is the original noise audio. Ground truth is the
labeled mask.

###  5.2 Results

####  5.2.1 Evaluation metrics

We use three metrics ($F1,IoU$ and $Dice$) to evaluate the performance of
image segmentation as follows.

$\displaystyle F1=\frac{TP}{TP+\frac{1}{2}(FP+FN)},$ (7) $\displaystyle
IoU=\frac{m\cap\tilde{m}}{m+\tilde{m}}\ \ \ \
Dice=2\times\frac{m\cap\tilde{m}}{m+\tilde{m}},$

where $TP$ is the number of true positives, $FP$ is the number of false
positives, and $FN$ is the number of false negatives. For audio denoising, we
use signal-to-distortion ratio (SDR) to evaluate our DVAD model. The higher
these four metrics, the better the segmentation model is.

$SDR=10\ \text{log}_{10}\frac{||m||^{2}}{||\tilde{m}-m||^{2}}$ (8)

###  5.3 Performance comparisons

We first show the comparisons of six different segmentation models in Fig. 6.
The segmented masks of the DeepLabV3 model are better than the other five
models. Similarly, we could observe that DeepLabV3 has the highest $F1,IoU$,
and $Dice$ scores in Tab. 2. Therefore, we can infer that the DeepLabV3 model
is the best segmentation model for our BirdSoundsDenoising among all six
segmentation models. In addition, we also reported the mean SDR of all bird
sounds in both validation and test datasets. As shown in Tab. 2, the SDR score
of our DVAD model with DeepLabV3 as the segmentation model achieves the
highest value. Notably, the performance level of three audio denoising methods
(R-CED, Noise2Noise, and TS-U-Net) is relatively lower than all other
segmentation models. The comparisons of raw bird audio, true labeled denoised
audio, and denoised audio from other models are shown in Fig. 7. The denoised
signal of DVAD (with DeepLabV3) is also closer to the labeled denoised signal.
Therefore, our DVAD architecture is effective in improving the audio denoising
performance.

##  6 Discussion

We compared six different state-of-the-art segmentation models and three deep
audio denoising methods. One obvious strength of our model is its better
performance than other methods. Especially the different variants of our DVAD
model are significantly better than the three audio denoising methods in terms
of SDR score. The compelling advantage of the DVAD model lies in the image
segmentation section. We can maintain the crucial clean signal via a segmented
mask, as shown in Fig. 6. Given a clean area in the segmented masks, the clean
signal will be preserved during the ISTFT process. Therefore, converting audio
denoising into an image segmentation problem can be a new stream to further
improve the performance of audio denoising.

Table 2: Results comparisons of different methods ($F1,IoU$, and $Dice$ scores are multiplied by 100. “$-$” means not applicable.  Networks | Validation | Test
---|---|---
$F1$ | $IoU$ | $Dice$ | $SDR$ | $F1$ | $IoU$ | $Dice$ | $SDR$
U2-Net [23] | 60.8 | 45.2 | 60.6 | 7.85 | 60.2 | 44.8 | 59.9 | 7.70
MTU-NeT [35] | 69.1 | 56.5 | 69.0 | 8.17 | 68.3 | 55.7 | 68.3 | 7.96
Segmenter [31] | 72.6 | 59.6 | 72.5 | 9.24 | 70.8 | 57.7 | 70.7 | 8.52
U-Net [25] | 75.7 | 64.3 | 75.7 | 9.44 | 74.4 | 62.9 | 74.4 | 8.92
SegNet [4] | 77.5 | 66.9 | 77.5 | 9.55 | 76.1 | 65.3 | 76.2 | 9.43
DeepLabV3 [7] | 82.6 | 73.5 | 82.6 | 10.33 | 81.6 | 72.3 | 81.6 | 9.96
R-CED [21] | $-$ | $-$ | $-$ | 2.38 | $-$ | $-$ | $-$ | 1.93
Noise2Noise [13] | $-$ | $-$ | $-$ | 2.40 | $-$ | $-$ | $-$ | 1.96
TS-U-Net [20] | $-$ | $-$ | $-$ | 2.48 | $-$ | $-$ | $-$ | 1.98

##  7 Conclusion

In this paper, we are the first to convert audio denoising into an image
segmentation problem. We then propose a deep visual audio denoising (DVAD)
network to remove the noise from a larger-scale BirdSoundsDenoising dataset.
In addition, we design an audio ImageMask tool and propose to use few-shot
generation to label all datasets. Extensive experimental results demonstrate
that the proposed DVAD model outperforms many state-of-the-art methods. As for
future work, a novel segmentation model could be developed to further improve
the audio denoising performance.

##  8 Broad impact

The application of our proposed DVAD model is not limited to bird sound
denoising. It can be easily extended to the following tasks for real-life
applications.

###  8.1 Adaptation to speech denoising

Speech denoising is increasingly important as speech has become a predominant
medium of daily communication and the main aspect of technology advancement.
We use bird sounds as training samples for our model, but our model can also
be applied to human speech denoising or even other non-audio signals. We
applied the pre-trained DeepLabV3 model to segment the speech audio image,
then converted the segmented image to get the denoised speech audio. As shown
in Fig. 8, the noise in human speech audio can be significantly reduced.
Therefore, our DVAD model demonstrates high-quality performance in human
speech audio denoising.

Figure 8: Example of speech denoising. Left is the original speech with noise
and the right is the denoised speech audio.

###  8.2 Audio separation

Audio separation tasks are important in many scenarios, such as high-quality
conference video production, surveillance system use, audio identification,
etc. In the birds sound denoising task, we treat it as a binary image
segmentation problem. For audio separation, we could treat it as a multi-class
image segmentation problem. As shown in Fig. 9, we can separate two different
bird sounds. We also added separated bird sounds in the supplementary
material. This aspect of our model is significant when trying to detect or
identify desired audio signal in a noisy mix. Hence, our DVAD model can be
easily applied to the audio separation problem.

Figure 9: Example of audio separation. Left is the audio image and right is
the overlay of audio image with two different segmented bird masks. Yellow
color is one bird, while blue color is another bird.

###  8.3 Audio enhancement

Audio enhancement has always been a challenging task since the noise signal
would also be enhanced if we did not properly remove noise signals. This
challenge presents in many models designed for speech enhancement. In our DVAD
model, we only preserve the clean signal mask. Hence, higher quality audio
enhancement can be achieved. Given the denoised audio $\hat{x_{t}}$, we can
enhance the audio by enlarging the denoised signal with $l\hat{x_{t}}$, where
$l$ is the number of times to enlarge the signal. As shown in Fig. 10, we can
enhance the pure bird signal by $l=200$ times. An enhanced audio example can
also be found in the supplementary material. Audio enhancement has a much
wider application, such as hearing aid, recording production, long-distance
signal transmission like cellular communication, etc.

Figure 10: Example of audio enhancement. The blue line is the original noise
signal, and the red line is the denoised signal. On the right, it is the
enhanced signal of 200 times of signal in the red line on the left. x and y
axes represent the length and magnitude of the signal, respectively. Figure
11: Two examples of noise estimation. The blue color is the noise audio, and
the cyan color is the estimated noise.

###  8.4 Noise estimation and audio identification

In denoised audio, we could still occasionally hear the noise since the noise
signal is difficult to be completely removed. Noise estimation will be useful
if we can learn the patterns of noise, then we can further remove noise from
the denoised audio. As shown in Fig. 11, we can estimate the noise signal by
using original noise audio to subtract the clean denoised signal. Sampling
these extracted noises can be used to further improve the quality of denoised
signals. Noise estimation is particularly useful for model training, it could
also be extended to audio identification. Noise in one scenario could become
intended signals in others [38]. As each audio signal has its own pattern,
learning the pattern of different audio signals can be useful to match
intended signals with those in training sets. This application is important in
medical applications, e.g., identifying disease-likely sound patterns from
regular sound patterns.

## References

  * [1] Houda Abouzid, Otman Chakkor, Oscar Gabriel Reyes, and Sebastian Ventura.  Signal speech reconstruction and noise removal using convolutional denoising audioencoders with neural deep learning.  Analog Integrated Circuits and Signal Processing, 100(3):501–512, 2019.
  * [2] Nasim Alamdari, Arian Azarang, and Nasser Kehtarnavaz.  Improving deep speech denoising by noisy2noisy signal mapping.  Applied Acoustics, 172:107631, 2021.
  * [3] Arian Azarang and Nasser Kehtarnavaz.  A review of multi-objective deep learning speech denoising methods.  Speech Communication, 122:1–10, 2020.
  * [4] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
  * [5] Steven Boll.  Suppression of acoustic noise in speech using spectral subtraction.  IEEE Transactions on acoustics, speech, and signal processing, 27(2):113–120, 1979.
  * [6] Jingdong Chen, Jacob Benesty, Yiteng Huang, and Simon Doclo.  New insights into the noise reduction wiener filter.  IEEE Transactions on audio, speech, and language processing, 14(4):1218–1234, 2006.
  * [7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [8] Tian Gao, Jun Du, Li-Rong Dai, and Chin-Hui Lee.  Snr-based progressive learning of deep neural network for speech enhancement.  In Interspeech, pages 3713–3717, 2016.
  * [9] Tian Gao, Jun Du, Li-Rong Dai, and Chin-Hui Lee.  Densely connected progressive learning for lstm-based speech enhancement.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5054–5058. IEEE, 2018.
  * [10] Francois G Germain, Qifeng Chen, and Vladlen Koltun.  Speech denoising with deep feature losses.  arXiv preprint arXiv:1806.10522, 2018.
  * [11] Juan S Gómez, Jakob Abeßer, and Estefanía Cano.  Jazz solo instrument classification with convolutional neural networks, source separation, and transfer learning.  In ISMIR, pages 577–584, 2018.
  * [12] John HL Hansen, Vinod Radhakrishnan, and Kathryn Hoberg Arehart.  Speech enhancement based on generalized minimum mean square error estimators and masking properties of the auditory system.  IEEE Transactions on Audio, Speech, and Language Processing, 14(6):2049–2063, 2006.
  * [13] Madhav Mahesh Kashyap, Anuj Tambwekar, Krishnamoorthy Manohara, and S Natarajan.  Speech denoising without clean training data: A noise2noise approach.  Proc. Interspeech 2021, pages 2716–2720, 2021.
  * [14] Qiuqiang Kong, Haohe Liu, Xingjian Du, Li Chen, Rui Xia, and Yuxuan Wang.  Speech enhancement with weakly labelled data from audioset.  arXiv preprint arXiv:2102.09971, 2021.
  * [15] Haoran Kui, Jiahua Pan, Rong Zong, Hongbo Yang, and Weilian Wang.  Heart sound classification based on log mel-frequency spectral coefficients features and convolutional neural networks.  Biomedical Signal Processing and Control, 69:102893, 2021.
  * [16] Andong Li, Minmin Yuan, Chengshi Zheng, and Xiaodong Li.  Speech enhancement using progressive learning-based convolutional recurrent neural network.  Applied Acoustics, 166:107347, 2020.
  * [17] Bingbing Li.  A principal component analysis approach to noise removal for speech denoising.  In 2018 International Conference on Virtual Reality and Intelligent Systems (ICVRIS), pages 429–432. IEEE, 2018.
  * [18] Jae Soo Lim and Alan V Oppenheim.  Enhancement and bandwidth compression of noisy speech.  Proceedings of the IEEE, 67(12):1586–1604, 1979.
  * [19] Rainer Martin.  Speech enhancement based on minimum mean-square error estimation and supergaussian priors.  IEEE transactions on speech and audio processing, 13(5):845–856, 2005.
  * [20] Eloi Moliner and Vesa Välimäki.  A two-stage u-net for high-fidelity denoising of historical recordings.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 841–845. IEEE, 2022.
  * [21] Se Rim Park and Jin Won Lee.  A fully convolutional neural network for speech enhancement.  Proc. Interspeech 2017, pages 1993–1997, 2017.
  * [22] Mozhde Firoozi Pouyani, Mansour Vali, and Mohammad Amin Ghasemi.  Lung sound signal denoising using discrete wavelet transform and artificial neural network.  Biomedical Signal Processing and Control, 72:103329, 2022.
  * [23] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and Martin Jagersand.  U2-net: Going deeper with nested u-structure for salient object detection.  Pattern recognition, 106:107404, 2020.
  * [24] Shivangi Raj, P Prakasam, and Shubham Gupta.  Multilayered convolutional neural network-based auto-codec for audio signal denoising using mel-frequency cepstral coefficients.  Neural Computing and Applications, 33(16):10199–10209, 2021.
  * [25] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [26] Nasir Saleem, Muhammad Irfan Khattak, Muhammad Yousaf Ali, and Muhammad Shafi.  Deep neural network for supervised single-channel speech enhancement.  Archives of Acoustics, 44, 2019.
  * [27] Nasir Saleem and Muhammad Irfan Khattak.  Deep neural networks for speech enhancement in complex-noisy environments.  2020\.
  * [28] Pascal Scalart et al.  Speech enhancement based on a priori signal to noise estimation.  In 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings, volume 2, pages 629–632. IEEE, 1996\.
  * [29] Samuel Sonning, Christian Schüldt, Hakan Erdogan, and Scott Wisdom.  Performance study of a convolutional time-domain audio separation network for real-time speech denoising.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 831–835. IEEE, 2020.
  * [30] Madhur Srivastava, C Lindsay Anderson, and Jack H Freed.  A new wavelet denoising method for selecting decomposition levels and noise thresholds.  IEEE access, 4:3862–3877, 2016.
  * [31] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7262–7272, 2021.
  * [32] Keiichi Tamura, Akitada Omagari, and Shuichi Hashida.  Novel defense method against audio adversarial example for speech-to-text transcription neural networks.  In 2019 IEEE 11th International Workshop on Computational Intelligence and Applications (IWCIA), pages 115–120. IEEE, 2019.
  * [33] Ke Tan and DeLiang Wang.  A convolutional recurrent neural network for real-time speech enhancement.  In Interspeech, volume 2018, pages 3229–3233, 2018.
  * [34] Tyler Vuong, Yangyang Xia, and Richard M Stern.  A modulation-domain loss for neural-network-based real-time speech enhancement.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6643–6647. IEEE, 2021.
  * [35] Hongyi Wang, Shiao Xie, Lanfen Lin, Yutaro Iwamoto, Xian-Hua Han, Yen-Wei Chen, and Ruofeng Tong.  Mixed transformer u-net for medical image segmentation.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2390–2394. IEEE, 2022.
  * [36] Kai Wang, Bengbeng He, and Wei-Ping Zhu.  Tstnn: Two-stage transformer based neural network for speech enhancement in the time domain.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7098–7102. IEEE, 2021.
  * [37] Li Wang, Weiguang Zheng, Xiaojun Ma, and Shiming Lin.  Denoising speech based on deep learning and wavelet decomposition.  Scientific Programming, 2021, 2021.
  * [38] Ruilin Xu, Rundi Wu, Yuko Ishiwaka, Carl Vondrick, and Changxi Zheng.  Listening to sounds of silence for speech denoising.  Advances in Neural Information Processing Systems, 33:9633–9648, 2020.
  * [39] Yong Xu, Jun Du, Li-Rong Dai, and Chin-Hui Lee.  A regression approach to speech enhancement based on deep neural networks.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(1):7–19, 2014.
  * [40] Hongjiang Yu, Zhiheng Ouyang, Wei-Ping Zhu, Benoit Champagne, and Yunyun Ji.  A deep neural network based kalman filter for time domain speech enhancement.  In 2019 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1–5. IEEE, 2019.
  * [41] Rui-Mei Zhao and Hui-min Cui.  Improved threshold denoising method based on wavelet transform.  In 2015 7th International Conference on Modelling, Identification and Control (ICMIC), pages 1–4. IEEE, 2015.


A 98 1 Yang YanmlabelA1111111 Qiuyan WangnlabelB[labelC] [labelA]The author is
with the Faculty of Tianjing University of Technology and Education
[labelB]The author is with the Faculty of Tiangong University

# ISHNE: Influence Self-attention for Heterogeneous Network Embedding

###### keywords:

Graph Attention Networks, Heterogeneous Networks, Graph Representation
Learning, Passive Influence Relation

††titlenote: This paper was presented at …

In recent years, Graph Neural Networks has received enormous attention from
academia for its huge potential of modeling the network traits such as
macrostructure and single node attributes. However, prior mainstream works
mainly focus on homogeneous network and lack the capacity to characterize the
network heterogeneous property. Besides, most previous literature cannot model
the influence under microscope vision, making it infeasible to model the joint
relation between the heterogeneity and mutual interaction within multiple
relation type. In this paper, we propose an Influence Self-attention network
to address the difficulties mentioned above. To model heterogeneity and mutual
interaction, we redesign attention mechanism with influence factor on the
single-type relation level, which learns the importance coefficient from its
adjacent neighbors under the same meta-path based patterns . To incorporate
the heterogeneous meta-path in a unified dimension, we developed a self-
attention based framework for meta-path relation fusion according to the
learned meta-path coefficient. Our experimental results demonstrate that our
framework not only achieve higher results than current state-of-the-art
baselines, but also show promising vision on depicting heterogeneous
interactive relations under complicated network structure.

##  1 Introduction

We are now living in a world where all connections come with graph structure,
such as traffic network, social network, etc. To characterize network topology
structure, vertex content and other information, graph embedding
representation theory is proposed. Graph embedding theory projects the whole
graph into a low-dimensional vector space in the form of deep learning
paradigm. Those methodologies include Node2vec [1], DeepWalk [2], Line [3],
etc.

In recent years , one special type of graph called heterogeneous information
graph (HIN for short) [4] has became a hot spot for network mining
researchers. The miscellaneous edges and nodes in heterogeneous graph
represent the complex relations such as recommendation system , paper citation
network etc. The most important feature of HIN is the meta path [5], which
shows the semantic relations in the node-edge tuple. Taking paper citation
network for example, the relation between two papers can be illustrated as
Paper-Author-Paper (coauthor relations) and Paper-Subject-Paper (peer
relations). From the tuple listed up, we can see the different patterns of
connection revel different relation in a heterogeneous graph. Previous
homogeneous graph deep network cannot handle the complex and across-pattern
interactions in the heterogeneous network.

In order to handle such difficulties, various methodology are proposed [2] [5]
[4]. However, those works still face several limitations. First, the problem
of hidden interactions, such as social influence have not been fully addressed
in previous studies, which means valuable social influence of hidden
interaction are neglected. A case in point is the paper citation network.If
author A has published many influential works, and B is a follower of
A.Surely, author A has latent impact on author B in such network. Second,
hidden engagement of adjacent nodes across meta-path relations have not been
discussed in previous studies. Although earlier approach has proposed semantic
level attention mechanism, their methods cannot address the latent influence
issue on ther proposed neural network model.

Based on the previous analysis, we propose a Influence Self-attention for
Heterogeneous Network Embedding framework (denoted as ISHNE) to model the
latent influence in the heterogeneous network in this letter. Our contribution
can be summarized as:

(1)To our best knowledge, this is the first attempt to study the latent impact
of heterogeneous network based on attention mechanism. Our work show its
promises on crossing meta-path relations in inductive learning where our model
achieves better result for the unseen nodes than the SOTA. (2)To integrate the
latent influence relations into graph learning framework, we propose a self-
attention model on hierarchical model to fuse the embedding space containing
social influence from different meta-path relations. (3)We conduct experiments
on real heterogeneous graph dataset to evaluate the performance of ISHNE. The
experimental results show its superiority comparing with the the state-of-the
art methodology. Our method also demonstrates its interruptibility in its
result analyze.

##  2 Related Works

###  2.1 Heterogeneous Information Network

To tackle the ubiquitous and pervasive multi-modal interactions in real world,
heterogeneous networks have been proposed and widely used in numerous network
mining scenarios. In HIN [6] , meta-paths are proposed to define the semantic
indirect relations in data ming scenarios such as classification [7],
clustering [8] [9], recommendation [10] [11] and outlier detection [12]. elf

###  2.2 HIN Embedding

In order to depict the complex interactions regarding multi-typed links and
higher-order meta-path, numerous methodologies are proposed. Those approaches
mainly fall on three categories: proximity-preserving methods, message-passing
methods and relation-learning methods. The goal of proximity-preserving
approach is to contain different types of proximity among nodes. To achieve
this, metapath2vec [2] proposes a meta-path guided random walks model to learn
the context of node regarding heterogeneous semantics. Hin2vec [13] considers
the possibility of the meta-path between two node s and generate positive and
negative tuple on the path generated by random walk algorithm. PTE [14]
decomposes the heterogeneous network into bipartite networks, each of which
describes one edge type. Message-passing methods learn the embedding of
attributes via aggregating the information from its adjacent neighbors. Graph
Neural Networks [15] are widely used in those approaches. R-GCN [16] utilizes
multiple convolutional layer. At each convolutional layer, representation
vector is updated by accumulating its vectors of neighboring nodes. HAN [4]
uses second order proximity to model meta-path relations. The neighbor’s
weight coefficient are learned by the attention mechanism. Relation learning
methods model each edge as a tuple and design a scoring function. TransE [17]
designs translating form of the embedding to minimize the margin-based ranking
loss. ConvE [18] designs a neural network model to score for the relation
tuple. In summary, although many aforementioned literature has discussed HIN
embedding, the effect of latent influence mechanism has not addressed in
previous heterogeneous graph representation learning model.

##  3 Preliminaries

In this section, we will specify the definition of HIN and our framework.

Definition 1: Heterogeneous Graph. A heterogeneous graph is defined as a
network associated with multiple types of nodes and edges. Heterogeneous graph
can be mathematically defined as $G=\\{\mathcal{V},\mathcal{E}\\}$, where
$\mathcal{V}$ and $\mathcal{E}$ respectively denotes the set of nodes and
edges. Each element of $\mathcal{V}$ is associated with node type function:
$f_{N}:\mathcal{V}\rightarrow\mathcal{A}$. Likewise, function
$f_{E}\mathcal{\epsilon}\rightarrow\mathcal{R}$ denotes the link mapping.
Here, $\mathcal{A}$ and $\mathcal{R}$ symbolizes node and link type.

Figure 1 is an example of heterogeneous graph to for ACM author-paper-subject
relations heterogeneous network. As shown in Figure 1(A) and (B), there are
three different types of nodes (Authors, Papers, Subjects) and two types of
linking relations (written-in relation between papers and authors, published-
in relation between papers and subjects).

Figure 1: Figure 1: An example of heterogeneous graph. Figure 1(A): Three
types of nodes in heterogeneous graph. Figure 1(B): Heterogeneous graph
consists of multiple nodes and edges. Figure 1(C): Two example of meta-paths.

Definition 2: Meta-path relations. A standard form of meta-path relations is
$V_{1}\stackrel{{\scriptstyle
R_{1}}}{{\longrightarrow}}V_{2}\stackrel{{\scriptstyle
R_{2}}}{{\longrightarrow}}V_{3}$, which characterize the relation type
$R_{1}R_{2}$ series between the vertice $V_{1}V_{2}V_{3}$.

As Figure 1(C) illustrated, two papers can be indirectly connected via two
different meta-paths: Paper-Author-Paper (denoted as PAP) and Paper-Subject-
Paper (denoted as PSP). In Figure 1(C), we can see that node $A$ and node $B$
are connected by meta-path PAP. Likewise, node $C$ and $D$ are connected by
meta-path PSP. Here, we call $A-B$ and $C-D$ are meta-path neighbors on PAP
and PSP. Based on the above preliminary knowledge, we will present a novel
neural network, which enables us to exploit the latent influence in the
heterogeneous network. All the symbol we use in this paper are summarized in
Table 1.

Table 1: Notation and Explanations Notation | Explanation
---|---
$\phi$ | single meta-path
h | Initial node feature
$\textbf{M}_{\phi}$ | Transformation matrix for type $\phi$
$\textbf{P}_{\phi}$ | influence feature Transformation Matrix
$\textbf{h}^{p}$ | Projected influence features vector
$e_{ij}^{\Phi}$ | importance coefficient on meta-path $\Phi$ for nodes $i$ and $j$
$\textbf{a}_{\Phi}$ | neighborhood attention vector for meta-path
$a_{ij}^{\Phi}$ | Weight coefficient meta-path $\Phi$ for nodes $i$ and $j$
$\textbf{N}_{i}^{\Phi}$ | Meta-path based neighbors
$\textbf{x}_{\Phi}$ | Semantic relation embedding
$w_{\Phi}$ | Weight coefficient Semantic relation embedding
X | Final embedding

##  4 Our Model

In this section, we will describe our proposed Influence Self-embedding for
Heterogeneous Network Embedding framework. The basic idea of ISHNE is to
project every node into different meta-path based embedding space. To achieve
this, local proximities and individual influence of each node are calculated
and integrated by aggregating the meta-path neighborhood information. The
comprehensive embedding of the vertex are fused from all its meta-path
embedding by self-attention. We elaborately design the architecture of ISHNE
shown in Figure 2.

Figure 2: Figure 2: The framework of ISHNE. (A) Nodes on all meta-paths are
projected into a separate space with its influential feature.(B) Embedding on
meta-paths are merged with self-attention module.(C) Compute the loss in
classification task with our ISHNE

###  4.1 Meta-Path Based Neighborhood Influence Attention

Before merging embedding results from each semantic component we should learn
that all nodes interacts with its meta-path neighbors. Here, we define our
neighborhood influence attention model. For each type of HIN meta path $\phi$,
we introduce transformation matrix $M_{\phi}$ to project the node $i$ into the
corresponding space of meta-path $\phi$. The projection function is defined as
:

$\displaystyle\textbf{h}_{i}^{\prime}=\textbf{M}_{\phi}\cdot\textbf{h}_{i}$
(1)

Here $h_{i}^{\prime}$ and $h_{i}$ are projected and original feature vector of
node $i$. Different from [4], we introduce neighborhood influence factor into
the attention mechanism. Specifically, given a node pair $(i,j)$ which is
connected by meta-path $\phi$ in academic network. As we stated in Section 1,
from the perspective of node $j$, node $i$ certainly has some latent effect on
him, especially when node $i$ is a well-known researcher. To character such
phenomenon, we impose a influence component on attention computing mechanism.
The influence component is computed as:

$\displaystyle\textbf{h}^{p}_{i}=\textbf{P}_{\phi}\cdot\textbf{h}_{i}$ (2)

After that, we leverage the weight coefficient for $(i,j)$ on meta-path
$a^{\phi}_{ij}$ in the form of:

$\displaystyle
a_{ij}^{\phi}=\frac{exp(\sigma(\textbf{a}^{T}_{\phi}\cdot[\textbf{h}^{\prime}_{i}||(\textbf{h}^{\prime}_{j}+\textbf{h}^{p}_{i})]))}{\sum\limits_{k\in
N^{\phi}_{i}}exp(\sigma(\textbf{a}^{T}_{\phi}\cdot[\textbf{h}^{\prime}_{i}||(\textbf{h}^{\prime}_{k}+\textbf{h}^{p}_{i})]))}$
(3)

Here $\sigma$ denotes the activation function. $|\cdot|$ operator denotes the
concatenating operator. $\textbf{a}_{\phi}$ denotes neighborhood influence
coefficient on meta path $\phi$. On th right part of Equation 3, we add
$\textbf{h}^{p}_{i}$ to $\textbf{h}_{j}^{\prime}$ to impose the latent
influence on node $j$ from node $i$ . We note that coefficient $a_{ij}$ is
asymmetric because node $i$ and $j$ have different impact to each other. For
node $i$, we will aggregate the feature vector of its meta-path based
neighbors in $\textbf{N}_{i}^{\Phi}$ in the form of Equation 4 .

$\displaystyle\textbf{x}_{i}^{\phi}=\sigma(\sum\limits_{j\in
N_{i}^{\phi}}a_{ij}^{\phi}\cdot\textbf{h}_{j}^{\prime})$ (4)

In order to capture the complicated, rich nature of node features and
stabilize the training results, we adopt the multihead attention. Here, we
repeat $K$ times like Equation 5 to concatenate the embedding vectors as the
input of semantic relation self-attention model in subsection 4.2 .

$\displaystyle\textbf{x}_{i}^{\phi}=\|_{k=1}^{K}\sigma(\sum\limits_{j\in
N_{i}^{\phi}}a_{ij}^{\phi}\cdot\textbf{h}_{j}^{\prime})$ (5)

Given the meta-path set $\\{\Phi_{0},\Phi_{1},...,\Phi_{P}\\}$, after
neighborhood influence attention, we get $P$ group for semantic relation
embedding for each node, denoted as
$\textbf{X}_{\phi_{0}},\textbf{X}_{\phi_{1}},,...,\textbf{X}_{\phi_{p}}$.

###  4.2 Semantic Relation Self-Attention Model

Considering each node can have multiple meta-based relations, it is necessary
to merge the embedding from every meta-path based relations $\phi$. We here
adopt the self-attention mechanism to address the problem of merging the
latent influence into a unified semantic space. To learn the importance of
each meta-path, we device a self-attention based meta-path embedding merging
approach. First, the query matrix $Q$, key matrix $K$ and value matrix $V$ are
computed in Equation 6.

$\displaystyle\textbf{Q}=\textbf{W}_{Q}\cdot\textbf{x}_{i}^{\phi}~{}~{}~{}~{}~{}\textbf{K}=\textbf{W}_{K}\cdot\textbf{x}_{i}^{\phi}~{}~{}~{}~{}~{}\textbf{V}=\textbf{W}_{V}\cdot\textbf{x}_{i}^{\phi}$
(6)

Next, we leverage the importance of all the semantic relation self-attention
node embedding which can be explained as the importance of each meta-path. For
each meta-path $\phi$, the importance of meta path $\textbf{w}_{\phi}$ is
computed as Equation 7.

$\displaystyle w_{\phi_{i}}=\frac{1}{|V|}\sum\limits_{i\in
V}\textbf{q}^{T}\cdot\mathit{softmax}(\frac{\textbf{Q}\cdot\textbf{K}^{T}}{\sqrt{d}})\cdot\textbf{V}$
(7)

Here, $|V|$ denotes the number of nodes in meta-path $\phi$, $d$ is the $2$nd
of dimension, q is semantic relation self-attention vector. The weight
$\beta_{\phi_{i}}$ of meta-path $\phi_{i}$ is computed as:

$\displaystyle\beta_{\phi_{i}}=\frac{exp(w_{\phi_{i}})}{\sum\limits_{i=1}^{P}exp(w_{\phi_{i}})}$
(8)

With the learned weight of each meta-path, we can obtain the final result of
semantic embedding in Equation 9.

$\displaystyle\textbf{X}=\sum\limits_{i=1}^{P}\beta_{\phi_{i}}\cdot\textbf{X}_{\phi_{i}}$
(9)

The final embedding is applied to a semi-supervised node classification task.
We utilize Cross-Entropy loss function in the task in Equation 10

$\displaystyle
L=-\sum\lim\limits_{l\in\textbf{y}_{l}}Y^{l}ln(C\cdot\textbf{Z})$ (10)

Here, $C$ is the parameter of the classifier, $y_{l}$ is the set of node
indices that have labels, $Y^{l}$ and $X^{L}$ are the labels of nodes and
embedding of labeled nodes respectively.

##  5 Experiments

We use ACM and IMDB as experimental datasets. ACM dataset comprises 3025
papers, 5835 authors and 56 subjects, which constitutes two types of meta-
path: $PAP$ (Paper-Author-Paper) and $PSP$ (Paper-Subject-Paper). Paper
features correspond to elements of a bag-of-words represented of keywords.
IMDB dataset contains 4780 movies, 5841 actors and 2269 directors. IMDB
dataset has three category for the movie nodes: Action, Comedy, Drama. Meta-
paths $MAM$ (Movie-Actor-Movie) and $MDM$ (Movie-Director-Movie) are
incorporated in semantic relations. Movie features employ correspond to
elements of a bag-of- words. We use Macro-F1 and Macro-F1 as evaluation
metrics. We split whole data into three parts. For ACM, $600$ papers are for
training set. $300$ papers are for validation set. The rest are for testing
set. For IMDB, $300$ movies are for training set. $300$ movies are for
validation set. The rest are for testing set.

Table 2: Micro-F1 results(%) Node classification tasks Datasets | GCN | GAT | HAN | ISHNE
---|---|---|---|---
ACM | 75.67 | 26.40 | 82.26 | 83.86
IMDB | 45.78 | 27.99 | 50.58 | 53.44
Table 3: Macro-F1 results(%) Node classification tasks Datasets | GCN | GAT | HAN | ISHNE
---|---|---|---|---
ACM | 75.60 | 19.26 | 81.92 | 83.44
IMDB | 45.31 | 22.49 | 38.71 | 51.00

Here, GCN [15] and GAT [19] are Homogeneous graph embedding approaches. We
perform experiments on all of the meta-paths based homogeneous graphs and pick
the best results. From Table 2 and 3, we can see that our ISHNE achieves
$83.86~{}\%$ and $53.44~{}\%$ Micro-F1 on ACM and IMDB datasets respectively.
It also achieves $83.44~{}\%$ and $51.00~{}\%$ Macro-F1 on two datasets. We
can see the heterogeneous graph methods such as HAN and ISHNE surpass
homogeneous graph approaches such as GCN and GAT. Compared with HAN [4], our
ISHNE incorporates latent influence in the neighborhood attention computation
and fusing meta-path relation process. Such improvements help us to achieve
better results than traditional other attention based mechanism. We also
noticed that on IMDB, ISHNE performs significant superiority than on ACM
dataset. This is mainly because ISHNE is capable of mining rich latent
relation on IMDB.

Through the above analysis, we can find that the proposed SHNE achieves the
best performance on all datasets. The results show that it is quite important
to capture the latent impact of nodes and meta-paths in heterogeneous graph
analysis.

##  6 Conclusion

This letter proposes a influence self-attention network embedding mechanism to
tackle the problem of characterizing latent influence relations in
heterogeneous graph embedding problem. Experimental results demonstrate that
our approach achieves more satisfying results than SOTA.

## References

  * [1] A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” Proc. of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp.855–864, 2016.
  * [2] B. Perozzi, R. Al-Rfou, and S. Skiena, “Deepwalk: Online learning of social representations,” Proc. of the 20th ACM SIGKDD international conference on knowledge discovery and data mining, pp.701–710, 2014.
  * [3] J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei, “Line: Large-scale information network embedding,” Porc. of The World Wide Web Conference, pp.1067–1077, 2015.
  * [4] X. Wang, H. Ji, C. Shi, B. Wang, Y. Ye, P. Cui, and P.S. Yu, “Heterogeneous graph attention network,” Proc. of The World Wide Web Conference, pp.2022–2032, 2019.
  * [5] Y. Dong, N.V. Chawla, and A. Swami, “metapath2vec: Scalable representation learning for heterogeneous networks,” Proc. of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp.135–144, 2017\.
  * [6] Y. Sun and J. Han, “Mining heterogeneous information networks: principles and methodologies,” Synthesis Lectures on Data Mining and Knowledge Discovery, vol.3, no.2, pp.1–159, 2012.
  * [7] M. Ji, Y. Sun, M. Danilevsky, J. Han, and J. Gao, “Graph regularized transductive classification on heterogeneous information networks,” Proc. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp.570–586, 2010.
  * [8] Y. Sun, C.C. Aggarwal, and J. Han, “Relation strength-aware clustering of heterogeneous information networks with incomplete attributes,” arXiv preprint arXiv:1201.6563, 2012.
  * [9] Y. Sun, B. Norick, J. Han, X. Yan, P.S. Yu, and X. Yu, “Pathselclus: Integrating meta-path selection with user-guided object clustering in heterogeneous information networks,” ACM Transactions on Knowledge Discovery from Data, vol.7, no.3, pp.1–23, 2013.
  * [10] T. Chen and Y. Sun, “Task-guided and path-augmented heterogeneous network embedding for author identification,” Proc. of the 10th ACM International Conference on Web Search and Data Mining, pp.295–304, 2017.
  * [11] X. Yu, X. Ren, Y. Sun, Q. Gu, B. Sturt, U. Khandelwal, B. Norick, and J. Han, “Personalized entity recommendation: A heterogeneous information network approach,” Proc. of the 7th ACM international conference on Web search and data mining, pp.283–292, 2014.
  * [12] M. Gupta, J. Gao, and J. Han, “Community distribution outlier detection in heterogeneous information networks,” Proc. of Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp.557–573, 2013.
  * [13] T.y. Fu, W.C. Lee, and Z. Lei, “Hin2vec: Explore meta-paths in heterogeneous information networks for representation learning,” Proc. of the 2017 ACM on Conference on Information and Knowledge Management, pp.1797–1806, 2017.
  * [14] J. Tang, M. Qu, and Q. Mei, “Pte: Predictive text embedding through large-scale heterogeneous text networks,” Proc. of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp.1165–1174, 2015.
  * [15] T.N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
  * [16] M. Schlichtkrull, T.N. Kipf, P. Bloem, R. Van Den Berg, I. Titov, and M. Welling, “Modeling relational data with graph convolutional networks,” Proc. o f the European semantic web conference, pp.593–607, 2018.
  * [17] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko, “Translating embeddings for modeling multi-relational data,” Proc. of Neural Information Processing Systems, pp.1–9, 2013.
  * [18] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel, “Convolutional 2d knowledge graph embeddings,” Proc. of the AAAI Conference on Artificial Intelligence, 2018.
  * [19] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” arXiv preprint arXiv:1710.10903, 2017\.

Yang Yan (Ph.D. 2018) received the Ph.D. degree from University of Chinese
Acedemy of Sciences in 2016. He is currently a lecturer at School of
Information Technology Engineering, Tianjin University of Technology and
Education. His research interests include Social Network, Data mining, coding
theory and cryptography.

Qiuyan Wang (Ph.D. 2016) received the Ph.D. degree from University of Chinese
Acedemy of Sciences in 2016\. She is currently a lecturer at School of
Computer Sciences and Technology, Tiangong University. Her research interests
include coding theory and cryptography.


# Audio Spectral Enhancement: Leveraging Autoencoders for Low Latency
Reconstruction of Long, Lossy Audio Sequences

Darshan Deshpande<EMAIL_ADDRESS>
Department of Computer Engineering
Thadomal Shahani Engineering College
Mumbai University
Mumbai, India. Harshavardhan Abichandani
<EMAIL_ADDRESS>
Department of Computer Engineering
Thadomal Shahani Engineering College
Mumbai University
Mumbai, India

###### Abstract

With active research in audio compression techniques yielding substantial
breakthroughs, spectral reconstruction of low-quality audio waves remains a
less indulged topic. In this paper, we propose a novel approach for
reconstructing higher frequencies from considerably longer sequences of low-
quality MP3 audio waves. Our technique involves inpainting audio spectrograms
with residually stacked autoencoder blocks by manipulating individual
amplitude and phase values in relation to perceptual differences. Our
architecture presents several bottlenecks while preserving the spectral
structure of the audio wave via skip-connections. We also compare several task
metrics and demonstrate our visual guide to loss selection. Moreover, we show
how to leverage differential quantization techniques to reduce the initial
model size by more than half while simultaneously reducing inference time,
which is crucial in real-world applications.

Keywords: Audio Reconstruction, Audio Enhancement, Autoencoders, Quantization,
Short-Time Fourier Transform

##  1 Introduction

Data compression is utilised in a variety of ways, from streaming music to
transferring text files, but it has had the greatest impact on the audio and
image domains. Among all audio compression algorithms, MP3 is a well-known
codec, primarily because it drastically decreases the file size while having
little effect on audio quality. It uses a perceptual model which removes
frequencies that are imperceptible to humans, usually involving the high
frequencies. The way it does this is by changing the input from the time
domain to the frequency domain by using FFT and subsequently determining the
threshold values of these frequencies. After this, Huffman encoding is applied
to further compress the file by assigning lower number of bits to the
frequently occurring frequencies. MP3 is a lossy compression format since it
removes information from the audio that is difficult to recover. The loss of
information is not perceptible by humans at high enough bitrates, but at low
bitrates artifacts like pre-echo are noticeable.

Some techniques have been established previously, in which the algorithm would
iteratively estimate the phase source and reconstruct the audio (Gunawan and
Sen, 2010; Mowlaee et al., 2012; Krawczyk and Gerkmann, 2014). These
techniques of estimating the phase show considerable improvement in the SSNR
and PESQ score, but all of them ignore other characteristics of the audio like
the magnitude and power spectrum (Tan and Wang, 2019). In recent times machine
learning methods have been used to outperform these algorithmic techniques.
CNN models were utilised, with the real and imaginary spectrograms of the
audio serving as separate input channels for the model (Fu et al., 2017). The
resultant spectrograms were then used for the reconstruction of the signal,
this type of reconstruction outperformed the algorithmic approaches.

Convolutional Recurrent Neural Network (CRN) model proposed by (Tan and Wang,
2019) had a substantially higher STOI and PESQ value compared to the CNN based
approach mentioned above (Fu et al., 2017). Later, (Kuleshov et al., 2017)
proposed a technique in which time-series data is fed into an autoencoder with
residual connections, which reconstructs the high resolution audio with a
sampling rate higher than the low resolution audio. This model was inspired by
the image super resolution paper (Dong et al., 2016) and demonstrates an
improvement of about 5dB in some circumstances (Kuleshov et al., 2017). Resnet
and Unets (Sulun and Davies, 2021) have also been tested for the task of
bandwidth extension. Multiple low pass filters were utilised to enhance the
data, allowing these models to generalize effectively on new unseen filters
(Sulun and Davies, 2021). The code for the sections that follow can be found
at: https://github.com/DarshanDeshpande/audio-spectral-enhancement.

##  2 Processing the Data

The usage of extended temporal sequences is the fundamental focus of our
research (above 100,000 data points per training sample). Because of the
limited input size and repeated sweeps of segments through the decoder network
that are necessary to rebuild lengthier audio sequences, upscaling with some
previous reconstruction algorithms like (Pascual et al., 2017) is
significantly slower. Although parallelism helps speed up the process, larger
audio files still take a long time to process. To help mitigate this major
constraint, one of the most practical solutions is to analyse and reconstruct
larger audio sequences in a single inference which is what we use for our
proposed model. This section provides details about the creation of training
data and the structuring of the pipeline.

###  2.1 Dataset

We use the GTZAN (Tzanetakis and Cook, 2002) dataset as a base for our
experiments. It includes 1,000 WAV tracks, each of which lasts 30 seconds and
are sampled at 22050 Hz. Each of these songs was divided into three parts,
augmenting the total number of songs in the dataset to 3,000. Using the LAME
encoder (Cheng, 1998) and the FFMPEG library(Tomar, 2006), all 3,000 mono
16bit WAV files were first converted to MP3 files at a constant bitrate of
128kbps. We then downscaled the 128kbps audio files to 32kbps audio files
using the same LAME encoder and the FFMPEG library(Tomar, 2006). We have used
$85\%$ of the data for training, $8\%$ data for validation and $7\%$ data for
testing.

###  2.2 Loading the Data

To accurately capture the differences between high and low bitrate audio, the
data was preprocessed using the Short Time Fourier Transform (STFT). The STFT
function returns a tensor of complex STFT values containing information about
the signal’s phase and amplitude, which was then separated into real and
imaginary parts and stacked on top of each other. Stacking was done because
the real and imaginary values are directly mappable to the new real and
imaginary values, respectively.

We experimented with various frame length and hop size configurations to
retrieve the most data while minimizing processing time. We found that a
Hanning window length of 1023, hop size of 248, a frame length of 1024 without
padding or centering provided the best results. This configuration gives us
400 frames and 512 frequency bins for every sample of 100,000 data points. The
hop size should be small and the frame length should be large to achieve
accurate inversion. Lowering the hop size to a very small value increases the
time resolution but it is very computationally heavy. This leads to increased
inference time and as a result, our setup finds a good compromise between the
two.

##  3 Model Creation

Popular purely convolutional networks such as ResNets (He et al., 2016) and
UNets (Ronneberger et al., 2015), rely heavily on residual connections and
lower dimensional projections. When training data is scarce and fast inference
is of high priority, our experiments revealed that these large models are
significantly less efficient. In our study, we found that 2-D Convolutional
UNets (Ronneberger et al., 2015) consistently produced poor latent
representations, drawing more attention to the background noise instead of the
missing frequencies. Furthermore, 1-D UNet based models like Wave-U-Net
(Stoller et al., 2018), Temporal FiLM (Birnbaum et al., 2019) and Audio-SR-Net
(Sulun and Davies, 2021) have only been tested for sequences with 10,000 or
fewer data points per sample. We discovered that 1-D convolutions applied to
training samples temporally ten times that size are extremely inefficient and
computationally costly to train. The subsections that follow detail the
experiments and model designs that were employed.

###  3.1 Model Architecture

Since one-dimensional models are unusable as discussed above, it is more cost
and time effective to directly convolve on the stacked STFT output. Previously
mentioned 2-D UNet architecture (Ronneberger et al., 2015) was designed with a
fixed positive range of 0 to 1 for images in mind. Since we had to scale our
data in order to feed it to the 2-D UNet model (Ronneberger et al., 2015), we
had two options for dealing with the issue of negative values in the scaled
data: the first was to change the range of our data to 0 to 1, and the second
was to use activation functions like tanh, which support negative values. In
comparison to the first method, which resulted in insufficient weighting of
negative values, our comparative trials demonstrated that the second strategy
resulted in significantly improved spectral structure retention. Both the
PReLU (He et al., 2015) and tanh functions work well with the 2-D U-Net
(Ronneberger et al., 2015), but they quickly overfit due to the large number
of trainable parameters and small dataset size, resulting in an extremely
saturated spectral plot and poor audio output.

Figure 1: A single Depthwise Separable Convolution Block(B) and it’s
arrangement as a residually connected autoencoder network

Our proposed model consists of a two-layer 2-D depthwise separable
convolutional (SIfre and Mallat, 2014) block, as seen in figure 1. This
architecture functions as a tiny autoencoder unit with a 256-dimensional
latent space. Each block uses a PReLU (He et al., 2015) activation which
prevents the gradients from exploding while performing significantly better
than tanh function in our experiments. With PReLU (He et al., 2015) and
$\alpha$ parameter initialization of zero, the model attains more freedom to
explore negative values which is not possible with the standard ReLU or
sigmoid activation functions. This activation function is sandwiched between
separable convolutional layers (SIfre and Mallat, 2014), each with a filter
size of $5\times 5$ and step size of 1. Our model uses constant padding which
persists the input rows in the output, along with a randomly uniform
initialization in accordance with the initialization proposed by (He et al.,
2015). The use of separable convolutions (SIfre and Mallat, 2014) rather than
general spatial convolutions stems from their low likelihood of overfitting
(SIfre and Mallat, 2014), as demonstrated by our experiments with considerably
higher trainable parameters and various custom experiments that are shown in
table 4.2. Each block can be stacked N times with skip connections as shown in
figure 1 to provide effective protection against the vanishing gradient
problem (He et al., 2016). A significant observation is that with the increase
of N, the coherence of the output audio decreases despite the phase and power
spectra remaining consistent. Through our experiments, we could also conclude
that a model consisting upto seven autoencoder blocks does not lead to
overfitting on our experiments with our augmented GTZAN dataset (Tzanetakis
and Cook, 2002). This explains the robust architectural design of the
configuration primarily due to the use of separable convolutions (SIfre and
Mallat, 2014).

Figure 2: Outputs of different loss functions

###  3.2 Training Losses

Our experimentation with loss functions involved permuting between six losses
and testing their combinations. The figure 2 provides a summary of the
individual metric tests. Considering these effects, we propose a weighted
combination of two losses for our training process. First is a modified
version of the sum of squared residuals (SSE) which is scaled by the variance
of the ground truth label.

$\text{Pixel
Loss}(y_{i},\hat{y_{i}})=\displaystyle\frac{\sum_{i=1}^{n}{(y_{i}-\hat{y_{i}}})^{2}}{\sum_{i=1}^{n}{(y_{i}-\bar{y_{i}}})^{2}}$
(1)

This loss function provides a much better spectral structure as compared to
the standard Mean Squared Error (MSE) for unscaled data or Kullback-Leibler
Divergence (KDL) for scaled data which produced faded textures. Similarly,
Mean Absolute Error (MAE) produced results that neglected the higher
frequencies altogether because of it’s inherent nature of ignoring outliers.
Some commonly employed spectral reconstruction losses, such as Log Spectral
Loss (Hu and Loizou, 2008), resulted in oversaturated low-frequency regions
that were not affected by compression, to begin with, as seen in figure 2.

The second loss that we use is the Structural Similarity Index (SSIM) (Wang et
al., 2004). The idea for this function is derived from its large applicability
in domains such as image reconstruction and image super-resolution. SSIM
highly influences the spectral luminosity and contrast of the output
spectrogram and ensures that the higher frequency intensity ranges are equally
stressed on unlike the Log Spectral Loss (Hu and Loizou, 2008)

$\text{SSIM}(y_{i},\hat{y_{i}})=\displaystyle\frac{(2\mu_{y_{i}}\mu_{\hat{y_{i}}}+C_{1})(2\sigma_{y_{i}\hat{y_{i}}}+C_{2})}{(\mu_{y_{i}}^{2}+\mu_{\hat{y_{i}}}^{2}+C_{1})(\sigma_{y_{i}}^{2}+\sigma_{\hat{y_{i}}}^{2}+C_{2})}$
(2)

Here, the SSIM loss is calculated on the magnitude of the output.
$C_{1}=(k_{1}L)^{2}$ and $C_{2}=(k_{2}L)^{2}$ are constants for stabilization
determined by the dynamic pixel value ranges, L, and pre-determined constants
$k_{1}$, $k_{2}$. Changing the value of $k_{1}$ even slightly, changes the
luminosity by a large factor and hence this value is most optimal if set
between 0.01 and 0.005. Additionally, altering the value of $k_{2}$ gives a
better definition to the output spectral plot at the cost of low intensity of
higher frequencies. Thus, the appropriate value of $k_{2}$ can range from 0.01
to 0.03. The value of L should be set to less than $1/2$ of the actual pixel
range to allow for better luminosity. (Wang et al., 2004) recommend that the
technique should be applied locally for better results. Our exhaustive
experiments conclude that applying the loss using a smaller convolutional
window size of $3\times 3$ improves the structural detail of the output and
prevents checkerboard patterns.

###  3.3 Training

The training process involves using the Adam (Kingma and Ba, 2014) optimizer
and takes approximately 200 epochs to converge with a learning rate of
$1\times 10^{3}$ and a mini batch size of 32. Other popular optimizers, such
as mini-batch SGD and RMSProp, are prone to get trapped in local minima,
resulting in insufficient loss landscape mapping. The training time heavily
depends on the model configuration, number of parameters and the value of N
mentioned in subsection 3.1. Additionally, a cyclical learning rate with a
decay factor of $5\times 10^{-4}$ can assist in providing a smoother descent.

At every step, the losses are weighted as follows:

$\text{Total Loss}=\displaystyle\text{Pixel Loss}+0.5\times\text{SSIM}$ (3)

This weighting is used to counteract SSIM’s tendency to dominate the
spectrogram by filling it with unnecessary high-intensity pixels. This
prevents the spectrogram from being overly saturated and producing a noisy
audio output.

###  3.4 Model Quantization

Since the primary goal of this research is to provide a low latency solution
for spectral mapping, we adopt quantization techniques (Jacob et al., 2018) to
shrink the model size as well as decrease inference times. We used a dynamic
8-bit precision quantization to achieve a compression of up to 52.15%,
resulting in a compressed size of 956KB per block. In this quantization
technique, the activations are dynamically quantized and un-quantized between
8-bit and float precisions during and after training respectively, while the
weights are stored with 8-bit precision. The differences in spectrograms after
compression is minuscule, with 16-bit float quantization techniques retaining
the complete spectral structure and the 8-bit integer quantization techniques
altering the intensity of the higher frequencies just slightly, while ensuring
the persistence of the quality of the audio output.

##  4 Evaluation

The evaluation section is divided into two parts. First, the spectrograms of
high, low and approximated audio signals are compared and finally, some of the
most common objective and perceptual based metrics are used to evaluate the
audio signals.

###  4.1 Spectrograms

Inverse STFT of the approximated signal was taken and then its frequency,
phase, magnitude and power spectrograms were plotted against the high bitrate
audio signal as shown in figure 4. Figure 3 shows that the reconstructed
spectrogram has kept the structure of higher frequencies. Figure 4 shows a
two-sided phase spectrogram that displays how the reconstructed phase differs
just slightly from the high bitrate audio.

Figure 3: From left to right: low bitrate audio spectrogram, high bitrate
audio spectrogram and reconstructed audio spectrogram

Figure 4: From left to right: magnitude spectrogram, power spectrogram and
two-sided phase spectrogram

###  4.2 Metrics

The performance of the reconstructed audio signal $\hat{y}$ in comparison to
the high bitrate audio signal $y$ was evaluated using the following metrics.

  1. 1.

Signal to Noise Ratio (SNR)

$\text{SNR}(y,\hat{y})=\displaystyle
10\cdot\log_{10}\frac{||y||_{2}^{2}}{||y-\hat{y}||_{2}^{2}}$ (4)

Signal to Noise ratio is the most widely used metric for analyzing the clarity
of the audio signals. The higher the value of the ratio, the better is the
audio quality.

  2. 2.

Perceptual Evaluation of Speech Quality (PESQ)
It is calculated by taking the linear combination of the average disturbance
values $D_{ind}$ and the average asymmetrical disturbance values $A_{ind}$ (Hu
and Loizou, 2008; Rix et al., 2001).

$\text{PESQ}(y,\hat{y})=a_{0}+a_{1}D_{ind}+a_{2}A{ind}$ (5)

We have taken all the default values of $a_{0}$, $a_{1}$, and $a_{2}$ and used
a wideband configuration with the sampling frequency of 16000Hz as mentioned
in (Wang, 2021).

  3. 3.

Log-Spectral Distance (LSD)

$\text{LSD}(y,\hat{y})=\frac{1}{L}\sum\limits_{l=1}^{L}\sqrt{\frac{1}{K}\sum\limits_{k=1}^{K}(S(k)-\hat{S}(k))^{2}}$
(6)

here $S$ and $\hat{S}$ are calculated by first taking the STFT of $y$ and
$\hat{y}$ respectively (Gray and Markel, 1976). For the STFT, a window length
of 2048 was chosen, followed by the log of the square of absolute values of
the STFT. Here $K$ and $L$ represents the frequencies and the total number of
frames respectively.

  4. 4.

Short-Time Objective Intelligibility (STOI)
STOI was calculated based on the parameters mentioned in (Taal et al., 2010,
2011) with the high bitrate audio, the reconstructed audio and the sampling
rate equal to $44100Hz$ as the inputs to the function.

Table 1: Inference time in milliseconds and metric comparison of different models. | Audio-SR-Net | FFmpeg | TFiLM | WaveNet | N = 1 | N = 5
---|---|---|---|---|---|---
| (27.7ms) |  | (75.9ms) | (86.9ms) | (7.74ms) | (20.8ms)
SNR | 16.7353 | 18.0077 | 9.9723 | 13.6910 | 21.2532 | 20.8868
PESQ | 2.6296 | 4.3039 | 3.6352 | 4.1743 | 4.3599 | 4.1197
LSD | 2.3340 | 2.6845 | 2.6497 | 3.1490 | 1.5909 | 1.8152
STOI | 0.9996 | 0.9913 | 0.9635 | 0.9881 | 0.9910 | 0.9894

In table 4.2, a single autoencoder block (N = 1) outperforms the majority of
the models compared. Even though it contains six times the number of trainable
parameters as a single block, the five block autoencoder (N = 5) shows a
slight variation in metric values and does not overfit. This demonstrates that
a small sized model is adequate to generalize and is sufficiently robust that,
when the size is increased, it shows no major evidence of overfitting when
compared to the other techniques.

##  5 Conclusion and Improvements

In this paper, we propose a novel architecture involving nested, residually
connected autoencoder blocks for the spectral reconstruction of monaural audio
samples. The preprocessing pipeline involves the conversion of a higher,
128Kbps audio wave to a lower, 32Kbps audio using FFMPEG (Tomar, 2006) whose
reconstruction was then attempted using the residual network. We compare our
method against FFMPEG (Tomar, 2006) based reconstruction and a variety of
preexisting and new machine learning methods using popular perceptual metrics
such as PESQ (Rix et al., 2001) and LSD (Gray and Markel, 1976). Furthermore,
we discuss how the proposed model can be pruned to a minimum size of 956KB
(52.15%) per autoencoder block, resulting in perceptually comparable
inferences.

Despite the benefits of the architecture and techniques used, the output wave
sometimes tends to be slightly incoherent with the source. Furthermore, due to
a lack of open sourced trainable data, the model could only be trained on
3,000 samples, resulting in poor generalisation of the silent regions in the
audio spectra. Additionally, Generative Adversarial Networks (GANs)
(Goodfellow et al., 2014) and Audio Transformers (Gong et al., 2021), and the
use of coherence-based comparative metrics are two noteworthy approaches to
consider for future trials.

## References

  * Birnbaum et al. (2019) Sawyer Birnbaum, Volodymyr Kuleshov, S. Enam, Pang Wei Koh, and S. Ermon.  Temporal film: Capturing long-range sequence dependencies with feature-wise modulations.  In _NeurIPS_ , 2019.
  * Cheng (1998) Mike Cheng.  LAME MP3 Encoder, 1998.  URL https://lame.sourceforge.io/.
  * Dong et al. (2016) Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 38(2):295–307, 2016.  doi: 10.1109/TPAMI.2015.2439281.
  * Fu et al. (2017) Szu-Wei Fu, Ting-yao Hu, Yu Tsao, and Xugang Lu.  Complex spectrogram enhancement by convolutional neural network with multi-metrics learning.  In _2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP)_ , pages 1–6, 2017.  doi: 10.1109/MLSP.2017.8168119.
  * Gong et al. (2021) Yuan Gong, Yu-An Chung, and James R. Glass.  AST: audio spectrogram transformer.  _CoRR_ , abs/2104.01778, 2021.  URL https://arxiv.org/abs/2104.01778.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gray and Markel (1976) A. Gray and J. Markel.  Distance measures for speech processing.  _IEEE Transactions on Acoustics, Speech, and Signal Processing_ , 24(5):380–391, 1976.  doi: 10.1109/TASSP.1976.1162849.
  * Gunawan and Sen (2010) David Gunawan and D. Sen.  Iterative phase estimation for the synthesis of separated sources from single-channel mixtures.  _IEEE Signal Processing Letters_ , 17(5):421–424, 2010.  doi: 10.1109/LSP.2010.2042530.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1026–1034, 2015.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hu and Loizou (2008) Yi Hu and Philipos C. Loizou.  Evaluation of objective quality measures for speech enhancement.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 16(1):229–238, 2008.  doi: 10.1109/TASL.2007.911054.
  * Jacob et al. (2018) Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2704–2713, 2018.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Krawczyk and Gerkmann (2014) Martin Krawczyk and Timo Gerkmann.  Stft phase reconstruction in voiced speech for an improved single-channel speech enhancement.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 22(12):1931–1940, 2014.  doi: 10.1109/TASLP.2014.2354236.
  * Kuleshov et al. (2017) Volodymyr Kuleshov, S. Zayd Enam, and Stefano Ermon.  Audio super resolution using neural networks, 2017.
  * Mowlaee et al. (2012) Pejman Mowlaee, Rahim Saeidi, and Rainer Martin.  Phase estimation for signal reconstruction in single-channel source separation.  In _Thirteenth Annual Conference of the International Speech Communication Association_ , 2012.
  * Pascual et al. (2017) Santiago Pascual, Antonio Bonafonte, and Joan Serrà.  Segan: Speech enhancement generative adversarial network.  In _Proc. Interspeech 2017_ , pages 3642–3646, 2017.  doi: 10.21437/Interspeech.2017-1428.  URL http://dx.doi.org/10.21437/Interspeech.2017-1428.
  * Rix et al. (2001) A.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra.  Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs.  In _2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221)_ , volume 2, pages 749–752 vol.2, 2001.  doi: 10.1109/ICASSP.2001.941023.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015_ , pages 234–241, Cham, 2015. Springer International Publishing.  ISBN 978-3-319-24574-4.
  * SIfre and Mallat (2014) Laurent SIfre and Stéphane Mallat.  Rigid-motion scattering for texture classification, 2014.
  * Stoller et al. (2018) D. Stoller, S. Ewert, and S. Dixon.  Wave-u-net: A multi-scale neural network for end-to-end audio source separation.  _ArXiv_ , abs/1806.03185, 2018.
  * Sulun and Davies (2021) Serkan Sulun and Matthew E. P. Davies.  On filter generalization for music bandwidth extension using deep neural networks.  _IEEE Journal of Selected Topics in Signal Processing_ , 15(1):132–142, 2021.  doi: 10.1109/JSTSP.2020.3037485.
  * Taal et al. (2010) Cees H. Taal, Richard C. Hendriks, Richard Heusdens, and Jesper Jensen.  A short-time objective intelligibility measure for time-frequency weighted noisy speech.  In _2010 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 4214–4217, 2010.  doi: 10.1109/ICASSP.2010.5495701.
  * Taal et al. (2011) Cees H. Taal, Richard C. Hendriks, Richard Heusdens, and Jesper Jensen.  An algorithm for intelligibility prediction of time–frequency weighted noisy speech.  _IEEE Transactions on Audio, Speech, and Language Processing_ , 19(7):2125–2136, 2011.  doi: 10.1109/TASL.2011.2114881.
  * Tan and Wang (2019) Ke Tan and DeLiang Wang.  Complex spectral mapping with a convolutional recurrent network for monaural speech enhancement.  In _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6865–6869, 2019.  doi: 10.1109/ICASSP.2019.8682834.
  * Tomar (2006) Suramya Tomar.  Converting video formats with ffmpeg.  _Linux Journal_ , 2006, 06 2006.
  * Tzanetakis and Cook (2002) G. Tzanetakis and P. Cook.  Musical genre classification of audio signals.  _IEEE Transactions on Speech and Audio Processing_ , 10(5):293–302, 2002.  doi: 10.1109/TSA.2002.800560.
  * Wang (2021) Wang.  GitHub - ludlows/python-pesq: PESQ (Perceptual Evaluation of Speech Quality) Wrapper for Python Users (narrow band and wide band), 2021.  URL https://github.com/ludlows/python-pesq.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.