# Distributed Processing for Encoding and Decoding of Binary LDPC codes using
MPI

Bhargav Gokalgandhi1, Ivan Seskar2
This project was funded by the NSF ”COSMOS” Project under grant number
CNS-1827923. WINLAB, Rutgers University
671, US-1, North Brunswick Township, NJ 08902
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Low Density Parity Check (LDPC) codes are linear error correcting codes used
in communication systems for Forward Error Correction (FEC). But, intensive
computation is required for encoding and decoding of LDPC codes, making it
difficult for practical usage in general purpose software based signal
processing systems. In order to accelerate the encoding and decoding of LDPC
codes, distributed processing over multiple multi-core CPUs using Message
Passing Interface (MPI) is performed. Implementation is done using Stream
Processing and Batch Processing mechanisms and the execution time for both
implementations is compared w.r.t variation in number of CPUs and number of
cores per CPU. Performance evaluation of distributed processing is shown by
variation in execution time w.r.t. increase in number of processors (CPU
cores).

††publicationid: pubid: 978-1-7281-1878-9/19/$31.00 ©2019 IEEE

##  I Introduction

Low Density Parity Check (LDPC) codes are a type of Error Correcting codes
which were developed by Robert Gallager in early 1960s [1]. Currently, LDPC
codes are used widely for Forward Error Correction (FEC) in current and next-
gen wireless standards such as 5G, 802.11, etc, as well as in video
broadcasting, due to their capacity approaching performance with increase in
block length. But, with increasing block length, the computation time of
encoding and decoding processes of LDPC codes increases leading to high
latency and limitation in throughput. Considering the increasing usage of LDPC
codes, and the current focus in ’softwarization’ of front-end signal
processing and networking elements for wireless systems, accelerating and
distributing the computation for encoding and decoding of LDPC codes becomes
important to achieve practical usage in next-gen software defined
communication systems.

Over the years, various methods have been used for accelerating the processing
of LDPC codes using general purpose hardware such as multi-core Central
Procssing Units (CPUs) and General Purpose Graphics Processing Units (GPGPUs).
[2, 3, 4, 5, 6, 7, 8, 9, 10] use GPGPU, or a combination of GPGPU and OpenMP
for acceleration of encoding and Sum-Product decoding of Binary LDPC codes.
These papers also compare the GPGPU and multi-core CPU implementations in
terms of acceleration provided. But the above papers show implementation of
LDPC encoding and decoding on single CPU with shared memory parallelization
model, and single GPGPU. While such systems provide good acceleration, it
becomes difficult to distribute the processing to multi-server and multi-CPU
systems. In such cases distributed memory mechanisms with message passing
models can be used for ease of processing distribution over multiple CPUs for
providing acceleration. LDPC encoding and decoding using message passing model
has not been implemented or evaluated as much. In [11] distributed LDPC
decoding is done using MPI, but it aims at optimization of coarse grain graph
search problem to select the LDPC Parity Check matrix with least Bit Error
Rate (BER). It does not evaluate the computational performance of encoding and
decoding LDPC codes.

In this paper, we use distributed memory message passing model by using
Message Passing Interface (MPI) to distribute the encoding and Sum-Product
decoding processes of LDPC codes over multiple multi-core CPUs. We perform the
processing using Stream Processing and Batch Processing mechanisms. We show
the acceleration provided by using message passing model for both mechanisms,
and compare the execution time required for processing w.r.t. increase in
number of CPUs and cores per CPU.

The paper is organized as follows. Section II describes the theoretical
background related to encoding and decoding of LDPC codes. Section III
describes the various components used for experimental evaluation, and it
explains the distributed implementation for each part of the algorithms
mentioned in Section II. Section IV shows the results of the various
experiments conducted. Lastly, Section V concludes the paper and states future
directions to be taken.

##  II Theoretical Background

We consider a vector $\mathrm{m}$ of $k$ information bits
$[m_{1},m_{2},....m_{k}]$. Using LDPC codes, $k$ information bits can be
encoded into a vector $\mathrm{p}$ of $n$ bits $[p_{1},p_{2},...,p_{n}]$, with
$n\geq k$. The $n$ bits consist of $k$ information bits from vector $m$ and
$n-k$ parity bits. So the rate of code is defined as $Rate(R)\geq k/n$. Here,
$n>k>0$.

The parity bits are formed by a linear combination of the $k$ information bits
known as parity check equations. LDPC codes are defined by putting these
parity check equations in a Parity Check matrix, denoted by $\mathrm{H}$,
which is a $k\times(n-k)$ matrix. Each row of $\mathrm{H}$ denotes a parity
check equation $c$, and each column defines whether a bit from the vector
$\mathrm{p}$ is present in the parity check equation or not. Depending on the
number of ones in each row and column, the Parity Check matrix is defined as
$(n_{c},n_{v})$, where $n_{c}$ is the number of ones per column, and $n_{v}$
is the number of ones per row. Then, the rate of code can be defined as
$Rate(R)\geq 1-n_{c}/n_{v}$. LDPC codes with this type of matrix are known as
regular LDPC codes.

$\mathrm{H}\mathrm{p}^{T}=0$ (1)

Where $T$ denotes transpose of a vector or a matrix. This means that the
additional parity bits must take values $0$ or $1$ such that each parity check
equation equals $0$.

###  II-A LDPC Encoding

From Eq. 1, we can say that each codeword is present in the null space of the
Parity Check matrix. So, we form the Generator matrix $\mathrm{G}$, which is a
$k\times n$ matrix, for encoding of LDPC codes by using the equation,

$\mathrm{HG}^{T}=0$ (2)

A Generator matrix is created by using Eq. 2 i.e. by finding the null space of
the Parity Check matrix. Then, by using the Generator matrix encoding of the
input information bits can be done using,

$\mathrm{m}^{T}\mathrm{G}=\mathrm{p}^{T}$ (3)

###  II-B LDPC Decoding

$\mathrm{H}=\begin{bmatrix}1&1&0&0&0&1&1&0\\\ 0&0&1&1&1&0&0&1\\\
1&1&0&0&0&1&0&1\\\ 0&0&1&1&1&0&1&0\\\ \end{bmatrix}$

Variable nodesCheck nodes

Figure 1: Bipartite graph for decoding of LDPC codes for a random $(2,4)$
Parity Check matrix with $n=8$ and $Rate(R)\geq 1/2$

We have taken the steps of Sum-Product decoding algorithm from [1, 12, 13,
14]. The aforementioned papers contain the sum-product algorithm description
in detail. We explain the gist of the algorithm. For decoding, a bipartite
graph $\mathcal{G(V,C,E)}$ is created from the Parity Check matrix, where
$\mathcal{V}$ is a set of $n$ variable nodes with index $v$, $\mathcal{C}$ is
a set of $n-k$ check nodes with index $c$. If there is a $1$ on the $i^{th}$
row and $j^{th}$ column of the Parity Check matrix, an edge connects the
$i^{th}$ check node and $j^{th}$ variable node. An example is shown in Fig. 1.
We represent the variable nodes using $v$ and check nodes using $c$. The
variable nodes and check nodes transfer messages and update Log-Likelihood
Ratio (LLR) values of each bit. These LLR values are input to the variable
nodes as,

$L_{v}=R_{v}=\frac{\text{prob(bit}_{v}=1)}{\text{prob(bit}_{v}=0)}=\frac{2y_{i}}{\sigma^{2}}$
(4)

where $L_{v}$ is the total LLR value and $R_{v}$ is the input LLR value of
node $v\in\mathcal{V}$, prob(.) is the probability value, $y_{i}$ is the
received noisy bit value, $\sigma^{2}$ is the Additive White Gaussian Noise
(AWGN) power at the receiver. These LLR values are sent to all the check nodes
connected to each variable nodes. After the check nodes receive the LLR
values, the check node values are updated using the equation,

$E_{cv}=log_{e}\left(\frac{1+\prod_{\hat{v}\in\mathcal{V},\hat{v}\neq
v}tanh(L_{\hat{v}c}/2)}{1+\prod_{\hat{v}\in\mathcal{V},\hat{v}\neq
v}tanh(L_{\hat{v}c}/2)}\right)$ (5)

where $E_{cv}$ is the partial LLR update that each check node
$c\in\mathcal{C}$ calculates for variable node $v$, and $L_{\hat{v}c}$ is the
partial LLR update each variable node $v$ calculates for check node $c$ using
the value from Eq. 5,

$L_{vc}=\sum_{\hat{c}\in\mathcal{C},\hat{c}\neq c}E_{\hat{c}v}+R_{v}$ (6)

After a predefined number of iterations, using the values gained from Eq. 5,
the total LLR value is calculated using equation,

$L_{v}=\sum_{c\in C}E_{cv}+R_{v}$ (7)

and then a hard-decision for each bit is made using,

$z_{v}=\left\\{\begin{array}[]{lr}1&:L_{v}\geq 0\\\
0&:L_{v}<0\end{array}\right.$ (8)

where $z_{v}$ is the final bit value for variable node $v$.

##  III Implementation Details

###  III-A Components used for implementation

####  III-A1 ORBIT Testbed

Open-Access Research Testbed (ORBIT) is a large-scale academic indoor wireless
testbed consisting of a 20x20 grid of computing nodes, more than 100 Software
Defined Radios (SDRs), and a set of inter-connected servers for large-scale
distributed computing and high performance computing applications. All
components are connected to a centralized server system using which control
functions and data transfer can be performed. More information related to the
testbed can be found in [15, 16].

####  III-A2 Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz

This CPU has 16 physical cores and 32 logical cores. It has a base frequency
of 2.3 GHz and max turbo frequency of 3.6 GHz, maximum memory bandwidth of 68
GBps with a 40 MB cache for fast memory access. More information can be seen
at [17]. We use 2 server nodes, each consisting of 2 CPUs. These server nodes
are present in ORBIT testbed. Both servers are connected using a 25 Gigabit
ethernet link which will be utilized when multi-server distributed processing
is implemented.

####  III-A3 MPI

MPI is a standardized interface for message passing between processors in
applications running on distributed memory systems. MPI can be used by
processes running independently on clusters of multiple CPUs for communicating
with each other and passing data using messages. We use MPICH [18], an open-
source and widely portable implementation of MPI standard for distributing the
processing among multiple cores and multiple servers. Some communication
routines which are used in this paper for implementation are,

  * •

MPI_Send():  Used to send data from a specific process to another.

  * •

MPI_Recv():  Used to receive data from a specific process.

  * •

MPI_Gatherv():  Used to gather data from all processes to one. Each process
can contain variable amount of data to be gathered.

  * •

MPI_Bcast():  Used to broadcast data from one process to all other processes.

###  III-B Distributed Stream Processing implementation

In Stream Processing, a single vector from a stream of multiple vectors is
computed in a distributed manner. The data within each vector is distributed
among multiple processors which perform computations and then the result is
gathered in the master processor. We assume the availability of $N_{proc}$
processors denoted by $Proc(0),Proc(1),...,Proc(N_{proc}-1)$, where $Proc(x)$
means processor with index $x$. The division of elements to be processed among
the processors is done in the following manner,

$NumElems(q)=\left\\{\begin{array}[]{ll}floor(N_{proc}/l)+1&:x<N_{procs}\%l\\\
floor(N_{proc}/l)&:x\geq N_{procs}\%l\end{array}\right.$ (9)

where $NumElems(q)$ represent the number of elements to be given to processor
$q$, $a\%b$ means the remainder of $a/b$, and $l$ is the total number of
elements to be divided.

####  III-B1 Encoding

$\mathrm{m}^{T}\mathrm{G}=\begin{bmatrix}1&0&1&1\end{bmatrix}\begin{bmatrix}1&0&0&0&{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}0}&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1}&{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color@cmyk@fill{0}{0}{1}{.5}0}&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}0}\\\
0&1&0&0&{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}1}&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0}&{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color@cmyk@fill{0}{0}{1}{.5}0}&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}0}\\\
0&0&1&0&{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}0}&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0}&{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color@cmyk@fill{0}{0}{1}{.5}1}&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}0}\\\
0&0&0&1&{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}1}&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1}&{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color@cmyk@fill{0}{0}{1}{.5}1}&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}1}\\\
\end{bmatrix}=\mathrm{p}^{T}$ $Proc(0)$$Proc(1)$$Proc(2)$$Proc(3)$

${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}p_{4}}=\begin{bmatrix}1&0&1&1\end{bmatrix}\begin{bmatrix}{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}0}\\\
{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}1}\\\
{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}0}\\\
{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}1}\\\
\end{bmatrix}$
$Proc(0)$${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}p_{5}}=\begin{bmatrix}1&0&1&1\end{bmatrix}\begin{bmatrix}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1}\\\
{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0}\\\
{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0}\\\
{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}1}\\\
\end{bmatrix}$
$Proc(1)$${\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color@cmyk@fill{0}{0}{1}{.5}p_{6}}=\begin{bmatrix}1&0&1&1\end{bmatrix}\begin{bmatrix}{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color<EMAIL_ADDRESS>{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color<EMAIL_ADDRESS>{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color<EMAIL_ADDRESS>{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color<EMAIL_ADDRESS>\end{bmatrix}$
$Proc(2)$${\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}p_{7}}=\begin{bmatrix}1&0&1&1\end{bmatrix}\begin{bmatrix}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}0}\\\
{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}0}\\\
{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}0}\\\
{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}1}\\\
\end{bmatrix}$ $Proc(3)$

$\mathrm{p}^{T}=\begin{bmatrix}p_{0}&p_{1}&p_{2}&p_{3}&{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}p_{4}}&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}p_{5}}&{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color@cmyk@fill{0}{0}{1}{.5}p_{6}}&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}p_{7}}\end{bmatrix}$
$=\begin{bmatrix}1&0&1&1&{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}1}&{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0}&{\color[rgb]{0.5,0.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.5,0.5,0}\pgfsys@color@cmyk@stroke{0}{0}{1}{.5}\pgfsys@color@cmyk@fill{0}{0}{1}{.5}0}&{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}1}\end{bmatrix}$

Figure 2: Example of Distributed Matrix-Vector multiplication for LDPC
Encoding with a random Generator matrix.

For encoding, we use the Generator matrix which is converted to standard
$[I_{k}:P]$ where $I_{k}$ sub-matrix is a $k\times k$ identity matrix formed
by column permutations of Generator matrix, and $P$ is a sub-matrix consisting
of remaining $k\times(n-k)$ values of the Generator matrix. The input
information bits $m_{1},...,m_{k}$ are first copied to the output vector bits
$p_{1},..,p_{k}$ for each processor. Then, the columns of the $P$ sub-matrix
of the Generator matrix are divided among the $N_{proc}$ processors using Eq.
9. An example of division of $4$ columns of the $P$ matrix is shown in Fig. 2.
Each processor selects a subset of columns from the $k$ columns of sub-matrix
$P$. Each processor then takes the input vector $\mathrm{m}$ and multiplies it
with the columns of sub-matrix $P$ selected by that processor. The partial
output calculated by each processor is then gathered at the master processor
by using the MPI_Gatherv() command for further processing. The encoded bits
are then converted to Binary Phase Shift Keying (BPSK) form for transmission
through AWGN channel. For BPSK, we convert the $0$ bits to $-1$ and $1$ bits
remain as $1$.

####  III-B2 Decoding

Algorithm 1 Distributed Sum-Product Decoding using MPI

1: Define $\mathcal{V}$ and $\mathcal{C}$ as sets containing variable nodes
$v$ and check nodes $c$ respectively, $E_{cv},L_{vc},L_{v},R_{v},z_{v}$,
$N_{proc}$, $Proc(x)$

2: Divide the check and variable nodes to all processors in an interleaved
manner

3: for $v\in\mathcal{V}$ do

4: Find the initial LLR values $L_{v}$ and $R_{v}$ and broadcast to all
processors using MPI_Bcast()

5: end for

6: for $i=1$ to #iterations do

7: // Updating LLR values at check nodes

8: for ($c=0;c<|\mathcal{C}|;c=c+N_{proc}$) do

9: Find $E_{(c+x)v}$

10: end for

11: for ($c=0;c<|\mathcal{C}|;c=c+1$) do

12: Send to processor which has the variable node $v$ in its list, (i.e.
$Proc(v\%N_{proc})$, using MPI_Send())

13: Receive the message using MPI_Recv() at $Proc(v\%N_{proc})$

14: end for

15: // Updating LLR values at variable nodes

16: for ($v=0;v<$size of V$;v=v+N_{proc}$) do

17: Find $L_{(v+x)c}$

18: end for

19: for ($v=0;v<|\mathcal{V}|;v=v+1$) do

20: Send to processor which has the check node $c$ in its list, (i.e.
$Proc(c\%N_{proc})$, using MPI_Send())

21: Receive the message using MPI_Recv() at $Proc(c\%N_{proc})$

22: end for

23: end for

24: // Calculating final LLR values and then performing hard-decision decoding

25: for ($v=0;v<|\mathcal{V}|;v=v+N_{proc}$) do

26: Calculate $L_{v+x}$

27: Broadcast output to all processes using MPI_Bcast()

28: Calculate $z_{v+x}$ value

29: Gather at master processor using MPI_Gatherv()

30: end for

$Proc(0)$:MPI_Send()$Proc(2)$:MPI_Recv()Variable node partial LLR ($L_{vc}$)
update and data transfer$L_{02}$

$Proc(3)$:MPI_Recv()$Proc(1)$:MPI_Send()Check node partial LLR ($E_{cv}$)
update and data transfer$E_{13}$

| Variable node
---|---
| Check node

Figure 3: Example of communication between 4 processors while performing Sum-
Product Decoding for Parity Check matrix shown in Fig. 1.

For Sum-Product Decoding, we create a graph and an adjacency list of the
$\mathrm{H}$ matrix as shown in Section II-B. Now, for each iteration of the
decoding algorithm, we first divide the variable nodes and check nodes among
all processors in an interleaved manner. For each iteration the processors
calculate the partial LLR values $E_{cv}$ of check nodes for which the check
node index $c\%N_{proc}=x$ using Eq. 5. Then, after all processors have
calculated their respective updated values, each processor sends the updated
value of check nodes only to the processor with variable node index
$v\%N_{proc}=x$. In the same way, the variable node LLR values are updated and
sent. An example of communication between processors to update LLR values is
shown in Fig. 3. The decoding algorithm runs for preset number of iterations
after which the variable nodes update their final LLR values and hard decision
decoding is done using Eq. 8. The complete Distributed Sum-Product Decoding is
shown in Algorithm 1. The LLR updation steps and the LLR transfer steps are
separate for all processors. This separation ensures minimum switching between
computation and communication for all processors.

###  III-C Distributed Batch Processing implementation

Batch Processing here means each processor takes a batch of vectors for
encoding or decoding, and the number of total vectors of information bits is
greater than the number of processors.

####  III-C1 Encoding

Now, for encoding, we assume an input vector $M$ which is a $kN_{e}$ length
input vector consisting of $k$ length vectors. Here $N_{e}$ is the number of
$k$ length vectors to be encoded. The $M$ vector divided into smaller vectors
of size $k$ which are then distributed among all processors. Each processor
then uses Eq. 3 to encode the batch of $k$ length $\mathrm{m}$ vectors. The
output of the batch of vectors at each processor is then gathered in the
master processor using MPI_Gatherv(). The distribution of vectors is done
using Eq. 9, where $l=N_{e}$.

####  III-C2 Decoding

For decoding, a noisy vector of length $nN_{d}$ is taken, where $N_{d}$ is the
number of vectors to be decoded. This vector is then divided into multiple $n$
length vectors which are distributed to multiple processors using Eq. 9. Then,
each processor performs Sum-Product decoding, as shown in Section II-B, on the
batch of $n$ length vectors in a serial manner. The decoded output is then
gathered in the master processor using MPI_Gatherv().

##  IV Experiments and Results

Parameter | Value
---|---
Code length | 1032
Design Rate | $(0.25,0.5,0.75)$
Number of $1$s per row | 12
Number of $1$s per column | Depending on rate $(9,6,3)$
Number of decoding iterations | 10
Number of vectors processed | 1000
Number of servers | $1,2$
Number of CPUs per server | $2$
Number of processors per CPU | 1 to 32
TABLE I: Parameters for the experiments

(a) Encoding

(b) Decoding

Figure 4: Per-vector execution time for Stream Processing and Batch Processing
of 1032 length LDPC codes with rates $(1/4,1/2,3/4)$ over single server

(a) Encoding

(b) Decoding

Figure 5: Per-vector execution time for Stream Processing and Batch Processing
of 1032 length LDPC codes with rates $(1/4,1/2,3/4)$ over two servers

The parameters used for performing experiments are shown in Table I. For the
experiments, we design the Parity Check matrix for LDPC codes as shown in [1].
We design $(N_{c},12)$ Parity Check matrix where $N_{c}$ is dependent on the
code rate. Based on the designed matrix, encoding and decoding is performed as
shown in previous sections. We conduct experiments in two parts. Firstly, we
distribute the processing on a single server consisting of two CPUs. Secondly,
we distribute the processing among all processors of two such servers. For
both cases, we compare Stream Processing and Batch Processing based on the
execution time taken for processing of a single input vector. To get per-
vector execution time, the total execution time is divided by the number of
vectors.

###  IV-A Single server case

The execution time for distributed processing on multiple processors of a
single server consisting of two CPUs is shown in Fig. 4. The difference in
execution time between Stream and Batch Processing starts increases as the
number of processors increase. Batch Processing starts performing better than
Stream Processing because the processors only communicate for taking the input
vector and giving the output vector. While for Stream Processing, the
processors communicate after encoding each vector, and during each iteration
of decoding of each vector. Also, when the number of processors is increased
beyond the maximum logical cores per CPU, the performance of Stream Processing
either saturates or worsens due to the added latency of inter-CPU
communication.

Even though Batch Processing has lesser execution time per vector, if high
efficiency in utilization of hardware resources is to be maintained, the
number of vectors to be processed must always be greater than the number of
processors. So, the initial latency of Batch Processing can be higher than
Stream Processing when higher utilization efficiency is to be maintained.

###  IV-B Two server case

The execution time for distributed processing on multiple processors of two
servers, with two CPUs each, is shown in Fig. 5. For two servers case, the
x-axis of Fig. 5 shows the number of processors used per server. Which means
that if the x-axis shows $4$ then $4$ processors are being used per server
i.e. total of $8$ processors are being utilized. It can be seen that Batch
Processing performs better than Stream Processing, especially for decoding,
due to the minimal communication latency between processors Also, the
performance of Stream Processing using two servers for decoding is
approximately an order of magnitude worse than performance of Stream
Processing using a single server, and for encoding the performance worsens
with increase in number of processors. So, for multi-server scenario, due to
the high communication latency between servers, Stream Processing performance
is dependent on the speed of inter-server communication links.

Since the communication between processors in Batch Processing is much lesser
than the computation per processor, its dependence on the type of inter-server
communication link is lesser as compared to that of Stream Processing, giving
higher acceleration. Performance for decoding using Distributed Processing for
two server case is either similar to or better than that of single server case
with increase in number of processors. Performance of encoding using
Distributed Processing for two server case is still worse than that of single
server case. This is due to the computation time being negligible as compared
to communication time.

##  V Conclusion and future Work

Considering the intensive computation required for processing of LDPC codes,
we used MPI to distribute the processing over LDPC codes over multiple multi-
core CPUs. Using the distributed implementation, acceleration was provided for
encoding and decoding processes of LDPC codes. Evaluation and comparison for
Stream processing and Batch Processing based mechanisms for distributed
processing of LDPC codes was done, and the advantages and limitations of both
methods were shown.

While using a distributed memory based message passing model for distributed
processing provides acceleration, increasing the number of processors to more
than the logical cores in the system adds high processing latency as well as
inter-processor communication latency due to context switching between
processes. Also, for multi-server systems, the processing latency is highly
dependent on the type of communication link between servers. So, to decrease
the dependence on communication between CPUs, combination of shared memory and
distributed memory based systems will be considered for distributed processing
of LDPC codes.

## References

  * [1] https://web.stanford.edu/class/ee388/papers/ldpc.pdf.
  * [2] L. Zhen, W. Yunpei, and L. Lulu, “Parallel algorithmic optimization and achievement for ldpc encoding and decoding on cuda platform,” in 2015 International Conference on Computational Science and Computational Intelligence (CSCI), pp. 632–636, 12 2015.
  * [3] Y. Zhao and F. C. M. Lau, “Implementation of decoders for ldpc block codes and ldpc convolutional codes based on gpus,” IEEE Transactions on Parallel and Distributed Systems, vol. 25, pp. 663–672, 3 2014.
  * [4] Y. Lin and W. Niu, “High throughput ldpc decoder on gpu,” IEEE Communications Letters, vol. 18, no. 2, pp. 344–347, 2014.
  * [5] B. Jiang, J. Bao, and X. Xu, “Efficient simulation of qc ldpc decoding on gpu platform by cuda,” in Wireless Communications & Signal Processing (WCSP), 2012 International Conference on, pp. 1–5, IEEE, 2012.
  * [6] S. Kang and J. Moon, “Parallel ldpc decoder implementation on gpu based on unbalanced memory coalescing,” in Communications (ICC), 2012 IEEE International Conference on, pp. 3692–3697, IEEE, 2012.
  * [7] C.-C. Chang, Y.-L. Chang, M.-Y. Huang, and B. Huang, “Accelerating regular ldpc code decoders on gpus,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 4, no. 3, p. 653, 2011.
  * [8] G. Falcao, L. Sousa, and V. Silva, “Massively ldpc decoding on multicore architectures,” IEEE Transactions on Parallel and Distributed Systems, vol. 22, pp. 309–322, 2 2011.
  * [9] H. Ji, J. Cho, and W. Sung, “Massively parallel implementation of cyclic ldpc codes on a general purpose graphics processing unit,” in Signal processing systems, 2009. SiPS 2009. IEEE workshop on, pp. 285–290, IEEE, 2009\.
  * [10] S. Wang, S. Cheng, and Q. Wu, “A parallel decoding algorithm of ldpc codes using cuda,” in Signals, systems and computers, 2008 42nd asilomar conference on, pp. 171–175, IEEE, 2008.
  * [11] V. S. Nittoor and R. Suda, “Parallelizing a coarse grain graph search problem based upon ldpc codes on a supercomputer,” in 2011 Sixth International Symposium on Parallel Computing in Electrical Engineering, pp. 7–12, IEEE, 2011\.
  * [12] S. J. Johnson, “Introducing low-density parity-check codes,” University of Newcastle, Australia, 2006.
  * [13] S. J. Johnson and S. R. Weller, “Low-density parity-check codes: Design and decoding,” U. o. Newcastle, Ed. Callaghan, Australia, pp. 1308–1318, 2002\.
  * [14] A. Shokrollahi, “Ldpc codes: An introduction,”
  * [15] D. Raychaudhuri, I. Seskar, M. Ott, S. Ganu, K. Ramachandran, H. Kremo, R. Siracusa, H. Liu, and M. Singh, “Overview of the orbit radio grid testbed for evaluation of next-generation wireless network protocols,” in Wireless Communications and Networking Conference, 2005 IEEE, vol. 3, pp. 1664–1669, IEEE, 2005.
  * [16] “Orbit website.”
  * [17] “Intel webpage for cpu information.”
  * [18] “Mpich website and overview.”


# HARP-Net: Hyper-Autoencoded Reconstruction Propagation
for Scalable Neural Audio Coding

###### Abstract

We propose a novel autoencoder architecture that improves the architectural
scalability of general-purpose neural audio coding models. An autoencoder-
based codec employs quantization to turn its bottleneck layer activation into
bitstrings, a process that hinders information flow between the encoder and
decoder parts. To circumvent this issue, we employ additional skip connections
between the corresponding pair of encoder-decoder layers. The assumption is
that, in a mirrored autoencoder topology, a decoder layer reconstructs the
intermediate feature representation of its corresponding encoder layer. Hence,
any additional information directly propagated from the corresponding encoder
layer helps the reconstruction. We implement this kind of skip connections in
the form of additional autoencoders, each of which is a small codec that
compresses the massive data transfer between the paired encoder-decoder
layers. We empirically verify that the proposed hyper-autoencoded architecture
improves perceptual audio quality compared to an ordinary autoencoder
baseline.

Index Terms—  audio coding, deep learning, U-Net, autoencoders

##  1 Introduction

Data compression is an essential aspect of information and communication
systems nowadays. Its main aim is to reduce the bitrate by eliminating the
redundancy of the data present in the signal by mapping the raw data samples
in the original high precision representation (e.g., single-precision floating
points) into a compact discrete representation (e.g., a bitstring). In this
work, we focus on the audio coding applications where the reconstructed signal
on the receiver side is allowed to have reconstruction error once its
perceptual quality is above the desired level. While in speech communication
low bitrate codecs can achieve the common intelligibility goal, as for music
signals, the required bitrate tends to be much higher since the user’s
listening experience can be deteriorated even by a subtle perceptual
degradation. Hence, scalability to various use cases and bitrates is an
important goal in modern speech or audio codecs, such as in unified speech and
audio coding (USAC) [1, 2].

With the recent breakthrough in deep learning, neural speech coding emerged as
a new research area. Despite their higher computationally complexity than the
conventional speech codecs, such as AMR-WB [3] and Opus [4], neural speech
codecs show merits in terms of coding gain. For example, fully-convolutional
autoencoders have been successfully transformed into a codec, whose bottleneck
layer is quantized to produce bitstings out of waveforms [5]. These relatively
compact waveform codecs start to compete with AMR-WB and Opus after being
coupled with linear predictive coding (LPC) [6]. Meanwhile, generative models,
such as WaveNet [7], have proven to be effective towards speech coding
reducing bitrates down to 2.4kbps, while retaining reasonable speech quality
[8, 9]. In return, the system is burdened by the complex WaveNet-based
decoders. A more recent approach proposed to use LPCNet [10], achieving as low
as 1.6kbps [11]. Likewise, neural speech coding has advanced by encompassing
traditional technology.

However, most promising advancements have been made for speech coding rather
than general-purpose audio coding. For example, the WaveNet decoder is
designed for very low bitrate cases, while it is too heavy for real-time
decoding, e.g., for on-device music players. Meanwhile, the autoencoder-based
waveform codecs display significant drawback as they rely on objective loss
functions, e.g., the mean-squared error, which often result in perceptual
discrepancy among decoded signals. Hence, perceptually more meaningful loss
functions show promising results for neural audio coding by calibrating the
loss with a psychoacoustic weighting scheme [12]. In [12], the idea of
cascading multiple autoencoders [13] also show performance improvement, while
the concatenation linearly increases model complexity.

(a) The baseline autoencoder

(b) A hypothetical U-Net codec

(c) The proposed HARP-Net architecture

Figure 1: The neural audio coding systems discussed in this work. Note that in
the hypothetical U-Net codec each identity shortcut requires a substantial
bitrate as each $\bm{x}^{(l)}$ is part of code. HARP-Net compresses them
further and reduces the total bitrate.

Our proposed method tackles the neural audio coding problem via architectural
improvement. Our codec covers various bitrates, while its complexity is
suppressed to be low. To this end, we posit a mirrored autoencoder (AE)
architecture, where the feature maps on the decoder side are an approximation
of the encoder’s. Hence, we conjecture that direct communication between each
pair of encoder and decoder layers can improve the layer-wise approximation
quality of the decoder. Then, the better approximation performance is
propagated to the decoder’s output layer, leading to a better total
reconstruction. To this end, we employ additional autoencoding paths that
skip-connect the corresponding feature map pairs on the opposite sides. Since
each skip AE is a small codec producing a code from its own bottleneck layer,
the final bitstream is the sum of all codes from those bottleneck layers. We
call the proposed architecture hyper-autoencoded reconstruction propagation
network (HARP-Net). HARP-Net shares the similar advantage introduced by the
U-Net’s skip connections [14]: a superior reconstruction quality for
autoencoding-like tasks, such as music source separation in Wave-U-Net [15].
The difference is that U-Net’s identity shortcuts disqualify the architecture
as a codec due to the high datarate. HARP-Net replaces the bitrate-consuming
identity shortcuts with compact AEs, whose goal is to deliver the encoder
feature map in a compressed form to the decoder side.

##  2 Neural Audio Coding

###  2.1 Basic Autoencoders

Figure 1(a) depicts an AE-based neural audio coding system. Its encoder
function ${\mathcal{F}}_{\text{enc}}(\cdot)$ converts the input signal into a
feature vector, which is subsequently quantized into a bitstring, or the code
$\bm{z}$. Compression is achieved if $\bm{z}$ is with a smaller number of bits
than the amount necessary to represent the raw input. The decoder
${\mathcal{F}}_{\text{dec}}(\cdot)$ follows to recover the original signal out
of the code vector $\bm{z}$. Hence, the AE’s goal is to best predict the
reconstruction $\bar{\bm{x}}$ that approximates the input $\bm{x}$ using a
short enough bitstring $\bm{z}$. We write this procedure by employing a
quantizer $Q$ and its inversion:

$\bm{x}\approx\bar{\bm{x}}\leftarrow{\mathcal{F}}_{\text{dec}}\circ
Q^{-1}(\bm{z}),\quad\bm{z}\leftarrow
Q\circ{\mathcal{F}}_{\text{enc}}(\bm{x}).$ (1)

Neural coding systems employ a deep neural network (DNN) to implement the
autoencoding function. Hence, the encoder function
${\mathcal{F}}_{\text{enc}}(\cdot)$ can be defined as a series of non-linear
transforms, whose $l$-th layer converts its input $\bm{x}^{(l-1)}$ into a new
feature space $\bm{x}^{(l)}$:

$\displaystyle\bm{z}\leftarrow{\mathcal{F}}_{\text{enc}}(\bm{x})=Q\circ
f_{\text{enc}}^{(L)}\circ\cdots\circ f_{\text{enc}}^{(2)}\circ
f_{\text{enc}}^{(1)}(\bm{x}),$ (2)

where $L$ denotes the number of the encoder layers. Hence, we can also denote
the raw input as $\bm{x}=\bm{x}^{(0)}$. The encoder output $\bm{x}^{(L)}$ goes
through a quantization module $Q$ that discretizes the floating-point feature
vector $\bm{x}^{(L)}$ into a bitstring $\bm{z}$. Furthermore, $\bm{z}$ is
losslessly compressed by using entropy coding, e.g., Huffman coding.

The de-quantization process $Q^{-1}(\bm{z})$ reverts the code vector $\bm{z}$
back to the floating-point feature $\bar{\bm{x}}^{(L)}$ with a certain amount
of quantization error, i.e., $\bar{\bm{x}}^{(L)}$ approximates $\bm{x}^{(L)}$.
Then, a series of decoder layers recover the raw signal:

$\bar{\bm{x}}^{(0)}\leftarrow{\mathcal{F}}_{\text{dec}}(\bm{x})=f_{\text{dec}}^{(1)}\circ\cdots\circ
f_{\text{dec}}^{(L-1)}\circ f_{\text{dec}}^{(L)}\circ Q^{-1}(\bm{z}).$ (3)

Note that layer indices decrease from the bottleneck to the output layer.

Although it is not common in the recent deep learning literature, a hard
association between the encoder and decoder layers was popular in some earlier
models. For example, in the restricted Boltzmann machines (RBM) [16], we write
the $l$-th layer transform
$\bm{x}^{(l)}\\!\leftarrow\\!\bm{W}_{\text{enc}}^{(l)}\bm{x}^{(l-1)}+\bm{b}_{\text{enc}}^{(l)}$.
Then, its corresponding decoder layer is defined with the encoder weight
matrix $\bm{W}_{\text{enc}}^{(l)}$, but by transposing it:
$\bar{\bm{x}}^{(l-1)}\\!\leftarrow\\!\left(\bm{W}_{\text{enc}}^{(l)}\right)^{\top}\bar{\bm{x}}^{(l)}+\bm{b}_{\text{dec}}^{(l)}$.
Since the transpose operation is equivalent to matrix inversion for
orthonormal matrices, this association loosely implies an inversion
relationship.

The basic AE in Figure 1(a) is with a mirrored architecture, while the
coupling of the encoder and decoder layers is not assumed. In the proposed
HARP-Net architecture, we postulate the ties between the encoder-decoder
layers must be helpful for coding, although the coupling is indirect using
skip connections rather than inversion.

###  2.2 Quantization and Bitrate Control

The quantization function $Q$ maps a floating-point value $x^{(L)}_{i}$ to a
pre-defined, finite set of quantization bins. Instead of post-training
quantization, we include $Q$ and $Q^{-1}$ as part of the AE, so training of
the AE can also suppress the quantization artifact. Since $Q$ is not a
differentiable process, we employ soft-to-hard quantization [17], which
softens the bin assignment during training. First, it estimates the
probability of assigning $i$-th code to $j$-th bin,
$\bm{P}=\text{softmax}(\alpha\bm{S})$, where $S_{ij}$ defines the similarity
(e.g. negative absolute difference) between $i$-th code value and $j$-th
representative $\mu_{j}$. $\alpha$ controls the “hardness” of the logits,
i.e., the smaller the softer. The softmax function turns the $i$-th similarity
values $\bm{S}_{i:}$ to all $J$ quantization bins into a probability vector
$\bm{P}_{i:}$. Then, the convex combination of the soft assignment probability
and bin representatives recovers the feature vector:
$\tilde{x}^{(L)}_{i}=\sum_{j=1}^{J}P_{ij}\mu_{j}$. The decoder takes
$\tilde{\bm{x}}^{(L)}$ as input. This whole process is differentiable.

Conversely, the test-time inference uses the hard quantization. First, from
$\bm{S}$, the closest bin index for all scalar codes forms the discrete code
$\bm{z}$, i.e., $j^{*}=\operatornamewithlimits{\arg\max}_{j}\bm{S}_{ij}$. Note
that we reduce the gap between the soft and hard assignments by annealing
$\alpha$, i.e., by gradually increasing it during training:
$\lim_{\alpha\rightarrow\infty}\tilde{\bm{x}}^{(L)}=\bar{\bm{x}}^{(L)}$.

The code $\bm{z}$ is further compressed via Huffman coding losslessly, the
entropy of $p(x^{(L)}_{i})$ serves as the lower bound of the bitrate, which is
unknown. As an alternative, we predict the overall frequency of $j$-th bin
being selected, $p_{j}$, by observing the soft assignment probability $\bm{P}$
for all training examples. Hence, the approximate entropy is
${\mathcal{H}}=-\sum_{j}p_{j}\log p_{j}$, which we regularize during training
to control the bitrate. For example, the encoder adjusts the distribution of
$\bm{x}^{(L)}$, so its entropy matches the target entropy, i.e., the lower
bound of the per-frame bitrate.

###  2.3 The Hypothetical Codec with the U-Net Architecture

In the original U-Net [14] and its variants for audio processing [15, 18] skip
connections pair up the encoder and decoder layers through identity shortcuts
to improve the AE’s reconstruction performance. Hence, the encoder feature
maps are delivered to the decoder layers intact. Figure 1(b) depicts a
hypothetical U-Net architecture for neural coding, whose decoder layer now
takes as input both the corresponding encoder and decoder feature maps by
concatenating them together along the channel dimension:

$\bar{\bm{x}}^{(l-1)}\leftarrow
f_{\text{dec}}^{(l)}\left(\big{[}\bm{x}^{(l)},\bar{\bm{x}}^{(l)}\big{]}\right).$
(4)

The performance gain entailed by these identity shortcuts is without question.
In addition to the information decoded from the binary code vector $\bm{z}$,
the decoder layer directly utilizes the encoder feature maps, which can be
regarded as less abstract feature representations. However, it is an
unrealistic coding system since each identity shortcut consumes a significant
amount of bits.

##  3 The Proposed HARP-Net Architecture

We address the hypothetical U-Net codec’s high data rate issue by replacing
the identity shortcuts with additional AEs, which we call skip autoencoders.
These skip connections can still be considered a way to provide less
processed, i.e., rawer information to the decoder as in U-Net. In this view,
since a decoder feature map $\bar{\bm{x}}^{(l)}$ is an approximation of its
corresponding encoder representation $\bm{x}^{(l)}$, information coming
directly from the encoder will help its reconstruction job. In neural coding,
these skip connections can compensate the quantization artifact introduced in
the bottleneck layer, i.e., $\bm{z}\leftarrow Q(\bm{x}^{(L)})$, which
propagates to all decoder layers.

HARP-Net employs layer-wise skip AEs ${\mathcal{G}}^{(l)}$ to compress the
data transferred through U-Net’s skip connections:

$\bm{x}^{(l)}\approx\bar{\bm{x}}^{(l)}\leftarrow{\mathcal{G}}_{\text{dec}}^{(l)}\circ
Q^{-1}(\bm{z}^{(l)}),\quad\bm{z}^{(l)}\leftarrow
Q\circ{\mathcal{G}}_{\text{enc}}^{(l)}(\bm{x}^{(l)}).$ (5)

Now a skip connection is divided into its own encoder
${\mathcal{G}}_{\text{enc}}^{(l)}(\cdot)$ and decoder
${\mathcal{G}}_{\text{dec}}^{(l)}(\cdot)$ parts. Hence, instead of
transmitting $\bm{x}^{(l)}$ directly, a skip AE compresses it down to
$\bm{z}^{(l)}$, a layer-specific code vector. To this end, first, the encoder
defines a feature transformation function
$\bm{y}^{(l)}\leftarrow{\mathcal{G}}_{\text{enc}}^{(l)}(\bm{x}^{(l)})$, which
produces a a processed version of the original feature map. $\bm{y}^{(l)}$ is
followed by the ordinary quantization and Huffman coding that construct the
layer-wise bitstring $\bm{z}^{(l)}$. Here, the encoder function
${\mathcal{G}}_{\text{enc}}^{(l)}(\cdot)$ is a placeholder: it can employ any
adequate structure, probably with multiple layers. De-quantization recovers
$\bar{\bm{y}}^{(l)}$ from $\bm{z}^{(l)}$. Then, the decoder recovers the
feature map
$\bm{x}^{(l)}\approx\bar{\bm{x}}^{(l)}\leftarrow{\mathcal{G}}_{\text{dec}}^{(l)}(\bar{\bm{y}}^{(l)})$

Once again, the goal here is to recover the original feature $\bm{x}^{(l)}$
with little information loss, using only few bits to represent $\bm{z}^{(l)}$.
The final code of the entire system is defined by concatenating all layer-wise
codes including the ordinary bottleneck code at $L$-th layer:
$\bm{z}=[\bm{z}^{(L)};\bm{z}^{(L-1)};\cdots\bm{z}^{(1)}]$.

HARP-Net’s advantage can be summarized as follows:

  * •

Performance: The additional skip connections boost the reconstruction quality
on the decoder side than a single bottleneck code.

  * •

Scalability: Having $\bm{z}^{(L)}$ as the most abstract form of
representation, it is natural to perform scalable coding by employing $M$
additional skip AEs as neeeded, i.e.,
$\bm{z}=[\bm{z}^{(L)};\bm{z}^{(L-1)};\cdots\bm{z}^{(L-M)}]$. $M$ can control
the bitrates.

  * •

Flexibility: The proposed method can be applied to any DNN with a mirrored
autoencoding architecture. Furthermore, the skip AE’s architecture is open to
various choices.

Models | # Layers | # Filters | # Params.
---|---|---|---
Baseline 1 | 10 | 30 | 218k
HARP-Net (1 Skip AE) | 12 | 24 | 216k
Baseline 2 | 9 | 36 | 275k
HARP-Net (2 Skip AEs) | 12 | 24 | 257k
Baseline 3 | 10 | 36 | 315k
HARP-Net (3 Skip AEs) | 12 | 24 | 298k
Baseline 4 | 10 | 38 | 350k
HARP-Net (4 Skip AEs) | 12 | 24 | 340k
Table 1: The network topology of variant HARP-Net versions with a varying
number of skip AEs. Their associated plain AE baselines are presented with a
matching number of parameters. The number of layers do not include the skip
AEs while the number of parameters take them into account.

##  4 Experiments

###  4.1 The Experimental Setup

We assess the performance of HARP-Net on musical signals. To this end, we use
$6,000$ commercially released songs of 13 genres, totalling $5.5$ hours of
audio data, which is divided into $5,700$ and $300$ songs for training and
validation, respectively. $16$ songs are set aside as a test set. All signals
are in mono with a sampling rate of $44.1$kHz. We use a frame size of 16384
with a hop size 32 samples.

The proposed HARP-Net codec operates in two target bitrate setups, 24kbps and
48kbps, but they are defined in the LPC residual domain. To this end, the
corresponding LPC coefficients need to be accounted for in the final system
bitrate, which amount to 16kbps. Eventually, the resulting bitrates are 40kbps
and 64kbps. We opt to remain in relatively high bitrates as we limit our
experiment to 44.1kHz signals. We leave lower bitrate cases for future work.

In order to assess the impact of the proposed skip AEs, we train multiple
model variants that are with varying number of skip AEs from $M=1$ to $4$. For
a fair comparison, we assess each of the HARP-Net models against a vanilla AE
with matching number of parameters. Table 1 summarizes the various models
trained for our experiments. Note that HARP-Net consists of 12 layers, each of
which is with 24 1-d convolutional kernels of size $1\times 15\times 24$. The
exception is the final encoder layer, where only one kernel is used to
collapse the 24 channels down to 1. Eventually, the dimension of the input
frame and encoder output are the same, i.e.,
$\bm{x}^{(L)},\bm{x}^{(0)}\in\mathbb{R}^{16384}$, meaning the encoder does not
perform dimension reduction. This choice is to avoid artifact commonly caused
by the downsampling and upsampling processes in convolutional models [19].
Instead, we control the bitrate by regularizing the entropy of $\bm{x}^{(L)}$
as discussed in Sec. 2.2.

HARP-Net’s skip AEs all follow the similar architecture to the baseline AE,
but in a smaller scale with only three hidden layers and 24 filters per layer.
The bottleneck is once again reduced to the original input dimension before
quatization. We learn $32$ cluster means per each bottleneck quantization task
$\bm{y}^{(l)}$ as well as the main bottleneck $\bm{x}^{(L)}$, leading to a
$5$-bit quantization. To limit the dynamic range of the codes, we use the
hyperbolic tangent function as the final activation of the encoders.

###  4.2 Training Process

We first compute the residual LPC signals from the entirety of our dataset
along their respective LPC coefficients. The residual signals are then scaled
up by $100$ to make up the low amplitudes before being fed to the networks.
Following the same method proposed in [5], we first let the network train
without quantization for about 8 epochs, then introduce the soft-to-hard
quantization module. We opt to progressively anneal $\alpha$ at a constant
rate ($0.3$ per epoch). We define our network loss function as a combination
of the sum of squared error and entropy regularizer:

$\pazocal{L}=\sum_{t}(x_{t}-\hat{x}_{t})^{2}+\lambda\sum_{m=0}^{M}{\mathcal{H}}^{(L-m)},$
(6)

where $\lambda$ controls the amount of entropy regularization, which we
dynamically change during the training process. We define the entropy
regularizer as the sum of estimated entropy values of all $M$ participating
layers. Given that, a stronger regularization (with a large $\lambda$)
suppresses the totaly entropy more and vice versa. The actual entropy control
works by monitoring all ${\mathcal{H}}^{(L-m)}$ values and by adjusting
$\lambda$ accordingly until the sum reaches the target entropy.

###  4.3 Experimental Results and Discussion

As an objective measure, we first compute the signal-to-noise ratio (SNR) on
the test set. While SNR values are not completely reliable, their correlation
with subjective score can sometime be found high [20]. Hence, we use SNR
scores to get a general idea of the models’ performance among the variants of
HARP-Net. We eventually choose the most promising variant for the subjective
tests.

(a) 40kbps

(b) 64kbps

Figure 2: SNR results of HARP-Net models and their associated baselines. Note
that the baselines do not have skip connections. Figure 3: MUSHRA evaluation
results in boxplots; green dotted line depicts the mean while the red line
represent the median.

Fig.2 shows the average test SNR scores. We found that the low bitrate systems
are affected more by the architectural choice, so we choose HARP-Net 3 and
Baseline 3 as the best models used for subjective tests. However, note that
this choice is not completely reliable due to its objective nature. Meanwhile,
HARP-Net systems are not affected too much by the number of skip connections
in the high bitrate cases, so we choose HARP-Net 1. It is noticeable that the
baselines’ performance does not improve much by increasing target bitrate. We
choose Baseline 3 as the best model for 64kbps.

The subjective test follows ITU’s MUSHRA guideline [21]. A session consists of
nine different trials, each of which includes song segments from the same test
set we have set aside. We select these segments in a manner that the subjects
are exposed to a wide variety of music genres, consequently covering a range
of auditory qualities and aspects. Each trial includes a hidden reference
along with two low-pass filtered anchors per the MUSHRA specification. The
systems in comparison are two HARP-Net models in two bitrates and their
corresponding baselines that are chosen based on the SNR scores. Finally, we
also include the MP3 decoded signals using Adobe Audition ®(licensed from
Fraunhofer IIS and Thomson) at 64kbps. Note, however, that the MP3 signals are
downsampled to $32$kHz by default. We recruit 12 audio experts, but excluded
one of them who did not meet the post-screening guideline [21].

In Fig.3, both of our proposed systems (HARP-Net 64kbps and 40kbps) outperform
their baseline counterparts. Moreover, we observe that the 40kbps HARP-Net
performs within the same score span as the 64kbps baseline. Lastly, we note
that HARP-Net 64kbps results lay close to the MP3 performance. Given that our
LPC part is not optimal and our models cover wider bandwidth, i.e., 44.1kHz
instead of 32kHz, the HARP-Net results are promising.

##  5 Conclusions

In this work we proposed a novel, lightweight, neural audio coding system,
HARP-Net. It is defined as a mirrored AE with interconnected encoder-decoder
layers via additional skip AEs. We found that those additional skip AEs create
information paths that circumvent the lossy bottleneck quantization process,
thus improving the total AE reconstruction performance. The skip AEs are
carefully designed to work as a mini codec not to consume too much bitrates.
Listening tests verified that the subjects prefer HARP-Net to the basic AE-
based codecs. In the future, we will improve the LPC module’s bitrate, which
can go down to 2.4kbps, and incorporate perceptually-motivated losses as
proposed in [12]. Source codes and sound examples are available at
https://saige.sice.indiana.edu/research-projects/harp-net.

## References

  * [1] ISO/IEC DIS 23003-3, “Information technology – MPEG audio technologies – part 3: Unified speech and audio coding,” 2011.
  * [2] ISO/IEC 14496-3:2009/PDAM 3, “Transport of unified speech and audio coding (USAC),” 2011.
  * [3] B. Bessette, R. Salami, R. Lefebvre, M. Jelinek, J. Rotola-Pukkila, J. Vainio, H. Mikkola, and K. Jarvinen, “The adaptive multirate wideband speech codec (AMR-WB),” _IEEE Transactions on Speech and Audio Processing_ , vol. 10, no. 8, pp. 620–636, 2002.
  * [4] J. M. Valin, K. Vos, and T. Terriberry, “Definition of the opus audio codec,” _IETF, September_ , 2012.
  * [5] S. Kankanahalli, “End-to-end optimized speech coding with deep neural networks,” in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2018.
  * [6] K. Zhen, M. S. Lee, J. Sung, S. Beack, and M. Kim, “Efficient and scalable neural residual waveform coding with collaborative quantization,” in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2020.
  * [7] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” _arXiv preprint arXiv:1609.03499_ , 2016.
  * [8] W. B. Kleijn, F. S. C. Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and T. C. Walters, “WaveNet based low rate speech coding,” in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2018, pp. 676–680.
  * [9] Y. L. C. Garbacea, A. van den Oord, “Low bit-rate speech coding with VQ-VAE and a wavenet decoder,” in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2019.
  * [10] J.-M. Valin and J. Skoglund, “LPCNet: Improving neural speech synthesis through linear prediction,” in _Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)_ , 2019.
  * [11] ——, “A real-time wideband neural vocoder at 1.6 kb/s using LPCNet,” in _Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech)_ , 2019.
  * [12] K. Zhen, M. S. Lee, J. Sung, S. Beack, and M. Kim, “Psychoacoustic calibration of loss functions for efficient end-to-end neural audio coding,” _IEEE Signal Processing Letters_ , vol. 27, pp. 2159–2163, 2020.
  * [13] K. Zhen, J. Sung, M. S. Lee, S. Beack, and M. Kim, “Cascaded cross-module residual learning towards lightweight end-to-end speech coding,” in _Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech)_ , 2019.
  * [14] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [15] D. Stoller, S. Ewert, and S. Dixon, “Wave-u-net: A multi-scale neural network for end-to-end audio source separation,” in _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , 2018\.
  * [16] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” _Science_ , vol. 313, no. 5786, pp. 504–507, 2006\.
  * [17] E. Agustsson, F. Mentzer, M. Tschannen, L. Cavigelli, R. Timofte, L. Benini, and L. V. Gool, “Soft-to-hard vector quantization for end-to-end learning compressible representations,” in _Advances in Neural Information Processing Systems (NIPS)_ , 2017, pp. 1141–1151.
  * [18] A. Jansson, E. Humphrey, N. Montecchio, R. Bittner, A. Kumar, and T. Weyde, “Singing voice separation with deep u-net convolutional networks,” in _Proceedings of the International Society for Music Information Retrieval Conference (ISMIR)_ , October 2017.
  * [19] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” in _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 1874–1883.
  * [20] V. Emiya, E. Vincent, N. Harlander, and V. Hohmann, “Subjective and objective quality assessment of audio source separation,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 7, pp. 2046–2057, 2011\.
  * [21] ITU-R Recommendation BS 1534-1, “Method for the subjective assessment of intermediate quality levels of coding systems (MUSHRA),” 2003.