# Virtual Avatar Stream: A cost-down approach to the metaverse experience

Joseph Chang
<EMAIL_ADDRESS>

###### Abstract

The Metaverse through VR headsets is a rapidly growing concept, but the high
cost of entry currently limits access for many users. This project aims to
provide an accessible entry point to the immersive Metaverse experience by
leveraging web technologies. The platform developed allows users to engage
with rendered avatars using only a web browser, microphone, and webcam. By
employing the WebGL and MediaPipe face tracking AI model from Google, the
application generates real-time 3D face meshes for users. It uses a client-to-
client streaming cluster to establish a connection, and clients negotiate SRTP
protocol through WebRTC for direct data streaming. Additionally, the project
addresses backend challenges through an architecture that is serverless,
distributive, auto-scaling, highly resilient, and secure. The platform offers
a scalable, hardware-free solution for users to experience a near-immersive
Metaverse, with the potential for future integration with game server
clusters. This project provides an important step toward a more inclusive
Metaverse accessible to a wider audience.

_K_ eywords Metaverse $\cdot$ Face Mesh $\cdot$ Streaming

##  1 Introduction

The Metaverse is a digital universe that seeks to replicate the physical world
through technologies [22]. With the advancement of virtual reality (VR)
technologies, VR headsets have become a primary interface for accessing the
Metaverse. The immersive experience provided by VR headsets has the potential
to revolutionize human interactions. These devices can capture upper body
movement data from the user and create a virtual avatar, allowing users to
move in a virtual digital world by moving their physical bodies. In addition,
VR headsets have two screens and speakers. The screens produce parallax based
on the distance of objects in the virtual world, mimicking the sense of 3D
vision humans have in the real world. The speakers produce stereophonic sound
to simulate the 3D vision and sound direction of the real world. By combining
body movement data with the digital interface, VR headsets create a virtual
reality experience that simulates the spatial environment of the physical
world. This technology enables users to interact with one another in a virtual
yet spatial environment, regardless of their physical location. This has
numerous applications, including virtual conferences, gaming, and education.

However, the high cost of entry has hindered consumers from investing in this
technology, particularly those who do not understand or have not experienced
the potential of the immersive Metaverse. Here, we propose to provide an
entry-level immersive Metaverse experience by leveraging face tracking, 3D
mesh rendering, and live streaming technologies. Furthermore, our solution
leads to other applications, such as a virtual Omegle chat and a setup for
virtual YouTubers. The former provides a safe space for one-on-one video chats
with random users while displaying facial movements and expressions on avatar
mesh. The latter eliminates the need for costly tracking headsets by using
software to track and animate characters with a webcam.

To offer a partially immersive Metaverse experience, we provide a solution to
replace the 3D conference call feature from Meta, which requires a costly VR
headset [6]. In brief, we use an AI Face Mesh model to track the user’s facial
expressions, position, and orientation, to render 3D mesh data into a 2D
screen through a rendering pipeline done on WebGL, allowing for a simulated 3D
conference experience using only a webcam, microphone, and web browser. The
current implementation enables two users to engage in a virtual video call
while displaying their avatar faces.

In summary, we demonstrate that companies like Meta can use this product to
offer an entry-level Metaverse experience through a web portal rather than
requiring consumers to invest in expensive hardware. By providing these
accessible applications, our solution is expected to bridge the gap for
consumers to experience the immersive Metaverse.

##  2 Face Tracking

This product utilizes the Google MediaPipe library’s AI model, specifically,
the MediaPipe Face Mesh model. The model takes images as input and generates
468 feature coordinates for each frame on the user’s face as output [12]. The
model is composed of two deep neural network sub-models: the Face Detection
Model and the Face Landmark Model. The Face Detection Model employs the
BlazeFace model [5] to locate, crop, and predict rotations for the faces in a
video. The cropped face with rotation is then transformed into a flat upright
rectangle of 128 or 256 square pixels, which is fed into the Face Landmark
Model to generate 3D landmark vectors [19]. These vectors can be used to
calculate the predicted 3D facial landmark coordinates and mapped back to the
cropped face with rotation. The Face Detection Model uses a CNN architecture
[5], while the Face Landmark Model employs ResNet, which is also a CNN
architecture [19]. Both models have been optimized for GPU acceleration to
enable applications on personal devices [5, 19]. Examples of the models’
outputs can be seen in figures 2 and 2. The system captures real-time images
from the user’s webcam and processes them with the Face Mesh model to retrieve
facial landmarks, which are then used for rendering.

Figure 1: Face Detection Model output example adapted from BlazeFace [5]. The
green line indicates the output rectangle used in the Face Mesh pipeline.

Figure 2: Face Landmark Model output examples adapted from [19].

##  3 Rendering

###  3.1 Graphic Pipeline

The platform uses WebGL for graphic rendering. WebGL is a JavaScript
implementation of OpenGL, a widely-used graphics API that enables developers
to construct GPU-accelerated rendering pipelines. These pipelines consist of
several stages, including transformation, vertex shading, fragment shading,
and perspective projection. The transformation stage involves a series of
operations that are represented by equations 1 to 5. Homogeneous coordinates,
a 4x4 matrix representation for 3D computation, are used in all 3D operations
to facilitate translation.

$\displaystyle Homogeneous\ coordinates\quad$ $\displaystyle P_{init}$
$\displaystyle=\begin{bmatrix}x\\\ y\\\ z\\\ 1\end{bmatrix}$ (1)
$\displaystyle Scale\quad$ $\displaystyle S$
$\displaystyle=\begin{bmatrix}S_{x}&0&0&0\\\ 0&S_{y}&0&0\\\ 0&0&S_{z}&0\\\
0&0&0&1\end{bmatrix}$ (2) $\displaystyle Rotation\quad$ $\displaystyle
R_{x}(\theta)$ $\displaystyle=\begin{bmatrix}1&0&0&0\\\
0&cos\theta&-sin\theta&0\\\ 0&sin\theta&cos\theta&0\\\ 0&0&0&1\end{bmatrix}$
(3) $\displaystyle Translation\quad$ $\displaystyle T$
$\displaystyle=\begin{bmatrix}1&0&0&T_{x}\\\ 0&1&0&T_{y}\\\ 0&0&1&T_{z}\\\
0&0&0&1\end{bmatrix}$ (4) $\displaystyle Transformation\quad$ $\displaystyle
P_{after}$ $\displaystyle=M_{trans}\times P_{init}$ (5)

However, homogeneous coordinates are not required for perspective projection,
and the projection matrix can be reduced to a 3x3 matrix to reduce
computational overhead. Once a camera and projection screen are defined, the
perspective projection algorithm can be implemented using equations 6 and 7.
This approach allows for more efficient rendering of 3D scenes.

$\displaystyle\begin{split}\begin{bmatrix}f_{x}\\\ f_{y}\\\
f_{w}\end{bmatrix}&=\begin{aligned}
\begin{bmatrix}1&&0&&\frac{e_{x}}{e_{z}}\\\ 0&&1&&\frac{e_{y}}{e_{z}}\\\
0&&0&&\frac{1}{e_{z}}\end{bmatrix}\begin{bmatrix}1&&0&&0\\\
0&&cos\theta_{x}&&sin\theta_{x}\\\
0&&-sin\theta_{x}&&cos\theta_{x}\end{bmatrix}\begin{bmatrix}cos\theta_{y}&&0&&-sin\theta_{y}\\\
0&&1&&0\\\ sin\theta_{y}&&0&&cos\theta_{y}\end{bmatrix}\\\
\begin{bmatrix}cos\theta_{z}&&sin\theta_{z}&&0\\\
-sin\theta_{z}&&cos\theta_{z}&&0\\\
0&&0&&1\end{bmatrix}\begin{pmatrix}\begin{bmatrix}p_{x}\\\ p_{y}\\\
p_{z}\end{bmatrix}-\begin{bmatrix}c_{x}\\\ c_{y}\\\
c_{z}\end{bmatrix}\end{pmatrix}\end{aligned}\end{split}$ (6)
$\displaystyle\begin{bmatrix}b_{x}\\\ b_{y}\end{bmatrix}$
$\displaystyle=\begin{bmatrix}f_{x}/f_{w}\\\ f_{y}/f_{w}\end{bmatrix}$ (7)

$\displaystyle p:\ The\ coordinates\ of\ a\ vertex$ $\displaystyle c:\ The\
coordinates\ of\ the\ camera$ $\displaystyle e:\ The\ display\ surface\ of\
the\ projection\ screen$ $\displaystyle f:Homogenous\ coordinates\ of\ 2D\
point$ $\displaystyle b:2D\ display\ coordinates$

###  3.2 Vertices and Calibration

The vertices of the face mesh are computed by grouping the landmarks into
triangles, as shown in figure 3. The vertices are generated by performing both
clockwise and counterclockwise rotations, resulting in a mesh that is filled
from both sides. As a result, if the mesh is rotated by 180 degrees, it will
not appear transparent.

To ensure the user’s face is calibrated with the 3D-rendered avatar face,
spatial translation and rotation are applied. It expects the user to have
their face-centered and upright at the initial frame. And it transforms the
avatar’s face so that the initial frame’s nose is centered, and the triangle
formed by the nose and two eyelashes lies on a flat surface in the graphic
space with its normal pointing outward from the screen. The calibration
conditions are expressed in equations 8, 9, 10, and 11. The facial landmarks
used to construct the face mesh are depicted in figure 3.

$\displaystyle landmark_{5}$ $\displaystyle\Leftarrow\begin{bmatrix}0\\\ 0\\\
0\end{bmatrix}$ (8) $\displaystyle\vec{up}$
$\displaystyle=(\frac{landmark_{52}-landmark_{282}}{2})-landmark_{1}$ (9)
$\displaystyle\vec{right}$ $\displaystyle=landmark_{282}-landmark_{52}$ (10)
$\displaystyle\hat{up}\times\hat{right}$
$\displaystyle\Leftarrow\begin{bmatrix}0\\\ 0\\\ 1\end{bmatrix}=\hat{z}$ (11)

$\displaystyle landmark_{5}$ $\displaystyle:\quad center\ nose$ $\displaystyle
landmark_{1}$ $\displaystyle:\quad lower\ nose$ $\displaystyle landmark_{52}$
$\displaystyle:\quad left\ eyelash$ $\displaystyle landmark_{282}$
$\displaystyle:\quad right\ eyelash$

Figure 3: Face mesh with landmark number labels adapted from [27].

To achieve the condition given in equation 11, the calibration algorithm
measures the off-angles of the up and right vectors and rotates the face mesh
by these off-angles using quaternion angles. The calibration algorithm
performs equations 12, 13, and 14 to ensure the orientation condition is met.

$\displaystyle Q_{up}$ $\displaystyle=QuaternionAngle(\hat{up},\hat{y})$ (12)
$\displaystyle Q_{right}$ $\displaystyle=QuaternionAngle(\hat{right},\hat{x})$
(13) $\displaystyle Q_{FaceMeshRotation}$ $\displaystyle=Q_{up}\times
Q_{right}$ (14)

###  3.3 Quaternions

Quaternions are a four-dimensional system that uses $\vec{x}$, $\vec{y}$,
$\vec{z}$, and $\vec{w}$ to represent a rotation. An alternative way to
represent a rotation is to use Euler angles of $\phi$, $\theta$, and $\psi$
with a rotation order such as XYZ to achieve $roll$, $yaw$, and $pitch$. The
quaternion system is used because it avoids Gimbal lock. The matrix
representation of Euler rotation in XYZ rotating sequence and the derivation
of quaternions from the Euler angles are shown in equations 15 and 16, adapted
from [17].

$\displaystyle\begin{split}\begin{bmatrix}x_{after}\\\ y_{after}\\\
z_{after}\end{bmatrix}&\begin{aligned}
&=R_{z}(\psi)R_{y}(\theta)R_{x}(\phi)\begin{bmatrix}x_{init}\\\ y_{init}\\\
z_{init}\end{bmatrix}\\\ &=\begin{bmatrix}cos\psi&-sin\psi&0\\\
sin\psi&cos\psi&0\\\ 0&0&1\end{bmatrix}\begin{bmatrix}cos\theta&0&sin\theta\\\
0&1&0\\\ -sin\theta&0&cos\theta\end{bmatrix}\begin{bmatrix}1&0&0\\\
0&cos\phi&-sin\phi\\\ 0&sin\phi&cos\phi\end{bmatrix}\begin{bmatrix}x_{init}\\\
y_{init}\\\ z_{init}\end{bmatrix}\end{aligned}\end{split}$ (15)
$\displaystyle\begin{bmatrix}Q_{x}\\\ Q_{y}\\\ Q_{z}\\\ Q_{w}\end{bmatrix}$
$\displaystyle=\begin{bmatrix}sin\frac{\phi}{2}cos\frac{\theta}{2}cos\frac{\psi}{2}-cos\frac{\phi}{2}sin\frac{\theta}{2}sin\frac{\psi}{2}\\\
cos\frac{\phi}{2}sin\frac{\theta}{2}cos\frac{\psi}{2}+sin\frac{\phi}{2}cos\frac{\theta}{2}sin\frac{\psi}{2}\\\
cos\frac{\phi}{2}cos\frac{\theta}{2}sin\frac{\psi}{2}-sin\frac{\phi}{2}sin\frac{\theta}{2}cos\frac{\psi}{2}\\\
cos\frac{\phi}{2}cos\frac{\theta}{2}cos\frac{\psi}{2}+sin\frac{\phi}{2}sin\frac{\theta}{2}sin\frac{\psi}{2}\end{bmatrix}$
(16)

##  4 Stream

###  4.1 Infrastructure

The platform is deployed using AWS (Amazon Web Services), and the architecture
described is a best practice cluster deployment suggested by AWS [1]. The
resources for the application are grouped within a VPC (Virtual Private Cloud)
consisting of public and private subnets. The ALB (Application Load Balancer)
is deployed within the public subnets to receive the traffic from the public
network and distribute them to the cluster. The ECS (Elastic Container
Service) cluster and Aurora database are deployed within the private subnets
to that both of them are not directly accessible from the public network. The
redundant subnets ensure the application survives on a zone outage.

The system is distributive and horizontally scalable, with multiple instances
monitored for CPU and memory usage to achieve resiliency and auto-scaling.
Terraform, an infrastructure-as-code software tool [16], is used to manage the
deployment procedure, which allows for robust infrastructure management. This
architecture is highly versatile and can be used to support a wide range of
applications, including popular services like Discord or online games that
need to handle millions of users concurrently. The architecture diagram is
shown in figure 4.

Figure 4: architecture diagram of virtual avatar stream

###  4.2 Protocols

In this system, WebSocket, a TCP data streaming protocol, is used to establish
and maintain the connection between the front end and the backend. The
streaming protocol is essential in helping front-end clients find each other
and negotiate peer-to-peer protocols. Since negotiation requires back-and-
forth data exchange until an agreement is reached, streaming becomes
necessary. As the system is distributed, clients with the same unique room ID
may not always land on the same cluster instance. In the event that clients
open WebSocket on different distributed cluster instances, the instance will
look up the private IP in the database and create a proxy connection between
instances to enable data swapping.

WebRTC, an SRTP protocol that streams data from client to client directly
[10], is used to avoid the backend handling sensitive data such as user voice
and face mesh. The front end uses the backend to negotiate the offer, answer,
and ICE candidate for WebRTC, and the client starts to send audio and face
mesh data, including the mesh translation and rotation, to another client once
the peer-to-peer connection is established using WebRTC. The voice is sent
through a streaming track, and the face mesh is sent through a data channel.

###  4.3 Compression

When the internet speed is slow or the distance between clients is long, the
data channel can suffer from network congestion, which is solved by
compressing the landmark data to float 16, the half-precision floating-point.
The payload size is compressed to 2838 bytes per frame, and the max fps is set
to 30, so the data rate is about 85 KB/s. The data rate of a typical YouTube
video, 1080p 30fps with an H264 compression ratio of 2000:1 [26], is 93 KB/s
which shows the platform data rate of 85 KB/s is operable. The derivation of
the H264 video data rate is shown in equation 17.

$1080\times 1920\times 3\frac{bytes}{pixel}\times
30fps\times\frac{1}{2000}=93KB/s$ (17)

The IEEE standards for float 32 and float 16 are depicted in figures 6 and 6,
respectively. To convert a float 32 to float 16, the sign bit remains
unchanged, the mantissa is truncated, and the leftmost 10 bits from float 32
are stored in the mantissa of float 16. Meanwhile, the exponent undergoes the
arithmetic in equation 18, but special conditions of overflow and underflow
must be taken into account.

$E_{float16}=E_{float32}-\text{0x7F}+\text{0xF}=E_{float32}-127+15$ (18)

Figure 5: IEEE single-precision floating-point (float 32) standards adapted
from [2].

Figure 6: IEEE half-precision floating-point (float 16) standards adapted from
[3]. The "T" here is mantissa.

##  5 Results

The project successfully demonstrates the potential of delivering a VR headset
Metaverse experience on the web using conventional hardware that most
customers already have. It utilizes a combination of open-source libraries and
web technologies, including Google MediaPipe, ThreeJS, WebSocket, WebRTC, and
AWS. The final product is available on the website:
virtualavatar.trip1elift.com. The source code is on the repository:
github.com/Trip1eLift/virtual-avatar, with a user guide:
github.com/Trip1eLift/virtual-avatar/tree/main/userguide.

Upon the first visit, the user is prompted to grant browser access to the
camera and microphone. The camera is used to capture the image of the user’s
face, which is processed by the 3D face tracking AI model of MediaPipe to
obtain facial landmarks. These landmarks are rendered into 2D graphics using
the ThreeJS library, with randomly generated colors. The microphone is
utilized when the user engages in a room chat with another user to collect
audio data. The website aligns the user’s nose and eyelashes with a flat
surface of a normal vector pointing out of the screen in the space. The user
can modify, store, and load calibration settings manually. Figure 7 shows the
calibration
interface.https://www.overleaf.com/project/6408de81f53a680b6c52b908

Figure 7: virtual avatar stream mesh control panel for calibration

Once calibration settings are configured, the user can begin streaming. Two
types of users are supported: room owners and guests. The room owner can
create a room by clicking the "create room" button in the streaming control
panel. The streaming cluster generates a unique room ID that the room owner
must share with the guest through an external channel. Since the website
doesn’t have a user account system, it doesn’t support friend calling. The
room owner maintains a WebSocket connection to the backend and waits for the
guest to join. During scale-down, both the room owner and guest may land on
the same distributive cluster instance, allowing the front ends to negotiate
the WebRTC protocol on the same instance. During traffic scaling, the room
owner and guest may land on different distributive cluster instances. The
instance the guest is on uses the room ID to obtain the private IP of the
instance the room owner is on from the database, and the guest instance
proxies WebSocket data to the owner’s instance. The two parties negotiate the
WebRTC protocol through the proxy.

After the WebRTC connection between the front ends is established, both hang
up their WebSocket connections to the backend cluster. Mesh data streaming and
audio data streaming commence. The mesh data is compressed to 43 KB/s to
prevent network congestion. Figure 8 illustrates the streaming interface. This
product demonstrates that an immersive Metaverse experience with VR headsets
can be brought to the web without requiring a VR headset.

Figure 8: virtual video call between two users in room 90 using virtual avatar
stream

The front end of the product is developed using ReactJS [28] and is
supplemented with various API documentation resources [21, 11, 14, 23, 25,
10]. Additionally, the front end is bolstered with the use of npm packages
[29, 13, 8, 24, 20, 30]. The back-end is built with golang [4] and leverages
open-source packages [7, 9]. The system is containerized using Docker [18] and
is deployed to AWS [1] with the help of Terraform [16]. The product employs
Postgres [15] as its database system.

##  6 Discussion

The current product demonstrates that it is possible to replace the facial
movements and expressions of a VR headset on a web-based platform using
conventional hardware. However, the potential of this product is much higher
than what it currently offers. Many VR headsets on the market come with VR
controllers that allow users to control their hand movements in the virtual
world. While Google MediaPipe supports upper body spine, arms, and finger
tracking, this platform does not support these body parts due to the
limitations of its basic graphic engine. To render the details of arm and
finger movements, a more advanced physics engine is needed to avoid mesh
clipping or other graphic glitches.

Another area that can be improved upon is face mesh rendering. The Metaverse
allows users to become whatever they want, like a humanoid with a face of a
cat or a dog. The challenge of implementing such a feature lies in the graphic
engine. A more advanced physics/graphic engine is needed to support vertices
mapping with weight to map the face mesh onto a dog face while interpolating
the facial expression.

Additionally, there is a gap between this platform and the Metaverse that runs
on a VR headset in terms of users’ visual perception due to the output
interface. In a VR headset, users can see things in 3D because the two lenses
on the VR headset create parallax to mimic 3D visualization. This project uses
a traditional computer screen as the output interface, and while the data was
captured in 3D, it projects the 3D data into a 2D space to show it to the
user. This limitation can be solved by using a red-blue 3D image rendering,
which any colored 2D monitor can output. The cost of red-blue 3D glasses made
out of cardboard is minimal, so with an upgrade of the graphic engine and a
minimal cost, users can experience 3D visuals that they could only experience
on a VR headset.

Another limitation of the current platform is that it only supports one-on-one
video chat and does not offer 3D spatial sound, which is a standard feature of
most Metaverse platforms. However, in the future, this product can be scaled
to support a conference room of many people. Implementing 3D spatial sound can
be achieved by offsetting the sound of left and right sound tracks and
decaying the intensity of the sound based on the energy decay of the inverse-
square law.

##  7 Conclusion

In this project, we have demonstrated the potential of creating an immersive
Metaverse experience with the usage of conventional cameras and microphones.
We have built a scalable client-to-client streaming backend that can handle
millions of users, which can be extended to other applications. Our product
leverages Google MediaPipe’s 3D face tracking model to convert webcam data
into face mesh landmarks, which are rendered using ThreeJS into 3D computer
graphics. We use WebSocket and WebRTC protocols to enable direct streaming of
the face mesh and audio data between two users for a virtual avatar video
call. Our compression techniques ensure that network congestion is prevented
and the streaming quality is maintained at an acceptable level.

Although the current experience provided by our product is not as immersive as
that provided by a VR headset, it offers an entry-level experience for
beginners to understand the concept of the Metaverse. With further
development, it is possible to enhance the product to provide an experience
that is nearly as good as that of a VR headset. We believe that our platform
can serve as a stepping stone for future innovations in the field of the
Metaverse.

## References

  * [1] Amazon  “Introduction - Amazon Elastic Container Service”  In _AWS_ Amazon, 2023  URL: https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/intro.html
  * [2] arm  “6.1. Floating-point basics and the IEEE-754 standard”  In _arm Developer_ , 2023  URL: https://developer.arm.com/documentation/den0042/a/Floating-Point/Floating-point-basics-and-the-IEEE-754-standard
  * [3] arm  “Half-precision floating-point number format”  In _arm Developer_ , 2023  URL: https://developer.arm.com/documentation/100067/0607/Other-Compiler-specific-Features/Half-precision-floating-point-number-format
  * [4] The Go Authors  “Documentation”  In _Go_ , 2009  URL: https://go.dev/doc/
  * [5] Valentin Bazarevsky et al.  “BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs”  In _arXiv preprint arXiv:1907.05047 [cs.CV]_ , 2019
  * [6] Meghan Bobrowsky  “Meta Quest Pro VR Headsets Get Price Cut Amid TikTok Parent Competition”  In _Wall Street Journal_ , 2023  DOI: https://www.wsj.com/articles/meta-slashes-price-for-quest-pro-vr-headset-months-after-launch-d61f8e30
  * [7] Gary Burd, Alexander Emelin, Matt Silverlock and Joachim Bauch  “Gorilla WebSocket”  In _GitHub_ , 2022  URL: https://github.com/gorilla/websocket
  * [8] Ricardo Cabello and Michael Herzog  “three.js”  In _npm_ , 2023  URL: https://www.npmjs.com/package/three
  * [9] Jack Christensen  “pgx - PostgreSQL Driver and Toolkit”  In _GitHub_ , 2023  URL: https://github.com/jackc/pgx
  * [10] Google  “Getting started with WebRTC”  In _WebRTC_ , 2019  URL: https://webrtc.org/getting-started/overview
  * [11] Google  “Google/mediapipe: Cross-platform, customizable ML solutions for live and streaming media.”  In _GitHub_ Google, 2019  URL: https://github.com/google/mediapipe
  * [12] Google  “MediaPipe Face Mesh”  In _mediapipe_ , 2023  URL: https://google.github.io/mediapipe/solutions/face_mesh.html
  * [13] Google  “MediaPipe JavaScript Solutions”  In _npm_ , 2023  URL: https://www.npmjs.com/package/@mediapipe/pose
  * [14] Khronos Group  “WebGL: 2D and 3D graphics for the web - web apis: MDN”  In _Web APIs | MDN_ Khronos Group, 2011  URL: https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API
  * [15] The PostgreSQL Global Development Group  “PostgreSQL 14.7 documentation”  In _PostgreSQL Documentation_ , 2023  URL: https://www.postgresql.org/docs/14/index.html
  * [16] HashiCorp  “AWS Provider”  In _Terraform registry_ , 2023  URL: https://registry.terraform.io/providers/hashicorp/aws/latest/docs
  * [17] D.. Henderson  “Shuttle Program. Euler angles, quaternions, and transformation matrices working relationships”  In _NASA Technical Reports Server_ , 1977  DOI: https://ntrs.nasa.gov/citations/19770024290
  * [18] Docker Inc.  “Docker Overview”  In _Docker Documentation_ , 2023  URL: https://docs.docker.com/get-started/overview/
  * [19] Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko and Matthias Grundmann  “Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs”  In _arXiv preprint arXiv:1907.06724 [cs.CV]_ , 2019
  * [20] Moz Morris  “React Webcam”  In _npm_ , 2022  URL: https://www.npmjs.com/package/react-webcam
  * [21] MUI  “Overview - material UI”  In _Overview - Material UI_ , 2023  URL: https://mui.com/material-ui/getting-started/overview/
  * [22] Stylianos Mystakidis  “Metaverse”  In _Encyclopedia_ 2.1, 2022, pp. 486–497  URL: https://www.mdpi.com/2673-8392/2/1/31
  * [23] Three.js Org  “Creating a scene”  In _three.js docs_ URL: https://threejs.org/docs/index.html#manual/en/introduction/Creating-a-scene
  * [24] Nikhil Saraf, Faraz Shaikh, Cody Bennett and Issaac Mason  “@react-three/fiber”  In _npm_ , 2023  URL: https://www.npmjs.com/package/@react-three/fiber
  * [25] Open Source  “Introduction”  In _React Three Fiber Documentation_ , 2023  URL: https://docs.pmnd.rs/react-three-fiber/getting-started/introduction
  * [26] RGB Spectrum  “H.264 Profiles”  In _RGB Spectrum_ , 2023  URL: https://www.rgb.com/h264-profiles#:~:text=High%20Profile-,H.,ratio%20of%20about%202000%3A1.
  * [27] Google MediaPipe Team  “canonical-face-model-uv-visualization.png”  In _GitHub_ , 2020  URL: https://github.com/google/mediapipe/blob/master/mediapipe/modules/face_geometry/data/canonical_face_model.fbx
  * [28] Jordan Walke  “React – a JavaScript library for building user interfaces”  In _React_ Meta  URL: https://reactjs.org/
  * [29] Albert Yu, Benny Joo, Matthew Brookes and Olivier Tassinari  “@mui/material”  In _npm_ , 2023  URL: https://www.npmjs.com/package/@mui/material
  * [30] Matt Zabriskie and Samuel Reed  “React-Draggable”  In _npm_ , 2022  URL: https://www.npmjs.com/package/react-draggable


# STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency

Zhong-Qiu Wang, Gordon Wichern, Shinji Watanabe, and Jonathan Le Roux
Manuscript received on Mar. 14, 2022. Z.-Q. Wang and S. Watanabe are with the
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA
15213, USA (e-mail<EMAIL_ADDRESS>shinjiw@cmu.edu). G. Wichern
and J. Le Roux are with Mitsubishi Electric Research Laboratories, Cambridge,
MA 02139, USA (e-mail: {wichern,leroux}@merl.com).

###### Abstract

Deep learning based speech enhancement in the short-term Fourier transform
(STFT) domain typically uses a large window length such as 32 ms. A larger
window contains more samples and the frequency resolution can be higher for
potentially better enhancement. This however incurs an algorithmic latency of
32 ms in an online setup, because the overlap-add algorithm used in the
inverse STFT (iSTFT) is also performed based on the same 32 ms window size. To
reduce this inherent latency, we adapt a conventional dual window size
approach, where a regular input window size is used for STFT but a shorter
output window is used for the overlap-add in the iSTFT, for STFT-domain deep
learning based frame-online speech enhancement. Based on this STFT and iSTFT
configuration, we employ single- or multi-microphone complex spectral mapping
for frame-online enhancement, where a deep neural network (DNN) is trained to
predict the real and imaginary (RI) components of target speech from the
mixture RI components. In addition, we use the RI components predicted by the
DNN to conduct frame-online beamforming, the results of which are then used as
extra features for a second DNN to perform frame-online post-filtering. The
frequency-domain beamforming in between the two DNNs can be easily integrated
with complex spectral mapping and is designed to not incur any algorithmic
latency. Additionally, we propose a future-frame prediction technique to
further reduce the algorithmic latency. Evaluation results on a noisy-
reverberant speech enhancement task demonstrate the effectiveness of the
proposed algorithms. Compared with Conv-TasNet, our STFT-domain system can
achieve better enhancement performance for a comparable amount of computation,
or comparable performance with less computation, maintaining strong
performance at an algorithmic latency as low as 2 ms.

###### Index Terms:

Frame-online speech enhancement, complex spectral mapping, microphone array
processing, deep learning.

##  I Introduction

Deep learning has dramatically advanced speech enhancement in the past decade
[1]. Early studies estimated target magnitudes via time-frequency (T-F)
masking [2] or directly predicted target magnitude via spectral mapping [3],
both using the mixture phase for signal re-synthesis. Subsequent studies
strove to improve phase modelling by performing phase estimation via
magnitude-driven iterative phase reconstruction [4]. Recent effort focuses on
complex- and time-domain approaches [5, 6, 7, 8, 9, 10, 11], where magnitude
and phase are modelled simultaneously through end-to-end optimization.

Many application scenarios such as teleconferencing and hearing aids require
low-latency speech enhancement. Deep learning based approaches [12, 8, 13, 14,
15, 16] handle this by using causal DNN blocks, such as uni-directional LSTMs,
causal convolutions, causal attention layers, and causal normalization layers.
Although many previous studies along this line advocate that their system with
a causal DNN model is causal, one should be aware that most of these systems
are, to be more precise, frame-online, and the amount of look-ahead depends on
the frame length. One major approach that can potentially achieve sample-level
causal processing is by using WaveNet-like models [17]. However, their
effectiveness in dealing with noise and reverberation in a sample-causal setup
is unclear [18]. In addition, at run time such models need to run a forward
pass for each sample, resulting in a humongous and likely unnecessary amount
of computation. Popular STFT- and time-domain approaches typically split
signals into overlapped frames with a reasonably large hop length before
processing. One advantage is that the forward pass then only needs to be run
every hop-length samples. The latency is however equal to the window length
due to the use of overlap-add in signal re-synthesis, plus the running time of
processing one frame (see Fig. 1 and its caption for a detailed explanation of
the latency). In the recent Deep Noise Suppression (DNS) challenge [19], one
key requirement was that the latency of producing an estimate for the sample
at index $n$ cannot exceed 40 ms on a standard Intel Core i5 processor. For a
typical STFT based system with a 32 ms window and an 8 ms hop size, a frame-
online DNN based system satisfies the requirement if the processing of each
frame can finish within 8 ms on the specified processor. We define the latency
due to algorithmic reasons (such as overlap-add) as algorithmic latency, and
the computing time needed to process one frame as hardware latency. The
overall latency is the summation of the two and is denoted as processing
latency.

Figure 1: Illustration of processing latency in systems based on regular STFT
and iSTFT. Each rectangular band denotes a segment of time-domain signals. We
use 75% frame overlap as an example. Because of the overlap-add in the iSTFT,
to get the prediction at sample $n$ (marked in the top of the figure) at frame
$t$, one has to first observe all the samples of frame $t+3$ and then wait
until the DNN finishes processing frame $t+3$. The processing delay is hence
the window length plus the running time of processing one frame.

Although a 40 ms processing latency can meet the demand of many applications,
for hearing aids this latency is too large to deliver a good listening
experience. In the recent Clarity challenge, proposed for hearing aid design
[20], the required algorithmic latency was 5 ms. Such a low-latency constraint
requires new designs and significant modifications to existing enhancement
algorithms. To meet this constraint, our study aims at an enhancement system
with a window that looks ahead at most 4 ms of samples, and a 2 ms hop size.
We assume that the hardware latency can be within 2 ms, leading to a maximum
processing delay of 6 ms, even though this computational capability might not
be available right now for a computationally demanding DNN model on resource-
constrained edge devices. We emphasize that this study focuses on improving
enhancement performance and reducing algorithmic latency rather than
computational efficiency or feasibility on current hardware.

In the literature, some STFT-domain beamforming studies [21, 22] use small
window and hop sizes to achieve enhancement with a low algorithmic latency,
and a low frequency resolution is used for STFT following the short window
length. However, this low frequency resolution may limit the enhancement
performance [1, 23, 24]. In the recent Clarity challenge, almost all the top
teams [25, 26, 27] adopt time-domain networks such as Conv-TasNet [7, 28],
which can use very short window and hop sizes to potentially realize very low-
latency enhancement. Conv-TasNet leverages DNN based end-to-end optimization
to learn a set of bases for a small window of samples respectively for its
encoder and decoder to replace the conventional STFT and iSTFT operations. The
number of bases is set to be much larger than the number of samples in the
window. Enhancement is then performed in the higher-dimensional encoded space
and the decoder is used for overlap-add based signal re-synthesis. While
achieving good separation performance in monaural anechoic speaker separation
tasks, Conv-TasNet performs less impressively in reverberant conditions and in
multi-microphone scenarios than frequency-domain approaches [29, 11, 30]. In
addition, the basis learned by Conv-TasNet is not narrowband [7]. It is not
straightforward how to combine Conv-TasNet with conventional STFT-domain
enhancement algorithms to achieve further gains, without incurring additional
algorithmic latency. Such conventional algorithms include beamforming and
weighted prediction error (WPE), which rely on the narrow-band assumption and
can produce reliable separation through their per-frequency processing [31,
32, 11, 33]. One way of combining them [34, 23, 35] is by iterating Conv-
TasNet, which uses a very short window, with STFT-domain beamforming, which
uses a regular, longer window. To use Conv-TasNet outputs to compute signal
statistics for STFT-domain beamforming, one has to first re-synthesize time-
domain signals before extracting STFT spectra for beamforming. Similarly, to
apply Conv-TasNet on beamforming results for post-filtering, one has to apply
iSTFT to get time-domain signals before feeding them to Conv-TasNet. Such an
iterative procedure would however gradually build up the algorithmic latency,
because the overlap-add algorithms are used multiple times in Conv-TasNet and
iSTFT.

Figure 2:  Illustration of overlap-add with dual window sizes. This example
uses a 16 ms input window size for STFT, a 4 ms output window size for
overlap-add, and a 2 ms hop size.

It is commonly perceived that regular STFT based systems, which suffer from a
large algorithmic latency equal to the STFT’s typically long window length,
are not ideal for very low-latency speech enhancement, and time-domain models
such as Conv-TasNet, which can achieve strong performance using very short
windows, appear a more appropriate choice [7]. In this study, we show that our
STFT based system can also produce a comparable or better enhancement
performance at an algorithmic latency as low as 4 or 2 ms. This is partially
achieved by combining STFT-domain, deep learning based speech enhancement with
a conventional dual window approach [36, 37], which uses a regularly long
window length for STFT and a shorter window length for overlap-add. This
approach is illustrated in Fig. 2. More specifically, assuming a hop size (HS)
of 2 ms, we use a 16 ms input window size (iWS) for STFT, and an output window
size (oWS) of 4 ms for the overlap-add in iSTFT. The 16 ms input window looks
4 ms ahead and 12 ms in the past in this case. After obtaining the predicted
signal at each frame, i.e., after performing inverse discrete Fourier
transform (iDFT), we throw away the first 12 ms of waveforms, apply a
synthesis window, and perform overlap-add based on the last 4 ms of signals at
each frame. The DNN module is designed to be frame-online. Therefore the
entire system has an algorithmic latency of 4 ms. Later in Section V-B, we
will introduce a future-frame prediction technique to further reduce the
algorithmic latency to 2 ms.

When used with DNNs, this dual window size approach has several advantages.
First, using a long window for STFT leads to higher frequency resolution,
meaning that we could have more estimated filters (or mask values) per frame
to obtain more fine-grained enhancement. In addition, higher frequency
resolution can better leverage the speech sparsity property in the T-F domain
for enhancement [1, 24]. Second, using a longer input window can capture more
reverberation at each frame, potentially leading to better dereverberation. In
addition, it could lead to better spatial processing, as the inter-channel
phase patterns could be more stable and salient for longer signals. Third,
STFT bases are narrowband in nature, meaning that we can readily use our DNN
outputs (in this study, the estimated target real and imaginary components) to
compute a conventional frequency-domain beamformer, whose results can be used
as extra features for another DNN to better predict the target speech.

Our study makes three major contributions. First, we adapt a conventional dual
window size approach [36, 37] to reduce the algorithmic latency of STFT-domain
deep learning based speech enhancement. Second, we utilize the outputs from
the first DNN for frequency-domain frame-online beamforming, and the
beamforming result is fed to a second DNN for better enhancement. Compared
with using the outputs from time-domain models for frequency-domain
beamforming, our approach does not incur algorithmic latency because the two
DNNs and the beamformer all operate in the complex T-F domain. Third, we
propose a future-frame prediction technique that can further reduce the
algorithmic latency caused by the output window size. In our experiments, we
find that the proposed STFT-domain system can achieve comparably good or
better performance than the popular Conv-TasNet [7, 28, 25], using a similar
amount of computation and at an algorithmic latency as low as 4 or 2 ms. There
is a recent dual window size study on deep learning based speaker separation
[24]. However, it is monaural, uses real-valued mask estimation, only reduces
the algorithmic latency to 8 ms, and does not compare with time-domain models.

An overview of our system is given in the following section.

##  II System Overview

Given an utterance of a speaker recorded in noisy-reverberant conditions by a
$P$-microphone array, the physical model in the STFT domain can be formulated
as

$\displaystyle\mathbf{Y}(t,f)$ $\displaystyle=\mathbf{X}(t,f)+\mathbf{V}(t,f)$
$\displaystyle=\mathbf{S}(t,f)+\mathbf{H}(t,f)+\mathbf{V}(t,f),$ (1)

where $\mathbf{Y}(t,f)$, $\mathbf{V}(t,f)$, $\mathbf{X}(t,f)$,
$\mathbf{S}(t,f)$ and $\mathbf{H}(t,f)\in{\mathbb{C}}^{P}$ respectively denote
the STFT vectors of the mixure, reverberant noise, reverberant target speech,
direct-path and non-direct signals of the target speaker at time $t$ and
frequency $f$. In the rest of this paper, when dropping $t$ and $f$ from the
notation, we refer to the corresponding spectrogram. In our experiments, the
default iWS, oWS, and HS for STFT are respectively set to 16, 4, and 2 ms, and
the sampling rate is 16 kHz. A 256-point DFT is applied to extract
129-dimensional STFT coefficients at each frame. The analysis window and
synthesis window will be described in Section V-A.

Based on the input $\mathbf{Y}$, we aim at recovering the target speaker’s
direct-path signal captured at a reference microphone $q$, i.e., $S_{q}$. We
use the corresponding time-domain signal of $S_{q}$, denoted as $s_{q}$, as
the reference signal for metric computation. Note that early reflections are
not considered as part of target speech. In multi-microphone cases, we assume
that the same array geometry is used for training and testing, following [38,
39]. This is a valid assumption as real-world products such as smart speakers
have a fixed array configuration.

Figure 3: DNN overview. Our system contains two frame-online DNNs with a
frame-online multi-channel Wiener filter (MCWF) in between.

Our best performing system, illustrated in Fig. 3, has two DNNs. Using the
real and imaginary (RI) components of multiple input signals as input
features, the DNNs are trained sequentially based on single- or multi-
microphone complex spectral mapping [5, 38, 39] to predict the RI components
of $S_{q}$. The estimated speech by DNN1 is used to compute, at each
frequency, a multi-channel Wiener filter (MCWF) [23] for the target speaker.
$\text{DNN}_{2}$ concatenates the RI components of the beamforming results,
the outputs of $\text{DNN}_{1}$, and $\mathbf{Y}$ as features to further
estimate the RI components of $S_{q}$. The $\text{DNN}_{1}$, $\text{DNN}_{2}$,
and MCWF modules are all designed to be frame-online, so that we can readily
plug our two-DNN system into Fig. 2 to achieve enhancement with very low
algorithmic latency. Note that such two-DNN systems with a beamformer in
between have been explored in our previous studies [38, 40, 39, 11], but their
target was offline processing. This paper extends them for frame-online
processing with a very low algorithmic latency.

The rest of this paper is organized as follows. Section III details the DNN
configurations, Section IV describes the DNN-supported beamforming, and
Section V presents the proposed enhancement system with low algorithmic
latency. Experimental setup and evaluation results are presented in Sections
VI and VII. Section VIII concludes this paper.

##  III DNN Configurations

Our DNNs are trained to do complex spectral mapping [5], where the real and
imaginary (RI) components of multiple signals are concatenated as input for
DNNs to predict the target RI components at a reference microphone. This
section describes the loss functions and the DNN architectures. The key
differences from our earlier studies [38, 40, 39, 11] include the facts that
(1) we train through the dual window size approach and define the loss
function on the re-synthesized signals; and (2) we dramatically reduce the
per-frame amount of computation of the DNN models used in our earlier studies.

###  III-A Loss Functions

The two DNNs in Fig. 3 are trained using different loss functions. For DNN1,
following [41, 38, 39] the loss function is defined on the predicted RI
components and their magnitude

$\displaystyle\mathcal{L}_{\text{RI+Mag}}=\|\hat{R}_{q}^{(1)}-\text{Real}$
$\displaystyle(S_{q})\|_{1}+\|\hat{I}_{q}^{(1)}-\text{Imag}(S_{q})\|_{1}$
$\displaystyle+\Big{\|}\sqrt{\hat{R}_{q}^{(1)^{2}}+\hat{I}_{q}^{(1)^{2}}}-|S_{q}|\Big{\|}_{1},$
(2)

where $\hat{R}_{q}^{(1)}$ and $\hat{I}_{q}^{(1)}$ are the predicted RI
components by DNN1, $\text{Real}(\cdot)$ and $\text{Imag}(\cdot)$ extract RI
components, and $\|\cdot\|_{1}$ computes the $L_{1}$ norm. The estimated
target spectrogram at the reference microphone $q$ is
$\hat{S}_{q}^{(1)}=\hat{R}_{q}^{(1)}+j\hat{I}_{q}^{(1)}$, where $j$ denotes
the imaginary unit.

Given the predicted RI components $\hat{R}_{q}^{(2)}$ and $\hat{I}_{q}^{(2)}$
by DNN2, we denote $\hat{S}_{q}^{(2)}=\hat{R}_{q}^{(2)}+j\hat{I}_{q}^{(2)}$
and compute the re-synthesized signal
$\hat{s}_{q}^{(2)}=\text{iSTFT}(\hat{S}_{q}^{(2)})$, where
$\text{iSTFT}(\cdot)$ uses a shorter output window for overlap-add to reduce
the algorithmic latency (see Fig. 2). The loss function is then defined on the
re-synthesized time-domain signal and its STFT magnitude

$\displaystyle\mathcal{L}_{\text{Wav+Mag}}=$
$\displaystyle\|\hat{s}_{q}^{(2)}-s_{q}\|_{1}+\Big{\|}|\text{STFT}_{\mathcal{L}}(\hat{s}_{q}^{(2)})|-|\text{STFT}_{\mathcal{L}}(s_{q})|\Big{\|}_{1},$
(3)

where $\text{STFT}_{\mathcal{L}}(\cdot)$ extracts a complex spectrogram. The
loss on magnitude is found to consistently improve objective metrics such as
PESQ and STOI [41]. Note that $\text{STFT}_{\mathcal{L}}(\cdot)$ here can use
any window types and window and hop sizes, and can be different from the ones
we use to extract $\mathbf{Y}$ and $S_{q}$, since it is only used for loss
computation. In our experiments, we use the square-root Hann window, a 32 ms
window size and an 8 ms hop size to compute this magnitude loss. Please do not
confuse these STFT parameters with the STFT parameters we used to extract
$\mathbf{Y}$ and $S_{q}$.

In our experiments, we will compare the two-DNN system with a single-DNN
system (i.e., without the beamforming module and the second DNN). In the
single-DNN case, DNN1 can be trained using either Eq. (III-A) or (3).
Differently, in the two-DNN case, DNN1 is trained only using Eq. (III-A). This
is because the beamformer we will derive later in Eq. (4) is based on a DNN-
estimated target in the complex domain.

Figure 4:  Network architecture of $\text{DNN}_{2}$. Each one of Conv2D,
Deconv2D, Conv2D+PReLU+BN, dsConv2D+PReLU+BN, and Deconv2D+PReLU+BN blocks is
specified in the format: kernelTime$\times$kernelFreq, (strideTime,
strideFreq), (padTime, padFrequency), (dilationTime, dilationFreq),
featureMaps. During training, the tensor shape after each block in the encoder
and decoder is denoted in the format:
featureMaps$\times$timeSteps$\times$freqChannels.

###  III-B Network Architecture

Our DNN architecture, denoted as LSTM-ResUNet, is illustrated in Fig. 4. It is
a long short-term memory (LSTM) network clamped by a U-Net [42]. Residual
blocks are inserted at multiple frequency scales in the encoder and decoder of
the U-Net. The motivation of this network design is that U-Net can maintain
fine-grained local structure via its skip connections and model contextual
information along frequency through down- and up-sampling, LSTM can leverage
long-range information, and residual blocks can improve discriminability. We
stack the RI components of different input and output signals as features maps
in the network input and output. DNN1 and DNN2 differ only in their network
input. DNN1 uses the RI components of $\mathbf{Y}$ to predict the RI
components of $S_{q}$, and DNN2 additionally uses as input the RI components
of $\hat{S}_{q}^{(1)}$ and a beamforming result $\hat{S}_{q}^{\text{MCWF}}$,
which will be described later in Section IV. The encoder contains one two-
dimensional (2D) convolution followed by causal layer normalization (cauLN)
for each input signal, and six convolutional blocks, each with 2D convolution,
parametric ReLU (PReLU) non-linearity, and batch normalization (BN), for down-
sampling. The LSTM contains three layers, each with 300 units. The decoder
includes six blocks of 2D deconvolution, PReLU, and BN, and one 2D
deconvolution, for up-sampling. Each residual block in the encoder and decoder
contains five depth-wise separable 2D convolution (denoted as dsConv2D)
blocks, where the dilation rate along time are respectively 1, 2, 4, 8 and 16.
Linear activation is used in the output layer to obtain the predicted RI
components.

All the convolution and normalization layers are causal (i.e., frame-online)
at run time. We use $1\times 3$ or $1\times 4$ kernels along time and
frequency for the down- and up-sampling convolutions, following [16]. Causal
$2\times 3$ convolutions are used in the residual blocks, following [16].

Note that this architecture is similar to the earlier TCN-DenseUNet
architecture [38, 40, 39, 11]. Major changes include replacing the DenseNet
blocks with residual blocks and replacing regular 2D convolutions with
depthwise separable 2D convolutions. These changes dramatically reduce the
amount of computation and the number of trainable parameters. The network
contains around 2.3 million parameters. It uses similar amount of computation
compared with the Conv-TasNet used in our experiments.

##  IV Frequency-Domain Beamforming

Based on the DNN-estimated target RI components, we compute an online multi-
channel Wiener filter (MCWF) [33] to enhance target speech (see Fig. 3). The
MCWF is computed per frequency, leveraging the narrow-band property of STFT.
Although the beamforming result of such a filter usually does not show better
scores in terms of enhancement metrics than the immediate DNN outputs, it can
provide complementary information to help DNN2 obtain better enhancement
results [39, 11, 43]. Note that our main contribution here is to show that
frame-online frequency-domain beamforming can be easily integrated with our
STFT-domain DNNs to improve enhancement, while not incurring any algorithmic
latency. We can use more advanced beamformers, or dereverberation algorithms
such as WPE [31, 32, 11], to achieve even better enhancement than MCWF. They
are however out of the scope of this study. This section will first describe
the offline time-invariant implementation of the MCWF beamformer and then
extend it to frame-online.

The MCWF [33] computes a linear filter per T-F unit or per frequency to
project the mixture to target speech. Assuming the target speaker does not
move within each utterance and based on the DNN-estimated target speech
$\hat{S}_{q}^{(1)}$, we compute a time-invariant MCWF per frequency through
the following minimization problem

$\displaystyle\underset{\mathbf{w}(f;q)}{{\text{min}}}\sum_{t}\big{|}\hat{S}_{q}^{(1)}(t,f)-\mathbf{w}(f;q)^{{\mathsf{H}}}\mathbf{Y}(t,f)\big{|}^{2},$
(4)

where $q$ denotes the reference microphone and
$\mathbf{w}(f;q)\in{\mathbb{C}}^{P}$ is a $P$-dimensional filter. Since the
objective is quadratic, a closed-form solution is available

$\displaystyle\hat{\mathbf{w}}(f;q)$
$\displaystyle=\Big{(}\hat{\mathbf{\Phi}}^{(yy)}(f)\Big{)}^{-1}\hat{\mathbf{\Phi}}^{(ys)}(f)\mathbf{u}_{q},$
(5) $\displaystyle\hat{\mathbf{\Phi}}^{(yy)}(f)$
$\displaystyle=\sum_{t}\mathbf{Y}(t,f)\mathbf{Y}(t,f)^{{\mathsf{H}}},$ (6)
$\displaystyle\hat{\mathbf{\Phi}}^{(ys)}(f)$
$\displaystyle=\sum_{t}\mathbf{Y}(t,f)\hat{\mathbf{S}}^{(1)}(t,f)^{{\mathsf{H}}},$
(7)

where $\hat{\mathbf{\Phi}}^{(yy)}(f)$ denotes the observed mixture spatial
covariance matrix, $\hat{\mathbf{\Phi}}^{(ys)}(f)$ the estimated covariance
matrix between the mixture and the target speaker, and $\mathbf{u}_{q}$ is a
one-hot vector with element $q$ equal to one. Notice that we do not need to
first fully compute the matrix $\hat{\mathbf{\Phi}}^{(ys)}(f)$ and then take
its $q^{\text{th}}$ column by multiplying it with $\mathbf{u}_{q}$, because

$\displaystyle\hat{\mathbf{\Phi}}^{(ys)}(f)\mathbf{u}_{q}=\sum_{t}\mathbf{Y}(t,f)\Big{(}\hat{S}_{q}^{(1)}(t,f)\Big{)}^{*},$
(8)

where $(\cdot)^{*}$ computes complex conjugate. The beamforming result is
computed as

$\displaystyle\hat{S}_{q}^{\text{MCWF}}(t,f)=\hat{\mathbf{w}}(f;q)^{{\mathsf{H}}}\mathbf{Y}(t,f).$
(9)

We point out that the DNN-estimated magnitude and phase can both be used for
computing the beamformer and we only need to use our DNN to estimate the
target speech at the reference microphone $q$. Differently, to leverage DNN-
estimated magnitude and phase to compute a beamformer, earlier studies
estimate the target speech at all the microphones by training a multi-channel
input and multi-channel output network that can predict the target speech at
all the microphones at once [38, 15], or by running a well-trained multi-
channel input and single-channel output network $P$ times at inference time
[39, 30], where each microphone is considered as the reference microphone in
turn. However, the former approach produces worse separation at each
microphone than the latter, probably because there are many more signals to
predict [38], and the latter dramatically increases the amount of computation
[39].

Differently from Eqs. (5)-(7), in a frame-online setup the statistics are
accumulated online, similarly to [44], and the beamformer at each time step is
computed as

$\displaystyle\hat{\mathbf{w}}(t,f;q)$
$\displaystyle=\Big{(}\hat{\mathbf{\Phi}}^{(yy)}(t,f)\Big{)}^{-1}\hat{\mathbf{\Phi}}^{(ys)}(t,f)\mathbf{u}_{q},$
(10) $\displaystyle\hat{\mathbf{\Phi}}^{(yy)}(t,f)$
$\displaystyle=\hat{\mathbf{\Phi}}^{(yy)}(t-1,f)+\mathbf{Y}(t,f)\mathbf{Y}(t,f)^{{\mathsf{H}}},$
(11) $\displaystyle\hat{\mathbf{\Phi}}^{(ys)}(t,f)$
$\displaystyle=\hat{\mathbf{\Phi}}^{(ys)}(t-1,f)+\mathbf{Y}(t,f)\hat{\mathbf{S}}^{(1)}(t,f)^{{\mathsf{H}}},$
(12)

with $\hat{\mathbf{\Phi}}^{(yy)}(0,f)$ and $\hat{\mathbf{\Phi}}^{(ys)}(0,f)$
initialized to be all-zero. Based on the online time-varying filter
$\hat{\mathbf{w}}(t,f;q)$, the beamforming result is obtained as

$\displaystyle\hat{S}_{q}^{\text{MCWF}}(t,f)=\hat{\mathbf{w}}(t,f;q)^{{\mathsf{H}}}\mathbf{Y}(t,f).$
(13)

Similarly to [44], in a frame-online setup
$\hat{\mathbf{\Phi}}^{(yy)}(t)^{-1}$ in Eq. (10) can be computed iteratively
according to the Woodbury formula, i.e.,

$\displaystyle\hat{\mathbf{\Phi}}^{(yy)}(t)^{-1}$
$\displaystyle=\big{(}\hat{\mathbf{\Phi}}^{(yy)}(t-1)+\mathbf{Y}(t)\mathbf{Y}(t)^{{\mathsf{H}}}\big{)}^{-1}$
$\displaystyle=\hat{\mathbf{\Phi}}^{(yy)}(t-1)^{-1}$
$\displaystyle-\frac{\hat{\mathbf{\Phi}}^{(yy)}(t-1)^{-1}\mathbf{Y}(t)\mathbf{Y}(t)^{{\mathsf{H}}}\hat{\mathbf{\Phi}}^{(yy)}(t-1)^{-1}}{1+\mathbf{Y}(t)^{{\mathsf{H}}}\hat{\mathbf{\Phi}}^{(yy)}(t-1)^{-1}\mathbf{Y}(t)},$
(14)

where the frequency index $f$ is dropped to make the equation less cluttered.
This way, expensive matrix inversion at each T-F unit is avoided in the frame-
online case.

##  V Enhancement with Low Algorithmic Latency

In Fig. 3, there are two DNNs and an MCWF in between. Since the DNNs and
beamformer all operate in the complex T-F domain, without going back and forth
to the time domain, we can use the same STFT resolution for all of them to
obtain a two-DNN system with a low algorithmic latency. This is different from
earlier studies [34, 23, 35] that combine time-domain models with beamforming
and have to switch back and forth to the time domain. Given a small hop size
(say 2 ms), we can use a regular, large iWS (for example 16 ms) for STFT to
have a reasonably high frequency resolution for frequency-domain beamforming.
To re-synthesize $\hat{S}_{q}^{(2)}$ to a time-domain signal, we use the last
4 ms of the 16 ms signals produced by iDFT at each frame for overlap-add,
following the procedure illustrated in Fig. 2. The resulting system has an
algorithmic latency of 4 ms, even though the STFT spectrograms are extracted
using a window size of 16 ms. The rest of this section describes the analysis
and synthesis windows, and a future-frame prediction technique that can
further reduce the 4 ms algorithmic latency to 2 ms.

###  V-A Analysis and Synthesis Window Design

In our experiments, we will investigate various analysis windows such as the
square-root Hann (sqrtHann) window, rectangular (Rect) window, asymmetric
sqrtHann (AsqrtHann) window [36, 37, 24], and Tukey window [45]. See Fig. 5
for an illustration of the windows. Our consideration for this investigation
is that for windows such as the sqrtHann window, where all the last 4 ms are
in the tapering range, the tapering could make the extracted frequency
components in the STFT spectrograms less representative of the last 4 ms of
signals, where we aim to make predictions. One solution is to use a
rectangular analysis window, which does not taper samples. However, it is
well-known that rectangular windows lead to more spectral leakage due to their
higher sidelobes than windows with a tapering shape [45]. Such leakage could
degrade per-frequency beamforming as well as the performance of DNNs. On the
other hand, the rectangular window does not taper any samples in the right end
and hence the signal in that region is not modified by the window. This could
be helpful for very low-latency processing, as we need to make predictions for
these samples. Our study also considers the Tukey window [45], defined as

$\displaystyle g[n]$ $\displaystyle=0.5-0.5\,\text{cos}\Big{(}\frac{\pi
n}{\alpha N}\Big{)},$ $\displaystyle\text{if\,\,\,}0\leq n\leq\alpha N;$
$\displaystyle=g[N-n],$ $\displaystyle\text{if\,\,\,}N-\alpha N\leq n<N;$
$\displaystyle=1,$ $\displaystyle\text{otherwise},$ (15)

where $0\leq n<N$ and tapering only happens at the first and the last $\alpha
N$ samples. Given a 16 ms analysis window, we set $\alpha$ to $\frac{1}{16}$,
meaning 1 ms of tapering on both ends. We also consider the AsqrtHann window
proposed in [36, 37, 24], and construct a 16 ms long AsqrtHann window by
combining the first half of a 30 ms sqrtHann window and the second half of a 2
ms sqrtHann window.

Figure 5:  Illustration of analysis windows (assuming a window size of 16 ms
and a sampling rate of 16 kHz).

We compute a synthesis window that can achieve perfect reconstruction when
used with an analysis window $g$, following [46]. Suppose that the run-time
oWS in samples is $A$ and the hop size in samples is $B$, and that $A$ is a
multiple of $B$, we obtain the synthesis window $l\in{\mathbb{R}}^{A}$ based
on the last $A$ samples of the analysis window

$\displaystyle l[n]$
$\displaystyle=\frac{g[N-A+n]}{\sum_{k=0}^{A/B-1}{g[N-A+(n\text{\,\,mod\,\,}B)+kB]}^{2}},$
(16)

where $0\leq n<A$.

###  V-B Future-Frame Prediction

We can further reduce the algorithmic latency by training the DNN to predict,
say, one frame ahead. That is, at time $t$, the DNN predicts the target RI
components at frame $t+1$. This can reduce the algorithmic latency from 4 to 2
ms. We use Fig. 2 to explain the idea. To get the prediction at sample $n$
(see the top of Fig. 2), we need to overlap-add the last 4 ms of frame $t$ and
$t+1$. If the DNN only predicts the current frame, we can only do the overlap-
add after we fully observe frame $t+1$ and finish feed-forwarding frame $t+1$.
In contrast, if the DNN predicts frame $t+1$ at frame $t$, we can do the
overlap-add after we observe and finish feed-forwarding frame $t$. The
algorithmic latency is hence reduced by 2 ms. We can predict more frames ahead
to reduce the algorithmic latency to 0 ms or negative, but this comes with a
degradation in performance as predicting the future is often a difficult task.

In our experiments, we find that predicting one frame ahead does not
dramatically degrade the performance. This is possibly because when predicting
one frame ahead and using a loss function like Eq. (3) which requires training
through iSTFT, at frame $t$ we essentially use the input signals up to frame
$t$ to predict the sub-frame (marked in the top of Fig. 2) at frame $t$ so
that a 2 ms algorithmic latency (i.e., the length of the sub-frame) can be
achieved111In other words, our DNN model in this case can fully observe the
mixture signals of the sub-frame at frame $t$, and it has the opportunity to
correct at frame $t$ the errors made when predicting at frame $t-1$ the (then
in the future) sub-frame at frame $t$. This could be the key reason why
predicting one frame ahead performs reasonably well in our experiments.. We
find that it is then important to use the rectangular window as the analysis
window and that using tapering-shaped windows produces noticeable artifacts
near the boundary of each frame. If we use an analysis window which
significantly tapers the right end of the input signal, such as the Tukey or
sqrtHann window, the DNN model would have difficulty predicting the signals at
the right boundary of the sub-frame at frame $t$, because the input
information especially near the right boundary would be lost due to the
tapering.

Performance degradation is however significant when predicting two frames
ahead (i.e., 4 ms in the future), and even more so when predicting three,
possibly because the model now has to fully predict part of the future signal.
If performance could be improved to the point that three (the value of
$\text{oWS}/\text{HS}+1$ in our setup) frames ahead can be accurately
predicted, for example via a more powerful DNN architecture, the algorithmic
latency could be reduced to $-2$ ms. In this case, the enhancement system
could potentially achieve $0$ ms processing latency if the hardware latency of
processing each frame can be less than the $2$ ms hop size (which would be
necessary to maintain real-time processing anyway).

In our two-DNN system in Fig. 3, only DNN2 can choose to predict future frames
and DNN1 always predicts the current frame.

##  VI Experimental Setup

We evaluate the proposed algorithm on a noisy-reverberant speech enhancement
task, using a single microphone or an array of microphones. This section
describes the dataset, benchmark systems, and miscellaneous configurations.

###  VI-A Dataset for Noisy-Reverberant Speech Enhancement

Due to the lack of a widely adopted benchmark for multi-channel noisy-
reverberant speech enhancement, we build a custom dataset based on the WSJCAM0
[47] and FSD50k [48] corpora. The clean signals in WSJCAM0 are used as the
speech sources. The corpus contains 7,861, 742, and 1,088 utterances
respectively in its training, validation, and test sets. Using the same split
of clean signals as in WSJCAM0, we simulate 39,245 ($\sim$77.7 hours), 2,965
($\sim$5.6 hours), and 3,260 ($\sim$8.5 hours) noisy-reverberant mixtures as
our training, validation, and test sets, respectively. The noise sources are
from the FSD50k dataset, which contains around 50,000 Freesound clips with
human-labeled sound events distributed in 200 classes drawn from the AudioSet
ontology [49]. We sample the clips in the development set of FSD50k to
simulate the noises for training and validation, and those in the evaluation
set to simulate the noises for testing. Since our task is single-speaker
speech enhancement, following [50] we filter out clips containing any sounds
produced by humans, based on the provided sound event annotation of each clip.
Such clips have annotations such as Human_voice, Male_speech_and_man_speaking,
Chuckle_and_chortle, Yell, etc222See
https://github.com/etzinis/fedenhance/blob/master/fedenhance/dataset
_maker/make_librifsd50k.py for the full list.. To generate multi-microphone
noise signals, for each mixture we randomly sample up to seven noise clips. We
treat each sampled clip as a point source, convolve each source with the
corresponding RIR, and summate the convolved signals to create the mixture.
The directions of each noise source and the target speaker to the array are
independently sampled from the range $[0,2\pi]$, so the speaker and some
noises could have a similar azimuth. Following the setup in the FUSS dataset
[51], which is designed for universal sound separation, we consider noise
clips as background noises if they are more than 10 seconds long, and as
foreground noises otherwise. Each simulated mixed noise file has one
background noise and the rest are foreground noises. The energy level between
the dry background noise and each dry foreground noise is drawn from the range
$[-3,9]$ dB. Considering that some FSD50k clips contain silence or digital
zeros, the energy level is computed by first removing silent segments in each
clip, next computing a sample variance from the remaining samples, and then
scaling the clips to an energy level based on the sample variances. After
summing up all the spatialized noises, we scale the summated reverberant noise
such that the SNR between the target direct-path speech and the summated
reverberant noise is equal to a value sampled from the range $[-8,3]$ dB.
Besides the FSD50k clips, in each mixture we always include a weak, diffuse,
stationary air-conditioning noise drawn from the REVERB corpus, where the SNR
between the target direct-path speech and the noise is equal to a value
sampled from the range $[10,30]$ dB.

The distance between each source and the array center is drawn from the range
$[0.75,2.5]$ m. The reverberation time (T60) is drawn from the range
$[0.2,1.0]$ s. The simulated array is a six-microphone uniform circular array
with a 20 cm diameter. The sampling rate is 16 kHz.

TABLE I: #params (M), FLOPs (G), SI-SDR (dB), PESQ, and eSTOI (%) Results for Monaural Enhancement. |  |  DNN1 | Window |  |  |  |  | Last DNN predicts | Algorithmic |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Entry | Systems | Loss | type | #DFT | iWS | oWS | HS | #frames ahead | latency (ms) | #params | FLOPs | SI-SDR | PESQ | eSTOI
0 | Unprocessed | - | - | - | - | - | - | - | - | - | - | $-6.2$ | $1.44$ | $41.1$
1a |  DNN1 | RI+Mag | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $2.32$ | $27.756\,68$ | $2.756\,05$ | $1.849\,317$ | $66.9980$
1b |  DNN1 | Wav+Mag | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $2.32$ | $27.756\,68$ | $2.889\,300$ | $1.910\,38$ | $68.377$
1c |  DNN1 | Wav+Mag | Tukey | 256 | 16 | 4 | 2 | 1 | $2$ | $2.32$ | $27.756\,68$ | $2.185$ | $1.792\,04$ | $64.8963$
2a |  DNN1 | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 0 | $4$ | $2.32$ | $27.756\,68$ | $2.8341$ | $1.8997$ | $68.235$
2b |  DNN1 | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 1 | $2$ | $2.32$ | $27.756\,68$ | $2.5297$ | $1.845\,75$ | $66.637$
2c |  DNN1 | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 2 | $0$ | $2.32$ | $27.756\,68$ | $-3.6493$ | $1.7138$ | $62.202$
2d |  DNN1 | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 3 | $-2$ | $2.32$ | $27.756\,68$ | $-5.457\,38$ | $1.626\,974$ | $58.7145$
3a |  Conv-TasNet [7] | Wav | - | - | 4 | 4 | 2 | 0 | $4$ | $6.18$ | $29.350\,856$ | $2.2581$ | $1.5752$ | $61.7139$
3b |  Conv-TasNet [7] | Wav+Mag | - | - | 4 | 4 | 2 | 0 | $4$ | $6.18$ | $29.350\,856$ | $2.1584$ | $1.7841$ | $65.729$
3c |  Conv-TasNet [7] | Wav+Mag | - | - | 4 | 4 | 1 | 0 | $4$ | $6.18$ | $54.4938$ | $2.418$ | $1.8326$ | $66.68$
3d |  Conv-TasNet [7] | Wav+Mag | - | - | 2 | 2 | 1 | 0 | $2$ | $6.14$ | $52.207\,439$ | $2.181$ | $1.7743$ | $65.078$
4a | Oracle Monaural SMM | - | sqrtHann | 512 | 32 | 32 | 8 | - | - | - | - | $1.0$ | $3.31$ | $90.3$
4b | Oracle Monaural PSM | - | sqrtHann | 512 | 32 | 32 | 8 | - | - | - | - | $5.9$ | $3.55$ | $90.3$
TABLE II: SI-SDR (dB), PESQ, and eSTOI (%) Results Using Various Analysis Windows for Six-Microphone Enhancement. |  |  DNN1 | Window |  |  |  |  | Last DNN predicts | Algorithmic |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Entry | Systems | Loss | type | #DFT | iWS | oWS | HS | #frames ahead | latency (ms) | #params | FLOPs | SI-SDR | PESQ | eSTOI
0 | Unprocessed | - | - | - | - | - | - | - | - | - | - | $-6.2$ | $1.44$ | $41.1$
1a |  DNN1 | RI+Mag | sqrtHann | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $5.8787$ | $2.166\,40$ | $75.67$
1b |  DNN1 | RI+Mag | AsqrtHann | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $5.965\,21$ | $2.130\,58$ | $75.9366$
1c |  DNN1 | RI+Mag | Rect | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $5.991\,293$ | $2.131\,858$ | $75.148$
1d |  DNN1 | RI+Mag | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $6.019\,40$ | $2.162\,64$ | $76.0204$
2a |  DNN1 | Wav+Mag | sqrtHann | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $6.403\,26$ | $2.252\,724$ | $77.2638$
2b |  DNN1 | Wav+Mag | AsqrtHann | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $6.338\,66$ | $2.252\,33$ | $77.1877$
2c |  DNN1 | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $6.197\,91$ | $2.230\,000\,8$ | $76.7019$
2d |  DNN1 | Wav+Mag | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $6.376\,042$ | $2.263\,25$ | $77.255\,47$

###  VI-B Benchmark Systems

We consider the frame-online Conv-TasNet [7] as the monaural benchmark system.
Conv-TasNet is an excellent model. It can achieve enhancement with very low
algorithmic latency through its very short window length, using a very small
amount of computation. We considered other monaural time-domain approaches
such as [13], which uses window sizes as large as typical STFT window sizes
and also leverages overlap-add for signal re-synthesis. It has the same
algorithmic latency as regular STFT based systems due to the overlap-add. The
proposed dual window size approach can be straightforwardly applied to reduce
its latency, but the model itself requires drastically more computation than
Conv-TasNet, mainly due to its DenseNet modules. Another recent study [52]
proposes to use low-overlap window for Wave-U-Net. However, a large window is
used and their algorithmic latency is at least 38.4 ms. We therefore only
consider Conv-TasNet as the monaural baseline. We also considered other frame-
online T-F domain models such as the winning solutions in the DNS challenges
[19]. However, they are targeted at teleconferencing scenarios, where a
processing latency as large as 40 ms is allowed. For example, DCCRN [14] has
an algorithmic latency of 62.5 ms and TSCN-PP [53] 20 ms. In addition, these
models share many similarities with our complex T-F domain DNN models and can
straightforwardly leverage our proposed techniques to reduce their algorithmic
latency. We therefore do not include them as baselines.

We consider the frame-online multi-channel Conv-TasNet [28, 25], denoted as
MC-Conv-TasNet, as the main multi-channel baseline. Compared with monaural
Conv-TasNet, MC-Conv-TasNet introduces a spatial encoder in addition to the
spectral encoder in the original Conv-TasNet to exploit spatial information.
The spatial embedding produced by the spatial encoder is used as extra
features for the network to better mask the spectral embedding. Following [28,
25], we set the spatial embedding dimension to 60 for two-channel processing
and to 360 for six-channel processing. Note that MC-Conv-TasNet is also the
enhancement component within the winning system [25] of the recent Clarity
challenge, a major effort in advancing very low-latency speech enhancement.

The Conv-TasNet models are trained either by

$\displaystyle\mathcal{L}_{\text{Wav}}=$
$\displaystyle\|\hat{s}_{q}-s_{q}\|_{1},$ (17)

where $\hat{s}_{q}$ denotes the predicted signal, or by Eq. (3). The magnitude
loss in Eq. (3) can significantly improve speech intelligibility and quality
metrics [41]. Using the notations of Conv-TasNet (in Table I of [7]), the
hyper-parameters are set to $N=512,B=158,S_{c}=158,H=512,P=3,X=8$, and $R=3$
for the single- and multi-channel Conv-TasNets. $B$ and $S_{c}$ here are
slightly larger than the default 128 in [7], since in multi-channel processing
there are additional spatial embeddings concatenated to the 512-dimensional
spectral embedding as the input to the TCN module of Conv-TasNet.

###  VI-C Miscellaneous Configurations

The MCWF beamforming filter is updated at each frame. We pad
$(\text{iWS}-\text{HS})$ ms of zero samples at the beginning of each mixture.
Without the padding, the algorithmic latency for the starting samples would be
higher.

For metric computation, we always use the target direct-path signal as the
reference. It is obtained by setting the T60 parameter to zero when generating
RIRs. Our main evaluation metric is the scale-invariant signal-to-distortion
ratio (SI-SDR) [54], which measures the quality of time-domain sample-level
predictions. We also report extended short-time objective intelligibility
(eSTOI) [55] and perceptual evaluation of speech quality (PESQ) scores. For
PESQ, narrow-band MOS-LQO scores based on the ITU P.862.1 standard [56] are
reported using the python-pesq toolkit333https://github.com/ludlows/python-
pesq, v0.0.2.

For the DNN models, we use the ptflops
toolkit444https://github.com/sovrasov/flops-counter.pytorch to count the
number of floating-point operations (FLOPs) to process a 4-second mixture.
When reporting the FLOPs of an STFT-based system, we summate the DNN FLOPs and
the FLOPs of beamforming, STFT, and iSTFT. Note that two FLOPs is roughly
equivalent to one multiply–accumulate operation.

##  VII Evaluation Results

Tables I, III and IV report the results of one-, six- and two-microphone
enhancement, respectively. In each table, we provide the algorithmic latency
of each model, along with the number of model parameters and FLOPs. When
comparing the results, we always take into account the algorithmic latency and
the amount of computation.

###  VII-A Comparison of Loss Functions

We observe that training through the proposed overlap-add procedure using the
Wav+Mag loss function in Eq. (3) leads to clear improvement over using the
RI+Mag loss in Eq. (III-A), which does not train through the signal re-
synthesis procedure. This can be observed from entries 1a vs. 1b in Table I,
1a-1d vs. 2a-2d in Table II, and 1a vs. 1b in Table III and IV. Using the
Wav+Mag loss in Eq. (3) rather than the Wav loss in Eq. (17) dramatically
improves Conv-TasNet’s scores on PESQ and STOI (see 3a vs. 3b in Table I, and
5a vs. 5b in III and IV). This aligns with our findings in [41], which only
deals with offline enhancement.

TABLE III: #params (M), FLOPs (G), SI-SDR (dB), PESQ, and eSTOI (%) Results on Six-Microphone Enhancement. |  |  DNN1 |  DNN2 | Window |  |  |  |  | Last DNN predicts | Algorithmic |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Entry | Systems | Loss | Loss | type | #DFT | iWS | oWS | HS | #frames ahead | latency (ms) | #params | FLOPs | SI-SDR | PESQ | eSTOI
0 | Unprocessed | - | - | - | - | - | - | - | - | - | - | - | $-6.2$ | $1.44$ | $41.1$
1a |  DNN1 | RI+Mag | - | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $6.019\,40$ | $2.162\,64$ | $76.0204$
1b |  DNN1 | Wav+Mag | - | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $6.376\,042$ | $2.263\,25$ | $77.255\,47$
1c |  DNN1 | Wav+Mag | - | Tukey | 256 | 16 | 4 | 2 | 1 | $2$ | $2.33$ | $28.316\,568$ | $5.202\,35$ | $2.096\,59$ | $73.981\,83$
2a |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $4.67$ | $57.1338$ | $7.8954$ | $2.698\,490$ | $83.9732$
2b |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Tukey | 256 | 16 | 4 | 2 | 1 | $2$ | $4.67$ | $57.1338$ | $6.837\,408$ | $2.474\,82$ | $80.7017$
3a |  DNN1 | Wav+Mag | - | Rect | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $28.316\,568$ | $6.197\,91$ | $2.230\,000\,8$ | $76.7019$
3b |  DNN1 | Wav+Mag | - | Rect | 256 | 16 | 4 | 2 | 1 | $2$ | $2.33$ | $28.316\,568$ | $5.8899$ | $2.201\,16$ | $76.1948$
3c |  DNN1 | Wav+Mag | - | Rect | 256 | 16 | 4 | 2 | 2 | $0$ | $2.33$ | $28.316\,568$ | $-2.0777$ | $1.942\,96$ | $69.9531$
4a |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 0 | $4$ | $4.67$ | $57.1338$ | $7.684\,495$ | $2.644\,126$ | $83.191$
4b |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 1 | $2$ | $4.67$ | $57.1338$ | $7.1327$ | $2.5006$ | $81.0619$
4c |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 2 | $0$ | $4.67$ | $57.1338$ | $-1.220\,62$ | $2.232\,36$ | $76.4172$
4d |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 3 | $-2$ | $4.67$ | $57.1338$ | $-2.753\,39$ | $2.1132$ | $73.3670$
5a |  MC-Conv-TasNet [25] | Wav | - | - | - | 4 | 4 | 2 | 0 | $4$ | $6.37$ | $30.141\,88$ | $5.4543$ | $1.959\,41$ | $73.238$
5b |  MC-Conv-TasNet [25] | Wav+Mag | - | - | - | 4 | 4 | 2 | 0 | $4$ | $6.37$ | $30.141\,88$ | $5.166$ | $2.2353$ | $76.352$
5c |  MC-Conv-TasNet [25] | Wav+Mag | - | - | - | 4 | 4 | 1 | 0 | $4$ | $6.37$ | $56.075\,85$ | $5.6833$ | $2.328\,69$ | $77.895$
5d |  MC-Conv-TasNet [25] | Wav+Mag | - | - | - | 2 | 2 | 1 | 0 | $2$ | $6.27$ | $53.234\,67$ | $5.6747$ | $2.293\,89$ | $77.265$
TABLE IV: #params (M), FLOPs (G), SI-SDR (dB), PESQ, and eSTOI (%) Results on Two-Microphone Enhancement. |  |  DNN1 |  DNN2 | Window |  |  |  |  | Last DNN predicts | Algorithmic |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Entry | Systems | Loss | Loss | type | #DFT | iWS | oWS | HS | #frames ahead | latency (ms) | #params | FLOPs | SI-SDR | PESQ | eSTOI
0 | Unprocessed | - | - | - | - | - | - | - | - | - | - | - | $-6.2$ | $1.44$ | $41.1$
1a |  DNN1 | RI+Mag | - | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $2.32$ | $27.95$ | $4.023\,44$ | $1.971\,91$ | $70.768$
1b |  DNN1 | Wav+Mag | - | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $2.32$ | $27.95$ | $4.153\,133$ | $2.047\,13$ | $72.1385$
1c |  DNN1 | Wav+Mag | - | Tukey | 256 | 16 | 4 | 2 | 1 | $2$ | $2.32$ | $27.95$ | $3.2806$ | $1.921\,12$ | $68.678$
2a |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Tukey | 256 | 16 | 4 | 2 | 0 | $4$ | $4.66$ | $56.1238$ | $4.750\,017$ | $2.174\,24$ | $74.467\,70$
2b |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Tukey | 256 | 16 | 4 | 2 | 1 | $2$ | $4.66$ | $56.1238$ | $4.0779$ | $2.063\,51$ | $72.1747$
3a |  DNN1 | Wav+Mag | - | Rect | 256 | 16 | 4 | 2 | 0 | $4$ | $2.33$ | $27.95$ | $4.251\,84$ | $2.0581$ | $72.587$
3b |  DNN1 | Wav+Mag | - | Rect | 256 | 16 | 4 | 2 | 1 | $2$ | $2.33$ | $27.95$ | $3.6793$ | $1.9704$ | $70.1381$
3c |  DNN1 | Wav+Mag | - | Rect | 256 | 16 | 4 | 2 | 2 | $0$ | $2.33$ | $27.95$ | $-3.187\,76$ | $1.805\,65$ | $65.501$
4a |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 0 | $4$ | $4.66$ | $56.1238$ | $4.784\,20$ | $2.185\,119$ | $74.911\,73$
4b |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 1 | $2$ | $4.66$ | $56.1238$ | $4.2521$ | $2.070\,683\,3$ | $72.412\,84$
4c |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 2 | $0$ | $4.66$ | $56.1238$ | $-2.446\,41$ | $1.879\,92$ | $67.7077$
4d |  DNN1+MCWF+DNN2 | RI+Mag | Wav+Mag | Rect | 256 | 16 | 4 | 2 | 3 | $-2$ | $4.66$ | $56.1238$ | $-4.050\,506$ | $1.818\,999$ | $65.5451$
5a |  MC-Conv-TasNet [25] | Wav | - | - | - | 4 | 4 | 2 | 0 | $4$ | $6.19$ | $29.421\,16$ | $3.6275$ | $1.729\,968$ | $66.9558$
5b |  MC-Conv-TasNet [25] | Wav+Mag | - | - | - | 4 | 4 | 2 | 0 | $4$ | $6.19$ | $29.421\,16$ | $3.561\,02$ | $1.9950$ | $71.1390$
5c |  MC-Conv-TasNet [25] | Wav+Mag | - | - | - | 4 | 4 | 1 | 0 | $4$ | $6.19$ | $54.634\,41$ | $3.821\,33$ | $2.042\,37$ | $71.968$
5d |  MC-Conv-TasNet [25] | Wav+Mag | - | - | - | 2 | 2 | 1 | 0 | $2$ | $6.15$ | $52.317\,17$ | $3.7618$ | $2.015\,09$ | $71.4074$

###  VII-B Comparison of Analysis Windows

We first look at the case where, at each frame, the DNNs are trained to make
predictions for the current frame. When using the Wav+Mag loss and training
through the re-synthesis procedure, we find that using different window
functions including sqrtHann, AsqrtHann, rectangular, and Tukey windows does
not produce notable differences in performance. This can be observed from
2a-2d in Table II. This is likely because, via the training-through procedure,
the DNNs could learn to deal with the slight differences in the synthesis
windows. Among all the considered windows, the Tukey window appears slightly
better than the others. This can also be observed from 1a-1d in Table II.

We then look at the results when using the DNNs to predict one frame ahead,
which can reduce the algorithmic latency from 4 to 2 ms. We observe that using
the Tukey window leads to more degradation (see 1b vs. 1c in Tables I, III and
IV and check the “Last DNN predicts #frames ahead” column) than the
rectangular window (see 2a vs. 2b in Table I, 3a vs. 3b in III and IV). In the
end, the Tukey window leads to worse performance than the rectangular window
(see 1c vs. 2b in Table I, 1c vs. 3b in III and IV). Predicting two frames
ahead, which can reduce the algorithmic latency to 0 ms, does not work very
well (see 2a and 2b vs. 2c in Table I, and 3a and 3b vs. 3c in III and IV),
especially in terms of SI-SDR.

###  VII-C Effectiveness of Beamforming

Comparing 1b and 2a, and 3a and 4a of Table III, we can see that with a
beamformer and a post-filtering network, DNN1+MCWF+DNN2 leads to clear
improvements especially on PESQ and eSTOI over using DNN1. Similar trend is
observed by comparing 1b and 2a, as well as 3a and 4a of Table IV.

###  VII-D Comparison with Conv-TasNet

In Tables I, III, and IV, we provide the results obtained by single- or multi-
channel Conv-TasNet [7, 25]. We experiment with 4/2, 4/1, and 2/1 ms
window/hop sizes. Their algorithmic latencies are respectively 4, 4, and 2 ms,
and the latter two approximately double the amount of computation used by the
first one due to their reduced hop size. In all tables, we found that using
4/1 ms window/hop sizes yields consistently better enhancement scores than the
other two, possibly because of its higher frame overlap.

Let us first look at Table I, the monaural results. Our models use fewer
number of parameters than Conv-TasNet (i.e., 2.32 M vs. 6.18 M). When the
algorithmic latency is restrained to 4 ms, our system 1b (and 2a) produces
better enhancement performance not only than 3b, using a similar number of
FLOPs, but also than the best Conv-TasNet model 3c, using around half of the
FLOPs. When the algorithmic latency is limited to 2 ms, the proposed 2b, which
predicts one frame ahead, shows better scores than 3d, using around half of
the FLOPs.

We now look at Table III. At 4 ms algorithmic latency, 1b and 3a show slightly
better (or comparable in some metrics) results than 5b, using a similar number
of FLOPs. The DNN1+MCWF+DNN2 model contains two DNNs and hence at least
doubles the amount of computation of DNN1. At 4 ms algorithmic latency, our
systems in 2a and 4a show clearly better enhancement scores than 5c, using a
comparable number of FLOPs. In 4b, DNN2 predicts one frame ahead and reduces
the algorithmic latency to 2 ms. The enhancement scores are clearly better
than those in 5d, which also have an algorithmic latency of 2 ms, again using
a similar number of FLOPs. Indeed, 4b uses two DNNs, each operating at a hop
size of 2 ms, and 5d only uses one DNN but the DNN operates at a hop size of 1
ms. The comparison between 4b and 5d suggests a new and promising way of
achieving speech enhancement with a very low algorithmic latency. Earlier
studies like time-domain methods [7, 9] tend to use very small window and hop
sizes to reduce the algorithmic latency and improve the performance, but this
significantly increases the amount of computation due to an increased number
of frames. Differently, we could use larger window and hop sizes (and hence
fewer frames) together with more powerful DNN models (such as the proposed
two-DNN system with a beamformer in between, which uses more computation at
each frame), and at the same time use the proposed future-frame prediction
technique to reduce the algorithmic latency.

Similar trends as in Table III can be observed in IV for two-microphone
enhancement.

These comparisons suggest that we can achieve reasonably good speech
enhancement with an algorithmic latency as low as 2 ms in the STFT domain, and
that operating in the STFT domain may rival or even outperform processing in
the time domain for speech enhancement with very low algorithmic latency.

###  VII-E Towards Zero Processing Latency

In systems 2c and 2d of Table I and 4c and 4d of Tables III and IV, we train
our DNNs to predict two or three frames ahead. This further reduces the
algorithmic latency at the cost of a degradation in performance, compared with
the case when we predict one frame ahead. The degradation is particularly
large for SI-SDR, likely because predicting the phase of future frames is
difficult. PESQ and eSTOI, which are less influenced by phase, maintain a
decent level of performance, even rivaling at 0 ms algorithmic latency in the
six-microphone case with a single-DNN system or an MC-Conv-TasNet system with
4 ms algorithmic latency (see 4c vs. 1b and 5c in Table III). One notable
advantage of predicting three frames ahead is that the enhancement system
could potentially have a zero processing latency, if the hardware is powerful
enough such that the hardware latency can be less than the 2 ms hop size.

##  VIII Conclusion

We have adapted a dual window size approach for deep learning based speech
enhancement with very low algorithmic latency in the STFT domain. Our approach
can easily integrate complex T-F domain DNNs with frequency-domain beamforming
to achieve better enhancement, without introducing additional algorithmic
latency. A future-frame prediction technique is proposed to further reduce the
algorithmic latency. Evaluation results on a simulated speech enhancement task
in noisy-reverberant conditions demonstrate the effectiveness of our
algorithms, and show that our STFT-based system can work well at an
algorithmic latency as low as 2 ms. The proposed algorithms can be
straightforwardly utilized by, or modified for, many T-F domain or time-domain
speech enhancement and source separation systems to reduce their algorithmic
latency.

The major limitation of our current study comes from the assumption that each
frame can be processed within the hop time by hardware in an online streaming
setup. This assumption may not be realistic for edge devices, such as
standalone hearing aids with limited computing capability, or even for modern
GPUs, unless there is careful design that can enable the system to achieve
frame-by-frame online processing, especially for heavy-duty DNN models. An
ideal speech enhancement system would have a small number of trainable
parameters and require a small amount of run-time memory and computation, at
the same time achieving high enhancement performance with very low processing
latency. A practical system will likely have to strike a trade-off among these
goals, and requires good engineering skills. Our current study focuses on
improving enhancement performance and achieving very low algorithmic latency.
Moving forward, we will consider (a) reducing the DNN complexity and using
light-weight DNN blocks [57]; (b) pruning network connections [16] or
quantizing DNN weights [58]; (c) reducing frequency resolution by using a
shorter analysis window; and (d) performing less frequent updates of the
beamforming filters.

## References

  * [1] D. Wang and J. Chen, “Supervised Speech Separation Based on Deep Learning: An Overview,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 26, no. 10, pp. 1702–1726, 2018.
  * [2] Y. Wang and D. Wang, “Towards Scaling Up Classification-Based Speech Separation,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 21, no. 7, pp. 1381–1390, 2013.
  * [3] Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, “A Regression Approach to Speech Enhancement Based on Deep Neural Networks,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 23, no. 1, pp. 7–19, 2015.
  * [4] J. Le Roux, G. Wichern, S. Watanabe, A. Sarroff, and J. R. Hershey, “Phasebook and Friends: Leveraging Discrete Representations for Source Separation,” _IEEE Journal on Selected Topics in Signal Processing_ , vol. 13, no. 2, pp. 370–382, 2019.
  * [5] D. S. Williamson, Y. Wang, and D. Wang, “Complex Ratio Masking for Monaural Speech Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , pp. 483–492, 2016.
  * [6] S. Pascual, A. Bonafonte, and J. Serr, “SEGAN : Speech Enhancement Generative Adversarial Network,” in _Proc. Interspeech_ , 2017, pp. 3642–3646.
  * [7] Y. Luo and N. Mesgarani, “Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 27, no. 8, pp. 1256–1266, 2019.
  * [8] K. Tan and D. Wang, “Learning Complex Spectral Mapping With Gated Convolutional Recurrent Networks for Monaural Speech Enhancement,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 28, pp. 380–390, 2020\.
  * [9] Y. Luo, Z. Chen, and T. Yoshioka, “Dual-Path RNN: Efficient Long Sequence Modeling for Time-Domain Single-Channel Speech Separation,” in _Proc. ICASSP_ , 2020, pp. 46–50.
  * [10] C. Subakan, M. Ravanelli, S. Cornell _et al._ , “Attention Is All You Need In Speech Separation,” in _Proc. ICASSP_ , 2021, pp. 21–25.
  * [11] Z.-Q. Wang, G. Wichern, and J. Le Roux, “Convolutive Prediction for Monaural Speech Dereverberation and Noisy-Reverberant Speaker Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 29, pp. 3476–3490, 2021.
  * [12] K. Wilson, M. Chinen _et al._ , “Exploring Tradeoffs in Models for Low-Latency Speech Enhancement,” in _Proc. IWAENC_ , 2018, pp. 366–370.
  * [13] A. Pandey and D. Wang, “Densely Connected Neural Network with Dilated Convolutions for Real-Time Speech Enhancement in The Time Domain,” in _Proc. ICASSP_ , 2020, pp. 6629–6633.
  * [14] Y. Hu, Y. Liu, S. Lv _et al._ , “DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement,” in _Proc. Interspeech_ , 2020, pp. 2472–2476.
  * [15] C. Han _et al._ , “Real-Time Binaural Speech Separation with Preserved Spatial Cues,” in _Proc. ICASSP_ , 2020, pp. 6404–6408.
  * [16] K. Tan, X. Zhang, and D. Wang, “Deep Learning Based Real-Time Speech Enhancement for Dual-Microphone Mobile Phones,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 29, pp. 1853–1863, 2021.
  * [17] A. van den Oord, S. Dieleman, H. Zen _et al._ , “WaveNet: A Generative Model for Raw Audio,” in _arXiv preprint arXiv:1609.03499_ , 2016.
  * [18] D. Rethage, J. Pons _et al._ , “A WaveNet for Speech Denoising,” in _Proc. ICASSP_ , 2018, pp. 5069–5073.
  * [19] C. K. A. Reddy, V. Gopal, R. Cutler _et al._ , “The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Speech Quality and Testing Framework,” in _Proc. Interspeech_ , 2020, pp. 2492–2496.
  * [20] “Clarity Challenge: Machine learning Challenges for Hearing Devices.” [Online]. Available: http://claritychallenge.org/
  * [21] Y. A. Huang and J. Benesty, “A Multi-Frame Approach to The Frequency-Domain Single-Channel Noise Reduction Problem,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 20, no. 4, pp. 1256–1269, 2012.
  * [22] H. Gode, M. Tammen, and S. Doclo, “Combining Binaural LCMP Beamforming and Deep Multi-Frame Filtering for Joint Dereverberation and Interferer Reduction in The Clarity-2021 Challenge,” in _Proc. Clarity_ , 2021, pp. 2–3.
  * [23] Z.-Q. Wang, H. Erdogan, S. Wisdom _et al._ , “Sequential Multi-Frame Neural Beamforming for Speech Separation and Enhancement,” in _Proc. SLT_ , 2021, pp. 905–911.
  * [24] S. Wang, G. Naithani, A. Politis, and T. Virtanen, “Deep Neural Network Based Low-latency Speech Separation with Asymmetric analysis-Synthesis Window Pair,” in _Proc. EUSIPCO_ , 2021.
  * [25] Z. Tu, J. Zhang _et al._ , “A Two-Stage End-to-End System for Speech-in-Noise Hearing Aid Processing,” in _Proc. Clarity_ , 2021, pp. 3–5.
  * [26] K. Zmolikova and J. H. Cernock, “BUT System for the First Clarity Enhancement Challenge,” in _Proc. Clarity_ , 2021, pp. 1–3.
  * [27] T. Gajecki and W. Nogueira, “Binaural Speech Enhancement Based on Deep Attention Layers,” in _Proc. Clarity_ , 2021, pp. 6–8.
  * [28] J. Zhang, C. Zorila, R. Doddipatla, and J. Barker, “On End-to-End Multi-Channel Time Domain Speech Separation in Reverberant Environments,” in _Proc. ICASSP_ , 2020, pp. 6389–6393.
  * [29] J. Heitkaemper, D. Jakobeit _et al._ , “Demystifying TasNet: A Dissecting Approach,” in _Proc. ICASSP_ , 2020, pp. 6359–6363.
  * [30] W. Zhang, J. Shi, C. Li, S. Watanabe, and Y. Qian, “Closing The Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions,” in _Proc. WASPAA_ , 2021, pp. 146–150.
  * [31] T. Yoshioka, A. Sehr, M. Delcroix _et al._ , “Making Machines Understand Us in Reverberant Rooms: Robustness against Reverberation for Automatic Speech Recognition,” _IEEE Signal Processing Magazine_ , vol. 29, no. 6, pp. 114–126, 2012.
  * [32] R. Haeb-Umbach, J. Heymann, L. Drude _et al._ , “Far-Field Automatic Speech Recognition,” _Proc. IEEE_ , 2020.
  * [33] S. Gannot, E. Vincent, S. Markovich-Golan, and A. Ozerov, “A Consolidated Perspective on Multi-Microphone Speech Enhancement and Source Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 25, pp. 692–730, 2017\.
  * [34] T. Ochiai, M. Delcroix, R. Ikeshita _et al._ , “Beam-TasNet: Time-Domain Audio Separation Network Meets Frequency-Domain Beamformer,” in _Proc. ICASSP_ , 2020, pp. 6384–6388.
  * [35] H. Chen and P. Zhang, “Beam-Guided TasNet: An Iterative Speech Separation Framework with Multi-Channel Output,” in _arXiv preprint arXiv:2102.02998v3_ , 2021.
  * [36] D. Mauler and R. Martin, “A Low Delay, Variable Resolution, Perfect Reconstruction Spectral Analysis-Synthesis System for Speech Enhancement,” in _Proc. EUSIPCO_ , 2007, pp. 222–226.
  * [37] S. U. Wood and J. Rouat, “Unsupervised Low Latency Speech Enhancement With RT-GCC-NMF,” _IEEE Journal on Selected Topics in Signal Processing_ , vol. 13, no. 2, pp. 332–346, 2019.
  * [38] Z.-Q. Wang and D. Wang, “Multi-Microphone Complex Spectral Mapping for Speech Dereverberation,” in _Proc. ICASSP_ , 2020, pp. 486–490.
  * [39] Z.-Q. Wang, P. Wang, and D. Wang, “Multi-Microphone Complex Spectral Mapping for Utterance-Wise and Continuous Speech Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 29, pp. 2001–2014, 2021.
  * [40] ——, “Complex Spectral Mapping for Single- and Multi-Channel Speech Enhancement and Robust ASR,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 28, pp. 1778–1787, 2020.
  * [41] Z.-Q. Wang, G. Wichern, and J. Le Roux, “On The Compensation Between Magnitude and Phase in Speech Separation,” _IEEE Signal Process. Lett._ , vol. 28, pp. 2018–2022, 2021.
  * [42] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in _Proc. MICCAI_ , 2015.
  * [43] Z.-Q. Wang, G. Wichern, and J. Le Roux, “Leveraging Low-Distortion Target Estimates for Improved Speech Enhancement,” _arXiv preprint arXiv:2110.00570_ , 2021.
  * [44] T. Higuchi, K. Kinoshita, N. Ito, S. Karita, and T. Nakatani, “Frame-by-Frame Closed-Form Update for Mask-Based Adaptive MVDR Beamforming,” in _Proc. ICASSP_ , 2018, pp. 531–535.
  * [45] “Window Function.” [Online]. Available: https://en.wikipedia.org/wiki/Window_function
  * [46] D. W. Griffin and J. S. Lim, “Signal Estimation from Modified Short-Time Fourier Transform,” _IEEE Trans. Acoust., Speech, Signal Process._ , vol. 32, no. 2, pp. 236–243, 1984.
  * [47] T. Robinson, J. Fransen, D. Pye, J. Foote, and S. Renals, “WSJCAM0: A British English Speech Corpus for Large Vocabulary Continuous Speech Recognition,” in _Proc. ICASSP_ , vol. 1, 1995, pp. 81–84.
  * [48] E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: An Open Dataset of Human-Labeled Sound Events,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , 2021.
  * [49] J. F. Gemmeke, D. P.W. Ellis, D. Freedman _et al._ , “Audio Set: An Ontology and Human-Labeled Dataset for Audio Events,” in _Proc. ICASSP_ , 2017, pp. 776–780.
  * [50] E. Tzinis, J. Casebeer, Z. Wang, and P. Smaragdis, “Separate But Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data,” in _Proc. WASPAA_ , 2021.
  * [51] S. Wisdom, H. Erdogan, D. P. W. Ellis _et al._ , “What’s All The Fuss About Free Universal Sound Separation Data?” in _Proc. ICASSP_ , 2021, pp. 186–190.
  * [52] S. Nakaoka, L. Li, S. Makino, and T. Yamada, “Reducing Algorithmic Delay using Low-Overlap Window for Online Wave-U-Net,” in _Proc. APSIPA_ , 2021, pp. 1210–1214.
  * [53] A. Li, W. Liu, X. Luo _et al._ , “ICASSP 2021 Deep Noise Suppression Challenge: Decoupling Magnitude and Phase Optimization with a Two-Stage Deep Network,” in _Proc. ICASSP_ , 2021, pp. 6628–6632.
  * [54] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR - Half-Baked or Well Done?” in _Proc. ICASSP_ , 2019, pp. 626–630.
  * [55] C. H. Taal, R. C. Hendriks _et al._ , “An Algorithm for Intelligibility Prediction of Time-Frequency Weighted Noisy Speech,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 19, no. 7, pp. 2125–2136, 2011.
  * [56] “P.862.1 : Mapping function for transforming P.862 raw result scores to MOS-LQO,” 2003. [Online]. Available: https://www.itu.int/rec/T-REC-P.862.1-200311-I/en
  * [57] Y. Luo, C. Han, and N. Mesgarani, “Group Communication with Context Codec for Lightweight Source Separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 29, pp. 1752–1761, 2021.
  * [58] M. Kim and P. Smaragdis, “Bitwise Neural Networks for Efficient Single-Channel Source Separation,” in _Proc. ICASSP_ , 2018, pp. 701–705.