fixmeTotalCounter

# Experiments with Encoding Structured Data for Neural Networks

Sujay Nagesh Koujalgi<EMAIL_ADDRESS>Penn State UniversityWestgate
BuildingUniversity ParkPAUSA16802 and  Jonathan Dodge<EMAIL_ADDRESS>Penn
State UniversityWestgate BuildingUniversity ParkPAUSA16802

(2023)

###### Abstract.

The project’s aim is to create an AI agent capable of selecting good actions
in a game-playing domain called Battlespace. Sequential domains like
Battlespace are important testbeds for planning problems, as such, the
Department of Defense uses such domains for wargaming exercises. The agents we
developed combine Monte Carlo Tree Search (MCTS) and Deep Q-Network (DQN)
techniques in an effort to navigate the game environment, avoid obstacles,
interact with adversaries, and capture the flag. This paper will focus on the
encoding techniques we explored to present complex structured data stored in a
Python class, a necessary precursor to an agent.

Explainable AI, State Encoding, MCTS, Reinforcement Learning, Sequential
Domain, Wargame

††journalyear: 2023††copyright: none††conference: Scholarly Paper for M.S.;
Submitted August 7; Penn State University††booktitle: Scholarly Paper for
M.S., submitted August 7, 2023 to Penn State University††price: 15.00††doi:
10.1145/3490099.3511115††isbn: 978-1-4503-9144-3/22/03

##  1\. Introduction

Decisionmaking in sequential domains entails considering possible future
states that may arrive as a consequence of decisions in the present. This is
the essence of a planning problem, and one instance of planning problems of
particular import is wargaming, which is a simulated military exercise to test
strategies and operational plans in a controlled environment. Within the
context of planning problems, it is easy to envision a variety of applications
for AI, ranging from decision support systems (DSS), intelligent opponents,
scenario generation, and so on. This work will focus primarily on agents
usable for the first two applications: DSS and intelligent opponents. As a
result, we seek not only agents that can select a good quality action, but
also agents that can analyze multi-player interactions, adapt rapidly to
changing conditions, and provide interpretable insights.

The U.S. Department of Defense introduced Battlespace (Hare et al., 2021) as a
platform for wargaming (Hung et al., 2022). It includes key features like
partial observability (Russell and Norvig, 2016), multi-player modes (both
cooperative and competitive modes), multi-level terrains, playable units with
different properties, malleability to test AI agents, and so on. Our goal is
to train an agent capable enough to be the engine behind a DSS intended to
work primarily _in-situ_ , with a goal of making better decisions while
wargaming, and secondarily during _retrospective analysis_ , with a goal of
making better decisions in the next wargaming session.

The Battlespace domain has several key challenges, due to its complex
structure, which we will detail in Section 2. Among the important challenges
to handle are the following.

  * •

Action Space Sparsity: Battlespace has a literal “do nothing,” but also has
many other moves that do not impact the game state much (spinning around).

  * •

State Space Sparsity: Battlespace uses a grid structure and most of the cells
are unoccupied.

  * •

Large Input Space: Battlespace’s structured data has a lot of properties,
which leads to a proliferation of layers required to encode the state class
and also leads to computational bottlenecks.

  * •

Multi-Agent Collaboration Strategies: Battlespace includes multiple players
who each control multiple units, learning strategies in this landscape is
difficult, which worsens the computational bottlenecks

This project involves experiments with encoding the structured data from the
Battlespace domain, training the AI agent on a simulated game environment. Our
goal in training was to enable the agent to learn how to navigate the
environment, avoid obstacles, interact with adversaries, and capture the flag.
By using our encoding algorithm, we can convert observations of the board,
present in the form of structured data (in this case, a state class), into a
tensor. Having done so, we can present complex states as input to a neural
network (NN), which acts as a function approximator to learn game dynamics. In
this project report, we will provide an overview of the project, describe the
methods we developed, and present the results obtained.

##  2\. Overview of Battlespace

Figure 1. An example Battlespace board, snapshot from Mid Game: Four players
represented by different colors are playing the game. Purple and Red units are
in one team, deployed in the northern half of the board. Yellow and Green are
in the second team, deployed on the southern half of the board. We can see
that players have employed a safe playing strategy by placing their flags at
the edge of the board as far away from the enemy territory as possible, with
units in front of the flag. In this figure, we see both the air layer and the
ground layer superimposed because that is how the built-in UI renders the
game.

The Battlespace board game is a complex game with two modes, namely
Annihilation and Capture the Flag. The goal of Annihilation is to completely
destroy enemy units to win the game. Capture the Flag is an extension of
Annihilation. This game is won either by destroying all enemy units or
reaching the enemy’s flag. We estimate the state space of the game to be
incredibly large, with around $50,000$ possible states based on the number of
units and unit properties (See Appendix A).

Our version of the game is playable on a board of dimensions of $10\times
11\times 2$, where the first two dimensions are the width and the length of
the board shown in Figure 1. The third dimension shows if the object is on
land or in the air. We can imagine the aerial dimension as a grid of shape
$10\times 11$ placed above the land grid of $10\times 11$. Players only have
imperfect information due to partial observability, as players cannot see the
opponent’s pieces until they are within the visible range of x squares in each
ordinal direction.

Each team can have one or two players and must strategically place their
units, including a flag, soldier, tank, truck, and airplane, on their segment
of the board in the _deployment phase_ (detailed later in Appendix Figures 6
and 7. Note that game parameters like board size and number of units are
mutable and we have made simplifying assumptions for our experiments which is
mentioned in Section 5. After the deployment phase, the game begins.

We define move as a selection of an action for a unit, turn as a collection of
moves for all units for a player and round as a collection of all the turns.
The players enter the moves for their units on the terminal synchronously and
all the moves are compiled at the end of a round. The simulator resolves all
players’ actions at the end of a round and updates the game state. The UI
updates after each round and players can see the changes as a consequence of
their previous turn.

Figure 2. Class hierarchy used in Battlespace. We represented the _abstract_
classes (each having no fill color) via the following shapes: Unit with a
triangle, Movable with a rectangle, and Immovable with an oval. The _player-
deployed_ Units are textured, meaning those with a solid fill are not deployed
by the user during the deployment phase. The _player-controlled_ units have a
blue fill color, meaning those with a red fill color are not under user
control after the deployment phase.

The units in the game can be classified into movable and non-movable units.
The movable units can change their position (and possibly orientation.
Reference Fig 2) in the game. Soldiers, tanks, trucks, airplanes, missiles,
and bombs are the exhaustive list of movable units (missiles and bombs are not
player-deployed or player-controlled). By player-deployed, we mean the units
are deployed in the game deployment phase by the player. However, missiles and
bombs are shot by users when they select actions.. Figure 2 shows the class
hierarchy. Once immovable units are placed on the map, they cannot be moved or
re-oriented. In the default setting, each player must place 5 units on the
board that include a soldier, tank, truck, plane, and flag in their quadrant
of the grid, resulting in a board that might look like the one shown in Figure
1. In our case, the game places 10 walls at the center of the grid on land
terrain with an opening in the middle to set up a breaching scenario (Hung et
al., 2022). In other words, to go from one side of the board to another, there
is only one possible grid unit on the land terrain which is at the center of
the board. This contrasts with objects in the air like airplanes and missiles
shot in the air that can move over the walls.

| _Actions_ |  _Definitions_
---|---|---
_Land
 _Only__ | ram |  Ram in the direction of current orientation. Enemy unit present in the location of the rammed square will be destroyed. If a wall is present, then the unit ramming will be destroyed.
advance1 |  Move one square forward toward the current orientation.
_Land _AND_ Air Units_ | shoot |  Shoot a missile in the direction of the current orientation.
doNothing |  Take no action, maintaining position, orientiation, etc.
turn-135 |  The unit takes its current orientation as the x-axis reference and then rotates by the specified values relative around the height axis.
turn-90
turn-45
turn0
turn45
turn90
turn135
turn180
_Air Units _Only__ | bomb |  Drop a bomb downwards.
advance0,-2 |  Advance by $(a,b)$ squares, while maintaining the same orientation.
advance-1,-1
advance0,-1
advance1,-1
advance-2,0
advance-1,0
advance0,0
advance1,0
advance2,0
advance-1,1
advance0,1
advance1,1
advance0,2
Table 1. List of actions available to both land and air units in Battlespace,
with a description of each action. For each playable unit, the player can
choose an action based on it being a land unit like Soldier, Tank, or Truck
(top two blocks of actions) or if it’s an air unit like an Airplane (bottom
two blocks of actions). Each playable unit will perform one of these actions
in one turn.

Some of the movable units of the game have a different set of actions, shown
in Table 1. Land units (soldier, tank and truck) have 12 actions: moving
forward, rotating, shooting, etc. Air units (airplanes) have 26 actions: most
of the land actions but more flexible in movement. “Projectile units” (bombs,
missiles) have only 1 action which is to continue moving in a direction once
fired. This action is not controlled by the user and is encoded as a property
of projectile units. In particular, missiles continue in the direction of
their projection and bombs drop downwards till they reach land. Walls can stop
projectiles, but only if they are on the ground layer, meaning missiles shot
by planes will go above the walls. Immovable units (flags, walls) stay in
their deployment location till the game is over and cannot be destroyed.

We define the reward system for this game to be +1 for a win, 0 for a draw,
and -1 for a loss. While we used a “sparse reward” approach, there are
multiple ways to shape the rewards. For example, we could define rewards for
different milestones like positive rewards for capturing an enemy piece,
moving towards the enemy territory, or spotting the enemy flag; with negative
rewards for losing a piece, being spotted in enemy territory, or having the
enemy spot your flag. Note that these kinds of shaping rewards would lend
themselves naturally to rewards decomposition, as used by Anderson et al.
(Anderson et al., 2019, 2020). In previous work on Battlespace (Hare et al.,
2021; Hung et al., 2022), the authors suggested that reward shaping biased the
agent to choose to shoot repeatedly over other actions. To avoid this and
other biases that might come from ill-specified shaping reward (e.g., the
famous boat race example), we stuck to the sparse rewards at the end of a full
trajectory listed previously.

Battlespace is playable on a local host with multiple players playing against
each other or against an agent (Random, Bayesian, Q-learning). Battlespace can
run the game in headless mode with just the agents playing against each other
on their own without a server configuration. Alternatively, to play a game on
localhost, we must create a server using the ServerWithUI.py file. After we
create a server, we can run two other terminal windows on a local host and
connect to that server using the HumanInterface.py file. Once the players are
connected, they can see a UI grid. The system then prompts the players to
place their units in their respective playable areas to start the game
(Appendix B details this process a bit more).

Headless mode is accessible from a separate git branch. The users can checkout
to ‘test1’ branch and run the scripts starting with test_<agent_name>.py for
pitting different agents against a Random Agent.

##  3\. Encoding

Based on our initial brainstorming, we started with an encoding approach based
on concatenating binary strings. Over the course of the project, we made
changes to this approach and ran experiments with different encodings,
eventually arriving at a solution that used many more layers. In addition, we
also examined an approach used by a built-in agent that uses a list as input.

###  3.1. Encoding #1 - Binary Strings, Few Layers

We developed an encoding algorithm to convert observations to 3D tensors of
shape $(m,n,numPlayers^{*}2+2)$, where $m$ and $n$ are the width and length of
the board, respectively. Each player has two layers in the third dimension.
Each player’s first layer, calculated by $(playerID^{*}2)$, contains the land
units of the respective player. Each player’s second layer, calculated
similarly $(playerID^{*}2+1)$, contains the air units of the respective
player. The extra layers encode (1) the obstacles and (2) the actions taken by
the players (Note: we did not finish implementing this). An example of this
encoding is available in Appendix C.

Variable |  Description |  Value Range
---|---|---
width |  The width of the board. |  width $\in\mathbb{N}$
|  _Simplifying Assumption_ : $width=10$ (later shrunken further to width $=5$) |
length |  The length of the board. |  length $\in\mathbb{N}$
|  _Simplifying Assumption_ : length$=11$ (later shrunken further to length$=5$) |
height |  The height of the board. |  height $\in\mathbb{N}$
|  _Simplifying Assumption_ : height $=2$ (later shrunken further to height $=1$) |
ownerID |  Each team of players receives a team ID starting from 0. Walls are not assigned to any team, but projectiles are. |  ownerID $\in\mathbb{W}$
|  _Simplifying Assumption_ : ownerID$\leq 2$ |
playerID |  Each player receives a player ID starting from 0. |  playerID $\in\mathbb{W}$
|  _Simplifying Assumption_ : playerID $\leq 4$ |
unitID |  Each unit placed on the board receives an ID, for both movable and immovable objects. If new units spawn during the game, for example projectiles, they receive the next counter value. Note that under our other simplifying assumptions, unitID is stable between episodes and we can also use it as an indicator of the unit’s type. |  unitID $\in\mathbb{W}$
|  _Simplifying Assumption_ : unitID $\leq 32$ (to ensure the binary encoding has a fixed length of 5) |
position |  Position of a unit on the board. Each element of the position 3-tuple ranges from width, length, or height, meaning all simplifying assumptions flow downstream from those variables. |  $(x,y,z),\,\,$where $x,y,z\in\mathbb{W}$
orientation |  Orientation of a unit. In Battlespace, most units are only oriented in the plane parallel to the ground. The exception is bombs, whose orientation is always $(0,0,-1)$. All other units can take on the eight ordinal orientations: {(0,1,0), (1,1,0), (-1,1,0), (1,0,0), (-1,0,0), (0,-1,0), (1,-1,0), (-1,-1,0)}. Note that $(0,0,0)$ is not a legal orientation. |  $(x,y,z),\,\,$where $x,y,z\in\\{-1,0,1\\}$
health |  Represents the health of a unit. |  health $\in\mathbb{W}$
|  _Simplifying assumption #1_ : health $=1$ for units which are alive. |
|  _Simplifying assumption #2_ : There are no mechanisms by which to heal, meaning that we need not separately store maximum health and current health. |
visibleRange |  Represents the number of squares in each of the eight ordinal directions a unit can see. Note that this means diagonal visibility is allowed. |  visibleRange $\in\mathbb{W}$
|  _Simplifying assumption_ : visibleRange $=1$ for all units. |
unitClass |  Binary representation of a unit class (either flag or not flag). If a unit class is of flag type, it is represented by 1 else, it’s represented by 0. |  unitClass $\in\\{0,1\\}$
Table 2. Table defining variables stored in the game and unit classes,
alongside their value ranges.

This algorithm takes in the board and calculates a binary string for each cell
of the tensor encoding the properties of any unit in that location under the
appropriate player’s ownership. To do so, we use the properties found in Table
2, such as unitId, Orientation, and Health, following Equation 1 to form the
binary string and then casting to integer for storage.

(1) $binaryString=Concat(Binary(unitID),Binary(orientation),Binary(health))$

Our implementation makes several simplifying assumptions, all of which are
listed in Table 2. In some cases, these assumptions improve the tractability
of computation (e.g., board size is relatively small and of fixed size), and
in other cases bound the range of numbers to allow the binarization process to
generate a fixed length string. In particular, we capped the unitID to 5 bits,
orientation to 4 bits, and health to 1 bit. This configuration translates to
$2^{5}$ unique possible unitIDs, $2^{4}$ different orientations, and $2^{1}$
health values. The Battlespace domain requires 9 different orientations and we
created a dictionary mapping of these orientations to numbers ranging (0–8).

###  3.2. Encoding #2 - Int/Bool, More Layers

With our encoding strategy, we observed that the unitId was overpowering the
orientation while binarizing data. Upon examining Equation 1, we see that
unitID is stored in the most significant bits, while orientation is in the
middle bits. In response, we separated each property into its own layer
containing either an integer or a boolean and discarded unitID, meaning each
player now has _four_ layers in the tensor instead of _two_. The first layer’s
two layers represent the player’s land units, the first ground layer will
contain unitID while the second ground layer will contain orientation in the
position of any existing units. The same approach would work for the air units
as well, but instead, we made a simplifying assumption to just use land units
and drop air units for this case.

###  3.3. Encoding #3 - list of properties

Finally, we also experiment with encoding packaged with the Battlespace
creators. An encoded state vector is initialized with properties of the unit
like [’owner’, ’unitID’, ’unitClass’, ’visibleRange’, ’health’, ’positionX’,
’positionY’,’positionZ’, ’orientationX’, ’orientationY’, ’orientationZ’ ].
Then, we check the region of visibleRange for friendly or enemy units. We
encode each square in this range with 0 if no unit is present there, 1 if a
friendly unit is present, or 2 if enemy unit is present. This encoding is
appended to the state vector listed previously. For example, if we have
visibleRange of 1, then we have a total of 8 squares around the unit in its
visible range to encode. We repeat the same process is repeated for each
playable unit.

##  4\. Agents

Battlespace includes a random agent designed to generate random actions. We
then created two additional agents, one based on search and the other based on
a NN, which is the focus of this paper.

###  4.1. Random Agent

The Random Agent is characterized by a list of weights that control the level
of preference for each available action. Depending on the specified weights,
we can have either a uniform random agent or a weighted random agent. A
Weighted Random Agent might choose actions based on the distribution of human
choices (as in (Hung et al., 2022)), or based on actions desirable for
training because they modify the game state more, like advancing as opposed to
doing nothing. We used Random Agents when creating other more complex agents,
such as when generating rollouts for MCTS (Monte Carlo Tree Search).

###  4.2. MCTS Agent

We use MCTS (Monte Carlo Tree Search) agents generate a win, loss, and draw
probability for each action available from the given state. MCTS searches
different states in the game by simulating making random moves many times from
the current state and recording the results. Then, once the search tree is
constructed, we use it to select an action by taking an arg max based on the
recorded win, loss, and draw results. The core of MCTS is managing the
explore-exploit tradeoff to try to focus sampling on parts of the search tree
that appear promising. An alternative that samples more uniformly is called
Pure Monte Carlo Game Search (PMCGS) (Russell and Norvig, 2016).

The first component where we used MCTS is the MCTS Agent, which generates
rollouts based on (both uniform and weighted) Random Agents’ choices and gives
the win percentage for each move. We experimented with different quantities of
rollouts, observing the expected result; as the number of rollouts increases,
we see better convergence. For example, if we have 12 actions and 500
rollouts, each action can be explored 42 times on average, but for the 26
actions for an air unit, that same 500 rollouts only results in around 19
samples for each action. Due to computational limitations, we ran experiments
for 200 to 1000 rollouts (each game lasts for around 40 to 50 moves on
average).

The second component where we used MCTS is to provide a training signal for
the Neural Network agent, which is responsible for generating the same win,
loss, draw probability; but in a different way. Our goal in this usage is a
little different from the MCTS Agent usage, where the goal is to select a good
action. Here, we want the information stored at the top level of the search
tree for _all_ moves, both good and bad.

###  4.3. Neural Network Agent

Figure 3. How the AI agent makes a decision, showing nouns in black boxes,
verbs on arrows, and the data involved above each step in the pipeline
(adapted from (Dodge et al., 2022)). The process begins with a board, which
gets converted to a board tensor by separating game objects by faction. Here,
the green tank is about to move, and we assume the green plane and blue tank
are friendly toward the green tank, while the red tank is an enemy, so we
encode their type and heading into the tensor. A convolutional neural network
then takes the board tensor as input and outputs outcome probabilities, which
the agent then scores for each action, resulting in a matrix. After enforcing
domain constraints on the score matrix, the agent applies a softmax to the
scores. Finally, to select an action, the agent samples from the resulting
distribution. Parts in cyan activate only during training. To contrast with
typical approaches, most AI architectures for this task will directly map
either board $\rightarrow$ action (an opaque box) or (board, action)
$\rightarrow$ value (which must be run many times in order to generate
actions/explanations).

Our initial decisionmaking process for the Neural Network Agent is based on
the one found in Dodge, et al. (Dodge et al., 2022), as illustrated in Figure
3. After running an encoding process on a board, we have a board tensor. We
feed board tensors to the NN, which is trained on the scores calculated by the
MCTS procedure. We interpret the output from the NN as a set of Outcome
Probabilities because that is what the MCTS training signal provides. In the
first step of our post-processing, we put the Outcome Probabilities through a
(non-learned) scoring function to calculate Action Scores. Next, we enforce
domain constraints on the Action Scores by masking illegal moves. Last, a
softmax function on the Action Scores produces a predicted best move for each
playable unit.

Having chosen NN as our function approximator, we experimented with various
architectures based on the input encoding. To train our NNs we rely on the
Deep Q-learning Network (DQN) procedure, using a target derived by MCTS. Our
experiments, detailed in Section 5, focused primarily on convolutional NN
(CNN) and dense NN. The choice of architecture and hyperparameters can have a
substantial impact on the performance of the NN, as it affects the network’s
ability to capture the relevant features of the board and the game state.

In sequential domains, one common problem when training agents is that the
states along a single trajectory are highly correlated. Thus, in order to
reduce unwanted biases, one should seek uncorrelated states, which we used two
methods to do. The first method was to generate a random board, get an action
from the agent, perform a simulation step, and a backward pass on the network;
then we would throw that board away, generate a new random board, and repeat.
The second method was the same process with respect to random board
generation, but forming mini-batches of size 4–8 before performing the
backward pass in an effort to stabilize the gradient.

During the course of the project, we made several changes to the architecture,
described in Section 5, based on lessons from experimentation and ideas we had
after reading Perolat et al. (Perolat et al., 2022).

##  5\. Experiments

The base version of Battlespace has 4 players competing on a $10\times 11$
grid containing around 30 units (player-deployed units and system-deployed
units like walls). To make our experiments more computationally tractable, we
made the simplifying assumptions listed in Table 2. In particular, we reduced
the number of units and set the number of players to 2, while also reducing
the board size to $5\times 5$. In addition, we sought to avoid wasting
computation repeatedly learning from the same (or similar) initial conditions,
so we built a random board generator that placed units randomly (both position
and orientation) on the board while playing in headless mode. This board
generator lifted the restriction of units being deployed in the players’ own
quadrants, meaning that the random board generator can produce states from the
mid-game or end-game phase.

###  5.1. Experiments with encoding

In parallel, we explored multiple encoding techniques among the three
described in Section 3 to gain a better understanding of the impact of
encoding on the outputs of the neural network. Encoding #1 involved converting
metadata such as unitID, orientation, and health to binary format,
concatenating the bit strings, and then converting the concatenated long bit
string to an integer. We used this integer value as the representation of a
particular unit, placing it in the tensor the grid at the unit’s position.
However, this method generated some problems due to the concatenation order
during the prior conversion. For instance, since Equation 1 illustrates how
unitID is contained in the most significant bits, that property received
higher importance in the binary-to-integer conversion, while health and
orientation received lesser importance.

###  5.2. Experiments with neural network

Our approach was to treat this problem from a computer vision perspective,
where we embedded each unit into separate layers and considered these layers
as channels of a CNN. CNNs are well-suited for the Battlespace domain, because
it has a grid structure and most aspects of its rules are translationally-
invariant. CNNs apply small filters (kernels) to local regions of the input
data, enabling them to efficiently detect and capture local patterns and
features. This capability makes CNNs highly beneficial for tasks in domains
with partial observability (like Battlespace), where extracting meaningful
information from limited visual data is crucial for decision-making and
situational awareness. While experimenting with list encoding, we had to
switch to Dense NN because there was no spatial correlation between
consecutive elements of an array.

Figure 4. Histogram showing actions selected by the NN agent after training
for 20 epochs with a batch size of 4 and MCTS rollouts set to 500 in 4
different games. Each image shows all moves from the start until the game is
over, with the number of rounds played in each game shown in the right-hand
top corner. This chart is for the simplified version of the game where the
game was played between two players with one land unit per player. Hence, the
number of unique actions is 12. For better visual clarity some moves (e.g.,
’turn0’, ’turn45’, ’advance1’, ’ram’) do not appear in the figure because
these actions went unused in all of these games. The actions in red are “non-
impactful” or non-offensive while the actions in blue are transitional or
offensive actions.

Through our experiments, we observed that the CNN responded poorly to changes
in orientation. To address this issue, we separated the position and
orientation into different layers as mentioned in Section 3.2. These revisions
provide a clearer and more cohesive description of the process and the actions
taken during the experimentation.

During our experimentation with different neural network architectures, we
made several changes to improve model performance. For the CNNs, we explored
varying the number of convolutional layers to find the best configuration.
Additionally, we experimented with adding one or two dense layers after the
reshaping operation. In the case of Dense Neural Networks, we conducted
multiple trials by adjusting the number of layers and layer sizes to determine
the optimal depth and width for the architecture. We used Hare et al.’s
(Defense Innovation Board, 2019) network as a reference point, drawing
insights from its design choices for my experiments with dense networks.
Moreover, the basic CNN architecture from the PyTorch image tutorial served as
a starting point, which we customized and modified to suit the specific tasks
and datasets at hand.

###  5.3. Experiments with MCTS agent

The Battlespace domain has state space of $\approx 8\times 10^{514}$, as we
detail in Appendix A. With our limited computational resources, running MCTS
such a large state space proved challenging, even with the speed of generating
rollouts quickly with the Random Agent. Therefore, we conducted experiments
with around 500 rollouts for each board input to the MCTS agent. The size of
the action space for each state of the game depends on the number of units in
play. In particular, the total number of unique actions per turn will be a
Cartesian product of all the available actions for each unit. For example, if
there are 3 units left on the board and each unit has 12 actions, then the
combination of unique actions will be $12^{3}=1728$. Further simplifying the
game, we experimented with just 1 and 2 units per player where the action
space sizes were 12 and 144, respectively.

We observed that out of the 12 moves (just considering land units for this
example), 9 did not impact the state very much, merely changing the
orientation of the unit or doing literally nothing. As we were using a Uniform
Random Agent during rollouts, low-impact moves were more likely to occur than
offensive ones that translated a unit or created a projectile.

The agents learned that sedentary and safe options were better than offensive
and risky moves, essentially developing a bias toward safety (Figure 4). In
this figure, we can see that the action highlighted in red are non-impactful
actions like turning and doing nothing. The ones highlighted in blue are
actions that are offensive or change positions like shoot and advance. We see
that most of the time agent prefers to stay in the same position and chooses
non-offensive actions. As a result, it is hard to get the agent into positions
that allow it to explore unseen states, and so the agent’s grasp of the game
dynamics is necessarily limited.

To address the issue with exploration, we experimented with a Non-Uniform
Random Agent to generate moves for the MCTS agent, assigning higher weights to
actions that create projectiles or adjust positions. We saw similar results as
with Non-Uniform Random Agent. Specifically, computational challenges to
achieve sufficient MCTS rollouts remained our greatest barrier to an agent
that selects quality moves.

##  6\. Discussion - Comparing how each encoding represents the structured
data

In the initial encoding, we latched the variables together by concatenating
them into a long string (essentially superimposing the properties). However,
in the subsequent encodings, we placed them one on top of the other
(essentially juxtaposing the properties). In this case, there is a trade-off
between complexity in the form of many cells with simple contents vs few cells
with complex contents111Consider that most vision applications use either 3 or
4 channel images, while our “many layers” encoding uses 6 under our
simplifying assumptions, but could require many more than 6 channels..
However, we found that employing the first method resulted in agents that
prioritized the ordering of the concatenation whereas the second one resulted
in agents that prioritized the position of the units in making decisions.
Further, partway through this process we discovered the paper by Perolat et
al. (Perolat et al., 2022), which used a very similar approach to our Encoding
#2 (detailed later in Appendix Figure 8). Note that both DeepMind’s approach
and approaches are similar in that the number of layers required grows rapidly
as the number and range of unit properties grows, so building more compact
representations for structured data remains an open question.

##  7\. Conclusion and Future Work

We have described a series of experiments on various types of encodings,
architecture, and scoring functions, arriving at insights for each topic.

Our scoring function relied on calculating win percentages. Due to the
reduction in unit count many games ended in a draw because of timeouts or
mutual destruction. We did not expect this because draws seem to be rare in
human games in the Battlespace domain.

We have tested various convolutional neural networks (CNNs) and observed a
substantial difference in outputs before and after resizing the board and re-
sampling the number of units. Our model’s losses decrease initially indicating
a good training signal. However, as most of the action space has stationary
actions, agents developed a bias for safe plays (i.e., playing to not lose
instead of playing to win). We used outputs from Monte Carlo tree search
(MCTS) to generate scores for games. Our Neural Network learned to predict
these scores given a board as input. As we increased the number of playable
units, the action space grew exponentially, creating a computational
bottleneck. One solution to this bottleneck that we would like to consider in
future work is to abandon the search tree entirely and attempt to adapt the
game-theoretic approach found in the DeepNash Stratego agent (Perolat et al.,
2022) to this domain.

Our initial encoding approach was to concatenate the binary representation of
unit’s metadata information. This approach demonstrated the importance of
concatenation order in the encoding process. However, we acknowledge the need
for further research to improve encoding strategies. In our current
architecture, we have implemented only the state encodings. We were inspired
by the Deep Nash paper and we plan to encode move history in our neural
network architecture in future work.

## References

  * (1)
  * Anderson et al. (2019) Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Alan Fern, and Margaret Burnett. 2019.  Explaining Reinforcement Learning to Mere Mortals: An Empirical Study. In _International Joint Conference on Artificial Intelligence_. IJCAI, Macau, China.
  * Anderson et al. (2020) Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis, Evan Newman, Jed Irvine, Souti Chattopadhyay, Matthew Olson, Alan Fern, and Margaret Burnett. 2020\.  Mental Models of Mere Mortals with Explanations of Reinforcement Learning.  _ACM Trans. Interact. Intell. Syst. (TIIS)_ 10, 2, Article 15 (2020), 37 pages.  https://doi.org/10.1145/3366485
  * Defense Innovation Board (2019) Defense Innovation Board. 2019\.  AI Principles: Recommendations on the Ethical Use of Artificial Intelligence by the Department of Defense: Supporting Document.  _United States Department of Defense_ (2019).
  * Dodge et al. (2022) Jonathan Dodge, Andrew Anderson, Matthew Olson, Rupika Dikkala, and Margaret Burnett. 2022.  How Do People Rank Multiple Mutant Agents?. In _ACM Intelligent User Interfaces (IUI)_ (Helsinki, Finland). 21 pages.  https://doi.org/10.1145/3490099.3511115
  * Hare et al. (2021) J Zachary Hare, B Christopher Rinderspacher, Sue E Kase, Simon Su, and Chou P Hung. 2021\.  Battlespace: using AI to understand friendly vs. hostile decision dynamics in MDO. In _Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III_ , Vol. 11746. International Society for Optics and Photonics.
  * Hung et al. (2022) Chou P Hung, J Zachary Hare, B Christopher Rinderspacher, Sue E Kase, Simon M Su, Walter Peregrim, Olena Tkachenko, Tomer Krayzman, Adrienne J Raglin, and John T Richardson. 2022.  _Final Report of DEVCOM ARL Directors Future Ventures (FY21): Novel AI Decision Aids for Decision Dynamics, Deception, and Game Theory (Summary Technical Report, Oct 2020-Sep 2021)_.  Technical Report. DEVCOM Army Research Laboratory.
  * Perolat et al. (2022) Julien Perolat, Bart De Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T. Connor, Neil Burch, Thomas Anthony, Stephen McAleer, Romuald Elie, Sarah H. Cen, Zhe Wang, Audrunas Gruslys, Aleksandra Malysheva, Mina Khan, Sherjil Ozair, Finbarr Timbers, Toby Pohlen, Tom Eccles, Mark Rowland, Marc Lanctot, Jean-Baptiste Lespiau, Bilal Piot, Shayegan Omidshafiei, Edward Lockhart, Laurent Sifre, Nathalie Beauguerlange, Remi Munos, David Silver, Satinder Singh, Demis Hassabis, and Karl Tuyls. 2022\.  Mastering the game of Stratego with model-free multiagent reinforcement learning.  _Science_ 378, 6623 (2022), 990–996.  https://doi.org/10.1126/science.add4679 arXiv:https://www.science.org/doi/pdf/10.1126/science.add4679
  * Russell and Norvig (2016) Stuart J Russell and Peter Norvig. 2016.  _Artificial intelligence: a modern approach_.  Malaysia; Pearson Education Limited,.

##  Appendix A State space calculation

To calculate an estimate of the state space size for the original game
including two teams with two players in each team, we begin with the number of
objects that might occupy a single square:

  1. (1)

4 soldiers (one of each team’s color)

  2. (2)

4 tanks

  3. (3)

4 trucks

  4. (4)

4 flags

  5. (5)

4 airplanes

  6. (6)

$M$ missiles (we estimate that around 6 missiles is reasonable)

  7. (7)

$B$ bombs (estimate around 1)

  8. (8)

$W$ walls (estimate around 10)

  9. (9)

nothing

This results in a total unit count of:

$totalUnits=20+M+B+W\approx 37$

Most of these units can have 8 different orientations, leading to a total
possible unit combinations at one square of:

$combinationsPerSquare=8^{*}(20+M)+B+W=160+8M+B+W\approx 219$

The board is $10\times 11\times 2=220$ squares, so we have:

$|StateSpace|=combinationsPerSquare^{squares}\approx 219^{220}\approx 8\times
10^{514}$

At this point, we observe that we discarded health, which would make this
estimate would grow even larger.

##  Appendix B Game Deployment and Usage

Steps to run the game on server:

  1. (1)

Git clone https://gitlab.kitware.com/mixtape/battlespace.git

  2. (2)

Install dependencies from the requirement.txt file.

  3. (3)

Run 3 terminals in parallel and change to the test directory in all of them.

  4. (4)

Use first terminal to create a server by using the command ‘`python
ServerWithUi.py --test`’.

  5. (5)

Run the game from second and third terminal by using the command ‘`python
HumanInterface.py`’.

  6. (6)

Enter the localhost id shown on the first terminal in the second and third
terminal to see the deployment phase.

  7. (7)

Place your units by clicking on the cells in the highlighted grid in yellow
(see Appendix Figures 6 and 7).

  8. (8)

After all the players place their units, the game begins. Players can enter
the action for each of their units base on the prompts shown on their
terminal, as shown in Appendix Figure 5.

Figure 5. A human player makes a move by selecting move options for each
playable unit from terminal Figure 6. Screenshot of Battlespace’s UI just
prior to deployment phase. Left board: Player 1 is about to place their units
in the region highlighted with yellow. Right board: Player 2 waits till Player
1 completes deployment. Figure 7. Screenshot of Battlespace’s UI during the
deployment phase. Left board: Player 1 has placed their units. Right board:
Player 2 initiates deployment. Note that Player 2 is able to see Player 1’s
units because they are teammates. Player 3 will not be able to see either of
these choices as they perform deployment.

##  Appendix C Our Tensor Representation

The following is an example of tensor output from Encoding #1. This example is
for the original game running with 10$\times 11$ board size, 2 teams, 2
players per team, and 5 units per player. The tensor_mapper() function has
embedded the board by using Equation 1 to convert the metadata representation
of each unit to an integer number which we placed at the same position where
the unit is located on the board. Each player is represented by two layers.
For example, the first and second layer form the encoding represents the land
and air units of Player 1. The next 6 layers represent the other 3 players.
The second to last layer represents the walls and the last layer was supposed
to encode the previous moves, but we did not finish that aspect of
implementation and left it containing zeroes.

        [[[  0.   0.  75.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0. 137.   0.   0.  13.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.  37.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]

     [[  0.   0.   0. 111.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]

     [[  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0. 201. 235.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0. 297.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [163.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]

     [[  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0. 265.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]

     [[  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0. 457.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]

     [[  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0. 423.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]

     [[  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0. 617.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]

     [[  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0. 583.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]

     [[  0.   0.   0.   0.   0. 649.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0. 681.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0. 713.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0. 745.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0. 809.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0. 841.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0. 873.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0. 905.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0. 937.   0.   0.   0.   0.   0.]]

     [[  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]
      [  0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.]]]

##  Appendix D DeepMind’s Tensor Representation

Figure 8 illustrates the encoding process DeepMind used in Perolat et al.
(Perolat et al., 2022). Note that the approach illustrated here is a mixture
of boolean (e.g., player pieces) and integer (e.g., opponent visible
information). Also note that the number of layers required for this approach
seems to grow as a Cartesian product of the properties game objects exhibit.

Figure 8. _(Image and caption credit to Perolat et al.(Perolat et al., 2022))_
: The input of the neural network is a single tensor encoding the position of
pieces, the currently known information of both opponent and own pieces
(whether a piece moved or was revealed), a limited move history and the
position of the lakes.


# harmonic gated compensation network plus for ICASSP 2022 DNS Challenge

###### Abstract

The harmonic structure of speech is resistant to noise, but the harmonics may
still be partially masked by noise. Therefore, we previously proposed a
harmonic gated compensation network (HGCN) to predict the full harmonic
locations based on the unmasked harmonics and process the result of a coarse
enhancement module to recover the masked harmonics. In addition, the auditory
loudness loss function is used to train the network. For the DNS Challenge, we
update HGCN with the following aspects, resulting in HGCN+. First, a high-band
module is employed to help the model handle full-band signals. Second, cosine
is used to model the harmonic structure more accurately. Then, the dual-path
encoder and dual-path rnn (DPRNN) are introduced to take full advantage of the
features. Finally, a gated residual linear structure replaces the gated
convolution in the compensation module to increase the receptive field of
frequency. The experimental results show that each updated module brings
performance improvement to the model. HGCN+ also outperforms the referenced
models on both wide-band and full-band test sets.

Index Terms—  Speech Enhancement, Harmonic, Deep Learning, Pitch

##  1 Introduction

††⋆Work done during internship at China Mobile Research Institute.

Speech enhancement (SE) aims to improve speech quality. Many researchers
introduce intuitive signal processing ideas to deep learning [1, 2]. [3]
explores an SE model combining signal processing and deep learning, but simple
structure limits its performance. [4] introduces the complex operations, which
improve the performance but with less auditory characteristic. [5, 6]
introduce the auditory feature, but spectra with wide frequency bandwidth
degrade the performance. [7] verifies the effectiveness of the hearing
pipeline structure. [8] verifies the necessity of modeling the harmonic.
Inspired by these works, we proposed the HGCN for SE [9]. In HGCN, a high-
resolution harmonic integration algorithm is proposed to predict the harmonic
locations. Then the locations are used as a gate to help the subsequent module
compensate for the result of the coarse enhancement module to obtain a refined
result. To make the enhancement more consistent with human hearing, we
previously proposed a loss function based on auditory loudness power
compression (APC-SNR) [10]. Both HGCN and APC-SNR have proved their
effectiveness for SE.

Compared to the HGCN, each module of the HGCN+ is updated and the HGCN+ can
handle full-band (FB, 0~24 KHz) signals. Since the wide-band (WB, 0~8 KHz) is
more likely to contain high energies, tonalities and long sustained sounds,
while the high-band (HB, 8~24 KHz) tends to have low energies, noise and
rapidly decaying sounds [11], the HB and WB spectra are modeled separately
[5]. The HB spectrum is enhanced by a lightweight NSNet [2] and the WB
spectrum is enhanced by an HGCN that is updated in the following aspects. 1)
The dual-path encoder and DPRNN [12, 13] are introduced to take full advantage
of the features. 2) Cosine is adopted to model the harmonic peak-valley
structure, and the voiced region detection (VRD) is judged based on the
harmonic integration significance. 3) The gated convolution is replaced by a
residual gated structure comprised of linear layers and Gated Recurrent Units
(GRUs) [14] to increase the receptive field of frequency. Since we model the
WB and HB spectra separately, HGCN+ can h andle both WB and FB signals without
resampling. Experimental results show that HGCN+ outperforms the referenced
methods on test sets.

Fig. 1: Architecture of the proposed HGCN+.

##  2 PROPOSED HGCN+

The overall diagram of the HGCN+ is as shown in Fig. 1, which is comprised of
four parts, namely the high-band module (HBM), coarse enhancement module
(CEM), harmonic locations prediction module (HM), and gated harmonic
compensation module (GHCM). The noisy signal is split into WB and HB spectra
after a short-time Fourier transform (STFT). The HBM enhances the HB spectrum.
And the WB spectrum is firstly passed to CEM to obtain a coarse result.
Subsequently, HM predicts harmonic locations based on the coarse result. Then,
GHCM compensates for the coarse result based on the harmonic location gates to
get the refined WB result. Finally, the enhanced WB and HB spectra are
concatenated and converted to waveform by an inverse STFT (iSTFT).

###  2.1 High-band model

HB spectra contain less speech information, and elaborate operation on them
could bring a computational burden, so a lightweight magnitude module [2] is
used. The HB spectrum
$\bm{S}^{\text{HB}}=\text{Cat}(\bm{S}^{\text{HB}}_{r},\bm{S}^{\text{HB}}_{i})\in\mathbb{R}^{T\times
2F}$ is firstly converted to $|\bm{S}^{\text{HB}}|$ and
$\bm{S}^{\text{HB}}_{\text{phase}}$, where
$|\bm{S}^{\text{HB}}|=({\bm{S}^{\text{HB}~{}2}_{r}}+{\bm{S}^{\text{\text{HB}~{}2}}_{i}})^{0.5}$
and
${\bm{S}^{\text{HB}}_{\text{phase}}}=\text{arctan}({\bm{S}^{\text{HB}}_{i}},{\bm{S}^{\text{HB}}_{r}})$
are the magnitude and phase, $\bm{S}^{\text{HB}}_{r}$ and
$\bm{S}^{\text{HB}}_{i}$ are the real and imaginary parts, $T$ and $F$ denote
the number of frames and the STFT bins respectively. The magnitude is used as
input to predict the mask $\bm{M}^{\text{HB}}$. And HBM consists of fully
connecting (FC), GRUs and ReLU [15], as shown in Fig. 1. Finally, the HB
enhanced result $\bm{S}^{\text{HB}^{\prime}}$ is obtained by the Mask Apply M
as follow,

$\bm{S}^{\text{HB}^{\prime}}=|\bm{S}^{\text{HB}}|\odot\sigma(\bm{M}^{\text{HB}})\odot
e^{j\bm{S}^{\text{HB}}_{\text{phase}}}$ (1)

where $\odot$ is the element-wise multiplication, $\sigma(\cdot)$ is sigmoid.

###  2.2 Coarse enhancement module

The noise with higher energy than speech is catastrophic for HM. So CEM is
designed to roughly suppress the noise of WB spectra, which is an encoder-
decoder structure. Before being input to the encoder, the WB spectrum
$\bm{S}^{\text{WB}}=\text{Cat}(\bm{S}^{\text{WB}}_{r},\bm{S}^{\text{WB}}_{i})$
is compressed by a power of 0.23 and is input to the dual-path encoder
together with the original spectrum, as shown in Fig. 1. Both the encoder and
decoder are comprised of 2D causal convolution, batch normalization [16], and
PReLU [17]. Between the encoder and decoder, DPRNN [13, 12] is inserted to
model the multidimensional dependencies and Skip Connection concatenates the
output of each encoder to the input of the corresponding decoder (red line in
Fig. 1). The output channel of the last decoder is $(\text{C}_{\text{A}}+2)$,
where 2 is the estimated mask of CEM
$\bm{M}^{\text{CEM}}=\text{Cat}(\bm{M}^{\text{CEM}}_{r},\bm{M}^{\text{CEM}}_{i})$,
$\text{C}_{\text{A}}$ is introduced in the next section. The Mask Apply E of
CEM is as follows,

$\bm{S}^{\text{WB}^{\prime}}=|\bm{S}^{\text{WB}}|\odot\text{tanh}(|\bm{M}^{\text{CEM}}|)\odot
e^{j(\bm{S}^{\text{WB}}_{\text{phase}}+\bm{M}^{\text{CEM}}_{\text{phase}})}$
(2)

###  2.3 Harmonic locations prediction module

The harmonics that are masked by noise can be deduced from the unmasked
harmonics (red box in Fig. 3). [18, 19] proposed to model the peak-valley
structure of the harmonics on spectra to detect the pitch. The pitch
candidates are set first, and the integral of the multiple locations is taken
as the significance $Q_{t,f_{c}}$ of each candidate $f_{c}$,

$Q_{t,f_{c}}=\sum_{k=1}^{\text{8000}/f_{c}}(\frac{1}{\sqrt{k}}\cdot{|S^{\text{WB}^{\prime}}_{t,kf_{c}}|}^{0.5}-\frac{1}{\sqrt{k}}{|S^{\text{WB}^{\prime}}_{t,(k-\frac{1}{2})f_{c}}}|^{0.5})$
(3)

where $k$ denotes the multiple of the pitch. The candidate with the highest
significance is regarded as the pitch.

To detect the pitch with fine-resolution based on STFT bins with wide-
bandwidth, we proposed a high-resolution harmonic integration matrix $\bm{U}$
in HGCN, which sets candidates in 60~420 Hz (normal pitch range of speech)
with a resolution of 0.1 Hz. In this paper, the peak-valley modeling is
improved by cosine function, and the $\bm{U}$ is designed as Algorithm 1 and
Fig. 2(a), where $[\cdot]$ is a rounding operation, $\text{linspace}(a,b,c)$
generates an arithmetic progression between $a$ and $b$ of length $c$. Then
the Eq. (3) is updated to,

$\bm{Q}_{t}=|\bm{S}^{\text{WB}^{\prime}}_{t}|^{0.5}\cdot\bm{U}^{\top}$ (4)

where $\bm{Q}_{t}\in\mathbb{R}^{1\times 3600}$ denotes the pitch candidate
significances of the $t$-th frame. The candidate corresponding to the maximum
value in $\bm{Q}_{t}$ is regarded as the pitch, and then the corresponding
harmonic peak-valley structure is deduced based on the pitch as
$\bm{R}_{\text{H}}\in\mathbb{R}^{T\times F}$, as shown in Fig. 3.

There is no harmonic in unvoiced and silent frames (orange box in Fig. 3), so
we apply the voiced region detector (VRD) to filter $\bm{R}_{\text{H}}$. In
addition, the energy is low even if it’s harmonic (gray box in Fig. 3), which
needs to be filtered out. Therefore, the final harmonic gate $\bm{G}$ is
calculated as follows,

$\bm{G}=\bm{R}_{\text{VRD}}\odot\bm{R}_{\text{A}}\odot\bm{R}_{\text{H}}$ (5)

where $\bm{R}_{\text{A}}\in\mathbb{R}^{T\times F}$ denotes the non-low energy
locations of speech and is detected by a speech energy detector (SED). Since
SED needs to resist noise, we change the output channel number of the last CEM
decoder to $(2+\text{C}_{\text{A}})$, and $\text{C}_{\text{A}}$ is the
channels number of the input $\bm{X}^{{}^{\prime}}\in\mathbb{R}^{T\times
F\times\text{C}_{\text{A}}}$ for an FC layer ($\text{FC}_{\text{A}}$) to get a
2-D (low-high) classification probabilities $P_{t,f}=[p_{0},p_{1}]$ for every
T-F point $\bm{P}\in\mathbb{R}^{T\times F\times 2}$. And the
$\bm{R}_{\text{A}}$ is obtained by $\bm{R}_{t,f}=\text{argmax}(P_{t,f})$.

The SED is designed to filter out the low energy parts, so we generate labels
for SED based on energy. The mean of each bin in the clean logarithmic
magnitude is counted as $\bm{\mu}=\sum_{t=1}^{T}\log{|\dot{\bm{S}}_{t}|}/T$,
where $|\dot{\bm{S}}|$ represents the clean magnitude. And the label is 1 if
the logarithmic magnitude of clean is larger than
$\bm{\mu}\in\mathbb{R}^{F\times 1}$, 0 otherwise, as shown in Fig. 2(b).

The significances of the voiced frames are higher than that of the unvoiced
and silent frames in the integral spectrum $\bm{Q}$, as shown in Fig. 3. So
the VRD is designed as,

$(\bm{R}_{\text{VRD}})_{t}=\textit{I}(\max{(\bm{Q}_{t})}>\left(\alpha\cdot\xi\right))$
(6)

where $\max({\cdot})$ denotes the maximum value in the vector. If the input is
true, $\textit{I}(\cdot)$ outputs 1. $\xi$ is the moving average which is
updated as
$\xi_{\text{new}}=0.9\xi_{\text{old}}+0.1\sum_{t=1}^{T}\max{(\bm{Q}_{t})}/T$.
$\alpha$ is the scale factor ($\alpha$ is 0.4 in our experiments).

(a)

(b)

Fig. 2: (a) High-resolution harmonic integral matrix $\bm{U}$. (b) Labels for
speech energy detection.

###  2.4 Gated harmonic compensation module

To recover the masked harmonics with the help of the unmasked harmonics, the
module needs to have a wide receptive field of frequency, so a gated residual
structure consisting of FCs and GRUs is adopted to compensate the coarsely
processed spectra according to the harmonic locations $\bm{G}$, as shown in
Fig. 1. And the harmonic gated compensation mask applying (Mask Apply H) is
used as,

$\begin{split}\bm{S}^{\text{WB}^{\prime\prime}}&=\left[1+\text{CC}(\bm{G})\odot\sigma(\bm{M}^{\text{GM}})\right]\odot|\bm{S}^{\text{WB}^{\prime}}|\odot
e^{j\bm{S}_{\text{phase}}^{\text{WB}^{\prime}}}\end{split}$ (7)

where $\bm{M}^{\text{GM}}$ is the estimated mask of GHCM. Since some harmonic
peaks maybe masked by noise, the magnitude that needs to be compensated is not
a complete harmonic structure, a causal convolution (CC) is used to process
the gate.

Finally, $\bm{S}^{\text{WB}^{\prime\prime}}$ and $\bm{S}^{\text{HB}^{\prime}}$
are concatenated into FB complex spectra and converted to the waveform by
iSTFT.

Algorithm 1 Integral matrix

1:$\bm{U}\leftarrow\bm{0}\in\mathbb{R}^{3600\times F}$

2:for $f_{c}\leftarrow 600\to 4200$ do

3: $\text{loc}_{\text{last}}\leftarrow 0$ ;
$\text{peak}_{\text{last}}\leftarrow 1$ ; $j\leftarrow f_{c}-600$

4: for $k\leftarrow 1\to\left[8000/(0.1\cdot f_{c})\right]$ do

5: $\text{loc}\leftarrow\left[0.1\cdot f_{c}\cdot k\cdot F/8000\right]$

6: $\text{peak}\leftarrow 1/\sqrt{k}$ ;
$\bm{U}_{j,\text{loc}}\leftarrow\text{peak}$

7: if $\text{loc}-\text{loc}_{\text{last}}>1$ then

8: $\text{num}_{\text{iner}}=\text{loc}-\text{loc}_{\text{last}}$

9:
$\bm{F^{\text{cos}}}\leftarrow\cos{(\text{linspace}(0,2\pi,\text{num}_{\text{iner}}))}$

10:
$\bm{F}\leftarrow\text{linspace}(\text{peak}_{\text{last}},\text{peak},\text{num}_{\text{iner}})$

11: for $i\leftarrow 1\to\text{num}_{\text{iner}}$ do

12:
$\bm{U}_{j,i+\text{loc}_{\text{last}}}=\bm{F^{\text{cos}}}_{i}\cdot\bm{F}_{i}$

13: else

14:
$\bm{U}_{j,\text{loc}}\leftarrow\bm{U}_{j,\text{loc}}-(\text{peak}_{\text{last}}+\text{peak})/2$

15:
$\bm{U}_{j,\text{loc}_{\text{last}}}\leftarrow\bm{U}_{j,\text{loc}_{\text{last}}}-(\text{peak}_{\text{last}}+\text{peak})/2$

16: $\text{loc}_{\text{last}}\leftarrow\text{loc}$ ;
$\text{peak}_{\text{last}}\leftarrow\text{peak}$

Fig. 3: The calculation process of harmonic gate.

###  2.5 Loss function

The magnitude loss is used to model the HB magnitude. To make the WB
enhancement results consistent with the human hearing, we introduce the time-
domain SI-SNR to measure the complex spectrum compressed by the auditory
loudness power exponent $\bm{\gamma}$ [10]. In addition, FocalLoss [20] is
used as a loss function for the SED. The loss function of the whole model is
defined as,

$L=L_{\text{HB}}+L_{\text{APC}}^{\bm{S}^{\text{WB}^{\prime}}}+L_{\text{APC}}^{\bm{S}^{\text{WB}^{\prime\prime}}}+L_{\text{focal}}$
(8)
$L_{\text{HB}}=\left\||\bm{S}^{\text{HB}^{\prime}}|-|\dot{\bm{S}}^{\text{HB}}|\right\|^{2}+\left\|\log|\bm{S}^{\text{HB}^{\prime}}|-\log|\dot{\bm{S}}^{\text{HB}}|\right\|^{2}$
(9) $L_{\text{focal}}=-\alpha(1-P_{t,f})^{\beta}\cdot\log{P_{t,f}}$ (10)
$\left\\{\begin{array}[]{ccl}\bm{S}^{\text{WB}^{\prime}}_{\text{C}}&=|\bm{S}^{\text{WB}^{\prime}}|\odot(|\bm{S}^{\text{WB}^{\prime}}|+1)^{\frac{\bm{\gamma}-1}{2}}\odot
e^{j\bm{S}^{\text{WB}^{\prime}}_{\text{phase}}}\\\
\dot{\bm{S}}^{\text{WB}}_{\text{C}}&=|\dot{\bm{S}}^{\text{WB}}|\odot(|\dot{\bm{S}}^{\text{WB}}|+1)^{\frac{\bm{\gamma}-1}{2}}\odot
e^{j\dot{\bm{S}}^{\text{WB}}_{\text{phase}}}\\\
\bm{S}_{\text{t}}&=(<\bm{S}^{\text{WB}^{\prime}}_{\text{C}},\dot{\bm{S}}^{\text{WB}}_{\text{C}}>\cdot\dot{\bm{S}}^{\text{WB}}_{\text{C}})/||\dot{\bm{S}}^{\text{WB}}_{\text{C}}||^{2}\\\
L_{\text{APC}}^{\bm{S}^{\text{WB}^{\prime}}}&=10\log_{10}{\left(||\bm{S}_{\text{t}}||^{2}/||\bm{S}^{\text{WB}^{\prime}}_{\text{C}}-\bm{S}_{\text{t}}||^{2}\right)}\\\
\end{array}\right.$ (11)

where $L_{\text{APC}}^{\bm{S}^{\text{WB}^{\prime}}}$ and
$L_{\text{APC}}^{\bm{S}^{\text{WB}^{\prime\prime}}}$ are the loss of CEM and
GHCM respectively, and they are calculated as Eq. (11). $||\cdot||$ denotes
the 2-norm of the vector. We set $\alpha\text{=}1$ and $\beta\text{=}2$ in
experiment.

##  3 Experiments

###  3.1 Dataset

To evaluate the performance of the updated WB part of HGCN+, we generate 100
hours of WB data with signal-to-noise ratios (SNR) ranging in 0~40 dB using
the speech and noise provided by INTERSPEECH 2020 DNS Challenge (DNS-2020)
[21]. We divide the data into training and validation set by 4:1. For testing,
we generate 540 noisy-clean pairs with 3 SNRs (-5 dB, 0 dB, 5 dB) using the
noise and speech that didn’t appear in the training or validation set.

For the ICASSP 2022 DNS Challenge (DNS-2022) [22], we generate a FB set with
3500 hours duration using provided 181 hours of noise, read-English, Russian,
French, and Spanish data, and the SNR ranges -5~25 dB. In addition, half of
the utterances are convolved with random synthetic and real room impulse
responses (RIRs) before being mixed. We divide the data into training and
validation by 4:1.

###  3.2 Training setup and comparison methods

For the WB experiments, we use HGCN (CEM+GHCM+HM) as the reference and
introduce the improvements of each module step by step for comparison
($\text{CEM}^{+}$, $\text{GHCM}^{+}$, $\text{HM}^{+}$, + denotes the updated
version). The $32$ ms hanning window with $25\%$ overlap and 512-point STFT is
used. The kernel size is $(5,2)$. The channel number and stride of encoder and
decoder are $\left\\{12,24,48,64,96,96\right\\}$ and $(2,1)$. For HGCN, a
512-units FC after 3-layer 128-units LSTM is adopted. The channel number and
stride of GHCM are $\left\\{8,16,8\right\\}$ and $(1,1)$ and the out channel
of last decoder is 22 ($\text{C}_{\text{A}}=\text{C}_{\text{B}}=10$). Since
the $\text{GHCM}^{+}$ and the DPRNN are residual, the hidden dimension is the
same as the input. The optimizer is Adam [23]. And the initial learning rate
is $0.001$, which decays $50\%$ when the valid loss plateaus for $5$ epochs,
and training is stopped if loss plateaus for 20 epochs.

For HGCN+ in DNS-2022, the $32$ ms Hanning window with $25\%$ overlap and
1536-point STFT is used. The kernel size and stride are $(5,2)$ and $(2,1)$.
The channel numbers of the encoder and decoder are
$\left\\{12,24,48,64,96,96\right\\}$ and the channel number of last decoder is
6 ($\text{C}_{\text{A}}=4$). The hidden cell of the HBM is 256 and 2-layer GRU
is used. HGCN+ is trained for 24 epochs on 3500 hours of data with a learning
rate of 0.000125. The audio speed is adjusted in 0.9~1.1 during training. The
number of parameter (Para.) is 5.29 M.

###  3.3 Experimental results and discussion

The Real-Time-Factor (RTF) of WB and FB models both are tested on a machine
with an Intel(R) Core(TM) i5-6200U<EMAIL_ADDRESS>GHz in a single thread
(implemented by ONNX).

Table 1: System comparison on the wide-band test set. Model |  Para. (M) | RTF | PESQ-WB | STOI(%)
---|---|---|---|---
-5dB | 0dB | 5dB | AVG | -5dB | 0dB | 5dB | AVG
Noisy | - | - | 1.11 | 1.20 | 1.37 | 1.23 | 72.6 | 81.2 | 88.2 | 80.7
HGCN | 0.93 | 0.11 | 1.59 | 1.95 | 2.37 | 1.97 | 80.2 | 88.2 | 93.6 | 87.4
+$\text{CEM}^{+}$ | 3.60 | 0.17 | 1.62 | 2.00 | 2.44 | 2.02 | 82.8 | 90.3 | 94.5 | 89.1
+$\text{GHCM}^{+}$ | 4.12 | 0.16 | 1.62 | 2.01 | 2.47 | 2.03 | 83.0 | 90.3 | 94.5 | 89.2
+$\text{HM}^{+}$ | 4.11 | 0.14 | 1.65 | 2.04 | 2.48 | 2.06 | 83.6 | 90.8 | 94.8 | 89.7

To evaluate the performance of WB models (100 hours training data), two
metrics are utilized, namely PESQ (PESQ-WB, PESQ-NB (narrow-band, 0~4 KHz))
and STOI [24, 25]. The comparison based on WB set is shown in Table 1. It can
be seen that the dual-path encoder and DPRNN in $\text{CEM}^{+}$ bring an
improvement in terms of feature utilization and frequency dependencies
modeling with more arithmetic complexity. $\text{GHCM}^{+}$ improves the
performance because the linear module has a wider receptive field with less
computational complexity than the convolution. In $\text{HM}^{+}$, the
significance-based VRD reduces the computation of CEM, and the introduction of
cosine makes gates more accurate and instructive.

Table 2: System comparison on DNS-2020 synthetic test set. Model | Para.(M) | PESQ-WB | PESQ-NB | STOI(%)
---|---|---|---|---
Noisy | - | 1.58 | 2.45 | 91.5
DCCRN [4] | 3.67 | - | 3.27 | -
GaGNet [7] | 5.94 | 3.17 | 3.56 | 97.1
HGCN+ | 5.29 | 3.19 | 3.65 | 97.2
Table 3: Challenge results for track 1 on DNS-2022. Model | Para.(M) | SIG | BAK | OVRL | WAcc | Final Score
---|---|---|---|---|---|---
NSNet2[2] | 6.17 | 3.62 | 3.93 | 3.26 | 0.63 | 0.60
HGCN+ | 5.29 | 4.01 | 4.55 | 3.81 | 0.65 | 0.68

We evaluate the HGCN+ (3500 hours training data) on the DNS-2020 and the
DNS-2022 test sets. Because HGCN+ processes the HB and WB spectra separately,
and the WB spectrum contains more speech information, the model has good
performance on the WB test set can bring gains to the processing of FB signal,
as shown in Table 2. In DNS-2022, ITU-T P.835 framework [26] and Word Accuracy
(WAcc) are used to evaluate the speech quality, and HGCN+ outperforms the
baseline [2] with less parameters in all metrics, as shown in Table 3. By
audiometric analysis, we also found that non-speech sound, such as breath,
sob, etc. is usually regarded as noise and filtered out, which results in
incomplete human voice but better speech and the BAK MOS which is relatively
larger than SIG MOS. In addition, our method makes a confusing pitch choice in
the case of multi-speaker and degrades the performance also. The RTF of HGCN+
in processing the FB signal is 0.16 and it consumes 5.22 ms per frame. The
frame size and overlap are 32 ms and 25%. Since the model is causal without
looking forward, the latency is $32\text{+}8\text{=}40$ ms, which satisfies
the requirements of the challenge.

##  4 Conclusion

In this paper, we improve each module of our HGCN. First, we model the
harmonic integration by cosine and propose a significance-based VRD to predict
the harmonic locations efficiently. Second, we introduce the dual-path
encoder, DPRNN, and residual linear structure to CEM and GHCM to enhance model
performance. Finally, we add a high-band module to help the model handle the
FB signal, resulting in HGCN+. HGCN+ outperforms the referenced models on the
DNS-2020 and DNS-2022 test sets.

## References

  * [1] K. Tan and D. Wang,  “Complex spectral mapping with a convolutional recurrent network for monaural speech enhancement,”  in IEEE ICASSP, 2019, pp. 6865–6869.
  * [2] S. Braun and I. Tashev,  “Data augmentation and loss normalization for deep noise suppression,”  in International Conference on Speech and Computer. Springer, 2020, pp. 79–86.
  * [3] JM Valin,  “A hybrid DSP/deep learning approach to real-time full-band speech enhancement,”  in IEEE MMSP, 2018, pp. 1–5.
  * [4] Y. Hu, Y. Liu, S. Lv, M. Xing, S. Zhang, Y. Fu, J. Wu, B. Zhang, and L. Xie,  “DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement,”  arXiv preprint arXiv:2008.00264, 2020.
  * [5] S. Lv, Y. Fu, M. Xing, J. Sun, L. Xie, J. Huang, Y. Wang, and T. Yu,  “S-dccrn: Super wide band dccrn with learnable complex feature for speech enhancement,”  arXiv preprint arXiv:2111.08387, 2021.
  * [6] H. Schröter, T. Rosenkranz, A. Maier, et al.,  “DeepFilterNet: A low complexity speech enhancement framework for full-band audio based on deep filtering,”  arXiv preprint arXiv:2110.05588, 2021.
  * [7] A. Li, C. Zheng, L. Zhang, and X. Li,  “Glance and Gaze: A collaborative learning framework for single-channel speech enhancement,”  arXiv preprint arXiv:2106.11789, 2021.
  * [8] D. Yin, C. Luo, Z. Xiong, and W. Zeng,  “PHASEN: A phase-and-harmonics-aware speech enhancement network,”  in AAAI, 2020, vol. 34, pp. 9458–9465.
  * [9] T. Wang, W. Zhu, Y. Gao, J. Feng, and S. Zhang,  “HGCN: Harmonic gated compensation network for speech enhancement,”  arXiv preprint arXiv:2201.12755, 2022.
  * [10] T. Wang and W. Zhu,  “A deep learning loss function based on auditory power compression for speech enhancement,”  arXiv preprint arXiv:2108.11877, 2021.
  * [11] N. Takahashi and Y. Mitsufuji,  “Multi-scale multi-band densenets for audio source separation,”  in IEEE WASPAA, 2017, pp. 21–25.
  * [12] X. Le, H. Chen, K. Chen, and J. Lu,  “DPCRN: Dual-path convolution recurrent network for single channel speech enhancement,”  in ISCA INTERSPEECH, 2021, pp. 2811–2815.
  * [13] Y. Luo, Z. Chen, and T. Yoshioka,  “Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation,”  in IEEE ICASSP, 2020, pp. 46–50.
  * [14] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio,  “Learning phrase representations using RNN encoder-decoder for statistical machine translation,”  arXiv preprint arXiv:1406.1078, 2014.
  * [15] X. Glorot, A. Bordes, and Y. Bengio,  “Deep sparse rectifier neural networks,”  in Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2011, pp. 315–323.
  * [16] S. Ioffe and C. Szegedy,  “Batch Normalization: Accelerating deep network training by reducing internal covariate shift,”  in IMLS ICML, 2015, vol. 1, pp. 448–456.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun,  “Delving Deep into Rectifiers: Surpassing human-level performance on imageNet classification,”  in IEEE ICCV, 2015, pp. 1026–1034.
  * [18] A. Camacho,  SWIPE: A sawtooth waveform inspired pitch estimator for speech and music,  University of Florida Gainesville, 2007.
  * [19] M. R Schroeder,  “Period histogram and product spectrum: New methods for fundamental-frequency measurement,”  The Journal of the Acoustical Society of America, vol. 43, no. 4, pp. 829–834, 1968.
  * [20] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár,  “Focal loss for dense object detection,”  in IEEE ICCV, 2017, pp. 2980–2988.
  * [21] C. K. A. Reddy, E. Beyrami, H. Dubey, V. Gopal, R. Cheng, R. Cutler, S. Matusevych, R. Aichner, A. Aazami, S. Braun, P. Rana, S. Srinivasan, and J. Gehrke,  “The interspeech 2020 deep noise suppression challenge: Datasets, subjective speech quality and testing framework,”  arXiv preprint arXiv:2001.08662, 2020.
  * [22] R. Cutler A. Aazami S. Matusevych S. Braun S. Eskimez M. Thakker T. Yoshioka H. Gamper R. Aichner H. Dubey, V. Gopal,  “ICASSP 2022 Deep Noise Suppression Challenge,”  in IEEE ICASSP, 2022, pp. 46–50.
  * [23] D. P. Kingma and J. L. Ba,  “Adam: A method for stochastic optimization,”  in ICLR, 2015.
  * [24] A. W Rix, M. P Hollier, A. P Hekstra, and J. G Beerends,  “Perceptual evaluation of speech quality (PESQ) the new itu standard for end-to-end speech quality assessment part i–time-delay compensation,”  Journal of the Audio Engineering Society, vol. 50, no. 10, pp. 755–764, 2002.
  * [25] C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen,  “An algorithm for intelligibility prediction of time–frequency weighted noisy speech,”  IEEE TASLP, vol. 19, no. 7, pp. 2125–2136, 2011.
  * [26] B. Naderi and R. Cutler,  “A crowdsourcing extension of the ITU-T recommendation P.835 with validation,”  arXiv e-prints, pp. arXiv–2010, 2020.