# JetSeg: Efficient Real-Time Semantic Segmentation Model for Low-Power GPU-
Embedded Systems

Miguel Lopez-Montiel1, Daniel Alejandro Lopez1, Oscal Montiel$*$, 1
<EMAIL_ADDRESS><EMAIL_ADDRESS>
∗Corresponding author<EMAIL_ADDRESS>
Department of Intelligent Systems
Instituto Politécnico Nacional, CITEDI-IPN1
Av. Instituto Politécnico Nacional No. 1310, Nueva Tijuana C.P. 22435

###### Abstract

Real-time semantic segmentation is a challenging task that requires high-
accuracy models with low-inference times. Implementing these models on
embedded systems is limited by hardware capability and memory usage, which
produces bottlenecks. We propose an efficient model for real-time semantic
segmentation called JetSeg, consisting of an encoder called JetNet, and an
improved RegSeg decoder. The JetNet is designed for GPU-Embedded Systems and
includes two main components: a new light-weight efficient block called
JetBlock, that reduces the number of parameters minimizing memory usage and
inference time without sacrificing accuracy; a new strategy that involves the
combination of asymmetric and non-asymmetric convolutions with depthwise-
dilated convolutions called JetConv, a channel shuffle operation, light-weight
activation functions, and a convenient number of group convolutions for
embedded systems, and an innovative loss function named JetLoss, which
integrates the Precision, Recall, and IoUB losses to improve semantic
segmentation and reduce computational complexity. Experiments demonstrate that
JetSeg is much faster on workstation devices and more suitable for Low-Power
GPU-Embedded Systems than existing state-of-the-art models for real-time
semantic segmentation. Our approach outperforms state-of-the-art real-time
encoder-decoder models by reducing 46.70M parameters and 5.14% GFLOPs, which
makes JetSeg up to 2x faster on the NVIDIA Titan RTX GPU and the Jetson Xavier
than other models. The JetSeg code is available at
https://github.com/mmontielpz/jetseg.

_Keywords_ Semantic Segmentation $\cdot$ Real-Time $\cdot$ NVIDIA Jetson,
$\cdot$ Embedded Systems, Deep Learning, Camvid

##  1 Introduction

Semantic segmentation is one of the critical computer vision tasks for
autonomous systems. It requires a complete understanding of the analyzed scene
to recognize and divide each object comprising the image, video, or 3D data.
Semantic segmentation aims to classify each pixel into its corresponding class
label to understand the image’s objects. The significance of a scene depends
heavily on the destined application that may range from topics such as human-
machine interaction [1], medical imaging [2], and autonomous driving methods
[3]. For instance, systems directed toward autonomous vehicles must be able to
segment roads and detect objects and traffic signs, which must operate under
real-time conditions and, often, limited computing capabilities.

Since the mid-2010s, most semantic segmentation methods have used some
variation of deep convolutional neural networks. Early networks such as fully
convolutional networks [4], SegNet [5], and U-Net [2] revolutionized the
capabilities of segmentation research. Nonetheless, these methods require high
computational resources, which is unsuitable for real-time embedded system
applications.

In robotics, real-time semantic segmentation must reach breakneck running
speeds for quick interaction and responses to aid decision-making. One of the
earliest networks for real-time semantic segmentation was ENet [6], which
implemented a bottleneck module in a residual layer that reduced it to up to
1% of SegNet [5] parameters.

Other proposals rely on pyramid feature extraction for real-time segmentation,
such as FPENet [7] that produces an efficient feature pyramid for multi-scale
context encoding, ESPNet that adopts efficient spatial pyramid dilated
convolutions [8], and DFANet [9] that mixes multibranch and spatial pyramid
pooling, and reduces complexity by reusing features enhancing feature
representation.

Although these methods can achieve real-time inference speed, the significant
loss of features hinders their performance considerably, sacrificing accuracy
for efficiency. Due to this, few works have attempted to obtain a favorable
trade-off between high-quality results and low-inference times for embedded
system applications. One of these models is ADSCNet [10], a lightweight
Asymmetric depthwise separable convolution network that connects sets of
dilated convolutional layers using dense dilated convolution connections; this
model achieves an mIoU of 67.5% on the Cityscapes dataset at 76.9 FPS while
performing 21.1 MFLOPS and having around 20K parameters while running on a GTX
1080 Ti GPU. Similarly, ESPNet V2 [11] implements group pointwise and depth-
wise dilated separable convolutions for representation learning.
Experimentation was carried out on a GTX 1080 Ti GPU and an NVIDIA Jetson TX2,
reaching an mIoU of 66.2-66.4% by performing 2.7 B FLOPs for the Cityscapes
dataset and 67-68% mIoU with 0.76 B FLOPs respectively. In recent work, MFENet
[12] uses modules for Spatial and Edge Extraction with a Laplace Operator to
improve low-level feature extraction, a Context Boost Module for high-level
features, and a Selective Refinement Module that combines their information.
This network obtains an mIoU of 76.7% on the Cityscape dataset performing 12.5
GLOPs at a speed of 47 FPS, and 73.5% on the CamVid dataset when executed on
an NVIDIA Titan Xp card. DSANet [13], a two-branch model that employs channel
split and shuffle to reduce computation and attain high accuracy. Its dual
attention and channel attention modules help with pixel-wise label prediction
using multi-level feature maps simultaneously. The experiments on a GTX 1080
GPU for the CamVid and Cityscapes datasets achieved mIoU of 69.9% and 71% at
speeds of 75.4 FPS and 34.04 FPS, respectively. MiniNetv2 [14] was developed
for low-power systems, improving the MiniNet efficient real-time semantic
segmentation. It attains an mIoU of 70.4% performing 12.89 GFLOPs and 6.45
GMACs at 50 FPS, having 0.52 M parameters and a memory footprint of 2.02 MB
for a resolution of 1024 x 512 of the CityScapes benchmark while running on an
NVIDIA Titan Xp card. Furthermore, several works regarding embedded system
implementation show promising results when compared to workstation devices,
such as ThunderNet [15] is an efficient and lightweight network with a minimal
ResNet18 backbone that unifies PSPNet pyramid pooling and a decoding phase
that achieves an mIoU of 64% at 96.2 FPS on a Titan Xp GPU and 20.9 FPS on a
Jetson TX2. Another instance is WFDCNet, a network of full-dimensional
continuous separation convolution modules (FCS) and lateral asymmetric pyramid
fusion modules (LAPF) that enable it to attain high accuracy without hindering
inference speed, reaching 73.7% of mIoU having 5.7 GFLOPs at 102.6 FPS and 0.5
M parameters on an RTX 2080 Ti GPU for the Cityscapes dataset while achieving
a speed of 17.2 FPS on a Jetson TX2. EFNet [16], a modified version of
ShuffleNet V2 , aims to further improve accuracy-speed trade-off by reducing
its downsampling operations, stride, and deconvolutions. Two thinner
convolutions replace its large convolution channel, and the final downsampling
and upsampling channels are substituted by more elaborate modules that boost
efficiency. As a result, it reaches an mIoU of 68% at 99 FPS and 0.18 M
parameters on an RTX 3090 GPU for the Cityscapes dataset. However, when run on
a Jetson Nano, the model achieves similar accuracy results but significantly
lower inference speeds of 0.9 FPS. Nevertheless, it was tested on a novel
memristor-based computing-in-memory accelerator to demonstrate its efficiency
on embedded systems further, obtaining 40.8 FPS and comparable accuracy
results.

Additionally to the trade-off between speed and accuracy of current real-time
semantic-segmentation models, there is limited research on execution time and
memory footprint for real-time models. Therefore, due to the necessities
mentioned above, our work focused on developing an efficient model capable of
achieving real-time semantic segmentation in low-power embedded devices;
specifically, we contribute to state-of-the-art with the following:

  * •

We introduce a novel, light, and efficient architecture called JetSeg that can
compete with state-of-the-art faster and bigger models, outperforming them in
most cases. JetSeg contains the novel encoder JetNet, the novel block
JetBlock, the novel operation JetConv, and features an original loss function
named JetLoss, which contributions will be explained.

  * •

JetConv is a novel convolutional operator designed for lightweight and
efficient feature extraction in real-time, low-power embedded systems. It
enhances spatial information representation, increases the receptive field,
and captures long-range dependencies without adding parameters. In addition,
by combining dilated asymmetric and non-asymmetric convolutions, JetConv
maintains spatial symmetry and enables balanced feature extraction in all
directions.

  * •

JetNet is an encoder that can extract information without compromising speed.
It consists of convolutional blocks comprised of JetConvs. By combining
dilated asymmetric depthwise convolutions and non-asymmetric traditional
convolutions, the encoder can acquire large amounts of context information on
fine and large dependencies without drastically increasing the number of
parameters.

  * •

JetBlock is an efficient unit for feature map extraction, achieving a balance
between inference time, memory usage, and model abstraction. It consists of
multiple stages for optimal performance, estimates the number of group
convolutions to prevent overfitting, and improves generalization. In addition,
batch normalization and lightweight activation functions like TanhExp maintain
representation while minimizing inference time.

  * •

JetLoss is a novel loss function for improved semantic segmentation. It
combines Precision, Recall, and IoUB losses, enhancing overall performance.
Adaptive weightings based on pixel counts per class are included to further
enhance effectiveness by focusing on challenging classes. As a result, JetLoss
enables faster convergence, efficient training, and reduced computational
requirements. Experimental results on the CamVid [17] benchmark dataset show
its edge over traditional loss functions, positioning JetLoss as a strong
competitor in semantic segmentation.

##  2 JetSeg

According to the revised related works, most real-time semantic segmentation
models are unsuitable for low-power embedded devices. Therefore, our work was
to design an efficient and lightweight model capable of running on resource-
constrained devices, such as the NVIDIA Jetson Nano and NVIDIA Jetson AGX, due
to most embedded devices possessing GPU as hardware accelerators [18].

Our proposed model JetSeg, see Figure 1, is an efficient real-time semantic
segmentation model for low-power GPU embedded systems, specifically, NVIDIA
Jetson devices. This model comprises two architectures: JetNet, our proposed
encoder, and an enhanced version of the RegSeg [19] decoder; hence the name
JetSeg for our model. JetSeg was designed for memory and operation efficiency
while preserving a competitive mIoU and FPS to the state-of-the-art (SOTA)
models for low-power real-time semantic segmentation without model
optimization or compression.

Figure 1: JetSeg framework: Our proposed JetSeg follows the encoder-decoder
architecture. The encoder component is JetNet, while the decoder is based on
RegSeg.

The JetSeg model has three configurations, one for the workstation, one for
the NVIDIA Jetson AGX, and one for the NVIDIA Jetson Nano. Input features are
first expanded and later reduced when nearing the output layers, which are
processed in different phases depending on the stage they are in. Four cases
dictate the stage and operations applied to the input features. S0 and S1
focus on high-level feature extraction, using three main operations: channel
shuffle to reduce the number of parameters; JetConv, which captures low and
high-level features without increasing model complexity; and a convolutional
block (CBAM) [20] that enables the model to learn more discriminative and
robust features. S0 also includes residual layers just as S2; however, S2
replaces the CBAM with the SAM [20], which allows the model to capture long-
range dependencies and enhances spatial awareness. S3 presents half the
residual layers compared to S2 and also includes an Efficient Channel
Attention Module [21] that highlights relevant channel-specific information.

###  2.1 JetConv

The JetConv novel operator is for lightweight and efficient feature extraction
in real-time low-power embedded systems. It was inspired by the EESP
(Extremely Efficient Spatial Pyramid) module [11]. The dilated Depthwise
Convolutions have been improved by introducing Combinated Dilated Depthwise
Convolution (CDDC), which combines dilated asymmetric convolution followed by
a non-asymmetric and non-dilated convolution. This method enables a more
effective and balanced spatial information representation. The receptive field
is also significantly increased, which allows capturing long-range
dependencies without increasing the total number of parameters and balancing
between power and computational efficiency. Asymmetric convolution limitations
are compensated through non-asymmetric convolutions, which preserve spatial
symmetry and allow equal feature extraction in all directions. This symmetry
is beneficial when dealing with objects lacking directional properties and
ensures balanced information among spatial dimensions. JetConv improves
contextual understanding without sacrificing spatial symmetry or robustness by
integrating the two types of convolutions, allowing it to capture long-range
dependencies and global spatial patterns critical for semantic segmentation.
Figure 2 shows the structure of the JetConv.

Figure 2: JetConv is based on Standard/non-Asymmetric Dilated Convolution by
the three levels that comprise JetBlock.

This balanced approach benefits objects without directional properties,
ensuring comprehensive spatial information. JetConv achieves improved
contextual understanding, capturing critical long-range dependencies and
global spatial patterns for semantics while preserving spatial symmetry and
robustness. Figure 2 is a residual block composed by five layers. In the first
layer, there are three CDDConv blocks; each CCDConv block is composed of two
different convolution operations. The first JetConv configuration consists of
a depthwise convolution (DConv) [22], ADDConv (Asymmetric Dilated Depthwise
Convolution) [23], and the novel convolution operation name ADDConv; they can
configure different size kernels with and without dilatation rate, they are:
$3\times 3$, $5\times 5$ and $7\times 7$ for non-asymmetric, and for dilated
asymmetric ($3\times 1$ \+ $1\times 3$), ($5\times 1$ \+ $1\times 5$) and
($7\times 1$ \+ $1\times 7$). The second is an additive layer that adds the
features obtain in the first layer, depending on the number of CDDConv blocks
we have. The objective of the third layer is to concatenate the result of the
second layer, i.e., we stack the obtained results of each CCDConv block to
combine the feature extraction of each configuration of CCDConv. The GPConv in
the fourth layer red reduces the overfitting, memory usage, and inference
time. The objective of the last layer is to perform feedback of the GPConv
result to the first layer to complete the residual calculus. The JetConv block
allows three basic modifiable configurations: a) the first one contains one
CDDConv block in the first, layer followed by the four layers shown in Figure
2, b) the second configuration adds a CCDConv block to the first
configuration, c) the third one adds a third CCDConv block the second
configuration.

###  2.2 JetBlock

JetBlock is a new efficient unit for feature map extraction comprised of
several stages that allow it to achieve an equilibrium between inference time,
memory usage, and model abstraction. First, the convenient number of group
convolutions is estimated, preventing overfitting and improving model
generalization. Group convolutions are followed by batch normalization and
activation functions suited for lightweight architectures, being TanhExp used
to reduce inference time while maintaining capture and representation. The
second stage focuses on computational efficiency through the channel shuffle
operator, which optimizes memory access patterns, enhances hardware resource
management, promotes feature diversity through layer interconnections, and
speeds up computation. The third stage extracts high-level features using the
JetConv layer and distinguishes between the important elements with the
attention module layer. Afterward, the block finishes with an activation
function and a grouped pointwise operator, reducing the computational cost and
memory requirements. The composition of the JetBlocks is defined in Figure 3.

Figure 3: Overview of JetNet Blocks: (a) Input block, and (b) Subsequent
blocks that comprise JetNet.

###  2.3 JetNet

JetNet is an encoder architecture that heavily relies on a set of efficient
convolutional blocks called JetConvs. It has four stages as S0 to S3, as shown
in Figure 1. The Stage S0 is composed of one configurable JetBlock followed by
a Batch Normalization (BN) and an Activation Function REU (Rectified
Exponential Units) [24] as shown in 3(a). The Stage S1 is a residual block
composed of four configurable JetBlocks as shown in 2(b); in this stage, the
four blocks are similar in structure. Its first layer is composed of a GPConv
of $1\times 1$ that expands the number of input channels in order to extract
more features, followed by BN, REU, Channel Shuffle (CS), JetConv, Attention
Module (AM), REU and $1\times 1$ GPConv in order to reduce the number of
features. Note that the AM layer is a CBAM module. Stage S2 is similar to
stage S1; the difference is the configuration, particularly in the AM module,
and the number of features used. We used only the spatial attention module in
the AM module because the image size was reduced. Stage S3 is similar to stage
S2; the block of this stage where reduced to improve the processing speed of
JetNet. In this stage, the attention module was also changed; it used ECAM
instead of SAM.

The main goal of the JetConv layer is to extract feature maps with as few
operations as possible. The JetConvs are the main blocks for the two JetNet
block variations; see Figure 3. The input block consists of a JetConv, a batch
normalization layer, and an activation function. The subsequent stages of
JetSeg are more intricate. The beginning and end of these blocks are similar
to the input block; there, we replace the JetConv with a $1\times 1$
convolution to reduce computational complexity. After that, we continue with
channel shuffle as proposed in [25] followed by another JetConv, then an
attention module, an activation function, and we finish by a $1\times 1$
convolution. The attention module can be any of the following: a spatial
attention module (SAM) [20], a CBAM, or an efficient channel attention module
(ECAM) [19]. Finally, we use a skip connection to concatenate the residual for
each block.

###  2.4 JetLoss

The JetLoss function is a novel loss function inspired by the work of Tian
[26], designed to enhance model performance in semantic segmentation. JetLoss
integrates three loss functions into one, combining elements from Precision
[27], Recall [26], and IoUB [28] losses to improve the overall performance,
particularly in terms of recall.

To further improve the effectiveness of JetLoss, adaptive weightings based on
the number of pixels per class, following Cui’s approach [29], are
incorporated. These adaptive weightings assign importance to each class during
training, enabling the model to focus on challenging classes and achieve
better overall performance. As a result, using JetLoss promotes faster model
convergence, efficient training, and reduced computational resource
requirements. Moreover, it ensures a fair evaluation of the semantic
segmentation model’s performance.

The experimental results on the CamVid [17] benchmark dataset demonstrate the
advantages of using JetLoss compared to conventional loss functions,
highlighting its potential as a robust contender in semantic segmentation loss
function proposals.

The components of the JetLoss function are described by the following set of
equations LABEL:eq:jetloss:

$\displaystyle R_{loss}$
$\displaystyle=\sum_{c=1}^{c}\frac{TP_{c}+\epsilon}{TP_{c}+FN_{c}+\epsilon}\times
W_{c}$ $\displaystyle P_{loss}$
$\displaystyle=\sum_{c=1}^{c}\frac{TP_{c}+\epsilon}{TP_{c}+FP_{c}+\epsilon}\times
W_{c}$ $\displaystyle B_{loss}$
$\displaystyle=\frac{{\sum_{i=1}^{N}TP_{b_{i}}}}{{\sum_{i=1}^{N}(TP_{b_{i}}+FP_{b_{i}}+FN_{b_{i}})}}\times
W_{c}$ $\displaystyle JetLoss$ $\displaystyle=R_{loss}+P_{loss}+B_{loss}$

In the equations above, $R_{loss}$ represents the recall loss, $P_{loss}$
represents the precision loss, and $B_{loss}$ represents the IoUB loss.
$TP_{c}$, $FN_{c}$, $FP_{c}$, and $W_{c}$ denote true positives, false
negatives, false positives, and class-specific weights, respectively. The
overall JetLoss is computed as the sum of the three component losses.

##  3 Experiments and results

The following metrics are used to evaluate JetSeg and are generally employed
to gauge the performance of real-time semantic segmentation models. The
computational complexity is essential when working with limited hardware
resources; hence the FLOPs metric is fundamental for evaluating a CNN model.
It measures the total number of arithmetic computations performed in a
process, emphasizing additions and multiplications. According to authors such
as [30], several factors play a part in FLOPs, for instance, the expense in
the computation of convolutions ($F_{conv}$), fully connected layers
($F_{fc}$), activation functions ($F_{act}$), as well as batch normalizations
($F_{bn}$). Equation 2 shows the total FLOPs formula broken down,

$\displaystyle F_{conv}$ $\displaystyle=\frac{K_{w}\times K_{h}\times
H_{in}\times W_{in}\times C_{in}\times C_{out}}{s^{2}\times g}$ $\displaystyle
F_{pool}$ $\displaystyle=C_{in}\times H_{out}\times W_{out}\times K_{w}\times
K_{h}$ $\displaystyle F_{fc}$ $\displaystyle=C_{in}\times H_{in}\times
W_{in}\times H_{out}\times W_{out}$ $\displaystyle F_{bn}$
$\displaystyle=4\times C_{in}\times H_{out}\times W_{out}$ $\displaystyle
F_{act}$ $\displaystyle=C_{in}\times H_{in}\times W_{in}$ $\displaystyle
FLOPs$ $\displaystyle=F_{conv}+F_{pool}+F_{fc}+F_{act}+F_{bn}$ (2)

where $K_{w}$, $K_{h}$, $H_{in}$, and $W_{in}$ represent the kernel size,
input height, and input width of the convolution layer, respectively; input
and output channels are represented by $C_{in}$ and $C_{out}$; $H_{in}$,
$W_{in}$, $H_{out}$ and $W_{out}$ represent the input height, input width,
output height and output width of the fully connected layer; activation
function cost is represented by $F_{act}$ and computed with the number of
activation functions in the input tensor given by $C_{in}\times H_{in}\times
W_{in}$, as well as the number of operations needed per activation; and
$F_{bn}$ represents batch normalization and is calculated based on the number
of activations in the input tensor, which is given by $C_{in}\times
H_{in}\times W_{in}$. The fundamental metric to evaluate semantic segmentation
performance is the mIoU [28], which refers to the total percentage of pixels
belonging to the ground truth class that the model was able to classify,
meaning the overlapping degree between the ground truth and model prediction.
Equation 3 describes the computation.

$\displaystyle mIoU$ $\displaystyle=\frac{1}{|C|}\sum_{c}\frac{\sum_{i\in
P}TP_{ic}}{\sum_{i\in P}TP_{ic}+FP_{ic}+FN_{ic}}$ (3)

The variables $TP_{c}$, $FP_{c}$, and $FN_{c}$ represent the counts of True
Positive, False Positive, and False Negative pixels, respectively, for class
$c$. The set $D={1,2,...,C}$ refers to all the categories present in the
dataset, while $P$ represents the set of all pixels within the dataset.

This section describes of the experimental setup and results analysis used to
evaluate our proposed JetSeg model. We conducted our experiments using the
widely recognized dataset CamVid, commonly used as a benchmark dataset for
Autonomous Driving Semantic Segmentation. CamVid [17] is a database for
understanding road and driving scenes initially from five videos, which add to
a total of 701 frames divided into 32 classes.

We conducted two phases of experiments: a classification evaluation and a
real-time embedded system implementation. We utilized the CamVid dataset for
both. A split of 67% of the data for training (468 samples) and 33% for
testing (233 samples). Additionally, we divided the training set into 78% (367
samples) for training and 22% (101 samples) for validation.

The hardware used for training and evaluation consisted of an NVIDIA TITAN RTX
GPU. We employed GPU-equipped embedded system boards to evaluate the real-time
embedded implementation: the NVIDIA Jetson Nano and the NVIDIA Jetson AGX. To
develop this work, we utilized a variety of software tools. Firstly, we
employed PyTorch [31], the currently most widely used framework for deep
learning. To ensure the reproducibility of our experiments, we utilized Docker
containers [32] for all our hardware, including both our workstation and
NVIDIA Jetsons.111https://catalog.ngc.nvidia.com/orgs/nvidia/containers/l4t-ml

Table 1: SoTA Results for Real-time Semantic Segmentation on the CamVid dataset (non-Jetson Devices). $\ast$ represents unclear information in some columns. G refers to Giga. Model | Input | Pretrained | Data | Model | Params | FLOPs | FPS
---|---|---|---|---|---|---|---
Size | Aug | Opt | (G) | (G)
$\ast$LinkNet [33] | $640\times 360$ | No | No | No | 11.5 | 21.2 | -
$\ast$CGNet [34] | $480\times 360$ | No | Yes | No | 0.5 | 6 | -
DSANet [13] | $480\times 360$ | No | Yes | No | 3.47 | 37.4 | 75.3
MFENet [12] | Original | Yes | Yes | No | 3.91 | 12.5 | -
JetSeg | $512\times 512$ | No | No | No | 0.00003 | 1.125 | 158
(Ours)

The following section presents our evaluation of the JetSeg model in a real-
time setting, showcasing the model’s efficiency on embedded devices with
limited resources. The evaluation was conducted on NVIDIA Jetson GPU-Embedded
Devices using NGC Docker containers to ensure the reproducibility and
generalizability of the experiments. For Jetson AGX Xavier, we utilized
JetPack version $5.0.2$ and the NGC Docker container
nvcr.io/nvidia/l4t-ml:r35.1.0-py3. For Jetson Nano 4GB, we used JetPack
$4.6.1$ and the NGC Docker container nvcr.io/nvidia/l4t-ml:r32.7.1-py3. These
configurations allowed us to effectively demonstrate the performance of JetSeg
in real-time scenarios with constrained resources.

The experiments are divided into two categories: workstation and embedded
devices. During the training phase, the model is trained from scratch for 15
epochs, without any pre-training, model compression, or data augmentation
techniques. Table 1 presents the performance metrics of the model on the
CamVid dataset with an input size of $512\times 512$. The model achieves a
considerably fast speed of 158 FPS, performing 1.125 G FLOPs with only 0.00003
G parameters. These results highlight that JetSeg is a compact and high-speed
model for semantic segmentation.

When evaluated on embedded devices, the NVIDIA Jetson AGX demonstrates real-
time performance, achieving a speed of 39.9 FPS. Despite its low hardware
requirements, the Jetson AGX performs 1.125 G FLOPs with 0.00003 G parameters,
making it suitable for real-time semantic segmentation in embedded system
applications. Moreover, the model also runs successfully on the NVIDIA Jetson
Nano, showcasing its lightweight nature and capability to operate on low-
memory devices.

##  4 Discussion

This work proposes a novel model called JetSeg for the task of semantic
segmentation for embedded devices in real-time, presenting an innovative,
efficient, and lightweight architecture comprised of an encoder called JetNet,
new model blocks named JetBlocks, original convolution operations called
JetConv, and a new loss function called JetLoss, which allows it to run on
embedded devices such as the NVIDIA Jetson AGX in real-time. This architecture
lets the model extract low and high-level features while reducing
computational complexity, maintaining spatial symmetry and balanced feature
extraction thanks to the JetConv convolutional operator. JetNet combines
dilated asymmetric depthwise convolutions and non-asymmetric traditional
convolutions to extract context information from fine and large dependencies.
JetBlock comprises JetNet and is an efficient unit for feature extraction,
preventing overfitting while minimizing inference time. Lastly, JetLoss
improves semantic segmentation performance by integrating three losses:
Precision, Recall, and IoUB, which enhances overall performance through
adaptive weighting based on pixel counts per class. This enables faster
convergence and efficient training with reduced computational requirements.
JetSeg can be executed in three configurations suited for workstations, such
as the NVIDIA Titan RTX GPU, the NVIDIA Jetson AGX, and NVIDIA Jetson Nano
embedded devices achieving high FPS, such as 148 FPS on the Titan RTX GPU and
39.1 FPS on the Jetson AGX, inference speeds suited for real-time
implementation. The comparative experiment results showed that JetSeg
outperformed the tested state-of-the-art models in inference speed up to
2$\times$ while executing less than 5$\times$ GFLOPs, as well as preserving a
small number of parameters without model optimization, compression techniques,
pretraining, or data augmentation.

## Authorship Note

The contribution for the realization of this paper is as follow: The idea and
project development are attributed to Miguel Angel Lopez Montiel and
collaborators Daniel Lopez Montiel and Oscar Montiel Ross. In the submitted
paper to NeurIPS2023, the authors Yoshio Rubio and Cynthia Olvera were
included to participate in the writing and document review; however, they
could not participate due to time constraints preventing them from fulfilling
these roles; hence, they do not appear in this paper.

## References

  * Oberweger et al. [2015] Markus Oberweger, Paul Wohlhart, and Vincent Lepetit.  Hands deep in deep learning for hand pose estimation.  2 2015.  URL https://arxiv.org/abs/1502.06807v2.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , 9351:234–241, 2015.  ISSN 16113349.  doi:10.1007/978-3-319-24574-4_28/COVER.  URL https://link.springer.com/chapter/10.1007/978-3-319-24574-4_28.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pages 3354–3361, 2012.  ISSN 10636919.  doi:10.1109/CVPR.2012.6248074.
  * Shelhamer et al. [2014] Evan Shelhamer, Jonathan Long, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39:640–651, 11 2014.  ISSN 01628828.  doi:10.1109/TPAMI.2016.2572683.  URL https://arxiv.org/abs/1411.4038v2.
  * Badrinarayanan et al. [2015] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39:2481–2495, 11 2015.  ISSN 01628828.  doi:10.1109/TPAMI.2016.2644615.  URL https://arxiv.org/abs/1511.00561v3.
  * Paszke et al. [2016] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello.  Enet: A deep neural network architecture for real-time semantic segmentation.  6 2016.  URL https://arxiv.org/abs/1606.02147v1.
  * Liu and Yin [2019] Mengyu Liu and Hujun Yin.  Feature pyramid encoding network for real-time semantic segmentation.  _30th British Machine Vision Conference 2019, BMVC 2019_ , 9 2019\.  URL https://arxiv.org/abs/1909.08599v1.
  * Watanabe et al. [2018] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai.  Espnet: End-to-end speech processing toolkit.  _Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH_ , 2018-September:2207–2211, 3 2018.  ISSN 19909772.  doi:10.21437/Interspeech.2018-1456.  URL https://arxiv.org/abs/1804.00015v1.
  * Li et al. [2019a] Hanchao Li, Pengfei Xiong, Haoqiang Fan, and Jian Sun.  Dfanet: Deep feature aggregation for real-time semantic segmentation.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2019-June:9514–9523, 4 2019a.  ISSN 10636919.  doi:10.1109/CVPR.2019.00975.  URL https://arxiv.org/abs/1904.02216v1.
  * Wang et al. [2020a] Jiawei Wang, Hongyun Xiong, Haibo Wang, and Xiaohong Nian.  Adscnet: asymmetric depthwise separable convolution for semantic segmentation in real-time.  _Applied Intelligence_ , 50:1045–1056, 4 2020a.  ISSN 15737497.  doi:10.1007/S10489-019-01587-1/TABLES/7.  URL https://link.springer.com/article/10.1007/s10489-019-01587-1.
  * Mehta et al. [2018] Sachin Mehta, Mohammad Rastegari, Linda Shapiro, and Hannaneh Hajishirzi.  Espnetv2: A light-weight, power efficient, and general purpose convolutional neural network.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2019-June:9182–9192, 11 2018.  ISSN 10636919.  doi:10.1109/CVPR.2019.00941.  URL https://arxiv.org/abs/1811.11431v3.
  * Zhang et al. [2020] Boxiang Zhang, Wenhui Li, Yuming Hui, Jiayun Liu, and Yuanyuan Guan.  Mfenet: Multi-level feature enhancement network for real-time semantic segmentation.  _Neurocomputing_ , 393:54–65, 6 2020.  ISSN 0925-2312.  doi:10.1016/J.NEUCOM.2020.02.019.
  * Elhassan et al. [2021] Mohammed A.M. Elhassan, Chenxi Huang, Chenhui Yang, and Tewodros Legesse Munea.  Dsanet: Dilated spatial attention for real-time semantic segmentation in urban street scenes.  _Expert Systems with Applications_ , 183:115090, 11 2021\.  ISSN 0957-4174.  doi:10.1016/J.ESWA.2021.115090.
  * Alonso et al. [2020] Inigo Alonso, Luis Riazuelo, and Ana C. Murillo.  Mininet: An efficient semantic segmentation convnet for real-time robotic applications.  _IEEE Transactions on Robotics_ , 36:1340–1347, 8 2020\.  ISSN 19410468.  doi:10.1109/TRO.2020.2974099.
  * Xiang et al. [2019] Wei Xiang, Hongda Mao, and Vassilis Athitsos.  Thundernet: A turbo unified network for real-time semantic segmentation.  _Proceedings - 2019 IEEE Winter Conference on Applications of Computer Vision, WACV 2019_ , pages 1789–1796, 3 2019.  doi:10.1109/WACV.2019.00195.
  * Dong et al. [2023] Shuai Dong, Zhen Fan, Yihong Chen, Kaihui Chen, Minghui Qin, Min Zeng, Xubing Lu, Guofu Zhou, Xingsen Gao, and Jun Ming Liu.  Performance estimation for the memristor-based computing-in-memory implementation of extremely factorized network for real-time and low-power semantic segmentation.  _Neural Networks_ , 160:202–215, 3 2023.  ISSN 0893-6080.  doi:10.1016/J.NEUNET.2023.01.008.
  * Brostow et al. [2008] Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla.  Segmentation and recognition using structure from motion point clouds.  In _ECCV (1)_ , pages 44–57, 2008.
  * Shuvo et al. [2023] Md. Maruf Hossain Shuvo, Syed Kamrul Islam, Jianlin Cheng, and Bashir I Morshed.  Efficient acceleration of deep learning inference on resource-constrained edge devices: A review.  _Proceedings of the IEEE_ , 111:42–91, 2023.  doi:10.1109/JPROC.2022.3226481.
  * Gao [2021] Roland Gao.  Rethink dilated convolution for real-time semantic segmentation, 2021\.
  * Woo et al. [2018] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module, 2018.
  * Wang et al. [2020b] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu.  Eca-net: Efficient channel attention for deep convolutional neural networks, 2020b.
  * Chollet [2016] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  _Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017_ , 2017-January:1800–1807, 10 2016.  doi:10.1109/CVPR.2017.195.  URL https://arxiv.org/abs/1610.02357v3.
  * Li et al. [2019b] Gen Li, Inyoung Yun, Jonghyun Kim, and Joongkyu Kim.  Dabnet: Depth-wise asymmetric bottleneck for real-time semantic segmentation.  _30th British Machine Vision Conference 2019, BMVC 2019_ , 7 2019b.  URL https://arxiv.org/abs/1907.11357v2.
  * Ying et al. [2019] Yao Ying, Jianlin Su, Peng Shan, Ligang Miao, Xiaolian Wang, and Silong Peng.  Rectified exponential units for convolutional neural networks.  _IEEE Access_ , 7:101633–101640, 2019.  ISSN 21693536.  doi:10.1109/ACCESS.2019.2928442.
  * Zhang et al. [2017] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices, 2017.
  * Tian et al. [2021] Junjiao Tian, Niluthpol Chowdhury Mithun, Zachary Seymour, Han Pang Chiu, and Zsolt Kira.  Striking the right balance: Recall loss for semantic segmentation.  _Proceedings - IEEE International Conference on Robotics and Automation_ , pages 5063–5069, 6 2021.  ISSN 10504729.  doi:10.1109/ICRA46639.2022.9811702.  URL https://arxiv.org/abs/2106.14917v2.
  * Flach and Kull [2015] Peter Flach and Meelis Kull.  Precision-recall-gain curves: Pr analysis done right.  volume 28. Curran Associates, Inc., 2015.  URL https://proceedings.neurips.cc/paper_files/paper/2015/file/33e8075e9970de0cfea955afd4644bb2-Paper.pdf.
  * Huang et al. [2020] Yifeng Huang, Zhirong Tang, Dan Chen, Kaixiong Su, and Chengbin Chen.  Batching soft iou for training semantic segmentation networks.  _IEEE Signal Processing Letters_ , 27:66–70, 2020.  ISSN 15582361.  doi:10.1109/LSP.2019.2956367.
  * Cui et al. [2019] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  pages 9260–9269, 2019.  doi:10.1109/CVPR.2019.00949.
  * Wang et al. [2021] Xiaojuan Wang, Xinlei Wang, Tianqi Lv, Lei Jin, and Mingshu He.  Harnas: Human activity recognition based on automatic neural architecture search using evolutionary algorithms.  _Sensors (Basel, Switzerland)_ , 21, 10 2021.  ISSN 1424-8220.  doi:10.3390/S21206927.  URL https://pubmed.ncbi.nlm.nih.gov/34696140/.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , 2019.
  * Merkel [2014] Dirk Merkel.  Docker: Lightweight linux containers for consistent development and deployment.  _Linux J._ , 2014, 3 2014.  ISSN 1075-3583.
  * Chaurasia and Culurciello [2018] Abhishek Chaurasia and Eugenio Culurciello.  Linknet: Exploiting encoder representations for efficient semantic segmentation.  _2017 IEEE Visual Communications and Image Processing, VCIP 2017_ , 2018-January:1–4, 2 2018.  doi:10.1109/VCIP.2017.8305148.
  * Wu et al. [2018] Tianyi Wu, Sheng Tang, Rui Zhang, Juan Cao, and Yongdong Zhang.  Cgnet: A light-weight context guided network for semantic segmentation.  _IEEE Transactions on Image Processing_ , 30:1169–1179, 11 2018.  ISSN 19410042.  doi:10.1109/TIP.2020.3042065.  URL https://arxiv.org/abs/1811.08201v2.


# Efficient dynamic filter for robust and low computational feature extraction

###### Abstract

The unseen noise signal is difficult to anticipate, and various approaches
have been developed to address this issue. In our earlier work, we proposed a
lightweight dynamic filter by splitting the filter into kernel and spatial
parts. This small footprint model showed robust results in an unseen noisy
environment. However, a simple pooling process for dividing the feature would
limit the performance. In this paper, we propose an efficient dynamic filter
to enhance the performance of the existing dynamic filter. Instead of the
simple feature mean, we separate the input features as non-overlapping chunks,
and separable convolutions take place for each feature direction. We also
propose a dynamic filter based attention pooling method. These methods are
applied to the kernel part in our previous work, and experiments are carried
out for keyword spotting and speaker verification. We confirm that our
proposed method performs better in unseen environments than the recently
developed models.

Index Terms—  dynamic filter, keyword spotting, speaker verification

##  1 Introduction

††footnotetext: This work was supported by Korea Environment Industry &
Technology Institute(KEITI) through Exotic Invasive Species Management
Program, funded by Korea Ministry of Environment(MOE) (2021002280004).
Corresponding Author:Hanseok Ko.

In audio-based deep learning applications, noise disturbance would occur
performance degradation since the noise signal may shift the domain of target
signals. In particular, unseen noise is difficult to anticipate and various
methods have been developed to mitigate its effects. Through adversarial loss
[1, 2, 3] and speech quality metric learning [4, 5, 6], a generative model
enhances the speech signal against noise. In addition, a data augmentation [7]
would help to enhance the speech signal and classification-driven approaches
[8, 9] would mitigate noisy signals through an end-to-end fashion. Assuming
that noise signals are cumbersome features for the classification task, a T-F
mask is estimated from an input T-F feature and it is recursively multiplied
with the input T-F feature to mitigate the noise effect. Although these
methods perform reasonably well in various applications, their implementations
are computationally expensive. In our previous work, we proposed a lightweight
dynamic filter [10] as an alternative to robust and low resource keyword
spotting. The filter was divided into two subtypes: Pixel Dynamic Filter (PDF)
and Instance-Level Dynamic Filter (IDF). For PDF, a single channel convolution
layer strides T-F features and the output of the convolution is considered
pixel-level scalar weights. In IDF, weight vectors are extracted by using
global averaging pooling and Fully Connected (FC) layers. These learning
methods reduce model complexity and improve Keyword Spotting (KWS) performance
in various noisy environments.

Although the feature averaging process would relieve the computational cost, a
simple feature averaging (e.g. feature mean) might be difficult for capturing
task salient features and it would degrade the performance of dynamic weights.
It would be particularly weak in noisy environments that are unseen. To compat
this issue, this paper propose Chunk Separated Convolution (CS-Conv) and
Dynamic Attention Pooling (DAP) as effective dynamic filters that are robust
against unseen signals. Based on the Dual-Path RNN framework [11], the T-F
features are split into non-overlapped chunk features, and separable
convolutions are performed on intra chunks and inter chunks, respectively.
Additionally, our proposed DAP maps the output feature of CS-Conv to a
temporal saliency-based embedding vector. A convolution layer in DAP produces
vector weights that are used to compute attention scores depending on temporal
frames. This process enables the dynamic filter to focus on temporally salient
features to obtain the embedding vector. The proposed method is applied to IDF
of our previous work and experiments are carried out on Speaker Verification
(SV) and KWS tasks. For KWS and SV evaluations, we use Speech command data
[12] and voxceleb data [13, 14]. We confirm that our proposed method enhances
the performance of KWS and SV in unseen environments (unseen noise and unseen
speakers) over our previous work with similar computation costs.

##  2 Related Works

Fig. 1: The pipeline of dynamic filter network:$F(X,W)$ denotes neural network
implementation with an input feature ($X$) and weight vector ($W$). $W_{fg}$
denotes the weight of the filter generator model. $W_{x}$ denotes the output
of the filter generator which is utilized as weights in the dynamic layer.

Dynamic Filter Network [15] (DFN) is a filter adaptation method that
dynamically modifies the filter values based on the input feature, and the
process is depicted in Fig 1. Instead of optimizing the initialized weight
vector, output features of the neural networks (filter generator) are used as
weight vectors of the main task model (dynamic layer). In this learning
process, weight vectors are produced by the manner of the filter generator and
the weights are independently generated by each instance basis. This property
would enhance the flexibility, where the weights would not be frozen, and it
has shown reasonably acceptable performance in various deep learning
applications [16, 9, 17]. However, the implementation of DFN requires a high
computational cost (time and space complexity) when the input example has a
lot of bases (time step or image patch) and the size of the input feature is
usually large in modern deep learning applications. To address this issue, a
small footprint dynamic model has been proposed. Instead of producing dynamic
weights for each patch unit, decoupled dynamic filter [18] produces the
weights as channel and spatial (kernel) units separately by splitting the
filter generation network as two branches (channel and spatial parts). Based
on the Global Average Pooling (GAP), the input features are divided into
channel and spatial, and the dynamic weights are produced in two different
models, respectively. The outputs of each model are then combined to perform
the convolution.

Fig. 2: The pipeline of our previous work: PDF, IDF denote Pixel Dynamic
Filter and Instance-level Dynamic Filter, respectively. $k$ and $N$ denote the
size of kernel and pixel, respectively. The IDF produces $1\times K$ weight
and the PDF process produces $N\times 1$ weight. These weights are multiplied
and the convolution is computed by using the generated weights

In our earlier work, we proposed a front-end light dynamic filter for KWS, and
the process is described in Fig 2. We divided the filter generator network
into pixel and kernel weight models respectively. In the pixel model (PDF), a
single layer of CNN and normalization processes produce pixel-level weights,
which represent the scale of each patch. In the kernel weight model (IDF),
temporal feature averaging and FC layers are employed to obtain the
convolution kernel weights depending on each utterance example. These two
weights are combined to perform single-channel CNN and it shows robust KWS
results against unseen noise environments. Although these learning frameworks
reduce the computational cost of the DFN, their performance would be limited
as feature averaging is employed to reduce the dimension of the feature. As a
simple mean operation might generate biased weights, the performance becomes
limited in a noisy environment. In this work, we aim to develop an efficient
dynamic filter, where the filter generator’s small footprint dynamic filter
would work efficiently over a simple feature average.

##  3 Efficient dynamic filter

To mitigate the biased weight problem, we aim to develop an efficient feature
mapping model which is fully trainable and low computation. To this end, we
employ DP-RNN and attention mechanisms to develop the chunk separable
convolution and dynamic attention pooling.

###  3.1 Chunk Separable Convolution

Our proposed CS-Conv is motivated by the Dual-Path Recurrent Neural Network
(DP-RNN) [11]. In DP-RNN, temporally overlapping T-F chunks are fed to two
ways of RNNs (inter and intra chunk), and the output features are assembled by
overlap adding. Instead of the RNN model, we utilized separable CNN by
reducing the temporal size, and it is computationally more efficient than RNN.
To this end, We divide T-F features into several chunks that do not overlap in
time, and separable convolutions are applied to intra chunk and inter chunk,
respectively. We change the format of the input T-F feature ($x\in R^{[F,T]}$,
where $F$ and $T$ denote frequency and temporal dimension respectively) as
non-overlapping chunks ($x_{c}\in R^{[F,C,T/C]}$, where $c$ denotes chunk
size). Then, separable convolutions are conducted to the intra and the inter-
chunk directions. In the intra chunk direction, $w_{itra}\in R^{[K,K,T/C]}$,
where $k$ denotes kernel size, strides each chunk feature and the outputs are
fed to the inter chunk convolution layer which has $w_{iter}\in R^{[K,K,C]}$
of CNN kernel. The CS-Conv procedure is as follows:

$CSconv(x_{c})=\sigma(conv(\sigma(conv(x_{c},w_{itra})),w_{iter})),$ (1)

where $\sigma(\cdot)$ and $conv(\cdot)$ denote the feature normalization and
the operation of CNN, respectively. In each convolution layer, the kernels are
put in stride to reduce the temporal dimension, and feature normalization is
applied to the output of CNN. The intra and inter-chunk CNNs would stride
various intervals of T-F patterns, and the output features might represent
long-term and short-term traits with preserving the original frequency
dimension.

###  3.2 Dynamic Attention Pooling

Fig. 3: The pipeline of dynamic attention pooling: Dot prod. denotes the dot
product computation.

In the deep learning framework, GAP is generally applied to obtain a low-
dimensional feature embedding. Based on this, mean-standard deviation pooling
[19] and a Self Attention Pooling (SAP) [20] were developed. In particular,
SAP adjusts temporal saliency using the dot product between T-F features and
learning variables. However, as SAP utilizes static weights, its performance
might be limited. In our proposed DAP, the idea of DFN is employed to produce
weights for SAP, and the process is described in Fig 3. The process of DAP is
as follows:

$w_{d}=TAP(conv(x,w)),$ (2)

$w^{Att}=\rho(x\times w_{d}),$ (3)

$DAP(x,w_{d})=\sum_{i}^{T}w^{Att}_{i}\times x_{i},$ (4)

where $\rho(\cdot)$ denotes softmax normalization. A single layer of 1D
convolution layer, which strides the temporal axis, is employed to produce a
temporally low-resolution feature, and Temporal Average Pooling (TAP) is
conducted to obtain dynamic weights ($w_{d}\in R^{[F,1]}$). Then, the weights
and the input features are multiplied to compute the temporal weights, and the
softmax normalization across the temporal step is performed. Finally, the
attention scores ($w^{Att}\in R^{[1,T]}$) are multiplied with the input
features to obtain low-dimensional feature embedding.

###  3.3 Network Architecture

Fig. 4: The pipeline of our proposed method: PDF, IDF, and CS-Conv denote
Pixel Dynamic Filter, Instance-level Dynamic Filter, and Chunk Separable
Convolution, respectively. $k$ and $N$ denote the size of kernel and pixel,
respectively. The IDF produces $1\times K$ weight and the PDF process produces
$N\times 1$ weight. These weights are multiplied and the convolution is
computed by using the generated weights.

As shown in Fig 4, we modify the IDF part from our previous work in Fig 2.
Instead of simple pooling and FC layers, the CS-Conv block is utilized to
compute long-term and short-term features, and then DAP is applied to map low-
dimensional embedding. Then, an FC layer and Swish activation [21] are used to
generate a weight vector. Our dynamic filter and main task model are both
optimized by classification loss (cross-entropy loss). For feature
normalization, we use the Instance Normalization (IN) [22] framework, where
the mean-standard deviation normalization and learnable affine transform are
applied to the frequency axis. With CS-Conv, we apply a feature normalization
to each chunk individually. In the other normalization layers, we divide the
input features into two temporal chunks, and apply feature normalization to
each chunk.

###  3.4 Learning details

In CS-Conv, we use $2\times 2$ of CNN kernels with a dilation size of 2 and
stride size of (1,2) in both convolution layers. In DAP, $1\times 25$ CNN
kernel with stride size of (1,10) is utilized to generate the dynamic weights,
which is a 40-dimensional vector. The output of DAP is fed to $40\times 9$ of
FC layer to produce the IDF weighs. In PDF, $3\times 3$ kernel with a dilation
size of 2 is used to produce the PDF weights.

##  4 Keyword Spotting

###  4.1 Experimental setup

Dateset. We used the speech command datasets v1 and v2 [12] for evaluating KWS
performance and followed the same data protocol in the DB guideline. We
utilized 10 keywords with two extra classes (unknown or silent) for model
training and injected background noise with random time-shifting. For
evaluating robustness against noise, we utilized the DCASE [23], Urbansound8K
[24] and WHAM [25] datasets. We randomly selected an audio sample from the
noise data and mixed it with the test data with 5 SNRs [20dB, 15dB, 10dB, 5dB,
and 0dB].
Implementation detail. 40 coefficients of Mel Frequency Cepstral are
constructed with 30ms of windows, 10ms overlap, and 64 Mel-filters. This
process gives a (40,98) size of audio features. In the training process, we
used a batch size of 100, 30k iterations, and an ADAM optimizer with a 0.001
initial learning rate. Every 10K iteration, the learning rate is decreased by
0.1 rates. In the PDF and the dynamic convolution process, we used $3\times 3$
CNN kernel ($k=9$) dilated by 2 with stride=1. In the IDF, $2\times 2$ kernel
dilated by 2 with stride of [1,2] is utilized to each CSconv and FC layer
follows $40\times k$ dimension. We do not use Spec-Augmentation [26] for fair
comparison.

###  4.2 Baselines

Four different baseline architectures are used for comparisons. We applied our
method to the front-end of the TENet12 architecture [27] and compared the KWS
result with the following baselines.
TCNet. TCNet [28] (or TC-Resnet) utilizes temporal convolution for low
computational cost model. TCNet8 contains 3 convolution blocks and 1 FC layer.
Each convolution block has two layers of temporal convolution with a skip-
connection. Similarly, TCNet14 contains 6 convolution blocks and 1 FC layer.
TENet. TENet [27] utilizes depth-separable temporal convolution. A convolution
block contains two 1D convolution layers and a single separable convolution
layer. TENet6 and TENet12 has 6 and 12 convolution blocks respectively. TENet
has 32 output channels for each convolution block, and TENet-n has 16 output
channels for each convolution block.
MHA-RNN. MHA-RNN[29] utilizes CRNN and self-attention models to compute the
keyword embedding vector. Two layers of FC produce KWS probabilities, and spec
augmentation [26] is used to train the model.
Neural Architecture Search. Neural Architecture Search (NAS) is a network
designing method that minimizes search costs (Flops, memory, accuracy, etc.).
We compared our method with several NAS methods including Differentiable
Architecture Search (DARTS). Please see details of the model in [30, 31].
Lightweight convolution. Lightweight Convolution [16] (LConv.) block is
designed to perform low computation separable convolution through weight
sharing and weight normalization. It contains two layers of linear, a Gated
Linear Unit (GLU), and the separable convolution. We use the single LConv
block in the front end of the TENet12 model. In the first linear layer, the
frequency dimension of the T-F features is increased by 80, and the GLU
activation is applied. Then, the separable convolution with $H=10$ and the
other linear layer is performed by preserving the feature dimension.
Additionally, instead of employing static weights in the separable
convolution, we utilized a single linear layer to produce the weight of the
convolution (Dyconv).
LDy-TENet. We previously developed LDy-TENet [10] which applied a lightweight
dynamic filter to the front-end of the TENET models. In the PDF process,
$3\times 3$ kernel with a dilation size of 2 is used. In IDF process $40\times
40$ and $40\times k$ dimension of weights are used.

###  4.3 Results and discussion

Model | (Par.,Flops.) | V1 | V2
---|---|---|---
Acc | Best | Acc | Best
TCNet8[28] | (145K,4.40M) | - | 96.2 | - | -
TCNet14[28] | (305K,8.26M) | - | 96.6 | 96.53 | 96.8
TENet6-n[27] | (17K,1.26M) | - | 96.0 | 96.30 | 96.5
TENet12-n[27] | (31K,1.97M) | - | 96.3 | 96.55 | 96.9
TENet6[27] | (54K,3.95M) | - | 96.4 | 96.83 | 97.0
TENet12[27] | (100K,6.42M) | - | 96.6 | 97.10 | 97.3
MHA-RNN†[29] | (743K,87.2M) | - | 97.2 | - | 98.0
NAS2[30] | (886K,-) | - | 97.2 | - | -
Random[31] | (196K,8.8M) | 96.58 | 96.8 | - | -
DARTS[31] | (93K,4.9M) | 96.63 | 96.9 | 96.92 | 97.1
F-DARTS[31] | (188K,10.6M) | 96.70 | 96.9 | 97.11 | 97.4
N-DARTS[31] | (109K,6.3M) | 96.79 | 97.2 | 97.18 | 97.4
LightConv[16] | (105K,7.40M) | 96.88 | 97.0 | 97.24 | 97.3
DyConv[16] | (107K,7.69M) | 96.89 | 97.1 | 96.26 | 97.4
LDy-TENet6-n[10] | (19K,1.48M) | 96.48 | 97.0 | 96.10 | 96.8
LDy-TENet12-n[10] | (33K,2.19M) | 96.69 | 96.9 | 96.80 | 97.2
LDy-TENet6[10] | (56K,4.17M) | 96.77 | 96.9 | 97.26 | 97.4
LDy-TENet12[10] | (102K,6.64M) | 96.95 | 97.1 | 97.35 | 97.6
EDy-TENet12 (Ours) | (102K,6.68M) | 97.07 | 97.4 | 97.42 | 97.8
Table 1: Comparison with lightweight models on Speech Command v1 and v2: Notation of $\dagger$ denotes the application of Spec-Augmentation [26]. For an accurate experiment, 8 times averaging accuracy and the best performance are presented. Acc | GAP | SAP | DAP
---|---|---|---
V1 | 97.02 | 97.04 | 97.07
V2 | 97.22 | 97.35 | 97.42
Table 2: Comparison of feature pooling methods depending on the Speech Commands datasets v1 and v2. Noise |  SNR (dB) | Model
---|---|---
Ours | LDy | Dconv | Lconv | TENet | TCNet
v1 | v2 | v1 | v2 | v1 | v2 | v1 | v2 | v1 | v2 | v1 | v2
DCASE [23] | 20 | 96.72 | 97.04 | 96.58 | 96.95 | 96.55 | 96.74 | 96.62 | 96.81 | 96.28 | 96.59 | 95.85 | 96.09
15 | 96.50 | 96.45 | 96.44 | 96.31 | 96.35 | 96.08 | 96.37 | 96.08 | 96.23 | 96.01 | 95.59 | 95.39
10 | 95.32 | 95.64 | 95.10 | 95.56 | 95.01 | 95.09 | 95.05 | 95.17 | 94.94 | 95.15 | 94.25 | 94.33
5 | 93.54 | 93.42 | 93.00 | 93.35 | 92.27 | 92.42 | 92.80 | 92.80 | 92.87 | 92.53 | 91.79 | 91.74
0 | 89.36 | 88.73 | 88.41 | 88.46 | 87.51 | 86.64 | 87.67 | 87.01 | 88.14 | 86.65 | 86.01 | 85.96
Urban [24] | 20 | 96.08 | 96.35 | 95.94 | 96.33 | 95.81 | 95.81 | 95.84 | 96.06 | 95.72 | 95.92 | 95.16 | 95.08
15 | 95.20 | 94.95 | 95.09 | 94.91 | 94.69 | 94.69 | 94.49 | 94.45 | 94.69 | 94.31 | 93.61 | 93.53
10 | 93.10 | 93.04 | 92.82 | 92.79 | 92.03 | 92.03 | 92.25 | 91.97 | 92.17 | 92.07 | 90.77 | 90.81
5 | 89.63 | 87.51 | 89.19 | 87.56 | 87.80 | 87.08 | 87.60 | 85.87 | 87.97 | 86.18 | 86.38 | 84.53
0 | 80.50 | 79.63 | 78.55 | 79.38 | 76.20 | 75.86 | 76.26 | 76.15 | 77.54 | 76.44 | 74.20 | 74.34
WHAM [25] | 20 | 96.17 | 96.30 | 96.09 | 96.27 | 95.96 | 96.08 | 96.02 | 96.13 | 95.75 | 95.99 | 95.43 | 95.35
15 | 95.50 | 95.40 | 95.47 | 95.40 | 96.12 | 94.61 | 95.23 | 94.84 | 95.12 | 94.74 | 94.44 | 93.45
10 | 93.06 | 93.39 | 92.91 | 93.44 | 92,39 | 92.25 | 92.69 | 92.71 | 92.67 | 92.53 | 91.46 | 91.47
5 | 89.10 | 88.10 | 88.19 | 87.95 | 87.36 | 86.26 | 87.83 | 86.60 | 87.88 | 86.37 | 85.72 | 85.42
0 | 77.79 | 76.97 | 75.92 | 76.37 | 74.52 | 73.80 | 75.18 | 74.30 | 75.19 | 74.05 | 73.15 | 72.90
Total-AVG. | 91.87 | 91.58 | 91.31 | 91.40 | 90.64 | 90.25 | 90.79 | 90.45 | 90.87 | 90.37 | 89.59 | 89.36
Table 3: Comparison with Unseen noise environment on the Speech Commands
datasets v1 and v2: experiments are performed on EDy-TENet12 LDy-TENet12,
TENet12, Lconv, Dconv, and TCNet14 model. SNR denotes signal-to-noise ratio.

Tables 1, 2, and 3 summarize the KWS results on the Speech Command datasets,
v1 and v2. The proposed method (EDy-TENet12) is applied to the IDF part of
[10]. For a more thorough model evaluation, we repeated the experiment eight
times.
Small footprint KWS. Table 1 compares small footprint KWS performances with
baseline methods. ”Par.” and ”Flops.” denote the number of training parameters
and the computational cost, respectively. The results indicate that our
proposed model improves the performance of the two datasets over the LDy
model. While the number of parameters is almost the same, 0.04M of Flops
increased. Particularly our method and the MHA-RNN method show similar
results. Given that the computational cost of the MHA-RNN is much higher than
our method, the performance improvement, which seems to be barely significant,
would be considered a meaningful result. In Table 2, we compare our method
with other feature pooling methods (GAP and SAP) to verify its effectiveness.
The GAP, SAP, and DAP are compared based on EDy-TENet12. The SAP computes the
temporal scores based on a 40-dimensional static weight vector. We confirm
that the DAP shows a small improvement over the SAP that utilizes static
weights.
Unseen noisy environment. Table 3 summarizes the KWS results on the unseen
noisy environments. These results indicate that our proposed model is more
robust than the LDy and Dconv models that use a dynamic filter. Compared with
LDy-TENet, our method takes 0.56% (v1) and 0.18% (v2) of improvement in
performance on average. Even though performance improvement is not significant
in the v2 dataset, v1 shows reasonably acceptable performance over the LDy-
TENet12 model in terms of computational cost. Particularly 2% performance
improvement is attained in the v1 dataset of the Urban 0dB condition.
In summary, our method requires 1.5k parameters with 257k Flops to implement,
whereas our previous work used 2k parameters with 220k Flops to implement.
Given the computational cost, we can conclude that our method efficiently
enhances the performance of KWS over our previous work and the other dynamic
filter approach with limited resources.

##  5 Speaker Verification

###  5.1 Experimental setup

Dateset. The speaker model is trained on the VoxCeleb2 dataset, and it is
evaluated by the VoxCeleb1 data. We use 2 seconds of speech segments which are
randomly chosen from each utterance, and 40 dimensions of the log-mel
spectrogram are extracted by a window of 25ms and step of 10ms. We perform
mean and variance normalization by instance normalization fashion. Also, we do
not apply data augmentation and Voice Activation Detection (VAD) during the
model training and evaluation.
Computational setup. Based on the TENet12 and ECAPA-TDNN [32], we compare our
the performance of front-end models. We modify the final convolution block of
the TENet12 model as 128 channels, and the vector format of the features is
extracted by the TAP process. Then, we use two FC layers to match the output
size to the number of speaker classes. A weights vector of $128\times 512$ is
used in the first FC layer, and a weights vector of $512\times 5994$ is used
to extract the probabilities for speaker identification. We utilize the output
of the first FC layer as speaker embedding for each utterance. In the ECAPA-
TDNN, we use 512 as the channel size of convolution layers and 256 dimensional
speaker embedding is used to evaluate the performance.
Implementation details. In the TENet training process, we use a batch size of
400, 500 epochs, and an ADAM optimizer with a 0.001 initial learning rate. In
the ECAPA-TDNN training, we utilize a batch size of 50, 100 epochs. In each
epoch, 500 utterances of samples are randomly selected from each speaker, and
the learning rate is decreased by 0.999. The training takes approximately four
days using the PyTorch package with RTX-3090 GPU. For the speaker model
training, we use Angular Prototypical loss [33] with an utterance of each
mini-batch as 2. We select ten samples of 2-second audio segments per test
utterance with fixed intervals and compute the cosine-based similarity between
all combinations (100 pairs) to evaluate the trained network. The averaging
similarity of all possible pairs is used as the similarity score. We utilize
Equal Error Rate (EER) and Minimal Detection Cost Function (Min. DCF) as the
evaluation metrics.
Evaluation protocol. In this work, two different evaluation protocols
(Voxceleb1-test and Voxceleb1-H) are used. Voxceleb1-test consists of 40
speaker identities which are equal to the original test set in Voxceleb1.
Voxceleb1-H contains 1251 speaker identities, and the sample pairs have the
same nationality and gender. It is noted that the development set of VoxCeleb2
does not contain the same speaker identities as VoxCeleb1. This means that the
evaluation is carried out in an open environment (unseen speaker).

###  5.2 Result discussion

Table 4 shows the SV results depending on the two different evaluation
protocols (Voxceleb1-test and Voxceleb1-H). We confirm that the dynamic filter
in the front-end enhances the SV performance. Particularly our proposed model
takes 0.53% and 0.31% of EER improvements in the Voxceleb1-H test set. As the
weights of CNN are generated by the filter generator, it shows robust
performance over unseen speakers. Especially our proposed method efficiently
extracts feature, and it would enhance the performance of the dynamic filter
better than our previous work.

SV | Front | Vox-test | Vox-H
---|---|---|---
EER | M. DCF | EER | M. DCF
TENet [27] | EDy | 4.79 | 0.36 | 9.77 | 0.59
LDy | 4.97 | 0.38 | 10.30 | 0.62
Base | 5.30 | 0.38 | 10.43 | 0.63
ECAPA. [32] | EDy | 2.90 | 0.24 | 6.16 | 0.41
LDy | 2.99 | 0.24 | 6.47 | 0.42
Base | 3.06 | 0.25 | 6.71 | 0.44
Table 4: Comparison of the speaker verification results.

##  6 Conclusion

In this paper, we proposed an efficient dynamic filter to extract robust and
low computational features. In our earlier work, we showed that a dynamic
neural architecture in the front-end of the model would enhance the robustness
against unseen noise. However, as the simple feature means was applied to
reduce the computational cost, the performance of the dynamic filter became
limited, and it would particularly reduce performance in an unseen
environment. To combat this limitation, we staged the separable convolutions
using temporal chunks (intra and inter-chunks) to extract local and global
features. Then, a dynamic filter-based feature pooling was employed to extract
the low dimensional feature vector. This approach was applied to the Instance-
level Dynamic Filter in our previous work, and performed relevant experiments
on the keyword spotting and speaker verification tasks. From the experimental
results, we verified that the proposed method enhances robustness against
unseen environments (noise and speaker) compared to the recently developed
methods with low computational cost.

## References

  * [1] Santiago Pascual, Antonio Bonafonte, and Joan Serra,  “Segan: Speech enhancement generative adversarial network,”  arXiv preprint arXiv:1703.09452, 2017.
  * [2] Szu-Wei Fu, Chien-Feng Liao, Yu Tsao, and Shou-De Lin,  “Metricgan: Generative adversarial networks based black-box metric scores optimization for speech enhancement,”  in International Conference on Machine Learning. PMLR, 2019, pp. 2031–2041.
  * [3] Chao-Yuan Kao, Sangwook Park, Alzahra Badi, David K Han, and Hanseok Ko,  “Orthogonal gradient penalty for fast training of wasserstein gan based multi-task autoencoder toward robust speech recognition,”  IEICE TRANSACTIONS on Information and Systems, vol. 103, no. 5, pp. 1195–1198, 2020.
  * [4] Yi Hu and Philipos C Loizou,  “Evaluation of objective quality measures for speech enhancement,”  IEEE Transactions on audio, speech, and language processing, vol. 16, no. 1, pp. 229–238, 2007.
  * [5] Szu-Wei Fu, Chien-Feng Liao, and Yu Tsao,  “Learning with learned loss function: Speech enhancement with quality-net to improve perceptual evaluation of speech quality,”  IEEE Signal Processing Letters, vol. 27, pp. 26–30, 2019.
  * [6] Ziyi Xu, Maximilian Strake, and Tim Fingscheidt,  “Deep noise suppression with non-intrusive pesqnet supervision enabling the use of real training data,”  arXiv preprint arXiv:2103.17088, 2021.
  * [7] Gwantae Kim, David K Han, and Hanseok Ko,  “Specmix: A mixed sample data augmentation method for training withtime-frequency domain features,”  arXiv preprint arXiv:2108.03020, 2021.
  * [8] Suwon Shon, Hao Tang, and James Glass,  “Voiceid loss: Speech enhancement for speaker verification,”  Proc. Interspeech, pp. 2888–2892, 2019.
  * [9] Donghyeon Kim, Jaihyun Park, David K Han, and Hanseok Ko,  “Dual stage learning based dynamic time-frequency mask generation for audio event classification,”  Proc. Interspeech 2020, pp. 836–840, 2020.
  * [10] Donghyeon Kim, Kyungdeuk Ko, Jeong-gi Kwak, David K Han, and Hanseok Ko,  “Lightweight dynamic filter for keyword spotting,”  arXiv preprint arXiv:2109.11165, 2021.
  * [11] Yi Luo, Zhuo Chen, and Takuya Yoshioka,  “Dual-path rnn: Efficient long sequence modeling for time-domain single-channel speech separation,”  in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 46–50.
  * [12] Pete Warden,  “Speech commands: A dataset for limited-vocabulary speech recognition,”  arXiv preprint arXiv:1804.03209, 2018.
  * [13] Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman,  “Voxceleb: Large-scale speaker verification in the wild,”  Computer Speech & Language, vol. 60, pp. 101027, 2020.
  * [14] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “Voxceleb2: Deep speaker recognition,”  arXiv preprint arXiv:1806.05622, 2018.
  * [15] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool,  “Dynamic filter networks,”  in Advances in neural information processing systems, 2016, pp. 667–675.
  * [16] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli,  “Pay less attention with lightweight and dynamic convolutions,”  arXiv preprint arXiv:1901.10430, 2019.
  * [17] Yuya Fujita, Aswin Shanmugam Subramanian, Motoi Omachi, and Shinji Watanabe,  “Attention-based asr with lightweight and dynamic convolutions,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7034–7038.
  * [18] Jingkai Zhou, Varun Jampani, Zhixiong Pi, Qiong Liu, and Ming-Hsuan Yang,  “Decoupled dynamic filter networks,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6647–6656.
  * [19] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5329–5333.
  * [20] Pooyan Safari and Javier Hernando,  “Self multi-head attention for speaker recognition,”  arXiv preprint arXiv:1906.09890, 2019.
  * [21] Prajit Ramachandran, Barret Zoph, and Quoc V Le,  “Searching for activation functions,”  arXiv preprint arXiv:1710.05941, 2017.
  * [22] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky,  “Instance normalization: The missing ingredient for fast stylization,”  arXiv preprint arXiv:1607.08022, 2016.
  * [23] Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen,  “Acoustic scene classification in dcase 2019 challenge: closed and open set classification and data mismatch setups,”  in Workshop on Detection and Classification of Acoustic Scenes and Events, 2019.
  * [24] Justin Salamon, Christopher Jacoby, and Juan Pablo Bello,  “A dataset and taxonomy for urban sound research,”  in Proceedings of the 22nd ACM international conference on Multimedia, 2014, pp. 1041–1044.
  * [25] Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux,  “Wham!: Extending speech separation to noisy environments,”  in Proc. Interspeech, Sept. 2019.
  * [26] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  arXiv preprint arXiv:1904.08779, 2019.
  * [27] Ximin Li, Xiaodong Wei, and Xiaowei Qin,  “Small-footprint keyword spotting with multi-scale temporal convolution,”  Proc. Interspeech, pp. 1987–1991, 2020.
  * [28] Seungwoo Choi, Seokjun Seo, Beomjun Shin, Hyeongmin Byun, Martin Kersner, Beomsu Kim, Dongyoung Kim, and Sungjoo Ha,  “Temporal convolution for real-time keyword spotting on mobile devices,”  Proc. Interspeech, pp. 3372–3376, 2019.
  * [29] Oleg Rybakov, Natasha Kononenko, Niranjan Subrahmanya, Mirko Visontai, and Stella Laurenzo,  “Streaming keyword spotting on mobile devices,”  arXiv preprint arXiv:2005.06720, 2020.
  * [30] Tong Mo, Yakun Yu, Mohammad Salameh, Di Niu, and Shangling Jui,  “Neural architecture search for keyword spotting,”  Proc. Interspeech, pp. 1982–1986, 2020.
  * [31] Bo Zhang, Wenfeng Li, Qingyuan Li, Weiji Zhuang, Xiangxiang Chu, and Yujun Wang,  “Autokws: Keyword spotting with differentiable architecture search,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 2830–2834.
  * [32] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,”  arXiv preprint arXiv:2005.07143, 2020.
  * [33] Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Lee, and Icksang Han,  “In defence of metric learning for speaker recognition,”  arXiv preprint arXiv:2003.11982, 2020.