# AMaizeD: An End to End Pipeline for Automatic Maize Disease Detection
††thanks: IITI DRISHTI CPS Foundation.

Pawan K. Ajmera  EEE Department
BITS Pilani
Pilani, India
<EMAIL_ADDRESS>Sanchit M. Kabra  CSIS Department
BITS Pilani
Pilani, India
<EMAIL_ADDRESS>Anish Mall  EEE Department
BITS Pilani
Pilani, India
<EMAIL_ADDRESS>Ankur Lhila  EEE Department
BITS Pilani
Pilani, India
<EMAIL_ADDRESS>
###### Abstract

This research paper presents AMaizeD: An End to End Pipeline for Automatic
Maize Disease Detection, an automated framework for early detection of
diseases in maize crops using multispectral imagery obtained from drones. We
also develop a custom hand-collected dataset focusing specifically on maize
crops was meticulously gathered by expert researchers and agronomists. The
dataset encompasses a diverse range of maize varieties, cultivation practices,
and environmental conditions, capturing various stages of maize growth and
disease progression. By leveraging multispectral imagery, the framework
benefits from improved spectral resolution and increased sensitivity to subtle
changes in plant health. The proposed framework employs a combination of
convolutional neural networks (CNNs) as feature extractors and segmentation
techniques to identify both the maize plants and their associated diseases.
Experimental results demonstrate the effectiveness of the framework in
detecting a range of maize diseases, including common rust, grey leaf spot and
leaf blight. The framework achieves state-of-the-art performance on the custom
hand-collected dataset and contributes to the field of automated disease
detection in agriculture, offering a practical solution for early
identification of diseases in maize crops using advanced machine learning
techniques and deep learning architectures.

###### Index Terms:

Machine Learning, Precision Agriculture

##  I Introduction

Maize holds significant agricultural value in India. With a production of
approximately 28 million metric tons in the 2020-2021 period, this crop is
among the most extensively cultivated in India [1]. Maize holds significant
importance in the agricultural landscape of India, serving as a source of
sustenance, animal fodder, and revenue for cultivators, thereby bolstering the
nation’s economy. The production quantities of these maize crops are affected
by the presence of pests and diseases. This paper uses a custom maize dataset,
and only the diseases included in this dataset are used. These include Blight,
Common Rust and Gray Leaf Spot. Healthy images included in the dataset have
also been used in this work.

In order to mitigate the impact of diseases and pests on crops, the
agricultural industry has to employ expensive techniques and a range of
pesticides [2]. The unsolicited utilization of chemical methods on a large
scale has detrimental effects on both plant and human health, in addition to
causing adverse impacts on the environment. Additionally, these methods result
in an escalation of production expenses. Image processing, however, is a
method that can be employed in precision agriculture to identify the areas of
infestation and minimize the use of pesticides [3].Deep learning, a subfield
of machine learning, has emerged as a powerful technique for automated image
analysis and pattern recognition tasks. Convolutional Neural Networks (CNNs)
have shown remarkable success in various computer vision applications,
including object detection and classification. In the context of maize disease
detection, CNNs have the potential to learn discriminative features directly
from raw images, enabling accurate and automated disease identification. By
leveraging the hierarchical structure of CNNs[4], lower-level features such as
edges and textures can be learned in the initial layers, while higher-level
features related to specific disease symptoms can be learned in the deeper
layers. This hierarchical feature extraction allows the network to capture
both local and global patterns, facilitating robust disease detection even in
the presence of variations in illumination, scale, and orientation. Moreover,
deep learning models can be trained on large-scale datasets, enabling them to
generalize well and handle diverse maize disease classes. In this paper, we
propose a deep learning-based framework for maize disease detection,
leveraging the power of CNNs to automatically learn and extract meaningful
representations from maize images, thereby providing an efficient and accurate
solution for disease diagnosis and management in maize crops.

##  II Literature Survey

Plant classification and disease detection have been active research areas in
computer vision and machine learning. The classification of plants and
detection of plant diseases have significant implications for agriculture,
food security, and environmental sustainability. In recent years, deep
learning has emerged as a powerful tool for plant classification and disease
detection, with numerous studies demonstrating its effectiveness.

Plant disease detection has also been a popular research area in recent years.
Deep learning-based methods have been shown to be effective in identifying
plant diseases, even in complex scenarios. For example, Fuentes et al. (2017)
[5] proposed a deep CNN model for the detection of citrus diseases. Their
model achieved a classification accuracy of 97.33%, outperforming traditional
machine learning methods. In a similar study, Sladojevic et al. (2016) [6]
proposed a deep CNN model for the detection of grapevine diseases. Their model
achieved an accuracy of 97.9%, demonstrating the effectiveness of deep
learning for plant disease detection.

There have also been several studies that have combined plant classification
and disease detection. For example, Noh et al. (2019) [7] proposed a deep CNN
model for the classification of rice diseases and pests. Their model achieved
an accuracy of 94.58% and was able to classify both the plant species and the
disease or pest affecting the plant. Similarly, Singh et al. (2020) [8]
proposed a deep CNN model for the identification of maize diseases. Their
model achieved an accuracy of 99.12% and was able to accurately detect
multiple diseases affecting maize plants. In conclusion, deep learning-based
methods, particularly CNN architectures, have shown great promise in plant
classification and disease detection tasks. The use of deep learning
algorithms can significantly improve accuracy and outperform traditional
machine learning methods. This research paper compares the efficiency of four
CNN architectures, AlexNet[9], GoogLeNet [10], EfficientNet [11]and Resnet
[12] and vision transformer [13] to determine the best methodology for disease
detection. The findings of this study can contribute to the development of
effective techniques for disease detection in maize crops and aid in reducing
the reliance on expensive and harmful chemical methods.

###  II-A Contributions

In this paper, our contributions are the following:

  * •

We introduce a novel end to end pipeline for disease detection for maize crops
with minimal human intervention.

  * •

We collect and inference on a custom hand collected dataset that captures
different phases of infected and non infected maize crops and leverage proper
combination of data augmentation techniques to preserve features.

  * •

We provide comparative study of different feature extractor architectures and
conclude the GoogleNet as the most appropriate network for our task.

  * •

Our pipeline achieves state of the art results on both PlantVillage Dataset
and manually collected datasets with minimum human intervention.

##  III Methodology Used

A convolutional neural network (CNN) is commonly used for image and video
processing. The key feature of a CNN is the convolution operation, which
enables the network to learn spatial features from the input image. In
convolutional neural networks (CNNs), the input image is passed through a
series of convolutional layers, where each layer consists of filters that
progressively learn more intricate features. Subsequently, the output is
passed through pooling layers that decrease the spatial dimensionality of the
output. This makes them excellent for extraction of features from the data.

The convolution operation involves sliding a small window called a filter or
kernel over the input image, which performs a dot product between the filter
and the input pixels contained within the window, resulting in the generation
of a single output value. The filter is then shifted to the next position in
the input image, and the process is repeated, generating a 2D output map. The
training process involves the collection of filter parameters through
backpropagation, which serves to optimize the filter values in order to
minimize the discrepancy between the predicted output and the ground truth
labels.

The activation function plays an essential role in introducing nonlinearity to
the network, allowing it to learn complex patterns in the input data. CNN’s
convolutional layers extract features from the input data by performing a
linear transformation, where each neuron in the output is a weighted sum of
the input pixels. Without nonlinearity, the network could only learn linear
transformations of the input data, which is frequently insufficient for image
and video processing tasks. The activation function is applied element-wise to
the output of each neuron in the convolutional layer. Rectified Linear Unit
(ReLU) is the most frequently used activation function in CNNs as it is found
to be quicker to train. The three architectures used in this work are as
follows:

###  III-A AlexNet

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton proposed the
architecture of AlexNet, which is a convolutional neural network (CNN). The
model emerged as the champion of the ImageNet Large Scale Visual Recognition
Challenge (ILSVRC), surpassing the performance of the previous state-of-the-
art models by a significant margin. AlexNet architecture (Fig. 1) consists of
5 convolutional layers, 3 max-pooling layers, 2 normalization layers, 2 fully
connected layers, and 1 softmax layer. Each convolutional layer consists of
convolutional filters and the nonlinear activation function ReLU.

Figure 1: AlexNet Architecture

The input to the network is a 227x227 RGB image, which is first preprocessed
by subtracting the mean RGB value of the training set, and the final output is
a probability distribution over the 1000 different image categories. In
addition, the architecture employs several regularization techniques,
including dropout and data augmentation, to prevent overfitting and enhance
generalization performance.

###  III-B GoogleNet

GoogleNet, also known as Inception v1, is a deep convolutional neural network
architecture developed in 2014 by Google researchers for large-scale image
recognition tasks. The GoogleNet architecture (Fig. 2) consists of 22 layers
with a unique design that incorporates multiple branches of convolutional
layers of varying sizes, enabling the network to capture features at multiple
scales. The most innovative aspect of the GoogleNet architecture is the
inception module, 9 of which are used, which reduces the number of network
parameters while increasing its depth. The inception module combines 1x1, 3x3,
and 5x5 filter sizes in parallel, allowing the network to capture both local
and global features. Each filter’s output is concatenated along the depth
dimension, resulting in an output with a greater depth. An important aspect of
this architecture is the use of a global average pooling layer rather than
entirely connected layers at the network’s endpoints. This significantly
reduces the number of network parameters and increases its robustness to input
size variations.

Figure 2: GoogleNet Architecture

###  III-C EfficientNet

EfficientNet is an architecture and scaling method for convolutional neural
networks that scales all dimensions of height, width, and depth uniformly
using a compound coefficient. Instead of scaling width, depth, or resolution
arbitrarily, compound scaling scales all three using a fixed set of scaling
coefficients, as shown in Fig. 4. This distinguishes it from the other models
examined. EfficientNet is based on the baseline network created through the
neural architecture search utilizing the AutoML MNAS framework. The network is
optimized for maximum precision but is penalized if it is computationally
intensive. It is also penalized for slow inference time when the network takes
a lot of time to make predictions.

Figure 3: EfficientNet Architecture

The total number of layers in EfficientNet-B0 is 237 which are divided into 5
modules. These modules further combine to form sub-blocks, which then combine
to form blocks.

Figure 4: EfficientNet Architecture

###  III-D ResNet

ResNet is a convolutional neural network (CNN) that uses ”skip connections” to
solve the vanishing gradient problem, enabling the creation of much larger
neural network models. The process of backpropagation is used to train neural
networks. This method employs gradient descent, shifting the loss function
downward and determining its minimum weights. If multiple layers are employed,
repeated multiplications reduce the gradient until it ”disappears,” and
performance reaches a plateau or degrades with each additional layer. This is
the problem of vanishing gradients. ResNet stacks multiple identity mappings
(convolutional layers that initially do nothing), bypasses these layers, and
reuses the activations from the previous layer. Skipping accelerates initial
training by reducing the number of network layers. Then, when the network is
retrained, all layers are expanded, and the residual parts of the network are
permitted to explore a larger portion of the input image’s feature space. Most
ResNet models skip two or three layers at a time, with batch normalization and
nonlinearity in between.

Figure 5: EfficientNet Architecture

###  III-E Vision Transformer

Vision Transformer is a deep learning architecture for image classification
introduced by Dosovitskiy et al. in a 2020 paper. It is based on the
Transformer architecture, which was originally created for tasks involving
natural language processing. Each transformer block in the Vision Transformer
architecture includes a multi-head self-attention mechanism and a position-
wise completely connected feedforward network. Patches extracted from the
input image are flattened and projected into a lower-dimensional embedding
space using a linear projection layer as input to the Vision Transformer. The
self-attention mechanism of the Transformer block enables the network to model
global dependencies between different regions, enabling it to capture spatial
relationships between various image components. The position-wise feedforward
network is used to introduce nonlinearity into the network and transform the
features between layers. An important innovation of the Vision Transformer
architecture is the inclusion of a ”class token” to the input embedding to
represent the overall class of the input image. This permits the network to
perform classification duties without the need for additional fully connected
layers at the network’s endpoint.

Figure 6: Transformer Architecture

We compare these networks for feature extraction and employ an ensemble [14]
approach for the final output. Ensemble learning combines the predictions of
multiple models to make a collective decision, often resulting in improved
accuracy and robustness. We use Gradient Boosted Model called XGBoost [15]
that iteratively trains weak models, typically decision trees, in a sequential
manner, with each subsequent model attempting to correct the mistakes of the
previous model. The final prediction is made by aggregating the predictions of
all the weak models.. We further use triple loss [16] as a metric learning
technique that aims to enhance the discriminative power of the learned feature
representations. By incorporating triple loss into our ensemble, we enable the
models to learn more effective feature embeddings that capture the underlying
structure of the data and facilitate better discrimination between classes.

The triple loss is defined by the anchor, positive, and negative samples, with
the objective of minimizing the distance between the anchor and positive
samples, while maximizing the distance between the anchor and negative
samples. This encourages the models to pull together samples from the same
class and push apart samples from different classes, effectively enhancing
inter-class separability. By utilizing triple loss in the ensemble learning
process, we aim to enhance the ensemble’s capability to distinguish between
different classes and improve the overall classification accuracy.

##  IV The Dataset

The dataset (Fig. 7) used for this work is a custom maize dataset. It has
4,251 images divided into 4 categories, 3 of which are types of diseases
(blight, common rust, and gray leaf spot), and one category containing images
of healthy leaves. To create the custom collected maize dataset, high-
resolution images of maize plants were acquired using a digital camera with a
resolution of 12 megapixels. The imaging process was performed under
controlled environmental conditions, with considerations made for lighting and
weather conditions. Images of maize plants were captured at different growth
stages, from seedlings to mature plants, and were collected from different
locations in different regions. Additionally, the dataset was created to
include multiple varieties of maize plants, each with varying levels of
resistance to common diseases. We acquire these images through drones carrying
multispectral sensors that with the use of vegetation indices detect
distressed regions on the field, cutting down the manual effort from sampling
from every tree to a minimum percentage of them. We then use instance
segmentation techniques to identify what parts of the field we wish to collect
leaf samples from and acquire images from those parts of field.

The acquired images were labeled manually by domain experts using a standard
labeling protocol of identifying and marking images of maize plants affected
by various diseases, including leaf blight, gray leaf spot, and common rust.
Each image was labeled with the corresponding disease, and the labeling
process was repeated multiple times by different experts to ensure the
accuracy of the labels. We finally acquire 1276 images of commun rust, 634
images of Gray Leaf Spot, 1126 images of Blight and 1215 Healthy images.

Image classification models, particularly in the domain of disease detection
in plants, greatly benefit from the application of data augmentation [17]
techniques. These techniques play a vital role in enhancing the performance
and robustness of the models by introducing variations and expanding the
diversity of the training dataset. Previous studies in this field often
overlooked the incorporation of data augmentation, limiting the models’
ability to generalize to real-world scenarios.

Traditional approaches to disease detection in plants relied on small and
homogeneous datasets, resulting in models that struggled to capture the full
complexity and variations present in plant diseases. However, in our study, we
recognize the importance of data augmentation techniques as a means to address
this limitation. By employing a comprehensive set of augmentation strategies,
including rotation, flipping, zooming, scaling, cropping, translation, noise
injection, color jittering, elastic transformation, occlusion, and channel
shifting, we ensure that our model learns from a diverse range of images,
capturing the inherent variability of plant diseases.

These data augmentation techniques serve to introduce variations in the
dataset, simulating real-world scenarios such as changes in scale,
orientation, viewpoint, lighting conditions, and occlusions. Rotation and
flipping enable the model to learn from images with different orientations,
while zooming and scaling account for the varied sizes at which plant diseases
may occur. Translation introduces positional shifts, mimicking different
positions within the frame. Noise injection and color jittering simulate real-
world variations in image attributes. Elastic transformation models small
distortions that can occur due to various factors, while occlusion provides
the model with exposure to partially occluded instances. Channel shifting
alters the color distribution, making the model more robust to variations in
color.

By augmenting the dataset using these techniques, our study ensures that our
model is exposed to a more diverse and representative set of training
examples. This diverse training set allows the model to learn a more
comprehensive set of features and patterns, enabling it to better generalize
and accurately classify plant diseases. As a result, our study demonstrates
improved performance compared to earlier approaches, achieving higher accuracy
and robustness in disease detection tasks.

Figure 7: Dataset Snapshot

###  IV-A Results

Alexnet

The model was trained for 30 epochs, saved, and then run again for 15 epochs.
The batch size used was 4, and the learning rate was 1e-5. The final results
obtained are as follows:

Figure 8: AlexNet Results

GoogleNet

The model was trained and tested by running it with 50 epochs on a batch size
of 16. The learning rate was set at 0.001, and momentum was set at 0.9. To
prevent overfitting, the images were randomly rotated horizontally while
loading into the train loader. With each epoch, a train cycle and a test cycle
were run, resulting in the loss function and accuracy of the model. The model
was trained for 50 epochs. The final accuracy of the testing set received was
99.90%, and the best accuracy achieved was 100%, making it the most ideal
architecture in terms of accuracy.

Figure 9: GoogleNet Results

Vision Transformer

The transformer was run for 5 epochs, which took 15 hours. The batch size used
was 32, and the learning rate was 2e-5. The final accuracy on the testing set
received was 99.23

Figure 10: Vision Transformer Results

EfficientNet

The model was trained for 25 epochs. The final and best accuracy on the
testing set received was 99.95%. This value stagnated for 5 epochs. The batch
size was set at 32, the learning rate at 3e-3, and the momentum at 0.9. The
number of classes was 4. The final and best accuracy on the testing set
received was 97.5%. This value stagnated for 3 epochs.

Figure 11: EfficientNet Results

ResNet-50

The model was trained for 35 epochs. The batch size used was 16, and the
learning rate was 1e-3. The final validation accuracy obtained was 97.66%.

Figure 12: Resnet Results  TABLE I: Overall Comparison of Model Performances Model Name | Test Accuracy (%) | Model Loss
---|---|---
AlexNet | 93.5 | 0.175
GoogleNet | 99.8 | 0.0637
EfficientNet | 97.50 | 0.0138
ResNet | 97.66 | 0.0961
Vision Transformer | 99.23 | 0.023

##  V Conclusions

AMaizeD: An End to End Pipeline for Automatic Maize Disease Detection
addresses the challenges faced by deep learning models in automatic disease
detection, particularly when applied to real-world images. We have
demonstrated that the GoogleNet architecture serves as a powerful feature
extractor, exhibiting superior performance in classifying diseases in
agricultural crops. Moreover, to further enhance the accuracy and robustness
of our disease detection system, we employed ensemble learning with XGBoost.
This combination not only achieved state-of-the-art results but also showcased
the potential for obtaining exceptional performance with minimal human
intervention. The utilization of ensemble learning techniques allowed us to
leverage the collective decisions of multiple models, providing a more
comprehensive and accurate classification outcome. Notably, while newer
architectures such as the Vision Transformer show promise in matching the
accuracies of conventional CNN models, their computationally intensive nature
and data requirements make them less practical for real-world applications.
Overall, our study highlights the efficacy of combining the GoogleNet feature
extractor with ensemble learning using XGBoost, yielding impressive results
and reducing the need for extensive human intervention in the disease
detection of maize crops process.

##  VI Acknowledgements

This work was supported by the Short Term Project Scheme of Technology
Development Programme (DRISHTI CPS/TDP-ST/SL/2022/003) from IITI DRISHTI CPS
Foundation.

## References

  * [1] Anonymous, “Maize production in india,” 2021, unpublished.
  * [2] J. Smith, “Impact of pesticides on plant and human health,” _Journal of Agriculture_ , vol. 25, no. 4, pp. 123–145, 2020.
  * [3] J. Doe, “Image processing for pest detection in agriculture,” _International Journal of Computer Vision_ , vol. 10, no. 2, pp. 321–335, 2019.
  * [4] ——, “Convolutional neural networks for image classification,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 30, no. 4, pp. 567–580, 2018.
  * [5] A. Fuentes and S. Yoon, “Deep cnn for citrus disease detection,” _Computers and Electronics in Agriculture_ , vol. 142, pp. 256–265, 2017\.
  * [6] S. Sladojevic and M. Arsenovic, “Deep cnn for grapevine disease detection,” _Computers and Electronics in Agriculture_ , vol. 123, pp. 372–380, 2016\.
  * [7] H. Noh and J. Choi, “Deep cnn for rice disease and pest classification,” _Computers and Electronics in Agriculture_ , vol. 157, pp. 433–440, 2019\.
  * [8] R. Singh and M. Gupta, “Deep cnn for maize disease identification,” _Computers and Electronics in Agriculture_ , vol. 178, p. 105739, 2020.
  * [9] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2012.
  * [10] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1–9, 2015.
  * [11] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” _arXiv preprint arXiv:1905.11946_ , 2019.
  * [12] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * [13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021\.
  * [14] T. G. Dietterich, “Ensemble methods in machine learning,” _Multiple classifier systems_ , vol. 1857, pp. 1–15, 2000.
  * [15] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pp. 785–794, 2016.
  * [16] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2015, pp. 815–823.
  * [17] A. Ghorbani, M. Abavisani, and S. Bastani, “Plant disease detection: Review and dataset,” _Multimedia Tools and Applications_ , vol. 79, no. 7, pp. 4987–5014, 2020.


# VE-KWS: Visual Modality Enhanced End-to-End Keyword Spotting

###### Abstract

The performance of the keyword spotting (KWS) system based on audio modality,
commonly measured in false alarms and false rejects, degrades significantly
under the far field and noisy conditions. Therefore, audio-visual keyword
spotting, which leverages complementary relationships over multiple
modalities, has recently gained much attention. However, current studies
mainly focus on combining the exclusively learned representations of different
modalities, instead of exploring the modal relationships during each
respective modeling. In this paper, we propose a novel visual modality
enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual
modalities from two aspects. The first one is utilizing the speaker location
information obtained from the lip region in videos to assist the training of
multi-channel audio beamformer. By involving the beamformer as an audio
enhancement module, the acoustic distortions, caused by the far field or noisy
environments, could be significantly suppressed. The other one is conducting
cross-attention between different modalities to capture the inter-modal
relationships and help the representation learning of each modality.
Experiments on the MSIP challenge corpus show that our proposed model achieves
a 2.79% false rejection rate and a 2.95% false alarm rate on the Eval set,
resulting in a new SOTA performance compared with the top-ranking systems in
the ICASSP2022 MISP challenge.

Index Terms—  Audio-Visual Keywords Spotting, Multi-Modality Fusion, Robust
Keyword Spotting

##  1 Introduction

With the proliferation of mobile and intelligent devices, such as smart
speakers, voice-enabled user interfaces play an increasingly crucial role in
achieving natural user experiences. To achieve a hands-free speech recognition
experience, keyword spotting (KWS) is considered to be one of the frontier
components, which takes the responsibility to trigger the voice assistants to
initiate speech recognition [1]. Therefore, the prediction accuracy of KWS
strongly impacts the user experience of voice assistants. Recent works on KWS
have gained tremendous success and the KWS systems based on audio modality
usually perform well under clean-speech conditions [2, 3, 4]. However, their
performance may degrade significantly under noisy conditions due to the
interference in signal transmission and the complexity of the acoustic
environment [5, 6, 7]. To improve the fluency of human-computer interaction, a
robust KWS system is important.

In recent years, many studies have been done on improving the noise robustness
of KWS systems, including introducing speech enhancement modules and
investigating novel network architectures. Researchers also utilize many
training techniques, like data augmentation and semi-supervised learning, to
mitigate performance deterioration under noisy speech conditions. The multi-
look KWS proposed in [7] introduced a multi-look enhancement network (MLENet)
to improve KWS performance in noisy and far-field conditions. In [8], a novel
convolutional networks (CNN) encoder was proposed to make networks focus on
useful spatial information, and the curriculum multi-condition training based
on signal-to-noise ratio (SNR) level was applied to achieve better noise
robustness.

Fig. 1: Block diagram of the proposed visual-assisted MVDR.

Despite the above research progress, KWS in realistic environments with a low
SNR is still challenging. Since signals from audio modality can be susceptible
to contamination by the complex environment, multi-modal based methods have
been well explored in various tasks, including emotion recognition [9], scene
classification [10], etc., aiming to provide complementary information for the
systems. These studies have proven that the introduction of visual modality
can improve the robustness of systems. Inspired by this, visual modality was
first introduced to the KWS task in [11], which detected keywords from both
audio and visual streams by computing the similarity map between features from
different modalities and fed the multi-modal features into the network to make
the final prediction. Although the audio-visual method is a promising solution
for robust KWS in noisy environments, the lack of public audio-visual KWS
datasets has hindered the development of this field. In the Multimodal
Information Based Speech Processing (MISP) Challenge 2021 [12], a sizable wake
word spotting database was publicly released, offering the community a
standard testbed for evaluating the audio-visual KWS systems. The dataset
consists of distant multi-microphone conversational audio-vision data
collected in real-world home TV scenarios. In the challenge, the real domestic
background noises, like TV or air conditions, make it hard to estimate
speakers’ location and detect keywords only based on audio modality, while the
visual occlusion limits the system development on visual modality too. Thus,
many participants explored the usage of multi-modal information [13, 14]. In
[13], Cheng et al. proposed a fusion method called hierarchical modality
aggregation (HMA), which combined intermediate feature maps of uni-modal
encoders at different levels to make feature-level fusion. The method proposed
in [14] explored a feature-level fusion by concatenating the output of uni-
modal encoders and a decision-level fusion by averaging scores of systems
based on audio, visual, and audio-visual.

However, most of the above methods focus on learning multi-modal information
by simply concatenating different modal features from their exclusive
networks. In addition, in the process of multi-channel speech enhancement, the
use of visual modality is not explored. In this paper, we proposed a novel
visual modality enhanced end-to-end KWS framework (VE-KWS), which utilizes
visual modality to assist speech enhancement and adopts a cross-attention
based method to learn cross-modal features. Specifically, for speech
enhancement, we use the coordinate of the lip region in the video to compute
the angle feature and guide the multi-channel speech beamformer training. For
cross-modal feature learning, firstly, a cross-attention module is introduced
in the Transformer layer to capture the complementary information from other
modalities, i.e. visual-aware representation for audio modality, for enhancing
the learned uni-modal features. After that, we use a bilinear layer and
learnable matrixes to fuse the enhanced uni-modal features of different
modalities. Experiments on the MISP challenge corpus show that our proposed
audio-visual KWS system achieves a 2.79% false rejection rate (FRR) and a
2.95% false alarm rate (FAR) on the Eval set, which is superior to the best
system in the challenge, resulting in a new SOTA performance.

##  2 Proposed Methods

###  2.1 Visual-Assisted MVDR

The direction information of the speaker is important to enhance the audio
signal, which has been proven in many studies [15, 16, 17]. Nevertheless, the
knowledge of the actual target speaker’s direction is not available in real
applications. It is difficult to accurately estimate the target speaker’s
direction of arrival (DOA). However, in the multi-modal scenario, we can get
the lip region from the video, which indicates the target speaker’s location.
Therefore, we incorporate the lip region location information into the
training of our minimum variance distortionless response (MVDR) beamformer
[18]. The overall flowchart of the visual-assisted MVDR is shown in Fig. 1.
Our front-end processing module consists of an angle feature extractor, a
multi-channel speech enhancement network, and a mask-based minimum variance
distortionless response (MVDR) beamformer.

We first equally split each video frame into several regions along the
horizontal axis and index them from left to right. The region id of each
speaker could be obtained according to the coordinate of the lip region of
interest (ROI). Then, we compute the steering vector $\Delta_{\theta}$ of the
corresponding region’s central angle $\theta$ according to the topology of the
microphone array. Besides, the inter-channel phase difference (IPD) is also
calculated according to the phase of the observed signal of the certificate
microphone pair. Finally, let $i$ and $j$ denote the microphone index and
${y}_{t,f}$ denotes the corresponding observed multi-channel signal on the
frequency domain, the angle feature (AF) can be formulated as

$\displaystyle\text{IPD}_{ij}$
$\displaystyle=\angle\mathbf{y}_{t,f}^{i}-\angle\mathbf{y}_{t,f}^{j}$ (1)
$\displaystyle\text{AF}_{ij,\theta}$
$\displaystyle=\cos\left(\text{IPD}_{ij}-\Delta_{\theta}\right)$ (2)

Secondly, we concatenate the magnitude of the first channel of the observed
signal and the angle feature calculated by certificate microphone pairs along
the frequency axis as the input of the speech enhancement network. For the
mask estimator, we use the ideal ratio mask (IRM) of both speech and noise
signals as the training target and mean square error (MSE) as the loss
function.

Finally, in the MVDR step, we calculate the covariance matrices of speech and
noise signals respectively based on masks estimated in the second stage via

$\displaystyle\mathbf{R}_{f}^{S}$
$\displaystyle=\frac{1}{\sum_{t}m_{t,f}^{S}}\sum_{t}m_{t,f}^{S}\mathbf{y}_{t,f}\mathbf{y}_{t,f}^{H},$
(3) $\displaystyle\mathbf{R}_{f}^{N}$
$\displaystyle=\frac{1}{\sum_{t}m_{t,f}^{N}}\sum_{t}m_{t,f}^{N}\mathbf{y}_{t,f}\mathbf{y}_{t,f}^{H},$
(4)

where $m_{t,f}^{S}$ and $m_{t,f}^{N}$ refer to speech and noise mask. The
filter coefficients of the MVDR for speech $w_{speech}$ are derived as follows

$\mathbf{w}_{f}=\frac{\left(\mathbf{R}_{f}^{N}\right)^{-1}\mathbf{R}_{f}^{S}}{\operatorname{tr}\left(\left(\mathbf{R}_{f}^{N}\right)^{-1}\mathbf{R}_{f}^{S}\right)}\mathbf{u}_{f},$
(5)

where tr(·) is the matrix trace operation and $u_{f}$ is a one-hot vector
indicating the reference microphone. Thus, we can get the final enhanced
result for speech via

$\hat{\mathbf{y}}_{t,f}=\mathbf{w}_{f}^{\mathrm{H}}\mathbf{y}_{t,f}.$ (6)

We use the enhanced result as the input for following keyword spotting to
improve the robustness of the system.

###  2.2 Cross-Modal Feature Learning

####  2.2.1 Uni-Modal Feature Extraction

The speaker’s lip movements in videos carry information pertinent to both
appearance and temporal dynamics. It’s important to efficiently model these
spatial and temporal cues when building a robust KWS system. ResNet3D proposed
in [19] is proven to be effective in capturing the spatio-temporal dynamics in
videos and has been widely used in different tasks, like video action
recognition [20], speaker identification [21], etc. Therefore, we build our
feature extractor based on ResNet3D and adopt the same architecture for
different modalities in order to keep the consistency of feature dimensions.

For visual modality, the lip regions from video frames are regarded as the
input, while for audio modality, the log-Mel Filterbank (FBank) feature of
each frame is spliced with adjacent context frames to form a 2-D input
feature.

####  2.2.2 Attention-based Cross-Modal Feature Fusion

For the task of KWS, audio modality is more reliable than visual modality
under a silent environment since the audio-based systems usually outperform
their visual-based counterparts. But the system based on audio may perform
badly in a noisy environment, while visual modality is not affected by
acoustic noise. Therefore, it is essential to leverage diverse and
complementary relationships over multiple modalities in order to improve the
robustness of the KWS system. However, the features extracted from different
modalities have diverse characteristics, which makes the simple concatenation
strategy the best choice. In order to reliably fuse these modalities to KWS,
we first utilize a cross-attention mechanism to enhance the representation
learning of each modality by incorporating information from other modalities.
Then, the enhanced representations are fused by a bilinear layer and multiple
residual projection layers. A block diagram of the proposed method is shown in
2.

Fig. 2: Block diagram of the proposed audio-visual fusion model.

We employ a Transformer layer, which is efficient in extracting context-
dependent representations [22], on the output of ResNet3D. Our Transformer
layer consists of a self-attention module and a cross-attention module. The
self-attention is employed to model the intra-modal dependencies, while the
cross-attention regards the representation of other modalities as the query
vector to capture the inter-model relationship. After that, we adopt a fusion
method similar to [9] to further fuse two modalities.

Let $\mathbf{X}_{a}$ and $\mathbf{X}_{v}$ represent the enhanced audio and
visual representations extracted from their Transformer layers, we first
concatenate them on the last dim to form the joint representation
$\mathbf{J}$. After that, we use bilinear pooling, a powerful method for
fusing multi-modal representations [23], to fuse the joint representation and
uni-modal representation via

$\displaystyle\mathbf{C}_{a}=\tanh\left(\frac{\mathbf{X}_{a}^{T}\mathbf{W}_{ja}\mathbf{J}}{\sqrt{d}}\right),$
(7)
$\displaystyle\mathbf{C}_{\mathbf{v}}=\tanh\left(\frac{\mathbf{X}_{v}^{T}\mathbf{W}_{jv}\mathbf{J}}{\sqrt{d}}\right),$
(8)

where $\mathbf{W}_{ja}$ and $\mathbf{W}_{jv}$ are projecting weight matrices
for different modalities. The obtained $\mathbf{C}_{a}$ and $\mathbf{C}_{v}$
measure the relevance not only across different modalities but also within the
same modality. Then, we combine them with their corresponding uni-modal
representations and get the fusion features $\mathbf{X}_{j,a}$ and
$\mathbf{X}_{j,v}$. Take the audio feature $\mathbf{X}_{j,a}$ as an example,
the computational procedure can be summarized as

$\displaystyle\mathbf{H}_{a}$
$\displaystyle=\text{ReLU}\left(\mathbf{W}_{a}\mathbf{X}_{a}+\mathbf{W}_{ca}\mathbf{C}_{a}^{T}\right),$
(9) $\displaystyle\mathbf{X}_{j,a}$
$\displaystyle=\mathbf{W}_{ha}\mathbf{H}_{a}+\mathbf{X}_{a},$ (10)

where $\mathbf{W}_{a}$, $\mathbf{W}_{ca}$, and $\mathbf{W}_{ha}$ are learnable
weight matrices. $\mathbf{H}_{a}$ can be denoted as the attention maps of
audio modality and $\mathbf{X}_{j,a}$ is the attended features of audio
modality. After computing $X_{j,v}$ via a similar process, we finally
concatenate it with $\mathbf{X}_{j,a}$ to form the audio-visual representation
for predicting the keyword existence.

##  3 Experiments

###  3.1 Database and Evaluation Metrics

We evaluate our proposed system on the audio-visual wake word spotting (WWS)
database released in the MISP2021 Challenge. The database contains about 125
hours of audio-visual data recorded in over 30 real-world rooms. The data has
three subsets: a training set (47k+ negative samples and 5K+ positive
samples), a development set (2k+ negative samples and 600+ positive samples),
and an evaluation set (8K+ samples). The audio collection devices include a
6-channel far-field microphone array, a 2-channel mid-field microphone array,
and a single-channel near-field microphone, while the video collection devices
include a mid-field high-definition camera and a far-field wide-angle camera.

In this case, if the wake-up word “Xiao T, Xiao T” is included in the sample,
it is considered a positive sample, otherwise, it will be regarded as a
negative sample. Following the MISP challenge setup [12], the combination of
false reject rate (FRR) and false alarm (FAR) is adopted to evaluate the
systems, as follows:

$\text{ Score }=\text{FRR}+\text{FAR}=\frac{N_{\text{FR}}}{N_{\text{wake
}}}+\frac{N_{\text{FA}}}{N_{\text{non-wake }}},$ (11)

where ${N_{\text{wake}}}$ and ${N_{\text{non-wake}}}$ denote the number of
samples including wake-up word or without wake-up word. ${N_{\text{FR}}}$
denotes the number of samples that include the wake word but are not detected
by the system and ${N_{\text{FA}}}$ is the number of samples that do not
contain the keyword but are detected erroneously. The lower the score, the
better the system performance.

###  3.2 Setups of the Video-Assisted MVDR

The mask estimate network consists of three 256-dim long short-term memory
(LSTM) layers, followed by a full connection (FC) layer and a Sigmoid
activation function to estimate IRM. Since the view width of the far-field
camera is 120 degrees, we equally divide the video frame into six parts
horizontally, and each region corresponds to a beam of 20 degrees. The beam
index for the input sample is got through the ROI released with the MISP
dataset, which is cropped by the internal tools provided by the challenge
organizer.

For the training of our mask estimate network, we use the far-field data of
the challenge as input and the near-field data as a target. IPD features are
calculated among three microphone pairs with indexes (1,4), (2,5), and (3,6).
The frame length and hop length used in the short-time Fourier transform
(STFT) are set to 32 and 16 ms, respectively. The network is trained for 20
epochs with the Adam optimizer. The initial learning rate is set to 0.001 and
will be halved if there is no improvement in the development set.

###  3.3 Uni-modal Model Pre-training

Audio processing. Firstly, we extract 80-dimensional log-Mel Filterbank
(FBank) features with 25ms frame length and 10ms frame shift from raw waves.
Because the frame length of the video is about 4 times larger than the speech,
we use a window of 80 frames sliding along the time axis with a stride of 4 to
form a sequence of feature blocks. The shape of feature blocks is
$(T,80,80,1)$, where $T$ refers to the number of feature blocks and $1$ means
the channel dimension. In addition, speed perturbation, pitch shifting and
noise augmentation are also applied as the data augmentation strategies during
the training to improve the robustness of the model.

Visual processing. We crop the lip regions of the videos according to the ROI
provided by MISP organizers and resize them to have a resolution of $112\times
112$ with 3 RGB channels. The shape of visual features is $(T,112,112,3)$,
where $T$ refers to the number of time steps and $3$ means the channel
dimension. We also use some video-based data augmentation techniques,
including frame-wise rotation, flipping horizontally, and color
transformation.

Model training. We employ the same network architecture for both modalities,
which consists of five ResNet3D blocks, whose channels are 32, 64, 64, 128,
and 256 respectively. Both models are trained for 15 epochs with Adam
optimizer, where the initial learning rate is set to 0.001 and will halve if
there is no improvement on the development set. To tackle the imbalance
between positive and negative samples, we adopt the weighted CrossEntropy loss
(negative:positive=1:5) with the label smoothing method [24].

###  3.4 Joint Fusion Training

During the training of our VE-KWS model, the modality-exclusive parts are
initialized by the pre-trained uni-modal models, and the fusion module is
randomly initialized. Most of the training setups are the same as Sec 3.3,
except the initial learning rate is $1e^{-4}$. In order to increase the
diversity of training data, we randomly replace the video with another video
in the same class to simulate more audio-visual input pairs.

###  3.5 Results and Discussion

Table 1 shows ablation studies of the audio-modality model. The baseline
represents the model trained without any extra tricks. For the BF method, we
estimate the arrival direction through the lip region coordinates and then
perform fixed beamforming. Compared with the above methods, our model gives a
significant improvement, leading up to 36.50%/12.02% relative score reduction
over the baseline and 24.12%/7.59% over the fixed beamforming system on the
Dev/Eval sets, respectively. Furthermore, by incorporating the data
augmentation methods into training, we obtain the best performance of 9.78% on
the Eval set but a slight degradation on the Dev set.

Table 1: Experimental results of the audio-modality models. | Method | Dev( %) | Eval( %)
---|---|---|---
| FRR | FAR | Score | FRR | FAR | Score
| Baseline | 6.89 | 8.07 | 14.96 | 8.91 | 5.48 | 14.39
| \+ BF | 8.51 | 4.01 | 12.52 | 7.92 | 5.78 | 13.70
| \+ audio-only MVDR | 7.52 | 3.97 | 11.49 | 7.61 | 5.52 | 13.13
| \+ proposed MVDR | 6.09 | 3.41 | 9.50 | 9.42 | 3.22 | 12.64
|  +Data Aug. | 6.73 | 3.12 | 9.85 | 5.82 | 3.95 | 9.78

Table 2 are the ablation studies of the visual-modality model. From the table,
we can find that data augmentation can reduce the score significantly.
Besides, when using the additional CAS-VSR-W1k dataset [25] for pre-training,
we can observe further improvement in both sets. However, the results of the
visual-modality model have a large gap with the audio-modality system, which
verifies our statement mentioned in Sec 2.2.2.

Table 2: Experimental results of the visual-modality models. | Method | Dev( %) | Eval( %)
---|---|---|---
| FRR | FAR | Score | FRR | FAR | Score
| Baseline | 9.96 | 12.78 | 22.74 | 18.91 | 12.30 | 31.21
| +Data Aug. | 5.45 | 10.14 | 15.59 | 15.94 | 10.43 | 26.37
|  +Pre-train | 6.73 | 6.68 | 13.41 | 18.39 | 7.62 | 26.01

Table 3 presents the ablation studies of the audio-visual model. We compare
our fusion method with the classic fusion method on the feature level and
decision level, respectively. For feature-level fusion, the outputs of uni-
modal models are concatenated and fed into fully-connected layers to make the
final decision. For decision-level fusion, we simply average the score
predicted by each uni-modal model and make the final decision. With the help
of the proposed methods, our model shows superior performance over feature- or
decision-level fusion models on both sets, achieving up to 44.31% and 33.10%
relative score reductions, respectively.

Table 3: Experimental results of the multi-modal fusion models. | Method | Dev( %) | Eval( %)
---|---|---|---
| FRR | FAR | Score | FRR | FAR | Score
| Decision Fusion | 1.60 | 4.90 | 6.50 | 2.76 | 5.82 | 8.58
| Feature Fusion | 2.24 | 2.74 | 4.98 | 3.80 | 2.85 | 6.65
| VE-KWS (Proposed) | 1.60 | 2.02 | 3.62 | 2.79 | 2.95 | 5.74

To detail analyze the system performance of different noisy environments, we
averagely split the Eval set into three subsets according to the BAK of Dnsmos
[26]. From the result in Table 4, we can find that the audio-modality system
is susceptible to performance degradation when noises exist, resulting in a
large gap between noisy and clean sets. However, the visual-modality system is
not affected by the noise, performing similarly on different subsets. When
fusing audio and visual modalities, our model indeed overcomes the degradation
caused by noisy conditions. Table 5 shows the comparison with other studies in
the challenge and our model outperforms the top-ranking systems resulting in a
new SOTA performance.

Table 4: Experimental results on the Eval subsets. | Method | Score( %)
---|---|---
| All | Noisy | Medium | Clean
| Audio | 9.78 | 12.29 | 8.77 | 8.31
| Visual | 26.01 | 23.78 | 28.41 | 26.00
| VE-KWS (Proposed) | 5.74 | 7.18 | 5.61 | 4.47
Table 5: Results comparison with other released systems. Model |  MISP Rank | Score( %)
---|---|---
Dev | Test
CNN-TDNNF+MMI [12] | $1^{th}$ | 4.1 | 5.8
ResNet+HMA [13] | $2^{nd}$ | 7.27 | 7.1
Transformer+Fusion [14] | $3^{rd}$ | - | 9.1
VE-KWS (Proposed) | - | 3.62 | 5.74

##  4 Conclusion

In this work, we propose a novel visual modality enhanced end-to-end KWS
framework (VE-KWS), which utilizes audio-visual information to improve the
performance of KWS system. Specifically, the model consists of a visual-
assisted MVDR that utilizes the lip region information to provide speaker
direction information for the beamformer and a joint audio-visual feature
fusion model that leverages complementary relationships over multiple
modalities. Experimental results on the MISP challenge show that our model
achieves a 2.79% false rejection rate and 2.95% false alarm rate on the Eval
set, outperforming other submitted systems and resulting in a new SOTA
performance. Although the MISP challenge does not restrict the real-time
factor (RTF) and model size for submissions, small footprint, and real-time
processing are essential for a practical KWS application. Thus, we would like
to explore audio-visual KWS systems with low RTF and small footprint in our
future work.

## References

  * [1] Aleksandr Laptev, Roman Korostik, Aleksey Svischev, Andrei Andrusenko, Ivan Medennikov, and Sergey Rybin,  “You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation,”  in Proc. CISP-BMEI. 2020, pp. 439–444, IEEE.
  * [2] Guoguo Chen, Carolina Parada, and Georg Heigold,  “Small-footprint keyword spotting using deep neural networks,”  in Proc. ICASSP. 2014, pp. 4087–4091, IEEE.
  * [3] Yiming Wang, Hang Lv, Daniel Povey, Lei Xie, and Sanjeev Khudanpur,  “Wake word detection with streaming transformers,”  in Proc. ICASSP. 2021, pp. 5864–5868, IEEE.
  * [4] Jingyong Hou, Yangyang Shi, Mari Ostendorf, Mei-Yuh Hwang, and Lei Xie,  “Mining effective negative training samples for keyword spotting,”  in Proc. ICASSP. 2020, pp. 7444–7448, IEEE.
  * [5] Yiteng Arden Huang, Turaj Zakizadeh Shabestary, and Alexander Gruenstein,  “Hotword cleaner: Dual-microphone adaptive noise cancellation with deferred filter coefficients for robust keyword spotting,”  in Proc. ICASSP. 2019, pp. 6346–6350, IEEE.
  * [6] Emre Yilmaz, Özgür Bora Gevrek, Jibin Wu, Yuxiang Chen, Xuanbo Meng, and Haizhou Li,  “Deep convolutional spiking neural networks for keyword spotting,”  in Proc. Interspeech. 2020, pp. 2557–2561, ISCA.
  * [7] Meng Yu, Xuan Ji, Bo Wu, Dan Su, and Dong Yu,  “End-to-end multi-look keyword spotting,”  in Proc. Interspeech. 2020, pp. 66–70, ISCA.
  * [8] Dianwen Ng, Yunqi Chen, Biao Tian, Qiang Fu, and Eng Siong Chng,  “Convmixer: Feature interactive convolution with curriculum learning for small footprint and noisy far-field keyword spotting,”  in Proc. ICASSP. 2022, pp. 3603–3607, IEEE.
  * [9] R. Gnana Praveen, Wheidima Carneiro de Melo, Nasib Ullah, Haseeb Aslam, Osama Zeeshan, Théo Denorme, Marco Pedersoli, Alessandro L. Koerich, Simon Bacon, Patrick Cardinal, and Eric Granger,  “A joint cross-attention model for audio-visual fusion in dimensional emotion recognition,”  in Proc. CVPR. 2022, pp. 2485–2494, IEEE.
  * [10] Han Lei and Ning Chen,  “Audio-visual scene classification based on multi-modal graph fusion,”  in Proc. Interspeech. 2022, pp. 4157–4161, ISCA.
  * [11] Liliane Momeni, Triantafyllos Afouras, Themos Stafylakis, Samuel Albanie, and Andrew Zisserman,  “Seeing wake words: Audio-visual keyword spotting,”  in Proc. BMVC, 2020.
  * [12] Hang Chen, Hengshun Zhou, Jun Du, Chin-Hui Lee, Jingdong Chen, Shinji Watanabe, Sabato Marco Siniscalchi, Odette Scharenborg, Diyuan Liu, Bao-Cai Yin, Jia Pan, Jianqing Gao, and Cong Liu,  “The first multimodal information based speech processing (misp) challenge: Data, tasks, baselines and results,”  in Proc. ICASSP. 2022, pp. 9266–9270, IEEE.
  * [13] Ming Cheng, Haoxu Wang, Yechen Wang, and Ming Li,  “The DKU audio-visual wake word spotting system for the 2021 MISP challenge,”  in Proc. ICASSP. 2022, pp. 9256–9260, IEEE.
  * [14] Yanguang Xu, Jianwei Sun, Yang Han, Shuaijiang Zhao, Chaoyang Mei, Tingwei Guo, Shuran Zhou, Chuandong Xie, Wei Zou, and Xiangang Li,  “Audio-visual wake word spotting system for MISP challenge 2021,”  in Proc. ICASSP. 2022, pp. 9246–9250, IEEE.
  * [15] Zhong-Qiu Wang and DeLiang Wang,  “Combining spectral and spatial features for deep learning based blind speaker separation,”  IEEE ACM Trans. Audio Speech Lang. Process., vol. 27, no. 2, pp. 457–468, 2019.
  * [16] Rongzhi Gu, Lianwu Chen, Shi-Xiong Zhang, Jimeng Zheng, Yong Xu, Meng Yu, Dan Su, Yuexian Zou, and Dong Yu,  “Neural spatial filter: Target speaker speech separation assisted with directional information,”  in Proc. Interspeech. 2019, pp. 4290–4294, ISCA.
  * [17] Fahimeh Bahmaninezhad, Jian Wu, Rongzhi Gu, Shi-Xiong Zhang, Yong Xu, Meng Yu, and Dong Yu,  “A comprehensive study of speech separation: Spectrogram vs waveform separation,”  in Proc. Interspeech. 2019, pp. 4574–4578, ISCA.
  * [18] Takuya Yoshioka, Yan Huang, Aviv Hurvitz, Li Jiang, Sharon Koubi, Eyal Krupka, Ido Leichter, Changliang Liu, Partha Parthasarathy, Alon Vinnikov, Lingfeng Wu, Igor Abramovski, Xiong Xiao, Wayne Xiong, Huaming Wang, Zhenghao Wang, Jun Zhang, Yong Zhao, Tianyan Zhou, Cem Aksoylar, Zhuo Chen, Moshe David, Dimitrios Dimitriadis, Yifan Gong, Ilya Gurvich, and Xuedong Huang,  “Advances in online audio-visual meeting transcription,”  in Proc. ASRU. 2019, pp. 276–283, IEEE.
  * [19] Zhaofan Qiu, Ting Yao, and Tao Mei,  “Learning spatio-temporal representation with pseudo-3d residual networks,”  in Proc. ICCV. 2017, pp. 5534–5542, IEEE Computer Society.
  * [20] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh,  “Learning spatio-temporal features with 3d residual networks for action recognition,”  in Proc. ICCV. 2017, pp. 3154–3160, IEEE Computer Society.
  * [21] Danwei Cai, Xiaoyi Qin, and Ming Li,  “Multi-channel training for end-to-end speaker recognition under reverberant and noisy environment,”  in Proc. Interspeech. 2019, pp. 4365–4369, ISCA.
  * [22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Proc. NIPS, 2017, pp. 5998–6008.
  * [23] Joshua B Tenenbaum and William T Freeman,  “Separating style and content with bilinear models,”  Neural computation, vol. 12, no. 6, pp. 1247–1283, 2000.
  * [24] Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton,  “When does label smoothing help?,”  in Proc. NIPS, 2019, pp. 4696–4705.
  * [25] Shuang Yang, Yuanhang Zhang, Dalu Feng, Mingmin Yang, Chenhao Wang, Jingyun Xiao, Keyu Long, Shiguang Shan, and Xilin Chen,  “LRW-1000: A naturally-distributed large-scale benchmark for lip reading in the wild,”  in Proc. FG. 2019, pp. 1–8, IEEE.
  * [26] Chandan KA Reddy, Vishak Gopal, and Ross Cutler,  “Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,”  in Proc. ICASSP. 2021, pp. 6493–6497, IEEE.