Subash<EMAIL_ADDRESS>Srikumar<EMAIL_ADDRESS>Aayush
<EMAIL_ADDRESS>Nathan<EMAIL_ADDRESS>Computer Science &
Engineering
Washington University in St. Louis
St. Louis, MO, USA GeoCLAP

# Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping

###### Abstract

We focus on the task of soundscape mapping, which involves predicting the most
probable sounds that could be perceived at a particular geographic location.
We utilise recent state-of-the-art models to encode geotagged audio, a textual
description of the audio, and an overhead image of its capture location using
contrastive pre-training. The end result is a shared embedding space for the
three modalities, which enables the construction of soundscape maps for any
geographic region from textual or audio queries. Using the SoundingEarth
dataset, we find that our approach significantly outperforms the existing
SOTA, with an improvement of image-to-audio Recall@100 from $0.256$ to
$0.450$. Our code is available at https://github.com/mvrl/geoclap.

##  1 Introduction

Sound is one of the fundamental senses that helps us reason about our
environment. There exists an intricate relationship between the visual
appearance and sound of a location [González et al.(2023)González, Morillas,
and Rey-Gozalo, Garzón et al.(2023)Garzón, Bravo-Moncayo, Arellana, and de
Dios Ortúzar]. Learning about the type of sound at a geographic location
allows one to understand many high-level concepts of the area. For example,
just by hearing the sound of traffic, we can imagine the location to be an
urban setting with a rush of cars and people, whereas the sound of sea waves
might elicit the beautiful scenery of a beach.

There have been several studies conducted on different cities around the world
attempting to understand human perception of various types of environmental
sound [Aramaki and Wakamiya(2023), Garzón et al.(2023)Garzón, Bravo-Moncayo,
Arellana, and de Dios Ortúzar, Ooi et al.(2023)Ooi, Ong, Watcharasupat, Lam,
Hong, and Gan, Picaut et al.(2019)Picaut, Fortin, Bocher, Petit, Aumond, and
Guillaume, Lionello et al.(2020)Lionello, Aletta, and Kang, Aiello et
al.(2016)Aiello, Schifanella, Quercia, and Aletta]. Moreover, it has been
established that there is a strong correlation between the physiological and
psychological health of a person and the environmental sound condition they
live in [Lercher and Dzhambov(2023), Cui et al.(2022)Cui, Li, Xia, and Dai,
Radicchi et al.(2021)Radicchi, Cevikayak Yelmi, Chung, Jordan, Stewart,
Tsaligopoulos, McCunn, and Grant]. Therefore, understanding the soundscape for
a given geographic area can be of great importance to policymakers focused on
urban planning and environmental noise management. Soundscapes also serve
value to the general public for whom environmental sound plays a vital role in
decisions such as buying a house or setting up a business.

Most of the existing works on creating soundscape focus on crowd-sourcing
human perception of sound in their surroundings [Aiello et al.(2016)Aiello,
Schifanella, Quercia, and Aletta, Lionello et al.(2020)Lionello, Aletta, and
Kang, Picaut et al.(2019)Picaut, Fortin, Bocher, Petit, Aumond, and Guillaume,
Aramaki and Wakamiya(2023), Yue et al.(2023)Yue, Meng, Yang, Wu, Liu, and
Yan]. While serving as an important tool for understanding the sound
distribution of a region, such approaches have two major limitations. First,
the abstraction of sound into a fixed set of indicators and psycho-acoustic
descriptors limits our ability to have a complete picture of underlying
physical factors associated with sound. Second, such soundscapes are usually
created for only highly visited places in the world, creating massive sparsity
of soundscapes on a global scale. In order to solve both of these limitations,
we propose to leverage the intrinsic relationship between sound and visual
cues of the location and learn to directly predict the most probable sound
that could be heard at any given location. Specifically, we train a multi-
modal deep learning framework that learns a shared embedding space where the
sound that is most likely to come from a given location, is pulled closer
while pushing other unlikely sounds farther apart. We represent the location
(latitude, longitude) by an overhead image of size $H\times W$ centered around
it. Once trained, our multi-modal embedding space and free availability of
overhead imagery makes it possible for us to create soundscape maps for any
area in the world.

One of the successful approaches to learning shared embedding space between
different modalities is contrastive learning. In recent years, contrastive
learning between image and text [Radford et al.(2021)Radford, Kim, Hallacy,
Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.]; image, text,
and audio [Guzhov et al.(2022)Guzhov, Raue, Hees, and Dengel]; text, audio
[Elizalde et al.(2023)Elizalde, Deshmukh, Al Ismail, and Wang, Deshmukh et
al.(2022)Deshmukh, Elizalde, and Wang, Wu* et al.(2023)Wu*, Chen*, Zhang*,
Hui*, Berg-Kirkpatrick, and Dubnov]; overhead image and audio [Heidler et
al.(2023)Heidler, Mou, Hu, Jin, Li, Gan, Wen, and Zhu] has been an effective
self-supervised training objective to learn a multi-modal embedding space.
Such a space has an understanding of the correspondence between the modalities
that can be transferred to various downstream tasks, where impressive results
have been observed. Motivated by these works, we also adopt contrastive
learning as our pre-training strategy to learn a multi-modal embedding space.
However, unlike the prior works, we are interested in incorporating geographic
knowledge into the embedding space learned by audio-language pre-training. We
achieve this by adding an overhead image, capturing the geographic context of
a scene, as an additional modality in our contrastive learning framework. With
the shared embedding space that has knowledge of correspondence between audio
and its corresponding overhead image, we can then formulate the task of
soundscape mapping as a cross-modal retrieval problem, where the objective is
to predict the most likely sound from a gallery of $N$ sounds given an
overhead image.

Our work builds upon a prior work [Heidler et al.(2023)Heidler, Mou, Hu, Jin,
Li, Gan, Wen, and Zhu] that introduced the SoundingEarth dataset containing
over $50k$ geotagged audios paired with their corresponding overhead image.
The objective of work by Heidler _et al_ [Heidler et al.(2023)Heidler, Mou,
Hu, Jin, Li, Gan, Wen, and Zhu] was to learn a good audio-visual embedding
space useful to be transferred for different downstream tasks in remote
sensing. However, in the interest of learning an embedding space to create
accurate soundscapes, our work is focused on improving the task of cross-modal
retrieval. In this regard, we utilise weights of publicly available modality-
specific SOTA models. Moreover, unlike Heidler _et al_ , who build an
embedding space capturing two modalities (overhead-image and audio), we
propose to also incorporate textual description of audio into the embedding
space. This essentially creates a tri-modal embedding space with richer
understanding of three modalities: overhead-image, audio, and text. We call
our framework GeoCLAP: Geography-Aware Contrastive Language Audio Pre-
training. As demonstrated by our results adding the textual modality improves
the representational capability of both overhead-image and audio encoders.
Moreover, with an understanding of three modalities, we are now able to create
soundscapes either from a textual or audio query for any geographic region.
The main contributions of our work are as follows:

  * •

We significantly improve the prior baseline on the task of cross-modal
retrieval of overhead image to sound and vice-versa.

  * •

We build a tri-modal embedding space that has an understanding of overhead
image, audio, and textual description of audio at a given location.

  * •

We demonstrate a simple and scalable way of creating soundscape for any
geographic area using either a textual or audio query.

##  2 Related Work

###  2.1 Soundscape Mapping

The soundscape of a geographic region can be defined as the acoustic
environment perceived by individuals within its context [for
Standardization(2014)]. There exists a large body of work focusing on the
problem of soundscape mapping [Aiello et al.(2016)Aiello, Schifanella,
Quercia, and Aletta, Lionello et al.(2020)Lionello, Aletta, and Kang, Picaut
et al.(2019)Picaut, Fortin, Bocher, Petit, Aumond, and Guillaume, Margaritis
and Kang(2017), Aramaki and Wakamiya(2023), Engel et al.(2021)Engel, Fiebig,
Pfaffenbach, and Fels, González et al.(2023)González, Morillas, and Rey-
Gozalo, Ooi et al.(2023)Ooi, Ong, Watcharasupat, Lam, Hong, and Gan, Zhao et
al.(2023)Zhao, Liang, Tu, Huang, and Biljecki, Yue et al.(2023)Yue, Meng,
Yang, Wu, Liu, and Yan]. In these works, soundscape mapping is formulated as a
framework containing three components: indicators, descriptors, and a
predictive model that maps indicators to descriptors. Indicators are psycho-
acoustic measures (for example, sound pressure level, loudness, spectral
slope, etc.) which determine the perceived value of descriptors (for example,
pleasant, unpleasant, eventful, etc.). In this paper, we refer to this line of
work as perceptual soundscape mapping.

One of the common findings from the literature of perceptual soundscape
mapping is that there exists a strong correlation between the human perception
of sound and the environmental variables of the scene such as building, road
category, etc. [Garzón et al.(2023)Garzón, Bravo-Moncayo, Arellana, and de
Dios Ortúzar]. Utilising this correlation between sound and visual cues, there
have been a few works that use deep learning to learn a shared embedding space
between sound and either ground level image [Owens et al.(2016)Owens, Wu,
McDermott, Freeman, and Torralba] or overhead image [Hu et al.(2020)Hu, Li,
Mou, Jin, Chen, Jing, Zhu, and Dou] of the scene. This multimodal learning
approach leads to improved performance on visual tasks such as aerial scene
recognition [Hu et al.(2020)Hu, Li, Mou, Jin, Chen, Jing, Zhu, and Dou], image
classification [Owens et al.(2016)Owens, Wu, McDermott, Freeman, and
Torralba], and object detection [Owens et al.(2016)Owens, Wu, McDermott,
Freeman, and Torralba]. Closer to our work, a few prior works [Mao et
al.(2018)Mao, Yuan, and Xiaoqiang, Chen et al.(2020)Chen, Lu, and Wang, Ning
et al.(2021)Ning, Zhao, and Yuan, Yang et al.(2022)Yang, Wang, Sun, Zhang,
Liao, Gu, Hou, and Jiao] focus on the task of cross-modal image-to-voice
retrieval. Such tasks require a dataset containing overhead imagery paired
with spoken audio captions, which is very limited. Moreover, instead of
learning from speech, we are interested in learning from free-form audio such
as field recordings, natural sounds, etc. which capture diverse concepts of
the location. Another closer work by Salem _et al_ [Salem et al.(2018)Salem,
Zhai, Workman, and Jacobs], proposed learning a shared embedding space between
audio, overhead image, and ground level image, enabling them to predict a
distribution over sound clusters from an overhead image. The problem
formulation of soundscape mapping in our work is similar to [Salem et
al.(2018)Salem, Zhai, Workman, and Jacobs]. However, the striking difference
as well as the strength of our work is that leveraging the power of
contrastive language audio pre-training (CLAP), we are able to create
soundscape conditioned on any textual or audio query. In doing so, we still
retain the ability to create soundscape with desired set of sound categories
in a zero-shot manner.

###  2.2 Contrastive Learning

Radford _et al_ , in their seminal work, CLIP [Radford et al.(2021)Radford,
Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.],
trained large image-text dataset using contrastive loss and demonstrated it’s
impressive zero-shot performance on many computer vision tasks. AudioCLIP
[Guzhov et al.(2022)Guzhov, Raue, Hees, and Dengel], extends CLIP to three
modalities: image, text, and audio. Such tri-modal embedding space enables one
to perform query between three pairs of modalities. Wav2clip [Wu et
al.(2022)Wu, Seetharaman, Kumar, and Bello], distilled the knowledge of CLIP
embedding space by freezing the image encoder of CLIP and contrastively
training an audio encoder to learn a new embedding space shared by audio and a
corresponding image. With similar training objective as CLIP, another work
CLAP [Elizalde et al.(2023)Elizalde, Deshmukh, Al Ismail, and Wang] performs
contrastive learning between audio and natural language. CLAP training has
proven to be an effective strategy with impressive audio retrieval performance
[Deshmukh et al.(2022)Deshmukh, Elizalde, and Wang]. Inspired by this, Wu _et
al_ [Wu* et al.(2023)Wu*, Chen*, Zhang*, Hui*, Berg-Kirkpatrick, and Dubnov]
further improved the CLAP’s performance by training on large-scale data with
effective audio feature fusion and text augmentation strategies. We refer the
work by Wu _et al_ [Wu* et al.(2023)Wu*, Chen*, Zhang*, Hui*, Berg-
Kirkpatrick, and Dubnov] as L-CLAP in our paper and use the pre-trained
encoders from L-CLAP to embed audio and text for GeoCLAP pre-training.

Our work takes motivation from the proven performance of contrastive learning
as an effective pre-training strategy. The focus of our work is soundscape
mapping. The embedding space for such tasks should have an understanding of
geography of a location where the sound is coming from [Ayush et
al.(2021)Ayush, Uzkent, Meng, Tanmay, Burke, Lobell, and Ermon]. Therefore, we
propose to learn an embedding space trained contrastively on three modalities:
overhead image, text, and audio.

###  2.3 Pretrained Models

Availability of modality specific pre-trained models trained with various
self-supervision objectives have proven to be crucial in bringing performance
improvement in various tasks in remote sensing [Wang et al.(2022)Wang,
Albrecht, Braham, Mou, and Zhu]. In the recent years, masked auto-encoders
(MAE) [He et al.(2022)He, Chen, Xie, Li, Dollár, and Girshick] based models
trained on satellite imagery have demonstrated to be a good starting
checkpoints to be fine-tuned for various downstream tasks [Cong et
al.(2022)Cong, Khanna, Meng, Liu, Rozi, He, Burke, Lobell, and Ermon, Reed et
al.(2022)Reed, Gupta, Li, Brockman, Funk, Clipp, Candido, Uyttendaele, and
Darrell]. In our work, we start with the pre-trained weights of Vision
Transformer (ViT) [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov,
Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit,
and Houlsby] encoder of SATMAE [Cong et al.(2022)Cong, Khanna, Meng, Liu,
Rozi, He, Burke, Lobell, and Ermon] as the overhead-image encoder for GeoCLAP.
SATMAE [Cong et al.(2022)Cong, Khanna, Meng, Liu, Rozi, He, Burke, Lobell, and
Ermon] was pre-trained on large-scale (over 700K) satellite imagery of the
world. To learn representations for audio and text, we use L-CLAP’s pre-
trained encoders. It uses HTSAT [Chen et al.(2022)Chen, Du, Zhu, Ma, Berg-
Kirkpatrick, and Dubnov] as the audio encoder and RoBERTa [Liu et
al.(2019)Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer, and
Stoyanov] as the text encoder. HTSAT is a swin-transformer [Liu et
al.(2021)Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and Guo] based model with SOTA
performance on various audio classification tasks. RoBERTa is a powerful
transformer-based language model trained with improved design choices than
BERT [Devlin et al.(2018)Devlin, Chang, Lee, and Toutanova]. L-CLAP [Wu* et
al.(2023)Wu*, Chen*, Zhang*, Hui*, Berg-Kirkpatrick, and Dubnov] was
contrastively pre-trained on over 630K audio-text paired dataset.

##  3 Approach

We present a detailed description of our approach, including the high-level
problem formulation, a description of our primary evaluation dataset, and a
detailed description of the network architecture and training procedure for
our method, GeoCLAP.

###  3.1 Problem formulation

The objective of our work is to learn a shared embedding space that allows us
to predict the most probable sounds that can be heard at a given geographic
location. This can be represented as $s^{*}=\max_{s}P(s|l)$ where $P(s|l)$
represents the conditional distribution of sounds for a given location $l$ and
$s^{*}$ is the most likely sound. Unfortunately, direct conditioning on
location does not generalize to regions without a large number of training
samples, which means truly global mapping wouldn’t be possible. On the other
hand, overhead imagery has a strong correlation to the type of sound at a
given location and is freely available across the globe. Therefore, in our
work, we represent the location indirectly, using an overhead image $I(l)$ of
the location. We learn a conditional distribution $P(s|I(l))$, which is able
to make high-resolution predictions even for regions without training samples.

###  3.2 Dataset

We use the SoundingEarth dataset to train and evaluate our method. The dataset
contains more than $50k$ geotagged audio recordings from $136$ countries and
overhead image pairs. The overhead images have size of ${1024}\times{1024}$
collected from Google Earth with an approximate ground-sample distance (GSD)
of $0.2$ meters (m). Audio data in the dataset was collected from the project
Radio Aporee:::Maps [Aporee()], which hosts an online platform dedicated to
creating a global soundmap. It contains diverse audio recordings from urban,
rural and natural environments, published under the creative commons license.
For our project, we remove the audio files with a sampling frequency less than
$16k$. This yields a dataset size of $50\,792$ samples.

The high-resolution Google Earth imagery is not available to be used freely.
Therefore, in order to have the ability to globally scale soundscape mapping,
we augment the existing SoundingEarth dataset by including freely available
lower-resolution images. Specifically, we use the RGB bands of the Sentinel-2
cloudless imagery with $10m$ $GSD$. For each location, we download a
$256\times 256$ image tile with the coverage radius of $512m$ centered at that
location.

###  3.3 GeoCLAP

Figure 1 represents the overall framework of GeoCLAP. Given a geotagged audio
$X_{k}^{a}$, textual description of the audio $X_{k}^{t}$, and an overhead
image at a given location $X_{k}^{i}$, where
($X_{k}^{a}$,$X_{k}^{t}$,$X_{k}^{i}$) is one audio-text-image triplet. We
obtain embeddings for each modalities by passing through modality-specific
encoder and linear projection layer, yielding embeddings with the same
dimension for audio, text, and overhead image, respectively.

Figure 1: GeoCLAP: A tri-modal contrastive learning framework to learn shared
embedding space between overhead image, sound, and textual description of the
corresponding sound.

$E_{k}^{a}=g_{audio}(f_{audio}(X_{k}^{a}))$ (1)
$E_{k}^{t}=g_{text}(f_{text}(X_{k}^{t}))$ (2)
$E_{k}^{i}=g_{image}(f_{image}(X_{k}^{i}))$ (3)

where $(f_{audio},g_{audio})$, $(f_{text},g_{text})$, $(f_{image},g_{image})$
are (encoder, linear projection layer) pairs producing $l2$-normalized $d$
dimensional embeddings: $E_{k}^{a}$, $E_{k}^{t}$, and $E_{k}^{i}$, for audio,
text, and overhead image respectively.

GeoCLAP is trained on embedding triplets using contrastive learning objective
similar to CLIP [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh,
Agarwal, Sastry, Askell, Mishkin, Clark, et al.] for all three pairs of
embeddings:

$L_{at}=\frac{1}{2N}\sum_{k=1}^{N}\bigg{(}log\frac{exp(E_{k}^{a}.E_{k}^{t}/\tau_{at})}{\sum_{j=1}^{N}exp(E_{k}^{a}.E_{j}^{t}/\tau_{at})}+log\frac{exp(E_{k}^{t}.E_{k}^{a}/\tau_{at})}{\sum_{j=1}^{N}exp(E_{k}^{t}.E_{j}^{a}/\tau_{at})}\bigg{)}$
(4)

$L_{ai}=\frac{1}{2N}\sum_{k=1}^{N}\bigg{(}log\frac{exp(E_{k}^{a}.E_{k}^{i}/\tau_{ai})}{\sum_{j=1}^{N}exp(E_{k}^{a}.E_{j}^{i}/\tau_{ai})}+log\frac{exp(E_{k}^{i}.E_{k}^{a}/\tau_{ai})}{\sum_{j=1}^{N}exp(E_{k}^{i}.E_{j}^{a}/\tau_{ai})}\bigg{)}$
(5)

$L_{ti}=\frac{1}{2N}\sum_{k=1}^{N}\bigg{(}log\frac{exp(E_{k}^{t}.E_{k}^{i}/\tau_{ti})}{\sum_{j=1}^{N}exp(E_{k}^{t}.E_{j}^{i}/\tau_{ti})}+log\frac{exp(E_{k}^{i}.E_{k}^{t}/\tau_{ti})}{\sum_{j=1}^{N}exp(E_{k}^{i}.E_{j}^{t}/\tau_{ti})}\bigg{)}$
(6)

where, $N$ is the training batch size and $\tau_{at}$, $\tau_{ai}$, and
$\tau_{ti}$ are learnable temperature parameters used to scale logits in loss
computation for each pairs of embeddings.

Combining equations 4, 5, and 6, the final loss for which GeoCLAP is trained
is as follows:

$L=L_{at}+L_{ai}+L_{ti}$ (7)

##  4 Experimental Details

###  4.1 Data Preprocessing

For audio preprocessing, we convert each audio sample into mel-spectrogram
using the default settings: {feature_size=64, sampling_rate=48000,
hop_length=480, max_length_s=10, fft_window_size=1024} provided in the
HuggingFace-wrapper:ClapProcessor for the pre-trained L-CLAP model clap-htsat-
fused.

In the SoundingEarth dataset, most of the audio recordings (except $6333$
samples) are also accompanied by a brief description and a title uploaded by
the contributor. In order to have textual description for all audio recordings
as well as to further encode geographic information in text, we use a python
client, geopy to obtain the address of the location and append an additional
sentence, _“The location of the sound is:{address}.”_ to the textual
description of each sample. For example, for the geolocation
$(52.509663,13.376481)$, the added sentence would be _“The location of the
sound is: Potsdamer Platz, Tiergarten, Mitte, Berlin, 10785, Germany”_.
Following L-CLAP, we use RobertaTokenizer with the parameter max_length set to
$77$.

For overhead imagery, we adopt the same data augmentation as SATMAE [Cong et
al.(2022)Cong, Khanna, Meng, Liu, Rozi, He, Burke, Lobell, and Ermon]. We
perform RandomResizedCrop with parameters:{input_size=224, scale=(0.2,1.0)},
followed by a RandomHorizontalFlip, during training. During inference, we
extract a $224\times 224$ center crop of the image.

###  4.2 Implementation and metrics

We implement our code in Pytorch and utilise HuggingFace for loading L-CLAP
encoders and their respective data pre-processing wrappers. We split the
dataset with ratio $70$:$10$:$20$ yielding a total of $35\,554$, $5\,079$, and
$10\,159$ samples into training, validation, and test split, respectively. For
experiments regarding the baseline, we ran the publicly available code for
[Heidler et al.(2023)Heidler, Mou, Hu, Jin, Li, Gan, Wen, and Zhu] using the
data splits of our study. We used the experimental setting for their best
reported results on cross-modal retrieval task, which is as
follows:{batch_size=256 encoders=ResNet18, latent_dim=128, loss=SymmetricCL,
tau=0.2}. The baseline was trained for $300$ epochs with Adam optimizer and
learning rate of $1e-3$.

####  4.2.1 Encoders

We use the pre-trained model clap-htsat-fused [Wu* et al.(2023)Wu*, Chen*,
Zhang*, Hui*, Berg-Kirkpatrick, and Dubnov] to encode audio and text. The
audio encoder used in our study, HTSAT, has $4$ swin-transformer blocks with
hidden feature dimension of $768$. The text encoder RoBERTa from [Wu* et
al.(2023)Wu*, Chen*, Zhang*, Hui*, Berg-Kirkpatrick, and Dubnov] used in our
study, has $12$ transformer blocks with hidden feature dimension of $768$. For
both audio and text encoders, we take the output of their respective L-CLAP’s
projection layer producing $512$-dimensional embeddings. For encoding overhead
image, we use the pre-trained vit_base_patch16 encoder of SATMAE [Cong et
al.(2022)Cong, Khanna, Meng, Liu, Rozi, He, Burke, Lobell, and Ermon]. It
processes input as a sequence of $16\times 16$ image patches passing through
$12$ layers of transformer blocks. In order to match dimension with audio and
text embeddings, we pass the output from SATMAE encoder to a ReLU activation
followed by a $512$-dimension linear layer. Starting from weights of these
pre-trained encoders, we conduct two set of experiments. First, we allow only
the overhead-image encoder to train while freezing L-CLAP. Second, we allow
fine-tuning of all encoders in our framework.

####  4.2.2 Training

We train GeoCLAP using the contrastive loss objective presented in Equation 7.
We initialize all three learnable temperature parameters to $0.07$. We also
run experiments with and without using text in our framework. While using
text, we further experiment the impact of adding an additional sentence
describing detailed address of the location to the text. For experiments where
we use overhead image and audio only, we train our model with image-audio
contrastive loss represented by Equation 5. Moreover, for experiments using
overhead image, audio, and text, while keeping the L-CLAP encoders frozen, we
train with $Loss=L_{ai}+L_{ti}$. We use a training batch size of $256$ for the
baseline, and our experiments with frozen L-CLAP, while using batch size of
$128$ for experiments allowing fine tuning of L-CLAP. We use the Adam
optimizer and set the initial learning rate to $5e-5$. We use weight_decay=0.2
and betas=(0.9,0.98). We use cosine annealing learning rate scheduler with
number of warm up iterations set to $2000$. We set max_epochs to $100$ for
experiments with frozen L-CLAP and $30$ for experiments allowing fine tuning
of L-CLAP.

####  4.2.3 Metrics

Following Heidler _et al_ [Heidler et al.(2023)Heidler, Mou, Hu, Jin, Li, Gan,
Wen, and Zhu], we use Recall@100 and Median Rank (Median-R) of the ground-
truth as the evaluation metrics of our approach. We use the test set
containing $10\,159$ samples as the gallery for both image-to-sound and sound-
to-image retrieval evaluation.

##  5 Evaluation

###  5.1 Experiments with SoundingEarth data

Table 1 shows the results of our experiments with the SoundingEarth dataset
while using the original overhead imagery of $0.2m$ resolution. One of the
interesting results from this table is that by just using frozen pre-trained
audio encoder from L-CLAP [Wu* et al.(2023)Wu*, Chen*, Zhang*, Hui*, Berg-
Kirkpatrick, and Dubnov], while allowing only overhead-image encoder to be
fine-tuned, we already get about $10$ points improvement in cross-modal
retrieval. This highlights the advantage of leveraging rich representation
space of pre-trained models like L-CLAP. However, when we introduce text
modality into training, while still keeping both text and audio encoders
frozen, the image-to-sound Recall@100 drops to $0.32$. L-CLAP was trained on
large corpus of text-audio pairs where textual description of audio have
relatively high quality. However, the primary focus of the SoundingEarth
dataset has been to collect geotagged audio from all around the world and
associate them with high-resolution overhead imagery. We observed that the
textual descriptions of audio in the SoundingEarth dataset are noisy and do
not reflect the type of textual prompts L-CLAP models were trained on. In our
experiments, we use three different types of texts: textual description of
audio, only address of the audio, and text containing both description and
address of the audio. We observed that for any type of text, learning with
frozen representation lowers the performance when compared to learning with
frozen representation of audio alone. With this observation, we decided to
allow fine-tuning of L-CLAP encoders. Accordingly, the performance of our
approach noticeably improves to image-to-sound Recall@100 of $0.384$ while
learning with overhead image and audio. The performance further improves to
Recall@100 of $0.423$ with Median Rank of $172$ when we learn with overhead
image, audio, and text. This performance is further improved to Recall@100 of
$0.434$ with Median Rank of $159$, when we add address of the audio location
in the text. This is an absolute improvement of the baseline performance by
$0.178$ points in image-to-sound Recall@100 and $655$ in Median Rank. We see
similar trends on sound-to-image retrieval task.

Method | Image2Sound | Sound2Image
---|---|---
Experiment | Image Encoder | Text-Audio Encoder | Text | Address | R@100 | Median-R | R@100 | Median-R
Baseline [Heidler et al.(2023)Heidler, Mou, Hu, Jin, Li, Gan, Wen, and Zhu] | ResNet18 | ResNet18 | ✗ | ✗ | 0.256 | 814 | 0.250 | 816
ours | SATMAE | L-CLAP-frozen | ✗ | ✗ | 0.352 | 360 | 0.348 | 369
ours | SATMAE | L-CLAP-frozen | ✓ | ✗ | 0.328 | 428 | 0.325 | 428
ours | SATMAE | L-CLAP-frozen | ✗ | ✓ | 0.298 | 546 | 0.295 | 544
ours | SATMAE | L-CLAP-frozen | ✓ | ✓ | 0.317 | 439 | 0.311 | 443
ours | SATMAE | L-CLAP | ✗ | ✗ | 0.384 | 230 | 0.385 | 237
ours | SATMAE | L-CLAP | ✓ | ✗ | 0.423 | 172 | 0.419 | 175
ours | SATMAE | L-CLAP | ✗ | ✓ | 0.432 | 166 | 0.431 | 167
ours | SATMAE | L-CLAP | ✓ | ✓ | 0.434 | 159 | 0.434 | 167
Table 1: Cross-modal retrieval performance for models using 0.2m GSD overhead
imagery.

###  5.2 Experiments with Sentinel data

Table 2 shows the results of our experiments with Sentinel-2 cloudless imagery
with $10m$ $GSD$. We found that performance in all of our experiments
noticeably improved while using lower-resolution overhead imagery. This choice
brought in $12.89\%$ relative improvement in the baseline Recall@100
performance as well. We believe the reason for this improvement is the larger
coverage of geographic area in a single overhead image with $10$m $GSD$.
Moreover, the lower-resolution sentinel imagery is inherently blurry offering
some regularization effect during training, leading to improved
generalizability of our models. Following similar trends as in Table 1, an
absolute Recall@100 improvement of about $10$ points is observed, when using a
pre-trained frozen audio encoder from L-CLAP. Similarly, the retrieval
performance improves to $0.396$ when the audio encoder is allowed to be fine-
tuned. We also observe gain in performance of fine-tuned GeoCLAP models
trained with text containing address. The best performance for GeoCLAP trained
with all three modalities, yields (Recall@100, Median Rank) of ($0.450$,$143$)
and ($0.447$,$144$) for image-to-sound and sound-to-image retrieval,
respectively. Compared to the baseline, this is a relative gain of $55.71\%$
and $57.95\%$ for Recall@100 on tasks: image-to-sound and sound-to-image
retrieval, respectively.

Method | Image2Sound | Sound2Image
---|---|---
Experiment | Image Encoder | Text-Audio Encoder | Text | Address | R@100 | Median-R | R@100 | Median-R
Baseline [Heidler et al.(2023)Heidler, Mou, Hu, Jin, Li, Gan, Wen, and Zhu] | ResNet18 | ResNet18 | ✗ | ✗ | 0.289 | 620 | 0.283 | 635
ours | SATMAE | L-CLAP-frozen | ✗ | ✗ | 0.384 | 274 | 0.381 | 271
ours | SATMAE | L-CLAP-frozen | ✓ | ✗ | 0.340 | 369 | 0.338 | 367
ours | SATMAE | L-CLAP-frozen | ✗ | ✓ | 0.311 | 453 | 0.304 | 461
ours | SATMAE | L-CLAP-frozen | ✓ | ✓ | 0.337 | 378 | 0.331 | 370
ours | SATMAE | L-CLAP | ✗ | ✗ | 0.396 | 199 | 0.396 | 205
ours | SATMAE | L-CLAP | ✓ | ✗ | 0.441 | 152 | 0.441 | 155
ours | SATMAE | L-CLAP | ✗ | ✓ | 0.441 | 153 | 0.440 | 156
ours | SATMAE | L-CLAP | ✓ | ✓ | 0.450 | 143 | 0.447 | 144
Table 2: Cross-modal retrieval performance for models using 10m GSD overhead
imagery.

###  5.3 Zero-Shot Soundscape Mapping

Utilising the rich representation space of our best-performing GeoCLAP model,
we demonstrate zero-shot soundscape mapping using both text and audio queries.
Soundscape maps, in our work, are the similarity-score heatmaps for a given
query. Specifically, we use the appropriate encoder from GeoCLAP to produce an
embedding of the query and embeddings for a dense set of overhead images in
the region of interest. Then, the cosine similarity score between the query
embedding and all overhead image embeddings is overlaid on the corresponding
region to yield a soundscape map (Figure 2). In Figure 3, we demonstrate a
country-scale soundscape map for the Netherlands. For this, we compute
soundscape for three prompts: {This is a sound of car horn; This is a sound of
chirping birds; This is a sound of animal farm} and overlay them together to
create a composite pseudo-color map. We compare this soundscape with ESRI’s
Sentinel-2 land cover classes. We observe a strikingly high correlation
between the related land-cover classes with the category of sound likely to be
heard at the location. More such soundscape maps can be found in the
supplemental material of this paper.

Figure 2: Soundscape maps along with reference overhead image for two regions.
Soundscape created for queries: (a) A textual prompt: This is a sound of sea
waves; (b) randomly selected sound from the class chirping_birds from ESC50
database [Piczak(2015)] (green: more probable, white: less probable). Figure
3: Comparison of (a) Soundscape map of the Netherlands with (b) Sentinel-2
land cover classes. The soundscape map was created by querying GeoCLAP with
textual prompts for three sound categories: car horn, chirping birds, and
animal farm.

##  6 Conclusion

We proposed GeoCLAP, a contrastive-learning framework capable of embedding the
modalities of overhead imagery, audio, and text into a common space. Our
approach significantly improves the state of the art for cross-modal retrieval
between overhead imagery and audio. We utilise the learned, multi-modal
representation space for soundscape mapping, demonstrating a simple and
scalable way to create soundscape maps for any geographic area using only
satellite imagery and audio or textual queries. With this approach, we can
construct global, high-resolution soundmaps with minimal effort.

## References

  * [Aiello et al.(2016)Aiello, Schifanella, Quercia, and Aletta] Luca Maria Aiello, Rossano Schifanella, Daniele Quercia, and Francesco Aletta.  Chatty maps: constructing sound maps of urban areas from social media data.  _Royal Society open science_ , 3(3):150690, 2016.
  * [Aporee()] Radio Aporee.  https://aporee.org/maps.
  * [Aramaki and Wakamiya(2023)] Eiji Aramaki and Shoko Wakamiya.  Image and sound of the city.  In _The Social City: Space as Collaborative Media to Enhance the Value of the City_ , pages 205–214. Springer, 2023.
  * [Ayush et al.(2021)Ayush, Uzkent, Meng, Tanmay, Burke, Lobell, and Ermon] Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon.  Geography-aware self-supervised learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [Chen et al.(2022)Chen, Du, Zhu, Ma, Berg-Kirkpatrick, and Dubnov] Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.  Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection.  In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2022.
  * [Chen et al.(2020)Chen, Lu, and Wang] Yaxiong Chen, Xiaoqiang Lu, and Shuai Wang.  Deep cross-modal image–voice retrieval in remote sensing.  _IEEE Transactions on Geoscience and Remote Sensing_ , 58(10):7049–7061, 2020.
  * [Cong et al.(2022)Cong, Khanna, Meng, Liu, Rozi, He, Burke, Lobell, and Ermon] Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon.  Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery.  _Advances in Neural Information Processing Systems_ , 35:197–211, 2022.
  * [Cui et al.(2022)Cui, Li, Xia, and Dai] Peng Cui, Tingting Li, Zhengwei Xia, and Chunyu Dai.  Research on the effects of soundscapes on human psychological health in an old community of a cold region.  _International Journal of Environmental Research and Public Health_ , 19(12):7212, 2022.
  * [Deshmukh et al.(2022)Deshmukh, Elizalde, and Wang] Soham Deshmukh, Benjamin Elizalde, and Huaming Wang.  Audio retrieval with wavtext5k and clap training.  _arXiv preprint arXiv:2209.14275_ , 2022.
  * [Devlin et al.(2018)Devlin, Chang, Lee, and Toutanova] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics_ , 2018.
  * [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _Proceedings of the International Conference on Learning Representations_ , 2021.
  * [Elizalde et al.(2023)Elizalde, Deshmukh, Al Ismail, and Wang] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang.  Clap learning audio concepts from natural language supervision.  In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2023.
  * [Engel et al.(2021)Engel, Fiebig, Pfaffenbach, and Fels] Margret Sibylle Engel, André Fiebig, Carmella Pfaffenbach, and Janina Fels.  A review of the use of psychoacoustic indicators on soundscape studies.  _Current Pollution Reports_ , pages 1–20, 2021.
  * [for Standardization(2014)] International Organization for Standardization.  Iso 12913-1: 2014 acoustics—soundscape—part 1: definition and conceptual framework.  _ISO, Geneva_ , 2014.
  * [Garzón et al.(2023)Garzón, Bravo-Moncayo, Arellana, and de Dios Ortúzar] Luis Garzón, Luis Bravo-Moncayo, Julián Arellana, and Juan de Dios Ortúzar.  On the relationships between auditory and visual factors in a residential environment context: A sem approach.  _Frontiers in Psychology_ , 14, 2023.
  * [González et al.(2023)González, Morillas, and Rey-Gozalo] David Montes González, Juan Miguel Barrigón Morillas, and Guillermo Rey-Gozalo.  Effects of noise on pedestrians in urban environments where road traffic is the main source of sound.  _Science of the total environment_ , 857:159406, 2023.
  * [Guzhov et al.(2022)Guzhov, Raue, Hees, and Dengel] Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel.  Audioclip: Extending clip to image, text and audio.  In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2022.
  * [He et al.(2022)He, Chen, Xie, Li, Dollár, and Girshick] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022.
  * [Heidler et al.(2023)Heidler, Mou, Hu, Jin, Li, Gan, Wen, and Zhu] Konrad Heidler, Lichao Mou, Di Hu, Pu Jin, Guangyao Li, Chuang Gan, Ji-Rong Wen, and Xiao Xiang Zhu.  Self-supervised audiovisual representation learning for remote sensing data.  _International Journal of Applied Earth Observation and Geoinformation_ , 116:103130, 2023.
  * [Hu et al.(2020)Hu, Li, Mou, Jin, Chen, Jing, Zhu, and Dou] Di Hu, Xuhong Li, Lichao Mou, Pu Jin, Dong Chen, Liping Jing, Xiaoxiang Zhu, and Dejing Dou.  Cross-task transfer for geotagged audiovisual aerial scene recognition.  In _Proceedings of the European Conference on Computer Vision_. Springer, 2020.
  * [Lercher and Dzhambov(2023)] Peter Lercher and Angel M Dzhambov.  Soundscape and health.  In _Soundscapes: Humans and Their Acoustic Environment_ , pages 243–276. Springer, 2023.
  * [Lionello et al.(2020)Lionello, Aletta, and Kang] Matteo Lionello, Francesco Aletta, and Jian Kang.  A systematic review of prediction models for the experience of urban soundscapes.  _Applied Acoustics_ , 170:107479, 2020.
  * [Liu et al.(2019)Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer, and Stoyanov] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * [Liu et al.(2021)Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and Guo] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [Mao et al.(2018)Mao, Yuan, and Xiaoqiang] Guo Mao, Yuan Yuan, and Lu Xiaoqiang.  Deep cross-modal retrieval for remote sensing image and audio.  In _10th IAPR workshop on pattern recognition in remote sensing_ , 2018.
  * [Margaritis and Kang(2017)] Efstathios Margaritis and Jian Kang.  Soundscape mapping in environmental noise management and urban planning: case studies in two uk cities.  _Noise mapping_ , 4(1):87–103, 2017.
  * [Ning et al.(2021)Ning, Zhao, and Yuan] Hailong Ning, Bin Zhao, and Yuan Yuan.  Semantics-consistent representation learning for remote sensing image–voice retrieval.  _IEEE Transactions on Geoscience and Remote Sensing_ , 60:1–14, 2021.
  * [Ooi et al.(2023)Ooi, Ong, Watcharasupat, Lam, Hong, and Gan] Kenneth Ooi, Zhen-Ting Ong, Karn N Watcharasupat, Bhan Lam, Joo Young Hong, and Woon-Seng Gan.  Araus: A large-scale dataset and baseline models of affective responses to augmented urban soundscapes.  _IEEE Transactions on Affective Computing_ , 2023.
  * [Owens et al.(2016)Owens, Wu, McDermott, Freeman, and Torralba] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba.  Ambient sound provides supervision for visual learning.  In _Proceedings of the European Conference on Computer Vision_ , 2016.
  * [Picaut et al.(2019)Picaut, Fortin, Bocher, Petit, Aumond, and Guillaume] Judicaël Picaut, Nicolas Fortin, Erwan Bocher, Gwendall Petit, Pierre Aumond, and Gwenaël Guillaume.  An open-science crowdsourcing approach for producing community noise maps using smartphones.  _Building and Environment_ , 148:20–33, 2019.
  * [Piczak(2015)] Karol J. Piczak.  ESC: Dataset for Environmental Sound Classification.  In _Proceedings of the Association for Computing Machinery Conference on Multimedia_ , 2015.
  * [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, et al.] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_. PMLR, 2021.
  * [Radicchi et al.(2021)Radicchi, Cevikayak Yelmi, Chung, Jordan, Stewart, Tsaligopoulos, McCunn, and Grant] Antonella Radicchi, Pınar Cevikayak Yelmi, Andy Chung, Pamela Jordan, Sharon Stewart, Aggelos Tsaligopoulos, Lindsay McCunn, and Marcus Grant.  Sound and the healthy city.  _Cities & Health_, 5(1-2):1–13, 2021.
  * [Reed et al.(2022)Reed, Gupta, Li, Brockman, Funk, Clipp, Candido, Uyttendaele, and Darrell] Colorado J Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell.  Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning.  _arXiv preprint arXiv:2212.14532_ , 2022.
  * [Salem et al.(2018)Salem, Zhai, Workman, and Jacobs] Tawfiq Salem, Menghua Zhai, Scott Workman, and Nathan Jacobs.  A multimodal approach to mapping soundscapes.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018.
  * [Wang et al.(2022)Wang, Albrecht, Braham, Mou, and Zhu] Yi Wang, Conrad Albrecht, Nassim Ait Ali Braham, Lichao Mou, and Xiaoxiang Zhu.  Self-supervised learning in remote sensing: A review.  _IEEE Geoscience and Remote Sensing Magazine_ , 2022.
  * [Wu et al.(2022)Wu, Seetharaman, Kumar, and Bello] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello.  Wav2clip: Learning robust audio representations from clip.  In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2022.
  * [Wu* et al.(2023)Wu*, Chen*, Zhang*, Hui*, Berg-Kirkpatrick, and Dubnov] Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Taylor Berg-Kirkpatrick, and Shlomo Dubnov.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2023.
  * [Yang et al.(2022)Yang, Wang, Sun, Zhang, Liao, Gu, Hou, and Jiao] Rui Yang, Shuang Wang, Yingzhi Sun, Huan Zhang, Yu Liao, Yu Gu, Biao Hou, and Licheng Jiao.  Multimodal fusion remote sensing image–audio retrieval.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 15:6220–6235, 2022.
  * [Yue et al.(2023)Yue, Meng, Yang, Wu, Liu, and Yan] Ran Yue, Qi Meng, Da Yang, Yue Wu, Fangfang Liu, and Wei Yan.  A visualized soundscape prediction model for design processes in urban parks.  In _Building Simulation_ , volume 16, pages 337–356. Springer, 2023.
  * [Zhao et al.(2023)Zhao, Liang, Tu, Huang, and Biljecki] Tianhong Zhao, Xiucheng Liang, Wei Tu, Zhengdong Huang, and Filip Biljecki.  Sensing urban soundscapes from street view imagery.  _Computers, Environment and Urban Systems_ , 99:101915, 2023.


# multimodal transformers for utterance-level code-switching detection

###### Abstract

An utterance that contains speech from multiple languages is known as a code-
switched sentence. In this work, we propose a novel technique to predict
whether given audio is monolingual or code-switched. We propose a multi-modal
learning approach by utilizing the phoneme information along with audio
features for code-switch detection. Our model consists of a Phoneme Network
(PN) that processes phoneme sequence and Audio Network (AN), which processes
the MFCC features. We fuse representation learned from both the Networks to
predict if the utterance is code-switched or not. The Audio Network and
Phonetic Network consist of initial convolutional, Bi-LSTM, and transformer
encoder layers. The transformer encoder layer helps in selecting important and
relevant features for better classification by using self-attention. We show
that utilizing the phoneme sequence of the utterance along with the MFCC
features improves the performance of code-switch detection significantly. We
train and evaluate our model on Microsoft’s code-switching challenge dataset
for Telugu, Tamil, and Gujarati languages. Our experiments show that the
multi-modal learning approach significantly improved accuracy over the uni-
modal approaches for Telugu-English, Gujarati-English, and Tamil-English
datasets. We also study the system performance using different neural layers
and show that the transformers help obtain better performance.

Index Terms—  multimodal learning, transformers, code-switching

##  1 Introduction

A continuous change between two or more languages within a single utterance is
known as code-switching [1]. Code-switching detection is about identifying
whether an utterance contains multiple languages. Low resource languages like
Indian languages most commonly have code-switches with English. Code-switched
utterances usually cause problems for automatic speech recognition systems,
which are built using monolingual data only. Many researchers in the speech
community are trying to develop systems that can handle code-switching speech.
Robust acoustic-modeling using Bi-lingual Deep Neural Networks are used in [2]
for Frisian code-switched automatic speech recognition task. [3] proposes to
use a shared acoustic model between the code-switched languages to obtain a
robust German speech recognition system. [4] proposes a one-pass recognition
strategy and shows we can eliminate the classical multi-pass technique, which
follows language identification, language boundary detection, and language-
independent speech recognition. Recently,[5] proposes using an end-to-end
neural network framework that uses the attention-CTC-based seq2seq model. They
propose to jointly predict the language label and the CTC labels for robust
code-switched speech recognition for Chinese. Language modeling techniques
have also been used for code-switched speech recognition tasks recently [7].
[6] proposed to use a recurrent neural network to predict the code-switches by
using textual features like Part of speech. Language identification seems to
be one of the main components of code-switch identification and code-switched
speech recognition. It is shown to have provided good improvement in the
system performance [8,9,10,11]. Some interesting works[1,2,3,4] have shown
different methods to detect the code-switching in a single
conversation/utterance. Multilingual training of DNN-based speech recognition
systems has shown some improvements in the low resource
languages[13,14,15,16]. Various other techniques like multilingual DNNs[12],
using latent space language models[13], using untranscribed data[14], and data
augmented acoustic and language models [15] are also proposed to code-switch
detection.

Recent developments in the area of deep learning have shown tremendous
improvement in many areas of speech processing, including speech recognition
[16,17,18,19,20], language identification [21,22,23,24], speaker
identification [25,26,27], emotion recognition [28,29,30], and many others.
Recently, the Attention mechanism [31] is shown to be a dominating approach in
both Natural language processing and speech due to their ability to focus on
specific parts of the input signal to extract relevant information for a
particular task. Sequence to sequence models become more powerful and
efficient when it is built using just attention layers. Work by [32] builds an
end-to-end architecture using many attention layers, thus eliminating the need
for a convolutional network or LSTM layers. These models are popularly known
as Transformer, and they are the state of the art models for most Natural
language processing problems. In the speech recognition field, state of the
art speech recognition uses attention based sequence to sequence model due to
its efficiency and capacity to align and predict. These models are also used
extensively in problems like emotion recognition [28], language
identification, and so on.

It is shown that the language identification problem can be tackled by using
phoneme sequences extracted from the speech recognition system with some post
processing[ 38]. [38],describes that we can use an English phoneme recognition
system to extract phoneme sequences for all the utterances with a different
language and apply some N-gram statistics post-processing to classify test
sentences. Motivated by the work of M. A. Zissman et al. [38], we propose to
use phoneme sequences as an additional guiding signal along with MFCC features
for the network architecture. We propose a multi-modal transformer model for
utterance-level code-switch detection. Our model consists of 2 streams, Audio
Network (AN) and Phoneme Network (PN). The Audio Network consists of a
sequence of CNN and LSTM layers followed by a Transformer encoder to process
MFCC features. The Phoneme Network also consists of a sequence of CNN and LSTM
layers followed by a Transformer Encoder layer to process phoneme embeddings.
We pool the output features from transformer layers from both Audio Network
and Phoneme Network using the statistics pooling layer, and we concatenate
these features to create utterance-level multi-modal feature representation.
We project this multi-modal feature representation to lower dimensional space
using a fully connected layer. Finally, we classify the lower-dimensional
representation using a softmax layer to predict the class label. We conduct
all our experiments on the Microsoft code-switching challenge dataset for
three Indic languages Telugu-English, Tamil-English, and Gujarati-English. Our
experimental study shows that our approach significantly improves the system
performance compared to the baseline model. We also show that our multi-modal
approach outperforms the uni-modal methods. Our open source implementation can
be found in https:/github.com/KrishnaDN/Code-Switch-Detection

The organization of the paper is as follows. In section 2, we explain our
proposed approach in detail. In section 3, we give a detailed analysis of the
dataset, and in section 4, we explain our experimental setup in detail.
Finally, in section 5, we describe our results.

##  2 Multi-Modal Learning

This paper proposes a multi-modal learning approach for utterance-level code-
switch detection using audio features and phoneme information. The model
consists of 2 stream architecture containing Audio Network and Phoneme
Network, as shown in figure 1. The Audio network takes MFCC features as input,
as shown in Fig 1. The Audio Network consists of Audio Encoder, Audio
Transformer, and statistics pooling blocks. The Audio Encoder converts the
MFCC features into high-level representations using Convolutional neural
networks followed by a Bi-LSTM layer, as shown in Fig. 1. This high-level
feature representation is fed into the Audio Transformer block, consisting of
2 multi-head self-attention layers. The multi-head self-attention layers use
self-attention to select important and relevant features for code-switching
detection using attention weighting. The statistics pooling layer in Audio
Network helps in aggregating frame-level features from Audio Transformer into
an utterance-level feature vector. Similarly, the Phoneme Network consists of
Phoneme Encoder, Phoneme Transformer, and statistic pooling layer, as shown in
Fig. 1. The Phoneme Encoder takes phoneme embeddings as input and produces
high-level contextual representation using convolutional neural networks and
Bi-LSTM. These features are processed by Phoneme Transformer to select
relevant features for better code-switch classification using multi-head self-
attention layers. The Phoneme network’s statistics-pooling layer acts as an
aggregation layer to pool variable length feature vectors from Phoneme
Transformer into a single feature vector for classification. We use Early
fusion to concatenate the utterance-level feature vector from both the Audio
Network and Phoneme Network to combine information from both audio features
and phoneme sequences for better classification performance. We explain all
the blocks in detail in the following section.

Fig. 1: Proposed model architecture.

###  2.1 Phoneme Recognition

Our proposed method consists of 2 streams known as Audio Network and Phoneme
network. The Phoneme Networks take a sequence of phonemes corresponding to the
audio data as its input. Due to the lack of ground truth phoneme labels in the
code-switch detection dataset, we generate phoneme labels by training a GMM-
HMM based speech recognition model for Telugu, Tamil, and Gujarati Languages
using Microsoft’s speech recognition challenge dataset. Using these speech
recognizers for the three languages, we create phoneme labels for Telugu-
English, Tamil-English, and Gujarati-English code-switching dataset. These
phoneme sequences and their corresponding MFCC features are used during the
training of code-switch identification models. The Audio Network takes MFCC
features for each audio file, while Phoneme Network takes the phoneme sequence
generated by the GMM-HMM speech recognition model for that input audio. All
the phoneme recognizers are trained in Kaldi.

###  2.2 Audio Encoder

The Audio Network consists of Audio Encoder, Audio Transformer, and statistics
pooling layer, as shown in the figure. The Audio Encoder takes a sequence of
MFCC frames as input and produces an utterance-level feature vector. We
extract 13-dimensional MFCC feature vectors for every 10ms using a 25ms
window. These low-level features are passed as input to Audio Encoder,
followed by Audio transformer to learn feature representation, which contains
information about code-switch in an utterance.

The Audio Encoder processes audio data to extract features that are useful for
code-switch identification. It consists of two 1D convolutional layers
followed by a Bi-LSTM layer. The Audio Encoder takes a sequence of
13-dimensional MFCC features and applies 1D convolution operations to extract
higher-level feature representations. Each convolutional block consists of a
1D convolution operation followed by Batch normalization and Relu activation,
as described in Fig.1. Each convolutional layer has its kernel size and
filters. We have [1x7,1x5] filter size for Conv-1 and Conv-2 layer,
respectively. Both Conv-1 and Conv-2 blocks are having stride [1x3].
Similarly, we have 64 filters for both Conv-1 and Conv-2. Since the
convolutional layer does not capture temporal information, we use Bi-LSTM
right after the convolutional block to capture temporal information.Let
$\boldsymbol{X=[x_{1},x_{2}..x_{n},...x_{T}]}$ be a sequence of
$\boldsymbol{T}$ MFCC feature vectors of dimension 13 and input vector
$\boldsymbol{x_{n}}$ is a MFCC frame extracted from the raw audio for every
25ms window with 10ms shift.

$\boldsymbol{F^{A}}=\textsf{Convolution}(\boldsymbol{X})$ (1)

Where Convolution is a sequence of two 1D convolutional blocks applied to the
MFCC sequence $\boldsymbol{X}$, as shown in Fig.1. After Convolution
operation, we obtain a feature sequence
$\boldsymbol{F^{A}=[f_{1},f_{2}.....f_{T}\prime]}$ of length
$\boldsymbol{T^{\prime}}$. The feature matrix, $\boldsymbol{F^{A}}$ has an
x-axis as the time dimension, and the y-axis is a feature dimension. The
feature dimension size will be the same as the number of filters in the last
convolutional layer. In our case, the feature dimension is 64, as the number
of filters in the last convolutional layer is 64. Since convolution cannot
capture temporal information, we use Bi-LSTM Layers to learn the temporal
representation of the feature matrix $\boldsymbol{F^{A}}$

$\boldsymbol{H^{A}}=\textsf{Bi-LSTM}(\boldsymbol{F^{A}})$ (2)

Where Bi-LSTM represents Bidirectional LSTM layer whose hidden dimension is
128. $\boldsymbol{H^{A}=[h_{1},h_{2}.....h_{T}\prime]}$ represents the output
sequence from the final Bi-LSTM layer. $\boldsymbol{h_{i}}$ represents the
$i^{\text{th}}$ timestep hidden activity from the last layer. Since the LSTM
is operating in both directions, the final hidden layer output feature will be
2 times the LSTM hidden feature vector in size.

###  2.3 Phoneme Encoder

The Phoneme Network consists of Phoneme Encoder, Phone Transformer, and
statistics pooling layer. The Phoneme Encoder consists of a learnable phoneme
embedding layer, two layers of convolution followed by a Bi-LSTM layer, as
shown in Figure 1. The Phoneme Encoder takes a phoneme sequence as inputs to
the embedding layers, and these fixed dimensional embeddings are learned
during training. The phoneme embeddings from the embedding layer are passed
into the convolutional network, consisting of 1D convolution layers. Each
convolutional layer has its kernel size and filters. we have [1x3,1x5] filter
size for Conv-1 and Conv-2 layer respectively. Similarly, we have 64 filters
for both Conv-1 and Conv-2. Since the convolutional layer does not capture
temporal information, we use Bi-LSTM right after the convolution block. The
Bi-LSTM learns contextual representations in the phoneme sequence and helps in
detecting code-switching.Let $\boldsymbol{P=[p_{1},p_{2}..p_{n},...p_{N}]}$ be
a sequence of $\boldsymbol{N}$ phoneme embeddings of dimension 128 and input
vector $\boldsymbol{p_{n}}$ is a phoneme embedding for $n^{\text{th}}$
phoneme.

$\boldsymbol{F^{P}}=\textsf{Convolution}(\boldsymbol{P})$ (3)

Where, Convolution is a sequence of two convolutional blocks applied to the
sequence of phoneme embeddings sequence $\boldsymbol{P}$ as shown in Figure 1.
$\boldsymbol{F^{P}=[f_{1},f_{2}.....f_{N}\prime]}$ is high-level feature
representation learned from convolution operation. Since convolution cannot
capture temporal information, we use a sequence of Bi-LSTM Layers to learn a
temporal representation of the feature matrix $\boldsymbol{F^{A}}$.

$\boldsymbol{H^{P}}=\textsf{Bi-LSTM}(\boldsymbol{F^{P}})$ (4)

Where Bi-LSTM represents 2 layers Bidirectional LSTM network whose hidden
dimension is 128. $\boldsymbol{H^{P}=[h_{1},h_{2}.....h_{N}\prime]}$
represents the output sequence from the final Bi-LSTM layer.
$\boldsymbol{h_{i}}$ represents the $i^{\text{th}}$ timestep hidden activity
from the last layer.

###  2.4 Transformer Network

Attention is a very well known concept in deep learning, specifically for
temporal sequences. These models are widely used in speech and NLP
applications due to their ability to attend and select relevant features that
are useful for solving a particular task. Recently, Multi-head self-attention
has become one of the widely used models in NLP and Speech due to its ability
to attend different parts of the input using multiple attention heads. Various
attention models have been invented over the past, including dot product
attention, locality-sensitive attention, additive attention, and so on. In
this work, we use a specific type of attention called multi-head self-
attention. A detailed multi-head self-attention block is shown in Fig. 2.

Fig. 2:  Multi-head self-attention: Audio Transformer (left) and Phoneme
Transformer (right)

The multi-head attention (MHA) layer consists of multihead scaled dot product
attention, matrix multiply, and position-wise feed-forward layer. The MHA
block consists of M linear layers for query Key and Value matrices, where M is
the number of heads in the multi-head attention. It takes feature
representations from Bi-LSTM layer and applies linear transform to create
$\boldsymbol{Q_{i}}$, $\boldsymbol{K_{i}}$ and $\boldsymbol{V_{i}}$ using
$\boldsymbol{i^{\text{th}}}$ linear transform where,
$\boldsymbol{i=[1,2.....M]}$ and M is the total number of attention heads. The
$\boldsymbol{Q_{i}}$, $\boldsymbol{K_{i}}$ and $\boldsymbol{V_{i}}$ are fed
into scaled dot product attention layer followed by matrix multiplication
between the value matrix and attention weights. The scaled dot product
attention $\boldsymbol{A_{i}}$ for $\boldsymbol{i^{\text{th}}}$ head is
defined as follows.

$\boldsymbol{A_{i}}=\textsf{Softmax}(\boldsymbol{\frac{Q_{i}K_{i}}{d_{q}}})\boldsymbol{V_{i}}$
(5)

Where $\boldsymbol{d_{q}}$ is the dimension of the query vector. We combine
the attention output from all the heads using simple concatenation and feed
them into the feed-forward layer.

$\boldsymbol{A}=\textsf{Concat}(\boldsymbol{A_{1},A_{2},A_{3}...A_{i}.....A_{M}}){W_{0}}$
(6)

Our proposed model consists of Audio Transformer and Phoneme Transformer. The
Audio Transformer takes output feature representation
$\boldsymbol{H^{A}=[h_{1},h_{2}.....h_{T}\prime]}$ from Audio Encoder as
input, while Phoneme Transformer takes output feature representation
$\boldsymbol{H^{P}=[h_{1},h_{2}.....h_{N}\prime]}$ from Phoneme Encoder as
input. The Query, Key, and Value are obtained using $\boldsymbol{H^{A}}$ for
Audio Transformer. Similarly, the Query, Key, and Value are obtained using
$\boldsymbol{H^{P}}$ for Phoneme Encoder. Both Audio Transformer and Phoneme
Transformer consist of 2 layers of Multi-head self-attention with four
attention heads each.

###  2.5 Statistics Pooling

The statistics pooling layers pools the frame-level features from both Audio
Transformer and Phoneme Transformer. It computes the mean and standard
deviation across time in a feature sequence to generate a single utterance-
level feature representation from Audio Network and Phoneme Network. The mean
and standard deviation vectors are concatenated to obtain the second-order
statistics of Audio Encoder and Phoneme Encoder.

$\boldsymbol{U^{A}}=\textsf{Concat}({\textsf{mean}(\boldsymbol{A^{a}})},{\textsf{std}(\boldsymbol{A^{a}})})$
(7)

$\boldsymbol{U^{P}}=\textsf{Concat}({\textsf{mean}(\boldsymbol{A^{p}})},{\textsf{std}(\boldsymbol{A^{p}})})$
(8)

Where, $\boldsymbol{A^{a}}$, $\boldsymbol{A^{p}}$ are the outputs from Audio
Transformer and Phoneme Transformer respectively. $\boldsymbol{U^{A}}$ is a
Utterance-level feature vector from $\boldsymbol{A^{a}}$ and
$\boldsymbol{U^{P}}$ is a Utterance-level feature vector from
$\boldsymbol{A^{p}}$ . Both $\boldsymbol{U^{A}}$ and $\boldsymbol{U^{P}}$ are
having same dimension.

###  2.6 Early Fusion

The utterance-level feature vectors $\boldsymbol{U^{A}}$ and
$\boldsymbol{U^{P}}$ are the representation learned from different modalities.
The early fusion block combines these two representations to combine
information from both audio and phoneme modality using simple concatenation.

$\boldsymbol{C}=\textsf{Concat}(\boldsymbol{U^{A}},\boldsymbol{U^{P}})$ (9)

Where $\boldsymbol{C}$ represents the fused feature representation.

##  3 Dataset

###  3.1 Speech Recognition dataset

The proposed model uses both audio features and phoneme sequences to train
multi-modal architecture for code-switching identification. We first build
LVSCR speech recognition models for Telugu, Tamil, and Gujarati languages to
extract the code-switching dataset’s phoneme sequences. We use the dataset
provided for Low resource speech recognition challenge by Microsoft. The
dataset consists of 40 hours of audio data for each of the languages. All the
audio files for each language are sampled at 16KHz to train speech recognition
models. The dataset contains both training evaluation split for each language.
We first train the GMM-HMM-based speech recognition model for each language
using the dataset split. We then evaluate each model with corresponding tests
split from their respective language.

###  3.2 Code-switching dataset

In this section, we discuss the code-switching dataset in detail. One of the
shared tasks’ main goal is to detect if an audio file is monolingual or code-
switched. The shared task contains two subtasks, 1) Utterance-level
identification of monolingual vs. code-switched utterances, and 2)Frame-level
identification of language in a code-switched utterance. We focus on subtask-1
in this paper. The dataset for subtask-1 contains 3 Indic languages, Telugu-
English, Tamil-English, and Gujarati-English. Each language is provided with
training and testing sets. Both training and test sets contain information
about whether an audio file is monolingual or code-switched. The statistics of
the dataset is given in Table 2. To use our approach, we need a phoneme
sequence for every utterance in the code-switching identification dataset. To
obtain the phoneme sequence, we use the prebuilt GMM-HMM speech recognition
models for their corresponding languages. We use Telugu, Tamil, and Gujarati
speech recognition models to extract phoneme sequences for Telugu-English,
Tamil-English, and Gujarati-English code-switched dataset, respectively.

Dataset Train Evaluation Duration Utterances Duration Utterances Gujarati
31.59 16780 3.59 2091 Telugu 31.59 16991 4.0 2135 Tamil 30.24 10933 7.312 2642

Table 1: Code-switching dataset: train and evaluation splits for different
languages

##  4 Experiments

###  4.1 Speech Recognition

Our speech recognition dataset consists of 40hrs of labeled audio data for 3
Indic languages Telugu, Tamil, and Gujarati. We extract 13-dimensional MFCC
features for every 25ms window with a 10ms shift. We use these features to
train the acoustic models. We build 3 different speech recognition models,
Telugu-ASR,Tamil-ASR, Gujarati-ASR for Telugu, Tamil and Gujarati language
respectively. The acoustic model for Telugu-ASR is trained using $\sim$40
hours of audio data containing $\sim$44K utterances. We train the Bi-Gram
language model using training utterances. Similarly, the acoustic models for
Tamil-ASR, Gujarati-ASR are trained using $\sim$40hours of data for each
language, and the dataset contains $\sim$39K utterance for Tamil and
$\sim$22K. The lexicon for Telugu-ASR, Tamil-ASR and Gujarati have $\sim$48K,
$\sim$58K and $\sim$43K words respectively. We evaluate speech recognition
models on $\sim$5 hours of evaluation data for each language.We use Kaldi
111www.kaldi.net [34] to train speech recognition models. We use
KenLM222www.kenLM.com [35] to train the language models.

###  4.2 Code-switching Identification

Our proposed approach consists of 2 streams, Audio Network and Phoneme
Network. The input to the audio network is a sequence of 13-dimensional MFCC
features. We use 25ms window length with 10ms hop length to extract the
features. The Audio Encoder has two convolutional neural layers containing
[1x7,1x5] kernels and 64 filters each. Each convolutional layer has a stride
of 3. Each of these convolutional layers consists of 1D convolution operation,
1D-batch normalization, and Relu activation function. The audio encoder’s
convolutional layers help reduce the temporal dimension of the input data and
extract higher-level features. The convolutional block’s output from the audio
encoder is fed to a Bi-LSTM with a hidden size of 128 and a dropout of 0.3.
Similarly, the phoneme encoder has two 1D convolutional layers with kernel
size [1x3,1x5] and 64 filters. Each convolution layer has stride 1. The input
to the text encoder is a sequence of phoneme embeddings of dimension 128 and
is learned as part of the training. The output of the Phoneme Encoder goes to
a Bi-LSTM with a hidden layer size of 128 and a dropout of 0.3. Both Audio
transformer and Phoneme Transformer contains two layers of multi-head
attention(MHA) blocks. Each MHA block uses four attention heads layer-
normalization before scaled dot product operation. The forward feed layer
inside MHA has a hidden dimension is 256. We use Adam optimizer [36] with a
scheduled learning rate. We use Pytorch [37] framework for implementation. All
our models are trained on RTX 2080Ti Graphics cards.

##  5 Results

In this section, we describe the experimental results in detail. We first
evaluate our speech recognition system performance for Telugu, Tamil, and
Gujarati using standard evaluation data. We then compare our multi-modal
approach with uni-modal approaches. We finally examine the effectiveness of
transformers for code-switching identification task. We train GMM-HMM-based
speech recognition on 40hrs of audio data for each language and evaluate the
word error rates on the corresponding test sets. We obtain, $\sim$24%,
$\sim$20% and $\sim$17% WER for Telugu, Tamil and Gujarati respectively.

We train our multi-modal model for Telugu-English, Tamil-English, and
Gujarati-English on the code-switching dataset. We evaluate the models on the
evaluation dataset for each language. The comparison between multi-modal and
uni-modal approaches are shown in Table 2. The experiment Multi-Modal
represents our approach, where we use both Audio network and Phoneme Network.
Uni-Modal-Audio represents the model where the classifier is trained using
only Audio Network with MFCC features as inputs. Similarly, Uni-Modal-Phoneme
is the model where the classifier is trained using only Phoneme Network with
Phoneme sequences as inputs. It can be shown that our approaches outperform
the uni-modal approaches. Our method outperforms the Uni-Modal-Audio approach
by 6.8%,7.75%, and 4.14% for Telugu, Tamil, and Gujarati, respectively.
Similarly, Our method outperforms the Uni-Modal-Phoneme approach by
5.46%,5.47%, and 13.85% for Telugu, Tamil, and Gujarati, respectively.

System Accuracy Telugu-English Multi-Modal 85.71% Uni-Modal-Audio 78.91% Uni-
Modal-Phoneme 80.25% Tamil-English Multi-Modal 86.02% Uni-Modal-Audio 78.27%
Uni-Modal-Phoneme 80.55% Gujarati-English Multi-Modal 88.85% Uni-Modal-Audio
84.71% Uni-Modal-Phoneme 75.80%

Table 2: Comparison of Multi-modal and Uni-Modal approaches. (Bold indicates
the best performance)

We experiment to see the effectiveness of the transformer for code-switching
identification task in multi-modal settings. We remove the Audio Transformer
and Phoneme Transformer from the original model to see the code-switching
identification task’s performance. We call this experiment CNN+Bi-LSTM.
Similarly, we remove Bi-LSTM from the original architecture to see the effect
of using Bi-LSTM in the system performance. The experiment is known as
CNN+Transformer. In the final experiment, we keep all the blocks from the
original architecture, and the experiment is called CNN+Bi-LSTM+Transformer.
We conduct these experiments for all the language pairs, and the results are
shown in Table 3. It can be seen that the CNN+Bi-LSTM+Transformer model is
shown to be the best model. It can be seen that the transformers help in
improving overall system performance.

System Accuracy Telugu-English CNN+Bi-LSTM 85.00% CNN+Transformer 85.30%
CNN+Bi-LSTM+Transformer 85.71% Tamil-English CNN+Bi-LSTM 85.70%
CNN+Transformer 86.02% CNN+Bi-LSTM+Transformer 85.60% Gujarati-English CNN+Bi-
LSTM 88.01% CNN+Transformer 87.90% CNN+Bi-LSTM+Transformer 88.85%

Table 3: Comparison of Different neural architectures. Bold indicates the best
performance

##  6 Conclusions

Code-switching detection is one of the most challenging and still unsolved
problems in the speech processing field. This work proposes a new approach for
improving code-switching detection performance using multi-modal learning
combined with transformer networks by utilizing the phonetic information. Our
method consists of two streams neural network where one stream processes audio
features like MFCC while the other stream processes phoneme sequences. Multi-
modal learning helps capture information shared between different modalities
to learn a better code-switching identification model. We also show that
transformer models help improve code-switching detection performance due to
their ability to attend and select features using a self-attention mechanism
for better code-switching detection performance. Our experiments show that our
multi-modal learning approach outperforms the uni-modal approaches on
Microsoft’s code-switching identification dataset.

##  7 Acknowledgements

We want to thank Microsoft India for providing the dataset for this research
work. Any opinion, findings, conclusions, or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views
of Freshworks Inc.

## References

  * [1] Peter Auer, “Code-switching in conversation: Language, interaction and identity”, Routledge, 2013.
  * [2] E. Yılmaz, H. Van den Heuvel, and D. A. Van Leeuwen, “Investigating bilingual deep neural networks for automatic speech recognition of code-switching Frisian speech,” in Proc. Workshop on Spoken Language Technology for Under-resourced Languages (SLTU), May 2016, pp. 159–166.
  * [3] G. Stemmer, E. Noth, and H. Niemann, “Acoustic modeling of foreign words in a German speech recognition system,” in Proc.EUROSPEECH, 2001, pp. 2745–2748
  * [4] D.-C. Lyu, R.-Y. Lyu, Y.-C. Chiang, and C.-N. Hsu, “Speech recognition on code-switching among the Chinese dialects”, in Proc. ICASSP, vol. 1, May 2006, pp. 1105–1108
  * [5] N. Luo, D. Jiang, S. Zhao, C. Gong, W. Zou, and X. Li, “Towards end-to-end code-switching speech recognition,” arXiv preprint arXiv:1810.13091, 2018.
  * [6] H. Adel, N. Vu, F. Kraus, T. Schlippe, H. Li, and T. Schultz, “Recurrent neural network language modeling for code switching conversational speech,” in Proc. ICASSP, 2013, pp. 8411–8415.
  * [7] Heike Adel, Katrin Kirchhoff, Dominic Telaar, Ngoc Thang Vu, Tim Schlippe, and Tanja Schultz, “Features for factored language models for code-switching speech,” in Spoken Language Technologies for Under-Resourced Languages, 2014.
  * [8] J. Weiner, N. T. Vu, D. Telaar, F. Metze, T. Schultz, D.-C. Lyu, E.-S. Chng, and H. Li, “Integration of language identification into a recognition system for spoken conversations containing codeswitches,” in Proc. SLTU, May 2012.
  * [9] K. R. Mabokela, M. J. Manamela, and M. Manaileng, “Modeling code-switching speech on under-resourced languages for language identification,” in Proc. SLTU, 2014, pp. 225–230.
  * [10] D.-C. Lyu, E.-S. Chng, and H. Li, “Language diarization for codeswitch conversational speech,” in Proc. ICASSP, May 2013, pp.7314–7318.
  * [11] K Bhuvanagiri and Sunil Kopparapu, “An approach to mixed language automatic speech recognition,” Oriental COCOSDA, Kathmandu, Nepal, 2010.
  * [12] E. Yılmaz, H. van den Heuvel and D. van Leeuwen, “Code-switching detection using multilingual DNNS,” 2016 IEEE Spoken Language Technology Workshop (SLT), San Diego, CA, 2016, pp. 610-616, doi: 10.1109/SLT.2016.7846326.
  * [13] C. Wu, H. Shen and C. Hsu, “Code-Switching Event Detection by Using a Latent Language Space Model and the Delta-Bayesian Information Criterion,” in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 11, pp. 1892-1903, Nov. 2015, doi: 10.1109/TASLP.2015.2456417.
  * [14] E. Yılmaz, H. Van den Heuvel, and D. A. Van Leeuwen, “Exploiting untranscribed broadcast data for improved code-switching detection,” in Proc. INTERSPEECH, Aug. 2017, pp. 42–46.
  * [15] Emre Yılmaz, Henk van den Heuvel, and David van Leeuwen, “Code-switching detection with data augmented acoustic and language models,” in the Sixth International Workshop on Spoken Language Technology for Under-resourced Languages (SLTU). Procedia Computer Science, 2018, pp. 127–131.
  * [16] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition” in ICASSP, 2016
  * [17] S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-task learning,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 4835–4839.
  * [18] C. Wang, Y. Wu, Y. Du, J. Li, S. Liu, L. Lu, S. Ren, G. Ye, S. Zhao, and M. Zhou, “Semantic mask for transformer based end-to-end speech recognition,” in Proc. Interspeech, 2020.
  * [19] S. Schneider, A. Baevski, R. Collobert, and M. Auli. “wav2vec: Unsupervised pre-training for speech recognition.” In Proc. of Interspeech, 2019.
  * [20] Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve, “Wav2letter: an end-to-end convnet based speech recognition system,” CoRR, vol. abs/1609.03193, 2016.
  * [21] D. Snyder, D. Garcia-Romero, A. McCree, G. Sell, D. Povey, and S. Khudanpur, “Spoken language recognition using x-vectors,” in Odyssey: The Speaker and Language Recognition Workshop, 2018
  * [22] J. Gonzalez-Dominguez, I. Lopez-Moreno, H. Sak, J. GonzalezRodriguez, and P. J. Moreno, “Automatic language identification using long short-term memory recurrent neural networks.” in INTERSPEECH, 2014, pp. 2155–2159.
  * [23] Ling, S., Salazar, J., Liu, Y., and Kirchhoff, K. “BERTphone: Phonetically-aware Encoder Representations for Utterance-level Speaker and Language Recognition”.
  * [24] Weicheng Cai, Danwei Cai, Shen Huang, and Ming Li, “Utterance-level end-to-end language identification using attention-based CNN-BLSTM,” in ICASSP, 2019, pp. 5991–5995.
  * [25] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur,“X-vectors: Robust dnn embeddings for speaker recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018
  * [26] Krishna D N, Ankita Patil, M.S.P Raj, Sai Prasad H S, Prabhu Aashish Garapati, “Identification of Indian Languages using Ghost-VLAD pooling”,arXiv:2002.01664
  * [27] Shuai Wang, Johan Rohdin, Lukas Burget, Oldrich Plchot, Yanmin Qian, Kai Yu, and Jan Cernock y, “On the usage of phonetic information for textindependent speaker embedding extraction,” INTERSPEECH, pp. 1148–1152, 2019
  * [28] M. Sarma, P. Ghahremani, D. Povey, N. K. Goel, K. K. Sarma,and N. Dehak, “Emotion identification from raw speech signals using DNNs,” in Proc. INTERSPEECH, Hyderabad, India,2018, pp. 3097–3101
  * [29] Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. “Learning Factorized Multimodal Representations”. In 7th International Conference on Learning Representations, ICLR 2019
  * [30] N, Krishna and Reddy, Sai “Multi-Modal Speech Emotion Recognition Using Speech Embeddings and Audio Features”, AVSP 2019 Melbourne ,Autralia
  * [31] D. Bahdanau, K. Cho, and Y. Bengio. “Neural machine translation by jointly learning to align and translate”, arXiv preprint arXiv:1409.0473, 2014
  * [32] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.
  * [33] D. Su, X. Wu and L. Xu, “GMM-HMM acoustic model training by a two level procedure with Gaussian components determined by automatic model selection,” 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, Dallas
  * [34] D. Povey et al., “The kaldi speech recognition toolkit,” in Workshop on Auto. Speech Recogn. and Understanding (ASRU), Hawaii, US, 2011, pp. 1–4.
  * [35] K. Heafield, I. Pouzyrevsky, J. H. Clark, and P. Koehn. “Scalable modified Kneser-Ney language model estimation”. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, 8 2013.
  * [36] Diederik P. Kingma and Jimmy Ba, “Adam: A Method for Stochastic Optimization”, In Proceedings of the International Conference on Learning Representations (ICLR), 2014
  * [37] Paszke, Adam and Gross, Sam and Chintala, Soumith and Chanan, Gregory and Yang, Edward and DeVito, Zachary and Lin, Zeming and Desmaison, Alban and Antiga, Luca and Lerer, Adam “Automatic differentiation in PyTorch” in NIPS, 2017
  * [38] M. A. Zissman and E. Singer, “Automatic language identification of telephone speech messages using phoneme recognition and N-gram modeling,” Proceedings of ICASSP ’94. IEEE International Conference on Acoustics, Speech and Signal Processing, Adelaide, SA, 1994, pp. I/305-I/308 vol.1, doi: 10.1109/ICASSP.1994.389377.