# SpeechVerse: A Large-scale Generalizable Audio Language Model

Nilaksh Das ∗ &Saket Dingliwal &Srikanth Ronanki &Rohit Paturi &Zhaocheng
Huang &Prashant Mathur &Jie Yuan &Dhanush Bekal &Xing Niu &Sai Muralidhar
Jayanthi &Xilai Li &Karel Mundnich &Monica Sunkara &Sundararajan Srinivasan
&Daniel Garcia-Romero &Kyu J. Han &Katrin Kirchhoff &
AWS AI Labs, Amazon Equal ContributionsCorresponding author<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) have shown incredible proficiency in performing
tasks that require semantic understanding of natural language instructions.
Recently, many works have further expanded this capability to perceive
multimodal audio and text inputs, but their capabilities are often limited to
specific fine-tuned tasks such as automatic speech recognition and
translation. We therefore develop SpeechVerse, a robust multi-task training
and curriculum learning framework that combines pre-trained speech and text
foundation models via a small set of learnable parameters, while keeping the
pre-trained models frozen during training. The models are instruction
finetuned using continuous latent representations extracted from the speech
foundation model to achieve optimal zero-shot performance on a diverse range
of speech processing tasks using natural language instructions. We perform
extensive benchmarking that includes comparing our model performance against
traditional baselines across several datasets and tasks. Furthermore, we
evaluate the model’s capability for generalized instruction following by
testing on out-of-domain datasets, novel prompts, and unseen tasks. Our
empirical experiments reveal that our multi-task SpeechVerse model is even
superior to conventional task-specific baselines on 9 out of the 11 tasks.

##  1 Introduction

Large language models (LLMs) [3, 1, 2] have achieved remarkable performance on
a variety of natural language tasks through self-supervised pre-training on
massive text corpora. They have also shown a striking ability to follow open-
ended instructions from users through further instruction tuning [5, 6, 4, 7],
enabling strong generalization capabilities. Despite the success, a
significant limitation lies in the inability of language models to perceive
non-textual modalities such as images and audio.

Speech in particular represents the most natural mode of human communication.
Empowering LLMs to deeply understand speech could significantly enhance human-
computer interaction [8] and multimodal dialog agents [9, 10]. As such,
enabling LLMs to comprehend speech has received substantial attention
recently. Some approaches first transcribe speech via an automated speech
recognition (ASR) system and then process the text with an LLM for improved
transcription [12, 13, 11]. However, such pipelines cannot capture non-textual
paralinguistic and prosodic features like speaker tone, intonation, emotion,
valence, etc.

Figure 1: Schematic diagram of the SpeechVerse framework.

A promising new paradigm directly fuses textual LLMs with speech encoders
within an end-to-end training framework [15, 14]. Enabling joint modeling of
speech and text holds promise for richer speech and audio comprehension versus
text-only methods. Particularly, instruction-following multimodal audio-
language models [18, 17, 16] are increasingly receiving more attention due to
their generalization capability. Despite some success, existing works in
multi-task audio-language models such as SpeechT5 [19], Whisper [20], VIOLA
[15], SpeechGPT [18], and SLM [17] are limited to processing a small number of
speech tasks.

We therefore propose SpeechVerse, a robust multi-task framework that leverages
supervised instruction finetuning to incorporate various speech tasks (see
Figure 1). In contrast to SpeechGPT [18], we propose using continuous
representations extracted from a self-supervised pre-trained speech foundation
model, focusing on tasks that generate text-only output. More recently, [16]
proposed Qwen-Audio, a multi-task audio-language model capable of perceiving
human speech and sound signals and is trained on 30 tasks from diverse audio
types including music and songs. However, this requires carefully designed
hierarchical tagging and a large-scale supervised audio encoder for fusion
making it sub-optimal for unseen speech tasks. In contrast, our training
paradigm incorporates multi-task learning and supervised instruction
finetuning in a unified curriculum, without the need for task-specific
tagging, allowing for generalization to unseen tasks using natural language
instructions.

We summarize our contributions below:

  1. 1.

Scalable multimodal instruction finetuning for diverse speech tasks.
SpeechVerse is a novel LLM-based audio-language framework to scalably exhibit
strong performance on as many as 11 diverse tasks. We extensively benchmark
our models on public datasets spanning ASR, spoken language understanding and
paralinguistic speech tasks.

  2. 2.

Versatile instruction following capability for novel open-ended tasks. We
demonstrate the SpeechVerse model’s capability to leverage the robust language
understanding of the LLM backbone in order to adapt to open-ended tasks that
were unseen during multimodal finetuning.

  3. 3.

Strategies for improving generalization to unseen tasks. We further study
prompting and decoding strategies including constrained and joint decoding,
that can enhance the model’s ability to generalize to completely unseen tasks,
improving absolute metrics by up to 21%.

##  2 Approach

###  2.1 Architecture

Figure 2: Block diagram of the SpeechVerse architecture.

As shown in Figure 2, our multimodal model architecture consists of three main
components: (1) a pre-trained audio encoder to encode an audio signal into a
feature sequence, (2) a 1-D convolution module that operates over the audio
feature sequence to abbreviate the sequence length, and (3) a pre-trained LLM
to use these audio features and textual instructions to perform the required
task. Details of each of these sub-systems are described below.

Audio encoder: To extract semantic features from a given audio, we use a large
pre-trained self-supervised speech foundation model as an audio encoder. We
can represent the audio encoder as a cascading collection of $L$ layers, where
each intermediate layer $l$ returns a feature sequence
$\mathbf{h}^{(l)}=f_{AE}(\mathbf{h}^{(l-1)};\theta^{(l)}_{AE})$, where
$\mathbf{h}^{(0)}=\mathbf{x}$ is the input audio. Here, $\theta^{(l)}_{AE}$
represents the learned weights for layer $l$ of the pre-trained speech model.
To capture a unified representation for multi-faceted forms of feature
semantics, we compute the output of the audio encoder as:

$\displaystyle
AE(\mathbf{x})=\frac{1}{L}\sum_{l=1}^{L}w^{(l)}\mathbf{h}^{(l)}\vspace{-0.2cm}$
(1)

where, the scalars $\\{w^{(1)},\dots w^{(L)}\\}$ are a set of learnable
parameters. As this approach concurrently encodes features from multiple
intermediate layers of the speech foundation model, it can simultaneously
capture different forms of semantics (higher-order as well as lower-order
features), leading to better generalization across a diverse set of tasks. We
also perform experiments by only taking the output from the last layer of the
audio encoder, i.e., $AE(\mathbf{x})=\mathbf{h}^{(L)}$.

Convolution downsampling module: LLMs trained on text-only input encode token
sequences which are typically of a considerably shorter length compared to the
feature sequences encoded by speech foundation models. To mitigate this large
discrepancy in length distribution between audio features and text tokens, we
downsample the encoded audio features through a learnable convolution module.
This module consists of successive blocks, each having a 1-D convolution layer
followed by layer normalization. For the 1-D convolution, we use a kernel size
of 3, which ensures that each output frame has both left and right context
from the input frames. In our experiments, we use as many number of these
downsampling blocks as necessary such that the resulting sampling rate for the
audio comes out to be 12.5 Hz, i.e., each output frame corresponds to 80ms of
audio. By fine-tuning the convolution downsampling module, the audio encoder
output can be transformed from an audio-only feature space to a joint audio-
text semantic space. Hence, we set the number of output channels for the 1-D
convolution to be equal to the feature dimension of the token embeddings for
the downstream LLM. We can represent the output of the convolution
downsampling module as $CNN(\mathbf{x})=f_{CNN}(AE(\mathbf{x});\theta_{CNN})$,
where $\theta_{CNN}$ represents learnable parameters of the 1-D convolution
blocks.

Large Language Model: An LLM typically takes in a sequence of input text
tokens $\mathbf{z}$ and models the probability of observing an output text
sequence $\mathbf{y}$ as the possible next tokens for the input text. The text
tokens are converted to vectorized embeddings $EMB(\mathbf{z})$ using a lookup
matrix that is learned during training. The output of the LLM can be
represented as $LLM(\mathbf{z})=f_{LLM}(EMB(\mathbf{z});\theta_{LLM})$, where
$\theta_{LLM}$ are the weights of the LLM. In this work, we leverage a pre-
trained LLM for multimodal tasks. For formulating the multimodal input with
audio $\mathbf{x}$ and text sequence $\mathbf{z}$, we simply concatenate the
downsampled audio features ($CNN(\mathbf{x})$) with the token embeddings
($(EMB(\mathbf{z})$) in the sequence dimension as shown in Figure 2. Hence, we
can represent the probability distribution over the output from our multi-
modal Spoken Language Model (SLM) as:

$\displaystyle
SLM(\mathbf{x},\mathbf{z})=f_{LLM}\Big{(}\big{[}CNN(\mathbf{x}),EMB(\mathbf{z})\big{]};\theta_{LLM}\Big{)}$
(2)

###  2.2 Multimodal Instruction Finetuning

Let
$\mathcal{D}^{\tau}=\\{\mathbf{x}^{\tau},\mathbf{y}^{\tau}\\}_{1}^{n^{\tau}}$
represent a labelled dataset for a task $\tau$ with $n^{\tau}$ samples where
each sample consists of audio sequence $\mathbf{x}^{\tau}$ and a corresponding
text label sequence $\mathbf{y}^{\tau}$. Let
$\mathcal{P}^{\tau}=\\{\mathbf{p}^{\tau}\\}_{1}^{m^{\tau}}$ be a set of
$m^{\tau}$ textual prompts/instructions sequences $\mathbf{p}^{\tau}$
describing the task. In our experiments, we do a weighted combination of
datasets from each of the $M$ training tasks
$\\{w^{\tau},\mathcal{D}^{\tau}\\}_{1}^{M}$ where $w^{\tau}$ is the weight
given to each sample of task $\tau$. This is done to ensure balance between
tasks with different complexities and training data sizes. Thus, each sample
can be represented as a tuple
$(\mathbf{x}^{\tau},\mathbf{p}^{\tau},\mathbf{y}^{\tau})$ where
$\mathbf{x}^{\tau}$ is the audio sample, $\mathbf{p}^{\tau}$ is a
prompt/instruction sampled uniformly from $\mathcal{P}^{\tau}$ and
$\mathbf{y}^{\tau}$ is the label. Then the probability for predicting the
labels $\mathbf{y}^{\tau}$ can be defined as:

$\displaystyle
p(\mathbf{y}^{\tau}|\mathbf{x}^{\tau},\mathbf{p}^{\tau};\Theta)=SLM(\mathbf{x}^{\tau},\mathbf{p}^{\tau})$
(3)

where $\Theta=\\{\theta_{AE},\theta_{CNN},\theta_{LLM}\\}$ are all the
parameters of our audio language model. The self-attention layers in the LLM
attend to both the audio and the textual instruction to generate the required
output on any audio-related task. We use the standard gradient descent method
to maximise the likelihood of producing the target label $\mathbf{y}^{\tau}$
for each sample in the training dataset as defined below:

$\displaystyle\mathcal{L}(\Theta)=-\log
p_{(\mathbf{x}^{\tau},\mathbf{p}^{\tau},\mathbf{y}^{\tau})\sim\\{\mathcal{D}^{\tau}\\}_{1}^{M}}(\mathbf{y}^{\tau}|\mathbf{x}^{\tau},\mathbf{p}^{\tau};\Theta)$
(4)

###  2.3 Curriculum Learning with Parameter Efficient Finetuning

To ensure faster convergence and avoid catastrophic forgetting and overfitting
by the pre-trained LLM, we adopt a parameter efficient approach based on low-
rank adaptation [21], or LoRA, for training our multimodal models. In this
work, we freeze the pre-trained audio encoder and LLM, and only train the
convolution downsampling module and the LoRA adapter. Since the bulk of the
parameters ($\theta_{AE}$ and $\theta_{LLM}$) are never updated throughout the
training process, it makes our framework compute efficient and allows us to
scale to a large number of diverse datasets and tasks with limited compute
resources. Moreover, it enables leveraging the existing capabilities of both
pre-trained audio and language models without catastrophic forgetting.
However, when training both the downsampling module and the LoRA adapter from
scratch on a diverse set of speech tasks, we observe frequent gradient
explosion, leading to suboptimal convergence. Hence, we carefully design a
curriculum of two stages for training.

In the first stage, we only train the convolution downsampling module and the
intermediate layer weights without introducing the LoRA adapters. Further,
only the samples from the automatic speech recognition (ASR) task are used in
this stage. Since the encoded speech feature vectors can be very different
from the token embeddings of the text input, this stage can help align them
more easily in a common embedding space by only learning the parameters for
the convolution downsampling module in the confined task space of ASR. This
enables a pre-trained text-based LLM to attend to the content of the audio
sequence and generate the speech transcription.

In the second stage, we now introduce the LoRA adapters for training the
model. In this stage, the intermediate layer weights, the downsampling module
as well as the LoRA adapters are unfrozen. Since the LoRA adapters are
training from scratch, we first allow the adapter weights to warmup by
training only on the ASR task, so as to get aligned to the common embedding
space learned by the convolution downsampling module in the first stage.
Finally, we introduce additional tasks on top of the ASR task and continue
training while keeping the pre-trained audio encoder and LLM weights frozen.
Since the warmup using only the ASR task enables the model to understand the
contents of the audio, our curriculum learning approach leads to faster
convergence on a variety of speech-based tasks that rely on the spoken
contents of the audio.

##  3 Experiments

###  3.1 Tasks

Table 1: Details of our tasks, training datasets and evaluation metrics. ST,
IC, SF, KWE and KWS tasks are referred as spoken language understanding (SLU)
tasks, while ER, ASC, SC, AC and SNS represents paralinguistic speech
processing (PSP) tasks

Task | Description | Metric | Dataset | Training
---|---|---|---|---
#Hours | #Samples
ASR |  Automatic Speech Recognition |  Word Error Rate (WER)$\downarrow$ |  Librispeech [22] | 960 | 281K
Mozilla Common Voice 5.1 [23] | 1.4K | 1.1M
VoxPopuli [24] | 385 | 160K
SLURP [25] | 84 | 119K
EuroParl [26] | 92 | 39K
|  |  |  MSP-Podcast 1.11 [27] | 135 | 83K
ST | Speech Translation | BLEU $\uparrow$ |  CoVost2 [28] | 426 | 576K
EuroParl [26] | 73 | 90K
IC | Intent Classification |  Accuracy (ACC)$\uparrow$ |  SLURP [25] | 35 | 47K
SF | Slot Filling |  SLU-F1 $\uparrow$ |  SLURP [25] | 25 | 30K
KWE | Keyword Extraction | Macro F1$\uparrow$ |  Librispeech [22] | 960 | 281K
Mozilla Common Voice 5.1 [23] | 426 | 288K
KWS | Keyword Search |  Accuracy (ACC) $\uparrow$ |  Librispeech [22] | 960 | 281K
Mozilla Common Voice 5.1 [23] | 426 | 288K
ER | Emotion Recognition |  Unweighted Average Recall (UAR) $\uparrow$ |  MSP-Podcast 1.11 [27] | 91 | 57K
ASC | Audio Sentiment Classification |  MSP-Podcast 1.11 [27] | 135 | 84K
SC | Speaker Counting |  Fisher [29, 30] | 690 | 755K
AC | Accent Classification |  Mozilla Common Voice 5.1 [23] | 190 | 123K
SNS | Speech/Non-Speech Detection | In-house VAD | 269 | 149K

In this work, we use a large collection of publicly available speech datasets
from a diverse set of tasks. A summary of the datasets and evaluation metrics
for these tasks is provided in the Table 1, while examples and prompts are
covered in the Table 10. Our training tasks include automatic speech
recognition (ASR), five spoken language understanding (SLU) tasks, and five
paralinguistic speech processing (PSP) tasks. The SLU tasks include those
tasks which can be solved by a cascaded system of a ASR model and an LLM,
while PSP tasks are classification tasks based on the audio, typically used in
audio analytics. For the IC/SL tasks, we split the SLURP dataset into seen and
unseen intent/slot label classes and study them separately to understand the
generalization capabilities of the model. The KWE task is about finding
important keywords from the audio, while in the KWS task, we learn to classify
whether a particular keyword was present in the audio or not. The target
labels were synthetically created for both these tasks using an LLM. All other
tasks are standard and an interested reader can refer to the Appendix A.3 for
more details. We create a list of at least 15 prompts per task describing the
goal of the task. To further add diversity to the set of tasks, we use a text-
to-speech (TTS) version of the Alpaca dataset [31]. This dataset contains a
diverse collection of prompt, input, output tuples, where the prompt describes
the task, input is the input for the task, and the output contains the target
labels. However, there are no corresponding audios associated with the
dataset. As in the existing work [17], we use a TTS system (AWS Polly in our
case) to generate synthetic audios for the input text using a pool of 10
different speakers.

###  3.2 Models

We train three different variants of multimodal models using our SpeechVerse
framework, namely, (1) Task-FT: These represent a set of models where each
model is trained individually for a particular task $\tau$. While most tasks
have separate model, datasets from certain related tasks like IC/SF, KWE/KWS,
and ER/ASC are trained together. (2) Multitask-WLM: This represents a single
multi-task model trained by pooling datasets for all the tasks together. Both
(1) and (2) uses a pretrained WavLM Large [32] as the backbone audio encoder
and only uses the last encoder layer output $AE(\mathbf{x})=\mathbf{h}^{(L)}$
as the representation of the audio. (3) Multitask-BRQ: This model is similar
to (2), but it uses the Best-RQ [33] architecture for the audio encoder.
Because the Best-RQ encoder is trained using a random projection quantizer,
the middle layer weights are more suited for downstream tasks if the encoder
is frozen during fine-tuning. We therefore used a unified representation that
combines representations from all layers via learnable weights for the
multitask model trained with the BEST-RQ encoder. The details about the
pretraining of this audio encoder are described in the Appendix A.1. All the
three variants of our models are trained using curriculum learning as
highlighted in the Section 2.3. All the models use Flan-T5-XL [5] as the
backbone LLM. The LoRA adapters introduced in Task-FT models have rank
$(r)=8$, while multitask models use $r=16$, allowing more learnable parameters
for solving a diverse collection of tasks. The complete list of hyper-
parameters and training setting is provided in the Appendix A.2.

Baselines: For the SLU tasks, we compare our models with a cascaded baseline
that uses an LLM on ASR hypotheses (ASR $\rightarrow$ LLM). For a fair
comparison, we use a parameter-efficient fine-tuned version of Flan-T5-XL as
the LLM for the baseline. The multi-task fine-tuning data is exactly the same
between our models and the baseline except that the latter uses ground truth
text in place of the audios. We benchmark the cascaded approach with ASR
hypotheses from (1) a strong publicly available Whisper-large-v2 [20] ASR
model and (2) our ASR Task-FT SpeechVerse model, enabling a true comparison
between a multimodal model vs cascaded approach. Finally, we also benchmark
the performance of the oracle ASR system by passing the ground truth
transcripts to the baseline LLM (GT $\rightarrow$ LLM). For the KWS task, we
use substring search of the keyword in ASR hypotheses as the baseline. For PSP
tasks, we train task-specific classifiers that use the last layer
representations from WavLM Large. The classifier contains a feed-forward
layer, followed by a 2-layer Gated Recurrent Unit (GRU) with mean pooling over
frames, followed by another 2 layers of feed-forward network and finally a
softmax operator. These models are trained on the same task-specific data
thereby allowing for direct comparison with our WavLM-based multimodal models.

Table 2: Results of ASR and spoken language understanding (SLU) tasks.
Datasets are defined as: LTC: Librispeech test-clean; LTO: Librispeech test-
other; Vox: Voxpopuli; MCV: Mozilla Common Voice; EN: English; DE: German; FR:
French;

Model | Training |  ASR (WER$\downarrow$) |  IC (ACC$\uparrow$) |  SF (SLU-F1$\uparrow$) |  ST (BLEU$\uparrow$) |  KWE (F1$\uparrow$) |  KWS (ACC$\uparrow$)
---|---|---|---|---|---|---|---
LTC | LTO | Vox | MCV | SLURP | SLURP | EN$\rightarrow$DE | EN$\rightarrow$FR | MCV | MCV
Whisper ASR $\rightarrow$ LLM |  LLM-FT | 2.5 | 4.9 | 7.0 | 8.2 | 86.6 | 72.3 | 23.5 | 30.6 | 45.1 | 87.7
SpeechVerse ASR $\rightarrow$ LLM | 2.1 | 4.4 | 6.5 | 10.5 | 88.8 | 80.5 | 24.4 | 30.9 | 54.3 | 87.3
SpeechVerse (E2E–SLM) | Task-FT | 2.1 | 4.4 | 6.5 | 10.5 | 90.4 | 82.4 | 27.8 | 35.7 | 46.9 | 98.7
Multitask-WLM | 2.5 | 4.7 | 6.8 | 12.0 | 90.3 | 82.2 | 25.9 | 33.7 | 46.6 | 98.5
Multitask-BRQ | 3.0 | 6.7 | 7.1 | 11.8 | 90.0 | 83.4 | 25.2 | 32.5 | 44.2 | 98.4
GT $\rightarrow$ LLM | LLM-FT | - | - | - | - | 94.2 | 84.3 | 27.2 | 34.8 | 70.3 | 100

Table 3: Results of paralinguistic speech processing (PSP) tasks. All reported
numbers are the value of the UAR metric.

Model | Training | ER | ASC | SC | AC | SNS
---|---|---|---|---|---|---
MSP | MSP | Fisher | MCV | VAD
WavLM | Task-FT | 60.3 | 57.5 | 98.5 | 57.9 | 96.3
SpeechVerse (E2E–SLM) | Task-FT | 64.6 | 61.3 | 99.6 | 60.1 | 96.9
Multitask-WLM | 62.0 | 60.1 | 98.9 | 60.0 | 97.6
Multitask-BRQ | 65.1 | 64.1 | 99.3 | 60.4 | 97.5

##  4 Results

###  4.1 Evaluation of SpeechVerse models

We evaluate end-to-end trained joint speech and language models (E2E-SLM)
leveraging the SpeechVerse framework on 11 unique tasks across multiple
domains and datasets. We first evaluate SpeechVerse’s core speech
understanding capability through ASR benchmarks. We then evaluate more complex
SLU tasks and paralinguistic speech tasks in Tables 2 and 3 respectively.

####  4.1.1 Performance on ASR and SLU tasks

First, we evaluate the performance of SpeechVerse models on four public ASR
benchmark datasets namely libri-test-clean, libri-test-other, Voxpopuli and
CommonVoice. The WER numbers for each of these datasets are reported in Table
2. SpeechVerse ASR in row 2 uses same model as task-specific pretrained ASR
model (Task-FT) in row 3. When comparing our task-specific pretrained ASR
model, which also serves as the initialization for multi-task finetuning, to
Whisper ASR, our model achieves slightly better performance on average.
However, the WER increases in both multitask models with Multitask-WLM
performing similarly to Whisper across three out of the four test sets. The
lower performance of the multi-task SpeechVerse model compared to the task-
specialized model is likely due to giving lower weight to ASR datasets when
constructing batches during multi-task training. This was done to balance the
performance across all tasks, since the data distribution is imbalanced
between the different tasks.

When it comes to SLU tasks, a frequent question posed is if an end-to-end
model can outperform a cascaded pipeline that transcribes speech via ASR and
then feeds it to a language model. To investigate this, we conducted
experiments on five semantic understanding tasks using the same foundation
models as SpeechVerse. The text foundation model was further fine-tuned on
data from the five SLU tasks separately, as we found the zero-shot performance
of Flan-T5 on these benchmark test sets to be quite poor. We also report
performance when feeding ground-truth transcripts into the fine-tuned LLM, to
provide upper bound results. On 4 of the 5 tasks, excluding Keyword
Extraction, the end-to-end trained models outperform the cascaded pipeline. In
particular, the more commonly used tasks like intent classification, slot
labeling, and speech translation are performing better than the cascaded
system, demonstrating the efficacy of our models trained using SpeechVerse. We
also observed that SpeechVerse models on KWS task are outperforming cascaded
pipeline by an absolute 10% in accuracy, while performing significantly behind
on KWE task. Since the keyword search task requires an attention span focused
on a specific word of interest, joint modeling helps improve accuracy by
overcoming error propagation present in a cascaded pipeline. We also conducted
an ablation study to determine if the KWE task benefits further from joint
decoding of ASR transcription and keywords. We noticed an improvement in
performance, closing the gap to the cascaded pipeline. The results from this
study are detailed further in the sub-section 4.3.2. When comparing the multi-
task models to the task-specific SpeechVerse models, there is a minor
degradation in performance, but the difference is not substantial. Overall,
the multitask model trained with either WavLM encoder or Best-RQ encoder
outperformed cascaded systems in majority tasks.

####  4.1.2 Performance on paralinguistic tasks

The results in Table 3 demonstrate clear improvements in performance on
various paralinguistic speech processing tasks when using multi-task learning
compared to fine-tuning the WavLM model independently for each task.
Specifically, the SpeechVerse model trained with multitask learning using
Best-RQ audio encoder (Multitask-BRQ) achieves gains over the baseline WavLM
model of 4.8% absolute on emotion recognition, 6.6% on audio sentiment
classification, and 2.5% on accent classification. More modest gains are seen
with the SpeechVerse model trained using multitask learning with WavLM encoder
(Multitask-WLM). The unified representation’s adaptive combination of all
encoder layers helps multitask BEST-RQ model improve diverse paralinguistic
task performance. Overall, multi-task learning provides noticeable
improvements in model generalization and effectiveness across a diverse set of
speech tasks compared to task-specific fine-tuning of the baseline WavLM
model. The results highlight the advantages of learning shared representations
across related tasks using multi-task learning techniques.

####  4.1.3 Comparison against SOTA models

Table 4: Comparison of SpeechVerse models to prior specialized SOTA models on
five diverse tasks: automatic speech recognition (ASR), speech translation
(ST), intent classification (IC), slot filling (SF), and emotion recognition
(ER).

Task | Dataset | Model | Performance
---|---|---|---
Metrics | Results
ASR |  Librispeech test-clean | test-other |  Whisper large-v2 [20] | WER $\downarrow$ |  2.5 | 4.9
SLM-FT [17] |  2.6 | 5.0
SpeechVerse Task-FT |  2.1 | 4.4
SpeechVerse Multitask-WLM |  |  2.5 | 4.7
VoxPopuli |  Whisper large-v2 [20] | WER $\downarrow$ | 7.0
mSLAM-CTC [34] | 7.0
SpeechVerse Task-FT | 6.5
| SpeechVerse Multitask-WLM |  | 6.8
CommonVoice 5.1 |  Whisper large-v2 [20] | WER $\downarrow$ | 8.2
SLM-FT [17] | 7.5
SpeechVerse Task-FT | 10.5
| SpeechVerse Multitask-WLM |  | 12.0
ST |  EuroParl EN$\rightarrow$DE | EN$\rightarrow$FR | EN$\rightarrow$RO |  SeamlessM4T 1.2B [35] | BLEU $\uparrow$ |  27.8 | 30.3 | 38.7
XMEF [36] | 22.5 | 30.0 | 32.3
SpeechVerse Task-FT |  27.8 | 35.7 | 32.2
SpeechVerse Multitask-WLM |  | 25.9 | 33.7 | 30.1
IC | SLURP |  E2E-SLU CTI [37] | ACC $\uparrow$ | 86.9
Frozen-hbt-large [38] | 74.4
PF-hbt-large [38] | 89.2
SpeechVerse Task-FT∗ | 84.6
SF | SLURP |  E2E-SLU CTI [37] | SLU-F1 $\uparrow$ | 74.7
Frozen-hbt-large [38] | 60.1
PF-hbt-large [38] | 78.9
SpeechVerse Task-FT∗ | 76.7
ER | MSP-Podcast 1.7 |  w2v2-L-robust[39] | UAR $\uparrow$ | 58
SpeechVerse Task-FT | 66.7
SpeechVerse Multitask-WLM | 61.2
∗SpeechVerse Task-FT model was re-trained for SLURP by including all intents
and slots for comparison with other models.

Table 4 benchmarks SpeechVerse models against state-of-the-art (SOTA) models
on five diverse tasks: automatic speech recognition (ASR), speech translation
(ST), intent classification (IC), slot filling (SF), and emotion recognition
(ER). Across these tasks, SpeechVerse demonstrates competitive or superior
performance compared to prior specialized models. When comparing our task-
specific pretrained ASR model, which also serves as the initialization for
multi-task finetuning, to Whisper ASR, our model achieves slightly better
performance on average. However, the multitask model (Multitask-WLM) performed
similarly to Whisper across three out of the four test sets. When evaluating
on speech translation across three language pairs, the task-specialized
SpeechVerse model surpassed SeamlessM4T on two pairs, while the multi-task
SpeechVerse model achieved competitive performance compared to prior work on
average. Both models did not perform well on English to Romanian pair. The
overall performance of the SpeechVerse models on speech translation is heavily
limited by the capabilities of the underlying language model FlanT5. The
speech translation capabilities cannot exceed the translation quality provided
by FlanT5 as the base language model. To evaluate SpeechVerse on spoken
language understanding tasks like intent classification (IC) and slot filling
(SF), we retrained the task-specialized SpeechVerse model by incorporating all
69 intents (both seen and unseen) as well as all slots. This allowed us to
compare SpeechVerse to prior work on the complete intent and slot sets. Our
SpeechVerse model achieved competitive performance to the previous SOTA (PF-
hbt-large) on slot filling, but was significantly behind on intent
classification with 5% lower absolute accuracy. However, SpeechVerse
outperformed the same SOTA model (Frozen-hbt-large) by 10% when the encoder
weights were frozen during fine-tuning. To further analyze the gap to prior
state-of-the-art, we conducted an experiment allowing the audio encoder
weights to be tunable during fine-tuning. This achieved 89.5% accuracy,
matching the prior SOTA. This suggests the intent classification performance
can overfit to the specific acoustic conditions of the SLURP dataset when full
fine-tuning is performed. The SpeechVerse model that was trained end-to-end
specifically for the task of emotion recognition achieved an 8% absolute
improvement in unweighted average recall over the previous state-of-the-art
model (w2v2-L-robust). In contrast, the multitask SpeechVerse model performed
3% better than the prior state-of-the-art. However, one key difference is that
the previous SOTA work trained on the MSP-Podcast 1.7 dataset, while we used
version 1.11 for training. The test set version remained the same between the
two approaches. Overall, the SpeechVerse model demonstrated competitive
performance compared to prior specialized models in some cases when evaluated
across the various tasks.

###  4.2 Generalization Across Instructions

We comprehensively study our Multitask-WLM model’s ability to generalize to
diverse forms of unseen instructions. As a first, we try to accomplish seen
tasks with differently worded instructions than those used for training. We
create novel prompts for some of the training tasks and evaluate the
robustness of the model to variations in the prompt. Next, we demonstrate the
model’s potential to leverage the robust language understanding of the
underlying LLM to generalize to completely new tasks that the model has not
seen at all during multimodal finetuning.

####  4.2.1 Measuring robustness to prompt variations

Table 5: Generalization to unseen prompts: The performance of each task is assessed on three different prompts, out of which, two are unseen during training. Task | Dataset |  Prompt (Unseen) | Performance
---|---|---|---
Metric | Results
ASR | Voxpopuli | P1 (✗) | WER$\downarrow$ | 6.8
P2 (✓) | 7.1
P3 (✓) | 6.8
ST |  EuroParl EN$\rightarrow$FR | P1 (✗) | BLEU$\uparrow$ | 33.7
P2 (✓) | 33.8
P3 (✓) | 33.5
AC | MCV | P1 (✗) | UAR$\uparrow$ | 60.0
P2 (✓) | 60.7
P3 (✓) | 60.6

To evaluate the effect of different prompts on the training task performance,
we tested our MultiTask-WLM model with additional prompts for 3 different
tasks: ASR, ST and AC. We test with three prompts per task, where one is
directly taken from the set of prompts used during training, while two
additional novel prompts are created by using a different wording and context.
As showcased in the Table 5, the model showecased similar performance across
prompts for each task. For the ASR task, we only see a minor variation of 0.3
in WER between seen and unseen prompts. Similarly for the ST and the AC task,
the metric values deviates very little with the change in prompt. These small
differences indicate that our model has generalized its core capabilities well
and is not strongly dependent on the specific wording or context of the
prompts. While prompt engineering remains important for optimal performance,
these results suggest that the model has learned robust task-specific skills
that transfer across the choice of prompts, at least for the tasks seen during
training. For the tasks beyond those seen during training, we study the prompt
design separately in a later subsection 4.3.1.

####  4.2.2 Open-ended Instruction Following

Table 6: Qualitative examples on open-ended tasks showcasing instruction-
following ability of the model. DC: Domain Classification; QA: Question
Answering; ST: Speech Translation

Task |  Instruction |  Prediction
---|---|---
|  [Transcription: are you able to update the personal details on my account]
Post Processing |  Transcribe the preceding audio |  are you able to update the personal details on my account
|  Transcribe the preceding audio with punctuations |  are you able to update the personal details on my account?
Response Generation |  Generate a response to the user query in the preceding audio |  Sure, I can update the personal details on your account. Can you please tell me which account you are referring to and the specific details you would like me to update?
ST unseen language |  Translate the preceding audio from english into spanish |  Puedes actualizar los datos personales de mi cuenta?
|  [Transcription: i need synthroid supply that will last me a month]
Contextual Biasing |  What is being said in the preceding audio? |  i need cynthroid supply that will last me a month
|  What is being said in the preceding audio? Hints: thyroxine, synthroid, thyroid, hormones |  i need synthroid supply that will last me a month
|  You are an AI assistant for a pharmaceutical company that specializes in thyroid treatment. Based on this context, what is being said in the preceding audio? |  i need synthroid supply that will last me a month
DC |  Classify the domain of the user audio input into finance, healthcare, travel, internet or food |  healthcare
QA |  What is the name of the medicine in the preceding audio? |  Synthroid
|  A 30-second audio generated using a text-to-speech system with contents of
a news article∗
Summar- ization |  Generate an abstractive summary for the content of the preceding audio in a single line |  Special counsel jack smith urges us supreme court to let former president donald trump’s election interference case proceed to trial without further delay.
|  Create a very crisp newspaper headline for the preceding audio |  Former President Donald Trump Faces Election Interference Charges in Court
|  [Transcription: Can you explain the process of composting?]
Factual QA |  Generate a short factual answer to the question in the preceding audio |  The process of composting is a gradual decomposition of organic matter into soil, with the goal being to decompose the organic matter into nutrients and water.
|  [Transcription: Compose a descriptive paragraph about a childhood memory.]
Creative QA |  Generate a short creative answer to the question in the preceding audio |  A child’s imagination is a powerful tool for creating memorable images, and one of my favorite memories is the creation of a dollhouse complete with all the pieces.
∗https://apnews.com/article/trump-special-counsel-election-interference-
january-6-c2dcc83e56a541804d4785f6bb6cd45c

To study the model’s ability to understand open-ended text-based as well as
speech-based instructions, we prompted the model with several unrestricted
creative requests that were not a part of our training curriculum. We
enumerate several such examples in the Table 4.2.2. In many of these examples,
the model is required to exhibit profound comprehension of both the spoken and
written directives to successfully execute the task. For example, in the
Creative QA task, the model has to understand the spoken request as well as
the instruction prompt in order to generate a related response. In the
Summarization task, the model has to correctly surmise the spoken content to
generate a summary. In the Contextual Biasing task, we observe that the model
is even able to correct its own output when provided with hints. The robust
responses of the multi-task model with such a distributional shift in the
input from the training data demonstrates the adaptability of its core
instruction following skills. Rather than overfitting to the training domain,
the multi-task learning approach enables the model to learn more versatile
capabilities in instruction comprehension and execution that better transfer
to new contexts. We provide some quantitative results on unseen tasks and
labels in the next section.

###  4.3 Strategies for Improving Performance

We further evaluate strategies to improve the multi-task model’s performance
specially for unseen tasks and class labels. First, we leverage contrained
decoding [40] for tasks that have a pre-defined set of finite outcomes. Next,
we also study joint decoding of the output of the task with the ASR hypotheses
of the audio for certain complex spoken language understanding tasks.

Table 7: Results on three unseen tasks with and without constrained decoding
(CD). We compare the settings wherein the instruction prompt only contained
the class labels v/s when descriptions were provided for each class label in
the prompt.

Class Label Description |  CD | IC |  | SL |  | DC
---|---|---|---|---|---|---
SLURP∗ | SNIPS |  |  SLURP∗ |  | Internal
(ACC) | (ACC) |  | (SD-F1) | (SLU-F1) |  | (ACC)
No | No | 51.7 | 54.9 |  | 73.12 | 46.45 |  | 56.9
Yes | 68.8 | 68.9 |  | 73.38 | 46.49 |  | 59.0
Yes | No | 47.7 | 61.0 |  | 69.79 | 47.34 |  | 44.0
Yes | 70.2 | 75.9 |  | 77.47 | 48.45 |  | 62.0
∗This is a subset of SLURP labels not seen during training

####  4.3.1 Constrained Decoding

The work in [40] introduced a model-agnostic technique to enforce domain-
specific knowledge and constraints during text generation. Building on this
prior approach, we have explored applying decoding constraints to the
SpeechVerse model to improve generalization to unseen speech classification
tasks. Rather than allowing the model to generate freely in response to a
prompt, the decoding is restricted to output from a predefined vocabulary of
class names. For example, in an intent classification task, the model would be
constrained to only generate intent labels such as “play_radio”,
“datetime_query” or “cooking_recipe”. By limiting the output space, the model
is more likely to produce the desired class label rather than unrelated text.

We meticulously benchmark the model’s performance on close-ended tasks, such
as a diverse set of classification tasks, that have a pre-defined set of
finite class labels. To understand the influence of the instruction prompts,
we divide this study into two parts: (1) where we only provide the class
labels in the prompt, and (2) where we provide an accompanying description of
each class label in the prompt. We ensure that none of these class labels were
seen during training, and hence these are all novel tasks for the model.
Further, we evaluate the efficacy of employing constrained decoding in each of
these two parts, as the class labels are known to us beforehand. Note here
that the SL task can be considered a harder task as the model has to correctly
classify a slot label as well as identify the corresponding slot value from
the speech. Hence, we report both, the SLU-F1 metric as well as the SD-F1
(Slot label Detection) metric for SL. The results of this study are presented
in Table 7.

We observe that including descriptions in the prompt has inconsistent results,
which can be attributed to the quality and subjectivity of the descriptions
provided in the prompt, especially as these descriptions were not seen during
training. However, we see that constrained decoding improves upon the results
in all cases, and most significant gains are observed only when descriptions
are provided with constrained decoding. This indicates that providing
descriptions indeed steers the model towards better comprehension of the task
semantics, but only constrained decoding is able to objectively prune the
noise introduced by any prompt bias. This phenomena is further revealed in the
SL task, where the SLU-F1 has a lower absolute value compared to SD-F1, as the
SLU-F1 metric incorporates both slot label and slot value, whereas constrained
decoding can only be applied to the slot label (hence the higher SD-F1).
Similarly, for a completely unseen task of Domain Classification (DC), where
the goal is to classify the content of the audio into five domains like
healthcare, technology etc, we observe a strong performance of 62% accuracy
with constrained decoding.

####  4.3.2 Joint Decoding

Certain SLU tasks require the model to understand the semantics of the audio
or perform a operation on the content of the audio. For example, KWE task is
about extracting important keywords from the ASR hypothesis of the audio.
Since this is a multi-step reasoning process for the model, we take
inspiration from the existing work [41] on Chain-of-Thought (CoT) prompting.
We train our model to first decode the ASR hypothesis of the audio, followed
by the output of the task. The prompts used for the joint elicitation of ASR
hypothesis and the task output are described in the Table 8. For a
representative set of SLU tasks including IC, KWE and ER, we re-train the
Task-FT models by adding a small portion of such multi-step examples along
with single-task examples. We compare the results with and without joint
decoding with ASR hypothesis in the Table 9.

Table 8: Example prompt for elicitation of compound goals for KWE and ER task Task |  Instruction |  Prediction
---|---|---
KWE |  Perform the following audio-based tasks in the order as described. |
=== Task: ASR === |  |
Perform speech recognition using the preceding audio. |  |
=== Task: KWE === |  |
Identify significant keywords in the provided audio. |  |
Make sure to format the output as "ASR: … | KWE: … |" |  ASR: paris is the capital of france | KWE: paris, france | |
ER |  Perform the following audio-based tasks in the order as described. |
=== Task: ASR === |  |
What is being said in the audio? |  |
=== Task: Emotion === |  |
Classify the tone of the speaker as happy, sad, angry or neutral |  |
Make sure to format the output as "ASR: … | Emotion: … |" |  ASR: can you shut up for a while | Emotion: angry | |
Table 9: Results on compound goal experiments when decoding performed with (w/) and without (w/o) ASR. Task | Dataset | Metric |  Joint Decoding
---|---|---|---
w/o | w/
IC | SLURP | ACC$\uparrow$ | 90.4 | 92.2
KWE | MCV | F1$\uparrow$ | 46.9 | 49.5
Vox | 53.1 | 56.7
ER | MSP |  UAR$\uparrow$ | 64.5 | 64.7

The results in the table showcase that augmenting the training data with
compound goals helps to improve the performance for all three tasks. The
improved performance can be attributed to the possibility of self-attention on
the already decoded ASR hypothesis in the decoder of our multimodal model.
Further, such multi-step training examples brings out the true multimodal
capabilities of the model to successfully complete the combination of tasks.
Further, such a paradigm can help save crucial inference latency by using a
single call of the large multi-modal model for obtaining both the transcript
and the task output. A more detailed analysis on understanding the benefits of
joint decoding will be conducted in future work.

Table 10: Qualitative examples of output from all the training tasks on a
audio file with transcription "turn on the radio now"

Task |  Instruction |  Prediction
---|---|---
ASR |  Transcribe the preceding audio |  turn on the radio now
|  What is being said in the preceding audio? |  turn on the radio now
ST |  Translate the english audio to german |  schalten sie jetzt den radio auf
|  Generate french translation for the english audio |  mettez le radio à l’onder maintenant
IC |  The previous audio needs to be mapped to exactly one of the following intents: |
\- datetime_query: user asks about date, time, schedules, etc; |  |
\- qa_definition: user wants to ask for general fact-based questions to receive definitions, explanations, descriptions, etc; |  |
\- calendar_remove: user wants to remove calendar events by voice; |  |
… |  |
\- play_radio: user wants to request playback of radio, stations, etc; |  play_radio |
SF |  Identify any of the following slots present in the speech: |
\- date: any reference to a date, month or day of the week; |  |
\- place_name: any reference of a place e.g., berlin, new york etc.; |  |
\- person: any reference of a person; |  |
\- time: any reference to time e.g., three am, two thirty etc.; |  time = now; |
KWE |  Identify important keywords in the preceding recording |  radio
KWS |  Is the word ’radio’ used in the audio? |  yes
ER |  Detect the primary emotion conveyed in this audio by the speaker’s tone - is it happiness, sadness, anger, or neutral? |  neutral
ASC |  Classify the sentiment of the speaker into one of the following - positivity, negativity, or neutrality |  neutral
AC |  Analyze the speaker voice and figure out their accent from one of American, Indian, Australian, Canadian or British. |  American
SC |  How many distinct speakers are there in this audio clip. |  one
SNS |  Is their speech in the preceding audio or not? |  speech

##  5 Related Work

Multi-task learning. Prior studies have shown that a single deep learning
model is capable of jointly learning multiple large-scale tasks across
different domains [42]. The key idea in multi-task learning is to leverage
shared representations across related tasks to improve overall generalization
and efficiency. Following this approach, the T5 model [43] frames all text
tasks as text-in to text-out, using a unified text-based framework that
facilitates shared representations across textual tasks. Similarly, SpeechNet
[44] and SpeechT5 [19] leverage a shared encoder-decoder framework to jointly
model speech and text modalities spanning 5 to 6 tasks like TTS, ASR, and
Voice Conversion (VC). VIOLA [15], a single auto-regressive Transformer
decoder-only network, unifies various cross-modal speech and text tasks as a
conditional codec language model via multi-task learning. Whisper [20] also
employs large-scale multi-task learning, training on related speech tasks
including language identification, speech recognition, and translation. In
this work, SpeechVerse utilizes multi-task training to transfer knowledge
between several related tasks while using natural language instructions to
perform each task. Unlike prior work that generated text, speech, or both, our
method focuses solely on producing textual output, while taking in audio and
text instructions.

Multimodal Large Language Models. Prior work on multimodal LLMs has focused
primarily on tasks involving images, such as image generation, visual question
answering, and image captioning [4, 45, 46, 9]. Multimodal models
incorporating modalities like audio and speech have received relatively less
attention compared to vision-and-language models [49, 47, 48]. However, there
has been growing interest in augmenting large language models with audio data,
leading to several proposed approaches [8, 18, 51, 14, 52, 50, 17]. SpeechGPT
[18] proposed a multimodal LLM combining discrete units of HuBERT with an LLM
to solve few understanding tasks like ASR, Spoken QA as well as generation
tasks like TTS. [17] introduces the novel capability of zero-shot instruction-
following for more diverse tasks such as dialog generation, speech
continuation and Question Answering. Most recently, [16] proposed Qwen-Audio,
a large-scale audio-language model trained using a multi-task learning
approach to handle a diverse range of tasks across various audio types
including human speech, natural sounds, music, and songs. Qwen-Audio employs a
single audio encoder to process various types of audio whose initialization is
based on the Whisper-large-v2 model [20] and performs full finetuning. In
contrast, our work utilizes two frozen pretrained models, one each for speech
encoder and text decoder to retain their intrinsic strengths. Also, we utilize
30+ instructions for each task during training for improved generalization
whereas [17] uses a single fixed instruction. Additionally, SpeechVerse
incorporates multi-task learning and instruction finetuning in a single
training stage.

##  6 Conclusion

In this work, we propose SpeechVerse, a multimodal framework that enables LLMs
to follow natural language instructions for performing diverse speech
processing tasks. Through supervised instruction finetuning and combining
representations from frozen pre-trained speech and text foundation models,
SpeechVerse achieves strong zero-shot generalization on unseen tasks.
Extensive benchmarking against conventional baselines show SpeechVerse’s
superiority on 9 out of 11 tasks, demonstrating its formidable instruction
following capability. Crucially, SpeechVerse maintains robust performance on
out-of-domain datasets, unseen prompts, and even unseen tasks. This highlights
the efficacy of our proposed training methodology in imbuing the model with a
generalizable skill for mapping text-based instructions to speech processing
outputs. Moving forward, we aim to expand SpeechVerse’s capabilities to follow
even more complex instructions and generalize to new domains. By separating
task specification from model design, SpeechVerse represents a versatile
framework that can dynamically adapt to new tasks through natural language
without retraining.

## Limitations

While this work demonstrated strong instruction following capabilities for the
multitask SpeechVerse model across a variety of tasks, some limitations
remain. The study relied on a single underlying LLM architecture (FlanT5)
rather than exploring more recent models tailored for instruction following.
Additionally, there is a trade-off between generalized capabilities on unseen
tasks versus specialized performance on original training tasks that poses
challenges for a single multitask model. While the model showed promise in
handling diverse unseen tasks, its limitations were not fully characterized
across the wide scope of possible instructions and the performance on these
unseen tasks is not quantitatively measured.

## Ethics Statement

All speech datasets we use have anonymous speakers. We do not have any access
to nor try to create any PII (Personal Identifiable Information) of speakers,
and our model neither identifies speakers nor uses speaker embeddings. Most of
the work used public open-source datasets for both training and testing. The
in-house datasets used for pre-training Best-RQ encoder and SNS task are
collected via third-party speech data vendors. No additional data collections
made concerning to the work carried in this paper.

## References

  * [1] Tom Brown et al. “Language models are few-shot learners” In _Advances in neural information processing systems_ 33, 2020, pp. 1877–1901
  * [2] Aakanksha Chowdhery et al. “Palm: Scaling language modeling with pathways” In _Journal of Machine Learning Research_ 24.240, 2023, pp. 1–113
  * [3] Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever “Improving language understanding by generative pre-training” OpenAI, 2018
  * [4] Josh Achiam et al. “Gpt-4 technical report” In _arXiv preprint arXiv:2303.08774_ , 2023
  * [5] Hyung Won Chung et al. “Scaling instruction-finetuned language models” In _arXiv preprint arXiv:2210.11416_ , 2022
  * [6] Long Ouyang et al. “Training language models to follow instructions with human feedback” In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 27730–27744
  * [7] Hugo Touvron et al. “Llama: Open and efficient foundation language models” In _arXiv preprint arXiv:2302.13971_ , 2023
  * [8] Rongjie Huang et al. “AudioGPT: Understanding and generating speech, music, sound, and talking head” In _arXiv preprint arXiv:2304.12995_ , 2023
  * [9] Team Gemini et al. “Gemini: a family of highly capable multimodal models” In _arXiv preprint arXiv:2312.11805_ , 2023
  * [10] Taicheng Guo et al. “Large Language Model based Multi-Agents: A Survey of Progress and Challenges” In _arXiv preprint arXiv:2402.01680_ , 2024
  * [11] W Ronny Huang et al. “Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study” In _arXiv preprint arXiv:2401.12789_ , 2024
  * [12] Yuang Li, Yu Wu, Jinyu Li and Shujie Liu “Prompting large language models for zero-shot domain adaptation in speech recognition” In _Proc. Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2023, pp. 1–8 IEEE
  * [13] Rao Ma et al. “Can generative large language models perform asr error correction?” In _arXiv preprint arXiv:2307.04172_ , 2023
  * [14] Paul K Rubenstein et al. “AudioPaLM: A Large Language Model That Can Speak and Listen” In _arXiv preprint arXiv:2306.12925_ , 2023
  * [15] Tianrui Wang et al. “VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation” In _arXiv preprint arXiv:2305.16107_ , 2023
  * [16] Yunfei Chu et al. “Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models” In _arXiv preprint arXiv:2311.07919_ , 2023
  * [17] Mingqiu Wang et al. “SLM: Bridge the thin gap between speech and text foundation models” In _Proc. Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2023, pp. 1–8 IEEE
  * [18] Dong Zhang et al. “Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities” In _arXiv preprint arXiv:2305.11000_ , 2023
  * [19] Junyi Ao et al. “SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing” In _arXiv preprint arXiv:2110.07205_ , 2021
  * [20] Alec Radford et al. “Robust speech recognition via large-scale weak supervision” In _Proc. ICML_ , 2023, pp. 28492–28518
  * [21] Edward J. Hu et al. “LoRA: Low-Rank Adaptation of Large Language Models”, 2021 arXiv:2106.09685 [cs.CL]
  * [22] Vassil Panayotov, Guoguo Chen, Daniel Povey and Sanjeev Khudanpur “Librispeech: an asr corpus based on public domain audio books” In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 2015, pp. 5206–5210 IEEE
  * [23] Rosana Ardila et al. “Common voice: A massively-multilingual speech corpus” In _arXiv preprint arXiv:1912.06670_ , 2019
  * [24] Ofer Dekel and Ohad Shamir “Vox Populi: Collecting High-Quality Labels from a Crowd.” In _COLT_ , 2009
  * [25] Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski and Verena Rieser “SLURP: A spoken language understanding resource package” In _arXiv preprint arXiv:2011.13205_ , 2020
  * [26] Philipp Koehn “Europarl: A parallel corpus for statistical machine translation” In _Proceedings of machine translation summit x: papers_ , 2005, pp. 79–86
  * [27] Reza Lotfian and Carlos Busso “Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings” In _IEEE Transactions on Affective Computing_ 10.4 IEEE, 2017, pp. 471–483
  * [28] Changhan Wang, Anne Wu and Juan Pino “Covost 2 and massively multilingual speech-to-text translation” In _arXiv preprint arXiv:2007.10310_ , 2020
  * [29] et al Cieri “Fisher English Training Speech Part 1 Speech LDC2004S13” In _Web Download. Philadelphia: Linguistic Data Consortium_ , 2004
  * [30] et al Cieri “Fisher English Training Part 2, Speech LDC2005S13” In _Web Download. Philadelphia: Linguistic Data Consortium_ , 2005
  * [31] Rohan Taori et al. “Stanford alpaca: An instruction-following llama model”, 2023
  * [32] Sanyuan Chen et al. “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing” In _IEEE Journal of Selected Topics in Signal Processing_ 16, 2021, pp. 1505–1518
  * [33] Chung-Cheng Chiu et al. “Self-supervised learning with random-projection quantizer for speech recognition” In _International Conference on Machine Learning_ , 2022, pp. 3915–3924 PMLR
  * [34] Ankur Bapna et al. “mslam: Massively multilingual joint pre-training for speech and text” In _arXiv preprint arXiv:2202.01374_ , 2022
  * [35] Seamless Communication et al. “SeamlessM4T: Massively Multilingual & Multimodal Machine Translation”, 2023 arXiv:2308.11596 [cs.CL]
  * [36] Xian Li et al. “Multilingual Speech Translation from Efficient Finetuning of Pretrained Models” In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ Online: Association for Computational Linguistics, 2021, pp. 827–838 DOI: 10.18653/v1/2021.acl-long.68
  * [37] Seunghyun Seo, Donghyun Kwak and Bowon Lee “Integration of pre-trained networks with continuous token interface for end-to-end spoken language understanding” In _Proc. ICASSP_ , 2022, pp. 7152–7156
  * [38] Yingzhi Wang, Abdelmoumene Boumadane and Abdelwahab Heba “A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding” In _arXiv preprint arXiv:2111.02735_ , 2022
  * [39] Anna Derington et al. “Testing Speech Emotion Recognition Machine Learning Models” In _arXiv preprint arXiv:2312.06270_ , 2023
  * [40] Brandon T Willard and Rémi Louf “Efficient Guided Generation for LLMs” In _arXiv preprint arXiv:2307.09702_ , 2023
  * [41] Jason Wei et al. “Chain-of-thought prompting elicits reasoning in large language models” In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 24824–24837
  * [42] Lukasz Kaiser et al. “One model to learn them all” In _arXiv preprint arXiv:1706.05137_ , 2017
  * [43] Colin Raffel et al. “Exploring the limits of transfer learning with a unified text-to-text transformer” In _The Journal of Machine Learning Research_ 21.1, 2020, pp. 5485–5551
  * [44] Yi-Chen Chen et al. “Speechnet: A universal modularized model for speech processing tasks” In _arXiv preprint arXiv:2105.03070_ , 2021
  * [45] Jean-Baptiste Alayrac et al. “Flamingo: a visual language model for few-shot learning” In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 23716–23736
  * [46] Junnan Li, Dongxu Li, Caiming Xiong and Steven Hoi “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation” In _International Conference on Machine Learning_ , 2022, pp. 12888–12900 PMLR
  * [47] Jing Yu Koh, Ruslan Salakhutdinov and Daniel Fried “Grounding language models to images for multimodal inputs and outputs” In _International Conference on Machine Learning_ , 2023, pp. 17283–17300 PMLR
  * [48] Zhiliang Peng et al. “Kosmos-2: Grounding Multimodal Large Language Models to the World” In _arXiv preprint arXiv:2306.14824_ , 2023
  * [49] Kaiyang Zhou, Jingkang Yang, Chen Change Loy and Ziwei Liu “Learning to prompt for vision-language models” In _International Journal of Computer Vision_ 130.9 Springer, 2022, pp. 2337–2348
  * [50] Soham Deshmukh, Benjamin Elizalde, Rita Singh and Huaming Wang “Pengi: An Audio Language Model for Audio Tasks” In _arXiv preprint arXiv:2305.11834_ , 2023
  * [51] Yuan Gong et al. “Listen, Think, and Understand” In _arXiv preprint arXiv:2305.10790_ , 2023
  * [52] Yu Shu et al. “Llasm: Large language and speech model” In _arXiv preprint arXiv:2308.15930_ , 2023
  * [53] J. Iranzo-Sánchez et al. “Europarl-ST: A Multilingual Corpus for Speech Translation of Parliamentary Debates” In _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020, pp. 8229–8233
  * [54] R. Lotfian and C. Busso “Building Naturalistic Emotionally Balanced Speech Corpus by Retrieving Emotional Speech From Existing Podcast Recordings” In _IEEE Transactions on Affective Computing_ 10.4, 2019, pp. 471–483 DOI: 10.1109/TAFFC.2017.2736999
  * [55] Rohit Paturi, Sundararajan Srinivasan and Xiang Li “Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correction” In _Proc. INTERSPEECH 2023_ , 2023, pp. 3567–3571 DOI: 10.21437/Interspeech.2023-1982
  * [56] et al Cieri “2000 HUB5 English Evaluation Speech LDC2002S09” In _Web Download. Philadelphia: Linguistic Data Consortium_ , 2002

##  Appendix A Appendix

###  A.1 Audio Encoder Pre-training

Our audio encoder is a 24-layer Conformer model with feature dimension of 768
and attention head of 8. The total number of parameters of this encoder model
is 300M. We adopt the BEST-RQ [33] method, which pre-trains the model to
predict the masked speech signals with labels generated from a random-
projection quantizer. The quantizer projects the speech inputs with a randomly
initialized matrix, and performs a nearest-neighbor lookup in a randomly-
initialized codebook. Neither the projection matrix nor the codebook is
updated during pre-training. We build an internal pre-training dataset
containing 300K hours English audios. The pre-training uses mask span of 10
with total effective masking ratio about 40%. The learning rate schedule
follows the transformer learning rate schedule with peak value of 0.0005 and
warm-up of 50K steps. AdamW optimizer is adopted with weight decay of 0.01.
Since the encoder has 4 times temporal-dimension reduction, the quantization
with random projections stacks every 4 frames for projections. We use 16
individual codeboooks, where the vocab size of each codebook is 8192 and the
dimension is 16. The model is pre-trained for 500K steps in total.

###  A.2 Hyper-parameters

We train all our models on a cluster of machines with 8 A100 GPUs, each having
40GB of memory. Pytorch Lightning framework111https://www.pytorchlightning.ai
is used for our implementation. A 5% subset of the complete training dataset
for a model is used as a validation set to choose the important hyper-
parameters. For the multi-task models, the sample weights for each dataset are
also chosen using this validation set. Further, all models are trained till
the validation loss converges and it does not improve for 5 consecutive
epochs. The details of the learning rate, warmup steps, batch size for the
Multitask-WLM and Multitask-BRQ are summarized in the Table 11. To adhere to
memory constraints of the GPUs, we filter out any training sample where the
sequence length of the audio is greater than 900 or the target label is
greater than 600. Since, the WavLM Large samples audio features at 50Hz rate,
we use two successive 1-D convolution blocks (kernel size=3, stride=2) for the
Task-FT and Multitask-WLM model to downsample the audio four times and achieve
the desired sampling rate of 12.5Hz. For BestRQ-based audio encoder, the
sampling rate is 25Hz and hence the stride of the second convolution block is
set to 1 to ensure the output sampling rate is 12.5Hz.

Table 11: Various hyper-parameters for our models

Parameter | Multitask-WLM | Multitask-BRQ
---|---|---
Audio encoder | WavLM-Large | Best-RQ
LLM | Flan-T5-XL | Flan-T5-XL
Convolution blocks | 2 | 2
Kernel sizes | [3, 3] | [3, 3]
Strides | [2, 2] | [2, 1]
Audio encoder ampling rate | 50Hz | 25Hz
Down-sampling factor | 4x | 2x
# parameters | 2.9B | 3.2B
# trainable parameters | 28.3M | 26.8M
LoRA rank(r) | 16 | 16
Learning rate | 0.001 | 0.005
Warm-up steps | 100 | 50000
Effective batch size | 768 | 2048

###  A.3 Tasks

We provide the details about our training tasks below as well as provide some
qualitative examples in the Table 10 to better understand the tasks.

ASR: We use a combination of 5 publicly available datasets for the ASR task,
which totals to 3k hours of paired audio and text data. We evaluate
performance on the standard benchmarks for ASR.

ST: We train our models to predict translations in multiple different
languages from the audios recorded with English speech. The tokenizer of the
backbone LLM limits the choice of what can be a potential target language. For
our case, we train and evaluate on German, French, and Romanian translations
from the EuroParl dataset [53]. We also augment the training data with German
and Catalan translations from the CoVost2 [28] dataset .

IC/SF: We train and evaluate our models on a subset of the SLURP dataset [25]
that consists of 10 intent classes and 4 slot labels. This also allows us to
study the generalization ability of our models to unseen class labels and we
separately study it in the Section 4.2. The intent classes and slot labels
that are chosen for the "seen" subset are the ones that occur most frequently
in the training data. The training prompt used for this task is designed to
contain the description of each class label.

KWE: The goal of this task is to identify important keywords in the content of
the speech in the audio. Since no publicly available dataset exists for this
task, we synthetically extract keywords from the ground truth transcripts
using a text-based keyword extraction
model222https://huggingface.co/Voicelab/vlt5-base-keywords. These are then
used as labels for training and evaluating our models.

KWS: This is a binary classification task to detect whether a specified
keyword was spoken in the audio or not. We create positive samples by randomly
selecting keywords from the ground truth transcripts and negative samples by
choosing a keyword that does not appear in the transcript. Positive and
negative examples are created in 70-30 ratio respectively for both training
and evaluation.

ER: For emotion recognition, we classify speech into one of four main emotion
classes: neutral, happy, sad, and angry, chosen based on the availability of
the training samples in the MSP-Podcast v1.11 dataset [54]. We report metrics
on the corresponding four-emotion subset of the Test1 split of the dataset.

ASC: For audio sentiment classification, we classify speech as positive,
negative, or neutral in sentiment. The sentiment labels were obtained by
thresholding the valence scale (annotated from 1 to 7) with 3 and 5. We train
on the entire training split of the MSP-Podcast v1.11 dataset, and evaluate on
the corresponding Test1 split.

SC: For speaker counting, we identify whether one or two speakers are present.
We train on segments from Fisher dataset transcripts [29, 30] with one or two
speakers, and evaluate on the Fisher test split used in [55].

AC: We train our models to classify speech into five accents of English
language: Canadian, Indian, Australian, British, and American, using metadata
from the Mozilla Common Voice dataset.

SNS: In this task, we identify whether speech is present in the audio. We
collect a diverse set of audios with and without speech for training our
models and evaluate them on a combination of speech segments from Hub5 [56]
dataset and held-out non-speech segments in our in-house collection.


# Sensing behavior of acetone vapors on TiO2 nanostructures — application of
density functional theory

V. Nagarajan, S. Sriram, R. Chandiramouli Corresponding author

(Received July 13, 2017, in final form September 5, 2017)

###### Abstract

The electronic properties of TiO2 nanostructure are explored using density
functional theory. The adsorption properties of acetone on TiO2 nanostructure
are studied in terms of adsorption energy, average energy gap variation and
Mulliken charge transfer. The density of states spectrum and the band
structure clearly reveals the adsorption of acetone on TiO2 nanostructures.
The variation in the energy gap and changes in the density of charge are
observed upon adsorption of acetone on n-type TiO2 base material. The results
of DOS spectrum reveal that the transfer of electrons takes place between
acetone vapor and TiO2 base material. The findings show that the adsorption
property of acetone is more favorable on TiO2 nanostructure. Suitable
adsorption sites of acetone on TiO2 nanostructure are identified at atomistic
level. From the results, it is confirmed that TiO2 nanostructure can be
efficiently utilized as a sensing element for the detection of acetone vapor
in a mixed environment.

Key words: TiO2, nanostructure, adsorption, acetone, energy gap

PACS: 71.15.Mb

###### Abstract

Äîñëäæóþòüñÿ åëåêòðîíí âëàñòèâîñò TiO2 íàíîñòðóêòóðè ç âèêîðèñòàííÿì òåîð¿
ôóíêöîíàëó ãóñòèíè. Âëàñòèâîñò àäñîðáö¿ àöåòîíó íà TiO2 íàíîñòðóêòóð âèâàþòüñÿ
â òåðìíàõ åíåðã¿ àäñîðáö¿, çìíè ñåðåäíüî¿ åíåðã¿ ùëèíè ïåðåíîñó çàðÿäó
Ìóëëêåíà. Ñïåêòð ãóñòèíè ñòàíâ òà ñòðóêòóðà çîíè òêî âêàçóþòü íà àäñîðáöþ
àöåòîíó íà TiO2 íàíîñòðóêòóðàõ. Çìíà åíåðã¿ ùëèíè òà çìíè ãóñòèíè çàðÿäó
ñïîñòåðãàþòüñÿ ïñëÿ àäñîðáö¿ àöåòîíó íà áàçîâîìó ìàòåðàë TiO2 n-òèïó.
Ðåçóëüòàòè ñïåêòðó ãóñòèíè ñòàíâ ïîêàçóþòü, ùî ïåðåíîñ åëåêòðîíâ âäáóâàòüñÿ ìæ
ïàðîþ àöåòîíó TiO2 áàçîâèì ìàòåðàëîì. Îòðèìàí äàí ïîêàçóþòü, ùî âëàñòèâîñò
àäñîðáö¿ àöåòîíó áëüø ñïðèÿòëèâèìè íà TiO2 íàíîñòðóêòóð. Çðóí ìñöÿ äëÿ
àäñîðáö¿ àöåòîíó íà TiO2 íàíîñòðóêòóð äåíòèôêóþòüñÿ íà àòîìàðíîìó ðâí. Çãäíî ç
îòðèìàíèìè ðåçóëüòàòàìè, ïäòâåðäæóòüñÿ, ùî TiO2 íàíîñòðóêòóðó ìîæíà åôåêòèâíî
âèêîðèñòîâóâàòè â ÿêîñò óòëèâîãî åëåìåíòà äëÿ âèÿâëåííÿ âèïàðâ àöåòîíó ó
çìøàíîìó äîâêëë.

Ключов слова: TiO2, íàíîñòðóêòóðà, àäñîðáöÿ, àöåòîí, åíåðãåòèíà ùëèíà

##  1 Introduction

The expansion of industries in recent years leads to emission of hazardous
gases and vapors into the atmosphere. Moreover, volatile organic compounds
(VOCs) are a major source of environmental pollutants and cause a serious
impact on humans. For instance, acetone (CH3COCH3) is a chemical reagent
utilized in laboratories and industries. Besides, this compound is widely used
in purifying paraffin, dissolving plastics and in pharmaceutics. Acetone may
cause damages to human noses, eyes and central nervous system when the
permissible exposure limit exceeds 1000 parts per million (ppm) according to
Occupational Safety and Health Administration [1]. The high exposure to
acetone to humans may cause mood swings, respiratory irritation and nausea. In
addition, breathing acetone in high ppm value may cause dizziness, respiratory
tract irritation and loss of strength [2]. Furthermore, acetone is also highly
inflammable. Meanwhile, acetone was found to be the final product for added
ketone bodies metabolism [3]. Among the transition metal oxide semiconductor,
titanium dioxide (TiO2) is extensively investigated as a key material for
technological application and fundamental research in the semiconductors,
solar cell [4] and lithium-ion batteries [5] owing to its excellent chemical
stability and low cost [6]. TiO2 is also a promising candidate in the field of
gas-sensor, photovoltaic, energy storage and photocatalysis due to its
photocatalytic properties, long-term stability and low toxicity [7, 8].
Besides, TiO2 polymorphs are mainly classified into three types, namely
anatase, rutile and brookite with corresponding space groups namely
I41/amd-$D_{4h}^{19}$ (tetragonal), P42/mnm-$D_{4h}^{14}$ (tetragonal) and
pbca-$D_{2h}^{15}$ (orthorhombic). Besides, only the first two crystal systems
play a vital role in industrial applications. Experimental work on brookite
crystal system is constrained owing to its difficulty in preparation [9]. The
optical energy gap values of anatase, rutile and brookite TiO2 crystal
structure are reported as 3.4 eV [10], 3.0 eV [11] and 3.3 eV [12],
respectively. The density functional theory (DFT) method has been widely used
by the researchers to study rutile and anatase phases of TiO2 nanostructure
[13]. The previous reports on TiO2 electronic properties were presented in the
literature using hybrid-functional schemes [14]. Bhowmik et al. [15] have
proposed TiO2 nanotubes as a good acetone sensor and the authors observed that
the response of the sensor reaches $3.35\%$ against 1000 ppm acetone. Chen et
al. [16] have reported the sensing performance of acetone based on nanoporous
morphology of TiO2 using a facile hydrothermal method. Rella et al. [17]
reported about the sensing properties of ethanol and acetone vapors on TiO2
nanoparticles prepared from pulsed laser deposition. Chen et al. [18] studied
the adsorption properties of acetone on pristine and transition metal doped
TiO2 clusters using DFT study. Thus, from the previous literatures, to our
knowledge, there are only limited reports based on DFT method to study the
adsorption properties of acetone on rutile TiO2 nanostructure. The motivation
behind the present work is to investigate acetone adsorption properties on
TiO2 nanostructure and to identify the most suitable adsorption site at the
atomistic level.

##  2 Computational methods

The electronic and adsorption properties of acetone on TiO2 nanostructures are
studied using DFT method utilizing SIESTA package [19]. The atomic position in
TiO2 nanostructures has been optimized to their ground state by decreasing
their Hellman-Feynman forces [20, 21] supported with conjugated gradient
algorithm. Moreover, all interatomic forces of rutile TiO2 nanostructures are
observed to be less than 0.05 eV/Å [22]. The generalized gradient
approximation (GGA) combined with Perdew-Burke-Ernzerhof (PBE) exchange
correlation functional is utilized to investigate the electron-electron
interaction [23, 24]. The energy cut-off of plane-wave basis set was adjusted
to 500 eV with energy convergence of $10^{-5}$ eV. The atomic positions in
TiO2 nanostructure were relaxed until the force of 0.02 eV/Å is achieved. The
$k$-points in Brillouin zones under the Monkhorst-pack scheme [25], are kept
as $10\times 10\times 10$ $k$-points with $3\times 3\times 3$ super-cell size
in the present study. The density of states, charge density, band structure
and electron localization function (ELF) of rutile TiO2 nanostructure were
calculated with the help of SIESTA code with Monkhorst-Pack $k$-point meshes
of $10^{-3}$ Å-1. The acetone adsorption properties on rutile TiO2
nanostructure is also studied using SIESTA package. Furthermore, the
electronic wave function of Ti and O atoms is expressed by basis set, which is
directly related with the numerical orbitals. The double zeta polarization
(DZP) basis set is utilized for relaxation of rutile TiO2 nanostructure in the
present work [26].

##  3 Results and discussion

###  3.1 Electronic properties of rutile TiO2 nanostructure

The present work concentrates on the investigation of electronic properties
and adsorption behavior of acetone on TiO2 nanostructure. Figure 1 represents
the schematic diagram of TiO2 nanostructure with the periodic boundary
condition (PBC). The optimized lattice constant and Wyckoff atomic positions,
including coordinates of TiO2 nanostructure are tabulated in table 1. In this
work, we have chosen a tetragonal — rutile TiO2 nanostructure (P42/mnm) for
calculating the electronic properties and adsorption behavior of acetone on
TiO2 nanostructure.

Figure 1: (Color online) Schematic diagram of an isolated TiO2 nanostructure with periodic boundary condition. Table 1:  The Wyckoff atomic positions of TiO2 nanostructure. Structure | Space group | Lattice | Wyckoff atomic positions
---|---|---|---
Tetragonal — rutile TiO2 | P42/mnm (136) |  $a=4.59$ Å | Ti: $2a$ (0, 0, 0)
|  |  $b=4.59$ Å | Ti: $16k$ (0.50, 0.50, 0.50)
|  |  $c=2.96$ Å | O: $8i$ (0.31, 0.31, 0)
|  |  | O: $8i$ (0.70, 0.70, 0)
|  |  | O: $16k$ (0.20, 0.81, 0.50)
|  |  | O: $16k$ (0.81, 0.20, 0.50)

Figure 2: (Color online) Band structure of an isolated TiO2 nanostructure.

The electronic properties of TiO2 base material are described in terms of band
structure [27]. Figure 2 illustrates the band structure of an isolated TiO2
nanostructure. The band gap of TiO2 base material is observed along the gamma
point (G) and it is observed that the band gap value of an isolated TiO2
nanostructure is found to be 2.43 eV with indirect gap. Moreover, the channels
along the conduction band minimum and the valence band maximum are out of
phase along the gamma point, which infers the indirect band gap. Furthermore,
the calculated band gap of TiO2 nanostructure is in good agreement with the
previously reported work [11]. In addition, based on the selection of
exchange-correlation (XC) functional, the electronic properties of TiO2
nanostructure can be fine-tuned. Previously, Deak et al. [14] reported the
electronic band gap of TiO2 with hybrid HSE06 functional for both anatase and
rutile crystal structure, which are overestimated from the present work. Table
2 shows that the energy band gap value of an isolated TiO2 nanostructure with
different exchange correlation functional such as PBEsol, BLYP, RPBE, PBE and
revPBE [28]. Kaloni and co-workers [29, 30, 31, 32, 33] have suggested that
the electronic and structural properties of 2D nanosheets and organic
compounds are in oligomer form, which can be fine-tuned with the interaction
of transition metal atoms.

Table 2:  Energy gap calculation for isolated TiO2 nanostructure with various exchange-correlation functional. XC functional |  $E_{\text{g}}$ (eV) | Type
---|---|---
PBE | 2.43 | GGA
BLYP | 2.50 | GGA
PBEsol | 2.38 | GGA
RPBE | 2.56 | GGA
revPBE | 2.48 | GGA

The band structure of TiO2 nanostructure is underestimated in the present work
since the density functional theory method with GGA/PBE exchange correlation
functional is utilized to calculate the electron-electron interaction in their
ground state. Moreover, the electronic and adsorption properties of acetone in
TiO2 nanostructure cannot be disturbed due to underestimation of the band gap,
since the isolated TiO2 base material is compared with acetone adsorbed TiO2
nanostructures. The density of states (DOS) spectrum gives the insights on
localization of charges in different energy intervals along TiO2
nanostructure. The visualization of band structure, PDOS and DOS spectrum are
shown in figures 2 and 3, respectively.

Figure 3: (Color online) Density of states (DOS) spectrum of an isolated TiO2
nanostructure.

The peak maxima are recorded near the Fermi energy level (EF), which is one of
the favorable conditions for the adsorption of target vapor/gas molecules.
Thereby, the free electrons can easily transfer between TiO2 base material and
acetone molecules. The peak maximum in different energy intervals arose owing
to the orbital overlapping between Ti and O atoms in TiO2 nanostructure.
Generally, for chemiresistive type of vapor/gas sensors, metal oxide
semiconducting materials are preferred owing to the fact that the transfer of
electrons is facilitated between target gas/vapor and TiO2 nanostructure and
the changes in the resistance can also be observed. From the observations of
band structure and DOS spectrum of TiO2 nanostructure it is inferred that TiO2
material can be used as a base material for the possible application of
chemical nanosensors.

###  3.2 Adsorption properties of acetone on TiO2 nanostructure

In the initial stage of acetone adsorption study on TiO2 base material,
acetone vapor should be studied in vapor phase. The bond length between Ti and
O atom in TiO2 nanostructure is 1.98 Å. Figure 4 (a) refers the adsorption of
carbon atom in acetone molecules adsorbed on Ti atom in TiO2 nanostructure and
it is referred to as position P. Figure 4 (b) illustrates the adsorption of H
atom in acetone molecules adsorbed on Ti atom in TiO2 nanostructure and it is
referred to as position Q. Similarly, positions R and S refer to the
adsorption of C and H atom in acetone molecules adsorbed on O atom in TiO2
nanostructure as shown in figure 4 (c) and (d), respectively. The adsorption
energy of acetone on TiO2 nanostructure can be calculated using equation (3.1)

$E_{\text{ad}}=[E(\text{TiO}_{2}/\text{CH}_{3}\text{COCH}_{3})-E(\text{TiO}_{2})-E(\text{CH}_{3}\text{COCH}_{3})+E(\text{BSSE})],$
(3.1)

where $E(\text{TiO}_{2}/\text{CH}_{3}\text{COCH}_{3})$ refers to the energy of
TiO2/CH3COCH3 complex. $E$(CH3COCH3) and $E$(TiO2) refers to the isolated
energy of CH3COCH3 and TiO2 molecules, respectively. BSSE represents the basis
set superposition error using counterpoise techniques in order to eliminate
the overlapping effect on basis functions during calculations.

Figure 4: (Color online) (a)–(d) Adsorption of acetone on position P, Q, R and
S.

When acetone molecules get adsorbed on TiO2 base material, negative values of
adsorption energy ($E_{\text{ad}}$) indicate a strong adsorption of acetone on
TiO2 nanostructure. Besides, in the present work, for all positions namely P–S
exhibits a negative value of $E_{\text{ad}}$. This clearly confirms the
adsorption of acetone molecules on TiO2 nanostructure. The adsorption energy
of TiO2 nanostructure for position P–S is observed to be $-1.48$ eV, $-1.46$
eV, $-1.46$ eV and $-1.29$ eV, respectively. In addition, the band gap of TiO2
nanostructure gets decreased due to the adsorption of acetone molecules on
TiO2 base material owing to the interaction of target VOCs with base material.
Thus, the conductivity of TiO2 nanostructure increases. The resistance of TiO2
base material decreases when it is exposed to the reducing vapors such as
acetone. The trend in the changes of resistance upon exposure towards acetone
vapor is in good agreement with the reported work of Sun et al. [34]. The
changes in band gap of TiO2 nanostructure for positions P–S are found to be
2.32 eV, 1.32 eV, 0.82 eV and 0.68 eV, respectively. Besides, the variation in
the energy gap and adsorption energy supports that TiO2 nanostructure can be
used for the detection of acetone vapor. Rella et al. [17] reported about the
detection of ethanol and acetone vapors using TiO2 nanoparticles synthesized
by pulsed laser deposition method. Chen et al. [16] studied the acetone
sensing performance of nanoporous titanium dioxide using a facile hydrothermal
method. Epifani et al. [35] reported about the design of an acetone sensor
based on titanium dioxide nanocrystals functionalized with tungsten oxide
species. From the literature, it is evident that TiO2 nanomaterial can be
utilized as a two probe device for the detection of VOCs upon adsorption,
which gives rise to change in the current. The variation in the current is
directly proportional to the concentration of acetone molecules present in the
atmosphere. From the previous reports, it is inferred that TiO2 nanostructure
can be efficiently used as acetone vapor sensor, which also strengthens the
present work. Besides, the present investigation strongly confirms the
adsorption of acetone on TiO2 nanostructure at atomistic level. In addition,
the most suitable adsorption site of acetone on TiO2 base material can be
concluded only after studying the percentage of average energy gap variation
($E_{\text{g}}^{\text{a}}$, %) compared with its isolated counterpart. Table 3
refers the Mulliken charge transfer, percentage of average energy gap
variation and adsorption energy. From the results, it is clearly observed that
the prominent adsorption sites for acetone molecule on TiO2 nanostructure are
positions Q, R and S. The average energy gap variation is observed to be
comparatively higher than P site. The transfer of electron between acetone
molecule and TiO2 nanostructure can be analyzed in terms of Mulliken
population analysis (Q) [36, 37, 38].

Table 3: Adsorption energy, Mulliken charge and average energy gap variation of TiO2 nanostructure. Nanostructures |  $E_{\text{ad}}$ (eV) | Q (e) |  $E_{\text{g}}$ (eV) |  $E_{\text{g}}^{\text{a}}$ (%)
---|---|---|---|---
Acetone adsorbed on TiO2 nanostructure
Isolated TiO2 | $-$ | $-$ | 2.43 | $-$
P |  $-$1.48 | 0.012 | 2.32 | 4.74
Q |  $-$1.46 | 0.105 | 1.32 | 84.09
R |  $-$1.46 | 0.171 | 0.82 | 196.34
S |  $-$1.29 | 0.013 | 0.68 | 257.35
H2O and O2 adsorbed on TiO2 nanostructure
A |  $-$1.62 | 0.224 | 2.39 | 1.67
B |  $-$1.60 | 0.17 | 2.41 | 0.83
C |  $-$1.78 | 0.029 | 2.18 | 11.47
D |  $-$1.71 |  $-$0.304 | 2.36 | 2.97

The positive value of Q show that the electrons are transferred from acetone
vapor molecule to TiO2 base material; whereas the negative value of Mulliken
charge shows that the electrons are transferred from TiO2 base material to
acetone molecule [39, 40, 41]. The Mulliken charge transfer values of TiO2
nanostructure for positions P–S are found to be 0.012 e, 0.105 e, 0.171 e and
0.013 e, respectively. From the observation, the positive value of Mulliken
charge is recorded for all the positions upon adsorption of acetone on TiO2
base material. Therefore, the concentration of electrons gets increased due to
the transfer of electrons from acetone to TiO2 base material [42, 43, 44, 45].
Moreover, the decrease in the energy gap and transfer of electrons from
acetone molecule to TiO2 nanostructures leads to an increase in the current
flowing across the two probe TiO2 device. The TiO2 base material can be used
for the design and development of acetone sensor. Figures 5–7 represents the
electron density of an isolated TiO2 nanostructure and for positions P, Q, R
and S.

Figure 5: (Color online) Electron density of an isolated TiO2 nanostructure.

Figure 6: (Color online) Electron density of position P and Q.

Figure 7: (Color online) Electron density of position R and S.

The variation in the electron density of an isolated TiO2 nanostructure and
acetone adsorbed TiO2 nanostructure clearly infers that transition of
electrons takes place between acetone and TiO2 nanostructure, which is also in
agreement with Mulliken charge transfer. The average energy gap value is found
to be high for positions R and S. Also the adsorption energy, Mulliken charge
transfer and energy gap variation are found to be favorable for position R.
However, for position S, the average energy gap variation is comparatively
high with low values of Q and $E_{\text{ad}}$. The Mulliken charge transfer
for position P is almost the same as that of position S, whereas the average
energy gap variation is comparatively very low. Figure 8 (a)–(d) illustrates
the band structure of TiO2 nanostructure for positions P–S, respectively.
Figures 9–12 represent the corresponding PDOS including DOS spectrum for
position P, Q, R and S. From the observation of energy band diagram for
positions P and Q, the energy gap is observed to be around 2.32 eV and 1.32
eV, respectively, along the gamma point (G). In the case of position R and S,
the energy gap is observed to be around 0.82 eV and 0.68 eV, respectively.
Comparing the changes in the energy gap upon adsorption of acetone on TiO2
nanostructure, the variation is observed to be significant for positions Q, R
and S. Meanwhile, on looking at the density of states (DOS) spectrum for
positions Q, R and S, the density of charge is observed to be larger than an
isolated TiO2 counterpart. (DOS spectrum is drawn in multi-curve fashion, the
magnitude is taken into consideration along $y$-axis).

Figure 8: (Color online) (a)–(d) Band structure of TiO2 nanostructure for
position P, Q, R and S.

An increase in the density of charge is due to the fact that since TiO2 is
n-type semiconductor, the adsorption of acetone molecules consequently gives
rise to a transfer of electrons between the acetone vapor and TiO2 base
material. The transfer of electrons will increase the electron concentration
in TiO2 base material, subsequently increasing the density of charge, which is
observed in DOS spectrum. The density of states spectrum gives the insight
that the electrons can freely transfer between acetone molecule and TiO2
nanostructure, which can be used as chemical sensor. Thus, the DOS spectrum of
TiO2 nanostructure strongly supports the adsorption of acetone molecule on
TiO2 material. In order to strengthen the above result, the partial (or
projected) density of states spectrum (PDOS) of an isolated TiO2 nanostructure
with adsorption sites for the positions P–S is illustrated in figures 9–12.
Moreover, the density of charges is noticed to be larger in positions Q, R and
S compared to an isolated TiO2 nanostructure.

Figure 9: (Color online) Projected density of states (PDOS) and (DOS) spectrum
of position P.

Figure 10: (Color online) Projected density of states (PDOS) and (DOS)
spectrum of position Q.

Figure 11: (Color online) Projected density of states (PDOS) and (DOS)
spectrum of position R.

Figure 12: (Color online) Projected density of states (PDOS) and (DOS)
spectrum of position S.

Furthermore, a prominent adsorption site of acetone vapor on TiO2
nanostructure can be concluded only after analyzing the adsorption energy,
Mulliken charge transfer, average energy gap variation and energy band gap.
From the observations, among all the positions it is found that when hydrogen
atom in acetone gets adsorbed on O atom in TiO2 nanostructure, position S is
found to be more prominent. Moreover, the selectivity of TiO2 nanostructure
towards acetone in the presence of other interfering gases in the ambient
condition is to be ascertained. It is well known to the sensor community that
sensitivity, selectivity and stability are important parameters to decide the
performance of metal oxide based gas sensors. Figure 13 depicts the
selectivity of acetone in ambient condition with other interfering gases,
namely H2O (humidity) and O2 gas molecules. The adsorption studies of O2 and
H2O on TiO2 nanostructure are carried out and the response towards O2 and H2O
molecules is observed.

Figure 13: (Color online) The coss-selectivity of acetone in ambient condition
with H2O (humidity) and O2 gas molecules.

The results show that the response of TiO2 nanostructure towards acetone is
found to be relatively high compared with H2O and O2 molecules. Besides, band
gap variation upon adsorption of H2O and O2 molecules on TiO2 nanostructure is
found to be low. Figure 14 shows the insight on the adsorption behavior of
acetone on TiO2 nanostructure, which can be used to design a simple two probe
device for a possible detection of acetone vapors present in the atmosphere.

Figure 14: (Color online) The insight on the adsorption behavior of acetone on
TiO2 nanostructure.

##  4 Conclusions

To sum up, the electronic properties and acetone adsorption properties on TiO2
nanostructure are studied using density functional theory method, which is
carried out with GGA/PBE exchange-correlation functional. The band gap of an
isolated TiO2 nanostructure is found to be around 2.43 eV. Furthermore, the
adsorption of acetone molecules on TiO2 nanostructures is confirmed by the
change in the adsorption energy, Mulliken charge transfer and average energy
gap variation. The results of DOS spectrum clearly reveal that the transfer of
electrons takes place between acetone vapor and TiO2 base material.
Furthermore, a prominent adsorption site is explored at atomistic level, which
confirms that when a hydrogen atom in acetone molecule gets adsorbed on the O
atom in TiO2 nanostructure, the adsorption is found to be favorable.
Furthermore, the selectivity of TiO2 nanostructure towards acetone molecules
with other interfering gases, namely H2O and O2, is also studied and reported.
The findings from the present investigation strongly support that TiO2
nanostructure can be efficiently used to detect the presence of acetone vapor
in the mixed environment. Thus, we conclude that a properly tailored TiO2
nanostructure can be used as a two-probe device to detect the presence of
acetone vapors.

## References

  * [1] Jia Q.Q., Ji H.M., Zhang Y., Chen Y.L., Sun X.H., Jin Z.G., J. Hazard. Mater., 2014, 276, 262, doi:10.1016/j.jhazmat.2014.05.044.
  * [2] Acetone: Health Information Summary, New Hampshire Department of Environmental Services, 2013, URL https://www.des.nh.gov/organization/commissioner/pip/factsheets/ard/documents/ard-ehp-7.pdf.
  * [3] Makisimovich N., Vorotyntsev V., Nikitina N., Kaskevich O., Karabun P., Martynenko F., Sens. Actuators B, 1996, 36, 419, doi:10.1016/S0925-4005(97)80107-4.
  * [4] Lim J.Y., Lee C.S., Lee J.M., Ahn J., Cho H.H., Kim J.H., J. Power Sources, 2016, 301, 18,
doi:10.1016/j.jpowsour.2015.09.109.

  * [5] Cai Y., Wang H.E., Jin J., Huang S.Z., Yu Y., Li Y., Feng S.P., Su B.L., Chem. Eng. J., 2015, 281, 844, doi:10.1016/j.cej.2015.06.122.
  * [6] Linsebigler A.L., Lu G., Yates J.T., Chem. Rev., 1995, 95, 735, doi:10.1021/cr00035a013.
  * [7] Volodin A.M., Catal. Today, 2000, 58, 103, doi:10.1016/S0920-5861(00)00245-5.
  * [8] Ochiai T., Fujishima A., J. Photochem. Photobiol. C, 2012, 13, 247, doi:10.1016/j.jphotochemrev.2012.07.001.
  * [9] Dambournet D., Belharouak I., Amine K., Chem. Mater., 2010, 22, 1173, doi:10.1021/cm902613h.
  * [10] Tang H., Lévy F., Berger H., Schmid P.E., Phys. Rev. B, 1995, 52, 7771, doi:10.1103/PhysRevB.52.7771.
  * [11] Amtout A., Leonelli R., Phys. Rev. B, 1995, 51, 6842, doi:10.1103/PhysRevB.51.6842.
  * [12] Mattsson A., Österlund L., J. Phys. Chem. C, 2010, 114, 14121, doi:10.1021/jp103263n.
  * [13] Landmann M., Rauls E., Schmidt W.G., J. Phys.: Condens. Matter, 2012, 24, 195503,
doi:10.1088/0953-8984/24/19/195503.

  * [14] Deák P., Aradi B., Frauenheim T., J. Phys. Chem. C, 2011, 115, 3443, doi:10.1021/jp1115492.
  * [15] Bhowmik B., Hazra A., Dutta K., Bhattacharyya P., IEEE Trans. Device Mater. Reliab., 2014, 14, 961, doi:10.1109/TDMR.2014.2347376.
  * [16] Chen N., Li Y., Deng D., Liu X., Xing X., Xiao X., Wang Y., Sens. Actuators B, 2017, 238, 491, doi:10.1016/j.snb.2016.07.094.
  * [17] Rella R., Spadavecchia J., Manera M.G., Capone S., Taurino A., Martino M., Caricato A.P., Tunno T., Sens. Actuators B, 2007, 127, 426, doi:10.1016/j.snb.2007.04.048.
  * [18] Chen Q., Zhu W., Hou X., Xu K., Vacuum, 2015, 119, 123, doi:10.1016/j.vacuum.2015.05.003.
  * [19] Soler J.M., Artacho E., Gale J.D., García A., Junquera J., Ordejón P., Sánchez-Portal D., J. Phys.: Condens. Matter, 2002, 14, 2745, doi:10.1088/0953-8984/14/11/302.
  * [20] Feynman R.P., Phys. Rev., 1939, 56, 340, doi:10.1103/PhysRev.56.340.
  * [21] Perdew J., Burke K., Wang Y., Phys. Rev. B, 1996, 54, 16533, doi:10.1103/PhysRevB.54.16533.
  * [22] Nagarajan V., Chandiramouli R., Microelectron. Eng., 2016, 162, 51, doi:10.1016/j.mee.2016.05.006.
  * [23] Perdew J., Chevary J., Vosko S., Jackson K., Pederson M., Singh D., Fiolhais C., Phys. Rev. B, 1992, 46, 6671, doi:10.1103/PhysRevB.46.6671.
  * [24] Dhivya G., Nagarajan V., Chandiramouli R., Chem. Phys. Lett., 2016, 660, 27, doi:10.1016/j.cplett.2016.07.061.
  * [25] Monkhorst H.J., Pack J.D., Phys. Rev. B, 1976, 13, 5188, doi:10.1103/PhysRevB.13.5188.
  * [26] Sriram S., Nagarajan V., Chandiramouli R., Chem. Phys. Lett., 2015, 636, 51, doi:10.1016/j.cplett.2015.07.018.
  * [27] Varthan S.H., Nagarajan V., Chandiramouli R., Process. Appl. Ceram., 2016, 10, 153,
doi:10.2298/PAC1603153S.

  * [28] Nagarajan V., Chandiramouli R., Chem. Phys., 2017, 495, 35, doi:10.1016/j.chemphys.2017.08.007.
  * [29] Kaloni T.P., Schreckenbach G., Freund M.S., Sci. Rep., 2016, 6, 36554, doi:10.1038/srep36554.
  * [30] Kaloni T.P., Schreckenbach G., Freund M.S., J. Phys. Chem. C, 2014, 118, 23361, doi:10.1021/jp505814v.
  * [31] Kaloni T.P., Schreckenbach G., Freund M.S., J. Phys. Chem. C, 2015, 119, 3979, doi:10.1021/jp511396n.
  * [32] Kaloni T.P., J. Phys. Chem. C, 2014, 118, 25200, doi:10.1021/jp5058644.
  * [33] Kaloni T.P., Singh N., Schwingenschlögl U., Phys. Rev. B, 2014, 89, 035409, doi:10.1103/PhysRevB.89.035409.
  * [34] Sun G.-J., Kheel H., Park S., Lee S., Park S.E., Lee C., Ceram. Int., 2016, 42, 1063,
doi:10.1016/j.ceramint.2015.09.031.

  * [35] Epifani M., Comini E., Díaz R., Genç A., Andreu T., Siciliano P., Morante J.R., J. Alloys Compd., 2016, 665, 345, doi:10.1016/j.jallcom.2016.01.077.
  * [36] Mulliken R.S., J. Chem. Phys., 1955, 23, 1833, doi:10.1063/1.1740588.
  * [37] Nagarajan V., Chandiramouli R., Appl. Surf. Sci., 2017, 419, 9, doi:10.1016/j.apsusc.2017.05.017.
  * [38] Beheshtian J., Baei M.T., Bagheri Z., Peyghan A.A., Appl. Surf. Sci., 2013, 264, 699,
doi:10.1016/j.apsusc.2012.10.100.

  * [39] Nagarajan V., Chandiramouli R., Comput. Theor. Chem., 2017, 1105, 52, doi:10.1016/j.comptc.2017.02.023.
  * [40] Nagarajan V., Chandiramouli R., Appl. Surf. Sci., 2017, 413, 109, doi:10.1016/j.apsusc.2017.04.008.
  * [41] Chandiramouli R., Srivastava A., Nagarajan V., Appl. Surf. Sci., 2015, 351, 662,
doi:10.1016/j.apsusc.2015.05.166.

  * [42] Nagarajan V., Chandiramouli R., Superlattices Microstruct., 2017, 101, 160, doi:10.1016/j.spmi.2016.11.032.
  * [43] Nagarajan V., Chandiramouli R., Condens. Matter Phys., 2017, 20, No. 2, 23705, doi:10.5488/CMP.20.23705.
  * [44] Chandiramouli R., Nagarajan V., Chem. Phys. Lett., 2016, 665, 22, doi:10.1016/j.cplett.2016.10.048.
  * [45] Nagarajan V., Chandiramouli R., Condens. Matter Phys., 2017, 20, No. 2, 23301, doi:10.5488/CMP.20.23301.

Ukrainian 0 Äåòåêòîð äëÿ âèçíàåííÿ ïîâåäíêè âèïàðâ àöåòîíó íà TiO2
íàíîñòðóêòóðàõ — çàñòîñóâàííÿ òåîð¿ ôóíêöîíàëó ãóñòèíè Â. Íàãàðàäæàí, Ñ.
Ñððàì, Ð. àíäðàìóë

Øêîëà åëåêòðîòåõíêè òà åëåêòðîíêè, Àêàäåìÿ ìèñòåöòâ, íàóêîâèõ òåõíîëîãíèõ
äîñëäæåíü Øàíìóãà (óíâåðñèòåò SASTRA), Òàíäæàâóð, Òàìë-Íàäó — 613 401, íäÿ