# BLSP-Emo: Towards Empathetic Large Speech-Language Models

Chen Wang1,3, Minpeng Liao2, Zhongqiang Huang2
Junhong Wu1,3, Chengqing Zong1,3, Jiajun Zhang1,322footnotemark: 2

1 Institute of Automation, Chinese Academy of Sciences
2 Alibaba Group
3 School of Artificial Intelligence, University of Chinese Academy of Sciences
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Work was done while at Alibaba.Corresponding author.

###### Abstract

The recent release of GPT-4o showcased the potential of end-to-end multimodal
models, not just in terms of low latency but also in their ability to
understand and generate expressive speech with rich emotions. While the
details are unknown to the open research community, it likely involves
significant amounts of curated data and compute, neither of which is readily
accessible. In this paper, we present BLSP-Emo (Bootstrapped Language-Speech
Pretraining with Emotion support), a novel approach to developing an end-to-
end speech-language model capable of understanding both semantics and emotions
in speech and generate empathetic responses. BLSP-Emo utilizes existing speech
recognition (ASR) and speech emotion recognition (SER) datasets through a two-
stage process. The first stage focuses on semantic alignment, following recent
work on pretraining speech-language models using ASR data. The second stage
performs emotion alignment with the pretrained speech-language model on an
emotion-aware continuation task constructed from SER data. Our experiments
demonstrate that the BLSP-Emo model excels in comprehending speech and
delivering empathetic responses, both in instruction-following tasks and
conversations.111Visit https://github.com/cwang621/blsp-emo for code and
https://cwang621.github.io/blsp-emo.github.io for demo.

BLSP-Emo: Towards Empathetic Large Speech-Language Models

Chen Wang1,3††thanks: Work was done while at Alibaba., Minpeng Liao2,
Zhongqiang Huang2††thanks: Corresponding author. Junhong Wu1,3, Chengqing
Zong1,3, Jiajun Zhang1,322footnotemark: 2 1 Institute of Automation, Chinese
Academy of Sciences 2 Alibaba Group 3 School of Artificial Intelligence,
University of Chinese Academy of Sciences<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in
intent understanding (Lu et al., 2023), instruction following (Chung et al.,
2022), and problem-solving (Achiam et al., 2023; Touvron et al., 2023),
revolutionizing human-machine interaction. Speech, as the primary mode of
human communication, conveys rich paralinguistic features related to emotions,
tones, and intentions that cannot be fully captured in text. Figure 1
illustrates that LLMs equipped with the ability to understand both linguistic
content and emotion cues in speech can enhance interaction experiences by
providing empathetic responses.

Figure 1: Illustrative example of an empathetic large language model
responding to speeches with identical linguistic content but different
emotional tones.

Recent work on end-to-end modeling of speech inputs with LLMs falls into two
categories. The first category focuses on adapting LLMs for a wide range of
speech and audio-related tasks, such as speech recognition, translation, and
emotion recognition (Rubenstein et al., 2023; Chen et al., 2023). However,
these models lack the ability to retain the general instruction-following
capabilities of LLMs and cannot engage in conversations with speech inputs.
The second category aims to extend LLMs’ instruction-following capability to
speech inputs, enabling direct speech interaction with LLMs (Zhang et al.,
2023; Wang et al., 2023a). Nevertheless, these approaches primarily focus on
the semantics in speech and fail to capture paralinguistic cues related to
emotions. Some studies have attempted to train models to understand emotions
in speech and respond empathetically (Xue et al., 2023; Lin et al., 2024).
However, these efforts rely on speech instruction data constructed with
expressive text-to-speech synthesis tools, which limits their generalization
capability with natural human speech. Annotating large quantities of new
emotion-sensitive instruction or conversation data for natural speech would be
costly.

In this paper, we present the BLSP-Emo approach, which aims to develop an end-
to-end speech-language model capable of understanding semantics and emotions
in speech and generating empathetic responses, using only existing speech
recognition (ASR) and speech emotion recognition (SER) datasets. BLSP-Emo
builds upon recent work on speech-language models developed with the BLSP
method (Wang et al., 2023a, 2024), which are bootstrapped from and aligned at
the semantic level with an LLM using ASR data. These speech-language models
exhibit generation behaviors consistent with the LLM when presented with
speech input containing the same linguistic content.

We propose to perform emotion alignment to understand emotions, in addition to
semantics, in speech and generate empathetic responses. Specifically, we first
prompt an LLM to generate emotion-aware continuations of transcripts in the
SER data given the reference emotion label. We then adapt a speech-language
model bootstrapped from the same LLM to generate these continuations directly
from speech. This adaptation step encourages the model to comprehend and react
to both the linguistic content and paralinguistic emotion cues in speech,
generating text continuations that are aligned with those the LLM would
produce if provided with the same linguistic content and emotion label.

The contributions of our work are as follows:

  * •

We introduce a new empathetic large speech-language model, adapted from an
instruction-following LLM, that can understand and respond to emotion cues in
speech with empathy, while maintaining its ability to follow speech
instructions and engage in conversations.

  * •

We develop a two-stage approach to adapt LLMs to empathetic large speech-
language models, using existing ASR data for semantic alignment and SER data
for emotion alignment, aiming to ensure that responses to speech input align
with those the LLMs would produce if provided with the same linguistic content
and emotion label.

  * •

We conduct quantitative evaluations and provide demonstrations to showcase
that the BLSP-Emo approach enables LLMs with competitive capabilities to
perform standalone speech emotion recognition, generate empathetic responses,
and engage in empathetic conversations.

Figure 2: Overview of the BLSP-Emo approach. In the first step, an LLM
generates emotion-aware text continuations using speech transcripts and
emotion labels as inputs. These generated continuations serve as supervisions
to train the model in the second step, where the corresponding speech is used
as input. Differences in the prompts used during data construction and the
training stage are highlighted in red font.

##  2 Method

Our proposed approach, termed BLSP-Emo, aims to develop an end-to-end speech-
language model that understands both linguistic content and paralinguistic
emotion cues in speech and generates empathetic responses. BLSP-Emo builds
upon bootstrapped speech-language models developed with the BLSP method (Wang
et al., 2023a, 2024), which are adapted from a text-only LLM using ASR data.
BLSP-Emo leverage SER data to enable these bootstrapped speech-language models
to also comprehend and react to the paralinguistic emotion cues. In what
follows, we will describe the model architecture and introduce how we achieve
semantic alignment and emotion alignment.

###  2.1 Architecture

BLSP-Emo models share a similar architecture as those in BLSP, comprising
three components: a speech encoder (with parameters $\psi$), an instruction-
following LLM (with parameters $\phi$), and a modality adapter (with
parameters $\theta$) between the speech encoder and LLM. Figure 2 provides an
overview of our model.

###  2.2 Semantic Alignment Stage

To achieve speech-text alignment at the semantic level and enable general
instruction-following capabilities for LLMs with speech inputs, we adopt the
behavior alignment approach used in BLSP Wang et al. (2023a). The core concept
is that if speech and text are well-aligned, the LLM’s text generation
behavior given speech input should closely match its behavior when given the
corresponding transcript. This alignment is accomplished by training on
synthesized speech instruction data derived from existing ASR datasets with a
continuation prompt as follows:

    User: Continue the following sentence in a
    coherent style: <transcript>
    Assistant:

This process extends an ASR training sample $(\mathbf{s},\mathbf{x})$ into a
tuple $(\mathbf{s},\mathbf{x},\mathbf{y})$, where $\mathbf{y}$ is the LLM’s
response, representing a natural continuation of the transcript $\mathbf{x}$
and the corresponding speech $\mathbf{s}$. The model is trained to generate
the same continuation when given speech input, using the same continuation
prompt. This is achieved by applying a KL-divergence loss according to the
knowledge distillation framework described in Wang et al. (2024), leading to
the semantic alignment loss:

$\displaystyle\ell_{\text{Semantic}}(\mathbf{s},\mathbf{x},\mathbf{y})=$
$\displaystyle\quad-\sum_{j,y}p_{\phi}(y|\mathbf{x},\mathbf{y}_{<j})\log
p_{\psi,\theta,\phi}(y|\mathbf{s},\mathbf{y}_{<j})$ (1)

In this semantic alignment stage, we focus on tuning the parameters $\theta$
of the modality adapter, keeping the parameters $\psi$ and $\phi$ of the
speech encoder and LLM frozen.

###  2.3 Emotion Alignment Stage

As studied in Busso et al. (2008); Castro et al. (2019), humans convey
emotions in speech through both linguistic and paralinguistic cues. A model
trained with the BLSP approach captures the linguistic cues for emotion but
lacks the ability to understand paralinguistic cues, as it is aligned at the
semantic level based on linguistic content. Ideally, an emotion-aware speech-
language model should be pretrained on large amounts of speech-text data to
understand the relationship between paralinguistic emotion cues and linguistic
context, and then fine-tuned on emotion-aware speech instruction data,
following the training paradigm used for text-only LLMs. However, this
approach requires extensive curated data and significant computational
resources, neither of which is readily accessible.

Our approach to emotion alignment builds upon and extends the behavior
alignment method by creating natural continuations of speech transcripts that
reflect the emotional tones in the speech. This is achieved by leveraging
existing speech emotion recognition (SER) datasets. Given a sample
$(\mathbf{s},\mathbf{x},e)$ from a SER dataset, where $e$ is the emotion label
annotated for speech $\mathbf{s}$, we prompt the LLM with the following
instruction:

    User: Continue the following sentence that reflects
    a <emotion> emotion tone in a coherent style:
    <transcript>
    Assistant:

This generates a text continuation $\mathbf{y}$ of the speech $\mathbf{s}$
that is consistent with the emotion label $e$. We then initialize the BLSP-Emo
model with parameters of the BLSP model trained from the semantic alignment
stage and fine-tune it to generate these continuations given only the speech
as input, as follows:

    User: Continue the following sentence based on the
    conveyed emotion tone in a coherent style:
    <speech features>
    Assistant: <text continuation>

This results in the primary emotion alignment loss based on emotion-aware
continuations:

$\displaystyle\ell^{\text{cont}}_{\text{Emotion}}(\mathbf{s},\mathbf{y})=-\sum_{j}\log
p_{\psi,\theta,\phi}(y_{j}|\mathbf{s},\mathbf{y}_{<j})$ (2)

We also introduce an auxiliary speech emotion recognition loss by directly
predicting the emotion label $e$ from the hidden states output by the modality
adapter, using pooling and a classification layer (with additional parameters
$\eta$):

$\displaystyle\ell^{\text{ser}}_{\text{Emotion}}(\mathbf{s},e)=-\log
p_{\psi,\theta,\eta}(e|\mathbf{s})$ (3)

In this emotion alignment stage, we unfreeze the parameters $\psi$ of the
speech encoder and parameters $\phi$ of the LLM, in addition to the parameters
$\theta$ of the modality adapter and $\eta$ of the classification layer. This
allows the speech encoder to capture paralinguistic emotion cues and provides
additional modeling power in the LLM to address the discrepancy between speech
and text. We follow the PLoRA approach proposed in Dong et al. (2024); Wang et
al. (2024) to adapt parameters $\phi$ of the LLM. The LoRA module is
selectively applied only to speech tokens, preserving the LLM’s ability to
encode text instructions and generate text.

##  3 Experiment Setup

###  3.1 Datasets

We use publicly available ASR datasets in the semantic alignment stage and SER
datasets in the emotion alignment stage.

The ASR datasets include LibriSpeech (Panayotov et al., 2015), CommonVoice
13.0 (Ardila et al., 2019), and the GigaSpeech M set (Chen et al., 2021),
totaling approximately 1.9 million English (speech, transcript) pairs, along
with a comparable number of Chinese ASR samples randomly selected from
WeNetSpeech (Zhang et al., 2022).

The details of the SER datasets and train/test splits can be found in Appendix
B. In summary, we train on IEMOCAP, MELD, CMU MOSEI, MEAD, and ESD, covering
approximately 70k utterances in English and Chinese, and evaluate SER
performance on IEMOCAP and MELD as in-domain test sets, on RAVDESS and
MerBench as out-of-domain test sets, as well as on three languages not seen in
training: AESDD for Greek, CaFE for French, and RESD for Russian. We focus on
five emotion categories: neutral, happy, sad, angry, and surprise across all
datasets.

We conduct evaluations on emotion-aware speech instruction capabilities based
on a synthesized version of Alpaca-52k (Taori et al., 2023), and emotion-aware
multi-turn conversation based on IEMOCAP (Busso et al., 2008), with details
presented in Section 4.

###  3.2 Training Details

We utilize the encoder part of Whisper-large-v2 (Radford et al., 2022) as the
speech encoder, convolution-based subsampler as the modality adapter, and
Qwen-7B-Chat (Bai et al., 2023) as the LLM. More details can be found in
Appendix C.

###  3.3 Baselines

Method | Tunable | Speech Emotion Recognition (Acc%)
---|---|---
Speech | Modality | LLM | IEMOCAP | MELD | RAVDESS | MerBench | MerBench
Encoder | Adapter | test1 | test2
_LLM-based Generative Models_ |  |  |  |  |  |
Text+LLM |  |  |  | 54.8 | 54.0 | 11.1 | n/a | n/a
Whisper+LLM |  |  |  | 57.1 | 53.8 | 13.7 | 49.4 | 46.9
BLSP |  | ✓ |  | 52.8 | 53.1 | 11.1 | 44.9 | 45.3
BLSP-SER | ✓ | ✓ | ✓ | 78.6 | 56.4 | 70.5 | 51.5 | 56.0
BLSP-Emo | ✓ | ✓ | ✓ | 76.0 | 57.3 | 72.0 | 60.0 | 54.7
_Encoder-based Classification Models_ |  |  |  |  |  |
HuBERT-Large | ✓ | ✓ |  | 64.6 | 53.2 | 70.5 | 55.6 | 45.3
wav2vec2-Large | ✓ | ✓ |  | 69.3 | 54.8 | 64.0 | 41.2 | 40.6
WavLM-Large | ✓ | ✓ |  | 68.9 | 54.6 | 70.3 | 48.3 | 42.8
SALMONN-7B |  | ✓ | ✓ | 67.0 | 32.9 | 38.8 | 45.8 | 41.7
Table 1: SER results on various datasets. "n/a" used for Text+LLM when
reference transcripts are not available.

We compare with the following baselines:

#### Text|Whisper+LLM

These are cascaded systems where the LLM input is either the ground-truth
transcript or the recognition output from Whisper-large-v2, which includes a
speech encoder, as used in BLSP-Emo, and a speech decoder.

#### BLSP

This model undergoes the semantic alignment stage described in Section 2.2 and
initializes BLSP-Emo before the emotion alignment stage.

#### BLSP-SER

This model is initialized from BLSP and fine-tuned directly on the SER task.
The only difference between BLSP-SER and BLSP-Emo is that the former is fine-
tuned to predict the ground-truth emotion label, while the latter generates
emotion-aware continuations, both utilizing the same SER training datasets.

#### HuBERT|wav2vec2|WavLM+Whisper+LLM

These are cascaded systems composed of a standalone SER module in addition to
the Whisper+LLM pipeline. The SER component is fine-tuned on the SER training
datasets from respective speech encoder models, including HuBERT large (Hsu et
al., 2021), Wav2Vec 2.0 large (Baevski et al., 2020), or WavLM large (Chen et
al., 2022), with the addition of an average pooling layer and a linear
classifier to predict the ground-truth emotion label. During evaluation, we
directly report the performance of the SER module for the SER task. For other
tasks, we first use the SER module and the Whisper model to respectively
predict the emotion label and transcript, and then use the following prompt to
generate responses:

    User: The user’s speech instruction, transcribed as
    "<transcript>", conveys a <emotion> emotion tone.
    Please provide a response.
    Assistant:

##  4 Experiments

Although BLSP-Emo is trained only on continuation tasks, we have found that
the resulting model has the ability to comprehend both linguistic content and
paralinguistic emotion cues in speech and respond accordingly. This enables
the model to not only follow task instructions but also demonstrate empathy
toward the emotional tone conveyed in the speech. Next, we will present
results on speech emotion recognition, instruction-following with empathetic
responses, multi-turn conversation, and generalization to other languages.

###  4.1 Main Results

#### Speech Emotion Recognition

To prompt the LLM-based generative models to perform the SER task, we use the
following prompt:

    User: Please identify the emotion tone of the
    sentence provided below. Select from the following
    options: neutral, sad, angry, happy, or surprise.
    \n\nSentence: <transcript|speech>
    Assistant:

where <transcript|speech> represents the transcript for cascaded systems or
speech features for end-to-end systems. Results are shown in Table 1.

The BLSP-Emo model achieves the highest overall recognition accuracy across
five test sets, along with the BLSP-SER model, which is fine-tuned from the
same BLSP model but specifically for the SER task. BLSP-Emo significantly
outperforms all other models, including SALMONN-7B (Tang et al., 2023), which
adapts a large language model to various speech tasks, including speech
emotion recognition.

The Text|Whisper+LLM cascaded systems achieve comparable or better results
than the encoder-based classification models on the MELD and MerBench test
sets, but they perform the worst on the IEMOCAP and RAVDESS test sets. This
suggests that while an LLM can capture linguistic cues for emotions, the text-
only mode limits its ability for comprehensive emotion recognition. The BLSP
model can process speech input but cannot pick up paralinguistic cues for
emotion as it is only trained with semantic alignment. Conversely, the
encoder-based classification models can capture paralinguistic cues but lack a
semantic understanding of emotion. In contrast, BLSP-Emo can simultaneously
model linguistic and paralinguistic emotion cues in speech, thanks to its end-
to-end modeling and two-stage alignment process.

#### Empathetic Response

Method | SER | Empathetic Response
---|---|---
Quality | Empathy
Text+LLM | 40.0 | 8.9 | 7.4
Whisper+LLM | 40.1 | 8.9 | 7.4
BLSP | 36.8 | 8.6 | 7.1
BLSP-SER | 80.3 | 1.9 | 2.1
BLSP-Emo | 83.8 | 8.8 | 7.7
HuBERT+Whisper+LLM | 76.3 | 8.9 | 7.6
wav2vec2+Whisper+LLM | 83.3 | 9.0 | 7.7
WavLM+Whisper+LLM | 80.8 | 8.9 | 7.8
SALMONN-7B | 43.8 | 2.4 | 1.9
Table 2: Results on SpeechAlpaca.

Beyond speech emotion recognition, our primary concern is whether the model
can understand both the semantic content and paralinguistic emotion cues in
speech and generate high-quality, empathetic responses. To evaluate this, we
construct a synthetic emotion-aware speech instruction dataset named
SpeechAlpaca, derived from the open-source instruction dataset Alpaca-52k
(Taori et al., 2023). Additionally, we use a modified system prompt222System
prompt: _You are a helpful assistant. Your response should fulfill requests
with empathy toward the user’s emotional tone._ that emphasizes both quality
and empathy for all systems. We then employ GPT-4 as an evaluator to
independently score the responses generated by different systems in terms of
quality and empathy on a scale from 0 to 10. For details on test set
construction and evaluation prompts, please refer to Appendix D. The results
are shown in Table 2.

Consistent with findings in the SER evaluation on natural speech, BLSP-Emo
achieves the highest emotion recognition accuracy of 83.8% on synthetic
speech. Additionally, BLSP-Emo scores competitively in both quality (8.8) and
empathy (7.7) as measured by GPT-4. In contrast, the BLSP-SER model, fine-
tuned specifically for the SER task, achieves a lower performance in SER
(80.3%) and performs poorly in empathetic response (quality: 1.9, empathy:
2.1), as it loses the ability to follow speech instructions learned during
semantic alignment.

The BLSP model, despite having a significantly lower SER score (36.8%),
achieves decent ratings in quality (8.6) and empathy (7.1), as it is able to
comprehend semantics and linguistic emotion cues thanks to semantic alignment.
The improvements from BLSP to BLSP-Emo in all three metrics—SER (36.8% to
83.8%), quality (8.6 to 8.8), and empathy (7.1 to 7.7)—suggest that the BLSP-
Emo approach effectively understands both linguistic and paralinguistic
emotion cues in speech while maintaining its instruction-following capability,
resulting in overall better responses.

The Text|Whisper+LLM systems achieve a slightly higher quality score (8.9 vs.
8.8) than BLSP-Emo but a lower empathy score (7.4 vs. 7.7) and significantly
lower SER scores (40.0% vs. 83.8%). This signifies that while LLMs have a
strong capability to capture linguistic emotion cues, they are limited by
their inability to understand paralinguistic emotion cues. As the examples in
Appendix D show, a text-only LLM can provide an empathetic response to the
instruction "Suggest the best way to avoid a traffic jam" based on the
semantic content alone. However, it cannot provide empathetic responses to a
neutral instruction "Come up with a 5-step process for making a decision"
stated in an angry voice.

The HuBERT|wav2vec2|WavLM+Whisper+LLM systems with standalone SER modules
achieve comparable quality ratings to the Text|Whisper+LLM systems but higher
empathy ratings (7.6$\sim$7.8 vs 7.4), further underlining the importance of
capturing paralinguistic emotion cues in generating empathetic responses. It
is worth noting that these cascaded systems also have slightly higher ratings
in quality than BLSP-Emo. We attribute this to the room for improvement in
semantic alignment for BLSP pretraining, as the Whisper model contains a
separate speech decoder that is trained on significantly more speech data
(Wang et al., 2023a, 2024). Additionally, despite being trained on various
speech tasks, large speech-language models like SALMONN (Tang et al., 2023)
exhibit limitations in following general speech instructions.

#### Multi-Turn Conversation

Figure 3: Results on multi-turn conversation.

We next evaluate multi-turn conversations, an important application scenario
for empathetic large speech-language models. This evaluation allows us to
determine if the emotion understanding capability of BLSP-Emo, learned from a
simple emotion-aware continuation task, can generalize to scenarios with
extended conversational context. Following a setup similar to Lin et al.
(2024), whose test set is not publicly available, we extract 3-turn dialogues
between two speakers from IEMOCAP (Busso et al., 2008), treating the first
speaker as the user and the second as the assistant. The conversation history
consists of the reference dialog transcripts from the first two turns, plus
the current input—either a transcript for a cascaded system or speech features
for an end-to-end model—from the user, along with the predicted emotion label
if the system has a standalone SER module. The LLM is then prompted to
generate a response. For examples, please refer to Appendix E.

Given that typical user inputs in conversations are not specific task
instructions, we found it difficult for GPT-4 to separately assess quality and
empathy as done on SpeechAlpaca. Instead, we employ GPT-4 as an evaluator to
determine which system’s output is better, based on reference transcripts in
the conversation history and the emotion label of the user’s most recent
input. For details, please refer to Appendix E.

As shown in Figure 3, BLSP-Emo demonstrates higher win rates compared to
Whisper+LLM, BLSP, and WavLM+Whisper+LLM. This advantage mirrors BLSP-Emo’s
comparative performance on SpeechAlpaca, highlighting its capability to
understand and respond to paralinguistic emotion cues in speech. Notably,
BLSP-Emo’s superiority over WavLM+Whisper+LLM is somewhat unexpected, given
that the latter performed comparably or slightly better on SpeechAlpaca in
both quality and empathy ratings. We speculate that this discrepancy may be
attributed to the specific prompt used, which incorporates both the transcript
and the recognized emotion tone for the user’s last speech input (as
illustrated in Appendix E). This could introduce inconsistency compared to the
simpler transcript representation of the conversation history. In contrast,
BLSP-Emo does not necessitate special prompting for speech input, as it
implicitly captures emotion cues in the speech features. While prompt
engineering could potentially enhance the performance of WavLM+Whisper+LLM,
this also underscores the simplicity and advantage of the BLSP-Emo approach.

#### Language Generalization

To explore whether the knowledge learned about emotion cues can generalize
across languages, we evaluate zero-shot SER performance on three languages not
included during training. As shown in Table 3, BLSP-Emo achieves the best
overall performance across the languages, performing comparably or better than
BLSP-SER and significantly better than the other models.

Method | AESDD | CaFE | RESD | Avg.
---|---|---|---|---
(Gr) | (Fr) | (Ru)
Whisper+LLM | 25.3 | 16.2 | 35.4 | 25.6
BLSP |  06.8 | 17.3 | 27.2 | 17.1
BLSP-SER | 68.9 | 76.7 | 41.4 | 62.3
BLSP-Emo | 68.8 | 75.3 | 46.2 | 63.4
HuBERT-Large | 53.9 | 66.5 | 43.0 | 54.5
wav2vec2-Large | 31.2 | 61.7 | 39.2 | 44.0
WavLM-Large | 47.0 | 70.7 | 37.3 | 51.7
SALMONN-7B | 31.4 | 36.3 | 39.2 | 35.6
Table 3: SER results on other languages. Method | SER | Empathetic Response
---|---|---
IEMOCAP | RAVDESS | SpeechAlpaca | Quality | Empathy
BLSP-Emo | 76.0 | 72.0 | 83.8 | 8.8 | 7.7
w/o pretraining | 68.5 | 68.6 | 80.3 | 6.7 | 7.0
w/o SER | 72.2 | 66.6 | 83.3 | 8.8 | 7.7
Table 4: Ablation study on the BLSP pretraining stage for semantic alignment and the auxiliary SER loss. Method | Training Task | Data Construction | SER | Empathetic Response
---|---|---|---|---
IEMOCAP | RAVDESS | SpeechAlpaca | Quality | Empathy
BLSP | continuation | same LLM | 57.1 | 11.1 | 36.8 | 8.6 | 7.1
BLSP-SER | SER | Human | 78.6 | 70.5 | 80.3 | 1.9 | 2.1
BLSP-Emo | emotion-aware continuation | same LLM | 76.0 | 72.0 | 83.8 | 8.8 | 7.7
BLSP-ChatGPT | emotion-aware continuation | GPT-3.5-turbo | 68.9 | 54.2 | 68.0 | 6.1 | 6.0
BLSP-MultiTask | continuation + SER | same LLM + Human | 75.3 | 71.5 | 77.8 | 8.3 | 7.2
Table 5: Comparison with ChatGPT data construction and multi-task learning.

###  4.2 Ablation Study

We conduct ablation studies to understand the impact of two training
strategies within the BLSP-Emo approach, with results presented in Table 4.
Directly applying emotion alignment without first performing BLSP semantic
alignment leads to a significant drop in both standalone SER performance and
quality/empathy ratings in empathetic response. This underscores the
importance of having a bootstrapped speech-language model that is aligned at
the semantic level before attending to paralinguistic cues.

Furthermore, incorporating the auxiliary SER classification task proves
beneficial for achieving higher performance in speech emotion recognition on
natural speech, even though it does not lead to any noticeable differences on
the SpeechAlpaca test set or in the evaluation of empathetic responses.

###  4.3 Analysis

We perform additional analysis comparing our training strategies against two
recent approaches in the literature of speech-language models with emotion-
aware capabilities.

First, we compare our approach to the method of E-chat (Xue et al., 2023) and
Spoken-LLM (Lin et al., 2024), which constructed synthesized emotion-aware
speech instruction data using expressive text-to-speech tools and ChatGPT. As
noted previously and found in our preliminary studies, models trained on
synthesized speech fail to generalize to natural human speech. Given that our
approach also requires constructing synthesized emotion-aware continuation
data for natural speech, a critical question arises: is it better to use
ChatGPT for data construction, as commonly done in the literature, or to use
the same LLM that BLSP-Emo is adapted from?

To address this, we trained a new model named BLSP-ChatGPT, utilizing ChatGPT
to generate emotion-aware continuations for emotion alignment, starting from
the same pretrained BLSP model as BLSP-Emo. As shown in Table 5, while BLSP-
ChatGPT achieves higher SER performance than BLSP, its quality and empathy
ratings in empathetic responses are notably lower. BLSP-ChatGPT performs worse
than BLSP-Emo across all metrics. We hypothesize that the emotion-aware
continuations generated by ChatGPT may not align well with the likely
responses generated by the internal LLM in BLSP-Emo. Consequently, the
alignment process may focus on narrowing the distribution gap between ChatGPT
and the internal LLM, rather than learning to capture the paralinguistic
emotion cues in speech to fit into the aligned semantic space established
during semantic alignment.

Next, we compare our approach against the multi-task learning strategy
employed by other large speech-language models, such as SALMONN (Tang et al.,
2023), which aims to understand semantic content and various paralinguistic
cues. As demonstrated in previous sessions, BLSP-Emo significantly outperforms
SALMONN-7B in both standalone emotion recognition and emotion-aware
instruction following. However, a question remains: can we replace the
emotion-aware continuation task employed in the emotion alignment stage with a
multi-task framework involving two tasks: emotion-agnostic continuation and
speech emotion recognition?

To answer this, we use the SER training datasets to construct two tasks: one
for standalone SER and another for emotion-agnostic continuation. The
resulting model is named BLSP-MultiTask. As shown in Table 5, while BLSP-
MultiTask significantly improves the SER accuracy of the BLSP model, its
response quality is lower than that of BLSP. BLSP-MultiTask also performs
worse than BLSP-Emo across all metrics. This comparison highlights the
importance of the emotion-aware continuation task in developing effective
empathetic speech-language models.

##  5 Related Works

See Appendix A for a discussion on related works.

##  6 Conclusion

In summary, this paper presents BLSP-Emo, a novel approach to build empathetic
large speech-language models by utilizing existing speech recognition and
speech emotion recognition datasets, through a two stage alignment process:
semantic alignment and emotion alignment. Through quantitative evaluations, we
demonstrate that the BLSP-Emo approach extends instruction-following LLMs with
competitive abilities to understand both semantics and emotions in speech and
perform standalone speech emotion recognition, generate empathetic responses,
and engage in multi-turn conversations.

## Limitations

#### Evaluation of Empathy.

While our methods for assessing empathetic responses provide valuable
insights, there are several limitations. Synthesized speech, as in
SpeechAlpaca, lacks variations in factors such as speaker ids and emotion
expressions, potentially limiting the accuracy of model performance evaluation
on natural human speech. Additionally, in the evaluation of multi-turn
conversations on IEMOCAP, we only assess a single-turn response within a
multi-turn context. This may not fully capture the model’s performance in
continuous conversations and how empathetic responses, sometimes repetitive,
are perceived from a user experience perspective.

#### Broader Applicability.

Our current approach to modeling emotions in speech relies on a limited number
of emotion states annotated in SER datasets. However, human speech has rich
expressions of emotions that are more nuanced and may include variations of
emotion in lengthy speech segments. Additionally, there are other types of
paralinguistic cues in human speech, such as tones and intentions, that are
important in communication but not addressed in this work. The two-stage
alignment approach, however, could be expanded to achieve general modeling of
paralinguistic cues through end-to-end modeling on large speech-text datasets,
while retaining instruction-following capabilities. We leave this to future
work.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ardila et al. (2019) Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019.  Common voice: A massively-multilingual speech corpus.  _arXiv preprint arXiv:1912.06670_.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_ , 33:12449–12460.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.
  * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008.  Iemocap: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_ , 42:335–359.
  * Castro et al. (2019) Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019.  Towards multimodal sarcasm detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4619–4629, Florence, Italy. Association for Computational Linguistics.
  * Chen et al. (2021) Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. 2021.  Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio.  _arXiv preprint arXiv:2106.06909_.
  * Chen et al. (2023) Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, et al. 2023.  Lauragpt: Listen, attend, understand, and regenerate audio with gpt.  _arXiv preprint arXiv:2310.04673_.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE Journal of Selected Topics in Signal Processing_ , 16(6):1505–1518.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  H. chi, jeff dean, jacob devlin, adam roberts, denny zhou, quoc v. le, and jason wei. 2022. scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Dong et al. (2024) Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. 2024.  Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.  _arXiv preprint arXiv:2401.16420_.
  * Gournay et al. (2018) Philippe Gournay, Olivier Lahaie, and Roch Lefebvre. 2018.  A canadian french emotional speech dataset.  In _Proceedings of the 9th ACM multimedia systems conference_ , pages 399–402.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Hu et al. (2024) Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, et al. 2024.  Wavllm: Towards robust and adaptive speech large language model.  _arXiv preprint arXiv:2404.00656_.
  * Lian et al. (2024) Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, and Jianhua Tao. 2024.  Merbench: A unified evaluation benchmark for multimodal emotion recognition.  _arXiv preprint arXiv:2401.03429_.
  * Lin et al. (2024) Guan-Ting Lin, Cheng-Han Chiang, and Hung-yi Lee. 2024.  Advancing large language models to capture varied speaking styles and respond properly in spoken conversations.  _arXiv preprint arXiv:2402.12786_.
  * Livingstone and Russo (2018) Steven R Livingstone and Frank A Russo. 2018.  The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english.  _PloS one_ , 13(5):e0196391.
  * Lu et al. (2023) Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023.  # instag: Instruction tagging for analyzing supervised fine-tuning of large language models.  In _The Twelfth International Conference on Learning Representations_.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: an asr corpus based on public domain audio books.  In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5206–5210. IEEE.
  * Poria et al. (2018) Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018.  Meld: A multimodal multi-party dataset for emotion recognition in conversations.  _arXiv preprint arXiv:1810.02508_.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision. arxiv.  _arXiv preprint arXiv:2212.04356_.
  * Rubenstein et al. (2023) Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. 2023.  Audiopalm: A large language model that can speak and listen.  _arXiv preprint arXiv:2306.12925_.
  * Shu et al. (2023) Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. 2023.  Llasm: Large language and speech model.
  * Tang et al. (2023) Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023.  Salmonn: Towards generic hearing abilities for large language models.  _arXiv preprint arXiv:2310.13289_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vryzas et al. (2018) Nikolaos Vryzas, Rigas Kotsakis, Aikaterini Liatsou, Charalampos A Dimoulas, and George Kalliris. 2018.  Speech emotion recognition for performance interaction.  _Journal of the Audio Engineering Society_ , 66(6):457–467.
  * Wang et al. (2023a) Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. 2023a.  Blsp: Bootstrapping language-speech pre-training via behavior alignment.
  * Wang et al. (2024) Chen Wang, Minpeng Liao, Zhongqiang Huang, and Jiajun Zhang. 2024.  Blsp-kd: Bootstrapping language-speech pre-training via knowledge distillation.
  * Wang et al. (2020) Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. 2020.  Mead: A large-scale audio-visual dataset for emotional talking-face generation.  In _European Conference on Computer Vision_ , pages 700–717. Springer.
  * Wang et al. (2023b) Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. 2023b.  Viola: Unified codec language models for speech recognition, synthesis, and translation.  _arXiv preprint arXiv:2305.16107_.
  * Xue et al. (2023) Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Qian Chen, and Lei Xie. 2023\.  E-chat: Emotion-sensitive spoken dialogue system with large language models.  _arXiv preprint arXiv:2401.00475_.
  * Zadeh et al. (2018) AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018.  Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2236–2246.
  * Zhang et al. (2022) Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et al. 2022.  Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6182–6186. IEEE.
  * Zhang et al. (2023) Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023.  Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.  _arXiv preprint arXiv:2305.11000_.
  * Zhou et al. (2022) Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. 2022.  Emotional voice conversion: Theory, databases and esd.  _Speech Communication_ , 137:1–18.

Dataset | Source | Language |  Emotion | #Utts
---|---|---|---|---
_Train Data_ |  |  |  |
IEMOCAP Session 1-4 | Act | English |  neutral, happy, sad, angry, excited, frustrated, fear, surprise, disappointed | 2610
MELD train | Friends TV | English |  neutral, happy (joy), sad (sadness), angry (anger), surprise, disgust, fear | 5920
ESD | Act | English&Chinese |  neutral, happy, sad, angry, surprise | 33443
CMU MOSEI | YouTube | English |  neutral, happy (happiness), sad (sadness), angry (anger), surprise, disgust, fear | 13557
MEAD | Act | English |  neutral, happy, sad, angry, surprise, contempt, disgust, fear | 15345
_Test Data_ |  |  |  |
IEMOCAP Session 5 | Act | English |  neutral, happy, sad, angry, excited, frustrated, fear, surprise, disappointed | 942
MELD test | Friends TV | English |  neutral, happy (joy), sad (sadness), angry (anger), surprise, disgust, fear | 1577
RAVDESS | Act | English |  neutral, happy, sad, angry, surprise, calm, fearful, disgust | 864
SpeechAlpaca | Microsoft TTS | English |  neutral, happy (cheerful), sad, angry | 400
MerBench test1 | Movies&TV series | Chinese |  neutral, happy (happiness), sad (sadness), angry (anger), surprise, worried | 354
MerBench test2 | Movies&TV series | Chinese |  neutral, happy (happiness), sad (sadness), angry (anger), surprise, worried | 360
AESDD | Act | Greek |  happy (happiness), sad (sadness), angry (anger), disgust, fear | 724
CaFE | Act | French |  neutral, happy (happiness), sad (sadness), angry (anger), surprise, disgust, fear | 648
RESD test | Act | Russian |  neutral, happy (happiness), sad (sadness), angry (anger), fear, enthusiasm, disgust | 158
Table 6: Overview of SER datasets. Emotion categories in parentheses indicate
original labels that are renamed for consistency, while struck-out labels
signify emotion categories not considered in our experiment.

##  Appendix A Related Works

#### Large Speech-Language Models

Large Language Models (LLMs) have achieved remarkable performance on various
natural language processing tasks (Achiam et al., 2023; Touvron et al., 2023).
Ongoing research aims to integrate speech signals into pre-trained, decoder-
only text-based LLMs, creating unified models capable of handling diverse
speech processing tasks. Models like AudioPaLM (Rubenstein et al., 2023),
VIOLA (Wang et al., 2023b), and LauraGPT (Chen et al., 2023) have emerged from
such efforts, primarily trained through multi-task learning for various speech
processing tasks, without utilizing conversational competencies inherent in
LLMs. Recent models like SALMONN (Tang et al., 2023) and WavLLM (Hu et al.,
2024), despite their conversational audio processing abilities using textual
instructions, still struggle with following general speech instructions. Other
efforts focus on generalized cross-modal instruction-following capabilities
through end-to-end frameworks, enabling direct interaction with LLMs via
speech, such as SpeechGPT (Zhang et al., 2023), LLaSM (Shu et al., 2023), and
BLSP (Wang et al., 2023a, 2024). However, these models primarily base
responses on linguistic content and cannot utilize paralinguistic features.

#### Interact with LLMs through Emotional Speech

Recent advancements in GPT-4o underscore the significance of integrating
paralinguistic emotion cues from user speech into LLM interactions. There are
multiple efforts to train LLMs to comprehend emotions in speech and deliver
empathetic responses. For instance, E-chat (Xue et al., 2023) developed an
emotion-aware speech instruction dataset for training models in this domain.
Similarly, Spoken-GPT (Lin et al., 2024) introduced a dataset covering various
speech styles, facilitating speech-to-speech conversations in a cascaded
manner. However, these approaches rely on TTS-synthesized speech for training,
posing challenges in generalizing to natural human speech.

##  Appendix B SER Datasets

A summary of the SER datasets employed in our experiments is presented in
Table 6, with each dataset categorized based on the following attributes:

  * •

Source: The origin of the collected samples.

  * •

Language: The language of the transcript.

  * •

Emotion: The labeled emotion categories.

  * •

#Utts: The number of utterances.

The SER datasets used during emotion alignment consist of sessions 1-4 of
IEMOCAP (Busso et al., 2008), the training set of MELD (Poria et al., 2018),
CMU MOSEI (Zadeh et al., 2018), MEAD (Wang et al., 2020), and ESD (Zhou et
al., 2022). Together, these datasets contribute to a corpus of approximately
70k utterances in English and Chinese. It’s worth noting that CMU MOSEI is a
multi-emotion-labeled dataset, meaning a speech segment could be annotated
with multiple emotions. However, we only utilize the single-label samples from
this dataset. In this work, we focus on the five emotion categories that are
widely annotated across datasets: neutral, happy, sad, angry, and
surprise333Due to the scarcity of the ”surprise” category in the IEMOCAP
dataset, we also excluded samples of this category.. To ensure the transcripts
provide sufficient semantic content for LLMs to generate meaningful
continuations, we filter out samples whose transcript contains fewer than 5
words in English or fewer than 5 characters in Chinese.

We evaluate SER performance on both in-domain datasets (IEMOCAP session 5,
MELD test set) and out-of-domain datasets (RAVDESS (Livingstone and Russo,
2018), MerBench (Lian et al., 2024)). Additionally, we report the
generalizability of SER performance on three other languages: AESDD (Vryzas et
al., 2018) for Greek, CaFE (Gournay et al., 2018) for French, and RESD (Vryzas
et al., 2018) for Russian.

##  Appendix C Training Details

We utilize the encoder part of Whisper-large-v2 (Radford et al., 2022) as the
speech encoder and employ Qwen-7B-Chat (Bai et al., 2023) as the LLM. The
modality adapter is composed of three 1-dimensional convolution layers
followed by a bottleneck layer with a hidden dimension of 512. The convolution
layers are designed to reduce the length of the speech features by a factor of
8, with each layer having a stride size of 2, a kernel size of 5, and a
padding of 2.

During the semantic alignment stage, we freeze the speech encoder and LLM, and
fine-tune the modality adapter for 1 epoch with a batch size of 768. This
process takes about 2.5 days on 4 A100 GPUs. During the emotion alignment
stage, we fine-tune the speech encoder, modality adapter, LLM444Using Partial
LoRA with hyperparameters $R=16$ and $\alpha=16$ for the key, query, value,
and output projection matrices that are activated only for speech tokens., and
SER classifier for 3 epochs with a batch size of 128. This process takes about
3 hours on 4 A100 GPUs.

##  Appendix D Evaluation on Empathetic Responses

Due to the lack of publicly available emotion-aware speech instruction
datasets to evaluate performance on empathetic responses, we construct a test
set named SpeechAlpaca from the open-source instruction dataset Alpaca-52k
(Taori et al., 2023). Specifically, we employ GPT-4 to deduce a set of
plausible emotional tones from a text instruction in Alpaca-52k, focusing on
four distinct emotions (neutral, cheerful, sad, and angry) that are supported
by Microsoft’s Text-to-Speech (TTS) API555https://azure.microsoft.com/en-
us/products/ai-services/text-to-speech. On average, GPT-4 suggests 1.4
plausible emotions per utterance due to ambiguities in determining the emotion
state from linguistic content alone. From these, we randomly select one as the
emotion label for the instruction. This process is used to select 100
instructions for each of the four emotion categories. Subsequently, we
synthesize expressive speech using the selected emotion label with Microsoft’s
TTS API.

We present examples of model outputs on the SpeechAlpaca test set in Table 7.
To evaluate the empathetic responses, we use GPT-4 to assess the quality of
responses with the prompt in Listing 1 and the empathy of responses with the
prompt in Listing 2.

Listing 1: Prompt for response quality evaluation on SpeechAlpaca

⬇

Given the original instruction provided by the user, the user’s emotion tone
when delivering the instruction, and the model’s response to the instruction.
You are a helpful and precise assistant for checking the quality of the
response.

<instruction>

{instruction}

</instruction>

<emotion>

{emotion}

</emotion>

<response>

{response}

</response>

Please evaluate the response with your justification having less than three
sentences, and provided a score ranging from 0 to 10 after your justification.
When evaluate the response, you should consider the helpfulness, harmlessness,
honesty of the response.

The score should be wrapped by <score> and </score>.

Listing 2: The prompt used to evaluate the empathy of response.

⬇

Given the original instruction provided by the user, the user’s emotional tone
when delivering the instruction, and the model’s response to the instruction.
You are a helpful and precise assistant for checking the empathy of the
response.

<instruction>

{instruction}

</instruction>

<emotion>

{emotion}

</emotion>

<response>

{response}

</response>

Please evaluate the response with your justification having less than three
sentences, and provided a score ranging from 0 to 10 after your justification.
When evaluate the response, you should consider whether it show empathy
towards the user’s emotional state.

The score should be wrapped by <score> and </score>.

Example 1 |  User: <speech>Suggest the best way to avoid a traffic jam.<sad/></speech>
---|---
Whisper+LLM |  Assistant: I’m sorry to hear that you’re experiencing traffic congestion. Here are some tips that may help you avoid a traffic jam: …
WavLM+Whisper+LLM |  Assistant:  I’m sorry to hear that you’re feeling sad about the possibility of getting stuck in a traffic jam. Here are some tips that may help you avoid one: …
BLSP-Emo |  Assistant: I’m sorry to hear that you’re feeling sad about the traffic jam. It can be really frustrating and stressful to be stuck in traffic for an extended period of time. Here are a few suggestions that might help you avoid traffic jams in the future: …
Example 2 |  User: <speech>Come up with a 5-step process for making a decision.<angry/></speech>
Whisper+LLM |  Assistant: Sure, I’d be happy to help you come up with a five-step process for making a decision. Here are the steps: …
WavLM+Whisper+LLM |  Assistant: I’m sorry to hear that you’re feeling angry. It’s understandable that making decisions can be frustrating and overwhelming at times. Here’s a five-step process that may help you make a decision:`\n\n`1\. Define the problem: … 2. Gather information: … 3. Consider your options: … 4. Make a decision: … 5. Take action: …
BLSP-Emo |  Assistant: I’m sorry to hear that you’re feeling angry. It’s important to take the time to process your emotions and make a decision that feels right for you. Here’s a five-step process that you can use to make a decision when you’re feeling angry:`\n\n` 1\. Take a deep breath and allow yourself to feel your anger. … 2. Identify the source of your anger. … 3. Consider the potential consequences of each option. … 4. Seek advice from a trusted friend or mentor. … 5. Make a decision and take action. …
Table 7: Examples of model outputs on the SpeechAlpaca test set. Each user
utterance, as enclosed in the <speech> tag, is synthesized into a waveform
using Microsoft’s TTS API with the indicated emotion label.

##  Appendix E Evaluation on Multi-turn Conversation

We present examples in Table 8 to illustrate the differences in responses
among various systems. To assess the comparative quality, we employ GPT-4 with
the prompt specified in Listing 3 for pairwise evaluation. To mitigate the
order bias of the GPT-4 evaluator, we conduct two evaluations for the outputs
of models A and B for the same sample: one in the AB sequence and the other in
the BA sequence. Model A is deemed the winner only if it is consistently
judged as better than B in both evaluations, while a loss is assigned only if
B is consistently superior in both. Otherwise, it is considered a tie.

Previous two turns |  User: yeah
---|---
Assistant: i don’t okay okay we can do this we can do this it’s okay we’ll
make it work
User: yeah
Assistant: i love you
Current turn |  Whisper +LLM |  User: i love you i’m going to miss you so much sigh
Assistant: i love you too and i’ll miss you too
BLSP |  User: <speech>i love you i’m going to miss you so much sigh<sad/></speech>
Assistant: i love you too and i’ll miss you too. but we can still stay in
touch and talk whenever we want. and who knows, maybe we’ll even be able to
see each other again someday.
WavLM +Whisper +LLM |  User: The user’s speech input, transcribed as "i love you i’m going to miss you so much sigh", conveys a "sad" emotion tone. Please provide a response.
Assistant: I’m sorry to hear that. Is there anything I can do to help ease
your sadness? Remember that you are not alone and that there are people who
care about you.
BLSP-Emo |  User: <speech>i love you i’m going to miss you so much sigh<sad/></speech>
Assistant:  i’m sorry to hear that you’re going to miss me. i’m here for you
and i’ll always be here to support you. remember that even though we may be
apart, our love is still strong and real.
Table 8: Examples of model outputs in multi-turn conversation constructed from
IEMOCAP. The user inputs shown for the current turn is the predicted
transcript for Whisper+LLM, extracted speech features for BLSP and BLSP-Emo,
and a constructed prompt for WavLM+Whisper+LLM in order to represent both
transcript and emotion. Listing 3: The prompt used to evaluate the win rate of
response.

⬇

Based on the dialogue history and the emotional tone expressed by the user in
their last statement, you are tasked to precisely evaluate two possible
responses (responses A and B) from Assistants A and B, respectively. You
should act as a thorough and accurate evaluator to determine which assistant’s
response better aligns with the preceding context and the emotional tone
expressed.

<history>

User: {text_u1}

Assistant: {text_a1}

User: {text_u2}

Assistant: {text_a2}

User: {text_u3}

</history>

<emotion>

{emotion}

</emotion>

<response_A>

Assistant: {response_a}

</response_A>

<response_B>

Assistant: {response_b}

</response_B>

Provide a concise justification for your choice in no more than three
sentences and conclude with a definitive selection between Response A and
Response B. Your evaluation should reflect how well each assistant’s response
adheres to the previous elements of the conversation, including the most
recent emotional tone presented by the user.

The choice should be wrapped by <choice> and </choice>.


# TransVisDrone: Spatio-Temporal Transformer for Vision-based Drone-to-Drone
Detection in Aerial Videos

Tushar Sangam1†, Ishan Rajendrakumar Dave1, Waqas Sultani2 and Mubarak Shah1
1Center for Research in Computer Vision lab (CRCV), University of Central
Florida, USA2Information Technology University of the Punjab, Lahore,
Pakistan†Corresponding Author<EMAIL_ADDRESS>
(August 2022)

###### Abstract

Drone-to-drone detection using visual feed has crucial applications, such as
detecting drone collisions, detecting drone attacks, or coordinating flight
with other drones. However, existing methods are computationally costly,
follow non-end-to-end optimization, and have complex multi-stage pipelines,
making them less suitable for real-time deployment on edge devices. In this
work, we propose a simple yet effective framework, TransVisDrone, that
provides an end-to-end solution with higher computational efficiency. We
utilize CSPDarkNet-53 network to learn object-related spatial features and
VideoSwin model to improve drone detection in challenging scenarios by
learning spatio-temporal dependencies of drone motion. Our method achieves
state-of-the-art performance on three challenging real-world datasets (Average
Precision@0.5IOU): NPS 0.95, FLDrones 0.75, and AOT 0.80, and a higher
throughput than previous methods. We also demonstrate its deployment
capability on edge devices and its usefulness in detecting drone-collision
(encounter). Project: https://tusharsangam.github.io/TransVisDrone-project-
page/

##  I Introduction

Drones have seen great popularity in various real-world applications such as
surveillance, package delivery, military applications, agricultural robotics
[1], etc. From the perception (computer vision) point of view, drone visual
feed can be used to address various problems such as human action recognition
[2, 3, 4, 5, 6], behavioral understanding [7], ground object detection and
tracking [8, 9], etc. Apart from the above ground object/actors detection, it
is also crucial to detect other airborne objects like the other drones or
birds to prevent collisions during flight. [10], tackle a drone attack [11],
or coordinate flights with other drones.[12]. Although drone detection from
aerial videos has crucial prospects, it is an under-explored research problem.

Drone-to-Drone detection has a more challenging nature compared to standard
object detection problems. The major challenges are: First, small-sized object
(drone): in the captured aerial videos, typical target drones are only 0.07%
of the frame-size [13], whereas in standard detection problems, the object
size is about 20% [14, 15]. Second, Movement of target drones can be fast and
erratic which often blurs video frames and make it difficult to detect
objects. Third, Egomotion or the source drone movement also makes the
detection and tracking of the target drone difficult.

(a) Target-drone is above-the-horizon with the clouds in its background.

(b) Abrupt illumination change due to egomotion and facing the sun.

(c) Target-drone is below-the-horizon which results in a cluttered
background with movements in vegetation or other ground vehicles.

Figure 1: Qualitative Visualization. Our TransVisDrone method is successfully
able to detect the drone even in various challenging scenarios. Green Box
indicates ground-truth, Red Box indicates output prediction box. (Best view
with 300% zoom-in)

Fourth, Uncontrolled Surroundings: aerial videos are often captured in an
outdoor setting, which brings many challenges like variable lighting,
cluttered background, occlusion, etc. Apart from these challenges,
computational efficiency (i.e. throughput) is highly desirable since most
drones need to process the data on edge-computing resources.

There are a few standard object detection methods such as Mask-RCNN [16], YOLO
[17], and De-DETR [18] which can be employed to detect drones in a drone
video. However, since they do not exploit the temporal dynamics of the video,
they perform at a sub-optimal level due to a lack of temporal coherence in the
predictions and can miss the target drone when there is motion due to source
or target drone or both.

A recent method DogFight [19] utilizes a two-stage segmentation approach. As a
part of preprocessing, a frame is first divided into overlapping patches. In
the first stage, contextual information in each frame is learned through a 2-D
convolution network and channel-pixel-wise attention. The frame-wise
detections obtained through the first stage are connected through a connected
component analysis and an off-the-shelf tracker. These obtained tracks are
processed through 3D-Convolution networks and channel-pixel-wise attention.
Although [19] achieves the state of art results, it has several downsides:
First, High computational requirement: since their framework utilizes 2D and
3D convolution networks along with channel-pixel attention modules in both
stages, it requires large computational resources. Second, Low throughput/
FPS: the framework takes multiple overlapping crops from a single frame and
processes them individually. It also utilizes connected component analysis and
an off-the-shelf tracker which are implemented on CPU and do not get benefit
from the parallel operations of GPU. Third, Non-Differential Components: It is
not an end-end approach; i.e. between stage-1 and stage-2, the method requires
non-differentiable components like connected component analysis and off-the-
shelf tracker, which makes it complicated to train, since first, we need to
store tracks from stage-1 for all videos and then start training stage-2. The
non-differentiable components also require hand-crafted filters which
introduce dataset-specific inductive biases. Rozantsev et al. [10] also face
similar issues to [19].

In the spirit of tackling the challenges of [19, 10] and other prior work for
real-time applications, we propose a new framework: spatio-temporal
TRANSformer for VISion-based DRONE-to-Drone Detection (TransVisDrone). We
propose a simple end-to-end framework based on CSP-DarkNet53 [20] and Video-
Swin transformer [21]. Our overall framework is shown in Fig. 2. First, a clip
is sampled from the flight video within a temporal window and processed
through stochastic temporally-consistent transformations. The transformed clip
is flattened across time dimension and fed to CSP-DarkNet53 to obtain spatial
features in each frame. Finally, to exploit video temporal information, drone
features of a short video clip are passed through Video-Swin model to learn
the spatio-temporal dependencies among them. The final output is obtained
through standard detection head and non-maximum suppression operations.

The major contributions of our work can be summarized as follows:

  * •

We propose a simple, efficient, and end-to-end trainable framework for drone
detection in the videos captured from a drone. Our approach detects drones
using only video feed without relying on expensive payloads, Lidars, etc, [22,
23]. Our method is fully differentiable and does not require any handcrafted
algorithms as used in the prior work [19, 24].

  * •

Our method establishes new state-of-the-art on three publicly available drone
detection datasets: NPS [13], FL-Drone [24], and AOT dataset [25]. These
datasets differ significantly in complexity, drone sizes, and a number of
videos.

  * •

We perform detailed ablation studies of various design choices of our
framework and conduct experiments on edge-device to demonstrate the usefulness
of our approach for real-world applications like drone encounter detection.

##  II Related Works

###  II-A Drone detection

Drone detection has been studied by various works in the aerial robotics
community, however, they mainly involve non-visual sensor data or static
cameras from the ground. Dressel and Kochenderfer [22] try to detect other
intruder drones for security purposes using RF sensors of the target drones.
Since this approach is constrained to the attached sensors of the target
drones, it is not applicable to airborne objects that do not bear the RF
sensors such as birds and balloons. [26] proposes a self-supervised learning
approach for visual localization of a quadrotor using its own noise as a
source of supervision. [27] presents an efficient self-supervised deep neural
network approach for monocular multi-robot relative localization. Yang and
Quan [28] use visual feed and computer vision models to guide the servo and
intercept intruder drones. However, their data is collected from a static
camera on the ground in a controlled environment and it only captures low-
altitude samples. Since their image acquisition setup is highly controlled, it
is not suitable for the drone camera and detecting challenging surroundings.
Chen et al. [29] uses point cloud data to segment the voxels and avoid
obstacles. However, obtaining the point cloud data is not inexpensive and
requires LIDAR sensors. Dogru and Marques [23] also use LIDAR sensors placed
on the ground to detect UAVs in the air. Cao et al. [30] use the Siamese
network to perform visual tracking of the objects in the UAV camera feed.
However, they focus on the objects on the ground instead of other airborne
objects. Also, they only process the spatial information and do not take the
temporal context into consideration. Yu et al. [31] uses visual sensors along
with radars to avoid obstacles. However, their visual computing is based on
optical flow which is computationally slow for a real-time system. Also, their
work is not demonstrated on real-world datasets. Rozantsev et al. [24] propose
a sptio-temporal(ST) cube that can combine the spatial & motion features. To
achieve that they employ two CNNs in a sliding window fashion & followed by a
third CNN to detect UAVs in each ST cube. This approach suffers from the
similar problems of being multi-stage, complex post-processing, and
computationally expensive to deploy.

Figure 2: TransVisDrone Framework: First, a set of frames is sampled within a
temporal window (shown in blue color) from the captured flight video into a
clip. This sampled clip is transformed through temporal-consistent
augmentations to obtain the input clip for our detection model $I_{x,y,t}$,
here $x,y$ indicates pixel coordinates, and $t$ indicates the timestamp of the
frame. Each frame of input clip is passed through CSP-DarkNet53 ($f_{x,y}$)
[20] to obtain object-related spatial features from different layers
($P_{3},P_{4}$, and $P_{5}$). To learn the spatio-temporal dependencies from
these frame-wise spatial features, we use Spatio-Temporal Swin
Transformer($g_{x,y,t}$) [21]. Finally, output tokens of the transformer are
sent to the detection head and non-maximum suppression module to obtain the
final detection output object bounding box sequence.

###  II-B Transformers for temporal context

We want to improve drone detection in videos by learning drone motion from the
temporal context. Recently, transformer-based self-attention is gaining
popularity in the computer vision community because of its ability to focus on
a non-local affinity of the data, which was previously a major limitation of
convolution neural networks. A self-attention can successfully learn global
temporal context from the sequence of frames and can encode the appearance and
motion information of the moving object [32]. However, learning the global
temporal context via transformer is computationally expensive, hence, Liu et
al. [33, 21] develop a method to cut the cost of global self-attention by
introducing a shifted window-based attention mechanism. Considering the
ability to deploy on edge-computing devices, we choose VideoSwin instead of a
full self-attention-based transformer.

##  III Method

The proposed method is based on the following three insights: First, to tackle
the large variation of drones and background scenes, data augmentation which
maintains the video information in a short clip creates challenging scenarios;
Second, due to real time applications and tiny object sizes, a fast and multi-
scale, multi-level features extractor should be used for accurate detections;
Third, temporal (video) information should be exploited while attending the
important regions in the videos. To accomplish these goals, our framework
consists of three components: (1) Temporally consistent preprocessing, (2)
Spatial feature extractor module (3) Spatio-temporal SwinTransformer [21]
module. A schematic diagram of our framework is shown in Fig. 2.

###  III-A Preprocessing

From any $i^{th}$ flight video $V^{i}$, frames within a temporal window,
$\tau$ are selected, which we call sampled clip $S_{x,y,t}$. In the training
mode, we select the sampled clip from a random temporal location, whereas,
during the inference mode, we use the sliding window method to cover the whole
video. The sampled clip is then transformed through Temporally-Consistent
Augmentation (TCA) module. The main goal of this module is to first select a
random subset of augmentations (details in Sec. IV-A) and apply the same
augmentations on every frame of $S_{x,y,t}$. Note that, if each frame of the
clip is transformed differently, it destroys the temporal dynamics of video
[34, 35].

###  III-B Spatial feature module

In CNN, feature map resolution is decreased in subsequent layers using max-
pooling operation which retains only the maximum values in local regions,
resulting in the loss of fine geometrical details required for small object
detection. Thus we use CSPDarknet53 [36] which doesn’t downscale features by
the common max-pooling operation & produces multi-scale features using spatial
pyramid pooling block SPP [37]. Each frame sampled at the end of the TCA III-A
module is then passed through the CSPDarknet53 [36] backbone and we obtain
multi-scale ($P_{3},P_{4},P_{5}$) spatial features for each frame in the clip
samples.

###  III-C Spatio-Temporal Transformer

Spatial features obtained in Sec. III-B lack temporal video information and
often miss drones due to their abrupt shape change and motion blur. Therefore,
these multi-scale spatial features are fed to corresponding VideoSwin [21]
branches to learn spatio-temporal dependencies. Figure 2 shows three different
VideoSwin [21] branches being applied at three different feature scales. The
standard Multi Head Self Attention MSA [38] layer works by dividing the input
into fixed-sized small patches & then learning their relations through self-
attention. SwinTransformer [21] blocks work more efficiently than the standard
MSA head[38] by dividing the data into bigger patch sizes. In our case, we set
the default patch size of $8\times 8\times\tau$. If the resolution of the
spatial feature is $H\times W$, then the input feature size of the VideoSwin
branch will be $H\times W\times\tau$. Thus after patching the input spatial
feature with $\frac{H}{8}\times\frac{W}{8}$ patches and each patch size
becomes $8\times 8\times\tau$. It then applies the MSA [38] inside each 3D
patch, by further dividing this patch into smaller $M\times M\times\tau$
patches. This is the first layer of the SwinTransformer [21] block referred to
as 3DW-MSA (3D Window-MSA) layer. For the next layer, windows are shifted by
$(\frac{8}{2},\frac{8}{2},0)$ and MSA [38] is applied again. This layer is
referred to as 3DSW-MSA (3D Shifted Window-MSA). Since there is a spatial
overlap between layer 1 & layer 2, it can capture the spatio-temporal global
cues over the large receptive field effectively. Since there is a relatively
small translation motion in the consecutive frames, attending over the local
area instead of the entire spatial map is cost-effective & performant. Details
of the design choice of the attention window size are analyzed in Sec IV-E3.

###  III-D Loss functions

To optimize our framework, we utilize standard loss functions introduced by
YOLO [17], which are: (1) objectness loss (2) classification loss, and (3)
localization loss. The feature map $g(f(I_{x,y,t}))$ is considered as a
$S\times S$ grid, and in each cell, $B$ bounding boxes are predicted and
prediction losses are applied. As shown in the following equation, the
objectness loss is calculated based on the condition if the object is present
or not in the cell.

$\small\mathcal{L}_{objectness}=\sum_{k=0}^{S^{2}}\sum_{j=0}^{B}{\mathbb{1}}_{kj}^{\text{obj}}\left(C_{k}-\hat{C}_{k}\right)^{2}\\\
+\lambda_{\textrm{noobj}}\sum_{k=0}^{S^{2}}\sum_{j=0}^{B}({1-\mathbb{1}}_{kj}^{\text{obj}})\left(C_{k}-\hat{C}_{k}\right)^{2},$
(1)

where, ${\mathbb{1}}_{kj}^{\text{obj}}$ is indicator binary function which
takes value of 1 if $j^{th}$ bounding box in cell k contains the object.
$\lambda_{\textrm{noobj}}$ is a hyperparameter which is set to 5 as per [17].
As shown in the Equation 2, a class-specific loss is computed using square of
the error between predicted conditional class probability $\hat{p}_{k}(c)$ and
ground-truth $p_{k}(c)$ for cell $k$.

$\mathcal{L}_{classification}=\sum_{k=0}^{S^{2}}\mathbb{1}_{k}^{\text{obj}}\sum_{c\in\textrm{classes}}\left(p_{k}(c)-\hat{p}_{k}(c)\right)^{2},$
(2)

where, $\mathbb{1}_{k}^{\text{obj}}$ is indicator binary function which takes
value of 1 if an object is present in the $k^{th}$ cell. The third loss is
computed from the discrepancy between the predicted bounding box and the
ground-truth as shown in the equation below.

$\mathcal{L}_{localization}=\sum_{k=0}^{S^{2}}\sum_{j=0}^{B}{\mathbb{1}}_{kj}^{\text{obj}}\left[\left(x_{k}-\hat{x}_{k}\right)^{2}+\left(y_{k}-\hat{y}_{k}\right)^{2}\right]\\\
+\lambda_{\textbf{coord}}\sum_{k=0}^{S^{2}}\sum_{j=0}^{B}{\mathbb{1}}_{kj}^{\text{obj}}\left[\left(\sqrt{w_{k}}-\sqrt{\hat{w}_{k}}\right)^{2}+\left(\sqrt{h_{k}}-\sqrt{\hat{h}_{k}}\right)^{2}\right],$
(3)

where, ($\hat{x}_{k}$, $\hat{y}_{k}$) and (${x}_{k}$, ${y}_{k}$) are predicted
top-left corner coordinates of predicted and groundtruth bounding box,
respectively. Whereas, ($\hat{w}_{k}$, $\hat{w}_{k}$) and (${h}_{k}$,
${h}_{k}$) are predicted width and height of predicted and groundtruth
bounding box, respectively. $\lambda_{\textbf{coord}}$ is a hyperparameter set
to 5 as per [17].

##  IV Experiments

###  IV-A Implementation Details

Since target drones are not always present in every frame of flight videos,
drone datasets [13, 24, 25] do not provide any annotations for such empty
frames. Therefore, for training, we use only frames that provide the
annotations, and for evaluation, we test all frames with a skip rate of 4
following the protocols of prior work [19]. In Temporal Consistent
Augmentation module, we use standard augmentations like perspective
transforms, cutout, and color jittering. In order to reduce the extensive
hyperparameter search for the augmentations, we follow the augmentation
strengths of Zhu et al. [39] which deals with small ground-object detection.
After augmentation module, we resize the frame to $1920\times 1280$. We use
Adam [40] optimizer with a momentum of 0.843 & learning rate of 3e-5. We apply
cosine-decay learning rate scheduler [41] during training. Following the prior
work [19], we start the training from publicly available pretrained model
weights of yolov5l [42] on MS-COCO [43]. For the non-maximum suppression
module, we set IoU threshold at 0.6 and the confidence threshold at 0.001. Our
codebase can be found on GitHub
111https://github.com/tusharsangam/TransVisDrone.

###  IV-B Evaluation Protocol

For evaluation, we set the IoU-threshold between predictions and ground truth
to 0.5. Therefore detections matching with ground-truth with IoU$\geq$0.5 are
counted as the true positives. Following the prior work, we report average
precision (AP) and a Precision-Recall pair corresponding to the best F1-Score.
Since AP is averaged over uniformly spaced 11 operating points of the
precision-recall curve, it is more reliable compared to the best precision-
recall pair. Following prior work [19], we evaluate on every $4^{th}$ frame.

###  IV-C Datasets

We use three challenging drone datasets. Following Dogfight [19], we use NPS
[13] and Fl-drones [24]. In addition, we also provide evaluations on the newly
released Airborne Object Tracking Dataset (AOT) [25] [44].

NPS-Drones dataset [13] This dataset has 50 videos of custom delta wing air
frames with a total number of frames adding up to 70250. Videos are captured
from GoPro 3 camera with an HD resolution of $1920\times 1280$ or $1280\times
760$. The resolution is sufficient for capturing objects at a far distance
range. Objects in this dataset are mainly small drones as shown in Fig. 1(a) &
Fig. 1(b). Object size typically ranges from $10\times 8$ to $65\times 21$. We
use the clean version annotations released by Ashraf et al. [19]. Following
the train/val/test split of [19], we use video-id #01-#36 for training, videos
#37-#40 for validation and #41-#50 videos for testing split.
FL-Drones dataset [24] This small-scale dataset is proposed by EPFL with 14
videos which totals up to 38,948 frames. It is captured from a camera mounted
on the flying drones with a mix of indoor and outdoor scenes. Frames are at
the resolution of $640\times 480$ or $752\times 480$ in grayscale.
Additionally drone object sizes vary from $9\times 9$ to $259\times 197$.
Following Ashraf et al. [19] we use half of the frames of each video for
training and the other half for testing. We use the cleaned version of
annotations released by Ashraf et al. [19].

Airborne Object Tracking AOT dataset[25] This was released for the ICCV 2021
workshop challenge [44] hosted by Amazon Prime Air. This dataset is collected
from a high-resolution camera mounted onboard aerial vehicles. It contains up
to 5.9M+ frames, collected at the resolution of $2448\times 2048$ in
grayscale. The planned encounters in the flight sequences have trajectories
that are intended to generate a broad range of distances, closing speeds, and
angles of approach. Airborne objects include helicopters, airplanes, drones,
and other unplanned airborne objects such as birds, flocks, and balloons. The
objects have labels, but information about their distance is not provided.
Refer to Fig. 1(c) for a sample frame from this dataset. The original
challenge test set is sequestered and the evaluation server is no longer
active. Since there is no test set released, we use part#1 of the dataset
where 516 videos are for training & 171 videos for testing, while the
remaining 300 videos are for validation.

###  IV-D Comparison with Prior works

We compare our method with several recent state-of-art methods on NPS and
FLDrones datasets as shown in Table I. Performance wise our TransVisDrone
outperforms the prior method by 3% and 1% absolute on AP metric on NPS [13]
and FLDrone datasets [24], respectively. As shown in Fig. 4 and Table I, with
the comparable performance, our method is significantly better than prior
methods in terms of throughput which is measured in frames-per-second.
Experiments performed on NVIDIA RTX A6000 48G GPU.

We also propose evaluation on AOT dataset [25], where we show results of our
method and train a new model of Dogfight [19] using their official code
repository. As shown in Table II, we outperform the prior method by 6%
absolute on AOT dataset [25]. Various operating points of our method can be
seen at the precision-recall curve in Fig. 3.

Methods | AP-NPS [13] | AP-FLDrones [24] | FPS
---|---|---|---
SCRDet-H [45] | 0.65 | 0.52 |
SCRDet-R [45] | 0.61 | 0.52 |
FCOS [46] | 0.83 | 0.62 |
Mask-RCNN [16] | 0.89 | 0.68 | 17.55
MEGA [47] | 0.83 | 0.65 |
SLSA [48] | 0.46 | 0.61 |
De-DETR [18] | 0.76 | _ | 10.69
VisTR [49] | 0.66 | _ | 1.6
yolov5-tph [39] | 0.92 | 0.69 | 25
Dogfight [19] | 0.89 | 0.72 | 1.0
Ours | 0.95 | 0.75 | 24.6
TABLE I:  Drone-to-drone detection results on NPS [13] and FL-drones [24] datasets. The best method is shown in Red and the second best method is shown in Blue. Methods | AP | Precision | Recall
---|---|---|---
Dogfight [19] | 0.74 | 0.82 | 0.65
Ours | 0.80 | 0.82 | 0.72
TABLE II:  Drone-to-drone detection results on AOT [25]. Figure 3: Precision
vs Recall Curve of our method on there benchmark datasets.

###  IV-E Ablations

In this section, we study various building blocks and design choices of our
framework and try to report their performance and throughput trade-off.

####  IV-E1 Size of Temporal Window

In order to understand the impact of temporal context in our framework, we
carry out experiments with three different sizes of the temporal window
($\tau$): {1,3,5}. Experiments are performed on NPS and AOT datasets with 1280
resolution. We do not perform $\tau=5$ on AOT due to its immense training
time. As shown in Table III, increasing the temporal window size from 1 to 3,
gets noticeable performance improvement, whereas, further increasing the size
to 5 slightly improves the precision and recall. This implies that short-term
motion ($\tau=3$) is helpful in is very helpful in learning the spatio-
temporal features of drone motion which results in improving the detection
performance over single-frame detection, but the improvement diminishes with
longer temporal context. In our final setting, we use $\tau=5$ due to its
better performance on precision-recall.

Our model on NPS [13] | AP | Precision | Recall
---|---|---|---
$\tau=1$ | 0.93 | 0.88 | 0.89
$\tau=3$ | 0.95 | 0.91 | 0.90
$\tau=5$ | 0.95 | 0.92 | 0.91
Our model on AOT [25] | AP | Precision | Recall
$\tau=1$ | 0.69 | 0.73 | 0.63
$\tau=3$ | 0.73 | 0.80 | 0.67
TABLE III: Ablation study of Temporal Window($\tau$) on NPS [13] test set &
AOT [25] val set. Image@1280 reso.

####  IV-E2 Spatial resolution of frame

We find that the spatial resolution of the frame provides a spectrum of
operating points to choose the right trade-off between performance and
throughput. We use three different resolutions: 1280, 800, and 640. As shown
in Table IV, when we decrease the resolution from 1280 to 640, it only costs
3% absolute drop in AP while gaining 250% in throughput (FPS).

Methods | AP | Precision | Recall | fps
---|---|---|---|---
Image@640 | 0.91 | 0.90 | 0.84 | 87.71
Image@800 | 0.92 | 0.88 | 0.88 | 57.14
Image@1280 | 0.95 | 0.92 | 0.91 | 24.6
TABLE IV:  Ablation of Frame resolution on NPS [13].

####  IV-E3 Attention Window of VideoSwin

We test two configurations of the VideoSwin windowed attention: (1) Attention
Window = 3, and Window Shift = 2; and (2) Attention Window = 5, and Window
Shift = 0. We keep the spatial size of the window at $8\times 8$ and their
shift stride at $4\times 4$. We vary only the Depth and its corresponding
shift stride. We fix $\tau$ to 5 and image resolution to 640 $\times$ 640\. As
shown in Table V, we can see that results are very close to each other in both
performance and throughput. In our best experiment setting, we use attention
window = 5, shift = 0 setting due to its higher AP.

Transformer Settings | AP | Precision | Recall | FPS
---|---|---|---|---
Att. Window= 3, shift= 2 | 0.914 | 0.875 | 0.83 | 88
Att. Window= 5, shift= 0 | 0.92 | 0.86 | 0.84 | 87.7
TABLE V:  Ablation study of depth of 3D attention on NPS dataset [13].
Comparison on val data.

####  IV-E4 Effect of Temporal Consistent Augmentation

We train two models on NPS [13] at the resolution of 640 $\times$ 640 &
$\tau=5$. The results (Table VI) demonstrate that the proposed Temporal
Consistency Augmentation module has better results. It indicates that applied
augmentations should be temporally consistent in order to learn robust spatio-
temporal features of the drone motion.

Temporal Augmentations | AP | Precision | Recall
---|---|---|---
inconsistent | 0.90 | 0.84 | 0.81
consistent | 0.92 | 0.86 | 0.84
TABLE VI:  Ablation study of temporal consistency in augmentations on NPS
[13]. Comparison on val data.

##  V Tackling real-world drone challenges

###  V-A Deployment for edge-computing: NVIDIA Jetson Xavier

To show the deployment capability of our model on edge-computing devices, we
use NVIDIA Jetson Xavier NX board [50]. It has 7025Mb of GPU memory & 6 CPU
cores. Our 640 resolution model obtains the real-time fps of 33 without any
complex TensorRT optimizations & keeping the board temperature well below
50∘c.

###  V-B False Positive Per Image (FPPI)

In real autonomous flight applications, low false positives are required [25]
as there is a significant cost attached to flight path maneuvers. FPPI can be
calculated by dividing the number of false positives encountered by the number
of frames processed. On 171 testing flights (194,193 frames) of the AOT
dataset, we obtained the FPPI as low as 0.000437 vs 0.018 [19], 0.02474 [18],
0.0105 [49].

###  V-C Encounter Detection Rate

To accurately predict the collision path, airborne objects need to be
continuously detected and tracked for 3 seconds time within the distance range
of 300m [25]. Even though we do not track objects in our method, our model can
successfully predict objects within a distance of 700m range. We have
continuous detections for 3 seconds in 82/175 = 46% closed encounter flights.

Figure 4: Tradeoff for performance<EMAIL_ADDRESS>vs Throughput (FPS): Our method
TransVisDrone consistently outperforms prior works under different input
resolutions. Other methods are evaluated on 1280 resolution. No off-the-shelf
optimizations.

##  VI Conclusion

We have developed an end-to-end trainable and computationally efficient
framework for drone-to-drone detection from videos. Spatio-temporal
transformer improves drone detection by learning the motion dependencies.
Apart from its state-of-the-art performance on three real-world datasets, we
have shown that our method significantly improves in throughput as well. We
have demonstrated that it is suitable to be deployed on edge-computing devices
like NVIDIA Jetson Xavier NX and useful to detect drone encounters
(collisions). We will make our code repository publicly available. Since our
model is capable of learning spatio-temporal features, in the future, it can
be extended to other problems like drone-to-drone tracking or estimating the
distance of the airborne objects from a monocular camera.

## References

  * [1] A. King, “Technology: The future of agriculture,” _Nature_ , vol. 544, no. 7651, pp. S21–S23, 2017.
  * [2] W. Sultani and M. Shah, “Human action recognition in drone videos using a few aerial training examples,” _Computer Vision and Image Understanding_ , vol. 206, p. 103186, 2021.
  * [3] I. R. Dave, M. N. Rizve, C. Chen, and M. Shah, “Timebalance: Temporally-invariant and temporally-distinctive video representations for semi-supervised action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * [4] M. N. Rizve, U. Demir, P. Tirupattur, A. J. Rana, K. Duarte, I. R. Dave, Y. S. Rawat, and M. Shah, “Gabriella: An online system for real-time activity detection in untrimmed security videos,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 4237–4244.
  * [5] I. Dave, Z. Scheffer, A. Kumar, S. Shiraz, Y. S. Rawat, and M. Shah, “Gabriellav2: Towards better generalization in surveillance videos for action detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2022, pp. 122–132.
  * [6] I. R. Dave, C. Chen, and M. Shah, “Spact: Self-supervised privacy preservation for action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 20 164–20 173.
  * [7] T. Li, J. Liu, W. Zhang, Y. Ni, W. Wang, and Z. Li, “Uav-human: A large benchmark for human behavior understanding with unmanned aerial vehicles,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 16 266–16 275.
  * [8] A. Kumar, J. Kini, A. Mian, and M. Shah, “Self supervised learning for multiple object tracking in 3d point clouds,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 3754–3761.
  * [9] Y. Cao, Z. He, L. Wang, W. Wang, Y. Yuan, D. Zhang, J. Zhang, P. Zhu, L. Van Gool, J. Han, _et al._ , “Visdrone-det2021: The vision meets drone object detection challenge results,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2847–2854.
  * [10] A. Rozantsev, V. Lepetit, and P. Fua, “Detecting flying objects using a single moving camera,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 5, pp. 879–892, 2016.
  * [11] R. H. Jacobsen and A. Marandi, “Security threats analysis of the unmanned aerial vehicle system,” in _MILCOM 2021-2021 IEEE Military Communications Conference (MILCOM)_. IEEE, 2021, pp. 316–322.
  * [12] V. Hassija, V. Chamola, A. Agrawal, A. Goyal, N. C. Luong, D. Niyato, F. R. Yu, and M. Guizani, “Fast, reliable, and secure drone communication: A comprehensive survey,” _IEEE Communications Surveys & Tutorials_, vol. 23, no. 4, pp. 2802–2832, 2021.
  * [13] J. Li, D. H. Ye, T. Chung, M. Kolsch, J. Wachs, and C. Bouman, “Multi-target detection and tracking from a single camera in unmanned aerial vehicles (uavs),” in _2016 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2016, pp. 4992–4997.
  * [14] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [15] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [16] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [17] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [18] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” _arXiv preprint arXiv:2010.04159_ , 2020.
  * [19] M. W. Ashraf, W. Sultani, and M. Shah, “Dogfight: Detecting drones from drones videos,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 7067–7076.
  * [20] C.-Y. Wang, H.-Y. M. Liao, Y.-H. Wu, P.-Y. Chen, J.-W. Hsieh, and I.-H. Yeh, “Cspnet: A new backbone that can enhance learning capability of cnn,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2020, pp. 390–391.
  * [21] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu, “Video swin transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3202–3211.
  * [22] L. Dressel and M. J. Kochenderfer, “Hunting drones with other drones: Tracking a moving radio target,” in _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 1905–1912.
  * [23] S. Dogru and L. Marques, “Drone detection using sparse lidar measurements,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 3062–3069, 2022\.
  * [24] A. Rozantsev, V. Lepetit, and P. Fua, “Detecting flying objects using a single moving camera,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 5, pp. 879–892, 2016.
  * [25] (2021) The airborne object tracking challenge. [Online]. Available: https://www.aicrowd.com/challenges/airborne-object-tracking-challenge
  * [26] M. Nava, A. Paolillo, J. Guzzi, L. M. Gambardella, and A. Giusti, “Learning visual localization of a quadrotor using its noise as self-supervision,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 2218–2225, 2022\.
  * [27] S. Li, C. De Wagter, and G. C. De Croon, “Self-supervised monocular multi-robot relative localization with efficient deep neural networks,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 9689–9695.
  * [28] K. Yang and Q. Quan, “An autonomous intercept drone with image-based visual servo,” 05 2020, pp. 2230–2236.
  * [29] F. Chen, Y. Lu, Y. Li, and X. Xie, “Real-time active detection of targets and path planning using uavs,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 391–397.
  * [30] Z. Cao, C. Fu, J. Ye, B. Li, and Y. Li, “Siamapn++: Siamese attentional aggregation network for real-time uav tracking,” 06 2021.
  * [31] H. Yu, F. Zhang, P. Huang, C. Wang, and L. Yuanhao, “Autonomous obstacle avoidance for uav based on fusion of radar and monocular camera,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 5954–5961.
  * [32] D. Neimark, O. Bar, M. Zohar, and D. Asselmann, “Video transformer network,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_ , October 2021, pp. 3163–3172.
  * [33] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.
  * [34] I. Dave, R. Gupta, M. N. Rizve, and M. Shah, “Tclr: Temporal contrastive learning for video representation,” _Computer Vision and Image Understanding_ , p. 103406, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1077314222000376
  * [35] K. Ranasinghe, M. Naseer, S. Khan, F. S. Khan, and M. S. Ryoo, “Self-supervised video transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2874–2884.
  * [36] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” _arXiv preprint arXiv:2004.10934_ , 2020.
  * [37] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 37, no. 9, pp. 1904–1916, 2015\.
  * [38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017\. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [39] X. Zhu, S. Lyu, X. Wang, and Q. Zhao, “Tph-yolov5: Improved yolov5 based on transformer prediction head for object detection on drone-captured scenarios,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_ , October 2021, pp. 2778–2788.
  * [40] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [41] T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie, and M. Li, “Bag of tricks for image classification with convolutional neural networks,” 2018. [Online]. Available: https://arxiv.org/abs/1812.01187
  * [42] G. Jocher, A. Stoken, J. Borovec, NanoCode012, ChristopherSTAN, L. Changyu, Laughing, tkianai, A. Hogan, lorenzomammana, yxNONG, AlexWang1900, L. Diaconu, Marc, wanghaoyang0106, ml5ah, Doug, F. Ingham, Frederik, Guilhen, Hatovix, J. Poznanski, J. Fang, L. Yu, changyu98, M. Wang, N. Gupta, O. Akhtar, PetrDvoracek, and P. Rai. (2020, Oct.) ultralytics/yolov5: v3.1 - Bug Fixes and Performance Improvements. [Online]. Available: https://doi.org/10.5281/zenodo.4154370
  * [43] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [44] (2021) The airborne object tracking challenge. [Online]. Available: https://zontakm9.github.io/aot-iccvw21/
  * [45] X. Yang, J. Yang, J. Yan, Y. Zhang, T. Zhang, Z. Guo, X. Sun, and K. Fu, “Scrdet: Towards more robust detection for small, cluttered and rotated objects,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 8232–8241.
  * [46] Z. Tian, C. Shen, H. Chen, and T. He, “Fcos: Fully convolutional one-stage object detection,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 9627–9636.
  * [47] Y. Chen, Y. Cao, H. Hu, and L. Wang, “Memory enhanced global-local aggregation for video object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 337–10 346.
  * [48] H. Wu, Y. Chen, N. Wang, and Z. Zhang, “Sequence level semantics aggregation for video object detection,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 9217–9225.
  * [49] Y. Wang, Z. Xu, X. Wang, C. Shen, B. Cheng, H. Shen, and H. Xia, “End-to-end video instance segmentation with transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8741–8750.
  * [50] (2021) Nvidia jetson xavier nx. [Online]. Available: https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/