#  Bat: Learning to Reason about Spatial Sounds with Large Language Models

Zhisheng Zheng  Puyuan Peng  Ziyang Ma  Xie Chen  Eunsol Choi  David Harwath

###### Abstract

Spatial sound reasoning is a fundamental human skill, enabling us to navigate
and interpret our surroundings based on sound. In this paper we present Bat,
which combines the spatial sound perception ability of a binaural acoustic
scene analysis model with the natural language reasoning capabilities of a
large language model (LLM) to replicate this innate ability. To address the
lack of existing datasets of in-the-wild spatial sounds, we synthesized a
binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we developed
SpatialSoundQA, a spatial sound-based question-answering dataset, offering a
range of QA tasks that train Bat in various aspects of spatial sound
perception and reasoning. The acoustic front end encoder of Bat is a novel
spatial audio encoder named Spatial Audio Spectrogram Transformer, or Spatial-
AST, which by itself achieves strong performance across sound event detection,
spatial localization, and distance estimation. By integrating Spatial-AST with
LLaMA-2 7B model, Bat transcends standard Sound Event Localization and
Detection (SELD) tasks, enabling the model to reason about the relationships
between the sounds in its environment. Our experiments demonstrate Bat’s
superior performance on both spatial sound perception and reasoning,
showcasing the immense potential of LLMs in navigating and interpreting
complex spatial audio environments. 111Our dataset, code, and model weights
will be released upon publication.

Machine Learning, ICML

##  1 Introduction

The evolution of Large Language Models (LLMs) has enabled their application
beyond textual data, giving rise to multimodal language models capable of
performing image understanding tasks such as visual question answering and
image captioning (Li et al., 2023; Liu et al., 2023; OpenAI, 2023; Peng et
al., 2024). Other work has integrated sound perception ability into LLMs,
enabling them to perform tasks such as audio question answering, speech
recognition, speech translation, and speech synthesis (Deshmukh et al., 2023;
Wang et al., 2023a; Chu et al., 2023; Borsos et al., 2023; Gong et al., 2024).
However, despite these LLM-based models’ ability to handle numerous visual and
audio tasks with impressive performance, so far none of them can handle in-
the-wild spatial audio input. Humans are equipped with binaural hearing
ability, allowing us to not only identify the type of sound we are hearing,
but also infer how far away the sound is, what direction it is coming from,
and whether multiple sound sources at different locations are present at once.
Consider scenarios such as locating someone calling from upstairs, hearing
your phone ringing from behind a sofa, or sensing footsteps approaching from
behind; these situations all demand spatial audio perception and reasoning, a
capability not yet present in the aforementioned models.

To bridge this gap, we introduce Bat, the first spatial audio-based LLM
designed to reason about sounds in a 3-D environment. Recognizing the current
lack of large-scale datasets for in-the-wild spatial audio and spatial audio-
based question answering, we synthesized a large-scale binaural audio dataset
using Audioset (Gemmeke et al., 2017) clips as sound sources and Soundspaces
2.0 (Chen et al., 2022) for simulating diverse, 3-D, reverberant acoustic
environments. In conjunction with this dataset, we developed SpatialSoundQA, a
diverse collection of question answering tasks designed to train and evaluate
spatial sound understanding across varying levels of complexity.

The successful training and evaluation of Bat hinge on the ability to encode
rich spatial audio information accurately. However, existing encoders often
specialize in either monaural event detection (Baade et al., 2022; Huang et
al., 2022; Chen et al., 2023, 2024b) or are limited to first-order ambisonics
(Shimada et al., 2021; Wang et al., 2023b; Kim et al., 2023), focusing on a
limited range of sound events. Other approaches are solely dedicated to
direction-of-arrival (DoA) estimation in specific multi-channel formats (Yang
et al., 2022a; Wang et al., 2023c), lacking the breadth needed for
comprehensive spatial audio understanding tasks. To overcome these
limitations, we developed Spatial-AST, a novel multi-task spatial encoder that
is not only capable of sound event detection and spatial localization but also
excels in distance perception, thereby providing a comprehensive solution for
spatial audio tasks.

In our experiments, we found that for Spatial-AST is both effective and
parameter-efficient. Using multi-step training curriculum, the Spatial-AST
encoder by itself can achieve strong performance across multiple tasks: a mean
Average Precision (mAP) of 50.03% for audio event classification, a Mean
Angular Error (MAE) of 17.94° for direction of arrival estimation, and a
Distance Error Rate (DER) within 0.5 meters of the actual location at 32.54%
for distance estimation. By fusing the Spatial-AST encoder to the LLaMA-2
(Touvron et al., 2023b) and adopting an appropriate training curriculum, our
Bat not only excels in tasks involving perception but also demonstrates
spatial reasoning abilities in scenarios with mixed sound sources (e.g. “Is
the stereo system to the left of the barking dog?”), achieving a 76.89%
accuracy in answering spatial sound reasoning questions.

Our key contributions are summarized as follows:

  * •

We present the first spatial audio-based question answering dataset
SpatialSoundQA, offering a range of 3-D audio understanding tasks from
perception to reasoning, which provides a platform for training models to
perform spatial sound understanding tasks.

  * •

We propose Spatial-AST, a binaural spatial audio encoder architecture that
jointly performs sound event detection, spatial localization, and distance
estimation that achieves strong performance across all three tasks.

  * •

We introduce Bat, which integrates Spatial-AST with the LLaMA-2 LLM, resulting
in a model capable of answering complex reasoning questions about multiple
sound sources situated within a 3-D environment.

Table 1: SpatialSoundQA Overview: The first four types focus on perception, while the last emphasizes reasoning. DP: Distance Prediction; DoA: Direction-of-Arrival. Numbers (e.g., 139K, 15.9%) indicate the QA sample count and their percentages in the dataset. Type | #Sources | Example
---|---|---
A: Detection (139K, 15.9%) | 1 |  Q: Identify the sound events in the audio clip. / A: baby laughter; laughter; speech
Q: What are the distinct sounds present in this audio clip? / A: heart sounds,
heartbeat
B: DP & DoA (139K, 15.9%) | 1 |  Q: How would you describe the location of this audio clip? / A: right, front, below; 2.5m
Q: At what distance and in which direction, is the music’s sound originating?
/ A: left, behind, below; 5m
C: Detection (118K, 13.5%) | 2 |  Q: Identify the sound events in the audio clip coming from the right, front, below, approximately
3 meters away. / A: slosh; speech
Q: What sound events can you detect in the audio recording emanating from the
left, behind, above,
roughly 0.5 meters away? / A: music; musical instrument; steelpan
D: DP & DoA (118K, 13.5%) | 2 |  Q: In which direction and how far away is the source of the heart sounds, heartbeat’s sound?
A: left, behind, below; 1m
Q: Where is the sound of the music coming from? / A: left, behind, below; 3m
E: Reasoning (358K, 41.2%) | 2 |  Q: When measuring the direct line distance, is the sound produced by wheeze closer to you than
the sound from bird flight, flapping wings? / A: No
Q: Is the source of both explosion and speech’s sounds from your left side? /
A: Yes
Q: Is the sound from the music on the front and the sound from the rattle on
the behind? / A: No
Q: Does the sound of electric shaver, electric razor appear on the behind of
the sound of waterfall? / A: Yes
Q: Can you estimate the distance from the sound of the speech to the sound of
the dog? / A: 1.5m
Q: What is the sound on the above side of the sound of the vibration? / A:
croak; frog
Q: Could you determine whether the singing’s sound is to the left or right of
the steam’s sound? / A: left

##  2 Related Work

###  2.1 Spatial Audio

Spatial audio is a family of techniques used to create the illusion of sound
sources existing within a 3-D space around the listener. Encapsulating
traditional stereo and surround sound systems to more recent methods such as
ambisonic audio, spatial audio is used in applications ranging from virtual
reality (Mystakidis, 2022) to advanced theater systems. In the realm of
artificial intelligence and machine learning, spatial audio presents unique
challenges for intelligent agents situated in 3-D physical spaces in
accurately localizing and interpreting sound sources (Evers et al., 2020;
Guizzo et al., 2022). To address these challenges, researchers have developed
acoustic simulation techniques (Scheibler et al., 2018; Chen et al., 2022) and
algorithms that leverage spatial audio (Yang et al., 2022b; Wang et al.,
2023c). Existing spatial audio datasets, such as YouTube-360 (Morgado et al.,
2020), YouTube-ASMR (Yang et al., 2020), Pano-AVQA (Yun et al., 2021), and
STARSS23 (Shimada et al., 2023), offer varying levels of spatial audio
information. However, they often suffer from inconsistent quality, lack
crucial ground truth labels like sound source distance and direction, and are
limited in scope, hindering the development of spatial audio-based machine
learning models.

###  2.2 Sound Event Localization and Detection

The Sound Event Localization and Detection (SELD) task (Adavanne et al., 2018)
merges sound source localization with sound event detection (SED), a task
which the DCASE community222https://dcase.community/community_info later
incorporated into their challenges as Task 3. Subsequent works (Wang et al.,
2020; Zhang et al., 2021; Wang et al., 2023b) adapted architectures like GRU
(Cho et al., 2014), TDNNF (Povey et al., 2018), and Conformer (Gulati et al.,
2020) for SELD tasks, achieving strong performance. However, these models have
primarily focused on shallow spatial audio perception, and the potential of
leveraging large language models to enable spatial reasoning remains
unexplored in prior work, which is the main focus of our work.

###  2.3 Multimodal Large Language Models

Recent models (OpenAI, 2023; Li et al., 2023; Liu et al., 2023; Peng et al.,
2024) have equipped LLMs with the ability to reason about and create images
and videos. For the audio modality, AudioGPT (Huang et al., 2023) integrates
ChatGPT as a versatile interface for a wide array of audio and speech
applications. Pengi (Deshmukh et al., 2023) proposes the use of transfer
learning to frame all audio tasks as text-generation challenges, utilizing
both audio and text inputs to produce results without additional fine-tuning.
LTU (Gong et al., 2024) proposes a monaural audio QA dataset, and combines an
Audio Spectrogram Transformer (AST) (Gong et al., 2021) feature extractor with
a LoRA-adapted (Hu et al., 2022) LLaMA (Touvron et al., 2023a) LLM to train
the model to reason and answer questions about the sounds in a clip.
Crucially, LTU differs from our work in that LTU is trained only on monaural
sound data, whereas we consider multi-channel spatial sound rendered by a
reverberant, 3-D environment. SALMONN (Tang et al., 2024) uses OpenAI’s
Whisper model (Radford et al., 2023) and BEATs (Chen et al., 2023) as dual
auditory encoders to extract speech and audio representations, and then
concatenates them to LLMs to enable the generation of responses. Qwen-audio
(Chu et al., 2023) focuses solely on training the Whisper-based audio encoder
using large-scale datasets, achieving proficiency in universal audio
understanding across a diverse range of tasks and audio types. However,
despite their impressive performance in the audio domain, none of these models
have the capability to perceive and reasoning about spatial audio that is
situated in diverse, reverberant, and complex 3-D environments.

##  3 SpatialSoundQA

The training of Bat requires a spatial audio-based question answering dataset.
Current datasets like SpatialVLM (Chen et al., 2024a) and Pano-AVQA (Yun et
al., 2021) are focused on vision and lack spatial audio information (or audio
altogether). To fill this void, we introduce SpatialSoundQA, the first spatial
audio-based question answering dataset, which is designed to emphasize the
complexities of the spatial audio, free from visual influences, and caters to
the unique demands of spatial audio perception and reasoning.

###  3.1 Spatial Audio Generation

Collecting and annotating spatial audio in the real world is a time consuming
task that is made more complex by environmental acoustic variability and
limitations in recording equipment. In order to efficiently create a dataset
with a large amount of variability across environments and types of sound
sources that is also rich in ground-truth metadata about these sources, we
have adopted a simulation-based approach for generating spatial audio data for
the dataset.

Spatial Audio Simulator. We use the state-of-the-art audio simulator,
SoundSpaces 2.0 (Chen et al., 2022). This platform performs on-the-fly
geometry-based sound rendering, enabling realistic acoustic reverberation with
arbitrary source-receiver locations. Many simulation parameters can be easily
modified, such as the material properties of a room’s walls and the objects
within the room, as well as the geometry of the receiver’s microphone array.
This enables us to curate a diverse and highly realistic dataset that also
offers easy access to the ground-truth sound generation parameters, such as
the spatial location and orientation of each source within the environment.
Specifically, we leverage Matterport3D (Chang et al., 2017) for our
environmental meshes, which includes highly detailed mesh renderings of 90
complete buildings, each featuring an average of 24.5 rooms across 2.61 floors
with an average floorspace of 517.34 $m^{2}$. For a given arbitrary source
location $s$, a monaural sound source $A^{s}$, receiver location $r$, and the
receiver’s heading direction $\theta$ in a specific mesh environment, the
audio signal $A^{r}$ received by a microphone is given by the convolution of
the room impulse response with the sound source, as shown in the following
equation:

$A^{r}_{m}(t)=R_{m}(t,s,r,\theta)*A^{s}(t)$ (1)

where $t\in[1,T]$ represents the time sample index, $m\in[1,2,...,n]$
represent the microphone index and $R_{m}(t,s,r,\theta)$ is the impulse
response between the source and receiver.

To minimize perceptual dissonance in auditory localization when visual cues
are absent, we ensure that both the sound source and the receiver are located
within the same room. We position the receiver at coordinates that allow for
an upright stance. As for the sound sources, they are placed at random
locations throughout the room. In total, 21,131 reverberations are generated,
averaging approximately 9.58 reverberations per room.

Sound Sources. Previous spatial audio datasets (Yang et al., 2020; Shimada et
al., 2023) often encompass a limited range of audio events, typically
restricted to categories like music, speech, and domestic sounds. To broaden
our scope and better represent real-world acoustic scenarios, we sample from
AudioSet (Gemmeke et al., 2017) to specify our monaural sound source $A^{s}$.
AudioSet consists of roughly 2 million 10-second in-the-wild YouTube clips
used for audio classification, with each clip weakly annotated with 527 types
of audio events, often featuring multiple overlapping events within a single
clip. However, certain categories among these 527 types are discernible only
through visual cues (e.g., “Single-lens reflex camera” vs. “Camera”). To
enhance the dataset’s reliability, we exclude labels that require visual
information by manual inspection. Additionally, to mitigate the impact of
poor-quality training samples, we remove labels with less than 50%
quality.333For example, the estimated quality of the “clatter” label is 20%.
Source: https://research.google.com/audioset/dataset/clatter.html. Ultimately,
given that the input audio undergoes convolution with reverberation, we
further exclude most noise-related labels, such as “Noise”, “Echo” and
“Outside, urban or manmade”. This curation results in a set of 355 audio event
labels that are appropriate for our setting, identifiable solely through audio
cues. After filtering by these categories, we are left with 1,861,750 clips in
the AudioSet-2M split, while our filtered AudioSet-20K split contains 18,373
clips, and our evaluation set consists of 17,148 clips. Lastly, to counteract
any volume variability in the source audio, we applied loudness normalization
across all clips.

###  3.2 Question-Answer Pair Generation

We curated SpatialSoundQA to include diverse question-answer pairs, all
centered around the challenges of spatial audio perception and reasoning.
Similar to LTU (Gong et al., 2024), each sample in the dataset is structured
as a tuple of (audio, question, answer), where the audio and question serve as
inputs to the model, and the answer acts as the target label. Table 1
illustrates the array of tasks in SpatialSoundQA, ranging from basic
perception to complex spatial reasoning. To enhance variety, the questions in
these tasks are paraphrased using GPT-4, ensuring a broad spectrum of queries.
The answers, on the other hand, are generated uniformly through a systematic
rule-based approach, maintaining consistency across the dataset. We enumerate
each question type in the following paragraphs. Sound Event Detection (Type A
& C). For this task, the question is “What sound events can you detect in the
audio recording?” and its paraphrases. The answer may consist of a sorted list
of sound classes present in the audio clip, arranged in alphabetical order.
Question type A utilizes audio with 1 sound source, while C uses 2.

Direction and Distance Estimation (Type B & D). This task aims to identify the
direction and distance of a sound source. For establishing ground truth, the
three-dimensional space is segmented into eight regions. Each region is
defined by a tuple representing the directional axis (left/right,
front/behind, above/below) with respect to the receiver. Additionally,
distance is quantified in increments of 0.5 meters, spanning a range from 0 to
10 meters. A typical question might be, “Where is the sound of the music
coming from?” accompanied by GPT-generated paraphrased versions. The format
for answers is constrained to expressions like “left, front, above; 2.5m”.
Question type B utilizes audio with 1 sound source, while D uses 2.

Spatial Reasoning (Type E). This task differs from the previous perception-
focused tasks by introducing scenarios where two sound events occur
concurrently, originating from distinct distances and directions. When
generating these scenarios, we ensure that each mix contains two sound sources
with entirely different sound events and orientations. The challenge here is
to discern the spatial differences between these overlapping sounds. For
example, a question might be, “What is the sound on the left side of the sound
of the dog barking?” This task demands both perception and complex reasoning.
The model must implicitly separate the sound sources based on their unique
classes, spatially localize each source, and then analyze the relationship
between the sources in the context of the question. However, the encoder
Spatial-AST was not been exposed to such mixed data types during its pre-
training stage, implying a lack of audio separation ability. This task is thus
designed to test the model’s in-context learning capabilities and effectively
utilize the latent representations from the encoder to address this complex
multi-source audio reasoning challenge. To make evaluation straightforward, we
have chosen a binary (Yes/No) response format for these questions. By adopting
binary answers, we can more effectively gauge the model’s reasoning
performance and maintain consistent assessment standards across different
question types.

Figure 1: Illustration of the Bat model structure. The rightmost part shows
the binaural audio generation process.

##  4 Method

Figure 1 illustrates our overall architecture for the Bat model. We will first
introduce Spatial-AST, our novel spatial audio encoder that acts as a front
end for Bat. We then discuss how it is integrated with the LLaMA-2 LLM in
order to build spatial reasoning ability on top of Spatial-AST’s auditory
perception ability.

###  4.1 Spatial Audio Encoder: Spatial-AST

We propose a novel architecture Spatial-AST to capture spatial audio
information. The model is shown on the left-hand side of Figure 1.

#### Front-End.

Our encoder integrates both Mel-Spectrogram and Interaural Phase Difference
(IPD). As detailed in the Spatial-AST section of Figure 1, we initially
transform the time-domain signal $x(n)$ into the frequency domain $X(t,f)$
using the short-time Fourier transform (STFT), as represented by Equation 2:

$X_{i}(t,f)=\sum_{n=0}^{N-1}x_{i}(n)w(n-t)e^{-j\frac{2\pi fn}{N}},\
i\in\\{1,2\\}$ (2)

Here $w(n)$ denotes an N-point window function, and $i$ indicates the left or
right channel in the binaural recording.

Subsequently, we compute both the Mel-Spectrogram and the Interaural Phase
Difference (IPD) based on these frequency domain signals $X_{i}(t,f)$. The
Mel-Spectrogram for each channel is calculated as per Equation 3, and the IPD
is derived from the phase spectrograms of the left and right channels, as
indicated in Equation 4:

$S_{i}(t,m)=log\left(|X_{i}(t,f)|^{2}\times\text{melW}\right)$ (3)

where melW is a $M$-bin Mel filter bank.

$IPD(t,f)=\angle\frac{X_{2}(t,f)}{X_{1}(t,f)}$ (4)

To address the numerical instabilities due to phase wraparound in IPD, we
apply cosine and sine transformations, which are then weighted by melW to
align with the dimension of the Mel-spectrogram. The final front-end output,
$\mathcal{Z}$, is a concatenation of these processed components. It includes
both the Mel-Spectrograms of the left and right channels, as well as the
cosine and sine transformations of the IPD, all combined as outlined in
Equation 5:

$\begin{split}\mathcal{Z}=[&S_{1};S_{2};\cos(\text{IPD})\times\text{melW};\\\
&\sin(\text{IPD})\times\text{melW}],\quad\mathcal{Z}\in\mathbb{R}^{4\times
T\times M}\end{split}$ (5)

Backbone. The initial input $\mathcal{Z}$ is first passed to a 3x3 2D
convolution followed by a batch normalization (Ioffe & Szegedy, 2015) layer
and a GELU (Hendrycks & Gimpel, 2016) layer to fuse the inter-channel
information. Subsequently, we employ a Patch-Embed CNN with a kernel size of
$(16\times 16)$ and corresponding strides in both time and frequency
dimensions, ensuring non-overlapping segmentation of the features into
distinct patch tokens. To extract spatial cues, we concatenate three [CLS]
tokens at the beginning of the audio tokens, each specifically designated for
extracting information about the audio’s category, distance, and direction
respectively. All tokens are then fed into 12-layer Transformer encoder
blocks. Finally, we process the three [CLS] tokens using three separate linear
layers to get predictions for their corresponding tasks.

Pre-Training Objective. The spatial encoder is pre-trained to handle three
tasks, namely sound event detection, distance prediction, and directional
prediction, with the latter being further divided into azimuth and elevation
angle predictions. We employ cross-entropy loss for all three tasks. To
discretize prediction targets for distance and directional prediction, we
categorize distance into intervals of 0.5 meters and angle into intervals of 1
degree for both azimuth and elevation.

The final pre-training objective of the spatial encoder is:

$\mathcal{L}=\lambda_{1}\mathcal{L}_{cls}+\lambda_{2}\mathcal{L}_{dis}+\lambda_{3}\mathcal{L}_{doa}$
(6)

where $\mathcal{L}_{cls}$, $\mathcal{L}_{dis}$, $\mathcal{L}_{doa}$ represent
the losses for detection, distance, and direction respectively, while
$\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ are hyperparameters.

###  4.2  Bat: A Model for Reasoning about Spatial Sounds

On its own, Spatial-AST is capable of predicting the type, direction, and
distance for an input spatial sound. To extend this ability to encompass
spatial reasoning about multiple sound sources present within an environment,
we fuse Spatial-AST to the LLaMA-2 7B LLM (Touvron et al., 2023b). Input
spatial audio is first processed by the Spatial-AST encoder, and then a
projection module is used to map its set of output tokens into LLaMA-2’s input
text embedding space. For efficient fine-tuning, we use the LLaMA-adapter v2
(Gao et al., 2023).

The fine-tuning objective of Bat is to predict the answer text conditioned on
its paired question text and the corresponding audio input. The model seeks to
maximize the probability of predicting the next answer token, applying cross-
entropy loss for each token position $1\leq t\leq T$ in the sequence, which
can be mathematically represented as:

$\mathcal{P}_{\theta}\left(y_{t}\mid
y_{t-1},\cdots,y_{1};a_{l},a_{l-1},\cdots,a_{1}\right)$ (7)

where $a_{l}$ is the l-th token of audio embedding, and $y_{t}$ denotes the
t-th token of text output.

Table 2: The perception-to-reasoning curriculum. Stage | Qestion Type | # Tr. Samples | Percentages
---|---|---|---
1 | A, B | 278K | 31.8%
2 | A, B, C, D | 514K | 58.8%
3 | A, B, C, D, E | 872K | 100%

For the training of Bat, we devised a perception-to-reasoning curriculum, as
detailed in Table 2. The process begins with a “warm-up” phase focusing on
single-source perception tasks (question types A and B), enabling the model to
adapt to audio modality inputs and infer basic properties of a single audio
source including its class, distance, and direction. In the second stage, we
introduce multi-source scenarios (question types C and D), where the model is
taught to implicitly perform sound source separation by answering questions
about specific sources within mixed audio. The final stage unlocks the full
capabilities of Bat, integrating reasoning tasks (question type E), such as
interpreting spatial relationships and distances between multiple sound
sources. The importance of this curriculum is evidenced in Table 10 in
Appendix H.

##  5 Experiments

###  5.1 Implementation Details

Input Audio Processing. The 10-second audio sources are first loudness
normalized by scaling them so that each clip has the same total sound energy.
After convolution with the room impulse response from SoundSpaces, we trim or
pad these clips to 10 seconds to align with AudioMAE (Huang et al., 2022). The
resulting waveforms are binaural with 2 channels at a 32kHz sampling rate. We
use a window size of 1024, a hop size of 320, and 128 mel-bins to compute the
Short-Time Fourier Transforms (STFTs) and mel-spectrograms. As a result, for a
10-second recording from AudioSet, the concatenated Mel-spectrogram and IPD
feature dimension is (4, 1024, 128).

Spatial Audio Encoder. We implement a patch masking ratio of 0.25 in both time
and frequency during training, and the unmasked tokens are then concatenated
with three learnable [CLS] tokens. We initialize the weights of the
transformer blocks using the official pretrained AudioMAE (Huang et al., 2022)
checkpoint. The encoder is trained on 8 RTX 3090 GPUs, with each epoch taking
approximately 10 minutes. Recognizing the greater challenge posed by sound
event detection compared to distance and directional prediction, we train our
encoder in two stages. In the first stage, we focus exclusively on detection
loss, setting the weights $\lambda_{2}$ and $\lambda_{3}$ to zero as per
Equation 6. In the second stage, we reintroduce the losses for distance and
direction, adjusting the loss weights to $\lambda_{1}=1250,\lambda_{2}=1$, and
$\lambda_{3}=2$ to broaden the capabilities of the model. This two-stage
training approach is crucial for achieving a balanced performance across
different tasks. The effects of varying these weights on model performance for
various tasks are detailed in Appendix E.

LLM. Diverging from the original two-stage learnable parameters design of
LLaMA-adapter v2, We concurrently train parameters including zero-initialized
attention, projection, norm, bias, and scale to simplify the training process.
The training is completed on 8 V100 GPUs. For generation, we employ a greedy
decoding and set the temperature to 0.1 and nucleus sampling (Holtzman et al.,
2020) top p to 0.75.

Table 3: This table presents a comparative analysis of various models using both monaural and binaural data, focusing on key performance metrics: mean Average Precision (mAP, $\uparrow$), Error Rate at 20° (ER20°, $\downarrow$), Mean Angular Error (MAE, $\downarrow$), and Distance Error Rate (DER, $\downarrow$). Results for AudioMAE models were derived through inference using their official checkpoints, without further training. Models that utilize anechoic audio data (no reverberation) are grayed-out for clarity. |  | Data | Performance Metrics
---|---|---|---
|  | Audio Type | Input Features | mAP (%) $\uparrow$ | ER20° (%) $\downarrow$ | MAE (°) $\downarrow$ | DER (%) $\downarrow$
AudioMAE (Huang et al., 2022) | Anechoic | Mel-spectrograms | 53.56 | - | - | -
AudioMAE (Huang et al., 2022) | Monaural | Mel-spectrograms | 47.18 | - | - | -
Spatial-AST (ours) | Joint-Training | 51.39 | 95.85 | 88.52 | 26.87
Stage-1 | 52.26 | - | - | -
Stage-2 | 50.69 | 95.35 | 89.42 | 28.25
SELDnet (Adavanne et al., 2018) | Binaural | Mel-spectrograms, IPD | 42.66 | 25.19 | 19.21 | 38.46
Spatial-AST (ours) |  | Mel-spectrograms | 49.94 | 25.50 | 18.57 | 34.85
Spatial-AST (ours) | Joint-Training | Mel-spectrograms, IPD | 50.57 | 24.81 | 18.16 | 35.29
Spatial-AST (ours) | Stage-1 | Binaural | Mel-spectrograms | 51.70 | - | - | -
Stage-2 | 50.86 | 30.94 | 21.66 | 28.60
Spatial-AST (ours) | Stage-1 | Mel-spectrograms, IPD | 52.26 | - | - | -
Stage-2 | 50.03 | 23.89 | 17.94 | 32.54

Baseline. For our encoder baseline comparisons, we select established models
such as AudioMAE (Huang et al., 2022) and SELDnet (Adavanne et al., 2018). To
ensure a fair comparison, SELDnet’s feature extraction network is augmented
with a 12-layer transformer block, aligning the model parameters of the
compared architectures to approximately 90M. Additionally, for Bat, we use a
version with monaural input for comparison, which has an architecture similar
to LTU (Gong et al., 2024), serving as a baseline.

Evaluation Metrics. In the evaluation of our spatial audio encoder, outlined
in Table 3, we use the mean Average Precision (mAP) for sound event detection;
Error Rate at 20° (ER20°), as referenced in (Mesaros et al., 2019), and Mean
Angular Error (MAE) for Direction of Arrival (DoA) accuracy, evaluating
whether predictions deviate more than 20° from the reference point and
quantifying the average angular deviation between predictions and ground
truth; and Distance Error Rate (DER) for measuring the accuracy of distance
predictions within 0.5 meters of the actual location. For the evaluation of
Bat, as shown in Table 4, we apply similar metrics, including mAP and DER. For
the DoA task, where the three-dimensional space is segmented into eight
distinct sections, we use accuracy (Acc) as the performance metric.
Additionally, for reasoning tasks in question type E, Binary Accuracy (BA)
serves as the evaluation metric.

###  5.2 Performance of Spatial-AST on SELD Tasks

Table 3 presents the experimental results for our proposed Spatial-AST audio
encoder. We compare Spatial-AST against current state-of-the-art models such
as AudioMAE (Huang et al., 2022) and SELDnet (Adavanne et al., 2018). We also
compare spatial cue extraction between monaural and binaural data.
Additionally, we explore the impact of two training methodologies, including
joint-training and two-stage approaches, and highlight the importance of IPD.

Table 4: Performance evaluation on SpatialSoundQA. The metrics include mAP for
detection accuracy, represented by two columns corresponding to question types
A and C, respectively; Accuracy (Acc) for assessing the accuracy of
identifying directions and Distance Error Rate (DER) for distance prediction,
with two columns each for question types B and D; and Binary Accuracy (BA) for
the accuracy of binary (Yes/No) responses in reasoning (question type E).
Input types include monaural audio (M), binaural audio (B), prompt (P), or
ground truth sound source parameters (GT). The “Random” and “Oracle” rows
serve as baseline and ideal performance benchmarks.

Model | Input |  Detection (mAP $\uparrow$)* |  Direction-of-Arrival (Acc $\uparrow$) |  Distance (DER $\downarrow$) | Reasoning (BA $\uparrow$)
---|---|---|---|---|---
Direction | Distance | Avg.
Random | P | 0.61 | 0.59 | 12.57 | 12.41 | 67.33 | 67.46 | 50.00 | 50.00 | 50.00
Monaural Bat (ours)1 | M + P | 24.15 | 6.42 | 14.31 | 11.93 | 34.17 | 56.26 | 57.69 | 51.36 | 54.53
One-stage Bat1 | B + P | 26.23 | 9.06 |  76.49 | 37.09 | 29.28 | 51.62 | 65.12 | 81.98 | 73.55
One-stage Bat1, 2 | 25.50 | 5.67 | 72.74 | 35.63 | 31.51 | 73.68 | 59.59 | 49.76 | 54.68
Bat | P | 0.69 | 0.65 | 12.88 | 13.14 | 51.65 | 71.73 | 58.28 | 50.67 | 54.48
B + P | 26.34 | 9.89 | 75.54 | 37.65 | 29.16 | 47.90 | 69.77 | 84.01 | 76.89
Oracle1 | GT + P | 94.16 | 94.26 | 99.20 | 99.81 | 0.00 | 0.00 | 98.21 | 100.00 | 99.10

  * *

Like LTU (Gong et al., 2024), we use a text encoder (text-embedding-3-small)
to encode Bat’s output and the labels from the evaluation dataset, then
compute the cosine similarity between these text embeddings for evaluation.
mAP potentially underestimates Bat’s performance.

  * 1

Only train stage 3. 2 We removed question types C and D from SpatialSoundQA.

Monaural vs. Binaural. To support monaural input, we modified the 3x3
convolution module, setting its input channel to 1, and only feed a monaural
mel spectrogram as input. Despite binaural data typically offering richer
spatial information, our encoder demonstrates robust performance with monaural
inputs, especially in sound event detection (SED) and distance prediction (DP)
tasks. The encoder achieves a mAP of 51.39 for monaural input, surpassing both
its binaural counterpart, which scores 49.94, and AudioMAE, which achieves
47.18 in mAP. Additionally, our encoder with monaural input even attains the
best Distance Error Rate (DER) compared to models using binaural input.
However, as monaural audio cannot capture spatial information, it shows low
performance in the direction-of-arrival (DoA) task, with an ER20° of 95.82 and
an MAE of 88.52. The performance of monaural data suggests it is sufficiently
effective for SED and DP tasks. The effectiveness of monaural input can be
attributed to the fact that binaural audio does not necessarily provide
additional benefits for sound category perception. The results from the
single-task setting detailed in Appendix D corroborate this conclusion.

Joint-training vs. Two-stage training. Joint-training emerges as the more
effective approach for monaural data, outperforming two-stage training in both
detection and distance prediction tasks, achieving a mAP of 51.39 and DER of
26.87. However, for binaural data, two-stage training consistently
demonstrates improvements in its second stage, regardless of whether IPD is
used. The enhancement is particularly significant with the usage of IPD,
achieving the lowest MAE of 23.89 and DER of 17.94, demonstrating the
importance of including multi-microphone phase information for spatial audio
perception. This also underscores the importance of employing a two-stage
training approach for enhanced performance in binaural settings.

Model Architecture. When compared to AudioMAE and SELDnet, our encoder
exhibits enhanced performance in reverberant environments. As AudioMAE is
tailored for monaural input, we focus our comparison on monaural scenarios. In
these contexts, our proposed model surpasses both joint-training and two-stage
training methods. This achievement is especially significant given that our
model has been finetuned for much fewer training hours. For simplicity, we
compare our model with SELDnet in joint-training, where it surpasses SELDnet
on all metrics.

###  5.3 Performance of Bat on SpatialSoundQA

Table 4 shows the performance of various configurations of our Bat model on
SpatialSoundQA, broken down across the different question types. Bat performs
strongly across all question types, outperforming the random baseline by a
very large margin. We observe that the one-stage Bat generally lags behind the
three-stage Bat in performance across various metrics. Intriguingly, we
observed a significant drop in the model’s reasoning ability when question
types C and D, related to sound source separation, were removed from the
training set. This is evident from the Binary Accuracy (BA) results, which
approximate random chance in the reasoning tasks. Analysis of the one-stage
Bat’s outputs reveals that the model struggles to differentiate between sound
sources and their respective positions when confronted with reasoning
questions. This finding highlights the importance of sound source separation
tasks in training Bat to effectively handle complex reasoning, underscoring
the necessity of a multi-stage curriculum that gradually introduces the model
to increasingly sophisticated spatial audio perception and reasoning tasks.

The “Monaural Bat” row shows the performance of Bat trained with monaural
audio. The architecture of monaural Bat is similar to that of LTU (Gong et
al., 2024), so we use it as a baseline for comparison. The results reveal that
monaural Bat faces significant challenges in reasoning tasks, achieving a
Binary Accuracy (BA) of only 54.53%. This suggests that monaural audio input
alone cannot provide sufficient spatial information for complex reasoning
challenges, highlighting the usefulness of SpatialSoundQA for learning spatial
audio reasoning capabilities. More ablations can be found in Appendix H.

##  6 Conclusion

In this work, we have presented Bat, the first LLM capable of processing
spatial audio input. To train and evaluate Bat, we introduced SpatialSoundQA,
the first extensive spatial audio-based question answering dataset. We also
proposed Spatial-AST, a novel spatial audio encoder capable of efficiently
handling sound event detection, spatial localization, and distance estimation.
Bat and SpatialSoundQA showcase the immense potential of LLMs for reasoning
about spatial sound. SpatialSoundQA also enables rich future work, such as
reasoning about the environment itself (materials, shape, layout), modeling
moving sounds to for tracking problems, or incorporating the visual modality
which SoundSpaces2.0 natively supports.

## Broader Impact

Our research on Bat, the first spatial audio-based large language model,
alongside the SpatialSoundQA dataset, holds significant potential for broad
impact across various fields:

Advancements in Spatial Audio: Our work addresses a key gap in spatial audio
perception and reasoning. By developing an LLM specifically for spatial audio,
we open up new possibilities for applications where spatial sound cues play a
crucial role, such as virtual reality, gaming, and audio engineering. This can
lead to more immersive and realistic experiences in these domains.

Enhancements in Multimodal Large Language Models: The integration of spatial
audio into LLMs represents a significant step towards truly multimodal AI
systems. Our model not only demonstrates the capability of LLMs in processing
complex spatial audio information but also paves the way for future research
in combining auditory and other sensory modalities, such as visual or tactile,
to create even more sophisticated AI models.

Empowering Embodied AI: The ability to interpret and reason about spatial
sounds can significantly enhance embodied AI systems, like robots or
autonomous vehicles. These systems can utilize spatial audio cues for better
navigation and interaction with their environment, leading to more effective
and safer applications in real-world scenarios.

## References

  * Adavanne et al. (2018) Adavanne, S., Politis, A., Nikunen, J., and Virtanen, T.  Sound event localization and detection of overlapping sources using convolutional recurrent neural networks.  _Proc. JSTSP_ , 2018.
  * Baade et al. (2022) Baade, A., Peng, P., and Harwath, D. F.  Mae-ast: Masked autoencoding audio spectrogram transformer.  _Proc. Interspeech_ , 2022.
  * Borsos et al. (2023) Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., et al.  AudioLM: A language modeling approach to audio generation.  _Proc. TASLP_ , 2023.
  * Chang et al. (2017) Chang, A., Dai, A., Funkhouser, T., Halber, M., Niessner, M., et al.  Matterport3D: Learning from RGB-D data in indoor environments.  _Proc. 3DV_ , 2017.
  * Chen et al. (2024a) Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., et al.  SpatialVLM: Endowing vision-language models with spatial reasoning capabilities.  _arXiv preprint arXiv:2401.12168_ , 2024a.
  * Chen et al. (2022) Chen, C., Schissler, C., Garg, S., Kobernik, P., Clegg, A., et al.  SoundSpaces 2.0: A simulation platform for visual-acoustic learning.  _Proc. NeurIPS_ , 2022.
  * Chen et al. (2023) Chen, S., Wu, Y., Wang, C., Liu, S., Tompkins, D., Chen, Z., and Wei, F.  BEATs: Audio pre-training with acoustic tokenizers.  _Proc. ICML_ , 2023.
  * Chen et al. (2024b) Chen, W., Liang, Y., Ma, Z., Zheng, Z., and Chen, X.  EAT: Self-supervised pre-training with efficient audio transformer.  _arXiv preprint arXiv:2401.03497_ , 2024b.
  * Cho et al. (2014) Cho, K., Van Merriënboer, B., Bahdanau, D., and Bengio, Y.  On the properties of neural machine translation: Encoder-decoder approaches.  _Proc. SSST_ , 2014.
  * Chu et al. (2023) Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., et al.  Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models.  _arXiv preprint arXiv:2311.07919_ , 2023.
  * Deshmukh et al. (2023) Deshmukh, S., Elizalde, B., Singh, R., and Wang, H.  Pengi: An audio language model for audio tasks.  _Proc. NeurIPS_ , 2023.
  * Evers et al. (2020) Evers, C., Löllmann, H. W., Mellmann, H., Schmidt, A., Barfuss, H., et al.  The LOCATA challenge: Acoustic source localization and tracking.  _Proc. TASLP_ , 2020.
  * Gao et al. (2023) Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., et al.  LLaMA-Adapter v2: Parameter-efficient visual instruction model.  _arXiv preprint arXiv:2304.15010_ , 2023.
  * Gemmeke et al. (2017) Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., et al.  AudioSet: An ontology and human-labeled dataset for audio events.  _Proc. ICASSP_ , 2017.
  * Gong et al. (2021) Gong, Y., Chung, Y.-A., and Glass, J.  AST: Audio spectrogram transformer.  _Proc. Interspeech_ , 2021.
  * Gong et al. (2024) Gong, Y., Luo, H., Liu, A. H., Karlinsky, L., and Glass, J.  Listen, think, and understand.  _Proc. ICLR_ , 2024.
  * Guizzo et al. (2022) Guizzo, E., Marinoni, C., Pennese, M., Ren, X., Zheng, X., et al.  L3DAS22 challenge: Learning 3d audio sources in a real office environment.  _Proc. ICASSP_ , 2022.
  * Gulati et al. (2020) Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., et al.  Conformer: Convolution-augmented transformer for speech recognition.  _Proc. Interspeech_ , 2020.
  * Hak et al. (2012) Hak, C. C., Wenmaekers, R. H., and Van Luxemburg, L.  Measuring room impulse responses: Impact of the decay range on derived room acoustic parameters.  _Proc. Acta Acustica_ , 2012.
  * Hendrycks & Gimpel (2016) Hendrycks, D. and Gimpel, K.  Gaussian error linear units (GELUs).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Holtzman et al. (2020) Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y.  The curious case of neural text degeneration.  _Proc. ICLR_ , 2020.
  * Hu et al. (2022) Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., et al.  LoRA: Low-rank adaptation of large language models.  _Proc. ICLR_ , 2022.
  * Huang et al. (2022) Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., et al.  Masked autoencoders that listen.  _Proc. NeurIPS_ , 2022.
  * Huang et al. (2023) Huang, R., Li, M., Yang, D., Shi, J., Chang, X., et al.  AudioGPT: Understanding and generating speech, music, sound, and talking head.  _arXiv preprint arXiv:2304.12995_ , 2023.
  * Ioffe & Szegedy (2015) Ioffe, S. and Szegedy, C.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  _Proc. ICML_ , 2015.
  * Kim et al. (2023) Kim, J. S., Park, H. J., Shin, W., and Han, S. W.  AD-YOLO: You look only once in training multiple sound event localization and detection.  _Proc. ICASSP_ , 2023.
  * Li et al. (2023) Li, J., Li, D., Savarese, S., and Hoi, S.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _Proc. ICML_ , 2023.
  * Liu et al. (2023) Liu, H., Li, C., Wu, Q., and Lee, Y. J.  Visual instruction tuning.  _Proc. NeurIPS_ , 2023.
  * Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F.  SGDR: Stochastic gradient descent with warm restarts.  _Proc. ICLR_ , 2017.
  * Loshchilov & Hutter (2019) Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  _Proc. ICLR_ , 2019.
  * Mesaros et al. (2019) Mesaros, A., Adavanne, S., Politis, A., Heittola, T., and Virtanen, T.  Joint measurement of localization and detection of sound events.  _Proc. WASPAA_ , 2019.
  * Morgado et al. (2020) Morgado, P., Li, Y., and Nvasconcelos, N.  Learning representations from audio-visual spatial alignment.  _Proc. NeurIPS_ , 2020.
  * Mystakidis (2022) Mystakidis, S.  Metaverse.  _Encyclopedia_ , 2022.
  * OpenAI (2023) OpenAI.  GPT-4V(ision) system card.  2023\.
  * Peng et al. (2024) Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., et al.  Grounding multimodal large language models to the world.  _Proc. ICLR_ , 2024.
  * Povey et al. (2018) Povey, D., Cheng, G., Wang, Y., Li, K., Xu, H., et al.  Semi-orthogonal low-rank matrix factorization for deep neural networks.  _Proc. Interspeech_ , 2018.
  * Radford et al. (2023) Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I.  Robust speech recognition via large-scale weak supervision.  _Proc. ICML_ , 2023.
  * Scheibler et al. (2018) Scheibler, R., Bezzam, E., and Dokmanic, I.  Pyroomacoustics: A python package for audio room simulation and array processing algorithms.  _Proc. ICASSP_ , 2018.
  * Shimada et al. (2021) Shimada, K., Koyama, Y., Takahashi, N., Takahashi, S., and Mitsufuji, Y.  ACCDOA: Activity-coupled cartesian direction of arrival representation for sound event localization and detection.  _Proc. ICASSP_ , 2021.
  * Shimada et al. (2023) Shimada, K., Politis, A., Sudarsanam, P., Krause, D., Uchida, K., et al.  STARSS23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events.  _arXiv preprint arXiv:2306.09126_ , 2023.
  * Tang et al. (2024) Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., et al.  SALMONN: Towards generic hearing abilities for large language models.  _Proc. ICLR_ , 2024.
  * Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., et al.  LLaMA: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., et al.  LLaMA 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Wang et al. (2023a) Wang, J., Du, Z., Chen, Q., Chu, Y., et al.  LauraGPT: Listen, attend, understand, and regenerate audio with GPT.  _arXiv preprint arXiv:2310.04673_ , 2023a.
  * Wang et al. (2020) Wang, Q., Wu, H., Jing, Z., Ma, F., Fang, Y., Wang, Y., Chen, T., Pan, J., Du, J., and Lee, C.-H.  The USTC-Iflytek system for sound event localization and detection of dcase2020 challenge.  _Proc. DCASE 2020_ , 2020.
  * Wang et al. (2023b) Wang, Q., Jiang, Y., Cheng, S., Hu, M., Nian, Z., et al.  The nerc-slip system for sound event localization and detection of dcase2023 challenge.  _Proc. DCASE 2023_ , 2023b.
  * Wang et al. (2023c) Wang, Y., Yang, B., and Li, X.  FN-SSL: Full-band and narrow-band fusion for sound source localization.  _Proc. Interspeech_ , 2023c.
  * Yang et al. (2022a) Yang, B., Ding, R., Ban, Y., Li, X., and Liu, H.  Enhancing direct-path relative transfer function using deep neural network for robust sound source localization.  _Proc. TIT_ , 2022a.
  * Yang et al. (2022b) Yang, B., Liu, H., and Li, X.  SRP-DNN: Learning direct-path phase difference for multiple moving sound source localization.  _Proc. ICASSP_ , 2022b.
  * Yang et al. (2020) Yang, K., Russell, B., and Salamon, J.  Telling left from right: Learning spatial correspondence of sight and sound.  _Proc. CVPR_ , 2020.
  * Yun et al. (2021) Yun, H., Yu, Y., Yang, W., Lee, K., and Kim, G.  Pano-AVQA: Grounded audio-visual question answering on 360deg videos.  _Proc. ICCV_ , 2021.
  * Zhang et al. (2021) Zhang, Y., Wang, S., Li, Z., Guo, K., Chen, S., and Pang, Y.  Data augmentation and class-based ensembled cnn-conformer networks for sound event localization and detection.  _Proc. DCASE 2021_ , 2021.

##  Appendix A Why Do We Use Large Language Models?

The use of Large Language Models (LLMs) in our spatial audio reasoning
framework is necessary due to several critical reasons. First, LLMs enable
complex questions to be posed using natural language, allowing for a more
flexible and intuitive query format, such as “Is the dog further away from you
than the stereo?” or “Is the dog to the left or to the right of the stereo?”
This capability eliminates the need for a cascade of separate processing steps
like source separation, classification, DoA, distance classifiers, and rule-
based templates, which would significantly limit the types of questions that
could be addressed. Additionally, LLMs facilitate the joint training of the
spatial audio encoder and reasoning module in an end-to-end manner, reducing
the risk of error propagation inherent in segmented processing approaches.
Moreover, LLMs offer the ability to generalize across different questions with
overlapping expressions, enhancing the model’s adaptability and versatility in
responding to a wide range of spatial audio-related queries. Last but not
least, this task also intends to further expand the application boundaries of
LLMs, demonstrating their versatility and effectiveness in domains beyond
traditional text and image processing.

##  Appendix B Hyperparameters

We present the specific training hyperparameter configurations for Spatial-AST
& Bat in Table 5.

Table 5: Training hyperparameters of Spatial-AST & Bat Configuration | Spatial-AST | Bat
---|---|---
Stage-1 | Stage-2 | Stage-1 | Stage-2 | Stage-3
Sound Source | AudioSet-2M | AudioSet-20K
Audio Normalization | ✓ | ✓
Augmentation | ✓ | ✗
Weighted sampling | ✓ | ✗
Optimizer | AdamW (Loshchilov & Hutter, 2019)
Optimizer momentum | $\beta_{1}=0.9,\beta_{2}=0.95$
Weight decay | 0.0001 | 0.05
Base learning rate | 0.001 | 0.001
Learning rate schedule | half-cycle cosine decay (Loshchilov & Hutter, 2017)
Warm-up epochs | 10 | 5 | 2
Epoch partitioning factor | 10 | 10
Epochs | 60 | 40 | 20 | 20 | 25
Batch size | 64 | 2
GPUs | 8 RTX 3090 | 8 Tesla V100

##  Appendix C Room Acoustics Characteristics

Table 6 presents the average RT60 values across some common room types in the
Matterport3D (Chang et al., 2017) training settings. RT60 is a standard
acoustic measurement that is defined as the time it takes for the sound
pressure level to reduce by 60 dB (Hak et al., 2012). This metric varies
across different rooms due to factors such as room size, object arrangement,
and construction materials, each influencing the room’s acoustic
characteristics.

Table 6: Average reverberation time (RT60) for different rooms. | Living Room | Bedroom | Closet | Bathroom | Toilet | Garage | Lounge | Kitchen | Dining Room
---|---|---|---|---|---|---|---|---|---
RT60 | 0.2076 | 0.1917 | 0.1950 | 0.1906 | 0.1706 | 0.2190 | 0.2224 | 0.2061 | 0.2201

##  Appendix D Performance of Spatial-AST in a Single-Task Setting

We evaluate the performance of our encoder in single-task scenarios,
specifically focusing on classification, distance prediction, and direction-
of-arrival (DoA). As shown in Table 7, we analyze the impact of including or
omitting Interaural Phase Difference (IPD), as well as comparing monaural
versus binaural formats, on the performance of each task. Our findings reveal
that IPD generally enhances performance across all tasks, indicating the value
of spatial information. Moreover, we observe that monaural data provides
sufficient information for tasks like sound event detection and distance
prediction, achieving the best performance in Distance Prediction and
comparable results to binaural data in Sound Event Detection. However,
binaural data significantly improves performance in tasks involving direction
perception, underscoring the encoder’s potential to handle multiple tasks
effectively and the importance of audio format in spatial audio perception.

Table 7: Performance of the encoder in single-task scenarios: classification, distance prediction, and Direction-of-Arrival (DoA). We compare the impact of including or excluding Interaural Phase Difference (IPD) on the performance of each task. Key evaluation metrics include mean Average Precision (mAP, $\uparrow$), Error Rate at 20° (ER20°, $\downarrow$), Mean Angular Error (MAE, $\downarrow$), and Distance Error Rate (DER, $\downarrow$). Task | Data | Performance Metrics
---|---|---
Audio Format | Input Features | mAP (%) $\uparrow$ | ER20° (%) $\downarrow$ | MAE (°) $\downarrow$ | DER (%) $\downarrow$
Detection | Monaural | Mel-spectrogram | 52.13 | - | - | -
Direction-of-Arrival | - | 96.80 | 85.47 | -
Distance Prediction | - | - | - | 21.44
Detection | Binaural | Mel-spectrogram | 51.70 | - | - | -
Mel-spectrogram, IPD | 52.26 | - | - | -
Direction-of-Arrival | Mel-spectrogram | - | 21.42 | 15.98 | -
Mel-spectrogram, IPD | - | 20.11 | 15.36 | -
Distance Prediction | Mel-spectrogram | - | - | - | 25.55
Mel-spectrogram, IPD | - | - | - | 22.60
Generalist | Binaural | Mel-spectrogram, IPD | 50.03 | 23.89 | 17.94 | 32.54

##  Appendix E Ablations of the Spatial Audio Encoder Spatial-AST

Loss Weights. As detailed in Equation 6, we fix $\lambda_{2}=1$ and
$\lambda_{3}=2$, and by adjusting the value of $\lambda_{1}$, we monitor
changes in the model’s performance. As depicted in Figure 2, with the increase
of $\lambda_{1}$, the mAP consistently improves. However, for the MAE metric
in direction-of-arrival (DoA), the performance gradually deteriorates. This
presents a trade-off between the two metrics. Prioritizing classification, we
opt for settings that yield the highest mAP. In subsequent experiments, we fix
the hyperparameters at $\lambda_{1}=1250,\lambda_{2}=1,\lambda_{3}=2$.

Figure 2: Classification (mAP) and Direction-of-Arrival (MAE) for different
$\lambda_{1}$ values.

Audio Normalization and Reverberation. In this ablation study, we assess the
effects of audio normalization and reverberation on the encoder’s performance,
using monaural and fixing $\lambda_{1},\lambda_{2},\lambda_{3}=1250,1,2$ for
simplicity, as outlined in Table 8. The addition of reverberation indeed
impacts the model’s ability to detect sound events, as it makes the task more
challenging without adding extra event information. A comparison between rows
1 and 2 with rows 3 and 4 indicates that removing reverberation significantly
enhances mean Average Precision (mAP), increasing from 51.13% to 52.64%.
However, it’s noteworthy that audio without reverberation lacks spatial
information, leading to a Distance Error Rate (DER) for distance perception
that is nearly equivalent to random guessing. Based on the distance
distribution described in Figure 3, random sampling yields approximately 67%
DER, and when sampling is limited to specific distances like 1.0, 1.5, and 2.0
meters, it results in around 53% DER. The implementation of audio
normalization further improves the encoder’s performance in both
classification and distance tasks. Comparing rows 3 and 4, the mAP increases
slightly from 51.13% to 51.39%, while DER significantly drops from 39.57% to
26.87%, underscoring the benefits of audio normalization in our model.

Figure 3: Frequency distribution of training reverberations’ distances ranging
from 0 to 10 meters.

Table 8: Ablation study on monaural audio input: We employ our proposed
architecture to train and infer under four distinct scenarios, with and
without audio normalization and reverberation.

Normalization | Reverberation | mAP (%) ↑ | DER (%) ↓
---|---|---|---
✗ | ✗ | 52.64 | 49.49
✓ | ✗ | 52.87 | 49.51
✗ | ✓ | 51.13 | 39.57
✓ | ✓ | 51.39 | 26.87

##  Appendix F SpatialSoundQA Generation

Figure 4: Sound events A and B occur in different directions and distances
simultaneously.

In the process of generating SpatialSoundQA, we simulate spatial audio
scenarios where multiple sound events occur simultaneously, each at distinct
directions and distances from the receiver. For example, as depicted in Figure
4, consider a situation where two sound sources, A and B, are active
concurrently. Sound event A occurs at an azimuth of 115 degrees and an
elevation of 40 degrees, situated 2.5 meters away from the receiver. In
contrast, sound source B occurs at coordinates of -40 degrees azimuth and -25
degrees elevation, positioned 6 meters distant. It’s important to note that in
the actual dataset construction, the direction and distance parameters are
often not as distinct as in this example. With these ground truth spatial
parameters, we can then construct a diverse array of questions and answers
concerning the spatial relationships between the two sound sources, as shown
in Table 1.

In supplement to Table 1, we provide details for each question type, including
the initial prompt used and the corresponding answer template.

Table 9: Prompts and answer templates for SpatialSoundQA Type |  Prompt | Answer template
---|---|---
A: Detection |  Identify the sound events in the audio clip. | sound event1; sound event2
B: DP & DoA |  Where is the audio clip coming from? | {d1}, {d2}, {d3}; {distance}
How would you describe the location of the {sound event}’s sound?
C: Detection |  Identify the sound events in the audio clip coming from the {d1}, {d2}, {d3}, approximately {distance} away. | sound event1; sound event2
D: DP & DoA |  Where is the sound of the {sound event} coming from? | {d1}, {d2}, {d3}; {distance}
E: Reasoning |  Do the sound of {sound event1} and the sound of {sound event2} both come from your {d} side? | Yes/No
Can you hear the {sound event1}’s sound from the {d1} and the {sound event2}’s
from the {d2}?
Would you find the sound of {sound event1} on the {d} side of the sound of
{sound event2}?
In terms of straight-line distance, does the sound of {sound event1} reach you
from a closer point compared to the sound of {sound event2}?

##  Appendix G Prompt Details

For our spatial audio-based large language model Bat, we designed a specific
prompt to guide its responses to various spatial audio tasks. The prompt
structure is as follows:

    "Based on the audio you’ve heard, refer to the instruction and provide a response.\n\n
    ### Instruction:\n{instruction}\n\n### Response:"

To form the input, we replace “{instruction}” with our specific question, and
then concatenate a fixed length of learnable query tokens at the beginning.

##  Appendix H Ablations of Bat

In Table 10, we thoroughly examine the influence of different training
curriculum on Bat’s reasoning ability. Initially, the model encounters
difficulties with reasoning tasks, particularly when facing unfamiliar
reasoning-style questions, as evidenced by its inability to produce accurate
binary (Yes/No) responses. A one-stage version of Bat surprisingly achieves a
respectable overall Binary Accuracy (BA) of 73.55% in reasoning tasks.
However, when we remove question types C and D, which are related to sound
source separation, from the one-stage training, the model’s reasoning ability
drops significantly, exhibiting near-random performance with a BA of only
54.68%. Intriguingly, when Bat is first trained on sound source separation
tasks before proceeding to the final reasoning stage, it retains its ability
to reason effectively, even in the absence of separation-related questions
during the final stage. The last row of Table 10 showcases the optimal results
achieved through our three-stage curriculum, demonstrating the importance of a
comprehensive training approach in equipping Bat with robust reasoning skills.

Table 10: The influence of different training curriculum on Bat’s reasoning
ability.

Model | #Sources | Stage | Reasoning (BA $\uparrow$)
---|---|---|---
1 | 2 | 3 | Direction | Distance | Avg.
Random | - | - | - | - | 50 | 50 | 50
Bat | 2 | ✓ | ✗ | ✗ | 0 | 0 | 0
✓ | ✓ | ✗ | 0 | 0 | 0
✗ | ✗ | ✓ | 65.12 | 81.98 | 73.55
✗ | ✗ | ✓* | 59.59 | 49.76 | 54.68
✗ | ✓ | ✓* | 64.69 | 79.16 | 71.93
✓ | ✓ | ✓* | 69.56 | 83.51 | 76.54
✓ | ✓ | ✓ | 69.77 | 84.01 | 76.89

  * *

We removed question types C and D from SpatialSoundQA.

Table 12 demonstrates that the combination of both CLS and audio tokens yields
the best performance in reasoning tasks. However, it is noteworthy that using
only three CLS tokens as a representation for spatial audio also results in
impressive performance. As per Table 12, a larger learnable query size in the
projection module correlates with improved reasoning efficacy.

Table 11: Impact of different tokens from Spatial-AST on One-stage Bat’s reasoning ability. CLS tokens | Audio Tokens | Reasoning (BA $\uparrow$)
---|---|---
Direction | Distance | Avg.
✓ | ✗ | 60.32 | 72.92 | 66.62
✗ | ✓ | 61.70 | 78.95 | 70.33
✓ | ✓ | 65.12 | 81.98 | 73.55

Table 12: Impact of learnable query size on reasoning accuracy. Learnable Query | Reasoning (BA $\uparrow$)
---|---
Direction | Distance | Avg.
32 | 61.42 | 83.49 | 72.45
64 | 69.77 | 84.01 | 76.89

##  Appendix I Limitations and Future Work

We are confident that Bat will significantly contribute to the development of
spatial audio perception and reasoning as well as multimodal LLM. However, as
with any research tool, it is crucial to acknowledge certain limitations and
assumptions inherent in our approach.

One of the primary challenges we encounter is extracting as much information
as possible from spatial audio, with a specific focus on how to better utilize
phase information. Currently, our model handles a maximum of two sound
sources, with an emphasis primarily on audio. Looking ahead, there is
potential to expand into multi-source scenarios and to integrate both audio
and speech processing for a more holistic approach. Additionally, while our
current framework is limited to binaural audio, exploring ambisonics could
provide a more immersive and realistic spatial audio experience. Moreover,
expanding SpatialSoundQA to be more open-ended would better align with human
usage patterns and preferences, allowing for a wider range of queries and
responses.

Another important consideration is the integration of additional modalities,
such as visual information, to complement auditory cues. This multimodal
approach might enhance the model’s understanding and interpretation of complex
environments. Lastly, the sim2real gap remains an aspect that requires further
investigation. Observing how our model performs in real-world scenarios, as
opposed to simulated environments, will be crucial in assessing its practical
applicability and effectiveness.


# Interface transparency to orbital current

Igor Lyalin  Department of Physics, The Ohio State University, Columbus, Ohio
43210, USA  Roland K. Kawakami<EMAIL_ADDRESS>Department of Physics, The
Ohio State University, Columbus, Ohio 43210, USA

###### Abstract

The transport of spin currents across interfaces is relatively well studied,
while the transport properties of orbital currents are just starting to be
examined. In Cr/Ni bilayers, the spin-orbit torque (SOT) due to the orbital
current generated in the Cr layer is believed to dominate over torques of
other origins. In this work, we study SOT in Cr/X/Ni trilayers, where X is an
ultra-thin spacer of a different material. Using the SOT as a proxy for the
orbital current transferred from the Cr to the Ni layer, we compare Cr/X/Ni
results to Pt/X/Ni, the system in which spin current generated in the Pt layer
plays a dominant role. We find that across 12 different spacers the apparent
interface transparency to the orbital current is comparable or larger than to
the spin current.

††preprint: APS/123-QED

##  I Introduction

The orbital Hall effect (OHE) describes a phenomenon in which an electrical
charge current generates a transverse flow of orbital angular momentum. It is
predicted theoretically that the OHE can exist in solids even in the absence
of spin-orbit coupling (SOC) and give rise to the spin Hall effect (SHE) when
SOC is present [1, 2, 3, 4, 5]. Thus, one can argue that the OHE is more
fundamental than SHE [2, 3]. Recently, there has been a growing amount of
experimental observations supporting the idea of the orbital Hall effect.
Unexpectedly large spin-orbit-torque (SOT) signals has been measured in
seemingly trivial systems without heavy metals, e.g. CuO/FM, Cr/FM, and
attributed to the orbital currents generated in non-magnetic layers [6, 7, 8,
9, 10]. It has also been observed that while SOT is usually large and negative
in Ta/FM bilayers, it depends on the choice of FM and is positive in Ta/Ni
[11]. This was attributed to the torque due to the OHE dominating over the
SHE-induced torque. More recently, an orbital accumulation due to the OHE on
the surface of a single layer of light metals Ti [12] and Cr [13] was detected
by magneto-optic Kerr effect, confirming the existence of the OHE more
directly.

While spin transport is relatively well studied, the transport properties of
orbital currents have just started to be investigated. Interface transparency
to orbital current is an important question from both, fundamental and
application, points of view, since to utilize orbital currents for spintronic
applications the efficient injection across NM/FM is required. It has been
suggested theoretically that orbital transport being dependent on the orbital
hybridization at the interface is more sensitive to interface quality and
disorder than spin transport [14].

In this work, we study interface transparency to orbital current in Cr/X/Ni
trilayers, where X is an ultra-thin spacer of a different material. In Cr/Ni
bilayers, the SOT due the orbital current generated in Cr layer is believed to
dominate over torques of the other origin [8, 9]. We use the experimentally
measured SOT as a proxy for the orbital current transferred from Cr to Ni
layer. Comparing Cr/X/Ni to a trilayer system in which spin current plays the
major role, Pt/X/Ni, we find that across 12 different spacers the apparent
interface transparency to the orbital current is comparable or larger than to
the spin current.

##  II Experiment

Using current-modulated magneto-optic Kerr effect (MOKE) technique [15], we
measure the damping-like spin-orbit torque in Cr/X/Ni (20/0.8/5 nm) and
Pt/X/Ni (20/0.8/5 nm) trilayers, where the numbers in parentheses represent
the thickness in nanometers. We study 15 different spacer layers X: Al, Ti,
Fe, Co, Cu, Ge, Pd, Ag, Sn, Ta, W, Pt, Au, NiO, MgO. The films are deposited
at room temperature on Mg(100) substrates by a combination of electron beam
evaporation and magnetron sputtering in a chamber with a base pressure of
$2\times 10^{-7}$ Torr. To protect the films from oxidation, they are capped
with MgO/Cr (3/3 nm). An example of the full stack sequence is
MgO/Cr/Cu/Ni/MgO/Cr (substrate/20/0.8/5/3/3 nm). For Pt/X/Ni samples a 1 nm
thick Cr seed layer is used to improve adhesion between a Pt layer and an MgO
substrate. During the growth, the thickness of the deposited materials is
monitored by a quartz crystal, which readings are calibrated by X-ray
reflectivity measurements. We estimate the error of the spacer layer thickness
to be about $20\%$.

The films are patterned into 20 $\mu$m wide devices by photolithography and
argon ion milling. Measuring the resistivity of Cr/MgO/Cr (1/3/3 nm) stack, we
estimate that less than 1% of total current applied to the devices is flowing
through the seed layer and partially oxidized Cr cap. The average
resistivities of the entire Cr/X/Ni and Pt/X/Ni stacks are $(96\pm
19)\,\mu\Omega$cm and $(31\pm 3)\,\mu\Omega$cm, respectively. We do not
attempt to estimate the resistivities of the individual layers since the
resistivity of thin multilayers depends strongly on interfacial scattering,
while the growth, and as a result, the resistivity of Ni layer, is affected by
the choice of the underlying spacer X.

A mode-locked Ti:Sapphire laser tuned to $\lambda=800$ nm wavelength is
utilized for the polar MOKE measurements. The sample is mounted on a motorized
XY stage for scanning in the sample plane. To exclude contributions from
quadratic MOKE, the light is linearly polarized at $45\degree$ relative to the
applied magnetic field [16]. The beam is focused on the sample to a spot size
of about 2 $\mu$m. The laser power incident on the sample is $800\,\mu$W. The
Kerr rotation of the reflected light is measured by a combination of a half
wave plate, Wollaston prism, and balanced photodetector. The signal from the
balanced photodetector is passed to a lock-in amplifier, which measures the
current-induced Kerr rotation by demodulating at the first harmonic of the
applied sinusoidal current of 41227 Hz frequency. The schematic of the
experimental geometry is shown in Fig. 1.

Figure 1:  Schematics of the current-modulated polar MOKE experiment. A laser
is scanned across a Cr/X/Ni device. The Kerr rotation detects the current-
induced out-of-plane component of magnetization due to the damping-like
effective SOT field $\bm{H_{DL}}\sim[\bm{m}\times\bm{\sigma}]$ and the out-of-
plane Oersted field.

##  III Results and Discussion

In Fig. 2a, we show the current-modulated MOKE data measured for Cr/Ni.
Measuring line scans across the 20 $\mu$m wide device for the Ni magnetization
parallel or antiparallel to the applied current, we separate the MOKE data
into the symmetric part (blue dots in Fig. 2a), and antisymmetric part (red
squares in Fig. 2a).

(a)left-0pt+/2right-3pt-/2up+10pt-/2down3pt+/2[left,up][vc,hc]

(a)

(b)left-0pt+/2right-3pt-/2up+10pt-/2down3pt+/2[left,up][vc,hc]

(b)

(c)left-0pt+/2right-3pt-/2up+10pt-/2down3pt+/2[left,up][vc,hc]

(c)

(d)left-0pt+/2right-3pt-/2up+10pt-/2down3pt+/2[left,up][vc,hc]

(d)

Figure 2:  Current-modulated MOKE line scan across a 20 $\mu$m wide channel of
(a) Cr/Ni (20/5 nm) and (b) Cr/Ge/Ni (20/0.8/5 nm) at current $I=20$ mA,
corresponding to current density $j=4\times 10^{10}$ A/m2. Effective damping-
like SOT field, $\mu_{0}H_{DL}$, as a function of applied current for (c)
Cr/Ni and (d) Cr/Ge/Ni.

The symmetric component is proportional to the effective damping-like SOT
field $\sim$$H_{DL}$, while antisymmetric component is proportional to the
out-of-plane Oersted field $\sim$$H^{oop}_{Oe}$ [15]. Using calibration by the
Oersted field, the DL-SOT signal is converted from MOKE voltages to actual
magnetic field values. Effective DL-SOT field as a function of applied current
$I$ for the Cr/Ni device is plotted in Fig. 2c. We observe that the signal
scales linearly with the current, as expected for the DL-SOT. The slope of the
current dependence, $\mu_{0}H_{DL}/I$, dimensions of the device, and the
resistivity of the whole stack $\rho_{xx}$ is used to determine the damping-
like SOT efficiency per applied electric field [17]:

$\xi_{E}=\dfrac{2e}{\hbar}M_{s}t_{FM}\dfrac{\mu_{0}H_{DL}}{E}\,,$ (1)

where $e$ is the electron charge, $\hbar$ is the reduced Planck constant,
$t_{FM}$ is the thickness of the Ni layer, $M_{s}$ is its saturation
magnetization, $E$ is the applied electric field. The surface magnetization
$M_{s}t_{FM}$ is measured by superconducting quantum interference device
(SQUID) magnetometry. We note, that we chose $t_{NM}$, the thickness of NM
layer, to be larger than the orbital (spin) diffusion length
$\lambda^{Cr}_{o}\approx 6$ nm [8, 13] ($\lambda^{Pt}_{s}\approx$ 1–11 nm [17,
18]). Since DL-SOT efficiency saturates with the thickness of the NM layer as
$\sim[1-\text{sech}(t_{NM}/\lambda^{NM}_{o(s)}]$, for $t_{NM}=20$ nm it is
close to its saturated value.

Measuring three Cr/Ni samples grown at different times we find that
$\xi_{E}=(0.17\pm 0.05)\times 10^{3}\,\Omega^{-1}$cm-1, consistent with
previous reports [8, 9]. The positive sign of the torque agrees with the
picture of the orbital torque dominating over the negative spin torque
generated by the SHE in Cr. Performing measurements on Pt/Ni we obtain an
efficiency $\xi_{E}=(2.43\pm 0.11)\times 10^{3}\,\Omega^{-1}$cm-1 of the same
sign.

Figure 2b and 2d shows the current-modulated MOKE data for Cr/X/Ni with one of
the spacers, X = Ge. A histogram in Figure 3 summarizes DL-SOT efficiency
results for all spacers. Cr/X/Ni and Pt/X/Ni results are plotted side-by-side
for each X to compare the behavior of the system dominated by orbital current
with the system dominated by the spin current. Each Cr/X/Ni column is the
average of two samples grown at different times. In the following, we discuss
the results summarized in the Figure 3, making a number of observations.

(a)  Figure 3:  Damping-like SOT efficiency per electric field for Cr/X/Ni and
Pt/X/Ni trilayers. The results are plotted side-by-side for each X to compare
the behavior of the system dominated by orbital current, Cr/X/Ni, with the
system dominated by the spin current, Pt/X/Ni. The DL-SOT efficiency for all
Pt/X/Ni trilayer is scaled by 0.07 to match the Pt/Ni value to the Cr/Ni
value.

###  III.1 Non-magnetic metal spacers

Across 11 non-magnetic spacer materials (Al, Ti, Cu, Ge, Pd, Ag, Sn, Ta, W,
Pt, Au) apparent interface transparency to orbital current is comparable or
larger than to spin current. We define the apparent orbital interface
transparency ratio as $\mathcal{T}_{o}^{\rm
X}=\frac{\xi^{\rm{Cr/X/Ni}}_{E}}{\xi^{\rm{Cr/Ni}}_{E}}$ and the apparent spin
interface transparency ratio as $\mathcal{T}_{s}^{\rm
X}=\frac{\xi^{\rm{Pt/X/Ni}}_{E}}{\xi^{\rm{Pt/Ni}}_{E}}$.

Light metals and interface transparencies. The DL-SOT efficiency due to the
SHE can be expressed as $\xi_{E}=T_{s}\sigma^{\rm NM}_{\rm SH}$, where $T_{s}$
is the spin interface transparency, $\sigma^{\rm NM}_{\rm SH}$ is spin Hall
conductivity of the NM layer. According to the current understanding [14], the
orbital torque is a result of the orbital current generated by the OHE in the
NM layer, transferred across NM/FM interface, and converted to the spin
current by the SOC of the FM layer. The resultant spin current exerts a torque
on the FM magnetization, as the orbital current cannot interact with the
magnetization directly [14]. Thus, the DL-SOT efficiency due to the OHE can be
expressed as $\xi_{E}=T_{o}\eta^{\rm FM}_{so}\sigma^{\rm NM}_{\rm OH}$, where
$T_{o}$ is the orbital interface transparency, $\eta^{\rm FM}_{so}$ is the
conversion ratio from the orbital to spin in the FM layer, proportional to SOC
in the FM, $\sigma^{\rm NM}_{\rm OH}$ is orbital Hall conductivity of the NM
layer, see schematics in Fig. 4.

Figure 4:  Schematics of the orbital current transferred across an NM/FM
interface. The orbital current generated by a NM layer is reduced by a factor
$T_{o}$ at the interface, before being converted into a spin current by the
spin-orbit coupling in the FM layer, with the conversion ratio
$\eta_{so}^{FM}$. The converted spin current exerts an orbital torque on the
FM magnetization.

For light 3d transition metals Ti and Cu with weak SOC, it would be reasonable
to assume that within the ultra-thin 0.8 nm spacer a negligible amount of
orbital and spin current is generated or destroyed since their orbital and
spin diffusion lengths are much larger than the thickness of the spacer [12,
7, 19], thus $\sigma^{\rm NM}_{\rm OH}$ should not be modified. It is also
logical to assume that, having weak SOC, Ti and Cu do not significantly modify
$\eta^{\rm FM}_{so}$ of Ni, as a spacer with strong SOC, for instance, Pt,
does. Therefore, in a simplistic model of two perfect interfaces, the apparent
transparency for X = Ti can be written as $\frac{\xi_{E}^{\rm
Cr/Ti/Ni}}{\xi_{E}^{\rm Cr/Ni}}=\frac{T_{o}^{\rm Cr/Ti}T_{o}^{\rm
Ti/Ni}}{T_{o}^{\rm Cr/Ni}}$. Under the assumption that the three interfaces
between the three similar $3d$ transition metals have the same transparency,
we estimate the order of magnitude of orbital transparency as $T_{o}^{\rm
Cr/Ni}\approx T_{o}^{\rm Cr/Ti}\approx T_{o}^{\rm Ti/Ni}\approx 40\%$. The
same logic applied to X = Cu yields an unrealistic value of $\approx 130\%$.
For comparison, the spin interface transparency $T_{s}$ of Py/Pt and Py/Ni/Pt
interfaces (Py = Ni80Fe20) was investigated in Ref. [20], and estimated to be
$\approx 25\%$. These estimates of orbital interface transparency for X = Ti,
Cu, are comparable to or larger than the spin interface transparency of Py/Pt
and Py/Ni/Pt. We emphasize that we do not attempt to give a precise
quantitative evaluation of the orbital interface transparency by this
oversimplified reasoning, but rather to perform a qualitative analysis. For
discussion later in the paper, we take the lower estimate of $\approx 40\%$
for the Cr/Ni orbital interface transparency $T_{o}^{\rm Cr/Ni}$. Lastly, we
note that the model above implies that the two interfaces can be treated
independently which might not be a good assumption for a very thin spacer. We
also observe that $\mathcal{T}_{o}^{\rm X}\gtrsim\mathcal{T}_{s}^{\rm X}$ for
the other light elements X = Al and Ge.

Based on these observations for light elements with weak SOC, we tentatively
conclude that not only the apparent orbital interface transparency
$\mathcal{T}_{o}^{\rm X}$ (which could include SOC effects in the spacer or FM
as discussed later), but also the actual orbital interface transparency could
be comparable or larger than the spin interface transparency.

Orbital-to-spin conversion in the Ni layer. Using $\xi_{E}=T_{o}\eta^{\rm
FM}_{so}\sigma^{\rm NM}_{\rm OH}$, theoretically calculated $\sigma^{\rm
Cr}_{\rm OH}=7\times 10^{5}\,\Omega^{-1}\rm m^{-1}$ [21], and experimentally
measured $\xi_{E}=(0.17\pm 0.05)\times 10^{5}\,\Omega^{-1}\rm m^{-1}$ we can
estimate $T^{\rm Cr/Ni}_{o}\eta^{\rm Ni}_{so}$ to be $\approx 2.4\%$. Using
the estimate of $40\%$ for $T^{\rm Cr/Ni}_{o}$ translates into $\eta^{\rm
Ni}_{so}\approx 6\%$. Therefore, while increasing interface transparency is
important for utilizing orbital torque in SOT-based applications, increasing
the orbital-to-spin conversion in FM layer might be a more promising route, as
there is more room for improvement. For example, this can be achieved by
alloying Ni and a HM with a large and positive SOC, e.g. using Ni90Pt10 as a
ferromagnetic layer.

Orbital character of the spacer (p vs. d). We observe that the apparent
orbital transparency for Cu and Ge spacers is larger than their apparent spin
transparency. The SOC strength increases with atomic number $Z$, thus one
could try to explain the monotonic increase Ti $\rightarrow$ Cu $\rightarrow$
Ge of the $\mathcal{T}_{o}^{X}$, by enhanced $\eta^{\rm Ni}_{so}$ due to
additional orbital-to-spin conversion in the spacer layer. However, the
$\mathcal{T}_{o}^{Cu}>\mathcal{T}_{o}^{Ag}$ and
$\mathcal{T}_{o}^{Ge}>\mathcal{T}_{o}^{Sn}$, of the heavier elements in the
same group in the periodic table with even larger SOC strength. Thus, the SOC
of these spacers alone cannot account for the enhanced orbital torque.
Instead, we attribute it to the enhanced interface transparency by Cu and Ge
spacers. The 2.5-fold increase of the interface transparency for X = Ge
(compared to without a spacer) might be due to $4p$-orbitals of Ge, indicating
that $p$-$d$ hybridization at the interface facilitates orbital transport in
comparison with the case of Cr/Ni interface that have only $s$\- and
$d$-orbitals. If $T^{\rm Cr/Ni}_{o}\approx 40\%$, then Cr/Ge/Ni interface is
close to being fully transparent. Thus, to maximize the orbital interface
transparency, and, as a result, orbital torque efficiency, elements with
different orbital character on the opposite sides of NM/FM bilayers might be
preferable. We note, that so far, the orbital torque has been studied mostly
in $3d/3d$, $4d/3d$, and $5d/3d$ bilayers.

Heavy metals with positive SOC. The insertion of $4d$ and $5d$ elements with
positive SOC (Pd, Ag, Sn, Pt, Au) seems to preserve or increase the
$\mathcal{T}_{o}^{\rm X}$. We attribute this to the increase of the effective
orbital-to-spin conversion parameter $\eta^{\rm Ni}_{so}$ in the Ni layer, by
being adjacent to a material with a larger positive SOC strength. Consistent
with this picture, $\mathcal{T}_{o}^{\rm X}$ is larger for the heavier $5d$
spacers (Pt and Au) than for $4d$ elements (Pd, Ag, Sn). Pt and Pd are known
to be strong spin sources, so the charge current even in 0.8 nm-thick layers
can generate a positive spin current that can enhance the measured torque.
This could explain why $\mathcal{T}_{o}^{\rm Pd}\gtrsim\mathcal{T}_{o}^{\rm
Ag}$ and $\mathcal{T}_{o}^{\rm Pt}\gtrsim\mathcal{T}_{o}^{\rm Au}$. For X =
Sn, both, enhanced $\eta^{\rm Ni}_{so}$ due to the Sn SOC and enhanced $T_{o}$
due to $5p$-$3d$ hybridization, can play a role.

Heavy metals with negative SOC. The behavior of the apparent transparency for
Ta and W is consistent with the interpretation given in the previous
paragraph. Having a negative SOC, Ta, and W decrease the positive value of
$\eta^{\rm Ni}_{so}$. As a result, the measured DL-SOT torque is reduced. The
spacer thickness is likely not large enough to fully convert the positive
orbital current into a negative spin current. Furthermore, it is not obvious
that for a larger thickness of Ta (W), the torque would change the sign, as
for thicker Ta layers the positive orbital torque due to the OHE in Ta has
been shown to dominate over the negative SHE-induced torque in Ta/Ni bilayers
[11].

###  III.2 Ferromagnetic metal spacers

The only spacers that decrease orbital torque in Cr/X/Ni to zero, or even
reverse its sign, are magnetic spacers, Fe and Co. We attribute this to the
$\eta^{\rm FM}_{so}$ being negative in Fe, $\eta^{\rm Fe}_{so}<0$, and almost
zero in Co, $\eta^{\rm Co}_{so}\approx 0$. This agrees with the experimental
observation of the self-induced anomalous SOT (ASOT) in single layers of Fe,
Co and Ni [22]. The ASOT arises from the SOC of FM layer, that generates the
spin current that acts on its own magnetization. In the reference [22], the
anomalous SOT was found to be negative in Fe, a small positive in Co, and the
largest positive in Ni, although this contradicts theoretical calculations in
ref. [11], where the spin-orbit correlation at the Fermi level were calculated
to be positive for all the three ferromagnetic materials. We observe that the
DL-SOT is negative in Cr/Fe/Ni and half of the value of Cr/Ni. The DL-SOT in
Cr/Co/Ni is close to zero. If the orbital diffusion length in Fe and Co would
be much larger than 0.8 nm, then assuming similar to Cr/Ni orbital
transparency at the interface, the orbital current would reach Ni layer and be
converted to the positive spin current, thus generating a positive DL-SOT.
However, we observe the opposite. Two possible scenarios can explain the
reversal of the torque for the X = Fe: a) the orbital diffusion length in Fe
is ultra-short, comparable with the thickness of the spacer $\sim$ $0.8$ nm,
which leads an efficient orbital-to-spin conversion before the orbital current
reaches the Ni layer, b) the orbital transparency of the Cr/Fe interface is
much smaller than $T_{o}^{\rm Cr/Ni}$ which leads to the negative spin torque
due to the SHE in Cr dominating the measurement. The observation of the
apparent orbital transparency being comparable or higher than spin
transparency for all other spacers makes us to believe that the scenario a) is
more likely. This scenario can also explain the nearly zero DL-SOT in
Cr/Co/Ni. The small positive $\eta^{\rm Co}_{so}$ and ultra-short
$\lambda^{\rm Co}_{o}$ results in a smaller positive orbital torque than in
Cr/Ni which is compensated by the negative SHE-induced torque. We note, that a
very short $\lambda_{o}$ in FMs agrees with earlier theoretical calculations
[14], but contradicts to later calculations [23].

###  III.3 Antiferromagnetic insulator spacers

The spin current from heavy metals can interact with magnons in magnetic
insulators. It has been shown that a thin layer of NiO can transfer spin
current from HM to FM for temperatures above and below the Néel temperature
[24, 25, 26, 27]. Using NiO as a spacer, we test whether the orbital current
can be transferred by magnons as well. Unlike other spacers, the 0.8 nm of NiO
is synthesized by oxidation. We reverse the stack sequence, first growing 5.5
nm of Ni on MgO substrate and then oxidizing the top 0.5 nm of Ni in 1 mTorr
of oxygen atmosphere for 1 minute to the self-limiting value of 0.8 nm of NiO.
This oxygen exposure is more than 100 times of the minimal oxygen dose
required [28, 29, 30]. After that we deposit 20 nm of Cr on the top and
perform current modulated MOKE measurement through the MgO substrate side. We
find that the apparent transparency of NiO is about 60% which is larger than
the value for the Ti and Al spacers. A NiO spacer grown in the same way for
Pt/NiO/Ni reduces DL-SOT efficiency to the 30% of the Pt/Ni value. Thus, the
apparent orbital interface transparency of NiO is larger than its spin
transparency. In addition, we perform measurements on Cr/MgO/Ni as a control
sample. We observe that DL-SOT is 3 times smaller with the MgO, than with the
NiO spacer. This indicates that electron tunneling across a 0.8 nm insulating
layer cannot account for the large observed $\mathcal{T}_{o}^{\rm NiO}$ and a
magnon-mediated mechanism of transferring orbital angular momentum from Cr to
Ni probably plays a role. We attribute the non-zero torque in Cr/MgO/Ni
trilayers to the presence of pinholes in the MgO layer. We note, that NiO
grown by oxidation is pinhole free [28, 31]. A detailed study is needed to
investigate the exact mechanism by which conduction electrons in a metal can
transfer their orbital angular momentum to magnons in a magnetic insulator.
Recent theoretical works have investigated the orbital angular momentum of
magnons [32, 33], furthermore, an intrinsic magnon OHE was proposed [34].

##  IV Conclusions

In this work, we studied the spin-orbit torque efficiency in Cr/X/Ni trilayers
with different ultra-thin spacers. Using the SOT as a proxy for the orbital
current transferred from Cr to Ni layer, we compared Cr/X/Ni to Pt/X/Ni, the
system in which spin current plays a dominant role. The interpretation of the
experimental results relies on an assumption that the positive SOT in Cr/Ni is
due to the orbital torque dominating over the torques of other origin [8, 9]:
spin torque, interfacial torque, and anomalous torque. Contrary to the
theoretical suggestion that the orbital current is more sensitive to interface
quality and disorder than the spin current [14], we found that across 12
different spacers, the apparent orbital interface transparency is comparable
to or larger than the spin interface transparency. In addition to this
principal result, the most interesting spacers are: a) Ge, which increases the
apparent orbital interface transparency by a factor of 2.5, b) ferromagnetic
Fe and Co, which suppress or even reverse the sign of the spin-orbit torque,
and c) insulating NiO, capable of a magnon-mediated transfer of orbital
angular momentum.

Our work provides a broad materials perspective on the orbital transport
across interfaces and can serve as a starting point for future in-depth
studies focused on a specific material combination, thicknesses, and growth
optimization.

## Acknowledgements

We thank Peter M. Oppeneer for insightful discussions. This research was
primarily supported by the Center for Emergent Materials, an NSF MRSEC, under
award number DMR-2011876. I.L. acknowledges support from the Ohio State
University Presidential Fellowship.

## References

  * Tanaka  _et al._ [2008] T. Tanaka, H. Kontani, M. Naito, T. Naito, D. S. Hirashima, K. Yamada, and J. Inoue, Intrinsic spin Hall effect and orbital Hall effect in $4d$ and $5d$ transition metals, Phys. Rev. B 77, 165117 (2008).
  * Kontani  _et al._ [2009] H. Kontani, T. Tanaka, D. S. Hirashima, K. Yamada, and J. Inoue, Giant Orbital Hall Effect in Transition Metals: Origin of Large Spin and Anomalous Hall Effects, Phys. Rev. Lett. 102, 016601 (2009).
  * Go  _et al._ [2018] D. Go, D. Jo, C. Kim, and H.-W. Lee, Intrinsic Spin and Orbital Hall Effects from Orbital Texture, Phys. Rev. Lett. 121, 086602 (2018).
  * Jo  _et al._ [2018] D. Jo, D. Go, and H.-W. Lee, Gigantic intrinsic orbital Hall effects in weakly spin-orbit coupled metals, Phys. Rev. B 98, 214405 (2018).
  * Bhowal and Satpathy [2020] S. Bhowal and S. Satpathy, Intrinsic orbital moment and prediction of a large orbital Hall effect in two-dimensional transition metal dichalcogenides, Phys. Rev. B 101, 121112 (2020).
  * Ding  _et al._ [2020] S. Ding, A. Ross, D. Go, L. Baldrati, Z. Ren, F. Freimuth, S. Becker, F. Kammerbauer, J. Yang, G. Jakob, Y. Mokrousov, and M. Kläui, Harnessing orbital-to-spin conversion of interfacial orbital currents for efficient spin-orbit torques, Phys. Rev. Lett. 125, 177201 (2020).
  * Kim  _et al._ [2021] J. Kim, D. Go, H. Tsai, D. Jo, K. Kondou, H.-W. Lee, and Y. Otani, Nontrivial torque generation by orbital angular momentum injection in ferromagnetic-metal/Cu/Al2O3 trilayers, Phys. Rev. B 103, L020407 (2021).
  * Lee  _et al._ [2021a] S. Lee, M.-G. Kang, D. Go, D. Kim, J.-H. Kang, T. Lee, G.-H. Lee, J. Kang, N. J. Lee, Y. Mokrousov, S. Kim, K.-J. Kim, K.-J. Lee, and B.-G. Park, Efficient conversion of orbital Hall current to spin current for spin-orbit torque switching, Commun. Phys. 4, 234 (2021a).
  * Sala and Gambardella [2022] G. Sala and P. Gambardella, Giant orbital Hall effect and orbital-to-spin conversion in $3d$, $5d$, and $4f$ metallic heterostructures, Phys. Rev. Research 4, 033037 (2022).
  * Hayashi  _et al._ [2023] H. Hayashi, D. Jo, D. Go, T. Gao, S. Haku, H.-W. Mokrousov, Yuriy Lee, and K. Ando, Observation of long-range orbital transport and giant orbital torque, Commun. Phys. 6, 32 (2023).
  * Lee  _et al._ [2021b] D. Lee, D. Go, H.-J. Park, W. Jeong, H.-W. Ko, D. Yun, D. Jo, S. Lee, G. Go, J. H. Oh, K.-J. Kim, B.-G. Park, B.-C. Min, H. C. Koo, H.-W. Lee, O. Lee, and K.-J. Lee, Orbital torque in magnetic bilayers, Nat. Commun. 12, 6710 (2021b).
  * Choi  _et al._ [2023] Y.-G. Choi, D. Jo, K.-H. Ko, D. Go, K.-H. Kim, H. G. Park, C. Kim, B.-C. Min, G.-M. Choi, and H.-W. Lee, Observation of the orbital Hall effect in a light metal Ti, Nature 619, 52 (2023).
  * Lyalin  _et al._ [2023] I. Lyalin, S. Alikhah, M. Berritta, P. M. Oppeneer, and R. K. Kawakami, Magneto-optical detection of the orbital hall effect in chromium, Phys. Rev. Lett. 131, 156702 (2023).
  * Go and Lee [2020] D. Go and H.-W. Lee, Orbital torque: Torque generation by orbital current injection, Phys. Rev. Res. 2, 013177 (2020).
  * Fan  _et al._ [2014] X. Fan, H. Celik, J. Wu, C. Ni, K.-J. Lee, V. O. Lorenz, and J. Q. Xiao, Quantifying interface and bulk contributions to spin–orbit torque in magnetic bilayers, Nat. Commun. 5, 3042 (2014).
  * Fan  _et al._ [2016] X. Fan, A. R. Mellnik, W. Wang, N. Reynolds, T. Wang, H. Celik, V. O. Lorenz, D. C. Ralph, and J. Q. Xiao, All-optical vector measurement of spin-orbit-induced torques using both polar and quadratic magneto-optic Kerr effects, Applied Physics Letters 109, 122406 (2016).
  * Nguyen  _et al._ [2016] M.-H. Nguyen, D. C. Ralph, and R. A. Buhrman, Spin torque study of the spin Hall conductivity and spin diffusion length in platinum thin films with varying resistivity, Phys. Rev. Lett. 116, 126601 (2016).
  * Stamm  _et al._ [2017] C. Stamm, C. Murer, M. Berritta, J. Feng, M. Gabureac, P. M. Oppeneer, and P. Gambardella, Magneto-Optical Detection of the Spin Hall Effect in Pt and W Thin Films, Phys. Rev. Lett. 119, 087203 (2017).
  * Bass and Pratt [2007] J. Bass and W. P. Pratt, Spin-diffusion lengths in metals and alloys, and spin-flipping at metal/metal interfaces: an experimentalist’s critical review, J. Phys.: Condens. Matter 19, 183201 (2007).
  * Zhang  _et al._ [2015] W. Zhang, W. Han, X. Jiang, S.-H. Yang, and S. S. P. Parkin, Role of transparency of platinum–ferromagnet interfaces in determining the intrinsic magnitude of the spin Hall effect, Nature Physics 11, 496 (2015).
  * Salemi and Oppeneer [2022] L. Salemi and P. M. Oppeneer, First-principles theory of intrinsic spin and orbital Hall and Nernst effects in metallic monoatomic crystals, Phys. Rev. Mater. 6, 095001 (2022).
  * Wang  _et al._ [2019a] W. Wang, T. Wang, V. P. Amin, Y. Wang, A. Radhakrishnan, A. Davidson, S. R. Allen, T. J. Silva, H. Ohldag, D. Balzar, B. L. Zink, P. M. Haney, J. Q. Xiao, D. G. Cahill, V. O. Lorenz, and X. Fan, Anomalous spin–orbit torques in magnetic single-layer films, Nat. Nanotechnol. 14, 819 (2019a).
  * Go  _et al._ [2023a] D. Go, D. Jo, K.-W. Kim, S. Lee, M.-G. Kang, B.-G. Park, S. Blügel, H.-W. Lee, and Y. Mokrousov, Long-range orbital torque by momentum-space hotspots, Phys. Rev. Lett. 130, 246701 (2023a).
  * Wang  _et al._ [2019b] Y. Wang, D. Zhu, Y. Yang, K. Lee, R. Mishra, G. Go, S.-H. Oh, D.-H. Kim, K. Cai, E. Liu, S. D. Pollard, S. Shi, J. Lee, K. L. Teo, Y. Wu, K.-J. Lee, and H. Yang, Magnetization switching by magnon-mediated spin torque through an antiferromagnetic insulator, Science 366, 1125 (2019b).
  * Zhu  _et al._ [2021a] L. Zhu, L. Zhu, and R. A. Buhrman, Fully spin-transparent magnetic interfaces enabled by the insertion of a thin paramagnetic NiO layer, Phys. Rev. Lett. 126, 107204 (2021a).
  * Zhu  _et al._ [2021b] L. Zhu, D. C. Ralph, and R. A. Buhrman, Maximizing spin-orbit torque generated by the spin Hall effect of Pt, Appl. Phys. Rev. 8 (2021b).
  * Zhu  _et al._ [2022] D. Zhu, T. Zhang, X. Fu, R. Hao, A. Hamzić, H. Yang, X. Zhang, H. Zhang, A. Du, D. Xiong, K. Shi, S. Yan, S. Zhang, A. Fert, and W. Zhao, Sign change of spin-orbit torque in $\mathrm{Pt}/\mathrm{NiO}/\mathrm{CoFeB}$ structures, Phys. Rev. Lett. 128, 217702 (2022).
  * Holloway [1981] P. H. Holloway, Chemisorption and oxide formation on metals: Oxygen–nickel reaction, Journal of Vacuum Science and Technology 18, 653 (1981).
  * Mitchell [1982] D. Mitchell, A kinetic study of the initial oxidation of Ni (111) and (211) surfaces by RHEED and X-ray emission, Surface science 114, 546 (1982).
  * Lambers  _et al._ [1996] E. S. Lambers, C. N. Dykstal, J. M. Seo, J. E. Rowe, and P. H. Holloway, Room-temerature oxidation of Ni(110) at low and atmospheric oxygen pressures, Oxidation of Metals 45, 301 (1996).
  * Kitakatsu  _et al._ [1998] N. Kitakatsu, V. Maurice, C. Hinnen, and P. Marcus, Surface hydroxylation and local structure of NiO thin films formed on Ni(111), Surface Science 407, 36 (1998).
  * Neumann  _et al._ [2020] R. R. Neumann, A. Mook, J. Henk, and I. Mertig, Orbital magnetic moment of magnons, Phys. Rev. Lett. 125, 117209 (2020).
  * Fishman  _et al._ [2022] R. S. Fishman, J. S. Gardner, and S. Okamoto, Orbital angular momentum of magnons in collinear magnets, Phys. Rev. Lett. 129, 167202 (2022).
  * Go  _et al._ [2023b] G. Go, D. An, H.-W. Lee, and S. K. Kim, Intrinsic magnon orbital Hall effect in honeycomb antiferromagnets (2023b), arXiv:2303.11687 [cond-mat.mes-hall] .