# Real-time auralization for performers on virtual stages

Ernesto Accolti  Instituto de Automática (Institute of Automation), National
University of San Juan (UNSJ) and the National Scientific and Technical
Research Council (CONICET), Argentina  Lukas Aspöck  Institute for Hearing
Technology and Acoustics, RWTH Aachen University, Aachen, Germany  Manuj Yadav
Institute for Hearing Technology and Acoustics, RWTH Aachen University,
Aachen, Germany  Michael Vorländer  Institute for Hearing Technology and
Acoustics, RWTH Aachen University, Aachen, Germany

###### Abstract

This article presents an interactive system for stage acoustics
experimentation including considerations for hearing one’s own and others’
instruments. The quality of real-time auralization systems for psychophysical
experiments on music performance depends on the system’s calibration and
latency, among other factors (e.g. visuals, simulation methods, haptics, etc).
The presented system focuses on the acoustic considerations for laboratory
implementations. The calibration is implemented as a set of filters accounting
for the microphone-instrument distances and the directivity factors, as well
as the transducers’ frequency responses. Moreover, sources of errors are
characterized using both state-of-the-art information and derivations from the
mathematical definition of the calibration filter. In order to compensate for
hardware latency without cropping parts of the simulated impulse responses,
the virtual direct sound of musicians hearing themselves is skipped from the
simulation and addressed by letting the actual direct sound reach the listener
through open headphones. The required latency compensation of the interactive
part (i.e. hearing others) meets the minimum distance requirement between
musicians, which is 2 m for the implemented system. Finally, a proof of
concept is provided that includes objective and subjective experiments, which
give support to the feasibility of the proposed setup.

##  1 Introduction

Auralization has a long tradition [1, 2, 3]. Furthermore, it has greatly
developed in the last decade [4, 5, 6, 7, 8, 9, 10, 11]. Although realism has
reached a great state of development, also for inclusion of musical instrument
directivities [4], many improvements are still possible [12].

Currently, auralization with high plausibility is emerging in many
applications – some even involving real-time processing – primarily for cases
where the sources are external to the listener, and/or relatively far away.
However, some details still remain as open research topics. That is especially
the case of real-time auralization for applications involving musicians
hearing themselves and/or their own instruments, musicians hearing other
musicians/instruments who are very close, or other applications involving
short source-receiver distances [13, 14, 15, 16, 17, 18]. A similar scheme but
intended for voice communication was introduced by [19] as an overview of
services infrastructure including multiparty immersive audio mixing and
management, as well as immersive sound rendering.

Figure 1:  Auralization scheme. DAW: Digital audio workstation, BRIR: Binaural
room Impulse Response, TF meas.: Transfer function measurement.

This paper aims to present a real-time system for auralizing musical
performance on a virtual stage by a single musician or an interactive ensemble
between a group of musicians. The main contribution of this paper is outlining
an elaborate framework for interactions between an arbitrary number of
musicians on a virtual stage, which is limited only by availability of
computing resources and electroacoustic hardware. This includes rigorous
considerations of system calibration and latency, which is currently lacking
in comparable systems for real-time auralization in literature (e.g., [15].) A
proof of concept with two interconnected musicians on a virtual stage with
several acoustic variations is also presented. The system is primarily
intended for studying stage acoustics in virtual settings wherein it is easier
to vary the acoustics than in real rooms. However, another potential uses
include enabling rehearsal/performance on virtual stages.

Fig. 1 shows the workflow used in this paper for realizing a virtual acoustics
scenario. The first step is the experimental design for stage acoustics
experiments or for another related purpose (e.g. rehearsal session, recording
session, training session, etc). This design includes the definition of both
real and virtual rooms including number, locations, and orientations of both
real and virtual sources and receivers as well as microphones and headphones.
Once the experimental design is set, the second block of activities is the
laboratory setup which includes both a physical setup and a digital setup. The
physical setup involves placing and connecting all the equipment. The digital
setup includes both the modeling (or measurement) of the binaural room impulse
responses (BRIRs) for each virtual source-receiver combination and the setup
of a Digital Audio Workstation (DAW).

The third block includes the measurements of latencies, microphone
sensitivities, and headphones transfer functions, shown in steps 3.a, 3.b, and
3.c, respectively. The step 3c / 4.a involving measurement of headphone
sensitivity can be considered as part of both blocks 3 and 4 because it is
both a calibration measurement and a part of the activities with participants.
The fourth block is called ”activities with participants” and its main part is
the listening/playing tests labeled 4.c (Run test). Before running the test,
the calibration and latency compensation is applied in step 4.b. Although the
step 4.b does not strictly require the participants to be present, it is
grouped in block 4 because the adaption can be carried out quickly while
participants remain in the laboratory after step 4.a, which may include time
for briefing and completing some general questions forms, as was done in the
current case. As part of step 4.b, the adapted BRIRs are uploaded to the
previously set up DAW session. Finally, step 4.c consists of the main test.

The rest of the paper is organized as follows. Sec. 2 introduces the
experimental design (block 1) as well as the laboratory setup (block 2) and
the BRIR adaption (block 4b). Sec. 3 describes the required adaption of the
BRIRs in terms of block 3 (namely latency compensation and calibration
filters), and also an analysis of the main sources of uncertainty. Finally, in
Sec. 4, a proof of concept is carried out with six guitar duets, followed by
the discussion and conclusions in Secs. 5 and 6, respectively.

##  2 Setup

The goal here is to simulate a virtual stage environment for a solo musician,
or for two or more musicians playing simultaneously while each of them is
located in an individual isolated booth. To that end, each musician hears both
the instrument of the other musicians and his or her own instrument in a
simulated room in real-time. For simplicity, each musician is defined with a
listener and a player role. Furthermore, each musician is assumed to play a
musical instrument that can be considered as one source (Fig. 2).

Figure 2: Example of an experimental design scheme. $s_{\text{i},m}$: input
signal from player role $m$, $s_{\text{o},n}$: output signal for listener role
$n$, $h_{m,n}$: impulse response for listener $n$ due to player $m$,
$m\in\\{1,2\\}$, $n\in\\{1,2\\}$

Fig. 2 shows an example of the experimental setup (see block 2 in Fig. 1) for
a system with two players and two listeners, i.e. a system for two musicians
in which each of them play one sound source. The upper part represents the
virtual situation in which a guitarist and a cellist play music in a simulated
concert hall. The lower part shows the electroacoustic setup (e.g. headphones,
microphones, and processing blocks) where the core parts are the DAW mixes
which perform the real-time convolutions and the signal summations.

Let $n\in\\{1,\cdots,N\\}$ denote the $n$th listener and
$m\in\\{1,\cdots,M\\}$ denote the $m$th player. Hence, based on the digital
input signal $\textbf{s}_{\text{i},m}$ from each player $m$ and the binaural
room impulse response $\textbf{h}_{m,n}$ at the listener $n$ due to each
player $m$, the output signal $\textbf{s}_{\text{o},n}$ to be sent to the
listener $n$ when $M$ players are simultaneously playing is computed as

$\textbf{s}_{\text{o},n}=\textbf{h}_{1,n}\ast\textbf{s}_{\text{i},1}+\cdots+\textbf{h}_{m,n}\ast\textbf{s}_{\text{i},m}+\cdots+\textbf{h}_{M,n}\ast\textbf{s}_{\text{i},M}$
(1)

where $\ast$ is the convolution operation. This equation can be
straightforwardly implemented in a DAW session. For example, the output signal
$\textbf{s}_{\text{o},n}$ for each listener role can be generated by routing
$M$ stereo subgroups to a mix channel $n$. Each subgroup $(m,n)$ contains a
convolution plug-in which applies the convolution of each of the two channels
of $\textbf{h}_{m,n}$ with the monaural signal $\textbf{s}_{\text{i},m}$
routed from the source input channel $m$. Setting up a system for implementing
this equation is step 2.b in Fig. 1.

The system should be calibrated and latency-compensated in order to avoid the
effect of the system itself in the signal that arrives at the musicians’ ears.
In this paper, the calibration $\textbf{k}_{m,n}$ and the latency-compensation
due to both the audio system latency $t_{\text{l}}$, and the physical source-
microphone distance $d_{\text{M-S}}$ are applied to the simulated BRIRs
$\textbf{h}^{\prime}_{m,n}$ as

$\textbf{h}_{m,n}=\textbf{h}^{\prime}_{m,n}\ast\operatorname{sinc}(\textbf{t}_{\text{s}}-t_{\text{l}}-\frac{{d}_{\text{M-S},m}}{c})\ast\textbf{k}_{m,n}$
(2)

where $\textbf{t}_{\text{s}}=[0,1/f_{\text{s}},\cdots,T]$ is the time vector
with sampling frequency $f_{\text{s}}$ and $c$ is the speed of sound. The
calibration filter $\textbf{k}_{m,n}$, the latency compensation $t_{\text{l}}$
and distance compensation $d_{\text{M-S}}/c$ are analyzed in detail in the
next section. This equation is applied in step 4.b in Fig. 1. Although the
result of eq. (2) is required in eq. (1), it is simpler to first set up the
implementation of eq. (1) in a DAW and then just adapting the BRIRs
$\textbf{h}_{m,n}$ for each participating musician according to eq. (2). In
most cases the experimental design involves fixed gains of the preamplifiers
and the distance between microphone and musical instrument; hence, the only
element of the calibration $\textbf{k}_{m,n}$ that is modified for each
participant is the headphone transfer function.

Due to latency issues (Sec. 3.1), the direct sound is skipped in the impulse
response $\textbf{h}_{m,n}$ with $m=n$ (listener $n$ receives the direct sound
from his or her own instrument), and is included in the impulse response
$\textbf{h}_{m,n}$ with $m\neq n$ (because listener $n$ is not in the same
room than player $m$). For example, in Fig. 2, the direct sound of cello is
skipped in $\textbf{h}_{1,1}$ and included in $\textbf{h}_{2,1}$.

In the example of Fig. 2, the guitarist is the listener $n=1$ and hears the
signal
$\textbf{s}_{\text{o},1}=\textbf{h}_{1,1}\ast\textbf{s}_{\text{i},1}+\textbf{h}_{2,1}\ast\textbf{s}_{\text{i},2}$
which is processed in real time in the mix 1 of the DAW according to Eq. (1).
For that reason, the signals $\textbf{s}_{\text{i},1}$ and
$\textbf{s}_{\text{i},2}$ are picked using microphones in front of each
instrument for the guitar and the cello, respectively. Simultaneously, the mix
2 of the DAW renders audio for the cellist who represents the listener
$n~{}=~{}2$. For that purpose, the mix 2 of the DAW implements Eq. (1) by
processing the same $\textbf{s}_{\text{i},1}$ and $\textbf{s}_{\text{i},2}$
signals, but this time convolved with $\textbf{h}_{1,2}$ and
$\textbf{h}_{2,2}$, respectively (i.e.
${{\textbf{s}_{\text{o},2}=\textbf{h}_{1,2}\ast\textbf{s}_{\text{i},1}+\textbf{h}_{2,2}\ast\textbf{s}_{\text{i},2}}}$).

The considered calibration accounts for both conversions from acoustic signals
to digital signals and from digital signals to acoustic signals. Sec. 3.2
presents a method for estimating the calibration filters and the analysis of
the effects of errors in the input data for this estimation.

##  3 Latency compensation and system calibration

The calibration of the virtual acoustic system can be calculated as the
frequency-dependent gain for sending the signal at the microphone in front of
player $m$ to the headphones of listener $n$. This represent the signal that
would be received at the entrance of his or her ear canal in a real situation
in free field without the headphones. The calibration filter is defined in the
frequency domain of the signals as $K_{m,n}(f)$.

Fig. 3 shows a scheme of the proposed situation for deducing the calibration
filters in a hypothetical anechoic environment. The left part in Fig. 3 shows
the hypothetical real situation which represents the playing situation with
$N=2$ listeners, but isolated in an anechoic environment. In this example,
just $M=1$ player plays a guitar, so the input signal is just $s_{\text{i},1}$
picked at a microphone placed in front of the guitar. The right part in Fig. 3
shows the auralization situation.

Figure 3: Scheme of calibration factors required for real-time auralization.
Left part: player role. right part: listener roles. $p_{\text{ec},n}$ sound
pressure at the entrance of the ear canal of listener $n$,
${p}_{\text{E,ff},n}$: free-field sound pressure at the position of the head
of listener $n$, $d_{\text{M-S},m}$ distance from the acoustic center of
source $m$ to microphone $m$, $d_{\text{E-S},m,n}$ distance from the acoustic
center of source $m$ to the head of player $n$, ${p}_{\text{M},m}$: sound
pressure at a microphone in front of player $m$, $s_{\text{i},m}$: input
signal from player role $m$, $s_{\text{o},n}$: output signal for listener role
$n$, $m\in\\{1\\}$, $n\in\\{1,2\\}$

Let $\hat{s}(f)$, $\hat{p}(f)$, and $\hat{h}(f)$ be the elements of frequency
$f$ of the Fourier transforms of digital signals s, sound pressure signals p,
and impulse responses h, respectively. In the case of Fig. 3 the right side of
Eq. (1) keeps just one term corresponding to the only musical instrument in
the proposed situation. The element of frequency $f$ of the Fourier transform
of Eq. (1) is

$\hat{s}_{\text{o},n}(f)=\hat{h}_{\text{ane},m,n}(f)\hat{s}_{\text{i},m}(f),$
(3)

with latency compensated and frequency calibrated impulse response

$\hat{h}_{\text{ane},m,n}(f)=\hat{h}^{\prime}_{\text{ane},m,n}(f)K_{m,n}(f)e^{-j\omega(t_{\text{l}}+\frac{\tilde{d}_{\text{M-S},m}}{c})}$

as an alternative of Eq. (2). The notation below will drop $f$ for brevity.

###  3.1 Latency

The latency $t_{\text{l}}$ of off-the-shelf sound interfaces is usually
greater than the wave travel-time for listening to a musical instrument played
by oneself (e.g. shorter than 0.2 ms for a violinist or about 1.5 ms for a
guitarist; some exceptions include pipe-organs, bagpipes, and some percussion
instruments).

This is demonstrated in Table 1, which shows the latency of three interfaces
measured using the ITA Toolbox [20], and where each interface was set to its
smallest buffer size option.

Table 1: Measured latency of RME Fireface UC, FOCUSRITE Scarlett 2i2, and M-AUDIO Fast Track Ultra interfaces  Interface |  $S_{\text{B}}$ (samples) |  $t_{\text{l}}$ (ms) |  $e_{\mathrm{d}}$ (m)
---|---|---|---
Fireface UC | 48 | 4 | 1
Scarlett 2i2 | 48 | 13 | 4
Fast Track Ultra | 256 | 17 | 6

To overcome the hardware latency issue for simulating hearing one’s own
instrument, one approach includes latency compensation, i.e. time shifting the
impulse response. This may be applied without cropping the impulse response
for source-receiver distances no closer than an equivalent distance
$e_{\mathrm{d}}=c\times t_{\text{l}}$. Another approach, which is also used in
previous systems (e.g., [13, 14]), includes skipping the simulation of the
direct sound for the hearing one’s own instrument. This approach involves
allowing the actual direct sound to reach the musician through open
headphones.

Hence, once the direct sound for hearing oneself is skipped, the compensation
time limit is the propagation time due to distances between musicians for
hearing others, or due to floor reflection path for hearing oneself. For the
current implementation, the RME Fireface UC interface meets both these limits:
for distances greater than 2 m between musicians and for receiver heights
greater than 1.3 m (these estimations include a compensation for a microphone-
source distance of $d_{\text{M-S}}=1$ m), and represents a suitable off-the-
shelf interface (Table 1).

###  3.2 Calibration

During auralization of the proposed situation (Fig. 3) with headphones,
listeners receive the same sound pressure at the ear canal $p_{\text{ec},n}$
that they would receive in an anechoic environment due to her or his own
musical instrument (as in the right-upper part of Fig. 3) or the instrument of
another musician (as in the right-lower part of Fig. 3). To that end, the
output signal $s_{\text{o},n}$ should contain the required calibration to
yield a sound pressure $p_{\text{ec},n}$ at the entrance of the ear canal of
each musician.

####  3.2.1 Filter components

In this section, the filter components of Eq. (3) are discussed and estimates
of the filter functions are derived. For this purpose, estimates of the sound
pressure at the microphone in front of each player and at the ears of each
listener are defined first.

The sound pressure at the microphone $p_{\text{M},m}$ can be obtained from the
input signal $s_{\text{i},m}$ as

$\hat{p}_{\text{M},m}=\frac{\hat{s}_{\text{i},m}}{S_{\text{M},m}}$ (4)

where $S_{\text{M},m}$ is the digital sensitivity of the recording system
measured as the ratio of the digital signal caused by 1 Pa of sound pressure
at the microphone to the digital full scale of the system (i.e. it includes
microphone, amplifier, ADC). Fig. 4 shows the digital sensitivity of the
recording system consisting of an Oktava MK-012 microphone with an RME
FIREFACE UC interface which is used below for a proof of concept (section IV).
It was measured by comparison to a calibrated class 1 measurement microphone
in the center of a medium-sized room with reverberation time below $0.1\
\text{s}$. Reflections were croped with a 15-ms-length rectangular time-window
centered in the direct sound as proposed in [21].

Figure 4: Recording system sensitivity $S_{\text{M},m}$ (dB) of Oktava MK-012
microphone with RME FIREFACE UC interface.

Let $W_{m}$ be the sound power for source $m$, $Q(\Omega)$ be the directivity
factor for direction $\Omega$, and $\Omega_{\text{M-S},m}$ be the
microphone–source direction for source $m$. Then, the free-field squared sound
pressure $\hat{p}^{2}_{\text{M}}$ measured by a microphone located at a
distance $d_{\text{M-S},m}$ from the acoustic center of source $m$ is

$\hat{p}^{2}_{\text{M},m}=\frac{\rho cW_{m}Q(\Omega_{\text{M-S},m})}{4\pi
d^{2}_{\text{M-S},m}}e^{2\pi f\frac{d_{\text{M-S},m}}{c}}$ (5)

where $\rho$ is the air density and $c$ is the speed of sound. Note that the
location of the acoustic center of the source depends on frequency [22].
Hence, $W_{m}$, $Q(\Omega_{\text{M-S},m})$, and also $d_{\text{M-S},m}$ are
functions of frequency.

Let $\Omega_{\text{E-S},m,n}$ be the direction and $d_{\text{E-S},m,n}$ the
distance from source $m$ to listener $n$. Then, the free-field squared sound
pressure at the position of the head of listener $n$ but in absence of the
listener is

$\hat{p}^{2}_{\text{E,ff},n}=\frac{\rho cW_{m}Q(\Omega_{\text{E-S},m,n})}{4\pi
d^{2}_{\text{E-S},m,n}}$ (6)

On the one hand, the nominal sound pressure at the blocked entrance of the
ear-canal $\hat{p}_{\text{ec},n}$ is

$\hat{p}_{\text{ec},n}=\hat{p}_{\text{E,ff,n}}H_{\text{S},m,n}$ (7)

where $H_{\text{S},m,n}$ is the head related transfer function (HRTF) of
listener $n$ in the direction to the source $m$. On the other, the actual
sound pressure at the blocked entrance of the ear canal
$\hat{p}_{\text{ec},n}$ can be obtained from an output signal
$\hat{s}_{\text{o},n}$ as

$\hat{p}_{\text{ec},n}=\hat{s}_{\text{o},n}H_{\text{E},n}e^{j2\pi
t_{\text{l}}}$ (8)

where $H_{\text{E},n}$ is the headphone transfer function (HpTF) of the
playback system for the listener $n$ measured in terms of sound pressure with
reference to the digital full scale (i.e. it includes the response of
headphones, amplifier, and DAC). Fig. 5 shows the modulus in decibels of the
HpTF of the playback system consisting of a LAMBDA STAX headphone with an RME
FIREFACE UC interface. These measurements were carried out with a dummy-head
with ten repetitions in order to estimate the deviations related to headphones
placement. Besides, these results are congruent with results previously
obtained by [23] in blocked ear canals of humans.

Figure 5: Headphone transfer function $H_{\text{E},n}$ (dB) of LAMBDA STAX
headphones with RME FIREFACE UC interface.

The calibration is based on a situation of direct sound in free field because
it avoids propagating errors from estimations of the acoustic properties of
room surfaces which would be the case for calibration based on reflections.
Although, the actual direct sound for hearing oneself passes through the open
headphone during the auralizations, the calibration is valid for the simulated
impulse response that skips direct sound. Continuing with headphones
properties, the sound insulation is important when skipping the direct sound
of simulations because the actual direct sound is expected to reach ears
through them (Subsec. 4.2).

Let $\ell=e^{-2\pi f({(}d_{\text{M-s},m}/{c}+t_{\text{l}})}$ be the latency
compensation. Then, substituting Eqs. (4), (5), (6), and (7) in (8) leads to

$\hat{s}_{\text{o},n}=\frac{d_{\text{M-S},m}\sqrt{Q(\Omega_{\text{E-S},m,n})}}{H_{\text{E},n}S_{\text{M},m}d_{\text{E-S},m,n}\sqrt{Q(\Omega_{\text{M-S},m})}}H_{\text{S},m,n}\ell\hat{s}_{\text{i},m}$
(9)

where directivity factor $Q$ is referred to the mean squared pressure
$\overline{p^{2}}$ across directions $\Omega$ over a spherical surface
centered at the source, i.e.

$Q(\Omega)=\frac{p^{2}(\Omega)}{\overline{p^{2}}}$ (10)

However, the directional factor

$\Gamma(\Omega)=\frac{p(\Omega)}{p(\Omega_{\text{ref}})}$ (11)

is usually preferred as input-data for geometrical acoustics (GA) software.
Hence, substituting eq. (10) and eq. (11) in eq. (9), yields to

$\hat{s}_{\text{o},n}=\frac{d_{\text{M-S},m}\Gamma(\Omega_{\text{E-S},m,n})}{H_{\text{E},n}S_{\text{M},m}d_{\text{E-S},m,n}\Gamma(\Omega_{\text{M-S},m})}H_{\text{S},m,n}\ell\hat{s}_{\text{i},m}$
(12)

Furthermore, since the reference direction coincides with the position of the
microphone in front of the source, the respective sound pressures are
equivalent, i.e. $p(\Omega_{\text{M-S},m})=p(\Omega_{\text{ref}})$. Thus, the
directional factor in the source-microphone direction is unity, i.e.
$\Gamma(\Omega_{\text{M-S},m})=1$.

The distance parameters $d_{\text{M-S},m}$ and $d_{\text{E-S},m,n}$ should be
measured from the acoustical center of the source $m$ to the acoustical center
of the microphone in front of that source and to the head of listener $n$,
respectively. Hence, all the parameters in Eq. (12) are frequency-dependent,
because the acoustical center of a complex source such as a musical instrument
is different for each frequency. However, source-receiver distance is usually
a single value parameter in the state-of-the-art GA simulators [24] which can
be used to estimate the impulse responses $h_{m,n}$. In order to have an
estimate of this effect in the calibration, in this work it is assumed

$d_{\text{M-S},m}/d_{\text{E-S},m,n}=\tilde{d}_{\text{M-S},m}/\tilde{d}_{\text{E-S},m,n}E_{\text{K},m,n}$
(13)

where $\tilde{d}_{\text{M-S},m}$ and $\tilde{d}_{\text{E-S},m,n}$ are single
value estimations of $d_{\text{M-S},m}$ and $d_{\text{E-S},m,n}$,
respectively. Besides, $E_{\text{K},m,n}$ is introduced as the error due to
these two estimations (subsec. 3.2.2 shows a detailed description of the
effect of this source of error on the calibration).

Eq. (12) can be written as Eq. (3) (i.e.
$\hat{s}_{\text{o},n}=\hat{h}_{\text{ane},m,n}\hat{s}_{\text{i},n}$ and
$\hat{h}_{\text{ane},m,n}=\hat{h}^{\prime}_{\text{ane},m,n}K_{m,n}\ell$) with

$K_{m,n}=\frac{\tilde{d}_{\text{M-S},m}E_{\text{K},m,n}}{H_{\text{E},n}S_{\text{M},m}\Gamma(\Omega_{\text{M-S},m})}$
(14)

and

$\hat{h}^{\prime}_{\text{ane},m,n}=\frac{\Gamma(\Omega_{\text{E-S},m,n})H_{\text{S},m,n}}{\tilde{d}_{\text{E-S},m,n}}$
(15)

Note that Eq. (14) can be used for estimating the calibration $K_{m,n}$ for
hearing oneself (i.e. $m=n$) as well as for hearing others (i.e. $m\neq n$).
Besides, the calibration for the same listener $n$ for each source $m$ would
modify the values of all parameters in Eq. (14) except $H_{\text{E},n}$ that
holds the same for hearing oneself as well as for hearing others.

In auralizations, Eq. (3) also applies but the free-field transfer function
$\hat{h}_{\text{ane},m,n}$ which was assumed for deducing the calibration data
is replaced by the room transfer function $\hat{h}_{m,n}$. Hence, the head
related transfer function $H_{\text{S}}$, the source-head distance
$\tilde{d}_{\text{E-S}}$, and source directional factor
$\Gamma(\Omega_{\text{E-S}})$ may be applied to the binaural room impulse
responses (BRIRs) which is the state-of-the-art of GA simulators. In this
article the BRIRs are calculated with RAVEN [25] and the input data include
the source directivity data reported in [26].

####  3.2.2 The effect of source-receiver distance

The calibrated output signal depends on two distances measured from the same
source position as shown above in Eq. (12). One of these distances is measured
to the head of a listener (i.e. $d_{\text{E-S}}$) and the other one to the
center of a microphone (i.e. $d_{\text{M-S}}$).

Let $\tilde{d}_{\text{M-S}}$ and $\tilde{d}_{\text{E-S}}$ be estimations of
$d_{\text{M-S}}$ and $d_{\text{E-S}}$, respectively. These estimations – which
assume the acoustic center is placed in a fixed point for all frequencies –
can be the geometric center, the mean of the acoustic center for all
frequencies in which it was aligned for the directivity database in [27], or
any other suitable approximation. Then, the errors $e_{1}$ and $e_{2}$
indicate the estimation error for ${d}_{\text{M-S}}$ and ${d}_{\text{E-S}}$,
respectively, and the ratio of this two distances is

$\frac{d_{\text{M-S}}}{d_{\text{E-S}}}=\frac{\tilde{d}_{\text{M-S}}+e_{1}}{\tilde{d}_{\text{E-S}}+e_{2}}$
(16)

In order to study the effect of the distance estimation error on the
calibration $K_{m,n}$, eq. (13) was proposed. Hence, inserting eq. (13) in eq.
(16) gives the effect of the estimation error

$E_{\text{K}}=\frac{1+e_{1}/\tilde{d}_{\text{M-S}}}{1+e_{2}/\tilde{d}_{\text{E-S}}}$
(17)

Fig. 6 shows results of equation (17) as contour lines in dB units related to
distance relative errors $R_{\mathrm{e,1}}=e_{1}/\tilde{d}_{\text{M-S}}$ and
$R_{\mathrm{e,2}}=e_{2}/\tilde{d}_{\text{E-S}}$, .

Figure 6: Effect of distances estimation error in dB units ($E_{\text{K}}$) on
calibration

The errors $e_{1}$ and $e_{2}$ for each frequency depend on the position of
the acoustic center of the source and do not depend on the position of the
receiver111Actually the acoustic center of the microphone can vary by about a
few centimeters, but it is assumed negligible compared to the variation of the
acoustic center of the source in this article for a given source-receiver
direction. Therefore the relative errors $R_{\mathrm{e,1}}$ and
$R_{\mathrm{e,2}}$ are smaller for larger source-receiver distances
($\tilde{d}_{\text{M-S}}$ and $\tilde{d}_{\text{E-S}}$). In other words, the
effect of these relative errors in the calibration estimated with single value
distances can be smaller for receivers in the audience than for receivers on
the stage, because the former are closer to the sources than the latter.

In the case of a violin, the acoustic center for low frequencies can be about
15 cm apart from the acoustic center for high frequencies, as reported in
[28]. Considering a distance of about 1 m this could yield a relative error
$R_{\mathrm{e,1}}\approx 15\%$.

The relative error $|R_{\mathrm{e,1}}|$ can reach about $15\%$ for a cello
player hearing himself or herself; thus the magnitude of the effect of the
distance estimation error $E_{\text{K}}$ can reach values below $2.6$ dB for
$|R_{\mathrm{e,2}}|\leq|R_{\mathrm{e,1}}|$. For singers, woodwind instruments,
brass instruments, and other instruments in which the source is very close to
the ears, the $|R_{\mathrm{e,2}}|$ error is likely greater. Hence, the proof
of concepts in this paper is carried out with acoustic guitars that are
usually played at a considerable distance from the ears.

For a given relative error $R_{\mathrm{e,1}}$, the effect is null when
$R_{\mathrm{e,2}}=~{}R_{\mathrm{e,1}}$; it can be the case that both receivers
are at the same point or a geometric coincidence which are both unlikely to
occur in practice. This is shown in the curve for 0 dB effect in Fig. 6.
Furthermore, for $R_{\mathrm{e,1}}>R_{\mathrm{e,2}}$ the effect is positive in
magnitude, whereas, for $R_{\mathrm{e,1}}<R_{\mathrm{e,2}}$ the effect is
negative in magnitude.

####  3.2.3 Source directivity approach

The directivity of musical instruments occurs due to their complex physical
systems which involve a large number of variables. However, the state-of-the
art assumes fractional octave bands as a simplified representation of the
radiation patterns because it reduces computational costs of simulations.
Nevertheless, this simplification may not be totally accurate for some
partials in some musical instruments [29, 30, 31, 32]. Besides, some partials
of the same frequency may have strong dissimilarities for certain instruments.
For instance, Fig. 7 shows balloon plots of a clarinet for the same $440\
\text{Hz}$ frequency due to the fundamental tone of an A4 note in the upper
register and due to the first harmonic of a A3 note in the lower register.

Figure 7: Comparison of balloon plots for a partial tone generated with the
low register of a clarinet and a fundamental tone with almost the same
frequency but played in the high register. Legend in dB scale (data extracted
from [26]).

The input data representing directivity can be extracted from several
available databases [33, 34, 29, 35, 31, 26, 36]. In this article the
directivity database by [26] is chosen because of its high resolution and its
centering processing. This database provides up to 4-th order spherical
harmonics coefficients for 41 musical instruments of different historical
periods considering up to 10-th order partials for each note in a chromatic
scale covering the typical range of each instrument. Then, the directivity for
each band is estimated by averaging the directional index of all the partials
that fall in that band for that instrument [32, sec. II-C].

The directivity for the same frequency and two different partials can be both
as similar as about $0.1$ dB for certain directions and as different as about
$40\ \text{dB}$ for certain other directions However, it is shown below that
the relative likelihood for such huge discrepancies is small compared to that
for similarities.

Fig. 8 shows the probability density function (PDF) of the estimation errors
of the directivity for each third octave frequency band for a guitar [32].
These estimation errors are defined in each direction as the level differences
between the actual directional index for each partial and the average of the
the directional index for the corresponding third octave band. Then, the
errors are evaluated at $2,592$ directions over a grid of azimuth and
elevation with a 5° resolution. Finally, the PDFs are estimated for each
frequency band with the kernel method222Gaussian 0.1 dB Kernel on the errors
for each of the partials that fall in that band.

Figure 8: Estimated probability density functions of directional factor
estimation errors for a guitar. The dashed area shows the percentile 25 and
75, the red solid line shows the mean and the white solid line shows the
median.

The error of the $50\ \%$ of the evaluated directions is at most about 6 dB,
as shown in Fig. 8. It should be noted that the error for partials is likely
less noticeable than for fundamental tones, because the latter usually carry
more energy than higher order partials. As shown, errors are (surprisingly)
large but this is what is currently possible with the state-of-the-art
databases.

Musicians movements may also cause an effect on the radiation pattern as well
as the source position and orientation. Thus, the movements of musicians can
be incorporated as in [37] by the RAVEN’s animation module. However, in this
article the sound field is simulated just with static sources as a proof of
concept.

##  4 Proof of concept

In this section, the implementation and validation of a two-players – two-
listeners setup in an virtual-acoustics interactive system is described. To
that end, the three subsections show an analysis comparing a real measurement
with a simulation, an objective determination of the calibration functions,
and a subjective evaluation of the system, respectively.

###  4.1 Simulation of a measured setup

In this subsection a comparison of simulation versus measurement of a setup
for one musician hearing oneself is documented. Therefore, a real room with a
dummy head and a loudspeaker was both measured and simulated. The room is
about $6\ \text{m}$ wide, $8\ \text{m}$ long, and $3\ \text{m}$ high; the
inner surfaces are acoustically hard with mean absorption coefficient of
$0.06$ [[, See Scene 9 in]]BRAS2019. Fig. 9 shows the room geometry.

Figure 9: Room geometry

A binaural room impulse response (BRIR) was measured with a dummy head using a
directional loudspeaker (Genelec 8020c) as a source placed near the dummy
head. Fig. 10 shows a photo of the measurement setup inside the empty room,
i.e. the only objects in the room were the dummy head and the loudspeaker; the
sound interface and the computer were placed outside the room.

Figure 10: Measurement setup for binaural room impulse response using a
Genelec 8020c loudspeaker and a dummy head in a $146\text{ m}^{3}$ room

The probability density functions of directivity estimation errors for the
Genelec 8020c are shown in Fig. 11. The source data consisting of the
radiation patterns in a 1-degree resolution spherical grid are available in
[38]. The analysis was performed as in [32], by averaging the radiation
pattern in frequency bands and comparing the resulting value with the
radiation pattern for each frequency within each band.

Figure 11: Estimated probability density functions of directivity’s estimation
errors of a Genelec 8020c. The dashed area shows the percentile 25 and 75, the
red solid line shows the mean and the white solid line shows the median.

About 50 % of the errors are within the $\pm 6\ \textrm{dB}$ range for each
1/3 octave band below 4 kHz. This error range increases with frequency likely
due to the higher variation of directivity within wider 1/3 octave bands at
higher frequencies.

Fig. 12 shows the envelope of the measured BRIR. It was measured and processed
with the ITA-toolbox [39, 20]. The envelope was estimated with the Hilbert
transform as in [40] because it provides a good descriptor for analyzing the
texture of room impulse responses.

Figure 12: Measured binaural room impulse response. Envelope estimated with
Hilbert transform at mid the frequencies range from 353 Hz to 2828 Hz (i.e.
from 500 Hz to 2 kHz Hz octave bands)

The same setup was also simulated using GA modelling using RAVEN [25]. The
measured HRTF of the same dummy head and the measured directivity of the same
loudspeaker (Genelec 8020c) were included as RAVEN input data. Then, the
resulting BRIR was convolved with the loudspeaker impulse response because the
measured directivity of the loudspeakers are normalized to the radiation in
the axial direction. Fig. 13 shows envelope of the simulated BRIR.

Figure 13: Simulated binaural room impulse response. Envelope estimated with
Hilbert transform at mid the frequencies range from 353 Hz to 2828 Hz (i.e.
from 500 Hz to 2 kHz Hz octave bands)

Comparing the results in Figs. 12 and 13, the measurement shows a direct sound
at $\Delta t\approx 2\ \text{ms}$ that is not present in the simulation
because the direct sound is excluded in the hearing oneself case. Then, the
first reflection is from the floor for both the measurement and the
simulation. Furthermore, the energy of the subsequent early reflections in the
simulation are more-or-less similar to that of the measurement. However, these
subsequent reflections are not as similar as for the first reflection, which
occurs due to the mentioned calibration uncertainties, simulation data inputs,
and simulation modeling. The modeling of the sound diffusion is achieved by
RAVEN as a gradual mix of the ray tracing model with the image source model.
Hence, the sound energy is similarly distributed in a statistical sense across
short time intervals.

The technique for the hearing-oneself case is to keep direct sound as in the
measured case (Fig. 12) and replace the following sound energy with the
simulation excluding the direct sound (Fig. 13). This is carried out by
placing each musician in a test room with open headphones that allow the
direct sound of her or his own instrument to pass through the cups of the
headphones and playback the sound processed in real time to simulate the
reflections over the headphones. The following two subsections explore this
technique.

###  4.2 Objective evaluation

In this subsection, the calibration is estimated and the effect of headphones
attenuation on real direct sound are shown for a general case. This case
considers the dummy head and the loudspeaker in Fig. 10 placed in a free field
environment for emulating a virtual environment for a soloist musician ($M=1$)
in hearing oneself configuration ($N=1$).

Fig. 14 shows the modulus in decibels of the calibration obtained for this
configuration. Besides, it includes $S_{\text{M},m}$ from Fig. 4 and
$H_{\text{E},n}$ from Fig. 5 as references.

Figure 14: Calibration for a simulated guitarist in a hearing oneself configuration Table 2: Configuration of the scenarios and relevant acoustic parameters  Scenario | $\text{R}_{1}$ | $\text{R}_{2}$ | A | B | C | D | M
---|---|---|---|---|---|---|---
Stage height (m) | 4 to 9 | – | 9 | 18 | 9 | 18 | 13.5
Stage width (m) | 12 | 20 | 20 | 20 | 12 | 12 | 16
Reverberation time (s) | 0.9 | – | 1.0 | 1.1 | 0.9 | 1.0 | 1.0
Early stage support (${ST}_{\text{E}}$; dB) | -12.4 | – | -13.6 | -16.6 | -11.8 | -14.2 | -14.4
Late stage support (${ST}_{\text{L}}$; dB) | -18.1 | – | -16.0 | -16.2 | -15.7 | -16.0 | -16.6

The calibration is carried out by eq. (14) and assuming an error
$E_{\text{K},m,n}$ (i.e. replacing $E_{\text{K},m,n}=1$). The microphones used
for estimating $H_{\text{E},n}$ and $S_{\text{M},m}$ in eq. (14) (Sennheiser
KE4 and Oktava MK-012, respectively) were calibrated by comparison to a
calibrated class 1 measurement microphone in the center of a medium sized room
with a reverberation time below $0.2\ \text{s}$ [41, see room called Virtual
laboratory in Sec. 2.3]).

The actual direct sound arrives from outside the headphones and just the
reflections are simulated through the loudspeakers of the headphones. Hence,
the attenuation of the headphones is assumed as an error in representing the
direct sound for the hearing oneself case (i.e. it does not apply when
listener $n$ hears musician $m\neq n$). Fig. 15 reports the sound attenuation
measured for LAMBDA STAX and Sennheiser HD 650 with a dummy-head and a
loudspeaker in twelve equidistantly distributed positions in a sphere of 2.5 m
radius. LAMBDA STAX headphones were used for the following proof of concepts
as they involve lesser attenuation of the direct sound.

Figure 15: Attenuation of LAMBDA STAX and Sennheiser HD 650

###  4.3 Subjective evaluation

In order to provide subjective support, an experiment with six actual guitar
duets was conducted. Each duet interacted over several simulated scenarios.
Per scenario, they also evaluated the overall quality, and the similarity to
playing in a real room.

The twelve guitarists’ age was 30.4 years in average (4.8 years standard
deviation) with self-reported normal hearing. Although only one of them is a
professional musician, every guitarist reported some experience with guitar
ensemble either during the past up to ten years or during the current year.
Musicians received a monetary compensation.

Each duet performed in eight different scenarios, whose main characteristics
are shown in Fig. 16. The eight scenarios have the same audience plan
dimensions ($25\ \text{m}\times 20\ \text{m}$) and height (16 m). The width
and height of stages are shown in Tab. 2.

(a) Floor plan scheme

(b) Longitudinal section scheme

Figure 16: Scheme of the virtual scenario. Solid line: bigger room with high
level of height and width. Red dotted line: low level width. Black dash-dotted
line: low level height. Blue long dashed line: tight-shell

The first two scenarios ($\text{R}_{1}$ and $\text{R}_{2}$) were provided for
the musicians to get familiar with the experiment. The first scenario
$\text{R}_{1}$ had a tight stage shell (see blue long dashed line in Fig. 16)
which provides acoustic support (with walls 1 m apart from musicians and a
splayed ceiling) and the second one $\text{R}_{2}$ is just the stage and the
audience surfaces as in an outdoors situation (i.e. without any walls or
ceiling). This was followed by the six experimental scenarios (A, B, C, D, and
two times M) in random order. The experimental scenarios are shoebox-shaped
stages whose dimensions were set by the four possible combinations (A, B, C,
and D) of a $2^{k}$ statistical design varying two factors $k$ at two levels
plus a central point (M) with a repetition. The statistical design involves
two levels of width (9 m and 18 m, see red dotted line and solid line,
respectively in Fig. 16), two levels of height (12 m and 20 m, see black dash
dotted line and solid line, respectively in Fig. 16), and one level of depth
(11 m).

Tab. 2 also shows acoustical descriptors of the virtual scenarios. Due to
identical audience simulation (i.e., geometry and materials are the same), the
reverberation time is similar for all the scenarios except for the free-field
case $\text{R}_{\text{2}}$. Early stage support ${ST}_{\text{E}}$ and late
stage support ${ST}_{\text{L}}$ are reported in Tab. 2 [42]. They are the
average values for octave bands from 250 Hz to 2 kHz and for three source
positions, each with three receiver positions.

The scenarios’ dimensions were chosen in order to have some diversity in stage
acoustic quality as reported in previous work on stage acoustics of real rooms
[43]. Besides, $ST_{\text{E}}$ range of the scenarios is $[-11.8,-16.6]$ with
about 5 dB difference between scenarios B and C. $ST_{\text{E}}$ of scenarios
A and D are similar with about 0.6 dB difference.

The distance between guitarists was set at a constant virtual distance of 10
m, i.e., $d_{\text{S-E},{1,2}}=d_{\text{S-E},{2,1}}=10\ \text{m}$ between
musicians. This was primarily to ensure that the differences between
experimental conditions were easier to perceive. Smaller distances between
musician increase the ratio of direct to reflected field, in turn making
reflected field harder to be perceived. The latter was also confirmed during
pre-experiment explorations where for distances more typical in real ensembles
(e.g., around 2 m), it was hard to notice differences in experimental
variations. This does not imply that perceiving differences in stage acoustics
is hard in general, but just that they were relatively harder to perceive with
shorter inter-musician distances for some of the current set of experimental
conditions.

For each duet, the two musicians were located in separate rooms connected only
aurally by their instruments (i.e., with microphones and headphones
implementing eq. (1)). The microphones were Oktava MK-012 and the headphones
were STAX LAMBDA (Figs. 4, 5, and 15 show the microphone sensitivity, the
HpTF, and the headphones cups’ attenuation, respectively). Besides, the
source-microphone distance was $\hat{d}_{\text{M-S},m}=0.9\ \text{m}$ for both
guitarists $m\in\\{1,2\\}$.

Musicians were informed that they would play on a virtual stage, positioned 3
m from the edge of the stage and 10 m apart from each other. Since guitar
duets usually play with shorter inter-musician distances (roughly 2 m apart
typically), they were instructed to imagine that the 10 m distance was due to
them being part of a bigger ensemble. They were also informed on their virtual
orientations, which were quite different for each musician (Fig. 16a). Looking
from the stage to the audience, the head of the musician at the right (Mus. 1)
is turned left to the fretboard and the head of the musician at the left (Mus.
2) is directed to the audience in front and between the fretboard and the
position of the other musician333This setup for the orientation of the
receivers was decided based on recorded videos of several professional duets.
All participants were right-handed players; hence no other configuration was
used..

Participant duets were asked to play two short (about 60 s long) musical
pieces for guitar ensemble (one of them allegro and the other one adagietto or
andante) in the simulated scenarios. Although the participants communicated
between the scenarios using their voices, they were asked to base their
judgments just on guitar signals, acoustics, and ensemble playing, and not on
visuals. To that end, they also tested the scenarios with their guitars
playing single tones, chords, arpeggios, or improvising at the beginning and
the end of each scenario.

After playing in each scenario, musicians answered a questionnaire that
included items, each on a linear scale, about aspects related to the
acoustical quality. While most of the questionnaire items will be analyzed in
future work, an item asking about the ’overall quality’ (’bad acoustics’ –
’good acoustics’ at the extremes of the scale) per scenario is analyzed here.
Since distributional assumptions of parameteric tests were not met, non-
parameteric tests are used throughout.

Wilcoxon signed-rank test showed that the participants’ ratings for the two
instances of condition M were not significantly different ($Z$ = -0.83, $p$ =
.41). This indicates that participants provided similar ratings to repeated
instance of the same condition, which is at least an encouraging sign for the
subjective evaluation of the system. Henceforth, given their similar values,
the ratings for repetitions of condition M were averaged. Following this,
Friedman’s test indicated that the overall effect of experimental condition
over ratings was statistically significant $(\chi^{2}(4)=11.14,p=.025)$. Fig.
17 shows the PDFs for each condition. Finally, post-hoc tests (Wilcoxon test
with $p$–value adjustment using the FDR method) showed that none of the
conditions were statistically significant from each other (all $p>$ .5). To
summarize, while there is an overall effect of experimental condition,
pairwise or other comparisons (e.g., between different groups of heights and
widths) did not reach statistical significance. There can be many reasons for
the latter, including small sample sizes (limiting the statistical power)
which is a clear limitation of the current experiment, and the overall
significance being driven by specific group differences that are not
meaningful comparisons based on experimental design, etc. However, from the
perspective of the system evaluation, since there are meaningful trends among
conditions, the results seem promising for future testing with a larger sample
size.

Figure 17: Probability density function of the Overall quality for each
scenario (4 % Gaussian kernel)

Finally, after playing in all the proposed scenarios, a linear scale was
presented to collect responses on the similarity of playing in the proposed
scenarios to playing in a real room. An histogram of the collected responses
is shown in Fig. 18. Let the extreme values of the scale be 0 and 100,
respectively; then, the mean is $73$, the mode is $80$, and the standard
deviation is $13$.

Figure 18: Subjective judgement of the 12 participants. Black: presented
scale. Grey: histogram of collected responses

##  5 Discussion

The findings indicate that the presented real-time system for auralizing
musical performance of two or more musicians interacting in an ensemble is
feasible. Hence, the presented system substantially extends the possibilities
of multi-musician auralizations compared to existing systems for one musician
interacting in real-time with recorded signals [13, 14]. A key advantage of
the presented system is the increased interactivity: musicians can modify
their performances due to the real-time ensemble. This represents a more
dynamic and closer-to-reality playing condition compared to interacting with
recordings.

The critical latency issue for the hearing oneself configuration (about less
than 3 ms) is addressed in the presented system by avoiding the digital
simulation of direct sound. This technique, which has previously been used in
non-interactive systems (e.g., [13, 14]), allows for a larger latency of about
6 ms considering a 2 m distance between musicians.

On the one hand, some latency may not interfere with tempo of the musical
performance, although it may change the perceived room acoustics. Hence,
proper latency compensation is an important consideration in systems intended
for stage acoustics studies. On the other hand, the drawback with not
simulating the direct sound is that there will be some direct sound
attenuation due to the headphones. To minimize this effect, headphones with
relatively smaller attenuation (about 6 dB at frequencies higher than 2 kHz;
Fig. 15), compared to an alternative, were used in the subjective evaluation
of this paper. Further, in the case of a guitarist in a hearing oneself
configuration, the quickest reflection comes from the guitar-floor-ear path,
which was accurately simulated with a suitable off-the-shelf hardware
interface. It is possible that with more customized hardware, the latency
issues could be curtailed further in future implementations.

Besides the latency issue, the implications of the calibration filter were
analyzed. To that end, a comprehensive theoretical derivation of the
components of a calibration filter for the system was presented. It was
determined that the three main sources of error are: the directivity of
sources, the distances from source to the microphone to the head, and the
transfer function of headphones. Although not considered in this paper, head
movement of the guitarists may have an effect on these three sources of error
(i.e. the headphones transfer function may be slightly modified if the
headphones move during performance).

The uncertainty due to band averaging the source directivity is
comprehensively analyzed elsewhere [32]. However, it is important to note
that, as far as the authors know, no study has investigated the directivity
differences among units of the same instrument. Furthermore, each musician
played a different guitar in the experiments because musicians were encouraged
to play with their own instruments for comfort. This is a clear weakness of
this study and other similar studies. Hence, investigating this uncertainty
further is an open research topic.

The effects of the single number estimation of source-microphone and source-
head distances derived in this paper (Fig. 6 also apply for listeners in the
audience of performance spaces, conference halls, or other rooms. This effect
grows with relative errors in distances. Therefore, larger real source-
microphone and virtual source-head distances are less sensitive to this
effect.

The objective proof of concepts showed encouraging results, hence a subjective
experiment was carried out. The subjective proof showed that playing in the
simulated scenarios was judged as having a score above the middle of a scale
measuring similarity to playing in a real room (Fig. 18) by all the
participants. Furthermore, for the more experienced musicians, this rating was
closer to the maximum score.

The other weaknesses of this setup are related to the lack of visual feedback
and the lack of response to source and head movements (especially for head
movements of the guitarist whose left hand is closer to the side wall).
Improving both these weaknesses is challenging due to latency issues as
discussed in the following two paragraphs.

On the one hand, preliminary experiments carried out previous to the proof of
concepts with experienced guitarist participants showed that static visual
feedback did not improve the ratings of similarities to playing in a real room
(Fig. 18). On the other hand, latency compensation of real-time video feedback
is challenging. A latency mismatch between video and audio could introduce
further uncontrolled variables, which was avoided here by excluding visual
feedback altogether. Visual feedback can be improved by positioning musicians
in the same room or in separated rooms with glass windows, hence visually
connecting them like in recording studios. However a recording-studio-like
setup could also interfere with aural cues of distance between musicians.
Nevertheless, improving audio-visual integration represents future work.

Although not studied in this article, placing two or more musicians in the
same room may be useful to reduce the distance between them to be below 2 m.
Of course, in this case the direct sound would be skipped in the room impulse
responses for hearing others (i.e. $m\neq n$). This, however, could not be
achieved in the current implementation with musicians playing in separated
rooms. Moreover, placing musicians in the same room reduces the flexibility in
varying the distances between them across experimental scenarios, and/or in
setting virtual distances beyond the dimensions of the actual laboratory
rooms; both of these are possible within the current system.

Tracking head movements represents the current state-of-the-art technique for
auralization of prerecorded material. However, when the audio signals are
generated just 6 ms before, the tracking of movements becomes a challenging
task.

Despite these limitations, the system was able to provide a realistic and
engaging listening and playing experience. The results of this study suggest
that real-time auralization of musical performance is a promising technology
with the potential to provide experimental platform for stage acoustics
investigations as well as musical performance training.

##  6 Conclusions

An interactive and real-time system for stage acoustics experimentation is
proposed and validated. The system enables interactive listening, including
hearing oneself as well as hearing others, while playing simultaneously on a
virtual stage. The latency issues are circumvented by time compensation of the
simulated impulse responses. The main sources of uncertainty are discussed
based on state-of-the-art practices and databases. Future studies are
recommended to address knowledge gaps regarding source directivities including
their interaction with typical movements by musicians during playing. The
results from a pilot study with guitar duets show promise in terms of
feasibility of the system for duets. Moreover, the current system can be
scaled to accommodate a larger number of musicians and musical instruments.
This can be useful in enabling virtual stage acoustics studies with a large
number of musicians concurrently, which currently represents a challenge.

## Acknowledgment

The work of E. A. in Germany was supported by the Alexander von Humboldt
Foundation. M.Y. was supported by a DFG (German Research Foundation) grant -
Project number 503914237.

## References

  * [1] H. Lehnert, J. Blauert, Principles of binaural room simulation, Applied Acoustics 36 (3-4) (1992) 259–291.  doi:10.1016/0003-682X(92)90049-X.
URL https://linkinghub.elsevier.com/retrieve/pii/0003682X9290049X

  * [2] M. Kleiner, B.-I. Dalenbäck, P. Svensson, Auralization-an overview, Journal of The Audio Engineering Society 41 (1993) 861–875.
  * [3] J. Blauert, Spatial Hearing. The psychophysics of human sound localization, revised Edition, MIT Press, Camebridge, Massachusetts, 1997.
  * [4] P. Kyoungsoo, J. Seo, K. Jeunghun, C. Jun, Auralization of musical instruments in virtual halls considering source directivity, in: Proceedings - 2014 10th International Conference on Intelligent Information Hiding and Multimedia Signal Processing, IIH-MSP 2014, 2014, pp. 614–617.  doi:10.1109/IIH-MSP.2014.159.
  * [5] T. Funkhouser, I. Carlbom, G. Elko, G. Pingali, M. Sondhi, J. West, A beam tracing approach to acoustic modeling for interactive virtual environments, Computer Graphics, SIGGRAPH ‘98, 21 (1998) 21–32.
  * [6] L. Savioja, J. Huopaniemi, T. Lokki, R. Väänänen, Creating interactive virtual acoustic environments, Journal of the Audio Engineering Society 47 (9) (1999) 675–705.
  * [7] S. Shelley, D. Murphy, Openair: An interactive auralization web resource and database, in: Audio Engineering Society Convention 129. Audio Engineering Society, 2010.
  * [8] J. Ahrens, Authentic auralization of acoustic spaces based on spherical microphone array recordings, Proceedings of the Institute of Acoustics 40 (2018) 8.
  * [9] J. Llorca-Bofí, M. Vorländer, Multi-detailed 3d architectural framework for sound perception research in virtual reality, Frontiers in Built Environment 7 (2021).  doi:10.3389/fbuil.2021.687237.
  * [10] M. Vorländer, Auralization: Fundamentals of acoustics, modelling, simulation, algorithms and acoustic virtual reality, 2nd Edition, Springer, 2020.
  * [11] H. Fırat, L. Maffei, M. Masullo, 3d sound spatialization with game engines: the virtual acoustics performance of a game engine and a middleware for interactive audio design, Virtual Reality 26 (2) (2022) 539–558.  doi:10.1007/s10055-021-00589-0.
  * [12] M. Blau, A. Budnik, M. Fallahi, H. Steffens, S. Ewert, S. Van De Par, Toward realistic binaural auralizations - perceptual comparison between measurement and simulation-based auralizations and the real room for a classroom scenario, Acta Acustica 5 (2021).  doi:10.1051/aacus/2020034.
  * [13] D. Pelegrín-García, O. Fuentes-Mendizábal, J. Brunskog, C.-H. Jeong, Equal autophonic level curves under different room acoustics conditions, Journal of the Acoustical Society of America 130 (1) (2011) 228–238, 00000.
  * [14] M. Yadav, D. Cabrera, W. L. Martens, A system for simulating room acoustical environments for one’s own voice, Applied Acoustics 73 (4) (2012) 409–414.  doi:10.1016/j.apacoust.2011.10.001.
  * [15] S. V. A. Garí, M. Kob, T. Lokki, Investigations on stage acoustic preferences of solo trumpet players using virtual acoustics, in: 14th Sound and Music Computing Conference, 2017, p. 9.
  * [16] M. Yadav, D. Cabrera, Autophonic Loudness of Singers in Simulated Room Acoustic Environments, Journal of Voice 31 (3) (2017) 388.e13–388.e25.  doi:10.1016/j.jvoice.2016.09.016.
URL https://linkinghub.elsevier.com/retrieve/pii/S0892199716303216

  * [17] K. S. Kim, H. Wang, L. Max, It’s About Time: Minimizing Hardware and Software Latencies in Speech Research With Real-Time Auditory Feedback, Journal of Speech, Language, and Hearing Research 63 (8) (2020) 2522–2534.
URL http://pubs.asha.org/doi/10.1044/2020_JSLHR-19-00419

  * [18] K. Kittimathaveenan, M. Park, The inﬂuence of stage acoustics on the singers’ performance and perception: A pilot study, journal of the audio engineering society (2021) 7.
  * [19] Y. Huang, J. Chen, J. Benesty, Immersive Audio Schemes, IEEE Signal Processing Magazine 28 (1) (2011) 20–32.  doi:10.1109/MSP.2010.938754.
URL http://ieeexplore.ieee.org/document/5670620/

  * [20] M. Berzborn, R. Bomhardt, J. Klein, J.-G. Richter, M. Vorländer, The ITA-Toolbox: An Open Source MATLAB Toolbox for Acoustic Measurements and Signal Processing, in: 43th Annual German Congress on Acoustics, Kiel (Germany), 6 Mar 2017 - 9 Mar 2017, 2017.
URL http://publications.rwth-aachen.de/record/687308

  * [21] E. Accolti, F. di Sciascio, On the use of time windows for the determination of sound strength parameter g from uncalibrated room impulse responses measurements, Applied Acoustics 178 (2021) 12.  doi:10.1016/j.apacoust.2021.108023.
URL https://www.sciencedirect.com/science/article/pii/S0003682X2100116X

  * [22] I. B. Hagai, M. Pollow, M. Vorländer, B. Rafaely, Acoustic centering of source measured by surrounding spherical microphone arrays, The Journal of the Acoustical Society of America 130 (4) (2011) 2003–2015, number: 4.  doi:https://dx.doi.org/10.1121/1.3624825s.
  * [23] D. Hammershøi, H. Møller, Methods for binaural recording and reproduction, in: Acta Acustica united with Acustica, Vol. 88, 2002, pp. 303–311.
  * [24] L. Savioja, U. P. Svensson, Overview of geometrical room acoustic modeling techniques, J. Acoust. Soc. Am. 138 (2) (2015) 708–730, number: 2.  doi:https://doi.org/10.1121/1.4926438.
  * [25] D. Schröder, M. Vorländer, RAVEN: A real-time framework for the auralization of interactive virtual environments, in: Forum Acusticum, Aalborg Denmark, 2011, pp. 1541–1546.
URL https://www2.ak.tu-
berlin.de/akgroup/ak_pub/seacen/2011/Schroeder_2011b_P2_RAVEN_A_Real_Time_Framework.pdf

  * [26] S. Weinzierl, M. Vorländer, G. Behler, F. Brinkmann, H. von Coler, E. Detzner, J. Krämer, A. Lindau, M. Pollow, F. Schulz, N. R. Shabtai, A database of anechoic microphone array measurements of musical instruments, last viewed June 15, s2021 (2017).  doi:http://dx.doi.org/10.14279/depositonce-5861.2.
URL http://dx.doi.org/10.14279/depositonce-5861.2

  * [27] N. R. Shabtai, M. Vorländer, Acoustic centering of sources with high-order radiation patterns, The Journal of the Acoustical Society of America 137 (4) (2015) 1947–1961, number: 4.  doi:10.1121/1.4916594.
  * [28] N. R. Shabtai, G. Behler, M. Vorländer, Generation of a reference radiation pattern of string instruments using automatic excitation and acoustic centering, Journal of the Acoustical Society of America 138 (6) (2015) 480–486.
  * [29] F. Hohl, Kugelmikrofonarray zur Abstrahlungsvermessung von Musikinstrumenten (spherical microphone array for capturing soundradiation from musical instruments), master’s thesis, Institute of Electronic Music and Acoustics, University of Music and Performing Arts (2009).
  * [30] F. Hohl, F. Zotter, Similarity of musical instrument radiation-patterns in pitch and partial, in: 36th German Conference on Acoustics: Fortschritte der Akustik, DAGA, Berlin, 2010, p. 2.
  * [31] J. Patynen, T. Lokki, Directivities of symphony orchestra instruments, Acta Acustica united with Acustica 96 (1) (2010) 138–167.  doi:10.3813/AAA.918265.
  * [32] E. Accolti, J. Gimenez, M. Vorländer, Uncertainties of room acoustics simulation due to directivity data of musical instruments, Techrxiv. Preprint (2023) 1–12doi:10.36227/techrxiv.20858596.
URL https://doi.org/10.36227/techrxiv.20858596.v2

  * [33] F. Otondo, J. Rindel, The influence of the directivity of musical instruments in a room, Acta Acustica united with Acustica 90 (6) (2004) 1178–1184.
  * [34] B. F. G. Katz, Directivity measurements of the singing voice, in: 19th International Congress on Acoustics (ICA 2007), Madrid, Spain, 2007, p. 7.
  * [35] M. Pollow, B. Masiero, Measuring directivities of natural sound sources with a spherical microphone array, in: Ambisonics Symposium, 2009, p. 7.
  * [36] N. R. Shabtai, G. Behler, M. Vorländer, S. Weinzierl, Generation and analysis of an acoustic radiation pattern database for forty-one musical instruments, The Journal of the Acoustical Society of America 141 (2) (2017) 1246–1256.  doi:10.1121/1.4976071.
URL http://aip.scitation.org/doi/10.1121/1.4976071

  * [37] D. Ackermann, C. Böhm, F. Brinkmann, S. Weinzierl, The acoustical effect of musicians’ movements during musical performances, Acta Acustica united with Acustica 105 (2) (2019) 356–367.  doi:10.3813/AAA.919319.
  * [38] L. Aspöck, F. Brinkmann, D. Ackermann, S. Weinzierl, M. Vorländer, BRAS - Benchmark for Room Acoustical Simulation (2019).  doi:10.14279/depositonce-6726.2.
  * [39] P. Dietrich, B. Masiero, M. Müller-Trapet, M. Pollow, R. Scharrer, Matlab toolbox for the comprehension of acoustic measurement and signal processing, in: Fortschritte der Akustik – DAGA, 2010.
  * [40] T. Hidaka, N. Nishihara, Favorable reverberation time in concert halls revisited for piano and violin solos, The Journal of the Acoustical Society of America 151 (3) (2022) 2192–2206.  doi:10.1121/10.0009931.
URL https://asa.scitation.org/doi/10.1121/10.0009931

  * [41] F. Pausch, Documentation of the experimental environments and hardware used in the dissertation ”Spatial audio reproduction for hearing aid research : System design, evaluation and application”, Tech. rep., RWTH Aachen University (2022).  doi:10.18154/RWTH-2022-01536.
URL https://publications.rwth-aachen.de/record/841181

  * [42] International Organization for Standardization, ISO 3382-1: Acoustics - Measurement of Room Acoustic Parameters. Part 1 : Performance Rooms, ISO, 2009\.
  * [43] L. Panton, M. Yadav, D. Cabrera, D. Holloway, Chamber musicians’ acoustic impressions of auditorium stages: Relation to spatial distribution of early reflections and other parameters, The Journal of the Acoustical Society of America 145 (6) (2019) 3715–3726, number: 6.  doi:10.1121/1.5111748.
URL http://asa.scitation.org/doi/10.1121/1.5111748