# Pulsar Timing Constraints on the Fermi Massive Black-Hole Binary Blazar
Population

A. Miguel Holgado,1 Alberto Sesana,2 Angela Sandrinelli,3,4 Stefano Covino,4
Aldo Treves,3,4 Xin Liu,1 and Paul Ricker1
1Department of Astronomy and National Center for Supercomputing Applications,
University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA
2School of Physics and Astronomy and Institute for Gravitational-Wave
Astronomy, University of Birmingham, Birmingham, B15 2TT, UK
3Universitá dell’Insubria, Dipartimento di Scienza ed Alta Technologia, Via
Valleggio 11, I-22100, Como, Italy
4INAF-Istituto Nazionale di Astrofisica, Osservatorio Astronomico di Brera,
Via Bianchi 46, I-23807 Merate (LC), Italy Contact e-mail:
<EMAIL_ADDRESS>
(Last updated xx; in original form xx)

###### Abstract

Blazars are a subpopulation of active galactic nuclei whose jets are nearly
aligned with the line-of-sight, which tend to exhibit multi-wavelength
variability on a variety of timescales. Quasi-periodic variability on year-
like timescales is sometimes thought to be due to binary orbital motion. If
this is indeed the case, then such blazar binaries would contribute to the
nanohertz gravitational-wave stochastic background. We test the binary
hypothesis for the blazar population observed by the Fermi Gamma-Ray Space
Telescope, which consists of BL Lacertae objects and flat-spectrum radio
quasars. We calculate what the inferred GW background would be from mock
populations informed by the luminosity functions for BL Lacertae objects and
flat-spectrum radio quasars with redshifts $z\leq 2$ and compare the amplitude
distributions to recent pulsar timing array upper limits. To make the inferred
background amplitudes consistent with pulsar-timing upper limits, we obtain
upper limits on the binary fraction for the blazar population, which
approximately range between $\approx 0.01\%-0.1\%$. We discuss implications
for the binary fraction for the unaligned AGN population and conclude that
binarity alone cannot explain year-like quasi-periodicity in blazars.

###### keywords:

quasars, blazars — gravitational waves, stochastic background – pulsar timing
arrays

††pubyear: 2018††pagerange: Pulsar Timing Constraints on the Fermi Massive
Black-Hole Binary Blazar Population–Pulsar Timing Constraints on the Fermi
Massive Black-Hole Binary Blazar Population

##  1 Introduction

Current observational searches for supermassive black-hole (SMBH) binaries on
a variety of scales, i.e., sub-pc to kpc separations, and with masses ranging
within $10^{6}M_{\odot}-10^{9}M_{\odot}$ are motivated to better understand
both galaxy formation and gravitational-wave (GW) astrophysics. There are
several methods to characterize individual SMBH binaries that are thought to
reside in some active galactic nuclei (AGN), including direct imaging (e.g.,
Rodriguez et al., 2006; Bansal et al., 2017), long-term photometric monitoring
(e.g., Graham et al., 2015; Charisi et al., 2016; Liu et al., 2016;
Sandrinelli et al., 2017), and multi-epoch spectroscopy of AGN broad lines
(Shen et al., 2013; Liu et al., 2014; Runnoe et al., 2015, 2017; Wang et al.,
2017). Sub-pc SMBH binaries are of particular interest since they are in the
regime where GW emission starts to significantly drive the inspiral and are at
frequencies where pulsar timing arrays (PTAs) are directly sensitive to. In
this Letter, we focus on the blazar population and its contribution to the
nanohertz GW stochastic background assuming a fraction of the population are
SMBH binaries.

Blazars are a particular type of AGN in which the jet is almost aligned with
the line-of-sight (i.e., $\lesssim 5^{\circ}$) as described by the AGN
unification model (e.g., Urry & Padovani, 1995; Urry, 2000). The Fermi Gamma-
Ray Space Telescope has provided the most complete all-sky catalog of blazars
out to redshift $z\lesssim 4$ (e.g., Acero et al., 2015; Ackermann et al.,
2017). Blazars may exhibit multi-wavelength quasi-periodicities on a variety
of timescales which have been proposed to possibly range from days to years
(e.g., Ackermann et al., 2015; Sandrinelli et al., 2016; Sandrinelli et al.,
2018). Quasi-periodic light curves on timescales ranging from months to years
has been interpreted by some to be due to binary orbital motion (e.g., De
Paolis et al., 2002; Rieger, 2007). The quasi-periodic features may also be
due to intrinsic variability in the accretion disk such as low-frequency
quasi-periodic oscillations (e.g., King et al., 2013). Even though binary
models have been used to explain the observations and tests for relativistic
Doppler boosts have been performed (e.g., D’Orazio et al., 2015; Charisi et
al., 2018; Yan et al., 2018), most candidates cannot definitively be confirmed
nor ruled out with such tests alone. With the advent of GW astronomy, a GW
signal or lack thereof can provide an additional constraint on such binaries
and help to confirm or rule out the binary hypothesis as described below.

Pulsar timing arrays (PTAs) are expected to detect the stochastic nanohertz
gravitational-wave background (GWB) from the inspiral of the SMBH binary
population out to redshifts $z\lesssim 2$. The three international
collaborations that aim to detect GWs in the nanohertz regime are the North
American Nanohertz Observatory for Gravitational Waves (NANOGrav, Demorest et
al. (2013); Arzoumanian et al. (2014)), the Parkes Pulsar Timing Array (PPTA,
Reardon et al. (2016)), and the European Pulsar Timing Array (EPTA, Desvignes
et al. (2016)), which, as a whole, form the International Pulsar Timing Array
(IPTA,Verbiest et al. (2016)).

The binary hypothesis, i.e., the supposition that quasars host a SMBH binary
if they exhibit certain photometric and/or spectroscopic features, has
recently received attention in the context of GW astrophysics. Sesana et al.
(2018), S18 herein, tested the binary hypothesis for SMBH binary candidates
selected via photometric periodicity from the Catalina Real-time Transient
Survey (CRTS, Graham et al. (2015)) and the Palomar Transient Factory (PTF,
Charisi et al. (2016)). S18 computed the implied SMBH binary merger rate from
CRTS and PTF candidates and found the inferred GW background to be in
moderate-to-severe tension with current PTA upper limits, depending on the
corrections for incompleteness, selection effects, virial-mass distributions,
and mass-ratio distributions. There are also a number of binary blazar
candidates that have been reported in the literature based on their quasi-
periodic oscillations (e.g., Romero et al., 2000; Valtonen et al., 2008).
Sandrinelli et al. (2014, 2018) and Ackermann et al. (2015) recently reported
two binary blazar candidates, PKS 2155-304 and PG1553+113, which exhibit
evidence for multi-wavelength quasi-periodic modulations. The gamma-ray light
curve in particular shows quasi-periodic behavior on year-like timescales.
Follow-up studies have used binary models to explain the quasi-periodic light-
curve features (Caproni et al., 2017; Tavani et al., 2018). The evidence for
quasi-periodic features in some blazars may not always be statistically
significant.

In this Letter, we test the binary hypothesis for the blazar population, which
consists of BL Lacertae objects (BL Lacs) and flat-spectrum radio quasars
(FSRQs). Ajello et al. (2012) (A12,herein) and Ajello et al. (2014) (A14,
herein) obtained models for the blazar luminosity functions for FSRQs and BL
Lacs, respectively. We use these luminosity functions to estimate the cosmic
merger rate of SMBH binaries, assuming some binarity fraction. The cosmic
merger rate is then integrated to get the background strain amplitudes and
compared to the most recent PTA upper limits. For our test of the binary
hypothesis, we will consider the scenario in which as much as 10% of blazars
show quasi-periodicity that is statistically significant (Sandrinelli et al.,
2018), which we will assume to be due to binarity. We adopt the same cosmology
as A12 and A14 ($H_{0}=71\ {\rm km}\,{\rm s}^{-1}\,{\rm Mpc}^{-1}$ and
$\Omega_{\rm M}=1-\Omega_{\Lambda}=0.27$).

##  2 The Cosmic Blazar Population and Gravitational-Wave Backgrounds

###  2.1 Blazar subpopulations

Blazars consist of two subpopulations: BL Lacertae objects (BL Lacs), which
have a relatively featureless optical emission continuum, and flat-spectrum
radio quasars (FSRQs), which show optical emission lines. A12 and A14 obtained
models for the blazar luminosity functions for FSRQs and BL Lacs,
respectively, which are defined as

$\frac{{\rm d}^{3}N}{{\rm d}L_{\gamma}\,{\rm d}z\,{\rm d}\Gamma}=\frac{{\rm
d}^{3}N}{{\rm d}L_{\gamma}\,{\rm d}V\,{\rm d}\Gamma}\frac{{\rm d}V}{{\rm
d}z}=\Phi\left(L_{\gamma},V(z),\Gamma\right)\frac{{\rm d}V}{{\rm d}z}\ ,$ (1)

where $L_{\gamma}$ is the gamma-ray luminosity, $\Gamma$ is the photon index,
and ${\rm d}V/{\rm d}z$ is the differential comoving volume per differential
redshift. We use the luminosity-dependent density evolution model for the
observed blazar population that A12 and A14 presents. A12 and A14 also present
beaming corrections to these observed luminosity function to obtain intrinsic
luminosity functions that represent the parent population of blazars. We
account for these beaming corrections when we estimate the cosmic merger rate
as described in §2.3. Our main quantity of interest from the luminosity
function is the number density of blazars, which we obtain by integrating the
luminosity function over all photon indices and luminosities:

$n(z)=\frac{{\rm d}N}{{\rm d}V}=\int_{L_{\gamma,\rm min}}^{L_{\gamma,\rm
max}}\int_{\Gamma_{\rm min}}^{\Gamma_{\rm max}}\Phi(L_{\gamma},z,\Gamma)\ {\rm
d}\Gamma\,{\rm d}L_{\gamma}\ .$ (2)

The bounds of integration are reported in A12 and A14. We plot the number
density for the BL Lacs and FSRQs in Figure 1, considering only the population
at redshifts $z\leq 2$, which is the approximate redshift limit that PTAs are
currently sensitive to.

Figure 1:  Realizations of the luminosity-dependent density evolution model
for FSRQs (A12) and BL Lacs (A14). The red and green lines correspond to the
FSRQ and BL Lac models evaluated at the best-fit parameters reported in A12
and A14, respectively. The shaded regions represent the $1\sigma$
uncertainties in these models.

As A14 notes, the increase in the BL Lac number density at $z\lesssim 0.5$ is
due to contributions from the high-synchrotron-peaked BL Lacs. A14 also notes
that the turnover in the FSRQ number density at $z\lesssim 0.5$ may imply an
evolutionary connection between BL Lacs and FSRQs (e.g., Böttcher & Dermer,
2002; Ghisellini et al., 2011). We do not consider such implications in this
Letter and solely focus on the binary fraction of each population as
independently contributing to the nanohertz GW background.

###  2.2 Blazar mass distributions

The BH mass is a particularly important parameter for our calculations because
the strain amplitude scales with the chirp mass as $h_{\rm c}\propto{\cal
M}^{5/3}$. Since BL Lacs do not show emission lines in their spectra, host-
galaxy relations, like the $M-\sigma$ relation, are used to infer the BH mass
(e.g., Gebhardt et al., 2000; Ferrarese & Merritt, 2000; Barth et al., 2002;
Falomo et al., 2002; Tremaine et al., 2002; Barth et al., 2003; Plotkin et
al., 2011; Kormendy & Ho, 2013). The BH masses for FSRQs are typically
obtained with the virial method (e.g., Castignani et al., 2013). We apply the
same corrections as described in Shen et al. (2008) and S18 to obtain
estimates for the true FSRQ mass distribution from the measured FSRQ virial
masses. We plot the mass distributions for the BL Lacs and FSRQs in the top
panel of Figure 2.

Figure 2:  Top panel: mass distribution of Fermi blazars. BL Lac masses are
often obtained via the $M-\sigma$ relation while FSRQ masses are often
obtained with the virial method. We have applied bias corrections to the FSRQ
virial-mass distribution (Shen et al., 2008; Sesana et al., 2018) to obtain an
estimate of the true distribution of FSRQ masses. Bottom panel: mass-ratio
distribution for SMBH binaries from the Millennium simulation (e.g., Springel
et al., 2005).

The mass distributions for the FSRQ and BL Lac populations are fairly similar,
which is expected given that there is no evidence for BH masses for these
blazar subpopulations to differ due to spectral features, variability, or
inclination.

###  2.3 Generating blazar mock populations and computing the GW background

Following Phinney (2001), the characteristic strain spectrum from a population
of circular SMBH binaries undergoing GW-dominated inspiral is

$h_{\rm c}^{2}=\frac{4G}{\pi c^{2}f^{2}}\int_{0}^{\infty}{\rm
d}z\int_{0}^{\infty}{\rm d}{\cal M}\,\frac{{\rm d}^{2}n}{{\rm d}z\,{\rm
d}{\cal M}}\frac{1}{1+z}\frac{{\rm d}E_{\rm GW}({\cal M})}{{\rm d}\ln f_{\rm
r}}\ ,$ (3)

where $z$ is the redshift, ${\cal M}$ is the chirp mass and obeys ${\cal
M}=Mq^{3/5}/(1+q)^{6/5}$, with $M=M_{1}+M_{2}$ as the total mass and
$q=M_{1}/M_{2}\leq 1$ as the mass ratio, where ${\rm d}^{2}n/{\rm d}z{\rm
d}{\cal M}$ is the cosmic merger rate, and where the energy radiated per
logarithmic frequency interval is

$\frac{{\rm d}E}{{\rm d}\ln f_{\rm r}}=\frac{1}{3}G^{2/3}{\cal M}^{5/3}(\pi
f_{\rm r})^{2/3}\ .$ (4)

We discuss in §3 why our assumption of a circular SMBH binary population,
i.e., no eccentricity due to environmental coupling does not significantly
affect our estimates. Given the A12 and A14 models for the complete blazar
population, some fraction may be binaries, which contributes to the GW
background. We thus approximate the cosmic merger rate as

$\frac{{\rm d}^{2}n}{{\rm d}z\,{\rm d}{\cal M}}=\frac{{\rm d}^{2}n}{{\rm
d}{\cal M}\,{\rm d}t_{\rm r}}\frac{{\rm d}t_{\rm r}}{{\rm
d}z}\approx\frac{\eta}{\zeta\tau_{\rm co}}\frac{{\rm d}n}{{\rm d}{\cal
M}}\frac{{\rm d}t_{\rm r}}{{\rm d}z}\ ,$ (5)

where $\eta$ is the average binary fraction, $\zeta$ is the beaming correction
factor, and $\tau_{\rm co}$ is the coalescence timescale for a GW-driven,
circular SMBHB emitting at a rest-frame frequency $f_{\rm r}=2f_{\rm orb}$,
given by

$\tau_{\rm co}=\frac{5}{256}\left(\frac{G{\cal M}}{c^{3}}\right)^{-5/3}(\pi
f_{\rm r})^{-8/3}\ .$ (6)

To complete the calculation, we need to estimate the chirp-mass distribution,
${\cal P}({\cal M})$, and the coalescence time, $\tau_{\rm co}$. The chirp-
mass distribution is estimated from the FSRQ and BL Lac mass functions by
assuming a SMBH binary mass-ratio distribution from the Millennium simulation
Springel et al. (2005), which is plotted in the in bottom panel of Figure 2.
Cosmological hydrodynamical simulations like Illustris may predict a flatter
mass-ratio distribution (Kelley et al., 2017), though we discuss later in §3
that such differences in reasonable mass-ratio distributions do not have a
significant effect on the GW background amplitude. Drawing from the blazar
luminosity functions, mass distributions, and mass-ratio distribution of the
Millennium simulation, we generate $N=5{\times}10^{5}$ realizations for both
blazar populations. For each value of $({\cal M},z)$, the coalescence time is
then the one corresponding to the longest orbital period that Fermi probes,
which we take to be $P_{\rm orb}=5\ {\rm yr}$. For the average binary
fraction, we take $\eta=0.1$, which is motivated by the fraction of blazars
that exhibit year-like quasi-periodic behavior (Sandrinelli et al., 2018). A12
reported a beaming correction factor of $\zeta_{\rm F}\approx 0.001$ for
FSRQs, corresponding to their reported average jet tilt angle of $\sim
5^{\circ}$. For BL Lacs, we use $\zeta_{\rm B}\approx 0.0025$, corresponding
to A14’s reported jet tilt angle of $\sim 10^{\circ}$.

The strain can be expressed in terms of the GW amplitude at a nominal per-
frequency as $h_{\rm c}=A_{\rm yr}\left(f/1\,{\rm yr}^{-1}\right)^{-2/3}$.
After obtaining the characteristic strain spectra with Eq. (3), we evaluate
the amplitudes at the nominal per-year frequency, $A_{\rm yr}$, which we then
compare to PTA upper limits. Given that PTAs are most sensitive to the most
massive binaries at lower redshifts, we expect that the GW amplitude
distributions we compute will be most sensitive to what the demographics of
our mock populations are at lower redshifts.

###  2.4 Quantifying tension with PTA upper limits

We quantify tension in the same manner as S18, whereby the odds ratio for the
null-hypothesis model over the binary-hypothesis model is

$\Lambda_{\rm NB}=\frac{{\cal P}({\rm D}|{\rm N})}{{\cal P}({\rm D}|{\rm
B})}\frac{{\cal P}({\rm N})}{{\cal P}({\rm B})}=\frac{{\cal P({\rm D}|{\rm
N})}}{{\cal P}({\rm D}|{\rm B})}\ ,$ (7)

where we have taken the null hypothesis and binary hypothesis to be equally
probable such that ${\cal P}({\rm N})={\cal P}({\rm B})$. The likelihood of
model X is

${\cal P}({\rm D}|{\rm X})=\int{\cal P}_{\rm PTA}(A){\cal P}_{\rm X}(A)\ {\rm
d}A\ ,$ (8)

where ${\cal P}_{\rm PTA}(A)$ is the posterior amplitude distribution measured
by PTAs. In the same manner as S18, we quantify tension with the PPTA upper
limits (Shannon et al., 2015), where we approximate the PPTA posterior
amplitude distribution as a Fermi function of the form

${\cal P}_{\rm PTA}(A)=\frac{C_{1}}{e^{(A-C_{2})/C_{3}}+1}\,.$ (9)

where $C_{1}=1.63$, $C_{2}=1.2{\times}10^{-16}$, and
$C_{3}=2.6{\times}10^{-16}$. The probability of the null hypothesis is $p_{\rm
N}=p({\rm D}|{\rm N})/(p({\rm D}|{\rm N})+p({\rm D}|{\rm B}))$ and the
probability of the binary hypothesis is $p_{\rm B}=1-p_{\rm N}$.

##  3 Results and Discussion

With amplitude distributions for the BL Lac and FSRQ populations, we compare
these to recent PTA upper limits. We plot the amplitude distributions and the
reported 95% upper limits from the PPTA NANOGrav 11-yr, and EPTA in Figure 3.

Figure 3:  Amplitude distributions for the BL Lac and FSRQ populations
assuming that 10% of blazars are binaries (Sandrinelli et al., 2018), i.e.,
$\eta=0.1$, and with the beaming correction factors of $\zeta_{\rm B}=0.0025$
(A14) and $\zeta_{\rm F}=0.001$ (A12) for BL Lacs and FSRQs, respectively. The
dotted, dashed, and dot-dashed vertical lines are 95% upper limits on $A_{\rm
yr}$ from PPTA, NANOGrav (11-yr), and EPTA, respectively.

Both the BL Lac and FSRQ amplitude distributions show severe tension with all
upper limits from the EPTA, PPTA, and NANOGrav. Using the most recent NANOGrav
11-year upper limits, we obtain binary fraction upper limits and plot the
distributions in Figure 4.

Figure 4:  Binary fraction upper limits for the BL Lac and FSRQ populations
using the NANOGrav 11-yr 95% upper limit of $A_{\rm yr}=1.45{\times}10^{-15}$.

Both the FSRQ and BL Lac amplitude distributions are in severe tension with
the PTA upper limits. We quantify these tensions in Table 1 using the PPTA
posterior distributions.

Table 1: Model selection using posterior distributions from the PPTA between the null hypothesis and binary hypothesis for the FSRQ and BL Lac populations with their respective beaming corrections and assuming $\eta=0.1$. Model Pair | $\log_{10}(\Lambda_{\rm NB})$ | $p_{\rm B}$
---|---|---
N/FSRQ | 7.14 | $7.23{\times}10^{-8}$
N/BL-Lac | 8.31 | $4.81{\times}10^{-9}$

The odds ratios indicate that the null hypothesis is significantly favored
over the binary hypothesis.

Although this calculation relies on a number of assumptions, they are unlikely
to significantly alter this main result. Despite our assumption of a specific
mass-ratio distribution, S18 and Inayoshi et al. (2018) have shown that
reasonable mass-ratio distributions can affect the overall GW background
amplitude by no more than a factor of $\approx 2$, thus leaving our main
result effectively unchanged. Similarly, the GW background amplitude has a
strong dependence on the assumed maximum orbital period to which Fermi is
sensitive to, which we conservatively assumed to be 5 years, thus making our
results robust. We also notice that PTAs are sensitive to sub-pc SMBH binaries
regardless of whether they are active or quiescent. Our calculation is
representative of the true GW background only if SMBH binaries are assumed to
form solely in gas-rich mergers, thus being harbored in quasars or blazars,
i.e., in AGNs in general. If conversely, SMBH binaries are not connected to
AGN activity, our estimates need to be corrected for the active duty cycle of
Fermi blazars. Again, such correction goes in the direction of making the GW
background larger, thus strengthening our results. We also assumed that the
observed periodicity is related to the binary orbital period, whereas binarity
might induce periodicity through precession. We note, however, that precession
timescales are much longer than the binary orbital period. An observed
precession on year-like timescales would imply a much more compact binary,
thus implying a much shorter coalescence time $\tau_{\rm co}$, and
consequently a much higher GW background.

There are also a few dynamical effects that can partially suppress the
expected GW background in the PTA band. In fact, we did not consider the
effects of environmental coupling in this work since S18 finds that the GW
signal amplitude would decrease only by a factor of ${\sim}1.5$ at $6\ {\rm
nHz}$. Moreover, their estimate does not account for the fact that increased
eccentricity due to environmental coupling decreases the coalescence timescale
for SMBH binaries, which increases the cosmic merger rate, and ultimately, the
GW background strain amplitude.

##  4 Conclusions

In this Letter, we have used PTA upper limits on the stochastic GW background
at nanohertz frequencies to constrain the nature of quasi-periodic blazar
candidates that Fermi observes. The Fermi blazar catalog contains the most
complete all-sky census of $\gamma$-ray blazars out to redshifts of $z\lesssim
4$ and statistically significant quasi-periodicity on year-like timescales has
been detected in some of the brightest sources. One interpretation of this
observed quasi-periodicity on year-like timescales is that it stems from the
orbital motion of a SMBH binary. Is this is indeed the case, then the
collective population of periodic blazars would produce a stochastic GW
background detectable in the PTA frequency band.

As previously demonstrated by S18, we have shown that even though PTAs have
not yet detected the nanohertz GW stochastic background, the upper limits that
they place provide astrophysically relevant constraints for characterizing the
SMBH binary population. We have also presented upper limits on the binary
fraction of BL Lacs and FSRQs of the blazar population, where we have used
blazar luminosity functions, distributions for the measured masses from host-
galaxy relations for BL Lacs and corrected virial masses for FSRQs, and mass-
ratio distributions informed by the Millennium simulation. We conclude that
binarity alone cannot explain the quasi-periodic behavior of blazars.

Further study of current binary candidates and future searches for sub-pc SMBH
binaries will continue. We expect the methods presented in S18 and this Letter
to be pivotal tests to favor or disfavor SMBH binary candidates.

## Acknowledgements

AMH is supported by the DOE NNSA Stockpile Stewardship Graduate Fellowship.
AMH gratefully acknowledges the NANOGrav PIRE program for supporting travel to
the University of Birmingham, UK, where much of this work was carried out. AMH
also thanks the School of Physics and Astronomy and the Institute for
Gravitational-Wave Astronomy for their hospitality during his visit to the
University of Birmingham. AS is supported by the Royal Society. We thank Maria
Charisi, Eliu Huerta, Vicky Kalogera, Xavier Siemens, Joseph Lazio, Stephen
Taylor, Chiara Mingarelli, and Maura McLaughlin for helpful comments and
fruitful discussions.

## References

  * Acero et al. (2015) Acero F., et al., 2015, ApJS, 218, 23
  * Ackermann et al. (2015) Ackermann M., et al., 2015, ApJL, 813, L41
  * Ackermann et al. (2017) Ackermann M., et al., 2017, ApJL, 837, L5
  * Ajello et al. (2012) Ajello M., et al., 2012, ApJ, 751, 108
  * Ajello et al. (2014) Ajello M., et al., 2014, ApJ, 780, 73
  * Arzoumanian et al. (2014) Arzoumanian Z., et al., 2014, ApJ, 794, 141
  * Bansal et al. (2017) Bansal K., Taylor G. B., Peck A. B., Zavala R. T., Romani R. W., 2017, ApJ, 843, 14
  * Barth et al. (2002) Barth A. J., Ho L. C., Sargent W. L. W., 2002, ApJL, 566, L13
  * Barth et al. (2003) Barth A. J., Ho L. C., Sargent W. L. W., 2003, ApJ, 583, 134
  * Böttcher & Dermer (2002) Böttcher M., Dermer C. D., 2002, ApJ, 564, 86
  * Caproni et al. (2017) Caproni A., Abraham Z., Motter J. C., Monteiro H., 2017, ApJL, 851, L39
  * Castignani et al. (2013) Castignani G., Haardt F., Lapi A., De Zotti G., Celotti A., Danese L., 2013, A&A, 560, A28
  * Charisi et al. (2016) Charisi M., Bartos I., Haiman Z., Price-Whelan A. M., Graham M. J., Bellm E. C., Laher R. R., Márka S., 2016, MNRAS, 463, 2145
  * Charisi et al. (2018) Charisi M., Haiman Z., Schiminovich D., D’Orazio D. J., 2018, MNRAS, 476, 4617
  * D’Orazio et al. (2015) D’Orazio D. J., Haiman Z., Schiminovich D., 2015, Nature, 525, 351
  * De Paolis et al. (2002) De Paolis F., Ingrosso G., Nucita A. A., 2002, A&A, 388, 470
  * Demorest et al. (2013) Demorest P. B., et al., 2013, ApJ, 762, 94
  * Desvignes et al. (2016) Desvignes G., et al., 2016, MNRAS, 458, 3341
  * Falomo et al. (2002) Falomo R., Kotilainen J. K., Treves A., 2002, ApJ, 569, L35
  * Ferrarese & Merritt (2000) Ferrarese L., Merritt D., 2000, ApJL, 539, L9
  * Gebhardt et al. (2000) Gebhardt K., et al., 2000, ApJL, 539, L13
  * Ghisellini et al. (2011) Ghisellini G., Tavecchio F., Foschini L., Ghirlanda G., 2011, MNRAS, 414, 2674
  * Graham et al. (2015) Graham M. J., et al., 2015, MNRAS, 453, 1562
  * Inayoshi et al. (2018) Inayoshi K., Ichikawa K., Haiman Z., 2018, preprint, 1805, arXiv:1805.05334
  * Kelley et al. (2017) Kelley L. Z., Blecha L., Hernquist L., 2017, MNRAS, 464, 3131
  * King et al. (2013) King O. G., et al., 2013, MNRAS Letters, 436, L114
  * Kormendy & Ho (2013) Kormendy J., Ho L. C., 2013, ARA&A, 51, 511
  * Liu et al. (2014) Liu X., Shen Y., Bian F., Loeb A., Tremaine S., 2014, ApJ, 789, 140
  * Liu et al. (2016) Liu T., et al., 2016, ApJ, 833, 6
  * Phinney (2001) Phinney E. S., 2001, arXiv:astro-ph/0108028
  * Plotkin et al. (2011) Plotkin R. M., Markoff S., Trager S. C., Anderson S. F., 2011, MNRAS, 413, 805
  * Reardon et al. (2016) Reardon D. J., et al., 2016, MNRAS, 455, 1751
  * Rieger (2007) Rieger F. M., 2007, Astrophys Space Sci, 309, 271
  * Rodriguez et al. (2006) Rodriguez C., Taylor G. B., Zavala R. T., Peck A. B., Pollack L. K., Romani R. W., 2006, ApJ, 646, 49
  * Romero et al. (2000) Romero G. E., Chajet L., Abraham Z., Fan J. H., 2000, A&A, 360, 57
  * Runnoe et al. (2015) Runnoe J. C., et al., 2015, ApJS, 221, 7
  * Runnoe et al. (2017) Runnoe J. C., et al., 2017, MNRAS, 468, 1683
  * Sandrinelli et al. (2014) Sandrinelli A., Covino S., Treves A., 2014, ApJL, 793, L1
  * Sandrinelli et al. (2016) Sandrinelli A., Covino S., Dotti M., Treves A., 2016, AJ, 151, 54
  * Sandrinelli et al. (2017) Sandrinelli A., et al., 2017, A&A, 600, A132
  * Sandrinelli et al. (2018) Sandrinelli A., Covino S., Treves A., Holgado A. M., Sesana A., Lindfors E., Ramazani V. F., 2018, arXiv:1801.06435 [astro-ph]
  * Sesana et al. (2018) Sesana A., Haiman Z., Kocsis B., Kelley L. Z., 2018, ApJ, 856, 42
  * Shannon et al. (2015) Shannon R. M., et al., 2015, Science, 349, 1522
  * Shen et al. (2008) Shen Y., Greene J. E., Strauss M. A., Richards G. T., Schneider D. P., 2008, ApJ, 680, 169
  * Shen et al. (2013) Shen Y., Liu X., Loeb A., Tremaine S., 2013, ApJ, 775, 49
  * Springel et al. (2005) Springel V., et al., 2005, Nature, 435, 629
  * Tavani et al. (2018) Tavani M., Cavaliere A., Munar-Adrover P., Argan A., 2018, ApJ, 854, 11
  * Tremaine et al. (2002) Tremaine S., et al., 2002, ApJ, 574, 740
  * Urry (2000) Urry C. M., 2000, AIP Conference Proceedings, 522, 299
  * Urry & Padovani (1995) Urry C. M., Padovani P., 1995, Publications of the Astronomical Society of the Pacific, 107, 803
  * Valtonen et al. (2008) Valtonen M. J., et al., 2008, Nature, 452, 851
  * Verbiest et al. (2016) Verbiest J. P. W., et al., 2016, MNRAS, 458, 1267
  * Wang et al. (2017) Wang L., Greene J. E., Ju W., Rafikov R. R., Ruan J. J., Schneider D. P., 2017, ApJ, 834, 129
  * Yan et al. (2018) Yan D., Zhou J., Zhang P., Zhu Q., Wang J., 2018, arXiv:1804.05342 [astro-ph]


# Expressivity-aware Music Performance Retrieval using
Mid-level Perceptual Features and Emotion Word Embeddings

Shreyan Chowdhury  Institute of Computational PerceptionJohannes Kepler
University LinzAustria<EMAIL_ADDRESS>and  Gerhard Widmer  Institute of
Computational PerceptionJohannes Kepler University LinzAustria
<EMAIL_ADDRESS>
(2023)

###### Abstract.

This paper explores a specific sub-task of cross-modal music retrieval. We
consider the delicate task of retrieving a performance or rendition of a
musical piece based on a description of its style, expressive character, or
emotion from a set of different performances of the same piece. We observe
that a general purpose cross-modal system trained to learn a common text-audio
embedding space does not yield optimal results for this task. By introducing
two changes – one each to the text encoder and the audio encoder – we
demonstrate improved performance on a dataset of piano performances and
associated free-text descriptions. On the text side, we use emotion-enriched
word embeddings (EWE) and on the audio side, we extract mid-level perceptual
features instead of generic audio embeddings. Our results highlight the
effectiveness of mid-level perceptual features learnt from music and emotion
enriched word embeddings learnt from emotion-labelled text in capturing
musical expression in a cross-modal setting. Additionally, our interpretable
mid-level features provide a route for introducing explainability in the
retrieval and downstream recommendation processes.

††journalyear: 2023††copyright: rightsretained††conference: Forum for
Information Retrieval Evaluation; December 15–18, 2023; Panjim,
India††booktitle: Forum for Information Retrieval Evaluation (FIRE 2023),
December 15–18, 2023, Panjim, India††doi: 10.1145/3632754.3632761††isbn:
979-8-4007-1632-4/23/12

##  1\. Introduction

Music performance involves an interplay between a composer, a performer, and a
listener. The performer imbues a composition with their own expression and
style, and the listener perceives the emotion or mood being communicated
through the music by the composer and the performer (Juslin, 2013). Thus the
expressive quality of performed music becomes an important attribute through
which a musical performance is characterised (Gabrielsson and Juslin, 1996;
Akkermans et al., 2019). The present paper provides a step towards building
music retrieval and recommendation systems that are sensitive to this
expressive quality of music.

We aim to develop a system for retrieving a desired performance of a musical
piece from a set of different performances based on a description of its
style, expressive character, or emotion. This is an important problem when
dealing with certain genres of music such as Western classical music, where we
typically have several renditions of compositions performed by different
artists111As stated by a major music streaming platform, “Classical music is
different. It has longer and more detailed titles, multiple artists for each
work, and hundreds of recordings of well-known pieces” (app, 2023)., and
possibly very distinguishing listeners. These interpretations, while all
representing the same musical content (piece), could vary subtly or vastly in
their expressive character, and traditional recommendation and retrieval
algorithms are not designed to be sensitive to such performance
characteristics.

Figure 1. A system for retrieving the best-matching performance of a musical
piece based on a text description of its expressive character.

In our work, we look at text-based music retrieval. Formally, given a set of
$K$ performances $P=\\{p_{1},p_{2},\ldots,p_{K}\\}$ of a musical piece and a
query $Q=\\{w_{1},w_{2},\ldots,w_{N}\\}$ comprised of $N$ “descriptive” words,
we would like to retrieve the performance (say $p_{3}$, as shown in Figure 1)
that we believe a general audience might consider matches best with the
description. This is a highly subjective task. We thus evaluate our models on
the Con Espressione dataset (Cancino-Chacón et al., 2020), which is a dataset
comprised of different piano performances, by famous pianists, of a number of
classical piano pieces, and associated free-text descriptions provided by
contributors through an online questionnaire (more details in Section 3). In
this dataset, we have on an average of five performances per piece, which
thereby constitute the search space for each query. The assumption here is
that we already know the piece (and thus the search space for that piece) for
each query, and we want to retrieve the closest matching performance in terms
of its expressive character. Inferring the piece itself from the query, while
a necessary component of an end-to-end retrieval system, is not the focus of
the present paper. The reader is encouraged to look at works on music search
and retrieval such as (Schedl et al., 2014) for more details on this topic.

The subjective nature of the task also motivated us to consider human-
oriented, perceptually relevant descriptors for representing expressive
qualities extracted from audio recordings. In particular, we focus on so-
called Mid-level Perceptual Features – relatively high-level musical qualities
that are considered to be perceptually important (Aljanaki and Soleymani,
2018); in our work, these are learnt from human annotations. Previous research
has shown these features to have the capacity to predict musical emotions as
well as to disentangle different performances based on emotion (Chowdhury and
Widmer, 2021a; Chowdhury, 2022; Cancino-Chacón et al., 2020).

Furthermore, on the text side, we note the inefficiency of traditional word
embeddings (Word2Vec, GloVe) to capture the intended emotion in descriptions
of musical performances. Such language representation models carry the
inductive bias that words used in the same context tend to possess similar
meanings, thus resulting in emotionally dissimilar words like happy and sad
having close proximity in the representation space, due to them often
occurring in similar contexts in the training data. To derive correct
emotional meanings from our text queries, we experiment with emotion-enriched
word embeddings from Agrawal et al. (Agrawal et al., 2018) and find that they
improve retrieval results significantly, particularly when combined with mid-
level features on the audio side.

##  2\. Related Work

Our work sits in the broader area of cross-modal retrieval. This is an area of
active research in general, and is of great significance to music information
retrieval (Müller et al., 2018; Doh et al., 2023; Li and Kumar, 2019).
Language-based audio retrieval is currently witnessing much interest from the
research community as can be seen from active participation in competitions
like DCASE (DCASE Challenge, 2023). In this section, we look at some recent
work in text-based retrieval for music.

Audio-Language Learning for Music: Text-based audio retrieval is typically
done by learning a common embedding space of aligned audio and text
embeddings. In “MusCALL” (Manco et al., 2022), this is done using a multimodal
contrastive learning (MCL) approach. Two encoders, $f_{a}(\cdot)$ for audio
and $f_{t}(\cdot)$ for text, are learnt such that for any audio-text pair
$(a_{i},t_{i})$, the resulting embeddings $z_{a,i}=f_{a}(a_{i})$ and
$z_{t,i}=f_{t}(t_{i})$ lie close in the joint embedding space (Radford et al.,
2021). A ResNet-50 (He et al., 2015) is used for the audio backbone, and a
downsized-Transformer (Vaswani et al., 2017) for the text backbone. They use
250k audio-text pairs from a production music library as the dataset. In
“MuLan” (Huang et al., 2022) a much larger dataset of 44m recordings and
weakly-associated, free-form text annotations is used. They use a pre-trained
BERT (Devlin et al., 2018) as their text encoder and experiment with two
different audio encoders – ResNet-50 and Audio Spectrogram Transformer (Gong
et al., 2021).

A more recent work is “Music and Text Representation” learning (MTR) (Doh et
al., 2023). It lays out an investigation into effective design choices for
universal representation learning for text-to-music retrieval systems. This
work uses a set of 500k music-text pairs. They use a modified version of the
Music Tagging Transformer (Won et al., 2021a) as the audio encoder, and two
different text encoders – pre-trained GloVe (Pennington et al., 2014) and pre-
trained BERT. We were able to obtain the trained models for this work, which
we use as a baseline for our work.

Emotion Embedding Spaces: In (Won et al., 2021b), the authors propose “emotion
embeddings” for retrieval of musical pieces that match the emotional
characteristic of stories. They use a ResNet model (Won et al., 2020) as the
audio backbone and a pre-trained DistilBERT (Sanh et al., 2019) as the text
backbone. However, they do not use natural language in the audio-text pairs in
their experiments; rather the text component comes from the labels or tags
associated with the audio clips, which are mapped to the embedding space using
a Word2Vec model or emotion lexicons.

###  2.1. Music-Text Representation (MTR)

We consider the Music-Text Representation (MTR) model by Doh et al. (Doh et
al., 2023) as the baseline system. It is a cross-modal retrieval system that
projects audio and text representations onto a common embedding space and
reduces the distance between paired vectors during training. It is trained on
500k audio-text pairs with the text formed by concatenating tags from
different sources. This data is a subset of the Million Song Dataset (Bertin-
Mahieux et al., 2011) and contains songs from a mix of different genres. We
use weights of the best version of their system according to the results on
their paper, which is a contrastive model type with a BERT text encoder and
stochastically sampled text representations.

##  3\. Datasets

The Con Espressione Dataset (Cancino-Chacón et al., 2020) consists of
recordings of 9 piano pieces played by different famous pianists (making a
total of 45 performances), and associated free-text descriptions for each
performance, collected from a large number of listeners. The study
participants were asked to describe the expressive character of each
performance. Typical characterisations in the dataset are adjectives like
“cold”, “playful”, “dynamic”, “passionate”, but also more complex phrases such
as “controlled with speed”, “smooth tempo variation”, “emotional with
dynamics” etc. In this work, we use the aggregated answers for each
performance. That is, all words or phrases used by different participants for
a performance are concatenated into a single text description of the
performance.

The Mid-level Perceptual Features Dataset (Aljanaki and Soleymani, 2018)
consists of 5k song snippets of 15 seconds each annotated according to seven
mid-level descriptors: melodiousness, articulation, rhythm stability, rhythm
complexity, dissonance, tonal stability, and modality (or ‘minorness’). The
ratings for the seven mid-level perceptual features were collected through
crowd-sourcing. We use a model trained on this dataset as our audio encoder,
and we refer to this trained model as the “mid-level model”, or “mid-level
encoder” in the following paragraphs. We add onset density as an additional
feature according to (Chowdhury and Widmer, 2023) (this gives the best results
in our case).

Additional Datasets: MusicCaps and Pitchfork Track Reviews. In order to
effectively train a model, we require a substantial amount of audio-text
paired data, and unfortunately, the Con Espressione dataset is insufficient in
size for this particular task. To expand our dataset, we look towards two
sources: the MusicCaps dataset, and a trusted and well-known music review
website, Pitchfork (www.pitchfork.com).

The MusicCaps dataset (Agostinelli et al., 2023) consists of 5.5k music clips
from diverse genres with paired text descriptions

##  4\. Approach

We define performance retrieval as the task of retrieving a particular musical
performance from a set of $K$ performances $P=\\{p_{1},p_{2},\ldots,p_{K}\\}$
of a musical piece such that the returned performance best matches the query
$Q=\\{w_{1},w_{2},\ldots,w_{N}\\}$ comprised of $N$ words. In our case, the
query is a description of the desired expressive character of a performance
provided in the form of text. For each query, the piece is known, so the
search space for the system is the set of performances of that piece. This is
arguably a hard task; the retrieval system will have to be sensitive to very
subtle musical differences. In our experiments, we will consider a retrieval
result ”correct” if the output of the system ranks the performance
corresponding to the input text the highest.

We take the basic framework of Music-Text Representation (MTR) by Doh et al.
(Doh et al., 2023) and modify the text and audio encoders. We hypothesise that
two kinds of modifications might improve the model’s effectiveness for
expressivity-aware performance retrieval. First we need an audio encoder
trained to extract features that are more attentive to the expressive
qualities in music. Mid-level perceptual features (Aljanaki and Soleymani,
2018) have a significant capacity to capture such musical qualities and hold
good predictive power for music emotion (Chowdhury, 2022). We thus replace the
audio encoder in MTR with a mid-level feature model. This pre-trained model
takes audio spectrograms as inputs and outputs seven mid-level features (seven
real-valued scalars). To this we add an eighth feature: onset density
(Chowdhury and Widmer, 2023), intended to model the ‘perceptual speed’ of a
performance. The assumption here is that points close together in the space
spanned by the mid-level features are similar in their expressive quality.
Second, we investigate if the system works better with a text encoder trained
using emotion labels. While the text encoder in MTR is a state-of-the-art BERT
sentence model, in our task, the sentence structure has less of a consequence
than obtaining accurate representations of the descriptive words. We thus
experiment with an emotion-enriched word embedding model (EWE) (Agrawal et
al., 2018).

###  4.1. Mid-level Features Audio Encoder and Emotion Enriched Text Encoder

As reasoned earlier in this paper, we propose replacing the audio encoder in
MTR with a trained mid-level feature model ($f_{m}$ in Figure 2). The input to
this model is an audio spectrogram, the output is a vector of 8 mid-level
features (including onset density). This model is domain-adapted for piano
music, since that is our domain of interest for this paper. As shown in
(Chowdhury and Widmer, 2021b), domain adaptation can improve mid-level
prediction for piano audio without compromising the performance for other
styles of music.

The text encoder is also replaced by an emotion enriched word embedding (EWE)
model ($g_{\text{EWE}}$ in Figure 2), which outputs an embedding of dimension
300 for each word. For a set of descriptive words, we take the resultant
vector by adding all individual embeddings element-wise.

Now since these two encoders are pre-trained, they do not project to a common
embedding space. Hence we need a model to project the outputs of the encoders
to a common space for enabling cross-modal retrieval. We choose to project the
text embeddings onto the mid-level feature space using a linear model $h$.
Preserving the mid-level feature predictions has the additional advantage of
providing explainable insight into the retrieval process, due to the
dimensions possessing interpretable musical meanings. This is described in the
next section. For $h$, a simple linear regression model proved sufficient. In
some cases (see Table 2), transforming the text embeddings with principal
component analysis (PCA) improved the results.

###  4.2. Experiments and Results

We perform piece-wise cross-validation over the Con Espressione dataset to
fully utilise the available data, i.e., in each run the test set is the set of
all performances of a piece in the dataset. The rest of the dataset is the
train set for that run. This results in 9-fold cross-validation as we have 9
pieces in the dataset. For retrieval, we use the cosine similarity in the mid-
level feature space:

(1)
$\text{cosine\\_similarity}(\mathbf{m_{1}},\mathbf{m_{2}})=\frac{\mathbf{m_{1}}\cdot\mathbf{m_{2}}}{\|\mathbf{m_{1}}\|\cdot\|\mathbf{m_{2}}\|}$

We use top-$k$ ratio and Mean Reciprocal Rank (MRR) as evaluation metrics.
Top-$k$ ratio (ranges from 0 to 1, with higher scores indicating better
retrieval performance) is defined as the number of queries for which the
correct performance has a rank equal to or better than $k$ among all
performances of the piece for which the query is made. We use $k=1$ and $k=2$.

Mean Reciprocal Rank (MRR) is defined as the average of the reciprocal rank of
the correct item retrieved over a set of queries:

(2) $\text{MRR}=\frac{1}{|M|}\sum_{i=1}^{|M|}\frac{1}{\text{rank}_{i}}$

where $|M|$ is the total number of queries, and $\text{rank}_{i}$ is the rank
of the correct item in the result list for the $i$-th query. MRR ranges from 0
to 1, with higher scores indicating better performance of the ranking
algorithm.

We also investigate the impact of augmenting the train set with additional
data from the MusicCaps and Pitchfork datasets (see Table 2).

Figure 2. In our system, the audio and text encoders of a MTR model (Doh et al., 2023) are replaced by a mid-level feature model and emotion enriched word representation model respectively. Table 1. Retrieval results using different audio and text encoders. The values here are for a model trained with both Pitchfork and MusicCaps augmentation and no PCA on text embeddings (see Table 2). For the random baseline, performances are chosen at random.  Text Enc | Audio Enc | Top-1 | Top-2 | MRR
---|---|---|---|---
(Random Baseline) | 0.18 | 0.37 | 0.44
MTR | MTR | 0.20 | 0.42 | 0.46
MTR | Mid-level | 0.22 | 0.53 | 0.50
EWE | MTR | 0.22 | 0.42 | 0.47
EWE | Mid-level | 0.38 | 0.64 | 0.61
Table 2. Effect of data augmentation and PCA on retrieval based on EWE embeddings and Mid-level features Augmentation | Text Emb | Metrics
---|---|---
Pitchfork | MusicCaps | Transform (PCA) | Top-1 | Top-2 | MRR
✗ | ✗ | ✗ | 0.22 | 0.46 | 0.48
✗ | ✗ | ✓ | 0.29 | 0.49 | 0.52
✓ | ✗ | ✗ | 0.24 | 0.44 | 0.48
✓ | ✗ | ✓ | 0.40 | 0.58 | 0.60
✗ | ✓ | ✗ | 0.33 | 0.57 | 0.57
✗ | ✓ | ✓ | 0.27 | 0.53 | 0.52
✓ | ✓ | ✗ | 0.38 | 0.64 | 0.61
✓ | ✓ | ✓ | 0.36 | 0.56 | 0.57
Figure 3.  Seven performances of Bach’s C Major Prelude, with human textual
characterisation and corresponding mid-level feature values predicted by
mapping $h(g_{\text{EWE}})$. Text and feature values relate to the performance
identified by ”true = …”

Despite reasonable performance of traditional models on the general purpose
task of music retrieval using text input, there remains much room for
improvement for expressivity-aware music performance retrieval. We observe
(see Table 1, second row) that for the MTR model the mean reciprocal rank for
performance retrieval on the Con Espressione dataset is only 0.46, which means
the average rank of the correct performance is greater than 2 (on sets of, on
average, 5 performances per piece); in effect, this is similar to what we
obtain with random guessing (Table 1, 1st row). Moreover, only 20% of the
queries return the correct performance as the top result.

On the other hand, our model with emotion word embeddings (EWE) and mid-level
features returns the correct performance for 38% of queries, with an MRR value
of 0.61 (Table 1, last row), meaning the average rank of the correct
performance is about 1.63. While our method shows significant improvement over
the baseline, it is important to note that this is a highly subjective task
and the dataset we have at hand is not large. The main objective of this paper
is to provide a proof-of-concept that models trained using domain-specific
perceptual features can lend a significant advantage in cross-modal retrieval
applications.

From Table 2, we see that augmenting with additional data has a minor but
positive effect on the results whereas PCA tends to improve performance for
non-augmented datasets. For the augmented case, PCA actually worsened the
performance.

##  5\. Mid-level Features as Explanations

It is instructive to look at an individual example. The mid-level features
that our model predicts from the audio recordings, and which it uses to
establish a relationship to textual descriptions, can be viewed as a kind of
explanation of the model’s retrieval choice, pointing to musical qualities in
the performance that may have influenced the decision. In the example shown in
Figure 3, the piece in question is J.S.Bach’s Praeludium in C major from his
Well-Tempered Clavier (Book I), for which our database contains seven
different recordings, by Walter Gieseking, Glenn Gould, Hélène Grimaud,
Wilhelm Kempff, Sviatoslav Richter, Martin Stadtfeld, and one completely
expressionless, mechanical MIDI performance derived from the score. Given the
query ”shy, magical, deep, delicate, …”, the system returns the Grimaud
performance (which is the ”correct” one, as these descriptions were indeed
associated with her performance by the human annotators); ”mechanical, boring,
beginner” returns, again correctly, the MIDI rendering. (The baseline MTR
model also gets the Grimaud right, but returns Sviatoslav Richter in response
to ”mechanical, boring, beginner”). The mid-level feature profiles predicted
from these text queries by our model (i.e., its ‘translation’ of free text
into mid-level feature space) explain its decisions quite well and conform
both to our intuition, and to what we hear in the recordings. In particular,
Grimaud is described as having high ”melodiousness” and very low
”articulation” whereas the MIDI rendering is characterised by rather neutral
values throughout, and negative rather than positive ”melodiousness”. Such
high-level musical descriptions could be useful as explanations in a real
music search and recommendation application.

##  6\. Conclusion

We present an experimental demonstration showing that mid-level perceptual
features and emotion enriched text embeddings are useful in capturing some of
the expressive musical character in audio recordings and relating them to
descriptive text. We see significant improvements in all three metrics
(particularly, the number of times the correct performance was retrieved
almost doubled) and we note the effect of modifying both the audio and the
text encoders. While this is not a large-scale study, our results point
towards the importance of perceptually-driven features and emotion-aware
models in music retrieval and consequently in music recommendation. Features
that are interpretable and musically meaningful are also crucial for
explainability and provide a route for training downstream interpretable
models of music retrieval and recommendation.

###### Acknowledgements.

This research was supported by the European Research Council (ERC) under the
European Union’s Horizon 2020 research and innovation programme, grant
agreements No 670035 (“Con Espressione”) and 101019375 (“Whither Music?”).

## References

  * (1)
  * app (2023) 2023\.  Apple Music Classical.  https://support.apple.com/en-us/HT213415
  * Agostinelli et al. (2023) Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matthew Sharifi, Neil Zeghidour, and Christian Havnø Frank. 2023.  MusicLM: Generating Music From Text.  _ArXiv_ abs/2301.11325 (2023).
  * Agrawal et al. (2018) Ameeta Agrawal, Aijun An, and Manos Papagelis. 2018.  Learning emotion-enriched word representations. In _Proceedings of the 27th international conference on computational linguistics_. 950–961.
  * Akkermans et al. (2019) Jessica Akkermans, Renee Schapiro, Daniel Müllensiefen, Kelly Jakubowski, Daniel Shanahan, David Baker, Veronika Busch, Kai Lothwesen, Paul Elvers, Timo Fischinger, et al. 2019\.  Decoding emotions in expressive music performances: A multi-lab replication and extension study.  _Cognition and Emotion_ 33, 6 (2019), 1099–1118.
  * Aljanaki and Soleymani (2018) Anna Aljanaki and Mohammad Soleymani. 2018.  A Data-driven Approach to Mid-level Perceptual Musical Feature Modeling. In _Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018_. Paris, France, 615–621.
  * Bertin-Mahieux et al. (2011) Thierry Bertin-Mahieux, Daniel P.W. Ellis, Brian Whitman, and Paul Lamere. 2011.  The Million Song Dataset. In _Proceedings of the 12th International Society for Music Information Retrieval Conference, ISMIR 2011_. Miami, Florida.
  * Cancino-Chacón et al. (2020) Carlos Cancino-Chacón, Silvan David Peter, Shreyan Chowdhury, Anna Aljanaki, and Gerhard Widmer. 2020.  On the Characterization of Expressive Performance in Classical Music: First Results of the Con Espressione Game. In _Proceedings of the 21st International Society for Music Information Retrieval Conference, ISMIR 2020_. Online.
  * Chowdhury (2022) Shreyan Chowdhury. 2022.  _Modelling Emotional Expression in Music Using Interpretable and Transferable Perceptual Features_.  Ph. D. Dissertation. Johannes Kepler University Linz, Austria.
  * Chowdhury and Widmer (2021a) Shreyan Chowdhury and Gerhard Widmer. 2021a.  On Perceived Emotion in Expressive Piano Performance: Further Experimental Evidence for the Relevance of Mid-level Perceptual Features. In _Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021_. Online.
  * Chowdhury and Widmer (2021b) Shreyan Chowdhury and Gerhard Widmer. 2021b.  Towards Explaining Expressive Qualities in Piano Recordings: Transfer of Explanatory Features Via Acoustic Domain Adaptation. In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing_. Online, 561–565.  https://doi.org/10.1109/ICASSP39728.2021.9413638
  * Chowdhury and Widmer (2023) Shreyan Chowdhury and Gerhard Widmer. 2023.  Decoding and Visualising Intended Emotion in an Expressive Piano Performance. In _Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Late-breaking Demo Session_. Bangalore, India.
  * DCASE Challenge (2023) DCASE Challenge. 2023.  Language-based audio retrieval.  https://dcase.community/challenge2023/task-language-based-audio-retrieval
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_. New Orleans, Louisiana, USA.
  * Doh et al. (2023) SeungHeon Doh, Minz Won, Keunwoo Choi, and Juhan Nam. 2023.  Toward Universal Text-To-Music Retrieval. In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 1–5.  https://doi.org/10.1109/ICASSP49357.2023.10094670
  * Gabrielsson and Juslin (1996) Alf Gabrielsson and Patrik N. Juslin. 1996.  Emotional Expression in Music Performance: Between the Performer’s Intention and the Listener’s Experience.  _Psychology of Music_ 24, 1 (1996), 68–91.  https://doi.org/10.1177/0305735696241007
  * Gong et al. (2021) Yuan Gong, Yu-An Chung, and James Glass. 2021.  AST: Audio Spectrogram Transformer. In _Proceedings of Interspeech 2021_. Brno, Czech Republic, 571–575.
  * He et al. (2015) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. 2015.  Deep Residual Learning for Image Recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2015), 770–778.
  * Huang et al. (2022) Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel P. W. Ellis. 2022.  MuLan: A Joint Embedding of Music Audio and Natural Language. In _Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022_. Bangalore, India.
  * Juslin (2013) Patrik N Juslin. 2013.  What does music express? Basic emotions and beyond.  _Frontiers in psychology_ 4 (2013), 596.
  * Li and Kumar (2019) Bochen Li and Aparna Kumar. 2019.  Query by Video: Cross-modal Music Retrieval.. In _Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019_. Delft, The Netherlands, 604–611.
  * Manco et al. (2022) Ilaria Manco, Emmanouil Benetos, Elio Quinton, and György Fazekas. 2022.  Contrastive Audio-Language Learning for Music. In _Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022_. Bangalore, India.
  * Müller et al. (2018) Meinard Müller, Andreas Arzt, Stefan Balke, Matthias Dorfer, and Gerhard Widmer. 2018.  Cross-modal music retrieval and applications: An overview of key methodologies.  _IEEE Signal Processing Magazine_ 36, 1 (2018), 52–62.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global Vectors for Word Representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Doha, Qatar, 1532–1543.  https://doi.org/10.3115/v1/D14-1162
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning Transferable Visual Models from Natural Language Supervision. In _International Conference on Machine Learning_. PMLR, Online, 8748–8763.
  * Sanh et al. (2019) V. Sanh, L. Debut, J. Chaumond, and T. Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In _Neural Information Processing Systems Workshop on Energy Efficient Machine Learning and Cognitive Computing_. Vancouver, BC, Canada.
  * Schedl et al. (2014) Markus Schedl, Emilia Gómez, Julián Urbano, et al. 2014\.  Music information retrieval: Recent developments and applications.  _Foundations and Trends® in Information Retrieval_ 8, 2-3 (2014), 127–261.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  _Advances in Neural Information Processing Systems_ 30\.
  * Won et al. (2021a) Minz Won, Keunwoo Choi, and Xavier Serra. 2021a.  Semi-supervised music tagging transformer. In _Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online_.
  * Won et al. (2020) Minz Won, Andrés Ferraro, Dmitry Bogdanov, and Xavier Serra. 2020.  Evaluation of CNN-based Automatic Music Tagging Models.  _Proceedings of Sound and Music Computing (SMC)_ abs/2006.00751 (2020).
  * Won et al. (2021b) Minz Won, Justin Salamon, Nicholas J. Bryan, Gautham J. Mysore, and Xavier Serra. 2021b.  Emotion Embedding Spaces for Matching Music to Stories. In _Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021_. Online.