# Conformal Weyl gravity via two stages of quasinormal ringing and late-time
behavior

R. A. Konoplya<EMAIL_ADDRESS>Institute of Physics and Research
Centre of Theoretical Physics and Astrophysics, Faculty of Philosophy and
Science, Silesian University in Opava, CZ-746 01 Opava, Czech Republic
Peoples Friendship University of Russia (RUDN University), 6 Miklukho-Maklaya
Street, Moscow 117198, Russian Federation

###### Abstract

Black hole (BH) solution in the conformal Weyl gravity is a generalization of
the Schwarzschild spacetime which includes two additional constants appearing
when integrating the third order differential equations for gravitational
field. One constant looks like the effective cosmological constant providing
the de Sitter asymptotic of the solution. The other constant allows one to
describe flat rotation of galaxies without introducing of the dark matter.
Here we show that the effective “dark matter” term in the metric function
drastically changes the asymptotic behavior of the evolution of the wave
function of a scalar field: after the Schwarzschild-like ringing phase, the
ringing at another, non-Schwarzschildian, longer-lived frequency takes place
before the beginning of the exponential asymptotic tail. Thus the evolution of
the scalar field consists of the three qualitatively different stages: the
Schwarzschild-like ringing phase, the effective dark matter ringing phase and
the de Sitter phase characterized by exponential tails. The late-time behavior
of the electromagnetic field is qualitatively different as well: the
exponential tails appear even in the absence of the effective de Sitter term.

###### pacs:

04.50.Kd,04.70.-s

##  I Introduction

Evolution of perturbations around black holes can be conditionally divided
into the three stages: initial outburst, quasinormal ringing, which is
followed by power-law (for asymptotically flat spacetimes) or exponential (for
asymptotically de Sitter spacetimes) tails at late times. Although the stage
of quasinormal ringing Konoplya:2011qq ; Kokkotas:1999bd ; Berti:2009kk  is
the most important for current observations of gravitational waves LIGO , the
asymptotic regime at late times, represented by tails, also attracted
considerable interest Price267 ; Bicak268 ; Ching266 ; Ching:1994bd ;
Burko:2004jn ; Brady270 ; Churilova:2019qph ; Konoplya:2006gq ; Rogatko:2008ut
; Gibbons:2008gg ; Gibbons:2008rs ; Rogatko:2007zz ; Moderski:2001gt ; 275Jing
; Brady:1996za ; 274Koyama ; Konoplya:2005et ; Konoplya:2013rxa , because only
by achieving the asymptotic regime one can have the full picture of the
compact object’s response to perturbations and judge about stability, duration
of the ring-down phase and echoes Cardoso:2017cqb .

In a seminal work by R. Price Price267  it was shown that massless scalar and
gravitational fields around the Schwarzschild black hole decay according to
the power law

$|\Psi|\sim t^{-(2\ell+3)}.$ (1)

For Schwarzschild-de Sitter solution the asymptotic decay of the massless
field is not power law anymore, but the exponential one Brady270

$|\Psi_{s}|\sim e^{-\ell k_{c}t},\quad\ell=1,2,...,$ (2)
$|\Psi_{s}|\sim|\Psi_{0}|+|\Psi_{1}|e^{-2k_{c}t},\quad\ell=0.$ (3)

The electromagnetic field falls off in the Schwarzschild-de Sitter background
at asymptotically late times according to the exponential law as well
Molina:2003dc ,

$|\Psi_{el}|\sim e^{-\ell k_{el}t},\quad\ell=1,2,...,$ (4)

where $k_{el}$ is some constant.

An interesting model of gravity is the Weyl conformal gravity Bach , where the
effective cosmological constant appears in the background solution as an
integration constant MK , that is, without introduction of the cosmological
constant into action. The latter has the form

$\displaystyle S=\int d^{4}x\sqrt{-g}C_{abcd}C^{abcd},$ (5)

where $g$ is the determinant of the metric. Birkoff’s theorem holds in the
Weyl conformal gravity as well Riegert . The static and spherically symmetric
vacuum solution describing a black hole in this theory was obtained by
Mannheim and Kazanas MK . This solution depends on the three parameters,
$\beta$, $\gamma$ and $k$, so that the metric function $B(r)$, used in the
line element

$\displaystyle
ds^{2}=-B(r)dt^{2}+B^{-1}(r)dr^{2}+r^{2}(d\theta^{2}+\sin^{2}\theta
d\phi^{2}),$ (6)

can be written in the following form:

$\displaystyle B(r)=1-3\beta\gamma-\frac{2\beta-3\gamma\beta^{2}}{r}+\gamma
r-kr^{2}.$ (7)

Mannheim and Kazanas argued that the Weyl gravity can explain the flat
rotation of galaxies without introducing dark matter, for which $\gamma$ is of
the order of the inverse of the Hubble radius MK . The astrophysical relevance
of this solution was further confirmed in a number of works Islam:2018ymd ;
Dutta:2018oaj ; Christodoulou:2018xxw . This black hole solution MK  has been
recently studied in a number of papers, with the emphasis to lensing and
particle motion Takizawa:2020dja ; Kasikci:2018mtg ; Fathi:2020sey ;
Turner:2020gxo ; Li:2020wvn ; Fathi:2019jgd ; Fathi:2020sfw , thermodynamics
Lanteri:2020trb ; Xu:2018liy  and quasinormal modes Momennia:2019cfd ; Mehrab-
Momennia ; Momennia:2019edt ; Sharif:2020icx .

Quasinormal modes of the Mannheim-Kazanas black hole have been recently
studied in Momennia:2019cfd ; Mehrab-Momennia ; Momennia:2019edt  with some
flaws and omissions. Thus, in Momennia:2019cfd  the wave equation for a test
scalar field was identified with the Pöshl-Teller equation for the near
extremal values of the cosmological constant and it was stated that the
stability of the scalar perturbations is proved. However, as we will show
here, the effective potential at the lowest multipole number $\ell=0$ has a
negative gap which is deeper exactly for the near extremal regime. Therefore,
the fitting to the Pöshl-Teller potential is not possible in this case and,
moreover, the existence of bound states with negative energy leading to
possible instability must be separately studied. Then, in Momennia:2019edt
the obtained master wave equation for gravitational perturbations cannot be
accepted, because the perturbations were fulfilled not in the conformal Weyl
theory, but in another theory allowing for the same family of metrics.

However, the most interesting phenomenon which was omitted in these studies is
connected with the evolution of perturbations at late times. Here we will show
that the decay of a signal in the Mannheim-Kazanas background at late times is
qualitatively different from that in the Schwarzschild or Schwarzschild-de
Sitter cases. We will show that once the effective cosmological term is zero,
the late times tails of the massless scalar field are oscillatory enveloped by
the universal power-law decay. When the effective cosmological constant is
turned on, this oscillatory tail becomes exponential and represents
quasinormal ringing dominated by a non-Schwarzschildian frequency. On the
contrary, electromagnetic perturbations decay according to the exponential law
even when the effective cosmological term is zero.

The paper is organized as follows. In Sec. II we briefly discusses the wave
equations for the scalar and electromagnetic perturbations. Section III
relates the WKB and time-domain integration methods we used as well as the
main results on quasinormal modes and late-time tails. In the conclusions we
summarize the obtained results and discusses the open questions.

##  II The wave equations

The general covariant equation for a massless scalar field has the form

$\frac{1}{\sqrt{-g}}\partial_{\mu}\left(\sqrt{-g}g^{\mu\nu}\partial_{\nu}\Phi\right)=0,$
(8)

while for an electromagnetic field it can be written as follows:

$\frac{1}{\sqrt{-g}}\partial_{\mu}\left(F_{\rho\sigma}g^{\rho\nu}g^{\sigma\mu}\sqrt{-g}\right)=0\,,$
(9)

where $\mu,\nu=0,1,2,3$ and
$F_{\rho\sigma}=\partial_{\rho}A_{\sigma}-\partial_{\sigma}A_{\rho}$ and
$A_{\mu}$ is a vector potential.

After some algebra one can separate the angular variables in Eqs. (8,9) and
rewrite the wave equations in the following general master form

$\dfrac{d^{2}\Psi}{dr_{*}^{2}}+(\omega^{2}-V(r))\Psi=0,$ (10)

in terms of the “tortoise coordinate” $r_{*}$ Konoplya:2011qq :

$dr_{*}=\frac{dr}{f(r)}.$ (11)

The effective potentials for the scalar and electromagnetic fields are:

$V_{scal}(r)=f(r)\left(\frac{\ell(\ell+1)}{r^{2}}+\frac{1}{r}\frac{df(r)}{dr}\right),$
(12) $V_{em}(r)=f(r)\frac{\ell(\ell+1)}{r^{2}}.$ (13)

Thus, $r^{*}\rightarrow-\infty$ corresponds to the black hole event horizon
$r_{+}$.

The effective potentials have the form of a positive definite potential
barrier with a single maximum, except for the case of $\ell=0$ scalar
perturbations, for which the effective potential has a negative gap (see Fig.
1). This means that the stability for this case is not evident and we will
test it in the next section.

Quasinormal modes $\omega_{n}$ correspond to solutions of the master wave
equation (10) with the requirement of the purely outgoing waves at infinity
and purely incoming waves at the event horizon (see, for example,
Konoplya:2011qq ; Kokkotas:1999bd ):

$\Psi_{s}\sim\pm e^{\pm i\omega r^{*}},\quad r^{*}\rightarrow\pm\infty.$ (14)

When the solution is asymptotically de Sitter, purely outgoing waves are
required at the de Sitter horizon instead of infinity.

##  III Quasinormal ringing via the WKB and time-domain integration methods

In order to analyze evolution of perturbations in time-domain we will use the
method for integration of the wave equation in time domain, that is, before
introduction of the stationary ansatz, at a given point in space
Gundlach:1993tp . We will integrate the wavelike equation rewritten in terms
of the light-cone variables $u=t-r_{*}$ and $v=t+r_{*}$. The appropriate
discretization scheme was suggested in Gundlach:1993tp :

$\displaystyle\Psi\left(N\right)$ $\displaystyle=$
$\displaystyle\Psi\left(W\right)+\Psi\left(E\right)-\Psi\left(S\right)-$ (15)
$\displaystyle\Delta^{2}\frac{V\left(S\right)\left(\Psi\left(W\right)+\Psi\left(E\right)\right)}{8}+{\cal
O}\left(\Delta^{4}\right)\,,$

where the following notation for the points were used:
$N=\left(u+\Delta,v+\Delta\right)$, $W=\left(u+\Delta,v\right)$,
$E=\left(u,v+\Delta\right)$ and $S=\left(u,v\right)$. The initial data are
given on the null surfaces $u=u_{0}$ and $v=v_{0}$. This method was used in a
great number of works and proved its efficiency (see for example
Konoplya:2020jgt ; Konoplya:2020bxa ; Konoplya:2019hml ; Churilova:2020bql
and references therein).

In the frequency domain we will use the WKB method of Will and Schutz
Schutz:1985zz , which was extended to higher orders in Iyer:1986np ;
Konoplya:2003ii ; Matyjasek:2017psv  and made even more accurate by the usage
of the Padé approximants in Matyjasek:2017psv ; Hatsuda:2019eoj . The higher-
order WKB formula Konoplya:2019hlu  has the form:

$\omega^{2}=V_{0}+A_{2}({\cal K}^{2})+A_{4}({\cal K}^{2})+A_{6}({\cal
K}^{2})+\ldots-$ $i{\cal K}\sqrt{-2V_{2}}\left(1+A_{3}({\cal
K}^{2})+A_{5}({\cal K}^{2})+A_{7}({\cal K}^{2})\ldots\right),$

where ${\cal K}$ takes half-integer values. The corrections $A_{k}({\cal
K}^{2})$ of order $k$ to the eikonal formula are polynomials of ${\cal K}^{2}$
with rational coefficients and depend on the values of higher derivatives of
the potential $V(r)$ in its maximum. In order to increase accuracy of the WKB
formula, we will follow Matyjasek and Opala Matyjasek:2017psv  and use Padé
approximants.

As both methods are very well known (Konoplya:2019hlu ; Konoplya:2011qq ), we
will not describe them in this paper in more detail, but will simply show that
data obtained by both methods are in a very good agreement.

$\gamma$ | Time-Domain | WKB
---|---|---
0 |  $0.585817-0.193680i$, asymptotic tail |  $0.585691-0.195298i$, $0.529608-0.612039i$
0.1 |  $0.626607-0.198156i$, $0.1086120-0.0316408i$ |  $0.626432-0.199957i$, $0.561854-0.630399i$
0.2 |  $0.666572-0.202136i$, $0.244780-0.057896i$ |  $0.666554-0.202376i$, $0.592440-0.642855i$
0.4 |  $0.745891-0.201371i$, $0.447305-0.0651241i$ |  $0.745880-0.200337i$, $0.650951-0.649154i$
0.6 |  $0.825988-0.181181i$, $0.658562-0.0641326i$ |  $0.825624-0.188484i$, $0.711067-0.624698i$
0.8 |  $0.906628-0.170434i$, $0.862459-0.059459i$ |  $0.907968-0.165472i$, $0.742103-0.570844i$
1 |  $1.05984-0.124847i$, $1.05006-0.0556556i$ |  $0.994072-0.124870i$, $0.767872-0.513711i$
Table 1: Quasinormal modes for scalar $s=0$ perturbations for various values of $\gamma$; $r_{+}=1$ $\ell=1$, $k=10^{-4}$. The second mode in the time domain data represents the second stage of quasinormal ringing induced by the effective dark matter (DM) term. $\gamma$ | Time-Domain | WKB
---|---|---
0 | $0.496520-0.184975i$ | $0.496520-0.184975i$
0.1 | $0.520591-0.193820i$ | $0.520538-0.193898i$
0.2 | $0.543096-0.202234i$ | $0.543098-0.202233i$
0.4 | $0.584386-0.217545i$ | $0.584385-0.217556i$
0.6 | $0.620287-0.231584i$ | $0.620243-0.231503i$
0.8 | $0.648245-0.243741i$ | $0.648244-0.243729i$
1 | $0.661441-0.249998i$ | $0.661437-0.250000i$
Table 2: Fundamental ($n=0$) quasinormal modes for electromagnetic $s=1$
perturbations for various values of $\gamma$; $r_{+}=1$, $\ell=1$, $k=0$.

Figure 1: Effective potentials for scalar perturbations $\gamma=0.2$,
$k=10^{-2}$ (left) and $k=0.39$ (right), $\ell=0$, $r_{+}=1$.  Figure 2:
Logarithmic plot of the time-domain evolution of scalar perturbations
$\gamma=0.5$, $k=0$, $\ell=0$ (left); $r_{+}=1$.  Figure 3: Semilogarithmic
plot of the time-domain evolution of scalar perturbations $\gamma=0.2$,
$k=10^{-4}$, $\ell=1$; $r_{+}=1$.

Figure 4: Semilogarithmic plot of the time-domain evolution of electromagnetic
perturbations for $\gamma=0.05$ (left) and $\gamma=0.1$ (right); $\ell=1$,
$r_{+}=1$, $k=0$. The straight lines represent the exponential decays $\sim
e^{-0.055t}$ (left) and $\sim e^{-0.11t}$ (right).

First of all, the time-domain integration shows that once the effective
cosmological constant is zero, which corresponds to $k=0$, we have the
following decay law of a scalar field at asymptotically late times (see an
example on Fig. 2 for $\ell=0$ perturbations):

$|\Psi_{s}|\sim t^{-3/2}\sin(At),\quad\ell=0,1,2,\ldots,\quad k=0.$ (16)

Here the constant $A$ depends on the black hole parameters. The enveloping
decay law $\sim t^{-3/2}$ is the same for all multipole numbers $\ell$ and
black hole parameters $\gamma$ and $\beta$. The electromagnetic perturbations
decay according to the exponential law at asymptotic times even when the
effective cosmological constant is zero:

$|\Psi_{el}|\sim e^{-Ct},\quad\ell=1,2,\ldots,\quad k=0,$ (17)

where the factor $C$ depends on $\gamma$ and $\ell$ (see, as an example, Fig.
4), for example,

$C\approx 1.1\gamma,\quad\ell=1.$ (18)

When $k$ is not zero, we have the following asymptotic decay law for a
massless scalar field:

$|\Psi_{s}|\sim|\Psi_{0}|+|\Psi_{1}|e^{-p_{s}t},\quad\ell=0,1,\ldots,k\neq 0,$
(19)

where $p_{s}$ depends on the black hole parameters (see, for instance, Fig.
3). The electromagnetic field decays in the presence of the effective de
Sitter term according to the following law:

$|\Psi_{el}|\sim e^{-Ct},\quad\ell=1,2,\ldots,k\neq 0,$ (20)

where the constant $C$ is the same as in the case $k=0$ above at least for
relatively small and moderate values of $k$.

When both $k$ and $\gamma$ are nonzero, the evolution of perturbations of the
scalar field consists of the three stages:

  * •

The quasinormal ringing with the perturbative Schwarzschild frequency, that
with the frequency which slowly changes from its Schwarzschild value when $k$
and $\gamma$ are turned on.

  * •

The second stage of quasinormal ringing with another dominant frequency which
is slower damped than the Schwarzschild one (see Table I). When $\gamma$ goes
to zero, this long-lived frequency goes over into the purely imaginary "mode"
representing the asymptotic de Sitter tail Brady270 . When $k$ vanishes, these
modes are reduced to oscillatory tails enveloped by the power law decay given
by the universal law (Eq. 16).

  * •

The asymptotic exponential tails. Here, unlike the well-known de Sitter tails
Brady270 , the constant $|\Psi_{0}|$ is added to the exponentially decaying
term not only for zero multipole, but also for higher multipoles $\ell$.

Special attention must be paid to the case $\ell=0$ and the near extremal
values of $k$, because the effective potential has a deep negative gap (see
Fig. 1). There are a number of examples when such negative gaps lead to the
unbounded growth of the perturbations, signifying the dynamical instability
Konoplya:2014lha ; Zhu:2014sya ; Konoplya:2008au ; Konoplya:2013sba .
Therefore, the stability must be checked numerically for this case by the
time-domain integration which includes contribution of all the overtones. Any
numerical method in the frequency domain will not exclude the possibility of
missing the mode leading to instability. A recent study of this near extremal
case Momennia:2019cfd  simply ignores the $\ell=0$ modes and claims the
stability based on the positiveness of the effective potential for
$\ell=1,2,...$. Time-domain profiles obtained here for near extremal cases as
well show that the scalar field is apparently stable even for $\ell=0$,
because the wave function decays in time.

It is worth mentioning that the dominant quasinormal modes extracted from the
time domain profiles by the Prony method are in a very good agreement with
those obtained via the 7th order WKB method with further usage of the Padé
approximants as prescribed in Matyjasek:2017psv . This can be seen in data
presented in Tables I and II. The choice of the Padé approximants was such
that the known accurate quasinormal modes of Schwarzschild black hole is
reproduced with the best accuracy, which corresponds to $\tilde{m}=7$, where
$\tilde{m}$ is defined in Konoplya:2019hlu .

##  IV Discussion

In the present paper we considered the evolution of scalar and electromagnetic
perturbations in the vicinity of the Mannheim-Kazanas black hole solution MK
in the conformal Weyl gravity. We found a number of peculiarities, which were
omitted in previous studies of quasinormal modes in this theory in the
frequency domain Momennia:2019cfd ; Mehrab-Momennia ; Momennia:2019edt :

  * •

We have shown that $\ell=0$ scalar field perturbations of the nearly extremal
black holes are governed by the effective potential which has a deep negative
gap, and, that, nevertheless, time-domain profiles are decaying, which points
to the stability of the scalar field.

  * •

When the effective de Sitter term vanishes ($k=0$), the asymptotic tails of
the massless scalar fields are not power-law, as it happens for the
Schwarzschild case Price267 ; Bicak268 , but the oscillatory ones with a
power-law enveloping of oscillations, like it happens for massive fields in
the background of asymptotically flat black holes Burko:2004jn ; 274Koyama .

  * •

When the effective de Sitter term is turned on, this oscillatory tail goes
over into the exponential quasinormal ringing dominated by an essentially non-
Schwarzschildian, longer-lived, frequency.

  * •

The asymptotic decay law at $t\rightarrow\infty$ for the scalar field is,
then, exponential, so that the whole evolution of the signal consists of the
three stages: the first stage of quasinormal ringing at the Schwarzschild-like
frequency, the second stage of quasinormal ringing at the long-lived non-
Schwarzschild frequency and the exponential tail.

  * •

The asymptotic tails for electromagnetic field are exponential even when the
effective cosmological term is tuned off ($k=0$).

Our work could be extended in a number of ways. First of all, one could make
an analytical derivation of the asymptotic behavior via the analysis of the
asymptotic behavior of the wave equations in a similar fashion with Ching266 .
Then the same analysis could be done for the Dirac perturbations describing
the neutrino field. In the latter case we would face the stability problem as
well LopezOrtega:2012hx ; Konoplya:2020zso . A much more complicated problem
would be the case of gravitational quasinormal modes which has not been
performed in Momennia:2019edt , because the Weyl equations were not perturbed
and, instead, perturbation of the same class of metrics, but in an essentially
different theory was considered.

###### Acknowledgements.

The author acknowledges 19-03950S GAČR grant and the RUDN University Program
5-100.

## References

  * (1) R. A. Konoplya and A. Zhidenko, Rev. Mod. Phys. 83, 793 (2011) [arXiv:1102.4014 [gr-qc]].
  * (2) K. D. Kokkotas and B. G. Schmidt, Living Rev. Rel. 2 (1999) 2 [gr-qc/9909058].
  * (3) E. Berti, V. Cardoso and A. O. Starinets, Class. Quant. Grav. 26, 163001 (2009) [arXiv:0905.2975 [gr-qc]].
  * (4) B. P. Abbott et al. Phys. Rev. Lett. 116, no. 6, 061102 (2016) [arXiv:1602.03837 [gr-qc]];
  * (5) Price, R. H., Phys. Rev. D 5, 2419 (1972); Phys. Rev. D5, 2439 (1972).
  * (6) Bicak, J., Gen. Relativ. Gravit. 3, 331 (1972).
  * (7) Ching, E. S. C., Leung, P. T., Suen, W. M. and Young, K., Phys. Rev. D 52, 2118 (1995) [arXiv:gr-qc/9507035].
  * (8) E. S. C. Ching, P. T. Leung, W. M. Suen and K. Young, Phys. Rev. Lett. 74, 2414 (1995) [gr-qc/9410044].
  * (9) L. M. Burko and G. Khanna, Phys. Rev. D 70, 044018 (2004) [gr-qc/0403018].
  * (10) Brady, P. R., Chambers, C. M., Laarakkers, W. G., and Poisson, E., Phys. Rev. D60, 064003 (1999).
  * (11) M. S. Churilova, R. A. Konoplya and A. Zhidenko, Phys. Lett. B 802, 135207 (2020) [arXiv:1911.05246 [gr-qc]].
  * (12) R. A. Konoplya, A. Zhidenko and C. Molina, Phys. Rev. D 75, 084004 (2007) [gr-qc/0602047].
  * (13) M. Rogatko and A. Szyplowska, Gen. Rel. Grav. 41, 1611 (2009) [arXiv:0812.1644 [hep-th]].
  * (14) G. W. Gibbons, M. Rogatko and A. Szyplowska, Phys. Rev. D 77, 064024 (2008) [arXiv:0802.3259 [hep-th]].
  * (15) G. W. Gibbons and M. Rogatko, Phys. Rev. D 77, 044034 (2008) [arXiv:0801.3130 [hep-th]].
  * (16) M. Rogatko and A. Szyplowska, Phys. Rev. D 76, 044010 (2007).
  * (17) R. Moderski and M. Rogatko, Phys. Rev. D 63, 084014 (2001).
  * (18) Jing, J., Phys. Rev. D 72, 027501 (2005) [arXiv:grqc/0408090].
  * (19) P. R. Brady, C. M. Chambers, W. Krivan and P. Laguna, Phys. Rev. D 55, 7538 (1997) [gr-qc/9611056].
  * (20) Koyama, H. and Tomimatsu, A., Phys. Rev. D 63, 064032 (2001) [arXiv:gr-qc/0012022]; Phys. Rev. D 64, 044014 (2001) [arXiv:gr-qc/0103086]; Phys. Rev. D 65, 084031 (2002) [arXiv:gr-qc/0112075].
  * (21) R. A. Konoplya and C. Molina, Phys. Rev. D 71, 124009 (2005) [gr-qc/0504139].
  * (22) R. A. Konoplya and A. Zhidenko, Phys. Rev. D 88, 024054 (2013) [arXiv:1307.1812 [gr-qc]].
  * (23) V. Cardoso and P. Pani, Nature Astron. 1, no. 9, 586 (2017) [arXiv:1709.01525 [gr-qc]].
  * (24) P. D. Mannheim, D. Kazanas, Astrophys. J. 342, 635 (1989).
  * (25) R. Bach, Math. Zeit. 9, 110 (1921).
  * (26) R. J. Riegert, Phys. Rev. Lett. 53, 315 (1984).
  * (27) C. Molina, D. Giugno, E. Abdalla and A. Saa, Phys. Rev. D 69, 104013 (2004) [gr-qc/0309079].
  * (28) T. Islam, Mon. Not. Roy. Astron. Soc. 488, no. 4, 5390 (2019) [arXiv:1811.00065 [gr-qc]].
  * (29) K. Dutta and T. Islam, Phys. Rev. D 98, no. 12, 124012 (2018) [arXiv:1808.06923 [gr-qc]].
  * (30) D. M. Christodoulou and D. Kazanas, Mon. Not. Roy. Astron. Soc. 479, no. 1, L143 (2018) [arXiv:1806.09778 [gr-qc]].
  * (31) K. Takizawa, T. Ono and H. Asada, Phys. Rev. D 102, no. 6, 064060 (2020) [arXiv:2006.00682 [gr-qc]].
  * (32) O. Kaşıkçı and C. Deliduman, Phys. Rev. D 100, no. 2, 024019 (2019) [arXiv:1812.01076 [gr-qc]].
  * (33) M. Fathi, M. Kariminezhad, M. Olivares and J. R. Villanueva, Eur. Phys. J. C 80, no. 5, 377 (2020) [arXiv:2009.03399 [gr-qc]].
  * (34) G. E. Turner and K. Horne, Class. Quant. Grav. 37, no. 9, 095012 (2020).
  * (35) Z. Li, G. Zhang and A. Övgün, Phys. Rev. D 101, no. 12, 124058 (2020) [arXiv:2006.13047 [gr-qc]].
  * (36) M. Fathi, M. Olivares and J. R. Villanueva, Eur. Phys. J. C 80, no. 1, 51 (2020) [arXiv:1910.12811 [gr-qc]].
  * (37) M. Fathi, M. Olivares and J. R. Villanueva, arXiv:2009.03404 [gr-qc].
  * (38) D. Lanteri, S. S. Wan, A. Iorio and P. Castorina, arXiv:2009.14087 [hep-th].
  * (39) H. Xu and M. H. Yung, Phys. Lett. B 793, 97 (2019) [arXiv:1811.07309 [gr-qc]].
  * (40) M. Momennia and S. H. Hendi, Phys. Rev. D 99, no. 12, 124025 (2019) [arXiv:1905.12290 [gr-qc]].
  * (41) M. Momennia, S. H. Hendi, F. S. Bidgoli, Phys. Lett. B, Vol. 813, 136028 (2021)
  * (42) M. Momennia and S. H. Hendi, Eur. Phys. J. C 80, no. 6, 505 (2020) [arXiv:1910.00428 [gr-qc]].
  * (43) M. Sharif and Z. Akhtar, Phys. Dark Univ. 29, 100589 (2020) [arXiv:2005.09430 [gr-qc]].
  * (44) C. Gundlach, R. H. Price and J. Pullin, Phys. Rev. D 49, 883 (1994) [gr-qc/9307009].
  * (45) R. A. Konoplya, A. F. Zinhailo and Z. Stuchlik, Phys. Rev. D 102, no. 4, 044023 (2020) [arXiv:2006.10462 [gr-qc]].
  * (46) R. A. Konoplya and A. F. Zinhailo, Eur. Phys. J. C 80, no. 11, 1049 (2020) [arXiv:2003.01188 [gr-qc]].
  * (47) R. A. Konoplya, A. F. Zinhailo and Z. Stuchlík, Phys. Rev. D 99, no. 12, 124042 (2019) [arXiv:1903.03483 [gr-qc]].
  * (48) M. S. Churilova, Phys. Rev. D 102, no. 2, 024076 (2020) [arXiv:2002.03450 [gr-qc]].
  * (49) B. F. Schutz and C. M. Will, Astrophys. J. 291, L33 (1985).
  * (50) S. Iyer and C. M. Will, Phys. Rev. D 35, 3621 (1987).
  * (51) R. A. Konoplya, Phys. Rev. D 68, 024018 (2003) [gr-qc/0303052].
  * (52) J. Matyjasek and M. Opala, Phys. Rev. D 96, no. 2, 024011 (2017) [arXiv:1704.00361 [gr-qc]].
  * (53) Y. Hatsuda, arXiv:1906.07232 [gr-qc].
  * (54) R. A. Konoplya, A. Zhidenko and A. F. Zinhailo, Class. Quant. Grav. 36, 155002 (2019) [arXiv:1904.10333 [gr-qc]].
  * (55) R. A. Konoplya and A. Zhidenko, Phys. Rev. D 90, no. 6, 064048 (2014) [arXiv:1406.0019 [hep-th]].
  * (56) Z. Zhu, S. J. Zhang, C. E. Pellicer, B. Wang and E. Abdalla, Phys. Rev. D 90, no. 4, 044042 (2014) Addendum: [Phys. Rev. D 90, no. 4, 049904 (2014)] [arXiv:1405.4931 [hep-th]].
  * (57) R. A. Konoplya and A. Zhidenko, Phys. Rev. Lett. 103, 161101 (2009) [arXiv:0809.2822 [hep-th]].
  * (58) R. A. Konoplya and A. Zhidenko, Phys. Rev. D 89, no. 2, 024011 (2014) [arXiv:1309.7667 [hep-th]].
  * (59) A. Lopez-Ortega, Int. J. Mod. Phys. D 21, 1250092 (2012) [arXiv:1211.1801 [gr-qc]].
  * (60) R. A. Konoplya and M. S. Churilova, arXiv:2004.05879 [gr-qc].


[datatype=bibtex, overwrite=true] [fieldsource=booktitle,
match=.*Interspeech.*, replace=Proc. Interspeech] [fieldsource=journal,
match=.*INTERSPEECH.*, replace=Proc. Interspeech] [fieldsource=booktitle,
match=.*ICASSP.*, replace=Proc. ICASSP] [fieldsource=booktitle,
match=.*icassp_inpress.*, replace=Proc. ICASSP (in press)]
[fieldsource=booktitle, match=.*Computer.*Vision.*and.*Pattern.*Recognition.*,
replace=Proc. CVPR] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Acoustics,.*Speech.*and.*Signal.*Processing.*,
replace=Proc. ICASSP] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Learning.*Representations.*,
replace=Proc. ICLR] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Machine.*Learning.*, replace=Proc.
ICML] [fieldsource=booktitle,
match=.*Automatic.*Speech.*Recognition.*and.*Understanding.*, replace=Proc.
ASRU] [fieldsource=booktitle, match=.*Spoken.*Language.*Technology.*,
replace=Proc. SLT] [fieldsource=booktitle,
match=.*Speech.*Synthesis.*Workshop.*, replace=Proc. SSW]
[fieldsource=booktitle, match=.*workshop.*on.*speech.*synthesis.*,
replace=Proc. SSW] [fieldsource=booktitle,
match=.*Advances.*in.*neural.*information.*processing.*, replace=Proc. NIPS]
[fieldsource=booktitle,
match=.*Advances.*in.*Neural.*Information.*Processing.*, replace=Proc.
NeurIPS] [fieldsource=journal,
match=.*Advances.*in.*Neural.*Information.*Processing.*, replace=Proc.
NeurIPS] [fieldsource=booktitle, match=.*Workshop.*on.* Applications.* of.*
Signal.*Processing.*to.*Audio.*and.*Acoustics.*, replace=Proc. WASPAA]
[fieldsource=booktitle,
match=.*International.*Conference.*on.*Language.*Resources.*and.*Evaluation.*,
replace=Proc. LREC] [fieldsource=booktitle,
match=.*International.*Workshop.*on.*Machine.*Learning.*for.*Signal.*Processing.*,
replace=Proc. MLSP] [fieldsource=booktitle,
match=.*European.*Conference.*on.*Computer.*Vision.*, replace=Proc. ECCV]
[fieldsource=booktitle, match=.*Applications.*of.*Computer.*Vision.*,
replace=Proc. WACV] [fieldsource=booktitle,
match=.*Detection.*and.*Classification.*of.*Acoustic.*Scenes.*and.*Events.*,
replace=Proc. DCASE] [fieldsource=booktitle,
match=.*International.*Conference.*on.*Human-Computer.*Interaction.*,
replace=Proc. HCII] [fieldsource=booktitle,
match=.*Symposium.*on.*Interactive.*3D.*Graphics.*and.*Games.*, replace=Proc.
I3D] [fieldsource=publisher, match=.+, replace=] [fieldsource=month, match=.+,
replace=] [fieldsource=location, match=.+, replace=] [fieldsource=address,
match=.+, replace=] [fieldsource=organization, match=.+, replace=]
[fieldsource=doi, match=.+, replace=] [fieldsource=url, match=.+, replace=]
[fieldsource=editor, match=.+, replace=] 1National Institute of Technology,
Tokuyama College, Japan. 2The University of Tokyo, Japan.
3Doshisha University, Japan. 4Ritsumeikan University, Japan.

# Visual onoma-to-wave: environmental sound synthesis
from visual onomatopoeias and sound-source images

###### Abstract

We propose a method for synthesizing environmental sounds from visually
represented onomatopoeias and sound sources. An onomatopoeia is a word that
imitates a sound structure, i.e., the text representation of sound. From this
perspective, onoma-to-wave has been proposed to synthesize environmental
sounds from the desired onomatopoeia texts. Onomatopoeias have another
representation: visual-text representations of sounds in comics,
advertisements, and virtual reality. A visual onomatopoeia (visual text of
onomatopoeia) contains rich information that is not present in the text, such
as a long-short duration of the image, so the use of this representation is
expected to synthesize diverse sounds. Therefore, we propose visual onoma-to-
wave for environmental sound synthesis from visual onomatopoeia. The method
can transfer visual concepts of the visual text and sound-source image to the
synthesized sound. We also propose a data augmentation method focusing on the
repetition of onomatopoeias to enhance the performance of our method. An
experimental evaluation shows that the methods can synthesize diverse
environmental sounds from visual text and sound-source images.

Index Terms—  Environmental sound synthesis, onomatopoeia, visual text, deep
neural network

##  1 Introduction

Environmental sound synthesis aims to synthesize natural and diverse
environmental sounds for sound-content production such as games [1]. Among
several deep learning methods for environmental sound synthesis, synthesis
from sound event labels (i.e., discrete symbols representing sound events) is
the basic approach [2, 3]. This method synthesizes a sound that appropriates
the given label but cannot control the detailed structure of the sound, e.g.,
duration and temporal changes in tone. Another method is synthesis from a text
[4, 5, 6]. For synthesizing sounds, onoma-to-wave [6] (Figure 1 (a)) uses an
onomatopoeia, a word imitating the sound (i.e., the text representation of
sounds). This method can control the detailed structure of the synthesized
sound by the input onomatopoeia text. Also, the overall impression of the
sound can be controlled by conditioning the auxiliary sound event label.

Fig. 1: Method comparison. キイン pronounced /kiin/, which is onomatopoeia of
hitting baseball with metal bat.

Onomatopoeias have another representation: visual-text representation of the
sound. For example, in comics and mangas, visual onomatopoeias are drawn with
arbitrary shapes, sizes, and fonts for delivering sounds to the reader [7]. A
visual onomatopoeia enables dramatic expression and draws the reader in a way
that descriptions of non-textual images (e.g., sounding objects or people)
alone cannot. Visual concepts, e.g., size and text stretch, of a visual
onomatopoeia evokes in the reader sound concepts, e.g., loudness and duration
[8]. Another example is in virtual reality (VR). Presenting visual
onomatopoeias along with environmental sounds in immersive VR can enhance the
immersive qualities of VR [9]. These examples show the possibility of applying
environmental sound synthesis from arbitrary visual onomatopoeias to audible
comics [10] and immersive VR and motivate us to develop a synthesis method.
Also, since visual onomatopoeias frequently pair with the sound source image
to express its sound, we expect that the auxiliary use of the sound source
image will guide the audio synthesis.

In this paper, we propose visual onoma-to-wave, environmental sound synthesis
from visual onomatopoeias. Our method synthesizes sounds from onomatopoeia
visual-text rather than onomatopoeia text. This enables us to control the
detailed structure of the synthesized sound by transferring the visual concept
of the visual onomatopoeia. In other words, we introduce image stretching of
visual onomatopoeias to control the duration of the environmental sound.
Furthermore, we introduce to condition the sound synthesis model by sound
source images for controlling the overall impression of the sound. This image-
based auxiliary control makes it possible to synthesize more diverse sounds
than label-based control with the conventional onoma-to-wave. We also propose
a data augmentation method to enhance the performance of environmental sound
synthesis. We conducted experimental evaluations to investigate 1) whether our
methods appropriately transfer visual concepts to the sound and 2) whether
they synthesizes diverse and natural sounds. The source code and demo are
open-sourced on our project pages111https://sarulab-
speech.github.io/demo_visual-onoma-to-wave/. The contributions of this study
are as follows:

  * •

We propose visual onoma-to-wave, a new task and method of synthesizing
environmental sounds from visual representations of onomatopoeias and sound
sources.

  * •

The proposed method and data augmentation allow synthesizing a more diverse
sound with the same or better naturalness than the conventional method.

##  2 Comparing sound synthesis methods

###  2.1 Sound synthesis from discrete symbols

The basic methods for environmental sound synthesis involve synthesizing from
discrete labels representing sound events, e.g., “cup” or “drum.” [2, 3] Such
methods use sound event labels to represent the overall impression of the
synthesized sounds. The label is suitable in representing the overall
impression of the synthesized sound but has no information to represent the
detailed structure of the sound. In contrast, text is a good means to describe
the detailed structure. There are two types of methods for text-based control:
describing all attributes of the sound in a sentence and separately describing
the overall impression and detailed structure. Yang’s method [4] is of the
former, and onoma-to-wave [6], is of the latter. Since onomatopoeias
efficiently imitate temporal changes of the sound, they are considered
straightforward to control the detailed structure of the sound [11, 12].
Onoma-to-wave also uses labels as auxiliary information in order to control
the overall impression, following the label-based synthesis.

###  2.2 Sound synthesis from images

Methods for speech and sound synthesis from images have been proposed. Image-
to-audio converts visual concepts in given images to synthesized sound [13,
14]. This image-based control of the overall expression of the sound is better
in sound diversity than the aforementioned label-based control. In text-to-
speech, visual text-to-speech (vTTS) replaces text of the standard text-to-
speech with visual text (i.e., text as an image) and converts visual concepts
in given visual text to synthesized speech [15]. This image-based control of
the detailed structure of the speech enables a variety of speech expressions
that are not possible with standard text-to-speech.

In light of these methods, we introduce image representation into onoma-to-
wave to enable the synthesis of diverse environmental sounds. Instead of
onomatopoeia text, onomatopoeia visual-text (visual onomatopoeia) is used to
synthesize. Not only textual but also visual concepts determine the detailed
structure of the sound. Also, instead of a sound event label, a sound event
image is used as an auxiliary to enhance the diversity in overall expression.
Considering the application to comics and VR described in Section 1, the
combination of visual onomatopoeia and auxiliary images is appropriate.
Usually, a visual onomatopoeia corresponds to the source image in comics and
VR. For example, in a baseball comic, the image of a baseball bat and the
visual onomatopoeia of hitting a baseball are adjacent to each other. The bat
image and visual onomatopoeia give the impression of audio to the reader.

##  3 Proposed methods

###  3.1 Visual onoma-to-wave

Basic architecture. Our method uses the Fastspeech2-inspired architecture [16,
15] to learn models from visual onomatopoeias and sound event images as shown
in Figure 2. First, the visual onomatopoeia is sliced into individual
characters. As with vTTS [15], we use visual onomatopoeias artificially
generated from onomatopoeia texts. Assuming a monospaced font of the visual
text, we slice the visual onomatopoeia into $n$ images of $h$-by-$w$ size,
where $n,h,w$ are the number of characters and the height and width of each
visual character, respectively. These are a bit far from the realistic
settings but sufficient to investigate the ideal performance of sound
synthesis from visual onomatopoeias. The sliced images are fed to the visual
feature extractor, followed by the FastSpeech2-inspired encoder, variance
adaptor222Since environmental sounds have no pitch-related feature, our
variance adaptor does not have a pitch predictor that is used in the original
FastSpeech2 in text-to-speech., and decoder. The sound waveform is synthesized
from the generated mel-spectrogram, using a neural vocoder [17].
Conditioning by sound event image. A sound event image is used for controlling
the overall expression of the sound. The pre-trained CLIP [18] image encoder
encodes the input image to the sound event feature. The obtained feature is
added to the output of the variance adaptor. The CLIP pre-training involves a
text-image contrastive objective; the pre-trained encoder is expected to
extract meaningful features from an input image333We used the pre-trained CLIP
image encoder to control the sound. Another control option is using the pre-
trained CLIP text encoder, i.e., controlling the sound by natural language..
For example, a feature extracted from an image of a metal baseball bat
corresponds to the text “a metal baseball bat.” This allows us to synthesize
more diverse sounds than the sound event label, e.g., “a baseball bat.”

Fig. 2: Overall architecture of the proposed visual onoma-to-wave.

###  3.2 Transferring visual effects in visual onomatopoeia to sound

We introduce duration-informed stretch of a visual onomatopoeia for
transferring its visual concept to sound. As explained in Section 1, shrinking
and expanding a visual onomatopoeia changes the duration of the imagined
sound.

Fig. 3: Duration-informed stretch of visual onomatopoeia.

We first calculate the average sounding rate $P$ [character/sec] in each sound
event cluster (sound event category in our study). This value indicates the
number of onomatopoeia characters used for describing $1$-second sound. A
width of a visual onomatopoeia corresponding to $D$-second sound is stretched
at a rate of $P\cdot D$. Figure 3 shows an example of two different
onomatopoeias corresponding to one sound. The onomatopoeias are visualized
with the same shape, regardless of their number of characters. This method is
applied to the training data, and there is no change in training. During
inference, sound duration can be controlled by shrinking or expanding the
visual onomatopoeia.

###  3.3 Data augmentation

Frequent appearance of repetition is a unique characteristic of onomatopoeias
[19], e.g., “ding-ding-ding-…” (bell ringing repeatedly) and “zzzzzz…”
(continued snoring sound). We categorize repetition types into word- (e.g.,
“ding-ding…”) and character-level (e.g., “zzzz…”). Our proposed data
augmentation method synthesizes sounds from visual onomatopoeias444This can be
also applied to the conventional onoma-to-wave. with repetitions.

Fig. 4: Example of our data augmentation for character repetition.

For word-level repetitions, we simply duplicate and concatenate a sound
waveform corresponding to non-repetition onomatopoeias. For example, the
“ding-ding” sound is made by concatenating the “ding” sound. For character-
level repetition, we first find repeating ($\geq$ $3$ times) characters in the
training data and temporally align characters and sound waveform. We chose the
middle part of the continuation, which is assumed more stable than the other
parts, and duplicate the waveform segment at the subsequent position as shown
in Figure 4. This causes discontinuous sounding, but this may be alleviated by
model training.

##  4 Experimental evaluation

###  4.1 Experimental Setup

Dataset. We used pairs of sounds and onomatopoeia texts of $10$ different
sound events from RWCP-SSD [20] (environmental sound corpus) and RWCP-SSD-
Onomatopoeia [21] (onomatopoeia corpus aligning to RWCP-SSD), and used
$13,170$ and $664$ audio samples for training and evaluation, respectively. We
filtered out onomatopoeias with confidence scores [21] lower than $3$ on a
$5$-point scale. Alignment between onomatopoeias and sound waveforms was
obtained by training hidden Markov models using HTK [22]. We confirmed they
were accurately aligned.
Visual onomatopoeias. We used the ahaha-
mojimoji555https://www.flopdesign.com/font7/ahahamojimoji-font font and the
Pillow module of python to artificially generate visual onomatopoeia from the
onomatopoeia text. We used $24$-pt font, so $h=w=24$ when no image stretching
was used. When using image stretching, the stretched image was adjusted to the
same $w$ across all data using zero-padding.
Model and learning configuration. The learning rate was set to $0.001$. In
Sections 4.2.1–4.2.3, we used sound event labels as the auxiliary information
rather than images. We used the same method as for speaker embedding [23] to
obtain 256-dimensional sound event features from the sound event labels. We
used images as the auxiliary information in Section 4.2.4. The sound event
feature was added to the output of the variance adaptor via a linear layer.
Other configurations, e.g., model configuration and training schedule,
followed the open-sourced implementation of vTTS
[15]666https://github.com/Yoshifumi-Nakano/visual-text-to-speech. Original
onoma-to-wave paper [6] uses a Tacotron-inspired synthesis model [24], but in
this paper, we used FastSpeech2-inspired model for both conventional onoma-to-
wave and the proposed visual onoma-to-wave for fair comparison.

###  4.2 Evaluations

In Section 4.2.1, we investigate whether visual onoma-to-wave is comparable in
the basic performance to the conventional onoma-to-wave. In Section
4.2.3–Section 4.2.4 we discuss our data augmentation method, image stretch-
based control, and image-conditioned synthesis, respectively.

Following the onoma-to-wave paper [6], we conducted five-scale mean opinion
score (MOS) evaluations on three criteria: acceptance of synthetic sounds
relative to onomatopoeias, expressiveness of the sounds relative to
onomatopoeia, and naturalness of the sounds. Native Japanese speakers
recruited through crowdsourcing participated in this evaluation. At least $20$
listeners participated in each MOS test.

####  4.2.1 Text input vs. Visual-text input

To evaluate the basic quality of the visual-text input, we compared synthetic
sound samples from text (i.e., onoma-to-wave) or visual text (i.e., visual
onoma-to-wave). We used $664$ samples for each synthesis method.

Table 1: MOS test results (mean$\pm$std) comparing text input and visual-text input. “Reconstructed” indicates reconstructed sound from ground-truth mel-spectrograms by neural vocoder. | Acceptance | Expressiveness | Naturalness
---|---|---|---
Reconstructed | $3.50\pm 1.14$ | $3.25\pm 1.25$ | $3.40\pm 1.15$
Onomatopoeia | $3.61\pm 1.21$ | $3.32\pm 1.08$ | $3.21\pm 1.22$
Visual onomatopoeia | $3.63\pm 1.12$ | $3.42\pm 1.17$ | $3.27\pm 1.22$

Table 1 lists the results. Since there is no statistical significance between
text input and visual-text input for all criteria, we can say that visual
onoma-to-wave is comparable to the conventional onoma-to-wave in basic
performance.

####  4.2.2 Impact of data augmentation

We then evaluated the impact of the proposed data augmentation. For word-level
repetition, the number of repetitions was set from $1$ through $2$ e.g.,
/dong/ $\rightarrow$ /dongdong/ (1) or /dongdongdong/ (2), in training and $0$
through $4$ for evaluation. Short words ($\leq 7$ characters) were augmented.
Namely, the evaluation data include sounds repeated more than in the training
data. For character-level repetition, the number of repetitions was set from
$1$ through $5$, e.g., /dong/ $\rightarrow$ /doong/ (1) or /doooooong/ (5), in
training and $0$ through $10$ for evaluation. We chose $11$ characters to be
augmented, which covers $90$% of repeated characters in the corpus. We
compared synthesized sounds with and without data augmentation. Our main
target is to evaluate the impact of data augmentation on visual onoma-to-wave,
but we also applied data augmentation to the conventional onoma-to-wave. We
used $1,000$ and $2,000$ audio samples for evaluation of word-level and
character-level augmentation, respectively.

For objective evaluation, we calculated sound duration along with the word or
character repetitions. The duration of non-repeated onomatopoeia was
normalized to $1.0$, and the relative changes in duration was calculated. For
subjective evaluation, we conducted MOS tests as in Section 4.1.

Fig. 5: Results of objective and subjective evaluation for data augmentation
of word-level repetition. Fig. 6: Results of objective and subjective
evaluation for data augmentation of character-level repetition.

Figure 5 (a) and Figure 6 (a) show the results of the objective evaluation.
Training without data augmentation did not increase the duration along with
the number of repetitions. Since FastSpeech2 explicitly assigns duration to
each character, the repeated words and characters had non-zero durations.
However, we found that the durations of the repeated ones were very close to
zero; increasing repetitions no longer change the total duration. In contrast,
our data augmentation method solves this problem, and the duration changes
near-linearly.

Figure 5 (b)–(d) and Figure 6 (b)–(d) show the results of the subjective
evaluation. As shown in Figure 5 (d), our data augmentation method did not
lose the MOS in naturalness compared with no data augmentation. In contrast,
from Figure 5 (b)–(d), our data augmentation method had better MOSs in
acceptance and expressiveness than no augmentation. As mentioned above,
synthesized sounds without data augmentation do not respond to the increase in
repetitions. This causes perceptual mismatch between the onomatopoeia and
synthetic sound. Our data augmentation method efficiently solves this and
preserves MOSs even if synthesizing from onomatopoeias with many repetitions.
The same tendency was observed in character-level repetitions, as shown in
Figure 6 (b)–(d), but significant improvements with our data augmentation
method were observed for naturalness.

####  4.2.3 Duration control by image stretching

We evaluated duration control by image stretching. The stretch ratios were set
to ${0.5,1.0,1.5,2.0}$, and $200$ audio samples were used for each stretch
ratio. For comparison, we trained models without pre-processing of image
stretching. The objective evaluation was on the relative duration of the
sound, setting the duration without stretching (i.e., the ratio of $1.0$) to
$1.0$. The subjective evaluation was conducted in the same manner as in
Section 4.2.2.

Fig. 7: Results of objective and subjective evaluations for image stretching-
based duration control.

Figure 7 shows the results. From Figure 7 (a), the sound duration changed
linearly, so we can say that stretching-based duration control performs well.
In contrast, as shown in Figure 7 (b)–(d), while the MOS on naturalness were
maintained among the ratios, those on acceptance and expressiveness degraded
when the ratio was $0.5$. We will investigate the reason for this in the
future.

####  4.2.4 Synthesis of diverse sounds from sound event image

We evaluated the quality and diversity of sound synthesized from visual
onomatopoeias and auxiliary images. We prepared additional sound-onomatopoeia-
image pairs for the evaluation. For sound events in RWCP-SSD, we crawled the
Internet and collected $50$ photos per event, which matched the sounds.
Additionally, we prepared a new sound event class “baseball bat” to evaluate
the synthesized sound. We crawled the Internet to collect $150$ photos of
baseball bats made of metal, wood, and plastic. GameSynth777http://tsugi-
studio.com/web/en/products-gamesynth.html, a sound effect creation tool, was
used to artificially generate the corresponding sounds. The corresponding
onomatopoeias were made from crowdsourcing as in the onoma-to-wave paper [6],
and at least eight onomatopoeias were obtained for each sound. $267$ sound-
onomatopoeia-image pairs were used for evaluation, and the remaining were for
the training data. The RWCP-SSD sound event data was used only for training
and not for evaluation. There was no overlap in sounds, onomatopoeias, and
images between the training and evaluation data. Using these data, we trained
the photo-conditioned synthesis model and evaluated the synthetic sounds. For
application to comics, we also trained the line-drawing-conditioned synthesis
model. Anime2Sketch [25]888https://github.com/Mukosame/Anime2Sketch was used
to convert the obtained photos to the line-drawing style. For comparison, we
also trained a label-conditioned model, i.e., the “baseball bat” label was
used for synthesis, the same as with the conventional onoma-to-wave. We used
ViT-L/14 [26], as the CLIP [18] image
encoder999https://github.com/openai/CLIP. The objective evaluation involved
sound diversity as a mean squared distance between all combinations of the
synthesized sounds [27]. The subjective evaluation was conducted in the same
manner as in Section 4.2.3.

Fig. 8: Sound diversity in synthesis from sound event image. Table 2: MOSs comparing auxiliary information. “Reconstructed” indicates reconstructed sound from ground-truth mel-spectrograms by neural vocoder. Bold means that there is significant difference between photo- or line-drawing-conditioned and label-conditioned. | Acceptance | Expressiveness | Naturalness
---|---|---|---
Reconstructed | $3.60\pm 1.13$ | $3.43\pm 1.16$ | $3.32\pm 1.14$
Line drawing | $\mathbf{2.86\pm 1.25}$ | $\mathbf{2.97\pm 1.12}$ | $3.46\pm 0.90$
Photo | $\mathbf{2.87\pm 1.35}$ | $\mathbf{2.90\pm 1.11}$ | $3.31\pm 0.97$
Label | $2.49\pm 1.11$ | $2.50\pm 1.08$ | $\mathbf{3.70\pm 0.91}$

Figure 8 and Table 2 show the results. From Figure 8, the distributions of
sounds of photo-conditioned and line-drawing-conditioned synthesis were close
to that of the reconstructed sounds. On the other hand, label-conditioned
synthesis was less in diversity compared with the others. We confirmed that
the use of sound event images enables us to synthesize more diverse sounds
than using labels. From Table 2, the photo- and line-drawing-conditioned
synthesis achieved significantly higher MOSs on acceptance and expressiveness
than the label-conditioned synthesis. Therefore, we can say that sound event
images help synthesize appropriate sounds. In contrast, we also observed that
it degrades the naturalness MOSs. This may be because label-conditioned
synthesis uniquely determines features, whereas photo- and line-drawing-
conditioned synthesis do not, thus making the synthesis slightly unstable. We
will address this for future work.

##  5 Conclusion

We proposed visual onoma-to-wave, which uses image representations to
synthesize environmental sounds from onomatopoeias. For future work, we plan
to synthesize sound from real data.

## References

  * [1] D Brandon Lloyd, Nikunj Raghuvanshi and Naga K Govindaraju “Sound synthesis for impact sounds in video games” In _Proc. I3D_ , 2011, pp. 55–62
  * [2] Qiuqiang Kong et al. “Acoustic scene generation with conditional SampleRNN” In _Proc. ICASSP_ , 2019, pp. 925–929
  * [3] Xubo Liu et al. “Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learning” In _Proc. MLSP_ , 2021, pp. 1–6
  * [4] Dongchao Yang et al. “Diffsound: Discrete Diffusion Model for Text-to-sound Generation” In _arXiv preprint arXiv:2207.09983_ , 2022
  * [5] Felix Kreuk et al. “AudioGen: Textually Guided Audio Generation” In _arXiv preprint arXiv:2209.15352_ , 2022
  * [6] Yuki Okamoto et al. “Onoma-to-wave: Environmental sound synthesis from onomatopoeic words” In _APSIPA Transactions on Signal and Information Processing_ 11.1, 2022
  * [7] Jeonghun Baek, Yusuke Matsui and Kiyoharu Aizawa “COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts” In _Proc. ECCV_ , 2022
  * [8] Cheng-Wen Huang and Arlene Archer “Fluidity of modes in the translation of manga: The case of Kishimoto’s Naruto” In _Journal of Visual Communication_ 13.4, 2014, pp. 471–486
  * [9] Jiwon Oh and Gerard J Kim “The effect of onomatopoeia to enhancing user experience in virtual reality” In _Proc. HCII_ , 2019, pp. 143–152
  * [10] Yujia Wang et al. “Comic-guided speech synthesis” In _ACM Transactions on Graphics (TOG)_ 38.6, 2019, pp. 1–14
  * [11] Guillaume Lemaitre and Davide Rocchesso “On the effectiveness of vocal imitations and verbal descriptions of sounds” In _The Journal of the Acoustical Society of America_ 135.2, 2014, pp. 862–873
  * [12] Shiva Sundaram and Shrikanth S Narayanan “Vector-based Representation and Clustering of Audio Using Onomatopoeia Words” In _AAAI Fall Symposium: Aurally Informed Performance_ , 2006, pp. 55
  * [13] Andrew Owens et al. “Visually indicated sounds” In _Proc. CVPR_ , 2016, pp. 2405–2413
  * [14] Yipin Zhou et al. “Visual to sound: Generating natural sound for videos in the wild” In _Proc. CVPR_ , 2018, pp. 3550–3558
  * [15] Yoshifumi Nakano et al. “vTTS: visual-text to speech” In _Proc. SLT_ , 2022
  * [16] Yi Ren et al. “Fastspeech 2: Fast and high-quality end-to-end text to speech” In _arXiv preprint arXiv:2006.04558_ , 2020
  * [17] Jungil Kong, Jaehyeon Kim and Jaekyoung Bae “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis” In _Proc. NeurIPS_ 33, 2020, pp. 17022–17033
  * [18] Alec Radford et al. “Learning transferable visual models from natural language supervision” In _Proc. ICML_ , 2021, pp. 8748–8763
  * [19] Akio Nasu “The word-final moraic obstruent in Japanese mimetics” In _Journal of the Phonetic Society of Japan_ 11.1, 2007, pp. 47–57
  * [20] Satoshi Nakamura et al. “Sound scene data collection in real acoustical environments” In _Journal of the Acoustical Society of Japan (E)_ 20.3, 1999, pp. 225–231
  * [21] Yuki Okamoto et al. “RWCP-SSD-Onomatopoeia: Onomatopoeic Word Dataset for Environmental Sound Synthesis” In _Proc. DCASE_ , 2020, pp. 125–129
  * [22] “Hidden Markov Model Toolkit (HTK)”, https://htk.eng.cam.ac.uk/
  * [23] Chung-Ming Chien et al. “Investigating on incorporating pretrained and learnable speaker representations for multi-speaker multi-style text-to-speech” In _Proc. ICASSP_ , 2021, pp. 8588–8592
  * [24] Yuxuan Wang et al. “Tacotron: Towards end-to-end speech synthesis” In _arXiv preprint 1703.10135_ , 2017
  * [25] Xiaoyu Xiang et al. “Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis” In _Proc. WACV_ , 2022, pp. 1434–1444
  * [26] Alexey Dosovitskiy et al. “An image is worth 16x16 words: Transformers for image recognition at scale” In _arXiv preprint arXiv:2010.11929_ , 2020
  * [27] Hiroki Tamaru et al. “Generative moment matching network-based neural double-tracking for synthesized and natural singing voices” In _IEICE Transactions on Information and Systems_ 103.3, 2020, pp. 639–647