# Detecting quadrupole: a hidden source of magnetic anisotropy for Manganese
alloys

Jun Okabayashi1111Corresponding author, e-mail address;
<EMAIL_ADDRESS>Yoshio Miura2, Yohei Kota3, Kazuya Suzuki4,5,
Akimasa Sakuma6,5,7, and Shigemi Mizukami4,5,7 1Research Center for
Spectrochemistry, The University of Tokyo, 113-0033 Tokyo, Japan  2Research
Center for Magnetic and Spintronic Materials, National Institute for Materials
Science (NIMS), Tsukuba 305-0047, Japan  3National Institute of Technology,
Fukushima Collage, Iwaki, Fukushima 970-8034, Japan  4WPI-Advanced Institute
for Materials Research, Tohoku University, Sendai 980-8577, Japan  5Center for
Spintronics Research Network (CSRN), Tohoku University, Sendai 980-8579, Japan
6Department of Applied Physics, Tohoku University, Sendai 980-8579, Japan
7Center for Science and Innovation in Spintronics (CSIS), Tohoku University,
Sendai 980-8577, Japan

###### Abstract

Mn-based alloys exhibit unique properties in the spintronics materials
possessing perpendicular magnetic anisotropy (PMA) beyond the Fe and Co-based
alloys. It is desired to figure out the quantum physics of PMA inherent to Mn-
based alloys, which have never been reported. Here, the origin of PMA in
ferrimagnetic Mn3-δGa ordered alloys is investigated to resolve antiparallel-
coupled Mn sites using x-ray magnetic circular and linear dichroism
(XMCD/XMLD) and a first-principles calculation. We found that the contribution
of orbital magnetic moments in PMA is small from XMCD and that the finite
quadrupole-like orbital distortion through spin-flipped electron hopping is
dominant from XMLD and theoretical calculations. These findings suggest that
the spin-flipped orbital quadrupole formations originate from the PMA in
Mn3-δGa and bring the paradigm shift in the researches of PMA materials using
x-ray magnetic spectroscopies.

## Introduction

Perpendicular magnetic anisotropy (PMA) is desired for the development of
high-density magnetic storage technologies. Thermal stability of ultrahigh
density magnetic devices is required to overcome the superparamagnetic
limit1-3. Recently, research interests using PMA films have focused on not
only magnetic tunnel junctions4-7 toward the realization of spin-transfer
switching magneto-resistive random-access memories but also antiferromagnetic
or ferrimagnetic devices8,9. To design PMA materials, heavy-metal elements
that possess large spin-orbit coupling are often utilized through the
interplay between the spins in 3$d$ transition-metals (TMs) and 4$d$ or 5$d$
TMs. The design of PMA materials without using the heavy-metal elements is an
important subject in future spintronics researches. Recent progress has
focused on the interfacial PMA in CoFeB/MgO10 or Fe/MgO11,12. However, a high
PMA of over the order of MJ/m3 with a large coercive field is needed to
maintain the magnetic directions during device operation13. Therefore, the
materials using high PMA constants and without using heavy-metal atoms are
strongly desired.

Mn-Ga binary alloys are a candidate that could overcome these issues. Mn3-δGa
alloys satisfy the conditions of high spin polarization, low saturation
magnetization, and low magnetic damping constants14-18. Tetragonal Mn3-δGa
alloys are widely recognized as hard magnets, which exhibit high PMA,
ferromagnetic, or ferrimagnetic properties depending on the Mn composition15.
Two kinds of Mn sites, which couple antiferromagnetically, consist of Mn3-δGa
with the $D0_{22}$-type ordering. Meanwhile, the $L1_{0}$-type Mn1Ga ordered
alloy possesses a single Mn site. These specific crystalline structures
provide the elongated $c$-axis direction, which induces the anisotropic
chemical bonding, resulting in the anisotropy of electron occupancies in 3$d$
states and charge distribution. There are many reports investigating the
electronic and magnetic structures of Mn3-δGa alloys to clarify the origin of
large PMA and coercive field19-21. To investigate the mechanism of PMA and
large coercive fields in Mn3-δGa, site-specific magnetic properties must be
investigated explicitly.

X-ray magnetic circular/ linear dichroism (XMCD/ XMLD) may be a powerful tool
to study the orbital magnetic moments and magnetic dipole moments of higher
order term of spin magnetic moments22,23. However, the difficulty in the
deconvolution of two kinds of Mn sites prevents site-selected detailed
investigations. Within the magneto-optical spin sum rule, using the
formulation proposed by C.T. Chen $et$ $al$.24, the orbital magnetic moments
are expressed as proportional to $r/q$, where $q$ and $r$ represent the
integral of the x-ray absorption spectra (XAS) and XMCD spectra, respectively,
for both $L_{2}$ and $L_{3}$ edges. In the cases of two existing components,
the orbital moments are not obtained from the whole integrals of spectra; by
using each component $r_{1}$, $r_{2}$, $q_{1}$, and $q_{2}$, the value of
$(r_{1}/q_{1})$ \+ $(r_{2}/q_{2})$ should be the average value. The value of
($r_{1}+r_{2}$)/($q_{1}+q_{2}$) does not make sense as an average in the case
of core-level atomic excitation, leading to the wrong value in the XMCD
analysis. As a typical example, for the mixed valence compound CoFe2O4, the
Fe3+ and Fe2+ sites can be deconvoluted by the ligand-field theory
approximation25. However, the deconvolution of featureless line shapes in a
metallic Mn3-δGa case is difficult by comparison with the theoretical
calculations. To detect the site-specific anti-parallel-coupled two Mn sites,
systematic investigations using Mn3-δGa of ${\delta}$ = 0, 1, and 2 provide
the information of site-specific detections. In previous reports, although
Rode $et$ $al.$ performed XMCD measurements of Mn3Ga and Mn2Ga19, the
comparison of XMCD line shapes with that of the single Mn sites in Mn1Ga is
necessary. Recently, the crystal growth techniques of Mn3-δGa were developed
using CoGa buffer layers to synthesize composition-controlled Mn3-δGa thin
films26, which enables discussion of the electronic structures of Mn3-δGa in
${\delta}$ = 0, 1, and 2. We adopted this growth technique and performed XMCD.
By contrast, the XMLD in Mn $L$-edges enables detection of the element of
quadrupole tensor $Q_{zz}$ by adopting the XMLD sum rule27. Although many
reports of XMLD for in-plane magnetic easy axis cases exist, perpendicularly
magnetized cases are attempted firstly in Mn3-δGa using perpendicular remnant
magnetization states.

The first-principles calculations based on the density functional theory (DFT)
suggest the unique band structures in the Mn sites of mixed spin-up and -down
bands at the Fermi level ($E_{\mathrm{F}}$), which allow the spin transition
between up and down spin states, resulting in the stabilization of the
PMA28-30. Usually, the PMA originates from the anisotropy of orbital magnetic
moments in the large exchange-split cases, such as Fe and Co, using second-
order perturbation for spin-orbit interaction31,32. Meanwhile, for Mn
compounds, the contribution from orbital moment anisotropy for PMA is smaller
than the spin-flipped contribution to PMA33-35. However, this picture has not
been guaranteed completely from an experimental viewpoint until now.

In this study, we performed the deconvolution of each Mn site using the
systematic XMCD and XMLD measurements for different Mn contents in Mn3-δGa. We
discuss the site-specific spin and orbital magnetic moments with magnetic
dipoleterm, which corresponds to electric quadrupoles. These are deduced from
the angular-dependent XMCD and XMLD and compared with the DFT calculations to
understand the PMA microscopically.

## Results

### X-ray magnetic spectroscopies

The Mn $L$-edge XAS and XMCD for L10-type Mn1Ga with a single Mn site (MnI),
and D022-type Mn2Ga and Mn3Ga with two kinds of Mn sites (MnI and MnII) are
shown in Fig. 1. The XAS were normalized to be one at the post-edges. With
increasing Mn concentrations (decreasing ${\delta}$), the intensities of XAS
increased and the difference between ${\mu}^{+}$ and ${\mu}^{-}$ became small,
resulting in the suppression of XMCD intensities because of the increase of
antiparallel components. In the case of Mn2Ga and Mn3Ga, the XMCD line shapes
in the $L_{3}$ and $L_{2}$ edges, of slightly split and doublet structures,
became clear because of the increase of another MnII component with opposite
sign. Furthermore, the element-specific hysteresis curves in XMCD at a fixed
photon energy of Mn $L_{3}$-edge exhibit similar features with the results of
the magneto-optical Kerr effects. Coercive fields ($H_{\mathrm{c}}$) of 0.5 T
were obtained for the Mn2Ga and Mn3Ga cases because the two kinds of Mn sites
enhance the antiparallel coupling.

To deconvolute the MnI and MnII sites in the XMCD spectra, we performed the
subtraction of XMCD between Mn1Ga and Mn3Ga. Figure 2 displays the XMCD of
Mn1Ga and Mn3Ga, and their differences after the normalization considering the
Mn compositions. The XMCD signal with opposite sign was clearly detected for
MnI and MnII components. As the lattice volume of Mn3-δGa on the CoGa buffer
layer remained almost unchanged with different $\delta$ 26, the validity of
the subtraction of XMCD is warranted because the density of states (DOS) for
MnI is similar in all ${\delta}$ regions as shown in latter (Fig. 4). To apply
the magneto-optical sum rule for effective spin magnetic moments
($m_{s}^{\mathrm{eff}}$) including magnetic dipole term and orbital magnetic
moments ($m_{\mathrm{orb}}$), the integrals of the XMCD line shapes are
needed24. Further, the integrals of XAS were also estimated for MnI and MnII,
divided by the composition ratios. The electron numbers for 3$d$ states of MnI
and MnII were estimated from the band-structure calculations to be 5.795 and
5.833, respectively. Thus, $m_{s}^{\mathrm{eff}}$ and $m_{\mathrm{orb}}$ for
MnI were estimated to be 2.30 and 0.163 ${\mu}_{\mathrm{B}}$, respectively.
For MnII, 2.94 ${\mu}_{\mathrm{B}}$ ($m_{s}^{\mathrm{eff}}$) and 0.093
${\mu}_{\mathrm{B}}$ ($m_{\mathrm{orb}}^{\perp}$) were obtained for
perpendicular components with the error bars of 20 % because of the
ambiguities estimating spectral background.

Here, we claim the validity of $m_{s}^{\mathrm{eff}}$ and $m_{\mathrm{orb}}$
in Mn3-δGa deduced from XMCD. First, these $m_{\mathrm{orb}}$ values are too
small to explain stabilizing the PMA because the magnetic crystalline energy
$E_{\mathrm{MCA}}{\propto}{\frac{1}{4}}{\alpha}{\xi}_{\mathrm{Mn}}$
$(m_{\mathrm{orb}}^{\perp}-m_{\mathrm{orb}}^{\parallel})$ within the scheme of
the Bruno relation32, assuming the spin-orbit coupling constant
$\xi_{\mathrm{Mn}}$ of 41 meV and the band-state parameter ${\alpha}=0.2$ for
Mn compounds, which is estimated from the DFT calculation. For Mn1Ga, as the
saturation magnetic field along hard axis direction was less than 1 T, the
projected component $m_{\mathrm{orb}}^{\parallel}$ could be deduced as
${\Delta}m_{\mathrm{orb}}(=m_{\mathrm{orb}}^{\perp}-m_{\mathrm{orb}}^{\parallel})$
of less than 0.01 ${\mu}_{\mathrm{B}}$, resulting in
$E_{\mathrm{MCA}}=1{\times}10^{-5}$ eV/atom, that is, 5.7 $\times$ 104 J/m3
using the unit cell of MnGa. Therefore, orbital moment anisotropy cannot
explain the PMA of the order of 106 J/m3 in Mn3-δGa26. As the electron
configuration is close to the half-filled 3$d^{5}$ case, the quenching of the
orbital angular momentum occurs in principle. In Mn3-δGa, since the electron
filling is not complete half-filled cases, small orbital angular momentum
appears. Second, another origin for the large PMA is considered as the spin-
flipped contribution between the spin-up and -down states in the vicinity of
the $E_{\mathrm{F}}$. The magnetic dipole term ($m_{\mathrm{T}}$) also
stabilizes the magneto-crystalline anisotropy energy ($E_{\mathrm{MCA}}$) by
the following equation33:

$E_{\mathrm{MCA}}{\simeq}{\frac{1}{4}}{\alpha}{\xi}{\Delta}m_{\mathrm{orb}}-\frac{21}{2}\frac{{\xi}^{2}}{{\Delta}E_{\mathrm{ex}}}{\Delta}m_{\mathrm{T}},$
(1)

where ${\Delta}E_{\mathrm{ex}}$ denotes the exchange splitting of 3$d$ bands.
Positive values of $E_{\mathrm{MCA}}$ stabilize the PMA. The second term
becomes dominant when proximity-driven exchange split cases, such as the 4$d$
and 5$d$ states, are dominant36,37. In the case of Mn3-δGa, the Mn 3$d$ states
were delicate regarding the mixing of the spin-up and -down states at the
$E_{\mathrm{F}}$, which corresponds to the quadrupole formation and the band
structure ${\alpha}$ values. The second term is expressed by $m_{\mathrm{T}}$
in the XMCD spin sum rule of $m_{s}+7m_{\mathrm{T_{z}}}$ along the out-of-
plane $z$ direction38. For Mn1Ga, if $m_{T_{z}}$ is negative, resulting in
$Q_{zz}\verb|>|$ 0 in the notation of $m_{\mathrm{Tz}}=-Q_{zz}{\cdot}m_{s}$,
which exhibits the prolate shape of the spin density distribution; the second
term favors PMA because of the different sign for the contribution of orbital
moment anisotropy in the first term. Since 7$m_{\mathrm{Tz}}$ is estimated to
be in the order of 0.1 ${\mu}_{\mathrm{B}}$ from angular-dependent XMCD
between surface normal and magic angle cases, $Q_{zz}$ is less than 0.01,
resulting that the orbital polarization of less than 1% contributes to
stabilize PMA. In this case, the contribution of the second term in eq.(1) is
one order larger than the orbital term, which is essential for explaining the
PMA of Mn3-δGa. Third, in a previous study19, quite small
${\Delta}m_{\mathrm{orb}}$ and negligible $m_{\mathrm{T_{z}}}$ were reported
for Mn2Ga and Mn3Ga. Their detailed investigation claims that
${\Delta}m_{\mathrm{orb}}$ of 0.02 ${\mu}_{\mathrm{B}}$ in MnI site
contributes to PMA and MnII site has the opposite sign. These are
qualitatively consistent with our results. The difference might be derived
from the sample growth conditions and experimental setup. Fourth, the reason
why $H_{\mathrm{c}}$ in Mn1Ga is small can be explained by the $L1_{0}$-type
structure, due to the stacking of the Mn and Ga layers alternately, which
weakened the exchange coupling between the Mn layers. Finally, we comment on
the XMCD of the Ga $L$-edges. This also exhibits the same sign as the MnI
component, suggesting that the induced moments in the Ga sites were derived
from the MnI component (Fig. S1), which was substituted by the MnII for Mn2Ga
and Mn3Ga.

To determine the effect of $m_{\mathrm{T_{z}}}$, we performed XMLD
measurements. Figure 3 shows the $\bf{E}$ vector polarization dependent XAS,
where $\bf{E}$ is perpendicular and horizontal to the magnetization direction.
After magnetizing perpendicular to easy-axis direction by the pulse of 1 T,
the XMLD was measured at the remnant states. The XMLD between the vertical and
horizontal polarized excitations were detected in grazing incident beams,
where the sample surface normal is tilted 60∘ from the incident beam. The
differential line shapes were similar to those of other Mn compounds39,40. We
estimate that the x-ray linear dichroism (XLD) components are less than 20 %
by the measurements at the same geometry without magnetizing. With increasing
Mn composition, the XMLD signal intensities were enhanced because XMLD detects
the ${\langle}M^{2}\rangle$ contribution. In Mn3Ga, XMLD includes the
summation of both
${\langle}M_{\mathrm{MnI}}^{2}\rangle+{\langle}M_{\mathrm{MnII}}^{2}\rangle$
contributions. Therefore, the contributions from XLD are not dominant factor
in spectral analyses. We note that the integrals of the XMLD line shapes are
proportional to $Q_{zz}$ along the sample surface normal direction. We
confirmed that the integral converges to a positive value, deducing that the
sign of $Q_{zz}$ is positive with the order of 0.01 for both MnI and MnII
components by applying the XMLD sum rule in the notation of
$m_{\mathrm{Tz}}=-Q_{zz}{\langle}S{\rangle}$; that is, 3$z^{2}-r^{2}$ orbitals
are strongly coupled with $\bf{E}$ and are elongated to an easy-axis direction
after subtracting the XLD contribution. The detail of estimation of $Q_{zz}$
is explained in the Supplemental Note and ref 41. This value is consistent
with that estimated from XMCD spin sum rule. These suggest the orbital
polarization of Mn 3$d$ states along $z$-axis direction forming the cigar-type
prolate unoccupied orbital orientation. Therefore, combining both XMCD and
XMLD, the order of $m_{\mathrm{Tz}}$ can be estimated as two order smaller
than the spin moments.

### DFT calculation

Figure 4 shows the DOS of Mn1Ga and Mn3Ga with site and orbital-resolved
contributions by the DFT calculation. The contributions for the
$E_{\mathrm{MCA}}$ of each atom and spin transition processes through the
second-order perturbation of the spin-orbit interaction are also shown for the
MnI and MnII sites using a tetragonal unit cell. In the DOS of the MnI and
MnII sites, all orbital states were split through exchange interaction.
However, exchange splitting was incomplete where complete spin splitting was
required, in the Bruno formula32, which enabled the transitions by spin mixing
between occupied spin-up and unoccupied spin-down states. Four types of spin
transition processes occurred between the occupied and unoccupied states, as
shown in the bottom panel of Fig. 4. The positive values in the presented bar
graphs stabilize the PMA. The ’up-down’ process implies a virtual excitation
from an occupied up-spin state to an unoccupied down-spin state in the second-
order perturbation, which forms the magnetic dipoles. While spin conserved
transition terms are slightly positive, spin-flipped transition terms show
dominant contribution to PMA. The further orbital-resolved anatomy of the
spin-flipped transition revealed the transition between the $yz$ and 3$z^{2}$
states, which induces the prolate-type spin distribution. The matrix elements
of $L_{x}$ between $m={\pm}1$ and 0 through the transitions of different
magnetic quantum numbers $m$ were predominant for the MnI site in Mn1Ga (see
Supplemental Materials). As shown in Fig. S2, the matrix elements of $L_{x}$
between $yz$ and 3$z^{2}$ in spin-flipped transition (blue bar graph) have
large positive values, indicating contribution to the PMA. Meanwhile, for
$D0_{22}$-type Mn3Ga, the contributions to MCA energy were different. Although
the spin-conserved transition terms are enhanced as compared with Mn1Ga, the
spin-flipped contributions were still dominant, which explains the suppression
of the orbital moment anisotropy in Mn1Ga. The MnI 3$z^{2}$ orbitals were
located near the $E_{\mathrm{F}}$ in Mn3Ga in the spin-down states, strongly
affecting the appearance of the finite matrix elements. Figure S3 shows the
large matrix elements of $L_{x}$ between $yz$ and 3$z^{2}$ in the spin-flipped
transition of Mn I, which are similar to those of Mn1Ga. The difference in the
MnII sites between Mn1Ga and Mn3Ga can be derived from the location of
neighboring Mn atoms, which promotes exchange interaction between the MnI and
MnII sites.

For Mn1Ga, the orbital moments along the $c$\- and $a$-axis,
$m_{\mathrm{orb}}^{z}$ and $m_{\mathrm{orb}}^{x}$, were estimated to be 0.022
and 0.0207 ${\mu}_{\mathrm{B}}$, respectively, by the DFT calculation. The
magnetic dipole moments of $m_{\mathrm{T_{z}}}$ and $m_{\mathrm{T_{x}}}$ were
$-0.039$ and $-0.022$ ${\mu}_{\mathrm{B}}$, respectively. Using Eq. (1),
${\Delta}m_{\mathrm{orb}}$ and ${\Delta}m_{\mathrm{T}}$ were estimated to be
$0.014$ ${\mu}_{\mathrm{B}}$ and $-0.017$ ${\mu}_{\mathrm{B}}$, respectively.
We note that $Q_{zz}$ was estimated to be 0.015 from the DFT calculation,
which is similar to the results of the XMLD through the relation of
$m_{\mathrm{T}}=-Q_{ij}{\cdot}S$. These values provide the anisotropic
energies of the first and second terms in Eq. (1) as $0.014$ meV and 0.13 meV,
respectively, by using ${\Delta}E_{\mathrm{ex}}$ of 2.3 eV and
${\xi}_{\mathrm{Mn}}$ of 41 meV for Mn atoms. The amplitude of the spin-
flipped term is larger than the orbital moment anisotropy to stabilize the PMA
energetically. These estimations are consistent with the values deduced from
the XMCD and XMLD analyses. Eq. (1) was modified using the energy difference
of each component $i$: ${\Delta}E_{i}={E_{i}}^{\perp}-{E_{i}}^{\parallel}$ and
the relation of:
$E_{\mathrm{MCA}}={\Delta}E_{L{\uparrow}}+{\Delta}E_{L{\downarrow}}+{\Delta}E_{\mathrm{T}}+{\Delta}E_{\mathrm{LS}}$,
where $E_{\mathrm{MCA}}$ is expressed by the summation of the orbital parts of
spin up and down states (${\Delta}E_{L{\uparrow}}+{\Delta}E_{L{\downarrow}}$),
the $m_{\mathrm{T}}$ term, and the residual of ${\Delta}E_{\mathrm{LS}}$33. We
confirmed that electron number dependence mainly obeys the
${\Delta}E_{\mathrm{T}}$ term (Fig. S4). Therefore, the finite value of
$T_{z}$, which contributes to the second term in Eq. (1), is indispensable for
the PMA in MnGa.

## Discussion

Considering the results of the XMCD, XMLD, and DFT calculation, we discuss the
origin of PMA in Mn3-δGa. As the orbital magnetic moments and their
anisotropies are small, the contribution of the first term in Eq. (1) is also
small, which is a unique property of Mn alloy compounds and contradicts the
cases of Fe and Co compounds exhibiting PMA. Beyond Bruno’s formula31, the
mixing of majority and minority bands in Mn 3$d$ states enables the spin-
flipped transition and $Q_{zz}$. However, comparing with the CoPd or FePt
cases, where the exchange splitting was induced in the 4$d$ or 5$d$ states, a
small ${\xi}_{\mathrm{Mn}}$ and large ${\Delta}E_{\mathrm{ex}}$ in the Mn 3$d$
states suppress the contribution of the second term. Large $Q_{zz}$ values
were brought by the crystalline distortion accompanied by the anisotropic spin
distribution, resulting in the PMA energy of Mn3-δGa exhibiting a similar
order with those in heavy-metal induced magnetic materials. Therefore, the
large PMA in Mn3-δGa originates from the specific band structure of the Mn
3$d$ states, where the orbital selection rule for the electron hopping through
spin-flipped ${\langle}yz{\uparrow}|L_{x}|z^{2}{\downarrow}{\rangle}$ provides
the cigar-type spin distribution28. As the spin-flipped term of
${\Delta}E_{\mathrm{T}}+{\Delta}E_{\mathrm{LS}}(={\Delta}E_{\mathrm{sf}})$ for
PMA energy, except the orbital contributions, can be written as:

${\Delta}E_{\mathrm{sf}}=\sum_{u{\uparrow},o{\downarrow}}\frac{{\xi}^{2}}{{\Delta}E_{\mathrm{ex}}}[{\langle}u{\uparrow}|{L_{x}}^{2}|o{\downarrow}{\rangle}-{\langle}u{\uparrow}|{L_{z}}^{2}|o{\downarrow}{\rangle}]$
(2)

the difference between the $L_{x}^{2}$ and $L_{z}^{2}$ terms through the spin-
flipped transitions between the occupied ($o$) to unoccupied ($u$) states is
significant for the gain of the PMA energy. The matrix elements of
${\langle}u{\uparrow}|{L_{x}}^{2}|o{\downarrow}{\rangle}$ were enhanced in the
spin-flipped transition between $yz$ and $z^{2}$, and those of
${\langle}u|{L_{z}}^{2}|o{\rangle}$ were enhanced in the spin-conserved case
between $xy$ and $x^{2}-y^{2}$ 28. These transitions favor the magnetic dipole
moments of prolate shapes
(${\langle}Q_{zz}{\rangle}={\langle}3L_{z}^{2}-L^{2}{\rangle}\verb|>|0$)
described by the Mn 3$d$ each orbital angular momenta. We emphasize that the
signs of ${\Delta}m_{\mathrm{orb}}$ and $Q_{zz}$ are opposite, which is
essential to stabilize the PMA by the contribution of the second term in Eq.
(1). The PMA energy of FePt exhibits around MJ/m3 and the contribution of the
second term in Pt is four times larger than the Fe orbital anisotropy
energy28. Therefore, MnGa has a specific band structure by crystalline
anisotropy elongated to the $c$-axis and intra-Coulomb interaction in Mn sites
to enhance the PMA without using heavy-metal atoms.

In conclusion, we investigated the origin of PMA in Mn3-δGa by decomposing
into two kinds of Mn sites for XMCD, XMLD, and the DFT calculation. The
contribution of the orbital moment anisotropy in Mn3Ga is small and that of
the mixing between the Mn 3$d$ up and down states is significant for PMA,
resulting in the spin-flipped process through the electron hopping between
finite unforbidden orbital symmetries in the 3$d$ states through the quadratic
contribution. Composition dependence reveals that the orbital magnetic moments
of the two antiparallel-coupled components in Mn sites were too small to
explain the PMA. These results suggest that the quadrupole-like spin-flipped
states through the anisotropic $L1_{0}$ and $D0_{22}$ crystalline symmetries
are originated to the PMA in Mn3-δGa. The present study provides a promising
strategy to investigate quadrupoles in antiferro or ferrimagnetic materials
with PMA.

## MATERIALS AND METHODS

### Sample growth and characterization

The samples were prepared by magnetron sputtering. The 40-nm-thick Cr buffer
layers were deposited on single-crystal MgO (001) substrates at room
temperature (RT), and in situ annealing at 700 ∘C was performed. Subsequently,
a 30-nm-thick Co55Ga45 buffer layer was grown at RT with $in$-$situ$ annealing
at 500 ∘C, then 3-nm-thick Mn3-δGa layers were grown at RT. Here, the
composition of Mn3-δGa films were controlled by Ar gas pressure during
deposition and co-sputtering technique with MnGa and Mn target. Finally, a
2-nm-thick MgO capping layer was deposited. Using the CoGa buffer layer,
ultra-thin Mn3-δGa layer deposition was achieved26. Curie temperatures of
samples are higher than RT. Magnetic anisotropy energy of Mn1Ga and Mn2Ga was
estimated to be approximately 0.13 and 0.9 MJ/m3, respectively, at RT, by
using vibrational sample magnetometer and tunnel magnetoresistance curves
shown in the Ref. 26.

### XMCD and XMLD measurements

The XMCD and XMLD were performed at BL-7A and 16A in the Photon Factory at the
High-Energy Accelerator Research Organization (KEK). For the XMCD
measurements, the photon helicity was fixed, and a magnetic field of $\pm$1.2
T was applied parallel to the incident polarized soft X-ray beam, defined as
${\mu}^{+}$ and ${\mu}^{-}$ spectra. The total electron yield mode was
adopted, and all measurements were performed at room temperature. The XAS and
XMCD measurement geometries were set to normal incidence, so that both the
photon helicity and the magnetic field were parallel and normal to the
surface, enabling measurement of the absorption processes involving the normal
components of the spin and orbital angular momenta36. In the XMLD
measurements, the remnant states magnetized to PMA were adopted. For grazing
incident measurements in XMLD and XLD, the angle between incident beam and
sample surface normal was kept at 60∘ tilting as shown in the inset of Fig. 3.
The direction of the electric field of the incident synchrotron beam $\bf{E}$
was tuned horizontally and vertically with respect to the magnetization
$\bf{M}$. We define the sign of XMLD by the subtraction of the
($\bf{M}{\parallel}{\bf{E}}$)$-$(${\bf{M}}{\perp}{\bf{E}}$) spectra35.

### First-principles calculation

The first-principles calculations of MCA energies for Mn1Ga and Mn3Ga were
performed using the Vienna $ab$ $initio$ simulation package (VASP). We
calculated the second-order perturbation of the spin-orbit interaction to MCA
energies for each atomic site using wave functions in the VASP calculations.
Details of the perturbation calculation are described in Ref. 42. In this
paper, we estimated the spin-orbit coupling constant of Mn and Ga atom as 41
and 35 meV, respectively, by the calculation.

###### Acknowledgements.

This work was partially supported by JSPS KAKENHI (Grant No. 16H06332), and
Spintronics Research Network of Japan. Part of research was performed under
the support from Toyota Physical and Chemical Research Institute. Parts of the
synchrotron radiation experiments were performed under the approval of the
Photon Factory Program Advisory Committee, KEK (Nos. 2017G060 and 2019G028).

## Author Contributions

J. O. and S. M. planned the study. Kz. S. prepared the samples and
characterized the properties. J.O. set up the XMCD and XMLD measurement
apparatus at Photon Factory and collected and analyzed the data. Y.M.
performed the first-principles calculation. Y. K. and A. S. also performed the
density-functional-theory calculation. J.O and Y.M. constructed the scenario
of quadrupole physics. All authors discussed the results and wrote the
manuscript.

## Competing interests

The authors declare no competing financial interests.

## Data availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * (1)
  * (2)

## References

* (3) Wolf, S. A. et al. Spintronics: A Spin-Based Electronics Vision for the Future, $Science$ $\bf{294}$, 1488 (2001).
* (4) Dieny, B. & Chshiev, M. Perpendicular magnetic anisotropy at transition metal/oxide interfaces and applications, $Rev.$ $Mod.$ $Phys.$ $\bf{89}$, 025008 (2017).
* (5) Mangin, S. et al. Current-induced magnetization reversal in nanopillars with perpendicular anisotropy, $Nat.$ $Mater.$ $\bf{5}$, 210 (2006).
* (6) Locatelli, N. Cros, V. & Grollier, J. Spin-torque building blocks, $Nat.$ $Mater.$ $\bf{13}$, 11 (2014)
* (7) Kwak, W.-Y. Kwon, J.-H, Grünberg, P. Han, S. H. & Cho, B. K. Current-induced magnetic switching with spin-orbit torque in an interlayer-coupled junction with a Ta spacer layer, $Sci.$ $Rep.$ $\bf{8}$, 3826 (2018).
* (8) Worledge, D. C. et al. Spin torque switching of perpendicular Ta/CoFeB/MgO -based magnetic tunnel junctions, $Appl.$ $Phys.$ $Lett.$ $\bf{98}$, 022501 (2011).
* (9) Kurebayashi, H. et al. An antidamping spin-orbit torque originating from the Berry curvature, $Nature$ $Nanotech.$ $\bf{9}$, 211 (2014).
* (10) Fukami, S. et al. Magnetization switching by spin-orbit torque in an antiferromagnet-ferromagnet bilayer system, $Nat.$ $Mater.$ $\bf{15}$, 535 (2016).
* (11) Marti, X. et al. Room-temperature antiferromagnetic memory resistor, $Nat.$ $Mater.$ $\bf{13}$, 367 (2014).
* (12) Ikeda, S. et al. A perpendicular-anisotropy CoFeB-MgO magnetic tunnel junction, $Nat.$ $Mater.$ $\bf{9}$, 721 (2010).
* (13) Koo, J. W. et al. Large perpendicular magnetic anisotropy at Fe/MgO interface, $Appl.$ $Phys.$ $Lett.$ $\bf{103}$, 192401 (2013).
* (14) Okabayashi, J. et al. Perpendicular magnetic anisotropy at the interface between ultrathin Fe film and MgO studied by angular-dependent x-ray magnetic circular dichroism, $Appl.$ $Phys.$ $Lett.$ $\bf{105}$, 122408 (2014).
* (15) Mizukami, S. Sakuma, A. Sugihara, A. Suzuki, K.Z. & Ranjbar, R. Mn-based hard magnets with small saturation magnetization and low spin relaxation for spintronics, $Scripta$ $Materia.$ $\bf{118}$, 70 (2016).
* (16) Mizukami, S. et al. Long-Lived Ultrafast Spin Precession in Manganese Alloys Films with a Large Perpendicular Magnetic Anisotropy, $Phys.$ $Rev.$ $Lett.$ $\bf{106}$, 117201 (2011).
* (17) Mizukami, S. et al. Composition dependence of magnetic properties in perpendicularly magnetized epitaxial thin films of Mn-Ga alloys, $Phys.$ $Rev.$ $B$ $\bf{85}$, 014416 (2012).
* (18) Wu, F. et al. Epitaxial thin films with giant perpendicular magnetic anisotropy for spintronic devices, $Appl.$ $Phys.$ $Lett.$ $\bf{94}$, 122503 (2009).
* (19) Zhu, L. et al. Multifunctional $L1_{0}$-Mn1.5Ga Films with Ultrahigh Coercivity, Giant Perpendicular Magnetocrystalline Anisotropy and Large Magnetic Energy Product, $Adv.$ $Mater.$ $\bf{24}$, 4547 (2012).
* (20) Kurt, H. Rode, K. Venkatesan, M. Stamenov, P. & Coey, J. M. D. Mn3-xGa (0$\leq$x$\leq$1): Multifunctional thin film materials for spintronics and magnetic recording, $Phys.$ $Status$ $Solidi$ $B$ $\bf{248}$, 2338 (2011),
* (21) Rode, K. et al. Site-specific order and magnetism in tetragonal Mn3Ga thin films, $Phys.$ $Rev.$ $B$ $\bf{87}$, 184429 (2013)
* (22) Glas, M. et al. X-ray absorption spectroscopy and magnetic circular dichroism studies of $L1_{0}$-Mn-Ga thin films, $J.$ $Appl.$ $Phys.$ $\bf{114}$, 183910 (2013).
* (23) Oshima, D. et al. Ion Irradiation-Induced Magnetic Transition of MnGa Alloy Films Studied by X-Ray Magnetic Circular Dichroism and Low-Temperature Hysteresis Loops, $IEEE$ $Trans.$ $Magn.$ $\bf{52}$, 1 (2016).
* (24) Thole, B. T. Carra, P. Sette, F. & van der Laan, G. X-ray circular dichroism as a probe of orbital magnetization, $Phys.$ $Rev.$ $Lett.$ $\bf{68}$, 1943 (1992).
* (25) Carra, P. Thole, B. T. Altarelli, M. & Wang, X. X-Ray Circular Dichroism and Local Magnetic Fields, $Phys.$ $Rev.$ $Lett.$ $\b{3}{70}$, 694 (1993).
* (26) Chen, C.T. et al. Experimental confirmation of the X-ray magnetic circular dichroism sum rules for iron and cobalt, $Phys.$ $Rev.$ $Lett.$ $\bf{75}$, 152 (1995).
* (27) Moyer, J. A. et al. Magnetic structure of Fe-doped CoFe2O4 probed by x-ray magnetic spectroscopies, $Phys.$ $Rev.$ $B$ $\bf{84}$, 054447 (2011).
* (28) Suzuki, K.Z. et al. Perpendicular magnetic tunnel junction with a strained Mn-based nanolayer, $Sci.$ $Rep.$ $\bf{6}$, 30249 (2016).
* (29) Carra, P. König, H. Thole, B.T. & Altarelli, M. Magnetic X-ray dichroism General features of dipolar and quadrupolar spectra, $Physica$ $B$ $\bf{192}$, 182 (1993).
* (30) Kota, Y. & Sakuma, A. Mechanism of Uniaxial Magnetocrystalline Anisotropy in Transition Metal Alloys, $J.$ $Phys.$ $Soc.$ $Jpn.$ $\bf{83}$, 034715 (2014).
* (31) Kim, D. Hong, j. & Vitos, L. Epitaxial strain and composition-dependent magnetic properties of MnxGa1-x alloys, $Phys.$ $Rev.$ $B$ $\bf{90}$, 144413 (2014).
* (32) Yun, W. S. Cha, G. B. Kim, I. G. Rhim, S. H. & Hong, S. C. Strong perpendicular magnetocrystalline anisotropy of bulk and the (001) surface of $D0_{22}$Mn3Ga: a density functional study, $J.$ $Phys.:$ $Condens.$ $Matter$ $\bf{24}$, 416003 (2012).
* (33) Yoshida, K. Okiji, A. & Chikazumi, S. Magnetic Anisotropy of Localized State in Metals, $Prog.$ $Theoretical$ $Phys.$ $\bf{33}$, 559 (1965).
* (34) Bruno, P. Tight-binding approach to the orbital magnetic moment and magnetocrystalline anisotropy of transition-metal monolayers, $Phys.$ $Rev.$ $B$ $\bf{39}$, 865(R) (1989).
* (35) van der Laan, G. Microscopic origin of magnetocrystalline anisotropy in transition metal thin films, $J.$ $Phys.:$ $Condens.$ $Matter$ $\bf{10}$, 3239 (1998).
* (36) Wang, D.S. Wu, R. & Freeman, A. J. First-principles theory of surface magnetocrystalline anisotropy and the diatomic-pair model, $Phys.$ $Rev.$ $B$ $\bf{47}$, 14 932 (1993).
* (37) Stöhr, J. & Siegmann, H.C. Springer series in solid state sciences $\bf{152}$, Springer-Verlag Berlin Heidelberg, Magnetism, From Fundamentals to Nanoscale Dynamics (2006).
* (38) Ikeda, K. et al. Magnetic anisotropy of $L1_{0}$-ordered FePt thin films studied by Fe and Pt L2,3-edges x-ray magnetic circular dichroism, $Appl.$ $Phys.$ $Lett.$ $\bf{111}$, 142402 (2017).
* (39) Okabayashi, J. Miura, Y. & Munekata, H. Anatomy of interfacial spin-orbit coupling in Co/Pd multilayers using X-ray magnetic circular dichroism and first-principles calculations, $Sci.$ $Rep.$ $\bf{8}$, 8303 (2018).
* (40) Stöhr, J. & Konig, H. Determination of Spin- and Orbital-Moment Anisotropies in Transition Metals by Angle-Dependent X-Ray Magnetic Circular Dichroism, $Phys.$ $Rev.$ $Lett.$ $\bf{75}$, 3748 (1995).
* (41) Huang, D. J. et al. Orbital Ordering in La0.5Sr1.5MnO4 Studied by Soft X-Ray Linear Dichroism, $Phys.$ $Rev.$ $Lett.$ $\bf{92}$, 087202 (2004).
* (42) Aruta, C. et al. Strain induced x-ray absorption linear dichroism in La0.7Sr0.3MnO3 thin films, $Phys.$ $Rev.$ $B$ $\bf{73}$, 235121 (2006).
* (43) Okabayashi, J. Iida, Y. Xiang, Q. Sukegawa, H. & Mitani, S. Perpendicular orbital and quadrupole anisotropies at Fe/MgO interfaces detected by x-ray magnetic circular and linear dichroisms, $Appl.$ $Phys.$ $Lett.$ $\bf{115}$, 252402 (2019).
* (44) Miura, Y. et al. The origin of perpendicular magneto-crystalline anisotropy in $L1_{0}$-FeNi under tetragonal distortion, $J.$ $Phys.:$ $Condens.$ $Matter$ $\bf{25}$, 106005 (2013).
Figure 1: XAS and XMCD of Mn3-δGa for ${\delta}$ = 0, 1, and 2. Spectra were
measured at the normal incident setup where the incident beam and magnetic
field were parallel to the sample film normal. ${\mu}^{+}$ and ${\mu}^{-}$
denote the absorption in different magnetic field direction. The insets show
the magnetic field dependence of the hysteresis curves taken by fixed
$L_{3}$-edge photon energy. All measurements were performed at RT.  Figure 2:
Deconvoluted XMCD spectra of Mn3-δGa by subtraction from Mn1Ga. The MnI and
MnII components were separated in this procedure. Illustrations of the unit-
cell structures of Mn1Ga and Mn3Ga are also displayed.  Figure 3: XAS and XMLD
of Mn3-δGa for ${\delta}$ = 0, 1, and 2. Spectra were taken at the grazing
incident setup where $\bf{E}$ of the incident beam and direction of
magnetization $\bf{M}$ were parallel and perpendicular, respectively.
${\mu}^{\perp}$ and ${\mu}^{\parallel}$ denote the absorption in different
electric-field directions. The inset shows an illustration of the XMLD
measurement geometry. The angle between sample surface normal and incident
beam is set to 60∘. All measurements were performed at RT.  Figure 4: DFT
calculation of Mn3-δGa. Density of states of MnI and MnII components for each
3$d$ orbital state (upper panel) and bar graphs of the second-order
perturbative channels of the spin-orbit interaction to the magneto-crystalline
anisotropy energy (lower panel): (a) Mn1Ga for MnI component and (b) Mn3Ga for
MnI and MnII components.


# Can we steal your vocal identity from the Internet?: Initial investigation
of cloning Obama’s voice using GAN, WaveNet and low-quality found data

###### Abstract

Thanks to the growing availability of spoofing databases and rapid advances in
using them, systems for detecting voice spoofing attacks are becoming more and
more capable, and error rates close to zero are being reached for the
ASVspoof2015 database. However, speech synthesis and voice conversion
paradigms that are not considered in the ASVspoof2015 database are appearing.
Such examples include direct waveform modelling and generative adversarial
networks. We also need to investigate the feasibility of training spoofing
systems using only low-quality found data. For that purpose, we developed a
generative adversarial network-based speech enhancement system that improves
the quality of speech data found in publicly available sources. Using the
enhanced data, we trained state-of-the-art text-to-speech and voice conversion
models and evaluated them in terms of perceptual speech quality and speaker
similarity. The results show that the enhancement models significantly
improved the SNR of low-quality degraded data found in publicly available
sources and that they significantly improved the perceptual cleanliness of the
source speech without significantly degrading the naturalness of the voice.
However, the results also show limitations when generating speech with the
low-quality found data.

##  1 Introduction

ASVspoof2015 [1] is a commonly used database for developing and evaluating
methods for preventing spoofing of automatic speaker verification (ASV)
systems. Recent studies have shown very good spoofing detection rates for the
ASVspoof2015 corpus with equal error rates (EERs) close to zero [2].

However, with recent advances in text-to-speech (TTS) and voice conversion
(VC) techniques in the speech synthesis field, it has become clear that the
TTS and VC systems included in the corpus are out of date. Good examples of
state-of-the-art speech synthesis techniques include direct waveform modeling
[3, 4, 5] and generative adversarial networks (GANs) [6]. In a recent study
[7], speech synthesized using an end-to-end direct waveform model was rated as
natural as human speech. A preliminary study showed that direct waveform
models do not fool speaker recognition systems; however, Mel-spectra generated
by a GAN have successfully fooled a speaker recognition system [8]111But the
experiment was not fair as the attackers had knowledge of the features used in
the recognition system and directly input generated features to the system,
which are impractical assumptions..

One drawback of the studies using the ASVspoof2015 corpus is that they used an
unrealistic assumption: only studio-recorded speech was used for generating
spoofing materials. If we think of more potential spoofing scenarios, it is
reasonable to expect attacks using unwillingly obtained speech (i.e.,
recordings made in non-controlled environments). This motivated us to study
the feasibility of training a speech enhancement system by obtaining publicly
available data and then using tools to enhance the speech quality of the found
data with the aim of creating reasonable state-of-the-art spoofing TTS or VC
materials.

In the case of using publicly available found data for generating spoofed
speech, it is easy to understand that there are large amounts of speech data
in publicly available sources and that it is likely that data for almost
anybody can be found one way or another. If we are talking about public
personalities like President Barack Obama (a common target for identity theft
research [9, 10]), the amount of data publicly available is immense.

Such data is commonly recorded in non-professional acoustic environments such
as homes and offices. Moreover, the recordings are often made using consumer
devices such as smartphones, tablets, and laptops. Therefore, the speech
portions of the recordings are typically of poor quality and contain a large
amount of ambient noise and room reverberation. However, applications
developed for speaker adaptation of speech synthesis and for voice conversion
have been normally designed to work only on clean data with optimal acoustic
quality and properties. Therefore, the quality of systems trained using data
found in publicly available sources is unknown.

Our specific objective was to answer two questions. First, how well can we
train a speech enhancement system to enhance low-quality data found in
publicly available sources? Second, can we use such enhanced data to produce
effective spoofing materials using the best available TTS and VC systems?

##  2 GAN-based speech enhancement

Generative adversarial networks consist of two “adversarial” models: a
generative model $G$ that captures the data distribution and a discriminative
model $D$ that estimates the probability that a sample came from the training
data rather than $G$. This GAN structure has been used to enhance speech [11,
12]. In the research reported here, we worked on improving the speech
enhancement generative adversarial network (SEGAN) [11]. More specifically, we
attempted to make the training process more robust and stable by introducing a
modified training strategy for SEGAN’s generator.

###  2.1 SEGAN-based speech enhancement

SEGAN [11] exploits the generative adversarial structure in a particular
fashion. The speech enhancement itself is carried out mainly by using model
$G$, which follows an encoder-decoder structure that takes noisy speech as an
input and produces enhanced speech as the output, similar to the U-net
architecture used in the pix2pix framework [13]. The function of model $D$ is
to determine, during training, whether the enhanced speech is detected as fake
(enhanced) or real (clean). If model $D$ can be fooled by the enhanced speech,
there is no gradients through model $G$. On the other hand, if model $D$
cannot be fooled by the enhanced speech, the gradient is back-propagated
through the model $G$ and update it in order to fool the model $D$, thus
biasing the structure towards producing enhanced speech closer to the clean
speech.

We found that the SEGAN structure is sensitive to noise variations during
training, making convergence difficult. We thus made two modifications to
achieve more robust training. For the first one, we created some pre-trained
baseline speech enhancement models (which may be simpler signal processing
methods or easier-to-train neural networks than GANs). They are used to
enhance the speech, and then the content loss of the initial iterations of the
generator model is computed on the basis of the baseline enhanced speech
instead of on clean speech.

For the second modification, a skip connection was added around the generator
so that its task is not to generate enhanced speech from scratch but to
generate a residual signal that refines the input noisy speech [6]. This
should encourage the generator to learn the detailed differences between clean
and enhanced speech waveforms.

##  3 Speech Corpora

We used two types of speech corpora. The corpora used to train the speech
enhancement module were constructed using publicly available data so that the
training process would be replicable. The corpus used as the source for the
cloned voice was constructed using a number of President Barack Obama’s public
interventions obtained from various sources.

###  3.1 Corpora for speech enhancement training

For training the speech enhancement module, we used a subset (28 speakers; 14
male and 14 female; British accent; 400 utterances per speaker) of the Centre
for Speech Technology Research (CSTR) voice cloning toolkit (VCTK)
corpus222http://datashare.is.ed.ac.uk/handle/10283/1942 [14] as the clean
speech corpus. We used different noisy iterations of this corpus to create
four additional corpora for use in making the speech enhancement signal robust
against noisy and/or reverberant environments (see table 3). These corrupted
corpora were recorded as a collaboration between CSTR and the National
Institute of Informatics of Japan and are publicly available in the DataShare
repository of the University of Edinburgh.

Table 1: Corpora for speech enhancement training. Corpus Name | Abbrev. | #Files | Total Time
---|---|---|---
VCTK | clean | 11572 | 8h54m56s
Noisy VCTK | n | 11572 | 8h54m56s
Reverberant VCTK | r | 11572 | 8h54m56s
Noisy Reverberant VCTK | nr | 11572 | 8h54m56s
Device Recorded VCTK | DR | 11572 | 8h54m56s

####  3.1.1 Device-recorded VCTK corpus

To create the device-recorded (DR) VCTK
corpus333https://datashare.is.ed.ac.uk/handle/10283/2959 [15] we re-recorded
the high-quality speech signals in the original VCTK corpus by playing them
back and recording them in office environments using relatively inexpensive
consumer devices. This corpus enables our speech enhancement system to learn
the nuanced relationships between high quality and device-recorded versions of
the same audio. Eight different microphones were used simultaneously for the
re-recording, which was carried out in a medium-sized office under two
background noise conditions (i.e., windows either opened or closed). This
resulted in 16 different conditions.

####  3.1.2 Noisy, Reverberant, and Noisy and reverberant VCTK corpora

We used three other artificially corrupted variations of the CSTR VCTK corpus:
Noisy VCTK444https://datashare.is.ed.ac.uk/handle/10283/2791 [16, 17],
Reverberant VCTK555https://datashare.is.ed.ac.uk/handle/10283/2031 [18], and
Noisy and reverberant VCTK666https://datashare.is.ed.ac.uk/handle/10283/2826
[19]. This diverse portfolio of possible speech corruptions enables our speech
enhancement system to learn how to target different possibilities, i.e., plain
noisy, reverberation compensation, and a mixture of both. As mentioned above,
all the corpora are based on the CSTR VCTK corpus, so the speakers and
utterances represented in the Edinburgh noisy speech dataset are similar to
those of the DR-VCTK corpora.

###  3.2 Obama’s found data

Obama’s data was found online, mainly in YouTube videos with transcriptions as
part of the description and from diverse sources such as interviews and
political meetings. The recording conditions and environments were diverse,
ranging from very noisy with large amounts of reverberation to not so noisy or
not so reverberant, and never achieving recording studio standards. The audio
channel was split from the video channel, automatically segmented on long
pauses, and down-sampled to 16 kHz. The transcription was copied over as a
text file. Table 2 shows a brief characterization of the data.

Table 2: Characterization of the used Obama’s found data. Sources | Public speeches, interviews…
---|---
Total length (w. silence) | 3h 7m 39s
Minimum segment duration | 0.54s
Maximum segment duration | 24.4s
Average segment duration | 5.4s
Estimated SNR mean | 17.15 dB
Estimated SNR variance | 171.22 dB

A histogram of the signal-to-noise ratio (SNR) in dB, estimated using the NIST
SNR measurement tool 777https://www.nist.gov/information-technology-
laboratory/iad/mig/nist-speech-signal-noise-ratio-measurements is shown in
figure 1. It is evident that the vast majority of the speech signals had a
very low SNR compared with conventional speech generation corpus standards.
For instance, the mean and variance of the SNRs estimated for the VCTK corpus
were 30.19 and 23.73 dB, respectively, whereas those of the data used as the
source for the cloned voice were 17.15 and 171.22 dB.

Figure 1: SNR histogram of original Obama’s found data.

##  4 Enhancing Obama’s found data

As mentioned above, the aim of this research was to create reasonable state-
of-the-art spoofing TTS or VC materials. More specifically, we aimed to train
a waveform generation model that can replicate the target speaker’s voice, in
this case, the very recognizable voice of President Barack Obama. Moreover,
the training should be done using only easily available low-quality resources,
as explained in section 3.2. But that kind of data is generally too poor to
ensure reasonably good training of speech synthesis systems. We thus developed
a generative adversarial network-based speech enhancement system for improving
low-quality data to the point where it can be used to properly train TTS
systems.

###  4.1 Design of the speech enhancement models

As we had a large amount of free, publicly available resources for training
our speech enhancement system, we did testing to determine the best training
regime strategy. We trained our speech enhancement system using seven
different sources with various amounts of data, as summarized in table 3.

Table 3: Data sources used for training speech enhancement model. (See table 1 for the meanings of the abbreviations.) SOURCES | #Files | Total Time
---|---|---
DR | 11572 |  8h54m56s
n | 11572 |  8h54m56s
r | 11572 |  8h54m56s
nr | 11572 |  8h54m56s
DR+n | 23144 | 17h49m52s
DR+nr | 23144 | 17h49m52s
All (DR+n+r+nr) | 46288 | 35h39m44s

Our motivation for using three single-category sources and four combinations
was our expectation that training using each single-category source would work
well for the corresponding type of disturbance (i.e., training using source
“n” should be good for cleaning noise, “r” should be good for cleaning
reverberation, and “DR” should be good for compensating for low-quality
recording devices). Since most of the data found will come from noisy poor-
quality sources, it made sense to combine “DR” with the different noisy
corpora. Moreover, since having as much varied data as possible helps neural
networks generalize better, the combination of all the corpora should also be
effective.

###  4.2 Training of the speech enhancement models

Similar to the original SEGAN training strategy, we extracted chunks of
waveforms by using a sliding window of $2^{14}$ samples at every $2^{13}$
samples (i.e., 50% overlap). At testing time, we concatenated the results at
the end of the stream without overlapping. For the last chunk, instead of zero
padding, we pre-padded it with the previous samples. For batch optimization,
RMSprop with a 0.0002 learning rate and a batch size of 100 was used. The
modified SEGAN model converged at 120 epochs.

For selecting the pre-enhancement method, we conducted preliminary
experiments, using the Postfish [20] and the HRNR [21]. Their sequential use
enhanced the quality of the samples, so we used this compound method to
generate the baseline models described in section 2.1.

The source code and extended documentation for the SEGAN implementation are
available online888https://github.com/ssarfjoo/improvedsegan.

###  4.3 Objective evaluation of the speech enhancement

After the speech enhancement models were trained, they were applied to the
noisy data. The effect of the enhancement process was evaluated by estimating
the SNR once again using the NIST tool. Although SNR is most likely not the
best measure of enhancement, the lack of a clean reference limited the
availability of tools.

Table 4: Average SNR in dB estimated with NIST tool for the results of the different speech enhancement models. SOURCES | average SNR (dB)
---|---
Obama source | 17.2
n | 49.8
r | 22.7
nr | 43.1
DR | 28.24
DR+n | 40.1
DR+nr | 41.37
all (DR+n+r+nr) | 37.89

The SNR estimation results (table 4) show a clear picture: the enhancement
process, regardless of which training data were used, improved the average SNR
of the original Obama voice data. In particular, training using noisy data
(i.e., “n”, “nr”, and their mixtures) was considerably more effective than
training using the other two possibilities (i.e., “r” and “DR”). This is
attributed to the fact that their use reduces the noise levels in the signal,
which is what the SNR measure targets. Their use may not actually improve the
perceptual quality of the voice signals. Histograms showing the improvement in
SNR when “n” and when all the variants were used are shown in figures 2 and 3
respectively.

Figure 2: SNR histogram of original Obama voice data after enhancement using
noisy VCTK. Figure 3: SNR histogram of original Obama voice data after
enhancement using all VCTK variants.

###  4.4 Perceptual evaluation of the speech enhancement

As mentioned in the previous section, SNR is most likely not the best measure
of enhancement. Since the ultimate objective of our research is to produce
high quality synthetic speech, be it through speech synthesis or through voice
conversion, it makes sense to evaluate perceptual quality from the viewpoint
of human users. We thus carried out a crowdsourced perceptual evaluation with
Japanese native listeners. We presented the listeners with a set of 16
screens, each corresponding to one of the eight evaluated conditions (original
plus seven enhanced versions) for one of two utterances. The evaluators were
given two tasks: 1) rate the perceived quality of each speech sample on a MOS
scale and 2) rate the cleanliness of each speech sample (i.e., free from
noise, reverberation, and artifacts), also on a MOS scale (with 1 being very
noisy speech and 5 being clean speech).

The participants were able to listen to each speech sample as many times as
they wanted, but they could not proceed to the next sample until they
completed both tasks. They were not allowed to return to previous samples. The
samples were selected on the basis of their length, evaluating in the all the
utterances (i.e., 530 utterances) between 5.0 and 5.9 seconds long. In total
this meant that 265 sets for evaluating all the evaluation utterances, which
was done 3 times for a total of 795 sets. The participants were allowed to
repeat the evaluation up to 8 times, thereby ensuring that there were at least
100 different listeners. A total of 129 listeners took part in the evaluation
(72 male, 57 female).

####  4.4.1 Results

The results of the perceptual evaluation (table 5) also show a clear picture.
For the original Obama voice data, there was a clear perception of noisiness
and related factors (MOS score of 2.42 for cleanliness) even though the
perceived quality was reasonably high (3.58). Studio recorded clean speech is
normally rated 4.5 on average [22], whereas the original Obama voice data was
rated 3.5, that is, one point less, most likely due to the poor conditions on
which these sources were recorded.

Table 5: Results of the perceptual evaluation (MOS score). Non-statistically significant differences are marked with *. SOURCES | Quality | Cleanliness
---|---|---
Obama source | 3.58* | 2.42
n | 2.73 | 3.35
r | 3.55* | 3.17
nr | 3.11 | 3.42*
DR | 3.51 | 3.31
n+DR | 3.26 | 3.02
nr+DR | 3.30 | 3.34
all (n+r+nr+DR) | 3.41 | 3.40*

Use of the enhanced versions, improved the cleanliness of the source data,
with different degrees of improvement depending on the source data used. Most
noteworthy is the cleanliness result for “noisy-reverberant” (3.42), which had
the largest improvement. This is attributed to the original data being
recorded mostly in noisy environments with reverberation, so a speech
enhancement system targeting this condition gives the best improvement in that
field. The cleanliness result for “all” was similarly high, which we attribute
to the training being done for all possible situations.

On the other hand, there was a cost to applying these speech enhancements: a
consistent degradation in the perceived speech quality. This implies that
speech enhancement focused on cleanliness can greatly reduce the naturalness
of the speech. This means that the approaches providing the biggest
improvements in SNR, such as the “noisy” condition with a quality score of
2.73 or the “noisy-reverberant” condition with a quality score of 3.11, may
not be the best way to produce clean speech for further speech processing.

In short, there seems to be a trade-off between quality degradation and
cleanliness improvement, which is not encouraging. But, if we look at the
results for the ”all” condition, combining all possible data sources, we see
that it provided one of the best cleanliness scores (3.40) with one of the
smallest quality degradations (0.17 degradation). This strongly suggests that
having trained our speech enhancement system in a variety of degradation
conditions gave the system enough generalization capability and enough
knowledge of human speech to reduce noisiness while maintaining as much as
possible voice naturalness.

##  5 Generation of the synthetic samples

We used two approaches to generate spoofed speech waveforms: CycleGAN [23] for
VC (section 5.1) and an autoregressive (AR) neural network for TTS acoustic
modeling (section 5.2). While both approaches generate mel-spectrograms,
CycleGAN converts the mel-spectrogram from a source speaker into a mel-
spectrogram that retains the speech contents and overlays the voice
characteristics of the target speaker. In contrast, the AR approach converts
the linguistic features extracted from text into the mel-spectrogram of the
target speaker. Given the generated mel-spectrogram for the target speaker,
the WaveNet neural network generates the speech waveforms (section 5.3). The
process for generating spoofed speech is illustrated in figure 4.

The decision to use the mel-spectrogram as the acoustic feature was based on
the expected limitations of traditional features (e.g., F0, Mel-generalized
cepstrum, and aperiodicity bands) as the estimation of F0 is problematic in
both the original noisy speech signals and the enhanced signals when
considering the noisy data we found. We also used an increased number of mel
bands compared to other approaches [7] (80 vs. 60) with the expectation that
it would help the waveform model better cope with corrupted or noisy segments.

Figure 4: Process for generating spoofed speech.

###  5.1 VC based on CycleGAN

CycleGAN was originally developed for unpaired image-to-image translation,
which consists of two generators ($G$ and $F$) and two discriminators ($D_{X}$
and $D_{Y}$), as shown in figure 5. Generator $G$ serves as a mapping function
from distribution $X$ to distribution $Y$, and generator $F$ serves as a
mapping function from $Y$ to $X$. The discriminators aim to estimate the
probability that a sample came from real data $x\in X$ (or $y\in Y$) rather
than from the generated sample $\hat{x}=F(y)$ (or $\hat{y}=G(x)$).

Figure 5: Diagram of CycleGAN. $G$ and $F$ are generators; $D_{X}$ and $D_{Y}$
are discriminators. $X$ and $Y$ are real distributions, and $\hat{Y}$ and
$\hat{X}$ represent the corresponding generated distributions. ‘FF’ means
feed-forward neural network.

As shown in the same figure, CycleGAN has two translation directions: forward
($X\rightarrow\hat{Y}\rightarrow\hat{X}$) and backward
($Y\rightarrow\hat{X}\rightarrow\hat{Y}$). This means that $X$ to $Y$
translation and $Y$ to $X$ translation can be learned simultaneously.
Furthermore, an input is carried back to its original form through each
translation direction and thereby minimizing consistency loss:

$\displaystyle\mathcal{L}_{cyc}(G,F)$
$\displaystyle=\operatorname{\mathbb{E}}_{x\sim
p_{data}(x)}[\|F(G(x))-x\|_{1}]$
$\displaystyle+\operatorname{\mathbb{E}}_{y\sim
p_{data}(y)}[\|G(F(y))-y\|_{1}],$ (1)

where $\operatorname{\mathbb{E}}$ is the expectation and $\|\cdot\|_{1}$ is
the L1 norm. With this structure, it is possible to keep part of the
information unchanged when an input is translated by $G$ or $F$. When applying
this model to VC, $X$ and $Y$ can be thought of as the feature distribution of
the source speaker and that of the target speaker, respectively. By
reconstructing the input data, linguistic information may be retained during
translation. Additionally, speaker individuality can be changed by adversarial
learning using an adversarial loss [24]. By integrating consistency loss and
adversarial loss, we can learn a mapping function for VC using a non-parallel
database [25].

To train a CycleGAN-based VC system, we used a female speaker (Japanese-
English bilingual) as the source speaker. Her speech was recorded in a studio.
The target speaker was President Barack Obama999Audio samples are available at
https://fangfm.github.io/crosslingualvc.html. Both his original voice data and
the enhanced data with SNR $>$ 30 dB (9240 utterances) were used. In
accordance with the source speaker’s data sets, we implemented three VC
systems: one using 611 Japanese utterances, one using 611 English utterances,
and one using a mixture of Japanese and English utterances. The generator and
discriminator of the CycleGAN were a fully connected neural network with six
layers. A 240-dimension vector consisting of a 80-dimension mel-spectrogram
and the first and second derivatives were input into the CycleGAN. There were
256, 512, 512, and 256 units in the hidden layers. A sigmoid was used as the
activation function. Batch normalization [26] was conducted for each hidden
layer of the generators. The batch size and learning rate for the generators
and discriminators were randomly selected 128 and 4096 frames and 0.001 and
0.0001, respectively.

###  5.2 TTS acoustic modeling based on AR neural network

An acoustic model for TTS converts the linguistic features extracted from a
text into acoustic features such as a mel-spectrogram. Specifically for this
work, given a sequence of linguistic feature
$\bm{l}_{1:N}=\\{\bm{l}_{1},\cdots,\bm{l}_{N}\\}$ of $N$ frames, an acoustic
model needs to generate a sequence of acoustic features
$\bm{a}_{1:N}=\\{\bm{a}_{1},\cdots,\bm{a}_{N}\\}$ with the same number of
frames. Here, $\bm{l}_{n}$ and $\bm{a}_{n}$ denote the linguistic features and
the mel-spectrogram for the $n$-th frame.

Figure 6: AR neural network for TTS acoustic modeling. FF Tanh and Linear
denote feedforward layers with Tanh and identity activation function,
respectively. Bi-LSTM and Uni-LSTM denote bi-directional and uni-directional
LSTM layers. The “Time delay” block keeps the Mel-spectrogram for frame $n$
and sends it to the Uni-LSTM at frame $n+1$.

The AR neural network illustrated in figure 6 was used to convert
$\bm{l}_{1:N}$ into $\bm{a}_{1:N}$. This network has two feedforward layers
and a bi-directional long-short-term-memory (LSTM) unit recurrent layer near
the input side. Following these three layers, it uses another uni-directional
LSTM layer. Different from the first LSTM layer, this one not only takes the
output of the previous layer but also the previous output of the whole network
as input. For example, it takes $\bm{a}_{n-1}$ as input when it generates
$\bm{a}_{n}$ for the $n$-th frame. This type of data feedback is widely used
in neural text generation and machine translation [27, 28], and a network with
this type of feedback loop is referred to as an autoregressive model. Note
that, while the natural $\bm{a}_{n-1}$ from the training set is fed back
during training, the generated $\widehat{\bm{a}}_{n-1}$ is fed back during
generation. Also note that a simple trick is used here that improves network
performance: it is to randomly drop out the feedback data in both the training
and generation stages [29].

The linguistic features used for both training and synthesis were extracted
using Flite [30]. The dimension of these feature vectors was 389. The
alignment information was obtained using forced alignment with a hidden-semi
Markov model trained using HTS [31] on the mel-spectrograms. In addition to
the linguistic features, a numeric variable characterizing the enhancement
condition was used as input. Together with the input features, the network was
trained using the mel-spectrograms obtained with the enhanced speech method
explained in section 4. The dropout rate was set to 25%.

###  5.3 WaveNet vococder

Building a state-of-the-art data-driven vocoder such as WaveNet represents a
big challenge when trying to use the types of data we found: it is not easy to
gather sufficient data that are good enough for the process. This is where the
advantage of having used another data-driven speech enhancement system comes
into play. As hinted at in the introduction to section 4, we can take
advantage of our GAN-based speech enhancement system to generate multiple
enhanced versions of the noisy speech data, effectively multiplying the amount
of training data available for training our system.

For this research, we trained our WaveNet vocoder on the enhanced version of
the original Obama voice data. The network structure is illustrated in figure
7. The vocoder works at a sampling rate of 16 kHz. The $\mu$-law compressed
waveform is quantized into ten bits per sample. Similar to that in a previous
study [5], the network consists of a linear projection input layer, 40 WaveNet
blocks for dilated convolution, and a post-processing block. The $k$-th
dilation block has a dilation size of $2^{\mathrm{mod}(k,10)}$, where
$\mathrm{mod}(\cdot)$ is modulo operation. In addition, a bi-directional LSTM
and a 1D convolution layer are used to process the input acoustic features.
The acoustic features fed to every WaveNet block contain the 80-dimensional
mel-spectrogram plus an additional component specifying which of the different
speech-enhancing models produced that speech waveform101010The tools for the
TTS and WaveNet implementation are based on a modified CURRENNT toolkit [32]
and can be found online http://tonywangx.github.io.

Figure 7: Structure of WaveNet vocoder. Each WaveNet block is surrounded by
dotted line. Details of first WaveNet block are shown.

##  6 Perceptual evaluation of the generated speech

To evaluate the generation capabilities of the proposed system, we carried out
a second crowd-sourced perceptual evaluation with Japanese native listeners.
The evaluation was conducted in the same way as the one described in section
4.4. Only the tasks were different: 1) rate the quality of the voice in each
speech sample, ignoring the effects of background noise and 2) rate the
similarity of the voice to that of the spoofed speaker (i.e., Obama), ignoring
speech quality and cleanliness. To compare with the target speaker, we
presented the participants two additional samples (one generated by the
evaluated system and one with natural speech), both containing the same
linguistic content but different than the one used for rating speech quality.
The participants were asked to rate on a MOS scale the similarity of the two
speakers.

###  6.1 Systems evaluated

We evaluated six systems plus natural speech and copy synthesis of the mel-
spectrograms of the natural speech with the trained WaveNet vocoder:

  * •

Voice conversion system 1 (VC1): Japanese-to-English bilingual VC

  * •

Voice conversion system 2 (VC2): English-to-English monolingual VC

  * •

Voice conversion system 3 (VC3): Japanese and English mixed utterances
multilingual VC

  * •

Text to speech system 1 (TT1): text to speech from estimated mel-spectrogram
for noisy condition

  * •

Text to speech system 2 (TT2): text to speech from estimated mel-spectrogram
for enhanced-all condition

  * •

Text to speech system 3 (TT3): Speech generated with TT2 with a small added
amount of reverberation

With the three different voice conversion systems, we aimed to evaluate the
effect of noise and to investigate whether CycleGAN can be used for cross-
lingual VC systems. With the three TTS systems, we aimed to analyze the effect
of generating spectrograms on the basis of different conditions for speech
enhancement and to determine the importance of mimicking the environmental
conditions of the reference natural speech when considering human perception.

###  6.2 Results

Table 6: Results of second perceptual evaluation (MOS score). Non-statistically significant differences are marked with superscripts. Sources | Quality | Similarity
---|---|---
Natural | 4.40 | 4.70
Copy-synthesis |  2.451 | 2.99
VC1 |  2.662 |  1.563
VC2 |  2.672 |  1.553
VC3 | 2.83 |  1.563
TT1 |  2.491 |  1.434
TT2 |  2.511 |  1.404
TT3 |  2.632 |  1.454

The results for the second perceptual evaluation are summarized in table 6.
There was a total of 103 unique participants (52 male and 51 female). The
results for natural speech indicate that the participants were able to
identify the actual Obama voice regardless of the environmental conditions
(MOS score of 4.70 for similarity). They also indicate that they were able to
distinguish the naturalness and frequency of the speech regardless of the
background noise and/or reverberation (MOS score of 4.40 for quality).

The results for copy synthesis using the trained WaveNet vocoder were quite
different (MOS score of 2.45 for quality). The WaveNet system and scripts had
previously been successful at generating speech when clean speech was used for
training [33], suggesting that the difference was due to the nature of the
data used for training. One possibility is that using mixture density networks
for generating output waveforms is problematic for modeling the variance of
noisy data.

Looking at the TTS system results, we see that the quality of the generated
speech did not change significantly with the generation condition (MOS score
of 2.5) and was similar to that for copy synthesis. Adding a small amount of
reverberation improved the perceived speech quality so that it was even higher
than with copy synthesis (2.63 versus 2.45). This means that reverberation can
mask part of the noise generated by the WaveNet vocoder. The significant drop
in the similarity score means that we cannot say whether the evaluators were
capable of identifying Obama’s voice.

Looking at the VC system results, we see a similar pattern. In terms of
quality, the VC systems were slightly but significantly better than both TTS
and copy synthesis. This is probably because the VC systems were trained on
selected data (i.e., only data with estimated SNR $>$ 30 dB), and clean data
was used for the source speaker. In terms of similarity, they were slightly
but significantly better than TTS but far worse than copy synthesis.

Comparing VC1 (Japanese-to-English bilingual VC) with VC2 (English-to-English
mono-lingual VC), we see that they achieved similar MOS scores for both speech
quality and speaker similarity. This suggests that CycleGAN can be used to
train a bilingual VC model. When Japanese and English utterances were mixed
(VC3), the speech quality was slightly higher for the other VC systems. This
is probably because twice the amount of source speaker training data was used.

##  7 Evaluation based on anti-spoofing countermeasures

We have also evaluated the built TTS and VC systems based on anti-spoofing
countermeasures. The countermeasure used is a common Gaussian mixture models
(GMMs) back-end classifier with constant Q cepstral coefficient (CQCC)
features [34]. Training of the countermeasure is speaker-independent and we
have used two alternative training sets to train the two GMMs (one for natural
or bona fide speech; another one for synthetic or converted voices). The first
one contains the training portion of the ASVspoof2015 data consisting of 5
(now outdated) spoofing attacks, and the second one consists of the converted
audio samples submitted by the 2016 Voice Conversion Challenge (VCC2016)
participants [35]. The latter contains 18 diverse stronger VC
attacks111111http://dx.doi.org/10.7488/ds/1575.

Table 7 shows the evaluation results for the CQCC-GMM countermeasure when
scored on the found data of Obama (that is, the same data used for the
listening tests in a previous section). The results are presented in terms of
equal error rate (EER, %) of the spoofing countermeasure. The higher the EER,
the more confused the countermeasure will be in telling apart our generated
voices from natural human speech.

As we can see from a table 7, although the VC and TTS systems in this paper
are more advanced methods than ones included in the current ASVspoof 2015, the
countermeasure models can still detect both the proposed VC and TTS samples
using the found data easily. This is because not only the VC and TTS process
but also the additional speech enhancement process caused noticeable
artifacts.

Table 7: Evaluation results based on anti-spoofing countermeasures (EER in percentages). 32-mix CQCC-GMMs were trained on ASVspoof2015 or VCC2016 sets. Sources | ASVspoof2015 | VCC2016
---|---|---
Copy-synthesis | 4.63 | 8.46
VC1 | 2.32 | 1.08
VC2 | 2.16 | 0.00
VC3 | 2.25 | 1.01
TT1 | 1.60 | 0.00
TT2 | 2.01 | 0.00
TT3 | 0.79 | 0.00

##  8 Conclusions and future work

We have introduced a number of publicly available and known datasets that
proved to be extremely useful for training speech enhancement models.
Application of these models to a corpus of low-quality considerably degraded
data found in publicly available sources significantly improved the SNR of the
data.

A perceptual evaluation revealed that the models can also significantly
improve the perceptual cleanliness of the source speech without significantly
degrading the naturalness of the voice as is common when speech enhancement
techniques are applied. Speech enhancement was most effective when the system
was trained using the largest amount of data available as doing so covered a
wide variety of environmental and recording conditions, thereby improving the
generalization capabilities of the system.

A second perceptual evaluation revealed that, while generating synthetic
speech from noisy publicly available data is starting to become possible,
there are still obvious perceptual problems in both text-to-speech and voice
conversion systems that must be solved to achieve the naturalness of systems
trained using very high quality data. Therefore, we cannot recommend yet that
next-generation ASVspoof data be generated using publicly available data even
if adding this new paradigm of speech generation systems is a must.

Acknowledgements: This work was partially supported by MEXT KAKENHI Grant
Numbers (15H01686, 16H06302, 17H04687).

## References

  * [1] Z. Wu, J. Yamagishi, T. Kinnunen, C. Hanilçi, M. Sahidullah, A. Sizov, N. Evans, M. Todisco, and H. Delgado, “Asvspoof: The automatic speaker verification spoofing and countermeasures challenge,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, no. 4, pp. 588–604, June 2017\.
  * [2] H. Muckenhirn, M. Magimai-Doss, and S. Marcel, “End-to-end convolutional neural network-based voice presentation attack detection,” in _Proc. IJCB 2017_ , 2017.
  * [3] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” _arXiv preprint arXiv:1609.03499_ , 2016.
  * [4] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron: Towards end-to-end speech synthesis,” in _Proc. Interspeech_ , 2017, pp. 4006–4010.
  * [5] A. Tamamori, T. Hayashi, K. Kobayashi, K. Takeda, and T. Toda, “Speaker-dependent WaveNet vocoder,” in _Proc. Interspeech 2017_ , 2017, pp. 1118–1122.
  * [6] T. Kaneko, H. Kameoka, N. Hojo, Y. Ijima, K. Hiramatsu, and K. Kashino, “Generative adversarial network-based postfilter for statistical parametric speech synthesis,” in _Proc. ICASSP 2017_ , 2017, pp. 4910–4914.
  * [7] J. Shen, M. Schuster, N. Jaitly, R. Skerry-Ryan, R. A. Saurous, R. J. Weiss, R. Pang, Y. Agiomyrgiannakis, Y. Wu, Y. Zhang, Y. Wang, Z. Chen, and Z. Yang, “Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions,” in _Proc. ICASSP 2018_ , 2018.
  * [8] W. Cai, A. Doshi, and R. Valle, “Attacking speaker recognition with deep generative models,” _arXiv preprint arXiv:1801.02384_ , 2018.
  * [9] R. Kumar, J. Sotelo, K. Kumar, A. de Brebisson, and Y. Bengio, “Obamanet: Photo-realistic lip-sync from text,” in _NIPS 2017 Workshop on Machine Learning for Creativity and Design_ , 2017.
  * [10] S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman, “Synthesizing Obama: learning lip sync from audio,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 4, pp. 95:1–95:13, 2017.
  * [11] S. Pascual, A. Bonafonte, and J. Serrà, “SEGAN: Speech enhancement generative adversarial network,” in _Proc. Interspeech 2017_ , 2017, pp. 3642–3646.
  * [12] C. Donahue, B. Li, and R. Prabhavalkar, “Exploring speech enhancement with generative adversarial networks for robust speech recognition,” _Proc. ICASSP 2018_ , 2018.
  * [13] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proc. CVPR 2017_ , 2017.
  * [14] C. Veaux, J. Yamagishi, and S. King, “The voice bank corpus: Design, collection and data analysis of a large regional accent speech database,” in _Proc. Oriental COCOSDA 2013_ , 2013, pp. 1–4.
  * [15] S. S. Sarfjoo and J. Yamagishi, _Device Recorded VCTK (Small subset version)_ , University of Edinburgh, 2017.
  * [16] C. Valentini-Botinhao, X. Wang, S. Takaki, and J. Yamagishi, “Speech enhancement for a noise-robust text-to-speech synthesis system using deep recurrent neural networks,” in _Interspeech 2016_ , 2016, pp. 352–356.
  * [17] ——, “Investigating RNN-based speech enhancement methods for noise-robust text-to-speech,” in _9th ISCA Speech Synthesis Workshop_ , 2016, pp. 146–152.
  * [18] C. Valentini-Botinhao, “Reverberant speech database for training speech dereverberation algorithms and tts models,” University of Edinburgh, 2016.
  * [19] ——, “Noisy reverberant speech database for training speech enhancement algorithms and tts models,” University of Edinburgh, 2017.
  * [20] M. Montgomery. (2005) Postfish by Xiph.org, Available: https://svn.xiph.org/trunk/postfish/README.
  * [21] C. Plapous, C. Marro, and P. Scalart, “Improved signal-to-noise ratio estimation for speech enhancement,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 14, no. 6, pp. 2098–2108, 2006.
  * [22] S. King, L. Wihlborg, and W. Guo, “The Blizzard Challenge 2017,” in _The Blizzard Challenge workshop 2017_ , August 2017.
  * [23] J. Zhu, T. Park, P. Isola, and A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proc. ICCV 2017_ , 2017\.
  * [24] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley, “Least squares generative adversarial networks,” in _Proc. ICCV 2017_. IEEE, 2017, pp. 2813–2821.
  * [25] F. Fang, J. Yamagishi, I. Echizen, and J. Lorenzo-Trueba, “High-quality nonparallel voice conversion based oncycle-consistent adversarial network,” in _Proc. ICASSP 2018_ , 2018.
  * [26] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_ , 2015, pp. 448–456.
  * [27] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _Proc. NIPS_ , 2014, pp. 3104–3112.
  * [28] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,” in _Proc. NIPS_ , 2015, pp. 1171–1179.
  * [29] X. Wang, S. Takaki, and J. Yamagishi, “An RNN-based quantized F0 model with multi-tier feedback links for text-to-speech synthesis,” in _Proc. Interspeech_ , 2017, pp. 1059–1063.
  * [30] HTS Working Group, “The English TTS system Flite+HTS_engine,” 2014. [Online]. Available: http://hts-engine.sourceforge.net/
  * [31] H. Zen, T. Toda, M. Nakamura, and K. Tokuda, “Details of Nitech HMM-based speech synthesis system for the Blizzard Challenge 2005,” _IEICE Trans. Inf. & Syst._, vol. E90-D, no. 1, pp. 325–333, Jan. 2007.
  * [32] F. Weninger, J. Bergmann, and B. Schuller, “Introducing CURRENNT–the Munich open-source CUDA recurrent neural network toolkit,” _Journal of Machine Learning Research_ , vol. 16, no. 3, pp. 547–551, 2015.
  * [33] X. Wang, J. Lorenzo-Trueba, S. Takaki, L. Juvela, and J. Yamagishi, “A comparison of recent waveform generation and acoustic modelingmethods for neural-network-based speech synthesis,” in _Proc. ICASSP 2018_ , April 2018\.
  * [34] M. Todisco, H. Delgado, and N. Evans, “A new feature for automatic speaker verification anti-spoofing: Constant q cepstral coefficients,” in _Odyssey 2016_ , 2016, pp. 283–290.
  * [35] T. Toda, L.-H. Chen, D. Saito, F. Villavicencio, M. Wester, Z. Wu, and J. Yamagishi, “The voice conversion challenge 2016,” in _Interspeech 2016_ , 2016, pp. 1632–1636.