# The Contribution of Evolved Stars to PAH Heating and Implications for
Estimating Star Formation Rates

Lulu Zhang Kavli Institute for Astronomy and Astrophysics, Peking University,
Beijing 100871, China<EMAIL_ADDRESS>Department of Astronomy, School
of Physics, Peking University, Beijing 100871, China  Luis C. Ho Kavli
Institute for Astronomy and Astrophysics, Peking University, Beijing 100871,
China<EMAIL_ADDRESS>Department of Astronomy, School of Physics,
Peking University, Beijing 100871, China

###### Abstract

Emission from polycyclic aromatic hydrocarbons (PAHs) is a promising tool for
estimating star formation rate (SFR) in galaxies, but the origin of its
sources of excitation, which include not only young but possibly also old
stars, remains uncertain. We analyze Spitzer mid-infrared mapping-mode
spectroscopic observations of the nuclear and extra-nuclear regions of 33
nearby galaxies to study the contribution of evolved stars to PAH emission. In
combination with photometric measurements derived from ultraviolet, H$\alpha$,
and infrared images, the spatially resolved spectral decomposition enables us
to characterize the PAH emission, SFR, and stellar mass of the sample galaxies
on sub-kpc scales. We demonstrate that the traditional empirical correlation
between PAH luminosity and SFR has a secondary dependence on specific SFR, or,
equivalently, stellar mass. Ultraviolet-faint regions with lower specific SFRs
and hence greater fraction of evolved stars emit stronger PAH emission at
fixed SFR than ultraviolet-bright regions. We reformulate the PAH-based SFR
estimator by explicitly introducing stellar mass as a second parameter to
account for the contribution of evolved stars to PAH excitation. The influence
of evolved stars can explain the sub-linear correlation between PAH emission
and SFR, and it can partly account for the PAH deficit in dwarf galaxies and
low-metallicity environments.

dust, extinction — galaxies: ISM — galaxies: star formation — infrared: ISM

††journal: ApJ

##  1 Introduction

Estimating the strength of star formation is of great importance for
characterizing the formation and evolution of galaxies. Normal star-forming or
starburst galaxies exhibit significant mid-infrared (IR) broad emission
features (e.g., Smith et al. 2007b; Galliano et al. 2008b; Gordon et al.
2008), the carriers of which are widely attributed to polycyclic aromatic
hydrocarbons (PAHs; Duley & Williams 1981; Leger & Puget 1984; Allamandola et
al. 1985; see review in Tielens 2008). PAHs radiate through IR fluorescence
following electronic excitation after absorbing a single ultraviolet (UV)
photon (Allamandola et al., 1989; Tielens, 2005). Thus, PAH emission provides
an indirect estimate of the strength of the UV radiation field, and hence the
star formation rate (SFR), which is relatively free from dust extinction
(Förster Schreiber et al., 2004; Peeters et al., 2004). Accordingly, numerous
studies have provided SFR prescriptions based on PAH emission (e.g., Calzetti
et al. 2005, 2007; Wu et al. 2005; Treyer et al. 2010; Shipley et al. 2016;
Maragkoudakis et al. 2018; Xie & Ho 2019). The utility of PAH emission as a
SFR indicator has motivated different strategies for measuring PAH strength.
Some employ broadband photometric observations that capture strong PAH
features (e.g., Wu et al. 2005; Calzetti et al. 2007; Donoso et al. 2012; Lee
et al. 2013; Cluver et al. 2017; Mahajan et al. 2019; Zhang & Ho 2022), which,
while expedient, may be susceptible to uncertainties from the underlying
continuum contamination (e.g., Donoso et al. 2012; Crocker et al. 2013; Cluver
et al. 2014; Lu et al. 2014), while others have advanced methods to extract
individual PAH features from spectroscopic observations, after subtracting the
local continuum or simultaneously fitting the PAH features and the continuum
using model decomposition (e.g., Peeters et al. 2002; Marshall et al. 2007;
Smith et al. 2007b; Xie et al. 2018; Zhang et al. 2021).

Although PAH emission is widely used as a tracer of star formation activity,
the theoretical assumption that PAHs are primarily excited by ionizing UV
photons from newborn stars, with little or no absorption of visible photons
from old stars (Sellgren, 2001), is not always valid (Draine & Li, 2001). Li &
Draine (2002) gave a satisfactory model fit to the observed IR emission of vdB
133, a very UV-poor reflection nebula, demonstrating that astronomical PAHs
can be pumped even by cool stars with weak UV radiation. Haas et al. (2002)
studied the spatial coincidence between the strength of the PAH 7.7 ${\rm\mu
m}$ feature and the 850 ${\rm\mu m}$ continuum emission for five galaxies with
a range of star formation activity, and they suggested that PAHs are
preferentially distributed in regions dominated by cold dust and molecular
clouds, where PAHs are excited mainly by the interstellar radiation field (see
also Bendo et al. 2008, 2010; Cortzen et al. 2019). The excitation of PAHs may
vary depending on galaxy type. For example, Jones et al. (2015) studied the
correlation between the 8 ${\rm\mu m}$ PAH emission band, as captured in the
IRAC4 band of the Spitzer Space Telescope (Houck et al., 2004), and the dust
continuum in NGC 2403 and M 83. They concluded that PAHs in NGC 2403 are
coupled with the diffuse dust and are heated by the radiation from the total
stellar population, while the excitation of PAHs within M 83 is connected more
strongly with star-forming regions. Similarly, Bendo et al. (2020) studied the
PAH excitation for a sample of 25 spiral galaxies by analyzing the ratio of
PAH surface brightness to dust surface density and its correlation with
tracers of young and evolved stars. They found that PAH emission is correlated
straightforwardly with young stars (as traced by the UV+24 ${\rm\mu m}$
continuum) in 11/25 galaxies, whereas for 5/25 galaxies PAH emission is linked
with young stars only in the inner parts of the galaxies. Moreover, PAH
emission better traces evolved stars (3.6 ${\rm\mu m}$ continuum) in 6/25
galaxies; no clear trends can be discerned in the remaining 3/25 galaxies.
Some case studies have tried to quantify the specific contribution of evolved
stars to PAH excitation. For example, Crocker et al. (2013) concluded that
$30\%-43\%$ of the 8 ${\rm\mu m}$ band PAH emission of NGC 628 is related to
evolved stars, while Lu et al. (2014) proposed that 67% of the 8 ${\rm\mu m}$
band PAH emission in M 81 is heated by evolved stars.

The aforementioned results testify that a single source cannot completely
account for the excitation of PAHs in different galaxies, and even of
different environments within a single galaxy. The contribution of evolved
stars should be considered, to ascertain the degree to which it might
influence the SFRs based on PAH emission. To this end, spatially resolved
analysis is critical to help isolate the contribution of PAHs from different
environments, in an attempt to identify potentially different sources of
excitation. We perform this exercise using spatially resolved measurements of
PAH emission extracted from galaxies from the Spitzer Infrared Nearby Galaxies
Survey111https://irsa.ipac.caltech.edu/data/SPITZER/SINGS/ (SINGS; Kennicutt
et al. 2003). We quantify the influence of evolved stars on PAH emission and
propose a revised calibration of the PAH-based SFR estimator, which has
promising applications for observations from the James Webb Space Telescope
(Rigby et al., 2022).

Figure 1: Illustration of the spatial coverage of the IRS mapping-mode data
cube using the FUV image of NGC 4254 (top), whose center is marked with a
star. The bottom panel delineates the UV-bright spaxels (blue) and UV-faint
spaxels (red) within the LL data cube that bisects the galaxy; all the
remaining spaxels are in gray. Analogous UV-bright and UV-faint spaxels can be
defined for the SL data cube that samples the extranuclear H II region on the
spiral arm. Table 1: Multiwavelength Photometric Images

Telescope | Filter | $\lambda_{\rm eff}$ | FWHM | $\sigma_{\rm cal}$ | References
---|---|---|---|---|---
|  | $(\mu m)$ | (″) | (%) |
(1) | (2) | (3) | (4) | (5) | (6)
GALEX | FUV | 0.153 | 4.2 | 5 | 1, 2
GALEX | NUV | 0.231 | 5.3 | 3 | 1, 2
KPNO/CTIO | H$\alpha$ | 0.657 | 1.9 | 15 | 3, 4
2MASS | $J$ | 1.25 | 2.5 | 5 | 5, 6
2MASS | $H$ | 1.65 | 2.5 | 5 | 5, 6
2MASS | $K_{s}$ | 2.16 | 2.5 | 5 | 5, 6
WISE | W1 | 3.35 | 6.1 | 2.4 | 7, 8
WISE | W2 | 4.60 | 6.4 | 2.8 | 7, 8
WISE | W3 | 11.56 | 6.5 | 4.5 | 7, 8
Spitzer | IRAC1 | 3.55 | 1.7 | 10 | 3, 9
Spitzer | IRAC2 | 4.49 | 1.7 | 10 | 3, 9
Spitzer | IRAC3 | 5.73 | 1.9 | 10 | 3, 9
Spitzer | IRAC4 | 7.87 | 2.0 | 10 | 3, 9
Spitzer | MIPS24 | 23.68 | 6.0 | 4 | 3, 10

Note. — Col. (1): Telescope. Col. (2): Filter. Col. (3): Effective wavelength
of filter. Col. (4): FWHM of the PSF. Col. (5): Calibration uncertainty. Col.
(6): References.

References. — (1) Morrissey et al. 2007; (2) Gil de Paz et al. 2007; (3) Dale
et al. 2007; (4) Kennicutt et al. 2008; (5) Jarrett et al. 2003; (6) Skrutskie
et al. 2006; (7) Wright et al. 2010; (8) Jarrett et al. 2013; (9) Fazio et al.
2004; (10) Engelbracht et al. 2007.

Table 2: Properties of the Galaxy Sample

Galaxy | Morphology | Class | $D_{L}$ | [N II]/H$\alpha$ | $E(B-V)$ | $N_{\rm H~{}II}$ | Galaxy | Morphology | Class | $D_{L}$ | [N II]/H$\alpha$ | $E(B-V)$ | $N_{\rm H~{}II}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---
(1) | (2) | (3) | (4) | (5) | (6) | (7) | (1) | (2) | (3) | (4) | (5) | (6) | (7)
NGC 24 | Sc | SFG | 8.2 | 0.37 | 0.017 | 0 | NGC 3521 | SABbc | AGN | 11.2 | 0.56 | 0.050 | 4
NGC 337 | SBd | SFG | 19.3 | 0.23 | 0.096 | 0 | NGC 3627 | SABb | AGN | 9.38 | 0.55 | 0.029 | 3
NGC 628 | Sc | SFG | 7.2 | 0.34 | 0.061 | 4 | NGC 3938 | Sc | SFG | 17.9 | 0.42 | 0.018 | 1
NGC 855 | E | SFG | 9.73 | 0.18 | 0.061 | 0 | NGC 4254 | Sc | SFG | 14.4 | 0.45 | 0.033 | 2
NGC 925 | SABd | SFG | 9.12 | 0.20 | 0.066 | 0 | NGC 4321 | SABbc | AGN | 14.3 | 0.43 | 0.023 | 3
NGC 1097 | SBb | AGN | 14.2 | 0.69 | 0.023 | 3 | NGC 4536 | SABbc | SFG | 14.5 | 0.45 | 0.016 | 0
NGC 1482 | S0 | SFG | 22.6 | 0.69 | 0.034 | 0 | NGC 4559 | SABcd | SFG | 6.98 | 0.28 | 0.015 | 0
NGC 1566 | SABbc | AGN | 18.0 | 0.62 | 0.008 | 3 | NGC 4625 | SABm | SFG | 9.3 | 0.53 | 0.016 | 0
NGC 2403 | SABcd | SFG | 3.5 | 0.22 | 0.034 | 6 | NGC 4631 | SBd | SFG | 7.62 | 0.28 | 0.015 | 3
NGC 2798 | SBa | SFG | 25.8 | 0.36 | 0.017 | 0 | NGC 4736 | Sab | AGN | 4.66 | 0.71 | 0.015 | 3
NGC 2915 | I0 | SFG | 3.78 | 0.14 | 0.236 | 0 | NGC 5055 | Sbc | AGN | 7.94 | 0.49 | 0.015 | 1
NGC 2976 | Sc | SFG | 3.55 | 0.36 | 0.064 | 2 | NGC 5194 | SABbc | AGN | 8.2 | 0.59 | 0.031 | 11
NGC 3034 | I0 | SFG | 3.50 | 0.30 | 0.134 | 0 | NGC 5713 | SABbc | SFG | 21.4 | 0.55 | 0.034 | 2
NGC 3049 | SBab | SFG | 19.2 | 0.40 | 0.033 | 0 | NGC 6946 | SABcd | SFG | 6.8 | 0.45 | 0.294 | 0
NGC 3184 | SABcd | SFG | 11.7 | 0.52 | 0.014 | 0 | NGC 7552 | Sc | SFG | 22.3 | 0.43 | 0.012 | 0
NGC 3198 | SBc | SFG | 14.1 | 0.30 | 0.011 | 0 | NGC 7793 | Sd | SFG | 3.91 | 0.31 | 0.017 | 4
NGC 3351 | SBb | SFG | 9.33 | 0.66 | 0.024 | 0 |  |  |  |  |  |  |

Note. — Col. (1): Galaxy name. Col. (2): Optical morphology from the NASA/IPAC
Extragalactic Database. Col. (3): Nuclear classification into SFG or AGN
according to the [N II]/H$\alpha$ versus [O III]/H$\beta$ emission-line
diagnostic diagram, based on spectra extracted from a central aperture of
2$\farcs$5$\times$2$\farcs$5 (Moustakas et al. 2010). Col. (4): Luminosity
distance, in units of Mpc, from Dale et al. (2017). Col. (5): [N II]
$\lambda\lambda 6548,6584/$H$\alpha$ ratio adopted from Kennicutt et al.
(2008). Col. (6): $E(B-V)$, in units of mag, from Schlafly & Finkbeiner
(2011). Col. (7): Number of extra-nuclear H II regions in each galaxy.

Table 3: Spatially Resolved Multiband Photometry

Region | log $f_{\rm FUV}$ | log $f_{\rm NUV}$ | log $f_{\rm H\alpha}$ | log $f_{J}$ | log $f_{H}$ | log $f_{Ks}$ | log $f_{\rm W1}$ | log $f_{\rm W2}$ | log $f_{\rm W3}$ | log $f_{\rm IRAC1}$ | log $f_{\rm IRAC2}$ | log $f_{\rm IRAC3}$ | log $f_{\rm IRAC4}$ | log $f_{\rm MIPS24}$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| (mJy) | (mJy) | (mJy) | (mJy) | (mJy) | (mJy) | (mJy) | (mJy) | (mJy) | (mJy) | (mJy) | (mJy) | (mJy) | (mJy)
(1) | (2) | (3) | (4) | (5) | (6) | (7) | (8) | (9) | (10) | (11) | (12) | (13) | (14) | (15)
NGC628_UVB_C | 0.09 $\pm$ 0.02 | 0.29 $\pm$ 0.01 | 1.37 $\pm$ 0.07 | 1.89 $\pm$ 0.03 | 1.95 $\pm$ 0.04 | 1.86 $\pm$ 0.02 | 1.58 $\pm$ 0.01 | 1.38 $\pm$ 0.01 | 1.90 $\pm$ 0.02 | 1.60 $\pm$ 0.04 | 1.43 $\pm$ 0.04 | 1.66 $\pm$ 0.04 | 2.02 $\pm$ 0.04 | 2.26 $\pm$ 0.02
NGC628_UVF_C | 0.49 $\pm$ 0.02 | 0.72 $\pm$ 0.01 | 1.68 $\pm$ 0.07 | 2.43 $\pm$ 0.03 | 2.48 $\pm$ 0.03 | 2.39 $\pm$ 0.02 | 2.05 $\pm$ 0.01 | 1.85 $\pm$ 0.01 | 2.39 $\pm$ 0.02 | 2.10 $\pm$ 0.04 | 1.95 $\pm$ 0.04 | 2.09 $\pm$ 0.04 | 2.40 $\pm$ 0.04 | 2.42 $\pm$ 0.02
NGC628_UVB_H1 |  $-$0.87 $\pm$ 0.02 |  $-$0.72 $\pm$ 0.01 | 0.76 $\pm$ 0.07 | 0.52 $\pm$ 0.05 | 0.48 $\pm$ 0.10 | 0.46 $\pm$ 0.03 | 0.33 $\pm$ 0.01 | 0.24 $\pm$ 0.01 | 1.09 $\pm$ 0.02 | 0.33 $\pm$ 0.04 | 0.36 $\pm$ 0.04 | 0.95 $\pm$ 0.04 | 1.19 $\pm$ 0.04 | 1.88 $\pm$ 0.02
NGC628_UVF_H1 |  $-$0.80 $\pm$ 0.02 |  $-$0.55 $\pm$ 0.01 | 0.76 $\pm$ 0.07 | 1.03 $\pm$ 0.04 | 1.01 $\pm$ 0.06 | 0.90 $\pm$ 0.03 | 0.73 $\pm$ 0.01 | 0.55 $\pm$ 0.01 | 1.34 $\pm$ 0.02 | 0.78 $\pm$ 0.04 | 0.58 $\pm$ 0.04 | 0.92 $\pm$ 0.04 | 1.37 $\pm$ 0.04 | 1.57 $\pm$ 0.02
NGC628_UVB_H2 |  $-$0.81 $\pm$ 0.02 |  $-$0.66 $\pm$ 0.01 | 0.71 $\pm$ 0.07 | 0.14 $\pm$ 0.13 | 0.03 $\pm$ 0.32 |  $-$0.02 $\pm$ 0.10 | 0.12 $\pm$ 0.01 | 0.04 $\pm$ 0.01 | 0.96 $\pm$ 0.02 | 0.15 $\pm$ 0.05 | 0.21 $\pm$ 0.04 | 0.85 $\pm$ 0.04 | 1.12 $\pm$ 0.04 | 1.65 $\pm$ 0.02
NGC628_UVF_H2 |  $-$0.94 $\pm$ 0.02 |  $-$0.72 $\pm$ 0.01 | 0.66 $\pm$ 0.07 | 0.40 $\pm$ 0.12 | 0.24 $\pm$ 0.34 |  $\cdots$ | 0.17 $\pm$ 0.01 |  $-$0.03 $\pm$ 0.01 | 1.07 $\pm$ 0.02 | 0.36 $\pm$ 0.05 | 0.15 $\pm$ 0.05 | 0.59 $\pm$ 0.04 | 1.14 $\pm$ 0.04 | 1.37 $\pm$ 0.02
NGC628_UVB_H3 |  $-$0.11 $\pm$ 0.02 |  $-$0.01 $\pm$ 0.01 | 1.26 $\pm$ 0.07 | 0.52 $\pm$ 0.09 | 0.46 $\pm$ 0.19 | 0.16 $\pm$ 0.10 | 0.43 $\pm$ 0.01 | 0.33 $\pm$ 0.01 | 1.24 $\pm$ 0.02 | 0.47 $\pm$ 0.04 | 0.44 $\pm$ 0.04 | 0.97 $\pm$ 0.04 | 1.33 $\pm$ 0.04 | 1.80 $\pm$ 0.02
NGC628_UVF_H3 |  $-$0.93 $\pm$ 0.02 |  $-$0.73 $\pm$ 0.01 | 0.58 $\pm$ 0.07 | 0.14 $\pm$ 0.18 | 0.08 $\pm$ 0.41 |  $\cdots$ |  $-$0.02 $\pm$ 0.01 |  $-$0.21 $\pm$ 0.01 | 0.82 $\pm$ 0.02 | 0.16 $\pm$ 0.05 |  $-$0.04 $\pm$ 0.05 | 0.38 $\pm$ 0.04 | 0.92 $\pm$ 0.04 | 0.95 $\pm$ 0.02
NGC628_UVB_H4 |  $-$1.19 $\pm$ 0.02 |  $-$1.10 $\pm$ 0.01 | 0.48 $\pm$ 0.07 |  $-$0.67 $\pm$ 0.51 |  $-$0.50 $\pm$ 0.68 |  $\cdots$ |  $-$0.56 $\pm$ 0.01 |  $-$0.62 $\pm$ 0.01 | 0.21 $\pm$ 0.02 |  $-$0.53 $\pm$ 0.06 |  $-$0.40 $\pm$ 0.05 | 0.23 $\pm$ 0.05 | 0.40 $\pm$ 0.05 | 0.90 $\pm$ 0.02
NGC628_UVF_H4 |  $-$1.09 $\pm$ 0.02 |  $-$0.89 $\pm$ 0.01 | 0.66 $\pm$ 0.07 |  $-$0.67 $\pm$ 1.17 |  $-$0.38 $\pm$ 1.19 |  $\cdots$ |  $-$0.39 $\pm$ 0.02 |  $-$0.52 $\pm$ 0.02 | 0.64 $\pm$ 0.02 |  $-$0.09 $\pm$ 0.06 |  $-$0.24 $\pm$ 0.06 | 0.32 $\pm$ 0.05 | 0.73 $\pm$ 0.04 | 1.00 $\pm$ 0.02

Note. — Spatially resolved multiband flux densities and corresponding
uncertainties. (This table is available in its entirety in machine-readable
form.)

##  2 Observational Material

###  2.1 Sample and Data Sets

This work is based on the Spitzer and ancillary data from SINGS (SINGS Team,
2020), a comprehensive IR imaging and spectroscopic survey that provides
mapping-mode Infrared Spectrograph (IRS; Werner et al. 2004) observations of
the central regions of 75 nearby galaxies, as well as of the extra-nuclear H
II regions of a subset of 23 galaxies therein. The IRS has both low-resolution
and high-resolution modules, each of which includes two modes, namely the
short-low/long-low (SL/LL) modes and the short-high/long-high (SH/LH) modes.
Each mode of the low-resolution module further consists of two orders (SL1/SL2
and LL1/LL2). Mapping-mode observations are conducted by scanning the slit of
each mode to map a section of a galaxy. This paper only uses the low-
resolution data, which cover the wavelength range $\sim 5-38\ {\rm\mu m}$ with
a full width at half maximum (FWHM) spectral resolution of
$\lambda/\Delta\lambda\approx 64-128$, depending on the observing mode. Our
analysis is based on spectral data cubes constructed using CUBISM (Smith et
al., 2007a).

Figure 1 schematically illustrates the spatial coverage of the observations.
The LL mode data cubes, covering $14-38\ {\rm\mu m}$, map an area
$50\arcsec\times 300\arcsec$ across the nuclear region of each galaxy with a
point-spread function (PSF) of ${\rm FWHM}\approx 4\farcs 5-9\farcs 0$. The SL
mode, capturing $5-14\ {\rm\mu m}$, samples a more limited area of
$15\arcsec\times 50\arcsec$ with ${\rm FWHM}\approx 1\farcs 5-3\farcs 5$
resolution centered on extra-nuclear H II regions. Altogether, the spatially
resolved spectra of the nuclear regions and H II regions provide a broad
census of the behavior of PAH emission across a diverse range of environments.
The large dynamic range in star-forming activity permits us to disentangle the
contribution of evolved stars to the excitation of PAH emission.

None of the mapping-mode data cubes provides sufficient, continuous spectral
coverage that can simultaneously measure the principal PAH features at 6.2,
7.7, 8.6, 11.3, and 17.0 ${\rm\mu m}$. The central region of each galaxy was
mapped only in the LL mode, while the extra-nuclear regions were confined to
the SL mode. Zhang et al. (2021) devised a method that supplements the gaps in
spectral coverage with mid-IR photometric measurements from WISE (Wright et
al., 2010; NASA/IPAC Infrared Science Archive, 2020) and Spitzer IRAC (Fazio
et al., 2004) and MIPS (Rieke et al., 2004). The combination of the photometry
and partial spectroscopy yields a spectral decomposition that can produce a
robust measurement of the integrated $5-20\ {\rm\mu m}$ PAH flux, as well as
the strengths of the individual PAH features. Zhang et al. (2022) further
extended the method to deliver estimates of the stellar mass by adding the
near-IR bands of 2MASS (Skrutskie et al., 2006). Table 1 summarizes the
multiwavelength imaging data sets used in this paper, which for the SINGS
galaxies include images from the 2MASS Large Galaxy
Atlas222https://irsa.ipac.caltech.edu/data/LGA/ (Jarrett et al., 2003, 2020),
the WISE Atlas333https://irsa.ipac.caltech.edu/applications/wise/ (Wright et
al., 2010; NASA/IPAC Infrared Science Archive, 2020), and MIPS24 (Engelbracht
et al., 2007). We use H$\alpha$
imaging444https://irsa.ipac.caltech.edu/data/SPITZER/SINGS/doc/sings_fifth_delivery_v2.pdf
to set the benchmark for star formation activity, after correcting for
contamination to the narrowband filter from [N II] $\lambda\lambda 6548,6584$
using the [N II]/H$\alpha$ ratios of Kennicutt et al. (2008). We additionally
use broadband far-ultraviolet (FUV; 1350 Å–1750 Å) and near-ultraviolet (NUV;
1750 Å–2750 Å) images from the GALEX Ultraviolet Atlas of Nearby
Galaxies555https://archive.stsci.edu/prepds/galex_atlas/ (Gil de Paz et al.
2007) to define two categories of regions (UV-bright and UV-faint), as
illustrated in the bottom panel of Figure 1 and described in Section 2.2.

The above data requirements can be fulfilled by the central regions of 25 SFGs
and 55 extra-nuclear H II regions drawn from 10 star-forming galaxies (SFGs)
and 8 galaxies hosting active galactic nuclei (AGNs) in the parent sample of
SINGS, as listed in Table 2. We have excluded four dwarf galaxies (IC 4710,
NGC 1705, NGC 6822, and Tol 89), whose PAHs are likely systematically depleted
by photo-destruction in the hard radiation field of low-metallicity
environments (Xie & Ho 2019; Zhang et al. 2022).

###  2.2 Data Processing and Definition of UV-bright and UV-faint Regions

The ancillary multi-wavelength images summarized in Table 1 form an integral
part of our analysis, both in providing information on star formation (UV and
H$\alpha$) and stellar mass ($JHK_{s}$), as well as supplementing the
incomplete spectral coverage of the IRS spectra to enable spectral
decomposition of the PAH emission. For the bands that display evident
background residuals (i.e., H$\alpha$, 2MASS, WISE, IRAC and MIPS), we remove
the artifacts by fitting and subtracting a two-dimensional polynomial function
after masking out the real sources. The spatially resolved analysis of the IRS
mapping-mode observations is performed using the method developed by Zhang et
al. (2021, 2022), as summarized below.

To enable spatially resolved analysis, each IRS data cube and its respective
ancillary images are convolved to the same angular resolution, which is set to
${\rm FWHM}=6\farcs 5$ for the SL data and ${\rm FWHM}=10\farcs 0$ for the LL
data. These resolutions are chosen to be slightly larger than the largest PSF
of the slices of the spectral data cubes (SL for the extra-nuclear H II
regions; LL for the nuclear regions) and the relevant ancillary images (Table
1). We perform the convolution using the convolve_fft function in
astropy.convolution (Astropy Collaboration et al. 2013), using convolution
kernels constructed following Aniano et al. (2011). All the convolved images
are then reprojected into the same coordinate frame as the IRS data cubes,
which has a pixel size of 1$\farcs$8 for the SL mode and 5$\farcs$1 for the LL
mode. The spatially resolved multiband photometry is given in Table 3. All the
extracted IRS spectra are scaled to mitigate against the systematics produced
by the convolution of the spectral data cube. The scaling factor for each
integrated spectrum is determined individually by minimizing the weighted
difference between the observed flux densities and those synthesized from the
integrated IRS spectrum using the mid-IR bands that encompass the
corresponding spectral range.

The main goal of this paper is to investigate whether evolved stars
participate in the excitation of PAH emission, and, if so, to devise a means
to quantify and remove its contribution from the PAH-based SFR estimator.
Toward this end, we classify the spaxels into two categories according to
their level of star formation activity, as judged by their UV brightness in
the GALEX images: (1) UV-bright spaxels correspond to those that have flux
densities that are $>20\,\sigma$, where $\sigma$ is the background noise of
the FUV and NUV images, and (2) UV-faint spaxels have flux densities between 3
$\sigma$ and 20 $\sigma$. The threshold between the two categories is, to some
degree, arbitrary, but careful examination reveals that our choice effectively
separates obvious, intense star-forming regions associated with spiral arms on
the one hand, and more diffuse, less active inter-arm regions on the other
hand. We use UV emission instead of H$\alpha$ emission as the benchmark
because the UV continuum traces star formation activity on longer timescales.
Moreover, in the case of H II regions in AGN host galaxies, we wish to avoid
potential contamination from nebular emission from the AGN-photoionized
narrow-line region, which can extend to sizable portions of the galactic disk
(e.g., Molina et al. 2022).

Then, we merge all the UV-bright spaxels of the nuclear region into a combined
SED of high signal-to-noise ratio, and similarly for the UV-faint spaxels. The
same procedure is applied to each of the extra-nuclear H II regions, if
present. In the end, we have at least one UV-bright and one UV-faint SED for
each galaxy (the exact number depends on the number of extra-nuclear H II
regions, which varies from 0 to 11; see Table 2), which represent,
respectively, the SEDs for regions with minimal versus substantial
contributions from evolved stars.

Figure 2: Illustration of the PAH decomposition for the SFG NGC 4254, showing
(a) an SL spectrum of a spaxel from an H II region and (b) an LL spectrum of a
spaxel from the nuclear region. The gray solid line is the IRS spectrum, and
the black points are the photometric measurements. The red solid line is the
best-fit model, which is composed of a PAH template, a stellar continuum
component (Bruzual & Charlot 2003; BC03), and three dust continuum components
each represented by a modified blackbody (MBB).

###  2.3 PAH, Stellar Mass, and SFR Measurements

Our method for measuring PAH is based on the technique of Xie et al. (2018).
After masking the strong ionic emission lines that are not blended with the
main PAH features, we fit the $\sim 5-38\,{\rm\mu m}$ spectrum with a
multicomponent model consisting of a theoretical PAH template plus dust
continuum represented by three modified blackbodies of different temperatures,
all subject to dust attenuation by foreground extinction. This method can
decompose robustly the complex PAH features from low-resolution IRS spectra,
as has been applied extensively in wide range of galactic environments,
including the SINGS galaxies (Xie & Ho, 2019, 2022; Zhang et al., 2022). Zhang
et al. (2021) modified the technique to enable reliable PAH measurement even
when only partial low-resolution IRS spectra are available, by combining
either the SL or LL spectra with properly matched mid-IR photometry to
complement the missing spectral region. This improvement is essential to take
advantage of the mapping-mode observations of SINGS, which lack full spectral
coverage for most of the mapped regions of their sample galaxies. Zhang et al.
(2022) further extended the SED coverage into the near-IR using the $JHK_{s}$
2MASS photometric data in order to estimate the stellar mass. The stellar
component is represented by a stellar population model from Bruzual & Charlot
(2003), assuming solar metallicity, a Chabrier (2003) stellar initial mass
function, and a fixed stellar age of 6 Gyr, which is otherwise difficult to
solve using IR information alone. Fortunately, the choice of stellar age does
not affect our conclusions because the near-IR stellar emission is governed
mostly by evolved stars, which have similar SEDs in the near-IR.

Figure 2 illustrates how the SED is decomposed by the multicomponent model. We
use the Bayesian Markov chain Monte Carlo procedure emcee in the Python
package to determine the posterior distribution of each best-fit parameter
(see Section 4.2 in Shangguan et al. 2018 for more details). The median and
standard deviation of the posterior distribution of each best-fit parameter
are taken as the final estimate and corresponding uncertainty, respectively.
Following extensive experiments performed in Zhang et al. (2021), we add an
extra 20% uncertainty to the IRS spectrum to balance the weighting of the
spectroscopic and photometric data. The PAH luminosity, $L_{\rm PAH}$, listed
in Table 4 pertains to the integral of the best-fitting PAH template over the
$5-20\ {\rm\mu m}$ region. The integrated PAH emission is less affected by the
potential variation of individual PAH features due to variations in radiation
intensity, which, in any case, are not strong for the range of conditions
sampled by the nearby galaxies studied here (Xie et al., 2018). The strength
of individual PAH features is strongly correlated with the strength of the
integrated PAH emission (see, e.g., Table 4 in Zhang et al. 2022), but in this
study we will not consider them further.

We use H$\alpha$ emission to quantify the strength of star formation activity,
adopting the SFR indicator given by Kennicutt & Evans (2012). To correct for
foreground extinction, we use the Galactic extinction curve of Cardelli et al.
(1989) with $R_{V}=3.1$, for which $A_{\rm H\alpha}=2.535\,E(B-V)$ (Kennicutt
et al., 2007), and reddening values from Schlafly & Finkbeiner (2011).
Correction for internal extinction follows the prescription of Calzetti et al.
(2007) for resolved H II regions, $L_{\rm H\alpha}=L_{\rm H\alpha}^{\rm
obs}+0.031\,L_{\rm 24\,{\rm\mu m}}$, where $L_{\rm H\alpha}^{\rm obs}$ is the
Galactic extinction-corrected H$\alpha$ luminosity and $L_{\rm 24\,{\rm\mu
m}}$ is derived from MIPS24 images. The spatially resolved measurements of PAH
luminosity, ${\rm SFR}=5.3\times\,10^{-42}L_{\rm H\alpha}$, and stellar mass
are provided in Table 4.

Table 4: Spatially Resolved Measurements

Region | log $L_{\rm PAH}$ | log $L_{\rm H\alpha}$ | log $M_{\rm\star}$
---|---|---|---
| ($\rm erg\ s^{-1}$) | ($\rm erg\ s^{-1}$) | ($M_{\odot}$)
(1) | (2) | (3) | (4)
NGC628_UVB_C | 41.22 $\pm$ 0.09 | 39.79 $\pm$ 0.11 | 8.75 $\pm$ 0.13
NGC628_UVF_C | 41.62 $\pm$ 0.10 | 40.00 $\pm$ 0.10 | 9.27 $\pm$ 0.13
NGC628_UVB_H1 | 40.39 $\pm$ 0.09 | 39.36 $\pm$ 0.13 | 7.29 $\pm$ 0.13
NGC628_UVF_H1 | 40.57 $\pm$ 0.09 | 39.12 $\pm$ 0.11 | 7.83 $\pm$ 0.13
NGC628_UVB_H2 | 40.36 $\pm$ 0.09 | 39.16 $\pm$ 0.11 | 6.82 $\pm$ 0.14
NGC628_UVF_H2 | 40.36 $\pm$ 0.09 | 38.96 $\pm$ 0.10 | 6.32 $\pm$ 0.25
NGC628_UVB_H3 | 40.59 $\pm$ 0.09 | 39.46 $\pm$ 0.08 | 7.20 $\pm$ 0.14
NGC628_UVF_H3 | 40.14 $\pm$ 0.09 | 38.69 $\pm$ 0.07 | 6.10 $\pm$ 0.25
NGC628_UVB_H4 | 39.71 $\pm$ 0.09 | 38.62 $\pm$ 0.07 | 6.08 $\pm$ 0.16
NGC628_UVF_H4 | 40.00 $\pm$ 0.09 | 38.76 $\pm$ 0.07 | 5.67 $\pm$ 0.19

Note. — Col. (1): Resolved region. Col. (2): Integrated ($5-20\ {\rm\mu m}$)
PAH luminosity. Col. (3): Extinction-corrected H$\alpha$ luminosity. Col. (4):
Stellar mass. (This table is available in its entirety in machine-readable
form.)

Figure 3: (a) The correlation between integrated PAH luminosity and
extinction-corrected H$\alpha$ luminosity for the UV-bright (UVB) and UV-faint
(UVF) regions, color-coded according to the sSFR. The black solid line is the
best linear regression fit for both regions combined (Equation 1). (b) The
dependence of the residual PAH luminosity between the observed PAH luminosity
and the best-fit value (the black solid line in panel a) on the sSFR, color-
coded according to the sSFR. The lower-right shows the Spearman correlation
coefficient $\rho$ and its statistical significance $p$. The median
uncertainty is shown in the upper-right corner of each panel. Figure 4: (a)
The correlation between observed SFR and PAH luminosity for the UV-bright
(UVB) and UV-faint (UVF) regions, color-coded according to the sSFR. The black
solid line is the direct calibration between observed SFR and PAH luminosity
(Equation 3), and the black dashed line is the calibration for the sample of
SFGs studied by Xie & Ho (2019). (b) The difference between the revised SFR
estimates (Equation 4) and the direct SFR estimates (Equation 3).

##  3 Results

###  3.1 Contribution of Evolved Stars to PAH Emission

We begin by examining the correlation between PAH luminosity and extinction-
corrected H$\alpha$ luminosity, or, equivalently, SFR. For UV-bright and UV-
faint regions combined, PAH emission tightly correlates with H$\alpha$
emission (Figure 3a). A linear regression fit with the Python package of the
Bayesian method linmix (Kelly 2007) gives

$\displaystyle\begin{aligned} &{\rm log}\,L_{\rm PAH}=(0.99\pm 0.024)({\rm
log}\,L_{\rm H\alpha}-39.5)+(40.92\pm 0.020),\end{aligned}$ (1)

with a total scatter of $\epsilon_{t}=0.21$ dex and an intrinsic scatter of
$\epsilon_{i}=0.16$ dex. The total scatter is obtained as the root-mean-square
error of the best-fit, and the intrinsic scatter is calculated assuming that
the observed PAH luminosity is the dependent variable. Color coding the points
by the specific SFR (${\rm sSFR\equiv SFR}/M_{*}$) reveals a subtle, but
distinctive pattern. We observe that sSFR increases from the top-left to the
lower-right of Figure 3a, perpendicular to the correlated points. At a given
strength of star formation activity, UV-faint regions, which have
characteristically lower sSFR and hence a higher fraction of evolved stars,
show stronger PAH emission than UV-bright regions, which tend to have higher
values of sSFR. Figure 3b highlights this trend more explicitly by plotting as
a function of sSFR the residual difference between the observed PAH luminosity
and the PAH luminosity predicted by Equation 1. According to Figure 3b, there
is a strong inverse correlation between the relative PAH excess and sSFR: the
Spearman correlation coefficient is $\rho=-0.66$, and the corresponding
statistical significance parameter is $p<10^{-15}$.

The above analysis shows that at given a SFR, star-forming regions with lower
sSFR, namely those that contain more evolved stars, exhibit relatively
stronger PAH emission. We suggest that this reflects the contribution of
evolved stars to PAH heating. Although massive star formation activity can
destroy PAHs and result in PAH suppression (Gordon et al., 2008; Lebouteiller
et al., 2011), note that weaker PAH emission is found in regions with higher
sSFR, rather than in those with higher SFR (Figure 4). On average, UV-faint
regions ($\langle\log{\rm sSFR}\rangle=-10.03$) have a positive residual PAH
luminosity ($\langle\Delta\log L_{\rm PAH}\rangle=+0.049$) while UV-bright
regions ($\langle\log{\rm sSFR}\rangle=-9.86$) exhibit the opposite behavior
($\langle\Delta\log L_{\rm PAH}\rangle=-0.024$). We conclude that evolved
stars contribute to PAH emission, and that such an effect, while relatively
small, is systematic and should be considered whenever possible when using PAH
emission to estimate SFR.

Having established that stellar mass contributes as a second parameter to the
systematic scatter in the $L_{\rm PAH}-L_{\rm H\alpha}$ relation, we formally
introduce $M_{\star}$ into a multiple regression analysis using the Python
package statsmodels (Seabold & Perktold, 2010):

$\displaystyle\begin{aligned} &{\rm log}\,L_{\rm PAH}=(0.71\pm 0.035)({\rm
log}\,L_{\rm H\alpha}-39.5)\ +\\\ &(0.28\pm 0.032)({\rm
log}\,M_{\star}-6.5)+(40.47\pm 0.056).\end{aligned}$ (2)

Compared with Equation 1, Equation 2 yields a slightly reduced total
($\epsilon_{t}=0.17$ dex) and intrinsic ($\epsilon_{i}=0.11$ dex) scatter.

###  3.2 Revised PAH-based SFR Calibration

Figure 4a gives the conventional relationship between SFR and PAH luminosity
for our sample. The best-fit linear regression (solid line), with
$\epsilon_{t}=0.21$ dex and $\epsilon_{i}=0.16$ dex, is

$\displaystyle\begin{aligned} &{\rm log}\,{\rm SFR}=(0.97\pm 0.023)({\rm
log}\,L_{\rm PAH}-39.5)-(3.14\pm 0.041).\end{aligned}$ (3)

The influence of evolved stars again becomes quite clear once we color code
the points by their sSFR. The effect is further accentuated by superposing (in
dashed line) the calibration of Xie & Ho (2019)666Xie & Ho (2019) define their
integrated PAH emission over the region $5-15\ {\rm\mu m}$; their values need
to be multiplied by a factor of 1.15 to be consistent with our definition,
which is defined over $5-20\ {\rm\mu m}$. The SFRs of Xie & Ho (2019) also
need to be multiplied by a factor of 0.68 due to be consistent with our choice
of initial mass function., which has a higher zero point because their sample
of massive SFGs has on average a factor of $\sim 2$ higher sSFR than our
sample ($\langle\log{\rm sSFR}\rangle=-9.60$ versus $-9.93$).

Whenever possible, stellar mass should be included explicitly as an additional
parameter to estimate SFRs, in order to remove the contribution of evolved
stars to the excitation of PAH emission. We propose the following revised SFR
calibration ($\epsilon_{t}=0.21$ dex, $\epsilon_{i}=0.14$ dex):

$\displaystyle\begin{aligned} &{\rm log}\,{\rm SFR}=(1.08\pm 0.052)({\rm
log}\,L_{\rm PAH}-39.5)\ -\\\ &(0.12\pm 0.048)({\rm
log}\,M_{\star}-6.5)-(3.11\pm 0.040).\end{aligned}$ (4)

Although the revised SFR calibration shows no signficant improvement in
intrinsic scatter compared to the original calibration (Equation 3), the
revised calibration provides more physically accurate and more robust SFR
estimates, as evidenced by the systematic trend in Figure 4b. At a given
observed PAH luminosity, the revised SFR estimates deviate systematically from
the original values according to the sSFR (a proxy for the relative
contribution from evolved stars). The revised calibration provides higher SFRs
for regions with higher sSFR (blueish points), while the opposite situation
holds for regions with lower sSFR (reddish points). This trend naturally
arises if evolved stars contribution to PAH emission. The deviation on average
is only at the level $\pm 0.05$ dex, but in the extreme it can reach $\pm 0.2$
dex.

##  4 Discussion

Figure 5: The correlation between integrated PAH luminosity and extinction-
corrected H$\alpha$ luminosity for the resolved, sub-kpc-scale spaxels from
the central $\sim$30″$\times$50″ regions of SINGS SFGs (Zhang et al., 2022).
The black solid line is the best linear regression fit for the UV-bright (UVB)
and UV-faint (UVF) regions (Equation 1), while the black dashed line is the
best linear regression fit for the resolved spaxels (Equation 5). Figure 6:
The correlation between observed SFRs and revised SFRs (Equation 4) for (a)
UV-bright (UVB) and UV-faint (UVF) regions and (b) resolved spaxels from the
central $\sim$30″$\times$50″ regions of SINGS SFGs (Zhang et al., 2022),
color-coded according to the sSFR.

A number of studies have noted that the correlation between PAH strength and
SFR is sub-linear. For instance, Calzetti et al. (2005, 2007) and Wu et al.
(2005) reported a best-fit slope of $\sim 0.9$ between the 8 ${\rm\mu m}$ band
PAH emission and different SFR indicators for resolved H II regions and SFGs.
In their spatially resolved analysis of M 51, Zhang et al. (2021) also found a
sub-linear slope of 0.85 for the correlation between PAH emission and star
formation indicators. A similar result appeared in Zhang et al. (2022), who
performed spatially resolved analysis of the SINGS sample, which includes the
galaxies included in this study.

The sub-linear PAH–SFR correlation arises naturally if, as we contend, evolved
stars contribute to PAH emission. We demonstrate this in Figure 5, which
repeats the $L_{\rm PAH}-L_{\rm H\alpha}$ relation of Figure 3a, but now
emphasizes instead the spaxels resolved on sub-kpc scales from the central
$\sim$30″$\times$50″ regions of the SINGS SFGs (Zhang et al., 2022). The
corresponding linear regression fit ($\epsilon_{t}=0.21$ dex,
$\epsilon_{i}=0.19$ dex),

$\displaystyle\begin{aligned} &{\rm log}\,L_{\rm PAH}=(0.86\pm 0.022)({\rm
log}\,L_{\rm H\alpha}-39.5)+(40.93\pm 0.018),\end{aligned}$ (5)

is clearly flatter (slope 0.86; dashed line) than that derived in Equation 1
for the much larger, spatially integrated UV-bright and UV-faint regions,
whose slope is effectively unity (0.99, dashed line). Evidently, the chief
culprit for the flatter correlation is the difference in relative contribution
of evolved stars to PAH emission at the two ends of the SFR distribution.
Unlike the UV-bright and UV-faint regions, which exhibit sSFR variation
perpendicular to the correlation between PAH emission and SFR (Figure 3a), the
resolved spaxels show sSFR varying along the correlation. The increase of sSFR
along the correlation elevates PAH emission at the low-SFR end, all the while
diminishing PAH emission at the high-SFR end, the combination of which
produces a flattening of the PAH–SFR correlation. While the destruction of
PAHs within the resolved spaxels of the strongest SFR potentially can
contribute to the flattening, this effect should play a secondary role here.
In contrast to the sensitivity of the slope of the $L_{\rm PAH}-L_{\rm
H\alpha}$ relation to the degree to which the galaxy is resolved, the
differences disappear when we use the revised SFR estimates, which take into
consideration the contribution of evolved stars (Figure 6).

Given that the sum of the two coefficients in Equation 2 is nearly unity,
galaxies on the star-forming main sequence, whose SFR scales approximately
linearly with stellar mass (e.g., Brinchmann et al. 2004; Salim et al. 2007;
Peng et al. 2010), naturally present a linear correlation between PAH emission
and SFR, if they are not biased by sample selection. Namely, the linear
correlation between PAH and H$\alpha$ emission for UV-bright and UV-faint
regions merely reflects the linear correlation between SFR and stellar mass.
Similarly, the sub-linear correlation between PAH emission and SFR indicator
implies a super-linear correlation between SFR and stellar mass (i.e., a sub-
linear correlation between stellar mass and SFR). As the slope of the resolved
star-forming main sequence spans a large range ($\sim 0.5-1.5$; Maragkoudakis
et al. 2017; Hall et al. 2018), we argue that the correlation between SFR and
PAH luminosity can be either linear or non-linear, depending on the sSFR, or
the relative contribution of evolved stars, of the sample used for the
empirical calibration.

We close with a brief remark concerning the possible relevance of our results
to the PAH deficit associated with metal-poor galactic environments (e.g.,
Engelbracht et al. 2005; Jackson et al. 2006; Madden et al. 2006; Smith et al.
2007b; Wu et al. 2007; Khramtsova et al. 2013; Xie & Ho 2019). The destruction
of PAH molecules by the hard radiation field under low-metallicity conditions
is widely invoked as the main reason for the observed deficit (Jackson et al.,
2006; Madden et al., 2006; Hunt et al., 2010; Khramtsova et al., 2013).
However, low-metallicity environments might also lead to the delayed formation
of PAH molecules, if PAHs are produced by asymptotic giant branch stars
(Latter, 1991; Dwek, 1998), which may not have accumulated in sufficient
numbers in the young stellar population of low-metallicity environments
(Engelbracht et al., 2005; Wu et al., 2007; Galliano et al., 2008a).
Regardless of the mechanism responsible for the reduction of PAH content, the
results of this work suggest that the characteristically lower fraction of
evolved stars in low-metallicity environments should contribute, at least in
part, to their observed PAH deficit.

##  5 Summary

PAH emission, a commonly used indicator of star formation in galaxies, may be
excited by both young and old stars. To assess the contribution of evolved
stars to PAH emission, we perform a spatially resolved analysis of the nuclear
and extra-nuclear regions of 33 nearby galaxies selected from the SINGS
project. Employing the technique of Zhang et al. (2021, 2022), we successfully
combine mapping-mode low-resolution IRS spectra with complementary near-IR and
mid-IR images to derive robust meaurements of PAH luminosity and stellar mass.
Together with SFRs obtained from narrowband H$\alpha$ images, we reinvestigate
the correlation between PAH luminosity and SFR and its possible dependence on
a second parameter. Classifying the level of star formation activity with the
intensity of UV emission measured from GALEX images, we show that UV-faint
regions, which have characteristically lower sSFR and hence more prominent
contribution from evolved stars, tend to exhibit stronger PAH emission at a
given SFR than UV-bright regions. We propose a revised calibration for
estimating SFR based on both PAH luminosity and stellar mass. This new
calibration, which explicitly takes into account the contribution of evolved
stars to PAH emission, provides more accurate SFRs, especially for systems of
low sSFR. The contribution of evolved stars to PAH emission naturally explains
the well-documented sub-linear slope of the correlation between PAH emission
and SFR, and it partly alleviates the PAH deficit observed in dwarf galaxies
and other low-metallicity environments.

We thank the anonymous referee for constructive suggestions. This work was
supported by the National Science Foundation of China (11721303, 11991052,
12011540375) and the China Manned Space Project (CMS-CSST-2021-A04, CMS-
CSST-2021-A06).

## References

  * Allamandola et al. (1985) Allamandola, L. J., Tielens, A. G. G. M., & Barker, J. R. 1985, ApJ, 290, L25
  * Allamandola et al. (1989) Allamandola, L. J., Tielens, A. G. G. M., & Barker, J. R. 1989, ApJS, 71, 733
  * Aniano et al. (2011) Aniano, G., Draine, B. T., Gordon, K. D., et al. 2011, PASP, 123, 1218
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33
  * Bendo et al. (2008) Bendo, G. J., Draine, B. T., Engelbracht, C. W., et al. 2008, MNRAS, 389, 629
  * Bendo et al. (2020) Bendo, G. J., Lu, N., & Zijlstra, A. 2020, MNRAS, 496, 1393
  * Bendo et al. (2010) Bendo, G. J., Wilson, C. D., Warren, B. E., et al. 2010, MNRAS, 402, 1409
  * Brinchmann et al. (2004) Brinchmann, J., Charlot, S., White, S. D. M., et al. 2004, MNRAS, 351, 1151
  * Bruzual & Charlot (2003) Bruzual, G., & Charlot, S. 2003, MNRAS, 344, 1000
  * Calzetti et al. (2005) Calzetti, D., Kennicutt Jr., R. C., Bianchi, L., et al. 2005, ApJ, 633, 871
  * Calzetti et al. (2007) Calzetti, D., Kennicutt Jr., R. C., Engelbracht, C. W., et al. 2007, ApJ, 666, 870
  * Cardelli et al. (1989) Cardelli, J. A., Clayton, G. C., & Mathis, J. S. 1989, ApJ, 345, 245
  * Chabrier (2003) Chabrier, G. 2003, PASP, 115, 763
  * Cluver et al. (2017) Cluver, M. E., Jarrett, T. H., Dale, D. A., et al. 2017, ApJ, 850, 68
  * Cluver et al. (2014) Cluver, M. E., Jarrett, T. H., Hopkins, A. M., et al. 2014, ApJ, 782, 90
  * Cortzen et al. (2019) Cortzen, I., Garrett, J., Magdis, G., et al. 2019, MNRAS, 482, 1618
  * Crocker et al. (2013) Crocker, A. F., Calzetti, D., Thilker, D. A., et al. 2013, ApJ, 762, 79
  * Dale et al. (2017) Dale, D. A., Cook, D. O., Roussel, H., et al. 2017, ApJ, 837, 90
  * Dale et al. (2007) Dale, D. A., Gil de Paz, A., Gordon, K. D., et al. 2007, ApJ, 655, 863
  * Donoso et al. (2012) Donoso, E., Yan, L., Tsai, C., et al. 2012, ApJ, 748, 80
  * Draine & Li (2001) Draine, B. T., & Li, A. 2001, ApJ, 551, 807
  * Duley & Williams (1981) Duley, W. W., & Williams, D. A. 1981, MNRAS, 196, 269
  * Dwek (1998) Dwek, E. 1998, ApJ, 501, 643
  * Engelbracht et al. (2007) Engelbracht, C. W., Blaylock, M., Su, K. Y. L., et al. 2007, PASP, 119, 994
  * Engelbracht et al. (2005) Engelbracht, C. W., Gordon, K. D., Rieke, G. H., et al. 2005, ApJ, 628, L29
  * Fazio et al. (2004) Fazio, G. G., Hora, J. L., Allen, L. E., et al. 2004, ApJS, 154, 10
  * Förster Schreiber et al. (2004) Förster Schreiber, N. M., Roussel, H., Sauvage, M., et al. 2004, A&A, 419, 501
  * Galliano et al. (2008a) Galliano, F., Dwek, E., & Chanial, P. 2008a, ApJ, 672, 214
  * Galliano et al. (2008b) Galliano, F., Madden, S. C., Tielens, A. G. G. M., et al. 2008b, ApJ, 679, 310
  * Gil de Paz et al. (2007) Gil de Paz, A., Boissier, S., Madore, B. F., et al. 2007, ApJS, 173, 185
  * Gordon et al. (2008) Gordon, K. D., Engelbracht, C. W., Rieke, G. H., et al. 2008, ApJ, 682, 336
  * Haas et al. (2002) Haas, M., Klaas, U., & Bianchi, S. 2002, A&A, 385, L23
  * Hall et al. (2018) Hall, C., Courteau, S., Jarrett, T., et al. 2018, ApJ, 865, 154
  * Houck et al. (2004) Houck, J. R., Roellig, T. L., Van Cleve, J., et al. 2004, SPIE, 5487, 62
  * Hunt et al. (2010) Hunt, L. K., Thuan, T. X., Izotov, Y. I., et al. 2010, ApJ, 712, 164
  * Jackson et al. (2006) Jackson, D. C., Cannon, J. M., Skillman, E. D., et al. 2006, ApJ, 646, 192
  * Jarrett et al. (2003) Jarrett, T. H., Chester, T., Cutri, R., et al. 2003, AJ, 125, 525
  * Jarrett et al. (2020) Jarrett, T. H., Chester, T., Cutri, R., et al. 2020, 2MASS Large Galaxy Atlas, IPAC, doi:10.26131/IRSA122
  * Jarrett et al. (2013) Jarrett, T. H., Masci, F., Tsai, C. W., et al. 2013, AJ, 145, 6
  * Jones et al. (2015) Jones, A. G., Bendo, G. J., Baes, M., et al. 2015, MNRAS, 448, 168
  * Kelly (2007) Kelly, B. C. 2007, ApJ, 665, 1489
  * Kennicutt et al. (2003) Kennicutt Jr., R. C., Armus, L., Bendo, G., et al. 2003, PASP, 115, 928
  * Kennicutt et al. (2007) Kennicutt Jr., R. C., Calzetti, D., Walter, F., et al. 2007, ApJ, 671, 333
  * Kennicutt & Evans (2012) Kennicutt Jr., R. C., & Evans, N. J. 2012, ARA&A, 50, 531
  * Kennicutt et al. (2008) Kennicutt Jr., R. C., Lee, J. C., Funes, J. G., et al. 2008, ApJS, 178, 247
  * Khramtsova et al. (2013) Khramtsova, M. S., Wiebe, D. S., Boley, P. A., et al. 2013, MNRAS, 431, 2006
  * Latter (1991) Latter, W. B. 1991, ApJ, 377, 187
  * Lebouteiller et al. (2011) Lebouteiller, V., Bernard-Salas, J., Whelan, D. G., et al. 2011, ApJ, 728, 45
  * Lee et al. (2013) Lee, J. C., Hwang, H. S., & Ko, J. 2013, ApJ, 774, 62
  * Leger & Puget (1984) Leger, A., & Puget, J. L. 1984, A&A, 500, 279
  * Li & Draine (2002) Li, A., & Draine, B. T. 2002, ApJ, 572, 232
  * Lu et al. (2014) Lu, N., Bendo, G. J., Boselli, A., et al. 2014, ApJ, 797, 129
  * Madden et al. (2006) Madden, S. C., Galliano, F., Jones, A. P., et al. 2006, A&A, 446, 877
  * Mahajan et al. (2019) Mahajan, S., Ashby, M. L. N., Willner, S. P., et al. 2019, MNRAS, 482, 560
  * Maragkoudakis et al. (2018) Maragkoudakis, A., Ivkovich, N., Peeters, E., et al. 2018, MNRAS, 481, 5370
  * Maragkoudakis et al. (2017) Maragkoudakis, A., Zezas, A., Ashby, M. L. N., et al. 2017, MNRAS, 466, 1192
  * Marshall et al. (2007) Marshall, J. A., Herter, T. L., Armus, L., et al. 2007, ApJ, 670, 129
  * Molina et al. (2022) Molina, J., Ho, L. C., Wang, R., et al. 2022, ApJ, 935, 72
  * Morrissey et al. (2007) Morrissey, P., Conrow, T., Barlow, T. A., et al. 2007, ApJS, 173, 682
  * Moustakas et al. (2010) Moustakas, J., Kennicutt Jr., R. C., Tremonti, C. A., et al. 2010, ApJS, 190, 233
  * NASA/IPAC Infrared Science Archive (2020) NASA/IPAC Infrared Science Archive, 2020, WISE All-Sky 4-band Atlas Coadded Images, IPAC, doi:10.26131/IRSA151
  * Peeters et al. (2002) Peeters, E., Hony, S., Van Kerckhoven, C., et al. 2002, A&A, 390, 1089
  * Peeters et al. (2004) Peeters, E., Spoon, H. W. W., & Tielens, A. G. G. M. 2004, ApJ, 613, 986
  * Peng et al. (2010) Peng, Y.-J., Lilly, S. J., Kovač, K., et al. 2010, ApJ, 721, 193
  * Rieke et al. (2004) Rieke, G. H., Young, E. T., Engelbracht, C. W., et al. 2004, ApJS, 154, 25
  * Rigby et al. (2022) Rigby, J., Perrin, M., McElwain, M., et al. 2022, arXiv:2207.05632
  * Salim et al. (2007) Salim, S., Rich, R. M., Charlot, S., et al. 2007, ApJS, 173, 267
  * Schlafly & Finkbeiner (2011) Schlafly, E. F., & Finkbeiner, D. P. 2011, ApJ, 737, 103
  * Seabold & Perktold (2010) Seabold, S., & Perktold, J. 2010, Proceedings of the 9th Python in Science Conference, ed. van der Walt, S., & Millman, J. (Austin, Texas: Enthought, Inc.), 92
  * Sellgren (2001) Sellgren, K. 2001, Spectrochimica Acta Part A: Molecular Spectroscopy, 57, 627
  * Shangguan et al. (2018) Shangguan, J., Ho, L. C., & Xie, Y. 2018, ApJ, 854, 158
  * Shipley et al. (2016) Shipley, H. V., Papovich, C., Rieke, G. H., et al. 2016, ApJ, 818, 60
  * SINGS Team (2020) SINGS Team, 2020, Spitzer Infrared Nearby Galaxy Survey, IPAC, doi:10.26131/IRSA424
  * Skrutskie et al. (2006) Skrutskie, M. F., Cutri, R. M., Stiening, R., et al. 2006, AJ, 131, 1163
  * Smith et al. (2007a) Smith, J. D. T., Armus, L., Dale, D. A., et al. 2007a, PASP, 119, 1133
  * Smith et al. (2007b) Smith, J. D. T., Draine, B. T., Dale, D. A., et al. 2007b, ApJ, 656, 770
  * Tielens (2005) Tielens, A. G. G. M. 2005, The Physics and Chemistry of the Interstellar Medium (Cambridge: Cambridge Univ. Press)
  * Tielens (2008) Tielens, A. G. G. M. 2008, ARA&A, 46, 289
  * Treyer et al. (2010) Treyer, M., Schiminovich, D., Johnson, B. D., et al. 2010, ApJ, 719, 1191
  * Werner et al. (2004) Werner, M. W., Roellig, T. L., Low, F. J., et al. 2004, ApJS, 154, 1
  * Wright et al. (2010) Wright, E. L., Eisenhardt, P. R. M., Mainzer, A. K., et al. 2010, AJ, 140, 1868
  * Wu et al. (2005) Wu, H., Cao, C., Hao, C.-N., et al. 2005, ApJ, 632, L79
  * Wu et al. (2007) Wu, H., Zhu, Y.-N., Cao, C., et al. 2007, ApJ, 668, 87
  * Xie & Ho (2019) Xie, Y., & Ho, L. C. 2019, ApJ, 884, 136
  * Xie & Ho (2022) Xie, Y., & Ho, L. C. 2022, ApJ, 925, 218
  * Xie et al. (2018) Xie, Y., Ho, L. C., Li, A., et al. 2018, ApJ, 860, 154
  * Zhang & Ho (2022) Zhang, L., & Ho, L. C. 2022, ApJ, in press
  * Zhang et al. (2022) Zhang, L., Ho, L. C., & Li, A. 2022, ApJ, 939, 22
  * Zhang et al. (2021) Zhang, L., Ho, L. C., & Xie, Y. 2021, AJ, 161, 29


# UnifySpeech: A Unified Framework for Zero-shot Text-to-Speech and Voice
Conversion

Haogeng Liu11footnotemark: 1, Tao Wang11footnotemark: 1, Ruibo
Fu22footnotemark: 2, Jiangyan Yi22footnotemark: 2, Zhengqi Wen22footnotemark:
2
Chinese Academy of Science
Beijing, China
&Jianhua Tao111Demo for this paper can be found atxxx
Department of Automation, Tsinghua University
School of Artificial Intelligence, University of Chinese Academy of Sciences,
China
Beijing, China

###### Abstract

Text-to-speech (TTS) and voice conversion (VC) are two different tasks both
aiming at generating high quality speaking voice according to different input
modality. Due to their similarity, this paper proposes UnifySpeech, which
brings TTS and VC into a unified framework for the first time. The model is
based on the assumption that speech can be decoupled into three independent
components: content information, speaker information, prosody information.
Both TTS and VC can be regarded as mining these three parts of information
from the input and completing the reconstruction of speech. For TTS, the
speech content information is derived from the text, while in VC it’s derived
from the source speech, so all the remaining units are shared except for the
speech content extraction module in the two tasks. We applied vector
quantization and domain constrain to bridge the gap between the content
domains of TTS and VC. Objective and subjective evaluation shows that by
combining the two task, TTS obtains better speaker modeling ability while VC
gets hold of impressive speech content decoupling capability.

Index Terms: decoupling, zero-shot learning, text-to-speech, voice conversion,
vector quantization

##  1 Introduction

Cloning the voice of the target speaker is an attractive technology, which can
be applied to various scenes (Sisman et al., 2020), such as entertainment
creation, personalized mobile assistants, security field, etc. The most ideal
voice cloning operation is to give only one relatively short speech of the
unseen target speaker as a reference and then any speech of the target speaker
can be synthesized, which is called zero-shot voice clone. In the speech
research community, text-to-speech (TTS) and voice conversion (VC) are two
mainstream ways to realize voice clone (Sisman et al., 2020). Therefore, a
variety of techniques for zero-shot TTS and VC have been proposed recently
(Gorodetskii & Ozhiganov, 2022; Tang et al., 2021).

However, although TTS and VC techniques are two important ways of voice clone
with same output form, the research of TTS and VC is more or less independent.
There isn’t much interaction between them. But they are both speech synthesis
task. In terms of speech generation, we categorize the information of the
target speaker’s speech into three kinds of information: (1) speech content,
the characters of phonemes or phonetic posteriorgram (PPG) in voice
conversion, represents the content of the speech. (2) speaker information,
which represents the characteristics of speakers, is related to the speaker’s
articulation organ. (3) prosody information, which covers the intonation,
stress, and rhythm of speech. According to FastSpeech2 (Ren et al., 2020),
pitch, energy and duration information can reflected them certainly. As TTS
extracts speech content directly from phonemes, it is easier to obtain content
information irrelevant to speaker than VC. As VC encounters more speakers,
it’s possible to obtain more robust speaker modeling ability. So, by
integrating TTS and VC into a unified framework and combining their training
data, it can help the model learn these three kinds of information better.

Unfortunately, investigating TTS and VC in the same framework is challenging,
as the speech content are extracted from different modality. Specifically, the
speech content in TTS is obtained through phoneme information while the
phonemes and speech in TTS are unequal sequences, needing attention mechanism
to align them. However, the attention mechanism is often affected by the
speaker’s information, so it is impossible to learn the speech content
representation completely irrelevant to the speaker. While in VC, the source
speech and target speech are aligned in speech content, so the speech content
can be extracted directly from the source speech, which is different from the
TTS. In contrast to speech content information, speaker information and
prosody information can be modeled using the same network in TTS and VC.
Therefore, the difficulty here is to keep speech content for TTS and VC
consistent. With the development of speech synthesis, the recently proposed
Adaspeech2 (Yan et al., 2021) can combine text information with speech
information, which can effectively decouple the speech content from the input.
The unified framework becomes possible.

Overall, the main contributions of this paper are:

  * •

We propose UnifySpeech, a unified framework for TTS and VC. VC enables
unlabeled data to join training process, making TTS encounters more speakers.
TTS enhance the ability for voice content decoupling in VC. Thus, both
pipeline benefits from the other one.

  * •

We apply vector quantization and domain constrain to bridge the gap between
the content domains of TTS and VC. Ablation experiment shows this method’s
effectiveness.

  * •

We perform extensive experiments: zero-shot TTS, zero-shot VC. Results proves
that jointly trained TTS outperformes StyleSpeech (Min et al., 2021)and
jointly trained VC gains better speech decoupling ability.

Demos for this paper are available at
https://liuhaogeng.github.io/UnifySpeech/.

##  2 Background

In this section, we will briefly review the background of this work, including
neural TTS and VC models, and the zero-shot learning for TTS and VC tasks.

###  2.1 Text-to-speech task

TTS task is to model the mapping between text and speech, which is a modeling
problem between two unequal length sequences. According to the alignment
mechanism (Battenberg et al., 2020) between text and speech in the model, the
end-to-end TTS model can be divided into two categories: 1) Using a neural
network to learn the alignment information between text and speech, such as
local sensitive attention in Tacotron (Wang et al., 2017b). Various
improvements to the attention mechanism have been proposed. In addition,
inspired by CTC (Kim et al., 2017) in ASR, glow-TTS (Kim et al., 2020), VITS
(Kim et al., 2021) can automatically learn the alignment information. 2) By
introducing the duration information (Ren et al., 2019) of phonemes as prior
knowledge, the text is upsampled to achieve alignment (McAuliffe et al.,
2017). Since the upsampled information based on text is independent of the
speaker, the speech content can be well separated from the speech. Therefore,
we introduce the duration information to build the TTS model in this paper.

###  2.2 Voice conversion task

Voice conversion can be seen as two steps (Sisman et al., 2020). Firstly,
extract the speaker-independent speech content information from the source
speech, and then embed the target speaker information to the speech content to
reconstruct the speech of the target speaker. According to the way of
extracting speech content, the VC model can be divided into two categories:
(1) text-based approach. (2) Information bottleneck approach. The first
approach requires an additional pre-trained ASR model. Since the ASR is
trained in a supervised manner, it demands a lot of paired text and speech.
Additionally, pipeline modeling is easy to accumulate errors and affects the
performance of the system. Therefore, a lot of research work is focused on the
latter. By adding some restrictions to the information bottleneck, different
kinds of information can be decoupled. However, if the information bottleneck
can not be decoupled well, the performance of the model will be significantly
reduced.

###  2.3 Zero shot learning for TTS and VC

The research of zero-shot learning based on TTS and VC focused on how to
extract effective speaker information and then embed it into TTS or VC model
for joint training or segmented training. Typical speaker features include
i-vector (Wang et al., 2017a), d-vector (Variani et al., 2014), x-vector
(Snyder et al., 2018) and so on. In addition, the modules that extract
speaker-style information through the specially designed network structure,
such as GST (Wang et al., 2018), VAE (Van Den Oord et al., 2017), can also
achieve good results.

##  3 UnifySpeech

In this section, we introduce the details of UnifySpeech for zero-shot TTS and
VC tasks. We first give the key idea of UnifySpeech: speech factorization, and
then introduce the formulation of UnifySpeech. Finally, we will describe the
model structure of UnifySpeech.

Figure 1: Structure of UnifySpeech.

###  3.1 Speech representation disentanglement

The core of the controllable and migratable speech generation task is to
decouple the components of the generated speech first, and then control and
transfer each component. Although some end-to-end models can directly model
the relationship between text and speech (TTS) or speech-to-speech task (VC),
due to the mutual coupling of various components of the end-to-end model, it
brings great difficulties to the transfer learning of the model. Therefore, we
first decouple the speech generation task into three independent components
and then input them into the decoder to synthesize speech. This idea is also
the main architecture of UnifySpeech. The three components and their sources
will be described in detail below.

Speech Content: To generate intelligible speech signals, it is important to
model accurate speech content information. Speech content is linguistic
information, which is irrelevant to the speaker. Due to the different types of
input signals of TTS and VC, the ways of extracting speech content are
different. For TTS, the source of speech content is text. Firstly, to learn
the context information of the text, a text encoder is used to encode the text
to obtain the context representation. The context representation is up-sampled
according to the phoneme’s duration information to obtain the speech content
information. For VC, since the source speech is aligned with the target
speech, we directly use a content encoder to extract speech content
information from the source speech.

Speaker: The speaker information includes the speaker’s characteristics, such
as the timbre, volume, etc. We can extract the speaker information from the
given speech of the target speaker. This is common for TTS and VC tasks, so
the speaker extraction network can be shared in the two tasks. Since the
speaker extraction network can directly extract information from speech
without text, so a large number of data without text annotation in the VC task
can be used for training, which can help to improve the transfer learning
ability of the model.

Prosody: The prosody information represents how the speaker says the content
information. It is independent of the speaker information and related to the
way of expression. Since the pitch information can reflect the rhythm of
speech, therefore, the pitch information is used to extract prosody
information. The prosody information, like speaker information, can be shared
by both TTS and VC. In addition, in the training process, we can obtain pitch
information from the ground truth speech, but there is no ground truth in the
inference stage. Therefore, a pitch prediction module (Ren et al., 2020) is
proposed in the training stage, which takes the speech content information and
speaker information as the input to predict the pitch information.

###  3.2 Speech content with vector quantization

Since there are different ways to obtain the speech content in TTS and VC, it
is very easy to deviate between the two domains. If this deviation occurs, it
will cause devastating damage to some downstream shared modules (such as pitch
predictor and decoder). Therefore, to ensure that the consistency of the two
speech content domains as close as possible, we reconstruct them from two
aspects:

  * •

First, we use the shared codebook to quantify the continuous feature space.

  * •

Second, we use the labeled data to narrow two discrete feature spaces.

The detailed process is described below. Suppose that the vector obtained by
the text encoder in TTS pipeline is
$C_{p}=(C_{p}^{1},C_{p}^{2},\cdots,C_{p}^{T})$ with length $T$, the vector
obtained by the content encoder in VC pipeline is
$C_{s}=(C_{s}^{1},C_{s}^{2},\cdots,C_{s}^{T^{{}^{\prime}}})$ with length
$T^{{}^{\prime}}$. It should be noted that we add a length regulator module in
the text encoder to solve the problem of length mismatch between the text and
speech sequence, which is introduced in FastSpeech (Ren et al., 2019).
Therefore, if text and speech are paired, $T^{{}^{\prime}}=T$. The vector
$C_{p}$ and $C_{s}$ is a sequence of continuous vector in Eq. Due to the large
representation range of continuous features, $C_{p}$ and $C_{s}$ are difficult
to match. We borrow the discretization method for latent variables from Vector
Quantized Variational AutoEncoder (Van Den Oord et al., 2017). Specifically,
for each time step $t$, the continuous latent representations $C_{p}^{t}$ in
$C_{p}$ can be mapped into $\overline{C}_{p}^{t}$ by finding the nearest pre-
defined discretized embedding in the dictionary as:

$\displaystyle\overline{C}_{p}^{t}=e_{k},\quad
k=argmin_{j}\left\|C_{p}^{t}-e_{j}\right\|_{2}$ (1)

where $e_{j}$ is the j-th embedding in the codebook dictionary, and $j\in
1,2,\cdots,V$. Since selecting the entry with the minimum distance will cause
the operation to be non-differentiable, the straight-through gradient
estimator can be used to approximate the gradient, which can be expressed as:

$\displaystyle\bar{h}_{t}=h_{t}+e_{v}-\operatorname{sg}\left(h_{t}\right),\quad
v=\underset{k}{\arg\min}\left\|h_{t}-e_{k}\right\|_{2}$ (2)

where sg(·) is the stop-gradient (Van Den Oord et al., 2017) operation that
treats its input as constant during back-propagation.

After vector quantization, the quantized sequence
$\overline{C_{p}}=(\overline{C}_{p}^{1},\overline{C}_{p}^{2},\cdots,\overline{C}_{p}^{T})$
and
$\overline{C_{s}}=(\overline{C}_{s}^{1},\overline{C}_{s}^{2},\cdots,\overline{C}_{s}^{T})$
can be obtained. It should be noted that when $C_{p}$ and $C_{s}$ are
quantized into $\overline{C_{p}}$ and $\overline{C_{s}}$, they share the same
codebook $e$. The advantage of this is that since the speech content features
$\overline{C_{p}}$ in the TTS pipeline are independent of the speaker, sharing
the same codebook can help learn the speech content features
$\overline{C_{s}}$ independent of the speaker in the VC pipeline, which is
essential for the VC task.

Although both pipeline use the same codebook for coding, there is no guarantee
that there is no deviation between the two fields after coding. Therefore, to
further eliminate the deviation between the two domains, we use the labeled
speech in the TTS pipeline to supervise the training of the two domains.
Specifically, for paired text and speech data, we constrain the feature
distance between the quantized sequence $\overline{C_{p}}$ and
$\overline{C_{s}}$:

$\displaystyle\mathcal{L}_{\text{pair}}=\left\|\overline{C_{p}}-\overline{C_{s}}\right\|_{2}^{2}$
(3)

In this way, we can efficiently minimize the domain discrepancy by using
limited labeled data.

Figure 2: Structure of vector quantized operation.

###  3.3 UnifySpeech pipeline

An overview of our proposed UnifySpeech architecture is illustrated in Fig. 2.
It consists of a sequence-to-sequence TTS, and a sequence-to-sequence VC. The
key idea is to share most of the module parameters (speaker encoder, prosody
encoder, decoder and pitch predictor) and map the speech content in TTS and VC
to the same space. As mentioned above, the UnifySpeech allows us to train the
model on the concatenation of both the labeled and unlabeled data. For
supervised training with labeled data, both models can be trained
independently by minimizing the loss between their predicted speech and the
ground truth. For unsupervised training with unlabeled data, the VC pipeline
can be trained, and the parameters are shared with TTS.

To further clarify the training process, we unrolled the framework as follows.

####  3.3.1 TTS pipeline

Denote the text and speech sequence pair $(x,y,F0)\in D$, where $D$ is the
paired text and speech corpus. Each element in the text sequence $x$
represents a phoneme or character, while each element in the speech sequence
$y$ represents a frame of speech. $F0$ is the pitch information of y. The
representation obtained after three encoders are speech content $C$, speaker
$S$ and prosody $P$.

Then, the three parts are added and input into a decoder to obtain the
predicted speech $y^{\prime}$. In addition, to obtain the pitch information in
the interference stage, we use the content information and speaker information
to predict the F0. These processes can be expressed as:

$\displaystyle y^{\prime}$ $\displaystyle=\text{decoder}(C+S+P)$ (4)
$\displaystyle F0^{\prime}$ $\displaystyle=\text{pitch\\_predictor}(C+S)$ (5)

Therefore, the reconstruction loss in TTS process includes two parts:

$\displaystyle\mathcal{L}^{VC}_{rec}=MSE(y,y^{\prime})+MSE(F0,F0^{\prime})$
(6)

In addition, we use the content encoder in the VC pipeline to extract the
content representation $C_{s}$ for the training speech in TTS, and close the
distance between the two domains by calculating the distance loss of $C_{s}$
and $C_{p}$, which is explained in Sec. 3.2.

$\displaystyle\mathcal{L}_{\text{pair}}=\left\|\overline{C_{p}}-\overline{C_{s}}\right\|_{2}^{2}$
(7)

The loss of TTS pipeline can be expressed as:

$\displaystyle\mathcal{L}^{TTS}=\mathcal{L}^{TTS}_{rec}+\mathcal{L}_{\text{pair}}$
(8)

where $MSE$ denotes the mean squared errors.

####  3.3.2 VC pipeline

Denote all the unlabeled or labeled speech $(y,F0)\in Y$. We first extract
three information from $(y,F0)$, which are speech content $C_{s}$, speaker
$S_{s}$ and prosody $P_{s}$.

Then, similar to the TTS pipeline, the shared decoder and pitch predictor
module is used to predict the speech signal $y^{\prime}$ and $F0^{\prime}$,
which is similar to the Eq.1 and Eq.2.

The loss of VC pipeline only includes reconstruction loss, which can be
expressed as:

$\displaystyle\mathcal{L}^{VC}=MSE(Y,Y^{\prime})+MSE(F0,F0^{\prime})$ (9)

where $MSE$ denotes the mean squared errors.

####  3.3.3 Training process

With such a unified framework, TTS and VC can learn from each other through
joint training. The details of the algorithm can be found below.

Procedure 1 UnifySpeech training algorithm

1: Input: Paired speech and text dataset $(x,y)$, speech only dataset
$y^{{}^{\prime}}$

2: repeat

3: # A. TTS pipeline with speech-text data pairs

  1. 1.

Sample paired speech and text $(x,y)$

  2. 2.

Extract speech content information from $x$ for domain loss

  3. 3.

Generate the predict speech y, pitch F0 and speech content from text

  4. 4.

Calculate the loss for TTS $\mathcal{L}^{TTS}_{rec}$

4: # B. VC pipeline with speech-only data

  1. 1.

Sample paired speech and text $(x,y)$

  2. 2.

Extract speech content information from $y$ for domain loss

  3. 3.

Calculate the domain loss for the two pipeline $\mathcal{L}_{pair}$

  4. 4.

Sample speech $y^{{}^{\prime}}$ in speech only dataset

  5. 5.

Generate the predict speech y, pitch F0 and speech content from speech
$y^{{}^{\prime}}$

  6. 6.

Calculate the loss for VC $\mathcal{L}^{VC}$

5: # C. Loss combination:

  1. 1.

Combine all loss ($\mathcal{L}^{TTS}_{rec}$, $\mathcal{L}_{pair}$,
$\mathcal{L}^{VC}$) into a single loss variable

  2. 2.

Calculate TTS and VC parameters gradient

  3. 3.

Update TTS and VC parameters with gradient descent optimization

6: until convergence

##  4 Experiments and results analysis

In this section, we conduct experiments to evaluate our proposed methods. The
experiments are carried out from two aspects: zero-shot TTS, zero-shot VC.

###  4.1 Datasets

Two datasets are used to simulate labeled data and unlabeled data,
respectively. VCTK dataset, an English language dataset containing 44 hours of
speech and 109 speakers is used as labeled data. Each speaker has
approximately 400 sentences. LibriTTS (Zen et al., 2019) are used as unlabeled
data, which consists of 585 hours of speech data from 2484 speakers. We only
use the speech data in LibriTTS and discard the text for unsupervised
training. In this way, it can simulate the scene where a large number of
speech that we can obtain are unlabeled. We use a 16-bit, 22050 Hz sampling
rate for all experiments. The 80-dim Mel spectrogram is extracted with Hann
windowing, frame shift of 12.5-ms, frame length of 50-ms, and 1024-point
Fourier transform. In this experiment, we use hifigan (Kong et al., 2020) as
vocoder.

###  4.2 Model Details

Figure 3: Structure of each module in UnifySpeech. FFT (Ren et al., 2019)
means feed-forward Transformer.

The detail of each module in our proposed method is shown in the Fig. 3.
Specifically, to make the sequence of speech content extracted from the text
encoder and content encoder equal, a length regulator is added to the text
encoder, which is inspired by the FastSpeech (Ren et al., 2019). The structure
of the duration predictor is the same as that in FastSpeech. The structure of
the decoder and content encoder is similar, but the dimensions of input and
output are opposite. For the prosody encoder, we first quantize F0 of each
frame to 32 possible values and encode them into a learnable embedding vector
according to the value. And we change the output of the pitch predictor into a
distribution. By this way, pitch prediction becomes a classification task,
reducing the difficulty of frame-level pitch prediction. We use the speaker
encoder in StyleSpeech and fuse features with Style Adaptive Layer Norm(SALN)
(Min et al., 2021) method to make the comparision fairly.

The number of feed-forward Transformer (Vaswani et al., 2017) (FFT) blocks in
the text encoder is 4 and it is 6 in the decoder module. In each FFT block,
the dimension of hidden states is 256. The kernel sizes of all the
1D-convolution are set to 3. The dropout rate is set to 0.5. The dimension of
the last linear layer in the decoder is 256. The dimension of last linear
layer in encoders (text encoder, pitch encoder, content encoder) is 256. An
Adam optimizer (Kingma & Ba, 2014) is used to update the parameters. The
initial learning rate is 0.001 and the learning rate decreased exponentially.

###  4.3 Zero-shot TTS

We first carried out zero-shot TTS task. We choose four speakers from VCTK
that are not used during training process as target speakers. For each
speaker, we randomly select about 20 sentences to be our target. Then we
caculate F0 RMSE (root of mean square errors of F0), MCD (Kubichek, 1993)
(Mel-cepstrum distortion), V/UV (the error rate of voicing/unvoicing flags)
and F0 CORR (correlation factor of F0) between synthesized speech and ground
truth speech as the objective metrics.

Table 1: Objective evaluation results for zero-shot TTS. Model | F0 RMSE (Hz) | MCD (db) | V/UV | F0 CORR
---|---|---|---|---
UnifySpeech | $17.84$ | $2.51$ | $16.9\%$ | $0.93$
StyleSpeech | $19.02$ | $2.63$ | $18.06\%$ | $0.92$

Subjective evaluation was also conducted to compare the speech’s quality and
similarity. We choose mean opinion scores(MOS) for naturalness and similarity
mean opinion scores(SMOS) for similarity. Both metrics are rated in 1-to-5
scale and reported with the 95% confidence intervals (CI).

Table 2: Mean opinion score (MOS) of the models. With VC means the model is jointly trained. Model | MOS | SMOS
---|---|---
GT | $4.32\pm 0.15$ | $-$
GT mel + Vocoder | $4.09\pm 0.15$ | $-$
StyleSpeech | $3.52\pm 0.13$ | $3.82\pm 0.13$
UnifySpeech-TTS (with VC) | $3.76\pm 0.12$ | $3.95\pm 0.13$

To better show our method’s effectiveness, we demonstrate the t-SNE projection
of the speaker embedding vectors from speakers in both VCTK and LibriTTS. (Van
der Maaten & Hinton, 2008) Fig. 4 shows the speaker visualization. For the
seen speakers (x) and unseen speakers (o), the corresponding speaker embedding
form a cluster and distinct from others. The boundary between different
speakers is clear. This shows that the speaker encoder performs well.

Figure 4: Speaker visualization of generated speeches,where the circle and
triangle indicate unseen speaker,while x and square indicate seen speaker.

###  4.4 Zero-shot VC

We carried out zero-shot VC task, using unseen speakers voice to be the
reference speech. As we are lack of parallel corpus, we only conduct
subjective evaluation. But VC and TTS shares the same speaker encoder, Fig. 4
in zero-shot TTS can also be a reference.
Just as in zero-shot TTS task, we choose mean opinion scores(MOS) for
naturalness and similarity mean opinion scores(SMOS) for similarity. Both
metrics are rated in 1-to-5 scale and reported with the 95% confidence
intervals (CI).

Table 3: Mean opinion score (MOS) of the VC models. With TTS means the model is jointly trained. Model | MOS | SMOS
---|---|---
GT | $4.32\pm 0.15$ | $-$
GT mel + Vocoder | $4.09\pm 0.15$ | $-$
UnifySpeech-VC(without TTS) | $3.63\pm 0.13$ | $1.31\pm 0.06$
UnifySpeech-VC(with TTS) | $3.58\pm 0.12$ | $3.31\pm 0.13$

It can be found that when VC pipeline is trained alone, its performance is
poor. In other words, it doesn’t have the ability to discriminate. But jointly
training with TTS improves its speech decoupling ability, indirectly improving
the speaker modeling ability, which we will analysis later.

###  4.5 Ablation experiment

To figure out whether jointly training is effective, we separately train the
TTS and VC parts, carrying out objective evaluation on them. We also remove
the VQ parts and test the model. For TTS, we caculate F0 RMSE (root of mean
square errors of F0), MCD (Kubichek, 1993) (Mel-cepstrum distortion), V/UV
(the error rate of voicing/unvoicing flags) and F0 CORR (correlation factor of
F0) between synthesized speech and ground truth speech as the objective
metrics.

Table 4: Objective evaluation results for zero-shot TTS. Model | F0 RMSE (Hz) | MCD (db) | V/UV | F0 CORR
---|---|---|---|---
UnifySpeech-TTS(without VC) | 19.31 | 2.55 | 16.9% | 0.91
UnifySpeech-TTS-novq(with VC) | $19.41$ | $2.58$ | $17.2\%$ | $0.92$
UnifySpeech-TTS(with VC) | $17.84$ | $2.51$ | $16.9\%$ | $0.93$

For VC, we separately calculate the Average Cosine Similarity (ACS) (Lei et
al., 2022) for the embedding from same (S-ACS) and different speakers (D-ACS).
And then their ratio is used as an evaluation metric.

Table 5: Ratio of ACS from same and different speaker. Unseen or seen means whether the speaker is from test set. With or without TTS means whether the VC pipeline is trained with TTS pipeline. Model |  $\frac{S-ACS}{D-ACS}$(unseen) |  $\frac{S-ACS}{D-ACS}$(seen)
---|---|---
UnifySpeech-VC (with TTS) | $2.8$ | $6.0$
UnifySpeech-VC (without TTS) | 1.0 | 1.0

To validate whether VQ can bridge the gap between the speech content parts in
TTS and VC, we caculate the L2 distance between the phoneme representation
from TTS and VC. As in VC there are many frames represent same phoneme, so we
choose their clustering center as the corresponding phoneme representation. We
randomly select 4 sentences from validation set to carry out the experiment.

Table 6: L2 distance between the phoneme representation in TTS and VC. $S_{1}$ means $sentence_{1}$. Model | S1 | S2 | S3 | S4 | average
---|---|---|---|---|---
UnifySpeech-VC(without VQ) | $0.464$ | $0.476$ | $0.474$ | $0.553$ | $0.492$
UnifySpeech-VC(with VQ) | $0.152$ | $0.205$ | $0.208$ | $0.205$ | $0.193$

###  4.6 Result analysis

Above results show that jointly training actually improves TTS’s speaker
modeling ability and VC’s speech decoupling ability. And VQ enhances the
consistency of representation of the same content from phonemes and speeches,
ensuring the model’s correctly working.

For TTS, sharing modules with VC enables unlabeled data to participate in its
training process. Along with the richer speaker style pattern, the speaker
style modeling capability has been enhanced.

For VC, when trained alone, it doesn’t have the speaker modeling ability. As
the self-supervised training process aims at reducing the reconstruction loss
of source audio, the reference audio of the target speaker isn’t crucial in
the process. Only the content encoder and the decoder are enough for the
process, that’s possibly why the speaker embeddings are all similar, though
they are from different speakers. When jointly trained, the text loss plays a
role of regularization factor, resulting that the content encoder just
extracting the content information from the source speech (speaker information
is reserved and others are discarded). This makes the reference speech with
rich speaker information become indispensable for the reconstruction process.
Thus the model’s speaker modeling ability has been improved.

##  5 Conclusions

In this paper, we propose UnifySpeech, a unified framework for TTS and VC.
Both task benefits from the other one. Due to training with large amounts of
unlabeled data, their few-shot modeling ability makes progress as well as the
synthesized speech’s quality. In the future, further improving the synthesized
speech’s quality and making the generated speech’s style more similar to
target speaker will be our endeavor.

## References

  * Battenberg et al. (2020) Eric Battenberg, RJ Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby.  Location-relative attention mechanisms for robust long-form speech synthesis.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6194–6198. IEEE, 2020.
  * Gorodetskii & Ozhiganov (2022) Artem Gorodetskii and Ivan Ozhiganov.  Zero-shot long-form voice cloning with dynamic convolution attention.  _arXiv preprint arXiv:2201.10375_ , 2022.
  * Kim et al. (2020) Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon.  Glow-tts: A generative flow for text-to-speech via monotonic alignment search.  _Advances in Neural Information Processing Systems_ , 33:8067–8077, 2020.
  * Kim et al. (2021) Jaehyeon Kim, Jungil Kong, and Juhee Son.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.  In _International Conference on Machine Learning_ , pp. 5530–5540. PMLR, 2021.
  * Kim et al. (2017) Suyoun Kim, Takaaki Hori, and Shinji Watanabe.  Joint ctc-attention based end-to-end speech recognition using multi-task learning.  In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pp. 4835–4839. IEEE, 2017.
  * Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kong et al. (2020) Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ , 33:17022–17033, 2020.
  * Kubichek (1993) Robert Kubichek.  Mel-cepstral distance measure for objective speech quality assessment.  In _Proceedings of IEEE pacific rim conference on communications computers and signal processing_ , volume 1, pp. 125–128. IEEE, 1993.
  * Lei et al. (2022) Yi Lei, Shan Yang, Jian Cong, Lei Xie, and Dan Su.  Glow-wavegan 2: High-quality zero-shot text-to-speech synthesis and any-to-any voice conversion.  _arXiv preprint arXiv:2207.01832_ , 2022.
  * McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger.  Montreal forced aligner: Trainable text-speech alignment using kaldi.  In _Interspeech_ , volume 2017, pp. 498–502, 2017.
  * Min et al. (2021) Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang.  Meta-stylespeech: Multi-speaker adaptive text-to-speech generation.  In _International Conference on Machine Learning_ , pp. 7748–7759. PMLR, 2021.
  * Ren et al. (2019) Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech: Fast, robust and controllable text to speech.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Ren et al. (2020) Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  _arXiv preprint arXiv:2006.04558_ , 2020.
  * Sisman et al. (2020) Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li.  An overview of voice conversion and its challenges: From statistical modeling to deep learning.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:132–157, 2020.
  * Snyder et al. (2018) David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur.  X-vectors: Robust dnn embeddings for speaker recognition.  In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pp. 5329–5333. IEEE, 2018.
  * Tang et al. (2021) Chuanxin Tang, Chong Luo, Zhiyuan Zhao, Dacheng Yin, Yucheng Zhao, and Wenjun Zeng.  Zero-shot text-to-speech for text-based insertion in audio narration.  _arXiv preprint arXiv:2109.05426_ , 2021.
  * Van Den Oord et al. (2017) Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Van der Maaten & Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Variani et al. (2014) Ehsan Variani, Xin Lei, Erik McDermott, Ignacio Lopez Moreno, and Javier Gonzalez-Dominguez.  Deep neural networks for small footprint text-dependent speaker verification.  In _2014 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pp. 4052–4056. IEEE, 2014.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2017a) Shuai Wang, Yanmin Qian, and Kai Yu.  What does the speaker embedding encode?  In _Interspeech_ , pp. 1497–1501, 2017a.
  * Wang et al. (2017b) Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al.  Tacotron: Towards end-to-end speech synthesis.  _arXiv preprint arXiv:1703.10135_ , 2017b.
  * Wang et al. (2018) Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous.  Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis.  In _International Conference on Machine Learning_ , pp. 5180–5189. PMLR, 2018.
  * Yan et al. (2021) Yuzi Yan, Xu Tan, Bohan Li, Tao Qin, Sheng Zhao, Yuan Shen, and Tie-Yan Liu.  Adaspeech 2: Adaptive text to speech with untranscribed data.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6613–6617. IEEE, 2021.
  * Zen et al. (2019) Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu.  Libritts: A corpus derived from librispeech for text-to-speech.  _arXiv preprint arXiv:1904.02882_ , 2019.