# Enhanced visibility through microbubble-induced photoacoustic fluctuation
imaging

Marco A. Inzunza-Ibarra1 J. Angel Navarro-Becerra1 Venkatalakshmi Narumanchi2
Nick Bottenus1,3 Todd W. Murray1,3 Mark A. Borden1,3

(1Department of Mechanical Engineering, University of Colorado Boulder, 80309
2Department of Electrical Engineering, University of Colorado Boulder, 80309
3Biomedical Engineering Program, University of Colorado Boulder, 80309
)

###### Abstract

A photoacoustic contrast mechanism is presented based on the photoacoustic
fluctuations induced by microbubbles flowing inside a micro vessel filled with
a continuous absorber. It is demonstrated that the standard deviation of a
homogeneous absorber mixed with microbubbles increases non-linearly as the
microbubble concentration and microbubble size is increased. This effect is
then utilized to perform photoacoustic fluctuation imaging with increased
visibility and contrast of a blood flow phantom.

##  1 Introduction

Photoacoustic (PA) imaging is based on the PA effect where ultrasound signals
are generated by the transient absorption of light and subsequent thermo-
elastic expansion [1]. The biomedical applications using photoacoustics have
grown significantly in recent years due to intrinsic optical absorption
sensitivity at the molecular level [2, 3, 4]. Some of these applications have
significant and widely translatable implications, which include PA molecular
imaging [5, 6, 7, 8], cancer detection and diagnostics [9, 10, 11], and
photodynamic [12, 13] and photothermal [14, 15] cancer therapy. One of the key
advantages of PA imaging is the optical wavelength-dependent absorption of
endogenous chromophores found in blood, including oxygenated and deoxygenated
hemoglobin, making it ideal for angiography [16]. Moreover, imaging of the
blood vessels with high spatial resolution is extremely important in the
detection of angiogenesis sites located in the vicinity of tumors [11]. PA
tomography (PAT) systems [17] typically employ an array of ultrasound
transducers placed outside the object, and images are formed by applying
reconstruction algorithms on the detected signals.

In many practical applications of PAT, access to the imaging object is
severely restricted and signals cannot be collected from large detection
angles, giving rise to the “limited-view” problem [18, 19, 20]. Most commonly,
conventional linear array detectors are employed which lack visibility of
structures due to a finite-aperture and coherence artifacts generated from PA
signals from sub-acoustic wavelength absorbers [21]. Deán-Ben and Razansky
[22] provided a useful link between PA images acquired under the limited-view
scenario as well as the coherent behavior of PA signals from sub-acoustic
absorbers. In brief, finite-aperture PA images from objects containing high
absorber densities are devoid of structural information belonging to the
interior of objects due to destructive interference, and only the edges
associated with constructive interference are retained.

In the recent years, PA fluctuation imaging has demonstrated to enhance
visibility and spatial resolution using multiple speckle illumination [23, 24,
25, 26], SOFI inspired statistical approaches [27, 28], and localization [29,
30, 31, 32, 33]. The last two examples demonstrated that the PA fluctuations
elicited from flowing absorbers could be leveraged to provide structural
information about the object. Flow-induced PA fluctuations, however, can be
difficult to detect in blood, where partial waves emitted from densely packed
red blood cells interfere destructively and can be perceived as a homogeneous
signal at the ultrasound transducer. Recent work by Vilov et al. [34] provided
a comprehensive theory on PA fluctuation imaging and experimental results
demonstrating enhanced visibility using blood at physiological concentrations.

Since their initial regulatory approval for echocardiography in the mid
1990’s, microbubbles (MBs) have been used routinely as ultrasound contrast
agents for a variety of clinical applications [35]. Photoacoustics is highly
sensitive to small changes in optical absorption variations, meaning that a
given percent change in the optical absorption coefficient yields the same
percentage change in the PA amplitude [36]. Acting as weak absorbers, randomly
distributed MBs in the excitation volume would hypothetically induce
fluctuations in the absorption profile and PA response. Here, we
experimentally demonstrate increased imaging visibility and contrast under
limited-view conditions using PA fluctuation imaging induced by MBs.

##  2 Methods

Consider the PA response from a homogeneous absorber (ink) and MBs flowing
through a cylindrical vessel as shown in Fig. 1(a). Laser pulse illumination
excites a PA response from the cylinder and the detected signal by the
ultrasound transducer is the sum of the signals generated by the absorbers
surrounding the MBs. Assuming that the optical absorption from the MBs is
negligible, they therefore do not emit partial PA waves to contribute to the
overall signal. Instead, they interfere with the absorption properties of the
object therefore developing a heterogenous absorption profile. Since ink is a
homogeneous absorber by itself, we attribute the heterogeneity in the overall
PA response to the random distribution of transparent MBs.

###  2.1 Single-element transducer experiments

To measure the PA response from a homogeneous sample and MBs, ink and MBs were
flowed through a 200 $\mu$m inner diameter tube using a syringe pump. Fig. S1
depicts the experimental setup. The flow speed is set to an average velocity
of 10 mm/s to ensure that the spatial distributions of MBs in adjacent single-
shots are uncorrelated. A micro-chip diode pumped laser with one nanosecond
laser pulses firing at a pulse repetition frequency (PRF) of 1 kHz and 532 nm
wavelength is used to generate the PA signals. Using a F = 150 mm lens, we set
the laser spot-size to 50 $\mu$m and the laser fluence was set to 20 mJ/cm2. A
single-element focused ultrasound transducer (Olympus V317) with a 20 MHz
center frequency and a receive-only full-width at half-maximum spot-size of
208 $\mu$m was used to detect the PA waves. A microscope glass cover-slip was
used as an acoustic reflector in order to align both the acoustical and
optical focal spots. A 1 GHz amplifier (FEMTO, HSA-X) with 40 dB gain was used
to amplify the radio-frequency (RF) signals detected by the single-element
transducer. The PA signals were acquired using a high-speed digitizer at 2.5
GHz sampling frequency and were post-processed by a low-pass Butterworth
filter with a 35 MHz cutoff frequency.

Figure 1: MB induced PA fluctuation measurements. (a) Experimental concept and
geometry. (b) In color are five PA single-shots for the homogeneous sample
(ink) and the homogeneous sample plus MBs with a polydisperse distribution and
concentration $\rho_{0}$=3×107 MB/mL. In black are the mean signals of the
5000 single-shots. (c) Normalized standard deviation $\sigma_{N}$ as a
function of MB concentration for four different MB size distributions.

Five thousand single-shot PA signals were collected in order to compute the
mean and standard deviation over time. Five PA single-shots along with the
mean response (black curve) for the homogeneous case (ink), as well as ink and
polydisperse MBs of concentration $\rho_{0}$=3x107 MB/mL with diameters
ranging from 1-10 $\mu$m, are shown in Fig. 1(b). The homogeneous case
demonstrates variations in the single-shots that are associated with the
electronic and system noise. A notable increase in the single-shot variations
above the system noise is seen in the ink sample with the polydisperse MB
distribution. The fluctuations cancel in the mean response for both cases as
only the edges of the object, the front and back of the micro vessel, are
observed. We also note high variations in the single-shots around the back
wall of micro vessel. We quantify the fluctuation level in the five thousand
PA single-shots by computing the normalized standard deviation $\sigma_{N}$,
expressed as $\sigma_{N}=\sqrt{\sigma_{m}^{2}-\sigma_{s}^{2}}/A_{pp}$ [37].

The peak-to-peak amplitude $A_{pp}$ is computed from the mean response as a
function of time. Similarly, the measured standard deviation (SD)
$\sigma_{m}$, is taken as the average value from a 50 ns time window centered
at 8.75 $\mu$s in the SD versus time plot. The system SD $\sigma_{s}$, is a
fixed value of the SD versus time before the PA arrival of the micro vessel.
$\sigma_{N}$ was measured for different MB concentrations and size
distributions, as shown in Fig. 1(c). The preparation and characterization of
the MBs are described in more detail in the supplemental material and Fig. S2.
In Fig. 1(c), we demonstrate a nonlinear increase of the $\sigma_{N}$ as a
function of concentration. As ink responds as a homogeneous medium, the
increase in $\sigma_{N}$ can be attributed exclusively to the MBs as they
perturb the absorption profile. Interestingly, a square-root regression was
applied to $\sigma_{N}$ versus MB concentration, shown as the straight lines
in Fig. 1(c), with all the fits having an adjusted R2 value greater than 0.95.

This square-root dependence of $\sigma_{N}$ is inversely related to
fluctuations induced by absorbers. In previous works [21, 22, 34, 37], it has
been reported that the PA fluctuations arising from the interior of micro-
vessels decrease as the absorber concentration increases and follows an
inverse square-root dependence. Fundamentally , this makes sense based on
previous work in that an increase in the density of the microbubbles leads to
a decrease in the density of absorbers.

A significant increase in $\sigma_{N}$ was also observed as we increased the
average MB radius. For example, the fluctuation level $\sigma_{N}$ for the
concentration of 6x107 MB/mL is about 230% higher for R0 = 2.3 $\mu$m compared
to R0 = 1 $\mu$m. This is an important result as higher fluctuation levels are
preferable as we move towards PAT with acoustic resolution, where the
excitation and detection regions are significantly larger than in the
experiment described here. This result motivates the next section of our study
in which we utilized the fluctuations generated by MBs in ink as well as blood
to perform PA fluctuation imaging.

###  2.2 Photoacoustic fluctuation imaging experiment

A PA imaging system was built as shown in Fig. S3. A 20 MHz center frequency
linear array (128 elements, L22-14vx) with a 60 percent bandwidth at -6 dB and
an elevational focus at 8 mm was used to receive the PA RF signals. The probe
was connected to an ultrasound scanner (Verasonics Vantage) where the
128-channel raw frames sampled at frequency $f_{s}$ = 62.5 MHz were sent via
PXIe to a host computer. Laser pulses from an optical parametric oscillator
(OPO) tuned at 688 nm wavelength, pumped by a Q-switched laser (Surelite I-20,
PRF = 20 Hz, 7 ns pulse-width at 532 nm wavelength), were used to illuminate a
micro vessel phantom. A knife-edge measurement was made to determine the 1/e
optical spot-size of 7.5 mm at the micro vessel location, and the fluence was
set to 20 mJ/cm2. The micro vessel phantom was surgical tubing (Scientific
Commodities) made from low-density polyethylene with an inner diameter of 0.58
mm. The linear array was aligned so that the direction of the transducer
elements was perpendicular to the micro vessel as shown in Fig. S3(b).
Finally, a syringe pump was used to control the fluid flow speed, which was
set to an average velocity of 15 mm/s, providing uncorrelated MB spatial
distributions for every laser shot. Additionally, a stir plate was used to mix
the absorber and MB solution before flowing into the vessel. In the previous
section, we characterized the MB induced PA fluctuations $\sigma_{N}$ for
different MB size distributions. Based on those results, we chose the size-
isolated MBs with an average radius of R0 = 2.3 $\mu$m, which resulted in the
largest $\sigma_{N}$. The protocol for MB synthesis and size isolation is
described in the supplemental material. The measured size distribution for R0
= 2.3 $\mu$m for the imaging experiments was similar to that in Fig. S2.

The PA RF frame acquisition was triggered by sampling part of the 688 nm beam
onto a photodiode. The photodiode then triggered a digital delay generator,
which provided the input trigger TTL signal to the ultrasound scanner. In
order to compensate for the jitter between the laser pulse and PA acquisition,
an output sync signal from the ultrasound scanner was recorded on a second
computer along with the laser pulse. The precise difference in time between
the laser pulse and the PA acquisition was used to shift the frames in time.
Each frame consists of 128 time resolved RF signals, and an ensemble of 1,000
frames was acquired in order to generate the mean and SD images. A
conventional delay-and-sum (DAS) beamforming algorithm with an axial and
lateral step-size of 10 $\mu$m was performed on the Hilbert transformed RF
data to produce the reconstructed in-phase and quadrature (IQ) PA images. A
flow diagram of the image reconstruction steps is provided in supplemental
Fig. S4.

Parasitic noise in the IQ frames associated with fluctuations in the laser
energy, for example, can arise as dominating features in the fluctuation
images. Therefore, spatiotemporal filtering using singular value decomposition
(SVD) was a necessary step in our data processing. A detailed description of
the SVD processing is provided in the supplemental material. Briefly, the SVD
of the images can be written as $\boldsymbol{S=U\Sigma V^{T}}$, where the
first singular values in $\boldsymbol{\Sigma}$ correspond to highly coherent
spatial features in the IQ images, such as the object’s edges. We therefore
used the SVD to suppress the highly correlated features by setting the first
singular values in $\boldsymbol{\Sigma}$ equal to zero. We then computed a new
set of images $\boldsymbol{S}$ and calculated the SD, resulting in the final
PA fluctuation images presented here.

###  2.3 Imaging results

The imaging results from the experiment can be seen in Fig. 2. The top row
(a-c) shows the resulting images from ink, which served as a homogeneous
absorber control sample. The bottom row (d-f) depicts the resulting images
from a mixture of ink and size-isolated MBs (similar distribution as shown in
Fig. S2) of mean radius R0 = 2.3 $\mu$m and concentration $\rho_{0}$=6×106
MB/mL. The absolute value of the mean images (Fig. 2(a) and (d)) for both
samples were dominated by the limited-view artifact, in which only the front
and back edges of the object were resolved, and the interior of the object was
hidden. The SD image in both cases (Fig. 2(b) and (e)) demonstrated a notable
increase in the visibility through the interior of the object, but the images
were mostly dominated by a strong fluctuation feature associated with laser
fluctuations at the object’s front wall. This feature was suppressed in the SD
after SVD images in Fig. 2(c) and (f). We note that the first two singular
values in the SVD were set to zero. We expected that the SD images of the
homogeneous sample would not produce an increased noise level, but this was
not the case in our results. We attribute this non-zero SD level of the
homogeneous sample to variations of the laser pulse energy, which were not
entirely compensated for by the SVD. However, as expected from our single-
element transducer experimental results, MBs can be leveraged to induce PA
fluctuations resulting in increased image contrast well above that of the
homogeneous control sample. Additional clutter can be seen after the object in
Fig. 2(f) which is associated with single or multiple scattering off the
microbubbles of the generated photoacoustic waves.

Figure 2: Reconstructed photoacoustic images for ink and MBs. (a-c) Images of
only ink of the mean, SD and SD after SVD, respectively. (d-f) Images for ink
and MB concentration of $\rho_{0}$=6×106 MB/mL and radius R0 = 2.3 $\mu$m. For
both cases, N = 1,000 frames were acquired to compute the statistics and the
first two singular values were set to zero in the SVD.

PA mean and SD images using blood and MBs were also acquired using the same
experimental conditions. Undiluted, defibrinated bovine blood (Carolina
Biological) with a hematocrit Hct = 0.46 mixed with MBs was prepared at room
temperature. For these experiments, we again used size-isolated MBs with a
mean radius of R0 = 2.3 $\mu$m. The resulting images for blood and a mixture
of blood and MBs at a concentration of $\rho_{0}$=6×106 MB/mL are depicted in
Fig. 3. Again, N = 1,000 RF frames were acquired to compute the mean and SD
(after SVD), and the first two singular values in the SVD were set to zero.
The mean and SD (after SVD) images of blood are shown in Fig. 3(a-b),
respectively. The SD (after SVD) for the blood and MBs mixture is shown in
Fig. 3(c). A notable increase in the visibility of the object’s interior is
seen in the SD (after SVD) image for blood and MBs when compared to blood by
itself. The mean image of blood is dominated by the limited-view artifacts and
the interior of the object is hidden, like the case of blood and MBs (mean not
shown here).

Figure 3: Reconstructed images for blood and MBs. (a-b) Mean and SD images
(after SVD) for blood. (c) SD (after SVD) of blood and MBs $\rho_{0}$=6×106
MB/mL and R0 = 2.3 $\mu$m.

In order to quantitatively demonstrate the fluctuation image contrast
enhancement as a function of MB concentration, we took images using four
different concentrations in ink and blood. The contrast $C$ induced by the MBs
was calculated on the SD (after SVD) images using the equation
$C=20\log_{10}(\mu_{target}/\mu_{noise})$ where $\mu_{target}$ and
$\mu_{noise}$ are the mean pixel intensity inside a region of interest (ROI)
of our target object and of the noise background, respectively. The ROI for
the target was chosen to be a 0.58 mm diameter circle with an origin centered
about the cross-sectional area of our micro-vessel. The mean pixel intensity
for the same ROI was calculated for the noise on the top-left corner of our
images, where PA contributions from the target were negligible. In Fig. 4, we
plot the contrast as a function of MB concentration. The contrast value for
the control samples of ink and blood were subtracted from the contrast with
MBs. The square markers in black represent the contrast of ink, while the
circle markers in red represent contrast of blood. We also plot the SD error
bars for three measurements using 1,000 frames per measurement. The contrast
for both the ink and blood absorbers increased as a function of MB
concentration. Similar to our experimental results in Fig. 1(c), the
fluctuation level followed a square-root dependence with MB concentration. The
solid lines are a square-root regression fit to the contrast parameter with
over 0.95 adjusted R2 values. Remarkably, blood with a MB concentration of
$\rho_{0}$=1×107 MB/mL expresses a difference in contrast of about 8 dB, or
about 250%, compared to blood alone.

Figure 4:  Contrast as a function of MB concentration for ink and blood
samples. Error bars represent the SD of three measurements.

Introducing MBs in blood to produce incoherence in the PA signals can also be
beneficial to improve the accuracy of Doppler based flow velocimetry
techniques, for example [38, 39], since these techniques rely on high
spatiotemporal incoherence for accuracy. However, some of the limitations in
combining MBs with photoacoustics is that MBs will inherently exacerbate light
scattering through the volume as well as scatter the PA waves propagating
towards the detector. As shown in our mean image in Fig. 2(d), the
introduction of MBs slightly reduces the amplitude of the coherent feature
corresponding to the back of the object. This could possibly lead to
deleterious resolution effects when imaging larger objects or objects placed
deep in tissue. Moreover, the clutter artifact in our fluctuation images
caused by acoustic scattering from the MBs can potentially be compensated for
and is the direction of our future work. Nevertheless, we have demonstrated an
interesting result which is that an increase in the photoacoustic image
contrast can be achieved by MB concentrations as low as 1×106 MB/mL in blood,
which is already a standard dose for contrast-enhanced ultrasound imaging in
humans.

##  3 Conclusion

In conclusion, we have demonstrated a novel PA image contrast technique in
vitro by exploiting the signal fluctuations induced by conventional lipid-
coated MBs. The PA fluctuations of different MB concentrations and size
distributions were characterized experimentally. We found that the fluctuation
level $\sigma_{N}$ increases with MB concentration as well as size. Finally,
fluctuation imaging was performed to overcome the limited-view problem, and it
was demonstrated that MBs provided an increase in the contrast and visibility
of a blood-filled micro-vessel phantom.

##  4 Acknowledgements

This research was funded by NIH grant R01CA195051 to MAB. We thank Dr.
Francesco Guidi for his advice on the instrumentation and fruitful
discussions.

Supplemental material for Enhanced visibility through microbubble-induced
photoacoustic fluctuation imaging

##  1 Microbubble synthesis and sample preparation

A phospholipid shell was used to stabilize the microbubble which consisted of
1, 2-dipalmitoyl- sn -glycero -3- phosphocholine (DPPC) and 1, 2- distearoyl-
sn- glycero-3- phosphoethanolamine -N- [methoxy (polyethylene glycol) – 2000]
(DSPE - PEG2000) at a concentration of 2 mg/mL and a molar ratio 9:1 in
phosphate buffer saline (PBS). High density perfluorobutane (PFB, C4F10), was
used as the gas core of the microbubbles. The lipid solution and PFB gas
interface was placed onto a probe tip sonication device at a constant, high
output power for 45 seconds to form the microbubbles. Centrifugation was used
to wash away residual lipid solution and to size isolate the microbubbles into
different size distributions [40]. The average of three measurements from a
Multisizer Coulter Counter was used to characterize the size distribution and
concentration of the microbubble distributions as shown in Fig. S2.
Immediately before each photoacoustic experiment, 3 mL of ink and blood
samples were mixed with the microbubbles at the desired concentration, keeping
the 3 mL of volume constant.

##  2 Singular value decomposition

Singular value decomposition (SVD) has been applied in numerous ultrasound
techniques including clutter rejection in ultrafast Doppler [41, 42] and
localization-based super-resolution imaging [43, 44]. As we proposed a
fluctuation-based imaging approach, SVD was required to separate unwanted
fluctuation sources, such as laser fluctuations, that tend to dominate the
fluctuation image.

SVD processing followed the acquisition and delay-and-sum reconstruction of
the 1,000 RF frames, where the in-phase and quadrature (IQ) frames were
written as a complex matrix $s(n_{z},n_{x},n_{t})$, where $n_{z}$ and $n_{x}$
correspond to the number of points in the $z$ and $x$ directions of the
reconstructed IQ frames, respectively, and $n_{t}$ is the number of frames
collected. We then reshaped the IQ frames into a complex 2D space-time matrix
$\boldsymbol{S}$ with dimensions $(n_{z}\times n_{x},n_{t})$. According to the
SVD, we can write $\boldsymbol{S}$ as:

$\boldsymbol{S=U\Sigma V^{T}}$ (1)

where $\boldsymbol{\Sigma}$ is a diagonal matrix of size $(n_{t}\times n_{t})$
containing singular values ordered from largest to smallest, $\boldsymbol{U}$
and $\boldsymbol{V}$ are orthonormal matrices with respective dimensions
$(n_{z}\times n_{x},n_{t})$ and $(n_{t}\times n_{t})$. The columns of matrices
$\boldsymbol{U}$ and $\boldsymbol{V}$ are the spatial and temporal singular
vectors of $\boldsymbol{\Sigma}$, respectively. Once $\boldsymbol{S}$ is
decomposed into these three matrices, we chose which singular components to
keep in $\boldsymbol{\Sigma}$.

Fig. S1:  Experimental setup for single element transducer measurements for
$\sigma_{N}$ as a function of microbubble concentration and size distribution.
Fig. S2:  Microbubble size distributions measured in a Multisizer Coulter
Counter. Reporting mean and standard deviation of N = 3 measurements. (a)
Number percent of three size isolated and polydisperse microbubble size
distributions with different mean diameters. (b) Volume percent of the same
microbubble size distributions.

Fig. S3:  Experimental set-up for photoacoustic imaging. (a) A flow-phantom
was illuminated by a pulsed-laser (7 ns pulses, $\lambda$=688 nm) and the
photoacoustic signals were detected by a linear-array transducer (128
elements, 20 MHz center frequency, L22-14vx). A syringe pump was used to
control the flow of the MB sample through the phantom. (b) Cross-sectional
view of the phantom and acoustic detection.

Fig. S4:  Photoacoustic fluctuation imaging flow diagram.

## References

  * [1] P. Beard, “Biomedical photoacoustic imaging,” Interface Focus 1(4), 602–631 (2011).
  * [2] C. Kim, C. Favazza, and L. V. Wang, “In vivo photoacoustic tomography of chemicals: high-resolution functional and molecular optical imaging at new depths,” Chemical Reviews 110(5), 2756–2782 (2010).
  * [3] J. Weber, P. C. Beard, and S. E. Bohndiek, “Contrast agents for molecular photoacoustic imaging,” Nature Methods 13(8), 639–650 (2016).
  * [4] V. Gujrati, A. Mishra, and V. Ntziachristos, “Molecular imaging probes for multi-spectral optoacoustic tomography,” Chemical Communications 53(34), 4653–4672 (2017).
  * [5] K. E. Wilson, T. Y. Wang, and J. K. Willmann, “Acoustic and photoacoustic molecular imaging of cancer,” Journal of Nuclear Medicine 54(11), 1851–1854 (2013).
  * [6] S. Zackrisson, S. M. W. Y. v. d. Ven, and S. S. Gambhir, “Light in and sound out: emerging translational strategies for photoacoustic imaging,” Cancer Research 74(4), 979–1004 (2014).
  * [7] Y. Liu, L. Nie, and X. Chen, “Photoacoustic molecular imaging: from multiscale biomedical applications towards early-stage theranostics,” Trends in Biotechnology 34(5), 420–433 (2016).
  * [8] J. Yao and L. V. Wang, “Recent progress in photoacoustic molecular imaging,” Current Opinion in Chemical Biology 45, 104–112 (2018).
  * [9] S. Mallidi, G. P. Luke, and S. Emelianov, “Photoacoustic imaging in cancer detection, diagnosis, and treatment guidance,” Trends in Biotechnology 29(5), 213–221 (2011).
  * [10] E. Herzog, A. Taruttis, N. Beziere, A. A. Lutich, D. Razansky, and V. Ntziachristos, “Optical imaging of cancer heterogeneity with multispectral optoacoustic tomography,” Radiology 263(2), 461–468 (2012).
  * [11] L. Lin, P. Hu, J. Shi, C. M. Appleton, K. Maslov, L. Li, R. Zhang, and L. V. Wang, “Single-breath-hold photoacoustic computed tomography of the breast,” Nature Communications 9(1), 2352 (2018).
  * [12] T. J. Dougherty, C. J. Gomer, B. W. Henderson, G. Jori, D. Kessel, M. Korbelik, J. Moan, and Q. Peng, “Photodynamic therapy,” Journal of the National Cancer Institute 90(12), 889–905 (1998).
  * [13] S. C. Hester, M. Kuriakose, C. D. Nguyen, and S. Mallidi, “Role of ultrasound and photoacoustic imaging in photodynamic therapy for cancer,” Photochemistry and Photobiology 96(2), 260–279 (2020).
  * [14] J. Shah, S. Park, S. R. Aglyamov, T. Larson, L. Ma, K. V. Sokolov, K. P. Johnston, T. E. Milner, and S. Y. Emelianov, “Photoacoustic imaging and temperature measurement for photothermal cancer therapy,” Journal of Biomedical Optics 13(3), 034024 (2008).
  * [15] Y. Liu, P. Bhattarai, Z. Dai, and X. Chen, “Photothermal therapy and photoacoustic imaging via nanotheranostics in fighting cancer,” Chemical Society Reviews 48(7), 2053–2108 (2019).
  * [16] P. Wray, L. Lin, P. Hu, and L. V. Wang, “Photoacoustic computed tomography of human extremities,” Journal of Biomedical Optics 24(2), 026003 (2019).
  * [17] C. Li and L. V. Wang, “Photoacoustic tomography and sensing in biomedicine,” Physics in Medicine and Biology 54(19), R59–R97 (2009).
  * [18] Y. Xu, L. V. Wang, G. Ambartsoumian, and P. Kuchment, “Reconstructions in limited-view thermoacoustic tomography,” Medical Physics 31(4), 724–733 (2004).
  * [19] G. Paltauf, R. Nuster, M. Haltmeier, and P. Burgholzer, “Experimental evaluation of reconstruction algorithms for limited view photoacoustic tomography with line detectors,” Inverse Problems 23(6), S81–S94 (2007).
  * [20] X. L. Deán-Ben, L. Ding, and D. Razansky, “Dynamic particle enhancement in limited-view optoacoustic tomography,” Optics Letters 42(4), 827–830 (2017).
  * [21] Z. Guo, L. Li, and L. V. Wang, “On the speckle-free nature of photoacoustic tomography,” Medical Physics 36(9), 4084–4088 (2009).
  * [22] X. L. Deán-Ben and D. Razansky, “On the link between the speckle free nature of optoacoustics and visibility of structures in limited-view tomography,” Photoacoustics 4(4), 133–140 (2016).
  * [23] J. Gateau, T. Chaigne, O. Katz, S. Gigan, and E. Bossy, “Improving visibility in photoacoustic imaging using dynamic speckle illumination,” Optics Letters 38(23), 5188–5191 (2013).
  * [24] T. Chaigne, J. Gateau, M. Allain, O. Katz, S. Gigan, A. Sentenac, and E. Bossy, “Super-resolution photoacoustic fluctuation imaging with multiple speckle illumination,” Optica 3(1), 54–57 (2016).
  * [25] E. Hojman, T. Chaigne, O. Solomon, S. Gigan, E. Bossy, Y. C. Eldar, and O. Katz, “Photoacoustic imaging beyond the acoustic diffraction-limit with dynamic speckle illumination and sparse joint support recovery,” Optics Express 25(5), 4875–4886 (2017).
  * [26] T. W. Murray, M. Haltmeier, T. Berer, E. Leiss-Holzinger, and P. Burgholzer, “Super-resolution photoacoustic microscopy using blind structured illumination,” Optica 4(1), 17–22 (2017).
  * [27] T. Dertinger, R. Colyer, G. Iyer, S. Weiss, and J. Enderlein, “Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI),” Proceedings of the National Academy of Sciences 106(52), 22287–22292 (2009).
  * [28] T. Chaigne, B. Arnal, S. Vilov, E. Bossy, and O. Katz, “Super-resolution photoacoustic imaging via flow-induced absorption fluctuations,” Optica 4(11), 1397–1404 (2017).
  * [29] E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science 313(5793), 1642–1645 (2006).
  * [30] S. T. Hess, T. P. K. Girirajan, and M. D. Mason, “Ultra-high resolution imaging by fluorescence photoactivation localization microscopy,” Biophysical Journal 91(11), 4258–4272 (2006).
  * [31] S. Vilov, B. Arnal, and E. Bossy, “Overcoming the acoustic diffraction limit in photoacoustic imaging by the localization of flowing absorbers,” Optics Letters 42(21), 4379–4382 (2017).
  * [32] X. L. Deán-Ben and D. Razansky, “Localization optoacoustic tomography,” Light: Science & Applications 7(4), 18004 (2018).
  * [33] P. Zhang, L. Li, L. Lin, J. Shi, and L. V. Wang, “In vivo superresolution photoacoustic computed tomography by localization of single dyed droplets,” Light: Science & Applications 8(1), 36 (2019).
  * [34] S. Vilov, G. Godefroy, B. Arnal, and E. Bossy, “Photoacoustic fluctuation imaging: theory and application to blood flow imaging,” Optica 7(11), 1495–1505 (2020).
  * [35] P. Frinking, T. Segers, Y. Luan, and F. Tranquart, “Three decades of ultrasound contrast agents: a review of the past, present and future improvements,” Ultrasound in Medicine & Biology 46(4), 892–908 (2020).
  * [36] L. V. Wang, “Tutorial on photoacoustic microscopy and computed tomography,” IEEE Journal of Selected Topics in Quantum Electronics 14(1), 171–179 (2008).
  * [37] M. A. Inzunza-Ibarra, E. Premillieu, C. Gruensteidl, R. Piestun, and T. W. Murray, “Sub-acoustic resolution optical focusing through scattering using photoacoustic fluctuation guided wavefront shaping,” Optics Express 28(7), 9823–9832 (2020).
  * [38] J. Brunker and P. Beard, “Acoustic resolution photoacoustic doppler velocimetry in blood-mimicking fluids,” Scientific Reports 6(1), 20902 (2016).
  * [39] J. Brunker and P. Beard, “Velocity measurements in whole blood using acoustic resolution photoacoustic doppler,” Biomedical Optics Express 7(7), 2789–2806 (2016).
  * [40] J. A. Feshitan, C. C. Chen, J. J. Kwan, and M. A. Borden, “Microbubble size isolation by differential centrifugation,” Journal of Colloid and Interface Science 329(2), 316–324 (2009).
  * [41] C. Demené, T. Deffieux, M. Pernot, B. Osmanski, V. Biran, J. Gennisson, L. Sieu, A. Bergel, S. Franqui, J. Correas, I. Cohen, O. Baud, and M. Tanter, “Spatiotemporal clutter filtering of ultrafast ultrasound data highly increases doppler and fultrasound sensitivity,” IEEE Transactions on Medical Imaging 34(11), 2271–2285 (2015).
  * [42] J. Baranger, B. Arnal, F. Perren, O. Baud, M. Tanter, and C. Demené, “Adaptive spatiotemporal svd clutter filtering for ultrafast doppler imaging using similarity of spatial singular vectors,” IEEE transactions on medical imaging 37(7), 1574–1586 (2018).
  * [43] C. Errico, J. Pierre, S. Pezet, Y. Desailly, Z. Lenkei, O. Couture, and M. Tanter, “Ultrafast ultrasound localization microscopy for deep super-resolution vascular imaging,” Nature 527(7579), 499–502 (2015).
  * [44] G. Zhang, S. Harput, S. Lin, K. Christensen-Jeffries, C. H. Leow, J. Brown, C. Dunsby, R. J. Eckersley, and M. X. Tang, “Acoustic wave sparsely activated localization microscopy awsalm: Super-resolution ultrasound imaging using acoustic activation and deactivation of nanodroplets,” Applied Physics Letters 113(1), 014101 (2018).


# UTTS: Unsupervised TTS with Conditional Disentangled Sequential Variational
Auto-encoder

‡Jiachen Lian, , ‡Chunlei Zhang, , Gopala K. Anumanchipalli, , Dong Yu Work
done when Jiachen was an intern at Tencent AI Lab, Bellevue, WA. ‡ Equal
contribution. Chunlei Zhang is the corresponding author.

###### Abstract

In this paper, we propose a novel unsupervised text-to-speech (UTTS) framework
which does not require text-audio pairs for the TTS acoustic modeling (AM).
UTTS is a multi-speaker speech synthesizer that supports zero-shot voice
cloning, it is developed from a perspective of disentangled speech
representation learning. The framework offers a flexible choice of a speaker’s
duration model, timbre feature (identity) and content for TTS inference. We
leverage recent advancements in self-supervised speech representation learning
as well as speech synthesis front-end techniques for system development.
Specifically, we employ our recently formulated Conditional Disentangled
Sequential Variational Auto-encoder (C-DSVAE) as the backbone UTTS AM, which
offers well-structured content representations given unsupervised alignment
(UA) as condition during training. For UTTS inference, we utilize a lexicon to
map input text to the phoneme sequence, which is expanded to the frame-level
forced alignment (FA) with a speaker-dependent duration model. Then, we
develop an alignment mapping module that converts FA to UA. Finally, the
C-DSVAE, serving as the self-supervised TTS AM, takes the predicted UA and a
target speaker embedding to generate the mel spectrogram, which is ultimately
converted to waveform with a neural vocoder. We show how our method enables
speech synthesis without using a paired TTS corpus. Experiments demonstrate
that UTTS can synthesize speech of high naturalness and intelligibility
measured by human and objective evaluations. Audio samples are available at
our demo page 111https://neurtts.github.io/utts_demo/.

###### Index Terms:

unsupervised TTS acoustic modeling, self-supervised learning, wavLM, C-DSVAE,
representation learning

##  I Introduction

Text-to-speech (TTS) synthesis plays an important role for human computer
interaction. With the continuous development of neural-based TTS systems,
e.g., Tacotron [1], DurIAN [2], FastSpeech [3, 4, 5] or more recent Glow-TTS
series [6, 7, 8], high-fidelity synthetic speech has been produced and the gap
between machine generated speech and real speech is reduced. This is
especially true for languages with rich resources, which refers to a sizeable
high quality parallel speech and textual data. Usually, the supervised TTS
system requires dozens of hours of single-speaker high quality data to retain
a good performance. However, collecting and labeling such data is a non-
trivial task, which is time-consuming and expensive. In that sense, current
supervised solutions are still have their limitations on the demanding needs
of ubiquitous deployment of customized speech synthesizers for AI assistants,
gaming or entertainment industries. Natural, flexible and controllable TTS
pathways become more essential when facing these diverse needs.

One possible way to relax the aforementioned limitation is to develop
unsupervised text-to-speech (UTTS) synthesis, which does not require high
quality parallel speech and textual data for training TTS acoustic models
(AM). In this preliminary study, we propose a novel UTTS framework based on
the recent advancement of self-supervised learning (SSL) techniques for speech
representation [9, 10, 11, 12]. Like many conventional supervised TTS systems,
our proposed UTTS framework is essentially a two-stage TTS process, namely an
acoustic model that converts a raw text to a mel spectrogram and a vocoder
which transforms the mel spectrogram to the speech waveform. While the
supervised TTS AM achieves the text to mel spectrogram transformation by
explicitly composing an external aligner or by learning a latent alignment [2,
4, 3, 8], the UTTS AM must resolve this mapping with an indirect way since no
paired TTS training data is available.

Figure 1:  Supervised TTS AM to UTTS AM

For conceptual comparison purpose, Fig.1 illustrates the necessary components
of a typical supervised TTS AM (e.g., DurIAN [2] or FastSpeech2 [4] which
shares a similar structure) and our UTTS AM. Although UTTS follows most of the
procedures of supervised AM during inference, the UTTS AM differs the
supervised AM in three major ways: 1) For supervised AM, the neural network
maps forced alignment of phoneme sequence to frame-by-frame acoustic features.
We replace forced alignment with unsupervised alignment, which is derived from
acoustic partitioning with the SSL features (e.g., kmeans labels of WavLM [10]
features). 2) supervised AMs (DurIAN and FastSpeech2) are only trained with
TTS speech data, since UTTS investigates the Conditional Disentangled
Sequential Variational Auto-encoder (C-DSVAE [11]) as the backbone, the self-
supervised/unsupervised model possibly enables more generic speech data to
build a more generalized AM instead of being restricted with the TTS data
only. 3) UTTS uses external speech recognition data to train the multi-speaker
duration model and the projection between forced alignment to unsupervised
alignment (FA2UA), while traditional TTS AMs rarely introduces such
variability.

For system implementation and experimental studies, the multi-speaker VCTK and
LibriTTS dataset [13, 14] are utilized to demonstrate the effectiveness of the
proposed method. We show the possibility that UTTS can generate speech of high
naturalness and intelligibility in terms of objective and subjective tests.

Although more detailed explanations and analysis can be found throughout this
paper, the main contributions of this work are summarized as follows:

  * •

By exploring the conditions to end to end disentangled speech representation
learning, we justify that unsupervised TTS AM is feasible.

  * •

UTTS and voice conversion are unified under a generative model (C-DSVAE),
showing the potential of employing self-supervised (disentangled)
representation learning for voice generation tasks.

  * •

With disentangled representations, UTTS offers a flexible framework that can
bridge control signal (speaker identity, text, style etc.) and generated
voice, which enables more fine grained processes for controllable and
customized TTS.

The rest of this paper is organised as follows. Sec II introduces the
background and related work. Section III illustrates the C-DSVAE and its
development for voice generation. The UTTS framework is presented in Sec IV.
Experimental setup and results are shown in Sec V, with conclusions in Sec.
VI.

##  II Related Work

In this section, we introduce the background and relation to our proposed
method.

Supervised TTS: the neural network based supervised TTS systems can be
generally categorised into two forms, the end-to-end methods [15, 4, 16] that
directly maps text to speech waveform and the two-stage solutions [1, 2, 3, 4,
5, 7, 8, 16] that utilize an acoustic model to convert text to frame by frame
acoustic features (e.g., mel spectrogram) and a vocoder transforms the
intermediate features to the waveform. While the end-to-end TTS draws an
increased attention for some tasks, most of the current studies still follow
the two-stage methods, mainly because of the less constraint from the data and
optimization. As mentioned in the Introduction, the proposed UTTS system
assembles like the two-stage supervised TTS, with the differences in some of
the critical building blocks.

Unsupervised TTS: there are not so many successful tries of the unsupervised
TTS system development until recently. Thanks to the recent development of
self-supervised speech representation learning, both systems in [17, 18] used
an unsupervised ASR model to transcribe the TTS speech data. With the pseudo
labels, they once again can adopt the supervised TTS recipe to build the
unsupervised TTS systems. The result indicates that it is possible to produce
speech with high intelligibility even without paired TTS data. One notable
concern for such unsupervised TTS systems remains in the development of
unsupervised ASR, which is still requires necessary improvement to close the
gap between the supervised counterparts [19, 20]. Contrasting with our
unsupervised approach, instead of transcribing the training speech data and
performing supervised network mapping, our TTS AM is a variant of variational
auto-encoder (i.e., C-DSVAE [11]), with unsupervised alignment as the
condition to regularize the learned latent representation to follow the
phonetic structure. With this design, our UTTS system requires an additional
mapping for text (forced alignment) space to the unsupervised alignment space.
For that purpose, we just conduct a data driven process to map forced
alignment to unsupervised alignment with the employment of an ASR system.

Disentangled speech representation for voice generation: speech
disentanglement algorithms, such as DSVAE and C-DSVAE [21, 22, 12, 11],
provide a way to isolate global timbre/speaker feature and local content
features. Our work is inspired from disentangled speech representation
learning and its downstream tasks, such as reconstruction/resynthesis and
voice conversion [12, 11, 23, 24]. The difference between those studies and
UTTS is that, resyntheis or VC are speech driven, the process keeps the
content unchanged, while UTTS has to make a linkage between raw text and the
intermediate content embeddings in order to generate meaningful output speech.
And how to connect the front-end text with unsupervised TTS AM (C-DSVAE)
remains the key in building our UTTS system.

##  III C-DSVAE for Voice Generation

Figure 2: The system diagram of conditional DSVAE.

###  III-A Overview

Disentangling speaker information from speech content is of great importance
to speech tasks such as speaker recognition, automatic speech recognition,
voice conversion and text-to-speech [25, 26, 27, 28, 29]. Of the existed
speech disentanglement methods, DSVAE [22] is a compelling framework that
generates the disentangled speaker and content representation in an
unsupervised manner. A recent DSVAE study in speech [12] presents impressive
speaker verification and voice conversion performance by balancing the
information flow between speaker and content. As the content prior being
assumed independent of the input data, early versions of DSVAE suffer from
discontinuous and unstable speech generation with damaged phonetic structures
[30, 31, 32, 22]. More recently, C-DSVAE [11] was proposed to introduce the
content-dependent acoustic alignment as a condition to regularize the DSVAE
training, from which the phonetically discriminative content representations
are learned and the stabilized vocalization is achieved, overcoming the
architecture drawback of earlier explorations in DSVAE [30, 31, 32] and
delivering enhanced zero-shot VC performances. In the following sections, we
first present a systematic overview of the DSVAE framework, which serves as
the backbone of the improved C-DSVAE. Then, we presents detailed development
methods of C-DSVAE by deliberating different types of content conditions as
well as training objectives. Furthermore, we extend C-DSVAE by introducing the
masked prediction training to capture the robust and long-range temporal
relations over acoustic units. In the last two sections, we discuss how the
voice conversion and alignment-driven voice generation are performed within
the C-DSVAE framework, the latter exploration serves as the cornerstone of our
UTTS system to be discussed in Sec. IV. The detailed model architectures can
be found in Sec. V-B1.

###  III-B DSVAE Framework

####  III-B1 Backbone

The backbone DSVAE architecture is presented in Fig. 2, which consists of a
shared encoder $E_{share}$, a posterior speaker encoder $E_{sq}$, a posterior
content encoder $E_{cq}$, a prior speaker encoder $E_{sp}$, a prior content
encoder $E_{cp}$ and a decoder $D$. The input utterance $X$ is passed into
$E_{share}$, followed by $E_{sq}$ and $E_{cq}$ which encode the speaker
posterior distribution $q(z_{s}|X)$ and the content posterior distribution
$q(z_{c}|X)$, respectively. $z_{s}$ denotes the speaker embedding and $z_{c}$
denotes the content embedding. For the prior modeling, $E_{sp}$ encodes the
speaker prior $p(z_{s})$ and $E_{cp}$ encodes the content prior $p(z_{c})$.
Depending on different voice generation tasks, during the decoding/generation
stage, the speaker embedding $z_{s}$ and content embedding $z_{c}$ are sampled
from either the posteriors $q(z_{s}|X)$ and $q(z_{c}|X)$ or the priors
$p(z_{s})$ and $p(z_{c})$, and the concatenation of them is passed into
decoder $D$ to generate the synthesized speech $\hat{X}=D(z_{s},z_{c})$.

####  III-B2 Probabilistic Graphical Models and Training Objective

As introduced in [22, 12, 11], DSVAE is designed to follow independence
assumption for both priors and posteriors to implicitly leverage
disentanglement between speaker and content representations. The probabilistic
graphical models are formulated in Eq. 1 and Eq. 2. Note that the
factorization of $p_{\theta}(z_{c})$ and $q_{\theta}(z_{c}|X)$ can either
follow the streaming manner [22, 32, 12] or non-streaming manner [11],
depending on the practical implementation. Eq. 1 and Eq. 2 correspond to the
latter one, where $t$ is the frame index and $\theta$ denotes model
parameters.

$p_{\theta}(z_{s},z_{c})=p(z_{s})p_{\theta}(z_{c})=p(z_{s})\prod_{t=1}^{T}p_{\theta}(z_{ct}|z_{c})$
(1)

$\begin{split}q_{\theta}(z_{s},z_{c}|X)&=q_{\theta}(z_{s}|X)q_{\theta}(z_{c}|X)\\\
&=q_{\theta}(z_{s}|X)\prod_{t=1}^{T}q_{\theta}(z_{ct}|X)\end{split}$ (2)

The overall training objective for DSVAE is a VAE loss, but with two latent
variables i.e. $z_{s}$ and $z_{c}$, as shown in Eq. 5. $L_{DSVAE}$ forces
strong disentanglement between $z_{s}$ and $z_{c}$ without additional
constraints such as mutual information regularization [30, 32, 31, 33], as
explained in [12]. KL divergence is denoted as $KLD$,
$p_{\theta}(X|z_{s},z_{c}))$ is modeled by the decoder $D$, representing the
generation process. $\alpha$ and $\beta$ are two balancing factors which are
used to control the disentanglement [12].

$\mathcal{L}_{\mathit{KLD_{s}}}=\mathbb{E}_{p(X)}[\mathit{KLD}(q_{\theta}(z_{s}|X)||p(z_{s}))]$
(3)

$\mathcal{L}_{\mathit{KLD_{c}}}=\mathbb{E}_{p(X)}[\mathit{KLD}(q_{\theta}(z_{c}|X)||p_{\theta}(z_{c}))]$
(4)

$\begin{split}\mathcal{L}_{\mathit{DSVAE}}=&\mathbb{E}_{p(X)}\mathbb{E}_{q_{\theta}(z_{s},z_{c}|X)}[-\log(p_{\theta}(X|z_{s},z_{c}))]\\\
&+\alpha\mathcal{L}_{\mathit{KLD_{s}}}+\beta\mathcal{L}_{\mathit{KLD_{c}}}\end{split}$
(5)

###  III-C C-DSVAE System

In the DSVAE systems [22, 30, 32, 12], the content prior $p(z_{c})$ is
independent of $X$. As formulated in Eq.4, the training objective is to
minimize the KL divergence between $p(z_{c})$ and $q(z_{c}|X)$. Since
$q(z_{c}|X)$ explicitly models the phonetic structure of the speech content,
the content-independent prior will force the content representation to lose
phonetic discriminativity in the latent space, which is detrimental to the
disentanglement. In order to generate phonetically meaningful and continuous
speech with stable vocalizations, we introduce the acoustic alignment $A_{X}$
as the condition for content prior distribution. The motivations are two-
folds. First, acoustic alignment carries the phonetic structure of the
utterance, which will be encoded into the content prior model. Such alignment-
aware prior acts as a “teacher” model to force the “student”, which is the
content posterior $q_{\theta}(z_{c}|X)$, to inherit the phonetic structure of
the original utterance. Second, during the inference, the acoustic alignment
$A_{X}$ could be sampled from the alignment-aware prior so as to perform
alignment-driven speech synthesis, which is the basis for our UTTS system to
be introduced in Sec. IV.

####  III-C1 Acoustic Alignment as Content Condition

We consider two types of acoustic alignment: forced alignment (FA) and
unsupervised alignment (UA). For a specific utterance $X$, we denote the
acoustic alignment as $A_{X}^{FA}$ and $A_{X}^{UA}$ respectively.

Forced Alignment: $A_{X}^{FA}$ is the forced alignment of the utterance $X$.
We use Montreal forced alignment (MFA) [34] to extract the forced alignment
given audio-text pair. In consistency with [5], the forced alignment is a
sequence of monophones and there is a total number of 72 monophones. In the
next phase, the content prior encoder $E_{cp}$ takes the one-hot form of
alignment sequence as input at each time step and predicts the frame-wise
content prior distribution $p(z_{c}|A_{X}^{FA})$.

Unsupervised Alignment: The forced alignment $A_{X}^{FA}$ is not always
available since audio-transcription pair does not naturally exist. And because
of this, the existing supervised TTS models are not scaled-well with respect
to audio samples under diverse acoustic conditions, which limits their
potential in synthesizing natural and expressive speech in the generalized
open domains. To further utilize the advantage of untranscribed audio samples
that are largely existed, we switch to the unsupervised alignment that is
derived via the self-supervised pre-training. Of the popularly proposed self-
supervised speech pre-training models, WavLM [10] delivers state-of-art
performance in “full stack” downstream speech tasks such as speaker
recognition, speaker diarization, ASR and speech separation. WavLM employs a
denoising masked speech prediction training strategy by feeding noisy and
overlapped speech as inputs to predict the pseudo-labels of original speech on
masked region, which improves both the speech content modeling capability and
model robustness. The WavLM-Base model is pretrained from Librispeech-960h
[35], and we adopt the WavLM-Base [10] model to extract the acoustic features.
With WavLM features adopted, Kmeans++ [36] algorithm is performed on the
features to generate frame-wise pseudo labels, denoting as unsupervised
alignment $A_{X}^{UA}$, which is also assumed to capture the phonetic
structure of the utterance $X$. The pretrained WavLM-Base model together with
Kmeans algorithm acts as the acoustic aligner as indicated in Fig. 2 (a).

####  III-C2 Masked Unit Prediction Training

In order to capture the robust and long-range temporal relations over acoustic
units and to generate more continuous speech, we adopt the Masked Unit
Prediction (MUP) when training the content prior encoder $E_{cp}$. Denote
$M(A_{X})\subset[T]$ as the collection of $T$ masked indices for a specific
condition $A_{X}$, where the masking configuration is consistent with [9]. Let
$\tilde{A_{X}}$ be a corrupted version of $A_{X}$, in which $A_{X_{t}}$ will
be masked out if $t\in M(A_{X})$. Denote $z_{cp}$ as the sample of output of
the content prior encoder $E_{cp}$, i.e. $z_{cp}\sim E_{cp}(A_{X})$. The
negative loglikelihood loss (NLL) $\mathcal{L}_{\mathit{MUP-C}}$ for condition
modeling is defined in Eq. 6, where $p(z_{{cp}_{i}}|\tilde{A_{X_{i}}})$ is the
softmax categorical distribution. $\mathbb{E}_{A_{X}}$ denotes the expectation
over all $A_{X}$. Since it is reported in [9] that masked-only prediction
delivers impressive ASR performances, we do not incorporate the prediction
loss for unmasked regions.

$\mathcal{L}_{\mathit{MUP-C}}=-\mathbb{E}_{A_{X}}\Sigma_{i\in M(A_{X})}\log
p(z_{{cp}_{i}}|\tilde{A_{X_{i}}})$ (6)

####  III-C3 Training Objective and Configuration

We adopt the DSVAE training objective as expressed in Eq. 5, with the
exception that the $L_{KLD_{c}}$ is formulated with conditional content prior,
as shown in Eq. 7. We design the content prior encoder $E_{cp}$ such that
$p_{\theta}(z_{c}|A_{X})$ follows a non-streaming factorization. The masked
unit prediction loss $L_{MUP-C}$ is also incorporated. The overall loss
objective for C-DSVAE is expressed in Eq. 8. When $\gamma=0$, it is vanilla
C-DSVAE training. We set $\alpha=0.01$, $\beta=10$ and $\gamma=1$ if masked
unit prediction training is involved. We adopt Adam optimizer [37] with an
initial learning rate of 5e-4 which is decayed by 0.95 every 5 epochs. The
batch size is fixed to be 256. The training configurations are in line with
[12, 11].

$\mathcal{L}_{\mathit{KLD_{c}-C}}=\mathbb{E}_{p(X)}[\mathit{KLD}(q_{\theta}(z_{c}|X)||p_{\theta}(z_{c}|A_{X}))]$
(7)
$\begin{split}\mathcal{L}_{\mathit{C-DSVAE}}=&\mathbb{E}_{p(X)}\mathbb{E}_{q_{\theta}(z_{s},z_{c}|X)}[-log(p_{\theta}(X|z_{s},z_{c}))]\\\
&+\alpha\mathcal{L}_{\mathit{KLD_{s}}}+\beta\mathcal{L}_{\mathit{KLD_{c}-C}}+\gamma
L_{\mathit{MUP-C}}\end{split}$ (8)

###  III-D Applications of C-DSVAE

Voice Conversion: VC seeks to convert the speaker information of source
utterance to the target speaker while keeping the speech content unchanged. As
presented in Fig. 2 (b), given the arbitrary source utterance $X_{1}$ and
arbitrary target utterance $X_{2}$, the speaker embedding $z_{s2}$ is
extracted from $X_{2}$ using the posterior speaker encoder $E_{sq}$ and the
content embedding $z_{c1}$ is extracted from $X_{1}$ using the posterior
content encoder $E_{cq}$. The decoder then takes the concatenation of $z_{s2}$
and $z_{c1}$ and generates the converted mel spectrogram $\hat{X_{12}}$, which
is converted to waveform via a pre-trained neural vocoder.

Alignment driven Voice Generation: Given acoustic alignment $A_{C}$ (either
forced alignment or unsupervised alignment) and arbitrary target utterance
$X_{T}$, alignment-driven voice generation aims to synthesize the speech from
the alignment with the target speaker embedding, as shown in Fig. 2 (c).
Specifically, the speaker embedding $z_{s}$ is extracted from $X_{T}$ using
posterior speaker embedding $E_{sq}$ and the content embedding is sampled from
$p_{\theta}(z_{c}|A_{C})$ generated via the content prior encoder $E_{cp}$,
which takes $A_{C}$ as input. The alignment-driven voice generation is an
essential trick that bridge the textual condition to the acoustic
representations in our study, which serves as a cornerstone of our UTTS
framework.

##  IV UTTS System

Figure 3: The core components of the proposed UTTS system.

Our proposed UTTS pipeline is presented in Fig. 3. We first obtain the phoneme
sequence of the text transcription with Librispeech Lexicon
222https://www.openslr.org/11/, which is consistent with [5]. The phoneme
sequence is then converted to a list of token indices. The phoneme sequence
(token indices) together with a speaker embedding (from a speaker pool for
duration model training) is passed into the Speaker-Aware Duration Prediction
module which delivers the predicted forced alignment (FA). The FA2UA module
takes FA as input and predicts unsuperviserd alignment (UA). The UA along with
a target speaker embedding is fed into the C-DSVAE to generate mel
spectrogram. A neural vocoder is then applied to convert mel spectrogram to
waveform. It is noted that the proposed UTTS system performs zero-shot voice
cloning for the target utterance, which is a very difficult setting for TTS.
In the following sections, we present the Speaker-Aware Duration Prediction
and FA2UA modules. Both the modules are trained separately, the detailed model
architecture of each module are presented in Sec. V-B1.

###  IV-A Speaker-Aware Duration Prediction

The duration predictor takes the phoneme sequence as well as speaker embedding
as input to predict the speaker-aware duration for each phoneme [2, 4], as
shown in Fig. 3 (a). Specifically, the phoneme sequence is first passed into a
trainable look-up table to obtain the phoneme embeddings. Afterwards, a four-
layer multi-head attention (MHA) module is followed to extract the latent
phoneme representation. A two-layer conv-1D module is then used to take the
summation of latent phoneme representation and speaker embedding sampled from
the speaker pool. A linear layer is finally applied to generate the predicted
duration in the logarithmic domain. During training, the Mean Squared Error
(MSE) is utilized to calculate the difference between the predicted duration
and the target duration. For training, the target duration of a phoneme
sequence is obtained from the forced alignment extracted by Montreal Forced
Alignment (MFA) [34]. Note that the target duration is also in the logarithmic
scale. During inference, we round up the duration generated by the predictor
and expand the phoneme sequence to form an estimated forced alignment. We also
present an example of how the duration predictor works during inference.
Assume a random speaker is sample and the phoneme sequence is [55 2 7]. The
predicted duration sequence is [2.2 1.8 0.9], correspondingly. We first round
up the duration to make it become [3 2 1]. The phoneme sequence is then
expanded to [55 55 55 2 2 7] as the predicted forced alignment (FA). The
duration predictor works similarly to [2, 4, 5].

###  IV-B FA2UA with Masked Prediction Training

The FA2UA module takes forced alignment (FA) as input and predicts
corresponding unsupervised alignment (UA). Specifically, FA is first passed
into a learnable look-up table to obtain the FA embeddings. Subsequently, a
3-layer Bi-LSTM module is employed to predict the UA embeddings given FA
embeddings. During training, we adopt a masked prediction training strategy to
train our FA2UA module as masked prediction is expected to be good at capture
the long-range time dependency across tokens and encode more contextualized
information for each token. Similar as Sec. III-C2, we denote
$M(\mathit{FA})\subset[T]$ as the collection of $T$ masked indices for a
specific unsupervised alignment $\mathit{FA}$. Let $\tilde{FA}$ be a corrupted
version of $\mathit{FA}$, in which $\mathit{FA}_{i}$ will be masked out if
$i\in M(\mathit{FA})$. Denote ${\mathit{UA}}_{i}$ as the corresponding frame
level unsupervised alignment of ${\mathit{FA}}_{i}$, which serves as the
ground truth for the masked prediction training. The negative loglikelihood
loss (NLL) $\mathcal{L}_{\mathit{FA2UA}}$ for masked prediction training is
defined in Eq. 9, where $p({\mathit{UA}}_{i}|\tilde{{\mathit{FA}}_{i}})$ is
the softmax categorical distribution. $\mathbb{E}_{\mathit{(FA,UA)}}$ denotes
the expectation over all $\mathit{(FA,UA)}$ pairs. Following Sec. III-C2,
masked-only prediction loss is incorporated for unsupervised alignment
prediction. Fig. 3 (b) presents an example, where [55 2] are masked out from
the input FA and the model only predicts the corresponding UA targets [6 12]
in the training phase. During the inference, we pick the token with the
maximum probability $p({\mathit{UA}}_{i}|{\mathit{FA}}_{i})$ at each time step
$i$ to form the predicted UA sequence.

$\mathcal{L}_{\mathit{FA2UA}}=-\mathbb{E}_{\mathit{(FA,UA)}}\Sigma_{i\in
M(\mathit{FA})}\log p({\mathit{UA}}_{i}|\tilde{{\mathit{FA}}_{i}})$ (9)

###  IV-C Dual Content Encoders for Reconstruction

It is noted that the content representation to the decoder of C-DSVAE is
sampled from the posterior content encoder $E_{cq}$ in the original C-DSVAE
training, which is suitable for the voice conversion tasks but not necessary
ideal for alignment-driven voice generation (i.e., UTTS inference). There
exists a mismatch between the reconstruction based training and the UTTS
inference. To alleviate this mismatch, we propose to use dual content encoders
for reconstruction in the C-DSVAE training. Specifically, we run the normal
C-DSVAE training guided by Eq. 8 until convergence, and then we start a second
round of training where we also sample latent representation $z_{cp}$ from the
prior content encoder for reconstruction. And the second round of training is
initialed with the pretrained C-DSVAE model. With this kind of training
strategy, the dual content encoders for reconstruction training can be
described as Eq. 10:

$\begin{split}\mathcal{L}_{\mathit{C-DSVAE}}=&\frac{1}{2}\\{\mathbb{E}_{p(X)}\mathbb{E}_{q_{\theta}(z_{s},z_{cq}|X)}[-log(p_{\theta}(X|z_{s},z_{cq}))]+\\\
&\mathbb{E}_{p(X)}\mathbb{E}_{q_{\theta}(z_{s},z_{cp}|X,A_{X})}[-log(p_{\theta}(X|z_{s},z_{cp}))]\\}\\\
&+\alpha\mathcal{L}_{\mathit{KLD_{s}}}+\beta\mathcal{L}_{\mathit{KLD_{c}-C}}+\gamma
L_{\mathit{MUP-C}}\end{split}$ (10)

where the new reconstruction term is formulated as an average loss with both
posterior content embedding and prior content embedding for reconstruction. In
our experiment, we find a clear quality improvement of the UTTS samples over
the posterior content only reconstruction models.

##  V Experiments

###  V-A datasets

##### VCTK

For VCTK corpus [13], we exclude speaker p315 since the transcriptions are not
available. we then randomly select 90% of the speakers as training set and the
remaining 10% as test set. The data is downsampled to 16K Hz. We extract mel
spectrogram with the window size/hop size of 64ms/16ms, and the feature
dimension is 80. 36900 trials are randomly generated for speaker verification
subtasks. The settings are consistent with [12, 11].

##### LibriTTS

We downsample the original LibriTTS [14] data to 16k Hz. The original training
split is taken as training set for C-DSVAE, FA2UA and duration predictor
training. The mel spectrogram that is extracted for training follows the same
configurations as the VCTK dataset.

###  V-B Implementation details

####  V-B1 Model Architectures

Table. I presents details of UTTS system. In the C-DSVAE architecture, for
posterior speaker encoder $E_{sq}$, posterior content encoder $E_{cq}$ and
prior content $E_{cp}$, there is one dense layer used to predict the mean and
one other dense layer used to predict the standard deviation. A linear
classifier will be added at the end of $E_{cp}$ if masked prediction training
is involved. Both speaker and content dimension are set to 64. The decoder $D$
consists of a prenet $D_{pre}$ and a postnet $D_{post}$, which were introduced
in [27, 12, 11]. We use HiFi-GAN pretrained with LibriTTS as the universial
vocoder [38, 14] for voice conversion and text-to-speech synthesis. We use the
same mask configuration as [9] for all mask prediction experiments.

UTTS Model Architecture
---
C-DSVAE | $E_{Share}$ | (Conv1D(256, 5, 2, 1)$\rightarrow$ InstanceNorm2D$\rightarrow$ ReLU)$\times$3
$E_{sq}$ | BiLSTM(512, 2)$\rightarrow$ Average Pooling $\rightarrow$ (Dense(64)$\Rightarrow$ mean, Dense(64)$\Rightarrow$ std)
$E_{sp}$ | Identity Mapping
$E_{cq}$ | BiLSTM(512, 2)$\rightarrow$RNN(512, 1)$\rightarrow$ (Dense(64)$\Rightarrow$ mean, Dense(64)$\Rightarrow$ std)
$E_{cp}$ | BiLSTM(512, 2)$\rightarrow$ (Dense(64)$\Rightarrow$ mean, Dense(64)$\Rightarrow$ std)($\rightarrow$ Linear Classifier)
$D_{pre}$ | (InstanceNorm2D$\rightarrow$ Conv1D(512, 5, 2, 1)$\rightarrow$ ReLU)$\times$3
LSTM(512, 1) $\rightarrow$ LSTM(1024, 2) $\rightarrow$ Dense(80)
$D_{post}$ | (Conv1D(512, 5, 2, 1)$\rightarrow$ tanh$\rightarrow$ InstanceNorm2D)$\times$4
Duration Predictor | nn.Embedding$\rightarrow$MHA(256, 128, 128, 2)$\rightarrow$ Conv1D(256, 3, 2, 1) $\rightarrow$ Dense(1)
FA2UA | nn.Embedding$\rightarrow$BiLSTM(256, 3)$\rightarrow$ Linear Classifier
TABLE I: The UTTS network architecture in detail. For Conv1D, the
configuration is (filter size, kernel size, padding, stride). For Multi-Head-
Attention (MHA), the configuration is (model dimension, key dimension, value
dimension, $\\#$heads). For LSTM/BiLTSM/RNN, the configuration is (hidden dim,
layers). For Dense layer, the configuration is (output dim).

####  V-B2 Training Experiments of each components

For C-DSVAE system, we first train it without content condition (i.e. DSVAE
[12]). Next, we explore the C-DSVAE training with different content
conditions, training strategies and datasets, as noted in the bracket. For
instance, C-DSVAE(UA)/C-DSVAE(FA) indicates that the training is conditioned
on UA/FA, and C-DSVAE(+MP) represents the mask prediction training along with
either C-DSVAE(UA) or C-DSVAE(FA). By default C-DSVAE is trained on VCTK
corpus unless otherwise specified. For example, C-DSVAE(++LibriTTS) shows that
C-DSVAE is pretrained on LibriTTS inherited with the C-DSVAE(+MP) setting and
is then fine-tuned on VCTK under the same configuration.

In our experiments, the speaker-aware duration predictor and the FA2UA module
are trained with LibriTTS for two major reasons: 1) LibriTTS serves as a big
external dataset introducing more speaker and phonetic varibilities to the
UTTS AMs; 2) we justify that using out of domain speech data (instead of
paired VCTK speech and textual data), UTTS can build robust text-to-speech
transformations. And the following experimental results demonstrate more
details about the systems we developed.

###  V-C Evaluations on C-DSVAEs

####  V-C1 C-DSVAE content embeddings and speaker embeddings

We evaluate the disentanglement performance of C-DSVAE system via both the
t-SNE [39] analysis and objective analysis on content and speaker embeddings.

##### t-SNE Analysis

Given pre-trained models, the same utterance that is randomly selected from
VCTK test set is taken as input and the t-SNE visualization of the content
embeddings is made, where the labels are the forced alignment. We assume that
better phonetic discriminativity indicates better disentanglement. As shown in
Fig. 4, C-DSVAE(UA) outperforms DSVAE significantly in terms of phoneme
discriminativity, which was also reported in [11]. It is reasonable that
C-DSVAE(+MP) and C-DSVAE(++LibriTTS) delivers better phonetic structure than
C-DSVAE(UA) because masked prediction training does help encode the
contextualized and structured phonetic information into the content
embeddings. It is also observable that C-DSVAE(+MP) delivers better inter-
class but worse intra-class phoneme discriminativity, which is assumed to be
caused by involving out-of-domain LibriTTS corpus during the training.

(a) DSVAE

(b) C-DSVAE(UA)

(c) C-DSVAE(+MP)

(d) C-DSVAE(++LibriTTS)

Figure 4: Visualizations of learned content embeddings.

##### Objective Analysis

We perform speaker verification (SV) on both speaker and content embeddings
given pre-trained C-DSVAEs. The experiments are conducted on the VCTK SV
trials mentioned in Sec.V-A, equal error rate (EER) is the metric. Typically,
lower EER on speaker embeddings and higher EER on content embeddings indicate
better disentanglement. Additionally, the pre-trained C-DSVAE model is fine-
tuned with forced alignment on VCTK training set and the phoneme
classification accuracy (Top1) for test set is reported. As observed in
Table.II, C-DSVAEs outperform DSVAE consistently in all metrics, which is
consistent with [11]. The C-DSVAE with FA condition outperforms that with UA
condition, which is reasonable because FA is expected to capture better
phonetic structure then unsupervised labels. However, such difference is small
and UA-based C-DSVAE is capable of encoding disentangled speaker and content
representation which is also semantically meaningful. This provides the
prerequisite for the success of the UTTS systems.

| content embd | spk embd
---|---|---
model | Phn ACC % | SV EER % | SV EER %
DSVAE | 30.5 | 40.5 | 4.8
C-DSVAE(UA) | 53.6 | 43.2 | 3.9
C-DSVAE(+MP) | 68.2 | 41.6 | 4.1
C-DSVAE(++LibriTTS) | 72.1 | 44.5 | 3.6
C-DSVAE(FA) | 54.6 | 43.4 | 3.8
C-DSVAE(+MP) | 70.5 | 42.3 | 3.6
C-DSVAE(++LibriTTS) | 73.3 | 45.6 | 3.3

TABLE II: Objective analysis of content embeddings and speaker embeddings.

####  V-C2 Voice conversion

The other way to evaluate the disentanglement of C-DSVAE system is to perform
voice conversion, as mentioned in Sec. III-D.

| seen to seen | unseen to unseen
---|---|---
model | naturalness | similarity | naturalness | similarity
DSVAE(WaveNet) [12] |  3.38$\pm$0.08 |  3.46$\pm$0.07 |  3.36$\pm$0.10 |  3.56$\pm$0.08
DSVAE(HiFi-GAN) [11] |  3.56$\pm$0.07 |  3.60$\pm$0.08 |  3.52$\pm$0.08 |  3.62$\pm$0.11
C-DSVAE(UA) |  3.78$\pm$0.07 |  3.86$\pm$0.06 |  3.75$\pm$0.07 |  3.82$\pm$0.10
C-DSVAE(+MP) | 3.85$\pm$0.06 | 3.92$\pm$0.08 | 3.86$\pm$0.06 | 3.88$\pm$0.09
C-DSVAE(++LibriTTS) |  3.80$\pm$0.07 |  3.78$\pm$0.09 |  3.82$\pm$0.09 |  3.86$\pm$0.08

TABLE III:  The MOS (95% CI) test on voice conversion samples with different
C-DSVAE models.

We conduct a MOS test to evaluate the proposed C-DSVAE models for zero-shot
non-parallel voice conversion. The evaluation setup is similar to [11] and we
split the evaluation into to two sets. The first set is a collection of seen
speakers sampled from training set and the second set is a collection of
unseen speakers sampled from test set. Details on train/test splition of VCTK
is introduced in V-A. We randomly draw 30 pairs from each set, with 10 pairs
for male to male conversion, 10 pairs for female to female conversion and 10
pairs for cross-gender conversion. The listener needs to give a score for each
sample in a test case according to the criterion: 1 = Bad; 2 = Poor; 3 = Fair;
4 = Good; 5 = Excellent. The final score for each model is calculated by
averaging the collected results from 18 subjects. As introduced in the paper,
we employ HiFi-GAN [38] as the neural vocoder to convert the mel spectrogram
to the waveform. We utilized the samples from two DSVAE based systems for a
fair comparison [11]. Table III shows the MOS results of different models.
Since we have different subjects for MOS test and there is always inevitable
randomness in subjective evaluation. That is why we observe a difference in
C-DSVAE performance between Table III and [11]. However, the trends that
revealed from Table III are still in line with [11].

As illustrated in the Table III, comparing with the WaveNet, HiFi-GAN produces
consistently better MOS scores given the same acoustic features. C-DSVAE with
unsupervised alignment (UA) achieves 0.24 performance boost in contrast with
the best DSVAE voice conversion system. Of all the systems, the C-DSVAE(+MP)
produced the best objective scores in terms of naturalness and similarity
under both seen to seen and unseen to unseen scenarios. And
C-DSVAE(++LibriTTS) with additional LibriTTS training data does not outperform
the VCTK only C-DSVAE(+MP) system, which indicates that the domain mismatch
exists in the self-supervised representation learning frameworks. It is noted
that, the developed voice conversion systems are not sensitive to seen and
unseen speech utterances, which is a good attribute from system generalization
perspective. However, the relatively larger variance values in the unseen
condition shows that unseen cases are still more challenging than the seen
cases.

###  V-D Evaluations on UTTSs

####  V-D1 MOS test

We conduct MOS tests in terms of naturalness and audio quality on VCTK
speakers. 18 subjects are asked to make judgement about the synthesized speech
samples. Since our UTTS system is a zero-shot voice cloning format, we also
ask the subjects to evaluate the similarity between the UTTS voice and the
target voice. We use a standard 5-scale rate to label the speech.

| seen target | unseen target | ASR intelligibility
---|---|---|---
UTTS model | naturalness | similarity | naturalness | similarity | CER | WER
GT | 4.45 | 2.3 | 7.0
C-DSVAE(UA) | 3.22 | 3.56 | 3.28 | 3.51 | 11.3 | 19.3
C-DSVAE(+MP) | 3.52 | 3.72 | 3.56 | 3.64 | 9.7 | 16.5
C-DSVAE(++Dual) | 3.76 | 3.82 | 3.72 | 3.74 | 8.3 | 15.1
C-DSVAE(++LibriTTS) | 3.58 | 3.63 | 3.61 | 3.72 | 10.8 | 18.2

TABLE IV: A multi-speaker UTTS MOS test and ASR intelligibility test on VCTK.

As shown from Table IV, compared with the vanilla C-DSVAE(UA) AM, the masked
unit prediction multi-task training in the content prior model of C-DSVAE does
helps the overall speech quality. Having LibriTTS for pretraining helps but
not so significantly. If the dual prior encoder training is applied to the
C-DSVAE(+MP) system, both naturalness and similarity are improved clearly,
showing the importance of prior modeling in this VAE based non-autoregressive
TTS model. At the same time, the gap between synthesised speech and the ground
truth in naturalness indicates that some form of prosody are currently missing
in the proposed UTTS systems.

####  V-D2 Speech intelligibility

Results of intelligibility evaluation using a Librispeech ASR
model333https://huggingface.co/espnet/simpleoier_librispeech_asr_train_asr_conformer7_hubert_ll60k_large_raw_en_bpe5000_sp
are also listed in Table IV. We generated 1600 utterances each model for this
portion of study. Basically, the objective and subjective evaluation show a
similar trend. Speech from the best UTTS model (C-DSVAE(++Dual)) achieves a
15.1% WER, the gap between ground truth and UTTS speech suggests that the
proposed method can be further improved by either improving the FA2UA mapping
module or the content prior modeling. In Table V, we also list the ASR based
speech intelligibility test results on UTTS samples by gender. As shown from
the table, the ASR system does not show much performance preference w.r.t.
different genders on real speech. However, the ASR results for male and female
synthesised samples are showing a big difference. The much worse CER and WER
for female generated speech across all 4 UTTS systems indicates that
generating female voice seems to be more difficult in the multi-speaker TTS
systems.

| overall | male | female
---|---|---|---
UTTS model | CER | WER | CER | WER | CER | WER
GT | 2.3 | 7.0 | 2.4 | 7.3 | 2.2 | 6.7
C-DSVAE(UA) | 11.3 | 19.3 | 8.8 | 16.9 | 13.8 | 21.6
C-DSVAE(+MP) | 9.7 | 16.5 | 6.1 | 12.8 | 13.3 | 20.2
C-DSVAE(++Dual) | 8.3 | 15.1 | 5.2 | 11.5 | 11.4 | 18.7
C-DSVAE(++LibriTTS) | 10.8 | 18.2 | 7.3 | 14.7 | 14.5 | 21.8

TABLE V: An ASR intelligibility test on gender dependent UTTS samples.

####  V-D3 Speaker similarity between the real and UTTS speech

Since UTTS is a zero-shot voice cloning TTS system, Table III demonstrates how
human evaluates the similarity between the synthesised and target speech. In
the meantime, we are interested to know how machine perceives the similarity
and difference. For this purpose, we use a pretrained speaker embedding system
to extract speaker embeddings from real and synthesised utterances, and
project them to a 2D T-SNE [40, 39] space. Fig.5 illustrates the speaker
embedding projections of 4 speakers (P259, P275, P293 and P343). P259 and P293
are used for the UTTS AM training, while P275 and P343 are never used in the
training stage.

Figure 5: T-SNE visualization of utterances of 4 VCTK speakers. Real and
synthetic utterances appear nearby when they are from the same speaker,
however real and synthetic utterances consistently form distinct clusters.

As shown in Fig.5, the speaker similarity between real and UTTS speech is
showing some randomness. For example, P259 (male & seen) and P343 (female &
unseen) are showing much obvious speaker similarities in speaker embedding
space, while P275 (female & unseen) and P293 (male & seen) demonstrates a more
separated speaker attribute in the same space. All the synthesised utterances
are formulating a distinct distribution in contrast with the target speech
distribution, which indicates that the synthetic speech from the current UTTS
systems still can be detected with a normal speaker verification system.

##  VI Conclusion

In this study, we proposed a novel method to develop TTS systems without using
the paired TTS training data. Specifically, we explored C-DSVAE, a self-
supervised model that learned disentangled intermediate content and speaker
representations with unsupervised alignment as the prior condition, and
generated high quality speech with the latent speaker and content features
using a decoder network. We employed C-DSVAE as the backbone module for
developing the UTTS systems. For that purpose, we proposed a simple yet
effective forced alignment to unsupervised alignment module to bridge the TTS
front-end and the C-DSVAE. To further improve the speech quality and
naturalness, we proposed to have a dual prior encoder reconstruction training
strategy in our VAE based AM training, which significantly alleviated the
mismatch between training and test phase of UTTS inference. Preliminary
experiments demonstrated that our proposed UTTS systems can produce speech
with an impressive quality. The newly developed method shows great potential
for its self-supervised/unsupervised nature that can utilize general quality
speech data for future needs.

At the same time, we found the UTTS generated samples still have some
limitations in speech quality, speech expressiveness and speaker similarity.
The the speech fidelity related issues can be addressed by introducing more
advanced text encoding networks or methods [8, 41]. While the latter one can
be improved with more data involved in the self-supervised or semi-supervised
speaker representation training [25]. Thus, our study serves as the
preliminary research work, proving the concept of unsupervised TTS AM, while
leaving observed inadequacies as the future work.

## References

  * [1] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio _et al._ , “Tacotron: Towards end-to-end speech synthesis,” _arXiv preprint arXiv:1703.10135_ , 2017.
  * [2] C. Yu, H. Lu, N. Hu, M. Yu, C. Weng, K. Xu, P. Liu, D. Tuo, S. Kang, G. Lei _et al._ , “Durian: Duration informed attention network for speech synthesis.” in _INTERSPEECH_ , 2020, pp. 2027–2031.
  * [3] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech: Fast, robust and controllable text to speech,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [4] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” _arXiv preprint arXiv:2006.04558_ , 2020.
  * [5] C.-M. Chien, J.-H. Lin, C.-y. Huang, P.-c. Hsu, and H.-y. Lee, “Investigating on incorporating pretrained and learnable speaker representations for multi-speaker multi-style text-to-speech,” in _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 8588–8592.
  * [6] J. Kim, S. Kim, J. Kong, and S. Yoon, “Glow-tts: A generative flow for text-to-speech via monotonic alignment search,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 8067–8077, 2020.
  * [7] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2018, pp. 4779–4783.
  * [8] J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5530–5540.
  * [9] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [10] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” _arXiv preprint arXiv:2110.13900_ , 2021.
  * [11] J. Lian, C. Zhang, G. K. Anumanchipalli, and D. Yu, “Towards improved zero-shot voice conversion with conditional dsvae,” _arXiv preprint arXiv:2205.05227_ , 2022.
  * [12] J. Lian, C. Zhang, and D. Yu, “Robust disentangled variational speech representation learning for zero-shot voice conversion,” in _IEEE ICASSP_. IEEE, 2022.
  * [13] C. Veaux, J. Yamagishi, and K. MacDonald, “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,” 2017.
  * [14] H. Zen, R. Clark, R. J. Weiss, V. Dang, Y. Jia, Y. Wu, Y. Zhang, and Z. Chen, “Libritts: A corpus derived from librispeech for text-to-speech,” in _Interspeech_ , 2019. [Online]. Available: https://arxiv.org/abs/1904.02882
  * [15] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” _arXiv preprint arXiv:1609.03499_ , 2016.
  * [16] J. Cong, S. Yang, L. Xie, and D. Su, “Glow-wavegan: Learning speech representations from gan-based variational auto-encoder for high fidelity flow-based speech synthesis,” in _2021 ISCA Interspeech_ , 2021.
  * [17] A. H. Liu, C.-I. J. Lai, W.-N. Hsu, M. Auli, A. Baevskiv, and J. Glass, “Simple and effective unsupervised speech synthesis,” _arXiv preprint arXiv:2204.02524_ , 2022.
  * [18] J. Ni, L. Wang, H. Gao, K. Qian, Y. Zhang, S. Chang, and M. Hasegawa-Johnson, “Unsupervised text-to-speech synthesis by unsupervised automatic speech recognition,” _arXiv preprint arXiv:2203.15796_ , 2022.
  * [19] A. Baevski, W.-N. Hsu, A. Conneau, and M. Auli, “Unsupervised speech recognition,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 27 826–27 839, 2021.
  * [20] A. H. Liu, W.-N. Hsu, M. Auli, and A. Baevski, “Towards end-to-end unsupervised speech recognition,” _arXiv preprint arXiv:2204.02492_ , 2022\.
  * [21] W.-N. Hsu, Y. Zhang, and J. Glass, “Unsupervised learning of disentangled and interpretable representations from sequential data,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [22] Y. Li and S. Mandt, “Disentangled sequential autoencoder,” _arXiv preprint arXiv:1803.02991_ , 2018.
  * [23] A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” _arXiv preprint arXiv:2104.00355_ , 2021\.
  * [24] K. Qian, Y. Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, “Autovc: Zero-shot voice style transfer with only autoencoder loss,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5210–5219.
  * [25] C. Zhang and D. Yu, “C3-dino: Joint contrastive and non-contrastive self-supervised learning for speaker verification,” _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * [26] A. Nagrani, J. S. Chung, S. Albanie, and A. Zisserman, “Disentangled speech embeddings using cross-modal self-supervision,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6829–6833.
  * [27] K. Qian, Y. Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, “Autovc: Zero-shot voice style transfer with only autoencoder loss,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5210–5219.
  * [28] Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, P. Nguyen, R. Pang, I. Lopez Moreno, Y. Wu _et al._ , “Transfer learning from speaker verification to multispeaker text-to-speech synthesis,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [29] S. Wegmann, D. McAllaster, J. Orloff, and B. Peskin, “Speaker normalization on conversational telephone speech,” in _1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings_ , vol. 1. IEEE, 1996, pp. 339–341.
  * [30] Y. Zhu, M. R. Min, A. Kadav, and H. P. Graf, “S3vae: Self-supervised sequential vae for representation disentanglement and data generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6538–6547.
  * [31] J. Han, M. R. Min, L. Han, L. E. Li, and X. Zhang, “Disentangled recurrent wasserstein autoencoder,” 2021. [Online]. Available: https://arxiv.org/abs/2101.07496
  * [32] J. Bai, W. Wang, and C. P. Gomes, “Contrastively disentangled sequential variational autoencoder,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [33] S. Yuan, P. Cheng, R. Zhang, W. Hao, Z. Gan, and L. Carin, “Improving zero-shot voice style transfer via disentangled representation learning,” 2021\. [Online]. Available: https://arxiv.org/abs/2103.09420
  * [34] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal forced aligner: Trainable text-speech alignment using kaldi.” in _Interspeech_ , vol. 2017, 2017, pp. 498–502.
  * [35] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [36] D. Arthur and S. Vassilvitskii, “k-means++: The advantages of careful seeding,” Stanford, Tech. Rep., 2006.
  * [37] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2014. [Online]. Available: https://arxiv.org/abs/1412.6980
  * [38] J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 17 022–17 033, 2020.
  * [39] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [40] C. Zhang, K. Koishida, and J. H. Hansen, “Text-independent speaker verification based on triplet convolutional neural network embeddings,” _IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP)_ , vol. 26, no. 9, pp. 1633–1644, 2018.
  * [41] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 6840–6851, 2020.