# Amphion: An Open-Source Audio, Music and Speech Generation Toolkit

Xueyao Zhang*,1, Liumeng Xue*,1, Yicheng Gu*,1, Yuancheng Wang*,1, Haorui He3,
Chaoren Wang1, Xi Chen1, Zihao Fang1, Haopeng Chen1, Junan Zhang2, Tze Ying
Tang1,
Lexiao Zou3, Mingxuan Wang1, Jun Han1, Kai Chen2, Haizhou Li1, Zhizheng
Wu${\dagger}$,1,2,3
1School of Data Science, The Chinese University of Hong Kong, Shenzhen
2Shanghai AI Lab
3Shenzhen Research Institute of Big Data

###### Abstract

Amphion is an open-source toolkit for Audio, Music, and Speech Generation,
targeting to ease the way for junior researchers and engineers into these
fields. It presents a unified framework that is inclusive of diverse
generation tasks and models, with the added bonus of being easily extendable
for new incorporation. The toolkit is designed with beginner-friendly
workflows and pre-trained models, allowing both beginners and seasoned
researchers to kick-start their projects with relative ease. Additionally, it
provides interactive visualizations and demonstrations of classic models for
educational purposes. The initial release of Amphion v0.1 supports a range of
tasks including Text to Speech (TTS), Text to Audio (TTA), and Singing Voice
Conversion (SVC), supplemented by essential components like data
preprocessing, state-of-the-art vocoders, and evaluation metrics. This paper
presents a high-level overview of Amphion.

Amphion: An Open-Source Audio, Music and Speech Generation Toolkit

Xueyao Zhang*,1, Liumeng Xue*,1, Yicheng Gu*,1, Yuancheng Wang*,1, Haorui He3,
Chaoren Wang1, Xi Chen1, Zihao Fang1, Haopeng Chen1, Junan Zhang2, Tze Ying
Tang1, Lexiao Zou3, Mingxuan Wang1, Jun Han1, Kai Chen2, Haizhou Li1, Zhizheng
Wu${\dagger}$,1,2,3 1School of Data Science, The Chinese University of Hong
Kong, Shenzhen 2Shanghai AI Lab 3Shenzhen Research Institute of Big Data

##  1 Introduction

**footnotetext: Equal Contribution.${\dagger}$${\dagger}$footnotetext:
Correspondence to<EMAIL_ADDRESS>
The development of deep learning LeCun et al. (2015) has greatly improved the
performance of generative models Hinton and Salakhutdinov (2006); Kingma and
Welling (2014); Goodfellow et al. (2020); Ho et al. (2020); Vaswani et al.
(2017). Leveraging these models has enabled researchers and practitioners to
explore innovative possibilities, leading to notable progress and
breakthroughs across various fields, including computer vision Rombach et al.
(2022) and natural language processing Radford et al. (2019); Brown et al.
(2020); Ren et al. (2019, 2020); Shen et al. (2018, 2024). The potential in
tasks related to audio111The term “audio” encompasses two distinct
interpretations. The first interpretation adopts a narrow view, specifically
associating “audio” with sound effects, a definition frequently employed
alongside music and speech throughout this paper. The second interpretation
presents a broader perspective, where “audio” collectively refers to sound
effects, music, and speech, embracing all aspects of auditory content., music,
and speech generation has spurred the scientific community to actively publish
new models and ideas Liu et al. (2023a, b); Ren et al. (2019, 2020); Shen et
al. (2018, 2024).

There is an increasing presence of both official and community-driven open-
source repositories that replicate these models. However, the quality of these
repositories varies significantly Stamelos et al. (2002); Borges and Valente
(2018); Aghajani et al. (2020); Wolf et al. (2020), and they are often
scattered, focusing on specific papers. These scattered repositories introduce
several obstacles to junior researchers or engineers who are new to the
research area. First, attempts to replicate an algorithm using different
implementations or configurations can result in inconsistent model
functionality or performance Paszke et al. (2019); Wolf et al. (2020); Lhoest
et al. (2021). Second, while many repositories focus on the model
architectures, they often neglect crucial steps such as detailed data pre-
processing, feature extraction, model training, and systematic evaluation.
This lack of systematic guidance poses substantial challenges for beginners
Georgiou et al. (2022), who may have limited technical expertise and
experience in training large-scale models. In summary, the scattered nature of
these repositories hampers efforts towards reproducible research and fair
comparisons among models or algorithms.

Figure 1: The north-star goal of the Amphion toolkit: Any to Audio.

Toolkit | Audio | Music/Singing | Speech | Visualization |
---|---|---|---|---|---
Amphion | ✓ | ✓ | ✓ | ✓ |
AudioCraft | ✓ | ✓ |  |  |
Bark | ✓ | ✓ | ✓ |  |
Coqui TTS |  |  | ✓ |  |
EmotiVoice |  |  | ✓ |  |
ESPnet |  | ✓ | ✓ |  |
Festival |  |  | ✓ |  |
HTS |  |  | ✓ |  |
Merlin |  |  | ✓ |  |
Mocking Bird |  |  | ✓ |  |
Muskits |  | ✓ |  |  |
Muzic |  | ✓ |  |  |
OpenVoice |  |  | ✓ |  |
PaddleSpeech |  | ✓ | ✓ |  |
SoftVC VITS |  | ✓ |  |  |
SpeechBrain |  |  | ✓ |  |
TANGO | ✓ | ✓ |  |  |
TorToiSe |  |  | ✓ |  |
WeTTS |  |  | ✓ |  |

Table 1: Representative open-source toolkits related to audio, music and
speech generation currently available in the open-source community (sort
alphabetically).

Motivated by that, we introduce Amphion, an open-source platform dedicated to
the north-star objective of “Any to Audio” (Figure 1). The initiative aims to
facilitate the conversion of any input into an audible audio signal. Compared
with the existing open-source toolkits as presented in Table 1, Amphion
integrates audio, music, and speech generation into a unified framework that
provides shared workflow across all models, including dataset processing,
feature extraction, model training, inference, evaluation, and open-source
pre-trained models. To aid junior engineers and researchers in understanding
the internal mechanism of generative models, Amphion provides visualizations
and interactive demonstrations of classic models. In a nutshell, Amphion, with
its foundation in education, holds four unique strengths that set it apart
from current open-source tools or repositories, in particular,

  * •

Unified Framework: Amphion provides a unified framework for audio, music, and
speech generation and evaluation. It is designed to be adaptable, flexible,
and scalable, supporting the integration of new models.

  * •

Beginner-friendly End-to-End Workflow: Amphion offers a beginner-friendly end-
to-end workflow with straightforward documentation and instructions. It
establishes itself as an accessible one-stop research platform suitable for
both novices and experienced researchers, including plenty of engineering
know-how.

  * •

Open Pre-trained Models: To promote reproducible research, Amphion commits to
strict standards to the release of pre-trained models. In partner with
industry, Amphion aims to make large-scale pre-trained models widely available
for various applications.

  * •

Visualization and Interactivity: Amphion provides visualization tools to
interactively illustrate the internal processing mechanism of classic models.
This provides an invaluable resource for educational purposes and for
facilitating understandable research.

The Amphion v0.1 toolkit, now available under the MIT
license,222https://github.com/open-mmlab/Amphion has supported a diverse array
of generation tasks. This paper presents a high-level overview of the Amphion
v0.1 toolkit.

##  2 Unified Framework

The north-star goal of Amphion is to unify various audible waveform generation
tasks. To make the framework adaptable and flexible to varied forms of tasks,
from the perspective of input, we formulate audio generation tasks into three
categories,

  1. 1.

Text to Waveform: The input consists of discrete textual tokens, which
strictly constrain the content of the output waveform. The representative
tasks include Text to Speech (TTS) Tan et al. (2021) and Singing Voice
Synthesis (SVS)333In addition to textual tokens, the input of SVS also
includes musical tokens such as MIDI notes Liu et al. (2022a). However, these
inputs in SVS are also discrete and strictly constrain the content (both
linguistic and musical content) like TTS. Therefore, we categorize SVS under
the Text to Waveform classification. Liu et al. (2022a).

  2. 2.

Descriptive Text to Waveform: The input consists of discrete textual tokens,
which generally guide the content or style of the output waveform. The
representative tasks include Text to Audio (TTA) Kreuk et al. (2022); Yang et
al. (2023) and Text to Music (TTM) Copet et al. (2023).

  3. 3.

Waveform to Waveform: Both the input and output are continuous waveform
signals. The representative tasks include Voice Conversion (VC) Sisman et al.
(2021), Singing Voic Conversion (SVC) Liu et al. (2021); Zhang et al. (2023),
Emotion Conversion (EC) Zhou et al. (2022), Accent Conversion (AC) Felps et
al. (2009), and Speech Translation (ST) Song et al. (2023a).

This section will introduce the system architecture design and multiple
classic generation tasks that are released in v0.1.

###  2.1 System Architecture Design

To achieve the north-star objective, the Amphion architecture is designed with
the following three principles,

  * •

To establish a fair comparison platform for a multitude of models. In
particular, Amphion provides consistent data preprocessing, unified underlying
training framework, and unified vocoders to create a fair experimental
environment for comparison.

  * •

To save practitioners from the tedious and repetitive task of “reinventing the
wheel". Amphion abstracts various classic components, such as feature
extraction, neural network modules, and batch sampling, into a unified
infrastructure.

  * •

To facilitate the unification of research across the entire audio domain. In
the context of today’s era of big data and large-scale pre-trained models, a
unified data preprocessing, model extraction, and training framework can
facilitate the development of the entire audio field.

Figure 2: System architecture design of Amphion.

In particular, the system architecture design is presented in Figure 2. From
the bottom up,

  1. 1.

We unify the data processing (Dataset, Feature Extractor, Sampler, and
DataLoader), the optimization algorithms (Optimizer, Scheduler, and Trainer),
and the common network modules (Module) as the infrastructure for all the
audio generation tasks.

  2. 2.

For each specific generation task, we unify its data/feature usage
(TaskLoader), task framework (TaskFramework), and training pipeline
(TaskTrainer).

  3. 3.

Under each generation task, for every specific model, we specify its
architecture (ModelArchitecture) and training pipeline (ModelTrainer).

  4. 4.

Finally, we provide a textitrecipe of each model for users. We unify the
recipe format across all models, ensuring it is as self-contained and novice-
friendly as possible. Besides, we provide the visualizations to demonstrate
the internal mechanisms of some typical models. On top of pre-trained models,
we also offer interactive demos for users to explore.

###  2.2 Classic Audio Generation Tasks

Amphion v0.1 toolkit includes a representative from each of the three
generation task categories (namely TTS, TTA, and SVC) for integration. This
ensures that Amphion’s framework can be conveniently adaptable to other audio
generation tasks during future development. Notably, most audio generation
models usually adopt a two-stage generation process, where they generate some
intermediate acoustic features (e.g. Mel Spectrogram) in the first stage, and
then generate the final audible waveform using a Vocoder in the second stage
(Figure 3(d)). Motivated by that, Amphion v0.1 also integrates a variety of
vocoder models.

(a) Text to Speech

(b) Text to Audio

(c) Singing Voice Conversion

(d) Vocoder

Figure 3: Pipelines of the classic audio generation tasks in Amphion.

Specifically, the pipelines of these classic audio tasks are designed as
follows:

  * •

Text to Speech: TTS aims to convert written text into spoken speech Tan et al.
(2021). The traditional TTS considers only textual tokens as input Wang et al.
(2017); Ren et al. (2020); Kim et al. (2021). Recently, zero-shot TTS attracts
more attentions from the research community. In addition to text, zero-shot
TTS requires a reference audio as a prompt (Figure 3(a)). By utilizing in-
context learning techniques, it can imitate the timbre and speaking style of
the reference audio Shen et al. (2024); Wang et al. (2023a).

  * •

Text to Audio: TTA aims to generate sounds that are semantically in line with
descriptions Kreuk et al. (2022); Yang et al. (2023). It usually requires a
pre-trained text encoder like T5 Kale and Rastogi (2020) to capture the global
information of the input descriptive text first, and then utilizes an acoustic
model, such as diffusion model Huang et al. (2023a); Liu et al. (2023a); Wang
et al. (2023b), to synthesize the acoustic features (Figure 3(b)).

  * •

Singing Voice Conversion: SVC aims to transform the voice of a singing signal
into the voice of a target singer while preserving the lyrics and melody Huang
et al. (2023b). To empower the reference speaker to sing the source audio, the
main idea is to extract the speaker-specific representations from the
reference, extract the speaker-agnostic representations (including semantic
and prosody features) from the source, and then synthesize the converted
features using acoustic models Zhang et al. (2023) (Figure 3(c)).

Current supported tasks, models and algorithms of Amphion v0.1 are presented
in Table 2.

Datasets | Tasks, Models and Algorithms | Evaluation Metrics
---|---|---
• Academic Datasets: $\circ$ AudioCaps Kim et al. (2019) $\circ$ CSD Choi et al. (2020) $\circ$ KiSing $\circ$ Libri-light Kahn et al. (2020) $\circ$ LibriTTS Zen et al. (2019) $\circ$ LJSpeech $\circ$ M4Singer Zhang et al. (2022) $\circ$ MLS Pratap et al. (2020) $\circ$ NUS-48E Duan et al. (2013) $\circ$ Opencpop Wang et al. (2022) $\circ$ OpenSinger Huang et al. (2021) $\circ$ Opera Black et al. (2014) $\circ$ PopBuTFY Liu et al. (2022b) $\circ$ PopCS Liu et al. (2022a) $\circ$ PJS Koguchi et al. (2020) $\circ$ SVCC Huang et al. (2023b) $\circ$ VCTK Yamagishi et al. (2019) Amphion also provides a unified data format for user custom dataset |  Text to Speech • Transformer-based: FastSpeech 2 Ren et al. (2020) • Flow-based: VITS Kim et al. (2021) • Diffusion-based: NaturalSpeech 2 Shen et al. (2024) • Autoregressive-based: VALL-E Wang et al. (2023a) Singing Voice Conversion • Features Extraction: $\circ$ Semantic Features: WeNet Yao et al. (2021), Whisper Radford et al. (2023), HuBERT Hsu et al. (2021), ContentVec Qian et al. (2022) $\circ$ Prosody Features: F0 extracted by Parselmouth Jadoul et al. (2018), Energy calculated based on STFT spectrograms Ren et al. (2020) • Acoustic Models: $\circ$ Diffusion-based: DiffWaveNetSVC Zhang et al. (2023), DiffComoSVC Song et al. (2023b); Lu et al. (2024) $\circ$ Flow-based: VitsSVC Kim et al. (2021); SVC-Develop-Team (2023) $\circ$ Transformer-based: TransformerSVC Vaswani et al. (2017) |  Text to Audio • Text Encoder: $\circ$ T5 Kale and Rastogi (2020) • Acoustic Models: $\circ$ Diffusion-based: AudioLDM Liu et al. (2023a); Wang et al. (2023b) Vocoder • Autoregressive-based: WaveNet van den Oord et al. (2016), WaveRNN Kalchbrenner et al. (2018) • Diffusion-based: DiffWave Kong et al. (2021) • Flow-based: WaveGlow Prenger et al. (2019) • GAN-based: $\circ$ Generators: MelGAN Kumar et al. (2019), HiFi-GAN Su et al. (2020), NSF-HiFiGAN Yoneyama et al. (2023), BigVGAN Lee et al. (2023), APNet Ai and Ling (2023) $\circ$ Discriminators: MSD Kumar et al. (2019), MPD Su et al. (2020), MRD Jang et al. (2020), MS-STFTD Défossez et al. (2022), MS-SB-CQTD Gu et al. (2024) |  • F0 Modeling: F0 Pearson Coefficients (FPC), F0 Periodicity Root Mean Square Error (Periodicity) Morrison et al. (2022), F0 Root Mean Square Error (F0RMSE), Voiced/Unvoiced F1 Score (V/UV F1). • Energy Modeling: Energy Root Mean Square Error (EnergyRMSE), Energy Pearson Coefficients (EnergyPC). • Spectrogram Distortion: PESQ Rix et al. (2001), STOI Taal et al. (2010), FAD Kilgour et al. (2019), MCD Kubichek (1993), SI-SNR Luo and Mesgarani (2018), SI-SDR Roux et al. (2019) • Intelligibility: Word Error Rate (WER) and Character Error Rate (CER) based on Whisper Radford et al. (2023) • Speaker Similarity: Cosine similarity based on RawNet3 Jung et al. (2022), Resemblyzer, and WavLM Chen et al. (2022).

Table 2: Supported datasets, models, and metrics of Amphion v0.1.

##  3 Beginner-Friendly End-to-End Workflow

Amphion provides a beginner-friendly, end-to-end workflow structured into five
stages: data preprocessing, feature extraction, model training, model
inference and evaluation, and visualization. Notably, Amphion workflow
incorporates plenty of engineering know-how for training models on large-scale
datasets such as Libri-light Kahn et al. (2020) and MLS Pratap et al. (2020).
This enables novice researchers to replicate advanced, proprietary systems.
Figure 4 presents the primary workflow of Amphion, outlined briefly as
follows,

Figure 4: An overview of Amphion workflow.

  * •

Data Preprocessing: Amphion offers pre-implemented, dataset-specific
preprocessing scripts for a wide range of academic datasets in audio, music,
and speech generation research (Table 2), enabling a seamless transition from
dataset-specific structures to Amphion’s unified data format.

  * •

Feature Extraction: After preprocessing, Amphion allows for both offline and
on-the-fly feature extraction. Offline extraction, which occurs before model
training, saves extracted features locally for later use in training. However,
for large-scale datasets, it is infeasible to perform the lengthy offline
extraction. Motivated by that, Amphion offers an on-the-fly extraction that
processes raw waveforms in batches and utilizes GPU as much as possible to
accelerate feature extraction during model training.

  * •

Model Training: Amphion enhances training efficiency by supporting multi-GPU
training with a Distributed Data Parallel (DDP) trainer Gugger et al. (2022),
dynamic batch sizes, and other user-friendly training infrastructures,
simplifying the replication of state-of-the-art models.

  * •

Model Inference and Evaluation: Amphion provides various pre-trained model
checkpoints of top-performing vocoders, one-click evaluation scripts, and
interactive demonstrations, enabling high-quality generation and easy
experimentation. The supported evaluation metrics can be seen in Table 2.

  * •

Visualization: Lastly, Amphion introduces a unique interactive visual analysis
of some classical models for educational purposes, helping newcomers
understand their inner workings.

##  4 Open Pre-trained Models

Pre-trained checkpoints can empower practitioners to reproduce experimental
results and build applications. Public pre-trained models could facilitate
innovation and learning for practitioners. For that purpose, Amphion has
released a variety of models pertaining to Audio, Music, and Speech
Generation, such as TTS, TTA, SVC, and Vocoder. Our standards for releasing
pre-trained models are designed to ensure transparency, reproducibility, and
usability for the community. Each model is released with a model card that
includes not only the checkpoint but also detailed descriptions of the
following criteria:

  * •

Model Metadata: Detail the model architecture and the number of parameters.

  * •

Training Datasets: List all the training corpus and their sources.

  * •

Training Configuration: Detail the training hyberparameters (like batch size,
learning rate, and number of training steps) and the computational platform.

  * •

Evaluation Results: Display the evaluation results and the performance
comparison to other typical baselines.

  * •

Usage Instructions: Instruct how to inference and fine-tune based on the pre-
trained model.

  * •

Interactive Demo: Provide an online interactive demo for users to explore.

  * •

License: Clear the licensing details including how the model can be utilized,
shared, and modified.

  * •

Ethical Considerations: Address ethical considerations related to the model’s
application, focusing on privacy, consent, and bias, to encourage responsible
usage.

As of now, the released pre-trained models of Amphion v0.1 are listed in Table
3. We are partnering with industry for larger models on larger datasets.

Task | Application Scenario | Model Architecture | #Parameters | Training Datasets
---|---|---|---|---
TTS | Multi-speaker TTS |  VITS Kim et al. (2021) | 30M | Hi-Fi TTS Bakhturina et al. (2021)
Zero-shot TTS |  NaturalSpeech2 Shen et al. (2024) | 201M | Libri-light Kahn et al. (2020)
TTA |  Sound effects synthesis |  AudioLDM Wang et al. (2023b) | 710M | AudioCaps Kim et al. (2019)
SVC |  Vocalist singing conversion |  DiffWaveNetSVC Zhang et al. (2023) | 31M | Vocalist Dataset (private)
Vocoder | Speech synthesis |  HiFi-GAN Su et al. (2020) | 24M |  LJSpeech, VCTK Yamagishi et al. (2019), LibriTTS Zen et al. (2019)
Singing synthesis |  BigVGAN Lee et al. (2023) | 112M |  Opencpop Wang et al. (2022), M4Singer Zhang et al. (2022), PJS Koguchi et al. (2020), CSD Choi et al. (2020), OpenSinger Huang et al. (2021), PopBuTFy Liu et al. (2022b), Opera Black et al. (2014), NUS-48E Duan et al. (2013), KiSing, PopCS Liu et al. (2022a)

Table 3: Supported pre-trained models of Amphion v0.1.

##  5 Visualization and Interactivity

Aiming to serve as a platform for exploring and understanding generative
models with educational purposes, Amphion provides intuitive and engaging
visualization tools for classic models or algorithms.

Visualizations act as a bridge between abstract theoretical concepts and their
practical applications, enabling users to see the “how" and “why" behind model
outputs Hohman et al. (2018). By providing a visual representation of model
architectures, data flow, and parameter effects, Amphion demystifies the inner
workings of generative models, making the technology more approachable.
Amphion v0.1 has developed the visualization system for diffusion model under
SVC task, namely SingVisio444https://dsvc.openmmlab.org.cn/ Xue et al. (2024).
The interface of the interactive visualization is shown in the appendix. This
tool offers a dynamic and interactive visualization of the generation process
in diffusion models, illustrating the gradual denoising of the noisy spectrum
and its evolution into a clear spectrum that embodies the target singer’s
timbre.

Amphion also utilizes online platforms like Gradio555https://www.gradio.app/
to support a series of interactive demos. With these interactive demos, users
can input text or select predefined prompts, and the underlying models will
generate corresponding audio outputs instantly. The interactive interface
provides users with immediate feedback and allows them to modify input
parameters, select options, or make choices using drop-down menus, check-
boxes, or sliders, depending on the specific task and model being utilized. We
provide an interactive demo example in the appendix.

Amphion’s visualization capabilities stand out as one of its most unique
features, designed to demystify complex generative models for those at the
beginning of their research journey. By merging the capabilities of Gradio
with the visualization features of Amphion, the toolkit provides a seamless
and interactive experience for users to explore and comprehend the inner
workings of generative models. This combination of visualizations and
interactivity empowers researchers, engineers, and enthusiasts to experiment,
iterate, and gain insights into the complex world of audio generation.

##  6 Experiments

We compare the performance of Amphion v0.1 on three tasks, namely TTS, TTA and
SVC, and the vocoder with public open repositories.

We use both objective and subjective evaluations to evaluate different tasks.
The objective evaluation metrics are presented in Table 2. Regarding the Mean
Opinion Score (MOS) and the Similarity Mean Opinion Score (SMOS) tests,
listeners are required to grade ranging from 1 (“Bad”) to 5 (“Excellent”) and
from 1 ("Different speaker, sure") to 5 ("Same speaker, sure") individually
with an interval of 1 to evaluate the generated audio’s overall quality and
the similarity to the reference speaker; regarding the Naturalness and
Similarity tests, listeners are required to grade ranging from 1 (“Bad”) to 5
(“Excellent”) and from 1 ("Different speaker, sure") to 4 ("Same speaker,
sure") individually with an interval of 1 to evaluate the conversed singing
utterance’s naturalness and the similarity to the reference singer. All the
subjective evaluation score listed are within a 95% Confidence Interval.

###  6.1 Text to Speech

Systems | CER $\downarrow$ | WER $\downarrow$ | FAD $\downarrow$ | MOS $\uparrow$
---|---|---|---|---
Coqui TTS (VITS) | 0.06 | 0.12 | 0.54 | 3.69 $\pm$ 0.10
SpeechBrain (FastSpeech 2) | 0.06 | 0.11 | 1.71 | 3.54 $\pm$ 0.11
TorToiSe (Diffusion-based model) | 0.05 | 0.09 | 1.90 | 3.61 $\pm$ 0.11
ESPnet (VITS) | 0.07 | 0.11 | 1.28 | 3.57 $\pm$ 0.11
Amphion v0.1 (VITS) | 0.06 | 0.10 | 0.84 | 3.61 $\pm$ 0.10
Table 4: Evaluation results of classic TTS in Amphion v0.1.

Systems | Training Dataset | MOS$\uparrow$ | SMOS $\uparrow$
---|---|---|---
Ground Truth | - | 4.72 $\pm$ 0.10 | 4.15 $\pm$ 0.26
YourTTS | LibriTTS | 3.11 $\pm$ 0.24 | 3.03 $\pm$ 0.19
VALL-E | Self-collected | 3.88 $\pm$ 0.23 | 3.45 $\pm$ 0.27
Amphion v0.1 (NaturalSpeech2) | Libri-light | 3.56 $\pm$ 0.18 | 3.37 $\pm$ 0.19

Table 5: Evaluation results of zero-shot TTS in Amphion v0.1.

####  6.1.1 Results of Classic TTS

We evaluate the performance of Amphion v0.1 TTS following the evaluation
pipeline in Amphion v0.1. Specifically, we construct testing sets with 100
text transcriptions and then generate the corresponding speech using Amphion
v0.1 VITS and other four popular open-source speech synthesis toolkits,
including Coqui TTS666https://github.com/coqui-ai/TTS accessed in Dec 2023
(VITS is used in this system),
SpeechBrain777https://github.com/speechbrain/speechbrain accessed in Dec 2023
(FastSpeech 2 is used because FastSpeech 2 is the most advanced model in this
system), TorToiSe888https://github.com/neonbjb/tortoise-tts accessed in Dec
2023 (an autoregressive and diffusion-based model), and
ESPnet999https://github.com/espnet/espnet accessed in Dec 2023 (VITS is used
in this system). The evaluation results are shown in Table 4. The evaluation
results show that the performance of Amphion v0.1 VITS is comparable to
existing open-source systems.

Systems | Dataset | FAD $\downarrow$ | IS $\uparrow$ | KL $\downarrow$
---|---|---|---|---
Diffsound |  AudioSet AudioCaps | 47.68 | 4.01 | 2.52
AudioGen |  AudioSet AudioCaps | - | - | 2.09
Make-an-Audio |  AudioSet AudioCaps 13 others | - | - | 2.79
AudioLDM-L |  AudioCaps | 27.12 | 7.51 | 1.86
AudioLDM-L-Full |  AudioSet AudioCaps 2 others | 23.31 | 8.13 | 1.59
AUDIT |  AudioSet96 AudioCaps 2 others | 20.19 | 9.23 | 1.32
Amphion v0.1 (AudioLDM) | AudioCaps | 20.47 | 8.78 | 1.44

Table 6: Evaluation results of Text to Audio in Amphion v0.1. “# others”
indicates the use of additional # datasets from different sources. More
information is available in Huang et al. (2023a); Liu et al. (2023a).

####  6.1.2 Results of Zero-shot TTS

We also evaluate the speech quality and speaker similarity of Amphion v0.1
zero-shot TTS systems alongside other open-source zero-shot TTS models. We use
the samples from NaturalSpeech2 Shen et al. (2024) demo
page101010https://speechresearch.github.io/naturalspeech2 accessed in Jan 2024
as our test set. We compare our Amphion v0.1 NaturalSpeech2 with YourTTS
Casanova et al. (2022), and VALL-E Wang et al. (2023a) from the reproduced
VALL-E repository111111https://github.com/Plachtaa/VALL-E-X accessed in Jan
2024. The results are shown in Table 5. The evaluation results show that
Amphion v0.1 NatrualSpeech2 is significantly better than YourTTS in terms of
speech quality and speaker similarity. Compared with the open-source
reproduced VALL-E, we have comparable speaker similarity, and the difference
in speech quality may come from the difference in training data.

###  6.2 Text to Audio

Systems | Naturalness $\uparrow$ | Similarity $\uparrow$
---|---|---
Ground truth | 4.67 $\pm$ 0.18 | 3.17 $\pm$ 0.29
SoftVC VITS | 2.98 $\pm$ 0.31 | 2.74 $\pm$ 0.31
Amphion v0.1 (DiffWaveNetSVC) | 3.52 $\pm$ 0.26 | 2.95 $\pm$ 0.3

Table 7: Evaluation results of Singing Voice Conversion in Amphion v0.1. The
full scores of naturalness and similarity are 5 and 4 respectively.

Systems | PESQ $\uparrow$ | M-STFT $\downarrow$ | F0RMSE $\downarrow$ | FPC $\uparrow$
---|---|---|---|---
HiFi-GAN ( UNIVERSAL_V1) | 3.43 | 1.98 | 177 | 0.88
ESPnet (libritts_hifigan.v1) | 3.55 | 1.12 | 188 | 0.86
Amphion v0.1 (HiFi-GAN) | 3.55 | 1.09 | 188 | 0.88

Table 8: Objective evaluation results of Vocoder in Amphion v0.1.

We use inception score (IS), Fréchet Distance (FD), and Kullback–Leibler
Divergence (KL) to evaluate our text-to-audio generation model. FD measures
the fidelity between the generated samples and target samples. IS measures the
quality and diversity of the generated samples. KL measures the correlation
between output samples and the target samples. FD, IS, and KL are based on the
state-of-the-art audio classification model PANNs Kong et al. (2020). We use
the test set of AudioCaps as our test set. The evaluation results of Amphion
v0.1 TTA are shown in Table 6. The results demonstrate that our Amphion v0.1
AudioLDM system achieves similar results to state-of-the-art models.

###  6.3 Singing Voice Conversion

To evaluate the effectiveness of the SVC models of Amphion v0.1, we adopt the
in-domain evaluation task of the Singing Voice Conversion Challenge (SVCC)
2023121212SVCC 2023: http://vc-challenge.org/ Huang et al. (2023b).
Specifically, there are 48 evaluated singing utterances, with 24 from a male
and 24 from a female. Our task is to convert them into two target singers (one
male and one female). For the training data, we utilize five datasets:
Opencpop Wang et al. (2022), SVCC training data Huang et al. (2023b), VCTK
Yamagishi et al. (2019), OpenSinger Huang et al. (2021), and M4Singer Zhang et
al. (2022). There are 83.1 hours of speech and 87.2 hours of singing data in
total. We compare the DiffWaveNetSVC Zhang et al. (2023) of Amphion v0.1 with
SoftVC VITS SVC-Develop-Team (2023), which is one of the most popular open-
source project in the SVC area. The MOS evaluation results for Amphion v0.1
SVC are shown in Table 7. We can see the DiffWaveNetSVC model of Amphion v0.1
owns better performance in both naturalness and speaker similarity.

###  6.4 Vocoder

We compare the Amphion v0.1 Vocoder to the two widely used open-source HiFi-
GAN checkpoints. One is the UNIVERSAL_V1 from the official HiFi-GAN
repository131313https://github.com/jik876/hifi-gan accessed in Dec 2023; the
other is the libritts_hifigan.v1 used by the
ESPnet141414https://github.com/kan-bayashi/ParallelWaveGAN accessed in Dec
2023. All of the checkpoints are trained on around 600 hours of speech data.
The whole evaluation set and the test set of LibriTTS are used for evaluation,
with a total of 20,306 utterances. Objective evaluation is conducted with
M-STFT, PESQ, F0RMSE, and FPC. The results are illustrated in Table 8. With
the assistance of additional guidance from Time-Frequency Representation-based
Discriminators Gu et al. (2024), the Amphion v0.1 HiFi-GAN achieves superior
performance in spectrogram reconstruction and F0 modeling.

##  7 Conclusions

This paper presented Amphion, an open-source toolkit dedicated to audio,
music, and speech generation. Amphion’s primary objective is to facilitate
reproducible research and serve as a stepping stone for junior researchers and
engineers entering the field of audio, music, and speech generation. Besides
focusing on specific generation tasks, Amphion provides visualizations of
classic models or architectures to empower users with a deeper understanding
of the underlying processes.

Large-scale datasets and large-scale models have proven to achieve state-of-
the-art performance in various tasks. In the future, Amphion plans to release
a few large-scale dataset in the area of audio, music and speech generation
area. Also, we plan to partner with industry for large-scale and production-
oriented pre-trained models. Since the release of Amphion in Dec 2023, Amphion
has received more than 3, 500 stars on GitHub and received a significant
number of requests and feedback. We will actively work on Amphion to serve
junior researchers and engineers.

## Limitations

In its initial release, Amphion has certain limitations that call for further
exploration and enhancement. While Amphion v0.1 has successfully incorporated
Text to Speech, Text to Audio, Singing Voice Conversion, and a variety of
Vocoders, there remains a multitude of audio generation tasks yet to be
integrated. Moreover, the audio processing field is progressing at a rapid
pace, with new models and techniques being developed on a continuous basis.
Ensuring that Amphion stays up-to-date and seamlessly integrates these
innovations will be an ongoing challenge that needs to be addressed in future
iterations.

## Ethical Considerations

In providing an open-source toolkit for audio, music, and speech generation,
we strive to balance innovation with ethical responsibility. We acknowledge
the ethical implications associated with our model’s ability to generate a
wide array of audio signals. Such capabilities, while powerful, carry
potential risks of misuse, such as the production of misinformation, deepfake
audio, or harmful content. We fervently advocate for the responsible use of
our code and pre-trained models, emphasizing the imperative of adhering to
regulatory standards. Our commitment to ethical considerations is not merely
an afterthought, but rather a guiding principle in our pursuit of advancing
the field of audio signal generation.

## References

  * Aghajani et al. (2020) Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C. Shepherd. 2020.  Software documentation: The practitioners’ perspective.  In _ICSE_ , page 590–601.
  * Ai and Ling (2023) Yang Ai and Zhen-Hua Ling. 2023.  APNet: An all-frame-level neural vocoder incorporating direct prediction of amplitude and phase spectra.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 31:2145–2157.
  * Bakhturina et al. (2021) Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg, and Yang Zhang. 2021.  Hi-Fi Multi-Speaker English TTS Dataset.  _INTERSPEECH_ , pages 2776–2780.
  * Black et al. (2014) Dawn AA Black, Ma Li, and Mi Tian. 2014.  Automatic identification of emotional cues in chinese opera singing.  In _ICMPC_.
  * Borges and Valente (2018) Hudson Borges and Marco Tulio Valente. 2018.  What’s in a github star? understanding repository starring practices in a social coding platform.  _Journal of Systems and Software_ , 146:112–129.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  In _NIPS_ , pages 1877–1901.
  * Casanova et al. (2022) Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. 2022.  Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone.  In _ICML_ , pages 2709–2720. PMLR.
  * Chen et al. (2022) Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  _IEEE J. Sel. Top. Signal Process._ , 16(6):1505–1518.
  * Choi et al. (2020) Soonbeom Choi, Wonil Kim, Saebyul Park, Sangeon Yong, and Juhan Nam. 2020.  Children’s song dataset for singing voice research.  In _ISMIR_.
  * Copet et al. (2023) Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023.  Simple and controllable music generation.  In _NeurIPS_.
  * Défossez et al. (2022) Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022.  High fidelity neural audio compression.  _arXiv_ , abs/2210.13438.
  * Duan et al. (2013) Zhiyan Duan, Haotian Fang, Bo Li, Khe Chai Sim, and Ye Wang. 2013.  The NUS sung and spoken lyrics corpus: A quantitative comparison of singing and speech.  In _APSIPA_ , pages 1–9.
  * Felps et al. (2009) Daniel Felps, Heather Bortfeld, and Ricardo Gutierrez-Osuna. 2009.  Foreign accent conversion in computer assisted pronunciation training.  _Speech Commun._ , 51(10):920–932.
  * Georgiou et al. (2022) Stefanos Georgiou, Maria Kechagia, Tushar Sharma, Federica Sarro, and Ying Zou. 2022.  Green ai: Do deep learning frameworks have different costs?  In _ICSE_ , pages 1082–1094.
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144.
  * Gu et al. (2024) Yicheng Gu, Xueyao Zhang, Liumeng Xue, and Zhizheng Wu. 2024.  Multi-scale sub-band constant-q transform discriminator for high-fidelity vocoder.  In _ICASSP_.
  * Gugger et al. (2022) Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022.  Accelerate: Training and inference at scale made simple, efficient and adaptable.  https://github.com/huggingface/accelerate.
  * Hinton and Salakhutdinov (2006) G. E. Hinton and R. R. Salakhutdinov. 2006.  Reducing the dimensionality of data with neural networks.  _Science_ , 313(5786):504–507.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  In _NIPS_ , pages 6840–6851.
  * Hohman et al. (2018) Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018.  Visual analytics in deep learning: An interrogative survey for the next frontiers.  _IEEE Transactions on Visualization and Computer Graphics_ , 25(8):2674–2693.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Huang et al. (2021) Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. 2021.  Multi-singer: Fast multi-singer singing voice vocoder with A large-scale corpus.  In _ACM Multimedia_ , pages 3945–3954.
  * Huang et al. (2023a) Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. 2023a.  Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models.  In _ICML_ , volume 202, pages 13916–13932. PMLR.
  * Huang et al. (2023b) Wen-Chin Huang, Lester Phillip Violeta, Songxiang Liu, Jiatong Shi, Yusuke Yasuda, and Tomoki Toda. 2023b.  The singing voice conversion challenge 2023.  In _ASRU_. IEEE.
  * Jadoul et al. (2018) Yannick Jadoul, Bill Thompson, and Bart de Boer. 2018.  Introducing Parselmouth: A Python interface to Praat.  _Journal of Phonetics_ , 71:1–15.
  * Jang et al. (2020) Won Jang, Dan Lim, and Jaesam Yoon. 2020.  Universal MelGAN: A robust neural vocoder for high-fidelity waveform generation in multiple domains.  _arXiv_ , abs/2011.09631.
  * Jung et al. (2022) Jee-weon Jung, Youjin Kim, Hee-Soo Heo, Bong-Jin Lee, Youngki Kwon, and Joon Son Chung. 2022.  Pushing the limits of raw waveform speaker recognition.  In _INTERSPEECH_ , pages 2228–2232.
  * Kahn et al. (2020) Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, and Emmanuel Dupoux. 2020.  Libri-light: A benchmark for ASR with limited or no supervision.  In _ICASSP_ , pages 7669–7673. IEEE.
  * Kalchbrenner et al. (2018) Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aäron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. 2018.  Efficient neural audio synthesis.  In _ICML_ , volume 80, pages 2415–2424.
  * Kale and Rastogi (2020) Mihir Kale and Abhinav Rastogi. 2020.  Text-to-text pre-training for data-to-text tasks.  In _INLG_ , pages 97–102.
  * Kilgour et al. (2019) Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. 2019.  Fréchet Audio Distance: A reference-free metric for evaluating music enhancement algorithms.  In _INTERSPEECH_ , pages 2350–2354.
  * Kim et al. (2019) Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019.  AudioCaps: Generating captions for audios in the wild.  In _NAACL-HLT_ , pages 119–132.
  * Kim et al. (2021) Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.  In _ICML_ , volume 139, pages 5530–5540.
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014.  Auto-encoding variational bayes.  In _ICLR_.
  * Koguchi et al. (2020) Junya Koguchi, Shinnosuke Takamichi, and Masanori Morise. 2020.  PJS: phoneme-balanced japanese singing-voice corpus.  In _APSIPA_ , pages 487–491.
  * Kong et al. (2020) Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. 2020.  PANNs: Large-scale pretrained audio neural networks for audio pattern recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2880–2894.
  * Kong et al. (2021) Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. 2021.  DiffWave: A versatile diffusion model for audio synthesis.  In _ICLR_.
  * Kreuk et al. (2022) Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2022.  AudioGen: Textually guided audio generation.  In _ICLR_.
  * Kubichek (1993) Robert Kubichek. 1993.  Mel-cepstral distance measure for objective speech quality assessment.  In _Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing_ , volume 1, pages 125–128.
  * Kumar et al. (2019) Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C. Courville. 2019.  MelGAN: Generative adversarial networks for conditional waveform synthesis.  In _NeurIPS_ , pages 14881–14892.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015.  Deep learning.  _Nature_ , 521(7553):436–444.
  * Lee et al. (2023) Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. 2023.  Bigvgan: A universal neural vocoder with large-scale training.  In _ICLR_.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021.  Datasets: A community library for natural language processing.  In _EMNLP_ , pages 175–184.
  * Liu et al. (2023a) Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo P. Mandic, Wenwu Wang, and Mark D. Plumbley. 2023a.  Audioldm: Text-to-audio generation with latent diffusion models.  In _ICML_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 21450–21474. PMLR.
  * Liu et al. (2023b) Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. 2023b.  AudioLDM 2: Learning holistic audio generation with self-supervised pretraining.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Liu et al. (2022a) Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. 2022a.  Diffsinger: Singing voice synthesis via shallow diffusion mechanism.  In _AAAI_ , pages 11020–11028. AAAI Press.
  * Liu et al. (2022b) Jinglin Liu, Chengxi Li, Yi Ren, Zhiying Zhu, and Zhou Zhao. 2022b.  Learning the beauty in songs: Neural singing voice beautifier.  In _ACL (1)_ , pages 7970–7983. Association for Computational Linguistics.
  * Liu et al. (2021) Songxiang Liu, Yuewen Cao, Dan Su, and Helen Meng. 2021.  DiffSVC: A diffusion probabilistic model for singing voice conversion.  In _ASRU_ , pages 741–748.
  * Lu et al. (2024) Yiwen Lu, Zhen Ye, Wei Xue, Xu Tan, Qifeng Liu, and Yike Guo. 2024.  Comosvc: Consistency model-based singing voice conversion.  _arXiv preprint arXiv:2401.01792_.
  * Luo and Mesgarani (2018) Yi Luo and Nima Mesgarani. 2018.  TaSNet: Time-domain audio separation network for real-time, single-channel speech separation.  In _ICASSP_ , pages 696–700.
  * Morrison et al. (2022) Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron C. Courville, and Yoshua Bengio. 2022.  Chunked Autoregressive GAN for conditional waveform synthesis.  In _ICLR_.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _NIPS_.
  * Pratap et al. (2020) Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020.  MLS: A large-scale multilingual dataset for speech research.  In _INTERSPEECH_ , pages 2757–2761. ISCA.
  * Prenger et al. (2019) Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 2019.  Waveglow: A flow-based generative network for speech synthesis.  In _ICASSP_ , pages 3617–3621.
  * Qian et al. (2022) Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David D. Cox, Mark Hasegawa-Johnson, and Shiyu Chang. 2022.  ContentVec: An improved self-supervised speech representation by disentangling speakers.  In _ICML_ , volume 162, pages 18003–18017.
  * Radford et al. (2023) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023.  Robust speech recognition via large-scale weak supervision.  In _ICML_ , volume 202, pages 28492–28518.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Ren et al. (2020) Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020.  FastSpeech 2: Fast and high-quality end-to-end text to speech.  In _ICLR_.
  * Ren et al. (2019) Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019.  FastSpeech: Fast, robust and controllable text to speech.  In _NeurIPS_ , volume 32.
  * Rix et al. (2001) Antony W. Rix, John G. Beerends, Michael P. Hollier, and Andries P. Hekstra. 2001.  Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs.  In _ICASSP_ , pages 749–752.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , pages 10684–10695.
  * Roux et al. (2019) Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey. 2019.  SDR - half-baked or well done?  In _ICASSP_ , pages 626–630.
  * Shen et al. (2018) Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018.  Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions.  In _ICASSP_ , pages 4779–4783.
  * Shen et al. (2024) Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2024.  Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers.  In _ICLR_. OpenReview.net.
  * Sisman et al. (2021) Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li. 2021.  An overview of voice conversion and its challenges: From statistical modeling to deep learning.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:132–157.
  * Song et al. (2023a) Kun Song, Yi Ren, Yi Lei, Chunfeng Wang, Kun Wei, Lei Xie, Xiang Yin, and Zejun Ma. 2023a.  Styles2st: Zero-shot style transfer for direct speech-to-speech translation.  _arXiv preprint arXiv:2305.17732_.
  * Song et al. (2023b) Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023b.  Consistency models.  _arXiv preprint arXiv:2303.01469_.
  * Stamelos et al. (2002) Ioannis Stamelos, Lefteris Angelis, Apostolos Oikonomou, and Georgios L Bleris. 2002.  Code quality analysis in open source software development.  _Information Systems Journal_ , 12(1):43–60.
  * Su et al. (2020) Jiaqi Su, Zeyu Jin, and Adam Finkelstein. 2020.  Hifi-gan: High-fidelity denoising and dereverberation based on speech deep features in adversarial networks.  In _INTERSPEECH_ , pages 4506–4510.
  * SVC-Develop-Team (2023) SVC-Develop-Team. 2023.  SoftVC VITS Singing Voice Conversion.  https://github.com/svc-develop-team/so-vits-svc.
  * Taal et al. (2010) Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. 2010.  A short-time objective intelligibility measure for time-frequency weighted noisy speech.  In _ICASSP_ , pages 4214–4217. IEEE.
  * Tan et al. (2021) Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. 2021.  A survey on neural speech synthesis.  _arXiv preprint arXiv:2106.15561_.
  * van den Oord et al. (2016) Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. 2016.  WaveNet: A generative model for raw audio.  In _Speech Synthesis Workshop_ , page 125.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NIPS_ , pages 5998–6008.
  * Wang et al. (2023a) Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023a.  Neural codec language models are zero-shot text to speech synthesizers.  _arXiv preprint arXiv:2301.02111_.
  * Wang et al. (2022) Yu Wang, Xinsheng Wang, Pengcheng Zhu, Jie Wu, Hanzhao Li, Heyang Xue, Yongmao Zhang, Lei Xie, and Mengxiao Bi. 2022.  Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis.  In _INTERSPEECH_ , pages 4242–4246. ISCA.
  * Wang et al. (2023b) Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, and Sheng Zhao. 2023b.  AUDIT: Audio editing by following instructions with latent diffusion models.  In _NeurIPS_.
  * Wang et al. (2017) Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. 2017.  Tacotron: Towards end-to-end speech synthesis.  In _INTERSPEECH_ , pages 4006–4010. ISCA.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _EMNLP_ , pages 38–45.
  * Xue et al. (2024) Liumeng Xue, Chaoren Wang, Mingxuan Wang, Xueyao Zhang, Jun Han, and Zhizheng Wu. 2024.  Singvisio: Visual analytics of diffusion model for singing voice conversion.  _arXiv preprint 2402.12660_.
  * Yamagishi et al. (2019) Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. 2019.  CSTR VCTK Corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92).  _University of Edinburgh. The Centre for Speech Technology Research (CSTR)_.
  * Yang et al. (2023) Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. 2023.  Diffsound: Discrete diffusion model for text-to-sound generation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Yao et al. (2021) Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei. 2021.  WeNet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit.  In _INTERSPEECH_ , pages 4054–4058.
  * Yoneyama et al. (2023) Reo Yoneyama, Yi-Chiao Wu, and Tomoki Toda. 2023.  Source-filter hifi-gan: Fast and pitch controllable high-fidelity neural vocoder.  In _ICASSP_ , pages 1–5.
  * Zen et al. (2019) Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019.  LibriTTS: A corpus derived from librispeech for text-to-speech.  In _INTERSPEECH_ , pages 1526–1530.
  * Zhang et al. (2022) Lichao Zhang, Ruiqi Li, Shoutong Wang, Liqun Deng, Jinglin Liu, Yi Ren, Jinzheng He, Rongjie Huang, Jieming Zhu, Xiao Chen, and Zhou Zhao. 2022.  M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus.  In _NeurIPS_.
  * Zhang et al. (2023) Xueyao Zhang, Yicheng Gu, Haopeng Chen, Zihao Fang, Lexiao Zou, Liumeng Xue, and Zhizheng Wu. 2023.  Leveraging content-based features from multiple acoustic models for singing voice conversion.  In _Machine Learning for Audio Workshop, NeurIPS 2023_.
  * Zhou et al. (2022) Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. 2022.  Emotional voice conversion: Theory, databases and ESD.  _Speech Commun._ , 137:1–18.

##  Appendix A Visualization and Interactive Demo Cases in Amphion

###  A.1 Visualization for DiffWaveNetSVC

Amphion has provided a visualization tool called
SingVisio151515https://dsvc.openmmlab.org.cn/ Xue et al. (2024) to demonstrate
the internal mechanism of DiffWaveNetSVC Zhang et al. (2023). Specifically, it
can demystify the entire diffusion process by extracting and rendering the
hidden features of the diffusion model for singing voice conversion. We
present the screenshot of the visualization system in Figure 5.

Figure 5: Interactive visualization of diffusion model for singing voice
conversion.

Figure 6: Online interactive demo for singing voice conversion.

###  A.2 Interactive Demo for DiffWaveNetSVC

Amphion has supported various online interactive demos for users to
explore.161616https://huggingface.co/amphion For example, we pre-train a
DiffWaveNetSVC Zhang et al. (2023) model which can conduct the singing voice
conversion among fifteen vocalists (as is shown in Table 3) and also provide
the interactive
demo171717https://huggingface.co/spaces/amphion/singing_voice_conversion. The
screenshot of it is presented in Figure 6.