11institutetext: Princeton University, Princeton NJ 08544, USA
22institutetext: Springer Heidelberg, Tiergartenstr. 17, 69121 Heidelberg,
Germany 22email<EMAIL_ADDRESS>
http://www.springer.com/gp/computer-science/lncs 33institutetext: ABC
Institute, Rupert-Karls-University Heidelberg, Heidelberg, Germany
33email<EMAIL_ADDRESS>
# BACK UP Pyrus Base: An Open Source Python Framework for the RoboCup 2D
Soccer Simulation

First Author 11 0000-1111-2222-3333 Second Author 2233 1111-2222-3333-4444
Third Author 33 2222–3333-4444-5555

###### Abstract

Soccer, also known as football in some parts of the world, involves two teams
of eleven players whose objective is to score more goals than the opposing
team. To simulate this game and attract scientists from all over the world to
conduct research and participate in an annual computer-based soccer world cup,
Soccer Simulation 2D (SS2D) was one of the leagues initiated in the RoboCup
competition. In every SS2D game, two teams of 11 players and one coach connect
to the RoboCup Soccer Simulation Server and compete against each other. Over
the past few years, several C++ base codes have been employed to control
agents’ behavior and their communication with the server. Although C++ base
codes have laid the foundation for the SS2D, developing them requires an
advanced level of C++ programming. C++ language complexity is a limiting
disadvantage of C++ base codes for all users, especially for beginners. To
conquer the challenges of C++ base codes and provide a powerful baseline for
developing machine learning concepts, we introduce Pyrus, the first Python
base code for SS2D. Pyrus is developed to encourage researchers to efficiently
develop their ideas and integrate machine learning algorithms into their
teams. Pyrus base is open-source code, and it is publicly available under MIT
License on GitHub111https://github.com/Cyrus2D/Pyrus2D.

###### Keywords:

Soccer Simulation Machine Learning Python Base Code.

##  1 Introduction

Soccer is one of the most popular team sports in the world. The main purpose
of this game is to achieve more goals than the opposing team in a multi-
player, real-time, strategic, and partially observable game. Players must
manage different tactical and technical strategies in addition to cooperative
behavior [1, 2, 3]. Considering the challenging and exciting nature of soccer,
computer simulation of this game creates an interesting environment for
developing machine learning algorithms that can address problems such as
multi-agent learning [5].

On this matter, the World Cup Robot Soccer Initiative was established to
create a realistic environment similar to the real game of soccer, encouraging
researchers to employ robotics and artificial intelligence (AI) to solve a
wide range of problems[6]. The primary goal of this tournament is designing a
robotic team to compete against the best human team by 2050 [4]. It was during
IJCAI-97 when the first RoboCup competition was held, and three competition
tracks were offered: the real robot league, the software robot competition,
and the expert robot competition[7, 8]. The Soccer Simulation 2D league (SS2D)
presents a wide range of research challenges, including autonomous decision-
making, communication, coordination, tactical planning, collective behavior,
and behavior prediction of opponents[9, 10, 11, 12, 13, 14, 15, 16, 17].

In this league, the RoboCup Soccer Simulation Server (RCSSServer) is
responsible for executing and managing a 2D soccer game between two teams of
twelve autonomous software programs(agents). Agents receive relative and noisy
information about the environment, and based on their logic and algorithms,
they produce basic commands (such as dashing, turning, or kicking) to
influence the environment. A visual example of the game is shown in Figure 1.

Developing an agent program from scratch in SS2D is a challenging task that
requires resolving technical issues such as stable network communication,
synchronization, and world modelling. However, having a base code and sample
team of agents can accelerate progress and allow teams to concentrate on
developing effective multi-agent teamwork techniques. The base code is
responsible for handling the complexity of the game The base code handles the
game’s complexity, such as communicating with the server, modelling the server
world, and making multi-agent decisions. This operational base code is
essential for advancing research in multi-agent systems in the context of
soccer simulation[18, 19].

Several teams have released base codes for RoboCup soccer simulation,
including the ”CMUnited” team from Carnegie Mellon University (USA)[20, 21],
the ”UvA Trilearn” team from the University of Amsterdam (The
Netherlands)[22], the ”MarliK” team from the University of Guilan (Iran)[26],
the ”HELIOS” team from AIST Information Technology Research Institute (Japan),
the ”Wrighteagle” team from University of Science and Technology of China[23],
”Gliders2d” team from the University of Sydney (Australia)[24, 25], and
”Cyrus2d” team from the Dalhousie University (Canada)[17, 19]. Using these
base codes, teams have been able to focus on developing high-level strategies
and algorithms rather than dealing with low-level implementation issues. Among
the published base codes, the ”Helios” base code is the most referenced, and
the ”Cyrus2D” is the most powerful base code for SS2D players. It was
implemented based on ”Helios”, and ”Gliders2D” and significant features of the
CYRUS team222The championship of RoboCup 2021. All of the mentioned bases are
developed in C++.

333For simplicity , throughout this paper we will use the ”base” term instead
of base code.

C++ has been the primary language for developing base codes for RoboCup soccer
simulation, given its speed and efficiency in handling computationally
demanding tasks. Although C++ is a powerful language, it can be challenging
for beginners due to the difficulty of working with C++. C++ is a powerful but
complex low-level programming language that provides a high level of control
and flexibility to the programmer. However, this also means that it can be
more difficult to maintain and debug than higher-level languages that offer
more abstraction and automation. As a result, learning and mastering C++ can
be a time-consuming process, especially for beginners who are new to
programming.

In contrast, Python is a more user-friendly language that prioritizes
simplicity and readability. With its extensive community support, simpler
debugging capabilities, and easy-to-use syntax, Python’s popularity has
increased significantly in recent years.

Although Python may not be as fast as C++, it offers numerous libraries and
frameworks that can speed up development and reduce programming time. As a
result, many developers are now exploring Python as an alternative language
for creating soccer simulation agents.

In this regard, Half Field Offense (HFO) framework was developed to work as a
Python interface on the Helios C++ base [27]. This framework uses the ”Helios”
base to connect to the server, receive and store observations, and send
actions. Due to its open-source availability and Python base interface, the
”HFO” encourages machine learning researchers to work on the soccer simulation
2D server. however, to apply any changes to the base code, researchers must
work with C++, since it is tied to the ”Helios” agent base code. Additionally,
”HFO” is not compatible with the latest soccer simulation server version.

To break any dependence from C++ base codes, we have designed the first
complete Python SS2D agent baseline called PYRUS, which has been implemented
from scratch using this language. In this paper, we will explain the details
of the PyrusGeom library, the base code, and the Pyrus Log Analyzer.

Figure 1: Visualization of Soccer Simulation 2D League.

##  2 Pyrus Geom Library

Pyrus Geom Library is a Python library that simplifies two-dimensional
geometric calculations in Python for the Pyrus. Many geometrical objects are
implemented in this library, including Angle2D, Circle2D, ConvexHull, Line2D,
Matrix2D, Polygon2D, Ray2D, Rect2D, Region2D, Sector2D, Segment2D, Size2D,
Triangle2D, and Vector2D444Researchers can install and use this library using
Python Pip or https://github.com/Cyrus2D/PyrusGeom/. This library can be
adapted in other Python applications and domains.

##  3 Pyrus Base Code

Members of the CYRUS soccer simulation 2D team started Pyrus Project in 2019.
This SS2D sample team or base code has been implemented in Python for the
first time. In addition to supporting player v18, coach v18, and trainer v18
in server v18, we will update the project to support future versions of the
server. In this base code, players can support full state (observation without
noise) and normal observation mode in the synchronous and standard timer of
the server. While the structure of the base code is similar to ”Helios”, some
improvements have been made to simplify it for beginners. In the following
subsections, we will discuss C++ base codes, and explain the features,
advantages, challenges, and structure of the Pyrus base code.

###  3.1 C++ Base Code Pros and Cons

”Helios” base is the most popular base code among all C++ base codes in soccer
simulation 2D league. ”Gliders” and ”Cyrus2D” base codes have been implemented
based on the ”Helios” base. These base codes are compatible with the old
versions of the server, so they include more than 150k lines of code. These
frameworks’ common object-oriented programming (OOP) structures are a benefit
since they help to ensure code understandability and maintainability.

Various informative behavioral algorithms are also included in these
frameworks, which researchers can learn and expand more efficiently. Due to
the enormous quantity of code and the complicated C++ syntax (such as
pointers), comprehending, debugging, and implementing new algorithms is
difficult, especially for beginners. Also, the development of machine learning
algorithms imposes additional complication to the development of C++ base
codes. In recent years, Python has become the most commonly used language for
machine learning research.

###  3.2 Features

The Pyrus base code is implemented in Python 3.9. Due to Python’s syntax
simplicity, researchers can focus on the development of their ideas rather
than language complexity management. Also, debugging the implemented
algorithms is faster since Python is simpler than C++. Python is one of the
most popular languages for machine learning, and it comes with a huge variety
of well-implemented machine-learning libraries that researchers can use
directly in Pyrus.

Pyrus contains about 30k lines of code, but w.r.t the C++ bases such as
”Helios” or ”Cyrus2D”, it is easier to read and understand the framework and
its internal processing.

Besides supporting the Python logging framework, this code supports the debug-
client and file-logging of soccer window debugging systems. Using the Python
multi-process and shared memory framework, players and trainers can share
data. This feature is useful at the training phase of reinforcement learning
which requires passing the observations among agents, which simplifies the
training phase of reinforcement learning.

###  3.3 Challenges

Considering the performance of Python, Pyrus processing time takes longer than
the C++ base codes. We are planning to address this problem by compiling the
code with Cython. Additionally to enhance the processing speed of mathematical
calculations teams can employ the libraries that are supported by C/C++
backends such as NumPy[29].

###  3.4 Structure

The Pyrus structure is based on Object Oriented Programming techniques and it
is similar to the ”Helios” base code. The structure of Pyrus is shown in
Figure 2.

The details of several important classes are provided in the following.

####  3.4.1 SoccerAgent.

This abstract class is designed to be a foundation for other agent classes.
This class provides

  * •

methods for managing the connection between the server and an agent,

  * •

updating the state of the agent and the field, and

  * •

making decisions based on these observations.

Inherited classes include ”PlayerAgent,” ”CoachAgent,” and ”TrainerAgent.”

####  3.4.2 WorldModel.

Players’ observations are stored and modelled using this class. The
”WorldModel” class contains information about the ball, the player, and other
players. The class is also responsible for localizing objects based on the
player’s sensor and updating the object named ”InterceptTable.”

####  3.4.3 InterceptTable.

The class calculates the number of necessary cycles for players to intercept
the ball based on their locations and velocities by simulating their future
actions. This enables precise prediction of player movements.

Figure 2: UML of main Classes in Pyrus

###  3.5 Algorithm

In order to maintain the simplicity of the code, Pyrus does not support old or
deprecated features of the server, such as asynchronous see mode. In order to
complete one cycle, a Pyrus agent receives messages from the server, parses
them, updates variables, makes decisions, and sends commands to the server.
The simplified flowchart of the Pyrus is shown in Figure 3.

Figure 3: FlowChart of Pyrus

##  4 Conclusion and Future Works

Using these base codes, teams have been able to focus on developing high-level
strategies and algorithms rather than dealing with low-level implementation
issues. The Robocup has provided a platform for researchers to employ robotics
and artificial intelligence solutions to solve various problems. The Soccer
Simulation 2D league presents a variety of research challenges that require
multi-agent decision-making, communication, coordination, tactical planning,
and behavior prediction of opponents. In order to decrease the amount of time
spent dealing with low-level implementation issues in working with the Soccer
Simulation Server, some C++ clients and base code have been released. C++,
however, is not the most appropriate language for implementing deep learning
and reinforcement learning algorithms, and Python has become the most popular
language for machine learning research. In this paper, we presented the Pyrus
base code, the first Python sample team for the 2D Soccer Simulation League.
This base code has several advantages: simpler syntax, faster debugging, and
access to various machine-learning libraries. By utilizing Pyrus, more
researchers can use RoboCup Soccer Simulation as a stable environment for
machine learning research.

For the future of this research, we plan to enhance localization and decision-
making algorithms to help beginners. To facilitate the usage of our framework
for machine learning researchers, we intend to create a package to use Pyrus
agents and RoboCup Soccer Simulation Server as an environment in the OpenAI
gym toolkit[28]. Additionally, we plan to implement a Python monitor and log
analyzer software to improve the usability of Pyrus.

## References

  * [1] Bangsbo J, Peitersen B. Soccer systems and strategies. Human Kinetics; 2000.
  * [2] Pollard R, Reep C. Measuring the effectiveness of playing strategies at soccer. Journal of the Royal Statistical Society: Series D (The Statistician). 1997 Dec;46(4):541-50.
  * [3] Rein R, Memmert D. Big data and tactical analysis in elite soccer: future challenges and opportunities for sports science. SpringerPlus. 2016 Dec;5(1):1-3.
  * [4] Burkhard, H.D., Duhaut, D., Fujita, M., Lima, P., Murphy, R., Rojas, R.: The road to RoboCup 2050. IEEE Robotics Automation Magazine 9(2), 31–38 (Jun 2002)
  * [5] Liu, S., Lever, G., Wang, Z., Merel, J., Eslami, S.M., Hennes, D., Czarnecki, W.M., Tassa, Y., Omidshafiei, S., Abdolmaleki, A. and Siegel, N.Y., 2021. From motor control to team play in simulated humanoid football. arXiv preprint arXiv:2105.12196.
  * [6] Noda, I. and Matsubara, H., 1996, November. Soccer server and researches on multi-agent systems. In Proceedings of the IROS-96 Workshop on RoboCup (pp. 1-7).
  * [7] Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I. and Osawa, E., 1997, February. Robocup: The robot world cup initiative. In Proceedings of the first international conference on Autonomous agents (pp. 340-347).
  * [8] Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., Osawa, E. and Matsubara, H., 1997. RoboCup: A challenge problem for AI. AI magazine, 18(1), pp.73-73.
  * [9] Noda, I., Stone, P.: The RoboCup Soccer Server and CMUnited Clients: Implemented Infrastructure for MAS Research. Autonomous Agents and Multi-Agent Systems 7(1–2), 101–120 (July–September 2003)
  * [10] Riley, P., Stone, P., Veloso, M.: Layered disclosure: Revealing agents’ internals. In: Castelfranchi, C., Lesperance, Y. (eds.) Intelligent Agents VII. Agent Theories, Architectures, and Languages — 7th. International Workshop, ATAL-2000, Boston, MA, USA, July 7–9, 2000, Proceedings. Lecture Notes in Artificial Intelligence, Springer, Berlin, Berlin (2001)
  * [11] Stone, P., Riley, P., Veloso, M.: Defining and using ideal teammate and opponent models. In: Proc. of the 12th Annual Conf. on Innovative Applications of Artificial Intelligence (2000)
  * [12] Butler, M., Prokopenko, M., Howard, T.: Flexible synchronisation within RoboCup environment: A comparative analysis. In: RoboCup 2000: Robot Soccer World Cup IV. pp. 119–128. Springer, London, UK (2001)
  * [13] Reis, L.P., Lau, N., Oliveira, E.: Situation based strategic positioning for coordinating a team of homogeneous agents. In: Balancing Reactivity and Social Deliberation in Multi-Agent Systems, From RoboCup to Real-World Applications. pp. 175–197. Springer (2001)
  * [14] Prokopenko, M., Wang, P.: Disruptive Innovations in RoboCup 2D Soccer Simulation League: From Cyberoos’98 to Gliders2016. In: Behnke, S., Sheh, R., Sariel, S., Lee, D.D. (eds.) RoboCup 2016: Robot World Cup XX [Leipzig, Germany, June 30 - July 4, 2016]. Lecture Notes in Computer Science, vol. 9776, pp. 529–541. Springer (2017)
  * [15] Prokopenko, M., Wang, P., Marian, S., Bai, A., Li, X., Chen, X.: Robocup 2d soccer simulation league: Evaluation challenges. In: Akiyama, H., Obst, O., Sammut, C., Tonidandel, F. (eds.) RoboCup 2017: Robot World Cup XXI [Nagoya, Japan, July 27-31, 2017]. Lecture Notes in Computer Science, vol. 11175, pp. 325–337. Springer (2018)
  * [16] Prokopenko, M., Wang, P., Obst, O.: Gliders2015: Opponent avoidance with bio-inspired flocking behaviour. In: RoboCup 2015 Symposium and Competitions: Team Description Papers, Hefei, China, July 2015 (2015)
  * [17] Zare, N., Sayareh, A., Sarvmaili, M., Amini, O., Matwin, S., Soares, A.: Engineering Features to Improve Pass Prediction in 2D Soccer Simulation Games. In: RoboCup 2021: Robot World Cup XXIV, Springer (2021)
  * [18] Akiyama, H., Nakashima, T.: Helios base: An open source package for the robocup soccer 2d simulation. In Robot Soccer World Cup 2013 Jun 24 (pp. 528-535). Springer, Berlin, Heidelberg.
  * [19] Zare, N., Amini, O., Sayareh, A., Sarvmaili, M., Firouzkouhi, A., Rad, S.R., Matwin, S. and Soares, A., 2023. Cyrus2D Base: Source Code Base for RoboCup 2D Soccer Simulation League. In RoboCup 2022: Robot World Cup XXV (pp. 140-151). Cham: Springer International Publishing.
  * [20] Stone, P., Asada, M., Balch, T.R., Fujita, M., Kraetzschmar, G.K., Lund, H.H., Scerri, P., Tadokoro, S., Wyeth, G.: Overview of robocup-2000. In: Stone, P., Balch, T.R., Kraetzschmar, G.K. (eds.) RoboCup 2000: Robot Soccer World Cup IV. Lecture Notes in Computer Science, vol. 2019, pp. 1–28. Springer (2000)
  * [21] Stone, P., Riley, P., Veloso, M.: The CMUnited-99 champion simulator team. In: Veloso, M., Pagello, E., Kitano, H. (eds.) RoboCup-99: Robot Soccer World Cup III, Lecture Notes in Artificial Intelligence, vol. 1856, pp. 35–48. Springer Verlag, Berlin (2000)
  * [22] Kok, J.R., Vlassis, N., Groen, F.: UvA Trilearn 2003 team description. In: Polani, D., Browning, B., Bonarini, A., Yoshida, K. (eds.) Proceedings CD RoboCup 2003. Springer, Padua (2003)
  * [23] Bai A, Chen X, MacAlpine P, Urieli D, Barrett S, Stone P. Wrighteagle and ut austin villa: RoboCup 2011 simulation league champions. InRoboCup 2011: Robot Soccer World Cup XV 15 2012 (pp. 1-12). Springer Berlin Heidelberg.
  * [24] Prokopenko, M., Wang, P.: Gliders2d: Source Code Base for RoboCup 2D Soccer Simulation League. CoRR abs/1812.10202 (2018)
  * [25] Prokopenko, M. and Wang, P., 2019, July. Fractals2019: Combinatorial optimisation with dynamic constraint annealing. In Robot World Cup (pp. 616-630). Springer, Cham.
  * [26] Tavafi, A., Nozari, N., Vatani, R., Yousefi, M.R., Rahmatinia, S., Pirdir, P.: MarliK 2012 Soccer 2D Simulation Team Description Paper. In: RoboCup 2012 Symposium and Competitions: Team Description Papers, Mexico City, Mexico, June 2012 (2012)
  * [27] Hausknecht, M., Mupparaju, P., Subramanian, S., Kalyanakrishnan, S. and Stone, P., 2016, May. Half field offense: An environment for multiagent learning and ad hoc teamwork. In AAMAS Adaptive Learning Agents (ALA) Workshop (Vol. 3). sn.
  * [28] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J. and Zaremba, W., 2016. Openai gym. arXiv preprint arXiv:1606.01540.
  * [29] Harris, C.R., Millman, K.J., Van Der Walt, S.J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N.J. and Kern, R., 2020. Array programming with NumPy. Nature, 585(7825), pp.357-362.


# LE-SSL-MOS: Self-Supervised Learning MOS Prediction with Listener
Enhancement

###### Abstract

Recently, researchers have shown an increasing interest in automatically
predicting the subjective evaluation for speech synthesis systems. This
prediction is a challenging task, especially on the out-of-domain test set. In
this paper, we proposed a novel fusion model for MOS prediction that combines
supervised and unsupervised approaches. In the supervised aspect, we developed
an SSL-based predictor called LE-SSL-MOS. The LE-SSL-MOS utilizes pre-trained
self-supervised learning models and further improves prediction accuracy by
utilizing the opinion scores of each utterance in the listener enhancement
branch. In the unsupervised aspect, two steps are contained: we fine-tuned the
unit language model (ULM) using highly intelligible domain data to improve the
correlation of an unsupervised metric - SpeechLMScore. Another is that we
utilized ASR confidence as a new metric with the help of ensemble learning. To
our knowledge, this is the first architecture that fuses supervised and
unsupervised methods for MOS prediction. With these approaches, our
experimental results on the VoiceMOS Challenge 2023 show that LE-SSL-MOS
performs better than the baseline. Our fusion system achieved an absolute
improvement of 13% over LE-SSL-MOS on the noisy and enhanced speech track. Our
system ranked 1st and 2nd, respectively, in the French speech synthesis track
and the challenge’s noisy and enhanced speech track.

Index Terms—  Listener enhancement, MOS prediction, Self-supervised model,
Speech synthesis

##  1 Introduction

In speech synthesis, evaluating the quality of synthesized speech has relied
on subjective evaluations such as Mean Opinion Scores (MOS) by human
listeners. During an MOS listening test, listeners are asked to rate a speech
sample’s naturalness and quality based on their subjective impressions. The
resulting score is divided into five levels, representing poor, bad, average,
good, and excellent. Labeling datasets for synthesized speech can be very
expensive and time-consuming, as it often involves having many listeners
listen to many synthesized speech samples. Therefore, researchers have been
developing methods that can automatically evaluate the quality of speech
synthesis systems.

With the continuous development of deep neural network (DNN) techniques,
researchers have gradually converted the evaluation task into a way to learn
the mapping relationship between speech and its quality score through these
powerful techniques and achieved promising results. MOSNet, a model for
predicting speech quality scores based on convolutional neural networks (CNNs)
and bidirectional long short-term memory networks (BLSTMs), was proposed for
speech conversion tasks [1]. MBNet and LDNet were proposed to use each
listener’s opinion scores [2, 3]. With its high capability of representations
in speech, self-supervised learning (SSL) was also applied to MOS predictions
[4]. It was found that fine-tuning the MOS prediction with the wav2vec2.0
model had better model generalization and accuracy [5]. Similar results were
obtained using a multi-task learning structure with SSL model features [6]. In
response to the VoiceMOS Challenge 2022 [7], many innovative model
architectures based on pre-trained SSL models have been proposed, including
the Fusion model, UTMOS, DDOS, and ZevoMOS, which further improved the
accuracy of the MOS predictions [8, 9, 10, 11]. As acquiring training data is
a challenge, Maiti et al. proposed an unsupervised evaluation metric called
SpeechLMScore, which evaluates the quality of speech generation by calculating
the likelihood of a speech-language model [12]. A recent study on a dataset
with over 1 million ratings found that models trained with data from multiple
locales consistently outperformed the baseline for mono-locale data [13].

This paper proposes a novel fusion architecture that improves estimation
accuracy by utilizing supervised and unsupervised subsystems through ensemble
learning. In supervised subsystems, we employ a novel SSL-based MOS predictor,
LE-SSL-MOS, by leveraging the listener-enhanced branch to learn independent
opinion scores of listeners and then improving the prediction performance
using a shared encoder by multi-task learning. We utilized two unsupervised
metrics, finetuned-ULM SpeechLMScore, and ASR Confidence, in unsupervised
subsystems. The former evaluates the perplexity of speech through a fine-tuned
unit language model by the highly intelligible target domain data. The latter
estimates the machine’s opinion score by calculating the average probability
of recognized tokens.

##  2 Related Works

SSL speech models, such as wav2vec2.0, can learn meaningful abstract
representations of speech from a large corpus of unlabeled speech data.

The results of the VoiceMOS Challenge 2022 indicate that top-ranking teams in
the main track have all used methods to fine-tune SSL models, and it was
observed that SSL-based models showed overwhelming effectiveness for this task
[7].

Among the effective SSL-based approaches, a so-called SSL-MOS
111https://github.com/nii-yamagishilab/mos-finetune-ssl [14] is actively
utilized for MOS prediction and achieved good performances in the Challenge.

Fig. 1: Block diagram of SSL-MOS approach.

Therefore, we adopted the SSL-based approach for MOS prediction in this study
and took the SSL-MOS as our baseline system. The block diagram in Fig.1 shows
the structure of the SSL-MOS system. Here, We will give a brief description of
it.

Suppose we have a MOS dataset with $N$ speech samples, denoted as
$\boldsymbol{X}=\\{\boldsymbol{x_{1},x_{2},...,x_{i},...,x_{N}}\\}$, with $L$
listeners, each speech sample has been rated by $M$ listeners with $M$ opinion
scores. To use the average opinion score during training, for each speech
sample $\boldsymbol{x_{i}}$, we need to first calculate the average of $M$
independent opinion scores as the average opinion score $y_{i}$ for this
sample. During training, for each speech sample $\boldsymbol{x_{i}}$, we need
to encode the input speech into SSL representation vectors at the frame level
using an encoder $\mathcal{E}$ based on the SSL model. Then, we need to
convert the frame-level representation vectors to utterance-level vectors
through mean pooling $\mathcal{M}$, and finally map the utterance-level vector
to the predicted speech quality score $\hat{y_{i}}$ through a linear layer
$\mathcal{H}_{ssl}$ [5].

$\displaystyle\hat{y_{i}}$
$\displaystyle=\mathcal{H}_{ssl}(\mathcal{M}(\mathcal{E}(\boldsymbol{x_{i}})))$
(1)

The goal of SSL-MOS is to estimate the utterance-level MOS, so its objective
function can be defined as follows:

$\displaystyle\mathcal{L}_{ssl}$
$\displaystyle=\frac{1}{N}\sum^{N}_{i=1}|y_{i}-\hat{y_{i}}|$ (2)

##  3 Proposed Methods

In this section, we first present our proposed LE-SSL-MOS model. Next, we will
formulate the fusion model for track 3 of the VocieMOS Challenge 2023.
Finally, we present an improvement to the unsupervised metric SpeechLMScore by
fine-tuning a unit language model (ULM) and comparing it with SSL-based
methods in experiments.

###  3.1 Listener-Enhanced SSL-MOS

Fig. 2: Architecture of the LE-SSL-MOS model

From Section 2, we can see that SSL-MOS only utilizes the mean opinion score
from the MOS dataset, while the subjective scores provided by many listeners
are not fully utilized. Considering the diversity in subjective opinion
distribution among each listener, we believe that improving the performance of
the model by considering the opinion scores of each utterance for each
listener would be beneficial. In previous studies, to make full use of the
training data, LDNet (non-SSL-based method) uses the listener-dependent (LD)
model to capture each listener’s preferences [3].

Following it, we propose a Listener-Enhanced SSL-MOS(LE-SSL-MOS) structure to
realize that purpose, utilizing an additional branch to learn independent
opinion scores of listeners, then enhance the performance of the SSL-MOS
branch by using a shared encoder. its architecture shown in Fig. 2. This
multi-task MOS predictor uses a shared SSL encoder to enhance the speech
feature representation and two independent decoders, one for estimating MOS
and the other for estimating independent opinion scores of listeners.

Compared to Fig.1, Fig.2 adds a listener-enhanced branch. This branch predicts
the subjective opinion scores of input speech for each listener through
concatenating SSL features and the embedding vector of the listener’s ID.
Assume there are $M$ listeners who provide subjective scores of input speech
$\boldsymbol{x_{i}}$, which can be represented as
$\boldsymbol{S}=\\{s_{1},s_{2},...,s_{M}\\}$. The subjective opinion score
$S_{m}$ of listener $l_{m}$ for input speech $\boldsymbol{x_{i}}$ is given by
$S_{m}$. During training, input speech $\boldsymbol{x_{i}}$ is extracted by an
encoder (SSL model) $\mathcal{E}$ and a mean pooling $\mathcal{M}$ layer to
obtain utterance-level feature and the ID of listener $l_{m}$ is obtained by
an embedding module $\mathcal{B}$ to obtain its embedding vector. After
concatenating these two vectors, they are input into a linear layer
$\mathcal{H}_{le}$ to obtain the prediction of listener $l_{m}$’s opinion
score $\hat{y}_{i}^{m}$ for $\boldsymbol{x_{i}}$, as shown in Equ. (3).

$\displaystyle\hat{y}_{i}^{m}$
$\displaystyle=\mathcal{H}_{le}(Concat(\mathcal{M}(\mathcal{E}(\boldsymbol{x_{i}})),\mathcal{B}(l_{m})))$
(3)

The objective function for the Listener Enhancement(LE) branch can be defined
as follows:

$\displaystyle\mathcal{L}_{le}$
$\displaystyle=\frac{1}{MN}\sum^{N}_{i=1}\sum^{M}_{m=1}|y_{i}^{m}-\hat{y}_{i}^{m}|$
(4)

And the loss of LE-SSL-MOS can be written as:

$\displaystyle\mathcal{L}$
$\displaystyle=\alpha\mathcal{L}_{ssl}+\beta\mathcal{L}_{le}$ (5)

where $\alpha$ and $\beta$ are hyperparameters balancing the losses.

###  3.2 SpeechLMScore with fine-tuned ULM

SpeechLMscore was proposed as an extension metric for text generation to
speech generation [12]. The block diagram of the speechLMscore system is shown
in Fig.3. From Fig.3, we can see that the system consists of three parts:
encoder (CPC, HuBERT or wav2vec2) [15, 16, 17], quantizer (KMeans) and unit
language model (ULM). SpeechLMScore measures the average log probability of a
set of unit tokens.

Fig. 3: The block diagram of SpeechLMScore system.

In natural language tasks, we often fine-tune the LM to learn the data
distribution features of the target task, thereby improving its performance.
It is natural to extend the idea of fine-tuning text-language models to
speech-language models. Therefore, fine-tuning a unit language model(ULM) in
the SpeechLMScore system with a small amount of data from the domain can
further improve the correlation between SpeechLMScore and human subjective
scoring.

###  3.3 Fusion model

Based on the work of [8], different MOS predictors can capture different
information from training sets, and ensemble learning can effectively utilize
this information further to enhance the stability and accuracy of the system.
Recently, researchers have proposed many methods for self-supervised learning
of speech, such as MMS, Wav2Vec2.0, XLSR, and WavLM [17, 18, 19, 20]. In this
study, we selected multiple pretrained models, for example, $P$ models, from
these methods to serve as SSL models in our SSL-based methods (SSL-MOS and LE-
SSL-MOS). As a result, we can obtain $2P$ MOS predictors based on SSL-based
methods. [21] shows that the utterance-level metrics are more useful than
system-level metrics by analyzing their baseline system. Therefore, we
selected the top $Q$($Q<2P$) MOS predictors under utterance-level SRCC metric
as supervised subsystems for the fusion model.

Fig. 4: The system architecture of the fusion method

In the Deep Noise Suppression (DNS) Challenge [22], word accuracy (WAcc) is
considered to be correlated with human intelligibility scores. Additionally,
we also obtained similar conclusions through the analysis of dataset TMHINT-QI
[23], which was on a new Chinese speech dataset for speech intelligibility and
quality assessment, and we found that there is a medium correlation between
the average opinion score of noise-enhanced speech and recognition confidence
by an ASR system. We believe that the speech recognition confidence by the ASR
system will provide helpful information that differs from MOS prediction
models for noise-enhanced speech datasets. Therefore, we use multiple ASR
systems as unsupervised subsystems of the fusion model. The unsupervised
subsystems also contain finetuned-ULM SpeechLMScore described in section 3.2.

The system framework diagram of the fusion model is shown in Fig. 4. Assuming
that the number of ASR systems we used is $R$ and the number of SpeechLMScore
metrics is $S$. We will obtain our final fusion model by concatenating
$(Q+R+S)$ results as input feature and using a Linear layer as a fuser.

##  4 Experiments

Table 1: Results of SSL-based MOS predictors(SSL-MOS and LE-SSL-MOS) on the
validation set of distributed TMHINT-QI

SSL-based Arch | SSL models | Utterance-level | System-level
---|---|---|---
MSE | LCC | SRCC | KTAU | MSE | LCC | SRCC | KTAU
SSL-MOS | MMS Base | 0.959 | 0.529 | 0.507 | 0.380 | 0.891 | 0.618 | 0.590 | 0.448
MMS-1B FL102 | 1.034 | 0.475 | 0.459 | 0.341 | 0.955 | 0.550 | 0.505 | 0.391
Wav2Vec2.0 Base | 1.005 | 0.483 | 0.461 | 0.343 | 0.830 | 0.540 | 0.503 | 0.363
Wav2Vec2.0 Large | 1.004 | 0.504 | 0.484 | 0.361 | 0.620 | 0.682 | 0.655 | 0.491
Xlsr53 56k | 1.087 | 0.441 | 0.426 | 0.314 | 1.120 | 0.424 | 0.374 | 0.283
WavLM Base | 0.991 | 0.500 | 0.473 | 0.352 | 0.872 | 0.569 | 0.536 | 0.395
WavLM Base+ | 1.025 | 0.505 | 0.482 | 0.358 | 0.845 | 0.585 | 0.556 | 0.401
WavLM Large | 1.143 | 0.447 | 0.427 | 0.315 | 0.991 | 0.552 | 0.519 | 0.374
LE-SSL-MOS | MMS Base | 0.951 | 0.537 | 0.517 | 0.388 | 0.875 | 0.607 | 0.566 | 0.426
MMS-1B FL102 | 1.080 | 0.509 | 0.490 | 0.366 | 1.217 | 0.584 | 0.542 | 0.413
Wav2Vec2.0 Base | 1.026 | 0.497 | 0.473 | 0.353 | 0.967 | 0.579 | 0.551 | 0.408
Wav2Vec2.0 Large | 0.993 | 0.505 | 0.485 | 0.362 | 0.635 | 0.666 | 0.634 | 0.474
XLSR53 56k | 1.162 | 0.374 | 0.368 | 0.270 | 0.916 | 0.574 | 0.553 | 0.393
WavLM Base | 0.964 | 0.526 | 0.500 | 0.373 | 0.660 | 0.639 | 0.611 | 0.459
WavLM Base+ | 0.969 | 0.529 | 0.511 | 0.382 | 0.736 | 0.620 | 0.591 | 0.435
WavLM Large | 0.985 | 0.496 | 0.478 | 0.355 | 0.754 | 0.638 | 0.639 | 0.474

###  4.1 Evaluation criteria

The evaluation criteria include mean squared error (MSE), Linear Correlation
Coefficient (LCC), Spearman Rank Correlation Coefficient (SRCC), and Kendall
Tau Rank Correlation (KTAU) at the utterance level and system level [1]. MSE
measures the difference between the predicted and ground-truth scores, while
the other criteria evaluate the correlation.

###  4.2 Training phase

This section provides detailed information about the experiments and results
we conducted during our training phase. During the training phase, the
organizers of the VoiceMOS Challenge 2023 released a noise-enhanced speech
dataset called TMHINT-QI [23]. To ensure that the testing set contained
systems unseen in the training set, the organizers re-split the training set
(8201 utterances) and validation set (4736), with 36 systems from the
validation set not appearing in the training set. Before training, we
resampled all audio data to 16kHz and normalized the amplitude of speech by
using the toolkit sv56 [7].

####  4.2.1 SSL-based MOS Predictors

During training, we first selected 8 SSL models (MMS Base, MMS-1B FL102,
Wav2Vec2.0 Base, Wav2Vec2.0 Large, XLSR53 56k, WavLM Base, WavLM Base+ and
WavLM Large)222https://github.com/facebookresearch/fairseq to be used for two
SSL-based architectures (SSL-MOS and LE-SSL-MOS), and fine-tuned on the above
datasets [17, 18, 19, 20]. These models are trained for 1000 epochs, using
L1-loss, batch size of 4, learning rate of 1e-4, an SGD optimizer, and
stopping training early if the loss on the validation set does not decrease in
10 epochs. In addition, for the hyperparameters of LE-SSL-MOS, $\alpha$ and
$\beta$ are both equal to 1, and the dimension of the listener ID embedding
feature is equal to 128. During the prediction phase, we only keep the SSL-MOS
branch for LE-SSL-MOS. Table 1 shows the results of all SSL-based models.

From Table 1, we can find that under the same model architecture, MMS Base
provides a higher correlation than other models under the Utterance-level SRCC
metric. Meanwhile, when using the same SSL model, the LE-SSL-MOS architecture
outperforms the baseline significantly.

Table 2: Results of confidence score, SpeechLMScore, and fusion model on the
validation set of distributed TMHINT-QI. $Confidence$ represent ASR system.
$SpeechLMScore_{ft-ULM}$ represent finetuned ULM SpeechLMScore metric,
$FusionModel_{Q}$ represent mono SSL-based MOS predictors subsystems.
$FusionModel_{QR}$ add ASR subsystems based on $FusionModel_{Q}$.
$FusionModel_{QRS}$ add finetuned-ULM SpeechLMScore subsystems based on
$FusionModel_{QR}$.

Method type | Models | Utterance-level | System-level
---|---|---|---
MSE | LCC | SRCC | KTAU | MSE | LCC | SRCC | KTAU
Unsupervised metric | Confidence | - | 0.405 | 0.418 | 0.308 | - | 0.576 | 0.558 | 0.418
SpeechLMScoreft-ULM | - | 0.463 | 0.425 | 0.317 | - | 0.5837 | 0.563 | 0.424
Fusion model | Fusion ModelQ | 0.929 | 0.548 | 0.525 | 0.395 | 0.928 | 0.631 | 0.592 | 0.448
Fusion ModelQR | 0.925 | 0.550 | 0.529 | 0.397 | 0.911 | 0.632 | 0.595 | 0.448
Fusion ModelQRS | 1.010 | 0.540 | 0.514 | 0.384 | 0.987 | 0.662 | 0.639 | 0.472

Table 3: System-level Results of VoiceMOS Challenge 2023 Tracks(since two
isolated listener test results in track 1, we separately reported their
performance, track 1a and track 1b. $multi$ and $track3$ respectively using
the $multi$ dataset and the distributed TMHINT-QI dataset during training. The
bold and italicized parts are our final submission(T06) results in the
VoiceMOS Challenge 2023.)

Models | System-level SRCC | System-level MSE
---|---|---
Track 1a | Track 1b | Track 2 | Track 3 | Track 1a | Track 1b | Track 2 | Track 3
SSL-MOSmulti | _0.790_ | _0.750_ | _0.693_ | - | _0.253_ | _0.659_ | _1.631_ | -
LE-SSL-MOSmulti | 0.878 | 0.627 | 0.717 | - | 0.236 | 0.494 | 2.190 | -
LE-SSL-MOStrack3 | - | - | - | _0.750_ | - | - | - | _0.404_
Fusion-Modeltrack3 | - | - | - | 0.880 | - | - | - | 0.206

####  4.2.2 Fusion Model

We use the experimental results of section 4.2.1 to select the MOS predictor
subsystems for the fusion model. We selected the top 5($Q=5$) mono-models
(bold predictors in table 1) under utterance-level SRCC metric. For the ASR
subsystem($R=1$), we use the fine-tuning model MMS-1B:FL102 based on MMS
project [18] and compute the average log-probability of output tokens
predicted by the acoustic model as the confidence. For the SpeechLMScore
subsystem($S=1$), we select librispeech clean-100 and TMHINT-QI data with MOS
greater than four as the domain data. To compute SpeechLMScore, we used pre-
trained HuBERT BASE as an encoder to output the third layer features as speech
features and kMeans (200 clusters) as a quantizer, and a fine-tuned ULM by the
above domain data [24, 16].

During training, we obtain the prediction results of the training set and
validation set in each subsystem and concatenate them as input features for
the model fuser, a fully connected layer without bias. We trained the fuser
model for 1000 epochs, using MSE-Loss, batch size of 4, the learning rate of
1e-5, an RMSProp optimizer, and stopping training early if the loss on the
validation set does not decrease in 20 epochs.

Compared to the best mono-model in Table 1, the results of $FusionModel_{Q}$
show that utilizing ensemble learning from SSL-base MOS predictors can achieve
higher utterance-level correlation. Table 2 also indicates that the fusion
model with recognition confidence($FusionModel_{QR}$) can further improve
performance than only used MOS predictors. And the fusion model with all
subsystems($FusionModel_{QRS}$) provided the highest system-level SRCC.

###  4.3 Evaluation phase

This section will describe the experiment setups and results for the three
released tracks in VoiceMOS Challenge2023 using SSL-based approaches and a
fusion model.

As shown in Table 1, we selected MMS BASE as the SSL model in the SSL-based
evaluation predictor.

For track1 and track2, since the organizers did not provide training and
validation sets, we collected SOMOS and the main track data from the VoiceMOS
challenge 2022 and combined the training and validation sets from both
datasets as the training and validation set for these tasks [7, 25]. We named
this dataset as _multi_. Using the same training method and parameters in
section 4.2.1 and train data _multi_ , we obtained the final evaluation models
SSL-MOSmulti and LE-SSL-MOSmulti. For track3, we directly use the MOS
predictor trained in section 4.2, and select LE-SSL-MOS(with MMS BASE in
section 4.2.1) and the fusion model(Fusion ModelQRS in section 4.2.2) as our
final MOS predictor.

The system-level SRCC and system-level MSE of the challenge dataset are shown
in Table 3.

Our submission ranked 1st and 2nd, respectively, in track1a and track3 of the
challenge. The final fusion model achieved an absolute improvement of 13% over
mono LE-SSL-MOS on track 3.

##  5 Conclusions

In this work, we propose a novel MOS predictor, LE-SSL-MOS, which constructs
the predictor by utilizing the pre-trained speech self-supervised learning
model, the MOS of each utterance, and the independent opinion scores of the
listener. We will also use ensemble learning to fuse mono SSL-based MOS
predictors, ASR systems, and fine-tuned-ULM SpeechLMScore metric to further
improve the performance of the fusion system on noisy enhanced datasets. In
the future, we will further explore the possibility of achieving a general-
purpose MOS predictor by constructing a more extensive dataset and a robust
architecture.

## References

  * [1] Chen-Chou Lo, Szu-Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and H. Wang,  “Mosnet: Deep learning based objective assessment for voice conversion,”  ArXiv, vol. abs/1904.08352, 2019.
  * [2] Yichong Leng, Xu Tan, Sheng Zhao, Frank K. Soong, Xiang-Yang Li, and Tao Qin,  “Mbnet: Mos prediction for synthesized speech with mean-bias network,”  in Proc. IEEE-ICASSP 2021, pp. 391–395.
  * [3] Wen-Chin Huang, Erica Cooper, Junichi Yamagishi, and Tomoki Toda,  “Ldnet: Unified listener dependent modeling in MOS prediction for synthetic speech,”  in Proc. IEEE-ICASSP 2022, pp. 896–900.
  * [4] A. Oord, Y. Li, and O. Vinyals,  “Representation learning with contrastive predictive coding,”  in Proc. NeurIPS, 2018.
  * [5] Erica Cooper, Wen-Chin Huang, Tomoki Toda, and Junichi Yamagishi,  “Generalization ability of MOS prediction networks,”  in Proc. IEEE-ICASSP 2022, pp. 8442–8446.
  * [6] Xiaohai Tian, Kaiqi Fu, Shaojun Gao, Yiwei Gu, Kai Wang, Wei Li, and Zejun Ma,  “A transfer and multi-task learning based approach for MOS prediction,”  in Proc. Interspeech 2022, pp. 5438–5442.
  * [7] Wen-Chin Huang, Erica Cooper, Yu Tsao, Hsin-Min Wang, Tomoki Toda, and Junichi Yamagishi,  “The voicemos challenge 2022,”  in Proc. Interspeech 2022, pp. 4536–4540.
  * [8] Zhengdong Yang, Wangjin Zhou, Chenhui Chu, Sheng Li, Raj Dabre, Raphael Rubino, and Yi Zhao,  “Fusion of self-supervised learned models for MOS prediction,”  in Proc. Interspeech 2022, pp. 5443–5447.
  * [9] Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari,  “UTMOS: utokyo-sarulab system for voicemos challenge 2022,”  in Proc. Interspeech 2022, pp. 4521–4525.
  * [10] Wei-Cheng Tseng, Wei-Tsung Kao, and Hung-yi Lee,  “DDOS: A MOS prediction framework utilizing domain adaptive pre-training and distribution of opinion scores,”  in Proc. Interspeech 2022, pp. 4541–4545.
  * [11] Adriana Stan,  “The zevomos entry to voicemos challenge 2022,”  in Proc. Interspeech 2022, pp. 4516–4520.
  * [12] Soumi Maiti, Yifan Peng, Takaaki Saeki, and Shinji Watanabe,  “Speechlmscore: Evaluating speech generation using speech language model,”  CoRR, vol. abs/2212.04559, 2022.
  * [13] Thibault Sellam, Ankur Bapna, Joshua Camp, Diana Mackinnon, Ankur P. Parikh, and Jason Riesa,  “Squid: Measuring speech naturalness in many languages,”  in Proc. IEEE-ICASSP, 2023, pp. 1–5.
  * [14] Erica Cooper, Wen-Chin Huang, Tomoki Toda, and Junichi Yamagishi,  “Generalization ability of mos prediction networks,”  in Proc. IEEE-ICASSP, 2021, pp. 8442–8446.
  * [15] Morgane Rivière and Emmanuel Dupoux,  “Towards unsupervised learning of speech features in the wild,”  in Proc. IEEE Spoken Language Technology Workshop SLT, pp. 156–163.
  * [16] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. PP, pp. 1–1, 10 2021.
  * [17] Alexei Baevski, Henry Zhou, Abdel rahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  ArXiv, vol. abs/2006.11477, 2020.
  * [18] Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Mamdouh Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli,  “Scaling speech technology to 1, 000+ languages,”  ArXiv, vol. abs/2305.13516, 2023.
  * [19] Qiantong Xu, Alexei Baevski, and Michael Auli,  “Simple and effective zero-shot cross-lingual phoneme recognition,”  in Proc. Interspeech 2022, pp. 2113–2117.
  * [20] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Micheal Zeng, and Furu Wei,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, pp. 1505–1518, 2021.
  * [21] Michael Chinen, Jan Skoglund, Chandan K. A. Reddy, Alessandro Ragano, and Andrew Hines,  “Using rater and system metadata to explain variance in the voicemos challenge 2022 dataset,”  in Proc. Interspeech 2022, pp. 4531–4535.
  * [22] Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Hannes Gamper, Mehrsa Golestaneh, and Robert Aichner,  “Deep speech enhancement challenge at icassp 2023,”  in Proc. IEEE-ICASSP, 2023.
  * [23] Yu-Wen Chen and Yu Tsao,  “Inqss: a speech intelligibility and quality assessment model using a multi-task learning network,”  in Proc. Interspeech 2022, pp. 3088–3092.
  * [24] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli,  “fairseq: A fast, extensible toolkit for sequence modeling,”  in Proc. NAACL-HLT 2019, pp. 48–53.
  * [25] Georgia Maniati, Alexandra Vioni, Nikolaos Ellinas, Karolos Nikitaras, Konstantinos Klapsas, June Sig Sung, Gunu Jho, Aimilios Chalamandaris, and Pirros Tsiakoulis,  “SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis,”  in Proc. Interspeech 2022, pp. 2388–2392.