# Towards a more comprehensive open-source model for interdisciplinary smart
integrated energy systems ††thanks: Funding: This research was funded by the
I3-Programme of the Hamburg University of Technology (TUHH) in the project
‘CyEntEE’ and the Federal Ministry for Economic Affairs and Climate Action in
the project ‘EffiziEntEE’ under the project number 03EI1050A.

Béla Wiegel, Tom Steffen, Davood Babazadeh and Christian Becker  Hamburg
University of Technology
Institute of Electric Power and Energy Technology
Hamburg, Germany
Email: {bela.wiegel, tom.steffen, davood.babazadeh<EMAIL_ADDRESS>
###### Abstract

The energy transition has recently experienced a further acceleration. In
order to make the integration of renewable energies as cost-effective, secure
and sustainable as possible and to develop new paradigms for the energy
system, many energy system models have been developed in research in the past
to evaluate the solutions. While model identification and dissemination of
results are widely discussed in the literature, a detailed view of the
methodology is often missing. This paper addresses this topic and proposes a
methodology to build a comprehensive, publicly accessible database for
modeling a multi-modal integrated energy system. The focus hereby is dynamic
modeling of low- and medium-voltage grids consisting of prosumers, battery
storages, heat pumps and electric cars. In addition, a district heating
network is parameterized to match the electricity grid. Modelica and the
TransiEnt-Library serves as the modeling tool. The methodology for creating
the grid models is available via GitLab. A study case that uses the
methodology to analyze the congestion situation within a medium-voltage
distribution grid is presented.

###### Index Terms:

energy system modeling, integrated energy system, multi-modal, open-source,
database, Modelica

††publicationid: pubid: 979-8-3503-3682-5/23/$31.00 ©2023 IEEE

##  I Introduction

The transition towards a modern and carbon-free energy system accelerates
worldwide. While the shutdown of single centralized conventional power plants
and their substitution by wind and photovoltaic power plants is feasible in
the current system, the comprehensive enrollment of renewable energies
requires a paradigm change in construction and operation concepts of the
future energy system [1]. Integrated energy systems, also known as multi-modal
energy systems, couple the electricity, heat and gas grids together. This
holds the possibility of multiple energy provisioning paths, reinforcing the
system evolution. For the concept of Smart Integrated Energy Systems (SIES),
the complexity is additionally expanded by aspects of Information and
Communication Technology (ICT) investigated as a component necessary for good
functioning [2, 3].

In order to understand and support the holistic design of complex systems of
this kind – both in terms of structural and operational planning – modeling is
an important tool. Therefore, many different energy system models were
developed in the past, representing lots of concepts and use cases. For each
model to be developed, the research question must be posed in advance to
clearly design the requirements and create the model-specific methodology.

While research questions and gap identification is project-specific, the work
also consists of recurring tasks. Despite macroscopic differences, modeling
often uses similar approaches and algorithms to solve the research question.
This already starts with data collection and preparation, and continues with
model building and evaluation methods. So that recurring tasks in the
multitude of research projects do not have to be tackled in future projects
again, the open research initiative has formed in the research community to
make details of data collection and model-specific methodologies and
implementation freely available. In Germany, a association called the National
Research Data Infrastructure (nfdi) [4] with the sub-group _nfdi4energy_ has
founded with the aim of endorsing the research cycle and utilizing community
services [5]. Also the European EERAdata and EnerMaps projects address this
topic. In the ERIGrid projects, harmonizing scenario development and model
validation is promoted [6]. A main aspect in context of developing and
publishing open source methodologies is the FAIR principle: findability,
accessibility, interoperability and reusability. These principles are
facilitated in current research [7, 8], and especially challenges in
data111The importance of data is also emphasized during the annual global
_Love Data Week_. provision using open source tools in [9].

In this context, the aim of this work is to develop a comprehensive energy
system description in the form of a open source data and modeling framework
that is suitable for the goal of generating detailed grid models with their
node interconnecting energy conversion plants. In addition to modeling the
electric power grid, a methodology is proposed to create a District Heating
Network (DHN) suitably parameterized to existing benchmark models for the
electric power grid. While usually the dissemination of the modeling and
research question investigation is the main part, this paper acts one step
before, depicted in Fig. 1. Data and models are published and maintained via
GitLab [10]. The main contributions are:

  * •

Developing a comprehensive open-source database description for modeling of
integrated energy systems at distribution level

  * •

Creation of a DHN appropriately parameterized to existing benchmark models for
electricity grids

  * •

Tool for generation of models for dynamic simulation of integrated
distribution systems

Thus, the rest of the paper is organized as follows: section II gives an
overview of existing approaches in public available modeling tools for
integrated energy systems, section III provides the data and modeling
framework description and concepts for model creation which is applied in a
use case in section IV, and section V gives final remarks and an outlook.

Figure 1: Scope of this paper within the research cycle.

##  II State and challenges in open modeling of integrated energy systems

Over the past decades, a large variety of energy system models have been
created. Diverse requirements for the models in the complex environment of
energy system research have led to manifold approaches. The ontology of models
can be described with categories of spatial and temporal resolution, modeled
energy carriers, physical detail, market representation and much more [11].
Over the years, constantly updated review articles classified the models and
place them in context. Thereby, it becomes apparent that integrated energy
systems, which couple different energy and application sectors, are becoming
more and more important, especially in open source modeling [12].

In order to analyze the physical energy system, two main methods have been
established in current research: optimization and simulation [13].
Optimization is suitable for planning a system configuration or an operational
schedule that satisfies certain requirements. Objective functions with
different weightings of the energy policy triangle consisting of economic
efficiency, sustainability and reliability resp. resiliency are applied for
this purpose. Methodologically, mixed-integer linear programming is often used
here, or also dynamic programming [13]. Simulation aims to analyze time-
dependent and mostly coupled processes and their interactions. In the case of
quasi-stationary simulation, balance equations are used in which at each point
in time the system is considered steady state. If storage-related time-
dependent processes, e.g. in the form of an oscillation or transient processes
within or across sectors are to be investigated, dynamic simulation in which
transient processes are furthermore represented via numerical solution of
differential equation systems are to be chosen. Only this approach gives
insight in interactions between interconnected and controlled components. For
optimization and simulation, it is necessary to capture the power system in
the form of a mathematical model.

In the planning and analysis of future energy systems, it is necessary in a
preliminary step to design a system description in the form of energy system
scenarios. Scenarios represent possible development paths and form the basis
for the creation of the models used for simulation and optimization.
Especially in interdisciplinary collaboration, which has many touch points in
the planning phase, a common understanding of the energy system is of high
importance so that consistent and concurring models can be created. In this
context, multi-modality as a concept of the modern integrated energy system
and its analysis requires the formation of grid benchmark models that are
parameterized together with their energy conversion plants, such as heat
pumps, combined heat and power or electrolyzers in order to be able to analyze
their interactions during operation properly. In ERIGrid and ERIGrid 2.0, a
EU-wide transnational program with lab-access and knowledge sharing in context
of SIES, concepts for harmonizing energy system validation approaches based on
modeling and simulation are developed [6].

Benchmark models for the electricity sector are established and well known.
The SimBench project provides documented electricity grids and time series for
load and consumption on all relevant voltage levels [14], although some parts
of the methodology are not fully explained. The Cigré benchmark system is
another well established methodology and described in [15]. For the heat
sector, different models for DHN exist, e.g. in the _pandapipes_ library, for
investigation of different research tasks, but there is no open benchmark
model existing. This is accompanied by the lack of a benchmark system for the
multi-modal resp. integrated energy system, which is a severe lack in
literature. Taking this up, the following section describes a methodology to
create a multi-modal test system for multi-modal grids, especially on
distribution level.

##  III Open Database Description

The overall goal of this paper is to provide a freely available basis for
modeling of lower-level integrated energy systems. Although the explained
models are designed for dynamic simulation, it is also possible to use the
same database to build linear optimization models, which was successfully
tested. The special feature here is that the methodology as a whole is freely
available, especially the data basis and the models itself, and large parts of
the physical structure of the integrated energy system are covered. This is
achieved by using the freely available TransiEnt-Library [16], which contains
component models of electricity, gas and heat grids and their most important
coupling elements using differential-algebraic equation systems. The library
is implemented in the modeling language Modelica [17]. With the dynamic energy
system modeling, a bottom-up approach is chosen to gain a high degree of
detail and represent the physical behavior of all components. The following
described methodology takes this requirement as a basis.

###  III-A Scenario Generation

The overall methodology for creating bottom-up energy system models on living
quarter level is depicted in Fig. 2. To obtain an overarching system
description and prosumer parameterization, information on future energy
systems is gathered in the form of a collection of different scenarios at
first. Main sources for this are the TYNDP studies by european grid operators
[18], scientific studies [19] or results in form of exploratory scenarios of
the before mentioned optimization models, e.g. REMod [20]. The qualitative was
former published in [3]. After this, representative grid topologies for the
electricity grid are underlain since these are well established in research.
The SimBench [14] benchmark model is the first chosen, but open-access Cigré
or IEEE benchmark models are suitable as well. With a defined grid topology,
the building structure is designed. Assumptions from the scenarios together
with ISO standards – or in case of Germany DIN standards – are used as a basis
for the parameterization of variables relevant for the model. Important
variables here are the demand for electrical energy, the ground area of the
building, thermodynamic properties such as heat transfer coefficients and
their area fractions, and the resulting Nominal Heat Load (NHL) for the
reference ambient temperature.

Figure 2: Methodology to create integrated test models. The scenario
description is used to gain qualitative and – where accessible – quantitative
information about the energy grid, which is quantified with all relevant
components afterwards.

With the defined building structure, a DHN with its topology is parameterized.
Since well established benchmark models for DHNs are scarce, a separate
methodology was build. The following steps describe the procedure roughly:
First, from different cycle-free grid topologies one is selected. Currently, a
topology based on the power grid or a minimum spanning forest with the vertex
spacing as weighting factor are available. Based on the NHL values of each
vertex and a comparatively low nominal supply temperature for modern DHNs in
the nominal design case, the maximum required mass flow per vertex is
calculated, which specifies the edge flows and thus the nominal width of the
pipes at a defined maximum flow velocity.

All remaining components, especially Battery Electric Storage (BES), Electric
Heat Pump (EHP), Battery Electric Vehicle (BEV), Photovoltaic Plant (PV) and
properties for sensors like the Smart Meter (SM) components are designed
afterwards. Fig. 3 gives an overview of the existing modules in the so called
Scenario Generator, which constructs the scenario based on the qualitative
scenario definition and different modules for the components of the modeled
energy system. Technically, every aspects is implemented in single Python
scripts, connected in a overall Scenario Generator script [10]. Scenario
studies are possible this way by analyzing the effect of input data in the
scenario description.

Figure 3: Scenario Generator: Overview of the modules contained in the
repository for database and model creation.

###  III-B Database description

During the design phase mentioned before, all processed data is written into a
database, called CyEntEE Database (CDB). In this case, a SQL database as back
end is used, but any other back end like other database engines or file-based
storages like JSON- or XML-files can be implemented. Advantage of a SQL
database is the possibility for using in a co-simulation environment in which
the modeling and simulation does not only take place on one single computer,
but concurrently on multiple simulation platforms that need the same system
information.

The modeling database described was created as part of a project in a way
that, in addition to modeling the physical energy system, also considers
aspects of ICT and decentralized markets [3] in form of a SIES. In order to
simulate the interactions between these different modeling entities, a co-
simulation platform, which is part of current investigation, is developed.

All components of the platform require data about the system configuration at
simulation time. For this case, a relational database management system
implementing SQL222In this case _MariaDB_ as open-source database management
system. was used as a back end to allow simultaneous read and write operations
across multiple workstations, and always provide fast data access.

In the case of the living quarter described in this context, the Entity
Relationship Model (ERM) is shown in Fig. 4. The tabular structure is shown
exemplary in Fig. 5.

Figure 4: Entity Relationship Model for the table structure describing one
living quarter. Each household, having a load profile, is connected by one
line and by zero or one pipe to a node in the grid model, which itself is
connected to a cell associated with a grid layer. Figure 5: Exemplary entity
relationship model for the relationship between the household, node and line
table. Details about the table structure are given in the code [10].

###  III-C Model Generation

As stated before, the database holds the possibility to export and combine the
scenario information in various different modeling environments, such as
pandapower, PyPSA, Matlab and various more. The simulation modeling approach
with highest detail supported are dynamic models, which is choosen in this
paper. Linear optimization models are also successfully implemented using the
CDB. This models, in contrast to quasi stationary or steady-state models,
captures within time based simulations the time dependence of the state
variables and their derivatives. The CDB is prepared for the generation of
those models. In order to make the conversion of the database contents into
dynamic models particularly easy, a python module was implemented within the
Scenario Generator, see Fig. 3. This module effortlessly export and combine
the scenario information within the CDB into a model within the modeling
language Modelica. Modelica is a language for modeling of cyber-physical
systems, which supports acausal connection of components described by
differential and algebraic equations [17].

This module includes the logic to extract the realistic load profiles, study-
based driving profiles, associated weather data as well as the topology
including the detailed household specification from the database. The
extracted information is then linked to existing and parameterizable Modelica
models from the CyEntEE Models package, which is planned to be released with
this publication within the TransiEnt Library [16].

The model with the most significance in this package is the Prosumer model,
which is the representation of future households with the capability to at
different points in time produce as well as consume power. The Prosumer is
described in higher detail in [21]. This model consists in full configuration
of six main components, a PV, BES, one or more BEVs, an EHP, inflexible load
and SM. Additionally, the module offers to choose which of the elements BEV
and BES on household level are externally controllable. In the model itself
this is realized by means of _RealInputs_ from the Modelica Standard Library,
which offer the external specification of target values while simulating.
Furthermore, one can choose to enable the SM models full potential meaning
normal distributed Gaussian noise with no systematic deviation, known mean and
known standard deviation, which is applied to the measurement outputs from the
SM model.

Fig. 6 and Fig. 7 show the Prosumer models composition as well as the example
parameter set for the configuration of household heating systems. The
individually configured Prosumers are then integrated in grid structures, as
mentioned in section III. The graphical representation of a simple 13 Prosumer
low voltage grid in Dymola, which is a commercial modeling and simulation
environment based on Modelica, is given in Fig. 8. The grid model is based on
the topology, meaning cable parameters, buses, household position and
transformer, given by the SimBench dataset. The Prosumer configuration is
obtained from the scenario generation, see section III-A, and is therefore,
based on the stochastic realization of the model, different for each Prosumer.

Figure 6: Structure of the Prosumer model from the CyEntEE Models package.
Figure 7: Example parameter configuration for Prosumer models household
heating system from the CyEntEE Models package. Figure 8: 13-Household rural
low voltage grid with topology based on SimBench LV-rural1–0-no_sw.

###  III-D Model preparation for Co-Simulation

According to the goal of the database to obtain common system descriptions
usable for different kinds of research studies the preparation of dynamic
models for Co-Simulation purposes was investigated and implemented. This is
achieved by means of the Functional Mock-up Interface (FMI) Standard [22].
“The Functional Mock-up Interface is a free standard that defines a container
and an interface to exchange dynamic simulation models using a combination of
XML files, binaries and C code, distributed as a ZIP file.” [23]. The
transformation from dynamic model within Modelica to the Functional Mock-up
Unit (FMU) is again achieved by a python module within the Scenario Generator,
see Fig. 3.

##  IV Use Case

To show the potential given by the database, a use case model in form of a
rural medium voltage ring based on the SimBench MV-rural-2-no-switches with
detailed rural low voltage sub grids, generated from the SimBench LV-rural-1
and LV-rural-2 was generated within Modelica. Fig. 9 shows the schematic
representation of this model, where the green grid represents the medium
voltage ring with its lines and buses. The dark blue points and lines
represent the buses within the underlying low voltage grids. They are
connected by the grey lines, which represent a transformer with on spatial
expansion. One bus, especially in the larger low voltage grid on the far left
side, can connect more than one Prosumer to the grid. The single red point
represents the interface, in form of an 630 MVA transformer to the high
voltage grid which is in this simulation modeled as boundary condition.

The grids specifications, meaning share of BEV, PV and other components, is
based on the _Distributed Energy Scenario_ from [3]. The weather data is taken
from the Deutsche Wetterdienst (DWD) for Hamelin, a city in Lower Saxony in
Germany, from the year 2020. The used load- and electric vehicle driving
profiles are study-based generated and selected.

When investigating the situation in this grid, by means of the bus voltages
given in Fig. 10, line loadings given in Fig. 11 and transformer loading given
in Fig. 12, within a 48-hour simulation based on the historical weather from
April 2020, it can be seen that even in an scenario with air temperature of
about 20°C and with high PV generation, future medium and low voltage grids
will likely face congestion situations on a daily bases. Especially in the
evening hours, between 5 and 10pm, recurrently increasing power increases can
be observed due to high share of BEVs.

Figure 9: Rural medium voltage ring based on the SimBench MV-rural-2-no-
switches with detailed rural low voltage sub cells, generated from the
SimBench LV-rural-1 and LV-rural-2. Blue points are prosumers at low voltage
level. Figure 10: Voltages in per unit at low and medium voltage buses in the
rural medium voltage ring scenario. Figure 11: Line loading in percent based
on the current-carrying capacity for low and medium voltage cables in the
rural medium voltage ring scenario. Figure 12: Transformer loading in percent
for transformers between the low and medium voltage level in the rural medium
voltage ring scenario.

##  V Conclusion and Outlook

In modern energy systems, different energy grids are coupled since renewable
energy arises mostly as electricity, and the other energy grids (heat, gas and
transportation) have to be supplied by the electricity sector, leading to the
concept of integrated resp. multi-modal energy systems. The purpose of this
paper is to describe a database and developed models to analyze the operation
of an integrated power system at living quarter level. Since research is
increasingly moving towards making data and models available, all developed
algorithms and modeling tools are made open source.

Public available data is used to create a qualitative scenario description of
the physical energy system, which then is converted to a quantitative scenario
description in different modules. The grid module parameterizes the energy
grids electricity and heat, and the prosumer module parameterizes the relevant
building energy components BES, EHP, BEV and PV, its thermodynamics and
measuring devices. Finally, the model generation module transfers the database
using the Modelica Language and TransiEnt-Library into a mathematical model.
In a use case as an exemplary scenario the congestion situation in a low- and
medium-voltage grid is analyzed.

Due to the modular structure and high level of detail of the model
description, users can build different scenarios and, for example, investigate
the penetration of renewable energies, interactions of different controller
structures or new operating strategies.

In the future, aim is to further expand the described scenario generator and
to specify and validate the created models within the framework of the ERIGrid
2 project.

## References

  * [1] S. S. Torbaghan, G. Suryanarayana, H. Hoschle, R. D'hulst, F. Geth, C. Caerts, and D. V. Hertem, “Optimal Flexibility Dispatch Problem Using Second-Order Cone Relaxation of AC Power Flows,” _IEEE Transactions on Power Systems_ , vol. 35, no. 1, pp. 98–108, jan 2020.
  * [2] D. Babazadeh, P. T. Baboli, M. Brand, C. Mayer, C. Becker, and S. Lehnhoff, “Resilience of smart integrated energy systems,” in _Handbook of Smart Energy Systems_. Springer International Publishing, 2022, pp. 1–27.
  * [3] K. Hoth, T. Steffen, B. Wiegel, A. Youssfi, D. Babazadeh, M. Venzke, C. Becker, K. Fischer, and V. Turau, “Holistic simulation approach for optimal operation of smart integrated energy systems under consideration of resilience, economics and sustainability,” _Infrastructures_ , vol. 6, no. 11, p. 150, oct 2021.
  * [4] N. Hartl, E. Wössner, and Y. Sure-Vetter, “Nationale forschungsdateninfrastruktur (NFDI),” _Informatik Spektrum_ , vol. 44, no. 5, pp. 370–373, aug 2021.
  * [5] A. Nieße, A. Weidlich, R. German, J. Lilliestam, C. Weinhardt, V. Hagenmeyer, S. Lehnhoff, A. Monti, B. Vogel, S. Auer, M. Richter, and S. Decker. (2023) National Research Data Infrastructure for the Interdisciplinary Energy System Research. [Online]. Available: https://nfdi4energy.uol.de/
  * [6] P. Raussi, K. Mäki, C. Evens, E. Grönroos, F. Pröstl Andrén, E. Widl, M. Calin, T. Strasser, E. Rikos, K. Heussen, O. Gehrke, T. V. Jensen, H. T. Nguyen, A. van der Meer, V. Subramaniam Rajkumar, A. Khavari, K. Anastasakis, A. Kontou, I. Orue, H. Taxt, S. D’Acro, J. E. Rodríguez Seco, J. Merino, A. Felipe Cortes, V. Efthymiou, V. Venizelou, M. Kynigos, and S. Theocharides, “D-NA4.1 Functional Scenarios,” ERIGrid, Tech. Rep., 2020.
  * [7] Y. Deng, K.-K. Cao, W. Hu, R. Stegen, K. von Krbek, R. Soria, P. R. R. Rochedo, and P. Jochem, “Harmonized and Open Energy Dataset for Modeling a Highly Renewable Brazilian Power System,” _Scientific Data_ , vol. 10, no. 1, feb 2023.
  * [8] D. Süsser, B. Pickering, L. Hülk, and S. Pfenninger, “Open energy system modelling to support the European Green Deal,” _F1000Research_ , vol. 11, p. 531, may 2022.
  * [9] S. Berendes, S. Hilpert, S. Günther, C. Muschner, S. Candas, K. Hainsch, J. van Ouwerkerk, S. Buchholz, and M. Söthe, “Evaluating the usability of open source frameworks in energy system modelling,” _Renewable and Sustainable Energy Reviews_ , vol. 159, p. 112174, may 2022.
  * [10] B. Wiegel, T. Steffen, and J. Rücker, “CyEntEE Models – Open Source Living Quarter Modeling Database,” https://collaborating.tuhh.de/cbs9803/CyEntEE_Models, 2023.
  * [11] M. Fodstad, P. C. del Granado, L. Hellemo, B. R. Knudsen, P. Pisciella, A. Silvast, C. Bordin, S. Schmidt, and J. Straus, “Next frontiers in energy system modelling: A review on challenges and the state of the art,” _Renewable and Sustainable Energy Reviews_ , vol. 160, p. 112246, may 2022\.
  * [12] J. van Ouwerkerk, K. Hainsch, S. Candas, C. Muschner, S. Buchholz, S. Günther, H. Huyskens, S. Berendes, K. Löffler, C. Bußar, F. Tardasti, L. von Köckritz, and R. Bramstoft, “Comparing open source power system models - a case study focusing on fundamental modeling parameters for the german energy transition,” _Renewable and Sustainable Energy Reviews_ , vol. 161, p. 112331, jun 2022.
  * [13] M. Groissböck, “Are open source energy system optimization tools mature enough for serious use?” _Renewable and Sustainable Energy Reviews_ , vol. 102, pp. 234–248, mar 2019.
  * [14] S. Meinecke, D. Sarajlić, S. R. Drauz, A. Klettke, L.-P. Lauven, C. Rehtanz, A. Moser, and M. Braun, “SimBench — A Benchmark Dataset of Electric Power Systems to Compare Innovative Solutions based on Power Flow Analysis,” _Energies_ , vol. 13, no. 12, p. 3290, Jun. 2020.
  * [15] Cigré, “Benchmark Systems for Network Integration of Renewable and Distributed Energy Resources,” _Cigré Task Force C6.04_ , 2014.
  * [16] A. Senkel, C. Bode, J.-P. Heckel, O. Schülting, G. Schmitz, C. Becker, and A. Kather, “Status of the transient library: Transient simulation of complex integrated energy systems,” _14th Modelica Conference 2021_ , pp. 187–196, 09 2021.
  * [17] M. A. et al. (2021) Modelica® and the modelica standard library. [Online]. Available: https://modelica.org/modelicalanguage.html
  * [18] O. Lebois, P. Boersma, A. Kättlitz, and M. C. Cavarretta, “TYNDP 2022. Scenarios Final Storyline Report,” ENTSOG, ENTSO-E, Tech. Rep., 2021. [Online]. Available: https://tyndp.entsoe.eu/
  * [19] J. DeAngelo, I. Azevedo, J. Bistline, L. Clarke, G. Luderer, E. Byers, and S. J. Davis, “Energy systems in scenarios at net-zero CO2 emissions,” _Nature Communications_ , vol. 12, no. 1, oct 2021.
  * [20] P. Sterchele, J. Brandes, J. Heilig, D. Wrede, C. Kost, T. Schlegl, A. Bett, and H.-M. Henning, “Paths to a Climate-neutral Energy System. The German Energy Transition in its Social Context,” Fraunhofer ISE, Tech. Rep., 2020.
  * [21] T. Steffen, B. Wiegel, D. Babazadeh, A. Youssfi, C. Becker, and V. Turau, “Generation of realistic smart meter data from prosumers for future energy system senarios,” _10th Conference on Sustainable Energy Supply and Energy Storage Systems, NEIS 2022_ , 2022.
  * [22] A. Junghanns, C. Gomes, C. Schulze, K. Schuch, P. R, M. Blaesken, I. Zacharias, A. Pillekeit, K. Wernersson, T. Sommer, C. Bertsch, T. Blochwitz, and M. Najafi, “The functional mock-up interface 3.0 - new features enabling new applications,” in _Proceedings of 14th Modelica Conference 2021, Linköping, Sweden, September 20-24, 2021_ , ser. Linköping Electronic Conference Proceedings. Linköping University Electronic Press, 2021, pp. 17–26.
  * [23] Modelica-Association. (2022) Functional mock-up interface specification. [Online]. Available: https://fmi-standard.org/docs/3.0/

  *[SIES]: Smart Integrated Energy Systems
  *[ICT]: Information and Communication Technology
  *[DHN]: District Heating Network
  *[NHL]: Nominal Heat Load
  *[DHNs]: District Heating Network
  *[BES]: Battery Electric Storage
  *[EHP]: Electric Heat Pump
  *[BEV]: Battery Electric Vehicle
  *[PV]: Photovoltaic Plant
  *[SM]: Smart Meter
  *[CDB]: CyEntEE Database
  *[ERM]: Entity Relationship Model
  *[BEVs]: Battery Electric Vehicle
  *[FMI]: Functional Mock-up Interface
  *[FMU]: Functional Mock-up Unit


# Accent Conversion in Text-To-Speech Using Multi-Level VAE and Adversarial
Training ††thanks: This project has received funding from SUTD Kickstarter
Initiative no. SKI 2021_04_06.

Jan Melechovsky1, Ambuj Mehrish1, Berrak Sisman2, and Dorien Herremans1
1Audio, Music, and AI Lab, Singapore University of Technology and Design,
Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>2Speech & Machine Learning Lab, The University
of Texas at Dallas, USA
<EMAIL_ADDRESS>
###### Abstract

With rapid globalization, the need to build inclusive and representative
speech technology cannot be overstated. Accent is an important aspect of
speech that needs to be taken into consideration while building inclusive
speech synthesizers. Inclusive speech technology aims to erase any biases
towards specific groups, such as people of certain accent. We note that state-
of-the-art Text-to-Speech (TTS) systems may currently not be suitable for all
people, regardless of their background, as they are designed to generate high-
quality voices without focusing on accent. In this paper, we propose a TTS
model that utilizes a Multi-Level Variational Autoencoder with adversarial
learning to address accented speech synthesis and conversion in TTS, with a
vision for more inclusive systems in the future. We evaluate the performance
through both objective metrics and subjective listening tests. The results
show an improvement in accent conversion ability compared to the baseline.

###### Index Terms:

Accent, Text-to-Speech, Accent Conversion, Multi-Level Variational Autoencoder

##  I Introduction

In a globalized world, we encounter speakers of various accents on a daily
basis, yet there is still a lack of representation of all these accents in
speech technology. Accent is an important attribute that influences mutual
intelligibility of a spoken dialogue. It refers to a peculiar way of speaking
a language that can be described on phonemic, phonetic, rhythmical, and
structural levels [1]. As part of one’s idiolect, accent forms a part of a
person’s identity. Given the importance of accent in speech synthesis,
paradoxically, there has not been much research done in this field when it
comes to TTS.

Figure 1: The proposed model architecture with D-Step (discriminator) and
G-Step (generator) illustrations.

The task of converting one’s accent into another is a domain of Accent
Conversion (AC), or its more specific variant – Foreign Accent Conversion
(FAC) [2, 3]. In FAC, the input audio of an L2 (second language) speaker is
converted to an output audio of the same speaker with an L1 (native) accent.
One of the main challenges in FAC is the disentanglement of speaker identity
and accent, which is crucial to achieving good conversion, but challenging
since the two overlap substantially. The main application area of FAC is
Computer-Assisted Pronunciation Training [4], which aims to help L2 speakers’
pronunciation to be closer to L1. Our work, however, puts emphasis on foreign
accents, as they are scarcely present in current speech synthesis systems. As
in FAC, we aim to keep the original speaker’s identity, while changing the
speaker’s accent into one of many other accents, which might in fact change
the person’s perceived identity accent-wise.

In controllable TTS, studies have focused on modelling speaker attributes such
as pitch, pace, and intonation, e.g. in [5] or [6], where Tacotron models are
enhanced with modules of Global Style Tokens (GST), and Gaussian Mixture
Variational Autoencoder, respectively. When it comes to modelling accent in
TTS specifically, one of the first papers is by Melechovsky et al. [7]. They
proposed a Tacotron2 model enhanced with a Multi-Level Variational Autoencoder
(MLVAE) to capture speaker characteristics and accent. This framework showed
promising results, but seemed to lack a bit in its accent conversion strength.
Since the change of accent is entirely dependent on the accent embeddings, we
hypothesize that putting more weight on these accent embeddings might
strengthen the accent conversion ability. In an FAC model by Wang et al. [2],
adversarial learning is adopted to wipe out speaker-related information from
the extracted speech recognition bottleneck features before passing them on to
the conversion model. Inspired by this, we believe adversarial loss can be
used to enhance the accent conversion ability of the MLVAE model.

In this paper, we propose an MLVAE-based accented TTS system with a two-step
training procedure based on adversarial loss. The purpose of adversarial
learning is to wipe out accent information from the speaker embeddings so as
to lessen the dominance of speaker identity during inference. This better
disentanglement of the speaker from the accent representation may result in a
potentially stronger accent conversion. The contributions of this paper are as
follows: 1) we propose a novel MLVAE-based Tacotron2 system for accented TTS
that utilizes adversarial learning; 2) the proposed method shows stronger
accent conversion ability than its predecessor; and 3) we discuss the
shortcomings of current accented TTS research and propose future directions.

The rest of the paper is organized as follows: In Section II, related work is
presented. Section III dives into our proposed method, and in Sections IV and
V we present the results, discuss their meaning, and touch on the limitations
of this work and research field.

##  II Related Work

Few studies have focused on capturing accents in TTS. In [6], a Gaussian
Mixture VAE was used to control speech attributes, including accent, though
this was not the primary focus of the model. Liu et al. [8] tackled accent
modification by adjusting phoneme energy, pitch, and duration through an
accent variance adaptor. Melechovsky et al. [7] specifically addressed
accented TTS using an MLVAE module to disentangle speaker and accent
information, producing robust, speaker-independent accent embeddings. The
model included an accent classifier to cluster these embeddings further, but
its impact was not demonstrated. However, these accent embeddings are not
forced to be utilized by the framework to reconstruct the audio, as sole
speaker embeddings are sufficient as they can still contain original accent
information, which results in a poor accent conversion performance, hence we
aim to improve the accent conversion ability through adversarial learning, as
proposed in the next section.

##  III Proposed Method

In this section, we formulate the accent conversion problem for TTS, with a
two-step training procedure based on adversarial loss as shown in Fig. 1. The
proposed approach is motivated by the fact that accent and speaker identity
are highly correlated attributes of speech, and the MLVAE model can, to a
large extent, learn both the attributes in one of the two designated
embeddings only. Therefore, we propose to utilize an accent classifier with
adversarial loss to increase disentanglement and minimize the accent
information present in the speaker representation to strengthen the model’s
accent conversion ability. We refer to our model as MLVAE-ADV.

Figure 2: Detailed view of the MLVAE encoder, $R_{o}$ is the output of the
Reference Encoder.

###  III-A MLVAE Encoder

The Multi-Level Variational Autoencoder (ML-VAE) architecture [9] is used for
encoding disentangled representations of a set of grouped observations based
on different accents, as illustrated in Fig. 2 with Tacotron2 [10] as the base
model. The accents are considered as a variation factor between different
speakers. To disentangle accent and speaker attributes in the latent
representation, two latent variables are used: $z_{s}$ and $z_{a}^{g}$, where
$z_{a}^{g}$ represents the factor of variation among the accents $a$,
superscript $g$ represents the grouping, and $s$ represents speaker. The
observations are assumed to be independent and identically distributed (i.i.d)
at the group level, and the average marginal log-likelihood over different
accents is defined as $\mathcal{A}$ as
$\frac{1}{|\mathcal{A}|}\sum_{a\in\mathcal{A}}\log p(X_{a}|\theta)$, where
$\theta$ are the network parameters.

The marginal log-likelihood $\log p(X_{a}|\theta)$ can be expressed in terms
of Kullback-Leibler (KL) divergence between the true posterior
$p(z_{a}^{g},z_{s}|X_{a};\theta)$ and the variational approximation
$q(z_{a}^{g},z_{s}|X_{a};\phi_{a})$. The group Evidence Lower Bound
$ELBO(a;\theta,\phi_{s},\phi_{a})$ is also defined.

$\begin{split}\log p(X_{a}|\theta)=&ELBO(a;\theta,\phi_{s},\phi_{a})+\\\
&KL(q(z_{a}^{g},z_{s}|X_{a};\phi_{a})||p(z_{a}^{g},z_{s}|X_{a};\theta))\\\
\geq&ELBO(a;\theta,\phi_{s},\phi_{a})\end{split}$ (1)

(a) Grouped accent embeddings $z_{a}$.

(b) Speaker embeddings with $\gamma$ too high.

(c) Speaker embeddings with optimal $\gamma$.

Figure 3: A t-SNE projection of speaker and accent embeddings from the MLVAE-
ADV model.

$ELBO$ is a lower bound on the marginal log-likelihood, and the KL divergence
is always positive. The variational parameters $\phi_{s}$ and $\phi_{a}$ are
utilized in this context. The traditional VAE learns the model parameters by
maximizing the average $ELBO$ over individual samples, while the ML-VAE
maximizes the group $ELBO$ using mini-batches of the group. We can observe the
effect of grouping on accent embeddings in Fig. 3(a). In this work, we
maximize the average group $ELBO$ to train the model parameters, as discussed
in [9]. The KL loss $\mathcal{L}_{kl}$ is computed as illustrated in Fig. 1,
and the reconstruction loss between the predicted mel spectrogram $\hat{X}$
and the ground truth mel spectrogram $X$ is computed as follows:

$L_{recon}=||\hat{X}-X||_{2}$ (2)

where $||.||_{2}$ denotes $L_{2}$ norm. During the $G$-Step (generator), the
model is trained to generate mel spectograms ($\hat{X}$) from phoneme
sequences ($P$) and the reference spectrograms ($X$), and all the model
parameters are updated. In the $D$-Step (discriminator), we update only the
accent classifier. The detailed training procedure is outlined in Algorithm 1.

Algorithm 1 Adversarial Training

1: %% $P$: Phoneme Embeddings %%

2: %% $\hat{y}$: accent labels %%

3: $\mathcal{G}:=$ {phoneme_encoder (pe), mel_decoder (md), reference_encoder
(re), MLVAE, accent_classifier (ac)}

4: $\mathcal{D}:=$ {reference_encoder (re), MLVAE, accent_classifier (ac)}

5: for epochs = $1$ to $N$ do

6: Sample $\\{X^{(i)}\\}^{m}_{i=1}\sim\mathcal{P}_{r}$ a batch from the real
data

7: require_grad: $\mathcal{G}\backslash\\{ac\\}$ $\rightarrow$ True

8: $G$-Step:

9: Compute the output $\hat{X^{i}}=\mathcal{G}(X^{i},P)$

10: Compute $\mathcal{L}_{kl}$, $\mathcal{L}_{adv}$, $\mathcal{L}_{rec}$

11: Update
$\theta^{(\mathcal{G})}\leftarrow\theta^{(\mathcal{G})}+\eta\frac{d\mathcal{L}\theta^{\mathcal{G}}}{d\theta^{\mathcal{G}}}$

12: require_grad: $\mathcal{G}$ $\rightarrow$ False,
$\mathcal{D}\backslash\\{ac\\}$ $\rightarrow$ False

13: $D$-Step:

14: Compute the output $\hat{y}=\mathcal{D}(X^{i})$

15: Compute $\mathcal{L}_{ce}$

16: Update
$\theta^{(\mathcal{D})}_{ac}\leftarrow\theta^{(\mathcal{D})}_{ac}+\eta\frac{d\mathcal{L}\theta^{\mathcal{D}}_{ac}}{d\theta^{\mathcal{D}}_{ac}}$

17: end for

To identify and disentangle accent-related information from $z_{s}$, we
utilize the adversarial accent classifier, as we incorporate an adversarial
loss function, defined as follows:

$L_{adv}=||e-\hat{p}^{a}||^{2}_{2}$ (3)

We define $e$ as a uniform distribution vector, where each element is equal to
$1/\lvert A\rvert$, $\lvert A\rvert$ being the total number of accents in the
set $\mathcal{A}$; and $\hat{p}^{a}$ represents the output probability of the
adversarial accent classifier. The total loss for training the model during
the $G$-Step can be computed as follows:

$\mathcal{L}_{G}=\mathcal{L}_{recon}+\beta\mathcal{L}_{kl}+\gamma\mathcal{L}_{adv}$
(4)

where $\beta$ and $\gamma$ are coefficients associated with the KL and
adversarial loss. Finally, $\mathcal{L}_{D}=\alpha\mathcal{L}_{ce}$ is used
during the $D$-Step, where $\alpha$ is the coefficient associated with cross
entropy loss. The tuning of hyperparameters $\alpha$, $\beta$ and, $\gamma$ is
further explained in Section IV-B.

##  IV Experiments and results

###  IV-A Dataset and Baselines

In our experiments, we used two datasets: L2Arctic and CMUArctic [11, 12].
L2Arctic includes 24 speakers (balanced by sex) with 6 accents (Arabic,
Chinese, Hindi, Korean, Spanish, and Vietnamese), each represented by 4
speakers. We added 4 American-accented speakers (BDL, CLB, RMS, and SLT) from
CMUArctic. Our dataset comprised 28 speakers, 7 accents, and approximately 31
hours of data. We used 10 unseen utterances per speaker for testing, 20 for
validation, and the rest for training.

For our first baseline, we chose GST [5] with an embedding size of $256$, $10$
tokens, and $8$ attention heads; combined with the same Tacotron2 model we use
in our proposed model. We note that GST is unsupervised and has not been
modified for accent conversion, thus we use it without converting accent for
non-conversion-related comparison. While it is possible to naively interpolate
between a source speaker and an average of target accent speakers in their
embedding space, it would produce speech with a new speaker identity. Our
second baseline is the MLVAE model from [7], without their originally proposed
accent classifier.

###  IV-B Experimental Setup and Inference

All models were trained with a batch size of 64 for 200k steps and ADAM
optimizer. Both MLVAE models had accent and speaker embedding sizes of 128.
The $\beta$ coefficient started at $10^{-6}$ and increased to $10^{-4}$
between 5k and 15k steps to aid training. The MLVAE-ADV model used $\alpha$ of
$10^{-1}$ for the accent classifier in D-step and $\gamma$ of $10^{-2}$
throughout training. We experimented with $\gamma$ coefficients from $10^{-4}$
to 0.5. Low $\gamma$ resulted in no accent conversion, while high $\gamma$
(e.g., $10^{-1}$) caused chaotic speaker embeddings (Fig. 3(b)) and
unintelligible output speech. Optimal speaker embeddings showed visible but
not isolated speaker clusters, reflecting partial identity subtraction
(accent) (Fig. 3(c)).

Figure 4: XAB accent and speaker similarity test results.

During inference, speaker and accent embeddings can be extracted from either
single or multiple reference audio files. In our study, we passed the complete
validation set through the model to extract and preserve the embeddings.
Subsequently, during inference, we load these embeddings and compute the
average representation of the selected speaker and the target accent
(reproducing the grouping operation). A pre-trained HiFiGAN vocoder is used to
process outputs into speech.

###  IV-C Objective Evaluation

To assess the effectiveness of reconstructing the mel spectrogram and the
intelligibility of the synthesized speech, we utilize mel-cepstral distortion
(MCD) [13] and word error rate (WER), respectively. The results are presented
in Table I. Our analysis indicates that the proposed MLVAE-ADV outperforms
MLVAE and GST in mel spectrogram reconstruction ability. However, with a WER
of $0.2124$, the performance of MLVAE-ADV falls short compared to $0.1668$ and
$0.1647$ for GST and MLVAE, respectively. We hypothesize that the superior
accent conversion capability of MLVAE-ADV could impact its lower WER compared
to other baselines.

###  IV-D Subjective Evaluation

We conducted listening experiments to further assess the performance of MLVAE-
ADV. A total of 15 participants joined the study and each listened to a total
of 77 samples. 111Audio samples and source code are available at amaai-
lab.github.io/Accented-TTS-MLVAE-ADV/.

In the first part of the test, listeners rated the voice quality of samples
from two baseline models, the proposed method, and the ground truth on a five-
point Likert Scale (1: bad, 2: poor, 3: fair, 4: good, 5: excellent). The
resulting mean Opinion Score (MOS) is shown in Table I. A paired t-test shows
a statistically significant difference in voice quality when comparing ground
truth to all models ($p<0.001$); MLVAE-ADV to GST ($p<0.01$); and MLVAE-ADV to
MLVAE ($p<0.001$).

In the second part, listeners rated accent-converted samples for speaker
similarity in an XAB test. They were presented with an original speaker sample
X, and samples from the MLVAE (A) and MLVAE-ADV (B) models. Listeners picked
which sample (A or B) retains more of the original speaker’s (X) identity. The
results in Fig. 4 show that the MLVAE model is preferred over MLVAE-ADV in
terms of speaker similarity, suggesting that the accent conversion in MLVAE is
weaker, thus preserving the original identity better.

Finally, we test for accent similarity in an XAB test. The listeners were
presented with the same set of accent-converted samples (A, B) as in the
previous test, but this time, we also presented them with a target accent
reference sample Y from one of the speakers of that accent. Listeners were to
determine whether A or B sounds more similar in terms of accent similarity to
the reference accent Y. In this test (Fig. 4), MLVAE-ADV outperforms the
baseline of MLVAE.

TABLE I: Objective Evaluation results (top part; the lower, the better) and Subjective evaluation results (bottom part) for voice quality, Mean Opinion Score with 95% Confidence Interval. Metric | GT | GST | MLVAE | MLVAE-ADV
---|---|---|---|---
MCD $\downarrow$ | - | 7.0576 | 7.0160 | 6.9422
WER $\downarrow$ | 0.1370 | 0.1647 | 0.1668 | 0.2124
MOS | 4.493 | 3.236 | 3.273 | 2.941
95% CI | 0.419 | 0.411 | 0.323 | 0.283

##  V Conclusion

We proposed a MLVAE-ADV framework, a Tacotron2-based MLVAE model with
adversarial training to for accented TTS to improve accent conversion.
Evaluations show increased accent conversion but some loss in speaker identity
and voice quality. Fine-tuning loss coefficients may mitigate this quality
decrease. Since accent is a key part of an individual’s idiolect, changing it
may alter perceived speaker identity. This trade-off could be due to the
limited dataset, which includes only 4 speakers per accent. This work offers
new directions for future research in accented TTS. Future studies should use
larger datasets and better balance accent conversion with speaker identity
preservation to advance inclusive speech technology.

## References

  * [1] J. C. Wells and J. C. Wells, _Accents of English: Volume 1_. Cambridge University Press, 1982.
  * [2] Z. Wang, W. Ge, X. Wang, S. Yang, W. Gan, H. Chen, H. Li, L. Xie, and X. Li, “Accent and speaker disentanglement in many-to-many voice conversion,” in _2021 12th Int. Symposium on Chinese Spoken Language Processing (ISCSLP)_. IEEE, 2021, pp. 1–5.
  * [3] S. Ding, G. Zhao, and R. Gutierrez-Osuna, “Accentron: Foreign accent conversion to arbitrary non-native speakers using zero-shot learning,” _Computer Speech & Language_, vol. 72, p. 101302, 2022.
  * [4] P. M. Rogerson-Revell, “Computer-assisted pronunciation training (capt): Current issues and future directions,” _RELC Journal_ , vol. 52, no. 1, pp. 189–205, 2021.
  * [5] Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,” in _Int. Conf. on Machine Learning_. PMLR, 2018, pp. 5180–5189.
  * [6] W.-N. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia, Z. Chen, J. Shen _et al._ , “Hierarchical generative modeling for controllable speech synthesis,” _arXiv:1810.07217_ , 2018.
  * [7] J. Melechovsky, A. Mehrish, D. Herremans, and B. Sisman, “Learning accent representation with multi-level vae towards controllable speech synthesis,” in _2022 IEEE SLT_. IEEE, 2023, pp. 928–935.
  * [8] R. Liu, B. Sisman, G. Gao, and H. Li, “Controllable accented text-to-speech synthesis with fine and coarse-grained intensity rendering,” _IEEE/ACM TASLP_ , 2024.
  * [9] D. Bouchacourt, R. Tomioka, and S. Nowozin, “Multi-level variational autoencoder: Learning disentangled representations from grouped observations,” in _Proc. of the AAAI_ , vol. 32, no. 1, 2018.
  * [10] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in _2018 IEEE ICASSP_. IEEE, 2018, pp. 4779–4783.
  * [11] G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev, J. Levis, and R. Gutierrez, “L2-arctic: A non-native english speech corpus.” in _INTERSPEECH_ , 2018, pp. 2783–2787.
  * [12] J. Kominek and A. W. Black, “The cmu arctic speech databases,” in _Fifth ISCA workshop on speech synthesis_ , 2004.
  * [13] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory,” _IEEE TASLP_ , vol. 15, no. 8, pp. 2222–2235, 2007.