# Thermodynamics of firms’ growth

Eduardo Zambrano<EMAIL_ADDRESS>Max-Planck-Institut für Physik komplexer
Systeme, Nöthnitzer Str. 38, 01187 Dresden, Germany  Alberto Hernando
<EMAIL_ADDRESS>SThAR, EPFL Innovation Park, Bâtiment C, 1015
Lausanne, Switzerland  Laboratory of Theoretical Physical Chemistry, Institut
des Sciences et Ingénierie Chimiques, École Polytechnique Fédérale de Lausanne
(EPFL), CH-1015 Lausanne, Switzerland  Aurelio Fernández-Bariviera  Department
of Business, Universitat Rovira i Virgili, Av. Universitat 1, 43204 Reus,
Spain  Ricardo Hernando  SThAR, EPFL Innovation Park, Bâtiment C, 1015
Lausanne, Switzerland  Angelo Plastino  National University of La Plata,
Physics Institute (IFLP-CCT-CONICET) C.C.737, 1900 La Plata, Argentina

###### Abstract

The distribution of firms’ growth and firms’ sizes is a topic under intense
scrutiny. In this paper we show that a thermodynamic model based on the
Maximum Entropy Principle, with dynamical prior information, can be
constructed that adequately describes the dynamics and distribution of firms’
growth. Our theoretical framework is tested against a comprehensive data-base
of Spanish firms, which covers to a very large extent Spain’s economic
activity with a total of $1\,155\,142$ firms evolving along a full decade. We
show that the empirical exponent of Pareto’s law, a rule often observed in the
rank distribution of large-size firms, is explained by the capacity of the
economic system for creating/destroying firms, and can be used to measure the
health of a capitalist-based economy. Indeed, our model predicts that when the
exponent is larger that 1, creation of firms is favored; when it is smaller
that 1, destruction of firms is favored instead; and when it equals 1
(matching Zipf’s law), the system is in a full macroeconomic equilibrium,
entailing “free” creation and/or destruction of firms. For medium and smaller
firm-sizes, the dynamical regime changes; the whole distribution can no longer
be fitted to a single simple analytic form and numerical prediction is
required. Our model constitutes the basis of a full predictive framework for
the economic evolution of an ensemble of firms that can be potentially used to
develop simulations and test hypothetical scenarios, as economic crisis or the
response to specific policy measures.

##  I Introduction

Many natural, social and economic phenomena follow power laws. It has been
previously ascertained that the distribution of incomes Pareto (1896), size of
cities Krugman (1995); Gabaix (1999); Bettencourt (2013); Hernando and
Plastino (2012a); Hernando  _et al._ (2013); Hernando and Plastino (2013);
Hernando, Hernando, and Plastino (2014); Hernando  _et al._ (2015), evolution
of human language Cancho and Solvé (2003), internet and genetic networks
Barabási and Albert (1999), and scientific publications and citations Katz
(1999); van Raan (2013), all follow power laws. Finding a complete theory for
describing these kind of systems seem an impractical task, given the huge
amount of degrees of freedom involved these social systems. Notwithstanding,
remarkable regularities were reported and studied, such as Zipf’s law Zipf
(1949); Newman (2005); Hernando  _et al._ (2009); Cristelli, Batty, and
Pietronero (2012), or the celebrated Gibrat’s Law of proportional growth
Gibrat (1931), which constitutes important milestones on the quest for a
unified framework that mathematically describe predictable tendencies.

Firm size distributions (FSD) are the outcome of the complex interaction among
several economic forces. Entry of new firms, growth rates, business
environment, government regulations, etc., may shape different FSD. The
underlying dynamics that drives the distribution of firms’ sizes is still an
issue under intense scrutiny. According to Gaffeo et al. Gaffeo  _et al._
(2012), there is an active debate going on among industrial organization’s
scholars, in which log-normal, Pareto, Weibull, or a mixture of them, compete
for the best-fitting distributions of FSD. One of the controversial issues is
the very definition of “size”, which can be measured by different proxies such
as annual sales, number of employees, total assets, etc.

The seminal contribution by Gibrat Gibrat (1931) initiated a research line
concerning the formal model that governs firms’ sizes and industry structure.
The introduction of a theoretical model that would underlie the industrial
demography could be of great help for authorities interested in maintaining
fair competence and/or antitrust policies.

Hart and Prais Hart and Prais (1956) find, using a database of large firms,
that average growth rates and sizes are independent variables. Quandt Quandt
(1966) states that Pareto’s distribution is often rejected when analyzing
industries sub-sectors. Other independent empirical studies, carried out by
Simon and Bonnini Simon and Bonnini (1958), Mansfield Mansfield (1962), and
Bottazzi and Secchi Bottazzi and Secchi (2006), among others, confirm that
firms’ growth rates are not related to firm size and that FSD follow a log-
normal distribution. Jacquemin and Cardon de Lichtbuer Jacquemin and de
Lichtbuer (1973) study the degree of firms and industry concentration in
British firms using Fortune’s 200 largest industrial companies outside the
United States, ranked according to sales. This study detects an increasing
degree of concentration.

Kwasnicki Kwasnicki (1998) affirms that skewed size distributions could be
found even in the absence of economies of scale, and that the shape of the
distribution is the outcome of innovation in firms. In particular, according
to his simulations, cost improving innovations generate Pareto-like skewed
distributions. This work also reconciles the finding by Ijiri and Simon Ijiri
and Simon (1977) about the concavity toward the origin of the log-log rank
size plot. Such concavity could be produced by the evolutionary forces and
innovation in the market. Jovanic Jovanic (1982) finds that rates of growth
for smaller firms are larger and more variable than those of bigger firms.
Similar results are found empirically for Dutch companies by Marsili Marsili
(2005). On contrary, Vining Vining (1976) had argued that the origin of the
concavity is the existence of decreasing returns to scale.

Segal and Spivak Segal and Spivak (1989) develop a theoretical model in which,
under the presence of bankruptcy costs, the rate of growth of small firms is
prone to be higher and more variable than that of larger firms. The same model
also predicts that, for the largest firms, the sequence of growth rates is
convergent satisfying Gibrat’s law, namely

$\dot{x}_{i}(t)=v_{i}(t)x_{i}(t),$ (1)

where $x_{i}(t)$ is the size of the $i$th firm at time $t$, $\dot{x}_{i}(t)$
its change in time, and $v_{i}(t)$ a size-independent growth rate. This model
is consistent with some previous empirical evidence, as that of Mansfield
Mansfield (1962).

Sutton Sutton (1997) has published a review of the literature on markets’
structure, highlighting the current challenges concerning FSD modelling.

During the 1990s, the interest in FSD experienced a revamp, with the
availability of new data-bases. A drawback of early studies was a biased
selection of firms. Typically, data comprise only publicly traded firms, i.e.,
the largest ones. In recent years, new, more comprehensive data sources became
available.

Stanley et al. Stanley  _et al._ (1995), use the Zipf-plot technique in order
to verify fittings of selected data for US manufacturing firms and find a non-
lognormal right tail. Shortly afterwards, Stanley et al. Stanley  _et al._
(1996) encounter that the distribution of growth rates has an exponential
form. Kattuman Kattuman (1996) studies intra-enterprise business size
distributions, finding also a skewed distribution. Axtell Axtell (2001), using
Census data for all US firms, encounters that the FSD is right skewed, giving
support for the workings of Pareto’s law. A similar finding is due to Cabral
and Mata Cabral and Mata (2003) for Portuguese manufacturing firms, although a
log-normal distribution underestimates the skewness of the distribution and is
not suitable for its lower tail. In this line, Fu et al. Fu  _et al._ (2005)
find that, for pharmaceutical firms in 21 countries, and for US publicly
traded firms, growth rates exhibit a central-portion distributed according to
a Laplace distribution, with power law tails. Palestrini Palestini (2007)
agrees with a power law distribution for firm sizes, although he models firm
growth as a Laplace distribution, that could change over business cycles.

According to Riccaboni et al. Riccaboni  _et al._ (2008), the simultaneous
study of firm sizes and growth presents an intrinsic difficulty, arising from
two facts: (i) the size distribution follows a Pareto Law and (ii) firms’
growth rate is independent of the firm’s size. This latter property is known
as the “Law of Proportionate Effect”. Growiec et al. Growiec  _et al._ (2008)
study firms’ growth and size distributions using firms’ business units as
units of measurements. This study reveals that the size of products follow a
log-normal distribution, whereas firm-sizes decay as a power law.

Gaffeo et al. Gaffeo  _et al._ (2012), using data from 38 European countries,
find that log mean and log variance size are linearly related at sectoral
levels, and that the strength of this relationship varies among countries. Di
Giovanni et al. di Giovanni, Levchenko, and Ranciere (2011) find that the
exponent of the power law for French exporting firms is lower than for non-
exporting firms, raising the argument of the influence of firm heterogeneity
in the industrial demography. Additionally, Gallegati and Palestrini Gallegati
and Palestrini (2010) and Segarra and Teruel Segarra and Teruel (2012), show
that sampling sizes influence the power-law distribution.

One can fairly assert that the concomitant literature has not yet reached a
consensus regarding what model could best fit empirical data. An overview of
several alternative models is detailed in Ref. de Wit (2005), and references
therein. As shown in the above literature review, previous attempts to model
growth and sizes of firms have not been entirely successful. In particular,
there is a dispute concerning the underlying stochastic process that steers
FSD. A possible solution in terms of agent-based model was proposed Farmer and
Foley (2009); these models are remarkable as descriptive tools, but they do
not furnish an overall panorama because are single-purpose models. Besides,
they are sensitive to the initial conditions, and, in some cases, their
outcome depends on the length of the simulation time.

The aim of this paper is twofold. First, to develop a thermodynamic-like
theoretical model, able to capture typical features of firms’ distributions.
We try to uncover the putative universal nature of FSD, which could be
characterized by general laws, independent of “microscopic” details. Secondly,
to validate our theoretical model using an extensive database of Spanish
manufacturing firms during a long time-period.

This paper contributes to the literature in several aspects. First, it
provides an explanation for the stochastic distribution of firms’ sizes. The
understanding of FSD is relevant for economic policy because it deals with
market concentration, and thus, with competition and antitrust policy measures
—for example, Naldi Naldi (2003) exhibits a relationship between Zipf’s law
and some concentration indices. Second, we apply our model to a large sample
of Spanish firms. Third, this work expands the literature on industrial
economics modelling.

The paper is organized as follows. First, we present the theoretical framework
and perform numerical experiments to validate our analytic approach. Then, we
show the empirical application to the Spanish firms. Finally, we draw some
discussions and conclusions of our work.

##  II Theoretical framework

Our framework is based on two fundamental hypothesis:

  1. 1.

a micro-economic dynamical hypothesis for individual firm growth; and

  2. 2.

using the maximum entropy principle, with dynamical prior information, for
describing macroeconomic equilibrium.

###  II.1 Microdynamics

For the micro-economical hypothesis, we assume Gibrat’s law of proportional
growth (Eq. (1)) as the main mechanism underlying firms’ size evolution. A
finite-size term, due to the central limit theorem Hernando  _et al._ (2013),
becomes dominant for medium and small sizes, being proportional to the square
root of the size. In addition to these two terms, we also assume that non-
proportional forces become eventually effective, being dominant for the
smallest sizes. Thus, our full dynamical equation is written as

$\dot{x}_{i}(t)=v_{1i}(t)|x_{i}(t)|+v_{1/2i}(t)|x_{i}(t)|^{1/2}+v_{0i}(t),$
(2)

where $v_{qi}(t)$ ($q=1$, $1/2$ and $0$) are independent growth rates. It is
expected that the growth rates are of a stochastic nature. Thus, a temperature
can be defined from their variance $T_{q}=\text{Var}[v_{q}]$. Accordingly,
assuming that the variation in the growth rates is much larger than the
variation in the observable $x$ —as done in Refs. Hernando and Plastino
(2012a); Hernando  _et al._ (2015) — the variance of the growth for several
realizations becomes

$\text{Var}[\dot{x}]=T_{1}|x|^{2}+T_{1/2}|x|+T_{0}.$ (3)

This equation defines three regimes, according to the size: small sizes
$x<T_{0}/T_{1/2}$; medium sizes $T_{0}/T_{1/2}<x<T_{1/2}/T_{1}$; and large
sizes $T_{1/2}/T_{1}<x$. Because of the existence of the non-proportional
term, $x$ is allowed to eventually take negative values. This defines an
additional set of temperatures for the negative domain, which in principle can
be independent of those at the positive one. Since all these temperatures can
be measured from the raw data, their properties can be empirically determined.
In Fig. 1 we show a conceptual sketch of the ensemble of firms evolving in
time as random walkers along the different regimes, with the corresponding
temperature and dynamics for each of them.

###  II.2 MaxEnt Principle

For an ensemble of firms following Eq. (1), we assume that dynamical
equilibrium is asymptotically reached when some macroscopic constraints are
obeyed. We cite the average total number of firms $N$, the typical wealthiness
of a given particular region, or any other objective observable. In view of
the success of an entropic procedure for describing equilibrium distributions
in other social systems (as, e.g., city population distributions), we take as
our macroeconomic hypothesis the Principle of Maximum Entropy (MaxEnt) with
dynamical prior information Hernando and Plastino (2012a); Hernando  _et al._
(2013); Hernando and Plastino (2012b); Hernando, Plastino, and Plastino
(2012), to predict the equilibrium density of the system. We focus our
analytic derivation on that particular regime that has received most attention
in the literature: the proportional growth-one: $T_{1/2}/T_{1}<x$ for the
largest sizes. According to Ref. Hernando and Plastino (2012a), the entropy of
a system following Gibrat’s law is measured in terms of the new dynamical
variable $u(t)=\log[x(t)/x_{c}]$ (where $x_{c}$ is some reference value, in
our case, the transition size $x_{c}=T_{1/2}/T_{1}$) which linearizes the
dynamical equation as $\dot{u}_{i}(t)=v_{1i}(t)$. Thus, we write the
macroscopic entropy for the system’s density distribution $\rho(u)$ for $N$
firms as

$S[\rho]=-\int du\rho(u)\log[\rho(u)/N],$ (4)

The equilibrium density is obtained by extremization of $S$ under the
empirical constraints Hernando and Plastino (2012b); Hernando, Plastino, and
Plastino (2012), such as the total number of firms, the minimum size of a
firms, among others. Lacking them, as sometimes happens in physics, we will
use a symmetry criterion Griffiths (1987): employ constraints that preserve a
symmetry of scale of $x(t)$, i.e. translation symmetry in $u(t)$. For this, we
define an _energy function_ , $\mathcal{E}[\rho]$, that depends on powers of
the dynamical variable $u$, namely

$\mathcal{E}[\rho]=\sum_{n}\lambda_{n}\int du\rho(u)(u-\langle
u\rangle)^{n}:=\sum_{n}\lambda_{n}m_{n},$ (5)

where $m_{n}$ are the central moments of $\rho$ and $\lambda_{n}$ the coupling
constants. The maximization problem is written as
$\delta_{\rho}(S[\rho]-\beta\mathcal{E}[\rho])=0$, where $\beta$ is a Lagrange
multiplier ($\beta\lambda_{n}$ become then the multipliers for each term), and
the general solution is of the form

$\rho(u)=N\exp\left[-1-\beta\sum_{n}\lambda_{n}(u-\langle
u\rangle)^{n}\right].$ (6)

The values of the multipliers are obtained by solving the system of Lagrange
equations, $m_{n}=\int du\rho(u)(u-\langle u\rangle)^{n}$ for the distribution
of Eq. (6).

###  II.3 Connection with Thermodynamics

We consider, for simplicity, the linear regime with only the first two moments
$n=0$ (a constraint on the average total number of firms $m_{0}=N$) and $n=1$
(a constraint on the mean value $\langle u\rangle$ written as $m_{1}=0$).
Since the equations are formally equivalent to those found in Thermodynamics,
and traditionally the multipliers associated with these constraints are Reif
(1965); Balian (2006) $\beta:=1/T_{1}$, $\lambda_{0}:=-\mu$,
$\lambda_{1}:=\lambda$, we have a thermodynamic potential

$\Omega=-T_{1}S-\mu N+\lambda U,$ (7)

where $U=\langle u-\langle u\rangle\rangle$. The variational problem becomes
$\delta_{\rho}\Omega[\rho]=0$. We obtain the distribution

$\rho(u)=Ne^{-(\lambda u-\mu)/T_{1}}.$ (8)

The distribution is cast in terms of the observable $x$ as

$\rho_{X}(x)dx=Ne^{\mu^{*}}\frac{x_{c}^{\lambda^{*}}}{x^{1+\lambda^{*}}}dx$
(9)

where $\mu^{*}=\mu/T_{1}$ and $\lambda^{*}=\lambda/T_{1}$. Accordingly, we
obtain a power law density. Useful for analyzing the empirical data is the
complementary of the cumulative distribution
$F(x)=\int^{x}_{x_{c}}dx^{\prime}\rho(x^{\prime})$, that reads

$N-P(x)=N\left(\frac{x_{c}}{x}\right)^{\lambda^{*}}.$ (10)

The solutions of the Lagrange equations lead to

$e^{-\mu/T_{1}}=\langle u\rangle;\text{ and }\lambda=T_{1}/\langle u\rangle,$
(11)

and to the equation of state:

$e^{\mu^{*}}=\lambda^{*}.$ (12)

This is the relevant equation for interpreting the empirical data, since
$\lambda^{*}$ can be measured from the data and $\mu^{*}$ can be interpreted
thanks to the thermodynamic analogy.

Indeed, comparing our results with those of a physical system, one can
identify $\mu$ with the chemical potential. We interpret $\mu$ as the “cost”
for including/creating or excluding/extinguishing firms in the proportional
large-size regime. Following MaxEnt Hernando and Plastino (2012a); Hernando
_et al._ (2013); Hernando and Plastino (2012b), the system is in contact with
a reservoir of firms, and tends to minimize $\Omega$. Since
$\partial\Omega/\partial N=-\mu$, for $\mu>0$ any new firm, in the
proportional regime, decreases $\Omega$, making it more likely the emergence
of a flow of firms entering into the system. However, for $\mu<0$, any new
firm will increase the value of $\Omega$, allowing for a flow of firms exiting
the system. In the particular case $\mu=0$ there is no cost for the flow of
firms, in what we expect to be an equilibrium, stable, and healthy situation
for a capitalist economy.

The thermodynamic variable $\lambda$ defines the exponent of the distribution,
and can be interpreted as a measure of the typical wealth of a region.
Specifically, it determines the scale of the size of firms, because it
constraints the geometric mean of $x$. Indeed, the use of the geometric mean
instead of the mean is common for systems with scale invariance, where long-
tailed distributions have undefined moments but well-defined log-moments IHD
(2014); RPI (2013). This value will change from one economy to other one.

Thanks to the equation of state Eq. (12), we can provide an intuitive,
physically-based interpretation of that exponent:

  * •

for $\lambda<1$ ($\mu<0$) the system favors the extinction of firms;

  * •

for $\lambda>1$ ($\mu>0$) the system favors the creation of firms;

  * •

for $\lambda=1$ ($\mu=0$) the system freely creates and extinguishes firms.

This last particular case corresponds to the Zipf’s law distribution, namely

$N-P(x)=N\frac{x_{c}}{x}.$ (13)

##  III Numerical experiments

In aim of testing our theoretical procedure we have performed numerical
experiments in terms of random walkers via a Monte Carlo simulation (MC). At
the initial time, the $N$ random walkers are randomly located using a uniform
distribution. We assume independent stochastic Wiener coefficients for
different firms, within each of the three regimes or, more explicitly,

$\langle
v_{qi}(t)v_{q^{\prime}j}(t^{\prime})\rangle=T_{q}\delta_{ij}\delta_{qq^{\prime}}\delta(t-t^{\prime}),$
(14)

where $q$ defines the specific dynamical regime, as in Eq. (2). In order to
make explicit the mechanisms that govern the dynamics, we will use a reduced
approach, in which each of the regimes, according to the size $x$, evolves
independently. Therefore, instead of simulating the whole dynamics, i.e. Eq.
(2), we aim to understand the particular contribution of each term in that
equation. Henceforth, we will focus in the interplay between the linear and
proportional growth regimes, disregarding the intermediary regime. To achieve
this goal, we use the following equation for the microscopic dynamics:

$\dot{x}_{i}(t)=\begin{cases}v_{0,i}(t),&\textrm{for }x<x_{c},\\\
v_{1,i}(t)x_{i}(t),&\textrm{for }x>x_{c},\end{cases}$ (15)

where $x_{c}$ defines the border between the linear and the proportional
regimes. Since the number of walkers in the proportional regime is not
constrained, we follow here the given recipe for a Grand canonical ensemble
Reif (1965); Balian (2006) where $\mu$ is fixed and the fluctuation in the
number of walkers is determined by the probabilities of including ($P_{+}$) or
extracting ($P_{-}$) a walker as

$P_{+}\propto e^{\mu^{*}}\quad\quad\text{ and }\quad\quad P_{-}\propto
e^{-\mu^{*}}.$ (16)

According to these probabilities, in an ensemble with $\mu^{*}=0$ any walker
can leave or enter into the system without any restriction. Additionally,
following Eq. (11) the constraint $\langle
u\rangle=\langle\ln(x/x_{c})\rangle=1$ should be fulfilled. We have performed
several realizations with different initial conditions, and let the system
evolve until reaching equilibrium. In Fig. 2, we show the rank plot, Eq. (10),
for different simulation times, measured in MC steps. Here we choose
$x_{c}=100$. We see that the equilibrium distribution (for $\sim 3000$ MC
cycles) follows the Zipf’s law. Moreover, we show in Fig. 2 the chemical
potential for the equilibrium distribution. We see that, up to some
fluctuations, the constraint $\mu^{\ast}=0$ is respected.

We find that the equilibrium distribution does not depend on the initial
conditions, and follows Zipf’s law: for large values, the complementary of the
cumulative distribution follows Eq. (13), as predicted by our thermodynamic
framework. The distribution deviates from the analytic result, Eq. (13), as
the size of $x$ reaches the transition critical valued $x_{c}$. In view of
these results, we numerically validate our analytic procedure.

##  IV Empirical application

In order to empirically verify our theoretical model, we consider the Spanish
SABI database Dijk (2001); SAB , which is a comprehensive one for all firms
that have the obligation to disclose balance sheets in the Spanish Mercantile
Register. Our sample consists of $1\,155\,142$ firms along a decade, with more
than $500\,000$ firms per year. We select those firms which have been active
at any time during the last 10 years and use as our observable $x_{i}(t)$ for
the $i$th firm at year $t$ the so-called Earnings Before Interest, Taxes,
Depreciation, and Amortization (EBITDA). This quantity is widely employed for
assessing companies’ performances. It is homogeneous across companies and is
not affected by different forms of financing. We believe that our proxy for
size is a clear indicator of both corporate performance and size.

###  IV.1 Microdynamics

We first test our microscopic dynamical hypothesis by measuring the variance
of the EBITDA growth for each year. Since the EBITDA values can be negative,
we study separately positive and negative domains. We first analyze all the
Spanish firms in the same set, displaying in Fig. 3 the dependence of the
growth-variance on the EBITDA for the year 2009. We find a remarkable match to
Eq. (2) for both positive and negative domains. The transition to proportional
growth takes place, in this case, at $x=T_{1/2}/T_{1}=109\times 10^{3}$ euro.
Additionally, in both domains, the linear regime temperature, $T_{0}$, is of
the same order of magnitude. As shown in the Supporting Information, all the
available data for 10 years match Eq. (2), with slightly changing
temperatures. A similar analysis, made per each Spanish autonomous community,
shows that the dynamics is also obeyed individually by regions, as shown in
Fig. 4. We do not find any exception for all the 15 Spanish autonomous
communities during these 10 years. In view of these results, we empirically
confirm the validity of the dynamical equation (2). Additionally, we find that
the temperatures $T_{1}$ and $T_{1/2}$ for the negative domain are
significantly higher than those for the positive one. Remarkable, as shown in
Fig. 5, $T_{0}$ can be considered the same for positive and negative EBITDA,
indicating that the same non-proportional regime is connecting both domains.

###  IV.2 Macroequilibrium

Once the dynamical equation has been validated, we pass to the rank-
distributions. We plot in Fig. 6 the complementary cumulative function
$N-P(x)$ for all Spanish firms in 2009, including those with positive and
negative EBITDA. We observe that for large values of this function the power
law Eq. (13) is followed, as predicted by our thermodynamic equilibrium
hypothesis, with an exponent very close to that of Zipf’s law $\lambda=1$. For
smaller values, the distribution deviates from the power law. We have checked
that this deviation takes place at about the same transition value
$x_{c}=T_{1/2}/T_{1}$, as predicted by our numerical experiments. This is a
compelling evidence for the relation between the dynamics and the distribution
given by our theoretical framework. We also measure the chemical potential
$\mu^{*}$, via the equation of state Eq. (12). This chemical potential is, in
general, with some deviations, close to $\mu=0$, value for which the
creation/extinction of firms has no cost for the energy potential function
$\mathcal{E}$. We find the same picture when studying the firms’ distribution
per each community. In general, all distributions are very close to the Zipf’s
regime. In view of these results, we consider that our theoretical framework
properly describes the dynamics and equilibrium of the ensemble of Spanish
firms.

##  V Discussion

Some interesting assertions can be made with regards to our theoretical
framework. The most relevant, the thermodynamic interpretation of the exponent
$\lambda^{*}$ of the long-tail in size distribution. Thanks to the equation of
state Eq. (12) we provide for the first time a clear explanation of this
exponent, linking that adimensional number with a dynamical, intuitive
mechanism as the cost to the system of creating or extinguish a firm, measured
by the chemical potential $\mu^{*}$. This interpretation can be used to
measure the macroscopic effect of particular economical policies, and to
measure how healthy is a capitalist-based economy. We find that, in general,
the value of $\mu^{*}$ in Spanish regions is close to zero —as shown in Fig.
6— indicating the freedom of creating or extinguishing a firm.

Additionally, we settle here the form of the microscopic dynamics by Eq. (2),
and its dependence with the size. Contrarily to other social systems following
proportional growth Hernando and Plastino (2012a); Hernando  _et al._ (2015);
Eliazar (2015), here exists the possibility of _negative_ values. This
requires an additional dynamical mechanism for the evolution of firms, that is
successfully included in our current approach as a linear term dominant for
small sizes. Firms can be classified according to the dynamical regime, even
if they are in the negative (losses) or positive (gain) domain. In a pictorial
way, we can talk about _heaven_ (positive proportional regime), _hell_
(negative proportional regime), or _purgatory_ (linear regime). The fact that
the temperatures in the positive domain are systematically smaller than in the
negative one, as illustrated in Fig. 5, can be summarized as _hell is warmer
than heaven_. Thus, a firm in hell losses money in a faster fashion than it
would equivalently earn it in heaven.

We also find useful as macroeconomic indicator the position of the transition
zone between medium and proportional growth regime in the negative domain,
that gives an estimate of the minimum losses a firm can afford before going
bankrupt —or metaphorically, the _hell’s gate_. Similarly, the same transition
but in the positive domain provides an estimation of the success region for
firms —that we might wish to call _heaven’s door_. Fig. 7 shows both
transition values from 2003 to 2012 as measured by the respective temperature
ratios $T_{1/2}/T_{1}$. We observe that, before the 2008 Global Financial
Crisis, both transitions were approximately equivalent in size, exhibiting a
symmetry between positive and negative regimes. Right before this crisis, the
negative value reached its maximum, indicating some abnormal economic growth,
potentially related with the speculative bubble. The confidence interval for
this specific year is higher than the absolute value, indicating that this
phenomenon did not happen with the same intensity among all the autonomous
communities. Finally, in the succeeding years, the negative value was reduced
to a half, augmenting the probability of firms to go bankrupt; whereas the
positive transition also decreased, although not as rapidly as in the negative
domain. After the burst of the crisis, both transitions tend to converge again
to a similar value, but lower than before the crisis. Because the equation of
state Eq. (12) and the constrained value of $\langle u\rangle$, this lower
value reflects a general reduction of the wealth in the whole system, because
it diminishes, on average, the scale of the successful firms at the
proportional regime.

As a final remark, the numerical simulation provided here based on walkers
under the Grand canonical ensemble opens the possibility of developing
simulation tools where economical forces can be introduced in the same fashion
as done for physical forces for gases and liquids. Indeed, we open a bridge
between the mathematical tools used in statistical mechanics and firms’
dynamics. Our analytic and numerical procedures can be used to deeply analyze
the empirical data measuring and parameterizing the economic forces in play,
and develop a full quantitative theory about these dynamics. Work in this line
is in progress.

##  VI Conclusions

We advanced in this paper a complete thermodynamic framework which
accommodates the firm size distribution of a given region. We propose an
empirically proof a microscopic dynamical hypothesis, and show how the firms
obey the maximum entropy principle at the macroscopic level. We analytically
proof the connection between microscopic dynamics and equilibrium firm’s size-
distributions via MaxEnt, and formulate the equation of state that relates the
exponent of the long tail in size distributions with a well-known
thermodynamic observable as the chemical potential. This lead to a clear and
intuitive interpretation of the exponents, showing that can be used to measure
the health of a economy. Indeed, the emergence of Zipf’s law is associated
with the free cost to the system of creating and extincting firms, as expected
in a capitalist-based economy. All these theoretical considerations have been
validated by comparison with empirical data concerning Spanish firms, in a
window of a decade. We expect this work to be a first step towards the
formalization of a theory of the evolution of firms that will yield underlying
forces and laws of evolution.

## References

  * Pareto (1896) V. Pareto,  _Cours d’Economie Politique_ (Genève: Droz, 1896).
  * Krugman (1995) P. R. Krugman, Japan and the World Economy 7, 371 (1995).
  * Gabaix (1999) X. Gabaix, Q J Econ 114, 739 (1999).
  * Bettencourt (2013) L. M. A. Bettencourt, Science 340, 1438 (2013).
  * Hernando and Plastino (2012a) A. Hernando and A. Plastino, Phys Rev E - Stat, Nonlinear, Soft Matter Phys 86, 066105 (2012a).
  * Hernando  _et al._ (2013) A. Hernando, R. Hernando, A. Plastino, and A. R. Plastino, J. R. Soc. Interface 10, 20120758 (2013).
  * Hernando and Plastino (2013) A. Hernando and A. Plastino, Phys Lett A 377, 176 (2013).
  * Hernando, Hernando, and Plastino (2014) A. Hernando, R. Hernando, and A. Plastino, J. R. Soc. Interface 11, 20130935 (2014).
  * Hernando  _et al._ (2015) A. Hernando, R. Hernando, A. Plastino, and E. Zambrano, J. R. Soc. Interface 12, 20141185 (2015).
  * Cancho and Solvé (2003) R. F. Cancho and R. V. Solvé, Proc Natl Acad Sci U S A 100, 788 (2003).
  * Barabási and Albert (1999) A. L. Barabási and R. Albert, Science 286, 509 (1999).
  * Katz (1999) S. Katz, Res Policy 28, 501 (1999).
  * van Raan (2013) A. F. J. van Raan, PLoS ONE 8, e59384 (2013).
  * Zipf (1949) G. K. Zipf,  _Human Behavior and the Principle of Least Effort_ (Addison-Wesley, 1949).
  * Newman (2005) M. J. Newman, Contemporary physics 46, 323 (2005).
  * Hernando  _et al._ (2009) A. Hernando, D. Puigdomèech, D. Villuendas, C. Vesperinas, and A. Plastino, Phys Lett A 374, 18 (2009).
  * Cristelli, Batty, and Pietronero (2012) M. Cristelli, M. Batty, and L. Pietronero, Scientific Reports 2, 812 (2012).
  * Gibrat (1931) R. Gibrat,  _Les Inegalité, economiques: applications aux inéqualité, des richesses, à la concentration des entreprises, aux populations des villes, aux statistiques des familles, etc., d’une loi nouvelle: la loi de l’effect proportionnel_ (Paris, Librairie du Recueil Sirey, 1931).
  * Gaffeo  _et al._ (2012) E. Gaffeo, C. D. Guilmi, M. Gallegati, and A. Russo, Ecol Comp 11, 109 (2012).
  * Hart and Prais (1956) P. E. Hart and S. J. Prais, J R Stat Soc 119, 150 (1956).
  * Quandt (1966) R. E. Quandt, Am Econ Rev 56, 416 (1966).
  * Simon and Bonnini (1958) H. A. Simon and C. P. Bonnini, Am Econ Rev 48, 607 (1958).
  * Mansfield (1962) E. Mansfield, Am Econ Rev 52, 1023 (1962).
  * Bottazzi and Secchi (2006) G. Bottazzi and A. Secchi, RAND Journal of Economics 37, 235 (2006).
  * Jacquemin and de Lichtbuer (1973) A. P. Jacquemin and M. C. de Lichtbuer, Eur Econ Rev 4, 393 (1973).
  * Kwasnicki (1998) W. Kwasnicki, Struc C Econ Dynam 9, 135 (1998).
  * Ijiri and Simon (1977) Y. Ijiri and H. A. Simon,  _Skew distribution and the size of business firms_ (North-Holland, Amsterdam, 1977).
  * Jovanic (1982) B. Jovanic, Econometrica 50, 649 (1982).
  * Marsili (2005) O. Marsili, Rev Ind Org 27, 303 (2005).
  * Vining (1976) D. R. Vining, J Pol Econ 84, 369 (1976).
  * Segal and Spivak (1989) U. Segal and A. Spivak, Eur Econ Rev 33, 159 (1989).
  * Sutton (1997) J. Sutton, J Econ Lit 35, 40 (1997).
  * Stanley  _et al._ (1995) M. H. R. Stanley, S. V. Buldyrev, S. Havlin, R. N. Mantegna, M. A. Salinger, and H. E. Stanley, Econ Lett 49, 453 (1995).
  * Stanley  _et al._ (1996) M. H. R. Stanley, L. A. N. Amaral, S. V. Buldyrev, S. Havlin, H. Leschhorn, P. Maass, M. A. Salinger, and H. E. Stanley, Nature 379, 804 (1996).
  * Kattuman (1996) P. A. Kattuman, Struc C Econ Dynam 7, 479 (1996).
  * Axtell (2001) R. L. Axtell, Science 293, 818 (2001).
  * Cabral and Mata (2003) L. M. B. Cabral and J. Mata, Am Econ Rev 93, 1075 (2003).
  * Fu  _et al._ (2005) D. Fu, F. Pammolli, S. V. Buldyrev, M. Riccaboni, K. Matia, K. Yamasaki, and H. E. Stanley, Proc Natl Acad Sci U S A 102, 18801 (2005).
  * Palestini (2007) A. Palestini, Econ Lett 94, 367 (2007).
  * Riccaboni  _et al._ (2008) M. Riccaboni, F. Pammolli, S. V. Buldyrev, L. Ponta, and H. E. Stanley, Proc Natl Acad Sci U S A 105, 19595 (2008).
  * Growiec  _et al._ (2008) J. Growiec, F. Pammolli, M. Riccaboni, and H. E. Stanley, Econ Lett 98, 207 (2008).
  * di Giovanni, Levchenko, and Ranciere (2011) J. di Giovanni, A. A. Levchenko, and R. Ranciere, J Int Econ 85, 42 (2011).
  * Gallegati and Palestrini (2010) M. Gallegati and A. Palestrini, Transdiscipl Perspect Econ Complex 75, 69 (2010).
  * Segarra and Teruel (2012) A. Segarra and M. Teruel, J Econ Behav Org 82, 314 (2012).
  * de Wit (2005) G. de Wit, Int J Ind Organ 23, 423 (2005).
  * Farmer and Foley (2009) J. D. Farmer and D. Foley, Nature 460, 685 (2009).
  * Naldi (2003) M. Naldi, Econ Lett 78, 329 (2003).
  * Hernando and Plastino (2012b) A. Hernando and A. Plastino, Eur Phys J B 85, 293 (2012b).
  * Hernando, Plastino, and Plastino (2012) A. Hernando, A. Plastino, and A. R. Plastino, Eur Phys J B 85, 147 (2012).
  * Griffiths (1987) D. J. Griffiths,  _Introduction to Elementary Particles_ (Wiley, John & Sons, 1987).
  * Reif (1965) F. Reif,  _Fundamentals of Statistical and Thermal Physics_ (McGraw–Hill, 1965).
  * Balian (2006) R. Balian,  _From microphysics to macrophysics. Vol I and II_ (Springer Science & Business Media, 2006).
  * IHD (2014) “Human development report,” United Nations (2014).
  * RPI (2013) “News release,” Office for National Statistics – UK (10 January, 2013).
  * Dijk (2001) B. V. Dijk,  _SABI_ (Bureau van Dijk, Paris, 2001).
  * (56) “SABI Sistema de Análisis de balances ibéricos,” Bureau van Dijk Electronic Pub, Brussels [online database] http://www.bvdinfo.com/en-gb/products/company-information/national/sabi.
  * Eliazar (2015) I. Eliazar, Annals of Physics , (in press) (2015).

Figure 1: Conceptual sketch of firms’ dynamics according with Eq. (2) and (3).
Firms (represented here by dots) evolve as random walkers in time —similar to
particles in a gas— according to the regime defined by they earnings
(delimited by dashed horizontal lines) and the empirical temperature at that
regime. Brown solid arrows show the path for two of the firms.

Figure 2: Rank plot (left) and chemical potential (right) for the simplified
dynamics of Eq. (15). We use $N=10^{4}$ random walkers with transition to
proportional growth at $x_{c}=100$ (dashed vertical line). The equilibrium
distribution is reached after $\sim 3000$ MC cycles.

Figure 3: Variance of EBITDA for the set of all Spanish firms in 2009. The
colored lines correspond to the fit according to Eq. (2). The black solid line
sketchs the trend-regime: linear for small firm-sizes, FSC for medium, and
proportional growth for the largest sizes.

Figure 4:  Variance of EBITDA for some Spanish autonomous communities in 2009.
The colored-surfaces correspond to the fit according to Eq. (2).

Figure 5: Temperatures $T_{1}$, $T_{1/2}$, and $T_{0}$ as function of the
year. Faded lines represent the temperature evolution for every autonomous
community, bold blue and red lines represent the Spanish mean temperature.

Figure 6: Rank plot (left) and chemical potential (right) for the empirical
data of Spain in 2009. The observed temperatures for positive EBITDA are
$T_{1}=1.27\pm 0.15$, $T_{1/2}=140\pm 20$, and $T_{0}=1100\pm 200$, and for
negative EBITDA $T_{1}=6.1\pm 0.9$, $T_{1/2}=250\pm 50$, and $T_{0}=1500\pm
500$. The vertical dashed lines display the transition to proportional growth
regimes, where the equation of state (12) holds.

Figure 7: Spain’s annual transition to the proportional regime. This is
measured by the rate $x_{c}=T_{1/2}/T_{1}$.


# Accurate Emotion Strength Assessment for Seen and Unseen Speech
Based on Data-Driven Deep Learning

###### Abstract

Emotion classification of speech and assessment of the emotion strength are
required in applications such as emotional text-to-speech and voice
conversion. The emotion attribute ranking function based on Support Vector
Machine (SVM) was proposed to predict emotion strength for emotional speech
corpus. However, the trained ranking function doesn’t generalize to new
domains, which limits the scope of applications, especially for out-of-domain
or unseen speech. In this paper, we propose a data-driven deep learning model,
i.e. StrengthNet, to improve the generalization of emotion strength assessment
for seen and unseen speech. This is achieved by the fusion of emotional data
from various domains. We follow a multi-task learning network architecture
that includes an acoustic encoder, a strength predictor, and an auxiliary
emotion predictor. Experiments show that the predicted emotion strength of the
proposed StrengthNet is highly correlated with ground truth scores for both
seen and unseen speech. We release the source codes at:
https://github.com/ttslr/StrengthNet.

Index Terms: Emotion strength, deep learning, data-driven

##  1 Introduction

Accurate emotion classification of speech and assessment of its strength are
essential to profile human behaviors, which has many potential applications,
such as human-robot interface, human-machine dialogue, and social media. In
recent years, there is an increasing interest in emotion control in expressive
speech synthesis, such as emotional text-to-speech, emotional voice
conversion, where accurate control of emotional strength in speech becomes
critically important.

The simplest emotion strength control method is to linearly scale the emotion
representation vector [1]. The effect of such a linear scale is hardly
interpretable. To obtain a meaningful strength descriptor, some followed the
idea of “relative attributes” [2, 3, 4] and quantify the emotion strength by
learning from the $<$neutral, emotional$>$ speech pairs. Support-Vector-
Machine (SVM) based attribute ranking [4] learns the difference between two
samples that are significantly different in a particular attribute, that has
been widely studied in computer vision [5, 6].

In speech processing, Zhu et al. [2] proposed to learn an emotion attribute
ranking function $R(\cdot)$ from the $<$neutral, emotional$>$ paired speech
features, then weight the emotional feature with a learnable weighting vector
and return a weighted sum as the indicator of the emotion strength of one
specific emotional speech. Lei et al. [3] further extended the utterance level
emotion attribute ranking function to phoneme level and obtain a fine-grained
ranking function. In this way, we can obtain a meaningful strength score for
emotional speech on a specific dataset, which correlates with human
perception.

Figure 1: The proposed StrengthNet that consists of Acoustic Encoder, Strength
Predictor and Emotion Predictor.

We note that a trained ranking function $R(\cdot)$ on specific data is not
easily generalized to new domains. In other words, $R(\cdot)$ is not able to
calculate an accurate or appropriate strength score for unseen or out-of-
domain speech. To extend to a new data, we need to retrain a new ranking
function on the $<$neutral, emotional$>$ paired samples from the new data.
Furthermore, the learning of ranking functions for new data requires parallel
samples. All these limit the scope of applications. Recently, it was shown
that deep learning has the ability to learn a mapping function effectively [7,
8]. The neural solution learns complex non-linear mapping relationships, and
exhibits good generalization ability with the support of a large number of
model parameters [9]. Most importantly, the data-driven training strategy
appears to be more powerful and have the potential to achieve good performance
for out-of-domain data [10].

In this paper, we propose a novel neural solution to emotion strength
assessment, termed StrengthNet. To improve the model generalization, we employ
a data-driven strategy, named “domain fusion”, to mix emotional data from
various domains for model training. StrengthNet is a multi-task framework that
includes a convolutional neural network (CNN) based acoustic encoder, a
bidirectional long short-term memory (BiLSTM) based strength predictor and an
auxiliary BiLSTM based emotion predictor. The acoustic encoder extracts the
high-level features from the input mel-spectrum. The strength predictor aims
to predict the strength score for the input mel-spectrum. The emotion
predictor is used to predict the emotion category, which serves as an
auxiliary task.

The main contributions of this paper include, 1) We propose a novel data-
driven deep learning based speech emotion strength assessment model, i.e.,
StrengthNet; 2) We show that the predicted emotion strength of seen and unseen
speech is highly correlated with the ground truth; To our best knowledge, this
is the first deep learning model for accurate emotion strength assessment for
seen and unseen speech.

This paper is organized as follows: In Section 2, we formulate the proposed
StrengthNet and the domain fusion training strategy. Section 3 report the
experimental results. Finally, Section 4 concludes the study.

##  2 StrengthNet

We first describe the overall StrengthNet architecture, then explain the
domain fusion training details. Lastly, we describe the model inference.

###  2.1 Model Overview

We formulate StrengthNet under the multi-task framework, which consists of an
acoustic encoder, a strength predictor and an emotion predictor as in Fig. 1.

To improve the model generalization ability, we adopt a domain fusion training
strategy. During training, the Mel-Spectrum features come from multiple
emotional domains. The ground-truth strength scores, which is normalized to
(0,1), are derived from a ranking function learned separately. The details
will be described in Section 2.2. We employ a multi-task framework with an
auxiliary emotion category prediction task. Next we will describe the model
architecture first.

####  2.1.1 Acoustic Encoder

The acoustic encoder takes the acoustic feature sequence, that is mel-spectrum
in this work, as input to extract a high-level feature representation. The
acoustic encoder consists of 12 convolution layers. The strategy of stacking
more convolutional layers to expand the receptive field of a CNN has been
widely used to model time series data and yield satisfactory performance [7].
Given an input mel-spectrum sequence $X$, the CNN based acoustic encoder aims
to extract a high-level feature $\mathcal{H}$. The high-level feature
$\mathcal{H}$ is then fed to two predictors to predict the emotion strength
score and emotion category, respectively.

####  2.1.2 Strength Predictor

The strength predictor then reads the high-level feature representation to
predict the emotion strength. Recent studies have confirmed the effectiveness
of combining CNN and BiLSTM for classification [7], and recognition [11]
tasks.

The strength predictor consists of a BiLSTM layer, two FC layers, followed by
an average pooling layer. The BiLSTM layer takes the high-level feature
$\mathcal{H}$ as input to output the hidden states $\mathcal{S}$ for each time
step. We then use two FC layers to convert the frame-wise hidden states
$\mathcal{S}$ into the frame level scalar $\alpha_{f}$ to indicate the
strength score of each frame. Finally, an average pooling layer is applied to
the frame-level scores to obtain the utterance-level strength score $\alpha$.

To supervise the training of the strength predictor, we define a mean absolute
error (MAE) loss $\mathcal{L}_{u\\_str}$ behind the average pooling layer to
force the predicted utterance-level strength score $\alpha$ close to the
ground-truth value.

To improve the convergence of StrengthNet, we further define another MAE
loss[7] $\mathcal{L}_{f\\_str}$, denoted as frame-wise constraint, behind the
last FC layer to minimize the difference between the predicted frame-level
strength score $\alpha_{f}$ and the ground-truth strength.

####  2.1.3 Emotion Predictor

The auxiliary emotion predictor is devised to predict the emotion category.
Similar to the strength predictor, the emotion predictor also consists of a
BiLSTM layer and an additional softmax layer. The BiLSTM summarizes the
temporal information of high-level feature $\mathcal{H}$ into another latent
states $\hat{\mathcal{S}}$. Finally, the softmax layer converts the latent
states $\hat{\mathcal{S}}$ to the output probability for all emotion
categories. Accordingly, we can obtain the predicted emotion category
$\theta$. We define a “categorical_cross_entropy” loss $\mathcal{L}_{cat}$ to
restrain the emotion predictor. We formulate the total objective function
$\mathcal{L}_{total}$ for training the StrengthNet as:

$\mathcal{L}_{total}=\mathcal{L}_{f\\_str}+\mathcal{L}_{u\\_str}+\mathcal{L}_{cat}.$
(1)

###  2.2 Domain Fusion

As shown in Fig. 2, to improve the model generalization, we employ a domain
fusion strategy [12, 13], which mix multiple emotional speech datasets from
various domains to train our StrengthNet. The fused data mixed from various
domains will represent a more comprehensive set, thus minimizing the distance
between the training and validation set, as well as any future testing sets.

Let us denote $\mathcal{D}_{1}$, $\mathcal{D}_{2}$, …, $\mathcal{D}_{K}$ as
the $K$ emotional speech datasets. We train the emotion attribute ranking
function $R(\cdot)$ for all datasets, since they all provide $<$neutral,
emotional$>$ pairwise speech samples.

To achieve this, we first build two sets, that are $O$ and $S$, which contain
ordered and similar paired samples, respectively. Specifically, for each
dataset $\mathcal{D}_{k}$ ($k\in$ [1,K]), we pick up one sample from neutral
speech and another sample from emotional speech to build the ordered set $O$.
We expect that the emotion strength of the emotional sample is higher than
that of the neutral sample. For the similar set $S$, we pick up two samples
from ‘neutral’ speech or emotional speech (such as ‘happy’). We assume that
two samples from the same emotion category have more similar emotion strength
than two others from different categories.

We follow [2] and build a support vector machine (SVM) [14] to learn the
ranking function $R(\cdot)$ for the emotion strength attribute. Finally, we
derive the strength scores across all datasets to serve as the ground-truth in
the training objective of StrengthNet. Note that the strength score is
normalized to (0,1) with 1 having the strongest emotion. We prepare mel-
spectrum features for all datasets as the input of StrengthNet. The
StrengthNet can be seen as the aggregation of multiple domain experts.

###  2.3 Run-time Inference

During inference, the StrengthNet takes a mel-spectrum extracted by any
emotional speech as the input feature to predict its emotion strength score,
as well as its emotion category. Furthermore, our StrengthNet can be used
directly to predict emotion strength for a new emotional speech dataset
without retraining, that is a clear advantage.

Figure 2: The shown domain fusion strategy is utilized to improve the model
generalization.

(a) StrengthNet (b) StrengthNet w/o $\mathcal{L}_{cat}$ (c) StrengthNet w/o
$\mathcal{L}_{f\\_str}$

Figure 3: Histogram of the utterance level strength predictions for (a)
StrengthNet; (b) StrengthNet w/o $\mathcal{L}_{cat}$; (c) StrengthNet w/o
$\mathcal{L}_{f\\_str}$. The X-axis and Y-axis of subfigures represent the
strength scores and the utterance number, respectively.

##  3 Experiments

###  3.1 Datasets

We use the ESD dataset [15] to validate the performance of StrengthNet in
terms of strength prediction. The English corpus with a total of nearly 13
hours of speech is used in our experiments. In addition, we use two additional
English SER datasets: the Ryerson Audio-Visual Database of Emotional Speech
and Song (RAVDESS) [16] and the Surrey Audio-Visual Expressed Emotion (SAVEE)
database [17] to achieve domain fusion and test the model generalization.

The datasets are summarized as follows:

  * •

ESD [15]: ESD is a multilingual emotional speech dataset and has 350 parallel
utterances spoken by 10 native English and 10 native Mandarin speakers. We use
the English corpora with a total of nearly 13 hours of speech by 5 male and 5
female speakers in five emotions, namely happy, angry, neutral, sad and
surprise. All speech samples are sampled at 16 kHz and coded in 16 bits.

  * •

RAVDESS [16]: The RAVDESS dataset contains two emotional parts: a speech part
and a song part. We mainly focus on its speech part, which has 1440 utterances
acted by 24 professional actors (12 female, 12 male). There are 8 classes of
emotion contained in the speech part (calm, happy, sad, angry, fear, surprise,
disgust, and neutral state). In the process of collecting data, actors are
asked to speak two fixed sentences with different classes of emotion.

  * •

SAVEE [17]: The SAVEE database consists of recordings from four male actors in
seven different emotions, 480 British English utterances in total with seven
emotional states (i.e., surprise, happy, sad, angry, fear, disgust, and
neutral). And the sentences are selected from the standard TIMIT corpus and
phonetically-balanced for each emotion.

Please note that the emotional speech databases can be divided into two types:
acted and improvised [18]. All above three mentioned datasets belongs to acted
family which widely used in emotional TTS [19]. We believe that it is
appropriate to employ acted emotional speech dataset for our experiments, due
to the improvised speech is hard to induce strong and well-differentiated
emotions [20].

###  3.2 Experimental Setup

For all datasets, we take 5 emotion classes, that are happy, sad, angry,
surprise, and neutral, to build the $<$neutral, emotional$>$ paired speech to
train the ranking function. During training of StrengthNet, we take 4 emotion
classes, that are happy, sad, angry and surprise, into account.

We extract 80-channel mel-spectrum features with a frame size of 50ms and
12.5ms frame shift, that are further normalized to zero-mean and unit-
variance, to serve as the model input. We employ openSMILE [21] to extract
384-d features for each utterance and follow the relative attributes algorithm
111https://github.com/chaitanya100100/Relative-Attributes-Zero-Shot-Learning
to train the ranking function. The utterance level strength scores for all
datasets are obtained using their ranking function and serve as the ground
truth score, or training target, for the strength predictor in our
StrengthNet. To calculate the frame level MAE $\mathcal{L}_{f\\_str}$, the
ground-truth strength score is used for all the frames in the speech
utterance. The emotion predictor of StrengthNet aims to predict 4 emotion
categories, including happy, sad, angry and surprise.

The acoustic encoder consists of 4 Conv2D blocks with filters size [16, 32,
64, 128], respectively. Each block includes 3 Conv2D layers with strides shape
{[1,1], [1,1], [1,3]}, respectively. All layers share the same kernel size [3
$\times$ 3] and ReLU activation function. For BiLSTM layers in two predictors,
each direction contains 128 cells.

All speech samples are resampled to 16 kHz. We train the models using the Adam
optimizer with a learning rate=0.0001 and $\beta_{1}$ = 0.9, $\beta_{2}$ =
0.98. We set the batch size to 64. The dropout rate is set to 0.3. For each
dataset, we partition the speech data into training, validation, and test set
at a ratio of 8:1:1. We apply early stopping based on the MAE of the
validation set with 30 epochs patience.

###  3.3 Experimental Results

####  3.3.1 Architecture Comparison Results

First, we intend to validate the effectiveness of the multi-task framework and
frame constraint in our StrengthNet, in terms of strength prediction
performance on the ESD dataset. We develop StrengthNet and two variants for an
ablation study, reported as follows: 1) StrengthNet (proposed), which is our
proposed model that consists of an acoustic encoder, strength predictor and
auxiliary emotion predictor; 2) StrengthNet w/o $\mathcal{L}_{cat}$, which is
the proposed model without an auxiliary emotion predictor; and 3) StrengthNet
w/o $\mathcal{L}_{f\\_str}$, which is the proposed model without a frame
strength constraint term $\mathcal{L}_{f\\_str}$ within the strength
predictor.

We report the performance in terms of MAE values. Note that the MAE can range
from 0 to $\infty$. It is negatively-oriented scores and lower values are
better. Fig. 3 shows the overall performance of these systems at the utterance
level. We report the MAE value of emotion strength prediction and the accuracy
of emotion category prediction on test data. We observe in Fig. 3 that our
proposed StrengthNet outperforms Strength w/o $\mathcal{L}_{cat}$ and Strength
w/o $\mathcal{L}_{f\\_str}$ and achieves the best performance, that is
attributed to the multi-task and frame constraint strategy. Specifically, we
find that StrengthNet achieves the lowest MAE score of 0.072 and highest
emotion recognition accuracy (denoted as SER Acc as shown in the axis titles
of Fig. 3) of 0.874. To sum up, the above results indicate that the
combination of multi-task learning and frame constraint can effectively learn
emotion strength cues from the input mel-spectrum to perform emotion strength
prediction well, along with category prediction.

Table 1: MAE results for the RAVDESS and SAVEE datasets in the comparison. Method | MAE
---|---
RAVDESS | SAVEE
$R_{{\rm RAVDESS}}(\cdot)$ | NA | 0.304
$R_{{\rm SAVEE}}(\cdot)$ | 0.283 | NA
$R_{{\rm ESD}}(\cdot)$ | 0.266 | 0.272
$\rm StrengthNet_{{\rm ESD}}$ | 0.238 | 0.243
$\rm StrengthNet_{{\rm ESD+RAVDESS}}$ | NA | 0.173
$\rm StrengthNet_{{\rm ESD+SAVEE}}$ | 0.102 | NA

####  3.3.2 Domain Fusion Results

We further verify the effectiveness of the proposed domain fusion for model
generalization in terms of emotion strength prediction.

We train StrengthNet on three dataset settings, that are ESD, ESD+RAVDESS,
ESD+SAVEE. $\rm StrengthNet_{ESD}$, $\rm StrengthNet_{ESD+RAVDESS}$ and $\rm
StrengthNet_{ESD+SAVEE}$ are used to represent three trained models,
respectively. $R_{{\rm RAVDESS}}(\cdot)$, $R_{{\rm SAVEE}}(\cdot)$ and
$R_{{\rm ESD}}(\cdot)$ refer to the three trained ranking functions for three
datasets, respectively. We adopt all trained ranking functions and models to
predict the emotion strength score on unseen test data from RAVDESS or SAVEE,
and report the MAE results in Table 1.

As shown in Table 1, we observe that our StrengthNet performs a lower MAE than
ranking functions on both, the RAVDESS and SVAEE datasets. More importantly,
in the case of $\rm StrengthNet_{ESD+RAVDESS}$ and $\rm
StrengthNet_{ESD+SAVEE}$, the MAE achieves the lowest values of 0.173 and
0.102 on SVAEE and RAVDESS respectively. As can be seen from the results, our
proposed strength method can reduce the overall MAE on unseen data with the
domain fusion strategy, which performs better model generalization than the
attribute ranking function.

####  3.3.3 Human Perception Evaluation

We further evaluate how close the results of StrengthNet for unseen speech can
get to the emotional strength as perceived by humans.

Since the RAVDESS dataset includes the manual emotion intensity label, which
each emotion is produced at two levels of emotion intensity: normal and
strong. Therefore, we adopt $R_{{\rm SAVEE}}(\cdot)$, $R_{{\rm ESD}}(\cdot)$
and $\rm StrengthNet_{ESD+SAVEE}$ to predict the emotion intensity for the
test set of RAVDESS. We consider the predicted intensity scalars from 0 to 0.5
as ‘normal’ and 0.5 to 1.0 as ‘strong’ in two categories. We select 20
utterances from the test set.

Figure 4: Confusion matrices between predicted and perceived emotion strength
categories for unseen speech. The X-axis and Y-axis of the figures represent
the predicted and perceived category, namely “normal” and “strong”.

Fig. 4 presents the intensity confusion matrices. It is observed that the
StrengthNet shows a higher correlation between the predicted and perceived
emotion intensity categories, with a correlation of over 85%, that is
considered a competitive result against the trained ranking function. The
experiments confirm the superiority of the proposed domain fusion based
StrengthNet for unseen speech in terms of human perception.

##  4 Conclusion

This paper presents a data-driven deep learning-based speech emotion strength
assessment model for the emotional speech synthesis task, referred to as
StrengthNet. Experimental results demonstrate that our StrengthNet can achieve
accurate emotion strength prediction for both seen and unseen speech with the
help of domain fusion strategy. As per our knowledge, the proposed StrengthNet
is the first end-to-end speech emotion strength assessment model. In future
work, we intend to integrate our StrengthNet as a front-end or back-end for
emotional speech synthesis models to enhance the emotion expressiveness of
output emotional speech.

## References

  * [1] T. Li, S. Yang, L. Xue, and L. Xie, “Controllable emotion transfer for end-to-end speech synthesis,” in _2021 12th International Symposium on Chinese Spoken Language Processing (ISCSLP)_. IEEE, 2021, pp. 1–5.
  * [2] X. Zhu, S. Yang, G. Yang, and L. Xie, “Controlling emotion strength with relative attribute for end-to-end speech synthesis,” in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2019, pp. 192–199.
  * [3] Y. Lei, S. Yang, and L. Xie, “Fine-grained emotion strength transfer, control and prediction for emotional speech synthesis,” in _2021 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2021, pp. 423–430.
  * [4] D. Parikh and K. Grauman, “Relative attributes,” in _2011 International Conference on Computer Vision_. IEEE, 2011, pp. 503–510.
  * [5] B. Siddiquie, R. S. Feris, and L. S. Davis, “Image ranking and retrieval based on multi-attribute queries,” in _CVPR 2011_. IEEE, 2011, pp. 801–808.
  * [6] Z. Meng, N. Adluru, H. J. Kim, G. Fung, and V. Singh, “Efficient relative attribute learning using graph neural networks,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 552–567.
  * [7] C. Lo, S. Fu, W. Huang, X. Wang, J. Yamagishi, Y. Tsao, and H. Wang, “Mosnet: Deep learning-based objective assessment for voice conversion,” in _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019_ , G. Kubin and Z. Kacic, Eds. ISCA, 2019, pp. 1541–1545. [Online]. Available: https://doi.org/10.21437/Interspeech.2019-2003
  * [8] S. Fu, Y. Tsao, H. Hwang, and H. Wang, “Quality-net: An end-to-end non-intrusive speech quality assessment model based on BLSTM,” in _Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 September 2018_ , B. Yegnanarayana, Ed. ISCA, 2018, pp. 1873–1877. [Online]. Available: https://doi.org/10.21437/Interspeech.2018-1802
  * [9] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.
  * [10] H. Dinkel, S. Wang, X. Xu, M. Wu, and K. Yu, “Voice activity detection in the wild: A data-driven approach using teacher-student training,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 1542–1555, 2021.
  * [11] R. Liu, B. Sisman, G. Gao, and H. Li, “Expressive tts training with frame and style reconstruction loss,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 1806–1818, 2021.
  * [12] R. Volpi, H. Namkoong, O. Sener, J. Duchi, V. Murino, and S. Savarese, “Generalizing to unseen domains via adversarial data augmentation,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2018, pp. 5339–5349.
  * [13] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” _Journal of Big Data_ , vol. 6, no. 1, pp. 1–48, 2019.
  * [14] T. Joachims, “Optimizing search engines using clickthrough data,” in _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_ , 2002, pp. 133–142.
  * [15] K. Zhou, B. Sisman, R. Liu, and H. Li, “Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 920–924.
  * [16] S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english,” _PloS one_ , vol. 13, no. 5, p. e0196391, 2018.
  * [17] P. Jackson and S. Haq, “Surrey audio-visual expressed emotion (savee) database,” _University of Surrey: Guildford, UK_ , 2014.
  * [18] P. N. Juslin and P. Laukka, “Impact of intended emotion intensity on cue utilization and decoding accuracy in vocal expression of emotion.” _Emotion_ , vol. 1, no. 4, p. 381, 2001.
  * [19] C. Cui, Y. Ren, J. Liu, F. Chen, R. Huang, M. Lei, and Z. Zhao, “EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model,” in _Proc. Interspeech 2021_ , 2021, pp. 2766–2770.
  * [20] R. Banse and K. R. Scherer, “Acoustic profiles in vocal emotion expression.” _Journal of personality and social psychology_ , vol. 70, no. 3, p. 614, 1996\.
  * [21] F. Eyben, F. Weninger, F. Gross, and B. Schuller, “Recent developments in opensmile, the munich open-source multimedia feature extractor,” in _Proceedings of the 21st ACM international conference on Multimedia_ , 2013, pp. 835–838.