# An xAI Approach for Data-to-Text Processing with ASP ††thanks: Partially
supported by the _Interdepartmental Project on Artificial Intelligence_
(Strategic Plan UniUD–22-25) and by INdAM–GNCS projects CUP E55F22000270001
and CUP E53C22001930001.

Alessandro Dal Palù Università di Parma, ItalyGNCS-INdAM, Roma, Italy
<EMAIL_ADDRESS>Università di Udine, ItalyGNCS-INdAM, Roma, Italy
Agostino Dovier Andrea Formisano Università di Udine, ItalyGNCS-INdAM, Roma,
Italy<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The generation of natural language text from data series gained renewed
interest among AI research goals. Not surprisingly, the few proposals in the
state of the art are based on training some system, in order to produce a text
that describes and that is coherent to the data provided as input. Main
challenges of such approaches are the proper identification of _what_ to say
(the key descriptive elements to be addressed in the data) and _how_ to say:
the correspondence and accuracy between data and text, the presence of
contradictions/redundancy in the text, the control of the amount of synthesis.

This paper presents a framework that is compliant with xAI requirements. In
particular we model ASP/Python programs that enable an explicit control of
accuracy errors and amount of synthesis, with proven optimal solutions. The
text description is hierarchically organized, in a top-down structure where
text is enriched with further details, according to logic rules. The
generation of natural language descriptions’ structure is also managed by
logic rules.

##  1 Introduction

The last decades witnessed a remarkable attention to the impact of Artificial
Intelligence (AI) on many aspects of everyday life. The ensuing debate
underscores the social and ethical implications of the use of AI-based systems
and is maturing into new regulations in USA and in Europe [16, 5]. The
European proposal should be operational by 2024 and it introduces a risk-based
classification of AI systems based on the principles of ethical AI [10].

The term _explainable AI_ (xAI, see, for instance, [4, 2]) has emerged to
capture desirable properties of high-risk systems: e.g., transparency (or
glass box approach), ethics, the capability to support results in terms of
intelligible descriptions, accountability, security, privacy, and fairness. In
the near future, the adoption of AI systems will depend on the capability of
providing a high-level description of inner activities, which promotes the
interpretability and transparency of the decisions that lead to a result.
Hence, domain experts could easily understand and criticize the contributions
coming from an AI system, in a similar way colleague’s opinions would be
considered.

In general terms, the data-to-text generation task consists in automatically
generating descriptions from non-linguistic data. In recent years, there has
been growing interest in such systems (see for instance the work [7],
supported by the Alexa AI organization). Systems able to textually summarize
data, such as time-series, can help in making data more accessible to non-
experts, compared to traditional forms of presentations. Indeed, natural
language descriptions of data can help readers even in cases where data are
depicted by means of forms of representations usually considered “self-
explanatory”, such as charts, histograms, pies, etc, that might be difficult
to read if the amount and/or complexity of data is large.

Text-to-speech tools, such as screen readers, might exploit textual generator
to extract information from charts contained in a digital document. This is
also relevant in situations where the interpretation of visualizations is made
difficult or hindered for people with visual impairments or when readers have
limited cognitive abilities in comprehending and analyzing complex charts.

Variants of this problem have been widely studied and solutions have been
proposed in various domains. For example, [14] and [17] apply a data-to-text
generation system to interpret data-records concerning NBA basketball games.
In the financial field, [11] proposes an approach to the problem of comment
generation for time-series of stock prices. Different solutions have been
proposed expressively to treat specific forms of diagrammatic representations
of data such as charts and histograms [12, 3] or tables [9]. In the healthcare
domain, [13] presents the prototype BT-45 aimed at generating textual
summaries of physiological clinical data (acquired automatically from sensors
or entered routinely by the medical staff). For space limits, we cannot
provide an exhaustive description of the literature concerning data-to-text
generation. We refer the reader to the previously mentioned papers and to
references therein.

Almost all the recent approaches to data-to-text are based on Machine Learning
(ML) and ultimately rely on Deep Learning (DL) techniques, usually exploiting
Neural Networks training algorithms. For example, the paper [7], mentioned
earlier, proposes DataTuner a neural data-to-text generation system.

There are a few exceptions worth mentioning. Among them we mention the seminal
work [8] that proposes a rule-based approach (actually, an expert system
written in the OPS5 production system language) to generate descriptions of
inputs from a Dow Jones stock quotes database. Another notable exception is
the BT-45 system cited earlier. It exploits signal analysis and pattern
detection techniques to process input time series and relies on a corpus of
human-authored sample linguistic expression of concepts to be used in
generating textual summaries. The system is one of the few proposals appeared
in the literature that do not rely on ML/DL techniques. In this sense, it is
similar to our approach. Nevertheless, it is conceived and designed for a very
specific application field. Indeed, it requires the development, by human
experts, of an ontology of medical concepts. Such an ontology is essential
both for assessing the importance of patterns in input data and for generating
expert-oriented output descriptions (see [13] for a detailed presentation of
the system).

It should be noted that Machine Learning and Deep Learning methods optimize
inner parameters (sub-symbolic representation) to adhere to training targets,
while they struggle to produce a high-level description (or symbolic) about
how and why the network learned specific parameters (black box approach). This
limitation favoured post-hoc explanations [4], e.g., textual captioning and/or
visual explanation by examples, still powered by an additional sub-symbolic
machinery. Even though explanations in Deep Learning approaches are one of the
next big challenges for AI, [15] expresses some concerns about the accuracy
achievable by this type of explainability and postulates that interpretable
models for high-stakes decisions should be sub-symbolic free systems.

The urge for explainability represents an opportunity of discontinuity with
respect to current AI methods. We divert from mainstream methods based on ML
and DL, and the recent trend of producing a high-level explanation of such
black box systems. The design of an architecture that is explainable and ML/DL
free represents an innovative and unexplored research direction.

Our strategic choice is to use robust and off-the-shelf technologies to reach
xAI compliance. We select Answer Set Programming (ASP) and Clingo [6] as the
explainable core of the system, because they allow knowledge representation at
a high-level and enables reasoning about such knowledge. This framework not
only is explainable, but also domain experts, in a scientists-in-the-loop
fashion, can detect flaws in the automated process and review the knowledge
base accordingly. Handling of incoherent points of view, inconsistent rules,
uncertainty and lack of knowledge can be explicitly supported by the system.

##  2 System’s Design

Our systems models the selection of _what_ to say in (which descriptions are
interesting to be narrated) and _how_ to say (how to build the qualitative
features and how to create a narration structure) the final narration. The
core of the system is explainable by design, since we develop a transparent
deductive handling of semantic rules that describe features, by exploiting
ASP. The descriptions selection is done through two optimization steps
performed by two ASP programs which determine, possibly multiple, descriptions
for each relevant region of the input. Then, the best candidate descriptions
(for different levels of detail) are selected. The outcome of this selection
phase is then processed, again by using ASP, to determine structural relations
among the fragments (narration order, dependencies, etc) to be used for the
generation of the narration. A Python script controls the pre-computation of
input, the handling of ASP programs execution and the conversion of ASP’s
output to text and enriched plots.

###  2.1 Descriptions and curve fitting

Let us present the overall idea and specific explainability targets we set in
the process of conversion of a data series to text. In this work we focus on
scalar data acquired on a time domain (represented by a sequence of floats).
Each element can be associated to a timestamp. A description of the data is a
simple and peculiar feature that generalizes a particular distribution along
time. A possible bridge from data to robust descriptions is to rely on fitting
functions. We select function prototypes that are fitted to data (i.e., least
square curve fitting) and that are described by few and intuitive parameters,
in order to promote a simple textual description. In the paper we refer to
_description/descriptor_ as the function model and its parameters, while we
refer to _narration_ as the corresponding natural language wording of such
description. As an example let us consider the concept of a valley in a
series: a sudden decrease followed by another sudden increase of values can be
modeled by a specific function (e.g., a V like function). The feature
extraction, based of curve fitting, allows to retrieve which optimal
parameters provide minimal Root Mean Square Error (RMSE) of the curve w.r.t.
the original data. Parameters, that control the degrees of freedom of the
shape of the curve, allow to build a high level narration. For example, if the
two sides of the valley have a high angular coefficient (or vertical
orientation), the valley can be described as sharp. Potential domain-dependent
descriptors can be encoded (e.g., specific patterns) and added with no
relevant modification of the system.

In our model, we allow an arbitrary large number of functions prototypes, to
support a modular and general usage. Each of them allows to investigate
particular behaviors. Each prototype requires to encode its fitting function
and its conversion to text as natural language description (we discuss about
it later). In our system we currently encoded (number of parameters inside
brackets): line (2), two segments polyline (4), a tooth like function (5) and
the most prominent sinusoid (3). We plan for the future to include the
analysis of transformed spaces (e.g., Fourier transform and wavelets) and
patterns occurring with characteristic shapes (e.g., EEG tracks).

Such choice allows to accomplish some steps towards explainability. In
particular, measuring the accuracy of a description, allows the system to
provide a self-judgement feedback. There is a measurable error (the fitting
accuracy) and a more qualitative error caused by conversion of descriptions
(numerical) to narrations (textual).

When interpreting the evolution of a measure along time, often we extrapolate
a multi-resolution description: in the simplest case we focus on the overall
trend and also on remarkable details. Therefore, we pre-process descriptors at
different sub-intervals, in order to capture potential features at different
time scales. Formally, we divide the time domain into regular $n=2^{\ell}$
zones. Each zone covers a subset of time samples and it is the smallest unit
for RMSE optimizations. The parameter $\ell$ (levels of divisions) controls
the amount of time resolution is provided to descriptors. Each descriptor $d$
covers a zone range from $i$ to $j$: $Z_{d}=\\{z:0\leq i\leq z\leq j<n\\}$. In
Figure 1 we show an example for some descriptors (bilinear, sinusoid and
squared peak), colored by zones size, for a series resulting to the search
term “Concert” in the last 5 years (from Google Trends) and normalized between
0 and 1. Given a descriptor $d$, we measure RMSE for each zone $i\in Z_{d}$,
namely $Err(d,i)$. The error is set to infinity for zones not in $Z_{d}$.

Figure 1: Example of three function prototypes fitting for 16 zones

###  2.2 Optimal descriptions covering

The problem of finding optimal descriptions that minimize the overall RMSE
error is rather trivial and it is the first phase in the overall optimization
process. In a practical scenario, we can expect thousands of description
candidates (set $\mathcal{D}$) to be processed. They originate from a rather
small set of function prototypes combined with a large set of sub-intervals of
$2^{\ell}$ to be investigated.

The goal is to represent the series through a set of descriptions
$D\subseteq\mathcal{D}$ out of the description pool. Each description $d$ has
an associated set of zones $Z_{d}$, the curve type, optimal parameters and the
RMSE $Err(d,i)$.

We wish to control the level of synthesis by an explicit parameter. In other
DL approaches this is rather complex to handle, especially in case of rather
long descriptions because of the incoherence in long text generation. We
introduce the concept of _verbosity_ ($v=|D|$) that defines the maximal number
of descriptions to be employed to describe the series. The verbosity
corresponds to the number of sentences used in the final narration.

The problem can be stated as follows. Given a verbosity $v$, select a set
$D_{v}$ such that $|D_{v}|=v$ and the sum of errors $Err(d,i)$ of each
description $d\in D_{v}$ for each zone $i\in Z_{d}$ is minimal. We also ask
for no zones overlaps ($Z_{d}\cap Z_{d}^{\prime}=\emptyset$, for each
$d,d^{\prime}\in D_{v}$) and that each description covers at least $n/2^{v}$
zones. This last requirement ensures that for low verbosity the descriptions
cover a sufficient time span, as a preference for general descriptions rather
than extremely small details.

In Figure 2 it is shown, for the same dataset of Figure 1, the optimization
results for $v\in[1..5]$. It can be noted that the red functions (descriptors)
provide a better accuracy on details as more descriptors can be included.
E.g., starting by a tooth function ($v=1$), the sharp peak reaching 1.0 is
better approximated with $v>3$. Figure 3 shows in a black-red-yellow palette
the RMSE for each zone (on the left, 16 zones, $\ell=4$) and for each $v$. It
can be noted how the error is significantly reduced by higher verbosity.

Figure 2: Example of optimal descriptions for verbosity 1–5

Figure 3: Example of optimal RMSE for verbosity 1–5 (left 16 zones, right 32
zones)

We believe that a fluent and effective description should find a balance
between general synthesis and offered details. Details should be selected in
order to avoid redundant information and/or no RMSE improvement w.r.t. larger
descriptions. Finally, relationships among descriptions (hierarchical
structure) could improve the structure of the narration. In conclusion,
selecting a low (general description) and a high (details) verbosity, e.g.,
$1$ and $v$, would result into an unnatural flat list of sentences, with the
possible outcome of repetitions and loss of interesting details.
Unfortunately, extending the previous model by these additional requirements
creates a combinatorial explosion that can not be easily handled, essentially
because every descriptor is a valid candidate.

###  2.3 The right balance between summary and details

We present the second optimization problem that models the creation of a
summary and a set of details that describe the series only where necessary and
accounting for the relationships among the other details.

We define _summary_ descriptors as the ones that cover the series with a
minimal verbosity and a specified accuracy. This selection allows to control
the minimum accuracy for cases in which the fitting functions encounter a
noisy dataset and/or a distribution very different from the prototypes.
Formally, the set $S=D_{s}$ of summary descriptors is selected among the
output of the previous optimization (sets $D_{1}\dots D_{v}$, for verbosity up
to $v$). We select the minimal verbosity level $s$ such that a threshold on
maximal RMSE allowed ($max\\_thr$) is satisfied by all zones:
$Err(d,i)<max\\_thr$ for $i\in Z_{d}$ and $d\in S$.

The decision variables of the optimization are the _detail_ descriptors. We
ask for at most $v$ details that will compose the set of details of the
narration. In principle, such details could be taken from $D_{v}$ (as
mentioned above), being $|D_{v}|=v$. We prefer selecting details from
$\cap_{i\leq v}D_{i}$, with the introduction of some constraints among them
and a cost function that accounts for accuracy improvements. The inclusion of
lower verbosity levels allows us to consider coarser descriptions of details
that may be suitable for narration as well as a more hierarchical narration
structure that covers some overlaps. Moreover, some details from $D_{v}$ can
be redundant and other aspects captured by lower verbosity could be
introduced. Finally, comparing different verbosity configurations, the
introduction of details can be balanced in terms of minimal amount of RMSE
improvement as opposed to the cost of the introduction itself.

Let us start modeling the introduction of details. If two descriptions
(including summary) taken from two different verbosity intersect on some
zones, i.e., there is an additional more detailed description, there must be a
minimal RMSE improvement on any zone in order to be willing to talk about the
detail. This threshold ($min\\_thr$) ensures that a new detail is introduced
for a minimal improvement. Formally, $d\in D_{i}$ and $d^{\prime}\in D_{j}$
with $i<j$ $\exists z.z\in Z_{d}\cap
Z_{d}^{\prime},Err(d,z)-Err(d^{\prime},z)>min\\_thr$.

Another constraint is that if a detail $d$ is selected, it can not be
completely covered in all of its zones $Z_{d}$ by other details of higher
verbosity. In this case $d$ would be completely re-worded and thus redundant
in the final textual description.

Finally, we design a maximization problem with a cost function related to the
amount of RMSE improvement provided by details. Ideally, we try to select
those details that improve the RMSE coverage of the summary. In particular, we
focus on each zone. If a zone $i$ is covered by at least a detail, the cost is
the difference between the best and the worst $Err(d,i)$ $\forall d.i\in
Z_{d}$. We also reduce the cost by a tiny penalty $p$ proportional to the
number of details that cover the same zone ($p=|{d.i\in Z_{d}}|$) in order to
favor more distributed coverings.

Let us conclude with a consideration about the decision variables size. The
current model needs only $\Sigma_{i\leq v}|D_{i}|=v(v-1)/2$ candidates. We
believe that exploiting this hierarchy provided by verbosity levels is rather
beneficial for an efficient selection of most relevant descriptors. If the two
optimization problems were joined, the decision variables would become in the
order of thousands, depending on the number of zones and function prototypes.

###  2.4 Towards natural language

We present now the final processing performed by the last ASP program. The
goal is to create a narration structure that orders and relates the
descriptions computed by the last optimization and to retrieve qualitative
properties associated to descriptions. This ASP program allows a single Answer
Set and it does not require any optimization. This crucial step allows to
clearly define the properties and rules in use, to deliver a transparent
program.

The paragraph structure is retrieved by computing lexicographic ordering of
summary and details descriptions, based on zones coverage. A narration
predicate enumerates properties associated to a description. In particular it
stores description location inside the narration order, grammar connective
relation with next description (immediate/separated), inclusion relation with
a description from lower verbosity and specific qualitative properties.

The larger part of this ASP code is devoted to the conversion of a description
into a set of qualitative properties (accompanied by some quantitative
measures). Each function prototype requires a dedicated ASP fragment that
produces a shape property that describes the type of function in relation to
its parameters. Moreover, it classifies qualitatively the extension of the
shape.

As an example, let us show in Figure 4, on the left, the mapping of parameters
for a description made of two consecutive segments into an interpreted shape.
In the lower left gray corner we depict two consecutive segments,
characterized by a normal $N$ angle and an aperture angle $A$. Given the
constraints of the three points forming the segments, $N\in[0..180]$ degrees
and $A\in[0..360]$ degrees. We report the case when the two segments are
rather similar in length; other cases are approximated by a single main line.
As an example, the combination $N=90$ and $A=270$ would result in a peak
(green area) with an smaller angle of 90 degrees. We depict the arrangements
of the two segments at various combination of the two angles. Gray areas
represent impossible combinations, since one of the two segments would invert
the time order along x-axis. The blue area is interpreted as rather constant
shape, being $A$ similar to 180. Adverbs that report on the quality the shape
(e.g., rather constant, moderately constant, etc.) can be generated and the
choice of intensity can be modulated by distances from optimum. Red and green
areas represent valleys and peaks, with a rather symmetric normal. White areas
represent drop and rise lines, with different flavours. Also, qualitative
amount of shape extension along y axis is associated to an ordered list of
adjectives (e.g., very mild, mild, steep, very steep) that are selected by
quantizing the measures and mapping them on the list. In the same Figure, on
the right, we report a fragment of ASP encoding for valley detection with
conversion of the numerical quantification of its depth to a qualitative
description (Strength in the last atom in the body of the first rule).

    select_bilin_type(ID,Angle,Normal,valley,Strength):-
          angle(Angle), angle2(Normal),  sol_desc(ID,_),
          |Angle-180|>=30,                 %%% not too flat
          Angle<180,       %%% there is a second order zero
          Normal+Angle/2 < 170,           %%% valley region
          10 < Normal+Angle/2,            %%% valley region
          adj_smooth_sharp(Strength,180-Angle,180).

    adj_smooth_sharp(very_smooth,V,Max):-   V*6<Max.
    adj_smooth_sharp(smooth,V,Max):-        V*6>=1*Max, V*6<2*Max.
    adj_smooth_sharp(rather_smooth,V,Max):- V*6>=2*Max, V*6<3*Max.
    ...

Figure 4: Example of mapping parameters (normal and angle) to type of curve

This _modus operandi_ , even if crafted for each function prototype, can
result in a robust general library. From the explainability point of view, the
error introduced from translating a description (still a mathematical function
with associated RMSE) to a qualitative narration, even if not easily
measurable, it can be manually assessed by ASP code analysis and conversion
maps like the one shown in Figure 4.

###  2.5 Narration of descriptions

Once the narration structure is output by last ASP program, the facts are
ready to be embedded in a text generation.

As a proof of concept, we used a Python procedure that follows the structure
and merges strings, to show that the only problem left is the actual string
processing. Connectives are handled according to the information received and
notably, no processing of relevant adjectives and nouns is performed anymore.
This is to show that with minimal imperative intervention is possible to
produce the text. A more fluent version could be reached if grammar rules,
variants, style rules and synonyms would be implemented in a more general ASP
program. This is planned as future work.

Another positive side effect of our processing is the capability of plotting
an enriched and summarized version of the series, based on the selected
descriptions, as shown in Figure 5. On the left we depict the summary function
(in this case a valley) and on the right we highlight some peaks and valleys.
On the bottom of the figures there is a red-green colored bar that reports the
relative errors of each zone (16 for this example), where the red value
corresponds to $max\\_thr$ (0.15 in this examples).

Figure 5: Example of enriched chart. On the left the Summary and on the right
the Details

The text associated to the Figure 5 is:

In general, the series presents a very deep valley reaching an average of 0.09
between 0.42 and 0.69 out of a general average of 0.42 among the whole
dataset. In detail, a rather sharp valley reaching a value of 0.24 occurs at
0.69; followed by a very steep increase reaching a value of 0.99 at 0.75; then
by a very sharp valley reaching a value of 0.45 at 0.81; and finally by a
sharp peak reaching a value of 0.39 at 1.0.

Notice that, as depicted in Figure 5, in processing the series we used a
normalized scale between 0 and 1 for the x axis. A further improvement could
map positions to timestamps for a more fluent processing of ranges and
positions in time.

##  3 Implementation

In implementing our tool, we took advantage of Potassco Clingo Python’s API.
The process is started by a Python script that downloads a tagged time series
from Google Trends service.

The main program consists of a Python configurator that sends the descriptor
facts to the Control object for grounding jointly with the first optimization
ASP program. For the control structure of the first optimization program, we
opted for an explicit branch and bound inner loop, since we want to recover
optimal covering at different verbosity levels. As soon as a solution is
found, the verbosity value is incremented by means of an external feature in
the outer loop.

The second optimization and description structured are located on the same ASP
program, while each function prototype conversion is stored on a separate ASP
file.

Results are plotted with Matplotlib and enriching visual description are
computed directly from the description structure output by the program.

The code is available at ahead-lab.unipr.it/files-for-iclp2023/

Figure 6: Comparison between 16 zones (left) and 32 zones (right). On top
details on the enriched plot and at bottom the heatmaps with accuracy of
descriptor candidates

##  4 Results

Let us present some results and discuss the impact of some parameters changes.
Defaults parameters are $max\\_thr=0.15$ and $min\\_thr=0.02$.

###  4.1 Impact of zones on accuracy

The number of zones clearly affects the quality of fitting functions as well
as the number of descriptors. We compare the results in terms of overall
accuracy of the narration (the sum of best contributions for each zone
position, as optimized by the second program). We compare 16 vs 32 zones, that
give rise to 408 and 1584 descriptors respectively (we used 3 functions
prototypes).

Starting with our running example of “Concert” series, we can see on Figure 3
the _heatmaps_ for 16 (left) and 32 (right) zones as results of the first
optimization. In general colors for 32 zones are darker, meaning that
descriptors are more accurate. Measuring the global RMSE (the sum for each
position of the best error covered by any selected description) gives a metric
to judge the accuracy of the output. Recall that the optimization goal
included a penalty for overlapped descriptions. Therefore, global RMSE could
be improved with a cost of more awkward narration. Nevertheless, penalties are
rather low and therefore global RMSE is not too distant from the optimization
cost.

Solution with 32 zones allows a selection of better descriptors and the ones
selected by second optimization improve the global RMSE (0.041 with 16 zones
and 0.036 with 32 zones). The global RMSE for 8 zones is 0.048, which is still
acceptable, given the ability of descriptors to find optimal fitting inside
the zone. However, when data has interesting patterns across zones, the
ability of capturing them is lost if too few zones are used. We found that a
good compromise could be 16 zones.

From a computational time point of view the complete program wall time,
including function fitting (80% of the time) is a couple of seconds for 8
zones, around 30 seconds for 16 zones and around 2 minutes for 32 zones on a
standard laptop.

Let us present a second test case where the “microsoft teams” search term was
retrieved on Google Trends. In this case global RMSE for verbosity 5 is 0.043
(8 zones), 0.030 (16 zones) and 0.024 (32 zones). In Figure 6 we depict on top
the two detail results for 16 zones (left) and 32 zones (right). It can be
noticed that more accurate zones allow to better describe the last two
features (valley and drop on the right) rather than corresponding a mild
decrease (left). In order to maintain the number of descriptions equal to 5,
there is an introduction of a small peak plateau (left) in place of two sharp
peaks (right). The overall RMSE is improved, as depicted on the green-red bar
at the bottom of the chart. The two heatmaps at the bottom of the figure show
the accuracy with a black-red-yellow scale depending on the zones. The
descriptors selected by the second optimization are colored green.

###  4.2 Verbosity comparison and text generation

We show here the impact of verbosity, which is the most influential control
for sentence summary.

We setup another test case with the series obtained with the search term
“blockchain” from Google Trends (last 5 years) with 16 zones. We launched the
program with verbosity 3, 4, and 5. Figure 7 depicts the corresponding
enriched plots. On the top left the summary, which is common to all solutions,
presents the main description (a deep plateau). On the top left we show
verbosity 3 with the presence of an initial drop, a narrow peak and a sharp
peak. It is interesting to note on bottom left (verbosity 4) that the initial
drop is better approximated by a sequence of two steep drops. At their
intersection a new peak is determined. As future work we plan to implement the
abstraction of consecutive functions, which could result in a higher level
identification of other patterns as post processing. Finally, at bottom right
the introduction of another description captures the last emerging peak at the
end of the series.

Figure 7: Enriched charts with different verbosity (3 = top right, 4 = bottom
left, 5 = bottom right). At top left the summary, shared by all solutions.

Let us report here the text generation from the selected descriptors.

Verbosity 3: In general, the series presents a very deep lower peak plateau
reaching an average of 0.12 between 0.16 and 0.57 out of a general average of
0.45 among the whole dataset. In detail, a steep drop reaching a value of 0.05
occurs at 0.44; followed by a very high peak reaching an average of 0.59
between 0.58 and 0.66 out of a general average of 0.17 between 0.44 and 0.69;
and finally by a sharp peak reaching a value of 0.1 at 1.0.

Verbosity 4: In general, the series presents a very deep lower peak plateau
reaching an average of 0.12 between 0.16 and 0.57 out of a general average of
0.45 among the whole dataset. In detail, a very steep decrease reaching a
value of 0.32 occurs at 0.25; followed by a steep drop reaching a value of
0.04 at 0.44; then by a very high peak reaching an average of 0.59 between
0.58 and 0.66 out of a general average of 0.17 between 0.44 and 0.69; and
finally by a sharp peak reaching a value of 0.1 at 1.0.

Verbosity 5: In general, the series presents a very deep lower peak plateau
reaching an average of 0.12 between 0.16 and 0.57 out of a general average of
0.45 among the whole dataset. In detail, a very steep decrease reaching a
value of 0.32 occurs at 0.25; followed by a steep drop reaching a value of
0.04 at 0.44; then by a very high peak reaching an average of 0.59 between
0.58 and 0.66 out of a general average of 0.17 between 0.44 and 0.69; a very
sharp peak reaching a value of 0.36 at 0.88; and finally by a very steep
decrease reaching a value of 0.13 at 1.0.

###  4.3 Conclusion

As future work we plan to optimize the Python fitting pre-processing, being
the most time demanding (several seconds). The goal is to optimize the system
to deliver almost real time answers. We also plan to develop a comprehensive
interpretation of consecutive descriptions, with the goal of inferring
additional descriptions. In this case, some higher-level description could
highlight interesting information that emerges, e.g., if two sharp peaks are
adjacent, a valley is probably in between. Another ASP program could take care
of the translation of the narration structure into natural language, porting
some Logic Programming ideas already appeared in the literature. In the
results, we showed a simple translation to natural language in Python, but a
richer rule-based model (e.g., a DCG in Prolog) could handle the variants and
nuances of a natural language in a more elegant and effective form.

In general, we plan to move from single time series and to extend the work in
order to handle multiple and synchronized time series. Such analysis could be
extended to high level interpretation of multiple source events that could be
described by an ontology. This could result in critical systems monitoring
(e.g., in healthcare), where explainability is fundamental. Moreover,
comparisons of series as well as comparisons of multidimensional data are
other domains for future investigation.

In conclusion, the paper presented an explainable methodology that produces a
natural language text, coming from a structured organization of descriptions,
as result of the interpretation of a time series. Reasoning about descriptions
is ruled by an ASP core (transparent and domain-independent): choices about
_what to say_ depend on an optimization problem that is controlled by an ASP
concise and transparent program; the definition of _how to say_ is handled by
an ASP program that organizes descriptions into a narration structure, assigns
qualitative evaluations and is able to abstract low level features into more
general ones. The output is associated to a RMSE measure on accuracy of each
sentence; the conversion from descriptors to qualitative interpretation is
described by means of a logic program; full control on minimal and maximal
errors is provided; verbosity is a key input parameter that models the whole
process; the narration structure is handled by a logic program and its
description can be easily converted to natural language and to enriched
graphs.

## References

  * [1]
  * [2] A. Adadi & M. Berrada (2018): _Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)_.  IEEE Access 6, pp. 52138–52160, 10.1109/ACCESS.2018.2870052.
  * [3] A. Balaji, T. Ramanathan & V. Sonathi (2018): _Chart-Text: A Fully Automated Chart Image Descriptor_.  CoRR abs/1812.10636.  Available at https://arxiv.org/abs/1812.10636.
  * [4] A. Barredo Arrieta, N.D. Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila & F. Herrera (2020): _Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI_.  Inf. Fusion 58, pp. 82–115, 10.1016/j.inffus.2019.12.012.
  * [5] European Commission (2021): _Proposal for a Regulation laying down harmonised rules on artificial intelligence_.  Available at https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence.
  * [6] M. Gebser, R. Kaminski, B. Kaufmann & T. Schaub (2012): _Answer Set Solving in Practice_.  Synthesis Lectures on Artificial Intelligence and Machine Learning, Morgan & Claypool Publishers.
  * [7] H. Harkous, I. Groves & A. Saffari (2020): _Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity_.  In D. Scott, N. Bel & C. Zong, editors: Proc. of the 28th International Conference on Computational Linguistics, Int. Committee on Computational Linguistics, Barcelona, Spain (Online), pp. 2410–2424, 10.18653/v1/2020.coling-main.218.
  * [8] K. Kukich (1983): _Design of a Knowledge-Based Report Generator_.  In M.P. Marcus, editor: 21st Annual Meeting of the Association for Computational Linguistics, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA, June 15-17, 1983, ACL, pp. 145–150, 10.3115/981311.981340.
  * [9] T. Liu, K. Wang, L. Sha, B. Chang & Z. Sui (2018): _Table-to-Text Generation by Structure-Aware Seq2seq Learning_.  In S.A. McIlraith & K.Q. Weinberger, editors: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, AAAI Press, pp. 4881–4888, 10.1609/aaai.v32i1.11925.
  * [10] J. Meszaros, J. Minari & I. Huys (2022): _The future regulation of artificial intelligence systems in healthcare services and medical research in the European Union_.  Frontiers in Genetics 13, 10.3389/fgene.2022.927721.  Available at https://www.frontiersin.org/articles/10.3389/fgene.2022.927721.
  * [11] S. Murakami, A. Watanabe, A. Miyazawa, K. Goshima, T. Yanase, H. Takamura & Y. Miyao (2017): _Learning to Generate Market Comments from Stock Prices_.  In R. Barzilay & M. Kan, editors: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, ACL, pp. 1374–1384, 10.18653/v1/P17-1126.
  * [12] J. Obeid & E. Hoque (2020): _Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model_.  In B. Davis, Y. Graham, J.D. Kelleher & Y. Sripada, editors: Proceedings of the 13th International Conference on Natural Language Generation, INLG 2020, Dublin, Ireland, December 15-18, 2020, Association for Computational Linguistics, pp. 138–147.  Available at https://aclanthology.org/2020.inlg-1.20.
  * [13] F. Portet, E. Reiter, A. Gatt, J. Hunter, S. Sripada, Y. Freer & C. Sykes (2009): _Automatic generation of textual summaries from neonatal intensive care data_.  Artif. Intell. 173(7-8), pp. 789–816, 10.1016/j.artint.2008.12.002.
  * [14] R. Puduppully, L. Dong & M. Lapata (2019): _Data-to-Text Generation with Content Selection and Planning_.  In: The 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, The 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, AAAI Press, pp. 6908–6915, 10.1609/aaai.v33i01.33016908.
  * [15] C. Rudin (2019): _Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead_.  Nature Machine Intell. 1(5), pp. 206–215, 10.1038/s42256-019-0048-x.
  * [16] USA Various Agencies (2016-2021): _USA National Artificial Intelligence Initiative_.  Available at https://www.ai.gov/.
  * [17] S. Wiseman, S.M. Shieber & A.M. Rush (2017): _Challenges in Data-to-Document Generation_.  In M. Palmer, R. Hwa & S. Riedel, editors: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, Association for Computational Linguistics, pp. 2253–2263, 10.18653/v1/D17-1239.


# ReCAB-VAE: Gumbel-Softmax
Variational Inference based on Analytic Divergence

Sangshin Oh
Yonsei University
<EMAIL_ADDRESS>
&Seyun Um
Yonsei University
<EMAIL_ADDRESS>
&Hong-Goo Kang
Yonsei University
<EMAIL_ADDRESS>

###### Abstract

The Gumbel-softmax distribution, or Concrete distribution, is often used to
relax the discrete characteristics of a categorical distribution and enable
back-propagation through differentiable reparameterization. Although it
reliably yields low variance gradients, it still relies on a stochastic
sampling process for optimization. In this work, we present a relaxed
categorical analytic bound (ReCAB), a novel divergence-like metric which
corresponds to the upper bound of the Kullback-Leibler divergence (KLD) of a
relaxed categorical distribution. The proposed metric is easy to implement
because it has a closed form solution, and empirical results show that it is
close to the actual KLD. Along with this new metric, we propose a relaxed
categorical analytic bound variational autoencoder (ReCAB-VAE) that
successfully models both continuous and relaxed discrete latent
representations. We implement an emotional text-to-speech synthesis system
based on the proposed framework, and show that the proposed system flexibly
and stably controls emotion expressions with better speech quality compared to
baselines that use stochastic estimation or categorical distribution
approximation.

##  1 Introduction

Inspired by recent successes in generating high fidelity images or audio using
deep neural networks, modeling and manipulating specific class information and
its fine attributes has become an active research area [1, 2, 3, 4, 5, 6].
Since many attributes of real world data such as object identity, gender, and
emotion are often annotated categorically, a natural choice is to model them
using categorical or discrete latent representations. When the discrete
structure of a data set is modeled in this manner, the latent representation
can be easily interpreted since each dimension represents one of the classes
or categories. For example, conditional generative adversarial networks
(cGANs) [1] and conditional variational autoencoders (cVAEs) [3] take
categorical auxiliary inputs, or _conditions_ , which are fed to both the
discriminator (or recognition) network and generator network to control the
attributes of the generated data.

An alternative approach was proposed by Kingma et al. [2] in which a
generative model was expanded to a variational approach in a semi-supervised
training framework. Here, it is possible to infer the class label by using an
auxiliary encoder even when the annotated label is not available. The
auxiliary encoder estimates the probability that a sample belongs to each
category, after which the estimated probability is used to calculate the
marginal likelihood of the sample. Recently, the Gumbel-softmax distribution
was proposed as a differentiable relaxation for discrete or categorical
variables [7, 8]. In this framework, unannotated labels are directly inferred
using relaxed categorical variables and back-propagated to the encoder using a
reparameterization trick. This allows for a significant reduction in inference
time because it does not need a marginalization over all possible latent
variables, unlike the semi-supervised method.

The relaxed categorical distribution is frequently applied for variational
inference [7, 8, 9, 10, 11, 12], in which a distribution of latent
representation estimated from observed data (a variational posterior) needs to
be fitted to the prior distribution. The Kullback-Leibler divergence (KLD)
[13] plays an important role in this optimization process because it
quantitatively measures the similarity between two distributions. The
calculation of the KLD is easy in the Gaussian distribution case because of
the existence of a simple analytic solution. However, no analytic solution
exists for the relaxed categorical distribution case; therefore, the KLD must
be calculated from a stochastic estimation process or approximated using a
normal categorical distribution.

In this work, we solve this problem by calculating the upper bound of the KLD
using a closed-form solution, which is more accurate than conventional
stochastic estimation or distribution approximation methods. Specifically, we
propose a relaxed categorical analytic bound variational autoencoder (ReCAB-
VAE), which has two distinctive features from previously suggested VAEs or the
jointVAE framework [7, 8, 9]. The relaxed categorical analytic bound (ReCAB),
which is the training objective for ReCAB-VAE, has a very similar value to the
true KLD of relaxed categorical distributions, but can be calculated
analytically and efficiently. The proposed metric is lower-bounded by the true
KLD, so it can be easily used to optimize the variational bound. In other
words, unlike conventional approximation-based approaches, the proposed metric
reliably sets a lower bound for the logarithmic evidence in the training
objective. In addition, we also propose a _discriminative prior_ that utilizes
the information from a class label of the data. Unlike uninformative priors
such as the standard Gaussian or uniform priors, the proposed discriminative
prior considers a class label to determine its corresponding parameters. By
giving distinctive information to its prior, it prevents the variational
posterior from collapsing into a non-informative uniform distribution. This
means that we do not need to provide any additional classification loss when
training the model because the latent representations are distinguishable due
to the guidance of the discriminative prior.

To show that our model is able to generate high fidelity samples while
flexibly controlling their attributes, we implement an emotional text-to-
speech (TTS) synthesis system with the proposed framework. An emotional TTS
system is a suitable application for this framework because it needs to
faithfully control emotional expressions as well as textual content. The
textual content is represented in the continuous space and the emotional
expressions are often annotated with class labels in the discrete latent space
[14, 15, 16]. Experimental results show that the synthesized speech quality of
the proposed system is much higher than that of conventional ones that use
stochastic estimation or categorical distribution approximation. In addition,
the proposed system stably controls emotional content while faithfully
maintaining the textual content.

The rest of this paper is organized as follows. In Section 2, we briefly
describe the fundamentals of the relaxed categorical distribution and the
jointVAE algorithm that uses both continuous and discrete latent spaces. In
Section 3, we explain the proposed ReCAB-VAE model in detail. Experiments and
results are described in Section 4, and conclusions follow in Section 5.

##  2 Preliminaries

We first describe the relaxation of categorical or discrete variables which
enables direct back-propagation through the stochastic node [7, 8]. Then, we
review the jointVAE algorithm that was recently developed to improve the
performance of conventional VAE approaches [9].

###  2.1 Relaxation of categorical random variables

Motivated by the Gumbel-Max trick [17], Jang et al. [7] and Maddison et al.
[8] concurrently suggested a relaxation method for categorical random
variables. If a categorical variable takes the class probability of
$(\alpha_{1},\alpha_{2},\dots,\alpha_{n})$ for each category, a relaxation of
this variable is defined as the softmax of
$\nicefrac{{\log\alpha_{k}+g_{k}}}{{\lambda}}$, where $g_{k}$ is independent
standard Gumbel noise and $\lambda$ is the temperature parameter. Precisely,
the $k$-th element of latent variable $z$ is defined as:

$\displaystyle
z_{k}=\frac{\exp\left(\nicefrac{{(\log\alpha_{k}+g_{k})}}{{\lambda}}\right)}{\sum_{{i}=1}^{n}\exp\left(\nicefrac{{(\log\alpha_{i}+g_{i})}}{{\lambda}}\right)}.$
(1)

Since the latent variable is properly reparameterized with the Gumbel noise,
the gradient can be calculated and back-propagated through this stochastic
node. This relaxation process is considered to be a generalized version of the
original categorical distribution because it converges to the exact
categorical distribution when the temperature $\lambda$ goes to 0. The
probability density function of this relaxation process is defined as follows:

$\displaystyle
p(z)=\Gamma(n)\lambda^{n-1}\prod_{k=1}^{n}\frac{\alpha_{k}z_{k}^{-\lambda-1}}{\sum_{{i}=1}^{n}\alpha_{i}z_{i}^{-\lambda}},$
(2)

where $\Gamma(\cdot)$ is a Gamma function. In this paper, we use $\alpha$ and
$\lambda$ to denote the logit and temperature parameters of the relaxed
categorical distribution, respectively.

###  2.2 JointVAE framework

Recently, Dupont [9] proposed the jointVAE model, which embeds its inputs in
both continuous and discrete latent spaces. Here, it was shown that it is
beneficial to model some attributes in a disentangled space from the
continuous latent space. The continuous and discrete latent variables are
assumed to be independent of each other, so the objective of this variational
model is given as follows:

$\displaystyle\mathbb{E}_{q_{\phi}(z_{c},z_{d}|x)}\big{[}\log
p_{\theta}(x|z_{c},z_{d})\big{]}-D_{KL}\left(q_{\phi}(z_{c}|x)\|p(z_{c})\right)-D_{KL}\left(q_{\phi}(z_{d}|x)\|p(z_{d})\right),$
(3)

where $z_{c}$ and $z_{d}$ refer to the continuous and discrete latent
variables, respectively, and $\phi$ and $\theta$ denotes encoder and decoder
parameters.

##  3 Relaxed Categorical Analytic Bound Variational Autoencoder

###  3.1 Relaxed Categorical Analytic Bound

We propose a relaxed categorical analytic bound (ReCAB), which is lower-
bounded by the KLD of the relaxed categorical distribution. To this end, we
decompose the equation from the definition of the KLD into a summation of
several expectation terms. Then, we reformulate or bound each term so that the
bound can be solved in a closed form. We first provide a brief mathematical
derivation for our proposed bound, and then compare it with conventional
estimation or approximation approaches for the KLD.

####  3.1.1 Derivation of the bound

From the density function in Eq. (2), the KLD of the relaxed categorical
distributions from prior $p(z)=\text{RelaxedCategorical}(a,l)$ to posterior
$q(z|x)=\text{RelaxedCategorical}(\alpha,\lambda)$ is defined as follows:

$\displaystyle\begin{split}D_{KL}\left(q(z|x)\|p(z)\right)&=\mathbb{E}_{z}\big{[}\log
q(z|x)-\log p(z)\big{]}\\\
&=\mathbb{E}_{z}\big{[}(n-1)\log\frac{\lambda}{l}+\sum_{{k}=1}^{n}\bigg{(}(\log\alpha_{k}-\lambda\log
z_{k})-(\log a_{k}-\lambda\log z_{k})\bigg{)}\\\
&\qquad{}-n\operatorname{LSE}_{{i}=1}^{n}\left(\log\alpha_{i}-\lambda\log
z_{i}\right)+n\operatorname{LSE}_{{i}=1}^{n}\left(\log a_{i}-\lambda\log
z_{i}\right)\big{]},\end{split}$ (4)

where $z_{k}$ is the $k$-th element of discrete latent variable $z_{d}$ and
$\operatorname{LSE}$ denotes a log-sum-exponential operator. In here, $a$ and
$l$ are the logit and temperature parameters of the prior distribution, while
$\alpha$ and $\lambda$ denote those of the posterior. Note that terms that
appear in both the posterior and prior cancel out. For notational simplicity
and clarity, we omit the subscript of the discrete latent variable $z$ and
succinctly write $\mathbb{E}_{z\sim q(z|x)}$ as $\mathbb{E}_{z}$. In order to
simplify the above equation, we can substitute logit terms with the following
equation, which is derived from Eq. (1).

$\displaystyle\begin{split}\log\alpha_{k}-\lambda\log
z_{k}&=-g_{k}+\lambda\operatorname{LSE}_{{i}=1}^{n}\left(\nicefrac{{(log\alpha_{i}+g_{i})}}{{\lambda}}\right)\\\
\log a_{k}-l\log z_{k}&=\log
a_{k}-\frac{l}{\lambda}\log\alpha_{k}+l\operatorname{LSE}_{{i}=1}^{n}\left(\nicefrac{{(log\alpha_{i}+g_{i})}}{{\lambda}}\right).\end{split}$
(5)

This results in the following equation:

$\displaystyle\begin{split}D_{KL}\left(q(z|x)\|p(z)\right)&=-(n-1)\log\frac{l}{\lambda}-\sum_{{k}=1}^{n}A_{k}-(1-\frac{1}{\lambda})\mathbb{E}_{g}\big{[}\sum_{{k}=1}^{n}g_{k}\big{]}\\\
&\qquad{}-n\cdot\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(g_{k}\right)\big{]}+n\cdot\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}-\frac{l}{\lambda}g_{k}\right)\big{]},\end{split}$
(6)

where $A_{k}=(\log a_{k}-\frac{l}{\lambda}\log\alpha_{k})$. Since the last
term cannot be solved analytically, we apply Jensen’s inequality:

$\displaystyle\begin{split}\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}-\frac{l}{\lambda}g_{k}\right)\big{]}&\leq\log\mathbb{E}_{g}\big{[}\sum_{{k}=1}^{n}\left(\exp
A_{k}\cdot\exp(-\frac{l}{\lambda}g_{k})\right)\big{]}\\\
&=\log\sum_{{k}=1}^{n}\left(\exp
A_{k}\cdot\mathbb{E}_{g}\big{[}\exp(-\frac{l}{\lambda}g_{k})\big{]}\right),\end{split}$
(7)

where the last RHS term is obtained by reordering the expectation and
summation operations. Since the expectation term becomes a constant value once
evaluated, the upper bound of the last RHS term can be solved analytically. By
calculating all of the remaining expectations, it is possible to obtain the
upper bound of the KLD as follows:

$\displaystyle\begin{split}D_{ReCAB}\big{(}q(z|x)\|p(z)\big{)}&=-(n-1)\log\frac{l}{\lambda}+n\left(\gamma\frac{l}{\lambda}+\log\Gamma(1+\frac{l}{\lambda})-\sum^{n-1}_{k=1}\frac{1}{k}\right)\\\
&\qquad{}-\sum_{{k}=1}^{n}\text{log-
softmax}_{k}(\log{a}-\frac{l}{\lambda}\log{\alpha})\\\ &\geq
D_{KL}\left(q(z|x)\|p(z)\right).\end{split}$ (8)

Since the last $\text{log-softmax}(\cdot)$ term is the only term that is
related to logits $\alpha$ and $a$, the other terms in Eq. (8) can be
calculated in advance when the temperatures are fixed throughout training.
Therefore, it is easy to implement a training algorithm because we only need
to evaluate one function at each epoch. The general algorithm for training is
depicted in Algorithm 1.

Detailed derivations are provided in the supplementary material.

####  3.1.2 Comparison with conventional approaches

In previous works, the KLD of the relaxed categorical distribution was
estimated via the Monte-Carlo method (Eq. (9), MC) or roughly approximated by
using the KLD of the categorical distribution (Eq. (10), CA) as follows:

$\displaystyle D_{KL}\left(q(z|x)\|p(z)\right)$
$\displaystyle\approx\sum^{L}_{i=1}\bigg{(}\log q(z^{(i)}|x)-\log
p(z)\bigg{)}=D_{MC}$ (9)
$\displaystyle\approx\sum_{{k}=1}^{n}\hat{\alpha}_{k}\bigg{(}\log\hat{\alpha}_{k}-\log{a}_{k}\bigg{)}=D_{CA},$
(10)

where $L$ is the number of samples for stochastic estimation and
$\hat{\alpha}_{k}$ denotes the $k$-th element of the normalized logit
parameter or categorical probability,
$\hat{\alpha}_{k}=\nicefrac{{\alpha_{k}}}{{\sum_{{i}=1}^{n}\alpha_{i}}}$. Eq.
(9) must be evaluated from the density function given in Eq. (2), so it is
somewhat complicated to calculate. In addition, it provides noisy and
inaccurate values for the KLD when $L$ is not large enough. Because of this
complexity and inaccuracy, it is a tempting choice to use the categorical
distribution to approximate the KLD as in Eq. (10). Unlike stochastic
estimation, this gives the approximated value in a deterministic and efficient
way. However, because the categorical distribution does not have any
temperature or scale parameters, this approximation cannot consider the
temperature parameter of the relaxed categorical distribution. Therefore, this
method only provides a rough approximation for the KLD which is inaccurate and
results in an irregular loss landscape. In addition, as pointed out in
previous works [7, 8], the training objective with this approximation is not
guaranteed to be an evidence lower bound.

###  3.2 Discriminative prior

In this section, we propose a _discriminative prior_ which can be used for
semi-supervised or supervised training of ReCAB-VAE. Unlike the uniform
distribution or its relaxed variants (e.g., a distribution with
$\alpha=[\frac{1}{n},\frac{1}{n},\cdots,\frac{1}{n}]$,
$\lambda\in(0,\infty)$), this prior provides some information about data
points as in Sohn et al. [3] or Casale et al. [18]. Since the prior
distribution is now conditioned on auxiliary code or label $c$, the evidence
lower bound is revised as follows:

$\displaystyle\begin{split}\mathcal{L}_{ELBo}&=\mathbb{E}_{z}\big{[}\log
p_{\theta}(x,z|c)-\log q_{\phi}(z|x)\big{]}\\\ &=\mathbb{E}_{z}\big{[}\log
p_{\theta}(x|z)\big{]}-D_{KL}\big{(}q_{\phi}(z|x)\|p(z|c)\big{)},\end{split}$
(11)

where $p(z|c)$ is the proposed discriminative prior. Note that we assume the
code $c$ to be independent of both the encoder $q_{\phi}$ and decoder
$p_{\theta}$ here. The proposed prior is a usual relaxed categorical
distribution whose logit $a$ is a function of the auxiliary code
$f_{a}(c;\epsilon)$. The function for the logit parameter can be defined as
follows:

$\displaystyle\begin{split}f_{a}(c;\epsilon)=\begin{cases}1-(n-1)\epsilon&\text{if
}c_{i}=1\\\ \epsilon&\text{otherwise}\end{cases},\end{split}$ (12)

where $c\in\\{0,1\\}^{n}$ is a one-hot discrete code and $\epsilon>0$ is a
small smoothing constant. The resulting prior is defined as
$p(z|c)=\text{GumbelSoftmax}(f_{a}(c;\epsilon),l)$, where the temperature $l$
is selected freely and usually fixed throughout training. Using this prior,
the model can learn distinctive latent representations for each class without
auxiliary classification loss, unlike in Kingma et al. [2] or Habib et al.
[15].

###  3.3 Model structure

$\theta$, $\phi$ $\leftarrow$ Initialize parameters

$l$, $\lambda$ $\leftarrow$ Set temperatures for prior and posterior
distributions

$\mathcal{L}_{1}$ $\leftarrow$
$-(n-1)\log\frac{l}{\lambda}+n\gamma\frac{l}{\lambda}+\log\Gamma(1+\frac{l}{\lambda})-\sum^{n-1}_{k=1}\frac{1}{k}$

while  _not converges_ do

$\mu$, $\sigma$, $\alpha$ $\leftarrow$ Encoder($x$; $\phi$)

$z_{c}$, $z_{d}$ $\leftarrow$ Sample via reparameterization trick

$\hat{x}$ $\leftarrow$ Decoder($z_{c}$, $W_{d}z_{d}$; $\theta$)

$\mathcal{L}_{2}$ $\leftarrow$ $\mathbb{E}\big{[}\log
p_{\theta}(x|z_{c},z_{d})\big{]}-D_{KL}\big{(}q_{\phi}(z_{c}|x)\|p(z_{c})\big{)}-\text{log-
softmax}(\log a-\frac{l}{\lambda}\log\alpha$)

$\mathcal{L}$ $\leftarrow$ $\mathcal{L}_{1}+\mathcal{L}_{2}$

$\theta$, $\phi$ $\leftarrow$ Update parameters with
$\nabla_{\theta,\phi}\mathcal{L}_{2}$

end while

Algorithm 1 Training procedure for ReCAB-VAE. Note that Encoder is separated
into continuous and discrete parts in a real implementation. $\mathcal{L}_{1}$
can be ignored in parameter updates since it is a constant throughout the
training procedure.

In this section, we propose ReCAB-VAE, a relaxed categorical analytic bound
variational autoencoder based on ReCAB and the aforementioned discriminative
prior. Although we apply our proposed features in a VAE framework, the
features explained above can be adopted in other frameworks that use the
relaxed categorical distribution. Notably, ReCAB can be used as a closed-form
surrogate for the KLD of the relaxed categorical distribution.

ReCAB-VAE consists of two independent latent encoders as in jointVAE [9]: a
continuous latent and a discrete latent encoder. Assuming independence between
the continuous and discrete latent variables, the training objective
$\mathcal{L}_{Prop}$ is written as follows:

$\displaystyle\begin{split}\mathcal{L}_{Prop}&=\mathbb{E}\big{[}\log
p_{\theta}(x|z_{c},z_{d})\big{]}-D_{KL}\big{(}q_{\phi}(z_{c}|x)\|p(z_{c})\big{)}-D_{ReCAB}\big{(}q_{\phi}(z_{d}|x)\|p(z_{d}|c)\big{)}\\\
&\leq\mathbb{E}\big{[}\log
p_{\theta}(x|z_{c},z_{d})\big{]}-D_{KL}\big{(}q_{\phi}(z_{c}|x)\|p(z_{c})\big{)}-D_{KL}\big{(}q_{\phi}(z_{d}|x)\|p(z_{d}|c)\big{)}\\\
&=\mathcal{L}_{ELBo}\end{split}$ (13)

where $c$ is an additional condition code for the discriminative prior. Note
that our proposed objective is bounded above by ELBo. With the learning
objective specified above, a structure for the proposed model is as follows:

$\displaystyle[\mu;\log\sigma^{2}]$ $\displaystyle=\text{Cont.Encoder}(x)$
$\displaystyle\alpha$ $\displaystyle=\text{Disc.Encoder}(x)$ $\displaystyle
z_{c}$ $\displaystyle\sim\mathcal{N}(\mu,\sigma^{2})$ $\displaystyle z_{d}$
$\displaystyle\sim\text{RelaxedCategorical}(\alpha,\lambda)$ (14)
$\displaystyle\hat{x}$ $\displaystyle=\text{Decoder}(z_{c},W_{d}z_{d}),$

where $\hat{x}$ is the reconstruction of sample $x$ and $\lambda$ is fixed
throughout the training process as in previous works [7, 8]. Both latent
encoders, Cont.Encoder and Disc.Encoder, take the data sample $x$ and
represent it in parallel in the latent spaces. The latent variables are then
sampled via the reparameterization trick. In order to match the dimension of
continuous and discrete latent representations, we project the discrete latent
variable with embedding matrix $W_{d}$. Then, the decoder reconstructs the
data sample from the continuous latent representation $z_{c}$ and projected
discrete latent representation $W_{d}z_{d}$.

##  4 Experiments and Results

In this section, we show that ReCAB provides a better estimation of the true
KLD compared to conventional stochastic and distribution approximation
approaches. We also conduct experiments to evaluate the performance of an
emotional text-to-speech (TTS) system that is implemented using the proposed
ReCAB-VAE framework.

###  4.1 Basic empirical studies

Figure 1: Heatmaps of KLD estimations from each method when the target logit
is [0.57, 0.14, 0.28]. The temperatures for the target and proposal
distributions are equivalently set to 1. Each point in the heatmap indicates
the logit of the proposal distribution after normalization. (a) True KLD
obtained from MC estimation with 300k samples, (b) MC estimation with a small
batch size (32 samples), (c) CA estimation, (d) ReCAB. Figure 2: Density plots
of target and optimum points from each method. The temperature of the target
distribution was set to 0.2, while those of all other distributions were set
to 1. (a) Target distribution with logit of [0.57, 0.14, 0.28]. Densities for
optimum logit from (b) MC with 32 samples, (c) CA, and (d) ReCAB.

Figure 1 shows 2-dimensional simplex heatmaps that depict the true KLD (a) and
estimated values from three different methods (b-d): a Monte-Carlo (MC)
estimator, a categorical distribution approximation (CA) estimator, and the
proposed ReCAB estimator. The three vertices of each heatmap denote the logits
for the relaxed categorical distribution representing the three categories.
The color at each point represents the estimated KLD with the target
distribution. As shown in the figure, the estimated results obtained by the
ReCAB estimator are almost identical to those computed using the exact KLD. On
the other hand, MC estimation with a small batch size results in noisy
outputs, and CA estimation results in a differently shaped, distorted heatmap.

Figure 2 illustrates the density plots of optimal logit parameters from the
MC, CA, and ReCAB estimators. When the temperatures are set to be identical to
the prior (or target), the densities from all the estimators fit well to the
target distribution (shown in supplementary material). When the temperatures
are different from the prior, however, each density plot shows distinctive
results. As shown in Eq. (10), the categorical approximation causes the local
optimum to ignore the temperature parameter; this results in a density that is
very different from the target distribution. Meanwhile, MC estimation and
ReCAB show relatively similar densities to the target distribution, but MC
estimation results in a slightly worse optimum because of its noisiness.

###  4.2 Emotional speech synthesis

We performed experiments on emotional speech synthesis using an internal
emotional TTS database consisting of 26.4 hours of speech. The database
contains 21,678 utterances, where each sentence is recorded by a male and a
female speaker. About one third of the utterances (6,604) are annotated with
one of 4 emotional classes (angry, happy, neutral, sad). The others are not
annotated because of ambiguity in labeling.

##### Architectures.

To generate emotional speech with our proposed model, we chose Tacotron 2 [19]
as our baseline system, an end-to-end TTS system based on an encoder-decoder
framework. Similar to previous approaches for Tacotron-based controllable
speech synthesis [15, 20, 21, 22, 23], our ReCAB-VAE model is an augmentation
to the baseline that allows control over the emotional expression of the
generated speech signal. In particular, we borrowed the _reference encoder_
architecture for the latent encoder of ReCAB-VAE. Since we have two separate
encoders for continuous and discrete latent representations, we halve the
number of channels in the convolutional layers and the number of cells in the
recurrent layer. The representations from each latent encoder are then
concatenated and added to the embedding from the text encoder of Tacotron 2.
We compared versions of this model trained with Monte-Carlo estimation (MC),
categorical distribution approximation (CA) and ReCAB. We used WaveGlow [24]
as a neural vocoder to transform the predicted mel-spectrogram into waveforms
for all models.

####  4.2.1 Subjective tests

We conducted two subjective tests to evaluate the performance of our proposed
system. In the first subjective test, raters were asked to assess the overall
quality of the synthesized speech signal on a scale from 1 to 5 to obtain the
mean opinion score (MOS). The second test was an ABX test in which raters were
asked to choose one of two speech samples from different models according to
their preferences. Emotion labels were shown to the raters in the ABX test,
while they were not shown in the MOS test.

##### Mean opinion score test.

Table 1: Speech quality and naturalness mean opinion scores (MOS) with 95% confidence intervals. Neutral voices indicates scores that only consider speech samples with the neutral emotion, while all of the speech samples are considered in Total. VAEs with Monte-Carlo estimation (MC) and normal categorical distribution approximation (CA) have same architecture as ReCAB-VAE, but are trained with different training objectives. Model | Neutral voices | Total
---|---|---
Recorded | 4.84 $\pm$ 0.10 | 4.87 $\pm$ 0.05
GT-Mel | 4.69 $\pm$ 0.08 | 4.57 $\pm$ 0.09
Tacotron 2 | 3.58 $\pm$ 0.13 | 3.50 $\pm$ 0.13
VAE-Tacotron | 3.76 $\pm$ 0.12 | 3.47 $\pm$ 0.13
VAE w/ MC | 3.61 $\pm$ 0.13 | 3.41 $\pm$ 0.13
VAE w/ CA | 3.42 $\pm$ 0.13 | 3.30 $\pm$ 0.14
ReCAB-VAE | 3.97 $\pm$ 0.12 | 3.71 $\pm$ 0.13

In this test, raters were directed to consider the overall quality and
naturalness of the speech signals, and assessed them on a 5-point scale with
1-point granularity. Each rater listened to and rated 18 sentences for each
system. In order to provide a reference score, we included real human voices
and the waveforms which were directly reconstructed from ground truth mel-
spectrograms (shown as Recorded and GT-Mel in Table 1, respectively). We used
Tacotron 2 with and without Gaussian VAE [25] as the baseline. The results are
summarized in Table 1.

For all of the models except the recorded signal, the total MOS is slightly
lower than the corresponding neutral voice MOS. Applying a continuous VAE
(VAE-Tacotron) results in better scores as a result of the flexibility in
controlling continuous latent representation. On the other hand, when the
discrete VAE is augmented and trained with the KLD from categorical
approximation (CA) or stochastic estimation (MC), the results do not improve
over the baseline Tacotron 2 model. When the KLD is approximated by the normal
categorical distribution (CA), the naturalness and quality are even worse than
the baseline with no latent encoder. This is due to the inaccurate values from
the CA method, which makes the training objective unable to manage the trade-
off between reconstruction and regularization. In the case of the MC method,
performance slightly improves, but is still worse than the continuous-only
VAE-Tacotron. This is because of the halved dimension of the continuous latent
representation and the poor quality of the discrete latent representation
resulting from the inaccurate KLD estimation.

ReCAB shows the best results, which are a significant improvement over the
baseline. Similar to the VAE with MC estimation, the training objective with
ReCAB is a lower bound of the variational bound. Since ReCAB provides accurate
values for the KLD, ReCAB-VAE is able to synthesize natural and high-fidelity
speech samples. This improvement is obtained without any architectural
changes, showing that ReCAB acts as a better metric for estimating the KLD of
the relaxed categorical distribution.

##### ABX preference test.

Figure 3: ABX preference test results. Proposed in the figure indicates the
model using ReCAB-VAE, and N/P stands for no preference. Samples from the
proposed model are preferred over those from both baseline models.

In the ABX preference test, two speech samples of the same sentence and same
emotion from different models were presented to raters, who listened to both
samples and were asked to make one of the following choices: (A) sample A is
preferred, (B) sample B is preferred, or (X) no preference. To evaluate
emotional expression and tonal variations, raters considered both emotional
expressiveness and naturalness, but were directed to prioritize emotional
expression. Speech samples from ReCAB-VAE were shown to have higher preference
percentages over both the VAE with CA and VAE with MC in these head-to-head
comparisons. The results of the test are presented in Figure 3.

##  5 Conclusion

In this paper, we have proposed a relaxed categorical analytic bound (ReCAB)
with a closed-form solution for estimating the Kullback-Leibler divergence
(KLD) of a relaxed categorical distribution. Since ReCAB can be solved
analytically, it does not depend on a sampling procedure for latent variables
and is able to provide a reliable estimation of the KLD. We also proposed a
discriminative prior, which enables discriminative learning in variational
autoencoders without an additional loss function. To evaluate the performance
of the proposed metric and prior distribution, we adopted them in a
variational autoencoder framework, which we refer to as ReCAB-VAE. In an
emotional speech synthesis task, ReCAB-VAE was able to generate high fidelity
speech samples with rich emotional expressions, and outperformed baseline
models in terms of both speech quality and emotion despite only a change to
the training objective. This indicates that ReCAB is an effective metric for
training models that embed latent representations in both continuous and
discrete latent spaces, and can improve their performance in various other
tasks.

## References

  * Mirza and Osindero [2014] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ , 2014.
  * Kingma et al. [2014] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling.  Semi-supervised learning with deep generative models.  In _Advances in neural information processing systems_ , pages 3581–3589, 2014.
  * Sohn et al. [2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  In _Advances in neural information processing systems_ , pages 3483–3491, 2015.
  * Van den Oord et al. [2016] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al.  Conditional image generation with pixelcnn decoders.  In _Advances in neural information processing systems_ , pages 4790–4798, 2016.
  * Luong et al. [2017] Hieu-Thi Luong, Shinji Takaki, Gustav Eje Henter, and Junichi Yamagishi.  Adapting and controlling dnn-based speech synthesis using input codes.  In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 4905–4909. IEEE, 2017.
  * Choi et al. [2018] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8789–8797, 2018.
  * Jang et al. [2016] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Maddison et al. [2016] Chris J Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  _arXiv preprint arXiv:1611.00712_ , 2016.
  * Dupont [2018] Emilien Dupont.  Learning disentangled joint continuous and discrete representations.  In _Advances in Neural Information Processing Systems_ , pages 710–720, 2018.
  * Yang et al. [2017] Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick.  Improved variational autoencoders for text modeling using dilated convolutions.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 3881–3890. JMLR. org, 2017.
  * Siddharth et al. [2017] Narayanaswamy Siddharth, Brooks Paige, Jan-Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr.  Learning disentangled representations with semi-supervised deep generative models.  In _Advances in Neural Information Processing Systems_ , pages 5925–5935, 2017.
  * Kipf et al. [2018] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel.  Neural relational inference for interacting systems.  _arXiv preprint arXiv:1802.04687_ , 2018.
  * Kullback and Leibler [1951] Solomon Kullback and Richard A Leibler.  On information and sufficiency.  _The annals of mathematical statistics_ , 22(1):79–86, 1951.
  * Lee et al. [2017] Younggun Lee, Azam Rabiee, and Soo-Young Lee.  Emotional end-to-end neural speech synthesizer.  _arXiv preprint arXiv:1711.05447_ , 2017.
  * Habib et al. [2019] Raza Habib, Soroosh Mariooryad, Matt Shannon, Eric Battenberg, RJ Skerry-Ryan, Daisy Stanton, David Kao, and Tom Bagby.  Semi-supervised generative modeling for controllable speech synthesis.  _arXiv preprint arXiv:1910.01709_ , 2019.
  * Kwon et al. [2019] Ohsung Kwon, Inseon Jang, ChungHyun Ahn, and Hong-Goo Kang.  An effective style token weight control technique for end-to-end emotional speech synthesis.  _IEEE Signal Processing Letters_ , 26(9):1383–1387, 2019.
  * Gumbel [1954] Emil Julius Gumbel.  Statistical theory of extreme values and some practical applications.  _NBS Applied Mathematics Series_ , 33, 1954.
  * Casale et al. [2018] Francesco Paolo Casale, Adrian Dalca, Luca Saglietti, Jennifer Listgarten, and Nicolo Fusi.  Gaussian process prior variational autoencoders.  In _Advances in Neural Information Processing Systems_ , pages 10369–10380, 2018.
  * Shen et al. [2018] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 4779–4783. IEEE, 2018.
  * Skerry-Ryan et al. [2018] RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron J Weiss, Rob Clark, and Rif A Saurous.  Towards end-to-end prosody transfer for expressive speech synthesis with tacotron.  _arXiv preprint arXiv:1803.09047_ , 2018.
  * Wang et al. [2018] Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ Skerry-Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Fei Ren, Ye Jia, and Rif A Saurous.  Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis.  _arXiv preprint arXiv:1803.09017_ , 2018.
  * Zhang et al. [2019] Ya-Jie Zhang, Shifeng Pan, Lei He, and Zhen-Hua Ling.  Learning latent representations for style control and transfer in end-to-end speech synthesis.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6945–6949. IEEE, 2019.
  * Sun et al. [2020] Guangzhi Sun, Yu Zhang, Ron J Weiss, Yuan Cao, Heiga Zen, and Yonghui Wu.  Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis.  _arXiv preprint arXiv:2002.03785_ , 2020.
  * Prenger et al. [2019] Ryan Prenger, Rafael Valle, and Bryan Catanzaro.  Waveglow: A flow-based generative network for speech synthesis.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 3617–3621. IEEE, 2019.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.

## Supplementary Material

##  A Detailed Derivation of ReCAB

###  A.1 Detailed Derivation

We start from the definition of the KLD of relaxed categorical distribution.

$\displaystyle\begin{split}D_{KL}\big{(}q(z|x)\|p(z)\big{)}&=\mathbb{E}_{z}\big{[}-(n-1)\log\frac{l}{\lambda}+\sum_{{k}=1}^{n}\big{(}(\log\alpha_{k}-\lambda\log
z_{k})-(\log a_{k}-l\log z_{k})\big{)}\\\
&\qquad-n\operatorname{LSE}_{{i}=1}^{n}\left(\log\alpha_{i}-\lambda\log
z_{i}\right)+n\operatorname{LSE}_{{i}=1}^{n}\left(\log a_{i}-l\log
z_{i}\right)\big{]},\end{split}$ (s.1)

where $\alpha$ and $\lambda$ are parameters of posterior distribution
$q(z|x)$, and $a$ and $l$ are for prior $p(z)$. From the definition of the
$k$-th element of relaxed categorical random variable $z_{k}$, we have:

$\displaystyle\begin{split}\log\alpha_{k}-\lambda\log
z_{k}&=-g_{k}+\lambda\operatorname{LSE}_{{i}=1}^{n}\left(\frac{\log\alpha_{i}+g_{i}}{\lambda}\right)\\\
\log a_{k}-l\log z_{k}&=\log
a_{k}-\frac{l}{\lambda}\log\alpha_{k}-\frac{l}{\lambda}g_{k}+l\operatorname{LSE}_{{i}=1}^{n}\left(\frac{\log\alpha_{i}+g_{i}}{\lambda}\right),\end{split}$
(s.2)

where $g_{k}$ in here is i.i.d. standard Gumbel noise. Since the
$\operatorname{LSE}_{{i}=1}^{n}\left(\cdot\right)$ term is constant with
respect to $k$, following equations can be derived:

$\displaystyle\begin{split}\operatorname{LSE}_{{k}=1}^{n}\left(\log\alpha_{k}-\lambda\log
z_{k}\right)&=\operatorname{LSE}_{{k}=1}^{n}\left(-g_{k}\right)+\lambda B\\\
\operatorname{LSE}_{{k}=1}^{n}\left(\log a_{k}-l\log
z_{k}\right)&=\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}-\frac{l}{\lambda}g_{k}\right)+lB,\end{split}$
(s.3)

where $A_{k}=\log a_{k}-\frac{l}{\lambda}\log\alpha_{k}$ and
$B=\operatorname{LSE}_{{i}=1}^{n}\left(\frac{\log\alpha_{i}+g_{i}}{\lambda}\right)$.
Applying the above equations, the KLD have the following form:

$\displaystyle\begin{split}D_{KL}\big{(}q(z|x)\|p(z)\big{)}&=\mathbb{E}_{g}\big{[}-(n-1)\log\frac{l}{\lambda}+\sum_{{k}=1}^{n}\big{\\{}\big{(}-g_{k}+\lambda
B\big{)}-\big{(}A_{k}-\frac{l}{\lambda}g_{k}+lB\big{)}\big{\\}}\\\
&\qquad{}-n\cdot\big{(}\operatorname{LSE}_{{k}=1}^{n}\left(-g_{k}\right)+\lambda
B-\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}-\frac{l}{\lambda}g_{k}\right)-lB\big{)}\big{]}.\\\
&=-(n-1)\log\frac{l}{\lambda}-\sum_{{k}=1}^{n}A_{k}-(1-\frac{l}{\lambda})\mathbb{E}_{g}\big{[}\sum_{{k}=1}^{n}g_{k}\big{]}\\\
&\qquad{}-n\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(-g_{k}\right)\big{]}+n\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}-\frac{l}{\lambda}g_{k}\right)\big{]}.\end{split}$
(s.4)

From Jensen’s inequality, we can derive the bound of the last term in above
equation.

$\displaystyle\begin{split}\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}-\frac{l}{\lambda}g_{k}\right)\big{]}&\leq\log\big{(}\mathbb{E}_{g}\big{[}\sum_{{k}=1}^{n}\exp(A_{k}-\frac{l}{\lambda}g_{k})\big{]}\big{)}\\\
&=\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}\right)+\log\mathbb{E}_{g}\big{[}\exp(-\frac{l}{\lambda}g_{k})\big{]}\big{)}\end{split}$
(s.5)

Now, we have the inequality for the KLD as following:

$\displaystyle\begin{split}D_{KL}\big{(}q(z|x)\|p(z)\big{)}&\leq-(n-1)\log\frac{l}{\lambda}-\sum_{{k}=1}^{n}\big{(}A_{k}-\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}\right)\big{)}-(1-\frac{l}{\lambda})\mathbb{E}_{g}\big{[}\sum_{{k}=1}^{n}g_{k}\big{]}\\\
&\qquad{}-n\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(-g_{k}\right)\big{]}+n\log\mathbb{E}_{g}\big{[}\exp(-\frac{l}{\lambda}g_{k})\big{]}.\end{split}$
(s.6)

The expectation terms in above equation have closed-form solutions.

##### First expectation: $\mathbb{E}_{g}\big{[}\sum_{{k}=1}^{n}g_{k}\big{]}$.

Since the expectation of sum of random variables is equivalent to sum of
expectations,

$\displaystyle\mathbb{E}_{g}\big{[}\sum_{{k}=1}^{n}g_{k}\big{]}=n\mathbb{E}_{g}\big{[}g\big{]}=n\gamma,$
(s.7)

where $\gamma\approx 0.57721$ is Euler-Mascheroni constant.

##### Second expectation:
$\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(-g_{k}\right)\big{]}$.

To evaluate this expectation, we first prove that $h=\exp(-g)$ follows
exponential distribution. From the CDF of Gumbel distribution
$F_{G}(g^{\prime})=\exp(\exp(-g^{\prime}))$, the CDF of $h$ is

$\displaystyle\begin{split}F_{H}(h^{\prime})&=P(h\leq
h^{\prime})=P(\exp(-g)\leq h^{\prime})=P(g\geq-\log h^{\prime})\\\
&=1-P(g\leq-\log h^{\prime})=1-F_{G}(-\log h^{\prime})\\\
&=1-\exp(-h^{\prime}),\end{split}$ (s.8)

which is a CDF of exponential distribution with scale parameter of 1,
$\text{Exp}(1)$. The summation over $h$, i.e.,
$\sum_{{k}=1}^{n}h_{k}=\sum_{{k}=1}^{n}\exp(-g_{k})$, follows gamma
distribution, whose logarithmic expectation is

$\displaystyle\begin{split}\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(-g_{k}\right)\big{]}=\psi(n),\end{split}$
(s.9)

where $\psi(n)$ is a digamma function. The digamma function is a log-
derivative of gamma function $\Gamma(\cdot)$ and is calculated as
$-\gamma+\sum_{k=1}^{n-1}\frac{1}{k}$ for $n\in\mathbb{N}$.

##### Third expectation:
$\mathbb{E}_{g}\big{[}\exp(-\frac{l}{\lambda}g_{k})\big{]}$.

The exponential of weighted Gumbel random variable,
$w=\exp(-\frac{l}{\lambda}g)=h^{\frac{l}{\lambda}}$, follows Weibull
distribution. The CDF of $w$ is

$\displaystyle\begin{split}F_{W}(w^{\prime})&=P(w\leq
w^{\prime})=P(h^{\frac{l}{\lambda}}\leq
w^{\prime})=P(h\leq(w^{\prime})^{\frac{\lambda}{l}})\\\
&=1-\exp(-(w^{\prime})^{\frac{\lambda}{l}}),\end{split}$ (s.10)

which is a CDF of $\text{Weibull}(1,\frac{\lambda}{1})$. The expectation of
the distribution is

$\displaystyle\mathbb{E}_{g}\big{[}\exp(-\frac{l}{\lambda}g_{k})\big{]}=\Gamma(1+\frac{l}{\lambda}).$
(s.11)

##### Resulting ReCAB.

From the results in Eq. (s.7, s.9, and s.11), the resulting bound of the KLD
is

$\displaystyle\begin{split}D_{KL}\big{(}q(z|x)\|p(z)\big{)}&\leq-(n-1)\log\frac{l}{\lambda}+n\big{(}\gamma\frac{l}{\lambda}-\sum_{k=1}^{n-1}\frac{1}{k}+\log\Gamma(1+\frac{l}{\lambda})\big{)}\\\
&\qquad{}-\sum_{{k}=1}^{n}\big{(}A_{k}-\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}\right)\big{)}.\\\
&=D_{ReCAB}\big{(}q(z|x)\|p(z)\big{)}\end{split}$ (s.12)

###  A.2 Tightness of ReCAB

Introducing auxiliary mass $\sum_{{k}=1}^{n}p_{k}=1$, we have following lower
bound.

$\displaystyle\begin{split}\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}-\frac{l}{\lambda}g_{k}\right)&=\mathbb{E}_{g}\big{[}\log\sum_{{k}=1}^{n}p_{k}\frac{\exp(A_{k}-\frac{l}{\lambda}g_{k})}{p_{k}}\big{]}\\\
&\geq\sum_{{k}=1}^{n}p_{k}\mathbb{E}_{g}\big{[}\log\frac{\exp(A_{k}-\frac{l}{\lambda}g_{k})}{p_{k}}\\\
&=\sum_{{k}=1}^{n}p_{k}\big{(}A_{k}-\gamma\frac{l}{\lambda}-\log
p_{k}).\end{split}$ (s.13)

Maximizing this bound with respect to $p_{k}$ results in the selection of
$p_{k}=\text{softmax}(A_{k}-\gamma\frac{l}{\lambda})$. Applying this value,
the bound is

$\displaystyle\begin{split}\mathbb{E}_{g}\big{[}\operatorname{LSE}_{{k}=1}^{n}\left(A_{k}-\frac{l}{\lambda}g_{k}\right)&\geq\operatorname{LSE}_{{i}=1}^{n}\left(A_{i}-\gamma\frac{l}{\lambda}\right).\end{split}$
(s.14)

Note that this bound relates to the lower bound of the KLD. The difference
between the upper bound in Eq. (s.12) and this bound is

$\displaystyle\begin{split}n\cdot\big{(}\log\Gamma(1+\frac{l}{\lambda})-\gamma\frac{l}{\lambda}\big{)},\end{split}$
(s.15)

which is a constant with respect to logit parameter. This is the interval
between upper and lower bound of desired KLD, or the maximal gap between
proposed ReCAB and true KLD, equivalently.

##  B Additional Experimental Results

###  B.1 Sample spectrograms

Figure s1: Sample spectrogram and alignment plots of (left) female speaker and
(right) male speaker. Each row is for angry, happy, neutral and sad voices,
respectively.

###  B.2 Heatmaps of KLD estimations

Figure s2: Additional results related to Figure 1 in the main paper. In each
row, temperature parameters of target distributions are set to (top) 0.5 and
(bottom) 2, while proposal distribution temperatures are all set to 1. Note
that CA does not use the temperature parameter.

###  B.3 Density plots

Figure s3: Additional results related to Figure 2 in the main paper. In each
row, temperature parameters of target distributions are set to (top) 0.5,
(middle) 1, and (bottom) 2, while temperatures of all others equal to 1. Note
that same densities are shown in the second row.

##  C Architectures and Settings for Emotional Speech Synthesis

Table s1 summarizes architectural details, and Figure s4 illustrates the block
diagram of the proposed system. The continuous and discrete embeddings are
first obtained by the ReCAB-VAE based encoder, after which the embeddings are
added to the output embedding of the text encoder in the Tacotron2 framework.

Table s1: Architectural details of the emotional TTS system based on ReCAB-VAE. Model | Module | Hyperparameters
---|---|---
Data | Input | Text embedding (512)
| Output | Mel-spectrogram
|  | (80, 64 ms frame size with 16 ms hop)
Tacotron 2 | Encoder | 3 Conv. layers (512)
|  |  $\rightarrow$ Bi-LSTM (512)
| Attention type | Location sensitive attention
| Decoder | 2-layer Pre-Net (FC, 256-dim)
|  |  $\rightarrow$ AttentionRNN (1024, LSTM)
|  |  $\rightarrow$ DecoderRNN (512, LSTM)
|  |  $\rightarrow$ FC layer (80)
|  |  $\rightarrow$ Post-Net (80-512*3-80, 5 Conv.)
ReCAB-VAE | Cont.Encoder | 6 Conv. layers (16-16-32-32-64-64 channels)
|  |  $\rightarrow$ GRU (256)
|  |  $\rightarrow$ FC (256)
| Disc.Encoder | 6 Conv. layers (16-16-32-32-64-64 channels)
|  |  $\rightarrow$ GRU (256)
|  |  $\rightarrow$ FC layer (256)
Figure s4: Structure of (a) ReCAB-VAE and (b) ReCAB-based emotional TTS
system.