# Fluctuation theorems and thermodynamic uncertainty relations

Gianluca Francica<EMAIL_ADDRESS>
###### Abstract

Fluctuation theorems are fundamental results in non-equilibrium
thermodynamics. Considering the fluctuation theorem with respect to the
entropy production and an observable, we derive a new thermodynamic
uncertainty relation which also applies to non-cyclic and time-reversal non-
symmetric protocols. Furthermore, we investigate the relation between the
thermodynamic uncertainty relation and the correlation between the entropy and
the observable.

##  I Introduction

In the last decades the non-equilibrium thermodynamics has received a great
attention kosloff13 ; Vinjanampathy16 ; bookthermo18 . Among the several
results achieved in this field, the fluctuation theorems (see, e.g., Ref.
campisi11  for a review) undoubtedly are of fundamental importance. For
instance, these theorems lead to the Jarzynski equality Jarzynski97  and the
second law of thermodynamics. Recently, it is shown how, for a cyclic and
time-reversal symmetric protocol, these theorems imply the so-called
thermodynamic uncertainty relations Hasegawa19 ; Timpanaro19 , i.e. they give
a lower bound of fluctuations of observables, which involves the entropy
produced in the process. Conversely, for the case of general fluctuation
theorems, hysteretic thermodynamic uncertainty relations Proesmans19 ; potts19
and lower bounds for the entropy produced in the forward process vo20 ;
campisi21  are derived.

In this paper, we consider a more general fluctuation relation and derive a
new thermodynamic uncertainty relation which is tighter than the one of Ref.
Proesmans19 ; potts19  and reduces to the one of Ref. Timpanaro19  for a
cyclic and time-reversal symmetric protocol. Furthermore, we investigate how
it depends on the correlation between the entropy and the observable.

##  II Thermodynamic uncertainty relation

We consider a generic system that starts in a thermal equilibrium state, such
that for the entropy production $\sigma$ and the observable $\phi$ we have the
multivariate fluctuation theorem

$\frac{p(\sigma,\phi)}{\tilde{p}(-\sigma,-\phi)}=e^{\sigma}\,,$ (1)

where $p(\sigma,\phi)$ and $\tilde{p}(\sigma,\phi)$ are the probability
distribution functions for the joint occurrence of $\sigma$ and $\phi$ in the
forward and backward process, respectively. In Ref. Timpanaro19  a cyclic and
time-reversal symmetric protocol is considered, for which
$\tilde{p}(\sigma,\phi)=p(\sigma,\phi)$, and it is shown that in this case the
Eq. (1) implies

$\frac{\langle\phi\rangle^{2}}{\langle\phi^{2}\rangle}\leq\left\langle\tanh^{2}\left(\frac{\sigma}{2}\right)\right\rangle\,,$
(2)

from which it follows the thermodynamic uncertainty relation

$\frac{\langle\phi^{2}\rangle}{\langle\phi\rangle^{2}}\geq\frac{1}{f^{2}(\langle\sigma\rangle)}\,,$
(3)

where the function $f$ is the inverse of $h(x)=2x\tanh^{-1}x$ and the averages
$\langle\cdots\rangle$ are calculated with respect to $p(\sigma,\phi)$, for
instance $\langle\phi\rangle=\int\phi p(\sigma,\phi)d\sigma d\phi$.

Here, we consider the general case (i.e. we can have
$\tilde{p}(\sigma,\phi)\neq p(\sigma,\phi)$), for which we have that the
fluctuation theorem of Eq. (1) implies

$\frac{(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}}{\langle\phi^{2}\rangle+\langle\phi^{2}\rangle_{R}}\leq\left\langle\tanh^{2}\left(\frac{\sigma}{2}\right)\right\rangle+\left\langle\tanh^{2}\left(\frac{\sigma}{2}\right)\right\rangle_{R}\,,$
(4)

where the averages $\langle\cdots\rangle_{R}$ are calculated with respect to
$\tilde{p}(\sigma,\phi)$. We recall that an inequality equivalent to Eq. (4)
has already been derived in Ref. potts19 , where it is proved that from it
follows the hysteretic thermodynamic uncertainty relation of Ref. Proesmans19
; potts19

$\frac{2(\langle\phi^{2}\rangle+\langle\phi^{2}\rangle_{R})}{(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}}\geq\coth\left(\frac{\langle\sigma\rangle+\langle\sigma\rangle_{R}}{4}\right)\,.$
(5)

Here, we prove that from Eq. (4) follows the thermodynamic uncertainty
relation

$\frac{2(\langle\phi^{2}\rangle+\langle\phi^{2}\rangle_{R})}{(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}}\geq\frac{1}{f^{2}\left(\frac{\langle\sigma\rangle+\langle\sigma\rangle_{R}}{2}\right)}\,.$
(6)

Of course Eq. (4) and $\eqref{bound1}$ reduce to Eq. (2) and (3) when
$\tilde{p}(\sigma,\phi)=p(\sigma,\phi)$. Thus, the bound is saturated by the
minimal distribution of Ref. Timpanaro19

$\displaystyle p_{min}(\sigma,\phi)$ $\displaystyle=$
$\displaystyle\frac{1}{2\cosh(a/2)}\big{(}e^{a/2}\delta(\sigma-a)\delta(\phi-b)$
(7) $\displaystyle+e^{-a/2}\delta(\sigma+a)\delta(\phi+b)\big{)}\,.$

In particular, we have the relation (for $x\geq 0$)

$\frac{2}{x}+1\geq\frac{1}{f^{2}(x)}\geq\coth\left(\frac{x}{2}\right)\,,$ (8)

thus, the bound is smaller than
$4/(\langle\sigma\rangle+\langle\sigma\rangle_{R})+1$ and tighter than the one
of Eq. (5), derived in Ref. Proesmans19 ; potts19 .

We note that, by defining the arithmetic mean
$\langle\cdots\rangle_{M}=(\langle\cdots\rangle+\langle\cdots\rangle_{R})/2$,
i.e. the average with respect to
$p_{M}(\sigma,\phi)=(p(\sigma,\phi)+\tilde{p}(\sigma,\phi))/2$, the Eq. (6)
reads

$\frac{\langle\phi^{2}\rangle_{M}}{\langle\phi\rangle^{2}_{M}}\geq\frac{1}{f^{2}\left(\langle\sigma\rangle_{M}\right)}\,,$
(9)

which is our main result. It is worth noting that the Eq. (9) is in agreement
with Eq. (3), since the distribution function $p_{M}(\sigma,\phi)$ is such
that $p_{M}(\sigma,\phi)/p_{M}(-\sigma,-\phi)=e^{\sigma}$.

We observe that how far the fluctuation is from the bound depends on the
degree of correlation between the variables $\sigma$ and $\phi$, which can be
characterized by the Pearson correlation coefficient defined as

$\rho_{\sigma,\phi}=\frac{\langle\sigma\phi\rangle-\langle\sigma\rangle\langle\phi\rangle}{\sqrt{(\langle\sigma^{2}\rangle-\langle\sigma\rangle^{2})(\langle\phi^{2}\rangle-\langle\phi\rangle^{2})}}\,.$
(10)

If there is no correlation we have the probability distribution
$p(\sigma,\phi)=p(\sigma)p(\phi)$. Then, from the fluctuation theorem we have
$\tilde{p}(\sigma,\phi)=e^{-\sigma}p(-\sigma)p(-\phi)$, such that
$\langle\phi\rangle_{R}=-\langle\phi\rangle$ and
$\langle\phi^{2}\rangle_{M}/\langle\phi\rangle_{M}^{2}$ tends to infinity.
Thus, we expect that the fluctuation is very far from the bound as the two
variables are weakly correlated. On the other hand, the bound is saturated for
the minimal distribution. In this case, it is easy to see that
$\rho_{\sigma,\phi}=1$ and there is a perfect correlation. For completeness,
we give the proofs of Eq. (4),(6) and (8).

Proof of Eq. (4) – We start by noting that for an arbitrary function
$F(\sigma,\phi)$ we have

$\displaystyle\langle F(\sigma,\phi)\rangle$ $\displaystyle=$
$\displaystyle\int F(\sigma,\phi)p(\sigma,\phi)d\sigma d\phi$ (11)
$\displaystyle=$ $\displaystyle\int
F(\sigma,\phi)e^{\sigma}\tilde{p}(-\sigma,-\phi)d\sigma d\phi\,,$ (12)

where we have used the fluctuation theorem. Thus, by performing the
substitutions $\sigma\to-\sigma$ and $\phi\to-\phi$ we get

$\langle F(\sigma,\phi)\rangle=\langle
F(-\sigma,-\phi)e^{-\sigma}\rangle_{R}\,.$ (13)

In particular, from this relation we have that

$\langle x^{n}\rangle=(-1)^{n}\langle x^{n}e^{-\sigma}\rangle_{R}$ (14)

for $x=\sigma,\phi$, thus

$(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}=\langle\phi(1-e^{-\sigma})\rangle_{R}^{2}=\left\langle\phi\sqrt{1+e^{-\sigma}}\frac{1-e^{-\sigma}}{\sqrt{1+e^{-\sigma}}}\right\rangle_{R}^{2}$
(15)

and by using the Cauchy-Schwartz inequality we have

$(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}\leq\langle\phi^{2}(1+e^{-\sigma})\rangle_{R}\left\langle\frac{(1-e^{-\sigma})^{2}}{1+e^{-\sigma}}\right\rangle_{R}\,.$
(16)

Since from Eq. (14) it results that

$\langle\phi^{2}\rangle+\langle\phi^{2}\rangle_{R}=\langle\phi^{2}(1+e^{-\sigma})\rangle_{R}\,,$
(17)

we obtain

$\frac{(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}}{\langle\phi^{2}\rangle+\langle\phi^{2}\rangle_{R}}\leq\left\langle\tanh^{2}\left(\frac{\sigma}{2}\right)e^{-\sigma}\right\rangle_{R}+\left\langle\tanh^{2}\left(\frac{\sigma}{2}\right)\right\rangle_{R}\,.$
(18)

By noting that the function $\tanh^{2}(\sigma/2)$ is even and by applying the
relation in Eq. (13) we get

$\left\langle\tanh^{2}\left(\frac{\sigma}{2}\right)e^{-\sigma}\right\rangle_{R}=\left\langle\tanh^{2}\left(\frac{\sigma}{2}\right)\right\rangle\,,$
(19)

and thus from Eq. (18) it follows the Eq. (4).

Proof of Eq. (6)– We define the functions $r(x)=\tanh^{2}(x/2)$,
$k(x)=x\tanh(x/2)$, $g$ as a right inverse function of $k$ and $w$ as the
composition of the functions $r$ and $g$, $w(x)=(r\circ g)(x)=r(g(x))$, which
is concave, i.e. $w^{\prime\prime}<0$. Since $g$ is a right inverse function
of $k$ and $r$ is an even function, we have that $r(x)=r(g(k(x)))=w(k(x))$,
where we have used the definition of $w$. Then, by using the Jensen’s
inequality, we have that $\langle r(\sigma)\rangle=\langle
w(k(\sigma))\rangle\leq w(\langle k(\sigma)\rangle)$, such that from Eq. (4)
we have

$\frac{(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}}{2(\langle\phi^{2}\rangle+\langle\phi^{2}\rangle_{R})}\leq\frac{1}{2}w(\langle
k(\sigma)\rangle)+\frac{1}{2}w(\langle k(\sigma)\rangle_{R})\,.$ (20)

Since $w$ is concave, we have

$\frac{(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}}{2(\langle\phi^{2}\rangle+\langle\phi^{2}\rangle_{R})}\leq
w\left(\frac{\langle k(\sigma)\rangle+\langle
k(\sigma)\rangle_{R}}{2}\right)\,.$ (21)

By noting that the function $k$ is even and by applying the relation in Eq.
(13) we get

$\langle k(\sigma)\rangle+\langle
k(\sigma)\rangle_{R}=\langle\sigma\tanh\left(\frac{\sigma}{2}\right)(1+e^{-\sigma})\rangle_{R}=\langle\sigma(1-e^{-\sigma})\rangle_{R}$
(22)

and from Eq. (14) we get

$\langle k(\sigma)\rangle+\langle
k(\sigma)\rangle_{R}=\langle\sigma\rangle+\langle\sigma\rangle_{R}\,.$ (23)

Thus, from Eq. (21) we obtain

$\frac{(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}}{2(\langle\phi^{2}\rangle+\langle\phi^{2}\rangle_{R})}\leq
w\left(\frac{\langle\sigma\rangle+\langle\sigma\rangle_{R}}{2}\right)\,.$ (24)

Taking the square root of both sides, applying then the increasing function
$\tanh^{-1}$, multiplying them by $2$ and applying $k$, we get

$h\left(\sqrt{\frac{(\langle\phi\rangle+\langle\phi\rangle_{R})^{2}}{2(\langle\phi^{2}\rangle+\langle\phi^{2}\rangle_{R})}}\right)\leq\frac{\langle\sigma\rangle+\langle\sigma\rangle_{R}}{2}\,,$
(25)

from which it follows Eq. (6).

Proof of Eq. (8) – Let’s start considering the inequality
$1/f^{2}(x)\geq\coth(x/2)$ with $x\geq 0$, which is equivalent to
$\sqrt{\tanh(x/2)}\geq f(x)$. By defining $y=\sqrt{\tanh(x/2)}$, such that
$0\leq y\leq 1$, and applying the function $h$ to both sides, we get the
inequality $y\tanh^{-1}y\geq\tanh^{-1}y^{2}$. We have the series
$\tanh^{-1}z=\sum_{k=0}^{\infty}z^{2k+1}/(2k+1)$ for $\left|z\right|\leq 1$,
thus the inequality becomes
$\sum_{k=0}^{\infty}y^{2k}/(2k+1)\geq\sum_{k=0}^{\infty}y^{4k}/(2k+1)$ which
is true for $\left|y\right|\leq 1$ since $y^{2k}\geq y^{4k}$ in this interval.
Nextly, we consider the inequality $2/x+1\geq 1/f^{2}(x)$ with $x\geq 0$,
which is equivalent to $f(x)\geq\sqrt{x/(2+x)}$. By defining
$y=\sqrt{x/(2+x)}$, such that $0\leq y\leq 1$, and applying the function $h$
to both sides, we get the inequality $y/(1-y^{2})\geq\tanh^{-1}y$. We have the
series $1/(1-z)=\sum_{k=0}^{\infty}z^{k}$ for $\left|z\right|\leq 1$, thus the
inequality becomes
$\sum_{k=0}^{\infty}y^{2k}\geq\sum_{k=0}^{\infty}y^{2k}/(2k+1)$ which is true
for $\left|y\right|\leq 1$.

##  III Physical example

We are interested to study how the difference
$\Delta=\langle\phi^{2}\rangle_{M}/\langle\phi\rangle_{M}^{2}-1/f^{2}(\langle\sigma\rangle_{M})$
is related to the correlation between $\sigma$ and $\phi$ characterized by the
correlation coefficient $\rho_{\sigma,\phi}$. Thus, we proceed our
investigation by considering a physical example that is a closed quantum
system made of two parties $A$ and $B$. In the forward protocol, the time-
dependent Hamiltonian $H(t)=H_{A}(t)+H_{B}(t)+H_{int}(t)$ generates the
unitary time evolution operator $U_{t,0}$ defined by the Schrödinger equation
$i\dot{U}_{t,0}=H(t)U_{t,0}$ with initial condition $U_{0,0}=\mathds{1}$. In
particular we consider the case in which the interaction $H_{int}(t)$ between
the systems $A$ and $B$ is turned-off at the initial time $t=0$ and final time
$t=\tau$. The initial state is the equilibrium state $\rho^{G}_{i}=e^{-\beta
H(0)}/Z_{i}$ where $Z_{i}$ is the partition function
$Z_{i}=\text{Tr}\left\\{e^{-\beta H(0)}\right\\}$. By considering $\phi$ as
the energy difference $\Delta E_{A}$ of the part $A$, we have the probability
distribution function

$\displaystyle p(\sigma,\Delta E_{A})$ $\displaystyle=$ $\displaystyle\sum
p^{(i)}_{mm^{\prime}}P_{mm^{\prime}nn^{\prime}}\delta(\sigma-\sigma_{mm^{\prime}nn^{\prime}})$
(26) $\displaystyle\times\delta(\Delta
E_{A}-\epsilon^{A}_{n}(\tau)+\epsilon^{A}_{m}(0))\,,$

where the initial populations are
$p^{(i)}_{mm^{\prime}}=\langle\epsilon^{A}_{m},\epsilon^{B}_{m^{\prime}}(0)|\rho^{G}_{i}|\epsilon^{A}_{m},\epsilon^{B}_{m^{\prime}}(0)\rangle$,
$|\epsilon^{X}_{m}(t)\rangle$ are eigenstates of $H_{X}(t)$ with eigenvalues
$\epsilon^{X}_{m}(t)$, with $X=A,B$, and
$\sigma_{mm^{\prime}nn^{\prime}}=\beta(\epsilon^{A}_{n}(\tau)+\epsilon^{B}_{n^{\prime}}(\tau)-\epsilon^{A}_{m}(0)-\epsilon^{B}_{m^{\prime}}(0)-\Delta
F)$, where the difference of free energy is $\Delta F=-\ln(Z_{f}/Z_{i})/\beta$
with $Z_{f}=\text{Tr}\left\\{e^{-\beta H(\tau)}\right\\}$. The transition
probability is given by
$P_{mm^{\prime}nn^{\prime}}=\left|\langle\epsilon^{A}_{n},\epsilon^{B}_{n^{\prime}}(\tau)|U_{\tau,0}|\epsilon^{A}_{m},\epsilon^{B}_{m^{\prime}}(0)\rangle\right|^{2}$.
On the other hand, in the backward protocol, the initial state is
$\rho^{G}_{f}=e^{-\beta H(\tau)}/Z_{f}$ and the time evolution from the
initial to the final time is described by the operator $U_{\tau,0}^{\dagger}$.
We have the probability distribution function

$\displaystyle\tilde{p}(\sigma,\Delta E_{A})$ $\displaystyle=$
$\displaystyle\sum
p^{(f)}_{nn^{\prime}}P_{mm^{\prime}nn^{\prime}}\delta(\sigma+\sigma_{mm^{\prime}nn^{\prime}})$
(27) $\displaystyle\times\delta(\Delta
E_{A}-\epsilon^{A}_{m}(0)+\epsilon^{A}_{n}(\tau))\,,$

where
$p^{(f)}_{nn^{\prime}}=\langle\epsilon^{A}_{n},\epsilon^{B}_{n^{\prime}}(\tau)|\rho^{G}_{f}|\epsilon^{A}_{n},\epsilon^{B}_{n^{\prime}}(\tau)\rangle$,
such that the Tasaki-Crooks fluctuation theorem of Eq. (1) is satisfied. In
order to proceed with our study, we consider the particular case of two qubits
with an Ising Hamiltonian, such that $H_{X}(t)=\omega(t)\sigma^{X}_{z}$, with
$X=A,B$, and the interaction is
$H_{int}(t)=\lambda(t)\sigma^{A}_{x}\otimes\sigma^{B}_{x}$, where
$\sigma_{x}$, $\sigma_{y}$ and $\sigma_{z}$ are the Pauli matrices. As shown
in the Fig. 1, the difference $\Delta$ is always positive and the bound of Eq.
(9) is satisfied. Furthermore, we have that the fluctuation deviates from the
value of the bound when the correlation coefficient $\rho_{\sigma,\Delta
E_{A}}$ decreases and so the correlation weakens.

Figure 1:  The plots of the normalized difference $\Delta_{n}$ (blue line) and
the correlation coefficient $\rho_{\sigma,\Delta E_{A}}$ (red dashed line) in
function of the duration time $\tau$. In detail, the normalized difference
$\Delta_{n}$ is obtained by dividing the difference $\Delta$ by its maximum
value in the interval under consideration. We consider
$\omega(t)=\omega_{i}+(\omega_{f}-\omega_{i})t/\tau$ and
$\lambda(t)=\lambda_{0}\sin(\pi t/\tau)$ with $\omega_{i}=1$, $\omega_{f}=2$,
$\lambda_{0}=4$ and $\beta=1$.

Of course, how strong the relation between the difference $\Delta$ and the
correlation is depends on the particular system under consideration. In order
to give another illustrative example, we note that given two non-negative
functions $g_{1}(\sigma,\phi)$ and $g_{2}(\sigma,\phi)$, we can construct the
probability

$p(\sigma,\phi)=n(\hat{g}_{1}(\sigma,\phi)\theta(\sigma)+\hat{g}_{2}(-\sigma,-\phi)e^{\sigma}\theta(-\sigma))\,,$
(28)

such that satisfies the fluctuation relation. In detail, $n$ is a constant
defined such that the probability $p(\sigma,\phi)$ is normalized, i.e. $\int
p(\sigma,\phi)d\sigma d\phi=1$, $\theta$ is the Heaviside step function, and
the functions $\hat{g}_{i}$ are defined by

$\hat{g}_{i}(\sigma,\phi)=\frac{g_{i}(\sigma,\phi)}{\int_{-\infty}^{\infty}d\phi\int_{0}^{\infty}d\sigma
g_{i}(\sigma,\phi)(1-e^{-\sigma})}\,,$ (29)

with $i=1,2$, such that also
$\tilde{p}(\sigma,\phi)=p(-\sigma,-\phi)e^{\sigma}$ is normalized. For
simplicity, we consider
$g_{1}(\sigma,\phi)=g_{2}(\sigma,\phi)=e^{-\sigma^{2}-\phi^{2}+\gamma\sigma\phi}$,
where the parameter $\gamma$ determines the amount of correlation. In this
simple case (in general for Gaussian functions) the difference $\Delta$ tends
to decrease as the correlation tends to increase showing a very strong
relation between the two (see Fig. 2).

Figure 2:  The plots of the normalized difference $\Delta_{n}$ (blue line) and
the correlation coefficient $\rho_{\sigma,\phi}$ (red dashed line) in function
of the parameter $\gamma$.

##  IV Conclusions

In summary, we have investigated the relation between fluctuation theorems and
thermodynamic uncertainty relations in the most general case of non-cyclic and
time-reversal non-symmetric protocols, deriving a new lower bound of the
observable fluctuation. The bound is tighter than the one of Ref. Proesmans19
; potts19 , in detail the difference $d(x)=1/f^{2}(x)-\coth(x/2)$ gets its
maximum value $1/3$ at $x=0$, is decreasing for $x\geq 0$ and $d(x)\to 0$ as
$x\to\infty$, such that the two bounds get closer as the average
$\langle\sigma\rangle_{M}$ increases. Furthermore, we have performed a study
in order to clarify the relation between the thermodynamic uncertainty
relation and the correlation between the two variables $\sigma$ and $\phi$. In
particular, we find that the fluctuation tends to be far from the bound as the
two variables are weakly correlated.

## References

  * (1) R. Kosloff, Entropy 15, 2100 (2013)
  * (2) S. Vinjanampathy and J. Anders, Contemp. Phys. 57, 545 (2016)
  * (3) Thermodynamics in the Quantum Regime - Fundamental Aspects and New Directions, (eds.) F. Binder, L. A. Correa, C.Gogolin, J. Anders, G. Adesso, Springer (2018)
  * (4) M. Campisi, P. Hänggi, and P. Talkner, Rev. Mod. Phys. 83, 771 (2011)
  * (5) C. Jarzynski, Phys. Rev. Lett. 78, 2690 (1997)
  * (6) Y. Hasegawa and T. Van Vu, Phys. Rev. Lett. 123, 110602 (2019)
  * (7) A. M. Timpanaro, G. Guarnieri, J. Goold and G. T. Landi, Phys. Rev. Lett. 123, 090604 (2019)
  * (8) K. Proesmans and J. M. Horowitz, J. Stat. Mech. (2019) 054005
  * (9) P. P. Potts and P. Samuelsson, Phys. Rev. E 100, 052137 (2019)
  * (10) V. T. Vo, T. Van Vu and Y. Hasegawa, Phys. Rev. E 102, 062132 (2020)
  * (11) M. Campisi and L. Buffoni, Phys. Rev. E 104, L022102 (2021)


# Continuous descriptor-based control for deep audio synthesis

###### Abstract

Despite significant advances in deep models for music generation, the use of
these techniques remains restricted to expert users. Before being democratized
among musicians, generative models must first provide expressive control over
the generation, as this conditions the integration of deep generative models
in creative workflows. In this paper, we tackle this issue by introducing a
deep generative audio model providing expressive and continuous descriptor-
based control, while remaining lightweight enough to be embedded in a hardware
synthesizer. We enforce the controllability of real-time generation by
explicitly removing salient musical features in the latent space using an
adversarial confusion criterion. User-specified features are then reintroduced
as additional conditioning information, allowing for continuous control of the
generation, akin to a synthesizer knob. We assess the performance of our
method on a wide variety of sounds including instrumental, percussive and
speech recordings while providing both timbre and attributes transfer,
allowing new ways of generating sounds.

**footnotetext: These authors contributed equally to this work

Index Terms—  Deep audio synthesis, continuous control, timbre transfer,
representation learning, adversarial training

##  1 Introduction

In recent years, deep generative models have offered exciting new ways to
synthesize sound and accomplished impressive results in generating high-
quality audio samples. Early works on auto-regressive (AR) models successfully
achieved high-quality raw waveform synthesis [1], but at the cost of expensive
computation [2]. Subsequent approaches have leveraged a time-frequency
representation of the signals and reduce significantly the inference cost [3],
but remain hard to control. These methods mainly rely on Generative
Adversarial Networks (GAN) which produces highly realistic samples [4] or
Variational AutoEncoders (VAE) [5] which provides a latent representation that
captures high-level signal features.

The RAVE model [6] leverages both VAE representation abilities and GAN
training to achieve high-quality waveform synthesis in real-time on a standard
laptop CPU. However, controlling the generation process over non-
differentiable attributes remains a challenging task. Prior works tried to
enhance generative models with control attributes [7, 8], notably the
Conditional VAE (C-VAE) [9], which adds an attribute vector as input to the
encoder and decoder to condition the generation [9]. However, it can lead to
poor control abilities as the model could ignore the conditioning and sample
directly from the latent variables. To address this, Fader Networks [10] force
the latent representation to be invariant to given target attributes through
adversarial learning. This work has been extended for VAEs to control real-
valued attributes instead of binary tags for symbolic generation [11].

Yet, providing high-level, continuous and time-varying controls with
perceptually meaningful parameters over the raw audio waveform remains an open
challenge. In this paper, we propose to tackle this issue by applying the
Fader Networks approach on top of the state-of-art deep audio model RAVE [6]
in order to provide intuitive controls for sound synthesis. After explicitly
disentangling salient musical features in the RAVE latent space by relying on
the Fader adversarial confusion criterion, we reintroduce them as additional
inputs to the RAVE decoder to condition the generation, allowing for
continuous time-varying control akin to a synthesizer knob. We show that our
model provides more accurate control compared to our baselines on various
speech and instrumental datasets. We evaluate the impact of control on the
generation quality using various quality metrics. Unlike prior works, our
method orthogonalizes descriptor-based control attributes from the latent
representation. Hence, it provides independent priors on both the latent
representation and the control attributes, allowing to separately (or jointly)
perform timbre transfer and attribute transfer. Hence, the user can choose any
set of audio descriptors to condition the generation process. Our approach
remains lightweight and can be embedded into highly-constrained hardware
synthesizers111All of our source code and experiments are available on a
supplementary webpage: https://github.com/neurorave/neurorave.

Fig. 1: Overall workflow of our proposed method combining RAVE, Fader network
and joint prior training.

##  2 Proposed model

We aim to provide continuous attributes control on any type of audio
descriptor for raw waveform generation. To do so, we propose to regularize the
latent space of a generative model with an adversarial criterion and condition
the generation with this given set of continuous controls. This overall
architecture is depicted in Figure 1.

### Generative models.

We base our work on RAVE [6] as it allows fast and high quality audio waveform
synthesis (with sampling rates up to 48kHz) on any type of signal while being
usable with real-time constraints on a laptop CPU. To do so, it leverages a
multi-band decomposition of the raw waveform with a Pseudo Quadrature Mirror
Filter bank (PQMF) [12, 8], which allows to decrease the temporal
dimensionality of the data. Similarly to the RAVE [6] model (depicted in blue
on Figure 1), the training is split between a representation learning stage
and an adversarial fine-tuning stage.

The first stage aims to learn relevant features for the latent space by
optimizing a multiscale spectral distance [13]. The encoder and decoder are
trained to minimize the loss $\mathcal{L}_{vae}$ derived from the ELBO [5].
Hence, for a given signal $\mathbf{x}\in\mathbb{R}^{n}$ (where $n$ is the
initial discrete time length), we train a VAE to reconstruct this input signal
by learning an informative and compact latent representation. This
representation keeps the temporal dimension as it produces a matrix
$\mathbf{z}\in\mathbb{R}^{d\times m}$, where $d<n$ is the number of latent
dimensions and $m\in\mathbb{N}$ is the compressed discrete time length. This
dimension depends on the sampling rate and compression ratio applied by the
encoder. Therefore, the embedded representation can be seen as a temporal
trajectory in a $d$-dimensional space. Once the latent representation is
learned, the encoder is frozen to perform the adversarial fine-tuning stage.

During the second stage, the latent space is seen as a base GAN distribution
and the decoder learns to generate more realistic signals by relying on a
discriminator $D$ optimizing the Hinge loss $\mathcal{L}_{g}$ [14]. To ensure
that the synthesized signal $\mathbf{\hat{x}}$ does not diverge too much from
the ground truth $\mathbf{x}$, the model keeps minimizing the multiscale
spectral distance, but also adds the feature matching loss $\mathcal{L}_{FM}$
proposed in [3], which minimizes the $L_{1}$ distance between the
discriminator feature maps of real and synthesized audio. Hence, the full
objective loss for the decoder becomes

$\begin{split}\mathcal{L}_{dec}=\mathcal{L}_{vae}+\mathcal{L}_{g}+\mathcal{L}_{FM}.\end{split}$
(1)

### Control.

Within the image field, one of the first approach of control [9] adds the
features information as conditioning input to a VAE. GAN models also
incorporated attributes control, such as in StyleGAN [15] which splits high-
level features from stochastic variation, or AttGans [16] which includes an
attribute classification constraint to the generated image. Within the audio
generation domain, DDSP [13] relies on explicit control signals, but only
offers pitch and loudness modifications over the generation.

Fader Networks [10] is a particularly interesting approach as it provides
explicit control on any set of desired attributes. It is achieved by applying
an adversarial discriminator in the latent space, forcing the learning of
invariant representations with respect to the varying attributes. This
discriminator is trained to predict the real attributes from the latent
representation, while the encoder is forced to bypass all features information
in order to prevent the discriminator from achieving its goal. This implies
the adversarial losses

$\begin{split}\mathcal{L}_{dis}(\theta_{dis}|\theta_{enc})=-\mathbb{E}_{p_{\textit{enc}}(\mathbf{z}|\mathbf{x})}[\text{log}(p_{\textit{dis}}(\mathbf{y}|\mathbf{z}))]\\\
\mathcal{L}_{dis}(\theta_{enc}|\theta_{dis})=-\mathbb{E}_{p_{\textit{enc}}(\mathbf{z}|\mathbf{x})}[\text{log}(p_{\textit{dis}}(1-\mathbf{y}|\mathbf{z}))],\end{split}$
(2)

where $\theta_{enc}$ denotes the encoder parameters, $\theta_{dis}$ those of
the discriminator and $p_{\textit{dis}}(\mathbf{y}|\mathbf{z})$ represents the
(discriminator) probability of an attribute vector $\mathbf{y}$ given a latent
representation $\mathbf{z}$. The drawback of this method is that it only
considers binary and static attributes, which are insufficient in the context
of audio generation control. However, [11] relies on an extension of Faders to
provide control over symbolic musical attributes. Although their method allows
for real-valued parameters instead of binary attributes, they still do not
address the problem of continuous temporal control. Hence, we propose to
extend this method to provide both continuous high-level attributes control
and raw audio generation.

### Our approach.

The representation learning stage of training perfectly fits with the first
objective of Fader networks, which is to re-organize an encoded
representation. Once the model converges, the encoder and fader discriminator
are frozen and the second adversarial fine-tuning stage begins in order to
improve the quality of the generation. In order to introduce control (depicted
in orange in Figure 1), we add an adversarial discriminator to the latent
space during the first stage similarly to Fader Networks [10]. The encoder is
enforced to learn a representation invariant to the chosen control attributes
$\mathbf{y}_{n}\in\mathbb{R}$, $\forall n\in[1,N]$, where $N$ is the number of
considered attributes. However, contrarily to previous approaches, we perform
on-the-fly attribute computation from input $\mathbf{x}$, such that
$\mathbf{a}_{n}=f_{a}(\mathbf{x})$, with $f_{a}$ being the descriptor
computation function. Then, the decoder receives the audio descriptor
$\mathbf{a}_{n}$, which is resampled to match the temporal dimension, as
additional information to the latent matrix $\mathbf{z}$.

In order to constrain the decoder to use the attribute information to
reconstruct the signal, the latent discriminator gives feedback to the encoder
by optimizing the adversarial criterion, which forces the encoder to remove
any attribute information from the latent space. The attributes
$\mathbf{y}_{n}$ used for this training are obtained from the conversion of
continuous attributes $\mathbf{a}_{n}$ into discrete categorical quantities by
performing a quantization. It is achieved by applying an RMS threshold to
remove silent parts of the sound and then sorting these values to obtain bins
with equal density. The final training loss is defined by the combination of
Eq. 1 and Eq. 2

$\begin{split}\mathcal{L}&=\mathcal{L}_{vae}+\lambda\cdot\mathcal{L}_{dis}(\theta_{enc}|\theta_{dis}),\end{split}$
(3)

where the $\lambda$ hyperparameter controls the trade-off between the ELBO and
the invariance of the latent representation.

##  3 Experiments

### Audio descriptors.

Descriptors provide quantitative measures on the properties of a given sound
[17]. To perform a direct descriptor-based synthesis, we selected a subset of
local descriptors (providing a value for each frame of signal), which are also
complex to differentiate, while being linked to perceptual properties. We only
retain descriptors that are pairwise independent to successfully disentangle
the selected attributes from the latent representation. In addition to
spectral descriptors (RMS perceptually similar to loudness, Centroid to
brightness, and Bandwidth to richness), we rely on more advanced timbral
features provided by Audio Commons222
http://github.com/AudioCommons/timbral_models: Sharpness and Boominess, which
we adapt to be computed frame-wise rather than globally.

### Datasets.

We experiment on a variety of dataset divided between train (80%), validation
(10%) and test (10%) splits.

### Instrumental (NSynth [18])

is a collection of instruments with 4-seconds monophonic sounds sampled at
16kHz. We use only string family, which corresponds to 20594 samples.

### Percussive (Darbuka)

is an internal dataset composed of about 2 hours of raw recordings of darbuka
sampled at 48kHz.

### Speech (SC09)

is part of the Speech Commands [19] set containing the spoken digits from
multiple speakers in various recording conditions sampled at 16kHz (23666
samples).

| Reconstruction | Control | Cycle | Reconstruction | Control | Cycle
---|---|---|---|---|---|---
| _JND_ | _Mel_ | _mSTFT_ | _Corr._ | _L1_ | JND | _JND_ | _Mel_ | _mSTFT_ | _Corr._ | _L1_ | JND
RAVE | 0.264 | 15.097 | 7.754 | - | - | - | 0.264 | 15.097 | 7.754 | - | - | -
C-RAVE | 0.225 | 12.567 | 5.842 | 0.890 | 0.135 | 0.603 | 0.208 | 12.236 | 4.584 | 0.425 | 0.361 | 0.752
F-RAVE | 0.238 | 13.876 | 5.394 | 0.917 | 0.112 | 0.581 | 0.187 | 13.050 | 4.681 | 0.445 | 0.357 | 0.776
Table 1: Comparison of various models on the mono-attribute rms training
(left), and the complete multi-attribute training (right). We compare these on
their quality of reconstruction, ability to control attributes and cycle
consistency.

### Implementation details

All experiments are conducted using the ADAM optimizer [20] with
$\beta_{1}=0.5$, $\beta_{2}=0.9$, a batch size of 8 and a learning rate of
$0.0001$ as suggested in [6]. The weights are initialized from a Normal
distribution $\mathcal{N}(0,0.02)$ and activations are LeakyReLU with a leak
of $0.2$. We implemented the same architecture as the original RAVE paper for
the encoder, decoder and discriminator with an $8$-band PQMF decomposition.
For the latent discriminator architecture, we implemented a series of three
1-d convolutional blocks per attributes along the latent dimensions $d$, in
order to preserve the temporal dimension $m$ of the latent space. Each block
is composed of convolution, BatchNorm and LeakyReLU, with an additional
softmax activation on the last convolution layer. We train the representation
learning stage with Fader discriminator for $1000k$ steps. The $\beta$ warmup
from $0$ to $0.1$ and fader $\lambda$ warmup from $0$ to $0.5$, start
respectively at step $5k$ and $15k$, and end at step $10k$ and $30k$.

### Baselines.

To evaluate our method, we implement the RAVE model as a baseline for
reconstruction accuracy. Then we add to it a conditioning mechanism
(concatenating to the latent vector the repeated temporal attributes as extra
channels) leading to a conditional-RAVE (C-RAVE) model. Finally, we compare
them to Fader-Rave (F-RAVE) described in Sec 2.

### Evaluation.

For the evaluation of the reconstruction, we use the multiscale-STFT (mSTFT).
As this is also our training criterion, we rely on two independent spectral
metrics: the $L_{1}$ distance between Mel-spectrograms (Mel) and the Just
Noticeable Difference (JND) score [21]. Regarding the efficiency of attributes
control, we compute the correlation between changed control attributes and
resulting output attributes through the Spearman rank-order correlation
coefficient. We also compute the $L_{1}$ distance to evaluate quantitatively
how the generation reflects the control. Additionally, we perform a cycle
consistency distance to ensure the model is able to reproduce the original
audio when reverting the attributes of a transformed signal to their original
values.

##  4 Results

### Models quality and control.

We compare the quality of reconstruction for inputs from the test set for
RAVE, C-RAVE and F-RAVE. Then, we evaluate the control behaviour by changing
the attributes of the input sound with those of out-of-distribution examples
(e.g. we switch the attributes of the violin for those of the darbuka). We
compute this for mono-attribute training (swapping only the RMS) or for multi-
attribute training (swapping all attributes) cases. We detail our results
obtained on Nsynth [18] in Table 1.

Interestingly, introducing conditioning (C-RAVE) seems to improve the overall
reconstruction quality, in both the mono- (left) and multi- (right) cases.
This could be explained by the fact that this extraneous information
simplifies the work of the decoder, by providing structured information on the
generation. Although the C-RAVE model is able to adequately control the single
RMS attribute case, it loses in control quality for the more complicated
multi-attribute case (although it still improves reconstruction). Oppositely,
F-RAVE provides stronger correlation in mono-attribute RMS control, while
maintaining equivalent audio quality. It seems also more resilient to the
multi-attribute changes, providing the strongest correlation in control and
the lowest reconstruction error. This implies that the model could be trained
once for a whole set of descriptors, while still maintaining control quality.
This is also reflected by the cycle consistency, which appears more coherent
than the C-RAVE model.

### Multiple control attributes.

We also analyze the control quality by training a separate model for each of
the 4 descriptors, and a model for all descriptors at once (termed C-RAVE (m.)
and F-RAVE (m.)). We analyze the correlation between target and output
attributes when changing a single descriptor, or when changing random sets of
2, 3, or 4 attributes at once. The results obtained with Nsynth [18] are
displayed in Table 2.

Mono-attribute | _RMS_ | _Centro_ | _Sharp._ | _Boom._
---|---|---|---|---
C-RAVE | 0.890 | 0.310 | 0.796 | 0.760
F-RAVE | 0.917 | 0.282 | 0.760 | 0.773
Multi-attribute | 1 | 2 | 3 | 4
C-RAVE (m.) | 0.602 | 0.524 | 0.463 | 0.425
F-RAVE (m.) | 0.616 | 0.539 | 0.478 | 0.445
Table 2: Comparison of the control provided by various models on different
descriptors for mono-attribute (top) or multi-attribute (bottom) setups.

As we can see (up), models that are trained on single attributes provide a
stronger control over that descriptor. In these, it seems that RMS, the most
salient descriptor is the easiest to control, while more complex concepts
(Sharpness, Boominess, Centroid) are harder to grasp for the control models.
As can be expected, relying on a multi-attribute model rather than
specifically-trained models impact the quality of control, especially for
simple conditioning (C-RAVE). However, F-RAVE model seems to be less affected
by this property. Similarly, when increasingly changing a larger number of
attributes (bottom), the C-RAVE (m.) steadily degrades in terms of control
quality, while the F-RAVE (m.) model maintains a higher quality of control,
even when switching all attributes of a sound.

### Comparing various datasets.

We finally evaluate how our proposed F-RAVE can be used on a wide diversity of
sounds in the multi-attribute setup. We display the reconstruction (Rec.) and
control (Ctr.) results in Table 3.

| NSynth | Darbouka | SC09
---|---|---|---
| _Rec._ | _Ctr._ | _Rec._ | _Ctr._ | _Rec._ | _Ctr._
C-RAVE | 4.58 | 0.42 | 5.31 | 0.62 | 6.41 | 0.35
F-RAVE | 4.68 | 0.45 | 5.14 | 0.61 | 6.33 | 0.34
Table 3: Results on the instrumental (NSynth), percussive (Darbouka) and
speech (SC09) datasets.

RAVE appears versatile enough to maintain a high quality on any type of
sounds, and confirm that the reconstruction quality stands even under multiple
attribute conditioning. Regarding other models, the trends from the previous
results seem to be maintained across different datasets, with a slight
advantage for the F-RAVE model. The percussive dataset appears to be the
easiest to control, which could be explained by the larger variance in the
descriptor values for this set.

##  5 Conclusion

In this paper, we combined Fader Networks and the recent RAVE model in order
to achieve continuous descriptor-based control on real-time deep audio
synthesis. We showed that our method outperforms previous proposals in both
quantitative and qualitative analyses. By orthogonalizing the continuous time-
varying attributes from the latent representation, our approach provides
independent priors which allows to separately perform both timbre transfer and
attribute transfer enabling new creative prospects. Hence, the user can choose
a set of descriptors to condition the generation process and create a large
variety of sounds. Altogether, we hope this expressive control approach will
benefit both experts and non-experts musicians and provide new means of
exploring audio synthesis while promoting co-creative musical applications.

## References

  * [1] Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio,  “Samplernn: An unconditional end-to-end neural audio generation model,”  arXiv preprint arXiv:1612.07837, 2016.
  * [2] Philippe Esling, Ninon Devis, Adrien Bitton, Antoine Caillon, Constance Douwes, et al.,  “Diet deep generative audio models with structured lottery,”  arXiv preprint arXiv:2007.16170, 2020.
  * [3] Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville,  “Melgan: Generative adversarial networks for conditional waveform synthesis,”  Advances in neural information processing systems, vol. 32, 2019\.
  * [4] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial nets,”  Advances in neural information processing systems, vol. 27, 2014\.
  * [5] Diederik P Kingma and Max Welling,  “Auto-encoding variational bayes,”  arXiv preprint arXiv:1312.6114, 2013.
  * [6] Antoine Caillon and Philippe Esling,  “RAVE: A variational autoencoder for fast and high-quality neural audio synthesis,”  CoRR, vol. abs/2111.05011, 2021.
  * [7] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville,  “Film: Visual reasoning with a general conditioning layer,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2018, vol. 32.
  * [8] Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie,  “Multi-band melgan: Faster waveform generation for high-quality text-to-speech,”  in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 492–498.
  * [9] Kihyuk Sohn, Honglak Lee, and Xinchen Yan,  “Learning structured output representation using deep conditional generative models,”  Advances in neural information processing systems, vol. 28, 2015\.
  * [10] Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc’Aurelio Ranzato,  “Fader networks: Manipulating images by sliding attributes,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [11] Lisa Kawai, Philippe Esling, and Tatsuya Harada,  “Attributes-aware deep music transformation,”  in Proceedings of the 21st international society for music information retrieval conference, ismir, 2020.
  * [12] Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei, et al.,  “Durian: Duration informed attention network for multimodal synthesis,”  arXiv preprint arXiv:1909.01700, 2019.
  * [13] Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts,  “Ddsp: Differentiable digital signal processing,”  arXiv preprint arXiv:2001.04643, 2020.
  * [14] Jae Hyun Lim and Jong Chul Ye,  “Geometric gan,”  arXiv preprint arXiv:1705.02894, 2017.
  * [15] Samaneh Azadi, Matthew Fisher, Vladimir G Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell,  “Multi-content gan for few-shot font style transfer,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7564–7573.
  * [16] Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen,  “Attgan: Facial attribute editing by only changing what you want,”  IEEE Transactions on Image Processing, vol. 28, no. 11, pp. 5464–5478, 2019.
  * [17] Geoffroy Peeters, Bruno L Giordano, Patrick Susini, Nicolas Misdariis, and Stephen McAdams,  “The timbre toolbox: Extracting audio descriptors from musical signals,”  The Journal of the Acoustical Society of America, vol. 130, no. 5, pp. 2902–2916, 2011.
  * [18] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan,  “Neural audio synthesis of musical notes with wavenet autoencoders,”  in International Conference on Machine Learning. PMLR, 2017, pp. 1068–1077.
  * [19] Pete Warden,  “Speech commands: A dataset for limited-vocabulary speech recognition,”  arXiv preprint arXiv:1804.03209, 2018.
  * [20] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Pranay Manocha, Adam Finkelstein, Richard Zhang, Nicholas J Bryan, Gautham J Mysore, and Zeyu Jin,  “A differentiable perceptual audio metric learned from just noticeable differences,”  arXiv preprint arXiv:2001.04460, 2020.