# On the theoretical framework for meniscus-guided manufacturing of large-area
OPV modules

Fabian Gumpert  Annika Janßen  Robin Basu  Christoph J. Brabec  Hans-Joachim
Egelhaaf  Jan Lohbreier  Andreas Distler

###### Abstract

For the manufacturing of thin films of solution-processable organic
semiconductors, e.g. for organic photovoltaics (OPV), meniscus guided-coating
techniques are the method of choice for large-scale industrial applications.
However, the process requires an in-depth understanding of the respective
fluid dynamics to control the resulting film thickness. In this article, we
derive an analytical expression to describe the layer thickness of coatings
manufactured with a trapezoidal-shaped applicator as a function of various
fluid and process parameters. The analytical calculations are compared with
results from computational fluid dynamics (CFD) simulations and experimental
data for an industrially relevant OPV active material system. The analytical
calculations are compared with results from computational fluid dynamics (CFD)
simulations and experimental data for an industrially relevant OPV active
material system. The good agreement of all three approaches demonstrates the
potential of the analytical and simulative methods to minimize the number of
time- and resource-consuming experiments. Furthermore, our theoretical model
can be used to enhance the homogeneity of large-area coatings by means of an
acceleration profile of the applicator that can compensate the liquid loss
during the coating process. The respective analytical expression is validated
by simulated and experimentally obtained data for long-distance coatings.
Finally, this approach is used to fabricate a large-area OPV module with new
world record efficiency.

###### keywords:

Organic photovoltaics , Meniscus-guided coating , Numerical simulation ,
Analytical description , Thin films , Accelerated coating velocity , Uniform
film thickness , Organic semiconductor , Computational fluid dynamics

††journal: Progress in Organic Coatings

[THN]organization=Faculty of Applied Mathematics, Physics and Humanities,
Nuremberg Institute of Technology,addressline=Keßlerplatz 12, city=Nuremberg,
postcode=90489, state=Bavaria, country=Germany

[FAU]organization=Faculty of Engineering, Department of Material Science,
Materials for Electronics and Energy Technology (i-MEET), Friedrich-Alexander-
Universität Erlangen-Nürnberg,addressline=Martensstraße 7, city=Erlangen,
postcode=91058, state=Bavaria, country=Germany

[HIERN]organization=Helmholtz Institute Erlangen-Nürnberg for Renewable Energy
(HI-ERN), Forschungszentrum Jülich GmbH,addressline=Immerwahrstraße 2,
city=Erlangen, postcode=91058, state=Bavaria, country=Germany

##  1 Introduction

The importance of photovoltaics has grown in recent years as the demand for
renewable energy continues to increase. Solution-processable organic
photovoltaics (OPV) receive more and more scientific and economical attention
due to their unique characteristics, like light weight, high throughput, and
semi-transparency Hu et al. (2022). The respective printing inks comprise the
photoactive material dissolved in a solvent. After the deposition of the
coating ink, the solvent evaporates during the drying process, leaving a dry
film as the functional layer. Recent material developments result in
photoelectric conversion efficiencies (PCEs) above 19 $\%$ Sun et al. (2022);
Zhan et al. (2022); Zhu et al. (2022) for small-scale cells (few $mm^{2}$) at
laboratory level. Uniform and even layers with a predefined thickness are
required to achieve these record efficiencies.
Solution-processable techniques are preferred processes for industrial
applications, since they enable higher throughput (e.g. roll-to-roll
manufacturing) and thus, lower costs. For roll-to-roll manufacturing, meniscus
guided coating techniques are established methods to deposit the OPV materials
NG et al. (2022); Gu et al. (2017); Berny et al. (2015). Among these
techniques, blade coating is scientifically and economically the most relevant
technique, especially for the research on upscaling from small-scale cells to
large-area modules. The upscaling to module size is still a major challenge
that requires further investigation of the process Zhang et al. (2022).
Insufficient control and/or understanding of the deposition process results in
uneven and non-uniform layer thicknesses and thus, to a decrease of the power
conversion efficiency of the module.
In the blade coating process, an initial fluid volume is deposited between the
applicator and the substrate to be coated. During the coating process, the
applicator is moved horizontally and, by this, fluid is partially deposited
onto the substrate. Consequently, the fluid volume beneath the applicator
steadily decreases with coating distance. However, the fluid volume beneath
the applicator influences the radius of the down-stream meniscus, which is
essential for the deposited wet film thickness Gumpert et al. (2023). As a
consequence, the consumption of the fluid during the coating process leads to
an uneven film thickness, i.e. an initially thicker film that becomes
gradually thinner with coating distance, as shown in Figure 1.

Figure 1: Normalized thickness profile of a P3HT:O-IDTBR film, which is
manufactured by blade coating with constant velocity, over a coating distance
of 15 $cm$. The steady decrease of film thickness with coating distance can be
observed in the experimentally obtained data points and is also clearly
visible in the photograph of the respective sample.

Besides laboratory experiments, numerical simulation and analytical
description are established methods to better understand the formation of the
wet film and to optimize the deposition process. Moreover, simulation and
theory approaches further decrease the number of time- and resource-expensive
experiments. For example the influence of different blade geometries on the
blade coating process are numerically investigated in Iliopoulos and Scriven
(2005). CFD models are reported to investigate the blade coating process as a
function of different variables (e.g. substrate speed and fluid properties)
Singh and Ormiston (2021); Schmidt et al. (2009); Mitsoulis and Athanasopoulos
(2010). The theory of wet film formation is based on the work of Landau,
Levich and Derjaguin Landau and Levich (1988); Derjaguin (1993). Based on this
theory, Gutenev et al. derived analytical expressions in Gutenev et al. (2003)
to further describe the blade coating process. According to the theory of wet
film formation, the coating velocity can compensate the fluid loss beneath the
applicator. In Park and Han (2009); Tsai et al. (2015), the influence of the
coating velocity on the wet film thickness was experimentally investigated. In
both articles, the theoretically predicted dependency of the film thickness on
the coating velocity is confirmed. In Gumpert et al. (2023), we developed a
numerical model of the coating process where the simulated thicknesses are in
very good agreement with the experimental data. Furthermore, a linear function
was derived which is able to predict the wet film thickness for a wide range
of velocity and initial volume combinations.
In this article, we propose a complete theoretical framework for blade coating
with a trapezoidal applicator shape, which does not only describe the
influence of coating speed and ink volume, but also includes the ink
properties, namely viscosity and surface tension.
In addition, for both the linear function of the previous work and the
analytical formula, an expression is derived to describe an accelerated
coating process to generate an even and uniform wet film with a constant
targeted thickness. Finally, experiments are performed where the velocity of
the applicator is increased according to the derived expressions. The
resulting wet films provide a thickness as uniform as predicted and thus
validate our simulations and analytical model. Thus, they prove to be powerful
tools for all kinds of research and development using blade coating, e.g. OPV
module production.

##  2 Material and Methods

###  2.1 Theoretical description of wet film thickness for trapezoidal
applicator geometry

The theory of wet film formation dates back to the work of Landau-Levich
Landau and Levich (1988) and Derjaguin Derjaguin (1993). In their work, they
describe how the wet film is generated on a flat plate which is dragged out of
a liquid (mixture of solvent and solute) reservoir. They assume a relatively
fast coating velocity, so effects of the drying process can be neglected
(Landau-Levich regime). Both articles propose an analytical expression for the
generated wet film

$h=1.34\cdot R\cdot Ca^{2/3},$ (1)

where $h$ denotes the wet film thickness (SI-unit: $m$) and $R$ the radius of
the down-stream meniscus (SI-unit: $m$). The capillary number $Ca$ is defined
as

$Ca=\frac{\mu\cdot u}{\sigma},$ (2)

where $\mu$ is the viscosity of the liquid (SI-unit: $Pa$ $s$), $u$ is the
velocity of the plate (SI-unit: $m/s$) and $\sigma$ is the surface tension of
the liquid in air environment (SI-unit: $N/m$). The theory was applied to
meniscus-guided coating processes where cylindrical applicator geometries are
used to predict the resulting wet film thickness Nickel et al. (2012); Park
and Han (2009).
It is found that the liquid volume beneath the applicator affects the radius
of the down-stream meniscus and consequently also the resulting wet film
thickness. Moreover, for cylindrical applicators also the contact angle
between the applicator and the coating liquid varies with the volume, which
makes the system even more complex. In contrast, applicators with trapezoidal-
shaped tips provide constant contact angles between applicator and liquid as
long as the volume is greater than a critical minimal value. In Gutenev et al.
(2003), a tilted plate moves along a substrate. The gap between plate and
substrate is filled with liquid. The authors derive an expression for the
radius from fundamental geometrical considerations, if $h/R<<1$

$R(S)=\sqrt{\frac{S}{\tan(\frac{\pi-\varphi}{2})+\frac{\varphi-\pi}{2}}}.$ (3)

Here, $S$ denotes the cross-sectional area of the liquid (SI-unit: $m^{2}$)
and $\varphi$ the opening angle in radians between the substrate and the
plate. In Figure 2, the cross-sectional view of the coating process is shown
and all relevant parameters for the calculation of the down-stream meniscus
radius are indicated, apart from the applicator width $w$ (SI-unit $m$) that
is orthogonal to the drawing plane.

Figure 2: Cross-sectional view of the meniscus-guided coating process. In the
process, the coating liquid (light blue) is injected between the applicator
tip (red) and the substrate (dark blue). The down-stream meniscus $R$, which
mainly determines the wet film thickness, is indicated and is a function of
the area $S$ (hatched area).

The applicator moves with a velocity $u$ at a height of $g$ over the substrate
(SI-unit: $m$). Together with the length of the applicator tip $l$ (SI-unit:
$m$), a minimal volume can be defined. The angle $\varphi$ can be derived from
the applicator geometry. Assuming a symmetrical liquid distribution beneath
the applicator, which is valid for small velocities, the relevant liquid area
$S$ (SI-unit: $m^{2}$) can be derived from basic geometrical considerations

$S(V_{0})=\frac{\frac{g^{2}}{\tan(\varphi)}+\frac{V_{0}}{w}-g\cdot l}{2}.$ (4)

Inserting equations 2, 3, and 4 in 1, an analytical expression to predict the
wet film thickness as a function of the coating velocity $u$ and the initial
volume $V_{0}$ can be derived

$h_{theo}(u,V_{0})=0.95\sqrt{\frac{\left[\frac{g^{2}}{\tan(\varphi)}+\frac{V_{0}}{w}-g\cdot
l\right]}{\tan(\frac{\pi-\varphi}{2})+\frac{\varphi-\pi}{2}}}\left(\frac{\mu\cdot
u}{\sigma}\right)^{2/3}.$ (5)

However, this theoretical expression is only able to predict layer thicknesses
for small coating distances where the volume loss is neglectable.

###  2.2 The accelerated coating process for a constant film thickness

For OPV module-relevant coating distances (i.e. $>$10 $cm$), the initial
volume $V_{0}$ will decrease considerably by the process itself. For a coating
distance $x$, the fluid volume beneath the applicator $V(x)$ can be written as

$V(x)=V_{0}-h\cdot w\cdot x.$ (6)

In a previous work Gumpert et al. (2023), we presented a computational fluid
dynamics (CFD) simulation as a digital twin of the coating process which only
requires a few, experimentally easily accessible, liquid properties to predict
the wet film thickness with good accuracy. Based on that, we derived a linear
fit function for one specific OPV material system to predict the wet film
thickness

$h_{fit}(u,V_{0})=(b+c\cdot V_{0})\cdot u^{2/3},$ (7)

where $b$ and $c$ are fit parameters which can be determined by the simulation
results.
In both, Eq. 5 and Eq. 7, it can be seen that the wet film thickness is a
function of the fluid volume and the velocity. Therefore, the loss of liquid
during the coating process can be compensated by gradually increasing the
velocity of the applicator during the coating process, a technique that can
easily be implemented. For a constant film thickness $h_{c}$, an expression
$u_{theo}(x)$ to describe the velocity at position $x$ can be derived from
Equation 5

$u_{theo}(x)=1.08\frac{\sigma}{\mu}\left\\{h_{c}\sqrt{\frac{\tan(\frac{\pi-\varphi}{2})+\frac{\varphi-\pi}{2}}{\left[\frac{g^{2}}{\tan(\varphi)}+\frac{V_{0}}{w}-h\cdot
x-g\cdot l\right]}}\right\\}^{3/2},$ (8)

and a respective expression $u_{fit}(x)$ can be derived from Equation 7

$u_{fit}(x)=\left[\frac{h_{c}}{b+c(V_{0}-h\cdot w\cdot x)}\right]^{3/2}.$ (9)

###  2.3 Material system P3HT:O-IDTBR

Different experiments are carried out to validate the proposed accelerated
coating approach. In all of these experiments, the material system
P3HT:O-IDTBR is used which is based on the photoactive material combination of
the donor polymer poly(3- hexylthiophene) (P3HT) and the non-fullerene
acceptor (5Z,5’Z)-5,5’-((7,7’-(4,4,9,9-tetraoctyl-4, 9-dihydros-
indaceno[1,2-b:5,6-b’]dithiophene-2,7- diyl)bis(benzo
[c][1,2,5]thiadiazole-7,4-diyl))bis(methanylylidene bis
(3-ethyl-2-thioxothiazolidin-4-one))) (O-IDTBR). The material system offers
good processing and stability properties Strohm et al. (2018); Armin et al.
(2021) and thus, has a high relevance for large-area industrial OPV
applications. In our previous work Gumpert et al. (2023), we experimentally
determined the different fluid-related properties of P3HT:O-IDTBR (dynamic
viscosity $\mu$, surface tension $\sigma$, and the contact angles with the
applicator $\alpha$ and the glass substrate $\beta$) which are required for
both, the numerical simulation and the theoretical approach. Further
information about the material system can be found in our previous paper.

##  3 Results and Discussion

###  3.1 Comparison of theory, simulation and experiment

In Gumpert et al. (2023), experiments are performed to determine the wet film
thickness of a blade-coated P3HT:O-IDTBR layer as a function of $V_{0}$ and
$u$. The developed CFD simulation is shown to reproduce the experimental
results with high precision. In Figure 3, we compare these experimental and
simulated results with calculations from our newly derived analytical
expression (Eq. 5).

Figure 3: Theoretically predicted (dashed lines), simulated (solid lines) and
experimentally measured (squares with corresponding error bars) wet film
thicknesses as a function of the coating velocity for different initial liquid
volumes.

Overall, the theoretical prediction shows great agreement with the simulation
and experimental results. For velocities below 50 $mm/s$, a maximal difference
between simulated and theoretically predicted wet film thickness of $\sim$ 0.5
$\mu m$ is obtained. For faster coating velocities, the theoretically
calculated wet film thicknesses slightly differ from simulation and
experimental results. For such high velocities, the assumption of a
symmetrically distributed liquid volume beneath the applicator (as shown in
Figure 2) has been shown not to be fulfilled anymore Gumpert et al. (2023).
The meniscus is dragged more to the down-stream side, which can explain the
differences in wet film thickness obtained by the different investigation
methods. Photographs from the experiment confirm this explanation (see Figure
4).

Figure 4: Cross-sectional view of the experimental setup, where the tip of the
applicator (red) and the coating ink (black) are visible. a) Symmetric fluid
distribution, if the applicator is at rest (0 $mm/s$). b) Asymmetric fluid
distribution, if the applicator moves with high coating velocity (90 $mm/s$).

If the applicator is at rest (Figure 4 a)), the coating fluid is symmetrically
distributed beneath the applicator, whereas a unsymmetrical fluid distribution
can be observed for a coating velocity of 90 $mm/s$ (Figure 4 b)). The fluid
distribution in the CFD simulations can be seen in the supplementary data.
The results clearly confirm the validity of our theoretical model, which has
been specially adapted to the geometry and physics of our system of
investigation.

###  3.2 Influence of fluid parameters on the wet film thickness

In Gumpert et al. (2023), all simulations and experiments are performed with
the same coating fluid, namely a P3HT:O-IDTBR ink, which is an established
material for the active layer in OPV. For this material, the surface tension
and viscosity have been determined experimentally, since these crucial fluid
parameters for the coating process and the respective simulation. However, the
development of novel materials for OPV is a current and very active research
topic Zhang et al. (2022). The creation of a large variety and number of new
materials aims at further pushing the power conversion efficiency Zhan et al.
(2022); Zhu et al. (2022) or to improve the stability of the active material
Xian et al. (2022); Liu et al. (2022). However, each new material provides
different fluid properties impacting the wet film thickness.
Therefore, our proposed simulation/theory approach is tested with respect to
its flexibility regarding variation of fluid properties in the following. In
Figure 5, the theoretical and simulated results for an initial volume $V_{0}$
of 80 $\mu l$ are shown. The wet film thicknesses are determined for different
viscosities (Figure 5 a)) and surface tensions of the fluid (Figure 5 b)) by
numerical simulations and the analytical expression.

Figure 5: The wet film thickness is plotted as a function of viscosity $\mu$
(a)) and surface tension $\sigma$ (b)) for an initial liquid volume $V_{0}$ of
80 $\mu l$ for different coating speeds $u$. In both a) and b), square markers
indicate the simulation results and the theoretical calculations are plotted
as dashed lines.

In Figure 5 a), results from the CFD simulation and the theory for velocities
are very consistent for velocities below 50 $mm/s$. For faster coating
velocities, the theory predicts slightly higher thicknesses compared to the
simulation. This can again be explained by an asymmetrical fluid distribution
under the applicator. For the dependence on the surface tension of the fluid,
numerical simulation and analytical expression show an excellent agreement for
all values of $\mu$ and $\sigma$ (see Figure 5 b)).
Both, our CFD simulation model as well as our adapted theoretical function,
are shown to provide the same results regarding variations in fluid
properties, which strongly suggests that these results also match the real
experimental findings, and thus both be used to predict and manipulate the
coating thickness, as will be shown in the following chapter.
In the supplementary information, a similar figure to Fig. 5 can be found for
an initial volume $V_{0}$ of 70 $\mu l$.

###  3.3 Homogeneous wet films processed by accelerated coating

For the investigation and implementation of an accelerated coating strategy,
we used a Zehntner ZAA 2300 blade coating device that was electrically
modified by Automatic Research GmbH to enable the pre-programming of a time-
dependent speed profile. The velocity of the applicator $u$ can be changed
every 100 $ms$. As a consequence, we can apply our proposed fluid-specific
acceleration strategy to compensate for the steadily decreasing film thickness
upon coating due to the constant loss of fluid volume. Thus, the steadily
decreasing film thickness could be compensated by using a fluid-specific
accerlation profile. To apply the compensation strategy, the previous
equations needs to be reformulated to describe the velocity as a function of
time. Based on Equation 8, the velocity as a function of time can be written
as

$u_{theo}(t)=1.08\frac{\sigma}{\mu}\left\\{h_{c}\sqrt{\frac{\tan(\frac{\pi-\varphi}{2})+\frac{\varphi-\pi}{2}}{\left[\left(\frac{g^{2}}{\tan(\varphi)}+\frac{V_{0}}{w}-g\cdot
l\right)^{7/4}-1.9\frac{t\cdot\sigma\cdot
h_{c}^{5/2}\left(\tan\left(\frac{\pi-\varphi}{2}\right)+\frac{\varphi-\pi}{2}\right)^{3/4}}{\mu}\right]^{4/7}}}\right\\}^{3/2}.$
(10)

An almost equivalent, but more compact, equation can be derived from the fit
function approach

$u_{fit}(t)=\left\\{\frac{h_{c}}{[(c\cdot V_{0}+b)^{5/2}-2.5\cdot c\cdot
h_{c}^{5/2}\cdot w\cdot t]^{2/5}}\right\\}^{3/2}.$ (11)

With these equations, the necessary velocities for a coating process with
constant thickness $h_{c}$ can be calculated and the respective time-velocity
tables can be programmed in the device. For three different initial volumes
$V_{0}$ (70, 80, and 90 $\mu l$) and a targeted constant wet film thickness
$h_{c}$ of 10 $\mu m$ the time-velocity tables are calculated with the fully
theoretical- (Eq. 10) and the fit function- (Eq. 11) approach for 15 $cm$. The
resulting velocity profiles are shown in the following figure as a function of
the corresponding substrate position $x$.

Figure 6: The coating velocity $u$ as a function of the substrate position is
determined by the theoretical expression (solid lines) and the fit function-
approach (dashed lines) for three different $V_{0}$ (70, 80, and 90 $\mu l$).

For all initial volumes $V_{0}$, the velocities profiles, based on theoretical
considerations and on the fit function, show some differences. The reason for
the discrepancies can be seen in Figure 3. Each approach predicts a slightly
different velocity to manufacture a 10 $\mu m$ thick wet film e.g. for an
initial volume of 90 $\mu l$.
For the experimental validation of the accelerated coating strategy, the
coating velocities for three different initial volumes (70, 80, and 90 $\mu
l$) are calculated with Eq. 11. In Figure 7, a photograph of a substrate after
the drying process is shown. The accelerated coating strategy was applied to
create a uniform film thickness of 10 $\mu m$ onto the substrate, the initial
volume of the coating ink was 90 $\mu l$.

Figure 7: Photograph of a P3HT:O-IDTBR film deposited with accelerated balde
coating using 90 $\mu l$ initial volume. The coating direction $x$ is
indicated as well as the regarding distance from the coating start point
($x=0mm$).

The dry film thicknesses are measured to validate the accelerated coating
strategy. From the composition of the material system, we can conclude the wet
film thickness from the dry film thickness Gumpert et al. (2023). The starting
point of the measurements is 20 $mm$ to exclude effects of the initial
movement of the applicator on the film thickness. The thickness of the coating
is measured every 10 $mm$ until a coating distance of 110 $mm$ has been
reached. The respective calculated wet film thicknesses are plotted in Figure
8 a) - c) (blue squares and error bars). CFD simulations were conducted to
compare the two methods of calculating the velocity profiles. CFD results,
where the velocity profile is calculated according to the theoretically
derived equation are plotted as green dashed lines whereas purple dash-dotted
lines indicate the CFD results, where the velocity profile is based on the
fluid-specific fit function.

Figure 8: a) - c) The experimentally determined wet film thicknesses (blue
squares with error bars, averaged over two samples per variation) are shown
for the initial volumes 70 $\mu l$, 80 $\mu l$, and 90 $\mu l$, respectively.
CFD results, which are based on the acceleration strategy according to Eq. 10
(green dashed lines) and Eq. 11 (purple dash-dotted lines). The targeted wet
film thickness (10 $\mu m$) is plotted as solid orange line.

For all investigated initial volumes, the measured film thickness is constant
over the whole evaluable distance and very close to the targeted value (orange
line). In addition, the results of the corresponding CFD simulations, which
also incorporate the respective acceleration profiles, are plotted as dashed
line, and show as well an excellent agreement with the targeted and
experimentally obtained values. Note: The deviation of the data point at 110
mm for an initial volume of 80 $\mu l$ is due to an experimental error caused
by an inhomogeneous drying process close to the end position of the applicator
bar, which also explains the relatively large error bar.
This data proves that both our simulative model and our analytical equation
can be used to create a full set of process parameters, which can be easily
applied in practice to fabricate homogeneous coating over large distances.
In fact, we already implemented this approach very recently into the
manufacturing process of high-performance large-area OPV modules, which
enabled us to yield a new world record efficiency for OPV modules with 14.5
$\%$ on 200 $cm^{2}$ NREL ; FAU . This module is depicted in Figure 9 and
demonstrates the excellent film homogeneity over the whole module area, which
is crucial for its high power conversion efficiency.

Figure 9: Photograph of the organic photovoltaic (OPV) module with a new world
record power conversion efficiency (PCE) of 14.5 $\%$ on 204 $cm^{2}$ enabled
by accelerated blade coating. The substrate size is 165 mm x 165 mm.

CFD simulations are performed where the applicator acceleration is determined
once with the theoretical expression and once with the formula, which is based
on the fit function. The results of the CFD simulations are plotted in Figure
8 a) - c) as green dashed and purple dash-dotted lines for the theoretical-
and fit function-approaches, respectively. Both approaches result in almost
identical simulated wet film thicknesses. According to the CFD simulation, the
acceleration, which is based on theoretical considerations, leads to slightly
thicker wet films compared to the acceleration, according to Equation 11.

##  4 Conclusion

In this article, we derived a theoretical expression to predict the height of
the deposited wet film in the doctor blading process. The predicted wet film
thicknesses of the theoretical description have been compared with
experimental data and results of a corresponding CFD simulation and show great
agreement in all investigated cases, which includes variations of the ink’s
viscosity and surface tension. Thus, both the proposed theoretical description
and the simulation model enable an extremely time- and resource-efficient
approach for high-throughput research, which is crucial for fields like OPV,
where a large number of novel materials is newly developed in a very short
time.
Expressions have been derived for both the theoretical equation and the fit
function to describe an acceleration of the applicator to compensate the
consumption of fluid during the coating process and to consequently achieve
uniform layers with predefined thickness. This acceleration strategy was
experimentally tested for long coating distances and the results of
experiment, simulation, and calculation all show similar values close to the
targeted thickness over the whole coating distance.
This newly developed approach was finally used to fabricate uniform large-area
coatings for high-performance organic solar modules and allowed us to break
the world record for OPV module efficiency, which underlines the importance
and applicability of this work.

## CRediT authorship contribution statement

Fabian Gumpert: Investigation, Methodology, Visualization, Writing - Original
Draft. Annika Janßen: Investigation, Validation, Writing - Review $\&$
Editing. Robin Basu: Validation, Writing - Review $\&$ Editing. Christoph J.
Brabec: Resources, Writing - Review $\&$ Editing. Hans-Joachim Egelhaaf:
Resources , Writing - Review $\&$ Editing. Jan Lohbreier: Supervision, Writing
- Review $\&$ Editing. Andreas Distler: Supervision, Writing - Review $\&$
Editing.

## Declaration of competing interest

The authors declare no competing financial interest or personal relationship
that could have appeared to influence the work reported in this paper.

## Data availability

Data will be made available on request.

## Acknowledgements

Funding: This work was supported by the Bavarian State Ministry for Science
and Art via the Energy Campus Nürnberg (EnCN); the European Union’s Horizon
2020 research and innovation program [Grant number 952911 and 101007084].

## References

  * Armin et al. (2021) Armin, A., Li, W., Sandberg, O.J., Xiao, Z., Ding, L., Nelson, J., Neher, D., Vandewal, K., Shoaee, S., Wang, T., Ade, H., Heumüller, T., Brabec, C., Meredith, P., 2021.  A history and perspective of non‐fullerene electron acceptors for organic solar cells.  Advanced Energy Materials 11.  URL: http://dx.doi.org/10.1002/aenm.202003570, doi:10.1002/aenm.202003570.
  * Berny et al. (2015) Berny, S., Blouin, N., Distler, A., Egelhaaf, H.J., Krompiec, M., Lohr, A., Lozman, O.R., Morse, G.E., Nanson, L., Pron, A., Sauermann, T., Seidler, N., Tierney, S., Tiwana, P., Wagner, M., Wilson, H., 2015.  Solar trees: First large-scale demonstration of fully solution coated, semitransparent, flexible organic photovoltaic modules.  Advanced Science 3.  URL: https://doi.org/10.1002/advs.201500342, doi:10.1002/advs.201500342.
  * Derjaguin (1993) Derjaguin, B., 1993.  On the thickness of the liquid film adhering to the walls of a vessel after emptying.  Progress in Surface Science 43, 134–137.  doi:10.1016/0079-6816(93)90022-n.
  * (4) FAU, .  World record in solar energy.  URL: https://www.fau.eu/2023/12/19/news/research/world-record-in-solar-energy/. accessed: 2024-01-08.
  * Gu et al. (2017) Gu, X., Zhou, Y., Gu, K., Kurosawa, T., Guo, Y., Li, Y., Lin, H., Schroeder, B.C., Yan, H., Molina-Lopez, F., Tassone, C.J., Wang, C., Mannsfeld, S.C.B., Yan, H., Zhao, D., Toney, M.F., Bao, Z., 2017.  Roll-to-roll printed large-area all-polymer solar cells with 5% efficiency based on a low crystallinity conjugated polymer blend.  Advanced Energy Materials 7.  URL: https://doi.org/10.1002/aenm.201602742, doi:10.1002/aenm.201602742.
  * Gumpert et al. (2023) Gumpert, F., Janßen, A., Brabec, C.J., Egelhaaf, H.J., Lohbreier, J., Distler, A., 2023.  Predicting layer thicknesses by numerical simulation for meniscus-guided coating of organic photovoltaics.  Engineering Applications of Computational Fluid Mechanics 17.  doi:10.1080/19942060.2023.2242455.
  * Gutenev et al. (2003) Gutenev, P.I., Pyatnitskii, A.M., Klimova, N.V., 2003.  Liquid entrainment from the meniscus of a liquid wedge by a moving horizontal plate.  Colloid Journal 65, 301–304.  URL: https://doi.org/10.1023/a:1024246520784, doi:10.1023/a:1024246520784.
  * Hu et al. (2022) Hu, Y., Wang, J., Yan, C., Cheng, P., 2022.  The multifaceted potential applications of organic photovoltaics.  Nature Reviews Materials 7, 836–838.  URL: https://doi.org/10.1038/s41578-022-00497-y, doi:10.1038/s41578-022-00497-y.
  * Iliopoulos and Scriven (2005) Iliopoulos, I., Scriven, L.E., 2005.  A blade-coating study using a finite-element simulation.  Physics of Fluids 17.  URL: https://doi.org/10.1063/1.2140226, doi:10.1063/1.2140226.
  * Landau and Levich (1988) Landau, L., Levich, B., 1988.  Dragging of a liquid by a moving plate.  Dynamics of Curved Fronts , 141–153doi:10.1016/b978-0-08-092523-3.50016-2.
  * Liu et al. (2022) Liu, Y.F., Zhang, S.W., Li, Y.X., Li, S.L., Huang, L.Q., Jing, Y.N., Cheng, Q., Xiao, L.G., Wang, B.X., Han, B., et al., 2022.  Solution-processed molybdenum oxide hole transport layer stabilizes organic solar cells.  Chinese Journal of Polymer Science 41, 202–211.  doi:10.1007/s10118-022-2873-3.
  * Mitsoulis and Athanasopoulos (2010) Mitsoulis, E., Athanasopoulos, G., 2010.  Numerical simulation of blade-over-roll coating forming flows.  Comput. Methods Mater. Sci 10, 214–224.
  * NG et al. (2022) NG, L.W.T., Lee, S.W., Chang, D.W., Hodgkiss, J.M., Vak, D., 2022.  Organic photovoltaics’ new renaissance: Advances toward roll-to-roll manufacturing of non-fullerene acceptor organic photovoltaics.  Advanced Materials Technologies 7.  URL: https://doi.org/10.1002/admt.202101556, doi:10.1002/admt.202101556.
  * Nickel et al. (2012) Nickel, F., Sprau, C., Klein, M.F., Kapetana, P., Christ, N., Liu, X., Klinkhammer, S., Lemmer, U., Colsmann, A., 2012.  Spatial mapping of photocurrents in organic solar cells comprising wedge-shaped absorber layers for an efficient material screening.  Solar Energy Materials and Solar Cells 104, 18–22.  URL: http://dx.doi.org/10.1016/j.solmat.2012.04.026, doi:10.1016/j.solmat.2012.04.026.
  * (15) NREL, .  Champion photovoltaic module efficiency chart.  URL: https://www.nrel.gov/pv/module-efficiency.html. accessed: 2024-01-08.
  * Park and Han (2009) Park, B., Han, M.y., 2009.  Photovoltaic characteristics of polymer solar cells fabricated by pre-metered coating process.  Optics Express 17, 13830.  doi:10.1364/oe.17.013830.
  * Schmidt et al. (2009) Schmidt, M., Schloßer, U., Schollmeyer, E., 2009.  Computational fluid dynamics investigation of the static pressure at the blade in a blade coating process.  Textile Research Journal 79, 579–584.  URL: http://dx.doi.org/10.1177/0040517508093440, doi:10.1177/0040517508093440.
  * Singh and Ormiston (2021) Singh, A.R., Ormiston, S.J., 2021.  Cfd analysis of blade coating from a reservoir onto a horizontal substrate using a homogeneous two-phase model.  The Canadian Journal of Chemical Engineering 100, 349–362.  URL: https://doi.org/10.1002/cjce.24096, doi:10.1002/cjce.24096.
  * Strohm et al. (2018) Strohm, S., Machui, F., Langner, S., Kubis, P., Gasparini, N., Salvador, M., McCulloch, I., Egelhaaf, H.J., Brabec, C.J., 2018.  P3ht: non-fullerene acceptor based large area, semi-transparent pv modules with power conversion efficiencies of 5$\%$ processed by industrially scalable methods.  Energy $\&$ Environmental Science 11, 2225–2234.  URL: http://dx.doi.org/10.1039/c8ee01150h, doi:10.1039/c8ee01150h.
  * Sun et al. (2022) Sun, R., Wu, Y., Yang, X., Gao, Y., Chen, Z., Li, K., Qiao, J., Wang, T., Guo, J., Liu, C., Hao, X., Zhu, H., Min, J., 2022.  Single-junction organic solar cells with 19.17% efficiency enabled by introducing one asymmetric guest acceptor.  Advanced Materials 34.  URL: https://doi.org/10.1002/adma.202110147, doi:10.1002/adma.202110147.
  * Tsai et al. (2015) Tsai, P.T., Yu, K.C., Chang, C.J., Horng, S.F., Meng, H.F., 2015.  Large-area organic solar cells by accelerated blade coating.  Organic Electronics 22, 166–172.  URL: https://doi.org/10.1016/j.orgel.2015.03.001, doi:10.1016/j.orgel.2015.03.001.
  * Xian et al. (2022) Xian, K., Zhang, S., Xu, Y., Liu, J., Zhou, K., Peng, Z., Li, M., Zhao, W., Chen, Y., Fei, Z., et al., 2022.  Refining acceptor aggregation in nonfullerene organic solar cells to achieve high efficiency and superior thermal stability.  Science China Chemistry 66, 202–215.  doi:10.1007/s11426-022-1394-y.
  * Zhan et al. (2022) Zhan, L., Li, S., Li, Y., Sun, R., Min, J., Bi, Z., Ma, W., Chen, Z., Zhou, G., Zhu, H., Shi, M., Zuo, L., Chen, H., 2022.  Desired open-circuit voltage increase enables efficiencies approaching 19% in symmetric-asymmetric molecule ternary organic photovoltaics.  Joule 6, 662–675.  URL: https://doi.org/10.1016/j.joule.2022.02.001, doi:10.1016/j.joule.2022.02.001.
  * Zhang et al. (2022) Zhang, G., Lin, F.R., Qi, F., Heumüller, T., Distler, A., Egelhaaf, H.J., Li, N., Chow, P.C.Y., Brabec, C.J., Jen, A.K.Y., Yip, H.L., 2022.  Renewed prospects for organic photovoltaics.  Chemical Reviews 122, 14180–14274.  URL: http://dx.doi.org/10.1021/acs.chemrev.1c00955, doi:10.1021/acs.chemrev.1c00955.
  * Zhu et al. (2022) Zhu, L., Zhang, M., Xu, J., Li, C., Yan, J., Zhou, G., Zhong, W., Hao, T., Song, J., Xue, X., Zhou, Z., Zeng, R., Zhu, H., Chen, C.C., MacKenzie, R.C.I., Zou, Y., Nelson, J., Zhang, Y., Sun, Y., Liu, F., 2022.  Single-junction organic solar cells with over 19% efficiency enabled by a refined double-fibril network morphology.  Nature Materials 21, 656–663.  URL: https://doi.org/10.1038/s41563-022-01244-y, doi:10.1038/s41563-022-01244-y.


# Interpretable SincNet-based Deep Learning for
Emotion Recognition from EEG brain activity

Juan Manuel Mayor-Torres1, Mirco Ravanelli2, Sara E. Medina-DeVilliers3,
Matthew D. Lerner3,
and Giuseppe Riccardi1 1University of Trento, Department of Information
Engineering and Computer Science (DISI), Via Sommarive 5, Povo, Trento 38123,
2Mila - Quebec Artifical Intelligence Institute, Montreal, QC, Canada, 3
StonyBrook University, NY, USA, Department of Psychology corresponding author:
<EMAIL_ADDRESS>
###### Abstract

Machine learning methods, such as deep learning, show promising results in the
medical domain. However, the lack of interpretability of these algorithms may
hinder their applicability to medical decision support systems. This paper
studies an interpretable deep learning technique, called SincNet. SincNet is a
convolutional neural network that efficiently learns customized band-pass
filters through trainable sinc-functions. In this study, we use SincNet to
analyze the neural activity of individuals with Autism Spectrum Disorder
(ASD), who experience characteristic differences in neural oscillatory
activity. In particular, we propose a novel SincNet-based neural network for
detecting emotions in ASD patients using EEG signals. The learned filters can
be easily inspected to detect which part of the EEG spectrum is used for
predicting emotions. We found that our system automatically learns the
high-$\alpha$ (9-13 Hz) and $\beta$ (13-30 Hz) band suppression often present
in individuals with ASD. This result is consistent with recent neuroscience
studies on emotion recognition, which found an association between these band
suppressions and the behavioral deficits observed in individuals with ASD. The
improved interpretability of SincNet is achieved without sacrificing
performance in emotion recognition.

Figure 1: The proposed architecture for EEG-based Face Emotion Recognition.
The neural pipeline is composed of a _SincConv_ layer, three standard conv-
pool blocks (_Conv1_ , _Conv2_ , and _Conv3_), and a fully-connected network
(_DNN1_) connected to a softmax classifier.

##  I INTRODUCTION

The recent development of artificial intelligence fosters unprecedented
innovation in the healthcare domain. The availability of big data
repositories, the development of robust learning algorithms, and the
availability of appropriate computational resources are making technologies
such as deep learning, applicable and challenging the state-of-the-art
systems. Deep learning (DL) has indeed been used in many medical applications
[1], including disease diagnosis [2], and personalized medicine [3], just to
name a few. In such critical cases, the lack of interpretability of this
technology limits its widespread end-user adoption and may even lead to
adverse consequences [4].
Current deep neural networks map low-level data into higher-level concepts
using a pipeline of non-linear transformations [5]. Therefore, the final
prediction is normally not explained by the network, and end-users ( i.e.
healthcare professionals ) do not know if the outcome is based on solid
evidence or due to some statistical biases [6]. Moreover, the inspection of
the intermediate representations learned by the network rarely helps to
explain the neural predictions. Improving the interpretability of the current
technology is an essential condition for overcoming understandable skepticism,
reluctance, and hesitations of the medical personnel.
Interpretable deep learning has been the object of increasing research efforts
over the last years [7]. Historically, there was a trade-off between
performance and interpretability. Simple models like linear regression are
transparent but not competitive in terms of performance with deeper models
such as fully-connected, convolutional, and recurrent neural networks. A
possible solution may be post-hoc interpretability [7], in which, a complex
neural model is built and analyzed afterward . The interpretability can also
be achieved with surrogate models (e.g., locally interpretable model
explanations - LIME [8]) and with gradient-based methods like in the saliency
maps [9]. In contrast, the alternative is to train machine learning models
that are interpretable by design. Along this line, a novel model called
SincNet has been proposed [10]. SincNet is a convolutional neural network that
learns a custom filter-bank using sinc-based convolutional kernels. Following
the network training, the filters can be inspected in the frequency domain to
identify which parts of the spectrum are used by the neural network to perform
a prediction. The filter inspection is straightforward and often insightful,
as emerged in some recent studies [11, 12, 13]. The interpretability insights
are directly mapping the neural network parameters to the input signal.
SincNet has been originally proposed for processing audio sequences [10], but
it has been recently used for EEG-based brain signals as well. In particular,
SincNet has been successfully adopted for EEG-based motor-imagery tasks [12,
13]. The full potential of this model in high-level processing of input
stimuli (e.g. visual or audio), however, is yet to be explored.
In this paper, we propose SincNet for studying the brain activity of patients
with Autism Spectrum Disorder (ASD). In particular, we analyze EEG signals of
ASD and non-ASD individuals while performing a Facial Emotion Recognition
(FER) task. The EEG recordings feed a deep learning model based on SincNet,
which tries to guess patients’ emotions from their brain waves. From the study
of filters learned by SincNet, we found an interesting consistency with the
Power-Spectral-Density analysis of previous EEG studies on ASD. The improved
interpretability is achieved without sacrificing the performance of the
machine learning models. The proposed system provides a contribution to the
interpretability-by-design alternative by applying deep learning techniques to
the medical domain.

##  II Facial Emotion Recognition with SincNet

The Facial Emotion Recognition (FER) task measures the ability to identify
basic emotions in facial expressions. It is particular important to asses this
ability in individuals with Autism Spectrum Disorder (ASD) because they
frequently demonstrate impairments in the ability to accurately categorize and
label the emotional facial expressions of others. In this section, we first
describe the FER task. Then, we describe the proposed architecture for EEG-
based emotion recognition in ASD and typically developing (TD) individuals.

###  II-A FER Task Description

Eighty-eight 14-to-17-year old adolescent participants (48 without ASD, while
40 with ASD) completed a FER task. The FER task, called Diagnostic Analysis of
Nonverbal Behavior (DANVA-2) [14], consists of presenting 48 photographs in
random order. Each photograph represents one of following emotions: happy,
sad, angry, and fear. The picture was first shown to the users for 2 second
and, within this interval, their EEG signal was recorded. Subsequently, each
photograph was shown again (for up 4 seconds) to give the participant enough
time to select the emotion they have seen in the picture using a button box.
The EEG activity of each participant was fed into the proposed machine
learning system, which is designed to predict the actual emotion from the
given brain recordings. The full pipeline for EEG-based emotion recognition is
shown in Figure 1.

###  II-B EEG pre-processing

The EEG signals were captured with a 32-channel ActiCHAMP device from Brain
Products and digitized with a sampling rate of 500 Hz (16-bit resolution). The
data were recorded continuously using the BrainVision Recorder software and
processed using the BrainVision Analyzer 2.0 for offline data reduction. We
converted all the input channels averaging them into a single sequence or a
grand-average Event- Related Potential (ERP) representation for each
participant and for each emotion. This helps reducing noise and statistical
variability of EEG trials [15]. We employed a standard EEG pre-processing
pipeline based on bandpass filtering, amplitude normalization, and Zero
Component Analysis (ZCA) whitening [16]. We also apply the ADJUST algorithm
[17] and the PREP pipeline [18] to clean the EEG signal before feeding the
neural network.

###  II-C Intepretable SincNet

The resulting grand-average ERP representation feeds the proposed SincNet-
based system. SincNet is a convolutional neural network whose first layer,
called SincConv, is designed to learn tunable Finite Impulse Response (FIR)
filters. In standard CNNs, all the elements (taps) of each filter are learned
from the data. In SincConv, instead, we parametrize the kernels in order to
implement rectangular bandpass filters, whose cut-off frequencies are the only
two parameters learned from data. This can be achieved with the following
parametrization:

$y[n]=x[n]*g[n,f_{1},f_{2}],$ (1)

where $x[n]$ is the input EEG signal and $y[n]$ is the output of the SinConv
layer. The convolution is performed with the kernel $g$, which is defined in
this way:

$g[n,f_{1},f_{2}]=2f_{2}sinc(2\pi f_{2}n)-2f_{1}sinc(2\pi f_{1}n),$ (2)

where the frequencies $f_{1}$ and $f_{2}$ are the learned parameters. This
technique not only saves a lot of parameters but naturally leads to a more
interpretable model. Moreover, the filters depend on human-readable parameters
with a clear physical meaning. At the end of the training, the filters are
inspected to identify which parts of the spectrum are covered by filters. This
helps users better understand what the network has learned [11, 13]. In
addition to the SincConv layer, the proposed architecture employs three 2-D
convolutional blocks based on standard convolution, batch normalization,
pooling, ReLU activations, and dropout. Finally, we plug a fully connected
layer followed by a softmax classifier.

##  III EXPERIMENTAL SETUP

In this section, we describe our SincNet-based emotion recognition
architecture and its training procedure.

###  III-A Architecture details

The adopted SincConv layer employed 100 filters with a kernel size of 250
points (which represent the impulse response of the filter). As in the
original paper, [10], a Hamming window was used to mitigate ripples in the
filters. The 2-D convolutional blocks were based on 32, 64, and 128 channels
with kernel sizes of (100 x 10), (20 x 5), (5 x 2), respectively. Max-pooling
used kernel sizes of (10 x 5), (5 x 2), (2 x 2). Batch normalization was added
between the convolution and the ReLU activations. Next, we employed a single
fully connected layer composed of 1024 ReLU neurons. Dropout was used in both
convolutional and fully connected layers with a rate of 0.5. The final
prediction over the four emotions was performed with a softmax classifier.

###  III-B Network Training

We trained and evaluated the SincNet-based pipeline using a Leave-One-Trial-
Out (LOTO) approach [16]. Thus, for each participant we used 47 out of the 48
trials for training, leaving a different test trial out every time. Therefore,
we completed a total of 48 training/validation experiments. This modality was
needed due to the lack of in-domain data for this specific task. The neural
network was initialized with the standard Glorot’s initialization scheme [19].
We used categorical cross-entropy as a loss function. The gradient was
computed with the backpropagation algorithm, while parameters are updated
using the Adam optimizer [20]. We used a learning rate of $0.001$ with a
weight-decay penalty of $1e-05$. We trained the neural network for 400 epochs
with a batch size of 30. For more information on the neural architecture and
training modality, please refer to the open-source code repository of this
project111https://github.com/meiyor/SincNet-for-Autism-EEG-based-Emotion-
Recognition.

Figure 2: Power Spectral Density (PSD) learned by SincNet filters on TD and
ASD individuals. Significant differences are observed in high-$\alpha$ (9-13
Hz) and $\beta$ (13-30 Hz) bands. Figure 3: Accuracy comparison between FER
(humans), CNN, and SincNet. Human accuracy is significantly lower than the one
of deep learning systems.

##  IV RESULTS

In the following section, we report the experimental evidence that emerged
from the FER task using SincNet.

###  IV-A Filter Analysis

After training, we inspected the filters learned by SincNet for ASD and non-
ASD (TD) participants. To analyze which EEG frequency bands were used for the
emotion prediction, we performed a Fourier Transform of the learned filters.
We then averaged their frequency responses and computed the cumulative Power
Spectral Density (PSD) reported in Fig. 2. We observed significant differences
in the filters learned from ASD and non-ASD participants. In particular, an
attenuation in the high-$\alpha$ (9-13 Hz) and $\beta$ (13-30 Hz) emerged in
the cumulative PSD spectrum of ASD participants. We observed significant
differences between non-ASD and ASD groups on high-$\alpha$ (9-13 Hz)
$F(1,87)=3.331,p=0.000267$ and $\beta$ (13-30 Hz) $F(1,87)=2.05,p=0.00102$
bands after Bonferroni-Holm correction. Our findings are consistent with
previous studies on ASD, indicating that high-$\alpha$ and $\beta$
attenuations are related to FER deficits on individuals with ASD [21, 22].

Figure 4: Confusion Matrix for the SincNet pipeline on ASD.

Although this frequency-band attenuation has been observed in some EEG studies
including individuals with ASD, the causes of this are not fully understood in
the scientific community. Some authors hypothesized that this is associated
with multiple behavioral deficits of individuals with ASD [23, 24]. However,
it is worth notice that SincNet learns that these bands are not useful to
predict emotions in individuals with ASD. Notably, these predictions were
learned automatically from raw EEG data only, without providing any additional
information to the network.

###  IV-B Performance Analysis

Figure 3 compares the accuracy achieved by humans in FER with the one reached
by the deep learning systems based on CNNs and SincNet. As for the human
performance or FER, we found a small difference between ASD and non-ASD
accuracies (79% vs 81%, respectively), thus, showing some consistent deficits
in individuals with ASD performing FER tasks [23, 24]. Interestingly, deep
learning systems outperformed the FER human accuracy. This suggests that there
were some cases where a participant wrongly labeled the photograph, but the
SincNet-based system was able to detect the correct emotion from participants’
EEG brain activity. This difference is more evident in ASD participants, where
the SincNet improvement is greater (79% vs 92%) than FER. The SincNet
confusion matrix for the ASD group is shown in Figure 4. The proposed SincNet
pipeline turned out to slightly outperforming a CNN-based system (90% vs 85%
for the non-ASD group and 92% vs 91% for the ASD group). The CNN architecture
was obtained by replacing the SincConv layer with a standard convolution. This
improvement is consistent with what was observed for audio [10] and motor-
based EEG signal in previous studies [13]. In sum, our results confirm that
SincNet improves the interpretability of the model without sacrificing
performance.

##  V CONCLUSIONS

This paper has proposed the application of an interpretable deep-learning
architecture, SincNet, in a medical domain. We applied this model to study EEG
activity patterns of ASDs and non-ASDs in a FER task. Our results indicate
that SincNet transparently learns the high-$\alpha$ and $\beta$ suppressions
observed in ASD individuals when perceiving and recognizing emotional faces.
SincNet improves the interpretability of the neural model without affecting
its performance, thus offering a convenient way to avoid the performance
versus interpretability dilemma.

## ACKNOWLEDGMENT

The authors would like to thank the University of Trento High-Performance
Computing and Stony Brook Research Computing and Cyber-infrastructure which
was made possible by a $1.4M National Science Foundation grant (1531492). This
research was supported by NIMH grant R01MH110585, grants from the AAF Fund for
Communication.

## References

  * [1] Juan M Mayor Torres, Tessa Clarkson, Evgeny A Stepanov, Christian C Luhmann, Matthew D Lerner, and Giuseppe Riccardi.  Enhanced error decoding from error-related potentials using convolutional neural networks.  In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 360–363. IEEE, 2018\.
  * [2] Shu Lih Oh, Yuki Hagiwara, U. Raghavendra, Yuvaraj Rajamanickam, N. Arunkumar, M. Murugappan, and U. Acharya.  A deep learning approach for parkinson’s disease diagnosis from eeg signals.  Neural Computing and Applications, pages 1–7, 2018.
  * [3] G. Z. Papadakis, A. Karantanas, Tsiknakis H, Tsatsakis M, Spandidos A, and Marias D. A.  Deep learning opens new horizons in personalized medicine (review).  Biomedical Reports, 10(4):215–217, 2019.
  * [4] Muhammad Aurangzeb Ahmad, Carly Eckert, and Ankur Teredesai.  Interpretable machine learning in healthcare.  In Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, page 559–560, 2018.
  * [5] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning, volume 1.  MIT press Cambridge, 2016.
  * [6] Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Unmasking clever hans predictors and assessing what machines really learn.  Nature communications, 10(1):1–8, 2019.
  * [7] Christoph Molnar.  Interpretable Machine Learning.  2019\.  https://christophm.github.io/interpretable-ml-book/.
  * [8] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  "why should i trust you?": Explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 1135–1144, 2016.
  * [9] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In Proc. of ICLR, 2014.
  * [10] Mirco Ravanelli and Yoshua Bengio.  Speaker recognition from raw waveform with sincnet.  In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 1021–1028. IEEE, 2018.
  * [11] Mirco Ravanelli and Yoshua Bengio.  Interpretable convolutional filters with sincnet.  arXiv preprint arXiv:1811.09725, 2018.
  * [12] Hong Zeng, Zhenhua Wu, Jiaming Zhang, Chen Yang, Hua Zhang, Guojun Dai, and Wanzeng Kong.  Eeg emotion classification using an improved sincnet-based deep learning model.  Brain sciences, 9(11):326, 2019.
  * [13] Davide Borra, Silvia Fantozzi, and Elisa Magosso.  Interpretable and lightweight convolutional neural network for eeg decoding: application to movement execution and imagination.  Neural Networks, 129:55–74, 2020.
  * [14] Stephen Nowicki.  Manual for the receptive tests of the diagnostic analysis of nonverbal accuracy 2.  Atlanta, GA: Department of Psychology, Emory University, 2000.
  * [15] Arnaud Delorme, Makoto Miyakoshi, Tzyy-Ping Jung, and Scott Makeig.  Grand average erp-image plotting and statistics: A method for comparing variability in event-related single-trial eeg activities across subjects and conditions.  Journal of neuroscience methods, 250:3–6, 2015.
  * [16] Juan Manuel Mayor Torres, Tessa Clarkson, Kathryn M Hauschild, Christian C Luhmann, Matthew D Lerner, and Giuseppe Riccardi.  Facial emotions are accurately encoded in the neural signal of those with autism spectrum disorder: A deep learning approach.  Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 2021\.
  * [17] Andrea Mognon, Jorge Jovicich, Lorenzo Bruzzone, and Marco Buiatti.  Adjust: An automatic eeg artifact detector based on the joint use of spatial and temporal features.  Psychophysiology, 48(2):229–240, 2011.
  * [18] Nima Bigdely-Shamlo, Tim Mullen, Christian Kothe, Kyung-Min Su, and Kay A Robbins.  The prep pipeline: standardized preprocessing for large-scale eeg analysis.  Frontiers in neuroinformatics, 9:16, 2015.
  * [19] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Jaime Pineda, Ashley Juavinett, and Michael Datko.  Self-regulation of brain oscillations as a treatment for aberrant brain connections in children with autism.  Medical hypotheses, 79(6):790–798, 2012.
  * [22] Elisabeth VC Friedrich, Aparajithan Sivanathan, Theodore Lim, Neil Suttie, Sandy Louchart, Steven Pillen, and Jaime A Pineda.  An effective neurofeedback intervention to improve social interactions in children with autism spectrum disorder.  Journal of autism and developmental disorders, 45(12):4084–4100, 2015.
  * [23] Geraldine Dawson, Sara Jane Webb, and James McPartland.  Understanding the nature of face processing impairment in autism: insights from behavioral and electrophysiological studies.  Developmental neuropsychology, 27(3):403–424, 2005.
  * [24] Melissa H Black, Nigel TM Chen, Kartik K Iyer, Ottmar V Lipp, Sven Bölte, Marita Falkmer, Tele Tan, and Sonya Girdler.  Mechanisms of facial emotion recognition in autism spectrum disorders: Insights from eye tracking and electroencephalography.  Neuroscience & Biobehavioral Reviews, 80:488–515, 2017.


# CDPAM: Contrastive learning for perceptual audio similarity

###### Abstract

Many speech processing methods based on deep learning require an automatic and
differentiable audio metric for the loss function. The Dpam approach of
Manocha et al. [1] learns a full-reference metric trained directly on human
judgments, and thus correlates well with human perception. However, it
requires a large number of human annotations and does not generalize well
outside the range of perturbations on which it was trained. This paper
introduces Cdpam – a metric that builds on and advances Dpam. The primary
improvement is to combine contrastive learning and multi-dimensional
representations to build robust models from limited data. In addition, we
collect human judgments on triplet comparisons to improve generalization to a
broader range of audio perturbations. Cdpam correlates well with human
responses across nine varied datasets. We also show that adding this metric to
existing speech synthesis and enhancement methods yields significant
improvement, as measured by objective and subjective tests.

Index Terms—  perceptual similarity, audio quality, deep metric, speech
enhancement, speech synthesis

##  1 Introduction

Humans can easily compare and recognize differences between audio recordings,
but automatic methods tend to focus on particular qualities and fail to
generalize across the range of audio perception. Nevertheless, objective
metrics are necessary for many automatic processing tasks that cannot
incorporate a human in the loop. Traditional objective metrics like Pesq [2]
and Visqol [3] are automatic and interpretable, as they rely on complex
handcrafted rule-based systems. Unfortunately, they can be unstable even for
small perturbations – and hence correlate poorly with human judgments.

Recent research has focused on learning speech quality from data [1, 4, 5, 6,
7, 8]. One approach trains the learner on subjective quality ratings, e.g.,
MOS scores on an absolute (1 to 5) scale [4, 5]. Such “no-reference” metrics
support many applications but do not apply in cases that require comparison to
ground truth, for example for learned speech synthesis or enhancement [9, 10].

Manocha et al. [1] propose a “full-reference” deep perceptual audio similarity
metric (Dpam) trained on human judgments. As their dataset focuses on just-
noticeable differences (JND), the learned model correlates well with human
perception, even for small perturbations. However, the Dpam model suffers from
a natural tension between the cost of data acquisition and generalization
beyond that data. It requires a large set of human judgments to span the space
of perturbations in which it can robustly compare audio clips. Moreover,
inasmuch as its dataset is limited, the metric may generalize poorly to unseen
speakers or content. Finally, because the data is focused near JNDs, it is
likely to be less robust to large audio differences.

To ameliorate these limitations, this paper introduces Cdpam: a contrastive
learning-based multi-dimensional deep perceptual audio similarity metric. The
proposed metric builds on Dpam using three key ideas: (1) contrastive
learning, (2) multi-dimensional representation learning, and (3) triplet
learning. Contrastive learning is a form of self-supervised learning that
augments a limited set of human annotations with synthetic (hence unlimited)
data: perturbations to the data that should be perceived as different (or
not). We use multi-dimensional representation learning to separately model
_content_ similarity (e.g. among speakers or utterances) and _acoustic_
similarity (e.g. among recording environments). The combination of contrastive
learning and multi-dimensional representation learning allows Cdpam to better
generalize across content differences (e.g. unseen speakers) with limited
human annotation [11, 12]. Finally, to further improve robustness to large
perturbations (well beyond JND), we collect a dataset of judgments based on
triplet comparisons, asking subjects: “Is A or B closer to reference C?”

We show that Cdpam correlates better than Dpam with MOS and triplet comparison
tests across nine publicly available datasets. We observe that Cdpam is better
able to capture differences due to subtle artifacts recognizable to humans but
not by traditional metrics. Finally, we show that adding Cdpam to the loss
function yields improvements to the existing state of the art models for
speech synthesis [9] and enhancement [10]. The dataset, code, and resulting
metric, as well as listening examples, are available here:
https://pixl.cs.princeton.edu/pubs/Manocha_2021_CCL/

Fig. 1: Training architecture: (a) We first train an audio encoder using
contrastive learning, then (b) train the loss-net on JND data [1]. (c)
Finally, we fine-tune the loss-net on the newly collected dataset of triplet
comparisons.

##  2 Related Work

###  2.1 Perceptual audio quality metrics

Pesq [2] and Visqol [3] were some of the earliest models to approximate the
human perception of audio quality. Although useful, these had certain
drawbacks: (i) sensitivity to perceptually invariant transformations [13];
(ii) narrow focus (e.g. telephony); and (iii) non-differentiable, making it
impossible to be optimized for as an objective in deep learning. To overcome
the last concern, researchers train differentiable models to approximate Pesq
[7, 14]. The approach of Fu et al. [7] uses GANs to model Pesq, whereas the
approach of Zhang et al. [14] uses gradient approximations of Pesq for
training. Unfortunately, these approaches are not always optimizable, and may
also fail to generalize to unseen perturbations.

Instead of using conventional metrics (e.g. Pesq) as a proxy, Manocha et al.
[1] proposed Dpam that was trained directly on a new dataset of human JND
judgments. Dpam correlates well with human judgment for small perturbations,
but requires a large set of annotated judgments to generalize well across
unseen perturbations. Recently, Serra et al. [8] proposed Sesqa trained using
the same JND dataset [1], along with other datasets and objectives like Pesq.
Our work is concurrent with theirs, but direct comparison is not yet
available.

###  2.2 Representation learning

The ability to represent high-dimensional audio using compact representations
has proven useful for many speech applications (e.g. speech recognition [15],
audio retrieval [16]).

Multi-dimensional learning: Learning representations to identify the
underlying explanatory factors make it easier to extract useful information on
downstream tasks [17]. Recently in audio, Lee et al. [18] adapted Conditional
Similarity Networks (CSN) for music similarity. The objective was to
disentangle music into genre, mood, instrument, and tempo. Similarly, Hung et
al. [19] explored disentangled representations for timbre and pitch of musical
sounds useful for music editing. Chou et al. [20] explored the disentanglement
of speaker characteristics from linguistic content in speech signals for voice
conversion. Chen et al. [21] explored the idea of disentangling phonetic and
speaker information for the task of audio representation.

Contrastive learning: Most of the work in audio has focused on audio
representation learning [22, 23], speech recognition [24], and phoneme
segmentation [25]. To the best of our knowledge, no prior method has used
contrastive learning together with multi-dimensional representation learning
to learn audio similarity.

##  3 The CDPAM Metric

This section describes how we train the metric in three stages, depicted in
Fig 1: (a) pre-train the audio encoder using contrastive learning; (b) train
the loss-net on the perceptual JND data; and (c) fine-tune the loss-net on the
new perceptual triplet data.

###  3.1 Dataset

Self-supervised dataset: We borrow ideas from the SimCLR framework [26] for
contrastive learning. SimCLR learns representations by maximizing agreement
between differently augmented views of the same data example via a contrastive
loss in the latent space. An audio is taken and transformations are applied to
it to get a pair of augmented audio waveforms $x_{i}$ and $x_{j}$. Each
waveform in that pair is passed through an encoder to get representations.
These representations are further passed through a projection network to get
final representations $z$. The task is to maximize the similarity between
these two representations $z_{i}$ and $z_{j}$ for the same audio, in contrast
to the representation $z_{k}$ on an unrelated piece of audio $x_{k}$.

In order to combine multi-dimensional representation learning with contrastive
learning, we force our audio encoder to output two sets of embeddings:
acoustic and content. We learn these separately using contrastive learning. To
learn acoustic embedding, we consider data augmentation that takes the same
acoustic perturbation parameters but different audio content, whereas to learn
content embedding, we take different acoustic perturbation parameters but the
same audio content. The pre-training dataset consists of roughly 100K
examples.

JND dataset: We use the same dataset of crowd-sourced human perceptual
judgments proposed by Manocha et al. [1]. In short, the dataset consists of
around 55K pairs of human subjective judgments, each pair coming from the same
utterance, with annotations of whether the two recordings are _exactly the
same or different?_ Perturbations consist of additive linear background noise,
reverberation, coding/compression, equalization, and various miscellaneous
noises like pops and dropouts.

Fine-tuning dataset: To make the metric robust to large (beyond JND)
perturbations, we create a triplet comparison dataset. This improves the
generalization performance of the metric to include a broader range of
perturbations and also enhances the ordering of the learned space. We follow
the same framework and perturbations as Manocha et al. [1]. This dataset
consists of around 30K paired examples of crowd-sourced human judgments.

Type | Name |  VoCo [27] |  FFTnet [28] |  BWE [29] | Dereverb | HiFi-GAN | PEASS | VC | Noizeus
---|---|---|---|---|---|---|---|---|---
Conventional | MSE | 0.18 | 0.18 | 0.00 | 0.14 | 0.00 | 0.25 | 0.00 | 0.20
Pesq | 0.43 | 0.49 | 0.21 | 0.85 | 0.70 | 0.71 | 0.56 | 0.68
VISQOL | 0.50 | 0.02 | 0.13 | 0.75 | 0.69 | 0.66 | 0.49 | 0.61
1-10 JND metric | Dpam | 0.71 | 0.63 | 0.61 | 0.45 | 0.30 | 0.63 | 0.45 | 0.10
1-10 Ours (default) | Cdpam | 0.73 | 0.68 | 0.65 | 0.93 | 0.68 | 0.74 | 0.61 | 0.71
Table 1: Spearman correlation of models with various MOS tests. Models include
conventional metrics, DPAM and ours. Higher is better.

###  3.2 Training and Architecture

Encoder Fig. 1(a): The audio encoder consists of a 16 layer CNN with $15\times
1$ kernels that is downsampled by half every fourth layer. We use global
average pooling at the output to get a 1024-dimensional embedding, equally
split into acoustic and content components. The projection network is a small
fully-connected network that takes in a feature embedding of size 512 and
outputs an embedding of 256 dimensions. The contrastive loss is taken over
this projection. We use Adam [30] with a batch size of 16, learning rate of
$10^{-4}$, and train for 250 epochs.

We use the NT-Xent loss (Normalized Temperature-Scaled Cross-Entropy Loss)
proposed by Chen et al. [26]. The key is to map the two augmented versions of
an audio (positive pair) with high similarity and all other examples in the
batch (negative pairs) with low similarity. For measuring similarity, we use
cosine distance. For more information, refer to SimCLR [26].

Loss-network Fig. 1(b-c): Our loss-net is a small 4 layer fully connected
network that takes in the output of the audio encoder (acoustic embedding) and
outputs a distance (using the aggregated sum of the L1 distance of deep
features). This network also has a small classification network at the end
that maps this distance to a predicted human judgment. Our loss-net is trained
using binary cross-entropy between the predicted value and ground truth human
judgment. We use Adam with a learning rate of $10^{-4}$, and train for 250
epochs. For fine-tuning on triplet data, we use MarginRankingLoss with a
margin of $0.1$, using Adam with a learning rate of $10^{-4}$ for 100 epochs.

As part of online data augmentation to make the model invariant to small
delay, we decide randomly if we want to add a 0.25s silence to the audio at
the beginning or the end and then present it to the network. This helps to
provide shift-invariance property to the model, to disambiguate that in fact
the audio is similar when time-shifted. To also encourage amplitude
invariance, we also randomly apply a small gain (-20dB to 0dB) on the training
data.

Type | Name | FFTnet | BWE | HiFiGAN | Simulated
---|---|---|---|---|---
Conventional | MSE | 55.0 | 49.0 | 70.2 | 43.0
Pesq | 67.1 | 38.1 | 88.5 | 86.1
Visqol | 64.2 | 44.4 | 96.1 | 84.2
1-6 JND metric | Dpam | 61.5 | 87.7 | 93.2 | 71.8
1-6 Ours (default) | Cdpam | 88.5 | 75.9 | 96.5 | 87.7

Table 2: 2AFC accuracy of various models, including conventional metrics, DPAM
and ours. Higher is better.

##  4 Experiments

###  4.1 Subjective Validation

We use previously published diverse third-party studies to verify that our
trained metric correlates well with their task. We show the results of our
model and compare it with Dpam as well as more conventional objective metrics
such as MSE, Pesq [2], and Visqol [3].

We compute the correlation between the model’s predicted distance with the
publicly available MOS, using Spearman’s Rank order correlation (SC). These
correlation scores are evaluated per speaker where we average scores for each
speaker for each condition.

As an extension, we also check for 2AFC accuracy where we present one
reference recording and two test recordings and ask subjects which one sounds
more similar to the reference? Each triplet is evaluated by roughly 10
listeners. 2AFC checks for the exact ordering of similarity at per sample
basis, whereas MOS checks for aggregated ordering, scale, and consistency. In
addition to all evaluation datasets considered by Manocha et al. [1], we
consider additional datasets:

  1. 1.

Dereverberation [31]: consists of MOS tests to assess the performance of 5
deep learning-based speech enhancement methods.

  2. 2.

HiFi-GAN [32]: consists of MOS and 2AFC scores to assess improvement across 10
deep learning based speech enhancement models (denoising and dereverberation).

  3. 3.

PEASS [33]: consists of MOS scores to assess audio source separation
performance across 4 metrics: global quality, preservation of target source,
suppression of other sources, and absence of additional artifacts. Here, we
only look at global quality.

  4. 4.

Voice Conversion (VC) [34]: consists of tasks to judge the performance of
various voice conversion systems trained using parallel (HUB) and non-parallel
data (SPO). Here we only consider HUB.

  5. 5.

Noizeus [35]: consists of a large scale MOS study of non-deep learning-based
speech enhancement systems across 3 metrics: SIG-speech signal alone; BAK-
background noise; and OVRL-overall quality. Here, we only look at OVRL.

Results are displayed in Tables 1 and 2, in which our proposed metric has the
best performance overall. Next, we summarize with a few observations:

  * •

Similar to findings by Manocha et al. [1], conventional metrics like Pesq and
Visqol perform better on measuring large distances (e.g. Dereverb, HiFi-GAN)
than subtle differences (e.g. BWE), suggesting that these metrics do not
correlate well with human perception when measuring subtle differences.

  * •

We observe a natural compromise between generalizing to large audio
differences well beyond JND (e.g. FFTnet, VoCo, etc.) and focusing only on
small differences (e.g. BWE). As we see, Cdpam is able to correlate well
across a wide variety of datasets, whereas Dpam correlates best where the
audio differences are near JND. Cdpam scores higher than Dpam on BWE on MOS
correlation, but has a lower 2AFC score suggesting that Dpam might be better
at ordering individual pairwise judgments closer to JND.

  * •

Compared to Dpam, Cdpam performs better across a wide variety of tasks and
perturbations, showing higher generalizability across perturbations and
downstream tasks.

Name | ComArea$\downarrow$ | Mono$\uparrow$ | VoCo$\uparrow$ | FFTnet$\uparrow$ | BWE$\uparrow$
---|---|---|---|---|---
Dpam [1] | 0.76 | 0.89 | - | - | -
1-6 self-sup. (triplet m.learning) | 0.52 | 0.31 | 0.21 | 0.10 | 0.00
contrastive w/o mul-dim. rep. | 0.43 | 0.48 | 0.38 | 0.17 | 0.38
self-sup. (contrastive) | 0.34 | 0.53 | 0.34 | 0.58 | 0.60
+JND | 0.32 | 0.88 | 0.65 | 0.65 | 0.79
+JND+fine-tune (default) | 0.32 | 0.89 | 0.73 | 0.68 | 0.65

Table 3: Ablation studies. Sec 4.2 describes common area, monotonicity, and
Spearman correlations for 3 datasets. $\uparrow$ or $\downarrow$ is better.

###  4.2 Ablation study

We perform ablation studies to better understand the influence of different
components of our metric in Table 3. We compare our trained metric at various
stages: (i) after self-supervised training; (ii) after JND training; and (iii)
after triplet finetuning (default). To further compare amongst self-supervised
approaches, we also show results of self-supervised metric learning using
triplet loss. To also show improvements due to learning multi-dimensional
representations, we show results of a model trained using contrastive learning
without _content_ dimension. The metrics are compared on (i) robustness to
content variations; (ii) monotonic behavior with increasing perturbation
levels; and (iii) correlation with subjective ratings from a subset of
existing datasets.

Robust to content variations To evaluate robustness to content variations, we
create a test dataset of two groups: one consisting of pairs of recordings
that have the same acoustic perturbation levels but varying audio content; the
other consisting of pairs of recordings having different perturbation levels
and audio content. We calculate the common area between these normalized
distributions. Our final metric has the lowest common area, suggesting that it
is more robust to changing audio content. Decreasing common area also
corresponds with increasing MOS correlations across downstream tasks,
suggesting that the task of separating these two distribution groups may be a
factor when learning acoustic audio similarity.

Clustered Acoustic Space: To further quantify this learned space, we also
calculate the Precision of Top $K$ retrievals which measures the quality of
top $K$ items in the ranked list. Given 10 different acoustic perturbation
groups - each group consisting of 100 files having the same acoustic
perturbation levels but different audio content, we take randomly selected
queries and calculate the number of correct class instances in the top $K$
retrievals. We report the mean of this metric over all queries ($MP^{k}$).
Cdpam gets $MP^{k=10}$ = 0.92 and $MP^{k=20}$ = 0.87, suggesting that these
acoustic perturbation groups all cluster together in the learned space.

Monotonicity To show our metric’s monotonicity with increasing levels of
noise, we create a test dataset of recordings with different audio content and
increasing noise perturbation levels (both individual and combined
perturbations). We calculate SC between the distance from the metric and the
perturbation levels. Both Dpam and Cdpam behave equally monotonically with
increasing levels of noise.

MOS Correlations Each of the key components of Cdpam namely _contrastive-
learning_ , _multi-dimensional representation learning_ , and _triplet
learning_ have a significant impact on generalization across downstream tasks.
Surprisingly, even our self-supervised model has a non-trivial positive
correlation with increasing noise as well as MOS correlation across datasets.
This suggests that a self-supervised model not trained on any classification
or perceptual task is still able to learn useful perceptual knowledge. This is
true across datasets ranging from subtle to large differences suggesting that
contrastive learning can be a useful pre-training strategy.

###  4.3 Waveform generation using our learned loss

We show the utility of our trained metric as a loss function for the task of
waveform generation. We use the current state-of-the-art MelGAN [9] vocoder.
We train two models: i) single speaker model trained on LJ [36], and ii)
cross-speaker model trained on a subset of VCTK [37]. Both the models were
trained for around _2 million_ iterations until convergence. We take the
existing model and just add Cdpam as an additional loss in the training
objective.

We randomly select 500 unseen audio recordings to evaluate results. For the
single-speaker model, we use LJ dataset, whereas for the cross-speaker model,
we use the 20 speaker DAPS [38] dataset. We perform A/B preference tests on
Amazon Mechanical Turk (AMT), consisting of _Ours vs baseline_ pairwise
comparisons. Each pair is rated by 6 different subjects and then majority
voted to see which method performs better per utterance. As shown in Fig 2(a),
our models outperform the baseline in both categories. All results are
statistically significant with p $<$ $10^{-4}$. Our model is strongly
preferred over the baseline, but the maximum improvement is observed in the
cross-speaker scenario where our model performs best. Specifically, we observe
that MelGAN enhanced with Cdpam detects and follows the reference pitch better
than the baseline.

| Pesq | STOI | CSIG | CBAK | COVL | MOS
---|---|---|---|---|---|---
Noisy | 1.97 | 91.50 | 3.35 | 2.44 | 2.63 | 2.61
DEMUCS [10] | 3.01 | 95.00 | 4.39 | 3.44 | 3.73 | 3.73
DEMUCS+Cdpam | 3.10 | 95.07 | 4.46 | 3.55 | 3.82 | 3.86
Finetune Cdpam | 3.06 | 94.93 | 4.30 | 3.56 | 3.70 | 4.07

Table 4: Evaluation of denoising models using the VCTK [37] test set with four
objective measures and one subjective measure. Fig. 2: Subjective tests: (a)
In pairwise tests, ours is typically preferred over MelGAN for single speaker
and cross-speaker synthesis. (b) MOS tests show denoising methods are improved
by CDPAM.

###  4.4 Speech Enhancement using our learned loss

To further demonstrate the effectiveness of our metric, we use the current
state-of-the-art DEMUCS architecture based speech denoiser [10] and supplement
Cdpam in two ways: (i) DEMUCS+CDPAM: train from scratch using a combination of
L1, multi-resolution STFT and Cdpam; and (ii) Fintune Cdpam: pre-train on L1
and multi-resolution STFT loss and finetune on Cdpam. The training dataset
consists of VCTK [37] and DNS [39] datasets. For a fair comparison, we only
compare real-time (causal) models.

We randomly select 500 audio clips from the VCTK test set and evaluate scores
on that dataset. We evaluate the quality of enhanced speech using both
objective and subjective measures. For the objective measures, we use: i) Pesq
(from 0.5 to 4.5); (ii) Short-Time Objective Intelligibility (STOI) (from 0 to
100); (iii) CSIG: MOS prediction of the signal distortion attending only to
the speech signal (from 1 to 5); (iv) CBAK: MOS prediction of the
intrusiveness of background noise (from 1 to 5); (v) COVL: MOS prediction of
the overall effect (from 1 to 5). We compare the baseline model with both our
models. Results are shown in Table 4.

For subjective studies, we conducted a MOS listening study on AMT where each
subject is asked to rate the sound quality of an audio snippet on a scale of 1
to 5, with _1=Bad, 5=Excellent_. In total, we collect around 1200 ratings for
each method. We provide studio-quality audio as reference for high-quality,
and the input noisy audio as low-anchor. As shown in Fig 2(b), both our models
perform better than the baseline approach. We observe that our Finetune Cdpam
model scores the highest MOS score. This highlights the usefulness of using
Cdpam in audio similarity tasks. Specifically, Cdpam can identify and
eliminate minor human perceptible artifacts that are not captured by
traditional losses. We also note that higher objective scores do not guarantee
higher MOS, further motivating the need for better objective metrics.

##  5 Conclusion and Future work

In this paper, we present Cdpam, a contrastive learning-based deep perceptual
audio metric that correlates well with human subjective ratings across tasks.
The approach relies on multi-dimensional and self-supervised learning to
augment limited human-labeled data. We show the utility of the learned metric
as an optimization objective for speech synthesis and enhancement, but it
could be applied in many other applications. We would like to extend this
metric to include content similarity as well, in general going beyond acoustic
similarity for applications like music similarity. Though we showed two
applications of the metric, future works could also explore other applications
like audio retrieval and speech recognition.

## References

  * [1] P. Manocha, A. Finkelstein, et al.,  “A differentiable perceptual audio metric learned from just noticeable differences,”  in Interspeech, 2020.
  * [2] A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra,  “Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment,”  in ICASSP, 2001.
  * [3] A. Hines, J. Skoglund, A C. Kokaram, et al.,  “ViSQOL: an objective speech quality model,”  EURASIP Journal on Audio, Speech, and Music Processing, 2015.
  * [4] C-C. Lo, S-W. Fu, W-C. Huang, X. Wang, Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang,  “MOSNet: Deep learning based objective assessment for voice conversion,”  Interspeech, 2019.
  * [5] B. Patton, Y. Agiomyrgiannakis, M. Terry, K. Wilson, R. A. Saurous, and D. Sculley,  “AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech,”  in NIPS Workshop, 2016.
  * [6] S-W. Fu, C-F. Liao, and Y. Tsao,  “Learning with learned loss function: Speech enhancement with quality-net,”  SPS, vol. 27, pp. 26–30, 2019.
  * [7] S-W. Fu, C-F. Liao, Y. Tsao, and S.D. Lin,  “MetricGAN: Generative adversarial networks based black-box metric scores optimization for speech enhancement,”  in ICML, 2019.
  * [8] J. Serrà, J. Pons, et al.,  “SESQA: semi-supervised learning for speech quality assessment,”  arXiv:2010.00368, 2020.
  * [9] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brebisson, Y. Bengio, and A. Courville,  “Melgan: Generative adversarial networks for conditional waveform synthesis,”  in NeurIPS, 2019.
  * [10] A. Defossez, G. Synnaeve, and Y. Adi,  “Real time speech enhancement in the waveform domain,”  in Interspeech, 2020.
  * [11] S V. Steenkiste, F. Locatello, J. Schmidhuber, and O. Bachem,  “Are disentangled representations helpful for abstract visual reasoning?,”  in NeurIPS, 2019.
  * [12] I. Higgins, D. Amos, D. Pfau, S. Racaniere, L. Matthey, D. Rezende, and A. Lerchner,  “Towards a definition of disentangled representations,”  arXiv:1812.02230, 2018.
  * [13] A. Hines, J. Skoglund, A. Kokaram, and N. Harte,  “Robustness of speech quality metrics: Comparing ViSQOL, PESQ and POLQA,”  in ICASSP, 2013.
  * [14] H. Zhang, X. Zhang, and G. Gao,  “Training supervised speech separation system to improve STOI and PESQ directly,”  in ICASSP, 2018.
  * [15] A. Conneau, A. Baevski, et al.,  “Unsupervised cross-lingual representation learning for speech recognition,”  arXiv:2006.13979, 2020.
  * [16] P. Manocha, R. Badlani, A. Kumar, A. Shah, B. Elizalde, and B. Raj,  “Content-based representations of audio using siamese neural networks,”  in ICASSP, 2018.
  * [17] Y. Bengio, A. Courville, and P. Vincent,  “Representation learning: A review and new perspectives,”  PAML, 2013.
  * [18] J. Lee, N J. Bryan, J. Salamon, Z. Jin, and J. Nam,  “Disentangled multidimensional metric learning for music similarity,”  in ICASSP, 2020.
  * [19] Y N. Hung, Y A. Chen, and Y H. Yang,  “Learning disentangled representations for timber and pitch in music audio,”  in arXiv:1811.03271, 2018.
  * [20] J. Chou, C. Yeh, H. Lee, and L. Lee,  “Multi-target voice conversion without parallel data by adversarially learning disentangled audio representations,”  in arXiv:1804.02812, 2018.
  * [21] Y.C. Chen, S.F. Huang, H.Y. Lee, Y.H. Wang, and C.H. Shen,  “Audio word2vec: Sequence-to-sequence autoencoding for unsupervised learning of audio segmentation and representation,”  ASLP, 2019.
  * [22] A. Oord, Y. Li, and O. Vinyals,  “Representation learning with contrastive predictive coding,”  in arXiv:1807.03748, 2018.
  * [23] P. Chi, P. Chung, T. Wu, et al.,  “Audio Albert: A lite BERT for self-supervised learning of audio representation,”  in arXiv:2005.08575, 2020.
  * [24] S. Schneider, A. Baevski, R. Collobert, and M. Auli,  “wav2vec: Unsupervised pre-training for speech recognition,”  in Interspeech, 2019.
  * [25] F. Kreuk, J. Keshet, and Y. Adi,  “Self-supervised contrastive learning for unsupervised phoneme segmentation,”  in Interspeech, 2020.
  * [26] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton,  “A simple framework for contrastive learning of visual representations,”  in ICML, 2020.
  * [27] Z. Jin, G J. Mysore, S. Diverdi, J. Lu, and A. Finkelstein,  “Voco: Text-based insertion and replacement in audio narration,”  TOG, 2017.
  * [28] Z. Jin, A. Finkelstein, G J. Mysore, and J. Lu,  “FFTNet: A real-time speaker-dependent neural vocoder,”  in ICASSP, 2018.
  * [29] B. Feng, Z. Jin, J. Su, and A. Finkelstein,  “Learning bandwidth expansion using perceptually-motivated loss,”  in ICASSP, 2019.
  * [30] D P. Kingma and J. Ba,  “Adam: A method for stochastic optimization,”  in ICLR, 2015.
  * [31] J. Su, A. Finkelstein, and Z. Jin,  “Perceptually-motivated environment-specific speech enhancement,”  in ICASSP, 2019.
  * [32] J. Su, Z. Jin, et al.,  “HiFi-GAN: High-fidelity denoising and dereverberation,”  Interspeech, 2020.
  * [33] V. Emiya, E. Vincent, N. Harlander, et al.,  “Subjective and objective quality assessment of audio source separation,”  ASLR, 2011.
  * [34] J. Lorenzo-Trueba, J. Yamagishi, T. Toda, D. Saito, F. Villavicencio, T. Kinnunen, and Z. Ling,  “The voice conversion challenge 2018,”  arXiv:1804.04262, 2018.
  * [35] Y. Hu and P. C. Loizou,  “Subjective comparison and evaluation of speech enhancement algorithms,”  Speech communication, 2007.
  * [36] K. Ito et al.,  “The LJ speech dataset,” 2017.
  * [37] C. Valentini-Botinhao et al.,  “Noisy speech database for training speech enhancement algorithms and TTS models,”  2017\.
  * [38] G J. Mysore,  “Can we automatically transform speech recorded on common consumer devices in real-world environments into professional production quality speech?—a dataset, insights, and challenges,”  SPS, 2014.
  * [39] C K. Reddy, E. Beyrami, H. Dubey, V. Gopal, et al.,  “The interspeech 2020 deep noise suppression challenge: Datasets, subjective speech quality and testing framework,”  Interspeech, 2020.