# Limitations to Electrical Probing of Spontaneous Polarization in
Ferroelectric-Dielectric Heterostructures

Mattia Segatto Riccardo Fontanini Francesco Driussi Daniel Lizzit David Esseni

###### Abstract

An accurate estimate of the ferroelectric polarization in ferroelectric-
dielectric stacks is important from a materials science perspective, and it is
also crucial for the development of ferroelectric based electron devices. This
paper revisits the theory and application of the PUND technique in Metal-
Ferroelectric-Dielectric-Metal (MFDM) structures by using analytical
derivations and numerical simulations. In an MFDM structure the results of the
PUND technique may largely differ from the polarization actually switched in
the stack, which in turn is different from the remnant polarization of the
underlying ferroelectric. The main hindrances that prevent PUND measurements
from providing a good estimate of the polarization switching in MFDM stacks
are thus discussed. The inspection of the involved physical quantities, not
always accessible in experiments, provides a useful insight about the main
sources of the errors in the PUND technique, and clarifies the delicate
interplay between the depolarization field and the charge injection and
trapping in MFDM stacks with a thin dielectric layer.

###### Index Terms:

HZO, Ferroelectric, MFDM, Dielectric, PUND, Depolarization

© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  I Introduction

Thanks to the discovery of a robust ferroelectricity in hafnium oxide thin
films,${}^{\text{\cite[cite]{[\@@bibref{}{Boscke_APL2011}{}{}]}}}$ several
intriguing applications of ferroelectricity in CMOS electron devices have been
proposed and are being presently scrutinized. Device concepts include
nanoscale CMOS FETs exploitating the effective negative capacitance to improve
the
subthresholdswing,${}^{\text{\cite[cite]{[\@@bibref{}{Salahuddin_NL2008,Jain_TED2014,
Khan_TED2016, Rollo_EDL2018, Rollo_EDL2018_trap, Pentapati_TED2020}{}{}]}}}$
as well as Ferroelectric Tunnelling Junctions
(FTJs),${}^{\text{\cite[cite]{[\@@bibref{}{Max_JEDS2019,Fontanini_ESSDERC2021}{}{}]}}}$
and ferroelectric
FETs,${}^{\text{\cite[cite]{[\@@bibref{}{Mulaosmanovic_EDL2020,Lizzit_ESSDERC2021}{}{}]}}}$
which may be used as non–volatile memories or as memristors for neuromorphic
computing
applications.${}^{\text{\cite[cite]{[\@@bibref{}{Slesazeck_Nanotechnology2019}{}{}]}}}$

In most of the above material systems and electron devices, one of the
materials adjacent to the ferroelectric is a dielectric (see also Figure 1),
and the working principle of the devices relies on the influence that the
ferroelectric polarization, $P$, exerts on the band bending inside the device.
Quite understandably, a dependable determination of $P$ is of primary
importance in ferroelectric materials and ferroelectric based electron
devices. To this purpose the Positive-Up-Negative-Down (PUND) measurement
technique was originally conceived for Metal-Ferroelectric-Metal (MFM)
structures${}^{\text{\cite[cite]{[\@@bibref{}{Rabe_book,
Fukunaga_JPSJ2008}{}{}]}}}$ (see also PUND waveforms in Figure 2), and it is
still routinely used also in Metal-Ferroelectric-Dielectric-Metal (MFDM)
device structures.${}^{\text{\cite[cite]{[\@@bibref{}{Lomenzo_AVS2014,
Mikheev_ACS2019, Ryu_Nature2019, Deshpande_SD2021}{}{}]}}}$ The main goal of
the PUND technique for an MFM structure is an accurate determination of the
difference, 2$P_{r}$, between the polarization at zero ferroelectric field for
the positive and negative polarization state. In an MFM stack with ideal metal
electrodes the field can be zeroed by using a zero external voltage, and the
PUND measurements essentially intend to minimize the contributions to 2$P_{r}$
due to the background ferroelectric polarization and to possible leakage
currents (see also the discussion in Section II). In an MFDM structure,
however, a zero external voltage cannot force a zero ferroelectric field due
to the depolarization field, and the application of the PUND technique to MFDM
devices is not straightforward in several respects. In fact the interpretation
of PUND results in an MFDM stack can lead to artifacts and to a misleading
information about the polarization of the underlying ferroelectric layer. We
here revisit the theory and application of PUND measurements in MFDM
structures, and to this purpose we use both analytical derivations and a
comprehensive modelling framework, that has been previously validated and
calibrated against experiments. Our results show that: a) the determination of
the spontaneous ferroelectric polarization is challenging in MFDM structures
even if the charge injection and trapping in the dielectric stack is
negligible; b) in the presence of a non negligible charge injection and
trapping, the variations of spontaneous polarization and trapped charge are
inextricably entangled, which further complicates the extraction of the
switched polarization; c) the $t_{D}$ dependence of the extracted polarization
can be an artifact due to the ($t_{D}$ dependent) interplay between the
depolarization field and charge trapping. The paper is organized as follows.
In Section II we present the theoretical background behind the extraction of
the spontaneous polarization from the terminal currents measured by the PUND
technique. In Section III we provide an overview of our in house modelling
framework and then in Secs. III-B and III-C we report simulation results about
the ability of the PUND technique to determine the switched polarization in
MFDM structures, and offer an insight about the main sources of errors. In
Section IV we propose a few concluding remarks and in the Appendix provide
additional details about some aspects of the modelling framework.

Figure 1: Ferroelectric capacitors and related symbols. a) Three dimensional
sketch of an MFDM capacitor, where $t_{F}$ and $t_{D}$ are the ferroelectric
and dielectric thicknesses, and $V_{T}$ is the external bias. b) Cross section
of an MFDM structure. The polarization, $P$, is taken positive when it points
to the dielectric. $Q_{MF}$, $I_{MF}$, $Q_{MD}$, $I_{MD}$ denote the charges
and currents at the MF and MD electrodes, respectively, whereas $Q_{S}$ is the
charge trapped at the ferroelectric-dielectric interface.  Figure 2: a)
Examples of the $V_{T}$ waveform used in PUND measurements. The 250 $\mu$s
pulse width has been used in all the simulation results reported in Section
III (so $t_{U}$ = $2t_{P}$), if not otherwise stated. A preset pulse at
$V_{T}$ = $-5$ V and for 125 $\mu$s is used to set an initial negative
polarization state; b) Charge waveform sketch during the PUND simulation. Some
key points are defined in order to simplify the notation of the paper.

##  II Charge and current at the electrodes in the MFDM structure

Let us consider the MFDM structure sketched in Figure 1(a), where
$\bf{r}$$=$$(x,y)$ and $z$ are the coordinate in the plane of the
ferroelectric-dielectric interface and in the direction normal to the
interface, while $Q_{MF}$, $Q_{MD}$ are the charges per unit area respectively
at the MF and MD electrodes, respectively (see Figure 1). Assuming a perfect
screening in the metal electrodes, the electrostatic problem is linear and
$Q_{MF}$, $Q_{MD}$ can be written by using appropriate Green’s functions for
the charges in the structure. More specifically, for $Q_{MF}$ we have

$Q_{MF}(t)=\dfrac{1}{A}\int\displaylimits_{A}P(\textbf{r},t)\,d\textbf{r}+\dfrac{1}{A}\int\displaylimits_{A}\varepsilon_{0}\varepsilon_{F}\,E_{FT}(\textbf{r},t)d\textbf{r}$
(1)

where $A$ is the device area, $P$ is the ferroelectric spontaneous
polarization, $E_{FT}(\textbf{r},t)$ denotes the $z$ component of the electric
field at the position r of the MF-FE interface (namely at $z$$=$$-t_{F}$).

At any time $t$, the $E_{FT}(\textbf{r},t)$ is determined by the external bias
$V_{T}$ and by the charges in the dielectric stack, that are charges located
at coordinate $\textbf{r}_{0}$ defined as r at $z$ = 0. In this latter
respect, we here define the depolarization field, $E_{DP}(\textbf{r},t)$, at
the MF-FE interface as the field produced by the distribution of the total
charge $[P(\textbf{r}_{0},t)$ \+ $Q_{S}(\textbf{r}_{0},t)]$ at the FE-DE
interface, where $Q_{S}(\textbf{r}_{0},t)$ is an interface charge due to fixed
Coulomb centers or to interface traps. We also similarly introduce
$E_{\rho}(\textbf{r},t)$ as the field produced by the remaining charge
densities $\rho(\textbf{r}_{0},z_{0},t)$ in the dielectric stack at $z_{0}\neq
0$. The $E_{FT}(\textbf{r},t)$ can thus be written as

$\varepsilon_{0}\varepsilon_{F}\,E_{FT}(\textbf{r},t)=C_{S}\,V_{T}+\varepsilon_{0}\varepsilon_{F}[\,E_{DP}(\textbf{r},t)+E_{\rho}(\textbf{r},t)\,]$
(2)

where $C_{S}$$=$$(1/C_{D}+1/C_{F})^{-1}$, with
$C_{D}$$=$$\varepsilon_{0}\varepsilon_{D}/t_{D}$,
$C_{F}$$=$$\varepsilon_{0}\varepsilon_{F}/t_{F}$ and $t_{D}$,
$\varepsilon_{D}$ being the thickness and relative permittivity of the
dielectric, and $t_{F}$, $\varepsilon_{F}$ being the thickness and background
permittivity of the ferroelectric (see Figure 1(b)). By substituting Equation
2 in Equation 1 we obtain

$\begin{split}Q_{MF}(t)=&C_{S}\,V_{T}(t)+P_{AV}(t)+\\\
&+\varepsilon_{0}\varepsilon_{F}\,E_{DP,AV}(t)+\varepsilon_{0}\varepsilon_{F}\,E_{\rho,AV}(t)\end{split}$
(3)

where $P_{AV}$, $E_{DP,AV}$ and $E_{\rho,AV}$ denote respectively the average
polarization and average fields at the MF-FE interface. From A of the Appendix
it can be inferred that the average $E_{DP}$ and $E_{\rho}$ can be written as:

$\varepsilon_{0}\varepsilon_{F}\,E_{DP,AV}=\dfrac{1}{A}\int\displaylimits_{A}[P(\textbf{r}_{0})+Q_{S}(\textbf{r}_{0})]\,G_{MF}(\textbf{r}_{0},z_{0})\,d\textbf{r}_{0}\vspace{-0.2truecm}$
(4a)
$\varepsilon_{0}\varepsilon_{F}\,E_{\rho,AV}=\dfrac{1}{A}\int\displaylimits_{A}\int\displaylimits_{-t_{F}}^{t_{D}}\rho(\textbf{r}_{0},z_{0})\,G_{MF}(\textbf{r}_{0},z_{0})\,dz_{0}\,d\textbf{r}_{0}$
(4b)

where $G_{MF}(\textbf{r}_{0},z_{0})$ is the Green’s function defined as

$G_{MF}(\textbf{r}_{0},z_{0})=\dfrac{\varepsilon_{0}\varepsilon_{F}}{e}\int\displaylimits_{A}E_{FT}\left[\textbf{r}_{0},z_{0}\right](\textbf{r})\,d\textbf{r}\,\,.$
(5)

with $E_{FT}\left[\textbf{r}_{0},z_{0}\right](\textbf{r})$ being the field
$E_{FT}(\textbf{r})$ produced by a point charge $e$ located at
$(\textbf{r}_{0},z_{0})$. In B of Appendix we also demonstrate that, under
realistic assumptions, the $G_{MF}(\textbf{r}_{0},z_{0})$ for the MFDM
structure in Figure 1(b) is independent of $\textbf{r}_{0}$ and it can be
evaluated analytically. For a charge located at the FE-DE interface, for
example, we have $G_{MF}(\textbf{r}_{0},0)$$\simeq$$-(C_{F}/C_{0})$, which
allows to rewrite Equation 4a as

$\varepsilon_{0}\varepsilon_{F}\,E_{DP,AV}\simeq-\dfrac{C_{F}}{C_{0}}(P_{AV}+Q_{S,AV})$
(6)

At any $z_{0}\neq 0$ the $G_{MF}(\textbf{r}_{0},z_{0})$ can be similarly
expressed with a $z_{0}$ dependent capacitance ratio. We now recall that PUND
measurements are based on the integral of the transient current at the
electrodes, hence by definition the experiments can probe only the variations
of the polarization and charges in the device stack. Here below the discussion
is carried out in terms of the current $I_{MF}$ at the MF terminal; in B of
Appendix we report the corresponding expression for $I_{MD}$. In the presence
of a trapping distributed throughout the device, it is thus difficult to
express the influence of $\rho(\textbf{r}_{0},z_{0},t)$ on $I_{MF}$, because
Equation 4b shows that the information about the distribution along $z_{0}$ is
required. Consequently, hereafter we simplify the picture and assume that the
time derivative of the charge trapped in the dielectric stack is dominated by
the $(\partial Q_{S}$$/$$\partial t)$ term due to traps at the FE-DE
interface, which implies
$\partial E_{\rho,AV}/\partial t$$\ll$$\partial E_{DP,AV}/\partial t$. We also
assume that $Q_{S}(t)$ can change only through the terminal currents
$I_{QS,MF}$, $I_{QS,MD}$ shown in Figure 3, and we let $I_{lkg}$ denote a
possible leakage current, not contributing to trapping. The current at the MF
electrode can thus be written as

Figure 3: a) Sketch of the sheet charges in the MFDM structure of Figure 1(b),
where $Q_{MF}$, $Q_{MD}$ denote the overall charges at the electrodes. The
picture illustrates an example corresponding to a negative polarization $P$
and a positive interface charge $Q_{S}$. $I_{QS,MF}$ and $I_{QS,MD}$ denote
the currents due to trapping and detrapping at the FE-DE interface, and
$I_{lkg}$ is a possible leakage current through the whole structure. $I_{MF}$,
$I_{MD}$ denote the overall currents at the electrodes (see Equation 7), that
are used in the PUND characterization technique. b) Sketch of the band diagram
in an MFDM stack, where $\chi_{D}$, $\chi_{F}$ are the electron affinity of
the dielectric and ferroelectric material, while $\Phi_{MD}$, $\Phi_{MF}$ are
the workfunctions of the MD and MF electrodes (that are equal in the
simulations of this work, Table I). The energy position of acceptor and donor
type traps is also depicted. The tunnelling coefficients $T_{MD}$ and $T_{MF}$
depend on the energy, $\sigma_{\text{E}}$, and geometric, $\sigma_{\text{T}}$,
cross sections of the traps, as well as on the traps energy $E_{T}$.

$\begin{split}I_{MF}=&\dfrac{\partial Q_{MF}}{\partial
t}+I_{QS,MF}+I_{lkg}=C_{S}\dfrac{\partial V_{T}}{\partial t}+\dfrac{\partial
P_{AV}}{\partial t}\\\ &+\varepsilon_{0}\varepsilon_{F}\,\dfrac{\partial
E_{DP,AV}}{\partial t}+I_{QS,MF}+I_{lkg}=\\\ =&C_{S}\dfrac{\partial
V_{T}}{\partial t}+\dfrac{C_{D}}{C_{0}}\dfrac{\partial P_{AV}}{\partial
t}-\dfrac{C_{F}}{C_{0}}\dfrac{\partial Q_{S,AV}}{\partial
t}+I_{QS,MF}+I_{lkg}\end{split}$ (7)

where $Q_{MF}$ has been expressed via Equation 3 assuming $\partial
E_{\rho,AV}/\partial t$$\ll$$\partial E_{DP,AV}/\partial t$, and in the last
equality we have used Equation 6.
If we now consider the Positive (P) pulse of a PUND experiment starting at $t$
= 0 s with $V_{T}(0)$ = 0 V (see the waveform in Figure 2(a)), we can evaluate
the charge $Q_{P}(t)$ (with $0\leq t\leq t_{P}$) by integrating the expression
for $I_{MF}$ in Equation 7 and obtain

$\begin{split}Q_{P}(t)=&\int_{0}^{t}\,I_{MF}(t^{\prime})\,dt^{\prime}\approx\\\
&\approx
C_{S}V_{T}(t)+P_{AV}(t)+\varepsilon_{0}\varepsilon_{F}E_{DP,AV}(t)+\\\
&+Q_{QS,MF}(t)+Q_{lkg}(t)\end{split}$ (8)

where $Q_{QS,MF}$ and $Q_{lkg}$ are the integral of $I_{QS,MF}$ and $I_{lkg}$,
respectively. Moreover it is understood that $P_{AV}$, $E_{DP,AV}$ and
$Q_{QS,MF}$ in Equation 8 denote the variations from the corresponding values
at $t$$=$0 or, equivalently, that Equation 8 conventionally assumes
$P_{AV}$$=$$E_{DP,AV}$$=$$Q_{S,MF}$$=$$Q_{MF}$$=$0 at $t$$=$0\. The charges
$Q_{U}(t)$, $Q_{N}(t)$, $Q_{D}(t)$ during respectively the Up (U), Negative
(N) and Down (D) pulses of the PUND technique have expressions equivalent to
Equation 8.
The first and last term at the right hand side of Equation 8 are the
contributions due to, respectively, the linear polarization of the dielectrics
and the leakage. Even in an MFM structure these contributions complicate the
extraction of the remnant polarization $2P_{r}$$=$$P_{AV}(t=t_{P})$, and PUND
measurements address this issue by subtracting from $Q_{P}$ the charge $Q_{U}$
during the U pulse at the same external bias $V_{T}$. In our theoretical
framework such an approach results in the charge
$Q_{PU}$$=$$(Q_{P}$$-$$Q_{U})$, that can be written as

$\begin{split}Q_{PU}\approx&P_{AV}^{(P)}-P_{AV}^{(U)}+\varepsilon_{0}\varepsilon_{F}\left(E_{DP,AV}^{(P)}-E_{DP,AV}^{(U)}\right)+\\\
&+Q_{QS,MF}^{(P)}-Q_{QS,MF}^{(U)}+Q_{lkg}^{(P)}-Q_{lkg}^{(U)}\end{split}$ (9)

where the apices $(P)$, $(U)$ identify the P and U pulse and all charges are
evaluated at times corresponding to the same $V_{T}$ value during either a
rising or a falling $V_{T}$ ramp. We recall that, as already mentioned about
Equation8, the $P_{AV}$, $E_{DP,AV}$ and $Q_{QS,MF}$ in Equation 9 denote the
variations from the corresponding values at the beginning of either the P or
the U pulse.

The second and third term in the right hand side of Equation 9 are due
respectively to the depolarization field and the current at the MF electrode
contributing to trapping at the FE-DE interface. In an MFM structure both
these terms are negligible and, moreover, it is typically assumed that the
polarization can be stabilized after the P pulse, so that $P_{AV}^{(U)}$ is
much smaller than $P_{AV}^{(P)}$. Furthermore, it is also usually assumed that
the leakage affects the measurements to a similar extent during the P and U
pulse, leading to
$Q_{lkg}^{(U)}$$\approx$$Q_{lkg}^{(P)}$.${}^{\text{\cite[cite]{[\@@bibref{}{Schenk_TUFF2015}{}{}]}}}$
Under these circumstances Equation 9 shows that the $Q_{PU}$ in an MFM stack
can be interpreted as the $P_{AV}^{(P)}$ that we wish to determine. In an MFDM
structure, instead, the terms in Equation 9 due to the depolarization field
can be comparable to $P_{AV}^{(P)}$, and the term
$(Q_{QS,MF}^{(P)}-Q_{QS,MF}^{(U)})$ may also give a sizeable contribution to
$Q_{PU}$. Hence in an MFDM stack the interpretation of $Q_{PU}$ and the
determination of $P_{AV}^{(P)}$ appear much more delicate that in the MFM
counterpart. This is systematically investigated in SectionIII by using
numerical simulations.

##  III Numerical simulations and discussion

The goal of the PUND measurements is to accurately determine the spontaneous
polarization switched by the external bias, which is very challenging in an
MFDM structure due to the depolarization field and the possible charge
trapping. In this section, we use numerical simulations to investigate the
possible errors and artifacts produced by PUND measurements in MFDM
structures, and to provide some useful physical insights.

###  III-A Modelling framework and validation

Our in house developed simulation framework comprises models for the
ferroelectric dynamics, a dynamic equation for the traps at the FE-DE
interface and a description of a tunnelling injection from the MF and MD
electrodes to the traps. The dynamics of the ferroelectric domains is
described by a formulation of the multi-domain Landau-Ginzburg-Devonshire
(LGD) model more thoroughly discussed in
${}^{\text{\cite[cite]{[\@@bibref{}{Rollo_Nanoscale2020}{}{}]}}}$

$\begin{split}t_{F}\,\rho\,\dfrac{\partial P_{i}}{\partial
t}=&-\left(2\alpha_{i}\,P_{i}+4\beta_{i}\,P_{i}^{3}+6\gamma_{i}\,P_{i}^{5}\right)t_{F}+\\\
&-\dfrac{t_{F}\,k}{d\,w}\sum_{n}\left(P_{i}-P_{n}\right)+\\\
&-\dfrac{1}{2}\sum_{j=1}^{n_{D}}\left(\dfrac{1}{C_{i,j}}+\dfrac{1}{C_{j,i}}\right)\,P_{j}+\dfrac{C_{D}}{C_{0}}V_{T}\end{split}$
(10)

where $\alpha$, $\beta$, $\gamma$ are the anisotropy constants, $\rho$ denotes
the resistivity that sets a time scale $t_{\rho}$$=$$\rho/(2|\alpha|)$ for the
ferroelectric switching. Moreover, the parameters $1/C_{i,j}$ describe the
depolarization energy and depolarization field in the MFDM structure, while
$k$ and $w$ are the coupling constant and the domain wall width involved in
the formulation of the domain wall energy. The mean values for $\alpha$,
$\beta$, $\gamma$ used in simulations are reported in Table I and are referred
to Hafnium-Zirconium-Oxide (HZO) ferroelectric which is the current state-of-
the art ferrolectric used in FTJs. In all simulations the domain wall coupling
$k$ was set to zero, by following recent first principles calculations for
HfO2.${}^{\text{\cite[cite]{[\@@bibref{}{Lee_Science_2020}{}{}]}}}$ Moreover,
we assumed a resistivity $\rho$$=$115 $\Omega$m, which is consistent with
recently reported values for Hafnium-Zirconium (HZO) based
capacitors,${}^{\text{\cite[cite]{[\@@bibref{}{Kobayashi_IEDM2016,
Kim_IEDM2020}{}{}]}}}$ and results in a time scale for the ferroelectric
dynamics $t_{\rho}$$\approx$ $119.8$ ns. All simulations include a $n_{D}$$=$
1024 and a domain size d=5nm; we verified that results are insensitive to a
further increase of $n_{D}$. The modelling for the ferroelectric dynamics has
been extensively compared to transient negative capacitance measurements,
demonstrating a good agreement with experiments in asymmetric MFDM dielectric
stacks,${}^{\text{\cite[cite]{[\@@bibref{}{Rollo_Nanoscale2020,Esseni_Nanoscale2021}{}{}]}}}$
and also in symmetric MFDFM
structures.${}^{\text{\cite[cite]{[\@@bibref{}{Hoffmann_AdvFunctionalMaterials_2021}{}{}]}}}$
The charge trapping model follows a first-order dynamic equation for the
occupation $f_{T}$ of either acceptor or donor type traps at the FE-DE
interface. By denoting with $c_{MD0}$, $c_{MF0}$ the capture rate from the
metal MD and MF electrodes the equation governing $f_{T}$ can be written as

$\dfrac{\partial f_{T}}{\partial
t}=c_{MD0}\,[\,f_{0,MD}-f_{T}\,]+c_{MF0}\,[\,f_{0,MF}-f_{T}\,]\,\,\vspace{-0.08truecm}$
(11)

where $f_{0,M}(E_{T})$$=$$1/[1+\text{exp}((E_{T}-E_{f,M})/(K_{B}T))]$ is the
Fermi occupation function in the metal electrodes, with
$E_{f,MF}$=$E_{f,MD}-qV_{B}$. In the derivation of Equation 11 we used a
detailed balance condition, ensuring that the steady state $f_{T}$ value at
the equilibrium (i.e. for $V_{T}$=$0$ V) is given by the Fermi function. In
this work, the capture rates were attributed to tunnelling from and to the
electrodes, and the tunnelling transmission described according to a WKB
approximation that involves the tunnelling effective mass in the two
dielectrics $m_{D}$, $m_{F}$, and the area and energy cross sections
$\sigma_{T}$ [m2], $\sigma_{E}$ [eV] (see also Figure 3). More details about
the trapping and tunnelling models may be found
in${}^{\text{\cite[cite]{[\@@bibref{}{Fontanini_ESSDERC2021,
Fontanini_JEDS2021}{}{}]}}}$ reporting also a good agreement with the
polarization versus voltage curves in FTJs with an MFDM structure and a thin
dielectric layers. The values of $m_{D}$, $m_{F}$, $\sigma_{T}$, $\sigma_{E}$
used in this work are reported in Table I.
From the occupations $f_{T}$ one can readily calculate the charges $Q_{acc}$
and $Q_{don}$ respectively in acceptor and donor type traps, and finally the
overall interface trapped charge $Q_{S}$$=$($Q_{acc}$$+$$Q_{don})$. The
knowledge of the time dependent polarization and trapped charge, in turn,
allows one to numerically calculate all the quantities discussed in Section
II, such as $I_{MF}$, $P_{AV}$, $Q_{S,AV}$, and also $I_{MF,QS}$, $Q_{MF,QS}$.

Table I: Material parameters used in simulations for the Hf0.5Zr0.5O2$-$Al2O3 MFDM system. Here $\alpha$, $\beta$ and $\gamma$ are the mean values of the anisotropic constants, and calculations include domain to domain variations of the $\alpha_{i}$, $\beta_{i}$, $\gamma_{i}$ parameters (with $i$=$1,\,2,\ldots,n_{D}$) corresponding to a ratio $\sigma_{EC}=10$% between the standard deviation and the mean value of the coercive field $E_{C}$. m${}_{\text{D}}$ and m${}_{\text{F}}$ are the effective tunnelling masses for respectively Al2O3 and HZO, while $\sigma_{\text{E}}$, $\sigma_{\text{T}}$ denote respectively the energy and geometric cross section of the traps, that are used in the tunnelling model. The maximum energy values for the traps are $0.6$ and $1.3$ eV below the conduction band minimum at the FE-DE interface for acceptor and donor traps, respectively. Both traps type extend in energy for $2$ eV below their maximum. The electron affinity was set to $\chi_{D}$$=$ 1.4 eV for Al2O3${}^{\text{\cite[cite]{[\@@bibref{}{Arya_JTAC2015}{}{}]}}}$ and to $\chi_{F}$$=$ 2.4 eV for HZO,${}^{\text{\cite[cite]{[\@@bibref{}{Zheng_JPC2005}{}{}]}}}$ while the workfunction for both TiN metal electrodes was taken as $\Phi_{M}$$=$ 4.5 eV.${}^{\text{\cite[cite]{[\@@bibref{}{Vitale_TED2010}{}{}]}}}$ $\alpha$ [m/F] |  $\beta$ [m5/C2/F] |  $\gamma$ [m9/C4/F] | $\epsilon_{F}$ | $\epsilon_{D}$ | m${}_{\text{F}}$ [m0] | m${}_{\text{D}}$ [m0] |  $\sigma_{\text{T}}$ [m2] |  $\sigma_{\text{E}}$ [eV]
---|---|---|---|---|---|---|---|---
$-4.8\cdot 10^{8}$ | $\quad 1.46\cdot 10^{9}$ | $\quad 3.14\cdot 10^{10}$ | $34$ | $10$ | $0.4$ | $0.18$ | $1\cdot 10^{-19}$ | $7\cdot 10^{-3}$

###  III-B Simulations results

We used numerical simulations to emulate PUND measurements with a 1 kHz
waveform and a 5 V peak voltage in an MFDM structure with different dielectric
thicknesses $t_{D}$ and trap densities $N_{acc}$ and $N_{don}$. The
ferroelectric HZO layer is 10 nm thick in all simulations.

We do not account for a possible leakage current flowing directly from MD to
MF. Here we did not attempt to model the leakage current because we think that
leakage is strongly technology dependent, frequently governed by Poole-Frenkel
and hopping mechanisms in the the HZO and, as such, difficult to describe in
simulations.${}^{\text{\cite[cite]{[\@@bibref{}{Schroeder_JAP2015}{}{}]}}}$
Moreover, while it is understood that a failure of the condition
$Q_{lkg}^{(U)}$$\approx$$Q_{lkg}^{(P)}$ can induce artifacts in PUND
experiments even in an MFM stack, this issue goes beyond the scope of the
present paper, that is focused on the influence that the depolarization field
and charge trapping have on the results of PUND measurements in MFDM
structures.

In Figure 4 we report simulation results for $t_{D}=$ 1.5 nm and for different
trap densities. The $Q_{PUND}$ is here defined as either
$Q_{PU}$$=$$(Q_{P}-Q_{U})$ or $Q_{ND}$$=$$(Q_{N}-Q_{D})$, respectively for the
positive and negative $V_{T}$ values. The sign of the interface charge
$Q_{S,AV}$ is typically opposite to the sign of the polarization, and Figure 4
reports $-Q_{S,AV}$, which together with $P_{AV}$ determines $E_{DP,AV}$
according to Equation 6. All charges in Figure 4 are referred to the
corresponding value at the beginning of the P pulse, namely at $t$$=$0 s and
$V_{T}$$=$0 V in Figure 2 (see also the discussion about Equations 8 and 9).
The $Q_{PUND}$ in Figure 4(a) shows a hysteresis loop that is much more tilted
and stretched than in the corresponding MFM curves (filled triangles). The
features for an MFDM are similar to those experimentally observed in the
$P$-$V$ curves for an HZO capacitor serially connected to a discrete ceramic
capacitor ensuring a negligible charge
injection,${}^{\text{\cite[cite]{[\@@bibref{}{Park_Nanoscale2021}{}{}]}}}$ or
to measurements in MFDM structures with thicker Al2O3
layers.${}^{\text{\cite[cite]{[\@@bibref{}{Hoffmann_AdvFunctionalMaterials_2021}{}{}]}}}$
In fact, the relatively low density of traps in Figure 4(a) results in an
interface charge $Q_{S}$ (green diamonds) that is practically negligible
compared to the ferroelectric polarization (red squares). The lack of any
compensation of the polarization results in a large depolarization field
$E_{DP,AV}$ (see Equation 6), which in turn leads to a vast discrepancy
between $Q_{PUND}$ and $P_{AV}$. In fact, Figure 4(a) shows that for $|V_{T}|$
above about 4 V a complete polarization switching occurs. Nevertheless the
corresponding $Q_{PUND}$ is much smaller than $P_{AV}$, mainly because the
$E_{DP,AV}$ term in Equation 9 subtracts from the $P_{AV}$ term due to the
opposite sign. The results for $Q_{PUND}$ are quite different in Figure 4(b),
because the $Q_{S,AV}$ can now compensate $P_{AV}$ to a large extent, thus
drastically reducing the depolarization field. The hysteresis loop of the
$Q_{PUND}$ curve in Figure 4(b) is qualitatively similar to the experimental
behaviour observed in FTJ structures with a thin tunnel
oxide,${}^{\text{\cite[cite]{[\@@bibref{}{Fontanini_ESSDERC2021,Li_VLSI_2020}{}{}]}}}$
and the discrepancy between $Q_{PUND}$ and $P_{AV}$ is much smaller than in
Figure 4(a).

Figure 4: Simulated charges corresponding to a 1 kHz PUND waveform applied to
an MFDM structure. The $Q_{PUND}$ is either $Q_{PU}$ = $(Q_{P}$ \- $Q_{U})$ or
$Q_{ND}$ = $(Q_{N}$ \- $Q_{D})$, respectively during the P pulse (i.e. for a
positive $V_{T}$) or during the N pulse (i.e. for a negative $V_{T}$). The
average polarization $P_{AV}$ and trapped charge $-Q_{S,AV}$ during the P and
N pulses are also shown. The Al2O3 layer thickness is $t_{D}$$=$1.5 nm. a)
Results for acceptor and donor type trap densities $N_{acc}$ = $N_{don}$ =
$0.5$ $\times$ $10^{13}$ [cm-2eV-1]; b) Results for
$N_{acc}$$=$$N_{don}$$=$$4\times 10^{13}$ [cm-2eV-1].

Figure 5 illustrates the same analysis as in Figure 4, but for a larger
dielectric thickness $t_{D}$ = $2.5$ nm. In this case the results of both
small and large interface traps densities have a qualitative behaviour similar
to Figure 4(a), namely the $Q_{S}$ is very small (green diamonds) and the
compensation of the ferroelectric polarization is minimal. For both traps
densities, the depolarization field results in a $Q_{PUND}$ much smaller than
$P_{AV}$.

Figure 5: Same simulations for a PUND waveform in an MFDM structure as in
Figure 4, but for an oxide thickness $t_{D}$ of 2.5 nm. a) Results for
acceptor and donor type trap densities $N_{acc}$$=$$N_{don}$$=$$0.5\times
10^{13}$ [cm-2eV-1]; b) Results for $N_{acc}$$=$$N_{don}$$=$$4\times 10^{13}$
[cm-2eV-1].

The different behaviour in Figure 5(b) compared to Figure 4(b) is due to the
fact that, according to the tunneling effective masses and traps cross-
sections reported in Table I, the trapping and de-trapping dynamics cannot
follow the 1 kHz $V_{T}$ waveform for $t_{D}$$=$2.5 nm or larger. In fact,
because the HZO layer is 10 nm thick, in the simulations of this work the
trapping dynamics is essentially set by the tunnelling through the much
thinner dielectric layer. The lack of $Q_{S}$ modulation in Figure 5(b) is
thus a dynamic effect. This emphasizes that the trapping induced compensation
of the ferroelectric polarization requires both a large enough trap density at
the FE-DE interface, and a trapping dynamics fast enough to respond to the
$V_{T}$ waveform. This latter observation has been crucial in transient
negative capacitance experiments, where thick dielectrics and fast bias
waveforms were used to avoid the undesired compensation of the ferroelectric
polarization and to achieve a hysteresis free
behaviour.${}^{\text{\cite[cite]{[\@@bibref{}{Rollo_Nanoscale2020,Hoffmann_Nature2019,Esseni_Nanoscale2021}{}{}]}}}$

While the $t_{D}$ values at which traps can no longer respond to a given
$V_{T}$ waveform depend on the tunnelling model and the corresponding
parameters in Table I, the qualitative trend is expected to be independent of
the modelling details.

Figure 6: $Q_{PU}$ extracted from PUND simulations in MFDM structures for
$V_{T}$ = 0 V, for different dielectric thickness $t_{D}$ and different traps
density $N_{acc}$$=$$N_{don}$ (in units of cm-2eV-1). The remnant polarization
$2P_{r}$ extracted for an MFM structure is also reported for comparison.

###  III-C Discrepancies between $Q_{PUND}$ and $P_{AV}$

The discrepancies between $Q_{PUND}$ and $P_{AV}$ shown in Figures 4 and 5
correspond to errors in the outcome of the PUND method. Hence in this section
we evaluate the relative error
$E_{PU}$$=$$|Q_{PU}-P_{AV}^{(P)}|$$/$$P_{AV}^{(P)}$, where from hereon all the
quantities are evaluated at the end of the P or the U pulse, namely when
$V_{T}$ is zero. Similar definitions apply to the N and D pulses, and the
simulation results are also completely similar (not shown).

Figure 7: Error $|Q_{PU}-P_{AV}^{(P)}|$$/$$P_{AV}^{(P)}$ of PUND measurements
in an MFDM structure for different thicknesses $t_{D}$ and different trap
densities $N_{acc}$ = $N_{don}$. The error is calculated for the P and U
pulses. a) Error evaluated at the end of the P pulse (see inset); b) Error
evaluated at the peak of the P pulse pulse.

Figure 6 shows the $Q_{PU}$ of the MFDM structures. It can be seen that a
combination of a large $t_{D}$ and low concentrations of traps lead to low
simulated $Q_{PU}$ values, because the corresponding
$\varepsilon_{0}\varepsilon_{F}E_{DP,AV}^{(P)}$ term is comparable to
$P_{AV}^{(P)}$ (as later shown in Figure 8). Figure 7 reports the evaluation
of the error $E_{PU}$ for different dielectric thicknesses and trap densities.
As it can be seen, the error tends to decrease for increasing trap densities,
due to the corresponding reduction of the depolarization field $E_{DP}$. For
the same reason the error increases for thicker dielectrics. This latter
behavior results in a $t_{D}$ dependence of the $P_{AV}$ estimated by the PUND
method, which is an artifact of the method when it is applied to an MFDM
structure.
For tD = 2.5 nm the error is fairly insensitive to the trap density, because
the $Q_{S}$ in the traps cannot respond to the $V_{T}$ waveform according to
our tunnelling model. To gain an insight about the main causes of the errors
shown in Figure 7, we first rewrite Equation 9 as

$\begin{split}Q_{PU}\approx&\dfrac{C_{D}}{C_{0}}\left(P_{AV}^{(P)}-P_{AV}^{(U)}\right)-\dfrac{C_{F}}{C_{0}}\left(Q_{S,AV}^{(P)}-Q_{S,AV}^{(U)}\right)+\\\
&+Q_{QS,MF}^{(P)}-Q_{QS,MF}^{(U)}\end{split}$ (12)

where we have used Equation 6 to express the depolarization field
$E_{DP,AV}^{(P)}$; here we have omitted the leakage part because the leakage
current is not included in our simulations, and all the quantities in Equation
12 are evaluated at the end of the P or the U pulse. Then we report in Figure
8(a) the quantities in the right hand side of Equations 6, 9 and 12, for a
dielectric thickness $t_{D}$ = 1.5 nm and evaluated in the same condition used
to evaluate the PUND error in Figure 7 (i.e. $V_{T}=0$ V).
Figure 8(a) conveys several important messages. The terms $Q_{QS,MF}^{(P)}$,
$Q_{QS,MF}^{(U)}$ (diamonds) related to the trapping and de-trapping current
at the MF electrode are very small even for large trap densities, hence they
do not appreciably influence $Q_{PU}$ in Equations 9 and 12. This is not
surprising because, in the MFDM structures at study, traps exchange electrons
primarily with the MD electrode as the dielectric is much thinner than the
ferroelectric layer.
Moreover, at large trap densities the $Q_{S,AV}^{(P)}$ (filled circles) in the
P pulse is comparable to $P_{AV}^{(P)}$ (filled squares), whereas
$Q_{S,AV}^{(P)}$ becomes negligible at low trap densities. The
$Q_{S,AV}^{(U)}$ in the $U$ pulse, instead, is always negligible compared to
$P_{AV}^{(P)}$. This is because, for the case at study in Figure 8, the band
bending in the dielectric at the end of the P pulse is such that the energy
levels of both acceptor and donor traps fall below the Fermi level of the MD
contact (see Figure 3(b)). Hence, essentially all traps have been filled at
the end of the P pulse, and their occupation is not appreciably changed during
the following U pulse. Figure 8(a) shows that also $P_{AV}^{(U)}$ in the $U$
pulse is much smaller than $P_{AV}^{(P)}$. This is because the $P_{AV}$ in the
P and U pulse is a measure of the non reversible switching, whereas most of
the switching in the $U$ pulse is reversible in nature because it is the
switching of those domains that have back switched after the $P$ pulse.
As mentioned above, Figure 8(a) shows that at low trap densities we have
$|Q_{S,AV}^{(P)}|$$\ll$$P_{AV}^{(P)}$ and Equation 6 suggests that this
results in a
$\varepsilon_{0}\varepsilon_{F}\,E_{DP,AV}^{(P)}$$\simeq$$-(C_{F}/C_{0})P_{AV}^{(P)}$,
as it is confirmed by Figure 8(b). These are the conditions that in Figure 7
correspond to the maximum discrepancy between $Q_{PU}$ and $P_{AV}^{(P)}$.
Equation 12 shows that for $|Q_{S,AV}^{(P)}|$$\ll$$P_{AV}^{(P)}$ the extracted
$Q_{PU}$ tends to $(C_{D}/C_{0})P_{AV}^{(P)}$, in fact resulting in a large
underestimate of $P_{AV}^{(P)}$. At large trap densities, instead,
$|Q_{S,AV}^{(P)}|$ becomes comparable to $P_{AV}^{(P)}$ and Equation 6
predicts a drastic reduction of the
$|\varepsilon_{0}\varepsilon_{F}\,E_{DP,AV}^{(P)}/P_{AV}^{(P)}|$ term, which
can be observed in Figure 8(b). The error in Figure 7 is correspondingly
reduced at large trap densities; in fact Equation 9 suggests that $Q_{PU}$
tends to $P_{AV}^{(P)}$.

Figure 8: Charge components contributing to $Q_{PU}$ according to Equation 12
and Equation 9 evaluated for $t_{D}$$=$1.5 nm and in the same conditions as in
Figure 7 (i.e. at the end of the P pulse with $V_{T}$ = 0 V). The sign of
$Q_{S,AV}$ and $\varepsilon_{0}\varepsilon_{F}E_{DP,AV}^{(P)}$ is opposite to
the sign of $P_{AV}$ and the figure displays $-$$Q_{S}$ and
$-\varepsilon_{0}\varepsilon_{F}E_{DP,AV}^{(P)}$. All the quantities shown in
the figure are difference between the values at the end and at the start of
the P pulse.

##  IV Conclusions

We have revisited the theory and application of the PUND technique in MFDM
structures by using analytical derivations and numerical simulations. The
interplay between the depolarization field and charge trapping in an MFDM
stack makes it difficult to obtain from the terminal currents alone an
accurate estimate of the spontaneous polarization switched in the $P$ or in
the $N$ pulse.

The discrepancies between $Q_{PU}$ and $P_{AV}^{(P)}$, for example, were
analyzed for different thicknesses $t_{D}$ of the dielectric layer and
different traps densities at the FE-DE interface, that in turn result in
different trapping induced compensations of the ferroelectric polarization.
Because in simulations one can inspect all the physical quantities at play,
even those that are not usually accessible in experiments, our analysis
allowed us to gain an insight about the main sources of error for the PUND
technique in MFDM structures. Besides the discrepancies between $Q_{PU}$ and
$P_{AV}^{(P)}$ that can be identified as an error of the PUND technique, it
should be understood that neither the $Q_{PU}$ nor the $P_{AV}^{(P)}$ of an
MFDM structure are a good estimate of the $2P_{r}$ of the underlying
ferroelectric. This is because the depolarization field can be large at zero
external bias, so that the MFDM structure at $V_{T}$$=$0 is not at all
representative of the ferroelectric material at zero ferroelectric field. More
precisely the $Q_{PU}$ of the PUND technique tends to underestimate the non-
reversible switched polarization $P_{AV}^{(P)}$, which in turn is an
underestimate of the $2P_{r}$ of the ferroelectric. The differences between
these quantities depend on $t_{D}$ and on the density of traps, which may lead
to artifacts in the characterization of a possible $t_{D}$ dependence of the
properties of the underlying ferroelectric layer. Of course, we acknowledge
that it would be very useful to suggest corrections to the PUND technique or
to propose a novel technique for MFDM structures in order to duly account for
the depolarizing field, and maybe even separate the switched polarization from
the trapped charge. At the time of writing, however, we are not able to
suggest a clear way of achieving such targets in an MFDM structure, and by
relying exclusively on quantities accessible in experiments. In this respect,
we cannot but conclude that more work is needed to improve the electrical
probing of spontaneous polarization in ferroelectric-dielectric
heterostructures.

## Acknowledgements

This work was supported by European Union through the BeFerroSynaptic project
(GA:871737).

##  Appendix A Green’s function of a point charge in the MFDM stack

Figure 9: Sketch of the MFDM stack, where $E_{FT}$ and $E_{DB}$ denote the $z$
component of the electric field respectively at MF-FE interface and at MD-DE
interface.

In this section we discuss the analytical expression for the Green’s function
of the point charge defined in Equation 5. The potential $\psi(\textbf{r},z)$
produced by a point charge located in $(\textbf{r}_{0},z_{0})$ in a dielectric
material having a relative dielectric constant $\varepsilon_{r}$ can be
obtained by solving the Poisson
equation${}^{\text{\cite[cite]{[\@@bibref{}{NanoscaleMOS_Cambridge2011}{}{}]}}}$

$\nabla^{2}\psi(\textbf{r},z)=-\dfrac{e}{\varepsilon_{0}\varepsilon_{r}}\delta(\textbf{r}-\textbf{r}_{0})\delta(z-z_{0}).$
(13)

where $e$ is the elementary charge. We now introduce the 2D Fourier transform
of $\psi(\textbf{r},z)$ with respect to the coordinates $\bf{r}$$=$($x$,$y$),
and define the Fourier pair

$\begin{split}\psi(\textbf{r},z)&\approx\int_{{\bf q}}\psi({\bf
q},z)\,\text{exp}\left(-i\,{\bf q}\cdot{\bf r}\right)\,d{\bf q}\\\ \psi({\bf
q},z)&\approx\frac{1}{(2\pi)^{2}}\int_{A}\psi(\textbf{r},z)\,\text{exp}\left(i\,{\bf
q}\cdot{\bf r}\right)\,d\bf{r}\,\,.\end{split}$ (14)

Equation 14 assumes that the device area $A$ is large enough that the integral
over $A$ is a good approximation of the indefinite integral over the entire
($x$,$y$) plane. 111The formalism may be rephrased in terms of a Fourier
series by assuming periodic boundary conditions for $\psi(\textbf{r},z)$ at
the edges of the area $A$, that would however lead to identical
results.${}^{\text{\cite[cite]{[\@@bibref{}{NanoscaleMOS_Cambridge2011}{}{}]}}}$

If we now recall the identity

$\delta(\textbf{r}-\textbf{r}_{0})\approx\dfrac{1}{(2\pi)^{2}}\int_{{\bf
q}}\text{exp}\left[-i\,{\bf q}\cdot(\bf{r}-{\bf r}_{0})\right]\,d{\bf q}$ (15)

and substitute Equations 14, 15 into Equation 13, we can readily infer that
the unknown potential $\Psi({\bf q},z)$ takes the
form${}^{\text{\cite[cite]{[\@@bibref{}{NanoscaleMOS_Cambridge2011}{}{}]}}}$

$\psi({\bf q},z)=\dfrac{\text{exp}\left(i\,{\bf q}\cdot{\bf
r}_{0}\right)}{(2\pi)^{2}}\,\phi(q,z)$ (16)

where $\phi(q,z)$ must satisfy the differential equation

$\bigg{[}\frac{\partial^{2}}{\partial
z^{2}}-q^{2}\bigg{]}\,\phi=-\frac{e}{\varepsilon_{0}\varepsilon_{r}}\,\,\delta(z-z_{0})$
(17)

Let us now assume that the point charge is located at $z_{0}$$=$0, namely at
the FE-DE interface (see Figure 9). In this case the potential $\phi_{F}(q,z)$
in the ferroelectric and $\phi_{D}(q,z)$ in the dielectric region can be
written as

$\displaystyle\phi_{F}(z)$
$\displaystyle=C_{1}\,\text{exp}\left(q\,z\right)+C_{2}\,\text{exp}\left(-q\,z\right)\quad\quad
z<0$ (18a) $\displaystyle\phi_{D}(z)$
$\displaystyle=C_{3}\,\text{exp}\left(q\,z\right)+C_{4}\,\text{exp}\left(-q\,z\right)\quad\quad
z>0$ (18b)

where the four $q$ dependent constants $C_{1}$, $C_{2}$, $C_{3}$ and $C_{4}$
can be determined by using appropriate boundary conditions. At the interface
with metal electrodes we used $\phi_{F}(-t_{F})$$=$0, $\phi_{D}(t_{D})$$=$0 ,
whereas at the FE-DE interface we employed the conditions
$\phi_{F}(0)$$=$$\phi_{D}(0)$ and
$[\varepsilon_{0}\varepsilon_{F}(\partial\phi_{F}(0)/\partial
z)-\varepsilon_{0}\varepsilon_{D}(\partial\phi_{D}(0)/\partial z)]$$=$$e$. By
doing so we obtain

$\displaystyle\phi_{F}(z)$
$\displaystyle=C_{Fe}\,\left\\{\text{exp}\left(q\,z\right)-\text{exp}\left[-q\left(z+2\,t_{F}\right)\right]\right\\}\quad
z<0$ (19a) $\displaystyle\phi_{D}(z)$
$\displaystyle=C_{De}\,\left\\{\text{exp}\left(q\,z\right)-\text{exp}\left[-q\left(z-2\,t_{D}\right)\right]\right\\}\quad
z>0$ (19b)

with

$\displaystyle C_{Fe}$
$\displaystyle=\dfrac{1-\text{e}_{D}}{\varepsilon_{F}\left(1+\text{e}_{F}\right)\left(1-\text{e}_{D}\right)-\varepsilon_{D}\left(1-\text{e}_{F}\right)\left(1+\text{e}_{D}\right)}\cdot\dfrac{e}{q}$
(20a) $\displaystyle C_{De}$
$\displaystyle=\dfrac{1-\text{e}_{F}}{1-\text{e}_{D})}\cdot\,C_{Fe}$ (20b)

where we have the notation more compact by introducing
$\text{e}_{F}$$=$$\text{exp}\left(-2\,q\,t_{F}\right)$ and
$\text{e}_{D}$$=$$\text{exp}\left(2\,q\,t_{D}\right)$. The Green’s function
$G_{MF}(\textbf{r}_{0},z_{0})$ that we wish to determine is defined as

$G_{MF}(\textbf{r}_{0},z_{0})=\dfrac{\varepsilon_{0}\varepsilon_{F}}{e}\int\displaylimits_{A}E_{FT}(\textbf{r})\,d\textbf{r}\,\,$
(21)

where $E_{FT}(\textbf{r})$ denotes the $z$ component of the electric field at
the MF-FE interface (i.e. at $z$$=$$-t_{F}$) produced by a point charge $e$
located at $(\textbf{r}_{0},z_{0})$. By recalling the definition of the
Fourier transform pairs in Equation 14 and then using Equation 16, we have

$\begin{split}\int\displaylimits_{A}E_{FT}(\textbf{r})\,d\textbf{r}&=(2\pi)^{2}\,\lim_{q\to
0}E_{FT}({\bf q})=\\\ &=-\,(2\pi)^{2}\,\lim_{q\to 0}\dfrac{\partial\psi({\bf
q},z_{0})}{\partial z}=\\\ &=-\,\lim_{q\to 0}\dfrac{\partial\phi_{F}({\bf
q},z_{0})}{\partial z}\end{split}$ (22)

Equations21, 22 finally provide

$G_{MF}(\textbf{r}_{0},z_{0})=-\dfrac{\varepsilon_{0}\varepsilon_{F}}{e}\,\lim_{q\to
0}\dfrac{\partial\phi_{F}({\bf q},z_{0})}{\partial z}\,\,.$ (23)

For $z_{0}$$=$$-t_{F}$ the limit in Equation23 can be readily calculated by
using Equations 19a and 20a, so as to obtain

$G_{MF}(\textbf{r}_{0},0)=-\dfrac{C_{F}}{C_{F}+C_{D}}$ (24)

The corresponding Green’s function $G_{MD}$ at the MD electrode, defined in
Equation 32, can be derived with an entirely similar procedure. The result is

$G_{MD}(\textbf{r}_{0},0)=-\dfrac{\varepsilon_{0}\varepsilon_{D}}{e}\,\lim_{q\to
0}\dfrac{\partial\phi_{D}({\bf q},t_{D})}{\partial
z}=-\dfrac{C_{D}}{C_{F}+C_{D}}$ (25)

so that $(G_{MF}(\textbf{r}_{0},0)+G_{MD}(\textbf{r}_{0},0))$$=$$-$1.

Similar derivations apply to the case of a point charge located in the
ferroelectric (i.e. for $-t_{F}<z_{0}<0$) or in the dielectric (i.e. for
$0<z_{0}<t_{d}$). In the former case we obtain

$\displaystyle G_{MF}(\textbf{r}_{0},z_{0})$
$\displaystyle=-\dfrac{C_{F}+C_{D}\dfrac{\lvert
z_{0}\rvert}{t_{F}}}{C_{F}+C_{D}}$ (26) $\displaystyle
G_{MD}(\textbf{r}_{0},z_{0})$
$\displaystyle=-\dfrac{C_{D}\,\left(1-\dfrac{\lvert
z_{0}\rvert}{t_{F}}\right)}{C_{F}+C_{D}}$ (27)

whereas in the latter case we have

$\displaystyle G_{MF}(\textbf{r}_{0},z_{0})$
$\displaystyle=-\dfrac{C_{F}\,\left(1-\dfrac{z_{0}}{t_{D}}\right)}{C_{F}+C_{D}}$
(28) $\displaystyle G_{MD}(\textbf{r}_{0},z_{0})$
$\displaystyle=-\dfrac{C_{D}+C_{F}\dfrac{z_{0}}{t_{D}}}{C_{F}+C_{D}}.$ (29)

As it can be seen, even for an $z_{0}$$\neq$0 we have
$[G_{MF}(\textbf{r}_{0},z_{0})+G_{MD}(\textbf{r}_{0},z_{0})]=-1$.

##  Appendix B Charge and current at the MD electrode

The analysis of PUND measurements presented in the main paper is based on the
current $I_{MF}$ at the MF electrode. According to the $I_{MF}$, $I_{MD}$
definitions sketched in Figure 1, we see that $I_{MD}$ must be equal to
$I_{MF}$. For the completeness of definitions and derivations, we here report
a concise analysis about the charge $Q_{MD}$ and current $I_{MD}$ at the MD
electrode. We start with Q${}_{\text{MD}}$ written as (see Figure 9)

$Q_{MD}\,(t)=\dfrac{1}{A}\int\displaylimits_{A}-\varepsilon_{0}\varepsilon_{D}E_{DB}(\textbf{r},t)\,d\textbf{r}\,=-\varepsilon_{0}\varepsilon_{D}E_{DB,AV}(t)$
(30)

where $E_{DB}(\textbf{r},t)$ is the $z$ component of the field at the DE-MD
interface at $z$$=$$t_{D}$. The term
$\varepsilon_{0}\varepsilon_{D}E_{DB}(\textbf{r})$ can be expressed as

$\varepsilon_{0}\varepsilon_{D}E_{DB}(\textbf{r})=C_{S}\,V_{T}+\varepsilon_{0}\varepsilon_{D}E_{DI}(\textbf{r})$
(31)

where $E_{DI}(\textbf{r})$ is the contribution to the field due to the total
charge $\left[P(\textbf{r}_{0})+Q_{S}(\textbf{r}_{0})\right]$ at the FE-DE
interface. As already discussed in Section II of the main paper, we are here
assuming that trapping is dominated by interface traps at the FE-DE interface.
We can now define the Green’s function $G_{MD}(\textbf{r}_{0},z_{0})$ at the
MD electrode

$G_{MD}(\textbf{r}_{0},z_{0})=\dfrac{\varepsilon_{0}\varepsilon_{D}}{e}\int\displaylimits_{A}-E_{DB}(\textbf{r},\textbf{r}_{0}z_{0})\,d\textbf{r}$
(32)

that allows us to write the average $E_{DI,AV}$ as

$\varepsilon_{0}\varepsilon_{D}E_{DI,AV}=-\dfrac{1}{A}\int\displaylimits_{A}\left[P(\textbf{r}_{0})+Q_{S}(\textbf{r}_{0})\right]G_{MD}(\textbf{r}_{0},z_{0})\,d\textbf{r}_{0}$
(33)

For a charge at the FE-DE interface we have $G_{MD}(\textbf{r}_{0},0)\simeq-
C_{D}/C_{0}$ and thus

$\varepsilon_{0}\varepsilon_{D}E_{DI,AV}\simeq\dfrac{C_{D}}{C_{0}}\left(P_{AV}+Q_{S,AV}\right)$
(34)

so that $Q_{MD}(t)$ in Equation30 becomes

$Q_{MD}(t)=-C_{S}V_{T}(t)-\varepsilon_{0}\varepsilon_{D}E_{DI,AV}(t).$ (35)

By using similar assumptions as those embraced in Section II, we can write
$I_{MD}$ as

$\begin{split}I_{MD}&=-\dfrac{\partial Q_{MD}}{\partial
t}+I_{QS,MD}+I_{lkg}\\\ &=C_{S}\dfrac{\partial V_{T}}{\partial
t}+\varepsilon_{0}\varepsilon_{D}\dfrac{\partial E_{DI,AV}}{\partial t}+\\\
&+I_{QS,MD}+I_{lkg}\\\ &=C_{S}\dfrac{\partial V_{T}}{\partial
t}+\dfrac{C_{D}}{C_{0}}\dfrac{\partial P_{AV}}{\partial
t}+\dfrac{C_{D}}{C_{0}}\dfrac{\partial Q_{S,AV}}{\partial t}+\\\
&+I_{QS,MD}+I_{lkg}\\\ \end{split}$ (36)

where in the last equality we have used Equation 34.

By recalling the $I_{MF}$ expression in Equation 7 and the relation
$[I_{QS,MF}-I_{QS,MD}]$$=$$\partial Q_{S,AV}/\partial t$, we readily obtain
$(I_{MF}$$-$$I_{MD})$$=$0, thus confirming that $I_{MD}$ is equal to $I_{MF}$.

## References

  * [1] T. S. Böscke, J. Müller, D. Bräuhaus, U. Schröder, and U. Böttger, “Ferroelectricity in hafnium oxide thin films,” _Applied Physics Letters_ , vol. 99, no. 10, p. 102903, 2011. doi: 10.1063/1.3634052
  * [2] S. Salahuddin and S. Datta, “Use of Negative Capacitance to Provide Voltage Amplification for Low Power Nanoscale Devices,” _Nano Letters_ , vol. 8, no. 2, 2008. doi: 10.1021/nl071804g
  * [3] A. Jain and M. A. Alam, “Stability constraints define the minimum subthreshold swing of a negative capacitance field-effect transistor,” _IEEE Transactions on Electron Devices_ , vol. 61, no. 7, pp. 2235–2242, 2014\. doi: 10.1109/TED.2013.2286997
  * [4] A. I. Khan, U. Radhakrishna, K. Chatterjee, S. Salahuddin, and D. A. Antoniadis, “Negative capacitance behavior in a leaky ferroelectric,” _IEEE Transactions on Electron Devices_ , vol. 63, no. 11, pp. 4416–4422, Nov 2016.
  * [5] T. Rollo and D. Esseni, “New Design Perspective for Ferroelectric NC-FETs,” _IEEE Electron Device Letters_ , vol. 39, no. 4, pp. 603–606, April 2018\. doi: 10.1109/LED.2018.2795026
  * [6] T. Rollo and D. Esseni, “Influence of Interface Traps on Ferroelectric NC-FETs,” _IEEE Electron Device Letters_ , vol. 39, no. 7, pp. 1100–1103, July 2018. doi: 10.1109/LED.2018.2842087
  * [7] S. Pentapati, R. Perumal, S. Khandelwal, M. Hoffmann, S. K. Lim, and A. I. Khan, “Cross-domain optimization of ferroelectric parameters for negative capacitance transistors—part i: Constant supply voltage,” _IEEE Transactions on Electron Devices_ , vol. 67, no. 1, pp. 365–370, 2020. doi: 10.1109/TED.2013.2286997
  * [8] B. Max, M. Hoffmann, S. Slesazeck, and T. Mikolajick, “Direct correlation of ferroelectric properties and memory characteristics in ferroelectric tunnel junctions,” _IEEE Journal of the Electron Devices Society_ , vol. 7, pp. 1175–1181, 2019. doi: 10.1109/JEDS.2019.2932138
  * [9] R.Fontanini, J.Barbot, M.Segatto, S.Lancaster, Q.Duong, F.Driussi, L.Grenouillet, F.Triozon, J.Coignus, T.Mikolajick, S.Slesazeck, and D. Esseni, “Polarization switching and interface charges in BEOL compatible ferroelectric tunnel junctions,” in _2021 IEEE 51st European Solid-State Device Research Conference (ESSDERC)_ , no. n/a, 2021. doi: n/a p. n/a.
  * [10] H. Mulaosmanovic, S. Dünkel, J. Müller, M. Trentzsch, S. Beyer, E. T. Breyer, T. Mikolajick, and S. Slesazeck, “Impact of read operation on the performance of hfo2-based ferroe-lectric fets,” _IEEE Electron Device Letters_ , vol. 41, no. 9, pp. 1420–1423, 2020. doi: 10.1109/LED.2018.2846570
  * [11] D. Lizzit and D. Esseni, “Operation and design of Ferroelectric FETs for a BEOL compatible device implementation,” in _2021 IEEE 51st European Solid-State Device Research Conference (ESSDERC)_ , no. n/a, 2021. doi: n/a p. n/a.
  * [12] S. Slesazeck and T. Mikolajick, “Nanoscale resistive switching memory devices: a review,” _Nanotechnology_ , vol. 30, no. 35, p. 352003, jun 2019. doi: 10.1088/1361-6528/ab2084. [Online]. Available: https://doi.org/10.1088/1361-6528/ab2084
  * [13] K. M. Rabe, M. Dawber, C. Lichtensteiger, C. Ahn, and J. Triscone, _Physiscs of Ferroelectrics, a Modern Perspective_. Springer, 2007.
  * [14] M. Fukunaga and Y. Noda, “New technique for measuring ferroelectric and antiferroelectric hysteresis loops,” _Journal of the Physical Society of Japan_ , vol. 77, no. 6, p. 064706, 2008. doi: 10.1143/JPSJ.77.064706. [Online]. Available: https://doi.org/10.1143/JPSJ.77.064706
  * [15] P. D. Lomenzo, P. Zhao, Q. Takmeel, S. Moghaddam, T. Nishida, M. Nelson, C. M. Fancher, E. D. Grimley, X. Sang, J. M. LeBeau, and J. L. Jones, “Ferroelectric phenomena in si-doped hfo2 thin films with tin and ir electrodes,” _Journal of Vacuum Science & Technology B_, vol. 32, no. 3, p. 03D123, 2014. doi: 10.1116/1.4873323. [Online]. Available: https://doi.org/10.1116/1.4873323
  * [16] V. Mikheev, A. Chouprik, Y. Lebedinskii, S. Zarubin, Y. Matveyev, E. Kondratyuk, M. G. Kozodaev, A. M. Markeev, A. Zenkevich, and D. Negrov, “Ferroelectric second-order memristor,” _ACS Applied Materials & Interfaces_, vol. 11, no. 35, pp. 32 108–32 114, 2019. doi: 10.1021/acsami.9b08189 PMID: 31402643. [Online]. Available: https://doi.org/10.1021/acsami.9b08189
  * [17] H. Ryu, H. Wu, F. Rao, and W. Zhu, “Ferroelectric tunneling junctions based on aluminum oxide/ zirconium-doped hafnium oxide for neuromorphic computing,” _Scientific Reports_ , vol. 9, no. 1, p. 20383, Dec 2019. doi: 10.1038/s41598-019-56816-x. [Online]. Available: https://doi.org/10.1038/s41598-019-56816-x
  * [18] V. Deshpande, K. S. Nair, M. Holzer, S. Banerjee, and C. Dubourdieu, “Cmos back-end-of-line compatible ferroelectric tunnel junction devices,” _Solid-State Electronics_ , vol. 186, p. 108054, 2021. doi: https://doi.org/10.1016/j.sse.2021.108054. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S003811012100099X
  * [19] T. Schenk, U. Schroeder, and T. Mikolajick, “Correspondence - dynamic leakage current compensation revisited,” _IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control_ , vol. 62, no. 3, pp. 596–599, 2015. doi: 10.1109/TUFFC.2014.006774
  * [20] T. Rollo, F. Blanchini, G. Giordano, R. Specogna, and D. Esseni, “Stabilization of negative capacitance in ferroelectric capacitors with and without a metal interlayer,” _Nanoscale_ , vol. 12,, pp. 6121–6129, 2020\. doi: 10.1039/c9nr09470a
  * [21] H.-J. Lee, M. Lee, K. Lee, J. Jo, H. Yang, Y. Kim, S. C. Chae, U. Waghmare, and J. H. Lee, “Scale-free ferroelectricity induced by flat phonon bands in HfO2,” _Science_ , vol. 369, no. 6509, p. 1343–1347, 2020. doi: 10.1126/science.aba0067
  * [22] M. Kobayashi, N. Ueyama, K. Jang, and T. Hiramoto, “Experimental Study on Polarization-Limited Operation Speed of Negative Capacitance FET with Ferroelectric HfO2,” in _2016 IEEE International Electron Devices Meeting (IEDM)_ , Dec 2016, pp. 314–317.
  * [23] J. A. T. Kim and D. A. Antoniadis, “Dynamics of HfZrO2 Ferroelectric Structures: Experiments and Models,” in _2020 IEEE International Electron Devices Meeting (IEDM)_ , Dec 2020, pp. 441–444.
  * [24] D. Esseni and R. Fontanini, “Macroscopic and microscopic picture of negative capacitance operation in ferroelectric capacitors,” _Nanoscale_ , vol. 13, pp. 9641–9650, 2021. doi: 10.1039/D0NR06886A
  * [25] M. Hoffmann, M. Gui, S. Slesazeck, R. Fontanini, M. Segatto, D. Esseni, and T. Mikolajick, “Intrinsic nature of negative capacitance in multidomain Hf0.5Zr0.5O2-based ferroelectric/dielectric heterostructures,” _Advanced Functional Materials_ , vol. n/a, no. n/a, p. 2108494, 2021. doi: https://doi.org/10.1002/adfm.202108494. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/adfm.202108494
  * [26] R. Fontanini, M. Segatto, M. Massarotto, R. Specogna, F. Driussi, M. Loghi, and D. Esseni, “Modelling and design of FTJs as multi-level low energy memristors for neuromorphic computing,” _IEEE Journal of the Electron Devices Society_ , pp. 1–1, 2021. doi: 10.1109/JEDS.2021.3120200
  * [27] S. K. Arya, B. Kaur, G. Kaur, and K. Singh, “Optical and thermal properties of (70-x) SiO2-xNa2o-15CaO-10Al2O3-5TiO2 (10 $\leq$ x $\leq$ 25) glasses,” _Journal of Thermal Analysis and Calorimetry_ , vol. 120, no. 2, pp. 1163–1171, May 2015. doi: 10.1007/s10973-015-4392-8. [Online]. Available: https://doi.org/10.1007/s10973-015-4392-8
  * [28] W. Zheng, K. H. Bowen, J. Li, I. Dabkowska, and M. Gutowski, “Electronic structure differences in ZrO2 vs HfO2,” _The Journal of Physical Chemistry A_ , vol. 109, no. 50, pp. 11 521–11 525, Dec 2005. doi: 10.1021/jp053593e. [Online]. Available: https://doi.org/10.1021/jp053593e
  * [29] S. A. Vitale, J. Kedzierski, P. Healey, P. W. Wyatt, and C. L. Keast, “Work-function-tuned tin metal gate fdsoi transistors for subthreshold operation,” _IEEE Transactions on Electron Devices_ , vol. 58, no. 2, pp. 419–426, 2011. doi: 10.1109/TED.2010.2092779
  * [30] H. Schroeder, “Poole-Frenkel-effect as dominating current mechanism in thin oxide films—An illusion?!” _Journal of Applied Physics_ , vol. 117, no. 21, p. 215103, 2015. doi: 10.1063/1.4921949. [Online]. Available: https://doi.org/10.1063/1.4921949
  * [31] H. W. Park, S. D. Hyun, I. S. Lee, S. H. Lee, Y. B. Lee, M. Oh, B. Y. Kim, S. G. Ryoo, and C. S. Hwang, “Polarizing and depolarizing charge injection through a thin dielectric layer in a ferroelectric–dielectric bilayer,” _Nanoscale_ , vol. 13, pp. 2556–2572, 2021. doi: 10.1039/D0NR07597C
  * [32] J. Li, Y. Qu, M. Si, X. Lyu, and P. D. Ye, “Multi-Probe characterization of ferroelectric/dielectric interface by C-V, P-V and conductance methods,” in _2020 IEEE Symposium on VLSI Technology_ , 2020. doi: 10.1109/VLSITechnology18217.2020.9265069 pp. 1–2.
  * [33] M. Hoffmann, F. P. G. Fengler, M. Herzig, T. Mittmann, B. Max, U. Schroeder, R. Negrea, P. Lucian, S. Slesazeck, and T. Mikolajick, “Unveiling the double-well energy landscape in a ferroelectric layer,” _Nature_ , vol. 565, pp. 464–467, 2019. doi: 10.1038/s41586-018-0854-z. [Online]. Available: https://doi.org/10.1038/s41586-018-0854-z
  * [34] D. Esseni, P. Palestri, and L. Selmi, _Nanoscale MOS Transistors_. Cambridge University Press, 2011.


arrows

# Embarrassingly Simple MixUp for Time-series

Karan Aggarwal
Department of Computer Science
University of Minnesota
Minneapolis, MN, USA
<EMAIL_ADDRESS>
&Jaideep Srivastava
Department of Computer Science
University of Minnesota
Minneapolis, MN, USA
<EMAIL_ADDRESS>
This author’s work was completed prior to his affiliation with Amazon.

###### Abstract

Labeling time series data is an expensive task because of domain expertise and
dynamic nature of the data. Hence, we often have to deal with limited labeled
data settings. Data augmentation techniques have been successfully deployed in
domains like computer vision to exploit the use of existing labeled data. We
adapt one of the most commonly used technique called MixUp, in the time series
domain. Our proposed, MixUp++ and LatentMixUp++, use simple modifications to
perform interpolation in raw time series and classification model’s latent
space, respectively. We also extend these methods with semi-supervised
learning to exploit unlabeled data. We observe significant improvements of 1%
- 15% on time series classification on two public datasets, for both low
labeled data as well as high labeled data regimes, with LatentMixUp++.

##  1 Introduction

Time series data is one of the most commonly found data in nature across
various domains like healthcare, finance, astronomy, and meteorology. Labeling
time series data is quite an intensive process owing to the temporal and
dynamic nature of data, unlike other domains like images or text.

Time series data labeling usually requires domain expertise and time to gather
labeled data in time series. This is particularly an acute problem in domains
like healthcare where labeling has to be quite precise owing to high stakes.
Hence, for many applications, particularly in healthcare, only limited labeled
data is available.

Recently, there is an advent of machine learning models in time series
analysis, in particular deep learning, which require large datasets for
performing well. Also, training on small datasets suffers from issues like
over-fitting [13]. These issues can significantly impact the performance as
well as generalizability of time series models.

Data Augmentation is a technique to augment a training dataset with
artificially created examples. It has been used extensively in computer vision
to expand the size of training image datasets [9, 14]. It involves either
using data transformations or using generative models for creating artificial
training samples. Commonly used data transformations on images are rotating
the image, flipping, or modulating colors.

These data transformations are quite simple but specific to image domain, as
such transformations can completely change the meaning of a time-series. This
is compounded by the huge diversity in domains of time series data, with very
limited generic data transformations. There have been limited attempts in time
series domain to come up with such augmentations, mostly in the wearables
domains [18, 15]. They have found permutation or rotation based
transformations to be most effective.

Deep generative models like Generative Adversarial Networks [5] are used to
generate images from noise distribution or conditional on desired style [7] or
labels [11]. These have recently been used for time series data augmentation
as well [3, 12]. However, these methods are quite challenging to train and to
generate real world images.

MixUp [20] was proposed in computer vision, as a training method to
interpolate two real data points to generate a synthetic sample. Model is
trained on the interpolated data, and not on the original training data.
However, it is specific to the image domain as two images can be superimposed
over each other, but a linear interpolation of two time series can be quite
different. It has been adapted to other domains like natural language
processing interpolation of sentences in latent space for text classification
[6].

In this work, we adapt MixUp by simple extensions to time series
classification problems; we train the model with the interpolated data as well
as original data, in a true data augmentation fashion. Next, we also perform
multiple MixUp steps for each batch of original data. We perform the same
operation in latent space, called LatentMixUp. Our proposed methods, MixUp++
and LatentMixUp++ outperform baselines: non-augmentation based as well as
permutation based time series augmentation. The proposed methods, especially
LatentMixUp works particularly well in a low labeled data regime. We further
extend MixUp to semi-supervised settings with pseudo-labeling based MixUp.
These methods do not need any additional effort or hyper-parameter tuning
apart from classifier tuning while improving performance considerably.

In summary, we make following contributions:

  * •

We propose simple but effective extensions to MixUp, MixUp++ and LatentMixUp++
for time series classification;

  * •

We propose Pseudo-labeling based MixUp for semi-supervised learning to
leverage unlabeled data; and

  * •

Our results on two datasets: human activity and sleep staging, shows that our
methods outperform baselines considerably in both low labeled data regime as
well as high labeled data setting.

We organize the rest of this work as follows. Section 2.2 discusses relevant
literature in data augmentation. Section 3 discusses background concepts like
MixUp and our proposed methods. Section 4 presents details on our experimental
settings. We present our results in Section 5. Finally, we summarize our
conclusions in Section 6.

##  2 Related Work

In this section, we give an overview of data augmentation literature and data
augmentation in time series.

###  2.1 Data Augmentation in Machine Learning

Data augmentation is a technique of adding existing data with synthetically
generated data. This synthetic generation of data can be done through: 1)
Simple data transformations or, 2) using Generative machine learning methods.

##### Simple Data Transformation Augmentations

These simple data transformations encode human knowledge about the data
domain. For example, in computer vision tasks like classification, we know
that a transformation like rotating an image or adjusting its colors would not
change its class. These geometric image transformations like flipping,
rotating, cropping, or noise injection have been extensively used in the
computer vision domain to increase data set sizes [14]. Even the seminal
ImageNet paper [9], used these transformations to increase the training
dataset size. Taylor et al. [17] performed a empirical study on effect of
different transformations on the data augmentation. AutoAugment [2] is a
general augmentation technique for images based on reinforcement learning that
learns appropriate augmentation from the given image data from a list of image
augmentations. However, these methods are quite limited to images and do not
consider characteristics like temporal nature in time series. In this work we
explore MixUp [20], which interpolates two images for time-series data
augmentation. We further explore MixUp in latent model embedding space as has
been explored in domains like natural language processing with sentence mixup
[6].

##### Generative Machine Learning Models

Generative machine learning models generate synthetic data from scratch by
modeling the distribution of input data. The most prominent family of models
are Generative Adversarial Networks (GANs) [5]. GANs can generate image data
from a noise distribution by learning a function that transforms noise into an
image by learning over real world images through adversarial learning. Various
methods in GANs have been proposed like Neural Style Transfer [7], which can
create synthetic examples by converting a given input image with another theme
like adding a hat to a person’s headshot. Further, Conditional Adversarial
Networks (CGANs) [11], have been proposed to generate images conditioned on a
class label. Other generative methods include Variational Auto-Encoders (VAEs)
[8]. Generative models are however, quite difficult to train and to be used
for generating realistic data.

###  2.2 Data Augmentation for Time Series Data

There is limited work in the time series domain for data augmentation. Since
time series data is so diverse and so domain specific, there are very limited
generic geometric transformations in time series. Um et al. [18] explored data
transformations like rotation, scaling, permutation, or distance based dynamic
time warping for data augmentation on wearables data. They found rotation and
permutation transformations to be the most effective augmentations. While
permutation involves shuffling time series segments to create another sample,
rotation involves 3D rotation of wearable time series. Steven Eyobu et al.
[15] also explored using scaling and jitter for wearable time series data
augmentation.

Generative models like GANs have been used for time series data augmentation
like Esteban et al. [3]. They generate medical time series like EEG using GANs
with a recurrent neural network (RNN) backbone for both generator and
discriminator to model the time series sequence. They showed that synthetic
time series performed as well as real world time series data. Ramponi et al.
[12] propose Time-Conditional Generative Adversarial Networks (T-CGAN), and
use it to study the data augmentation for time series. They show considerable
improvement on time series datasets by augmenting with GAN generated time
series data. However, these methods are challenging to train and tune to
produce desirable results.

In this work, we explore MixUp for time series data augmentation. We compare
our method with permutation augmentation proposed by Um et al. [18] and show
the effectiveness of our simple method.

##  3 Methodology

In this section, we present the methodology used in this work. We first give a
background on MixUp technique, and next discuss our methodology.

###  3.1 Supervised Setting: MixUp++ and LatentMixUp++

####  3.1.1 Mixup

MixUp is a simple data augmentation technique proposed by Zheng et al. [20]
used in computer vision. It is based on a simple idea of a linear
interpolation of two labeled training samples to produce a synthetic training
sample. This results in: 1) increased number of labeled training samples; and
2) regularization in the supervised model’s embedding space. It has shown
encouraging results by improving the accuracy of supervised algorithms on
image classification datasets.

Given two training samples, ($x_{1}$, $y_{1}$) and ($x_{2}$, $y_{2}$),
synthetic interpolated example can be given by:

$\displaystyle~~\tilde{x}$ $\displaystyle=$ $\displaystyle\lambda
x_{1}+(1-\lambda)x_{2}$ (1) $\displaystyle~~\tilde{y}$ $\displaystyle=$
$\displaystyle\lambda y_{1}+(1-\lambda)y_{2}$ (2)

where, $\tilde{x}$ and $\tilde{y}$ are generated by example and label,
respectively. $\lambda$ is ratio of mixing, which is sampled from a beta
distribution as follows with a parameter $\alpha$:

$\displaystyle\lambda$ $\displaystyle\sim$
$\displaystyle\mathrm{Beta}(\alpha,\alpha)$ (3)

####  3.1.2 LatentMixUp

MixUp performs an interpolation in the raw input space to create a new
example. However, while it is intuitive to do such interpolation in the image
domain, it is not as intuitive in domains in natural language or even time
series. For example, interpolation of a cosine function and its opposite phase
cosine, can produce a constant value time series, which might be unlikely to
be observed. Hence, recent methods have proposed to perform mixup in latent
space of classification model in domains like Natural language with SenMixUp
[6] where two sentences can not be interpolated directly. We refer to it as
_LatentMixUp_ , to perform a mixup of two time series in the latent space,
described as follows.

Let $f(x)$ be the classifier with
$f:\mathbb{R}^{T}\longrightarrow\mathbb{R}^{|C|}$, where $C$ is the set of
output classes. We can further define $f(x)=g(h(x))$, where $h(x)$ is an
intermediate neural network representation. $g(x)$ is the remainder of neural
network layers that are built on top of $h(x)$ to perform the classification.
For example, in a 6 layered neural network, $h(x)$ can be the first 5 layers
of the network, and $g(x)$ is the last layer of the neural network that
performs the softmax for the classification task.

Given two training samples, ($x_{1}$, $y_{1}$) and ($x_{2}$, $y_{2}$),
synthetic interpolated example can be given by:

$\displaystyle~~h(\tilde{x})$ $\displaystyle=$ $\displaystyle\lambda
h(x_{1})+(1-\lambda)h(x_{2})$ (4) $\displaystyle~~\tilde{y}$ $\displaystyle=$
$\displaystyle\lambda y_{1}+(1-\lambda)y_{2}$ (5)

Final prediction on interpolated $h(\tilde{x})$ is done by passing it through
$g(.)$. Hence, prediction for the newly interpolated example,
$\tilde{y}_{pred}=g(h(\tilde{x}))$. The key idea behind LatentMixUp is that it
provides interpolation in the model’s latent space which is arguably more
linear than raw feature space, making interpolation operation more
representative of the data manifold.

####  3.1.3 Proposed: MixUp++ and LatentMixUp++

In MixUp and LatentMixUp, the model is trained _only_ on the interpolated
data, while the original data is not used. We hypothesize that this could
impact the model performance as original training data is real world data and
is important for creation of data manifold on which decision boundary is
created. Further, MixUp is done by randomly permuting a training batch and
interpolating it with the training batch examples in the given batch, with a
sampled value of $\lambda$. While with enough epochs of training, the model
could get to see enough pairs from all possible pairs with a good proportion
of mixing coefficient $\lambda$, it is not as efficient as the number of
epochs is usually limited.

We propose two embarrassingly simple but effective additions to the MixUp
training:

  * •

Train with original data: We keep the original data during training instead of
discarding the original data in MixUp [20].

  * •

Train with multiple MixUps for a single data batch: We perform mixup,
$k\in\mathbb{N}$ times for a single training data batch, each with different
sampled values of mixing coefficient $\lambda$ to increase the possible mixup
examples model is trained on.

We hypothesize that training entirely on interpolated examples is not optimal
for time series data, as interpolated examples can represent an entirely
different phenomenon for time series. Hence, it is important to use the
original real world data during training, and use the interpolated data as a
regularizer.

###  3.2 Semi-supervised Setting

While MixUp is entirely based in a supervised learning setting, we further
extend the notion of MixUp in the semi-supervised setting. For this purpose,
we use pseudo-labeling, a simple but commonly used semi-supervised learning
technique.

####  3.2.1 Pseudo-Labeling

Pseudo-labeling [10] is a simple technique to leverage unlabeled data in
presence of limited labeled data. The idea behind pseudo-labeling is simple:
we train the model on the limited labeled data, perform inference on unlabeled
data to choose the examples the model is quite confident on. These examples
are then added to the training set with model predicted labels for the next
training iteration. This process is continued until we the training stabilises
or maximum number of epochs are reached.

More formally, let labeled data be given as
$\mathcal{D}_{l}=\\{x_{i}^{l},y_{i}^{l}\\}_{i=1}^{n}$, where $n$ is the number
of labeled samples. Let, $\mathcal{D}_{u}=\\{x_{j}^{u}\\}_{j=1}^{m}$ be an
unlabeled dataset, with $m$ being the number of unlabeled samples. We train a
classifier $f(x)$ with labeled and unlabeled examples. Firstly, labeled
examples are used to train the model in the epoch, which is used to do
inference over each unlabeled example $x_{j}^{u}$. If predicted
$f(x^{u})=\hat{y}^{u}\geq\tau$, then the pair $(x^{u},y^{u})$ is used to train
the model further. $\tau$ is the confidence threshold to only select highly
confident examples in training. $y^{u}\in\mathbb{R}^{|C|}$ is one hot encoding
of highest probability class above the confidence threshold $\tau$ given by:

$~~y_{c}^{u}=~~~~\begin{cases}~~~1,&\text{if $c$ =
$\mathrm{argmax}_{c\in|C|}\hat{y}^{u}$}\\\
~~~0,&\text{otherwise}\end{cases}~~~$ (6)

Hence, we given a label of $y^{u}$ to the highest confident class of an
unlabeled sample $x^{u}$, which is then used for training the model. Pseudo-
label classification loss can be written as follows:

$~~\mathcal{L}(f,\mathcal{D}_{l},\mathcal{D}_{u})=\sum_{x^{l}\in\mathcal{D}_{l}}l(f,x^{l},y^{l})+\sum_{x^{u}\in\mathcal{D}_{u}}\mathbb{I}(f(x^{u})\geq\tau)l(f,x^{u},y^{u})$
(7)

where, $l(f,x,y)$ is the standard cross-entropy classification loss. Note,
value of threshold, $\tau$ is usually set to be high. We use a $\tau$ of 0.99
in this work. A low value of $\tau$ would allow wrongly labeled samples from
unlabeled data to find their way into the training data. This can destabilize
the model, further reinforced by subsequent pseudo-labeling rounds.

Figure 1:  Schematic view of Pseudo-labeling with MixUp. First, we get the
most confident samples from the unlabeled data, called pseudo-labels. We use
these pseudo-labels and labeled data for MixUp. Finally, we train the model on
three sources: labeled data, pseudo-labeled data, and MixUp data.

####  3.2.2 Pseudo-Labeling with MixUp

We propose to leverage MixUp during pseudo-labeling in order to leverage the
unlabeled data. The idea is simple: we add highly confident samples from
unlabeled datasets with their pseudo-labels to the examples MixUp has access
to. This way, MixUp interpolates between the labeled as well as pseudo-labeled
samples from unlabeled data. Pseudo-Labeling with MixUp is shown in Figure 1.

Next, we discuss our experimental settings to evaluate the effectiveness of
our proposed methods: _MixUp++_ and _LatentMixUp++_.

##  4 Experimental Settings

In this section, we present our experimental settings: datasets used, data
augmentation baselines, and hyper-parameters tuned.

###  4.1 Datasets

We use the following two datasets in this work related to human activity
recognition and sleep staging:

  * •

Sleep EDF111https://physionet.org/challenge/2012/: Sleep-EDF dataset [4]
contains polysomnography (PSG) for 20 subjects. We follow Supartak et al. [16]
by using the EEG channels (Fpz-Cz) sampled at 100 Hz, for sleep stage
classification. Goal is to detect the sleep stage of each person for every 30
second window into Awake (W), deep sleep (N1, N2, N3), and Rapid Eye Movement
(REM) sleep. This data has 25612 training and 8910 test samples. Note, the
training and test sets are independent in terms of subjects following the
implementation in DeepSleepNet222https://github.com/akaraspt/deepsleepnet.

  * •

UCI
HAR333https://archive.ics.uci.edu/ml/datasets/human+activity+recognition+using+smartphones:
UCI HAR dataset [1] contains multivariate time series across 6 channels of
motion state recording of 30 volunteers, aged 19-48 years. It tracks 6 kinds
of activities like walking, walking upstairs, walking downstairs, lying,
sitting, or standing up. Data was collected using a waist bound Galaxy SII
device with a sampling rate of 50 Hz. Task is to detect the state of the
person for each 2.56 second window. This splitting gives about 7352 training
set windows, and 2947 test set windows.

###  4.2 Baselines Used

We use the following data augmentation methods as a comparison with our
proposed approach:

  * •

Supervised Learning: Baseline model without using any data augmentation,
trained only on original real world labeled data.

  * •

PermuteAugment: In this data augmentation technique, we permute segments of
time series, based on Um _et al._ [18] where permutation and rotation were the
most effective augmentation methods for time-series data. Since, rotation is
specific to the dataset they used, we use permutation as the augmentation
method as implemented in their code444https://github.com/terryum/Data-
Augmentation-For-Wearable-Sensor-Data.

  * •

MixUp: We compare MixUp++ with the conventional MixUp algorithm.

  * •

LatentMixup: This is just the conventional MixUp in latent space as described
in Section 3. We implement this on the last layer before the softmax of our
transformer neural network, described next.

For all these baselines, we use the same classifier, a transformer neural
network [19]. Using the same classifier ensures that we can confidently
measure the difference between data augmentations of different methods.

###  4.3 Hyper-parameters

We use the same train-test split as provided in the datasets. For hyper-
parameter tuning we randomly sample 80% of the train set for training, and
rest 20% for validation. For the Sleep-EDF dataset, we use subject stratified
split to ensure no overlap of subjects in training and validation sets. For
transformer architecture, we tried the number of layers
$\in\\{1,2,3,4,5,6\\}$, and selected 5 layers and 5 as the number of heads.
For layer size, we searched $\in\\{32,64,100,128,160\\}$ and selected 100 with
a dropout of 0.15. We use Adam optimizer with a learning rate of $0.0002$. We
searched for $\tau\in\\{0.95,0.97,0.98,\\\ 0.99,0.995\\}$ and selected 0.99 as
the confidence threshold for pseudo labels. Further, all experiments were
performed 10 times with 10 different seeds to report the mean and standard
deviations of all the results reported in the results section, Sec 5.

##  5 Results and Analysis

In this section, we present results on the time series classification for two
datasets: HAR and Sleep-EDF. Firstly, we present results in a fully supervised
setting. Next, we show ablation results of performance as the proportion of
training data. Finally, we show the performance of semi-supervised pseudo-
labeling mixup.

Table 1: Time series classification results with various data augmentation methods measured on Accuracy/F1 micro, F1 Macro, and Cohen’s Kappa for two datasets: HAR and Sleep-EDF. Note, differences between supervised and LatentMixUp++ are statistically significant with a $p<0.01$ with student $t$-test. Method |  Accuracy (%) |  F1 Macro (%) |  Kappa (%)
---|---|---|---
_HAR_
Supervised | 92.95 $\pm$ 0.83 | 92.99 $\pm$ 0.89 | 91.52 $\pm$ 1.0
PermuteAugment | 91.8 $\pm$ 0.91 | 91.97 $\pm$ 0.92 | 90.14 $\pm$ 1.09
PermuteAugment++ | 93.1 $\pm$ 0.43 | 93.26 $\pm$ 0.46 | 91.71 $\pm$ 0.51
MixUp | 92.63 $\pm$ 0.56 | 92.71 $\pm$ 0.65 | 91.14 $\pm$ 0.67
MixUp++ | 93.29 $\pm$ 0.80 | 93.38 $\pm$ 0.85 | 91.94 $\pm$ 0.97
MixUp++ (w/ 2 Iterations) | 93.45 $\pm$ 0.41 | 93.57 $\pm$ 0.45 | 92.13 $\pm$ 0.50
LatentMixUp | 94.07 $\pm$ 0.70 | 94.17 $\pm$ 0.73 | 92.87 $\pm$ 0.84
LatentMixUp++ | 94.41 $\pm$ 0.92 | 94.46 $\pm$ 0.95 | 93.28 $\pm$ 1.1
LatentMixUp++ (w/ 2 Iterations) |  94.44 $\pm$ 0.72 |  94.52 $\pm$ 0.75 |  93.32 $\pm$ 0.87
_Sleep-EDF_
Supervised | 80.57 $\pm$ 0.34 |  73.52 $\pm$ 0.85 | 73.48 $\pm$ 0.42
PermuteAugment | 74.21 $\pm$ 2.09 |  67.59 $\pm$ 2.31 |  64.3 $\pm$ 2.71
PermuteAugment++ |  78.89 $\pm$ 0.35 |  71.75 $\pm$ 0.52 | 71.63 $\pm$ 0.43
MixUp | 79.14 $\pm$ 0.96 |  66.3 $\pm$ 0.91 | 70.52 $\pm$ 1.47
MixUp++ | 80.47 $\pm$ 0.70 |  70.82 $\pm$ 1.62 | 72.8 $\pm$ 0.92
MixUp++ (w/ 2 Iterations) | 80.00 $\pm$ 0.57 |  68.7 $\pm$ 1.00 | 72.13 $\pm$ 0.67
LatentMixUp | 80.83 $\pm$ 0.82 | 72.71 $\pm$ 1.04 | 73.56 $\pm$ 1.09
LatentMixUp++ | 81.08 $\pm$ 0.56 |  73.74 $\pm$ 1.05 |  73.89 $\pm$ 0.65
LatentMixUp++ (w/ 2 Iterations) |  81.12 $\pm$ 0.47 |  73.79 $\pm$ 0.82 |  73.94 $\pm$ 0.62
Figure 2:  Classification Performance on HAR dataset as a function of
percentage of training data. Note, MixUp++ and LatentMixUp++ performance is
reported for models with 2 mixup batches. Figure 3:  Classification
Performance on Sleep-EDF dataset as a function of percentage of training data.
Note, MixUp++ and LatentMixUp++ performance is reported for models with 2
mixup batches.

###  5.1 Data Augmentation in Supervised Setting

Classification results for the two datasets are presented in Table 1. We
observe that LatentMixUp with two batches of mixup, shows the best results,
for the both datasets compared with the supervised baselines and other data
augmentation baselines. Original MixUp [20], performs even lower than the
vanilla supervised baseline while vanilla LatentMixUp performs only slightly
better than the vanilla supervised baseline. The performance difference is
especially stark in Sleep-EDF dataset.

Adding batches of MixUp with original data helps MixUp, though all MixUp
variants perform below or similar to supervised baseline. LatentMixUp
standalone performs better than supervised baseline, while also benefiting
significantly from the addition of original data in LatentMixUp++ variants. We
can attribute this difference between MixUp and LatentMixUp to the fact that
addition of two time series in raw time space could produce something that is
completely unrecognizable, e.g., addition of two time out of sync consine
functions could produce a zero-valued time series. Hence, mixing time series
in the latent space with features more relevant to the time series class, is
expected to produce better examples. Our hypothesis is confirmed by the
empirical results.

###  5.2 Ablation: Performance as a proportion of training Data

We experiment to see the performance of our methods as a function of amount of
training data, as shown in Figure 2 and Figure 3. We report MixUp++ and
LatentMixUp++ performance with 2 mixup batches. We observe that LatentMixUp++
still beats all the baseline methods for all the percentages of training data.
Additionally, LatentMixUp++ has a much higher F1 score differential low data
regime (1% or 5%), than in high label data regime. This shows the
effectiveness of LatentMixUp in regularizing the latent space for better
classification performance. We note that MixUp++ is only marginally better
than purely supervised method in low data regime for HAR dataset, but
significantly better for Sleep-EDF dataset. We could attribute this to
avoidance of over-fitting by label regularization in low data regimes which
are prone to over-fitting.

Figure 4: Classification Performance on HAR dataset as a function of
percentage of labeled data in the semi-supervised pseudo-labeling setting.
Note, MixUp++ and LatentMixUp++ performance is reported for models with 2
mixup batches. Figure 5:  Classification Performance on Sleep-EDF dataset as a
function of percentage of labeled data in the semi-supervised pseudo-labeling
setting. Note, MixUp++ and LatentMixUp++ performance is reported for models
with 2 mixup batches.

###  5.3 Data Augmentation in Semi-Supervised Setting with Pseudo-labeling
Mixup

Next, we experiment with semi-supervised learning by using pseudo-labeling
MixUp, as described in Section 3. We used a confidence score of 0.99 to select
highly confident examples for pseudo-labeling MixUp. Results are shown in
Figure 4 and Figure 5 for HAR and sleep datasets, respectively. We report
MixUp++ and LatentMixUp++ performance with 2 mixup batches. LatentMixUp++
still outperforms all other semi-supervised baselines, especially in the low
labeled data regime with 6-7% absolute increase in F1 score for the 1% label
data scenario versus traditional pseudo labeling for the two datasets. MixUp++
performs similar to pseudo labeling baseline. The out-performance reduces to
0.5% - 1% in the higher labeled data regime like 50%.

Comparing performance of semi-supervised pseudo-labeling scenarios with purely
supervised scenarios we see a drastic jump in low labeled data scenarios,
especially for Sleep-EDF dataset. However, we do notice a drop in performance
versus purely supervised method in the Sleep-EDF dataset as we increase
labeled data to 25%. This can be attributed to issues with pseudo-labeling as
highly confident examples can bias the dataset, and propagate errors in
initial training [21].

Based on these results, we can conclude that LatenMixUp++ with pseudo-labeling
MixUp improves performance considerably in semi-supervised settings,
especially for low labeled data regime.

Figure 6:  Classification Performance on HAR dataset for MixUp methods as a
function of number of MixUp batches per original data batch.

###  5.4 Ablation Study: Number of MixUp Batches

In the previously shown results, we used MixUp methods with 2 batches of mixup
data per original data batch. We present an ablation study in Figure 6, on HAR
dataset, as a function of the number of mixup data batches per original data
batch. We observe a maximum when the number of batches is equal to two;
performance drops subsequently as we increase the number of mixup batches.
While this is empirical observation, we speculate that this is due to loss of
decision boundary in the data manifold of the base model as the number of
synthetic mixuped examples far exceeds the number of real examples.

##  6 Conclusions

Time series are ubiquitous in nature, and commonly found in various domains.
Labeling time series data is a challenging task as it needs domain expertise
and more effort owing to the dynamic and temporal nature of the data. This
often leads to situations with limited labeled data, with limited scope for
getting more labeled data. Machine learning models are particularly data
intensive. Data augmentation techniques have been used extensively in computer
vision to overcome these issues. In this work, we extend MixUp to time series
classification. MixUp is a commonly used simple technique in computer vision
to interpolate input samples. We propose MixUp++ and LatentMixUp++ which use
simple modifications over MixUp to perform interpolation in raw time series
and classification model’s latent space. We also propose extension of our
methods in semi-supervised learning with pseudo labeling. Our results on two
public datasets, indicate considerable improvement in performance for both low
labeled data regime as well as high labeled data using LatentMixUp++.

## References

  * Anguita et al. [2013] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortiz, et al.  A public domain dataset for human activity recognition using smartphones.  In _Esann_ , volume 3, page 3, 2013.
  * Cubuk et al. [2018] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation policies from data.  _arXiv preprint arXiv:1805.09501_ , 2018.
  * Esteban et al. [2017] Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch.  Real-valued (medical) time series generation with recurrent conditional gans.  _arXiv preprint arXiv:1706.02633_ , 2017.
  * Goldberger et al. [2000] Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley.  Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals.  _circulation_ , 101(23):e215–e220, 2000.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _NIPS 2014_ , pages 2672–2680, 2014.
  * Guo et al. [2019] Hongyu Guo, Yongyi Mao, and Richong Zhang.  Augmenting data with mixup for sentence classification: An empirical study.  _arXiv preprint arXiv:1905.08941_ , 2019.
  * Jing et al. [2019] Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song.  Neural style transfer: A review.  _IEEE transactions on visualization and computer graphics_ , 26(11):3365–3385, 2019.
  * Kingma et al. [2019] Diederik P Kingma, Max Welling, et al.  An introduction to variational autoencoders.  _Foundations and Trends® in Machine Learning_ , 12(4):307–392, 2019.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _NIPS_ , pages 1097–1105, 2012.
  * Lee et al. [2013] Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _Workshop on challenges in representation learning, ICML_ , volume 3, page 896, 2013.
  * Mirza and Osindero [2014] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ , 2014.
  * Ramponi et al. [2018] Giorgia Ramponi, Pavlos Protopapas, Marco Brambilla, and Ryan Janssen.  T-cgan: Conditional generative adversarial network for data augmentation in noisy time series with irregular sampling.  _arXiv preprint arXiv:1811.08295_ , 2018.
  * Salman and Liu [2019] Shaeke Salman and Xiuwen Liu.  Overfitting mechanism and avoidance in deep neural networks.  _arXiv preprint arXiv:1901.06566_ , 2019.
  * Shorten and Khoshgoftaar [2019] Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  _Journal of big data_ , 6(1):1–48, 2019.
  * Steven Eyobu and Han [2018] Odongo Steven Eyobu and Dong Seog Han.  Feature representation and data augmentation for human activity classification based on wearable imu sensor data using a deep lstm neural network.  _Sensors_ , 18(9):2892, 2018.
  * Supratak et al. [2017] Akara Supratak, Hao Dong, Chao Wu, and Yike Guo.  Deepsleepnet: A model for automatic sleep stage scoring based on raw single-channel eeg.  _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , 25(11):1998–2008, 2017.
  * Taylor and Nitschke [2018] Luke Taylor and Geoff Nitschke.  Improving deep learning with generic data augmentation.  In _2018 IEEE symposium series on computational intelligence (SSCI)_ , pages 1542–1547. IEEE, 2018.
  * Um et al. [2017] Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić.  Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks.  In _Proceedings of the 19th ACM international conference on multimodal interaction_ , pages 216–220, 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Zhang et al. [2017] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _International Conference on Learning Representations_ , 2017.
  * Zou et al. [2019] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang.  Confidence regularized self-training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5982–5991, 2019.