# On Delay-Doppler Plane Orthogonal Pulse

Hai Lin
Osaka Metropolitan University
Sakai, Osaka, 599-8531, Japan
Email<EMAIL_ADDRESS>Jinhong Yuan
The University of New South Wales
Sydney, NSW, 2052, Australia
Email<EMAIL_ADDRESS>
###### Abstract

In this paper, we analyze the recently discovered delay-Doppler plane
orthogonal pulse (DDOP), which is essential for delay-Doppler plane multi-
carrier modulation waveform. In particular, we introduce a _local
orthogonality_ property of pulses corresponding to Weyl-Heisenberg (WH)
_subset_ and justify the DDOP’s existence, in contrast to _global
orthogonality_ corresponding to WH _set_ governed by the WH frame theory.
Then, sufficient conditions for locally-orthogonal pulses are presented and
discussed. Based on the analysis, we propose a general DDOP design. We also
derive the frequency domain representation of the DDOP, and compare the DDOP-
based orthogonal delay-Doppler division multiplexing (ODDM) modulation with
other modulation schemes, in terms of TF signal localization. Interestingly,
we show perfect local orthogonality property of the DDOP with respect to
delay-Doppler resolutions using its ambiguity function.

##  I Introduction

In digital communications, a modulation scheme usually requires a set of
(bi)orthogonal _analog_ pulses or continuous time functions, each of which
carries an information-bearing _digital_ symbol, to synthesize the signal
waveform[1]. Therefore, the modulation process can be intuitively thought of
as placing these pulses in the time-frequency (TF) plane, and the
(bi)orthogonality can be achieved by placing them with proper TF distance.
Such modulation schemes include single carrier (SC) modulation with temporally
spaced pulses, and multi-carrier (MC) modulation whose pulses are spaced both
temporally and spectrally. Meanwhile, for a communication system, a transmit
signal always consists of a finite number of pulses and occupies a finite TF
region in the TF plane, determining the signal’s duration and bandwidth.

In the context of MC modulation, the pulses are typically generated by TF
shifting a _prototype pulse_ in accordance with a frequency resolution
$\mathcal{F}$ and a time resolution $\mathcal{T}$. The minimum TF distance
among these pulses can be quantified by $\mathcal{R}=\mathcal{T}\mathcal{F}$,
called as joint TF resolution (JTFR) in this paper. The fundamental issue of
designing an MC modulation scheme is to find the prototype pulse that can form
(bi)orthogonal pulses with respect to $\mathcal{T}$ and $\mathcal{F}$.
Conventionally, these TF-shifted pulses are considered as _Wely-Heisenberg_
(WH) or _Gabor_ function set[2, 3, 4]. According to the WH frame theory, the
(bi)orthogonal WH function sets only exist for $\mathcal{R}\geq 1$[5, 6], and
therefore most of orthogonal MC modulation schemes are designed with
$\mathcal{R}\geq 1$ [7, 8].

Recently, a delay-Doppler plane MC (DDMC) modulation named as the orthogonal
delay-Doppler division multiplexing (ODDM) modulation, was proposed in [9,
10]. Considering that linear time-varying (LTV) channels in a stationary
region that can be modelled as a delay-Doppler (DD) channel with a
deterministic spreading function, the ODDM modulation employs a newly
discovered DD plane orthogonal pulse (DDOP) to couple the modulated MC signal
with the DD channel. It achieves superior performance by harvesting both time
and frequency diversity, while it is shown in [9, 10] that the DDOP can form
an orthogonal function set with respect to the DD plane resolutions. Because
the DD plane’s TF resolutions result in a JTFR $\mathcal{R}_{\textrm{DD}}<1$,
the DDOP seems inconsistent with current (bi)orthogonal pulses design
principles. Although its orthogonality has been proved, a rational explanation
for the DDOP’s unique properties is still missing.

In this paper, we take an in-depth look into the DDOP and justify its
existence. We introduce a _local orthogonality_ property and clarify that the
DDOP only needs to satisfy local orthogonality, in contrast to _global
orthogonality_ governed by the WH frame theory. Then, sufficient conditions
for pulse to achieve local orthogonality are analyzed. Based on the analysis,
we propose a general DDOP design. Our contributions can be summarized as
follows:

  * •

We point out that only local (bi)orthogonality in the finite TF region rather
than global (bi)orthogonality in the whole TF plane is required by a
modulation scheme. Accordingly, we show that a WH _subset_ rather than a WH
set is required in the pulse design.

  * •

We reformulate the (bi)orthogonal pulse design problem, based on the local
(bi)orthogonality. We show that the DDOP forms a WH subset that satisfies the
local orthogonality.

  * •

We analyze the local orthogonality with respect to TF resolutions, and discuss
the corresponding sufficient conditions. We reveal that for a limited number
of subcarriers, surprisingly, there are _infinite_ pulses orthogonal with
respect to $\mathcal{F}$, as long as they are periodic functions with a
specified period related to the number of subcarriers.

  * •

By introducing cyclic prefix (CP) and cyclic suffix (CS) to achieve the
specified periodicity, we propose a general DDOP design, which releases the
duration constraint of square-root Nyquist (SRN) sub-pulses in our previously
designed DDOP.

  * •

We derive the frequency domain representation of the DDOP. Together with the
DDOP’s time domain representation, we illustrate the DDOP-based ODDM’s TF
signal localization, and schematically compare it with those of other
modulation schemes. The ambiguity function shows perfect local orthogonality
property of the DDOP with respect to delay-Doppler resolutions.

Notations: In this paper, $\operatorname{\Pi}_{\mathfrak{T}}(t)$ stands for
the rectangular pulse with unit energy and support $[0,\mathfrak{T}]$. Given
the number of subcarriers $N$, $a(t)$ denotes the SRN pulse for interval
$\mathcal{T}$, with energy $\frac{1}{N}$ and support $[-T_{a}/2,T_{a}/2]$.
$\mathcal{A}_{g,\gamma}(\cdot)$ is the (cross)ambiguity function of $g(t)$ and
$\gamma(t)$.

##  II WH set based pulse design principles

Let us first introduce main parameters and their notations for an MC
modulation in Table I. The transmit pulses in an MC modulation can be
represented by the function set

$\left(g,\mathcal{T},\mathcal{F}\right)=\left\\{g_{m,n}\right\\}_{m,n\in\mathbb{Z}},$
(1)

where $g_{m,n}\coloneqq g(t-m\mathcal{T})e^{j2\pi
n\mathcal{F}(t-m\mathcal{T})}$ and $g(t)$ is the prototype pulse. Similarly,
we can form the receive pulses $\left(\gamma,\mathcal{T},\mathcal{F}\right)$
using another prototype pulse $\gamma(t)$ with the same TF resolutions. Note
that because a time-limited signal cannot be strictly band-limited, the
bandwidth of $g(t)$, $B_{g}$, is defined in an essential sense [11].

TABLE I: MC Modulation Parameters  Notation |  Parameter
---|---
$\mathcal{F}$ |  frequency resolution, frequency spacing
$\mathfrak{T}$ | symbol period, $\mathfrak{T}=1/\mathcal{F}$
$\mathcal{T}$ | time resolution, symbol interval
$\mathcal{R}$ | JTFR, $\mathcal{R}=\mathcal{T}\mathcal{F}$
$N$ | number of subcarriers
$M$ | number of symbols
$g(t)$ | transmit (prototype) pulse
$T_{g}$ | duration of $g(t)$, symbol duration
$B_{g}$ | bandwidth of $g(t)$

Given $\mathcal{T}$ and $\mathcal{F}$, the fundamental issue of an MC
modulation is to find $g(t)$ and $\gamma(t)$ satisfying the orthogonal
condition of

$\langle
g_{m,n},g_{\dot{m},\dot{n}}\rangle=\delta(m-\dot{m})\delta(n-\dot{n}),$ (2)

or the biorthogonal condition of

$\langle
g_{m,n},\gamma_{\dot{m},\dot{n}}\rangle=\delta(m-\dot{m})\delta(n-\dot{n}).$
(3)

By considering the TF plane as a 2D phase space, the function set in (1) forms
a discrete lattice “sampling” the phase space[12, 2], where the “sampling”
resolution is the JTFR $\mathcal{R}$. Then, the function set in (1) can be
treated as a WH set. According to the WH frame theory, the existence of
(bi)orthogonal WH set depends on the “sampling” resolution and can be
summarized as [13, 2, 4, 3, 5, 12, 14, 7, 15]:

  * •

Critical sampling ($\mathcal{R}=1$) : Orthogonal WH sets exist. However, they
have either infinite time or frequency energy spread according to the Balian-
Low theory [16], and therefore are not TF well-localized.

  * •

Undercritical sampling ($\mathcal{R}>1$) : TF well-localized orthogonal or
biorthogonal WH sets exist, if $\mathcal{R}$ is sufficiently larger than $1$.

  * •

Overcritical sampling ($\mathcal{R}<1$) : Neither orthogonal nor biorthogonal
WH sets exist.

With the transmit pulses in (1), the transmit waveform of an MC modulation can
be represented as

$\displaystyle
x(t)=\sum_{m=0}^{M-1}\sum_{n=0}^{N-1}X_{m,n}g(t-m\mathcal{T})e^{j2\pi
n\mathcal{F}(t-m\mathcal{T})},$ (4)

where $X_{m,n}$’s are the information-bearing digital symbols.

##  III ODDM modulation

In the design of modulation schemes, the primary concern is the dispersive
effect of the channel. A doubly-selective wireless channel with both time and
frequency dispersion is usually considered as a LTV system, and represented by
its time-varying channel impulse response (TV-CIR) or DD spread function[17].

###  III-A DD channel model

Since the transmit signal is band- and time-limited, we always apply an
appropriate bandpass filtering and a subsequent sampling at the receiver. As a
result, we observe an equivalent channel that is the band- and time-limited
version of the physical channel. Let the sampling rate and duration be $W_{0}$
and $T_{0}$, respectively. The equivalent DD channel can be written as [17]

$h(\tau,\nu)=\sum_{p=1}^{P}h_{p}\delta(\tau-\tau_{p})\delta(\nu-\nu_{p}),$ (5)

with $\tau_{p}=\frac{l_{p}}{W_{0}}$, $\nu_{p}=\frac{k_{p}}{T_{0}}$,
$l_{p},k_{p}\in\mathbb{Z}$, where $\frac{1}{W_{0}}$ and $\frac{1}{T_{0}}$ are
the delay and Doppler resolutions, respectively.

Figure 1: $u(t)$, the transmit pulse of ODDM modulation.

###  III-B ODDM modulation and DDOP

To couple the MC signal with the DD channel in (5), the ODDM matches its
signal resolutions to the delay and Doppler resolutions, namely set
$\mathcal{T}=\frac{1}{W_{0}}$ and $\mathcal{F}=\frac{1}{T_{0}}$, respectively.
Note that for an ODDM signal, we have $W_{0}=\frac{M}{T}$ and $T_{0}=NT$.
Then, an ODDM frame without the frame-wise CP can be written as[10]

$\displaystyle
x(t)=\sum_{m=0}^{M-1}\sum_{n=0}^{N-1}X_{m,n}u\left(t-m\frac{T}{M}\right)e^{j2\pi
n\frac{1}{NT}(t-m\frac{T}{M})},$ (6)

where $u(t)$ is the DDOP given by

$u(t)=\sum_{\dot{n}=0}^{N-1}a(t-\dot{n}T).$ (7)

As shown in Fig. 1, the duration of $a(t)$ in $u(t)$ is $T_{a}=2Q\frac{T}{M}$.
When $2Q\ll M$ and therefore $T_{a}\ll T$, it has been proved in [10] that
$u(t)$ satisfies the orthogonal property

$\mathcal{A}_{u,u}\left(\bar{m}\frac{T}{M},\bar{n}\frac{1}{NT}\right)=\delta(\bar{m})\delta(\bar{n}),$
(8)

for $|\bar{m}|\leq M-1$ and $|\bar{n}|\leq N-1$. Because the corresponding
JTFR of
$\mathcal{R}_{\textrm{DD}}=\frac{T}{M}\times\frac{1}{NT}=\frac{1}{MN}\ll 1$
does not allow the existence of (bi)orthogonal WH set, a natural question
arises: How to explain the existing DDOP in [10] and whether is there any
general DDOP design principle?

##  IV Global and Local (Bi)Orthogonality

From (8), one can see that this orthogonality is regarding $M$ symbols with
$N$ subcarriers, and therefore it only applies to a part of TF plane. Since an
MC modulation has a limited number of symbols and subcarriers, the
orthogonality within this signal bandwidth and duration is sufficient for an
MC modulation. As a result, we can reformulate its pulse design problem, and
introduce a concept of local orthogonality.

###  IV-A Global and local (bi)orthogonality

Analogous to (2) and (3), the (bi)orthogonal pulse design problem taking the
limited number of symbols and subcarriers into account is to find WH subsets
$\left(g,\mathcal{T},\mathcal{F},M,N\right)$ and
$\left(\gamma,\mathcal{T},\mathcal{F},M,N\right)$ that satisfy the orthogonal
condition of

$\langle
g_{m,n},g_{\dot{m},\dot{n}}\rangle=\delta(m-\dot{m})\delta(n-\dot{n}),\,\,m,\dot{m}\in\mathbb{Z}_{m},n,\dot{n}\in\mathbb{Z}_{N},$
(9)

or the biorthogonal condition of

$\langle
g_{m,n},\gamma_{\dot{m},\dot{n}}\rangle=\delta(m-\dot{m})\delta(n-\dot{n}),\,\,m,\dot{m}\in\mathbb{Z}_{m},n,\dot{n}\in\mathbb{Z}_{N},$
(10)

where

$\displaystyle\mathbb{Z}_{M}=\\{0,1,\cdots,M-1\\},\,\,\mathbb{Z}_{N}=\\{0,1,\cdots,N-1\\}.$
(11)

We call (9) and (10) the local orthogonal condition and local biorthogonal
condition, respectively. Because of

$\displaystyle\langle
g_{m,n},g_{\dot{m},\dot{n}}\rangle=\mathcal{A}_{g,g}(\bar{m}\mathcal{T},\bar{n}\mathcal{F})e^{j2\pi
n\bar{m}\mathcal{F}\mathcal{T}},$ (12)

where $\bar{m}=\dot{m}-m$ and $\bar{n}=\dot{n}-n$, the local orthogonal
condition in (9) is equivalent to

$\displaystyle\mathcal{A}_{g,g}(\bar{m}\mathcal{T},\bar{n}\mathcal{F})=\delta(\bar{m})\delta(\bar{n}),$
(13)

for $|\bar{m}|\leq M-1,|\bar{n}|\leq N-1$. Similar result can be obtained for
the local biorthogonal condition in (10).

It is noteworthy that the WH frame theory based results regarding
(bi)orthogonal WH sets are rigorously correct. Since the WH set is a time-
frequency analysis tool for functions in $L^{2}(\mathbb{R})$, it considers the
whole TF plane where $m,n\in\mathbb{Z}$, and corresponds to the signal without
the limitation of bandwidth and duration. To make this possible, given
$\mathcal{T}$ and $\mathcal{F}$, $g(t)$ is independent of the number of
symbols $M$ and the number of subcarriers $N$, to be shifted over the whole TF
plane. In other words, to achieve the global (bi)orthogonality in (2) and (3),
$g(t)$ is parameterized only by $\mathcal{T}$ and/or $\mathcal{F}$.

On the other hand, for MC modulation, a WH subset that satisfies the local
(bi)orthogonality in (9) and (10) is sufficient. Obviously, $g(t)$ that
achieves the global (bi)orthogonality can form a such WH subset. However, what
we really need is just a WH subset, and it is not necessarily bounded by the
WH frame theory for the WH set. In fact, the pulses parameterized by not only
$\mathcal{T}$ and $\mathcal{F}$ but also $M$ and $N$, can achieve the local
orthogonality. An example is the DDOP in (7).

###  IV-B Orthogonality with respect to $\mathcal{F}$

Let us consider a fixed $m$ in $g_{m,n}$, and investigate the orthogonality
with respect to the frequency resolution $\mathcal{F}$. We want to find $g(t)$
that can achieve the orthogonality among $g(t-m\mathcal{T})e^{j2\pi
n\mathcal{F}(t-m\mathcal{T})}$ with a given $m$ but variable $n$, where $0\leq
t\leq T_{g}$ and $T_{g}=\mathfrak{T}=1/\mathcal{F}$. Without loss of
generality, let $m=0$. We can obtain the following results:

  1. (F1)

Unbounded $n$ ($n\in\mathbb{Z}$): $g(t)$ is the rectangular pulse
$\Pi_{\mathfrak{T}}(t)$, which is independent of $N$.

  2. (F2)

Bounded $n$ ($|n|\leq N-1$): We have the following lemma:

######  Lemma 1.

When $g(t)$ is a periodic function with period $\frac{\mathfrak{T}}{N}$ for
$0\leq t\leq T_{g}$ and $T_{g}=\mathfrak{T}$, it satisfies the orthogonal
property that

$\mathcal{A}_{g,g}\left(0,n\mathcal{F}\right)=\delta(n),$ (14)

for $|n|\leq N-1$.

###### Proof:

Since the period of $g(t)$ is $\frac{\mathfrak{T}}{N}$, $g(t)$ can be written
as

$\displaystyle
g(t)=g\left(t+\dot{n}\frac{\mathfrak{T}}{N}\right),\,\,\,0\leq\dot{n}\leq
N-1.$ (15)

for $0\leq t<\frac{\mathfrak{T}}{N}$. Then, bearing in mind that
$\mathfrak{T}=1/\mathcal{F}$, we have

$\displaystyle\mathcal{A}_{g,g}(0,n\mathcal{F})$
$\displaystyle=\int_{0}^{T_{g}}g(t)g^{*}(t)e^{-j2\pi n\mathcal{F}t}dt,$
$\displaystyle=\sum_{\dot{n}=0}^{N-1}\int_{\dot{n}\frac{\mathfrak{T}}{N}}^{(\dot{n}+1)\frac{\mathfrak{T}}{N}}g(t)g^{*}(t)e^{-j2\pi
n\mathcal{F}t}dt,$
$\displaystyle=\sum_{\dot{n}=0}^{N-1}e^{-j2\pi\frac{n\dot{n}}{N}}\int_{0}^{\frac{\mathfrak{T}}{N}}g(t)g^{*}(t)e^{-j2\pi
n\mathcal{F}t}dt,$ $\displaystyle=\delta(n),$ (16)

for $|n|\leq N-1$, which completes the proof. ∎

Lemma 1 indicates that once there is a constraint imposed on the number of
subcarriers, there are _infinite_ pulses that can satisfy the orthogonality
with respect to $\mathcal{F}$. In particular, _regardless of $B_{g}$_, $g(t)$
can achieve the orthogonality among $N$ subcarriers with a subcarrier spacing
$\mathcal{F}$, as long as it is an aforementioned periodic function. An
example of such $g(t)$ for $N=4$ is shown in Fig. 2.

Figure 2: $g(t)$ orthogonal w.r.t $\mathcal{F}=\frac{1}{\mathfrak{T}}$ for
$|n|\leq N-1$ and fixed $m$.

It is noteworthy that in contrast to (F1) where $B_{g}$ is proportional to
$\mathcal{F}$, (F2) _decouples_ $B_{g}$ and $\mathcal{F}$, and consequently
allows pulses with much wider bandwidth to achieve orthogonality among $N$
subcarriers. On the other hand, to avoid the intersymbol interference (ISI)
and achieve the orthogonality among MC symbols time-multiplexed by
$\mathcal{T}$, we need $B_{g}$ to be comparable to $\frac{1}{\mathcal{T}}$.
The decoupling of $\mathcal{F}$ and $B_{g}$ in (F2) actually paves a way to
design orthogonal pulse with respect to _independent TF resolutions_.

###  IV-C Orthogonality with respect to $\mathcal{T}$

Similarly, we can consider a fixed $n$ in $g_{m,n}$, and investigate the
orthogonality with respect to the time resolution $\mathcal{T}$. Our target
now is to find $g(t)$ that can achieve the orthogonality among
$g(t-m\mathcal{T})e^{j2\pi n\mathcal{F}(t-m\mathcal{T})}$ with a fixed $n$ but
different $m$. When $n\neq 0$, we have the following straightforward answer
with _isolated_ pulses/sub-pulses:

  1. (T1)

Unbounded $m$ ($m\in\mathbb{Z}$) : $g(t)$ can be any function with duration
$T_{g}\leq\mathcal{T}$, which is independent of $M$.

  2. (T2)

Bounded $m$ ($|m|\leq M-1$) : $g(t)$ consists of $\dot{N}>1$ sub-pulses
$b_{\dot{n}}(t),0\leq\dot{n}\leq N-1$, where these sub-pulses are temporally
spaced by $M\mathcal{T}$ and each sub-pulse has a duration of
$T_{b_{\dot{n}}}\leq\mathcal{T}$.

Meanwhile, when $n=0$, we have another answer with _overlapped_ pulse/sub-
pulses:

  1. (T3)

Unbounded $m$ ($m\in\mathbb{Z}$) : SRN pulse for symbol interval
$\mathcal{T}$, which is also independent of $M$.

  2. (T4)

Bounded $m$ ($|m|\leq M-1$) : $g(t)$ consists of $\dot{N}>1$ SRN sub-pulses
for symbol interval $\mathcal{T}$, where these sub-pulses are temporally
spaced by $M\mathcal{T}$. The SRN sub-pulse can have any duration.

It is interesting to note that $g(t)$ in (T4) actually can form a periodic
function that satisfies (F2), when $\dot{N}$ is large enough.

Figure 3: $u_{c}(t)$ for $D=1$.

##  V General DDOP design

Recall that the orthogonal property of the DDOP in (8) is subject to a
duration constraint of SRN sub-pulse, given by $T_{a}\ll T$. In practice, it
is desirable to relax such constraint to enable flexible design. In this
section, we propose a general DDOP design, where the SRN sub-pulse’s duration
constraint is released.

Let $\dot{N}=N$ and $\mathcal{T}=\frac{T}{M}$, $g(t)$ in (T4) becomes the DDOP
$u(t)$ in (8), except the unbounded $T_{a}$. From (F2), we know that for the
frequency resolution $\mathcal{F}=\frac{1}{NT}$, the key to achieve the
orthogonality among $N$ subcarriers is to form a periodic function with period
$\frac{1}{N\mathcal{F}}=T$. This observation inspires us to use $u_{c}(t)$, a
cyclically extended version of $u(t)$, as the transmit pulse, while the
receive pulse is still $u(t)$. Furthermore, because
$\mathcal{A}_{u_{c},u}(m\frac{T}{M},n\frac{1}{NT})$ is calculated between
$u_{c}(t)$ and $u(t-m\frac{T}{M})e^{j2\pi\frac{n}{NT}(t-m\frac{T}{M})}$, the
problem becomes how can we let $u_{c}(t)$ have the specified periodicity
within the range of
$u(t-m\frac{T}{M})e^{j2\pi\frac{\bar{n}}{NT}(t-m\frac{T}{M})}$ for $|m|\leq
M-1$. We have the following lemma:

######  Lemma 2.

Let $u(t)$ consist of $N$ SRN pulses $a_{T/M,N}(t)$ temporally spaced by $T$,
it satisfies the orthogonal property that

$\mathcal{A}_{u_{c},u}\left(m\frac{T}{M},n\frac{1}{NT}\right)=\delta(m)\delta(n),$
(17)

for $|m|\leq M-1$ and $|n|\leq N-1$, where $u_{c}(t)$ is a cyclically extended
version of $u(t)$ that is a periodic function with period $T$ during
$-(M-1)\frac{T}{M}\leq t\leq(MN-1)\frac{T}{M}+T_{a}$.

###### Proof:

Let us first check the periodicity of $u_{c}(t)$ within the range of
$-(M-1)\frac{T}{M}\leq t\leq(MN-1)\frac{T}{M}+T_{a}$, which correspond to the
start of the first sub-pulse of $u(t+(M-1)\frac{T}{M}$ and the end of the last
sub-pulse of $u(t-(M-1)\frac{T}{M})$, respectively. From (7), we can divide
$u(t)$ into $N$ segments, where $u(t)=\sum_{n=0}^{N-1}u_{n}(t)$ and the $n$th
segment is given by $u_{n}(t)=u(t)$ for $nT\leq t<(n+1)T$.

Figure 4: $u_{c}(t)$ for $D=2$.

Let $D=\lceil T_{a}/T\rceil$. If $D=1$, we have $u_{n}(t)=a(t-nT),$ which
implies that the periodicity within $-(M-1)\frac{T}{M}\leq
t\leq(MN-1)\frac{T}{M}+T_{a}$ can be obtained by cyclically extending $u(t)$
to $u_{c}(t)=\sum_{n=-1}^{N}a(t-nT)$. Similarly, when $D>1$, the periodicity
can be obtained by further extending to

$\displaystyle u_{c}(t)=\sum_{n=-D}^{N-1+D}a(t-nT).$ (18)

Two examples of $u_{c}(t)$ with $D=1,2$ are shown in Fig. 3 and Fig. 4,
respectively, where the first sub-pulse of $u(t+(M-1)\frac{T}{M})$ and the
last sub-pulse of $u(t-(M-1)\frac{T}{M})$ are also plotted with dashed lines.

Next, let us verify the ambiguity functions. Due to the aforementioned
periodicity of $u_{c}(t)$, we have

$\displaystyle u_{c}(t)=u_{c}(t+\dot{n}T),\,\,\,0\leq\dot{n}\leq N-1,$ (19)

for $m\frac{T}{M}\leq t\leq m\frac{T}{M}+T_{u}$, where $|m|\leq M-1$ and
$T_{u}=(N-1)T+T_{a}$. Then, using (19), the ambiguity function between
$u_{c}(t)$ and $u(t)$ for $|n|\leq N-1$ and $|m|\leq M-1$ can be calculated
similarly to (16), and given by

$\displaystyle\mathcal{A}_{u_{c},u}(m\frac{T}{M},n\frac{1}{NT})$
$\displaystyle=\int_{m\frac{T}{M}}^{m\frac{T}{M}+T_{u}}u_{c}(t)u^{*}(t-m\frac{T}{M})e^{-j2\pi
n\frac{1}{NT}(t-m\frac{T}{M})}dt,$ $\displaystyle=\delta(n)\delta(m).$ (20)

(20) completes the proof. ∎

Lemma 2 indicates that the constraint of $T_{a}$ in $u(t)$ can be removed.
Once the appropriate CP and CS are added in accordance with (18), the desired
local orthogonality can be achieved as well. As a result, generally the
transmit pulse of ODDM modulation is $u_{c}(t)$, where the extension parameter
$D=\lceil T_{a}/T\rceil=\lceil 2Q/M\rceil$. When $M\gg 2Q$, we have
$2Q/M\approx 0$. Then, as proved in [10], the ODDM can just employ the DDOP
$u(t)$ without cyclic extension ($D=0$).

##  VI TF signal localization and numerical results

###  VI-A Frequency domain representation of DDOP

The frequency domain representation plays an important role in the analysis of
pulse. In the following, we will derive $U(f)$, the frequency domain
representation of $u(t)$.

It is well-known that the frequency domain representation of an impulse train
$\dot{u}(t)=\sum_{n=-\infty}^{\infty}\delta(t-nT),$ is a Fourier series and
also can be written as an impulse train in frequency domain
$\dot{U}(f)=\frac{1}{T}\sum_{n=-\infty}^{\infty}\delta(f-\frac{n}{T}).$ It is
interesting to observe that the DDOP can be obtained from $\dot{u}(t)$ by
applying a rectangular window
$\operatorname{\Pi}_{NT}\left(t+\frac{T}{2}\right)$ followed by a $a(t)$ based
filtering. Then, we have

$\displaystyle
u\left(t+\frac{T_{a}}{2}\right)=\left(\dot{u}(t)\times\operatorname{\Pi}_{NT}\left(t+\frac{T}{2}\right)\right)\star
a(t),$ (21)

where $\star$ denotes the convolution. Since the multiplication and
convolution in time domain correspond to the convolution and multiplication in
frequency domain, respectively, we have

$\displaystyle U(f)$ $\displaystyle=e^{-j2\pi
f\frac{T_{a}}{2}}A(f)\left(\dot{U}(f)\star e^{-j2\pi
f\frac{(N-1)T}{2}}\operatorname{Sinc}(fNT)\right),$
$\displaystyle=\frac{e^{-j2\pi
f\tilde{T}}}{T}A(f)\sum_{n=-\infty}^{\infty}e^{j2\pi\frac{n(N-1)}{2}}\operatorname{Sinc}(fNT-
nN),$ (22)

where $\tilde{T}=(T_{a}+(N-1)T)/2$ and $A(f)$ is the Fourier transform of
$a(t)$. Without loss of generality, let $M$ be an even number. Then, the shape
of $|U(f)|$ in plotted in Fig. 5, where the shape of
$|\operatorname{Sinc}(fNT-nN)|$ is truncated for the purpose of display. Now,
it becomes clear that $\operatorname{Sinc}(fNT-nN)$ and $A(f)$ correspond to
the orthogonality with respect to $\mathcal{F}=\frac{1}{NT}$ and
$\mathcal{T}=\frac{T}{M}$, respectively.

Figure 5: $|U(f)|$.

###  VI-B TF signal localization comparison

Figure 6: TF signal localization comparison of modulation waveforms.

For the TF region bounded by the sampling rate and duration of
$W_{0}=\frac{M}{T}$ and $T_{0}=NT$, the corresponding degrees of freedom (DoF)
of the signal is around $W_{0}T_{0}=MN$. Then, an MC modulation scheme employs
$MN$ orthogonal pulses corresponding to its TF resolutions to transmit $MN$
digital symbols, resulting in its own TF localization structure.

With $u(t)$ in (7) and $U(f)$ in (22), like that of OFDM in [7], the TF signal
localization structure of ODDM modulation can be schematically illustrated in
Fig. 6, where those of other modulation waveforms are also given for
comparison. It can be observed that :

  1. 1)

For SC modulation, which is a time-division multiplexing (TDM) scheme, the
$MN$ digital symbols are conveyed by $MN$ SRN pulses for symbol interval
$\frac{T}{M}$. The pulses are overlapped only in time domain.

  2. 2)

For frequency-division multiplexing (FDM) scheme, an example is the OFDM
modulation with frequency resolution $\frac{1}{NT}$, where $MN$ digital
symbols are conveyed by $MN$ rectangular pulses $\operatorname{\Pi}_{NT}(t)$
modulated by $MN$ subcarriers, respectively. The pulses are overlapped only in
frequency domain.

  3. 3)

For the conventional OFDM modulation with frequency resolution $\frac{1}{T}$
and time resolution $T$, $MN$ digital symbols are conveyed by $N$ OFDM
symbols, where each OFDM symbols has $M$ rectangular pulses
$\operatorname{\Pi}_{T}(t)$ modulated by $M$ subcarriers, respectively. Since
$N$ OFDM symbols are isolated in time domain, these pulses also are overlapped
only in frequency domain.

  4. 4)

For ODDM modulation with frequency resolution $\frac{1}{NT}$ and time
resolution $\frac{T}{M}$, $MN$ digital symbols are conveyed by $M$ pulse
trains $u(t)$ modulated by $N$ subcarriers, respectively. These pulses are
overlapped in both time and frequency domains to achieve the local
orthogonality with respect to $\frac{T}{M}$ and $\frac{1}{NT}$.

###  VI-C Numerical results

Now, we present the numerical results for the ambiguity function of the DDOP.
A three-dimensional plot of the ambiguity function in (17) is shown in Fig. 7,
where $\mathcal{F}=\frac{1}{NT}$, $\mathcal{T}=\frac{T}{M}$ with $M=32$,
$N=8$. $a(t)$ is a root raised cosine pulse with roll-off factor $\rho=0.1$
and $Q=20$. Because $D=2$ for this parameter setting, we adopt the general
DDOP design. The corresponding 2D plot of
$\left|\mathcal{A}_{u_{c},u}\left(m\frac{T}{M},n\frac{1}{NT}\right)\right|$
with $n=0$ is also given in Fig. 8. One can see that with appropriate CP and
CS, the DDOP can achieve the local orthogonality within $|m|\leq M-1$ and
$|n|\leq N-1$. For $|m|\geq M$ or $|n|\geq N$, the ambiguity function repeats
with time period $T$ and frequency period $\frac{1}{T}$, if we further extend
the CP and CS.

The elegant TF localization of ODDM schemes shown in Fig. 6 demonstrates that
every information symbol is evenly distributed over its TF region. Thus, it is
flexible for allocating TF resources for multi-user communications system
design. In addition, the perfect local orthogonality of the DDOP’s ambiguity
function with respect to DD resolutions, shown in Figs. 7 and 8, can be
exploited for design integrated sensing and communication (ISAC) systems. We
will investigate these topics in our future work.

Figure 7:
$\left|\mathcal{A}_{u_{c},u}\left(m\frac{T}{M},n\frac{1}{NT}\right)\right|$,
$M=32$ and $N=8$. Figure 8:
$\left|\mathcal{A}_{u_{c},u}\left(m\frac{T}{M},n\frac{1}{NT}\right)\right|$,
$M=32$ and $N=8$, $n=0$.

##  VII Conclusion

In this paper, the recently discovered DDOP is analyzed in terms of local
orthogonality, frequency-domain representation and ambiguity function. We
clarified the DDOP’s local orthogonality and justified its existence as a WH
subset, without violating the WH frame theory which governs the global
orthogonality corresponding to the WH set. Several sufficient conditions for
locally-orthogonal pulses were presented, and a general DDOP design was
proposed by introducing CP and CS to the DDOP. We derived the DDOP’s frequency
domain representation, and compared the DDOP-based ODDM modulation with other
modulation schemes, in terms of TF signal localization. We demonstrated the
perfect local orthogonality of DDOP with respect to DD resolutions by its
ambiguity function.

## References

  * [1] J. M. Wozencraft and I. M. Jacobs, _Principles of Communication Engineering_. Wiley, 1965.
  * [2] I. Daubechies, “The wavelet transform, time-frequency localization and signal analysis,” _IEEE Trans. Inf. Theory_ , vol. 36, no. 5, pp. 961–1005, 1990\.
  * [3] H. G. Feichtinger and T. Strohmer, Eds., _Gabor Analysis and Algorithms: Theory and Applications_. Birkhäuser, Boston, MA, 1998.
  * [4] K. Gröchenig, _Foundations of Time-Frequency Analysis_. Birkhäuser, Boston, MA, 2001.
  * [5] J. Wexler and S. Raz, “Discrete Gabor expansions,” _Signal Process._ , vol. 21, no. 3, pp. 207–220, 1990.
  * [6] A. Janssen, “Duality and biorthogonality for Weyl-Heisenberg frames,” _J. Fourier Anal. Applicat._ , vol. 1, no. 4, pp. 403–436, 1995.
  * [7] G. Matz, H. Bolcskei, and F. Hlawatsch, “Time-frequency foundations of communications: Concepts and tools,” _IEEE Signal Process. Mag._ , vol. 30, no. 6, pp. 87–96, 2013.
  * [8] A. Sahin, I. Guvenc, and H. Arslan, “A survey on multicarrier communications: Prototype filters, lattice structures, and implementation aspects,” _IEEE Commun. Surveys Tuts._ , vol. 16, no. 3, pp. 1312–1338, 2014\.
  * [9] H. Lin and J. Yuan, “Multicarrier modulation on delay-Doppler plane: Achieving orthogonality with fine resolutions,” in _Proc. of IEEE ICC_ , 2022, pp. 1–6.
  * [10] H. Lin and J. Yuan, “Orthogonal delay-Doppler division multiplexing modulation,” _IEEE Trans. Wireless Commun._ , 2022, to appear.
  * [11] D. Slepian, “On bandwidth,” _Proc. IEEE_ , vol. 64, no. 3, pp. 292–300, 1976\.
  * [12] R. Hass and J.-C. Belfiore, “A time-frequency well-localized pulse for multiple carrier transmisson,” _Wireless Personal Commun._ , vol. 5, no. 1, pp. 1–18, 1997.
  * [13] B. Le Floch, M. Alard, and C. Berrou, “Coded orthogonal frequency division multiplex,” _Proc. IEEE_ , vol. 83, no. 6, pp. 982–996, 1995.
  * [14] W. Kozek and A. Molisch, “Nonorthogonal pulseshapes for multicarrier communications in doubly dispersive channels,” _IEEE J. Sel. Areas Commun._ , vol. 16, no. 8, pp. 1579–1589, 1998.
  * [15] T. Strohmer and S. Beaver, “Optimal OFDM design for time-frequency dispersive channels,” _IEEE Trans. Commun._ , vol. 51, no. 7, pp. 1111–1122, 2003.
  * [16] I. Daubechies, _Ten Lectures on Wavelets_. SIAM, 1992.
  * [17] P. Bello, “Characterization of randomly time-variant linear channels,” _IEEE Trans. Commun. Syst._ , vol. 11, no. 4, pp. 360–393, 1963.


# An Efficient Temporary Deepfake Location Approach Based Embeddings for
Partially Spoofed Audio Detection

###### Abstract

Partially spoofed audio detection is a challenging task, lying in the need to
accurately locate the authenticity of audio at the frame level. To address
this issue, we propose a fine-grained partially spoofed audio detection
method, namely Temporal Deepfake Location (TDL), which can effectively capture
information of both features and locations. Specifically, our approach
involves two novel parts: embedding similarity module and temporal convolution
operation. To enhance the identification between the real and fake features,
the embedding similarity module is designed to generate an embedding space
that can separate the real frames from fake frames. To effectively concentrate
on the position information, temporal convolution operation is proposed to
calculate the frame-specific similarities among neighboring frames, and
dynamically select informative neighbors to convolution. Extensive experiments
show that our method outperform baseline models in ASVspoof2019 Partial Spoof
dataset and demonstrate superior performance even in the cross-dataset
scenario. The code is released online111https://github.com/xieyuankun/TDL-ADD.

Index Terms—  partially spoofed audio detection, temporal deepfake location,
embedding learning.

##  1 Introduction

AI generated content (AIGC) technology has witnessed swift progress in recent
years, particularly in speech-related applications like text-to-speech (TTS)
[1, 2, 3] and voice conversion (VC) [4, 5, 6]. Although these technologies
have brought about convenience, they have also posed significant security
threats. Thus, various initiatives and challenges, such as ASVspoof [7, 8],
have been established to foster research on countermeasure solutions that
safeguard speech applications and human listeners against spoofing attacks.
Nevertheless, a significant scenario has been overlooked in most datasets and
challenges where a bonafide speech utterance is contaminated by synthesized
speech segments, leading to partial spoofing (PS). Attackers can use PS to
alter sentence semantics, and such modifications can be easily accomplished at
low cost. For instance, attackers can easily modify single word such as time,
place, and characters in sentence to dramatically change the semantics.
Furthermore, If attackers have knowledge of phonology, they can manipulate
vowels and even consonants such as “pan,”“pin,”“pen,” which are smaller than
the word level. Therefore, defending against such fine-grained PS scenarios
poses significant challenges for defenders.

In recent years, there are several studies about PS scenarios for Audio
Deepfake Detection (ADD). Yi et al. [9] create a dataset that focuses on
changing a few words in an utterance for half-truth audio detection. At the
same time, Zhang et al. [10] construct a speech database called ‘PartialSpoof’
designed for PS scenarios. The above two datasets are the beginning of the
research for PS scenario in ADD task. Afterward, Zhang et al. [11] propose the
SELCNN network to enhance the ability of the accuracy of the utterance. Lv et
al. [12] use Wav2Vec2 (W2V2) [13] as front-end, ECAPA-TDNN [14] as back-end
achieving the first rank in ADD 2022 Track 2[15]. Although the above research
shows effectiveness at the utterance level detection in PS, they do not
pinpoint specific segments with precision. Thus, Zhang et al. [16] extended
the previous utterance-level PS dataset labels to frame-level and proposed
corresponding W2V2-based countermeasures to enhance frame-level detection
capability.

Fig. 1: The entire structure of our proposed Temporal Deepfake Location (TDL)
method.

The aforementioned methods solely utilize existing ADD models such as LCNN,
currently lacking specific approaches tailored to the PS scenario,
particularly in terms of precise frame-level localization. To address this
challenge, we propose a novel Temporal Deepfake Location (TDL) method. For
front-end, we take advantage of W2V2 [17]. By training on a vast corpus of
genuine speech from diverse source domains, W2V2 can effectively discriminate
the real and fake in complex acoustic scenarios. For back-end, our primary
focus is on fine-grained locating the genuine and spoofed speech segment. To
clearly distinguish the real and fake in feature level, we first design the
embedding similarity module to separate the real and fake frames in embedding
space and get a high-quality embedding similarity vector. Then, we propose
temporal convolution operation to locate the region from the embedding vector.
The local similarity for each temporal position is calculated from the
embedding. By this means, we can obtain a frame-specific weight to guide the
convolution making a temporal sensitive calculation. Our main contributions
can be summarized as follows:

  * •

We propose TDL method, an efficient and effective ADD method for PS scenarios
which combines a embedding similarity module and temporal convolution
operation to effectively capture both feature and positional information.

  * •

The proposed method outperforms baseline models in ASV spoof 2019PS dataset
and demonstrate superior performance even in cross-dataset experiments.

##  2 Proposed Method

###  2.1 Problem statement and overview

In PS scenarios, the fake audio segment is inserted within the genuine speech.
Our target is to detect the real and fake segments at frame level. Given the
large-scale self-supervised audio feature $f=(f_{1},f_{2},...f_{T})\in
R^{D\times T}$, where $D$ and $T$ denote the dimension of audio feature and
the number of frames respectively. The whole task is defined as input feature
$f$ and output the frame level label
$y=(y_{1},y_{2},...y_{T})\in\\{0,1\\}^{T}$, where 1 represents the real frames
and 0 represents the fake frames.

The framework of our proposed TDL is depicted in Figure 1. First, we utilize
Wav2Vec-XLS-R to extract the frame level feature from the raw audio. Then, for
enhanced identification of genuine and fake distinctions at the embedding
level, we devise an embedding similarity module to segregate authentic and
synthetic frames within the embedding space. Next, to capture the position
information, we adopt temporal convolution operation by focusing on frame-
specific similarities among neighboring frames. Finally, we employ 1D
convolutional layers and fully connected layers for downsampling to the frame
level label to compute the Binary Cross-Entropy (BCE).

###  2.2 W2V2 front-end

W2V2 based front-end is trained by solving a contrastive task over a masked
feature encoder. Firstly, speech signals in various lengths are passed through
a feature extractor consisting of seven convolutional neural network (CNN)
layers. Subsequently, context representations are obtained using a Transformer
network [18] comprising of 24 layers, 16 attention heads, and an embedding
size of 1024. In practice, we utilize the Hugging Face version of
wav2vec2-XLS-R-300M222https://huggingface.co/facebook/wav2vec2-xls-r-300m and
freeze the weights of the front-end. The front-end model is pre-trained with
436k hours of unannotated genuine speech data in 128 languages. Consequently,
the last hidden states from the transformer can effectively represent the
contextualized information of genuine speech which is different from the
partially fake speech.

###  2.3 Embedding similarity module

To better capture feature-level information, we first distinguish the real and
fake frames in the embedding space. Specifically, the W2V2 features are fed
into a CONV module, consisting of two sequential 1D-CNNs, which downsamples
the embedding dimension from 1024 to 32. The embedding vector is
$L2$-normalized. Then we get a embedding vector $e=(e_{1},e_{2},...e_{T})\in
R^{D\times T}$. In the embedding similarity module, we utilize cosine
similarity to measure the similarity of two embedding vector $e_{u}$ and
$e_{v}$ as follows:

$\mathcal{S}\left(\mathbf{e}_{u},\mathbf{e}_{v}\right)=\frac{\mathbf{e}_{u}^{T}\cdot\mathbf{e}_{v}}{\left\|\mathbf{e}_{u}\right\|_{2}\cdot\left\|\mathbf{e}_{v}\right\|_{2}}.$
(1)

To increase the distance between genuine and fake frames in the embedding
space and improve generalizability, we computed the cosine similarities
between genuine frames, between fake frames, and between genuine and fake
frames. Specifically, we ensured that genuine frames from different positions
exhibited similarity, fake frames from different positions exhibited
similarity, while genuine and fake frames are dissimilar to each other.

Thus, $\mathcal{L}_{ESM}^{Real}$ and $\mathcal{L}_{ESM}^{Fake}$ are proposed
to make the real frames and fake frames in different positions similar:

$\mathcal{L}_{\mathrm{ESM}}^{\mathrm{Real}}=\max_{\forall e_{x},e_{y},x\neq
y}\left\lfloor\tau_{\text{same
}}-\mathcal{S}\left(\mathbf{e}_{x},\mathbf{e}_{y}\right)\right\rfloor_{+},$
(2) $\mathcal{L}_{\mathrm{ESM}}^{\mathrm{Fake}}=\max_{\forall
e_{m},e_{n},m\neq n}\left\lfloor\tau_{\text{same
}}-\mathcal{S}\left(\mathbf{e}_{m},\mathbf{e}_{n}\right)\right\rfloor_{+},$
(3)

where $e_{x}$ and $e_{y}$ refer to distinct positions of real frames, while
$e_{m}$ and $e_{n}$ refer to those of fake frames. $\tau_{\text{same }}$ is
the similarity threshold between frames from the same category,
$\left\lfloor\\\ \dots\right\rfloor_{+}$ represents clipping below at zero. It
is noteworthy that although we know the positions of frame-level authenticity
labels, the temporal dimension of W2V2-XLS-R features does not inherently
align with these frame-level labels. To tackle this issue, we ascertain the
temporal authenticity in the time dimension of the embedding vector by
calculating the ratio between the temporal dimensions of the label and the
embedding vector.

$\mathcal{L}_{ESM}^{Diff}$ is proposed to separate the real and fake frames,
which can be formulated as:

$\mathcal{L}_{\mathrm{ESM}}^{\mathrm{Diff}}=\max_{\forall
e_{r},e_{f}}\left\lfloor\mathcal{S}\left(\mathbf{e}_{r},\mathbf{e}_{f}\right)-\tau_{\text{Diff
}}\right\rfloor_{+},$ (4)

where $e_{r}$ and $e_{f}$ refer to the embedding vector of real frames and
fake frames. $\tau_{\text{diff }}$ is the similarity threshold to constraint
the distance between real and fake frames. Finally, the embedding similarity
module is optimized by $\mathcal{L}_{ESM}$, which takes into account the three
aforementioned losses in a joint manner. The $\mathcal{L}_{ESM}$ is calculated
as follows:

$\mathcal{L}_{ESM}=\mathcal{L}_{ESM}^{Real}+\mathcal{L}_{ESM}^{Fake}+\mathcal{L}_{ESM}^{Diff}.$
(5)

###  2.4 Temporal convolution operation

To effectively capture the positional information, we use the embedding vector
as an local attention mask to perform temporal convolution operations.
Consider a audio feature ${\mathbf{X}}\in R^{D_{in}\times T}$, where $D_{in}$
and $T$ represent the dimension of the vector and number of frames
respectively. The temporal convolution layer learns a dynamic convolution
kernel $\Bbbk\in R^{k\times D_{in}\times D_{out}}$, where $k$ is the size of
temporal kernel, $D_{out}$ is the dimension of output feature. We only utilize
the dynamic kernel $\Bbbk^{m}\in R^{k\times D_{in}}$ to compute $m^{th}$
channel of the output for convenient. Thus, the temporal convolution operation
for the $t^{th}$ feature can be expressed as:

$f_{t}^{m}=\sum_{i=0}^{k-1}\mathcal{\Bbbk}^{m}[i,:]\cdot\overline{\mathbf{X}}\left[:,t-\frac{k}{2}+i\right],$
(6)

where $f_{t}^{m}$ is the value in the $m^{th}$ channel of output feature
vector, $[\cdots]$ means a slice of a matrix, $(\cdot)$ denotes the inner
product. $\overline{\mathbf{X}}$ is the modulated feature processed by
neighbor similarity calculation:

$\displaystyle\overline{\mathbf{X}}\left[:,t-\frac{k}{2}+i\right]=\mathbf{X}\left[:,t-\frac{k}{2}+i\right]\times\mathbf{a}[i,t],$
(7) $\displaystyle i\in[0,\ldots,k-1],$

where matrix $\mathbf{a}\in R^{k\times T}$ is a similarity matrix that
calculate the local similarity for each temporal position, $\mathbf{a}[i,t]$
indicates the similarity between the $t^{th}$ feature vector and its $k$
neighbors.

In practice, we determine the dynamic kernel weight based on the embedding
vector generated by ESM module. We apply temporal convolution operation to the
W2V2 features on two sequence 1D-CNNs, where both input channel and output
channel remain unchanged to maintain consistency in temporal dimension.

###  2.5 Total loss

Following two consecutive temporal convolution operation layers, to capture
additional temporal information and align with the label dimensions, we
subsequently employ 1D-CNN, fully connected (FC) layers, and sigmoid
activation functions to calculate the BCE loss. The architecture details of
TDL is shown in Table 1. The total loss is defined as follow:

$\mathcal{L}_{all}=\mathcal{L}_{BCE}+\lambda\mathcal{L}_{ESM},$ (8)

where $\lambda$ is set to 0.1 to balance the value of two losses.

Table 1: Architecture of TDL network. module | kernel/stride | output shape
---|---|---
W2V2 | - | (batch,1024,1050)
CONV | 3/1 | (batch,512,1050)
3/1 | (batch,32,1050)
TCONV | 3/1 | (batch,1024,1050)
TCONV | 3/1 | (batch,1024,1050)
CONV | 1/1 | (batch,2,1050)
Flatten/FC | - | (batch,132)

##  3 Experiments

###  3.1 Database

Our experiments for PS scenario include two public datasets: ASVspoof2019PS
(19PS) [10] and LAV-DF [19]. 19PS is constructed based on the ASVspoof2019 LA
database [20]. All experiments on the 19PS dataset are conducted using 160ms
resolution labels. The training, validation, and testing sets are distributed
according to the original dataset allocation, consisting of 25,380, 24,844,
and 71,237 utterances respectively.

To evaluate the model’s generalizability, we conduct additional testing of the
19PS-trained model using the LAV-DF test set. LAV-DF represents a multi-modal
temporal forgery dataset, containing a total of 26,100 videos in test set. We
extract the audio track of each video and create 160ms frame level genuine and
fake labels.

We calculated the percentage of samples belonging to fake class at both the
frame and sentence levels, as shown in the Table 2. We can observe that the
frame-level labels in 19PS are balanced, facilitating model training. However,
the LAV-DF dataset exhibits a lower proportion of spoof segments, making it
unbalanced and presenting greater challenges for detection.

Table 2: Percentages(%) of fake class in each dataset. dataset | subset | frame-level | utterance-level
---|---|---|---
19PS | train | 53.00 | 89.83
19PS | dev | 52.31 | 89.74
19PS | test | 48.03 | 89.68
LAV-DF | test | 10.01 | 48.82

###  3.2 Implementation details

In order to address the issue of variable-length audio inputs, we employ the
technique of zero-padding to the maximum length of training set. For the frame
of genuine speech, we set the label to one, while for spoofing frame, the
label is set to zero. In the case of 19PS, the maximum duration of speech in
the training set is 21.03 seconds with a W2V2 feature dimension of (1050,1024)
and the number of frames at a resolution of 160 ms is 132. For LFCC, we
extracted 60-dimensional LFCC with a combination of static, delta and delta-
delta coefficients.

For training strategy, the Adam optimizer is adopted with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, $\varepsilon$ = $10^{-9}$ and weight decay is $10^{-4}$. We
train all of the models for 100 epochs. The learning rate is initialized as
$10^{-5}$ and halved every 5 epochs. It is worth mention that no data
augmentation method is used for experiment.

###  3.3 Evaluation metrics

In our experiment, we employ four evaluation metrics to assess model
performance: Equal error rate (EER), precision, recall, and $F_{1}$ score. All
metrics are computed based on frame-level authenticity labels of the partially
spoofed audio. Precision, recall, and $F_{1}$ score are defined as follow:

$Precison=\frac{TP}{TP+FP},$ (9) $Recall=\frac{TP}{TP+FN},$ (10)
${F_{1}score}=\frac{2\cdot Precison\cdot Recall}{Precison+Recall},$ (11)

where $TP$, $TN$, $FP$, $FN$ represent the numbers of true positive, true
negative, false positive, and false negative samples, respectively.

##  4 Results and discussions

###  4.1 Results

Results on 19PS. We compare the performance of several baseline models in
terms of EER metric, as presented in Table 3. All models are trained on the
19PS training dataset. TDL (w/o ESM) represents our model without ESM module.
As shown in Table 3, our model achieve the lowest EER 1.92% in partially
spoofed audio detection task.

Based on the experimental results, We first observe that the impact of feature
is greater than backbone. For instance, as seen in first and third row in
Table 3, where the backbone is LCNN-BLSTM, the utilization of W2V2 features
resulted in a 12.60% EER decrease compared to LFCC. Conversely, when feature
remain consistent, as demonstrated in first and second row of the in Table 3,
both employing the shared LFCC attribute, SELCNN-LSTM exhibited a marginal EER
reduction of 0.28% in comparison to LCNN-LSTM. Furthermore, we find that the
architecture design of the TDL network aligns well with partial spoofed
detection. Specifically, when the features utilized W2V2-XLS-R, the TDL
(without ESM module) still exhibits a 1.02% reduction in EER compared to the
LCNN-BLSTM.

Results on LAV-DF. To validate the generalizability of our proposed model, we
train on 19PS and evaluate on the test set of LAV-DF for 4 evaluation metrics.
The results of the testing are presented in the Table 4. Although LAV-DF is an
unbalanced dataset, our proposed model achieve the best performance of 5.17%
EER compared to baseline models.

Table 3: EER results (%) on ASVspoof2019 PS dataset. Model | Feature | EER
---|---|---
LCNN-BLSTM [10] | LFCC | 16.21
SELCNN-BLSTM [11] | LFCC | 15.93
5gMLP [16] | W2V2-Large | 9.24
LCNN-BLSTM [10] | W2V2-XLS-R | 3.61
TDL (w/o emb) | W2V2-XLS-R | 2.59
TDL | W2V2-XLS-R | 1.92
Table 4: The four evaluation metrics results (%) for training on 19PS and testing on the LAV-DF test set. Model | Feature |  EER$\downarrow$ |  Precision$\uparrow$ |  Recall$\uparrow$ |  $F_{1}$ score$\uparrow$
---|---|---|---|---|---
LCNN-BLSTM | LFCC | 10.94 | 95.93 | 73.73 | 83.38
LCNN-BLSTM | W2V2-XLS-R | 8.26 | 99.05 | 62.32 | 76.50
TDL | W2V2-XLS-R | 5.17 | 98.73 | 75.42 | 85.51
Table 5: EER results (%) of different label configuration on 19PS. Label Setting |  EER$\downarrow$ |  Precision$\uparrow$ |  Recall$\uparrow$ |  $F_{1}$ score$\uparrow$
---|---|---|---|---
Boundary 1 | 3.85 | 79.72 | 82.01 | 80.85
real 0 fake 1 | 3.01 | 81.87 | 84.52 | 83.17
real 1 fake 0 | 1.92 | 87.62 | 94.34 | 90.86
Table 6: Parameters (in thousands) comparison. Model | Parameters |
---|---|---
TDL | 8,718 |
LCNN-BLSTM | 21,511 |

###  4.2 Disscussion

Label Setting. As we mentioned in Section 3.2, we set real frames, fake frames
for 1 and 0. To the best of our knowledge, there has been no prior research
discussing which label configuration will be beneficial to the final
prediction. Therefore, we experiment with three different label settings on
our proposed TDL model as shown in Table 5.

“Boundary 1” indicates that we set the boundary frames between genuine and
fake segments as 1, while other positions are set as 0. In practice, due to
the sparsity of boundary frames, we set 4 boundary frames at the transition
between genuine and fake segments. Additionally, we employ a weighted BCE
loss, assigning a weight value of 100 to the boundary values, as a replacement
for standard BCE. Experimental results demonstrate that this method is less
effective compared to directly predicting the authenticity of individual
frames. Additionally, since predicting boundaries often requires further
verification of the genuineness of the segments on both sides, we did not
adopt the boundary setting.

For the frame-level direct prediction of authenticity, we conducted
experiments by setting real frames as 0 and fake frames as 1, and
alternatively by setting real frames as 1 and fake frames as 0, as shown in
the “real 0 fake 1” and “real 1 fake 0” of the Table 5 respectively.
Experiments results show that “real 1 fake 0” outperform “real 0 fake 1” in
four evaluation metrics, especially in recall metric, which indicates that TDL
can accurately identify genuine speech. When setting real frames as “1” and
fake frames along with padding frames as “0”, we can better concentrate on the
real segment. This is similar to previous works [21, 22] which also focus on
the real speech distribution in fully-spoofed ADD task. Through our
experiments, we have demonstrated that it is also significant in partially-
spoofed ADD task. This is also why W2V2 features are effective in the field of
ADD which only extracted by rich real source domains.

Complexity Comparision. Apart from evaluating the performance, we measured the
complexity of the models. For frame-level detection task, particularly for
fine-grained prediction, the large final output dimension can result in
excessive parameterization and low efficiency. Unlike LCNN, which convolves
overall values, our proposed TDL model uses temporal convolution operation to
selectively focus only on high-weight regions. It can be observed that the
parameter count of TDL is only 40.53% of that of LCNN-BLSTM, which is shown in
Table 6.

##  5 Conclusion

In this paper, we propose an efficient temporary deepfake location approach
based embeddings for partially spoofed audio detection. TDL can achieve
outstanding performance benefits from two designed core modules: embedding
similarity module and temporal convolution operation, which can effectively
capture both feature and positional information. The experimental results
demonstrate that TDL achieves the best performance in the 19PS dataset and
also perform well in cross-dataset scenarios.

## References

  * [1] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren,  “Prodiff: Progressive fast diffusion model for high-quality text-to-speech,”  in Proceedings of ACM MM, 2022, pp. 2595–2605.
  * [2] X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y. Liu, X. Wang, Y. Leng, Y. Yi, L. He, et al.,  “Naturalspeech: End-to-end text to speech synthesis with human-level quality,”  arXiv preprint arXiv:2205.04421, 2022.
  * [3] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.,  “Neural codec language models are zero-shot text to speech synthesizers,”  arXiv preprint arXiv:2301.02111, 2023.
  * [4] Chak Ho Chan, Kaizhi Qian, Yang Zhang, and Mark Hasegawa-Johnson,  “Speechsplit2. 0: Unsupervised speech disentanglement for voice conversion without tuning autoencoder bottlenecks,”  in Proceedings of ICASSP, 2022, pp. 6332–6336.
  * [5] Y. Chen, D. Wu, T. Wu, and H. Lee,  “Again-vc: A one-shot voice conversion using activation guidance and adaptive instance normalization,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 5954–5958.
  * [6] Huaizhen Tang, Xulong Zhang, Jianzong Wang, Ning Cheng, and Jing Xiao,  “Avqvc: One-shot voice conversion by vector quantization with applying contrastive learning,”  in Proceedings of ICASSP. IEEE, 2022, pp. 4613–4617.
  * [7] Andreas Nautsch, Xin Wang, Nicholas Evans, Tomi H Kinnunen, Ville Vestman, Massimiliano Todisco, Héctor Delgado, Md Sahidullah, Junichi Yamagishi, and Kong Aik Lee,  “Asvspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech,”  IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 3, no. 2, pp. 252–265, 2021.
  * [8] H. Delgado, N. Evans, T. Kinnunen, K. Lee, X. Liu, A. Nautsch, J. Patino, M. Sahidullah, M. Todisco, X. Wang, et al.,  “Asvspoof 2021: Automatic speaker verification spoofing and countermeasures challenge evaluation plan,”  arXiv preprint arXiv:2109.00535, 2021.
  * [9] Jiangyan Yi, Ye Bai, Jianhua Tao, Haoxin Ma, Zhengkun Tian, Chenglong Wang, Tao Wang, and Ruibo Fu,  “Half-truth: A partially fake audio detection dataset,”  in Proceedings of Interspeech, 2021, pp. 1654–1658.
  * [10] Lin Zhang, Xin Wang, Erica Cooper, Junichi Yamagishi, Jose Patino, and Nicholas Evans,  “An initial investigation for detecting partially spoofed audio,”  in Proceedings of Interspeech, 2021, pp. 4264–4268.
  * [11] Lin Zhang, Xin Wang, Erica Cooper, and Junichi Yamagishi,  “Multi-task learning in utterance-level and segmental-level spoof detection,”  arXiv preprint arXiv:2107.14132, 2021.
  * [12] Zhiqiang Lv, Shanshan Zhang, Kai Tang, and Pengfei Hu,  “Fake audio detection based on unsupervised pretraining models,”  in Proceedings of ICASSP, 2022, pp. 9231–9235.
  * [13] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in neural information processing systems, vol. 33, pp. 12449–12460, 2020.
  * [14] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck,  “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,”  arXiv preprint arXiv:2005.07143, 2020.
  * [15] Jiangyan Yi, Ruibo Fu, Jianhua Tao, Shuai Nie, Haoxin Ma, Chenglong Wang, Tao Wang, Zhengkun Tian, Ye Bai, Cunhang Fan, et al.,  “Add 2022: the first audio deep synthesis detection challenge,”  in Proceedings of ICASSP. IEEE, 2022, pp. 9216–9220.
  * [16] Lin Zhang, Xin Wang, Erica Cooper, Nicholas Evans, and Junichi Yamagishi,  “The partialspoof database and countermeasures for the detection of short fake speech segments embedded in an utterance,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022.
  * [17] Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al.,  “Xls-r: Self-supervised cross-lingual speech representation learning at scale,”  arXiv preprint arXiv:2111.09296, 2021.
  * [18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [19] Zhixi Cai, Kalin Stefanov, Abhinav Dhall, and Munawar Hayat,  “Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization,”  in 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA). IEEE, 2022, pp. 1–10.
  * [20] Xin Wang, Junichi Yamagishi, Massimiliano Todisco, Héctor Delgado, Andreas Nautsch, Nicholas Evans, Md Sahidullah, Ville Vestman, Tomi Kinnunen, Kong Aik Lee, et al.,  “Asvspoof 2019: A large-scale public database of synthesized, converted and replayed speech,”  Computer Speech & Language, vol. 64, pp. 101114, 2020.
  * [21] Y. Zhang, F. Jiang, and Z. Duan,  “One-class learning towards synthetic voice spoofing detection,”  IEEE Signal Processing Letters, vol. 28, pp. 937–941, 2021.
  * [22] Yuankun Xie, Haonan Cheng, Yutian Wang, and Long Ye,  “Learning A Self-Supervised Domain-Invariant Feature Representation for Generalized Audio Deepfake Detection,”  in Proc. INTERSPEECH 2023, 2023, pp. 2808–2812.