# Exploiting Temporal Structures of Cyclostationary Signals for Data-Driven
Single-Channel Source Separation

###### Abstract

We study the problem of single-channel source separation (SCSS), and focus on
cyclostationary signals, which are particularly suitable in a variety of
application domains. Unlike classical SCSS approaches, we consider a setting
where only examples of the sources are available rather than their models,
inspiring a data-driven approach. For source models with underlying
cyclostationary Gaussian constituents, we establish a lower bound on the
attainable mean squared error (MSE) for any separation method, model-based or
data-driven. Our analysis further reveals the operation for optimal separation
and the associated implementation challenges. As a computationally attractive
alternative, we propose a deep learning approach using a U-Net architecture,
which is competitive with the minimum MSE estimator. We demonstrate in
simulation that, with suitable domain-informed architectural choices, our
U-Net method can approach the optimal performance with substantially reduced
computational burden.

††footnotetext:  Research was sponsored by the United States Air Force
Research Laboratory and the United States Air Force Artificial Intelligence
Accelerator and was accomplished under Cooperative Agreement Number
FA8750-19-2-1000. The views and conclusions contained in this document are
those of the authors and should not be interpreted as representing the
official policies, either expressed or implied, of the United States Air Force
or the U.S. Government. The U.S. Government is authorized to reproduce and
distribute reprints for Government purposes notwithstanding any copyright
notation herein.
The authors acknowledge the MIT SuperCloud and Lincoln Laboratory
Supercomputing Center for providing HPC resources that have contributed to the
research results reported within this paper.
Alejandro Lancho is funded by EU’s Horizon 2020 programme under the Marie
Sklodowska-Curie grant agreement No. 101024432.

Index Terms—  Source separation, cyclostationary signal processing, deep
neural network, supervised learning.

##  1 Introduction

Source separation is a well-studied problem with many important applications
in radio-frequency systems, wireless communications and biomedical signal
monitoring [1, 2, 3, 4], to name a few. A common formulation is the blind
source separation problem, for which perhaps the most popular framework is
independent component analysis [5]. Many approaches rely on some degree of
spatial diversity present in multi-channel observations. In contrast, a
particularly challenging problem of interest is the _single-channel_ (or
single-sensor) source separation (SCSS). The goal in SCSS is to separate the
latent sources from a single-channel observed mixture. In this regime, the
aforementioned algorithms are irrelevant due to the absence of spatial
diversity. Instead, other characteristics of the latent sources must be
exploited.

Recent efforts demonstrate that machine learning (ML) techniques can serve as
powerful tools for source separation, even in the single-channel regime, in
image and audio counterparts [6, 7, 8, 9]. Solutions typically exploit the
inherent structure specific to the signal type. For example, natural images
may be separable by color features and local dependencies [8], whereas speech
signals are commonly addressed by time-frequency spectrogram masking [10, 11,
12]. Furthermore, for time series ($1$-dimensional) data, if the sources are
separable in time and/or frequency, appropriate spectrogram-based masking and
classical filtering methods can be adopted, e.g. [13, 14]. In contrast, a key
challenge is the separation of signals that overlap (partially/fully) in both
time and frequency; such a setting is rarely addressed in aforementioned
works. Further, in such a regime, it is no longer trivial to identify the
properties that are helpful in separating the signals, and what separation
performance is attainable.

One particular type of time series data is the class of _cyclostationary_
signals, a relevant model adopted in many of the earlier mentioned
applications (e.g., [15]). Interestingly, under some conditions, perfect
separation of cyclostationary signals could be achieved despite having
components with overlapping time-frequency spectra [16, 17]. The challenge,
however, is in modeling such cyclostationarities in a way that properly
describes the true statistics of the observed mixtures. A more realistic
scenario is one where the signal model is unknown, but examples of the signals
(by measurements or generation111Note that the ability to generate a signal
(with some device) does not necessarily mean that one can also analytically
characterize its true statistics.) are available. Further, despite
cyclostationarity properties, the available examples are usually finite
unsynchronized time segments, meaning that they are extracted at possibly
random different “start times”, relative to an arbitrarily chosen time
instance. In other words, there are additional latent variables—the random
time shifts—that are unobservable, adding a layer of difficulty to the
problem.

In this paper, we study the SCSS problem involving mixtures of two signals.
Specifically, we focus on the scenario where each component is a randomly
time-shifted and scaled segment from a cyclostationary complex Gaussian
(hereafter, simply referred to as _Gaussian_ in this work) process, a regime
in which an analytical form of the optimal estimator, in the sense of minimum
mean-square-error (MMSE), can be derived. Nonetheless, we also highlight the
challenges in implementing the optimal estimator, thereby motivating less
computationally demanding alternatives. We propose a deep learning (DL)
approach to the SCSS problem, and demonstrate its performance in two
representative examples—one of which is based on real applications in wireless
communications. Our simulations show that the performance of our proposed DL
strategy is competitive to the optimal estimator.

We recognize that many signals of practical relevance generally depart from
the Gaussian model. However, the analysis involving cyclostationary Gaussian
signals allows us to better understand and characterize the performance of DL-
based methods. In particular, it focuses our attention to the exploitation of
temporal correlation structures of the latent source components, so as to
accurately assess the ability of ML approaches in capturing such structures.
This would shed light as to how far we are from the optimal performance, and
how to close that performance gap.

##  2 SCSS Problem Formulation

We consider the following model of an observed 1D signal of length $N$, which
is a noisy mixture of two latent sources,

${\mbox{\boldmath$y$}}={\mbox{\boldmath$s$}}_{\tau_{s}}+\underbrace{\kappa\,{\mbox{\boldmath$b$}}_{\tau_{b}}}_{\text{``interference''}}+\underbrace{\sigma{\mbox{\boldmath$z$}}}_{\text{``noise''}}\;\in\mathbb{C}^{N},$
(1)

where ${\mbox{\boldmath$s$}}_{\tau_{s}},{\mbox{\boldmath$b$}}_{\tau_{b}}$ are
the (unobservable) independent components, $\kappa\in\mathbb{R}_{+}$ is
distributed according to some unknown (and for simplicity, discrete)
distribution on $\mathcal{K}\subset\mathbb{R}_{+}$, $0$ denotes the all zeros-
vector and $I$ denotes the identity matrix (with context-dependent
dimensions). The noise component, $\sigma{\mbox{\boldmath$z$}}$, is an
additive white Gaussian noise, where
${\mbox{\boldmath$z$}}\sim\mathcal{C}\mathcal{N}({\mbox{\boldmath$0$}},{\mbox{\boldmath$I$}})$
and $\sigma^{2}$ corresponds to variance of the noise. Without loss of
generality, for the purposes of our discussion,
${\mbox{\boldmath$s$}}_{\tau_{s}}$ is termed the “reference” signal and
$\kappa\,{\mbox{\boldmath$b$}}_{\tau_{b}}$ the interference. Additionally, we
assume that ${\mbox{\boldmath$s$}}_{\tau_{s}}$ and
${\mbox{\boldmath$b$}}_{\tau_{b}}$ have unit average power; hence,
$\kappa^{2}$ is related to the inverse of the signal-to-interference ratio
(SIR). The goal in SCSS is to produce an estimate
$\widehat{{\mbox{\boldmath$s$}}}_{\tau_{s}}$ based on $y$ so that given some
metric $d$, the value
$\mathbb{E}\left[d(\widehat{{\mbox{\boldmath$s$}}}_{\tau_{s}},{\mbox{\boldmath$s$}}_{\tau_{s}})\right]$
is minimized. We limit our scope to the conventional metric
$d({\mbox{\boldmath$u$}},{\mbox{\boldmath$v$}})=\|{\mbox{\boldmath$u$}}-{\mbox{\boldmath$v$}}\|^{2}_{2}$,
leading to the MMSE criterion.

We focus our discussion on signal components that are segments extracted from
cyclostationary Gaussian processes. We consider two independent, discrete-
time, zero-mean circularly-symmetric complex Gaussian processes
$\tilde{s}[\cdot]$, $\tilde{b}[\cdot]$, with autocovariance functions
satisfying

$\displaystyle C_{\tilde{s}}[n,l]$
$\displaystyle\triangleq\mathbb{E}\left[\tilde{s}[n]\,\tilde{s}[n+l]\right],\;\;C_{\tilde{s}}[n,l]$
$\displaystyle=C_{\tilde{s}}[n+N_{s},l],$ $\displaystyle C_{\tilde{b}}[n,l]$
$\displaystyle\triangleq\mathbb{E}\left[\tilde{b}[n]\,\tilde{b}[n+l]\right],\;\;C_{\tilde{b}}[n,l]$
$\displaystyle=C_{\tilde{b}}[n+N_{b},l],$

i.e., cyclostationary with fundamental periods $N_{s},N_{b}>1$ .

We denote the time offsets by $\tau_{s}$, $\tau_{b}$. Hence,

$\displaystyle{\mbox{\boldmath$s$}}_{\tau_{s}}=[s_{\tau_{s}}[0],\ldots,s_{\tau_{s}}[N-1]]^{{\rm
T}}\in\mathbb{C}^{N},\;s_{\tau_{s}}[n]\triangleq\tilde{s}[n+\tau_{s}],$

and similarly for ${\mbox{\boldmath$b$}}_{\tau_{b}}$ and $\tilde{b}$. We
consider the case where time offsets are random, drawn from a discrete uniform
distribution, i.e., $\tau_{s}\sim\mathcal{U}[0,N_{s}-1]$ and
$\tau_{b}\sim\mathcal{U}[0,N_{b}-1]$, and assume $\tau_{s}$, $\tau_{b}$,
$\tilde{s}[\cdot]$, $\tilde{b}[\cdot]$ and $\kappa$ are statistically
independent. Consequently, note that ${\mbox{\boldmath$s$}}_{\tau_{s}}$ and
${\mbox{\boldmath$b$}}_{\tau_{b}}$ are Gaussian mixtures.

As mentioned in Section 1, we are particularly interested in the case where we
do not have full knowledge of the signal models—the distributions of
$\tilde{{\mbox{\boldmath$s$}}}$ and $\tilde{{\mbox{\boldmath$b$}}}$ (and
thereby of ${\mbox{\boldmath$s$}}_{\tau_{s}}$ and
${\mbox{\boldmath$b$}}_{\tau_{b}}$) are unknown. Nevertheless, we assume we
have a dataset of $M$ independent, identically distributed (iid) copies of
$\\{({\mbox{\boldmath$y$}}^{(i)},{\mbox{\boldmath$s$}}_{\tau_{s}}^{(i)})\\}_{i=1}^{M}$,
allowing for a data-driven approach. Note that this dataset contains
_unsynchronized_ examples, wherein the corresponding time offsets
$\tau^{(i)}_{s}$ and $\tau^{(i)}_{b}$ (of the $i$-th pair) are unknown.

Under this formulation, an ML approach is to learn a regressor, i.e., to
estimate ${\mbox{\boldmath$s$}}_{\tau_{s}}$ from $y$. Note that the dimension
of the regression output is the same as that of the input. Thus, it can be
thought of as a transformation or filtering (through some parametric model) of
$y$ to obtain ${\mbox{\boldmath$s$}}_{\tau_{s}}$, which is generally more
challenging than a conventional single-output regression problem [18]. We will
thus investigate the use of deep neural networks (DNNs) to learn such a
regressor from data. However, we begin with an in-depth examination of the
SCSS problem through approaches to optimal estimation.

##  3 Optimal Model-Based Estimators

We now derive, for several cases, optimal estimators that achieve the lower
bounds for their respective cases. Specifically, the simplified expression of
the MMSE estimator reveals the challenges in attaining it, but also provides
valuable intuition that informs and justifies our proposed data-driven
methodology for the SCSS problem.

A possible “classical signal processing” approach for this setting, is a two-
step process, where one first estimates the model parameters, and then adopts
MMSE estimation based on the empirical model. However, estimation of these
parameters requires synchronization of the given dataset, which can be a very
challenging task in itself. Nevertheless, in this section we assume access to
an oracle regarding the signal model, so as to establish a lower bound on the
mean squared error (MSE) for our problem.

Specifically, in the following we assume oracle knowledge of the signal
models—i.e., the first and second-order statistics (SOSs) of $\tilde{s}$ and
$\tilde{b}$, and the marginal distributions of $\tau_{s},\tau_{b}$. We denote
the conditional (temporal) covariance of ${\mbox{\boldmath$s$}}_{\tau_{s}}$
given $\tau_{s}$ by

$\displaystyle{\mbox{\boldmath$C_{s}$}}(\tau_{s})\triangleq\mathbb{E}\left[{\mbox{\boldmath$s$}}_{\tau_{s}}{\mbox{\boldmath$s$}}_{\tau_{s}}^{\rm{H}}|\tau_{s}\right]\in\mathbb{C}^{N\times
N},$

which we highlight that it is a function of $\tau_{s}$; likewise,
${\mbox{\boldmath$C_{b}$}}(\tau_{b})$ denotes the conditional covariance of
${\mbox{\boldmath$b$}}_{\tau_{b}}$. We denote the entries of
${\mbox{\boldmath$C_{s}$}}(\tau_{s})$ by
$({\mbox{\boldmath$C_{s}$}}(\tau_{s}))_{i,j}=C_{\tilde{s}}[i+\tau_{s},i-j]$.

###  3.1 Case 1: The Optimal Linear MMSE Solution

To obtain the SOS of ${\mbox{\boldmath$s$}}_{\tau_{s}}$, one must account for
the uniform randomness in $\tau_{s}$. Hence, the covariance of
${\mbox{\boldmath$s$}}_{\tau_{s}}$ is given by

$\displaystyle{\mbox{\boldmath$\check{C}_{s}$}}=\mathbb{E}\left[{\mbox{\boldmath$s$}}_{\tau_{s}}{\mbox{\boldmath$s$}}_{\tau_{s}}^{\rm{H}}\right]=\mathbb{E}_{\tau_{s}}\left[\mathbb{E}\left[{\mbox{\boldmath$s$}}_{\tau_{s}}{\mbox{\boldmath$s$}}_{\tau_{s}}^{\rm{H}}|\tau_{s}\right]\right]=\frac{1}{N_{s}}\sum_{\tau_{s}=0}^{N_{s}-1}{\mbox{\boldmath$C_{s}$}}(\tau_{s}).$

Note that this corresponds to a Toeplitz covariance structure, due to the fact
that $s_{\tau_{s}}[\cdot]$ is a wide-sense stationary process (unlike
$\tilde{s}[\cdot]$). The same applies for the covariance of
${\mbox{\boldmath$b$}}_{\tau_{b}}$.

The SOSs of ${\mbox{\boldmath$s$}}_{\tau_{s}}$ and
${\mbox{\boldmath$b$}}_{\tau_{b}}$ are sufficient for optimal _linear_
estimation. For the sake of practicality, if we restrict our attention to
linear operators, the optimal estimator is given by

$\widehat{{\mbox{\boldmath$s$}}}_{\tau_{s},\text{\tiny
LMMSE}}={\mbox{\boldmath$\check{C}_{s}$}}\,\left[{\mbox{\boldmath$\check{C}_{s}$}}+\mathbb{E}\left[\kappa^{2}\right]\cdot{\mbox{\boldmath$\check{C}_{b}$}}+\sigma^{2}{\mbox{\boldmath$I$}}\right]^{-1}{\mbox{\boldmath$y$}},\\\
$ (2)

where we used the statistical independence assumption of the sources and noise
to compute the covariance of $y$. This would indeed correspond to the
estimator that achieves the MMSE among the family of linear estimators.
Further, given a sufficiently large dataset, one can implement an accurate
approximation of (2) by replacing the covariance matrices with empirical
estimates (i.e., sample covariances) from the datasets. However, due to the
random time offsets, ${\mbox{\boldmath$s$}}_{\tau_{s}}$ and
${\mbox{\boldmath$b$}}_{\tau_{b}}$ are not Gaussian—but rather, are Gaussian
mixtures—and hence the optimal linear estimator does not coincide with the
MMSE estimator.

###  3.2 Case 2: The Oracle MMSE Solution

In this subsection, we develop the optimal solution for the case in which
$(\tau_{s},\tau_{b},\kappa)$ are known (or, observable). This would have been
the case if we would have at our disposal an oracle that provides side
information of perfect synchronization to both the reference and interference
signals, and the SIR-related coefficient $\kappa$. Under this setting, $y$ and
${\mbox{\boldmath$s$}}_{\tau_{s}}$ are jointly Gaussian, hence the optimal
solution is given by

${\widehat{{\mbox{\boldmath$s$}}}_{\tau_{s}}}|_{(\tau_{s},\tau_{b},\kappa)}=\mathbb{E}\left[{\mbox{\boldmath$s$}}_{\tau_{s}}|{\mbox{\boldmath$y$}},\tau_{s},\tau_{b},\kappa\right]={\mbox{\boldmath$H$}}(\tau_{s},\tau_{b},\kappa){\mbox{\boldmath$y$}},$
(3)

where

$\displaystyle{\mbox{\boldmath$H$}}(\tau_{s},\tau_{b},\kappa)\triangleq{\mbox{\boldmath$C_{s}$}}(\tau_{s})\left[{\mbox{\boldmath$C_{s}$}}(\tau_{s})+\kappa^{2}{\mbox{\boldmath$C_{b}$}}(\tau_{b})+\sigma^{2}{\mbox{\boldmath$I$}}\right]^{-1}$
(4)

is the optimal linear (generally) time-varying filter. Since (4) is a function
of $(\tau_{s},\tau_{b},\kappa)$, we subsequently refer to (3) as an oracle-
synchronized MMSE solution. Nevertheless, it should be noted that (3) is not a
realizable estimator, as it is a function of latent, unobservable variables.
Also, note that for cyclostationary signals, (3) can also be represented as a
frequency-shift filter or a cyclic Wiener filter, i.e., expressed as a linear
function of frequency shifted copies of the observation [2, 17].

###  3.3 Case 3: The Optimal MMSE Estimator

In general, the time offsets $\tau_{s}$ and $\tau_{b}$, and the SIR parameter
$\kappa$, are not known at inference time, and a realizable estimator must
account for their inherent randomness, by explicit/implicit (non-linear)
estimation. However, upon conditioning on these quantities, i.e., considering
(4) as fixed, the resulting optimal MMSE estimator would also be the linear
one. With this observation in mind, the true, realizable MMSE estimator is
expressed as

$\displaystyle\widehat{{\mbox{\boldmath$s$}}}_{\tau_{s},\text{\tiny MMSE}}$
$\displaystyle=\mathbb{E}\left[{\mbox{\boldmath$s$}}_{\tau_{s}}|{\mbox{\boldmath$y$}}\right]=\mathbb{E}_{(\tau_{s},\tau_{b},\kappa)|{\mbox{\boldmath$y$}}}\left[\mathbb{E}\left[{\mbox{\boldmath$s$}}_{\tau_{s}}|{\mbox{\boldmath$y$}},\tau_{s},\tau_{b},\kappa\right]\right]$
$\displaystyle=\sum_{\tau_{s}=0}^{N_{s}-1}\sum_{\tau_{b}=0}^{N_{b}-1}\sum_{\kappa\in\mathcal{K}}p(\tau_{s},\tau_{b},\kappa|{\mbox{\boldmath$y$}})\,\widehat{{\mbox{\boldmath$s$}}}_{\tau_{s}}|_{(\tau_{s},\tau_{b},\kappa)},$
(5)

which essentially corresponds to the sum of oracle-synchronized MMSE solutions
(3) for each set of parameters, weighted by the corresponding posterior
probabilities given the observation $y$. We emphasize that (5), unlike (3), is
a legitimate estimator, i.e., a function of the observed data only. However,
the MSE achieved by (3) serves as a lower bound of that by (5), by data
procecessing inequality of MMSE.

It should be noted that, in cases when the true posterior is a Kronecker delta
function, (5) in fact coincides with (3),

$\displaystyle p(\tau_{s},\tau_{b},\kappa|{\mbox{\boldmath$y$}})$
$\displaystyle=\delta[\tau_{s}-\tau^{*}_{s},\tau_{b}-\tau^{*}_{b},\kappa-\kappa^{*}]$
(6)
$\displaystyle\Longrightarrow\;\;\widehat{{\mbox{\boldmath$s$}}}_{\tau_{s},\text{\tiny
MMSE}}$
$\displaystyle={\widehat{{\mbox{\boldmath$s$}}}_{\tau_{s}}}|_{(\tau^{*}_{s},\tau^{*}_{b},\kappa^{*})},$

namely the performance of the oracle-aided solution is achievable. Indeed, for
the noiseless case ($\sigma^{2}=0$), it is easy to construct examples, where
perfect separation is attained.

Fig. 1: The U-Net architecture used for SCSS in our simulations.

##  4 Methodologies for Data-Driven SCSS

The methods discussed thus far provide an understanding of the functional
structure for the MMSE estimator, in terms of the (unknown) time offsets and
SIR parameters. Nevertheless, implementing the MMSE estimator may not be
possible in practice, due to the challenges outlined below.

###  4.1 Difficulties in Realizing the MMSE Estimator

First, as seen in (5), the MMSE estimator involves the computation of a
posterior term over the latent variables $(\tau_{s},\tau_{b},\kappa)$.
However, obtaining this posterior becomes computationally intensive as the
space of parameters increases.

Second, (5) also involves a sum of different (conditionally) linear operators,
each corresponding to a set of parameter values. However, each of these
operators involves an inversion of a large covariance matrix, which is
infeasible in regimes of signals from long observation periods (large $N$).

Lastly, and importantly, in practice we do not have oracle knowledge of the
signal model, corresponding to the first and second-order statistics in the
case of the Gaussian models. Instead, we are given many examples, through
which we can obtain empirical estimates of the statistics. However, as
mentioned, the dataset of the signals is not synchronized. Thus, obtaining the
SOSs of the underlying cyclostationary signal, $C_{\tilde{s}}$ and
$C_{\tilde{b}}$, requires the estimation of the latent variable $\tau_{s}$ and
$\tau_{b}$, for each example in the dataset, corresponding to the
synchronization of the entire dataset, which is generally an extremely
challenging task of independent interest in itself.

We now show that ML methods can be successfully used to circumvent the
aforementioned issues. Specifically, we propose the use of DNNs, trained on
_unsynchronized_ datasets, to solve the SCSS problem. We use two
representative examples to demonstrate our approach, and compare it with the
performance of an optimal MMSE solution that utilizes oracle knowledge. The
goal henceforth is to establish a practical pipeline, and benchmark it against
a theoretical lower bound.

###  4.2 Supervised Separation with U-Net

Given the formulation in Section 2, a natural approach is to use a DNN to
learn a regression model with multivariate output for source separation. We
propose to use the so-called “U-Net” architecture for SCSS (architecture shown
in Fig. 1).222 https://github.com/RFChallenge/SCSS_CSGaussian Such a DNN was
first proposed for biomedical image segmentation [19], but has found its use
in other applications, including spectrogram-based RF interference
cancellation [1] and audio source separation [6, 12]—all of which also
corresponding to a multivariate regression setup with the same dimensions on
the input and output. Similar to the latter works, we use 1D-convolutional
layers to capture features relating to the time series data. The U-Net
architecture contains downsampling blocks that operate on successively coarser
timescales, and possesses skip connections that combine features at these
various timescales with the upsampling blocks.

To technically handle complex-valued signals, borrowing inspiration from
widely linear estimation [20], we stack the real and imaginary parts as
separate channels to the U-Net.

As these methods are applied to time series signals in practice, the use of
domain knowledge to craft an appropriate neural network architecture may be
crucial in attaining performance gain, as evident in our experiments and
architectural choices. For example, we made the intentional choice of longer
kernel sizes on the first convolutional layers. This further reinforces the
relevance of this work, that is in identifying and characterizing DNN
architectures under study relative to the best possible performance. In our
experiments, we observe that kernel sizes that match the effective correlation
length (i.e., timescales in which the covariance magnitudes are non-
negligible) are required to attain the best performance. This may be an
indication to how some partial, though important, information about the signal
model is helpful (or even essential) in seeking the appropriate DNN
architecture.

##  5 Simulation Results

We now consider two examples for SCSS. For each setting, we train a U-Net to
estimate the corresponding signal ${\mbox{\boldmath$s$}}_{\tau_{s}}$ from the
mixture $y$ in an end-to-end fashion, and with no supervision regarding the
time-shifts $\tau_{s}$ and $\tau_{b}$, and the gain $\kappa$.

In the examples below, we describe how long (but finite) segments of the
processes $\tilde{s}[\cdot],\tilde{b}[\cdot]$ are generated, from which
$N$-length segments are extracted to create the datasets. The training set is
processed as such to yield a labeled dataset of iid copies—mixture and ground-
truth reference signal,
$\\{({\mbox{\boldmath$y$}}^{(i)},{\mbox{\boldmath$s$}}_{\tau_{s}}^{(i)})\\}_{i=1}^{M}$—as
is done in the supervised learning framework. Our training set comprises
$10,000\times|\mathcal{K}|$ pairs of mixtures $y$ and ground-truth
${\mbox{\boldmath$s$}}_{\tau_{s}}$, and the validation set comprises
$500\times|\mathcal{K}|$ pairs, where the cardinality $|\mathcal{K}|$ is the
total number of levels for $\kappa$ under consideration. Subsequently, we test
the performance across $1,000$ examples per $\kappa$ level, reporting the
average MSE in dB. Note that varying $\kappa$ results in different levels of
SIR. In our simulations, we assess the separation performance across different
SIR levels. We also compare the performance of using the aforementioned U-Net
against that of the optimal model-based estimators. For these optimal
estimators, $\kappa$ is assumed to be known.

#### Implementation Details:

Keras and Tensorflow 2 are used to implement and train the U-Net [21, 22]. For
training, we use empirical MSE as the loss function. We also use Adam
optimizer[23] and an exponentially decaying learning rate schedule, batch size
of 32 with shuffled training samples, and trained for 2,000 epochs with early
stopping if there is no improvement for 100 epochs on the validation set. We
train the neural networks on a computing cluster with Intel Xeon Gold 6248,
192 GB RAM, and an NVidia Volta V100 GPU.

###  5.1 Signals with Randomly Generated Covariances

We consider (1) with $N=256$, $N_{s}=11$, $N_{b}=5$, $\mathcal{K}$
corresponding to 5 equidistant SIR levels in $[-6,6]$ dB. The reference and
interference signals are generated as

${\mbox{\boldmath$\tilde{s}$}}={\mbox{\boldmath$G_{s}$}}{\mbox{\boldmath$a$}}_{1},\;\;{\mbox{\boldmath$\tilde{b}$}}={\mbox{\boldmath$G_{b}$}}{\mbox{\boldmath$a$}}_{2},$

where
${\mbox{\boldmath$a$}}_{1},{\mbox{\boldmath$a$}}_{2}\sim\mathcal{C}\mathcal{N}({\mbox{\boldmath$0$}},{\mbox{\boldmath$I$}})$,
and
${\mbox{\boldmath$G_{s}$}},{\mbox{\boldmath$G_{b}$}}\in\mathbb{C}^{\tilde{N}\times\tilde{N}}$,
with $\tilde{N}=550$, are block-diagonal matrices with repeating $N_{s}\times
N_{s}$ and $N_{b}\times N_{b}$ blocks respectively. Each entry in the blocks
is drawn (once) independently from the Gaussian distribution, and is fixed for
the rest of this experiment. Full details on the signal generation are
provided in our Github repository.22footnotemark: 2 Fig. 2(a) shows the
covariance structures of the resulting sources.

(a) Simple Cyclostationary Example
---
|  |  |
Signal ${\mbox{\boldmath$s$}}_{\tau_{s}}$ | Signal ${\mbox{\boldmath$b$}}_{\tau_{b}}$ | Signal ${\mbox{\boldmath$s$}}_{\tau_{s}}$ | Signal ${\mbox{\boldmath$b$}}_{\tau_{b}}$
($\tau_{s}=0$) | ($\tau_{b}=0$) | (Unsynchronized) | (Unsynchronized)
(b) Communication-inspired Example
|  |  |
RRC Signal | OFDM Signal | RRC Signal | OFDM Signal
($\tau_{s}=0$) | ($\tau_{b}=0$) | (Unsynchronized) | (Unsynchronized)
Fig. 2: Visualization of (a sub-matrix of) the covariance matrices for the
signals used in the respective examples.

Fig. 3 compares the MSE achieved by the U-Net against that obtained by the
linear and the “global” MMSE estimators. We also include the oracle-
synchronized MMSE (3), which, as evident from the figure, is indistinguishable
(in terms of its MSE) from the true, nonoracle MMSE estimator (5). This occurs
when all the mass of the posterior is (approximately) concentrated at the
point of the true values of $(\tau_{s},\tau_{b},\kappa)$, as in (6).

We observe that a U-Net trained on an unsynchronized dataset of signals is
capable of obtaining results close to the MMSE performance. This means that
the U-Net necessarily learned a significant part of the model, which enables
high-quality estimation of the reference signal. We reiterate that the U-Net
did not have access to the true statistics of the signal model or any form of
synchronization of the signals during training and inference. The slight
deterioration performance could be attributed to approximation errors
introduced from a DL-based function approximator, or due to the trade-off from
lack of access to synchronized dataset. Future work entails identifying
factors to close the performance gap.

###  5.2 Communication-like Waveforms

The problem of SCSS is particularly relevant in the application of wireless
communications, where we may be interested in separating multiplex of signals,
or extracting a signal-of-interest while mitigating interference [24, 3]. In
this example, we consider two types of signals with SOS properties identical
to a single-carrier communication waveform and an orthogonal frequency
division multiplexed (OFDM) waveform, respectively. The single-carrier signal
is modeled as,

$\displaystyle\tilde{s}[n]=\sum_{p=-\infty}^{\infty}a_{p}\,g[n-pN_{s}],$ (7)

where $a_{p}\sim\mathcal{C}\mathcal{N}(0,1)$, $N_{s}$ is the symbol period,
and $g[n]$ is the root-raised cosine (RRC) filter. Here, we consider a RRC
filter that corresponds to 16 samples per symbol—i.e., $N_{s}=16$—and spans 8
symbols, with a roll-off factor of 0.5. This also corresponds to an example
where a cyclostationary signal’s covariance (Fig. 2(b)) is not block-diagonal,
which leads to additional computational burden in the matrix inversion.

Fig. 3: Separation performance of the U-Net separator vs. optimal model-based
estimators for waveforms with randomly generated covariance structures. Fig.
4: Separation performance of the U-Net separator vs. optimal model-based
estimators for communication-like waveforms.

The second source, an OFDM waveform, is modeled as,

$\displaystyle\tilde{b}[n]={\frac{1}{\sqrt{N_{\textrm{sc}}}}}\sum_{p=-\infty}^{\infty}\sum_{\ell=0}^{N_{\textrm{sc}}-1}a_{p,\ell}\,q[n-pN_{b},\,\ell],$
(8) $\displaystyle q[n,\ell]=\mathbbm{1}_{\left\\{0\leq n\leq
N_{b}-1\right\\}}\,\cdot\,\exp\left(j2\pi\ell\frac{n-N_{\textrm{cp}}}{N_{\textrm{sc}}}\right),$

where $a_{p,\ell}\sim\mathcal{C}\mathcal{N}(0,1)$ for
$\ell\in\mathcal{L}_{\textrm{sc}}$, where $\mathcal{L}_{\textrm{sc}}$ refers
to the set of nonzero subcarrier indices, and $a_{p,\ell}=0$ otherwise,
$N_{\textrm{sc}}$ is the number of subcarriers per OFDM symbol,
$N_{\textrm{cp}}$ is the cyclic prefix (CP) length, and $N_{b}$ is the OFDM
symbol period, i.e., $N_{b}=N_{\textrm{sc}}+N_{\textrm{cp}}$. In our specific
example, $N_{\textrm{sc}}=64$, $N_{\textrm{cp}}=16$, and thus $N_{b}=80$. More
details on the signal specifications are provided in our Github
repository.22footnotemark: 2 Fig. 2(b) shows the covariance structures of the
resulting sources, whose cyclostationarity is evident.

We consider segments of length $N=1280$, and $\mathcal{K}$ corresponding to
SIR levels $[-30,3]$ dB at 1.5 dB steps ($|\mathcal{K}|=23$), namely we focus
on the more challenging low SIR regime. We emphasize that the models (7) and
(8) are assumed to be _unknown_ once we have generated the dataset. Rather, we
only have access to a dataset of unsynchronized samples.

Fig. 4 compares the MSE achieved by the linear MMSE and the oracle-
synchronized MMSE. We note that, for this example, the true MMSE curve could
not be obtained in practice due to the size of the parameter space, rendering
the computation of the posterior infeasible. Nevertheless, the MSE of the
oracle-synchronized MMSE solution—albeit not a realizable estimator, as
established earlier—serves as a lower bound.

As observed from Fig. 4, the best performing U-Net, which is trained on
unsynchronized data, outperforms the linear MMSE estimator, and is close to
the performance of the oracle MMSE (e.g., about $1.2$ dB away at SIR levels
between $-9$ and $-30$ dB). We also highlight that the choice of U-Net
architecture to achieve such a performance benefits from specific domain
knowledge. For example, capturing the temporal structures on the order of the
signals’ effective correlation length yielded significantly improved
performance—for which long kernels on the first layer is a way of doing
so.333For comparisons with other DNNs for the communication example:
https://github.com/RFChallenge/SCSS_DNN_Comparison

##  6 Conclusion

We study the SCSS problem when signal models of the underlying latent sources
are unknown, and develop a data-driven, DL approach as a prospective solution.
Further, we characterize the setup for signals corresponding to segments
extracted from cyclostationary Gaussian processes, and present optimal MMSE
estimators. We then outline practical challenges associated with implementing
the MMSE estimator, and through two examples, demonstrate how a trained U-Net
with a careful choice of first-layer kernel size is able to achieve comparable
performance, motivating DNNs as a competitive, computationally attractive
approach.

We reiterate that in this work, the Gaussianity assumption allowed us to have
a simplified closed-form expression for the MMSE estimator (5), and in turn a
theoretical benchmark to compare against. In practice, signals depart from
Gaussianity—e.g., digital communication signals possess symbols from discrete
sets. In these cases, the MMSE estimator is no longer linear that solely
depends on (up to) second-order statistics, and higher-order statistics can be
exploited to attain performance better than what is attained by (5). However,
an analytical form of the optimal estimator is less obvious. This strongly
motivates learning-based methods that hold the promise of improving upon
implementable, optimal model-based estimators by learning those higher-order
structures. Characterization and exploration of such performance gains via DL
are of high interest for future work.

## References

  * [1] J. Akeret, C. Chang, A. Lucchi, and A. Refregier, “Radio frequency interference mitigation using deep convolutional neural networks,” _Astronomy and computing_ , vol. 18, pp. 35–39, 2017.
  * [2] N. Shlezinger and R. Dabora, “Frequency-shift filtering for OFDM signal recovery in narrowband power line communications,” _IEEE Trans. Commun._ , vol. 62, no. 4, pp. 1283–1295, 2014.
  * [3] M. Zhao, X. Yao, J. Wang, Y. Yan, X. Gao, and Y. Fan, “Single-channel blind source separation of spatial aliasing signal based on stacked-LSTM,” _Sensors_ , vol. 21, no. 14, 2021.
  * [4] X. Chen, X. Xu, A. Liu, S. Lee, X. Chen, X. Zhang, M. J. McKeown, and Z. J. Wang, “Removal of muscle artifacts from the EEG: A review and recommendations,” _IEEE Sensors Journal_ , vol. 19, no. 14, pp. 5353–5368, 2019.
  * [5] P. Comon and C. Jutten, _Handbook of Blind Source Separation: Independent component analysis and applications_. Academic press, 2010.
  * [6] D. Stoller, S. Ewert, and S. Dixon, “Wave-U-Net: A multi-scale neural network for end-to-end audio source separation,” in _Proceedings of the 19th ISMIR_ , 2018, pp. 334–340.
  * [7] A. A. Nugraha, A. Liutkus, and E. Vincent, “Multichannel audio source separation with deep neural networks,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 24, no. 9, pp. 1652–1664, 2016.
  * [8] Y. Gandelsman, A. Shocher, and M. Irani, ““Double-DIP”: Unsupervised image decomposition via coupled deep-image-priors,” in _Proc. of IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , 2019, pp. 11 026–11 035.
  * [9] Y. Lyu, Z. Cui, S. Li, M. Pollefeys, and B. Shi, “Reflection separation using a pair of unpolarized and polarized images,” _Adv. Neural Inf. Process. Syst._ , vol. 32, pp. 14 559–14 569, 2019.
  * [10] P.-S. Huang, M. Kim, M. Hasegawa-Johnson, and P. Smaragdis, “Joint optimization of masks and deep recurrent neural networks for monaural source separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 23, no. 12, pp. 2136–2147, 2015.
  * [11] Y. Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,” _IEEE/ACM Trans. Audio, Speech, Lang. Process._ , vol. 27, no. 8, pp. 1256–1266, 2019.
  * [12] E. Tzinis, Z. Wang, and P. Smaragdis, “Sudo RM-RF: Efficient networks for universal audio source separation,” in _Proc. of MLSP_ , 2020, pp. 1–6.
  * [13] M. G. Amin, “Interference mitigation in spread spectrum communication systems using time-frequency distributions,” _IEEE Trans. Signal Process._ , vol. 45, no. 1, pp. 90–101, 1997.
  * [14] M. G. Amin, D. Borio, Y. D. Zhang, and L. Galleani, “Time-frequency analysis for GNSSs: From interference mitigation to system monitoring,” _IEEE Signal Process. Mag._ , vol. 34, no. 5, pp. 85–95, 2017.
  * [15] A. Punchihewa, Q. Zhang, O. A. Dobre, C. Spooner, S. Rajan, and R. Inkol, “On the cyclostationarity of OFDM and single carrier linearly digitally modulated signals in time dispersive channels: Theoretical developments and application,” _IEEE Trans. Wirel. Commun._ , vol. 9, no. 8, pp. 2588–2599, 2010.
  * [16] G. B. Giannakis, “Cyclostationary signal analysis,” in _Digital Signal Processing Handbook_ , V. K. Madisetti and D. Williams, Eds. CRC press, 1998.
  * [17] W. Gardner, _Cyclostationarity in Communications and Signal Processing_. IEEE Press, 1994.
  * [18] A. Baietto, J. Boubin, P. Farr, T. J. Bihl, A. M. Jones, and C. Stewart, “Lean neural networks for autonomous radar waveform design,” _Sensors_ , vol. 22, no. 4, 2022.
  * [19] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in _MICCAI 2015_. Springer, 2015, pp. 234–241.
  * [20] B. Picinbono and P. Chevalier, “Widely linear estimation with complex data,” _IEEE Trans. Signal Process._ , vol. 43, no. 8, pp. 2030–2033, 1995.
  * [21] F. Chollet _et al._ , “Keras,” https://keras.io, 2015.
  * [22] M. Abadi _et al._ , “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, tensorflow.org.
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. of 3rd Int. Conf. Learn. Represent._ , 2015.
  * [24] MIT RLE, “RF Challenge - AI Accelerator,” accessed 2021-11-01. [Online]. Available: https://rfchallenge.mit.edu