# Auditory Attention Decoding with Task-Related Multi-View Contrastive
Learning

Xiaoyu Chen  0000-0001-8945-3150 Laboratory of Brain Atlas and Brain-Inspired
Intelligence, State Key Laboratory of Multimodal Artificial Intelligence
Systems, Institute of Automation, Chinese Academy of Sciences& School of
Artificial Intelligence, University of Chinese Academy of ScienceBeijingChina
<EMAIL_ADDRESS>,  Changde Du  0000-0002-0084-433X Laboratory of Brain
Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal
Artificial Intelligence Systems, Institute of Automation, Chinese Academy of
SciencesBeijingChina<EMAIL_ADDRESS>,  Qiongyi Zhou  0000-0001-9498-9523
Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key
Laboratory of Multimodal Artificial Intelligence Systems, Institute of
Automation, Chinese Academy of Sciences& School of Artificial Intelligence,
University of Chinese Academy of ScienceBeijingChina<EMAIL_ADDRESS>and  Huiguang He  0000-0002-0684-1711 Laboratory of Brain Atlas and Brain-
Inspired Intelligence, State Key Laboratory of Multimodal Artificial
Intelligence Systems, Institute of Automation, Chinese Academy of Sciences&
School of Artificial Intelligence, University of Chinese Academy of
ScienceBeijingChina<EMAIL_ADDRESS>
(2018)

###### Abstract.

The human brain can easily focus on one speaker and suppress others in
scenarios such as a cocktail party. Recently, researchers found that auditory
attention can be decoded from the electroencephalogram (EEG) data. However,
most existing deep learning methods are difficult to use prior knowledge of
different views (that is attended speech and EEG are task-related views) and
extract an unsatisfactory representation. Inspired by Broadbent’s filter
model, we decode auditory attention in a multi-view paradigm and extract the
most relevant and important information utilizing the missing view.
Specifically, we propose an auditory attention decoding (AAD) method based on
multi-view VAE with task-related multi-view contrastive (TMC) learning.
Employing TMC learning in multi-view VAE can utilize the missing view to
accumulate prior knowledge of different views into the fusion of
representation, and extract the approximate task-related representation. We
examine our method on two popular AAD datasets, and demonstrate the
superiority of our method by comparing it to the state-of-the-art method.

neural decoding, auditory attention decoding, multi-view learning

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

In the acoustic environments people face every day, one’s brain can focus
auditory attention on a particular stimulus while filtering out other stimuli.
For example, people can focus on their interest speaker during a cocktail
party (see Figure 1). This marvel phenomenon, the cocktail party effect
(Cherry, 1953), has attracted long-standing research interest (McDermott,
2009; Conway et al., 2001; Golumbic et al., 2013). And the mechanism behind it
is often called selective auditory attention (Dai et al., 2018; Mesgarani and
Chang, 2012; Golumbic et al., 2013).

Figure 1. In a cocktail party, one can focus on the interested speaker while
ignoring other interference sounds. (Image from The Great Gatsby: Warner Bros.
Pictures and Roadshow Films.)

Recently, with the development of the brain-computer interface (BCI),
researchers are interested in decoding auditory attention through brain
activities, which is known as auditory attention decoding (AAD). Auditory
attention can be decoded from several brain signals, such as
electrocorticography (ECoG) (Mesgarani and Chang, 2012),
magnetoencephalography (MEG) (Ding and Simon, 2012) or electroencephalography
(EEG) (O’Sullivan et al., 2015). Since it is economical and non-invasive,
methods based on EEG have the most promising application potential and may
affect hearing aids and active noise cancellation (ANC) headphones in the
future.

In the dual-speaker scenario, which is the most popular experimental form in
recent AAD research, the subject will hear two different speeches and choose
one speech as the attended one actively or passively (see Figure 2 for an
example). And the task of AAD methods is to infer the subject’s attended
speech based on the EEG and two speeches. Most existing AAD methods resort to
extracting the representation using all the information in the data (Kuruvila
et al., 2021; Cai et al., 2021; Xu et al., 2022). However, the prior knowledge
of the AAD task is that the attended speech and the EEG are two related views,
which contain information about auditory attention. And such a relationship
has been ignored in the existing deep learning AAD methods. According to
Broadbent’s filter model (Broadbent, 1957, 2013), the attentional processing
system in the human brain has an early selection process that uses a selective
filter to avoid unrelated information getting involved in the higher-level
processing. With this filtering mechanism, our human brain can have a
remarkable capability to pay attention to a particular sound source and ignore
surrounding noise, such as focusing on the attended speaker at a cocktail
party intentionally or making conversation with your friends on a noisy train.
Therefore, we argue that the representation should be extracted from the task-
related part of the data.

Inspired by Broadbent’s filter model, we developed our method in a multi-view
structure and filter the unrelated information when fusion the representation.
Specifically, our work refers to the EEG and speeches as different views of
data and decoding auditory attention based on multi-view variational
autoencoder (VAE) (Kingma and Welling, 2013; Sutter et al., 2021; Wu and
Goodman, 2018; Shi et al., 2019). The multi-view VAE will transform the
different views of data into different single-view representations at the
beginning, and fuse them to a common representation space. Then several
decoders will be trained, and map the representation from the common space
back to different views of data. The common space can encode the distribution
of multi-view data effectively after training. Since that, the multi-view VAE
can leverage the underlying relationship between different views of data and
improve the performance of AAD methods.

Figure 2. Auditory attention decoding in the dual-speaker scenario. The
subject will hear two different speeches when during the EEG recording. The
AAD methods can infer the subject’s attention based on the speeches and EEG.

When implementing the multi-view VAE in the AAD task, a critical problem is
how to effectively utilized the prior knowledge about different views of data.
In fact, the information about selective attention is contained in the
attended speech and EEG, which we called task-related representations (or
views). So it is important to retain more information from the task-related
views and minimize the interference of task-unrelated views (unattended
speech) during the fusion of single-view representations. Since the multi-view
VAEs support learning a representation of data with the missing view, a
straightforward thought is fusing the task-related representation on attended
speech and EEG views. But unfortunately, the multi-view VAE needs the fused
common representation to carry out the AAD task, while fusing a task-related
common representation needs the result of AAD. This dilemma causes a great
obstacle in the application of task-related representation in the AAD task. To
solve this problem, we propose task-related multi-view contrastive (TMC)
learning to extract the approximate task-related representation.

The TMC learning consists of two main ideas: 1) utilizing the support of
missing view in multi-view VAE to fuse a task-related representation and 2)
approximate task-related representation using contrastive learning.
Specifically, we first fuse the task-related representation based on the
attended speech and EEG according to the label in the training stage. Since
the label is unavailable in testing, we then fuse a complete representation,
which depends on all the speeches and EEG, and align the complete
representation with the task-related one using contrastive learning. Through
that, the TMC can approximate the task-related representation by the complete
one. Since the fusion of complete representation does not need label
information, we can get an approximate solution to the non-trivial problem
above.

Contributions. Our main contributions are: 1) By applying the multi-view VAE,
we construct our method to exploit the information in the multi-view data and
learn a more comprehensive representation (see figure 3 for an overview of the
framework). 2) We propose task-related multi-view contrastive (TMC) learning
which can utilize the prior knowledge about different views of data to learn
an approximate task-related representation effectively. 3) The experiments
show that our method is comparable to or much better than the state-of-the-art
methods on two popular AAD datasets.

##  2\. Related Work

Figure 3. Overview of our model architecture. (a) The raw EEG and speech
stimuli. (b) Preprocessed EEG and speech stimuli. We use the spectrogram for
speech stimuli and the filter bank feature for the EEG. (c) We use a multi-
view VAE architecture to extract the representation from the EEG and speech
stimuli. (d) The reconstructed inputs. (e) Our classification network is
constructed by a three-layer MLP. (f) The classifier will output a one-hot
vector that indicates the AAD result. (g) The process of extracting filter
bank EEG feature. The raw EEG will be passed through different band-pass
filters to form the filter bank feature.

###  2.1. Auditory Attention Decoding

####  2.1.1. The Traditional Methods.

The traditional correlation-based methods can be divided into two categories,
forward encoding methods and backward decoding methods. The ideas of forward
encoding methods and backward decoding methods are consistent with neural
encoding and decoding, that is, predicting the brain activities (EEG) given
stimuli, or reconstructing the stimuli given brain activities (Geirnaert et
al., 2021). In the forward encoding methods, an encoder will be trained to
infer the EEG given different speech signals (Alickovic et al., 2019; Wong et
al., 2018) and correlated to the real EEG to decide the attended speech. While
for the backward decoding methods (de Cheveigné et al., 2018; Das et al.,
2018; Katthi et al., 2020; De Cheveigné et al., 2021) , a stimulus
reconstruction pattern is wildly accepted (O’Sullivan et al., 2015; Mirkovic
et al., 2015; Biesmans et al., 2016). The envelope of attended speech will be
reconstructed from the EEG, and then compare with all the speeches using the
Pearson correlation coefficient. The traditional methods are mostly based on a
linear model, which fails to capture the nonlinear characteristics of the
human auditory system (Zwicker and Fastl, 2013).

####  2.1.2. Deep Learning Methods.

With the applications of deep learning in the brain-computer interface, many
works decoded auditory attention using deep neural networks. Although the
stimulus reconstruction pattern can be easily transferred to the deep learning
AAD methods (de Taillez et al., 2020), most of the works choose a more direct
and end-to-end way, i.e. classify the speeches directly (Ciccarelli et al.,
2019; Vandecappelle et al., 2021; Cai et al., 2021; Kuruvila et al., 2021; Su
et al., 2022). For example, in (Cai et al., 2021) and (Su et al., 2022), the
authors build different attention mechanisms and apply them to the channel,
band, or temporal of EEG to extract effective representation for AAD. However,
these methods do not accumulate the prior knowledge of different views in the
extraction of representation. As we mentioned before, the task-related
information is contained in the attended speech and EEG, and ignoring such
prior knowledge will hinder the performance of AAD methods. Different from
those methods, we propose a multi-view auditory decoding method based on
multi-view VAE, and use the TMC learning to accumulate the prior knowledge in
the fusion of representation and learn an approximate task-related
representation.

###  2.2. Multi-View VAEs

Recently, there has been a research interest in using VAE for self-supervised
multi-view generative models, and produced a lot of important research
progress (Kurle et al., 2019; Wu and Goodman, 2018; Shi et al., 2019). The
greatest advantage of multi-view VAEs is that they can infer the complete
representation given incomplete views of data. And the fundamental difference
between these works is in the formulation of constructing the complete
representation space, i.e. the complete posteriors. In MVAE (Wu and Goodman,
2018), the researchers use a product of single-view posteriors (Product-of-
Experts, PoE (Hinton, 2002)) to formulate the complete posterior. While in
MMVAE (Shi et al., 2019), the complete posterior is formulated using a mixture
of single-view posteriors (Mixture-of-Experts, MoE). After that, several works
have been proposed to improve the performance of MVAE and MMVAE (Wu and
Goodman, 2019; Kurle et al., 2019; Sutter et al., 2020; Shi et al., 2020;
Daunhawer et al., 2021). In order to effectively combine the advantages of
MVAE and MMVAE, MoPoE-VAE (Sutter et al., 2021) use the Mixture-of-Products-
of-Experts (MoPoE) which first conducts PoE on subsets of complete views and
then form the complete posterior using MoE on these subsets.

##  3\. Methodology

The primary purpose of AAD is to find the attended speaker in multiple
speakers. We design our method based on the prior knowledge that the
information about the attended speaker is contained in EEG and the attended
speech. In contrast, unattended speech is unrelated to the goal when decoding
auditory attention. Based on this idea, the main challenges are: 1) The method
we use to construct the representation space; 2) How does our method reduce
the interference of unrelated information while retaining the task-related
information during the training? We will specify our method to address these
two challenges in the following subsections.

###  3.1. Decoding Auditory Attention with Multi-View VAE

We construct our representation space using multi-view VAE. Specifically, we
consider the EEG and speech stimuli (both the attended and unattended ones) as
different views of data that may contain information about the subject’s
auditory attention, and use multi-view VAE to fuse different views into a
common representation space. The overview of our method architecture is
illustrated in figure 3.

Given the raw EEG and speech stimuli, we extract different features from EEG
and speech stimuli in the preprocessing stage. We extract the speech
spectrogram using the short-time Fourier transform (STFT) from the lowpass-
filtered raw speech stimuli. While for the EEG signal, we extract different
frequency bands to construct a more comprehensive feature. We consider five
EEG bands including the
$\delta(1-4\text{Hz}),\theta(4-8\text{Hz}),\alpha(8-12\text{Hz}),\beta(12-30\text{Hz}),\text{and
low }\gamma(30-50\text{Hz})$ (Buzsaki and Draguhn, 2004). The detailed
implementation of the data preprocessing can be found in Section 4.

After data preprocessing, we mapped the different views of data into different
single-view posteriors. All the single-view posteriors will be fused into a
complete-view posterior $q_{\phi}(z|e,s_{1},s_{2})$ using Mixture-of-Products-
of-Experts(MoPoE) (Sutter et al., 2021). Specifically, let $q_{\phi}(z|e)$
denote the posterior given EEG, $q_{\phi}(z|s_{1})$ and $q_{\phi}(z|s_{2})$
denote the posterior given different speeches, the MoPoE compute the complete
posterior $q_{\phi}(z|e,s_{1},s_{2})$ as:

(1) $\displaystyle
q_{\phi}(z|\mathbf{X}_{k})=p_{\theta}(z)\prod\limits_{m=1}^{N_{k}}q_{\phi}(z|x_{m}),\quad
x_{m}\in\mathbf{X}_{k},$ (2) $\displaystyle
q_{\phi}(z|e,s_{1},s_{2})=\frac{1}{K}\sum\limits_{k=1}^{K}q_{\phi}(z|\mathbf{X}_{k}),$

where $\mathbf{X}_{k}$ is a subset of complete view $\\{e,s_{1},s_{2}\\}$
which has $N_{k}$ element, and $p_{\theta}(z)\sim\mathcal{N}(0,I)$ is an
isotropic Gaussian.

Figure 4. An example of TMC learning between two multi-view samples $x^{i}$
and $x^{j}$. We set the positive pair as the complete representation and the
task-related representation in the same sample, and set the negative pair
between different samples.

The samples from the complete posterior will be fed into three decoders to
reconstruct the original input, and the multi-view VAE will be trained by
maximizing the evidence lower bound (ELBO):

$\displaystyle L_{ELBO}=\mathbf{E}_{z\sim
q_{\phi}(z|e,s_{1},s_{2})}[\log(p_{\theta}(e,s_{1},s_{2}|z))]$ (3)
$\displaystyle\qquad\qquad-D_{KL}(q_{\phi}(z|e,s_{1},s_{2})\|p_{\theta}(z)),$

where the $D_{KL}(\cdot)$ is the KL-divergence that is used to measure the
statistical distance between complete posterior and the isotropic Gaussian
$p_{\theta}(z)$.

The multi-view VAE can extract powerful representation in a self-supervision
way. To approach the auditory attention in the representation space
constructed by multi-view VAE, we apply a simple classifier in the complete
representation space to separate samples with different attended speeches. The
classifier contains a 3-layer MLP, which can map the samples from the complete
posterior into a one-hot vector that indicates the attended speech in the
input speeches.

We minimized the binary cross entropy (BCE) loss for the classifier during the
training. Let the $\mathcal{C}(z)$ denote the classification result, and y is
the label related to it, we compute classification loss as:

$\displaystyle L_{\mathcal{C}}=$ $\displaystyle-\mathbf{E}_{z\sim
q_{\phi}(z|e,s_{1},s_{2})}\left[y\log(\mathcal{C}(z))\right.$ (4)
$\displaystyle\left.+(1-y)\log(1-\mathcal{C}(z))\right].$

###  3.2. Approximate the Task-Related Representation Using TMC

Although the multi-view VAE can extract powerful representation from the
complete view data, it has an inherent drawback in representation fusion. In
fact, as we mentioned before, even if the unattended speech has task-unrelated
information, we have no choice but to include it in the complete
representation. Since removing the task-unrelated unattended view needs the
information from the label that is not available in the testing stage, it is
impossible to get the task-related representations (posterior given EEG and
attended speech) without ground-truth.

To solve this problem and make better use of prior knowledge, we propose TMC
learning that encourages multi-view VAE to learn an approximate task-related
representation. A simple TMC instantiated between two multi-view samples
$x^{i}$ and $x^{j}$ is shown in Figure 4.

####  3.2.1. Task-Related Multi-View Contrastive (TMC) Learning.

In general, we use $\\{x^{i}_{m}\\}_{m=1}^{M}=\mathbf{X}^{i}$ to denote a
sample from the general multi-view data which have M views. Moreover, we
assume that there is a subset of views $\\{x^{i}_{u}\\}_{u\in
U}=\mathbf{X}^{i}_{U}\subset\mathbf{X}^{i}$ that is task-unrelated. We refer
to the single-view representations $z^{i}_{m}=f(x^{i}_{m})$ for the
presentations extracted from the single-view data, and using
$z^{i}_{c}=f(\mathbf{X}^{i})$ for the complete representation which extracts
from all the given views. Also, with the information from the gound-truth of
the task, we have the task-related representation
$z^{i}_{t}=f(\mathbf{X}^{i}_{T})$ which is fused by the representations from
several single-views $\\{x^{i}_{t}\\}_{t\in
T}=\mathbf{X}^{i}_{T}\subseteq\mathbf{X}$ that are related to the task.

TMC uses contrastive learning to align the complete representation to the
task-related one. Specifically, we compute the similarity of positive pair as:

(5) $S_{p}=s_{c,t}(i,i)=exp(sim(z_{c}^{i},z_{t}^{i})/\tau),$

where we choose cosine for $sim(\cdot,\cdot)$ and $\tau$ is the temperature
hyperparameter.

And for the negative pairs, we set the similarity between two different
samples as the negative one. We consider the similarity between the complete
representation and the task-related one as:

(6) $S_{n}=s_{c,c}(i,j)+s_{c,t}(i,j)+s_{t,t}(i,j),i\neq j.$

So the TMC loss has the following form:

(7) $L_{\mathcal{TMC}}=-\mathbf{E}_{z}\left[\log\frac{S_{p}}{S_{n}}\right].$

####  3.2.2. Approximation Task-Related Representation.

As we mentioned before, the ideology task-related representation is
unavailable in the testing stage, but it can be approximated through the
complete representation by using the TMC learning.

Specifically, in the AAD task, where we use multi-view VAE as the backbone
network, the single-view representation $z^{i}_{m}=f(x^{i}_{m})$ are sampled
from the single-view posteriors learned by encoders related to different
views:

(8) $z^{i}_{m}\sim q_{\phi}(z|x_{m}),x_{m}\in\\{e,s_{1},s_{2}\\}.$

Take the $s_{1}$ as the attended speech for an example, the complete
representation $z^{i}_{c}=f(\mathbf{X}^{i})$ and task-related representation
$z^{i}_{t}=f(\mathbf{X}^{i}_{T})$ are extracted by the multi-view VAE from
different fused posteriors:

(9) $\displaystyle z^{i}_{c}\sim q_{\phi}(z|e,s_{1},s_{2}),$ (10)
$\displaystyle z^{i}_{t}\sim q_{\phi}(z|e,s_{1}).$

And the TMC can encourage the multi-view VAE to approximate the task-related
representation by aligning the complete representation to the task-related
one, which is fused by attended speech and EEG. We implement that by using the
TMC loss to joint training multi-view VAE. So the loss function we used for
the AAD task in our method is:

(11) $L=-L_{ELBO}+\alpha L_{\mathcal{C}}+\beta L_{\mathcal{TMC}},$

the $\alpha$ and $\beta$ are the weights of classification loss and TMC loss.

Although we propose TMC learning for the AAD task, we must point out that TMC
is a general learning method. And the intuitive idea behind TMC can be applied
to any multi-view data which bothered by the task-unrelated views.

In the implementation, we take the advantage of MoPoE that MoE and PoE are
special cases of MoPoE. Specifically, when we constraint all the subsets of
complete view only have single-view
$\mathbf{X}_{1}=\\{e\\},\mathbf{X}_{2}=\\{s_{1}\\}\mathbf{X}_{3}=\\{s_{2}\\}$,
we can have the MoE posterior:

(12)
$q_{\phi}(z|e,s_{1},s_{2})=\frac{1}{3}\sum\limits_{m=1}^{3}q_{\phi}(z|x_{m}),x_{m}\in\\{e,s_{1},s_{2}\\},$

and when we constraint the MoPoE to have only one subset which is the complete
view itself, we can have the PoE posterior:

(13)
$q_{\phi}(z|e,s_{1},s_{2})=p_{\theta}(z)\prod\limits_{m=1}^{3}q_{\phi}(z|x_{m}),x_{m}\in\\{e,s_{1},s_{2}\\}.$

In Section 4, we give a thorough evaluation of our TMC learning with different
fusion methods of multi-view VAE.

Figure 5. A topographic map of the EEG filter bank feature we used.

##  4\. Experiments

###  4.1. Experiment Setup

####  4.1.1. AAD Datasets.

We test our method on two popular AAD datasets. The first one is the KUL
dataset (Das et al., 2019) which collects EEG data from 16 normal-hearing
subjects in a soundproof and electromagnetically shielded room. The EEG data
are collected by a 64-channel BioSemi ActiveTwo system at 8196 Hz sample rate.
The stimuli are Dutch short stories narrated by different male speakers. To
help the subjects focus on the experiment, the KUL dataset truncates the
silences longer than 500 ms to 500 ms. The stimuli have two presentation
conditions, HRTF (head-related transfer function) and dry. In the HRTF,
stimuli applied to the subject’s left and right ears are simulated by HRTF
functions. While in the dry condition, the different story tracks are
presented separately in the left ear or the right ear. We use the dry
condition in our experiments, which has 4 trials with 6 mins duration for
every subject.

The DTU dataset (Fuglsang et al., 2018) contains EEG data from 18 subjects who
take the experiment in a soundproof room. The EEG data are collected by a
64-channel BioSemi ActiveTwo system at 512 Hz sample rate. Different from the
KUL dataset, the DTU datasets use Danish speeches narrated by a male and a
female speaker. Every subject will experience 60 trials of speech stimuli, and
every trial last for 50 s.

####  4.1.2. Data Preprocessing.

The speech stimuli are filtered and down-sampled before extracting the
spectrogram. We first pass the speeches through a Chebyshev (type II) low-pass
filter with 8 kHz cut-off frequency and downsampled the speeches to 16 kHz.
Then we split the speeches into many decision windows and extract the
spectrogram using the short-time Fourier transform with 32 ms Hann window and
12 ms hop length.

For the EEG signal, we form a 3D filter bank by passing the EEG signal into
the different Chebyshev (type II) band-pass filters, and concatenate the
different frequency bands in one tensor. We use frequency bands of 1-4 Hz, 4-8
Hz, 8-12 Hz, 12-30 Hz, and 30-50 Hz which are known as $\delta$, $\theta$,
$\alpha$, $\beta$, and low-$\gamma$ bands in EEG (Buzsaki and Draguhn, 2004).
While for the EEG channel, we follow the Joint CNN-LSTM (Kuruvila et al.,
2021) to use F7, F3, F4, F8, T7, C3, Cz, C4, T8, Pz instead of all the
electrode. We refer to figure 5 for the topographic map of the EEG filter bank
features.

In our experiments, we use two different decision window settings, 2 s and 3
s. With the longer decision window, the signal will contain more auditory
attention information. Since our method is based on deep learning, we adopt
the data augmentation by adding overlap between two windows. The overlap is
set to 1 s for the 2 s decision window and 2 s for the 3 s decision window.
The data volume after performing data augmentation is listed in Table 1.

Table 1. Data volume of two datasets after data augmentation. Dataset | 2s | 3s
---|---|---
train | val | test | train | val | test
KUL | 18719 | 3120 | 3120 | 18719 | 3120 | 3120
DTU | 38879 | 6480 | 6480 | 38879 | 6480 | 6480
|  |  |  |  |  |

Since we use the same hop length for different decision windows, the total
amount of data under different decision window lengths is the same. This
setting can eliminate the impact on the performance caused by training data
volume in different decision window lengths, especially for deep learning
methods where data is a critical factor.

####  4.1.3. Network Settings.

Our method is implemented based on Pytorch (Paszke et al., 2019). For the
encoder, we adapt the CNN part from Joint CNN-LSTM (Kuruvila et al., 2021),
which uses 4 convolution layers for the EEG encoder and 5 for the speech
encoder. We add one common linear layer and two private linear layers after
the CNNs for the mean and variance of single-view posteriors. While for the
decoder, we use a linear layer and several deconvolution layers (the same
number as the single-view encoder) to reverse the process of encoding. For the
classification part, we use a 3 layers MLP as our classifier.

We keep the network architecture identical in different decision windows. When
decoding the auditory attention in 2 s decision windows, we just repeat and
truncate the 2 s signal to make the input of the encoders have a 3 s length.

####  4.1.4. Parameter Settings.

For all the experiments, we use $\alpha=1$, $\beta=1$ for the weight of
classification loss and TMC loss, and set the temperature hyperparameter
$\tau=1.5$. Moreover, for the dimension of representation learned by multi-
view VAE, we use 128-dim for all the posterior fusion methods and conducted
the experiment on a batch of 128 samples.

###  4.2. Quantitative and Qualitative Results.

In this part, we first evaluate the performance of task-related representation
in the testing stage. Then we compare our method to several previous works in
different decision windows on both datasets. We also include the MoPoE-VAE
(Sutter et al., 2021) and make a comparison of the representation similarity
to evaluate the effectiveness of our TMC learning. After that, we take close
scrutiny to our TMC learning by evaluating TMC learning with different fusion
methods. In all the tables except Table 6, * denotes the TMC-VAE performance
is significantly better than the compared method (one-tailed unpaired t-test,
p¡0.05).

(a) KUL 3s.

(b) KUL 2s.

(c) DTU 3s.

(d) DTU 2s.

Figure 6. Representation visualization of task-related representation. The
$\cdot$ and $\times$ in the figure denote samples with different attended
speeches.

####  4.2.1. Evaluation of Task-Related Representation.

To verify the reliability of our ideas, we first evaluate the decoding
performance using task-related representation. We evaluate the performance in
two aspects: 1) the decoding accuracy and 2) the visualization of task-related
representation.

We use the label to construct task-related representation and classify the
auditory attention based on it. The decoding performances are shown in Table
2. We find that the task-related representation can yield $100\%$ accuracy on
both testing sets. We also visualize the task-related representation in the
testing stage. As is shown in Figure 6, samples with different attended
speeches are well separated in the task-related representation space.

Even though these results can not prove the performance of our method, the
superior separation in the task-related representation space supports our
motivation, which aims to construct an approximate task-related
representation.

Table 2. Decoding the auditory attention using task-related representation.  Dataset | 2s | 3s
---|---|---
KUL | 100% | 100%
DTU | 100% | 100%

(a) MVAE.

(b) MMVAE.

(c) MoPoE-VAE.

(d) MVAE+TMC.

(e) MMVAE+TMC.

(f) TMC-VAE.

Figure 7. Representation visualization of different multi-view VAE with or
without TMC learning. We perform all the visualization on the KUL dataset with
a 3 s decision window. In the first row, we present the visualization of
representation without TMC learning. We use blue for the task-related
representations and red for the complete ones. The $\cdot$ and $\times$ in the
figure denote samples with different attended speeches.

####  4.2.2. Comparative Study

In this part, we compare our method to several existing AAD methods. The joint
CNN-LSTM (Kuruvila et al., 2021) uses a convolution network (CNN)
collaborative with a long short-term memory (LSTM) (Hochreiter and
Schmidhuber, 1997) to identify the attended speech in two input speeches. Deep
CCA (Katthi et al., 2020) performs the correlation base AAD using the deep
neural network with regularization. CNN-FC (Cai et al., 2021) learn a
discriminative representation for AAD by using the attention mechanism in
their network, which is the state-of-the-art method on the KUL and the DTU
datasets according to our knowledge. And the MoPoE-VAE refers to using the
same settings as our method but without TMC loss. We choose MoPoE as the
fusion method of our approach in this study and use TMC-VAE to represent this
configuration.

Table 3. Accuracy on the KUL dataset in different decision windows. Method | 2s | 3s
---|---|---
Joint CNN-LSTM | $72.7\%_{*}$ | $78.2\%_{*}$
Deep CCA | $66.4\%_{*}$ | $68.1\%_{*}$
CNN-FC | 86.9% | $89.7\%_{*}$
MoPoE-VAE | $83.6\%_{*}$ | $\underline{95.6\%}_{*}$
TMC-VAE | 85.5% | 96.6%

We report the results in 2 s and 3 s decision windows on the KUL dataset in
table 3. Our TMC-VAE yields state-of-the-art result on the KUL dataset under 3
s decision window. Moreover, our TMC-VAE outperforms the existing methods by a
large margin. Compared with the joint CNN-LSTM (Kuruvila et al., 2021), which
has the same CNN encoder and more robust sequence data modeling capability
with its LSTM module (Hochreiter and Schmidhuber, 1997), TMC-VAE can improve
the decoding performance by $18.4\%$. Also, the comparison between MoPoE-VAE
and existing methods can prove the advantages of using multi-view VAE to
decode auditory attention. We notice that our method has a performance drop
under a smaller decision window, but we must point out that our main
contribution is using multi-view VAE and TMC learning to learn an approximate
task-related representation, rather than carefully designing the networks. And
our method can be easily adopted with a well-designed VAE backbone.

We present the results on the DTU dataset in table 4. The TMC-VAE also
outperforms existing methods with a large margin under 3 s decision window.
Although we use ordinary CNN architecture in our encoders and decoders, our
method can perform comparable results with elaborately designed attention-
based architecture (CNN-FC (Cai et al., 2021)) under 2 s decision window.
Also, in both datasets and all the decision windows, the comparison between
the MoPoE-VAE and our method can demonstrate the effectiveness of TMC
learning.

Table 4. Accuracy on the DTU dataset in different decision windows. Method | 2s | 3s
---|---|---
Joint CNN-LSTM | $54.1\%_{*}$ | $55.6\%_{*}$
Deep CCA | $58.9\%_{*}$ | $60.7\%_{*}$
CNN-FC | 82.9% | $86.4\%_{*}$
MoPoE-VAE | $78.4\%_{*}$ | $\underline{91.5\%}_{*}$
TMC-VAE | 80.8% | 92.1%

####  4.2.3. Representation Similarity.

To make a better evaluation of the TMC learning and validate the performance
improvement of TMC-VAE originates from the approximate representation, we
compare the similarity between the task-related representation and the
approximate one (complete representation) using the cosine similarity. We use
the representations from two models in the testing stage in this part: 1) the
MoPoE-VAE which trained without the TMC learning and 2) the TMC-VAE which
trained with the TMC learning. The results are shown in Table 5 in which we
can find that the similarity between the complete representation and task-
related representation is increased significantly.

Table 5. Representation similarity. Method | KUL 2s | KUL 3s | DTU 2s | DTU 3s
---|---|---|---|---
MoPoE-VAE | $0.037_{*}$ | $0.041_{*}$ | $0.008_{*}$ | $0.008_{*}$
TMC-VAE | 0.356 | 0.361 | 0.311 | 0.291

####  4.2.4. Effectiveness of TMC with Different Fusion Methods.

To make a thorough study of the effectiveness of our TMC learning, we present
the ablation study of TMC learning with different multi-view VAEs. We choose
three typical multi-view VAEs here: 1) MVAE (Wu and Goodman, 2018) which uses
PoE to fuse the single-view posteriors 2) MMVAE (Shi et al., 2019) which
proposes MoE in the fusion of complete posteriors and 3) MoPoE-VAE (Sutter et
al., 2021) which take the advantages from PoE and MoE, and proposed a general
fusion modal.

Table 6. AAD accuracy of different fusion methods on the KUL and the DTU datasets. * denotes the performance with TMC is significantly better than without TMC (one-tailed unpaired t-test, p¡0.05). Method | KUL | DTU
---|---|---
2s | 3s | 2s | 3s
MVAE | $83.6\%_{*}$ | $93.9\%_{*}$ | $77.1\%_{*}$ | $89.9\%_{*}$
MVAE+TMC | 84.5% | 94.3% | 78.2% | 90.7%
MMVAE | $84.4\%_{*}$ | $92.7\%_{*}$ | $78.3\%_{*}$ | $87.7\%_{*}$
MMVAE+TMC | 84.7% | 93.1% | 78.8% | 88.4%
MoPoE-VAE | $83.6\%_{*}$ | $95.6\%_{*}$ | $78.4\%_{*}$ | $91.5\%_{*}$
TMC-VAE | 85.5% | 96.6% | 80.8% | 92.1%

It is shown in table 6 that TMC learning can encourage multi-view VAE to learn
an approximate task-related representation with different posterior fusion
formulas. Also, the TMC-VAE yields the best results, which also gives
quantitative support for the advantage of choosing MoPoE in TMC-VAE.

####  4.2.5. Representation Visualization.

In this part of the study, we present some qualitative results in figure 7 to
give intuitive evidence of the effectiveness of TMC learning. Specifically, we
visualize the representation learned by different multi-view VAEs with or
without TMC learning. We use the same multi-view VAEs in Section 4.2.4.

All the visualizations are yields in the testing stage on the KUL dataset
under a 3 s decision window. To perform the visualizations, we first feed a
batch of data into the trained encoders to extract the 128-dim
representations, and then map these representations to 2-dim using t-SNE (Van
der Maaten and Hinton, 2008). We apply blue for the task-related
representations and red for the complete ones, and distinguish the samples
with different attended speeches using circle and cross.

As shown in figure 7, the task-related representations (in blue) are more
separable in the representation space than the complete one (in red). While
TMC learning can encourage the multi-view VAEs to learn a more separable
complete representation by aligning it to the task-related one. We also
observed that the TMC-VAE has the most prominent consistency and separability
in all the visualizations, which suggests that MoPoE is the optimal choice of
the fusion formula in our method. We report the decoding performance of
different posterior fusion methods in the next part study.

##  5\. Conclusions.

In this work, we first introduce a multi-view VAE and a classifier to learn
the multi-view representation for AAD efficiently. Then, inspired by
Broadbent’s filter model, we define the task-related representation in the AAD
task, and propose the TMC learning to encourage the complete representation
aligning with the task-related one. Finally, the experiments on the KUL and
the DTU datasets prove the advantages of our method.

###### Acknowledgements.

This work was supported in part by the Scientific and Technological Innovation
(STI) 2030–Major Projects under Grant 2021ZD0201503; in part by the National
Natural Science Foundation of China under Grant 82272072; and in part by the
CAAI-Huawei MindSpore Open Fund.

## References

  * (1)
  * Alickovic et al. (2019) Emina Alickovic, Thomas Lunner, Fredrik Gustafsson, and Lennart Ljung. 2019.  A tutorial on auditory attention identification methods.  _Frontiers in neuroscience_ (2019), 153.
  * Biesmans et al. (2016) Wouter Biesmans, Neetha Das, Tom Francart, and Alexander Bertrand. 2016.  Auditory-inspired speech envelope extraction methods for improved EEG-based auditory attention detection in a cocktail party scenario.  _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ 25, 5 (2016), 402–412.
  * Broadbent (1957) Donald Eric Broadbent. 1957\.  A mechanical model for human attention and immediate memory.  _Psychological review_ 64, 3 (1957), 205.
  * Broadbent (2013) Donald Eric Broadbent. 2013\.  _Perception and communication_.  Elsevier.
  * Buzsaki and Draguhn (2004) Gyorgy Buzsaki and Andreas Draguhn. 2004.  Neuronal oscillations in cortical networks.  _science_ 304, 5679 (2004), 1926–1929.
  * Cai et al. (2021) Siqi Cai, Enze Su, Longhan Xie, and Haizhou Li. 2021\.  EEG-Based Auditory Attention Detection via Frequency and Channel Neural Attention.  _IEEE Transactions on Human-Machine Systems_ 52, 2 (2021), 256–266.
  * Cherry (1953) E Colin Cherry. 1953\.  Some experiments on the recognition of speech, with one and with two ears.  _The Journal of the acoustical society of America_ 25, 5 (1953), 975–979.
  * Ciccarelli et al. (2019) Gregory Ciccarelli, Michael Nolan, Joseph Perricone, Paul T Calamia, Stephanie Haro, James O’Sullivan, Nima Mesgarani, Thomas F Quatieri, and Christopher J Smalt. 2019.  Comparison of two-talker attention decoding from EEG with nonlinear neural networks and linear methods.  _Scientific reports_ 9, 1 (2019), 1–10.
  * Conway et al. (2001) Andrew RA Conway, Nelson Cowan, and Michael F Bunting. 2001\.  The cocktail party phenomenon revisited: The importance of working memory capacity.  _Psychonomic bulletin & review_ 8 (2001), 331–335.
  * Dai et al. (2018) Bohan Dai, Chuansheng Chen, Yuhang Long, Lifen Zheng, Hui Zhao, Xialu Bai, Wenda Liu, Yuxuan Zhang, Li Liu, Taomei Guo, et al. 2018\.  Neural mechanisms for selectively tuning in to the target speaker in a naturalistic noisy situation.  _Nature communications_ 9, 1 (2018), 2405.
  * Das et al. (2018) Neetha Das, Alexander Bertrand, and Tom Francart. 2018\.  EEG-based auditory attention detection: boundary conditions for background noise and speaker positions.  _Journal of neural engineering_ 15, 6 (2018), 066017\.
  * Das et al. (2019) Neetha Das, Tom Francart, and Alexander Bertrand. 2019.  Auditory attention detection dataset KULeuven.  _Zenodo_ (2019).
  * Daunhawer et al. (2021) Imant Daunhawer, Thomas M Sutter, Ričards Marcinkevičs, and Julia E Vogt. 2021.  Self-supervised disentanglement of modality-specific and shared factors improves multimodal generative models. In _DAGM German Conference on Pattern Recognition_. Springer, 459–473.
  * De Cheveigné et al. (2021) Alain De Cheveigné, Malcolm Slaney, Søren A Fuglsang, and Jens Hjortkjaer. 2021.  Auditory stimulus-response modeling with a match-mismatch task.  _Journal of Neural Engineering_ 18, 4 (2021), 046040\.
  * de Cheveigné et al. (2018) Alain de Cheveigné, Daniel DE Wong, Giovanni M Di Liberto, Jens Hjortkjaer, Malcolm Slaney, and Edmund Lalor. 2018.  Decoding the auditory brain with canonical component analysis.  _NeuroImage_ 172 (2018), 206–216.
  * de Taillez et al. (2020) Tobias de Taillez, Birger Kollmeier, and Bernd T Meyer. 2020\.  Machine learning for decoding listeners’ attention from electroencephalography evoked by continuous speech.  _European Journal of Neuroscience_ 51, 5 (2020), 1234–1241.
  * Ding and Simon (2012) Nai Ding and Jonathan Z Simon. 2012.  Neural coding of continuous speech in auditory cortex during monaural and dichotic listening.  _Journal of neurophysiology_ 107, 1 (2012), 78–89.
  * Fuglsang et al. (2018) SA Fuglsang, DDE Wong, and J Hjortkjær. 2018.  EEG and audio dataset for auditory attention decoding.  _Zenodo_ (2018).
  * Geirnaert et al. (2021) Simon Geirnaert, Servaas Vandecappelle, Emina Alickovic, Alain de Cheveigne, Edmund Lalor, Bernd T Meyer, Sina Miran, Tom Francart, and Alexander Bertrand. 2021.  Electroencephalography-based auditory attention decoding: Toward neurosteered hearing devices.  _IEEE Signal Processing Magazine_ 38, 4 (2021), 89–102.
  * Golumbic et al. (2013) Elana M Zion Golumbic, Nai Ding, Stephan Bickel, Peter Lakatos, Catherine A Schevon, Guy M McKhann, Robert R Goodman, Ronald Emerson, Ashesh D Mehta, Jonathan Z Simon, et al. 2013\.  Mechanisms underlying selective neuronal tracking of attended speech at a “cocktail party”.  _Neuron_ 77, 5 (2013), 980–991.
  * Hinton (2002) Geoffrey E Hinton. 2002\.  Training products of experts by minimizing contrastive divergence.  _Neural computation_ 14, 8 (2002), 1771–1800.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Katthi et al. (2020) Jaswanth Reddy Katthi, Sriram Ganapathy, Sandeep Kothinti, and Malcolm Slaney. 2020.  Deep canonical correlation analysis for decoding the auditory brain. In _2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_. IEEE, 3505–3508.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ (2013).
  * Kurle et al. (2019) Richard Kurle, Stephan Günnemann, and Patrick Van der Smagt. 2019\.  Multi-source neural variational inference. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 4114–4121.
  * Kuruvila et al. (2021) Ivine Kuruvila, Jan Muncke, Eghart Fischer, and Ulrich Hoppe. 2021.  Extracting the auditory attention in a dual-speaker scenario from EEG using a joint CNN-LSTM model.  _Frontiers in Physiology_ 12 (2021).
  * McDermott (2009) Josh H McDermott. 2009\.  The cocktail party problem.  _Current Biology_ 19, 22 (2009), R1024–R1027.
  * Mesgarani and Chang (2012) Nima Mesgarani and Edward F Chang. 2012.  Selective cortical representation of attended speaker in multi-talker speech perception.  _Nature_ 485, 7397 (2012), 233–236.
  * Mirkovic et al. (2015) Bojana Mirkovic, Stefan Debener, Manuela Jaeger, and Maarten De Vos. 2015.  Decoding the attended speech stream with multi-channel EEG: implications for online, daily-life applications.  _Journal of neural engineering_ 12, 4 (2015), 046007\.
  * O’Sullivan et al. (2015) James A O’Sullivan, Alan J Power, Nima Mesgarani, Siddharth Rajaram, John J Foxe, Barbara G Shinn-Cunningham, Malcolm Slaney, Shihab A Shamma, and Edmund C Lalor. 2015\.  Attentional selection in a cocktail party environment can be decoded from single-trial EEG.  _Cerebral cortex_ 25, 7 (2015), 1697–1706.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019).
  * Shi et al. (2019) Yuge Shi, Brooks Paige, Philip Torr, et al. 2019\.  Variational mixture-of-experts autoencoders for multi-modal deep generative models.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Shi et al. (2020) Yuge Shi, Brooks Paige, Philip HS Torr, and N Siddharth. 2020\.  Relating by contrasting: A data-efficient framework for multimodal generative models.  _arXiv preprint arXiv:2007.01179_ (2020).
  * Su et al. (2022) Enze Su, Siqi Cai, Longhan Xie, Haizhou Li, and Tanja Schultz. 2022.  STAnet: A spatiotemporal attention network for decoding auditory spatial attention from EEG.  _IEEE Transactions on Biomedical Engineering_ 69, 7 (2022), 2233–2242.
  * Sutter et al. (2020) Thomas Sutter, Imant Daunhawer, and Julia Vogt. 2020\.  Multimodal generative learning utilizing jensen-shannon-divergence.  _Advances in Neural Information Processing Systems_ 33 (2020), 6100–6110.
  * Sutter et al. (2021) Thomas M Sutter, Imant Daunhawer, and Julia E Vogt. 2021\.  Generalized multimodal ELBO.  _arXiv preprint arXiv:2105.02470_ (2021).
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Vandecappelle et al. (2021) Servaas Vandecappelle, Lucas Deckers, Neetha Das, Amir Hossein Ansari, Alexander Bertrand, and Tom Francart. 2021.  EEG-based detection of the locus of auditory attention with convolutional neural networks.  _Elife_ 10 (2021), e56481.
  * Wong et al. (2018) Daniel DE Wong, Søren A Fuglsang, Jens Hjortkjær, Enea Ceolini, Malcolm Slaney, and Alain De Cheveigne. 2018.  A comparison of regularization methods in forward and backward models for auditory attention decoding.  _Frontiers in neuroscience_ 12 (2018), 531.
  * Wu and Goodman (2018) Mike Wu and Noah Goodman. 2018.  Multimodal generative models for scalable weakly-supervised learning.  _Advances in Neural Information Processing Systems_ 31 (2018).
  * Wu and Goodman (2019) Mike Wu and Noah Goodman. 2019.  Multimodal generative models for compositional representation learning.  _arXiv preprint arXiv:1912.05075_ (2019).
  * Xu et al. (2022) Zihao Xu, Yanru Bai, Ran Zhao, Qi Zheng, Guangjian Ni, and Dong Ming. 2022\.  Auditory attention decoding from EEG-based Mandarin speech envelope reconstruction.  _Hearing Research_ 422 (2022), 108552.
  * Zwicker and Fastl (2013) Eberhard Zwicker and Hugo Fastl. 2013.  _Psychoacoustics: Facts and models_. Vol. 22.  Springer Science & Business Media.