# The Brain’s Bitter Lesson: Scaling Speech Decoding With Self-Supervised
Learning

Dulhan Jayalath1 Gilad Landau1 Brendan Shillingford2
Mark Woolrich3 Oiwi Parker Jones1
$\\{^{1}$PNPL, 3OHBA$\\}$, University of Oxford 2Google DeepMind
{dulhan<EMAIL_ADDRESS>
###### Abstract

The past few years have produced a series of spectacular advances in the
decoding of speech from brain activity. The engine of these advances has been
the acquisition of labelled data, with increasingly large datasets acquired
from single subjects. However, participants exhibit anatomical and other
individual differences, and datasets use varied scanners and task designs. As
a result, prior work has struggled to leverage data from multiple subjects,
multiple datasets, multiple tasks, and unlabelled datasets. In turn, the field
has not benefited from the rapidly growing number of open neural data
repositories to exploit large-scale data and deep learning. To address this,
we develop an initial set of neuroscience-inspired self-supervised objectives,
together with a neural architecture, for representation learning from
heterogeneous and unlabelled neural recordings. Experimental results show that
representations learned with these objectives scale with data, generalise
across subjects, datasets, and tasks, and are also learned faster than using
only labelled data. In addition, we set new benchmarks for two foundational
speech decoding tasks. Taken together, these methods now unlock the potential
for training speech decoding models with orders of magnitude more existing
data.

Figure 1: Leveraging unlabelled data for speech decoding. We pre-train a
neural network using tasks that generate implicit labels from abundant
unlabelled neuroimaging data, permitting learning from large heterogeneous
datasets. The tasks apply a randomly selected transformation $T$ to the data
and the network predicts the transformation. We fine-tune the pre-trained
network on labelled data, achieving faster generalisation owing to the
strength of the representation learned in pre-training.

##  1 Introduction

In his Bitter Lesson, Richard Sutton argues that a major conclusion of 70
years of AI research is that general methods exploiting large-scale
computation will outperform model-based approaches as the availability of
compute increases [50]. In line with this, the generality of deep learning,
via statistical learning from ever bigger datasets, has allowed the field to
leverage computation in a way that appears to scale arbitrarily, leading to
astounding advances across a diverse set of domains [29, 7, 42, 46].

In the domain of brain data, and of tasks like speech decoding, the bitter
lesson has not yet been fully assimilated. State-of-the-art brain-computer
interfaces (BCIs) have tried to scale up labelled datasets for individual
subjects, using either invasive [40, 60] or non-invasive brain recordings
[52], mapping these to transcripts of attempted or imagined speech. Yet, a
number of obstacles to scale remain. With few exceptions at present (e.g.
[17]), speech decoding models tend not to train on data from more than one
subject. Moreover, they do not combine data from multiple datasets and in
general do not utilise unlabelled data, or data from diverse tasks. Thus the
size of training data has been limited to how much can be acquired for a
single subject, and data from other subjects, or from the growing number of
public data repositories, has not been leveraged. There are many reasons for
these limitations; individual brains and data from different neuroimaging
scanners differ, for example. But overcoming these limitations, as has begun
to happen in neighbouring sub-fields (e.g. [27]), holds the promise of
training models on collective, internet-scale data.

Given the scarcity of labelled data, self-supervised learning (SSL) appears
promising as an avenue for domains where such data is rare or hard to obtain
[3]. In the SSL paradigm, _pretext_ tasks pre-train a model on unlabelled data
by generating implicit training labels through transformations of the input
data in order to help a downstream task. We develop a set of these tasks,
informed by advances in neuroscience, for learning with unlabelled brain data
(Figure 1) and design an architecture for processing continuous multi-sensor
neuroimaging signals. In order to scale existing non-invasive datasets, we
provide a unified method that allows us to leverage data from other
experiments that do not have the same labels (by treating them as unlabelled)
and that come from different subjects and neuroimaging scanners. We evaluate
the representations learned with our approach on heard speech datasets
acquired with non-invasive magnetoencephalography (MEG), setting the baselines
for speech detection and voicing classification on this data. The results not
only demonstrate that scaling with unlabelled data works in speech decoding,
but also shows that these representations can generalise across datasets,
tasks, and even novel subjects for the first time. Our main contributions are:

  * •

A set of domain specific self-supervised pretext tasks for representation
learning that can scale speech decoding over multiple subjects, multiple
studies, and unlabelled data;

  * •

A neural architecture for learning these self-supervised objectives and
training downstream speech decoding from brain data; and

  * •

A comprehensive experimental evaluation providing evidence that our approach
can scale up with data and enable cross-dataset, task, and subject
generalisation.

##  2 Related Work

Prior work in speech decoding has focused almost entirely on supervised
learning with decoding models that typically do not generalise across
participants or experiments. This is true both in recent state-of-the-art
invasive studies [40, 39, 60, 9] and non-invasive studies [52]. These prior
works have scaled up the experimental data collected within individual
subjects, but are unable to leverage data from other subjects and experiments.
Focusing on semantic rather than phonetic decoding, work by Tang et al. [52]
is remarkable for showing an ability to generalise across labelled task data
when listening to speech, imagining speech, or even watching videos. They do
not, however, leverage unlabelled data and are unable to show generalisation
between subjects.

Specific studies into the limitations of generalising models between subjects
show that while performance decreases on average when subjects are pooled,
there are exceptions. Csaky et al. [11] find that a subset of individuals
perform better when evaluated with a group-level model than with individual
models. Exploiting audio data in a multi-modal framework, Défossez et al. [17]
show that decoding performance improves for a segment identification task as
data from multiple subjects listening to connected speech are aggregated.
Although they repeat the result within two MEG and two EEG datasets, Défossez
et al. [17] do not show any improvements for pooling data across datasets.
Moreover, they do not combine data from studies with different labels either;
cf. [58, 16, 56]. Unfortunately, two of these papers [58, 16] included a bug
in their evaluation code. As such, their methods may perform no better than a
baseline that provides pure noise inputs to the model [28].

In general, speech decoding has centred on different kinds of speech:
listening, imagining, speaking out loud, and, for paralysed patients,
attempting to speak aloud. We focus here on listening because it is easier to
decode than imagined speech (e.g. [37]). There is also some evidence of a
functional overlap between listening and imagined speech representations in
the brain [55], though we acknowledge that the question of overlap has been
contested [30]. Prior work has also investigated the two tasks that we focus
on here [13, 40, 23]. The first of these, speech detection, formed the
backbone to Moses et al. [40], where a speech detection model was trained and
subsequently used to detect isolated words, which were in turn classified and
checked against a language model to generate acceptable sentences. Hamilton et
al. [26] further elaborated on the neural anatomy underlying speech detection,
categorising neural responses in the superior temporal gyrus (STG) to
sustained speech and speech onset. As for the second task, voicing
classification, Gwilliams et al. [23] used this task as a proxy for phoneme
classification, as pooling phonemes into unvoiced or voiced segments (e.g. /p
t k f s/ vs /b d g v z/) improves data efficiency. We note that voicing
classification and speech detection are related tasks as voicing is a subclass
of speech. This makes them foundational for building hierarchical speech
decoding pipelines similar to prior surgical decoding work [40, 60].

In the computer vision literature, there have been a plethora of methods that
use self-supervised pretext tasks for representation learning [1, 15, 41, 31,
62, 21]. Until now, similar approaches have not translated to the brain
decoding literature. However, prior work has used other methods to leverage
unlabelled brain data. For example, Jiang et al. [27] succeeded in cross-
dataset and cross-task generalisation, using a transformer with tokenised
brain signals and a masked token prediction objective. Although this work
combined unlabelled EEG datasets, it only achieved improvements on non-speech
tasks. Wang et al. [57] used a similar approach, replacing tokens with
contextualised embeddings of time-frequency input representations. They
attained impressive speech detection results but with invasive neural
recordings, which are comparatively rare and thus have much less potential to
scale than non-invasive data.

##  3 Method

To process continuous neuroimaging data, we introduce a neural architecture
for encoding heterogeneous brain signals into latent representations. By
developing pretext tasks with the objective of learning generalisable brain
representations, we leverage this architecture for self-supervised learning
from unlabelled data, hoping to replicate similar successes in computer vision
[21, 8].

###  3.1 Network Architecture

We design a two-stage neural network architecture (Figure 2). The pre-training
stage uses pretext tasks to train a representation with unlabelled brain data.
Then, the fine-tuning stage uses this representation to learn the downstream
task by fine-tuning with labelled data.

Figure 2: Architecture overview. Inputs are projected into a shared dimension,
then encoded. In pre-training, all weights are trainable except for modules in
light-red, while in shallow fine-tuning, modules with light-blue borders are
frozen and modules with light-red borders are unfrozen. Deep fine-tuning is
identical, except the encoder is trainable. Dashed borders indicate optional
components.

Normalisation. We divide recordings into windows of length $w$ seconds or $t$
samples. At train time, each batch of windows is standardised such that each
sensor has zero mean and unit variance.

Backbone. The network takes as input the standardised sample windows. To
combine heterogeneous datasets, which have different numbers of sensors $S$,
we apply a dataset-conditional linear layer to the sensor dimension,
projecting the signal into a shared space with dimension
$d_{\mathrm{shared}}$. Then, to encode the signal, we construct a wave-to-wave
convolutional encoder architecture, the cortex encoder, inspired by work in
neural audio codecs [61, 14]. Specifically, our convolutional encoder adapts
the implementation of the SEANet architecture [51] used in Défossez et al.
[14]. As these codecs typically operate on mono audio signals in
$\mathbb{R}^{1\times t}$, while our signals are in
$\mathbb{R}^{d_{\mathrm{shared}}\times t}$, we increase the convolutional
channel dimension from $1$ to match $d_{\mathrm{shared}}$ while also inflating
the channel dimension of subsequent convolutions. We refer to the output
dimension of embeddings from this backbone as $d_{\mathrm{backbone}}$. Thus,
the backbone takes as input a window in $\mathbb{R}^{S\times t}$, and encodes
this into $\tau$ embeddings (where $\tau<t$), each of dimension
$d_{\mathrm{backbone}}$ (i.e. an
$\mathbb{R}^{d_{\mathrm{backbone}}\times\tau}$ output).

Pre-training. Following the advice of Balestriero et al. [3, Section 3.2], we
use a two-layer feedforward projector to alleviate misalignment between our
pretext and downstream tasks in the representation. After the projector,
linear classifiers make predictions for each of the pretext tasks.

Fine-tuning. In this stage, we introduce classifiers for the downstream tasks
and train with labelled data and supervised learning (Figure 2). Depending on
the experiment, we fine-tune in one of two ways. In the first, we train an MLP
classifier from scratch on top of the pre-trained representation, which
remains frozen. Thus, we backpropagate only through the classifier. We call
this shallow fine-tuning. In the other, we fine-tune by training an MLP on top
of the pre-trained representation again, but do not freeze the backbone,
training end-to-end. Here, we backpropagate through the classifier and the
backbone. We refer to this as deep fine-tuning. In both cases, a new trainable
dataset-specific linear layer can be introduced for a novel dataset.

For speech detection, our classifier makes a prediction for each individual
embedding. For voicing classification, where there is only one label for each
sample window, the embeddings are flattened into a tensor in
$\mathbb{R}^{d_{\mathrm{backbone}}\times\tau}$ representing the entire window.
This is the input to the voicing classifier and is referred to as full epoch
decoding in neuroimaging literature [12].

Subject conditioning. Just as speakers have different voices, neural responses
between subjects have different characteristics. Consequently, individual
variation leads to models that do not generalise well across subjects [11]. In
the speech literature, models include speaker conditioning to account for
these differences [20]. We take a similar approach by introducing subject
conditioning, represented as a subject embedding, to our model. With a SEANet-
based architecture, Zeghidour et al. [61] find that conditioning is equally as
effective at the encoder bottleneck as in other stages of the model. Hence, we
place ours at the cortex encoder bottleneck for simplicity. We evaluate
multiple types of conditioning including subject embeddings and feature-wise
linear modulation (FiLM) [44].

###  3.2 Pretext Tasks

Our pretext tasks are unsupervised feature learning tasks for continuous
neuroimaging signals that aim to learn generalisable speech decoding features.
Since different datasets use different hardware and varied numbers of sensors,
we construct these tasks with labels that are agnostic to the number of
sensors in the signal. This means that these tasks do not require identifying
specific sensor channels.

Band prediction. In the literature, neural responses can be broadly segmented
into functional frequency bands [22, 45, 36]. Delta ($\delta$) waves (0.1–4
Hz) are commonly associated with the rhythmic structure of heard speech [35],
Theta ($\theta$) waves (4–8 Hz) reliably track [34] and phase-lock to the
amplitude envelope of heard sentences [43], Alpha ($\alpha$) waves (8–12 Hz)
relate to attentional processes and the inhibition of irrelevant information,
helping to focus on relevant speech signals [49], Beta ($\beta$) waves
(12–30Hz) are implicated in top-down predictive coding [5] which affects
lexical processing [59], Gamma ($\gamma$) waves (30–70 Hz) occur with higher
cognitive functions (e.g. memory, learning, reasoning, and planning) [19, 6],
and High Gamma ($\gamma^{\mathrm{high}}$) waves ($>$70 Hz) have been linked
specifically to speech detection [26] and phonemic feature classification in
the STG [38] as well as phonemic feature classification in the ventral
sensorimotor cortex (vSMC) [10]. As High Gamma is a relatively wide band, we
have split it into two sub-bands: Lower High Gamma
($\gamma^{\mathrm{high}}_{\mathrm{lower}}$) waves (70–100 Hz) and Upper High
Gamma ($\gamma^{\mathrm{high}}_{\mathrm{upper}}$) waves (100–150 Hz).

To learn representations that can distinguish between these, our band
prediction task applies a band-stop filter for a randomly selected band
$\omega$ to the sample $x$, passes the filtered sample $x^{\omega^{\prime}}$
through the network backbone $g$ and the corresponding predictor
$f_{\mathrm{band}}$, requiring the network to predict the frequency band that
was rejected. This yields the loss

$\mathcal{L}_{\mathrm{band}}=\sum_{x\in
B}\mathcal{L}_{\mathrm{CE}}(f_{\mathrm{band}}(g(x^{\omega^{\prime}})),\omega),$
(1)

where $B$ is a mini-batch of samples,
$\omega\in\\{\delta,\theta,\alpha,\beta,\gamma,\gamma^{\mathrm{high}}_{\mathrm{lower}},\gamma^{\mathrm{high}}_{\mathrm{upper}}\\}$,
and $\mathcal{L}_{\mathrm{CE}}$ is the cross-entropy loss as this is a multi-
class classification task.

Phase shift prediction. Phase coupling between networks of neuron populations
is necessary for coordinating brain activity [18, 54]. Thus, since phase often
synchronises between communicating brain areas, phase coupling between
spatially distant sensors is likely to be a useful feature. Supporting this
insight, recent work [27] also finds phase to be an essential component of the
signal.

To learn representations that encode phase differences between brain areas,
this task applies a discrete uniform random phase shift
$\phi\in\\{0,\frac{\pi}{8},\frac{\pi}{4},\frac{3\pi}{8},\frac{\pi}{2},\frac{5\pi}{8},\frac{3\pi}{4},\frac{7\pi}{8}\\}$
to a uniformly randomly selected proportion $\rho$ of the sensors. Applying
this shift to random sensors is critical since sensors are placed in different
positions, capturing different regions of the brain. Uniform random selection
ensures differences between any two regions of the brain are represented. The
objective of this task is to predict the phase shift. This leads to a similar
loss

$\mathcal{L}_{\mathrm{phase}}=\sum_{x\in
B}\mathcal{L}_{\mathrm{CE}}(f_{\mathrm{phase}}(g(x^{\phi})),\phi),$ (2)

where $x^{\phi}$ describes the signal with a phase shift $\phi$ applied to a
proportion of the sensors. We use a discrete number of possible phase shifts,
treating it as a multi-class task rather than a regression task, to ease the
difficulty of the problem as MEG scanners typically have a large number of
sensors.

Amplitude scale prediction. MEG and EEG signals use an array of sensors at
different spatial locations, capturing different signal sources more
intensely. Representing the relative amplitude difference between sensors
could be important for differentiating between neural responses originating
from distinct parts of the brain. Within speech, Hamilton et al. [26] find
that localised regions of the STG respond to sustained speech and speech
onsets. Differentiating between neural responses from this region and others
may be essential for decoding speech perception.

Thus, this pretext task focuses on learning representations that encode
relative sensor amplitude differences. Similar to the phase shift task, we
select a random proportion of the sensors $\rho$ and apply a discrete random
amplitude scaling coefficient $A\in[-2,2]$, discretised into 16 scaling
factors, to the signal. The objective is to predict the scaling factor,
leading to the loss

$\mathcal{L}_{\mathrm{amplitude}}=\sum_{x\in
B}\mathcal{L}_{\mathrm{CE}}(f_{\mathrm{amplitude}}(g(x^{A})),A),$ (3)

where $x^{A}$ is the signal scaled with $A$.

These pretext tasks capture complementary time- and frequency-domain
properties of the signal. Hence, during pre-training, we combine them,
creating an augmented version of the input for every pretext task by applying
the matching transformation. We feed the augmented inputs through the network
backbone and apply the corresponding classifier to predict the transformation,
summing the weighted losses such that our final pre-training loss is given by

$\mathcal{L}_{\mathrm{SSL}}=w_{1}\mathcal{L}_{\mathrm{band}}+w_{2}\mathcal{L}_{\mathrm{phase}}+w_{3}\mathcal{L}_{\mathrm{amplitude}},$
(4)

where $w_{i}$ is a constant coefficient for each self-supervised loss.

##  4 Experiments

In this section, we evaluate the representations learned with our pretext
tasks by measuring their ability to scale downstream performance with
unlabelled data. This includes understanding how well they can generalise
across datasets, subjects, and tasks. We focus our evaluation on MEG data as
the signal is rich, with better spatial resolution than EEG [32] and faster
sampling rates than fMRI [25].

###  4.1 Experimental setup

Datasets. Unless specified otherwise, our experiments use Cam-CAN [48, 53] as
an unlabelled representation learning dataset for pre-training. This is a
study containing 641 subjects with resting and sensorimotor tasks, totalling
approximately 160 hours of MEG recordings. For our downstream tasks, we use
two labelled heard speech MEG datasets where participants listen to short
stories or audiobooks. Armeni et al. [2] contains 3 subjects who listen to 10
hours of recordings each (30 hours total) while Gwilliams et al. [24] has 27
subjects, each recorded for 2 hours (54 hours total). Overall, we utilise over
200 hours of data. To the best of our knowledge, this is the largest volume of
MEG data ever used for speech decoding.

Preprocessing. Each recording is in $\mathbb{R}^{S\times T}$ where $S$ is the
number of sensors and $T$ is the number of time points sampled by the scanner.
To eliminate high-frequency muscle movement artifacts, we apply a low-pass
filter at 125Hz as well as a high-pass filter at 0.5Hz to remove slow-drift
artifacts. Since the datasets were recorded in Europe, where the electric grid
frequency is 50Hz, we apply a notch filter at multiples of 50Hz to account for
line noise. Next, we downsample the signal to 250Hz, avoiding aliasing at
frequencies up to our low-pass filter threshold. Finally, we detect bad sensor
channels, those with significant noise and artifacts, using a variance
threshold and replace them by interpolating the spatially nearest sensors.

Downstream tasks. We evaluate our methods with two fundamental speech decoding
tasks of increasing difficulty. The first, speech detection, determines
whether speech occurs in the auditory stimulus using the neural response. The
second task is voicing classification. Given data aligned at the occurrence of
a phoneme, the task is to recognise whether the phoneme is voiced or
voiceless, where voicing is a binary phonetic feature that categorises whether
a speech sound is associated with vocal cord vibration. We select these tasks
as they are simpler than phoneme recognition, but are foundational because
they must be solved to decode speech accurately into natural language.

Training. We pre-train all models to completion and then shallow or deep fine-
tune on labelled data for each task, using this to analyse the
generalisability of the pre-trained representation. Appendix E provides
complete training details for all experiments.

###  4.2 Learning Generalisable Representations Using Pretext Tasks

We investigate whether our self-supervised objectives produce useful
representations by first learning a representation with each pretext task
independently and then analysing its downstream generalisation using shallow
fine-tuning (i.e. with the representation frozen). In Table 1, we show that
all of our pretext tasks lead to better than chance accuracy on speech
detection, and critically, outperform a randomly initialised and shallow fine-
tuned baseline. The results are statistically significant in all cases,
providing initial evidence that all of our tasks are helpful in learning
representations that generalise downstream to speech decoding.

Interestingly, the combination of all pretext tasks leads to better
generalisation than any task on its own. As we hypothesised earlier, this may
be because our pretext tasks capture complementary properties in time- and
frequency-space, enforcing that our representation includes more salient
features for speech decoding. For our other downstream task, voicing
classification, we were not able to use shallow fine-tuning to generalise to a
statistically significant degree with any of the pretext tasks nor with a
random backbone. It may be too difficult to learn with shallow fine-tuning and
so we turn to using deep fine-tuning for this task in the next section.

Pretext task | Speech detection balanced accuracy
---|---
| Armeni | $t$ | $p$ | Gwilliams | $t$ | $p$
Amp($\rho=0.2$) | $0.5846\pm 0.0032$ | $26$ | $7\mathrm{e}{-4}$ | $0.5116\pm 0.0004$ | $29$ | $6\mathrm{e}{-4}$
Phase($\rho=0.5$) | $0.5831\pm 0.0029$ | $29$ | $6\mathrm{e}{-4}$ | $0.5093\pm 0.0014$ | $7$ | $1\mathrm{e}{-2}$
Band | $0.5941\pm 0.0024$ | $39$ | $2\mathrm{e}{-3}$ | $0.5177\pm 0.0028$ | $6$ | $1\mathrm{e}{-2}$
All | $\mathbf{0.6011}\pm 0.0018$ | $56$ | $2\mathrm{e}{-4}$ | $\mathbf{0.5206}\pm 0.0010$ | $22$ | $1\mathrm{e}{-3}$
None (rand. backbone) | $0.5000\pm 0.0000$ | $-$ | $-$ | $0.5000\pm 0.0000$ | $-$ | $-$
Chance-level | $0.5$ | $-$ | $-$ | $0.5$ | $-$ | $-$
Table 1: Pre-training with pretext tasks produces generalisable
representations. We provide the test accuracy at the point of best validation
accuracy (early stopping). In the none baseline, the network backbone is
randomly initialised before fine-tuning (i.e. there is no pre-training). When
all pretext tasks are used, their losses are weighted equally. The uncertainty
is the standard error of the mean over three random seeds. We quote the
$t$-score and $p$-value for each result from a one-sample one-sided $t$-test
where the population mean is chance-level accuracy. Models are pre-trained on
all of Cam-CAN [48, 53] and fine-tuned on Armeni et al. [2] or Gwilliams et
al. [24] respectively.

Among the pretext tasks, band prediction performs best. This could be because
the representation easily identifies phase-locking to speech onset in theta
waves [43]. However, further investigation is necessary to determine which
frequency band is most informative. The choice of the proportion of sensors to
apply transformations to, $\rho=0.5$ for phase shift prediction and $\rho=0.2$
for amplitude prediction, were determined through a hyperparameter search
(Appendix F). We conjecture that a smaller $\rho$ is optimal for amplitude
scale prediction since this leads to representations that are especially
strong at discriminating amplitude differences among small groups of sensors.
Perhaps this makes it easier to distinguish between neural responses from
distinct parts of the brain such as the STG, which is associated with speech
onset [26]. In contrast, a larger $\rho$ for phase shift prediction may be
best because phase is descriptive of neural synchrony which is distributed
across the brain. Unlike localised responses, a large proportion of the
sensors can detect this feature.

Whilst we have shown that our representations generalise, do they outperform
supervised baselines when trained to saturation? Using shallow fine-tuning
once more, we compare our approach directly to similarly parameterised
supervised classifiers in Table 2. A representation pre-trained with all
pretext tasks significantly outperforms chance, a randomly initialised
backbone, and supervised training on both downstream datasets. Here, the
supervised classifier has significantly more parameters than the models that
probe the network backbone because the input dimension is far larger without
an encoder. Even with this bias favouring the supervised classifier, the self-
supervised case performs better. Thus, it is apparent that pre-training with
unlabelled data significantly improves downstream generalisation.

Model | Speech detection balanced accuracy
---|---
| Armeni | $t$ | $p$ | Gwilliams | $t$ | $p$
Supervised |  |  |  |  |  |
Linear | $0.5222\pm 0.0015$ | $12$ | $3\mathrm{e}{-3}$ | $0.5006\pm 0.0001$ | $5$ | $2\mathrm{e}{-2}$
Self-supervised \+ fine-tuning |  |  |  |  |
Random backbone |  |  |  |  |  |
MLP (two layers) | $0.5000\pm 0.0000$ | $-$ | $-$ | $0.5000\pm 0.0000$ | $-$ | $-$
Pre-trained backbone |  |  |  |  |  |
Linear | $0.5853\pm 0.0017$ | $52$ | $3\mathrm{e}{-4}$ | $0.5011\pm 0.0001$ | $9$ | $6\mathrm{e}{-3}$
MLP (two layers) | $\mathbf{0.6011}\pm 0.0018$ | $56$ | $2\mathrm{e}{-4}$ | $\mathbf{0.5206}\pm 0.0010$ | $22$ | $1\mathrm{e}{-3}$
Chance-level | $0.5$ | $-$ | $-$ | $0.5$ | $-$ | $-$
Table 2: Self-supervised pre-training outperforms supervised classifiers. We
use shallow fine-tuning on top of a randomly initialised and pre-trained
backbone, comparing these to a supervised classifier trained to completion. We
quote test accuracy at the point of best validation accuracy (early stopping)
and show uncertainty as the standard error of the mean from three seeds. We
calculate the $t$-score and $p$-value from one-sample one-sided $t$-tests
where the population mean is chance. Self-supervised models are pre-trained on
all of Cam-CAN [48, 53] and shallow fine-tuned on Armeni et al. [2] or
Gwilliams et al. [24] while the supervised models are trained entirely on the
latter datasets.

###  4.3 Scaling Speech Decoding With Unlabelled Data

Figure 3: Scaling unlabelled data improves downstream generalisation. We pre-
train the model on increasing amounts of unlabelled data from Cam-CAN [48, 53]
and fine-tune on voicing and speech detection on the Armeni et al. [2] and
Gwilliams et al. [24] datasets. No pre-training shows best test accuracy for a
randomly initialised network fine-tuned for the same number of epochs (i.e.
without pre-training). We show only one no pre-training line as the result is
the same for both datasets. The shaded area shows the standard error of the
mean across three seeds.

Previously, we showed that the combination of all pretext tasks leads to the
best representations and that this outperforms a randomly initialised backbone
as well as a supervised classifier. In this section, we examine how these
pretext tasks fare as we increase the amount of unlabelled data, analysing
scaling performance on downstream tasks. As before, we pre-train with the
combined pretext tasks using Cam-CAN [48, 53], which does not have any speech
decoding labels, and now fine-tune for ten epochs on Armeni et al. [2] and
Gwilliams et al. [24], showcasing transfer with minimal fine-tuning. This
simulates the scenario in which models need to generalise quickly with very
little calibration data from new patients in order to be deployed as a speech
BCI. Where earlier we analysed only speech detection, here, we also focus on
voicing classification by using deep fine-tuning for this task only. As
evidenced by Section 4.2, we require this because voicing classification is a
more difficult task, necessitating end-to-end training with more parameters to
observe generalisation.

Figure 3 shows balanced accuracy as we increase the amount of unlabelled data
in pre-training up to approximately 160 hours. For both tasks, pre-training
outperforms the no pre-training baseline, which fails to generalise at all,
showing that any unlabelled data is sufficient to pre-train a useful
representation. In both tasks, there is a clear improvement in accuracy as the
amount of unlabelled data increases. Thus, adding unlabelled data improves
generalisation and scales performance. With speech detection, the improvement
in accuracy is more significant for Armeni et al. [2]. Noting that there is
more within-subject data in this dataset than Gwilliams et al. [24], speech
detection may be more sensitive to subject variation than voicing
classification.

We scaled up the pre-training dataset by increasing the number of subjects.
Since this led to consistent and almost monotonic improvements in downstream
accuracy, our self-supervised method is an exception to the common consensus
that pooling subjects worsens generalisation. Moreover, as we pre-trained our
model with a different dataset to those we fine-tuned on, our representation
shows cross-dataset generalisation. This is particularly notable as the Armeni
et al. [2], Gwilliams et al. [24], and pre-training datasets all use different
scanners entirely. Performing well across these datasets indicates that,
together, our architecture and pretext tasks successfully generate
representations that are generalisable across heterogeneous scanners.

Beyond hardware, our results also show improvements on distinct tasks, namely
speech detection and voicing classification, thus illustrating cross-task
generalisation. Thus, our pretext tasks are sufficiently generic to produce
representations that generalise to multiple downstream tasks, indicating their
applicability to speech decoding more generally. Finally, our pre-training
dataset contained no language data whatsoever yet still improved downstream
accuracy on language tasks. Remarkably, this shows that brain data collected
from any task (including those that are not linguistic) can be used to scale
downstream speech decoding performance.

###  4.4 Scaling Unlabelled Data Improves Zero-Shot Generalisation To Novel
Subjects

Brain data is variable across participants, leading to difficulty transferring
models to novel subjects [11]. Whilst we have shown generalisation across
subjects with our method, here, we investigate how well we can generalise to
novel subjects—an even more difficult challenge. This is critical in order to
widely deploy speech BCIs for new patients. In the following experiment, we
pre-train using the same setup as Figure 3, but fine-tune only on Gwilliams et
al. [24]. We use FiLM conditioning as this performs best for seen and unseen
subjects (Appendix B). During training, we hold out three subjects with which
we evaluate novel subject generalisation (i.e. unseen subject performance).

Figure 4: Scaling pre-training data improves novel subject generalisation. As
before, we pre-train with increasing amounts of unlabelled data from Cam-CAN
[48, 53], but now fine-tune only on Gwilliams et al. [24]. In the seen case,
we evaluate on held-out sessions from subjects that were used in the training
data. For the unseen case we evaluate on three completely held-out subjects.
No pre-training shows the best test accuracy for a randomly initialised
network fine-tuned for the same number of epochs (i.e. without pre-training).
We show only one no pre-training line as the result is the same for both
datasets. The shaded area is the standard error of the mean over three seeds.

Figure 4 shows that scaling up the amount of unlabelled data used in pre-
training not only improves accuracy on subjects previously seen, but also
demonstrates a positive trend in performance for unseen (novel) subjects. This
is a promising result, indicating that scaling with our method is an
encouraging direction for resolving the issues related to individual variance
faced by prior work.

###  4.5 Limitations

Although our results are significant in demonstrating a viable path forward to
scale up speech BCIs, there remain a number of limitations to the present
work. We focused here on two downstream tasks: speech detection and voice
classification. Ultimately, we would like to expand this work to predict full
transcripts from brain recordings (i.e. brain-to-text). This has been achieved
with surgical data [40, 60] but not yet convincingly with non-invasive methods
like MEG or EEG [28]. Speech detection has played an important role in the
development of full brain-to-text in a surgical context [40] and we hope may
play a similar role for non-invasive methods. Prior work has further used
voice classification as a stand in for phoneme classification [23], and we
have been able to improve on these results here. In future work, we would like
to expand this to all English phonemes. Secondly, while we have been able to
demonstrate the utility of a few pretext tasks here, we do not claim to have
exhausted the full set of useful tasks. Rather, we conjecture that more useful
pretext tasks remain to be found and believe a useful avenue of research will
be into other input representations for brain recordings. For example, this
paper did not make use of spatial features. Another limitation is our emphasis
on heard speech over other types of speech, such as attempted or imagined
speech. We hypothesise that the same methods presented here will generalise to
these other varieties of speech, though this has yet to be shown. But, perhaps
the biggest limitation of the present work is that, while it surpasses the
amount of data used in other studies, it remains to be seen how much speech
decoding tasks can be improved by scaling up the number of datasets used in
training. In sharing this work now, we believe that the current proof of
concept will be sufficiently impactful to the field as we continue to actively
scale up the datasets that we can leverage.

##  5 Conclusion

Ultimately, solving speech decoding could transform the lives of patients with
severe communication difficulties. Yet, this promise has not materialised
because the field has been blocked by its inability to scale up data to
leverage deep learning. Prior methods have been unable to aggregate data
across different datasets, labels, or subjects to scale up because of
heterogeneity in recording hardware, experiment design, and participants. A
handful of studies have shown weak signals towards alleviating these issues.
But until now, no one has developed a general solution. We provided a unified
method that leverages unlabelled data using generic pretext tasks that shows
that all of these problems can be solved. We verified this with experiments
showing that our method not only scales with heterogeneous data but even
generalises across datasets, subjects, and tasks. Our method unlocks the
potential of the bitter lesson, providing a general method to exploit more
computation by using more data. We implore the research community to employ
the vast quantities of data and compute available to realise this potential.
If scale is all you need in speech decoding, then the bitter lesson may not be
so bitter.

## Acknowledgments and Disclosure of Funding

We would like to thank Botos Csaba for many insightful discussions and
creative ideas which helped shaped the direction of this work. Thanks also to
Minqi Jiang for an encouraging conversation on unsupervised representation
learning, Mats W.J. van Es for technical assistance with the OSL library,
Brian Liu for technical contributions which did not reach the final paper, and
Miran Özdogan for reviewing a draft of this work. The authors would like to
acknowledge the use of the University of Oxford Advanced Research Computing
(ARC) facility in carrying out this work.
http://dx.doi.org/10.5281/zenodo.22558.

DJ is supported by an AWS Studentship from the EPSRC CDT in Autonomous
Intelligent Machines and Systems (AIMS). GL is supported by an EPSRC
Studentship. MW is supported by the Wellcome Trust (106183/Z/14/Z,
215573/Z/19/Z), the New Therapeutics in Alzheimer’s Diseases (NTAD) study
supported by UK MRC, the Dementia Platform UK (RG94383/RG89702) and the NIHR
Oxford Health Biomedical Research Centre (NIHR203316). The views expressed are
those of the author(s) and not necessarily those of the NIHR or the Department
of Health and Social Care. OPJ is supported by the MRC (MR/X00757X/1), Royal
Society (RG\R1\241267), NSF (2314493), NFRF (NFRFT-2022-00241), and SSHRC
(895-2023-1022).

## References

  * Agrawal et al. [2015] Pulkit Agrawal, João Carreira, and Jitendra Malik.  Learning to see by moving.  In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_ , pages 37–45. IEEE Computer Society, 2015.  doi: 10.1109/ICCV.2015.13.  URL https://doi.org/10.1109/ICCV.2015.13.
  * Armeni et al. [2022] Kristijan Armeni, Umut Güçlü, Marcel van Gerven, and Jan-Mathijs Schoffelen.  A 10-hour within-participant magnetoencephalography narrative dataset to test models of language comprehension.  _Scientific Data_ , 9(1):278, June 2022.  ISSN 2052-4463.  doi: 10.1038/s41597-022-01382-7.  URL https://www.nature.com/articles/s41597-022-01382-7.
  * Balestriero et al. [2023] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Grégoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum.  A cookbook of self-supervised learning.  _CoRR_ , abs/2304.12210, 2023.  doi: 10.48550/ARXIV.2304.12210.  URL https://doi.org/10.48550/arXiv.2304.12210.
  * Bordes et al. [2023] Florian Bordes, Samuel Lavoie, Randall Balestriero, Nicolas Ballas, and Pascal Vincent.  A surprisingly simple technique to control the pretraining bias for better transfer: Expand or narrow your representation.  _CoRR_ , abs/2304.05369, 2023.  doi: 10.48550/ARXIV.2304.05369.  URL https://doi.org/10.48550/arXiv.2304.05369.
  * Bressler and Richter [2015] Steven L Bressler and Craig G Richter.  Interareal oscillatory synchronization in top-down neocortical processing.  _Current Opinion in Neurobiology_ , 31:62–66, 2015.
  * Buzsáki and Wang [2012] György Buzsáki and Xiao-Jing Wang.  Mechanisms of gamma oscillations.  _Annual Review of Neuroscience_ , 35:203–225, 2012.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pages 9630–9640. IEEE, 2021.  doi: 10.1109/ICCV48922.2021.00951.  URL https://doi.org/10.1109/ICCV48922.2021.00951.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1597–1607. PMLR, 2020.  URL http://proceedings.mlr.press/v119/chen20j.html.
  * Chen et al. [2024] Xupeng Chen, Ran Wang, Amirhossein Khalilian-Gourtani, Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, and Adeen Flinker.  A neural speech decoding framework leveraging deep learning and speech synthesis.  _Nature Machine Intelligence_ , pages 1–14, April 2024.  ISSN 2522-5839.  doi: 10.1038/s42256-024-00824-8.  URL https://www.nature.com/articles/s42256-024-00824-8.
  * Cheung et al. [2016] Connie Cheung, Liberty S Hamilton, Keith Johnson, and Edward F Chang.  The auditory representation of speech sounds in human motor cortex.  _eLife_ , 5:e12577, 2016.
  * Csaky et al. [2022] Richard Csaky, Mats W. J. van Es, Oiwi Parker Jones, and Mark W. Woolrich.  Group-level brain decoding with deep learning.  _Human Brain Mapping_ , 44:6105 – 6119, 2022.  URL https://doi.org/10.1002/hbm.26500.
  * Csaky et al. [2023] Richard Csaky, Mats W.J. van Es, Oiwi Parker Jones, and Mark Woolrich.  Interpretable many-class decoding for MEG.  _NeuroImage_ , 282:120396, November 2023.  ISSN 10538119.  doi: 10.1016/j.neuroimage.2023.120396.  URL https://linkinghub.elsevier.com/retrieve/pii/S1053811923005475.
  * Dash et al. [2020] Debadatta Dash, Paul Ferrari, Satwik Dutta, and Jun Wang.  NeuroVAD: Real-Time Voice Activity Detection from Non-Invasive Neuromagnetic Signals.  _Sensors_ , 20(8):2248, January 2020.  ISSN 1424-8220.  doi: 10.3390/s20082248.  URL https://www.mdpi.com/1424-8220/20/8/2248.
  * Défossez et al. [2022] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi.  High fidelity neural audio compression.  _CoRR_ , abs/2210.13438, 2022.  doi: 10.48550/ARXIV.2210.13438.  URL https://doi.org/10.48550/arXiv.2210.13438.
  * Doersch et al. [2015] Carl Doersch, Abhinav Gupta, and Alexei A. Efros.  Unsupervised visual representation learning by context prediction.  In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_ , pages 1422–1430. IEEE Computer Society, 2015.  doi: 10.1109/ICCV.2015.167.  URL https://doi.org/10.1109/ICCV.2015.167.
  * Duan et al. [2023] Yiqun Duan, Charles Chau, Zhen Wang, Yu-Kai Wang, and Chin-Teng Lin.  DeWave: Discrete encoding of EEG waves for EEG to text translation.  In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, _Advances in Neural Information Processing Systems 36 (NeurIPS 2023), New Orleans, LA, USA, December 10 - 16_ , 2023.  URL http://papers.nips.cc/paper_files/paper/2023/hash/1f2fd23309a5b2d2537d063b29ec1b52-Abstract-Conference.html.
  * Défossez et al. [2023] Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King.  Decoding speech perception from non-invasive brain recordings.  _Nature Machine Intelligence_ , 5(10):1097–1107, October 2023.  ISSN 2522-5839.  doi: 10.1038/s42256-023-00714-5.  URL https://www.nature.com/articles/s42256-023-00714-5.
  * Fries [2005] Pascal Fries.  A mechanism for cognitive dynamics: neuronal communication through neuronal coherence.  _Trends in Cognitive Sciences_ , 9(10):474–480, October 2005.  ISSN 1364-6613.  doi: 10.1016/j.tics.2005.08.011.  URL https://www.sciencedirect.com/science/article/pii/S1364661305002421.
  * Fries [2009] Pascal Fries.  Neuronal gamma-band synchronization as a fundamental process in cortical computation.  _Annual Review of Neuroscience_ , 32:209–224, 2009.
  * Gibiansky et al. [2017] Andrew Gibiansky, Sercan Ömer Arik, Gregory Frederick Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou.  Deep voice 2: Multi-speaker neural text-to-speech.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 2962–2970, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/c59b469d724f7919b7d35514184fdc0f-Abstract.html.
  * Gidaris et al. [2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net, 2018.  URL https://openreview.net/forum?id=S1v4N2l0-.
  * Giraud and Poeppel [2012] Anne-Lise Giraud and David Poeppel.  Cortical oscillations and speech processing: emerging computational principles and operations.  _Nature Neuroscience_ , 15(4):511–517, April 2012.  ISSN 1546-1726.  doi: 10.1038/nn.3063.  URL https://www.nature.com/articles/nn.3063.
  * Gwilliams et al. [2022] Laura Gwilliams, Jean-Rémi King, Alec Marantz, and David Poeppel.  Neural dynamics of phoneme sequences reveal position-invariant code for content and order.  _Nature Communications_ , 13(1):6606, November 2022.  ISSN 2041-1723.  doi: 10.1038/s41467-022-34326-1.  URL https://www.nature.com/articles/s41467-022-34326-1.
  * Gwilliams et al. [2023] Laura Gwilliams, Graham Flick, Alec Marantz, Liina Pylkkänen, David Poeppel, and Jean-Rémi King.  Introducing MEG-MASC a high-quality magneto-encephalography dataset for evaluating natural speech processing.  _Scientific Data_ , 10(1):862, December 2023.  ISSN 2052-4463.  doi: 10.1038/s41597-023-02752-5.  URL https://www.nature.com/articles/s41597-023-02752-5.
  * Hall et al. [2014] Emma L. Hall, Siân E. Robson, Peter G. Morris, and Matthew J. Brookes.  The relationship between MEG and fMRI.  _NeuroImage_ , 102:80–91, 2014.  URL https://doi.org/10.1016/j.neuroimage.2013.11.005.
  * Hamilton et al. [2018] Liberty S. Hamilton, Erik Edwards, and Edward F. Chang.  A Spatial Map of Onset and Sustained Responses to Speech in the Human Superior Temporal Gyrus.  _Current Biology_ , 28(12):1860–1871.e4, June 2018.  ISSN 09609822.  doi: 10.1016/j.cub.2018.04.033.  URL https://linkinghub.elsevier.com/retrieve/pii/S0960982218304615.
  * Jiang et al. [2024] Weibang Jiang, Liming Zhao, and Bao liang Lu.  Large brain model for learning generic representations with tremendous EEG data in BCI.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=QzTpTRVtrP.
  * Jo et al. [2024] Hyejeong Jo, Yiqian Yang, Juhyeok Han, Yiqun Duan, Hui Xiong, and Won Hee Lee.  Are EEG-to-text models working?  _arXiv_ , 2024.  doi: https://arxiv.org/abs/2405.06459.
  * Jumper et al. [2021] John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596:583 – 589, 2021.  URL https://doi.org/10.1038/s41586-021-03819-2.
  * Langland-Hassan and Vicente [2018] Peter Langland-Hassan and Agustín Vicente.  _Inner Speech: New Voices_.  Oxford University Press, 2018.
  * Larsson et al. [2016] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich.  Learning representations for automatic colorization.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV_ , volume 9908 of _Lecture Notes in Computer Science_ , pages 577–593. Springer, 2016.  doi: 10.1007/978-3-319-46493-0\\_35.  URL https://doi.org/10.1007/978-3-319-46493-0_35.
  * Lopes da Silva [2013] Fernando Lopes da Silva.  EEG and MEG: Relevance to Neuroscience.  _Neuron_ , 80(5):1112–1128, December 2013.  ISSN 0896-6273.  doi: 10.1016/j.neuron.2013.10.017.  URL https://www.sciencedirect.com/science/article/pii/S0896627313009203.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.  URL https://openreview.net/forum?id=Bkg6RiCqY7.
  * Luo and Poeppel [2007] Huan Luo and David Poeppel.  Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex.  _Neuron_ , 54(6):1001–1010, 2007.
  * Luo et al. [2010] Huan Luo, Zuxiang Liu, and David Poeppel.  Auditory cortex tracks both auditory and visual stimulus dynamics using low-frequency neuronal phase modulation.  _PLOS Biology_ , 8(8):e1000445, 2010.
  * Mai et al. [2016] Guangting Mai, James W. Minett, and William S. Y. Wang.  Delta, theta, beta, and gamma brain oscillations index levels of auditory sentence processing.  _NeuroImage_ , 133:516–528, June 2016.  ISSN 1053-8119.  doi: 10.1016/j.neuroimage.2016.02.064.  URL https://www.sciencedirect.com/science/article/pii/S1053811916001737.
  * Martin et al. [2014] Stéphanie Martin, Peter Brunner, Chris Holdgraf, Hans-Jochen Heinze, Nathan E Crone, Jochem Rieger, Gerwin Schalk, Robert T Knight, and Brian N Pasley.  Decoding spectrotemporal features of overt and covert speech from the human cortex.  _Frontiers in Neuroengineering_ , 7:14, 2014.
  * Mesgarani et al. [2014] Nima Mesgarani, Connie Cheung, Keith Johnson, and Edward F. Chang.  Phonetic feature encoding in human superior temporal gyrus.  _Science_ , 343(6174):1006–1010, 2014.  doi: DOI:10.1126/science.1245994.
  * Metzger et al. [2023] Sean L. Metzger, Kaylo T. Littlejohn, Alexander B. Silva, David A. Moses, Margaret P. Seaton, Ran Wang, Maximilian E. Dougherty, Jessie R. Liu, Peter Wu, Michael A. Berger, Inga Zhuravleva, Adelyn Tu-Chan, Karunesh Ganguly, Gopala K. Anumanchipalli, and Edward F. Chang.  A high-performance neuroprosthesis for speech decoding and avatar control.  _Nature_ , 620:1037–1046, 2023.
  * Moses et al. [2021] David A. Moses, Sean L. Metzger, Jessie R. Liu, Gopala K. Anumanchipalli, Joseph G. Makin, Pengfei F. Sun, Josh Chartier, Maximilian E. Dougherty, Patricia M. Liu, Gary M. Abrams, Adelyn Tu-Chan, Karunesh Ganguly, and Edward F. Chang.  Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria.  _New England Journal of Medicine_ , 385(3):217–227, July 2021.  ISSN 0028-4793.  doi: 10.1056/NEJMoa2027540.  URL https://doi.org/10.1056/NEJMoa2027540.
  * Noroozi and Favaro [2016] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI_ , volume 9910 of _Lecture Notes in Computer Science_ , pages 69–84. Springer, 2016.  doi: 10.1007/978-3-319-46466-4\\_5.  URL https://doi.org/10.1007/978-3-319-46466-4_5.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _CoRR_ , abs/2303.08774, 2023.  doi: 10.48550/ARXIV.2303.08774.  URL https://doi.org/10.48550/arXiv.2303.08774.
  * Peelle et al. [2012] Jonathan E. Peelle, Joachim Gross, and Matthew H. Davis.  Phase-locked responses to speech in human auditory cortex are enhanced during comprehension.  _Cerebral Cortex_ , 23(6):1378–1387, 2012.
  * Perez et al. [2018] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville.  FiLM: Visual reasoning with a general conditioning layer.  In Sheila A. McIlraith and Kilian Q. Weinberger, editors, _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 3942–3951. AAAI Press, 2018.  doi: 10.1609/AAAI.V32I1.11671.  URL https://doi.org/10.1609/aaai.v32i1.11671.
  * Piai et al. [2014] Vitória Piai, Ardi Roelofs, and Eric Maris.  Oscillatory brain responses in spoken word production reflect lexical frequency and sentential constraint.  _Neuropsychologia_ , 53:146–156, January 2014.  ISSN 0028-3932.  doi: 10.1016/j.neuropsychologia.2013.11.014.  URL https://www.sciencedirect.com/science/article/pii/S0028393213004119.
  * Radford et al. [2023] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  In _International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 28492–28518. PMLR, 2023.  URL https://proceedings.mlr.press/v202/radford23a.html.
  * Schoffelen et al. [2019] Jan-Mathijs Schoffelen, Robert Oostenveld, Nietzsche H. L. Lam, Julia Uddén, Annika Hultén, and Peter Hagoort.  A 204-subject multimodal neuroimaging dataset to study language processing.  _Scientific Data_ , 6(1):17, April 2019.  ISSN 2052-4463.  doi: 10.1038/s41597-019-0020-y.  URL https://www.nature.com/articles/s41597-019-0020-y.
  * Shafto et al. [2014] Meredith A. Shafto, Lorraine K. Tyler, Marie Dixon, Jason R. Taylor, James Benedict Rowe, Rhodri Cusack, Andrew J. Calder, William D. Marslen-Wilson, John S. Duncan, T. Dalgleish, Richard N. A. Henson, Carol Brayne, and Fiona E. Matthews.  The Cambridge centre for ageing and neuroscience (Cam-CAN) study protocol: a cross-sectional, lifespan, multidisciplinary examination of healthy cognitive ageing.  _BMC Neurology_ , 14, 2014.
  * Strauß et al. [2015] Antje Strauß, Molly J Henry, Mathias Scharinger, and Jonas Obleser.  Alpha phase determines successful lexical decision in noise.  _Journal of Neuroscience_ , 35(7):3256–3262, 2015.
  * Sutton [2019] Richard Sutton.  The bitter lesson.  _Incomplete Ideas (blog)_ , 2019.  URL http://www.incompleteideas.net/IncIdeas/BitterLesson.html.
  * Tagliasacchi et al. [2020] Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, and Dominik Roblek.  SEANet: A multi-modal speech enhancement network.  In Helen Meng, Bo Xu, and Thomas Fang Zheng, editors, _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_ , pages 1126–1130. ISCA, 2020.  doi: 10.21437/INTERSPEECH.2020-1563.  URL https://doi.org/10.21437/Interspeech.2020-1563.
  * Tang et al. [2023] Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G. Huth.  Semantic reconstruction of continuous language from non-invasive brain recordings.  _Nature Neuroscience_ , 26(5):858–866, May 2023.  ISSN 1546-1726.  doi: 10.1038/s41593-023-01304-9.  URL https://www.nature.com/articles/s41593-023-01304-9.
  * Taylor et al. [2017] Jason R. Taylor, Nitin Williams, Rhodri Cusack, Tibor Auer, Meredith A. Shafto, Marie Dixon, Lorraine K. Tyler, Cam-CAN Group, and Richard N. A. Henson.  The Cambridge centre for ageing and neuroscience (Cam-CAN) data repository: Structural and functional MRI, MEG, and cognitive data from a cross-sectional adult lifespan sample.  _Neuroimage_ , 144:262 – 269, 2017.
  * Vidaurre et al. [2018] Diego Vidaurre, Laurence T. Hunt, Andrew J. Quinn, Benjamin A. E. Hunt, Matthew J. Brookes, Anna C. Nobre, and Mark W. Woolrich.  Spontaneous cortical activity transiently organises into frequency specific phase-coupling networks.  _Nature Communications_ , 9(1):2987, July 2018.  ISSN 2041-1723.  doi: 10.1038/s41467-018-05316-z.  URL https://www.nature.com/articles/s41467-018-05316-z.
  * Wandelt et al. [2024] Sarah K Wandelt, David A. Bjånes, Kelsie Pejsa, Brian Lee, Charles Y Liu, and Richard Andersen.  Representation of internal speech by single neurons in human supramarginal gyrus.  _Nature human behaviour_ , 2024.  URL https://doi.org/10.1038/s41562-024-01867-y.
  * Wang et al. [2023a] Bo Wang, Xiran Xu, Longxiang Zhang, Boda Xiao, Xihong Wu, and Jing Chen.  Semantic reconstruction of continuous language from MEG signals.  _CoRR_ , abs/2309.07701, 2023a.  doi: 10.48550/ARXIV.2309.07701.  URL https://doi.org/10.48550/arXiv.2309.07701.
  * Wang et al. [2023b] Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, and Andrei Barbu.  Brainbert: Self-supervised representation learning for intracranial recordings.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net, 2023b.  URL https://openreview.net/pdf?id=xmcYx_reUn6.
  * Wang and Ji [2022] Zhenhailong Wang and Heng Ji.  Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification.  In _Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Virtual Event, February 22 - March 1_ , pages 5350–5358. AAAI Press, 2022.  doi: 10.1609/AAAI.V36I5.20472.  URL https://doi.org/10.1609/aaai.v36i5.20472.
  * Weiss and Mueller [2012] Sabine Weiss and Horst M. Mueller.  “Too many betas do not spoil the broth”: the role of beta brain oscillations in language processing.  _Frontiers in Psychology_ , 3, 2012.  doi: https://doi.org/10.3389/fpsyg.2012.00201.
  * Willett et al. [2023] Francis R. Willett, Erin M. Kunz, Chaofei Fan, Donald T. Avansino, Guy H. Wilson, Eun Young Choi, Foram Kamdar, Matthew F. Glasser, Leigh R. Hochberg, Shaul Druckmann, Krishna V. Shenoy, and Jaimie M. Henderson.  A high-performance speech neuroprosthesis.  _Nature_ , 620(7976):1031–1036, August 2023.  ISSN 1476-4687.  doi: 10.1038/s41586-023-06377-x.  URL https://www.nature.com/articles/s41586-023-06377-x.
  * Zeghidour et al. [2022] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi.  Soundstream: An end-to-end neural audio codec.  _IEEE ACM Trans. Audio Speech Lang. Process._ , 30:495–507, 2022.  doi: 10.1109/TASLP.2021.3129994.  URL https://doi.org/10.1109/TASLP.2021.3129994.
  * Zhang et al. [2016] Richard Zhang, Phillip Isola, and Alexei A. Efros.  Colorful image colorization.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III_ , volume 9907 of _Lecture Notes in Computer Science_ , pages 649–666. Springer, 2016.  doi: 10.1007/978-3-319-46487-9\\_40.  URL https://doi.org/10.1007/978-3-319-46487-9_40.

##  Appendix A Supervised Heard Speech Decoding

While this work is primarily about scaling speech decoding, we also set the
first baselines for speech detection and voicing classification from MEG with
connected auditory speech stimuli. Therefore, in this section we provide fully
supervised results where our architecture is trained to completion using all
available labelled data in the corresponding dataset (except the validation
and test splits) and compare this to a linear classifier baseline. These
results are not comparable with our self-supervised models because the entire
architecture is trained end-to-end. In our SSL work with shallow fine-tuning,
only the final classifier is trained with labels.

Experiment | Balanced accuracy
---|---
| Speech detection | Voicing
Armeni (linear) | $0.5222\pm 0.0015$ | $0.5207\pm 0.0007$
Armeni (ours) | $\mathbf{0.7253}\pm 0.0005$ | $\mathbf{0.5282}\pm 0.0006$
Gwilliams (linear) | $0.5006\pm 0.0001$ | $0.5166\pm 0.0002$
Gwilliams (ours) | $\mathbf{0.5946}\pm 0.0016$ | $\mathbf{0.5211}\pm 0.0005$
Table 3: Supervised baselines. Uncertainty is standard error of the mean over
three random seeds.

The results in Table 3 show that we can achieve strong speech detection
results on Armeni et al. [2] and Gwilliams et al. [24] with our architecture.
The difference between the two may be due to a combination of different
scanners, numbers of sensors (208 vs 269 magnetometers), varied difficulty in
the audio stimulus, and more within-subject data in Armeni et al. [2]. We also
achieve good results on voicing classification for both datasets, easily
outperforming the linear models.

One potential way to improve on both supervised and self-supervised approaches
is by using a temporally aware model such as an LSTM or an attentional model
such as a Transformer as the classifier. We do not investigate this within
this paper as it is out of scope. Rather, we choose to use simple classifiers
as the purpose of this paper is to demonstrate the potential of scaling.

##  Appendix B Subject Conditioning

To investigate subject conditioning approaches, we pre-train then fine-tune
our models on a subset of Gwilliams et al. [24], holding out three subjects
for evaluation. We conduct these experiments without conditioning, with
subject embeddings, and with FiLM to see if they have any influence on subject
generalisation. We represent embeddings as learnable vectors, unique to each
subject, that are concatenated to the backbone representation. For FiLM, we
use the same embeddings as inputs to the functions that modulate the backbone
representation.

Subject | Subject voicing balanced accuracy
---|---
conditioning | Seen | $t$ | $p$ | Unseen (zero-shot) | $t$ | $p$
None | $0.5118\pm 0.001$ | $8.4$ | $0.007$ | $0.5057\pm 0.001$ | $4.5$ | $0.02$
Embedding | $0.5168\pm 0.0006$ | $27.9$ | $0.0006$ | $0.5052\pm 0.0007$ | $7.5$ | $0.009$
FiLM [44] | $\mathbf{0.5197}\pm 0.001$ | $18.6$ | $0.001$ | $\mathbf{0.5058}\pm 0.002$ | $3.3$ | $0.04$
Chance-level | $0.5$ | $-$ | $-$ | $0.5$ | $-$ | $-$
Table 4: Subject conditioning improves generalisation on seen subjects but not
novel subjects. We quote test accuracy at the point of best validation
accuracy (early stopping). Uncertainty is the standard error of the mean and
we show the $t$-score and $p$-value from a one-sample one-sided $t$-test where
the population mean is chance-level accuracy.

Table 4 reveals that in-distribution subject generalisation is helped by using
subject conditioning methods. Both subject embeddings and FiLM conditioning
outperform no conditioning, with FiLM being best. However, for novel subject
generalisation, using a subject conditioning method makes little difference.
Regardless of conditioning, we can generalise beyond chance-level, but suffer
a reduction in accuracy compared to in-distribution subjects. Conditioning may
be negligible in the unseen case as the embeddings are random rather than
learned for the held-out subjects.

##  Appendix C Aggregating Unlabelled Datasets

To scale up unlabelled data further than individual studies, we must be able
to combine many existing datasets. As a preliminary investigation, we combine
two of the largest public MEG datasets: MOUS [47] and Cam-CAN [48, 53]. We
investigate how pre-training with these combined datasets affects downstream
performance using the same experimental setup as Figure 3.

|  | Balanced accuracy
---|---|---
Labels | Pre-training data (hours) | Speech detection | Voicing
Armeni | MOUS (160 hours) | $0.5736\pm 0.0016$ | $0.5220\pm 0.0013$
| Cam-CAN (159 hours) | $\mathbf{0.5911}\pm 0.0009$ | $\mathbf{0.5241}\pm 0.0004$
| Cam-CAN + MOUS (319 hours) | $0.5872\pm 0.0012$ | $0.5236\pm 0.0016$
Gwilliams | MOUS (160 hours) | $0.5060\pm 0.0006$ | $0.5175\pm 0.0008$
| Cam-CAN (159 hours) | $\mathbf{0.5118}\pm 0.0011$ | $\mathbf{0.5196}\pm 0.0010$
| Cam-CAN + MOUS (319 hours) | $0.5116\pm 0.0017$ | $0.5187\pm 0.0013$
Table 5: Combining two unlabelled datasets does not improve over the best
study. We quote test accuracy at the point of best validation accuracy (early
stopping). The uncertainty is the standard error of the mean calculated over
three seeds.

The results in Table 5 suggest that combining these two datasets is not
strictly better than the best dataset on its own. Pre-training on Cam-CAN
alone leads to the best performance in all instances. This suggests that Cam-
CAN is a better pre-training dataset than MOUS—a surprising result given that
MOUS and Armeni et al. [2] share the same scanning hardware and MOUS includes
data from subjects listening to speech while Cam-CAN has no such language
tasks. This could be due to Cam-CAN being a cleaner dataset as we found
artefacts can significantly affect learning. Combining Cam-CAN with MOUS
performs marginally worse or the same as Cam-CAN alone. For all but one
instance they are within the standard error of the mean of each other. The
combined datasets always outperform MOUS on its own.

In general, further investigation is necessary to determine how datasets
should be best aggregated. Increasing the number of datasets could enable the
network to model the variance between them and eventually improve over the
best singular dataset. This remains an open question as we were only able to
study up to two datasets in this paper.

##  Appendix D Task Labels Are Hard To Scale; Generic SSL Scales Best

Figure 5: Cross-task semi-supervised learning does not improve generalisation.
We pre-train with increasing amounts of labelled data from Armeni et al. [2],
using speech labels when evaluating voicing and voicing labels for speech. We
also use a constant 38 hours of unlabelled data in pre-training. No pre-
training shows test accuracy for a randomly initialised network fine-tuned for
the same number of epochs (i.e. without pre-training). SSL is when pre-
training with only the unlabelled data. The shaded area shows the standard
error of the mean across three seeds.

Our self-supervised tasks led to clearly generalisable and scalable
representations by providing an inductive bias towards speech decoding.
However, could these representations be made more powerful by utilising labels
from other speech decoding tasks to improve this bias? This moves us into the
semi-supervised learning regime, where we use a mix of labelled and unlabelled
data.

With the same experimental setup as Figure 3, we add speech detection or
voicing as an additional loss during pre-training. Figure 5 shows that no
amount of additional labelled data improves downstream accuracy, with all data
points remaining near the range of the standard error of the mean of the SSL
baseline for both tasks. There also does not appear to be any indication of
scaling within the range of labelled data available to us.

This outcome indicates that our self-supervised losses are the main
contributor to downstream accuracy. The MEG signal appears rich, encapsulating
lots of information, with the speech component being a subset distributed
across the representation space. Learning with specific labels produces
representations that are too specialised to improve on other speech decoding
tasks. This does not preclude there being some—or a combination of—labelled
tasks that provide more general inductive biases for speech; nor does it mean
more labelled data will not scale. However, the point remains salient.
Labelled data is more difficult to scale given its scarcity and the generic
property is lost with specific labels while it is well-preserved with SSL,
providing another reminder of the bitter lesson.

##  Appendix E Experiment Details

To generate the data in Table 1, we pre-train our representations with non-
overlapping sample windows from all subjects and sessions 001 to 009 of Armeni
et al. [2] for 200 epochs with a $0.8:0.1:0.1$ train/val/test split. We
shallow fine-tune with labels for 30 epochs using all subjects, but only
session 010, which was held-out during pre-training.

For the data in Figure 3, we use the same number of epochs and the same ratios
for data splitting. We pre-train only with Cam-CAN [48, 53]. We adjust the
amount of unlabelled data used by increasing the number of subjects in the
sequence 1, 2, 4, 8, 17, 36, 74, 152, 312, and 641, successively randomly
selecting more subjects to include. Each seed uses a different set of subjects
to reduce negative effects from outlier subjects. We fine-tune with all data
in Armeni et al. [2] and Gwilliams et al. [24] with the same train/val/test
ratios as before.

For the results shown in Figure 5, we use the same setup as in Figure 3. In
addition to the unlabelled data, we add labelled data from Armeni et al. [2],
increasing the number of sessions used in the sequence 1, 3, 6, 9, 12, and 27.
For this experiment, we calculate the number of labelled data hours using an
estimate of 1 hour per session per subject.

Finally, for the results in Table 4, we pre-train and deep fine-tune to
saturation with subjects 01-07 of Gwilliams et al. [24] and test zero-shot
generalisation using held-out subjects 08-10. All other experimental details
are the same as the previous two experiments.

In all experiments, we use three randomly selected seeds for each pre-training
and corresponding fine-tuning run. For speech detection, since our encoder
reduces the temporal dimension from 125 samples (the number of samples in a
0.5 second window with a sample rate of 250Hz) down to 5 embeddings, we
downsample our speech detection labels to match using PyTorch’s
torch.nn.functional.interpolate. Therefore, each speech detection label
represents a 0.1 second period of time.

##  Appendix F Hyperparameters

We conduct a search over hyperparameters of interest to optimise our self-
supervised objectives and neural architecture. In all experiments in this
section, we measure downstream performance (using shallow fine-tuning) with
best validation accuracy for speech detection unless explicitly specified,
training and evaluating only on data from subject 001 of Armeni et al. [2] to
minimise training time, over three random seeds. All uncertainties in this
section are standard deviations and not standard error of the mean as in the
main body.

Table 6 shows the results of a search over the proportion $\rho$ of sensors to
apply the phase and amplitude pretext task transformations to. We only search
up to $\rho=0.5$ as anything above this is equivalent to applying the inverse
transformation to a proportion $1-\rho$ of the sensors. We find that for phase
shift prediction, $\rho=0.5$ is optimal, while for amplitude scale prediction
$\rho=0.2$ is best.

Task | $\rho$ | Balanced accuracy
---|---|---
Phase shift prediction | 0.1 | $0.5452\pm 0.0048$
| 0.2 | $0.5566\pm 0.0014$
| 0.3 | $0.5578\pm 0.0021$
| 0.4 | $0.5530\pm 0.0014$
| 0.5 | $\mathbf{0.5655}\pm 0.0026$
Amplitude scale prediction | 0.1 | $0.5440\pm 0.0012$
| 0.2 | $\mathbf{0.5796}\pm 0.0050$
| 0.3 | $0.5625\pm 0.0063$
| 0.4 | $0.5684\pm 0.0016$
| 0.5 | $0.5650\pm 0.0026$
Table 6: Best $p$ for phase and amplitude pretext tasks.

In our architecture, we define $d_{\mathrm{backbone}}$ to be the bottleneck
dimension of the cortex encoder (i.e. the embedding dimension) as well as the
size of $d_{\mathrm{shared}}$. Balestriero et al. [3], using the results from
work by Bordes et al. [4], suggest that increasing the backbone output
dimension in self-supervised learning leads to better downstream transfer. We
investigate this for our architecture by varying the encoder dimension in
Table 7. Our results are consistent with their findings. Balanced accuracy is
highest at the largest encoder dimension we evaluated. Encoder dimensions
larger than $2048$ led to unstable training and were not feasible given the
compute resources available to us.

$d_{\mathrm{backbone}}$ | Balanced accuracy
---|---
$128$ | $0.5775\pm 0.0037$
$300$ | $0.5715\pm 0.0012$
$512$ | $0.5835\pm 0.0016$
$1024$ | $0.5800\pm 0.0050$
$2048$ | $\mathbf{0.5987}\pm 0.0024$
Table 7: Optimal encoder dimension.

In Table 8, we show the effect of varying the length $w$, i.e. the window size
of each data sample. Increasing the window to five seconds led to unstable
losses in pre-training so we did not evaluate above a window length of two
seconds. This is likely to be because increasing the window size reduces the
number of training samples when only non-overlapping samples are used. For
speech detection, there appears to be a U-shaped distribution, where very
short and very long windows lead to better downstream accuracy. For voicing
classification, the window lengths we evaluated seem to make little
difference. However, windows smaller than 0.1 seconds may lead to
significantly lower accuracy as most phonemes will occur for at least this
length of time in the neural response [23].

Window length (s) | Balanced accuracy
---|---
| Speech detection | Voicing classification
$0.2$ | $0.5919\pm 0.0014$ | $0.5033\pm 0.0014$
$0.4$ | $0.5804\pm 0.0036$ | $\mathbf{0.5077}\pm 0.0027$
$0.5$ | $0.5715\pm 0.0012$ | $0.5055\pm 0.0023$
$1.0$ | $0.5790\pm 0.0012$ | $0.5062\pm 0.0020$
$2.0$ | $\mathbf{0.6116}\pm 0.0019$ | $0.5065\pm 0.0011$
Table 8: Ideal sample window length.

We also evaluated several different configurations of the SEANet architecture
for the cortex encoder, converging on convolutional blocks with channel
dimensions $(512,512,512,512)$.

While these ablations indicate a theoretically ideal architectural
configuration, in practice, we altered our final experimental architecture due
to instabilities during training when data was scaled up. Our final
architecture hyperparameters achieve a balance between the best values from
our hyperparameter search and stable training. These values are detailed in
Table 9.

Hyperparameter | Value
---|---
Window length (s) | $0.5$
$\rho$ (phase) | $0.5$
$\rho$ (amplitude) | $0.2$
$\\{w_{1},w_{2},w_{3}\\}$ | $\\{1.0,1.0,1.0\\}$
$d_{\mathrm{shared}}$ | $512$
$d_{\mathrm{backbone}}$ | $512$
SEANet convolution channels | $(512,512,512,512)$
SEANet downsampling ratios | $(5,5,1)$
FiLM conditioning dimension | $16$
Subject embedding dimension | $16$
Pre-training epochs | $200$
Optimizer | AdamW [33]
Learning rate | $0.000066$
Train ratio | $0.8$
Validation ratio | $0.1$
Test ratio | $0.1$
Table 9: Experimental hyperparameters.

##  Appendix G Compute Resources

All experiments were run on individual NVIDIA V100 and A100 GPUs with up to
40GiB of GPU memory on a system with up to 1TiB of RAM. Each pre-training run
with the maximum amount of pre-training data took approximately 200 hours (8.3
days). Fine-tuning following pre-training took up to another 12 hours. We
estimate that we used approximately 3000 hours of compute for the final
experimental runs, including hyperparameter searches. In total, over the
course of developing this work from idea to final paper, we used around 10,000
hours of GPU compute.

##  Appendix H Licences For Datasets And Code

The Armeni et al. [2] dataset is distributed under CC-BY-4.0 while the
Gwilliams et al. [24] dataset is distributed under the CC0 1.0 Universal
licence. The Schoffelen et al. [47] dataset is distributed with a RU-DI-HD-1.0
licence from the Donders institute. The licence for the Cam-CAN [48, 53]
dataset is unknown. The SEANet code adapted from Défossez et al. [14] is
distributed under the MIT licence, and the OSL library, which we use for
preprocessing, is under the BSD-3-Clause licence.

##  Appendix I Broader Impacts

Decoding speech from non-invasive brain recordings is likely to bring about
significant positive societal impacts. Research in this field could enable
paralysed patients to communicate freely and materially assist those with
minor communication difficulty (e.g. stammering). As the technology matures,
it could also enable new ways of communicating with others and interacting
with devices without the risks of invasive surgical implants. Nevertheless,
the maturity of this technology could also present potential negative societal
impacts. For one, reading inner speech creates new concerns over data controls
as this information is likely to be highly sensitive and personal to
individuals. Given access to this technology, there is also the risk that bad
actors could extract sensitive information from target individuals without
consent. Moreover, there are possible long horizon effects associated with
speech decoding research. Broad adoption of this technology could lead to the
gradual erosion of privacy over inner speech within society. In addition,
asymmetric effects, where some individuals or organisations can read inner
speech but others are unable to, could worsen societal inequality. Within the
scope of this paper, we mitigate risks associated with inner speech by
focusing on decoding heard speech where there is low potential for abuse.
Nonetheless, we acknowledge that this is still a stepping stone towards
solving inner speech decoding.