# Language Reconstruction with Brain Predictive Coding from fMRI Data

Congchi Yin1,2 Ziyi Ye3 Piji Li1,2
1Nanjing University of Aeronautics and Astronautics
2MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing,
China
3Tsinghua University, Beijing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding author.

###### Abstract

Many recent studies have shown that the perception of speech can be decoded
from brain signals and subsequently reconstructed as continuous language.
However, there is a lack of neurological basis for how the semantic
information embedded within brain signals can be used more effectively to
guide language reconstruction. The theory of predictive coding suggests that
human brain naturally engages in continuously predicting future word
representations that span multiple timescales. This implies that the decoding
of brain signals could potentially be associated with a predictable future. To
explore the predictive coding theory within the context of language
reconstruction, this paper proposes a novel model PredFT for jointly modeling
neural decoding and brain prediction. It consists of a main decoding network
for language reconstruction and a side network for predictive coding. The side
network obtains brain predictive coding representation from related brain
regions of interest with a multi-head self-attention module. This
representation is fused into the main decoding network with cross-attention to
facilitate the language models’ generation process. Experiments are conducted
on the largest naturalistic language comprehension fMRI dataset Narratives.
PredFT achieves current state-of-the-art decoding performance with a maximum
BLEU-1 score of $27.8\%$.

##  1 Introduction

Reconstructing natural language from functional magnetic resonance imaging
(fMRI) signals offers potential insights into understanding the formation of
language in the human brain. Recent studies have attempted to leverage brain
signals with computational language models to generate coherent, naturally
flowing languages [3, 37, 1, 44]. This advancement is achieved based on the
ability of computational language models to craft coherent language, with
brain responses serving as guidance in the generation process, thereby
aligning the model’s output with human perception. For example, Tang et al.
[33] used a GPT [27] model to generate semantic candidates, and then brain
signals are employed to select the content that is more aligned with the
semantic content perceived by humans. Xi et al. [41] proposed to fuse the
brain representation as the input for language model and achieves language
reconstruction through sequence-to-sequence machine translation [32].

Despite relatively good decoding quality, existing works often overlook how
the natural language is encoded within human brain as well as its
representation within language models, which is significant for understanding
the language reconstruction process. Predictive coding [20, 29, 9] provides an
attractive theory for an unified view of neural encoding and decoding. It
suggests that human brain naturally makes predictions of future words over
multiple timescales when receiving current phonetic stimuli. Previous
neuroscience studies [39, 24] have already evidenced such speech prediction in
human brain through fMRI. Caucheteux et al. [5] further investigated the
correlation between language model representation and brain predictive coding.
After confirming the activations of modern language models could be linearly
mapped to brain responses, they demonstrated that such brain mapping would be
enhanced by adding predictive content to language model input. These findings
in neural encoding remind us that in the language reconstruction process,
treating brain signals as inputs without considering temporal relationships
related to the predictive process may not be sufficient. However, whether
brain predictive coding could really help facilitate fMRI-to-text decoding and
how to exploit the prediction in decoding still remain unexplored.

To investigate predictive coding from the perspective of decoding, we first
conduct a preliminary experiment to analyze the capability of brain signals in
predicting future content and their associative relationship with the
representations of language models. We verify the predictive coding ability of
brain signals and identify six brain regions of interests that are most
related to the predictive coding functions. Based on the observations, we
propose PredFT, abbreviation of FMRI-to-Text decoding with Predictive coding,
for jointly modeling language reconstruction and brain predictive coding.
PredFT is an end-to-end model with a main decoding network for language
reconstruction and a side network for providing brain predictive coding
heuristics. The main decoding network consists of an encoding model for
spatial-temporal feature extraction and a Transformer [35] variant for
generation. The encoding model applies a 3D convolutional neural network [14]
to extract spatial features of fMRI and a finite impulse response model [13]
to compensate for the delay of blood-oxygen-level-dependent signal. At the
same time, the side network extracts and fuses six regions of interest related
to brain predictive coding, and then builds mappings with the main decoding
network through cross-attention mechanism.

Experiments are conducted on a large naturalistic language comprehension fMRI
dataset Narratives [23]. First, we present the overall decoding performance of
PredFT. We show that its decoding performance outperforms existing proposed
methods in terms of a series of language similarity metrics. Second, we
explore whether the selection of regions of interest for the side network will
affect the decoding performance of PredFT. This experiment aims to confirm
that the brain predictive coding functions only originate in certain spatial
regions from decoding perspective. Last, we analyze the distance in predictive
coding to better understand how the human brain makes predictions over
multiple timescales and its influence on language reconstruction performance.

The main contributions of this paper can be summarized as follows: (i) To the
best of our knowledge, we first investigate the brain predictive coding issue
from the perspective of decoding brain signals into natural language. (ii) We
propose an end-to-end model PredFT for fMRI-to-text decoding, which features
effectively utilizing brain predictive coding representation to improve
decoding performance through a side network. (iii) Experimental results show
that PredFT benefits from the joint modeling of brain predictive coding and
achieves state-of-the-art decoding performance.

##  2 Predictive Coding Verification

In this section, we elaborate on predictive coding in human brain by analyzing
the correlation between brain responses stimulated by spoken words and
activations of language model with the spoken words as input. Following
previous study [4], the brain score
$R(\mathbf{X})=corr(f(\mathbf{X}),\mathbf{Y})$ is first defined, which
measures the pearson correlation between language model activations
$\mathbf{X}\in\mathbb{R}^{M\times D}$ and brain responses
$\mathbf{Y}\in\mathbb{R}^{N\times V}$, where $f$ is a linear ridge regression
model with $\ell_{2}$-regularization for linear mapping. $M$ and $N$ stand for
the number of words and fMRI frames; $D$ and $V$ stand for the output
dimension of language model and number of voxels in brain. Similar to [5],
prediction score
$P_{(d,l)}(\mathbf{X})=R(\mathbf{X}\oplus\mathbf{X}^{d}_{l})-R(\mathbf{X})$ is
proposed, which is the difference between brain score of the concatenation of
current word with future words and brain score of the current word.
$\mathbf{X}^{d}_{l}$ indicates the representation of $l$ continuous future
heard words with prediction distance $d$ which measures the distance from
current word to the last future heard word. Sub-figure (a) of Figure 1 shows
an example of $\mathbf{X}^{5}_{2}$. Prediction score reflects the degree of
predictive coding. A positive value suggests long-range prediction helps
improve the correlation between language model activations and brain
responses. In experiment, the output of eighth layer in GPT-2 [28] is applied
for $\mathbf{X}$. The brain score is averaged across subjects and computed
within one fMRI frame, namely $N$ is set as 1. The prediction score is
calculated with fixed window length $l=2$ and distance $d$ ranging from 0 to
11.

Figure 1: Example and results of the predictive coding verification
experiment.

The sub-figure (b) in Figure 1 reports the prediction score across individuals
with $95\%$ confidence intervals. Results show prediction score
$P_{(d,l)}(\mathbf{X})$ first increases and peaks at $d=4$, then decreases as
the prediction distance $d$ increases, and finally goes down below zero when
the prediction distance comes to $d=10$. We also conduct regions of interest
(ROI) analysis. Six regions related to brain predictive coding, including
superior temporal sulcus, angular gyrus, supramarginal gyrus, and opercular,
triangular, and orbital part of the inferior frontal gyrus in the left
hemisphere, are selected for experiments. Sub-figure (c) shows the prediction
score of six ROIs across individuals with $95\%$ confidence intervals. The
prediction distance is set at d = 4 for the best predictive performance, as
reflected in sub-figure (b). All the selected ROIs show positive response to
prediction words.

The above verification experiment highlights the correlation between language
model representation and brain predictive coding. One possible explanation for
the phenomenon is that both language models and brain predictive coding are
similar in the objective of upcoming words prediction. This inspires us with
the following motivation: While predictive coding has been verified from the
perspective of encoding, could it help in decoding brain signals into natural
language with auto-regressive generation? We propose PredFT for investigating
the effectiveness of utilizing brain prediction in fMRI-to-text decoding.
Details are introduced in the next section.

##  3 Methodology

We first formalize the fMRI-to-text decoding task. Given a naturalistic
language comprehension fMRI dataset $\mathcal{D}:=\\{\left\langle
F_{ij},U_{j}\right\rangle\\}$, where $U_{j}$ is $j$-th fragment of word
stimuli and $F_{ij}$ is the fMRI data collected while the $i$-th subject is
hearing the content of $U_{j}$. The fMRI-to-text decoding task aims to build a
model $\mathcal{M}$ that decodes $U_{j}^{\prime}=\mathcal{M}(F_{ij})$ to
maximize the similarity between $U_{j}$ and $U_{j}^{\prime}$, which is
measured by a text similarity metric $D$. Specifically, the word stimuli
segment $U_{j}=\\{u_{j}^{T},u_{j}^{T+1},\ldots,u_{j}^{T+k}\\}$ is the auditory
content presented to the subject from time step $T$, where each $u_{j}^{T}$
contains a series of corresponding word tokens. Similarly,
$F_{ij}:=\\{f_{i,j}^{T},f_{i,j}^{T+1},\ldots,f_{i,j}^{T+k}\\}$ consists of
$k{+1}$ continuous fMRI images starting from time step $T$. Each $u_{j}$
matches $f_{i,j}$ at corresponding time step $t$.

We propose PredFT, which integrates brain predictive coding function in the
language reconstruction process. PredFT is denoted as
$\mathcal{M}_{\theta,\phi}$, containing a main decoding network
$\mathcal{M}_{\theta}$ and a side network $\mathcal{M}_{\phi}$ for brain
predictive coding. We first introduce the main decoding network
$\mathcal{M}_{\theta}$ which can reconstruct accurate text from fMRI with
computational language models. Then we elaborate on the side network
$\mathcal{M}_{\phi}$ which extracts and exploits brain predictive coding.
Finally, the fusion of brain predictive coding representation and the joint
training of $\mathcal{M}_{\theta}$ and $\mathcal{M}_{\phi}$ are introduced.

###  3.1 Main Decoding Network

As shown in Figure 2, the main decoding network $\mathcal{M}_{\theta}$
consists of an encoder $\mathcal{M}_{\theta_{\text{Enc}}}$ and a decoder
$\mathcal{M}_{\theta_{\text{Dec}}}$. The encoder
$\mathcal{M}_{\theta_{\text{Enc}}}$ is stacked with a 3D-CNN module, a finite
impulse response (FIR) model [13], and a Transformer encoder [35]. For each
fMRI image $f_{i,j}^{T}\in\mathbb{R}^{w\times h\times d}$, where $w,h,d$
represents the width, height, and depth, voxel-level normalization is first
applied. Details are described in Appendix A. The fMRI image after
normalization is denoted as $\hat{f}_{i,j}^{T}$, which is then fed into the
3D-CNN module. The 3D-CNN module contains $L$ layers of group normalization
[40], ReLU activation, and convolution layer [16], with residual connection
[11]. The size of fMRI image $\hat{f}_{i,j}^{T}$ is progressively reduced by
convolution layer and finally downsized to
$\hat{f}_{i,j}^{T}\in\mathbb{R}^{w^{\prime}\times h^{\prime}\times
d^{\prime}\times c}$ where $c$ is the number of output channels. A flatten
layer and a linear layer are used to obtain a one-dimensional vector
$x_{i,j}^{T}\in\mathbb{R}^{d_{m}}$ as the output of the 3D-CNN module. After
that, a FIR model $g_{t}$ is applied to compensate for the latency of blood-
oxygen-level-dependent (BOLD) signal. For $k{+1}$ continuous fMRI images with
$t\in\\{T,T{+1},\ldots,T{+k}\\}$, the temporal transformation $g_{t}$
concatenates $k{-k^{*}}$ future frames to form the representation in time step
$t$:

$\displaystyle g_{t}:\mathbb{R}^{k\times d_{m}}$
$\displaystyle\rightarrow\mathbb{R}^{k^{*}\times(d_{m}(k-k^{*}))}$ (1)
$\displaystyle x_{i,j}^{t}$ $\displaystyle\mapsto
concat(x_{i,j}^{t},x_{i,j}^{t+1},\ldots,x_{i,j}^{t+(k-k^{*})}),t\in\\{T,T{+1},\ldots,T{+k^{*}}\\}.$

where $k-k^{*}$ is the number of delays. Then a linear layer
$\mathbf{W}_{ll}\in\mathbb{R}^{(d_{m}\dot{(}k-k^{*}))\times d_{m}}$ is used to
fuse delayed brain responses and recover $x_{i,j}^{t}$ into its original
dimension $d_{m}$. After learning the spatial features of fMRI images, the
representations with learnable time positional embeddings, denoted as
$\mathbf{H}^{0}_{\theta_{\text{Enc}}}\in\mathbb{R}^{k^{*}\times d_{m}}$, are
sent into a Transformer encoder to capture temporal features within given
intervals. The output of the Transformer encoder is
$\mathbf{H}^{P}_{\theta_{\text{Enc}}}=\mathcal{M}_{\theta_{\text{Enc}}}(\mathbf{H}^{0}_{\theta_{\text{Enc}}})$,
where $P$ is the number of Transformer encoder layers.

Figure 2: The general framework of PredFT. The italic type words in the input
word sequence stand for the first heard word of each fMRI image while the bold
words stand for the prediction words.

Finally, the output from $\mathcal{M}_{\theta_{\text{Enc}}}$, i.e.,
$\mathbf{H}^{P}_{\theta_{\text{Enc}}}$, is fed into the decoder
$\mathcal{M}_{\theta_{\text{Dec}}}$. $\mathcal{M}_{\theta_{\text{Dec}}}$
contains modules within a standard Transformer decoder consisting of masked
multi-head self-attention, multi-head encoder-decoder attention, and feed-
forward layer. Besides, additional masked multi-head cross-attention layer and
feed-forward layer are designed for $\mathcal{M}_{\theta_{\text{Dec}}}$. The
cross-attention layer integrates brain predictive coding representations
inherited from the side network. More details will be introduced in Section
3.3 and here we just view it as an encoder-decoder attention layer. The
shifted right text $U_{j}^{r}$ is first tokenized and sent into a word
embedding layer $\mathbf{W}_{\text{word}}$ to get representations
$\mathbf{H}^{0}_{\theta_{\text{Dec}}}$. We denote the input of the
$(l{+1})$-th self-attention layer as $\mathbf{H}^{l}_{\theta_{\text{Dec}}}$,
so the self-attention is calculated by

$\mathrm{Self\text{-}Attn}(\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{Q}^{l},\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{K}^{l},\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{V}^{l})=\mathrm{softmax}(\frac{\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{Q}^{l}(\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{K}^{l})^{\top}}{\sqrt{d_{k}}})\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{V}^{l},$
(2)

where
$\mathbf{W}_{Q}^{l},\mathbf{W}_{K}^{l},\mathbf{W}_{V}^{l}\in\mathbb{R}^{d_{m}\times
d_{k}}$ are the parameter matrices of projecting query, key, value in the
$(l{+1})$-th corresponding layer (i.e. here is the self-attention layer) for
simplicity. The encoder-decoder attention aims to integrate fMRI
representations. It takes $\mathbf{H}^{l}_{\theta_{\text{Dec}}}$ as query and
$\mathbf{H}^{P}_{\theta_{\text{Enc}}}$ for key and value:

$\mathrm{ED\text{-}Attn}(\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{Q}^{l},\mathbf{H}^{P}_{\theta_{\text{Enc}}}\mathbf{W}_{K}^{l},\mathbf{H}^{P}_{\theta_{\text{Enc}}}\mathbf{W}_{V}^{l})=\mathrm{softmax}(\frac{\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{Q}^{l}(\mathbf{H}^{P}_{\theta_{\text{Enc}}}\mathbf{W}_{K}^{l})^{\top}}{\sqrt{d_{k}}})\mathbf{H}^{P}_{\theta_{\text{Enc}}}\mathbf{W}_{V}^{l}.$
(3)

The attention masks for self-attention and encoder-decoder attention are the
same as vanilla Transformer. The final output of
$\mathcal{M}_{\theta_{\text{Dec}}}$ is denoted as
$\mathbf{H}^{Q}_{\theta_{\text{Dec}}}$ where $Q$ is the number of decoder
layers.

###  3.2 Side Network for Predictive Coding

The idea of designing a side network $\mathcal{M}_{\phi}$ for brain predictive
coding is motivated by predictive coding theory [20, 29, 9], which indicates
the human brain naturally makes predictions about words over multiple
timescales. Since brain predictive coding has been verified from the
perspective of encoding (the linear mapping between language models and brain
responses), we hope to exploit it by training a neural network to well
represent regions involved in prediction and fusing brain predictive coding
representations in decoding. A vanilla Transformer $\mathcal{M}_{\phi}$ with
encoder $\mathcal{M}_{\phi_{\text{Enc}}}$ and decoder
$\mathcal{M}_{\phi_{\text{Dec}}}$ is applied for brain predictive coding.
Specifically, it takes
$R_{ij}:=\\{r_{i,j}^{T},r_{i,j}^{T+1},\ldots,r_{i,j}^{T+k}\\}$ as encoder
input and $V_{j}^{r}:=\\{$<$s$>$,v_{j}^{T},v_{j}^{T+1},\ldots,v_{j}^{T+k}\\}$
as decoder input. As shown in Figure 2, each
$r_{i,j}^{T}\in\mathbb{R}^{d_{r}}$ in $R_{ij}$ extracted from $f_{i,j}^{T}$ in
$F_{ij}$ is one-dimensional cerebral cortex data. It stands for the
concatenation of regions of interest (ROIs) related to brain predictive coding
as verified in Section 2. Each $v_{j}^{T}$ in $V_{j}^{r}$ stands for the $l$
future words with prediction distance $d$ (recall the definition in Section 2)
of $u_{j}^{T}$ in $U_{j}^{r}$ that human brain predicts. The side network
$\mathcal{M}_{\phi}$ is trained to learn the mapping from brain predictive
responses to ground truth future words.

Figure 3: Illustration of attention masks. Grey color indicates mask. The
cross-attention mask is transposed for simplicity of expression.

The ROI fusion layer is a fully connected feed-forward network that outputs
the representation $r_{i,j}^{T}\in\mathbb{R}^{d_{m}}$. The same FIR model in
the main decoding network is applied to compensate for the delays of the BOLD
signal. Learnable time positional embedding is added to $r_{i,j}^{T}$ before
entering into the Transformer encoder $\mathcal{M}_{\phi_{\text{Enc}}}$ with
$M$ layers. The output of $\mathcal{M}_{\phi_{\text{Enc}}}$ is denoted as
$\mathbf{H}^{M}_{\phi_{\text{Enc}}}$, which stands for the representation of
brain predictive coding. $\mathbf{H}^{M}_{\phi_{\text{Enc}}}$ plays an
essential role in PredFT, as it will be fused into the main decoding network
to verify the effectiveness of brain predictive coding in the decoding task.
The Transformer decoder $\mathcal{M}_{\phi_{\text{Dec}}}$ follows the
conventional practice of masked self-attention and encoder-decoder attention,
which has been elaborated in Equation 2 and Equation 3 of Section 3.1,
respectively.

###  3.3 Prediction Fusion and Joint Training

The subsection details how the brain predictive coding representation from the
side network fuses into the main decoding network. As shown in Figure 2, the
brain predictive coding representation $\mathbf{H}^{M}_{\phi_{\text{Enc}}}$,
which is the output of $\mathcal{M}_{\phi_{\text{Enc}}}$, plays as keys and
values for the masked cross-attention in the main decoding network. The
queries of cross-attention layer are the output from the previous encoder-
decoder attention layer and feed-forward layer. The cross attention if
formularized as:

$\mathrm{Cross\text{-}Attn}(\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{Q}^{l},\mathbf{H}^{M}_{\phi_{\text{Enc}}}\mathbf{W}_{K}^{l},\mathbf{H}^{M}_{\phi_{\text{Enc}}}\mathbf{W}_{V}^{l})=\mathrm{softmax}(\frac{\mathbf{H}^{l}_{\theta_{\text{Dec}}}\mathbf{W}_{Q}^{l}(\mathbf{H}^{M}_{\phi_{\text{Enc}}}\mathbf{W}_{K}^{l})^{\top}}{\sqrt{d_{k}}})\mathbf{H}^{M}_{\phi_{\text{Enc}}}\mathbf{W}_{V}^{l}.$
(4)

The mask $\mathbf{M}_{\text{ca}}\in\mathbb{R}^{k_{t}\times k^{*}}$ of cross-
attention is shown in Figure 3(a). $k_{t}$ and $k^{*}$ are the numbers of
input tokens and fMRI signals, respectively. The cross-attention mask
$\mathbf{M}_{\text{ca}}$ is designed in this way: for each token in the text
fragment $u_{j}^{t}$, all the predictive coding representations after time
step $t$ are allowed to attend, while previous representations are masked.

PredFT is trained in an end-to-end manner. The main decoding network
$\mathcal{M}_{\theta}$ and the side network $\mathcal{M}_{\phi}$ share the
same word embedding layer $\mathbf{W}_{\text{word}}$, whose parameters are
only updated with the gradient flow from $\mathcal{M}_{\theta}$ during
training. The training objective follows a left-to-right auto-regressive
language modeling manner for both $\mathcal{M}_{\theta}$ and
$\mathcal{M}_{\phi}$. Specifically, following $\mathcal{M}_{\theta}$ and
$\mathcal{M}_{\phi}$ are two language model heads $\mathbf{W}_{\theta}$ and
$\mathbf{W}_{\phi}$. The training loss for $\mathcal{M}_{\theta}$ is

$\mathcal{L}_{\text{Main}}=-\sum_{t=1}^{n}\log
P(y_{t}|y_{<t},U_{j}^{r};\theta),$ (5)

where $U_{j}^{r}$ is the input and $y_{t}$ is the generated token. Similarly,
the training loss for $\mathcal{M}_{\phi}$ is

$\mathcal{L}_{\text{Side}}=-\sum_{t=1}^{n}\log
P(z_{t}|z_{<t},V_{j}^{r};\phi).$ (6)

The joint training of $\mathcal{M}_{\theta}$ and $\mathcal{M}_{\phi}$ is to
optimize
$\mathcal{L}=\mathcal{L}_{\text{Main}}+\lambda\mathcal{L}_{\text{Side}}$,
where $\lambda$ is a hyper-parameter.

##  4 Experimental Settings and Results

Several experiments are conducted to (i) evaluate the decoding performance of
PredFT (ii) detailedly analyze how brain predictive coding work in PredFT.
First we briefly introduce the experimental settings, while more details are
shown in Appendix A. Then the results of decoding performance, regions of
interest selection analysis, and prediction distance analysis are displayed.
Besides, we investigate the decoding error distribution of different models in
Appendix B and conduct ablation study in Appendix C.

Table 1: Performance of different models in fMRI-to-text decoding. Length denotes to the length of continuous fMRI frames. Length | Model | BLEU-N ($\%$) | ROUGE-1 ($\%$)
---|---|---|---
$N=1$ | $N=2$ | $N=3$ | $N=4$ | $R$ | $P$ | $F$
10 | UniCoRN | 20.64 | 5.03 | 1.40 | 0.45 | 15.56 | 25.47 | 19.23
PredFT | 24.73 | 8.39 | 3.92 | 1.86 | 14.07 | 35.28 | 19.53
20 | UniCoRN | 18.02 | 4.71 | 1.32 | 0.4 | 18.01 | 29.46 | 20.82
PredFT | 25.98 | 5.61 | 1.36 | 0.21 | 19.61 | 25.43 | 22.09
40 | UniCoRN | 21.76 | 5.43 | 1.17 | 0.34 | 19.76 | 35.33 | 25.30
PredFT | 27.80 | 8.29 | 2.00 | 0.54 | 19.53 | 38.95 | 25.96
Table 2: The decoding performance of PredFT when different ROIs are selected for the side network. Random means randomly picking ROIs. Whole stands for using whole human cerebral cortex. BPC denotes to the ROIs related to brain predictive coding. Length | ROIs Selection | BLEU-N ($\%$) | ROUGE-1 ($\%$)
---|---|---|---
$N=1$ | $N=2$ | $N=3$ | $N=4$ | $R$ | $P$ | $F$
10 | None | 18.08 | 3.98 | 1.05 | 0.28 | 14.96 | 26.21 | 18.96
Random | 16.83 | 3.04 | 0.63 | 0.13 | 16.71 | 19.23 | 17.28
Whole | 21.99 | 4.51 | 0.83 | 0.25 | 17.36 | 22.83 | 19.43
BPC | 24.73 | 8.39 | 3.92 | 1.86 | 14.07 | 35.28 | 19.53
20 | None | 20.37 | 3.86 | 1.03 | 0.19 | 17.42 | 22.15 | 19.45
Random | 16.11 | 3.28 | 0.55 | 0.12 | 19.27 | 24.29 | 21.44
Whole | 23.55 | 6.39 | 1.33 | 0.39 | 15.66 | 30.98 | 20.72
BPC | 25.98 | 5.61 | 1.36 | 0.21 | 19.61 | 25.43 | 22.09
40 | None | 18.01 | 4.72 | 1.27 | 0.34 | 16.41 | 34.36 | 22.16
Random | 19.71 | 5.01 | 1.22 | 0.39 | 20.02 | 29.61 | 24.55
Whole | 24.67 | 5.81 | 1.14 | 0.39 | 20.53 | 29.46 | 24.16
BPC | 27.80 | 8.29 | 2.00 | 0.54 | 19.53 | 38.95 | 25.96

###  4.1 Baselines and Evaluation Metrics

The decoding performance of PredFT is compared to current state-of-the-art
model UniCoRN [41]. We don’t test [33] and [42] in experiments because their
model settings aren’t compatible with the dataset splitting method or
preprocessed version we applied. Details are discussed in Appendix A.

Automatic evaluation metrics including BLEU [25] and ROUGE [19] are applied to
measure decoding performance of different models. BLEU measures the n-gram
overlap between decoded content and ground truth. ROUGE-N computes the
precision, recall, and F1-score based on the n-gram overlap. Specifically,
BLEU-1/2/3/4 and ROUGE1-R/P/F are used for all experiments.

###  4.2 Decoding Performance

We compare the decoding performance of different models on automatic
evaluation metrics. To further investigate the effect of fMRI sequence length
on decoding model performance, experiments are separately conducted with fMRI
sequence length of 10, 20, and 40, which equals to 15s, 30s, and 60s of fMRI
with 1.5s TR. To the best of our knowledge, we are the first to try decoding
one-minute of fMRI data. Results are shown in Table 1. Despite UniCoRN’s good
performance in some evaluation index like ROUGE1-R, PredFT achieves the best
overall performance on all three experiments with different fMRI sequence
lengths. Specifically, it gets the highest BLEU-1 score of $27.8\%$ on
decoding 40 continuous fMRI frames. From the relative low results of
BLEU-2/3/4, we find all the models struggle to generate long accurate text.
It’s still a challenging task to decode fMRI to text. Furthermore, we don’t
find clear evidence indicating that the fMRI sequence length has an impact on
the performance of the decoding models from experiments.

###  4.3 Regions of Interest Selection

To better understand whether PredFT benefits from introducing brain predictive
coding, we select different regions of interest (ROIs) for the side network to
test their influence on decoding performance. Results are listed in Table 2.
Specifically, four types of ROIs are selected: (a) “None” indicates no ROIs
are chosen. The side network together with the cross-attention layer in the
main decoding network is detached from PredFT under this setting. (b) “Random”
means we randomly pick six ROIs apart from brain predictive coding regions.
Specifically, G_and_S_cingul-Ant, G_and_S_subcentral,
G_and_S_transv_frontopol, G_orbital, S_front_middle, S_subparietal are chosen
in experiment. (c) “Whole” means the whole human cerebral cortex (74 ROIs in
the used atlas, see Appendix A) is applied for brain prediction. (d) “BPC”
denotes to the six ROIs related to brain predictive coding as verified in
Section 2. BPC leads to the best performance while whole ROIs selection also
performs well. However, both random selection and non-selection lead to poor
decoding accuracy, and random selection causes worse performance in most
cases. Two conclusions could be drawn from the above ROIs analysis. First, the
predictive coding information only contains in specific regions of human
brain. Second, brain predictive coding can be beneficial to decoding models
with proper network architecture design (e.g. PredFT).

###  4.4 Prediction Distance Analysis

In this section, we investigate the influence of prediction distance to the
decoding performance of PredFT. Experiments are conducted with a fixed window
length $l=2$ and the influence of prediction distance is reflected through
BLEU-1 score. Figure 4 shows the results with different fMRI sequence lengths.
We notice a similar phenomenon as the predictive coding verification
experiment in Section 2. The trend of BLEU-1 score first rises then falls as
prediction distance $d$ increases. PredFT achieves the best performance with
prediction distance around four. The prediction distance analysis highlights
two findings: (i) The design of a side network in PredFT successfully fuses
brain prediction for decoding. (ii) The influence of prediction distance in
fMRI-to-text decoding is consistent with that in linearly mapping language
models to brain response.

Figure 4: The influence of prediction distance to decoding performance. Table 3: Two cases of decoded content. Bold type words indicate key concepts or phrases. Case1 |  Truth: It was more real than any dream he had ever had in his life. He could still hear and feel that sharp metal ripping explosion that searing wave of heat. He sat
---|---
|  UniCoRN: It’s than I just a said of a a the hand. You have I the the the
you I I I in to at the to to sit.
|  PredFT: It’s a more than normal just good dream about to said. He open the
eyes I have be her I like she and and and he and of under the next and I look
the platform
Case2 |  Truth: He couldn’t shake the thought out of his mind. It persisted all through the day until dinner. He was still brooding as he and Mary got ready for bed. Guy dear. Hm oh no. Anything wrong
|  UniCoRN: And I know a Dean and a eyes it. And of the first and the and a to
guy him. I said I to to him out and And to in that in the end.
|  PredFT: He don’t know my girl you of the eyes but his girl sleep he and he
said and he said and the to the and and which I not wrong. But the Guy

###  4.5 Case Study

Some fMRI-to-text decoding cases with fMRI sequence length equals to 10 are
listed in Table 3. From the cases, we find PredFT successfully decodes some
high-level semantic concepts and key words compared to UniCoRN. For example,
in case1 PredFT conveys the meaning of “the best dream” while UniCoRN fails
to. In case2, PredFT decodes the meaning of “he and Mary prepare to sleep”.
Overall, PredFT performs better than UniCoRN. However, despite the relative
good automatic evaluation performance, all models struggle to decode generally
accurate content, especially in (i) generating fluent and coherent sentences
(ii) capturing fine-grained semantic meanings (iii) decoding terminology (e.g.
name “Mary”) or complicated phrases (e.g. “sharp metal ripping explosion”).

##  5 Related Work

#### fMRI-to-Text Decoding.

Previous studies focused on aligning fMRI signal to a limited vocabulary of
hundred tokens and performing word-level decoding [3, 37, 1, 44], or
incorporating fMRI representation into sentence embedding and applying
pairwise classification for sentence-level generation [26, 31]. Recently,
researchers turned to powerful pretrained language models for open-vocabulary
fMRI-to-text decoding. Tang et al. [33] designed a pipeline model where the
encoder is responsible for identifying the most possible word sequence among
candidates generated by GPT-2 model with beam search. Xi et al. [41] proposed
a three-phase training framework UniCoRN which applies BART [17] model for
generation. Ye et al. [42] proposed BrainLLM by concatenating fMRI embedding
with word embedding as input prompt to fine-tune a Llama2 [34] model.

#### Brain Predictive Coding.

Predictive coding theory [20, 29, 9] aims to propose a potential unifying
theory for computational and cognitive neuroscience [21]. It was initially
proposed as a neuroscientific theory [22] and subsequently developed into its
mathematical form of cortical responses [8]. Although originally formulated to
investigate brain visual processing, it was also extended to language
processing in human brain in previous work [10, 36]. Predictive coding
suggests that human brain naturally makes predictions about future words and
sentences when it perceives natural language stimuli. Such hypothesis has
already been evidenced by correlating word or phonetic surprisal with fMRI or
EEG [39, 24, 7, 12]. Caucheteux et al. [5] further proved a predictive coding
hierarchy in the human brain listening to speech by investigating the linear
mapping between modern language models and brain responses.

##  6 Discussion

In this presented study, we focus on brain predictive coding theory and neural
decoding. Specifically, we step further on the basis of current predictive
coding theory by investigating its effectiveness in decoding fMRI signals into
natural language. Predictive coding is first verified by linearly mapping the
activations of current auto-regressive language model to brain responses. Then
fMRI-to-text decoding model PredFT is proposed, which features utilizing a
side network to capture and fuse brain prediction. Comprehensive experiments
demonstrate the good decoding performance of PredFT benefits from integrating
brain predictive coding.

While lots of studies have confirmed the representations drawn from auto-
regressive language models are effective at predicting brain responses
stimulated by hearing spoken words, the reasons behind such phenomenon are
still controversial. A possible convincing explanation is both of the language
models and human brain follow a next words prediction pattern. But Antonello
and Huth [2] questioned this hypothesis and claimed the language models are
effective at predicting brain responses because they generally capture a wide
variety of linguistic phenomena. In this paper, we don’t seek to find a proper
explanation but try to investigate predictive coding from the perspective of
decoding. This finding offers a novel view for future research in predictive
coding.

Neural decoding is another popular research topic. Thanks to the development
of deep learning, decoding non-invasive signals is becoming possible. However,
despite the improvement in decoding performance in PredFT, we find it’s still
extremely difficult to reconstruct natural language from fMRI signals during
experiments. The reasons can be summarized as follows: First, the noise
inherited from collecting fMRI data is a natural barrier to decoding. Second,
different from fMRI-to-image decoding [38, 30] whose experimental setting is
letting subjects look at pictures one by one with certain intervals, the fast
spoken word rate isn’t compatible with the poor temporal resolution of fMRI
data in fMRI-to-text decoding. So part of the brain responses are not recorded
in fMRI. Such hypothesis has been evidenced through experiments in Appendix B.
Building a dataset with high-resolution devices (e.g. 7T MRI scanner) will
probably alleviate this problem.

The limitations of this work include: (i) Experiments are only conducted on
Narratives. It’s better to try other datasets like [15] or [26]. (ii) We apply
cross-subject dataset splitting for all experiments. It’s better to supplement
experiments on per-subject splitting to get more comprehensive comparisons.
(iii) The window length of future words is set as $l=2$ in experiments due to
the limited computational resources. We leave the exploration of future words
window length to future work. (iv) Surprising content might become distraction
in brain predictive coding. We leave it as future work to analyze.

## Acknowledgements

This research is supported by the National Natural Science Foundation of China
(No.62106105), the CCF-Baidu Open Fund (No.CCF-Baidu202307), the CCF-Zhipu AI
Large Model Fund (No.CCF-Zhipu202315), the Scientific Research Starting
Foundation of Nanjing University of Aeronautics and Astronautics
(No.YQR21022), and the High Performance Computing Platform of Nanjing
University of Aeronautics and Astronautics.

## References

  * Affolter et al. [2020] Nicolas Affolter, Beni Egressy, Damian Pascual, and Roger Wattenhofer.  Brain2word: decoding brain activity for language generation.  _arXiv preprint arXiv:2009.04765_ , 2020.
  * Antonello and Huth [2024] Richard Antonello and Alexander Huth.  Predictive coding or just feature discovery? an alternative account of why language models fit brain data.  _Neurobiology of Language_ , 5(1):64–79, 2024.
  * Bhattasali et al. [2019] Shohini Bhattasali, Murielle Fabre, Wen-Ming Luh, Hazem Al Saied, Mathieu Constant, Christophe Pallier, Jonathan R Brennan, R Nathan Spreng, and John Hale.  Localising memory retrieval and syntactic composition: an fmri study of naturalistic language comprehension.  _Language, Cognition and Neuroscience_ , 34(4):491–510, 2019.
  * Caucheteux et al. [2022] Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King.  Deep language algorithms predict semantic comprehension from brain activity.  _Scientific reports_ , 12(1):16327, 2022.
  * Caucheteux et al. [2023] Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King.  Evidence of a predictive coding hierarchy in the human brain listening to speech.  _Nature human behaviour_ , 7(3):430–441, 2023.
  * Destrieux et al. [2010] Christophe Destrieux, Bruce Fischl, Anders Dale, and Eric Halgren.  Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature.  _NeuroImage_ , 53(1):1–15, 2010.  ISSN 1053-8119.  doi: https://doi.org/10.1016/j.neuroimage.2010.06.010.  URL https://www.sciencedirect.com/science/article/pii/S1053811910008542.
  * Donhauser and Baillet [2020] Peter W Donhauser and Sylvain Baillet.  Two distinct neural timescales for predictive speech processing.  _Neuron_ , 105(2):385–393, 2020.
  * Friston [2008] Karl Friston.  Hierarchical models in the brain.  _PLoS computational biology_ , 4(11):e1000211, 2008.
  * Friston and Kiebel [2009] Karl Friston and Stefan Kiebel.  Predictive coding under the free-energy principle.  _Philosophical transactions of the Royal Society B: Biological sciences_ , 364(1521):1211–1221, 2009.
  * Garrido et al. [2009] Marta I Garrido, James M Kilner, Klaas E Stephan, and Karl J Friston.  The mismatch negativity: a review of underlying mechanisms.  _Clinical neurophysiology_ , 120(3):453–463, 2009.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Heilbron et al. [2022] Micha Heilbron, Kristijan Armeni, Jan-Mathijs Schoffelen, Peter Hagoort, and Floris P De Lange.  A hierarchy of linguistic predictions during natural language comprehension.  _Proceedings of the National Academy of Sciences_ , 119(32):e2201968119, 2022.
  * Huth et al. [2016] Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant.  Natural speech reveals the semantic maps that tile human cerebral cortex.  _Nature_ , 532(7600):453–458, 2016.
  * Ji et al. [2013] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu.  3d convolutional neural networks for human action recognition.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 35(1):221–231, 2013.  doi: 10.1109/TPAMI.2012.59.  URL https://doi.org/10.1109/TPAMI.2012.59.
  * LeBel et al. [2023] Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G Huth.  A natural language fmri dataset for voxelwise encoding models.  _Scientific Data_ , 10(1):555, 2023.
  * [16] Yann LeCun, Yoshua Bengio, et al.  Convolutional networks for images, speech, and time series.
  * Lewis et al. [2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.703.  URL https://aclanthology.org/2020.acl-main.703.
  * Liao et al. [2002] Chien Heng Liao, Keith J Worsley, J-B Poline, John AD Aston, Gary H Duncan, and Alan C Evans.  Estimating the delay of the fmri response.  _NeuroImage_ , 16(3):593–606, 2002.
  * Lin [2004] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81, 2004.
  * McClelland and Rumelhart [1981] James L McClelland and David E Rumelhart.  An interactive activation model of context effects in letter perception: I. an account of basic findings.  _Psychological review_ , 88(5):375, 1981.
  * Millidge et al. [2022] Beren Millidge, Anil Seth, and Christopher L Buckley.  Predictive coding: a theoretical and experimental review, 2022.
  * Mumford [1991] David Mumford.  On the computational architecture of the neocortex: I. the role of the thalamo-cortical loop.  _Biological cybernetics_ , 65(2):135–145, 1991.
  * Nastase et al. [2021] Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al.  The “narratives” fmri dataset for evaluating models of naturalistic language comprehension.  _Scientific data_ , 8(1):250, 2021.
  * Okada et al. [2018] Kayoko Okada, William Matchin, and Gregory Hickok.  Neural evidence for predictive coding in auditory cortex during speech production.  _Psychonomic bulletin & review_, 25:423–430, 2018.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * Pereira et al. [2018] Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko.  Toward a universal decoder of linguistic meaning from brain activation.  _Nature communications_ , 9(1):963, 2018.
  * Radford et al. [2018] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Rao and Ballard [1999] Rajesh PN Rao and Dana H Ballard.  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects.  _Nature neuroscience_ , 2(1):79–87, 1999.
  * Scotti et al. [2024] Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al.  Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Sun et al. [2019] Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong.  Towards sentence-level brain decoding with distributed representations.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):7047–7054, Jul. 2019.  doi: 10.1609/aaai.v33i01.33017047.  URL https://ojs.aaai.org/index.php/AAAI/article/view/4685.
  * Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to sequence learning with neural networks.  In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , pages 3104–3112, 2014.  URL https://proceedings.neurips.cc/paper/2014/hash/a14ac55a4f27472c5d894ec1c3c743d2-Abstract.html.
  * Tang et al. [2023] Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth.  Semantic reconstruction of continuous language from non-invasive brain recordings.  _Nature Neuroscience_ , 26(5):858–866, 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wacongne et al. [2011] Catherine Wacongne, Etienne Labyt, Virginie Van Wassenhove, Tristan Bekinschtein, Lionel Naccache, and Stanislas Dehaene.  Evidence for a hierarchy of predictions and prediction errors in human cortex.  _Proceedings of the National Academy of Sciences_ , 108(51):20754–20759, 2011.
  * Wang et al. [2020] Shaonan Wang, Jiajun Zhang, Haiyan Wang, Nan Lin, and Chengqing Zong.  Fine-grained neural decoding with distributed word representations.  _Information Sciences_ , 507:256–272, 2020.
  * Wang et al. [2024] Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang.  Mindbridge: A cross-subject brain decoding framework.  _arXiv preprint arXiv:2404.07850_ , 2024.
  * Willems et al. [2016] Roel M Willems, Stefan L Frank, Annabel D Nijhof, Peter Hagoort, and Antal Van den Bosch.  Prediction during natural language comprehension.  _Cerebral cortex_ , 26(6):2506–2516, 2016.
  * Wu and He [2018] Yuxin Wu and Kaiming He.  Group normalization.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 3–19, 2018.
  * Xi et al. [2023] Nuwa Xi, Sendong Zhao, Haochun Wang, Chi Liu, Bing Qin, and Ting Liu.  UniCoRN: Unified cognitive signal ReconstructioN bridging cognitive signals and human language.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13277–13291, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.741.  URL https://aclanthology.org/2023.acl-long.741.
  * Ye et al. [2023] Ziyi Ye, Qingyao Ai, Yiqun Liu, Min Zhang, Christina Lioma, and Tuukka Ruotsalo.  Language generation from human brain activities.  _arXiv preprint arXiv:2311.09889_ , 2023.
  * Yin et al. [2023] Congchi Yin, Qian Yu, Zhiwei Fang, Jie He, Changping Peng, Zhangang Lin, Jingping Shao, and Piji Li.  Data contamination issues in brain-to-text decoding.  _arXiv preprint arXiv:2312.10987_ , 2023.
  * Zou et al. [2021] Shuxian Zou, Shaonan Wang, Jiajun Zhang, and Chengqing Zong.  Towards brain-to-text generation: Neural decoding with pre-trained encoder-decoder models.  In _NeurIPS 2021 AI for Science Workshop_ , 2021.

##  Appendix A Experimental Settings

#### Dataset and Splitting Method

All experiments are conducted on the largest naturalistic language
comprehension fMRI dataset Narratives [23], which contains recordings from 345
subjects listening to 27 diverse stories. Since the data collection process
involves different machines, only fMRI data with $64\times 64\times 27$ voxels
is considered, which results to 230 subjects. For the dataset preprocess
method, we choose the fMRIPrep version. Xi et al. [41] also applied fMRIPrep
while Ye et al. [42] applied the AFNI-smooth version.

How to split datasets for training and evaluation is a matter of debate in
fMRI-to-text decoding [41]. Ye et al. [42] and Tang et al. [33] trained and
evaluated models per subject. Xi et al. [41] applied cross-subject splitting
but has been identified to have data leakage issue [43]. To avoid data leakage
and test model’s cross-subject generalization ability, we apply the splitting
method proposed in [43].

#### Implementation Details

The latest version of Destrieux atlas [6] is applied for cortical
parcellation, which leads to 74 regions per hemisphere. We use six regions of
interests that have been proven in [5] to contribute to brain prediction,
including superior temporal sulcus, angular gyrus, supramarginal gyrus, and
opercular, triangular, orbital part of the inferior frontal gyrus.

Voxel-level normalization is performed to raw fMRI data, which separately
normalizes the values of each voxel over the time domain.This normalization
highlights the relative activation of a specific voxel within given intervals.
In the main decoding network, the numbers of Transformer encoders and decoders
are set to $P{=4}$ and $Q=12$ respectively. For the side network, both are set
as $M{=N{=6}}$. We apply the tokenizer in BART[17] for our model. PredFT is
trained over 20 epochs with an initial learning rate of 5e-4 which eventually
decays to 1e-5. The hyper-parameter $\lambda$ for jointly training the main
and side networks is set to 1 for fMRI sequence of length 10, and 0.5 for
sequence of length 20 and 40. For baseline methods we strictly follow the
settings in the proposed paper [41]. All experiments are conducted on NVIDIA
A100-80G GPUs.

##  Appendix B Decoding Error Analysis

In this section, we design an experiment that analyzes the positional
distribution of incorrectly decoded words. Figure 5 is an example of two
successive fMRI frames containing seven and four spoken words respectively.
Three kinds of errors during decoding are defined: (i) The decoding model
$\mathcal{M}$ fails to generate the correct word (e.g. “could” is incorrectly
decoded as “should”). (ii) The decoding model $\mathcal{M}$ generates
redundant words (e.g. repetitive “and”). (iii) Corresponding words are missing
during decoding (e.g. “sharp metal ripping” is missing in output). Although
some of the generated words are semantically consistent with the ground truth,
we apply the strict exact match to facilitate automatic evaluation. Three
position counting methods for corresponding errors are proposed: (i) If
decoded word is wrong, position index of the corresponding truth word is
marked as wrong. (ii) If decoded words are redundant, position index of the
last matched truth word is marked as wrong. (iii) If decoded words are
missing, position indices of all the missing truth words are marked as wrong.
Since different fMRI frames contain different numbers of spoken words, the
relative positions of incorrect words within each frame, namely the percentage
of index (PosPCT), are considered. The error probability of one specific
position is the proportion of errors at this position to the total number of
errors. Table (a) in Figure 5 illustrates the positional distribution of
incorrectly decoded words in the example. The distribution is calculated at
ten percentiles from $10\%$ to $100\%$. As some positions like $10\%$ or
$90\%$ are minority in all statistical positions, we also add the error
probabilities of the first and last $50\%$ respectively, as shown in table
(b).

Figure 5: An example of the experiment for decoding error analysis. PosID and
PosPCT stand for the word position index of truth and the percentage of index
respectively.

Such experiment is conducted on UniCoRN and PredFT. Positional decoding error
distribution and sum of error probability are analyzed. Results are shown in
Figure 6. We find the error probability of last heard words in TR is
significantly higher than words heard at the beginning. However, the error
probabilities of decoding the first and last half of text are supposed to be
the same in normal cases. This phenomenon leads to the hypothesis that the
information of some heard words, especially the last few words in each TR, is
lost in fMRI data. It’s caused by the discrete sampling feature of fMRI: Due
to the constraints of MRI scanner in strength and speed of switching the
magnetic gradients, the fMRI signal is sampled discretely with a fixed time
interval called repetition time (TR) in order to achieve the balance between
spacial and temporal resolution. The repetition time in fMRI-to-text decoding
task is usually around two seconds. However, the average speaking rate of
human is about three words per second. If pauses between sentences are
excluded, the word rate within one sentence will increase to five per second.
This feature of fMRI leads to the information loss problem in fMRI-to-text
decoding: While brain responses of the first few heard words are recorded in
one fMRI frame, information of the last heard words is lost due to the low
temporal resolution, making decoding these words difficult. The latency of
BOLD signal complicates the theoretical explanation of this phenomenon. But
based on experimental results and previous study [18] which indicates the
latency of fMRI response is about six seconds, exactly an integer multiple of
repetition time (1.5s or 2s), the hypothesis of information loss is
reasonable.

From sub-figure (a), (b), (c) in Figure 6, we surprisingly find PredFT
successfully reduces the error probability of the last few decoded words
compared to UniCoRN. This implies the predictive coding information in brain
could be utilized to alleviate the information loss, and such alleviation of
information loss is closely related to the decoding accuracy. To better
illustrate the degree of information loss, we propose a novel index
information loss slope $\varphi$ measuring the growth rate of error
probability from the first half of decoded content to the last half,

$\varphi=\frac{\sum_{i=6}^{10}p_{i}-\sum_{i=1}^{5}p_{i}}{0.5},$ (7)

where $p_{i}$ stands for the error probability of $i\%$ position. $\varphi$ is
expected to be around zero, as the error probabilities of different positions
are supposed to be the same without information loss. However, the $\varphi$
values of all compared models are high, indicating that all models suffer from
information loss. PredFT successfully mitigates information loss to some
extent. As shown in sub-figure (d), (e), (f) of Figure 6, the $\varphi$ score
of PredFT is lower than compared models on all the three experiments with
different fMRI sequence lengths.

Figure 6: Information loss of different models under different fMRI sequence
lengths.

##  Appendix C Ablation Study

Three aspects of ablation experiments are conducted to analyze PredFT. First
we test whether the side network for brain prediction really improves decoding
accuracy. The model PredFT without side network (PredFT w/o SideNet) is built
with the same settings as PredFT during training except for only keeping the
main decoding network (the cross-attention layers in main decoding network are
removed). The results of this model’s decoding performance are listed in Table
2, same as not using ROIs in side network (“None” in the table). The
performance of PredFT w/o SideNet is significantly worse than PredFT in all
the three experiments with different fMRI sequence length. It also performs
worse than UniCoRN under most cases, which might be attributed to the
pretrained language model used in UniCoRN.

Besides, the decoding error distribution of PredFT w/o SideNet is counted to
verify whether the side network helps alleviate information loss. As shown in
Figure 6, the error distribution of PredFT w/o SideNet across different
positions is similar to that of UniCoRN. The probability of decoding error
increases as the word position moves backward within one fMRI frame, peaking
at the position of the last word. PredFT w/o SideNet severely suffers from
information loss as shown in sub-figure (d), (e), (f) of Figure 6, with the
highest information loss slope. The ablation experiments provide solid
evidence on the effectiveness of the side network in PredFT.

We also test the influence of hyper-parameter $\lambda$ to decoding
performance of PredFT. As shown in Table 4, four different $\lambda$ values
ranging from 0.25 to 1 are tested in experiments with different fMRI sequence
length. Empirically, PredFT achieves relatively good decoding accuracy with
$\lambda=0.5$ in all experiments. For sequence length of 10, PredFT performs
best when $\lambda$ is set as 1.

Table 4: Decoding performance of PredFT under different hyper-parameter $\lambda$. Length | $\lambda$ | BLEU-N ($\%$) | ROUGE-1 ($\%$)
---|---|---|---
$N=1$ | $N=2$ | $N=3$ | $N=4$ | $R$ | $P$ | $F$
10 | 1 | 24.73 | 8.39 | 3.92 | 1.86 | 14.07 | 35.28 | 19.53
0.75 | 22.32 | 4.44 | 0.87 | 0.12 | 16.57 | 19.77 | 17.96
0.5 | 22.56 | 4.59 | 1.26 | 0.41 | 15.84 | 19.54 | 17.44
0.25 | 21.13 | 5.21 | 1.26 | 0.35 | 14.00 | 26.67 | 18.25
20 | 1 | 18.33 | 5.00 | 1.37 | 0.48 | 15.60 | 31.97 | 20.90
0.75 | 21.15 | 4.71 | 1.22 | 0.44 | 20.58 | 27.13 | 23.35
0.5 | 25.98 | 5.61 | 1.36 | 0.21 | 19.61 | 25.43 | 22.09
0.25 | 25.21 | 5.59 | 1.35 | 0.24 | 20.46 | 26.24 | 22.95
40 | 1 | 20.56 | 5.20 | 1.24 | 0.26 | 21.92 | 28.74 | 24.82
0.75 | 26.73 | 7.13 | 1.55 | 0.49 | 19.21 | 31.17 | 23.72
0.5 | 27.80 | 8.29 | 2.00 | 0.54 | 19.53 | 38.95 | 25.96
0.25 | 20.28 | 4.73 | 0.84 | 0.21 | 22.12 | 28.40 | 24.82
  *[fMRI]: functional magnetic resonance imaging