# Data Contamination Issues in Brain-to-Text Decoding

Congchi Yin1,2∗, Qian Yu3, Zhiwei Fang3, Jie He3, Changping Peng3
Zhangang Lin3, Jingping Shao3, Piji Li1,2†
1 College of Computer Science and Technology,
Nanjing University of Aeronautics and Astronautics, China
2 MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing,
China
3 JD.com, Beijing, China
<EMAIL_ADDRESS>
###### Abstract

Decoding non-invasive cognitive signals to natural language has long been the
goal of building practical brain-computer interfaces (BCIs). Recent major
milestones have successfully decoded cognitive signals like functional
Magnetic Resonance Imaging (fMRI) and electroencephalogram (EEG) into text
under open vocabulary setting. However, how to split the datasets for
training, validating, and testing in brain-to-text decoding still remains
controversial. In this paper, we conduct systematic analysis on current
dataset splitting methods and find data contamination largely exaggerates
model performance. Specifically, first we find the leakage of test subjects’
cognitive signals corrupts the training of a robust encoder. Second, we prove
the leakage of text stimuli causes the auto-regressive decoder to memorize
seen information in test set. To eliminate the influence of data contamination
and fairly evaluate different models’ generalization ability, we propose a new
splitting method for different types of cognitive dataset (e.g. fMRI, EEG). We
also test the performance of SOTA brain-to-text decoding models under the
proposed dataset splitting paradigm as baselines for further research.

11footnotetext: Work was done when the author was an intern in
JD.com.22footnotetext: Corresponding author.

##  1 Introduction

Brain-computer interface (BCI) builds connections between human brain and
external devices (e.g. computer). It has been widely researched in the field
of neuroscience and has gained remarkable success like repairing damaged sight
or restoring movement of disabled people (Polikov et al., 2005; Hochberg et
al., 2012; Bouton et al., 2016). However, it’s still challenging in decoding
cognitive signals to corresponding natural language when subjects read or hear
text stimuli, especially for non-invasive cognitive signals like functional
Magnetic Resonance Imaging (fMRI) or electroencephalogram (EEG) that are noisy
and of low resolution (Mridha et al., 2021).

Figure 1: General frameworks of brain-to-text decoding and possible situations
of data contamination.

Recent methods (Makin et al., 2020; Wang and Ji, 2022; Xi et al., 2023; Tang
et al., 2023) typically viewed brain-to-text decoding as machine translation
(Sutskever et al., 2014; Bahdanau et al., 2015) and adopted an encoder-decoder
framework, where the encoder is responsible for converting cognitive signals
into low-dimensional representations and the decoder learns to map the
representations to natural language. Specifically, as shown in Figure1, the
encoder usually consists of a spatial and time series feature extractor. It
can be trained either in an end-to-end manner with decoder (Figure 1 a) or
first pre-trained through a signal reconstruction task and then applied in
decoder training (Figure 1 b). Despite recent success in model design, it
still remains controversial in how to split the dataset for training,
validating, and testing (Xi et al., 2023). Addressing this issue is urgent and
meaningful, as fair evaluation of models is impossible without a widely
recognized dataset splitting paradigm.

A cognitive dataset is usually formatted in signal-sentence pair. In most
cases for brain-to-text decoding task, each sentence belongs to a certain
task, so signal-sentence pair can be further divided into signal-task and
task-sentence pair. Current dataset splitting methods (Wang and Ji, 2022; Xi
et al., 2023) can be summarized into five categories: (1) split by subjects,
(2) split by tasks, (3) split by randomly picking signal frames, (4) split by
randomly picking signal frames under certain task, (5) split by randomly
picking consecutive signal frames under certain task. However, all these
methods suffer from data contamination on encoder side, decoder side, or both.
As shown in Figure 1, to encoder if subjects’ cognitive signals in test set
are mixed into training set, encoder will become overfitted and fail to well
represent unseen subjects’ cognitive signals. As to decoder, situation gets
worse if text stimuli are leaked. Since the decoder generates token by token
in an auto-regressive manner, data contamination will cause the decoder to
memorize seen paragraphs and probability distribution, which means given the
first few tokens the decoder is able to predict next token regardless of
encoded cognitive signal representations.

To address the above-mentioned problems, we propose a new dataset splitting
method that eradicates data contamination from both encoder and decoder sides.
We focus on fMRI and EEG signals in experiments, although the proposed
splitting method could be applied to any cognitive signals satisfying the
given format. In our method, the dataset is split according to subject-stimuli
pairs with the following rules: (1) Cognitive signals collected from subjects
in validation set and test set won’t appear in training set, which means the
trained encoder can’t get access to any brain information belonging to
subjects in test set. (2) Text stimuli in validation set and test set won’t
appear in training set. The decoder learns the mapping between cognitive
signal representation and token embedding instead of memorizing seen text.

Our contributions can be summarized as follows:

  * •

We investigate current dataset splitting methods and analyze their influence
on popular frameworks in brain-to-text decoding.

  * •

We prove the existence of data contamination in current dataset splitting
methods through analysis and experiments, which seriously exaggerates model
performance.

  * •

We propose the first splitting method without data contamination on public
cognitive datasets. We also release a fair benchmark to evaluate different
models’ generalization performance for further research in this domain.

Figure 2: Different cognitive dataset splitting methods.

##  2 Related Work

Cognitive Signal Cognitive signals can be classified into three categories:
invasive, partially invasive, and non-invasive according to how close
electrodes get to brain tissue. Due to the high cost and complexity of
invasive and partially invasive methods, it’s hard to apply them in building
generic and practical BCIs. In this paper, we mainly focus on non-invasive
signals EEG and fMRI. EEG signal is electrogram of the spontaneous electrical
activity of the brain. Its frequencies usually range from 1 to 30 Hz, divided
into several groups like alpha (4-13 Hz), beta (13-30 Hz), delta (0.5-4 Hz),
theta (4-7 Hz). EEG is of high temporal resolution and relatively tolerant of
subject movement, but its spatial resolution is low and it can’t display
active areas of the brain directly. fMRI measures brain activity by detecting
changes of blood flow. Blood flow of a specific region increases when this
brain area is in use. The spatial resolution of fMRI is measured by the size
of voxel, which is a three-dimensional rectangular cuboid ranging from 3mm to
5mm (Vouloumanos et al., 2001; Noppeney and Price, 2004). Unlike EEG which
samples brain signals continuously, fMRI samples according to the smallest
time period named TR, usually at second level.

Brain-to-text Decoding Previous research on brain-to-text decoding (Herff et
al., 2015; Anumanchipalli et al., 2019; Zou et al., 2021; Moses et al., 2021;
Défossez et al., 2023) mainly focused on word-level decoding in a restricted
vocabulary with hundreds of words (Panachakel and Ramakrishnan, 2021). These
models typically apply recurrent neural network or long short-term memory
(Hochreiter and Schmidhuber, 1997) network to build mapping between cognitive
signals and words in vocabulary. Despite relatively good accuracy, these
methods fail to generalize to unseen words. Some progress (Sun et al., 2019)
has been made by expanding word-level decoding to sentence-level through
encoder-decoder framework, or use less noisy ECoG data (Burle et al., 2015;
Anumanchipalli et al., 2019). However, these models struggle to generate
accurate and fluent sentences limited by decoder ability. Wang and Ji (2022)
introduced the first open vocabulary EEG-to-text decoding model by leveraging
the power of pre-trained language models. Xi et al. (2023) improved the model
design and proposed a unified framework for decoding both fMRI and EEG
signals.

##  3 Methodology

In this section, we will first introduce the definition of decoding cognitive
signals to text and the general description of dataset format. Then we
systematically analyze current dataset splitting methods and point out that
all existing methods suffer from two kinds of data contamination: cognitive
signal leakage and text stimuli leakage. Finally, a new dataset splitting
method is proposed to avoid the above-mentioned two kinds of data
contamination.

###  3.1 Task Definition

Given the cognitive signal $F_{ij}$ stimulated by $i$-th subject $S_{i}$
hearing or reading certain text $T_{j}$, brain-to-text decoding aims to decode
$F_{ij}$ back to text $T_{j}^{\prime}$ and make $T_{j}^{\prime}$ as similar as
possible to $T_{j}$. The composition of $F_{ij}$ and $T_{j}$ is different as
to fMRI and EEG. The former samples brain information discretely with a fixed
time interval TR, while the latter samples continuously. To fMRI, $L$
consistent sentence segments $s_{j}$ with corresponding fMRI frames $f_{ij}$
are concatenated to form a sample pair $\left\langle
F_{ij},T_{j}\right\rangle$. Here
$T_{j}=concat(s_{j},s_{j+1},\ldots,s_{j+L-1})$ and
$F_{ij}=concat(f_{ij},f_{i,j+1},\ldots,f_{i,j+L-1})$. To EEG, since signals
corresponding to a complete sentence are available, we bond sentence (aka text
stimuli) $T_{j}$ and EEG signal $F_{ij}$ together to form a sample pair
$\left\langle F_{ij},T_{j}\right\rangle$. Under most circumstances, each
sentence $T_{j}$ belongs to one certain task $M_{k}$. So the signal-sentence
pair $\left\langle F_{ij},T_{j}\right\rangle$ can be further split into
$\left\langle F_{ij},M_{k}\right\rangle$ and $\left\langle
M_{k},T_{kj}\right\rangle$.

Figure 3: The process of our proposed dataset splitting method.

In brain-to-text decoding, the ultimate goal of trained BCI models is to
generalize to unseen subjects with unseen text stimuli (Huang et al., 2010;
Handiru and Prasad, 2016; Gao et al., 2021). As a result, if cognitive signal
$F_{ij}$ appears in test set $S_{test}$, then any subject $i$’s signal $F_{i}$
should not appear in training set $S_{train}$. Similarly, text stimuli
$T_{kj}$ in $S_{test}$ should not appear in $S_{train}$. So the dataset
splitting rules can be formally defined as:

$S_{train}=\\{\left\langle F_{ij},T_{kj}\right\rangle|F_{i},T_{kj}\notin
S_{val},S_{test}\\}$ (1) $S_{val}=\\{\left\langle
F_{ij},T_{kj}\right\rangle|F_{i},T_{kj}\notin S_{train},S_{test}\\}$ (2)
$S_{test}=\\{\left\langle F_{ij},T_{kj}\right\rangle|F_{i},T_{kj}\notin
S_{train},S_{val}\\}$ (3)

###  3.2 Dataset Splitting Methods

Current dataset splitting methods can be summarized as five categories
according to classifying objectives $S_{i},M_{k},T_{kj},F_{ij}$. More
specifically, the five dataset splitting methods are characterised as (1)
split by subjects, (2) split by tasks, (3) split by randomly picking signal
frames, (4) split by randomly picking signal frames under certain task, (5)
split by randomly picking consecutive signal frames under certain task,
corresponding to image (a), (b), (c), (d), (e) in Figure 2. Figure 2 vividly
displays the differences between current dataset splitting methods. For
simplicity of expression, we choose four subjects with three tasks each
containing four, three, four sentences. The line connecting two symbols
indicates they are related to one sample. Take path
$S_{1},M_{1},T_{11},F_{11}$ for example, it’s one sample where subject $S_{1}$
listens to text stimuli $T_{11}$ belonging to task $M_{1}$ and $S_{1}$’s
corresponding brain signal is recorded as $F_{11}$. Some symbols are connected
with several lines. For example, the four lines between $S_{1}$ and $M_{1}$
correspond to $\left\langle M_{1},T_{11}\right\rangle$, $\left\langle
M_{1},T_{12}\right\rangle$, $\left\langle M_{1},T_{13}\right\rangle$,
$\left\langle M_{1},T_{14}\right\rangle$ counting from left to right.
Similarly, the three lines between $M_{1}$ and $T_{11}$ correspond to
$\left\langle S_{1},M_{1}\right\rangle$, $\left\langle
S_{2},M_{1}\right\rangle$, $\left\langle S_{3},M_{1}\right\rangle$
respectively. The same rules can be extended to other lines and symbols. The
green lines and orange lines stand for training samples and testing samples.
The grey dotted line means the sample is abandoned, which will be introduced
in our dataset splitting method. As the splitting of validation set is similar
to test set, we only consider training set and test set in this section.

We will use method (a), (b), (c), (d), (e) to represent five current dataset
splitting methods in the rest of the paper. Method (a) splits the dataset
according to subjects, which can be described as

$S_{train}=\\{\left\langle F_{ij},T_{kj}\right\rangle|S_{i}\notin
S_{val},S_{test}\\}$ (4)

for training set. Method (b) splits the dataset according to tasks, which can
be described as

$S_{train}=\\{\left\langle F_{ij},T_{kj}\right\rangle|M_{k}\notin
S_{val},S_{test}\\}$ (5)

for training set. Method (c), (d), and (e) all split the dataset according to
cognitive signal frames

$S_{train}=\\{\left\langle F_{ij},T_{kj}\right\rangle|F_{ij}\notin
S_{val},S_{test}\\}.$ (6)

However, there are slight differences between these methods. Method (c) views
all the cognitive signal frames in dataset as a whole and splits according to
the default proportion (e.g. 8:1:1). Method (d) views signal frames under
certain task $M_{k}$ as a whole and splits proportionally, and then union all
training sets under different tasks to form a complete set for training.
Method (e) is similar to method (d). They both first split training,
validation, and test set under certain task proportionally and then union
them. The difference lies in that method (d) randomly picks signal frames
while method (e) picks consecutive signal frames.

Type | Method | Narratives / ZuCo | Average
---|---|---|---
seed$1$ | seed$2$ | seed$3$ | seed$4$
CSLR$(\%)$ | (a) |  0.00 / 0.00 |  0.00 / 0.00 |  0.00 / 0.00 |  0.00 / 0.00 |  0.00 / 0.00
(b) | 6.73 / - | 6.32 / - | 7.7 / - | 17.93 / - | 9.67 / -
(c) | 12.55 / 12.52 | 12.52 / 12.55 | 12.48 / 12.48 | 12.44 /12.46 | 12.50 / 12.50
(d) | 12.81 / 12.60 | 12.8 / 12.58 | 12.78 / 12.56 | 12.79 / 12.61 | 12.795 / 12.59
(e) | 12.28 / - | 12.27 / - | 12.26 / - | 12.27 / - | 12.27 / -
(f) |  0.00 / 0.00 |  0.00 / 0.00 |  0.00 / 0.00 |  0.00 / 0.00 |  0.00 / 0.00
TSLR$(\%)$ | (a) | 100.00 / 23.43 | 100.00 / 20.25 | 100.00 / 23.38 | 100.00 / 22.95 | 100.00 / 22.50
(b) |  0.00 / - |  0.00 / - |  0.00 / - |  0.00 / - |  0.00 / -
(c) | 100.00 / 13.21 | 100.00 / 13.06 | 100.00 / 12.91 | 100.00 / 13.1 | 100.00 / 13.07
(d) | 99.93 / 0.00 | 99.81 / 0.00 | 99.54 / 0.00 | 99.99 / 0.00 | 99.82 / 0.00
(e) | 9.19 / - | 9.31 / - | 9.36 / - | 9.29 / - | 9.29 / -
(f) |  0.00 / 0.00 |  0.00 / 0.00 |  0.00 / 0.00 |  0.00 / 0.00 |  0.00 / 0.00

Table 1: Results of Cognitive Signal Leakage Rate (CSLR) and Text Stimuli
Leakage Rate (TSLR).

The goal of brain-to-text decoding models is to generalize to unseen subjects
with unseen text stimuli, which means both subject’s brain information and
received text stimuli are new to the trained model. In this sense, we define
two kinds of data contamination: cognitive signal leakage and text stimuli
leakage. The data contamination situation of different methods is reflected in
Figure 2. If lines associated with $S_{i}$ or $T_{kj}$ are of different
colours, data in test set leaks into training set. Lines between $S_{i}$ and
$M_{k}$ indicate cognitive signal leakage situation and lines between $T_{kj}$
and $M_{k}$ indicate text stimuli leakage situation. Remind the composition of
samples differs as to fMRI signal and EEG signal, so the dataset splitting
methods are different for two cognitive signals too. Since fMRI signals need
to be sampled continuously with a certain length $L$, the path of a sample
shown in Figure 2 is actually the first part of one fMRI sample, with $L-1$
continuous part following. In this sense, for EEG cognitive signal leakage
doesn’t exist in method (a), but method (a) suffers from text stimuli leakage.
The situation of method (b) is opposite to that of method (a), where there’s
no text stimuli leakage but cognitive signal leakage. Method (c) and method
(d) are similar. They suffer from both cognitive signal leakage and text
stimuli leakage. Method (e) is in the same situation as method (b). For fMRI,
method (c), (d), and (e) which seem the same for EEG are actually different
splitting ways. The situation of data contamination for different methods is
similar to EEG, except for method (e) there still exists slight text stimuli
leakage in the overlap between training samples and test samples.

###  3.3 Our Method

To eliminate data contamination from both cognitive signal leakage and text
stimuli leakage, we split the dataset by $\left\langle
S_{i},T_{j}\right\rangle$ pairs as shown in (f) of Figure 2. Since EEG and
fMRI are different in the composition of dataset, we treat them separately and
propose two dataset splitting methods. As to EEG dataset where $F_{ij}$ and
$T_{j}$ form a sample, we consider a bipartite graph
$\mathcal{G}_{1}=(\mathcal{U},\mathcal{V},\mathcal{E})$ where
$\mathcal{U}=\\{S_{i}\\}_{i=1}^{N}$, $\mathcal{V}=\\{T_{j}\\}_{j=1}^{M}$.
$\mathcal{E}$ is the edge between node in $\mathcal{U}$ and node in
$\mathcal{V}$, indicating $\left\langle S_{i},T_{j}\right\rangle$ pair in the
dataset. $N$ is the total number of subjects and $M$ is the total number of
unique text stimuli. We assert $M>N$, so $e=(u,v)\in\mathcal{E}$ exists for
every $v\in\mathcal{V}$, as each text stimuli is listened or read by at least
one subject. As shown in step 2 of Figure 3, first we pick one edge for each
node $v\in\mathcal{V}$ and build a new bipartite graph
$\mathcal{G}_{2}=(\mathcal{U},\mathcal{V},\mathcal{E}^{\prime})$. Then we
split graph $\mathcal{G}_{2}$ by subject $\mathcal{U}$ with the given
splitting ratio and form three disjoint graphs
$\mathcal{G}_{train},\mathcal{G}_{val},\mathcal{G}_{test}$. In step 4, some
edges satisfying zero data contamination condition are not included in the
graph. We add these edges to corresponding graphs, extending each graph
$\mathcal{G}_{train},\mathcal{G}_{val},\mathcal{G}_{test}$ to its maximally
scalable state and finishing the dataset splitting process.

$F_{ij}=concat(f_{ij},f_{i,j+1},\ldots,f_{i,j+L-1})$ and
$T_{j}=concat(s_{j},s_{j+1},\ldots,s_{j+L-1})$ form a sample pair in fMRI
dataset. If we follow the same process as EEG, text stimuli leakage will occur
in the overlapping part of two samples, when one sample is assigned to
training set and the other is assigned to validation or test set. We propose a
simple solution that achieves the balance between abandoning as little data as
possible and ensuring zero data contamination. Instead of $\left\langle
S_{i},T_{j}\right\rangle$ pair, we consider $\left\langle
S_{i},M_{k}\right\rangle$ pair and apply the above-mentioned algorithm. More
details and pseudo-code are available in Appendix B.

Model | Epoch+lr+Method | BLEU-N ($\%$) | ROUGE-1 ($\%$)
---|---|---|---
$N=1$ | $N=2$ | $N=3$ | $N=4$ | $F$ | $P$ | $R$
UniCoRN | 10+1e-3+(a) | 49.56 | 30.49 | 21.07 | 15.49 | 44.83 | 50.41 | 40.65
10+1e-3+(b) | 26.37 | 7.50 | 2.48 | 0.99 | 22.28 | 25.99 | 19.62
10+1e-3+(c) | 50.24 | 30.83 | 21.23 | 15.60 | 44.68 | 49.44 | 41.01
10+1e-3+(d) | 49.63 | 30.29 | 20.85 | 15.32 | 45.06 | 50.47 | 41.03
10+1e-3+(e) | 28.94 | 9.39 | 4.07 | 1.53 | 21.68 | 24.64 | 19.49
UniCoRN∗ | 20+1e-4+(a) | 50.19 | 34.25 | 25.98 | 21.00 | 46.59 | 50.36 | 43.62
30+1e-4+(a) | 55.46 | 40.99 | 32.85 | 27.56 | 52.08 | 55.02 | 49.68
20+1e-4+(b) | 25.91 | 8.80 | 3.84 | 1.66 | 20.65 | 27.74 | 16.57
30+1e-4+(b) | 25.91 | 8.80 | 3.84 | 1.66 | 20.65 | 27.74 | 16.57
20+1e-4+(c) | 72.44 | 60.84 | 53.35 | 47.88 | 70.52 | 74.10 | 67.53
30+1e-4+(c) | 72.82 | 61.42 | 53.95 | 48.44 | 71.24 | 74.41 | 68.57
20+1e-4+(d) | 65.31 | 51.02 | 42.54 | 36.72 | 62.76 | 67.09 | 59.29
30+1e-4+(d) | 66.56 | 53.00 | 44.75 | 39.02 | 63.89 | 67.51 | 60.95
20+1e-4+(e) | 32.15 | 12.34 | 5.57 | 2.45 | 24.28 | 30.43 | 20.35
30+1e-4+(e) | 32.15 | 12.34 | 5.57 | 2.45 | 24.28 | 30.43 | 20.35
Table 2: Generation quality of UniCoRN model for fMRI under different training
settings. Here UniCoRN∗ indicates the encoder of UniCoRN is randomly
initialized instead of pre-trained through signal reconstruction task.

##  4 Experiments and Analysis

We test state-of-the-art brain-to-text decoding models on two popular
cognitive datasets. Comprehensive experiments are conducted to prove the
existence of the following phenomena: (1) Cognitive signals and text stimuli
in test set leak into training set in all current dataset splitting methods.
(2) The model’s generalization ability, particularly that of the auto-
regressive decoder, has been overestimated due to data contamination. Because
the number of tasks in EEG dataset is too small and method (e) makes no
difference to EEG as method (d), we only consider method (a), (c), (d).

###  4.1 Datasets

We apply the "Narratives" (Nastase et al., 2021) dataset for fMRI-to-text
decoding and the ZuCo (Hollenstein et al., 2018) dataset for EEG-to-text
decoding in experiments. The "Narratives" dataset contains fMRI data from 345
subjects listening to 27 diverse stories. Since the data collection process
involves different machines, we only consider fMRI data with $64\times
64\times 27$ voxels. The ZuCo dataset includes 12 healthy adult native English
speakers reading English text for 4 to 6 hours. It contains simultaneous EEG
and Eye-tracking data. The reading tasks include Normal Reading (NR) and Task-
specific Reading (TSR) extracted from movie views and Wikipedia. Both datasets
are split into training, validation, and test set with a ratio of 80$\%$,
10$\%$, 10$\%$ in all experiments.

###  4.2 Implementation

We follow the same settings of UniCoRN (Xi et al., 2023) and EEG2Text (Wang
and Ji, 2022), except the datasets are split to the ratio of 8:1:1 for fair
comparison. All experiments are conducted on NVIDIA A100-SXM4-40GB GPUs. More
details are shown in Appendix A.

Model | Epoch+lr+Method | BLEU-N ($\%$) | ROUGE-1 ($\%$)
---|---|---|---
$N=1$ | $N=2$ | $N=3$ | $N=4$ | $F$ | $P$ | $R$
UniCoRN | 50+1e-4+(a) | 58.09 | 49.23 | 43.23 | 38.43 | 63.88 | 61.12 | 67.50
80+1e-4+(a) | 60.88 | 50.52 | 43.42 | 37.84 | 65.17 | 61.16 | 70.72
50+1e-4+(c) | 52.30 | 42.89 | 36.80 | 32.17 | 57.39 | 51.09 | 67.29
80+1e-4+(c) | 60.78 | 55.92 | 53.18 | 51.10 | 84.64 | 63.16 | 71.50
50+1e-4+(d) | 22.90 | 7.36 | 2.71 | 0.95 | 17.73 | 19.90 | 17.33
80+1e-4+(d) | 22.90 | 7.36 | 2.71 | 0.95 | 17.73 | 19.90 | 17.33
EEG2Text | 50+1e-4+(a) | 51.22 | 33.83 | 22.99 | 16.05 | 46.40 | 46.85 | 46.58
80+1e-4+(a) | 63.32 | 52.52 | 45.19 | 39.50 | 65.96 | 64.74 | 68.01
50+1e-4+(c) | 53.83 | 38.99 | 29.57 | 23.01 | 53.64 | 54.19 | 53.56
80+1e-4+(c) | 65.42 | 57.56 | 52.56 | 48.60 | 73.00 | 69.99 | 77.01
50+1e-4+(d) | 23.92 | 8.16 | 3.21 | 1.20 | 20.78 | 19.96 | 23.89
80+1e-4+(d) | 23.92 | 8.16 | 3.21 | 1.20 | 20.78 | 19.96 | 23.89
Table 3: Generation quality of UniCoRN and EEG2Text model for EEG under
different training settings.

###  4.3 Data Contamination

We have analyzed two kinds of data contamination, cognitive signal leakage and
text stimuli leakage in Methodology section. In this part, we will quantify
data contamination situation through experiments.

To better illustrate the extent of data contamination across different dataset
splitting methods, we design two novel evaluation metrics named Cognitive
Signal Leakage Rate (CSLR) and Text Stimuli Leakage Rate (TSLR) for detecting
cognitive signal leakage and text stimuli leakage. Note that the situation for
validation set is similar as test set, we only consider test set in
experiments. CSLR indicates the average percentage of each subject’s cognitive
signals in test set appearing in training set, which could be formulated as

$\frac{1}{N}\sum_{i=1}^{N}\min(1,\frac{|\\{F_{ij}|F_{ij}\in S_{test}\cap
S_{train}\\}|}{|\\{F_{ij}|F_{ij}\in S_{train}\\}|})$ (7)

where $N$ stands for the total number of subjects in test set. $|\cdot|$
stands for the cardinality of a set. Function $\min(\cdot,\cdot)$ is applied
to make sure for each subject the data leakage rate is less than 1.

The definition of TSLR is somewhat different for EEG signal and fMRI signal.
As to EEG signal where cognitive signals are sampled continuously, it’s easy
to match certain sentence stimuli with corresponding signals. Its TSLR is
similar to CSLR, which indicates the average percentage of certain text in
test set appearing in training set. TSLR for EEG data can be calculated
through

$\frac{1}{M}\sum_{j=1}^{M}\min(1,\frac{|\\{T_{ij}|T_{ij}\in S_{test}\cap
S_{train}\\}|}{|\\{T_{ij}|T_{ij}\in S_{train}\\}|})$ (8)

where $M$ stands for the total number of unique text periods in test set and
$T_{ij}$ stands for $j$-th period of text stimuli received by $i$-th subject.

Figure 4: Validation loss of encoder under different dataset splitting methods
in two datasets.

The fMRI signal is sampled discretely with a deterministic interval TR, making
it hard to acquire signals corresponding to sentences. Previous methods
instead concatenated continuous fMRI frames of certain length with their
corresponding sentence segments as training samples. As a result, we consider
the average percentage of the same sentence segments in test set appearing in
training set as TSLR of fMRI signal. It can be formulated as

$\frac{1}{M}\sum_{j=1}^{M}\tau\frac{|\\{T_{ij}|T_{ij}\in S_{test}\cap
S_{train}\\}|}{|S_{test}|\times L}$ (9)

where $\tau=0$ if $\\{T_{ij}|T_{ij}\in S_{test}\cap S_{train}\\}=\emptyset$
else

$\tau=\min(1,\frac{|\\{T_{ij}|T_{ij}\in S_{train}\\}|}{|\\{T_{ij}|T_{ij}\in
S_{test}\cap S_{train}\\}|}).$ (10)

We test current dataset splitting methods and our method on fMRI dataset
Narratives and EEG dataset ZuCo. Considering the influence of randomness in
splitting, we select four seeds for experiments. The results are shown in
Table 1 and are consistent with theoretical analysis. For fMRI, current
methods apart from method (a) suffer from cognitive signal leakage, while
method (a) has serious text stimuli leakage. Method (b) gets no text stimuli
leakage but has slight cognitive signal leakage. The situation for EEG is
similar to that of fMRI. Apart from our proposed method (f), there is no way
to achieve zero cognitive signal leakage and text stimuli leakage at the same
time.

Dataset | Model | BLEU-N ($\%$) | ROUGE-1 ($\%$)
---|---|---|---
$N=1$ | $N=2$ | $N=3$ | $N=4$ | $F$ | $P$ | $R$
Narratives | UniCoRN | 22.83 | 5.69 | 1.43 | 0.48 | 15.55 | 24.80 | 19.04
ZuCo | UniCoRN | 23.32 | 7.78 | 3.01 | 1.09 | 18.47 | 20.00 | 17.92
EEG2Text | 24.49 | 7.49 | 2.28 | 0.62 | 23.98 | 23.95 | 25.74
Table 4: A fair benchmark for evaluating brain-to-text decoding.

###  4.4 Overestimated Performance

Cognitive signal leakage and text stimuli leakage affect models from encoder
side and decoder side.

####  4.4.1 Encoder

As shown in Figure 1, encoder in current models is trained in two different
ways: either jointly trained with decoder or solely trained through a
reconstruction task. In the former end-to-end training scenario, it’s hard to
evaluate encoder performance separately. So we mainly focus on the latter, in
which case the encoder is trained through an encoder-decoder framework to
reconstruct input cognitive signals. The decoder here doesn’t refer to the
decoder for text generation. It’s similar to the structure of the encoder and
will be abandoned once the encoder is trained. Although a proper evaluation
index of the encoder’s representation ability is missing, validation loss is
used to measure the influence of data contamination.

We test different splitting methods on two cognitive datasets. The validation
loss of encoder is shown in Figure 4. For fMRI, influenced by leakage of
cognitive signals, the validation loss of method (b), (c), (d), (e) keeps
dropping even with long training epochs. The encoder is actually overfitting
and degrading. For method (a) and (f) without cognitive signal leakage, the
validation loss quickly rises after reaching the lowest point with a few
epochs, satisfying the basic rule of machine learning. For EEG, we find
validation loss keeps dropping for all methods even with very long training
epochs, regardless of cognitive signal leakage or not. We think the poor
spatial resolution of EEG signal might lead to this phenomenon.

####  4.4.2 Decoder

All state-of-the-art models choose pre-trained language model BART (Lewis et
al., 2020) as decoder. On one hand, the powerful auto-regressive decoder is
able to achieve fluent sentence-level open vocabulary text generation. On the
other hand, if data contamination occurs, due to the feature of auto-
regressive generation, the decoder will generate memorized text given the
first few words, which is obviously an act of cheating.

The influence of text stimuli leakage on decoder is detected through BLEU
scores (Papineni et al., 2002) and ROUGE-1 scores (Lin, 2004), which measure
text similarity between generated text and ground truth. If evaluation
indicators keep improving as training epochs increase, we believe part of the
test set is leaked into training set and the model is overfitting. For fMRI
signal, we test five current dataset splitting methods under different
training settings. As shown in Table 2, we test two kinds of UniCoRN models.
One is UniCoRN with finely tuned hyper-parameters claimed in the original
paper, and the other is UniCoRN∗ with a randomly initialized encoder.
Empirically, the former will perform much better than the latter. However, in
method (a), (c), (d), due to text stimuli leakage, if we reduce the learning
rate and extend training epochs, UniCoRN∗ performs much better than UniCoRN
and its performance keeps rising with longer training epochs. As to method (b)
and (e) with no text stimuli leakage, changing training epochs or learning
rates makes no obvious difference to model performance. For EEG signal, the
conclusion is similar as shown in Table 3. For method (a) and (c) with text
stimuli leakage, model performance keeps rising with longer training epochs.
For method (d) without text stimuli leakage, both models reach optimal
performance after the first few rounds of training epochs.

###  4.5 Fair Benchmark

We evaluate current SOTA models for brain-to-text decoding under our dataset
splitting method and release a fair benchmark. UniCoRN is tested for both fMRI
and EEG decoding, EEG2Text model is tested for EEG decoding. The results are
listed in Table 4. For EEG dataset, UniCoRN achieves higher results in
BLEU-2,3,4 while EEG2Text is better in BLEU-1 and ROUGE-1.

##  5 Conclusion

In this paper, we explore a controversial topic: Due to the complexity of
cognitive datasets, no consensus has been reached on how to split the dataset
for training, validating, and testing in brain-to-text decoding. We analyze
current dataset splitting methods and find serious data contamination largely
exaggerates model performance and leads to poor generalization. Sufficient
experiments and analysis are conducted to prove the existence of the above
phenomena. Moreover, we propose a new dataset splitting method which can avoid
both cognitive signal and text stimuli leakage to fix this problem. Current
state-of-the-art models are reevaluated under this setting and a fair
benchmark is released for further research in the domain.

## Limitations

The "Narratives" dataset and the ZuCo dataset provide researchers with precise
cognitive signal resources stimulated by text or voice. However, in brain-to-
text decoding task, both subject’s cognitive signals and text stimuli in
validation and test set need to be invisible to training set, which makes
splitting these public datasets difficult. Our proposed dataset splitting
method meets the above requirements at the expense of discarding some data in
the dataset. We recommend future datasets in this domain follow these
guidelines. The division of the training set, validation set, and test set
should be provided when the dataset is released. Besides, we suggest hiring
new subjects with unique stimuli for validation set and test set, which is
good for testing the generalization ability of models without loss of data
(Tang et al., 2023). What’s more, we find existing models rely more on the
strong auto-regressive decoder to achieve good generation quality. The encoder
is of limited use in all SOTA models, which might become a research point in
the future.

## Ethics Statement

In this paper, we introduce a new dataset splitting method to avoid data
contamination for decoding cognitive signals to text task. Experiments are
conducted on public accessible cognitive datasets Narratives and ZuCo1.0 with
the authorization from their respective maintainers. Both datasets have been
de-identified by dataset providers and used for researches only.

## Acknowledgements

This research is supported by the National Natural Science Foundation of China
(No.62106105), the CCF-Baidu Open Fund (No.CCF-Baidu202307), the CCF-Zhipu AI
Large Model Fund (No.CCF-Zhipu202315), the Scientific Research Starting
Foundation of Nanjing University of Aeronautics and Astronautics
(No.YQR21022), and the High Performance Computing Platform of Nanjing
University of Aeronautics and Astronautics. We appreciate JD.com for providing
powerful GPUs for experiments.

## References

  * Anumanchipalli et al. (2019) Gopala K Anumanchipalli, Josh Chartier, and Edward F Chang. 2019.  Speech synthesis from neural decoding of spoken sentences.  _Nature_ , 568(7753):493–498.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Bouton et al. (2016) Chad E Bouton, Ammar Shaikhouni, Nicholas V Annetta, Marcia A Bockbrader, David A Friedenberg, Dylan M Nielson, Gaurav Sharma, Per B Sederberg, Bradley C Glenn, W Jerry Mysiw, et al. 2016.  Restoring cortical control of functional movement in a human with quadriplegia.  _Nature_ , 533(7602):247–250.
  * Burle et al. (2015) Boris Burle, Laure Spieser, Clémence Roger, Laurence Casini, Thierry Hasbroucq, and Franck Vidal. 2015.  Spatial and temporal resolutions of eeg: Is it really black and white? a scalp current density view.  _International Journal of Psychophysiology_ , 97(3):210–220.
  * Défossez et al. (2023) Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. 2023.  Decoding speech perception from non-invasive brain recordings.  _Nature Machine Intelligence_ , 5(10):1097–1107.
  * Gao et al. (2021) Xiaorong Gao, Yijun Wang, Xiaogang Chen, and Shangkai Gao. 2021.  Interface, interaction, and intelligence in generalized brain–computer interfaces.  _Trends in Cognitive Sciences_ , 25(8):671–684.
  * Handiru and Prasad (2016) Vikram Shenoy Handiru and Vinod A. Prasad. 2016.  Optimized bi-objective eeg channel selection and cross-subject generalization with brain–computer interfaces.  _IEEE Transactions on Human-Machine Systems_ , 46(6):777–786.
  * Herff et al. (2015) Christian Herff, Dominic Heger, Adriana De Pesters, Dominic Telaar, Peter Brunner, Gerwin Schalk, and Tanja Schultz. 2015.  Brain-to-text: decoding spoken phrases from phone representations in the brain.  _Frontiers in neuroscience_ , 9:217.
  * Hochberg et al. (2012) Leigh R Hochberg, Daniel Bacher, Beata Jarosiewicz, Nicolas Y Masse, John D Simeral, Joern Vogel, Sami Haddadin, Jie Liu, Sydney S Cash, Patrick Van Der Smagt, et al. 2012.  Reach and grasp by people with tetraplegia using a neurally controlled robotic arm.  _Nature_ , 485(7398):372–375.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Comput._ , 9(8):1735–1780.
  * Hollenstein et al. (2018) Nora Hollenstein, Jonathan Rotsztejn, Marius Troendle, Andreas Pedroni, Ce Zhang, and Nicolas Langer. 2018.  Zuco, a simultaneous eeg and eye-tracking resource for natural sentence reading.  _Scientific data_ , 5(1):1–13.
  * Huang et al. (2010) Gan Huang, Guangquan Liu, Jianjun Meng, Dingguo Zhang, and Xiangyang Zhu. 2010.  Model based generalization analysis of common spatial pattern in brain computer interfaces.  _Cognitive neurodynamics_ , 4:217–223.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7871–7880. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Makin et al. (2020) Joseph G Makin, David A Moses, and Edward F Chang. 2020.  Machine translation of cortical activity to text with an encoder–decoder framework.  _Nature neuroscience_ , 23(4):575–582.
  * Moses et al. (2021) David A Moses, Sean L Metzger, Jessie R Liu, Gopala K Anumanchipalli, Joseph G Makin, Pengfei F Sun, Josh Chartier, Maximilian E Dougherty, Patricia M Liu, Gary M Abrams, et al. 2021.  Neuroprosthesis for decoding speech in a paralyzed person with anarthria.  _New England Journal of Medicine_ , 385(3):217–227.
  * Mridha et al. (2021) Muhammad F Mridha, Sujoy Chandra Das, Muhammad Mohsin Kabir, Aklima Akter Lima, Md Rashedul Islam, and Yutaka Watanobe. 2021.  Brain-computer interface: Advancement and challenges.  _Sensors_ , 21(17):5746.
  * Nastase et al. (2021) Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Keshavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. 2021.  The “narratives” fmri dataset for evaluating models of naturalistic language comprehension.  _Scientific data_ , 8(1):250.
  * Noppeney and Price (2004) Uta Noppeney and Cathy J Price. 2004.  An fmri study of syntactic adaptation.  _Journal of Cognitive Neuroscience_ , 16(4):702–713.
  * Panachakel and Ramakrishnan (2021) Jerrin Thomas Panachakel and Angarai Ganesan Ramakrishnan. 2021.  Decoding covert speech from eeg-a comprehensive review.  _Frontiers in Neuroscience_ , 15:392.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA_ , pages 311–318. ACL.
  * Polikov et al. (2005) Vadim S Polikov, Patrick A Tresco, and William M Reichert. 2005.  Response of brain tissue to chronically implanted neural electrodes.  _Journal of neuroscience methods_ , 148(1):1–18.
  * Sun et al. (2019) Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2019.  Towards sentence-level brain decoding with distributed representations.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 7047–7054. AAAI Press.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , pages 3104–3112.
  * Tang et al. (2023) Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. 2023.  Semantic reconstruction of continuous language from non-invasive brain recordings.  _Nature Neuroscience_ , pages 1–9.
  * Vouloumanos et al. (2001) Athena Vouloumanos, Kent A Kiehl, Janet F Werker, and Peter F Liddle. 2001.  Detection of sounds in the auditory stream: event-related fmri evidence for differential activation to speech and nonspeech.  _Journal of Cognitive Neuroscience_ , 13(7):994–1005.
  * Wang and Ji (2022) Zhenhailong Wang and Heng Ji. 2022.  Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification.  In _Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022_ , pages 5350–5358. AAAI Press.
  * Xi et al. (2023) Nuwa Xi, Sendong Zhao, Haochun Wang, Chi Liu, Bing Qin, and Ting Liu. 2023.  Unicorn: Unified cognitive signal reconstruction bridging cognitive signals and human language.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 13277–13291. Association for Computational Linguistics.
  * Zou et al. (2021) Shuxian Zou, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2021.  Towards brain-to-text generation: Neural decoding with pre-trained encoder-decoder models.  In _NeurIPS 2021 AI for Science Workshop_.

##  Appendix A Implementation Details

More details in experiments are supplemented in this section. We perform the
same filtering steps to Narratives dataset as UniCoRN paper (Xi et al., 2023)
and the same filtering steps to ZuCo1.0 as EEG2Text paper (Wang and Ji, 2022).
In CSLR and TSLR calculation, the number of four different seeds are set as
$1,2,3,4$ respectively. In signal reconstruction task for encoder of UniCoRN,
the batch size of EEG and fMRI data is 512 and 320 respectively. The learning
rate is set as 1e-4 and 1e-3 separately as the author claimed in the original
paper. In the fair benchmark, for fMRI data, encoder of UniCoRN is trained
through 1e-4 learning rate and decaying to 1e-6 finally for 30 training
epochs. Decoder is trained through 1e-4 learning rate and decaying to 1e-6
finally for 10 training epochs with 90 batch size. Sample length $L$ is set as
10 for all experiments related to fMRI. For EEG data, EEG2Text model is
trained with 1e-6 learning rate for 80 epochs. UniCoRN model is trained with
the same settings as fMRI data.

##  Appendix B Our Dataset Splitting Method

In this part, we release the pseudo-code of two dataset splitting methods for
EEG and fMRI signal. As shown in Figure 3, our proposed dataset splitting
method consists of four steps. The blue lines stand for the situation of
original dataset. The main difference between two methods lies in the how
$\mathcal{G}_{2}$ is generated. We always choose the side with fewer nodes in
bipartite graph $\mathcal{G}_{1}$ to perform $\mathcal{G}_{2}$ generation. For
example, in Algorithm 1 where we assert $|\mathcal{U}|<|\mathcal{V}|$, the
adjacency matrix is initialized as $M\times N$. In Algorithm 2 where
$|\mathcal{V}|<|\mathcal{U}|$, the adjacency matrix is initialized as $N\times
K$. All hypotheses are based on analysis of cognitive datasets.

One more thing to notice is that in Line 14 of both pseudo-code, the loop
indicates extending training set, validation set, and test set respectively.
So the names of variable should be alternated in the repeat loop and the
displayed part in pseudo-cod is a case example of extending training set. We
write it in this way for simplicity of expression.

1 Initialize: Bipartite graph
$\mathcal{G}_{1}=(\mathcal{U},\mathcal{V},\mathcal{E})$,
$\mathcal{G}_{2}=(\mathcal{U},\mathcal{V},\mathcal{E}^{\prime})$ where
$\mathcal{U}=\\{S_{i}\\}_{i=1}^{N}$ and $\mathcal{V}=\\{T_{j}\\}_{j=1}^{M}$,
Adjacency matrix $A_{1}$ of $\mathcal{G}_{1}$ where $A_{1}[i][j]=1$ if node
$i$ and node $j$ is connected else $A_{1}[i][j]=0$, Adjacency matrix $A_{2}$
of $\mathcal{G}_{2}$ where $A_{2}[i][j]=0$, Array $C$ where
$len(C)=len(\mathcal{U})$ and $C[i]=0$;

2 for  _$u\leftarrow U_{1}$ to $U_{N}$_ do

3 $C_{copy}\leftarrow C$;

4 for  _$v\leftarrow A_{1}[u][0]$ to $A_{1}[u][M]$_ do

5 if  _$v=0$_ then

6 $C_{copy}[v.index]\leftarrow\infty$;

7

8  _Minimum_ $=\min(C_{copy})$;

9 $A_{2}[u][$_Minimum_ $.index]\leftarrow 1$;

10 $C[$_Minimum_ $.index]\leftarrow C[$_Minimum_ $.index]+1$; // Make degree
of nodes balanced

11Split by subjects $\mathcal{U}$ according to default ratio;

12
$\mathcal{G}_{2}=\mathcal{G}_{train}\cup\mathcal{G}_{val}\cup\mathcal{G}_{test},\,\mathcal{U}_{train}\cap\mathcal{U}_{val}\cap\mathcal{U}_{test}=\emptyset,\,\mathcal{V}_{train}\cap\mathcal{V}_{val}\cap\mathcal{V}_{test}=\emptyset$;

13 repeat // To three sets respectively, below is for training set

14 for  _$u$ in $\mathcal{U}$ _ do

15 for  _$v$ in $\mathcal{V}$_ do

16 if  _$e=(u,v)\in\mathcal{E}$ and
$e=(u,v)\notin\mathcal{E}^{\prime}_{train}$ and
$u\notin\mathcal{U}_{val}\cup\mathcal{U}_{test}$_ then

17
$\mathcal{E}_{train}^{\prime}\leftarrow\mathcal{E}_{train}^{\prime}\cup\\{e\\}$;

18

19

20

21until  _$\mathcal{G}_{train},\mathcal{G}_{val},\mathcal{G}_{test}$ _are all
extended__ ;

22return $\mathcal{G}_{train},\mathcal{G}_{val},\mathcal{G}_{test}$;

Algorithm 1 Dataset splitting method for EEG signal

23 Initialize: Bipartite graph
$\mathcal{G}_{1}=(\mathcal{U},\mathcal{V},\mathcal{E})$,
$\mathcal{G}_{2}=(\mathcal{U},\mathcal{V},\mathcal{E}^{\prime})$ where
$\mathcal{U}=\\{S_{i}\\}_{i=1}^{N}$, $\mathcal{V}=\\{M_{k}\\}_{k=1}^{K}$,
Adjacency matrix $A_{1}$ of $\mathcal{G}_{1}$ where $A_{1}[i][j]=1$ if node
$i$ and node $j$ is connected else $A_{1}[i][j]=0$, Adjacency matrix $A_{2}$
of $\mathcal{G}_{2}$ where $A_{2}[i][j]=0$, Array $C$ where
$len(C)=len(\mathcal{V})$ and $C[i]=0$;

24 for  _$v\leftarrow V_{1}$ to $V_{K}$_ do

25 $C_{copy}\leftarrow C$;

26 for  _$u\leftarrow A_{1}[v][0]$ to $A_{1}[v][K]$_ do

27 if  _$u=0$_ then

28 $C_{copy}[u.index]\leftarrow\infty$;

29

30  _Minimum_ $=\min(C_{copy})$;

31 $A_{2}[v][$_Minimum_ $.index]\leftarrow 1$;

32 $C[$_Minimum_ $.index]\leftarrow C[$_Minimum_ $.index]+1$; // Make degree
of nodes balanced

33Split by tasks $\mathcal{V}$ according to default ratio;

34
$\mathcal{G}_{2}=\mathcal{G}_{train}\cup\mathcal{G}_{val}\cup\mathcal{G}_{test},\,\mathcal{U}_{train}\cap\mathcal{U}_{val}\cap\mathcal{U}_{test}=\emptyset,\,\mathcal{V}_{train}\cap\mathcal{V}_{val}\cap\mathcal{V}_{test}=\emptyset$;

35 repeat // To three sets respectively, below is for training set

36 for  _$v$ in $\mathcal{V}$ _ do

37 for  _$u$ in $\mathcal{U}$_ do

38 if  _$e=(u,v)\in\mathcal{E}$ and
$e=(u,v)\notin\mathcal{E}^{\prime}_{train}$ and
$v\notin\mathcal{V}_{val}\cup\mathcal{V}_{test}$_ then

39
$\mathcal{E}_{train}^{\prime}\leftarrow\mathcal{E}_{train}^{\prime}\cup\\{e\\}$;

40

41

42

43until  _$\mathcal{G}_{train},\mathcal{G}_{val},\mathcal{G}_{test}$ _are all
extended__ ;

44return $\mathcal{G}_{train},\mathcal{G}_{val},\mathcal{G}_{test}$;

Algorithm 2 Dataset splitting method for fMRI signal


# Improving Document Classification with Multi-Sense Embeddings

Vivek Gupta University of Utah, email<EMAIL_ADDRESS>Karaggpur, email:
<EMAIL_ADDRESS>of Utah, email<EMAIL_ADDRESS>Guwahati, email<EMAIL_ADDRESS>Bangalore, email<EMAIL_ADDRESS>Ankit Kumar IIT Karaggpur, email<EMAIL_ADDRESS>of Utah,
email<EMAIL_ADDRESS>Guwahati, email<EMAIL_ADDRESS>Bangalore, email<EMAIL_ADDRESS>Pegah Nokhiz University of Utah, email:
<EMAIL_ADDRESS>Guwahati, email<EMAIL_ADDRESS>Bangalore,
email<EMAIL_ADDRESS>Harshit Gupta IIT Guwahati, email:
<EMAIL_ADDRESS>Bangalore, email<EMAIL_ADDRESS>Partha Talukdar
IISC Bangalore, email<EMAIL_ADDRESS>
###### Abstract

Efficient representation of text documents is an important building block in
many NLP tasks. Research on long text categorization has shown that simple
weighted averaging of word vectors for sentence representation often
outperforms more sophisticated neural models. Recently proposed Sparse
Composite Document Vector (SCDV) [32] extends this approach from sentences to
documents using soft clustering over word vectors. However, SCDV disregards
the multi-sense nature of words, and it also suffers from the curse of higher
dimensionality. In this work, we address these shortcomings and propose SCDV-
MS. SCDV-MS utilizes multi-sense word embeddings and learns a lower
dimensional manifold. Through extensive experiments on multiple real-world
datasets, we show that SCDV-MS embeddings outperform previous state-of-the-art
embeddings on multi-class and multi-label text categorization tasks.
Furthermore, SCDV-MS embeddings are more efficient than SCDV in terms of time
and space complexity on textual classification tasks. We have released SCDV-MS
source code with the paper. 111https://github.com/vgupta123/SCDV-MS

##  1 Introduction

Distributed word embeddings such as word2vec [35] are effective in capturing
the semantic meanings of the words by representing them in a lower-dimensional
continuous space. A smooth inverse frequency-based word vector averaging
technique for sentence embeddings was proposed by [4]. However, because the
final representation is in the same space as the word vectors, these methods
are only capable of capturing the meaning of a single sentence. Thus,
embedding a large text document in a dense, low-dimensional space is a
challenging task.

[32] attempted to resolve this problem by proposing a clustering-based word
vector averaging approach (SCDV) for embedding larger text documents. SCDV
embeds each document by cluster-based averaging, thus representing each
document in a more representative space than the original vectors. This model
combines the word embedding models with a latent topic model where the topic
space is learned efficiently using a soft clustering technique over
embeddings. The final document vectors are also made sparse to reduce time and
space complexities in several downstream tasks.

SCDV has many shortcomings: applying thresholding-based sparsity on the
document representations can be unreliable since it is highly sensitive to the
number of documents in the corpus. SCDV does not utilize the word contexts to
disambiguate word sense for learning sense-aware document representations.
Ignoring the multi-sense nature of the words during the representation leads
to cluster ambiguity. SCDV neglects the negative additive effect of common
words such as ‘and’, ‘the’, etc. during final document representations.
Lastly, the documents represented by SCDV suffer from the curse of high
dimensionality and cannot be utilized for deep learning applications which
require low-dimensional continuous representations. To overcome these
challenges, we proposed a novel document representation technique, namely
SCDV-MS. Our proposed SCDV-MS mitigated the above shortcomings by the
following contributions.

  1. 1.

To overcome the problem of cluster ambiguity SCDV-MS replaced the single sense
word vector representations with multi-sense context-sensitive word
representations to resolve word sense disambiguation.

  2. 2.

SCDV-MS removed the noise in the final representation by applying a threshold-
based sparsity directly on fuzzy word cluster assignments instead of the
document representations. Sparser representations result in better
performance, lower time and space complexities.

  3. 3.

To overcome the noisy negative additive effect of common words such as ‘and’,
‘the’, etc. SCDV-MS learned and used Doc2VecC-initialized [9] robust word
vectors to zero out common and high frequent words.

  4. 4.

Lastly, we showed that the sparse word-topic vectors can be projected into a
non-linear local neighborhood preserving a manifold to learn continuous
distributed representations much more effectively and efficiently than SCDV
which proves to be useful for deep learning application.

Overall, we show that: disambiguating the multiple senses of words based on
their context words (adjacent words) can lead to better document
representations. Sparsity in representations is helpful for effective and
efficient lower-dimensional manifold representation learning. Representation
noise at words’ level has a significant impact on the final downstream tasks.

In section 1, we provided a brief introduction to the problem statement. In
section 2 we discuss the related work in document representations. In section
3, we describe the proposed algorithm and then discuss the proposed
modifications compared to SCDV in section 4. We move on to experiments in
section 5, followed by conclusions in section 6.

##  2 Related Work

For document representation, averaging of word-vectors with an unweighted
scheme was proposed by [45, 36, 37, 34]. [44] extended the previous simple
averaging model by tf-idf weighting of word vectors to form document vectors.
[26] proposed paragraph models (PV-DM and PV-DBOW) similar to word vector
models (CBoW and SGNS) by treating each paragraph as a pseudoword. [46] used a
parse tree to train a Recursive Neural Network (RNN) with supervision. In
addition, several neural network models such as seq2seq models: Recurrent
Neural Networks (RNN) [33], Long Short Term Memory Networks (LSTM) [16] and a
hierarchical model: Convolutional Neural Networks (CNN) [23, 21] were proposed
to capture syntax while representing documents. [49] used supervised learning
over the Paraphrase Dataset (PPDB) to learn Paraphrastic Sentence embeddings
(PSL). Later, [48] also propose an optimization of word embeddings based on a
neural network and a cosine similarity measure.

Several models such as WTM [15], TWE [30], NTSG [30], LTSG [25], w2v-LDA [39],
TV+MeanWV [28], Topic2Vec [40], Gaussian-LDA [11], Lda2vec [38], ETM [14],
D-ETM [13] and MvTM [29] combine topic modeling [8] with word vectors to learn
better word and sentence representations. [24] cast the distributional
hypothesis to a sentence level by proposing skip-thought document vectors.
Recently, two deep contextual word embeddings namely ELMo [41] and BERT [12]
were proposed. These contextual embeddings perform as state of the art on
multiple NLP tasks since they are very effective in capturing the surrounding
context. Interestingly, [27] checks the effect of using multi-sense embeddings
on various NLP tasks. However, our goal is different and aim at effectively
using multi-sense words embeddings to learn better document representations. A
hard clustering-based averaging of word vectors was proposed by [17, 3] to
form document vectors. [18] extended the approach with a better partitioning
technique and tried it on other natural language tasks. [32] further improved
the state-of-the-art SCDV by using fuzzy clustering and tf-idf weighted word
averaging. Their method outperformed earlier models on several NLP tasks.

##  3 Proposed Algorithm SCDV-MS

In this section, we will describe our new proposed algorithm 1 in details. The
algorithm is similar to SCDV [32], but with important modifications and has
three main components as described below:

Data: Documents Dn, $n=1,\ldots,N$

Result: Document vectors $\vec{dv}_{D_{n}}$, $n=1,\ldots,N$

/* Word Sense Disambiguation */

1 Use adagram for word sense disambiguation;

2 Annotate each word with a sense according to the neighboring context words;

3 Obtain word vectors ($\vec{wv}_{i}$) on annotated corpus using Doc2VecC;

4 for  _each word $w_{i}$ $\in V$_ do

5 obtain idf values, idf(wi), $i=1..|V|$ ;

/* $|V|$ is the vocabulary */

6

7 end for

/* Word Vector Clustering */

8 Fuzzy clusters $\vec{wv}$ in $K$ clusters;

9 Each word $w_{i}$ and cluster $c_{k}$, obtain P$(c_{k}|w_{i})$ ;

10 $\vec{\text{SP}(c|w_{i})}$ = make-sparse($\vec{\text{P}(c|w_{i})}$);

/* Word Topic Vectors */

11 for  _each word $w_{i}$ $\in$ $V$_ do

12 for  _each cluster $c_{k}$_ do

13 $\vec{wcv}_{ik}$ $=$ $\vec{wv_{i}}$ $\times$ SP$(c_{k}|w_{i})$;

14

15 end for

16 $\vec{wtv}_{i}$ $=$ idf(wi) $\times$ $\bigoplus_{k=1}^{K}$ $\vec{wcv}_{ik}$
;

/* $\bigoplus$ is concatenation */

/* Optional: Manifold Learning */

17 $\vec{rwtv}_{w}$ = manifold-proj($\vec{wtv}_{w}$);

18

19 end for

/* SCDV-MS Representation */

20 for  _ $n\in(1..N)$_ do

/* initialize vectors */

21 $\vec{dv}_{D_{n}}$ = $\vec{0}$;

22 for  _word $w_{i}$ in Dn_ do

23 $\vec{dv}_{D_{n}}$ += $\vec{wtv}_{w_{i}}$;

24 end for

25

26 end for

Algorithm 1 SCDV-MS

###  3.1 Word Sense Disambiguation (Algo 1: 1 \- 1):

We employed the widely-used AdaGram [6] algorithm to disambiguate the multi-
sense words in our corpora. We chose AdaGram because it’s a nonparametric
Bayesian extension of Skip-gram which automatically learns the counts of the
senses of the multi-sense words and their representations. 222One could also
replace AdaGram with [10] and [5] We first trained the AdaGram algorithm on
the training corpora. 333 https://github.com/sbos/AdaGram.jl We used the
trained model to annotate the words with the corresponding word senses in all
train-test examples. We then trained the Doc2vecC algorithm on an annotated
corpus to obtain the final multi-sense word vectors. Lastly, we obtained the
idf values of words of the vocabulary which we will use as a means for
weighting the rare words (Lines 4-6 Algo 1).

###  3.2 Word Vector Clustering (Algo 1: 1 \- 1)

Similar to the SCDV approach, we clustered the word embeddings using Gaussian
Mixture Models (GMM), which is a soft clustering technique, and obtained the
word-cluster assignments probabilities P$(c_{k}|w_{i})$. Additionally, we made
use of the fact that GMMs have an irrelevant noisy tail and made the cluster
probability assignment $\vec{\text{P}(c|w_{i})}$ manually sparse by zeroing
the values of P$(c_{k}|w_{i})$. Retaining only the top $l$ maximum
P$(c_{k}|w_{i})$ and zeroing the rest $K-l$ values results in a sparse word-
cluster assignment vector $\vec{\text{SP}(c|w_{i})}$ for each word. Here, $K$
represents the total number of clusters and $l$ is the sparsity constant
($l<<K$). One can use different values of $l$ for each word ($w_{i}$)
depending on the values of P$(c_{k}|w_{i})$ . However, in our experiments we
did not observe significant performance difference when $l$ is varied with
respect to the words.

$\small\text{SP}(c_{k}|w_{i})=\begin{cases}\text{P}(c_{k}|w_{i})&\text{if
}k\in\\{k|\operatorname*{arg\,max}_{k}^{l}{{\text{P}(c_{k}|w_{i})}}\\}\\\
0&\text{otherwise}\end{cases}$ (1)

$\operatorname*{arg\,max}_{k}^{l}{{\text{P}(c_{k}|w_{i})}}$ outputs indices of
the top $l$ maximum assignments.

###  3.3 Word Topic Vectors (Algo 1: 1 \- 1)

Similar to SCDV, for each word $w_{i}$ $\in$ $V$, we created $K$ different
word-cluster vectors of $d$ dimensions ($\vec{wcv}_{ik}$) by weighting the
word’s embedding with sparse probability distribution for the $k^{th}$
cluster, i.e., $SP(c_{k}|w_{i})$. Next, we concatenated the word-cluster
vectors ($\vec{wcv}_{ik}$) which are $K$ word-topic vectors in total, into a
$K\times d$ embedding vector. We then weighted it with inverse document
frequency (idf) of $w_{i}$ to obtain word-topic vectors ($\vec{wtv}_{i}$). For
all words appearing in a given document $D_{n}$, we computed the average of
the corresponding projected lower dimensional word-topic vectors
$\vec{wtv}_{i}$ to obtain the document vector $\vec{dv}_{D_{n}}$. Furthermore,
one can optionally project the ($\vec{wtv}_{i}$) into a lower dimensional
continuous representation called reduced word topic vectors,
($\vec{rwtv}_{i}$), using manifold learning algorithms, namely Random
Projection [2], PCA [1] and Denoising Autoencoders [47]. We can then use them
instead of ($\vec{wtv}_{i}$) for document representation. We call this
reduction-based representation method as R-SCDV-MS. Refer to section 4.1 on
manifold learning for more details.

$\small\vec{wcv}_{ik}=\vec{wv}_{i}\times\text{SP}(c_{k}|w_{i})$ (2)
$\small\vec{wtv}_{i}=\text{idf}(w_{i})\times\oplus_{k=1}^{K}\vec{wcv}_{ik}$
(3) $\small\vec{rwtv}_{i}=\text{manifold-proj}(\vec{wtv}_{i})$ (4)

$\bigoplus$ is the concatenation and manifold-proj is the manifold learning
algorithm we utilized. Figures 3 and 3 show the flow-chart of high level flow
of our proposed SCDV-MS embedding.

##  4 Discussion on Proposed Modifications

In this section, we will describe the modifications applied to the SCDV
embeddings in details.

###  4.1 Word Representation: Single Sense vs Multi Sense

SCDV-MS used a multi-sense approach instead of single sense word embeddings
because SCDV does not disambiguate the senses of the words based on the
context words used in the documents. SCDV-MS performed an automatic word sense
disambiguation using multi-sense word embeddings according to the context
determined by the neighboring words to resolve cluster ambiguity for
polysemous words. Table 1 shows examples of multi-sense words along with their
fitting context and the prominent words of the assigned clusters.

Figure 1: Effect of sense disambiguation on word cluster assignment
probabilities

Figure 1 shows the effect of sense disambiguation on fuzzy GMM clustering on
the 20NewsGroup dataset. The same word is assigned to different clusters
depending on its context which helps in resolving the word cluster
ambiguities, e.g., without sense disambiguation, the word ’Subject’ belongs to
cluster $13$ with probability $0.25$ and cluster $50$ with probability $0.65$.
But after sense disambiguation we acquire two word embeddings of the word
‘Subject’, i.e., ‘Subject#1’ and ‘Subject#2’. ‘Subject#1’ belongs to cluster
$50$ with probability of $0.9$ and ‘Subject#2’ belongs to cluster $13$ with
probability $0.8$. So depending on the context in which word ‘Subject’ is
used, the algorithm assigns ‘Subject’ to a single cluster based on its sense;
thus word cluster ambiguity is resolved. We observe similar disambiguation
effects for other polysemous words in the corpus.

Table 1: Examples of multi-sense words along with their context words and the corresponding prominent cluster words Multi-Sense Words | Sentence (Context Words) | Prominent Cluster Words
---|---|---
Subject |  The math subject is a nightmare for many students | physics, chemistry, math, science
|  In anxiety, he sent an email without a subject | mail, letter, email, gmail
|  After promotion, he went to Maldives for spring break | vacation, holiday, trip, spring
Break |  Breaking government websites is common for hackers | encryption, cipher, security, privacy
|  Use break to stop growing recursion loops | if, elseif, endif, loop, continue
Unit |  The S.I. unit of distance is meter | calculation, distance, mass, length
|  Multimeter shows a unit of 5V | electronics, KWH, digital, signal
Interest |  His interest lies in astrophysics | information, enthusiasm, question
|  Bank’s interest rates are controlled by RBI | bank, market, finance, investment

###  4.2 Thresholding Word Cluster Assignments

In SCDV, the thresholding is applied to the final document vectors. However,
applying the hard thresholding in an earlier stage in word cluster assignments
($P(c_{i}|w)$) results in better heavy tail noise removal and yields more
robust representations. Thus, SCDV-MS applied the hard thresholding directly
on the word cluster assignments instead of the final document representations.
Furthermore, applying sparsity over vocabulary words with fewer dimensions
($O(VK)$) instead of millions of documents ($O(NKd)$) results in higher
efficiency ($Nd>>V$). Here, $N$ is the number documents, $V$ is vocabulary,
$K$ is number of clusters, and $d$ is word vector dimensions. Empirically, on
20NewGroups we observe that about $98\%$ of entries in word-cluster
assignments $(P(c_{i}|w))$ for all words are close to $0$ ($<$ $0.01$). For
each word on average, the probability of cluster assignment ($P(c_{i}|w)$) for
$58$ cluster assignments out of $60$ (variance of $1.56$) is less than $0.01$.
Thus, applying thresholding at word-cluster assignment level is reasonable.

###  4.3 Doc2VecC vs SGNS Representations

In the SCDV approach, the SGNS word vectors represent the common words (mainly
stop words) as non-zero vectors. This makes the clusters redundant and
generates a heavy tail noise. SCDV-MS addressed this issue by using Doc2VecC
[9] which introduces corruption while doing context addition in word
embeddings to help in learning robust word vector representations. In this
approach, the common words of the corpus are forcefully learned as zeroed
vectors. We observed that using Doc2VecC trained word vectors results in non-
redundant diverse clusters. Thus, using Doc2VecC trained word vectors not only
improves the performance but also reduces the feature size. There is no
running time overhead for Doc2VecC compared to the SGNS.

###  4.4 Low Dimensional Manifold Learning

SCDV [32] represents documents as high dimensional sparse vectors. The SCDV
approach showed that such vectors are useful for linear classification models.
However, being useful for many downstream applications (especially the ones
using deep learning models) requires a continuous low dimensional document
representation similar to word vectors. To overcome this issue, SCDV-MS
projected the sparse word-topic vectors into a lower-dimensional manifold
which preserves the local neighborhood using simple techniques such as random
projection. Furthermore, the manifold learning is applied over word
vocabularies instead of millions of documents, which is more efficient.
Dimensionality reduction for SCDV-MS is roughly ($O(\frac{N}{V})$) (where $N$
is the number of documents and $V$ is the size of vocabulary) faster than
SCDV.

Figure 2: Flowchart representing modified $\vec{wtv}$ computation.

Figure 3: Flowchart representing final document vector computation.

##  5 Experimental Results

Document embeddings obtained using SCDV-MS can be used as direct features for
downstream supervised tasks. We experimented with text classification tasks
(see Table 2) to show the effectiveness of our embeddings. We evaluated the
following questions through our experiments.

Table 2: Text classification datasets overview. Task | Dataset | #Classes | #train / #test
---|---|---|---
Multi-class | 20NewsGroup | 20 | 11K / 8K
Multi-label | Reuters-21578 | 444 | 13K / 6K

  1. Q1.

Does disambiguating word-cluster assignments using multi-sense embedding
improve classification performance?

  2. Q2.

Does hard thresholding over word-cluster assignments improve performance,
space and time complexities?

  3. Q3.

Is representational noise reduction using Doc2Vec initialization effective?

  4. Q4.

Can effective lower dimensional manifold be learned from the sparse high
dimensional word topic vectors?

Baselines:  We considered the following baselines: Bag-of-Words (BoW) [19],
Bag of Word Vector (BoWV) [17], 444https://bit.ly/2X0XfBH Sparse Composite
Document Vectors (SCDV) [32], 555https://bit.ly/36NxGZh paragraph vectors
[26], pmeans [43], ELMo [42], Topical word embeddings (TWE-1) [31], Neural
Tensor Skip-Gram Model (NTSG-1 to NTSG-3) [30], tf-idf weighted average word-
vector [44] and weighted Bag of Concepts (weight-BoC) [22], and BERT [12]. In
BoC we built topic-document vectors by counting the member words in each
topic. For BERT, we reported the results on the unsupervised pre-trained (pr)
model because of a fair comparison to our approach which is also unsupervised.
In Doc2VecC [9] averaging and training the vectors was done jointly with
corruption. Also, in SIF [4] we used the inverse frequency weights for
weighting while averaging word vectors, and finally removed the common
components from the average. The results of our proposed embeddings is
represented by SCDV-MS in Tables 5.1 and 3. We also compared our
representation with topic modeling-based embedding methods, described in the
related work.

The Experimental Setting: We learned the word embeddings with Skip-Gram
Negative Sampling (SGNS) using commonly used parameters, e.g., negative sample
size of $10$, minimum word frequency of $20$, and the window size of $10$. We
ensure for usual data cleansing like stop word removal, lemmetization and
stemming for all the baselines. In addition, we used simple models such as
LinearSVM for multi-class classification and Logistic regression with a
OneVsRest setting for the multi-label classification tasks so that we can
directly compare our results with the previous approaches which uses the same
classifiers. Similar to SCDV, to tune the hyperparameters, we employed a
5-fold cross-validation on the F1 score. We also used the Doc2VecC model [9]
to initialize the word embeddings on the annotated corpora for performance
improvement. To ensure a fair comparison with SCDV, we fixed the same number
of clusters to $60$ and used full covariance for GMM clustering for all
experiments based on our best empirical results with cross-validation. We
tuned the hard threshold sparsity constant $l$ from range $\\{3,5,7\\}$ with
cross-validation to select the best hyper-parameter for making the word
cluster assignments sparse. Moreover, we used AdaGram [6] for disambiguating
the sense of multi-sense words using a neighborhood of $5$ context words on
both sides, so that the window size is $10$. We first ranked our words based
on their tf-idf scores; we then selected a practicable number (top $5000$
words) as candidates for the polysemic words. Next, we selected the true
polysemic words by applying AdaGram on the candidates.
666https://bit.ly/2Jv6wxX The best parameter settings were used to generate
baselines results. We used $200$ dimensions for the tf-idf weighted word-
vector model, $400$ for the paragraph vector model, $80$ topics and $400$
dimensional vectors for TWE/NTSG/LTSG, and $60$ topics and $200$ dimensional
word vectors for SCDV and BOWV. We reported the average of $5$ runs. Our
results were robust across multiple runs with a variance of O($10^{-6}$).

###  5.1 Text Classification Results

We evaluated the classifier’s performance on multi-class classification using
several metrics such as accuracy, macro-averaging precision, recall, and macro
F1-score. Table 3 shows a comparison with multiple state-of-the-art document
representations (the first 7 except BERT/ELMo are clustering-based, the next
11 topic-word embeddings based, the next 6 are simple averaging or topic
modeling methods) on the 20NewsGroup dataset. We also reported the results
(micro F1) on the $20$ classes of 20NewsGroup in Table 4. Furthermore, we
evaluated the multi-label classification performance using several metrics
such as Precision@K, nDCG@k [7], Coverage error, Label ranking average
precision score (LRAPS)777Section $3.3.3.2$ of https://goo.gl/4GrR3M and macro
F1-score. Table 5.1 shows the results on the Reuters dataset.

Table 3: Performance on multi-class classification. Values in bold show the best performance using the SCDV-MS embeddings. Model | Accuracy | Precision | Recall | F-measure
---|---|---|---|---
SCDV-MS | 86.19 | 86.20 | 86.18 | 86.16
R-SCDV-MS | 84.9 | 84.9 | 84.9 | 84.9
BERT (pr)[12] | 84.9 | 84.9 | 85.0 | 85.0
SCDV [32] | 84.6 | 84.6 | 84.5 | 84.6
RandBin | 83.9 | 83.99 | 83 .9 | 83.76
BoWV [17] | 81.6 | 81.1 | 81.1 | 80.9
pmeans [43] | 81.9 | 81.9 | 81.9 | 81.5
Doc2VecC [9] | 84.0 | 84.1 | 84.1 | 84.0
BoE [20] | 83.1 | 83.1 | 83.1 | 83.1
NTSG-2 [30] | 82.5 | 83.7 | 82.8 | 82.4
LTSG [25] | 82.8 | 82.4 | 81.8 | 81.8
WTM [15] | 80.9 | 80.3 | 80.3 | 80.0
ELMo [42] | 74.1 | 74.0 | 74.1 | 73.9
w2v-LDA [39] | 77.7 | 77.4 | 77.2 | 76.9
TV+MeanWV [28] | 72.2 | 71.8 | 71.5 | 71.6
MvTM [29] | 72.2 | 71.8 | 71.5 | 71.6
TWE-1 [31] | 81.5 | 81.2 | 80.6 | 80.6
lda2Vec [38] | 81.3 | 81.4 | 80.4 | 80.5
lda [8] | 72.2 | 70.8 | 70.7 | 70.0
weight-AvgVec [44] | 81.9 | 81.7 | 81.9 | 81.7
BoW [44] | 79.7 | 79.5 | 79.0 | 79.0
weight-BOC [44] | 71.8 | 71.3 | 71.8 | 71.4
PV-DBoW [26] | 75.4 | 74.9 | 74.3 | 74.3
PV-DM [26] | 72.4 | 72.1 | 71.5 | 71.5
Table 4: Class-wise F1-Score on the 20newsgroup dataset with different document representations. Class Name | SCDV | SCDV-MS | R-SCDV-MS
---|---|---|---
alt.atheism | 80.14 | 81.35 | 80.39
comp.graphics | 78.99 | 76.84 | 76.95
comp.os.ms-windows.misc | 75.65 | 77.65 | 78.28
comp.sys.ibm.pc.hardware | 72.08 | 73.43 | 68.38
comp.sys.mac.hardware | 82.15 | 86.82 | 80.16
comp.windows.x | 81.8 | 82.97 | 83.27
misc.forsale | 82.8 | 85.13 | 84.99
rec.autos | 89.06 | 92.53 | 91.77
rec.motorcycles | 94.27 | 96.11 | 94.27
rec.sport.baseball | 93.57 | 96.47 | 93.68
rec.sport.hockey | 97.27 | 96.78 | 96.41
sci.crypt | 93.1 | 92.82 | 93.5
sci.electronics | 77.38 | 77.45 | 74.25
sci.med | 88.58 | 92.30 | 91.57
sci.space | 90.33 | 91.40 | 90.71
soc.religion.christian | 89.56 | 89.97 | 89.76
talk.politics.guns | 80.69 | 84.18 | 83.05
talk.politics.mideast | 95.96 | 95.95 | 96.1
talk.politics.misc | 69.33 | 73.49 | 73.67
talk.religion.misc | 65.53 | 65.54 | 60.48

Table 5: Performance on various metrics for multi-label classification on the Reuters dataset. Values in bold show the best performance using the SCDV-MS algorithm. Model |  Prec |  |  |  |  |
---|---|---|---|---|---|---
|  Prec |  |  |  |  |
|  nDCG |  |  |  |  |
|  Cover. |  |  |  |  |
| LRAPS |  F1 |  |  |  |
|  | Score |  |  |  |
SCDV-MS | 95.06 | 37.56 | 50.20 | 5.87 | 94.21 | 82.71
R-SCDV-MS | 93.56 | 37.00 | 49.47 | 6.74 | 92.96 | 81.94
BERT (pr) | 93.8 | 37 | 49.6 | 6.3 | 93.1 | 81.9
SCDV | 94.00 | 37.05 | 49.6 | 6.65 | 93.34 | 81.77
Doc2VecC | 93.45 | 36.86 | 49.28 | 6.83 | 92.66 | 81.29
pmeans | 93.29 | 36.65 | 48.95 | 7.66 | 91.72 | 77.81
BoWV | 92.90 | 36.14 | 48.55 | 8.16 | 91.46 | 79.16
TWE-1 | 90.91 | 35.49 | 47.54 | 8.16 | 91.46 | 79.16
PV-DM | 87.54 | 33.24 | 44.21 | 13.15 | 86.21 | 70.24
PV-DBoW | 88.78 | 34.51 | 46.42 | 11.28 | 87.43 | 73.68
tfidf AvgVec | 89.33 | 35.04 | 46.83 | 9.42 | 87.90 | 71.97

Datasets: We evaluated our approach by running multi-class classification
experiments on the 20NewsGroup dataset,
888http://qwone.com/j̃ason/20Newsgroups/ and multi-label classification
experiments on the Reuters-21578 dataset. 999https://goo.gl/NrOfu For more
details on dataset statistics refer to Table 2. We used script for datasets
preprocessing. 101010https://bit.ly/2PXDdXj

Table 6: Ablation Study reporting F1 scores. In $\pm x$, $x$ is the variance across several runs. Ablation (w/o) | 20NewsGroup | Reuters
---|---|---
Sparsity |  85.28 $\pm$ 0.002 |  82.17 $\pm$ 0.001
Doc2VecC |  85.41 $\pm$ 0.001 |  82.08 $\pm$ 0.002
MultiSense |  85.16 $\pm$ 0.001 |  82.43 $\pm$ 0.001
All |  84.61 $\pm$ 0.004 |  81.77 $\pm$ 0.003
None | 86.16 $\pm$ 0.002 | 82.71 $\pm$ 0.002
Table 7: Performance in terms of accuracy with various dimensionality reduction methods on the 20NewsGroup dataset. Similar results were acquired for precision, recall, and F1 score. Dimension |  Random |  |
---|---|---|---
|  PCA |  |
| Autoencoder |  |
200 | 78.97 | 80.62 | 81.44
500 | 82.19 | 83.14 | 83.83
1000 | 83.75 | 83.80 | 84.31
2000 | 84.47 | 84.34 | 84.80
3000 | 84.94 | 84.86 | 84.90

Figure 4: Percentage loss in F1-Score (%RL) after Random Projection-based
$\vec{wtv}$ dimensionality reduction on 20NewsGroup.

Figure 5: Percentage loss in F1-Score (%RL) after Autoencoder-based
$\vec{wtv}$ dimensionality reduction on 20NewsGroup.

Figure 6: Percentage loss in F1-Score (%RL) after PCA (Subspace)-based
$\vec{wtv}$ dimensionality reduction on 20NewsGroup.

Table 8: Performance of Convolutional Neural Network (CNN) for multi-class text classification on 20NewsGroup with the original word embeddings ($200$ dimensions) and the reduced word topic vectors ($2000$ dimensions). In $\pm x$, $x$ is the variance across several runs. Embedding | Dimen | Accuracy | Precision | Recall | F1-Score
---|---|---|---|---|---
Word2Vec | 200 |  82.07 $\pm$ 0.003 |  82.22 $\pm$ 0.005 |  81.9 $\pm$ 0.004 |  81.9 $\pm$ 0.005
Reduce Word Topic Vector | 200 | 82.13 $\pm$ 0.002 | 82.36 $\pm$ 0.003 | 82.06 + $\pm$ 0.003 | 82.05 $\pm$ 0.003
Word2Vec | 2000 |  82.31 $\pm$ 0.004 |  82.87 $\pm$ 0.005 |  82.31 $\pm$ 0.004 |  82.38 $\pm$ 0.005
Reduce Word Topic Vector | 2000 | 82.85 $\pm$ 0.001 | 83.18 $\pm$ 0.004 | 82.64 $\pm$ 0.002 | 82.68 $\pm$ 0.003
Table 9: Time and Space complexity analysis of embedding methods. Bold values represent the best results. Method | Vocab |  $\vec{wtv}$ |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
|  $\vec{wtv}$ | Dim |  |  |  |  |  |  |  |
|  $\vec{dv}$ |  |  |  |  |  |  |  |  |
|  Cluster |  |  |  |  |  |  |  |  |
|  Feature |  |  |  |  |  |  |  |  |
|  Predict |  |  |  |  |  |  |  |  |
|  Training |  |  |  |  |  |  |  |  |
|  Model |  |  |  |  |  |  |  |  |
|  $\vec{wtv}$ |  |  |  |  |  |  |  |  |
| Space(MB) |  |  |  |  |  |  |  |  |
SCDV | 15591 | 12000 | 1 | 81 | 242 | 2.56 | 119 | 82 | 1900 | 748
SCDV-MS | 25466 | 12000 | 98 | 74 | 569 | 0.06 | 111 | 79 | 1900 | 71
R-SCDV-MS | 25466 | 2000 | 0 | 0 | 576 | 0.86 | 14 | 66 | 333 | 203
Table 10: PCA based subspace rank reduction criteria. Red-Dim | Subspace Rank Reduction Criteria
---|---
$500$ |  rank $>$ $10$ reduce to $10$ else the original rank
$1000$ |  rank $>$ $20$ reduce to $15$ else the original rank
$2000$ |  rank $>$ $100$ reduce to $30$ else the original rank
$3000$ |  rank $>$ $100$ reduce to $50$ else the original rank
Table 11: Performance on reduced word topic vectors using several reduction techniques on various dimensions for multi-label classification – the Reuters dataset. Method | Dimen | Prec@1 | Prec | nDCG | Cover | LRAP | F1
---|---|---|---|---|---|---|---
|  | nDCG | @5 | @5 | Error |  | Score
| 500 | 92.14 | 36.53 | 48.74 | 8.02 | 91.34 | 79.96
Auto | 1000 | 92.95 | 36.82 | 49.17 | 7.14 | 92.32 | 81.05
Encoder | 2000 | 93.39 | 36.95 | 49.39 | 6.87 | 92.75 | 81.65
| 3000 | 93.56 | 37 | 49.47 | 6.74 | 92.96 | 81.94
| 500 | 91.98 | 36.26 | 48.47 | 7.41 | 91 | 79.03
Random | 1000 | 92.59 | 36.62 | 48.91 | 6.98 | 91.84 | 80.39
Projection | 2000 | 93.39 | 36.83 | 49.26 | 6.95 | 92.59 | 81.12
| 3000 | 93.32 | 36.91 | 49.33 | 6.78 | 92.75 | 81.39
| 500 | 90.73 | 36.03 | 48.06 | 8.40 | 90.11 | 78.06
PCA | 1000 | 92.15 | 36.55 | 48.78 | 7.44 | 91.48 | 80
(SubSpace) | 2000 | 92.95 | 36.87 | 49.22 | 6.86 | 92.30 | 81.2
| 3000 | 93.38 | 36.97 | 49.4 | 6.69 | 92.8 | 81.48
Table 12: Time complexity for dimensionality reduction of word topic vectors
to a $2000$-dimension dense representation using various reduction techniques
on 20NewsGroup

Method |  Random |  |
---|---|---|---
|  PCA |  |
| Autoencoder |  |
Time (sec) | 35 | 66 | 608

Results and Analysis. We observed that our modified embeddings (SCDV-MS) with
Doc2VecC-initialized word vectors, direct thresholding on word cluster
assignments, and multi-sense disambiguation using AdaGram outperforms all
earlier embeddings on both the 20NewsGroup and the Reuters datasets. From
class-wise results in Table 4, we notice a consistent performance improvement
where we are outperforming SCDV in 18 out of 20 classes. It should be noted
that the improvement on Reuters is not as great as the 20NewsGroup dataset due
to the fact that the number of unique polysemic words in Reuters ($250$) is
significantly fewer than 20NewsGroup ($1000$); thus each word is assigned to
only one cluster. Therefore, the use of AdaGram for sense disambiguation and
the sparsity operation over the word-cluster assignments does not improve the
performance by a large margin. We verified this claim in the ablation study
below. We can conclude that our modifications yield notable improvements if
the dataset has more multi-sense words.

Ablation Study:  To understand the contributions of each of the three
modifications, we compared five different versions of our embeddings. In the
first version, we ablated the sparsity of the word-cluster assignments and
applied sparsity directly on the document vectors similar to SCDV while
keeping the Doc2VecC multi-sense word embeddings and the sense annotated
corpus intact. In the second version, we ablated the Doc2VecC embeddings with
normal SGNS embedding while keeping the word topic vector sparsity, and the
sense annotated corpus intact. In the third version, we ablated multi-sense
embeddings and the annotation of the corpus while keeping the Doc2VecC word
training and the word topic vector sparsity intact. We also compared our
results with an all ablation approach, i.e., the SCDV baseline and a none
ablation approach, i.e., our new embeddings in SCDV-MS. Table 6 shows the
results obtained with ablation on 20NewsGroup and Reuters datasets. We
obtained the best performance with the none ablation approach, i.e., SCDV-MS.
Thus, we can conclude that all three modifications is needed to yield the best
performance. Multi-sense is the most pivotal improvement for 20newsgroup since
by ablating it we observed the lowest performance out of ablating each of the
three modifications. Whereas on the Reuters dataset, multi-sense was the least
important because of fewer multi-sense words. On Reuters, the noise removal at
word level representations was the most important.

Comparison with Contextual Embeddings:  SCDV-MS is a lot simpler than
unsupervised contextual embeddings like BERT (pr) and ELMo, but it
outperformed them. We presume that SCDV-MS’s concentration on capturing
semantics (local and global) in sparse high dimension representations instead
of capturing both semantics and syntax in single lower dimensional continuous
representations (what BERT does) is the reason behind our method’s superior
performance. Because understanding syntax is not as influential as semantics
in our classification and similarity tasks.

###  5.2 wtv’s Lower Dimensional Manifold Learning

We tried three popular lower-dimensional manifold learning algorithms to
reduce high dimensional sparse word topic vectors to lower-dimensional word
embeddings, namely Random Projection [2], PCA (Subspace) [1], and Autoencoders
[47]. For PCA (Subspace), we observed that not all subspaces ($200$
dimensions) of word topic vectors have a complete full rank ($200$). Most of
the subspaces were of ranks much smaller than $200$ (rank $\leq$ $200$).
Therefore, we applied PCA on each of $200$ dimension subspaces separately and
concatenated the subspace reduced vectors. We refer to Table 10 as the
criteria for PCA-based subspace rank reduction. For Auto-encoders, we used the
standard architecture for reducing the word topic vectors from $12000$ to
$2000$ dimensions, through a intermediate layer of $4000$ dimensions (see
Figure 7). We used the mean squared error minimization, $\tanh$ non-linear
activation, and Adam optimization routine for training the autoencoders. We
used the initial learning rate of $0.001$, $\beta$${}_{1}=0.9$ and
$\beta$${}_{2}=0.999$ for Adam optimization routine.

Figure 7: The AutoEncoder architecture ($12000-4000-2000-4000-12000$) used to
reduce the word topic vectors from $12000$ dimensions to $2000$ dimensions.

Results: Table 5.1 shows the performance of the dimensionality reduction
techniques such as Random Projection, PCA (Subspace), and Autoencoders on the
20NewsGroup dataset. We observed that autoencoders outperform other reduction
methods because they easily fit any non-linear function. Also, we observed
only a small decrease of $\leavevmode\nobreak\ 1\%$ in the performance after a
reduction to $2000$ dimensions with most methods. This decrease is associated
to loss due to data compression and can be explained by information theory.
However, word topic vectors can be efficiently projected into a lower-
dimensional manifold without a significant loss in the performance. We
compared the percentage loss in performance (F1-Score) %RL =
$\frac{\text{Orignal}-\text{Reduced}}{\text{Orignal}}$ on text classification
with a dimension-reduced $\vec{wtv}$ through random projection for both SCDV
and SCDV-MS on 20newsgroup. In Figures 6, 6 and 6 we observe that the $\%RL$
loss in SCDV-MS’s F1-Score is distinctively less compared to SCDV for all
reduction methods. Furthermore, we observed that the reduction time for SCDV-
MS was shorter than SCDV, particularly for random projection, because SCDV-MS
$\vec{wtv}$ are sparser than SCDV $\vec{wtv}$. We also tried a direct
reduction of final document representations which yielded a poor performance
and took a longer reduction time for both embeddings. Overall, reducing SCDV-
MS $\vec{wtv}$ is much more effective than reducing SCDV $\vec{wtv}$ or
document vectors. Similar observations for a multi-label classification task
on the Reuters dataset, see Table 11.

Application to Deep Learning: One significance of our reduction of $\vec{wtv}$
is that they can be used as direct word embeddings in popular deep learning
architectures such as CNNs on downstream classification tasks. We used the
same architecture (see the supplementary material, Figure 8 111111
https://bit.ly/33473wk) for both embeddings. Employing CNN, our results
outperformed the original word embeddings of the same dimension for the
20NewsGroup classification task, shown in Table 8.

###  5.3 Time and Space Complexity

Table 5.1 illustrates empirical results for time and space complexities on
SCDV ($12000$ dimensions), SCDV-MS ($12000$ dimensions) and reduced R-SCDV-MS
($2000$ dimensions).

Feature Formation Time: Due to the direct thresholding of word cluster
assignments in SCDV-MS, the word topic vectors ($\vec{wtv}$) are extremely
sparse. SCDV-MS $\vec{wtv}$ has only $2\%$ of active attributes ($98\%$
sparse), whereas SCDV $\vec{wtv}$ has $99\%$ of active attributes (only $1\%$
sparse). Therefore, we can use an efficient sparse operation (sparse addition
and multiplication) over sparse vectors to speedup feature formation. We
observed that by adding sparsity we can reduce the feature formation time by a
significant factor of $43$.

Overall Prediction Time: Overall, due to enhanced feature formation and
reduced $\vec{wtv}$ loading time, SCDV-MS will predict faster compared to the
original SCDV. However, we observed an insignificant difference in the
prediction time and model size as SCDV sparsifies the final document vectors
(both equally sparse). Furthermore, after reducing the SCDV-MS $\vec{wtv}$ to
$2000$-dense dimension features using auto-encoders, we observed a distinctive
reduction of $8.5$ times in the prediction time. One can directly store the
reduced $\vec{wtv}$ for the complete vocabulary instead of the reduction
model. Refer to Table 5.1 for SCDV-MS $\vec{wtv}$ reduction timing.
Furthermore, one can directly also reduce the words appearing in the documents
i.e. use a real-time reduction model during prediction.

Vector Space Complexity, Training Time, and Model Size: We only require $0.1$
of the original space to store sparse $\vec{wtv}$. We achieved these
improvements despite having $1.63$ times of the size of the original
vocabulary due to multi-sense word embeddings. The projected $\vec{wtv}$ is
$3$ times larger than the $\vec{wtv}$ of SCDV-MS; however, it is $3.7$ times
smaller than the $\vec{wtv}$ in SCDV. SCDV is marginally ($1.1$ times) sparser
than SCDV-MS due to manual thresholding of document vectors. SCDV-MS also has
a slightly faster training time compared to the original SCDV. We also
observed that our training model on reduced vectors (R-SCDV-MS) is $6$ times
smaller than SCDV, and the training process of our classifier is $1.25$ times
faster than SCDV.

##  6 Conclusion

In this paper, we proposed several novel modifications to overcome the
shortcomings of SCDV, a state-of-the-art document embedding method. Our
proposed SCDV-MS, outperformed the previous embedding (including SCDV and BERT
(pr)) on the downstream tasks of text classification. Overall, we have shown
that disambiguating multi-sense words based on context words (adjacent words)
can lead to better document representations. Sparsity in representation is
helpful for effective and efficient lower-dimensional manifold representation
learning. Representation noise in words level can have a significant impact on
the downstream tasks.

## References

  * [1] Hervé Abdi and Lynne J Williams, ‘Principal component analysis’, Wiley interdisciplinary reviews: computational statistics, (2010).
  * [2] Dimitris Achlioptas, ‘Database-friendly random projections: Johnson-lindenstrauss with binary coins’, Journal of computer and System Sciences, (2003).
  * [3] Hadi Amiri and Mitra Mohtarami, ‘Vector of locally aggregated embeddings for text representation’, NAACL 2019, (2019).
  * [4] Sanjeev Arora, Yingyu Liang, and Tengyu Ma, ‘A simple but tough-to-beat baseline for sentence embeddings’, International Conference of Learning Representation, (2017).
  * [5] Ben Athiwaratkun, Andrew Gordon Wilson, and Anima Anandkumar, ‘Probabilistic fasttext for multi-sense word embeddings’, arXiv preprint arXiv:1806.02901, (2018).
  * [6] Sergey Bartunov, Dmitry Kondrashkin, Anton Osokin, and Dmitry Vetrov, ‘Breaking sticks and ambiguities with adaptive skip-gram’, in Artificial Intelligence and Statistics, (2016).
  * [7] Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateek Jain, ‘Sparse local embeddings for extreme multi-label classification’, in NeurIPS, (2015).
  * [8] David M. Blei, David M. Ng, Michael I. Jordan, and John Lafferty, ‘Latent dirichlet allocation’, Journal of Machine Learning Research, 3, 2003, (2003).
  * [9] Minmin Chen, ‘Efficient vector representation for documents through corruption’, 5th International Conference on Learning Representations, (2017).
  * [10] Dat Quoc Nguyen Dai Quoc Nguyen, Ashutosh Modi, Stefan Thater, and Manfred Pinkal, ‘A mixture model for learning multi-sense word embeddings’, ACL 2017, (2017).
  * [11] Rajarshi Das, Manzil Zaheer, and Chris Dyer, ‘Gaussian lda for topic models with word embeddings’, in Proceedings of The 53th Annual Meeting of the Association of Computational Linguistic (ACL). Association of Computational Linguistic, (2015).
  * [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, ‘Bert: Pre-training of deep bidirectional transformers for language understanding’, arXiv preprint arXiv:1810.04805, (2018).
  * [13] Adji B Dieng, Francisco JR Ruiz, and David M Blei, ‘The dynamic embedded topic model’, arXiv preprint arXiv:1907.05545, (2019).
  * [14] Adji B Dieng, Francisco JR Ruiz, and David M Blei, ‘Topic modeling in embedding spaces’, arXiv preprint arXiv:1907.04907, (2019).
  * [15] Xianghua Fu, Ting Wang, Jing Li, Chong Yu, and Wangwang Liu, ‘Improving distributed word representation and topic model by word-topic mixture model’, in Proceedings of The 8th Asian Conference on Machine Learning, (2016).
  * [16] Felix A Gers, Nicol N Schraudolph, and Jürgen Schmidhuber, ‘Learning precise timing with lstm recurrent networks’, Journal of machine learning research, 3(Aug), (2002).
  * [17] Vivek Gupta, Harish Karnick, Ashendra Bansal, and Pradhuman Jhala, ‘Product classification in e-commerce using distributional semantics’, in COLING 2016, (2016).
  * [18] Vivek Gupta, Ankit Kumar Saw, Partha Pratim Talukdar, and Praneeth Netrapalli.  Unsupervised document representation using partition word-vectors averaging, 2019.
  * [19] Zellig Harris, ‘Distributional structure’, Word, 10, 146–162, (1954).
  * [20] Peng Jin, Yue Zhang, Xingyuan Chen, and Yunqing Xia, ‘Bag-of-embeddings for text classification’, in International Joint Conference on Artificial Intelligence, number 25, (2016).
  * [21] Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom, ‘A convolutional neural network for modelling sentences’, in Proceedings of the 52nd Annual Meeting of the ACL, volume 1, (2014).
  * [22] Han Kyul Kim, Hyunjoong Kim, and Sungzoon Cho, ‘Bag-of-concepts: Comprehending document representation through clustering words in distributed representation’, Neurocomputing, (2017).
  * [23] Yoon Kim, ‘Convolutional neural networks for sentence classification’, in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), (2014).
  * [24] Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler, ‘Skip thought vectors’, NeurIPS, (2015).
  * [25] Jarvan Law, Hankz Hankui Zhuo, Junhua He, and Erhu Rong, ‘Ltsg: Latent topical skip-gram for mutually learning topic model and vector representations’, arXiv preprint arXiv:1702.07117, (2017).
  * [26] Quoc V Le and Tomas Mikolov, ‘Distributed representations of sentences and documents.’, in ICML, volume 14, (2014).
  * [27] Jiwei Li and Dan Jurafsky, ‘Do multi-sense embeddings improve natural language understanding?’, arXiv preprint arXiv:1506.01070, (2015).
  * [28] Shaohua Li, Tat-Seng Chua, Jun Zhu, and Chunyan Miao, ‘Generative topic embedding: a continuous representation of documents’, in Proceedings of The 54th Annual Meeting of the ACL (ACL). Association of Computational Linguistic, (2016).
  * [29] Ximing Li, Jinjin Chi, Changchun Li, Jihong Ouyang, and Bo Fu, ‘Integrating topic modeling with word embeddings by mixtures of vmfs’, COLING 2016, (2016).
  * [30] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang, ‘Learning context-sensitive word embeddings with neural tensor skip-gram model.’, in Proceedings of IJCAI 2014, the 24th International Joint Conference on Artificial Intelligence, (2015).
  * [31] Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun, ‘Topical word embeddings.’, in Proceedings of AAAI 2015, (2015).
  * [32] Dheeraj Mekala, Vivek Gupta, Bhargavi Paranjape, and Harish Karnick, ‘Scdv: Sparse composite document vectors using soft clustering over distributional representations’, in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, (2017).
  * [33] Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur, ‘Recurrent neural network based language model’, in Eleventh Annual Conference of the International Speech Communication Association, (2010).
  * [34] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean, ‘Distributed representations of words and phrases and their compositionality’, in NeurIPS, (2013).
  * [35] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig, ‘Linguistic regularities in continuous space word representations.’, in HLT-NAACL, (2013).
  * [36] Jeff Mitchell and Mirella Lapata, ‘Vector-based models of semantic composition’, in Proceedings of ACL-08: HLT, Columbus, Ohio, (June 2008). ACL.
  * [37] Jeff Mitchell and Mirella Lapata, ‘Composition in distributional models of semantics’, Cognitive science, (2010).
  * [38] Christopher E Moody, ‘Mixing dirichlet topic models and word embeddings to make lda2vec’, arXiv preprint arXiv:1605.02019, (2016).
  * [39] Dat Quoc Nguyen, Richard Billingsley, Lan Du, and Mark Johnson, ‘Improving topic models with latent feature word representations’, Transactions of the ACL, 3, (2015).
  * [40] Liqiang Niu, Xinyu Dai, Jianbing Zhang, and Jiajun Chen, ‘Topic2vec: learning distributed representations of topics’, in 2015 International Conference on Asian Language Processing (IALP). IEEE, (2015).
  * [41] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer, ‘Deep contextualized word representations’, in Proc. of NAACL, (2018).
  * [42] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer, ‘Deep contextualized word representations’, in Proc. of NAACL, (2018).
  * [43] Andreas Rücklé, Steffen Eger, Maxime Peyrard, and Iryna Gurevych, ‘Concatenated $p$-mean word embeddings as universal cross-lingual sentence representations’, arXiv preprint arXiv:1803.01400, (2018).
  * [44] Pranjal Singh and Amitabha Mukerjee, ‘Words are not equal: Graded weighting model for building composite document vectors’, in ICON-2015, (2015).
  * [45] Paul Smolensky, ‘Tensor product variable binding and the representation of symbolic structures in connectionist systems’, Artificial intelligence, (1990).
  * [46] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts, et al., ‘Recursive deep models for semantic compositionality over a sentiment treebank’, in Proceedings of the conference on empirical methods in natural language processing (EMNLP), (2013).
  * [47] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol, ‘Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion’, Journal of machine learning research, (Dec), (2010).
  * [48] John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu, ‘Towards universal paraphrastic sentence embeddings’, International Conference of Learning Representation, (2016).
  * [49] John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Dan Roth, ‘From paraphrase database to compositional paraphrase model and back’, Transactions of the ACL, 3, (2015).