# Self-Attention: A Better Building Block for Sentiment Analysis Neural
Network Classifiers

Artaches Ambartsoumian
School of Computing Science
Simon Fraser University
Burnaby, BC, CANADA
<EMAIL_ADDRESS>
&Fred Popowich
School of Computing Science
Simon Fraser University
Burnaby, BC, CANADA
<EMAIL_ADDRESS>

###### Abstract

Sentiment Analysis has seen much progress in the past two decades. For the
past few years, neural network approaches, primarily RNNs and CNNs, have been
the most successful for this task. Recently, a new category of neural
networks, self-attention networks (SANs), have been created which utilizes the
attention mechanism as the basic building block. Self-attention networks have
been shown to be effective for sequence modeling tasks, while having no
recurrence or convolutions. In this work we explore the effectiveness of the
SANs for sentiment analysis. We demonstrate that SANs are superior in
performance to their RNN and CNN counterparts by comparing their
classification accuracy on six datasets as well as their model characteristics
such as training speed and memory consumption. Finally, we explore the effects
of various SAN modifications such as multi-head attention as well as two
methods of incorporating sequence position information into SANs.

##  1 Introduction

Sentiment analysis, also know as opinion mining, deals with determining the
opinion classification of a piece of text. Most commonly the classification is
whether the writer of a piece of text is expressing a position or negative
attitude towards a product or a topic of interest. Having more than two
sentiment classes is called fine-grained sentiment analysis with the extra
classes representing intensities of positive/negative sentiment (e.g. very-
positive) and/or the neutral class. This field has seen much growth for the
past two decades, with many applications and multiple classifiers proposed
Mäntylä et al. (2018). Sentiment analysis has been applied in areas such as
social media Jansen et al. (2009), movie reviews Pang et al. (2002), commerce
Jansen et al. (2009), and health care Greaves et al. (2013b) Greaves et al.
(2013a).

In the past few years, neural network approaches have consistently advanced
the state-of-the-art technologies for sentiment analysis and other natural
language processing (NLP) tasks. For sentiment analysis, the neural network
approaches typically use pre-trained word embeddings such as word2vec Mikolov
et al. (2013) or GloVePennington et al. (2014) for input, which get processed
by the model to create a sentence representation that is finally used for a
softmax classification output layer. The main neural network architectures
that have been applied for sentiment analysis are recurrent neural
networks(RNNs) Tai et al. (2015) and convolutional neural networks (CNNs) Kim
(2014), with RNNs being more popular of the two. For RNNs, typically gated
cell variants such as long short-term memory (LSTM) Hochreiter and Schmidhuber
(1997), Bi-Directional LSTM (BiLSTM) Schuster and Paliwal (1997), or gated
recurrent unit (GRU) Cho et al. (2014) are used.

Most recently, Vaswani et al. Vaswani et al. (2017) introduced the first
fully-attentional architecture, called Transformer, which utilizes only the
self-attention mechanism and demonstrated its effectiveness on neural machine
translation (NMT). The Transformer model achieved state-of-the-art performance
on multiple machine translation datasets, without having recurrence or
convolution components. Since then, self-attention networks have been
successfully applied to a variety of tasks, including: image classification
Parmar et al. (2018), generative adversarial networks Zhang et al. (2018),
automatic speech recognition Povey et al. (2018), text summarization Liu et
al. (2018), semantic role labeling Strubell et al. (2018), as well as natural
language inference and sentiment analysis Shen et al. (2018).

In this paper we demonstrate that self-attention is a better building block
compared to recurrence or convolutions for sentiment analysis classifiers. We
extend the work of Barnes et al. (2017) by exploring the behaviour of various
self-attention architectures on six different datasets and making direct
comparisons to their work. We set our baselines to be their results for LSTM,
BiLSTM, and CNN models, and used the same code for dataset pre-processing,
word embedding imports, and batch construction. Finally, we explore the
effectiveness of SAN architecture variations such as different techniques of
incorporating positional information into the network, using multi-head
attention, and stacking self-attention layers. Our results suggest that
relative position representations is superior to positional encodings, as well
as highlight the efficiency of the stacking self-attention layers.

Source code is publicly available111https://github.com/Artaches/SSAN-self-
attention-sentiment-analysis-classification.

##  2 Background

The attention mechanism was introduced by Bahdanau et al. (2014) to improve
the RNN encoder-decoder sequence-to-sequence architecture for NMT Sutskever et
al. (2014). Since then, it has been extensively used to improve various RNN
and CNN architectures (Cheng et al. (2016); Kokkinos and Potamianos (2017); Lu
et al. (2016)). The attention mechanism has been an especially popular
modification for RNN-based architectures due to its ability to improve the
modeling of long range dependencies (Daniluk et al. (2017); Zhou et al.
(2018)).

###  2.1 Attention

Originally Bahdanau et al. (2014) described attention as the process of
computing a context vector for the next decoder step that contains the most
relevant information from all of the encoder hidden states by performing a
weighted average on the encoder hidden states. How much each encoder state
contributes to the weighted average is determined by an alignment score
between that encoder state and previous hidden state of the decoder.

More generally, we can consider the previous decoder state as the query
vector, and the encoder hidden states as key and value vectors. The output is
a weighted average of the value vectors, where the weights are determined by
the compatibility function between the query and the keys. Note that the keys
and values can be different sets of vectors Vaswani et al. (2017).

The above can be summarized by the following equations. Given a query $q$,
values $(v_{1},...,v_{n})$, and keys $(k_{1},...,k_{n})$ we compute output
$z$:

$z=\sum_{j=1}^{n}\alpha_{j}(v_{j})$ (1)

$\alpha_{j}=\frac{\exp{f({k_{j},q})}}{\sum_{i=1}^{n}\exp{f({k_{i},q})}}$ (2)

$\alpha_{j}$ is computed using the softmax function where $f(k_{i},q)$ is the
compatibility score between $k_{i}$ and $q$,

For the compatibility function, we will be using using the scaled dot-product
function from Vaswani et al. (2017):

$f(k,q)=\frac{(k)(q)^{T}}{\sqrt{d_{k}}}$ (3)

where $d_{k}$ is the dimension of the key vectors. This scaling is done to
improve numerical stability as the dimension of keys, values, and queries
grows.

###  2.2 Self-Attention

Self-attention is the process of applying the attention mechanism outlined
above to every position of the source sequence. This is done by creating three
vectors (query, key, value) for each sequence position, and then applying the
attention mechanism for each position $x_{i}$, using the $x_{i}$ query vector
and key and value vectors for all other positions. As a result, an input
sequence $X=(x_{1},x_{2},...,x_{n})$ of words is transformed into a sequence
$Y=(y_{1},y_{2},...,y_{n})$ where $y_{i}$ incorporates the information of
$x_{i}$ as well as how $x_{i}$ relates to all other positions in $X$. The
(query, key, value) vectors can be created by applying learned linear
projections Vaswani et al. (2017), or using feed-forward layers.

This computation can be done for the entire source sequence in parallel by
grouping the queries, keys, and values in Q, K, V matricesVaswani et al.
(2017).

$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V$ (4)

Furthermore, instead of performing self-attention once for (Q,K,V) of
dimension $d_{\text{model}}$, Vaswani et al. (2017) proposed multi-head
attention, which performs attention $h$ times on projected (Q,K,V) matrices of
dimension $d_{\text{model}}/h$. For each head, the (Q,K,V) matrices are
uniquely projected to dimension $d_{\text{model}}/h$ and self-attetnion is
performed to yield an output of dimension $d_{\text{model}}/h$. The outputs of
each head are then concatenated, and once again a linear projection layer is
applied, resulting in an output of same dimensionality as performing self-
attention once on the original (Q,K,V) matrices. This process is described by
the following formulas:

$\displaystyle\mathrm{MultiHead}(Q,K,V)=\hskip 71.13188pt$
$\displaystyle\mathrm{Concat}(\mathrm{head_{1}},...,\mathrm{head_{h}})W^{O}$
(5) $\displaystyle\text{where}~{}\mathrm{head_{i}}=\hskip 99.58464pt$
$\displaystyle\mathrm{Attention}(QW^{Q}_{i},KW^{K}_{i},VW^{V}_{i})$ (6)

Where the projections are parameter matrices
$W^{Q}_{i}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}$,
$W^{K}_{i}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}$,
$W^{V}_{i}\in\mathbb{R}^{d_{\text{model}}\times d_{v}}$ and
$W^{O}\in\mathbb{R}^{hd_{v}\times d_{\text{model}}}$.

###  2.3 Position Information Techniques

The attention mechanism is completely invariant to sequence ordering, thus
self-attention networks need to incorporate positional information. Three main
techniques have been proposed to solve this problem: adding sinusoidal
positional encodings or learned positional encoding to input embeddings, or
using relative positional representations in the self-attention mechanism.

####  2.3.1 Sinusoidal Position Encoding

This method was proposed by Vaswani et al. (2017) to be used for the
Transformer model. Here, positional encoding ($PE$) vectors are created using
sine and cosine functions of difference frequencies and then are added to the
input embeddings. Thus, the input embeddings and positional encodings must
have the same dimensionality of ${d_{\text{model}}}$. The following sine and
cosine functions are used:

$\displaystyle PE_{(pos,2i)}=sin(pos/10000^{2i/d_{\text{model}}})$
$\displaystyle PE_{(pos,2i+1)}=cos(pos/10000^{2i/d_{\text{model}}})$

where $pos$ is the sentence position and $i$ is the dimension. Using this
approach, sentences longer than those seen during training can still have
positional information added. We will be referring to this method as $PE$.

####  2.3.2 Learned Position Encoding

In a similar method, learned vectors of the same dimensionality, that are also
unique to each position can be added to the input embeddings instead of
sinusoidal position encodingsGehring et al. (2017). There are two downsides to
this approach. First, this method cannot handle sentences that are longer than
the ones in the training set as no vectors are trained for those positions.
Second, the further position will likely not get trained as well if the
training dataset has more short sentences than longer ones. Vaswani et al.
(2017) also reported that these perform identically to the positional encoding
approach.

####  2.3.3 Relative Position Representations

Relative Position Representations ($RPR$) was introduced by Shaw et al. (2018)
as a replacement of positional encodings for the Transformer. Using this
approach, the Transformer was able to perform even better for NMT. Out of the
three discussed, we have found this approach to work best and we will be
referring to this method as $RPR$ throughout the paper.

For this method, the self-attention mechanism is modified to explicitly learn
the relative positional information between every two sequence positions. As a
result, the input sequence is modeled as a labeled, directed, fully-connected
graph, where the labels represent positional information. A tunable parameter
$k$ is also introduced that limits the maximum distance considered between two
sequence positions. Shaw et al. (2018) hypothesized that this will allow the
model to generalize to longer sequences at test time.

Figure 1: SSAN Model Architecture

##  3 Proposed Architectures

In this work we propose a simple self-attention (SSAN) model and test it in 1
as well as 2 layer stacked configurations. We designed the SSAN architecture
to not have any extra components in order to compare specifically the self-
attention component to the recurrence and convolution components of LSTM and
CNN models. Our goal is to test the effectiveness of the main building blocks.
We compare directly the results of two proposed architectures, 1-Layer-SSAN
and 2-Layer-SSAN, to the LSTM, BiLSTM, and CNN architectures from Barnes et
al. (2017).

SSAN performs self-attention only once, which is identical to 1-head multi-
head attention. SSAN takes in input word embeddings and applies 3 feed-forward
layers to obtain Q,K,V representations on which self-attention is performed.
The output of the self-attention layer is passed through another feed-forward
layer. This process is done twice for 2-Layer-SSAN, using the output of first
layer as input for the second. The output of the last self-attention layer is
averaged and a feed-forward layer is then applied to create a sentence
representation vector of fixed dimension $d_{\text{model}}$. Finally, the
sentence representation vector is passed through an output softmax layer that
has an output dimension of $d_{\text{classes}}$. Dropout Srivastava et al.
(2014) is applied on input word embeddings, output of self-attention layers,
on the sentence representation vector. The architecture is visualized in
Figure 1. All feed-forward layers use ReLU Nair and Hinton (2010) activation
functions. For relative positional representation, we set the parameter
$k$=10, which is the maximum relative position considered for each input
sequence position.

Finally, we also show results for other, more complex, self-attention
architectures that are based on the Transformer. We take a 2 layer Transformer
encoder as described by Vaswani et al. (2017), then just like for SSAN,
average the output of the second layer to create a sentence representation and
apply a feed-forward layer followed by an output softmax layer. Dropout is
applied as described in Vaswani et al. (2017) as well as on the sentence
representation vector.

##  4 Experiments

To reduce implementation deviations from previous work, we use the codebase
from Barnes et al. (2017) and only replace the model and training process. We
re-use the code for batch pre-processing and batch construction for all
datasets, accuracy evaluation, as well as use the same word
embeddings222https://github.com/jbarnesspain/sota_sentiment. All neural
network models use cross-entropy for the training loss.

All experiments and benchmarks were run using a single GTX 1080 Ti with an i7
5820k @ 3.3Ghz and 32Gb of RAM. For model implementations: LSTM, BiLSTM, and
CNN baselines are implemented in Keras_2.0.8 Chollet et al. (2015) with
Tensorflow_1.7 backend using cuDNN_5.1.5 and CUDA_9.1. All self-attention
models are implemented in Tensorflow_1.7 and use the same CUDA libraries.

|  Train |  Dev. |  Test |  # of Classes |  Average Sent. Length |  Max Sent. Length |  Vocab. Size |  Wiki Emb. Coverage |  300D Emb. Coverage
---|---|---|---|---|---|---|---|---|---
SST-fine | 8,544 | 1,101 | 2,210 | 5 | 19.53 | 57 | 19,500 | 94.4% | 89.0%
SST-binary | 6,920 | 872 | 1,821 | 2 | 19.67 | 57 | 17,539 | 95.0% | 89.6%
OpeNER | 2,780 | 186 | 743 | 4 | 4.28 | 23 | 2,447 | 94.2% | 99.3%
SenTube-A | 3,381 | 225 | 903 | 2 | 28.54 | 127 | 18,569 | 75.6% | 74.5%
SenTube-T | 4,997 | 333 | 1,334 | 2 | 28.73 | 121 | 20,276 | 70.4% | 76.0%
SemEval | 6,021 | 890 | 2,376 | 3 | 22.40 | 40 | 21,163 | 77.1% | 99.8%
Table 1: Modified Table 2 from Barnes et al. (2017). Dataset statistics,
embedding coverage of dataset vocabularies, as well as splits for Train, Dev
(Development), and Test sets. The ’Wiki’ embeddings are the 50, 100, 200, and
600 dimension used for experiments.

###  4.1 Datasets

In order to determine if certain neural network building blocks are superior,
we test on six datasets from Barnes et al. (2017) with different properties.
The summary for dataset properties is in Table 1.

The Stanford Sentiment Treebank (SST-fine) Socher et al. (2013) deals with
movie reviews, containing five classes [very-negative, negative, neutral,
positive, very-positive]. (SST-binary) is constructed from the same data,
except the neutral class sentences are removed, all negative classes are
grouped, and all positive classes are grouped. The datasets are pre-processed
to only contain sentence-level labels, and none of the models reported in this
work utilize the phrase-level labels that are also provided.

The OpeNER dataset Agerri et al. (2013) is a dataset of hotel reviews with
four sentiment classes: very negative, negative, positive, and very positive.
This is the smallest dataset with the lowest average sentence length.

The SenTube datasets Uryupina et al. (2014) consist of YouTube comments with
two sentiment classes: positive and negative. These datasets contain the
longest average sentence length as well as the longest maximum sentence length
of all the datasets.

The SemEval Twitter dataset (SemEval) Nakov et al. (2013) consists of tweets
with three classes: positive, negative, and neutral.

|  Model |  Dim. |  SST-fine |  SST-binary |  OpeNER |  SenTube-A |  SenTube-T |  SemEval |  Macro-Avg.
---|---|---|---|---|---|---|---|---|---
Baselines | Bow |  | 40.3 | 80.7 | 77.1 | 60.6 | 66.0 | 65.5 | 65.0
Ave | 300 | 41.6 | 80.3 | 76.3 | 61.5 | 64.3 | 63.6 | 64.6
LSTM | 50 | 43.3 (1.0) | 80.5 (0.4) | 81.1 (0.4) | 58.9 (0.8) | 63.4 (3.1) | 63.9 (1.7) | 65.2 (1.2)
100 | 44.1 (0.8) | 79.5 (0.6) | 82.4 (0.5) | 58.9 (1.1) | 63.1 (0.4) | 67.3 (1.1) | 65.9 (0.7)
200 | 44.1 (1.6) | 80.9 (0.6) | 82.0 (0.6) | 58.6 (0.6) | 65.2 (1.6) | 66.8 (1.3) | 66.3 (1.1)
300 | 45.3 (1.9) | 81.7 (0.7) | 82.3 (0.6) | 57.4 (1.3) | 63.6 (0.7) | 67.6 (0.6) | 66.3 (1.0)
600 | 44.5 (1.4) | 83.1 (0.9) | 81.2 (0.8) | 57.4 (1.1) | 65.7 (1.2) | 67.5 (0.7) | 66.5 (1.0)
BiLSTM | 50 | 43.6 (1.2) | 82.9 (0.7) | 79.2 (0.8) | 59.5 (1.1) | 65.6 (1.2) | 64.3 (1.2) | 65.9 (1.0)
100 | 43.8 (1.1) | 79.8 (1.0) | 82.4 (0.6) | 58.6 (0.8) | 66.4 (1.4) | 65.2 (0.6) | 66.0 (0.9)
200 | 44.0 (0.9) | 80.1 (0.6) | 81.7 (0.5) | 58.9 (0.3) | 63.3 (1.0) | 66.4 (0.3) | 65.7 (0.6)
300 | 45.6 (1.6) | 82.6 (0.7) | 82.5 (0.6) | 59.3 (1.0) | 66.2 (1.5) | 65.1 (0.9) | 66.9 (1.1)
600 | 43.2 (1.1) | 83.0 (0.4) | 81.5 (0.5) | 59.2 (1.6) | 66.4 (1.1) | 68.5 (0.7) | 66.9 (0.9)
CNN | 50 | 39.9 (0.7) | 81.7 (0.3) | 80.0 (0.9) | 55.2 (0.7) | 57.4 (3.1) | 65.7 (1.0) | 63.3 (1.1)
100 | 40.1 (1.0) | 81.6 (0.5) | 79.5 (0.9) | 56.0 (2.2) | 61.5 (1.1) | 64.2 (0.8) | 63.8 (1.1)
200 | 39.1 (1.1) | 80.7 (0.4) | 79.8 (0.7) | 56.3 (1.8) | 64.1 (1.1) | 65.3 (0.8) | 64.2 (1.0)
300 | 39.8 (0.7) | 81.3 (1.1) | 80.3 (0.9) | 57.3 (0.5) | 62.1 (1.0) | 63.5 (1.3) | 64.0 (0.9)
600 | 40.7 (2.6) | 82.7 (1.2) | 79.2 (1.4) | 56.6 (0.6) | 61.3 (2.0) | 65.9 (1.8) | 64.4 (1.5)
Self-Attention Models |  1-Layer SSAN + RPR | 50 | 42.8 (0.8) | 79.6 (0.3) | 78.6 (0.5) |  64.1 (0.4) | 67.0 (1.0) | 67.1 (0.5) | 66.5 (0.6)
100 | 44.6 (0.3) | 82.3 (0.3) | 81.6 (0.5) | 61.6 (1.3) | 68.6 (0.6) | 68.6 (0.5) | 67.9 (0.6)
200 | 45.4 (0.4) | 83.1 (0.5) | 82.3 (0.4) | 62.2 (0.6) | 68.4 (0.8) | 70.5 (0.4) | 68.6 (0.5)
300 |  48.1 (0.4) |  84.2 (0.4) | 83.8 (0.2) | 62.5 (0.3) | 68.4 (0.8) |  72.2 (0.8) |  69.9 (0.5)
600 | 47.7 (0.7) | 83.6 (0.4) | 83.1 (0.4) | 62.0 (0.4) |  68.8 (0.7) | 70.5 (0.8) | 69.2 (0.5)
2-Layer SSAN + RPR | 50 | 43.2 (0.9) | 79.8 (0.2) | 79.2 (0.6) | 63.0 (1.3) | 66.6 (0.5) | 67.5 (0.7) | 66.5 (0.7)
100 | 45.0 (0.4) | 81.6 (0.9) | 81.1 (0.4) | 63.3 (0.7) | 67.7 (0.5) | 68.7 (0.4) | 67.9 (0.5)
200 | 46.5 (0.7) | 82.8 (0.5) | 82.3 (0.6) | 61.9 (1.2) | 68.0 (0.8) | 69.6 (0.8) | 68.5 (0.8)
300 |  48.1 (0.8) | 83.8 (0.9) | 83.3 (0.9) | 62.1 (0.8) | 67.8 (1.0) | 70.7 (0.5) | 69.3 (0.8)
600 | 47.6 (0.5) | 83.7 (0.4) | 82.9 (0.5) | 60.7 (1.4) | 68.2 (0.7) | 70.3 (0.3) | 68.9 (0.6)
Transformer Encoder + RPR | 300 | 47.3 (0.4) | 83.8 (0.4) | 84.2 (0.5) | 62.0 (1.4) | 68.2 (1.6) | 72.0 (0.5) | 69.6 (0.8)
Transformer Encoder + PE | 300 | 45.0 (0.7) | 82.0 (0.6) | 83.3 (0.7) | 62.3 (2.4) | 66.9 (0.8) | 68.4 (0.8) | 68.0 (1.0)
1-Layer SSAN | 300 | 47.2 (0.5) | 83.9 (0.7) | 83.6 (0.6) | 62.1 (2.5) | 68.7 (1.0) | 70.2 (1.2) | 69.3 (1.1)
1-Layer SSAN + PE | 300 | 45.0 (0.3) | 82.9 (0.2) | 80.7 (0.6) | 62.6 (2.3) | 67.8 (0.4) | 69.1 (0.3) | 68.0 (0.7)
Table 2: Modified Table 3 from Barnes et al. (2017). Test accuracy averages and standard deviations (in brackets) of 5 runs. The baseline results are taken from Barnes et al. (2017); the self-attention models results are ours. Best model for each dataset is given in bold. Model | # of Parameters | GPU VRAM Usage (MB) |  Training Time (s) |  Inference Time (s)
---|---|---|---|---
LSTM | 722,705 | 419MB | 235.9s | 7.6s
BiLSTM | 1,445,405 | 547MB | 416.0s | 12.7s
CNN | 83,714 | 986MB | 21.1s | 0.85s
1-Layer SSAN + RPR | 465,600 | 381MB | 64.6s | 8.9s
1-Layer SSAN + PE | 453,000 | 381MB | 58.1s | 8.5s
2-Layer SSAN + RPR | 839,400 | 509MB | 70.3s | 9.3s
Transformer + RPR | 1,177,920 | 510MB | 78.2s | 9.7s
Table 3: Neural networks architecture characteristics. A comparison of number
of learnable parameters, GPU VRAM usage (in megabytes) during training, as
well as training and inference times (in seconds).

###  4.2 Embeddings

We use the exact same word embeddings as Barnes et al. (2017). They trained
the 50, 100, 200, and 600-dimensional word embeddings using the word2vec
algorithm described in Mikolov et al. (2013) on a 2016 Wikipedia dump. In
order to compare to previous work, they also used the publicly available
Google 300-dimensional word2vec embeddings, which are trained on a part of
Google News dataset333https://code.google.com/archive/p/word2vec/. For all
models, out-of-vocabulary words are initialized randomly from the uniform
distribution on the interval [-0.25 , 0.25].

###  4.3 Baselines

We take 5 classifiers from Barnes et al. (2017) and use their published
results as baselines. Two of the methods are based on logistic regression, Bow
and Ave, and 3 are neural network based, LSTM, BiLSTM, and CNN.

The (Bow) baseline is a L2-regularized logistic regression trained on bag-of-
words representation. Each word is represented by a one-hot vectors of size
$n=|V|$, where $|V|$ is the vocabulary size.

The (Ave) baseline is also a L2-regularized logistic regression classifier
except trained on the average of the 300-dimension word embeddings for each
sentence.

The LSTM baseline, input word embeddings are passed into an LSTM layer. Then a
50-dimensional feed-forward layer with ReLU activations is applied, followed
by a softmax layer that produces that model classification outputs. Dropout
Srivastava et al. (2014) is applied to the input word embeddings for
regularization.

The BiLSTM baseline is the same as LSTM, except that a second LSTM layer is
used to process the input word embeddings in the reverse order. The outputs of
the two LSTM layers are concatenated and passed a feed-forward layer,
following by the output softmax layer. Dropout is applied identically as in
LSTM. This modification improves the networks ability to capture long-range
dependencies.

The final baseline is a simple CNN network. The input sequence of $n$
embeddings is reshaped to an $n\times R$ dimensional matrix $M$, where $R$ is
the dimensionality of the embeddings. Convolutions with filter size of [2,3,4]
are applied to $M$, following by a pooling layer of length 2. As for LSTM
networks, a feed-forward layer is applied followed by an output softmax layer.
Here, dropout is applied to input embeddings as well as after the convolution
layers.

The LSTM, BiLSTM, and CNN baselines are trained using ADAM Kingma and Ba
(2014) with cross-entropy loss and mini-batches of size 32. Hidden layer
dimension, dropout amount, and the number of training epochs are tuned on the
validation set for each (model, input embedding, dataset) combination.

###  4.4 Self-Attention Architectures

We use 1-Layer SSAN + RPR and 2-Layer SSAN + RPR to compare the self-attention
mechanism to the recurrence and convolution mechanisms in LSTM, BiLSTM, and
CNN models. We compare these models using all word embeddings sizes.

Next, we explore the performance of a modified Transformer Encoder described
in 3. We do this to determine if a more complex architecture that utilized
multi-head attention is more beneficial.

Finally, we compare the performance of using positional encodings (+PE) and
relative positional representations (+RPR) for the Transformer Encoder and
1-Layer-SSAN architectures. We also test 1-Layer SSAN without using any
positional information techniques.

For the self-attention networks, we simplify the training process to only tune
one parameter and apply the same process to all models. Only the learning rate
is tuned for every (model, input embedding) pair. We fix the number of batches
to train for to 100,000 and pick the model with highest validation accuracy.
Each batch is constructed by randomly sampling the training set. Model
dimensionality $d_{\text{model}}$ is fixed to being the same as the input word
embeddings. Learning rate is tuned based on the size of $d_{\text{model}}$.
For $d_{\text{model}}$ dimensions [50, 100, 200, 300, 600] we use learning
rates of [0.15, 0.125, 0.1, 0.1, 0.05] respectively, because the larger
$d_{\text{model}}$ models tend to over-fit faster. Dropout of 0.7 is applied
to all models, and the ADADELTA Zeiler (2012) optimizer is used with cross-
entropy loss.

##  5 Analysis

Table 2 contains the summary of all the experimental results. For all neural
network models we report mean test accuracy of five runs as well as the
standard deviations. Macro-Avg results are the average accuracy of a model
across all datasets. We focus our discussion on the Macro-Avg column as it
demonstrates the models general performance for sentiment analysis.

Our results show general better performance for self-attention networks in
comparison to LSTM, BiLSTM and CNN models. Using the same word embedding, all
of the self-attention models receive higher Macro-Avg accuracy than all
baseline models. 1-Layer-SSAN+RPR models generally perform the best for all
(input embeddings, dataset) combinations, and getting top scores for five out
of six datasets. Transformer Encoder+RPR also performs comparatively well
across all datasets, and achieves top accuracy for the OpeNER dataset.

Using 2-Layer-SSAN+RPR does not yield better performance results compared to
1-Layer-SSAN+RPR. We believe that one self-attention layer is sufficient as
the datasets that we have tested on were relatively small. This is reinforced
by the results we see from Transformer Encoder + RPR since it achieves similar
accuracy as 2-Layer-SSAN+RPR and 1-Layer-SSAN+RPR while having greater
architectural complexity and more trainable parameters, see Table 3.

Using relative positional representations for 1-Layer-SSAN+RPR increases the
Macro-Avg accuracy by 2.8% compared to using positional encodings for 1-Layer-
SSAN+PE, and by 0.9% compared to using no positional information at all
(1-Layer-SSAN). Interestingly enough, we observe that using no positional
information performs better than using positional encodings. This could be
attributed once again to small dataset size, as Vaswani et al. (2017)
successfully used positional encodings for larger MT datasets.

Another observation is that SenTube dataset trials achieve a low accuracy
despite having binary classes. This is unexpected as generally with a low
number of classes it is easier to train on the dataset and achieve higher
accuracy. We suspect that this is because SenTube contains longer sentences
and very low word embedding coverage. Despite this, SSANs perform relatively
well on the SenTube-A dataset, which suggests that they are superior at
capturing long-range dependencies compared to other models.

Smaller $d_{\text{model}}$ SSAN models perform worse for lower dimension input
embeddings on SST-fine, SST-binary and OpeNER datasets while still performing
well on SenTube and SemEval. This is caused by the limitations of our training
process where we forced the network $d_{\text{model}}$ to be same size as the
input word embeddings and use the same learning rate for all datasets. We
found that working with smaller dimensions of $d_{\text{model}}$ the learning
rate needed to be tuned individually for some datasets. For example, using a
learning of 0.15 for 50D models would work well for SenTube and SemEval, but
would under-fit for SST-fine, SST-binary and OpeNER datasets. We decided to
not modify the training process for the smaller input embeddings in order to
keep our training process simplified.

###  5.1 Model Characteristics

Here we compare training and test efficiency, memory consumption and number of
trainable parameters for every model. For all models, we use the SST-fine
dataset, hidden dimension size of 300, Google 300D embeddings, batch sizes of
32 for both training and inference, and the ADAM optimizer Kingma and Ba
(2014). The Training Time test is the average time it takes every model to
train on 10 epochs of the SST-fine train set ( 2670 batches of size 32). The
Inference Time test is the average time it takes a model to produce
predictions for the validation set 10 times ( 344 batches of size 32). Table 3
contains the summary of model characteristics. The GPU VRAM usage is the
amount of GPU video memory that is used during training.

CNN has the lowest number of parameters but consumes the most GPU memory. It
also has the shortest training and inference time, which we attributed to the
low number of parameters.

Using relative position representations compared to positional encoding for
1-Layer-SSAN increases the number of trainable parameters by only 2.7%,
training time by 11.2%, and inference time by 4.7%. These findings are similar
to what Shaw et al. (2018) reported.

BiLSTM has double the number of parameters as well as near double training and
inference times compared to LSTM. This is reasonable due to the nature of the
architecture being two LSTM layers. Much like BiLSTM, going from 1-Layer-SSAN
to 2-Layer-SSAN doubles the number of trainable parameters. However, the
training and inference times only increase by 20.1% and 9.4% respectively.
This demonstrates the efficiency of the self-attention mechanism due to it
utilizing only matrix multiply operations, for which GPUs are highly-
optimized.

We also observe that self-attention models are faster to train than LSTM by
about 3.4 times, and 5.9 times for BiLSTM. However, inference times are slower
than LSTM by 15.5% and faster than BiLSTM by 41%.

##  6 Conclusion

In this paper we focused on demonstrating that self-attention networks achieve
better accuracy than previous state-of-the-art techniques on six datasets. In
our experiments, multiple SSAN networks performed better than CNN and RNN
architectures; Self-attention architecture resulted in higher accuracy than
LSTMs while having 35% fewer parameters and shorter training time by a factor
of 3.5. Additionally, we showed that SSANs achieved higher accuracy on the
SenTube datasets, which suggests they are also better at capturing long-term
dependencies than RNNs and CNNs. Finally, we reported that using relative
positional representation is superior to both using positional encodings, as
well as not incorporating any positional information at all. Using relative
positional representations for self-attention architectures resulted in higher
accuracy with negligible impact on model training and inference efficiency.

For future work, we plan to extend the SSAN networks proposed to achieve
state-of-the-art results on the complete SST dataset. We are also interested
to see the behaviour of the models explored in this work on much larger
datasets, we hypothesize that stacked multi-head self-attention architectures
will perform significantly better than RNN and CNN counterparts, all while
remaining more efficient at training and inference.

## Acknowledgments

We thank the anonymous reviewers for their insightful suggestions.

## References

  * Agerri et al. [2013] Rodrigo Agerri, Montse Cuadros, Seán Gaines, and German Rigau.  Opener: Open polarity enhanced named entity recognition.  _Procesamiento del Lenguaje Natural_ , 51(0):215–218, 2013.  ISSN 1989-7553.  URL http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/4891.
  * Bahdanau et al. [2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  _arXiv:1409.0473_ , 2014.
  * Barnes et al. [2017] Jeremy Barnes, Roman Klinger, and Sabine Schulte im Walde.  Assessing state-of-the-art sentiment models on state-of-the-art sentiment datasets.  In _Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , Copenhagen, Denmark, 2017\.
  * Cheng et al. [2016] Jianpeng Cheng, Li Dong, and Mirella Lapata.  Long short-term memory-networks for machine reading.  _CoRR_ , abs/1601.06733, 2016.  URL http://arxiv.org/abs/1601.06733.
  * Cho et al. [2014] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder–decoder approaches.  In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_ , pages 103–111. Association for Computational Linguistics, 2014.  doi: 10.3115/v1/W14-4012.  URL http://www.aclweb.org/anthology/W14-4012.
  * Chollet et al. [2015] François Chollet et al.  Keras.  https://keras.io, 2015.
  * Daniluk et al. [2017] Michał Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel.  Frustratingly short attention spans in neural language modeling.  _arXiv preprint arXiv:1702.04521_ , 2017.
  * Gehring et al. [2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin.  Convolutional sequence to sequence learning.  _arXiv preprint arXiv:1705.03122_ , 2017.
  * Greaves et al. [2013a] Felix Greaves, Daniel Ramirez-Cano, Christopher Millett, Ara Darzi, and Liam Donaldson.  Harnessing the cloud of patient experience: using social media to detect poor quality healthcare.  _BMJ Qual Saf_ , 22(3):251–255, 2013a.
  * Greaves et al. [2013b] Felix Greaves, Daniel Ramirez-Cano, Christopher Millett, Ara Darzi, and Liam Donaldson.  Use of sentiment analysis for capturing patient experience from free-text comments posted online.  _Journal of medical Internet research_ , 15(11), 2013b.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780, 1997.
  * Jansen et al. [2009] Bernard J Jansen, Mimi Zhang, Kate Sobel, and Abdur Chowdury.  Twitter power: Tweets as electronic word of mouth.  _Journal of the American society for information science and technology_ , 60(11):2169–2188, 2009.
  * Kim [2014] Yoon Kim.  Convolutional neural networks for sentence classification.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1746–1751. Association for Computational Linguistics, 2014.  doi: 10.3115/v1/D14-1181.  URL http://www.aclweb.org/anthology/D14-1181.
  * Kingma and Ba [2014] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2014.
  * Kokkinos and Potamianos [2017] Filippos Kokkinos and Alexandros Potamianos.  Structural attention neural networks for improved sentiment analysis.  _arXiv preprint arXiv:1701.01811_ , 2017.
  * Liu et al. [2018] Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Łukasz Kaiser, and Noam Shazeer.  Generating wikipedia by summarizing long sequences.  _arXiv preprint arXiv:1801.10198_ , 2018.
  * Lu et al. [2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh.  Hierarchical question-image co-attention for visual question answering.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems 29_ , pages 289–297. Curran Associates, Inc., 2016.  URL http://papers.nips.cc/paper/6202-hierarchical-question-image-co-attention-for-visual-question-answering.pdf.
  * Mikolov et al. [2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ , 2013.
  * Mäntylä et al. [2018] Mika V. Mäntylä, Daniel Graziotin, and Miikka Kuutila.  The evolution of sentiment analysis—a review of research topics, venues, and top cited papers.  _Computer Science Review_ , 27:16 – 32, 2018.  ISSN 1574-0137.  doi: https://doi.org/10.1016/j.cosrev.2017.10.002.  URL http://www.sciencedirect.com/science/article/pii/S1574013717300606.
  * Nair and Hinton [2010] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Proceedings of the 27th international conference on machine learning (ICML-10)_ , pages 807–814, 2010.
  * Nakov et al. [2013] Preslav Nakov, Zornitsa Kozareva, Alan Ritter, Sara Rosenthal, Veselin Stoyanov, and Theresa Wilson.  Semeval-2013 task 2: Sentiment analysis in twitter, 2013.
  * Pang et al. [2002] Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan.  Thumbs up?: Sentiment classification using machine learning techniques.  In _Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10_ , EMNLP ’02, pages 79–86, Stroudsburg, PA, USA, 2002. Association for Computational Linguistics.  doi: 10.3115/1118693.1118704.  URL https://doi.org/10.3115/1118693.1118704.
  * Parmar et al. [2018] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, and Alexander Ku.  Image transformer.  _arXiv preprint arXiv:1802.05751_ , 2018.
  * Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, 2014.  URL http://www.aclweb.org/anthology/D14-1162.
  * Povey et al. [2018] Daniel Povey, Hossein Hadian, Pegah Ghahremani, Ke Li, and Sanjeev Khudanpur.  A time-restricted self-attention layer for asr.  pages 5874–5878, 2018.
  * Schuster and Paliwal [1997] Mike Schuster and Kuldip K Paliwal.  Bidirectional recurrent neural networks.  _IEEE Transactions on Signal Processing_ , 45(11):2673–2681, 1997.
  * Shaw et al. [2018] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.  Self-attention with relative position representations.  _CoRR_ , abs/1803.02155, 2018.
  * Shen et al. [2018] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang.  Disan: Directional self-attention network for rnn/cnn-free language understanding.  In _AAAI Conference on Artificial Intelligence_ , 2018.
  * Socher et al. [2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642, 2013.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ , 15(1):1929–1958, 2014.
  * Strubell et al. [2018] Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum.  Linguistically-informed self-attention for semantic role labeling.  _CoRR_ , abs/1804.08199, 2018.
  * Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_ , pages 3104–3112. Curran Associates, Inc., 2014.  URL http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf.
  * Tai et al. [2015] Kai Sheng Tai, Richard Socher, and Christopher D Manning.  Improved semantic representations from tree-structured long short-term memory networks.  _arXiv preprint arXiv:1503.00075_ , 2015.
  * Uryupina et al. [2014] Olga Uryupina, Barbara Plank, Aliaksei Severyn, Agata Rotondi, and Alessandro Moschitti.  Sentube: A corpus for sentiment analysis on youtube social media.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation_. European Language Resources Association, 2014\.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 5998–6008. Curran Associates, Inc., 2017.  URL http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf.
  * Zeiler [2012] Matthew D Zeiler.  Adadelta: an adaptive learning rate method.  _arXiv preprint arXiv:1212.5701_ , 2012.
  * Zhang et al. [2018] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  _arXiv preprint arXiv:1805.08318_ , 2018.
  * Zhou et al. [2018] Yi Zhou, Junying Zhou, Lu Liu, Jiangtao Feng, Haoyuan Peng, and Xiaoqing Zheng.  Rnn-based sequence-preserved attention for dependency parsing.  2018\.  URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17176.


# Classification of Electrical Impedance Tomography Data Using Machine
Learning

Diogo Pessoa1 ID , Bruno Machado Rocha1 ID , Grigorios-Aris Cheimariotis2 ID ,
Kostas Haris2 ID ,
Claas Strodthoff3 ID , Evangelos Kaimakamis2,4 ID , Nicos Maglaveras2 ID ,
Inéz Frerichs3 ID ,
Paulo de Carvalho1 ID and Rui Pedro Paiva1 ID 1University of Coimbra, Centre
for Informatics and Systems of the University of Coimbra, Department of
Informatics Engineering, 3030-290 Coimbra, Portugal. {dpessoa, bmrocha,
carvalho<EMAIL_ADDRESS>Laboratory of Computing, Medical Informatics
and Biomedical Imaging Technologies, Aristotle University of Thessaloniki,
Thessaloniki, 54636, Greece. {ncheimar, kharis<EMAIL_ADDRESS>Department of
Anesthesiology and Intensive Care Medicine, University Medical Centre
Schleswig-Holstein, Campus Kiel, Kiel, Germany. {Claas.Strodthoff,
<EMAIL_ADDRESS>1st Intensive Care Unit, “G. Papanikolaou” General
Hospital, Thessaloniki, Greece<EMAIL_ADDRESS>
###### Abstract

Patients suffering from pulmonary diseases typically exhibit pathological lung
ventilation in terms of homogeneity. Electrical Impedance Tomography (EIT) is
a non-invasive imaging method that allows to analyze and quantify the
distribution of ventilation in the lungs. In this article, we present a new
approach to promote the use of EIT data and the implementation of new clinical
applications for differential diagnosis, with the development of several
machine learning models to discriminate between EIT data from healthy and non-
healthy subjects. EIT data from 16 subjects were acquired: 5 healthy and 11
non-healthy subjects (with multiple pulmonary conditions). Preliminary results
have shown accuracy percentages of 66% in challenging evaluation scenarios.
The results suggest that the pairing of EIT feature engineering methods with
machine learning methods could be further explored and applied in the
diagnostic and monitoring of patients suffering from lung diseases. Also, we
introduce the use of a new feature in the context of EIT data analysis
(Impedance Curve Correlation).

###### Index Terms:

EIT, Machine Learning, Feature Engineering

##  I Introduction

Lung-related diseases represent some of the most common medical conditions
worldwide [1]. Moreover, they are among the most significant causes of
morbidity and mortality and are responsible for a substantial strain on health
systems [2, 3]. Most of these diseases can be characterized by increased
ventilation inhomogeneity associated with the pathologically changed regional
lung function. Therefore, parameters that translate these changes in lung
function can be useful in the diagnostic and monitoring of such diseases.

Electrical Impedance Tomography (EIT) is a non-invasive, radiation-free,
imaging technique that relies on the application of alternating electrical
currents on the external surface of the body to assess its internal electrical
characteristics and generate bio-impedance images/maps [4, 5]. Hence, the
reconstructed images represent the electrical permittivity and conductivity
distributions inside the chest [4].

Despite the lower spatial resolution of EIT when compared to other traditional
well established imaging modalities, such as magnetic resonance imaging and
computed tomography scan, EIT can provide continuous monitoring and it has a
high temporal resolution, unlike the other methods [4]. Therefore, while
traditional imaging techniques can provide further insight regarding
structural changes of the pulmonary tissue, EIT is better suited to assess its
functionality [6].

Over the years, the number of studies focusing on EIT has experienced a steady
increase. However, the majority of these have not aimed at the development of
methods to promote differential diagnosis applications with EIT. This could
represent one of the greatest strengths of EIT, especially due to its
suitability to be deployed in wearable devices, allowing to closely monitor
patients with telemonitoring systems [7].

In [8], the authors suggested, through statistical inference, that the global
inhomogeneity index extracted from EIT might be useful for the identification
and follow up of ventilation problems in patients with COPD. Also, in [5], the
authors have assessed, through the extraction and analysis of several
measures, the usefulness of EIT in the characterization of regional
ventilation in idiopathic pulmonary fibrosis. In another study, the authors
have used raw EIT images paired with regression machine learning techniques to
infer several respiratory and circulatory parameters [9].

The main objective of this exploratory study is to understand if the
association of EIT and machine learning could be useful for differential
diagnosis applications, through the employment of multiple traditional machine
learning classification models. To the best of our knowledge, this is the
first study where EIT data is employed in association with a machine learning
classification pipeline, for this purpose. Therefore, we studied whether EIT
derived measures of spatial and temporal ventilation homogeneity, extracted
from periods of tidal and deep breathing, were able to discriminate between
EIT data coming from healthy and non-healthy subjects.

##  II Materials and Methods

###  II-A Data acquisition and protocol

The data collection process for this study was carried out in the pneumology
service at the George Papanikolaou General Hospital of Thessaloniki, Greece.
The study was approved by the Ethics Committee of the same hospital.

A total of 16 (6 female, 10 male) participants were considered in this study,
which included 5 healthy and 11 non-healthy subjects. The group of non-healthy
subjects suffered from a wide range of conditions (such as COPD, asthma,
pneumonia, etc.). The average age for all subjects was 60.3$\pm$20.4 years
with a mean BMI of 28.9$\pm$5.5 $kg/m^{2}$ (Healthy - 35.3$\pm$14.4 years and
31.0$\pm$6.5 $kg/m^{2}$; Non-healthy - 71.6$\pm$9.6 years and 28.0$\pm$5.1
$kg/m^{2}$).

EIT data were collected using the Goe-MF II EIT device (CareFusion, Höchberg,
Germany). An array of sixteen self-adhesive electrodes (Blue Sensor, Ambu,
Ballerup, Denmark) was attached to the chest circumference between the 5-6th
intercostal space (xipho-sternal line), with another reference electrode
placed on the abdomen. Small alternating electrical currents were delivered
through adjacent pairs of electrodes in a sequential rotating process and the
resulting potential differences were measured by the remaining electrode
pairs. EIT data were acquired at a sampling rate of 33 images/second.

During the acquisitions the participants were asked to perform two types of
recordings: 1) tidal breathing during two or three respiratory cycles,
followed by deep breathing, also during two or three respiratory cycles; 2)
tidal breathing followed by a period of about two minutes with both harms
elevated above the head. Each participant has repeated the two types of
recordings multiple times.

###  II-B EIT ROIs

In order to take advantage of the EIT capability to perform regional
ventilation analysis [10], several major Regions of Interest (ROIs) were
considered to extract meaningful parameters from the data. Therefore, based on
the model used to obtain the reconstructed images (subsection II-C), the
images were divided in multiple regions. A total of 16 different regions was
considered. These include major ROIs, such as the right/left and
anterior/posterior areas of the lung, as well as 4 different quadrants and
vertical and horizontal ROIs. In Figure 1, the vertical and horizontal ROIs
are represented, as well as the shape of the model used by the EIDORS library
(yellow area).

Figure 1: EIT horizontal and vertical ROIs (Left - horizontal ROIs, Right -
vertical ROIs; A-Anterior; P-Posterior; R-Right; L-Left; M-Mid).

###  II-C EIT reconstruction

The acquired raw EIT data were processed offline to obtain the reconstructed
images/frames using the Graz Consensus Reconstruction Algorithm for EIT
(GREIT) [11]. The reconstruction was performed using an adult thorax shaped
model with a single plane of 16 electrodes and the adjacent stimulation
pattern was selected from the models library of the EIDORS software [12]. The
resulting reconstructed EIT images consisted of 32 by 32 pixels. After
obtaining the reconstructed images for every time step, the Global Impedance
Curve (GIC) was computed through the sum of all individual pixel values for
each image. Using the GIC, the end-inspiration and end-expiration moments were
manually determined/annotated through the analysis of the curves of every EIT
file considered (see Figure 2).

Figure 2: Example of GIC with the identification of end-inspiratory and end-
expiratory moments (A.U.-Arbitrary Units; $\Delta Z$-Impedance variation).

After the determination of the respiratory cycles, the functional EIT images
were computed (fEIT images). These images were determined for each breathing
cycle and correspond to the difference of the EIT reconstructed images between
end-inspiration and end-expiration instants (see Figure 3). The regional
impedance curves (RIC) for each of the considered ROIs (see subsection II-B)
were also isolated between the above-mentioned moments.

Figure 3: Functional EIT image generation: difference between end-inspiration
(red square) and begin-inspiration (green square) frames (A.U.-Arbitrary
Units; $\Delta Z$-Impedance variation).

###  II-D EIT measures/features

In this study, several EIT features were extracted at a global and regional
level, based on the regions defined in subsection II-B. The features were
extracted for each inspiration cycle individually, from each fEIT image and
the corresponding impedance curves (as identified in subsection II-C). A total
of 112 features were extracted for each sample using methods developed in
MATLAB 2020a. A brief description of the extracted parameters/features is
presented below:

  * •

Ventilation Ratios: ratio between the sum of all pixel values from two certain
regions (e.g., ratio between left and right lungs, etc) [10];

  * •

Coefficient of Variation: statistical measure that characterizes the relative
magnitude of the standard deviation of a fEIT image with respect to its mean
[10]. This measure was determined for the multiple ROIs;

  * •

Global Inhomogeneity Index: measure that quantifies the homogeneity of the
volume distribution [13]. The higher it its, the more heterogeneous the
ventilation distribution is. This measure was determined for the multiple
ROIs;

  * •

Regional Ventilation Delay: the regional ventilation delay expresses the delay
between the global start of inspiration and the point in time where the
regional impedance curve reaches a certain threshold (set to 40%) [14]. Thus,
it can be used to translate the temporal heterogeneity of ventilation. This
measure is determined for multiple regional impedance curves in relation to
the GIC;

  * •

Impedance Curve Correlation: The impedance curve correlation is determined
through the correlation between the GIC and the regional impedance curves, and
between the regional impedance curves within themselves. This metric captures
the dependency in variation of the impedance curves, translating the spatial
and temporal homogeneity of ventilation in lung function.

###  II-E Classifiers

To develop the classification models, we used several machine learning
algorithms to classify the samples, such as: Linear discriminant analysis
(LDA); Support vector machine with radial basis function kernel (SVMrbf);
Decision Trees (DecTree); Random undersampling boosted trees (RUSBoost); and
Random Forest (RndForest). All the classifiers were trained 50 times with
different seeds to divide the complete dataset, and their hyperparameters were
optimized on a validation set containing 25% of the training set. Bayesian
optimization was used to optimize the models’ hyperparameters [15]. The models
with the best set of hyperparameters were then applied to the test set. Filter
feature selection algorithms (e.g., ReliefF and MRMR) were tested but did not
lead to a better performance of the algorithms. Therefore, the results for
these cases are not presented. All the models were developed in MATLAB 2020a.

##  III Results and Discussion

In this section, we present the performance evaluation of the algorithms. All
the classification methods were tested in two different scenarios: A) Repeated
stratified hold-out validation; B) Repeated stratified patient hold-out
validation.

Scenario A consists in the random division of all the samples available with
75% of the data being used to train the models and the remaining 25% for
testing. This is a scenario frequently found in many research approaches,
which however has significant limitations as will be explained later in this
section. On the other hand, in scenario B the division occurs patient wise,
with 75% of the total number of patients used in the training set and the
remaining 25% in the testing set, accordingly. The patient-wise split
guarantees that data from the same patient cannot be in the training and
testing datasets at the same time. This split is performed in such a way that
the number of available samples on the testing set is always equal, or
superior, to 25% of the total number of available samples. Both processes are
repeated 50 times, being that 25% percent of the training data is retained and
used for validation to perform hyperparameter optimization during the training
phase, as previously mentioned.

To develop the machine learning classification models a total of 1500 samples
were considered (1108 from non-healthy subjects and 392 from healthy ones,
respectively). In Table I and Table II the results obtained over 50
repetitions for both evaluation scenarios are presented.

TABLE I: Classification performance evaluation for Repeated hold-out
validation (Scenario A).

Class |  Acc | F1 H | Rec H | Prec H |  Spec H | F1 NH | Rec NH | Prec NH | Spec NH
---|---|---|---|---|---|---|---|---|---
DecisionTree  | 90.9$\pm$1.4  | 80.8$\pm$4.1 | 84.0$\pm$4.3 | 94.5$\pm$1.7 | 82.3$\pm$2.8  | 94.5$\pm$1.7 | 93.3$\pm$1.3 | 80.8$\pm$4.1 | 93.9$\pm$1.0
RUSBoost  | 95.1$\pm$1.5  | 89.0$\pm$3.9 | 92.1$\pm$4.2 | 97.2$\pm$1.7 | 90.4$\pm$2.9  | 97.2$\pm$1.7 | 96.2$\pm$1.3 | 89.0$\pm$3.9 | 96.7$\pm$1.1
LDA  | 86.2$\pm$4.1  | 77.6$\pm$5.1 | 73.8$\pm$9.6 | 89.3$\pm$6.6 | 75.0$\pm$4.8  | 89.3$\pm$6.6 | 91.9$\pm$1.5 | 77.6$\pm$5.1 | 90.4$\pm$3.4
SVMrbf  | 95.7$\pm$1.2  | 89.2$\pm$4.0 | 94.4$\pm$2.9 | 98.1$\pm$1.1 | 91.6$\pm$2.5  | 98.1$\pm$1.1 | 96.3$\pm$1.3 | 89.2$\pm$4.0 | 97.2$\pm$0.8
RndForest  |  96.0$\pm$1.2 | 88.7$\pm$3.8 | 96.0$\pm$3.7 | 98.6$\pm$1.4 |  92.1$\pm$2.4 | 98.6$\pm$1.4 |  96.1$\pm$1.2 | 88.7$\pm$3.8 | 97.3$\pm$0.8

Acc-Accuracy; F1-F1 Score; Rec-Recall; Prec-Precision; Spec-Specificity;
H-Healthy; NH-Non-healthy.
---

TABLE II: Classification performance evaluation for repeated patient hold-out
validation (Scenario B).

Class |  Acc | F1 H | Rec H | Prec H |  Spec H | F1 NH | Rec NH | Prec NH | Spec NH
---|---|---|---|---|---|---|---|---|---
DecisionTree  | 62.1$\pm$8.7  | 26.9$\pm$13.0 | 46.4$\pm$20.6 | 81.0$\pm$11.6 | 32.4$\pm$13.0  | 81.0$\pm$11.6 | 67.4$\pm$6.7 | 26.9$\pm$13.0 | 73.3$\pm$7.5
RUSBoost  | 65.1$\pm$8.1  | 35.5$\pm$15.9 | 50.9$\pm$19.8 | 80.8$\pm$10.8 |  40.3$\pm$15.4 | 80.8$\pm$10.8 | 70.3$\pm$6.3 | 35.5$\pm$15.9 | 74.8$\pm$6.7
LDA  | 62.3$\pm$8.4  | 25.7$\pm$16.0 | 52.6$\pm$26.3 | 82.2$\pm$14.8 | 30.3$\pm$14.6  | 82.2$\pm$14.8 | 67.5$\pm$6.0 | 25.7$\pm$16.0 | 73.4$\pm$7.5
SVMrbf  | 59.8$\pm$8.5  | 22.6$\pm$10.4 | 41.6$\pm$19.6 | 79.8$\pm$12.4 | 27.7$\pm$10.7  | 79.8$\pm$12.4 | 65.7$\pm$6.1 | 22.6$\pm$10.4 | 71.7$\pm$7.6
RndForest  |  66.0$\pm$7.3 | 28.7$\pm$15.3 | 54.2$\pm$22.8 | 85.9$\pm$11.0 | 35.4$\pm$15.9  | 85.9$\pm$11.0 | 69.4$\pm$5.5 | 28.7$\pm$15.3 | 76.4$\pm$6.2

Acc-Accuracy; F1-F1 Score; Rec-Recall; Prec-Precision; Spec-Specificity;
H-Healthy; NH-Non-healthy.
---

Through the analysis of both tables, the disparity between the two scenarios
is evident. In scenario A, all classifiers performed well, with the random
forest model presenting the highest score, reaching an average accuracy of
96.0$\pm$1.2%. On the other hand, in scenario B, the performance drops
significantly for all models and, once again, the random forest model presents
the highest score, reaching an average accuracy of 66.0$\pm$7.3%. From the
analysis of Table II, it is possible to observe that the performance of the
classifiers on the Healthy class decreases significantly, with lower recall
and F1 values. The precision of the Non-healthy class also decreases,
highlighting the bias of the models.

The fact that the obtained results are much lower when the data are divided in
a patient wise fashion suggests that inter-patient distribution variability
for the extracted metrics is very high. This stresses the need for further
studies to validate the normality and non-normality values. Notwithstanding,
it was already expected for the results to be lower in Scenario B, since it
presents a more realistic evaluation scenario and where there is no data
leakage during the training phase of the classifiers, unlike scenario A. These
results underline the importance of careful experimental design, as it can
have a high impact on the results that are obtained.

To understand which features were the most relevant for the classification in
each scenario, the importance of each feature for the best classification
model (Random Forrest) was estimated using MATLAB function predictorImportance
111https://www.mathworks.com/help/stats/compactclassificationensemble.predictorimportance.html.
The function computes estimates of the predictors’ importance for the
classification model by summing changes in the risk due to splits on every
predictor and dividing the sum by the number of branch nodes. Table III lists
the 10 most important features for the best classification model in both
scenarios. The importance value for each feature was determined as the average
value across all 50 runs.

TABLE III: Ten most important features for Random Forest Model in both evaluation scenarios. Scenario A | Scenario B
---|---
ratio_right_left | corr_verticalR_verticalL
corr_global_posterior | corr_horizontalA_horizontalMA
corr_horizontalA_horizontalMA | corr_horizontalMP_horizontalP
corr_verticalR_verticalL | ratio_right_left
corr_quadrant1_quadrant3 | corr_global_horizontalP
corr_horizontalMP_horizontalP | ratio_quadrant1_quadrant4
GI_horizontalP | corr_global_quadrant2
corr_global_horizontalP | corr_global_posterior
ratio_quadrant1_quadrant4 | corr_global_verticalL
corr_global_horizontalMA | GI_horizontalP

From the analysis of Table III, it possible to observe that the Impedance
Curve Correlation features are among the most important in both evaluation
scenarios. In addition, there is a significant level of accordance regarding
the most relevant features between the classification models in the two
scenarios, being 8 of the 10 most important common in both cases. This fact
highlights the discriminating power of those features. In Figure 4 and Figure
5, the distributions for the most important features in each of the evaluation
scenarios are presented (ratio_right_left and corr_verticalR_verticalL),
respectively). From the analysis of both figures it is noticeable that the two
features present significantly different distributions between healthy and
non-healthy subjects.

Figure 4: Boxplot with the distribution of the most important feature for the
Random Forest model in scenario A (Ratio between right and left ROIs). Figure
5: Boxplot with the distribution of the most important feature for the Random
Forest model in scenario B (Impedance curve correlation between vertical right
and left ROIs).

One of the major limitations of this study is related to the limited amount of
data that was used and the lack of class balance. Furthermore, the
characteristics of the healthy group are significantly different from the non-
healthy group, with the participants in the healthy group being much younger.
All of these factors combined can introduce significant bias in the
classification models. Since EIT is still not a fully established imaging
modality in the clinical practice, the number of studies conducted using large
amounts of data using this data-source is still scarce. Moreover, the publicly
available EIT datasets typically contain a small number of samples and are
mostly conceived for demonstration/education purposes.

Another possible limitation could be related to the labeling process of the
samples used to develop the classifiers. If a sample was from a subject who
was healthy, it was tagged as healthy. The same principle was applied to non-
healthy subjects. However, even if a subject is non-healthy, this does not
necessarily imply that some of the samples cannot be "normal". Even if this
fact might introduce bias on the training process of the machine learning
algorithms, it is very hard to avoid it in this particular case, since it
would require an extensive one-by-one validation of all the samples available.
Moreover, the default values of "normality" for the extracted features are not
established, making this process unfeasible.

##  IV Conclusion

With this work we were able to demonstrate the potential of pairing EIT data
processing and feature engineering with machine learning classification
models. The obtained results, while far from perfect in more challenging
evaluation scenarios, suggest that these methods might be useful to develop
differential diagnosis systems using EIT and exploit the main potentialities
of this imaging technique. We have also introduced a new feature/measure in
EIT data processing (Impedance Curve Correlation), which has proved to be
relevant to discriminate between the lung function of healthy and non-healthy
subjects.

The implementation of more refined and mature methods, similar to those
developed in this work, could potentially be integrated into telemonitoring
applications acting as a quick diagnosis-support tool. This highlights the
need to keep working on better methodologies, namely feature engineering
methods that extract clinically meaningful information from the EIT data.
Future work could also focus on multi-modal approaches, such as combining EIT
with lung sound data.

## ACKNOWLEDGMENT

This research is partially supported by Fundação para a Ciência e a Tecnologia
(FCT) Ph.D. scholarships (2020.04927.BD and SFRH/BD/135686/2018), by the
Horizon 2020 Framework Programme of the European Union project WELMO (under
grant agreement number 825572), and by FCT project Lung@ICU (under grant
reference DSAIPA/AI/0113/2020).

## References

  * [1] B. M. Rocha, D. Pessoa, A. Marques, P. Carvalho, and R. P. Paiva, “Automatic classification of adventitious respiratory sounds: A (un)solved problem?,” Sensors, vol. 21, no. 1, 2021.
  * [2] W. H. Organization, “The top 10 causes of death,” 2018.
  * [3] G. J. Gibson, R. Loddenkemper, B. Lundbäck, and Y. Sibille, “Respiratory health and disease in Europe: The new European Lung White Book,” European Respiratory Journal, vol. 42, no. 3, pp. 559–563, 2013\.
  * [4] T. A. Khan and S. H. Ling, “Review on Electrical Impedance Tomography: Artificial Intelligence Methods and its Applications,” Algorithms, vol. 12, p. 88, apr 2019.
  * [5] E. Krauss, D. van der Beck, I. Schmalz, J. Wilhelm, S. Tello, R. C. Dartsch, P. Mahavadi, M. Korfei, E. Teschner, W. Seeger, and A. Guenther, “Evaluation of Regional Pulmonary Ventilation in Spontaneously Breathing Patients with Idiopathic Pulmonary Fibrosis (IPF) Employing Electrical Impedance Tomography (EIT): A Pilot Study from the European IPF Registry (eurIPFreg),” Journal of Clinical Medicine, vol. 10, p. 192, jan 2021\.
  * [6] L. Lasarow, B. Vogt, Z. Zhao, L. Balke, N. Weiler, and I. Frerichs, “Regional lung function measures determined by electrical impedance tomography during repetitive ventilation manoeuvres in patients with COPD,” Physiological Measurement, vol. 42, p. 015008, jan 2021.
  * [7] I. Frerichs, B. Vogt, J. Wacker, R. Paradiso, F. Braun, M. Rapin, L. Caldani, O. Chételat, and N. Weiler, “Multimodal remote chest monitoring system with wearable sensors: a validation study in healthy subjects,” Physiological Measurement, vol. 41, p. 015006, feb 2020.
  * [8] F. Trenk, L. Mendes, P. Carvalho, R. P. Paiva, J. Henriques, N. Maglaveras, I. Chouvarda, V. Tsara, and C. A. Teixeira, “Evaluation of lung ventilation distribution in chronic obstructive pulmonary disease patients using the global inhomogeneity index,” in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), vol. 2016-Octob, pp. 5286–5289, IEEE, aug 2016.
  * [9] N. Strodthoff, C. Strodthoff, T. Becher, N. Weiler, and I. Frerichs, “Inferring respiratory and circulatory parameters from electrical impedance tomography with deep recurrent models,” IEEE Journal of Biomedical and Health Informatics, 2021.
  * [10] I. Frerichs, M. B. Amato, A. H. Van Kaam, D. G. Tingay, Z. Zhao, B. Grychtol, M. Bodenstein, H. Gagnon, S. H. Böhm, E. Teschner, O. Stenqvist, T. Mauri, V. Torsani, L. Camporota, A. Schibler, G. K. Wolf, D. Gommers, S. Leonhardt, A. Adler, E. Fan, W. R. Lionheart, T. Riedel, P. C. Rimensberger, F. S. Sipmann, N. Weiler, and H. Wrigge, “Chest electrical impedance tomography examination, data analysis, terminology, clinical use and recommendations: Consensus statement of the TRanslational EIT developmeNt stuDy group,” Thorax, vol. 72, no. 1, pp. 83–93, 2017.
  * [11] A. Adler, J. H. Arnold, R. Bayford, A. Borsic, B. Brown, P. Dixon, T. J. Faes, I. Frerichs, H. Gagnon, Y. Gärber, B. Grychtol, G. Hahn, W. R. Lionheart, A. Malik, R. P. Patterson, J. Stocks, A. Tizzard, N. Weiler, and G. K. Wolf, “GREIT: A unified approach to 2D linear EIT reconstruction of lung images,” Physiological Measurement, vol. 30, no. 6, 2009.
  * [12] A. Adler and W. R. Lionheart, “Uses and abuses of EIDORS: An extensible software base for EIT,” Physiological Measurement, vol. 27, no. 5, 2006\.
  * [13] Z. Zhao, S. Pulletz, I. Frerichs, U. Müller-Lisse, and K. Möller, “The EIT-based global inhomogeneity index is highly correlated with regional lung opening in patients with acute respiratory distress syndrome,” BMC Research Notes, vol. 7, no. 1, 2014.
  * [14] H. Wrigge, J. Zinserling, T. Muders, D. Varelmann, U. Günther, C. Von Der Groeben, A. Magnusson, G. Hedenstierna, and C. Putensen, “Electrical impedance tomography compared with thoracic computed tomography during a slow inflation maneuver in experimental models of lung injury,” Critical Care Medicine, vol. 36, no. 3, pp. 903–909, 2008.
  * [15] J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” 2012.