uq short=UQ, long=Uncertainty Quantification bnn short=BNN, long=Bayesian
Neural Networks mcdropout short=MC-Dropout, long=Monte Carlo Dropout vae
short=VAE, long=Variational Autoencoders de short=DE, long=Deep Ensembles ud
short=UD, long=Uncertainty Distillation kd short=KD, long=Knowledge
Distillation dudes short=DUDES, long=’Deep Uncertainty Distillation using
Ensembles for Segmentation’ sgd short=SGD, long=Stochastic Gradient Descent
dnn short=DNN, long=Deep Neural Network rmsle short=RMSLE, long=root mean
squared logarithmic error,

# U-CE: Uncertainty-aware Cross-Entropy for Semantic Segmentation

Steven Landgraf Markus Hillemann Kira Wursthorn Markus Ulrich
Karlsruhe Institute of Technology (KIT)
(steven.landgraf, markus.hillemann, kira.wursthorn<EMAIL_ADDRESS>
###### Abstract

Deep neural networks have shown exceptional performance in various tasks, but
their lack of robustness, reliability, and tendency to be overconfident pose
challenges for their deployment in safety-critical applications like
autonomous driving. In this regard, quantifying the uncertainty inherent to a
model’s prediction is a promising endeavour to address these shortcomings. In
this work, we present a novel Uncertainty-aware Cross-Entropy loss (U-CE) that
incorporates dynamic predictive uncertainties into the training process by
pixel-wise weighting of the well-known cross-entropy loss (CE). Through
extensive experimentation, we demonstrate the superiority of U-CE over regular
CE training on two benchmark datasets, Cityscapes and ACDC, using two common
backbone architectures, ResNet-18 and ResNet-101. With U-CE, we manage to
train models that not only improve their segmentation performance but also
provide meaningful uncertainties after training. Consequently, we contribute
to the development of more robust and reliable segmentation models, ultimately
advancing the state-of-the-art in safety-critical applications and beyond.
111Code will be made available once the publication process is complete.

##  1 Introduction

Humans often make poor decisions and reach erroneous conclusions while
overestimating their abilities, a phenomenon known as the Dunning-Kruger
effect [22]. Although deep neural networks are highly effective at solving
semantic segmentation problems [34], they also suffer from overconfidence
[13]. Additionally, neural networks lack interpretability [12] and struggle to
distinguish between in-domain and out-of-domain samples [26]. These flaws are
particularly relevant in safety-critical applications, such as autonomous
driving [32] and medical imaging [27], as well as in computer vision tasks
that have high demands on reliability, like industrial inspection [44, 16] and
automation [24, 45], where robust predictions are crucial. Misclassifying
pixels in these contexts can lead to severe consequences, emphasizing the need
for robust and trustworthy segmentation models.

Figure 1: U-CE introduces an uncertainty-aware cross-entropy loss that
dynamically incorporates the predictive uncertainties provided by Monte Carlo
Dropout (MC-Dropout) into the training process. As a result, we manage to
train models that are naturally capable of predicting meaningful uncertainties
after training while also improving their segmentation performance.

Previous work suggests that quantifying the uncertainty inherent to a model’s
prediction is a promising endeavour to enhance the safety and reliability of
such applications [25, 27, 26, 35, 36]. These uncertainties provide additional
insights beyond the common softmax probabilities, revealing regions where the
model is indecisive and likely to make errors. Surprisingly, the utilization
of these uncertainties during the training of segmentation models has not been
thoroughly explored.

In this work, we present a novel Uncertainty-aware Cross-Entropy loss,
referred to as U-CE, that addresses this gap by incorporating dynamic
uncertainty estimates into the training process as shown in Figure 1. Through
pixel-wise uncertainty weighting of the well-known cross-entropy loss (CE), we
harness the valuable insights provided by the uncertainties for more effective
training. With U-CE, we manage to train models that are naturally capable of
predicting meaningful uncertainties after training while simultaneously
improving their segmentation performance.

Our contributions can be summarized as follows: Firstly, we propose the U-CE
loss function, which utilizes uncertainty estimates to guide the optimization
process, emphasizing regions with high uncertainties. Secondly, we conduct
extensive experiments on two benchmark datasets, Cityscapes [8] and ACDC [41],
using two common backbones, ResNet-18 and ResNet-101 [15], demonstrating the
superiority of U-CE over regular CE training. Lastly, we present additional
insights, limitations, and potential improvements for U-CE through multiple
ablation studies and a thorough discussion.

##  2 Related Work

In this section, we briefly review the related work on uncertainty
quantification and uncertainty-aware segmentation.

###  2.1 Uncertainty Quantification

Deep neural networks, with their millions of model parameters and non-
linearities, have proven effective in solving complex tasks in natural
language processing [38] and computer vision, like semantic segmentation [34].
Unfortunately, due to their complexity, the computation of the exact posterior
probability distribution of the network’s output is infeasible [4, 30].
Consequently, approximate uncertainty quantification methods are employed to
offer a practical solution to tackle the intractability of the exact posterior
distribution. The most prominent methods include Bayesian Neural Networks
[31], Monte Carlo Dropout [10], and Deep Ensembles [23]. We will refer to
these methods as traditional uncertainty quantification techniques throughout
the following.

A mathematically grounded, though computationally complex, approach to
uncertainty quantification is provided by Bayesian Neural Networks, which
transform a deterministic network into a stochastic one using probabilistic
distributions placed over the activations or the weights [18]. For instance,
Bayes by Backprob [4] employs variational inference to learn approximate
distributions over the weights. These can be used to create an ensemble of
models with differently sampled weights to approximate the posterior
distribution of the predictions.

Gal and Ghahramani simplify this approximation process by using Monte Carlo
Dropout [10]. While dropout is usually applied as a regularization technique
[43], Monte Carlo Dropout uses this concept to sample from the posterior
distribution of a network’s prediction at test time. In its original form,
Monte Carlo Dropout only captures the epistemic uncertainty inherent to the
model. To obtain a more comprehensive measure of uncertainty that includes the
aleatoric uncertainty, which captures the noise inherent in the observations,
Monte Carlo Dropout can be combined with learned uncertainty predictions and
assumed density filtering [11, 21, 30].

The current state-of-the-art uncertainty quantification method are Deep
Ensembles, which consist of an ensemble of trained models that generate
diverse predictions at test time [23]. Due to the introduction of randomness
through random weight initialization or different data augmentations across
ensemble members [9], Deep Ensembles are well-calibrated [23]. Multiple
studies demonstrated that Deep Ensembles generally outperform other
uncertainty quantification methods across varying tasks [39, 48, 14]. However,
this performance gain is associated with high computational cost.

In addition to the aforementioned approximate uncertainty quantification
methods, there has been a growing interest in deterministic single forward-
pass approaches, which offer advantages in terms of memory usage and inference
time. For example, van Amersfoort _et al_. [46] and Liu _et al_. [29] explore
the concept of distance-aware output layers. While these methods demonstrate
good performance, they are not competitive with the current state-of-the-art
and require significant modifications to the training process [36]. Another
approach, proposed by Mukhoti _et al_. [36], simplifies the two previous
methods by employing Gaussian Discriminant Analysis for feature-space density
estimation after training. Although they perform on par with Deep Ensembles in
some settings, their approach still necessitates a more sophisticated training
approach. Additionally, fitting the feature-space density estimator is only
possible after training, which is not suitable for U-CE where meaningful
uncertainties are required during training.

Overall, uncertainty quantification remains an active and evolving field of
research, with various approaches offering their own advantages and
disadvantages. For our specific case, Monte Carlo Dropout emerges as the
preferred option due to its ease of use, minimal impact on the training
process, and computational efficiency compared to Deep Ensembles. Through
Monte Carlo Dropout sampling, we can compute the predictive uncertainty to
apply pixel-wise weighting of the well-known cross-entropy loss. With
predictive uncertainties, we refer to the standard deviation of the softmax
probabilities of the predicted class provided by Monte Carlo Dropout sampling.

Figure 2: A schematic overview of the training process of U-CE. U-CE
integrates the predictive uncertainties of a Monte Carlo Dropout (MC-Dropout)
model into the training process to enhance segmentation performance. In
comparison to most applications of Monte Carlo Dropout, U-CE utilizes the
uncertainties not only at test time but also dynamically during training by
applying pixel-wise weighting to the regular cross-entropy loss.

###  2.2 Uncertainty-aware Segmentation

In the domain of uncertainty-aware segmentation, researchers have explored
various techniques to incorporate uncertainty measures into the training
process. Surprisingly, traditional uncertainty quantification methods have
been largely overlooked or underutilized. We provide an overview of notable
works that leverage uncertainty-aware techniques for segmentation tasks in
various domains. Additionally, we discuss how U-CE addresses the gap towards
full utilization of traditional uncertainty quantification methods during
training.

Some of the earlier work on more effective training has originally been
designed for object detection. For example, Lin _et al_. [28] introduced the
focal loss that down-weights the contribution of easy examples to shift the
focus more towards hard examples. Another closely related technique is online
hard example mining by Shrivastava _et al_. [42]. They propose to
automatically select hard examples to only learn from them and completely
ignore the easy examples. By now, both methods have been successfully adapted
for semantic segmentation [17, 47].

Another line of work focuses on the identification and compensation of
ambiguities and label noise. Kaiser _et al_. [19] propose adding a learned
bias to a network’s logits and introducing a novel uncertainty branch to
induce the compensation bias only to relevant regions. However, unlike U-CE,
their approach does not utilize uncertainties to make training more robust,
rather they aim to avoid new noise during data annotation.

More closely related to our work, Bischke _et al_. [3] and Bressan _et al_.
[5] propose to leverage uncertainties to improve training on imbalanced aerial
image datasets. The former use the per-class uncertainty of the model together
with the median frequency to balance training [3]. We argue that dynamically
weighting each pixel individually during training, which is what U-CE does, is
even more valuable. The latter utilize pixel-wise weights, but only consider
the class and labeling uncertainty [5] instead of the predictive uncertainties
like U-CE.

In addition to these methods, Chen _et al_. [6] propose to transform the
embeddings of the last layer from Euclidean space into Hyperbolic space to
dynamically weight pixels based on the hyperbolic distance, which they
interpret as uncertainty. Similarly, Bian _et al_. [2] propose an uncertainty
estimation and segmentation module to estimate uncertainties that they use to
improve the segmentation performance. Unlike U-CE, however, these two works do
not incorporate traditional uncertainty quantification methods into training.

In contrast to existing literature, U-CE fully utilizes predictive
uncertainties dynamically during training. By pixel-wise uncertainty weighting
of the cross-entropy loss, U-CE harnesses valuable insights from the
uncertainties to guide the optimization process. This approach enables more
effective training, resulting in models that are naturally capable of
predicting meaningful uncertainties after training while also improving their
segmentation performance.

##  3 Methodology

In the following, we provide an overview of U-CE, explain our novel
uncertainty-aware cross-entropy loss and outline the implementation details.

###  3.1 Overview

The central idea of U-CE is to incorporate predictive uncertainties into the
training process to enhance segmentation performance. As depicted in Figure 2,
we propose two simple yet highly effective adaptions to the regular training
process:

  1. 1.

During training, we sample from the posterior distribution with Monte Carlo
Dropout to obtain predictive uncertainties alongside the regular segmentation
prediction.

  2. 2.

We apply pixel-wise weighting to the regular cross-entropy loss based on the
collected uncertainties.

To compute predictive uncertainties during training, we choose Monte Carlo
Dropout. It is straightforward to implement, requires minimal tuning, and is
computationally more efficient than Deep Ensembles. However, it is worth
noting that other uncertainty quantification methods could also be utilized
for U-CE. Exploring these alternatives is an interesting avenue for future
work, which we will discuss in Section 5.

###  3.2 Uncertainty-aware Cross-Entropy

Algorithm 1 PyTorch-like Pseudocode for a single training step with U-CE

$x,y=batch$

$\hat{y}=model(x)$

$\hat{y}_{\beta}=torch.empty(size=[\beta,\hat{y}.size()])$

for $\beta$ and with no_grad do

$\hat{y}_{\beta}[\beta]=model(x)$

end for

$p=torch.mean(softmax(\hat{y}_{\beta}),dim=0)$

$q=torch.std(softmax(\hat{y}_{\beta}),dim=0)$

$\sigma=q[torch.argmax(p,dim=1)]$

$loss=(1+\sigma)^{\alpha}\cdot torch.ce(\hat{y},y,reduction="none")$

return $torch.sum(loss)/torch.numel(loss)$

Segmentation Sampling. In contrast to typical usage of Monte Carlo Dropout,
U-CE incorporates the sampling process from the posterior distribution not
only at test time but also during training. To compute the necessary
uncertainties for our uncertainty-aware cross-entropy loss, we perform $\beta$
sampling iterations at each training step. This generates $\beta$ segmentation
samples in addition to the regular segmentation prediction. Notably, gradient
computation is disabled during the sampling process as it is unnecessary for
backward propagation, which relies solely on the regular segmentation
prediction. By disabling gradient computation during sampling, we reduce the
additional computational overhead of U-CE in terms of training time and GPU
memory usage.

Uncertainty-aware Cross-Entropy Loss. The final objective function of U-CE
builds upon the well-known categorical cross-entropy loss and can be defined
as:

$L_{u\mathchar 45\relax
ce}=-\frac{1}{N}\sum_{n=1}^{N}w_{n}\sum_{c=1}^{C}y_{n,c}\cdot\log(p_{n,c}),$
(1)

where $L_{u\mathchar 45\relax ce}$ is the uncertainty-aware cross-entropy loss
for a single image, $N$ is the number of pixels in the image, $C$ is the
number of classes, $y_{n,c}$ is the respective ground truth label, $p_{n,c}$
is the respective predicted softmax probability, and $w_{n}$ represents the
pixel-wise uncertainty weight. It is worth noting that Equation 1 simplifies
to the regular cross-entropy loss by setting $w_{n}$ to one for all pixels.

Pixel-wise Uncertainty Weight. The pixel-wise uncertainty weight $w_{n}$ can
be formulated as:

$w_{n}=(1+\sigma_{n})^{\alpha},$ (2)

where $\sigma_{n}$ denotes the predictive uncertainty, and $\alpha$ controls
the influence of the uncertainties in an exponential manner. The predictive
uncertainty $\sigma$ represents the standard deviation of the softmax
probabilities of the predicted class of the segmentation samples.

Pseudocode. Finally, Algorithm 1 shows how to use U-CE in the training step in
a simplified way. As mentioned earlier, the two key adaptations to the regular
training process are sampling with Monte Carlo Dropout during training and
applying pixel-wise uncertainty weighting to the regular cross-entropy loss.

##  4 Experiments

In this section, we conduct an extensive range of experiments to demonstrate
the value of incorporating predictive uncertainties into the training process.
Firstly, we provide quantitative results comparing regular CE to U-CE under
diverse settings. Secondly, we analyze qualitative examples. Lastly, we
provide multiple ablation studies.

###  4.1 Setup

Architecture. For all of our experiments, we employ DeepLabv3+ [7] as the
decoder and either a ResNet-18 or ResNet-101 [15] as the encoder. Both
backbones are commonly used for semantic segmentation [34, 49], making our
work highly comparable and serving as an excellent baseline for future
research.

Monte Carlo Dropout. In order to convert our architectures into Monte Carlo
Dropout models, we add a dropout layer after each of the four residual block
layers of the ResNets, inspired by Kendall _et al_. [20] and Gustafsson _et
al_. [14].

Training. For all training processes, we use a Stochastic Gradient Descent
(SGD) optimizer [40] with a base learning rate of 0.01, momentum of 0.9, and
weight decay of 0.0001. Additionally, we multiply the learning rate of the
decoder and segmentation head by ten. Finally, we employ polynomial learning
rate scheduling to decay the initial learning rate during the training
process, following the formula:

$lr=lr_{base}\cdot(1-\frac{iteration}{total\>iterations})^{0.9},$ (3)

where $lr$ is the current learning rate, and $lr_{base}$ is the initial base
learning rate. In all training processes, we use a batch size of 16 and train
on four NVIDIA A100 GPUs with 40 GB of memory using mixed precision [33].

Datasets. All of our experiments are based on either the Cityscapes dataset
[8] or the ACDC dataset [41]. Both datasets are publicly available street
scene datasets aimed at advancing the current state-of-the-art in autonomous
driving. The former consists of 2975 training images, 500 validation images,
and 1525 test images. The latter contains 1600 training images, 406 validation
images, and 2000 test images. Although both datasets share the same 19
evaluation classes and a void class, the ACDC dataset exclusively focuses four
adverse conditions: fog, nighttime, rain, and snow.

Data Augmentations. To prevent overfitting, we apply a common data
augmentation strategy for all training procedures, regardless of the dataset
or architecture used. The strategy includes the following steps:

  1. 1.

Random scaling with a factor between $0.5$ and $2.0$.

  2. 2.

Random cropping with a crop size of $768\times 768$ pixels.

  3. 3.

Random horizontal flipping with a flip chance of $50\%$.

Evaluation. Since both test splits are withheld for benchmarking purposes, we
utilize the validation images for testing in all our experiments. Unless
otherwise specified, we only report single forward pass results based on the
original validation images without resizing or sampling for a fair comparison
between all of the models. Also, we set the number of segmentation samples
$\beta$ to ten by default.

Metrics. For quantitative evaluations, we primarily report the mean
Intersection over Union (mIoU), also known as the Jaccard Index, to measure
the segmentation performance. In addition to the mIoU, we also utilize the
Expected Calibration Error (ECE) [37] to evaluate the calibration as well as
the mean class-wise predictive uncertainty (mUnc) to quantitatively compare
the resulting uncertainties.

###  4.2 Quantitative Evaluation

Encoder 200 Epochs 500 Epochs 3-8 CE U-CEα=1 U-CEα=10 CE U-CEα=1 U-CEα=10
Dropout (0%) RN18 70.0 - - 72.0 - - Dropout (10%) RN18 69.4 69.6 71.6 72.3
72.3 74.2 Dropout (20%) RN18 69.0 69.5 71.8 71.9 72.6 73.5 Dropout (30%) RN18
68.2 69.0 71.0 71.9 72.4 74.1 Dropout (40%) RN18 66.6 67.7 70.5 71.1 71.1 73.7
Dropout (50%) RN18 64.3 65.3 69.6 69.0 69.4 72.6 Dropout (0%) RN101 74.6 - -
76.1 - - Dropout (10%) RN101 74.8 75.1 76.1 76.3 76.6 77.5 Dropout (20%) RN101
74.6 74.8 76.6 76.3 77.0 77.7 Dropout (30%) RN101 74.5 74.7 76.1 76.4 76.6
77.5 Dropout (40%) RN101 74.7 74.0 75.8 76.1 76.5 78.2 Dropout (50%) RN101
74.1 73.7 75.9 76.6 76.6 77.3

Table 1: Quantitative comparison between regular CE and U-CE on the Cityscapes
dataset [8] for different dropout ratios. The provided numbers represent the
mIoU $\uparrow$. Best respective results are marked in bold.

Encoder 200 Epochs 500 Epochs 3-8 CE U-CEα=1 U-CEα=10 CE U-CEα=1 U-CEα=10
Dropout (0%) RN18 56.3 - - 62.2 - - Dropout (10%) RN18 55.5 56.4 60.0 62.1
62.8 65.0 Dropout (20%) RN18 54.6 56.1 60.5 61.5 62.0 65.0 Dropout (30%) RN18
52.2 54.3 59.2 59.6 61.6 64.3 Dropout (40%) RN18 48.9 50.8 58.2 56.8 58.8 63.9
Dropout (50%) RN18 47.7 49.3 56.3 53.3 56.0 62.4 Dropout (0%) RN101 65.0 - -
68.8 - - Dropout (10%) RN101 64.5 65.3 67.0 68.4 69.3 69.9 Dropout (20%) RN101
64.1 65.0 65.8 68.5 68.7 70.2 Dropout (30%) RN101 62.7 64.3 65.3 68.4 68.5
69.9 Dropout (40%) RN101 61.1 63.1 65.4 67.8 67.8 70.0 Dropout (50%) RN101
58.0 60.2 63.7 66.0 67.4 70.2

Table 2: Quantitative comparison between regular CE and U-CE on the ACDC
dataset [41] for different dropout ratios. The provided numbers represent the
mIoU $\uparrow$. Best respective results are marked in bold.

Encoder 200 Epochs 500 Epochs 3-8 mIoU $\uparrow$ ECE $\downarrow$ mUnc mIoU
$\uparrow$ ECE $\downarrow$ mUnc CE RN18 69.0 0.035 0.088 71.9 0.025 0.088
U-CEα=1 RN18 69.5 0.036 0.089 72.6 0.027 0.088 U-CEα=10 RN18 71.8 0.029 0.085
73.5 0.018 0.084 CE RN101 74.6 0.026 0.080 76.3 0.041 0.076 U-CEα=1 RN101 74.8
0.024 0.079 77.0 0.041 0.076 U-CEα=10 RN101 76.6 0.022 0.073 77.7 0.040 0.073

Table 3: A more detailed quantitative comparison between regular CE and U-CE
on the Cityscapes dataset [8] using a dropout ratio of 20%. The provided
numbers represent the mIoU $\uparrow$, ECE $\downarrow$, and mUnc.

CE

U-CEα=1

U-CEα=10

CE

U-CEα=1

U-CEα=10

(a) Image

(b) Ground Truth Label

(c) Segmentation Prediction

(d) Binary Accuracy Map

(e) Predictive Uncertainty

Figure 3: Example images from the Cityscapes and ACDC validation set (a),
corresponding ground truth labels (b), the model’s segmentation predictions
(c), a binary accuracy map (d), and the predictive uncertainty (e). White
pixels in the binary accuracy map are either incorrect predictions or void
classes, which appear black in the ground truth label. For the uncertainty
prediction, brighter pixels represent higher predictive uncertainties. The
first three rows depict results from models with a ResNet-18 backbone and
dropout ratio of $20\%$, trained for 200 epochs on Cityscapes [8]. The last
three rows show examples from models using a ResNet-101 backbone and a dropout
ratio of $20\%$, trained for 500 epochs on the ACDC dataset [41].

Tables 1 and 2 outline a quantitative comparison between regular CE and our
proposed U-CE loss using two different $\alpha$ values for various dropout
ratios and training lengths on the Cityscapes [8] and ACDC [41] datasets.
Remarkably, U-CEα=10 achieves the highest mIoU across all dropout ratios, even
outperforming the baseline models that do not use dropout in most cases.
Notably, U-CEα=10 achieves a maximum improvement of up to $9.3\%$ over regular
CE when training on ACDC [41] for 200 epochs using a ResNet-18 with a dropout
ratio of $40\%$. On average, U-CEα=10 outperforms CE by $2.0\%$ on Cityscapes
[8] and by $4.6\%$ on ACDC [41]. Interestingly, U-CEα=1 also matches or
improves upon regular CE training in most cases. On average, U-CEα=1
outperforms CE by $0.3\%$ on Cityscapes and by $1.3\%$ on ACDC.

Table 3 provides additional information on the ECE and mUnc for CE and U-CE
using a dropout ratio of $20\%$. In comparison to regular CE and U-CEα=1,
which exhibit similar results, U-CEα=10 not only improves segmentation
performance but also yields slightly better calibrated networks, as measured
by the ECE. Moreover, the mUnc is also slightly lower for U-CEα=10.

Overall, Tables 1, 2 and 3 provide strong evidence for the effectiveness of
leveraging predictive uncertainties in the training process.

###  4.3 Qualitative Evaluation

In addition to the quantitative evaluation, we also provide qualitative
examples in Figure 3 showing the original input image, the corresponding
ground truth label, the model’s segmentation prediction, a binary accuracy
map, and the student’s predictive uncertainty. The first three rows depict
results from models with a ResNet-18 backbone and a dropout ratio of $20\%$,
trained for 200 epochs with CE, U-CEα=1, U-CEα=10 on Cityscapes [8]. The last
three rows show examples from models using a ResNet-101 backbone and a dropout
ratio of $20\%$, trained for 500 epochs on the ACDC dataset [41]. The binary
accuracy map visualizes incorrectly predicted pixels and void classes in
white, and correctly predicted pixels in black.

Generally, for large areas and well-represented classes like road, building,
sky, and car, all models perform exceptionally well with minimal errors.
Furthermore, there is a strong correlation between the binary accuracy map and
the predictive uncertainty, indicating that all models provide meaningful
uncertainties.

Nonetheless, there are nuanced differences between the models. For example, in
the first two rows of Figure 3, which represent models trained CE and U-CEα=1,
there are noticeable misclassifications on top of the human standing in front
of the truck. Naturally, this area is also accompanied with high
uncertainties. In contrast, the model trained with U-CEα=10 exhibits
significantly fewer difficulties, resulting in a better segmentation
prediction and lower uncertainties.

A similar situation is observable in the last three rows, showing examples
from the more challenging ACDC dataset [41]. Here, the model trained with
regular CE struggles to correctly segment the truck on the left as well as
differentiate between the sidewalk and the terrain on the right side of the
image. The model trained with U-CEα=1 does slightly better in these areas, but
is equally uncertain. Only the model trained with U-CEα=10 successfully
classifies the truck and differentiates between the sidewalk and the terrain
decently. Consequently, the predictive uncertainty is also lower in these
areas.

In summary, the qualitative findings presented in Figure 3 concur with our
quantitative evaluation, manifesting the efficacy of U-CE across different
datasets and architectures.

###  4.4 Ablation Studies

$\alpha$ 1 2 4 6 8 10 12 14 16 RN18 (Cityscapes) 69.5 70.0 70.7 71.2 71.5 71.8
71.0 47.0 70.9 RN101 (Cityscapes) 74.8 75.2 75.6 76.1 76.4 76.6 76.3 75.8 72.6
RN18 (ACDC) 56.1 56.9 57.6 58.8 58.8 60.5 60.3 60.1 37.5 RN101 (ACDC) 65.0
65.0 65.7 65.5 66.0 65.8 66.7 64.5 19.9

Table 4: Ablation study on the impact of $\alpha$. The provided numbers
represent the mIoU $\uparrow$. Best respective results are marked in bold.

$\beta$ 0 2 6 10 14 18 CE 69.0 (1:49) - - - - - U-CEα=10 - 71.1 (1:52) 71.6
(2:01) 71.6 (2:27) 71.6 (2:53) 71.7 (3:17)

Table 5: Ablation study on the number of segmentation samples $\beta$. In
addition to the mIoU $\uparrow$, we provide the training time in hours:minutes
$\downarrow$ in paranthesis.

Random Flipping Random Scaling mIoU $\uparrow$ CE $\times$ $\times$ 66.1 ✓
$\times$ 67.0 $\times$ ✓ 68.6 ✓ ✓ 69.0 U-CEα=1 $\times$ $\times$ 65.8 ✓
$\times$ 67.8 $\times$ ✓ 69.1 ✓ ✓ 69.5 U-CEα=10 $\times$ $\times$ 69.6 ✓
$\times$ 70.1 $\times$ ✓ 71.8 ✓ ✓ 71.8

Table 6: Ablation study on the impact of various data augmentations
strategies. We use random cropping with a crop size of $768\times 768$ pixels
as a baseline for all strategies.

$lr_{base}$ $10^{-1}$ $10^{-2}$ $10^{-3}$ $10^{-4}$ $10^{-5}$ CE 50.5 69.0
55.9 35.6 18.9 U-CEα=1 56.0 69.5 57.6 36.9 19.3 U-CEα=10 2.0 71.8 65.0 47.6
25.3

Table 7: Ablation study on the base learning rate $lr_{base}$. The provided
numbers represent the mIoU $\uparrow$. Best results are marked in bold.

In addition to the quantitative and qualitative evaluation, we also present
multiple ablation studies. Unless otherwise noted, we confined all of the
ablation studies to models that use a ResNet-18 as the backbone, have a
dropout ratio of 20%, and were trained for 200 epochs.

Impact of $\alpha$. The most influential hyperparameter of U-CE is $\alpha$ as
it exponentially controls the weighting of the CE loss. Table 4 demonstrates
the impact of different $\alpha$ values on the mIoU for both backbones,
ResNet-18 (RN18) and ResNet-101 (RN101), on both Cityscapes and ACDC.
Evidently, the segmentation performance consistently improves as $\alpha$
increases until it reaches ten, which stands as the best value in three out of
four cases across the two datasets and architectures. Thus, using ten as the
default value for $\alpha$ seems to be a fair estimation to achieve the best
results, not only for the mentioned cases but potentially for other
applications as well. Further increasing $\alpha$ leads to a degradation in
mIoU. Additionally, training becomes more unstable as models overly focus on
uncertain pixels, resulting in some models failing to converge properly.
Nonetheless, U-CE exhibits robustness against changes in $\alpha$, offering a
wide range of valid hyperparameters that lead to improved segmentation results
compared to regular CE training.

Impact of $\beta$. Table 5 exhibits another ablation study on the number of
segmentation samples $\beta$. Interestingly, there is no clear benefit of
sampling more often than six times, especially with regard to the training
time. As indicated by the training times, U-CEβ=6 increases the necessary
training time by approximately $10\%$, whereas U-CEβ=10 extends it by roughly
$35\%$. For comparison, Gal and Ghahramani [10] recommend sampling ten times
to get a reasonable estimation of the predictive mean and uncertainty.

Impact of Data Augmentations. The impact of various data augmentation
strategies on CE and U-CE is demonstrated in Table 6. The results show that
incorporating additional data augmentations on top of the baseline strategy of
random cropping with a crop size of $768\times 768$ pixels improves the mIoU
across the board. More importantly, this ablation study confirms that U-CE
consistently outperforms CE across different data augmentation strategies,
indicating its effectiveness in improving segmentation performance.

Impact of $lr_{base}$. Table 7 shows the ablation study on the base learning
rate $lr_{base}$. The most notable comparison is between regular CE and
U-CEα=1, which demonstrates that U-CE is not limited to specific learning
rates. U-CEα=1 consistently outperforms regular CE for all examined base
learning rates, despite increasing the training loss by approximately $9\%$ as
indicated by the mUnc in Table 3. Moreover, U-CEα=10 exceeds the results of CE
and U-CEα=1 for all base learning rates except $10^{-1}$, which caused
divergence. Overall, this ablation study confirms the value of leveraging
predictive uncertainties during training, irrespective of the learning rate,
which is arguably the single most important hyperparameter in deep learning
[1].

##  5 Discussion

In contrast to previous approaches, U-CE fully leverages predictive
uncertainties obtained by Monte Carlo Dropout during training. As a result, we
manage to train models that not only improve their segmentation performance
but are also naturally capable of predicting meaningful uncertainties after
training as well.

While U-CE appears to have no apparent shortcomings, except for a minor
increase in training time, we acknowledge the need for a transparent
discussion about its potential limitations. Our aim is to effectively guide
future work in pushing the boundaries of state-of-the-art techniques,
especially in safety-critical applications like autonomous driving.

Limitations. One limitation of U-CE arises in the absence of densely annotated
ground truth labels. If most pixels are either labeled as background or
designated to be ignored while training, U-CE will likely offer next to no
benefit, except for a higher loss around object boundaries. Additionally, U-CE
may not contribute to improved segmentation performance if the network is
already overfitting the training data. Having said that, the impact of U-CE on
generalization needs further examination.

Future Work. With regards to future work, we have multiple suggestions that
might be worth investigating. Potentially, the results of U-CE could be
further improved if the quality of the uncertainty estimates would be better.
Therefore, it would be interesting to integrate Deep Ensembles [23], the
state-of-the-art uncertainty quantification method [39, 48, 14], with U-CE,
which we could not realize because of computational restrains. On a similar
note, it could be worth employing warmup epochs, which we omitted to refrain
from introducing another hyperparameter. Additionally, we would like to see
$\alpha$ removed from U-CE by incorporating statistical hypothesis testing.
This would be beneficial in two ways: Firstly, it would remove the most
influential hyperparameter of U-CE. Secondly, and maybe more importantly, it
would leverage all of the available uncertainties and not just the predictive
uncertainty. Finally, we encourage other researchers to incorporate U-CE into
state-of-the-art semantic segmentation approaches and to explore its
usefulness in other computer vision tasks that rely on pixel-wise predictions,
such as depth estimation.

Overall, we believe that U-CE presents a promising paradigm in semantic
segmentation by dynamically leveraging uncertainties to create more robust and
reliable models. Despite a minor increase in training time and room for
further improvement, we see no reason not to employ U-CE in comparison to
regular CE.

##  6 Conclusion

In this paper, we introduced U-CE, a novel uncertainty-aware cross-entropy
loss for semantic segmentation. U-CE incorporates predictive uncertainties,
based on Monte Carlo Dropout, into the training process through pixel-wise
weighting of the regular cross-entropy loss. As a result, we manage to train
models that are naturally capable of predicting meaningful uncertainties after
training while simultaneously improving their segmentation performance.
Through extensive experimentation on the Cityscapes and ACDC datasets using
ResNet-18 and ResNet-101 architectures, we demonstrated the superiority of
U-CE over regular cross-entropy training.

We hope that U-CE and our thorough discussion of potential limitations and
future work contribute to the development of more robust and trustworthy
segmentation models, ultimately advancing the state-of-the-art in safety-
critical applications and beyond.

## Acknowledgment

The authors acknowledge support by the state of Baden-Württemberg through
bwHPC.

This work is supported by the Helmholtz Association Initiative and Networking
Fund on the HAICORE@KIT partition.

## References

  * [1] Yoshua Bengio.  Practical recommendations for gradient-based training of deep architectures.  In Neural Networks: Tricks of the Trade: Second Edition, pages 437–478. Springer, 2012.
  * [2] Cheng Bian, Chenglang Yuan, Jiexiang Wang, Meng Li, Xin Yang, Shuang Yu, Kai Ma, Jin Yuan, and Yefeng Zheng.  Uncertainty-aware domain alignment for anatomical structure segmentation.  Medical Image Analysis, 64:101732, 2020.
  * [3] Benjamin Bischke, Patrick Helber, Damian Borth, and Andreas Dengel.  Segmentation of imbalanced classes in satellite imagery using adaptive uncertainty weighted class loss.  In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pages 6191–6194. IEEE, 2018.
  * [4] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1613–1622, Lille, France, 2015. PMLR.
  * [5] Patrik Olã Bressan, José Marcato Junior, José Augusto Correa Martins, Maximilian Jaderson de Melo, Diogo Nunes Gonçalves, Daniel Matte Freitas, Ana Paula Marques Ramos, Michelle Taís Garcia Furuya, Lucas Prado Osco, Jonathan de Andrade Silva, et al.  Semantic segmentation with labeling uncertainty and class imbalance applied to vegetation mapping.  International Journal of Applied Earth Observation and Geoinformation, 108:102690, 2022.
  * [6] Bike Chen, Wei Peng, Xiaofeng Cao, and Juha Röning.  Hyperbolic uncertainty aware semantic segmentation.  arXiv preprint arXiv:2203.08881, 2022.
  * [7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [8] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [9] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.  Deep Ensembles: A Loss Landscape Perspective.  arXiv:1912.02757, 2020.
  * [10] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, New York, New York, USA, 20–22 Jun 2016. PMLR.
  * [11] Jochen Gast and Stefan Roth.  Lightweight Probabilistic Deep Networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3369–3378, 2018.
  * [12] Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, Muhammad Shahzad, Wen Yang, Richard Bamler, and Xiao Xiang Zhu.  A Survey of Uncertainty in Deep Neural Networks.  arXiv:2107.03342, 2022.
  * [13] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.  On calibration of modern neural networks.  In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR, 06–11 Aug 2017\.
  * [14] Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schon.  Evaluating scalable bayesian deep learning methods for robust computer vision.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 318–319, 2020.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [16] Michael Heizmann, Alexander Braun, Markus Glitzner, Matthias Günther, Günther Hasna, Christina Klüver, Jakob Krooß, Erik Marquardt, Michael Overdick, and Markus Ulrich.  Implementing machine learning: chances and challenges.  at-Automatisierungstechnik, 70(1):90–101, 2022.
  * [17] Shruti Jadon.  A survey of loss functions for semantic segmentation.  In 2020 IEEE conference on computational intelligence in bioinformatics and computational biology (CIBCB), pages 1–7. IEEE, 2020.
  * [18] Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun.  Hands-on bayesian neural networks—a tutorial for deep learning users.  IEEE Computational Intelligence Magazine, 17(2):29–48, 2022.
  * [19] Timo Kaiser, Christoph Reinders, and Bodo Rosenhahn.  Compensation learning in semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3266–3277, 2023.
  * [20] Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla.  Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding.  arXiv preprint arXiv:1511.02680, 2015.
  * [21] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 5580–5590. Curran Associates Inc., 2017.
  * [22] Justin Kruger and David Dunning.  Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments.  Journal of personality and social psychology, 77(6):1121, 1999.
  * [23] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [24] Steven Landgraf, Markus Hillemann, Moritz Aberle, Valentin Jung, and Markus Ulrich.  Segmentation of industrial burner flames: A comparative study from traditional image processing to machine and deep learning.  arXiv preprint arXiv:2306.14789, 2023.
  * [25] Steven Landgraf, Kira Wursthorn, Markus Hillemann, and Markus Ulrich.  Dudes: Deep uncertainty distillation using ensembles for semantic segmentation.  arXiv preprint arXiv:2303.09843, 2023.
  * [26] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin.  Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples.  arXiv:1711.09325, 2018.
  * [27] Christian Leibig, Vaneeda Allken, Murat Seçkin Ayhan, Philipp Berens, and Siegfried Wahl.  Leveraging uncertainty information from deep neural networks for disease detection.  Scientific Reports, 7(1):17816, 2017.
  * [28] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [29] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan.  Simple and principled uncertainty estimation with deterministic deep learning via distance awareness.  Advances in Neural Information Processing Systems, 33:7498–7512, 2020.
  * [30] Antonio Loquercio, Mattia Segu, and Davide Scaramuzza.  A General Framework for Uncertainty Estimation in Deep Learning.  IEEE Robotics and Automation Letters, 5(2):3153–3160, 2020.
  * [31] David J. C. MacKay.  A Practical Bayesian Framework for Backpropagation Networks.  Neural Computation, 4(3):448–472, 1992.
  * [32] Rowan McAllister, Yarin Gal, Alex Kendall, Mark van der Wilk, Amar Shah, Roberto Cipolla, and Adrian Weller.  Concrete Problems for Autonomous Vehicle Safety: Advantages of Bayesian Deep Learning.  In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 4745–4753, Melbourne, Australia, 2017. International Joint Conferences on Artificial Intelligence Organization.
  * [33] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al.  Mixed precision training.  arXiv preprint arXiv:1710.03740, 2017.
  * [34] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3523–3542, 2022.
  * [35] Jishnu Mukhoti and Yarin Gal.  Evaluating bayesian deep learning methods for semantic segmentation.  arXiv preprint arXiv:1811.12709, 2018.
  * [36] Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal.  Deep deterministic uncertainty: A new simple baseline.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24384–24394, 2023.
  * [37] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
  * [38] Daniel W Otter, Julian R Medina, and Jugal K Kalita.  A survey of the usages of deep learning for natural language processing.  IEEE transactions on neural networks and learning systems, 32(2):604–624, 2020.
  * [39] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [40] Herbert Robbins and Sutton Monro.  A Stochastic Approximation Method.  The Annals of Mathematical Statistics, 22(3):400–407, 1951.
  * [41] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021.
  * [42] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick.  Training region-based object detectors with online hard example mining.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761–769, 2016.
  * [43] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research, 15(56):1929–1958, 2014.
  * [44] Carsten Steger, Markus Ulrich, and Christian Wiedemann.  Machine Vision Algorithms and Applications.  John Wiley & Sons, 2018.
  * [45] Markus Ulrich and Markus Hillemann.  Generic hand–eye calibration of uncertain robots.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11060–11066, 2021.
  * [46] Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal.  Uncertainty estimation using a single deep deterministic neural network.  In International conference on machine learning, pages 9690–9700. PMLR, 2020.
  * [47] Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le.  Semi-supervised semantic segmentation using unreliable pseudo-labels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4248–4257, June 2022.
  * [48] Kira Wursthorn, Markus Hillemann, and Markus Ulrich.  Comparison of uncertainty quantification methods for CNN-based regression.  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLIII-B2-2022:721–728, 2022.
  * [49] Rongyu Zhang, Lixuan Du, Qi Xiao, and Jiaming Liu.  Comparison of backbones for semantic segmentation network.  In Journal of Physics: Conference Series, volume 1544, page 012196\. IOP Publishing, 2020.


# Steiner systems S(2,6,121/126) based on difference families

Ivan Hetman

(Date: January 2024)

###### Abstract.

In this paper new Steiner systems $S(2,6,121)$, $S(2,6,126)$ are introduced.
Also some non-existence results for line lengths $7..9$ are presented. There
is no solid proof that presented algorithm is exhaustive or correct, but it
produces same results on already known difference families for line lengths
$3..5$. Due to calculation-based approach this paper probably won’t be
published, but will be submitted to arxiv as it contains some new results.

##  1\. Introduction

There exist enumerations for non-isomorphic difference families in [1] and
[2]. According to this results, number of non-isomorphic $(v,k,1)$ difference
families are known for $k=3$ and $v\leq 61$, $k=4$ and $v\leq 76$, $k=5$ and
$v\leq 85$, $k=6$ and $v\leq 96$ and $k=7$ and $v\leq 91$. Every difference
family produces balanced incomplete block design $BIBD(v,k,\lambda)$. Number
of BIBD is significantly larger than the family of difference families with
same parameters. Still, in [1] only 1 non-isomorphic $BIBD(121,6,1)$ is known
and only 2 non-isomorphic $BIBD(126,6,1)$.

##  2\. Algorithm description

We will construct base blocks over $\mathbb{Z}_{v}$ one by one. Blocks are
lexicographically ordered. If $k|v$ then last block is always
$(0,k,2k,...,(v-k))$. First item of block is always $0$, second item of block
should produce maximal difference between two consecutive members of the block
(including last and first = 0). Then on from second to last block we iterate
limited possible numbers. Also we have filter which contains ”forbidden”
numbers which will make construction of difference family impossible. If
specific block produces same difference $\Delta
B=\\{b_{i}-b_{j}|i,j=1,...,k;i\not=j\\}$ as already computed, we filter it out
because it will just repeat same calculation. In the end we add filtered
difference family to the final list by ”mirroring” procedure.

Difference family search algorithm is easily parallelizable by second number
in first block. This fact helps to utilize modern multi-core hardware. Search
was run on 24-core PC. Simple parallelization of the algorithm led to 10x
performance boost. Code is present [4] here.

Let’s consider $v=15$ and $k=3$. As $3|15$, then we automatically have the
last block $(0,5,10)$. Let’s calculate first two blocks. After the search
using base algorithm, we obtain first block $(0,7,9)$. Second block is then
$(0,11,12)$. We see that this two blocks form difference family with
$(0,5,10)$. Therefore it’s desired difference family. As we limited first step
by selecting only unique differences, we need to conduct ”mirroring”
procedure. To mirror any block, we fix first two numbers $b_{1}=0$ and
$b_{2}$, and apply transformation $b_{i}\rightarrow b_{1}+(v-b_{i})$ to the
rest. For instance, ”mirror” of $(0,7,9)$ is $(0,7,13)$ and ”mirror” of
$(0,11,12)$ is $(0,11,14)$. If we mirror all blocks simultaneously it can be
easily seen that obtained difference families are isomorphic. So, we need to
conduct $2^{\lfloor\frac{v}{k(k-1)}\rfloor-1}$ different mirrorings to obtain
all possibly non-isomorphic difference families. In the end we filter out
isomorphic difference families. For $v=15$ and $k=3$ nothing is filtered, but
for $v=13$ and $k=3$, two difference families obtained by mirroring are
isomorphic.

##  3\. Results

Applying described algorithm gives us same numbers that are in [1] for known
difference families. Still, number of non-isomorphic difference families for
$v=73$ and $k=4$ in [2] is $1426986$. But same calculation using algorithm
described above gives us $1428546>1426986$ number.

Also using same algorithm it was possible to get some other low-hanging
fruits. In addition to [3] result of non-existence of $(127,7,1)$ difference
family, it was established non-existence of $(133,7,1)$, $(113,8,1)$,
$(120,8,1)$, $(145,9,1)$ and $(153,9,1)$ difference families.

Bigger numbers require significantly bigger run times. For instance, together
with non-exhaustive search for $(121,6,1)$ and $(126,6,1)$ designs, search for
$(169,8,1)$ and $(176,8,1)$ was launched. And during 20-days computation no
difference family for last two was found. Other results of this computation
are listed below.

######  Example 3.1.

List of 6 difference families $(121,6,1)$. ”Mirrorings” of any of the sets
produce 8 non-isomorphic difference family yielding 48 non-isomorphic
difference families.

  * •

{{0, 25, 37, 55, 76, 99}, {0, 52, 57, 72, 110, 113}, {0, 54, 71, 81, 90, 97},
{0, 73, 75, 79, 107, 108}}

  * •

{{0, 25, 45, 66, 79, 97}, {0, 39, 46, 51, 83, 99}, {0, 62, 63, 65, 71, 98},
{0, 64, 74, 78, 93, 104}}

  * •

{{0, 26, 35, 54, 72, 97}, {0, 40, 47, 52, 91, 113}, {0, 42, 53, 57, 63, 80},
{0, 56, 76, 89, 90, 92}}

  * •

{{0, 26, 41, 53, 73, 96}, {0, 50, 52, 87, 111, 117}, {0, 63, 72, 77, 105,
108}, {0, 64, 75, 82, 83, 104}}

  * •

{{0, 29, 31, 56, 80, 93}, {0, 40, 46, 50, 66, 98}, {0, 45, 60, 67, 78, 79},
{0, 47, 68, 77, 82, 85}}

  * •

{{0, 38, 39, 56, 61, 85}, {0, 45, 51, 58, 66, 78}, {0, 49, 53, 79, 90, 93},
{0, 52, 54, 86, 102, 111}}

######  Example 3.2.

List of 8 difference families $(126,6,1)$. ”Mirrorings” of any of the sets
produce 8 non-isomorphic difference family yielding 64 non-isomorphic
difference families. Last block $\\{0,21,42,63,84,105\\}$ is omitted.

  * •

{{0, 26, 38, 56, 81, 103}, {0, 40, 48, 68, 99, 102}, {0, 41, 57, 93, 94, 107},
{0, 80, 82, 87, 91, 97}}

  * •

{{0, 28, 46, 68, 85, 101}, {0, 45, 52, 65, 95, 99}, {0, 49, 59, 64, 78, 115},
{0, 82, 88, 90, 91, 114}}

  * •

{{0, 30, 50, 69, 95, 123}, {0, 34, 59, 77, 82, 94}, {0, 64, 68, 79, 119, 120},
{0, 80, 88, 90, 104, 117}}

  * •

{{0, 35, 36, 64, 89, 103}, {0, 44, 60, 70, 77, 92}, {0, 55, 57, 61, 102, 107},
{0, 75, 83, 86, 95, 113}}

  * •

{{0, 36, 37, 47, 77, 108}, {0, 38, 52, 64, 91, 97}, {0, 46, 50, 66, 69, 94},
{0, 68, 70, 75, 83, 92}}

  * •

{{0, 36, 37, 64, 93, 115}, {0, 45, 52, 83, 106, 118}, {0, 67, 71, 77, 101,
117}, {0, 68, 82, 85, 87, 100}}

  * •

{{0, 36, 38, 48, 75, 95}, {0, 52, 68, 81, 82, 86}, {0, 53, 62, 70, 77, 103},
{0, 55, 61, 80, 83, 115}}

  * •

{{0, 47, 48, 67, 91, 123}, {0, 49, 66, 74, 111, 120}, {0, 57, 61, 68, 90, 95},
{0, 73, 85, 87, 103, 113}}

## References

  * [1] C. J. Coulborn, J. H. Dinitz Handbook of Combinatorial Designs, Second Edition, 2007.
  * [2] Baicheva T., Topalova S. Classification results for (v, k, 1) cyclic difference families with small parameters. In: Deza M., Petitjean M., Markov K. (eds.) Mathematics of Distances and Applications, pp. 24—30. International Book Series: Information Science and Computing, Book 25ITHEA Publication, Sofia
  * [3] Baicheva T., Topalova S. Classification of optimal (v, k, 1) binary cyclically permutable constant weight codes with k=5, 6 and 7 and small lengths. Des. Codes Cryptogr. 87, 365–374 (2019). https://doi.org/10.1007/s10623-018-0534-x
  * [4] Github repository, 2023.


11institutetext: University of Milano-Bicocca, Viale Sarca 336, 20126, Milano,
Italy
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>22institutetext: NeuroMI, Milan Center for
Neuroscience, Piazza dell’Ateneo Nuovo 1, 20126, Milano, Italy

# Inner speech recognition through electroencephalographic signals

Francesca Gasparini 1122 0000-0002-6279-6660 Elisa Cazzaniga 11 Aurora Saibene
1122 0000-0002-4405-8234

###### Abstract

This work focuses on inner speech recognition starting from EEG signals. Inner
speech recognition is defined as the internalized process in which the person
thinks in pure meanings, generally associated with an auditory imagery of own
inner “voice”. The decoding of the EEG into text should be understood as the
classification of a limited number of words (commands) or the presence of
phonemes (units of sound that make up words). Speech-related BCIs provide
effective vocal communication strategies for controlling devices through
speech commands interpreted from brain signals, improving the quality of life
of people who have lost the capability to speak, by restoring communication
with their environment. Two public inner speech datasets are analysed. Using
this data, some classification models are studied and implemented starting
from basic methods such as Support Vector Machines, to ensemble methods such
as the eXtreme Gradient Boosting classifier up to the use of neural networks
such as Long Short Term Memory (LSTM) and Bidirectional Long Short Term Memory
(BiLSTM). With the LSTM and BiLSTM models, generally not used in the
literature of inner speech recognition, results in line with or superior to
those present in the state-of-the-art are obtained.

###### Keywords:

EEG inner speech recognition BCI

##  1 Introduction

Human speech production is a complex motor process that starts in the brain
and ends with respiratory, laryngeal, and articulatory gestures for creating
acoustic signals of verbal communication. Physiological measurements using
specialized sensors and methods can be made at each level of speech
processing, including the central and peripheral nervous systems, muscular
action potentials, speech kinematics (tongue, lips, jaw), and sound pressure
[25]. However, there are cases of subjects suffering from neurodegenerative
diseases or motor disorders that prevent the normal activity of signal
transmission from the brain to the peripheral areas. These subjects are
prevented from communicating or carrying out certain actions.

Brain Computer Interfaces (BCIs) are promising technologies for improving the
quality of life of people who have lost the capability to move or speak, by
restoring communication with their environment. A BCI is a system that makes
possible the interaction between an individual and a computer without using
the brain’s normal output pathways of peripheral nerves and muscles. In
particular, speech-related BCI technologies provide neuro-prosthetic help for
people with speaking disabilities, neuro-muscular disorders and diseases. It
can equip these users with a medium to communicate and express their thoughts,
thereby improving the quality of rehabilitation and clinical neurology [23].
Speech-related paradigms, based on either silent, imagined or inner speech
provide a more natural way for controlling external devices [20].

There are different types of brain-signal recording techniques that are mainly
divided into invasive or non-invasive methods. The first ones involve
implanting electrodes directly into the brain. They provide better spatial and
temporal resolution, also increasing the quality of the signal obtained.
However, invasive technologies have problems related to usability and the need
for surgical intervention on the subject. This is why non-invasive techniques
are increasingly used in BCI research. Among the non-invasive technologies,
the electroencephalogram (EEG) is the most used method for measuring the
electrical activity of the brain from the human scalp. It has an exceedingly
high time resolution, it is simple to record and it is sufficiently
inexpensive [13]. Over the years, EEG hardware technology has evolved and
several wireless multichannel systems have emerged that deliver high quality
EEG and physiological signals in a simpler, more convenient and comfortable
design than the traditional, cumbersome systems.

This paper focuses on inner speech recognition starting from EEG signals,
where the basic definition of inner speech is [1] ”the subjective experience
of language in the absence of overt and audible articulation”.

As suggested in [3], there is evidence from past neuroscience research that
inner speech engages brain regions that are commonly associated with language
comprehension and production [2]. This includes temporal, frontal and
sensorimotor areas predominantly in the left hemisphere of the brain [2] [4].
Therefore, by monitoring these brain areas, it is theoretically possible to
develop an inner speech BCI that classifies neural representations of imagined
word [4].

In section 2 the studies in the field of inner speech are described. Section 3
presents the two publicly available datasets used for the analyses proposed in
section 4. In section 5 the results obtained with our models are presented and
discussed. Finally, in section 6 some conclusions are proposed.

##  2 Related works

Most studies on classification of inner speech focus on invasive methods such
as electrocorticography (ECoG) [17] as they provide higher spatial resolution
while fewer studies concerning inner speech classification using EEG data are
available [21]. It is important for a BCI application to be non-invasive,
accessible and easy to implement so that it can be used by a large number of
subjects.

Inner speech recognition is generally faced considering phonemes, in general
vowels or syllables, such as /ba/ or /ku/, or simple words such as left,
right, up and down, in subject-dependent approaches.

Preliminary works were conducted with very few participants and syllables by
D’Zmura et al. [11], where EEG waveform envelopes have been adopted to
recognize EEG patterns. Also Brigham and Kumar [5] and Deng et al. [10]
considered the recognition of two syllables. In the first work, the accuracy
obtained for the 7 subjects ranges from 46% to 88%. They preprocessed raw EEG
data to reduce the effects of artifacts and noise, and applied k-Nearest
Neighbor classifier to autoregressive coefficients extracted as features.
While Deng end colleagues using Hilbert spectra and linear discriminant
analysis recognized the two syllables imagined in three different rhythms, for
a 6 classes task, with accuracy ranging from 19% to 22%.

Considering works where recognition of phonemes have been investigated, Da
Salla et.al [9] and [8], analyzed the recognition of three tasks: /a/, /u/,
and rest, obtaining from 68% to 79% of accuracy by using common spatial
patterns. On the same dataset, several other researchers have been tested
different models obtaining promising results [14], [22].

Kim et al. [15], instead, consider three vowels /a/, /i/ and /u/ and applied
multivariate empirical mode decomposition and common spatial pattern for
feature extraction together with linear discriminant analysis, reaching around
70 % of accuracy.

Few representative studies that try to recognize imagined words using EEG data
are reported in the literature. Given the complexity of the task, the number
of terms considered is generally limited.
Suppes et al. [26] proposed an experiment in which five subjects performed the
internal speech considering the following words: first, second, third, yes,
no, right, and left for all subjects with the addition of to, too and hear for
the last three subjects.
In the work performed by Wang at al. [27], eight Chinese subjects were
required to read in mind two Chinese characters (that meant left and one).
They were able to distinguish between the two characters and the rest state.
Feature vectors of EEG signals were extracted using CSP, and then these
vectors were classified with SVM. Accuracies between 73.65% and 95.76% were
obtained when comparing between each of the imagined words and the rest state.
A mean accuracy of 82.3% was achieved between the two words themselves.
Salama et al. [24] implemented different types of classifiers such as SVM,
discriminant analysis, self-organizing map, feed-forward back-propagation and
a combination of them, to recognize two words (Yes and No). They used a single
electrode EEG device to collect data from seven subjects and the accuracy
obtained ranges from 57% to 59%.
In [18], Mohanchandra at al. constructed a one-against-all multiclass SVM
classifier to discriminate five subvocalized words (water, help, thanks, food,
and stop) and reported an accuracy ranging from 60% to 92%.
In the González-Castañeda at al. [12] analyses, some techniques of
sonification and textification are applied, which allows to characterize EEG
signals as either an audio signal or a text document. Five imagined words (up,
down, left, right) and 27 subjects are considered. The average accuracy rate
using the EEG textified signals were 83.34%.
Using the data from six subjects, [19] reported an average accuracy of 50.1%
for the three-short words (in, out and up) classification problem and 66.2%
for the two long words classification problem (cooperate and independent),
using a Multi-Class Relevance Vector Machine (MRVM). In order to evaluate the
effect of the sound, three phonemes were used, namely /a/, /i/ and /u/
obtaining an accuracy of 49.0%. Coretto et al. [7] that collected one of the
two datasets considered in this paper, reported a mean recognition rate of
22.32% in classifying five Spanish vowels and 18.58% in classifying six
Spanish words using a Random Forest (RF) algorithm.Using the same dataset, in
[6] 30.00% and 24.97% accuracies are obtained respectively for vowels and
words using CNNs.
Recently, Bram van den Berg et al. [3], working on the Thinking Out Loud
dataset [20], also considered in our work, reported an average accuracy of
29.7% for a four-word classification task using a 2D CNN based on the EEGNet
architecture [16].

##  3 Datasets

The testing of the proposed strategies is performed on two publicly available
datasets, i.e., the Thinking Out Loud [20] and the Imagined Speech [7]
datasets. In particular, the last dataset is used to check the validity of the
resulting best approach for the Thinking Out Loud one.

###  3.1 Thinking Out Loud dataset

The first literature dataset chosen to conduct the subsequent analyses is the
Thinking Out Loud [20] one, which is focused on an inner speech paradigm
intended for the control of a BCI system through imagination of Spanish words.
The selected Spanish words are arriba (up), abajo (down), derecha (right), and
izquierda (left). Notice that the words were presented randomly with a visual
cue.
Ten (four females) healthy right-handed subjects with mean $\pm$ std age 34
$\pm$ 10, without any hearing or speech loss, nor any previous BCI experience,
participated in the experiment, which consisted of three experimental
conditions, i.e., inner/pronounced speech and visualized condition. During the
inner speech condition the participant was asked to imagine his/her own voice,
repeating the corresponding word. Instead, the participant was asked to
repeatedly pronounce aloud the word corresponding to each visual cue during
the pronounced speech condition. Finally, the visualized condition
corresponded to the tasks during which the participant was asked to focus on
mentally moving a circle shown in the center of a screen in the direction
indicated by a visual cue.

Each subject participated in 3 consecutive sessions (200 words/session),
separated by a break. Each session consisted of a baseline recording (15s),
the pronounced speech run, two inner speech runs, and two visualized condition
runs. Each run was constituted by a series of trials containing the different
experimental tasks.
Figure 1 shows the organization of each inner speech trial, under
investigation in this paper. A white circle was shown in the center of the
screen and the subject was asked to stare at it without blinking.
Subsequently, a white triangle was shown pointing in one of the four
directions corresponding to the chosen Spanish words. When the triangle
disappeared and the white circle was presented again, the subject had to
perform the indicated task. The task execution had to be stopped when the
white circle turned blue. The subject has been asked to control eye blinking
until the circle disappeared. Finally, to evaluate the participants’
attention, the subjects were asked to indicate the last inner speech and
visualized conditions after a random number of trials. The subject answered
using keyboard arrows and feedback was displayed.

Figure 1: Trial workflow taken from the Thinking Out Loud dataset original
paper [20].

The data acquisition was performed using 128 active EEG wet electrodes and 8
external active EOG/EMG wet electrodes. The resolution was of 24 bits
resolution and 1024 Hz sampling rate applied.

The EEG signals of the Thinking Out Loud dataset were preprocessed by its
authors. The preprocessing included a band pass filter between 0.5-100 Hz, a
notch filter at 50 Hz and downsampling to 254 Hz.

Please, find further details on the original dataset paper [20].

###  3.2 Imagined Speech dataset

The Imagine Speech dataset [7] was chosen to confirm the validity of the model
obtaining the best results on the Thinking Out Loud dataset. In fact, similar
experimental conditions are presented considering Spanish words and also
vowels.
In fact, the vowels /a/, /e/, /i /, /o/ and /u/ have been selected due to
their acoustic stationarity, simplicity and lack of meaning by themselves.
While the Spanish words arriba (up), abajo (down), derecha (right), izquierda
(left), adelante (forward), and atras (backward) were chosen as possible BCI
commands to control the movements of an external device.

Fifteen (seven females) healthy subjects with mean age of 25 years old,
without any hearing or speech loss, participated in the experiment. Only one
of the subjects reported to be left-handed, while the rest were right-handed.
EEG signals were recorded under two conditions: imagined speech and pronounced
speech. During imagined speech, the subjects had to imagine pronouncing the
word without moving muscles or producing sounds.
Target stimuli were presented in a sequence comprised of four intervals of
predefined duration (Figure 2). During the ready interval (2 s), the subject
is informed that the rest interval finished and a new cue would be displayed
soon. Afterwards, the target word is presented, both visually and
acoustically, during the stimulus presentation interval (2 s). In the
Imagine/Pronounce stage an image displays the task requested (either imagined
or pronounced speech). It is in this stage that the subject has to imagine the
pronunciation or pronounce the word given as a cue. In the case that the word
is a vowel, the subject must perform the task during the complete 4 seconds of
this interval duration, while if the word is a command, a sequence of three
audible clicks will indicate when to imagine or pronounce the target word.
Finally, during the rest interval (4 s) the subject is allowed to move,
swallow or blink.

The number of trials performed by each subject for each word is fifty, with
forty corresponding to the imagined speech condition and the other ten
belonging to the pronounced speech one.

Figure 2: Sequence time course for the presentation of one stimulus, in this
particular case for the word adelante and under the imagined speech condition.
Graphic taken from the original dataset paper [7].

EEG signals were recorded using Ag-AgCl cup electrodes, attached to the scalp
according to the 10-20 international system and with conductive paste. No
electrode cap was used. F3, F4, C3, C4, P3, and P4 were chosen as active
electrodes, while reference and ground electrodes were placed on the left and
right mastoids. The EEG signals were acquired with 1024 Hz sampling rate and
16 bit resolution.

The EEG signals of the Imagined Speech dataset were preprocessed by its
authors. The preprocessing included a band pass filter between 2-40 Hz.

Please, find further details on the original dataset paper [7].

##  4 Proposed approaches

Some classification models are studied and implemented starting from basic
methods such as SVM, to ensemble methods such as the XGBoost classifier up to
the use of neural networks such as LSTM and BiLSTM.

###  4.1 Machine Learning approaches

Since inner speech recognition is a very complicated task, a simpler
preliminary analysis was performed, which consists of binary classification
between the resting state and the action interval. The first 1.5s of the rest
interval and the 2.5s of the action interval were considered for each trial
(Figure 1). Subsequently, multiclass classification considering the four words
(left, right, up and down) of inner speech was carried out. Action interval
2.5s were used for each trial (Figure 1).

The following ML analyses were performed for both classification tasks.
Power Spectral Density (PSD) was used as a feature extraction technique before
proceeding with the classification. PSD was calculated using Welch’s method
and based on relative power in specific frequency bands: alpha (8-13 Hz), beta
(13-30 Hz) and gamma (30-100 Hz).
The models were trained and tested on each subject individually, using K-fold
cross validation. In this study, the data was split into four folds, resulting
in a number of trials ranging from 237 to 285 trials (depending on the
subject) in test set and from 713 to 855 trials in the training set in binary
classification, while in multiclass classification from 118 to 142 trials in
test set and from 357 to 428 trials in the training set. The SVM and XGBoost
classifiers were trained on PSD features vector with dimension
$(n\\_epochs,n\\_channels*n\\_band\\_freqs)$ for each subject.
Since the number of features is too high compared to the number of trials of
each subject, three possible solutions were analysed:

  * •

to apply Principal Component Analysis (PCA);

  * •

to extract the most important features identified with the XGBoost classifier,
both considering the subjects individually and making an intersection of the
most important features in common to all subjects;

  * •

to choose a subset of meaningful electrodes, since the neural correlates of
inner speech processing are reported to be mainly present in the left
hemisphere (see Section 1).

The analysis carried out in the binary classification showed a difference
between the action interval and the rest interval. It was therefore verified
whether this difference could be associated with one or more time windows, in
order to identify a particularly significant area in which the inner speech
activity could be encoded. The action interval has been split into 0.5s wide
sliding windows with a 50% overlap. For each window of the action interval, a
binary SVM model was trained, considering all the resting state and using the
features extracted with XGBoost. The idea was to identify the best window for
binary classification and then use it in multiclass classification. Again a
subject-based approach was used.

The analyzes performed do not justify the choice of one interval over another
and this suggests continuing to consider the entire interval in the following
tests carried out with deep learning methods.

###  4.2 Deep Learning approaches

The DL models were trained and tested for the multiclass classification task
on each subject individually, using nested cross-validation.
Three different types of analysis were performed in order to obtain the input
data to train the models:

  * •

The PSD using the Welch’s method was calculated and the most important
features were extracted using the XGBoost features importance vector.

  * •

The raw data considering all the channels were used.

  * •

The raw data considering only the channels associated with the most important
features extracted using the XGBoost were used.

LSTM and BiLSTM networks were trained for each type of input data.

The Imagine Speech dataset [7] was chosen to confirm the validity of the model
that obtained the best results on the Thinking Out Loud dataset.

##  5 Results and discussion

###  5.1 Machine Learning Models results

In the binary classification task, among the various tests performed using the
PCA, the best results for SVM were obtained without PCA and an accuracy of 79%
is obtained, while with XGBoost an accuracy of 81% is obtained using PCA and
explaining 99% of the variance. Using the most important features extracted
with XGBoost considering each subject individually, the SVM performances
improve up to 80% accuracy with a gain of 0.9.

In the multiclass classification task, the results obtained with SVM and
XGBoost are very similar, respectively 26.2% and 27.9% accuracy. In this case,
using the most important features extracted with XGBoost both considering each
subject individually and in common to all subjects, the results are
approximately the same. This means that the subjects have common
characteristics relevant for classification. Furthermore, there are no
particular differences when using all channels or only those of the left
hemisphere.

The characteristics extracted in common to all subjects were analyzed and are
highlighted in the Figure 3. Considering all the channels, the most affected
areas are the occipital one, probably involved due to the visual signals
presented on the screen, the temporal and the frontal one. Both the right and
left hemispheres are involved. Since, even using only the electrodes
identified in the left hemisphere, the performance still remains good, this
could mean that there are electrodes in the left channels that compensate for
the absence of the right ones. The frequency band most involved is alpha,
usually associated with intense mental activity. These features were used
later for some analyzes carried out with the DL models.

Figure 3: Features in common to all subjects in the multiclass task identified
with XGBoost using all channels and gain=0.95

###  5.2 Deep Learning Models results

This paragraph summarizes the performances obtained with the different deep
learning models using the Thinking out loud dataset. In table 1 the results
are shown averaging over all subjects.

Input Type | LSTM Accuracy | BiLSTM Accuracy
---|---|---
Most Important Features | 30.4 % | 31.3%
Raw data (all channels) | 27.2% | 36.1%
Raw data (channels Most Important Features) | 26.7% | 33.1%

Table 1: Deep Learning Models comparison

In general, with BiLSTM the performances increases rather than using LSTM.
This is due to the fact that BiLSTM is able to capture the sequential
dependencies between data in both directions. The greatest improvement is
obtained using raw data from all channels as input.

Looking at the average performance subject by subject, we have a repeating
trend. Using both different models and different types of input data, the data
of some subjects are classified better than others. Specifically, subjects 4
and 5 achieve an average performance which is always slightly lower. This
could probably be related to the data acquisition phase of these volunteers.
Maybe the placement of the electrodes was not perfect or their data is
noisier. Analyzing the results of attention monitoring there are no
differences with the other subjects, so the lack of attention in carrying out
the task should not be the cause of the lower performance.

The best model network is composed of a BiLSTM layer followed by two dense
layer (with ReLU activation function) and a dense output layer (with softmax
activation function). Two dropout layers were used to reduce overfitting. SGD
was used as optimization method and categorical cross entropy as loss
function. Figure 4 shown the results obtained for each subject. As we can see,
all subjects achieve an accuracy above randomness (represented by the red line
- 25%). The mean accuracy considering all the subjects is 36.1%.

Figure 4: Accuracy of the BiLSTM network for multiclass classification using
the Thinking out loud raw data. The red line represents the chance level
(25%).

Table 2 summarizes the results of our best model for each subject.

Subject | Accuracy | Precision | Recall | F1-score
---|---|---|---|---
Sub 1 | 38.60% | 40.37% | 38.93% | 37.87%
Sub 2 | 40.17% | 40.06% | 40.43% | 39.56%
Sub 3 | 37.60% | 38.25% | 37.50% | 35.50%
Sub 4 | 33.67% | 34.44% | 33.69% | 32.81%
Sub 5 | 31.67% | 32.13% | 31.94% | 31.06%
Sub 6 | 34.81% | 37.69% | 33.00% | 33.75%
Sub 7 | 34.83% | 36.00% | 34.94% | 34.31%
Sub 8 | 36.60% | 37.69% | 36.88% | 34.88%
Sub 9 | 38.00% | 38.19% | 37.75% | 37.31%
Sub 10 | 35.33% | 34.50% | 34.94% | 34.38%
Average | 36.12% | 36.93% | 36.00% | 35.14%
Table 2: BiLSTM performance for each subject on the 4-class inner speech
classification task using the Thinking out loud raw data

Table 3 shows a comparison of our proposed approaches and works in the
literature using the Thinking out loud dataset.

Classifier | Input Data | Accuracy
---|---|---
SVM | PSD Features (channels left hemisphere) + PCA (0.99) | 26.2%
XGBoost | PSD Features + PCA (0.99) | 27.9%
LSTM | Most Important Features | 30.4%
Raw data (all channels) | 27.2%
Raw data (channels Most Important Features) | 26.7%
BiLSTM | Most Important Features | 31.3%
Raw data (all channels) | 36.1%
Raw data (channels Most Important Features) | 33.1%
EEGNet | Raw Data (channels left hemisphere) | 29.67% [3]
Table 3: Comparison of our ML and DL models with results in the literature
using the Thinking Out Loud dataset.

The Imagined Speech dataset was chosen to confirm the validity of the model
that obtained the best results on the Thinking Out Loud dataset. Figure 5
shown the results obtained for each subject. As we can see, all subjects
achieve an accuracy above chance (represented by the red line - 16.67%). The
mean accuracy considering all the subjects is 25.1%.

Figure 5: Accuracy of the BiLSTM network for multiclass classification using
the Imagined Speech raw data. The red line represents the chance level
(16.7%).

Table 4 shows a comparison of our proposed approach and works in the
literature using the Imagined Speech dataset.

Classifier | Input Data | Accuracy
---|---|---
BiLSTM | Raw data (all channels) | 25.1%
RF | Relative Wavelet Energy (RWE) | 18.58% [7]
EEGNet | Raw Data (all channels) | 24.97% [6]
Table 4: Comparison of our DL model with results in the literature using the
Imagined Speech dataset

##  6 Conclusions

Inner speech recognition decoding EEG signal is still an open field of
research. Few datasets are available in the literature and the classification
performance, even if above chance, is still very low. The results obtained
with this work confirm that the adoption of BiLSTM architecture increases the
performance of classification with respect to those in the state-of-the-art.
In particular, the model designed for the Thinking Out Loud dataset was tested
on a different one, acquired using a similar experimental protocol, confirming
the validity of our proposal. The best classifier is obtained considering raw
data of all channels, denoting that a deeper analysis on the most significant
features should be performed, recalling that inner speech recognition should
be considered in BCI applications where classification should be performed in
real time. Finally, the need of more numerous and less noisy datasets is
crucial for further development in this field of research.

## References

  * [1] Alderson-Day, B., Fernyhough, C.: Inner speech: development, cognitive functions, phenomenology, and neurobiology. Psychological bulletin 141(5), 931 (2015)
  * [2] Amit, E., Hoeflin, C., Hamzah, N., Fedorenko, E.: An asymmetrical relationship between verbal and visual thinking: Converging evidence from behavior and fmri. NeuroImage 152, 619–627 (2017)
  * [3] van den Berg, B., van Donkelaar, S., Alimardani, M.: Inner speech classification using eeg signals: A deep learning approach. In: 2021 IEEE 2nd International Conference on Human-Machine Systems (ICHMS). pp. 1–4. IEEE (2021)
  * [4] Bocquelet, F., Hueber, T., Girin, L., Chabardès, S., Yvert, B.: Key considerations in designing a speech brain-computer interface. Journal of Physiology-Paris 110(4), 392–401 (2016)
  * [5] Brigham, K., Kumar, B.V.: Imagined speech classification with eeg signals for silent communication: a preliminary investigation into synthetic telepathy. In: 2010 4th International Conference on Bioinformatics and Biomedical Engineering. pp. 1–4. IEEE (2010)
  * [6] Cooney, C., Korik, A., Folli, R., Coyle, D.: Evaluation of hyperparameter optimization in machine and deep learning methods for decoding imagined speech eeg. Sensors 20(16), 4629 (2020)
  * [7] Coretto, G.A.P., Gareis, I.E., Rufiner, H.L.: Open access database of eeg signals recorded during imagined speech. In: 12th International Symposium on Medical Information Processing and Analysis. vol. 10160, p. 1016002. SPIE (2017)
  * [8] DaSalla, C.S., Kambara, H., Koike, Y., Sato, M.: Spatial filtering and single-trial classification of eeg during vowel speech imagery. In: Proceedings of the 3rd International Convention on Rehabilitation Engineering & Assistive Technology. pp. 1–4 (2009)
  * [9] DaSalla, C.S., Kambara, H., Sato, M., Koike, Y.: Single-trial classification of vowel speech imagery using common spatial patterns. Neural networks 22(9), 1334–1339 (2009)
  * [10] Deng, S., Srinivasan, R., Lappas, T., D’Zmura, M.: Eeg classification of imagined syllable rhythm using hilbert spectrum methods. Journal of neural engineering 7(4), 046006 (2010)
  * [11] D’Zmura, M., Deng, S., Lappas, T., Thorpe, S., Srinivasan, R.: Toward eeg sensing of imagined speech. In: International Conference on Human-Computer Interaction. pp. 40–48. Springer (2009)
  * [12] González-Castañeda, E.F., Torres-García, A.A., Reyes-García, C.A., Villaseñor-Pineda, L.: Sonification and textification: Proposing methods for classifying unspoken words from eeg signals. Biomedical Signal Processing and Control 37, 82–91 (2017)
  * [13] Gu, X., Cao, Z., Jolfaei, A., Xu, P., Wu, D., Jung, T.P., Lin, C.T.: Eeg-based brain-computer interfaces (bcis): A survey of recent studies on signal sensing technologies and computational intelligence approaches and their applications. IEEE/ACM transactions on computational biology and bioinformatics 18(5), 1645–1666 (2021)
  * [14] Idrees, B.M., Farooq, O.: Vowel classification using wavelet decomposition during speech imagery. In: 2016 3rd International Conference on Signal Processing and Integrated Networks (SPIN). pp. 636–640. IEEE (2016)
  * [15] Kim, J., Lee, S.K., Lee, B.: Eeg classification in a single-trial basis for vowel speech perception using multivariate empirical mode decomposition. Journal of neural engineering 11(3), 036010 (2014)
  * [16] Lawhern, V.J., Solon, A.J., Waytowich, N.R., Gordon, S.M., Hung, C.P., Lance, B.J.: Eegnet: a compact convolutional neural network for eeg-based brain–computer interfaces. Journal of neural engineering 15(5), 056013 (2018)
  * [17] Martin, S., Iturrate, I., Millán, J.d.R., Knight, R.T., Pasley, B.N.: Decoding inner speech using electrocorticography: Progress and challenges toward a speech prosthesis. Frontiers in neuroscience 12, 422 (2018)
  * [18] Mohanchandra, K., Saha, S.: A communication paradigm using subvocalized speech: translating brain signals into speech. Augmented Human Research 1(1), 1–14 (2016)
  * [19] Nguyen, C.H., Karavas, G.K., Artemiadis, P.: Inferring imagined speech using eeg signals: a new approach using riemannian manifold features. Journal of neural engineering 15(1), 016002 (2017)
  * [20] Nieto, N., Peterson, V., Rufiner, H.L., Kamienkowski, J.E., Spies, R.: Thinking out loud, an open-access eeg-based bci dataset for inner speech recognition. Scientific Data 9(1), 1–17 (2022)
  * [21] Panachakel, J.T., Ramakrishnan, A.G.: Decoding covert speech from eeg-a comprehensive review. Frontiers in Neuroscience p. 392 (2021)
  * [22] Riaz, A., Akhtar, S., Iftikhar, S., Khan, A.A., Salman, A.: Inter comparison of classification techniques for vowel speech imagery using eeg sensors. In: The 2014 2nd International Conference on Systems and Informatics (ICSAI 2014). pp. 712–717. IEEE (2014)
  * [23] Saha, P., Abdul-Mageed, M., Fels, S.: Speak your mind! towards imagined speech recognition with hierarchical deep learning. arXiv preprint arXiv:1904.05746 (2019)
  * [24] Salama, M., ElSherif, L., Lashin, H., Gamal, T.: Recognition of unspoken words using electrode electroencephalograhic signals. In: The Sixth International Conference on Advanced Cognitive Technologies and Applications. pp. 51–5. Citeseer (2014)
  * [25] Schultz, T., Wand, M., Hueber, T., Krusienski, D.J., Herff, C., Brumberg, J.S.: Biosignal-based spoken communication: A survey. IEEE/ACM Transactions on Audio, Speech, and Language Processing 25, 2257–2271 (2017)
  * [26] Suppes, P., Lu, Z.L., Han, B.: Brain wave recognition of words. Proceedings of the National Academy of Sciences 94(26), 14965–14969 (1997)
  * [27] Wang, L., Zhang, X., Zhong, X., Zhang, Y.: Analysis and classification of speech imagery eeg for bci. Biomedical signal processing and control 8(6), 901–908 (2013)