# Hadamard Inverse Function Theorem
Proved by Variational Analysis

Milen Ivanov and Nadia Zlateva

(In memoriam to A. L. Dontchev)

###### Abstract

We present a proof of Hadamard Inverse Function Theorem by the methods of
Variational Analysis, adapting an idea of I. Ekeland and E. Séré [4].

##  1 Introduction

The classical example $(x,y)\to e^{x}(\cos y,\sin y)$ shows that – except in
dimension one – the derivative may be everywhere invertible while the function
itself is invertible only locally. Probably the historically first sufficient
condition for global invertibility is given by J. S. Hadamard, see (1) in
Theorem 1 below.

An excellent overview – both from research and educational perspective – of
this topic is given in [6]. Perhaps the easiest to understand – because of its
geometrical nature – proof involves application of the Mountain Pass Theorem
to the function $x\to\|f(x)-y\|$ to ensure the injectivity of $f$. However,
Mountain Pass Theorem – although “obvious” – is hard to verify and may impose
additional restrictions of technical nature.

Here we present a new proof based on a recent idea by I. Ekeland and E. Séré
[4]. This idea allows obtaining a continuous right inverse to $f$ on any
compact, see Proposition 1. The necessary key Proposition 1 is proved by
methods of Variational Analysis in the flavour of the monographs of A.
Dontchev [2], A. Dontchev and T. Rockafellar [3], and A. Ioffe [5].

We work in a Banach space $(X,\|\cdot\|)$ and denote its closed unit ball by
$B_{X}$. Recall that the function

$f:X\to X$

is called Fréchet differentiable at $x\in X$ if there is a bounded linear
operator $f^{\prime}(x):X\to X$ such that

$\lim_{\|h\|\to 0}\frac{f(x+h)-f(x)-f^{\prime}(x)h}{\|h\|}=0.$

The function $f$ is called smooth, denoted $f\in C^{1}$, if the function

$x\to f^{\prime}(x)$

is norm-to-norm continuous.

We present a modern proof to the following classical

######  Theorem 1.

(Hadamard) Let $f\in C^{1}$, $f^{\prime}(x)$ be invertible for all $x$ and
satisfying

$\|[f^{\prime}(x)]^{-1}\|\leq M,\quad\forall x\in X,$ (1)

for some $M>0$.

Then $f$ is $C^{1}$ invertible on $X$.

In other words, there is $g\in C^{1}$ such that

$g(f(x))=f(g(x))=x,\quad\forall x\in X.$

The work is organised as follows. In the next Section 2 we give the necessary
preliminary known facts. In Section 3 we prove the key Proposition 1 and in
the final Section 4 we complete the proof of Hadamard Theorem 1.

##  2 Preliminaries

We start with recalling the classical (local) Inverse Function Theorem.

######  Theorem 2.

Let $f\in C^{1}$ and let $f^{\prime}(x_{0})$ be invertible. Then there are
$\varepsilon,\delta>0$ such that for each $y$ such that

$\|y-f(x_{0})\|<\varepsilon$

there is unique $x=:g(y)$ such that $\|x-x_{0}\|<\delta$ and

$f(x)=y.$

Moreover, $g\in C^{1}$ and

$g^{\prime}(f(x_{0}))=[f^{\prime}(x_{0})]^{-1}.$ (2)

The following statements are also well-known.

######  Lemma 1.

Let $f$ be $C^{1}$. Let $K\subset X$ be compact and let $r>0$. Then

$f(x+th)=f(x)+tf^{\prime}(x)h+o(t)\mbox{ uniformly on }x\in K\mbox{ and }h\in
rB_{X}.$

More precisely, there is $\alpha(t)\to 0$ as $t\to 0$ such that

$\sup\\{\|f(x+th)-f(x)-tf^{\prime}(x)h\|:x\in K,\ h\in
rB_{X}\\}\leq\alpha(t)t.$

######  Lemma 2.

Let $X$ be a Banach space and let $A(x)$ be bounded linear operator for each
$x\in X$. Let the function $x\to A(x)$ be norm-to-norm continuous at $x_{0}$.
If $A(x_{0})$ is invertible then

$x\to A^{-1}(x)$

is continuous at $x_{0}$.

Next is a precursor to Ekeland Variational Principle, see [1, Chapter 5,
Section 1]. Of course, it easily follows from Ekeland Variational Principle
itself, see e.g. [5, Basic Lemma]. See also the comments concerning the “Basic
Lemma” on [5, p. 93]. Here we present a proof based on what is called in these
comments “simple iteration”.

######  Lemma 3.

Let $X$ be a Banach space and let $\mu:X\to\mathbb{R}^{+}\cup\\{\infty\\}$ be
lower semicontinuous and such that for some $r>0$

$\forall x:\ 0<\mu(x)<\infty\Rightarrow\exists y:\ \mu(y)<\mu(x)-r\|y-x\|.$

Then for each $x\in\mathrm{dom}\,\mu$ there is $y\in X$ such that

$\mu(y)=0\mbox{ and }r\|y-x\|\leq\mu(x).$

###### Proof.

Fix $x_{0}\in\mathrm{dom}\,\mu$ such that $\mu(x_{0})>0$. Let
$x_{1},x_{2},\ldots,x_{n}$ be already chosen in the following way.

Set

$\nu_{n}=\sup\\{\|x-x_{n}\|:\ \mu(x)<\mu(x_{n})-r\|x-x_{n}\|\\}.$ (3)

We are given that the set in the right hand side is nonempty, so $\nu_{n}>0$.
Also, since $\mu\geq 0$, we have that $\nu_{n}\leq\mu(x_{n})/r<\infty$.

Choose a $x_{n+1}$ such that

$\mu(x_{n+1})<\mu(x_{n})-r\|x_{n+1}-x_{n}\|\mbox{ and
}\|x_{n+1}-x_{n}\|>\nu_{n}/2.$ (4)

Note that

$\|x_{n+1}-x_{0}\|\leq\sum_{i=0}^{n}\|x_{i+1}-x_{i}\|\leq\sum_{i=0}^{n}(\mu(x_{i})-\mu(x_{i+1}))/r\leq\mu(x_{0})/r.$

If if $\mu(x_{n+1})=0$, we are done. If not, we continue by induction.

If we would end up with an infinite sequence $(x_{n})_{0}^{\infty}$, then from
the above inequality $\sum_{i=0}^{\infty}\|x_{i+1}-x_{i}\|\leq\mu(x_{0})/r$,
so $x_{n}\to\bar{x}$ as $n\to\infty$ and $\|\bar{x}-x_{0}\|\leq\mu(x_{0})/r$.
From (4) it follows that $\nu_{n}\to 0$.

If $\mu(\bar{x})>0$ then we can find $\bar{y}$ such that

$\mu(\bar{y})<\mu(\bar{x})-r\|\bar{y}-\bar{x}\|.$ (5)

Since $\mu$ is lower semicontinuous, we will have for all $n$ large enough
$\mu(\bar{y})<\mu(x_{n})-r\|\bar{y}-x_{n}\|$. Hence, see (3),
$\nu_{n}\geq\|\bar{y}-x_{n}\|$ for all $n$ large enough. Since $\nu_{n}\to 0$,
we get that $\bar{y}=\bar{x}$ which contradicts (5).

So, $\mu(\bar{x})=0$ and we are done. ∎

##  3 Right inverse à la Ekeland & Séré

The following is what distinguishes our proof of Hadamard Theorem.

######  Proposition 1.

Let $f\in C^{1}$, $f^{\prime}(x)$ be invertible for all $x$ and let $f$
satisfy (1). Let $K\subset X$ be compact. Then $f$ has a continuous right
inverse on $K$, that is, there is a continuous $g:K\to X$ such that

$f(g(x))=x,\quad\forall x\in K.$

Moreover, if $f(0)=0\in K$ then there is a continuous right inverse of $f$ on
$K$ that satisfies

$g(0)=0.$

###### Proof.

Let $C(K,X)$ be the space of all continuous functions from $K$ to $X$. It is
clear that when equipped with the norm

$\|g\|_{\infty}:=\max_{y\in K}\|g(y)\|$

it is a Banach space.

Consider the following measure

$\mu:C(K,X)\to\mathbb{R}^{+}$

of how much a given function $g$ differs from a right inverse of $f$:

$\mu(g):=\max_{y\in K}\|f(g(y))-y\|.$

It is clear that $\mu$ is lower semicontinuous. (It is easy to check that it
is continuous but we do not need this.)

The claim is that there exists $g$ such that $\mu(g)=0$.

In order to check the condition of Lemma 3, fix $\hat{g}\in C(K,X)$ such that

$\mu(\hat{g})>0.$

Set $u:K\to X$ as

$u(y):=y-f(\hat{g}(y)).$

By definition,

$\mu(\hat{g})=\|u\|_{\infty}.$

So, $u$ is not identically equal to zero, because $\mu(\hat{g})>0$.

Put

$w(y):=[f^{\prime}(\hat{g}(y))]^{-1}u(y),\quad\forall y\in K.$

Because $x\to f^{\prime}(x)$ is continuous, from Lemma 2 it follows that

$y\to[f^{\prime}(\hat{g}(y))]^{-1}$

is norm-to-norm continuous, so $w\in C(K,X)$.

Therefore, for $t>0$

$g_{t}:=\hat{g}+tw\in C(K,X).$

Note for future reference that form (1) it follows that $\|w\|_{\infty}\leq
M\|u\|_{\infty}$, that is

$\|w\|_{\infty}\leq M\mu(\hat{g}).$ (6)

Our next aim is to estimate $\mu(g_{t})$. By definition

$\mu(g_{t}):=\max_{y\in K}\|f(g_{t}(y))-y\|.$

For $y\in K$ define $\varphi_{y}:\mathbb{R}^{+}\to\mathbb{R}^{+}$ by

$\varphi_{y}(t):=\|f(g_{t}(y))-y\|,$

hence

$\mu(g_{t}):=\max_{y\in K}\varphi_{y}(t).$ (7)

Because the set $\hat{g}(K)$ is compact and the set $w(K)$ is bounded, from
Lemma 1 it follows that

$\max_{y\in
K}\|f(\hat{g}(y)+tw(y))-f(\hat{g}(y))-tf^{\prime}(\hat{g}(y))w(y)\|=\alpha(t)t,$

where $\alpha(t)\to 0$ as $t\to 0$. But

$f^{\prime}(\hat{g}(y))w(y)=f^{\prime}(\hat{g}(y))[f^{\prime}(\hat{g}(y))]^{-1}u(y)=u(y),$

so

$\|f(g_{t}(y))-f(\hat{g}(y))-tu(y)\|_{\infty}=\alpha(t)t.$

Therefore, for any $y\in K$

$\displaystyle\varphi_{y}(t)$ $\displaystyle=$
$\displaystyle\|f(g_{t}(y))-y\|$ $\displaystyle\leq$
$\displaystyle\|f(\hat{g}(y))+tu(y)-y\|+\|f(g_{t}(y))-f(\hat{g}(y))-tu(y)\|$
$\displaystyle\leq$ $\displaystyle\|(t-1)u(y)\|+\alpha(t)t.$

Since $\varphi_{y}(0)=\|u(y)\|$ we have that for small $t$

$\varphi_{y}(t)\leq(1-t)\varphi_{y}(0)+\alpha(t)t.$

Taking a maximum over $y\in K$, see (7), we get

$\mu(g_{t})\leq(1-t)\mu(g_{0})+\alpha(t)t,$

or, in other words,

$\mu(\hat{g}+tw)\leq\mu(\hat{g})-t\mu(\hat{g})+\alpha(t)t.$ (8)

Since $\mu(\hat{g})>0$, for some $\delta>0$ we then have
$|\alpha(t)|<\mu(\hat{g})/2$ for $t\in(0,\delta)$. So,

$\mu(\hat{g}+tw)<\mu(\hat{g})-(t/2)\mu(\hat{g}),\quad\forall t\in(0,\delta).$

From (6), which is $\mu(\hat{g})\geq(1/M)\|w\|_{\infty}$, we get

$\mu(\hat{g}+tw)<\mu(\hat{g})-(1/2M)\|tw\|_{\infty},\quad\forall
t\in(0,\delta),$

and we can apply Lemma 3 with $r=1/2M$, $x=\hat{g}$ and
$y=\hat{g}+(\delta/2)w$, to conclude that $\mu$ vanishes somewhere.

If $f(0)=0\in K$ then we can modify the above by considering instead of
$C(K,X)$ the Banach space of continuous $g:K\to X$ such that $g(0)=0$. It is
clear that in this case $u(0)=w(0)=0$ and everything else works in the same
way. ∎

##  4 Proof of Theorem 1

###### Proof.

It is enough to show that $f$ is bijective.

Let $y\in X$ be arbitrary and set $K=\\{y\\}$. From Proposition 1 it follows
that there is $x=g(y)$ such that $f(x)=y$. So, $f$ is surjective, i.e.
$f(X)=X$.

Let $a,b\in X$ be such that $f(a)=f(b)$. By considering instead of $f$ the
function

$x\to f(b-x)-f(b)$

we can assume without loss of generality that

$b=0\mbox{ and }f(0)=0.$

Then

$f(a)=0.$

Set

$K:=f([0,a]).$

Since $f$ is continuous, $K$ is compact. From Proposition 1 there is a
continuous

$g:K\to X,\mbox{ such that }g(0)=0\mbox{ and }f(g(y))=y,\quad\forall y\in K.$

Consider

$I:=\\{t\in[0,1]:\ g(f(ta))=ta\\}.$

Obviously, $0\in I$, because $g(0)=0$. Due to the continuity of $g$ and $f$
the set $I$ is closed and, therefore, compact. Let

$\bar{t}:=\max\\{t:t\in I\\}.$

Assume that $\bar{t}<1$. By the local Inverse Function Theorem, see Theorem 2,
there are $\delta,\varepsilon>0$ such that for each $y\in X$ such that

$\|y-f(\bar{t}a)\|<\varepsilon$

there is unique $x\in X$ such that $\|x-\bar{t}a\|<\delta$ and

$f(x)=y.$

From the continuity of $f$ there is $\mu>0$ such that for all
$t\in(\bar{t},\bar{t}+\mu)\subset(0,1)$ we have $\|ta-\bar{t}a\|<\delta$,
$\|f(ta)-f(\bar{t}a)\|<\varepsilon$. Moreover,
$\|g(f(ta))-\bar{t}a\|=\|ta-\bar{t}a\|<\delta$, since
$g(f(\bar{t}a))=\bar{t}a$.

Then, because of $f(ta)\in K$ we have that $f(g(f(ta)))=f(ta)$. From the
uniqueness of the solution to $f(\cdot)=f(ta)$ in this neighbourhood we get
$g(f(ta))=ta$ for all $t\in(\bar{t},\bar{t}+\mu)$ which contradicts the
definition of $\bar{t}$.

So, $\bar{t}=1$ meaning that $g(f(a))=a$. But $f(a)=0$. Since $g(0)=0$, it
follows that $a=0$.

We have proved that if $f(a)=f(b)$ then $a=b$, so $f$ is injective. ∎

## References

  * [1] J.-P. Aubin and I. Ekeland, _Applied nonlinear analysis_ , John Wiley & Sons, New York, 1984, ISBN: 0-486-45324-3
  * [2] Asen L. Dontchev, _Lectures on Variational Analysis_ , Book Series: Applied Mathematical Sciences, Springer, 2021, ISBN: 978-3-030-79910-6
  * [3] A. L. Dontchev and R. T. Rockafellar, _Implicit Functions and Solution Mappings: A View from Variational Analysis_ , Series in Operations Research and Financial Engineering, Springer, 2014, ISBN: 978-1-4939-1037-3
  * [4] I. Ekeland and E. Séré, A local surjection theorem, 2017, https://project.inria.fr/brenier60/files/2011/12/Brenier.pdf
  * [5] A. Ioffe, _Variational Analysis of Regular Mappings: Theory and Applications_ , Springer Monographs in Mathematics, 2017, SBN: 978-3-319-64277-2
  * [6] R. Plastock, Homeomorogisms between Banach spaces, Trans. Amer. Math. Soc., 200, 1974, 169–183.


#  Keyframe Segmentation and Positional Encoding
for Video-guided Machine Translation Challenge 2020

Tosho Hirasawa
Tokyo Metropolitan
University
hirasawa-tosho
@ed.tmu.ac.jp Zhishen Yang ††footnotemark:
Tokyo Institute
of Technology
zhishen.yang
@nlp.c.titech.ac.jp Mamoru Komachi
Tokyo Metropolitan
University
komachi
@tmu.ac.jp
Naoaki Okazaki
Tokyo Institute
of Technology
okazaki
@c.titech.ac.jp Equal contribution

###### Abstract

Video-guided machine translation as one of multimodal neural machine
translation tasks targeting on generating high-quality text translation by
tangibly engaging both video and text. In this work, we presented our video-
guided machine translation system in approaching the Video-guided Machine
Translation Challenge 2020. This system employs keyframe-based video feature
extractions along with the video feature positional encoding. In the
evaluation phase, our system scored 36.60 corpus-level BLEU-4 and achieved the
1st place on the Video-guided Machine Translation Challenge 2020.

##  1 Introduction

In multimodal machine translation (MMT), a target sentence is translated from
a source sentence together with related nonlinguistic information such as
images Specia et al. (2016) and videos Wang et al. (2019). The goal of Video-
guided Machine Translation (VMT) Challenge 2020 is to generate target-language
video descriptions given both the videos and its description in the source
languages.

Videos preserve rich visual information that guides textual translation. Since
video descriptions illustrate visual objects, scenes and actions in the
videos, we hypothesize that obtaining appearance features (objects and scenes)
and action features from videos will contribute to the quality of translation.
Additionally, keyframes store entire images in the video, helping us extract
high-quality video features.

A video consists of an ordered sequence of frames, while features extracted
from them often do not preserve such order information. We hypothesize that
incorporating such order information with visual feature facilities our model
in improving translation quality.

Based on the above hypotheses, in the proposed video-guided machine
translation system, we introduce a keyframe-based approach for video feature
extraction, along with positional encoding to inject order information into
video features. The core model in our system is a modified hierarchical
attention model Libovický and Helcl (2017) with encoder and decoder
architecture.

##  2 Hierarchical Attention with Positional Encoding

Our video-guided machine translation system is an extension of the
hierarchical attention model Libovický and Helcl (2017). The underlying model
has a simple encoder and a modified decoder from Bahdanau et al. (2015) that
uses two individual attention mechanisms to compute the textual context vector
and the auxiliary context vector (in our case, the context vector over
sequential video representations). However, the model is assumed to
incorporate with spatial image features (e.g., region of Interest feature from
Faster-RCNN models) and cannot leverage order information, which is a
distinguishing property of video features (e.g. I3D).

To address this problem, we add positional encodings Vaswani et al. (2017) to
the video representations at the beginning of the attention to make the model
use the order of the representations.

### Encoder

We first encode the $N$-tokens input sentence
$\textbf{x}=(x_{1},\cdots,x_{N})$ into encoder states
$\textbf{h}=(h_{1},\cdots,h_{N})$ by a bidirectional GRU, where each $h_{*}$
is a vector with $d$ dimension. The $T$-elements video representations
$\textbf{z}=(z_{1},\cdots,z_{T})$ is extracted from a video $\bm{v}$ using
either video or imagery encoder described in Section 3.

Additionally, we add positional encoding to video representations z to obtain
position-aware video representations
$\hat{\textbf{z}}=(\hat{z}_{1},\cdots,\hat{z}_{T})$ at each timestep
$pos\in(1,\cdots,T)$:

$\displaystyle\hat{z}_{pos}$ $\displaystyle=z_{pos}+PE_{pos}$ (1)
$\displaystyle PE_{(pos,2i)}$ $\displaystyle=sin(pos/10000^{2i/d})$ (2)
$\displaystyle PE_{(pos,2i+1)}$ $\displaystyle=cos(pos/10000^{2i/d})$ (3)

where $i$ is the dimension.

### Decoder

In each position $j$ while decoding, we first compute the decoder state
proposal $\bm{s}_{j}$ from previous word embedding $w_{j-1}$ and previous
decoder state $\hat{\bm{s}}_{j-1}$,

$\bm{s}_{j}=\mathrm{GRU}(w_{j-1},\hat{\bm{s}}_{j-1})$ (4)

Afterward, the textual context vector $\bm{c}_{j}^{(t)}$ and the video context
vector $\bm{c}_{j}^{(\bm{z})}$ are computed using two separate attention
mechanisms $\mathrm{att}_{\bm{t}}$ and $\mathrm{att}_{\bm{z}}$.

$\displaystyle\bm{c}_{j}^{(\bm{t})}=\mathrm{att}_{\bm{t}}(\bm{s}_{j},\textbf{h})$
(5)
$\displaystyle\bm{c}_{j}^{(\bm{z})}=\mathrm{att}_{\bm{z}}(\bm{s}_{j},\hat{\textbf{z}})$
(6)

The final context vector $\bm{c}_{j}$ is computed using another attention over
modalities $m\in\\{\bm{t},\bm{z}\\}$.

$\displaystyle\bm{e}_{j}^{(m)}$
$\displaystyle=o^{T}\mathrm{tanh}(\bm{W}_{1}s_{j}+\bm{U}^{(m)}\bm{c}_{j}^{(m)})$
(7) $\displaystyle\alpha_{j}^{(m)}$
$\displaystyle=\frac{\mathrm{exp}(\bm{e}_{j}^{(m)})}{\sum\limits_{m^{\prime}\in\\{t,z\\}}\bm{e}_{j}^{(m^{\prime})}}$
(8) $\displaystyle\bm{c}_{j}$
$\displaystyle=\sum\limits_{m\in\\{t,z\\}}\alpha_{j}^{(m)}\bm{Q}^{(m)}\bm{c}_{j}^{(m)}$
(9)

where $o^{T}$ and $\bm{W}_{1}$ are model parameters and shared among all
modalities, while $\bm{U}^{(m)}$ and $\bm{Q}^{(m)}$ are dedicated model
parameters for each modality. $\bm{U}^{(m)}$ and $\bm{Q}^{(m)}$ are a
projection matrices that map each single-modality context vector into a common
space. $o$ is the weight vector with the same dimensions of the common space.

The final context vector $\bm{c}_{j}$ is fed into the second GRU along with
the decoder state proposal $\bm{s}_{j}$ to generate the final decoder state
$\hat{\bm{s}}_{j}$ and output distribution $p(y_{j}|y_{<j})$

$\displaystyle\hat{\bm{s}}_{j}$
$\displaystyle=\mathrm{GRU}(\bm{c}_{j},\bm{s}_{j})$ (10) $\displaystyle
p(y_{j}|y_{<j})$
$\displaystyle=\mathrm{softmax}(\bm{W}_{2}\hat{\bm{s}}_{j}+b)$ (11)

where $\bm{W}_{2}$ and $b$ are model parameters.

### Multiple Video Feature Integration

Integrating various types of video features into a video-guided machine
translation system represents a potential way of improving translation quality
Wang et al. (2018b). In our system, we decided to ensemble models trained on
different types of video features. In section 5, we detail choices on ensemble
models.

##  3 Video and Imagery Encoders

Videos, as another input in our system, often possess visual clues that guide
the translation, such as actions, objects and scenes. Encoding video to
acquire information-rich video features acts as visual-guidance to text
translation.

We classified two types of video features: action features derived from
actions, and appearance features from visual objects and scenes. Keyframes in
videos store whole images, which often provide good visual representations of
objects and scenes. We used keyframes for appearance feature extraction and as
a basis to segment videos for obtaining motion features.

Our video-guided machine translation system consists of a video encoder that
outputs action features, and an imagery encoder generates appearance features.

Model Validation Set Public Test Set Wang et al. (2019) - 29.12 (1) Text-only
35.10 - Official I3D features 111https://eric-xw.github.io/vatex-
website/download.html (2) with positional encoding 35.28 35.26 (3) without
positional encoding 35.02 - Keyframe-based video feature extraction (3) Action
features 35.42 35.35 (4) Object features (Res-Net 152, ImageNet) 35.29 - (5)
Scene features (Res-Net 50, Place365) 35.14 - Ensemble Model 3 Action features
(3) 36.20 - 3 Object features (4) + 3 Scene features (5) 36.38 - 3 Action
features (3) + 3 Object features (4) + 3 Scene features (5) 36.48 36.60

Table 1:  Corpus-level BLEU on validation and public_test sets.

### Video Encoder

We first segmented a video based on keyframes
222https://github.com/dmlc/decord to build a segment list, and each segment
contains a keyframe and 31 consecutive frames after it. We then feed the video
segment list to obtain the action feature matrix from a non-local neural
network Wang et al. (2018a) with Res-Net 101 He et al. (2016) as the backbone
pre-trained on ImageNet and fine-tuned on Kinetics400 dataset
333https://gluon-cv.mxnet.io/model_zoo/action_recognition.html#id113. Each
feature vector in the matrix is in chronological order of appearance of its
video segment by the time. The action feature matrix
$\textbf{M}\in\mathbf{R}^{T\times d}$ for a video $\bm{v}$ is:

M $\displaystyle=\mathrm{Video\ Encoder}(S)$ (12) $\displaystyle S$
$\displaystyle=\mathrm{Segmentation}(\bm{v})$ (13)

where $S$ is the list of $T$ keyframe video segments with chronological order.

### Imagery Encoder

Keyframes in the video store complete imagery information that suites our need
to extracting high-quality appearance features. Frames in the video often
involve visual objects and visual scenes. Therefore we obtained these two
types of appearance features from keyframes. We employed a object-recognition
system that is a Res-Net 152 model pre-trained with ImageNet and a scene-
recognition system that is a Res-Net 50 model pre-trained with Place365 Zhou
et al. (2018) 444http://places2.csail.mit.edu/. The input to our imagery
encoder is the keyframes from the video.

##  4 Experiment setup

### Model

The encoders of our model have one layer with 512 hidden dimensions, and
therefore the bidirectional GRU has a dimension of 1024. The decoder state has
a dimension of 512. The input word embedding size and output vector size are
1024.

During training, we used Adam optimizer with a learning rate of 0.001,
clipping gradient norm to 1.0, the dropout rate of 0.5, batch size of 512, and
early stopping patience of 10. The loss function was cross entropy. In the
evaluation phase, we performed a beam search with a size of 5.

### Preprocess

We preprocessed both English and Chinese sentences in the same manner as in
the starter code 555https://github.com/eric-xw/Video-guided-Machine-
Translation, where English sentences are lower-cased, and Chinese sentences
are split into sequences of characters.

The vocabulary of either English or Chinese contains tokens that occur at
least five times in the training set, giving 7,947 types for English and 2,655
types for Chinese.

##  5 Experimental Result

The official evaluation metrics for VMT challenge 2020 is BLEU Papineni et al.
(2002). Table 1 shows the corpus-level BLEU-4 scores of each model on the
validation set and public test set.

Figure 1: Example translations from the validation set generated from three of
our system variants. Only our system with ensemble model (final submission)
could correctly translate the group people as 游泳运动员 (swimming athletes), and
their actions as 练习水下舞蹈技巧 (practicing underwater dance tricks).

Our model using textual feature achieved a score of 35.10 on validation set,
which makes it serve as a baseline (text-only baseline) for the following
experiments with inputs of both textual and video features. Engaging official
video features, we observed slight performance deterioration (-0.08 BLEU)
without positional encoding, while a score improvement (+0.18 BLEU) with it.
Based on above results, we decided to keep positional encoding in our system.

We extracted two types of video features based on keyframes: action features
and appearance features, in which appearance features consist of two types of
features: object features, and scene features. To evaluate how each type of
video feature helps in improving the translation quality, we trained our
system using each type of video feature with textual features.

On evaluation set, training our system using object features ((4) in the table
1) and textual features improves 0.19 BLEU scores over text-only baseline, and
0.01 BLEU compared to system trained on official I3D features ((2) in table
1). Training on scene features ((5) in table 1) and textual features give the
system a slight 0.04 BLEU score improvements over text-only baseline. Compared
to the appearance features, our system trained on action features and textual
features achieved the highest BLEU scores, particularly, 0.32 BLEU over text-
only baseline and 0.14 BLEU over our system trained on official I3D features.
Based on above results, all types of video features improve our system
performance compared to using textual feature only.

Compared to our system trained on single types of video feature and textual
feature, our system ensembling models trained on different types of video
features and textual features give another raise in the BLEU score. On
evaluation set, compared to best preform single video feature model (3 in
table 1), ensemble three models of (3) improves 0.78 BLEU score, while
ensemble 3 models of (4) and 3 models of (5) get 0.96 BLEU score boost. An
ensemble of three different models (3), (4), and (5) achieves the best BLEU
score on validation set, this ensemble model is also our final submission,
which obtains 36.60 BLEU score in the public test set an ranks the first
place. Figure LABEL:fig:my_label shows example translations generated from
three of our system variants.

##  6 Conclusion

In the Video-guided Machine Translation Challenge 2020, we revealed that
keyframe-based video feature extraction and positional encoding jointly
enhance the translation quality by showing a substantial improvement from the
text-only baseline.

We also demonstrated that the ensemble of multiple models trained on different
types of video features brought further performance improvements. In the
future, we will explore the best integration of different features to improve
translation quality under the video-guidance.

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _ICLR_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _CVPR_ , pages 770–778.
  * Libovický and Helcl (2017) Jindřich Libovický and Jindřich Helcl. 2017.  Attention strategies for multi-source sequence-to-sequence learning.  In _ACL_ , pages 196–202.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _ACL_ , pages 311–318.
  * Specia et al. (2016) Lucia Specia, Stella Frank, Khalil Sima’an, and Desmond Elliott. 2016.  A shared task on multimodal machine translation and crosslingual image description.  In _MT_ , pages 543–553.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NIPS_ , pages 5998–6008.
  * Wang et al. (2018a) Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018a.  Non-local neural networks.  In _CVPR_ , pages 7794–7803.
  * Wang et al. (2018b) Xin Wang, Yuan-Fang Wang, and William Yang Wang. 2018b.  Watch, listen, and describe: Globally and locally aligned cross-modal attentions for video captioning.  In _NAACL_ , pages 795–801.
  * Wang et al. (2019) Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In _ICCV_ , pages 4581–4591.
  * Zhou et al. (2018) Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2018\.  Places: A 10 million image database for scene recognition.  _TPAMI_ , 40(6):1452–1464.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

Italian Workshop on Artificial Intelligence for Human-Machine Interaction
(AIxHMI 2023), November 06, 2023, Rome, Italy

[orcid=<EMAIL_ADDRESS>url=, ] [1]

[orcid=0000-0002-4405-8234<EMAIL_ADDRESS>url=https://mmsp.unimib.it/aurora-saibene/, ]

[orcid=0000-0002-6279-6660<EMAIL_ADDRESS>url=https://mmsp.unimib.it/francesca-gasparini/, ]

[orcid=<EMAIL_ADDRESS>url=, ]

[orcid=0000-0002-1394-0507<EMAIL_ADDRESS>url=, ]

[1]Corresponding author.

# A multi-artifact EEG denoising by frequency-based deep learning

Matteo Gabardi  Aurora Saibene  Francesca Gasparini  Daniele Rizzo  Fabio
Antonio Stella  University of Milano-Bicocca, Viale Sarca 336, 20126, Milano,
Italy  Xnext S.p.A., Via Valtorta 48, 20127, Milano, Italy  NeuroMI, Milan
Center for Neuroscience, Piazza dell’Ateneo Nuovo 1, 20126, Milano, Italy

(2023)

###### Abstract

Electroencephalographic (EEG) signals are fundamental to neuroscience research
and clinical applications such as brain-computer interfaces and neurological
disorder diagnosis. These signals are typically a combination of neurological
activity and noise, originating from various sources, including physiological
artifacts like ocular and muscular movements. Under this setting, we tackle
the challenge of distinguishing neurological activity from noise-related
sources. We develop a novel EEG denoising model that operates in the frequency
domain, leveraging prior knowledge about noise spectral features to adaptively
compute optimal convolutional filters for noise separation. The model is
trained to learn an empirical relationship connecting the spectral
characteristics of noise and noisy signal to a non-linear transformation which
allows signal denoising. Performance evaluation on the _EEGdenoiseNet_ dataset
shows that the proposed model achieves optimal results according to both
temporal and spectral metrics. The model is found to remove physiological
artifacts from input EEG data, thus achieving effective EEG denoising. Indeed,
the model performance either matches or outperforms that achieved by benchmark
models, proving to effectively remove both muscle and ocular artifacts without
the need to perform any training on the particular type of artifact.

###### keywords:

electroencephalography (EEG) deep learning (DL) frequency-based neural network
EEG denoising

##  1 Introduction

The electroencephalographic (EEG) signal is a time series acquired with non-
invasive sensors (called electrodes) placed on a subject’s scalp and is
characterized by time, frequency and spatial information [1]. However, it
usually presents a mixture of neurological activity and signals deriving from
noise-related biological or non-physiological sources [2].
This means that besides recording neural signals, the EEG captures noise
generated from ocular, muscular, and cardiac movements as examples of
biological artifacts, and noise related to non-biological sources like cable
movement, electrical interference, and electrode bad positioning [3]. For
further information on other artifact types, please refer to the review papers
by Urigüen and Garcia-Zapirain [3], and Rashmi and Shantala [4].

Many attempts have been presented in the state-of-the-art to reduce or remove
these types of artifacts, but automatic EEG denoising remains an open
challenge [5, 6]. In particular, this paper focuses on the denoising of two
specific types of artifacts, i.e., _ocular_ (OAs) and _muscular artifacts_
(MAs).

OAs are the most easily detectable artifacts due to their spiking shape
similar to a V and their pronounced presence on signals recorded by frontal
electrodes [7]. Moreover, they have a frequency range between 0.5 and 3 Hz,
and high amplitudes (around 100 mV) [4]. Notice that reference electrodes,
called electrooculograms (EOG), are sometimes included in the experimental
setting to track eye movements. Similar references, i.e., electromyographic
(EMG) sensors, can be also placed to detect surface muscular activity, which
may introduce MAs [3]. These artifacts are related to movements like
swallowing, chewing, talking, clenching hands, and muscular tension [4].
Unfortunately, MAs are more difficult to detect and present spectral
characteristics overlapping with the neural ones, having that they usually
have a frequency lesser than or equal to 35 Hz [4].
Notice that even if reference electrodes (EOG and EMG) can be used to track
artifacts, the interference between them and the EEG related electrodes is
bidirectional, i.e., the artifacts contaminate the EEG signals, and the EOG
and EMG electrodes capture both artifacts and neural activity [7]. Thus the
removal of OAs and MAs exploiting these sensors could be prone to errors or
excessive neural signal removal [7]. Therefore, the initial exploitation of
EOG and EMG sensors in linear regression methods, shifted to methodologies
like filtering, blind source separation, source decomposition, empirical mode
decomposition, signal space projection, beamforming, and hybrid techniques [3,
8, 9]. However, in the last few years, some works have proposed to move from
the traditional denoising techniques previously reported to completely data-
driven techniques based on deep learning (DL) models. In this framework, this
paper presents a DL-based denoising model, which relies on the knowledge
related to the power spectral density (PSD) estimation of noisy EEG signals,
and EOG or EMG noise data. In particular, the proposed model follows the
guidelines provided by Zhang et al. [10], who present _EEGdenoiseNet_ , i.e.,
a dataset devised as a benchmark to train and test DL-based denoising
strategies. Therefore, the paper is structured as follows.
After a brief overview of current studies presenting DL-based denoising
techniques (Section 2), _EEGdenoiseNet_ is described to provide a better
understanding of the exploited data (Section 3). Afterwards, the proposed DL
model is detailed (Section 4). Section 5 reports the performed experiments,
discuss both obtained and literature results, and provide future developments.
Final considerations are presented in Section 6.

##  2 Related works

This study focuses on DL-based denoising techniques, which will be detailed in
this section, and thus will not provide a dissertation of traditional
processing methodologies. The readers are invited to consult insightful review
papers on these topics provided by the EEG research community [3, 11, 8, 9].

Starting from the work related to the exploited dataset, Zhang et al. [10] do
not produce only _EEGdenoiseNet_ , but also develop (i) a fully-connected
neural network (FCNN), (ii) a simple CNN, (iii) a complex CNN, and (iv) a
recurrent neural network (RNN) for benchmarking purposes. Moreover, in a later
publication [12], the authors propose a novel CNN to remove MAs. In
particular, the devised architecture is composed by seven blocks, of which the
first six contain two 1D convolutional layers with ReLu as the activation
function and a 1D average pooling layer. The last block has as well two 1D
convolutional layers, which are instead followed by a flatten layer. Finally,
a dense layer is inserted. Notice that the core of the proposal is related to
the learning process. As reported by the authors, the aim of the DL-based
denoising models is to define a function that projects the noisy signals to
the clean ones:

$\tilde{x}=f(\hat{y},\theta)$ (1)

where $\tilde{x}$ is the clean EEG, $\hat{y}$ is the normalized noisy EEG, and
$\theta$ is the parameter to be learned.
Notice that the authors [10] also report results obtained by applying
traditional denoising techniques, i.e., empirical mode decomposition and
filtering, and demonstrate that their DL-based model provides a better data
denoising. Therefore, in this paper only comparisons with this benchmark model
and other DL-based proposals will be provided.

Subsequently, other approaches have been proposed in the literature, like Yu
et al.’s end-to-end DL framework, called _DeepSeparator_ [13]. This model is
based on Inception-like blocks and composed of (i) an encoder deputed to
feature extraction, (ii) a decomposer exploited to detect and remove OAs and
MAs, and (iii) a decoder used to reconstruct the cleaned signal.
Notice that the authors propose a training strategy where three input and
output pairs are designed to learn from both clean signals and artifacts:
$\langle$ noisy EEG, clean EEG $\rangle$, $\langle$ clean EEG, clean EEG
$\rangle$, and $\langle$ artifacts, artifacts $\rangle$.

Another proposal is _EEGDnet_ [14], which considers both non-local and local
self-similarities of EEG signals. Notice that the model has a 2D transformer
structure devised to remove OAs and MAs from 1D EEG signals. The clean and
noise signals are summed up considering a specific signal-to-noise ratio (SNR)
and the resulting noisy signals fed to EEGDnet. Afterwards, the input is
reshaped in a 2D matrix and passed to a self-attention block, a normalization
layer, a feed-forward block, and another normalization layer to finally
reconstruct the signal.
Similarly, Wang, Li, and Wang [15] propose a network mainly composed by a
bidirectional gated recurrent unit, a self-attention, and a dense layer to
remove OAs and MAs.

A Multi-Module Neural Network (MMNN) [16] is developed to be used in real-time
environments and considering single-channel EEG data. MMNN has a modular
structure constituted by blocks containing convolutional and fully-connected
layers. The model convergence and learning ability is supported by the
residual connections intra- and inter-blocks.

Other proposals exploit Generative Adversarial Networks (GANs) to remove
noise. For example, Brophy et al. [17] sample the generator input directly
from noisy EEG signals and make a comparison with the corresponding clean EEG
signals in the discriminator. The generator is constituted by a Long-Short
Term Memory (LSTM) network, while the discriminator is composed by four 1D
convolutional layers and a fully-connected layer.
Similarly, Wang, Luo, and Shen [18] generator consists of a Bidirectional-LSTM
(BiLSTM) and a LSTM layer, while the discriminator comprises five CNN layers
plus a fully-connected layer. The noisy EEG are passed to the generator,
producing the denoised EEG, which is inputted to the discriminator with the
ground truth data. Therefore, the authors’ main aim is to map the
relationships between clean EEG and artifacts to iteratively reduce the noise.

Finally, OAs only removal strategies are reported. Ozdemir, Kizilisik, and
Guren [19] focus on the use of BiLSTM and propose a benchmark combining
_EEGDenoiseNet_ and the _DEAP_ dataset [20]. Notice that the inputs of the
BiLSTM are the time-frequency features extracted from the augmented data.
Instead, Yin et al. [21] propose a cross-domain framework integrating time and
frequency domain information, demonstrating that the extracted features are
able to improve the performance of state-of-the-art methods when provided as
input to DL models.

##  3 Dataset

The _EEGdenoiseNet_ [10] is used in this study, having that it has been
provided to the research community as a benchmark dataset to train and test
DL-based denoising models.

In fact, Zhang et al. construct a dataset exploiting EEG, EOG and EMG signals
of publicly available datasets, processing these data to obtain neural and
noise signals that could be considered unaffected by other sources and thus
_clean_.
In particular, the authors consider Cho et al.’s EEG dataset [22], presenting
signals collected with 64 electrodes on 52 subjects during a motor execution
and imagery experiment.
The EOG signals have been instead taken from Kanoga et al. [23], and the BCI
Competition IV dataset 2a and 2b [24], while the EMG signals are related to a
facial EMG dataset [25].

Afterwards, the data are pre-processed as follows:

  1. 1.

Signals are notch (50 Hz) and bandpass (EEG: 1-80 Hz, EOG: 0.3-10 Hz, and EMG:
1-120 Hz) filtered.

  2. 2.

Signals are re-sampled, considering a sampling rate of 256 Hz or 512 Hz for
the EEG, 256 Hz for the EOG, and 512 Hz for the EMG signals.

  3. 3.

Signals are divided in segments of 2 s to provide data as cleaner as possible,
and standardized. Segments are visually inspected by experts.

Notice that between point 1 and 2, EEG signals are processed with the
independent component analysis based _ICLabel_ toolbox [26] to obtain clean
ground truth data. The data resulting from this process are 4,514, 3,400, and
5,598 _pure_ (as defined by the authors) EEG, EOG, and EMG segments,
respectively.
The pure EEG data are used as the _ground truth_ and semi-synthetic data
produced by linearly combining these data with EOG or EMG segments, according
to the following formula:

$y=x+\lambda n$ (2)

where $y$ is the noisy signal, $x$ the pure EEG signal, $n$ the EOG or EMG
noise, and $\lambda$ a hyperparameter controlling the SNR. For further
information, please consult the original publication by Zhang et al. [10].

##  4 Proposed model

In this work a novel denoising model leveraging data in the frequency domain
is proposed.

The idea is that given a prior knowledge about the noise spectral features, an
optimal convolutional filter, or a cascade of filters, can be computed to
separate the noise from signal. Therefore the proposed model is trained to
learn an empirical relationship which connects the spectral characteristics of
noise and noisy signal to a non-linear transformation able to denoise that
signal. The assumptions under which the model can operate are:

  * •

the PSD estimate of the noise is given;

  * •

the relation between signal and noise is known.

For the EEGdenoiseNet dataset this relationship is a linear mixture of the
clean signal and the artifacts (OAs and MAs considered one at a time) as per
Eq. 2.

The PSD related to noise $\lambda n$, the PSD related to noisy signal $y$, and
the noisy signal are all given separately as multiple inputs to the model,
which consists of two major components, repeatedly applied: the kernel
evaluator and the convolutional filters applier. The kernel evaluator is used
to evaluate the best convolutional filters from the frequencies that are
characteristic of the noisy signal and noise. The convolutional filters
applier then effectively apply the filters estimated by the kernel evaluator
to the time domain signal. The overall pipeline of the model is depicted in
Figure 1.

Figure 1: Model pipeline presenting the input form, the model mainly
constituted by the kernel evaluator and the convolutional filter applier, and
the obtained output.

###  4.1 Model inputs

The model is inputted with two inputs not directly interacting with each
other: the PSDs and the time series. The former is the concatenation of the
PSD of the pure noise, the PSD of the noisy signal (which is always known) and
the ratio between the noisy signal and the pure noise PSDs, which does not add
any further information but facilitates model learning since the ratio is an
operation not easily reproducible by the following convolutional operations.
The PSDs and their ratio are processed by the kernel evaluator blocks only.
The latter, i.e., the time series, is the noisy signal in the time domain,
which is processed in cascade by the convolutional filters applier block in
order to obtain the denoised signal.

###  4.2 Kernel evaluator

For each convolutional step a kernel evaluator block, which structure is
depicted in Figure 2, evaluates a set of convolutional filters (i.e., the
kernels values) to be applied to the time series.

Each block is inputted with the PSDs, which are processed by two symmetric
series of 1D convolutional layers with tanh activations. These branches
independently estimate the real and imaginary part of the filters that will be
subsequently applied to the time series. Remind that in this case the model is
working in the frequency domain, dealing with PSDs.

Figure 2: Kernel evaluator block diagram depicting the input form, the kernel
evaluator, and its output.

To translate the filters into the time domain, where they will actually
operate on the noisy signal, an inverse real fast Fourier transformation
(IRFFT) is then applied to the complex 1D arrays obtained by assembling the
two branches.

The IRFFT operation adds algorithmic capabilities to the model since the
convolutional and activation layers in the pipeline cannot replace it by
performing an analogous transformation.

The inverse fast Fourier transform (IFFT) is an algorithm that efficiently
computes the inverse discrete Fourier transform (IDFT) of a sequence and is
given as follows:

$x[n]=\frac{1}{N}\sum_{k=0}^{N-1}X[k]e^{i\frac{2\pi}{N}kn}$ (3)

The IRFFT is a particular case of IFFT which returns real-valued sequences, as
would be desired for the time-domain filters, without sacrificing the useful
algorithmic capabilities of the Fourier transform.

As a final step, the filters outputted by the IRFFT are linearly combined with
a 1D convolutional layer with kernel size equal to 1. The last tanh activation
forces the filters values in the range $[-1;1]$, avoiding numerical problems
coming from too high values and acting as a normalizer on filters.

The length of the filters evaluated by the kernel evaluator block is equal to
the length of the input noisy signal itself, thus the filters are able to act
on any signal frequency and to extract both local and global features. The
kernel evaluator blocks contain all and only the trainable parameters of the
model, specifically the kernels of the 1D convolutions applied to the PSDs.
Therefore these parameters, which amount to 224192, depend only on the
frequency characteristics of the signal and noise. The choice of the number of
convolutional layers and the size of the kernels they apply are dictated by
having a sufficiently large receptive field that operates on correlated
frequencies that are close to each other, without taking into account very
distant frequencies.

###  4.3 Convolutional filters applier

The filters evaluated by the kernel evaluator blocks are inputted to the
convolutional filter applier, which uses them without further changes. As
depicted in Figure 3, on the first step this part of the model applies 1D
convolutions directly to the noisy signal using the filters of the first
kernel evaluator block and the resulting features are inputted to an ELU
activation function.

Figure 3: Convolutional filter applier schema using the previously obtained
filters as inputs. The output is the denoised signal.

Subsequently, these features are convoluted with the filters evaluated by the
second block and once again activated by the ELU function. This operation is
repeated in cascade until the last convolution, which returns directly the
denoised signal. No activation is applied in this last step and the range of
possible values is therefore $(-\infty;+\infty)$, which is also the range of
the possible clean signals.

The number of convolutions applied in series and the number of filters applied
at each convolutional step were sized based on domain knowledge and in a way
that limited the total number of parameters trained in the kernel evaluator
blocks.

##  5 Performed experiments

The denoising capabilities of the proposed model are evaluated on the basis of
the reconstruction performance of pure EEG signals from the EEGdenoiseNet
dataset when contaminated by EOG or EMG artifacts, separately, following the
paradigm proposed by the authors of the benchmark model [10] and considered by
most of the reported literature works (Section 2). In this section the data
preparation method, the metrics used to evaluate the model and the training
process are described. Finally, statistical and graphical results of the model
are reported.

###  5.1 Data preparation

The original dataset has been randomly partitioned into two mutually exclusive
subsets, i.e., a training dataset (60$\%$) and a test dataset (40$\%$).
Therefore, the training dataset consists of 2,708 EEG, 3,358 EMG and 2,040 EOG
samples while the test dataset consists of 1,806 EEG, 2,240 EMG and 1,360 EOG
samples.

To synthesize the noisy signals from the pure samples, the linear relation
defined in Eq. 2 has been used. The $\lambda$ values are randomly sampled to
obtain a uniform distribution of the signal-to-noise ratio (SNR) in the range
$[-7dB;4dB]$ for the model training and in the range $[-7dB;2dB]$ for the
model testing. This is a common range for OAs and MAs and the same range has
been used by the EEGdenoiseNet authors [10].

During the training phase, the noisy data synthesis is performed runtime in a
random way, allowing the model to be trained with constantly new combinations
of signal, noise and SNR.

Both noisy signal and pure noise are standardized using the mean and standard
deviation of the noisy signal.

###  5.2 Metrics

In order to quantitatively evaluate the performance of the model in the
denoising task, standard metrics used for benchmarking on _EEGdenoiseNet_ data
are adopted.

The Root Mean Square Error (RMSE) is used to measure the variance between the
output predicted by the model and the ground truth and it is defined by:

$RMSE=\sqrt{\frac{\sum_{i=1}^{N}(x_{i}-\tilde{y}_{i})^{2}}{N}}$ (4)

where $x$ denotes the EEG signal, $\tilde{y}$ the denoised signal and $N$ the
total number of data points of the signal.

To avoid a metric depending on the absolute value of the signals, the Relative
Root Mean Square Error (RRMSE) is used, which in the time domain is expressed
as follows:

$RRMSE_{t}=\sqrt{\frac{\sum_{i=1}^{N}(x_{i}-\tilde{y}_{i})^{2}}{\sum_{i=1}^{N}x_{i}^{2}}}$
(5)

while when considering the frequency domain we obtain the following:

$RRMSE_{f}=\sqrt{\frac{\sum_{i=1}^{N}(PSD(x)_{i}-PSD(\tilde{y})_{i})^{2}}{\sum_{i=1}^{N}PSD(x)_{i}^{2}}}$
(6)

The correlation coefficient (CC), also referred to as Pearson correlation
coefficient, measures the degree of the statistical relationship between two
variables, in this case the ground truth signal and the denoised signal. The
CC takes values in the range $[-1;1]$, where $\pm 1$ indicates complete linear
dependence between the variables, while $0$ could mean their independence, and
is defined as follows:

$CC=\frac{\sum_{i=1}^{n}(x_{i}-\overline{x})(\tilde{y}_{i}-\overline{y})}{\sqrt{\sum_{i=1}^{n}(x_{i}-\overline{x})^{2}}\sqrt{\sum_{i=1}^{n}(\tilde{y}_{i}-\overline{y})^{2}}}$
(7)

where $\overline{x}$ and $\overline{y}$ are the means of the ground truth and
of the model output, respectively.

###  5.3 Training process

Standardization helps to speed up the training process since input centering
and scaling operations improve the rate at which the neural network converges
[27]. Indeed, the learning algorithm is sensitive to the input scale, and if
the input data are not standardized, it may take longer for the algorithm to
find a good set of parameters, i.e., weights and thresholds of the network, or
the learning algorithm may get stuck in a local minima. Moreover, the
standardization of the input data makes the model capable of processing EEG
signals with wider amplitude ranges. Nevertheless, only the mean and standard
deviation of the noisy signal are always known. Therefore, the noisy signal
$y$, the pure EEG signal $x$ and the pure noise $n$ are processed in a similar
manner according to:

$\hat{x}=\frac{x-\overline{y}}{\sigma_{y}},\quad\hat{y}=\frac{y-\overline{y}}{\sigma_{y}},\quad\hat{n}=\frac{n-\overline{y}}{\sigma_{y}}$
(8)

These signals, as well as the PSDs evaluated by them, are the actual inputs of
the model.

The loss function minimized in the training phase is a combination of three
different terms, differently weighted:

$L=aL_{RRMSE_{t}}+bL_{RRMSE_{f}}+cL_{log-cosh}$ (9)

where $L_{RRMSE_{t}}$ is the temporal RRMSE defined in Eq. 5, $L_{RRMSE_{f}}$
is the spectral RRMSE defined in Eq. 6, and $L_{log-cosh}$ the log-cosh error
of the ground truth and predicted signals in the time domain, which is defined
by the equation:

$L_{log-cosh}(x,\tilde{y})=\sum_{i=1}^{N}\log(\cosh(\tilde{y}_{i}-x_{i}))$
(10)

This function provides a smooth approximation to the mean absolute error for
values near 0. The $a$, $b$, and $c$ coefficients are empirically chosen for
the training and are equal to $0.25$, $0.25$, and $0.5$, respectively.

The optimization method used to find the best weights of the kernel evaluator
blocks is AdaMax [28].

The code was developed in Python 3.8.10 and the proposed model was designed
using the TensorFlow library, version 2.8. The experiments were run on an
Nvidia Quadro RTX 4000 for a total training time of 61 hours.

###  5.4 Results and discussions

A single model has been trained on both EMG and EOG artifacts at the same time
in order to have a solution capable of handling both cases. In fact, the noisy
signals are affected by either OAs or MAs. Therefore, both EMG only and EOG
only affected signals are inputted to the model, as introduced at the
beginning of Section 5.

Figure 4: Loss of the training set and test set as training epochs increase.

Figure 4 shows the trend of loss as the epochs change for the training and
test sets. For both datasets the loss values monotonically decrease and no
significant overfit is present, indication that the model design and the
runtime data synthesis approach used are effective in avoiding this issue.

Figure 5: Test examples of noisy EEG signals corrupted by muscular or ocular
artifacts with a signal-to-noise ratio equal to -7 $dB$ denoised by the model.
The timestep denotes the signal sample.

Using the last epoch weights, we qualitatively demonstrate the denoising
capabilities of the model on EOG and EMG artifacts in Figure 5. We can observe
that both the high frequencies related to MAs and low frequencies related to
OAs are filtered by the model while the majority of the structures related to
the true EEG signals are preserved. Moreover, several amplitude ranges are
properly managed by the model as well as different SNRs.

The quantitative results of the proposed model and of the literature benchmark
models for MAs and OAs are reported in Table 1 and 2, respectively.

Muscular artifacts
---
Model | $RRMSE_{t}$ | $RRMSE_{f}$ | $CC$
FCNN [12] | 0.585 | 0.580 | 0.796
Simple CNN [12] | 0.646 | 0.649 | 0.783
Complex CNN [12] | 0.650 | 0.633 | 0.780
RNN [12] | 0.570 | 0.530 | 0.812
Novel CNN [12] | 0.448 | 0.442 | 0.863
DeepSeparator [13] | 0.712 | 0.717 | 0.734
EEGDnet [14] | 0.677 | 0.626 | 0.732
Proposed model | 0.573 | 0.496 | 0.805
Table 1: Metrics of the proposed denoising model and benchmark models averaged
on all SNRs ($[-7dB;2dB]$) with noisy signals affected by MAs. Ocular
artifacts
---
Model | $RRMSE_{t}$ | $RRMSE_{f}$ | $CC$
DeepSeparator [13] | 0.705 | 0.747 | 0.769
EEGDnet [14] | 0.497 | 0.491 | 0.868
Proposed model | 0.405 | 0.490 | 0.917
Table 2: Metrics of the proposed denoising model and benchmark models averaged
on all SNRs ($[-7dB;2dB]$) with noisy signals affected by OAs.

Regarding EEG affected by MAs, the reported values are in line with the
benchmark models results. The proposed model demonstrated robust performance,
achieving the third best results in the temporal domain metrics, $RRMSE_{t}$
and $CC$, and the second best result in the spectral metric $RRMSE_{f}$. This
latter metric provides insight into the method ability to effectively preserve
the spectral information of the clean EEG signal.

Concerning OAs, our method achieves the best performance on all three metrics
among the reported benchmark models. Furthermore, the similarity of the
proposed model results in $RRMSE_{f}$ values on both muscular and ocular
artifacts highlights the method stability and its ability to perform
indiscriminately well on both high and low frequencies.

In contrast to the current state of the art, the proposed model is able to
achieve these results using not only a single architecture but also a single
set of trained parameters, valid on both MAs and OAs. This allows the two main
sources of noise in EEG signals to be handled with a single solution. However,
while the need to have as input an estimate of the PSD of the noise afflicting
the signal makes the method more adaptable to new noise conditions with known
characteristics, it also makes it sensitive to precise a priori knowledge of
the frequency characteristics of this noise that cannot be easily estimated.
In the future we will develop methods trained on the reference dataset to best
estimate the PSD of the noise of each signal as a substitute for the exact PSD
provided in the current work. Moreover, different activation functions and
artifact amount estimation methodologies [29] will be considered to provide a
better denoising of the EEG signals.

##  6 Conclusions

The significant issue of muscle and ocular artifact removal in EEG data has
been tackled in this research. We introduced a unique solution capable of
dealing with both artifact types using a single model. Our proposed method
leverages dynamically assessed convolutional filters, which are determined
based on the frequency features of the noise and the noisy signal. With this
knowledge, our model has proven its effectiveness in both qualitatively and
quantitatively cleaning EEG signals from muscle and ocular artifacts. This
achievement either matches or exceeds the performance of existing state-of-
the-art models, which typically require specific training on either ocular or
muscle artifacts. Thus, this study marks a substantial step forward in EEG
data processing by offering a versatile spectral-based strategy for artifacts
elimination and provides a baseline for subsequent work addressing the problem
of estimating noise frequencies, whether with experimental solutions,
algorithmic solutions, or a combination of the two.

###### Acknowledgements.

This work was partially supported by the MUR under the grant “Dipartimenti di
Eccellenza 2023-2027" of the Department of Informatics, Systems and
Communication of the University of Milano-Bicocca, Italy.

## References

  * Saibene et al. [2023] A. Saibene, M. Caglioni, S. Corchs, F. Gasparini,  Eeg-based bcis on motor imagery paradigm using wearable technologies: A systematic review,  Sensors 23 (2023) 2798.
  * Zhang et al. [2020] J. Zhang, Z. Yin, P. Chen, S. Nichele,  Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review,  Information Fusion 59 (2020) 103–126.
  * Urigüen and Garcia-Zapirain [2015] J. A. Urigüen, B. Garcia-Zapirain,  Eeg artifact removal—state-of-the-art and guidelines,  Journal of neural engineering 12 (2015) 031001.
  * Rashmi and Shantala [2022] C. Rashmi, C. Shantala,  Eeg artifacts detection and removal techniques for brain computer interface applications: a systematic review,  International Journal of Advanced Technology and Engineering Exploration 9 (2022) 354\.
  * Jiang et al. [2019] X. Jiang, G.-B. Bian, Z. Tian,  Removal of artifacts from eeg signals: a review,  Sensors 19 (2019) 987.
  * Mumtaz et al. [2021] W. Mumtaz, S. Rasheed, A. Irfan,  Review of challenges associated with the eeg artifact removal methods,  Biomedical Signal Processing and Control 68 (2021) 102741.
  * Wallstrom et al. [2004] G. L. Wallstrom, R. E. Kass, A. Miller, J. F. Cohn, N. A. Fox,  Automatic correction of ocular artifacts in the eeg: a comparison of regression-based and component-based methods,  International journal of psychophysiology 53 (2004) 105–119.
  * Mannan et al. [2018] M. M. N. Mannan, M. A. Kamran, M. Y. Jeong,  Identification and removal of physiological artifacts from electroencephalogram signals: A review,  Ieee Access 6 (2018) 30630–30652.
  * Chen et al. [2019] X. Chen, X. Xu, A. Liu, S. Lee, X. Chen, X. Zhang, M. J. McKeown, Z. J. Wang,  Removal of muscle artifacts from the eeg: A review and recommendations,  IEEE Sensors Journal 19 (2019) 5353–5368.
  * Zhang et al. [2021] H. Zhang, M. Zhao, C. Wei, D. Mantini, Z. Li, Q. Liu,  Eegdenoisenet: a benchmark dataset for deep learning solutions of eeg denoising,  Journal of Neural Engineering 18 (2021) 056057.
  * Minguillon et al. [2017] J. Minguillon, M. A. Lopez-Gordo, F. Pelayo,  Trends in eeg-bci for daily-life: Requirements for artifact removal,  Biomedical Signal Processing and Control 31 (2017) 407–418.
  * Zhang et al. [2021] H. Zhang, C. Wei, M. Zhao, Q. Liu, H. Wu,  A novel convolutional neural network model to remove muscle artifacts from eeg,  in: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, pp. 1265–1269.
  * Yu et al. [2022] J. Yu, C. Li, K. Lou, C. Wei, Q. Liu,  Embedding decomposition for artifacts removal in eeg signals,  Journal of Neural Engineering 19 (2022) 026052.
  * Pu et al. [2022] X. Pu, P. Yi, K. Chen, Z. Ma, D. Zhao, Y. Ren,  Eegdnet: Fusing non-local and local self-similarity for eeg signal denoising with transformer,  Computers in Biology and Medicine 151 (2022) 106248.
  * Wang et al. [2022] W. Wang, B. Li, H. Wang,  A novel end-to-end network based on a bidirectional gru and a self-attention mechanism for denoising of electroencephalography signals,  Neuroscience 505 (2022) 10–20.
  * Zhang et al. [2022] Z. Zhang, X. Yu, X. Rong, M. Iwata,  A novel multimodule neural network for eeg denoising,  IEEE Access 10 (2022) 49528–49541.
  * Brophy et al. [2022] E. Brophy, P. Redmond, A. Fleury, M. De Vos, G. Boylan, T. Ward,  Denoising eeg signals for real-world bci applications using gans,  Frontiers in Neuroergonomics 2 (2022) 44.
  * Wang et al. [2022] S. Wang, Y. Luo, H. Shen,  An improved generative adversarial network for denoising eeg signals of brain-computer interface systems,  in: 2022 China Automation Congress (CAC), IEEE, 2022, pp. 6498–6502.
  * Ozdemir et al. [2022] M. A. Ozdemir, S. Kizilisik, O. Guren,  Removal of ocular artifacts in eeg using deep learning,  in: 2022 Medical Technologies Congress (TIPTEKNO), IEEE, 2022, pp. 1–6.
  * Koelstra et al. [2011] S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, I. Patras,  Deap: A database for emotion analysis; using physiological signals,  IEEE transactions on affective computing 3 (2011) 18–31.
  * Yin et al. [2022] J. Yin, A. Liu, C. Li, R. Qian, X. Chen,  Frequency information enhanced deep eeg denoising network for ocular artifact removal,  IEEE Sensors Journal 22 (2022) 21855–21865.
  * Cho et al. [2017] H. Cho, M. Ahn, S. Ahn, M. Kwon, S. C. Jun,  Eeg datasets for motor imagery brain–computer interface,  GigaScience 6 (2017) gix034.
  * Kanoga et al. [2016] S. Kanoga, M. Nakanishi, Y. Mitsukura,  Assessing the effects of voluntary and involuntary eyeblinks in independent components of electroencephalogram,  Neurocomputing 193 (2016) 20–32.
  * Tangermann et al. [2012] M. Tangermann, K.-R. Müller, A. Aertsen, N. Birbaumer, C. Braun, C. Brunner, R. Leeb, C. Mehring, K. J. Miller, G. Mueller-Putz, et al.,  Review of the bci competition iv,  Frontiers in neuroscience (2012) 55\.
  * Rantanen et al. [2016] V. Rantanen, M. Ilves, A. Vehkaoja, A. Kontunen, J. Lylykangas, E. Mäkelä, M. Rautiainen, V. Surakka, J. Lekkala,  A survey on the feasibility of surface emg in facial pacing,  in: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, 2016, pp. 1688–1691.
  * Pion-Tonachini et al. [2019] L. Pion-Tonachini, K. Kreutz-Delgado, S. Makeig,  Iclabel: An automated electroencephalographic independent component classifier, dataset, and website,  NeuroImage 198 (2019) 181–197.
  * LeCun et al. [2002] Y. LeCun, L. Bottou, G. B. Orr, K.-R. Müller,  Efficient backprop,  in: Neural networks: Tricks of the trade, Springer, 2002, pp. 9–50.
  * Kingma and Ba [2014] D. P. Kingma, J. Ba,  Adam: A method for stochastic optimization,  arXiv preprint arXiv:1412.6980 (2014).
  * Vialatte et al. [2008] F.-B. Vialatte, J. Solé-Casals, A. Cichocki,  Eeg windowed statistical wavelet scoring for evaluation and discrimination of muscular artifacts,  Physiological Measurement 29 (2008) 1435.