11institutetext: EURA NOVA, Mont-St-Guibert, Belgium
11email<EMAIL_ADDRESS>
# DAEMA: Denoising Autoencoder with Mask Attention

Simon Tihon equal contribution 0000-0002-3985-1967 Muhammad Usama Javaid*
0000-0001-9262-2250 Damien Fourure  0000-0001-5085-0052 Nicolas Posocco
0000-0002-1795-6039 Thomas Peel  0000-0003-2967-6381

###### Abstract

Missing data is a recurrent and challenging problem, especially when using
machine learning algorithms for real-world applications. For this reason,
missing data imputation has become an active research area, in which recent
deep learning approaches have achieved state-of-the-art results. We propose
DAEMA (Denoising Autoencoder with Mask Attention), an algorithm based on a
denoising autoencoder architecture with an attention mechanism. While most
imputation algorithms use incomplete inputs as they would use complete data -
up to basic preprocessing (e.g. mean imputation) - DAEMA leverages a mask-
based attention mechanism to focus on the observed values of its inputs. We
evaluate DAEMA both in terms of reconstruction capabilities and downstream
prediction and show that it achieves superior performance to state-of-the-art
algorithms on several publicly available real-world datasets under various
missingness settings.

Figure 1: The goal of DAEMA is to produce a representation of the input which
is robust to missingness, so that a Feature Decoder can impute the missing
data from it. To do so, a Feature Encoder produces multiple values for each
latent feature. These are then weighted by a Feature Selector based on the
missingness mask. This mask enables the attention mechanism to focus on the
values that are produced using only observed inputs.

##  1 Introduction

Machine learning researchers and practitioners frequently encounter the
problem of missing data. Data can be missing for many reasons. These include
system failures, loss of data, or the fact that data was never known, measured
or recorded. Missing data can introduce bias and alter the statistical
properties of a dataset. This can impact the performance of models learnt from
this data, both in obvious (e.g. poor performance due to lack of data) or
subtle (e.g. a bias is learnt) ways. Simple approaches consist in discarding
samples with missing data, removing an entire feature if it is too often
missing or imputing the mean/median value per feature. In practice, these
provide a satisfying solution when a small proportion of the data is missing.
Otherwise, it is likely to significantly alter the empirical data
distribution.

Missing data imputation has received a lot of attention and many approaches
have been proposed recently [21, 12, 9]. The literature can be divided into
two categories: discriminative and generative approaches. Discriminative
approaches model the conditional probability of each feature given the others.
That is, they use all features but one to impute the remaining feature, and
iterate over all features to produce a complete dataset. On the other hand,
generative approaches model the joint distribution of all features to impute
missing data all at once.

Generative approaches for missing data imputation generally involve some form
of denoising autoencoder [19]. A denoising autoencoder deals with noise,
taking noisy samples as input and learning to reconstruct the cleaned samples.
As missing data is a special case of noisy data, a denoising autoencoder can
be used to reconstruct the missing parts.

Attention is a very popular technique in the fields of natural language
processing and computer vision. Introduced in [18], attention mechanisms can
be summed up as paying attention to parts of the input which are relevant to
generate an output. Attention enables a model to understand the underlying
structure of data better, resulting in better generalisation [21].

To leverage this promising technique, we propose DAEMA (Denoising Autoencoder
with Mask Attention). It improves on a simple yet efficient denoising
autoencoder architecture by adding an attention mechanism based on the
missingness mask. Its architecture can be seen in Figure 1. Thanks to a mask-
based attention module, the latent representation produced by DAEMA is a
missingness-robust embedding of the original sample. This approach achieves
good performance with both randomly and systematically missing data, beating
previous state-of-the-art techniques.

##  2 Related Work

Early approaches have modelled the task of missing data imputation as a
predictive task. These are referred to as discriminative approaches. Some of
them are based on k-nearest neighbours [17] and support vector techniques
[20], using models whose capacity is often insufficient for the task. Others
are iterative, such as MissForest [16] and Multiple Imputation using Chained
Equation (MICE) [1], which respectively use random forests and linear
regressors to predict missing values for one feature at a time. Both
approaches impute a same dataset several times, using the previous result to
compute the next one more precisely. Since they produce multiple imputation of
said dataset, these are referred to as multiple-imputation methods. Having
multiple-imputed datasets can be an advantage as it helps to account for the
uncertainty in the imputation process. However, these iterative methods are
meant to impute whole datasets at once. This can be a drawback in real-world
situations, where new data comes everyday.

Due to recent advances in deep learning and more particularly in generative
deep-learning, more recent works have modelled the task of missing data
imputation as a generative task. The resulting methods are referred to as
generative approaches. One such method is called Multiple Imputation using
Denoising Autoencoders (MIDA) [4]. It applies the denoising autoencoder
approach to missing data imputation. Another method, called Generative
Adversarial Imputation Nets (GAIN) [22], leverages a conditional GAN [5] to
learn the real distribution of data through adversarial training. However,
GANs are known to be hard to train, suffering from non-convergence and mode
collapse. WGAN-GP [6] refines GANs to overcome these problems and ensure
better training. Other methods, such as VAE [8] and HI-VAE [10], use a
variational autoencoder architecture. More recently, MCFlow [12] leverages
normalising flows and Monte Carlo sampling for imputation. Finally, GINN [15]
exploits graph convolutional networks to take advantage of the information
contained in the nearest neighbours at imputation time instead of using only
the information contained in each sample being imputed.

A recent deep-learning discriminative method, AimNet [21], achieves state-of-
the-art results with a dot-product attention mechanism applied to the
individual embedding of each feature. Its attention weights depend only on
which feature is being imputed.

Finally, some variants of existing methods have been proposed. In [2],
embeddings for categorical input features and gumbel-softmax activation layers
for categorical output features are successfully applied to GAIN and to a VAE
architecture. In [9], the optimal transport distance between two batches is
proposed to be used as a training loss for missing data imputation models.
These approaches are complementary to our method.

We propose DAEMA, a generative method based on a denoising autoencoder and an
attention mechanism. In contrast to AimNet, our attention mechanism is input-
oriented: the attention weights of DAEMA depend on which feature values are
missing instead of depending on which feature we want to impute. Therefore,
the model can focus on non-missing values and distinguish them from
placeholder values. To our knowledge, DAEMA is the first method to use an
input-oriented attention mechanism for the imputation problem.

##  3 Problem Statement

A dataset is defined as $D=\\{\mathbf{X},\ C,\ N\\}$.
$\mathbf{X}\in\mathbb{R}^{n\times d}$ is a matrix of data composed of $n$
samples of $d$ features $\mathbf{x}_{i}=(x_{i}^{1},\ \ldots,\
x_{i}^{d})\in\mathbb{R}^{d}$. $\ C$ (resp. $N$) is the set of indices of
categorical (resp. numerical) features, that is, features taking discrete
(resp. continuous) values. In this work, in order to keep the loss and metrics
simple, we focus on datasets containing only numerical features, i.e.
$C=\emptyset$.

For the missing data imputation task, a dataset with missing values in
$\mathbf{X}$ is given. We define the missingness-mask matrix
$\mathbf{M}\in\\{0,1\\}^{n\times d}$ such that $x_{i}^{j}$ is missing if and
only if $m_{i}^{j}=1$. We denote $D^{*}=\\{\mathbf{X^{*}},\ C,\ N\\}$ the
ground truth dataset without missing data. Missing data are generally
classified into three different categories [13, 14]:

  * •

Missing completely at random (MCAR) means the probability that a value is
missing does not depend on any value in the dataset.

  * •

Missing at random (MAR) means the probability that a value is missing depends
only on the observed (non-missing) values.

  * •

Missing not at random (MNAR) means the probability that a value is missing
depends on unobserved values or latent variables.

We are interested in learning an imputation function
$f:\mathbb{R}^{d}\times\\{0,1\\}^{d}\to\mathbb{R}^{d};(\mathbf{x},\mathbf{m})\mapsto
f(\mathbf{x},\mathbf{m}).$ Using the dataset $D$ and the corresponding
missingness-mask matrix $\mathbf{M}$, the goal is to find the best function
$f^{{\dagger}}$ minimizing a reconstruction metric and a metric based on a
downstream machine-learning task as described in Section 6.2.

##  4 Our Approach

Figure 1 shows the architecture of DAEMA, which is based on a standard
denoising autoencoder. At its core is a mask-based attention mechanism,
designed to help the network efficiently use the available data to produce
robust latent representations. This in turn makes imputation possible by
decoding these very representations.

###  4.1 Denoising Autoencoder

As a denoising autoencoder, DAEMA takes a noisy input sample $\mathbf{x}$ and
produces a clean version $\mathbf{\hat{x}}$ of it. In our case, noise is
defined as missingness of data, meaning that ground truth values of the
missing data are unknown. It is trained using additional artificial
missingness in each batch, as the values of the missing features are needed to
train the network. As it is impossible for the model to distinguish between
originally missing values and artificially missing values, the model
reconstructs both. Let $\mathbf{m}$ be the original missingness mask of the
input sample $\mathbf{x}$, $\mathbf{\bar{m}}$ the one including artificial
missingness and
$\mathbf{\bar{x}}=\mathbf{x}\cdot(\mathbf{1}-\mathbf{\bar{m}})$ the sample
with artificially missing values.

Because the originally missing values are unknown, the reconstruction loss has
to take into account the missingness mask $\mathbf{m}$, as done in GAIN [22]
and MCFlow [12]. The masked reconstruction loss used is defined as follows:

$\ell(\mathbf{\hat{x}}_{i},\mathbf{x}_{i},\mathbf{m}_{i})=\sum_{j}(1-m_{i}^{j})\cdot(x_{i}^{j}-\hat{x}_{i}^{j})^{2}$
(1)

Minimising the loss implies correctly imputing artificially missing data as
well as correctly reconstructing observed values. However, the minimisation of
the loss is not impacted by originally missing data.

###  4.2 Mask-Based Attention Mechanism

To help the model focus on non-missing values, we add an attention mechanism
into the encoder part of the network. The intuition is that, for different
missingness patterns, the model has to focus on different non-missing values
to reconstruct the missing ones. By adding an attention mechanism based on the
missingness mask, the model can choose the values it has to focus on. As shown
in Figure 1, the attention mechanism is based on three elements: the Feature
Encoder, the Feature Selector and a Sum aggregation.

The Feature Encoder $f_{e}$ is a function that takes as input a sample with
artificial missingness $\mathbf{\bar{x}}$ and its corresponding missingness
mask $\mathbf{\bar{m}}$ and produces $d_{z}$ feature vectors of dimension
$d^{\prime}$:

$f_{e}(\mathbf{\bar{x}},\mathbf{\bar{m}})=(\mathbf{f}^{1},\ldots,\mathbf{f}^{d_{z}})\text{
with }\mathbf{f}^{j}\in\mathbb{R}^{d^{\prime}}$ (2)

In practice, the Feature Encoder is implemented as a multilayer perceptron
producing an output of size $d^{\prime}\cdot d_{z}$. This output is then
reshaped into a two-dimensional feature map of size $(d^{\prime},d_{z})$.

The Feature Selector $f_{s}$ is a function that takes the artificial
missingness mask $\mathbf{\bar{m}}$ as input and produces $d_{z}$ selection
vectors of dimension $d^{\prime}$:

$f_{s}(\mathbf{\bar{m}})=(\mathbf{s}^{1},\ldots,\mathbf{s}^{d_{z}})\text{ with
}\mathbf{s}^{j}\in\mathbb{R}^{d^{\prime}}$ (3)

It is also implemented using a multi-layer perceptron which produces an output
of size $d^{\prime}\cdot d_{z}$. This output is then reshaped into a two-
dimensional feature map of size $(d^{\prime},d_{z})$.

The objective of the Feature Encoder is to create multiple estimation values
for each latent feature (i.e. one feature vector $\mathbf{f}^{j}$ per latent
feature $z^{j}\in\mathbf{z}$) while the goal of the Feature Selector is to
give more attention to the most meaningful values from each vector
$\mathbf{f}^{j}$ according to the artificial missingness mask
$\mathbf{\bar{m}}$.

The Attention Mechanism combines feature vectors and selection vectors. To do
so, the selection vectors $\mathbf{s}^{j}$ are normalized with a softmax
function $\sigma$. Then, an element-wise product is used to combine each
feature vector $\mathbf{f}^{j}$ with its selection vector $\mathbf{s}^{j}$.
Finally, a summation aggregates each resulting vector to get the final latent
representation $\mathbf{z}$. Mathematically:

$\mathbf{z}=(\sigma(\mathbf{s}^{1})^{\mathsf{T}}\cdot\mathbf{f}^{1},\ldots,\sigma(\mathbf{s}^{d_{z}})^{\mathsf{T}}\cdot\mathbf{f}^{d_{z}})=(z^{1},\ldots,z^{d_{z}}).$
(4)

##  5 Technical Implementation

In order to ensure the reproducibility of our results, the architecture and
implementation details are thoroughly defined in this section. The full
training and testing code of DAEMA is
available111https://github.com/euranova/DAEMA.

###  5.1 Preprocessing Steps

To evaluate the imputation performance of DAEMA, complete datasets (i.e.
without missing values) are needed. To train the model, we create real-world-
like datasets $D$ (with missing values) from ground truth datasets $D^{*}$.
The preparation of all datasets follows these steps:

  * •

We separate the downstream target label from the other features. In real-world
applications, the target is unknown during the imputation process as it is
predicted by the downstream model afterwards.

  * •

For a few datasets, we remove some samples, such as samples containing
naturally missing data or extreme outliers, as detailed in Section 6.1.

  * •

We introduce missing values in the ground truth datasets. We use the two
uniform mechanisms described in the MIDA paper [4] to create both MCAR and
MNAR data. For the MCAR setting, each single value has 20% chance to be
removed, possibly removing none or all the values of a sample. For the MNAR
setting, we randomly choose two features and select the samples for which a)
the first feature has a value smaller or equal to the median of that first
feature or b) the second feature has a value bigger or equal to the median of
that second feature. Then, each single value of the selected samples has 20%
chance to be removed.

  * •

We randomly split each dataset into a train set and a test set. We use a 70-30
ratio for all experiments.

  * •

We apply a z-normalisation on the non-missing values by subtracting the mean
and dividing by the variance of the training set.

###  5.2 Detailed Architecture

Inspired by AimNet [21] and GAIN [22], DAEMA has been kept as shallow as
possible. The Feature Encoder is a multilayer perceptron with two layers both
using the hyperbolic tangent as activation. The input size is $2d$
($\mathbf{\bar{x}}$ concatenated with $\mathbf{\bar{m}}$). Both hidden and
output layers are of size $d^{\prime}\cdot d_{z}$. The output of the Feature
Encoder is reshaped into a $(d^{\prime},d_{z})$ feature map. The Feature
Selector is a single fully-connected layer without activation going from
dimension $d$ to dimension $d^{\prime}\cdot d_{z}$. The output vector is
reshaped into a two dimensional matrix of size $(d^{\prime},d_{z})$. The
Feature Decoder is a single fully-connected layer of dimension $d$ without
activation. For our experiments, $d^{\prime}$ and $d_{z}$ are set to $2d$.

During the training, the artificial missingness mask $\mathbf{\bar{m}}$ and
its corresponding sample $\mathbf{\bar{x}}$ are built on the fly by randomly
removing features with a $0.2$ probability. Besides, we use the masked
reconstruction loss defined in equation 1. The model is trained by gradient
descent using the Adam optimizer [7] with a learning rate of $0.001$ and a
batch-size of $64$. The training is stopped after $40,\\!000$ batch-steps.

All hyperparameters are chosen based on early experiments. Fine-tuning these
hyperparameters could probably improve the performance of the algorithm.
However, this is out of the scope of this paper and is left for future work.

##  6 Experimentation

To validate our approach, we compare it to state-of-the-art algorithms on
different datasets and with different missingness proportions.

###  6.1 Datasets

We use seven publicly available real-life datasets, six from the UCI
repository [3] and one from the sklearn.datasets module [11]: EEG Eye State
(14,976 samples, 14 features, 2 classes), Glass (214 samples, 9 features, 6
classes), Breast Cancer (683 samples, 9 features, 2 classes), Ionoshpere (351
samples, 34 features, 2 classes), Shuttle (58,000 samples, 9 features, 7
classes),Boston Housing (506 samples, 13 features, regression) and CASP
(45,730 samples, 9 features, regression). These datasets have different sizes
and different dimensions to perform comprehensive experimentation. Note that
the Breast Cancer dataset has samples with NA values and the EEG dataset has
four extreme outliers in its sixth feature. We remove these samples from the
datasets. Boston, Glass, Breast Cancer, Ionosphere and Shuttle datasets are
used for experimentation in MIDA [4]. EEG and CASP datasets are used for
experimentation in AimNet [21]. As these two models are state-of-the-art and
direct competitors to our model, it is more relevant to perform comparison on
these datasets.

###  6.2 Metrics

We use two metrics to compare the algorithms, namely a reconstruction metric
and a downstream metric. The reconstruction metric is the normalised root mean
square error (NRMS) over the missing values of the test set, as in [21]. The
normalisation is done using all the ground truth values of the test set.
Although NRMS is a direct metric to assess the performance of the models, it
favours predicting the mean of multi-modal distributions rather than one of
the possible modes. To assess better the ability of the models to capture the
structure of the data, we also use a complementary downstream metric. This
metric is more relevant regarding real-world situations as missing data
imputation is often used as a preprocessing step to train a model. However,
this metric is indirect and results in a less precise evaluation.

As downstream models, we use random forests trained and tested respectively on
the imputed train set and the imputed test set. The performance of a random
forest is measured by the NRMS in the case of a regression and the accuracy in
the case of a classification. Because of the randomness of the random forest
algorithm, we define the metric as the mean performance of ten random forests
trained with different seeds. We use random forests with 100 estimators and at
most 1000 leaf nodes per estimator for computational reasons.

To obtain more stable results from each run, the metrics are averaged on five
of the last training steps (see Section 6.3 for details). To account for the
randomness of the dataset preprocessing and the one of the models
initialisation and training procedure, each experiment is run ten times, using
ten different seeds. All compared algorithms are run on the same ten
preprocessed datasets to obtain more significant results. The mean of the ten
metric measurements obtained is reported in order to compare the algorithms
with each other. The sample variance of the ten values obtained is reported
for better reproducibility.

###  6.3 Compared Algorithms

To validate our approach, we have selected and implemented three state-of-the-
art algorithms for comparison: AimNet [21], MIDA [4] and MissForest [16]. We
have also implemented a custom denoising autoencoder, which we will refer to
as DAE. For DAEMA, we evaluate the model after 39200, 39400, 39600, 39800 and
40000 batch-steps. For AimNet, we use the hyperparameters described in the
paper [21]. We evaluate the model after 18, 19, 20, 21 and 22 epochs. For
MIDA, we have not found any indication on the batch size. Therefore, we use
the whole train set at each training step. Moreover, we use the Adam optimiser
with a learning rate of 0.0001 instead of the recommended one as we found it
achieves better results. We evaluate the model after 492, 494, 496, 498 and
500 epochs.

We also compare DAEMA with MissForest. However, MissForest targets a fix
dataset, as it reconstructs the whole dataset at once. Thus it can be seen as
a one shot process only, with a computational burden that makes it difficult
to apply on a data stream. To have a fair comparison we had to adapt the
algorithm. During the training procedure, we save all predictors produced by
each iteration. At test time, each predictor is applied one by one to unseen
data. We use random forests of 100 estimators each without any leaf-node
limit. The maximum number of iterations is set to 10. For this approach, as
the number of steps is dynamically chosen, we evaluate the model only once,
i.e. after convergence or after the maximum number of iterations has been
reached.

Finally, we also compare DAEMA against a classical denoising autoencoder
(DAE). The training procedure of the DAE is similar to the one of DAEMA. The
autoencoder is composed of three fully-connected layers, respectively of size
$2d$, $2d$ and $d$, followed by an hyperbolic tangent activation function for
the first two layers and no activation function for the last one. We evaluate
the model after 39200, 39400, 39600, 39800 and 40000 batch-steps.

###  6.4 Comparison with State-of-the-Art Algorithms

We compare DAEMA with the other algorithms on seven publicly available
datasets both in MCAR and MNAR settings (see Section 5.1 for dataset
preprocessing). The performance of a simple mean imputation Mean and a perfect
reconstruction Real are also reported as indicative lower and upper bounds.

Table 1: State-of-the-art comparison under the MCAR setting. NRMS is reported
in the top table and the performance of a downstream random forest using the
imputed datasets is reported in bottom one (accuracy for classification and
NRMS for regression).

| EEG | Glass | Breast | Ionosphere | Shuttle | Boston | CASP
---|---|---|---|---|---|---|---
DAEMA |  0.392 $\pm$.005 |  0.714 $\pm$.093 | 0.678 $\pm$.041 |  0.745 $\pm$.035 |  0.546 $\pm$.123 |  0.635 $\pm$.047 |  0.422 $\pm$.069
DAE | 0.467 $\pm$.005 | 0.754 $\pm$.092 |  0.645 $\pm$.037 | 0.799 $\pm$.034 | 0.597 $\pm$.111 | 0.645 $\pm$.057 | 0.460 $\pm$.066
AimNet | 0.440 $\pm$.005 | 0.926 $\pm$.108 | 0.681 $\pm$.029 | 0.846 $\pm$.036 | 0.568 $\pm$.123 | 0.704 $\pm$.063 | 0.431 $\pm$.070
MIDA | 0.925 $\pm$.015 | 0.954 $\pm$.108 | 0.797 $\pm$.033 | 0.901 $\pm$.020 | 1.470 $\pm$.401 | 0.845 $\pm$.064 | 0.861 $\pm$.051
MissForest |  0.364 $\pm$.006 | 0.741 $\pm$.076 | 0.677 $\pm$.050 | 0.756 $\pm$.040 | 0.673 $\pm$.182 |  0.601 $\pm$.052 |  0.379 $\pm$.077
Mean | 0.998 $\pm$.013 | 1.003 $\pm$.112 | 0.993 $\pm$.035 | 1.018 $\pm$.023 | 0.969 $\pm$.077 | 0.998 $\pm$.048 | 1.004 $\pm$.030

| Accuracy (higher is better) | NRMS (lower is better)
---|---|---
| EEG | Glass | Breast | Ionosphere | Shuttle | Boston | CASP
DAEMA |  0.861 $\pm$.004 |  0.684 $\pm$.043 |  0.973 $\pm$.011 | 0.932 $\pm$.022 |  0.996 $\pm$.001 |  0.481 $\pm$.047 |  0.708 $\pm$.004
DAE | 0.849 $\pm$.003 | 0.678 $\pm$.034 | 0.971 $\pm$.011 |  0.933 $\pm$.019 | 0.995 $\pm$.001 | 0.492 $\pm$.049 | 0.720 $\pm$.005
AimNet | 0.851 $\pm$.004 | 0.669 $\pm$.043 | 0.970 $\pm$.011 | 0.930 $\pm$.022 | 0.996 $\pm$.001 | 0.506 $\pm$.053 | 0.712 $\pm$.006
MIDA | 0.823 $\pm$.005 | 0.665 $\pm$.026 | 0.973 $\pm$.011 | 0.930 $\pm$.024 | 0.994 $\pm$.000 | 0.505 $\pm$.038 | 0.761 $\pm$.004
MissForest |  0.869 $\pm$.006 | 0.675 $\pm$.041 | 0.971 $\pm$.013 | 0.931 $\pm$.016 |  0.997 $\pm$.000 | 0.492 $\pm$.045 |  0.683 $\pm$.005
Mean | 0.824 $\pm$.005 | 0.669 $\pm$.044 | 0.968 $\pm$.013 | 0.927 $\pm$.024 | 0.996 $\pm$.000 | 0.495 $\pm$.033 | 0.752 $\pm$.004
Real | 0.925 $\pm$.005 | 0.750 $\pm$.040 | 0.973 $\pm$.009 | 0.940 $\pm$.020 | 1.000 $\pm$.000 | 0.372 $\pm$.051 | 0.615 $\pm$.006

Table 2: State-of-the-art comparison under the MNAR setting. NRMS is reported
in the top table and the performance of a downstream random forest using the
imputed datasets is reported in bottom one (accuracy for classification and
NRMS for regression).

| EEG | Glass | Breast | Ionosphere | Shuttle | Boston | CASP
---|---|---|---|---|---|---|---
DAEMA |  0.390 $\pm$.006 | 0.731 $\pm$.123 | 0.703 $\pm$.046 |  0.765 $\pm$.080 |  0.611 $\pm$.253 |  0.634 $\pm$.051 |  0.429 $\pm$.052
DAE | 0.467 $\pm$.006 |  0.725 $\pm$.129 |  0.656 $\pm$.047 | 0.807 $\pm$.078 | 0.683 $\pm$.217 | 0.644 $\pm$.045 | 0.465 $\pm$.044
AimNet | 0.440 $\pm$.008 | 0.822 $\pm$.128 | 0.687 $\pm$.056 | 0.853 $\pm$.072 | 0.642 $\pm$.247 | 0.683 $\pm$.041 | 0.435 $\pm$.053
MIDA | 0.962 $\pm$.023 | 0.897 $\pm$.119 | 0.820 $\pm$.069 | 0.884 $\pm$.060 | 1.481 $\pm$.244 | 0.842 $\pm$.033 | 0.862 $\pm$.032
MissForest |  0.363 $\pm$.006 | 0.786 $\pm$.231 | 0.688 $\pm$.051 | 0.768 $\pm$.085 | 0.653 $\pm$.178 |  0.591 $\pm$.059 |  0.386 $\pm$.058
Mean | 1.041 $\pm$.024 | 0.957 $\pm$.119 | 1.018 $\pm$.041 | 0.994 $\pm$.057 | 1.027 $\pm$.148 | 1.003 $\pm$.046 | 1.013 $\pm$.030

| Accuracy (higher is better) | NRMS (lower is better)
---|---|---
| EEG | Glass | Breast | Ionosphere | Shuttle | Boston | CASP
DAEMA |  0.870 $\pm$.004 |  0.710 $\pm$.037 | 0.968 $\pm$.011 | 0.935 $\pm$.018 |  0.997 $\pm$.001 |  0.469 $\pm$.045 |  0.696 $\pm$.011
DAE | 0.859 $\pm$.006 | 0.693 $\pm$.060 |  0.969 $\pm$.013 |  0.938 $\pm$.016 | 0.996 $\pm$.001 | 0.479 $\pm$.048 | 0.706 $\pm$.013
AimNet | 0.862 $\pm$.006 | 0.681 $\pm$.048 | 0.966 $\pm$.012 | 0.932 $\pm$.019 | 0.997 $\pm$.001 | 0.482 $\pm$.053 | 0.698 $\pm$.012
MIDA | 0.837 $\pm$.009 | 0.668 $\pm$.056 | 0.968 $\pm$.008 | 0.932 $\pm$.020 | 0.995 $\pm$.001 | 0.505 $\pm$.038 | 0.739 $\pm$.018
MissForest |  0.876 $\pm$.005 | 0.695 $\pm$.057 | 0.967 $\pm$.011 | 0.930 $\pm$.021 |  0.997 $\pm$.000 |  0.462 $\pm$.053 |  0.674 $\pm$.009
Mean | 0.837 $\pm$.010 | 0.672 $\pm$.034 | 0.963 $\pm$.012 | 0.932 $\pm$.019 | 0.997 $\pm$.001 | 0.509 $\pm$.053 | 0.732 $\pm$.017
Real | 0.924 $\pm$.003 | 0.748 $\pm$.039 | 0.969 $\pm$.013 | 0.940 $\pm$.017 | 1.000 $\pm$.000 | 0.392 $\pm$.058 | 0.618 $\pm$.004

As shown in Table 1, DAEMA achieves good results in the MCAR setting both in
terms of data reconstruction (top table) and downstream task (bottom table)
compared to DAE, AimNet and MIDA. It obtains the best performance for six of
the seven datasets, sometimes by a huge margin. DAEMA also performs well in
the more challenging MNAR setting as shown in Table 2, which can be explained
by the kind of patterns an attention mechanism can learn.

We also compare DAEMA with MissForest. We can see DAEMA is very competitive
for both missingness settings. The MNAR setting gives a slight advantage to
MissForest though. We hypothesise it is because the local nature of MissForest
makes it less sensitive to the introduced bias than DAEMA, which models the
data distribution on a global scale. However, the scope of MissForest is
limited, as it is meant to reconstruct a fix dataset, while DAEMA can process
new data, making DAEMA more suitable for real-world applications.

###  6.5 Missingness Sensitivity

(a) EEG

(b) Glass

(c) Boston

Figure 2: NRMS reconstruction metric on (2(a)) EEG, (2(b)) Glass and (2(c))
Boston datasets for varying missingness proportions.

Figure 2 shows the sensitivity of the algorithms with respect to the
missingness percentage. Mean imputation provides a lower baseline for
imputation. We can see that AimNet, MissForest and DAEMA achieve significantly
better performance than mean imputation for EEG and Boston datasets. However,
their superiority is limited for the Glass dataset. It can be explained by the
fact that the Glass dataset has only a few samples (214 samples), showing that
AimNet, MissForest and DAEMA need a sufficient amount of data to model the
data distribution. This hypothesis is confirmed by the fact that they also
seem to be more impacted by the amount of missing data.

Compared to MissForest, we can see that DAEMA achieves very competitive
performance. DAEMA seems to be less impacted by the amount of missingness,
giving it an advantage for high missingness rates. Furthermore, as explained
in Section 6.3, the scope of DAEMA is less limited than the one of MissForest
making it more adapted to real-world applications.

##  7 Conclusion

In this work, we propose a novel algorithm, DAEMA, a denoising autoencoder
with an attention mechanism. Unlike state-of-the-art algorithms that learn
from the entire input including placeholder values (mean-imputed or zero-
imputed), DAEMA focuses on observed values thanks to its specifically designed
attention mechanism. We show that, even when applied to a simple denoising
autoencoder without hyperparameter tuning, this new attention mechanism
outperforms state-of-the-art approaches on several instances of missing data
under missing completely at random (MCAR) and missing not at random (MNAR)
settings. These results also propagate nicely to improve the performance on
downstream tasks.

## References

  * [1] Buuren, S.v., Groothuis-Oudshoorn, K.: mice: Multivariate imputation by chained equations in r. Journal of statistical software pp. 1–68 (2010)
  * [2] Camino, R.D., Hammerschmidt, C.A., State, R.: Improving missing data imputation with deep generative models. arXiv preprint arXiv:1902.10666 (2019)
  * [3] Dua, D., Graff, C.: UCI machine learning repository (2017), http://archive.ics.uci.edu/ml
  * [4] Gondara, L., Wang, K.: Mida: Multiple imputation using denoising autoencoders. In: Pacific-Asia Conference on Knowledge Discovery and Data Mining. pp. 260–272. Springer (2018)
  * [5] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., Bengio, Y.: Generative adversarial nets. In: NIPS. pp. 2672–2680 (2014), http://papers.nips.cc/paper/5423-generative-adversarial-nets
  * [6] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: NIPS. pp. 5769–5779 (2017), http://papers.nips.cc/paper/7159-improved-training-of-wasserstein-gans
  * [7] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (Poster) (2015)
  * [8] McCoy, J.T., Kroon, S., Auret, L.: Variational autoencoders for missing data imputation with application to a simulated milling circuit. IFAC-PapersOnLine 51(21), 141–146 (2018)
  * [9] Muzellec, B., Josse, J., Boyer, C., Cuturi, M.: Missing data imputation using optimal transport. In: International Conference on Machine Learning. pp. 7130–7140. PMLR (2020)
  * [10] Nazabal, A., Olmos, P.M., Ghahramani, Z., Valera, I.: Handling incomplete heterogeneous data using vaes. Pattern Recognition 107, 107501 (2020)
  * [11] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011)
  * [12] Richardson, T.W., Wu, W., Lin, L., Xu, B., Bernal, E.A.: Mcflow: Monte carlo flow models for data imputation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14205–14214 (2020)
  * [13] Rubin, D.B.: Inference and missing data. Biometrika 63(3), 581–592 (1976)
  * [14] Seaman, S., Galati, J., Jackson, D., Carlin, J.D.: What is meant by ”missing at random”? Statistical Science 28, 257–268 (2013)
  * [15] Spinelli, I., Scardapane, S., Uncini, A.: Missing data imputation with adversarially-trained graph convolutional networks. Neural Networks 129, 249–260 (2020)
  * [16] Stekhoven, D.J., Bühlmann, P.: Missforest—non-parametric missing value imputation for mixed-type data. Bioinformatics 28(1), 112–118 (2012)
  * [17] Troyanskaya, O., Cantor, M., Sherlock, G., Brown, P., Hastie, T., Tibshirani, R., Botstein, D., Altman, R.B.: Missing value estimation methods for dna microarrays. Bioinformatics 17(6), 520–525 (2001)
  * [18] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NIPS. pp. 6000–6010 (2017), http://papers.nips.cc/paper/7181-attention-is-all-you-need
  * [19] Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th international conference on Machine learning. pp. 1096–1103 (2008)
  * [20] Wang, X., Li, A., Jiang, Z., Feng, H.: Missing value estimation for dna microarray gene expression data by support vector regression imputation and orthogonal coding scheme. BMC bioinformatics 7(1), 1–10 (2006)
  * [21] Wu, R., Zhang, A., Ilyas, I., Rekatsinas, T.: Attention-based learning for missing data imputation in holoclean. Proceedings of Machine Learning and Systems 2, 307–325 (2020)
  * [22] Yoon, J., Jordon, J., Schaar, M.: Gain: Missing data imputation using generative adversarial nets. In: International Conference on Machine Learning. pp. 5689–5698. PMLR (2018)