# A RECURRENT ENCODER-DECODER APPROACH WITH SKIP-FILTERING CONNECTIONS FOR
MONAURAL SINGING VOICE SEPARATION

Stylianos Ioannis Mimilakis∗, Konstantinos Drossos†, Tuomas Virtanen†, Gerald
Schuller∗‡
∗Fraunhofer IDMT, Ilmenau, Germany
{mis<EMAIL_ADDRESS>
†Tampere University of Technology, Tampere, Finland
<EMAIL_ADDRESS>
‡Technical University of Ilmenau, Ilmenau, Germany

## I. Introduction

The under-determined separation of audio and music signals from mixtures is an
active research area in the field of audio signal processing. The main
objective is to estimate individual sources contained in an observed single-
channel (i.e. monaural) mixture. An important task which has attracted a lot
of attention is the estimation of singing voice and background music [1]. The
most widely-used strategy to achieve the estimation of individual sources
employs time-varying filters, usually in the short-time Fourier transform
(STFT) domain. These filters, henceforth denoted as time-frequency masks, are
derived from rational models which incorporate prior information about the
spectral representation of each source in the mixture [2, 3].

When sources are already known (i.e. informed source separation) [4], the
process of source separation is almost trivial by employing an optimal time-
frequency masking strategy [2]. On the other hand, when the sources are not
known, a prior estimation of each individual source takes place. To that aim,
numerous approaches have been proposed. More specifically, in [5] it is
proposed to exploit phase information for separating sources that have
harmonic and impulsive characteristics, while in [6] the repetitive structure
of music sources is studied for separating singing voice and background music.
Alternative methods approximate the mixture magnitude via non-negative low
rank modeling [7, 8], and/or matrix completion via robust principal component
analysis (RPCA) [9].

Studies in music source separation have shown that supervised methods based on
deep learning can yield state-of-the-art results in singing voice separation
[1]. Such methods can be roughly discriminated into two main categories. In
the first category the goal is to train deep neural networks, to approximate
an ideal time-frequency mask from observed magnitude spectral representations
[10, 11, 12]. The second category exploits denoising auto-encoders (i.e.
neural networks trained to map from a noisy observation to a clean one), with
the main goal of estimating the magnitude spectral representation of
individual music sources from input mixtures [13, 14, 15]. Then, these
estimates are combined to derive [13, 15, 16] or optimize [14, 17] a time-
frequency mask that filters the input mixture. However, the quality of the
time-frequency mask is heavily depended on the computation of the sources [14,
16] and the time-frequency mask approximation is not part of the optimization
process (only the estimation of the sources is).

An exception to the latter is the work in [14]. In that work, in order to
predict the time-frequency mask the sources are first predicted and then
combined in a generalized Wiener filtering process. Although, and according to
[14], the approximation of the time-frequency mask is subject to optimization,
this optimization is based on the ability of the previous neural network
layers to estimate the sources.

In this work we propose a method for predicting a time-frequency mask from the
observed mixture magnitude spectrogram, and optimizing it according to its
performance on estimating the target source. During training, the only prior
knowledge is the mixture and the target source magnitude spectrograms. After
training, only the mixture magnitude spectrogram is required. This approach
differs from the existing ones in music source separation because: a) we do
not base the prediction of the time-frequency mask on the prior estimation of
the source(s), and b) we do not require a prior knowledge of the ideal time-
frequency mask for training. We let a recurrent neural network (RNN) to
predict the time-frequency mask. Then, based on that time-frequency mask, we
estimate the magnitude spectrogram of the target source using skip-filtering
connections, a highway network, and a generalized Wiener filtering process.
The RNN and the highway network are jointly trained.

The rest of the paper is organized as follows: Section II presents the
proposed method, followed by Section III which provides information about the
training and evaluation of the proposed method. Section IV presents the
obtained results from the experimental procedure, followed by discussion.
Conclusions are in Section V.

## II. Proposed method

Our method accepts as an input the time-domain samples of a monaural mixture
vector $\mathbf{x}$ and produces the time-domain samples of the $j$-th target
source vector $\mathbf{\hat{y}}^{j}$, as illustrated in Fig. 1. We calculate
the matrix $\mathbf{Y}$, which contains the complex-valued time-frequency
representation of $\mathbf{x}$. In order to use short sequences of frames with
context information (i.e. previous and next frames), we create the tensor
$\mathbf{Y}_{\text{in}}$ consisting of overlapping segments of the magnitude
of $\mathbf{Y}$. We use each matrix in $\mathbf{Y}_{\text{in}}$ as an input to
a single-layered, bi-directional gated recurrent unit (BiGRU), the encoder, in
order to let our method learn temporal inter-dependencies of the input data.
The output of the encoder is used as an input to a single layered GRU, the
decoder, in order to estimate a time-frequency mask. We combine the estimated
time-frequency mask and the input to the encoder using skip-filtering
connections, in order to estimate the magnitude spectrogram of the target
source, $\mathbf{Y}^{j}_{\text{filt}}$. The encoder and the decoder are
optimized by minimizing the generalized Kullback-Leibler divergence,
$\mathcal{L}_{KL}$, between the true and the estimated magnitude spectrogram
of the target source.

Figure 1: Illustration of the proposed encoder-decoder approach with skip-
filter connections.

The $\mathbf{Y}^{j}_{\text{filt}}$ is used as an input to a highway network in
order to reduce the interferences from the rest sources. The encoder, the
decoder, and the highway network are optimized using $\mathcal{L}_{KL}$
between the true magnitude spectrogram of the target source and the output of
the highway network, plus an $L_{2}$ regularization term. The output of the
highway network with the complex-valued mixture representation, $\mathbf{Y}$,
are given as an input to a generalized Wiener filtering process. The latter
produces the complex-valued time-frequency representation of the target source
and further reduces the interferences from the rest sources. The final output,
$\mathbf{\hat{y}}^{j}$, is calculated through an overlap and add synthesis
procedure [18].

### i. Input processing

Vector $\mathbf{x}$ is sliced in $M$ overlapping time frames of length $N$,
using a shift/hop-size of $H$ samples. The overlapping time frames are
element-wise multiplied by the Hamming windowing function. This yields a
matrix of overlapping segments of the original mixture, from which the
complex-valued time-frequency representation $\mathbf{Y}\in\mathbb{C}^{M\times
N}$ is acquired by applying the short-time Fourier (STFT) analysis for all $M$
time segments. From the STFT representation of the mixture $\mathbf{Y}$, we
compute the magnitude spectrum $|\mathbf{Y}|$, with $|\cdot|$ denoting the
element-wise magnitude of the matrix. Using a sliding window over the time-
frames of $|\mathbf{Y}|$ we form the tensor
$\mathbf{Y}_{\text{in}}\in\mathbb{R}_{\geq 0}^{B\times T\times N}$, where
$B=\lceil M/T\rceil$ and $T$ is an integer indicating the amount of STFT time-
frames used for encoding. $\mathbf{Y}_{\text{in}}$ consists of a $B$ number of
matrices that encompass $T\times N$ segments of the mixture magnitude
spectrogram $|\mathbf{Y}|$. Each matrix
$\mathbf{Y}_{\text{in}_{b}}\in\mathbb{R}_{\geq 0}^{T\times N}$ is a sequence
of $T$ overlapping time-frames, and is acquired from $|\mathbf{Y}|$ as

$\displaystyle\mathbf{Y}_{\text{in}_{b}}$
$\displaystyle=[|\mathbf{y}_{m^{\prime}+1}|,\ldots,|\mathbf{y}_{m^{\prime}+T}|]\text{,
where}$ (1) $\displaystyle m^{\prime}$ $\displaystyle=(b-1)\times
T-(b-1)\times 2\times L\text{, }$ (2)

and $L$ and $m^{\prime}$ are integers indicating the amount of time-frames
that will be used as context information, and the time-frame location in
$|\mathbf{Y}|$ ($|\mathbf{y}_{m^{\prime}}|$), respectively. In the case that
$m^{\prime}>M$, $|\mathbf{y}_{m^{\prime}}|$ is a vector of zeros and of length
$N$. The usage of overlapping time-frames is interpreted as having $2\times L$
context frames ($L$ before and $L$ after) for $T-(2\times L)$ length sequence
of time-frames and is necessary for the sub-sampling operation, imposed by the
decoding part of our proposed method. The term $2\times L$ is derived
experimentally.

### ii. Mask and source estimation

Each $\mathbf{Y}_{\text{in}_{b}}$ is used as an input to the BiGRU encoder,
which processes it according to the equations describing a GRU and using the
proposed non-linearities [19]. The BiGRU encoder consists of two GRUs, the
forward GRU which accepts $\mathbf{Y}_{\text{in}_{b}}$ as an input, and the
backward one which accepts
${\overleftarrow{\mathbf{Y}}_{\text{in}_{b}}}=[\mathbf{y}_{\text{in}_{b,T}},\ldots,\mathbf{y}_{\text{in}_{b,t}},\ldots,\mathbf{y}_{\text{in}_{b,1}}]\text{,}$
where $\mathbf{y}_{\text{in}_{b,t}}\in\mathbb{R}_{\geq 0}^{N}$ is a vector in
$\mathbf{Y}_{\text{in}_{b}}$ at time-step $t$, and $\overleftarrow{}$ denotes
the direction of recursion over the time sequences. The hidden outputs from
the bi-directional encoder at time-step $t$, $\mathbf{h}_{t}$ and
$\overleftarrow{\mathbf{h}}_{t}$, are concatenated and updated using residual
connections [20], as

${\mathbf{h}_{\text{enc}_{t}}}=[({\mathbf{h}}_{t}+{{\mathbf{y}}_{\text{in}_{b,t}}})^{\text{T}},({\overleftarrow{\mathbf{h}}_{t}}+{\overleftarrow{\mathbf{y}}_{\text{in}_{b,t}}})^{\text{T}}]^{\text{T}}$
(3)

and the output of the encoder is

$\mathbf{H}_{\text{enc}}=[{\mathbf{h}_{\text{enc}_{1}}},\ldots,{\mathbf{h}_{\text{enc}_{t}}},\ldots,{\mathbf{h}_{\text{enc}_{T}}}]\text{.}$
(4)

$\mathbf{H}_{\text{enc}}$ is used as an input to the decoder, producing an
output denoted as $\mathbf{H}_{\text{dec}}^{j}$. The subscripts “enc” and
“dec” stand for encoder and decoder, respectively. The superscript $j$
signifies that the values of the matrix are dependent on the $j$-th source.
The concatenation imposed by Eq. (3) increases the dimensionality of the
encoded information by a factor of two ($\mathbf{H}_{\text{enc}}$ is of shape
${T\times(N\times 2)}$). For this reason the decoder utilizes dimensionality
reduction, such the $\mathbf{H}_{\text{dec}}^{j}$ matches the dimensionality
of $\mathbf{Y}_{\text{in}_{b}}$. Furthermore, a sub-sampling operation over
the time-steps of $\mathbf{H}_{\text{dec}}^{j}$ is applied, defined as

$\tilde{\mathbf{H}}_{\text{dec}}^{j}=[{\mathbf{h}}^{j}_{\text{dec}_{1+L}},{\mathbf{h}}^{j}_{\text{dec}_{2+L}},\ldots,{\mathbf{h}}^{j}_{\text{dec}_{T-L}}],\vspace{-1mm}$
(5)

where $L$ in Eq. (5) indicates the amount of time steps to be discarded from
the start and end of ${\mathbf{H}}_{\text{dec}}^{j}$. The intuition behind Eq.
(5) is to allow the flow of information from previous and preceding context
frames $L$, without dramatically increasing the learning complexity of longer
time interdependencies [21]. After the sub-sampling operation, we apply the
skip-filtering connections between $\mathbf{\tilde{H}}_{\text{dec}}^{j}$ and a
sub-sampled version of the input data
$\tilde{\mathbf{\mathbf{Y}}}_{\text{in}_{b}}$ as

$\displaystyle\mathbf{Y}_{\text{filt}_{b}}^{j}=$
$\displaystyle\tilde{\mathbf{Y}}_{\text{in}_{b}}\odot|\tilde{\mathbf{H}}_{\text{dec}}^{j}|\text{,
where}$ (6) $\displaystyle\tilde{\mathbf{Y}}_{\text{in}_{b}}=$
$\displaystyle[{\mathbf{y}}_{\text{in}_{b,L}},{}_{\cdots},{\mathbf{y}}_{\text{in}_{b,T-L}}]\text{,}$
(7)

$\odot$ is the Hadamard product, $\tilde{\mathbf{H}}_{\text{dec}}^{j}$ is
interpreted as a time-frequency mask specific to the source $j$, and
$\tilde{\mathbf{Y}}_{\text{in}_{b}}$ is the sub-sampled version of
$\mathbf{Y}_{\text{in}_{b}}$. According to Eq. (6),
$\mathbf{Y}_{\text{filt}_{b}}^{j}$ can be considered as a filtered version of
the input sequence $\tilde{\mathbf{Y}}_{\text{in}_{b}}$ approximating the
magnitude spectrogram of the $j$-th target source, and thus the subscript
“filt”. We minimize the $\mathcal{L}_{KL}$ between the true and estimated
magnitude spectrogram of the $j$-th target source to train the encoder and the
decoder.

The skip-filtering connections directly filter the input mixture magnitude
sequence by the output of the decoder. The motivation behind this is to
enforce $\tilde{\mathbf{H}}_{\text{dec}}^{j}$ to be a time-frequency mask
derived through optimization, based on the prior knowledge of the magnitude
spectrogram of the target source and not on the prior knowledge of an ideal
time-frequency mask. This is inspired by the denoising source separation
framework presented in [22], where the separation of a stationary source in
the frequency domain is achieved by learning a matrix which contains on its
main diagonal the scalar values for each frequency sub-band (i.e. a filter).
The same operation can be realized by the Hadamard product of vectors in the
frequency domain, which can be implemented by the skip-filtering connections
presented in this work and the speech separation approach presented in [23]
which is denoted as signal approximation 111At the time of publication the
authors of this work were not aware of the speech separation method presented
in [23]. To acknowledge the findings of [23] we have addressed the necessary
changes in this manuscript version..

### iii. Enhancement of estimated magnitude spectrogram

In order to enhance the outcome of the filtering process of Eq. (6) [15],
$\mathbf{Y}_{\text{filt}_{b}}^{j}$ is used as an input to a single layer of
highway neural networks [24], with shared weights in time, as

$\begin{split}|\hat{\mathbf{Y}}^{\prime
j}_{b}|=\sigma(\mathbf{Y}^{j}_{\text{filt}_{b}}\,\mathbf{W}^{h}+\mathbf{b}^{h})\odot
g(\mathbf{Y}^{j}_{\text{filt}_{b}}\,\mathbf{W}^{tr}+\mathbf{b}^{tr})+\\\
\mathbf{Y}^{j}_{\text{filt}_{b}}\odot(1-\sigma(\mathbf{Y}^{j}_{\text{filt}_{b}}\,\mathbf{W}^{tr}+\mathbf{b}^{tr}))\text{,}\end{split}$
(8)

where $|\hat{\mathbf{Y}}^{\prime j}_{b}|$ is the enhanced estimated magnitude
spectrogram of the $j$-th target source [15], $\sigma$ is the sigmoid
function, $g$ is defined as $g(x)=\max(0,x)$, and $\mathbf{W}^{tr}$ and
$\mathbf{W}^{h}$ are the weight matrices of the transformation and gating
operations with their corresponding bias vectors $\mathbf{b}^{h}$ and
$\mathbf{b}^{tr}$, respectively. All the layers are trained by minimizing the
$\mathcal{L}_{KL}$ between the true and the enhanced estimated magnitude
spectrogram of the $j$-th target source, plus an additional $L_{2}$ penalty,
for regularizing $|\hat{\mathbf{Y}}^{\prime j}_{b}|$, scaled by
$\lambda=1e^{-4}$. Given that $L_{2}$ penalty is higher for non pseudo-
periodic information [9], we use it in order to enforce the highway network to
enhance the estimated magnitude spectrogram of the singing voice.

### iv. Final output and implementation details

We iterate through all the above presented equations for all $b$ in
$\mathbf{Y}_{\text{in}_{b}}$ and the result is the tensor
$|\hat{\mathbf{Y}}^{\prime j}|\in\mathbb{R}_{\geq 0}^{B\times T^{\prime}\times
N}$, where $T^{\prime}=T-2\times L$. Then, $|\hat{\mathbf{Y}}^{\prime j}|$ is
reshaped to $|\hat{\mathbf{Y}}^{\prime j}_{\text{flat}}|\in\mathbb{R}_{\geq
0}^{M\times N}$, by reversing the process in Eq. (1) and (2). The subscript
“flat” is used to refer to the aforementioned reshaping procedure. The final
output of the proposed method is the complex-valued spectral representation of
the $j$-th source $\hat{\mathbf{Y}}^{j}\in\mathbb{C^{M\times N}}$ computed as

$\displaystyle\hat{\mathbf{Y}}^{j}$
$\displaystyle=\mathbf{M}^{j}\odot\mathbf{Y}\text{, where}$ (9)
$\displaystyle{\mathbf{M}}^{j}$
$\displaystyle=\frac{|{\hat{\mathbf{Y}}^{\prime
j}_{\text{flat}}}|^{\alpha}}{|{\mathbf{Y}}|^{\alpha}}\text{,}$ (10)

$\alpha\in(0,2]$ is an exponent applied element-wise, signifying the
statistical assumption(s) about the sources and their additive property in the
magnitude time-frequency domain [2]. Furthermore, the division is also
performed element-wise. The reasoning behind Eqs. (9)–(10) and the $\alpha$
factor, is to infer the information that the mixture encapsulates about the
phase and the magnitude of the target source, practically improving the
interference reduction from other concurrently active sources. The
reconstruction of the time-domain function of the target source
$\mathbf{\hat{y}}^{j}$ is achieved by the STFT synthesis operation, for all M
time segments, followed by the overlap and add method presented in [18].

The highway network and the GRUs in the encoder consist of 1025 neurons each
and the GRU in the decoder has 2050 neurons. This results in approximately 24
million parameters in total for our method. All weight matrices are randomly
initialized using the method presented in [25] and jointly trained using the
adam algorithm [26], with an initial learning rate of $1e^{-3}$, over $b$
batches of 16, and an $L_{2}$ based gradient norm clipping equal to 0.35. The
training is terminated if after two consecutive iterations over all the
available training batches, no minimization took place. The implementation of
the proposed method is based on the keras [27] and Theano [28] frameworks. The
above parameters are chosen experimentally by informal listening tests and
subjectively evaluating the obtained quality of separation, with data drawn
from the development subset of DSD$100$ (presented in the next section).

## III. Evaluation

### i. Dataset and preprocessing

In order to assess the performance of the proposed method we focus on the task
of singing voice and background music separation. The Demixing Secret Dataset
(DSD$100$)222http://www.sisec17.audiolabs-erlangen.de is used for training and
evaluating our approach. DSD$100$ consists of $100$ professionally produced
multi-tracks of various music genres, sampled at $44.1$kHz. The dataset is by
default divided evenly into development and evaluation subsets, each
consisting of $50$ multi-tracks, and each multi-track contains four stereo
target sources forming the produced mixture.

For each multi-track contained in the development subset, a monaural version
of each of the four sources is generated by averaging the two available
channels. Afterwards, two signals are generated. One containing the
corresponding monaural mixture of all the monaural sources (singing voice,
bass, drums, etc) and a second containing the mixture of all the monaural
sources but the singing voice. The mixing gain values for each source are not
modified (no data augmentation is applied). An analysis operation using the
STFT is applied to the two preceding mixture signals and the target singing
voice, using $H=256$ and $N^{\prime}=2048$. Since we are concerned with real-
valued signals, their time-frequency representation using the DFT is Hermitian
and thus the redundant information is discarded, resulting into a
dimensionality of $N=1025$. The length of the sequences is set to $T=18$,
modeling approximately $120$ ms, and $L=3$. To avoid inconsistencies due to
the sub-sampling operation of Eq. (5), the considered shifts for acquiring
$\mathbf{Y}_{\text{in}}$ are overlapping by six time-frames ($2\times L$).
During training, the true source $|\mathbf{Y}^{j}|$ is the outcome of a
generalized Wiener filtering using the a priori knowledge from the dataset for
each source and a value of $\alpha=1$.

### ii. Metrics and evaluation procedure

To evaluate our method we use the standard metrics employed in the music
source separation evaluation campaign [1]. These are the signal to distortion
ratio (SDR) and the signal to interference ratio (SIR). These metrics are
computed by using the output of the proposed method on the evaluation sub-set
and the true sources acquired from the multi-tracks of the same sub-set.
Results from three different strategies using the proposed method are
reported. For the first one, we train our method using the magnitude
spectrogram of the singing voice $|\mathbf{Y}^{j=1}|$. After the training, we
obtain an estimation of the singing voice using the input mixture spectrogram
and an $\alpha$ value equal to $1.7$, as according to Eq. (9) and (10). For
the rest two strategies, we use two deep neural networks trained separately.
Both of them are based on our method. One to predict the singing voice
$|\mathbf{Y}^{j=1}|$ and another to predict the background music without the
singing voice $|\mathbf{Y}^{j=2}|$. The predicted magnitude spectral estimates
(i.e. $|\mathbf{\hat{Y}}^{j=1}|$ and $|\mathbf{\hat{Y}}^{j=2}|$) are used in
the Eq. (10) and the Eq. (9) to compute the final estimation of the
$|\mathbf{Y}^{j=1}|$. In Eq. (10), the numerator is defined as
$|\mathbf{\hat{Y}}^{j=1}|^{\alpha}$ and we replace the $|\mathbf{Y}|^{\alpha}$
of the denominator with
$|\hat{\mathbf{Y}}|^{\alpha}=|\mathbf{\hat{Y}}^{j=2}|^{\alpha}+|\mathbf{\hat{Y}}^{j=1}|^{\alpha}$.
For the second strategy we use $\alpha=1.7$ and for the third $\alpha=2$. We
will refer to these three strategies as GRU-S (the first), GRU-D (the second),
and GRU-DWF (the third). The values for $\alpha$ are chosen according to the
generalized Wiener filtering (i.e. $\alpha=2$) and its extension to heavy-
tailed distributions [2].

We compare with state-of-the-art methods dealing with monaural singing voice
separation. More specifically, we compare the results from our proposed method
against: i) two approaches based on deep feed-forward neural networks of
approximately $17$ million parameters each, trained using data augmentation to
predict both binary and soft time-frequency masks derived from [3], denoted as
GRA2 and GRA3 [10] ii) a convolutional based encoder-decoder approach with
approximately $27$ million parameters, trained using data augmentation to
predict a soft time-frequency mask, denoted as CHA [12], and iii) two
additional deep denoising auto-encoders operating on the common fate signal
representation [29] denoted as STO1, STO2. As for the oracle estimation, the
results from ideal binary masking (IBM) are also presented. We report results
for the overall performance on the employed dataset and for three clusters of
music genres, namely Jazz/Pop/Rock, Electronic/Rap, and Heavy Metal.

## IV. Results & Discussion

Results from the SDR and SIR metrics for the above methods and the oracle
estimation are illustrated in Figures 2 and 3. Examining the differences in
SDR and SIR metrics between the methods are trained to approximate the ideal
time-frequency masks (GRA3, CHA) and GRU-D, it can be observed that the median
SDR has been improved by approximately $5.4$ dB when compared to GRA3 and by
$2.1$ dB when compared to CHA. An average improvement of $3.5$ dB for the SIR
can be observed regarding the methods that predict of the time-frequency mask
(GRA2, GRA3, CHA). On the other hand, when it comes to comparison with common
denoising auto-encoders (STO1, STO2) and our method, a marginal average loss
of $0.3$ dB in the median SDR and a gain of $0.25$ dB and $0.9$ dB in the
median SIR can be observed according to the results over the GRU-D and GRU-DWF
cases, respectively. We believe (but we don’t have any evidence) that these
marginal differences can be attributed to the more sophisticated signal
representations, such as the common fate model [29], used in STO1, STO2. In
contrast, our method operates on top of a magnitude spectral representation
computed from a STFT.

Figure 2: Analysis of variance of SDR for previous approaches and the proposed
ones. Lines denote the median values. Figure 3: Analysis of variance of SIR
for previous approaches and the proposed ones. Lines denote the median values.

By inspecting the differences between the proposed strategies GRU-S and GRU-D,
it can be seen that SDR can be increased by approximately $0.9$ dB by
incorporating separate deep neural networks for approximating additional
sources contained in mixtures. On the other hand, dramatic differences in SIR
between the three strategies were not observed, unless the value for $\alpha$
was increased, like in the case of GRU-DWF. This shows, that combining
multiple deep neural networks in generalized Wiener filtering leads to
improved SDR. Additionally, the additivity property, acknowledged in
generalized Wiener filtering [2], of the estimates of deep neural networks
might not hold true without an explicit cost objective. Nonetheless, it can
used for improving the interference reduction. In Figure 4 are the average SDR
and SIR measures of the three employed strategies over the three employed
music genres.

Figure 4: Average objective performance of the three presented strategies over
three clusters of music genres.

As it can be seen, the performance over the cluster Jazz/Pop/Rock is higher
than the performance of for the other clusters. An explanation to this is that
the DSD$100$ dataset contains training multi-tracks mainly from pop and rock
music genres. This means that the poorer performance of our method can be
attributed to the fact that recurrent models might need additional data for
modeling more complicated structures of singing voice.

## V. Conclusions & Future Work

In this work we presented a deep learning method for music source separation.
The method used an encoder-decoder configuration based on GRUs and skip-
filtering connections between input spectral representations and their hidden
latent variables, forcing the GRUs to approximate a time-frequency masking
operation. Its application to monaural singing voice separation was studied
and assessed objectively. The obtained results signify that the skip-filtering
connections can be used for approximating time-frequency masks, providing
comparable results to state of the art deep learning approaches. Future work
will focus on psycho-acoustically motivated loss minimization and exploring
sparsity priors for improving the approximated time-frequency mask. Subjective
assessment of the plausible extensions of our methodology are also emerging
[30]. Source code and listening examples can be found under:
https://github.com/Js-Mim/mlsp2017_svsep_skipfilt.

## VI. Acknowledgements

We would like to thank the authors of [1] for making the results of the
evaluation available. The major part of the research leading to these results
has received funding from the European Union’s H2020 Framework Programme
(H2020-MSCA-ITN-2014) under grant agreement no 642685 MacSeNet. Konstantinos
Drossos was partially funded from the European Union’s H2020 Framework
Programme through ERC Grant Agreement 637422 EVERYSOUND. Minor part of the
computations leading to these results were performed on a TITAN-X GPU donated
by NVIDIA.

## References

  * [1] A. Liutkus, F.-R. Stöter, Z. Rafii, D. Kitamura, B. Rivet, N. Ito, N. Ono, and J. Fontecave,  “The 2016 signal separation evaluation campaign,”  in Latent Variable Analysis and Signal Separation: 13th International Conference, LVA/ICA 2017, 2017, pp. 323–332.
  * [2] A. Liutkus and R. Badeau,  “Generalized wiener filtering with fractional power spectrograms,”  in 40th International Conference on Acoustics, Speech and Signal Processing (ICASSP 2015), April 2015, pp. 266–270.
  * [3] H. Erdogan, J. R. Hershey, S. Watanabe, and J. Le Roux,  “Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks,”  in 40th International Conference on Acoustics, Speech and Signal Processing (ICASSP 2015), April 2015, pp. 708–712.
  * [4] G. Puy, A. Ozerov, N. Q. K. Duong, and P. Perez,  “Informed source separation via compressive graph sampling,”  in 42nd International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), Mar 2017, pp. 1–5.
  * [5] E. Cano, M. Plumbley, and C. Dittmar,  “Phase-based harmonic percussive separation,”  in Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), Sept. 2014, pp. 1628–1632.
  * [6] Z. Rafii and B. Pardo,  “Repeating pattern extraction technique (repet): A simple method for music/voice separation,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 21, no. 1, pp. 73–84, Jan 2013.
  * [7] A. Liutkus, D. Fitzgerald, and R. Badeau,  “Cauchy nonnegative matrix factorization,”  in Applications of Signal Processing to Audio and Acoustics (WASPAA 2015), Oct 2015, pp. 1–5.
  * [8] D. Fitzgerald, A. Liutkus, and R. Badeau,  “PROJET - Spatial Audio Separation Using Projections,”  in 41st International Conference on Acoustics, Speech and Signal Processing (ICASSP 2016), 2016, pp. 36–40.
  * [9] I.-Y. Jeong and K. Lee,  “Singing voice separation using RPCA with weighted $l_{1}$-norm,”  in Latent Variable Analysis and Signal Separation: 13th International Conference, LVA/ICA 2017, 2017, pp. 553–562.
  * [10] E.-M. Grais, G. Roma, A.J.R. Simpson, and M.-D. Plumbley,  “Single-channel audio source separation using deep neural network ensembles,”  in Audio Engineering Society Convention 140, May 2016.
  * [11] E.-M. Grais, G. Roma, A.J.R. Simpson, and M.-D. Plumbley,  “Combining mask estimates for single channel audio source separation using deep neural networks,”  in Proceedings of the 17th Annual Conference of the International Speech Communication Association (Interspeech), Sept. 8-12 2016, pp. 3339–3343.
  * [12] P. Chandna, M. Miron, J. Janer, and E. Gómez,  “Monoaural audio source separation using deep convolutional neural networks,”  in Latent Variable Analysis and Signal Separation: 13th International Conference, LVA/ICA 2017, 2017, pp. 258–266.
  * [13] S. Uhlich, F. Giron, and Y. Mitsufuji,  “Deep neural network based instrument extraction from music,”  in 40th International Conference on Acoustics, Speech and Signal Processing (ICASSP 2015), 2015, pp. 2135–2139.
  * [14] P.-S. Huang, M. Kim, M. Hasegawa-Johnson, and P. Smaragdis,  “Joint optimization of masks and deep recurrent neural networks for monaural source separation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 12, pp. 2136–2147, Dec 2015.
  * [15] S.-I. Mimilakis, E. Cano, J. Abeßer, and G. Schuller,  “New sonorities for jazz recordings: Separation and mixing using deep neural networks,”  in Audio Engineering Society 2nd Workshop on Intelligent Music Production, 2016.
  * [16] S. Uhlich, M. Porcu, F. Giron, M. Enenkl, T. Kemp, N. Takahashi, and Y. Mitsufuji,  “Improving music source separation based on deep neural networks through data augmentation and network blending,”  in 42nd International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), 2017, pp. 261–265.
  * [17] A.-A. Nugraha, A. Liutkus, and E. Vincent,  “Multichannel music separation with deep neural networks,”  in 24th European Signal Processing Conference (EUSIPCO), Aug 2016, pp. 1748–1752.
  * [18] D. Griffin and J. Lim,  “Signal estimation from modified short-time Fourier transform,”  IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236–243, Apr 1984.
  * [19] D. Bahdanau, K. Cho, and Y. Bengio,  “Neural machine translation by jointly learning to align and translate,”  in International Conference on Learning Representations (ICLR), 2015\.
  * [20] Y. Wu et al,  “Google’s neural machine translation system: Bridging the gap between human and machine translation,”  CoRR, vol. abs/1609.08144, 2016.
  * [21] A. Graves,  Hierarchical Subsampling Networks, pp. 109–131,  Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.
  * [22] J. Särelä and H. Valpola,  “Denoising source separation,”  J. Mach. Learn. Res., vol. 6, pp. 233–272, Dec. 2005.
  * [23] F. Weninger, J. R. Hershey and J. Le Roux and B. Schuller  “Discriminatively trained recurrent neural networks for single-channel speech separation,”  in Proceedings of the IEEE Global Conference on Signal and Information Processing (GlobalSIP 2014), 2014, pp. 577–581.
  * [24] R.-K. Srivastava, K. Greff, and J. Schmidhuber,  “Highway networks,”  CoRR, vol. abs/1505.00387, 2015.
  * [25] X. Glorot and Y. Bengio,  “Understanding the difficulty of training deep feedforward neural networks,”  in In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS’10), 2010, pp. 249–256.
  * [26] D.-P. Kingma and J. Ba,  “Adam: A method for stochastic optimization,”  CoRR, vol. abs/1412.6980, 2014.
  * [27] F. Chollet et al.,  “Keras (version 1.2.2),” https://github.com/fchollet/keras, 2015\.
  * [28] Theano Development Team,  “Theano: A python framework for fast computation of mathematical expressions (version 0.9),”  CoRR, vol. abs/1605.02688, 2016.
  * [29] F.-R. Stöter, A. Liutkus, R. Badeau, B. Edler, and P. Magron,  “Common fate model for unison source separation,”  in International Conference on Acoustics, Speech and Signal Processing (ICASSP 2016), 2016, pp. 126–130.
  * [30] E. Cano, D. Fitzgerald, and K. Brandenburg,  “Evaluation of quality of sound source separation algorithms: Human perception vs quantitative metrics,”  in 24th European Signal Processing Conference (EUSIPCO), Aug 2016, pp. 1758–1762.


11institutetext: Microsoft, Redmond WA, USA
11email<EMAIL_ADDRESS>
# Aligning benchmark datasets for table structure recognition

Brandon Smock 11 Rohith Pesala 11 Robin Abraham 11

###### Abstract

Benchmark datasets for table structure recognition (TSR) must be carefully
processed to ensure they are annotated consistently. However, even if a
dataset’s annotations are self-consistent, there may be significant
inconsistency across datasets, which can harm the performance of models
trained and evaluated on them. In this work, we show that _aligning_ these
benchmarks—removing both errors and inconsistency between them—improves model
performance significantly. We demonstrate this through a data-centric approach
where we adopt a single model architecture, the Table Transformer (TATR), that
we hold fixed throughout. Baseline exact match accuracy for TATR evaluated on
the ICDAR-2013 benchmark is 65% when trained on PubTables-1M, 42% when trained
on FinTabNet, and 69% combined. After reducing annotation mistakes and inter-
dataset inconsistency, performance of TATR evaluated on ICDAR-2013 increases
substantially to 75% when trained on PubTables-1M, 65% when trained on
FinTabNet, and 81% combined. We show through ablations over the modification
steps that canonicalization of the table annotations has a significantly
positive effect on performance, while other choices balance necessary trade-
offs that arise when deciding a benchmark dataset’s final composition. Overall
we believe our work has significant implications for benchmark design for TSR
and potentially other tasks as well. All dataset processing and training code
will be released.

##  1 Introduction

Table extraction (TE) is a long-standing problem in document intelligence.
Over the last decade, steady progress has been made formalizing TE as a
machine learning (ML) task. This includes the development of task-specific
metrics for evaluating table structure recognition (TSR) models [5, 26, 20] as
well as the increasing variety of datasets and benchmarks [3, 25, 26, 21].
These developments have enabled significant advances in deep learning (DL)
modeling for TE [18, 16, 25, 21, 13, 12].

In general, benchmarks play a significant role in shaping the direction of ML
research [17, 10]. Recently it has been shown that benchmark datasets for TSR
contain a significant number of errors and inconsistencies [21]. It is well-
documented that errors in a benchmark have negative consequences for both
learning and evaluation [6, 27, 4]. Errors in the test set for the ImageNet
benchmark [14], for instance, have led to top-1 accuracy saturating around 91%
[24]. Errors can also lead to false conclusions during model selection,
particularly when the training data is drawn from the same noisy distribution
as the test set [14].

Compared to annotation mistakes, inconsistencies in a dataset can be more
subtle because they happen across a collection of samples rather than in
isolated examples—but no less harmful. Even if a single dataset is self-
consistent there may be inconsistencies in labeling across different datasets
for the same task. We consider datasets for the same task that are annotated
inconsistently with respect to each other to be _misaligned_. Misalignment can
be considered an additional source of labeling noise. This noise may go
unnoticed when a dataset is studied in isolation, but it can have significant
effects on model performance when datasets are combined.

In this work we study the effect that errors and misalignment between
benchmark datasets have on model performance for TSR. We select two large-
scale crowd-sourced datasets for training—FinTabNet and PubTables-1M—and one
small expert-labeled dataset for evaluation—the ICDAR-2013 benchmark. For our
models we choose a single fixed architecture, the recently proposed Table
Transformer (TATR) [21]. This can be seen as a data-centric [22] approach to
ML, where we hold the modeling approach fixed and instead seek to improve
performance through data improvements. Among our main contributions:

(a) Original annotation

(b) Corrected annotation

Figure 1: An example table from us-gov-dataset/us-012-str.xml in the
ICDAR-2013 dataset. On the top are the column bounding boxes created when
using the dataset’s original annotations. On the bottom are the column
bounding boxes following correction of the cells’ column index labels.
Corresponding columns before and after correction are shaded with the same
color.

  * •

We remove both annotation mistakes and inconsistencies between FinTabNet and
ICDAR-2013, aligning these datasets with PubTables-1M and producing improved
versions of two standard benchmark datasets for TSR.

  * •

We show that removing inconsistencies between benchmark datasets for TSR
indeed has a substantial positive impact on model performance, improving
baseline models trained separately on PubTables-1M and FinTabNet from 65% to
78% and 42% to 65%, respectively, when evaluated on ICDAR-2013.

  * •

We perform a sequence of ablations over the steps of the correction procedure,
which shows that canonicalization has a clear positive effect on model
performance, while other factors create trade-offs when deciding the final
composition of a benchmark.

  * •

We train a single model on both PubTables-1M and the aligned version of
FinTabNet, establishing a new baseline DAR of 0.965 and exact match table
recognition accuracy of 81% on the corrected ICDAR-2013 benchmark.

  * •

We will release all of the dataset processing and training code.

##  2 Related Work

The first standard multi-domain benchmark dataset for table structure
recognition was the ICDAR-2013 dataset, introduced at the 2013 ICDAR Table
Competition [5]. In total it contains 158 tables111The dataset is originally
documented as having 156 tables but we found 2 cases of tables that need to be
split into multiple tables to be annotated consistently. in the official
competition dataset and at least 100 tables222This is the number of tables in
the practice set we were able to find online. in a practice dataset. The
competition also established the first widely-used metric for evaluation, the
directed adjacency relations (DAR) metric, though alternative metrics such as
TEDS [26] and GriTS [20] have been proposed more recently. At the time of the
competition, the best reported approach achieved a DAR of 0.946. More recent
work from Tensmeyer et al. [23] reports a DAR of 0.953 on the full competition
dataset.

The difficulty of the ICDAR-2013 benchmark and historical lack of training
data for the TSR task has led to a variety of approaches and evaluation
procedures that differ from those established in the original competition.
DeepDeSRT [18] split the competition dataset into a training set and a test
set of just 34 samples. Additionally, the authors processed the annotations to
add row and column bounding boxes and then framed the table structure
recognition problem as the task of detecting these boxes. This was a
significant step at the time but because spanning cells are ignored it is only
a partial solution to the full recognition problem. Still, many works have
followed this approach [7]. Hashmi et al. [8] recently reported an F1 score of
0.9546 on the row and column detection task, compared to the original score of
0.9144 achieved by DeepDeSRT.

While ICDAR-2013 has been a standard benchmark of progress over the last
decade, little attention has been given to complete solutions to the TSR task
evaluated on the full ICDAR-2013 dataset. Of these, none reports exact match
recognition accuracy. Further, we are not aware of any work that points out
label noise in the ICDAR-2013 dataset and attempts to correct it. Therefore it
is an open question to what extent performance via model improvement has
saturated on this benchmark in its current form.

To address the need for training data, several large-scale crowd-sourced
datasets [3, 26, 11, 25, 21] have been released for training table structure
recognition models. However, the quality of these datasets is lower overall
than that of expert-labeled datasets like ICDAR-2013. Of particular concern is
the potential for ambiguity in table structures [9, 19, 1, 15]. A specific
form of label error called oversegmentation [21] is widely prevalent in
benchmark TSR datasets, which ultimately leads to annotation inconsistency.
However, models developed and evaluated on these datasets typically treat them
as if they are annotated unambiguously, with only one possible correct
interpretation.

To address this, Smock et al. [21] proposed a canonicalization algorithm,
which can automatically correct and make table annotations more consistent.
However, this was applied to only a single source of data. The impact of
errors and inconsistency across multiple benchmark datasets remains an open
question. In fact, the full extent of this issue can be masked by using
metrics that measure the average correctness of cells [20]. But for table
extraction systems in real-world settings, individual errors may not be
tolerable and therefore the accuracy both of cells and of entire tables is
important to consider.

##  3 Data

Figure 2: An example table from HBI/2013/page_39.pdf in the FinTabNet dataset
with cell bounding boxes (shaded in orange) that are inconsistently annotated.
Some cells in the first column include the dot leaders in the bounding box
while others do not.

As our baseline training datasets we adopt PubTables-1M [21] and FinTabNet
[25]. PubTables-1M contains nearly one million tables from scientific
articles, while FinTabNet contains close to 113k tables from financial
reports. As our baseline evaluation dataset we adopt the ICDAR-2013 [5]
dataset. The ICDAR-2013 dataset contains tables from multiple document domains
manually annotated by experts. While its size limits its usefulness for
training, the quality of ICDAR-2013 and its diverse set of table appearances
and layouts make it useful for model benchmarking. For the ICDAR-2013 dataset,
we use the competition dataset as the test set, use the practice dataset as a
validation set, and consider each table "region" annotated in the dataset to
be a table, yielding a total of 256 tables to start.

Table 1: The high-level data processing steps and ablations for the FinTabNet and ICDAR-2013 datasets. Data cleaning steps are grouped into stages. Each stage represents an addition to the steps of the full processing algorithm. For each stage, a dataset ablation is created with the steps up to and including that stage. But the stages themselves do not occur in the same exact order in the full processing pipeline. For instance, when stage a5 is added to the processing it actually occurs at the beginning of stage a4 to annotate more column headers prior to canonicalization. Dataset |  Stage |  Description
---|---|---
FinTabNet |  |
— |  Unprocessed |  The original data, which does not have header or row/column bounding box annotations.
a1 |  Completion |  Baseline FinTabNet dataset; create bounding boxes for all rows and columns; remove tables for which these boxes cannot be defined.
a2 |  Cell box adjustment |  Iteratively refine the row and column boxes to tightly surround their coinciding text; create _grid cells_ at the intersection of each row and column; remove any table for which 50% of the area of a word overlaps with multiple cells in the table.
|
|
|
a3 |  Consistency adjustments |  Make the sizing and spanning of columns and rows more consistent; adjust cell bounding boxes to always ignore dot leaders (remove any tables for which this was unsuccessful); remove empty rows and empty columns from the annotations; merge any adjacent header rows whose cells all span the same columns; remove tables with columns containing just cent and dollar signs (we found these difficult to automatically correct).
|
|
|
|
|  |
|  |
a4 |  Canonicalization |  Use cell structure to infer the column header and projected row headers for each table; remove tables with only two columns as the column header of these is structurally ambiguous; canonicalize each annotation.
|
|
|
a5 |  Additional column header inference |  Infer the column header for many tables with only two columns by using the cell text in the first row to determine if the first row is a column header.
|
|
a6 |  Quality control |  Remove tables with erroneous annotations, including: tables where words in the table coincide with either zero or more than one cell bounding box, tables with a projected row header at the top or bottom (indicating a caption or footer is mistakenly included in the table), and tables that appear to have only a header and no body.
ICDAR-2013 |  |
— |  Unprocessed |  The original data, which does not have header or row/column bounding box annotations.
a1 |  Completion |  Baseline ICDAR-2013 dataset; create bounding boxes for all rows and columns; remove tables for which these boxes cannot be defined or whose annotations cannot be processed due to errors.
a2 |  Manual correction |  Manually fix annotation errors (18 tables fixed in total).
a3 |  Consistency adjustments and canonicalization |  Apply the same automated steps applied to FinTabNet.a2 through FinTabNet.a4; after processing, manually inspect each table and make any additional corrections as needed.
|
|

###  3.1 Missing Annotations

While the annotations in FinTabNet and ICDAR-2013 are sufficient to train and
evaluate models for table structure recognition, several kinds of labels that
could be of additional use are not explicitly annotated. Both datasets
annotate bounding boxes for each cell as the smallest rectangle fully-
enclosing all of the text of the cell. However, neither dataset annotates
bounding boxes for rows, columns, or blank cells. Similarly, neither dataset
explicitly annotates which cells belong to the row and column headers.

For each of these datasets, one of the first steps we take before correcting
their labels is to automatically add labels that are missing. This can be
viewed as making explicit the labels that are _implicit_ given the rest of the
label information present. We cover these steps in more detail later in this
section.

Table 2: Diversity and complexity of table instances in the baseline and ablation datasets. Dataset |  # Tables‡ |  # Unique Topologies |  Avg. Tables / Topology |  Avg. Rows / Table |  Avg. Cols. / Table |  Avg. Spanning Cells / Table
---|---|---|---|---|---|---
FinTabNet |  112,875 |  9,627 |  11.72 |  11.94 |  4.36 |  1.01
FinTabNet.a1 |  112,474 |  9,387 |  11.98 |  11.92 |  4.35 |  1.00
FinTabNet.a2 |  109,367 |  8,789 |  12.44 |  11.87 |  4.33 |  0.98
FinTabNet.a3 |  103,737 |  7,647 |  13.57 |  11.81 |  4.28 |  0.93
FinTabNet.a4 |  89,825 |  18,480 |  4.86 |  11.97 |  4.61 |  2.79
FinTabNet.a5 |  98,019 |  18,752 |  5.23 |  11.78 |  4.39 |  2.57
FinTabNet.a6 |  97,475 |  18,702 |  5.21 |  11.81 |  4.39 |  2.58
ICDAR-2013 |  256 |  181 |  1.41 |  15.88 |  5.57 |  1.36
ICDAR-2013.a1 |  247 |  175 |  1.41 |  15.61 |  5.39 |  1.34
ICDAR-2013.a2 |  258 |  181 |  1.43 |  15.55 |  5.45 |  1.42
ICDAR-2013.a3 |  258 |  184 |  1.40 |  15.52 |  5.45 |  2.08
‡The number of tables in the dataset that we were able to successfully read
and process.

###  3.2 Label Errors and Inconsistencies

Next we investigate both of these datasets to look for possible annotation
mistakes and inconsistencies. For FinTabNet, due to the impracticality of
verifying each annotation manually, we initially sample the dataset to
identify types of errors common to many instances. For ICDAR-2013, we manually
inspect all 256 table annotations.

For both datasets, we note that the previous action of defining and adding
bounding boxes for rows and columns was crucial to catching inconsistencies,
as it exposed a number of errors when we visualized these boxes on top of the
tables. For ICDAR-2013 we noticed the following errors during manual
inspection: 14 tables with at least one cell with incorrect column or row
indices (see Figure 1), 4 tables with at least one cell with an incorrect
bounding box, 1 table with a cell with incorrect text content, and 2 tables
that needed to be split into more than one table.

For FinTabNet, we noticed a few errors that appear to be common in crowd-
sourced table annotations in the financial domain. For example, we noticed
many of these tables use _dot leaders_ for spacing and alignment purposes.
However, we noticed a significant amount of inconsistency in whether dot
leaders are included or excluded from a cell’s bounding box (see Figure 2). We
also noticed that it is common in these tables for a logical column that
contains a monetary symbol, such as a $, to be split into two columns,
possibly for numerical alignment purposes (for an example, see
NEE/2006/page_49.pdf). Tables annotated this way are done so for presentation
purposes but are inconsistent with their logical interpretation, which should
group the dollar sign and numerical value into the same cell. Finally we
noticed annotations with fully empty rows. Empty rows are problematic because
they are used for spacing and serve no logical purpose. Even if spacing were
considered a valid use of an empty row, it is not clear when there would be
enough spacing to require an empty row or if more than one empty row should be
used when there is a large amount of space between rows. As a result, empty
rows in these tables should be considered a labeling error. For both datasets
we noticed oversegmentation of header cells, as is common in crowd-sourced
markup tables.

###  3.3 Dataset Corrections and Alignment

Mistakes noticed during manual inspection of ICDAR-2013 are corrected
directly. For both datasets, nearly all of the other corrections are done by a
series of automated processing steps. We list these steps at a high level in
Table 1. For each dataset the correction steps are grouped into a series of
macro-steps, which helps us to study some of their effects in detail using
ablation experiments in Section 4.

As part of the processing we adopt the canonicalization procedure [21] used to
create PubTables-1M. This helps to align all three datasets and minimize the
amount of inconsistencies between them. We make small improvements to the
canonicalization procedure that help to generalize it to tables in domains
other than scientific articles. These include small improvements to the step
that determines the column header and portions of the row header. Tables with
inconsistencies that are determined to not be easily or reliably corrected
using the previous steps are filtered out.

When creating a benchmark dataset there are many objectives to consider such
as maximizing the number of samples, the diversity of the samples, the
accuracy and consistency of the labels, the richness of the labels, and the
alignment between the labels and the desired use of the learned model for a
task. When cleaning a pre-existing dataset these goals potentially compete and
trade-offs must be made. There is also the added constraint in this case that
whatever steps were used to create the pre-existing dataset prior to cleaning
are unavailable to us and unchangeable. This prevents us from optimizing the
entire set of processing steps holistically. Overall we balance the competing
objectives under this constraint by correcting and adding to the labels as
much as possible when we believe this can be done reliably and filtering out
potentially low-quality samples in cases where we do not believe we can
reliably amend them. We document the effects that the processing steps have on
the size, diversity, and complexity of the samples in Table 2.

Table 3: Comparison of the original baseline model for PubTables-1M and the one we use in the current work with additional image cropping during training and a slightly longer training schedule. Training Data |  Baseline Version |  Epoch‡ |  Test Images |  $\textbf{GriTS}_{\textbf{Con}}$ |  $\textbf{GriTS}_{\textbf{Loc}}$ |  $\textbf{GriTS}_{\textbf{Top}}$ |  $\textbf{Acc}_{\textbf{Con}}$
---|---|---|---|---|---|---|---
PubTables-1M |  Original |  23.72 |  Padded |  0.9846 |  0.9781 |  0.9845 |  0.8138
|  Current |  29 |  Tight |  0.9855 |  0.9797 |  0.9851 |  0.8326
‡In the current work an epoch is standardized across datasets to equal 720,000
training samples.

##  4 Experiments

For our experiments, we adopt a data-centric approach using the Table
Transformer (TATR) [21]. TATR frames TSR as object detection using six classes
and is implemented with DETR [2]. We hold the TATR model architecture fixed
and make changes only to the data. TATR is free of any TSR-specific
engineering or inductive biases, which importantly forces the model to learn
to solve the TSR task from its training data alone.

We make a few slight changes to the original approach used to train TATR on
PubTables-1M in order to establish a stronger baseline and standardize the
approach across datasets. First we re-define an epoch as 720,000 training
samples, which corresponds to 23.72 epochs in the original paper, and extend
the training to 30 epochs given the new definition. Second, we increase the
amount of cropping augmentation during training. All of the table images in
the PubTables-1M dataset contain additional padding around the table from the
page that each table is cropped from. This extra padding enable models to be
trained with cropping augmentation without removing parts of the image
belonging to the table. However, the original paper evaluated models on the
padded images rather than tightly-cropped table images. In this paper, we
instead evaluate all models on more tightly-cropped table images, leaving only
2 pixels of padding around the table boundary in each table image’s original
size. The use of tight cropping better reflects how TSR models are expected to
be used in real-world settings when paired with an accurate table detector.

ICDAR-2013.a1 ICDAR-2013.a2 ICDAR-2013.a3
$0.62$$0.66$$0.7$$0.74$$0.78$$\text{Acc}_{\text{Con}}$ Figure 3: The
performance of TATR trained on PubTables-1M and evaluated on the three
versions of ICDAR-2013. Performance improves significantly after removing
errors and inconsistencies between the two datasets. FinTabNet.a1 FinTabNet.a2
FinTabNet.a3 FinTabNet.a4 FinTabNet.a5 FinTabNet.a6
$0.76$$0.78$$0.8$$0.8$$0.82$$0.84$$\text{Acc}_{\text{Con}}$ Figure 4: The
performance of TATR trained and evaluated on each version of FinTabNet. As can
be seen, performance of FinTabNet evaluated on itself generally improves as a
result of the modification steps. Performance is highest for FinTabNet.a5
rather than for the final version of FinTabNet. However, FinTabNet.a6 ends up
being better aligned with other benchmark datasets, highlighting the potential
trade-offs that must be considering when designing a benchmark dataset.

$2$$4$$6$$8$$10$$12$$14$$16$$18$$20$$22$$24$$26$$28$$30$$0.95$$0.95$$0.96$$0.96$$0.97$$0.97$$0.98$$0.98$$0.99$Epoch$\text{GriTS}_{\text{Con}}$ICDAR-2013.a3
Model
SelectionFinTabNet.a1FinTabNet.a2FinTabNet.a3FinTabNet.a4FinTabNet.a5FinTabNet.a6PubTables-1MPT1M
+ FTN.a1PT1M + FTN.a6

Figure 5: The performance of each model after every epoch of training
evaluated on its validation set. The training epoch with the best performance
for each model is circled. PT1M = PubTables-1M, FTN = FinTabNet. FinTabNet.a1
FinTabNet.a2 FinTabNet.a3 FinTabNet.a4 FinTabNet.a5 FinTabNet.a6 PubTables-1M
PubTables-1M + FinTabNet.a1 PubTables-1M + FinTabNet.a6
$0.39$$0.42$$0.45$$0.48$$0.51$$0.54$$0.57$$0.6$$0.63$$0.66$$0.69$$0.72$$0.75$$0.78$$0.81$$\text{Acc}_{\text{Con}}$ICDAR2013.a1ICDAR2013.a2ICDAR2013.a3
Figure 6: The performance of each model on each ICDAR-2013 dataset ablation.
These results show the impact of improvements to both FinTabNet and
ICDAR-2013, as well as the impact that combining data from multiple domains
has on model performance.

For evaluation, we use the recently proposed grid table similarity (GriTS)
[20] metrics as well as table content exact match accuracy
($\textrm{Acc}_{\textrm{Con}}$). GriTS compares predicted tables and ground
truth directly in matrix form and can be interpreted as an F1 score over the
correctness of predicted cells. Exact match accuracy considers the percentage
of tables for which all cells, including blank cells, are matched exactly. The
TATR model requires words and their bounding boxes to be extracted separately
and uses maximum overlap between words and predicted cells to slot the words
into their cells. For evaluation, we assume that the true bounding boxes for
words are given to the model and used to determine the final output.

In Table 3 we evaluate the performance of the modified training procedure to
the original procedure for PubTables-1M. We use a validation set to select the
best model, which for the new training procedure selects the model after 29
epochs. Using the modified training procedure improves performance over the
original, which establishes new baseline performance metrics.

In the rest of our experiments we train nine TATR models in total using nine
different training datasets. We train one model for each modified version of
the FinTabNet dataset, a baseline model on PubTables-1M, and two additional
models: one model for PubTables-1M combined with FinTabNet.a1 and one for
PubTables-1M combined with FinTabNet.a6. Each model is trained for 30 epochs,
where an epoch is defined as 720,000 samples. We evaluate each model’s
checkpoint after every epoch on a validation set from the same distribution as
its training data as well as a separate validation set from ICDAR-2013. We
average the values of $\text{GriTS}_{\text{Con}}$ for the two validation sets
and select the saved checkpoint yielding the highest score.

###  4.1 FinTabNet self-evaluation

In Table 4, we present the results of each FinTabNet model evaluated on its
own test set. As can be seen, the complete set of dataset processing steps
leads to an increase in $\textrm{Acc}_{\textrm{Con}}$ from 0.787 to 0.812.
Models trained on FinTabNet.a5 and FinTabNet.a6 both have significantly higher
$\textrm{Acc}_{\textrm{Con}}$ than models trained on FinTabNet.a1 through
FinTabNet.a3, despite the tables in the former datasets being much more
complex overall than the latter according to Table 2. This strongly indicates
that the results are driven primarily by increasing the consistency and
cleanliness of the data, and not by reducing their inherent complexity.

Something else to note is that while $\text{GriTS}_{\text{Loc}}$,
$\text{GriTS}_{\text{Con}}$, and $\text{Acc}_{\text{Con}}$ all increase as a
result of the improvements to the data, $\text{GriTS}_{\text{Top}}$ is little
changed. $\text{GriTS}_{\text{Top}}$ measures only how well the model infers
the table’s cell layout alone, without considering how well the model locates
the cells or extracts their text content. This is strong evidence that the
improvement in performance of the models trained on the modified data is
driven primarily by there being more consistency in the annotations rather
than the examples becoming less challenging. However, this evidence is even
stronger in the next section, when we evaluate the FinTabNet models on the
modified ICDAR-2013 datasets instead.

###  4.2 ICDAR-2013 evaluation

In the next set of results, we evaluate all nine models on all three versions
of the ICDAR-2013 dataset. These results are intended to show the combined
effects that improvements to both the training data and evaluation data have
on measured performance. Detailed results are given in Table 5 and a
visualization of the results for just $\text{Acc}_{\text{Con}}$ is given in
Figure 6.

Note that all of the improvements to the ICDAR-2013 annotations are verified
manually and no tables are removed from the dataset. So observed improvements
to performance from ICDAR-2013.a1 to ICDAR-2013.a3 can only be due to cleaner
and more consistent evaluation data. This improvement can be observed clearly
for each trained model from FinTabNet.a4 onward. Improvements to ICDAR-2013
alone are responsible for measured performance for the model trained on
PubTables-1M and FinTabNet.a6 combined increasing significantly, from 68% to
81% $\text{Acc}_{Con}$.

Evaluating improvements to FinTabNet on ICDAR-2013.a3, we also observe a
substantial increase in performance, from 41.1% to 64.6%
$\text{Acc}_{\text{Con}}$. This indicates that not only do improvements to
FinTabNet improve its own self-consistency, but also significantly improve its
performance on challenging real-world test data. The improvement in
performance on test data from both FinTabNet and ICDAR-2013 clearly indicates
that there is a significant increase in the consistency of the data across
both benchmark datasets.

These results also hold for FinTabNet combined with PubTables-1M. The final
model trained with PubTables-1M and FinTabNet.a6 combined outperforms all
other baselines, achieving a final $\text{Acc}_{\text{Con}}$ of 81%. Few prior
works report this metric on ICDAR-2013 or any benchmark dataset for TSR. As we
discuss previously, this is likely due to the fact that measured accuracy
depends not only on model performance but the quality of the ground truth
annotation. These results simultaneously establish new performance baselines
and indicate that the evaluation data is clean enough for this metric to be a
useful measure of model performance. Note also that we took no steps to
balance and optimize how the FinTabNet and PubTables-1M datasets were
combined. This suggests there is unrealized potential for more gains in
performance in the future even within the current modeling approach using
these newly-improved datasets.

$2$$4$$6$$8$$10$$12$$14$$16$$18$$20$$22$$24$$26$$28$$30$$0.3$$0.4$$0.5$$0.6$$0.7$$0.8$Epoch$\text{Acc}_{\text{Con}}$ICDAR-2013.a3
evaluationFinTabNet.a1FinTabNet.a2FinTabNet.a3FinTabNet.a4FinTabNet.a5FinTabNet.a6PubTables-1MPT1M
+ FTN.a1PT1M + FTN.a6

Figure 7: The performance of each model after every epoch of training evaluated on ICDAR-2013.a3. The training epoch with the best performance for each model on its own validation set is circled. PT1M = PubTables-1M, FTN = FinTabNet. Table 4: Object detection and TSR performance metrics for models trained and evaluated on each FinTabNet ablation’s training and test sets. For model selection, we train each model for 30 epochs, evaluate the model on its validation set after every epoch, and choose the model with the highest $\text{Acc}_{\text{Con}}$ on the validation set. Training Data | Epoch | AP | $\textbf{AP}_{\textbf{50}}$ | $\textbf{AP}_{\textbf{75}}$ | AR | $\textbf{GriTS}_{\textbf{Con}}$ | $\textbf{GriTS}_{\textbf{Loc}}$ | $\textbf{GriTS}_{\textbf{Top}}$ | $\textbf{Acc}_{\textbf{Con}}$
---|---|---|---|---|---|---|---|---|---
FinTabNet.a1 | 27 | 0.867 | 0.972 | 0.932 | 0.910 | 0.9796 | 0.9701 | 0.9874 | 0.787
FinTabNet.a2 | 22 | 0.876 | 0.974 | 0.941 | 0.916 | 0.9800 | 0.9709 | 0.9874 | 0.768
FinTabNet.a3 | 14 | 0.871 | 0.975 | 0.942 | 0.910 | 0.9845 | 0.9764 | 0.9893 | 0.799
FinTabNet.a4 | 30 | 0.874 | 0.974 | 0.946 | 0.919 | 0.9841 | 0.9772 | 0.9884 | 0.793
FinTabNet.a5 | 24 | 0.872 | 0.977 | 0.950 | 0.917 | 0.9861 | 0.9795 | 0.9897 | 0.828
FinTabNet.a6 | 26 | 0.875 | 0.976 | 0.948 | 0.921 | 0.9854 | 0.9796 | 0.9891 | 0.812
Table 5: Object detection and TSR performance metrics for models trained on each FinTabNet ablation’s training set and evaluated on each ICDAR-2013 ablation. Training Data | Epoch | Test Data | AP | $\textbf{AP}_{\textbf{50}}$ | $\textbf{AP}_{\textbf{75}}$ | AR | $\textbf{DAR}_{\textbf{C}}$ | $\textbf{GriTS}_{\textbf{C}}$ | $\textbf{GriTS}_{\textbf{L}}$ | $\textbf{GriTS}_{\textbf{T}}$ | $\textbf{Acc}_{\textbf{C}}$
---|---|---|---|---|---|---|---|---|---|---|---
FinTabNet.a1 | 28 | IC13.a1 | 0.670 | 0.859 | 0.722 | 0.737 | 0.8922 | 0.9390 | 0.9148 | 0.9503 | 0.417
| 28 | IC13.a2 | 0.670 | 0.859 | 0.723 | 0.737 | 0.9010 | 0.9384 | 0.9145 | 0.9507 | 0.411
| 20 | IC13.a3 | 0.445 | 0.573 | 0.471 | 0.491 | 0.8987 | 0.9174 | 0.8884 | 0.9320 | 0.411
FinTabNet.a2 | 27 | IC13.a1 | 0.716 | 0.856 | 0.778 | 0.776 | 0.9107 | 0.9457 | 0.9143 | 0.9536 | 0.436
| 27 | IC13.a2 | 0.714 | 0.856 | 0.778 | 0.775 | 0.9049 | 0.9422 | 0.9105 | 0.9512 | 0.430
| 29 | IC13.a3 | 0.477 | 0.576 | 0.516 | 0.516 | 0.8862 | 0.9196 | 0.8874 | 0.9336 | 0.405
FinTabNet.a3 | 25 | IC13.a1 | 0.710 | 0.851 | 0.765 | 0.771 | 0.9130 | 0.9462 | 0.9181 | 0.9546 | 0.462
| 25 | IC13.a2 | 0.710 | 0.850 | 0.764 | 0.770 | 0.9091 | 0.9443 | 0.9166 | 0.9538 | 0.456
| 28 | IC13.a3 | 0.470 | 0.571 | 0.505 | 0.510 | 0.8889 | 0.9229 | 0.8930 | 0.9346 | 0.418
FinTabNet.a4 | 30 | IC13.a1 | 0.763 | 0.935 | 0.817 | 0.834 | 0.9134 | 0.9427 | 0.9170 | 0.9516 | 0.551
| 30 | IC13.a2 | 0.763 | 0.935 | 0.818 | 0.834 | 0.9088 | 0.9409 | 0.9155 | 0.9503 | 0.544
| 25 | IC13.a3 | 0.765 | 0.944 | 0.832 | 0.836 | 0.9287 | 0.9608 | 0.9409 | 0.9703 | 0.589
FinTabNet.a5 | 30 | IC13.a1 | 0.774 | 0.939 | 0.838 | 0.840 | 0.9119 | 0.9412 | 0.9075 | 0.9456 | 0.494
| 30 | IC13.a2 | 0.774 | 0.940 | 0.840 | 0.839 | 0.9098 | 0.9400 | 0.9057 | 0.9443 | 0.487
| 16 | IC13.a3 | 0.773 | 0.956 | 0.854 | 0.839 | 0.9198 | 0.9548 | 0.9290 | 0.9624 | 0.551
FinTabNet.a6 | 23 | IC13.a1 | 0.760 | 0.927 | 0.830 | 0.830 | 0.9057 | 0.9369 | 0.9117 | 0.9497 | 0.500
| 21 | IC13.a2 | 0.757 | 0.926 | 0.818 | 0.828 | 0.9049 | 0.9374 | 0.9106 | 0.9491 | 0.506
| 20 | IC13.a3 | 0.757 | 0.941 | 0.840 | 0.830 | 0.9336 | 0.9625 | 0.9431 | 0.9702 | 0.646
PubTables-1M | 29 | IC13.a1 | 0.873 | 0.972 | 0.943 | 0.913 | 0.9440 | 0.9590 | 0.9462 | 0.9623 | 0.647
| 29 | IC13.a2 | 0.873 | 0.972 | 0.941 | 0.912 | 0.9392 | 0.9570 | 0.9448 | 0.9608 | 0.639
| 30 | IC13.a3 | 0.828 | 0.973 | 0.934 | 0.876 | 0.9570 | 0.9756 | 0.9700 | 0.9786 | 0.753
PubTables-1M + | 25 | IC13.a1 | 0.872 | 0.970 | 0.940 | 0.913 | 0.9543 | 0.9678 | 0.9564 | 0.9720 | 0.686
FinTabNet.a1 | 25 | IC13.a2 | 0.871 | 0.970 | 0.939 | 0.912 | 0.9501 | 0.9655 | 0.9543 | 0.9704 | 0.677
| 28 | IC13.a3 | 0.820 | 0.949 | 0.911 | 0.866 | 0.9630 | 0.9787 | 0.9702 | 0.9829 | 0.785
PubTables-1M + | 28 | IC13.a1 | 0.881 | 0.977 | 0.953 | 0.921 | 0.9687 | 0.9678 | 0.9569 | 0.9705 | 0.679
FinTabNet.a6 | 28 | IC13.a2 | 0.880 | 0.975 | 0.951 | 0.921 | 0.9605 | 0.9669 | 0.9566 | 0.9702 | 0.671
| 29 | IC13.a3 | 0.826 | 0.974 | 0.934 | 0.873 | 0.9648 | 0.9811 | 0.9750 | 0.9842 | 0.810

##  5 Conclusion

In this work we addressed the problem of _misalignment_ in benchmark datasets
for table structure recognition. We adopted the Table Transformer (TATR) model
and three standard benchmarks for TSR—FinTabNet, PubTables-1M, and
ICDAR-2013—and removed significant errors and inconsistencies between them.
After data improvements, performance of TATR on ICDAR-2013 increased
substantially from 42% to 65% exact match accuracy when trained on FinTabNet
and 65% to 75% when trained on PubTables-1M. In addition, we trained TATR on
the final FinTabNet and PubTables-1M datasets combined, establishing new
improved baselines of 0.965 DAR and 81% exact match accuracy on the ICDAR-2013
benchmark through data improvements alone. Finally, we demonstrated through
ablations that canonicalization has a significantly positive effect on the
performance improvements across all three datasets.

## References

  * [1] Broman, K.W., Woo, K.H.: Data organization in spreadsheets. The American Statistician 72(1), 2–10 (2018)
  * [2] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European Conference on Computer Vision. pp. 213–229. Springer (2020)
  * [3] Chi, Z., Huang, H., Xu, H.D., Yu, H., Yin, W., Mao, X.L.: Complicated table structure recognition. arXiv preprint arXiv:1908.04729 (2019)
  * [4] Frénay, B., Verleysen, M.: Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems 25(5), 845–869 (2013)
  * [5] Göbel, M., Hassan, T., Oro, E., Orsi, G.: ICDAR 2013 table competition. In: 2013 12th International Conference on Document Analysis and Recognition. pp. 1449–1453. IEEE (2013)
  * [6] Guyon, I., Matić, N., Vapnik, V.: Discovering informative patterns and data cleaning. In: Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining. pp. 145–156 (1994)
  * [7] Hashmi, K.A., Liwicki, M., Stricker, D., Afzal, M.A., Afzal, M.A., Afzal, M.Z.: Current status and performance analysis of table recognition in document images with deep neural networks. arXiv preprint arXiv:2104.14272 (2021)
  * [8] Hashmi, K.A., Stricker, D., Liwicki, M., Afzal, M.N., Afzal, M.Z.: Guided table structure recognition through anchor optimization. arXiv preprint arXiv:2104.10538 (2021)
  * [9] Hu, J., Kashi, R., Lopresti, D., Nagy, G., Wilfong, G.: Why table ground-truthing is hard. In: Proceedings of Sixth International Conference on Document Analysis and Recognition. pp. 129–133. IEEE (2001)
  * [10] Koch, B., Denton, E., Hanna, A., Foster, J.G.: Reduced, reused and recycled: The life of a dataset in machine learning research. arXiv preprint arXiv:2112.01716 (2021)
  * [11] Li, M., Cui, L., Huang, S., Wei, F., Zhou, M., Li, Z.: Tablebank: Table benchmark for image-based table detection and recognition. In: Proceedings of The 12th Language Resources and Evaluation Conference. pp. 1918–1925 (2020)
  * [12] Liu, H., Li, X., Liu, B., Jiang, D., Liu, Y., Ren, B.: Neural collaborative graph machines for table structure recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4533–4542 (June 2022)
  * [13] Nassar, A., Livathinos, N., Lysak, M., Staar, P.: Tableformer: Table structure understanding with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4614–4623 (June 2022)
  * [14] Northcutt, C.G., Athalye, A., Mueller, J.: Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14749 (2021)
  * [15] Paramonov, V., Shigarov, A., Vetrova, V.: Table header correction algorithm based on heuristics for improving spreadsheet data extraction. In: International Conference on Information and Software Technologies. pp. 147–158. Springer (2020)
  * [16] Prasad, D., Gadpal, A., Kapadni, K., Visave, M., Sultanpure, K.: CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 572–573 (2020)
  * [17] Raji, I.D., Bender, E.M., Paullada, A., Denton, E., Hanna, A.: Ai and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366 (2021)
  * [18] Schreiber, S., Agne, S., Wolf, I., Dengel, A., Ahmed, S.: DeepDeSRT: Deep learning for detection and structure recognition of tables in document images. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR). vol. 1, pp. 1162–1167. IEEE (2017)
  * [19] Seth, S., Jandhyala, R., Krishnamoorthy, M., Nagy, G.: Analysis and taxonomy of column header categories for web tables. In: Proceedings of the 9th IAPR International Workshop on Document Analysis Systems. pp. 81–88 (2010)
  * [20] Smock, B., Pesala, R., Abraham, R.: Grits: Grid table similarity metric for table structure recognition. arXiv preprint arXiv:2203.12555 (2022)
  * [21] Smock, B., Pesala, R., Abraham, R.: PubTables-1M: Towards comprehensive table extraction from unstructured documents. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4634–4642 (June 2022)
  * [22] Strickland, E.: Andrew Ng, AI Minimalist: The machine-learning pioneer says small is the new big. IEEE Spectrum 59(4), 22–50 (2022)
  * [23] Tensmeyer, C., Morariu, V.I., Price, B., Cohen, S., Martinez, T.: Deep splitting and merging for table structure decomposition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 114–121. IEEE (2019)
  * [24] Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022)
  * [25] Zheng, X., Burdick, D., Popa, L., Zhong, X., Wang, N.X.R.: Global table extractor (GTE): A framework for joint table identification and cell structure recognition using visual context. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 697–706 (2021)
  * [26] Zhong, X., ShafieiBavani, E., Yepes, A.J.: Image-based table recognition: data, model, and evaluation. arXiv preprint arXiv:1911.10683 (2019)
  * [27] Zhu, X., Wu, X.: Class noise vs. attribute noise: A quantitative study. Artificial intelligence review 22(3), 177–210 (2004)