11institutetext: Technology Innovation Hub, Indian Statistical Institute,
Kolkata, India
11email<EMAIL_ADDRESS>
22institutetext: Computer Vision Center, Computer Science Department,
Universitat Autònoma de Barcelona, Spain
22email: {sbiswas, abanerjee<EMAIL_ADDRESS>33institutetext: CVPR Unit,
Indian Statistical Institute, Kolkata, India
33email: {siladittya_r<EMAIL_ADDRESS>
44institutetext: Department of Electronics and Electrical Communication
Engineering,
Indian Institute of Technology Kharagpur, India
44email<EMAIL_ADDRESS>
# SelfDocSeg: A Self-Supervised vision-based Approach towards Document
Segmentation

Subhajit Maity∗ 11 0000-0002-0735-8406 Sanket Biswas∗ 22 0000-0001-6648-8270
Siladittya Manna 33 0000-0001-6364-8654 Ayan Banerjee 22 0000-0002-0269-2202
Josep Lladós 22 0000-0002-4533-4739 Saumik Bhattacharya 44 0000-0003-1273-7969
Umapada Pal 33 0000-0002-5426-2618

###### Abstract

Document layout analysis is a known problem to the documents research
community and has been vastly explored yielding a multitude of solutions
ranging from text mining, and recognition to graph-based representation,
visual feature extraction, _etc._ However, most of the existing works have
ignored the crucial fact regarding the scarcity of labeled data. With growing
internet connectivity to personal life, an enormous amount of documents had
been available in the public domain and thus making data annotation a tedious
task. We address this challenge using self-supervision and unlike, the few
existing self-supervised document segmentation approaches which use text
mining and textual labels, we use a complete vision-based approach in pre-
training without any ground-truth label or its derivative. Instead, we
generate pseudo-layouts from the document images to pre-train an image encoder
to learn the document object representation and localization in a self-
supervised framework before fine-tuning it with an object detection model. We
show that our pipeline sets a new benchmark in this context and performs at
par with the existing methods and the supervised counterparts, if not
outperforms. The code is made publicly available at:
github.com/MaitySubhajit/SelfDocSeg

###### Keywords:

Document Layout Analysis Document Segmentation Document Understanding Self-
supervised Learning

**footnotetext: These authors contributed equally to this work.

##  1 Introduction

From the early days of computer vision and document understanding research,
document layout analysis (DLA) had been a primitive problem and had been
conquered with a multitude of strategies [7] from classical methods [1, 46,
26] to state-of-the-art learning-based models [36, 55, 8]. While intelligent
document processing (IDP) has emerged as an essential step toward automatic
document understanding, bearing the rise of convolutional neural networks
(CNN) and sequence models, researchers have achieved near-perfect accuracy in
the context of DLA with models having vast deployability and reliability. With
a rapidly growing population, and the span of digital personal lives,
documents are becoming more unconventional and business applications for IDP
are encountering complex and never-before-seen layouts taking the already
solved challenge a level up with the requirement of deep features exploration
and exploitation. To meet that end, the state-of-the-art DLA strategies had
improved a lot over time and still continue to do so, while remaining one of
the most trendy topics for the research community [47].

The challenge of document segmentation or DLA had been a research interest for
nearly a decade and thus had been solved with classical heuristic rule-based
methods [26, 1] in the early days, while modern documents understanding
community treats it as a document object detection (DOD) problem and is
typically approached by a standard vision based object detector models [33,
43, 52]. With the breakthrough in sequence models and language models,
researchers had been using the same along with object detection to solve the
problem with better accuracy [36]. However, the research community has vastly
ignored the fact that the growth in the number of unconventional documents
raises the need for tedious annotation tasks to exploit the knowledge for
better document understanding through the conventional supervised setting, and
thus the self-supervised approaches toward the problem are highly relevant in
this context.

Figure 1:  Scope and Motivation. A basic methodological distinction between
SelfDocSeg and existing approaches. While earlier works utilize information
from visual, layout, and textual modalities for large-scale pre-training, we
deal with visual cues only for boosting representation learning.

Deploying self-supervision in DOD is quite non-trivial as the task inherently
cannot use the power of contrastive learning as the images contain multiple
document objects of different classes and thus naively using each picture as
its own class is not going to help. Apart from handling the class information,
the encoder needs to consider object localization which cannot be realized
through self-supervision without preliminary knowledge of the locations of
layout objects. Moreover, it had been observed that although deep visual
features extracted from document images are rich in information, they are
difficult to train and are usually guided with learned textual [29] or layout
[8] cues or both [36] from existing text spotter and detector models. And thus
the efficacy of self-supervision for document segmentation faces a big
question mark, which we address in this paper.

The core of our design consists of a self-supervised framework designed
following ’Bootstrap Your Own Latent’ (BYOL) [28] which actively tries to
fulfill both the objectives of representation and localization. As self-
supervision is employed for pre-training of the encoder, we do not have access
to the ground-truth document object locations and categories. Thus we use
classical image processing to approximate a rough physical layout mask for
each document image and use the same as guidance for both document object
localization and representation learning purpose. We use a backbone image
encoder to obtain feature maps, followed by a mask pooling operation to
extract encodings of all the possible relevant physical layout objects and
train them in a self-supervised representation learning framework using
negative cosine similarity. In parallel, we use a layout predictor module on
the encoded feature maps,l and the module is tasked with a classification task
of predicting if the salient features at every pixel of the feature map belong
to a document object or not. This module is trained using focal loss [43] with
the supervision of the generated physical layout mask.

The evidence had not been in favour of self-supervised vision-based approaches
for DLA tasks as visual representations needed to be guided with learned
textual and layout embeddings. However, we suggest that self-supervised visual
representation learning can still be explored as visual features prove to be
useful in the supervised settings for the task at hand. This had become the
core motivating factor for our work as we try to explore and exploit the rich
visual features extracted from the document images using the powerful CNN
backbones. The intuition behind the proposed framework is relatively simple as
we try to guide the learning of the backbone encoder using approximated visual
layouts for both document object localization and representation learning.
Unlike, the layout cues in the existing self-supervised strategies, we do not
use any layout information from any pre-trained text recognition model.
Instead, we devise the inherent visual information as a layout to guide the
visual representation learning. A clear distinction in working principles
between the proposed framework, SelfDocSeg, and the existing self-supervised
strategies can be realized in Fig. 1.

In summary, our contributions in this paper can be divided into three folds:
(a) A novel vision-based self-supervised framework, specifically designed to
pre-train an image encoder for DLA task;
(b) A pseudo physical layout guided strategy for self-supervision in the
region of interest localization for document segmentation;
(c) A data efficient pre-training strategy to learn multiple document object
representations simultaneously in the self-supervised setting.

The rest of the paper is organized as follows. A comprehensive literature
review is provided in Section 2. The proposed methodology is described in
Section 3. In Section 4 we discuss the experiments and results. Finally, the
conclusions are drawn in Section 5.

##  2 Related Work

###  2.1 Self-Supervision for Visual Representation

As the world of computer vision evolved it demanded undivided attention toward
the exploration and exploitation of complex visual features to learn
representations from images from a multitude of sources. Thus, data-centric
machine-learning models with immense capability of feature extraction and
correlation have emerged as modern-day solutions to the growing sophisticated
requirements. However, with the huge amount of data required for modern
network architectures, the need for data annotation has increased giving rise
to the plethora of self-supervised strategies. MoCo [32] provided the research
community with the idea of weight update using the exponential moving averages
and large memory banks in contrastive learning settings. SimCLR [16] improved
on it and introduced a large batch size as an alternative to the memory bank.
DINO [15] introduced self-supervision in vision transformers [23]. Following
suit, MoCov2 [17], SwAV [14] had achieved wonderful performance in the self-
supervised paradigm. On the other hand, BYOL [28], SimSiam [18] treat two
crops from the same image as similar pairs instead of contrastive learning,
while masked autoencoders [31] introduced a masking strategy to the age-old
autoencoders for learning representation via reconstruction.

Among this plethora of self-supervised methods, self-supervised object
detection and document segmentation remained vastly unexplored. Having
significantly superior performances from supervised object detection methods
like Mask RCNN [33], Yolo [52], Retinanet [43], DETR [13] _etc._ , their self-
supervised counterparts had been comparatively paler in diversity until
recently. In the last few years, we have seen end-to-end self-supervised
object detection models like UP-DETR [21], DETReg [6] and backbone pre-
training strategies like Self-EMD [44], Odin [35]. However, being closely
related to object detection, instance-level document layout analysis has
barely adopted self-supervision. Although there had been a few self-supervised
document segmentation approaches, none of them had been explicitly using self-
supervised visual representation or guidance. We address this lack of vision-
based self-supervision for document understanding in this paper.

###  2.2 Document Understanding

Document Understanding (DU) has been reformulated in the current document
analysis literature [11] as a landscape term covering different problems and
tasks related to Document Intelligence systems which majorly includes Key
Information Extraction [37, 49, 56], Classification [30], Document Layout
Analysis [63, 54], Question Answering [48, 59], and Machine Reading
Comprehension [58] whenever they involve visually rich documents (VRDs) in
contrast to plain texts or image-text pairs. Recent state-of-the-art DU
systems majorly rely on large-scale pre-training to combine both visual and
textual modalities as in [3, 36, 41, 29, 27] while methods like Donut [39] and
Dessurt [22] mostly rely on combining more effective visual features by
synthetic generation techniques [10, 61, 20, 38] for learning important layout
representation during document pre-training. In this work, we aim to identify
a new direction toward boosting visual representation through a self-
supervision strategy for document layout understanding.

###  2.3 Document Layout Analysis

DLA has evolved as a significant DU application, dedicated towards the
optimization of storage and large-scale information workflows [7]. Since the
advent of deep learning and CNN-based approaches [53, 55], the segmentation of
document layouts has been reformulated as DOD. With the launch of large-scaled
annotated DLA benchmarks [63, 54] opened a new direction for deep-learning-
based approaches. Later, Biswas et.al. [9] redefined the DLA task as an
instance-level segmentation task to detect both bounding boxes along with
segmentation masks, especially in layouts of pages containing overlapped
objects. More recently, transformer-based approaches [8, 36, 4] claimed the
recent state-of-the-art on DLA, especially on large-scaled document datasets.
However, there is still a huge scope for improvement of transformer methods,
especially in smaller annotated datasets like [19]. Recently, language-based
approaches like LayoutLMv3 [36] and UDoc [29] have been tested on PubLayNet
[63] benchmark to get the best performance. But they fail in document
benchmarks with more complex layouts and much smaller annotated samples. In
this work, we strive to propose SelfDocSeg a pure vision-based self-supervised
strategy to mitigate the aforementioned issues.

##  3 Methodology

###  3.1 Problem Formulation

In the context of document layout segmentation we have access to a relevant
dataset $\mathcal{D}=\\{x,y\\}$ where $x\in\mathcal{I}^{3\times H\times W}$ is
a standard RGB document image with height $H$ and width $W$, and
$y=\\{y_{1},\ldots,y_{p}\\}$ with $y_{l}$ having a set of coordinates
$y_{l}^{\texttt{mask}}$ of object pixels and $y_{l}^{\texttt{label}}$ for the
object $l=\\{1,\ldots,p\\}$ assuming $p$ objects in the image $x$. However,
according to the core principle of self-supervised learning, we do not use
this dataset as it contains the ground-truth annotations, and thus we derive a
dataset $\mathcal{D}^{\prime}=\\{x,m\\}$ from dataset $\mathcal{D}$ such that
it contains only the images without annotated data and a rough binary mask $m$
extracted from $x$ as described in Section 3.2 depicting the physical layout
of the document image.

The pre-training strategy for the image encoder briefly described in Section
3.3, has been adapted to our self-supervised framework, SelfDocSeg. Formally,
we train an image encoder $F_{\theta}$ parameterized by $\theta$ in a self-
supervised setting specifically designed to cater to document object
recognition. Once the pre-training is done we apply these weights to
initialize the backbone of an object detector model to be later fine-tuned for
the segmentation of document objects as described in Section 3.4.

Figure 2: Layout Mask Generation Pipeline. The figure illustrates a stepwise
approach with different strategies adapted for generating the final layout
mask for a given document image.

###  3.2 Layout Mask Generation

The layout mask generation is a crucial step for the proposed pipeline as our
self-supervised framework depends on it for some visual guidance. For any
document image $x$ we generate a mask $m$ using classical image processing
techniques as depicted in Fig. 2. At first, we convert the RGB image $x$ to
grayscale $x_{\text{gray}}$ as defined by CIE, followed by global thresholding
to obtain a binarized output $x_{\text{bin}}$. We then perform an erosion
operation over $x_{\text{bin}}$ to make the logical layout elements of the
document like characters, sub-figures, and plot lines a little thicker and
preferably a blob, so that the blobs get connected to form a rough physical
layout $\overline{m}$. The final layout mask $m$ is generated by an inversion
operation of this eroded rough layout $\overline{m}$.

###  3.3 Encoder Pre-Training

#### Overview:

The pre-training of the image encoder is quite difficult to achieve as we are
specifically considering the object detection problem, a multi-task challenge
that enforces the encoder to learn important salient features contributing
towards inter-class variance of different logical layout components and
simultaneously localizing the object regions. To overcome this problem, we use
the layout masks as visual guidance. However, there is no option to use layout
masks directly as we have two distinct yet related tasks at hand. Thus, the
challenge and also the scope of innovation remain on how the guidance is
provided for the two aforementioned tasks.

The proposed architecture of SelfDocSeg is developed inspired by the popular
BYOL [28] self-supervised framework. It mainly consists of two branches,
online and momentum, parameterized by two sets of weights $\theta$ and $\xi$
respectively. The overall learning strategy of the architecture is to use a
self-distillation technique that involves the two branches of the network
which exploits the similarity in salient features extracted from two different
views or variations of the same input. The feature similarity exploitation
principle demands the generation of significantly different views of the input
document image $x$. Thus we augment $x$ randomly to generate two different
versions $v_{1},v_{2}$ and feed them to the two branches of the SelfDocSeg
framework as illustrated in Fig. 3. Each of these branches employs multiple
modules to generate meaningful semantic embeddings from the image through a
mask pooling operation. This operation is designed in a way that
simultaneously extracts embeddings for all possible layout objects as a batch
from the input image $x$ using the guidance of the layout mask $m$ as we
roughly extract separate masks of each layout object to obtain an average
pooled vector for each.

In parallel, the layout prediction module ensures that the online network
learns to locate regions of interest in the document image feature map. We use
both the feature maps from $v_{1},v_{2}$ to feed the layout predictor module
and generate pixel-wise probability scores of a logical document object being
present at that pixel. This whole module is trained using $m$ with the help of
focal loss[43]. The overall architecture is depicted in Fig. 3 and the overall
loss function used to optimize the training is given by equation 1 where
$\mathcal{L}_{\text{Det}}$ comes from the Layout Prediction Module and
$\mathcal{L}_{\text{Sim}}$ comes from Layout Object Representation Learning as
discussed later in the following subsections.

$\displaystyle\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{Det}}+\mathcal{L}_{\text{Sim}}$
(1)

Figure 3:  Model Overview. A simple architectural design of SelfDocSeg pre-
training modeled after the BYOL [28] framework along with a layout prediction
module.

#### Augmentation:

The augmentations used to synthesize $v_{1},v_{2}$ from the input document
image $x$ are well crafted to match the task and are very similar to the
augmentations used in SimCLR [16]. From the set of augmentations used by
SimCLR, we carefully exclude random cropping and random horizontal flip. As we
are performing mask pooling on the feature map, it simultaneously tries to
learn all the possible physical layout object representations. And thus,
taking two patches from the image and using them as positive or contrastive
pairs is not required and is inherently done for multiple pairs
simultaneously. Moreover, the layout predictor module is designed to predict
layout all over the document image and thus the task does not align well with
taking randomly cropped patches from the document image. On the other hand,
the inputs, being document images, contain a large variety of texts and glyphs
which could break the clearly established pattern when flipped at random and
thus can hinder the training of our module. So, we exclude these two from the
set of augmentations and the final set includes the following: (a) Gaussian
blurring, (b) color jittering, (c) color dropping, and (d) solarization. The
augmentations are not manually controlled and are randomly applied on input
image $x$ with random parameters which provide a wide range of variations as
required to learn the similarity between the two views $v_{1},v_{2}$.

#### Image Encoder & Mask Pooling:

We define the image encoder $F:\mathcal{I}^{3\times H\times W}\rightarrow f$
which takes an input image and encodes it to a salient feature map
$f\in\mathbb{R}^{c\times h\times w}$ with $c$ channels and height and width of
$h$ and $w$ respectively. Considering both the online and the momentum
branches, we have two image encoders $F_{\theta}$ and $F_{\xi}$ providing
feature maps $f$ and $f^{\prime}$ respectively.

The mask pooling operation is performed on feature maps $f$ and $f^{\prime}$
separately on the two branches to get all the layout object representation
vectors simultaneously. The individual object layout masks $m_{k=1,\ldots,n}$
are obtained by detecting individual contours from $m$ assuming $n$ separate
contours in $m$. Once we have all the possible layout object masks, we get the
average pooled vector $y^{(k)}\in\mathbb{R}^{d}$ with dimension $d$ for each
of the objects from the feature map $f$ according to the established
definition of Mask Pooling[35] operation $MP:f\rightarrow y$ as per equation 2
where $i,j$ are pixel coordinates. Once we get $y^{(k)}$ for all
$k=1,\ldots,n$ estimated objects from $m_{k}$ in layout mask $m$, we neatly
stack the vectors to form $y\in\mathcal{R}^{k\times d}$.

$\displaystyle
y^{(k)}=\frac{1}{\sum_{i,j}m_{k}[i,j]}\sum_{i,j}m_{k}[i,j]f[i,j]$ (2)

#### Layout Object Representation Learning:

The overall framework of SelfDocSeg is designed after BYOL [28] and thus
follows a similar strategy to learn the representations from document images
using cosine similarity. The online branch comprises an image encoder
$F_{\theta}$, projector module $Z_{\theta}$, and a predictor module
$Q_{\theta}$ while the momentum branch consists of the image encoder $F_{\xi}$
and a projector module $Z_{\xi}$ where $\theta$ and $\xi$ are the sets of
parameters for online and momentum branch respectively.

For inputs $v_{1},v_{2}$ we get feature maps $f_{1},f_{2}$ and
$f^{\prime}_{1},f^{\prime}_{2}$ from $F_{\theta}$ and $F_{\xi}$ respectively
and we use mask pooling $MP$ as a strategy to get the encoded vector from the
feature map which is designed to pool out all the approximate physical layout
segments into individual encoded vectors simultaneously as discussed earlier
and neatly stack the encodings in a batch resulting in $y_{1},y_{2}$ and
$y^{\prime}_{1},y^{\prime}_{2}$, which contain average pooled vectors for all
$k=1,\ldots,n$ objects stacked in batches. The encoded vectors $y_{1},y_{2}$
and $y^{\prime}_{1},y^{\prime}_{2}$ are passed through projector modules
$Z_{\theta}$ and $Z_{\xi}$ to yield $z_{1},z_{2}$ and
$z^{\prime}_{1},z^{\prime}_{2}$ respectively. As the online branch of the
framework has a predictor layer $Q_{\theta}$, it takes input $z_{1},z_{2}$ and
produces $q_{1},q_{2}$.

$\displaystyle\theta\leftarrow\text{optimizer}\left(\theta,\nabla_{\theta}\mathcal{L}_{\text{total}},\eta\right)$
(3) $\displaystyle\xi\leftarrow\tau\xi+\left(1-\tau\right)\theta$

SelfDocSeg is designed to learn object representations without labels in a
self-distillation approach and uses exponentially moving average (EMA) to
update the weights of momentum branch $\xi$ from the online branch weights
$\theta$. Thus, $F_{\xi}$ and $Z_{\xi}$ are updated using EMA from
$F_{\theta}$ and $Z_{\theta}$ while $F_{\theta}$ and $Z_{\theta}$ are updated
using back-propagation on the loss function as given in equation 3 where
$\eta$ is the learning rate, $\tau$ is the momentum of EMA and
$\nabla_{\theta}\mathcal{L}_{\text{total}}$ denotes the gradient corresponding
to the total loss $\mathcal{L}_{\text{total}}$ from equation 1. It is also to
note that the back-propagation does not happen in the momentum branch as
denoted by the ’stop gradient’ sign in Fig. 3. The similarity loss
$\mathcal{L}_{\text{Sim}}$ uses cosine similarity to compare representations
from online and momentum branches as per equation 4.

$\displaystyle\mathcal{L}_{\text{Sim}}=4-2\left(\frac{\langle
q_{1},z^{\prime}_{2}\rangle}{||q_{1}||_{2}\cdot||z^{\prime}_{2}||_{2}}+\frac{\langle
q_{2},z^{\prime}_{1}\rangle}{||q_{2}||_{2}\cdot||z^{\prime}_{1}||_{2}}\right)$
(4)

#### Layout Prediction Module:

The layout prediction module $L$ is an auxiliary module facilitating the
layout object localization. This module helps the encoder to learn better
representation specifically for the detection task. The module takes input
from both the feature maps $f_{1},f_{2}$ and predicts a mask layout
$m_{\text{pred}}$, which is then compared to the approximated mask $m$ using
focal loss [43], $\mathcal{L}_{\text{Det}}$ given in equation 5 with hyper-
parameters $\alpha$ and $\gamma$. The intuition behind this module is to
instill a notion of localization of the layout objects, of which the encoder
is learning the representation from $\mathcal{L}_{\text{Sim}}$.

$\displaystyle\mathcal{L}_{\text{Det}}=-\frac{\alpha}{\sum_{i,j}m[i,j]}\cdot\sum_{i,j}(m[i,j](1-m_{\text{pred}}[i,j])^{\gamma}\log
m_{\text{pred}}[i,j]$ (5)
$\displaystyle+(1-m[i,j])m_{\text{pred}}[i,j]^{\gamma}\log(1-m_{\text{pred}}[i,j]))$

###  3.4 Fine-tuning

Once the image encoder is trained we proceed towards the main task of document
segmentation and for this purpose, we take the pre-trained weights of the
image encoder $F_{\theta}$ and use it to initialize the backbone weights of an
object detector model. This model is trained with the supervision of ground-
truth labels using whole annotated dataset $\mathcal{D}$ and we use Mask
RCNN[33] as our object detector model equipped with a feature-pyramid network
(FPN)[42] from intermediate layers for multi-scale detection.

##  4 Experiments

#### Datasets:

For the training and evaluation of our framework for document segmentation, we
used several datasets specifically designed for the task. We use the
DocLayNet[51] dataset for our pre-training experiments. It contains 69,375
training, 6,489 validation, and 4,999 test images from six different domains
with annotated ground-truth labels for 11 separate classes. However, we only
use the training split without the ground-truth labels for the pre-training
phase as mentioned in Section 3.

Once the pre-training is complete we move on to the document layout analysis
task to evaluate the efficacy of our pre-training strategy and for the same,
we use PRImA[2] dataset with 305 labeled images in total; Historic
Japanese[54] dataset, having 181,097 training, 39,410 validation and 39,109
test layout objects of seven different categories spanning over 2,271 document
images; and the extensive PubLayNet [63] dataset containing 335,703 training,
11,245 validation, and 11,405 test images with labeled layout masks of five
different layout object classes, along with the DocLayNet [51] dataset used in
pre-training.

#### Implementation Details:

The complete self-supervised framework is developed using a self-supervised
library named Lightly-AI [57], that is built on PyTorch Lightning [25] and
PyTorch [50]. All the models are trained on a 48 GB Nvidia RTX A40 GPU.

The mask generation process using classical image processing, described in
3.2, is developed using OpenCV [12]. We empirically found the global threshold
value of $239$ working well enough for the pre-training dataset, DocLayNet
[51] given that the grayscale images are having 8-bit unsigned integer
datatype. For the erosion operation, we use a $5\times 5$ rectangular kernel
as the structuring element.

Firstly, for the image encoders $F_{\theta},F_{\xi}$ we use a standard
ResNet50 [34] and we use the last residual block for extracting encoded
feature maps, leaving out the global average pooling and fully-connected
layers at the end, and thus the average pooled object vectors have dimension
$d=2048$ same as $c$ in the output of last residual block. The projectors
$Z_{\theta}$ and $Z_{\xi}$, and the predictor $Q_{\theta}$, are implemented as
two-layer multi-layer perceptrons (MLPs) with $4096$ hidden and $256$ output
dimensions. The auxiliary layout prediction module $L$ is implemented by a
$1\times 1$ convolution block to predict the layout mask. The focal loss
$\mathcal{L}_{\text{Det}}$ mentioned in equation 5 uses two hyper-parameters,
a weighing factor $\alpha=0.25$ and a focusing parameter $\gamma=2$.

In the pre-training phase, the optimal model parameters are learned by
optimizing the loss $\mathcal{L}_{\text{total}}$ from equation 1 using LARS
[62] optimizer with an initial learning rate of $\eta=0.2$, and a weight decay
of $0.0005$. The learning rate is decayed following a cosine annealing
schedule [45] for $800$ epochs. For the momentum branch, the value of the
momentum hyper-parameter $\tau$ is set to $0.99$.

The object detection model in fine-tuning with task-specific ground-truth
annotation uses the same backbone, ResNet50 [34] for Mask RCNN [33] with FPN
[42] and is trained with Detectron2 [60]. We use this framework because it is
built on top of PyTorch, is extremely easy to use, and is reliable for
deploying models. We have used Nvidia RTX A40 GPUs for all of our fine-tuning
purposes. Our trained models have been based on ResNet-50 weights pre-trained
as per our proposed framework, SelfDocSeg, on the DocLayNet [51] dataset. The
initial learning rate of the model was 0.0025, and it is trained for a total
of 300,000 iterations. A multitude of anchor scales and aspect ratios are
considered to cover the input image, resulting in the number of anchor boxes
$k=64$. Nesterov Momentum was used to train with the Stochastic Gradient
Descent (SGD) optimizer [24], and the batch size per image in the RoI heads
was 128. Our task had a detection non-maximum suppression (NMS) threshold of
$0.4$ and a detection minimum confidence score of $0.6$. The data loader’s
workforce was set at four. We set the model’s testing threshold in the RoI
heads to $0.6$ following the completion of the fine-tuning process. The values
mentioned are carefully chosen after reviewing empirical data.

###  4.1 Competitors

#### State-of-the-Art:

In this work, our contribution is towards a novel self-supervised pre-training
strategy, which cannot be directly compared to its counterparts without having
to look at the performance of the downstream task. Thus, we compare the
methodologies with existing models after fine-tuning the downstream task using
a document object detector.

In the context of self-supervised DLA task, the research community has yet to
see various strategies and thus we have only a few strategies to compare with.
We compared our methods with state-of-the-art (SOTA) self-supervised
methodologies: (a) LayoutLMv3 [36] uses a multi-modal transformer that uses a
masked language modeling and masked image modeling-based strategy using linear
image patch embeddings from the document and textual embeddings extracted from
a pre-trined optical character recognition (OCR) and also tries to align word
patches with image patches jointly with an image and word token classification
task. (b) UDoc [29] employs a region of interest alignment strategy that
aligns the visual features from the document images with the textual
embeddings and location information from a pre-trained OCR to generate joint
embeddings at the sentence level to use them with contextual masking to learn
with visual contrastive learning and visual-textual alignment through a cross
attention module. The noticeable difference between these existing SOTA with
the proposed pipeline can be identified as the use of pre-trained OCR to guide
the pre-training with textual and layout cues along with visual features while
we use only visual features for the purpose. (c) DiT [40] uses only visual
features like the proposed strategy using masked image modeling with BEiT [5]
pre-training over a massive dataset of size 42 million approximately.

In parallel, we show a comparative analysis with supervised SOTA
methodologies: (a) DocSegTr [8] implements a transformer encoder-decoder
architecture, being the first of its kind, along with a convolutional backbone
for document segmentation. (b) LayoutParser [55] uses a universal framework
built on top of CNN-based approaches for DOD evaluation (c) Biswas _et al._
[9] had used a Mask RCNN-styled architecture employing a fully convolutional
network, for instance, level segmentation and class prediction and bounding
box regression empowered by a region proposal network to suggest significant
regions of interest from the features extracted at multiscale by FPN on a
backbone.

Table 1: Quantitative analysis of the performance of the document object detection task along with the guidances used during supervision. V, L, and T stand for visual, layout, and textual cues respectively. | Methods | Cues | # Data | DocLayNet | PubLayNet | PRImA | HJ
---|---|---|---|---|---|---|---
V | L | T | mAP | mAP | mAP | mAP
Supervised | DocSegTr [8] | ✓ | ✗ | ✗ | - | - | 90.4 | 42.5 | 83.1
LayoutParser [55] | ✓ | ✓ | ✓ | - | - | 86.7 | 64.7 | 81.6
Biswas _et al._ [9] | ✓ | ✗ | ✗ | - | - | 89.3 | 56.2 | 82.0
Mask RCNN [33] | ✓ | ✗ | ✗ | - | 72.4 | 88.6 | 56.3 | 80.1
Self- Supervised | LayoutLMv3${}_{\text{Base}}$ [36] | ✓ | ✓ | ✓ | 11M | - | 95.1 | 40.3 | 82.7
UDoc [29] | ✓ | ✓ | ✓ | 1M | - | 93.9 | - | -
| DiT${}_{\text{Base}}$ [40] | ✓ | ✗ | ✗ | 42M | - | 93.5 | - | -
Proposed | BYOL [28] | ✓ | ✗ | ✗ | 81k | 63.5 | 79.0 | 28.7 | 59.8
SelfDocSeg | ✓ | ✗ | ✗ | 81k | 74.3 | 89.2 | 52.1 | 78.8

#### Comparable Baselines:

In terms of DOD, we also establish a few baselines, both supervised and
unsupervised. We used BYOL [28] as a pre-training strategy to train an image
encoder and used it to perform the fine-tune training on the document object
detector, Mask RCNN [33] with ground-truth and consider it as our potential
self-supervised baseline. On the other hand, we train Mask RCNN [33] in a
fully supervised setting with ground-truth annotations.

(a) Invoice

(b) Advertisement

(c) Industrial

(d) Leaflet

Figure 4: Qualitative analysis of the SelfDocSeg framework on the DocLayNet
datasets (Left: Predicted layout Right: Ground-truth)

###  4.2 Performance Analysis

The performance of our pre-training strategy is compared with the existing
methodologies in Table 1 in terms of mean average precision (mAP) score at the
pixel level segmentation of document objects along with the depiction of
guidance/cues used for each of the methods and amount of data used for self-
supervised pre-training. (a) LayoutLMv3 [36] and UDoc [29] have superior
performance on PubLayNet [63] dataset being the current self-supervised SOTAs.
LayoutLMv3 [36] also has superior performance over other works. However, both
methods use pre-trained OCR for textual and layout cues which provides them an
additional advantage in learning instance-level document layout component
embeddings coupled with the superior backbone of a multi-modal transformer. In
comparison, the proposed SelfDocSeg performs at par without any guidance from
pre-trained OCR. (b) DiT${}_{\text{Base}}$ [40] has a superior performance
compared to the proposed strategy due to its powerful Vision Transformer [23]
backbone coupled with the huge pre-training dataset of 42 million. (c)
DocSegTr [8] uses a convolutional backbone as well as a transformer
architecture with multi-headed attention that is inherently more powerful for
extraction of useful features and thus supervision with ground truth provides
superior performance in this scenario. It is noticed that transformer-based
models could not perform well in small datasets like PRImA [2]. (d)
LayoutParser [55] is a comprehensive layout analysis toolkit that offers
state-of-the-art performance with common CNN-based approaches, which shows
decent results owing to a powerful OCR module along with document image
augmentations. (e) The work of Biswas _et al._ [9] employs a modified Mask
RCNN architecture with end-to-end supervision and performs decently in the
context of Mask RCNN [33], compared to vanilla Mask RCNN in the fully
supervised setting. (f) The vanilla BYOL pre-training fails to learn
meaningful document object representations and thus performs poorly in Mask
RCNN. The possible justification for the performance could be attributed to
unrestricted random cropping and flipping in augmentations, which leads to two
different classes present in the same input document image being treated as
positive pairs. (g) Vanilla Mask RCNN performs decently in the fully
supervised setting. (h) It is evident that although Mask RCNN does not have
all-over superiority in performance, it easily outperforms all the
transformer-based models in small datasets like PRImA [2]. (i) It is also
noticed that modern self-supervised strategies use a large amount of data for
pre-training. In comparison, the proposed framework has achieved similar
performance with significantly less data in pre-training.

To this end, the SelfDocSeg framework proves to be performing at par with the
Mask RCNN competitors if not outperforming. And at the same time, it
establishes that self-supervised pre-training with only visual features using
a limited amount of data is efficient and effective for document segmentation
tasks. Some visual results of our proposed pipeline are depicted in Fig. 4.

Table 2: Semi-supervised scenario. We evaluate the performance of Mask RCNN fine-tuning with varying % of labeled data used in DocLayNet. % Annotations | mAP
---|---
10% | 41.3
50% | 65.1
100% | 74.3
Table 3: Ablation Study. Contribution of individual learning objectives during SelfDocSeg pre-training on DocLayNet. Loss | mAP
---|---
w/o $\mathcal{L}_{\text{Sim}}$ | 39.1
w/o $\mathcal{L}_{\text{Det}}$ | 69.7
Combined ($\mathcal{L}_{\text{total}}$) | 74.3

###  4.3 Ablation Study

#### How effective and generalizable is SelfDocSeg pretraining?

The importance of pre-training can be realized in the performance of the fine-
tuned document segmentation model and the natural question lingers around the
efficacy of the SelfDocSeg pre-training strategy. To answer the same, we
perform several experiments on the fine-tuning stage in a semi-supervised
setting. Meaning, we use multiple smaller sets of annotated training data in
fine-tuning Mask RCNN and record the performance in Table 2. Although using
all the available training data has the best result, it is evident that the
model generalizes well enough due to the learning of extensive and useful
visual feature extraction during pre-training as decreasing the amount of
labeled data during fine-tuning does not seem to affect the model accuracy
much. We see the mAP value drops 9% only while annotated training data is
reduced to half. However, it starts getting affected as the quantity of
labeled data drops drastically as we see more than 30% drop when the annotated
dataset is reduced to 10%.

#### How well do the individual losses contribute in SelfDocSeg?

We use two loss functions for the proposed pre-training strategy, focal loss
in equation 5 and a representation loss with cosine similarity in equation 4.
To dissect how much these losses individually contribute towards the pre-
training, we train two separate encoders from scratch, one with focal loss
only and another with representation loss only. The outcomes are depicted in
Table 3. It is evident that the focal loss although helps in localization, it
alone is not capable of learning meaningful object representations from the
document images. On the other hand, the representation loss is capable enough
to learn document object embeddings, but it falls short in the case of the
layout analysis tasks. And both the losses together help to pre-train an image
encoder for the document segmentation task.

##  5 Conclusion

While self-supervision is fairly new in the documents research community, in
this paper, we studied the exploitation of rich visual features present in the
document images using self-supervision. To this end, we designed a self-
supervised strategy to pre-train the image encoder in the document layout
analysis context. Our extensive experiments show that the proposed strategy
performs decently and generalizes well enough in document images despite
having no textual or layout guidance from pre-trained text recognition models.
In a word, we present a complete vision-based self-supervised approach towards
document segmentation which involves two specific strategies, _i.e._ a pseudo
physical layout guided technique for document object localization and a
document object representation learning in a self-supervised setting. We
further intend to explore the scope of performance improvements in DLA using
self-supervision.

## Acknowledgment

This work has been partially supported by IDEAS - Technology Innovation Hub,
Indian Statistical Institute, Kolkata; the Spanish project
PID2021-126808OB-I00, the Catalan project 2021 SGR 01559 and the PhD
Scholarship from AGAUR (2021FIB-10010). The Computer Vision Center is part of
the CERCA Program / Generalitat de Catalunya.

## References

  * [1] Agrawal, M., Doermann, D.: Voronoi++: A dynamic page segmentation approach based on voronoi and docstrum features. In: Proceedings of the International Conference on Document Analysis and Recognition. pp. 1011–1015 (2009)
  * [2] Antonacopoulos, A., Bridson, D., Papadopoulos, C., Pletschacher, S.: A realistic dataset for performance evaluation of document layout analysis. In: 2009 10th International Conference on Document Analysis and Recognition. pp. 296–300. IEEE (2009)
  * [3] Appalaraju, S., Jasani, B., Kota, B.U., Xie, Y., Manmatha, R.: Docformer: End-to-end transformer for document understanding. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 993–1003 (2021)
  * [4] Banerjee, A., Biswas, S., Lladós, J., Pal, U.: Swindocsegmenter: An end-to-end unified domain adaptive transformer for document instance segmentation. In: Document Analysis and Recognition–ICDAR 2023: 17th International Conference, San Jose, California, August 21–26, 2023, Proceedings. Springer (2023)
  * [5] Bao, H., Dong, L., Piao, S., Wei, F.: BEit: BERT pre-training of image transformers. In: International Conference on Learning Representations (2022)
  * [6] Bar, A., Wang, X., Kantorov, V., Reed, C.J., Herzig, R., Chechik, G., Rohrbach, A., Darrell, T., Globerson, A.: Detreg: Unsupervised pretraining with region priors for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)
  * [7] Binmakhashen, G.M., Mahmoud, S.A.: Document layout analysis: a comprehensive survey. ACM Computing Surveys (CSUR) 52(6), 1–36 (2019)
  * [8] Biswas, S., Banerjee, A., Lladós, J., Pal, U.: Docsegtr: an instance-level end-to-end document image segmentation transformer. arXiv preprint arXiv:2201.11438 (2022)
  * [9] Biswas, S., Riba, P., Lladós, J., Pal, U.: Beyond document object detection: instance-level segmentation of complex layouts. International Journal on Document Analysis and Recognition (IJDAR) 24(3), 269–281 (2021)
  * [10] Biswas, S., Riba, P., Lladós, J., Pal, U.: Docsynth: a layout guided approach for controllable document image synthesis. In: Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part III. pp. 555–568. Springer (2021)
  * [11] Borchmann, Ł., Pietruszka, M., Stanislawek, T., Jurkiewicz, D., Turski, M., Szyndler, K., Graliński, F.: Due: End-to-end document understanding benchmark. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021)
  * [12] Bradski, G.: The OpenCV Library. Dr. Dobb’s Journal of Software Tools (2000)
  * [13] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV (2020)
  * [14] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems (2020)
  * [15] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision (2021)
  * [16] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning (2020)
  * [17] Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 (2020)
  * [18] Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2021)
  * [19] Clausner, C., Antonacopoulos, A., Pletschacher, S.: Icdar2019 competition on recognition of documents with complex layouts-rdcl2019. In: Proceedings of the International Conference on Document Analysis and Recognition. pp. 1521–1526 (2019)
  * [20] Coquenet, D., Chatelain, C., Paquet, T.: Dan: a segmentation-free document attention network for handwritten document recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
  * [21] Dai, Z., Cai, B., Lin, Y., Chen, J.: Up-detr: Unsupervised pre-training for object detection with transformers. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2021)
  * [22] Davis, B., Morse, B., Price, B., Tensmeyer, C., Wigington, C., Morariu, V.: End-to-end document recognition and understanding with dessurt. In: Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IV. pp. 280–296. Springer (2023)
  * [23] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [24] Dozat, T.: Incorporating nesterov momentum into adam (2016)
  * [25] Falcon et al., W.: Pytorch lightning. GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning 3 (2019)
  * [26] Fang, J., Gao, L., Bai, K., Qiu, R., Tao, X., Tang, Z.: A table detection method for multipage pdf documents via visual seperators and tabular structures. In: ICDAR (2011)
  * [27] Gemelli, A., Biswas, S., Civitelli, E., Lladós, J., Marinai, S.: Doc2graph: a task agnostic document understanding framework based on graph neural networks. In: Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IV. pp. 329–344. Springer (2023)
  * [28] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.: Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems (2020)
  * [29] Gu, J., Kuen, J., Morariu, V.I., Zhao, H., Jain, R., Barmpalios, N., Nenkova, A., Sun, T.: Unidoc: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems 34, 39–50 (2021)
  * [30] Harley, A.W., Ufkes, A., Derpanis, K.G.: Evaluation of deep convolutional nets for document image classification and retrieval. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR). pp. 991–995. IEEE (2015)
  * [31] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)
  * [32] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020)
  * [33] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision (2017)
  * [34] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)
  * [35] Hénaff, O.J., Koppula, S., Shelhamer, E., Zoran, D., Jaegle, A., Zisserman, A., Carreira, J., Arandjelović, R.: Object discovery and representation networks. In: ECCV (2022)
  * [36] Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: Layoutlmv3: Pre-training for document ai with unified text and image masking. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 4083–4091 (2022)
  * [37] Jaume, G., Ekenel, H.K., Thiran, J.P.: Funsd: A dataset for form understanding in noisy scanned documents. In: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW). vol. 2, pp. 1–6. IEEE (2019)
  * [38] Kang, L., Riba, P., Rusinol, M., Fornes, A., Villegas, M.: Content and style aware generation of text-line images for handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(12), 8846–8860 (2021)
  * [39] Kim, G., Hong, T., Yim, M., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., Park, S.: Donut: Document understanding transformer without ocr. arXiv preprint arXiv:2111.15664 (2021)
  * [40] Li, J., Xu, Y., Lv, T., Cui, L., Zhang, C., Wei, F.: Dit: Self-supervised pre-training for document image transformer. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 3530–3539 (2022)
  * [41] Li, P., Gu, J., Kuen, J., Morariu, V.I., Zhao, H., Jain, R., Manjunatha, V., Liu, H.: Selfdoc: Self-supervised document representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5652–5660 (2021)
  * [42] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2117–2125 (2017)
  * [43] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision (2017)
  * [44] Liu, S., Li, Z., Sun, J.: Self-emd: Self-supervised object detection without imagenet. arXiv preprint arXiv:2011.13677 (2020)
  * [45] Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)
  * [46] Marinai, S., Gori, M., Soda, G.: Artificial neural networks for document analysis and recognition. TPAMI (2005)
  * [47] Markewich, L., Zhang, H., Xing, Y., Lambert-Shirzad, N., Jiang, Z., Lee, R.K.W., Li, Z., Ko, S.B.: Segmentation for document layout analysis: not dead yet. International Journal on Document Analysis and Recognition (IJDAR) pp. 1–11 (2022)
  * [48] Mathew, M., Karatzas, D., Jawahar, C.: Docvqa: A dataset for vqa on document images. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 2200–2209 (2021)
  * [49] Park, S., Shin, S., Lee, B., Lee, J., Surh, J., Seo, M., Lee, H.: Cord: a consolidated receipt dataset for post-ocr parsing. In: Workshop on Document Intelligence at NeurIPS 2019 (2019)
  * [50] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019), http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [51] Pfitzmann, B., Auer, C., Dolfi, M., Nassar, A.S., Staar, P.: Doclaynet: A large human-annotated dataset for document-layout segmentation. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 3743–3751 (2022)
  * [52] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)
  * [53] Schreiber, S., Agne, S., Wolf, I., Dengel, A., Ahmed, S.: Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR). vol. 1, pp. 1162–1167. IEEE (2017)
  * [54] Shen, Z., Zhang, K., Dell, M.: A large dataset of historical japanese documents with complex layouts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 548–549 (2020)
  * [55] Shen, Z., Zhang, R., Dell, M., Lee, B.C.G., Carlson, J., Li, W.: Layoutparser: A unified toolkit for deep learning based document image analysis. In: Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part I 16. pp. 131–146. Springer (2021)
  * [56] Stanisławek, T., Graliński, F., Wróblewska, A., Lipiński, D., Kaliska, A., Rosalska, P., Topolski, B., Biecek, P.: Kleister: key information extraction datasets involving long documents with complex layouts. In: Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part I. pp. 564–579. Springer (2021)
  * [57] Susmelj, I., Heller, M., Wirth, P., Prescott, J., et al., M.E.: Lightly. GitHub. Note: https://github.com/lightly-ai/lightly (2020)
  * [58] Tanaka, R., Nishida, K., Yoshida, S.: Visualmrc: Machine reading comprehension on document images. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 13878–13888 (2021)
  * [59] Tito, R., Karatzas, D., Valveny, E.: Hierarchical multimodal transformers for multi-page docvqa. arXiv preprint arXiv:2212.05935 (2022)
  * [60] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
  * [61] Yim, M., Kim, Y., Cho, H.C., Park, S.: Synthtiger: synthetic text image generator towards better text recognition models. In: Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part IV 16. pp. 109–124. Springer (2021)
  * [62] You, Y., Gitman, I., Ginsburg, B.: Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 (2017)
  * [63] Zhong, X., Tang, J., Yepes, A.J.: Publaynet: largest dataset ever for document layout analysis. In: Proceedings of the International Conference on Document Analysis and Recognition. pp. 1015–1022 (2019)


# Sensing of inspiration events from speech: comparison of deep learning and
linguistic methods

Aki Härmä1 , Ulf Grossekathöfer1, Okke Ouweltjes1 and Venkata Srikanth
Nallanthighal2 Currently with DACS, Maastricht University, The Netherlands.
1Philips Research, Eindhoven, The Netherlands. 2Philips Research, Bangalore,
India

###### Abstract

Respiratory chest belt sensor can be used to measure the respiratory rate and
other respiratory health parameters. Virtual Respiratory Belt, VRB, algorithms
estimate the belt sensor waveform from speech audio. In this paper we compare
the detection of inspiration events (IE) from respiratory belt sensor data
using a novel neural VRB algorithm and the detections based on time-aligned
linguistic content. The results show the superiority of the VRB method over
word pause detection or grammatical content segmentation. The comparison of
the methods show that both read and spontaneous speech content has a
significant amount of ungrammatical breathing, that is, breathing events that
are not aligned with grammatically appropriate places in language. This study
gives new insights into the development of VRB methods and adds to the general
understanding of speech breathing behavior. Moreover, a new VRB method,
VRBOLA, for the reconstruction of the continuous breathing waveform is
demonstrated.

Index Terms: Pathological speech sensing, speech breathing, respiratory health
and fitness, automatic speech recognition

##  1 Introduction

In recent year we have seen increase in speech-based health sensing methods
[1]. Often, the goal is to detect a condition, such as a respiratory infection
[2], or a specific diagnostic condition such as OSA [3], Alzheimer’s [4, 5] or
Parkinson’s disease [6]. One may also aim at estimating continuous values such
as the age of a talker [7], or simultaneous physiological sensor signal from
speech. For example, it would be very useful to be able to estimate the blood
sugar level of a diabetic patient [8] or respiratory parameters of a talker
[9] directly from the speech of a caller, for example, in a tele-health
application. The current paper studies the problem of respiratory health
sensing from speech using the VRB method. The Virtual Respiratory Belt, VRB,
processing uses speech audio to model a signal captured using a chest-worn
Respiratory Inductance Plethysmography, RIP, belt sensor. RIP measurements can
be used to get an estimate of the respiratory parameters such as respiratory
rate and tidal volume [10, 11]. McKenna et al demonstrated that it is also
possible to predict, from RIP sensor data, the values of spirometry
measurements during speechlike breathing [12].

Some of the early VRB work [9] was done using log-Mel spectrum data and RNN
neural networks. A VRB task was also included in the 2020 Paralinguistic
challenge [13]. Various CNN network architectures have been recently found
useful in the parallel problem of covid-19 detection [14] and it was shown in
a recent paper [anon] that the performance of the VRB modeling can be further
improved by using pre-trained transformer networks such as Hubert [15] or
Whisper [16]. In the current paper one such design is introduced, and we
demonstrate its performance in the task of the detection of inhaling, or
Inspiration Events, IEs, from read and spontaneous speech content. In
addition, we introduce and demonstrate the performance of a novel
modification, VRBOLA, of the conventional VRB method, which is using overlap-
add processing in the waveform reconstruction.

Ingressive phonation, that is, speech vocalization during inspiration, is
relatively rare [17], and the inspiration events in typical speech occur in
pauses between words. Speech pauses can be detected using speech activity
detection, SAD, algorithms. It is shown in this paper that by a selection of
long pauses in speech using a good SAD can give a rough estimate of the
occurrence of breathing events in speech.

There is a complex relationship between spoken language and our respiratory
behavior influenced by breathing planning [18, 19, 20, 21] which makes it
challenging to estimate the VRB signal directly from speech, see discussion in
[anon]. The breathing planning is expected to be guided by the rules of
language so that IEs are placed where there is a natural pause in speech, for
example, at end of a phrase or a sentence. In fact, some studies on read
speech respiration have found that IEs in speech occur almost exclusively at
grammatically appropriate places[18]. In conversational speech IEs occur also
elsewhere, e.g., upto 13 % in [22]. One could propose to use the detection of
grammatically appropriate places in speech as another method to detect IEs. In
this paper, the idea is tested and compared against the VRB method.

This paper compares the performance of the proposed VRB algorithm to a method
based on detection of speech pauses, and a method based on detection of
grammatically appropriate places in speech, respectively. As expected, the VRB
has the best performance in the detection of IEs in both read and spontaneous
speech. The analysis of the results gives very interesting insights about
speech breathing and design of algorithms for speech-based respiratory
sensing. It also seems to show that, in the data used in this study, the
assumption of grammatical breathing does not appear very strong but many IEs
in both read and spontaneous speech are examples of ungrammatical breathing,
i.e., inspiration events occurring in other parts of speech. Ungrammatical
breathing occurs in both read speech and spontaneous speech but is more
prevalent in spontaneous speech.

##  2 Breathing waveform estimation

The basic block diagram of a basic VRB algorithm is shown in Fig. 1a. The
speech preprocessing block contains typical steps of signal level
normalization by automatic gain control, pre-emphasis filtering to remove low
frequency noises and to flatten the spectrum, and possibly an insertion of a
background noise floor to make the models more robust for background noise
conditions. In the experiments reported in [23] the signal was transformed to
a Log-Mel-spectrogram representation. In [anon], it was demonstrated that the
performance of both instantaneous breathing waveform modeling and forecasting
breathing of events is significantly improved by using large pre-trained
wave2vec transformer models, such as Hubert [15], in the preprocessing of the
data before the neural network.

Figure 1: Block diagram for breathing waveform estimation from speech using a)
VRB b) ASR and logic

###  2.1 VRB wave modeling

The experiments of the current paper use a VRB model architecture that gave
the highest Pearson correlation in another study [23] between the obtained VRB
signal and the real belt signal. This model uses Hubert as the embedding model
to generate vectors corresponding to 20ms processing windows at the sample
rate of 16kHz. The vectors are stacked to a a matrix of 256 columns which are
further processed using a 3-layer Gated Recurrent Unit, GRU, network with 64
hidden units, and finally one dense linear layer to produce $K$ samples of the
VRB signal corresponding to the speech signal.

1: Use pretrained Hubert model to embed the content in 30ms windows to
$D=1024$-tap vectors

2: Stack $K=256$ vectors into a matrix ${\bf M}_{K\times D}$.

3: Train a cascade of a LSTM network and 1D CNN network to produce a $p$th
frame of a VRB signal $b_{p}(k),k=0,...K-1$.

Algorithm 1 Frame-based VRB modeling

The neural network was trained using the data from the dataset A introduced in
Section 4. The model was optimized using the smooth L1 loss function, and the
model optimization was performed by the Stochastic Gradient Descent, SGD,
method, both implemented in the pytorch library [24], respectively.

###  2.2 Continuous wave reconstruction

In previous work the VRB method reconstructed the signal sample-by-sample, or
in concatenated frames of $N$ samples. It turns out that it is very useful to
reconstruct the VRB signal using an overlap-add, OLA, method. In this paper,
this method is called VRBOLA. The last step of Algorithm 1 produces a VRB
signal fragment $b_{p}(k),k=0,...K-1$ for a $p^{th}$ block. In the VRBOLA
method reconstruction is carried out by applying the overlap-add method using
a sequence of window functions $w_{p}(t)$ so that the reconstruction of the
final VRB signal is given by

$b(t)=\sum_{p=-\infty}^{\infty}w_{p}(t)b_{p}(t-pS).$ (1)

The window sequence is defined so that $w_{p}(t)$ is, e.g., a squared sine, in
$t\in[pS,\,(p+1)S-1]$ and zero elsewhere. Overlap-add reconstruction reduces
the effects of the errors that are usually larger close to the frame
boundaries than in the center of the window.

In this paper, the focus is on the detection and characterization of
inspiration events, IEs. In inspiration phase the belt sensor signal typically
goes up, while during speech vocalization it goes down as lungs are depleted
from air. The detection of one IE in the respiration signal is defined by a
local maximum followed by local minimum in the waveform. In the following
experiments, the same algorithm was used for the detection of IEs from the
real belt waveform and the VRB signal estimated from speech.

1: Remove bias, sensor drift and high frequency noise by applying a third
order Butterworth band-pass filter from 0.08 Hz to 1.0 Hz using forward-
backward filtering.

2: Select local minima and maxima that have a minimal separation of 1s and
that have normalized peak prominence that exceeds a threshold of 0.8.

3: The average breathing rate is taken from the mean of the distance between
the maxima.

Algorithm 2 Detection of IEs

The average non-speech inspiration and expiration durations, for example, in
Chronic Obstructive Pulmonary Disease, COPD, patients are 1s and 1.7s,
respectively [25], which give a respiration rate of 0.2Hz. During speech, the
average respiratory rate is approximately half of that [26] and the duration
of an IE is may be less than 100ms. In this paper, the sample rate of the
embedding data matrix ${\bf M}$ and the modeled belt signal $b(t)$ is 50Hz,
corresponding to the 20ms input size of the Hubert model.

##  3 Respiratory sensing using ASR

In this paper, we use two databases of speech recordings with respiratory belt
measurements. The dataset A has 500 subjects and it has been collected at a
large medical center MAHE in India. The dataset B contains recordings of 40
talkers [27]collected in a research lab at Philips Research in Eindhoven, the
Netherlands. The VRB model used in this paper has been trained using the
dataset A, and all experiments reported in the paper have been performed on
dataset B.

The database B contains forty recordings of read speech and spontaneous
conversational speech. In early experiments we found that the WhisperX
algorithm [28] based on the Whisper ASR [16] and word-level alignment works
well for the data. The processing model for VRB estimation using ASR is shown
in Fig. 1b. After the ASR and word-level time-alignment the IE candidates are
selected by the punctuation points in the output of WhisperX. The durations of
IEs computed from the belt sensor data and the timestamps between punctuation
points and the start of the next word in ASR data, and pause durations from
the VRB signal, respectively, are shown in Fig. 2. The pauses in VRB follow
similar statistics to the real belt measurements, which can be expected
because the VRB model has been trained using similar belt data. The most
common duration of an IE is 200-250ms.

(

(

Figure 2: Histogram of inspiration event durations based on the belt sensor
data, ASR-punct, and VRB, respectively in a) read b) spontaneous speech.

a) b)

Let us define two ASR-based methods for the estimation of IEs from speech
data.

###  3.1 ASR-word method

The underlying idea is to assume that IEs occur in long word breaks in speech.
The method uses the whisperX to detect word boundaries and count all word-to-
word pauses that are longer than 150ms as IE candidates. This method is based
on assumption that long pauses in speech are IEs. The limit of 150ms was found
optimal for the best performance in IE detection accuracy in the dataset.

###  3.2 ASR-punct method

This method follows the observation in [18] that IEs usually occur at
grammatically appropriate places, grammatical stops. First all ends of clauses
and sentences are detected from the output of the whisperX algorithm. In most
cases, the end points are indicated by punctuation. Next, the end point of
each IE is selected to be the start point of the next word in the sequence
after the grammatical stop.

##  4 Experiments

The dataset B has speech from 40 healthy volunteers in a research institute in
the central Europe either reading ”The Rainbow” passage [29] or 1-2 minute
free speech in English about their current project or the last holiday trip.
The subjects were wearing a Respiratory Inductance Plethysmograph, RIP, belt
while speaking, which measures a continuous signal corresponding to the
circumference of the chest and abdomen of the subject. The VRB and VRBOLA
methods were trained using the dataset A with 200 recordings collected with a
medical center in India telling about a traditional fable in their local
language.

(

(

(

Figure 3: Examples of a) sensor belt signal and estimated VRB, b) detected IEs
in the belt and VRB signals, and c) word pauses and punctuation points at the
output of ASR.

a) b) c)

Fig 3a shows an example of a true sensor belt signal and an VRB estimate from
the VRB model. The IE detections based are shown in Fig. 3b. Fig. Fig. 3c
shows the long word pauses and grammatical stops based on the time-aligned ASR
output.

Based on the IE segments identified based on VRB, word pauses, or grammatical
stops, we can compute performance metrics of a method in comparison to the
ground truth IEs computed from the real sensor belt signal. True Positive
detection is counted if an IE estimate temporally overlaps with the true IE
and True Negative is counted when there are no IE estimates in regions between
ground truth IEs. False Positives and Negatives are counts of IEs between
ground truth IEs, and counts of missed ground truth IEs, respectively.

The results of read ”Rainbow” passage are shown in Table 1. The overall
performance, by the F1 score, is the highest, 0.85, for the VRB detections
followed by ASR-word and ASR-punct. The performance of ASR-punct is
significantly reduced by a high number of false negatives (fn). This is an
interesting observation because it is in opposition with the observation of
[18] that IEs occur, almost exclusively, in grammatically appropriate points.
By the relatively high specificity, 0.72, it indeed seems to be the case that
grammatical stops are often points of IEs. But, the high number of false
negatives is a clear indication of ungrammatical breathing, that is, a large
number of IEs in this read speech dataset occur in other parts of read content
than close to grammatical stops.

The ASR-word method based on long word pauses has a large number of both false
positives and false negatives but overall performance with F1 score 0.74 is
better than the ASR-punct method. Pauses in speech an IEs are indeed aligned,
but not all long pauses are IEs.

| ASR-word | ASR-punct | VRB | VRBOLA
---|---|---|---|---
tp | 960 | 752 | 885 | 929
tn | 985 | 762 | 866 | 915
fp | 343 | 319 | 101 | 93
fn | 347 | 568 | 290 | 232
Sensitivity | 0.73 | 0.57 | 0.75 | 0.80
Specificity | 0.74 | 0.70 | 0.90 | 0.91
F1-score | 0.74 | 0.63 | 0.82 | 0.85
Table 1: Performance of IE detection in read speech

The performance of the three methods, see Table 2, have a similar trend also
in the spontaneous speech dataset with the VRB method giving the highest
overall performance. However, the performance of the ASR-word method is here
significantly closer to VRB. One possible reason may be in the recording setup
where the subject was instructed to talk much and the interviewer was just
triggering more talk with brief questions. Many subjects were speaking in a
very relaxed way and had relatively longer IEs than in read speech. In the
ASR-punct one can see again increase in false negatives, that is,
ungrammatical breathing but also an increased number of false positives.

| ASR-word | ASR-punct | VRB | VRBOLA
---|---|---|---|---
tp | 513 | 370 | 512 | 532
tn | 517 | 389 | 488 | 507
fp | 269 | 312 | 136 | 135
fn | 207 | 266 | 168 | 175
Sensitivity | 0.71 | 0.58 | 0.75 | 0.75
Specificity. | 0.66 | 0.55 | 0.78 | 0.79
F1-score | 0.68 | 0.56 | 0.77 | 0.78
Table 2: Performance of IE detection in spontaneous speech

The comparison of the conventional concatenated VRB processing, and the new
VRBOLA method proposed in this paper shows a clear advantage for the proposed
overlap-add processing. The VRBOLA method increases the computational
requirements but it seems to increase the overall performance probably due to
elimination of transients in frame borders.

##  5 Discussion

Ingressive speech is rare in most languages and speech is largely phonetically
controlled expiration. Consequently, inspiration takes place, almost
exclusively, in pauses between words, or in nonverbal vocalization during
gasping or yawning. The results of the experiment reported in this paper shows
that detection of long pauses in speech can give a rough detection of
inspiration events, IEs, in speech. However, not all pauses are used for
inhaling and talkers are able to ventilate in very brief, less than 100ms,
pauses between words.

It is often suggested that IEs take mostly place in grammatically appropriate
places in speech. In a study by Winksworth et al. [18] breathing in read
speech is almost completely aligned with syntactic stops. The results of the
current paper do not support this observation but nearly half of the IEs seem
to take place in other parts of speech. In this paper we use the term
ungrammatical breathing for this phenomenon and it was shown that it is common
in both read and spontaneous speech. Obviously, not all paragraph, sentence,
clause, and phrase boundaries are IEs. In addition, ambiguities in ASR and
parsing, and ungrammatical speech may also lead to well-known errors, and
spontaneous speech may have ungrammatical passages of words.

Compared to the word pauses and grammatical stops, the best performance in the
detection of IEs was with the Virtual Respiratory Belt, VRB, algorithm trained
to model respiratory behavior of a talker from speech. The F1 score in the
correct detection of IEs was here over 0.85 while in the methods based on word
pauses or grammatical stops F1 score was below 0.75.

The spontaneous speech in the current paper consists of speech fragments from
a conversation but the actual turn-taking was not included in the analysis.
Breathing in conversation is also influenced by the turn-taking behavior, see,
e.g., [30], and in future work it would be interesting to explore the use of
the talk of the other party so support IE detection.

##  6 Acknowledgements

The authors wish to thank anonymous reviewers for their useful comments and
[anon] for the financial support of this work.

## References

  * [1] N. Cummins, A. Baird, and B. W. Schuller, “Speech analysis for health: Current state-of-the-art and the increasing impact of deep learning,” _Methods_ , vol. 151, pp. 41–54, Dec. 2018. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S1046202317303717
  * [2] M. Albes, Z. Ren, B. W. Schuller, and N. Cummins, “Squeeze for Sneeze: Compact Neural Networks for Cold and Flu Recognition,” in _Interspeech 2020_. ISCA, Oct. 2020, pp. 4546–4550. [Online]. Available: http://www.isca-speech.org/archive/Interspeech_2020/abstracts/2531.html
  * [3] E. Goldshtein, A. Tarasiuk, and Y. Zigel, “Automatic Detection of Obstructive Sleep Apnea Using Speech Signals,” _IEEE Transactions on Biomedical Engineering_ , vol. 58, no. 5, pp. 1373–1382, May 2011\.
  * [4] F. Haider, S. De La Fuente, and S. Luz, “An assessment of paralinguistic acoustic features for detection of Alzheimer’s dementia in spontaneous speech,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 14, no. 2, pp. 272–281, 2019.
  * [5] S. Luz, F. Haider, S. de la Fuente, D. Fromm, and B. MacWhinney, “Alzheimer’s Dementia Recognition through Spontaneous Speech: The ADReSS Challenge,” in _Proc. Interspeech_ , Shanghai, China, 2020. [Online]. Available: https://arxiv.org/abs/2004.06833
  * [6] L. Moro-Velazquez, J. Villalba, and N. Dehak, “Using x-vectors to automatically detect Parkinson’s disease from speech,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. Barcelona, Spain: IEEE, 2020, pp. 1155–1159.
  * [7] P. Ghahremani, P. S. Nidadavolu, N. Chen, J. Villalba, D. Povey, S. Khudanpur, and N. Dehak, “End-to-end deep neural network age estimation.” in _Proc. Interspeech_ , Hyderabad, India, 2018, pp. 277–281.
  * [8] J. Sidorova, P. Carbonell, and M. Čukić, “Blood Glucose Estimation From Voice: First Review of Successes and Challenges,” _Journal of Voice_ , vol. 36, no. 5, pp. 737.e1–737.e10, Sep. 2022. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0892199720303349
  * [9] V. S. Nallanthighal, A. Härmä, and H. Strik, “Deep Sensing of Breathing Signal During Conversational Speech,” in _Interspeech 2019_. ISCA, Sep. 2019, pp. 4110–4114. [Online]. Available: http://www.isca-speech.org/archive/Interspeech_2019/abstracts/1796.html
  * [10] C. Heyde, H. Leutheuser, B. Eskofier, K. Roecker, and A. Gollhofer, “Respiratory Inductance Plethysmography—A Rationale for Validity during Exercise,” _Medicine & Science in Sports & Exercise_, vol. 46, no. 3, pp. 488–495, Mar. 2014. [Online]. Available: https://journals.lww.com/00005768-201403000-00008
  * [11] B. Laufer, S. Krueger-Ziolek, P. D. Docherty, F. Hoeflinger, L. Reindl, and K. Moeller, “Tidal volume via circumferences of the upper body: a pilot study,” _Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference_ , vol. 2019, pp. 3559–3562, Jul. 2019.
  * [12] V. S. McKenna and J. E. Huber, “The Accuracy of Respiratory Calibration Methods for Estimating Lung Volume During Speech Breathing: A Comparison of Four Methods Across Three Adult Cohorts,” _Journal of speech, language, and hearing research: JSLHR_ , vol. 62, no. 8, pp. 2632–2644, Aug. 2019.
  * [13] B. W. Schuller, A. Batliner, C. Bergler, E.-M. Messner, A. Hamilton, S. Amiriparian, A. Baird, G. Rizos, M. Schmitt, L. Stappen, H. Baumeister, A. D. MacIntyre, and S. Hantke, “The INTERSPEECH 2020 Computational Paralinguistics Challenge: Elderly Emotion, Breathing & Masks,” in _Interspeech 2020_. ISCA, Oct. 2020, pp. 2042–2046. [Online]. Available: https://www.isca-speech.org/archive/interspeech_2020/schuller20_interspeech.html
  * [14] M. Effati and G. Nejat, “A Performance Study of CNN Architectures for the Autonomous Detection of COVID-19 Symptoms Using Cough and Breathing,” _Computers_ , vol. 12, no. 2, p. 44, Feb. 2023. [Online]. Available: https://www.mdpi.com/2073-431X/12/2/44
  * [15] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” Jun. 2021, arXiv:2106.07447 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2106.07447
  * [16] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” _arXiv preprint arXiv:2212.04356_ , 2022.
  * [17] R. Eklund, “Pulmonic ingressive phonation: Diachronic and synchronic characteristics, distribution and function in animal and human sound production and in human speech,” _Journal of the International Phonetic Association_ , vol. 38, no. 3, pp. 235–324, Dec. 2008, publisher: Cambridge University Press.
  * [18] A. L. Winkworth, P. J. Davis, E. Ellis, and R. D. Adams, “Variability and consistency in speech breathing during reading: Lung volumes, speech intensity, and linguistic factors,” _Journal of Speech, Language, and Hearing Research_ , vol. 37, no. 3, pp. 535–556, 1994.
  * [19] M. Włodarczak and M. Heldner, “Respiratory Constraints in Verbal and Non-verbal Communication,” _Frontiers in Psychology_ , vol. 8, May 2017\. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5434352/
  * [20] O. Egorow, T. Mrech, N. Weißkirchen, and A. Wendemuth, “Employing Bottleneck and Convolutional Features for Speech-Based Physical Load Detection on Limited Data Amounts,” in _Interspeech 2019_. ISCA, Sep. 2019, pp. 1666–1670. [Online]. Available: http://www.isca-speech.org/archive/Interspeech_2019/abstracts/2502.html
  * [21] S. Fuchs and A. Rochet-Capellan, “The respiratory foundations of spoken language,” _Annual Review of Linguistics_ , vol. 7, no. 1, pp. 13–30, 2021\.
  * [22] Y.-T. Wang, J. R. Green, I. S. B. Nip, R. D. Kent, and J. F. Kent, “Breath group analysis for reading and spontaneous speech in healthy adults,” _Folia phoniatrica et logopaedica: official organ of the International Association of Logopedics and Phoniatrics (IALP)_ , vol. 62, no. 6, pp. 297–302, 2010.
  * [23] V. S. Nallanthighal, Z. Mostaani, A. Härmä, H. Strik, and M. Magimai-Doss, “Deep learning architectures for estimating breathing signal and respiratory parameters from speech recordings,” _Neural Networks_ , vol. 141, pp. 211–224, Sep. 2021\. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0893608021001179
  * [24] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” in _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d. Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8024–8035.
  * [25] W. Chatila, T. Nugent, G. Vance, J. Gaughan, and G. J. Criner, “The Effects of High-Flow vs Low-Flow Oxygen on Exercise in Advanced Obstructive Airways Disease,” _Chest_ , vol. 126, no. 4, pp. 1108–1115, Oct. 2004\.
  * [26] M. Włodarczak, M. Heldner, and J. Edlund, “Breathing in conversation: An unwritten history,” in _2nd European and the 5th Nordic Symposium on Multimodal Communication, Tartu, Estonia, August 6-8, 2014_ , 2015, pp. 107–112.
  * [27] V. S. Nallanthighal, A. Härmä, and H. Strik, “Deep Sensing of Breathing Signal During Conversational Speech,” in _Interspeech 2019_. ISCA, Sep. 2019, pp. 4110–4114. [Online]. Available: http://www.isca-speech.org/archive/Interspeech_2019/abstracts/1796.html
  * [28] M. Bain and T. Han, “WhisperX,” 2022, publication Title: GitHub repository. [Online]. Available: https://github.com/m-bain/whisperX
  * [29] G. Fairbanks, “The rainbow passage,” _Voice and articulation drillbook_ , vol. 2, 1960.
  * [30] M. Wlodarczak and M. Heldner, “Breathing in Conversation,” _Frontiers in Psychology_ , vol. 11, 2020. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fpsyg.2020.575566