# Unified Pretraining Framework for Document Understanding

Jiuxiang Gu1, Jason Kuen1, Vlad I. Morariu1, Handong Zhao1,
Nikolaos Barmpalios2, Rajiv Jain1, Ani Nenkova1, Tong Sun1
1Adobe Research, 2Adobe Document Cloud
<EMAIL_ADDRESS>
###### Abstract

Document intelligence automates the extraction of information from documents
and supports many business applications. Recent self-supervised learning
methods on large-scale unlabeled document datasets have opened up promising
directions towards reducing annotation efforts by training models with self-
supervised objectives. However, most of the existing document pretraining
methods are still language-dominated. We present UDoc, a new unified
pretraining framework for document understanding. UDoc is designed to support
most document understanding tasks, extending the Transformer to take
multimodal embeddings as input. Each input element is composed of words and
visual features from a semantic region of the input document image. An
important feature of UDoc is that it learns a generic representation by making
use of three self-supervised losses, encouraging the representation to model
sentences, learn similarities, and align modalities. Extensive empirical
analysis demonstrates that the pretraining procedure learns better joint
representations and leads to improvements in downstream tasks.

##  1 Introduction

Document intelligence is a broad research area that includes techniques for
information extraction and understanding. Unlike plain-text documents in
natural language processing (NLP) yang2016hierarchical ; goldstein2000multi ,
a physical document can be composed of multiple elements: tables, figures,
charts,  _etc_. In addition, a document usually includes rich visual
information, and can be one of various types of documents (scientific paper,
form, resume,  _etc_.), with various combinations of multiple elements and
layouts. Complex content and layout, noisy data, font and style variations
make automatic document understanding very challenging. For example, to
understand text-rich documents such as letters, a system needs to focus almost
exclusively on text content, paying attention to a long sequential context,
while processing semi-structured documents such as forms requires the system
to analyze spatially distributed short words, paying particular attention to
the spatial arrangement of the words. Following the success of BERT
dai2019transformer  on NLP tasks, there has been growing interest in
developing pretraining methods for document understanding xu2020layoutlm ;
xu2020layoutlmv2 ; selfdoc2021 . Pretrained models have achieved state-of-the-
art (SoTA) performance across diverse document understanding tasks
park2019cord ; harley2015icdar .

Huge training datasets help pretraining models to learn a good representation
for downstream tasks. However, we observe three major problems with the
current pretraining setup: (1) documents are composed of semantic regions.
Most of the recent document pretraining works follow BERT and split documents
into words. However, unlike the sequence-to-sequence learning in NLP,
documents have a hierarchical structure (words form sentences, sentences form
a semantic region, and semantic regions form a document). Also, the importance
of words and sentences are highly context-dependent,  _i.e_., the same word or
sentence may have different importance in a different context. Moreover,
current transformer-based document pretraining models suffer from input length
constraints. Also, input length becomes a problem for text-rich documents or
multi-page documents. (2) documents are more than words. The semantic
structure of the document is not only determined by the text within it but
also the visual features such as table, font size and style, and figure,
_etc_. Moreover, the visual appearance of the text within a block are often
overlooked. Most of recent BERT-based pre-training works only take the words
as input without considering multimodal content and alignment of multimodal
information within semantic regions. (3) documents have spatial layout. Visual
and layout information is critical for document understanding. Recent works
encode spatial information via 2D position encoding and model spatial
relationships with self-attention, which computes attention weights for long
inputs xu2020layoutlm ; xu2020layoutlmv2 . However, for semi-structured
documents, such as forms and receipts, words are more related to their local
surroundings. This corresponds strongly with human intuition – when we look at
magazines or newspapers, the receptive fields are modulated by our reading
order and attention. Based on the above observations, we ask the following
question: Can unified document pretraining benefit all of these different
kinds of documents?

We propose a unified pretraining framework for document understanding, shown
in Fig. 1. Our model integrates image information in the pretraining stage by
taking advantage of the transformer architecture to learn cross-modal
interactions between visual and textual information. To handle textual
information, we encode sentences using a hierarchical transformer encoder. The
first level of the hierarchical encoder models the formation of the sentences
from words. The second level models the formation of the document from
sentences. With the help of the hierarchical structure, UDoc learns how words
form sentences and how sentences form documents. Meanwhile, it reduces model
computation complexity exponentially and increases the number of input words.
This also mimics human reading behaviors since the sentence/paragraph is a
reasonable unit for people to read and understand—people rarely check the
interactions between arbitrary words across different regions in order to
understand an article. Convolution has been very successful in the extraction
of local features that encode visual and spatial information gu2018recent , so
we use convolution layers as a more efficient complement to self-attention for
addressing local intra-region dependencies in a document image. Meanwhile,
self-attention uses all input tokens to generate attention weights for
capturing global dependencies. Thus, we combine convolution with self-
attention to form a mixed attention mechanism that combines the advantages of
the two operations.

We depart from previous vision-language pretraining lu2019vilbert ; gu2020self
by extracting both the textual and visual features for each semantic region.
We propose a novel gated cross-attentional transformer that enables
information exchange between modalities. A visually-rich region (figure,
chart,  _etc_) may have stronger visual information than textual information.
Instead of treating outputs from both modalities identically, we design a
gating mechanism that can dynamically control the influence of textual and
visual features. This approach enables cross-modal connections and allows for
variable highlight the relevant information in visual and textual modality and
enables cross-modal connections. During pretraining, the CNN-based visual
backbone and multi-layer gated cross-attention encoder are jointly trained in
both pretraining and fine-tuning phases.

Our contributions are summarized as follows: (1) We introduce UDoc, a powerful
pretraining framework for document understanding. UDoc is capable of learning
contextual textual and visual information and cross-modal correlations within
a single framework, which leads to better performance. (2) We present Masked
Sentence Modeling for language modeling, Visual Contrastive Learning for
vision modeling, and Vision-Language Alignment for pretraining. (3) We present
extensive experiments and analyses to validate the effectiveness of the
proposed UDoc. Extensive experiments and analysis provide useful insights on
the effectiveness of the pretraining tasks and show outstanding performance on
various downstream tasks.

Figure 1: Overview of the proposed approach, UDoc. UDoc first uses a CNN-based
visual backbone to learn visual representations. The model then extracts the
Region of Interest (RoI) features with OCR bounding boxes and generates a
multimodal embedding by combining the textual embedding and position encoding.
The transformer-based encoder takes a set of masked multimodal embeddings as
input and is pretrained with three pretraining tasks. All the network
parameters except those of the textual encoder are jointly trained during both
pretraining and fine-tuning phases.

##  2 Related Work

Self-supervised learning has shown great success in producing generic
representations that learn from large-scale unlabeled corpora
dai2019transformer . Like the development of pretraining in computer vision
deng2009imagenet  and NLP dai2019transformer , there has been a surging
interest in self-supervised learning for Vision-Language (VL) tasks
lu2019vilbert ; su2019vl ; li2019unicodervl ; gu2020self . Transformers
dai2019transformer  are the key technology that enables learning
contextualized representations from large-scale unlabeled training data. The
unique characteristics of document images (spatial layout and multiple
elements) distinguish document image pretraining from pretraining works in NLP
and VL domains. In the NLP domain, the inputs are pure texts without spatial
layouts (bounding boxes). In the VL domain, the inputs are the visual objects
and captions. While for document images, the input elements are spatially
distributed, and the visual and textual information co-occur within the
semantic regions.

Several recent works have explored pretraining on document images
xu2020layoutlm ; xu2020layoutlmv2 ; patil2020read . LayoutLM xu2020layoutlm
extends BERT to learn contextualized word representations for document images
through multi-task learning. It takes a sequence of OCR words as input during
pretraining and incorporates the 2D position embedding as input for each
token. However, LayoutLM only considers textual information during pretraining
without modeling the alignment between visual and textual information–visual
information is only incorporated into the model during the fine-tuning stage.
The most recent version, LayoutLMv2 xu2020layoutlmv2 , improves on this by
incorporating the image encoder into pretraining and jointly training the
image encoder along with the BERT model. LayoutLMv2 splits the document image
into several parts and concatenates the visual embeddings and text embeddings
into a single sequence. Apart from masked language learning (MLM), LayoutLMv2
also considers image-text alignment and image-text matching during
pretraining. The most related work to ours is SelfDoc selfdoc2021 , which
proposes a multimodal document pretraining framework. It first extracts the
document object proposals from pre-trained Faster R-CNN girshick2015fast  and
then applies OCR for each proposal to get the words. It takes the pre-
extracted RoI features and sentence embeddings as input, and models the
perform learning over the textual and visual information using the cross-
modality encoder.

There is a noticeable difference between our proposed method, UDoc, and other
concurrent works in document image pretraining. UDoc is a multimodal end-to-
end pretraining framework for document images. Unlike the fixed document
object detector in selfdoc2021 , the parameters of the image encoder with RoI
align, which derive the visual features for semantic regions, are also updated
in UDoc. In contrast to xu2020layoutlmv2 , our visual features come from the
semantic regions instead of splitting the image into fixed regions. Like the
object-level semantic elements in natural images, for document images, we
represent the typical document layout elements such as paragraph, title,
figure, and table as semantic regions. Moreover, to learn the contextualized
visual representations, UDoc masks visual information in the latent space and
learns contextualized representations by solving a contrastive learning task
defined over a quantization of the latent visual embeddings.

##  3 Method

###  3.1 Model Architecture

Fig. 1 illustrates our approach, UDoc, which consists of four components:
feature extraction, feature embedding, multi-layer gated cross-attention
encoder, and pretraining tasks. Given a document image and the locations of
document elements (sentence or RoI), UDoc takes image regions and words that
correspond to each document elements as inputs, and extracts their respective
embeddings through a visual feature extractor and a sentence encoder. These
embeddings are then fed into a transformer-based encoder to learn the cross-
modal contextualized embeddings that integrate both visual features and
textual features.

In the feature extraction step, we first employ an off-the-shelf OCR tool
easyocr  to extract text from a document image $\mathbf{I}$, where the words
are grouped into sentences $\mathcal{S}=\\{s_{1},\ldots,s_{N}\\}$ whose
corresponding bounding boxes are $\mathcal{P}=\\{p_{1},\ldots,p_{N}\\}$. For
each sentence bounding box $p_{i}$, we use a ConvNet-based backbone
$f_{\text{ImEnc}}$ and RoI Align he2017mask  $f_{\text{RoIAlign}}$ to extract
the pooled RoI features $\bm{v}_{i}$. To obtain a feature embedding, we
extract the sentence embedding $\bm{s}_{i}$ for each sentence $s_{i}$ via a
pretrained sentence encoder $f_{\text{SentEnc}}$. Each region’s RoI feature
$\bm{v}_{i}$ is discretized into a finite set of visual representations
$\bm{v}_{i}^{Q}\in\mathbf{V}^{Q}$ via product quantization jegou2010product .
The multi-layer Gated Cross-Attention encoder takes the position information,
masked visual features $\mathbf{\tilde{V}}$ and masked textual features
$\mathbf{\tilde{S}}$ as inputs, and then it generates the contextualized
multimodal representations ($\mathbf{H}_{V}^{l}$ and $\mathbf{H}_{S}^{l}$,
$l\in[1,L]$) and outputs the predicted features ($\mathbf{\hat{V}}$ and
$\mathbf{\hat{S}}$), where $L$ is the number of stacked transformer blocks.

More formally, the pretraining procedure can be decomposed into the following
steps:

$\displaystyle\mathbf{I}\xrightarrow[]{\text{OCR}}\binom{\mathcal{P}}{\mathcal{S}}\xrightarrow[f_{\text{SentEnc}}]{f_{\text{ImEnc}}+f_{\text{RoIAlign}}}\binom{\mathbf{V},\mathbf{V}^{Q}}{\mathbf{S}}\xrightarrow[f_{\text{Mask}}]{}\binom{\mathbf{\tilde{V}}}{\mathbf{\tilde{S}}}\xrightarrow[]{}\binom{\mathbf{H}_{V}^{l}}{\mathbf{H}_{S}^{l}}\xrightarrow[]{}\binom{\mathbf{\hat{V}}}{\mathbf{\hat{S}}}\xrightarrow[]{}\mathcal{L}_{\text{Pretraining}}$
(1)

where $f_{\text{Mask}}$ denotes the masking function that randomly masks RoI
features and sentence embeddings with the respective probabilities
$p_{\text{Mask}}^{v}$ and $p_{\text{Mask}}^{s}$.
$\mathcal{L}_{\text{Pretraining}}$ is composed of three pretraining tasks:
Masked Sentence Modeling (MSM), Visual Contrastive Learning (VCL), and Vision-
Language Alignment (VLA). Next, we provide details mentioned in Eq. 1.

#### Feature Extraction and Embedding.

Formally, a document image $\mathbf{I}\in\mathbb{R}^{W\times H}$ consists of
$N$ regions, where each region’s bounding box is characterized by a 6-d
vector, as
$p_{i}=\\{\frac{x_{\text{LT}}}{W},\frac{y_{\text{LT}}}{H},\frac{x_{\text{RB}}}{W},\frac{y_{\text{RB}}}{H},\frac{w}{W},\frac{h}{H}\\}$,
where $w$ and $h$ are of the width and height the region, $W$ and $H$ are the
width and height of $\mathbf{I}$, while $(x_{\text{LT}},y_{\text{LT}})$ and
$(x_{\text{RB}},y_{\text{RB}})$ denote the coordinates of the top-left and
bottom-right corners respectively. The 6-d vector is mapped into a high-
dimensional representation via a linear mapping function.

The visual embedding is the sum of the mapped RoI feature and position
embedding. Likewise, textual embedding is the sum of sentence embedding and
position embedding. We also have different types of segments to distinguish
different modalities. The input sequence to the transformer-based encoder
starts with a special start element ([CLS] and full visual features), then it
is followed by multimodal elements, and it ends with a special ending element
([SEP]+full visual features). For the special elements ([CLS] and [SEP]), the
corresponding full visual features are features extracted from the whole input
image, by applying $f_{\text{ImEnc}}$ to an RoI covering the whole input
image.

#### Quantization Module.

Unlike the fixed image encoder in selfdoc2021 , we jointly learn the image
encoder in an end-to-end fashion alongside the multimodal model. A visual
representation can be learned by predicting the visual features of the masked
regions, but it is challenging to predict such features exactly, since they
are unconstrained and of continuous representation. To constrain the
representation space of the visual features and facilitate the end-to-end
learning of image encoder (see Task #2 in Sec. 3.2), we follow oord2017neural
; baevski2020wav2vec  and use vector quantization to discretize the visual
features $\mathbf{V}=\\{\bm{v}_{1},\ldots,\bm{v}_{N}\\}$ into a finite set of
representations $\mathbf{V}^{Q}=\\{\bm{v}_{1}^{Q},\ldots,\bm{v}_{N}^{Q}\\}$.
Specifically, we define latent embedding spaces $\bm{e}\in\mathbb{R}^{C\times
E}$, where $C$ is the number of codebooks, and $E$ is the number of entries
for each codebook. For each $\bm{v}_{i}$, we first map it to logits
$\bm{v}^{\ell}_{i}\in\mathbb{R}^{C\times E}$, and calculate the probability
for the $j$-th codebook entry in $i$-th group as
$p_{c,e}=\text{exp}((v^{\ell}_{c,e}+g_{e})/\tau)/\sum_{k=1}^{E}\text{exp}((v^{\ell}_{c,k}+g_{k})/\tau)$,
where $\tau$ is a non-negative temperature, $g_{1:E}$ are i.i.d samples drawn
from Gumbel(0,1) distribution. During the forward pass, we choose one entry
vector from each codebook by $\bm{\tilde{e}}_{i}\sim\text{argmax}_{e}p_{c,e}$
and generate the quantized representation $\bm{v}^{Q}_{i}$ by a concatenation
of $\\{{\bm{\tilde{e}}_{1},\ldots,\bm{\tilde{e}}_{G}}\\}$ which is then
followed by a linear transformation. During the backward pass, the gradients
are computed through a Gumbel-Softmax estimator jang2016categorical .

#### Gated Cross-Attention.

To model the interactions among multimodal inputs, we introduce a multimodal
transformer with gated cross-attention to model the cross-modality
relationships. Let $\mathbf{H}_{m}^{l+1}$ be output features at the $l$-th
layer for one modality $m$, and let $n$ be another modality
($m,n\in\\{V,S\\}$). We obtain the features at $(l+1)$-th layer as:

$\displaystyle\mathbf{H}_{m}^{l+1}$
$\displaystyle=f_{\text{LN}}\Big{(}f_{\text{LN}}\big{(}\mathbf{H}_{m}^{l}+f_{\text{Cross-
Att}}^{l}(\mathbf{H}_{m}^{l}|\mathbf{H}_{n}^{l})\big{)}+f_{\text{FF}}^{l}\big{(}f_{\text{LN}}(\mathbf{H}_{m}^{l}+f_{\text{Cross-
Att}}^{l}(\mathbf{H}_{m}^{l}|\mathbf{H}_{n}^{l}))\big{)}\Big{)}$ (2)

where $f_{\text{LN}}$ denotes layer normalization ba2016layer . The feed-
forward sub-layer $f_{\text{FF}}$ in Eq. 2 is further composed of two fully-
connected sub-layers, both wrapped in residual adds and $f_{\text{LN}}$.

The core part of Eq. 2 is the cross-attention $f_{\text{Cross-Att}}(\cdot)$.
Given the intermediate representations $\mathbf{H}_{m}^{l}$ and
$\mathbf{H}_{n}^{l}$, the cross-attention output for modality $m$ is computed
as:

$\displaystyle f_{\text{Cross-Att}}(\mathbf{H}_{m}^{l}|\mathbf{H}_{n}^{l})$
$\displaystyle=[\text{Cross-
Att}^{1}(\mathbf{H}_{m}^{l}|\mathbf{H}_{n}^{l});\ldots;\text{Cross-
Att}^{h}(\mathbf{H}_{m}^{l}|\mathbf{H}_{n}^{l})]{\bm{U}}$ (3)
$\displaystyle\text{Cross-Att}^{i}(\mathbf{H}_{m}^{l}|\mathbf{H}_{n}^{l})$
$\displaystyle={\text{softmax}\left(f^{i}_{q}(\mathbf{H}_{m}^{l})f^{i}_{k}(\mathbf{H}_{n}^{l})^{T}/\sqrt{d}\right)f^{i}_{v}(\mathbf{H}_{n}^{l})}$
(4)

where $f^{i}_{q}(\mathbf{H}_{m}^{l})$, $f^{i}_{k}(\mathbf{H}_{n}^{l})$, and
$f^{i}_{v}(\mathbf{H}_{n}^{l})$ are the query, key, and value calculated by
linear mapping layers for the $i$-th head. $d$ is the model dimension, $h$ is
the number of heads, and ${\bm{U}}$ is the weight matrix that combines the
outputs of the heads.

Considering the substantial diversity of document images and the different
information needs of differing document types, we use a gating mechanism
hu2018squeeze  to dynamically weight the outputs of the visual and textual
branches. Specifically, we feed the concatenated the visual and textual
features to a non-linear network
$f_{\text{Gate}}([\mathbf{H}_{m}^{l+1};\mathbf{H}_{n}^{l+1}])$, which
generates the modality-specific attention weights $\alpha_{m}^{l}$ and
$\alpha_{n}^{l}$, and returns the weights separately to their respective
modality-specific branches to perform element-wise products. We multiply the
features for modality $m$ with its modality-specific attention weight, and
compute the updated feature as:
$\mathbf{H}_{m}^{l+1}=\mathbf{H}_{m}^{l+1}(1+\alpha_{m}^{l})$, same that for
modality $n$.

###  3.2 Training Tasks and Objectives

The full pretraining objective of UDoc (right block in Fig. 1) is defined as:
$\mathcal{L}_{\text{Pretraining}}=\mathcal{L}_{\text{MSM}}+\mathcal{L}_{\text{VCL}}+\mathcal{L}_{\text{VLA}}$.
In the rest of this section, we describe each task in detail.

#### Task #1$\colon$ Masked Sentence Modeling.

This task is similar to the MLM task utilized in BERT. The key difference is
that we mask sentences instead of tokens. During pretraining, each sentence
and RoI of the input document is randomly and independently masked. For the
masked sentence, its token is replaced with a special sentence of [MASK]. The
model is trained to predict the masked sentence feature, based on the unmasked
words and the visual features. The goal is to predict the masked sentence
embeddings based on the contextual information from the surrounding sentences
and image regions, by minimizing the smooth L1 loss girshick2015fast :

$\displaystyle\mathcal{L}_{\text{MSM}}(\Theta)=\sum_{i}\text{smooth}_{L_{1}}(\bm{s}_{i}-f_{\text{UDoc}}(\bm{s}_{i}|\bm{s}_{\setminus
i},\mathbf{\tilde{V}}))$ (5)

where $\Theta$ is the trainable parameters and $f_{\text{UDoc}}(.)$ outputs
the unmasked textual feature, $\bm{s}_{\setminus i}$ is the surrounding
features for the $i$-th input, $\mathbf{\tilde{V}}$ are the image features
with random masking.

#### Task #2$\colon$ Visual Contrastive Learning.

We learn visual feature representations by solving a visual contrastive
learning task which requires estimating the true quantized latent RoI
representation. Given a prediction $\bm{\hat{v}}_{i}\in\bm{\hat{V}}$ for the
masked RoI $\bm{\tilde{v}}_{i}\in\mathbf{\tilde{V}}$, the model needs to
estimate the positive quantized representation $\bm{v}^{Q}_{i}$ in a set of
quantized candidate representations $\mathbf{V}^{Q}$. Good representations are
learned by maximizing the agreement between output representation and
quantized representation of the same RoIs as follows:

$\mathcal{L}_{\text{VCL}}(\Theta)=-\sum_{\bm{\tilde{v}}_{i}\in\mathbf{\tilde{V}}}\Big{(}\log\frac{\exp(\text{sim}(\bm{{\hat{v}}}_{i},\bm{v}^{Q}_{i})/\kappa)}{\sum_{\bm{v}^{Q}_{j}}\exp(\text{sim}(\bm{{\hat{v}}}_{i},\bm{v}^{Q}_{j})/\kappa)}\Big{)}+\lambda\frac{1}{CE}\sum_{c=1}^{C}\sum_{e=1}^{E}p_{c,e}\log{p_{c,e}}$
(6)

where $\text{sim}(\cdot,\cdot)$ computes the cosine similarity between two
vectors, $\lambda$ is a hyperparameter, and $\kappa$ is a temperature scalar.
The second term encourages the model to use the codebook entries more equally.

#### Task #3$\colon$ Vision-Language Alignment.

To enforce the alignment among different modalities, we explicitly encourage
alignment between words and image regions via similarity-preserving knowledge
distillation tung2019similarity . Note that, unlike the text-image alignment
in LayoutLMv2 xu2020layoutlmv2  which splits the image into four regions and
predicts whether the given word is covered or not on the image side, we align
the image and text belonging to the same region. The goal is to minimize the
differences between the pairwise similarities of sentence embeddings and the
pairwise similarities of image region features:

$\displaystyle\mathcal{L}_{\text{VLA}}(\Theta)=\frac{1}{N\times
N}||f_{\text{Norm}}({\mathbf{S}\cdot\mathbf{S}^{\top}})-f_{\text{Norm}}(\mathbf{H}_{V}^{L}\cdot\mathbf{H}_{V}^{L\top})||^{2}_{F}$
(7)

where $\mathbf{S}$ is the unmasked input sentence embeddings,
$\mathbf{H}_{V}^{L}$ is the mapped visual representations of the final layer,
$||\cdot||_{F}$ is the Frobenius norm, and $f_{\text{Norm}}$ performs L2
normalization.

##  4 Experiment

###  4.1 Pretraining UDoc

#### Pretraining corpus.

We build our pretraining corpus based on IIT-CDIP Test Collection 1.0
10.1145/1148170.1148307 , which contains more than 11M scanned document
images. To differentiate pretraining from finetuning, we filter out the
document images of RVL-CDIP harley2015icdar  from IIT-CDIP since it is a
subset of IIT-CDIP, and sample 1M document images as our pretraining corpus.

Table 1: Comparison of the datasets used for pretraining and finetuning process. ‘Box’, ‘Label’, and ‘Text’ indicate the availability of location, label and text annotations for document entities. ‘Tag’ denotes the document class label availability. Dataset | Type | Size | Box | Label | Text | Tag
---|---|---|---|---|---|---
IIT-CDIP 10.1145/1148170.1148307  | Misc | 11M | ✗ | ✗ | ✓ | ✗
RVL-CDIP harley2015icdar  | Misc | 400K | ✗ | ✗ | ✗ | ✓
CORD park2019cord  | Receipt | 1K | ✓ | ✓ | ✓ | ✗
FUNSD 8892998  | Form | 0.2K | ✓ | ✓ | ✓ | ✗
PubLayNet zhong2019publaynet  | Article | 347K | ✓ | ✓ | ✗ | ✗

Table 1 shows the dataset statistics. IIT-CDIP only provides the OCR texts in
XML format. We extract words and their locations by applying EasyOCR easyocr
on document images. As shown in Fig. 3 (a), EasyOCR provides two kinds of
output modes: non-paragraph and paragraph. The paragraph mode groups the non-
paragraph results into text regions. We think document image pretraining
should be treated differently than sequence-based pretraining in NLP, since
the words in the document (2D) are arranged according to spatial layouts,
while the words in NLP corpora are sequential (1D). Considering the special
characteristics of documents (complex layout, multi-pages) and the limited
input length of BERT models, it is not intuitive to formulate the input at the
word level. Hence, we adopt the paragraph-level outputs as the basic input
elements since textual regions provide semantically more meaningful
information than independent words.

Figure 2: Distribution of words per region on RVL-CDIP according to the
categories.

There are some advantages to our design: (1) the region-level design
hierarchically encodes document elements and this facilitates the modeling of
latent relationships at the region level which has higher-level semantics than
the word level. (2) the hierarchical encoding also overcomes the input size
limitation of word-level BERT-based models xu2020layoutlm ; xu2020layoutlmv2 .
Fig. 2 shows the distribution of words per region on RVL-CDIP. It can be seen
that even though we consider region-level input, for some semi-structured
documents, single-words dominate the inputs; this somehow forces UDoc to pay
attention to word-level inputs. Unlike MLM that predicts the masked word, UDoc
predicts the textual embedding of the masked input with MSM.

#### Pretraining setting.

We initialize the sentence encoder $f_{\text{SentEnc}}$ with BERT-NLI-STSb-
base reimers-2019-sentence-bert  pretrained for NLI williams2017broad  and
STS-B cer2017semeval . The ResNet-50 backbone in the image encoder is
pretrained on the PubLayNet training set zhong2019publaynet . All the
parameters (except $f_{\text{SentEnc}}$ and $f_{\text{ImEnc}}$) are randomly
initialized. During pretraining, we freeze the parameters of
$f_{\text{SentEnc}}$ and jointly train the visual encoder and multi-modal UDoc
model in an end-to-end fashion. Such an end-to-end training allows the ConvNet
and Transformer to realize their full potentials in spatial and sequence
modeling for pretraining. UDoc contains 12 layers of gated cross-attention
transformer blocks. We set the hidden size to 768 and the number of heads to
12, the maximum number of regions $N$ to 64, and the maximum input sequence
length for $f_{\text{SentEnc}}$ to 512. The pretraining is conducted on 8
NVIDIA Tesla V100 32GB GPUs with a batch size of 64. It is trained with Adam
optimizer kingma2014adam , with an initial learning rate of 10-5, weight decay
of 10-4, and learning rate warmup in the first 20% iterations.

To learn a useful multimodal representation, random masking is applied to both
textual and visual inputs. For MSM, we set the mask probability
$p_{\text{Mask}}^{s}$ for input sentences to 15%. 80% among the masked
sentences are replaced by special sentence [CLS, MASK, SEP], while 10%
sentences are replaced by random sentences sampled from other documents, and
10% remains unchanged. For VCL, the $\lambda$ is set to 0.1, $\kappa$ is set
to 0.1, the mask probability $p_{\text{Mask}}^{v}$ is set to 7.5% and the
masked RoI features are filled with zeros. The temperature $\tau$ is annealed
from 2.0 to 0.5 by a factor of of 0.999995 at every iteration. We select the
pretraining checkpoint with the lowest $\mathcal{L}_{\text{Pretraining}}$ for
finetuning stage.

Figure 3: Document image samples. The boxes in red/green are OCR bounding
boxes obtained with/without paragraph mode, while the boxes in blue are
officially-provided bounding boxes.

###  4.2 Finetuning Tasks

#### Form Understanding.

Form understanding requires the model to predict the label for each semantic
entity. We use FUNSD 8892998  as the evaluation dataset. It contains 149/50
training/testing images. Fig. 3 (b) shows a sample from FUNSD. Each semantic
entity comprises a list of words, a label, and a bounding box. The officially-
provided OCR texts and bounding boxes are used during training and testing. We
take the semantic entities as input and feed the concatenated visual and
textual output representations to a classifier. We apply cross-entropy loss
for finetuning. The model is finetuned for 100 epochs with a learning rate of
10-5 and batch size of 16. All the parameters except $f_{\text{SentEnc}}$ are
trained. One of question, answer, header or other is predicted for each
semantic entity. We use entity-level F1 score as the evaluation metric.

#### Receipt Understanding.

Receipt understanding requires the model to recognize a list of text lines
with bounding boxes. The performance on this task is evaluated on CORD
park2019cord  dataset. The official data contains 800/100/100 receipts for
training/validation/testing. The receipts are labeled with 30 types of
entities under 4 categories: company, date, address, and total. Like FUNSD, we
feed the concatenated visual and textual output representations to the
classifier. The model is finetuned for 200 epochs with a batch size of 16 and
a learning rate of 10-5. The evaluation metric is entity-level F1 score.

#### Document Classification.

Document classification involves predicting the category for each document
image. We use RVL-CDIP harley2015icdar  as the target dataset. It consists of
320K/40K/40K training/validation/testing images under 16 categories. The OCR
words and bounding boxes are extracted by EacyOCR. To fine-tune UDoc on RVL-
CDIP, we compute the overall representation as an element-wise product between
the visual and textual representations averaged from all sentences/regions,
and learn a classifier on top of the overall representation with cross-entropy
loss. We fine-tune the model for 30 epochs with a batch size of 64 and a
learning rate of 10-5. Classification accuracy over 16 categories is used to
measure model performance.

#### Document Object Detection.

Document object detection involves decomposing a document image into semantic
units. We evaluate the effectiveness of our pretrained visual backbone on
PubLayNet zhong2019publaynet . As shown Fig. 3 (d), the documents in PubLayNet
are scientific articles. PubLayNet consists of 336K/11K training/validation
images with six category labels (text, title, list, figure, and table). We
train Faster-RCNN (F-RCNN) using Detectron2 wu2019detectron2  and initialize
the visual backbone with the pretrained ResNet-50 from UDoc. The model is
trained for 180k iterations with a base learning rate of 0.01 and a batch size
of 8. Mean average precision (MAP) @ intersection over union (IOU) [0.50:0.95]
of bounding boxes is used to measure the performance.

###  4.3 Results and Discussion

#### The importance of multimodal learning.

To study the effect of multimodal learning, we experiment in three different
settings (1) Vision only (V): this setting omits the textual components of
UDoc and adopts multilayer self-attention transformer to learn the visual
representation. (2) Language only (L): this setting omits the visual encoder
and keeps only the textual components. (3) Vision-Language (V+L): this setting
considers both vision and language information. We first train three settings
without pretraining. Table 2 shows consistent improvement across tasks for V+L
over the single-stream baselines (V or L). This demonstrates that our UDoc
model is able to learn important visual-linguistic relationships that benefit
downstream tasks even without pretraining.

In Table 2, we find that visual information dominates the performance of
document classification, while language information contributes a lot to form
understanding and receipt understanding. The results also indicate that
different document tasks rely on different information. For document entity
recognition tasks, language information is more important than visual
features. As can be seen in Fig. 3 (b) and (c), entity recognition is more
word-oriented. On the other hand, document classification is more focused on
global-level understanding. As a result, visual and layout information
contribute a lot to the final prediction of the document classification model.
This matches well with the innate abilities of humans to distinguish between
document types without fully understanding the words. We also observe that
gated cross-attention (V+L) achieves a better performance than the non-gated
version (V+L♯), as its gating mechanism can learn to adaptively determine how
much each modality contributes to the output features.

#### Effect of pretraining tasks.

We analyze the effectiveness of different pretraining settings through
ablation studies over FUNSD, CORD†, and RVL-CDIP, which are representative
document benchmarks. Table 2 ablates the key design choices in pretraining
UDoc. Note that, for CORD†, $\dagger$ indicates that we use different splits
for the ablation study instead of the official ones. For experimental
efficiency, UDoc models evaluated here are trained with 5 epochs on 300k
training corpus. Overall, the pretraining of UDoc consistently improves the
performance over all three downstream tasks. The improvement gains vary among
different tasks.

Table 2: Experimental results and comparison on FUNSD, CORD†, and RVL-CDIP test sets. Pretraining | FUNSD |  CORD† | RVL-CDIP
---|---|---|---
Enable | #Data | Modality | Max #Words | #Param. | Tasks | Epoch | F1 | F1 | Accuracy
✗ | – | V | – | 85M | – | – | 77.49 | 57.08 | 91.35
– | L | – | 153M | – | – | 78.46 | 71.52 | 86.82
– |  V+L♯ | – | 255M | – | – | 80.60 | 95.98 | 92.76
– | V+L | – | 267M | – | – | 83.34 | 96.59 | 92.93
✓ | 300K | V+L | $64\times 512$ | 270M | MSM + MVM | 5 | 84.37 | 97.44 | 93.10
300K | V+L | $64\times 512$ | 272M | MSM + VCL | 5 | 86.87 | 98.70 | 93.59
300K | V+L | $64\times 512$ | 272M | MSM + VCL + VLA | 5 | 87.38 | 98.75 | 93.92
300K | V+L | $64\times 512$ | 274M | MSM + VCL + VLA + REL | 5 | 87.20 | 98.13 | 93.64

We first establish two baselines: MSM+MVM in Table 2 indicates the combination
of masked sentence learning and masked visual feature prediction. Similar to
MSM, for MVM, we freeze the visual backbone and perform masked visual feature
prediction via RoI-feature regression. MSM+VCL jointly trains the visual
backbone end-to-end with contrastive learning. As shown in Table 2, MSM+MVM
achieves better results than the model without pretraining. Furthermore, when
combining VCL together with MSM, consistent performance gains are observed
across all the benchmarks. Among the three finetuning tasks, the improvements
on FUNSD and CORD are bigger than on RVL-CDIP. We think the local context
modeling capability of the ConvNet-based image encoder brings more benefits to
entity recognition, since entities are heavily linked and correlated to their
local surroundings. When MSM, VCL, and VLA are jointly trained, we observe
further performance gains across all the benchmarks. For VCL, instead of
sampling the negatives from the same input document, we also try including the
negative samples from other document images of the same batch. However, we
find that sampling negatives from the entire batch of document images hurt the
performance. This is likely because the negatives from other document images
are easy to distinguish from each other. We also consider the image-text
matching task (Rel) xu2020layoutlmv2 , and combine Rel with MSM+VCL+VLA. It
hurts the performance of all three downstream tasks. We conjecture that the
image-text matching task introduces mismatched pairs of image and OCR texts as
negative examples that potentially hamper the training of other tasks.

#### What if Masked Language Modeling is included?

To study the feasibility of that, we consider MLM during pretraining. Since
the number of words may be very large, we select the tokens by randomly
applying a sliding window (window size 128) across all sequenced OCR words.
Each word is formulated as a single-word sentence ([CLS] [Token] [SEP]). We
randomly mask 15% of those sampled words ([CLS] [MASK] [SEP]) and concatenate
them along with the region-based inputs. During pretraining, we add the word
prediction head on top of UDoc and predict the masked words. We conduct
finetuning experiments on entity recognition tasks (FUNSD and CORD), and find
that such a direct combination hurts the performance: FUNSD: 87.38 (UDoc) vs.
83.76$\downarrow$ (UDoc+MLM), CORD: 98.75 (UDoc) vs. 98.63$\downarrow$
(UDoc+MLM). There are consistent performance drops from adding MLM. One
possible reason is that the RoI features extracted by token bounding boxes
might not be discriminative enough due to the tiny word-level bounding boxes.

Figure 4: For (a) we show the samples from RVL-CDIP. The boxes in orange color are grouped OCR bounding boxes. For (b) we plot the accuracies on 16 classes achieved by different models that are represented by different colors in the bar chart. Table 3: Comparison with state-of-the-art methods. The symbol ${\ddagger}$ implies using Google OCR engine.. Method | Pretraining | FUNSD | CORD | RVL-CDIP
---|---|---|---|---
Source | #Data | Scale | Max #Words | Modality | #Param. | F1 | F1 | Accuracy
$\text{BERT}_{\text{BASE}}$ xu2020layoutlmv2  | – | – | Word | 512 | L | 110M | 60.26 | 89.68 | 89.81
$\text{BERT}_{\text{LARGE}}$ xu2020layoutlmv2  | – | – | Word | 512 | L | 340M | 65.63 | 90.25 | 89.92
$\text{LayoutLM}_{\text{BASE}}$ xu2020layoutlmv2  | IIT-CDIP | 11M | Word | 512 | L | 113M | 78.66 | 94.72 | 94.42
$\text{LayoutLM}_{\text{LARGE}}$ xu2020layoutlmv2  | IIT-CDIP | 11M | Word | 512 | L | 343M | 78.95 | 94.93 | 94.43
$\text{LayoutLMv2}_{\text{BASE}}$ xu2020layoutlmv2  | IIT-CDIP | 11M | Word | 512 | V+L | 200M | 82.76 | 94.95 | 95.25
$\text{LayoutLMv2}_{\text{LARGE}}$ xu2020layoutlmv2  | IIT-CDIP | 11M | Word | 512 | V+L | 426M | 84.20 | 96.01 | 95.64
SelfDoc selfdoc2021  | RVL-CDIP | 320K | Region |  50$\times$512 | V+L | – | 83.36 | – | 92.81
SelfDoc+VGG-16 selfdoc2021  | RVL-CDIP | 320K | Region |  50$\times$512 | V+L | – | – | – | 93.81
TILT-Base powalski2021going  | RVL-CDIP+ | 1.1M | Word | 512 | V+L | 230M | – | 95.11 | 95.25
TILT-Large powalski2021going  | RVL-CDIP+ | 1.1M | Word | 512 | V+L | 780M | – | 96.33 | 95.52
UDoc | IIT-CDIP | 1M | Region |  64$\times$512 | V+L | 272M | 87.96 | 96.64 | 93.96
UDoc∗ | IIT-CDIP | 1M | Region |  64$\times$512 | V+L | 272M | 87.93 | 96.86 |  95.05‡

#### Performance Comparison with SoTA.

We further pretrain UniDoc on 1M document images with 5 epochs and report the
finetuning results in Table 3. UniDoc outperforms previous models on the
official test set of FUNSD and CORD, by a significantly large margin,
demonstrating that our proposed approach is highly effective, partially due to
the end-to-end training of the image encoder that improves the semantic
alignments between images and texts. Note that UniDoc is pretrained on a
subset of IIT-CDIP (1M document images), which is considerably less than the
11M document images used in LayoutLM xu2020layoutlm  and LayoutLMv2
xu2020layoutlmv2 . TILT powalski2021going  builds a 1.1M pretraining corpus by
combining RVL-CDIP, UCSF Industry Documents Library, and Common Crawl. UniDoc
also achieves promising results on document classification. Note that both
LayoutLM v2 and TILT use Microsoft OCR, which is a commercial service with a
stronger OCR performance than EasyOCR, which is used in our experiments. We
find that OCR plays a key role in document classification performance. As
shown in Fig. 4, UniDoc performs the best on the ‘email’ category but worst on
the ‘form’ category. We also report the results with different OCR engines:
93.42 (Tesseract 10.5555/1288165.1288167 ) vs. 93.96 (EasyOCR easyocr ) vs.
94.10 (Google OCR googleocr ). UniDoc with EasyOCR achieves a better
performance than with Tesseract since EasyOCR is powered by an advanced neural
network, while Tesseract is based on less sophisticated techniques. Since
different tasks require task-specific input embeddings to perform well,
instead of finetuning the sentence encoder during pretraining, we explore
unfreezing the sentence encoder during the finetuning stage (named as UniDoc∗)
and report the results in Table 3. Unsurprisingly, we see performance
improvements on several downstream applications. _E.g_., RVL-CDIP: 93.96
(UniDoc) vs. 95.05$\uparrow$ (UniDoc∗). However, this also makes the training
more challenging in terms of computational resources and training time.

Table 4: MAP @ IOU [0.50:0.95] of the document detection models on PubLayNet dev set. Method | Text | Title | List | Table | Figure | mAP
---|---|---|---|---|---|---
F-RCNN (ResNet-101) zhong2019publaynet  | 91.0 | 82.6 | 88.3 | 95.4 | 93.7 | 90.0
M-RCNN (ResNet-101) zhong2019publaynet  | 91.6 | 84.0 | 88.6 | 96.0 | 94.9 | 90.7
F-RCNN (ResNet-50) | 92.2 | 84.4 | 89.5 | 96.5 | 94.5 | 91.4
F-RCNN (UDoc, ResNet-50) | 93.9 | 88.5 | 93.7 | 97.3 | 96.4 | 93.9

Effect of visual backbone. Additionally, we apply the trained visual backbone
to document object detection on PubLayNet. The performance of the F-RCNN on
the validation set is depicted in Table 4. To better compare, we establish two
F-RCNN models with: (1) backbone initialized with ResNet-50 pretrained on
ImageNet; (2) backbone initialized from UDoc’s pretrained visual backbone. It
can be seen that our pretrained backbone outperforms ImageNet-pretrained
backbones. By leveraging UDoc, we can train different variants of the visual
backbone and apply them to document-specific downstream applications, without
relying on incompatible pretrained backbones from other domains (_e.g_.,
natural image). Moreover, the visual backbone of UDoc does not require any
custom layers, and thus any ConvNet architecture can be used in place of
ResNet.

##  5 Conclusion, Limitations, and Future Works

We develop UDoc, a unified pretraining framework for document understanding.
Our model introduces a novel joint training framework that effectively
exploits the visual and textual information during pretraining and finetuning.
We evaluate the UDoc comprehensively on three downstream tasks: form
understanding, receipt understanding, and document image classification.
Extensive empirical analysis demonstrates that the pretraining procedure can
take advantage of multimodal inputs. Also, it can effectively aggregate and
align visual and textual information of document images with the proxy tasks.
This work has a broader impact on document applications. By finetuning the
pretrained UDoc on task-specific data, document processing systems can provide
better results and reduce the expensive data annotations costs. In terms of
negative social impact, the document images used for pretraining may contain
sensitive information and therefore the models trained on such data may
inappropriately leak some private information. To address the privacy leakage,
it is worthwhile to explore the combination of privacy-preserving learning and
self-supervised learning.

There are interesting short- and long-term research directions for UDoc: (1)
we freeze the sentence encoder during pretraining and fine-tuning phases due
to computational constraints. A better document representation can be learned
by jointly training the sentence encoder, visual backbone and cross-attention
encoder in a completely end-to-end fashion. (2) Although impressive
performance has been achieved in document entity recognition tasks such as
form and receipt understanding, the classification accuracy on semi-structured
documents such as forms is still inferior to that of rich-text documents. It
is possible to devise a better method to model the spatial relationship among
words. (3) An interesting direction is to extend UDoc to
multipage/multilingual document pretraining. Additionally, there exist many
text-based labeled document datasets in the NLP domain, such as document
summarization. Can we transfer the knowledge learned from the text-based
document domain to the image-based document domain? How to unify the
pretraining of the pure-text document (1D) and image-based document (2D) in a
single framework is also worth to try. Lastly, the use of different OCR tools
is one of the major sources of inconsistency among the existing document
pretraining works. It is worthwhile and essential to build standardized
pretraining document image datasets with preprovided OCR results. In addition
to scanned documents, using digital PDF as part of the pretraining data is a
direction worth exploring since it provides rich metadata which could be
beneficial for multimodal learning.

## References

  * [1] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy.  Hierarchical attention networks for document classification.  In NAACL, 2016.
  * [2] Jade Goldstein, Vibhu O Mittal, Jaime G Carbonell, and Mark Kantrowitz.  Multi-document summarization by sentence extraction.  In NAACL-ANLP Workshop, 2000.
  * [3] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In ACL, 2019.
  * [4] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou.  Layoutlm: Pre-training of text and layout for document image understanding.  In SIGKDD, 2020.
  * [5] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al.  Layoutlmv2: Multi-modal pre-training for visually-rich document understanding.  In ACL, 2021.
  * [6] Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu.  Selfdoc: Self-supervised document representation learning.  In CVPR, 2021.
  * [7] Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee.  Cord: A consolidated receipt dataset for post-ocr parsing.  In NeurIPS Workshop, 2019.
  * [8] Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis.  Evaluation of deep convolutional nets for document image classification and retrieval.  In ICDAR, 2015.
  * [9] Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, et al.  Recent advances in convolutional neural networks.  Pattern Recognition, 2018.
  * [10] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In NeurIPS, 2019.
  * [11] Jiuxiang Gu, Jason Kuen, Shafiq Joty, Jianfei Cai, Vlad Morariu, Handong Zhao, and Tong Sun.  Self-supervised relationship probing.  In NeurIPS, 2020.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [13] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In ICLR, 2019.
  * [14] Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In AAAI, 2020.
  * [15] Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar Averbuch-Elor.  Read: Recursive autoencoders for document layout generation.  In CVPR Workshop, 2020.
  * [16] Ross Girshick.  Fast r-cnn.  In CVPR, 2015.
  * [17] Easyocr.  https://github.com/JaidedAI/EasyOCR, 2020.
  * [18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [19] Herve Jegou, Matthijs Douze, and Cordelia Schmid.  Product quantization for nearest neighbor search.  PAMI, 2010.
  * [20] Aaron van den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  In NeurIPS, 2017.
  * [21] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In NeurIPS, 2020.
  * [22] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  In ICLR, 2017.
  * [23] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [24] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In CVPR, 2018.
  * [25] Frederick Tung and Greg Mori.  Similarity-preserving knowledge distillation.  In ICCV, 2019.
  * [26] D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard.  Building a test collection for complex document information processing.  In SIGIR, 2006.
  * [27] G. Jaume, H. Kemal Ekenel, and J. Thiran.  Funsd: A dataset for form understanding in noisy scanned documents.  In ICDAR Workshop, 2019.
  * [28] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes.  Publaynet: largest dataset ever for document layout analysis.  In ICDAR, 2019.
  * [29] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In EMNLP, 2019.
  * [30] Adina Williams, Nikita Nangia, and Samuel R Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In NAACL, 2018.
  * [31] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  In ACL, 2017.
  * [32] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2014.
  * [33] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [34] Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka.  Going full-tilt boogie on document understanding with text-image-layout transformer.  In ICDAR, 2021.
  * [35] Anthony Kay.  Tesseract: An open-source optical character recognition engine.  Linux J., 2007(159):2, July 2007.
  * [36] Google cloud vision ocr.  https://cloud.google.com/vision/docs/ocr, 2021.


Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC
BY 4.0).

BIR 2021: 11th International Workshop on Bibliometric-enhanced Information
Retrieval at ECIR 2021, April 1, 2021, online

<EMAIL_ADDRESS>] [orcid=0000-0002-9609-9505,
<EMAIL_ADDRESS>url=http://tmr.liacs.nl, ]

# Improving reference mining in patents with BERT

Ken Voskuil  Suzan Verberne  Leiden Institute of Advanced Computer Science,
Leiden University

(2021)

###### Abstract

In this paper we address the challenge of extracting scientific references
from patents. We approach the problem as a sequence labelling task and
investigate the merits of BERT models to the extraction of these long
sequences. References in patents to scientific literature are relevant to
study the connection between science and industry. Most prior work only uses
the front-page citations for this analysis, which are provided in the metadata
of patent archives. In this paper we build on prior work using Conditional
Random Fields (CRF) and Flair for reference extraction. We improve the quality
of the training data and train three BERT-based models on the labelled data
(BERT, bioBERT, sciBERT). We find that the improved training data leads to a
large improvement in the quality of the trained models. In addition, the BERT
models beat CRF and Flair, with recall scores around 97% obtained with cross
validation. With the best model we label a large collection of 33 thousand
patents, extract the citations, and match them to publications in the Web of
Science database. We extract 50% more references than with the old training
data and methods: 735 thousand references in total. With these
patent–publication links, follow-up research will further analyze which types
of scientific work lead to inventions.

###### Abstract

We

###### keywords:

Patent analysis Information Extraction Reference mining BERT

##  1 Introduction

References in patents to scientific literature provide relevant information
for studying the relation between science and technological inventions. These
references allow us to answer questions about the types of scientific work
that leads to inventions. Most prior work analysing the citations between
patents and scientific publications focuses on the front-page citations, which
are well structured and provided in the metadata of patent archives such as
Google Patents. It has been argued that in-text references provide valuable
information in addition to front-page references: they have little overlap
with front-page references [1] and are a better indication of knowledge flow
between science and patents [2, 3, 1].

In the 2019 paper by Verberne et al. [4], the authors evaluate two sequence
labelling methods for extracting in-text references from patents: Conditional
Random Fields (CRF) and Flair. In this paper we extend that work, by (1)
improving the quality of the training data and (2) applying BERT models to the
problem. We use error analysis throughout our work to find problems in the
dataset, improve our models and analyze the types of errors different models
are susceptible to.

We first discuss the prior work in Section 2. We describe the improvements we
make in the dataset in Section 3, and the new models proposed for this task in
Section 4. We compare the results of our new models with previous results,
both on the labelled dataset and a larger unlabelled corpus (Section 5). We
end with a discussion on the characteristics of the results of our new models
(Section 6), followed by a conclusion.

Our code and improved dataset are released under an open-source license on
github.111https://github.com/kaesve/patent-citation-extraction

##  2 Prior work

Reference analysis in patents has primarily been done using the references
that are listed on the patent’s front page. Patents often contain many more
references in the patent text themselves, but these are more difficult to
extract and analyze because their formatting is not standardized. Verberne et
al. [4] introduce a new labelled dataset consisting of 22 patents and 1,952
hand-labelled references. They apply two sequence labelling methods to the
reference extraction tasks.

Conditional Random Fields (CRF) model sequence labelling problems as an
undirected graph of observed and hidden variables, to find an optimal sequence
of hidden variables (labels) given a sequence of feature vectors [5]. Feature
vectors usually consist of several manually designed heuristics on the level
of individual tokens and small neighborhoods of tokens. For extracting
references, Verberne et al. [4] use a set of $11+6*4$ features. This includes
11 features derived from the current token, ranging from the part-of-speech
(POS) tag (extracted with NTLK), lexical features such as whether the token
starts with a capital or is a number, and pattern-based features to mark
tokens that look like a year or a page number.222The features are similar to
the ones used in https://sklearn-
crfsuite.readthedocs.io/en/latest/tutorial.html It also includes a subset of 6
features for each of the two preceding and following tokens.

As the authors note, CRF has limited capabilities to take context into
account. They chose to compare CRF with the Flair framework, which is better
able to use token contexts. Flair uses a BiLSTM-CRF model in combination with
pre-trained word embeddings [6]. One downside of Flair models is that they are
memory intensive, which limits the maximum sequence length it can process at
once. Where the CRF model can analyze a complete patent at once, the Flair
models required to split sequences up into subsequences of 20 to 40 tokens
[4]. Verberne et al. used the IOB labels during training to prevent splitting
within a reference.

The models were evaluated by measuring precision and recall using cross
validation on the labelled data. CRF performed better than Flair in all
measures except the recall of I-labels. The models were also applied to a
large corpus of 33,338 unlabelled USPTO biotech patents, and the resulting
extracted references were matched against the Web of Science (WoS) database.
Here, Flair performed significantly better. Counting references with a
definitive match in WoS that were not included in the patent front-page, CRF
was able to find 125,631 of such references compared to 493,583 references
found by Flair.

Recent developments in transfer learning have improved the state of the art in
numerous NLP tasks. BERT [7] is a large transformer model that is pre-trained
on a large corpus for multiple language modelling tasks. The resulting model
can be used as a basis for new tasks on different data sets. Even when the
contents of these data sets or the task deviate significantly from the pre-
training corpus and tasks, the pre-training is still beneficial. Several
authors have trained models with the same architecture as BERT on different,
more domain-specific corpora. These include SciBERT [8] and BioBERT [9].

##  3 Improving data quality

While exploring the results of our models, we found that several prediction
errors seemed to be caused by mistakes in the labelled data. These mistakes
result in a more pessimistic evaluation of our models and, more importantly,
could influence the effectiveness of training our models. We noticed two types
of problems; inconsistent or missing labels, and inconsistent tokenization. We
include examples of both kinds of problems below, and describe our attempts to
improve the data quality.

###  3.1 pre-processor inconsistencies

The patent dataset contains text from 22 patents taken from Google Patents.
Labels were added manually by one annotator using the BRAT annotation
tool333http://brat.nlplab.org/, and the text was subsequently transformed into
IOB files using a pipeline consisting of splitting the text into sentences,
then tokens and adding IOB and POS tags. Because tokenization was applied
after annotation, the labels produced by BRAT needed to be aligned with the
produced tokens. In some cases, this was done by recombining tokens. When
comparing the source text with the IOB data, we found that some sequences of
tokens seemed to have been accidentally reordered. An example of this is shown
in Figure 1 After reviewing the pre-processing pipeline we were able to find
the likely cause of this problem. We chose to replace this pipeline with a
simpler procedure, that does not do sentence splitting or combining of tokens.
Besides sentence boundaries, our method also ignores paragraph boundaries and
white space in general.

(Eskildsen et al., Nuc. Acids Res. 31:3166-3173, 2003;
---
Kakuta et al., J. Interferon & Cytokine Res. 22:981-993, 2002.)
(a) Original text Token | Eskildsen | et | al., | Nuc. | Acids | Res. | … | Res. | 22:981-993, | 2002.)(
---|---|---|---|---|---|---|---|---|---|---
Label | B | I | I | I | I | I | … | I | I | O
(b) Original tokenization Token | ( | Eskildsen | et | al. | , | Nuc | . | Acids | … | Res | . | 22:981-993 | , | 2002 | . | )
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Label | O | B | I | I | I | I | I | I | … | I | I | I | I | I | I | O
(c) New tokenization

Figure 1: Comparing the original with the new tokenization. Note that
punctuation marks are now treated as separate tokens. Also note the labelling
of the braces, and the wrongly labelled last token in the original
tokenization. Example taken from patent US8133710B2.

###  3.2 Inconsistent labelling

After improving the pre-processing, we still found examples of label
inconsistencies. Moreover, our models found several references that were not
included in the annotations. Finally, we found multiple instances of
references to patents and other non-academic literature. These are often hard
to distinguish from scientific literature references. We manually looked at
each difference between predicted and expected labels, and changed the
annotations where necessary. We repeated this process several times, with
different models and after retraining on the updated data. In this process, we
labelled 330 new references, resulting in a total of 2,318 references and
32,359 (I)nside tokens. We chose to include patent references when they
included author names or titles, and other non-literature references, when the
reference shares the format of a literature reference. This simplifies the
task, as the model does not have to disambiguate references by their type.
Since these extracted non-literature references will not match with the
publications in WoS, they will be filtered out in the next step of the
pipeline.

While we think this process has improved the data quality significantly, our
method does introduce biases in the training and evaluation of our models. By
only fixing labelling mistakes that our models find, we may overlook
unlabelled references that our models miss. This leads to an overestimation in
our evaluation, and biases in our model due to the feed back loop in the
training process. By using multiple different models for finding incorrect
labels, we mitigate the effect to some extent. Beside an intrinsic evaluation
using the labelled data, we will also evaluate our models on an extrinsic task
using unlabelled data. This allows us to still compare our model performance
with previous results, without biases in the dataset or overestimations.

##  4 Extracting references with BERT, BioBERT and SciBERT

We compare three different pre-trained models for extracting references from
our data set; BERT, BioBERT and SciBERT. Since our data set consists of
patents from the biomedical domain, we expect that these more domain-specific
pre-training corpora will have a positive effect on our task. Before comparing
the results between these models, we describe our method for fine tuning the
pre-trained model for reference extraction.

###  4.1 pre-processing

BERT-based models have two characteristics that require additional pre-
processing of our dataset. BERT uses its own neural subword tokenization. Our
dataset is already tokenized into words, as described above, so we apply the
BERT tokenizer to each token in our dataset. Transformer-based models such as
BERT also work on fixed sequence lengths, using padding for shorter sequences,
and are memory intensive. The models we train use a maximum sequence length of
64 tokens, limited by the memory available. Though this can be configured to
be higher depending on the available hardware and the size of the model, it is
infeasible to apply these models on complete patents, which can contain tens
of thousands of subword tokens. There are several common strategies to divide
text into shorter sequences. A natural approach is to use paragraph or
sentence splitting. We found this insufficient, as many sentences in our data
set run for much longer than the limit of 64 tokens. Our data set contains not
only long sentences; even references, the entities we are looking to extract,
can be longer than 64 tokens. Because of this observation we decide to not use
any semantic or structural information in splitting our text, except for our
original token boundaries.

Our BERT specific pre-processing can be summarized in the following steps:

  1. 1.

Collect the sequence of tokens $T$ and their respective labels $L$ for a given
patent

  2. 2.

Create two empty lists $T^{\prime}$ and $L^{\prime}$

  3. 3.

Add the sequence start token to $T^{\prime}$

  4. 4.

While there are tokens left in $T$:

    1. (a)

Get the next token $t$ and label $l$

    2. (b)

Use the word tokenizer to get sub tokens $t^{\prime}_{1},...,t^{\prime}_{n}$

    3. (c)

If $|T^{\prime}|+n+1$ is larger than our limit of 64 tokens or when we reach
the end of the document, add the sequence end token to $T^{\prime}$, pad both
sequences and add them to the data set. Set $T^{\prime}$ and $L^{\prime}$ to
new empty lists

    4. (d)

Add $t^{\prime}_{1},...,t^{\prime}_{n}$ to $T^{\prime}$, add $l$ to
$L^{\prime}$

We note that the retokenization changes our task from a one-to-one to a many-
to-many sequence-to-sequence task, as there could now be multiple subword
tokens associated with one label. Another implication of these pre-processing
steps, is that the entities that we seek to extract can be split across
multiple sequences of 64 subword tokens. As mentioned earlier, we have a total
of 2,318 references and 32,359 tokens labelled as (I)nside. This gives us a
total of $34,677$ reference tokens (labelled either B or I). We find that the
average reference contains $\frac{34,677}{2,318}\approx 15$ word tokens, and
thus at least that many subword tokens. We can expect a large number of
references to be split across two or more sequences. We expect that this could
have a significant effect on the performance of our models, as the model will
not always have access to the context of a reference.

###  4.2 Training the BERT models

We fine-tunet three different BERT models to our labelled data: BERT-base,
bioBERT, and sciBERT (all cased). We used To fine-tune the BERT models, we use
the open source BERT implementation by
HuggingFace444https://huggingface.co/transformers/model_doc/bert.html##bertfortokenclassification,
with a token classification head consisting of a single linear layer. In the
case that an input sequence is shorter than 64 tokens (which only occurs at
the end of a patent), we mask out the loss for the output past the input
sequence. We train the models for three epochs through our training data, with
a batch size of 32.555We published the trained models on
https://github.com/kaesve/patent-citation-extraction

##  5 Results

###  5.1 Intrinsic evaluation

We evaluate our models using a leave-one-out training scheme. For each patent
in the data set we train a new model using the other 21 patents as the
training data. Aside from the maximum sequence length, we used the default
hyperparameter configurations provided by the chosen framework. We evaluate on
both the original and updated dataset.

Table 1: Comparing three BERT models and two baseline models using leave-one-out evaluation on 22 patents (micro averages). Best results are printed in boldface. * indicates results as reported by [4].  | Dataset | Label |  Precision |  Recall |  F1 | Support
---|---|---|---|---|---|---
BERT | Original | B |  0.849 |  0.927 |  0.886 | 1,988
|  | I |  0.896 |  0.955 |  0.924 | 28,449
SciBERT | Original | B |  0.865 |  0.925 |  0.894 | 1,988
|  | I |  0.898 |  0.944 |  0.920 | 28,449
BioBERT | Original | B |  0.843 |  0.929 |  0.884 | 1,988
|  | I |  0.894 |  0.952 |  0.922 | 28,449
BERT | Updated | B |  0.934 |  0.948 |  0.941 | 2,318
|  | I |  0.985 |  0.972 |  0.978 | 32,359
SciBERT | Updated | B |  0.947 |  0.954 |  0.950 | 2,318
|  | I |  0.986 |  0.976 |  0.981 | 32,359
BioBERT | Updated | B |  0.944 |  0.957 |  0.951 | 2,318
|  | I |  0.986 |  0.974 |  0.980 | 32,359
CRF* | Original | B |  0.890 |  0.824 |  0.856 | 1,988
|  | I |  0.914 |  0.870 |  0.891 | 28,449
CRF | Updated | B |  0.922 |  0.893 |  0.907 | 2,318
|  | I |  0.964 |  0.938 |  0.951 | 32,359
Flair* | Original | B |  0.762 |  0.702 |  0.731 | 1,988
(Flair embeddings) |  | I |  0.814 |  0.890 |  0.850 | 28,449
Flair* | Original | B |  0.722 |  0.647 |  0.682 | 1,988
(Glove embeddings) |  | I |  0.789 |  0.840 |  0.814 | 28,449

Table 1 shows the results of evaluating the models on the labelled data using
leave-one-out validation. We also include the results of [4] as a baseline,
however, the results are not directly comparable as they used five-fold cross-
validation for evaluation. Their models therefore were trained on less data.
Finally, we include the results of applying the original CRF implementation on
our updated dataset, using the same leave-one-out validation strategy.

We see that our new models perform reasonably well on the original dataset.
Comparing to the baseline methods, we see that the BERT models consistently
achieve a much higher recall. This is especially useful for the WoS matching
task, as was discussed earlier.

When we compare the results of our models obtained with the updated dataset to
those obtained with the original data, we see that the changes in the dataset
lead to improvements in every metric. Especially in the precision column, we
see a large jump in quality. This jump is in part the direct result of our
relabelling process. Most changes in the dataset concerned changing labels
from ‘O’ to ‘I’ or ‘B’ tokens, where our models found references that were
missed during labelling.

Comparing the BERT-based models with each other, we find that the differences
are small. With the updated data the SciBERT and BioBERT models seem to
perform slightly better than the plain BERT model.

Finally, we can compare the results of the CRF model on the original and
updated dataset. We again see a clear jump in performance. This comparison
does suffer from the training bias and different evaluation strategy mentioned
earlier. Furthermore, the CRF model uses features designed for the original
dataset. As we changed the tokenization process, this means that some of the
pattern based features do not work as intended. Still, we think the results do
show that the changes to the dataset make this task easier.

###  5.2 Extrinsic evaluation

We also apply each model to an unlabelled data set of 33,338 patents [4]. For
this application, the models are trained on the complete labelled data set.
The references produced by these models are matched against the Web of Science
database, using the same procedures as reported in [4].

From the set of 33,338 patents, we extract references to papers published in
the years 1980–2010 (the ‘focus years’). This results in a list of extracted
references. We parse them into separate fields: first author, second author,
year, journal title, volume/issue, and page numbers. Then we match those
fields to publications in the database. If we find a non-ambiguous match for a
subset of the fields, we count this as a ‘definite match’ [4].

Figure 2: The number of found, parsed and matched references in the unlabelled
dataset. The results for CRF and Flair were obtained for the original dataset.
In-text citations are citations that only occur in the text and are not
included in the patent front page. The ‘focus years’ are the years 1980–2010,
for which we extract references. A ‘definite match’ is a non-ambiguously
matched publication to the reference. (also see [4] for details about the
matching procedure)

The results are displayed in Figure 2. There is a clear difference between the
new BERT-based models and the previous CRF and Flair models, but these results
are not directly comparable since CRF and Flair were trained on the original
data. The figure also shows that the three BERT models perform nearly
identical to each other. As with the results from our intrinsic evaluation,
SciBERT seems to perform better than the other two BERT models by a small
margin.

We found that our models do not always produce clean sequences of IOB tokens;
sometimes the beginning is not marked as a B, or a word in the middle of a
reference is labelled as O. We extract references from sequences of I tokens
starting with a B token or an I token preceded by an O token, and ending
before an O or B token. In the case that our model misses a word in the middle
of a reference, this means that we split this reference in two references
during extraction. Our matching script reports unique matches per patent, so
this does not lead to double-counting references. On the other hand, it could
mean that neither part of the split reference contains enough information to
make a definite match in the WoS database.

##  6 Discussion

Our results show that our BERT-based models outperform both CRF and Flair,
especially after improving the training data. While the increased precision
and recall is likely overestimated in our intrinsic evaluation, the new models
also perform better in our extrinsic evaluation, which does not have the same
training biases. Our models were able to extract roughly 240,000 more
references that could be matched with the WoS database from the unlabelled
data than Flair could, an increase of almost 50%.

The difference between the numbers of matched publications found by CRF and
BERT is striking given the small differences in quality of the models measured
with leave-one-out validation (Table 1). This can for a large part be
explained by the improved training data, but also by the higher recall for the
BERT models. In addition, we investigate two characteristics of errors made by
our models, and show the differences between BERT and CRF. We focus on
prediction errors _within_ references, as these have the largest effect on the
downstream task of parsing references. Specifically, we look at cases where
the model labels a token as O when that token is labelled as B or I in the
ground truth.

Figure 3: The relative positions of predicted ‘O’ labels within references,
per model.

Figure 3 shows the relative position of errors within references. This data
was captured during the leave-one-out evaluation. One major difference between
BERT and CRF-based models is that CRF explicitly learns ordered patterns in
sequences. We would expect CRF models to make errors by starting or ending a
label sequence too early or too late, but we do not usually expect errors to
occur in the middle of a reference, as CRF learns that an I never follows an
O. Without this structural prior, we expect the errors to occur more uniformly
across the references for the BERT models. The histograms seem to confirm
these intuitions. Leaving out mistakes in the first word, we see that the
distribution for especially the SciBERT and BioBERT models seem uniform. The
CRF model shows a clear drop in the first third of the distribution, and a
steady increase in the second half.

By manually looking at references where CRF predicts an O close to the middle,
we found we could categorize these mistakes almost completely in two groups:
CRF only labelled the first or last few tokens as part of the reference, or
the reference is very long and CRF finds two references at beginning and end
of the reference. In both scenarios CRF does produce coherent sequences of a B
label followed by I labels. On the other hand, our BERT models sometimes do
not predict a B at all, or in the wrong place. The models are also prone to
missing an I label in the middle of a reference.

Figure 4 is another way to visualize this difference. Here we plot the lengths
of sequences of O’s found within references. The median error sequence length
is one or two for the BERT models, and four for CRF. In other words, BERT
models not only make fewer mistakes than CRF, but the mistakes are smaller on
average, and more uniformly spread across the reference. We speculate that
this helps with the ultimate task of parsing and matching the references. CRF
errors almost always include the first or last few tokens, which often contain
important information for parsing the reference, such as the publication year
and the author names.

Figure 4: The relative positions of predicted ‘O’ labels within references,
per model.

##  7 Conclusion

We applied BERT-based models to extract references from patent texts. We found
that these models achieve better recall than CRF and Flair. We use an external
database of publications to match these references, which means that recall is
more important than precision, as imprecisions will be resolved during
matching. During the development of our models, we found that the original
dataset for this task had errors in labelling and pre-processing. We used our
models interactively to find these mistakes, and repaired them.

We find that the improved training data leads to a large improvement in the
quality of the trained models. In addition, the BERT models beat CRF and
Flair, with recall scores around 97% obtained with cross validation. Our
models were also applied to a large unlabelled dataset, and were able to
extract 50% more references than previous methods.

We also show that BERT models are prone to a different kind of errors than CRF
models. Combining these methods could potentially lead to a stronger model. We
think that the limited maximal sequence size that BERT can handle affects its
performance, due to the average length of references. Recent work focuses on
modifying the attention architecture underlying BERT to better accommodate
longer sequences. This includes new models such as the Reformer, Longformer,
Linformer, Big Bird and the Performer [10]. We think these models could
achieve even better results, with little modification to our method.

## References

  * Bryan et al. [2019] K. A. Bryan, Y. Ozcan, B. N. Sampat, In-Text Patent Citations: A User’s Guide, Technical Report, National Bureau of Economic Research, 2019.
  * Nagaoka and Yamauchi [2015] S. Nagaoka, I. Yamauchi,  The use of science for inventions and its identification: Patent level evidence matched with survey,  Research Institute of Economy, Trade and Industry (RIETI) (2015).
  * Bryan and Ozcan [2016] K. A. Bryan, Y. Ozcan,  The impact of open access mandates on invention,  Mimeo, Toronto (2016).
  * Verberne et al. [2019] S. Verberne, I. Chios, J. Wang,  Extracting and matching patent in-text references to scientific publications,  in: Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019), 2019, pp. 56–69.
  * Wallach [2004] H. M. Wallach,  Conditional random fields: An introduction,  Technical Reports (CIS) (2004) 22\.
  * Akbik et al. [2019] A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, R. Vollgraf,  Flair: An easy-to-use framework for state-of-the-art nlp,  in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), 2019, pp. 54–59.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  Bert: Pre-training of deep bidirectional transformers for language understanding,  in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186.
  * Beltagy et al. [2019] I. Beltagy, A. Cohan, K. Lo,  Scibert: Pretrained contextualized embeddings for scientific text,  CoRR abs/1903.10676 (2019). URL: http://arxiv.org/abs/1903.10676. arXiv:1903.10676.
  * Lee et al. [2019] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, J. Kang,  Biobert: a pre-trained biomedical language representation model for biomedical text mining,  CoRR abs/1901.08746 (2019). URL: http://arxiv.org/abs/1901.08746. arXiv:1901.08746.
  * Tay et al. [2020] Y. Tay, M. Dehghani, D. Bahri, D. Metzler,  Efficient transformers: A survey,  arXiv preprint arXiv:2009.06732 (2020).