#  Unifying Vision, Text, and Layout for Universal Document Processing

Zineng Tang1,2 Ziyi Yang2 Guoxin Wang3 Yuwei Fang2 Yang Liu2
Chenguang Zhu2 Michael Zeng2 Cha Zhang3 Mohit Bansal1∗
1University of North Carolina at Chapel Hill
2Microsoft Azure Cognitive Services Research
3Microsoft Azure Visual Document Intelligence Corresp. authors:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We propose Universal Document Processing (UDOP), a foundation Document AI
model which unifies text, image, and layout modalities together with varied
task formats, including document understanding and generation. UDOP leverages
the spatial correlation between textual content and document image to model
image, text, and layout modalities with one uniform representation. With a
novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-
domain downstream tasks into a prompt-based sequence generation scheme. UDOP
is pretrained on both large-scale unlabeled document corpora using innovative
self-supervised objectives and diverse labeled data. UDOP also learns to
generate document images from text and layout modalities via masked image
reconstruction. To the best of our knowledge, this is the first time in the
field of document AI that one model simultaneously achieves high-quality
neural document editing and content customization. Our method sets the state-
of-the-art on 9 Document AI tasks, e.g., document understanding and QA, across
diverse data domains like finance reports, academic papers, and websites. UDOP
ranks first on the leaderboard of the Document Understanding Benchmark
(DUE).111Code and models:
https://github.com/microsoft/i-Code/tree/main/i-Code-Doc

Figure 1:  UDOP unifies vision, text, and layout through vision-text-layout
Transformer and unified generative pretraining tasks including vision task,
text task, layout task, and mixed task. We show the task prompts (left) and
task targets (right) for all self-supervised objectives (joint text-layout
reconstruction, visual text recognition, layout modeling, and masked
autoencoding) and two example supervised objectives (question answering and
layout analysis).

##  1 Introduction

Document Artificial Intelligence studies information extraction,
understanding, and analysis of digital documents, e.g., business invoices, tax
forms, academic papers, etc. It is a multimodal task where text is
structurally embedded in documents, together with other vision information
like symbols, figures, and style. Different from classic vision-language
research, document data have a 2D spatial layout: text content is structurally
spread around in different locations based on diverse document types and
formats (e.g., invoices vs. tax forms); formatted data such as figures, tables
and plots are laid out across the document. Hence, effectively and efficiently
modeling and understanding the layout is vital for document information
extraction and content understanding, for example, title/signature extraction,
fraudulent check detection, table processing, document classification, and
automatic data entry from documents.

Document AI has unique challenges that set it apart from other vision-language
domains. For instance, the cross-modal interactions between text and visual
modalities are much stronger here than in regular vision-language data,
because the text modality is visually-situated in an image. Moreover,
downstream tasks are diverse in domains and paradigms, e.g., document question
answering [44], layout detection[55], classification [13], information
extraction [27], etc. This gives rises to two challenges: (1) how to utilize
the strong correlation between image, text and layout modalities and unify
them to model the document as a whole? (2) how can the model efficiently and
effectively learn diverse vision, text, and layout tasks across different
domains?

There has been remarkable progress in Document AI in recent years [36, 51, 53,
1, 10, 11, 28, 52, 25, 50, 29, 35, 23, 12, 15, 47, 16]. Most of these model
paradigms are similar to traditional vision-language frameworks: one line of
work [36, 51, 53, 1, 11, 28, 52, 50, 29, 35] inherits vision-language models
that encode images with a vision network (e.g., vision transformer) and feed
the encodings to the multimodal encoder along with text [43, 26, 17, 46];
another line of work uses one joint encoder [21, 45] for both text and image
[16]. Some models regard documents as text-only inputs [10, 25, 12, 15, 47].
In these works, the layout modality is represented as shallow positional
embeddings, e.g., adding a 2D positional embedding to text embeddings. The
strong correlation between modalities inherent in document data are not fully
exploited. Also to perform different tasks, many models have to use task-
specific heads, which is inefficient and requires manual design for each task.

To address these challenges, we propose Universal Document Processing (UDOP),
a foundation Document AI model that unifies vision, text, and layout and
different document tasks. Different from regarding image and document text as
two separate inputs in previous works, in UDOP we propose to model them with
the uniform layout-induced representation (Sec. 2.1): in the input stage, we
add embeddings of text tokens with the features of the image patch where the
tokens are located. This simple and novel layout-induced representation
greatly enhances the interaction between the text and vision modalities.

Besides the layout-induced representation, to form a uniform paradigm for
different vision, text, layout tasks, UDOP first builds a homogeneous
vocabulary for texts and document layout that converts layout, i.e. bounding
boxes, to discretized tokens. Second, we propose Vision-Text-Layout (VTL)
Transformer, consisting of a modality-agnostic encoder, text-layout decoder
and vision decoder. VTL Transformer allows UDOP to jointly encode and decode
vision, text, and layout. UDOP unites all downstream tasks with a sequence-to-
sequence generation framework.

Besides the challenges of modalities unification and task paradigms discussed
above, another issue is previous works utilized self-supervised learning
objectives that were originally designed for single-modality learning, e.g.,
masked language modeling, or classical vision-language pretraining, e.g.,
contrastive learning. We, on the other hand, propose novel self-supervised
learning objectives designed to allow holistic document learning, including
layout modeling, text and layout reconstruction, and vision recognition that
account for text, vision and layout modeling together (Sec. 3). Besides
sequential generation, UDOP can also generate vision documents by leveraging
masked autoencoders (MAE) [14] by reconstructing the document image from text
and layout modalities. With such generation capacity, UDOP is the first
document AI model to achieve high-quality customizable, joint document editing
and generation.

Finally, our uniform sequence-to-sequence generation framework enables us to
conveniently incorporate all major document supervised learning tasks to
pretraining, i.e., document layout analysis, information extraction, document
classification, document Q&A, and Table QA/NLI, despite their significant
differences in task and data format. In contrast, pretraining in previous
document AI works is constrained to unlabeled data only (or using one single
auxiliary supervised dataset such as FUNSD [53]), while abundant labeled
datasets with high quality supervision signals are ignored due to the lack of
modeling flexibility. Overall, UDOP is pretrained on 11M public unlabeled
documents, together with 11 supervised datasets of 1.8M examples. Ablation
study in Table 4 shows that UDOP only pretrained with the proposed self-
supervised objectives exhibits great improvements over previous models, and
adding the supervised data to pretraining further improves the performance.

We evaluate UDOP on FUNSD [18], CORD [33], RVL-CDIP [13], DocVQA [32], and
DUE-Benchmark [2]. UDOP ranks the 1st place on the DUE-Benchmark leaderboard
with 8 tasks, and also achieves SOTA on CORD, RVL-CDIP, and DocVQA, hence
making UDOP a powerful and unified foundation Document AI model for diverse
document understanding tasks,

To summarize, our major contributions include:

1\. Unified representations and modeling for vision, text and layout
modalities in document AI.

2\. Unified all document tasks to the sequence-to-sequence generation
framework.

3\. Combined novel self-supervised objectives with supervised datasets in
pretraining for unified document pretraining.

4\. UDOP can process and generate text, vision, and layout modalities
together, which to the best of our knowledge is first one in the field of
document AI.

5\. UDOP is a foundation model for Document AI, achieving SOTA on 9 tasks with
significant margins.

##  2 Universal Document Processing

We introduce UDOP, a novel document AI framework with unified learning
objectives and model architecture for text, vision, and layout as shown in
Figure 1. In this section, we will concretely discuss the proposed Vision-
Text-Layout Transformer in UDOP, and will introduce the unified generative
pretraining method in the next section. In document processing, given a
document image ${\bm{v}}$, typically optical character recognition (OCR) is
used on ${\bm{v}}$ to extract text tokens $\\{s_{i}\\}$ in the document and
their bounding boxes $\\{(x^{1}_{i},y^{1}_{i},x^{2}_{i},y^{2}_{i})\\}$, i.e.,
the layout information for each token. $(x^{1}_{i},y^{1}_{i})$ and
$(x^{2}_{i},y^{2}_{i})$ respectively represent the coordinates of the left-
upper and right-bottom corner of the bounding box. Thus, suppose we have $M$
word tokens, the input is the triple,
$({\bm{v}},\\{s_{i}\\}^{M}_{i=1},\\{(x^{1}_{i},y^{1}_{i},x^{2}_{i},y^{2}_{i})\\}^{M}_{i=1})$.
Figure 1 shows an example document (left) and downstream tasks (right).

###  2.1 A Unified Vision, Text, and Layout Encoder

We fuse the vision, text, and layout modalities in the input stage using one
unified transformer encoder. For traditional vision-text data, the text
modality is usually the high-level description of the corresponding image or
task prompt (e.g., question). While in document images, text is embedded
inside the image, i.e., text and image pixels have one-to-one correspondence.
To leverage this correspondence, we propose a new Vision-Text-Layout (VTL)
Transformer architecture to dynamically fuse and unite the image pixels and
text tokens based on the layout information.

Concretely, given the document image ${\bm{v}}\in\mathbb{R}^{H\times W\times
C}$, $M$ word tokens $\\{s_{i}\\}_{i=1}^{M}$ inside the image and the
extracted layout structure
$\\{(x^{1}_{i},y^{1}_{i},x^{2}_{i},y^{2}_{i})\\}_{i=1}^{M}$, we first
partition ${\bm{v}}$ into $\frac{H}{P}\times\frac{W}{P}$ image patches, where
each patch is of size $P\times P\times C$. We then encode each patch with a
$D$-dim vector and group all patch embeddings into a sequence of vectors
$\\{{\bm{v}}_{i}\in\mathbb{R}^{D}\\}_{i=1}^{N}$ where
$N=\frac{H}{P}\times\frac{W}{P}$. Text tokens are also converted to numerical
$D$-dim embeddings $\\{{\bm{s}}_{i}\\}_{i=1}^{M}$ by vocabulary look-up.

#### Layout-Induced Vision-Text Embedding.

Figure 2:  Layout-induced vision-text embedding.

Next, we build a unified representation for vision, text, and layout as shown
in Figure 2. We define the layout indicator function $\phi$ of image patch and
token embeddings as follows:

$\phi({\bm{s}}_{i},{\bm{v}}_{j})=\begin{cases}&1,\,\,\,\text{if the center of
${\bm{s}}_{i}$'s bounding box}\\\ &\,\,\,\,\,\,\,\,\,\text{is within the image
patch ${\bm{v}}_{j}$.}\\\ &0,\,\,\,\text{otherwise}.\end{cases}$ (1)

Then for each text token embedding ${\bm{s}}_{i}$, the joint representation is
the sum of its image patch feature222Some text token like manually crafted
prompts have no locations. So, we set their layout bounding boxes to be
$(0,0,0,0)$, i.e., they fall into a pseudo image patch. and the text feature:

${\bm{s}}^{\prime}_{i}={\bm{s}}_{i}+{\bm{v}}_{j},\,\text{where}\,\,\phi({\bm{s}}_{i},{\bm{v}}_{j})=1.$

For image patches ${\bm{v}}_{j}$ without any text tokens, i.e. $\forall
i,\,\phi({\bm{s}}_{i},{\bm{v}}_{j})=0$, the joint representation,
${\bm{v}}^{\prime}_{j}$ is itself:

${\bm{v}}^{\prime}_{j}={\bm{v}}_{j}.$

Note we do not have a designated joint representation for image patch
containing tokens, since features of these image patches are already
integrated with the text embeddings. Then $\\{{\bm{s}}_{i}^{\prime}\\}$ and
$\\{{\bm{v}}_{j}^{\prime}\\}$ are fed into the VTL transformer encoder. These
joint representations greatly enhance the interaction between vision, text and
layout in the model input stage by explicitly leveraging their spatial
correlations.

To further unify layout and text representation, inspired by the recent
progress in generative object detection [4, 48], we discretize the layout
modality, i.e., continuous coordinates text bounding box, to layout tokens.
Suppose we have bounding box $(x^{1}_{i},y^{1}_{i},x^{2}_{i},y^{2}_{i})$
normalized in $[0,1]$. The resulting layout token will be each coordinate
multiplied by vocabulary size and then rounded to nearest integer. For
example, if we have bounding box $(0.1,0.2,0.5,0.6)$ with layout vocabulary
size $500$, the layout tokens will then be <50><100><250><300>. Layout tokens
can be conveniently inserted into text context, and elegantly used for layout
generation tasks (e.g., location detection). More details are discussed in
Section 3.

Table 1: A summary of all generative pretraining objectives with task names,
task prompts, and task targets.

Self-Supervised Tasks |  Task Prompts |  Task Targets
---|---|---
Layout Modeling |  Layout Modeling. <layout_0> Ship Date to Retail </layout_0> Week of March 14, 1994 |  <layout_0> <100><350><118><372>
Visual Text Recognition |  Visual Text Recognition. <text_0> <100><350><118> <372> </text_0> to Retail: Week of March 14, 1994 |  <text_0> Ship Date
Joint Text-Layout Reconstruction |  Joint Text-Layout Reconstruction. <text_layout_0> to Retail: Week of March 14, 1994 |  <text_layout_0> Ship Date <100> <350><118><372>
Masked Image Reconstruction |  Masked Image Reconstruction. Ship Date to Retail: Week of March 14, 1994 |  [Pixels of the original image]
Supervised Tasks |  |
Classification |  Document Classification. Ship Date to Retail: Week of March 14, 1994 |  Memo.
Layout Analysis |  Layout Analysis. Paragraph. |  Paragraph <82><35><150><439>
Information Extraction |  Information Extraction. Ship Date to Retail |  Week of March 14, 1994
Question Answering |  Question Answering. What is the ship year? |  1994
Document NLI |  Document Natural Language Inference. Ship Date to Retail: Week of March 14, 1994 |  Entailment.

#### Position Bias.

We follow TILT [35] to encode 2D text token position as 2D relative attention
bias, similar to the relative attention bias used in T5. However, unlike T5,
TILT, or transformer models in previous Document AI works [16, 35], we do not
use 1D position embeddings in VTL transformer encoder, since the joint
embedding and the 2D position bias already incorporate the layout structure of
the input document.

#### Modality-Specific Model Variant.

In the field of multimodal learning, a common model architecture is the two-
tower model, where vision and text are encoded by two modality-specific
encoders respectively [37, 54]. Concretely, instead of having one unified
encoder, we separately use a text encoder (to encode both text and layout
tokens) and a vision encoder. Position bias are used in both encoders to
represent layout information following previous works. We name this variant
UDOP-Dual. More details are available in and Section 3.1 and Section 4.

###  2.2 Vision-Text-Layout Decoder

As introduced in the previous section, the VTL encoder is able to compactly
and jointly encode vision, text, and their layout. To perform various document
generative tasks (will be discussed in Section 3), the VTL decoder is designed
to jointly generate all vision, text, and layout modalities.

The VTL decoder consists of a text-layout decoder and a vision decoder, as
shown in Figure 1 (middle). The text-layout decoder is a uni-directional
Transformer decoder to generate text and layout tokens in a sequence-to-
sequence manner. For the vision decoder, we adopt the decoder of MAE [14] and
directly generate the image pixels with text and layout information. Details
of the image decoding process will be discussed in the segment “Masked Image
Reconstruction with Text and Layout ” of Section 3.1. Both text-layout decoder
and vision decoder will cross-attend to the VTL encoder (in the case of UDOP-
Dual, which has two modality-specific encoders, decoders cross-attend with the
concatenation of two encoders’ outputs). Information such as model
configurations are presented in Section 4.1.

##  3 Unified Generative Pretraining

To unify across different training objectives and datasets, we create a
universal generative task format with task prompt. We pretrain UDOP on large-
scale documents with and without human labels. We summarize the tasks prompts
and targets in Table 1 which includes all self-supervised and supervised tasks
respectively in upper and lower blocks.

###  3.1 Self-Supervised Pretraining Tasks

We propose various innovative self-supervised learning objectives for
unlabeled documents. The unlabeled document contains OCR text inputs with
token-level bounding boxes and the document image. In the rest of this
subsection, we use the following input text as example:
“Ship Date to Retail: Week of March 14, 1994”

(1) Joint Text-Layout Reconstruction requires the model to reconstruct the
missing texts and locate them in the document image. Concretely, we mask a
percentage of text tokens and ask the model to both the tokens and their
bounding boxes (i.e. layout tokens). E.g., assume masking “Ship Date” and
“of”, the input sequence and target sequence is given below:

Input Sequence:
---
“Joint Text-Layout Reconstruction. <text_layout_0>
to Retail: Week <text_layout_1> March 14, 1994”
Target Sequence:
“<text_layout_0> Ship Date <100><350><118><372>
<text_layout_1> of <100><370><118><382>”

Here <text_layout_0> and <text_layout_1> denote the text-layout sentinel
tokens, <100><350><118><372> and <100><370><118><382>” represent the layout
tokens of “Date to” and “of” respectively. We use masking ratio 15% similar to
Masked Language Modeling (MLM) [9] as this task can be interpreted as masked
text-layout modeling.

(2) Layout Modeling asks the model to predict positions of (group of) text
tokens, given the document image and context text. E.g., to predict positions
of “Ship Date” and “of”, the input sequence and target sequence is given
below:

Input Sequence:
---
“Layout Modeling. <layout_0> Ship Date </layout_0>
to Retail: Week <layout_1> of </layout_1> March 14,
1994”
Target Sequence:
“<layout_0> <100><350><118><372> <layout_1>
<100><370><118><382>”

Note this pretraining task has a different sentinel token, <layout_sent_0>,
from the previous task “Joint Text-Layout Reconstruction” because the
generation content is different (layout vs. text + layout). We use large
masking ratio 75% since masking with small ratio results in an easy task.

(3) Visual Text Recognition identifies text at given location in the image.
E.g., to recognize the text tokens at <100><350><118><372> and
<100><370><118><382>, the input and target is:

Input Sequence:
---
“Visual Text Recognition. <text_0> <100><350><118>
<372> </text_0> to Retail: Week <text_1> <100><370>
<118><382> </text_1> March 14, 1994”
Target Sequence:
“<text_0> Ship Date <text_1> of”

Note this pretraining task also has a different sentinel token, <text_0> . We
use masking ratio 50% to distinguish this task from “Joint Text-Layout
Reconstruction” and set the layout (bounding box) of sentinel token, e.g.,
<text_0>, and layout token, e.g., <0><10><2><20>, to (0,0,0,0). This objective
helps model learn joint vision-text embedding by understanding vision-text
correspondence.

Figure 3:  Masked autoencoding with text and layout.

(4) Masked Image Reconstruction with Text and Layout aims to reconstruct image
with text and layout as shown in Figure 3. We adopt the MAE objective [14] for
vision self-supervised learning. Originally, MAE masks a percentage of the
image patches and feed non-masked patches into a vision encoder. It then feeds
encoder outputs to a vision decoder to reconstruct masked patches. MAE uses
mean squared error and apply loss only on masked patches. We make the
following modifications to the MAE decoding process to customize it for
document image generation and our task unification framework:

(4.a) Cross-Attention with Character Embeddings. In document, the textual
content mostly consists of alphabetic characters, numbers and punctuation. The
character-level composition of text tokens should be helpful for the vision
generation. We add cross-attention in the vision decoder that it attends to
both the text token encoder features and embeddings of characters in the token
(Figure 3 left upper). These characters embeddings are trainable parameters
and not encoded by the encoder. This cross-attention with characters only adds
linear computation complexity but considerably improves the image generation
quality.

(4.b) Image Decoding. Next, we describe the MAE decoding process. We first
discuss the image decoding process in the two-tower encoder variant, UDOP-
Dual, which is similar to the original MAE setting. 1) The vision encoder and
text encoder encodes non-masked image patches and text tokens respectively. 2)
Same as in the original MAE model, placeholder embeddings, that represent
masked image patches, and encodings of non-masked image are input to the
vision decoder. The length and order of the input sequence is same as the
patches of target image. 3) The vision decoder cross-attends to the text
encoder and vision decoder output, together with character embeddings.

For UDOP, we cannot directly feed the unified encoder output to the vision
decoder, since the joint vision-text embedding only contains non-masked image
patches to the unified encoder (Section 2.1), and image patches are fused with
text tokens. Therefore, we propose that the vision decoder takes in a sequence
of trainable placeholder embeddings. The length and order of the placeholder
sequence is same as the patches of target image. We use two types of
placeholder embeddings to indicate whether the image patch is masked in the
input document image. The vision decoder attends to encoder vision-text output
AND character embeddings via cross-attention. The above process is illustrated
in Figure 3. We show the high quality generation visualization in Section 5.1.

###  3.2 Supervised Pretraining Tasks

Self-supervised tasks leverage large-scale unlabeled data to learn robust
representations. On the other hand, supervised tasks use labeled data for
fine-grained model supervision. We include the following supervised tasks in
pretraining: document classification, layout analysis, information extraction,
question answering, and document natural language inference. Due to space
limit, details of the following supervised dataset are in Appendix C. Note
that we do not conduct self-supervised tasks on the supervised datasets since
we already have large-scale and diverse unlabeled data.

Classification. The task is to predict the document type. The task prompt is
“Document Classification on (Dataset Name)” like “Document Classification on
RVLCDIP”, then followed by text tokens. The target is the document class. We
use RVL-CDIP [13] with 16 document categories.

Layout Analysis. This task is to predict locations of an entity in the
document like title, paragraph, etc. The task prompt is “Layout Analysis on
(Dataset Name)”, then followed by the entity name. The target are all bounding
boxes that cover the given entity. We use PubLayNet [55].

Information Extraction. This task predict the entity type and location of a
text query (e.g., the abstract paragraph). The task prompt is “Information
Extraction on (Dataset Name) (Text Query)”. The target is the entity label and
the bounding box of each token of the query. We use DocBank [27], Kleister
Charity (KLC) [40], PWC [19], and DeepForm [42].

Question Answering. The task is to answer a given question associated with the
document image. The task prompt is “Question Answering on (Dataset Name)”,
then followed by the question and all document tokens. The target is the
answer. We use WebSRC [3], VisualMRC [44], DocVQA [32], InfographicsVQA [31],
and WTQ (WikiTableQuestions) [34].

Document NLI. Document Natural Language Inference predicts the entailment
relationship between two sentences in a document. The prompt is “Document
Natural Language Inference on (Dataset Name)”, then followed by the sentence
pair. The target is the “Entailment” or ”Not Entailment”. We use TabFact [5]
for this task.

##  4 Experimental Setup

###  4.1 Model Pretraining

Model Configuration. In UDOP, the unified encoder and text-layout decoder
follows the encoder-decoder architecture of T5-large [38]. The vision decoder
is MAE-large decoder [14]. Overall UDOP has 794M trainable parameters. For
UDOP-Dual, the text-layout encoder-decoder follows T5-large, and the vision
encoder-decoder has the same configuration as MAE-large. It has in total 1098M
trainable parameters.

Data. For self-supervised learning, we use IIT-CDIP Test Collection 1.0 [24],
a large-scale document collections commonly-used in previous works [51, 53,
16]. It contain 11 million scanned document with contains text and token-level
bounding boxes extracted by OCR. Supervised datasets are as introduced in
Section 3.2.

Curriculum Learning. We use large image resolution, $1024$, in our final
settings since low resolution makes document text unidentifiable for both
detection and generation. It will result in $(1024/16)^{2}=4096$ image patch
sequence length which takes longer training time than small image resolution,
e.g., $224$. Therefore, we use curriculum learning to start from a relatively
small resolution and gradually scale up to 1024 resolution. In practice, we
use scale with 3 resolutions during the pretraining $224\rightarrow
512\rightarrow 1024$. We show the performance of the 3 stages in Appendix D.

Training. We use Adam [22] optimizer with learning rate 5e-5, 1000 warmup
steps, batch size 512, weight decay of 1e-2, $\beta_{1}=0.9$, and
$\beta_{2}=0.98$. For each curriculum learning stage, we train for 1 epoch.

###  4.2 Downstream Evaluations

Table 2:  Comparison with existing published models on the DUE-Benchmark.
Modality T, L, V denote text, layout, or vision.

Model | Modality | Question Answering | Information Extraction | Table QA/NLI | Avg.
---|---|---|---|---|---
DocVQA | InfoVQA | KLC | PWC | DeepForm | WTQ | TabFact
Donut | V | 72.1 | - | - | - | - | - | - | -
BERTlarge [9] | T | 67.5 | - | - | - | - | - | - | -
T5large [38] | T | 70.4 | 36.7 | 74.3 | 25.3 | 74.4 | 33.3 | 58.9 | 50.7
T5large+U [35] | T | 76.3 | 37.1 | 76.0 | 27.6 | 82.9 | 38.1 | 76.0 | 56.5
T5large+2D [35] | T+L | 69.8 | 39.2 | 72.6 | 25.7 | 74.0 | 30.8 | 58.0 | 50.4
T5large+2D+U [35] | T+L | 81.0 | 46.1 | 75.9 | 26.8 | 83.3 | 43.3 | 78.6 | 59.8
LAMBERT [10] | T+L | - | - | 81.3 | - | - | - | - | -
StructuralLMlarge[25] | T+L | 83.9 | - | - | - | - | - | - | -
LayoutLMv2large [53] | V+T+L | 78.8 | - | - | - | - | - | - | -
LayoutLMv3large [16] | V+T+L | 83.4 | 45.1 | 77.1 | 26.9 | 84.0 | 45.7 | 78.1 | 62.9
UDOP-Dual | V+T+L | 84.4 | 47.1 | 81.9 | 28.7 | 85.2 | 46.7 | 79.5 | 64.7
UDOP | V+T+L | 84.7 | 47.4 | 82.8 | 28.9 | 85.5 | 47.2 | 78.9 | 65.1

Table 3:  Performance on FUNSD, CORD, and RVL-CDIP datasets. Modality V, T, L
denote vision, text and layout.

Model | Modality | Info Ext. | Classification
---|---|---|---
FUNSD | CORD | RVL-CDIP
Donut | V | - | 91.6 | 95.3
BERTlarge | T | 65.63 | 90.25 | 89.92
BROSlarge[15] | T+L | 84.52 | 97.40 | -
StructuralLMlarge | T+L | 85.14 | - | 96.08
LiLT[47] | T+L | 88.41 | 96.07 | 95.68
FormNet [23] | T+L | 84.69 | 97.28 | -
LayoutLMlarge | T+L | 77.89 | - | 91.90
SelfDoc | V+T+L | 83.36 | - | 92.81
UDoc | V+T+L | 87.93 | 98.94 | 95.05
DocFormerlarge [1] | V+T+L | 84.55 | 96.99 | 95.50
TILTlarge | V+T+L | - | 96.33 | 95.52
LayoutLMv2large | V+T+L | 84.20 | 96.01 | 95.64
LayoutLMv3large | V+T+L | 92.08 | 97.46 | 95.93
UDOP-Dual | V+T+L | 91.20 | 97.64 | 96.22
UDOP | V+T+L | 91.62 | 97.58 | 96.00

We report the results on FUNSD [18], CORD [33], RVL-CDIP [13], and DocVQA [32]
in Table 3 and describe their respective settings in below. We also report the
results on 7 datasets of DUE-Benchmark [2] in Table 2. Finetuning training
details are available in Section C.6 and performance variance is available in
Table 6 and Table 7.

FUNSD (Form Understanding in Noisy Scanned Documents [18]) has 149 and 50
samples for train and test. We evaluate on the entity recognition task:
predicting the entity, "question", "answer", "header", or "other", for the
text token. The task format is, suppose we have the title, "The Title", and
its entity "[I-Header]", then the encoder input is "The Title" and the
generation target is "The Title [I-Header]". The metric is F1 scores.

CORD (Consolidated Receipt Dataset for Post-OCR Parsing) [33] is a key
information extraction dataset with 30 labels under 4 categories such as
"total" or "subtotal". It has 1,000 receipt samples. The train, validation,
and test splits contain 800, 100, and 100 samples respectively. The metric is
F1 and the task format is the same as FUNSD.

RVL-CDIP is the document classification dataset that we have discussed
previously. It has 320k/40k/40k images for training/validation/test. The
metric is classification accuracy.

DUE-Benchmark contains 7 datasets and 3 domains, including document question
answering (DocVQA [32], InfographicsVQA[31]), key information extraction
(KLC[40], PWC[19], DeepForm[42]), and Table QA/NLI (WTQ[34], TabFact[5]). Task
prompt formats can be found in Section 3.2 and details of datasets can be
found in the appendix.

Results. Pretrained models are finetuned on each evaluation dataset. As shown
in Table 2, our models UDOP and UDOP-Dual achieve SOTA performance on all 7
tasks of DUE-Benchmark, ranking the 1st place on the leaderboard as of
November 11, 2022. Our models also sets SOTA on CORD, RVL-CDIP (Table 3).

Curriculum learning on image resolution (appendix Table 5) shows that with
larger resolution, UDOP steadily gains stronger performance. E.g., UDOP
average performance on DUE-Benchmark with 224, 512 and 1024 resolution is
63.9, 64.3 and 65.1 respectively. Note our model with 224 resolution already
outperform previous best models (e.g., average 62.9 on DUE-Benchmark). Also
like many previous works that introduce supervised learning into
pretraining[48, 8], our evaluation datasets partially overlap with supervised
datasets used in pretraining. We then train UDOP and UDOP-Dual only with self-
supervised objectives (224 resolution). Their performance (Table 4) also
surpasses baselines, which shows the effectiveness of the unified
representations, TVL transformer and the proposed self-supervised objectives.

Figure 4:  Document generation with customized content (right). Left is the
original document. We show four document edits within the same figure
including title replacement, text addition, text replacement, and tilted text
replacement. All edits are done with one model run.  Figure 5:  Document
generation with customized layout (right). Left is the original document. We
change the layout of the document text including line breaks change and text
rearrangement. All edits are done with one model run.

##  5 Analysis

###  5.1 Visualization Analysis

Masked Image Reconstruction. Figure 6 presents masked image reconstruction.
Even with high masking ratio, the model can reconstruct the document image
from text and layout signals with high quality: reconstructed contents are
clear, consistent, and almost identical with the original image (all
demonstrations are conducted on unseen documents.).

Figure 6:  MAE demonstrations with 75% masking. Middle: reconstruction, Right:
original.

Document Generation & Editing. For the first time in Document AI, UDOP
achieves controllable high-quality document generation and editing. As shown
in Fig. 4), one can edit and add to the document image content with customized
contents. The generated content is of high resolution and is consistent with
context in font, size, style and orientation (e.g., vertical numbers in Fig.
4). More generation examples are available in Appendix B. This is done by
masking the regions to edit in the document image, and specifying the
customized content in the text input, and their positions through layout
embeddings. This novel functionality can generate augmentation document data
for future research.

Layout Customization. UDOP can perform controllable high-quality document
layout edits. We show examples in Figure 5, where our model can edit the
layout of the document by regenerating the document from scratch. This is done
by keeping only a few image patch as prompt, change the bounding boxes of the
content, and then regenerate the document image with the new layout.

###  5.2 Ablation Analysis

Table 4 presents the ablation study of pretraining objectives on DocVQA and
RVL-CDIP validation sets. Models pretrained with layout/text self-supervised
objectives (“Layout Modeling”, “Visual Text Dataition”, and “Joint Text-Layout
Reconstruction”) outperforms the one trained with masked language modeling
(MLM), confirming their effectiveness. Adding vision self-supervised learning
(masked image reconstruction) and supervised learning further improves the
performance.

Table 4:  Ablation study on pretraining objectives. Models are evaluated on
the validation set. Results on more datasets are presented in Table 8

Pretrain Objectives | #Pretrain Data | DocVQA | RVL-CDIP
---|---|---|---
MLM | 11.0M | 79.7 | 95.3
UDOP-Dual |  |  |
Self-Supervised | 11.0M | 83.5 | 95.8
\+ Supervised | 12.8M | 84.1 | 96.1
UDOP |  |  |
Self-Supervised | 11.0M | 84.4 | 96.2
\+ Supervised | 12.8M | 85.0 | 96.3

##  6 Related Work

Unifying Model Architectures in Multimodal Learning. Unifying model
architectures for different modalities, such as vision, language, and speech,
is an emergent direction. Inspired by the immense success in natural language
processing, computer vision and speech processing, model architectures in
multimodal learning is converging to Transformers. One type of works
concatenates text token embeddings and projected image patches as the input
[6, 41] to a multimodal Transformer. Other models uses two-tower or three-
tower architecture where each modality is encoded respectively. Projection
heads or fusion networks on top of the two-tower architecture generate
multimodal representations [37, 54].

Unifying Tasks with the Generative Framework. Research on unifying training
processes across different tasks and domains recently has made significant
progress. [8] finetunes language models with instructions on 1.8k tasks. [7]
unifies several vision-language tasks by converting training objectives to
sequence generation. [48, 49, 30] further combines more tasks, e.g., image
generation, by converting images and bounding boxes to discrete tokens.

Document Artificial Intelligence. LayoutLM [51] pretrains BERT models on
document data with masked language modeling and document classification task,
with 2D positional information and image embeddings integrated. Subsequent
works [53, 16, 15] also adopt VL-BERT alike architecture and includes
additional pretraining tasks, e.g., masked image/region modeling proposed, and
leverages the reading order in layout information [12]. [11, 28] use a
multimodal encoder to model region features extracted by CNN with sentence-
level text representations and train with self-supervised objectives. [20]
proposes an OCR-free model to directly generate textual output from document
images. [35] trains generative language models on both unlabeled and labeled
document data using generative training objectives. [10] proposed to model
documents as collections of tokens bounding boxes.

##  7 Conclusion

In this work, we propose UDOP, a foundation model for document AI. UDOP
unifies the vision, text and layout modalities of documents by utilizing their
strong spatial correlations through layout-induced vision-text representations
and Vision-Text-Layout transformer. It also unites all self-supervised and
supervised document tasks with a generative framework. UDOP achieves SOTA on 9
tasks and currently ranks the 1st place on the Document Understanding
Benchmark Leaderboard. For the first time in document AI, UDOP achieves
customizable realistic document generation and editing. We discuss the
limitations and societal impact of our work in the appendix.

## References

  * [1] Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha.  Docformer: End-to-end transformer for document understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 993–1003, 2021.
  * [2] Łukasz Borchmann, Michał Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Michał Turski, Karolina Szyndler, and Filip Graliński.  Due: End-to-end document understanding benchmark.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * [3] Lu Chen, Xingyu Chen, Zihan Zhao, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu.  Websrc: A dataset for web-based structural reading comprehension.  arXiv preprint arXiv:2101.09465, 2021.
  * [4] Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, and Geoffrey Hinton.  Pix2seq: A language modeling framework for object detection.  In International Conference on Learning Representations, 2022.
  * [5] Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang.  Tabfact: A large-scale dataset for table-based fact verification.  arXiv preprint arXiv:1909.02164, 2019.
  * [6] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In European conference on computer vision, pages 104–120. Springer, 2020.
  * [7] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal.  Unifying vision-and-language tasks via text generation.  In International Conference on Machine Learning, pages 1931–1942. PMLR, 2021.
  * [8] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  arXiv preprint arXiv:2210.11416, 2022.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2018.
  * [10] Łukasz Garncarek, Rafał Powalski, Tomasz Stanisławek, Bartosz Topolski, Piotr Halama, Michał Turski, and Filip Graliński.  Lambert: layout-aware language modeling for information extraction.  In International Conference on Document Analysis and Recognition, pages 532–547. Springer, 2021.
  * [11] Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun.  Unidoc: Unified pretraining framework for document understanding.  Advances in Neural Information Processing Systems, 34:39–50, 2021\.
  * [12] Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang.  Xylayoutlm: Towards layout-aware multimodal networks for visually-rich document understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4583–4592, 2022.
  * [13] Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis.  Evaluation of deep convolutional nets for document image classification and retrieval.  In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 991–995. IEEE, 2015.
  * [14] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  arXiv preprint arXiv:2111.06377, 2021.
  * [15] Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park.  Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10767–10775, 2022.
  * [16] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei.  Layoutlmv3: Pre-training for document ai with unified text and image masking.  arXiv preprint arXiv:2204.08387, 2022.
  * [17] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu.  Pixel-bert: Aligning image pixels with text by deep multi-modal transformers.  arXiv preprint arXiv:2004.00849, 2020.
  * [18] Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran.  Funsd: A dataset for form understanding in noisy scanned documents.  In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2, pages 1–6. IEEE, 2019.
  * [19] Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, and Robert Stojnic.  Axcell: Automatic extraction of results from machine learning papers.  arXiv preprint arXiv:2004.14356, 2020.
  * [20] Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park.  Ocr-free document understanding transformer.  In European Conference on Computer Vision, pages 498–517. Springer, 2022.
  * [21] Wonjae Kim, Bokyung Son, and Ildoo Kim.  ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision.  In ICML, 2021.
  * [22] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2014.
  * [23] Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, and Tomas Pfister.  Formnet: Structural encoding beyond sequential modeling in form document information extraction.  arXiv preprint arXiv:2203.08411, 2022.
  * [24] David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, David Grossman, and Jefferson Heard.  Building a test collection for complex document information processing.  In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 665–666, 2006.
  * [25] Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si.  Structurallm: Structural pre-training for form understanding.  arXiv preprint arXiv:2105.11210, 2021.
  * [26] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [27] Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou.  Docbank: A benchmark dataset for document layout analysis.  arXiv preprint arXiv:2006.01038, 2020.
  * [28] Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu.  Selfdoc: Self-supervised document representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5652–5660, 2021.
  * [29] Yulin Li, Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding.  Structext: Structured text understanding with multi-modal transformers.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 1912–1920, 2021.
  * [30] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  arXiv preprint arXiv:2206.08916, 2022.
  * [31] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar.  Infographicvqa.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022.
  * [32] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.  Docvqa: A dataset for vqa on document images.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021.
  * [33] Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee.  Cord: a consolidated receipt dataset for post-ocr parsing.  In Workshop on Document Intelligence at NeurIPS 2019, 2019.
  * [34] Panupong Pasupat and Percy Liang.  Compositional semantic parsing on semi-structured tables.  arXiv preprint arXiv:1508.00305, 2015.
  * [35] Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka.  Going full-tilt boogie on document understanding with text-image-layout transformer.  In International Conference on Document Analysis and Recognition, pages 732–747. Springer, 2021.
  * [36] Subhojeet Pramanik, Shashank Mujumdar, and Hima Patel.  Towards a multi-modal, multi-task learning based pre-training framework for document representation learning.  arXiv preprint arXiv:2009.14457, 2020.
  * [37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [38] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  JMLR, 2020.
  * [39] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text.  In EMNLP, 2016.
  * [40] Tomasz Stanisławek, Filip Graliński, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemysław Biecek.  Kleister: key information extraction datasets involving long documents with complex layouts.  In International Conference on Document Analysis and Recognition, pages 564–579. Springer, 2021.
  * [41] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In International Conference on Learning Representations, 2019.
  * [42] S Svetlichnaya.  Deepform: Understand structured documents at scale, 2020.
  * [43] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  In EMNLP, 2019.
  * [44] Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida.  Visualmrc: Machine reading comprehension on document images.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13878–13888, 2021.
  * [45] Zineng Tang, Jaemin Cho, Jie Lei, and Mohit Bansal.  Perceiver-vl: Efficient vision-and-language modeling with iterative latent attention.  arXiv preprint arXiv:2211.11701, 2022.
  * [46] Zineng Tang, Jie Lei, and Mohit Bansal.  Decembert: Learning from noisy instructional videos via dense captions and entropy minimization.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2415–2426, 2021.
  * [47] Jiapeng Wang, Lianwen Jin, and Kai Ding.  Lilt: A simple yet effective language-independent layout transformer for structured document understanding.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7747–7757, 2022.
  * [48] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In International Conference on Machine Learning, pages 23318–23340. PMLR, 2022.
  * [49] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  arXiv preprint arXiv:2208.10442, 2022.
  * [50] Te-Lin Wu, Cheng Li, Mingyang Zhang, Tao Chen, Spurthi Amba Hombaiah, and Michael Bendersky.  Lampret: Layout-aware multimodal pretraining for document understanding.  arXiv preprint arXiv:2104.08405, 2021.
  * [51] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou.  Layoutlm: Pre-training of text and layout for document image understanding.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1192–1200, 2020.
  * [52] Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei.  Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding.  arXiv preprint arXiv:2104.08836, 2021.
  * [53] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al.  Layoutlmv2: Multi-modal pre-training for visually-rich document understanding.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2579–2591, 2021.
  * [54] Ziyi Yang, Yuwei Fang, Chenguang Zhu, Reid Pryzant, Dongdong Chen, Yu Shi, Yichong Xu, Yao Qian, Mei Gao, Yi-Ling Chen, et al.  i-code: An integrative and composable multimodal learning framework.  arXiv preprint arXiv:2205.01818, 2022.
  * [55] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes.  Publaynet: largest dataset ever for document layout analysis.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1015–1022. IEEE, 2019.

Figure 7:  Document generation with customized content (right). Left is the
original document. We show different document edits within the same figure
including title replacement, text addition, text replacement, and tilted text
replacement. All edits are done with one model run.  Figure 8:  Document QA
and answer localization with UDOP on VisualMRC dataset. As shown, besides
generating the answer, UDOP can predict the region of interest (RoI) that
answer is located in by generating the layout tokens. Note that the the
labeled RoI VisualMRC dataset is at paragraph level.  Table 5:  Comparison of
different image size in curriculum learning on the DUE-Benchmark. Modality T,
L, V denote text, layout, or vision.

Model | Modality | Question Answering | Information Extraction | Table QA/NLI | Avg.
---|---|---|---|---|---
DocVQA | InfoVQA | KLC | PWC | DeepForm | WTQ | TabFact
UDOP (224) | V+T+L | 84.4 | 46.1 | 82.1 | 26.7 | 83.6 | 46.1 | 78.2 | 63.9
UDOP (512) | V+T+L | 84.5 | 47.3 | 82.0 | 27.1 | 84.7 | 46.2 | 78.3 | 64.3
UDOP (1024) | V+T+L | 84.7 | 47.4 | 82.8 | 28.9 | 85.5 | 47.2 | 78.9 | 65.1

Table 6:  Performance with standard deviations on on the DUE-Benchmark.
Modality T, L, V denote text, layout, or vision.

Model | Modality | Question Answering | Information Extraction | Table QA/NLI | Avg.
---|---|---|---|---|---
DocVQA | InfoVQA | KLC | PWC | DeepForm | WTQ | TabFact
Donut | V | 72.1 | - | - | - | - | - | - | -
BERTlarge [9] | T | 67.5 | - | - | - | - | - | - | -
T5large [38] | T | 70.4 | 36.7 | 74.3 | 25.3 | 74.4 | 33.3 | 58.9 | 50.7
T5large+U [35] | T | 76.3 | 37.1 | 76.0 | 27.6 | 82.9 | 38.1 | 76.0 | 56.5
T5large+2D [35] | T+L | 69.8 | 39.2 | 72.6 | 25.7 | 74.0 | 30.8 | 58.0 | 50.4
T5large+2D+U [35] | T+L | 81.0 | 46.1 | 75.9 | 26.8 | 83.3 | 43.3 | 78.6 | 59.8
LAMBERT [10] | T+L | - | - | 81.3 | - | - | - | - | -
StructuralLMlarge[25] | T+L | 83.9 | - | - | - | - | - | - | -
LayoutLMv2large [53] | V+T+L | 78.8 | - | - | - | - | - | - | -
LayoutLMv3large [16] | V+T+L | 83.4 | 45.1 | 77.1 | 26.9 | 84.0 | 45.7 | 78.1 | 62.9
UDOP-Dual | V+T+L | 84.4$\pm$0.1 | 47.1$\pm$0.2 | 81.9$\pm$0.4 | 28.7$\pm$0.5 | 85.2$\pm$0.2 | 46.7$\pm$0.4 |  79.5$\pm$0.3 | 64.7$\pm$0.3
UDOP | V+T+L |  84.7$\pm$0.2 |  47.4$\pm$0.2 |  82.8$\pm$0.3 |  28.9$\pm$0.4 |  85.5$\pm$0.2 |  47.2$\pm$0.2 | 78.9$\pm$0.1 |  65.1$\pm$0.2

Table 7:  Performance with standard deviations on FUNSD, CORD, and RVL-CDIP datasets.  Model | Modality | Info Ext. | Classification
---|---|---|---
FUNSD | CORD | RVL-CDIP
Donut | V | - | 91.6 | 95.3
BERTlarge | T | 65.63 | 90.25 | 89.92
BROSlarge[15] | T+L | 84.52 | 97.40 | -
StructuralLMlarge | T+L | 85.14 | - | 96.08
LiLT[47] | T+L | 88.41 | 96.07 | 95.68
FormNet [23] | T+L | 84.69 | 97.28 | -
LayoutLMlarge | T+L | 77.89 | - | 91.90
SelfDoc | V+T+L | 83.36 | - | 92.81
UDoc | V+T+L | 87.93 | 98.94 | 95.05
DocFormerlarge [1] | V+T+L | 84.55 | 96.99 | 95.50
TILTlarge | V+T+L | - | 96.33 | 95.52
LayoutLMv2large | V+T+L | 84.20 | 96.01 | 95.64
LayoutLMv3large | V+T+L | 92.08 | 97.46 | 95.93
UDOP-Dual | V+T+L | 91.20$\pm$0.21 |  97.64$\pm$0.12 |  96.22$\pm$0.27
UDOP | V+T+L |  91.62$\pm$0.34 |  97.58$\pm$0.15 |  96.00$\pm$0.26
Table 8:  Ablation study on pretraining objectives. Performance is reported on
validation sets.

Pretrain Objectives | #Pretrain Data | DocVQA | RVL-CDIP | FUNSD | CORD
---|---|---|---|---|---
MLM | 11.0M | 79.7 | 95.3 | 90.2 | 96.7
UDOP-Dual |  |  |  |  |
Self-Supervised | 11.0M | 83.5 | 95.8 | 91.5 | 97.2
\+ Supervised | 12.8M | 84.1 | 96.1 | 91.5 | 97.3
UDOP |  |  |  |  |
Self-Supervised | 11.0M | 84.4 | 96.2 | 91.0 | 97.2
\+ Supervised | 12.8M | 85.0 | 96.3 | 91.9 | 97.4

##  Appendix A Appendix Overview

The appendix has the following contents:

  * •

Vision demonstrations of UDOP localizing answers in documents, the
effectiveness of the cross attention with character embeddings in vision
generation, and more neural editing examples Appendix B.

  * •

More details for pretraining and evaluation datasets, and finetuning
experiment set up in Appendix C.

  * •

Experiment results of curriculum learning in Appendix D.

  * •

Ablation studies on pretraining objectives in Appendix E.

  * •

Performance variance of UDOP in Appendix F.

  * •

Additional supervised training stage in Appendix G.

  * •

Discussion of limitations and societal impacts in Appendix H.

##  Appendix B Visualization Analysis

Creative Image Generation. UDOP achieves controllable high-quality document
generation and editing as described in Section 5.1. We show additional
examples here in Fig. 7. Our model can edit and add to the document image
content with customized contents. Note that even if the document content is
vertical (the first subfigure of Fig. 7), UDOP can still achieve high
generation quality.

Answer Localization for Document QA. UDOP can perform question answering while
predicting the location of the answer. We show examples on VisualMRC in Figure
8 and our model can answer the questions regarding the document correctly
while locating the area of interest.

##  Appendix C Supervised Pretraining Tasks

In this section, we list more details about the supervised datasets in
pretraining and evaluations.

###  C.1 Classification

RVL-CDIP [13] contains 16 document categories, such as “invoice”, “scientific
publication” and “form”. The dataset has 320k training, 40k validation and 40k
test images.

###  C.2 Layout Analysis

PubLayNet [55] is a layout analysis dataset created from medical publications.
It contains over 360k document images and labeled with typical document layout
elements such as titles, paragraphs, etc.

###  C.3 Information Extraction

DocBank [27] is a richly-annotated large-scale IE dataset. It consists of 500K
document pages, where 400K for training, 50K for validation and 50K for
testing. It has 12 semantic structure labels like abstract, title, and author.
Each token has corresponding bounding box and semantic structure label.

Kleister Charity [40] is an IE dataset with complex invoice page layout and
has 21.6k entities and 2.7k document images from UK Charity Commission. Its
entities for extraction include invoice date, invoice number, net amount,
vendor name, etc.

PWC [19] is an IE dataset which has 2,291 leaderboards, where the data is
collected from the Papers with Code labelling interface. It asks information
like task, dataset, metric, etc. Different from original implementation, DUE-
Benchmark provides complete papers as input instead of tables.

DeepForm [42] is an IE dataset collected from political television ads in US
elections and has 20k receipts and over 100k document images. This task is to
extract entities like advertiser name, contract number, amount paid, etc.

###  C.4 Question Answering

WebSRC [3] stands for Web-based Structural Reading Comprehension. It consists
of 0.44M questions collected from 6.5K web pages with corresponding HTML,
screenshots and metadata. The answer is either the text span of context or
yes/no.

VisualMRC [44] stands for visual machine reading comprehension. It consists of
10,197 images 30,562 abstractive questions-answers.

DocVQA [32] is a QA dataset for excerpts from industry documents and has 50k
questions on 12k document images. It asks questions on topics like text
content, non-textual elements like marks or diagrams, layout, style, etc.

InfographicsVQA [31] is a QA dataset with a focus on infographic images and
has 30K questions on 5.3k document images. It requires reasoning on text
content, images, data visualizations, layout, etc.

WTQ [34] is a table-based QA dataset on HTML tables collected from Wikipedia.
It has 2.1k tables and 22k questions hand crafted by humans and cover a wide
range of topics like table lookup, superlatives, arithmetic operations, etc.

###  C.5 Document NLI

TabFact [5] is an open-domain table-based NLI task and has 16k Wikipedia
tables for 118k statements by human annotations.

###  C.6 Finetuning Experiment Setting

For all DUE-Benchmark finetuning experiments, we use Adam [22] optimizer with
learning rate 5e-5, 1000 warmup steps, batch size 16, weight decay of 1e-2,
$\beta_{1}=0.9$, and $\beta_{2}=0.98$. For FUNSD and CORD, we use learning
rate 3e-4 and for RVL-CDIP, we use learning rate 1e-3 both with 1000 warmup
steps, batch size 16, weight decay of 1e-2, $\beta_{1}=0.9$, and
$\beta_{2}=0.98$.

##  Appendix D Curriculum Learning

In this section, we present the results of curriculum learning of input image
resolution (224, 512, 1024) on the validations sets of evaluation benchmarks.
As shown in Table 5, while the model already performs competitively well on
224 resolution, its performance further increases on 512 and 1024.

##  Appendix E Ablation Study

We show ablation study of pretraining objectives on additional datasets
including FUNSD and CORD, i.e., the expanded version of Table 4 as shown in
Table 8.

##  Appendix F Performance Variance

For results in Table 2 and Table 3, we report their standard deviations as
shown in Table 6 and Table 7. The deviations are computed from 5 runs with
different seeds for parameter initialization.

##  Appendix G Additional Supervised Training Stage

TILT [35] performs additional training on a wide range of QA datasets, such as
reading comprehension dataset SQuAD [39], before the finetuning on DocVQA.
This results in considerable performance improvement of the TILT model on
DocVQA. To have a fair comparison, we also finetune UDOP on the same set of
datasets before testing on DocVQA. As shown in Table 9, UDOP is further
improved with this auxiliary training and outperforms TILT.

Table 9:  Training UDOP on auxiliary QA datasets as in TILT. The performance
of UDOP on DocVQAs further improved. (TILT performance without the auxiliary
training is not reported in TILT paper)

Model | Modality | DocVQA
---|---|---
TILTlarge(w/ the auxiliary training) | V+T+L | 87.1
UDOP (w/o the auxiliary training) | V+T+L | 84.7
UDOP (w/ the auxiliary training) | V+T+L | 87.8

##  Appendix H Limitations and Societal Impact

UDOP can assist users with document analysis, understanding and information
extraction. This automatic processing technology will make the document
processing workflow more efficient and potential more accurate. It is also
worth noting that, similar to all AI generation technology, the document
generation capacity of UDOP can be potentially abused for malicious document
counterfeit, e.g., signature forgery, tampering monetary amount in checks,
fake medical/financial records generation, etc. To avoid abuse, for model
release we plan to open source the vision generation model only with limited
access, e.g., through an API. Documents submitted by users that are classified
as sensitive (the classifier can be a finetuned UDOP model), such as checks
and personal ID, will be denied.

Applying UDOP on non-English data, especially those with non-Latin writing
systems, may require further modifications to the model. For example, in Sec.
3.1, the vision decoder cross-attends with character embeddings. Then for non-
English data, we need to include more character embeddings to attend with.


# Unveiling the Importance of Longer Paths in Quantum Networks

Xinqi Hu  Gaogao Dong<EMAIL_ADDRESS>School of Mathematical Sciences,
Jiangsu University, Zhenjiang, Jiangsu 212013, China  Renaud Lambiotte
<EMAIL_ADDRESS>Mathematical Institute, University of Oxford,
Oxford OX2 6GG, UK  Turing Institute, London NW1 2DB, UK  Kim Christensen
<EMAIL_ADDRESS>Blackett Laboratory and Centre for Complexity
Science, Imperial College London, London SW7 2AZ, UK  Jingfang Fan  School of
Systems Science and Institute of Nonequilibrium Systems, Beijing Normal
University, Beijing 100875, China  Lixin Tian  School of Mathematical
Sciences, Jiangsu University, Zhenjiang, Jiangsu 212013, China  Shlomo Havlin
Department of Physics, Bar-Ilan University, Ramat Gan 52900, Israel  Xiangyi
Meng<EMAIL_ADDRESS>Department of Physics and Astronomy, Northwestern
University, Evanston, Illinois 60208, USA

###### Abstract

The advancement of quantum communication technologies is calling for a better
understanding of quantum network (QN) design from first principles, approached
through network science. Pioneering studies have established a classical
percolation mapping to model the task of entanglement transmission across QN.
Yet, this mapping does not capture the stronger, yet not fully understood
connectivity observed in QNs, which facilitates more efficient entanglement
transmission than predicted by classical percolation. In this work, we explore
the critical phenomena of the potential statistical theory underlying this
enhanced connectivity, known as concurrence percolation. Compared to classical
percolation, the concurrence percolation mapping employs a unique approach of
“superposing” path connectivities, utilizing a different set of path
connectivity rules, thereby boosting the overall network connectivity.
Firstly, we analytically derive the percolation critical exponents for
hierarchical, scale-free networks, particularly the ($U,V$) flower model,
characterized by two distinct network length scales, $U\leq V$. Our analysis
confirms that classical and concurrence percolations, albeit both satisfying
the hyperscaling relation, fall into separate universality classes. Most
importantly, this separation stems from their different treatment of non-
shortest path contributions to overall connectivity. Notably, as the longer
path scale $V$ increases, concurrence percolation retains unignorable
dependence of both its critical threshold and critical exponents on $V$ and
thus, comparing with its classical counterpart, shows a higher resilience to
the weakening of non-shortest paths. This higher resilience is also observed
in real-world network topology, e.g., the Internet. Our findings reveal a
first principle for QN design: longer paths still contribute significantly to
QN connectivity—as long as they are abundant.

The emerging prospect of quantum Internet [1] is driving the field of quantum
communication to explore larger and more complex scales, entering the realms
of statistical physics [2] and network science [1]. As a response, a _quantum
network_ (QN) model [3] has been introduced to explore the potentials of
transmitting quantum resource—most notably, entanglement [4]—across complex
network topologies. In the QN model, each link represents an identical,
partially entangled state
$\left|\psi(\theta)\right\rangle=\cos\theta\left|00\right\rangle+\sin\theta\left|11\right\rangle$,
shared between two qubits that come from different nodes [Fig. 1a]. The
parameter $\theta\in[0,\pi/4]$ serves as a “link weight” that quantifies the
degree of entanglement. Specifically, $\theta=\pi/4$ signifies a perfectly
entangled state, while $\theta=0$ indicates zero entanglement. This
homogeneous, pure-state QN maps onto a classical percolation problem [3],
where each link can employ certain quantum operations as a sort of “gambling”
to enhance its level of entanglement, with allows a probability $p\equiv
2\sin^{2}\theta$ of becoming perfectly entangled and $1-p$ of losing all
entanglement [3]. Within this percolation analogy, entanglement transmission
between two distant nodes—say, Alice (A) and Bob (B)—translates to the
probability of having a path comprising only maximally entangled links between
A and B. The presence of such a path further ensures that infinite-distance
entanglement can be established between A and B [5], marking a successful
event of entanglement transmission.

In the thermodynamic limit (i.e., when the number of nodes $N\to\infty$),
classical percolation exhibits a critical threshold $p_{\text{th}}$ [6,
*galam1996universal, *cohen2002percolation]. It implies that if
$\theta<\sin^{-1}{\sqrt{p_{\text{th}}/2}}$, no entanglement transmission can
happen between A and B when their distance $L\to\infty$. This is, however,
refuted by recent discovery of an alternate, more effective mapping from QN to
a percolation-like statistical theory of _concurrence percolation_ [5].

The best way to understand the difference between classical and concurrence
percolation is to express the percolation connectivity between A and B as a
“superposition” of all _paths’ connectivity_ [Fig. 1b]: In classical
percolation, the overall probability of connection of A and B can be computed
through each individual path, contingent upon $p$. The superposition can be
simplified into connectivity rules, such as the series and parallel rules
[Fig. 1a, left]. Similarly, in concurrence percolation, a distinct parameter
$c$ called concurrence, a conventional measure of entanglement, can be defined
via $\theta$ in the QN, $c\equiv\sin 2\theta$. The superposition of
concurrence paths adheres to a distinct set of connectivity rules [Fig. 1a,
right]. Concurrence percolation also reveals a critical threshold
$c_{\text{th}}$, which, in terms of $\theta$, is always lower than
$p_{\text{th}}$:
$\left(\sin^{-1}{c_{\text{th}}}\right)/2\leq\sin^{-1}{\sqrt{p_{\text{th}}/2}}$
[9], indicating that QNs potentially have stronger connectivity than classical
percolation predicts.

What is the origin of this stronger connectivity, from a statistical physics
perspective? It is conceivable that concurrence percolation might simply be a
variant of classical percolation (albeit under a different set of variables),
thereby belonging to the same universality class. Alternatively, concurrence
percolation could represent a fundamentally distinct phenomenon from classical
percolation, characterized by distinct critical exponents. Previous research,
unfortunately, has been unable to identify the universality class of
concurrence percolation due to computational constraints [10]. Gaining a
deeper insight into the nature of this stronger concurrence connectivity could
shed light on QN design from _first principles_ , as explored through the lens
of network science.

In this study, we investigate the critical phenomena of concurrence
percolation in networks displaying hierarchical and scale-free structures,
typical characteristics found in numerous real-worlds networks such as the
_Internet_ [11]. Firstly, we exactly determine the critical exponents for a
family of hierarchical scale-free network models known as the ($U$,$V$)
flowers [12], which are characterized by two distinct network length scales,
$U\leq V$. Our analysis confirms that concurrence percolation falls into a
distinct universality class, separated from that of classical percolation.

Most importantly, we highlight that this separation is largely reflected in
how the two percolation theories respond to an increase in the longer length
scale $V$, controlling the lengths of _non-shortest paths_. When $V\to\infty$,
we find that the classical critical exponents become decoupled from $V$,
depending _only_ on the shorter length scale $U$. In contrast, the concurrence
critical exponents depend on _both_ $U$ and $V$. This distinction extends to
the behavior of critical thresholds: while both $p_{\text{th}}$ and
$c_{\text{th}}$ tend towards unity as $V\to\infty$, the concurrence threshold
$c_{\text{th}}$ shows a slower rate of approach. This implies a higher
resilience of concurrence connectivity against increase in $V$, a phenomenon
we also observe in real-world network topology. Our findings underscore the
role of longer paths in QN: despite the exponential decay of entanglement
along longer paths, these paths cannot be ignored in concurrence percolation.
In fact, _longer paths still contribute significantly to QN connectivity—as
long as they are abundant._ In practice, this principle suggests that an
effective design of QN should move beyond a focus solely on the shortest paths
to include a strategic incorporation of longer paths as well. This view may
open up opportunities to explore advanced quantum communication technologies,
such as path routing [13] and network coding [14], in a synergistic manner.

(a)

(b)

Figure 1: Percolation mappings of a quantum network (QN). (a) In a QN, a node
comprises several qubits, each entangled with those in other nodes, and a link
symbolizes a partially entangled bipartite state $\left|\phi\right\rangle$
shared between two nodes. Entanglement transmission across QN can be
understood from two distinct statistical physics mappings: classical
percolation (blue) vs. concurrence “quantum” percolation (red). Each mapping
is based on its own set of path connectivity rules—series and parallel
rules—which dictate the “superposition” of multiple path connectivities to
determine the total percolation connectivity between two distant nodes, say,
Alice (A) and Bob (B). (b) Different connectivity rules result in a varied
emphasis on path length in classical vs. concurrence percolations. While in
classical percolation, the shortest path between A and B dominates the overall
network connectivity, in concurrence percolation, longer paths also contribute
to the concurrence connectivity if they are abundant.  Figure 2: ($U$,$V$)
flower as hierarchical scale-free network. The ($U$,$V$) flower is a self-
similar network, constructed by iteratively replacing each link with a motif
comprising two nodes connected by two parallel paths of varying lengths $U\leq
V$. The $n$-th generation ($U$,$V$) flower can be fully decomposed into
$2^{n}$ non-overlapping paths between A and B, with varying lengths
$(U^{n},VU^{n-1},...,V^{n})$ and corresponding count of paths
$(C^{0}_{n},C^{1}_{n},...,C^{n}_{n})$. The example shows $U=2$ and $V=3$.
Figure 3: Critical phenomena on ($U$,$V$) flower. (a-e) Critical exponent
$\nu$ in classical (blue) and quantum (red) mappings. (f-j) Critical exponents
$\beta$ and $d_{f}$.

##  I Critical Exponents and Hyperscaling

We begin by focusing on ($U$,$V$) flowers [12], a canonical example of
hierarchical, scale-free networks. In a ($U$,$V$) flower, the $(n+1)$-th
generation is built by replacing each existing link in the $n$-th generation
by a basic motif, formed by a shorter path with $U$ links and a longer path
with $V$ links between two nodes A and B (Fig. 2). The two scales $U$ and $V$
determine different network characteristics: while the shorter length scale
$U$ governs the shortest path length $L=U^{n}$ between A and B, which is
asymptotically the same as the diameter of the network [15], the longer length
scale $V$ controls the lengths of non-shortest paths, thereby controlling the
dimension of the network, $d=\ln N/\ln L$, where $N\sim\left(U+V\right)^{n}$
is the number of nodes in the network [15]. Note that all length scales are
established in the chemical (“time”) space, thus differing from traditional
Euclidean length scales $\xi$ by an additional dynamic exponent $z$, related
by $L\sim\xi^{z}$ [12].

A promising feature of ($U$,$V$) flowers is that they are also series-parallel
networks [16]. Consequently, they allow for an analytical calculation of the
classical _sponge-crossing probability_ [6, 17], $P_{\text{sc}}$, which is the
probability of connection between the two boundary nodes A and B (a.k.a.
$\Pi_{\infty}(p;L)$ in boundary conformal field theories [18]). Specifically,
$P_{\text{sc}}$ can be calculated through iteratively applying an exact
renormalization-group (RG) function that is constructed from series and
parallel rules [Fig. 1a]:

$\displaystyle\mathcal{R}(P_{\text{sc}})=\text{para}(\text{seri}(\stackrel{{\scriptstyle
U}}{{\overbrace{P_{\text{sc}},\cdots,P_{\text{sc}}}}}),\text{seri}(\stackrel{{\scriptstyle
V}}{{\overbrace{P_{\text{sc}},\cdots,P_{\text{sc}}}}})).$ (1)

Equation (1) allows us to derive the $n$-th generation $P_{\text{sc}}$ by
nesting $\mathcal{R}$ $n$ times, given by
$P_{\text{sc}}=\stackrel{{\scriptstyle
n}}{{\overbrace{\mathcal{R}(\mathcal{R}(\mathcal{R}(\cdots\mathcal{R}}}}(p))))$.
The critical thresholds $p_{\text{th}}$ can be determined by locating the
nontrivial fixed point of $p$ that satisfies the RG fixed point equation,
$p=\mathcal{R}(p)$. It also means that the sponge-crossing probability
$P_{\text{sc}}$ at criticality is equal to $p_{\text{th}}$ when $n\to\infty$.

Similarly, the sponge-crossing concurrence $C_{\text{sc}}$ and the concurrence
threshold $c_{\text{th}}$ (hereafter referred to as “quantum” results) can be
calculated by replacing $P_{\text{sc}}$ to $C_{\text{sc}}$ in Eq. (1) and the
corresponding classical series/parallel rules to their quantum counterparts
[Fig. 1a]. For example, in a ($2$,$2$) flower, for concurrence percolation, we
find $c_{\text{th}}=0.759\cdots$, or $\theta\approx 0.549\pi/4$, while for
classical percolation, $p_{\text{th}}=(\sqrt{5}-1)/2\approx 0.618$ [15],
corresponding to $\theta\approx 0.750\pi/4$ [Fig. 3c]. The results reinforce
that the quantum threshold is lower than the classical one [5].

Near criticality, the thermal exponent $\nu$ characterizes the divergence of
the correlation length as one approaches the critical point. We can obtain an
exact expression for $\nu$ by evaluating [15]

$\frac{\partial R(p)}{\partial
p}\big{|}_{p=p_{\text{th}}}=\left[\left(UL\right)/L\right]^{1/\nu}\Rightarrow\nu=\frac{\ln{U}}{\ln\left({\partial
R(p)}/{\partial p}\right)}\big{|}_{p=p_{\text{th}}},$ (2)

where $L$ and $UL$ correspond to the diameters of the $n$-th and
$\left(n+1\right)$-th generations of the network, respectively. Compared to
the established classical value of $\nu=1.635\cdots$ [15], we find a distinct
quantum value of $\nu=1.352\cdots$. These exact values are numerically
verified by finite-size scaling analysis: assuming that the finite-size
critical threshold $p_{\text{th}}(L)$ deviates from the true threshold
$p_{\text{th}}$ by $|p_{\text{th}}(L)-p_{\text{th}}|\sim L^{-1/\nu}$, we find
$\nu\approx 1.63528(1)$ for classical and $\nu\approx 1.3530(1)$ for quantum
on both sides of the critical threshold of the ($2$,$2$) flower [Figs.
3a,b,d,e] (also see SI, Section IV).

In the traditional study of classical percolation, not only does the length
scale but the cluster size distribution shows critical behaviors as well.
Notably, the emergence of a giant component when $p\geq p_{\text{th}}$ [19],
where the strength $P_{\infty}=N_{g}/N$ represents the giant component’s
relative size $N_{g}$ to the total network size $N$, leads to a scaling
behavior as $P_{\infty}\sim(p-p_{\text{th}})^{\beta}$ for $p\to
p_{\text{th}}^{+}$. This cluster-based concept of $P_{\infty}$ has not been
translated into the quantum counterpart, which relies on path connectivity
rules and lacks a notion of “clusters.” Nevertheless, we notice that
$P_{\infty}$ can be alternatively considered as _the probability of a randomly
chosen node in the bulk to reach the boundaries of the network_ [15]. This
consideration allows us to redefine $P_{\infty}$, and crucially $C_{\infty}$,
in terms of path connectivity from a randomly chosen node (which must be
averaged across all nodes) to the boundaries. For the ($U,V$) flowers, the
boundaries are represented by A and B. The construction is detailed in SI,
Section I.

Using the new definition of $P_{\infty}$, we find an exponent $\beta\approx
0.1688290(1)$ [Fig. 3g], closely matching the cluster-defined value,
$\beta=0.165\cdots$ on ($2$,$2$) flowers. (The minor difference arises because
computing $P_{\infty}$ requires higher-order path connectivity rules (SI,
Section I) that extend beyond the basic series and parallel rules. However,
these higher-order rules can be approximated using only the series/parallel
rules through a technique known as star-mesh transformations [5], yielding
results that are close, but not precise.) In the quantum domain, we observe
$\beta\approx 0.0765379(1)$ [Fig. 3f], markedly distinct from its classical
counterpart.

Another critical exponent, $d_{f}$, defines the fractal dimension of the giant
cluster at the critical point, with $N_{g}\sim L^{d_{f}}$. Again, lacking a
“cluster” notion, $N_{g}=NP_{\infty}$ must be redefined through $P_{\infty}$
at the critical point. We find $d_{f}\approx 1.89675835(5)$ [Fig. 3j] in the
classical scenario, nearly identical to the cluster-defined value
$d_{f}=1.899\cdots.$ In contrast, the quantum scenario shows $d_{f}\approx
1.9434305(5)$ [Fig. 3i].

Promisingly, the three path-defined critical exponents still adhere to the
hyperscaling relation, $d_{f}=d-\beta/\nu$ [20, 21], in both classical and
concurrence percolation, underscoring a consistent path-connectivity framework
across different percolation theories.

##  II Asymptotic dependence on the longer length scale

To investigate the impact of the longer length scale $V$ on the two
percolation theories, we derive the asymptotic behaviors of both critical
thresholds $p_{\text{th}}$ and $c_{\text{th}}$ in the $V\to\infty$ limit (SI,
Section II), finding that for classical percolation,

$p_{\text{th}}\simeq 1-\left(\ln\frac{U}{U-1}\right)V^{-1}+O(V^{-2}),$ (3)

and, in comparison, for concurrence percolation,

$c_{\text{th}}\simeq 1-\left(\frac{1}{4}\ln V\right)V^{-1}+O(V^{-1}\ln\ln V).$
(4)

The $V^{-1}$ terms originate from the fact that as $V\to\infty$, loops within
the ($U$,$V$) flower extend to infinite lengths, and the network essentially
becomes tree-like. As a result, both thresholds should approach $1$ when
$V\to\infty$. For classical percolation, the prefactor for $V^{-1}$ depends
solely on $U$, signaling the significant role the _shorter paths_ play in
classical percolation near the critical point. In contrast, concurrence
percolation shows a unique behavior. The presence of a logarithmic prefactor
$\left(\ln V\right)/4$ highlights the importance of _longer paths_ in
concurrence percolation. This term indicates that connectivity in concurrence
percolation near criticality is enhanced compared to its classical
counterpart, implying $1-c_{\text{th}}\gg 1-p_{\text{th}}$ as $V\to\infty$.
Moreover, this term is fully decoupled from $U$, suggesting that the shorter
length scale no longer plays a determining role in $c_{\text{th}}$.

The asymptotic behaviors of the critical exponents are also different. We
analytically determine $\nu$ and $d_{f}$ directly via the solution of
$P_{\text{sc}}$ and $P_{\infty}$ (cf. $C_{\text{sc}}$ and $C_{\infty}$), and
$\beta$ from the hyperscaling relation $d_{f}=d-\beta/\nu$. The results are
summarized in Table 1. We find that in classical percolation, all dominant
terms governing $\nu$, $d-d_{f}$, and $\beta$ are fully determined by $U$,
once again signaling the dominance of shorter paths. In contrast, in
concurrence percolation, we notice the emergence of a very slow correction
($\sim\ln\ln V$) affecting the dominant terms in $\nu$ and $d_{f}$.
Interestingly, the corrections counterbalance each other in the calculation of
$\beta$, resulting in a constant $\beta=2$ that is independent on either $U$
or $V$. Since $\beta$ is intrinsically tied to the definition of an order
parameter, we hypothesize that this unique, constant value of $\beta$ reveals
an entirely distinct symmetry associated with the order parameter in
concurrence percolation on ($U$,$V$) flowers.

Table 1:  Critical exponents in the $V\to\infty$ limit. | Classical [$\sim\\!f(U)+O(V^{-1})$] | Quantum [$\sim\\!g(U,V)$]
---|---|---
$\nu$ | $\frac{\ln U}{\ln\left[1+\left(U-1\right)\ln\frac{U}{U-1}\right]}\\!+\\!O(\\!V^{-1}\\!)$ | $\frac{2\ln U}{\ln\ln V}\\!+\\!O\left(\frac{\ln\ln\ln V}{\left(\ln\ln V\right)^{2}}\right)$
$d\\!-\\!d_{f}\\!$ | $\frac{\\!\ln{U}\\!-\\!\ln{\left[\left(1\\!-\\!U\right)+2/\left(\ln\\!{\frac{U}{U-1}}\right)\right]}}{\ln{U}}\\!+\\!O(\\!V^{-1}\\!)$ | $\frac{\ln\ln V}{\ln U}\\!+\\!O\left(\frac{\ln\ln V}{\ln V}\right)$
$\beta$ | $\frac{\\!\ln{U}\\!-\\!\ln{\left[\left(1\\!-\\!U\right)+2/\left(\ln\\!{\frac{U}{U-1}}\right)\right]}}{\ln\left[1+\left(U-1\right)\ln\frac{U}{U-1}\right]}\\!+\\!O(\\!V^{-1}\\!)$ | $2\\!+\\!O\left(\frac{\ln\ln\ln V}{\ln\ln V}\right)$

##  III Superposition of paths in general

The above results have broader implications beyond the specific ($U$,$V$)
flower topology. To see this, it is worth noting that the $n$-th generation of
a ($U$,$V$) flower could be fully decomposed into several _non-overlapping but
intersecting_ paths between A and B. In the language of graph theory, these
paths are edge-disjoint, but not vertex-disjoint. Specifically, each of these
paths can have a length that is given by one of the options
$\\{U^{n},VU^{n-1},\cdots,V^{n}\\}$. The number of paths corresponding to each
length option is determined by the binomial coefficients
$\\{C^{0}_{n},C^{1}_{n},\cdots,C^{n}_{n}\\}$, summing up to $2^{n}$ paths in
total.

The exact decomposition goes as follows. Initially, in the $1$-st generation,
there are only two non-overlapping paths between A and B, characterized by
lengths $U$ and $V$. As progressing to the $2$-nd generation, each link from
the previous generation is systematically replaced by the basic motif,
necessitating a decision for each link within a previous path: to replace it
by either a $U$-length path or a $V$-length path. Consequently, there are two
non-overlapping options: replacing all links entirely by $U$-length paths, or
entirely by $V$-length paths. This bifurcation results in each earlier
generation’s path splitting into two distinct paths. Thus, as a shorthand,
each path can be encoded as a string of $n+1$ characters (e.g., “UVVUUV….”),
each designated as ‘U’ or ‘V’. This string reflects the sequence of choices
made at each generation, with ‘U’ for choosing all $U$-length paths and ‘V’
for $V$-length paths. Each non-overlapping path can be uniquely identified by
such a string. The corresponding path length is $U^{n-x}V^{x}$, with $x$
representing the count of ‘V’s in the string. For each possible length, there
are $C^{x}_{n}$ such choices, which scale as $C^{x}_{n}\simeq\left(2\pi
x\right)^{-1/2}x^{-x}n^{x}e^{x}$ for $x\ll n$, $x\to\infty$, suggesting that
longer paths are also exponentially more abundant.

When $V$ adjusts to $tV$, these lengths would adjust to
$(U^{n},tVU^{n-1},\cdots,t^{n}V^{n})$, i.e., changed by $(1,t,\cdots,t^{n})$
sequentially, highlighting different growth patterns between shorter and
longer paths. On the other hand, the number of these paths remains constant.
This allows us to investigate the response to adjustment of the longer length
scale $V$ for hierarchical, scale-free networks of more general topologies.
Specifically, given a real-world network, we select two hub nodes A and B and
non-overlapping (but potentially intersecting) paths between them (SI, Section
V). These paths are ranked by their lengths and sequentially placed into
different groups, such that each successive group encompasses an exponentially
larger number of longer paths. The lengths of these grouped paths are then
rescaled by $(1,t,t^{2},\cdots)$, reflecting the adjustment of $V$.

We analyzed a segment of the real-world Internet at the autonomous-system
level [11]. By rearranging Eq. (3) and expanding the equation near
$\theta_{\text{th}}(V\to\infty)=\pi/4$, we derive
$V\left[\theta_{\text{th}}(\infty)-\theta_{\text{th}}^{(p)}(V)\right]\simeq
V\left(1-p_{\text{th}}\right)/2\simeq\ln{\left[U/\left(U-1\right)\right]}/2$
as $V\to\infty$, where
$\theta_{\text{th}}^{(p)}\equiv\sin^{-1}{\sqrt{p_{\text{th}}/2}}$. Similarly,
from Eq. (4) we derive
$V\left[\theta_{\text{th}}(\infty)-\theta_{\text{th}}^{(c)}(V)\right]\simeq
V\sqrt{\left(1-c_{\text{th}}\right)/2}\simeq\sqrt{V\ln V/8}$ as $V\to\infty$,
where $\theta_{\text{th}}^{(c)}\equiv\left(\sin^{-1}{c_{\text{th}}}\right)/2$.
The distinct $\theta_{\text{th}}$ dependencies on $V$ (or $t$, assuming $V\sim
t$) are illustrated for both the ($U$,$V$) flower [Fig. 4b] and the Internet
[Fig. 4c]. The resemblance of the dependencies reveals a striking similarity
between the two network topologies, highlighting a shared quantum
characteristic: the slower approach of $\theta_{\text{th}}^{(c)}(V)$ towards
to $\theta_{\text{th}}^{(c)}(V\to\infty)$, which indicates a higher resilience
of concurrence connectivity. This observation underscores the presence of such
resilience in general network topologies, characterized by non-overlapping but
intersecting paths.

Given the very slow corrections $\sim\ln\ln V$ in the quantum critical
exponents (Table 1), this extreme effect becomes eminent only at a very large
length scale, far exceeding the Internet diameter. However, if we consider
“fundamental” networks, such as spin networks in loop quantum gravity [22],
where the length unit is extremely small (the Planck scale $10^{-35}$ m), the
corrections could potentially become detectable.

(a)

(b)

(c)

Figure 4: Real-world path connectivity. (a) Paths between two hub nodes A and
B in real-world networks (e.g., Zachary’s Karate club). Identifying non-
overlapping paths between the hubs enables investigation of how the
percolation critical thresholds ($p_{\text{th}}$ vs. $c_{\text{th}}$) depend
on non-shortest path lengths (governed by the longer length scale, $V$) in
general. (b) $p_{\text{th}}$ and $c_{\text{th}}$ show different scaling
behaviors when translating to $\theta_{\text{th}}$ in ($U$,$V$) flowers. As
$V$ increases, $p_{\text{th}}$ (blue) approaches unity at a constant rate
$\sim V^{-1}$. In contrast, $c_{\text{th}}$ (red) shows a slower rate of
approach ($\sim V^{-1}\ln V$). (c) Corresponding patterns are also observed in
the Internet network topology [11].

##  IV Discussion

We have found analytical expressions for the critical thresholds and exponents
of both classical percolation and concurrence percolation using the ($U,V$)
flowers model, a hierarchical scale-free network structure. Our findings,
which place concurrence percolation in a distinct universality class, rule out
the possibility that differences between classical and concurrence percolation
theories are confined merely to short-range (i.e., ultraviolet) details.
Instead, variations also stem from long-range (i.e., infrared) behaviors. This
distinction underscores the fact that concurrence percolation represents a
fundamentally different statistical framework compared to its classical
counterpart. The question of whether concurrence percolation can be described
by a field theory akin to the $\phi^{3}$ model [23] for classical percolation
remains open. A crucial step in this direction involves defining a cluster-
based order parameter (field) specific to concurrence percolation. Our
alternate definition of the giant “component” $C_{\infty}$ on ($U,V$) flowers
might offer valuable insights for this.

From the perspective of quantum communication, our research indicates that
having a path of maximally entangled states in QN is a _sufficient, but not
necessary criterion_. It is possible to achieve entanglement transmission via
longer, exponentially weaker paths composed of partially entangled links,
provided there is a compensatory increase in the number of such links. This
principle finds a parallel in classical branching processes, where infinite
connectivity is achievable if the growing number of trees with given size
exactly compensates for the exponential decline in the probability of such a
tree existing at the critical point [19]. Further exploration of this concept
through other models, such as the ($U,V,W$) flowers (with three different
length scales), could provide deeper insights on both theoretical and
practical aspects of entanglement distribution efficiency in QNs.

##  V Acknowledgements

S.H. thanks the EU H2020 DIT4Tram (Grant number 953783) and the Horizon Europe
grant OMINO (grant number 101086321).

## References

  * Wehner  _et al._ [2018] S. Wehner, D. Elkouss, and R. Hanson, Quantum internet: A vision for the road ahead, Science 362, eaam9288 (2018).
  * Brito  _et al._ [2020] S. Brito, A. Canabarro, R. Chaves, and D. Cavalcanti, Statistical properties of the quantum internet, Phys. Rev. Lett. 124, 210501 (2020).
  * Acín  _et al._ [2007] A. Acín, J. I. Cirac, and M. Lewenstein, Entanglement percolation in quantum networks, Nat. Phys. 3, 256 (2007).
  * Chitambar and Hsieh [2016] E. Chitambar and M.-H. Hsieh, Relating the resource theories of entanglement and quantum coherence, Phys. Rev. Lett. 117, 10.1103/PhysRevLett.117.020402 (2016).
  * Meng  _et al._ [2021] X. Meng, J. Gao, and S. Havlin, Concurrence Percolation in Quantum Networks, Phys. Rev. Lett. 126, 170501 (2021).
  * Kesten [1980] H. Kesten, The critical probability of bond percolation on the square lattice equals 1/2, Commun. Math. Phys. 74, 41 (1980).
  * Galam and Mauger [1996] S. Galam and A. Mauger, Universal formulas for percolation thresholds, Physical Review E 53, 2177 (1996).
  * Cohen  _et al._ [2002] R. Cohen, D. ben Avraham, and S. Havlin, Percolation critical exponents in scale-free networks, Physical Review E 66, 036113 (2002).
  * Meng  _et al._ [2023] X. Meng, X. Hu, Y. Tian, G. Dong, R. Lambiotte, J. Gao, and S. Havlin, Percolation Theories for Quantum Networks, Entropy 25, 1564 (2023).
  * Malik  _et al._ [2022] O. Malik, X. Meng, S. Havlin, G. Korniss, B. K. Szymanski, and J. Gao, Concurrence percolation threshold of large-scale quantum networks, Commun. Phys. 5, 10.1038/s42005-022-00958-4 (2022).
  * Tilch  _et al._ [2020] G. Tilch, T. Ermakova, and B. Fabian, A multilayer graph model of the internet topology, Int. J. Netw. Virtual Organ. 22, 219 (2020).
  * Rozenfeld  _et al._ [2007] H. D. Rozenfeld, S. Havlin, and D. ben Avraham, Fractal and transfractal recursive scale-free nets, New Journal of Physics 9, 10.1088/1367-2630/9/6/175 (2007).
  * Pirandola [2019] S. Pirandola, End-to-end capacities of a quantum communication network, Communications Physics 2, 1 (2019).
  * Kobayashi  _et al._ [2010] H. Kobayashi, F. Le Gall, H. Nishimura, and M. Rötteler, Perfect quantum network communication protocol based on classical network coding, in  _2010 IEEE International Symposium on Information Theory_ (2010) pp. 2686–2690.
  * Rozenfeld and ben Avraham [2007] H. D. Rozenfeld and D. ben Avraham, Percolation in hierarchical scale-free nets, Physical Review E 75, 10.1103/PhysRevE.75.061102 (2007).
  * Duffin [1965] R. J. Duffin, Topology of series-parallel networks, J. Math. Anal. Appl. 10, 303 (1965).
  * Wierman [1981] J. C. Wierman, Bond percolation on honeycomb and triangular lattices, Adv. Appl. Probab. 13, 298 (1981).
  * Cardy [2001] J. Cardy, Conformal Invariance and Percolation (2001), arxiv:math-ph/0103018 .
  * Christensen and Moloney [2005] K. Christensen and N. R. Moloney,  _Complexity and criticality_ , Vol. 1 (World Scientific Publishing Company, 2005).
  * Kapitulnik  _et al._ [1983] A. Kapitulnik, A. Aharony, G. Deutscher, and D. Stauffer, Introduction to percolation theory, 2nd ed, J. Phys. A 16, L269 (1983).
  * ben Avraham and Havlin [2000] D. ben Avraham and S. Havlin, Diffusion and reactions in fractals and disordered systems, 16, L269 (2000).
  * Ashtekar and Bianchi [2021] A. Ashtekar and E. Bianchi, A short review of loop quantum gravity, Rep. Prog. Phys. 84, 042001 (2021).
  * Amit [1976] D. J. Amit, Renormalization of the Potts model, J. Phys. A: Math. Gen. 9, 1441 (1976).