# Alignment-Enriched Tuning for Patch-Level Pre-trained Document Image Models

Lei Wang1, Jiabang He2, Xing Xu2, Ning Liu3, Hui Liu4 Corresponding Author

###### Abstract

Alignment between image and text has shown promising improvements on patch-
level pre-trained document image models. However, investigating more effective
or finer-grained alignment techniques during pre-training requires a large
amount of computation cost and time. Thus, a question naturally arises: Could
we fine-tune the pre-trained models adaptive to downstream tasks with
alignment objectives and achieve comparable or better performance? In this
paper, we propose a new model architecture with alignment-enriched tuning
(dubbed AETNet) upon pre-trained document image models, to adapt downstream
tasks with the joint task-specific supervised and alignment-aware contrastive
objective. Specifically, we introduce an extra visual transformer as the
alignment-ware image encoder and an extra text transformer as the alignment-
ware text encoder before multimodal fusion. We consider alignment in the
following three aspects: 1) document-level alignment by leveraging the cross-
modal and intra-modal contrastive loss; 2) global-local alignment for modeling
localized and structural information in document images; and 3) local-level
alignment for more accurate patch-level information. Experiments on various
downstream tasks show that AETNet can achieve state-of-the-art performance on
various downstream tasks. Notably, AETNet consistently outperforms state-of-
the-art pre-trained models, such as LayoutLMv3 with fine-tuning techniques, on
three different downstream tasks. Code is available at
https://github.com/MAEHCM/AET.

## Introduction

Self-pretraining techniques aiming to learn generic representations have
recently proved to be highly effective for document image understanding.
Notably, transfer learning based on pre-trained document image models yields
strong performance on various document related downstream tasks (Xu et al.
2020, 2021b; Garncarek et al. 2021; Hong et al. 2022; Wu et al. 2021; Li et
al. 2021a, c, d; Lee et al. 2022; Huang et al. 2022; Li et al. 2022). A
typical self-pretraining method in document image understanding is to pre-
train a model on a large amount of pairs of the document images and OCR texts
with layout information underlying the constraints of unsupervised losses.
Downstream tasks then usually leverage the pre-trained weights for
initialization. After that, the initialized model is fine-tuned for a
downstream task using a task-specific supervised objective.

As shown in Figure 1, a document image contains rich contextual text and
structural information, requiring fine-grained interaction modeling between
image and text. Although self-supervision achieves large progress in document
image related-tasks, most existing pre-trained document image models are
trained with coarse self-supervised losses, which ignore fine-grained
interaction modeling between image and text. Inspired by this, DocFormer
(Appalaraju et al. 2021) learns to reconstruct image pixels through a CNN
decoder, SelfDoc (Li et al. 2021c) proposes to regress the masked region
feature, and the latest model LayoutLMv3 (Huang et al. 2022) introduces a
word-patch alignment objective by reconstructing masked patch tokens of the
visual modality. Although alignment between image and text has shown promising
improvements on large-scale self-supervised pre-trained document image models,
investigating more effective or finer-grained alignment techniques during the
pre-training requires huge computational cost and time. Thus, a question
naturally arises: Could we fine-tune the pre-trained models adaptive to
downstream tasks with more alignment objectives and achieve comparable or
better performance?

Figure 1: A document image example containing rich contextual text and layout
information. Figure 2: Fine-tuning requires tuning the entire pre-trained
model for each downstream task. Prompt-tuning only requires tuning a trivial
amount of extra parameters for each downstream task. Alignment-enriched tuning
adds an extra alignment-aware image encoder and an extra alignment-aware text
encoder before the multimodal pre-trained model. Alignment-enriched tuning
trains the entire parameters together for each downstream task.

In recent few years, with the wide success of large-scale pre-trained models,
a range of tuning techniques has arisen to adapt these general-purpose models
to downstream tasks (Howard and Ruder 2018; Jiang et al. 2019; Houlsby et al.
2019; Gururangan et al. 2020; Xu et al. 2021a; Liu et al. 2021a, b; Li and
Liang 2021). The model tuning strategy (i.e., fine-tuning) (Howard and Ruder
2018) tunes all model parameters during adaptation. To improve training
efficiency, adapters (Houlsby et al. 2019) and prompt tuning (Li and Liang
2021; Liu et al. 2021b, a) tune pre-trained models by a trivial amount of
parameters, but they may suffer a small performance degradation compared with
fine-tuning. In addition, task-adaptive pre-training (Gururangan et al. 2020)
continues to train the pre-trained models with the unlabeled data of specific
tasks. However, it is inconvenient to incorporate alignment loss into existing
tuning strategies due to the limitation of fixed model architectures of pre-
trained models.

To enable the pre-trained models to have the ability of modeling alignment,
ALBEF (Li et al. 2021b) and TCL (Yang et al. 2022) encode the document image
and text independently with an extra image encoder and an extra text encoder
before fusion by a multimodal encoder. The model is then pre-trained with
alignment-aware strategies to improve the learned features’ expressiveness,
which is essential for joint multi-modal representation learning. However, as
mentioned before, investigating more effective or finer-grained alignment
techniques during the pre-training stage requires much computation cost and
time.

To bridge the above research gap, in this paper, we propose a new model
architecture with alignment-enriched tuning (termed AETNet), which tunes pre-
trained document image models adaptively to enable downstream tasks with the
joint task-specific supervised and alignment-aware contrastive objective. As
shown in Figure 2, compared with fine-tuning and prompt-tuning, in our
proposed AETNet, we introduce an extra visual transformer as the alignment-
ware image encoder and an extra text transformer as the alignment-ware text
encoder before multimodal fusion. We consider alignment in the following three
aspects: 1) document-level alignment by leveraging the cross-modal and intra-
modal contrastive loss; 2) global-local alignment for modeling localized and
structural information in document images; and 3) local-level alignment for
more accurate patch-level information.

We evaluate our AETNet method on various downstream document image
understanding tasks, including FUNSD (Jaume, Ekenel, and Thiran 2019) for form
understanding, CORD (Park et al. 2019) for receipt Understanding, DocVQA
(Mathew, Karatzas, and Jawahar 2021) for document visual question answering,
and a sampled subset RVL-CDIP-1 from RVL-CDIP (Harley, Ufkes, and Derpanis
2015) for document image classification. In terms of performance, the proposed
AETNet method consistently outperforms existing state-of-the-art pre-trained
models with fine-tuning on various downstream tasks. Notably, with the help of
alignment-enriched tuning, AETNet achieves better performance compared with
general fine-tuning and prompt tuning strategies. We also carry out ablation
studies with detailed analysis to investigate the effectiveness of each
alignment loss in AETNet. Lastly, we conduct case studies over real examples
from the FUNSD task to show that AETNet benefits from the alignment modeling.

## Related Work

Figure 3: An overview of the framework of AETNet. The model architecture
consists of a vision transformer, a text transformer, and a fusion encoder.
Image and text encoders have their paired momentum encoder updated by the
momentum-based moving average. The alignment objective contains four
contrastive losses (i.e., DITC, IMC, GLITC and PITA) for both alignment-
enriched cross-modal and intra-modal representation learning.

#### Multimodal Pre-training

Multimodal self-supervised pre-training has been successfully applied in
document images through effectively leveraging image, layout, contextual text
information (Xu et al. 2020, 2021b; Garncarek et al. 2021; Powalski et al.
2021; Wu et al. 2021; Li et al. 2021a, c; Appalaraju et al. 2021; Li et al.
2021d; Wang, Jin, and Ding 2022; Gu et al. 2022; Kim et al. 2022). LayoutLM
and its following works consider the layout information as a type of two-
dimensional positional vectors and fuse their transformed vectors with text
embeddings for the multimodal pre-trained model (Xu et al. 2020; Li et al.
2021a; Hong et al. 2022; Lee et al. 2022). Some works extract CNN grid
features (Xu et al. 2021b; Appalaraju et al. 2021) and some (Xu et al. 2020;
Powalski et al. 2021; Li et al. 2021c; Gu et al. 2021) rely on object
detectors to extract region features. However, these works are either limited
by heavy computation bottleneck or require region supervision. Recently
researchers make many efforts to overcome the above limitations caused by CNN.
Inspired by Vision Transformer (ViT) (Dosovitskiy et al. 2021), most rely on
separate self-attention networks to learn visual features to reduce
computational cost (Xue et al. 2021; Li et al. 2021b; Dou et al. 2021). For
instance, ViLT (Kim, Son, and Kim 2021), one of works in vision-and-language
pre-training utilizing ViT, learns visual features with a lightweight linear
layer and significantly reduce the model parameters and running time.
Following ViLT, LayoutLMv3 (Huang et al. 2022) is the first work to take
advantage of patch-level embeddings without CNNs in Document images. For
modeling fine-grained interaction and alignment between document image and
their OCR text, we introduce alignment-aware ViT (Dosovitskiy et al. 2021) and
RoBERRTa (Liu et al. 2019) encoders before fusion.

#### Tuning Techniques

Fine-tuning is the useful paradigm for tuning large pretrained language models
for downstream tasks (Devlin et al. 2019). In recent few years, different
fine-tuning techniques have been proposed. Model tuning (Howard and Ruder
2018) requires tuning the entire pre-trained model for each downstream task.
Mixout (Lee, Cho, and Kang 2019) randomly replaces part of the model
parameters with the pre-trained weights during fine-tuning. Child-Tuning (Xu
et al. 2021a) updates parameters within the child network via a gradient mask.
There are also studies focusing on parameter-efficient fine-tuning, such as
adapter-based methods (Houlsby et al. 2019), and the prompt tuning methods (Li
and Liang 2021; Liu et al. 2021b, a). These works tune pre-trained models by a
trivial amount of parameters while they may suffer a small performance
degradation compared with fine-tuning. Beyond, it is inconvenient to
incorporate alignment loss into these existing tuning strategies due to the
limitation of fixed model architectures of pre-trained models.

## Methodology

In this section, we present a new model architecture with the proposed
alignment-enriched tuning (AETNet) for the transfer of the patch-level pre-
trained models in document images to downstream tasks. We first introduce the
model architecture, followed by the tuning process. Then we detail the tuning
objectives including downstream task specific supervised objective, document-
level image-text contrastive objective, intra-modal contrastive objective,
global-local image-text contrastive objective, and patch-level image-text
alignment objective.

### Model Architecture

The model architecture of our method AETNet is shown in Figure 3. The model
architecture consists of a large-scale patch-level multimodal pre-trained
model, an extra alignment-aware image encoder, and an extra alignment-aware
text encoder. For the multimodal pre-trained model, we employ LayoutLMv3
(Huang et al. 2022) as our pre-trained model because of not only its state-of-
the-art performance on downstream tasks but also its patch-level image encoder
that is suitable for the AETNet process. We use a RoBERTa (Liu et al. 2019) as
the extra alignment-aware text encoder. Likewise, we use a ViT (Dosovitskiy et
al. 2021) as the extra alignment-aware image encoder, which is initialized
using weights pre-trained on ImageNet-1k from DeiT (Touvron et al. 2021). The
implementation details of extra text and image encoders, are described in
Experiment Section.

### Tuning Process

In the downstream tuning process, our model is trained with the joint loss of
downstream task specific objective and the proposed alignment-enriched
objective. As shown in Figure 3, an input document image $I$ is first encoded
into a sequence of patch-level visual representation vectors:
$\\{v_{cls},v_{1},...,v_{N}\\}$ through the alignment-aware ViT, where $cls$
denotes the special token [CLS] and $N$ is the number of patches. Meanwhile,
the input text (i.e., the textual content of this document image, which is
obtained by the open-source OCR toolkit,
Tesseract111https://github.com/tesseract-ocr/tesseract) is fed into the
alignment-aware RoBERTa to be transformed into a sequence of text token
representation vectors: $\\{w_{cls},w_{1},...,w_{L}\\}$, where $L$ is the
length of the input text tokens. Before fusion with LayoutLMv3, we compute
alignment-aware losses based on the obtained representations in terms of
alignment-aware objectives to let the obtained representations preserve
alignment information. Then, we derive the patch-level input image embeddings
for LayoutLMv3 by the sum of obtained alignment-enriched image representation
vectors and original input image embeddings of LayoutLMv3. Likewise, input
text embeddings for LayoutLMv3 are derived by the sum of obtained alignment-
enriched text representation vectors and original input text embeddings of
LayoutLMv3. Lastly, LayoutLMv3 transforms the newly fused embeddings into
output hidden representation for computing supervised task-specific loss.

### Tuning Objectives

The full tuning objective of AETNet consists of five objectives: the
downstream task-specific subjective objective (SO), global-level cross-modal
alignment (GCMA), global-level intra-modal contrastive (GIMC), Global-Local MI
Maximization (GLMI), and Local-level Cross-Modal Alignment (LLCMA), shown as
below:

$\mathcal{L}_{aet}=\mathcal{L}_{so}+\mathcal{L}_{ditc}+\mathcal{L}_{imc}+\mathcal{L}_{glitc}+\mathcal{L}_{pita}.$
(1)

In the following, we elaborate each objective in details.

#### Supervised Objective (SO)

In the vanilla fine-tuning framework, the model is first initialized with the
parameters of large pre-trained models (LPM). Then, the initialized model is
fine-tuned for a certain downstream task using the task-specific objective on
the corresponding dataset. Unlike vanilla fine-tuning, our model consists of
three modules: LPM, the alignment-aware image encoder, and the alignment-aware
text encoder. We train these three modules together in our model with the
task-specific objective: $\mathcal{L}_{so}(I,T,Y)$, where image $I$ and OCR
text $T$ are the input and $Y$ is for the ground-truth labels. We take the
semantic entity labeling task (Jaume, Ekenel, and Thiran 2019; Park et al.
2019), one of the downstream tasks mentioned in Experiment Section, as an
example. $L_{so}$ is the cross entropy loss based on OCR tokens’ predictions
and their corresponding ground-truth labels.

#### Document-Level Image-Text Contrastive (DITC)

The purpose of DITC is to learn better document(global)-level alignment cross-
modality through a contrastive loss, which pulls representations of matched
image-text pairs to be close and pushes the unmatched ones apart. To model
global information, we apply similarity functions
$sim(v_{cls},\tilde{w}_{cls})=f_{img}(v_{cls})^{\top}f_{txt}(\tilde{w}_{cls})$
and
$sim(w_{cls},\tilde{v}_{cls})=f_{text}(w_{cls})^{\top}f_{img}(\tilde{v}_{cls})$
for the DITC loss, where $f_{img}$ and $f_{txt}$ are two projection heads for
mapping hidden vectors into low-dimensional representations in the contrasitve
loss space and $v_{cls}$ and $w_{cls}$ are two [CLS] vectors from alignment-
aware image and text encoders, respectively. Next, we refer to MoCo (He et al.
2020) and ALBEF (Li et al. 2021b) to maintain two memory queues (i.e.,
$\tilde{I}_{cls}=\\{\tilde{v}^{1}_{cls},...,\tilde{v}^{K}_{cls}\\}$ for
images, $\tilde{T}_{cls}=\\{\tilde{w}^{1}_{cls},...,\tilde{w}^{K}_{cls}\\}$
for text) to record the most recent K image and text representations from the
momentum alignment-aware encoders. The document-level contrastive loss for a
pair of document image $I$ and OCR text $T$ can be defined as follows:

$\displaystyle\mathcal{L}_{cl}(v_{cls},\tilde{w}_{cls},\tilde{T}_{cls})=-\log\frac{\exp(sim(v_{cls},\tilde{w}_{cls})/\tau)}{\sum_{k=1}^{K}\exp(sim(v_{cls},\tilde{w}^{k}_{cls})/\tau)},$
(2)

where $\tau$ is the learnable temperature rate. Considering image-to-text and
text-to-image together, DITC loss is defined as:

$\displaystyle\mathcal{L}_{ditc}=\frac{1}{2}\mathbb{E}_{(I,T)\sim
B}\big{[}\mathcal{L}_{cl}(v_{cls},\tilde{w}_{cls},\tilde{T}_{cls})+$ (3)
$\displaystyle\mathcal{L}_{cl}(w_{cls},\tilde{v}_{cls},\tilde{I}_{cls})\big{]},$

where $B$ is a batch of image-text pairs. In addition, We follow ALBEF (Li et
al. 2021b) to guide the training of DITC loss by pseudo-targets generated by
the momentum model.

#### Intra-Modal Contrastive (IMC)

The goal of IMC is to learn more accurate representations within the same
modality. For the visual modality, we generate positive latent image
representation $\\{v^{+}_{cls},v^{+}_{1},...,v^{+}_{N}\\}$ for the anchor
augmented image $I^{+}$ by feeding the anchor image into the momentum
alignment-aware image encoder. For the text modality, the positive latent
representation $\\{w^{+}_{cls},w^{+}_{1},...,w^{+}_{L}\\}$ for the anchor text
$T$ is produced by the momentum model in the same way as image samples. Two
momentum queues used for the DITC loss also provide negative samples for the
IMC loss. The IMC loss is define as follows:

$\displaystyle\mathcal{L}_{imc}=\frac{1}{2}\mathbb{E}_{(I,T)\sim
B}\big{[}\mathcal{L}_{cl}(v_{cls},v^{+}_{cls},\tilde{I}_{cls})+$ (4)
$\displaystyle\mathcal{L}_{cl}(w_{cls},w^{+}_{cls},\tilde{T}_{cls})\big{]}.$

#### Global-Local Image-Text Contrastive (GLITC)

Inspired by TCL (Yang et al. 2022), we introduce the GLITC loss to capture
localized and structural information within the single modality by modeling
interactions between the document-level representation and local regions.
Specifically, for the visual modality, we use global representation $v_{cls}$
and compute the contrastive loss with momentum image patch
representations$\\{{\widetilde{v}}_{2},\ldots,{\widetilde{v}}_{N}\\}$.
Likewise, for the the text modality, we use the text [CLS] representation
$w_{cls}$ and momentum text token representations
$\\{{\widetilde{w}}_{2},\ldots,{\widetilde{w}}_{L}\\}$ for computing loss. We
encourage to model interactions between global information and lcoal
information within the same modality by minimizing the following two-side
interaction contrastive loss:

$\displaystyle\mathcal{L}_{glitc}=\frac{1}{2}\mathbb{E}_{(I,T)\sim
B}\big{[}\frac{1}{N}\sum^{N}_{i=1}\mathcal{L}_{cl}(v_{cls},\tilde{v}_{i},I^{-}_{B})+$
(5)
$\displaystyle\frac{1}{L}\sum^{L}_{j=1}\mathcal{L}_{cl}(w_{cls},\tilde{w}_{j},T^{-}_{B})\big{]},$

where $I^{-}_{B}$ and $T^{-}_{B}$ are negative image patch and text token
representation vectors, respectively. These negatives are from other data
examples in the same batch $B$.

#### Patch-Level Image-Text Alignment (PITA)

Although the introduced DPITC loss has the ability to capture certain cross-
modal localized and structural information in the input, it ignores patch-
level alignment between patch-level images and patch-level contextual text.
Patch-level alignment is critical to learning more accurate and finer-grained
representations for more accurate document understanding. Therefore, we
propose a novel patch-level alignment loss as a complement to the previous
three contrastive losses. As shown in Figure 1, to compute the patch-level
alignment loss, we first find text tokens matching image patches by a rule-
based strategy that locates text in the image patches by layout information
obtained from the OCR toolkit. Then, we average the representation vectors of
the matched text tokens to derive patch-level text representations for the
matched text $\\{t_{1},t_{2},...,t_{N}\\}$. The PITA loss is defined below:

$\mathcal{L}_{pita}=-\frac{1}{N}\big{(}\sum^{N}_{i=1}\frac{\langle
v_{i},t_{i}\rangle}{||v_{i}||\cdot||t_{i}||}\big{)}.$ (6)

## Experiment

### Experiment Setting

#### Model Configures

Since we aim to investigate the impact of the proposed fine-grained alignment
losses during the tuning process, we thus adopt the patch-level multimodal
pre-trained model (LayoutLMv3 (Huang et al. 2022)) instead of document-level
pretrained models as our fusion model. LayoutLMv3 is pre-trained on 11 million
document images of a large IIT-CDIP (Harley, Ufkes, and Derpanis 2015)
dataset. LayoutLMv3${}_{\text{BASE}}$ uses a 12-layer Transformer (Vaswani et
al. 2017) Encoder with 12 self attention heads, 768 hidden size, and 3,072
intermediate size for feed-forward network. LayoutLMv3${}_{\text{BASE}}$ uses
a 24-layer Transformer (Vaswani et al. 2017) Encoder with 16 self attention
heads, 1,024 hidden size, and 4,096 intermediate size for feed-forward
network. Patch size in LayoutLMv3 is 16. The alignment-aware ViT and RoBERTa
encoders before fusion are initialized with the model parameters released by
DeiT (Touvron et al. 2021) and RoBERTa (Liu et al. 2019), respectively. The
size of the momentum queues of all encoders is 65536.

#### Downstream Tasks

We evaluate the proposed alignment-enriched tuning (AET) method on three
document multimodal downstream tasks as follows:

Form and Receipt Understanding is a sequential labeling task, which aims to
assign a label to each word. Form and receipt understanding requires an
understanding of visual information and textual content extracted from
structural forms in document images. We follow fine-tuning settings in
LayoutLMv3 to evaluate AET on two public datasets, i.e., FUNSD (Jaume, Ekenel,
and Thiran 2019) and CORD (Park et al. 2019). FUNSD is a dataset sampled from
the RVL-CDIP dataset (Harley, Ufkes, and Derpanis 2015) about noisy scanned
form understanding. It consists of 199 documents (149 documents for training
and 50 documents for testing) and 9,707 semantic entities. CORD is a receipt
key information extraction dataset, including 1,000 receipts and 30 semantic
labels defined under 4 categories, where 800 samples are used for training,
100 for validation, and 100 for testing.

Document Visual Question Answering is to predict the answer given an document
image and a question, which requires models understanding the knowledge in
documents and learning to reason over documents to answer. We follow the task
formulation in LayoutLMv3, considering this task as an extractive QA problem.
Specifically, the model is required predicting start and end positions of the
answer in the document. This is a binary classification over each text token.
We follow the official partition of the DocVQA (Mathew, Karatzas, and Jawahar
2021) dataset, which consists of 10,194/1,286/1,287 images with
39,463/5,349/5,188 questions for training/validation/test, respectively.

Document Image Classification is a document classification task aiming to
predict the category of a given document, which requires of an understanding
of document contents. Due to the limitations of our servers, we sampled a
subset from RVL-CDIP (Harley, Ufkes, and Derpanis 2015), termed RVL-CDIP-1 ,
including 10,000 data examples with 16 categories from the RVL-CDIP datasets
of 400,000 document images as our evaluation dataset. RVL-CDIP-1 is divided
into 8000 training samples, 1000 validation samples, and 1000 test samples.

### AETNet on Downstream Tasks

Model | FUNSD | CORD
---|---|---
F1$\uparrow$ | F1$\uparrow$
$\textrm{BERT}_{base}$ (Devlin et al. 2019) | 60.26 | 89.68
$\textrm{RoBERTa}_{base}$ (Liu et al. 2019) | 66.48 | 93.54
$\textrm{BROS}_{base}$ (Hong et al. 2022) | 83.05 | 95.73
UDoc (Gu et al. 2021) | 87.93 | 98.94†
$\textrm{LayoutLMv2}_{base}$ (Xu et al. 2021b) | 82.76 | 94.95
$\textrm{DocFormer}_{base}$ (Appalaraju et al. 2021) | 83.34 | 96.33
LayoutLMv3base (Huang et al. 2022) | 89.82 | 95.97
AETNetbase (Ours) | 91.55 | 97.04
$\textrm{BERT}_{large}$ (Devlin et al. 2019) | 65.63 | 90.25
$\textrm{RoBERTa}_{large}$ (Liu et al. 2019) | 70.72 | 93.80
$\textrm{BROS}_{large}$ (Hong et al. 2022) | 84.52 | 97.40
$\textrm{LayoutLMv2}_{large}$ (Xu et al. 2021b) | 84.20 | 96.01
$\textrm{DocFormer}_{large}$ (Appalaraju et al. 2021) | 84.55 | 96.99
LayoutLMv3large (Huang et al. 2022) | 90.94 | 97.01
AETNetlarge (Ours) | 92.33 | 97.52

Table 1:  Performance comparison with existing published pre-trained models
with fine-tuning on FUNSD and CORD datasets. The results of LayoutLMv3 are
based on our implementations with the released UniLM (Dong et al. 2019). The
score† is not directly comparable to other scores.

The detailed description of hyper-parameters, including running epochs,
learning rate, batch size, and optimizer, for our method on three downstream
tasks and four datasets, are referred to in the supplementary.

#### Evaluation on Form and Receipt Understanding

We compare the proposed model architecture with alignment-enriched tuning
(AETNet) to three types of self-supervised pre-training approaches using fine-
tuning: (1) Pre-trained models only using text modality. BERT (Devlin et al.
2019) and RoBERTa (Liu et al. 2019) is based on Transformer architecture and
only use text information; (2) Pre-trained models with text and layout
modalities. BROS (Hong et al. 2022) incorporates layout information by
encoding relative layout positions; (3) Pre-trained models with text, layout
and image modalities. UDoc (Gu et al. 2021) adopts object proposals from
document images and concatenate region features and text embeddings.
LayoutLMv2 (Xu et al. 2021b) use a CNN network to extract image features and
then feed it with layout and text information into multimodal Transformer.
DocFormer (Appalaraju et al. 2021) extract image features with CNN. LayoutLMv3
(Huang et al. 2022) replaces CNN backbones with patch-level linear embedding
layers.

Table 1 reports the results of the comparison AETNet to fine-tuning for
existing published pre-trained models on FUNSD and CORD datasets. Our APTNet
achieves state-of-the-art performance while outperforming the previous fine-
tuning based methods. For transferring the pre-trained model to the downstream
tasks, we compared AETNetbase to the reported fine-tuning for LayoutLMv3base
(Huang et al. 2022) and our implemented fine-tuning for LayoutLMv3base.
AETNetbase improves $+1.25$ and $+1.73$ on FUNSD and $+0.51$ and $+1.07$ on
CORD, revealing the necessity of conducting document-level and patch-level
intra-modal and cross-modal alignment after pre-training.

#### Evaluation on Document Visual Question Answering

Model | DocVQA
---|---
ANLS$\uparrow$
$\textrm{BERT}_{base}$ (Devlin et al. 2019) | 63.72
$\textrm{RoBERTa}_{base}$ (Liu et al. 2019) | 66.42
$\textrm{LayoutLMv2}_{base}$ (Xu et al. 2021b) | 78.08
LayoutLMv3base (Huang et al. 2022) | 78.76
AETNetbase (Ours) | 79.73
Table 2:  Performance comparison on DocVQA dataset.  Figure 4: Performance
comparison on the sampled subset RVL-CDIP-1 dataset.

#Pre-train | Training tasks | FUNSD | CORD
---|---|---|---
Models | F1$\uparrow$ | F1$\uparrow$
LayoutLMv3base | Fine Tuning | 89.82 | 95.97
P-Tuning (Liu et al. 2021a) | 83.75 | 87.37
AETNet | 91.55 | 97.04

Table 3:  Comparison to different adaptation approaches on FUNSD and CORD
datasets. Fine Tuning: All pre-trained parameters are tuned with the
supervised objective. P-Tuning: Prompt parameters of each layer are tuned with
the supervised objective. AETNet: Pre-trained parameters and extra alignment-
aware image and text encoders are tuned with the supervised objective and our
proposed alignment loss. All results are based on our implementation.

Tuning Objective | FUNSD | CORD
---|---|---
F1$\uparrow$ | F1$\uparrow$
LayoutLMv3base (Fine-tuning) | 89.82 | 95.97
AETNetbase (w/ Supervised Objective) | 89.77 | 96.30
\+ DITC | 90.44 | 96.48
\+ IMC | 90.45 | 96.61
\+ GLITC | 90.18 | 96.37
\+ PITA | 90.69 | 96.78
AETNetbase | 91.55 | 97.04

Table 4:  Ablation study of each loss in the AETNet on FUNSD and CORD
datasets. The F1 is reported. DITC, IMC, DLITC, and PITA means Document-Level
Image-Text Contrastive, Intra-Modal Contrastive, Global-Local Image-Text, and
Patch-Level Image-Text Alignment, respectively. All results are obtained by
our implementation.  Figure 5: Visualization of two cases on FUNSD , which are
predicted by LayoutLMv3 (Column (a)) and AETNet (Column (b). Column (c) is the
Ground-truth annotations.

Table 2 reports the performance comparison on the DocVQA dataset. We compare
AETNetbase to methods only requiring text modality, i.e., BERTbase,
RoBERTabase, and methods requiring image and text as inputs, i.e.,
LayoutLMv2base (Xu et al. 2021b) and LayoutLMv3base (Huang et al. 2022).
AETNetbase substantially outperforms existing baseline methods, achieving an
absolute accuracy of $+0.97$ boost compared to LayoutLMv3base.

#### Evaluation on Document Image Classification

Figure 4 shows the performance comparison on the sampled subset classification
dataset, RVL-CDIP-1. We compare AETNet with the general fine-tuning and
prompt-tuning strategies for LayoutLMv3 in both base and large model sizes.
Although prompt-tuning is more parameter efficient, it cannot outperform fine-
tuning and alignment-enriched tuning. Likewise, fine-tuning outperforms prompt
tuning but is worse than alignment-enriched tuning. Overall, in terms of
performance, AETNet can achieve desirable results on document image
classification with the help of alignment modeling.

### Intrinsic Analysis

To learn the effectiveness of each component in AETNet, we first compare
AETNet with the general fine-tuning strategy and a commonly-used prompt tuning
strategy. Then, we perform ablation study of each loss in the full objective
on FUNSD and CORD datasets. Lastly, we do a case study to demonstrate the
ability of AETNet to model alignment information.

#### AETNet vs Other Tuning Methods

As shown in Table 3, we compare AETNet with fine-tuning based LayoutLMv3base
and P-tuning based LayoutLMv3base on FUNSD and CORD datasets. With the help of
alignment-enriched tuning, AETNet achieves the best performance among these
three tuning strategies. Specifically, AETNet achieves absolute improvements
of $+1.73$ points over fine-tuning and $+7.79$ over prompt-tuning on FUNSD,
and $+1.07$ points over fine-tuning and $+9.67$ over prompt-tuning on CORD. It
indicates that alignment-enriched tuning has the ability to improve the
quality of representations when transferring pre-trained models to downstream
tasks.

#### Ablation Study of each Loss in AETNet

To learn the effectiveness of the newly proposed losses (i.e., DITC, IMC,
GLITC, and PITA) in improving the document multi-modal representation
learning, we perform an ablation study of each each loss in AETNet on FUNSD
and CORD datasets. Table 4 report the results. Firstly, it is worth mentioning
that AETNetbase using supervised objective is slightly worse than
LayoutLMv3base using supervised objective. In other words, adding more
parameters during the fine-tuning process in this study cannot improve
performance. This observation further verifies that downstream tasks benefit
from alignment-enriched representations learned by our proposed alignment-
enriched objective during the tuning process. Secondly, combining with any
newly proposed loss in AETNetbase improves the performance on FUNSD and CORD,
which can be attributed to the consideration of document-level or patch-level
alignment-enriched representations.

#### Case Study

Figure 5 shows the visualization of two cases predicted by LayoutLMv3 and
AETNet. In case 1, AETNet predict three entities within the reb box as
“question” but LayoutLMv3 predict them as “header”. The possible reason is
that LayoutLMv3 makes the prediction based on the layout information rather
than content text, which requires models to have more accurate interaction and
alignment modeling ability to avoid this error. In case 2, LayoutLMv3 learns
from two neighbors above it to predict the entity within the red box as
“answer”, which indicates that the model utilizes too much layout information
to make predictions rather than based on understanding it.

## Conclusion

This paper proposes a new model architecture with alignment-enriched tuning
method named AETNet for transferring multimodal pre-trained models in document
images. AETNet consists of an extra visual transformer as the alignment-ware
image encoder and an extra text transformer as the alignment-ware text
encoder, and a multimodal fusion encoder initialized by pre-trained model
parameters. Based on this model architecture, AETNet tunes pre-trained models
for downstream tasks with joint task-specific supervised and alignment-aware
contrastive objective. AETNet achieves consistent improvements over existing
state-of-the-art pre-trained models with fine-tuning and different commonly-
used tuning techniques over various downstream tasks, which indicates the
effectiveness of our proposed AETNet.

## Acknowledgments

This work was supported in part by National Natural Science Foundation of
China under Grants (No. 61976049).

## References

  * Appalaraju et al. (2021) Appalaraju, S.; Jasani, B.; Kota, B. U.; Xie, Y.; and Manmatha, R. 2021.  DocFormer: End-to-End Transformer for Document Understanding.  In _ICCV_.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL_.
  * Dong et al. (2019) Dong, L.; Yang, N.; Wang, W.; Wei, F.; Liu, X.; Wang, Y.; Gao, J.; Zhou, M.; and Hon, H.-W. 2019.  Unified language model pre-training for natural language understanding and generation.  _Advances in Neural Information Processing Systems_ , 32.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _ICLR_.
  * Dou et al. (2021) Dou, Z.-Y.; Xu, Y.; Gan, Z.; Wang, J.; Wang, S.; Wang, L.; Zhu, C.; Liu, Z.; Zeng, M.; et al. 2021.  An Empirical Study of Training End-to-End Vision-and-Language Transformers.  _arXiv preprint arXiv:2111.02387_.
  * Garncarek et al. (2021) Garncarek, Ł.; Powalski, R.; Stanisławek, T.; Topolski, B.; Halama, P.; Turski, M.; and Graliński, F. 2021.  LAMBERT: Layout-Aware Language Modeling for Information Extraction.  In _ICDAR_.
  * Gu et al. (2021) Gu, J.; Kuen, J.; Morariu, V.; Zhao, H.; Jain, R.; Barmpalios, N.; Nenkova, A.; and Sun, T. 2021.  UniDoc: Unified Pretraining Framework for Document Understanding.  In _NeurIPS_.
  * Gu et al. (2022) Gu, Z.; Meng, C.; Wang, K.; Lan, J.; Wang, W.; Gu, M.; and Zhang, L. 2022.  XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understanding.  In _CVPR_.
  * Gururangan et al. (2020) Gururangan, S.; Marasović, A.; Swayamdipta, S.; Lo, K.; Beltagy, I.; Downey, D.; and Smith, N. A. 2020.  Don’t stop pretraining: adapt language models to domains and tasks.  _arXiv preprint arXiv:2004.10964_.
  * Harley, Ufkes, and Derpanis (2015) Harley, A. W.; Ufkes, A.; and Derpanis, K. G. 2015.  Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval.  In _ICDAR_.
  * He et al. (2020) He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 9729–9738.
  * Hong et al. (2022) Hong, T.; Kim, D.; Ji, M.; Hwang, W.; Nam, D.; and Park, S. 2022.  BROS: A Pre-Trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents.  In _AAAI_.
  * Houlsby et al. (2019) Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , 2790–2799. PMLR.
  * Howard and Ruder (2018) Howard, J.; and Ruder, S. 2018.  Universal language model fine-tuning for text classification.  _arXiv preprint arXiv:1801.06146_.
  * Huang et al. (2022) Huang, Y.; Lv, T.; Cui, L.; Lu, Y.; and Wei, F. 2022.  LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking.  _arXiv preprint arXiv:2204.08387_.
  * Jaume, Ekenel, and Thiran (2019) Jaume, G.; Ekenel, H. K.; and Thiran, J.-P. 2019.  Funsd: A dataset for form understanding in noisy scanned documents.  In _ICDARW_.
  * Jiang et al. (2019) Jiang, H.; He, P.; Chen, W.; Liu, X.; Gao, J.; and Zhao, T. 2019.  Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  _arXiv preprint arXiv:1911.03437_.
  * Kim et al. (2022) Kim, G.; Hong, T.; Yim, M.; Nam, J.; Park, J.; Yim, J.; Hwang, W.; Yun, S.; Han, D.; and Park, S. 2022.  OCR-Free Document Understanding Transformer.  In _European Conference on Computer Vision (ECCV)_.
  * Kim, Son, and Kim (2021) Kim, W.; Son, B.; and Kim, I. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _ICML_.
  * Lee, Cho, and Kang (2019) Lee, C.; Cho, K.; and Kang, W. 2019.  Mixout: Effective regularization to finetune large-scale pretrained language models.  _arXiv preprint arXiv:1909.11299_.
  * Lee et al. (2022) Lee, C.-Y.; Li, C.-L.; Dozat, T.; Perot, V.; Su, G.; Hua, N.; Ainslie, J.; Wang, R.; Fujii, Y.; and Pfister, T. 2022.  FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction.  In _ACL_.
  * Li et al. (2021a) Li, C.; Bi, B.; Yan, M.; Wang, W.; Huang, S.; Huang, F.; and Si, L. 2021a.  StructuralLM: Structural Pre-training for Form Understanding.  In _ACL_.
  * Li et al. (2021b) Li, J.; Selvaraju, R.; Gotmare, A.; Joty, S.; Xiong, C.; and Hoi, S. C. H. 2021b.  Align before fuse: Vision and language representation learning with momentum distillation.  In _NeurIPS_.
  * Li et al. (2022) Li, J.; Xu, Y.; Lv, T.; Cui, L.; Zhang, C.; and Wei, F. 2022.  DiT: Self-supervised Pre-training for Document Image Transformer.  _arXiv preprint arXiv:2203.02378_.
  * Li et al. (2021c) Li, P.; Gu, J.; Kuen, J.; Morariu, V. I.; Zhao, H.; Jain, R.; Manjunatha, V.; and Liu, H. 2021c.  SelfDoc: Self-Supervised Document Representation Learning.  In _CVPR_.
  * Li and Liang (2021) Li, X. L.; and Liang, P. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_.
  * Li et al. (2021d) Li, Y.; Qian, Y.; Yu, Y.; Qin, X.; Zhang, C.; Liu, Y.; Yao, K.; Han, J.; Liu, J.; and Ding, E. 2021d.  StrucTexT: Structured Text Understanding with Multi-Modal Transformers.  In _ACM Multimedia_.
  * Liu et al. (2021a) Liu, X.; Ji, K.; Fu, Y.; Du, Z.; Yang, Z.; and Tang, J. 2021a.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  _arXiv preprint arXiv:2110.07602_.
  * Liu et al. (2021b) Liu, X.; Zheng, Y.; Du, Z.; Ding, M.; Qian, Y.; Yang, Z.; and Tang, J. 2021b.  GPT understands, too.  _arXiv preprint arXiv:2103.10385_.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mathew, Karatzas, and Jawahar (2021) Mathew, M.; Karatzas, D.; and Jawahar, C. 2021.  Docvqa: A dataset for vqa on document images.  In _WACV_.
  * Park et al. (2019) Park, S.; Shin, S.; Lee, B.; Lee, J.; Surh, J.; Seo, M.; and Lee, H. 2019.  CORD: A Consolidated Receipt Dataset for Post-OCR Parsing.  In _Document Intelligence Workshop at Neural Information Processing Systems_.
  * Powalski et al. (2021) Powalski, R.; Łukasz Borchmann; Jurkiewicz, D.; Dwojak, T.; Pietruszka, M.; and Pałka, G. 2021.  Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer.  In _ICDAR_.
  * Touvron et al. (2021) Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and Jégou, H. 2021.  Training data-efficient image transformers & distillation through attention.  In _ICML_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NeurIPS_.
  * Wang, Jin, and Ding (2022) Wang, J.; Jin, L.; and Ding, K. 2022.  LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding.  In _ACL_.
  * Wu et al. (2021) Wu, T.-L.; Li, C.; Zhang, M.; Chen, T.; Hombaiah, S. A.; and Bendersky, M. 2021\.  LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding.  _arXiv preprint arXiv:2104.08405_.
  * Xu et al. (2021a) Xu, R.; Luo, F.; Zhang, Z.; Tan, C.; Chang, B.; Huang, S.; and Huang, F. 2021a.  Raise a child in large language model: Towards effective and generalizable fine-tuning.  _arXiv preprint arXiv:2109.05687_.
  * Xu et al. (2020) Xu, Y.; Li, M.; Cui, L.; Huang, S.; Wei, F.; and Zhou, M. 2020.  Layoutlm: Pre-training of text and layout for document image understanding.  In _KDD_.
  * Xu et al. (2021b) Xu, Y.; Xu, Y.; Lv, T.; Cui, L.; Wei, F.; Wang, G.; Lu, Y.; Florencio, D.; Zhang, C.; Che, W.; Zhang, M.; and Zhou, L. 2021b.  LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding.  In _ACL_.
  * Xue et al. (2021) Xue, H.; Huang, Y.; Liu, B.; Peng, H.; Fu, J.; Li, H.; and Luo, J. 2021.  Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training.  In _NeurIPS_.
  * Yang et al. (2022) Yang, J.; Duan, J.; Tran, S.; Xu, Y.; Chanda, S.; Chen, L.; Zeng, B.; Chilimbi, T.; and Huang, J. 2022.  Vision-Language Pre-Training with Triple Contrastive Learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 15671–15680.

##  Appendix A Supplementary Material

Due to the limitation of writing space, we provide more analysis of our
proposed AETNet in this supplementary material. More specifically, we include
the following three folds:

  * •

Detailed implementation of the proposed AETNet method on various downstream
tasks.

  * •

Additional intrinsic analysis of the impact of the number of patches after the
average pooling and the impact of number of layers of the proposed alignment-
ware image and text encoders.

  * •

More visualization of cases to demonstate the strong alignment ability of our
proposed AETNet.

### Implementation Details

This section describes the detailed implementation of the AETNet method on
various downstream tasks.

For all downstream tasks, we run experiments using $2$ NVIDIA Tesla 3090 GPUs.
For initialization of model parameters of AETNetbase and AETNetlarge, we use
DeiT (Touvron et al. 2021) to initialize the alignment-aware image encoder,
RoBERTa (Liu et al. 2019) to initialize the alignment-aware text encoder, and
LayoutLMv3 (Huang et al. 2022) to initialize the multimodal fusion encoder.

For the base model, we set the batch size and epochs for alignment-enriched
tuning to be $4$ and $30$ respectively for all downstream tasks. We set the
number of layers for the alignment-aware image encoder, the number of layers
for the alignment-aware text encoder, and hidden size to be 12, 12, and 768,
respectively. We follow LayoutLMv3base to set number of layers for the
multimodal fusion encoder.

For the large model, we set the batch size and epochs for alignment-enriched
tuning to be $2$ and $60$ respectively for all downstream tasks. We set the
number of layers for the alignment-aware image encoder, the number of layers
for the alignment-aware text encoder, and hidden size to be 6, 6, and 1024,
respectively. We follow LayoutLMv3large to set number of layers for the
multimodal fusion encoder.

#### Implementation Details on FUNSD

We evaluate the proposed AETNet method on the FUNSD dataset for the form
understanding task. We follow the official split of the FUNSD dataset (Jaume,
Ekenel, and Thiran 2019), which consists of 199 documents (149 documents for
training and 50 documents for testing) and 9,707 semantic entities. We adopt
the AdamW optimizer. The initial learning rate is 3e-2 and weight decay here
is 0.02. We use the learning rate scheduler, i.e., CosineLRScheduler, with the
minimum learning rate of 2e-2. Warmup learning rate and epochs are 2e-5 and
20, respectively.

#### Implementation Details on CORD

For the receipt understanding task, we evaluate on the CORD dataset (Park et
al. 2019), including 1,000 receipts and 30 semantic labels defined under 4
categories, where 800 samples are used for training, 100 for validation, and
100 for testing. We use AdamW with an initial learning rate of 5e-2,
CosineLRScheduler with the minimum learning rate of 3e-2, and weight decay of
1. The warmup learning rate we use is 2e-5, and warmup epochs are 20. We tune
models for 40 epochs.

#### Implementation Details on DocVQA

For the document visual question answering task, we conduct experiments on the
DocVQA dataset (Mathew, Karatzas, and Jawahar 2021). It consists of
10,194/1,286/1,287 images with 39,463/5,349/5,188 questions for
training/validation/test, respectively. A data example here includes a
document image, a question, and an answer. We use open-source OCR
Tesseract222https://github.com/tesseract-ocr/tesseract) to obtain the bounding
boxes for each document image. For efficiency, we employ fp16 rather than
fp32. We train the model with the training batch size of 8, epochs of 10,
learning rate of 3e-5, and same learning scheduler as our experiments on FUNSD
with the minimum learning rate of 2e-5. We follow LayoutLMv3 For the rest of
hyper-parameters.

#### Implementation Details on RVL-CDIP-1

For the document image classification task, we evaluate all experiments on a
small-scale dataset termed RVL-CDIP-1, which is a sampled subset of the RVL-
CDIP dataset (Harley, Ufkes, and Derpanis 2015). We sample examples from the
original datasetset according to the distribution of 16 categories. In the
end, RVL-CDIP-1 contains 8000/1000/1000 data samples for the
training/validation/test. We train the model with the training batch size of
4, epochs of 80, learning rate of 3e-2, and same learning scheduler as our
experiments on FUNSD with the minimum learning rate of 2e-2.

Figure 6:  Performance (F1) comparison w.r.t. different kernel sizes and
stride values for the average pooling over patch embeddings.

### Additional Intrinsic Analysis

#### Impact of the Number of Patches

In our all experiments, each document image is first split into 196 patches
with a size of $14\times 14$. Following TCL (Yang et al. 2022), we employ the
global average pooling over the last layer of image and text patch embeddings
for ruling out the probability that small document image patches may not
contain enough text information. Figure 6 shows that AETNet benefits the most
from $3\times 3$ averaged patches with kernel size of 5 and stride value of 4,
which indicates that there exist small document image patches which may not
include text information or only include trivial and useless text information.

#### Impact of Number of Layers of the Alignment-Aware Image Encoder

Figure 7:  Performance (F1) comparison w.r.t. different numbers of layers of
the alignment-ware image encoder (ViTbase) of AETNetbase with the fixed
12-layer text encoder (RoBERTabase) on the FUNSD dataset.

This subsection studies the impact of number of layers of the alignment-aware
image encoder (ViT). First, we fix the number of layers of the alignment-aware
text encoder (RoBEERTa) to be 12. We then change the number of layers of ViT
from 0 to 12 with the interval of 2.

Figure 7 presents the results of variants of the number of layers of the ViT
on the FUNSD dataset. We have the following observations. To our surprise,
AETNet with the 0-layer ViT encoder, i.e., only using patch-level embedding
layers, can achieve better performance than the one with 2-layer, 4-layer, or
8-layer ViT encoders. Although it can outperform LayoutLMv3 and run
efficiently, it is still worse than the one with the 12-layer ViT encoder. The
possible reason is that the model with more layers has a stronger ability to
model local information, which is essential for the proposed alignment loss.
Thus, we choose the 12-layer ViT encoder as the alignment-aware image encoder.

#### Impact of Number of Layers of the Alignment-Aware Text Encoder

Figure 8: Performance (F1) comparison w.r.t. different numbers of layers of
the alignment-ware text encoder (RoBERTabase) of AETNetbase with the fixed
12-layer image encoder (ViTbase) on the FUNSD dataset.

In this subsection, we study the impact of number of layers of the alignment-
aware text encoder (RoBERTa). First, we fix the number of layers of the
alignment-aware image encoder (ViT) to be 12. We then vary the number of
layers of RoBERTa from 0 to 12 with the interval of 2.

Figure 8 shows the results of variants of the number of layers of RoBERTa on
the FUNSD dataset. We observe that AETNet with the 0-layer RoBERTa encoder,
i.e., only using patch-level text embedding layers, can achieve better
performance than the one with 2-layer or 4-layer RoBERTa encoders. Beyond, it
can outperform LayoutLMv3. Same as the observation of results of varying the
number of ViT layers with fixed text encoder, it is still worse than the one
with the 12-layer RoBERTa encoder, which achieves the best performance among
all variants. We assume that the model with more text layers has a stronger
ability to model local information hidden in the text, which is also essential
for the proposed alignment loss. Thus, we choose the 12-layer RoBERTa encoder
as the alignment-aware text encoder.

### Additional Examples of Alignment-Enriched Cases

In Figure 9 and Figure 10, we show more visualization of cases on FUNSD and
CORD datasets to demonstrate the strong alignment ability of our proposed
AETNet to perform the document and receipt understanding.

Figure 9: Visualization of case $3\sim 6$ on the FUNSD dataset. Figure 10:
Visualization of case $7\sim 9$ on the CORD dataset.