# ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for
Document Information Extraction

Jiabang He1, Lei Wang2, Yi Hu1, Ning Liu3, Hui Liu4, Xing Xu1, Heng Tao Shen1
1 Center for Future Media & School of Computer Science and Engineering,
University of Electronic Science and Technology of China
2 Singapore Management University , 3 Beijing Forestry University
4 Beijing Rongda Technology Co., Ltd.
Corresponding author<EMAIL_ADDRESS>author.
<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs), such as GPT-3 and ChatGPT, have demonstrated
remarkable results in various natural language processing (NLP) tasks with in-
context learning, which involves inference based on a few demonstration
examples. Despite their successes in NLP tasks, no investigation has been
conducted to assess the ability of LLMs to perform document information
extraction (DIE) using in-context learning. Applying LLMs to DIE poses two
challenges: the modality and task gap. To this end, we propose a simple but
effective in-context learning framework called ICL-D3IE, which enables LLMs to
perform DIE with different types of demonstration examples. Specifically, we
extract the most difficult and distinct segments from hard training documents
as hard demonstrations for benefiting all test instances. We design
demonstrations describing relationships that enable LLMs to understand
positional relationships. We introduce formatting demonstrations for easy
answer extraction. Additionally, the framework improves diverse demonstrations
by updating them iteratively. Our experiments on three widely used benchmark
datasets demonstrate that the ICL-D3IE framework enables Davinci-003/ChatGPT
to achieve superior performance when compared to previous pre-trained methods
fine-tuned with full training in both the in-distribution (ID) setting and in
the out-of-distribution (OOD) setting. Code is available at
https://github.com/MAEHCM/ICL-D3IE.

##  1 Introduction

Figure 1: Two approaches for solving the DIE task: (a) previous pre-trained
document understanding models [15, 42] fine-tuned with full training examples,
and (b) in-context learning over LLMs with a few examples.

The task of visually rich document understanding (VRDU), which involves
extracting information from VRDs [2, 19], requires models that can handle
various types of documents, such as voice, receipts, forms, emails, and
advertisements, and various types of information, including rich visuals,
large amounts of text, and complex document layouts [28, 18, 26]. Recently,
fine-tuning based on pre-trained visual document understanding models has
yielded impressive results in extracting information from VRDs [41, 13, 22,
23, 15, 21], suggesting that the use of large-scale, unlabeled training
documents in pre-training document understanding models can benefit
information extraction from VRDs. As shown in Figure 13 (a), a pre-trained
model such as LayoutLMv3 [15] can predict labels for entities in a test VRD.

Large language models (LLMs), such as GPT-3 [1], OPT [45], and PaLM [5],
develop quickly and have shown remarkable results in various natural language
processing (NLP) tasks. As LLMs grow in model parameters and training corpus
size, they reveal emergent abilities that allow them to learn to reason from
just a few demonstration examples within a given context [38]. This paradigm
of learning is referred to as in-context learning (ICL) [8]. Recently,
approaches [43, 14] have been proposed to explore how to use LLMs to solve
vision-language (VL) tasks. However, to date, there has been no investigation
into the ability of LLMs to handle VRD understanding tasks, such as document
information extraction (DIE). Similar to VQA [12], Two main challenges arise
when applying LLMs to DIE: the modality gap and the task gap, as LLMs cannot
directly process images and may lack training on layout information in VRDs.

To address these challenges, one popular strategy in using LLMs for the VQA
task is to use demonstration QA pairs and convert their corresponding images
into image descriptions through image caption models [43, 14]. Subsequently,
the demonstration QA pairs and image descriptions are combined as a prompt for
the LLM to answer a test question. Figure 13 (b) shows this straightforward
strategy to apply LLMs to the DIE task. It first utilizes Optical Character
Recognition (OCR) tools to convert images of demonstration documents from the
training data into textual contents and corresponding entity bounding boxes.
The converted demonstrations with entity labels are then combined as a prompt
for LLMs to predict labels for entities in a test document. However, this
strategy may perform poorly, as it ignores positional relationships among
textual contents and is sensitive to examples selected for demonstrations.

In this paper, we propose ICL-D3IE, a simple and effective in-context learning
framework for LLMs to perform the DIE task with various types of demonstration
examples within a given context. Our method constructs different types of
demonstrations based on three criteria: (1) the demonstrations should benefit
all test documents rather than just a subset of them, (2) layout information
must be included, and (3) the demonstrations should predict labels in an
easily extractable format. To construct hard demonstrations for the first
criterion, we select challenging segments from the training documents that are
difficult for LLMs to accurately predict entities. To construct layout-aware
demonstrations for the second criterion, we use a prompt question to direct
LLMs to describe positional relationships between textual content boxes in
selected regions. To create formatting demonstrations for the third criterion,
we randomly choose training segments to guide LLMs to predict labels in a
desired format for easy extraction. Furthermore, the framework iteratively
enhances diverse demonstrations by updating hard demonstrations through in-
context learning with previous diverse demonstrations.

Experiments conducted on three widely used benchmark datasets (FUNSD [18],
CORD [28], and SROIE [16]), demonstrate that ICL-D3IE allows LLMs to achieve
DIE performance that is superior or comparable to previous pre-trained methods
fine-tuned with full training samples when tested in the in-distribution (ID)
setting. For example, ICL-D3IE with GPT-3 (97.88%) outperforms LayoutLMv3base
(96.89%) on SROIE. Moreover, in the out-of-distribution (OOD) setting,
ICL-D3IE performs much better than previous pre-trained methods on all
datasets, achieving superior performance. Together, these remarkable results
encourage new ways to leverage LLMs for solving VRD-related tasks.

Figure 2:  A detailed illustration of ICL-D3IE framework, including obtaining
nearest neighbor documents for test samples from the training dataset,
constructing iteratively updated diverse demonstrations, and performing
inference.

##  2 Related Work

Visually Rich Document Understanding (VRDU). The research topic of VRDU has
been a challenging area of research for many years, with numerous named entity
recognition (NER) methods proposed based on neural networks, such as recurrent
neural networks [20]. However, most of these methods only identify key
information in plain text, neglecting the visual and layout information
present in the document. To address this issue, convolutional and graph neural
networks have been introduced to model layout and semantic information [46,
24]. Recently, multimodal self-supervised pre-training and fine-tuning have
proven effective in visually rich documents by modeling visual, layout, and
textual information [40, 35, 11, 36, 15]. Huang et al. [15] were inspired by
the Vision Transformer (ViT) [10] to use patch-level embeddings to learn
visual features in LayoutLMv3. DIE involves automatically extracting
information from VRDs. The objective is to identify valuable information in
these complex documents and organize it in a format that can be easily
analyzed and used. The process of extracting information from VRDs requires
two essential steps: (1) text detection and recognition in document images,
and (2) entity labeling of the recognized text. The first step falls under the
area of research known as optical character recognition. This study focuses on
the second step and mainly discusses how to leverage GPT-3 to accurately label
entities in recognized text.

In-Context Learning. LLMs like GPT-3 [1], OPT [45], and PaLM [5] demonstrate
emergent abilities as model and corpus sizes increase [38]. These abilities
are learned from demonstrations containing a few examples in the context,
which is known as in-context learning [8]. To enable reasoning in LLMs, [39]
propose Chain-of-Thought (CoT) prompting, which adds multiple reasoning steps
to the input question. CoT prompting is a simple and effective few-shot
prompting strategy that improves LLMs’ performance on complex reasoning tasks.
Several works [34, 32, 31] have since aimed to improve CoT prompting in
different aspects, such as prompt format [4], prompt selection [25], prompt
ensemble [37], and problem decomposition [47]. While LLMs were originally
developed for NLP tasks, recent studies[43, 3, 44] have shown that LLMs with
in-context learning have few-shot or zero-shot abilities for multimodal
problems, including visual question answering tasks. Furthermore, Frozen [33]
demonstrates promising few-shot performance using pre-trained models for
vision-and-language tasks. However, to our knowledge, our work is the first to
explore the use of LLMs with in-context learning for information extraction
from VRDs. You can refer to [9] for more related works on in-context learning

Figure 3: Example of the input and output of in-context learning with diverse
demonstrations. The text highlighted in blue is not processed by LLMs, while
the text highlighted in red is fed into LLMs. The green-highlighted text
represents the output of LLMs. The text in red represents the prediction made
by the LLM. The final prompt comprises label mapping, hard demonstrations,
layout-aware demonstrations, formatting demonstrations, and a question prompt
of “What are the labels for these texts?”.

##  3 Our ICL-D3IE Method

###  3.1 Preliminary of In-Context Learning

In-context learning enables LLMs to quickly adapt to solve downstream tasks
using just a few examples during inference [1], requiring no training. In
contrast, fine-tuning LLMs necessitates training on as many samples as
feasible, resulting in redundant computation and time expenses. This section
describes how to formulate in-context learning for solving the DIE task.

A data sample consists of a document image $I$ and its corresponding entity
labels $Y=\\{y_{1},y_{2},...,y_{L}\\}$, where $L$ is the number of entities in
the document. To obtain textual contents and their corresponding boxes, we
process the document image $I$ using an OCR tool. We denote the set of textual
contents as $T=\\{t_{1},t_{2},...,t_{L}\\}$, where $t_{l}$ is a segment of
words, and denote the set of their corresponding boxes as
$B=\\{b_{1},b_{2},...,b_{L}\\}$, where $b_{l}$ is the coordinates
$\left[p_{1}^{l},p_{2}^{l},p_{3}^{l},p_{4}^{l}\right]\in\mathbb{Z}^{4}$ of the
box $b_{l}$. Note that the ordering of $T$ is crucial because GPT-3 is
sensitive to the permutation of words. We follow the approach of XYLayoutLM
[11] and use the XYCut algorithm to determine the ordering of textual regions.
The DIE task (This paper considers the task of entity labeling in VRDs)
involves generating labels $Y$ for the given entities $T$ in the document
image $I$ by maximizing the conditional probability as follows: $p(Y\mid
T)=\frac{1}{L}\sum_{l}^{L}p\left(y_{l}\mid t_{l}\right)$.

While previous state-of-the-art studies [40, 11] typically fine-tune pre-
trained models to downstream tasks, this paper proposes using LLMs with in-
context learning to solve the DIE task. Specifically, we define the
probability of generating the target entity labels $Y$ for a given document
image $\mathbf{I}$ and in-context string $C$ using a LLM $\mathcal{P}_{lm}$ as
follows:

$p(Y|I,C)=\sum_{l=1}^{L}\mathcal{P}_{lm}\left(\mathcal{V}(y_{l})|C,\mathcal{T}(I)\right).$
(1)

Here, $\mathcal{T(\cdot)}$ denotes a set of operations used to convert the
original document image into a text format as GPT-3 desire, $C$ is the in-
context examples obtained by concatenating $k$ input-output demonstration
examples
$\left\\{(\mathcal{T}(I_{1}),Y_{1}),(\mathcal{T}(I_{2}),Y_{2}),\ldots,(\mathcal{T}(I_{k}),Y_{k})\right\\}$,
and $\mathcal{V}$ is an operation for mapping an entity label $y_{l}$ to
natural language words that can be understood by GPT-3.

###  3.2 Overview Framework of ICL-D3IE

We present ICL-D3IE, a novel in-context learning framework for tackling the
DIE task, that enables GPT-3 to predict entity labels in a test document based
on different types of demonstrations. Constructing demonstrations is designed
to satisfy three criteria: (i) the demonstrations should benefit all test
documents, not just a subset, (ii) they should include layout information,
which is essential for solving VRD-related tasks., and (iii) they should
predict entity labels in an easily extracted and evaluated format.

The proposed ICL-D3IE framework involves four key steps as shown in Figure 2.
Firstly, the framework selects $n$ training documents most similar to the $n$
test documents. Secondly, ICL-D3IE constructs diverse demonstrations based on
the selected similar training documents. These demonstrations include initial
hard demonstrations for criterion (i), layout-aware demonstrations for
criterion (ii), and formatting demonstrations for criterion (iii). Thirdly,
the framework iteratively updates the diverse demonstrations by improving the
hard demonstrations through in-context learning with previous diverse
demonstrations. Lastly, ICL-D3IE performs inference using in-context learning
with the updated diverse demonstrations.

###  3.3 Nearest Neighbor Document Selection

To facilitate effective in-context learning, the proposed ICL-D3IE selects $n$
training documents that are most similar to the $n$ test documents. This
process involves several steps. Firstly, we leverage OCR tools to convert $m$
training and $n$ test document images into plain text with corresponding box
information. Subsequently, the plain text is fed into Sentence-BERT [30] to
obtain document representations, and cosine similarity scores are calculated
to identify the most similar training document for each test document.
Finally, we can identify $n$ training documents that are the closest match to
the $n$ test documents, which we refer to as nearest neighbor documents
$I^{\mathrm{nnd}}_{1},I^{\mathrm{nnd}}_{2},\dots,I^{\mathrm{nnd}}_{n}$.

###  3.4 Diverse Demonstrations Construction

Once we have obtained $n$ nearest neighbor documents from the training
dataset, we construct diverse demonstrations for effective in-context
learning. The standard approach to constructing in-context demonstrations
involves designing a template for the target task to convert data examples
into texts that LLMs can process. Unlike standard in-context learning that
relies solely on task-specific demonstrations, ICL-D3IE constructs diverse
demonstrations for each test instance: hard demonstrations that highlight
challenging aspects of a task, layout-aware demonstrations that describe the
positional relationship between textual contents, and formatting
demonstrations that provide output formatting examples.

Initial Hard Demonstrations. The first criterion for selecting hard
demonstrations is that they should highlight the most challenging aspects of
the DIE task to benefit all test documents. The process of obtaining initial
hard demonstrations involves several steps. First, we use a zero-shot
prompting technique, which involves using a prompt such as “What are the
labels for these texts?” $pt_{0}$ to ask GPT-3 to predict labels for entities
in $I^{\mathrm{nnd}}_{i}$. Next, we calculate entity-level F1 scores based on
the predicted labels and the corresponding ground truth labels. We then
identify the text segment $t_{\mathrm{hard}}$ with the lowest F1 scores from
the nearest neighbor documents. An initial hard demonstration can be
formulated as:

$C_{\mathrm{hard,0}}=\mathrm{CONCAT}(t_{\mathrm{hard}},b_{\mathrm{hard}},pt_{0},y_{\mathrm{hard}}),$
(2)

where $b_{\mathrm{hard}}$ and $y_{\mathrm{hard}}$ are the box coordinate and
answer of the text segment $t_{\mathrm{hard}}$, respectively.

Layout-Aware Demonstrations. The second criterion necessitates the inclusion
of layout information in the in-context demonstrations, which is crucial for
completing the DIE task. To acquire demonstrations mindful of layout, We
randomly select adjacent hard segments obtained in the construction of
$C_{\mathrm{hard,0}}$ to create a region $R_{l}$ for positional relationship
description. We use a prompt “Please describe the positional relationship of
these texts” $pt_{l}$ to guide GPT-3 to generate a description $\tilde{y}_{l}$
of the positional relationship between text segments in $R_{l}$. A layout-
aware demonstration can be formulated as:

$C_{l}=\mathrm{CONCAT}(R_{l},B_{l},pt_{l},\tilde{y}_{l}),$ (3)

where $B_{l}$ are the box coordinates for text segments of the selected region
$R_{l}$.

Formatting Demonstrations. The third criterion expects to provide examples to
guide GPT-3 to format the output for the DIE task. To achieve this, we first
randomly select a text segment $t_{\mathrm{f}}$ from the nearest neighbor
documents. Then, a formatting demonstration $C_{\mathrm{f}}$ consist of a text
segment $t_{\mathrm{f}}$, its corresponding box coordinate $b_{\mathrm{f}}$,
the formatting prompt $pt_{0}$, and the ground truth answer $y_{\mathrm{f}}$,
denoted as $C_{\mathrm{f}}$:

$C_{\mathrm{f}}=\mathrm{CONCAT}(t_{\mathrm{f}},b_{\mathrm{f}},pt_{0},y_{\mathrm{f}}).$
(4)

Label Mapping. The objective of label mapping is to translate unnatural word
labels to an answer space where GPT-3 can effectively function as a predictive
model. To achieve this, we gather text descriptions of the original labels
from the provided datasets, such as “total. cash price” representing “the
amount paid in cash.” Then, we include the original labels ($Y^{\prime}$) and
their corresponding descriptions ($Y$) in the context before various
demonstrations to prompt GPT-3 to solve the test sample. Label Mapping for
prompting can be formulated as:

$C_{\mathrm{m}}=\mathrm{CONCAT}(Y^{\prime},Y).$ (5)

###  3.5 Diverse Demonstrations Updating

To further highlight the most challenging aspects of the DIE task, ICL-D3IE
iteratively updates its diverse demonstrations by improving hard
demonstrations through in-context learning with previous diverse
demonstrations. Initial diverse demonstrations with initial hard
demonstrations $C_{\mathrm{hard},0}$ are used to perform inference for all
nearest neighbor documents
$I^{\mathrm{nnd}}_{1},I^{\mathrm{nnd}}_{2},\dots,I^{\mathrm{nnd}}_{n}$.
Entity-level F1 scores are computed for all entities, and the text segment
with the lowest F1 score is appended to the initial hard demonstrations to
obtain new hard demonstrations $C_{\mathrm{hard},1}$. This process is iterated
$k$ times to obtain final updated hard demonstrations $C_{\mathrm{hard,k}}$,
which are used to construct the final diverse demonstrations.

###  3.6 Inference

After diverse demonstrations updating, the obtained diverse and comprehensive
demonstrations can be used to direct GPT-3 to perform the test, which is
formulated as follows:

$p(Y|\mathbf{I},C)=\frac{1}{L}\sum_{l=1}^{L}\mathcal{P}_{lm}\left(\mathcal{V}(y_{l})|C_{\mathrm{m}},C_{\mathrm{hard,k}},C_{l},C_{\mathrm{f}},\mathcal{T}(\mathbf{I})\right).$
(6)

Finally, ICL-D3IE extracts the corresponding answers from the generated
predictions and then converts them into a suitable format for evaluation.

##  4 Experiment

###  4.1 Experimental Setup

Datasets. We experiment on three widely used DIE datasets. Here is a brief
introduction to these datasets: The FUNSD dataset [17] is a noisy scanned form
understanding dataset. It comprises 199 documents with varying layouts and
9,707 semantic entity annotations in total. In our study, we focus on the
semantic entity labeling task, which involves assigning labels such as
“question,” “answer,” “header,” or “other” to each semantic entity. The
training set comprises 149 samples, and the test set comprises 50 samples. The
CORD dataset [29] is a consolidated receipt understanding dataset that
includes 800 receipts for training, 100 receipts for validation, and 100
receipts for testing. The labels in this dataset have a hierarchy, comprising
30 semantic labels under four categories. However, the labels are more complex
than those in the FUNSD dataset and require label mapping. The SROIE dataset
[16] is another receipt understanding dataset, comprising 973 receipts
categorized into four classes. The dataset includes 626 training images and
347 test images. The labels in this dataset are “company,” “date,” “address,”
and “total.”

| Dataset | FUNSD | CORD | SROIE
---|---|---|---|---
Setting | Model | ID | OOD | Average | ID | OOD | Average | ID | OOD | Average
F1$\uparrow$ | F1$\uparrow$ | F1$\uparrow$ | F1$\uparrow$ | F1$\uparrow$ | F1$\uparrow$ | F1$\uparrow$ | F1$\uparrow$ | F1$\uparrow$
Full-Training |  $\textrm{BERT}_{\rm BASE}$ [7] | 60.26 | 51.02 | 55.64 | 89.68 | 55.68 | 72.68 | 90.99 | 72.36 | 81.68
$\textrm{LiLT}_{\rm BASE}$ [35] | 88.41 | 64.29 | 76.35 | 96.07 | 73.32 | 84.70 | 94.68 | 74.29 | 84.49
$\textrm{BROS}_{\rm BASE}$ [13] | 83.05 | 68.72 | 75.89 | 95.73 | 71.24 | 83.49 | 95.48 | 75.51 | 85.50
$\textrm{XYLayoutLM}_{\rm BASE}$ [11] | 83.35 | 61.24 | 72.30 | 94.45 | 69.12 | 81.79 | 95.74 | 75.91 | 85.83
$\textrm{LayoutLM}_{\rm BASE}$ [41] | 79.27 | 54.38 | 66.83 | 91.06 | 70.13 | 80.60 | 94.38 | 76.24 | 85.31
$\textrm{LayoutLMv2}_{\rm BASE}$ [42] | 82.76 | 59.66 | 71.21 | 94.95 | 76.39 | 85.67 | 96.25 | 78.57 | 87.41
$\textrm{LayoutLMv3}_{\rm BASE}$ [15] | 90.29 | 73.24 | 81.77 | 96.56 | 75.23 | 85.90 | 96.89 | 78.34 | 87.62
Few-Shot |  $\textrm{BERT}_{\rm BASE}$ [7] | 38.76 | 19.68 | 29.22 | 38.88 | 15.31 | 27.10 | 38.76 | 20.56 | 59.32
$\textrm{LiLT}_{\rm BASE}$ [35] | 54.88 | 25.32 | 40.10 | 69.12 | 29.94 | 49.53 | 84.03 | 61.25 | 72.64
$\textrm{BROS}_{\rm BASE}$ [13] | 59.46 | 27.49 | 43.48 | 72.78 | 36.34 | 54.56 | 76.78 | 57.28 | 67.03
$\textrm{XYLayoutLM}_{\rm BASE}$ [11] | 65.44 | 30.56 | 48.00 | 69.16 | 32.19 | 50.68 | 75.66 | 56.23 | 65.95
$\textrm{LayoutLM}_{\rm BASE}$ [41] | 32.49 | 17.66 | 25.08 | 40.19 | 23.62 | 31.91 | 76.79 | 55.44 | 66.12
$\textrm{LayoutLMv2}_{\rm BASE}$ [42] | 71.42 | 49.12 | 60.27 | 65.71 | 29.43 | 47.57 | 81.81 | 59.56 | 70.69
$\textrm{LayoutLMv3}_{\rm BASE}$ [15] | 70.67 | 48.33 | 59.50 | 70.13 | 32.88 | 51.51 | 79.13 | 56.08 | 67.61
Standard ICL (ChatGPT) | 72.76 | 69.32 | 71.04 | 68.34 | 65.68 | 67.01 | 82.11 | 80.31 | 81.21
Standard ICL (Davinci-003) | 71.52 | 67.31 | 69.42 | 67.96 | 64.28 | 66.12 | 79.34 | 76.12 | 77.73
| ICL-D3IE (ChatGPT) | 83.66 | 79.05 | 81.36 | 87.13 | 70.69 | 78.91 | 92.63 | 86.31 | 89.47
| ICL-D3IE (Davinci-003) | 90.32 | 88.71 | 89.52 | 94.12 | 91.23 | 92.68 | 97.88 | 93.76 | 95.82

Table 1:  Results of comparing ICL-D3IE with Standard ICL and existing pre-
trained VDU models fine-tuned with full training samples and a few samples on
three benchmark datasets in ID and OOD settings.

Baselines. We compare ICL-D3IE with three types of baselines. The first type
includes strong pre-trained models fine-tuned with full training samples,
while the second type includes those fine-tuned with only a few samples. The
third type includes standard in-context learning, where one of its
demonstrations includes one document’s textual contents, the corresponding box
coordinates, the prompt question $pt_{0}$, and the corresponding ground truth
answers.

For the text modality-based pre-trained baseline, we compare our method to
BERT [6]. For the text and layout modalities based on pre-trained baselines,
we employ LiLT [35] and BROS [13]. LiLT uses a language-independent layout
transformer that decouples text and layout modalities. BROS is a pre-trained
key information extraction model that encodes relative layout information.
Furthermore, we also consider pre-trained baselines that utilize text, layout,
and image modalities, including LayoutLM [40], XYLayoutLM [11], LayoutLMv2
[42], and LayoutLMv3 [15]. LayoutLM uses two objectives to learn language
representation during pre-training and incorporates image information during
the fine-tuning phase. XYLayoutLM employs a preprocessing algorithm called
Augmented XY Cut to generate proper reading orders. LayoutLMv2 uses CNN to
encode document images and utilizes image information during the pre-training
stage. Lastly, LayoutLMv3 can model patch-level document information.

Implementation Details. In our experiments, we use the public GPT-3 text-
davinci-003 (175B) and ChatGPT gpt-3.5-turbo with the
API111https://platform.openai.com/docs/models/gpt-3-5 as the backbone language
models due to their popularity and accessibility. To ensure consistent output,
we set the temperature parameter to 0. For evaluation, we employ the same
metrics as in LayoutLMv3 and reported entity-level F1 for all methods. For our
ICL-D3IE method, we use 4 hard demonstrations, 4 layout-ware demonstrations,
and 4 formatting demonstrations. For the fine-tuning-based baselines, we adopt
the hyper-parameters reported in their original papers. Note that our
demonstrations may be segments, and we use document examples that include
segments used in our method to fine-tune few-shot baseline models to ensure a
fair comparison. To demonstrate the generalization ability of in-context
learning over LLMs, we generate out-of-distribution (OOD) test data for three
benchmark datasets using TextAttack [27]. The original test data for these
datasets are referred to as in-distribution (ID) test data. Specifically, we
replace original words with words that are nearly identical in appearance yet
different in meaning and delete certain characters in words, such as “name”
$\longrightarrow$ “nme,” to generate OOD samples.

###  4.2 Main Results

Table 1 presents the performance comparison of ICL-D3IE with existing full-
training and few-shot baseline methods on both in-domain (ID) and out-of-
domain (OOD) settings. We can first observe that on the ID setting, ICL-D3IE
(Davinci-003) achieves a new state-of-the-art on FUNSD and SROIE datasets with
only a few data examples and without any training. It achieves 90.32% on FUNSD
and 97.88% on SROIE, beating all other VDU achieving SOTA. On the SROIE
dataset, ICL-D3IE (Davinci-003) reaches within 3% of the state-of-the-art
performance, which is comparable to pre-trained VDU models that are fine-tuned
with full training samples. On the other hand, ICL-D3IE has large performance
gains for DIE in the few-shot setting. For instance, in CORD, average
performance more than doubled for the VDU in the few-shot setting. Meanwhile,
compared to other full-training baselines, ICL-D3IE has greater robustness to
OCR errors in document content on the OOD settings, resulting in significantly
better performance.

Moreover, we can see that ICL-D3IE outperforms Standard ICL on three datasets,
with ICL-D3IE (Davinci-003) showing an 18.8 F1 score improvement over Standard
ICL (Davinci-003) on FUNSD. We experiment with GPT-3 (text-davinci-003) and
ChatGPT (gpt-3.5-turbo) to investigate the applicability of ICL-D3IE with
different backbone language models and find that ICL-D3IE substantially
improves the performance of ChatGPT compared with Standard ICL. However,
ChatGPT generation’s flexibility makes answer extraction harder, resulting in
slightly worse performance for ICL-D3IE (ChatGPT) compared to ICL-D3IE
(Davinci-003). These promising results demonstrate the effectiveness of
ICL-D3IE for the DIE task and its versatility across different backbone
language models.

Overall, our ICL-D3IE method shows consistent superiority over other methods
across all datasets and settings except for the ID setting on CORD, suggesting
its potential to effectively solve VRD-related tasks using LLMs. These
remarkable results not only highlight the effectiveness of ICL-D3IE but also
inspire the development of novel methods with LLMs that require less manual
effort.

| FUNSD | CORD | SROIE
---|---|---|---
F1$\uparrow$ | F1$\uparrow$ | F1$\uparrow$
ICL-D3IE | 90.32 | 94.12 | 97.88
w/o HD | 78.20 | 87.13 | 89.13
w/o LD | 87.25 | 84.13 | 96.83
w/o LM | 89.63 | 87.94 | 97.19
w/o FD | 88.73 | 93.07 | 90.58
Table 2:  The effect of different components in ICL-D3IE. HD means Hard
Demonstrations. LD means Layout-Aware Demonstrations. LM means Label Mapping.
FD means Formatting Demonstrations.

###  4.3 Further Analysis

In this section, we conduct a detailed analysis of ICL-D3IE and its
components.

Effect of Different Components in Diverse Demonstrations. ICL-D3IE’s
demonstrations consist of four components: hard demonstrations, layout-aware
demonstrations, formatting demonstrations, and label mapping. In this section,
we evaluate the impact of each component by removing one at a time and
measuring the effect on ICL-D3IE (Davinci-003) performance.

(a)

(b)

Figure 4: Further analysis on (a) the effect of the number of different
demonstrations on CORD and (b) the effect of the number of hard demonstrations
updating.

As shown in Table 2, removing any components drops DIE performance. Removing
hard demonstrations has the most significant impact, indicating the
effectiveness of iteratively updated hard demonstrations in benefiting all
test samples. Removing layout-aware demonstrations leads to a drop of around
10 F1 score on CORD but little on SROIE since CORD labels require more layout
information than SROIE. Removing label mapping results in a significant drop
in CORD due to its unnatural labels. ICL-D3IE’s performance without label
mapping suggests formatting demonstrations contribute to easier and better
answer extraction. Notably, ICL-D3IE (Davinci-003) outperforms Standard ICL
(Davinci-003) (Table 1), even with one component removed. Overall, these
results highlight the effectiveness of each component in ICL-D3IE’s in-context
demonstrations.

Effect of the Number of each Type of Demonstrations. In Table 1, we set the
number of different types of demonstrations in ICL-D3IE to 4. However, varying
the number of each type of demonstration in the in-context diverse
demonstrations may result in varying performance outcomes. To investigate
this, we vary the number of a specific type of demonstration from 0 to 4 while
keeping the number of other types of demonstrations constant at 4.

We present the F1 score of ICL-D3IE (Davinci-003) on CORD in Figure 4(a). We
can observe that the number of demonstrations of each type influences the
performance of ICL-D3IE. Besides, performance improves as the number of any
demonstration increases. Interestingly, we observe significant changes in
performance when varying the number of hard and layout-aware demonstrations,
suggesting that hard demonstrations are beneficial for solving all test
samples and that the DIE task on CORD requires a substantial amount of layout
information to solve.

Effect of the Number of Hard Demonstrations Updating. This study aims to
investigate the impact of the number of Hard Demonstrations Updating on three
different datasets. As highlighted in Figure 4(b), initial hard demonstrations
can help ICL-D3IE work very well, and hard demonstrations after 20 iterations
can achieve better performance. These findings demonstrate that incorporating
feedback from challenging aspects, as identified through predictions on
training data, to the prompt for LLMs is a useful strategy that can benefit
solving all test samples. Additionally, updating Hard Demonstrations through
in-context learning with previous diverse demonstrations can enhance the
performance of ICL-D3IE (Davinci-003).

Effect of Ordering of Diverse Demonstrations. This study investigates the
impact of the different ordering of demonstrations on ICL-DI3E (Davinci-003)
performance. Specifically, we change the ordering of hard and layout-ware
demonstrations and evaluate two different orderings: M-H-L-F (label mapping,
hard demonstrations, layout-aware demonstrations, and formatting
demonstrations) and M-L-H-F (label mapping, layout-aware demonstrations, hard
demonstrations, and formatting demonstrations).

Figure 5(a) presents a comparison of the performance of these two orderings.
In our case, M-H-L-F consistently outperforms M-L-H-F across all three
datasets. It suggests that in-context learning is highly sensitive to the
ordering of demonstrations and that finding the optimal ordering for in-
context learning is critical. Our study highlights the importance of
optimizing the ordering of demonstrations for in-context learning, and this
will be a focus of our future research.

Effect of the Number of Demonstration Examples. To further evaluate the
performance of ICL-D3IE in comparison to pre-trained VRDU models fine-tuned
with a few demonstrations, we varied the number of demonstrations for ICL-D3IE
(Davinci-003), ICL-D3IE (ChatGPT), and LayoutLMv3 from 1 to 12. Figure 5(b)
demonstrates that the performances of all three methods improve as the number
of demonstrations increases on CORD. Notably, ICL-D3IE (Davinci-003) and
ICL-D3IE (ChatGPT) consistently outperform LayoutLMv3 by a large margin across
all numbers of demonstrations. These results suggest that our proposed in-
context diverse demonstrations approach is effective and outperforms pre-
trained VRDU models fine-tuned with a few demonstrations.

(a)

(b)

Figure 5: Further analysis on (a) the performance effect of arranging
demonstrations in a different order and (b) the performance comparison of
increasing the number of demonstrations on ICL-D3IE (Davinci-003/ChatGPT) and
LayoutLMv3 on CORD.

(a)

(b)

Figure 6: Case study on comparison of (a) positional relationship description
and (b) predictions generated by Standard ICL (Davinci-003) and ICL-D3IE
(Davinci-003).

Case Study. Figure 6(a) presents two examples of asking positional
relationship descriptions with Standard ICL (Davinci-003) and ICL-D3IE
(Davinci-003) during the test phase. Our results illustrate that Standard ICL,
without layout-ware demonstrations, cannot accurately describe the positional
relationships between textual contents in a document, while ICL-D3IE can do so
effectively. In Figure 6(b), we observe that Standard ICL predicts the
entities in the blue box as “header,” while ICL-D3IE predicts the entities as
“question.”These findings highlight the importance of applying diverse
demonstrations such as hard and layout-aware demonstrations in DIE tasks.

##  5 Conclusion

In this paper, we proposed ICL-D3IE, an in-context learning framework that
addresses the challenges of applying LLMs to DIE tasks, specifically the
modality and task gap. We extracted challenging segments from hard training
documents to benefit all test instances, designed demonstrations that describe
positional relationships to enable LLMs to understand the layout of documents,
and introduced formatting demonstrations to facilitate easy answer extraction.
The framework also improves diverse demonstrations iteratively and uses label
mapping to convert unnatural words to words that GPT can process. Our
evaluation of three DIE datasets shows that ICL-D3IE consistently outperforms
other methods, except for the ID setting on CORD. These results highlight the
potential of in-context learning frameworks for VRD understanding tasks based
on LLMs, and we hope to inspire future research in this area.

##  6 Acknowledgments

This work was supported in part by National Natural Science Foundation of
China under Grants (No. 62222203 and 61976049).

## References

  * [1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [2] Haoyu Cao, Xin Li, Jiefeng Ma, Deqiang Jiang, Antai Guo, Yiqing Hu, Hao Liu, Yinsong Liu, and Bo Ren.  Query-driven generative network for document information extraction in the wild.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 4261–4271, 2022.
  * [3] Rui CAO, Roy Ka-Wei LEE, Wen-Haw CHONG, and Jing JIANG.  Prompting for multimodal hateful meme classification.  2022\.
  * [4] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen.  Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.  arXiv preprint arXiv:2211.12588, 2022.
  * [5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  ArXiv preprint, abs/2204.02311, 2022.
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In NAACL-HLT, pages 4171–4186, 2019.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL, 2019.
  * [8] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui.  A survey for in-context learning.  arXiv preprint arXiv:2301.00234, 2022.
  * [9] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui.  A survey on in-context learning.  2022\.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [11] Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang.  Xylayoutlm: Towards layout-aware multimodal networks for visually-rich document understanding.  In CVPR, 2022.
  * [12] Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi.  From images to textual prompts: Zero-shot vqa with frozen large language models.  arXiv preprint arXiv:2212.10846, 2022.
  * [13] Teakgyu Hong, DongHyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park.  Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents.  In AAAI, 2022.
  * [14] Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo.  Promptcap: Prompt-guided task-aware image captioning.  arXiv preprint arXiv:2211.09699, 2022.
  * [15] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei.  Layoutlmv3: Pre-training for document ai with unified text and image masking.  arXiv preprint arXiv:2204.08387, 2022.
  * [16] Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar.  ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction.  In ICDAR, pages 1516–1520, 2019.
  * [17] Guillaume Jaume, H. K. Ekenel, and J. Thiran.  FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents.  ICDARW, 2:1–6, 2019.
  * [18] Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran.  Funsd: A dataset for form understanding in noisy scanned documents.  In ICDARW, 2019.
  * [19] Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park.  Ocr-free document understanding transformer.  In European Conference on Computer Vision (ECCV), 2022.
  * [20] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer.  Neural architectures for named entity recognition.  In NAACL HLT, 2016.
  * [21] Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei.  Dit: Self-supervised pre-training for document image transformer.  arXiv preprint arXiv:2203.02378, 2022.
  * [22] Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu.  Selfdoc: Self-supervised document representation learning.  In CVPR, 2021.
  * [23] Yulin Li, Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding.  Structext: Structured text understanding with multi-modal transformers.  In ACM Multimedia, 2021.
  * [24] Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao.  Graph convolution for multimodal information extraction from visually rich documents.  arXiv preprint arXiv:1903.11279, 2019.
  * [25] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan.  Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.  arXiv preprint arXiv:2209.14610, 2022.
  * [26] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.  Docvqa: A dataset for vqa on document images.  In WACV, 2021.
  * [27] John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi.  Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119–126, 2020.
  * [28] Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee.  Cord: A consolidated receipt dataset for post-ocr parsing.  In Document Intelligence Workshop at Neural Information Processing Systems, 2019.
  * [29] Seunghyun Park, S. Shin, Bado Lee, Jihyo Lee, Jaeheung Surh, Minjoon Seo, and Hwal-Suk Lee.  CORD: A Consolidated Receipt Dataset for Post-OCR Parsing.  In Workshop on Document Intelligence at NeurIPS 2019, 2019.
  * [30] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [31] Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang.  On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning.  arXiv preprint arXiv:2212.08061, 2022.
  * [32] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al.  Challenging big-bench tasks and whether chain-of-thought can solve them.  ArXiv preprint, abs/2210.09261, 2022.
  * [33] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill.  Multimodal few-shot learning with frozen language models.  arXiv preprint arXiv:2106.13884, 2021.
  * [34] Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun.  Towards understanding chain-of-thought prompting: An empirical study of what matters.  arXiv preprint arXiv:2212.10001, 2022.
  * [35] Jiapeng Wang, Lianwen Jin, and Kai Ding.  Lilt: A simple yet effective language-independent layout transformer for structured document understanding.  In ACL, 2022.
  * [36] Lei Wang, Jiabang He, Xing Xu, Ning Liu, and Hui Liu.  Alignment-enriched tuning for patch-level pre-trained document image models.  arXiv preprint arXiv:2211.14777, 2022.
  * [37] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  arXiv preprint arXiv:2203.11171, 2022.
  * [38] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.  Emergent abilities of large language models.  CoRR, abs/2206.07682, 2022.
  * [39] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022), 2022.
  * [40] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou.  In LayoutLM: Pre-training of Text and Layout for Document Image Understanding, pages 1192–1200, 2020.
  * [41] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou.  Layoutlm: Pre-training of text and layout for document image understanding.  In KDD, 2020.
  * [42] Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou.  LayoutLMv2: Multi-modal pre-training for visually-rich document understanding.  In ACL, 2021.
  * [43] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang.  An empirical study of gpt-3 for few-shot knowledge-based vqa.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3081–3089, 2022.
  * [44] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  arXiv preprint arXiv:2204.00598, 2022.
  * [45] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  OPT: open pre-trained transformer language models.  CoRR, abs/2205.01068, 2022.
  * [46] Xiaohui Zhao, Zhuo Wu, and Xiaoguang Wang.  CUTIE: Learning to Understand Documents with Convolutional Universal Text Information Extractor.  arXiv preprint arXiv:1903.12363, 2019.
  * [47] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi.  Least-to-most prompting enables complex reasoning in large language models.  arXiv preprint arXiv:2205.10625, 2022.

##  Appendix A Additional Analysis of ICL-D3IE

Due to the limitations of space, we provide more experimental analysis of our
proposed ICL-D3IE in this supplement material. Specifically, the main content
includes:

  * •

More experimental results and discussion of ICL-D3IE performance on two
benchmark datasets.

  * •

More visual examples obtained by our ICL-D3IE are provided for easy reference.

For more implementation details, please kindly refer to our code at
https://github.com/MAEHCM/ICL-D3IE.

###  A.1 Effect of the Number of Different Demonstrations

We present the F1 score of ICL-D3IE (GPT-3 [1]) on FUNSD [17] in Figure 7(a)
and on SROIE [16] in Figure 7(b), respectively. We observed that the
performance of ICL-D3IE is influenced by the number of demonstrations of each
type. Moreover, the performance of ICL-D3IE is significantly improved as the
number of any demonstration increases. Notably, we observe that changes in the
number of hard demonstrations (HD) and layout-aware demonstrations (LD) on
FUNSD result in significant changes in performance. Conversely, on the SROIE
dataset, changes in the number of hard demonstrations and format
demonstrations (FD) have a greater impact on performance. It is worth
menioning that the DIE task on FUNSD requires a substantial amount of layout
and difficult text information to solve, whereas SROIE requires a more
favorable extraction format for optimal performance.

###  A.2 Effect of Increasing the Number of Demonstrations

We conduct experiments where we varied the number of demonstrations for
ICL-D3IE (GPT-3), ICL-D3IE (ChatGPT), and LayoutLMv3 from 1 to 12. The results
of our experiments are shown in Figure 8, which clearly demonstrates that the
performances of all three methods improve as the number of demonstrations
increases on FUNSD and SROIE. ICL-D3IE (GPT-3) and ICL-D3IE (ChatGPT)
consistently outperform LayoutLMv3 by a significant margin across all numbers
of demonstrations on two datasets. These findings suggest that our proposed
in-context diverse demonstrations approach is highly effective and outperforms
pre-trained VRDU models that have been fine-tuned with only a few
demonstrations.

(a)

(b)

Figure 7: The effect of the number of different demonstrations on (a) FUNSD
and on (b) SROIE.

(a)

(b)

Figure 8: The performance comparison of increasing the number of
demonstrations on (a) FUNSD and on (b) SROIE. Figure 9: Visualization of two
cases on FUNSD, which are predicted by ICL-D3IE with text-davinci-003 (90.32
F1). Blue: Header-label, Green: Answer, Red: Question, Purple: Other. The
entities contained within the red box are predicted inaccurately. Figure 10:
Visualization of two cases on FUNSD, which are predicted by ICL-D3IE with
text-davinci-003 (90.32 F1). Blue: Header-label, Green: Answer, Red: Question,
Purple: Other. Figure 11: Visualization of three cases on CORD, which are
predicted by GPT-3 (Column (a)), ChatGPT (Column (b)), and Ground Truth(Column
(c)). ICL-D3IE with text-davinci-003 (94.12 F1) versus ICL-D3IE with ChatGPT
(87.13 F1) on CORD test-set.  Table 3: Examples of prediction by GPT-3 and
ChatGPT on CORD. GPT-3:
---
{text:"Vanilla SLC",Box:[282 443 394
463],entity:MENU.SUB_ETC}{text:"1",Box:[261 387 275
401],entity:MENU.CNT}{text:"FL-Xmas 30 Off",Box:[284 384 434
408],entity:MENU.NM}{text:"68,180",Box:[520 390 587
407],entity:MENU.PRICE}{text:"1",Box:[261 405 275
421],entity:MENU.SUB_CNT}{text:"PAKET SLICES",Box:[310 405 460
425],entity:MENU.SUB_NM}{text:"0",Box:[574 410 587
425],entity:MENU.SUB_PRICE}{text:"1",Box:[261 425 275
441],entity:MENU.SUB_CNT}{text:" FL Cake - French",Box:[305 425 488
444],entity:MENU.SUB_NM}{text:"0",Box:[574 430 587
444],entity:MENU.SUB_PRICE}{text:"1",Box:[261 466 273
479],entity:MENU.SUB_CNT}…✗
ChatGPT:
{text:"Vanilla SLC",Box:[282 443 394
463],entity:MENU.SUB_NM}{text:"1",Box:[261 387 275
401],entity:MENU.SUB_CNT}{text:"FLXmas 30 Off",Box:[284 384 434
408],entity:MENU.NM}{text:"68,180",Box:[520 390 587
407],entity:MENU.PRICE}{text:"1",Box:[261 405 275
421],entity:MENU.SUB_CNT}{text:"PAKET SLICES",Box:[310 405 460
425],entity:MENU.SUB_NM}{text:"0",Box:[574 410 587
425],entity:MENU.SUB_PRICE}{text:"1",Box:[261 425 275
441],entity:MENU.SUB_CNT}{text:" FL Cake - French",Box:[305 425 488
444],entity:MENU.SUB_NM}{text:"0",Box:[574 430 587
444],entity:MENU.SUB_PRICE}{text:"1",Box:[261 466 273
479],entity:MENU.SUB_CNT}… ✗
Ground Truth:
{text:"Vanilla SLC",Box:[282 443 394
463],entity:MENU.SUB_NM}{text:"1",Box:[261 387 275
401],entity:MENU.CNT}{text:"FLXmas 30 Off",Box:[284 384 434
408],entity:MENU.NM}{text:"68,180",Box:[520 390 587
407],entity:MENU.PRICE}{text:"1",Box:[261 405 275
421],entity:MENU.SUB_CNT}{text:"PAKET SLICES",Box:[310 405 460
425],entity:MENU.SUB_NM}{text:"0",Box:[574 410 587
425],entity:MENU.SUB_PRICE}{text:"1",Box:[261 425 275
441],entity:MENU.SUB_CNT}{text:" FL Cake - French",Box:[305 425 488
444],entity:MENU.SUB_NM}{text:"0",Box:[574 430 587
444],entity:MENU.SUB_PRICE}{text:"1",Box:[261 466 273
479],entity:MENU.SUB_CNT}… ✓
Figure 12: Visualization of three cases on CORD, which are predicted by GPT-3
(Column (a)), ChaTGPT (Column (b)), Ground Truth(Column (c)). Column(a),
Column(b) and Column(c) corresponds to Table4. ICL-D3IE with text-davinci-003
(94.12 F1) versus ICL-D3IE with ChatGPT (87.13 F1) on CORD test-set Table 4:
Examples of prediction by GPT-3 and ChatGPT on CORD. GPT-3:
---
{text:"1",Box:[277 427 294 442],entity:MENU.CNT}{text:"[MD] SOFT MORNING
ROLL",Box:[305 427 579 444],entity:MENU.NM}{text:"18,000",Box:[649 427 729
443],entity:MENU.PRICE}{text:"2",Box:[275 445 296
460],entity:MENU.CNT}{text:"REAL HONEY PANCAKE BREAD",Box:[305 445 603
461],entity:MENU.NM}{text:"20,000",Box:[647 444 730
460],entity:MENU.PRICE}{text:"4",Box:[277 463 296
478],entity:MENU.CNT}{text:"CUSTARD BRIOCHE",Box:[305 462 494
478],entity:MENU.NM}{text:"36,000",Box:[648 461 729
478],entity:MENU.PRICE}{text:"TOTAL 74,000",Box:[304 495 733
526],entity:TOTAL.TOTAL_PRICE}{text:"CASH 100,000",Box:[305 524 732
540],entity:TOTAL.CASHPRICE}… ✓
ChatGPT:
{text:"1",Box:[277 427 294 442],entity:MENU.CNT}{text:"[MD] SOFT MORNING
ROLL",Box:[305 427 579 444],entity:MENU.NM}{text:"18,000",Box:[649 427 729
443],entity:MENU.PRICE}{text:"2",Box:[275 445 296
460],entity:MENU.CNT}{text:"REAL HONEY PANCAKE BREAD",Box:[305 445 603
461],entity:MENU.NM}{text:"20,000",Box:[647 444 730
460],entity:MENU.PRICE}{text:"4",Box:[277 463 296
478],entity:MENU.CNT}{text:"CUSTARD BRIOCHE",Box:[305 462 494
478],entity:MENU.NM}{text:"36,000",Box:[648 461 729
478],entity:MENU.PRICE}{text:"TOTAL 74,000",Box:[304 495 733
526],entity:SUB_TOTAL.SUBTOTAL_PRICE}{text:"CASH 100,000",Box:[305 524 732
540],entity:TOTAL.CASHPRICE}… ✗
Ground Truth:
{text:"1",Box:[277 427 294 442],entity:MENU.CNT}{text:"[MD] SOFT MORNING
ROLL",Box:[305 427 579 444],entity:MENU.NM}{text:"18,000",Box:[649 427 729
443],entity:MENU.PRICE}{text:"2",Box:[275 445 296
460],entity:MENU.CNT}{text:"REAL HONEY PANCAKE BREAD",Box:[305 445 603
461],entity:MENU.NM}{text:"20,000",Box:[647 444 730
460],entity:MENU.PRICE}{text:"4",Box:[277 463 296
478],entity:MENU.CNT}{text:"CUSTARD BRIOCHE",Box:[305 462 494
478],entity:MENU.NM}{text:"36,000",Box:[648 461 729
478],entity:MENU.PRICE}{text:"TOTAL 74,000",Box:[304 495 733
526],entity:TOTAL.TOTAL_PRICE}{text:"CASH 100,000",Box:[305 524 732
540],entity:TOTAL.CASHPRICE}… ✓
Figure 13: Visualization of three cases on CORD, which are predicted by
ICL-D3IE with text-davinci-003 (94.12 F1). Column(a) corresponds to Table
5Q(A), Column(b) corresponds to Table 5Q(B) and Column(b) corresponds to Table
6Q(C). Table 5: Examples of prediction by ICL-D3IE on CORD. The part with
green background is generated by LLM.  Q(A): {text:"1",Box:[262 481 276
506]}{text:"CIRENG PANDAWA",Box:[291 479 557 502]}{text:"26,818",Box:[719 478
818 504]}{text:"Subtotal 26,818",Box:[451 537 821 564]}{text:"Pb1
2,681",Box:[512 568 819 596]}{text:"Total 29,500",Box:[476 605 820
633]}{text:"CASH 29,500",Box:[469 632 822 663]}{text:"Total Item 1",Box:[418
665 824 690]}{text:"Total Qty 1",Box:[435 697 825 725]},What are the labels
for these texts?
---
A: {text:"1",Box:[262 481 276 506],entity:MENU.CNT}{text:"CIRENG
PANDAWA",Box:[291 479 557 502],entity:MENU.NM}{text:"26,818",Box:[719 478 818
504],entity:MENU.PRICE}{text:"Subtotal 26,818",Box:[451 537 821
564],entity:SUB_TOTAL.SUBTOTAL_PRICE}{text:"Pb1 2,681",Box:[512 568 819
596],entity:SUB_TOTAL.TAX_PRICE}{text:"Total 29,500",Box:[476 605 820
633],entity:TOTAL.TOTAL_PRICE}{text:"CASH 29,500",Box:[469 632 822
663],entity:TOTAL.CASHPRICE}{text:"Total Item 1",Box:[418 665 824
690],entity:TOTAL.MENUTYPE_CNT}{text:"Total Qty 1",Box:[435 697 825
725],entity:TOTAL.MENUQTY_CNT.} ✓
Q(B): {text:"1",Box:[292 615 300 632]}{text:"BLACK PAPPER MEATBALL
PAS",Box:[311 597 575 629]}{text:"79,000",Box:[601 591 668
606]}{text:"1",Box:[292 636 300 655]}{text:"TRUFFLE CREAM",Box:[311 629 454
650]}{text:"77,000",Box:[605 614 674 627]}{text:"1",Box:[291 660 299
676]}{text:"EARL GREY MILK TEA",Box:[310 645 509 675]}{text:"59,000",Box:[610
636 680 648]}{text:"SUBTTL 215,000",Box:[311 658 685 697]}{text:"Service
Charge 6% 12,900",Box:[312 684 693 725]}{text:"TAX 10% 22,790",Box:[311 706
696 752]}{text:"TOTAL 250,690",Box:[310 763 704 803]},What are the labels for
these texts?
A: {text:"1",Box:[292 615 300 632],entity:MENU.CNT}{text:"BLACK PAPPER
MEATBALL PAS",Box:[311 597 575 629],entity:MENU.NM}{text:"79,000",Box:[601 591
668 606],entity:MENU.PRICE}{text:"1",Box:[292 636 300
655],entity:MENU.CNT}{text:"TRUFFLE CREAM",Box:[311 629 454
650],entity:MENU.NM}{text:"77,000",Box:[605 614 674
627],entity:MENU.PRICE}{text:"1",Box:[291 660 299
676],entity:MENU.CNT}{text:"EARL GREY MILK TEA",Box:[310 645 509
675],entity:MENU.NM}{text:"59,000",Box:[610 636 680
648],entity:MENU.PRICE}{text:"SUBTTL 215,000",Box:[311 658 685
697],entity:SUB_TOTAL.SUBTOTAL_PRICE}{text:"Service Charge 6% 12,900",Box:[312
684 693 725],entity:SUB_TOTAL.SERVICE_PRICE}{text:"TAX 10% 22,790",Box:[311
706 696 752],entity:SUB_TOTAL.TAX_PRICE}{text:"TOTAL 250,690",Box:[310 763 704
803],entity:TOTAL.TOTAL_PRICE}. ✓
Table 6: Examples of prediction by ICL-D3IE on CORD. The part with green
background is generated by LLM. Q(C): {text:"2",Box:[150 416 173
439]}{text:"[MD] SOFT STEAMED CHEESEC",Box:[189 415 629
442]}{text:"20,000",Box:[678 419 789 445]}{text:"2",Box:[150 445 173
469]}{text:"[MD] SOFT STEAMED CHOCOCA",Box:[189 444 629
472]}{text:"20,000",Box:[675 447 787 473]}{text:"TOTAL 40,000",Box:[180 499
784 550]}{text:"CASH 100,000",Box:[180 546 784 572]}{text:"CHANGE
60,000",Box:[173 574 798 625]},What are the labels for these texts?
---
A: {text:"2",Box:[150 416 173 439],entity:MENU.CNT}{text:"[MD] SOFT STEAMED
CHEESEC",Box:[189 415 629 442],entity:MENU.NM}{text:"20,000",Box:[678 419 789
445],entity:MENU.PRICE}{text:"2",Box:[150 445 173
469],entity:MENU.CNT}{text:"[MD] SOFT STEAMED CHOCOCA",Box:[189 444 629
472],entity:MENU.NM}{text:"20,000",Box:[675 447 787
473],entity:MENU.PRICE}{text:"TOTAL 40,000",Box:[180 499 784
550],entity:TOTAL.TOTAL_PRICE}{text:"CASH 100,000",Box:[180 546 784
572],entity:TOTAL.CASHPRICE}{text:"CHANGE 60,000",Box:[173 574 798
625],entity:TOTAL.CHANGEPRICE}. ✓
Table 7: Examples of layout-aware demonstrations produced by ICL-D3IE on
FUNSD. The part with green background is generated by LLM.

Q: {text:"TO:",Box:[84 53 112 67],entity:question}{text:"R. B. SPELL",Box:[147
50 228 68],entity:answer}{text:"SUBMISSION DATE:",Box:[532 52 670
67],entity:header}{text:"FROM:",Box:[85 85 134 102],entity:question}{text:"S.
J. Farnham",Box:[147 85 239 102],entity:answer}{text:"OCT 3",Box:[535 85 581
103],entity:question}{text:"DEC 26",Box:[651 85 704
103],entity:question}{text:"SUBJECT:",Box:[87 122 157
139],entity:question}{text:"HARLEY DAVIDSON 100’S CIGARETTES PROGRESS
REPORT",Box:[161 122 571 139],entity:answer}{text:"OCT 31",Box:[531 105 581
122],entity:question}{text:"JAN 23, 1995",Box:[652 105 705
136],entity:question}{text:"X",Box:[725 105 740
119],entity:answer}{text:"DISTRIBUTION:",Box:[87 180 193
197],entity:question}{text:"EFFECTIVENESS OF SELL (REPORT ON OCT 3
ONLY).",Box:[197 182 596 198],entity:answer}{text:"DIRECT ACCOUNT AND CHAIN
VOIDS (USE X TO INDICATE A VOID).",Box:[85 272 546
292],entity:header}{text:"ACCOUNT",Box:[236 321 303
332],entity:question}{text:"NO. STORES",Box:[511 307 570
331],entity:question}{text:"100’S",Box:[623 318 657
333],entity:question}{text:"LTS. 100’S",Box:[669 306 706
332],entity:question}{text:"NONE",Box:[246 339 289 353],entity:answer}{text:"%
OF DISTRIBUTION ACHIEVED IN RETAIL OUTLETS:",Box:[87 541 447
559],entity:question}…, Please describe the positional relationship of these
text?
---
A: "TO:" is located in the upper left corner with a Box of [84 53 112 67], so
it can be labeled as "question".
"R. B. SPELL" is located on the right of "TO:" with a Box of [147 50 228 68],
so it can be labeled as "answer".
"SUBMISSION DATE:" is located at the top right corner with a Box of [532 52
670 67], so it can be labeled as "header".
"FROM:" is located below "TO:" with a Box of [85 85 134 102], so it can be
labeled as "question".
"S. J. Farnham" is located on the right of "FROM:" with a Box of [147 85 239
102], so it can be labeled as "answer".
"OCT 3" is located below "SUBMISSION DATE:" with a Box of [535 85 581 103], so
it can be labeled as "question".
"DEC 26" is located on the right of "OCT 3" with a Box of [651 85 704 103], so
it can be labeled as "question".

Figure 14: Visualization of two layout-aware demonstrations on FUNSD, which
are generated by ICL-D3IE. Column (a) corresponds to TableA.2 and Column (b)
corresponds to TableA.2. Blue: Header-label, Green: Answer, Red: Question,
Purple: Other.

Table 8: Examples of layout-aware demonstrations produced by ICL-D3IE on
FUNSD. The part with green background is generated by LLM.

Q: {text:"KOOL",Box:[54 41 100 54],entity:question}{text:"0595- 529- 1510
-0",Box:[58 55 193 72],entity:answer}{text:"PROJECT NO.",Box:[430 35 527
55],entity:question}{text:"1987- 144- K",Box:[558 31 653
51],entity:other}{text:"DESCRIPTION:",Box:[54 107 155
126],entity:question}{text:"KMDP ANALYSIS UTILIZING BLACK SWITCHING STUDY
DATA",Box:[197 100 625 125],entity:answer}{text:"MR PERSONNEL:",Box:[55 179
165 196],entity:question}{text:"C. B. Pugh",Box:[200 177 278
195],entity:answer}{text:"SUPPLIER:",Box:[54 224 132
238],entity:question}{text:"Market Facts",Box:[200 218 308
235],entity:answer}{text:"AUTHORIZED COST:",Box:[55 262 194
279],entity:question}{text:"12, 500 (10- 15- 87)",Box:[198 258 372
280],entity:answer}{text:"CARRYOVER TO 1988 – 0",Box:[460 275 659
296],entity:other}{text:"RELEASED TO ACCTG",Box:[75 517 151
551],entity:question}{text:"PAID OUT OF 1987 BUDGET",Box:[292 509 389
548],entity:question}{text:"CUMULATIVE",Box:[553 527 637
545],entity:question}{text:"JAN",Box:[195 568 227
583],entity:answer}{text:"466024372",Box:[548 884 671
906],entity:other}{text:"PRODUCED FROM B&W WEB SITE",Box:[5 950 228
970],entity:other}{text:"466024372",Box:[678 950 744 968],entity:other},
Please describe the positional relationship of these text?
---
A: "KOOL" has a Box of [54 41 100 54] and is located in the upper left corner,
so it can be labeled as "question".
"0595- 529- 1510 -0" has a Box of [58 55 193 72] and is located on the below
of "KOOL", so it can be labeled as "answer".
"PROJECT NO." has a Box of [430 35 527 55] and is located at the top right
corner, so it can be labeled as "question".
"1987- 144- K" has a Box of [558 31 653 51] and is located on the right of
"PROJECT NO.", so it can be labeled as "other".
"DESCRIPTION:" has a Box of [54 107 155 126] and is located below of "KOOL"
question, so it can be labeled as "question".

Figure 15: Visualization of three cases on SROIE, which are predicted by GPT-3
(Column (a)), ChaTGPT (Column (b)) and Ground Truth (Column (c)). ICL-D3IE
with text-davinci-003 (97.88 F1) versus ICL-D3IE with ChatGPT (92.63 F1).
Blue: Company-label, Green: Address, Red: Total, Purple: Date.

Table 9: Predictions produced by GPT-3 and ChatGPT on SROIE. The visualization
results are shown in Figure15. GPT-3:
---
{"KEDAI PAPAN YEW CHUAN"}.
{"LOT 276 JALAN BANTING"}{"43800 DENGKIL, SELANGOR."}.
{"283.55"}.
{"12/03/2018"}.
✓
ChatGPT:
{"KEDAI PAPAN YEW CHUAN"}.
{"LOT 276 JALAN BANTING"}{"43800 DENGKIL, SELANGOR."}{"TEL :
03-87686092"}{"FAX : 03-87686092"}.
{"283.55"}.
{"12/03/2018"}.
✗
Ground Truth:
{"KEDAI PAPAN YEW CHUAN"}.
{"LOT 276 JALAN BANTING"}{"43800 DENGKIL, SELANGOR."}.
{"283.55"}.
{"12/03/2018"}.
✓

Figure 16: Visualization of three cases on SROIE, which are predicted by
GPT-3. Blue: Company-label, Green: Address, Red: Total, Purple: Date.

Table 10: Predictions produced by GPT-3 on SROIE. The visualization results
are shown in Figure16. GPT-3:
---
{"STAR GROCER SDN BHD (1171881-W)"}.
{"NO 4, DESA PANDAN, OFF KAMPONG PANDAN,"}{"55100 KUALA LUMPUR"}.
{"66.15"}.
{"25/03/18"}.
✓
GPT-3:
{"SYARIKAT PERNIAGAAN GIN KEE"}.
{"NO 290. JALAN AIR PANAS."}{"SETAPAK."}{"53200, KUALA LUMPUR."}.
{"36.04"}.
{"01/02/2018"}.
✓
GPT-3:
{"GUARDIAN HEALTH AND BEAUTY SDN BHD"}.
{"LOT B-005-006, BASEMENT LEVEL 1"}{"THE STARLING MALL"}{"JALAN SS21/60,
DAMANSARA UTAMA."}.
{"108.21"}.
{"16/08/17"}.
✓

Table 11:  An Example of ICL-D3IE Prompt for FUNSD.Blue: Hard Demonstrations,
Green: Formatting Demonstrations, Red: Layout-Aware Demonstrations, Purple:
Label Mapping.

Prompt for FUNSD NER
---
There are four labels for selection, "question", "answer", "header", and
"other". sample 1 "Attendence" : other "140" : other "QUANTITY REQUIRED" :
question "CUSTOMER SHIPPING NUMBER" : question "DATE FORWARDED TO PROMOTION
SERVICES:" : question "REQFORM" : other "01/17/97" : other
sample 2
sample 3 "MARKETING RESEARCH PROJECT APPROVAL" : header
… Context:{text:"Name of Account",Box:[369 481 459
497],entity:question}{text:"Ind/ Lor Volume",Box:[494 472 537
503],entity:question}
…
Q:{text:"KOOL",Box:[54 41 100 54],entity:question}{text:"0595- 529- 1510
-0",Box:[58 55 193 72],entity:answer}{text:"PROJECT
A:"KOOL" has a Box of [54 41 100 54] and is located in the upper left corner,
so it can be labeled as "question". …
Q:{text:"ACUTE TOXICITY IN MICE",Box:[295 56 512 79]}{text:"COMPOUND",Box:[84
109 136 119]},What are the labels for these texts?
A:{text:"ACUTE TOXICITY IN MICE",Box:[295 56 512
79],entity:header}{text:"COMPOUND",Box:[84 109 136 119],entity:question}.
Q:

Table 12:  An Example of ICL-D3IE Prompt for CORD. Green: Formatting
Demonstrations, Red: Layout-Aware Demonstrations, Purple: Label Mapping.

Prompt for CORD NER
---
MENU.NM : name of menu Context: {text:"NASI PUTIH",Box:[137 446 366
475],entity:MENU.NM}{text:"13,000",Box:[730 446 871 476],entity:MENU.PRICE}
{text:"AYAM GR BUMBU",Box:[135 511 438
539],entity:MENU.NM}{text:"44,000",Box:[724 512 869 542],entity:MENU.PRICE}
{text:"2X",Box:[134 479 184 506],entity:MENU.CNT}{text:"22,000",Box:[500 480
638 508],entity:MENU.UNITPRICE}{text:"NESTLE 330 ML",Box:[138 573 433
601],entity:MENU.NM}{text:"16,000",Box:[725 576 865 605],entity:MENU.PRICE}
…
Context:{text:"BASO TAHU BIHUN",Box:[49 649 361
680],entity:MENU.NM}{text:"1",Box:[449 646 472 671],entity:MENU.CNT}
Q:What is difference between MENU.UNITPRICE and MENU.PRICE?
A:There are two texts "43.636" on the same line. The text on the left is
labeled as MENU.UNITPRICE and the text on the right is labeled as MENU.PRICE.
Context:{text:"Rp",Box:[608 584 667
610],entity:SUB_TOTAL.SUBTOTAL_PRICE}{text:"Rp",Box:[606 673 667
703],entity:TOTAL.
Q:which label selection for text "Rp" with [608 584 667 610]?
A:SUB_TOTAL.SUBTOTAL_PRICE,the text "Rp" with box [608 584 667 610] is located
on the same line and obscured by the text ’Subtotal 12.000’ with box [81 584
831 610].
…
Q:{text:"Berry Many-Low (P)",Box:[320 418 519 444]},What are the labels for
these texts?
A:{text:"Berry Many-Low (P)",Box:[320 418 519 444],entity:MENU.NM}.
Q:{text:"Kupon 1",Box:[135 548 250 576]}{text:"Total Item: 2",Box:[0 621 506
666]},What are the labels for these texts?
A:{text:"Kupon 1",Box:[135 548 250 576],entity:MENU.NM}{text:"Total Item:
2",Box:[0 621 506 666],entity:TOTAL.MENUQTY_CNT}.
Q:

Table 13:  An Example of ICL-D3IE Prompt for SROIE. Green: Formatting
Demonstrations, Red: Layout-Aware Demonstrations, Purple: Label Mapping.
Prompt for SROIE NER
---
There are four labels for selection, "company", "address", "date", and
"total". Context:"99 SPEED MART S/B" : company "LOT P.T. 2811, JALAN ANGSA,
TAMAN BERKELEY 41150 KLANG, SELANGOR 1243-JLN PUDU ULU" : address "AEON CO.
(M) SDN BHD" : company "3RD FLR, AEON TAMAN MALURI SC JLN JEJAKA, TAMAN MALURI
CHERAS, 55100 KUALA LUMPUR" : address "BENS INDEPENDENT GROCER SDN. BHD" :
company "LOT 6, JALAN BATAI, PLAZA BATAI, DAMANSARA HEIGHTS 50490, KUALA
LUMPUR" : address
…
Context:{text:"TOTAL SALES INCL GST",Box:[1598 2761 2305
2855],entity:other}{text:"38.59",Box:[2833 2763 3008
2848],entity:other}{text:"TOTAL AFTER ADJ INCL GST",Box:[1595 2876 2445
2957],entity:other}{text:"38.60",Box:[2827 2871 3012 2950],entity:total}.
Q:Why "38.60" labeled as total?
A:Because "38.60" is on the right of "TOTAL AFTER ADJ INCL GST".
Context:{text:"TOTAL INCLUSIVE GST:",Box:[84 716 352
741],entity:other}{text:"29.00",Box:[562 718 629 743],entity:total}.
Q:Why "29.00" labeled as total?
A:Because "29.00" is on the right of "TOTAL INCLUSIVE GST:".
…
Context:{text:"107",Box:[347 216 454 293],entity:other}{text:"GERBANG ALAF
RESTAURANTS SDN BHD",Box:[75 338 649
380],entity:company}{text:"(65351-M)",Box:[272 385 437
429],entity:other}{text:"GOLDEN ARCHES RESTAURANTS SDN BHD",Box:[64 469 662
511],entity:other}{text:"FORMERLY KNOWN AS",Box:[203 431 515
469],entity:other}{text:"LEVEL 6, BANGUNAN TH, DAMANSARA UPTOWN3",Box:[24 560
727 602],entity:address}{text:"LICENSEE OF MCDONALD’S",Box:[161 512 568
560],entity:other}{text:"NO.3, JALAN SS21/39, 47400 PETALING JAYA",Box:[26 604
723 648],entity:address}{text:"(GST ID NO: 000504664064)",Box:[143 693 594
737],entity:other}{text:"SETANGOR",Box:[286 649 441
693],entity:address}{text:"MCDONALD’S SHELL MAHKOTA CHERAS DT (#36",Box:[18
739 725 783],entity:other}{text:"TEL NO. 03-9010-9849",Box:[189 784 568
828],entity:other}{text:"QTY ITEM",Box:[20 1088 177
1126],entity:other}{text:"ORD #07 -REG #19- 10/03/2018",Box:[22 1048 526
1090],entity:date}{text:"17:24:07",Box:[536 1046 685
1088],entity:other}{text:"TOTAL",Box:[625 1098 712 1130],entity:other}{text:"1
M COKE",Box:[60 1140 213 1174],entity:other}{text:"3.50",Box:[641 1140 716
1176],entity:other}{text:"1 MCCHICKEN",Box:[56 1181 257
1215],entity:other}{text:"5.00",Box:[639 1179 712
1223],entity:other}{text:"TAKEOUT TOTAL (INCL GST)",Box:[20 1263 457
1311],entity:other}{text:"TOTAL ROUNDED",Box:[24 1308 260
1350],entity:other}{text:"8.50",Box:[639 1271 716
1311],entity:total}{text:"8.50",Box:[635 1314 716
1356],entity:total}{text:"CASH TENDERED",Box:[26 1358 272 1394],entity:other}
Q:Return text labeled as company, original address, total, and date?
A:{"GERBANG ALAF RESTAURANTS SDN BHD"}.
Q:Return one company and its original address, one total, and one date?
A:{"GERBANG ALAF RESTAURANTS SDN BHD"}.
Q:


# Motion Consistency Loss for Monocular Visual Odometry with Attention-Based
Deep Learning
††thanks:  André Françani was supported by CAPES – Coordination of Improvement
of Higher Education Personnel under grant 88887.687888/2022-00. Marcos Maximo
is partially funded by CNPq –- National Research Council of Brazil through
grant 307525/2022-8.

1st André O. Françani  Autonomous Computational Systems Lab (LAB-SCA)
Computer Science Division
Aeronautics Institute of Technology
São José dos Campos, SP, Brazil
<EMAIL_ADDRESS>2nd Marcos R. O. A. Maximo  Autonomous Computational
Systems Lab (LAB-SCA)
Computer Science Division
Aeronautics Institute of Technology
São José dos Campos, SP, Brazil
<EMAIL_ADDRESS>
###### Abstract

Deep learning algorithms have driven expressive progress in many complex
tasks. The loss function is a core component of deep learning techniques,
guiding the learning process of neural networks. This paper contributes by
introducing a consistency loss for visual odometry with deep learning-based
approaches. The motion consistency loss explores repeated motions that appear
in consecutive overlapped video clips. Experimental results show that our
approach increased the performance of a model on the KITTI odometry benchmark.

###### Index Terms:

deep learning, loss function, transformer, monocular visual odometry

††
979-8-3503-1538-7/23/$31.00©2023 IEEE

##  I Introduction

Deep learning (DL) techniques have shown to be state-of-the-art in diverse
complex applications, such as computer vision (CV) and natural language
processing (NLP). As evidence, DL models can be employed in image
classification tasks [1], video surveillance [2], text translation [3], and
others [4].

A fundamental component of deep learning techniques is the loss function,
being a core element in the optimization algorithm. The loss function is
minimized during the training step and the model’s parameters are adjusted
according to the error between the model’s prediction and the expected ground
truth. Different loss functions guide the training to a specific task, tuning
the model’s parameters to a good performance in regression or classification
tasks. This means that the choice of the loss function impacts what the model
is in fact learning. The focal loss [5] is a clear example where the loss
function played an important role in object detection applications, enabling
inferences as fast as one-stage detectors while reaching accuracy competitive
to two-stage detectors.

A consistency loss may introduce a gain of information capable of making the
model more robust and more reliable, increasing the performance. Thus, a
consistency loss may guide the learning process to be more accurate in its
predictions regarding a particular task, reducing the error between the
predicted output and the expected ground truth. Zhu _et al._ [6] achieved
outstanding qualitative results in generative models by introducing a cycle
consistency loss in the image-to-image translation task.

In this paper, our contribution is a novel consistency loss function that
deals with repeated motion in overlapped clips in the context of monocular
visual odometry with deep learning. We evaluate our method on the KITTI
odometry benchmark [7], showing that our approach increases the performance of
the model.

The remaining of this paper is organized as follows. Section II provides
theoretical background about visual odometry and attention mechanisms. Section
III introduces the proposed motion consistency loss. Section IV describes the
experimental setups. Section V shows the experimental results on the KITTI
odometry benchmark. Finally, Section VI concludes and shares our ideas for
future work.

##  II Background

###  II-A Monocular visual odometry

Visual odometry (VO) is widely applied in robotics and autonomous vehicles to
estimate the camera’s pose given a sequence of image frames [8]. The motion
between consecutive time steps $k-1$ and $k$ comprises a rotation matrix
$\mathbf{R}_{k}\in S\\!O(3)$ and a translation vector
$\mathbf{t}_{k}\in\mathbb{R}^{3\times 1}$. Note that $\mathbf{R}_{k}$ and
$\mathbf{t}_{k}$, respectively, depict the rotation and translation from time
step $k-1$ to $k$. The complete motion can be written as a transformation
$\mathbf{T}_{k}\in\mathbb{R}^{4\times 4}$, defined as

$\mathbf{T}_{k}=\begin{bmatrix}\mathbf{R}_{k}&\mathbf{t}_{k}\\\
\mathbf{0}&\mathbf{1}\\\ \end{bmatrix}.$

Furthermore, the poses in VO problems are typically described through 6
degrees of freedom (6-DoF): three rotational, and three translational. To
address the visual odometry problem of estimating the 6-DoF, one can rely on
traditional geometry-based approaches [9, 10], deep learning-based approaches
through end-to-end architectures [11, 12], and hybrid methods that mix deep
learning-based and geometry-based algorithms depending on the component of the
visual odometry [13, 14]. Fig. 1 illustrates the camera’s motion between
consecutive time steps. In Fig. 1, the blue dots are keypoints in the
scenario, which are used as a reference to estimate the motion via a geometry-
based method.

Figure 1: Camera’s motion between consecutive time steps.

In the monocular visual odometry case, where the image frames are captured
from a single camera, the motion can be obtained up to a scale, resulting in
scale ambiguity. This is caused by the loss of depth information when
projecting three-dimensional (3D) objects onto the two-dimensional (2D) image
space. The accumulation of scale errors over time is called scale drift, being
a crucial factor in decreasing the accuracy in monocular visual odometry
systems [8].

###  II-B Time-space attention in monocular visual odometry

Deep learning-based methods are able to estimate the 6-DoF camera’s pose
directly from RGB images. In an end-to-end manner, the models estimate the
pose given only the sequence of frames as input. DeepVO [11] is an example of
a deep learning-based method that uses a convolutional neural network (CNN) to
extract features from consecutive image frames, as well as long short-term
memory cells (LSTM) to deal with the temporal information. Furthermore,
unsupervised learning techniques can also be employed, as in the UnDeepVO [12]
that uses depth estimation to recover the scale.

For many years, LSTM cells have dominated tasks involving temporal
information. However, architectures based on attention mechanisms, such as the
Transformer architecture [3], have outperformed the accuracy of LSTM models,
and have become state-of-the-art in NLP applications.

The input of the attention block is composed of the query, key, and value
vectors, all with dimension $d$. Then, those vectors from different inputs are
packed together into the matrices $\mathbf{Q}$, $\mathbf{K}$, and
$\mathbf{V}$, which are the inputs of the attention layer. The processing
steps can be defined as the Attention function [3]:

$\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}}\right)\mathbf{V},$
(1)

where Softmax is the well-known Softmax function.

Another important structure is the multi-head self-attention layer (MHSA),
which has $N_{h}$ concatenated attention layers running in parallel [3]. Since
then, the deep learning community has started building models based on the
decoder and encoder blocks of the Transformer, and new state-of-the-art
networks have begun to emerge, such as BERT [15] and GPT [16]. The Transformer
architecture also achieved state-of-the-art performance in image recognition
benchmarks [1] and video understanding tasks [17, 18]. For vision tasks, the
images are decomposed into $N$ non-overlapping patches of size $P\times P$.
Each patch is flattened and embedded into tokens that are sent as input to the
network [1].

Video understanding tasks require space and time information to recognize
actions or events in videos. To address this problem with the attention
mechanism, G. Bertasius _et al._ [18] proposed mainly two different self-
attention architectures to extract spatio-temporal features. One of the
proposed architectures is the “joint space-time” self-attention, which relates
all tokens in space and time together. The other one, more efficient
computationally in terms of processing complexity, is the “divided space-time”
self-attention. It applies first the attention over tokens with the same
spatial index (temporal axis) followed by the attention over the tokens from
the same frame (spatial axis).

In the context of visual odometry, our previous work [19] uses a Transformer-
based network with the “divided space-time” to estimate the 6-DoF camera’s
pose for the monocular case. The TSformer-VO is based on the TimeSformer
architecture with “divided space-time” self-attention in its encoding blocks.
The model receives clips with $N_{f}$ frames of size $C\times H\times W$ as
input, where $C$, $H$, and $W$ are the number of channels, the height, and the
width of the image frames, respectively. Each clip is divided into
$N=HW/P^{2}$ patches, and each patch is embedded into tokens through a linear
map. Finally, the outputs of the embedding layer are the input tokens denoted
as $\mathbf{z}^{l}_{(s,t)}\in\mathbb{R}^{E_{d}}$, where $E_{d}$ is the
embedding dimension of the flattened patch at spatial location $s$, time index
$t$, and encoding block $l$. Notice that the model has $L_{x}$ encoder blocks
stacked, indicating the depth of the network in terms of encoding layers.

The encoder blocks follow the traditional layers present in the Transformers
encoders, namely the layer normalization (LN), multi-head self-attention
(MHSA), residual connections, fully-connected layer (FC), and multilayer
perceptron (MLP). For the specific “divided space-time” self-attention, those
layers are related as follows:

$\begin{split}\mathbf{a}^{l}_{t}&=\text{MHSA}\left(\text{LN}\left(\mathbf{z}_{(s,t)}^{l-1}\right)\right)+\mathbf{z}_{(s,t)}^{l-1},\\\
\mathbf{a}^{l}_{t_{FC}}&=\text{FC}\left(\mathbf{a}^{l}_{t}\right),\\\
\mathbf{a}^{l}_{s}&=\text{MHSA}\left(\text{LN}\left(\mathbf{a}^{l}_{t_{FC}}\right)\right)+\mathbf{a}^{l}_{t_{FC}},\\\
\mathbf{z}_{(s,t)}^{l}&=\text{MLP}\left(\text{LN}\left(\mathbf{a}^{l}_{s}\right)\right)+\mathbf{a}^{l}_{s}.\\\
\end{split}$ (2)

Further details about the exact computation of the query, key, and value can
be found in [3, 18]. Fig. 2 contains the block diagram of the encoder block
with “divided space-time” self-attention, summarizing (2).

Figure 2: Encoder block with “divided space-time” self-attention.

##  III Proposed Method

The proposed method relies on providing extra information during the training
step to increase overall performance. The information gain comes from the fact
that we are sampling consecutive clips with $N_{f}-1$ overlapped frames. This
means that the same motion is estimated from different input clips, as shown
in Fig. 3. With this in mind, we can conduct the learning process of the
network by introducing a consistency loss. The idea behind our approach is
that consecutive clips should estimate the same motion where they overlap,
that is, consecutive clips contain common motions due to the overlapping
introduced by sampling the clips. Without the consistency loss, it is not
directly clear to the network that consecutive clips contain similar
information, once our approach uses the information of the entire clip to
infer all the poses in a pair of consecutive frames.

Figure 3: Schematic representation of the proposed method. Input clips with
overlapped frames go to a model that estimates the camera’s motion between
consecutive time steps. During the training step, the motion consistency loss
is calculated using the estimated motions that appear in all input clips
($\hat{\mathbf{T}}_{2}$ in this example). The motion consistency loss is
weighted by a hyperparameter $\alpha$, and this result is added to the MSE of
all estimations. The final loss propagates back to the model during the
training step.

The main part of the total loss function is the mean squared error (MSE)
between all predicted motions and their ground truths, given by:

$\mathcal{L}_{\textsc{}{MSE}}=\frac{1}{N_{f}-1}\sum_{w=1}^{N_{f}-1}{\|\mathbf{y}_{w}^{k}-\hat{\mathbf{y}}_{w}^{k}\|_{2}^{2}},$
(3)

where $\|\cdot\|_{2}^{2}$ is the squared L2 norm, $\mathbf{y}_{w}^{k}$ is the
flattened 6-DoF relative pose at position $w$ of the clip at instant $k$, and
$\hat{\mathbf{y}}_{w}^{k}$ is its estimate by the network. When dealing with
batch processing, the loss is reduced by the mean over all $B_{s}$ batch
elements.

The second part of the total loss function is our motion consistency loss
$\mathcal{L}_{\textsc{}{MC}}$. It is the sum of the squared errors of all
predicted motions that appear in more than one input clip, that is, the same
motion but from different clips. Therefore, the $\mathcal{L}_{\textsc{}{MC}}$
is defined as:

$\mathcal{L}_{\textsc{}{MC}}=\sum_{j=1}^{2N_{f}-5}\sum_{n=m+1}^{\gamma}\sum_{m=\mu}^{\lambda}{\|\hat{\mathbf{y}}_{k-j}^{k-m}-\hat{\mathbf{y}}_{k-j}^{k-n}\|_{2}^{2}},$
(4)

where $\mu=\max{(j-N_{f}+2,0)},\lambda=\min{(N_{f}-3,j-1)}$, and
$\gamma=\min{(N_{f}-2,j)}$.

The final loss is the sum of the MSE of all estimations and the motion
consistency loss, this one weighted by a hyperparameter $\alpha$ to measure
and balance the influence of the consistency loss in the training process.
Therefore, the final loss is defined as follows:

$\mathcal{L}=\mathcal{L}_{\textsc{}{MSE}}+\alpha\mathcal{L}_{\textsc{}{MC}}.$
(5)

##  IV Experimental Setup

This section contains further information on the model configurations we
tested in our experiments, the training setup, the dataset we employed, as
well as the evaluation metrics we chose to compare the performance of the
models.

###  IV-A Model setup

This work aims to study the influence of the motion consistency loss on the
model’s performance. Therefore, different architectures were not evaluated,
and the model hyperparameters were selected based on our previous work [19].
All three models in Table I have $L_{x}=12$ encoders, each one with $N_{h}=6$
heads in the MHSA layer. The patch size is $P=16$, and the embedding dimension
is set to $E_{d}=384$. Moreover, we tested our approach for clips with
$N_{f}=3$ frames, meaning that consecutive clips have two overlapped frames,
i.e. one motion in common.

We tested our approach considering different values for $\alpha$ to weight the
motion consistency loss. Table I shows the definition of the models according
to their loss functions.

TABLE I: Definition of the models with their corresponding loss function Model | Loss function
---|---
A | $\mathcal{L}_{\text{MSE}}$
B | $\mathcal{L}_{\text{MSE}}+\mathcal{L}_{\text{MC}}$
C | $\mathcal{L}_{\text{MSE}}+10\mathcal{L}_{\text{MC}}$

###  IV-B Training setup

We used a computer with a Intel i9-7900X CPU 3.3GHz CPU and a GeForce GTX 1080
Ti GPU with 11GB VRAM for computations, and all the deep learning procedures
were implemented with PyTorch 1.10. The models were trained from scratch using
random initialization for 95 epochs with Adam optimization. The learning rate
was set to $1\times 10^{-5}$, and all the other hyperparameters of the Adam
algorithm were the default values.

Aiming to recycle the code from our previous work [19], we sampled the clips
with a sliding window of size $2N_{f}$ and stride $1$. After that, we shuffled
the sampled clips and created batches of size $B_{s}=2$. This way, clips in
the same batch are not consecutive to each other. However, our approach
requires consecutive clips only for the training step to calculate the motion
consistency loss. To solve this issue, we split the batches in half, each one
containing clips with $N_{f}$ frames. The main advantage of this sampling
process is that we ensure that the clips in the same batch are still shuffled
while having their consecutive clips in a second batch. After that, we
concatenated both small batches into only one to take advantage of the batch
processing ability of deep learning frameworks. Furthermore, this helped us to
fit our GPU memory capacity.

###  IV-C Dataset

We developed our algorithms considering the KITTI dataset [7], which is a
benchmark for evaluating visual odometry algorithms. The imagery is recorded
from a stereo camera on a moving vehicle. The scenario varies from roads and
streets on a normal day in the city, and the car’s speed varies from 0 to 90
km/h, challenging visual odometry algorithms at high-speed scenarios. To make
the system monocular for our study case, we selected only the RGB images
recorded by the left camera. In the KITTI dataset, there are 22 different
sequences with different lengths corresponding to different rides. However,
the camera’s position and orientation over time are only available for 11 of
them.

All image frames were resized to $192\times 640$, keeping the aspect ratio of
the dataset while making the dimensions multiple of the patch size ($P=16$) to
divide the frames into patches. Furthermore, sequences 00, 02, 08, and 09 are
used as training data since they are the four largest recordings.
Consequently, the test data is composed of sequences 01, 03, 04, 05, 06, 07,
and 10, following the choice in [11].

###  IV-D Evaluation metrics

We use the Python KITTI evaluation toolbox††https://github.com/Huangying-
Zhan/kitti-odom-eval to compare the performance of the models. The metrics are
defined as follows:

  * •

$t_{err}$: average translational error, given in percentage (%);

  * •

$r_{err}$: average rotational error, given in degrees per 100 meters (º/100
m);

  * •

ATE: absolute trajectory error, measured in meters;

  * •

RPE: relative pose error for rotation and translation, measured frame-to-frame
in degrees (º) and in meters, respectively for the rotation and the
translation.

According to [7], the average rotational and translational errors are computed
for all subsequences of length $(100,200,\dots,800)$ meters.

Prior works evaluate monocular methods with an optimization transformation to
align the predictions with the ground truths [10, 13]. Thus, our results are
under the 7-DoF alignment due to the scale ambiguity in monocular systems.

##  V Experimental results

We evaluated the performance of the models in Table I using the KITTI dataset.
After the training step, we obtained the quantitative results present in Table
II. For each sequence, the best values of each evaluation metric are
highlighted in bold, and the second-best values are underlined.

TABLE II: Quantitative results for the 11 KITTI sequences with ground truth.

|  | Sequence
---|---|---
| Model | 00 | 01 | 02 | 03 | 04 | 05 | 06 | 07 | 08 | 09 | 10
| A | 9.053 | 31.116 | 3.105 | 9.867 | 8.029 | 12.800 | 27.212 | 23.475 | 5.498 | 5.063 | 18.544
| B | 3.609 | 41.534 | 4.084 | 10.869 | 6.218 | 12.263 | 25.554 | 25.224 | 3.283 | 4.099 | 12.188
$t_{err}$ (%) | C | 4.942 | 28.324 | 5.062 | 15.651 | 4.316 | 13.198 | 22.334 | 18.300 | 5.028 | 5.785 | 12.602
| A | 2.018 | 6.885 | 1.063 | 4.295 | 3.630 | 4.472 | 9.264 | 9.085 | 2.264 | 1.715 | 6.289
| B | 1.433 | 6.137 | 1.528 | 4.397 | 3.412 | 4.137 | 7.766 | 9.190 | 1.366 | 1.351 | 5.496
$r_{err}$ (º/100m) | C | 2.076 | 6.231 | 1.660 | 7.540 | 1.733 | 4.157 | 6.147 | 6.299 | 1.977 | 1.889 | 4.595
| A | 62.718 | 218.055 | 37.431 | 8.696 | 3.998 | 60.149 | 84.704 | 35.929 | 47.987 | 27.426 | 26.291
| B | 19.424 | 152.154 | 80.454 | 11.412 | 3.971 | 51.414 | 69.889 | 36.430 | 14.708 | 15.553 | 23.749
ATE (m) | C | 48.945 | 133.490 | 48.249 | 15.583 | 2.469 | 68.946 | 73.715 | 30.908 | 37.251 | 27.547 | 21.791
| A | 0.107 | 0.601 | 0.026 | 0.112 | 0.104 | 0.141 | 0.366 | 0.164 | 0.030 | 0.037 | 0.146
| B | 0.037 | 1.060 | 0.024 | 0.138 | 0.113 | 0.141 | 0.290 | 0.178 | 0.031 | 0.033 | 0.150
RPE (m) | C | 0.038 | 0.646 | 0.040 | 0.130 | 0.066 | 0.140 | 0.307 | 0.157 | 0.041 | 0.031 | 0.166
| A | 0.361 | 0.320 | 0.283 | 0.288 | 0.170 | 0.263 | 0.252 | 0.289 | 0.281 | 0.229 | 0.324
| B | 0.356 | 0.299 | 0.277 | 0.285 | 0.165 | 0.262 | 0.241 | 0.290 | 0.274 | 0.221 | 0.327
RPE (º) | C | 0.339 | 0.296 | 0.267 | 0.288 | 0.148 | 0.263 | 0.237 | 0.284 | 0.258 | 0.219 | 0.321

Note that the results of sequences 00, 02, 08, and 09 should not be considered
in the analysis once they are used for training the models. However, they are
shown in Table II for a complete report, as well as to confirm that the models
are not completely overfitted to the training data.

By ignoring those sequences, it can be seen that the presence of the motion
consistency loss in models B and C increased the overall performance if
compared to model A (only the MSE loss). Considering the $t_{err}$, model B
outperforms model A in all sequences but in sequences 01 and 03. For the ATE,
model B showed worse performance than model A only in sequence 03. However,
this metric reveals that the consistency loss led to a significant
improvement, especially in sequences 01, 05, and 06. Similar conclusions can
be made for the rotational metric. For example, the $r_{err}$ shows that model
B outperforms model A in all sequences but in sequences 03 and 07. However,
the rotational metrics were not significantly affected by the consistency
loss, once the metric values are close to each other for all the models. The
same happens for the relative pose errors. Furthermore, similar conclusions
can be drawn for model C since it is also frequently present in the top 1 and
top 2 best values.

Therefore, the consistency loss improved the overall performance when
comparing models B and C with model A, especially regarding the translational
aspect.

Following the analysis of the impact of the consistency loss in visual
odometry, we displayed in Fig. 4 the predicted trajectories of models A, B,
and C for qualitative analysis. Fig. 4 reveals that model B is predominantly
closer to the ground truth than the other models. Moreover, it also reveals
that the scale drift problem still remains pronounced, typical of monocular
systems.

Figure 4: Trajectories obtained by model A(0.2ex$\blacksquare\blacksquare$),
model B (0.2ex$\blacksquare\blacksquare$), and model C
(0.2ex$\blacksquare\blacksquare$), compared with the ground truth
(0.2ex$\blacksquare\blacksquare$). All depicted sequences belong to the test
set, but sequence 09. Trajectories are obtained under the 7-DoF alignment.

##  VI Conclusion

We proposed a consistency loss for visual odometry systems with deep learning.
Experimental results have demonstrated that the addition of our motion
consistency loss increased the overall performance of the model considering
the KITTI dataset. Our results also showed that the consistency loss was most
prominent for the translational metrics, not being highly effective in the
rotational aspect. Despite an improvement in model performance for visual
odometry, the scale drift problem still remains significant for monocular
systems, which is expected since the model is not designed to infer depth
features to estimate the scale.

For future research, we expect to minimize the scale drift problem by
addressing a depth estimator together [14] with our approach. Furthermore, we
are looking for new deep learning-based architectures to estimate the camera’s
pose in an end-to-end manner.

## References

  * [1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [2] D. F. Dos Santos, A. O. Françani, M. R. O. A. Maximo, and A. S. C. Ferreira, “Performance comparison of convolutional neural network models for object detection in tethered balloon imagery,” in _2021 Latin American Robotics Symposium (LARS), 2021 Brazilian Symposium on Robotics (SBR), and 2021 Workshop on Robotics in Education (WRE)_. IEEE, 2021, pp. 246–251.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [4] T. Prangemeier, C. Wildner, A. O. Françani, C. Reich, and H. Koeppl, “Yeast cell segmentation in microstructured environments with deep learning,” _Biosystems_ , vol. 211, p. 104557, 2022.
  * [5] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proc. of the IEEE Int. Conf. on Computer Vision_ , 2017, pp. 2980–2988.
  * [6] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proc. of the IEEE Int. Conf. on Computer Vision_ , 2017, pp. 2223–2232.
  * [7] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite,” in _Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2012.
  * [8] D. Scaramuzza and F. Fraundorfer, “Visual odometry [tutorial],” _IEEE Robotics & Automation Magazine_, vol. 18, no. 4, pp. 80–92, 2011.
  * [9] A. Geiger, J. Ziegler, and C. Stiller, “StereoScan: Dense 3D reconstruction in real-time,” in _Intelligent Vehicles Symposium (IV)_ , 2011.
  * [10] R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras,” _IEEE Transactions on Robotics_ , vol. 33, no. 5, pp. 1255–1262, 2017.
  * [11] S. Wang, R. Clark, H. Wen, and N. Trigoni, “DeepVO: Towards end-to-end visual odometry with deep recurrent convolutional neural networks,” in _2017 IEEE Int. Conf. on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 2043–2050.
  * [12] R. Li, S. Wang, Z. Long, and D. Gu, “UnDeepVO: Monocular visual odometry through unsupervised deep learning,” in _2018 IEEE Int. Conf. on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 7286–7291.
  * [13] H. Zhan, C. S. Weerasekera, J.-W. Bian, and I. Reid, “Visual odometry revisited: What should be learnt?” in _2020 IEEE Int. Conf. on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 4203–4210.
  * [14] A. O. Françani and M. R. O. A. Maximo, “Dense prediction transformer for scale estimation in monocular visual odometry,” in _2022 Latin American Robotics Symposium (LARS), 2022 Brazilian Symposium on Robotics (SBR), and 2022 Workshop on Robotics in Education (WRE)_ , 2022, pp. 1–6.
  * [15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [16] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever _et al._ , “Improving language understanding by generative pre-training,” 2018.
  * [17] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid, “ViViT: A video vision transformer,” in _Proc. of the IEEE/CVF Int. Conf. on Computer Vision (ICCV)_ , October 2021, pp. 6836–6846.
  * [18] G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?” in _Proc. of the Int. Conf. on Machine Learning (ICML)_ , July 2021.
  * [19] A. O. Françani and M. R. O. A. Maximo, “Transformer-based model for monocular visual odometry: a video understanding approach,” _arXiv preprint arXiv:2305.06121_ , 2023.