# UniTable: Towards a Unified Framework for Table Recognition via Self-
Supervised Pretraining

ShengYun Peng1 Aishwarya Chakravarthy1 Seongmin Lee1 Xiaojing Wang2
Rajarajeswari Balasubramaniyan2 Duen Horng Chau1
1Georgia Tech 2ADP, Inc.
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Tables convey factual and quantitative data with implicit conventions created
by humans that are often challenging for machines to parse. Prior work on
table recognition (TR) has mainly centered around complex task-specific
combinations of available inputs and tools. We present UniTable, a training
framework that unifies both the training paradigm and training objective of
TR. Its training paradigm combines the simplicity of purely pixel-level inputs
with the effectiveness and scalability empowered by self-supervised
pretraining (SSP) from diverse unannotated tabular images. Our framework
unifies the training objectives of all three TR tasks — extracting table
structure, cell content, and cell bounding box (bbox) — into a unified task-
agnostic training objective: language modeling. Extensive quantitative and
qualitative analyses highlight UniTable’s state-of-the-art (SOTA) performance
on four of the largest TR datasets. UniTable’s table parsing capability has
surpassed both existing TR methods and general large vision-language modelss,
_e.g_., GPT-4o, GPT-4-turbo with vision, and LLaVA. Our code is publicly
available at https://github.com/poloclub/unitable, featuring a Jupyter
Notebook that includes the complete inference pipeline, fine-tuned across
multiple TR datasets, supporting all three TR tasks.

##  1 Introduction

Tables are ubiquitous in documents, as they serve to summarize factual and
quantitative data — information that is cumbersome to describe in text but
nevertheless crucial [10, 8]. Due to the implicit conventions used by humans
in creating tables, the representations within tables are often challenging
for machines to parse. Even the milestone VLM, GPT-4o [29], GPT-4V(vision)
[39] and LLaVA [21], still struggles with various document-related tasks
including TR. GPT-4V tends to omit content in large tables and performs worse
when faced with complex tables, _e.g_., spanning or empty cells and uneven
text distributions [35].

Prior work on TR has mainly centered around complex task-specific combinations
of available inputs and tools. Typically, table structure was predicted by an
image-to-text pipeline [42] and cell bbox was predicted by a detection head,
_e.g_., Faster R-CNN [36] or DETR [28]. The assumption regarding predicting
cell content varies: some studies assume the presence of a portable document
format (PDF) accompanying the tabular image [28], while others rely on
external text line detection and text recognition models [40, 12]. However,
training generic Transformers under a single language modeling objective,
_i.e_., predicting the next token, for diverse tasks has achieved remarkable
success across diverse tasks in language [32, 3], vision [4, 5], and vision-
and-language domains [1]. We wonder whether understanding visual language in
tables, _i.e_., extracting table structure, cell content, and cell bbox from
tabular images, can be seamlessly integrated into the language modeling
training framework. This integration is challenging because: (1) the fusion of
vision and language in tabular images demands high-fidelity reading and rich
high-level representations [15]; (2) the large amount of unannotated tabular
images in practice cannot be leveraged by existing supervised learning
approaches [28, 26]; (3) the diverse output of table-related tasks are
typically addressed by task-specific models [12]; and (4) direct application
of a linear projection Transformer results in the significant performance drop
[30], leading prior work to exclusively employ a hybrid convolutional neural
network (CNN)-Transformer architecture. We resolve the above challenges by
proposing UniTable, a training framework that unifies both training paradigm
and training objective of TR and make the following major contributions (Fig.
1):

Figure 1: UniTable, a training framework that unifies both training paradigm
and training objective of TR. In UniTable, the visual encoder is self-
supervised pretrained and then finetuned along with the task decoder on
supervised datasets. UniTable unifies the training objectives of all three TR
tasks — extracting table structure, cell bbox, and cell content — into a
unified task-agnostic training objective: language modeling. With UniTable,
the user inputs a tabular image and obtains the corresponding digitalized
table in HTML.  Figure 2: The table parsing capability of UniTable has
surpassed that of general large VLMs, _e.g_., GPT-4o, GPT-4-turbo with vision,
and LLaVA. For complex tables that include multiple spanning cells, UniTable
can successfully reconstruct the table in HTML, whereas general large VLMs
fail in various aspects.

  1. 1.

UniTable’s training paradigm combines the simplicity of purely pixel-level
inputs with the effectiveness and scalability empowered by SSP from diverse
unannotated tabular images. Specifically, UniTable unifies the training
paradigm for TR: pretraining the visual encoder by predicting the masked
tabular images in a self-supervised manner and finetuning the visual encoder
along with the task decoder on supervised datasets. With UniTable, the table
structure prediction on SynthTabNet [28], a comprehensive dataset with 600k
tables across finance, marketing, and academia in both dense and sparse
format, achieves the SOTA 99.18% when self-supervised pretrained on 2M images,
significantly lifting the original accuracy of 84.04% when trained from
scratch using a linear projection Transformer. Owing to the powerful SSP,
UniTable has also successfully mitigated the performance drop caused by
replacing the CNN backbone with the linear projection.

  2. 2.

UniTable unifies the training objectives of all three TR tasks — extracting
table structure, cell content, and cell bbox — into a unified task-agnostic
training objective: language modeling. Specifically, the input to our model is
an image in the form of raw pixels only and the output is text in the form of
token sequences, and the training objective is language modeling. UniTable’s
SOTA performance on the FinTabNet dataset [41] demonstrates our approach’s
generalizability to the PDF input modality as we can simply convert PDF to
images. Our framework also enables us to leverage the power of SSP on large-
scale unannotated tabular images as all models are finetuned from SSP.
UniTable’s unified training objective applies to both linear projection
Transformer and hybrid CNN-Transformer architectures conventionally used in
TSR.

  3. 3.

Extensive quantitative and qualitative analyses highlight UniTable’s SOTA
performance on four of the largest TR datasets: ICDAR 2019 B2 Modern [8],
PubTabNet [42], FinTabNet [41], and SynthTabNet [28]. UniTable’s table parsing
capability has surpassed both existing TR methods and general large VLMs (Fig.
2), _e.g_., GPT-4o, GPT-4V(ision), and LLaVA. Due to our unified language
modeling framework formulation, we discover three types of previously
unacknowledged inconsistencies in the groundtruth annotations of PubTables-1M
[36], one of the largest TSR datasets, accounting for more than 53.10% of its
training set. Our visualization of the visual tokens reveals the key reason
for why SSP works — the visual semantics captured by the visual codebook show
a fine-grained categorization to represent the implicit human conventions used
when creating the tables.

  4. 4.

Open-source code and UniTable in practice. To promote reproducible research,
enhance transparency, SOTA innovations, and facilitate fair comparisons in our
domain as tables are a promising modality for representation learning, we
open-source our code (anonymized)111https://github.com/poloclub/unitable . We
provide all the details regarding training, validation, testing, and ablation
studies. To enable users to easily try UniTable on their own tabular images
and obtain fully digitized HTML tables, we release the first-of-its-kind
Jupyter Notebook of the whole inference pipeline, fine-tuned across multiple
TR datasets, supporting all three TR tasks.

##  2 Background

###  2.1 Task Definition

The goal of TR is to translate the input tabular image $\mathbf{I}$ into a
machine-readable sequence $\mathbf{T}$, which typically consists of table
structure $\mathbf{S}$, table cell bbox $\mathbf{B}$, and table cell content
$\mathbf{C}$. Structure $\mathbf{S}=[s_{1},...,s_{m}]$ is a sequence of
tokenized HTML table tags $s$, cell bbox
$\mathbf{B}=[\mathbf{b}_{1},...,\mathbf{b}_{n}]$ is a sequence of bboxes
defined by $\mathbf{b}=(x_{min},y_{min},x_{max},y_{max})$, and cell content
$\mathbf{C}=[c_{1},...,c_{n}]$ comprises the content within each cell in
reading order. Note that the sequence length of $\mathbf{B}$ and $\mathbf{C}$
are the same, but shorter than $\mathbf{S}$ as the HTML tags contain both
empty and non-empty cells. Since each cell $i$ is defined by a single bbox,
$c_{i}$ can have either single or multiple text lines.

###  2.2 Model Architecture

tsr model architecture has two modules: visual encoder and task decoder. The
visual encoder extracts features from input $\mathbf{I}$, and the task decoder
predicts $\mathbf{T}$. For the visual encoder, prior work employed either an
off-the-shelf CNN backbone, _e.g_., ResNets and ResNet variants [11], or
hybrid CNN-Transformer architecture. EDD [42] explored five different
ResNet-18 variants, TableMaster [24] combined residual blocks with multi-
aspect global context attention, TableFormer [28] connected a ResNet-18 with
Transformer encoder layers, and VAST [12] adopted the first four stages of a
ResNet-31. These convolution layers cannot be replaced because a direct
employment of a vanilla Transformer with linear projection leads to a
significant performance drop [30]. However, linear projection, that divides
image into patches, is a widely used input image processor in SOTA vision
Transformers [7, 22], VLMs [16, 20] and multi-modal models [27]. Recent work
on visual language understanding has also successfully adopted the linear
projection [14, 15]. Thus, to avoid having a separate architecture design
solely for the table domain, we aim to keep the linear projection Transformer,
and mitigate the performance gap by SSP. Sec. 3 shows that pretraining the
visual encoder in a self-supervised manner significantly helps the model learn
how to parse the table structure and achieves performance even higher than
architectures with convolutions.

##  3 A Unified Framework for Pretraining and Finetuning TR Models

We pretrain the visual encoder by predicting the masked tabular images in a
self-supervised manner and finetune the visual encoder along with the task
decoder using the supervised dataset for each task, as shown in Fig. 1. Sec.
3.1 introduces the SSP of the visual encoder, and Sec. 3.2 details how we
unify all TR tasks and finetune with the pretrained visual encoder.

###  3.1 Self-Supervised Pretraining of the Visual Encoder

Before pretraining, each tabular image $I$ is tokenized into discrete visual
tokens. During pretraining, $I$ is divided into patches, and a portion of the
image patches are masked so that the visual encoder predicts which visual
token is chosen to replace the masked regions.

Image tokens. Define a visual codebook in the latent space
$Z\in\mathbb{R}^{K\times D}$, representing $K$ entries of visual tokens
$z_{i}\in\mathbb{R}^{D}$. The visual codebook is trained in such a way that an
input image, once it is embedded into an image grid, each pixel on the
embedded image grid can be substituted with a visual token from the codebook.
Decoding this modified image grid will then reconstruct the input image. We
use Vector Quantized-Variational AutoEncoder (VQ-VAE) [37] to train the visual
codebook. Specifically, the tabular image $I$ is tokenized into discrete
tokens $z$ after passing through the encoder $q_{\phi}(z|I)$, and the decoder
$p_{\psi}(I|z)$ takes these discrete tokens and rebuilds the original image.
The training objective of VQ-VAE is to maximize the $\mathbb{E}_{z\sim
q_{\phi}(z|I)}[\log p_{\psi}(I|z)]$ with respect to $\phi$ and $\psi$. The
training is non-differentialble due to the categorical distribution in
selecting visual tokens. Thus, we use Gumbel-Softmax [13] as a
reparameterization trick following DALL-E [33]. We have trained the VQ-VAE on
1M and 2M tabular images, where $K=8192$ for 1M, and $K=16384$ for 2M.

Image patches. Given an input tabular image $I\in\mathbb{R}^{H\times W\times
C}$, the linear projection divides $I$ into a sequence of flattened 2D patches
$I_{p}\in\mathbb{R}^{N\times(P^{2}\cdot C)}$, where $C$ is the number of
channels, $(P,P)$ is the size of each image patch, and $N=HW/P^{2}$ is the
number of patches. It is implemented by a kernel $P\times P$, stride $P$
convolution. We set $P=16$ and $I\in\mathbb{R}^{448\times 448\times 3}$, thus
the sequence length of the image patches is $28\times 28$. Approximately 40%
of the sequence is replaced with a masked token and the pretraining objective
is to maximize the log-likelihood of the visual tokens of the masked region
given the unmasked region. The image tokenizer’s codebook provides the
groundtruth visual tokens.

The pretraining task is inspired by the success of masked language [6] and
natural image [2] modeling, but our work differs in the following ways:

(1) Table incorporates both vision and language representation of data,
presenting concise human language with implicit conventions. Models for visual
language understanding are required to read with high fidelity while also
building rich high-level representation, relying on signals from both vision
and language. In contrast, our work first explores the feasibility of self-
supervised learning only on images for visual language tasks, using tables as
an example.

(2) A domain shift from semantic-rich natural images to predominantly black
texts on white background tabular images poses a difficult optimization
problem. It appears that tabular images are mainly text and lines separating
rows and columns, so the visual codebook can quickly exhaust the table
patterns with minimal tokens. Contrarily, the training of VQ-VAE can easily
diverge. We discover that training stability is achieved by increasing the
total number of tokens or introducing tabular images with colorful
backgrounds. Sec. 5.3 visualizes the semantics of the trained visual codebook
and provides an explanation for the substantial number of tokens required.

###  3.2 Unified Finetuning Framework

Prior work employed a task-specific decoder for each task, where $\mathbf{S}$
was predicted by an image-to-text pipeline and $\mathbf{B}$ was predicted by a
detection head, _e.g_., Faster R-CNN [36] or DETR [28]. The assumption on
predicting $\mathbf{C}$ varies: some assume a PDF is always accompanying the
tabular image [28], others rely on external text line detection and text
recognition models [40, 12]. We aim to provide a unified task-agnostic
training framework, where the input to our model is an image in the form of
raw pixels only and the output is text in the form of token sequences. This
setting is also generalizable to PDF input modality as we can simply take a
screenshot of the PDF. The framework also enables us to leverage the visual
encoder pretrained on the unannotated tabular images.

Table structure $\mathbf{S}$ . Predicting the table structure $\mathbf{S}$
already fits our training framework as $\mathbf{S}$ is defined by discrete
HTML table tags. For non-spanning cells, we use 	 and []	 to
denote empty and non-empty cells. For spanning cells, 	 and >[]	 marks the ending of empty and non-empty cells. The
specific tokens for spanning cells are rowspan="n" and colspan="n". We use
$n\in[2,19]$ as that covers most of the tables in practice. Apart from the
data cell tags, the vocabulary also contains the following tags that define a
table:
,
,
, and their corresponding closing tags.

Table cell bbox $\mathbf{B}$. Each cell bbox $\mathbf{b}$ are four continuous
coordinates, which are not naturally expressed as discrete tokens. Inspired by
Pix2Seq [4], we discretize the coordinate into an integer between 0 and image
size. The two directions of an image share the same vocabulary. Since we need
to predict all bboxes within a tabular image, each quantized bbox is
concatenated together in reading order: from left to right and top to bottom.
This formulation completes the quantization and serialization of all bboxes
into a sequence of discrete tokens. At inference time, we de-serialize the
predicted sequence into groups of four tokens.

Table cell content $\mathbf{C}$. After predicting all bbox coordinates, we
only need to perform optical character recognition (OCR) on the image region
within each bbox. Note that each cell is defined by a single bbox, thus the
cell content can have single or multiple lines of text. In the training stage,
the model is trained on a mixture of single line and multi-line dataset. At
inference time, we parse all text simultaneously as each cell bbox is
independent. Finally, we insert the cell content back into the non-empty cells
[]	 or >[]	 as the reading order is already preserved in both S
and B. We use WordPiece tokenizer [38] with charater-level granularity since
OCR requires the model to read instead of understanding the semantics, which
significantly reduces the total vocabulary size to less than 6k.

Up till now, we have completely digitalized a tabular image into HTML with a
unified image-to-text framework. Note that all visual encoders are initialized
from SSP. For cell content and cell bbox, there is an alternative solution
that we first generate all the cell content within a table, and then predict
the cell bbox via prompting the model with cell content. However, we find it
hard for the model to predict all the cell content first as all tabular images
are rescaled to a fixed size during augmentation, and such rescaling leads to
texts in various aspect ratios. Thus, we do not use this solution and instead
generate all cell bbox first.

Training Objective. Since all task outputs have been formulated into a
sequence of discrete tokens, the objective function is simply the maximum
likelihood of tokens conditioned on pixel inputs and the preceding tokens.
Denote the probability of the $i$th step prediction
$p(t_{i}|I,t_{1:i-1};\theta)$, we directly maximize the correct structure
prediction by using the following formulation:

$\theta^{\ast}=\operatorname*{arg\ max}_{\theta}\sum_{(I,T)}\log
p(T|I;\theta),$ (1)

where $\theta$ are model parameters.

Table 1: UniTable outperforms prior methods and achieves SOTA on four out of five largest publicly available TR datasets across all available tasks. Our method is trained with a task-agnostic language modeling loss and does not rely on external PDF for text extraction and bbox post-processing.  | IC19B2M | PubTabNet | FinTabNet | SynthTabNet | PubTables-1M
---|---|---|---|---|---
| IoU 0.6 | WAvg. F1 | $\text{AP}_{50}$ | S-TEDS | TEDS | S-TEDS | $\text{AP}_{50}$ | S-TEDS | $\text{AP}_{50}$ | $\text{AP}_{75}$
SOTA | GTE | GTE | VAST | VAST | VAST | VAST | TableFormer | DRCC | DETR | DETR
38.50 | 24.80 | 94.50 | 97.23 | 96.31 | 98.63 | 87.70 | 98.70 | 97.10 | 94.80
UniTable |  |  |  |  |  |  |  |
Base | 54.97 | 40.15 | 97.94 | 95.63 | 94.78 | 97.19 | 98.99 | 98.97 | 94.48 | 88.64
Large | 58.10 | 42.62 | 98.43 | 97.89 | 96.50 | 98.89 | 99.00 | 99.39 | 95.68 | 93.28

##  4 Experiments

###  4.1 Implementation

Architecture. We have trained two model variants: (1) a base model with 30M
parameters including 4 encoder layers, 8 attention heads with a hidden size of
512, (2) a large model with 125M parameters including 12 encoder layers, 12
attention heads with a hidden size of 768. Both base and large models have a
task decoder of 4 decoder layers. The maximum token sequence length is 512 for
table structure, 1024 for cell bbox, and 200 for cell content, as we find such
settings satisfy most tables.

Training and inference We have pretrained the VQ-VAE on 1M and 2M tabular
images. The 1M VQ-VAE is trained on PubTabNet [42] and SynthTabNet [28], and
the extra 1M datasets for training 2M VQ-VAE are PubTables-1M [36] and
TableBank [18]. In Sec. 4.3, we present the finetuning results of 2M VQ-VAE
for comparing with SOTA methods. In Sec. 5.1, we ablate the effectiveness and
scalability of SSP on both 1M and 2M VQ-VAE. All models are trained with the
AdamW optimizer [23]. We employ a cosine learning rate scheduler with a linear
warmup. All models are trained for 24 epochs for a fair comparison. We apply
teacher forcing during training and employ greedy decoding at inference time.

###  4.2 Evaluation Metrics

car was first proposed in ICDAR2013 competition [10] and improved by ICDAR2019
competition [8]. It aligns the the predicted bbox with the groundtruth bbox
for each table cell based on intersection over union (IoU) and generates a
list of adjacency relations between a non-empty cell and its nearest
horizontal and vertical neighbors. The precision, recall, and F1 score are
computed based on this converted 1-D adjacency relation list.

COCO average prevision (AP) [19] is a widely used metric for generic object
detection, which has been reported in other work for evaluating table cell
detection. We use the COCO evaluation
toolkit222https://github.com/cocodataset/cocoapi and report mean  (mAP),
$\text{AP}_{50}$, and $\text{AP}_{75}$.

teds was created by PubTabNet [42] to robustify the cell adjacency relations
(CAR) metric against cell shift perturbation and cell content perturbation.
teds converts the table HTML code into a tree structure and measures the edit
distance between the prediction $T_{pred}$ and the groundtruth $T_{gt}$. A
shorter edit distance indicates a higher degree of similarity, leading to a
higher tree-edit-distance-based similarity (TEDS) score. teds measures both
the table structure and table cell content, and we use S-TEDS [12] when only
the table structure is considered.

###  4.3 Results on Datasets

We evaluate UniTable on five of the largest publicly available TR datasets as
shown in Table 1. Comparing UniTable with the prior SOTA on each dataset
across all available tasks, we achieve new SOTA on four out of the five
datasets even without training with task-specific loss [12] or relying on
external PDF for text extraction and bbox post-processing [28]. Below is an
introduction of each dataset and comparisons with previous methods.

ICDAR 2019 B2 Modern (IC19B2M) [8] was originated from ICDAR 2019 table
competition. The dataset has two subsets for TR, archival and modern, and only
the modern subset has table cell content bbox annotations. The competition
computes CAR F1 score at $\text{\acs{iou}}\in[0.6,0.7,0.8,0.9]$ and ranks
method by weighted average F1 (WAvg. F1):

$\text{\acs{wf1}}=\frac{\sum_{i=1}^{4}\text{\acs{iou}}_{i}\times\text{F1@\acs{iou}}_{i}}{\sum_{i=1}^{4}\text{\acs{iou}}_{i}}$
(2)

We evalute on all 100 test tables from the modern subset and report both WAvg.
F1 and IoU 0.6 as in other work. Our method significantly improves the
previous SOTA GTE [41] by a large margin.

PubTabNet [42] contains 509k images of heterogeneous tables extracted from the
medical scientific articles. It is the first large-scale TR dataset that
provides annotations (in HTML format) of table cell bbox, table structure, and
table cell content. PubTabNet measures the table cell bbox by COCO
$\text{AP}_{50}$, table structure by S-TEDS, and full table including both
structure and cell content by TEDS. The authors of PubTabNet also developed
the EDD model, which consisted of a CNN encoder and dual recurrent neural
network (RNN) decoders for predicting table structure and cell content,
respectively. TableFormer [28] improved EDD by replacing the cell content
decoder with a cell bbox decoder and extracted all contents from the PDF
corresponding to the tabular image. VAST [12] added an auxiliary visual-
alignment loss while training the cell bbox decoder and achieved previous SOTA
on all three metrics. Our unified training framework with SSP achieves the new
SOTA on all tasks even without leveraging any external PDF as PDF
corresponding to the tabular image may not always exist. Specifically, both
UniTable-base and UniTable-large outperform VAST on $\text{AP}_{50}$ by more
than 3 percentage points (pp), which confirms the effectiveness of converting
the cell bbox detection to language modeling.

FinTabNet [41] is a dataset containing 113k tables from the annual reports of
the S&P 500 companies in PDF format. The major challenge of this dataset is
that financial tables largely differ from scientific and government document
tables in that the former has fewer graphical lines, larger gaps within each
table, and more color variations. Thus, FinTabNet mainly evaluates table
structure prediction accuracy, _i.e_., S-TEDS. VAST trained the model with an
auxiliary supervised signal and achieved the previous SOTA. We achieve the new
SOTA by leveraging SSP and finetuning without a task-specific loss objective.
The performance gain from base to large shows that our method scales with
model parameters.

SynthTabNet [28] is a large-scale synthetically generated dataset that offers
control over 1) dataset size, 2) table structure, 3) table style, and 4)
content type. The dataset aims to overcome the limitations of PubTabNet and
FinTabNet, which are skewed table distributions towards simpler tables,
limited variance in appearance styles, and restricted cell content domains.
Thus, SynthTabNet is organized into 4 subsets of 150k tables (600k in total),
namely Finance, PubTabNet, Marketing, and Sparse. The first two mimic the
appearance of FinTabNet and PubTabNet but encompass more complex table
structures. Marketing adopts a colorful appearance with high contrast that
resembles real-world marketing tables, and Sparse contains tables with sparse
content. The authors of SynthTabNet propose TableFormer as a baseline.
TableFormer predicts table structure and cell bbox and relies on external PDF
to extract cell content, so $\text{AP}_{50}$ and S-TEDS are the evaluation
metrics. Both our base and large models outperform previous SOTA on
$\text{AP}_{50}$ and S-TEDS. Since SynthTabNet is a comprehensive dataset that
can thoroughly evaluate the model under different table configurations, we use
it for ablations and present results on four subsets separately in Sec. 5.

PubTables-1M [36] aims to overcome the groundtruth inconsistency observed in
prior datasets using a new canonicalization procedure. The dataset has 947k
tables annotated with bbox and text within each bbox. The dataset differs from
previous datasets that the bbox is word-wise instead of cell-wise, thus each
cell can have more than one bbox. Besides, since all annotations are in bbox
format and no table structure labels, _e.g_., HTML, are provided, the baseline
DETR trained by the dataset creators report their performance in detection
metrics, _e.g_., $\text{AP}_{50}$ and $\text{AP}_{75}$. UniTable also achieves
competitive results compared with the baseline DETR. Visualizing the bbox
predictions shows that our model predicts more bboxes (longer sequence) than
the groundtruth, motivating us to delve deep into the dataset annotations and
discover previously unacknowledged inconsistencies in table annotations. We
conjecture the SOTA performance of DETR model on PubTables-1M may be due to
overfitting to the training set. Sec. F.1 describes the details of these
dataset annotation issues.

###  4.4 Using UniTable in Practice

We have also finetuned our UniTable-large across multiple datasets and
released a Jupyter Notebook as a demo of our inference pipeline. A user can
simply pass a table screenshot through our notebook and obtain a fully
digitalized HTML table. The table structure is trained across PubTabNet,
FinTabNet, and SynthTabNet, the cell bbox is trained across PubTabNet and
SynthTabNet, and the cell content is trained across PubTabNet, SynthTabNet,
and PubTables-1M. We have provided a public API hosted on HuggingFace to
facilitate the access to our UniTable: https://poloclub.github.io/magic-
table/. Appendix B visualizes several table examples in practice.

##  5 Deeper Dive into UniTable: Ablation and Analysis

###  5.1 Effectiveness and Scalability of SSP

Table 2: Effectiveness and scalability of SSP on all four subsets of SynthTabNet. Without SSP, the model performance suffers, and increasing the model complexity from base to large barely improves the performance. Pretraining the visual encoder on 1M tabular images provides an average increase of 14.40 pp. Pretraining on 2M images continues to increase the performance by 0.74 pp. Here we present S-TEDS of the table structure prediction, and the same trend also applies to other tasks as elaborated in Appendix C.  | Finance | PubTabNet | Marketing | Sparse
---|---|---|---|---
| Base | Large | Base | Large | Base | Large | Base | Large
No SSP | 88.95 | 90.75 | 89.10 | 91.67 | 68.05 | 70.60 | 85.50 | 87.72
SSP 1M | 98.73 | 99.56 | 99.02 | 99.55 | 95.14 | 99.05 | 97.20 | 99.29
SSP 2M | 99.41 | 99.58 | 99.44 | 99.56 | 98.35 | 99.08 | 98.69 | 99.34

We ablate the effectiveness and scalability of SSP on all four subsets of
SynthTabNet since SynthTabNet is a large-scale dataset that can
comprehensively evaluate the model under different table configurations. Table
2 presents the table structure task, and the same trend also applies to other
tasks as elaborated in Appendix C. Comparing row “No SSP” and “SSP 1M” or “SSP
2M”, both the base and the large models have benefited significantly from the
SSP. Specifically, Marketing and Sparse are two of the most challenging
subsets. Marketing has a colorful background with high-contrast texts, and
Sparse contains tables with sparse content. These variations make it
challenging to accurately predict the HTML table structure tags and bbox
surrounding the cell content. Without SSP, the model performance suffers, and
increasing the model complexity from base to large barely improves the
performance. Instead, after pretraining the visual encoder on 1M tabular
images, both the base and the large models have an average increase of 14.40
pp. Note the performance also scales along with the pretraining dataset size.
When the pretraining dataset increases from 1M to 2M, the average performance
continues to increase by 0.74 pp. Finally, all large models have an average
performance gain of 1.51 pp over the base models.

###  5.2 Generalization of the Unified Training Objective

Table 3: UniTable’s unified training objective applies to both linear projection Transformer and hybrid CNN-Transformer architectures conventionally used in TR. Results on all four subsets of the SynthTabNet for table structure prediction evaluated with S-TEDS, and the same conclusion also applies to other tasks as elaborated in Appendix D.  Model | Finance | PubTabNet | Marketing | Sparse
---|---|---|---|---
Base | 98.63 | 98.80 | 97.16 | 95.30
Large | 99.44 | 99.44 | 98.71 | 98.64

We showcase that our unified training objective, language modeling, not only
works on models with linear projection Transformer, but also works with the
hybrid CNN-Transformer architecture used in the TR literature. Table 3
presents the results on all four subsets of the SynthTabNet similar to the
settings in Sec. 5.1. The main difference is that the linear projection in the
visual encoder is replaced by a ResNet-18 CNN backbone. Such a modification
leads to an increase of 12M parameters. The performance of this hybrid CNN-
Transformer architecture is roughly on par with the linear projection model
initialized from the SSP 1M tabular images, which shows that our training
objective is agnostic to the choice of architecture. The significant
performance gap between hybrid CNN-Transformer and linear projection
Transformer trained from scratch verifies the observation identified in the
previous literature [30]. Though hybrid CNN-Transformer has also achieved
competitive results, we still recommend using the linear projection
Transformer because of 1) capability of leveraging the power of SSP, 2)
architectural compliance with VLM in natural image domain, and 3) the
performance of hybrid CNN-Transformer is still worse than the SSP on 2M
tabular images even with more total parameters. Though prior work has started
to propose SSP for hybrid CNN-Transformer, this direction is still in its
early stage as it is challenging in heavy computational cost and pretraining-
finetuning discrepancy [9]. Moreover, most work only demonstrates the
capability in discriminative tasks rather than the generative tasks used in
the TR domain. Besides, existing SOTA VLMs, _e.g_., BLIP-2 [17] and LLaVA
[20], still employ the linear projection and leave the Transformer to learn
the interactions between different patches. As we believe the compliance of
architecture is an important step towards generic VLMs, our success in
replacing the CNN backbone with linear projection and leveraging a task-
agnostic language modeling loss is a cornerstone of incorporating TR in modern
VLM training.

###  5.3 Why does SSP work?

((a)) Background patches (red)

((b)) Patches for separations

Figure 3:  The key reason that SSP works is because each tokens have visual
semantics and the codebook shows a fine-grained categorization to represent
the implicit conventions in the table. The codebook used in SSP has learned to
represent abstract concepts by using different groups of tokens to represent
different concepts: (a) empty background and (b) separations within a table.
Red highlights the token indices under investigation. Appendix E provides a
zoomed-in version of these images labeled with token indices. We lay the color
patches over the original tabular image to present the selected token indices
from the 2M VQ-VAE.

During SSP, the visual encoder is trained to fill in the masked tabular image
by selecting visual tokens from the VQ-VAE codebook. Thus, we visualize the
selected token indices and analyze whether the tokens have visual semantics.
Fig. 3 presents the original table overlaid by the selected token indices from
the 2M VQ-VAE codebook. We highlight the token indices under investigation in
red. Appendix E provides a zoomed-in version of these images labeled with
token indices. The input image size is $448\times 448$ and the patch size is
$16\times 16$, so each image has $28\times 28$ tokens chosen from the 16384
entries in the codebook. First, we highlight the tokens representing the blank
background (Fig 3a), which covers most of the empty space in the table;
leaving only texts and separation lines nonhighlited. Then, we highlight the
tokens representing the separation lines (Fig 3b), showing that they well-
separate the header row, and the shaded and non-shaded rows. Taking a closer
look at the token indices in Fig. 5 and 6 in Appendix E, we find the codebook
has learned to represent an abstract concept by assigning multiple tokens,
where each reflects a different scenario. For example, token “111” represents
the separation above the gray color shading and token “964” represents the one
below, and token “15282” represents the separation above a bold horizontal
line and “10807” represents the one below. A certain type of separation also
has multiple choices depending on the portion of the line within the patch,
_e.g_., token “14181”, “8714”, and “10807” all representing the “below a bold
horizontal line”, but the differences lie in the line thickness and amount of
black inside the image patch. Such a fine-grained categorization also explains
why the codebook needs so many tokens to represent the implicit conventions
created by humans in the table.

##  6 Conclusion

We present UniTable, a training framework that unifies both the training
paradigm and training objective of TR. Its training paradigm combines the
simplicity of purely pixel-level inputs with the effectiveness and scalability
empowered by SSP from diverse unannotated tabular images. Our framework
unifies the training objectives of all three TR tasks, extracting table
structure, cell content, and cell bbox, into a unified task-agnostic training
objective: language modeling. Extensive quantitative and qualitative analyses
highlights UniTable’s SOTA performance on four of the largest TR datasets To
promote reproducible research, enhance transparency, and SOTA innovations, we
open-source our code and release the first-of-its-kind Jupyter Notebook of the
whole inference pipeline, fine-tuned across multiple TR datasets, supporting
all three TR tasks.

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Bao et al. [2021] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ , 2021.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chen et al. [2021] Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton.  Pix2seq: A language modeling framework for object detection.  _arXiv preprint arXiv:2109.10852_ , 2021.
  * Chen et al. [2022] Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey E Hinton.  A unified sequence interface for vision tasks.  _Advances in Neural Information Processing Systems_ , 35:31333–31346, 2022.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Gao et al. [2019] Liangcai Gao, Yilun Huang, Hervé Déjean, Jean-Luc Meunier, Qinqin Yan, Yu Fang, Florian Kleber, and Eva Lang.  Icdar 2019 competition on table detection and recognition (ctdar).  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1510–1515. IEEE, 2019.
  * Gao et al. [2022] Peng Gao, Teli Ma, Hongsheng Li, Ziyi Lin, Jifeng Dai, and Yu Qiao.  Convmae: Masked convolution meets masked autoencoders.  _arXiv preprint arXiv:2205.03892_ , 2022.
  * Göbel et al. [2013] Max Göbel, Tamir Hassan, Ermelinda Oro, and Giorgio Orsi.  Icdar 2013 table competition.  In _2013 12th International Conference on Document Analysis and Recognition_ , pages 1449–1453. IEEE, 2013.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Huang et al. [2023] Yongshuai Huang, Ning Lu, Dapeng Chen, Yibo Li, Zecheng Xie, Shenggao Zhu, Liangcai Gao, and Wei Peng.  Improving table structure recognition with visual-alignment sequential coordinate modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11134–11143, 2023.
  * Jang et al. [2016] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Kim et al. [2022] Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park.  Ocr-free document understanding transformer.  In _European Conference on Computer Vision_ , pages 498–517. Springer, 2022.
  * Lee et al. [2023] Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova.  Pix2struct: Screenshot parsing as pretraining for visual language understanding.  In _International Conference on Machine Learning_ , pages 18893–18912. PMLR, 2023.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Li et al. [2020] Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, and Zhoujun Li.  Tablebank: Table benchmark for image-based table detection and recognition.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 1918–1925, 2020.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023.
  * Liu et al. [2024] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu et al. [2021] Ning Lu, Wenwen Yu, Xianbiao Qi, Yihao Chen, Ping Gong, Rong Xiao, and Xiang Bai.  Master: Multi-aspect non-local network for scene text recognition.  _Pattern Recognition_ , 117:107980, 2021.
  * Lysak et al. [2023] Maksym Lysak, Ahmed Nassar, Nikolaos Livathinos, Christoph Auer, and Peter Staar.  Optimized table tokenization for table structure recognition.  _arXiv preprint arXiv:2305.03393_ , 2023.
  * Ma et al. [2023] Chixiang Ma, Weihong Lin, Lei Sun, and Qiang Huo.  Robust table detection and structure recognition from heterogeneous document images.  _Pattern Recognition_ , 133:109006, 2023.
  * Mizrahi et al. [2023] David Mizrahi, Roman Bachmann, Oğuzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir.  4m: Massively multimodal masked modeling.  _arXiv preprint arXiv:2312.06647_ , 2023.
  * Nassar et al. [2022] Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Peter Staar.  Tableformer: Table structure understanding with transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4614–4623, 2022.
  * OpenAI [2024] OpenAI.  Hello gpt-4o, 2024.  URL https://openai.com/index/hello-gpt-4o/.
  * Peng et al. [2023] Anthony Peng, Seongmin Lee, Xiaojing Wang, Rajarajeswari Raji Balasubramaniyan, and Duen Horng Chau.  High-performance transformers for table structure recognition need early convolutions.  In _NeurIPS 2023 Second Table Representation Learning Workshop_ , 2023.
  * Prasad et al. [2020] Devashish Prasad, Ayan Gadpal, Kshitij Kapadni, Manish Visave, and Kavita Sultanpure.  Cascadetabnet: An approach for end to end table detection and structure recognition from image-based documents.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pages 572–573, 2020.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Shen et al. [2023] Huawen Shen, Xiang Gao, Jin Wei, Liang Qiao, Yu Zhou, Qiang Li, and Zhanzhan Cheng.  Divide rows and conquer cells: Towards structure recognition for large tables.  In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence_ , pages 1369–1377, 2023.
  * Shi et al. [2023] Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, and Lianwen Jin.  Exploring ocr capabilities of gpt-4v (ision): A quantitative and in-depth evaluation.  _arXiv preprint arXiv:2310.16809_ , 2023.
  * Smock et al. [2022] Brandon Smock, Rohith Pesala, and Robin Abraham.  Pubtables-1m: Towards comprehensive table extraction from unstructured documents.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4634–4642, 2022.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Wu et al. [2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_ , 2016.
  * Yang et al. [2023] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang.  The dawn of lmms: Preliminary explorations with gpt-4v (ision).  _arXiv preprint arXiv:2309.17421_ , 9(1):1, 2023.
  * Ye et al. [2021] Jiaquan Ye, Xianbiao Qi, Yelin He, Yihao Chen, Dengyi Gu, Peng Gao, and Rong Xiao.  Pingan-vcgroup’s solution for icdar 2021 competition on scientific literature parsing task b: table recognition to html.  _arXiv preprint arXiv:2105.01848_ , 2021.
  * Zheng et al. [2021] Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang.  Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 697–706, 2021.
  * Zhong et al. [2020] Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes.  Image-based table recognition: data, model, and evaluation.  In _European conference on computer vision_ , pages 564–580. Springer, 2020.

##  Appendix A Detail comparisons with milestone VLMs

Table 4: UniTable outperforms GPT-4V and LLaVA-v1.6 by a huge margin for table recognition. Model | Finance | PubTabNet | Marketing | Sparse
---|---|---|---|---
Samples | 100 | 300 | 500 | 100 | 300 | 500 | 100 | 300 | 500 | 100 | 300 | 500
LLaVA-v1.6-vicuna-7b | 37.72 | 35.57 | 34.16 | 41.66 | 42.19 | 41.80 | 36.54 | 34.34 | 34.14 | 36.95 | 37.42 | 38.49
LLaVA-v1.6-mistral-7b | 38.40 | 39.76 | 38.07 | 39.16 | 37.79 | 38.96 | 34.16 | 33.20 | 32.25 | 40.28 | 35.97 | 37.06
LLaVA-v1.6-vicuna-13b | 43.50 | 45.13 | 43.95 | 48.17 | 49.05 | 49.68 | 41.37 | 42.00 | 41.40 | 46.38 | 46.04 | 47.21
LLaVA-v1.6-34b | 40.86 | 42.96 | 40.69 | 46.30 | 47.99 | 49.30 | 35.08 | 35.75 | 33.99 | 39.13 | 39.60 | 39.85
GPT-4-1106-vision-pre. | 64.05 | 66.93 | 66.13 | 69.26 | 69.86 | 69.62 | 58.51 | 60.03 | 59.13 | 55.64 | 57.02 | 56.99
UniTable Large | 99.63 | 99.56 | 99.57 | 99.54 | 99.54 | 99.55 | 99.11 | 99.10 | 99.10 | 99.34 | 99.35 | 99.34

Table 4 shows detailed comparisons (S-TEDS score) with GPT-4V and the latest
LLaVA v1.6. In summary, UniTable outperforms GPT-4V and LLaVA-v1.6 by a huge
margin for table recognition.

Dataset: SynthTabNet [28], is a large-scale table benchmark that offers
control over dataset size, table structure, style, and content type. It has 4
subsets: Finance, PubTabNet, Marketing, Sparse. We incrementally evaluate more
samples up to 500 on each subset because no further notable changes in the
results are observed.

Models: We compare UniTable with GPT-4V and LLaVA. For GPT-4V, we use the
gpt-4-1106-vision-preview. For LLaVA, we test on all 4 variants of its latest
version, v1.6, provided by the official model weights on HuggingFace [21]:
vicuna-7b, mistral-7b, vicuna-13b, 34b. We use the same prompt as in the
latest work that evaluates GPT-4V’s OCR capabilities by Shi _et al_., [35]:
“Please read the table in this image and return an HTML-style reconstructed
table in text, do not omit anything.” The max generated sequence length is set
to 1500 to cover the largest table in the test.

##  Appendix B Using UniTable in Practice

A user can simply pass a table screenshot through our notebook and obtain a
fully digitalized HTML table. We visualize different types of table cell bbox
detection results of UniTable in Fig. 4, and an example of a digitalized HTML
table can be found in https://anonymous.4open.science/r/icml-
review/notebooks/full_pipeline.ipynb.

((a)) Table cell bbox detections results on Table 1 in the main paper.

((b)) Table cell bbox detections results on complex academic tables

((c)) Table cell bbox detections results on tables with colorful backgrounds
and spanning headers.

Figure 4:  Table cell bbox detection results of UniTable on unannotated tables
in practice.

##  Appendix C Effectiveness and Scalability of SSP

Table 10 demonstrates the effectiveness and scalability of SSP on all four
subsets of SynthTabNet. Without SSP, the model performance suffers, and
increasing the model complexity from base to large barely improves the
performance. Pretraining the visual encoder on 1M tabular images provides a
significant improvement, and pretraining on 2M images continues to increase
the performance on all TR tasks.

##  Appendix D Generalization of the Unified Training Objective

UniTable’s unified training objective applies to both linear projection
Transformer and hybrid CNN-Transformer architectures conventionally used in
TR. Table 11 shows results on all four subsets of the SynthTabNet for table
structure prediction evaluated with S-TEDS and table cell bbox detection
evaluated with mAP.

##  Appendix E Visualization of the Visual Codebook

Fig. 5 and 6 present the original table overlaid by the selected token indices
from the 2M VQ-VAE codebook. We highlight the token indices under
investigation in red and use blue for the others. The input image size is
$448\times 448$ and the patch size is $16\times 16$, so each image has
$28\times 28$ tokens chosen from the 16384 entries in the codebook. First, we
highlight the tokens representing the blank white background. Comparing the
red and blue indices, we observe that the red region has covered most of the
blank space inside the table, and the blue region has formulated a convex hull
tightly surrounding either the texts or separation lines. Next, we highlight
the tokens representing the separation lines. Comparing the red and blue
regions, we observe that the red regions align with all the separations within
the table, _e.g_., line separations and color shading separations. Taking a
closer look at the indices, we find the codebook has learned to represent an
abstract concept by assigning multiple tokens, where each reflects a different
scenario. For example, token “111” represents the separation above the gray
color shading and token “964” represents the one below, and token “15282”
represents the separation above a bold horizontal line and “10807” represents
the one below. A certain type of separation also has multiple choices
depending on the portion of the line within the patch, _e.g_., token “14181”,
“8714”, and “10807” all representing the “below a bold horizontal line”, but
the differences lie in the line thickness and amount of black inside the image
patch. Such a fine-grained categorization also explains why the codebook needs
so many tokens to represent the implicit conventions created by humans in the
table.

Figure 5: A zoomed-in version of the token indices from the 2M VQ-VAE. The
codebook used in SSP has learned to represent abstract concepts by using
different groups of tokens to represent empty white backgrounds within a
table. Red highlights the token indices representing the concept of “empty
white backgrounds”.

Figure 6: A zoomed-in version of the token indices from the 2M VQ-VAE. The
codebook used in SSP has learned to represent abstract concepts by using
different groups of tokens to represent separations within a table. Red
highlights the token indices representing the concept of “separations”.

##  Appendix F Discussions on TSR Dataset Annotations

Word-wise vs. cell-wise bbox annotations. Fig. 7(a) is the word-wise
annotation from table “PMC1574335 table 0” in PubTables-1M and Fig. 7(b) is
the cell-wise annotation from table “PMC5897438 004 00” in PubTabNet. In cell-
wise annotation, each cell has a unique bbox that can be matched to the non-
empty cells in table structure HTML tags. However, in word-wise annotation, a
bbox is matched to a single word, which cannot be easily combined with the
table structure as in cell-wise annotation, which limits its general
applicability.

((a)) Word-wise bbox from PubTables-1M

((b)) Cell-wise bbox from PubTabNet

Figure 7:  Word-wise vs. cell-wise bbox annotations: (a) is the word-wise
annotation from table “PMC1574335 table 0” in PubTables-1M and (b) is the
cell-wise annotation from table “PMC5897438 004 00” in PubTabNet. In cell-wise
annotation, each cell has a unique bbox that can be matched to the non-empty
cells in table structure HTML tags. However, in word-wise annotation, a bbox
is matched to a single word, which cannot be easily combined with the table
structure as in cell-wise annotation, which limits its general applicability.

###  F.1 Discussions on TSR Dataset Annotations

((a)) Unrelated text around table

((b)) Overlapping bbox

Figure 8: Our UniTable has successfully identified inconsistent table cell
bbox annotations in PubTables-1M, which can guide future data annotation
pipelines and qualities.

Our goal is to push forward the development of the document and table
understanding, thus we hope to see high-quality and consistent data
annotations. Because of the unified language modeling framework formulation,
we discover previously unacknowledged inconsistency in one of the largest TR
datasets, PubTables-1M.

Here, we have identified three types of inconsistent table cell bbox
annotations: (1) Fig. 8(a) (“PMC4964067 table 7”) shows bbox annotations for
unrelated text around table; (2) Fig. 8(b) (“PMC6202420 table 1”) shows
overlapping bbox annotations; (3) Bbox annotations exceed the boundary of
image size, _e.g_., [-4.6, 278.6, 19.5, 292.4] from “PMC4802837 table 1”. With
a rapid filter comparing the image size with the bbox, we find that a
surprising 53.10% (402914 over 758849) of the table annotations in the
training set have at least one bbox going beyond the image size. We hope that
these inconsistencies discovered by UniTable can help guide future data
annotation pipelines and qualities.

##  Appendix G SOTA results

Table 5: Comparisons on IC19B2M Model | IoU 0.6 | WAvg. F1
---|---|---
NLPR-PAL [8] | 30.50 | 20.60
CascadeTabNet [31] | 35.40 | 23.20
GTE [41] | 38.50 | 24.80
UniTable Base (Ours) | 54.97 | 40.15
UniTable Large (Ours) | 58.10 | 42.62
Table 6: Comparisons on PubTabNet Model | S-TEDS | TEDS | $\text{AP}_{50}$ | $\text{AP}_{75}$
---|---|---|---|---
EDD [42] | 89.90 | 88.30 | 79.20 | -
GTE [41] | 93.01 | - | - | -
TableFormer [28] | 96.75 | 93.60 | 82.10 | -
TableMaster [40] | 96.04 | 96.16 | - | -
OTSL [25] | 95.50 | - | - | 88.00
VAST [12] | 97.23 | 96.31 | 94.80 | -
UniTable Base (Ours) | 95.63 | 94.78 | 97.94 | 87.27
UniTable Large (Ours) | 97.89 | 96.50 | 98.43 | 92.44
Table 7: Comparisons on SynthTabNet Model | S-TEDS | $\text{AP}_{50}$ | $\text{AP}_{75}$
---|---|---|---
TableFormer [28] | 96.70 | 87.70 | -
DRCC [34] | 98.70 | - | -
UniTable Base (Ours) | 98.97 | 98.99 | 98.79
UniTable Large (Ours) | 99.39 | 99.00 | 98.87
Table 8: Comparisons on FinTabNet Model | S-TEDS
---|---
GTE [41] | 91.02
EDD [42] | 90.60
OTSL [25] | 95.90
TableFormer [28] | 96.80
VAST [12] | 98.63
UniTable Base (Ours) | 97.19
UniTable Large (Ours) | 98.89
Table 9: Comparisons on PubTables-1M Model | $\text{AP}_{50}$ | $\text{AP}_{75}$
---|---|---
Faster R-CNN [36] | 81.50 | 78.50
OTSL [25] | - | 89.60
DETR [36] | 97.10 | 94.80
UniTable Base (Ours) | 94.48 | 88.64
UniTable Large (Ours) | 95.68 | 93.28

Table 5, 6, 7, 8, and 9 show detailed comparisons across five of the largest
table benchmarks from our paper. In summary, UniTable outperforms prior
methods and achieves SOTA on four out of the five largest table datasets. (“-”
means not reported by the compared methods.)

Table 10: Effectiveness and scalability of SSP on all four subsets of SynthTabNet. Without SSP, the model performance suffers, and increasing the model complexity from base to large barely improves the performance. Here we present both S-TEDS of the table structure prediction and mAP of the cell bbox detection  | Finance | PubTabNet | Marketing | Sparse
---|---|---|---|---
| Base | Large | Base | Large | Base | Large | Base | Large
Table structure - S-TEDS |  |  |  |
No SSP | 88.95 | 90.75 | 89.10 | 91.67 | 68.05 | 70.60 | 85.50 | 87.72
SSP 1M | 98.73 | 99.56 | 99.02 | 99.55 | 95.14 | 99.05 | 97.20 | 99.29
SSP 2M | 99.41 | 99.58 | 99.44 | 99.56 | 98.35 | 99.08 | 98.69 | 99.34
Table cell bbox \- COCO mAP |  |  |  |
No SSP | 83.30 | 85.46 | 87.67 | 88.88 | 72.00 | 75.21 | 88.84 | 90.01
SSP 1M | 96.13 | 97.07 | 95.93 | 96.95 | 94.63 | 95.91 | 97.07 | 97.84
SSP 2M | 96.54 | 97.37 | 96.50 | 97.44 | 95.21 | 96.25 | 97.46 | 97.96
Table 11: UniTable’s unified training objective applies to both linear projection Transformer and hybrid CNN-Transformer architectures conventionally used in TR. Results on all four subsets of the SynthTabNet for table structure prediction evaluated with S-TEDS and cell bbox detection with mAP. Model | Finance | PubTabNet | Marketing | Sparse
---|---|---|---|---
Table structure - S-TEDS |
Base | 98.63 | 98.80 | 97.16 | 95.30
Large | 99.44 | 99.44 | 98.71 | 98.64
Table cell bbox \- COCO mAP |
Base | 94.61 | 94.39 | 91.42 | 95.38
Large | 95.90 | 96.50 | 94.96 | 97.55
  *[TR]: table recognition
  *[SSP]: self-supervised pretraining
  *[bbox]: bounding box
  *[SOTA]: state-of-the-art
  *[VLM]: vision-language models
  *[PDF]: portable document format
  *[CNN]: convolutional neural network
  *[VLMs]: vision-language models
  *[VQ-VAE]: Vector Quantized-Variational AutoEncoder
  *[OCR]: optical character recognition
  *[IC19B2M]: ICDAR 2019 B2 Modern
  *[IoU]: intersection over union
  *[AP]: average prevision
  *[]:
  *[mAP]: mean
  *[CAR]: cell adjacency relations
  *[TEDS]: tree-edit-distance-based similarity
  *[WAvg. F1]: weighted average F1
  *[RNN]: recurrent neural network
  *[pp]: percentage points


XX XX 8 Mon

# A Secure Future for Open-Source Computational Science and Engineering

Reed Milewicz  Jeffrey Carver  Samuel Grayson  Travis Atkison

(2022; July 2022)

††journal: Computing in Science & Engineering

Journalists, public policy analysts, and economists have called attention to
the growing importance that high-performance and scientific computing have to
national security and industrial leadership. As computing continues to power
scientific advances in virtually every discipline, so too does it improve our
economic productivity and quality of life. In 2020, a US government-funded
study by Hyperion Research estimated that every dollar invested into HPC
resulted in an average of $507 in new revenue and $47 in profit or cost-
savings [1].

The increasing social, political, and economic importance of research software
has also brought the question of software security to the fore. Just as
unintentional software errors can threaten the integrity of scientific
studies, malicious actors could leverage vulnerabilities to alter results,
exfiltrate data, and sabotage computing resources. A 2015 US Department of
Energy workshop report on cybersecurity in scientific computing vividly
captured these concerns, arguing that attacks on scientific computing could
inject flaws into critical infrastructure, inhibit public policy decision-
making, and weaken US economic competitiveness [2]. With concerns that the
world is returning to an era of great power competition, these threats could
ultimately harm the ability of researchers to collaborate openly, curtailing
the free flow of ideas that is key to innovation. Likewise, a climate of
uncertainty around scientific computing could amplify public distrust in
scientific institutions.

For these reasons, research has been proceeding to develop solutions to
detect, mitigate, and prevent security threats at the point of deployment on
HPC systems, such as through hardware-based trusted execution environments,
access controls, event logging, and secure data provenance [3]. While these
are encouraging advances, we must also look to addressing security earlier in
the research software development lifecycle. As software security expert Rod
Cope has noted, “While there is—rightly—a big focus on securing software that
is already deployed, the reality is that many future vulnerabilities stem from
the creation of that software… Not securing software right from the
development stage is like putting a deadbolt on a cardboard door” [4].

This emphasis on integrating security as early as possible in the development
process is representative of the “shift left” philosophy of the DevSecOps
movement which has recently gained traction in
industry111https://www.devsecops.org. In this new line of thinking, security
advocates argue that it is critical to understand where and how software is
created so security can be moved closer to that development — that is, earlier
in the software lifecycle and with developers engaging with security in their
work.

Increasingly, the development of research software occurs on the open web. In
the past decade, research software development has transitioned from
independently developed research scripts to community-driven, open-source
software that leverages modern software development technologies. Today, even
a “bare-bones” modeling and simulation code may rely on different open-source
packages for I/O, parallel communication, meshing, discretization, solvers,
and visualization. The code teams responsible for these packages belong to
many different institutions and can vary widely in terms of their size and
maturity. In short, the leading edge of research software development is
increasingly large-scale, open-source, and distributed across teams with
different values and perspectives on software development practice.

While these changes have accelerated the pace of progress, the shift towards
an interconnected research software ecosystem has also complicated security
considerations with a proliferation of environments, actors, and attack
surfaces. In terms of people, the production of research software can involve
many teams of researchers and research software engineers, rotating casts of
student interns and postdocs, and academic and industry partners. In terms of
infrastructure, the software makes its way through numerous environments, from
open-source hosting and continuous integration platforms to national labs and
university servers and finally to clusters and HPC facilities for code
execution. In terms of data, the large amounts of data produced by the
software must move to various types of storage for analysis and reuse. A
compromise of the actors, infrastructure, or data could have disastrous
consequences.

Most reported attacks on research computing institutions so far appear to be
financially motivated, where the attackers tried to extort ransoms, hijack
compute resources, or exfiltrate data. For example, in May 2020, hackers
leveraged a vulnerability in the VPN at Michigan State University to carry out
a ransomware attack against the Physics and Astronomy Department. Between 50
and 70% of research activities were temporarily halted due to the attack, and
some research groups lost years’ worth of data [5]. There have been other
similar ransomware-style attacks recently reported in the media that targeted
the University of Amsterdam, Amsterdam University of Applied Sciences, the UK
Research and Innovation Agency, Maastricht University, and the Netherlands
Organization for Scientific Research [6]. In another type of attack,
supercomputing centers in the UK, Germany, Switzerland, and Spain fell victim
to cryptocurrency-mining hijackers [7]. In the third type of attack, security
analysts at ESET identified Kobalos, a previously unknown malware that targets
HPC clusters and university servers by creating backdoors for hackers to steal
passwords [8]. The sophistication and targeted nature of these types of
malware suggests that malicious actors may be plotting focused attacks on
research computing infrastructure.

At the time of writing, there have been no reported attacks using research
software, systems, or data as the principal attack vector. However, as
researchers increasingly develop and use open-source software there is likely
to be a consequent increase in attacks on research software. A 2021 report by
analysts at Sonatype showed a 650% year-over-year increase in attacks on open-
source software supply chains [9]. The ubiquity of open-source software
packages means that these attacks can be far-reaching. For example, the US
government estimated a vulnerability disclosed late last year in Log4J, a
popular open-source infrastructure package, potentially affects hundreds of
millions of devices [10]. Security experts anticipate open-source software
will be the next frontier for cyberattacks [11].

Within the research software domain, there are multiple approaches attackers
could use to introduce malicious code or novel vulnerabilities. First,
attackers could hijack a trusted contributor’s account (e.g. on GitHub) and
modify source code hosted in an online repository. Second, attackers could
create a seemingly useful package with hidden malicious code, perhaps typo-
squatting on a well-known package. Third, an attacker could tamper with
continuous integration or deployment services (e.g., TravisCI or Jenkins) and
quietly alter software prior to deployment. The recent spate of cryptomining
and ransomware attacks shows these types of attacks are already possible.

Meanwhile, if an attacker can infiltrate a critical piece of a research
software stack, they have many options available to create mischief. While a
financially motivated attacker might commit the attacks seen above, an
attacker motivated by attacking research sensitive to national security could
be even more dangerous:

  * •

They could introduce code to intentionally produce inaccurate results that
could lead to sub-optimal decisions or faulty science.

  * •

They could exfiltrate confidential research data.

  * •

They could sabotage the compute infrastructure by introducing code to
uselessly consume compute cycles or clog disk drives with garbage.

It is even more concerning considering that all of these threats can be
multiplied across the open-source software ecosystem. Each third-party
dependency a scientific codebase has could have vastly different coding
practices and attack surfaces. If any one of those dependencies is vulnerable,
the whole system that uses it is vulnerable.

The spectre of cyberattacks on open-source research software is not just a
material threat but also an epistemic one: it endangers our ability to trust
our codes and the resulting science. As such, we need to build security into
scientific computing from the ground up, starting with research software
development. Shifting security left, however, means confronting the fact that
research software developers are not cybersecurity experts or vice versa. Not
only do most researchers lack knowledge of security, but the typical
cybersecurity analyst is also wholly unfamiliar with the languages, libraries,
and programming models of scientific computing. This disconnect leaves open
questions about how to move forward with incorporating cybersecurity
principles into the research software development process.

In recent years the rifts between scientific and mainstream software
engineering have begun to close with the rise of the research software
engineer (RSE) professional identity and the emergence of community
organizations advocating for SE best practices (e.g. Better Scientific
Software and the US-RSE Association). As a result, a cross-disciplinary
movement has emerged with the goal of translating insights from mainstream
software engineering and to pioneer new tools and practices well-suited to the
scientific domain.

We see a similar opportunity for addressing the disconnect between
cybersecurity requirements and research software development needs. By
incorporating cybersecurity principles throughout the research software
lifecycle, especially early in the process, teams can begin to address the
concerns raised above. The Confidentiality Integrity and Availability (CIA)
triad is a standard model for designing secure information systems. The CIA
triad defines the necessary characteristics for a system to be secure.
Confidentiality is the securing of private information from unauthorized
entities. Integrity is the preservation of the original message without any
alterations. Availability is the accessibility to an entity. Table 1 lists
examples of the types of attacks that fall into each of these categories. If
the information system can meet appropriate levels of these requirements, it
meets an acceptable threshold of security.

Table 1: Example attacks for each CIA category

Confidentiality | Integrity | Availability
---|---|---
Man-in-the-Middle | Masquerade | Jamming
Eavesdropping | Code Injection | DOS
Side-Channel | Code Modification | Delay of Information

We identify two types of steps to take moving forward: (1) porting and
adopting existing cybersecurity practices into the research software domain
and (2) conducting research on how those cybersecurity practices and others
work in the research software domain.

First, well-known, proven cybersecurity practices can be integrated into the
research software development process. In general, these practices fall under
the concept of “building security in” rather than trying to bolt it on at the
end. These practices include:

  * •

Considering cybersecurity in the requirements specification step

  * •

Architecting and designing software with cybersecurity considerations in mind

  * •

Using “safe” coding practices to reduce the potential of introducing
exploitable vulnerabilities

  * •

Incorporating security checks into build and test automation to identify
problems prior to release

  * •

Using vulnerability scanners to identify problematic code

Second, additional research into the adoption of cybersecurity practices in
research software is needed. The research agenda includes answering questions
like:

  * •

How do existing cybersecurity tools and practices need to be modified to work
in a research software ecosystem that is dominated by research software
developers and RSEs?

  * •

How do we encourage the adoption of cybersecurity tools and practices among
scientific computing professionals? What can we do to lower the barriers to
entry and increase security literacy?

  * •

What implications does the use of cybersecurity tools and practices have on
the performance, reproducibility, and openness of research software?

  * •

What are the right criteria by which we can judge the overall success of
security tool and practice use? That is, when will we know that we have
succeeded?

In the long run, we believe cybersecurity could eventually become “nativized”
into research software development praxis in the same way that software
engineering concepts and tools have been adopted. Many security principles
can, with some reframing, fit into what researchers already value, such as the
FAIR principles for research software [12].

  * •

Findable software means the software has a globally unique identifier (such as
a DOI or Software Heritage ID). This facilitates the creation of “manifest”
files that precisely specify dependencies. Manifest files improve security
because they can be automatically audited for recency and known
vulnerabilities.

  * •

Available software means it is retrievable by that identifier from known
repositories. This practice improves the security of the software supply
chain, since the dependencies come from reliable, trusted sources over an
encrypted channel.

  * •

Interoperable software means that software operates behind explicit interfaces
that can be incorporated in a wide range of applications, rather than each
author writing a library for their own use-case. This improves security by
reducing the attack surface and concentrating efforts on improving a small set
of core libraries.

  * •

Reusable software means the software has the metadata necessary to reuse
software results (provenance) and the ability to get the same results in a
reuse (reproducibility). Reproducibility helps researchers detect tampering,
since they can easily compare a hash of the results, while provenance can help
researchers debug security incidents when they occur.

Our hope is that researchers may become conversant in security and able to
work with professionals in cybersecurity and software engineering to achieve
it in practice.

††Sandia National Laboratories is a multimission laboratory managed and
operated by National Technology & Engineering Solutions of Sandia, LLC, a
wholly owned subsidiary of Honeywell International Inc., for the U.S.
Department of Energy’s National Nuclear Security Administration under contract
DE-NA0003525. Any subjective views or opinions that might be expressed in the
paper do not necessarily represent the views of the U.S. Department of Energy
or the United States Government. SAND2022-10431-O.

## References

  * [1] E. Joseph, “SC20 update on the ROI and ROR from investing in HPC,” https://web.archive.org/web/20220901063620/https://www.hpcuserforum.com/ROI/, November 2020.
  * [2] S. Peisert, T. E. Potok, and T. Jones, “ASCR cybersecurity for scientific computing integrity-research pathways and ideas workshop,” Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States), Tech. Rep., 2015. [Online]. Available: https://doi.org/10.2172/1223021
  * [3] K.-C. Li, N. Sukhija, E. Bautista, and J.-L. Gaudiot, _Cybersecurity and High-Performance Computing Environments: Integrated Innovations, Practices, and Applications_. CRC Press, 2022. [Online]. Available: https://doi.org/10.1201/9781003155799
  * [4] R. Cope, “Strong security starts with software development,” _Network Security_ , vol. 2020, no. 7, pp. 6–9, 2020. [Online]. Available: https://doi.org/10.1016/S1353-4858(20)30078-7
  * [5] A. Adams, T. Siu, J. Songer, and V. Welch, “Research at risk: Ransomware attack on physics and astronomy case study,” NSF Cybersecurity Center of Excellence, Trusted CI, trustedci.org, Tech. Rep., June 2021. [Online]. Available: https://hdl.handle.net/2022/26638
  * [6] M. Enserink, “Dutch research funding agency, paralyzed by ransomware attack, refuses to pay up,” https://www.science.org/content/article/dutch-research-funding-agency-paralyzed-ransomware-attack-refuses-pay, February 2021, accessed: 2022-03-17.
  * [7] C. Cimpanu, “Supercomputers hacked across europe to mine cryptocurrency,” https://web.archive.org/web/20220708214636/https://www.zdnet.com/article/supercomputers-hacked-across-europe-to-mine-cryptocurrency/, 2020\.
  * [8] M.-E. M.Léveillé and I. Sanmillan, “A wild kobalos appears: Tricksy linux malware goes after HPCs,” ESET, Tech. Rep., January 2021. [Online]. Available: https://www.welivesecurity.com/post_paper/a-wild-kobalos-appears-tricksy-linux-malware-goes-after-hpcs/
  * [9] H. N. Security, “Open source cyberattacks increasing by 650%, popular projects more vulnerable,” https://web.archive.org/web/20221017194124/https://www.helpnetsecurity.com/2021/09/17/open-source-cyberattacks/, September 2021.
  * [10] L. Barr, “Cybersecurity official warns software vulnerability could affect ’hundreds of millions of devices’,” https://web.archive.org/web/20221026045053/https://abcnews.go.com/Politics/cybersecurity-official-warns-software-vulnerability-affect-hundreds-millions/story?id=81751264, December 2021.
  * [11] J. Carder, “Open source code: The next major wave of cyberattacks,” https://web.archive.org/web/20221021152434/https://www.darkreading.com/vulnerabilities-threats/open-source-code-the-next-major-wave-of-cyberattacks, February 2022.
  * [12] N. P. Chue Hong, D. S. Katz, M. Barker, A.-L. Lamprecht, C. Martinez, F. E. Psomopoulos, J. Harrow, L. J. Castro, M. Gruenpeter, P. A. Martinez, T. Honeyman, A. Struck, A. Lee, A. Loewe, B. van Werkhoven, C. Jones, D. Garijo, E. Plomp, F. Genova, H. Shanahan, J. Leng, M. Hellström, M. Sandström, M. Sinha, M. Kuzak, P. Herterich, Q. Zhang, S. Islam, S.-A. Sansone, T. Pollard, U. D. Atmojo, A. Williams, A. Czerniak, A. Niehues, A. C. Fouilloux, B. Desinghu, C. Goble, C. Richard, C. Gray, C. Erdmann, D. Nüst, D. Tartarini, E. Ranguelova, H. Anzt, I. Todorov, J. McNally, J. Moldon, J. Burnett, J. Garrido-Sánchez, K. Belhajjame, L. Sesink, L. Hwang, M. R. Tovani-Palone, M. D. Wilkinson, M. Servillat, M. Liffers, M. Fox, N. Miljković, N. Lynch, P. Martinez Lavanchy, S. Gesing, S. Stevens, S. Martinez Cuesta, S. Peroni, S. Soiland-Reyes, T. Bakker, T. Rabemanantsoa, V. Sochat, Y. Yehudi, and R. F. WG, “FAIR Principles for Research Software (FAIR4RS Principles),” May 2022. [Online]. Available: https://doi.org/10.15497/RDA00068

Jeffrey Carver Ph.D. is a full professor in the Department of Computer Science
at The University of Alabama. His general research interest lie in the area of
empirical software engineering and human factors in software engineering. His
research focuses on topics including: software engineering for research
software, software quality assurance, peer code review, software security,
human errors, and software engineering education.

Reed Milewicz Ph.D. is a computer scientist and software engineering
researcher in the Department of Software Engineering and Research at Sandia
National Laboratories. His research focuses on developing better practices,
processes, and tools to improve software development in the scientific domain.
He leads software science research efforts within his department and is a
member of the Interoperable Design of Extreme-Scale Application Software
(IDEAS-ECP) project, an arm of the Exascale Computing Project, where he is
part of the Productivity and Sustainability Improvement Planning (PSIP) team.

Samuel Grayson is a Ph.D. candidate at the University of Illinois at Urbana-
Champaign advised by Darko Marinov and Daniel S. Katz. Sam’s research focus is
on improving the process of developing software for computational sciences
using software tools and process improvements. He is also a summer intern at
Software Engineering and Research at Sandia National Laboratories.

Travis Atkison Ph.D. is an associate professor in the Department of Computer
Science at The University of Alabama. His current research efforts focus on
the topics of cyber security, transportation infrastructure, and control
systems security. These efforts include malicious software detection, threat
avoidance, digital forensics, and security in control system environments (in
both power systems and transportation).


s_{0}}^{t}\text{\rm{e}}^{(t-s)A}Bg^{(2)}(s)\text{\rm{d}}s$

and

$\displaystyle g^{(2)}(t)$ $\displaystyle:=\frac{T-t}{T-t\wedge
s_{0}}\nabla_{\phi(X_{0})}X_{t\wedge s_{0}}^{\mu_{\varepsilon,\phi},(2)}$
$\displaystyle\quad-\frac{(t-t\wedge
s_{0})(T-t)}{T^{2}}B^{*}\text{\rm{e}}^{(T-t)A^{*}}(U^{s_{0}}_{T})^{-1}\text{\rm{e}}^{(T-s_{0})A}\nabla_{\phi(X_{0})}X_{t\wedge
s_{0}}^{\mu_{\varepsilon,\phi},(1)}$ (4.122)
$\displaystyle\quad-\frac{(t-t\wedge
s_{0})(T-t)}{T^{2}}B^{*}\text{\rm{e}}^{(T-t)A^{*}}(U^{s_{0}}_{T})^{-1}\int_{s_{0}}^{T}\frac{T-s}{T-s_{0}}\text{\rm{e}}^{(T-s)A}B\nabla_{\phi(X_{0})}X_{s_{0}}^{\mu_{\varepsilon,\phi},(2)}\text{\rm{d}}s.$

Then our main result in the current part can be stated in the following
theorem.

######  Theorem 4.9.

Assume that (C1), (C2) and (C3) hold. Then for any
$\mu\in\mathscr{P}_{2}(\mathbb{R}^{m+l})$ and
$f\in\mathscr{B}_{b}(\mathbb{R}^{m+l}),P_{T}f$ is $L$-differentiable at $\mu$
such that

$\displaystyle
D^{L}_{\phi}(P_{T}f)(\mu)=\mathbb{E}\left(f(X_{T}^{\mu})\int_{0}^{T}\langle
K_{H}^{-1}(R_{H}h^{\phi})(t),\text{\rm{d}}W_{t}\rangle\right),\ \ \phi\in
L^{2}(\mathbb{R}^{m+l}\rightarrow\mathbb{R}^{m+l},\mu),$

where $h^{\phi}\in\mathrm{Dom}\delta\cap\mathcal{H}$ and satisfies for every
$t\in[0,T]$,

$\displaystyle(R_{H}h^{\phi})(t)$
$\displaystyle=\int_{0}^{t}\sigma^{-1}(s)\left[\nabla_{g(s)}b^{(2)}(s,\cdot,\mathscr{L}_{X_{s}^{\mu}})(X_{s}^{\mu})\right.$
$\displaystyle\qquad\qquad\qquad\left.+\left(\mathbb{E}\langle
D^{L}b^{(2)}(s,y,\cdot)(\mathscr{L}_{X_{s}^{\mu}})(X_{s}^{\mu}),\nabla_{\phi(X_{0})}X_{s}^{\mu}\rangle\right)|_{y=X_{s}^{\mu}}-(g^{(2)})^{\prime}(s)\right]\text{\rm{d}}s$

and $g=(g^{(1)},g^{(2)})$ is given by (4.121) and (4.4) for $s_{0}=0$ and
$(\nabla_{\phi(X_{0})}X_{\cdot}^{\mu,(1)},\nabla_{\phi(X_{0})}X_{\cdot}^{\mu,(2)})$
replacing
$(\nabla_{\phi(X_{0})}X_{\cdot}^{\mu_{\varepsilon,\phi},(1)},\nabla_{\phi(X_{0})}X_{\cdot}^{\mu_{\varepsilon,\phi},(2)})$.

###### Proof.

Observe first that by (4.120), we can see that $U^{s_{0}}_{t}$ is invertible
with $\|(U^{s_{0}}_{t})^{-1}\|\leq 1/\rho(t)$ for every $t\in(s_{0},T]$, and
then $g^{(1)}(t)$ and $g^{(2)}(t)$ given respectively by (4.121) and (4.4) are
well-defined. Since $\nabla b^{(1)}=(A,B)$, we have that $g^{(1)}(t)$
satisfies (4.113). Owing to (4.4), it is readily checked that $g^{(2)}(T)=0$
and (4.114) holds. Besides, by (4.121) and (4.4) again, we have

$\displaystyle
g^{(1)}(T)=\text{\rm{e}}^{(T-s_{0})A}\nabla_{\phi(X_{0})}X_{s_{0}}^{\mu_{\varepsilon,\phi},(1)}+\int_{s_{0}}^{T}\text{\rm{e}}^{(T-s)A}Bg^{(2)}(s)\text{\rm{d}}s$
$\displaystyle=\text{\rm{e}}^{(T-s_{0})A}\nabla_{\phi(X_{0})}X_{s_{0}}^{\mu_{\varepsilon,\phi},(1)}+\int_{s_{0}}^{T}\frac{T-s}{T-s_{0}}\text{\rm{e}}^{(T-s)A}B\nabla_{\phi(X_{0})}X_{s_{0}}^{\mu_{\varepsilon,\phi},(2)}\text{\rm{d}}s$
$\displaystyle\quad-\int_{s_{0}}^{T}\frac{(s-s_{0})(T-s)}{T^{2}}\text{\rm{e}}^{(T-s)A}BB^{*}\text{\rm{e}}^{(T-s)A^{*}}\text{\rm{d}}s(U^{s_{0}}_{T})^{-1}\text{\rm{e}}^{(T-s_{0})A}\nabla_{\phi(X_{0})}X_{s_{0}}^{\mu_{\varepsilon,\phi},(1)}$
$\displaystyle\quad-\int_{s_{0}}^{T}\frac{(s-s_{0})(T-s)}{T^{2}}\text{\rm{e}}^{(T-s)A}BB^{*}\text{\rm{e}}^{(T-s)A^{*}}\text{\rm{d}}s(U^{s_{0}}_{T})^{-1}\int_{s_{0}}^{T}\frac{T-s}{T-s_{0}}\text{\rm{e}}^{(T-s)A}B\nabla_{\phi(X_{0})}X_{s_{0}}^{\mu_{\varepsilon,\phi},(2)}\text{\rm{d}}s$
$\displaystyle=0,$

where the last equality is due to the definition of $U^{s_{0}}_{T}$.
For any $\varepsilon\in[0,1]$ and $s_{0}\in[0,T)$, let

(4.123) $\displaystyle\widetilde{h}_{s_{0}}^{\varepsilon,\phi}(t)$
$\displaystyle=\int_{t\wedge
s_{0}}^{t}\sigma^{-1}(s)\left[\nabla_{g(s)}b^{(2)}(s,\cdot,\mathscr{L}_{X_{s}^{\mu_{\varepsilon,\phi}}})(X_{s}^{\mu_{\varepsilon,\phi}})\right.$
(4.124) $\displaystyle\left.+\left(\mathbb{E}\langle
D^{L}b^{(2)}(s,y,\cdot)(\mathscr{L}_{X_{s}^{\mu_{\varepsilon,\phi}}})(X_{s}^{\mu_{\varepsilon,\phi}}),\nabla_{\phi(X_{0})}X_{s}^{\mu_{\varepsilon,\phi}}\rangle\right)|_{y=X_{s}^{\mu_{\varepsilon,\phi}}}-(g^{(2)})^{\prime}(s)\right]\text{\rm{d}}s,$

where $g^{(1)}$ and $g^{(2)}$ are defined by (4.121) and (4.4), respectively.
Noting that the right-hand side of (4.123) belongs to
$I_{0+}^{H+\frac{1}{2}}(L^{2}([0,T],\mathbb{R}^{l}))$ due to (C1) and (C2),
there exists $h_{s_{0}}^{\varepsilon,\phi}\in\mathcal{H}$ such that
$R_{H}h_{s_{0}}^{\varepsilon,\phi}=\widetilde{h}_{s_{0}}^{\varepsilon,\phi}$.
Under (C1) and (C2), the adaptation of our calculations in Lemma 4.7 to the
present degenerate case are straightforward. Then resorting to the same
techniques as in (4.53) and [17, Theorem 3.2], we obtain that
$K_{H}^{-1}(R_{H}h_{s_{0}}^{\varepsilon,\phi})=K_{H}^{*}h_{s_{0}}^{\varepsilon,\phi}\in
L^{2}_{a}([0,T]\times\Omega,\mathbb{R}^{l})$ (which implies that
$\delta(h_{s_{0}}^{\varepsilon,\phi})=\delta_{W}(K_{H}^{*}h_{s_{0}}^{\varepsilon,\phi})=\int_{0}^{T}\langle
K_{H}^{-1}(R_{H}h_{s_{0}}^{\varepsilon,\phi})(t),\text{\rm{d}}W_{t}\rangle$),
and
$\int_{0}^{1}\left(\mathbb{E}\delta^{2}(h_{s_{0}}^{\tau,\phi})\right)^{\frac{1}{2}}\text{\rm{d}}\tau<\infty$
as well as $\mathbb{E}\delta^{2}(h^{\phi})\leq
C_{T,K,\widetilde{K},H}\|\phi\|^{2}_{L^{2}_{\mu}}$. Furthermore, with the help
of (C3) and by a similar analysis of (4.3) we derive that

$\displaystyle\lim_{\varepsilon\rightarrow
0^{+}}\mathbb{E}|\delta(h^{\varepsilon,\phi})-\delta(h^{\phi})|=0\ \
\mathrm{and}\ \ \lim_{\|\phi\|_{L^{2}_{\mu}}\rightarrow
0}\sup\limits_{\varepsilon\in(0,1]}\frac{\mathbb{E}|\delta(h^{\varepsilon,\phi})-\delta(h^{\phi})|}{\|\phi\|_{L^{2}_{\mu}}}=0.$

Therefore, the assertions follow from Proposition 4.8. ∎

######  Remark 4.4.

Similar to Remark 4.3(ii), we can obtain that there exist two positive
constants $c_{i}(T,K,\widetilde{K},H),i=1,2$ such that

$\displaystyle\|D^{L}(P_{T}f)(\mu)\|\leq
c(T,K,\widetilde{K},H)\left[(P_{T}f^{2})(\mu)-(P_{T}f(\mu))^{2}\right]^{\frac{1}{2}},\
\ f\in\mathscr{B}(\mathbb{R}^{m+l}),$
$\displaystyle\|\mathscr{L}_{X_{T}^{\mu}}-\mathscr{L}_{X_{T}^{\nu}}\|_{\mathrm{var}}\leq
c(T,K,\widetilde{K},H)\mathbb{W}_{2}(\mu,\nu),\ \
\mu,\nu\in\mathscr{P}_{2}(\mathbb{R}^{m+l}),$

where

$\displaystyle
c(T,K,\widetilde{K},H):=c_{1}(T,K,\widetilde{K},H)+c_{2}(T,K,\widetilde{K},H)\left(\frac{1}{T^{H}}+\frac{1}{\rho(T)}+\frac{1}{T^{H}\rho(T)}\right)$

with $c(T,K,\widetilde{K},H)=O\left(\frac{1}{T^{H}\rho(T)}\right)$ when
$T\rightarrow 0$. If the following Kalman rank condition

$\displaystyle\mathrm{Rank}[B,AB,\cdots,A^{k}B]=m$

holds for some integer number $k\in[0,m-1]$, then (4.120) is satisfied with
$\rho(t)=\frac{C_{1}(t\wedge 1)^{2(k+1)}}{T\text{\rm{e}}^{C_{2}T}}$ for two
positive constants $C_{i},i=1,2$ (see, e.g., [38, Theorem 4.2]), which implies
that $c(T,K,\widetilde{K},H)=O\left(\frac{1}{T^{2k+1+H}}\right)$ as
$T\rightarrow 0$.

Acknowledgement

X. Fan is supported by the DFG through the CRC 1283 Taming uncertainty and
profiting from randomness and low regularity in analysis, stochastics and
their applications, and the Natural Science Foundation of Anhui Province (No.
2008085MA10). X. Huang is supported by the National Natural Science Foundation
of China (No. 11801406).

## References

  * [1]
  * [2] S. Albeverio, Y. G. Kondratiev and M. Röckner, Differential geometry of Poisson spaces, C. R. Acad. Sci. Paris Sér. I Math. 323 (1996), 1129–1134.
  * [3] E. Alòs, O. Mazet and D. Nualart, Stochastic calculus with respect to Gaussian processes, Ann. Probab. 29 (2001), 766–801.
  * [4] E. Alòs and D. Nualart, Stochastic integration with respect to the fractional Brownian motion, Stochastics and Stochastic Reports 75 (2003), 129–152.
  * [5] D. Baños, The Bismut-Elworthy-Li formula for mean-field stochastic differential equations, Ann. Inst. H. Poincaré Probab. Statist. 54 (2018), 220–233.
  * [6] J. Bao, P. Ren and F.-Y. Wang, Bismut formula for Lions derivative of distribution-path dependent SDEs, J. Differential Equations 282 (2021), 285–329.
  * [7] J. Bismut, Large Deviation and The Malliavin Calculus, Birkhäuser, Boston, MA, 1984.
  * [8] R. Buckdahn and S. Jing, Mean-field SDE driven by a fractional Brownian motion and related stochastic control problem, SIAM J. Control Optim. 55 (2017), 1500–1533.
  * [9] R. Buckdahn, J. Li, S. Peng and C. Rainer, Mean-field stochastic differential equations and associated PDEs, Ann. Probab. 2 (2017), 824–878.
  * [10] P. Cardaliaguet, Notes on mean field games, P.-L. Lions lectures at Collège de France, https://www.ceremade.dauphine.fr/cardaliaguet/MFG20130420.pdf, 2013.
  * [11] R. Carmona and F. Delarue, Probabilistic analysis of mean-field games, SIAM J. Control Optim. 51 (2013), 2705–2734.
  * [12] R. Carmona and F. Delarue, Forward-backward stochastic differential equations and controlled McKean-Vlasov dynamics, Ann. Probab. 43 (2015), 2647–2700.
  * [13] D. L. Cohn, Measure Theory (Second Edition), Birkhäuser, Boston, Mass., 2013.
  * [14] D. Crisan and E. McMurray, Smoothing properties of McKean-Vlasov SDEs, Probab. Theory Related Fields 171 (2018), 97–148.
  * [15] L. Decreusefond and A. S. Üstünel, Stochastic analysis of the fractional Brownian motion, Potential Anal. 10 (1998), 177–214.
  * [16] K. Elworthy and X. Li, Formulae for the derivatives of heat semigroups, J. Funct. Anal. 125 (1994), 252–286.
  * [17] X. Fan, Derivative formulas and applications for degenerate stochastic differential equations with fractional noises, J. Theoret. Probab. 32 (2019), 1360–1381.
  * [18] X. Fan and Y. Ren, Bismut formulas and applications for stochastic (functional) differential equations driven by fractional Brownian motions, Stoch. Dyn. 17 (2017), 1750028.
  * [19] M. Huang, R. Malhamé and P. Caines, Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle, Commun. Inf. Syst. 6 (2006), 221–251.
  * [20] X. Huang, M. Röckner and F.-Y. Wang, Nonlinear Fokker-Planck equations for probability measures on path space and path-distribution dependent SDEs, Discrete Contin. Dyn. Syst. 39 (2019), 3017–3035.
  * [21] X. Huang and F.-Y. Wang, Distribution dependent SDEs with singular coefficients, Stochastic Process.Appl. 129 (2019), 4747–4770.
  * [22] X. Huang and F.-Y. Wang, Derivative estimates on distributions of McKean-Vlasov SDEs, Electron. J. Probab. 26 (2021), 1–12.
  * [23] M. Kac, Foundations of kinetic theory, in: Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, University California Press, Berkeley, 1956, 171–197.
  * [24] J. Li, Mean-field forward and backward SDEs with jumps and associated nonlocal quasi-linear integral-PDEs, Stochastic Process. Appl. 128 (2018), 3118–3180.
  * [25] W. Liu and M. Röckner, Stochastic Partial Differential Equations: An Introduction, Universitext, Springer, 2015.
  * [26] H. P. McKean, Propagation of chaos for a class of non-linear parabolic equations, in: Lecture Series in Differential Equations, Session 7, 1967, 41–57.
  * [27] J. Mémin, Y. Mishura and E. Valkeila, Inequalities for the moments of Wiener integrals with respect to a fractional Brownian motion, Statist. Probab. Lett. 51 (2001), 197–206.
  * [28] A. F. Nikiforov and V. B. Uvarov, Special Functions of Mathematical Physics, Birkhäuser, Boston, 1988.
  * [29] D. Nualart, The Malliavin Calculus and Related Topics, Second edition, Springer-Verlag, Berlin, 2006.
  * [30] P. Ren and F.-Y. Wang, Bismut formula for Lions derivative of distribution dependent SDEs and applications, J. Differential Equations 267 (2019), 4745–4777.
  * [31] P. Ren and F.-Y. Wang, Space-distribution PDEs for path independent additive functionals of McKean-Vlasov SDEs, to appear in Infin. Dimens. Anal. Quantum Probab. Relat. Top.
  * [32] S. G. Samko, A. A. Kilbas and O. I. Marichev, Fractional Integrals and Derivatives, Theory and Applications, Gordon and Breach Science Publishers, Yvendon, 1993.
  * [33] Y. Song, Gradient estimates and exponential ergodicity for Mean-Field SDEs with jumps, J. Theoret. Probab. 33 (2020), 201–238.
  * [34] E. M. Stein, Singular Integrals and Differentiability Properties of Functions, Princeton University Press, Princeton, 1970.
  * [35] A. Sznitman, Topics in propagation of chaos, in: Ecole d’Eté de Probabilités de Saint-Flour XIX-1989, Springer, Berlin, Heidelberg, 1991, 165–251.
  * [36] F.-Y. Wang, Harnack Inequalities for Stochastic Partial Differential Equations, Springer, 2013.
  * [37] F.-Y. Wang, Distribution dependent SDEs for Landau type equations, Stochastic Process. Appl. 128 (2018), 595–621.
  * [38] F.-Y. Wang and X. Zhang, Derivative formula and applications for degenerate diffusion semigroups, J. Math. Pures Appl. 99 (2013), 726–740.