# An Augmentation Strategy for Visually Rich Documents

Jing Xie  GoogleMountain ViewUSA<EMAIL_ADDRESS>,  James B. Wendt
GoogleMountain ViewUSA<EMAIL_ADDRESS>,  Yichao Zhou  GoogleMountain
ViewUSA<EMAIL_ADDRESS>,  Seth Ebner  Johns Hopkins
UniversityBaltimoreUSA<EMAIL_ADDRESS>and  Sandeep Tata  GoogleMountain
ViewUSA<EMAIL_ADDRESS>
###### Abstract.

Many business workflows require extracting important fields from form-like
documents (e.g. bank statements, bills of lading, purchase orders, etc.).
Recent techniques for automating this task work well only when trained with
large datasets. In this work we propose a novel data augmentation technique to
improve performance when training data is scarce, e.g. 10–250 documents. Our
technique, which we call FieldSwap, works by swapping out the key phrases of a
source field with the key phrases of a target field to generate new synthetic
examples of the target field for use in training. We demonstrate that this
approach can yield 1–7 F1 point improvements in extraction performance.

information extraction, data augmentation

††copyright: none††ccs: Computing methodologies Information extraction

##  1\. Introduction

Visually rich documents like invoices, receipts, paystubs, insurance
statements, and tax forms are pervasive in business workflows. Processing
these documents continues to involve manually extracting relevant structured
information, which is both tedious and error-prone. Consequently, several
recent papers have tackled the problem of automatically extracting structured
information from such documents (Lee et al., 2022a; Garncarek et al., 2021a;
Xu et al., 2020; Wu et al., 2018; Sarkhel and Nandi, 2019). Given a target
document type with an associated set of fields of interest, as well as a set
of human-annotated training documents, these systems learn to automatically
extract the values for these fields from unseen documents of the same type.

While recent models have shown impressive performance (Jaume et al., 2019;
Park et al., 2019; Huang et al., 2019; Stanisławek et al., 2021), a major
hurdle in the development of high-quality extraction models is the large cost
of acquiring and annotating training documents. In this paper, we examine the
question of improving the data efficiency for this task especially when only a
small number of labeled training documents is available. We propose a novel
data augmentation technique called FieldSwap loosely inspired by the success
of mixing approaches in the image domain (Naveed, 2021). We exploit the
observation that most target fields in a document are indicated by a short key
phrase (like “Amount Due” for the total due in an invoice or “SSN” for the
Social Security number in a US tax form). FieldSwap generates a synthetic
example for a target field $T$ using an example for a source field $S$ and
replacing the key phrase associated with $S$ with the phrase for $T$.

Figure 1 illustrates FieldSwap on a snippet of a paystub document with typical
fields like salary, bonus, and overtime amounts for the pay period and for the
year-to-date. As the example in Figure 1 shows, when the source field $S$ is
$current.salary$ (with value $3,308.62) with the key phrase “Base Salary” and
the target field $T$ is $current.overtime$ with the key phrase “Overtime”, the
FieldSwap algorithm generates a synthetic example for the target field. So
long as we correctly identify the key phrases associated with the fields $S$
and $T$, generating these synthetic examples serves to regularize the model by
presenting the field in different surrounding contexts. Note that the choice
of source-target pairs matters. If we instead chose the source field $S$ to be
the $ytd.salary$ (with value $72,789.64) with the same key phrase, the
substitution of the key phrase would create an instance of $ytd.overtime$,
which is not a good synthetic example for the field $T=current.overtime$.

Figure 1. Example of FieldSwap on a paystub document. The source field is
current.salary with key phrase “Base Salary”. In the Field-to-Field swap, the
phrase is replaced with “Base”, another key phrase of field current.salary,
and the label is kept. In the Type-to-Type swap, the phrase is replaced with
“Overtime”, the key phrase of a new target field, current.overtime.

The key questions are: (a) how do we infer the key phrases corresponding to
each field for a given document type, and (b) what field pairs should be
considered for generating these synthetic examples? We show that having a
human expert supply a few key phrases works surprisingly well. The challenge
then becomes one of inferring these key phrases automatically when only a
small number of labeled examples are present. We show that a small model pre-
trained for an extraction task on an out-of-domain corpus can be effectively
used to identify the key phrases. We also show that simply considering all
pairs of fields of the same type can work quite effectively. Experiments on a
diverse set of corpora show that depending on the document-type, FieldSwap can
produce an improvement of 1–7 F1 points. For context, novel architecture and
pre-training objectives in this space resulted in increases of 1–1.5 F1 points
(Huang et al., 2022). We believe this is an exciting step towards better data
efficiency in extraction tasks for visually-rich documents that is orthogonal
to larger models and larger pre-training corpora.

We make the following contributions in this paper:

  * •

We introduce a data augmentation strategy called FieldSwap that generates new
examples for a field $T$ using examples from another field $S$. To our
knowledge, this is the first instance of a data augmentation strategy designed
specifically for visually rich documents.

  * •

We present simple algorithms for automatically inferring key phrases and field
pairs for generating synthetic examples.

  * •

Through experiments on several real-world datasets, we show that FieldSwap is
effective—improving average F1 scores by 1–7 points completely automatically
even with small training sets (10–250 documents).

  * •

With simple human inputs like key phrases, we observe improvement up to 14.6
F1 points for some document types. We also present a careful ablation study
showing the contribution of each of our design choices.

##  2\. FieldSwap

FieldSwap exploits the property that form fields are very often indicated by
key phrases (Majumder et al., 2020). For example, the total due field on an
invoice document is often designated by phrases such as “total” or “amount
due”, among others. We leverage this observation to generate synthetic
examples by taking an instance of a source field, $S$, swapping its key phrase
for a key phrase of an intended target field, $T$, and relabeling the instance
as an example of the target field. Note that we create a single augmented
example for a target field at a time; we do not generate entire synthetic
documents with all fields labeled.

This augmentation is governed by two inputs:

  1. (1)

The set of valid key phrases for each field that can be swapped. For example,
“total” and “amount due” are valid key phrases for a total due field in
invoices.

  2. (2)

A list of source-to-target field pairs for which key phrases can be swapped
and result in a valid synthetic example for the target field.

These settings can be specified manually by a human or they can be inferred
automatically. We find that manually specifying these settings works
surprising well (see results in Section 3.5). The main challenge is in
automatically inferring these settings using only a few examples from a small
dataset. Below, we present methods for automatically inferring key phrases and
field mappings.

###  2.1. Preliminaries

FieldSwap, at a high level, is intended to be agnostic to the model
architecture for which its augmentations might be used for training. However,
there are implementation details that are particular to different model
classes.

For model classes that operate at the document-level, such as sequence
labeling approaches (Xu et al., 2020; Lee et al., 2022a), there are a number
of constraints that complicate the implementation of the FieldSwap
augmentation. For example, since field values are also read as inputs to these
models, we must also replace both field values along with field key phrases in
source examples, lest we introduce inconsistent examples—for example, the
values of fields such as tax due and total due have different relative
magnitudes, which should be preserved. Furthermore, since these model classes
operate at the document-level, the augmentations must also preserve important
document-level semantics. For example, augmented documents must be consistent
with the document type’s schema—if a field should occur only once in a
document, we must ensure FieldSwap does not introduce a second instance.
Accommodating these kinds of constraints leads to even more questions—e.g.,
must augmentations only be made when two fields can act as both sources and
targets for one another? Must both fields always be present on the same
document in order to perform an augmentation? How might we generate multiple
augmentations on a single document?

In this work, we build an implementation of FieldSwap around the binary
classifier architecture described in (Majumder et al., 2020) as it more easily
lends itself to this augmentation strategy than sequence labeling approaches
do, since it operates at the candidate-level. In this architecture (as shown
in Figure 2), the extraction problem is broken down into two stages:

  1. (1)

Candidate generation: Base types are extracted from an OCR’ed document using
common off-the-shelf annotators, such as number annotators, date annotators,
and address annotators.

  2. (2)

Candidate classification: A binary classifier determines which candidates from
the previous stage are the field(s) in question from the document type’s
schema.

The aforementioned issues that befall sequence labeling architectures are not
present in the candidate-level approach since each candidate is treated
independently. In fact, the source candidate and target candidate may not even
belong to the same document.

A candidate is represented by three features: (i) its position on the page,
(ii) its neighboring tokens, and (iii) the relative positions of those
neighboring tokens to the candidate. Note that the actual value of the
candidate is explicitly ignored. This representation is easy to manipulate for
the purposes of FieldSwap since we need only find the key phrase in the
neighborhood feature of a source field, replace it with the key phrase of a
target field, and relabel the example as the target field.

Thus, we opted to explore FieldSwap first with the candidate-based
architecture. We found that this was sufficient to outperform even the state
of the art sequence labeling architectures in low data settings. For example,
Wang et al. (2022) evaluated FormNet (Lee et al., 2022a) with a 10-shot
learning task on FCC Forms and achieved $55.15$ Macro-F1 score, which is $11$
points worse than this candidate-based model architecture (as shown in Figure
5). We leave it to future work to adapt FieldSwap for the complexities
inherent in sequence labeling approaches.

###  2.2. Automatically Inferring Key Phrases

Recall that, since most fields have short key phrases, only a small number of
tokens in a candidate’s neighborhood will indicate the candidate’s field. We
thus define a method for measuring neighbor importance—this metric allows us
to detect those tokens that the model pays most attention to when classifying
a particular candidate. We aggregate these importance scores across all of the
positive examples for a field in the training dataset to infer the set of key
phrases for the field. We define a _phrase_ as a continuous span of tokens in
reading order.

####  2.2.1. Neighbor Importance Measurement

We reuse the same model architecture (Majumder et al., 2020) described in
Section 2.1 for the purposes of measuring neighbor importance. This model
learns a field-agnostic representation for each example’s neighborhood. It
first encodes each neighbor token by concatenating its text embedding and
relative position embedding. Then, it employs self-attention and max-pooling
to generate the _Neighborhood Encoding_ of a candidate. To measure how each
neighbor contributes to the candidate’s representation, we calculate the
_cosine similarity_ between the model’s intermediate output on the encoding of
each individual neighbor token and the post max-pooling _Neighborhood
Encoding_ , as shown in Figure 2. We use a model pretrained on a large out-of-
domain dataset for this task. The intuition is that the neighbor’s relative
position plays a critical role in identifying important neighbors and those
positional signals are usually shared across domains. Empirical results show
that a pretrained model identifies a reasonable set of important neighbors for
each example.

Figure 2. Architecture of the candidate-based extraction model. Neighboring
tokens of a current.salary candidate (e.g. $\$3,308.62$) are fed into a
Transformer-based encoder and a max-pooling layer to generate a neighborhood
encoding for measuring the importance of every neighbor phrase. Meanwhile, the
neighborhood encoding is concatenated with a candidate position embedding to
make a binary prediction for the target field.

####  2.2.2. Inferring Important Phrases per Example

After obtaining the importance scores of each neighbor for each example, we
apply _Sparsemax_ (Martins and Astudillo, 2016) across the importance scores
to get a sparse output of the neighbors with non-zero scores. We consider
these neighbors as the set of _important neighbor tokens_. We use an OCR
service111https://cloud.google.com/vision that detects characters, tokens, and
lines in the document. Lines are groups of tokens on the same y-axis that are
typically separate from other lines by way of visual features (e.g. vertical
bars in a table) or long horizontal stretches of whitespace. Using these
signals we form _important phrases_ by concatenating all tokens on the same
OCR line as long as one token is considered an important neighbor token. This
is based on our observation that a key phrase is usually short and lies in a
single line. Leveraging OCR lines to infer important phrases also makes the
process more tolerant to the model’s recall loss on important neighbor tokens.
As long as the model finds one important token, we’ll be able to infer the
longer phrase, if it exists. We do post-processing of the OCR line inferred
phrases by cleaning up any leading and trailing punctuation. We define _phrase
importance score_ as the average token importance score in the phrase.

####  2.2.3. Aggregation and Ranking by Field

Once all important phrases and importance scores have been gathered for all
positive examples, we aggregate the results by field and phrase. For any field
$F$, we use $Score(F,P,C_{i})$ to denote the phrase importance score for a $F$
candidate $C_{i}$ that has important phrase $P$ in its neighborhood. We
calculate $Importance(F,P)=1-\exp(\Sigma_{i}log(1-Score(F,P,C_{i})))$ as the
measurement of how $P$ relates to $F$. This measurement prefers phrases with
higher importance scores and frequency across $F$ candidates. We then rank all
phrases for each field by their _Importance_ and select the top $k$ phrases as
the key phrases for the field, where $k$ is a tunable hyperparameter.

####  2.2.4. Fields without Key Phrases

Not all fields necessarily have key phrases. Fields such as company name,
company address, and statement date often appear in the top corners of
documents without any specific phrase indicators. When trying to infer key
phrases for such fields, the model may output wrong phrases (usually with low
importance scores). For example, the model might infer “LLC” as a key phrase
for the company address field since it may often find many company names with
“LLC” directly above the company address field value. However, the values of
other fields cannot be part of the key phrase of another field because field
values are variable across different documents while key phrases are
consistent across the documents (belonging to the same template). To avoid
such spurious correlations, we explicitly exclude tokens that are part of the
ground truth value for any field in the document. We also set a threshold
$\theta$ to filter out any inferred key phrases with importance score below
the threshold, where $\theta$ is a tunable hyperparameter.

###  2.3. Field Pair Mappings

We explored two options to determine which fields can be swapped with each
other.

Field-to-Field swap. The simplest and most straightforward option is to swap
only examples belonging to the same field. In this case, the _source_ field,
$S$, and _target_ field, $T$, denote the same field. With this approach, we
are less likely to generate out-of-distribution synthetic examples. The
downside is that we are usually unable to generate a sizeable number of
synthetic examples unless the field has a lot of key phrase variation. In the
extreme case where the field is associated with one key phrase, FieldSwap
cannot generate any synthetic examples for this field. In practice, it is the
rare fields that we are most interested in augmenting, and these are the
fields that benefit the least from this pair mapping approach.

Type-to-Type swap. Each field is associated with a general field type such as
date, price, number, or address. A simple heuristic is to consider similar
fields for FieldSwap, so considering pairs of fields that have the same type
is a natural idea to consider. We can generate synthetic examples for a target
field (e.g. salary) from other same-type fields (e.g. bonus, overtime) by
swapping the key phrases. This approach allows us to generate more synthetic
examples for rare fields by utilizing examples from other frequent fields. It
also regularizes the model against spurious correlations with nearby non-
related text.

However, we might generate bad synthetic examples if there exist contradictory
fields with the same type. For example, current.bonus and ytd.bonus have the
same key phrase “bonus”, and current.vacation and ytd.vacation have the same
key phrase “vacation”. FieldSwap would generate contradictory synthetic
examples when swapping between the four fields, such as by creating a
synthetic current.vacation example using a ytd.bonus source example. We
mitigate the bad effects by ignoring synthetic examples if no neighbor tokens
are actually changed and by adding a final training stage in which the model
is fine-tuned on the original examples without augmentations, which serves to
reduce the impact of any out-of-distribution examples we may have introduced
by training on bad synthetic examples (details in Section 2.5).

All-to-All swap. We also considered swapping between any pair of fields, but
found that this was nearly always worse than type-to-type swaps.

###  2.4. Generating Synthetic Examples

Once the Fieldswap configuration of key phrases and source-to-target pairs is
specified (either manually or automatically), we generate synthetic examples.
This is done by iterating through all positive examples of the source fields.
For each positive example, we iterate through each of the target fields in the
source-to-target mapping and attempt to generate one synthetic example for
each of the target’s key phrases.

Note that if no phrases in the source example’s neighborhood match a source
phrase in the configuration, then no synthetic examples are generated from the
source example. Furthermore, if, after replacing the source example’s key
phrase with the target’s, we are left with an unchanged neighborhood, we also
omit the synthetic example. This helps prevent us from creating semantically
incorrect examples, such as the case previously described in Section 2.3, when
two fields have the same key phrase but are semantically different (e.g.,
current.bonus vs. ytd.bonus).

When a source phrase and target phrase are exactly the same length in tokens,
each source phrase token is replaced with its counterpart target phrase token.
When the source phrase is longer than the target phrase, we replace the first
$n_{t}$ tokens, where $n_{t}$ is the number of tokens in the target phrase,
with the target phrase tokens, then PAD the remaining $n_{s}-n_{t}$ tokens
belonging to the source phrase, where $n_{s}$ is the number of source phrase
tokens. When the source phrase is shorter than the target phrase, we replace
the entire set of source phrase tokens with the first $n_{s}$ target phrase
tokens, then replace $n_{t}-n_{s}$ of the lowest important tokens in the
neighborhood by way of the same importance measurement methods described
above. In this scenario we also update the $n_{t}-n_{s}$ trailing target
phrase tokens’ position features to that of the position of the $n_{s}$th
token.

###  2.5. Training Schedule

We design a training schedule to avoid the model learning poorly from the
augmented instances, which may deviate from the original data distribution.
This is accomplished by first training the model on all of the augmented
instances together with the initial training dataset, then fine-tuning the
model on the initial training dataset. Empirically, we see large improvements
made by this training schedule, demonstrating that this fine-tuning step helps
the model learn from both augmented and training data while forgetting out-of-
distribution information that can hurt performance. We also down-weight the
augmented instances in the first training stage so that we can control how
aggressively the model learns from the synthetic examples. Detailed
experiments are discussed in Section 3.6.

##  3\. Experiments

###  3.1. Dataset

Document Type | # Fields | Train Docs Pool Size | Test Docs
---|---|---|---
Brokerage Statements | 18 | 294 | 186
Loan Payments | 24 | 2000 | 815
Earnings | 26 | 2000 | 1847
Service Bills | 25 | 1086 | 183
CORD (Park et al., 2019) | 23 | 800 | 100
FCC Forms (Wang et al., 2022) | 13 | 200 | 300
FARA (Wang et al., 2022) | 6 | 200 | 300

Table 1. Datasets. A subset of documents are selected at random from the
larger pool to create the training sets for our experiments.

Hyperparameter | Range explored | Best performer
---|---|---
learning rate | 0.0001-0.1 | 0.001
dropout rate | 0.1-0.5 | 0.1
batch size | {64, 128, 256} | 128
top-$k$ phrases per field | 1-3 | 3
importance score threshold $\theta$ | 0-0.3 | 0.2
downweight ratio | 0.1-1 | 0.4

Table 2. Hyperparameters.

We evaluate FieldSwap on seven datasets of form-like documents, including
three public datasets: CORD(Park et al., 2019), FCC Forms(Wang et al., 2022),
FARA(Wang et al., 2022), and four proprietary datasets: Earnings, Brokerage
Statements, Loan Payments and Service Bills. Each dataset corresponds to a
different document type. Dataset statistics are summarized in Table 1. We
herein also refer to document types as domains. For each domain, we evaluate
on a fixed hold-out test set. We vary the training dataset sizes to plot
learning curves.

###  3.2. Experimental Setup

In all experiments, we started with a pretrained model that was trained on an
out-of-domain document type (invoices) with approximately $5000$ training
documents. We fine-tuned on the training examples in the target domain, as
described in (Gunel et al., 2022). We tune the hyperparameters using grid
search and use the most performant values, listed in Table 2. When training
the model on the target document type, we split the dataset into 80%-20%
training-validation sets.

We train the model on the target domain in two stages. In the first stage we
generate FieldSwap augmentations and append them to the target domain training
data, and train for 10 epochs. In the second stage we fine-tune the model on
the target domain training data without any synthetic examples for 25 epochs.
We stop training early in each stage if the AUC-ROC on the validation set has
not improved over the last 3 epochs. We find it typically takes fewer than 10
epochs to trigger early stopping.

We pick the checkpoint from the fine-tuned stage with the best AUC-ROC on the
validation split. In order to fully utilize all examples from the sample
training set, we use a simple multi-model ensemble technique (Dong et al.,
2020). We train 3 separate models with the same architecture and the same
training schedule but with different seeded training-validation splits and
different parameter initialization. We ensemble the 3 models by averaging
their outputs. We find simultaneously training and ensembling multiple models
can improve the performance substantially.

We evaluate the ensembled model by measuring the end-to-end field extraction
performance on the test set using the maximum F1 in the precision-recall
curve. We report the macro average of the F1 scores across all fields.

###  3.3. Baseline and Variability

We build our baseline model using the same architecture, transfer learning,
and multi-model ensemble technique mentioned above. The only difference is
that we don’t incorporate FieldSwap augmentation in the baseline setting.

In order to capture the inherent variability that may arise in experiments
with such small dataset sizes, we repeat our experiments across three
different axes. For a given domain and dataset size $N$, we repeat the
experiment using (i) 3 different collections of $N$ documents from the
domain’s large pool of documents (see Table 1), (ii) 3 training-validation
splits of those $N$ documents, and (iii) 3 differently seeded random
initializations of the model’s parameters. This amounts to a total of 27
experiments for the given domain and dataset size. Each data point we report
on the learning curve corresponds to the median performance across these 27
experiments on the fixed hold-out test set.

###  3.4. Human Expert

A human familiar with a given document type can easily provide additional
inputs in lieu of additional labeled examples. For instance, a human can
provide typical key phrases that indicate a field as well as mark potential
pairs of fields that should be used for FieldSwap. This idea draws on the
literature in rule-based augmentations where rules are provided in addition to
training examples. We design a human expert approach by devising a FieldSwap
configuration with human inputs. Instead of relying only on the automatically
detected key phrases and field pairs to conduct FieldSwap augmentations, we
apply human inputs to protect FieldSwap from some error-prone situations. For
key phrases, fields such as debtor_name and debtor_address from the Loan
Payments corpora do not have clear key phrases, we remove these fields
entirely in this human expert setup. In some cases, particularly rare fields
such as pto_pay and incentive_pay from the Earnings corpora might not even
have positive examples (which is common in the few-shot setting), so we rely
on domain knowledge to supply the key phrases. For field pairs, we start with
type-to-type field pairs, then prune those that are mostly likely to appear in
different tables or sections in the document.

###  3.5. Results

The experiments are aimed at answering the following four questions: (1) Is
FieldSwap effective in its fully automatic version? (2) Does it work better
with human-supplied inputs? (3) How do improvements vary across document types
and field types? (4) Which design choices were critical?

(a) Public Datasets

(b) Proprietary Datasets

Figure 3. Experiment results on different domains with different train doc
sizes. Under each setting, we repeat the experiments with different random
seeds as mentioned in Section 3.3. We report the median Macro-F1 scores.

Automatic FieldSwap. FieldSwap improves macro average F1 by 1-7 points across
6 of the 7 datasets we evaluated on, while the performance for some individual
fields improves by as much as 22 F1 points. The magnitude of improvement
varies across document types and even across dataset sizes, but FieldSwap
never hurts performance. Most consistently, improvements are most apparent in
the lower data regimes ($<100$ documents), and for some document types
improvements continue up to larger dataset sizes.

We test both field-to-field and type-to-type field mappings with automatically
inferred key phrases. As shown in Figure 3, FieldSwap improves extraction
performance at small dataset sizes (10–250 documents). Type-to-type FieldSwap
typically performs best.

The biggest improvement we observe is on the Earnings domain. Compared to
other document types, most of the fields in Earnings are in tabular format,
with similar base type (i.e. numerical values), and have clear and succinct
phrase indicators. We believe FieldSwap is most helpful when dealing with
document types with such characteristics. Furthermore, since ordering of
fields in these tables is unimportant, FieldSwap is particularly well-aligned
for augmenting these structures. That being said, the Earnings domain also
poses a challenge since many field pairs can easily yield bad synthetic
examples, as discussed in Section 2.3 (e.g., current.X vs. ytd.X). Yet, even
in the presence of these potentially contradicting pairs the type-to-type
mapping still yields the best performance. This demonstrates that the proposed
method tolerates a small number of these contradictory synthetic examples.

FieldSwap is also not useful for all domains. For example, the FCC Forms
corpus contains only a handful of swappable field pairs, namely, name and
address types, many of which in fact do not have indicative key phrases. When
fields are not marked explicitly with key phrases, FieldSwap should explicitly
not generate augmentations. For the most part, our implementation does exactly
this, however at times it does fail and find spurious phrases for these
fields. However, in this scenarios, the additional safeguards put in place
(e.g., omitting ground truth text as described in Section 2.4) helps maintain
neutral results throughout the learning curves for these fields.

Figure 4. Learning curves of Baseline, FieldSwap (type-to-type) and FieldSwap
(human expert) on Earnings and Loan Payments domain.

Domain | Field | Frequency |  F1 (FieldSwap, automatic) |  F1 (FieldSwap, human expert) |  $\Delta$F1
---|---|---|---|---|---
Earnings | ytd.sales_pay | 3.9% | 0.62 | 51.28 | 50.66
| ytd.pto_pay | 15.9% | 7.05 | 50.88 | 43.84
| current.sales_pay | 2.85% | 3.28 | 33.71 | 30.43
| current.pto_pay | 9.5% | 7.41 | 35.57 | 28.16
| current.incentive_pay | 8.3% | 15.15 | 31.97 | 16.82
Loan Payments | overdue_charges | 32.1% | 78.66 | 86.86 | 8.20
| date_of_maturity | 25.55% | 46.05 | 50.77 | 4.71
| finance_charges | 32.65% | 88.68 | 90.73 | 2.05

Table 3. Fields with the largest Median F1 score gaps between Automatic
FieldSwap and Human Expert FieldSwap when trained on 10 training documents.
Frequency refers to the fraction of documents that contain said field in the
larger pool of training documents.

FieldSwap with Human Expert. We ran experiments to answer two questions: (1)
Can inputs provided by a human expert be easily incorporated into FieldSwap?
(2) How does this compare with the algorithm to automatically infer key
phrases? We compare the performance between automatic FieldSwap with type-to-
type mappings and FieldSwap using human expert-curated phrases and mappings.
We evaluate these two settings on the Earnings and Loan Payments domains.

In the human expert setting, we obtain the key phrases by inspecting examples
in the training set—one of the authors of this paper examined a small number
of training documents in each domain and recorded the key phrases they
observed for each field. The same person also constructed the field mappings
using the method described in Section 3.4, which avoids contradictory field
pairs.

Figure 4 shows that performance can be further improved with inputs from human
experts in the low data regime (10 documents). For instance, at the 10
document setting, human inputs result in an F1 score that is nearly 7 points
higher on the Earnings corpus and about 1 F1 point higher on the Loan Payments
domain. At larger dataset sizes (250 documents), this advantage dissipates and
the results are neutral when comparing with the automatic approach. In Table
3, we observe that the gap in the 10 document setting is mostly attributed to
rare occurring fields. If the human supplied key phrases that were not present
in the small training set, this resulted in a substantial advantage over the
automatic approach which has no hope of discovering this key phrase. With
larger dataset sizes, the key-phrases provided by the human were discoverable
from the data. These results lead us to conclude that inputs from a human
expert may be useful for zero-shot or few-shot scenarios.

Figure 5. Per-field median F1 at 10-document train set size with standard
deviation error bars.

Per-field Extraction Performance. Figure 5 shows a comparison between the
baseline and FieldSwap with different settings on 4 fields in the Earnings
domain. We find the value of FieldSwap can be particularly impressive in the
few-shot setting for some fields, with improvements of 22 F1 points, even
without human inputs. In general, we tend to find the largest improvements for
the rarer fields. Note that at small dataset sizes, the particular examples
that are present in the training data can have a large impact on downstream
evaluation metrics. For example, consider a rare field that occurs in
$\sim$20% of documents, and a dataset size of 10 documents—a difference of 1
or 2 occurrences can mean a 50% difference in available training signal.

Figure 6. Ablation study. FieldSwap (type-to-type) has both downweighting and
fine-tune features turned on. FieldSwap (no downweight) only enables fine-tune
stage on the initial dataset without downweighting augmented instances.
FieldSwap (no fine-tune) only downweights augmented instances without the
additional fine-tuning stage.

###  3.6. Ablations

Figure 6 presents an ablation study to understand the impact of the more
important design choices we made, in particular, the training schedule and
downweighting of synthetic examples, both described in Section 2.5.

We use auto-inferred key phrases and type-to-type swap in all FieldSwap
settings and also compare the results against the baseline. For FieldSwap, we
disable one feature at a time to test the effectiveness of down-weighting the
augmented instances and fine-tuning on the initial training dataset. Results
show that both features are helpful. We also notice that the performance of
FieldSwap could drop below the baseline as the training set size increases
when we disable fine-tuning. The largest drop is on the Earnings domain when
the training set reaches 50 documents. Recall that the Earnings domain
contains many contradictory fields. As the training set size increases,
FieldSwap with type-to-type swap has more chances to generate contradictory
synthetic examples that could hurt the model. We find that the additional
fine-tuning stage on the initial training dataset is effective at mitigating
such bad effects.

##  4\. Related Work

###  4.1. Data Augmentation

Data augmentation is a class of techniques for acquiring additional data
automatically. Two main categories of data augmentation are rule-based and
model-based techniques, which use hard-coded data transformations or pre-
trained models (typically language models), respectively. Rule-based
techniques—such as EDA (Wei and Zou, 2019)—are easier to implement but have
limited benefit, whereas model-based techniques—such as back-translation
(Sennrich et al., 2016) and example extrapolation (Lee et al., 2021)—are more
difficult to develop but offer greater benefit (Feng et al., 2021). FieldSwap
contains elements of both categories, as it changes (possibly automatically
inferred) key phrases based on a set of swap rules.

Feng et al. (2021) suggest that “the distribution of augmented data should
neither be too similar nor too different from the original”. FieldSwap
achieves this balance by placing known key phrases in the contexts of other
key phrases, which increases diversity in a controlled way. Data for
underrepresented fields can be extrapolated using patterns learned from multi-
shot fields (Lee et al., 2021), but our work focuses on the scenario in which
all fields are few-shot. The use of schema field types in FieldSwap is similar
to the use of entity types for mention replacement in named entity
recognition, which is effective especially in low-data settings (Dai and Adel,
2020). Generating data for intermediate tasks (for example, natural language
inference), especially when combined with self-training, is effective on a
wide array of natural language processing tasks in the few-shot setting but
assumes access to unlabeled data (Vu et al., 2021).

Other data augmentation techniques have been used for multimodal tasks that
combine text and vision, such as image captioning (Atliha and Šešok, 2020) and
visual question answering (Kafle et al., 2017; Yokota and Nakayama, 2018).
FieldSwap, like these other approaches, focuses on modifying the textual
component of each input rather than the visual component; that is, the key
phrase is replaced but the spatial layout remains the same.

Perhaps the most similar prior work to ours is (Andreas, 2020). The main idea
of that work is that if two items appear in similar contexts, then they can be
interchanged wherever one of them occurs to generate new examples. In our
work, the items we change are key phrases associated with schema fields, and
we determine interchangeability based on the identity or type of the field.
Rather than generate new labeled examples by changing the value of the field,
we generate examples by changing the surrounding context (via key phrases).

###  4.2. Few-Shot Learning

Because recent approaches to form extraction require substantial amounts of
data, we are interested in improving performance in the few-shot setting.
Gunel et al. (2022) demonstrate that transfer learning, especially multi-
domain transfer learning, can greatly improve form extraction performance in
the few-shot setting. A similar method for fine-tuning with data from other
domains has been shown to work for other information extraction tasks (Xu et
al., 2021). The FieldSwap method we propose does not rely on existing data in
other domains.

###  4.3. Form Extraction

Approaches for extracting information from form-like documents typically rely
on multimodal features: text, spatial layout, and visual patterns. Models
often make use of pre-trained encoders that incorporate such multimodal
signals (Appalaraju et al., 2021; Garncarek et al., 2021b; Huang et al.,
2022), but these encoders require a large amount of pre-training data,
although they do exhibit good downstream task data efficiency during fine-
tuning (Sage et al., 2021). Large amounts of training data are also required
by span classification approaches (Majumder et al., 2020; Tata et al., 2021),
sequence labeling approaches (Aggarwal et al., 2020; Lee et al., 2022b), and
end-to-end approaches (Cheng et al., 2022). Rather than suggest a new model
architecture, we propose a method for augmenting form extraction data.

##  5\. Conclusions

In this paper, we describe a data-augmentation technique designed for
extraction problems on visually rich documents. We exploit the fact that many
fields have a “key phrase” to indicate them: we generate an augmented example
for a target field by taking an example of a source field and replacing the
key phrase with that of the target field (while retaining the other
neighboring tokens). Experiments on a variety of datasets show that this
simple technique is very effective, particularly for small training sets
(10–250 documents). Results show improvements of 1–7 points on the average
macro F1 score, and up to 22 F1 points on some fields.

This result opens up two interesting directions for future work. First, how do
we design a version of FieldSwap that works for sequence-labeling based
modeling approaches given the challenges we described in Section 2? Second,
there are several extensions to FieldSwap that are worth investigating. When
does swapping across document types help? Can we use a pre-trained LLM instead
of a human expert to generate a set of key phrases given the name or
description of a field? Can we learn information about key phrases from an
unlabeled corpus to enable semi-supervised learning (Pryzant et al., 2022)?

## References

  * (1)
  * Aggarwal et al. (2020) Milan Aggarwal, Hiresh Gupta, Mausoom Sarkar, and Balaji Krishnamurthy. 2020.  Form2Seq : A Framework for Higher-Order Form Structure Extraction. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 3830–3840.  https://doi.org/10.18653/v1/2020.emnlp-main.314
  * Andreas (2020) Jacob Andreas. 2020\.  Good-Enough Compositional Data Augmentation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 7556–7566.  https://doi.org/10.18653/v1/2020.acl-main.676
  * Appalaraju et al. (2021) Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. 2021.  DocFormer: End-to-End Transformer for Document Understanding. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. 973–983.  https://doi.org/10.1109/ICCV48922.2021.00103
  * Atliha and Šešok (2020) Viktar Atliha and Dmitrij Šešok. 2020.  Text Augmentation Using BERT for Image Captioning.  _Applied Sciences_ 10, 17 (2020).  https://doi.org/10.3390/app10175978
  * Cheng et al. (2022) Zhanzhan Cheng, Peng Zhang, Can Li, Qiao Liang, Yunlu Xu, Pengfei Li, Shiliang Pu, Yi Niu, and Fei Wu. 2022.  TRIE++: Towards End-to-End Information Extraction from Visually Rich Documents.  _arXiv preprint arXiv:2207.06744_ (2022).
  * Dai and Adel (2020) Xiang Dai and Heike Adel. 2020.  An Analysis of Simple Data Augmentation for Named Entity Recognition. In _Proceedings of the 28th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Barcelona, Spain (Online), 3861–3867.  https://doi.org/10.18653/v1/2020.coling-main.343
  * Dong et al. (2020) Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. 2020.  A survey on ensemble learning.  _Frontiers of Computer Science_ 14, 2 (2020), 241–258.
  * Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021.  A Survey of Data Augmentation Approaches for NLP. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Association for Computational Linguistics, Online, 968–988.  https://doi.org/10.18653/v1/2021.findings-acl.84
  * Garncarek et al. (2021a) Łukasz Garncarek, Rafał Powalski, Tomasz Stanisławek, Bartosz Topolski, Piotr Halama, Michał Turski, and Filip Graliński. 2021a.  LAMBERT: layout-aware language modeling for information extraction. In _International Conference on Document Analysis and Recognition_. Springer, 532–547.
  * Garncarek et al. (2021b) Łukasz Garncarek, Rafał Powalski, Tomasz Stanisławek, Bartosz Topolski, Piotr Halama, Michał Turski, and Filip Graliński. 2021b.  LAMBERT: Layout-Aware Language Modeling for Information Extraction. In _Document Analysis and Recognition – ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part I_ (Lausanne, Switzerland). Springer-Verlag, Berlin, Heidelberg, 532–547.  https://doi.org/10.1007/978-3-030-86549-8_34
  * Gunel et al. (2022) Beliz Gunel, Navneet Potti, Sandeep Tata, James B Wendt, Marc Najork, and Jing Xie. 2022\.  Data-Efficient Information Extraction from Form-Like Documents.  _arXiv preprint arXiv:2201.02647_ (2022).
  * Huang et al. (2022) Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022.  LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking.  _arXiv preprint arXiv:2204.08387_ (2022).
  * Huang et al. (2019) Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. 2019.  Icdar2019 competition on scanned receipt ocr and information extraction. In _2019 International Conference on Document Analysis and Recognition (ICDAR)_. IEEE, 1516–1520.
  * Jaume et al. (2019) Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. 2019.  FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents. In _Accepted to ICDAR-OST_.
  * Kafle et al. (2017) Kushal Kafle, Mohammed Yousefhussien, and Christopher Kanan. 2017\.  Data Augmentation for Visual Question Answering. In _Proceedings of the 10th International Conference on Natural Language Generation_. Association for Computational Linguistics, Santiago de Compostela, Spain, 198–202.  https://doi.org/10.18653/v1/W17-3529
  * Lee et al. (2022a) Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, and Tomas Pfister. 2022a.  FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction. In _ACL_.
  * Lee et al. (2022b) Chen-Yu Lee, Chun-Liang Li, Timothy Dozat, Vincent Perot, Guolong Su, Nan Hua, Joshua Ainslie, Renshen Wang, Yasuhisa Fujii, and Tomas Pfister. 2022b.  FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 3735–3754.  https://doi.org/10.18653/v1/2022.acl-long.260
  * Lee et al. (2021) Kenton Lee, Kelvin Guu, Luheng He, Tim Dozat, and Hyung Won Chung. 2021.  Neural data augmentation via example extrapolation.  _arXiv preprint arXiv:2102.01335_ (2021).
  * Majumder et al. (2020) Bodhisattwa Prasad Majumder, Navneet Potti, Sandeep Tata, James Bradley Wendt, Qi Zhao, and Marc Najork. 2020.  Representation Learning for Information Extraction from Form-like Documents. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 6495–6504.  https://doi.org/10.18653/v1/2020.acl-main.580
  * Martins and Astudillo (2016) Andre Martins and Ramon Astudillo. 2016.  From softmax to sparsemax: A sparse model of attention and multi-label classification. In _International conference on machine learning_. PMLR, 1614–1623.
  * Naveed (2021) Humza Naveed. 2021\.  Survey: Image Mixing and Deleting for Data Augmentation.  https://doi.org/10.48550/ARXIV.2106.07085
  * Park et al. (2019) Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019.  CORD: A Consolidated Receipt Dataset for Post-OCR Parsing.  (2019).
  * Pryzant et al. (2022) Reid Pryzant, Ziyi Yang, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022.  Automatic Rule Induction for Efficient Semi-Supervised Learning.  _arXiv preprint arXiv:2205.09067_ (2022).
  * Sage et al. (2021) Clément Sage, Thibault Douzon, Alex Aussem, Véronique Eglin, Haytham Elghazel, Stefan Duffner, Christophe Garcia, and Jérémy Espinas. 2021.  Data-efficient information extraction from documents with pre-trained language models. In _International Conference on Document Analysis and Recognition_. Springer, 455–469.
  * Sarkhel and Nandi (2019) Ritesh Sarkhel and Arnab Nandi. 2019.  Visual Segmentation for Information Extraction from Heterogeneous Visually Rich Documents. In _Proceedings of the 2019 International Conference on Management of Data_. 247–262.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016\.  Improving Neural Machine Translation Models with Monolingual Data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 86–96.  https://doi.org/10.18653/v1/P16-1009
  * Stanisławek et al. (2021) Tomasz Stanisławek, Filip Graliński, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemysław Biecek. 2021.  Kleister: key information extraction datasets involving long documents with complex layouts. In _International Conference on Document Analysis and Recognition_. Springer, 564–579.
  * Tata et al. (2021) Sandeep Tata, Navneet Potti, James B Wendt, Lauro Beltrão Costa, Marc Najork, and Beliz Gunel. 2021\.  Glean: structured extractions from templatic documents.  _Proceedings of the VLDB Endowment_ 14, 6 (2021), 997–1005.
  * Vu et al. (2021) Tu Vu, Minh-Thang Luong, Quoc Le, Grady Simon, and Mohit Iyyer. 2021.  STraTA: Self-Training with Task Augmentation for Better Few-shot Learning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 5715–5731.  https://doi.org/10.18653/v1/2021.emnlp-main.462
  * Wang et al. (2022) Zilong Wang, Yichao Zhou, Wei Wei, Chen-Yu Lee, and Sandeep Tata. 2022.  A Benchmark for Structured Extractions from Complex Documents.  _arXiv preprint arXiv:2211.15421_ (2022).
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019\.  EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 6382–6388.  https://doi.org/10.18653/v1/D19-1670
  * Wu et al. (2018) Sen Wu, Luke Hsiao, Xiao Cheng, Braden Hancock, Theodoros Rekatsinas, Philip Levis, and Christopher Ré. 2018.  Fonduer: Knowledge Base Construction from Richly Formatted Data. In _Proceedings of the 2018 International Conference on Management of Data_. 1301–1316.
  * Xu et al. (2021) Haoran Xu, Seth Ebner, Mahsa Yarmohammadi, Aaron Steven White, Benjamin Van Durme, and Kenton Murray. 2021\.  Gradual Fine-Tuning for Low-Resource Domain Adaptation. In _Proceedings of the Second Workshop on Domain Adaptation for NLP_. Association for Computational Linguistics, Kyiv, Ukraine, 214–221.  https://aclanthology.org/2021.adaptnlp-1.22
  * Xu et al. (2020) Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. 2020\.  LayoutLMv2: Multi-modal pre-training for visually-rich document understanding.  _arXiv preprint arXiv:2012.14740_ (2020).
  * Yokota and Nakayama (2018) Masashi Yokota and Hideki Nakayama. 2018.  Augmenting Image Question Answering Dataset by Exploiting Image Captions. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_. European Language Resources Association (ELRA), Miyazaki, Japan.  https://aclanthology.org/L18-1436


# Bayesian quantification of the Quark-Gluon Plasma: Improved design and
closure demonstration††thanks: Presented at the XXIXth International
Conference on Ultra-relativistic Nucleus-Nucleus Collisions (Quark Matter
2022).

Matthew Heffernana Speaker Charles Galea Sangyong Jeona Jean-François Paquetb
aDepartment of Physics, McGill University, Montréal, QC, H3A 2T8, Canada
bDepartment of Physics, Duke University, Durham, NC 27708, USA

###### Abstract

We present a demonstration of the design sampling and closure for the first
comprehensive Bayesian model-to-data comparison of heavy-ion measurements with
IP-Glasma initial conditions, in which we combine with state-of-the-art
hydrodynamics (MUSIC), particlization (iS3D), and transport (SMASH). We
further introduce a systematically-improvable method of sampling design points
with better projection properties to explore the parameter space of the model
for the first time in heavy-ion collisions.

##  1 Introduction

The strongly-interacting matter produced in ultra-relativistic heavy ion
collisions is successfully modeled with a pre-equilibrium phase, followed by
viscous hydrodynamics, particlization, and a hadronic afterburner with the
totality of the collision taking place in $\mathcal{O}(10)$ fm/c [1]. Bayesian
analysis techniques are increasingly applied to constrain the physical
properties of this matter and to quantify their uncertainty (c.f. [2, 3] and
references therein). These provide a systematic, methodical way to solve the
statistical inverse problem. Due to computational cost, surrogate models are
trained on event-by-event hybrid simulations at specific choices of input
parameters, called design points, chosen with space-filling sampling
strategies.

IP-Glasma is a pre-equilibrium model in the Color Glass Condensate framework
where the gluon field evolution obeys the classical Yang-Mills equation [4].
In the hydrodynamic phase, (2+1)D MUSIC [5] implements second-order viscous
hydrodynamics until the medium cools to the particlization temperature, where
particles become the pertinent degrees of freedom. The transition from
hydrodynamics to particles is implemented using the 14-moment viscous
corrections in iS3D [6]. Finally, the particles are evolved from the freezeout
surface using SMASH [7]. The viscous parameters are parameterized as in Ref.
[3] with similar Bayesian priors.

In what follows, we discuss a different, improved approach to space-filling
designs for heavy ion collisions before demonstrating self-consistency
(closure) tests with varying fractions of the design space. The aim of this
analysis is to both illustrate the self-consistency of a hybrid model with an
IP-Glasma initial stage and to introduce an improved approach to design space
sampling that allows for reliable intermediary analysis. These first steps
establish the foundation of an upcoming complete study.

##  2 Computer Experiment Design

Previous Bayesian works in heavy ion collisions have used a Latin hypercube
[8] that maximizes the minimum Euclidean distance between points in
$d$-dimensional space However, some parameters in a study may not be well-
constrained and the remainder form an active subspace. To ensure that the
coverage of the active subspace is also considered, a design that maximizes
all projections of the $d$-dimensional design space is desirable. An
implementation of this is a Maximum Projection design [9]. These designs
minimize the Maximum Projection criterion for points $x$,

$\min_{D}\psi(D)=\left\\{\frac{1}{\left(\begin{array}[]{c}n\\\
\vspace{-0.1in}2\end{array}\right)}\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}\frac{1}{\prod_{l=1}^{p}\left(x_{il}-x_{jl}\right)^{2}}\right\\}^{1/p}.$
(1)

$p$ is the number of dimensions of the space and $n$ are the number of points
in the sample space. $\psi(D)=\infty$ for $x_{il}=x_{jl}$, $i\neq j$. These
points may also be ordered such that the first $N$ points are the subset of
the full design which minimizes $\psi(D)$ given the constraint that only $N$
points may be chosen. This ensures that partially-completed designs can be
used to reliably train surrogate models as the design more rapidly achieves
coverage of the design space. This is shown in Fig. 1, which shows that the
currently popular maximin algorithm prioritizes centrally-clustered points
until the design is complete while the ordered Maximum Projection Latin
hypercube features more even partial subsets.

Figure 1: Subsequent subsets from a one-dimensional projection of the unit
11-dimensional Maximum Projection design with 350 design points compared to
similar subsets for a maximin Latin hypercube algorithm.

##  3 Self-consistency testing

A surrogate model – in heavy ion collisions, typically Gaussian processes –
can be trained on a set of design points and be used as an inexpensive model
emulator, mapping input to output at a fraction of the computational cost of
the hybrid model. Importantly, those surrogate models must be validated before
they can be used. In self-consistency (or closure) testing, given model output
with known input, the surrogate model can solve the inverse problem and
recover the input. For a fixed validation point, where the model is compared
to pseudodata generated with known parameters, the viscous posterior is shown
in Fig. 2. By increasing the number of design points in the training set, the
posterior for both $\zeta/s$ and $\eta/s$ are more constrained around the
underlying truth.

Figure 2: Viscous posterior with underlying closure test truth indicated by
dashed line. The first 100 design points were used for the comparison in blue
(“partial design”), while the full design was used for the comparison in
orange (“full design”).

This demonstrates that, with care in choosing the design points, intermediate
analyses may be performed as demonstrated in our hybrid model with IP-Glasma,
reducing the time required for the production of physics output. This can also
serve to conserve computational resources by determining if a regime of
diminishing returns has been reached, in turn ensuring prudent usage of
limited and costly computational resources for maximal physics results.

##  4 Acknowledgements

This work was supported in part by the Natural Sciences and Engineering
Research Council of Canada, and in part by the U.S. Department of Energy Grant
no. DE-FG02-05ER41367. This research was enabled in part by support provided
by Compute Canada and its regional partners Calcul Quebec, Compute Ontario,
SciNet, SHARCNET, and WestGrid.

## References

  * [1] Björn Schenke  “The smallest fluid on Earth”  In _Rept. Prog. Phys._ 84.8, 2021, pp. 082301
  * [2] D. Everett  “Phenomenological constraints on the transport properties of QCD matter with data-driven model averaging”  In _Phys. Rev. Lett._ 126.24, 2021, pp. 242301
  * [3] D. Everett  “Multisystem Bayesian constraints on the transport coefficients of QCD matter”  In _Phys. Rev. C_ 103.5, 2021, pp. 054904
  * [4] Björn Schenke, Prithwish Tribedy and Raju Venugopalan  “Event-by-event gluon multiplicity, energy density, and eccentricities in ultrarelativistic heavy-ion collisions”  In _Phys. Rev. C_ 86 American Physical Society, 2012, pp. 034908
  * [5] Bjoern Schenke, Sangyong Jeon and Charles Gale  “(3+1)D hydrodynamic simulation of relativistic heavy-ion collisions”  In _Phys. Rev._ C82, 2010, pp. 014903
  * [6] Mike McNelis, Derek Everett and Ulrich Heinz  “Particlization in fluid dynamical simulations of heavy-ion collisions: The iS3D module”  In _Comput. Phys. Commun._ 258, 2021, pp. 107604
  * [7] J. Weil et al.  “Particle production and equilibrium properties within a new hadron transport approach for heavy-ion collisions”  In _Phys. Rev. C_ 94 American Physical Society, 2016, pp. 054905
  * [8] Furong Sun, Robert B. Gramacy, Benjamin Haaland, Siyuan Lu and Youngdeok Hwang  “Synthesizing simulation and field data of solar irradiance”  In _Statistical Analysis and Data Mining: The ASA Data Science Journal_ 12.4, 2019, pp. 311–324
  * [9] V. Roshan Joseph, Evren Gul and Shan Ba  “Maximum projection designs for computer experiments”  In _Biometrika_ 102.2 [Oxford University Press, Biometrika Trust], 2015, pp. 371–380