∎ @ignorepar labelinglabel

11institutetext: Pratik Kayal 22institutetext: Department of Computer Science
and Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar
382355, India
22email<EMAIL_ADDRESS>33institutetext: Mrinal Anand
44institutetext: Department of Computer Science and Engineering, Indian
Institute of Technology Gandhinagar, Gandhinagar 382355, India
44email<EMAIL_ADDRESS>55institutetext: Harsh Desai
66institutetext: Department of Computer Science and Engineering, Indian
Institute of Technology Gandhinagar, Gandhinagar 382355, India
66email<EMAIL_ADDRESS>77institutetext: Mayank Singh 88institutetext:
Department of Computer Science and Engineering, Indian Institute of Technology
Gandhinagar, Gandhinagar 382355, India
88email<EMAIL_ADDRESS>
# Tables to LaTeX: Structure and Content Extraction from Scientific Tables

Pratik Kayal  Mrinal Anand  Harsh Desai  Mayank Singh

(Received: date / Accepted: date)

###### Abstract

Scientific documents contain tables that list important information in a
concise fashion. Structure and content extraction from tables embedded within
PDF research documents is a very challenging task due to the existence of
visual features like spanning cells and content features like mathematical
symbols and equations. Most existing table structure identification methods
tend to ignore these academic writing features. In this paper, we adapt the
transformer-based language modeling paradigm for scientific table structure
and content extraction. Specifically, the proposed model converts a tabular
image to its corresponding LaTeX source code. Overall, we outperform the
current state-of-the-art baselines and achieve an exact match accuracy of
70.35% and 49.69% on table structure and content extraction, respectively.
Further analysis demonstrates that the proposed models efficiently identify
the number of rows and columns, the alphanumeric characters, the LaTeX tokens,
and symbols.

###### Keywords:

Scientific documents Transformer LaTeX Tabular information Information
Extraction

##  1 Introduction

Documents are ubiquitous nowadays, with around 2.5 trillion PDF documents
available on the web zhong2019publaynet . Besides, private, public, and
governmental institutions often publish reports in PDF format. These documents
primarily hold the data in natural language text and structures such as lists,
tables, and ontologies sarawagi2008information . Without a reliable document
processing technique, such precious data for decision support and object
search cannot be effectively used by search engines and other downstream data-
oriented applications. Automatically extracting meaningful information from
PDF documents is considered an arduous task niklaus2018survey ;
singh2019automated . In recent years, we witness the widespread usage of
several typesetting tools for PDF generation. In particular, tools like LaTeX
help in typesetting complex scientific document styles and subsequent
production of electronic exchange documents. Brischoux et al. brischoux2009don
showed that approximately 26% of submissions to 54 randomly selected scholarly
journals from 15 different scientific disciplines use LaTeX typesetter, with a
significant difference between LaTeX-using and non-LaTeX-using disciplines.

In contrast to natural language text, which is considered unstructured, tables
provide a natural way to present data in a structured manner douglas1995using
. Tables occur in numerous variations, especially visually, such as with or
without horizontal and vertical lines, spanning multiple columns or rows, non-
standard spacing and alignment, and text formatting embley2006table . In the
recent decade, we witness several ongoing efforts in table detection and table
recognition. The table detection task involves predicting the bounding boxes
around tables embedded within a PDF document, while the table recognition task
involves the table structure and content extraction. We find significantly
large number of datasets for table detection and table structure recognition
fang2012dataset ; gao2019icdar ; li2020tablebank ; siegel2018extracting ;
zhong2019publaynet . Comparatively, table content recognition is a least
explored task with a very few datasets 8978078 ; gobel2013icdar ;
zhong2019imagebased . Interestingly, the majority of these datasets are
insufficient to perform end-to-end neural training. For instance, ICDAR 2013
table competition task gobel2013icdar  includes only 150 table instances.
Besides, several datasets comprise simpler tables that might not generalize in
real-world extraction scenarios.

Singh et al. singh2019automated  showed the limitations of current information
extraction systems for PDF-to-table recognition task. Previous research
focuses on using general OCR software like Tesseract smith2007overview ,
coupled with standard vision-based segmentation techniques to recognize the
content of tabular images tablenet ; vasileiadis2017extraction . However, such
approaches have performed poorly for scientific tables (see illustrative
examples in Figure 1). In contrast to the HTML-based table extraction task
zhong2019imagebased , PDF-based table extraction poses several challenges
depending on the formatting complexity of the typesetting tool. For instance,
LaTeX utilizes libraries like booktabs, array, multirow, longtable, and
graphicx, for table creation and libraries like amsmath, amssymb, amsbsy, and
amsthm, for various scientific and mathematical writing. Such libraries help
in creating complex scientific tables that are difficult to recognize by the
current systems accurately singh2019automated . Deng et al. 8978078  proposed
a table recognition method for LaTeX generated documents. However, due to
minimal data processing like word-level tokenization and no fine-grained task
definitions, they predict the entire LaTeX code leading to poor recognition
capabilities.

Table |  |  | $\epsilon_{LJ}\tiny{[\frac{Kcal}{mol}]}$ |  $\sigma_{LJ}$ [Å] |  $r_{cut}$ [Å]
---|---|---|---
$C-C$ | 0.0951 | 3.473 | 15.0
$C-H$ | 0.0380 | 3.159 | 15.0
$H-H$ | 0.0152 | 2.846 | 15.0
OCR |  | ensla—] — obs Al — Teut 1A \\\n
---
C-C 0.0951 3.473 15.0 \\\n
C’— HAH 0.0380 3.159 15.0\\\n
H—-H 0.0152 2.846 15.0

(a) Scientific table in tsourtis2017parameterization .

Table |  |  | Search Strategies
---|---
| (RMSE, $\sigma$) |  | (PAcc, $\sigma$)
SkILL | (0.616, 0.063) |  | (0.661, 0.045)
SkILL+pruning | (0.581, 0.099) |  | (0.663, 0.045)
Aleph | (0.656, 0.047)
OCR |  | Search Strategies\\\n (RMSE, oc)
---
(PAcc, co)\\\n\\\n\\\n SkILL\\\n SkILL+pruning
\\\n\\\n (0.616, 0.063) (0.661, 0.045)\\\n
\\\n (0.581, 0.099) (0.663, 0.045)
\\\n Aleph\\\n\\\n\\\n (0.656, 0.047)

(b) Scientific table in corte2015skill .

Figure 1: Recognizing scientific tables using the Tesseract-based recognition
tool smith2007overview . The commercial tool performs well in recognizing
natural language text. But it fails to recognize scientific symbols and
structural information in tables.

In this paper, we address some of these limitations in the existing table
recognition tools. We adapt the transformer-based language modeling paradigm
for scientific table structure and content extraction. Specifically, the
proposed model converts a tabular image to a LaTeX source code. In addition to
tables generated from the LaTeX code, we showcase the proposed model’s
effectiveness in converting images generated from non-LaTeX typesetting tools
like Microsoft Word. Our main contributions are:

  * •

We present the table recognition task as two sub-tasks, Table Structure
Recognition and LaTeX-Optical Character Recognition.

  * •

We propose transformer-based models leveraging a powerful attention-on-
attention mechanism for generating code in LaTeX language from the
corresponding table images.

  * •

We propose a dataset and several text-based evaluation metrics to evaluate the
baselines and proposed models.

  * •

We present an adversarial case study with PubTabNet pubtabnet  dataset and
Microsoft Word tables to test the robustness of the proposed model.

Outline of the paper: The entire paper is organized as follows. Section 2
introduces the related work. In Section 3, we introduce the two table
recognition tasks. We describe our proposed methodology in Section 4. Section
5 presents experimental details. In Section 6, we present the experimental
analysis and key results. Section 7.2 describes an adversarial use case.
Finally, we conclude this paper in Section 8.

##  2 Related Work

In the recent two decades, we witness large-scale digital documents available
online for mass consumption due to an exponential surge in internet
infrastructure and communication facilities. The volume of scientific output
in terms of published research papers follows similar trends TENOPIR2014159 .
A large proportion of this scientific volume comprises complex tables to
highlight results concisely. However, we witness fewer research efforts to
locate and extract this tabular information embedded within digital documents.
The complete tabular information processing literature can be broadly
classified into three sub-tasks: (i) tabular region detection, (ii) table
structure recognition, and (iii) table content recognition.

###  2.1 Tabular Region Detection

Table region detection is the problem of identifying the bounding boxes around
tables in documents. Early works used sets of predefined rules and predefined
parameters to recognize tables. For example, Ramel et al. ramel2003detection
uses line rulings and position of text and arrangement of text to find table
structures from exchange format files. However, methods that use predefined
rules and predefined parameters to recognize tables fail to compete with the
advent of data-driven methods. DeepDeSRT schreiber2017deepdesrt  is simple and
effective end-to-end approach to detect tables but is not as accurate as
compared to other state-of-the art approaches Hashmi2021CurrentSA . Datasets
like Deepfigures siegel2018extracting , ICDAR 2019 gao2019icdar , PubLayNet
zhong2019publaynet  and TableBank li2020tablebank  are some of the large-scale
datasets for table region detection. We do not explore this task in our paper.

###  2.2 Table Structure Recognition

Table structure recognition is the problem of identifying the structural
layout of a table e2006design . Yildiz et al. yildiz2005pdf2table  proposed a
technique for recognizing and reforming columns and rows by using the
horizontal and vertical overlaps present between text blocks. Tensmeyer et al.
deepsplitting  used dilated convolutions but depend upon heuristics approaches
during post-processing Hashmi2021CurrentSA . Raja et al. rajaetal  used Mask
R-CNN with ResNet-101 as a backbone network. However, thier approach is
vulnerable in the case of empty cells Hashmi2021CurrentSA . Newer methods like
scitsr  assumes that the cell text is pre-segmented and it first convert the
tables to graphs and then predict the inherent structure. Li et al.
li2020tablebank  recognize table structures in HTML format using model
proposed in deng2017image . These methods predict the spanning rows and
spanning columns with good accuracy but fail to recognize the rich visual
features and complexities of LaTeX tables.

###  2.3 Table Content Recognition

Table content recognition is the problem of recognizing the textual content
embedded inside a table. PubTabNet zhong2019imagebased  provide an HTML based
table content recognition dataset. Deng et al. 8978078  created a dataset for
LaTeX based table recognition system. Their method recognizes the structure
and content simultaneously by predicting the entire LaTeX code of a given
table image. However, their method performs poorly due to minimal data
preprocessing and fine task definitions. Furthermore, the dataset is not
publicly available.

##  3 Table Recognition Tasks

The task of generating LaTeX code from table images is non-trivial. Similar
features of the table can be encoded in different ways in the LaTeX code.
While the content in a tabular image is in sequential form, structural
information is encoded in the form of structural tokens like `\multicolumn`
and column alignment tokens like `c`, `l`, and `r` in LaTeX. The proposed
recognition models generate a sequence of tokens given the input table image.
We examine the task of generating LaTeX code from table image by dividing it
into two sub-tasks — Table Structure Recognition and LaTeX Optical Character
Recognition. The extracted structure information aids in reconstructing the
table structure. In comparison, the extracted content information can be
helpful in filling the textual placeholders of the table.

####  3.0.1 Table Structure Recognition (TSR)

This subtask recognizes the structural information embedded inside the table
images. Figure 2b shows a TSR output from a sample tabular image (see Figure
2a). Structural information such as mutliple columns (defined using the
command `\multicolumn{cols}{pos}{text}`), multiple rows (defined using the
command `\multirow{rows}{width}{text}`), the column alignment specifiers (`c`,
`l`, or `r`), horizontal lines (`\hline`, `\toprule`, `\midrule`, or
`\bottomrule`), etc., are recognized in this task. Note that, content inside
the third argument (`{text}`) of `\multicolumn` or `\multirow` commands are
recognised in L-OCR task (defined in the next section). A special placeholder
token ‘CELL’ represents a content inside a specific cell of the table.
Specifically, the vocabulary of TSR task comprises `&`, `0`, `1`, `2`, `3`,
`4`, `5`, `6`, `7`, `8`, `9`, `CELL`, `\\`, `\hline`, `\hspace`, `\multirow`,
`\multicolumn`, `\toprule`, `\midrule`, `\bottomrule`, `c`, `l`, `r`, `|`,
`\{`, and `\}`.

(a) Sample table image

{ `|` c `|` `|` c `|` c `|` c `|` } `\`hline CELL & CELL & CELL & CELL
---
`\\` `\`hline CELL & CELL & CELL & CELL `\\` `\`hline CELL &
CELL & CELL & CELL `\\` `\`hline CELL & CELL & CELL &
CELL `\\` `\`hline CELL & CELL & CELL & CELL `\\` `\`hline
CELL & CELL & CELL & CELL `\\` `\`hline
(b) TSR output

A c c u r a c y ¦ ( % ) & C o l o n ¦ C a n c e r ¦ & D u k e ¦ C a n c e r
---
¦ & L e u k e m i a ¦ `\\` R V M ¦ & 8 5 . 4 8 ¦ & 8 0 . 9 5 ¦ & 9 3 . 0 6
¦ `\\` S V M ¦ & 8 3 . 8 7 ¦ & 8 5 . 7 1 ¦ & 8 7 . 5 0 ¦ `\\` R V M ¦ `$ _`
`{ P F C V M ¦ } $` & 8 7 . 1 0 ¦ & 9 2 . 8 6 ¦ & 9 5 . 8 3 ¦ `\\` S
V M ¦ `$ _ { P F C V M ¦ } $` & 8 5 . 4 8 ¦ & 9 7 . 6 2 ¦  & 9
5 . 8 3 ¦ `\\` P F C V M ¦ `$ _ { L P ¦ } $` & 9 6 . 7 7 ¦  & 9 5 .
2 4 ¦ & 9 8 . 6 1 ¦
(c) L-OCR output

Figure 2: A sample tabular image and the corresponding TSR and L-OCR outputs.

####  3.0.2 LaTeX -Optical Character Recognition (L-OCR)

This subtask extracts content information from the table images. Content
information includes alphanumeric characters, symbols, and LaTeX tokens. While
the table’s basic structure is recognized in the TSR task, the more nuanced
structural information is captured by the L-OCR task. Instead of predicting
all possible tokens leading to vocabulary size in millions, we demonstrate a
character-based recognition scheme. A delimiter token (`¦`) is used to
identify words among the characters. Figure 2c shows the L-OCR output from a
sample tabular image (see Figure 2a). The terms `(` `Accuracy`, `(%))`
together produces the first cell in the first row of the table. `Accuracy` is
recognized as `A` `c` `c` `u` `r` `a` `c` `y` `<dim>`. The keyword `<dim>`
(delimiter token) is removed in the post-processing step. Overall, 235 unique
tokens comprises the vocabulary of L-OCR sub-task. This includes all the
alphabets and their case variants (a-z and A-Z), digits (0-9), LaTeX
environment tokens (`\vspace`, `\hspace`, etc.), brackets (`{`, `}`, etc.),
modifier character (`\`), accents (`,`, `^`, etc.) and symbols (`%`, `#`, `&`,
etc.).

##  4 The Proposed Methodology

In this section, we formally describe the recognition problem and present two
variants of gated ResNet-based Transformers for tabular information
extraction.

Figure 3: Resnet-Transformer attention-on-attention model. (b) The self-
attention mechanism. (c) The gated attention mechanism.

###  4.1 The Problem Formulation

This section introduces some notations and formalizes the problem of
information extraction from tabular images as LaTeX token sequence prediction
problem. Given a source image $\mathscr{I}$ and a target sequence of tokens
$\bm{y}$ consisting of tokens $\bm{y}_{1},\bm{y}_{2},\cdots,\bm{y}_{n}$, where
n is the length of the output, the task is to learn to predict the target
sequence $\bm{y}$ using the source image $\mathscr{I}$. At test time, the
system is given a raw input $\mathscr{I}$, the model generates a hypothesis
$\bm{y^{\ast}}$. We evaluate generated sequence $\bm{y^{\ast}}$ against the
ground-truth sequence $\bm{y}$. From the perspective of generation, the
sequence prediction task can be modeled as finding an optimal label sequence
$\bm{y^{\ast}}$ that maximizes the conditional probability
$p(\bm{y}|\mathscr{I})$, which is calculated as follows:

$p(\bm{y}|\mathscr{I})=\prod_{i=1}^{n}p(y_{i}|y_{1},y_{2},\cdots,y_{i-1},\mathscr{I})$
(1)

###  4.2 Gated ResNet Transformers

In this section, we discuss our implementation of the neural models Fully-
Gated RT (FGRT) and Partially-Gated RT (PGRT), to address the automatic table
recognition problem. To generate a sequence of text from an image, a
generative model would consist of an encoder that processes the input image
features and transforms them into a context vector. This representation, in
other words, is a summary of the entire source image. A decoder would then be
initialized with the context vector to generate the sequence.

The encoding and decoding layers: Recently, Transformer vaswani2017attention
based architectures have shown state-of-the-art performance for several Image-
to-Text tasks feng2020scene ; lyu20192d ; yang2019simple . As depicted in
Figure 3 (a), we utilize the core Transformer model implementation for text
generation and propose significant structural alterations in the attention
module. We propose models that utilize the attention-on-attention mechanism
huang2019attention  (initially proposed for LSTM models) for image-to-token
sequence translation. Specifically, the model encodes the table image using
the partial ResNet-101 he2015deep  which is then passed to an embedding layer
to generate input embeddings. Considering the complexity of the LaTeX text and
the dimension of transformer we used, we adopt the first four layers of
ResNet-101 as the feature extractor module. These input embeddings added with
the learned positional embeddings of the same dimension are given as input to
the transformer’s vaswani2017attention  encoder. The encoder contains a multi-
head self-attention mechanism sublayer and a position-wise fully connected
feed-forward network sublayer. The decoder consists of a multi-head self-
attention mechanism sublayer that attends encoder’s output, a masked self-
attention mechanism sublayer that masks future tokens when predicting the
current token, and a position-wise fully connected feed-forward network
sublayer. The Add & Norm sublayer in both the encoder and decoder adds the
sublayer’s output to its input and normalizes it using Layer Normalization
ba2016layer . The decoder takes the text sequence embeddings added with
learned positional embeddings of the same dimension as input. The Linear layer
and Softmax layer follow the decoder’s output to generate output probabilities
for the tokens. We term this ResNet-101-transformer model as RT.

The different attention mechanisms: Attention mechanisms have become an
integral part of sequence modeling tasks, allowing to capture of long-range
dependencies regardless of their distance in the input or output sequences.
Specifically, we experiment with two attention mechanisms: (i) vanilla self-
attention and (ii) gated self-attention. The vanilla self-attention is the
fundamental attention mechanism used in traditional transformer models. The
self-attention module ($f_{SA}$) relates different positions of an input
sequence to compute the input sequence representation. This module is present
in both encoder and decoder layers. The cross-attention module ($f_{CA}$)
relates different input sequence positions to the output sequence. This module
connects the encoder and decoder components.

$f_{SA}=f_{dot}(Q_{e},K_{e},V_{e})=softmax\left(\frac{Q_{e}K_{e}^{T}}{\sqrt{d}}\right)V_{e}$
(2)
$f_{CA}=f_{dot}(Q_{d},K_{e},V_{e})=softmax\left(\frac{Q_{d}K_{e}^{T}}{\sqrt{d}}\right)V_{e}$
(3)

where $f_{dot}$ represents scaled dot product attention vaswani2017attention ,
($Q_{i},K_{i},V_{i}$) denotes sequence representations in terms of query, key
and value respectively, $i\in\\{e,d\\}$, where $e$ represents encoder and $d$
represents decoder.

The gated attention mechanism filters out the unnecessary part of the sequence
by attending over the generated attention scores $f_{CA}$ and determining the
relevant attention. The gated cross attention module ($f_{GA}$) uses a
sigmoidal gating mechanism for filtering out irrelevant attention while
decoding the output. It generates an information vector ($i$) which carries
relevant representation of the input vector and an attention gate ($g$) that
filters the relevant attention scores. Now this filtered attention is applied
to the information vector to obtain attended information, or the relevant
information.

$f_{GA}=\sigma(W_{q}^{g}Q_{e}+W_{v}^{g}f_{CA}+b^{g})\odot(W_{q}^{i}Q_{e}+W_{v}^{i}f_{CA}+b^{i})$
(4)

$\sigma$ denotes sigmoid activation, $W_{q}^{i}$ and $W_{v}^{i}$ represent
weight matrices corresponding to value query and value at information vector,
respectively, $W_{q}^{g}$ and $W_{v}^{g}$ represent weight matrices
corresponding to value query and value at attention gate, respectively. Note
that, $\\{W_{q}^{i},W_{v}^{i},W_{q}^{g},W_{v}^{g}\\}\in\mathbb{R}^{d\times d}$
and $\\{b^{i},b^{g}\\}\in\mathbb{R}^{d}$. Figure 3c shows the gated attention
architecture. We term the RT variant that uses gated attention mechanism on
$f_{SA}$ and $f_{CA}$ as FGRT. And RT variant that uses gated attention only
on $f_{CA}$ as PGRT.

##  5 Experiments

This section details the experimental settings. Specifically, we discuss the
experimental datasets, evaluation metrics, and relevant baselines. To
encourage reproducibility in scientific research, we also discuss
implementation details and plan to make the entire codebase and processed
datasets publicly available.

###  5.1 Datasets

Due to the unavailability of the publicly available corpus for the generation
of LaTeX code from table images, we create a new dataset ‘Tab-To-Tex’. We only
consider papers corresponding to the topics in Computer Science, from the
preprint repository ArXiv111http://arxiv.org/. We preprocess LaTeX code before
compiling them to images and post-process the LaTeX code to make them more
suitable for the text generation task. We, first of all, extract the table
snippet which begins with `\begin{tabular}` command and ends with
`\end{tabular}` command. We remove commands like `\cite{}`, `\ref{}`, etc.,
along with the commented text that cannot be predicted from the tabular
images. The filtered tabular LaTeX code is compiled as PDF and then converted
into JPG images using Wand library222https://github.com/emcconville/wand. Note
that, to keep a smaller vocabulary for faster training, we mask LaTeX
environment (beginning with `\` character) tokens with corpus frequency less
than 1000 as a special token `\LATEX_TOKEN`.

The table images have been rendered in two different ways based on two
possibilities of aspect ratios. The aspect conserved table images (ACT) are
prepared by rendering the LaTeX images containing tables and keeping the
aspect ratios conserved. The images have a maximum height and width of 400
pixels. In contrast, fixed aspect table images (FAT) are prepared by rendering
each of the LaTeX images containing tables to images of size 400$\times$400
pixels. For the L-OCR task, we create two variants of the dataset based on the
maximum number of tokens. For example, L-OCRD 250 comprises table sources
having less than 250 tokens. L-OCRD 500 comprises table sources having less
than 500 tokens. In contrast, for the TSR task, we create a single variant
TSRD with maximum tokens less than 250.333Defining tabular structure requires
lesser tokens. Thus, we only experimented with 250 variant. The statistics of
TSRD, L-OCRD 250, and L-OCRD 500 are shown in Table 1. Since each variant can
exist in two aspect ratios, i.e., ACT and FAT, we obtain a total of six
different datasets. The dataset and code is licensed under Creative Commons
Attribution 4.0 International License and available for download at
https://tinyurl.com/ydbw9asp

Table 1: Summary of datasets. ML denotes the maximum sequence length of
samples. Samples denote the total number of image-text pairs. T/S is the
average number of tokens per sample.

| ML | Samples | Train | Val | Test | T/S
---|---|---|---|---|---|---
TSRD | 250 | $46141$ | $43138$ | $800$ | $2203$ | 76.09
L-OCRD 250 | 250 | $24187$ | $22100$ | $300$ | $1787$ | $134.60$
L-OCRD 500 | 500 | $37917$ | $35500$ | $500$ | $1917$ | $213.95$

###  5.2 Evaluation Metrics

We use several intuitive metrics for the automatic evaluation of TSR and L-OCR
systems. Consider that if the model generates a hypothesis $\bm{y^{\ast}}$
consisting of tokens $y^{\ast}_{1}y^{\ast}_{2}\cdots y^{\ast}_{n_{1}}$, during
the test phase. We evaluate by comparing it with the ground truth token
sequence code $\bm{y}$ consisting of tokens $y_{1}y_{2}\cdots y_{n_{2}}$,
where $n_{1}$ and $n_{2}$ are the number of tokens in model-generated code and
ground truth code, respectively. We employ exact code match accuracy and code
match accuracy @95% as our core evaluation metric for both TSR and L-OCR
tasks.

  1. 1.

Exact Code Match Accuracy (EA): For a correctly identified sample,
$y^{\ast}_{1}=y_{1},y^{\ast}_{2}=y_{2},\cdots,y^{\ast}_{n_{1}}=y_{n_{2}}$.

  2. 2.

Code Match Accuracy @95% (E95): For a correctly identified sample,
$y^{\ast}_{i}=y_{j},y^{\ast}_{i+1}=y_{j+1},\cdots,y^{\ast}_{i+l}=y_{j+l}$ and
$l>=0.95\times n_{2}$.

In addition, we also employ task-specific metrics that provide intuitive
criteria to compare different models trained for the specific task. For the
TSR task, we use row accuracy and column accuracy as evaluation metrics.

  1. 1.

Row Accuracy (RA): For a correctly identified sample, the number of rows in
$\bm{y^{\ast}}$ is equal to the number of rows in $\bm{y}$.

  2. 2.

Column Accuracy (CA): For a correctly identified sample, the number of columns
in $\bm{y^{\ast}}$ is equal to the number of columns in $\bm{y}$.

  3. 3.

Multi-column Recall (MCR): For a correctly identified sample, the multicolumn
string \multicolumn{number of columns} is equal for $\bm{y^{\ast}}$ and
$\bm{y}$.

  4. 4.

Multi-row Recall (MRR): For a correctly identified sample, the multirow string
\multirow{number of rows} is equal for $\bm{y^{\ast}}$ and $\bm{y}$.

Table 2: Hyper-parameters used to train Recurrent and Transformer models,
unless differently specified.

Model | Encoder | Decoder | Embedding | Encoder | Decoder
---|---|---|---|---|---
base | Type | Type | size | depth | depth
Recurrent | CNN-RNN | LSTM | 100 | 7 | 4
Transformer | ResNet-101 | Transformer | 256 | 4 | 8

For the L-OCR task, we classify each token in $\bm{y^{\ast}}$ and $\bm{y}$
into four exhaustive categories: (i) Alpha-Numeric Tokens (Alphabets and
Numbers), (ii) LaTeX Tokens (tokens that are defined in and used by LaTeX
markup language like `\cdots`, `\times`, `\textbf`, etc.), (iii) LaTeX Symbols
(symbols with escape character (‘\’) like `\%`, `\$`, `\{`, `\}`, etc.), and
(iv) Non-LaTeX Symbols (symbols like `=`, `$`, `{`, `}`, etc.). The L-OCR
specific evaluation scheme is now defined as:

  1. 1.

Alpha-Numeric Tokens Evaluation (AN): We form strings $\bm{y^{\ast}_{AN}}$ and
$\bm{y_{AN}}$ from $\bm{y^{\ast}}$ and $\bm{y}$, respectively, by keeping the
alpha-numeric tokens and discarding the rest and preserving the order. Then,
for a correctly identified sample, $\bm{y^{\ast}_{AN}}=\bm{y_{AN}}$.

  2. 2.

LaTeX Tokens Evaluation (LT): We form strings $\bm{y^{\ast}_{LT}}$ and
$\bm{y_{LT}}$ from $\bm{y^{\ast}}$ and $\bm{y}$, respectively, by keeping the
LaTeX tokens and discarding the rest and preserving the order. Then, for a
correctly identified sample, $\bm{y^{\ast}_{LT}}=\bm{y_{LT}}$.

  3. 3.

LaTeX Symbols Evaluation (LS): We form strings $\bm{y^{\ast}_{LS}}$ and
$\bm{y_{LS}}$ from $\bm{y^{\ast}}$ and $\bm{y}$, respectively, by keeping the
LaTeX symbol tokens and discarding the rest and preserving the order. Then,
for a correctly identified sample, $\bm{y^{\ast}_{LS}}=\bm{y_{LS}}$.

  4. 4.

Non-LaTeX Symbols Evaluation (NLS): We form strings $\bm{y^{\ast}_{NLS}}$ and
$\bm{y_{NLS}}$ from $\bm{y^{\ast}}$ and $\bm{y}$, respectively, by keeping the
non-LaTeX symbol tokens and discarding the rest and preserving the order.
Then, for a correctly identified sample, $\bm{y^{\ast}_{NLS}}=\bm{y_{NLS}}$.

  5. 5.

Average Levenshtien Distance (ALD): We report the average Levenshtien Distance
levenshtein1966binary  between strings $\bm{y^{\ast}}$ and $\bm{y}$.

###  5.3 State-Of-The-Art Baselines

In this section, we describe three state-of-the-art baseline systems.

  1. 1.

Image-to-markup Model (CNNL): The image-to-markup model proposed by Deng et
al. deng2017image  extracts image features using a convolutional neural
network (CNN) and arranges the features in a grid. CNN consists of eight
convolutional layers with five interleaved max-pooling layers. Each row is
then encoded using a bidirectional LSTM with the initial hidden state (called
positional embeddings) kept as trainable to capture semantic information in
the vertical direction. An LSTM decoder then uses these encoded features with
a visual attention mechanism to predict the tokens given the previously
generated token history and the features as input.

  2. 2.

TableBank (TB): TableBank li2020tablebank  model recognizes the structure from
the table images. We compare this baseline model against our proposed TSR
model. It also follows the same architecture as the CNNL baseline. Since it is
trained for the TSR task, we use it as a baseline for the TSR task.

  3. 3.

ResNet Transformer (RT): feng2020scene  proposed a model that has a partial
ResNet-101 encoder followed by a Fully-Connected Layer as a feature extractor
module and a Transformer vaswani2017attention  module consisting of encoder-
decoder structure for generating the output sequence of tokens. ResNet-101
takes an input image and generates convolutional features maps as an input to
a fully-connected layer. The fully-connected layer then generates word
embeddings added with learned positional embeddings and given as input to the
transformer’s encoder. The decoder then generates the tokens in the output
sequence in an auto-regressive manner. We adapt this model proposed for the
scene text recognition tasks to the table recognition task.

###  5.4 Implementation details

This section describes the implementation-specific parameters used to train
our proposed architecture and different baseline systems (described in the
previous section). Table 2 presents the hyperparameters used for the baseline
and proposed methods. We train the CNNL and TB baselines using the available
image-to-text method in the OpenNMT system klein-etal-2017-opennmt . We keep
the word embedding size as 100 and use seven encoders and four decoders with
other hyperparameters as default. For RT and our proposed model variants, we
use the first four layers of ResNet-101 as the feature extractor that yields a
feature map of size = $25\times 25\times 1024$. Since embedding size is kept
as 256, it is further reduced to size = $25\times 25\times 256$. Additionally,
Adam Optimizer is used with Noam’s Decay vaswani2017attention  with a starting
learning rate of $0.1$. We trained our models with a cross-entropy loss
function. We used dropout (with the probability of $0.1$) and label smoothing
($\epsilon_{ls}=0.1$) as regularizers in our model. The complete model is
trained end-to-end to maximize the likelihood of the training data. All
experiments are performed on 4$\times$NVIDIA V100 GPUs.

Table 3: Comparing our proposed models (FGRT and PGRT) against state-of-the-
art baselines (TB, CNNL, and RT) for the TSR task.

Dataset | Models | EA | E95 | RA | CA | MCR | MRR
---|---|---|---|---|---|---|---
ACT | TB | - | - | 43.25 | 60.19 | - | -
CNNL | 35.00 | 46.62 | 60.15 | 80.89 | 58.34 | 39.42
RT | 66.40 | 83.29 | 92.28 | 86.88 | 71.27 | 47.22
FGRT | 70.35 | 85.24 | 92.78 | 87.69 | 73.20 | 48.36
PGRT | 66.54 | 83.88 | 91.87 | 85.92 | 73.20 | 47.28
FAT | CNNL | 66.68 | 79.71 | 92.42 | 86.88 | 72.81 | 50.24
RT | 64.04 | 79.07 | 86.29 | 86.51 | 73.48 | 50.00
FGRT | 69.40 | 85.24 | 93.69 | 86.88 | 73.48 | 52.77
PGRT | 67.04 | 84.15 | 91.96 | 86.42 | 73.20 | 55.55

Table 4: Comparing our proposed models (FGRT and PGRT) against state-of-the-
art baselines (CNNL and RT) for the L-OCR task.

Task | Dataset | Models | EA | E95 | AN | LT | LS | NLS | ALD
---|---|---|---|---|---|---|---|---|---
L-OCR 250 | ACT | CNNL | 29.32 | 51.76 | 63.51 | 64.07 | 93.84 | 43.42 | 43.73
RT | 42.97 | 60.71 | 67.15 | 67.93 | 93.34 | 51.53 | 31.51
FGRT | 43.98 | 62.05 | 68.66 | 67.93 | 93.56 | 51.53 | 27.95
PGRT | 45.94 | 63.17 | 69.83 | 68.99 | 93.45 | 53.16 | 28.39
FAT | CNNL | 47.34 | 68.83 | 78.85 | 72.02 | 95.41 | 53.89 | 25.86
RT | 48.01 | 65.13 | 77.00 | 70.78 | 94.85 | 54.95 | 26.54
FGRT | 48.51 | 65.64 | 77.95 | 71.29 | 95.13 | 56.35 | 22.82
PGRT | 49.69 | 67.31 | 78.56 | 72.07 | 95.18 | 57.97 | 22.90
L-OCR 500 | ACT | CNNL | 20.71 | 47.31 | 54.98 | 51.85 | 92.91 | 31.19 | 65.29
RT | 36.72 | 57.27 | 61.29 | 61.34 | 91.54 | 44.91 | 54.87
FGRT | 36.93 | 58.68 | 63.06 | 61.86 | 92.38 | 45.64 | 52.67
PGRT | 36.98 | 57.27 | 63.01 | 61.86 | 91.02 | 44.60 | 51.93
FAT | CNNL | 41.05 | 67.71 | 76.06 | 67.76 | 94.63 | 48.30 | 46.58
RT | 43.08 | 64.11 | 74.33 | 66.09 | 94.05 | 50.75 | 43.40
FGRT | 42.77 | 65.10 | 75.63 | 67.70 | 94.31 | 51.17 | 39.08
PGRT | 43.11 | 64.05 | 74.54 | 67.88 | 94.00 | 54.23 | 39.65

##  6 Results and Discussions

In this section, we report the results of the proposed model against the
baselines. We start by reporting and discussing the results for TSR and L-OCR
tasks in Sections 6.1 and 6.2, respectively.

###  6.1 Evaluating TSR

Table 3 presents the results on the TSR Task. Results show that transformer
models yield better performance than the recurrent models. FGRT model
outperforms all other models against the core evaluation metrics and the
additional performance metrics. Additional evaluation metrics show that both
FGRT and PGRT models performs better in Multi-column Recognition and Multi-row
Recognition tasks. This shows the efficacy of the gated attention mechanism
compared to the vanilla attention mechanism. A manual error analysis reveals
that the FGRT fails to predict the correct alignment token in most erroneous
cases. For example, predicting the left aligned `l` column as center-aligned
`c`. For a fair comparison, we only report TB’s performance on the ACT dataset
as TB was originally trained on the TableBank recognition dataset
li2020tablebank  that only contains images with conserved aspect ratio.

###  6.2 Evaluating L-OCR

Similar to the TSR task observations, models on the L-OCR task perform better
with fixed aspect ratio images (see Table 4). For the L-OCR 250 task, results
show that the PGRT model gives the best performance with an EA value of
49.69%. It is also interesting to note that the model attains an E95 value of
67.31%. The striking difference between the two values suggests that the model
can predict a large chunk of the LaTeX code with high accuracy. To further
analyze which non-LaTeX symbols the PGRT model fails to predict, we evaluated
the generated strings by removing curly brackets (‘{’,‘}’) and the dollar sign
(‘$’) from the generated strings and the ground truth strings. We found that
removing the dollar sign (‘$’) increases the EA value to 53.27% from 49.69%,
and removing both curly brackets (‘{’, ‘}’), and the dollar sign (‘$’)
increases the EA value to 57.64%. Our empirical analysis shows that this trend
is followed by all the models for both L-OCR 250 and L-OCR 500 tasks. We
attribute that the omission of curly brackets (‘{’,‘}’) and the dollar sign
(‘$’) do not change the visual layout of the table, resulting in an inaccurate
model. Table 4 also shows a significant difference between the performance of
L-OCR 250 models and L-OCR 500 models. The highest EA performance in the L-OCR
250 and 500 tasks is 49.69% and 43.11%, respectively. At the same time, the
highest E95 performance in the L-OCR 250 and 500 tasks is 68.83% and 67.71%,
respectively. It is also observed that CNNL performs better in the ALD metrics
compared to the transformer based models. Furthermore, the results of
additional evaluation metrics show that the models can correctly identify
alphanumeric strings, LaTeX tokens, and LaTeX symbols with high accuracy and
fails to identify the non-LaTeX symbols making it a bottleneck problem. Models
like PGRT and FGRT that predict non-LaTeX symbols with higher accuracy also
achieve higher EA scores.

###  6.3 The End-To-End Evaluation

We join the results of the TSR and L-OCR tasks by joining the two output
strings from both tasks such that the respective L-OCR contents replace the
placeholder [CELL] token from the TSR task output. Since the basic structure
tokens like row separation token (\\\\) and column separation token (&) is
produced by both L-OCR and TSR, combining the two becomes possible. The end-
to-end evaluation of our best performing model PGRT combining the TSR stage
with the L-OCR stage yields an overall exact match accuracy of 44.13%.

##  7 Adversarial Case Studies:

###  7.1 PubTabNet Dataset: Tables from medical domain

The tables in PubTabNet pubtabnet  dataset are obtained from PubMed Central
Open Access Subset. Table regions are identified by matching the PDF format
and the XML format of the articles in PubMed. We notice that the tables are
visually different from ones in Tab-To-Tex because of differences in line
widths and character fonts. Thus, we evaluate our best-performing model on the
validation set of PubTabNet pubtabnet  dataset as an adversarial case study of
our trained model. For this study, we re-generate table images in PubTabNet by
first converting the HTML code to LaTeX code using Pandas library
reback2020pandas . We then follow the same processing pipeline as described in
Section 5.1 to prepare the source images and target structure and content
tokens. After processing, we got 5062 table images from the validation set. We
evaluate the PGRT model trained on Tab-To-Tex on this subset of tables images.
We got an end-to-end exact match accuracy of less than 2% as the model was
predicting extra tokens like `\midrule`, `\bottomrule`, etc. which was not
present in the ground truth. Upon further analysis, we find that the model
recognizes the table structure for this validation set with row and column
accuracy of 82.7% and 88.75%, respectively. The model also recognizes
Alphanumeric characters with an accuracy of 36.3%.

###  7.2 MS Word tables

We study a particular use case of our application by predicting the LaTeX
structure code from images of tables written in Microsoft Word (MSW). We
curate 40 tables written in MSW for the TSR task and manually prepare the
ground truth LaTeX source code. On evaluation, the FGRT trained on the FAT
images recognizes the structure with an EA score of 28.12%. The model
identified the number of rows and columns with an accuracy of 90.62% and
93.75%, respectively. In most cases, FGRT produced the output with two
`\hline` tokens instead of one, which reduced the exact match accuracy. Even
though the model identifies the rows and columns with high accuracy, it failed
to correctly identify `\multicolumn` and `\multirow` tokens in several cases.
The high row and column recognition accuracy shows that our model can generate
the LaTeX code from tabular images constructed from MSW.

##  8 Conclusion

In this paper, we propose a scientific table reconstruction problem as two
subtasks, table structure recognition and LaTeX-optical character recognition.
It aims to extract both the structure and contents of a scientific table to
its corresponding LaTeX code using a ResNet-Transformer model. We postulate
that using different models for TSR and L-OCR tasks might result in better
accuracy scores. In the future, we plan to build models that consider the
sequential nature of the L-OCR task and the non-sequential nature of the TSR
task in an end-to-end fashion.

###### Acknowledgements.

This work was supported by The Science and Engineering Research Board (SERB),
under sanction number ECR/2018/000087.

## Declarations

### Funding

This work was supported by The Science and Engineering Research Board (SERB),
under sanction number ECR/2018/000087.

### Conflict of interest

The authors declare that they have no conflict of interest.

### Availability of data and material

The dataset is licensed under Creative Commons Attribution 4.0 International
License and available for download at https://tinyurl.com/ydbw9asp

### Code availability

The code is licensed under Creative Commons Attribution 4.0 International
License and available for download at https://tinyurl.com/ydbw9asp

## References

  * (1) Ba, L.J., Kiros, J.R., Hinton, G.E.: Layer normalization.  CoRR abs/1607.06450 (2016).  URL http://arxiv.org/abs/1607.06450
  * (2) Brischoux, F., Legagneux, P.: Don’t format manuscripts.  The Scientist 23(7), 24 (2009)
  * (3) Chi, Z., Huang, H., Xu, H., Yu, H., Yin, W., Mao, X.: Complicated table structure recognition.  CoRR abs/1908.04729 (2019).  URL http://arxiv.org/abs/1908.04729
  * (4) Côrte-Real, J., Mantadelis, T., Dutra, I., Roha, R., Burnside, E.: Skill-a stochastic inductive logic learner.  In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. 555–558. IEEE (2015)
  * (5) Deng, Y., Kanervisto, A., Ling, J., Rush, A.M.: Image-to-markup generation with coarse-to-fine attention.  In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 980–989. JMLR. org (2017)
  * (6) Deng, Y., Rosenberg, D., Mann, G.: Challenges in end-to-end neural scientific table recognition.  In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 894–901 (2019)
  * (7) Douglas, S., Hurst, M., Quinn, D., et al.: Using natural language processing for identifying and interpreting tables in plain text.  In: Proceedings of the Fourth Annual Symposium on Document Analysis and Information Retrieval, pp. 535–546 (1995)
  * (8) Embley, D.W., Hurst, M., Lopresti, D., Nagy, G.: Table-processing paradigms: a research survey.  International Journal of Document Analysis and Recognition (IJDAR) 8(2-3), 66–86 (2006)
  * (9) Fang, J., Tao, X., Tang, Z., Qiu, R., Liu, Y.: Dataset, ground-truth and performance metrics for table detection evaluation.  In: 2012 10th IAPR International Workshop on Document Analysis Systems, pp. 445–449. IEEE (2012)
  * (10) Feng, X., Yao, H., Yi, Y., Zhang, J., Zhang, S.: Scene text recognition via transformer.  arXiv preprint arXiv:2003.08077 (2020)
  * (11) Gao, L., Huang, Y., Déjean, H., Meunier, J.L., Yan, Q., Fang, Y., Kleber, F., Lang, E.: Icdar 2019 competition on table detection and recognition (ctdar).  In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1510–1515. IEEE (2019)
  * (12) Göbel, M., Hassan, T., Oro, E., Orsi, G.: Icdar 2013 table competition.  In: 2013 12th International Conference on Document Analysis and Recognition, pp. 1449–1453. IEEE (2013)
  * (13) Hashmi, K.A., Liwicki, M., Stricker, D., Afzal, M.A., Afzal, M.A., Afzal, M.Z.: Current status and performance analysis of table recognition in document images with deep neural networks.  IEEE Access 9, 87663–87685 (2021)
  * (14) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015)
  * (15) Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4634–4643 (2019)
  * (16) Klein, G., Kim, Y., Deng, Y., Senellart, J., Rush, A.: OpenNMT: Open-source toolkit for neural machine translation.  In: Proceedings of ACL 2017, System Demonstrations, pp. 67–72. Association for Computational Linguistics, Vancouver, Canada (2017).  URL https://www.aclweb.org/anthology/P17-4012
  * (17) Levenshtein, V.I., et al.: Binary codes capable of correcting deletions, insertions, and reversals.  In: Soviet physics doklady, vol. 10, pp. 707–710. Soviet Union (1966)
  * (18) Li, M., Cui, L., Huang, S., Wei, F., Zhou, M., Li, Z.: Tablebank: Table benchmark for image-based table detection and recognition.  In: LREC 2020 (2020).  URL https://www.microsoft.com/en-us/research/publication/tablebank-table-benchmark-for-image-based-table-detection-and-recognition/
  * (19) Lyu, P., Yang, Z., Leng, X., Wu, X., Li, R., Shen, X.: 2d attentional irregular scene text recognizer.  arXiv preprint arXiv:1906.05708 (2019)
  * (20) Niklaus, C., Cetto, M., Freitas, A., Handschuh, S.: A survey on open information extraction.  In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 3866–3878 (2018)
  * (21) Paliwal, S.S., Vishwanath, D., Rahul, R., Sharma, M., Vig, L.: Tablenet: Deep learning model for end-to-end table detection and tabular data extraction from scanned document images.  In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 128–133. IEEE (2019)
  * (22) Raja, S., Mondal, A., Jawahar, C.V.: Table structure recognition using top-down and bottom-up cues.  CoRR abs/2010.04565 (2020).  URL https://arxiv.org/abs/2010.04565
  * (23) Ramel, J.Y., Crucianu, M., Vincent, N., Faure, C.: Detection, extraction and representation of tables.  In: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings., pp. 374–378. IEEE (2003)
  * (24) Sarawagi, S.: Information extraction.  Foundations and trends in databases 1(3), 261–377 (2008)
  * (25) Schreiber, S., Agne, S., Wolf, I., Dengel, A., Ahmed, S.: Deepdesrt: Deep learning for detection and structure recognition of tables in document images.  In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 1162–1167. IEEE (2017)
  * (26) Siegel, N., Lourie, N., Power, R., Ammar, W.: Extracting scientific figures with distantly supervised neural networks.  In: Proceedings of the 18th ACM/IEEE on joint conference on digital libraries, pp. 223–232 (2018)
  * (27) e Silva, A.C., Jorge, A.M., Torgo, L.: Design of an end-to-end method to extract information from tables.  International Journal of Document Analysis and Recognition (IJDAR) 8(2-3), 144–171 (2006)
  * (28) Singh, M., Sarkar, R., Vyas, A., Goyal, P., Mukherjee, A., Chakrabarti, S.: Automated early leaderboard generation from comparative tables.  In: European Conference on Information Retrieval, pp. 244–257. Springer (2019)
  * (29) Smith, R.: An overview of the tesseract ocr engine.  In: Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), vol. 2, pp. 629–633. IEEE (2007)
  * (30) pandas development team, T.: pandas-dev/pandas: Pandas (2020).  DOI 10.5281/zenodo.3509134.  URL https://doi.org/10.5281/zenodo.3509134
  * (31) Tenopir, C., King, D.W.: 6 - the growth of journals publishing.  In: B. Cope, A. Phillips (eds.) The Future of the Academic Journal (Second Edition), second edition edn., pp. 159 – 178. Chandos Publishing (2014).  DOI https://doi.org/10.1533/9781780634647.159.  URL http://www.sciencedirect.com/science/article/pii/B9781843347835500069
  * (32) Tensmeyer, C., Morariu, V.I., Price, B., Cohen, S., Martinez, T.: Deep splitting and merging for table structure decomposition.  In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 114–121 (2019).  DOI 10.1109/ICDAR.2019.00027
  * (33) Tsourtis, A., Harmandaris, V., Tsagkarogiannis, D.: Parameterization of coarse-grained molecular interactions through potential of mean force calculations and cluster expansion techniques.  Thermodynamics and Statistical Mechanics of Small Systems 19, 245 (2017)
  * (34) Vasileiadis, M., Kaklanis, N., Votis, K., Tzovaras, D.: Extraction of tabular data from document images.  In: Proceedings of the 14th Web for All Conference on The Future of Accessible Work, pp. 1–2 (2017)
  * (35) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need.  In: Advances in neural information processing systems, pp. 5998–6008 (2017)
  * (36) Yang, L., Wang, P., Li, H., Gao, Y., Zhang, L., Shen, C., Zhang, Y.: A simple and strong convolutional-attention network for irregular text recognition.  arXiv preprint arXiv:1904.01375 (2019)
  * (37) Yildiz, B., Kaiser, K., Miksch, S.: pdf2table: A method to extract table information from pdf files.  In: IICAI, pp. 1773–1785 (2005)
  * (38) Zhong, X., Bavani, E.S., Yepes, A.J.: Image-based table recognition: data, model, and evaluation (2019)
  * (39) Zhong, X., ShafieiBavani, E., Jimeno Yepes, A.: Image-based table recognition: Data, model, and evaluation.  In: A. Vedaldi, H. Bischof, T. Brox, J.M. Frahm (eds.) Computer Vision – ECCV 2020, pp. 564–580. Springer International Publishing, Cham (2020)
  * (40) Zhong, X., Tang, J., Yepes, A.J.: Publaynet: largest dataset ever for document layout analysis.  In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1015–1022. IEEE (2019).  DOI 10.1109/ICDAR.2019.00166


# Herd Behaviour in Public Goods Games

María Pereda  Grupo de Investigación Ingeniería de Organización y Logística
(IOL), Departamento Ingeniería de Organización, Administración de empresas y
Estadística, Escuela Técnica Superior de Ingenieros Industriales, Universidad
Politécnica de Madrid, 28006 Madrid, Spain.
Grupo Interdisciplinar de Sistemas Complejos (GISC), 28911 Leganés, Madrid,
Spain<EMAIL_ADDRESS>
###### Abstract

The problem of free-riding arises when individuals benefit from a shared
resource, service, or public good without proportionately contributing to its
provision. This conduct often leads to a collective action problem, as
individuals pursue personal gains while relying on the contributions of
others. In this study, we present a Bayesian inference model to elucidate the
behaviour of participants in a Public Goods Game, a conceptual framework that
captures the essence of the free-riding problem. Here, individuals possess
information on the distribution of group donations to the public good. Our
model is grounded in the premise that individuals strive to harmonise their
actions with the group’s donation patterns. By successfully replicating the
behavioural patterns observed in experiments, our model provides a
straightforward mechanism for understanding individual contributions to public
goods. These insights contribute to the understanding of how cooperation is
achieved through alignment with group behaviour.

Public Goods Game, Cooperation, Game Theory, Simulation modelling

††preprint: APS/123-QED

##  I  Introduction

In delving into the intricate dynamics of human cooperation and decision-
making, the provision of public goods has garnered extensive attention in
recent decades. At the core of this research lies the profound dilemma between
individually rational decisions and the collective optimal outcome. The
extensively researched paradigm for this issue is the Public Goods Game (PGG)
[1], where $N\geq 2$ agents can voluntarily contribute to a common pool. The
pool is then multiplied or enhanced by a factor $1<r<N$, and the resulting
value is shared among all participants, regardless of their individual
contributions. The Nash equilibrium of the game suggests a tendency towards
non-contribution to the common good, while the collective optimal outcome
demands maximal contributions from all participants. Beyond its theoretical
intrigue, this dilemma is of vital importance in addressing real-world
challenges such as climate change mitigation, public infrastructure
management, ecosystem protection, and sustainable exploitation, where
solutions hinge on voluntary contributions from many. Despite game theory
predictions, experimental findings, both in controlled laboratory settings and
online, consistently show that participants contribute approximately half of
their endowment in one-shot games; and in iterated games, contributions
decline but persist, with a general pattern of gradual decrease as players
learn the consequences of their actions. The observed results are influenced
by the information the participants receive about the actions of other players
[2, 3, 4, 5, 6, 7].

Much research has explored individual contributions in traditional Public
Goods Games, where decisions hinge on the average contributions of the group
members. Studies conducted by [8, 9, 10] convincingly showcase the widespread
occurrence of conditional cooperation in the Public Goods game. Utilising the
strategy method, participants were categorised into two predominant groups:
conditional cooperators, who are inclined to contribute more as others
contribute, and free-riders, who abstain from contributions regardless of
others’ behaviour. In conditions where individuals cannot base their decisions
on others’ behaviour, conditional cooperators, as demonstrated by [8, 9],
determine their contributions relying on diverse beliefs about the
contributions of other group members. As far as we are aware, only one study
[7] has delved into comprehending human decisions when presented with the
distribution of decisions (not solely the average contribution of the group),
and to date, no models have explained the behaviour observed in such a setup.
Most of the modelling efforts have focused on understanding the group dynamics
based on the average contribution to the group. Our model diverges from those
standard cooperation models (see [11, 12] for two recent reviews on
cooperation), from theoretical modelling approaches [13, 14, 15, 16], and
models using the conditional cooperation mechanism [8, 17, 18]. In our
proposal, agents base their decisions on the distribution of decisions of the
group, employing Bayesian inference to converge toward the group’s patterns of
contribution behaviour, as individuals possess pronounced inclinations to
conform and mimic the group [19, 20].

Regarding the use of Bayesian inference to explain game-theoretical dilemmas,
a large number of articles are devoted to the study of Bayesian games, i.e.,
games with incomplete information [21, 22, 23] and extensive research is
focused on the field of decision-making under uncertainty. In [24], the
authors built a model based on the psychological dual-process theories of
intuition decision-making; combining two decision processes, a controlled one
using Bayesian inference and an intuitive one based on reinforced learning.
The model is used to draw conclusions about the frugality and quality of both
types of decision-making processes. In the same uncertain context,
experimental results also showed that the most frequent decision rule used by
people was Bayes’s rule [25].

In this article, we present a model that attempts to explain the results
observed in a non-traditional Public Goods Game experiment [7] in which
participants were given, each round, the distribution of donations from all
the members in the group, and given a window of 24 hours to make each round
decision, allowing for well-thought decisions [24], which require time. The
model we propose is grounded in the notion that individuals seek convergence
toward the group’s opinion [19, 20]. Consequently, they adjust their
information based on the collective decision-making process and decide using
these updated beliefs. In essence, agents are not driven to optimise their
individual or group gains throughout the game. Instead, they actively align
themselves with the group’s behaviour.

The paper is organised as follows. We present the definition of the model in
Sec. II. The results obtained utilizing simulation are analysed in Sec. III.
Finally, we conclude with Sec. IV summarising the insights offered by our
work.

##  II  MODEL DYNAMICS

In this section, we present four models for the observed human behaviour under
a Public Goods Game in which people are informed of the distribution of
contributions of the others. Then, agent donations are based on Bayesian
inference. The first model (A) uses a non-informative prior for the agents,
the second model (B) uses a data-based prior, the third (C) includes non-
Bayesian agents and uses the data-based prior from (B), and the fourth one (D)
includes non-Bayesian agents and the non-informative prior from A.

A crucial parameter in the linear Public Goods Games is the Marginal Per
Capita Return (MPCR), which dictates the individual payoff for contributing an
additional unit to the public good. A Prisoner’s Dilemma situation arises when
$1/N<MPCR=r/N<1$. In cases where $MPCR<1/N$, opting not to contribute becomes
the payoff-dominant strategy. Conversely, if $MPCR>1$, there is no dilemma and
contributing becomes the payoff-dominant strategy. In our model, while we do
not explicitly model or study the influence of $r$, we assume that the agents
are participating in a setup where $1/N<MPCR=r/N<1$.

In our models, as in [7], for each experimental round, $N$ agents receive 10
ECU (Experimental Conditioning Unit, the virtual currency inside the game) and
decide how much to donate to the common pool: 0, 2, 4, 6, 8, 10 ECU (from zero
to all the endowment). Therefore, the probability that an agent donates each
of the six quantities follows a categorical distribution $\bm{X}\sim
Categorical(\theta_{1},\dots,\theta_{m})$, where $m$ represents the number of
categories and $\theta_{i}$ the probability of choosing the category $i$.

To make this decision, each Bayesian agent uses a prior distribution. In the
four models, we use the Dirichlet distribution as prior, since it is a
conjugate prior for the categorical distribution. Hence, $\bm{\theta}\sim
Dir(\alpha_{1},\dots,\alpha_{m})$ and

$P(\bm{\theta})\propto\prod_{i=1}^{m}\theta_{i}^{\alpha_{i}-1}I_{\sum_{i=1}^{m}\theta_{i}=1,\theta_{i}\geq
0}$ (1)

where $I$ is the indicator function.

After each round, the agents observe the distribution of decisions of the
group, i.e., the percentage of people that chose each category, and compute
the posterior predictive distribution to make their new decision.

The model operates as follows:

  1. 1.

Initialise the prior $\bm{\theta}\sim Dir(\bm{\alpha})$.

  2. 2.

The agents make their first decision using the prior predictive distribution
$\bm{Y}\sim Categorical(\theta_{1},\dots,\theta_{m})$.

  3. 3.

From round 2 to the final round, the agents observe the decisions of the $N$
agents in the previous round (data), defining the likelihood for the data
$P(Y|\bm{\theta})\sim Categorical(\theta_{1},\dots,\theta_{m})$.

  4. 4.

The agents compute the posterior distribution, which is proportional to the
likelihood $\times$ the prior

$\displaystyle P(\bm{\theta}|Y)\propto P(Y|\bm{\theta})\times P(\bm{\theta})=$
$\displaystyle\prod_{i=1}^{m}\theta_{i}^{c_{i}+\alpha_{i}-1}\propto
Dir(\bm{\theta}|\bm{c}+\bm{\alpha})$ (2)

where the vector $\bm{c}$ summarises the number of decisions in each category
present in the data.

  5. 5.

The agents compute the posterior predictive distribution

$\displaystyle P(X^{\prime}|Y)=\int P(X^{\prime}|\bm{\theta},Y)\times
P(\bm{\theta}|Y)d\theta=$ $\displaystyle\int P(X^{\prime}|\bm{\theta})\times
P(\bm{\theta}|Y)d\theta=\mathbb{E}(\bm{\theta}_{X^{\prime}}|Y)=$
$\displaystyle(c_{x}+\alpha_{x})\mathbin{/}(n+\alpha_{0})$ (3)

where $c_{x}$ counts the number of decisions in category x, and
$\alpha_{0}=\sum_{i=1}^{m}\alpha_{i}$.

  6. 6.

The agents make their next decision using the posterior predictive
distribution. Then we repeat steps 3 to 6 until the last round is reached.

The model versions have been implemented using the _R_ language and the code
can be downloaded from github https://github.com/mpereda/Herding_in_PGG.

In the following, we present the four models.

###  II.1 Bayesian agents and non-informative prior

In this initial model, we assume the agents start their participation in the
game with no initial knowledge of the behaviour of people in this type of
game. Therefore, we parameterise the Dirichlet distribution for the prior to
be the least informative prior possible, with an invariant distribution, so we
use the Jeffreys prior. The Jeffreys prior for the categorical distribution is
a Dirichlet distribution with parameters $\alpha_{i}=1/2$ for $i=1,...,m$.
Therefore, the prior for the agents follows Eq. (1) with $\alpha_{i}=1/2$.

###  II.2 Model with Bayesian agents and data-based prior

In this second model, we assume the individuals possess knowledge about the
behaviour of people in this kind of dilemmas, and so can estimate a prior
distribution of donations to use in their first decision. This would be in
line with the observed first donations and donations in one-shot games, which
happen to be approximately half of the endowment for a great variety of games.
In the model, we use empirical data from [7] which can be found in the Zenodo
repository that accompany the referred paper. The prior distribution is then a
Dirichlet distribution with parameters $\alpha_{1}=0.1$, $\alpha_{2}=0.2$,
$\alpha_{3}=0.32$, $\alpha_{4}=0.2$, $\alpha_{5}=0.07$, and $\alpha_{6}=0.11$
(see the code that accompany this work).

###  II.3 Model with Bayesian agents, data-based prior, and non-Bayesian
agents

In experiments with real people, we find a percentage of people who contribute
the whole endowment (known as _full cooperators_) and another percentage that
contribute nothing (known as _full defectors_ or _free riders_).

In this third model, we incorporate agents that implement both types of
behaviours. We estimate the percentage of _full cooperators_ and _full
defectors_ from the data in [7]. To do so, we define the _full cooperator_ as
someone who donated the whole endowment in more than 80% rounds, and
analogously we define the _full defector_ as those people who donated zero in
more than 80% of the rounds. Then, we compute the average number of _full
cooperators_ and _full defectors_ in the experimental data, resulting in a 5%
of full cooperators and 6% of full defectors (for details, see the code
accompanying this work).

###  II.4 Model with Bayesian agents, non-informative prior, and non-Bayesian
agents

This last model incorporates the non-informative prior from model A and
includes the same percentage of non-Bayesian agents in model C.

##  III  Results

This section presents the simulation results of 100 replications for each
model, with populations of N=100 and N=1000 agents.

For each model, we perform cluster analysis on the simulation results (agent
behaviours) to explore whether the models can generate different types of
aggregated behavioural patterns, as seen in the experimental data (e.g.,
highly polarised behaviours or mean-centred behaviours). Specifically, we
perform hierarchical agglomerative clustering with Euclidean distance and
single linkage (using the R library _hclust_). Each data instance is an
84-dimensional vector that summarises each simulation run, where position
$d_{i*(j-6)}$ represents the percentage of agents making the $i-th$ decision
at round $j$.

In the supplementary material, we provide, for each model, a histogram of
histograms (aggregating data from multiple model replications) using
scatterplot-type graphs with jitter. For each cluster in the cluster analyses,
we provide examples of instances inside the clusters.

The experimental results we use for comparison, from [7], are shown in Fig.1
and Fig.2, which have been reproduced with the permission of the authors.

Figure 1:  Heatmap of the distribution of decisions in a 100-people PGG
experiment with feedback on the distribution of donations (PGG_H), reproduced
from [7] with permission. Figure 2:  Heatmap of the distribution of decisions
in a second 100-people PGG experiment with feedback on the distribution of
donations (PGG_H2), reproduced from [7] with permission.

###  III.1 Results for the basic model with Bayesian agents and noninformative
prior

After conducting 100 simulations with N=100 agents and N=1000 agents, we
proceeded to perform a cluster analysis on them. An overview of all
simulations is presented in Supplementary Figures S1 and S2.

Cluster analysis unveils the presence of six distinct behaviour clusters,
labelled as clusters 1 through 6. Their representativeness, quantified by the
percentage of replications belonging to each cluster, is as follows: 31% for
cluster 1, 32% for cluster 2, 11% for cluster 3, 21% for cluster 4, 3% for
cluster 5, and 2% for cluster 6. See Supplementary Figures S3 to S8 for
simulation examples of each cluster.

Cluster 1 comprises simulations in which the most common cooperative behaviour
is centred around the average contribution of the group, along with
simulations in which the majority of contributions are above the mean
endowment (5). See Fig. 3 as a reference. In Cluster 2, simulations are
notable for the presence of a high percentage of agents contributing low
values of the endowment, with a non-negligible number of simulations featuring
a very high percentage of agents who do not contribute at all (Fig. 4).
Cluster 4 groups simulations in which the predominant contribution is 20% of
the endowment (2) (see Supplementary Figure S6). The remaining clusters
encapsulate less frequent behaviours (see Supplementary Figures S5, S7 and
S8).

Figure 3:  Heatmap of the distribution of donations of a simulation of cluster
type 1 behaviour of the basic model (A) with N=100. The average donation is
shown as dots joined with lines, in black. Figure 4:  Heatmap of the
distribution of donations of a simulation of cluster type 2 behaviour of the
basic model (A) with N=100. The average donation is shown as dots joined with
lines, in black.

In simulations with N=1000 agents, a predominant cluster emerges, which
includes 98% of the simulations, in which individual behaviours have
homogenized (Fig. 5). See Supplementary Figures S2 and S9-S10 for simulation
examples from each cluster.

Figure 5:  Heatmap of the distribution of donations of a simulation of
principal cluster for N=1000 simulations of the basic model (A). The average
donation is shown as dots joined with lines, in black.

It can be observed that a population size that is not sufficiently large
prevents Bayesian agents from homogenising their behaviour, as they would do
if they had sufficient information with a larger sample size, as in the case
of N=1000.

###  III.2 Results for the model with Bayesian agents and data-based prior

An encompassing display of all simulations can be found in Supplementary
Figures S11 (N=100) and S12 (N=1000).

Cluster analysis reveals that simulations involving N=100 agents exhibit
distinctive behavioural patterns, with six distinct categories emerging with
the following percentages of representation: 49, 36, 1, 4, 4 and 6%. The
primary cluster, depicted in Supplementary Figure S13, constitutes 49% of the
behaviours and is characterised by cooperation values that converge around
half of the endowment (see, for example, Fig. 6). A secondary group,
illustrated in S14, represents 36% of the cases and displays patterns centred
around half of the endowment but with higher dispersion, with some examples
showing high percentages of agents donating the full endowment (Fig. 7). The
fourth cluster includes replications with the majority of agents contributing
40% of the endowment (Figure S15). The fifth group shows pronounced
polarisation towards significantly lower levels of cooperation (Figure S16).
Lastly, the sixth cluster demonstrates polarisation towards higher cooperation
values.

Figure 6:  Heatmap of the distribution of donations of a simulation of cluster
type 1 behaviour of the model with Bayesian agents and data-based prior (B)
with N=100. The average donation is shown as dots joined with lines, in black.
Figure 7:  Heatmap of the distribution of donations of a simulation of cluster
type 2 behaviour of the model with Bayesian agents and data-based prior (B)
with N=100. The average donation is shown as dots joined with lines, in black.

In the simulations involving N=1000 agents, a predominant cluster, as depicted
in Supplementary Figure S18 and Figure 8, comes to the forefront, encompassing
a remarkable 97% of the simulations. Within this cluster, individual
behaviours have significantly converged and homogenised around the
distribution used as a prior.

Figure 8:  Heatmap of the distribution of donations of a simulation of cluster
type 1 of the model with Bayesian agents and data-based prior (B) with N=1000.
The average donation is shown as dots joined with lines, in black.

###  III.3 Results for the model with Bayesian agents, data-based prior, and
non-Bayesian agents

The results of the cluster analysis with N=100 return 4 clusters (86%, 6%, 2%,
and 6%, respectively), shown in Supplementary Figures S21 to S24. The
behaviours are grouped into a main cluster that contains 86% of the
simulations. These show agent behaviour that becomes more polarised as the
rounds of the game progress towards highly polarised values (most agents
donate all or nothing at the end of the game), see Fig. 9 as an example.
Cluster 2 groups simulations with ultra-cooperative behaviour, i.e. those with
an extremely high percentage of agents donating all the endowment (Fig. S22).
The remaining clusters, which are less representative, show polarised
simulations but to a lesser extent than the previous ones.

Figure 9:  Heatmap of the distribution of donations of a simulation of cluster
type 1 behaviour of the model with bayesian and non-bayesian agents and data-
based prior (C) with N=100. The average donation is shown as dots joined with
lines, in black.

The outcomes of cluster analysis for N=1000 reveal four clusters, which make
up 81%, 17%, 1%, and 1% of the total, respectively. See Supplementary Figures
S25 to S28 for more details. A cluster appears that groups most of the
simulations 81%, in which the agents’ behaviours are grouped around average
values of the endowment, as reflected in the prior used in the first round of
the game. Non-negligible percentages of agents donating all or none of the
endowment are observed, but in much lower proportions than in the case with
N=100 agents. For reference, see the example in Fig. 10. The second cluster
shows very similar behaviour, but with a higher percentage of donations
concentrated in 40% of the endowment (Supplementary Figure S26).

Figure 10:  Heatmap of the distribution of donations of a simulation of
cluster type 1 behaviour of the model with bayesian and non-bayesian agents
and data-based prior (C) with N=1000. The average donation is shown as dots
joined with lines, in black.

An overview of all simulations is provided in Supplementary Figures S19 and
S20.

###  III.4 Results for the model with Bayesian agents, non-informative prior,
and non-Bayesian agents

Supplementary Figures S29 and S30 offer a comprehensive presentation of all
simulations.

The results of the cluster analysis with N=100 indicate the presence of six
clusters, representing 82%, 5%, 9%, 1%, 1%, and 2%, respectively.
Supplementary Figures S31 to S36 provide a visual representation of these
clusters. The majority cluster, with 82% of the simulations, shows ultra-
polarised behaviour, due to the presence of _free riders_ and _full
cooperators_ (see Fig 11 and S31). With a non-informative prior, the dynamics
of the agents are attracted by agents who do not change their opinion. The
remaining clusters have simulations that are similar to cluster 1, with some
particularities. They can be seen in Supplementary Figures S32 to S36.

Figure 11:  Heatmap of the distribution of donations of a simulation of
cluster type 1 behaviour of the model with bayesian and non-bayesian agents
and data-based prior (C) with N=1000. The average donation is shown as dots
joined with lines, in black.

Cluster analysis carried out with N=1000 delineates clusters with very similar
instances, so we can consider that all behaviours have homogenised. Figure 12
and Supplementary Figure S37 visually illustrate the characteristics of this
unique group.

Figure 12:  Heatmap of the distribution of donations of a simulation of
cluster type 1 behaviour of the model with bayesian and non-bayesian agents
and data-based prior (C) with N=1000. The average donation is shown as dots
joined with lines, in black.

##  IV  Discussion and conclusions

In this study, we have introduced four models to elucidate human behaviour in
the Public Goods Game when individuals possess more information than just the
group’s average contribution, incorporating details about the distribution of
donations from all participants in the previous round. Grounded on the
assumption that individuals in this scenario are not seeking to optimise their
gains, but rather to converge and imitate _the group’s_ behaviour. The first
model, model A, is the simplest, assuming that individuals have no knowledge
of how others behave in this game. Model B refines this assumption,
incorporating some form of knowledge, drawing insights from real experimental
data [7]. The last two models extend these initial models by introducing
individuals who are not influenced by the group but exhibit independent
behaviours such as _free riders_ or _full cooperators_ , as observed in
experiments.

Overall, our results demonstrate scale effects on behaviours. Various patterns
emerge for the distribution of donations in mid-sized groups (N=100), some
resembling those observed in real human data. However, these patterns tend to
disappear on larger scales (N = 1000), where a predominant and more
homogeneous behavioural pattern is observed across replications.

Model C with 100 agents exhibits behavioural patterns compatible with those
experimentally observed in groups of 100 individuals (see Figures 1 and 1 vs 9
and Supplementary Figure S21). We observe how agent behaviour and human
behaviour in the initial rounds stem from a prior assumption and subsequently
evolve, polarising towards situations with a high percentage of full
cooperators and free riders.

These findings could be attributed not only to the effect of Bayesian agents
but also to a small-scale effect. We note that for larger group sizes, as seen
in the results with 1000 agents presented throughout the study, agent
behaviours become much more homogeneous across all models.

Our results lead us to hypothesise that humans, in such scenarios, might
prefer imitation and convergence behaviours over profit optimisation,
providing a plausible explanation for the surprising results, high levels of
average cooperation and polarisation, in Public Goods Dilemmas of this kind
(individuals with more information).

Adhering to Occam’s Razor, our models, while limited, offer generic insights
into how cooperation is achieved through the convergence to group behaviour. A
potential extension of the models would be introducing noise or _trembling
hand_ in decisions, as found in [25], where ’people are prone to make errors
and random decisions’. This avenue presents an intriguing prospect for further
exploration.

###### Acknowledgements.

The genesis of this article emerged from informal discussions with José A.
Cuesta and Ignacio Tamarit during the GISC workshop in the pre-pandemic era.
Thanks to both for sparking up the initial idea.

## References

  * Ledyard [1995] J. O. Ledyard, Public goods: A survey of experimental research, in  _The Handbook of Experimental Economics_ (Princeton University Press, 1995) pp. 111–194, funding by JPL.
  * Zelmer [2003] J. Zelmer, Linear Public Goods Experiments: A Meta-Analysis, Experimental Economics 6, 299 (2003).
  * Peng [2022] H.-C. Peng, Punishment mechanisms and cooperation in public goods games: Experimental evidence, Annals of Public and Cooperative Economics 93, 533 (2022), publisher: John Wiley & Sons, Ltd.
  * Antonioni  _et al._ [2014] A. Antonioni, M. P. Cacault, R. Lalive, and M. Tomassini, Know Thy Neighbor: Costly Information Can Hurt Cooperation in Dynamic Networks, PLOS ONE 9, e110788 (2014), publisher: Public Library of Science.
  * Cuesta  _et al._ [2015] J. A. Cuesta, C. Gracia-Lázaro, A. Ferrer, Y. Moreno, and A. Sánchez, Reputation drives cooperative behaviour and network formation in human groups, Scientific Reports 5, 7843 (2015).
  * Pereda  _et al._ [2019a] M. Pereda, V. Capraro, and A. Sánchez, Group size effects and critical mass in public goods games, Scientific Reports 9, 5503 (2019a).
  * Pereda  _et al._ [2019b] M. Pereda, I. Tamarit, A. Antonioni, J. A. Cuesta, P. Hernández, and A. Sánchez, Large scale and information effects on cooperation in public good games, Scientific Reports 9, 15023 (2019b).
  * Fischbacher  _et al._ [2001] U. Fischbacher, S. Gächter, and E. Fehr, Are people conditionally cooperative? Evidence from a public goods experiment, Economics Letters 71, 397 (2001).
  * Fischbacher and Gächter [2010] U. Fischbacher and S. Gächter, Social Preferences, Beliefs, and the Dynamics of Free Riding in Public Goods Experiments, American Economic Review 100, 541 (2010).
  * Li and Noussair [2023] T. Li and C. N. Noussair, Conditional cooperation and group size: experimental evidence from a public good game, Journal of the Economic Science Association 10.1007/s40881-023-00152-4 (2023).
  * Perc  _et al._ [2013] M. Perc, J. Gómez-Gardeñes, A. Szolnoki, L. M. Floría, and Y. Moreno, Evolutionary dynamics of group interactions on structured populations: a review, Journal of The Royal Society Interface 10, 20120997 (2013), publisher: Royal Society.
  * Perc  _et al._ [2017] M. Perc, J. J. Jordan, D. G. Rand, Z. Wang, S. Boccaletti, and A. Szolnoki, Statistical physics of human cooperation, Statistical physics of human cooperation 687, 1 (2017).
  * Gokhale and Traulsen [2010] C. S. Gokhale and A. Traulsen, Evolutionary games in the multiverse, Proceedings of the National Academy of Sciences 107, 5500 (2010), publisher: Proceedings of the National Academy of Sciences.
  * Han  _et al._ [2012] T. A. Han, A. Traulsen, and C. S. Gokhale, On equilibrium properties of evolutionary multi-player games with random payoff matrices, Theoretical Population Biology 81, 264 (2012).
  * Duong and Han [2016] M. H. Duong and T. A. Han, Analysis of the expected density of internal equilibria in random evolutionary multi-player multi-strategy games, Journal of Mathematical Biology 73, 1727 (2016).
  * Szolnoki  _et al._ [2012] A. Szolnoki, M. Perc, and M. Perc, Conditional strategies and the evolution of cooperation in spatial public goods games, Physical Review E 85, 026104 (2012).
  * Chaudhuri [2011] A. Chaudhuri, Sustaining cooperation in laboratory public goods experiments: a selective survey of the literature, Experimental Economics 14, 47 (2011).
  * Battu and Srinivasan [2020] B. Battu and N. Srinivasan, Evolution of conditional cooperation in public good games, Royal Society Open Science 7, 191567 (2020).
  * Richerson and Boyd [2006] P. J. Richerson and R. Boyd,  _Not by Genes Alone: How Culture Transformed Human Evolution_ (University Of Chicago Press, 2006).
  * Mesoudi [2009] A. Mesoudi, How cultural evolutionary theory can inform social psychology and vice versa., Psychological Review 116, 929 (2009), place: US Publisher: American Psychological Association.
  * Harsanyi [1967] J. C. Harsanyi, Games with Incomplete Information Played by “Bayesian” Players, I–III Part I. The Basic Model, Management Science 14, 159 (1967), publisher: INFORMS.
  * Aumann  _et al._ [1995] R. Aumann, M. Maschler, and R. Stearns,  _Repeated Games with Incomplete Information_ (MIT Press, 1995).
  * Forges [2012] F. Forges, Folk theorems for Bayesian (public good) games, in  _Thirteenth annual conference (PET12) of the Association for Public Economic Theory (APET)_ (Taipei, China, 2012) p. 30.
  * Achtziger and Alós-Ferrer [2014] A. Achtziger and C. Alós-Ferrer, Fast or Rational? A Response-Times Study of Bayesian Updating, Management Science 60, 923 (2014).
  * El-Gamal  _et al._ [1995] M. A. El-Gamal, D. M. Grether, and D. M. Grether, Are People Bayesian? Uncovering Behavioral Strategies, Journal of the American Statistical Association 90, 1137 (1995).