# SemEval-2021 Task 9: Fact Verification and Evidence Finding for Tabular Data
in Scientific Documents (SEM-TAB-FACTS)

Nancy X. R. Wang Diwakar Mahajan††footnotemark:  Marina Danilevsky Sara
Rosenthal
IBM Research
<EMAIL_ADDRESS>{dmahaja, mdanile<EMAIL_ADDRESS>Equal
Contribution Corresponding Author

###### Abstract

Understanding tables is an important and relevant task that involves
understanding table structure as well as being able to compare and contrast
information within cells. In this paper, we address this challenge by
presenting a new dataset and tasks that addresses this goal in a shared task
in SemEval 2020 Task 9: Fact Verification and Evidence Finding for Tabular
Data in Scientific Documents (SEM-TAB-FACTS). Our dataset contains 981
manually-generated tables and an auto-generated dataset of 1980 tables
providing over 180K statement and over 16M evidence annotations. SEM-TAB-FACTS
featured two sub-tasks. In sub-task A, the goal was to determine if a
statement is supported, refuted or unknown in relation to a table. In sub-task
B, the focus was on identifying the specific cells of a table that provide
evidence for the statement. 69 teams signed up to participate in the task with
19 successful submissions to subtask A and 12 successful submissions to
subtask B. We present our results and main findings from the competition.

##  1 Introduction

Tables are ubiquitous in documents and presentations for conveying important
information in a concise manner. This is true in many domains, stretching from
scientific to government documents. In fact, surrounding text in these
articles are often statements summarizing or highlighting some information
derived from the primary source of data in tables. A relevant example is shown
in Figure 1 from a Business Insider article analyzing the impact of Covid-19
Aylin Woodward and Gal (2020). Describing all the information provided in this
table in a readable manner would be lengthy and considerably more difficult to
understand. Despite their importance, popular question answering (e.g. SQuAD
and Natural Question Rajpurkar et al. (2016); Kwiatkowski et al. (2019)) and
truth verification tasks (e.g. SemEval-2019 Fact Checking Task Mihaylova et
al. (2019)) have not focused on tables, being composed solely of written text.
This is likely due to their complexity to parse and understand, despite their
rich amount of information.

Figure 1: Surrounding text often highlights some information from the table
but does not capture all data. Alternately, the linked text may be subjective
or even misleading without the original table to check the claims.

Further, the structure of tables allows much more information to be presented
in an efficient manner as humans can interpret meaning in the spatial
relationship between cells. However, due to their challenging nature, recent
algorithms have been less successful at extracting Hoffswell and Liu (2019)
and understanding header and data structure in tables Cafarella et al. (2018).
In addition, any hierarchical and nested headers (common in printed documents)
increases the difficulty in interpreting data cells, as shown in Figure 2.

In this paper, we propose to bridge this gap with statement verification and
evidence finding using tables from scientific articles. This important task
promotes proper interpretation of the surrounding article. In fact, the
misunderstanding of tables can lead to the reporting of fake news that we see
as being all too prevalent today.

We present the first SemEval challenge to address table understanding. We
introduce a brand new dataset of 1980 tables from scientific articles that
addresses two challenging tasks important to table understanding:

A: Statement Fact Verification

Given a statement, determine whether it is supported, refuted or unknown
according to the table.

B: Cell Evidence Selection

Given a statement, select the cells in the table that provide evidence
supporting or refuting the statement.

The rest of this paper is formatted as follows: We first discuss related work.
We then present a new large table understanding dataset containing close to
2000 tables that is the first to provide evidence labels at the cell level for
statements and the first to focus on scientific articles. We provide a
detailed analysis of the dataset including several baseline results. We then
discuss the performance and approaches of the 19 participants in our challenge
and end with an aggregated analysis of participating teams. Finally, we
discuss future work.

Figure 2: A complex table sourced from East et al. (2018) with hierarchical
column and row structure. Additional difficulty follows from row hierarchy not
being delineated by separate columns.

##  2 Related Work

#### Natural Language Inference (NLI)

The table evidence task can be best understood as a variation of the natural
language inference task Dagan et al. (2005), but on tabular data. NLI asks
whether one (or more) sentence entails, refutes, or is unrelated to another
sentence; our framing asks whether a given table entails, refutes, or is
unrelated to a sentence. Several datasets have been created for studying NLI,
such as SNLI Bowman et al. (2015), MultiNLI Williams et al. (2018), and
SciTail Khot et al. (2018).

#### Table QA

This task is also closely related to the problem of search and question
answering on tables. The closest example would be, given a table that is known
to contain the relevant information, return cell values that answer a natural
language question Pasupat and Liang (2015). A variation requires analyzing a
collection of tables rather than a single one, along with the natural language
question Sun et al. (2016). Two of the most recent works are TAPAS Herzig et
al. (2020) and TaBERT Yin et al. (2020), which jointly pre-train over textual
and tabular data to facilitate table QA. However, such approaches have
previously focused on traditional natural language questions (“What is the
population of France?”) rather than inference statements (“France has the
highest population in Europe”), which may be entailed, refuted or unknowable
from the given table.

#### Related Datasets

The works closest to our dataset are TabFact Wenhu Chen and Wang (2020) and
INFOTABS Gupta et al. (2020). Both datasets were sourced from Wikipedia tables
and contain hypothesis and premise pairs. TabFact has entailment and refute
hypothesis types while INFOTABS has an additional “neutral” hypothesis
category, much like our “unknown” statements. Both works show that neural
models still lag far behind human performance for the fact checking task with
tables.

While both datasets have been great at kindling interest in fact verification
with tabular data, our dataset differs in two key aspects. First, we source
from scientific articles in a variety of domains rather than Wikipedia
infoboxes. Scientific tables have very specialized vocabulary and can be more
difficult to interpret. Additionally, scientific tables have much more complex
structure, like hierarchical column and row headers, rendering the assumption
that the first column/row is the header unhelpful. Finally, tables are often
directly referenced in scientific text unlike Wikipedia tables that are
generally stand-alone. This creates an opportunity to leverage natural
statements that depict the original author’s style and intent. The second key
differentiator of SEM-TAB-FACTS is the accompanying evidence annotations. We
believe the future of fact verification and AI in general will be in
cooperation with humans rather than in replacement. Thus, it is essential that
models are able to present explanations for decisions on the relationship
between the statement and table by showing the most relevant cells in a
potentially very large table.

Source | #Tables | #Entailed | #Refuted | #Unknown | #Relevant | #Irrelevant
---|---|---|---|---|---|---
Train Crowdsourced | 981 | 2,818 | 1,688 | 0 | 0 | 0
Train Auto-generated | 1,980 | 92,136 | 87,209 | 0 | 1,039,058 | 15,467,957
Development | 52 | 250 | 213 | 93 | 3,048 | 2,8495
Test | 52 | 274 | 248 | 131 | 3,458 | 26,724
Table 1:  Statistics for our SEM-TAB-FACTS dataset.

##  3 Dataset Details

Our dataset consists of two forms of generation: (1) a crowdsourced dataset,
and (2) an auto-generated dataset. Table 1 presents the statistics of the
dataset. We detail our dataset creation process in the following sections.

###  3.1 Data extraction and preprocessing

We sourced our tables from scientific articles belonging to active journals
that are currently being published by Elsevier and are available on
ScienceDirect111https://www.elsevier.com/__data/promis_misc/sd-
content/journals/jnlactive.xlsx. We utilized Elsevier ScienceDirect
APIs222https://dev.elsevier.com/sd_apis.html to scrape scientific articles
which belong to this list, and satisfy the following criteria: (1) the article
is open-access333https://www.elsevier.com/open-access, (2) the article is
available under “Creative Commons Attribution 4.0 (CC-BY)” user
license444https://www.elsevier.com/about/policies/open-access-licenses/user-
licences, and (3) the article has at least one table. We downloaded 1,920
articles belonging to 722 journals which contained 6,773 tables. We further
filtered out complicated tables (e.g. multiple tables in a single table) using
hand-written rules to get a set of 2,762 candidate tables from 1,085 articles
for annotation. We also extracted sentences mentioning the table within the
scientific article as candidate statements, which are corrected and then
labeled manually by the annotators.

Figure 3: Sample crowd-sourced statements for one table (sourced from Carney
et al. (2020)). Please note that these are the original statements without any
further corrections nor rephrasing.

###  3.2 Crowdsourced labeling

The manually generated statements were collected using the crowdsourcing
platform Appen555https://appen.com/. We collected five entailed and five
refuted statements for each table from the business preferred operators (BPO)
on Appen. The BPO crowd is composed of employees hired by Appen on an hourly
basis at a constant pay rate determined by Appen. We found that the workers
were much more motivated for the task as they were able to ask questions if
needed and we were also able to provide direct feedback to the workers. We
initially attempted generating statements with workers from the Appen open-
crowd, which is on-demand, but the quality was very poor as it was hard to
automatically validate naturally generated statements. Our instructions
explicitly lay out 7 types of statements and ask that workers attempt to make
one of each type. We encourage the use of different sets of cells whenever
possible. The types of statements are aggregation, superlative, count,
comparative, unique, all and usage of caption or common sense knowledge. These
are derived from the INFOTABS analysis Gupta et al. (2020). We asked workers
to avoid subjective adjectives like “best”, “worst”, “seldom” and look-up
statements that only require reasoning with one cell. The pay for each
statement set was 75 cents. In total, we collected 10000 statements for 1000
unique tables. See Figure 3 for an example table with its manually generated
and natural statements.

Additionally, for our training data, we conducted a verification task to check
for grammatical issues and doubly verify the statement label for both the
generated and natural in-text statements. The verification task was paid at 3
cents per statement, which equates to 30 cents per table. We restricted the
verification task to the workers in the open-crowd from English speaking
countries. After verification, we only preserved the statements that were
verified to be grammatically correct and the new label matched the original
label. Natural statements were also verified in the same process. Although
natural statements were generally factually correct, they were sometimes not
able to be verified by the referenced table. Additionally, these statements
often required rewording to ensure that all parts of the statement can be
verified by the table, which was a step taken only for the development and
test sets. This left us with 981 tables and 4506 statements. The majority of
the removals were due to grammatical errors as most BPO workers are not native
English speakers. See Table 1 (first row) for detailed statistics of the
crowd-sourced training set.

We initially attempted to collect the development and test sets as well as
evidence annotations via the same method as the training set. However, we
found that the quality was not gold-level and thus we (three of the authors)
decided to manually correct the statements and annotate the evidence
ourselves. All authors first annotated a small set of 102 statements to test
inter-annotator agreement for statement relationship and evidence labeling.
Out of 102 statements, we found 5 statements where at least one of three
annotators disagreed on the relationship and a further 5 statements where the
relationship was agreed but the evidence annotation differed. The other 92
were in complete agreement, indicating high agreement. Therefore, the
annotations for the rest of the dev set were annotated by just one person. The
test set was annotated fully by one author and the two other authors checked
the annotations with all disagreements being resolved. See Figure 4 for a
screenshot of the statement annotation correction and evidence annotation
interface. See the third and fourth rows of Table 1 for detailed statistics of
the dev and test sets.

Figure 4: Screenshot showing the labeling interface for statement rephrasing,
relationship labeling and evidence annotation.

###  3.3 Automatically generated statements

Input |  Template |  Evidence |  Example Statements
---|---|---|---
col_i, col_j |  ‘The’ + col_i_head + ‘is’ + col_i_val + ‘, when the’ + col_j_head + ‘is’ + col_j_val |  col_i_head, col_j_head, col_i_val, col_j_val |  The Code is AG3 when the Locality is Los Aguanances3.
col |  col_val + ‘is in’ + col_head |  col_val, col_head for entailed; col for refuted |  AG3 is in Code.
col |  unique or same values |  col for entailed; None for refuted |  Sup./Inf. has the same values.
col[#] |  ‘The maximum of’ + col_head +‘is’+val |  col[#] for entailed; None for refuted |  The maximum of Length(mm) is 2.22.
col[#] |  ‘The minimum of’ + col_head +‘is’+val |  col[#] for entailed; None for refuted |  The minimum of Length(mm) is 1.54.
col[#] |  ‘The mean of’ + col_head + ‘is’ + val |  col[#] |  The mean of Length(mm) is 1.83.
col[#] |  ‘The median of’ + col_head + ‘is’ + val |  col[#] |  The median of Length(mm) is 1.73.
col[#] |  ‘The mode of’ + col_head + ‘is’ + val |  col[#] |  The mode of Length(mm) is 1.54, 1.73, 2.22.
Table 2:  Template and evidence rules used for auto-generated ground truth.
The examples are derived from Table 4 in Figure 4.

IBM Watson™ Discovery666https://www.ibm.com/cloud/watson-discovery is an AI-
powered search and text analytics engine for extracting answers from complex
business documents. One of the available functionalities is a Table
Understanding service that produces a detailed enrichment of table data within
an html document. We use this service to identify the body and header cells,
as well as the _cell relationships_ , within our dataset. We then proceed to
use a set of templates to automatically create statements about each table. We
begin by identifying which cells and columns are numeric and non-numeric using
a simple regex. Unlike non-numeric cells, numeric cells and columns are
appropriate for specific templates that expect numeric values, such as ‘Value
[V] is the maximum of Column [C]’, where every value in column [C] has been
identified as numeric. We also generate evidence for some of these templates.
The template and evidence generation rules along with their inputs are
detailed in Table 2. This process generated 3,512,978 statements from 1,980
tables which were highly skewed in favor of refuted statements. This dataset
was then down-sampled to a maximum of 50 statements per table while ensuring a
more even distribution between the two classes to form our final released
auto-generated dataset. The full statistics for the auto-generated training
data is shown in the second row of Table 1.

##  4 Evaluation Metrics

###  4.1 Task A: Statement Fact Verification

The goal of task A is to determine if a statement is entailed or refuted by
the given table, or whether, as is in some cases, this cannot be determined
from the table. We show two evaluation results. The first is a standard 3-way
Precision / Recall / F1 micro evaluation of a multi-class classification that
evaluates whether each table was classified correctly as Entailed / Refuted /
Unknown. This tests whether the classification algorithm understands cases
where there is insufficient information to make a determination. The second,
simpler evaluation, uses the same P/R/F1 metric but is a 2-way classification
that removes statements with the “unknown” ground truth label from the
evaluation. The 2-way metric still penalizes misclassifying refuted/ entailed
statement as unknown.

###  4.2 Task B: Cell Evidence Selection

In Task B, the goal is to determine for each cell and each statement, if the
cell is within the minimum set of cells needed to provide evidence for the
statement (“relevant”) or not (“irrelevant”). In other words, if the table
were shown with all other cells blurred out, would this be enough for a human
to reasonably determine that the table entails or refutes the statement? The
evaluation calculates the recall and precision for each cell, with “relevant”
cells as the positive category. For some statements, there may be multiple
minimal sets of cells that can be used to determine statement entailment or
refusal. In such cases, our dataset contains all of these versions. We compare
the prediction to each ground truth version and count the highest score.

##  5 Experiments

We present our baseline experimental setup for each task below.

#### Task A

We employ state-of-the-art Table-BERT
implementation777https://github.com/wenhuchen/Table-Fact-Checking as proposed
by Wenhu Chen and Wang (2020). We utilize Table-BERT’s best performing
configuration (Table-BERT-Horizontal-T+F-Template) as (1) using entity-linking
to find the relevant columns for a statement, (2) flattening the table by
scanning horizontally to form natural statements from the relevant columns and
their cell values and (3) classifying the flattened table and the statement
using the sentence pair classification setting in BERT. To overcome the lack
of unknown statements in our dataset, we supplement each table with randomly
chosen statements from other tables. In Table-BERT, if the entity linking
results in no matches, the flattened table is marked as [UNK]. As our dataset
contains unknown statements, in such cases we consider all columns to be a
match and flatten the entire table.

Using the above process, we perform the following experiments (1) apply the
Table-BERT model out-of-the-box (2) re-train Table-BERT model with unknown
statement and apply on our test data (3) fine-tune the model in (2) with our
manual+auto-generated data and apply on our test data. We also compare these
experiments with a majority baseline with entailed as our majority class. The
results are presented in Table 3. Applying Table-BERT model out-of-the-box
provides some improvement over a majority-baseline. However, when the model is
retrained with previously missing unknown statements, the performance improves
for three-way classification. Further fine-tuning the model with our training
dataset (both manual and auto-generated) provides the best performance on the
two-way F1-score.

Experiment | Test
---|---
2-way | 3-way
majority-baseline | 52.42 | 42.16
original Table-BERT | 56.77 | 45.58
re-trained Table-BERT | 52.96 | 48.33
\+ FT with SEM-TAB-FACTS | 56.81 | 48.24
Table 3:  Task A baseline results using F1-score.

#### Task B

We present the following two baselines for Task B: (1) a random baseline where
each cell is marked relevant or irrelevant randomly (2) a simple word-match-
based baseline where a cell is marked relevant if it overlaps with the
statement. The baseline results are presented in Table 4.

Experiment | Dev | Test
---|---|---
random-baseline | 21.18 | 20.47
word-match | 49.53 | 47.39
Table 4:  Task B baseline results using F1-Score Team | 3-way F-Score | 2-way F-Score
---|---|---
Official Leaderboard
King001 | 84.48 | 88.74
THiFly_Queen | 83.76 | 84.55
RyanStark | 81.51 | 87.22
sattiy | 77.32 | 84.96
BreakingBERT@IITK | 69.31 | 76.81
Volta | 67.34 | 72.89
TAPAS | 66.81 | 73.13
AttesTable | 65.59 | 71.72
Yaoxu | 60.76 | 75.8
Beary-group | 58.37 | 72.56
ok-team | 57.79 | 71.84
SUNLP | 47.92 | 59.58
FishToucher | 41.83 | 52.01
KaushikAcharya | 36.23 | 23.08
Unverified Leaderboard
Skywalker | 92.55 | 95.15
MagicPai | 90.88 | 94.03
endworld | 82.35 | 88.16
Paima | 81.96 | 88.85
ravikranc | 57.90 | 71.99
Table 5:  Task A Leaderboard

##  6 Competition Results

We present two leaderboards for each task888We made the assumption that teams
would not make any use of the test data, as is usually the case for algorithm
evaluation, but we did not make this explicit ahead of time and some teams did
not realize this was an issue. We decided to have two leaderboards to have a
fair comparison for all teams.. The official leaderboard is from participants
who have given us detailed descriptions on their system and affirmed that they
did not incorporate any information from the test set that changed their final
model. This is a more accurate representation of system quality. The
unverified leaderboard is composed of participants who either did not give
enough detail or have affirmed that they incorporated some test data
information in their final model. The participants did not have access to
labels for test data but some teams altered their models upon examining the
input data in the test set. Although we discouraged this approach, we present
the results in hopes it can give some interesting information about how much
improvement might be possible with having access to input test data.

19 teams participated in Task A. Of the 14 teams on the official leaderboard,
King001 obtained the highest score for task A for both the 2-way (88.74) and
3-way (84.48) F-scores. However, the top three participants have comparable
scores. All teams except for the last two beat our best baseline in Table 3.
The unverified leaderboard includes 5 teams and contains higher scores thank
in the official leaderboard. However, due to the reasons outlined above, we
cannot say with certainty that the results are reproducible. The full
leaderboard results for all participants are in Table 5.

Task B is a much harder task and fewer teams participated in this challenge.
Of the 12 teams that participated, 8 are in the official leaderboard. The best
score is 65.17 by BreakingBERT@IITK(65.17) which is noticeably lower than the
F-scores in Task A. Similarly to Task A the results in the unverified
leaderboard are considerably higher. The full leaderboard results for all
participants are in Table 6.

Team | F-Score
---|---
Official Leaderboard
BreakingBERT@IITK | 65.17
Volta | 62.95
King001 | 62.14
FishToucher | 60.06
RyanStark | 54.96
Sattiy | 48.56
AttesTable | 43.02
KaushikAcharya | 33.81
Unverified Leaderboard
MagicPai | 88.74
SkyWalker | 73.05
endworld | 57.85
Paima | 51.97
Table 6:  Task B Leaderboard

We summarize the system details for all participating teams in Tables 7 (Task
A) and 8 (Task B). In general, deep learning was the most popular approach
used by the participants e.g. BiLSTM with attention, BERT Devlin et al. (2019)
etc. Most of the participants used transformer-based models to train their
systems with flavors ranging from general-domain BERT Devlin et al. (2019) to
table-understanding specific versions like TAPAS Herzig et al. (2020), TaBERT
Yin et al. (2020) and Table-BERT Wenhu Chen and Wang (2020). One third of the
participants employed some form of ensembling technique in their submission.

Team |  Description
---|---
AttesTable Varma et al. (2021) |  Extended TAPAS to 3 classes by fine-tuning it. Employed a novel way of synthesizing “unknown” samples.
BreakingBERT@IITK Jindal et al. (2021) |  Ensemble models with TAPAS and TableBERT Transformers in a hierarchical two-step method for 3-way classification (unknown vs not unknown first)
Beary-group |  Used TAPAS model with TabFact task, and added unique features. Employed prepossessing tricks like k-fold validation and replacing the characters and did hyperparameter tuning.
BOUN Köksal et al. (2021)* |  Used text augmentation techniques such as back translation and synonym swapping on the TAPAS model. Domain adaptation and joint learning using SemTabFacts and TabFact datasets.
endworld |  Data Cleaning. Ensemble combining 80 instances of trained TaPas-Large and label smoothing.
FishToucher |  Motivated by TaPas, used BERT and enriched the embedding layer with two new token type embeddings: row and column ids* (*The team mistakenly submitted an old model version, see paper for more accurate scores)
Kaushik Acharya Acharya (2021) |  Parsed statements into candidate logical form; mapped result to handwritten rules, to then execute over relevant cells (identified using string matching and universal dependency parsing)
King001 |  Trained 20 instances of TaPas, SAT and Table-Bert for an ensemble of 60 models. Used preprocessing like acronym completion, rules to align the table content with the question content, label smoothing.
MagicPai |  Multi-model training using models such as TaBERT, tapas_wikisql, tapas_TabFact, tapas_masklm. Finally rule amendments and aligning the distribution of training and test data
ok-team |  TAPAS pretrained on TabFact with preprocessing of data (like transforming English numerals to Arabic numerals, removing special characters etc.)
Paima |  Fine-tuned TAPAS optimized to perform window scanning on statement-related table data. Pre-processing to reduce abbreviations for table headers, and identifying operation expressions.
RyanStark |  Multi-model TaBERT pretrained Model fusion. Pre-processing such as case and abbreviations.
Sattiy Ruan et al. (2021) |  Ensemble of 6 fine-tuned pre-trained models on the augmented data with content snap-shot input. Augmented the data provided by expanding the labels. Used Fast Gradient Method and added disturbance to the embedding layer to obtain a more stable word representation and a more general model.
SkyWalker |  Deep learning, LPA rules, TAPAS dataset
SUNLP |  BERT for sequence classification, transfer learning
TAPAS Müller et al. (2021) |  Ensemble of TAPAS (BERT-large-like) models: trained with a Mask-LM task on Wikipedia tables, intermediate pre-training data and TabFact data. Hierarchical two-step method for 3-way classification. Added neutral statements during training: random and by removing one of the evidence columns.
THiFly_Queen Yuxuan et al. (2021) |  Ensemble models in a hierarchical two-step method. 8-model to identify unknown statements and 9-model ensemble to classify entailed/refuted. Incorporated different ensemble weights for various statement types (count, superlative, unique).
Volta Gautam et al. (2021) |  Finetuned TAPAS that was pretrained on TabFact. Pre-processing to standardize multiple header rows to a single header.
Yaoxu |  Added numeric and enumerate features to TAPAS and also statistic information (such as count) as a new row/column to the table.
Table 7: Descriptions of systems from participants for Task A. *Note: Team BOUN did not participate in the official leaderboard. Team |  Description
---|---
BreakingBERT @IITK |  An ensemble of an individual cell-based NLI approach and a similarity approach with the cells and statement
FishToucher |  BERT CLS tokens for statement and table cells are used to determine cell relationships to each other, and the statement (for relevant cells)
Kaushik Acharya |  Relevant cells are output as part of Task A
RyanStark |  BOW approach with rules applied based on word matches in header and data cells.
Volta |  Finetuned TAPAS for cell selection. Different models for entailed and refuted statements. Used transfer learning and header standardization.
Table 8: Descriptions of systems from participants for Task B (when provided)

Most of the participants have used the manually generated ground-truth in the
development of their systems, with only one team not finding it useful.
Further, a large percentage of participants have used the auto-generated
ground truth in their systems with three teams not finding it helpful in their
evaluation.

In terms of external resources, a majority of the participants used external
table understanding resources in their systems. Further, most of the
participants employed pre-processing techniques like acronym completion,
removing special characters, etc… A substantial percentage of participants
used techniques like incorporating word embeddings, entity resolution etc.
Finally, a large number of participants used TabFact Wenhu Chen and Wang
(2020) as an external dataset.

| Refuted | Entailed | Unknown
---|---|---|---
Refuted | 164 | 81 | 3
Entailed | 46 | 226 | 2
Unknown | 16 | 72 | 43
Table 9:  Task A average confusion matrix

We also conducted additional analyses on participant submissions on the
official leaderboard. We show through the average confusion matrix for Task A
in Table 9 that the Unknown label was the most difficult. In fact, there were
more unknown statements incorrectly labelled as entailed than were correctly
categorized. Naturally, the statements with the lowest accuracy ($<25\%$)
consist of mainly unknown statements, especially those statements that have
words overlapping with those in the table. Out of the entailed and refuted
statements, ones that require numerical reasoning, like range, count or
comparisons seemed to be most challenging. The statements with the highest
accuracy ($>95\%$) generally had most words or numbers exactly overlapping
with those in the table. In task B, out of the statements with less than 30%
evidence F-score, 86% were ones with a refuted relationship. Conversely, the
statements with greater than 70% F-score, 74% were ones with an entailed
relationship. This shows that it is more difficult to find the most direct
evidence to prove that a statement is refuted by a table than it is to show
the positive evidence that a particular statement is supported by it. We
believe this is an interesting line of research for future studies.

##  7 Conclusion and Future Works

In this paper, we presented the data and competition results for SEM-TAB-
FACTS, Shared Task 9 of SemEval 2021. We created a large dataset via automated
and crowdsourced fact verification as well as evidence finding for tables. Our
19 teams had a variety of techniques to tackle this unique but very relevant
problem. The evidence finding scores are still quite low and have a large
improvement potential. Additionally, the test set may be expanded in future
versions of this task with a combination of manually generated, natural, and
automated statements.

## References

  * Acharya (2021) Kaushik Acharya. 2021.  Kaushikacharya at SemEval-2021 task 9: Candidate generation for fact verification over tables.  In _Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021)_.
  * Aylin Woodward and Gal (2020) Ruobing Su Aylin Woodward and Shayanne Gal. 2020.  What to know about the coronavirus outbreak in 17 charts and maps.  _Business Insider_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Cafarella et al. (2018) Michael Cafarella, Alon Halevy, Hongrae Lee, Jayant Madhavan, Cong Yu, Daisy Zhe Wang, and Eugene Wu. 2018.  Ten years of webtables.  _Proceedings of the VLDB Endowment_ , 11(12):2140–2149.
  * Carney et al. (2020) Richard W Carney, Travers Barclay Child, and Xiang Li. 2020.  Board connections and crisis performance: Family, state, and political networks.  _Journal of Corporate Finance_ , 64:101630.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The PASCAL recognising textual entailment challenge.  In _Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment_ , MLCW’05, page 177–190, Berlin, Heidelberg. Springer-Verlag.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * East et al. (2018) Katherine East, Sara C Hitchman, Ioannis Bakolis, Sarah Williams, Hazel Cheeseman, Deborah Arnott, and Ann McNeill. 2018.  The association between smoking and electronic cigarette use in a cohort of young people.  _Journal of Adolescent Health_ , 62(5):539–547.
  * Gautam et al. (2021) Devansh Gautam, Kshitij Gupta, and Manish Shrivastava. 2021.  Volta at SemEval-2021 task 9: Statement verification and evidence finding with tables using TAPAS and transfer learning.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_.
  * Gupta et al. (2020) Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020.  INFOTABS: Inference on tables as semi-structured data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2309–2324, Online. Association for Computational Linguistics.
  * Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020.  TaPas: Weakly supervised table parsing via pre-training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4320–4333, Online. Association for Computational Linguistics.
  * Hoffswell and Liu (2019) Jane Hoffswell and Zhicheng Liu. 2019.  Interactive repair of tables extracted from pdf documents on mobile devices.  In _ACM Human Factors in Computing Systems (CHI)_.
  * Jindal et al. (2021) Aditya Jindal, Ankur Gupta, Jaya Srivastava, Preeti Menghwani, Vijit Malik, Vishesh Kaushik, and Ashutosh Modi. 2021.  Breakingbert@iitk at SemEval-2021 task 9: Statement verification and evidence finding with tables.  In _Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021)_.
  * Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.  Scitail: A textual entailment dataset from science question answering.  In _AAAI Conference on Artificial Intelligence_.
  * Köksal et al. (2021) Abdullatif Köksal, Yusuf Yüksel, Bekir Yıldırım, and Arzucan Özgür. 2021.  BOUN at SemEval-2021 Task 9: Text Augmentation Techniques for Fact Verification in Tabular Data.  In _Proceedings of the Fifteenth Workshop on Semantic Evaluation_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Mihaylova et al. (2019) Tsvetomila Mihaylova, Georgi Karadzhov, Pepa Atanasova, Ramy Baly, Mitra Mohtarami, and Preslav Nakov. 2019.  SemEval-2019 task 8: Fact checking in community question answering forums.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 860–869, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Müller et al. (2021) Thomas Müller, Julian Martin Eisenschlos, and Syrine Krichene. 2021.  TAPAS at SemEval-2021 task 9: Reasoning over tables with intermediate pre-training.  In _Proceedings of the Fifteenth Workshop on Semantic Evaluation_. International Committee for Computational Linguistics.
  * Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015.  Compositional semantic parsing on semi-structured tables.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1470–1480, Beijing, China. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ruan et al. (2021) Xiaoyi Ruan, mei Jin, Jian Ma, Lianxin Jiang, Mo Yang, and Jianping Shen. 2021.  Sattiy at SemEval-2021 task 9: Method for statement verification and evidence finding with tables based on multi-model ensemble.  In _Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021)_.
  * Sun et al. (2016) Huan Sun, Hao Ma, Xiaodong He, Wen-tau Yih, Yu Su, and Xifeng Yan. 2016.  Table cell search for question answering.  In _Proceedings of the 25th International Conference on World Wide Web_ , WWW ’16, page 771–782, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Varma et al. (2021) Harshit Varma, Aadish Jain, Pratik Ratadiya, and Abhishek Rathi. 2021.  Attestable at SemEval-2021 task 9: Extending statement verification with tables for unknown class, and semantic evidence finding.  In _Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021)_.
  * Wenhu Chen and Wang (2020) Jianshu Chen Yunkai Zhang Hong Wang Shiyang Li Xiyou Zhou Wenhu Chen, Hongmin Wang and William Yang Wang. 2020.  Tabfact : A large-scale dataset for table-based fact verification.  In _International Conference on Learning Representations (ICLR)_ , Addis Ababa, Ethiopia.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020.  TaBERT: Pretraining for joint understanding of textual and tabular data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8413–8426, Online. Association for Computational Linguistics.
  * Yuxuan et al. (2021) Zhou Yuxuan, Zhou Kaiyin, Liu Xien, Wu Ji, and Zhu Xiaodan. 2021.  Thifly_queen at SemEval-2021 task 9: Statement verification and evidence finding with tables.  In _Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021)_.


2\pi}\int_{y_{*}}^{y_{*}+1}dy\int_{\tilde{x}_{*}+ky}^{\tilde{x}_{*}+ky+m}dx\,\phi_{0}(x,y)\partial_{x}\partial_{y}\phi(x,y)$
(C.4) $\displaystyle-{i\over
2\pi}\int_{\tilde{x}_{*}+ky_{*}}^{\tilde{x}_{*}+ky_{*}+m}dx\,g_{\tilde{X}}^{0}(x,y_{*})\partial_{x}\phi(x,y_{*})$
$\displaystyle+{i\over
2\pi}\int_{y_{*}}^{y_{*}+1}ds\,\left[g_{X}^{0}(\tilde{x}_{*}+ks,s)k\partial_{x}\phi(\tilde{x}_{*}+ks,s)+\partial_{y}g_{X}^{0}(s)\,\phi(\tilde{x}_{*}+ks,s)\right]\Big{]}$

In the special case of an untwisted torus, $k=0,m=1$, and this operator
reduces to

$\displaystyle{\cal U}(\phi_{0})$ $\displaystyle=\exp\Big{[}{i\over
2\pi}\int_{x_{*}}^{x_{*}+1}dx\int_{y_{*}}^{y_{*}+1}dy\,\phi_{0}(x,y)\partial_{x}\partial_{y}\phi(x,y)$
(C.5) $\displaystyle-{i\over
2\pi}\int_{y_{*}}^{y_{*}+1}dy\,g_{X}^{0}(x_{*},y)\partial_{y}\phi(x_{*},y)-{i\over
2\pi}\int_{x_{*}}^{x_{*}+1}dx\,g_{Y}^{0}(x,y_{*})\partial_{x}\phi(x,y_{*})$
$\displaystyle+in_{xy}^{0}\,\phi(x_{*},y_{*})\Big{]}\,.$

It is straightforward to check that this operator satisfies the following
properties:

  * •

It is independent of the reference point $(x_{*},y_{*})$.

  * •

It is symmetric under exchange of $\phi$ and $\phi_{0}$.

  * •

It is invariant under the gauge transformation of $\phi_{0}$:

$\displaystyle\phi_{0}(x,y)\sim\phi_{0}(x,y)+2\pi n^{x}_{0}(x)+2\pi
n^{y}_{0}(y)\,,$ (C.6) $\displaystyle
n^{x}_{0}(x)\,,n^{y}_{0}(y)\in\mathbb{Z}\,.$

  * •

Since this operator is symmetric in $\phi\leftrightarrow\phi_{0}$, it is also
invariant under the gauge transformation of $\phi$:

$\displaystyle\phi(x,y)\sim\phi(x,y)+2\pi n^{x}(x)+2\pi n^{y}(y)\,,$ (C.7)
$\displaystyle n^{x}(x)\,,n^{y}(y)\in\mathbb{Z}\,.$

  * •

If $\partial_{x}\partial_{y}\phi_{0}=0$, the operator ${\cal U}(\phi_{0})$
depends only on the transition functions of $\phi$, where we have used the
second property above. Therefore, it is a conserved operator in the
$\phi$-theory.

When proving some of these statements, we drop integers of the form $\int
dxg(x)\partial_{x}n(x)$ for some integer-valued functions $g(x),n(x)$ in the
exponent of ${\cal U}(\phi_{0})$.

We are now ready to discuss the most general conserved winding charges, which
are ${\cal U}(\phi_{0})$ with $\partial_{x}\partial_{y}\phi_{0}=0$. As
discussed in Section 3.2.1, the most general such $\phi_{0}$ takes the form

$\displaystyle\phi_{0}={2\pi r\over
m}\left(\Theta^{\textrm{P}}(x,0,1)-k\Theta^{\textrm{P}}(y,0,1)\right)+f^{x}(x)+f^{y}(y)$
(C.8)

with $r$ an integer modulo $m$ and $f^{i}(x^{i}+1)=f^{i}(x^{i})$.

The most general winding charge is therefore

$\displaystyle{\cal U}(\phi_{0})=$ $\displaystyle\exp\Big{[}-{ir\over
m}\sum_{I=0}^{m-1}g_{\tilde{X}}(I,0)+{ir\over
m}\sum_{J=0}^{k-1}g_{X}(J,0)\Big{]}$ (C.9) $\displaystyle\times$
$\displaystyle\exp\left[{i}\oint dxf^{x}(x)Q^{x}(x)+i\oint
dyf^{y}(y)Q^{y}(y)\right]\,.$

where we have set $x_{*}=y_{*}=0$ for simplicity. We have thus unified the
$U(1)$ charges (LABEL:U1charges) and the $\mathbb{Z}_{m}$ charge (3.16) into a
single general winding operator ${\cal U}(\phi_{0})$.

The analogous operators associated with the momentum symmetry of $\phi$ can be
described using the dual field $\phi^{xy}$. (See [10] for details on the self-
duality of the $\phi$-theory.) More explicitly, these operators are given by
(LABEL:Uphi0) with $\phi$ replaced by $\phi^{xy}$. They shift $\phi$ by
$\phi_{0}$.

##  Appendix D Wilson operators of the $\mathbb{Z}_{N}$ theory

In this appendix, we construct the most general gauge-invariant Wilson
operator in the $\phi-A$ presentation of the $2+1$-dimensional
$\mathbb{Z}_{N}$ gauge theory on a twisted torus. For simplicity, we will set
$\ell_{x}^{\textrm{eff}}=\ell_{y}^{\textrm{eff}}=1$ in this appendix.

We follow a reasoning similar to that in Appendix C. We start with a
background profile circle-valued function $\phi_{0}(x,y)$ and attempt to
define

$\displaystyle``\exp\left[{iN\over
2\pi}\int_{T^{2}}\phi_{0}(x,y)A_{xy}(x,y)\right]"$ (D.1)

However, such an expression is generally not gauge-invariant, and depends on
the choice of the fundamental domain of the torus.

To remedy these issues, we define the following operator in a similar spirit
as in Appendix C:

$\displaystyle{\cal U}(\phi_{0})=\exp\Big{[}{iN\over
2\pi}\int_{y_{*}}^{y_{*}+1}dy\int_{\tilde{x}_{*}+ky}^{\tilde{x}_{*}+ky+m}dx\,\phi_{0}(x,y)A_{xy}(x,y)$
(D.2) $\displaystyle-{i\over
2\pi}\int_{\tilde{x}_{*}+ky_{*}}^{\tilde{x}_{*}+ky_{*}+m}dx\,g_{\tilde{X}}^{0}(x,y_{*})\partial_{x}\phi(x,y_{*})$
$\displaystyle-{i\over
2\pi}\int_{y_{*}}^{y_{*}+1}ds\,\left[g_{X}^{0}(\tilde{x}_{*}+ks,s)\partial_{y}\phi(\tilde{x}_{*}+ks,s)+k\partial_{x}g_{X}^{0}(\tilde{x}_{*}+ks,s)\,\phi(\tilde{x}_{*}+ks,s)\right]$
$\displaystyle+in^{0}_{x\tilde{x}}\,\phi(\tilde{x}_{*}+ky_{*},y_{*})\Big{]}$

When the equation of motion $A_{xy}={1\over N}\partial_{x}\partial_{y}\phi$ is
imposed, it becomes the operator ${\cal U}(\phi_{0})$ in the $\phi$-theory
(LABEL:Uphi0). A similar calculation shows that ${\cal U}(\phi_{0})$ is
independent of the choice of the reference point $x_{*},y_{*}$. For
simplicity, we will set $x_{*}=y_{*}=0$ from now on.

It is clearly invariant under $\phi\sim\phi+2\pi n^{x}(x)+2\pi n^{y}(y)$.
Under a gauge transformation $\phi\sim\phi+N\alpha,A_{xy}\sim
A_{xy}+\partial_{x}\partial_{y}\alpha$, this operator picks up a factor:

$\displaystyle\exp\Big{[}{iN\over
2\pi}\int_{y_{*}}^{y_{*}+1}dy\int_{\tilde{x}_{*}+ky}^{\tilde{x}_{*}+ky+m}dx\,\alpha(x,y)\partial_{x}\partial_{y}\phi_{0}(x,y)$
(D.3) $\displaystyle-{iN\over
2\pi}\int_{\tilde{x}_{*}+ky_{*}}^{\tilde{x}_{*}+ky_{*}+m}dx\,g_{\tilde{X}}^{\alpha}(x,y_{*})\partial_{x}\phi_{0}(x,y_{*})$
$\displaystyle-{iN\over
2\pi}\int_{y_{*}}^{y_{*}+1}ds\,\left[g_{X}^{\alpha}(\tilde{x}_{*}+ks,s)\partial_{y}\phi_{0}(\tilde{x}_{*}+ks,s)+k\partial_{x}g_{X}^{\alpha}(\tilde{x}_{*}+ks,s)\,\phi_{0}(\tilde{x}_{*}+ks,s)\right]$
$\displaystyle+i{N}n^{\alpha}_{x\tilde{x}}\,\phi_{0}(\tilde{x}_{*}+ky_{*},y_{*})\Big{]}$

where $g^{\alpha}_{\cal C}$ is the transition function and
$n^{\alpha}_{x\tilde{x}}$ is similarly defined . The condition for ${\cal
U}(\phi_{0})$ to be gauge invariant is

$\displaystyle\partial_{x}\partial_{y}\phi_{0}=0\,,~{}~{}{N\phi_{0}\over
2\pi}\in\mathbb{Z}\,.$ (D.4)

On top of these conditions, $\phi_{0}$ should still have
$2\pi\mathbb{Z}$-valued transition functions. As discussed in Section 4.1, the
most general such $\phi_{0}$ takes the form (4.3).

We now discuss several important examples of ${\cal U}(\phi_{0})$:

  * •

Consider

$\displaystyle\phi_{0}={2\pi\over
N}\left(\Theta^{\textrm{P}}(y,y_{1},1)-\Theta^{\textrm{P}}(y,y_{2},1)\right)\,.$
(D.5)

(Here, for simplicity, we assume $0<y_{2}-y_{1}\leq 1$.) This gives the Wilson
strip operator (LABEL:stripoperatorZN)

$\displaystyle\textbf{W}_{(y)}(y_{1},y_{2})=\exp\left[i\int_{y_{1}}^{y_{2}}dy\oint
dxA_{xy}(x,y)\right]\,.$ (D.6)

There is a similar choice of $\phi_{0}$ giving the Wilson strip that is
extended along the $y$ cycle. They generate the $\mathbb{Z}_{N}$ dipole global
symmetry of the $\mathbb{Z}_{N}$ gauge theory.

  * •

As another example, we can take $\phi_{0}$ to be:

$\displaystyle\phi_{0}={2\pi\over\text{gcd}(N,m)}\left(\Theta^{\textrm{P}}(x,0,1)-k\Theta^{\textrm{P}}(y,0,1)\right)\,.$
(D.7)

This leads to the $\mathbb{Z}_{\text{gcd}(N,m)}$ operator in (4.29):

$\displaystyle{\bf U}=$
$\displaystyle\exp\Big{[}\,{i}{N\over\text{gcd}(N,m)}\int_{0}^{1}dy\int_{ky}^{ky+m}dx\,\left(\Theta^{\textrm{P}}(x,0,1)-k\Theta^{\textrm{P}}(y,0,1)\right)A_{xy}(x,y)$
(D.8)
$\displaystyle-{i}{m\over\text{gcd}(N,m)}\int_{0}^{1}ds\,\partial_{y}\phi(ks,s)\,\Big{]}\,.$

Note that the first integral has no support in the rectangle $k-1<x<k,0<y<1$.

We have therefore unified the most general Wilson operators built from $A$ and
$\phi$ into ${\cal U}(\phi_{0})$ with different choices of $\phi_{0}$ obeying
(D.4). On a lattice, we have $L_{x}^{\rm eff}+L_{y}^{\rm eff}-1$ Wilson strip
operators $\textbf{W}_{(i)}$ (LABEL:stripoperatorZN) and one
$\mathbb{Z}_{\text{gcd}(N,M)}$ operator $\bf U$ (4.29).

##  Appendix E Winding configurations of $\hat{\phi}$

In this appendix, we place classical circle-valued fields $\hat{\phi}^{i(jk)}$
in the $\mathbf{2}$ of $S_{4}$ on a twisted three-torus. This, for example, is
the quantum field of the $\hat{\phi}$-theory of [11]. In contrast to the
parallel analysis for $\phi$ in Section 5, we will see that, on a lattice,
there is no new winding charge beyond those labeled by the $2(L_{x}^{\rm
eff}+L_{y}^{\rm eff}+L_{z})-3$ integer winding charges.

The $\hat{\phi}$-theory is dual to the $3+1$-dimensional $U(1)$ gauge theory
of $A$ [11], where the winding charges of $\hat{\phi}$ are mapped to the
electric charges of $A$. Similar to the analysis of the transition functions
in Section 4.2, there is no new electric charge in the $3+1$-dimensional gauge
theory of $A$ beyond those labeled by the $2(L_{x}^{\rm eff}+L_{y}^{\rm
eff}+L_{z})-3$ integers. Hence, the computation in this appendix provides
another check of the above duality.

Furthermore, the gauge parameters $\hat{\alpha}^{i(jk)}$ of the gauge field
$\hat{A}$ in [11, 12] are also in the $\mathbf{2}$ of $S_{4}$. The winding
configurations in this appendix were used as the large gauge transformations
(LABEL:hatgauge) in the gauge theory of $\hat{A}$ in Section 6.4.

We now proceed to analyze the winding charges of $\hat{\phi}^{i(jk)}$. The
winding charges obey $\partial_{i}\partial_{j}Q^{k}(x^{i},x^{j})=0$ with
$i\neq j\neq k$ and can be expressed as

$\displaystyle Q^{x}(y,z)={1\over 2\pi}\oint
dx\partial_{x}\hat{\phi}^{x(yz)}=W^{y}_{x}(y)+W^{z}_{x}(z)\,,$ (E.1)
$\displaystyle Q^{y}(z,x)={1\over 2\pi}\oint
dy\partial_{y}\hat{\phi}^{y(zx)}=W^{z}_{y}(z)+W^{x}_{y}(x)\,,$ $\displaystyle
Q^{z}(x,y)={1\over 2\pi}\oint
dz\partial_{z}\hat{\phi}^{z(xy)}=W^{x}_{z}(x)+W^{y}_{z}(y)\,.$

where $W^{i}_{j}(x^{i})\in\mathbb{Z}$. Pairwise they share a common zero mode:

$\displaystyle\left(\,W^{i}_{k}(x^{i}),W^{j}_{k}(x^{j})\,\right)\sim\left(\,W^{i}_{k}(x^{i})+1,W^{j}_{k}(x^{j})-1\,\right)$
(E.2)

Since $Q$’s are single-valued integer operators on the torus, the discussion
in Section 3.2.1 implies that
$W^{x}_{i}(x+\ell_{x}^{\textrm{eff}})=W^{x}_{i}(x),~{}W^{y}_{i}(y+\ell_{y}^{\textrm{eff}})=W^{y}_{i}(y),~{}W^{z}_{i}(z+\ell_{z})=W^{z}_{i}(z)$.

However, these integers are not all independent. To see this, consider the
following combination:

$\displaystyle\partial_{z}W^{z}_{y}(z)-\langle\tilde{X},Y\rangle\partial_{z}W^{z}_{x}(z)=$
$\displaystyle\partial_{z}Q^{y}-\langle\tilde{X},Y\rangle\partial_{z}Q^{x}$
(E.3) $\displaystyle=$ $\displaystyle{1\over 2\pi}\partial_{z}\oint
dy\partial_{y}\hat{\phi}^{y(zx)}-{\langle\tilde{X},Y\rangle\over
2\pi}\partial_{z}\oint dx\partial_{x}\hat{\phi}^{x(yz)}\,.$

We start by showing that the last expression can be rewritten as

$\displaystyle\hat{Q}^{y}(\hat{Y})\equiv{1\over 2\pi}\partial_{z}\oint_{\cal
C}\left(\partial_{x}\hat{\phi}^{y(zx)}dx+\partial_{y}\hat{\phi}^{y(zx)}dy\right)\,,$
(E.4)

where $\cal C$ is any curve homologous to $\hat{Y}\equiv
m\tilde{X}=Y+\langle\tilde{X},Y\rangle X$ (2.9). Note that (E.4) only depends
on the homology class of the curve $\cal C$, but not the explicit
representative. For any cycle $\cal S$, the charge $\hat{Q}^{y}({\cal S})$
computes the derivative of the winding charges of $\hat{\phi}^{y(zx)}$ along
$\cal S$ and takes the form

$\displaystyle\hat{Q}^{y}({\cal
S})=\sum_{\gamma}n_{\gamma}\delta^{\textrm{P}}(z,z_{\gamma},\ell_{z})\,,~{}~{}~{}~{}n_{\gamma}\in\mathbb{Z}\,.$
(E.5)

Let us choose $\cal C$ to be a curve that first goes around the $Y$ cycle once
and then goes around the $X$ cycle $\langle\tilde{X},Y\rangle$ times. With
this choice, we can write $\hat{Q}^{y}(\hat{Y})$ as

$\displaystyle\hat{Q}^{y}(\hat{Y})$ $\displaystyle={1\over 2\pi}\oint_{\cal
C}\left(\partial_{z}\partial_{y}\hat{\phi}^{y(zx)}dy-\partial_{x}\partial_{z}\hat{\phi}^{x(yz)}dx-\partial_{x}\partial_{z}\hat{\phi}^{z(xy)}dx\right)$
(E.6) $\displaystyle={1\over 2\pi}\oint
dy\partial_{z}\partial_{y}\hat{\phi}^{y(zx)}-{\langle\tilde{X},Y\rangle\over
2\pi}\oint dx\partial_{x}\partial_{z}\hat{\phi}^{x(yz)}$

In the first line we used
$\hat{\phi}^{x(yz)}+\hat{\phi}^{y(zx)}+\hat{\phi}^{z(xy)}=0$. In the second
line we used the fact that $\partial_{z}\hat{\phi}^{z(xy)}$ is a single-valued
operator, and therefore $\partial_{x}\partial_{z}\hat{\phi}^{z(xy)}dy$ does
not contribute to the integral along $\cal C$, which is always aligned with
the $x$ or the $y$ axes. We have thus shown that

$\displaystyle\partial_{z}W^{z}_{y}(z)-\langle\tilde{X},Y\rangle\partial_{z}W^{z}_{x}(z)=\hat{Q}^{y}(\hat{Y})=m\hat{Q}^{y}(\tilde{X})\,.$
(E.7)

Therefore, from the general form of the charge $\hat{Q}^{y}$ (E.5), we arrive
at the constraint:

$\displaystyle\partial_{z}W^{z}_{y}(z)-\langle\tilde{X},Y\rangle\partial_{z}W^{z}_{x}(z)=m\sum_{\gamma}M_{\gamma}\delta^{\textrm{P}}(z,z_{\gamma},\ell_{z})\,,$
(E.8)

for some integers $M_{\gamma}\in\mathbb{Z}$. Using (2.13), this constraint is
equivalent to

$\displaystyle\partial_{z}W^{z}_{x}(z)-\langle\tilde{Y},X\rangle\partial_{z}W^{z}_{y}(z)=m\sum_{\gamma}M^{\prime}_{\gamma}\delta^{\textrm{P}}(z,z_{\gamma},\ell_{z})\,,$
(E.9)

for some integers $M^{\prime}_{\gamma}\in\mathbb{Z}$.

We summarize that the winding charges of $\hat{\phi}^{i(jk)}$ on a twisted
torus are parameterized by $W^{i}_{j}(x^{i})$ subject to the constraint (E.8)
and the redundancy (E.2). On a lattice, we have $2(L_{x}^{\rm eff}+L_{y}^{\rm
eff}+L_{z})-3$ such integers.

Below we present the explicit winding configurations of $\hat{\phi}^{i(jk)}$
that realize all these charges. For convenience, we will use (E.2) to gauge
fix $W^{z}_{x}(z=0)=W^{z}_{y}(z=0)=W_{z}^{x}(x=0)=0$ below.

The most general winding configuration is

$\displaystyle{1\over
2\pi}\hat{\phi}^{x(yz)}={\tilde{y}\over\ell_{x}}W^{y}_{x}(y)+{x\over\ell_{x}}W^{z}_{x}(z)-{y\over\ell_{y}}\,W^{z}_{y}(z)-{z\over\ell_{z}}\,W^{y}_{z}(y),$
(E.10) $\displaystyle{1\over
2\pi}\hat{\phi}^{y(zx)}={y\over\ell_{y}}W^{z}_{y}(z)+{\tilde{x}\over\ell_{y}}W^{x}_{y}(x)-{z\over\ell_{z}}\,W^{x}_{z}(x)-{x\over\ell_{x}}\,W^{z}_{x}(z)\,,$
$\displaystyle{1\over
2\pi}\hat{\phi}^{z(xy)}={z\over\ell_{z}}\left[W^{x}_{z}(x)+W^{y}_{z}(y)\right]-{\tilde{x}\over\ell_{y}}\,W^{x}_{y}(x)-{\tilde{y}\over\ell_{x}}\,W^{y}_{x}(y)\,.$

where

$\displaystyle\tilde{x}=y-\langle\tilde{Y},X\rangle{\ell_{y}\over\ell_{x}}x\,,$
(E.11)
$\displaystyle\tilde{y}=x-\langle\tilde{X},Y\rangle{\ell_{x}\over\ell_{y}}y\,.$

These coordinates are shifted by
$(\tilde{x},\tilde{y})\to(\tilde{x}-\langle\tilde{Y},X\rangle\ell_{y},\tilde{y}+\ell_{x})$
along the $X$ cycle, and are shifted by
$(\tilde{x},\tilde{y})\to(\tilde{x}+\langle\tilde{X},\tilde{Y}\rangle\ell_{y},\tilde{y})$
along the $\tilde{X}$ cycle.

Let us check that the transition functions are $2\pi\mathbb{Z}$-valued. The
transition function around the $Z$ cycle is:

$\displaystyle\hat{g}^{x(yz)}_{Z}=-2\pi W^{y}_{z}(y)\,,$ (E.12)
$\displaystyle\hat{g}^{y(zx)}_{Z}=-2\pi W^{x}_{z}(x)\,,$
$\displaystyle\hat{g}^{z(xy)}_{Z}=2\pi W^{x}_{z}(x)+2\pi W^{y}_{z}(y)\,.$

The transition function around the $X$ cycle is:

$\displaystyle\hat{g}^{x(yz)}_{X}=2\pi W^{y}_{x}(y)+2\pi W^{z}_{x}(z)$ (E.13)
$\displaystyle\hat{g}^{y(zx)}_{X}=-2\pi\langle\tilde{Y},X\rangle
W^{x}_{y}(x)-2\pi W^{z}_{x}(z)$
$\displaystyle\hat{g}^{z(xy)}_{X}=2\pi\langle\tilde{Y},X\rangle
W^{x}_{y}(x)-2\pi W^{y}_{x}(y)\,.$

Finally, the transition function around the $\tilde{X}$ cycle is

$\displaystyle\hat{g}^{x(yz)}_{\tilde{X}}=2\pi{\langle\tilde{X},Y\rangle\over
m}W^{z}_{x}(z)-2\pi{1\over m}W^{z}_{y}(z)$ (E.14)
$\displaystyle\hat{g}^{y(zx)}_{\tilde{X}}=2\pi\langle\tilde{X},\tilde{Y}\rangle
W^{x}_{y}(x)-2\pi{\langle\tilde{X},Y\rangle\over m}W^{z}_{x}(z)+2\pi{1\over
m}W^{z}_{y}(z)$
$\displaystyle\hat{g}^{z(xy)}_{\tilde{X}}=-2\pi\langle\tilde{X},\tilde{Y}\rangle
W^{x}_{y}(x)$

In the gauge choice $W^{z}_{x}(z=0)=W^{z}_{y}(z=0)=0$, (E.8) is equivalent to
$-\langle\tilde{X},Y\rangle W^{z}_{x}(z)+W^{z}_{y}(z)\in m\mathbb{Z}$. Hence,
all these transition functions are indeed $2\pi\mathbb{Z}$-valued. The cocycle
conditions are trivially satisfied since all the transition functions are
single-valued.

## References

  * [1] C. Chamon, Quantum glassiness in strongly correlated clean systems: An example of topological overprotection, Phys. Rev. Lett. 94 (Jan, 2005) 040402.
  * [2] J. Haah, Local stabilizer codes in three dimensions without string logical operators, Phys. Rev. A 83 (Apr, 2011) 042330.
  * [3] R. M. Nandkishore and M. Hermele, Fractons, Ann. Rev. Condensed Matter Phys. 10 (2019) 295–313, [arXiv:1803.11196].
  * [4] M. Pretko, X. Chen, and Y. You, Fracton Phases of Matter, arXiv:2001.01722.
  * [5] S. Vijay, J. Haah, and L. Fu, Fracton Topological Order, Generalized Lattice Gauge Theory and Duality, Phys. Rev. B94 (2016), no. 23 235157, [arXiv:1603.04442].
  * [6] K. Slagle and Y. B. Kim, Quantum Field Theory of X-Cube Fracton Topological Order and Robust Degeneracy from Geometry, Phys. Rev. B96 (2017), no. 19 195139, [arXiv:1708.04619].
  * [7] K. Slagle, D. Aasen, and D. Williamson, Foliated Field Theory and String-Membrane-Net Condensation Picture of Fracton Order, SciPost Phys. 6 (2019), no. 4 043, [arXiv:1812.01613].
  * [8] K. Slagle, Foliated Quantum Field Theory of Fracton Order, arXiv:2008.03852.
  * [9] N. Seiberg, Field Theories With a Vector Global Symmetry, SciPost Phys. 8 (2020) 050, [arXiv:1909.10544].
  * [10] N. Seiberg and S.-H. Shao, Exotic Symmetries, Duality, and Fractons in 2+1-Dimensional Quantum Field Theory, arXiv:2003.10466.
  * [11] N. Seiberg and S.-H. Shao, Exotic $U(1)$ Symmetries, Duality, and Fractons in 3+1-Dimensional Quantum Field Theory, SciPost Phys. 9 (2020) 046, [arXiv:2004.00015].
  * [12] N. Seiberg and S.-H. Shao, Exotic $\mathbb{Z}_{N}$ Symmetries, Duality, and Fractons in 3+1-Dimensional Quantum Field Theory, arXiv:2004.06115.
  * [13] P. Gorantla, H. T. Lam, N. Seiberg, and S.-H. Shao, More Exotic Field Theories in 3+1 Dimensions, arXiv:2007.04904.
  * [14] P. Gorantla, H. T. Lam, N. Seiberg, and S.-H. Shao, FCC, Checkerboards, Fractons, and QFT, arXiv:2010.16414.
  * [15] K. Slagle and Y. B. Kim, X-cube model on generic lattices: Fracton phases and geometric order, Phys. Rev. B97 (2018), no. 16 165106, [arXiv:1712.04511].
  * [16] W. Shirley, K. Slagle, Z. Wang, and X. Chen, Fracton Models on General Three-Dimensional Manifolds, Phys. Rev. X 8 (2018), no. 3 031051, [arXiv:1712.05892].
  * [17] W. Shirley, K. Slagle, and X. Chen, Fractional excitations in foliated fracton phases, Annals Phys. 410 (2019) 167922, [arXiv:1806.08625].
  * [18] W. Shirley, K. Slagle, and X. Chen, Foliated fracton order in the checkerboard model, Phys. Rev. B 99 (2019), no. 11 115123, [arXiv:1806.08633].
  * [19] W. Shirley, K. Slagle, and X. Chen, Foliated fracton order from gauging subsystem symmetries, SciPost Phys. 6 (2019), no. 4 041, [arXiv:1806.08679].
  * [20] K. Slagle, A. Prem, and M. Pretko, Symmetric Tensor Gauge Theories on Curved Spaces, Annals Phys. 410 (2019) 167910, [arXiv:1807.00827].
  * [21] W. Shirley, K. Slagle, and X. Chen, Twisted foliated fracton phases, Phys. Rev. B 102 (2020), no. 11 115103, [arXiv:1907.09048].
  * [22] A. Dua, D. J. Williamson, J. Haah, and M. Cheng, Compactifying fracton stabilizer models, Phys. Rev. B 99 (2019), no. 24 245135, [arXiv:1903.12246].
  * [23] M. Qi, L. Radzihovsky, and M. Hermele, Fracton phases via exotic higher-form symmetry-breaking, arXiv:2010.02254.
  * [24] N. Manoj, K. Slagle, W. Shirley, and X. Chen, Screw dislocations in the X-cube fracton model, arXiv:2012.07263.
  * [25] A. Dua, D. J. Williamson, and M. Cheng , unpublished.
  * [26] D. S. Freed, G. W. Moore, and G. Segal, The Uncertainty of Fluxes, Commun. Math. Phys. 271 (2007) 247–274, [hep-th/0605198].
  * [27] D. S. Freed, G. W. Moore, and G. Segal, Heisenberg Groups and Noncommutative Fluxes, Annals Phys. 322 (2007) 236–285, [hep-th/0605200].
  * [28] A. Kapustin and N. Seiberg, Coupling a QFT to a TQFT and Duality, JHEP 04 (2014) 001, [arXiv:1401.0740].
  * [29] D. Gaiotto, A. Kapustin, N. Seiberg, and B. Willett, Generalized Global Symmetries, JHEP 02 (2015) 172, [arXiv:1412.5148].
  * [30] H. Ma, E. Lake, X. Chen, and M. Hermele, Fracton topological order via coupled layers, Phys. Rev. B95 (2017), no. 24 245126, [arXiv:1701.00747].
  * [31] S. Vijay, Isotropic Layer Construction and Phase Diagram for Fracton Topological Phases, arXiv:1701.00762.
  * [32] A. Paramekanti, L. Balents, and M. P. A. Fisher, Ring exchange, the exciton bose liquid, and bosonization in two dimensions, Phys. Rev. B 66 (Aug, 2002) 054526.
  * [33] M. Bauer, G. Girardi, R. Stora, and F. Thuillier, A Class of topological actions, JHEP 08 (2005) 027, [hep-th/0406221].
  * [34] C. Cordova, D. S. Freed, H. T. Lam, and N. Seiberg, Anomalies in the Space of Coupling Constants and Their Dynamical Applications I, SciPost Phys. 8 (2020), no. 1 001, [arXiv:1905.09315].