# Doc2Dict: Information Extraction as Text Generation

Benjamin Townsend11footnotemark: 1 Eamon Ito-Fisher22footnotemark: 2 Lily
Zhang33footnotemark: 3 Madison May11footnotemark: 1
11footnotemark: 1 Indico Data Solutions
22footnotemark: 2 Franklin W. Olin College of Engineering
33footnotemark: 3 New York University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Typically, information extraction (IE) requires a pipeline approach: first, a
sequence labeling model is trained on manually annotated documents to extract
relevant spans; then, when a new document arrives, a model predicts spans
which are then post-processed and standardized to convert the information into
a database entry. We replace this labor-intensive workflow with a transformer
language model trained on existing database records to directly generate
structured JSON. Our solution removes the workload associated with producing
token-level annotations and takes advantage of a data source which is
generally quite plentiful (e.g. database records). As long documents are
common in information extraction tasks, we use gradient checkpointing and
chunked encoding to apply our method to sequences of up to 32,000 tokens on a
single GPU. Our Doc2Dict approach is competitive with more complex, hand-
engineered pipelines and offers a simple but effective baseline for document-
level information extraction. We release our Doc2Dict model and code to
reproduce our experiments and facilitate future work. 111The code is released
at https://github.com/IndicoDataSolutions/Doc2Dict.

Figure 1: Information extraction with sequence-labeling models requires
gathering additional token-level labels as well as an additional span
selection and standardization step. Our proposed method, Doc2Dict, learns to
directly produce a normalized version of the database record that corresponds
to a given document, enabling the use of pre-existing database records as
supervision and minimizing the need for custom post-processing.

##  1 Introduction

The task of extracting structured information from documents is omnipresent
Zhang et al. (2020); Majumder et al. (2020); Liu et al. (2019). Although
documents are designed primarily for human-consumption, they often contain
values (e.g. amounts, dates, IDs) that are useful as inputs to downstream
software. While rigid forms permit straightforward automated processing via
the use of document templates, more varied documents like invoices, contracts,
and quarterly reports pose a challenge for machine-learning assisted
processing. For these documents, the information necessary to convert the
document to structured records is primarily textual and we turn to the field
of natural language processing for potential solutions.

###  1.1 Sequence-Labeling for Information Extraction

It is natural to frame information extraction problems as sequence-labeling
tasks Collobert et al. (2011); Huang et al. (2015); Lample et al. (2016).
However, sequence-labeling models rely on the availability of token-level
annotations — data not typically recorded by data entry software, which only
captures the standardized values derived from the document. Consequently,
documents that have already been manually processed need to undergo a
secondary annotation process where annotators denote document spans that
correspond to the derived values.

The cost of annotating large corpora with per-token labels is significant
Alonso et al. (2016); Settles et al. (2008), and this burden is exaggerated
for long documents where information of interest may be spread throughout the
document. Previous work has focused on minimizing the number of labeled
samples required to construct an effective model Howard and Ruder (2018);
Radford et al. (2019); Devlin et al. (2019), but annotating even a few hundred
lengthy documents still requires substantial effort and domain knowledge.

One might reasonably look to reconstruct token-level labels by automatically
finding the source of each field in the corresponding document, but this
alignment process is error prone. Searching for exact string matches between
database values and source document text is insufficient, as values are
frequently standardized prior to storage (e.g. “May 7th, 2019” may be
converted to “2019/05/07”).

In other cases, a stored value might spuriously match irrelevant tokens in the
source document. For instance, the “Subtotal” and “Total” lines on an invoice
may both match the value of “Invoice Total” stored in a database, but only the
latter should be provided as an annotation. More sophisticated systems do
exist for aligning the values stored in a database with likely matches in the
source document, but they rely on custom logic for each field type to perform
reliably Graliński et al. (2020) and must be tailored to the task of interest.

Even after constructing a dataset with token-level labels and training a
sequence-labeling model, it is necessary to post-process predicted spans to
prepare entities for storage in a database. In the case that a single value is
expected for a given field, a pool of candidate spans must be narrowed down to
a single response via a selection heuristic.

Values must also be standardized to match the expected format. If
standardization is handled by a rules-based system, additional software
engineering effort is necessary in order to benefit from the machine learning
system. If standardization is instead performed manually, extra overhead will
be incurred in perpetuity when incoming documents are processed. In
combination, these limitations pose significant barriers to adoption.

###  1.2 Information Extraction as Text Generation

One possible solution to address the difficulty of producing labeled training
data for sequence-labeling tasks is to forgo the requirement of token-level
labels altogether by framing information extraction as a sequence-to-sequence
task from document to a structured database record directly Sutskever et al.
(2014). Sequence-to-sequence learning presents a flexible framework under
which many diverse tasks can be unified - classification, regression, question
answering, and sequence-labeling tasks can all be treated as simple text
generation tasks McCann et al. (2018); Raffel et al. (2020); Radford et al.
(2019). Rather than iterating through a document and predicting which document
spans correspond to the fields of interest, we task a machine learning model
with generating a JSON-like representation of the extracted information when
conditioned on the text of the source document, permitting the direct use of
preexisting database records as supervision.

This paper introduces Doc2Dict: a general, end-to-end information extraction
system that directly utilizes database records generated by manual data entry
processes as training data for a text generation model. Doc2Dict requires
little to no custom configuration and eliminates the need for additional
manual annotation or a heuristic alignment step. We jointly learn to extract,
select, and standardize values of interest, enabling direct generation of a
JSON-like data format. Additionally, we leverage gradient checkpointing to
allow processing documents of up to 32k tokens on a single consumer GPU.
Finally, we systematically explore how different types of information loss in
the translation to database records impact model performance and suggest
avenues for minimizing impact.

##  2 Related Work

Other natural language processing application areas have already benefited
from the use of text-generation models to tasks that require structured
outputs. In 2013, Andreas et al. (2013) demonstrated that framing semantic
parsing as a machine translation task was competitive with existing methods
that were purpose-built for semantic-parsing. More recently, text-generation
methods have been successfully applied to the text-to-SQL task, which requires
the translation of a natural language question and a database schema to a SQL
query. Of particular note is SeqGenSQL Li et al. (2020), a finetuned T5 model
that treats the text-to-SQL problem as a pure text generation task and
currently leads the weakly supervised category of the WikiSQL benchmark Zhong
et al. (2017).

Similar to the text-to-SQL problem, task-oriented dialogue systems often
contain components that translate free-form conversations to structured
representations of dialogue state. While task-oriented dialogue systems have
traditionally been broken up into many disparate parts (dialogue state
tracking, dialogue management, and response generation), SimpleTOD Hosseini-
Asl et al. (2020) and its successors Peng et al. (2020); Lin et al. (2020b)
unify these parts under the umbrella of text generation and use a simple
language modeling objective to learn to produce dialogue state
representations.

Several prior works explore information extraction tasks in an end-to-end
manner. Paolini et al. (2021) proposes an augmented natural language format
for use in solving a wide variety of structured prediction language tasks,
including named entity recognition, relation extraction, and event extraction.
Their output format and framing of IE as a generative language modeling task
permits multi-task learning without task-specific heads, but requires span-
level information, preventing direct application to our task of interest. Zeng
et al. (2018) and Nayak and Ng (2020) use text-generation models to perform
end-to-end joint entity and relation extraction from sentences. Similarly, Lin
et al. (2020a) jointly tackles entity, relation, and event extraction tasks by
using a transformer encoder and sophisticated graph decoder. In all cases,
however, copy mechanisms or extractive task formulations prevent
straightforward application to our problem setting.

Most relevantly, a handful of prior works have approached information
extraction from business documents in an end-to-end manner. Palm et al. (2019)
leverages pointer networks to extract relevant fields from invoices. However,
they build custom neural modules to handle the standardization of each field
type, which limits the general usability of the approach to other output types
not yet considered (e.g. addresses) and prevents leveraging pre-trained models
to minimize training data requirements. Sage et al. (2020) similarly employs a
pointer-generator network for information extraction from purchase orders. In
contrast to our work, like Zeng et al. (2018) they assume that no
standardization has been applied to the extracted values and require that all
system outputs can be copied directly from the source document.

##  3 Methods

Our Doc2Dict model is based on the T5 transformer-based encoder-decoder
architecture of Raffel et al. (2020). We maintain the text-to-text formulation
of T5, enabling structured output generation by formatting the output as
Python’s string representation of built-in data types. In practice, we find
that a multitude of different structured output formats enable similar
performance (see Section 3.4). In the following subsections, we present
additional considerations raised by our specific formulation of information
extraction as text generation and our methods for addressing them.

###  3.1 Entity Order

One peculiarity of treating information extraction tasks in a text-to-database
formulation is that the correct output sequence is not uniquely determined; in
particular, any ordering of key-value pairs is a correct output. Language
models trained with teacher forcing, however, only consider one correct output
string. Using such a loss, our model may be penalized for producing valid
responses. Unfortunately, training on all possible permutations increases the
computational requirements by a factor of $n!$ where $n$ is the length of the
sequence. Instead, we choose to encourage robustness to entity order by
evaluating with respect to different randomly-ordered output sequences per
epoch. We find that this simple solution works reasonably well in practice,
achieving gains over fixed output orderings.

###  3.2 Standardization

The standardization of output values results in a discrepancy between the
supervision offered by database records and the text that exists in the
document. For instance, if the entity of interest is a due date that exists in
YY/MM/DD format in the database but in textual format (e.g. Month Day, Year)
in the document, correctly predicting the provided database label requires
that a model not only identify the relevant part of the text, but also
translate that text into the format given in the label. The latter must be
learned without explicit supervision of an intermediate value (e.g. the exact
value as seen in the document) and often requires learning non-trivial multi-
token to multi-token mappings. For instance, even translating between words
with different capitalization can change tokenization drastically, e.g.
(“Barack”, “ Obama”) vs (“B”, “AR”, “ACK”, “ ”, “OB”, “AMA”). To alleviate
this effect, we preprocess the input text before tokenization so that
candidate text spans more closely align with the database output, e.g. if
outputs are all lowercase, we would lowercase all the text before feeding it
into the model.

###  3.3 Processing Long Input Sequences

Transformer-based language models are typically trained on subsequences of 512
to 1024 tokens at a time because of the memory complexity of the attention
operation Vaswani et al. (2017); Wang et al. (2020), but many information
extraction tasks require processing documents that may be tens of thousands of
tokens in length. In the sequence-labeling problem formulation, this
limitation can be overcome by “chunking” the document into either sentences or
windows that contain a fixed number of tokens Dai et al. (2019).

Unfortunately, this chunking paradigm breaks down when applied to a text-
generation setting. Using each chunk and the target sequence independently is
ill-formed, as many document chunks may contain no information relevant to
producing the target sequence. If the information present in the target
sequence is not derivable from the input subsequence supplied, the loss
encourages the undesirable behavior of producing likely output sequences not
relevant to the input. Without knowing where in the document target values are
derived we cannot effectively break the document up into manageable
subsequences with appropriate labels per chunk.

####  3.3.1 Fusion-in-Decoder

As the aforementioned chunking scheme is not viable for our approach, we
address the issue of long input sequences by taking inspiration from the
fusion-in-decoder method Izacard and Grave (2020). First proposed for
applications in open-domain question answering, fusion-in-decoder provides a
means for encoder-decoder language models to independently encode many
subsequences but aggregate information from all subsequences in the decoder.
In Izacard and Grave (2020), fusion-in-decoder is applied to different
documents returned by a retriever, whereas we apply a similar method to permit
conditioning on many chunks from a single long document. Rather than a
quadratic dependency on overall sequence length, the fusion-in-decoder method
requires $O(c^{2}m)$ compute and memory, where $c$ is the chunk size and $m$
is the number of fixed-size chunks processed by the encoder. By only
permitting dense attention within a given chunk in the encoder and restricting
chunks to a fixed size, the memory and compute complexity is made linear with
the length of the document.

####  3.3.2 Gradient Checkpointing Scheme

Even with these algorithmic optimizations, the memory requirements of
processing long documents remain high and gradient checkpointing is required
at training time. Gradient checkpointing allows trading compute for memory by
caching only a portion of intermediate activations and recomputing the
remaining activations during the backwards pass Chen et al. (2016). We use
this standard approach for the decoder, but for the encoder we add an
additional layer of gradient checkpointing per-encoder stack, illustrated in
figure 2. When using a batch size of 1 at training time, this approach allows
us to train on 64 blocks of 512 tokens at full precision with only 11GB of
VRAM. This translates to an overall sequence length of over 32,000 tokens.

Figure 2: Gradient checkpointing scheme used to make training on long input
sequences tractable. We compute a full forward pass of the model, computing
the loss and caching the activations of each encoded chunk and each of the
inter-block activations of the decoder (indicated by dashed red lines). These
cached activations are used in conjunction with a second forward pass through
the decoder in order to compute the gradients of the decoders parameters. For
each block in the encoder, we then recompute the forward pass to cache the
inter-block activations (indicated by solid blue lines). A final forward pass
through the encoder allows computing gradients with respect to the parameters
of the encoder.

###  3.4 Output Format Experiments

To test what output formats are most easily generated by T5, we experiment
with multiple output formats on a joint intent detection and slot-filling
task, ATIS Hemphill et al. (1990). Output formats tested include XML, JSON,
and Python’s string representation, a close cousin to JSON. As the T5 vocab is
missing some tokens necessary for representing these formats (e.g. “{” and
“}”), some of the extra IDs present in the T5 vocab were assigned to these
tokens.

Although the lack of the requisite tokens in the vocabulary implies that T5
has not been exposed to data in these formats during pre-training, we found
that all output formats were viable targets for the task and performed
similarly, as shown in Table 1. We opt to use Python’s str format in
subsequent experiments, due in part to its shorter output sequence length
relative to XML. This shorter output length translates directly to shorter
training and prediction times.

Format | Intent | Entity | Parse Failure
---|---|---|---
YAML | .970 | .951 | .003
XML | .969 | .949 | .002
JSON | .972 | .949 | .002
Python string | .971 | .950 | .004
Table 1:  Performance on ATIS joint intent and slot filling task by output
format. Output format used has negligible impact on task performance. Reported
intent and entity scores are micro F1. Across all output formats, generated
content failed format validation only 0.26% of the time.

##  4 Results

We evaluate our method on a number of datasets covering direct extraction and
end-to-end information extraction, the results of which are included in Tables
2.

One of the primary strengths of our approach is its ability learn to extract
and standardize data from documents given only the desired answer for training
signal. We demonstrate these strengths on three datasets; Kleister-NDA,
Kleister-Charity Graliński et al. (2020) and DeepForm Stacey Svetlichnaya
(2020). Each of these datasets provides a set of documents paired with ground
truth values, but does not provide information about where ground truth values
occurred in the source document. Some of these ground truth values appear
verbatim whilst others have been transformed to a standardized format. For the
Kleister and DeepForm datasets the baselines used in the original papers apply
a pipeline approach, where custom heuristics are used to align records to text
spans in the input document and re-cast the problem as a sequence-labeling
task. At prediction time per-field standardization and selection rules are
applied to the model predictions in order to translate extracted spans back to
a canonical form that matches the desired output.

Additionally, we would like to draw some conclusions about how token-level
supervision compares to a sequence-to-sequence formulation when entity
standardization is not required. As an indicator of relative performance we
compare a RoBERTa model trained with token-level annotations to our Doc2Dict
method. We test on the ATIS intent detection dataset as well as the OntoNotes
5.0 NER dataset Pradhan et al. (2013).

###  4.1 Metrics

For the Kleister datasets we use uncased F1 metrics to match the original task
paper Graliński et al. (2020). For DeepForm we use uncased entity-level F1 and
report macro F1 as an overall score. For ATIS and OntoNotes entity-level micro
F1 is reported.

###  4.2 Data Preprocessing

For Kleister and DeepForm, we opt to lowercase both the source and the targets
and remove any commas that occur directly after a number in the source text.
This is an effort to rectify some of the problems resulting from inconsistent
tokenization, mentioned previously in Section 3.2 and discussed at length in
Section 5.1. The relevant metrics are insensitive to case so this
transformation does not effect metrics. Ablations for these transformations
can be seen in Table 4.

For the DeepForm and ATIS datasets, our output format is a Python dictionary
format.

    {’key’: value, ...}

For the Kleister and OntoNotes datasets, our output format is a set of tuples,
which allows us to represent keys that have multiple values.

    {(’key’, value), ...}

Dataset | Doc2Dict F1 | Pipeline F1
---|---|---
DeepForm | 0.900 | 0.761
Kleister-NDA | 0.809 | 0.777
Kleister-Charity | 0.561 | 0.801
Table 2: Results for Kleister-NDA, Kleister-Charity and DeepForm. In two of three tasks of interest, we outperform the baseline despite not requiring token-level supervision. Although we under-perform the baseline on Kleister-Charity, we believe that the comparative ease and reduced cost of obtaining the required training data has the opportunity to result in significant benefits for large scale practical applications. Doc2Dict | F1 | Baseline F1
---|---|---
OntoNotes | 0.865 | 0.896
ATIS | 0.959 | 0.987
Table 3: The baseline model is a RoBERTa model Liu et al. (2019) with CRF output layer that has access to gold standard token-level annotations as training signal. Even without this additional supervision, our model is able to produce very reasonable results on OntoNotes and ATIS datasets. Dataset |  | F1
---|---|---
Charity | Base | 0.503
Charity | \+ Lowercased | 0.533
Charity | \+ Comma Stripping | 0.561
NDA | Base | 0.716
NDA | \+ Shuffled Epochs | 0.759
NDA | \+ Lowercased | 0.797
NDA | \+ Comma Stripping | 0.809
Table 4: Ablations to explore the impact of our dataset preprocessing. Comma
stripping is stripping all commas that proceed a number, Lowercased is making
the input and targets lowercase and Shuffled Epochs is reordering target
values between every epoch (details in Section 3.1). Shuffled Epochs is not
included for Kleister-Charity as there is a one to one mapping between key and
value.

##  5 Limitations and Future Work

###  5.1 Tokenizer Limitations

Although our work shows that sequence to sequence information extraction
models are a viable alternative to token-level sequence-labeling tasks when
database records are available for use as supervision, we hypothesize that
T5’s choice of tokenizer may make learning to jointly extract and standardize
information more difficult than necessary.

In order to validate this hypothesis we opt to study the task of learning
standardization rules in isolation to understand what kinds of standardization
rules are most tractable to learn. Below, we experiment with three synthetic
datasets. For all experiments we use the pre-trained T5 base checkpoint and
measure how performance varies as dataset size is increased. Experiment
results are presented in Figure 3.

Figure 3: Name and number standardization is substantially more difficult than
date standardization, which we attribute to tokenization differences caused by
the standardization operation in these two tasks. See Section 5.1 for
experimental details. Figure 4: With T5’s default tokenizer, we find learned
standardization is imperfect even at 50k training examples. However,
constraining T5’s vocabulary to single character tokens makes the number
standardization task trivial by ensuring tokens can be copied directly from
the input. Unfortunately, this change increases overall sequence length
significantly, and we refrain from applying this modification in our main
experimental results.

####  5.1.1 Date Standardization

The first synthetic dataset explores the difficulty in translating from a
variety of date formats to a standard form. Random dates between 1950 and 2021
were generated using Python’s datetime module and formatted using a family of
10 unique format strings. The model is tasked with translating these varied
date formats to a canonical “%Y/%m/%d” form. As observed in our main results,
we found this translation to be easily learnable – the model completes the
task with nearly perfect accuracy after only observing 2500 examples.

####  5.1.2 Name Lowercasing

The second synthetic dataset tests the model’s ability to learn to translate
names with varying capitalization to a lowercase form. Of the input data, 5%
is lowercase, 20% is uppercase, and the remaining 75% has the first letter of
every word capitalized. Names are drawn at random from a gazetteer of roughly
18,000 names. In contrast to the date standardization task, at 2500 samples
our exact match accuracy is only around 80%. Even at the full 50,000 samples
we observe an error rate of 2.5%, an indicator that this mapping is not
trivial to learn. We hypothesize that this difficulty in learning to solve
this task is due to the difference in the input and output sequence
tokenization induced by the lowercasing operation (described previously in
Section 3.2).

####  5.1.3 Number Standardization

Finally, we test the model’s ability to convert from integer values sampled
uniformly between one thousand and one billion to a more human readable format
that includes commas and two decimal points of precision. A value of
“123456789” would be converted to “123,456,789.00” – the encoder-decoder model
must learn to insert commas where appropriate and emit “.00” at the end of the
sequence. Similar to the lowercasing task, we observe a failure rate of 5%
even when training on the full dataset of 50,000 samples.

####  5.1.4 Tokenizer Modifications

To validate our hypothesis that the poor performance on the number and name
standardization datasets are due to the tokenization differences in the input
and output sequences preventing a copying heuristic from being learned, we
experiment with restricting the vocabulary of T5 to aid in copying number
values. In particular, we modify the tokenizer to produce only single
character tokens for numeric values. In Figure 4, we show that this simple
modification makes the standardization task trivial for the T5 model to solve,
suggesting future work on tokenization may be key to making learned
normalization practical.

###  5.2 Conclusion

We devise a new formulation for information extraction which directly exploits
existing database records as training data, a much less labor-intensive
approach than existing sequence-extraction methods. Paired with simple data
preprocessing techniques and a strategy that enables processing long
documents, our method can achieve comparable results to hand-crafted pipelines
on several real-world datasets.

Future work could incorporate constrained generation to avoid unparseable
outputs, permutation-invariant losses to better address the issue of entity
order, and information extraction-specific pretraining to help with learning
implicit standardization. Moreover, as the fusion-in-decoder formulation does
not allow for attention across chunks in the encoder, we hypothesize that
methods which allow for long-range attention in the encoder during pre-
training may lead to better task performance. Additionally, since document
layout understanding is important for accurate information extraction (e.g.
extraction of cell values in tables and forms), incorporating strategies to
include the spatial location of tokens in their input representations, such as
done in Xu et al. (2020), could improve the general performance in our
approach for information extraction. Finally, although not explored in this
work, our flexible output format permits representing outputs with more
complex structure and may prove a promising direction for approaching other
information extraction tasks like relation extraction. Regardless, in its
current form, Doc2Dict represents an easy, reproducible baseline for future
information extraction tasks and paves the way towards practical information
extraction processes.

##  6 Acknowledgements

We thank Luke Metz, Rahil Dedhia, Slater Victoroff and Matthew Bayer for their
invaluable feedback on early drafts of this paper.

## References

  * Alonso et al. (2016) Héctor Martínez Alonso, Djamé Seddah, and Benoît Sagot. 2016.  From noisy questions to minecraft texts: Annotation challenges in extreme syntax scenario.  In _NUT@COLING_.
  * Andreas et al. (2013) Jacob Andreas, Andreas Vlachos, and Stephen Clark. 2013.  Semantic parsing as machine translation.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 47–52, Sofia, Bulgaria. Association for Computational Linguistics.
  * Chen et al. (2016) T. Chen, B. Xu, C. Zhang, and Carlos Guestrin. 2016.  Training deep nets with sublinear memory cost.  _ArXiv_ , abs/1604.06174.
  * Chiu and Nichols (2016) Jason P.C. Chiu and Eric Nichols. 2016.  Named entity recognition with bidirectional LSTM-CNNs.  _Transactions of the Association for Computational Linguistics_ , 4:357–370.
  * Collobert et al. (2011) Ronan Collobert, J. Weston, L. Bottou, Michael Karlen, K. Kavukcuoglu, and P. Kuksa. 2011.  Natural language processing (almost) from scratch.  _J. Mach. Learn. Res._ , 12:2493–2537.
  * Dai et al. (2019) Zihang Dai, Z. Yang, Yiming Yang, J. Carbonell, Quoc V. Le, and R. Salakhutdinov. 2019.  Transformer-xl: Attentive language models beyond a fixed-length context.  In _ACL_.
  * Devlin et al. (2019) J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_.
  * Ghaddar and Langlais (2018) Abbas Ghaddar and Phillippe Langlais. 2018.  Robust lexical features for improved neural network named-entity recognition.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1896–1907, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Graliński et al. (2020) Filip Graliński, Tomasz Stanisławek, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemysław Biecek. 2020.  Kleister: A novel task for information extraction involving long documents with complex layout.
  * Hemphill et al. (1990) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990.  The ATIS spoken language systems pilot corpus.  In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990_.
  * Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, B. McCann, Chien-Sheng Wu, Semih Yavuz, and R. Socher. 2020\.  A simple language model for task-oriented dialogue.  _ArXiv_ , abs/2005.00796.
  * Howard and Ruder (2018) J. Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _ACL_.
  * Huang et al. (2015) Zhiheng Huang, W. Xu, and Kai Yu. 2015.  Bidirectional lstm-crf models for sequence tagging.  _ArXiv_ , abs/1508.01991.
  * Izacard and Grave (2020) Gautier Izacard and E. Grave. 2020.  Leveraging passage retrieval with generative models for open domain question answering.  _ArXiv_ , abs/2007.01282.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 260–270, San Diego, California. Association for Computational Linguistics.
  * Li et al. (2020) Ning Li, Bethany Keller, Mark Butler, and Daniel Matthew Cer. 2020.  Seqgensql - a robust sequence generation model for structured query language.  _ArXiv_ , abs/2011.03836.
  * Lin et al. (2020a) Ying Lin, Huai zhong Ji, Fei Huang, and L. Wu. 2020a.  A joint neural model for information extraction with global features.  In _ACL_.
  * Lin et al. (2020b) Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020b.  Mintl: Minimalist transfer learning for task-oriented dialogue systems.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Majumder et al. (2020) Bodhisattwa Majumder, Navneet Potti, Sandeep Tata, James B. Wendt, Qi Zhao, and Marc Najork. 2020.  Representation learning for information extraction from form-like documents.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)_ , pages 6495–6504.
  * McCann et al. (2018) B. McCann, N. Keskar, Caiming Xiong, and R. Socher. 2018.  The natural language decathlon: Multitask learning as question answering.  _ArXiv_ , abs/1806.08730.
  * Nayak and Ng (2020) Tapas Nayak and H. T. Ng. 2020.  Effective modeling of encoder-decoder architecture for joint entity and relation extraction.  _ArXiv_ , abs/1911.09886.
  * Palm et al. (2019) R. Palm, Florian Laws, and O. Winther. 2019.  Attend, copy, parse end-to-end information extraction from documents.  _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 329–336.
  * Paolini et al. (2021) Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021\.  Structured prediction as translation between augmented natural languages.  In _International Conference on Learning Representations_.
  * Peng et al. (2020) Baolin Peng, C. Li, Jin chao Li, Shahin Shayandeh, L. Liden, and Jianfeng Gao. 2020\.  Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model.  _ArXiv_ , abs/2005.05298.
  * Pradhan et al. (2013) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013.  Towards robust linguistic analysis using OntoNotes.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_ , pages 143–152, Sofia, Bulgaria. Association for Computational Linguistics.
  * Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012.  CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes.  In _Joint Conference on EMNLP and CoNLL - Shared Task_ , pages 1–40, Jeju Island, Korea. Association for Computational Linguistics.
  * Radford et al. (2019) A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, M. Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Sage et al. (2020) Clément Sage, A. Aussem, V. Eglin, H. Elghazel, and Jérémy Espinas. 2020\.  End-to-end extraction of structured information from business documents with pointer-generator networks.  In _SPNLP@EMNLP_.
  * Settles et al. (2008) B. Settles, M. Craven, and Lewis A. Friedland. 2008.  Active learning with real annotation costs.
  * Stacey Svetlichnaya (2020) Jonathan Stray Stacey Svetlichnaya. 2020.  Project deepform: Extract information from documents.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _NIPS_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _ArXiv_ , abs/1706.03762.
  * Wang et al. (2020) Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020.  Linformer: Self-attention with linear complexity.  _ArXiv_ , abs/2006.04768.
  * Xu et al. (2020) Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and M. Zhou. 2020.  Layoutlm: Pre-training of text and layout for document image understanding.  _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_.
  * Zeng et al. (2018) Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018.  Extracting relational facts by an end-to-end neural model with copy mechanism.  In _ACL_.
  * Zhang et al. (2020) Ruixue Zhang, W. Yang, Luyun Lin, Zhengkai Tu, Yuqing Xie, Zihang Fu, Yuhao Xie, Luchen Tan, Kun Xiong, and Jimmy Lin. 2020.  Rapid adaptation of bert for information extraction on domain-specific business documents.  _ArXiv_ , abs/2002.01861.
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017.  Seq2sql: Generating structured queries from natural language using reinforcement learning.  _CoRR_ , abs/1709.00103.

##  Appendix A Task Descriptions and Additional Experimental Details

###  A.1 Kleister-NDA and Kleister-Charity

For the Kleister datasets, we use the train, dev and test splits provided by
the original paper and report uncased F1 scores Graliński et al. (2020).
Alongside the documents, the Kleister datasets provide outputs from several
OCR providers. For each dataset we use the OCR provider that yielded the best
metrics in the original paper: pdf2djvu for Kleister-NDA and Textract for
Kleister-Charity. For each input sequence the Kleister dataset provides a set
of ”slots” to be filled; we concatenate this list of expected values to the
beginning of the input sequence but do not enforce that responses for each
slot are generated.

##### Kleister-NDA Statistics

In order to validate the claim that heuristic alignment of normalized entities
with the source text of the document leads to sub-par targets, we manually
produce token-level annotations for a set of 100 Kleister-NDA documents. We
then attempt to automatically annotate the documents using a simple regular
expression to search for near-exact matches in the source document. The
matching procedure permits the inclusion of additional spaces, periods, or
commas as well as variation in capitalization. Results of this evaluation are
included in Table 5.

For the “Jurisdiction” and “Party” labels we observe high recall, with all
“Jurisdiction” labels and all but 7 “Party” labels having near-exact matches
in the source document. Term and Effective Date labels are more varied in
their presentation in the source document, with more than a fifth of relevant
labels being missed by automatic annotation. Perhaps more concerning, we also
observe a high false positive rate for all classes, with precision scores
varying from 0.25 to 0.72. In other words, automatic annotations frequently
match irrelevant mentions in the source document.

Table 5: Statistics relating to fuzzy matching on the Kleister-NDA dataset. Automatically-labeled spans that overlap with ground truth annotations are considered true positives, those that do not are considered false positives. | Recall | Precision
---|---|---
Effective Date | 0.60 | 0.58
Jurisdiction | 1.00 | 0.33
Party | 0.94 | 0.42
Term | 0.72 | 0.72

##### Kleister-Charity Results

We hypothesize that the poor performance of our approach on the Kleister-
Charity task is due to several factors. First, documents in this task are
significantly longer on average than the documents from other datasets (See
Table 8.) Approximately 6.5% of the documents in this task exceed our 32k
token limit and are truncated. This may require our model to produce a
response for which it does not have access to supporting evidence. Second,
several of the Kliester-Charity fields commonly appear in tables (namely,
”Income” and ”Spending”) instead of longer-form natural language. While this
should also be a challenge for the baseline, we note that the chunking scheme
used in Text2DB may exacerbate this issue by making it likely that a table
cell value is not included in the same chunk as its corresponding header.

##### Kleister Error Analysis

We selected a sample of 20 documents from the Kleister-Charity and Kleister-
NDA test sets in order to perform a qualitative error analysis. Overall,
errors were similar to those seen in any information extraction pipeline. We
most commonly observed missed, erroneous, or partial matches akin to those
produced by typical sequence-labeling models.

We also occasionally observed errors unique to the text-generation problem
framing. One such example are errors we attribute to a failure to copy a value
from input to output. This seemed to be most prevalent in the Kleister-NDA
post code field, where many of the extracted values were similar but not
exactly the same as values that appeared in the document. In one case, we
observed a field key predicted as a value in a case where the correct
extracted value started with the same few tokens as a key: ”Basic Income Earth
Network (Bien)” became ”basic income annually in british pounds”. Another
error mode unique to the text-generation framing is that of hallucinating
plausible but invalid continuations of observed entities. In one instance, our
model extended the name of the town ”Sutton” to ”Sutton-upon-Thames”.

It is worth noting that we observed no errors that could be described as
”standardization errors”, in cases such as dates and term where heavy
transformations are applied by the model, the outputs were always correctly
formed.

###  A.2 DeepForm

The DeepForm dataset is composed of PDF disclosures of advertising
expenditures from US political campaigns. The ground truth for the dataset
includes values which have been manually recorded in an effort to make the
political process more transparent. Information about the location on the page
where entities were found is not available, and existing benchmarks on the
task employ a complex fuzzy matching process that attempts to align extracted
values with candidate tokens from the source documents. We selected the labels
”Gross Amount”, ”Committee”, ”Agency” and ”Call Sign” to include in our
experiments. DeepForm does not provide any standardised splits for the data,
and because the source documents are continuously pulled from a 3rd party
website, the documents that are included in the dataset can vary depending on
the time in which the dataset is downloaded. For this reason we run our own
baselines for DeepForm. We use the pdf-to-text pipeline and fuzzy matching
procedure from the DeepForm codebase and finetune a RoBERTa model Liu et al.
(2019) with a conditional random field output layer as our baseline. At
prediction we select the most confident candidate prediction. Additionally, we
write simple output standardization rules for gross amount to remove any
leading dollar signs and to convert the numbers back into the format used for
input.

###  A.3 OntoNotes 5.0

For OntoNotes, we attempt to reproduce the train, dev and test splits as
reported in Chiu and Nichols (2016) and Ghaddar and Langlais (2018). To this
end, we utilized the train and dev splits as produced by the script provided
in the CoNLL-2012 task Pradhan et al. (2012), and the test split as produced
by the script provided alongside Pradhan et al. (2013). Finally, following
Ghaddar and Langlais (2018) we exclude the New Testament portion, as it does
not contain gold standard NER labels.

| Beam Size | Epochs | Chunks | Lowercase | Shuffled Epochs | Strip Commas
---|---|---|---|---|---|---
NDA | 2 | 128 | 32 | Yes | Yes | Yes
Charity | 4 | 32 | 64 | Yes | - | Yes
DeepForm | 4 | 8 | 16 | Yes | - | Yes
DeepForm Baseline | - | 8 | - | - | - | -
Table 6: Best hyper parameters for each run. Learning rate was always $6.25e-5$ and batch size always $1$. Dataset | Label | Doc2Dict F1 | Baseline F1
---|---|---|---
DeepForm | Gross Amount | 0.853 | 0.98
DeepForm | Comittee | 0.844 | 0.615
DeepForm | Agency | 0.907 | 0.721
DeepForm | Callsign | 0.996 | 0.727
Deepform |  | 0.9 | 0.761
NDA | Party | 0.761 | 0.701
NDA | Juristiction | 0.961 | 0.938
NDA | Effective Date | 0.836 | 0.82
NDA | Term | 0.56 | 0.608
NDA |  | 0.809 | 0.777
Charity | Street | 0.571 | 0.682
Charity | Charity Name | 0.665 | 0.724
Charity | Income | 0.168 | 0.709
Charity | Postcode | 0.580 | 0.836
Charity | Charity Number | 0.827 | 0.967
Charity | Town | 0.659 | 0.833
Charity | Report Date | 0.932 | 0.957
Charity | Spending | 0.124 | 0.685
Charity |  | 0.561 | 0.801
Table 7: Full Per-field Results for Kleister-NDA, Kleister-Charity and DeepForm. Dataset | Split | Num Docs | Min Length | Max Length | Mean length
---|---|---|---|---|---
DeepForm | Train | 5981 | 272 | 114994 | 3715
DeepForm | Test | 2567 | 268 | 93450 | 3676
NDA | Train | 254 | 817 | 26109 | 3802
NDA | Dev | 83 | 494 | 10418 | 4070
NDA | Test | 203 | 751 | 14074 | 3802
Charity | Train | 1729 | 509 | 96669 | 12246
Charity | Dev | 440 | 379 | 122258 | 13950
Charity | Test | 609 | 520 | 365630 | 14272
Table 8: Dataset statistics measures in tokens of T5’s tokenizer.


the four components, accurately reproduces the total energy resolution (black
curve) obtained directly from the standard deviation of the total PE spectrum.

Figure 13: Energy resolution decomposition. The black curve refers to the
energy resolution obtained with total PE. The blue, red, dark red, and yellow
curves represent the contributions from scintillation Poisson statistics,
quenching effect, Cherenkov radiation and the covariance between
scintillation, and Cherenkov processes, respectively. The gray curve which
superimposes with the black curve is the root squared sum of the four
components.

It is worth noting that the standard deviation of the total PE is
significantly larger than the Poisson standard deviation, indicating the
presence of significant systematic effects in the energy resolution beyond
statistical fluctuations. This highlights the importance of considering these
systematic effects in the analysis of the energy resolution in the detector
simulation.

###  7.2 Decomposition of the energy resolution in the reconstruction

There are a few key factors which could affect the energy resolution in the
event reconstruction. Due to energy non-uniformity, the vertex uncertainty
will propagate to the energy resolution. A more precise vertex could improve
the energy resolution. Another important contributor comes from the PMT dark
noise, which contaminates the signal photons and worsens the energy
resolution. Thirdly, the inputs to the event reconstruction, namely the charge
and time of PMTs which are obtained from the PMT waveform reconstruction, the
charge non-linearity and waveform reconstruction uncertainty will propagate to
the energy resolution as well. Finally, the PMT charge is used to estimate the
number of detected PE. The intrinsic PMT charge resolution, which includes
contributions from both Gaussian and exponential components (Sec. 5.1.1), will
lead to charge smearing and potentially worsen the energy resolution for the
charge-based energy reconstruction. The impact of these factors on the energy
resolution is decomposed in this section. Different cases listed below were
compared, in which the factors above were removed sequentially.

  * •

Default Case: vertex and energy are simultaneously reconstructed.

  * •

Case A: true vertex is used in the energy reconstruction.

  * •

Case B: PMT dark noise is removed in the samples and true vertex is used.

  * •

Case C: on top of Case B, the waveform reconstruction is removed in the
reconstruction.

  * •

Case D: in addition to the changes in Case C, SPE charge resolution and other
electronics effects are also removed.

In the default case, the positron vertex is simultaneously reconstructed with
its energy using the method described in Sec. 6.2. The vertex resolution is
around 10 cm at 1 MeV and decreases at higher energies. Meanwhile, the vertex
bias is less than 2 cm within the FV. In case A everything is the same as the
default case, except that the MC truth vertex is used in the energy
reconstruction, which is equivalent to a vertex resolution of 0 mm. Besides
using the true vertex, the PMT dark noise is removed in case B. Given that
there is no more dark noise, the 420 ns signal window is discarded in both the
construction of the nPE maps as well as in the energy reconstruction. In case
C, the waveform reconstruction is replaced by a toy simulation, which smears
the charge of each PE, provides charge and time. This smeared PMT charge from
the toy simulation is used in both the construction of the nPE maps as well as
the energy reconstruction. In comparison, the charge reconstructed from PMT
waveforms are used in previous cases. By using these toy electronic simulation
samples, we are able to remove the impact from the waveform reconstruction as
well as other electronics effects except the PMT SPE charge smearing. The true
vertex is used for case C. For case D, the PMT SPE charge smearing is removed
on top of case C by directly using the detector simulation samples for the
reconstruction. As a consequence, the new observable is the number of PE,
instead of the reconstructed charge. This corresponds to the most ideal
scenario and leads to the best energy resolution.

Figure 14: Comparison of the energy resolution (left) and energy non-linearity
(right) among the different cases. The energy resolution for each case is
better with respect to the previous one, indicating that the energy resolution
improves by moving from the real to the ideal situation.

Figure 14 shows the comparison of the energy resolution and energy non-
linearity among all the cases. We can see that the energy resolution improves
monotonically among the cases, indicating that the energy resolution keeps
improving by removing each contributing factor. The fitting results of the
energy resolution for all the cases are summarized in Tab. 5. The relative
improvement of the energy resolution at 1 MeV for each case with respect to
the previous case is also presented. We can see that at 1 MeV, the PMT dark
noise and SPE charge smearing are the two dominant contributing factors to the
energy resolution with a relative improvement of about 3.45% and 2.1%,
respectively. Removing the charge uncertainty from the PMT waveform
reconstruction together with charge non-linearity leads to a 0.92% relative
improvement. The impact of vertex resolution is relatively small, the relative
improvement is about 0.78% using the true vertex.

Table 5: Comparison of the energy resolution among all the cases. The relative improvement of the energy resolution at 1 MeV with respect to the previous case is also shown in the last column. Case |  a [%] |  b [%] |  c [%] |  $E_{\textrm{res}}$@1 MeV [%] | Sequential improvement
---|---|---|---|---|---
Default | 2.614 | 0.640 | 1.205 | 2.948 | -
A (- vertex uncert.) | 2.581 | 0.667 | 1.206 | 2.925 | 0.78%
B (- dark noise) | 2.571 | 0.671 | 0.956 | 2.824 | 3.45%
C (- waveform reco) | 2.542 | 0.647 | 0.973 | 2.798 | 0.92%
D (- SPE charge smear) | 2.445 | 0.600 | 1.079 | 2.739 | 2.1%

The comparison above shows the average impact of each factor on the energy
resolution. In addition, it also enables us to decompose the energy resolution
at each discrete energy point. With the energy resolution curves of all the
cases, the default energy resolution can be decomposed into five components,
as it is shown in Fig. 15. The black solid curve corresponds to the default
case, which is used for the NMO analysis [14]. The teal solid curve
corresponds to case D and represents the ideal energy resolution we could
obtain. The four dashed curves show the contribution of each factor to the
energy resolution, and the values at each energy point are calculated as
$\sqrt{(E^{X}_{\textrm{res}})^{2}-(E^{Y}_{\textrm{res}})^{2}}$, where Y goes
from case A to D and X corresponds to the preceding case Y. Two features stand
out immediately: first, the contribution from dark noise decreases as the
energy increases as expected. Moreover, in the energy range below 1.5 MeV,
dark noise is the major contributor. For the energy range above 1.5 MeV,
particularly the most sensitive region (1.5, 3) MeV for the NMO analysis, the
SPE charge smearing is the dominant factor. Preliminary studies indicate that
its impact could be partially mitigated by PE counting, especially for PMTs
with 1 or 2 PE.

Figure 15: Decomposition of the energy resolution. The black solid curve
represents the default case. It is decomposed into five parts: the four dotted
curves correspond to the contribution of vertex resolution (red), dark noise
(green), waveform reconstruction (blue), and SPE charge smear (purple)
respectively. The light blue solid curve represents case D (ideal case).
Taking 1.022 MeV as an example, the values for all the curves are 3.12%,
0.35%, 0.83%, 0.39%, 0.58% and 2.90%, and the decomposition can be written as
$3.12^{2}=0.35^{2}+0.83^{2}+0.39^{2}+0.58^{2}+2.90^{2}$.

##  8 Summary

The energy resolution is a crucial parameter in determining the sensitivity of
the JUNO experiment to NMO. However, in addition to the statistical
fluctuations in the detected number of PE, several other effects impact the
energy resolution in the detection of the IBD signals. This paper presents a
comprehensive study of the energy resolution in the JUNO experiment,
incorporating the latest knowledge and updates in the detector construction
stage. This includes a better understanding of the detector structures, more
precise measurements of the optical properties of the LS, comprehensive
evaluations of the characteristics of both LPMTs and SPMTs, improved modeling
of the spectral and angular dependencies of the LPMTs’ PDE, and better
constraints on the absolute scintillation and Cherenkov light yield based on
data from the Daya Bay experiment. All of these updates have been implemented
in the JUNO detector simulation, which also includes detailed modeling of the
PMT responses and readout electronics in the electronics simulation.

By using data samples generated with the full detector and electronics
simulation, a full-chain data processing of calibration and reconstruction is
performed to evaluate factors that can smear the energy resolution. These
factors include residual energy non-uniformity after reconstruction, accuracy
of reconstructed energies from reconstruction algorithms, impacts of dark
count rate and SPE charge resolution, among others. After considering these
factors, it is found that an overall energy resolution of 2.95% at 1 MeV can
be achieved for positrons from IBD signals, and the obtained energy resolution
curve has been applied in the recent NMO analysis of JUNO. Furthermore, the
contribution of major effects in the energy resolution budget is estimated.
This study serves as a reference for interpreting future measurements of
energy resolution in JUNO data taking and provides a guideline for
understanding the energy resolution of LS-based detectors.

After the data-taking phase of the JUNO experiment begins, further updates and
improvements are expected in the understanding of the energy resolution. The
analysis of the collected data will provide valuable insights and allow for
refinements in the modeling and calibration of the detector. These updates and
improvements will contribute to the ongoing efforts to optimize the
performance of the JUNO detector and enhance its sensitivity to NMO.

## Acknowledgement

We acknowledge the Daya Bay collaboration for providing the Monte Carlo
simulation software. We are grateful for the ongoing cooperation from the
China General Nuclear Power Group. This work was supported by the Chinese
Academy of Sciences, the National Key R&D Program of China, the CAS Center for
Excellence in Particle Physics, Wuyi University, and the Tsung-Dao Lee
Institute of Shanghai Jiao Tong University in China, the Institut National de
Physique Nucléaire et de Physique de Particules (IN2P3) in France, the
Istituto Nazionale di Fisica Nucleare (INFN) in Italy, the Italian-Chinese
collaborative research program MAECI-NSFC, the Fond de la Recherche
Scientifique (F.R.S-FNRS) and FWO under the “Excellence of Science – EOS” in
Belgium, the Conselho Nacional de Desenvolvimento Científico e Tecnològico in
Brazil, the Agencia Nacional de Investigacion y Desarrollo in Chile, the
Charles University Research Centre and the Ministry of Education, Youth, and
Sports in Czech Republic, the Deutsche Forschungsgemeinschaft (DFG), the
Helmholtz Association, and the Cluster of Excellence PRISMA+ in Germany, the
Joint Institute of Nuclear Research (JINR) and Lomonosov Moscow State
University in Russia, the joint Russian Science Foundation (RSF) and National
Natural Science Foundation of China (NSFC) research program, the MOST and MOE
in Taiwan, the Chulalongkorn University and Suranaree University of Technology
in Thailand, and the University of California at Irvine in USA.

## References

  * [1] Particle Data Group, R. L. Workman et al., Review of Particle Physics,  PTEP 2022, 083C01 (2022).
  * [2] Hyper-Kamiokande, J. Bian et al.,  Hyper-Kamiokande Experiment: A Snowmass White Paper,  in 2022 Snowmass Summer Study, 2022, arXiv:2203.02029.
  * [3] DUNE, A. Abud Abed et al., Low exposure long-baseline neutrino oscillation sensitivity of the DUNE experiment,  Phys. Rev. D 105, 072006 (2022), arXiv:2109.01304.
  * [4] IceCube, M. G. Aartsen et al., Development of an analysis to probe the neutrino mass ordering with atmospheric neutrinos using three years of IceCube DeepCore data,  Eur. Phys. J. C 80, 9 (2020), arXiv:1902.07771.
  * [5] JUNO, Z. Djurcic et al., JUNO Conceptual Design Report,  (2015), arXiv:1508.07166.
  * [6] JUNO, F. An et al., Neutrino Physics with JUNO,  J. Phys. G 43, 030401 (2016), arXiv:1507.05613.
  * [7] JUNO, A. Abusleme et al., JUNO physics and detector,  Prog. Part. Nucl. Phys. 123, 103927 (2022), arXiv:2104.02565.
  * [8] JUNO, A. Abusleme et al., Sub-percent precision measurement of neutrino oscillation parameters with JUNO,  Chin. Phys. C 46, 123001 (2022), arXiv:2204.13249.
  * [9] JUNO, A. Abusleme et al., JUNO sensitivity to 7Be, pep, and CNO solar neutrinos,  JCAP 10, 022 (2023), arXiv:2303.03910.
  * [10] JUNO, A. Abusleme et al., JUNO sensitivity to low energy atmospheric neutrino spectra,  Eur. Phys. J. C 81, 10 (2021), arXiv:2103.09908.
  * [11] JUNO, A. Abusleme et al., Feasibility and physics potential of detecting 8B solar neutrinos at JUNO,  Chin. Phys. C 45, 023004 (2021), arXiv:2006.11760.
  * [12] JUNO, A. Abusleme et al., Prospects for detecting the diffuse supernova neutrino background with JUNO,  JCAP 10, 033 (2022), arXiv:2205.08830.
  * [13] JUNO, A. Abusleme et al., JUNO Sensitivity on Proton Decay $p\to\bar{\nu}K^{+}$ Searches,  Chin. Phys. C 47, 113002 (2023), arXiv:2212.08502.
  * [14] The paper is going to be released.
  * [15] JUNO, A. Abusleme et al., TAO Conceptual Design Report: A Precision Measurement of the Reactor Antineutrino Spectrum with Sub-percent Energy Resolution,  (2020), arXiv:2005.08745.
  * [16] JUNO, A. Abusleme et al., Radioactivity control strategy for the JUNO detector,  JHEP 11, 102 (2021), arXiv:2107.03669.
  * [17] JUNO, A. Abusleme et al., Mass testing and characterization of 20-inch PMTs for JUNO,  Eur. Phys. J. C 82, 1168 (2022), arXiv:2205.08629.
  * [18] Y. Wang, G. Cao, L. Wen, and Y. Wang, A new optical model for photomultiplier tubes,  Eur. Phys. J. C 82, 329 (2022), arXiv:2204.02703.
  * [19] Borexino, G. Alimonti et al., The Borexino detector at the Laboratori Nazionali del Gran Sasso,  Nucl. Instrum. Meth. A 600, 568 (2009), arXiv:0806.2400.
  * [20] KamLAND, S. Abe et al., Precision Measurement of Neutrino Oscillation Parameters with KamLAND,  Phys. Rev. Lett. 100, 221803 (2008), arXiv:0801.4589.
  * [21] JUNO, Daya Bay, A. Abusleme et al., Optimization of the JUNO liquid scintillator composition using a Daya Bay antineutrino detector,  Nucl. Instrum. Meth. A 988, 164823 (2021), arXiv:2007.00314.
  * [22] JUNO, A. Abusleme et al., The design and sensitivity of JUNO’s scintillator radiopurity pre-detector OSIRIS,  Eur. Phys. J. C 81, 973 (2021), arXiv:2103.16900.
  * [23] M. He et al., Design of the PMT underwater cascade implosion protection system for JUNO,  JINST 18, P02013 (2023), arXiv:2209.08441.
  * [24] JUNO, A. Abusleme et al., Calibration Strategy of the JUNO Experiment,  JHEP 03, 004 (2021), arXiv:2011.06405.
  * [25] V. Cerrone et al., Validation and integration tests of the JUNO 20-inch PMT readout electronics,  Nucl. Instrum. Meth. A 1053, 168322 (2023), arXiv:2212.08454.
  * [26] A. Coppi et al., Mass testing of the JUNO experiment 20-inch PMT readout electronics,  Nucl. Instrum. Meth. A 1052, 168255 (2023), arXiv:2301.04379.
  * [27] R. Triozzi et al., Implementation and performances of the IPbus protocol for the JUNO Large-PMT readout electronics,  Nucl. Instrum. Meth. A 1053, 168339 (2023), arXiv:2302.10133.
  * [28] JUNO, S. Conforti et al., CATIROC: an integrated chip for neutrino experiments using photomultiplier tubes,  JINST 16, P05010 (2021), arXiv:2012.01565.
  * [29] JUNO, A. Abusleme et al., The JUNO experiment Top Tracker,  Nucl. Instrum. Meth. A 1057, 168680 (2023), arXiv:2303.05172.
  * [30] B. Valeur and M. N. Berberan-Santos,  Excitation Energy Transfer (John Wiley & Sons, Ltd, 2012), chap. 8, pp. 213–261, https://onlinelibrary.wiley.com/doi/pdf/10.1002/9783527650002.ch8.
  * [31] J. B. Birks, Scintillations from Organic Crystals: Specific Fluorescence and Relative Response to Different Radiations,  Proc. Phys. Soc. A 64, 874 (1951).
  * [32] I. M. Frank and I. E. Tamm, Coherent visible radiation of fast electrons passing through matter,  Compt. Rend. Acad. Sci. URSS 14, 109 (1937).
  * [33] M. Bellato et al., Embedded readout electronics R&D; for the large PMTs in the JUNO experiment,  Nucl. Instrum. Meth. A 985, 164600 (2021), arXiv:2003.08339.
  * [34] GEANT4, S. Agostinelli et al., GEANT4–a simulation toolkit,  Nucl. Instrum. Meth. A 506, 250 (2003).
  * [35] J. Allison et al., Geant4 developments and applications,  IEEE Trans. Nucl. Sci. 53, 270 (2006).
  * [36] J. Allison et al., Recent developments in Geant4,  Nucl. Instrum. Meth. A 835, 186 (2016).
  * [37] J. H. Zou et al., SNiPER: an offline software framework for non-collider physics experiments,  J. Phys. Conf. Ser. 664, 072053 (2015).
  * [38] https://github.com/SNiPER-Framework.
  * [39] T. Lin et al., Simulation software of the JUNO experiment,  Eur. Phys. J. C 83, 382 (2023), arXiv:2212.10741,  [Erratum: Eur.Phys.J.C 83, 660 (2023)].
  * [40] C. Cao et al., Mass production and characterization of 3-inch PMTs for the JUNO experiment,  Nucl. Instrum. Meth. A 1005, 165347 (2021), arXiv:2102.11538.
  * [41] M. Daimon and A. Masumura, Measurement of the refractive index of distilled water from the near-infrared region to the ultraviolet region,  Appl. Opt. 46, 3811 (2007).
  * [42] X. Yang et al., Study on acrylic transmittance for JUNO Central Detector,  Rad. Det. Tech. Meth. 5, 284 (2021).
  * [43] M. He et al., Mass production and performance study on the 20-inch PMT acrylic protection covers in JUNO,  JINST 19, T05003 (2024), arXiv:2402.16272.
  * [44] J. Jyothi et al., Optical properties of TiAlC/TiAlCN/TiAlSiCN/TiAlSiCO/TiAlSiO tandem absorber coatings by phase-modulated spectroscopic ellipsometry,  Applied Physics A: Materials Science & Processing 123, 496 (2017).
  * [45] J. Apostolakis et al., GEANT4 low-energy electromagnetic models for electrons and photons,  (1999).
  * [46] Daya Bay, D. Adey et al., A high precision calibration of the nonlinear energy response at Daya Bay,  Nucl. Instrum. Meth. A 940, 230 (2019), arXiv:1902.08241.
  * [47] Borexino, G. Bellini et al., Final results of Borexino Phase-I on low energy solar neutrino spectroscopy,  Phys. Rev. D 89, 112007 (2014), arXiv:1308.0443.
  * [48] https://physics.nist.gov/PhysRefData/Star/Text/ESTAR.html.
  * [49] http://www.srim.org/SRIM/SRIM2011.htm.
  * [50] F.-H. Zhang et al., Measurement of the liquid scintillator nonlinear energy response to electron,  Chinese Physics C 39, 016003 (2015).
  * [51] X. Zhou et al., Rayleigh scattering of linear alkylbenzene in large liquid scintillator detectors,  Rev. Sci. Instrum. 86, 073310 (2015), arXiv:1504.00987.
  * [52] O. Perevozchikov, Study of KamLAND liquid scintillator nonlinear response to low energy responses, APS meeting, April 2027.
  * [53] Y. Zhang, Z.-Y. Yu, X.-Y. Li, Z.-Y. Deng, and L.-J. Wen, A complete optical model for liquid-scintillator detectors,  Nucl. Instrum. Meth. A 967, 163860 (2020), arXiv:2003.12212.
  * [54] D. Dörflinger et al., Measuring the fluorescence time profile of the JUNO liquid scintillator using gamma radiation and a pulsed neutron beam, Neutrino 2020 meeting.
  * [55] L. Xiao-Bo et al., Timing properties and pulse shape discrimination of lab-based liquid scintillator,  Chinese Physics C 35, 1026 (2011).
  * [56] D. J. Coumou, E. L. Mackor, and J. Hijmans, Isotropic light-scattering in pure liquids,  Trans. Faraday Soc. 60, 1539 (1964).
  * [57] J. Goett et al., Optical attenuation measurements in metal-loaded liquid scintillators with a long-pathlength photometer,  Nucl. Instrum. Meth. A 637, 47 (2011).
  * [58] http://www.hzcphotonics.com/products/XP72B20.pdf.
  * [59] R. Zhao et al., Afterpulse measurement of JUNO 20-inch PMTs,  Nucl. Sci. Tech. 34, 12 (2023), arXiv:2207.04995.
  * [60] S. Jetter et al., PMT waveform modeling at the Daya Bay experiment,  Chin. Phys. C 36, 733 (2012).
  * [61] Y. Huang et al., The Flash ADC system and PMT waveform reconstruction for the Daya Bay Experiment,  Nucl. Instrum. Meth. A 895, 48 (2018), arXiv:1707.03699.
  * [62] Q. Liu, M. He, X. Ding, W. Li, and H. Peng, A vertex reconstruction algorithm in the central detector of JUNO,  JINST 13, T09005 (2018), arXiv:1803.09394.
  * [63] Z. Li et al., Event vertex and time reconstruction in large-volume liquid scintillator detectors,  Nucl. Sci. Tech. 32, 49 (2021), arXiv:2101.08901.
  * [64] W. Wu, M. He, X. Zhou, and H. Qiao, A new method of energy reconstruction for large spherical liquid scintillator detectors,  JINST 14, P03009 (2019), arXiv:1812.01799.
  * [65] G. Huang et al., Improving the energy uniformity for large liquid scintillator detectors,  Nucl. Instrum. Meth. A 1001, 165287 (2021), arXiv:2102.03736.
  * [66] G.-h. Huang, W. Jiang, L.-j. Wen, Y.-f. Wang, and W.-M. Luo, Data-driven simultaneous vertex and energy reconstruction for large liquid scintillator detectors,  Nucl. Sci. Tech. 34, 83 (2023), arXiv:2211.16768.