# Two Kinds of Recall

Yoav Goldberg
Bar-Ilan University  Allen Institute for AI
<EMAIL_ADDRESS>
@yoavgo  Yoav Goldberg
Bar-Ilan University  Allen Institute for AI
<EMAIL_ADDRESS>
twitter: @yoavgo

This paper was rejected from ACL-2023. See reviews after the references.

###### Abstract

It is an established assumption that pattern-based models are good at
precision, while learning based models are better at recall. But is that
really the case? I argue that there are two kinds of recall: d-recall,
reflecting diversity, and e-recall, reflecting exhaustiveness. I demonstrate
through experiments that while neural methods are indeed significantly better
at d-recall, it is sometimes the case that pattern-based methods are still
substantially better at e-recall. Ideal methods should aim for both kinds, and
this ideal should in turn be reflected in our evaluations.

##  1 Introduction

A common folk wisdom in applied natural language processing (NLP) is that
there is an inherent tradeoff between pattern-based methods and machine-
learning based methods, in which the pattern-based methods are more precise,
while the learning-based methods identify more cases (have better recall).
With recent advances in neural-network based models, the learning based
methods became significantly more precise, leaving them with both good
precision _and_ better recall, compared to the pattern based methods.

However, I argue that this story is incomplete, and that pattern-based
methods, _while being significantly worse at recall_ than learning-based
methods, _can also at the same time be significantly better at recall_ than
learning-based methods. This is possible due to the existence of two distinct
kind of recall. The first kind of recall favors _diversity_ : finding many
variations of the same main theme. Learning-based methods indeed excel in this
kind of recall. The second kind of recall favors _exhaustiveness_ : once a
variation is known, finding all occurrences of instances of this kind. Here,
pattern-based methods shine, while learning-based methods sometimes still
struggle, or—even at the best case—are less reliable than pattern-based
methods.

Thus, I argue that when evaluating systems for recall, we should distinguish
between the two kinds, d-recall, reflecting diversity, and e-recall,
reflecting exhaustiveness. Ideal systems should achieve both kinds of recall
(while remaining precise). However, current datasets and evaluation methods
focus primarily on d-recall, while leaving e-recall as an incidental
afterthought.

I will demonstrate the two kinds of recall and the tendencies of neural vs.
pattern based systems on a task that combines sentence retrieval with relation
extraction: identifying sentences that contain an instance of some binary
relation that holds between two sentence entities.

The neural system is a RoBERTA-large111Liu et al. (2019) based Question
Answering model, trained on the SQuAD v.2.0 dataset.222Rajpurkar et al. (2018)
The pattern based system will rely on pattern matching over syntactic
dependency trees (which are automatically obtained through a neural-based
parser).

Note, however, the the aim of this paper is not to compare neural and pattern-
based methods for information retrieval, but rather to highlight the lack of
systematicity of learning-based methods, which result in sub-optimal e-recall,
and to shine a light on e-recall as an important property of language learning
systems which is currently not being properly evaluated. While the case-study
here is specific to information extraction, I believe the general trend holds
(but may be harder to show) for all extractive NLP tasks, including question-
answering, coreference resolution, named entity recognition, and others.

##  2 Background

###  2.1 Dependency Trees and Syntactic Patterns

Sentences have a syntactic structure that governs how the different words
relate to each other through various constructions Valin (2001). Syntactic
theories define these structures, and syntactic parsers recover them from
sentences. The relations between words are structural rather than semantic:
phrasing that mean the same thing, such as “Sam closed the door”, “The door
was closed by Sam”, “Sam, who closed the door” and “The door which was closed
by Sam” all have different syntactic structures.

In this piece I focus on _dependency-based_ syntactic analysis Kubler et al.
(2009), in which each word is a node in a directed tree, where some nodes are
parents of other nodes. The edges between tree nodes are labeled, reflecting
the syntactic relation that holds between the words. These are the syntactic
structures of two of the sentences above, according to version 1.0 of the
_universal dependencies_ syntactic analysis scheme Nivre et al. (2016).

[theme = default] [column sep=0.2cm] Sam & closed & the & door
[edge above,edge height=3ex]21nsubj [edge above,edge height=3ex]24dobj [edge
above,edge height=1ex]43det

[theme = default] [column sep=0.2cm] The & door & was & closed & by & Sam
[edge above,edge height=1ex]21det [edge above,edge height=3ex]42nsubjpass
[edge above,edge height=1ex]43aux [edge above,edge height=3ex]46nmod:by3ex
[edge above,edge height=1ex]65case

Neural-network based automatic parsers are good at recovering the underlying
syntactic structure of sentences, with high accuracy and high consistency
Kiperwasser and Goldberg (2016); Dozat and Manning (2017); Honnibal and
Montani (2017).

#### Syntactic Patterns

Some paths in a syntax tree correspond to semantic relations. Thus, one can
extract information from text by first obtaining syntactic parse trees, and
then looking for specific patterns in these trees. For example, the pattern:

[theme=default] [column sep=0.2cm] ___a1 & was & educated & at & ___a2
[edge above,edge height=3ex]31nsubjpass [edge above,edge height=1ex]32aux
[edge above,edge height=3ex]35nmod:at [edge above,edge height=1ex]54case3ex

will match sentences whose syntactic trees contain the above subtree. This
includes sentences such as:

1\. _He a1 was born in London , and was educated at University College ,
Londona2, of which he subsequently became a fellow ._

2\. _Born to a Sikh family on 4 May 1922 , Talwara1 was educated for a short
time at the Lawrence School , Sanawara2 , along with his brother Rana Talwar
._

3\. _Paul a1 was primarily educated at St. Jarlath ’s Vocational Schoola2 ._

We can further restrict the pattern, requiring for example for the subject
position $a1$ (indicated by an incoming nsubjpass edge) to contain a proper-
noun and not a pronoun, ruling out sentence (1) above.

Syntactic patterns are interpretable, efficient and precise, but have a clear
drawback: there are many ways to express the same semantic relation, and we
need to tailor a distinct pattern for each one. In other words, _the patterns
have low d-recall_.

###  2.2 Extractive Question Answering and the SQuAD dataset

SQuAD (“Stanford Question Answering Dataset”) Rajpurkar et al. (2016, 2018) is
a collection of over 150,000 <question, passage> pairs, where the answer to
the question is a span in passage. The SQuAD dataset is used to train machine
learning models to perform _extractive QA_ : given a question and a passage,
mark the passage span that answers the question. In the real world, not every
passage contains an answer to every question. To make the task more realistic,
the SQuAD v 2.0 dataset Rajpurkar et al. (2018) also includes <question,
passage> pairs where the question is not answerable by the passage. Now, the
model is tasked with identifying the answer span, or indicating that no
relevant answer span exists (“no answer”). The answerable question-passage
pairs were created by showing human annotators a passage, and asking them to
ask a question and mark its answer in the text. The annotators were also
encouraged to not repeat the passage phrasing in their questions.

The result is a set of many entity-centric questions, where the question
contains an entity and the answer span contains a related entity. For example,
“Which gas makes up 20.8% of the Earth’s atmosphere?” (answer: diatomic
oxygen). The SQuAD dataset and the extractive-QA task became very popular, and
models “solve” the SQuAD benchmark with very high accuracies.333See e.g.
systems’ scores at https://rajpurkar.github.io/SQuAD-explorer/) As a result,
SQuaD-based models are often used as of-the-shelf NLP components. For example,
the model deepset/roberta-base-squad2 available on Huggingface models
hub,444https://huggingface.co/models has been downloaded over 650,000 times in
Jan 2023 alone, and the more accurate but also more compute intensive
deepset/roberta-large-squad2 was downloaded over 35,000 times in the same
month.

##  3 Experiments

The following experiments are all performed through the SPIKE system Shlain et
al. (2020); Taub Tabib et al. (2020), available at
https://spike.apps.allenai.org/. The system allows to perform various query
types over corpora which is pre-annotated with linguistic structures
(including syntax, POS-tags, lemmas and named entities). The syntactic
annotation in the SPIKE-indexed corpora are based on the universal
dependencies v1 scheme, enhanced with additional (non-tree) arcs by the
pattern-based pyBART system Tiktinsky et al. (2020). The SQuAD experiments are
based on a model downloaded from Huggingface model hub. The model is
deepset/roberta-large-squad2, the largest version of the most-popular squad2
model on huggingface. This model is based on the RoBERTA large pre-trained
transformer Liu et al. (2019), and is considered to be a strong extractive
question-answering model.

###  3.1 Demonstrating d-Recall

In the first experiment, I retrieve a collection of PubMed sentences
containing the words “pain” and “molecule”. I treat each of the 380 returned
sentences as a passage in a QA setup, and ask the SQuAD model “which molecule
relates to pain?”. This leaves us with 282 sentences marked as No-Answer, and
additional 98 sentences in which the QA model identified an answer span. Most
of these answer spans are correct, and the answers are diverse in terms of
their underlying structures and words, for example:

1\. _… we present evidence that … molecule , nitric oxide ( NO ) may act at
several levels of the nervous system during the development of experimental
neuropathic pain ._

2\. _TRPA1 , a member of the transient receptor potential channel ( TRP )
family , is genetically linked to pain in humans , and … _

3\. _TRPV1 : an important molecule involved in the peripheral sensitization
during chronic pain ._

4\. _The TRPV1 is a well-known pain transducer molecule and plays crucial
roles in the perception of inflammatory and thermal pain ._

In a similar experiment, I retrieve Wikipedia sentences including the words
“police”, “arrest”, and a named person, and ask the QA engine “who escaped?”.
This results in answers such as:

1\. _… reports Debbie to police , and Debbie is last seen running down an
alley to escape after officers arrive at the Gallagher house to arrest her for
…_

2\. _She calls the police with a story about how Dirk shot Johnny and then ran
away , but they arrest her when they …_

The diversity of these answers clearly demonstrate the d-recall abilities of
the neural QA models. Extracting these answers using syntactic patterns would
be a non-trivial challenge.

###  3.2 Demonstrating (lack of) e-Recall

In the second set of experiments, I start with queries which are based on
syntactic patterns. For the first query, I queried for the syntactic pattern:

____a1 _was educated at_ ____a2
demonstrated above, restricting the $a1$ position to be a person’s name.

Running this query over the Wikipedia corpus results in a huge number of
results, of which we take the first 5,000 matches. By design, all the results
clearly indicate the “PERSON educated at PLACE/School” relation, via a
recurring syntactic pattern. We now ask the SQuAD model, on each of these
sentences,

Figure 1: Sample sentences on which the roberta-large-squad2 model returned
No-Answer for the question “who was educated?”. The marked spans are based on
the syntactic-pattern match.

“Who was educated?”

While the SQuAD model got many answers correctly, it also identified 611 of
the sentences as No-Answer cases. Inspecting these sentences reveal that they
all contain explicit and clear answers to the question. A sample of such No-
Answer sentences is given in Figure 1. Note that the set on which the SQuAD
model returns _No-Answer_ includes very direct and concise sentences, such as:

\- _Evans was educated at Haverford College_.

Question variations such as “Who was educated somewhere?”, “Who was educated
at a school?”, “Who was educated at a university?”, “Who studied at a
university?” etc, result in very similar trends, with hundreds to thousands
(!!) of No-Answer results for each question.555Additionally, in some of these
question the model becomes less precise, marking spans such as “Eton College”
and “University of Glasgow” as answers.

Similarly, I ran the syntactic pattern:

_DISEASE was treated with CHEMICAL_

over the PubMed corpus (DISEASE and CHEMICAL are entity-type restrictions),
resulting in 3654 medical-condition / chemical pairs. Asking the QA model
“which condition was treated?” results in 721 No-Answer cases, including from
sentences such as:

\- _The infection was successfully treated with local voriconazole followed by
oral terbinafine_; and
\- _Early muscle pain was treated with Ibuprofen_.

Question variations such as “what was treated?” result in a similar number of
No-Answer cases, where the the sets of No-Answer cases overlap, but are not
the same.

This trend by which the QA model fails to identify many answers recovered by
the syntactic pattern query is remarkably consistent for pretty much all
syntactic patterns I tried.

There are possible explanations for the SQuAD model behavior: for example, the
questions refer to more abstract concepts than what the model is used to, and
the model’s training data led it to be especially careful around potential
answers where the text around the answer span is too similar to the question
text.

Yet, regardless of these reasons, the observed behavior remains: the model
manages to correctly identify the answer is some of the cases, but also fails
to identify the answer on very many others: it is failing in e-recall. The
pattern based methods, in contrast, are naturally exhaustive in these cases,
and can consistently identify cases which are not identified by the neural QA
model.

###  3.3 …but did you try GPT-3?

When working on the final version of this paper, on March 18, 2023, I also
performed a quick test with the text-davinci-003 model. For the first 100
wikipedia results for the _PERSON was educated at_ pattern, I queries the
model with the following prompt:

based on the following text, who was educated somewhere? write only the name,
and "None" if no one was educated.
text: {text}

While text-davinci-003 is obviously a very strong model, it still fails to
demonstrate perfect e-recall. Out of the 100 sentences with a very clear
person-educated-at pattern, text-davinci-003 failed to extract the person name
from 2 of them, extracting the school name instead.

While 98% e-recall is impressive, we should note that it was over a set in
which a pattern-based system obtained 100% e-recall, and with a weird failure
mode of identifying the wrong argument with a different semantic type.

##  4 Discussion

An obvious response to the SQuAD experiment above is “we just need to add some
results of the pattern-based queries to the QA model’s training data”. Indeed,
this will likely “solve” the issue described above, in the sense that it will
be hard to demonstrate low e-recall of the resulting QA model, using the same
technique. But I argue that such a “solution” will only hide the underlying
problem, and not solve it. The above experiment demonstrated one class of
instances in which the QA model is missing many valid answers. But do we know
it is the only one? I argue that it is very likely _not_ the only e-recall
failure mode of neural QA models, it is only one which was convenient to
demonstrate. If we block this particular demonstration technique, it does not
mean the models will stop having other blind-spots.

##  5 Going Forward

I have shown that there are two kinds of recall, one based on diversity, and
another that is based on exhaustiveness. Currently, pattern-based methods
still excel at ensuring exhaustiveness, while neural methods struggle.
Moreover, while it is almost trivial to bound the exhaustiveness of pattern-
based methods (for example, syntax-based methods are as exhaustive as the
accuracy of the underlying parse trees), it is currently impossible to do so
for learning-based methods. And crucially, our evaluation methods don’t test
for exhaustiveness (and for an understandable reason: it is hard to do well,
compared to diversity-based tests).

Recall of an ideal system should be both diverse _and_ exhaustive. Recognizing
the existence of e-recall and keeping it in our minds as we develop future
methods, is a good first step towards achieving this ideal. Vocabulary is
important in establishing norms, and I would be happy if the e-recall and
d-recall vocabulary (or a variation of it) would be adopted. However, while
the recognition of a problem and the establishment of vocabulary around it is
a good and necessary start, it is not sufficient. It is very hard to make
progress on what we cannot reliably measure. This is thus a call for a new
challenge to the community: developing robust methods of assessing e-recall,
for a wide set of language understanding problems.

## Limitations

The work presents an opinion, and like any opinion, should be assessed and not
taken for granted just because it is written. While the opinion is backed by
evidence, this evidence is reported as anecdotal cases, and not as a thorough
quantitative study (for example, I could have repeated the experiment in 3.2
with hundred different syntactic patterns, and counted the number of no-answer
cases in each. I chose not to do that. I hope the readers believe me that the
results are indeed consistent across many patterns). This could be considered
either is a limitation, or, as I rather see it, as a stylistic choice which is
fitting of opinion papers like this one, who aim to make a point through
demonstration, but where the point holds regardless of the exact numbers that
could be measured.

## Ethics Statement

To the best of my judgement, this work does not raise any significant ethical
concerns.

## Acknowledgements

This project has received funding from the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme,
grant agreement No. 802774 (iEXTRACT).

## References

  * Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017.  Deep biaffine attention for neural dependency parsing.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016.  Simple and accurate dependency parsing using bidirectional LSTM feature representations.  _Transactions of the Association for Computational Linguistics_ , 4:313–327.
  * Kubler et al. (2009) Sandra Kubler, Ryan McDonald, and Joakim Nivre. 2009.  _Dependency Parsing_.  Morgan and Claypool Publishers.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016.  Universal Dependencies v1: A multilingual treebank collection.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 1659–1666, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Shlain et al. (2020) Micah Shlain, Hillel Taub-Tabib, Shoval Sadde, and Yoav Goldberg. 2020.  Syntactic search by example.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 17–23, Online. Association for Computational Linguistics.
  * Taub Tabib et al. (2020) Hillel Taub Tabib, Micah Shlain, Shoval Sadde, Dan Lahav, Matan Eyal, Yaara Cohen, and Yoav Goldberg. 2020.  Interactive extractive search over biomedical corpora.  In _Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing_ , pages 28–37, Online. Association for Computational Linguistics.
  * Tiktinsky et al. (2020) Aryeh Tiktinsky, Yoav Goldberg, and Reut Tsarfaty. 2020.  pyBART: Evidence-based syntactic transformations for IE.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 47–55, Online. Association for Computational Linguistics.
  * Valin (2001) Robert D. van Valin, Jr. 2001.  _An Introduction to Syntax_.  Cambridge University Press.

##  Appendix A Reviews and (some) Responses

The paper received very low scores in ACL 2023, and I decided to post it on
arxiv rather than going through the author-response process. Here are the
reviews, with some responses inline. The aim of my responses is not to “defend
myself” but rather to highlight differences between me and “the community” (as
reflected by the reviewers) regarding our beliefs of what are desired
qualities of a position paper. I hope to convince future reviewers of
position-like papers to take a stance closer to the one I present here than to
the one the reviewers took. I also answer questions where I believe readers
will have a similar question to the reviewer, and may benefit from a clear
answer.

The crux of the issue is that I attempted to highlight a deficiency of neural
models and tell a story about evaluation and a metric which we do not
currently evaluate (e-recall). To demonstrate the point, I used an experiment
tha anecdotally demonstrated the point. Indeed, the reviewers “got” the point.
But they also focused on the more “expected story” of a competition between
models. As such, they were looking for a more “rigorous” setup for the
comparison, that will “better convince” them that “the results hold”. But for
me the results are just an anecdote, and are not the point. I argue that we
should be able to let go of our fixation on numbers and comparisons, and try
to identify broader stories when they exist. Especially in position-type
papers. (Some of the blame is on me for not making this clearer. In the
current version of the paper, I added the last paragraph of the introduction,
which was not present in the submitted version, in an attempt to state the
point of the paper more explicitly.)

###  A.1 Review 1

#### What is this paper about and what contributions does it make?

This paper argues that in NLP tasks, we should consider "two kinds of recall”:
"d-recall,” which emphasizes the diversity of cases correctly identified by
some system, and "e-recall,” which emphasizes the exhaustiveness of cases
correctly identified by some system. This distinction is motivated by a
supposed assumption that pattern-based systems for NLP are better for
optimizing precision than neural systems, and neural systems are better for
optimizing recall. I like to think of this distinction as follows: d-recall
pertains to how diverse are the correctly identified variations within some
class, while e-recall pertains to the proportion of some variation within a
class that are correctly classified. The paper presents some anecdotal
evidence that if we think of d-recall and e-recall separately, we see that
neural systems are good at d-recall, but less so at e-recall. The target task
is relation extraction on sentences retrieved from a corpus, which is done
through two comparison methods: 1) using a dependency parser-based retrieval
mechanism, and 2) using a neural extractive question answering model which is
a RoBERTa-based language model fine-tuned on SQuAD. Several experiments are
performed by first using the parser-based model to retrieve N examples of some
relation, e.g., "X was educated at Y” from some corpus, then applying the
neural QA model. Then the author examines the outputs of the neural QA model
to investigate their diversity and exhaustiveness.

Specifically, the author finds that if we consider a set of sentences of
diverse syntactic structure, the neural QA model can correctly extract a
target relation from a breadth of them. Meanwhile, if we retrieve sentences of
the same syntactic structure, the neural QA model misses many target relations
in them. This provides some evidence of the strength of the neural QA model
with respect to diversity, but the brittleness of it with respect to
exhaustiveness.

While I think the distinction of these "two kinds of recall” is novel and
important in the context of these fine-tuned language models, which are widely
known to spuriously change their outputs with small, inconsequential changes
to the inputs, this paper does not put in the effort to warrant publication.
The paper attempts to make conclusions based on anecdotal evidence, where
there are missed opportunities for proper empirical studies. With more effort
toward clear definitions of concepts introduced, and comprehensive empirical
studies and analysis, this could be a substantial contribution.

YG: Clearly the reviewer and myself see the role of position-papers
differently. The experiments were meant to demonstrate the issue and convince
people that it exists and is interesting, not to quantify it. As the reviewer
notes later, they understood the concepts and found them interesting. I don’t
see any benefit in such quantification, and, as a reader, would have found
them boring to read. I rather see concrete cases over aggregate numbers.

#### Reasons to accept

The paper proposes novel (to my knowledge) concepts of d-recall (recall
favoring diversity) and e-recall (recall favoring exhaustiveness). This
refinement of the recall metric could be a helpful way to measure the
spuriousness of neural text classifiers. While the experiments are not
conclusive or complete, I found the problem formulation to be an interesting
way to expose these two kinds of recall.

#### Reasons to reject

The most glaring issue with this paper is that the two kinds of recall are not
clearly defined evaluation metrics. I can think of a number of ways we can
attempt to do this, e.g., by measuring the lexical or syntactic diversity of
correctly classified examples within each class, but this effort was missing.

Almost all of the results presented are qualitative. The author only presents
anecdotal evidence and cherry-picked examples, which is not conclusive and not
rigorous enough for scientific publication. Even the quantitative results
presented are not conclusive, as they are simply counts of supposedly
misclassified examples by the neural QA model; but the data is not human-
annotated and has no ground truth labels to my understanding, which means it
could be generally noisier than the examples presented in the paper. This
paper would be much more convincing if applied to some existing, human-
annotated benchmarks, and we could calculate quantitative metrics to compare
the d-recall and e-recall on different subsets of the data.

YG: While “scientific rigor” is important in technical contributions, we
should be careful not to throw the baby with the bathwater when insisting on
it for position-like papers. The concepts of “d-recall” and “e-recall” I
present are broader than just the information-extraction task I demonstrate
them on. My intent was to raise awareness to a general phenomena, and
hopefully ignite research around its solution. By providing a clear definition
and evaluation metric, I’d be severely limiting the readers into a pre-
specified solution, rather than letting them find their own, which may be
better than mine (and I’d also be restricting them to the context of
information-extraction).
The call for using “an existing, human annotated benchmark” is also very
limiting in this case, as existing benchmarks are not well suited for
measuring e-recall (as stated in the intro “current datasets and evaluation
methods focus primarily on d-recall, while leaving e-recall as an incidental
afterthought”). More broadly, reliance on established benchmarks is both a
driving force and a restraining force. We should be careful not to restrain
too much.

The paper only considers one task, one type of neural model, and one type of
pattern-based model. In order to demonstrate the general conclusions made, a
more comprehensive study may be necessary. If not, then the paper should be
more clear about exactly where the conclusions apply. YG: This seems to view
the paper as a claim about the relative performance of different models. This
misses the broader—and to me substantially more important—message: there is a
kind of recall that we as a community do not properly evaluate, and which
current systems often fail on.

#### Questions for the Author(s)

\- Question A: In the experiment mentioned in L204, where are the sentences
retrieved from? How many were retrieved?
YG: Who cares? Does knowing it change the picture in any meaningful way? The
context is showing a bunch of sentences over which a SQuAD model works well,
in-line with the established assumption in the community that SQuAD models
work well. What does it matter where the sentences came from, or what the
larger group size was? (the answer is Wikipedia, which was removed for
submission version to fit in 4 pages, and then added back in the current
version).

\- Question B: Applying a model trained on SQuAD to single sentences is a bit
out-of-domain, as these models are trained on full passages. Could that
influence the trend you’re seeing? Did you try any other types of neural
models?
YG: If it does, does it change the large picture in any way? If a model often
work but sometimes fail when applied to a sentence rather than a paragraph,
when the sentence is sufficient to answer the query, what does it tell us
about the robustness of this model? Moreover, I’d argue that until we (NLP
community) have a good notion of what a “domain” is, we should be careful in
using “domain shift” as an excuse for model failures.

\- Question C: Is the comparison between pattern-based and neural approaches
fair? You seem to apply the neural model to only sentences retrieved by the
pattern-based approach. What do your results say about pattern-based
approaches, if anything?
YG: The results demonstrate that there is a set of sentences for which the
neural model has low e-recall. Note that the reviewer is again tempted to
force a story about model comparison, and not about two kinds of recall.

\- Question D: Why not try the experiments you suggest in the beginning of
Section 4 then run some out-of-domain evaluation setting? Then you can
actually support the claims here.
YG: Which claims, and how will it help the two kinds of recall story?

\- Question E: RoBERTa is a bit old at this point. Do you plan to apply this
study to any newer models to understand the variation with model size,
training data size, etc.?
YG: This continues to seek the “let’s compare models” story, rather than the
“two kinds of recall” story.

#### Missing References

While it’s a common thought that precision and recall can sometimes be at odds
when building models for NLP tasks, I’m not sure I’ve heard of this assumption
that pattern-based methods would be better at precision, and machine learning
methods better at recall. While I don’t find it surprising, a broad statement
like this about the field should be justified if it will motivate the entire
work. Please support this with some relevant references. I wasn’t able to find
any to point to myself… This paper could use a proper literature review. I
think some parts may be borrowed from Section 2 here.

#### Typos, Grammar, Style, and Presentation Improvements

In the abstract, please give some specific ideas about the task, models, and
final conclusions considered in this work.
YG: The abstract lists the conclusion very clearly: _“Ideal methods should aim
for both kinds [of recall], and this ideal should in turn be reflected in our
evaluations”_. The reviewer is again trying to fit a model-comparison story,
which should be “scoped”.

Recall is a metric used in all kinds of scientific disciplines, and this
notion of "two kinds of recall” may not apply to all of them, so it would help
to scope down a bit from the beginning. In Section 1, a figure visualizing the
difference between d-recall and e-recall would be helpful, or at least some
more details about how e-recall is defined.

L087: "autoamtic” -> "automatic”

L093: grammatical error; remove "a”?

Section 2 could use a bit more attention to organization. First, it may be
more appropriate to call it a "Methods” section rather than "Background.” Some
introductory sentences in each subsection, and some conclusive sentences about
how the syntactic patterns, SQuAD dataset, etc., are actually applied in the
work would help the reader follow this section. Section 2.2 is one long
paragraph - can we break it down a little?

L159: "experiements” -> "experiments”; "perfromed” -> "performed”

L164: "annoatated” -> "annotated”

I found the first paragraph of Section 3 hard to follow. In separate
paragraphs, please introduce the pattern-based system implementation, and the
neural-based system implementation.

Remove "(!!)” in L250.

The results in Section 3 need to be presented completely differently. There
are too many changes to list here, but it would be much easier to follow if
results were presented in a table and some actual evaluation metrics were
calculated - don’t just cherry pick examples and list counts of misclassified
examples. Reduce the amount of qualitative discussions about results, e.g., at
L246, L278, L282, and replace with proper empirical studies where possible.
This is hard to follow and completely inconclusive.

Somewhere before Section 3, the problem should be formally defined. Section 3
is a bit hard to follow because the problem is not clearly defined.

YG: (I fixed the typos and ignored the other stylistic suggestions.)

Soundness: 1
Excitement (Short Paper): 1.5
Reviewer Confidence: 4
Recommendation for Best Paper Award: No
Reproducibility: 3
Ethical Concerns: No

YG: I wouldn’t necessarily judge the soundness of a position paper by the
exhaustiveness of its experiments.

###  A.2 Review 2

#### What is this paper about and what contributions does it make?

This paper discusses the properties of recall-based evaluation, distinguishing
the recall into a "d-Recall" reflecting on response diversity and an
"e-Recall" reflecting on response exhaustiveness. The author argues that while
recent neural models are very good at d-Recall, they lack on e-Recall even
compared to pattern-based methods. However, it is desirable to have systems
whose responses are both diverse and exhaustive, i.e. complete. They present
an example to support this assumption by showing high diversity of answers of
a neural extractive QA model (RoBERTa-Large), and its inferiority on the
completeness of the answers when asked to answer questions about contexts
retrieved by a pattern-based method based on syntactic trees matching.
Finally, the author argues that current evaluation methods favour diversity in
evaluation and, thus, that future evaluation benchmarks should aim to take
both aspects of recall into consideration.

#### Reasons to accept

(i) The paper is easy to follow, giving priority to clear, short statements
with minimal turns from the main line of argumentation.
(ii) The distinction of "diversity" from "exhaustiveness" in the evaluation of
systems responses makes sense to me. I think its distinguishing in evaluation
testbeds could allow the community to identify important differences in the
functioning of NLP systems while motivating future work in clearing out blind
spots of some existing methods.

#### Reasons to reject

I understand that a more extensive evaluation might fall outside the scope of
a position paper, but having trouble relating to the main assumptions, I still
had to rely on the evidence given in the example evaluations. My first two
points refer to the problems that I identify in these.
(i) The paper supports "high d-Recall of neural systems" assumption by
presenting the example of neural QA behaviour in responding to the question
"which molecule relates to pain?" over a PubMed corpus and showing answers to
these questions. This evaluation shows the performance of QA system trained on
SQuAD under quite a large domain shift in both the question and contexts.
Further, it evaluates the diversity of the responses subjectively, without
comparison to the pattern-based responses. Therefore, from this case, it is
hard for me to relate to the claim that the diversity of QA answers "clearly
demonstrates" the diversity quality of neural models (L216).
YG: This reviewer again focuses on the comparison between neural and pattern
based system. And also, like reviewer 1 is suddenly having trouble believing
that “neural methods are good at diversity”.
(ii) "Lack of e-Recall" claim is in the paper supported by a high number of
"not-answerable" responses of QA system on pattern-retrieved responses to some
specific patterns. I am happy to believe that the situation is similar with
other patterns (mentioned in Limitations, L342), but this approach also does
not give a measured e-Recall of QA model into any context. Would the
conclusions be the same even if the QA responds "no answer" for, say, 200 or
20 examples? I can not reconstruct any general conclusions from such an
evaluation.
YG: This infatuation with numbers again, and with comparisons / looking for
thresholds. I really fail to see why the exact numbers matter here, or why a
number of 200 will tell a different story w.r.t the existence of an e-recall
failure of the neural model.
Even assuming that the pattern-based methods return all correct responses, to
show the difference, the same experiment could be done the other way around,
to get some context: assuming that the QA responses (excluding "not-
answerable") are all valid and matching them using the pattern-based method.
YG: No, this would be measuring d-recall of the pattern-based system (which we
know is bad).
(iii) The paper’s overall objective seems to be to motivate follow-up work to
create benchmark(s) that will reflect equally on both kinds of recall.
However, after reading the paper and looking at the examples, I have no idea
how I would proceed towards that goal. This raises doubts in me on the
usefulness of this work in a broader scope, as currently framed.
YG: This is what makes the challenge a hard one, and why I think it warrants a
position paper in the “reality check” theme track.

#### Questions for the Author(s)

\- Question A: On L317, you state that "our evaluation methods don’t test for
exhaustiveness", but I am having difficulty to see at this point how this is
the case. Can you give examples of this assumed blind spot on examples of
specific, existing NLP benchmarks?
YG: Models have very good performance on existing benchmarks. And benchmarks
are not collected with many examples that follow a shared and simple pattern.

\- Question B: Can you be more specific in proposing the evaluation technique
that would consider both kinds of recall while still being applicable to the
broad coverage of NLP tasks?
YG: I think this should be defined on a per-task basis. And if I knew the
answer, I’d be writing a resources-and-evaluation paper and not a position-
paper in the theme track.

#### Typos, Grammar, Style, and Presentation Improvements

Typos (some of which could be uncovered by simple type checking tools): L56:
SqUAD -> SQuAD

L93: Correspond to a semantic … -> Correspond to semantic

L98: Will match -> will match

L150: SQuaD based-models -> SQuAD-based models

Soundness: 2
Excitement (Short Paper): 2.5
Reviewer Confidence: 3
Recommendation for Best Paper Award: No
Reproducibility: 4
Ethical Concerns: No

###  A.3 Review 3

#### What is this paper about and what contributions does it make?

The paper explores the performance of pattern-based and learning-based models
in terms of diversity and exhaustiveness of recall, introducing the concepts
of d-recall and e-recall. Through experiments, the study suggests that neural
methods are better at d-recall, while pattern-based methods are still
substantially better at e-recall. The paper argues that ideal methods should
aim for both kinds of recall and proposes that evaluations should reflect this
ideal.

#### Reasons to accept

Introduces the concepts of d-recall and e-recall, which could potentially be
useful for evaluating and optimizing machine learning models in certain
contexts.

Proposes that evaluations should aim for both kinds of recall, which could
lead to more nuanced and comprehensive evaluations of machine learning models.

The paper is easy to follow

#### Reasons to reject

The paper seems to present a specific and limited contribution, focusing on
the performance of pattern-based and learning-based models on SQuAD in terms
of d-recall and e-recall. While this may be an interesting and relevant
problem, it is not clear how significant or generalizable the findings are,
and whether they justify a full paper.

The paper does not provide enough context or related work to understand the
novelty or importance of the proposed concepts of d-recall and e-recall.

Experiments are too simple to support that neural methods are better at
d-recall, while pattern-based methods are still substantially better at
e-recall. This is examined simply by two example questions: "which molecule
relates to pain?” (for d-recall) and "Who was educated?” (for e-recall). I
also feel why the pattern-based method can do e-recall better in this case is
because pattern for "was educated at" is too simple. If the authors test on
more difficult problems, the conclustion may be different.

Overall, the paper lacks depth, and the contributions and implications of the
study are not well-established.

#### Questions for the Author(s)

If we lower the threshold of prediction, could learning-based models achieve a
higher e-recall? Is it meaningful to evaluate the recall without considering
precision? YG: I evaluate e-recall on cases where the answer exists and is
easily identified by a parser-based method. Changing the threshold will indeed
result in fewer “no answer” cases, but will also be abysmall in precision on a
more diverse dataset.

#### Typos, Grammar, Style, and Presentation Improvements

Line 222: a queries

Soundness: 2
Excitement (Short Paper): 2
Reviewer Confidence: 4
Recommendation for Best Paper Award: No
Reproducibility: 4
Ethical Concerns: No


[a]Christopher Chamness

# Trivializing Flow in 2D O(3) sigma model

Konstantinos Orginos  Daniel Kovner

###### Abstract

The two dimensional O(3) sigma model, just as quantum chromodynamics, is an
asymptotically free theory with a mass gap. Therefore, it is an interesting
and simple toy model to investigate algorithms for Markov Chain Monte Carlo
simulations of quantum chromodynamics. In this talk, we discuss the
construction of a trivializing map, a field transformation from a given theory
to a trivial one, through a suitably chosen gradient flow. An analytic
solution for the generating functional of this trivializing flow has been
obtained by a perturbative expansion in the flow time. Utilizing this solution
allows for new approaches to be considered when proposing updates for a Markov
Chain Monte Carlo algorithm.

JLAB-THY-24-3981

##  1 Introduction

The idea of using a Trivializing Map for improving Markov Chain Monte Carlo
(MCMC) calculations in Quantum Chromodynamics (QCD) has been proposed more
than 10 years ago [1]. In this approach, the field variables that satisfy a
non-trivial distribution are mapped to new variables that satisfy a trivial
distribution. This is done by via a bijective map that transforms the trivial
probability density distributions to the target one. Although, a very
promising idea, up to now it has not found an efficient application to QCD. In
this work, we turn our attention to a much simpler model, the O(3) sigma model
in two dimensions. This model, shares with QCD the properties of asymptotic
freedom [2] as well as the existence of mass gap [3].

It is, therefore, a non-trivial theory that might help us understand the
properties of trivializing maps that could be relevant to QCD. Furthermore,
this toy model is less computationally intensive and allows for new algorithms
to be designed and tested much quicker than would be possible in QCD [4]. A
similarly motivated study has already been made in the case of the CP(N-1)
model in two dimensions [5], where the leading order approximation in the flow
time expansion was used to study the potential of improvement of the
corresponding Hybrid Monte Carlo algorithm. In this contribution, we discuss
the perturbative flow time expansion of the generating functional of the
gradient flow which defines our trivializing map.

##  2 Trivializing Map

A Trivializing Map [1] may be defined as a map between the initial condition
and the solution of a gradient flow at a specific flow time t. Without loss of
generality, this time is taken to be $t=1$. This map is then used to transform
random variables with a non-trivial distribution to random variables with a
trivial distribution (i.e. easy to sample from). The gradient flow is defined
by a generating functional, $\tilde{S}[s]$, that is a scalar function of the
field variables, $s(x)$. The generator of the flow is defined through its
gradient.

$\dot{s}(x)=\partial_{x}\tilde{S}[s]\quad\quad\partial_{x}\equiv\frac{\partial}{\partial
s(x)}$ (1)

Given that the initial conditions of the fields, $s(x,0)=\sigma$ at $t=0$, the
map is then defined as

$s(x,t)=\mathcal{F}^{-1}_{t}(\sigma)$ (2)

The field variables, $s(x,t)$, are time-dependent values that are determined
from the initial conditions. The map, $\mathcal{F}^{-1}_{t}$, due to the
uniqueness of solutions of the flow equation, is bijective and is therefore
invertible,
$\mathcal{F}_{t}(s(x,t))=\mathcal{F}_{t}(\mathcal{F}_{t}^{-1}(\sigma))=\sigma$.
This map can be seen as a change of variables between the distributions of the
$s$ and $\sigma$ fields.

$q(\sigma)\mathcal{D}\sigma=q(\mathcal{F}_{1}(s))\det\mathcal{J}(s)\mathcal{D}s$
(3)

where $\mathcal{J}$ is the Jacobian of the transformation. This is the
trivializing map if it transforms the trivial distribution into the target
distribution.

$p(s)\mathcal{D}s=q(\mathcal{F}_{1}(s))\det\mathcal{J}(s)\mathcal{D}s$ (4)

Taking the log of both sides results in

$S_{t}(s)=S_{0}(\mathcal{F}_{1}(s))-\ln[\det\mathcal{J}(s)]-C\,,$ (5)

where $S_{t}$ and $S_{0}$ are the target and trivial actions respectfully and
$C$ is a constant determined by the partition function of the distributions.
To go further, we must consider the Jacobian, $\mathcal{J}$, for the specific
change of variables we are interested in. In our case the 2D-O(3) sigma model.

##  3 The 2D-O(3) sigma model

The lattice 2D-O(3) sigma model is defined by the action, $\mathcal{S}[s]$,
and the probability distribution, $p[s]$, where the integration measure,
${\cal D}s$ is over the unit sphere ${\cal S}^{2}$

$\mathcal{S}[s]=\frac{\beta}{2}\sum_{x}\sum_{\mu}(1-s_{x}\cdot
s_{x+\mu}),\quad s_{x}\in\mathcal{S}^{2},\quad
x\in\mathbb{Z}^{2},\quad\mu\in\\{\mathbb{Z}^{2}\;\big{|}\;|\mu|=1\\}$ (6)
$\displaystyle p[s]=\frac{e^{-\mathcal{S}[s]}}{\cal
Z},\quad\quad\quad\quad\quad\quad{\cal Z}=\int{\cal D}s\,e^{-{\cal S}[s]}$ (7)

To define a Trivializing Map from a gradient flow we need to first consider
the complications that arise from the compact nature of the sigma model. The
map has to transform the sphere to a sphere i.e.
$\mathcal{S}^{2}\rightarrow\mathcal{S}^{2}$. This implies that any
transformation of the spin degrees of freedom must be an SO(3) rotation:

$\forall R\in SO(3)\quad\&\quad\forall s\in{\cal S}^{2},\quad R*s\in{\cal
S}^{2}$ (8)

Therefore, the most general map that takes an $\mathcal{S}^{2}$ valued vector
field $\sigma$ and maps to an $\mathcal{S}^{2}$ valued vector field $s$, can
be written as

$s(x)=R(x)\,\sigma(x)\,,$ (9)

where $R(x)$ is an element of SO(3) which is determined up to an SO(2)
rotation. Rotations about the axis of the vector are the identity
transformation. This quotient subgroup SO(3)/SO(2) of SO(3) is diffeomorphic
to the sphere ${\cal S}^{2}$ manifold.

It is useful to consider parametrizing the spin variables $s(x)$ as a rotation
of a reference vector. This way the Lie derivative defined through an
infinitesimal rotation can be expressed as following:

$s(x)=R(x)s_{0},\quad\quad\quad
R(x)=\exp(\sum_{\alpha}\omega_{x}^{\alpha}L_{\alpha})$ (10)
$\partial_{y}^{\alpha}R(x)=L_{\alpha}R(x)\;\delta_{x,y}\implies\partial_{y}^{\alpha}s(x)=L_{\alpha}s(x)\,\delta_{x,y}\,.$
(11)

where $L_{a}$ are the traceless antisymmetric generators of
$\mathfrak{so}(3)$. This derivative satisfies the expected property

$\displaystyle\sum_{\alpha}s^{a}(x)\partial^{a}_{x}s(x)=0\,.$ (12)

In other words, the derivatives are elements of the tangent bundle. Using a
scalar functional of the fields, $\tilde{S}$, whose gradient defines the
generator of the flow, we have a flow on the $S^{2}$ manifold as

$\displaystyle\dot{s}(x,t)=-\sum_{a}L_{a}\partial_{x}^{\alpha}\tilde{S}[s(x,t)]\,s(x,t).$
(13)

##  4 Trivializing Flows in the Sigma Model

With the fields and their derivatives defined, it is now clear what we mean by
the Jacobian in Eq. (5). The Jacobian Matrix is then given by:

$\displaystyle{\cal
J}^{a,b}[R^{\prime}(x)](x,y)=-\frac{1}{2}\operatorname{Tr}[\hat{\partial}_{y}^{b}R(x)R^{\mkern-1.5mu\mathsf{T}}(x)L_{a}]$
(14)

where $\hat{\partial}_{y}^{b}$ is the Lie derivative with respect to the
initial condition fields $\sigma(x)$.

In Eq. (5) we have written the target distribution as independent of flow time
while, in principle, it could have explicit flow time dependence in which it
interpolates between trivial and the final target distributions. The simplest
case is a linear interpolation between the two distributions.

$S_{t}(s(t),t)=t[S_{0}(s(t))-\mathcal{S}(s(t))]+\mathcal{S}(s(t))$ (15)

Where at $t=0$, $S_{t}(s(t),t)$ is the target distribution, $\mathcal{S}$, and
at $t=1$ it is the trivial distribution, $S_{0}$. Using this simplifying
ansatz, we may take the total time derivative of our matching condition Eq.
(5).

$\sum_{x}\partial_{x}S_{t}(s)\dot{s}(t)+\partial_{t}S_{t}(s)=\frac{d}{dt}S_{0}(\mathcal{F}_{1}(s))-\frac{d}{dt}\ln[\det\mathcal{J}(s)]-\dot{C}$
(16)

From here, some simplifying steps may be taken. For any compact group, such as
$SO(3)$, the trivial action is simply $S_{0}=0$, which is constant and thus
has no time dependence. It can also be shown that the total time derivative of
the Jacobian term may be simplified,

$\displaystyle\frac{d\ln\det[{\cal
J}(R_{t})]}{dt}=\frac{d\operatorname{Tr}\ln[{\cal
J}(R_{t})]}{dt}=\operatorname{Tr}[\dot{\cal J}(R_{t}){\cal
J}^{-1}(R_{t})]=\partial_{x}^{a}\partial_{x}^{a}\tilde{S}$ (17)

In which case, using Eqs. (13) and (15) it can be shown that the matching
condition reduces to:

$-\partial^{2}\tilde{S}+t\sum_{x}\partial_{x}\mathcal{S}\;\partial_{x}\tilde{S}=-\mathcal{S}-\dot{C}$
(18)

Exact solutions for the generating functional, $\tilde{S}$, are not possible.
However, the small flow time expansion lets us determine $\tilde{S}$ and
$\dot{C}$ in a power series of $t$.

$\tilde{S}(s(t),t)=\sum_{n}t^{n}S^{(n)}(s(t))\quad\quad\dot{C}(t)=\sum_{n}t^{n}\dot{C}^{(n)}$
(19)

Using this form and collecting like powers of $t$, we determine 2 classes of
equations

$\displaystyle n=0:$
$\displaystyle\;\;\;\;\;\;\;\;\;\;\;\;\;\;-\partial^{2}S^{(0)}=-\mathcal{S}-\dot{C}^{(0)}$
(20) $\displaystyle n\geq 1:$
$\displaystyle\;\;\;\;\;-\partial^{2}S^{(n)}+\sum_{x}\partial_{x}\mathcal{S}\;\partial_{x}S^{(n-1)}=-\dot{C}^{(n)}$
(21)

These equations have no explicit dependence on $t$, and because they must hold
for every $t$ we may consider these fields at any instant to have static
values. Much like in the case of $SU(3)$ gauge theory [1], this set of
equations allows us to construct an iterative process to compute $S^{(n)}$ to
any order, as each order depends only on our target action, $\mathcal{S}$, and
the previous order solution.

###  4.1 Perturbative Solution

The $0^{th}$ Order is a special case that only depends on the target action.

$-\partial^{2}S^{(0)}=-\mathcal{S}-\dot{C}^{(0)},\quad\quad\quad\quad\mathcal{S}=\frac{-\beta}{2}\sum_{x,\;\mu}s_{x}\cdot
s_{x+\mu}$ (22)

Consider the ansatz that $S^{(0)}$ has the same functional form as
$\mathcal{S}$.

$S^{(0)}=\gamma_{0}\sum_{x,\;\mu}s_{x}\cdot s_{x+\mu}$ (23)

Then, using Eqs. (11) and (22) it is easy to show,

$-\partial^{2}S^{(0)}=4S^{(0)}\implies
4\gamma_{0}=\frac{\beta}{2}\implies\gamma_{0}=\frac{\beta}{8}\implies
S^{(0)}=\frac{\beta}{8}\sum_{x,\;\mu}s_{x}\cdot s_{x+\mu}$ (24)

Using the $0^{th}$ order solution and Eq. 21 we can construct the $1^{st}$
order solution by noting that:

$\displaystyle\implies$
$\displaystyle-\partial^{2}S^{(1)}=\frac{\beta^{2}}{4}\sum_{x}\Big{[}\sum_{\mu,\nu}(s_{x}\cdot
s_{x+\mu+\nu})-(s_{x}\cdot s_{x+\mu})(s_{x}\cdot
s_{x+\nu})\Big{]}-\dot{C}^{(1)}$ (25)

We may assume that $S^{(1)}$ has the same functional form as the RHS of the
above equation. However, when applying $\partial^{2}$ this ansatz for
$S^{(1)}$, a new type of term arises. So we must change the functional form to
include the new term. This process is repeated until the set of terms in the
functional form is closed under $\partial^{2}$. This generates a good basis
for the solution space. Using this it is possible to solve for the $1^{st}$
order flow functional.

$S^{(1)}=\frac{\beta^{2}}{40}\Big{[}2\Psi^{(2)}-\tilde{\Psi}^{(1,1)}+\frac{1}{6}\Psi^{(1,1f)}\Big{]}$
(26)

where,

$\Psi^{(2)}=\sum_{x}\sum_{\mu,\nu}s_{x}\cdot
s_{x+\mu+\nu},\;\tilde{\Psi}^{(1,1)}=\sum_{x}\sum_{\mu,\nu}(s_{x}\cdot
s_{x+\mu})(s_{x}\cdot s_{x+\nu}),\;\Psi^{(1,1f)}=\sum_{x}\sum_{\mu}(s_{x}\cdot
s_{x+\mu})^{2}$ (27)

Following the same steps as the 1st Order, continually adding terms to the
ansatz until the set of terms is closed under $\partial^{2}$, gives:

$\displaystyle S^{(2)}$
$\displaystyle=\frac{\beta^{3}}{72000}\Big{[}-4467\Psi^{(1)}+1305\Psi^{(3)}-990\tilde{\Psi}^{(2,1)}-300\tilde{\Psi}^{(1,2)}_{d}+200\tilde{\Psi}^{(1,1,1)}_{branch}$
$\displaystyle\qquad+225\tilde{\Psi}^{(1,1,1)}_{chain}+246\tilde{\Psi}^{(1,2f)}-210\tilde{\Psi}^{(1,1f,1)}+35\tilde{\Psi}^{(1,1f,1f)}\Big{]}$

where,

$\displaystyle\Psi^{(1)}\equiv\sum_{x}\sum_{\mu}{s_{x}}\cdot{s_{x+\mu}}\quad\quad\Psi^{(3)}\equiv\sum_{x}\sum_{\mu,\nu,\lambda}{s_{x}}\cdot{s_{x+\mu+\nu+\lambda}}\quad\quad\tilde{\Psi}^{(2,1)}\equiv\sum_{x}\sum_{\mu,\nu,\lambda}({s_{x}}\cdot{s_{x+\mu+\nu}})({s_{x}}\cdot{s_{x+\lambda}})$
$\displaystyle\tilde{\Psi}^{(1,2)}_{disc}\equiv\sum_{x}\sum_{\mu,\nu,\lambda}({s_{x}}\cdot{s_{x+\mu}})({s_{x+\nu}}\cdot{s_{x+\lambda}})\quad\quad\tilde{\Psi}^{(1,1,1)}_{branch}\equiv\sum_{x}\sum_{\mu,\nu,\lambda}({s_{x}}\cdot{s_{x+\mu}})({s_{x}}\cdot{s_{x+\nu}})({s_{x}}\cdot{s_{x+\lambda}})$
$\displaystyle\tilde{\Psi}^{(1,1,1)}_{chain}\equiv\sum_{x}\sum_{\mu,\nu,\lambda}({s_{x}}\cdot{s_{x+\mu}})({s_{x+\mu}}\cdot{s_{x+\mu+\nu}})({s_{x+\mu+\nu}}\cdot{s_{x+\mu+\nu+\lambda}})\quad\quad\tilde{\Psi}^{(1,2f)}\equiv\sum_{x}\sum_{\mu,\nu}({s_{x}}\cdot{s_{x+\mu}})({s_{x}}\cdot{s_{x+\mu+\nu}})$
$\displaystyle\tilde{\Psi}^{(1,1f,1)}\equiv\sum_{x}\sum_{\mu,\nu}({s_{x}}\cdot{s_{x+\mu}}^{2})({s_{x}}\cdot{s_{x+\nu}})\quad\quad\tilde{\Psi}^{(1,1f,1f)}\equiv\sum_{x}\sum_{\mu}({s_{x}}\cdot{s_{x+\mu}})^{3}$

This process may be continued to arbitrary order, however, the set of terms
grows rapidly. The 3rd order has 40 unique types of terms. The 4th order has
176 unique types of terms. Therefore, the procedure becomes quickly
intractable as it proceeds to higher orders. A Rust and Python implementation
has been made to calculate the terms to arbitrary order (and cross-check), but
these terms were not implemented in our simulation 111 See
https://github.com/cdchamness/O3OrderExpansion.

##  5 Numerical Tests

Using the approximate solution detailed above defines a flow that
approximately maps trivial variables to those following the distribution of
the 2D-O(3) model. By applying this map on uniformly distributed fields of
unit vectors, it is possible to generate samples that are close to the target
distribution. Using Eq. (5) we measure the deviation from the true
distribution. A perfect flow would always result in the LHS being equal to the
RHS. Therefore the error on the map may be determined by the variance of the
differences between the approximate distribution and the target distribution.
A variance of the difference is used as any overall shift may be absorbed into
the definition of the action and has no effect on the distribution.

$\sigma_{\Delta S}=Var(\Delta S)^{1/2}\equiv
Var\Big{(}\mathcal{S}(s)-\Big{[}S_{0}(\mathcal{F}(s))-\ln\mathcal{J}(s)\Big{]}\Big{)}^{1/2}$
(28)

To investigate the accuracy of the flow, simulations were performed to apply
the flow on trivial fields. For each measurement, we considered 400
configurations, where the flow was applied via a fixed step size of $\delta
t=0.05$. Jackknife sampling was used to determine the errors of the variances.
We computed the scaling of the $\sigma_{\Delta S}$ as a function of the
lattice volume $V$ and the coupling $\beta$ of the target action. We observe
linear scaling of $\sigma_{\Delta S}$ with the lattice side length $L=V^{1/2}$
as shown in Figure 1. This test was performed at fixed $\beta=0.5$. As higher
order terms are included $\sigma_{\Delta S}$ is reduced for all lattice sizes
$L$. Furthermore, at fixed order in the flow time expansion, the scaling of
$\sigma_{\Delta S}$ is approximately linear with the lattice size $L$ in
lattice units. The linear scaling of $\sigma_{\Delta S}$ with lattice size
suggests that the quality of the approximation of the target distribution via
the approximate Trivializing map will drop exponentially with $L$. In Figure 1
we consider the scaling of $\sigma_{\Delta S}$ vs. $\beta$ while keeping the
lattice size fixed, $L=36$. Our solution is an expansion in $0\leq t\leq 1$
however, at every order, an additional factor of $\beta$ is obtained through
the contraction with $\mathcal{S}$. For that reason the truncation to a fixed
order results in polynomial corrections in $\beta$ at small $\beta$
($\sigma_{\Delta S}~{}\beta^{2}$(Ord. 0), $\sigma_{\Delta S}~{}\beta^{3}$(Ord.
1), $\sigma_{\Delta S}~{}\beta^{4}$(Ord. 2)). From our current experiments, it
is not clear what the behavior of the approximation is as one approaches
criticality at large $\beta$ and volumes.

We may improve the approximation by adding a subset of known terms to
approximate higher-order solutions with little increase in computational cost.
By noticing that $\Psi^{(1)}$ appears in both the 0th order and 2nd order
solutions it is then possible to know that all terms that appeared in the 1st
order solution will appear in the 3rd order solution. This is because the
process for determining the initial set of each order is the same,
specifically $\sum_{x}\partial_{x}\mathcal{S}\;\partial_{x}S^{(n-1)}$. This
also works with respect to the 2nd-order and 4th-order solutions. Therefore it
is possible to determine which terms appear in "even" order solutions and
terms that appear in "odd" order solutions.

$\displaystyle
S^{(2n)}=\beta^{2n+1}\sum_{a}\gamma^{(2n)}_{a}\Psi_{even}^{a}\quad\quad
S^{(2n+1)}=\beta^{2n+2}\sum_{a}\gamma^{(2n+1)}_{a}\Psi_{odd}^{a}$ (29)

By non-perturbative tuning of the $\\{\gamma_{\alpha}^{(n)}\\}$, it might be
possible to increase performance further with little cost to performing the
flow, as these terms would already be computed for previous orders.

Figure 1: The standard deviation of $\Delta S$ as a function of the Lattice
Size (left) and coupling $\beta$ (right). The lines are fitted to the data as
described in the text.

##  6 Conclusions

In this contribution, we present an analytic construction of a trivializing
flow for the 2D-O(3) sigma model. The construction is based on a gradient flow
on the $\mathcal{S}^{2}$ manifold, whose generating functional is determined
order-by-order in the small flow-time expansion. This perturbative expansion
is algorithmic and has been programmed in principle to all orders. However, it
is impractical beyond the first few orders. We numerically tested the
efficiency of the approximation and found evidence that the computational cost
of the approximation scales exponentially with the lattice size. We intend to
investigate this issue further both numerically and analytically. If such
behavior persists it is doubtful that trivializing maps constructed
perturbatively or non-perturbatively can eliminate critical showing down which
occurs close to criticality where large correlation lengths mandate large
lattices in lattice units.

Acknowledgements: This work was supported by the U.S. DOE Grant DE-
FG02-04ER41302 and by the US DOE Contract No. DE-AC05-06OR23177. We thank
Balin Armstrong for discussions related to his work on trivializing maps of
the O(2) sigma model.

## References

  * Luscher [2010] M. Luscher, Commun. Math. Phys. 293, 899 (2010), arXiv:0907.5491 [hep-lat] .
  * Polyakov [1975] A. Polyakov, Physics Letters B 59, 79 (1975).
  * Hasenfratz  _et al._ [1990] P. Hasenfratz, M. Maggiore, and F. Niedermayer, Phys. Lett. B 245, 522 (1990).
  * Bietenholz  _et al._ [2018] W. Bietenholz, P. de Forcrand, U. Gerber, H. Mejía-Díaz, and I. O. Sandoval, Phys. Rev. D 98, 114501 (2018), arXiv:1808.08129 [hep-lat] .
  * Engel and Schaefer [2011] G. P. Engel and S. Schaefer, Comput. Phys. Commun. 182, 2107 (2011), arXiv:1102.1852 [hep-lat] .