#  ferret: a Framework for Benchmarking Explainers on Transformers

Giuseppe Attanasio♣, Eliana Pastor♢, Chiara Di Bonaventura♠, Debora Nozza♣

♣Bocconi University, Milan, Italy
♢Politecnico di Torino, Turin, Italy
♠King’s College London, London, United Kingdom
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

As Transformers are increasingly relied upon to solve complex NLP problems,
there is an increased need for their decisions to be humanly interpretable.
While several explainable AI (XAI) techniques for interpreting the outputs of
transformer-based models have been proposed, there is still a lack of easy
access to using and comparing them. We introduce ferret, a Python library to
simplify the use and comparisons of XAI methods on transformer-based
classifiers. With ferret, users can visualize and compare transformers-based
models output explanations using state-of-the-art XAI methods on any free-text
or existing XAI corpora. Moreover, users can also evaluate ad-hoc XAI metrics
to select the most faithful and plausible explanations. To align with the
recently consolidated process of sharing and using transformers-based models
from Hugging Face, ferret interfaces directly with its Python library. In this
paper, we showcase ferret to benchmark XAI methods used on transformers for
sentiment analysis and hate speech detection. We show how specific methods
provide consistently better explanations and are preferable in the context of
transformer models.

##  1 Introduction

Transformers have revolutionized NLP applications in recent years due to their
strong performance on various tasks; their black-box nature remains an
obstacle for practitioners who need explanations about why specific
predictions were made and what features drove them. The development of
explainable AI (XAI) techniques on several NLP tasks Madsen et al. (2022) has
helped bridge this gap by providing insight into the inner workings of
transformers and helping users gain trust in their decisions. Several XAI
approaches have been proposed in the literature Ribeiro et al. (2016);
Lundberg and Lee (2017); Simonyan et al. (2014a); Pastor and Baralis (2019),
also tailored to Transformer models Wallace et al. (2019a); Li et al. (2016);
Jin et al. (2019); Ross et al. (2021). Despite the importance of making XAI
methods accessible to NLP experts and practitioners through practical tools,
there is still a lack of accessibility for transformer models. XAI for
transformers is mainly scattered and hard to operationalize. Methods come with
independent implementations or framework-specific libraries that do not allow
either evaluation or cross-method comparison. Further, existing
implementations are not integrated with widespread transformers libraries
(e.g., Hugging Face’s transformers Wolf et al. (2020)). The lack of
standardization and weak interoperability leaves practitioners with unsolved
questions, such as choosing the best method given a task and a model Attanasio
et al. (2022).

We introduce ferret (FramEwork foR benchmaRking Explainers on Transformers),
an open-source Python library that drastically simplifies the use and
comparison of XAI methods on transformers. The library stems from vertical
scientific contributions and focused engineering efforts. On the one hand,
ferret provides the first-of-its-kind API (see Figure 1) to use and compare
explanation methods along the established criteria of faithfulness and
plausibility Jacovi and Goldberg (2020). On the other hand, it integrates
seamlessly with transformers Wolf et al. (2020), making it an easy add-on to
existing Transformer-based pipelines and NLP tasks. ferret permits to run four
state-of-the-art XAI methods, compute six ad-hoc XAI evaluation metrics, and
easily load four existing interpretability datasets. Further, it offers
abstract interfaces to foster future integration of methods, metrics, and
datasets.

We showcase ferret on sentiment analysis and hate speech detection case
studies. Faithfulness and plausibility metrics highlight SHAP Lundberg and Lee
(2017) as the most consistent explainer on single- and multiple-samples
scenarios.

##### Contributions.

We release ferret, the first-of-its-kind benchmarking framework for
interpretability tightly integrated with Hugging Face’s transformers library.
We release our code and documentation,111https://github.com/g8a9/ferret an
interactive demo,222https://huggingface.co/spaces/g8a9/ferret and a video
tutorial.333https://youtu.be/kX0HcSah_M4

⬇

from transformers import AutoModelForSequenceClassification, AutoTokenizer

from ferret import Benchmark

name = "cardiffnlp/twitter-xlm-roberta-base-sentiment"

model = AutoModelForSequenceClassification.from_pretrained(name)

tokenizer = AutoTokenizer.from_pretrained(name)

bench = Benchmark(model, tokenizer)

explanations = bench.explain("You look stunning!", target=1)

evaluations = bench.evaluate_explanations(explanations, target=1)

Figure 1: Essential code to benchmark explanations on an existing Hugging Face
model using ferret.

##  2 Library Design

ferret builds on four core principles.

##### 1\. Built-in Post-hoc Interpretability

We include four state-of-the-art post-hoc feature importance methods and three
interpretability corpora. Ready-to-use methods allow users to explain any text
with an arbitrary model. Annotated datasets provide valuable test cases for
new interpretability methods and metrics. To the best of our knowledge, ferret
is first in providing integrated access to XAI datasets, methods, and a full-
fledged evaluation suite.

##### 2\. Unified Explanation Benchmarking

We propose a unified API to evaluate explanations. We currently support six
state-of-the metrics along the principles of faithfulness and plausibility
(Jacovi and Goldberg, 2020).

##### 3\. Transformers-readiness

ferret offers a direct interface with models from the Hugging Face Hub. Users
can load models using standard naming conventions and explain them with the
built-in methods effortlessly. Figure 1 shows the essential code to classify
and explain a string with a pre-existing Hugging Face model and evaluate the
resulting explanations.

##### 4\. Modularity and Abstraction

ferret counts three core modules, implementing Explainers, Evaluation, and
Datasets APIs. Each module exposes an abstract interface to foster new
development. For example, user can sub-class BaseExplainer or BaseEvaluator to
include a new feature importance method or a new evaluation metric
respectively.

Feature |  Category
---|---
Gradient |  Saliency
Integrated Gradient |  Saliency
LIME |  Surrogate Model
SHAP |  Shapley Values
Comprehensiveness |  Faithfulness
Sufficiency |  Faithfulness
Correlation with |
Leave-One-Out scores |  Faithfulness
Intersection-Over-Union |  Plausibility
Area Under |
Precision-Recall Curve |  Plausibility
Token-level F1 score |  Plausibility
HateXplain |  Hate Speech
MovieReviews |  Sentiment
SST |  Sentiment
Thermostat |  Generic
Table 1: ferret at a glance: built-in methods (top), metrics (middle), and
datasets (bottom).

ferret builds on common choices from the interpretability community and good
engineering practices. We report the most salient technical details (e.g.,
efficiency via GPU inference, visualization tools, etc.) in Appendix A.

###  2.1 Explainer API

We focus on the widely adopted family of post-hoc feature attribution methods
Danilevsky et al. (2020). I.e., given a model, a target class, and a
prediction, ferret lets you measure how much each token contributed to that
prediction. We integrate Gradient Simonyan et al. (2014b) (also known as
Saliency) and Integrated Gradient Sundararajan et al. (2017); SHAP Lundberg
and Lee (2017) as a Shapley value-based method, and LIME Ribeiro et al. (2016)
as representative of local surrogate methods.

We build on open-source libraries and streamline their interaction with
Hugging Face models and paradigms. We report the supported configurations and
functionalities in Appendix A.

###  2.2 Dataset API

Fostering a streamlined, accessible evaluation on independently released XAI
datasets, we provide a convenient Dataset API. It enables users to load XAI
datasets, explain individual or subsets of samples, and evaluate the resulting
explanations.

Currently, ferret includes three classification-oriented datasets annotated
with human rationales, i.e., annotations highlighting the most relevant words,
phrases, or sentences a human annotator attributed to a given class label
DeYoung et al. (2020); Wiegreffe and Marasovic (2021). Moreover, ferret API
gives access to the Thermostat collection Feldhus et al. (2021), a wide set of
pre-computed feature attribution scores.

##### HateXplain

Mathew et al. (2021). It contains 20,148 English instances labeled along three
axes: (i) hate (either hateful, offensive, normal or undecided), (ii) target
group (either race, religion, gender, sexual orientation, or miscellaneous),
and (iii) word-level human rationales (expressed only on hateful and offensive
texts).444If a model splits a relevant word into sub-words, we consider all of
them relevant as well.

##### MovieReviews

Zaidan and Eisner (2008); DeYoung et al. (2020). The dataset contains 2,000
movie reviews annotated with positive and negative sentiment labels and
phrase-level human rationales that support gold labels.

##### Stanford Sentiment Treebank (SST)

Socher et al. (2013). A sentiment classification dataset of 9,620 movie
reviews annotated with binary sentiment labels, including human annotations
for word phrases of the parse trees. We extract human rationales from
annotations following the heuristic approach proposed in Carton et al. (2020).

##### Thermostat Datasets

Thermostat Feldhus et al. (2021) provides pre-computed feature attribution
scores given a model, a dataset, and an explanation method. ferret currently
provides built-in access to pre-computed attributions on the news topic
classification and sentiment analysis tasks.

These datasets provide an initial example of what an integrated approach can
offer to researchers and practitioners.

###  2.3 Evaluation API

We evaluate explanations on the faithfulness and plausibility properties
(Jacovi and Goldberg, 2020; DeYoung et al., 2020). Specifically, ferret
implements three state-of-the-art metrics to measure faithfulness and three
for plausibility.

##### Faithfulness.

Faithfulness evaluates how accurately the explanation reflects the inner
working of the model (Jacovi and Goldberg, 2020).

ferret offers the following measures of faithfulness: comprehensiveness,
sufficiency, (DeYoung et al., 2020) and correlations with ‘leave-one-out’
scores (Jain and Wallace, 2019).

Comprehensiveness ($\uparrow$) evaluates whether the explanation captures the
tokens the model used to make the prediction. We measure it by removing the
tokens highlighted by the explainer and observing the change in probability as
follows.

Let $x$ be a sentence and let $f_{j}$ be the prediction probability of the
model $f$ for a target class $j$. Let $r_{j}$ be a discrete explanation or
rationale indicating the set of tokens supporting the prediction $f_{j}$.
Comprehensiveness is defined as $f(x)_{j}-f(x\setminus r_{j})_{j}$ where
$x\setminus r_{j}$ is the sentence $x$ were tokens in $r_{j}$ are removed. A
high value of comprehensiveness indicates that the tokens in $r_{j}$ are
relevant for the prediction.

While comprehensiveness is defined for discrete explanations, feature
attribution methods assign a continuous score to each token. We hence select
identify $r_{j}$ as follows. First, we filter out tokens with a negative
contribution (i.e., they pull the prediction away from the chosen label).
Then, we compute the metric multiple times, considering the $k\%$ most
important tokens, with $k$ ranging from 10% to 100% (step of 10%). Finally, we
aggregate the comprehensiveness scores with the average, called Area Over the
Perturbation Curve (AOPC) (DeYoung et al., 2020).

Sufficiency ($\downarrow$) captures if the tokens in the explanation are
sufficient for the model to make the prediction (DeYoung et al., 2020). It is
measured as $f(x)_{j}-f(r_{j})_{j}$. A low score indicates that tokens in
$r_{j}$ are indeed the ones driving the prediction. As for Comprehensiveness,
we compute the AOPC by varying the number of the relevant tokens $r_{j}$.

Correlation with Leave-One-Out scores ($\uparrow$). We first compute leave-
one-out (LOO) scores by omitting tokens and measuring the difference in the
model prediction. We do that for every token, once at a time. LOO scores
represent a simple measure of individual feature importance under the
linearity assumption (Jacovi and Goldberg, 2020). We then measure the Kendall
rank correlation coefficient $\tau$ between the explanation and LOO importance
(Jain and Wallace, 2019) ($taucorr\\_loo$). $taucorr\\_loo$ closer to 1 means
higher faithfulness to LOO.

##### Plausibility.

Plausibility reflects how explanations are aligned with human reasoning by
comparing explanations with human rationales (DeYoung et al., 2020) .

We integrate into ferret three plausibility measures of the ERASER benchmark
(DeYoung et al., 2020): Intersection-Over-Union (IOU) at the token level,
token-level F1 scores, and Area Under the Precision-Recall curve (AUPRC).

The first two are defined for discrete explanations. Given the human and
predicted rationale, IOU ($\uparrow$) quantifies the overlap of the tokens
they cover divided by the size of their union. Token-level F1 scores
($\uparrow$) are derived by computing precision and recall at the token level.
Following DeYoung et al. (2020) and Mathew et al. (2021), we derive discrete
explanations by selecting the top $K$ tokens with positive influence, where
$K$ is the average length of the human rationale for the dataset. While being
intuitive, IOU and Token-level F1 are based only on a single threshold to
derive rationales. Moreover, they do not consider tokens’ relative ranking and
degree of importance. We then also integrate the AUPRC ($\uparrow$), defined
for explanations with continuous scores (DeYoung et al., 2020). It is computed
by varying a threshold over token importance scores, using the human rationale
as ground truth.

###  2.4 Transformers-Ready Interface

ferret is deeply integrated with Hugging Face interfaces. Users working with
their standard models and tokenizers can easily integrate it for diagnostic
purposes. The contact point is the main Benchmark class. It receives any
Hugging Face model and tokenizer and uses them to classify, run explanation
methods and seamlessly evaluate the explanations. Similarly, our Dataset API
leverages Hugging Face’s datasets555https://github.com/huggingface/datasets to
retrieve data and human rationales.

⬇

from transformers import AutoModelForSequenceClassification, AutoTokenizer

from ferret import Benchmark

name = "cardiffnlp/twitter-xlm-roberta-base-sentiment"

model = AutoModelForSequenceClassification.

from_pretrained(name)

tokenizer = AutoTokenizer.

from_pretrained(name)

bench = Benchmark(model, tokenizer)

query = "Great movie for a great nap!"

scores = bench.score(query)

print(scores)

# Run built-in explainers

explanations = bench.explain(

query,

target=2 # "Positive" label

)

bench.show_table(explanations)

# Evaluate explanations

evaluations = bench.evaluate_explanations(

explanations, target=2

)

bench.show_evaluation_table(evaluations)

## Output

>> {’Negative’: 0.013735532760620117,

>> ’Neutral’: 0.06385018676519394,

>> ’Positive’: 0.9224143028259277}

Figure 2: Code to explain and evaluate explanations on a sentiment classifier
(top). Token attributions (middle): darker red (blue) show higher (lower)
contribution to the prediction. Faithfulness metrics (bottom): darker colors
show better performance.

##  3 Case Studies

We showcase ferret in two real-world tasks, focusing on benchmarking
explainers on individual samples or across multiple instances. In the
following, we describe how ferret highlights the best explainers in sentiment
analysis and hate speech detection tasks. Our running examples use an XLM-
RoBERTa model fine-tuned for sentiment analysis Barbieri et al. (2021) and a
BERT model fine-tuned for hate speech detection Mathew et al. (2021).

###  3.1 Faithfulness Metrics for Error Analysis

Explanations on individual instances are often used for model debugging and
error analysis Vig (2019); Feng et al. (2018). However, different explanations
can lead users to different conclusions, hindering a solid understanding of
the model’s flaws. We show how practitioners can alleviate this issue
including ferret in their pipeline.

Figure 2 shows explanations and faithfulness metrics computed on the sentence
“Great movie for a great nap!” for the “Positive” class label misclassified by
the model as “Negative”.

Faithfulness metrics show that SHAP adheres best to the model’s inner workings
since it returns the most comprehensive and relevant explanations. Indeed,
SHAP retrieves the highest number of tokens the model used to make the
prediction ($aopc\\_compr(\uparrow)=0.41$) that are relevant to drive the
prediction ($aopc\\_suff(\downarrow)=0.09$). Further,
$taucorr\\_loo(\uparrow)=0.43$ indicates that SHAP explanations capture the
most important tokens for the prediction under the linearity assumption.
Although Integrated Gradient (x Input) shows a higher $taucorr\\_loo$, it does
not provide comprehensive and sufficient explanations. Similarly, Gradient and
Integrated Gradient show bad sufficiency and comprehensiveness, respectively.
LIME and Gradient (x Input) do not return trustworthy explanations according
to all faithfulness metrics.

Once SHAP has been identified as the best explainer, its explanations enable
researchers to investigate possible recurring patterns or detect model biases
thoroughly. In this case, the explanations shed light on a type of lexical
overfitting: the word “great” skews the prediction toward the positive label
regardless of the context and semantics.

###  3.2 Multi-Instance Assessment

Instance-level analysis finds explainers that meet specific requirements
locally. However, the best local explainer might be unsatisfactory across
multiple instances. With ferret, users can easily produce and aggregate
evaluation metrics across multiple dataset samples—or the entire corpus.

We describe how to choose the explainer that returns the most plausible and
faithful explanations for the HateXplain dataset. For demonstration purposes,
we focus only on a sample of the dataset.

Figure 3 (Appendix C) shows the metrics averaged across ten samples with the
“hate speech” label. Results suggest again that SHAP yields the most faithful
explanations. SHAP and Gradient achieve the best comprehensiveness and
sufficiency scores, but SHAP outperforms all explainers for the $\tau$
correlation with LOO (taucorr_loo ($\uparrow$) = 0.41). Gradient provides the
most plausible explanations, followed by SHAP.

|  Multiple XAI approaches |  Transformers- readiness |  Evaluation APIs |  XAI datasets |  Built-in visualization
---|---|---|---|---|---
Captum | ✓ | ✗ | ✗ | ✗ | ✓
AllenNLP Interpret | ✓ | ✗ | ✗ | ✗ | ✗
Transformers-Interpret | ✗ | ✓ | ✗ | ✗ | ✓
Thermostat | ✓ | ✓ | ✗ | ✗ | ✓
ContrXT | ✗ | ✗ | ✗ | ✗ | ✗
OpenXAI | ✓ | ✗ | ✓ | ✗ | ✗
NLPVis | ✗ | ✗ | ✗ | ✗ | ✓
Seq2Seq-Vis | ✗ | ✗ | ✗ | ✗ | ✓
BertViz | ✗ | ✓ | ✗ | ✗ | ✗
ELI5 | ✗ | ✗ | ✗ | ✗ | ✓
LIT | ✓ | ✗ | ✗ | ✗ | ✓
ERASER | ✗ | ✗ | ✓ | ✓ | ✗
Inseq | ✓ | ✓ | ✗ | ✗ | ✓
ferret | ✓ | ✓ | ✓ | ✓ | ✓
Table 2: Comparing off-the-shelf features across different XAI libraries. When
assessing built-in visualization, we disregard tools that either do not
provide a unified interface or provide single data-point visualizations.

##  4 Related Work

This section provides a review of tools and libraries that offer a subset of
the ferret’s functionalities, namely the option to use multiple XAI methods
and datasets, evaluation API, transformer-readiness, and built-in
visualization. Table 2 summarizes them and compares ferret with similar
frameworks.

##### Tools for Post-Hoc XAI.

Toolkits for post-hoc interpretability offer built-in methods to explain model
prediction, typically through a code interface. ferret builds on and extends
this idea to a unified framework to generate explanations, evaluate and
compare them, with support to several XAI datasets. Moreover, ferret’s
explainers are integrated with transformers’s Wolf et al. (2020) principles
and conventions.

PyTorch’s Captum (Kokhlikyan et al., 2020) is a generic Python library
supporting many interpretability methods. However, the library lacks
integration with the Hugging Face Hub and offers no evaluation procedures.
AllenNLP Interpret Wallace et al. (2019b) provides interpretability methods
based on gradients and adversarial attacks for AllenNLP models Gardner et al.
(2018). We borrow the modular and extensible design and extend it to a wider
set of explainers. Transformers-
Interpret666https://github.com/cdpierse/transformers-interpret leverages
Captum to explain Transformer models, but it supports only a limited number of
methods. Thermostat Feldhus et al. (2021) exposes pre-computed feature
attribution scores through the Hugging-Face Hub but no features oriented to
implement or evaluate XAI. We support the Thermostat as a third-party add-on
and let users test and benchmark pre-computed explanations. Unlike our study,
Inseq (Sarti et al., 2023) focuses on post-hoc interpretability for sequence
generation models. Although researchers can use the library to add
interpretability evaluations to their models, the toolkit lacks built-in
evaluation metrics.

Other related approaches enable global (rather than local) explainability
Malandri et al. (2022), or explanation interfaces for non-transformers models
on non-NLP tasks Agarwal et al. (2022). Other approaches study model behavior
at the subgroup level Wang et al. (2021); Goel et al. (2021); Pastor et al.
(2021a, b), focusing more on model evaluation and robustness rather than its
interpretation.

##### Visualization.

Most studies that develop visualization tools to investigate the relationships
among the input, the model, and the output focus either on specific NLP models
- NLPVis Liu et al. (2018), Seq2Seq-Vis Strobelt et al. (2018), or explainers
- BertViz Vig (2019), ELI5777https://github.com/TeamHG-Memex/eli5. LIT Tenney
et al. (2020) streamlines exploration and analysis in different models.
However, it acts mainly as a graphical browser interface. ferret provides a
Python interface easy to integrate with pre-existing pipelines.

##### Evaluation.

Although prior works introduced diagnostic properties for XAI techniques,
evaluating them in practice remains challenging. Studies either concentrate on
specific model architectures Lertvittayakumjorn and Toni (2019); Arras et al.
(2019); DeYoung et al. (2020), individual datasets Guan et al. (2019); Arras
et al. (2019), or a single group of explainability methods Robnik-Šikonja and
Bohanec (2018); Adebayo et al. (2018). Hence, providing a generally applicable
and automated tool for choosing the most suitable method is crucial. To this
end, Atanasova et al. (2020) present a comparative study of XAI techniques in
three application tasks and model architectures. To the best of our knowledge,
we are the first to present a user-friendly Python interface to interpret,
visualize and empirically evaluate models directly from the Hugging Face Hub
across several metrics. We extend previous work from DeYoung et al. (2020),
who developed a benchmark for evaluating rationales on NLP models called
ERASER by offering a unified interface for evaluation and visual comparison of
the explanations at the instance- and dataset-level.

Closer to ferret, the OpenXAI framework Agarwal et al. (2022) enables a
systematic evaluation of feature attribution explanation, integrating multiple
explainers and XAI structured datasets. OpenXAI supports tabular datasets
while we focus on textual data and NLP models.

##  5 Conclusions

We introduced ferret, a novel Python framework to easily access XAI techniques
on transformer models. With ferret, users can explain using state-of-the-art
post-hoc explainability techniques, evaluate explanations on several metrics
for faithfulness and plausibility, and easily interact with datasets annotated
with human rationales.

We built ferret with modularity and abstraction in mind to facilitate future
extensions and contributions from the community (see Appendix B for an
overview of the ongoing development). As future work, we envision off-the-
shelf support for new NLP tasks and scenarios. Building on the classification
setup presented in this paper, we plan to add support to more NLP tasks that
can be framed as classification, such as Mask Filling Prediction, Natural
Language Inference, Zero-Shot Text Classification, Next Sentence Prediction,
Token Classification, and Multiple-Choice QA. One further direction would be
improving ferret’s interoperability with new libraries, e.g., Inseq Sarti et
al. (2023) for XAI on text generation tasks and models.

## Ethics Statement

ferret’s primary goal is to facilitate the comparison of methods that are
instead frequently tested in isolation. Nonetheless, we cannot assume the
metrics we currently implement provide a full, exhaustive picture, and we work
towards enlarging this set accordingly.

Further, interpretability is much broader than post-hoc feature attribution.
We focus on this family of approaches for their wide adoption and
intuitiveness.

Similarly, the evaluation measures we integrate are based on removal-based
criteria. Prior works pointed out their limitations, specifically the problem
of erased inputs falling out of the model input distribution (Hooker et al.,
2019).

## Acknowledgments

This project has partially received funding from the European Research Council
(ERC) under the European Union’s Horizon 2020 research and innovation program
(grant agreement No. 949944, INTEGRATOR), by Fondazione Cariplo (grant No.
2020-4288, MONICA), and by the grant “National Centre for HPC, Big Data and
Quantum Computing”, CN000013 (approved under the M42C Call for Proposals -
Investment 1.4 - Notice “National Centers” - D.D. No. 3138, 16.12.2021,
admitted for funding by MUR Decree No. 1031, 17.06.2022). DN and GA are
members of the MilaNLP group and the Data and Marketing Insights Unit of the
Bocconi Institute for Data Science and Analysis. EP did part of the work while
at CENTAI and is currently a member of the DataBase and Data Mining Group
(DBDMG) at Politecnico di Torino. CDB contributed to the work while at Bocconi
University and is currently part of the UKRI Centre for Doctoral Training in
Safe and Trusted Artificial Intelligence (www.safeandtrustedai.org).

## References

  * Adebayo et al. (2018) Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018.  Sanity checks for saliency maps.  _Advances in neural information processing systems_ , 31.
  * Agarwal et al. (2022) Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. 2022.  OpenXAI: Towards a transparent evaluation of model explanations.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Arras et al. (2019) Leila Arras, Ahmed Osman, Klaus-Robert Müller, and Wojciech Samek. 2019.  Evaluating recurrent neural network explanations.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 113–126, Florence, Italy. Association for Computational Linguistics.
  * Atanasova et al. (2020) Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020\.  A diagnostic study of explainability techniques for text classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3256–3274, Online. Association for Computational Linguistics.
  * Attanasio et al. (2022) Giuseppe Attanasio, Debora Nozza, Eliana Pastor, and Dirk Hovy. 2022.  Benchmarking post-hoc interpretability approaches for transformer-based misogyny detection.  In _Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP_ , pages 100–112, Dublin, Ireland. Association for Computational Linguistics.
  * Barbieri et al. (2021) Francesco Barbieri, Luis Espinosa Anke, and José Camacho-Collados. 2021\.  XLM-T: A multilingual language model toolkit for twitter.  _CoRR_ , abs/2104.12250.
  * Carton et al. (2020) Samuel Carton, Anirudh Rathore, and Chenhao Tan. 2020.  Evaluating and characterizing human rationales.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9294–9307, Online. Association for Computational Linguistics.
  * Danilevsky et al. (2020) Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. 2020.  A survey of the state of explainable AI for natural language processing.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 447–459, Suzhou, China. Association for Computational Linguistics.
  * DeYoung et al. (2020) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020.  ERASER: A benchmark to evaluate rationalized NLP models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4443–4458, Online. Association for Computational Linguistics.
  * Feldhus et al. (2021) Nils Feldhus, Robert Schwarzenberg, and Sebastian Möller. 2021.  Thermostat: A large collection of NLP model explanations and analysis tools.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 87–95, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Feng et al. (2018) Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018.  Pathologies of neural models make interpretations difficult.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3719–3728, Brussels, Belgium. Association for Computational Linguistics.
  * Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018.  AllenNLP: A deep semantic natural language processing platform.  In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_ , pages 1–6, Melbourne, Australia. Association for Computational Linguistics.
  * Goel et al. (2021) Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christopher Ré. 2021.  Robustness gym: Unifying the NLP evaluation landscape.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations_ , pages 42–55, Online. Association for Computational Linguistics.
  * Guan et al. (2019) Chaoyu Guan, Xiting Wang, Quanshi Zhang, Runjin Chen, Di He, and Xing Xie. 2019\.  Towards a deep and unified understanding of deep neural models in nlp.  In _International conference on machine learning_ , pages 2454–2463. PMLR.
  * Hooker et al. (2019) Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. 2019.  A benchmark for interpretability methods in deep neural networks.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205, Online. Association for Computational Linguistics.
  * Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019.  Attention is not Explanation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3543–3556, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jin et al. (2019) Xisen Jin, Zhongyu Wei, Junyi Du, Xiangyang Xue, and Xiang Ren. 2019.  Towards hierarchical importance attribution: Explaining compositional semantics for neural sequence models.  In _International Conference on Learning Representations_.
  * Kokhlikyan et al. (2020) Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. 2020.  Captum: A unified and generic model interpretability library for pytorch.  _arXiv preprint arXiv:2009.07896_.
  * Lertvittayakumjorn and Toni (2019) Piyawat Lertvittayakumjorn and Francesca Toni. 2019.  Human-grounded evaluations of explanation methods for text classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5195–5205, Hong Kong, China. Association for Computational Linguistics.
  * Li et al. (2016) Jiwei Li, Will Monroe, and Dan Jurafsky. 2016.  Understanding neural networks through representation erasure.  _CoRR_ , abs/1612.08220.
  * Liu et al. (2018) Shusen Liu, Tao Li, Zhimin Li, Vivek Srikumar, Valerio Pascucci, and Peer-Timo Bremer. 2018.  Visual interrogation of attention-based models for natural language inference and machine comprehension.  Technical report, Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States).
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Madsen et al. (2022) Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022.  Post-hoc interpretability for neural nlp: A survey.  _ACM Comput. Surv._ Just Accepted.
  * Malandri et al. (2022) Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, Navid Nobani, and Andrea Seveso. 2022.  Contrastive explanations of text classifiers as a service.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations_ , pages 46–53.
  * Mathew et al. (2021) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021.  Hatexplain: A benchmark dataset for explainable hate speech detection.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(17):14867–14875.
  * Pastor and Baralis (2019) Eliana Pastor and Elena Baralis. 2019.  Explaining black box models by means of local rules.  In _Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing_ , SAC ’19, page 510–517, New York, NY, USA. Association for Computing Machinery.
  * Pastor et al. (2021a) Eliana Pastor, Luca de Alfaro, and Elena Baralis. 2021a.  Looking for trouble: Analyzing classifier behavior via pattern divergence.  In _Proceedings of the 2021 International Conference on Management of Data_ , SIGMOD ’21, page 1400–1412, New York, NY, USA. Association for Computing Machinery.
  * Pastor et al. (2021b) Eliana Pastor, Andrew Gavgavian, Elena Baralis, and Luca de Alfaro. 2021b.  How divergent is your data? _Proc. VLDB Endow._ , 14(12):2835–2838.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Ribeiro et al. (2016) Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  "Why Should I Trust You?": Explaining the Predictions of Any Classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_ , pages 1135–1144. ACM.
  * Robnik-Šikonja and Bohanec (2018) Marko Robnik-Šikonja and Marko Bohanec. 2018.  Perturbation-based explanations of prediction models.  In _Human and machine learning_ , pages 159–175. Springer.
  * Ross et al. (2021) Alexis Ross, Ana Marasović, and Matthew Peters. 2021.  Explaining NLP models via minimal contrastive editing (MiCE).  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3840–3852, Online. Association for Computational Linguistics.
  * Sanyal and Ren (2021) Soumya Sanyal and Xiang Ren. 2021.  Discretized integrated gradients for explaining language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10285–10299, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sarti et al. (2023) Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Oskar van der Wal. 2023.  Inseq: An interpretability toolkit for sequence generation models.  _ArXiv_ , abs/2302.13942.
  * Simonyan et al. (2014a) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014a.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings_.
  * Simonyan et al. (2014b) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014b.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _2nd International Conference on Learning Representations, ICLR 2014_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Strobelt et al. (2018) Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander M Rush. 2018.  S eq 2s eq-v is: A visual debugging tool for sequence-to-sequence models.  _IEEE transactions on visualization and computer graphics_ , 25(1):353–363.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 3319–3328. JMLR.org.
  * Tenney et al. (2020) Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. 2020.  The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 107–118, Online. Association for Computational Linguistics.
  * Vig (2019) Jesse Vig. 2019.  Visualizing attention in transformer-based language representation models.  _arXiv preprint arXiv:1904.02679_.
  * Wallace et al. (2019a) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019a.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Wallace et al. (2019b) Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. 2019b.  AllenNLP interpret: A framework for explaining predictions of NLP models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_ , pages 7–12, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2021) Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, and Xuanjing Huang. 2021.  TextFlint: Unified multilingual robustness evaluation toolkit for natural language processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 347–355, Online. Association for Computational Linguistics.
  * Wiegreffe and Marasovic (2021) Sarah Wiegreffe and Ana Marasovic. 2021.  Teach me to explain: A review of datasets for explainable natural language processing.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yin et al. (2022) Fan Yin, Zhouxing Shi, Cho-Jui Hsieh, and Kai-Wei Chang. 2022.  On the sensitivity and stability of model interpretations in NLP.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2631–2647, Dublin, Ireland. Association for Computational Linguistics.
  * Zaidan and Eisner (2008) Omar Zaidan and Jason Eisner. 2008.  Modeling annotators: A generative approach to learning from annotator rationales.  In _Proceedings of the 2008 conference on Empirical methods in natural language processing_ , pages 31–40.

##  Appendix A Technical Details

###  A.1 Explainer API

Our implementation is built on top of original implementations (as for SHAP
and LIME) and open-source libraries (as Captum (Kokhlikyan et al., 2020) for
gradient-based explainers) to directly explain Transformer-based language
models.

Currently, we integrate Gradient (G) Simonyan et al. (2014b), Integrated
Gradient (IG) Sundararajan et al. (2017), SHAP Lundberg and Lee (2017), and
LIME Ribeiro et al. (2016). For G and IG, users can get explanations from
plain gradients or multiply gradients by the input token embeddings. For SHAP,
we use the Partition approximation to estimate Shapley
values.888https://shap.readthedocs.io/en/latest/generated/shap.explainers.Partition.html

####  A.1.1 Evaluation API

While human gold annotations are normally discrete, current explainers provide
continuous token attribution scores. Following previous work, we hence go from
continuous scores to a discrete set of relevant tokens (i.e., $r_{j}$ in
Section 2.3) as follows.

We consider only tokens with a positive contribution to the chosen label
(i.e., they push the prediction towards the chosen label). For the AOPC
comprehensiveness and sufficiency measures, the relevant tokens in the
discrete rationale are the most $k\%$ important tokens with $k$ ranging from
10% to 100% (step of 10%). For token-level IOU and F1 scores plausibility
measure, we follow the DeYoung et al. (2020) and Mathew et al. (2021)
approach, and we select the top $k$ tokens where $k$ is the average length of
human rationales for the dataset.

The evaluation measures at the dataset level are the average scores across
explanations. Differently than DeYoung et al. (2020) that use the F1 IOU
score, we directly compute the average token-level IOU.

All human rationales are at the token level, indicating the most relevant
tokens to a given class label.

Figure 3: Faithfulness and Plausibility metrics averaged across ten samples
with the “hateful” label of HateXPlain. Darker colors mean better performance.

###  A.2 Technical Features

ferret implements several functionalities to facilitate end users in using it.

  * •

High-level interface. Most of ferret’s features, such as interpretability
methods and evaluation measures, are accessible via a single entry point, the
Benchmark class.

  * •

GPU-enabled batched inference. ferret requires running inference for certain
executions. It uses batching and local GPUs transparently to the user whenever
that happens.

  * •

Visualization methods. The Benchmark class exposes several methods to
visualize attribution scores and evaluation results in tabular format. These
tables are plotted seamlessly on Jupyter Notebooks (see Figure 2 (bottom) for
an example).

##  Appendix B Ongoing Development

ferret is under active development. We are extending the core modules as
follows.

##### Explainers.

We plan to integrate two recent interpretability methods that require training
a complementary model. Sampling and Occlusion (SOC) Jin et al. (2019) provides
a hierarchical explanation to address compositional contributions. Minimal
Contrastive Editing (MiCE) Ross et al. (2021) trains a T5 Raffel et al. (2020)
model to implement contrastive edits to the input to change the model output.
Finally, we are including a third gradient-based algorithm. Integrated
Discretized Gradients Sanyal and Ren (2021) improve IG sampling intermediate
steps close to actual words in the embedding space.

##### Evaluators.

We plan to include additional evaluation measures such as sensitivity,
stability Yin et al. (2022), and Area Under the Threshold-Performance curve
(AUC-TP) Atanasova et al. (2020).

##  Appendix C Additional Results

Figure 3 shows a screenshot of dataset-level assessment from our demo web app.
It reports the evaluation metrics averaged across ten samples with the “hate
speech” label for the HateXplain dataset, discussed in Section 3.

The user specifies a model from the Hugging Face Hub (HF Model field), the
target class (Target), and the set of samples of interest (List of samples).
ferret web app directly computes explanation and their evaluation and
visualizes the results.


# Results on QCD jet production at ATLAS and CMS

Christopher J. Meyer, on behalf of the ATLAS and CMS Collaborations

The production of jets at the Large Hadron Collider (LHC) at $\sqrt{s}=7$ TeV
is summarized, including results from both the ATLAS and CMS detectors.
Current jet performance is described, followed by inclusive jet and multi-jet
measurements in various final state configurations. Finally some results on
heavy flavour and jet substructure are presented.

At both the ATLAS $\\!{}^{{\bf?}}$ and CMS $\\!{}^{{\bf?}}$ detectors in the
LHC, jets serve as a proxy to final state partons. Following the hard
collision they undergo parton showering, hadronisation, and subsequently
interact in the surrounding detector. To reconstruct and calibrate the
constituents of jets, CMS uses a particle flow method which employs several
subdetectors $\\!{}^{{\bf?}}$ while ATLAS forms jets using finely segmented
calorimeters.$\\!{}^{{\bf?}}$ In both cases a pileup offset correction is
applied to remove additional energy from multiple proton-proton collisions.

The anti-$k_{T}$ clustering algorithm $\\!{}^{{\bf?}}$ is the preferred choice
for jet reconstruction, with other methods such as Cambridge-Aachen
$\\!{}^{{\bf?}}$ used for jet substructure. For cross section measurements a
radius between $0.4\leq R\leq 0.7$ is used, while larger jet radii ($R\geq
1.0$) are used for jet substructure studies.

The 2010 jet energy calibration is derived using Monte Carlo (MC) simulation
tuned using test beam and early collision data. The response is derived by
comparing fully simulated and reconstructed jets to truth jets, with in situ
techniques such as multijet, photon-jet, and Z-jet balance used as cross
checks in data. The uncertainty on the derived jet energy calibration, shown
in Figure 1, is often the dominant source of experimental error on cross
section measurements.

The inclusive jet cross section measures the production rate of jets as a
function of both transverse momentum ($p_{T}$) and rapidity ($y$). In 2010
jets were measured with 20 GeV $<p_{T}<$ 1550 GeV out to rapidities of
$|y|=4.4$ using 37 pb-1 of integrated luminosity at ATLAS,$\\!{}^{{\bf?}}$
with similar results seen in CMS.$\\!{}^{{\bf?}}$ In 2011 the data sample of
4.8 fb-1 has extended the reach to a $p_{T}$ of almost 2 TeV in
CMS.$\\!{}^{{\bf?}}$ The large reach of this basic observable offers a
powerful test of the Standard Model over many orders of magnitude. A next-to-
leading order (NLO) calculation is performed using NLOJET++ $\\!{}^{{\bf?}}$,
with non-perturbative corrections applied to account for hadronisation and
underlying event. Monte Carlo events are also generated with POWHEG
BOX,$\\!{}^{{\bf?}}$ producing NLO matrix elements with parton showering which
are then interfaced to PYTHIA or HERWIG for hadronisation. There is generally
good agreement seen between acceptance corrected measurement and theory, as
shown in Figure 2. At high $p_{T}$, especially for large values of $y$, a
tension is observed with theory over estimating data.

Ratios of jet measurements are powerful because many systematics (jet energy
calibration uncertainty and luminosity for example) either paritally or fully
cancel when the ratio is taken. Taking the ratio of events with $N\geq 3$ jets
to $N\geq 2$ jets is an interesting probe of NLO effects. $\\!{}^{{\bf?}}$
Measured as a function of the scalar sum of jets $p_{T}$,
$H_{T}=\Sigma~{}jet~{}p_{T}$ where the sum is extended to all jets with
$p_{T}>50$ GeV and $|y|<2.5$, Figure 3 shows that for $H_{T}>500$ GeV a
variety of MC predicts the data well.

Figure 1: Fractional uncertainty on the jet energy calibration as a function
of jet $p_{T}$ in ATLAS $\\!{}^{{\bf?}}$ and CMS.$\\!{}^{{\bf?}}$

Figure 2: Measurement of the inclusive jet cross section in the ATLAS
detector.$\\!{}^{{\bf?}}$ A slight tension is observed between data and theory
at high $p_{T}$. Figure 3: Ratio of the cross section of events in CMS with
$N\geq 3$ jets to $N\geq 2$ jets, where only jets with $p_{T}>50$ GeV and
$|y|<2.5$ are considered. The ratio is shown as a function of
$H_{T}$.$\\!{}^{{\bf?}}$

The ratio of the inclusive dijet cross section (considering all combinations
of $N\geq 2$ jets in an event) to the exclusive dijet cross section (only
consider events with exactly $N=2$ jets) is sensitive to the resummation of
large $log(1/x)$ terms (BFKL evolution). All jets with $p_{T}>35$ GeV and
$|y|<4.7$ are considered, with the ratio plotted as a function of absolute
rapidity separation $|\Delta y|$ between jet pairs. $\\!{}^{{\bf?}}$ As seen
in Figure 4(a) PYTHIA gives the best agreement to data.

Heavy flavour at the LHC is important for understanding backgrounds in
searches for the Higgs boson and/or super-symmetric particles, as well as
providing a check of the hadronisation description in MC. Figure 4(b) shows
the ratio of jets containing a $D^{*\pm}$ meson to all jets as a function of
$z$, the $D^{*\pm}$ momentum along the jet axis divided by the jet energy.
$\\!{}^{{\bf?}}$ For this low $p_{T}$ slice the agreement between data and MC
is poor. At low $p_{T}$, $D^{*\pm}$ originate mostly from c-hadrons showing
that c-fragmentation in jets is not well modeled.

(a) Ratio of the inclusive dijet cross section to the exclusive dijet cross
section in CMS.$\\!{}^{{\bf?}}$

(b) Fraction of jets containing a $D^{*\pm}$ meson in ATLAS.$\\!{}^{{\bf?}}$

Figure 4: Two ratio measurements from ATLAS and CMS.

(a) $b\bar{b}$ dijet cross section as a function of dijet mass in ATLAS.

(b) $b\bar{b}$ dijet cross section as a function of $\Delta\phi$ in ATLAS.

Figure 5: Results on jets produced from $b$-hadrons.$\\!{}^{{\bf?}}$

The measurement of the dijet cross section for jets from b-hadrons tests the
production and hadronization of b-quarks. $\\!{}^{{\bf?}}$ Figure 5(a) shows
the dijet mass cross section from $b\bar{b}$ pairs, where theory is seen to
describe data well. For dijet systems which radiate a gluon, the azimuthal
angle $\Delta\phi$ between them will be reduced. Figure 5(b) shows that while
back-to-back systems (larger $\Delta\phi$) are well described, as $\Delta\phi$
decreases both POWHEG+Pythia and MC@NLO+Herwig begin to over estimate the
data.

Jet substructure is useful for identifying hadronic decays of boosted heavy
particles. Splitting/filtering using Cambridge-Aachen $R=1.2$ jets is one
example which undoes the clustering procedure until a large mass drop is
observed. This type of technique is robust against the effects of multiple
proton-proton interactions in a single bunch crossing. Figure 6 shows the
improved agreement between data and MC after splitting/filtering has been
performed,$\\!{}^{{\bf?}}$ giving confidence in the MC hadronisation
description for substructure studies.

Figure 6: Results from ATLAS on jet substructure for Cambridge-Aachen $R=1.2$
jets, before and after application of splitting/filtering.$\\!{}^{{\bf?}}$

## References

## References

  * [1] ATLAS Collaboration, JINST 3 (2008) S08003.
  * [2] CMS Collaboration, JINST 3 (2008) S08004.
  * [3] CMS Collaboration, JINST 6 (2011) 11002.
  * [4] ATLAS Collaboration, arXiv:1112.6426 [hep-ex].
  * [5] M. Cacciari, G. P. Salam and G. Soyez, JHEP 0804 (2008) 063.
  * [6] Y. L. Dokshitzer, G. D. Leder, S. Moretti and B. R. Webber, JHEP 9708 (1997) 001.
  * [7] ATLAS Collaboration, Phys. Rev. D 86 (2012) 014022.
  * [8] CMS Collaboration, Phys. Rev. Lett. 107 (2011) 132001
  * [9] CMS Collaboration, CMS-PAS-QCD-11-004 (2012), https://cdsweb.cern.ch/record/1431022.
  * [10] S. Catani and M. H. Seymour, Nucl. Phys. B 485 (1997) 291 [Erratum-ibid. B 510 (1998) 503].
  * [11] S. Alioli, K. Hamilton, P. Nason, C. Oleari and E. Re, JHEP 1104 (2011) 081.
  * [12] CMS Collaboration, Phys. Lett. B 702 (2011) 336.
  * [13] CMS Collaboration, arXiv:1204.0696 [hep-ex].
  * [14] ATLAS Collaboration, Phys. Rev. D 85 (2012) 052005.
  * [15] ATLAS Collaboration, Eur. Phys. J. C 71 (2011) 1846.
  * [16] ATLAS Collaboration, JHEP 1205 (2012) 128.


An Observation on the Beta Functions in Quadratic Gravity

Hikaru Kawaia,b,111e-mail address<EMAIL_ADDRESS>and Nobuyoshi
Ohtac,d,222e-mail address<EMAIL_ADDRESS>
aDepartment of Physics and Center for Theoretical Physics, National Taiwan
University, Taipei 106, Taiwan

bPhysics Division, National Center for Theoretical Sciences, Taipei 106,
Taiwan

cResearch Institute for Science and Technology, Kindai University, Higashi-
Osaka,
Osaka 577-8502, Japan

dSchool of Physics, Korea Institute for Advanced Study, 85 Hoegi-ro
Dongdaemun-gu,
Seoul 02455, Korea

Abstract

We study the beta functions for the dimensionless couplings in quadratic
curvature gravity, and find that there is a simple argument to restrict the
possible form of the beta functions as derived from the counterterms at an
arbitrary loop. The relation to the recent different results on beta functions
is also commented on.

##  1 Introduction

The perturbative calculation of beta functions in quadratic gravity has a long
history. The action of our concern is

$\displaystyle S=\int
d^{4}x\sqrt{-g}\left[\frac{1}{2\lambda}C_{\mu\nu\rho\lambda}^{2}+\frac{1}{\xi}R^{2}-\frac{1}{\kappa}E+\frac{M_{P}^{2}}{2}R+\rho\right],$
(1.1)

where $M_{P}$ is the Planck mass, and $C_{\mu\nu\rho\lambda}$ is the Weyl
tensor with
$C_{\mu\nu\rho\lambda}^{2}=R_{\mu\nu\rho\lambda}^{2}-2R_{\mu\nu}^{2}+\frac{1}{3}R^{2}$
and $E=R_{\mu\nu\rho\lambda}^{2}-4R_{\mu\nu}^{2}+R^{2}$.

The first attempt to derive the beta functions for $\lambda$ and $\xi$ was
made in [1], but this does not have contributions from additional ghosts. The
result was corrected in [2], with some further corrections in [3]. The final
correct answer is

$\displaystyle\beta_{\lambda}^{o}$ $\displaystyle=$
$\displaystyle-\frac{1}{(4\pi)^{2}}\frac{133}{10}\lambda^{2},$
$\displaystyle\beta_{\xi}^{o}$ $\displaystyle=$
$\displaystyle-\frac{1}{(4\pi)^{2}}\frac{5(72\lambda^{2}-36\lambda\xi+\xi^{2})}{36}.$
(1.2)

Since then, the result is confirmed by several works [4, 5, 6, 7, 8]. It was
noted that the beta function for the coefficient of the Gauss-Bonnet term
$\kappa$ does not depend on $\kappa$ itself because it is a topological term
which does not contribute to quantum effects [9]. Thus it is proportional to
$\kappa^{2}$ at the fixed point of other couplings, and it is either
asymptotically free or not depending on the sign of the proportionality
constant. It is claimed that the other couplings are asymptotically free, but
there is restriction on these couplings $\xi>0.5487\lambda>0$ to be
asymptotically free [10].

This result means that the simple pole part of the counterterms are given as

$\displaystyle\frac{1}{4\epsilon}\frac{\beta_{\lambda}^{n,o}}{\lambda^{2}}C^{2}+\frac{1}{2\epsilon}\frac{\beta_{\xi}^{n,o}}{\xi^{2}}R^{2},$
(1.3)

where $\frac{1}{\epsilon}=\frac{2}{4-d}$ is the regularization parameter in
the dimensional regularization, corresponding to
$\log\left(\frac{\Lambda_{UV}^{2}}{\mu^{2}}\right)$ in the cutoff
regularization. See, for example, the review in [11].

In a recent paper [12], it has been claimed that the above results on the beta
functions of the dimensionless couplings $\lambda$ and $\xi$ do not describe
the “physical” running of the couplings. The beta functions obtained in Ref.
[12] are given as

$\displaystyle\beta_{\lambda}^{n}$ $\displaystyle=$
$\displaystyle-\frac{1}{(4\pi)^{2}}\frac{(1617\lambda-20\xi)\lambda}{90},$
$\displaystyle\beta_{\xi}^{n}$ $\displaystyle=$
$\displaystyle-\frac{1}{(4\pi)^{2}}\frac{\xi^{2}-36\lambda\xi-2520\lambda^{2}}{36}.$
(1.4)

Motivated by this result, we have studied the structure of the beta functions
in perturbation theory in the mass-independent renormalization scheme [13],
and point out that there is a simple argument to restrict the possible form of
the beta functions as derived from the counterterms at an arbitrary loop. We
also comment on the difference from the above results.

##  2 Perturbation theory as a double expansion in $\lambda$ and $\xi$

Applying the perturbation expansion to the action (1.1), we see that the
$n$-loop correction is given by an $n$-th order homogeneous function of
$\lambda$ and $\xi$. Furthermore, as we will show, it is a degree $n$
homogeneous polynomial of $\lambda$ and $\xi$. In other words, the
perturbation series is nothing but a double expansion in $\lambda$ and $\xi$.

In order to see this, we rewrite (1.1) using an auxiliary field $\varphi$ as
follows:111The Gauss-Bonnet term is ignored hereafter.

$\displaystyle S^{\prime}=\int
d^{4}x\sqrt{-g}\left[\frac{1}{2\lambda}C_{\mu\nu\rho\lambda}^{2}+\left(\varphi+\frac{M_{P}^{2}}{2}\right)R-\xi\varphi^{2}+\rho\right].$
(2.1)

Then, by changing variables as

$\displaystyle g^{\prime}_{\mu\nu}=e^{\phi}g_{\mu\nu},\qquad
e^{\phi}=\varphi+\frac{1}{2}M_{P}^{2},$ (2.2)

we have

$\displaystyle S^{\prime}=\int
d^{4}x\sqrt{-g^{\prime}}\left[\frac{1}{2\lambda}{C^{\prime}}_{\mu\nu\rho\lambda}^{2}+R^{\prime}-\frac{3}{2}{g^{\prime}}^{\mu\nu}\partial_{\mu}\phi\
\partial_{\nu}\phi-\xi\left(1-\frac{1}{2}M_{P}^{2}\ e^{-\phi}\right)^{2}+\rho\
e^{-2\phi}\right],$ (2.3)

where ${C^{\prime}}_{\mu\nu\rho\lambda}$ and $R^{\prime}$ are the Weyl tensor
and scalar curvature for ${g^{\prime}}_{\mu\nu}$, respectively. Finally, by
shifting $\phi$ as

$\displaystyle\phi=\phi^{\prime}+\log\frac{M_{P}^{2}}{2},$ (2.4)

we have

$\displaystyle S^{\prime}=\int
d^{4}x\sqrt{-g^{\prime}}\left[\frac{1}{2\lambda}{C^{\prime}}_{\mu\nu\rho\lambda}^{2}+R^{\prime}-\frac{3}{2}{g^{\prime}}^{\mu\nu}\partial_{\mu}\phi^{\prime}\
\partial_{\nu}\phi^{\prime}-\xi(1-e^{-\phi^{\prime}})^{2}+\frac{4\rho}{M_{P}^{4}}\
e^{-2\phi^{\prime}}\right].$ (2.5)

The important fact is that the Weyl gravity action, the first term in (2.5),
does not depend on the conformal factor of the metric. Therefore, if we
decompose the metric ${g^{\prime}}_{\mu\nu}$ as

$\displaystyle{g^{\prime}}_{\mu\nu}=\eta_{\mu\nu}+\sqrt{\lambda}\bar{h}_{\mu\nu}+\frac{1}{4}\eta_{\mu\nu}h,\qquad\bar{h}_{\mu\nu}\eta^{\mu\nu}=0,$
(2.6)

the coefficients of the kinetic terms for $\bar{h}_{\mu\nu}$, $h$ and
$\phi^{\prime}$ are numerical constants and the other terms are associated
with positive powers in $\sqrt{\lambda}$ and $\xi$. This shows that the
perturbation series is given by a power series of $\lambda$ and $\xi$.222 We
note here that the cosmological constant term $\frac{4\rho}{M_{P}^{4}}\
e^{-2\phi^{\prime}}$ in (2.5) is a sort of mass term for $\phi^{\prime}$ and
it does not affect the beta functions in the minimal subtraction scheme.

##  3 Smoothness of $\xi\to\infty$ limit and simple structure of
$\beta_{\lambda}$

Next we consider the limit where $\xi$ is taken to infinity. First if we
formally take this limit in (1.1) (with the Gauss-Bonnnet term ignored), we
have

$\displaystyle S_{\xi\to\infty}=\int
d^{4}x\sqrt{-g}\left[\frac{1}{2\lambda}C_{\mu\nu\rho\lambda}^{2}+\frac{M_{P}^{2}}{2}R+\rho\right].$
(3.1)

From a quantum mechanical point of view, such a limit is not necessarily
smooth. In this case, however, we can show that this limit is indeed smooth as
follows. To see this, we consider the action (2.5), in which $\xi$ appears
only in the fourth term. It is clear that the field $\phi^{\prime}$ is frozen
to $\phi^{\prime}=0$ in the $\xi\to\infty$ limit. Then (2.5) is reduced to

$\displaystyle S^{\prime}_{\xi\to\infty}=\int
d^{4}x\sqrt{-g^{\prime}}\left[\frac{1}{2\lambda}{C^{\prime}}_{\mu\nu\rho\lambda}^{2}+R^{\prime}+\frac{4\rho}{M_{P}^{4}}\right],$
(3.2)

which is nothing but (3.1) with

$\displaystyle g_{\mu\nu}=\frac{M_{P}^{2}}{2}g^{\prime}_{\mu\nu}.$ (3.3)

This argument can be made more rigorous by changing the variable in (2.5) as

$\displaystyle\phi^{\prime}=-\log(1-\psi).$ (3.4)

Then (2.5) becomes

$\displaystyle S^{\prime}=\int
d^{4}x\sqrt{-g^{\prime}}\left[\frac{1}{2\lambda}{C^{\prime}}_{\mu\nu\rho\lambda}^{2}+R^{\prime}-\frac{3}{2}\frac{1}{(1-\psi)^{2}}{g^{\prime}}^{\mu\nu}\partial_{\mu}\psi\
\partial_{\nu}\psi-\xi\psi^{2}+\frac{4\rho}{M_{P}^{4}}(1-\psi)^{2}\right].$
(3.5)

This shows that the mass of $\psi$ becomes infinite in the $\xi\to\infty$
limit and that $\psi$ does not appear as an internal line. Therefore, $\psi$
decouples from the physical process.

This should be contrasted with the $\lambda\to\infty$ limit. In this limit, it
is clear from (2.5) and (2.6) that the traceless mode of the metric,
$\bar{h}_{\mu\nu}$, has large quantum fluctuations and couples with the other
degrees of freedom, $\phi^{\prime}$ and $h$. Therefore, the system is a
strongly coupled system.

In general, in the mass-independent renormalization scheme [13], the beta
functions of dimensionless parameters are independent of parameters with
positive mass dimension. Therefore, the beta function for $\lambda$ and $\xi$
are independent of $M_{P}$ and $\rho$ and are functions of $\lambda$ and $\xi$
only.

However, as we have seen, in the $\xi\to\infty$ limit, $\phi^{\prime}$
decouples from the system and the system is reduced to one with three
parameters $\lambda$, $M_{P}$ and $\rho$. This means that the beta function
for $\lambda$ in a mass-independent scheme must be finite in the
$\xi\to\infty$ limit. In other words, the perturbation series for
$\beta_{\lambda}$ must not have a term containing $\xi$. At the 1-loop level,
this is indeed the case, as shown in (1.2). The beta function of $\lambda$ has
not been calculated beyond 1-loop, and it is an interesting question to check
whether this holds at higher loop levels.

Finally, we note that this can be also understood from the meaning of $\xi$ in
(2.5) or (3.5). Actually, it is clear that $\xi$ plays the roll of mass for
$\phi^{\prime}$ or $\psi$. Therefore, in the minimal subtraction scheme it
does not appear in the beta function of the dimensionless coupling constant
$\lambda$.

##  4 Action (1.1) in terms of ordinary metric field

For later convenience, we consider the ordinary perturbation theory of (1.1)
with $\rho=0$ instead of introducing auxiliary field $\phi$. We take the
fluctuation field as

$\displaystyle g_{\mu\nu}=\eta_{\mu\nu}+h_{\mu\nu},$ (4.1)

and the gauge fixing term as

$\displaystyle{\cal
L}_{GF}=-\frac{1}{2a}\Big{(}\partial_{\mu}h^{\mu}{}_{\nu}-\frac{b}{2}\partial^{\nu}h^{\alpha}_{\alpha}\Big{)}^{2},$
(4.2)

where $a$ and $b$ are gauge fixing parameters. We then find the quadratic part
of the action is given by

$\displaystyle{\cal L}^{(2)}$ $\displaystyle=$
$\displaystyle\frac{1}{4}h^{\mu\nu}\Big{[}\left(\frac{1}{\lambda}\Box+\frac{M_{P}^{2}}{2}\right)P^{(2)}+\left(\frac{12}{\xi}\Box-
M_{P}^{2}+\frac{3b^{2}}{2a}\right)P^{(0,s)}+\frac{1}{a}P^{(1)}$ (4.3)
$\displaystyle\hskip
56.9055pt+\frac{(b-2)^{2}}{2a}P^{(0,w)}+\frac{\sqrt{3}b(b-2)}{2a}(P^{(0,sw)}+P^{(0,ws)})\Big{]}_{\mu\nu,\alpha\beta}\Box\,h^{\alpha\beta}.~{}~{}$

where

$\displaystyle P^{(2)}_{\mu\nu,\alpha\beta}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\left(\theta_{\mu\alpha}\theta_{\nu\beta}+\theta_{\mu\beta}\theta_{\nu\alpha}-\frac{2}{3}\theta_{\mu\nu}\theta_{\alpha\beta}\right),$
$\displaystyle P^{(1)}_{\mu\nu,\alpha\beta}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\left(\theta_{\mu\alpha}\omega_{\nu\beta}+\theta_{\mu\beta}\omega_{\nu\alpha}+\theta_{\nu\alpha}\omega_{\mu\beta}+\theta_{\nu\beta}\omega_{\mu\alpha}\right),$
$\displaystyle P^{(0,s)}_{\mu\nu,\alpha\beta}$ $\displaystyle=$
$\displaystyle\frac{1}{3}\theta_{\mu\nu}\theta_{\alpha\beta},\qquad
P^{(0,w)}_{\mu\nu,\alpha\beta}\,=\,\omega_{\mu\nu}\omega_{\alpha\beta},$
$\displaystyle P^{(0,sw)}_{\mu\nu,\alpha\beta}$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{3}}\theta_{\mu\nu}\omega_{\alpha\beta},\qquad
P^{(0,ws)}_{\mu\nu,\alpha\beta}\,=\,\frac{1}{\sqrt{3}}\omega_{\mu\nu}\theta_{\alpha\beta},$
(4.4)

are the spin projectors with

$\displaystyle\theta_{\mu\nu}=\eta_{\mu\nu}-\frac{\partial_{\mu}\partial_{\nu}}{\Box},\qquad\omega_{\mu\nu}=\frac{\partial_{\mu}\partial_{\nu}}{\Box}.$
(4.5)

Then the propagator for the theory (1.1) is given by

$\displaystyle\frac{1}{(2\pi)^{4}}\Big{[}\frac{P^{(2)}}{k^{2}\left[\frac{1}{\lambda}k^{2}-\frac{M_{P}^{2}}{2}\right]}-\frac{a}{k^{2}}P^{(1)}+\frac{P^{(0,s)}}{k^{2}\left[\frac{12}{\xi}k^{2}+M_{P}^{2}\right]}-\frac{\frac{24a}{\xi}k^{2}-3b^{2}+2aM_{P}^{2}}{(b-2)^{2}k^{2}\left[\frac{12}{\xi}k^{2}+M_{P}^{2}\right]}P^{(0,w)}$
$\displaystyle\hskip
56.9055pt-\frac{\sqrt{3}b}{(b-2)k^{2}\left[\frac{12}{\xi}k^{2}+M_{P}^{2}\right]}(P^{(0,sw)}+P^{(0,ws)})\Big{]}_{\mu\nu,\alpha\beta},$
(4.6)

This propagator becomes transparent in the Landau-type gauge, $a=b=0$, as

$\displaystyle\frac{1}{(2\pi)^{4}}\Big{[}\frac{P^{(2)}}{k^{2}\left[\frac{1}{\lambda}k^{2}-\frac{M_{P}^{2}}{2}\right]}+\frac{P^{(0,s)}}{k^{2}\left[\frac{12}{\xi}k^{2}+M_{P}^{2}\right]}\Big{]}_{\mu\nu,\alpha\beta}.$
(4.7)

The first term represents the propagation of the traceless mode, and the
second term represents the propagation of the trace mode (or conformal mode).
At a first glance, both of the limits $\lambda\to\infty$ and $\xi\to\infty$
are dangerous because the ultraviolet behavior is changed in both cases.
However, as we have discussed in section 3, the latter limit is safe. In fact,
by introducing an auxiliary scalar field $\phi$, the quartic behavior of the
second term in (4.7) is resolved as

$\displaystyle\frac{1}{k^{2}\left[\frac{12}{\xi}k^{2}+M_{P}^{2}\right]}=\frac{1}{M_{P}^{2}}\frac{1}{k^{2}}-\frac{1}{M_{P}^{2}}\frac{1}{k^{2}+\frac{\xi
M_{P}^{2}}{12}}.$ (4.8)

The system then becomes a coupled system of the conformal mode and $\phi$, and
the $\xi\to\infty$ limit is nothing but the large mass limit for $\phi$. The
important point is that this decoupling holds even when the interaction is
taken into account, as is discussed in section 3.

##  5  $\xi\to\infty$ limit and structure of bare action

In this section we start with the action (1.1), and examine the form of the
counterterms in the $\xi\to\infty$ limit.

As we have seen, the perturbation series is a double expansion with respect to
$\lambda$ and $\xi$. In particular the beta functions are degree $n+1$
polynomials of $\lambda$ and $\xi$ at the $n$-loop level. Because the simple
pole ($1/\epsilon$) part of the counterterms is related to the beta functions
as (1.3) in the minimal subtraction scheme, the $n$-loop counterterms must
have the form

$S_{n\text{-loop}}=\sum_{k=1}^{n}\frac{1}{\epsilon^{k}}\left[\lambda^{n-1}\left(a^{(n)(k)}_{1}+a^{(n)(k)}_{2}\frac{\xi}{\lambda}+...+a^{(n)(k)}_{n+2}\left(\frac{\xi}{\lambda}\right)^{n+1}\right)C_{\mu\nu\rho\lambda}^{2}+\xi^{n-1}\left(b^{(n)(k)}_{1}+b^{(n)(k)}_{2}\frac{\lambda}{\xi}+...+b^{(n)(k)}_{n+2}\left(\frac{\lambda}{\xi}\right)^{n+1}\right)R^{2}\right].$
(5.1)

Each coefficient of $C_{\mu\nu\rho\lambda}^{2}$ and $R^{2}$ can contain only
up to $\lambda^{-2}$ or $\xi^{-2}$, respectively, in order for the beta
functions not to be singular in the zero coupling constant limit. In
particular, at the $1$-loop level we have

$\displaystyle
S_{1\text{-loop}}=\frac{1}{\epsilon}\left[\left(a_{1}+a_{2}\frac{\xi}{\lambda}+a_{3}\left(\frac{\xi}{\lambda}\right)^{2}\right)C_{\mu\nu\rho\lambda}^{2}+\left(b_{1}+b_{2}\frac{\lambda}{\xi}+b_{3}\left(\frac{\lambda}{\xi}\right)^{2}\right)R^{2}\right].$
(5.2)

Now consider the limit $\xi\to\infty$. If we assume that the bare action
$S_{0}=S+S_{1\text{-loop}}+...$ remains finite in this limit, we have

$\displaystyle a^{(n)(k)}_{2}=...=a^{(n)(k)}_{n+2}=0,\qquad
b^{(n)(k)}_{1}=...=b^{(n)(k)}_{n-1}=0,$ (5.3)

for all $n=1,...$ and $k=1,...,n$. In other words, in (5.1) only the first
term for $C_{\mu\nu\rho\lambda}^{2}$ and the last three terms for $R^{2}$
remain. More explicitly, we can write as

$S_{n\text{-loop}}=\sum_{k=1}^{n}\frac{1}{\epsilon^{k}}\left[\lambda^{n-1}a^{(n)(k)}C_{\mu\nu\rho\lambda}^{2}+\left(\lambda^{n-1}b^{(n)(k)}_{n}+\xi^{-1}\lambda^{n}{b}^{(n)(k)}_{n+1}+\xi^{-2}\lambda^{n+1}{b}^{(n)(k)}_{n+2}\right)R^{2}\right].$
(5.4)

If this assumption is correct to all loop orders, from (1.3) the beta
functions should be given by

$\displaystyle\beta_{\lambda}=\sum_{n=1}^{\infty}4a^{(n)(1)}\lambda^{n+1},\qquad\beta_{\xi}=\sum_{n=1}^{\infty}2\left(b^{(n)(1)}_{n}\lambda^{n-1}\xi^{2}+{b}^{(n)(1)}_{n+1}\lambda^{n}\xi+{b}^{(n)(1)}_{n+2}\lambda^{n+1}\right).$
(5.5)

Therefore, $\beta_{\lambda}$ is a function of only $\lambda$, and
$\beta_{\xi}$ contains $\xi$ at most quadratically. It will be interesting to
see if this is correct at higher loop orders.

##  6 Relation between various beta functions

In the previous sections, we have seen that the action (1.1) is equivalent to
(2.5) or (3.5). Furthermore, if the cosmological constant is zero, $\rho=0$,
$\xi$ is the mass squared for $\psi$. Therefore, in the minimal subtraction
scheme, the beta function for $\lambda$ does not depend on $\xi$, which
confirms the form of the first equation of (1.2):

$\displaystyle\beta_{\lambda}^{o}=-\frac{1}{(4\pi)^{2}}\frac{133}{10}\lambda^{2}.$
(6.1)

However, as we will discuss now, this beta function is not exactly the same as
the beta function for the decoupled action (3.1) obtained by sending
$\xi\to\infty$ in the action, which we denote as $\beta_{\lambda}^{\infty}$.
In fact, it is known that $\beta_{\lambda}^{\infty}$ is given by [2]

$\displaystyle\beta_{\lambda}^{\infty}=-\frac{1}{(4\pi)^{2}}\frac{797}{60}\lambda^{2}.$
(6.2)

The difference is exactly the contribution of a scalar field $\psi$. Actually,
for the action (3.5) in the minimal subtraction scheme, the counterterm does
not depend on the mass of $\psi$. Therefore the beta function is the same as
the massless theory, that is, the theory with $\xi=0$. But this is nothing but
the action (3.1) plus a massless scalar $\phi^{\prime}$. The contribution of a
scalar field to the beta function $\Delta\beta_{\lambda}$ is easily calculated
from the induced Weyl gravity, and it is given by [2, 14]

$\displaystyle\Delta\beta_{\lambda}=-\frac{1}{(4\pi)^{2}}\frac{1}{60}\lambda^{2}.$
(6.3)

And we can indeed find

$\displaystyle\beta_{\lambda}^{o}=\beta_{\lambda}^{\infty}+\Delta\beta_{\lambda}.$
(6.4)

It is now clear that the discrepancy is understood as the failure of the
decoupling in the mass independent renormalization scheme. In fact in the
minimal subtraction scheme, the counterterm remains the same even if the mass
of a field becomes infinite and the field decouples physically. Therefore the
beta function for $\lambda$ of the original theory (1.1) in the minimal
subtraction scheme is not exactly the same as that of the decoupled action
(3.1), but we have an additional contribution from a scalar field.

Related to this is the difference of the coefficients of the beta functions
directly calculated in the conformal gravity [2, 15, 16], where the beta
function is given as

$\displaystyle\beta_{\lambda}=-\frac{1}{(4\pi)^{2}}\frac{199}{15}\lambda^{2},$
(6.5)

The difference between (6.2) and (6.5) is again the same as the contribution
of a scalar field (6.3). This arises because (6.5) is calculated by projecting
out a trace mode in the theory. This suggests that there might be some
subtlety in the $\xi\to 0$ limit related to the contribution of the spin 0
mode. This was studied in [16], but the result remained the same. This point
needs further study.

Finally what is the difference between the beta functions in the minimal
subtraction scheme and those in (1.4)? The beta functions in (1.4) are read
off from the scattering amplitudes. In this case, there are additional
contributions $\log(p^{2}/m^{2})$ with an IR regulator $m^{2}$ when $M_{P}$ is
set to zero. For the beta functions defined so, our simple observation may not
apply because such beta functions are not related to the counterterms.

##  7 Summary

In this paper, we have studied the beta functions for the dimensionless
couplings in the quadratic curvature theory. By considering the general
structure of the counterterms in the theory together with the $\xi\to\infty$
limit, we have argued that the beta function for $\lambda$ should be a
monomial $\lambda^{n+1}$ and that for $\xi$ consists of three terms of
$\lambda^{n+1},\lambda^{n}\xi$ and $\lambda^{n-1}\xi^{2}$ at $n$-loop. This
result is supported by 1-loop calculation, but there has not been any
calculation at 2-loop and beyond. It would be an interesting problem to check
explicitly whether this is indeed true or not at 2-loop and beyond. We also
commented on the difference in the coefficients of several calculations of the
beta functions for $\lambda$, and there seems to be subtlety in the
calculation. Sending $\xi\to\infty$ at the action level or in the beta
function give difference in the mass-independent renormalization. We also
noted the different definition of the beta function given in [12].

## Acknowledgments

We would like to thank Roberto Percacci for valuable comments. H.K. thanks
Prof. Shin-Nan Yang and his family for their kind support through the Chin-Yu
chair professorship. H.K. is partially supported by JSPS (Grants-in-Aid for
Scientific Research Grants No. 20K03970), by the Ministry of Science and
Technology, R.O.C. (MOST 111-2811-M-002-016), and by National Taiwan
University. The work of N.O. was supported in part by the Grant-in-Aid for
Scientific Research Fund of the JSPS (C) No. 20K03980, and by the Ministry of
Science and Technology, R. O. C. (Taiwan) under the grant MOST
112-2811-M-008-016. N.O. would like to thank Kimyeong Lee and Korea Institute
for Advanced Study for their hospitality, where this work was completed.

## References

  * [1] J. Julve and M. Tonin, “Quantum Gravity with Higher Derivative Terms,” Nuovo Cim. B 46 (1978) 137.
  * [2] E. S. Fradkin and A. A. Tseytlin, “Renormalizable Asymptotically Free Quantum Theory of Gravity,” Phys. Lett. B 104 (1981) 377; Nucl. Phys. B 201 (1982) 469.
  * [3] I. G. Avramidi and A. O. Barvinsky, “Asymptotic freedom in higher derivative quantum gravity,” Phys. Lett. B 159 (1985) 269.
  * [4] A. Codello and R. Percacci, “Fixed points of higher derivative gravity,” Phys. Rev. Lett. 97 (2006) 221301 [hep-th/0607128].
  * [5] M. Niedermaier, “Gravitational fixed points and asymptotic safety from perturbation theory,” Nucl. Phys. B 833 (2010) 226.
  * [6] F. Saueressig, K. Groh, S. Rechenberger and O. Zanusso, “Higher derivative gravity from the universal renormalization group machine,” PoS EPS-HEP 2011 (2011) 124, arXiv:1111.1743 [hep-th]
  * [7] G. de Berredo-Peixoto and I. L. Shapiro, “Higher derivative quantum gravity with Gauss-Bonnet term,” Phys. Rev. D 71 (2005) 064005 [hep-th/0412249].
  * [8] N. Ohta and R. Percacci, “Higher Derivative Gravity and Asymptotic Safety in Diverse Dimensions,” Class. Quant. Grav. 31 (2014) 015024 [arXiv:1308.3398 [hep-th]].
  * [9] K. Falls, N. Ohta and R. Percacci, “Towards the determination of the dimension of the critical surface in asymptotically safe gravity,” Phys. Lett. B 810 (2020) 135773 [arXiv:2004.04126 [hep-th]].
  * [10] H. Kawai and N. Ohta, “Wave function renormalization and flow of couplings in asymptotically safe quantum gravity,” Phys. Rev. D 107 (2023) 126025 [arXiv:2305.10591 [hep-th]].
  * [11] N. Ohta, “One-Loop Divergences in Higher-Derivative Gravity,” in C. Bambi, L.Modesto, I. Shapiro (eds) Handbook of Quantum Gravity (Springer, Singapore), https://doi.org/10.1007/978-981-19-3079-9_11-1 [arXiv:2210.02583 [hep-th]].
  * [12] D. Buccio, J. F. Donoghue, G. Menezes and R. Percacci, “Physical running of couplings in quadratic gravity,” [arXiv:2403.02397 [hep-th]].
  * [13] S. Weinberg, “New approach to the renormalization group,” Phys. Rev. D 8 (1973) 3497.
  * [14] I. Antoniadis and E. Mottola, “4-D quantum gravity in the conformal sector,” Phys. Rev. D 45 (1992) 2013.
  * [15] G. de Berredo-Peixoto and I. L. Shapiro, “Conformal quantum gravity with the Gauss-Bonnet term,” Phys. Rev. D 70 (2004) 044024 [arXiv:hep-th/0307030 [hep-th]].
  * [16] N. Ohta and R. Percacci, “Ultraviolet Fixed Points in Conformal Gravity and General Quadratic Theories,” Class. Quant. Grav. 33 (2016), 035001 [arXiv:1506.05526 [hep-th]].


* (6) C. Y. Seng, M. Gorchtein and M. J. Ramsey-Musolf, Dispersive evaluation of the inner radiative correction in neutron and nuclear $\beta$ decay, Phys. Rev. D100 (2019), no. 1 013001 [1812.03352].
  * (7) M. Gorchtein, $\gamma W$ Box Inside Out: Nuclear Polarizabilities Distort the Beta Decay Spectrum, Phys. Rev. Lett. 123 (2019), no. 4 042503 [1812.04229].
  * (8) A. Czarnecki, W. J. Marciano and A. Sirlin, Radiative Corrections to Neutron and Nuclear Beta Decays Revisited, 1907.06737.
  * (9) Particle Data Group Collaboration, P. Zyla et. al., Review of Particle Physics, PTEP 2020 (2020), no. 8 083C01.
  * (10) W. J. Marciano, Precise determination of —V(us)— from lattice calculations of pseudoscalar decay constants, Phys. Rev. Lett. 93 (2004) 231803 [hep-ph/0402299].
  * (11) V. Cirigliano and H. Neufeld, A note on isospin violation in Pl2(gamma) decays, Phys. Lett. B 700 (2011) 7–10 [1102.0563].
  * (12) Flavour Lattice Averaging Group Collaboration, S. Aoki et. al., FLAG Review 2019, 1902.08191.
  * (13) B. Belfatto, R. Beradze and Z. Berezhiani, The CKM unitarity problem: A trace of new physics at the TeV scale?, Eur. Phys. J. C 80 (2020), no. 2 149 [1906.02714].
  * (14) W. Tan, Laboratory tests of the ordinary-mirror particle oscillations and the extended CKM matrix, 1906.10262.
  * (15) Y. Grossman, E. Passemar and S. Schacht, On the Statistical Treatment of the Cabibbo Angle Anomaly, JHEP 07 (2020) 068 [1911.07821].
  * (16) A. M. Coutinho, A. Crivellin and C. A. Manzari, Global Fit to Modified Neutrino Couplings and the Cabibbo-Angle Anomaly, Phys. Rev. Lett. 125 (2020), no. 7 071802 [1912.08823].
  * (17) K. Cheung, W.-Y. Keung, C.-T. Lu and P.-Y. Tseng, Vector-like Quark Interpretation for the CKM Unitarity Violation, Excess in Higgs Signal Strength, and Bottom Quark Forward-Backward Asymmetry, JHEP 05 (2020) 117 [2001.02853].
  * (18) A. Crivellin and M. Hoferichter, Beta decays as sensitive probes of lepton flavor universality, 2002.07184.
  * (19) M. Endo and S. Mishima, Muon g-2 and CKM unitarity in extra lepton models, JHEP 08 (2020), no. 08 004 [2005.03933].
  * (20) B. Capdevila, A. Crivellin, C. A. Manzari and M. Montull, Explaining $b\to s\ell^{+}\ell^{-}$ and the Cabibbo Angle Anomaly with a Vector Triplet, 2005.13542.
  * (21) M. Kirk, The Cabibbo anomaly versus electroweak precision tests – an exploration of extensions of the Standard Model, 2008.03261.
  * (22) E. S. Ginsberg, Radiative Corrections to Kl-3 + /- Decays, Phys. Rev. 142 (1966) 1035–1040.
  * (23) E. S. Ginsberg, Radiative corrections to k-e-3-neutral decays and the delta-i=1/2 rule. (erratum), Phys. Rev. 171 (1968) 1675. [Erratum: Phys.Rev. 174, 2169 (1968)].
  * (24) E. S. Ginsberg, Radiative corrections to the k-l-3 +- dalitz plot, Phys. Rev. 162 (1967) 1570. [Erratum: Phys.Rev. 187, 2280 (1969)].
  * (25) E. Ginsberg, Radiative corrections to k-mu-3 decays, Phys. Rev. D 1 (1970) 229–239.
  * (26) T. Becherrawy, Radiative Correction to K(l3) Decay, Phys. Rev. D 1 (1970) 1452–1468.
  * (27) V. Bytev, E. Kuraev, A. Baratt and J. Thompson, Radiative corrections to the K+-(e3) decay revised, Eur. Phys. J. C 27 (2003) 57–71 [hep-ph/0210049]. [Erratum: Eur.Phys.J.C 34, 523–524 (2004)].
  * (28) T. C. Andre, Radiative corrections in K0(l3) decays, Annals Phys. 322 (2007) 2518–2544 [hep-ph/0406006].
  * (29) A. Garcia and M. Maya, model independent radiative corrections to M+-(l3) decays, Phys. Rev. D 23 (1981) 2603.
  * (30) C. Juarez-Leon, A. Martinez, M. Neri, J. Torres and R. Flores-Mendieta, Radiative corrections to the Dalitz plot of $K_{l3}^{\pm}$ decays, Phys. Rev. D 83 (2011) 054004 [1010.5547]. [Erratum: Phys.Rev.D 86, 059901 (2012)].
  * (31) J. Torres, A. Martinez, M. Neri, C. Juarez-Leon and R. Flores-Mendieta, Radiative corrections to the Dalitz plot of $K_{l3}^{\pm}$ decays: Contribution of the four-body region, Phys. Rev. D 86 (2012) 077501 [1209.5759].
  * (32) M. Neri, A. Martínez, C. Juárez-León, J. Torres and R. Flores-Mendieta, Radiative corrections to the Dalitz plot of $K_{l3}^{0}$ decays, Phys. Rev. D 92 (2015), no. 7 074022 [1510.00401].
  * (33) V. Cirigliano, M. Knecht, H. Neufeld, H. Rupertsberger and P. Talavera, Radiative corrections to K(l3) decays, Eur. Phys. J. C23 (2002) 121–133 [hep-ph/0110153].
  * (34) V. Cirigliano, H. Neufeld and H. Pichl, K(e3) decays and CKM unitarity, Eur. Phys. J. C 35 (2004) 53–65 [hep-ph/0401173].
  * (35) V. Cirigliano, M. Giannotti and H. Neufeld, Electromagnetic effects in K(l3) decays, JHEP 11 (2008) 006 [0807.4507].
  * (36) R. Urech, Virtual photons in chiral perturbation theory, Nucl. Phys. B433 (1995) 234–254 [hep-ph/9405341].
  * (37) M. Knecht, H. Neufeld, H. Rupertsberger and P. Talavera, Chiral perturbation theory with virtual photons and leptons, Eur. Phys. J. C12 (2000) 469–478 [hep-ph/9909284].
  * (38) B. Ananthanarayan and B. Moussallam, Four-point correlator constraints on electromagnetic chiral parameters and resonance effective Lagrangians, JHEP 06 (2004) 047 [hep-ph/0405206].
  * (39) S. Descotes-Genon and B. Moussallam, Radiative corrections in weak semi-leptonic processes at low energy: A Two-step matching determination, Eur. Phys. J. C42 (2005) 403–417 [hep-ph/0505077].
  * (40) C.-Y. Seng, D. Galviz and U.-G. Meißner, A New Theory Framework for the Electroweak Radiative Corrections in $K_{l3}$ Decays, JHEP 02 (2020) 069 [1910.13208].
  * (41) X. Feng, M. Gorchtein, L.-C. Jin, P.-X. Ma and C.-Y. Seng, First-principles calculation of electroweak box diagrams from lattice QCD, Phys. Rev. Lett. 124 (2020), no. 19 192002 [2003.09798].
  * (42) C.-Y. Seng, X. Feng, M. Gorchtein and L.-C. Jin, Joint lattice QCD–dispersion theory analysis confirms the quark-mixing top-row unitarity deficit, Phys. Rev. D 101 (2020), no. 11 111301 [2003.11264].
  * (43) C.-Y. Seng, X. Feng, M. Gorchtein, L.-C. Jin and U.-G. Meißner, New method for calculating electromagnetic effects in semileptonic beta-decays of mesons, JHEP 10 (2020) 179 [2009.00459].
  * (44) P.-X. Ma, X. Feng, M. Gorchtein, L.-C. Jin and C.-Y. Seng, Lattice QCD calculation of the electroweak box diagrams for the kaon semileptonic decays, 2102.12048.
  * (45) C.-Y. Seng, D. Galviz, M. Gorchtein and U.-G. Meißner, High-precision determination of the $K_{e3}$ radiative corrections, 2103.00975.
  * (46) NA48/2 Collaboration, J. R. Batley et. al., Measurement of the form factors of charged kaon semileptonic decays, JHEP 10 (2018) 150 [1808.09041].
  * (47) J. Erler, Electroweak radiative corrections to semileptonic tau decays, Rev. Mex. Fis. 50 (2004) 200–202 [hep-ph/0211345].
  * (48) Y.-J. Shi, C.-Y. Seng, F.-K. Guo, B. Kubis, U.-G. Meißner and W. Wang, Two-Meson Form Factors in Unitarized Chiral Perturbation Theory, 2011.00921.
  * (49) N. Meister and D. Yennie, Radiative Corrections to High-Energy Scattering Processes, Phys. Rev. 130 (1963) 1210–1229.
  * (50) A. Sirlin, General Properties of the Electromagnetic Corrections to the Beta Decay of a Physical Nucleon, Phys. Rev. 164 (1967) 1767–1775.
  * (51) D. Wilkinson and B. Macefield, The numerical evaluation of radiative corrections of order $\alpha$ to allowed nuclear $\beta$-decay, Nucl. Phys. A 158 (1970) 110–116.
  * (52) H. H. Patel, Package-X: A Mathematica package for the analytic calculation of one-loop integrals, Comput. Phys. Commun. 197 (2015) 276–290 [1503.01469].
  * (53) H. H. Patel, Package-X 2.0: A Mathematica package for the analytic calculation of one-loop integrals, Comput. Phys. Commun. 218 (2017) 66–70 [1612.00009].
  * (54) NA7 Collaboration, S. Amendolia et. al., A Measurement of the Space - Like Pion Electromagnetic Form-Factor, Nucl. Phys. B 277 (1986) 168.
  * (55) S. Amendolia et. al., A Measurement of the Kaon Charge Radius, Phys. Lett. B 178 (1986) 435–440.
  * (56) B. Moussallam, A Sum rule approach to the violation of Dashen’s theorem, Nucl. Phys. B 504 (1997) 381–414 [hep-ph/9701400].
  * (57) J. Bijnens and G. Ecker, Mesonic low-energy constants, Ann. Rev. Nucl. Part. Sci. 64 (2014) 149–174 [1405.6488].
  * (58) P. Baikov, K. Chetyrkin and J. Kuhn, Adler Function, Bjorken Sum Rule, and the Crewther Relation to Order $\alpha^{4}_{s}$ in a General Gauge Theory, Phys. Rev. Lett. 104 (2010) 132004 [1001.3606].
  * (59) W. J. Marciano and A. Sirlin, Radiative corrections to pi(lepton 2) decays, Phys. Rev. Lett. 71 (1993) 3629–3632.
  * (60) A. Sirlin, Current Algebra Formulation of Radiative Corrections in Gauge Theories and the Universality of the Weak Interactions, Rev. Mod. Phys. 50 (1978) 573. [Erratum: Rev. Mod. Phys.50,905(1978)].
  * (61) A. Sirlin, Large m(W), m(Z) Behavior of the O(alpha) Corrections to Semileptonic Processes Mediated by W, Nucl. Phys. B196 (1982) 83–92.
  * (62) V. Cirigliano, G. Ecker, H. Neufeld, A. Pich and J. Portoles, Kaon Decays in the Standard Model, Rev. Mod. Phys. 84 (2012) 399 [1107.6001].
  * (63) FlaviaNet Working Group on Kaon Decays Collaboration, M. Antonelli et. al., An Evaluation of $|V_{us}|$ and precise tests of the Standard Model from world data on leptonic and semileptonic kaon decays, Eur. Phys. J. C 69 (2010) 399–424 [1005.2323].
  * (64) A. Czarnecki, W. J. Marciano and A. Sirlin, Pion beta decay and Cabibbo-Kobayashi-Maskawa unitarity, Phys. Rev. D 101 (2020), no. 9 091301 [1911.04685].
  * (65) A. Bazavov et. al., Kaon semileptonic vector form factor and determination of $|V_{us}|$ using staggered fermions, Phys. Rev. D 87 (2013) 073012 [1212.4993].
  * (66) RBC/UKQCD Collaboration, P. A. Boyle et. al., The kaon semileptonic form factor in Nf = 2 + 1 domain wall lattice QCD with physical light quark masses, JHEP 06 (2015) 164 [1504.01692].
  * (67) A. Bazavov et. al., Determination of $|V_{us}|$ from a Lattice-QCD Calculation of the $K\to\pi\ell\nu$ Semileptonic Form Factor with Physical Quark Masses, Phys. Rev. Lett. 112 (2014), no. 11 112001 [1312.1228].
  * (68) N. Carrasco, P. Lami, V. Lubicz, L. Riggio, S. Simula and C. Tarantino, $K\to\pi$ semileptonic form factors with $N_{f}=2+1+1$ twisted mass fermions, Phys. Rev. D 93 (2016), no. 11 114512 [1602.04113].
  * (69) Fermilab Lattice, MILC Collaboration, A. Bazavov et. al., $|V_{us}|$ from $K_{\ell 3}$ decay and four-flavor lattice QCD, Phys. Rev. D99 (2019), no. 11 114509 [1809.02827].
  * (70) PACS Collaboration, J. Kakazu, K.-i. Ishikawa, N. Ishizuka, Y. Kuramashi, Y. Nakamura, Y. Namekawa, Y. Taniguchi, N. Ukita, T. Yamazaki and T. Yoshié, $K_{l3}$ form factors at the physical point on a $(10.9fm)^{3}$ volume, Phys. Rev. D 101 (2020), no. 9 094504 [1912.13127].
  * (71) M. Antonelli et. al., Flavor Physics in the Quark Sector, Phys. Rept. 494 (2010) 197–414 [0907.5386].
  * (72) KTeV Collaboration, E. Abouzaid et. al., Dispersive analysis of K (L mu3) and K (L e3) scalar and vector form factors using KTeV data, Phys. Rev. D 81 (2010) 052001 [0912.1291].
  * (73) V. Bernard, M. Oertel, E. Passemar and J. Stern, K(mu3)**L decay: A Stringent test of right-handed quark currents, Phys. Lett. B638 (2006) 480–486 [hep-ph/0603202].
  * (74) V. Bernard and E. Passemar, Matching chiral perturbation theory and the dispersive representation of the scalar K pi form-factor, Phys. Lett. B661 (2008) 95–102 [0711.3450].
  * (75) V. Bernard, M. Oertel, E. Passemar and J. Stern, Dispersive representation and shape of the K(l3) form factors: Robustness, Phys. Rev. D80 (2009) 034034 [0903.1654].
  * (76) R. J. Hill, Constraints on the form factors for K —$>$ pi l nu and implications for —V(us)—, Phys. Rev. D 74 (2006) 096006 [hep-ph/0607108].
  * (77) J. Gasser and H. Leutwyler, Low-Energy Expansion of Meson Form-Factors, Nucl. Phys. B 250 (1985) 517–538.
  * (78) G. Colangelo, S. Lanz, H. Leutwyler and E. Passemar, Dispersive analysis of $\eta\rightarrow 3\pi$, Eur. Phys. J. C 78 (2018), no. 11 947 [1807.11937].
  * (79) G. Ecker, J. Gasser, A. Pich and E. de Rafael, The Role of Resonances in Chiral Perturbation Theory, Nucl. Phys. B 321 (1989) 311–342.
  * (80) G. Ecker, J. Gasser, H. Leutwyler, A. Pich and E. de Rafael, Chiral Lagrangians for Massive Spin 1 Fields, Phys. Lett. B 223 (1989) 425–432.
  * (81) V. Cirigliano, G. Ecker, M. Eidemuller, R. Kaiser, A. Pich and J. Portoles, Towards a consistent estimate of the chiral low-energy constants, Nucl. Phys. B 753 (2006) 139–177 [hep-ph/0603205].
  * (82) J. Gasser and H. Leutwyler, Chiral Perturbation Theory to One Loop, Annals Phys. 158 (1984) 142.
  * (83) R. Gastmans and R. Meuldermans, Dimensional regularization of the infrared problem, Nucl. Phys. B 63 (1973) 277–284.
  * (84) W. Marciano and A. Sirlin, Dimensional Regularization of Infrared Divergences, Nucl. Phys. B 88 (1975) 86–98.