# An Investigation of Language Model Interpretability via Sentence Editing

Samuel Stevens
The Ohio State University
<EMAIL_ADDRESS>
&Yu Su
The Ohio State University
<EMAIL_ADDRESS>

###### Abstract

Pre-trained language models (PLMs) like BERT are being used for almost all
language-related tasks, but interpreting their behavior still remains a
significant challenge and many important questions remain largely unanswered.
In this work, we re-purpose a sentence editing dataset, where faithful high-
quality human rationales can be automatically extracted and compared with
extracted model rationales, as a new testbed for interpretability. This
enables us to conduct a systematic investigation on an array of questions
regarding PLMs’ interpretability, including the role of pre-training
procedure, comparison of rationale extraction methods, and different layers in
the PLM. The investigation generates new insights, for example, contrary to
the common understanding, we find that attention weights correlate well with
human rationales and work better than gradient-based saliency in extracting
model rationales. Both the dataset and code are available at
https://github.com/samuelstevens/sentence-editing-interpretability to
facilitate future interpretability research.

##  1 Introduction

Pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019;
Beltagy et al., 2019) are pervasively used in language-related tasks, but
interpreting their predictions is notoriously difficult because of their
parameters’ complex inter-dependencies. Given a specific prediction, we want
to know why a model made that decision, both to further improve performance
and to use the model in high-stakes scenarios such as healthcare or bank loan
approvals where interpretability is important. This has motivated efforts in
extracting model explanations, typically in the form of rationales, i.e.,
subsets of the original input that support a decision (Zaidan et al., 2007).
Attention heatmaps (Xu et al., 2015) and gradient-based saliency maps
(Simonyan et al., 2014) are common extraction methods.

The algorithm descripted described in the previous
sections has several advantages.
---
The algorithm descripted$\to$described in the previous sections has several
advantages.
However, we must note that we still have no means of deciding which
documents out of _MATH_ deserve to be in _MATH_ and _MATH_, respectively.
However, we must note that we still have no means of deciding which documents
out of _MATH_ deserve to be in _MATH_ and _MATH_, respectively.
Figure 1:  Two “need edit” examples from AESW in the original data format and
a human-readable format. The first example (a) has a spelling error
“descripted” and the second (b) is edited for concision.

There have been efforts on developing datasets for interpretability research,
for example, the recent ERASER benchmark DeYoung et al. (2020). However, the
majority of ERASER tasks use human rationales highlighted by a different
annotator after the original labeling process. Such rationales are not
necessarily faithful; a rationale highlighted by the second annotator may not
have been actually used by the first annotator while labeling. Manual
rationale labeling is also difficult and time-consuming; of the six datasets
in the ERASER benchmark, only one has more than 200 examples.

Our first contribution is the realization that AESW (Automatic Evaluation of
Scientific Writing; Daudaravicius et al., 2016), a sentence editing dataset,
contains thousands of faithful human rationales that can be automatically re-
purposed for interpretability research. See Figure 1 for examples. This
provides a new, large-scale dataset with truly faithful human rationales for
interpretability questions surrounding model rationales.

Our second contribution is investigating multiple factors in PLM rationale
plausibility. More plausible rationales are valuable in human-in-the-loop
systems where humans use model rationales to make a final decision. We compare
(1) pre-training procedures, (2) attention weight- and input gradient-based
methods of extracting model rationales, (3) correlation between model
rationale plausibility and model confidence, and (4) differences in
transformer layers. While previous work (Jain and Wallace, 2019; Serrano and
Smith, 2019) has shown that attention weights are not always faithful, we find
that they correlate with human rationales better than gradient-based methods.

##  2 Related Work

Human rationales (as defined by Zaidan et al., 2007) are subsets of input
highlighted by human annotators as evidence to support a decision. The same
annotator labeling an example might also highlight their rationale (Khashabi
et al., 2018; Thorne et al., 2018). In other cases, rationales are collected
for an existing dataset by different annotators (Zaidan et al., 2008; Camburu
et al., 2018; Rajani et al., 2019). As previously stated, such rationales may
not be faithful. Rationale length can vary from sub-sentence spans (Talmor et
al., 2019) to multiple sentences (Lehman et al., 2019).

Model rationales can be produced as an explicit training objective (Zaidan et
al., 2008) or extracted as a post-hoc explanation. Post-hoc methods typically
assign token-level importance scores: attention weights are often used in
attention-based models (Bahdanau et al., 2016), gradient-based explanations
are typical for differentiable models (Denil et al., 2015; Shrikumar et al.,
2017), and LIME is a model-agnostic method (Ribeiro et al., 2016). We follow
work using BERT’s attention (Clark et al., 2019; Kovaleva et al., 2019) to
extract rationales.

A model rationale is evaluated on faithfulness (if it is actually used to make
a decision) and plausibility (if it is easily understood by humans).
Faithfulness can be measured by perturbing inputs marked as evidence and
measuring change in outputs (Jain and Wallace, 2019; Serrano and Smith, 2019).
Plausibility can be measured through user studies, wherein users are given a
model rationale and asked either to predict the model’s decision (Kim et al.,
2016) or to rate rationale understandability (Nguyen, 2018; Ehsan et al.,
2018, 2019; Strout et al., 2019). Rationale plausibility can also be measured
by similarity to human rationales (DeYoung et al., 2020), but this requires
faithful human rationales. We use similarity to evaluate rationale
plausibility because we gather faithful human rationales from sentence editing
annotations.

##  3 Proposed Task

We propose re-purposing the AESW classification task for measuring model
interpretability. We gather examples from AESW from which we can automatically
extract faithful and sufficient human rationales, and then use said rationales
to investigate factors in PLM rationale plausibility, specifically BERT
(Devlin et al., 2019) and its variants. It is worth noting that our rationale
dataset can be used for other interpretability topics such as training with
rationales or evaluating rationale faithfulness; we will focus on plausibility
considering the scope of this paper.

###  3.1 Human Rationales

Human rationales are substrings used as evidence for a decision (Zaidan et
al., 2007). Faithful and sufficient (enough evidence to justify a decision)
human rationales can be used as gold labels for evaluating model rationale
plausibility.

The original AESW task is to predict if a sentence from a scientific paper
needs editing. Daudaravicius et al. extract spans of a sentence before and
after professional editing111 Native English speaking editors working at VTeX.
as deleted () or inserted () and provide 1.1M training, 140K
validation and 140K testing examples. Sentences without changes are assumed to
not require editing.

We exploit the data format to automatically extract faithful and sufficient
human rationales. Delete text (text between  tags) is always a faithful
rationale (such text is a source of the error). For edits with arbitrary
text alone and  text, the  alone is not always a sufficient
rationale to justify “need edit”. Consider a sentence where a verb is
incorrectly conjugated and replaced with the correctly conjugated verb. The
incorrectly conjugated verb is not sufficient to decide that the sentence
needs editing; the surrounding context is required. To find edits where
text is always a sufficient rationale, we use two criteria:

  1. 1.

A spelling error is corrected. (spelling error)

  2. 2.

Text is only deleted, not added. (deleted text)

Spelling errors are always a sufficient rationale to justify editing a
sentence (see Figure 1a). In edits with no insertions, removing the  text
leads to an error-free sentence, so the  text is sufficient explanation
for editing (see Figure 1b). These criteria lead to both a simple, lexical
task (spelling error) and a more challenging semantic task (deleted text) and
constitute a wider range of challenges for future interpretability research.
We extract faithful and sufficient human rationales for 1,321 spelling error
edits and 6,741 deleted text edits from the validation set of AESW.222 More
details on extracting human rationales, as well as additional examples, can be
found in Appendix C

###  3.2 Model Rationales

Model rationales are substrings provided by a model as evidence for a
decision. Given a model, an example $x_{i}$ and a prediction $y_{i}$, we
extract three model rationales.

First, we use attention maps (Xu et al., 2015; Kovaleva et al., 2019) to rank
word relevance. We measure the total attention weight from BERT’s final
layer’s initial [CLS] token to each token $t_{j}$ across $H$ attention heads.
Then we add those totals together for each token $t_{j}$ in a word $w$:

$\text{score}(w)=\sum_{t_{j}\in
w}\sum_{h=1}^{H}\text{Attn}_{h_{j}}\left(\text{[CLS]}\to t_{j}\right).$ (1)

Second, we use gradient-based saliency (specifically gradient$\times$input;
Denil et al., 2015; Shrikumar et al., 2017) to rank word relevance. We
calculate a saliency score for each token $t_{j}$ in $x_{i}$. First, change in
model output with respect to $t_{j}$’s input embedding
$\nabla_{e(t_{j})}f_{y_{i}}(x_{i})$ captures the sensitivity to token $t$. The
dot product with $e(t_{j})$ is then a scalar measure of each token’s marginal
impact on the model prediction (Han et al., 2020). Finally, we again compute a
word-level score by summing over each token $t_{j}$ in word $w$:

$\text{score}(w)=\sum_{t_{j}\in w}\nabla_{e(t_{j})}f_{y_{i}}(x_{i})\cdot
e(t_{j})$ (2)

In contrast to attention, gradient$\times$input faithfully measures the
marginal effect of each input token on the prediction (Bastings and Filippova,
2020). Finally, we extract a third set of rankings using
gradient$\times$input’s magnitude to rank words.

###  3.3 Evaluation

We evaluate extracted rationale plausibility using similarity to human
rationales (DeYoung et al., 2020). We use the continuous word scores generated
in the previous section to rank relevance, then use mean reciprocal rank as an
evaluation metric.333 We performed our analyses with additional, similar
metrics; our main observations remain consistent independent of the choice of
metric. Appendix D contains additional details and complete results.

##  4 Experiments

To demonstrate the utility of the AESW task for interpretability research, we
present four experiments, each with the goal of understanding factors in PLM
rationale plausibility.

For our experiments, we add a linear layer and sigmoid activation function on
top of the [CLS] token representation, fine-tune BERT-base end-to-end on the
AESW training set using the original training objective (classify a sentence
as “need edit” or “no edit”) and use validation loss to tune hyperparameters.
We do not add any interpretability or rationale-related objectives. The
classification F1 for the models on spelling error and deleted text edits is
82.0 and 74.0, respectively.444 Appendix B contains more details on fine-
tuning procedure and results.  We extract and evaluate model rationales on all
spelling error and deleted text edits, even edits that a model does not
correctly predict as “need edit.”

#### Does pre-training procedure affect rationale plausibility?

We are interested in how pre-training affects plausibility after fine-tuning
(to the best of our knowledge, this is a previously unexplored topic). We
compare BERT and two variants, RoBERTa Liu et al. (2019) and SciBERT Beltagy
et al. (2019). These models have the same architecture but differ in pre-
training: RoBERTa is pre-trained for 5x longer with 10x more data than BERT,
and SciBERT is pre-trained on a corpus of academic papers. We extract model
rationales using attention weights and evaluate their plausibility.

As seen in Figure 2, RoBERTa and BERT generate nearly equally plausible
rationales despite differences in pre-training corpus size. We hypothesize
that SciBERT generates less plausible rationales because it encodes “need
edit” representations in earlier layers (rather than in the final layer), then
attends to [SEP] as a no-op in later layers, as proposed in Clark et al.
(2019) and Kobayashi et al. (2020) and further confirmed in the subsequent
experiments.

Figure 2:  Left: BERT, RoBERTa and SciBERT’s attention weight rationale
plausibility. Right: BERT’s attention weight, gradient$\times$input and
$\lvert\text{gradient}\times\text{input}\rvert$ rationale plausibility (see
Section 3.2).

#### Do attention weights or input gradients produce better rationales?

In contrast to attention weights, gradient$\times$input scores are naturally
faithful with respect to individual feature importance (Jain and Wallace,
2019; Bastings and Filippova, 2020). However, attention weights can represent
word relevance in context, potentially leading to more plausible rationales.
We extract and evaluate rationales using attention weights and the two
gradient$\times$input methods described in Section 3.2.

Figure 2 shows that attention-based rationales are more plausible and that the
difference is more pronounced on deleted text edits. Using
$\lvert\text{gradient}\times\text{input}\rvert$ (right-most) also shows
improvements over directional gradient$\times$input (middle-right), in
contrast to Han et al. (2020).

#### Are plausibility and confidence correlated?

We are curious if BERT is more confident in its classification decisions when
it attends to the original evidence used by editors. We quantify BERT’s
classification confidence by treating the sigmoid activation function’s value
as the probability of a “need edit” decision. We only consider examples that
BERT correctly classifies as “need edit” and calculate BERT’s mean confidence.
We find that when human rationales are BERT’s most attended-to words, it is
11.7% and 11.9% more confident in its predictions for spelling error and
deleted text edits, respectively.

One possible explanation for such a correlation is that easy-to-classify
examples also have easy-to-identify rationales. Because agreement with human
rationales is not the training objective, however, we believe this correlation
suggests that BERT learns to classify edits similarly to humans.

#### How does transformer layer affect plausibility?

It is widely agreed that BERT’s early layers encode more lexical and phrasal
information than other layers (Jawahar et al., 2019; Rogers et al., 2020). We
hypothesize that rationales extracted from early layers will be more plausible
for spelling error than deleted text edits because it is a more lexical task.
We extract rationales from each layer’s attention weights and measure their
plausibility in Figure 4.

The results confirm the hypothesis and show that early layers in BERT models
are indeed more lexically-oriented. We also find that SciBERT strongly attends
to spelling errors in earlier layers. We believe it is because that, pre-
trained on well-formed academic text, SciBERT is not as well exposed to
spelling errors as BERT and RoBERTa so it learns to attend to spelling errors
in earlier layers during fine-tuning.

Figure 3:  Mean reciprocal rank for each layer (using the mean strategy) for
each model for spelling error and deleted text edits.44footnotemark: 4

##  5 Conclusion

We re-purpose the AESW task to gather thousands of inherently faithful human
rationales and investigate an array of questions regarding PLM
interpretability. We find, among other new insights, that attention weights
correlate well with human rationales and produce more plausible rationales
than input gradients, which is different from existing understanding.
Furthermore, we find that BERT is more confident in its predictions when it
attends to the same words that a human did, supporting the idea that while
attention is not inherently faithful, attention-based models might rely on the
same information as humans when making a prediction.

Future work might expand the subset of examples for which human rationales can
be automatically extracted, include human rationales during training or
evaluate faithful-by-design model rationales on this dataset.

##  6 Acknowledgements

The authors would like to extend their thanks to the anonymous reviewers for
their insightful feedback.

## References

  * Bahdanau et al. (2016) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016.  Neural machine translation by jointly learning to align and translate.  ArXiv preprint arXiv:1409.0473.
  * Bastings and Filippova (2020) Jasmijn Bastings and Katja Filippova. 2020.  The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 149–155, Online. Association for Computational Linguistics.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018\.  e-snli: Natural language inference with natural language explanations.  In _Advances in Neural Information Processing Systems_ , volume 31, pages 9539–9549. Curran Associates, Inc.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Daudaravicius et al. (2016) Vidas Daudaravicius, Rafael E. Banchs, Elena Volodina, and Courtney Napoles. 2016\.  A report on the automatic evaluation of scientific writing shared task.  In _Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 53–62, San Diego, CA. Association for Computational Linguistics.
  * Denil et al. (2015) Misha Denil, Alban Demiraj, and Nando de Freitas. 2015.  Extraction of salient sentences from labelled documents.  ArXiv preprint arXiv:1412.6815.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * DeYoung et al. (2020) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020.  ERASER: A benchmark to evaluate rationalized NLP models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4443–4458, Online. Association for Computational Linguistics.
  * Ehsan et al. (2018) Upol Ehsan, Brent Harrison, Larry Chan, and Mark O. Riedl. 2018.  Rationalization: A neural machine translation approach to generating natural language explanations.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’18, page 81–87, New York, NY, USA. Association for Computing Machinery.
  * Ehsan et al. (2019) Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O. Riedl. 2019\.  Automated rationale generation: A technique for explainable ai and its effects on human perceptions.  In _Proceedings of the 24th International Conference on Intelligent User Interfaces_ , IUI ’19, page 263–274, New York, NY, USA. Association for Computing Machinery.
  * Han et al. (2020) Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. 2020.  Explaining black box predictions and unveiling data artifacts through influence functions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5553–5563, Online. Association for Computational Linguistics.
  * Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019.  Attention is not Explanation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3543–3556, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 252–262, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kim et al. (2016) Been Kim, Rajiv Khanna, and Oluwasanmi Koyejo. 2016.  Examples are not enough, learn to criticize! criticism for interpretability.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , NIPS’16, page 2288–2296, Red Hook, NY, USA. Curran Associates Inc.
  * Kobayashi et al. (2020) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020.  Attention is not only a weight: Analyzing transformers with vector norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7057–7075, Online. Association for Computational Linguistics.
  * Kovaleva et al. (2019) Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019.  Revealing the dark secrets of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4365–4374, Hong Kong, China. Association for Computational Linguistics.
  * Lehman et al. (2019) Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C. Wallace. 2019.  Inferring which medical treatments work from reports of clinical trials.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3705–3717, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  ArXiv preprint arXiv:1907.11692.
  * Nguyen (2018) Dong Nguyen. 2018.  Comparing automatic and human evaluation of local explanations for text classification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1069–1078, New Orleans, Louisiana. Association for Computational Linguistics.
  * Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Explain yourself! leveraging language models for commonsense reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4932–4942, Florence, Italy. Association for Computational Linguistics.
  * Ribeiro et al. (2016) Marco Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  “why should I trust you?”: Explaining the predictions of any classifier.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_ , pages 97–101, San Diego, California. Association for Computational Linguistics.
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020.  A primer in bertology: What we know about how bert works.  _Transactions of the Association for Computational Linguistics_ , 8:842–866.
  * Serrano and Smith (2019) Sofia Serrano and Noah A. Smith. 2019.  Is attention interpretable? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2931–2951, Florence, Italy. Association for Computational Linguistics.
  * Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017.  Learning important features through propagating activation differences.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3145–3153, International Convention Centre, Sydney, Australia. PMLR.
  * Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  ArXiv preprint arXiv:1312.6034.
  * Strout et al. (2019) Julia Strout, Ye Zhang, and Raymond Mooney. 2019.  Do human rationales improve machine explanations? In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 56–62, Florence, Italy. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Huggingface’s transformers: State-of-the-art natural language processing.  ArXiv preprint arXiv:1910.03771.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015.  Show, attend and tell: Neural image caption generation with visual attention.  In _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pages 2048–2057, Lille, France. PMLR.
  * Zaidan et al. (2007) Omar Zaidan, Jason Eisner, and Christine Piatko. 2007.  Using “annotator rationales” to improve machine learning for text categorization.  In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference_ , pages 260–267, Rochester, New York. Association for Computational Linguistics.
  * Zaidan et al. (2008) Omar F. Zaidan, Jason Eisner, and Christine Piatko. 2008.  Machine learning with annotator rationales to reduce annotation cost.  In _Proceedings of the NIPS*2008 Workshop on Cost Sensitive Learning_.

##  Appendix A AESW Details

The original AESW task is to classify a sentence from an academic paper as
“need edit” or “no edit”. There is no special markup attached to the sentence
when it is given as input to a model. Task participants receive the training
and validation sets with  and  tokens for model development, and
Daudaravicius et al. provided an automatic test set evaluation through an
online portal. Daudaravicius et al. collected these sentences from 9,919
journal articles published by Springer Publishing Company and edited at VTeX.
Sentences before and after editing were automatically aligned using a modified
diff algorithm.555More details can be found in Daudaravicius et al. (2016)
After the challenge, Daudaravicius et al. released the dataset in its entirety
(all three datasets with all tokens/spans included).

##  Appendix B Training Details

The AESW task uses scientific papers written in LaTeX, which contains markup
characters that impact sentence meaning. The original authors (Daudaravicius
et al.) replace these characters with special tokens, as seen in Table 1. We
add these four special tokens (_MATH_, _MATHDISP_, _CITE_ and _REF_) to the
model vocabulary, fine-tuning the word representations during training.

LaTeX Example | Special Token
---|---
$\beta_{2}$ | _MATH_
$$2 + 3$$ | _MATHDISP_
\cite{google2018} | _CITE_
\ref{tab:results} | _REF_
Table 1: Special tokens found in the original AESW data that should not be
split further into bytes/tokens.

We train all models for a maximum of 30 epochs with a patience of 5 on a
single Tesla P100 GPU.

All models
(BERT666https://huggingface.co/transformers/v3.0.2/model_doc/bert.html#bertforsequenceclassification,
SciBERT777https://github.com/allenai/scibert#pytorch-huggingface-models,
RoBERTA888https://huggingface.co/transformers/v3.0.2/model_doc/roberta.html#robertaforsequenceclassification)
are based on their HuggingFace (Wolf et al., 2020) implementations.

We list all the key hyperparameters and tuning bounds for reproducibility in
Table 3. Our final results for all three BERT-based models along with the top
three models from Daudaravicius et al. (2016) can be found in Table 2.
Additionally, we will release code and instructions for reproducing our
results.

Model | Dev F1 | Test F1
---|---|---
CNN+LSTM | – | 0.628
CNN | – | 0.611
SVM | – | 0.555
BERT${}_{\textbf{base}}$ | 0.654 | 0.666
RoBERTa${}_{\textbf{base}}$ | 0.661 | 0.670
SciBERT${}_{\textbf{scivocab}}$ | 0.658 | 0.668
Table 2: Performance on the original AESW sentence classification task. Dev set results are not available for models reported in Daudaravicius et al. (2016). Model | Hyperparameters | Hyperparameter bounds
---|---|---
BERT${}_{\text{base}}$ |  learning rate: $1\times 10^{-6}$ batch size: 32 model: bert-base-uncased vocab size: 30526 (normally 30522) |  learning rate: ($2\times 10^{-7}$, $1\times 10^{-6}$, $2\times 10^{-5}$, $1\times 10^{-4}$)
RoBERTa${}_{\text{base}}$ |  learning rate: $1\times 10^{-6}$ batch size: 32 model: roberta-base vocab size: 50269 (normally 50265) |  learning rate: ($1\times 10^{-6}$)
SciBERT |  learning rate: $1\times 10^{-6}$ batch size: 32 model: allenai/scibert_scivocab_uncased vocab size: 31094 (normally 31090) |  learning rate: ($1\times 10^{-6}$)
Table 3: Hyperparameter options for each model. Note that each model had 4
special tokens added to the vocabulary. BERT was fine-tuned first. Because of
compute limitations, RobERTa and SciBERT were both fine-tuned using the same
hyperparameters as the optimal BERT configuration (learning rate of ($1\times
10^{-6}$)).

##  Appendix C Gathering Human Rationales

To find spelling error edits, we look for sentences with a deleted, misspelled
word followed by an inserted, correctly spelled word. The first two examples
in Figure 4 are examples of spelling error edits, while the third is not.

To find deleted text edits, we look for sentences where text is removed but
not added. The fourth and fifth examples in Figure 4 are examples of deleted
text edits.

The algorithm descripteddescribed in the previous
sections has several advantages.
---
A spelling error edit; “descripted” is a spelling error and is corrected.
For each energy point, the thereby obtained cross-section values and errors
from different experiments have been further averaged according to the
weigthedweighted average method used by the Particle
Data Group _CITE_, including error rescaling by _MATH_ in case of large
discrepancy.
A spelling error edit; “weigthed” is a spelling error and is corrected.
And the short notations for the denominators are _MATHDISP_, Furthermore, the
following relations are useful to shortshortcut the
expressions: _MATHDISP_.
Not a spelling error edit; “short” is incorrect in this context, but it is not
a spelling error.
However, we must note that we still have no means of deciding which
documents out of _MATH_ deserve to be in _MATH_ and _MATH_, respectively.
A deleted text edit; text is only removed, while no text is inserted.
Let _MATH_ be the conjugate Holder’s function of a
Holder’s function _MATH_.
A deleted text edit; text is only removed, while no text is inserted.
Figure 4: Additional examples of the two types of edits extracted from the
original AESW dataset.

##  Appendix D Evaluating Model Rationales

Although the main text presents our analyses using mean reciprocal rank, we
performed our analyses with multiple metrics. Here we provide specific
definitions for our metrics and our complete results for all models with every
metric.

#### Mean reciprocal rank

Because human rationales can be made up of multiple words, we need to modify
mean reciprocal ranking. For a single sentence, a model’s ordered ranking of
words $M$ and an unordered human rationale $H$:

  1. 1.

Find the top ranked word $w$ in $M$ from $H$ and record the rank.

  2. 2.

Remove $w$ from $M$ and $H$.

  3. 3.

Repeat until $H$ is empty.

  4. 4.

Use the reciprocal of the mean rank.

If we did not remove words from $M$, a perfect score for a rationale with
multiple words would be impossible. Consider the sentence “The boy ate the
found his ball.” If the rationale ranked ‘ate’ and ‘the’ as most important,
the mean rationale rank would be $(1+2)/2=1.5$ and the rationale’s reciprocal
rank would be $\nicefrac{{1}}{{1.5}}=0.66$, despite a perfect rationale.

We take the mean reciprocal ranking across all examples to evaluate
rationales.

#### Mean area under precision-recall curve

Using the model relevance scores for words in a sentence, we adjust the
threshold for classifying a word as part of the rationale, calculate a
precision-recall curve and measure the area underneath. We take the mean AUPRC
across all examples.

#### Mean top 1 match

We score a rationale as 1 if the rationale’s top ranked word is the human
rationale and 0 otherwise. This means that all multiple-word rationales are
automatically not a match and scored as 0. We take the mean of these scores
across all examples.

Our main observations are consistent across metrics: Table 4 contains all
results for models on spelling error edits and Table 5 contains all results
for models on deleted text edits.

Model | F1 | Method | Classification | Mean Recip. | AUPRC | Top 1 Match
---|---|---|---|---|---|---
BERT | 82.1 | Attention | All | 0.841 | 0.115 | 0.756
BERT | 100.0 | Attention | Correct | 0.895 | 0.081 | 0.824
BERT | 0.0 | Attention | Wrong | 0.717 | 0.194 | 0.602
BERT | 82.1 | gradient$\times$input | All | 0.407 | 0.332 | 0.325
BERT | 100.0 | gradient$\times$input | Correct | 0.401 | 0.335 | 0.322
BERT | 0.0 | gradient$\times$input | Wrong | 0.422 | 0.325 | 0.331
BERT | 82.1 | $\lvert\text{gradient}\times\text{input}\rvert$ | All | 0.701 | 0.201 | 0.579
BERT | 100.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Correct | 0.730 | 0.184 | 0.615
BERT | 0.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Wrong | 0.635 | 0.241 | 0.498
RoBERTa | 81.4 | Attention | All | 0.834 | 0.118 | 0.739
RoBERTa | 100.0 | Attention | Correct | 0.886 | 0.085 | 0.811
RoBERTa | 0.0 | Attention | Wrong | 0.721 | 0.192 | 0.581
RoBERTa | 81.4 | gradient$\times$input | All | 0.310 | 0.380 | 0.231
RoBERTa | 100.0 | gradient$\times$input | Correct | 0.303 | 0.383 | 0.229
RoBERTa | 0.0 | gradient$\times$input | Wrong | 0.324 | 0.375 | 0.236
RoBERTa | 81.4 | $\lvert\text{gradient}\times\text{input}\rvert$ | All | 0.675 | 0.219 | 0.546
RoBERTa | 100.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Correct | 0.707 | 0.201 | 0.583
RoBERTa | 0.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Wrong | 0.605 | 0.259 | 0.463
SciBERT | 86.8 | Attention | All | 0.739 | 0.203 | 0.577
SciBERT | 100.0 | Attention | Correct | 0.830 | 0.138 | 0.707
SciBERT | 0.0 | Attention | Wrong | 0.442 | 0.415 | 0.152
SciBERT | 86.8 | gradient$\times$input | All | 0.475 | 0.296 | 0.399
SciBERT | 100.0 | gradient$\times$input | Correct | 0.467 | 0.296 | 0.401
SciBERT | 0.0 | gradient$\times$input | Wrong | 0.501 | 0.295 | 0.392
SciBERT | 86.8 | $\lvert\text{gradient}\times\text{input}\rvert$ | All | 0.670 | 0.214 | 0.557
SciBERT | 100.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Correct | 0.698 | 0.197 | 0.591
SciBERT | 0.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Wrong | 0.575 | 0.268 | 0.447
Table 4: Results for all three models for spelling error edits across all methods of selecting rationales (attention, gradient$\times$input and $\lvert\text{gradient}\times\text{input}\rvert$) and all metrics to evaluate rationales (mean reciprocal ranking, AUPRC and mean top 1 match). Model | F1 | Method | Classification | Mean Recip. | AUPRC | Top 1 Match
---|---|---|---|---|---|---
BERT | 74.0 | Attention | All | 0.561 | 0.272 | 0.411
BERT | 100.0 | Attention | Correct | 0.680 | 0.211 | 0.528
BERT | 0.0 | Attention | Wrong | 0.391 | 0.359 | 0.244
BERT | 74.0 | gradient$\times$input | All | 0.189 | 0.447 | 0.074
BERT | 100.0 | gradient$\times$input | Correct | 0.195 | 0.442 | 0.089
BERT | 0.0 | gradient$\times$input | Wrong | 0.181 | 0.454 | 0.054
BERT | 74.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | All | 0.306 | 0.416 | 0.130
BERT | 100.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Correct | 0.354 | 0.399 | 0.166
BERT | 0.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Wrong | 0.238 | 0.440 | 0.078
RoBERTa | 74.8 | Attention | All | 0.598 | 0.263 | 0.426
RoBERTa | 100.0 | Attention | Correct | 0.662 | 0.230 | 0.487
RoBERTa | 0.0 | Attention | Wrong | 0.503 | 0.312 | 0.335
RoBERTa | 74.8 | gradient$\times$input | All | 0.168 | 0.456 | 0.058
RoBERTa | 100.0 | gradient$\times$input | Correct | 0.156 | 0.458 | 0.054
RoBERTa | 0.0 | gradient$\times$input | Wrong | 0.187 | 0.452 | 0.063
RoBERTa | 74.8 | $\lvert\text{gradient}\times\text{input}\rvert$ | All | 0.321 | 0.404 | 0.154
RoBERTa | 100.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Correct | 0.367 | 0.384 | 0.194
RoBERTa | 0.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Wrong | 0.252 | 0.434 | 0.094
SciBERT | 73.9 | Attention | All | 0.501 | 0.327 | 0.304
SciBERT | 100.0 | Attention | Correct | 0.605 | 0.270 | 0.414
SciBERT | 0.0 | Attention | Wrong | 0.353 | 0.406 | 0.148
SciBERT | 73.9 | gradient$\times$input | All | 0.226 | 0.426 | 0.111
SciBERT | 100.0 | gradient$\times$input | Correct | 0.260 | 0.410 | 0.145
SciBERT | 0.0 | gradient$\times$input | Wrong | 0.178 | 0.450 | 0.062
SciBERT | 73.9 | $\lvert\text{gradient}\times\text{input}\rvert$ | All | 0.322 | 0.400 | 0.162
SciBERT | 100.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Correct | 0.356 | 0.387 | 0.189
SciBERT | 0.0 | $\lvert\text{gradient}\times\text{input}\rvert$ | Wrong | 0.275 | 0.418 | 0.123
Table 5: Results for all three models for deleted text edits across all
methods of selecting rationales (attention, gradient$\times$input and
$\lvert\text{gradient}\times\text{input}\rvert$) and all metrics to evaluate
rationales (mean reciprocal ranking, AUPRC and mean top 1 match).


# A $U$-Spin Anomaly in Charm CP Violation

Stefan Schacht<EMAIL_ADDRESS>Department of Physics and
Astronomy, University of Manchester, Manchester M13 9PL, United Kingdom

###### Abstract

Recent LHCb data shows that the direct CP asymmetries of the decay modes
$D^{0}\rightarrow\pi^{+}\pi^{-}$ and $D^{0}\rightarrow K^{+}K^{-}$ have the
same sign, violating an improved $U$-spin limit sum rule in an unexpected way
at $2.1\sigma$. From the new data, we determine for the first time the
imaginary part of the CKM-subleading, $U$-spin breaking $\Delta U=1$
correction to the $U$-spin limit $\Delta U=0$ amplitude. The imaginary part of
the $\Delta U=0$ amplitude is determined by $\Delta a_{CP}^{\mathrm{dir}}$.
The corresponding strong phases are yet unknown and could be extracted in the
future from time-dependent measurements. Assuming $\mathcal{O}(1)$ strong
phases due to non-perturbative rescattering, we find the ratio of $U$-spin
breaking to $U$-spin limit contributions to the CKM-subleading amplitudes to
be $(173^{+85}_{-74})\%$. This highly exceeds the Standard Model (SM)
expectation of $\sim 30\%$ $U$-spin breaking, with a significance of
$1.95\sigma$. If this puzzle is confirmed with more data in the future, in the
SM it would imply the breakdown of the $U$-spin expansion in CKM-subleading
amplitudes of charm decays. The other solution are new physics models that
generate an additional $\Delta U=1$ operator, leaving the $U$-spin power
expansion intact. Examples for the latter option are an extended scalar sector
or flavorful $Z^{\prime}$ models.

##  I Introduction

After the discovery of CP violation in charm decays Aaij  _et al._ (2019),
recently, there have again been several important advances in the measurement
of mixing and CP violation in charm decays Serena Maccolini on behalf of the
LHCb Collaboration ; Nayak  _et al._ (2020); Aaij  _et al._ (2021a, b, c, d,
e, 2022), see Ref. Amhis  _et al._ (2022) for most recent world averages and
global fits. Also theoretically, charm CP violation obtains a lot of attention
right now Grossman and Schacht (2019a); Chala  _et al._ (2019); Li  _et al._
(2019); Soni (2019); Cheng and Chiang (2019); Buccella  _et al._ (2019); Dery
and Nir (2019); Calibbi  _et al._ (2020); Dery  _et al._ (2021); Bause  _et
al._ (2020); Kagan and Silvestrini (2021); Buras  _et al._ (2021); Acaroglu
and Blanke (2022); Schacht and Soni (2022); Gavrilova  _et al._ (2022), see
earlier Refs. Einhorn and Quigg (1975); Abbott  _et al._ (1980); Buras  _et
al._ (1986); Golden and Grinstein (1989); Buccella  _et al._ (1993); Falk  _et
al._ (2002); Brod  _et al._ (2012a); Bhattacharya  _et al._ (2012); Franco
_et al._ (2012); Hiller  _et al._ (2013); Nierste and Schacht (2017, 2015);
Müller  _et al._ (2015a, b); Grossman and Schacht (2019b); Buccella  _et al._
(1995); Grossman  _et al._ (2007); Artuso  _et al._ (2008); Khodjamirian and
Petrov (2017); Buccella  _et al._ (2013); Cheng and Chiang (2012); Feldmann
_et al._ (2012); Li  _et al._ (2012); Atwood and Soni (2013); Grossman and
Robinson (2013); Yu  _et al._ (2017); Brod  _et al._ (2012b); Pirtskhalava and
Uttayarat (2012); Altmannshofer  _et al._ (2012); Grossman  _et al._ (2012).
The most recent news is the first evidence of a non-vanishing CP asymmetry

$\displaystyle a_{CP}^{\mathrm{dir}}(f)$
$\displaystyle\equiv\frac{|\mathcal{A}(D^{0}\rightarrow
f)|^{2}-|\mathcal{A}(\overline{D}^{0}\rightarrow
f)|^{2}}{|\mathcal{A}(D^{0}\rightarrow
f)|^{2}+|\mathcal{A}(\overline{D}^{0}\rightarrow f)|^{2}}$ (1)

in a single decay Serena Maccolini on behalf of the LHCb Collaboration ,
namely $D^{0}\rightarrow\pi^{+}\pi^{-}$. The knowledge of both CP asymmetries
Serena Maccolini on behalf of the LHCb Collaboration

$\displaystyle a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow K^{+}K^{-})$
$\displaystyle=(7.7\pm 5.7)\cdot 10^{-4}\,,$ (2) $\displaystyle
a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})$ $\displaystyle=(23.2\pm
6.1)\cdot 10^{-4}$ (3)

gives important advantages compared to the combination

$\displaystyle\Delta a_{CP}^{\mathrm{dir}}$ $\displaystyle\equiv
a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow
K^{+}K^{-})-a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})$ (4)

only. The reason is that a separate measurement of both CP asymmetries allows
to test the $U$-spin expansion in the amplitude contributions which are
relatively suppressed by Cabibbo-Kobayashi-Maskawa (CKM) matrix elements
compared to the leading singly-Cabibbo-suppressed (SCS) amplitude. In fact, we
can now probe the $U$-spin limit sum rule for the sum of CP asymmetries
Grossman  _et al._ (2007); Pirtskhalava and Uttayarat (2012); Hiller  _et al._
(2013); Grossman and Robinson (2013)

$\displaystyle\Sigma a_{CP}^{\mathrm{dir}}$ $\displaystyle\equiv
a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow
K^{+}K^{-})+a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})\overset{\text{$U$-spin
limit}}{=}0\,,$ (5)

which is violated at $2.7\sigma$ Serena Maccolini on behalf of the LHCb
Collaboration . Remarkably, Eq. (5) predicts that
$a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow K^{+}K^{-})$ and
$a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})$ have opposite signs,
but in fact the measurement shows that they have the same sign.

An improved version of the sum rule Eq. (5) is given as Grossman  _et al._
(2007); Pirtskhalava and Uttayarat (2012); Hiller  _et al._ (2013); Grossman
and Robinson (2013)

$\displaystyle\frac{\Gamma(D^{0}\rightarrow
K^{+}K^{-})}{\Gamma(D^{0}\rightarrow\pi^{+}\pi^{-})}$
$\displaystyle\overset{\text{$U$-spin
limit}}{=}-\frac{a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})}{a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow
K^{+}K^{-})}\,.$ (6)

The sum rules Eqs. (5, 6) belong to a category of $U$-spin sum rules which are
based on the complete interchange of $s$ and $d$ quarks Gronau (2000);
Fleischer (1999); Gronau and Rosner (2000). Inserting the experimental
measurements listed in Table 1 below, we obtain

$\displaystyle\frac{\Gamma(D^{0}\rightarrow
K^{+}K^{-})}{\Gamma(D^{0}\rightarrow\pi^{+}\pi^{-})}$ $\displaystyle=2.81\pm
0.06$ (7)

and

$\displaystyle-\frac{a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})}{a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow
K^{+}K^{-})}$ $\displaystyle=-3.01^{+0.95}_{-5.95}\,,$ (8)

_i.e._ altogether

$\displaystyle-\frac{\Gamma(D^{0}\rightarrow
K^{+}K^{-})}{\Gamma(D^{0}\rightarrow\pi^{+}\pi^{-})}\frac{a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow
K^{+}K^{-})}{a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})}$
$\displaystyle=-0.93^{+0.62}_{-0.41}\neq+1.$ (9)

The improved $U$-spin limit sum rule Eq. (6) is broken at $2.1\sigma$, because
Eq. (9) has the “wrong” sign. While $U$-spin breaking is expected, because
$U$-spin is only an approximate symmetry of QCD, the amount of breaking goes
beyond the Standard Model expectations of $\varepsilon\sim
m_{s}/\Lambda_{\mathrm{QCD}}\sim 30\%$ at $1.9\sigma$.

In this article, we analyze the implications of the new charm CP measurements
in more detail, extracting the CKM-subleading $\Delta U=1$ contributions to
the amplitudes of $D^{0}\rightarrow K^{+}K^{-}$ and
$D^{0}\rightarrow\pi^{+}\pi^{-}$ decays. In the SM these are generated from
the tensor product of the $U$-spin limit $\Delta U=0$ operator with the
$U$-spin breaking triplet operator Jung and Mannel (2009); Gavrilova  _et al._
(2022).

After briefly reviewing the application of SU(3)F methods in charm decays in
Sec. II, we summarize our notation in Sec. III. In Sec. IV we recapitulate how
to completely solve the system of two-body $D^{0}$ decays to kaons and pions.
We also show explicitly how to extract in principle the strong phases of the
CKM-subleading $\Delta U=1$ and $\Delta U=0$ hadronic matrix elements from
time-dependent CP violation. In Sec. V we present our numerical results.
Finally, in Sec. VI we give predictions and options for interpretations in
terms of new physics models that can be tested with future and more precise
data. We conclude in Sec. VII.

##  II Review of SU(3)F-breaking in Charm Decays

The application of SU(3)F methods in particle physics have their roots in
spectroscopy, namely the “eightfold way” for the description of the spectrum
of the meson and baryon octets Gell-Mann (1961); Ne’eman (1961). In
spectroscopy, SU(3)F has proven to be an extremely useful ordering principle.
For example, SU(3)F-limit predictions agree with the baryon octet mass
splitting with an accuracy of $10\%$ Greiner and Muller (1989). Furthermore,
the Gell-Mann–Okubo mass formula Gell-Mann (1961); Okubo (1962) demonstrated
that by including SU(3)F-breaking effects in a systematic way the precision of
predictions can be significantly improved. We know therefore that SU(3)F is a
very trustable technique for the particle spectrum. The question is if the
same applies also to decay rates, in particular for charm decays.

The nominal size of SU(3)F-breaking for decay amplitudes can be estimated from
the ratio of the decay constants Aoki  _et al._ (2022)

$\displaystyle\frac{f_{K}}{f_{\pi}}-1\sim 0.2\,.$ (10)

Now, two important examples where it looks naively as if $U$-spin is broken by
$\mathcal{O}(1)$ are given in terms of the ratios

$\displaystyle\frac{\mathcal{B}(D^{0}\rightarrow
K^{+}K^{-})}{\mathcal{B}(D^{0}\rightarrow\pi^{+}\pi^{-})}$ $\displaystyle\sim
3\,,$ (11) $\displaystyle\frac{\mathcal{B}(D^{0}\rightarrow
K_{S}K_{S})}{\mathcal{B}(D^{0}\rightarrow K^{+}K^{-})}$ $\displaystyle\sim
0.03\,.$ (12)

In the strict SU(3)F limit, neglecting also differences from phase space
effects, we have:

$\displaystyle\frac{\mathcal{B}(D^{0}\rightarrow
K^{+}K^{-})}{\mathcal{B}(D^{0}\rightarrow\pi^{+}\pi^{-})}$
$\displaystyle=1\,,$ (13) $\displaystyle\frac{\mathcal{B}(D^{0}\rightarrow
K_{S}K_{S})}{\mathcal{B}(D^{0}\rightarrow K^{+}K^{-})}$ $\displaystyle=0\,,$
(14)

in clear contradiction with the experimental measurements Eqs. (11) and (12).
However, already in Ref. Savage (1991) it was realized that Eq. (11) can
actually be explained by $\varepsilon\sim 30\%$ SU(3)F-breaking on the
amplitude. This can be understood as follows: Already for $\varepsilon\sim
30\%$, very roughly the ratio of branching ratios can be estimated as

$\displaystyle\frac{\left(1+\varepsilon\right)^{2}}{\left(1-\varepsilon\right)^{2}}$
$\displaystyle\sim 3\,.$ (15)

That means, Eq. (11) can be consistently explained with SU(3)F breaking of
$\sim 30\%$ on the amplitude. Note that the rough illustration Eq. (15) also
demonstrates that higher order contributions may be important, as at linear
order the left-hand side of Eq. (15) results in $\sim 2$, and only at
$\mathcal{O}(\varepsilon^{2})$ it reaches $\sim 3$. Below, we extract first
and second order $U$-spin breaking from branching ratio data, see Eqs.
(30)–(33) and Table 3. For the CKM-leading amplitudes, which dominate the
branching ratios, our results support that $\varepsilon\sim 0.3$ and
$\varepsilon^{2}\sim 0.1$, consistent with the $U$-spin power counting.

Coming now to the second example, as $\mathcal{B}(D\rightarrow K_{S}K_{S})$
vanishes in the SU(3)F limit, we can estimate the corresponding amplitude-
level SU(3)F breaking roughly as

$\displaystyle\varepsilon^{\prime}$
$\displaystyle\sim\sqrt{\frac{\mathcal{B}(D^{0}\rightarrow
K^{0}\overline{K}^{0})}{\mathcal{B}(D^{0}\rightarrow
K^{+}K^{-})}}=\sqrt{\frac{2\mathcal{B}(D^{0}\rightarrow
K_{S}K_{S})}{\mathcal{B}(D^{0}\rightarrow K^{+}K^{-})}}\sim 0.26\,,$ (16)

again consistent with the nominal size of SU(3)F breaking. Here we use the
experimental values Workman (2022)

$\displaystyle\mathcal{B}(D^{0}\rightarrow K_{S}K_{S})$
$\displaystyle=(1.41\pm 0.05)\cdot 10^{-4}\,,$ (17)
$\displaystyle\mathcal{B}(D^{0}\rightarrow K^{+}K^{-})$
$\displaystyle=(4.08\pm 0.06)\cdot 10^{-3}\,,$ (18)

and, due to Bose symmetry, see e.g. Ref. Nierste and Schacht (2015)

$\displaystyle\mathcal{A}(D^{0}\rightarrow K_{S}K_{S})$
$\displaystyle=-\frac{1}{\sqrt{2}}\mathcal{A}(D^{0}\rightarrow\overline{K}^{0}K^{0})\,.$
(19)

It follows that Eqs. (11, 12) can not be used as an argument that SU(3)F is
broken at $\mathcal{O}(1)$ for charm decays.

We note that if one would adopt additional theory assumptions in terms of a
$1/N_{c}$ power counting ’t Hooft (1974); Buras  _et al._ (1986) on top of the
SU(3)F expansion, in the $1/N_{c}$ limit one can factorize the tree amplitude
of non-leptonic charm decays, see e.g. Ref. Müller  _et al._ (2015a). However,
in this case the factorizable $U$-spin breaking of the tree amplitudes alone
does not suffice in order to explain the SU(3)F breaking in Eq. (11) Cheng and
Chiang (2010). In the topological diagram approach, besides the tree amplitude
$T$, the branching ratios of $D^{0}\rightarrow K^{+}K^{-}$ and
$D^{0}\rightarrow\pi^{+}\pi^{-}$ depend also on exchange diagrams $E$ and
SU(3)F-breaking combinations of penguin contractions of the tree operator
$P_{\mathrm{break}}$, see the parametrizations in Refs. Brod  _et al._
(2012a); Müller  _et al._ (2015a). Therefore, under the assumption of a
$1/N_{c}$ power counting, in order to explain Eq. (11), additional
contributions to the SU(3)F-breaking have to come from these contributions. At
first glance this seems counterintuitive, as $E$ and $P_{\mathrm{break}}$ are
formally $1/N_{c}$ suppressed relative to $T$, which would also affect the
possible amount of SU(3)F breaking. However, there are two contributions to
these respective topological diagrams, which are both suppressed by $1/N_{c}$,
and which stem from the Hamiltonian Müller  _et al._ (2015a)

$\displaystyle H_{W}$ $\displaystyle\propto C_{1}Q_{1}+C_{2}Q_{2}\,,$ (20)

where

$\displaystyle C_{1}$ $\displaystyle\sim\mathcal{O}(1/N_{c})\,,$
$\displaystyle C_{2}$ $\displaystyle\sim\mathcal{O}(1)\,,$ (21)
$\displaystyle\left<PP^{\prime}\right|Q_{1}\left|D\right>$
$\displaystyle\sim\mathcal{O}(1)\,,$
$\displaystyle\left<PP^{\prime}\right|Q_{2}\left|D\right>$
$\displaystyle\sim\mathcal{O}(1/N_{c})\,.$ (22)

A priori, it is unclear how the two terms of order $\mathcal{O}(1/N_{c})$ from
Eq. (20) interfere. This depends on the assumptions one makes about the
respective matrix elements and can at this time not be determined from first
principles. The fit in Ref. Müller  _et al._ (2015a) shows the existence of a
solution that is compatible with $1/N_{c}$ counting, namely when both
contributions interfere constructively, see Fig. 3(c) therein. This leads then
to a large $E/T$ ratio, see also Refs. Bhattacharya and Rosner (2010); Cheng
and Chiang (2010); Brod  _et al._ (2012b).

The fit result can be understood already when considering the single decay
mode $\mathcal{B}(D\rightarrow K_{S}K_{S})$, which only depends on
SU(3)F-breaking exchange diagrams. Eq. (16) determines their relative size as
$\sim 0.26$. In case of a constructive interference of the matrix elements of
Eq. (20), together with the estimates Eq. (22), we obtain the rough estimate

$\displaystyle\varepsilon(|C_{1}|+|C_{2}/N_{c}|)\sim 0.24\,,$ (23)

where we use $C_{2}=1.2$ and $C_{1}=-0.4$ Müller  _et al._ (2015a). The
estimate Eq. (23) reproduces the measurement Eq. (16) up to 8%. At the same
time, the fit in Ref. Müller  _et al._ (2015a) finds that the large
SU(3)F-breaking exchange diagrams together with the broken penguin can also
explain Eq. (11).

Global fits in the pure group-theoretical approach Hiller  _et al._ (2013)
agree with the approach employing topological diagrams Müller  _et al._
(2015a) in that the maximal needed linear SU(3)F breaking in the CKM-leading
amplitudes is given as $\varepsilon\sim 30\%$.

We can test the SU(3)F expansion also beyond linear order breaking effects.
For the ratio

$\displaystyle R_{DPP}$
$\displaystyle\equiv\frac{|\mathcal{A}(D^{0}\rightarrow
K^{+}K^{-})/(V_{cs}V_{us})|+|\mathcal{A}(D^{0}\rightarrow\pi^{+}\pi^{-})/(V_{cd}V_{ud})|}{|\mathcal{A}(D^{0}\rightarrow
K^{+}\pi^{-})/(V_{cd}V_{us})|+|\mathcal{A}(D^{0}\rightarrow
K^{-}\pi^{+})/(V_{cs}V_{ud})|}-1$ (24)

the SU(3)F expansion predicts that it is proportional to second order
SU(3)F-breaking effects Brod  _et al._ (2012a); Grossman and Robinson (2013);
Falk  _et al._ (2002)

$\displaystyle R_{DPP}^{\mathrm{th}}$
$\displaystyle=\mathcal{O}(\varepsilon^{2})\,.$ (25)

The experimental branching ratio measurements give

$\displaystyle R_{DPP}^{\mathrm{exp}}$ $\displaystyle=0.046\pm 0.008\,,$ (26)

confirming the theory prediction Eq. (25). If $U$-spin breaking were
$\mathcal{O}(1)$, the second order $U$-spin breaking contributions that are
isolated in Eq. (26) would still be $\mathcal{O}(1)$. Instead, it is, as
expected, consistent with $\mathcal{O}(\varepsilon^{2})$.

Although we see many examples where the SU(3)F expansion, including $U$-spin,
is applied with great success, it is still an open question how trustable it
is in general. Therefore, we seek to test the validity of $U$-spin at every
possible opportunity.

Below, from recent data, we identify a new puzzle that appears in the CKM-
subleading amplitude contributions to charm decays, as opposed to the CKM-
leading contributions discussed above.

##  III Notation

We use the notation of Ref. Grossman and Schacht (2019a) which we shortly
summarize in this section. In the SM, the Hamiltonian of SCS charm decays has
the $U$-spin structure

$\displaystyle\mathcal{H}_{\mathrm{eff}}\sim\Sigma(1,0)-\frac{\lambda_{b}}{2}(0,0)\,,$
(27)

with $(i,j)=\mathcal{O}^{\Delta U=i}_{\Delta U_{3}=j}$ and the CKM matrix
element combinations

$\displaystyle\Sigma$
$\displaystyle\equiv\frac{V_{cs}^{*}V_{us}-V_{cd}^{*}V_{ud}}{2}\,,$
$\displaystyle-\frac{\lambda_{b}}{2}$
$\displaystyle\equiv-\frac{V_{cb}^{*}V_{ub}}{2}=\frac{V_{cs}^{*}V_{us}+V_{cd}^{*}V_{ud}}{2}\,.$
(28)

Amplitudes of SCS charm decays can then be written as

$\displaystyle\mathcal{A}$ $\displaystyle=\Sigma
A_{\Sigma}-\frac{\lambda_{b}}{2}A_{b}\,.$ (29)

We use the following parametrization of $U$-spin related two-body $D^{0}$
decays to kaons and pions Grossman and Schacht (2019a); Brod  _et al._ (2012a)

$\displaystyle\mathcal{A}(K\pi)$
$\displaystyle=\mathcal{A}(\overline{D}^{0}\rightarrow
K^{+}\pi^{-})=V_{cs}V_{ud}^{*}\left(t_{0}-\frac{1}{2}t_{1}\right)\,,$ (CF)
(30) $\displaystyle\mathcal{A}(\pi\pi)$
$\displaystyle=\mathcal{A}(\overline{D}^{0}\rightarrow\pi^{+}\pi^{-})=-\Sigma^{*}\left(t_{0}+s_{1}+\frac{1}{2}t_{2}\right)-\lambda_{b}^{*}\left(p_{0}-\frac{1}{2}p_{1}\right)\,,$
(SCS) (31) $\displaystyle\mathcal{A}(KK)$
$\displaystyle=\mathcal{A}(\overline{D}^{0}\rightarrow
K^{+}K^{-})=\Sigma^{*}\left(t_{0}-s_{1}+\frac{1}{2}t_{2}\right)-\lambda_{b}^{*}\left(p_{0}+\frac{1}{2}p_{1}\right)\,,$
(SCS) (32) $\displaystyle\mathcal{A}(\pi K)$
$\displaystyle=\mathcal{A}(\overline{D}^{0}\rightarrow\pi^{+}K^{-})=V_{cd}V_{us}^{*}\left(t_{0}+\frac{1}{2}t_{1}\right)\,,$
(DCS) (33)

where the decays are classified according to their suppression with
Wolfenstein-$\lambda$ as Cabibbo-favored (CF), SCS and doubly-Cabibbo
suppressed (DCS). The subscripts of the parameters in Eqs. (30)–(33) indicate
the corresponding order in the $U$-spin expansion. We employ the normalized
parameters

$\displaystyle\tilde{t}_{1}\equiv\frac{t_{1}}{t_{0}}\,,\qquad\tilde{t}_{2}\equiv\frac{t_{2}}{t_{0}}\,,\qquad\tilde{s}_{1}\equiv\frac{s_{1}}{t_{0}}\,,\qquad\tilde{p}_{0}\equiv\frac{p_{0}}{t_{0}}\,,\qquad\tilde{p}_{1}\equiv\frac{p_{1}}{t_{0}}\,.$
(34)

The amplitudes are normalized such that

$\displaystyle\mathcal{B}(D\rightarrow PP^{\prime})$
$\displaystyle=|\mathcal{A}|^{2}\cdot\mathcal{P}(D,P,P^{\prime})\,,$ (35)
$\displaystyle\mathcal{P}(D,P,P^{\prime})$
$\displaystyle=\frac{\tau_{D}}{16\pi
m_{D}^{3}}\sqrt{\left(m_{D}^{2}-(m_{P}-m_{P^{\prime}})^{2}\right)\left(m_{D}^{2}-(m_{P}+m_{P^{\prime}})^{2}\right)}\,,$
(36)

and Golden and Grinstein (1989); Nierste and Schacht (2017); Pirtskhalava and
Uttayarat (2012)

$\displaystyle a_{CP}^{\mathrm{dir}}$
$\displaystyle=\mathrm{Im}\left(\frac{\lambda_{b}}{\Sigma}\right)\mathrm{Im}\left(\frac{A_{b}}{A_{\Sigma}}\right)\,.$
(37)

Furthermore, we write the amplitudes without CKM factors as $A(f)$ for CF and
DCS decays and $A(f)\equiv A_{\Sigma}(f)$, $A_{b}(f)$ for SCS decays.

Following Ref. Grossman and Schacht (2019a) we also use the observable
combinations

$\displaystyle R_{K\pi}$ $\displaystyle\equiv\frac{|A(K\pi)|^{2}-|A(\pi
K)|^{2}}{|A(K\pi)|^{2}+|A(\pi K)|^{2}}\,,$ (38) $\displaystyle R_{KK,\pi\pi}$
$\displaystyle\equiv\frac{|A(KK)|^{2}-|A(\pi\pi)|^{2}}{|A(KK)|^{2}+|A(\pi\pi)|^{2}}\,,$
(39) $\displaystyle R_{KK,\pi\pi,K\pi}$
$\displaystyle\equiv\frac{|A(KK)|^{2}+|A(\pi\pi)|^{2}-|A(K\pi)|^{2}-|A(\pi
K)|^{2}}{|A(KK)|^{2}+|A(\pi\pi)|^{2}+|A(K\pi)|^{2}+|A(\pi K)|^{2}}\,.$ (40)

The strong phase between CF and DCS $D^{0}$ decays is defined as

$\displaystyle\delta_{K\pi}\equiv\mathrm{arg}\left(\frac{\mathcal{A}(D^{0}\rightarrow
K^{+}\pi^{-})}{\mathcal{A}(D^{0}\rightarrow K^{-}\pi^{+})}\right)\,.$ (41)

For convenience, we define the strong phases $\delta_{KK}$ and
$\delta_{\pi\pi}$ slightly different from Ref. Grossman and Schacht (2019a) as

$\displaystyle\delta_{KK}$
$\displaystyle\equiv\mathrm{arg}\left(\frac{A_{b}(KK)}{A_{\Sigma}(KK)}\right)\,,$
$\displaystyle\delta_{\pi\pi}$
$\displaystyle\equiv\mathrm{arg}\left(\frac{A_{b}(\pi\pi)}{A_{\Sigma}(\pi\pi)}\right)\,.$
(42)

##  IV Solving for underlying theory parameters

In the convention of Ref. Grossman and Schacht (2019a), the parametrization
Eqs. (30)–(33) has the following eight real parameters, not counting the
normalization $t_{0}$:

$\displaystyle\mathrm{Re}(\tilde{t}_{1}),\quad\mathrm{Im}(\tilde{t}_{1}),\quad\tilde{t}_{2},\quad\tilde{s}_{1},\quad\mathrm{Re}(\tilde{p}_{0}),\quad\mathrm{Im}(\tilde{p}_{0}),\quad\mathrm{Re}(\tilde{p}_{1}),\quad\mathrm{Im}(\tilde{p}_{1}).$
(43)

We can solve the complete system to order $\mathcal{O}(\varepsilon^{2})$ as
follows Grossman and Schacht (2019a)

$\displaystyle\mathrm{Re}(\tilde{t}_{1})$ $\displaystyle=-R_{K\pi}\,,$ (44)
$\displaystyle\mathrm{Im}(\tilde{t}_{1})$
$\displaystyle=-\tan\left(\delta_{K\pi}\right)\,,$ (45)
$\displaystyle\tilde{t}_{2}$
$\displaystyle=2R_{KK,\pi\pi,K\pi}-\frac{1}{4}R_{KK,\pi\pi}^{2}+\frac{1}{4}R_{K\pi}^{2}+\frac{1}{4}\tan^{2}(\delta_{K\pi})\,,$
(46) $\displaystyle\tilde{s}_{1}$ $\displaystyle=-\frac{1}{2}R_{KK,\pi\pi}\,,$
(47) $\displaystyle\mathrm{Im}(\tilde{p}_{0})$
$\displaystyle=\frac{1}{4\,\mathrm{Im}(\lambda_{b}/\Sigma)}\Delta
a_{CP}^{\mathrm{dir}}\,,$ (48) $\displaystyle\mathrm{Im}(\tilde{p}_{1})$
$\displaystyle=\frac{1}{2\,\mathrm{Im}(\lambda_{b}/\Sigma)}\left(\Sigma
a_{CP}^{\mathrm{dir}}+\frac{1}{2}R_{KK,\pi\pi}\Delta
a_{CP}^{\mathrm{dir}}\right)\,,$ (49)
$\displaystyle\mathrm{Re}(\tilde{p}_{0})$
$\displaystyle=\frac{1}{4}\left(\mathrm{Re}\left(\frac{A_{b}(D^{0}\rightarrow
K^{+}K^{-})}{A_{\Sigma}(D^{0}\rightarrow
K^{+}K^{-})}\right)-\mathrm{Re}\left(\frac{A_{b}(D^{0}\rightarrow\pi^{+}\pi^{-})}{A_{\Sigma}(D^{0}\rightarrow\pi^{+}\pi^{-})}\right)\right)\,,$
(50) $\displaystyle\mathrm{Re}(\tilde{p}_{1})$
$\displaystyle=\frac{1}{2}\left(\mathrm{Re}\left(\frac{A_{b}(D^{0}\rightarrow
K^{+}K^{-})}{A_{\Sigma}(D^{0}\rightarrow
K^{+}K^{-})}\right)+\mathrm{Re}\left(\frac{A_{b}(D^{0}\rightarrow\pi^{+}\pi^{-})}{A_{\Sigma}(D^{0}\rightarrow\pi^{+}\pi^{-})}\right)\right)$
$\displaystyle\quad+\frac{1}{4}R_{KK,\pi\pi}\left(\mathrm{Re}\left(\frac{A_{b}(D^{0}\rightarrow
K^{+}K^{-})}{A_{\Sigma}(D^{0}\rightarrow
K^{+}K^{-})}\right)-\mathrm{Re}\left(\frac{A_{b}(D^{0}\rightarrow\pi^{+}\pi^{-})}{A_{\Sigma}(D^{0}\rightarrow\pi^{+}\pi^{-})}\right)\right)\,.$
(51)

Note that $\tan\delta_{K\pi}\approx\delta_{K\pi}$. Furthermore, we have to
$\mathcal{O}(\varepsilon^{2})$:

$\displaystyle\frac{1/2\,\mathrm{Im}(\tilde{p}_{1})}{\mathrm{Im}(\tilde{p}_{0})}=\frac{\Sigma
a_{CP}^{\mathrm{dir}}}{\Delta
a_{CP}^{\mathrm{dir}}}+\frac{1}{2}R_{KK,\pi\pi}\,.$ (52)

The parameters $\mathrm{Re}(\tilde{p}_{0})$ and $\mathrm{Re}(\tilde{p}_{1})$
can be determined from time-dependent measurements. In the following, we write
the equations for $\mathrm{Re}(\tilde{p}_{0})$ and
$\mathrm{Re}(\tilde{p}_{1})$ in a more convenient form in terms of the phases
$\cot\delta_{KK}$ and $\cot\delta_{\pi\pi}$. These are related to the
parametrization Eqs. (30)–(33) as

$\displaystyle\cot\delta_{KK}$
$\displaystyle=\frac{\mathrm{Re}(A_{b}(KK)/A_{\Sigma}(KK))}{\mathrm{Im}(A_{b}(KK)/A_{\Sigma}(KK))}\,,$
$\displaystyle\cot\delta_{\pi\pi}$
$\displaystyle=\frac{\mathrm{Re}(A_{b}(\pi\pi)/A_{\Sigma}(\pi\pi))}{\mathrm{Im}(A_{b}(\pi\pi)/A_{\Sigma}(\pi\pi))}\,,$
(53)

and can be obtained from the subleading, non-universal contributions to the
time-dependent CP violation observable $\Delta Y_{f}$, where Aaij  _et al._
(2021c)

$\displaystyle A_{CP}(f,t)\approx a_{CP}^{\mathrm{dir}}+\Delta
Y_{f}\frac{t}{\tau_{D^{0}}}\,,$ (54)

and to very good precision Aaij  _et al._ (2021c)

$\displaystyle\Delta
Y_{f}=x\sin\phi-y\left(\left|\frac{q}{p}\right|-1\right)+ya_{CP}^{\mathrm{dir}}(f)\left(1+\frac{x}{y}\cot\delta_{f}\right)\,.$
(55)

Here, $x$, $y$, $\left|q/p\right|$ and $\phi$ are the parameters of
$D^{0}-\overline{D}^{0}$ mixing, see Refs. Aaij  _et al._ (2021c); Kagan and
Silvestrini (2021); Grossman  _et al._ (2009); Kagan and Sokoloff (2009) for
details. Rearranging Eq. (55), we extract $\cot\delta_{f}$ from $\Delta Y_{f}$
as

$\displaystyle\cot\delta_{f}$ $\displaystyle=\frac{y}{x}\left(\frac{\Delta
Y_{f}-x\sin\phi+y\left(\left|q/p\right|-1\right)}{ya_{CP}^{\mathrm{dir}}(f)}-1\right)\,.$
(56)

In terms of $\cot\delta_{KK}$, $\cot\delta_{\pi\pi}$, $\Delta
a_{CP}^{\mathrm{dir}}$ and $\Sigma a_{CP}^{\mathrm{dir}}$ we obtain the
following expressions for $\mathrm{Re}(\tilde{p}_{0})$ and
$\mathrm{Re}(\tilde{p}_{1})$ to order $\mathcal{O}(\varepsilon^{2})$:

$\displaystyle\mathrm{Re}(\tilde{p}_{0})$
$\displaystyle=\frac{1}{8\mathrm{Im}(\lambda_{b}/\Sigma)}\Delta
a_{CP}^{\mathrm{dir}}\left(\cot\delta_{KK}+\cot\delta_{\pi\pi}\right)\,,$ (57)
$\displaystyle\mathrm{Re}(\tilde{p}_{1})$
$\displaystyle=\frac{1}{8\mathrm{Im}(\lambda_{b}/\Sigma)}\Delta
a_{CP}^{\mathrm{dir}}\left(\cot\delta_{\pi\pi}(R_{KK,\pi\pi}-2)+\cot\delta_{KK}(R_{KK,\pi\pi}+2)\right)+$
$\displaystyle\qquad\frac{1}{4\mathrm{Im}(\lambda_{b}/\Sigma)}\Sigma
a_{CP}^{\mathrm{dir}}\left(\cot\delta_{KK}+\cot\delta_{\pi\pi}\right)\,.$ (58)

##  V Numerical Results

For the numerical determination of the hadronic matrix element parameters of
the parametrization Eqs. (30)–(33) we employ the experimental input data in
Tables 1, 2 and apply Eqs. (44)–(49) and (56)–(58). From the branching ratio
measurements we obtain the combinations

$\displaystyle R_{K\pi}$ $\displaystyle=-0.08\pm 0.01\,,$ (59) $\displaystyle
R_{KK,\pi\pi}$ $\displaystyle=0.532\pm 0.008\,,$ (60) $\displaystyle
R_{KK,\pi\pi,K\pi}$ $\displaystyle=0.083\pm 0.008\,.$ (61)

From time-dependent CP violation we obtain for the strong phases

$\displaystyle\cot\delta_{KK}$ $\displaystyle=-28^{+61}_{-126}\,,$
$\displaystyle\cot\delta_{\pi\pi}$ $\displaystyle=-28^{+30}_{-36}\,,$ (62)

_i.e._ , basically no constraint. This is understandable from the fact that
the phases only contribute to the subleading, final-state dependent
contributions of $\Delta Y_{f}$, and at the current precision $\Delta
Y_{K^{+}K^{-}}$ and $\Delta Y_{\pi^{+}\pi^{-}}$ do not yet show a significant
final-state dependence.

Note that in principle there is a further opportunity for constraining the
strong phases of $\tilde{p}_{0,1}$ by extracting $\cot\delta_{KK}$ and
$\cot\delta_{\pi\pi}$ from future precision determinations of the isolated
mixing parameters $y_{CP}^{KK}$ and $y_{CP}^{\pi\pi}$, where these phases also
appear in subleading, final-state dependent contributions, see Refs. Pajero
and Morello (2022); Kagan and Silvestrini (2021); Grossman  _et al._ (2009);
Kagan and Sokoloff (2009) for details. Recently, LHCb measured the
combinations $y_{CP}^{KK}-y_{CP}^{K\pi}$ and $y_{CP}^{\pi\pi}-y_{CP}^{K\pi}$
Aaij  _et al._ (2022). However, like $\Delta Y_{K^{+}K^{-}}$ and $\Delta
Y_{\pi^{+}\pi^{-}}$, they do not yet show a significant final-state
dependence.

Our results for all parameters in Eq. (34) are given in Table 3. As a result
of Eq. (62, we have basically no information on the real parts
$\mathrm{Re}(\tilde{p}_{0})$ and $\mathrm{Re}(\tilde{p}_{1})$. We will
therefore not include them in the discussion any further.

We make now the following assumption:

  * •

Due to non-perturbative rescattering Grossman and Schacht (2019a), the phases
of $\tilde{p}_{0}$ and $\tilde{p}_{1}$ are $\mathcal{O}(1)$, resulting in
$\left|\mathrm{Im}(\tilde{p}_{1})/\mathrm{Im}(\tilde{p}_{0})\right|\approx|\tilde{p}_{1}|/|\tilde{p}_{0}|$.

With future data on time-dependent CP violation this assumption can be tested
and improved. From Eq. (52), it follows then for the ratio of the magnitude of
the $U$-spin breaking contribution to the CKM-subleading amplitude
$A_{b}(\pi\pi)$ ($A_{b}(KK)$) to the corresponding $U$-spin limit
contribution:

$\displaystyle\frac{1/2\,|\tilde{p}_{1}|}{|\tilde{p}_{0}|}$
$\displaystyle\approx\left|\frac{1/2\,\mathrm{Im}(\tilde{p}_{1})}{\mathrm{Im}(\tilde{p}_{0})}\right|=1.73^{+0.85}_{-0.74}\,,$
(63)

which deviates at $1.95\sigma$ from the SM expectation of $\mathcal{O}(30\%)$.
Eq. (63) is our main result. We illustrate Eq. (63) and the dependence of Eq.
(52) on $\Sigma a_{CP}^{\mathrm{dir}}$ in Fig. 1.

The found $U$-spin breaking of $(173^{+85}_{-74})\%$ may lead to the question
if the $U$-spin power counting used for its extraction in Eqs. (44–49, 57, 58)
is actually still valid. Note that Eqs. (44–49, 57, 58) are all formally valid
at $\mathcal{O}(\varepsilon^{2})$. Now, if $\mathrm{Im}(\tilde{p}_{1})$ breaks
the power counting by being $\mathcal{O}(1)$ instead of
$\mathcal{O}(\varepsilon)$, these equations have the following power counting:

  * •

Eqs. (44–47, 49) for the extraction of $\mathrm{Re}(\tilde{t}_{1})$,
$\mathrm{Im}(\tilde{t}_{1})$, $\tilde{t}_{2}$, $\tilde{s}_{1}$, and
$\mathrm{Im}(\tilde{p}_{1})$ are in this case still valid at
$\mathcal{O}(\varepsilon^{2})$.

  * •

Eqs. (48, 58) for the extraction of $\mathrm{Im}(\tilde{p}_{0})$ and
$\mathrm{Re}(\tilde{p}_{1})$ are in this case valid at
$\mathcal{O}(\varepsilon)$.

  * •

Eq. (57) obtains $\mathcal{O}(1)$ corrections, _i.e._ is broken in this case
and can no longer be used for the extraction of $\mathrm{Re}(\tilde{p}_{0})$.

Note that also Eq. (52) is formally valid at $\mathcal{O}(\varepsilon^{2})$
and is still valid at $\mathcal{O}(\varepsilon)$ when
$\mathrm{Im}(\tilde{p}_{1})\sim\mathcal{O}(1)$. The above implies that for
$\mathrm{Im}(\tilde{p}_{1})\sim\mathcal{O}(1)$ the methodology of Sec. IV
still enables a consistent parameter extraction with the exception of the
parameter $\mathrm{Re}(\tilde{p}_{0})$. However, with current data we have in
any case no sensitivity to this parameter. As can be seen from Table 3, the
other $U$-spin breaking parameters are consistent with the $U$-spin power
counting.

Figure 1: Illustration of the dependence of
$1/2\,\mathrm{Im}(\tilde{p}_{1})/\mathrm{Im}(\tilde{p}_{0})$ on $\Sigma
a_{CP}^{\mathrm{dir}}$ according to Eq. (52). For this illustration we fix
$\Delta a_{CP}^{\mathrm{dir}}$ and $R_{KK,\pi\pi}$ to their central values and
vary $\Sigma a_{CP}^{\mathrm{dir}}$ away from its measured value (blue). In
red we show the current experimental data for $\Sigma a_{CP}^{\mathrm{dir}}$
and the resulting value for
$1/2\,\mathrm{Im}(\tilde{p}_{1})/\mathrm{Im}(\tilde{p}_{0})$ including
$1\sigma$ errors. For the estimate of the region of 30% $U$-spin breaking
(yellow) we assume that the strong phases are $\mathcal{O}(1)$, such that
$|1/2\,\mathrm{Im}(\tilde{p}_{1})/\mathrm{Im}(\tilde{p}_{0})|\approx
1/2|\tilde{p}_{1}|/|\tilde{p}_{0}|\leq 30\%$.  Direct CP Asymmetries
---
$a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow K^{+}K^{-})$ | $(7.7\pm 5.7)\cdot 10^{-4}$ | Serena Maccolini on behalf of the LHCb Collaboration
$a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})$ | $(23.2\pm 6.1)\cdot 10^{-4}$ | Serena Maccolini on behalf of the LHCb Collaboration
$\rho$ | 0.88 | Serena Maccolini on behalf of the LHCb Collaboration
Time-dependent CP Violation
$\Delta Y_{K^{+}K^{-}}$ | $(-2.3\pm 1.5\pm 0.3)\cdot 10^{-4}$ | Aaij  _et al._ (2021c)
$\Delta Y_{\pi^{+}\pi^{-}}$ | $(-4.0\pm 2.8\pm 0.4)\cdot 10^{-4}$ | Aaij  _et al._ (2021c)
$D^{0}-\overline{D}^{0}$ Mixing Parameters
$x$ | $\left(0.409_{-0.049}^{+0.048}\right)\cdot 10^{-2}$ | Amhis  _et al._ (2022)
$y$ | $\left(0.615^{+0.056}_{-0.055}\right)\cdot 10^{-2}$ | Amhis  _et al._ (2022)
$\delta_{K\pi}$ | $(7.2^{+7.9}_{-9.2})^{\circ}$ | Amhis  _et al._ (2022)
$|q/p|$ | $0.995\pm 0.016$ | Amhis  _et al._ (2022)
$\phi$ | $\left(-2.5\pm 1.2\right)^{\circ}$ | Amhis  _et al._ (2022)
Branching Ratios
$\mathcal{B}(D^{0}\rightarrow K^{+}K^{-})$ | $(4.08\pm 0.06)\cdot 10^{-3}$ | Workman (2022)
$\mathcal{B}(D^{0}\rightarrow\pi^{+}\pi^{-})$ | $(1.454\pm 0.024)\cdot 10^{-3}$ | Workman (2022)
$\mathcal{B}(D^{0}\rightarrow K^{+}\pi^{-})$ | $(1.363\pm 0.025)\cdot 10^{-4}$ | Workman (2022)
$\mathcal{B}(D^{0}\rightarrow K^{-}\pi^{+})$ | $(3.947\pm 0.030)\cdot 10^{-2}$ | Workman (2022)
Further Numerical Inputs
$\mathrm{Im}\left(\lambda_{b}/\Sigma\right)$ | $(-6.0\pm 0.3)\cdot 10^{-4}$ | Workman (2022)
Table 1: Experimental input data. We include the correlation between $a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow K^{+}K^{-})$ and $a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})$ which is given by $\rho$. We also include the correlations between the mixing parameters $x$, $y$, $\delta_{K\pi}$, $|q/p|$, and $\phi$, which are given in Table 2. We symmetrize all errors of the input data if applicable. Note that for $a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow K^{+}K^{-})$ and $a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})$ we use directly the most recent preliminary measurements by LHCb which contain all Run-1 and Run-2 measurements. Note further that the fit results by the Heavy Flavor Averaging Group (HFLAV) for the $D^{0}-\overline{D}^{0}$ mixing parameters Amhis  _et al._ (2022) do not yet include these latest results for the direct CP asymmetries. Both of these points can be improved in the future with updates of the world averages and global fits Amhis  _et al._ (2022); Aubert  _et al._ (2008); Aaltonen  _et al._ (2012); Aaij  _et al._ (2014, 2016); Staric  _et al._ (2008); Starič  _et al._ (2016).  | $x$ | $y$ | $\delta_{K\pi}$ | $|q/p|$ | $\phi$
---|---|---|---|---|---
$x$ | $1.0$ | $-0.075$ | $-0.029$ | $-0.122$ | $0.087$
$y$ | $-0.075$ | $1.0$ | $0.970$ | $-0.035$ | $0.071$
$\delta_{K\pi}$ | $-0.029$ | $0.970$ | $1.0$ | $-0.043$ | $0.079$
$|q/p|$ | $-0.122$ | $-0.035$ | $-0.043$ | $1.0$ | $0.558$
$\phi$ | $0.087$ | $0.071$ | $0.079$ | $0.558$ | $1.0$
Table 2: Correlation matrix for the needed $D^{0}-\overline{D}^{0}$ mixing parameters from Ref. Amhis  _et al._ (2022).  $\mathrm{Re}(\tilde{t}_{1})$ | $0.083\pm 0.010$
---|---
$\mathrm{Im}(\tilde{t}_{1})$ | $-0.11^{+0.15}_{-0.16}$
$\tilde{t}_{2}$ | $0.101^{+0.019}_{-0.016}$
$\tilde{s}_{1}$ | $-0.2658^{+0.0040}_{-0.0039}$
$\mathrm{Im}(\tilde{p}_{0})$ | $0.66\pm 0.13$
$\mathrm{Im}(\tilde{p}_{1})$ | $-2.27^{+0.96}_{-0.98}$
$\mathrm{Re}(\tilde{p}_{0})$ | $-18^{+23}_{-47}$
$\mathrm{Re}(\tilde{p}_{1})$ | $64^{+56}_{-55}$
Table 3: Results for hadronic matrix elements of the $U$-spin expansion Eqs.
(30)–(33), as extracted from the experimental data in Tables 1 and 2.

##  VI Predictions and New Physics Interpretations

The large $U$-spin breaking of $(173^{+85}_{-74})\%$ that we find in Eq. (63)
indicates large contributions from $\Delta U=1$ operators in the CKM-
subleading amplitude of SCS charm decays. This leads to an $\mathcal{O}(1)$
breaking of the $U$-spin limit sum rule Grossman  _et al._ (2007);
Pirtskhalava and Uttayarat (2012); Hiller  _et al._ (2013); Grossman and
Robinson (2013)

$\displaystyle\frac{\Gamma(D^{0}\rightarrow
K^{+}K^{-})}{\Gamma(D^{0}\rightarrow\pi^{+}\pi^{-})}$
$\displaystyle=-\frac{a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})}{a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow
K^{+}K^{-})}\,,$ (64)

see the discussion in Sec. I. As the decays $D^{0}\rightarrow K^{+}K^{-}$ and
$D^{0}\rightarrow\pi^{+}\pi^{-}$ are also connected to a wider class of decays
via SU(3)F symmetry, we expect that the $U$-spin limit sum rule Grossman  _et
al._ (2007); Pirtskhalava and Uttayarat (2012); Hiller  _et al._ (2013);
Grossman and Robinson (2013)

$\displaystyle\frac{\Gamma(D^{+}\rightarrow\overline{K}^{0}K^{+})}{\Gamma(D_{s}^{+}\rightarrow
K^{0}\pi^{+})}$
$\displaystyle=-\frac{a_{CP}^{\mathrm{dir}}(D_{s}^{+}\rightarrow
K^{0}\pi^{+})}{a_{CP}^{\mathrm{dir}}(D^{+}\rightarrow\overline{K}^{0}K^{+})}$
(65)

is also broken at $\mathcal{O}(1)$. In Ref. Müller  _et al._ (2015b) improved
versions of the sum rules Eqs. (64, 65) are formulated that account for the
first order SU(3)F breaking effects from all topological diagrams except for
the penguin contraction of tree operators ($P$ and $PA$ in the notation
therein). Therefore, we predict that also the sum rules Müller  _et al._
(2015b)

$\displaystyle\frac{\mathcal{S}(D^{0}\rightarrow
K^{+}K^{-})-\mathcal{S}(D^{0}\rightarrow\pi^{+}\pi^{-})}{e^{2i\delta(D^{0}\rightarrow
K^{+}K^{-})}-e^{2i\delta(D^{0}\rightarrow\pi^{+}\pi^{-})}}-\frac{\mathcal{S}(D^{0}\rightarrow
K^{+}K^{-})+\sqrt{2}\mathcal{S}(D^{0}\rightarrow\pi^{0}\pi^{0})}{e^{2i\delta(D^{0}\rightarrow
K^{+}K^{-})}-e^{2i\delta(D^{0}\rightarrow\pi^{0}\pi^{0})}}$
$\displaystyle=0\,,$ (66)
$\displaystyle\frac{\mathcal{S}(D^{+}\rightarrow\overline{K}^{0}K^{+})-\mathcal{S}(D_{s}^{+}\rightarrow
K^{0}\pi^{+})}{e^{2i\delta(D^{+}\rightarrow\overline{K}^{0}K^{+})}-e^{2i\delta(D_{s}^{+}\rightarrow
K^{0}\pi^{+})}}-\frac{\mathcal{S}(D^{+}\rightarrow\overline{K}^{0}K^{+})+\sqrt{2}\mathcal{S}(D_{s}^{+}\rightarrow
K^{+}\pi^{0})}{e^{2i\delta(D^{+}\rightarrow\overline{K}^{0}K^{+})}-e^{2i\delta(D_{s}^{+}\rightarrow
K^{+}\pi^{0})}}$ $\displaystyle=0$ (67)

are broken at $\mathcal{O}(1)$. Here,
$\delta(d)\equiv\mathrm{arg}(A_{\Sigma}(d))$ and the function $\mathcal{S}(d)$
can be found in Ref. Müller  _et al._ (2015b). Further SU(3)F sum rules are
given in Refs. Grossman and Robinson (2013); Grossman  _et al._ (2014). For a
general treatment of $U$-spin sum rules at any order see Ref. Gavrilova  _et
al._ (2022). In light of the puzzle posed by the $U$-spin expansion of charm
decays, also a further test of the respective isospin structure is very
important Grossman  _et al._ (2012).

As laid out in Ref. Hiller  _et al._ (2013), new physics models with
additional $\Delta U=1$ operators, so-called “$\Delta U=1$ models” Hiller  _et
al._ (2013) can explain the breaking of Eqs. (64, 65) beyond the $U$-spin
power counting. The same applies to Eqs. (66, 67). Such models generate
additional effective operators with the flavor content
$\overline{s}c\overline{u}s$ and/or $\overline{d}c\overline{u}d$ with non-
universal coefficients. They can _e.g._ arise from two-Higgs-doublet models
(2HDMs) Altmannshofer  _et al._ (2012) or flavorful $Z^{\prime}$ models
Altmannshofer  _et al._ (2012); Chala  _et al._ (2019); Bause  _et al._
(2020). Recently, in Ref. Bause  _et al._ (2020) it has been shown that
$Z^{\prime}$ models can induce large $U$-spin breaking between
$a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow K^{+}K^{-})$ and
$a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})$, depending on the
charge assignments of the quarks under an additional $U(1)^{\prime}$ group.
With the new data, charm CP asymmetries can be used effectively to probe and
explore the parameter space of such models further. For example, the specific
charge assignments of $Z^{\prime}$ models considered in Ref. Bause  _et al._
(2020) lead to opposite signs for $a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow
K^{+}K^{-})$ and $a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})$, see
Fig. 3 therein, whereas the most recent data indicates
$a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow K^{+}K^{-})>0$ and
$a_{CP}^{\mathrm{dir}}(D^{0}\rightarrow\pi^{+}\pi^{-})>0$.

The exploration of the $U$-spin puzzle with future and more precise
measurements including sum rule tests is important for a complete
understanding of the CKM-subleading amplitudes of SCS charm decays and in
order to further probe the parameter space of $\Delta U=1$ models.

##  VII Conclusions

Assuming the Standard Model, from recent measurements of charm CP violation in
the single decay channels $D^{0}\rightarrow K^{+}K^{-}$ and
$D^{0}\rightarrow\pi^{+}\pi^{-}$ we extract for the first time the imaginary
part $\mathrm{Im}(\tilde{p}_{1})$ of the $U$-spin breaking $\Delta U=1$
contribution to the CKM-subleading amplitudes. We obtain

$\displaystyle\frac{1/2\,\mathrm{Im}(\tilde{p}_{1})}{\mathrm{Im}(\tilde{p}_{0})}$
$\displaystyle=(-173^{+74}_{-85})\%\,,$ (68)

where $\mathrm{Im}(\tilde{p}_{0})$ is the $U$-spin limit $\Delta U=0$
contribution to the CKM-subleading amplitudes which is determined by $\Delta
a_{CP}^{\mathrm{dir}}$. The strong phases of $\tilde{p}_{0,1}$ are yet
unknown. Assuming $\mathcal{O}(1)$ strong phases due to non-perturbative
rescattering, the result implies very large $U$-spin breaking, which exceeds
the SM expectation of $\sim 30\%$ by almost a factor six, at $1.95\sigma$.

It is crucial to probe this anomaly further with more data and test the
$U$-spin expansion also in additional decays, using the sum rules listed in
Sec. VI. Most importantly, we need improved time-dependent measurements, such
that we can extract the strong phases of $\tilde{p}_{0,1}$ from data. In order
to test the pattern of the SU(3)F expansion, measurements of CP asymmetries of
basically all singly-Cabibbo suppressed decays are necessary.

We encourage experimental collaborations to extract the underlying theory
parameters using the methodology described in Sec. IV directly from the data,
enabling the most comprehensive treatment of all correlations.

If the $U$-spin anomaly is confirmed with more data in the future, this would
imply either a breakdown of the $U$-spin expansion in the Standard Model, or a
sign for new physics with an additional $\Delta U=1$ operator, for example
from additional scalar particles or a flavorful $Z^{\prime}$.

###### Acknowledgements.

We thank Laurent Dufour, Marco Gersabeck, Yuval Grossman, Ulrich Nierste,
Guillaume Pietrzyk and Alan Schwartz for useful discussions. S.S. is supported
by a Stephen Hawking Fellowship from UKRI under reference EP/T01623X/1 and the
Lancaster-Manchester-Sheffield Consortium for Fundamental Physics, under STFC
research grant ST/T001038/1. For the purpose of open access, the author has
applied a Creative Commons Attribution (CC BY) licence to any Author Accepted
Manuscript version arising. This work uses existing data which is available at
locations cited in the bibliography.

## References

  * Aaij  _et al._ (2019) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 122, 211803 (2019), arXiv:1903.08726 [hep-ex] .
  * (2) Serena Maccolini on behalf of the LHCb Collaboration, Talk at the “XLI International Conference on High Energy Physics” (ICHEP 2022), 6–13 July 2022, Bologna, Italy, https://agenda.infn.it/event/28874/contributions/169315/.
  * Nayak  _et al._ (2020) M. Nayak _et al._ (Belle), Phys. Rev. D 102, 071102 (2020), arXiv:1912.10912 [hep-ex] .
  * Aaij  _et al._ (2021a) R. Aaij _et al._ (LHCb), JHEP 06, 019 (2021a), arXiv:2103.11058 [hep-ex] .
  * Aaij  _et al._ (2021b) R. Aaij _et al._ (LHCb), Phys. Rev. D 104, L031102 (2021b), arXiv:2105.01565 [hep-ex] .
  * Aaij  _et al._ (2021c) R. Aaij _et al._ (LHCb), Phys. Rev. D 104, 072010 (2021c), arXiv:2105.09889 [hep-ex] .
  * Aaij  _et al._ (2021d) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 127, 111801 (2021d), arXiv:2106.03744 [hep-ex] .
  * Aaij  _et al._ (2021e) R. Aaij _et al._ (LHCb), JHEP 12, 141 (2021e), arXiv:2110.02350 [hep-ex] .
  * Aaij  _et al._ (2022) R. Aaij _et al._ (LHCb), Phys. Rev. D 105, 092013 (2022), arXiv:2202.09106 [hep-ex] .
  * Amhis  _et al._ (2022) Y. Amhis _et al._ (HFLAV), (2022), and online update Dec 2021, arXiv:2206.07501 [hep-ex] .
  * Grossman and Schacht (2019a) Y. Grossman and S. Schacht, JHEP 07, 020 (2019a), arXiv:1903.10952 [hep-ph] .
  * Chala  _et al._ (2019) M. Chala, A. Lenz, A. V. Rusov, and J. Scholtz, JHEP 07, 161 (2019), arXiv:1903.10490 [hep-ph] .
  * Li  _et al._ (2019) H.-N. Li, C.-D. Lü, and F.-S. Yu, (2019), arXiv:1903.10638 [hep-ph] .
  * Soni (2019) A. Soni, (2019), arXiv:1905.00907 [hep-ph] .
  * Cheng and Chiang (2019) H.-Y. Cheng and C.-W. Chiang, Phys. Rev. D 100, 093002 (2019), arXiv:1909.03063 [hep-ph] .
  * Buccella  _et al._ (2019) F. Buccella, A. Paul, and P. Santorelli, Phys. Rev. D 99, 113001 (2019), arXiv:1902.05564 [hep-ph] .
  * Dery and Nir (2019) A. Dery and Y. Nir, JHEP 12, 104 (2019), arXiv:1909.11242 [hep-ph] .
  * Calibbi  _et al._ (2020) L. Calibbi, T. Li, Y. Li, and B. Zhu, JHEP 10, 070 (2020), arXiv:1912.02676 [hep-ph] .
  * Dery  _et al._ (2021) A. Dery, Y. Grossman, S. Schacht, and A. Soffer, JHEP 05, 179 (2021), arXiv:2101.02560 [hep-ph] .
  * Bause  _et al._ (2020) R. Bause, H. Gisbert, M. Golz, and G. Hiller, Phys. Rev. D 101, 115006 (2020), arXiv:2004.01206 [hep-ph] .
  * Kagan and Silvestrini (2021) A. L. Kagan and L. Silvestrini, Phys. Rev. D 103, 053008 (2021), arXiv:2001.07207 [hep-ph] .
  * Buras  _et al._ (2021) A. J. Buras, P. Colangelo, F. De Fazio, and F. Loparco, JHEP 10, 021 (2021), arXiv:2107.10866 [hep-ph] .
  * Acaroglu and Blanke (2022) H. Acaroglu and M. Blanke, JHEP 05, 086 (2022), arXiv:2109.10357 [hep-ph] .
  * Schacht and Soni (2022) S. Schacht and A. Soni, Phys. Lett. B 825, 136855 (2022), arXiv:2110.07619 [hep-ph] .
  * Gavrilova  _et al._ (2022) M. Gavrilova, Y. Grossman, and S. Schacht, (2022), arXiv:2205.12975 [hep-ph] .
  * Einhorn and Quigg (1975) M. B. Einhorn and C. Quigg, Phys. Rev. D 12, 2015 (1975).
  * Abbott  _et al._ (1980) L. F. Abbott, P. Sikivie, and M. B. Wise, Phys. Rev. D 21, 768 (1980).
  * Buras  _et al._ (1986) A. J. Buras, J. M. Gerard, and R. Ruckl, Nucl. Phys. B 268, 16 (1986).
  * Golden and Grinstein (1989) M. Golden and B. Grinstein, Phys. Lett. B 222, 501 (1989).
  * Buccella  _et al._ (1993) F. Buccella, M. Lusignoli, G. Mangano, G. Miele, A. Pugliese, and P. Santorelli, Phys. Lett. B 302, 319 (1993), arXiv:hep-ph/9212253 .
  * Falk  _et al._ (2002) A. F. Falk, Y. Grossman, Z. Ligeti, and A. A. Petrov, Phys. Rev. D 65, 054034 (2002), arXiv:hep-ph/0110317 .
  * Brod  _et al._ (2012a) J. Brod, Y. Grossman, A. L. Kagan, and J. Zupan, JHEP 10, 161 (2012a), arXiv:1203.6659 [hep-ph] .
  * Bhattacharya  _et al._ (2012) B. Bhattacharya, M. Gronau, and J. L. Rosner, Phys. Rev. D 85, 054014 (2012), arXiv:1201.2351 [hep-ph] .
  * Franco  _et al._ (2012) E. Franco, S. Mishima, and L. Silvestrini, JHEP 05, 140 (2012), arXiv:1203.3131 [hep-ph] .
  * Hiller  _et al._ (2013) G. Hiller, M. Jung, and S. Schacht, Phys. Rev. D 87, 014024 (2013), arXiv:1211.3734 [hep-ph] .
  * Nierste and Schacht (2017) U. Nierste and S. Schacht, Phys. Rev. Lett. 119, 251801 (2017), arXiv:1708.03572 [hep-ph] .
  * Nierste and Schacht (2015) U. Nierste and S. Schacht, Phys. Rev. D 92, 054036 (2015), arXiv:1508.00074 [hep-ph] .
  * Müller  _et al._ (2015a) S. Müller, U. Nierste, and S. Schacht, Phys. Rev. D 92, 014004 (2015a), arXiv:1503.06759 [hep-ph] .
  * Müller  _et al._ (2015b) S. Müller, U. Nierste, and S. Schacht, Phys. Rev. Lett. 115, 251802 (2015b), arXiv:1506.04121 [hep-ph] .
  * Grossman and Schacht (2019b) Y. Grossman and S. Schacht, Phys. Rev. D 99, 033005 (2019b), arXiv:1811.11188 [hep-ph] .
  * Buccella  _et al._ (1995) F. Buccella, M. Lusignoli, G. Miele, A. Pugliese, and P. Santorelli, Phys. Rev. D 51, 3478 (1995), arXiv:hep-ph/9411286 .
  * Grossman  _et al._ (2007) Y. Grossman, A. L. Kagan, and Y. Nir, Phys. Rev. D 75, 036008 (2007), arXiv:hep-ph/0609178 .
  * Artuso  _et al._ (2008) M. Artuso, B. Meadows, and A. A. Petrov, Ann. Rev. Nucl. Part. Sci. 58, 249 (2008), arXiv:0802.2934 [hep-ph] .
  * Khodjamirian and Petrov (2017) A. Khodjamirian and A. A. Petrov, Phys. Lett. B 774, 235 (2017), arXiv:1706.07780 [hep-ph] .
  * Buccella  _et al._ (2013) F. Buccella, M. Lusignoli, A. Pugliese, and P. Santorelli, Phys. Rev. D 88, 074011 (2013), arXiv:1305.7343 [hep-ph] .
  * Cheng and Chiang (2012) H.-Y. Cheng and C.-W. Chiang, Phys. Rev. D 85, 034036 (2012), [Erratum: Phys.Rev.D 85, 079903 (2012)], arXiv:1201.0785 [hep-ph] .
  * Feldmann  _et al._ (2012) T. Feldmann, S. Nandi, and A. Soni, JHEP 06, 007 (2012), arXiv:1202.3795 [hep-ph] .
  * Li  _et al._ (2012) H.-n. Li, C.-D. Lu, and F.-S. Yu, Phys. Rev. D 86, 036012 (2012), arXiv:1203.3120 [hep-ph] .
  * Atwood and Soni (2013) D. Atwood and A. Soni, PTEP 2013, 093B05 (2013), arXiv:1211.1026 [hep-ph] .
  * Grossman and Robinson (2013) Y. Grossman and D. J. Robinson, JHEP 04, 067 (2013), arXiv:1211.3361 [hep-ph] .
  * Yu  _et al._ (2017) F.-S. Yu, D. Wang, and H.-n. Li, Phys. Rev. Lett. 119, 181802 (2017), arXiv:1707.09297 [hep-ph] .
  * Brod  _et al._ (2012b) J. Brod, A. L. Kagan, and J. Zupan, Phys. Rev. D 86, 014023 (2012b), arXiv:1111.5000 [hep-ph] .
  * Pirtskhalava and Uttayarat (2012) D. Pirtskhalava and P. Uttayarat, Phys. Lett. B 712, 81 (2012), arXiv:1112.5451 [hep-ph] .
  * Altmannshofer  _et al._ (2012) W. Altmannshofer, R. Primulando, C.-T. Yu, and F. Yu, JHEP 04, 049 (2012), arXiv:1202.2866 [hep-ph] .
  * Grossman  _et al._ (2012) Y. Grossman, A. L. Kagan, and J. Zupan, Phys. Rev. D 85, 114036 (2012), arXiv:1204.3557 [hep-ph] .
  * Gronau (2000) M. Gronau, Phys. Lett. B 492, 297 (2000), arXiv:hep-ph/0008292 .
  * Fleischer (1999) R. Fleischer, Phys. Lett. B 459, 306 (1999), arXiv:hep-ph/9903456 .
  * Gronau and Rosner (2000) M. Gronau and J. L. Rosner, Phys. Lett. B 482, 71 (2000), arXiv:hep-ph/0003119 .
  * Jung and Mannel (2009) M. Jung and T. Mannel, Phys. Rev. D 80, 116002 (2009), arXiv:0907.0117 [hep-ph] .
  * Gell-Mann (1961) M. Gell-Mann,  (1961), 10.2172/4008239.
  * Ne’eman (1961) Y. Ne’eman, Nucl. Phys. 26, 222 (1961).
  * Greiner and Muller (1989) W. Greiner and B. Muller,  _Theoretical physics. Vol. 2: Quantum mechanics. Symmetries_ (1989).
  * Okubo (1962) S. Okubo, Prog. Theor. Phys. 27, 949 (1962).
  * Aoki  _et al._ (2022) Y. Aoki _et al._ (Flavour Lattice Averaging Group (FLAG)), Eur. Phys. J. C 82, 869 (2022), arXiv:2111.09849 [hep-lat] .
  * Savage (1991) M. J. Savage, Phys. Lett. B 257, 414 (1991).
  * Workman (2022) R. L. Workman (Particle Data Group), PTEP 2022, 083C01 (2022).
  * ’t Hooft (1974) G. ’t Hooft, Nucl. Phys. B 72, 461 (1974).
  * Cheng and Chiang (2010) H.-Y. Cheng and C.-W. Chiang, Phys. Rev. D 81, 074021 (2010), arXiv:1001.0987 [hep-ph] .
  * Bhattacharya and Rosner (2010) B. Bhattacharya and J. L. Rosner, Phys. Rev. D 81, 014026 (2010), arXiv:0911.2812 [hep-ph] .
  * Grossman  _et al._ (2009) Y. Grossman, Y. Nir, and G. Perez, Phys. Rev. Lett. 103, 071602 (2009), arXiv:0904.0305 [hep-ph] .
  * Kagan and Sokoloff (2009) A. L. Kagan and M. D. Sokoloff, Phys. Rev. D 80, 076008 (2009), arXiv:0907.3917 [hep-ph] .
  * Pajero and Morello (2022) T. Pajero and M. J. Morello, JHEP 03, 162 (2022), arXiv:2106.02014 [hep-ph] .
  * Aubert  _et al._ (2008) B. Aubert _et al._ (BaBar), Phys. Rev. Lett. 100, 061803 (2008), arXiv:0709.2715 [hep-ex] .
  * Aaltonen  _et al._ (2012) T. Aaltonen _et al._ (CDF), Phys. Rev. Lett. 109, 111801 (2012), arXiv:1207.2158 [hep-ex] .
  * Aaij  _et al._ (2014) R. Aaij _et al._ (LHCb), JHEP 07, 041 (2014), arXiv:1405.2797 [hep-ex] .
  * Aaij  _et al._ (2016) R. Aaij _et al._ (LHCb), Phys. Rev. Lett. 116, 191601 (2016), arXiv:1602.03160 [hep-ex] .
  * Staric  _et al._ (2008) M. Staric _et al._ (Belle), Phys. Lett. B 670, 190 (2008), arXiv:0807.0148 [hep-ex] .
  * Starič  _et al._ (2016) M. Starič _et al._ (Belle), Phys. Lett. B 753, 412 (2016), arXiv:1509.08266 [hep-ex] .
  * Grossman  _et al._ (2014) Y. Grossman, Z. Ligeti, and D. J. Robinson, JHEP 01, 066 (2014), arXiv:1308.4143 [hep-ph] .