# E-NER: Evidential Deep Learning for Trustworthy Named Entity Recognition

Zhen Zhang1 Mengting Hu1 Shiwan Zhao Minlie Huang2 Haotian Wang1
Lemao Liu3 Zhirui Zhang3 Zhe Liu4 Bingzhe Wu3*
1 College of Software, Nankai University, 2 The CoAI group, Tsinghua
University
3 Tencent AI Lab, 4 Zhejiang Lab
<EMAIL_ADDRESS><EMAIL_ADDRESS>Mengting Hu and Bingzhe Wu are
the corresponding authors. Independent researcher.

###### Abstract

Most named entity recognition (NER) systems focus on improving model
performance, ignoring the need to quantify model uncertainty, which is
critical to the reliability of NER systems in open environments. Evidential
deep learning (EDL) has recently been proposed as a promising solution to
explicitly model predictive uncertainty for classification tasks. However,
directly applying EDL to NER applications faces two challenges, i.e., the
problems of sparse entities and OOV/OOD entities in NER tasks. To address
these challenges, we propose a trustworthy NER framework named E-NER
111https://github.com/Leon-bit-9527/ENER by introducing two uncertainty-guided
loss terms to the conventional EDL, along with a series of uncertainty-guided
training strategies. Experiments show that E-NER can be applied to multiple
NER paradigms to obtain accurate uncertainty estimation. Furthermore, compared
to state-of-the-art baselines, the proposed method achieves a better OOV/OOD
detection performance and better generalization ability on OOV entities.

##  1 Introduction

Named entity recognition (NER) aims to locate and classify entities in
unstructured text, such as extracting LOCATION information "New York" from the
sentence "How far is New York from me". Thanks to the development of deep
neural network (DNN), current NER methods have achieved remarkable performance
on a wide range of benchmarks Lample et al. (2016); Yamada et al. (2020); Li
et al. (2022).

Figure 1: Visualization of desired uncertainty estimations in the NER
application.

Despite this progress, current NER-related research typically focuses on
improving the model performance, such as recognition accuracy and F1 scores Yu
et al. (2020); Zhu and Li (2022). However, seldom works focus on investigating
the model’s reliability. The critical aspect of the model reliability is the
uncertainty estimation of the predictive results, which can characterize the
probability that the model prediction will be wrong. One natural way to
construct the predictive uncertainty is based on the maximum value of the
Softmax output Yan et al. (2021); Li et al. (2022); Zhu and Li (2022) (the
smaller this value, the larger the uncertainty). However, previous empirical
studies show that probabilistic predictions produced by DNN models (e.g.,
transformer and CNN) are often inaccurate Guo et al. (2017); Lee et al.
(2018); Pinto et al. (2022). Therefore, this natural way may over/under-
estimate the predictive uncertainty, hindering the model’s reliability.

High-quality uncertainty estimation helps to improve the model’s reliability
in an open environment and to find valuable samples to improve training sample
efficiency, thus reducing the cost of manual labeling. On the one hand, for
the reliability aspect, accurate uncertainty estimation can equip the NER
model with the ability to express “I do not know” to both the out-of-domain
(OOD) or out-of-vocabulary (OOV) samples Charpentier et al. (2020). A desired
uncertainty estimation is conceptually shown in Figure 1, wherein
misclassified OOV/OOD entities are assigned with significantly higher
uncertainty than the in-domain (ID) entities. Besides, the estimated
uncertainty can be further absorbed into the training process to improve the
model robustness against OOV/OOD samples. On the other hand, for the sample
efficiency aspect, prior work shows that high-quality uncertainty estimation
can also be used for selecting more "informative" samples and thus can reduce
the number of labeled samples required for training the NER model.

To attain high-quality uncertainty estimation, evidential deep learning (EDL)
Sensoy et al. (2018) provides a promising solution. EDL is superior to
existing Bayesian learning-based methods Blundell et al. (2015); Kingma et al.
(2015); Graves (2011) in that model uncertainty can be efficiently estimated
in a single forward pass that avoids inexact posterior approximation Kopetzki
et al. (2021) or time/storage-consuming Monte Carlo sampling Gal and
Ghahramani (2016). However, directly applying conventional EDL to NER
applications still faces two critical challenges: (1) sparse entities: In text
corpus, entities only take a minority. For example, only 16.8% of the words in
the commonly used CoNLL2003 dataset belong to entities. The remaining non-
entity types are labeled into the "others" (O) class. The imbalance between
entity and non-entity words can cause over-fitting and poor performance on the
entity types. (2) OOV/OOD entity discrimination: In the open environment, NER
training/test data typically comes with OOV/OOD entities. However, the
optimization objective of current EDL methods lacks explicit modeling of such
types of information.

To address these two issues, we present a trustworthy NER framework named
E-NER with a series of uncertainty-guided training strategies. For the issue
of sparse entities, we propose to use an uncertainty-guided importance
weighted (IW) loss, wherein samples with higher predictive uncertainties are
assigned larger weights. This loss helps the model training to pay more
attention to entities of interest (e.g., person and location). To solve the
issue of unknown entities, we present an additional regularization term to
penalize the case where labels are more prone to errors by assigning higher
uncertainties to corresponding samples. We empirically show these two
uncertainty-guided loss terms can improve both the quality of estimated
confidence and the robustness against OOV samples.

Our contributions are summarized as follows:

  * •

To the best of our knowledge, E-NER is the first work to explore how to
leverage evidential deep learning to improve the reliability of current NER
models. This work has successfully shown the potential of EDL to provide high-
quality uncertainty estimation in NER applications. The estimated uncertainty
can be further used for detecting OOD/OOV samples in the test phase.

  * •

For the technique contribution, we propose two uncertainty-guided loss terms
to mitigate sparse entities and OOV/OOD entity discrimination issues in the
NER task.

  * •

E-NER is extensively validated in a series of experiments. In contrast to
conventional NER methods, the result shows that E-NER comes with the following
superiority: (1) more accurate uncertainty estimation. (2) better OOV/OOD
detection performance. (3) better generalization ability on OOV entities. (4)
better sample efficiency (i.e., fewer samples are required to achieve the
same-level performance).

Figure 2:  Basic architecture of E-NER with different NER paradigms.

##  2 Preliminary

This section introduces a commonly-used EDL implementation based on the
Dirichlet-based model (DBM) Sensoy et al. (2018). We then describe how the DBM
computes the uncertainty in a closed form.

###  2.1 Dirichlet-based Model

Conventional neural network classifiers typically employ a Softmax layer to
provide a point estimation of the categorical distribution. In contrast,
Dirichlet-based models (DBM) output the parameters of a Dirichlet distribution
and then use it to estimate the categorical distribution. Specifically, for
the $i$-th sample $x^{(i)}$ (e.g., the $i$-th word in the NER task) in the
$C$-class classification task, the DBM replaces the Softmax of the neural
network with an activation function layer (e.g., Softplus) to ensure that the
network outputs non-negative values, which are considered as the evidence
$\mathbf{e}^{(i)}\in\mathbb{R}^{C}_{+}$ to support the classification. The
evidence is then used for constructing a Dirichlet distribution which models
the distribution over different classes. To this end, the parameter of a
Dirichlet distribution is obtained by:
$\boldsymbol{\alpha}^{(i)}=\mathbf{e}^{(i)}+\mathbf{1}$, where $\mathbf{1}$
represents the vector of $C$ ones. Finally, the density function of Dirichlet
distribution is given by:

$\displaystyle{\rm{Dir}}(\mathbf{p}^{(i)}|\boldsymbol{\alpha}^{(i)})=\frac{1}{B(\boldsymbol{\alpha}^{(i)})}\prod_{c=1}^{C}p_{c}^{(\alpha^{(i)}_{c}-1)},$
(1)

where $B(\boldsymbol{\alpha}^{(i)})$ is the $C$-dimensional multinomial beta
function.

To learn model parameters, given the sample $(x^{(i)},\mathbf{y}^{(i)})$,
where $\mathbf{y}^{(i)}$ is a one-hot $C$-dimensional label for sample
$x^{(i)}$, previous EDL methods build the optimization objective by combining
a cross-entropy classification loss $\mathcal{L}_{CLS}$ and a KL penalty loss
$\mathcal{L}_{KL}$:

$\displaystyle\mathcal{L}^{(i)}_{EDL}\\!=$
$\displaystyle\mathcal{L}^{(i)}_{CLS}+\mathcal{L}^{(i)}_{KL}$ (2)
$\displaystyle=$
$\displaystyle\underbrace{\sum_{c=1}^{C}y^{(i)}_{c}\left(\psi(S^{(i)})-\psi(\alpha^{(i)}_{c})\right)}_{\text{(a)
classification loss}}$ $\displaystyle+$
$\displaystyle\underbrace{\lambda_{1}KL[{\rm{Dir}}(\mathbf{p}^{(i)}|{\widetilde{\boldsymbol{\alpha}}^{(i)})}||{\rm{Dir}}(\mathbf{p}^{(i)}|\mathbf{1})]}_{\text{(b)
penalty loss}},$

where $\psi(\cdot)$ is the digamma function, and
$S^{(i)}\\!=\\!\sum^{C}_{c=1}\alpha^{(i)}_{c}$ denotes the Dirichlet strength,
$\lambda_{1}$ is the balance factor, ${\rm{Dir}}(\mathbf{p}^{(i)}|\mathbf{1})$
is a special case which is equivalent to the uniform distribution, and
$\widetilde{\boldsymbol{\alpha}}^{(i)}=\mathbf{y}^{(i)}+(1-\mathbf{y}^{(i)})\odot\boldsymbol{\alpha}^{(i)}$
denotes the masked parameters while $\odot$ refers to the Hadamard (element-
wise) product, which removes the non-misleading evidence from predicted
parameters $\boldsymbol{\alpha}^{(i)}$. Intuitively, the first term in Eq. 2
measures the classification performance while the second term can be seen as a
regularization term that penalizes misleading evidences by encouraging the
associate distribution to be close to uniform distribution (see more details
in Appendix §C.3).

###  2.2 Uncertainty Estimation of DBM

Once we obtain the Dirichlet distribution for prediction, we can estimate the
predictive uncertainty in a closed form. To this end, EDL provides two
probabilities: belief mass and uncertainty mass. The belief mass $\mathbf{b}$
represents the probability of evidence assigned to each category and the
uncertainty mass $u$ provides uncertainty estimation. Specifically, for the
sample $x^{(i)}$, the belief mass $b^{(i)}_{c}$ and uncertainty $u^{(i)}$ are
computed as:

$b^{(i)}_{c}=\frac{{e}^{(i)}_{c}}{S^{(i)}}\quad\text{and}\quad
u^{(i)}=\frac{C}{S^{(i)}},$ (3)

with the restrictions that $u^{(i)}+\sum^{C}_{c=1}b^{(i)}_{c}=1$. The belief
mass $\mathbf{b}$ and the uncertainty mass $u$ will be used to guide the
training process in our proposed framework (see Section §3.3).

##  3 E-NER Architecture

In this section, we describe the three core modules of E-NER and provide an
overview of the system architecture in Figure 2. Additionally, we revise the
learning strategy of EDL by incorporating importance weights (IW) to address
the sparse entities problem and uncertainty mass optimization (UNM) to model
the uncertainty of mispredicted entities.

###  3.1 NER Feature Extraction

Given a word sequence $X=\\{x^{(1)},...,x^{(n)}\\}$ and a target sequence
$Y=\\{y^{(1)},...,y^{(n)}\\}$. To obtain the hidden representation $H$ of $X$,
the words in the sentence $X$ are first preprocessed according to the input
form required by the corresponding NER method. Then the processed input is fed
into an Encoder module (e.g., BERT Devlin et al. (2019)) to compute the hidden
representation $H={\rm{Encoder}}(X)$, where $H\in\mathbb{R}^{n\times d_{h}}$
and $d_{h}$ denotes the dimension of the hidden representation. The input
format for NER models can vary depending on the paradigm used. Three NER
paradigms were considered for this study: sequence labeling (Figure 2(a)),
span-based (Figure 2(b)), and Seq2Seq (Figure 2(c)). The specific formats for
these paradigms are provided in the Appendix §A. Note that in the Seq2Seq
(sequence-to-sequence) paradigm, we choose a pointer-based model Yan et al.
(2021), so that we don’t need to learn on the entire vocabulary.

###  3.2 Dirichlet-based Prediction Layer

Once we obtain the hidden representation, we introduce a Dirichlet-based layer
to produce the final predictive distribution. Precisely, for the $i^{th}$
sample, the hidden representation $h$ is fed to the fully connected layer to
output logits, and then we can transform the logits into Dirichlet parameters
$\boldsymbol{\alpha}$ as described in Section §2.1. Finally, as shown in
Figure 2, only one forward step using Eq. 3 is sufficient to calculate the
uncertainty $u^{(i)}$, while the probability distribution $\mathbf{p}^{(i)}$
and prediction ${y}^{(i)}$ are calculated as follows:

${\mathbf{p}^{(i)}}\\!=\\!\frac{\boldsymbol{\alpha}^{(i)}}{S^{(i)}},\;\quad
y^{(i)}\\!=\\!\mathop{\mathrm{arg\,max}}\limits_{c\in
C}\left[{{p}^{(i)}_{c}}\right].$ (4)

###  3.3 E-NER Model Learning

Overview. The objective function of EDL training is to minimize the sum of
losses over all words. Due to the sparse entities and OOV/OOD entities issues,
directly applying EDL to NER leads suboptimal uncertainty estimates. We
improve conventional EDL methods by incorporating belief mass and uncertainty
into the network training process. Specifically, two key modifications are
introduced: (1) We compute importance weights for each sample based on the
belief mass to reweight the original classification loss in Eq. 2(a). (2) We
introduce an additional term to increase the uncertainty of mispredicted
instances, which explicitly improves the quality of uncertainty estimation and
helps OOD entity detection.

Figure 3: (a) Overview of uncertainty estimation for Dirichlet distributions.
(b-d) Typical patterns of Dirichlet distribution for an example 3-class
classification task.

Importance Weight. Due to the inherent imbalance between entities and non-
entities in NER datasets, conventional EDL methods tend to overfit non-
entities and assign high uncertainty estimates to entities. To make the
training focus more on the entities and increase the evidence corresponding to
the ground-truth category, we use the belief mass of the ground-truth category
to compute the category-level uncertainty for each instance to adjust the
loss. Specifically, for the $i^{th}$ sample, we use
$(\mathbf{1}-\mathbf{b}^{(i)})$ as the category-level uncertainty which serves
as the importance weights of entity categories during training. To this end,
we replace the ground truth $\mathbf{y}^{(i)}$ of one-hot representation with
an importance weight (IW)
$\mathbf{w}^{(i)}=(\mathbf{1}-\mathbf{b}^{(i)})\odot\mathbf{y}^{(i)}$, and
lastly, the Eq. 2(a) is adjusted to:

$\displaystyle\mathcal{L}^{(i)}_{IW}=$
$\displaystyle\sum_{c=1}^{C}{\color[rgb]{0.80,0.10,0.1}w^{(i)}_{c}}\left(\psi(S^{(i)})-\psi(\alpha^{(i)}_{c})\right).$
(5)

As illustrated in Figure 3(b), the belief mass of the ground-truth category is
high, indicating a high level of certainty in the prediction. In this case,
the importance weight (IW) assigned will be small. Conversely, Figure 3(c)
presents a small belief mass, indicating an uncertain prediction. IW will be
assigned a large value. In this manner, the training process can focus more on
sparse but valuable entities.

Uncertainty Mass Optimization. Assigning high uncertainty to OOV/OOD entities
(see Figure 3(d) as an example) facilitates OOV/OOD entity detection. However,
ground-truth OOV/OOD samples are not available during training. One solution
is to synthesize such data on the boundary of the in-domain region via a
generative model Lee et al. (2018). In this paper, we propose a more
convenient way to treat hard samples as OOV/OOD samples which are often
outliers and are mispredicted even after adequate model training. In this way,
we enable the model to detect OOV/OOD data. Specifically, uncertainty mass
optimization (UNM) assigns higher uncertainty to more error-prone samples for
the model to express a lack of evidence, by adding an uncertainty mass penalty
term $\mathcal{L}_{UNM}$ to the wrongly predicted samples:

$\displaystyle\mathcal{L}_{UNM}\\!=\\!-\\!\lambda_{2}\\!\sum_{i\in\\{\hat{y}^{(i)}\neq
y^{(i)}\\}}\\!{\rm{log}}(u^{(i)}).$ (6)

The coefficient $\lambda_{2}=\lambda_{0}\
\mathrm{exp}\\{-(\mathrm{ln}\lambda_{0}/T)t\\}$, where
$\lambda_{2}\in[\lambda_{0},1]$, $\lambda_{0}\ll 1$ is a small positive
constant, $t$ is the current training epoch, and $T$ is the total number of
training epochs. As the training epoch $t$ increases towards $T$, the factor
$\lambda_{2}$ will increase monotonically from $\lambda_{0}$ to 1.0. This
allows the network to initially focus on optimizing classification and
gradually shift its emphasis towards optimizing UNM as the training
progresses.

Overall Loss. The overall loss function combines three components: the
importance weighted classification loss $\mathcal{L}_{IW}$, the KL divergence
penalty loss $\mathcal{L}_{KL}$, and the uncertainty mass loss
$\mathcal{L}_{UNM}$ for mispredicted entities. Each element contributes to the
overall loss and is defined as follows:

$\mathcal{L}_{overall}=\sum_{i=1}^{N}(\mathcal{L}^{(i)}_{IW}+\mathcal{L}^{(i)}_{KL})+\mathcal{L}_{UNM}.$
(7)

##  4 Experiments

###  4.1 Research Questions

In this section, we design extensive experiments to validate whether the
proposed method obtains high-quality uncertainty estimation. Concretely, the
following four research questions will be investigated.

RQ1: Whether E-NER improves the quality of confidence estimation in contrast
to prior work?

RQ2: Can uncertainty provided by E-NER achieve better OOV/OOD detection
performance?

RQ3: Can E-NER improve the model generalization ability on OOV samples?

RQ4: Can E-NER help to find valuable instances to improve the sample
efficiency of NER model training?

Following these four research questions, we provide further discussions on our
method including ablation studies and limitations.

Dataset | Sentences | Types | Domain
---|---|---|---
CoNLL2003 | 22,137 | 4 | Newswire
OntoNotes 5.0 | 76,714 | 18 | General
WikiGold | 1,696 | 4 | General
Table 1: Statistics of the NER dataset. Dataset | Sentences | Entities | OOV Rate
---|---|---|---
TwitterNER | 3257 | 3990 | 0.62
CoNLL2003-Typos | 2676 | 4130 | 0.71
CoNLL2003-OOV | 3685 | 5648 | 0.96
Table 2: Statistics of OOV entities in the test set.

###  4.2 Datasets and Metrics

Datasets from Different Domains. To answer the above research questions, we
choose three widely-used datasets, including CoNLL2003 Tjong Kim Sang and De
Meulder (2003), OntoNotes 5.0 Weischedel et al.
(2013)222https://catalog.ldc.upenn.edu/LDC2013T19 and WikiGold Balasuriya et
al. (2009). The statistics are displayed in Table 1.

OOV Datasets. We further choose three public OOV datasets, including
TwitterNER Zhang et al. (2018), CoNLL2003-Typos Wang et al. (2021), and
CoNLL2003-OOV Wang et al. (2021). The statistics are displayed in Table 2.

Metrics. We evaluate the results using three metrics: F1, Expected Calibration
Error (ECE), and Area Under the ROC Curve (AUC). F1 is a commonly used
performance indicator in NER. ECE is a metric that measures the confidence
calibration of a model, with a low score indicating a well-calibrated model.
AUC is a commonly used metric for evaluating the performance of binary
classifiers, and we use it to evaluate the OOV/OOD detection performance.
Their detailed computations are described in the Appendix §C.2.

Setting | Typos | OOV | OOD |
---|---|---|---|---
Con | Unc | Con | Unc | Con | Unc |
BERT-Tagger Devlin et al. (2019) | 0.812 | 0.812 | 0.689 | 0.751 | 0.674 | 0.756 |
-EDL | 0.805 | 0.808 | 0.699 | 0.759 | 0.693 | 0.767 |
-E-NER(ours) | 0.820 | 0.817 | 0.700 | 0.760 | 0.769 | 0.799 |
1-8 SpanNERFu et al. (2021) | 0.717 | 0.783 | 0.614 | 0.773 | 0.623 | 0.799 |
-EDL | 0.701 | 0.759 | 0.607 | 0.760 | 0.620 | 0.792 |
-E-NER(ours) | 0.741 | 0.792 | 0.640 | 0.796 | 0.676 | 0.824 |
1-8 Seq2Seq Yan et al. (2021) | 0.825 | 0.833 | 0.724 | 0.794 | 0.797 | 0.820 |
-EDL | 0.829 | 0.830 | 0.729 | 0.787 | 0.793 | 0.818 |
-E-NER(ours) | 0.824 | 0.841 | 0.743 | 0.803 | 0.822 | 0.847 |
Table 3: Evaluation results of OOV/OOD detection in terms of AUC. The three
binary detection tasks can use either confidence (Con) or uncertainty (Unc)
for classification.

###  4.3 Experiment Setting

We conduct experiments on three popular NER paradigms: sequence labeling,
span-based, and Seq2Seq. The following three models are chosen for evaluating
each paradigm.

BERT-Tagger Devlin et al. (2019). It follows the classical paradigm,
recognizing entities via _sequence labeling_.

SpanNER333https://github.com/neulab/spanner Fu et al. (2021). It enumerates
all spans and detects entities from them. For simplicity, we use the original
span-based method, without any constraints or data processing.

Seq2Seq444https://github.com/yhcc/BARTNER Yan et al. (2021). It is a
generative model based on BART, which does not require additional labeling
strategies and entity enumeration.

In the experiments, all the reported results are the average of five runs. The
experiment details are introduced in Appendix §C.

Setting | CoNLL2003 | OntoNotes 5.0
---|---|---
F1($\uparrow$) | ECE($\downarrow$) | F1($\uparrow$) | ECE($\downarrow$)
BERT-Tagger | 91.32 | 0.0845 | 88.20 | 0.1053
-EDL | 91.36 | 0.0755 | 88.09 | 0.0838
-E-NER(ours) | 91.55 | 0.0739 | 88.74 | 0.0603
1-5 SpanNER | 91.94 | 0.0673 | 87.82 | 0.0609
-EDL | 91.97 | 0.0481 | 87.39 | 0.0474
-E-NER(ours) | 92.06 | 0.0414 | 88.44 | 0.0434
1-5 Seq2Seq | 93.05 | 0.0324 | 89.89 | 0.0375
-EDL | 92.84 | 0.0322 | 90.22 | 0.0329
-E-NER(ours) | 93.15 | 0.0225 | 90.64 | 0.0328
Table 4: Evaluation results in various NER systems, in terms of F1 (%) and ECE
for evaluating performance and confidence quality, respectively. Figure 4:
Model calibration curve. The basic encoder is SpanNER. This figure is depicted
by evaluating subsets separately, where each subset has the same range of
confidence.

###  4.4 Research Question Discussions

####  4.4.1 Confidence Estimation Quality

To answer the first research question, an important concept should be
clarified, i.e., _what is qualified confidence?_ This concept should have a
positive correlation with performance, meaning that higher confidence should
indicate better performance and vice versa, as depicted by the dashed line in
Figure 4. Our findings reveal that on both datasets, Softmax is far below the
perfectly calibrated line, indicating that confidence does not reflect
performance well, and it is an example of _over-confidence_. However, E-NER is
found to approach the perfect calibrated line. This suggests that E-NER can
produce well-qualified confidence.

We further evaluate all paradigms and present the results in Table 4.3. It can
be observed that E-NER consistently performs the best across all paradigms.
This demonstrates that E-NER can be effectively applied in various frameworks.
When comparing EDL to the original models, it is observed that while EDL
improves confidence estimation, it also results in a decline in performance.
For example, on OntoNotes 5.0 dataset, EDL performs worse than BERT-Tagger and
SpanNER in terms of the F1 metric. This highlights the limitations of directly
applying the EDL approach. In contrast, E-NER performs the best on both
metrics, demonstrating that it can provide better-qualified confidence without
negatively impacting performance, and even achieving slight improvements in
all settings. A typical reliability diagram is also included in Appendix §B.1
for a more detailed representation.

####  4.4.2 OOV/OOD Detection

The typical usage of uncertainty is to detect whether an instance is OOV/OOD
or not, as large uncertainty tends to reveal unnatural instances, such as OOV
and OOD. To evaluate uncertainty from this usage (RQ2), we choose three binary
detection tasks, including typos, OOV, and OOD. The results are shown in Table
4.2.

Firstly, it can be observed that, when compared to the original model of each
paradigm, EDL does not improve the performances in most experiments of the
three paradigms. This verifies that EDL is not effective in addressing the
_OOV/OOD entity discrimination_ challenge of NER. Then we found that E-NER
significantly outperforms the original models and EDL in various paradigms. In
particular, in span-based OOD detection, E-NER outperforms SpanNER by +5.3%
and EDL by +5.6% on AUC when using confidence for detection. This demonstrates
the effectiveness of E-NER in distinguishing whether an entity is OOV/OOD or
not. Note that using uncertainty is better than using confidence for OOV/OOD
detection in most cases.

Methods |  TwitterNER | CoNLL2003
---|---|---
Typos | OOV
VaniIB Alemi et al. (2017) | 71.19 | 83.49 | 70.12
DataAug Dai and Adel (2020) | 73.69 | 81.73 | 69.60
SpanNER (BERT large) | 71.57 | 81.83 | 64.43
SpanNER (RoBERTa large) | 71.70 | 82.85 | 64.70
SpanNER (AlBERT large) | 70.33 | 82.49 | 64.12
1-4 EDL-SpanNER (BERT large) | 74.14 | 82.89 | 68.40
E-SpanNER (BERT base) | 74.94 | 83.31 | 67.99
E-SpanNER (BERT large) | 75.64 | 83.64 | 69.71
1-4 $\Delta$ E-NER-NER vs. SpanNER | 4.07↑ | 1.81↑ | 5.28↑
Table 5: Evaluation results of generalization on OOV samples in terms of F1
(%). To compare fairly, we also choose SpanNER as the basic encoder.

####  4.4.3 Generalization on OOV Samples

Another benefit of well-qualified confidence is the robustness to noise, since
the model is properly calibrated without over or under-confidence. Thus, we
further investigate E-NER’s generalizing ability on OOV samples (RQ3). The
results on three OOV datasets are reported in Table 4.4.2.

It is first observed that E-NER (BERT large) achieves the best performances on
TwitterNER and CoNLL2003-Typos datasets, and competitive performance on
CoNLL2003-OOV. Compared with a strong baseline SpanNER (BERT large), E-NER
(BERT large) significantly outperforms it by +4.07%, +1.81% and +5.28% on
three datasets, respectively. This validates the generalizing ability of our
approach. Secondly, by comparing EDL (BERT large) and E-NER s(BERT large), our
method also achieves consistently better performances. This further validates
that our proposed two uncertainty-guided loss terms effectively promote the
robustness against OOV samples.

Setting | CoNLL2003 | OntoNotes 5.0
---|---|---
Ratio | F1($\uparrow$) | Ratio | F1($\uparrow$)
Random | 5.5% | 85.39 | 3.0% | 79.47
Entropy | 5.5% | 88.29 | 3.0% | 84.80
MC dropout | 5.5% | 88.67 | 3.0% | 86.06
EDL | 5.5% | 90.51 | 3.0% | 86.25
E-NER | 5.5% | 90.88 | 3.0% | 86.68
Table 6: Evluation results of in-domain data selection in terms of F1 (%). Ratio indicates the proportion of selected samples out of the whole training set. Setting | ${\rm{WikiGold}}_{\leftarrow{\rm{CoNLL.}}}$ | ${\rm{CoNLL2003}}_{\leftarrow{\rm{Onto.}}}$
---|---|---
Ratio | F1($\uparrow$) | Ratio | F1($\uparrow$)
Random | 4.8% | 53.67 | 4.7% | 84.23
Entropy | 4.8% | 80.63 | 4.7% | 88.81
MC dropout | 4.8% | 82.87 | 4.7% | 90.32
EDL | 4.8% | 83.32 | 4.7% | 90.12
E-NER | 4.8% | 84.08 | 4.7% | 90.52
Table 7: Evaluation results of cross-domain data selection in terms of F1 (%).
The left side of the arrow $\leftarrow$ is the target domain, and the right
side is the source domain.

####  4.4.4 Sample Efficiency

In active learning, a sample’s uncertainty can be utilized for data selection.
Then whether the selected samples are valuable also suggests the quality of
uncertainty. To evaluate E-NER from this perspective (RQ4), we design in-
domain and cross-domain sample selection experiments. The results are
displayed in Table 6 and Table 7, respectively.

It is found that using the same scale of samples, E-NER achieves consistently
the best performances in both the in-domain and cross-domain settings. This
verifies that uncertainty predicted by E-NER has better quality. Concretely,
MC dropout attains uncertainty with multiple runs of sub-models, which costs
time and memory. Though outperforming naive random selection and entropy of
softmax, MC dropout is still less performed than EDL and E-NER, which both
directly compute the uncertainty in one forward pass. Then we see that EDL
does not always outperform MC dropout, as the cross-domain experiment
${\rm{CoNLL2003}}_{\leftarrow{\rm{Onto}}}$ shown. Yet E-NER, concentrating on
two issues of NER task, is universally effective, and can better handle the
challenges of an open environment.

###  4.5 Further Analysis

Setting | CoNLL2003 | OntoNotes 5.0
---|---|---
F1 | ECE | F1 | ECE
E-NER | 92.06 | 0.041 | 88.44 | 0.043
-UNM | 92.10 | 0.058 | 88.21 | 0.051
-IW | 91.95 | 0.045 | 87.77 | 0.042
Table 8: Evaluation results of ablation study in terms of F1 (%) and ECE.

Ablation Study. To explore the effects of individual loss terms, the ablation
study is presented in Table 8. It is observed that removing each loss term
would cause performance declines in most evaluation metrics. Concretely,
removing IW causes the F1 score to decrease more than removing UNM. On the
contrary, removing UNM makes a significant degradation in ECE. Overall, this
study indicates that the proposed uncertainty-guided terms are both effective.

Why E-NER Works. We incorporate two uncertainty-guided loss terms into EDL.
Firstly, IW is designed for sparse entities which leads to an imbalance
problem. Using uncertainties as weights helps the model training to pay more
attention to entities of interest. As reported in Table 8, IW is effective in
improving the F1 score. Secondly, UNM is proposed to deal with OOV/OOD
entities. Such entities should have larger uncertainties compared to normal
ones, however, naive EDL does not model this explicitly. E-NER increases the
uncertainty of mispredictions which are relatively close to OOV/OOD entities.
As shown in Table 8, UNM helps to improve the quality of uncertainty
estimation. These two uncertainty-guided loss terms target different NER
issues, and using uncertainty (IW) and learning uncertainty (UNM)
interactively allows E-NER to perform well in various experimental settings.
Furthermore, we showcase actual predictions in Appendix §B.2.

##  5 Related Work

NER Paradigm. NER is a fundamental task in information extraction. The
mainstream methods of NER can be divided into three categories: sequence
labeling, span-based, and Seq2Seq. Sequence labeling methods assign a label to
each token in a sentence to identify flat entities, and are better at handling
longer entities with lower label consistency Fu et al. (2021). Span-based
methods, which enumerate and classify entity sets in a sentence according to
the maximum span length, perform better on sentences with OOV words and
entities of medium length Alemi et al. (2017); Dai and Adel (2020); Fu et al.
(2021). Seq2Seq methods directly generate the entities and corresponding
labels in the sentence, and are capable of handling various NER subtasks
uniformly Yan et al. (2021). Recently, NER systems are undergoing a paradigm
shift Akbik et al. (2018); Yan et al. (2019), using one paradigm to handle
multiple types of NER tasks. Zhang et al. (2022) analysis the incorrect bias
in Seq2Seq from the perspective of causality, and designed a data augmentation
method based on the theory of backdoor adjustment, making Seq2Seq more
suitable for unified NER tasks.

Uncertainty Estimation. Bayesian deep learning uses Bayesian principles to
estimate uncertainty in DNN parameters. However, modeling uncertainty in
network parameters does not guarantee accurate estimation of predictive
uncertainty Sensoy et al. (2021). Recently, there has been a trend in using
the output of neural networks to estimate the parameters of the Dirichlet
distribution for uncertainty estimation Sensoy et al. (2018); Malinin and
Gales (2018). The EDL Sensoy et al. (2018) has the advantages of
generalizability and low computational cost, making it applicable to various
tasks Han et al. (2021); Hu and Khan (2021). However, their uncertainty
estimates have difficulty expressing uncertainties outside the domain Amini et
al. (2020); Hu and Khan (2021). In contrast, the Prior Networks Malinin and
Gales (2018) require OOD data during training to distinguish in-distribution
(ID) and OOD data. When the NER model encounters unseen entities (e.g., OOV
and OOD), it is easy to make unreliable predictions, which are often
considered from the perspective of data augmentation or information theory
Fukuda et al. (2020); Wang et al. (2022), but there is no guarantee that these
methods will achieve a balance between performance and robustness.

##  6 Conclusion

In this work, we study the problem of trustworthy NER by leveraging evidential
deep learning. To address the issues of sparse entities and OOV/OOD entities,
we propose E-NER with two uncertainty-guided loss terms. Extensive
experimental results demonstrate that the proposed method can be effectively
applied to various NER paradigms. The uncertainty estimation quality of E-NER
is improved without harming performance. Additionally, the well-qualified
uncertainties contribute to detecting OOV/OOD, generalization, and sample
selection. These results validate the superiority of E-NER on real-world
problems.

## Limitations

Our work is the first attempt to explore how evidential deep learning can be
used to improve the reliability of current NER models. Despite the improved
performance and robustness, our work has limitations that may guide our future
work.

First, we propose a simple method to treat hard samples (such as outliers) in
the dataset as OOV/OOD samples, enabling the model to detect OOV/OOD data with
minimal cost. However, there is still a certain gap between these hard samples
and the real OOV/OOD data. OOV/OOD detection performance can still be improved
by further incorporating more real OOV/OOD samples, for example, real OOD data
from other domains, well-designed adversarial examples, generated OOV samples
by data augmentation techniques, etc.

Second, we evaluate the versatility of E-NER by applying it to mainstream NER
paradigms. However, there are still other paradigms, such as Hypergraph-based
methods Lu and Roth (2015) and the W2NER Li et al. (2022) approach in recent
work, that could be evaluated in the future.

## Acknowledgements

We sincerely thank all the anonymous reviewers for providing valuable
feedback. This work is supported by the youth program of National Science Fund
of Tianjin, China (Grant No. 22JCQNJC01340), the Fundamental Research Funds
for the Central University, Nankai University (Grant No. 63221028), and the
key program of National Science Fund of Tianjin, China (Grant No.
21JCZDJC00130)

## References

  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _Proceedings of the 27th International Conference on Computational Linguistics (COLING)_ , pages 1638–1649.
  * Alemi et al. (2017) Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. 2017.  Deep variational information bottleneck.  In _International Conference on Learning Representations (ICLR)_ , pages 1–19.
  * Amini et al. (2020) Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. 2020.  Deep evidential regression.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 14927–14937.
  * Balasuriya et al. (2009) Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R. Curran. 2009.  Named entity recognition in Wikipedia.  In _Proceedings of the 2009 Workshop on The People’s Web Meets NLP: Collaboratively Constructed Semantic Resources (People’s Web)_ , pages 10–18.
  * Blundell et al. (2015) Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015.  Weight uncertainty in neural network.  In _International conference on machine learning (ICML)_ , pages 1613–1622.
  * Charpentier et al. (2020) Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. 2020.  Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 1356–1367.
  * Dai and Adel (2020) Xiang Dai and Heike Adel. 2020.  An analysis of simple data augmentation for named entity recognition.  In _Proceedings of the 28th International Conference on Computational Linguistics (COLING)_ , pages 3861–3867.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_ , pages 4171–4186.
  * Fu et al. (2021) Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021.  SpanNER: Named entity re-/recognition as span prediction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP)_ , pages 7183–7195.
  * Fukuda et al. (2020) Nobukazu Fukuda, Naoki Yoshinaga, and Masaru Kitsuregawa. 2020.  Robust Backed-off Estimation of Out-of-Vocabulary Embeddings.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4827–4838.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning (ICML)_ , pages 1050–1059.
  * Graves (2011) Alex Graves. 2011.  Practical variational inference for neural networks.  In _Advances in neural information processing systems (NeurIPS)_ , page 2348–2356.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017.  On calibration of modern neural networks.  In _Proceedings of the 34th International Conference on Machine Learning (ICML)_ , pages 1321–1330.
  * Han et al. (2021) Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. 2021.  Trusted multi-view classification.  In _International Conference on Learning Representations (ICLR)_ , pages 1–16.
  * Hu and Khan (2021) Yibo Hu and Latifur Khan. 2021.  Uncertainty-aware reliable text classification.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (SIGKDD)_, pages 628–636.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , pages 1–15.
  * Kingma et al. (2015) Durk P Kingma, Tim Salimans, and Max Welling. 2015.  Variational dropout and the local reparameterization trick.  In _Advances in neural information processing systems (NeurIPS)_ , pages 2575–2583.
  * Kopetzki et al. (2021) Anna-Kathrin Kopetzki, Bertrand Charpentier, Daniel Zügner, Sandhya Giri, and Stephan Günnemann. 2021.  Evaluating robustness of predictive uncertainty estimation: Are dirichlet-based models reliable? In _International Conference on Machine Learning (ICML)_ , pages 5707–5718.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_ , pages 260–270.
  * Lee et al. (2018) Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. 2018.  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  In _International Conference on Learning Representations (ICLR)_ , pages 1–16.
  * Li et al. (2022) Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. 2022.  Unified named entity recognition as word-word relation classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence(AAAI)_ , pages 10965–10973.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations (ICLR)_ , pages 1–18.
  * Lu and Roth (2015) Wei Lu and Dan Roth. 2015.  Joint mention extraction and classification with mention hypergraphs.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 857–867.
  * Malinin and Gales (2018) Andrey Malinin and Mark Gales. 2018.  Predictive uncertainty estimation via prior networks.  In _Advances in neural information processing systems (NeurIPS)_ , page 7047–7058.
  * Pinto et al. (2022) Francesco Pinto, Philip HS Torr, and Puneet K Dokania. 2022.  An impartial take to the cnn vs transformer robustness contest.  In _European Conference on Computer Vision (ECCV)_ , pages 466–480.
  * Sensoy et al. (2018) Murat Sensoy, Lance M. Kaplan, and Melih Kandemir. 2018.  Evidential deep learning to quantify classification uncertainty.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , page 3183–3193.
  * Sensoy et al. (2021) Murat Sensoy, Maryam Saleki, Simon Julier, Reyhan Aydogan, and John Reid. 2021.  Misclassification risk and uncertainty quantification in deep classifiers.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 2484–2492.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 (HLT-NAACL)_ , pages 142–147.
  * Wang et al. (2022) Xiao Wang, Shihan Dou, Limao Xiong, Yicheng Zou, Qi Zhang, Tao Gui, Liang Qiao, Zhanzhan Cheng, and Xuanjing Huang. 2022.  MINER: Improving out-of-vocabulary named entity recognition from an information theoretic perspective.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL)_ , pages 5590–5600.
  * Wang et al. (2021) Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, and Zexiong and Pang. 2021.  TextFlint: Unified multilingual robustness evaluation toolkit for natural language processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations (ACL-IJCNLP)_ , pages 347–355.
  * Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013.  OntoNotes Release 5.0.  In _3_. Abacus Data Network.
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020\.  LUKE: Deep contextualized entity representations with entity-aware self-attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6442–6454.
  * Yan et al. (2019) Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019.  TENER: adapting transformer encoder for named entity recognition.  _CoRR_ , abs/1911.04474.
  * Yan et al. (2021) Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021.  A unified generative framework for various NER subtasks.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP)_ , pages 5808–5822.
  * Yu et al. (2020) Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020.  Named entity recognition as dependency parsing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 6470–6476.
  * Zhang et al. (2018) Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018.  Adaptive co-attention network for named entity recognition in tweets.  In _Thirty-Second AAAI Conference on Artificial Intelligence (AAAI)_ , page 5674–5681.
  * Zhang et al. (2022) Shuai Zhang, Yongliang Shen, Zeqi Tan, Yiquan Wu, and Weiming Lu. 2022.  De-bias for generative extraction in unified NER task.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL)_ , pages 808–818.
  * Zhu and Li (2022) Enwei Zhu and Jinpeng Li. 2022.  Boundary smoothing for named entity recognition.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 7096–7108.

| BERT-Tagger | SpanNER | Seq2Seq
---|---|---|---
Input |  $X=\\{x^{(1)},x^{(2)},...,x^{(n)}\\}$ |  $X=\\{x^{(1)},x^{(2)},...,x^{(n)}\\}$ |  $X=\\{x^{(1)},x^{(2)},...,x^{(n)}\\}$
Processing |  - |  Enumerate all spans $S=\\{s^{(1)},s^{(2)},...,s^{(m)}\\}$ |  Obtain start and end indexes of entities $Y=\\{y^{b}_{1},y^{e}_{1},y_{1},...,y^{b}_{k},y^{e}_{k},y_{k}\\}$
Hidden state |  $h={\rm{Encoder}}(X);$ $h\in\mathbb{R}^{n\times d}$ |  $h={\rm{Encoder}}(s^{(i)});$ $h\in\mathbb{R}^{d}$ |  $h_{t}=\mathrm{EncoderDecoder}(X,Y_{<t});$ $h_{t}\in\mathbb{R}^{d}$
Inference |  Token-level classification |  Span-level classification |  Target sequence $Y$ generation
Table 9: Explanation of the three NER paradigms.

##  Appendix A NER Paradigms

Here we introduce three popular NER paradigms, shown in Table 9.

BERT-Tagger. It follows the sequence labeling paradigm, which aims to assign a
tagging label $Y=\\{y^{(1)},...,y^{(n)}\\}$ to each word in a sequence
$X=\\{x^{(1)},...,x^{(n)}\\}$. We use BERT-Tagger Devlin et al. (2019) as the
baseline method for sequence labeling. The labeling method adopts a BIO tag
set, which indicates the beginning and interior of an entity, or other words.
$X$ is fed to BERT to obtain hidden states, followed by a nonlinear classifier
to classify each word.

SpanNER. Given an input sentence $X=\\{x^{1},...,x^{n}\\}$, SpanNER enumerates
all spans and obtains a set $S=\\{s^{(1)},...,s^{(i)},...,s^{(m)}\\}$. Then it
assigns each span an entity label ${y}$ Fu et al. (2021). The maximum length
$l$ of the span is artificially set. Assume a sentence’s length is n and the
maximum span length is set to 2, the subscript of the span set can be
expressed as $\\{(1,1),(1,2)...(n-1,n-1),(n-1,n),(n,n)\\}$. Each span is fed
into the encoder to obtain a vector representation.

Seq2Seq. As presented in Table 9, given an input sentence
$X=\\{x^{(1)},x^{(2)},...,x^{(n)}\\}$, the target sequence is represented as
$Y=\\{y^{b}_{1},y^{e}_{1},y_{1},...,y^{b}_{k},y^{e}_{k},y_{k}\\}$. This target
sequence indicates $X$ describes $k$ entities. Take the first entity as an
example, its beginning and end indexes are $y^{b}_{1}$ and $y^{e}_{1}$, with
entity category $y_{1}$. This method learns in a sequence-to-sequence manner
Yan et al. (2021).

(a) CoNLL2003 Softmax

(b) CoNLL2003 E-NER

(c) OntoNotes 5.0 Softmax

(d) OntoNotes 5.0 E-NER

Figure 5: Reliability diagrams.

(a) OOD Softmax

(b) OOD E-NER

Figure 6: Reliability diagrams of OOD entities. CoNLL2003 is used as the
training set. The testing set of WikiGold is used for evaluating the OOD
samples.

##  Appendix B Additional Experimental Analysis

###  B.1 Reliability Diagrams

We further depict the reliability diagrams to evaluate the quality of
uncertainty estimation. As shown in Figure 5 and Figure 6, the confidence
range is equally divided into ten bins. Then the subset within the same
confidence range is utilized to compute the accuracy.

As shown in Figure 5, the confidence of Softmax represents poor accuracy,
indicating it is over-confident. Then compared with Softmax, E-NER nearly
approaches the perfectly calibrated line and has a much smaller ECE score.
This suggests that E-NER can yield well-qualified confidence, showing it is
more trustworthy. Then the observations in Figure 6 are similar, which
demonstrates the reliability of the proposed approach for OOD entities.

Case | Sentence | Softmax+Entropy | E-NER
---|---|---|---
* |  Mapping: {MIS: miscellaneous; PER: person; ORG: organization; O: non-entity} |  Entity: {Predcition; Confidence%; Uncertainty%}
IID |  A visit to the computer centre offering $Internet^{\rm{E^{1}}}_{\rm{[MIS]}}$ services found a $European^{\rm{E^{2}}}_{\rm{[MIS]}}$ official clicking away on his mouse. |  ${\color[rgb]{0.80,0.10,0.1}{\rm{E^{1}}}}_{\\{\rm{O}\,;\,99.9\,;\,{\color[rgb]{0.80,0.10,0.1}8.0}\\}}$ ${{\color[rgb]{0.00,0.00,1.00}{\rm{E^{2}}}}}_{\\{\rm{MIS}\,;\,99.9\,;\,{\color[rgb]{0.00,0.00,1.00}3.0}\\}}$ |  ${\color[rgb]{0.80,0.10,0.1}{\rm{E^{1}}}}_{\\{\rm{O}\,;\,42.0\,;\,{\color[rgb]{0.80,0.10,0.1}70.8}\\}}$ ${{\color[rgb]{0.00,0.00,1.00}{\rm{E^{2}}}}}_{\\{\rm{MIS}\,;\,92.7\,;\,{\color[rgb]{0.0,0.0,1.0}8.9}\\}}$
IIID |  ${Lazio^{\rm{E^{1}}}_{\rm{[ORG]}}}$ have injury doubts about striker $Pierluigi$ $Casiragh^{\rm{E^{2}}}_{\rm{[PER]}}.$ |  ${\color[rgb]{0.80,0.10,0.1}{\rm{E^{1}}}}_{\\{\rm{O}\,;\,98.8\,;\,{\color[rgb]{0.00,0.00,1.00}7.3}\\}}$ ${{\color[rgb]{0.00,0.00,1.00}{\rm{E^{2}}}}}_{\\{\rm{PER}\,;\,99.9\,;\,{\color[rgb]{0.00,0.00,1.00}0.4}\\}}$ |  ${{\color[rgb]{0.00,0.00,1.00}\rm{E^{1}}}}_{\\{\rm{ORG}\,;\,88.9\,;\,{\color[rgb]{0.00,0.00,1.00}12.5}\\}}$ ${{\color[rgb]{0.00,0.00,1.00}{\rm{E^{2}}}}}_{\\{\rm{PER}\,;\,98.3\,;\,{\color[rgb]{0.00,0.00,1.00}2.3}\\}}$
IIIOOV |  But the ${{Inthrnet}^{\rm{E^{1}}}_{\rm{[MIS]}}}$ , a global computer network. |  ${\color[rgb]{0.80,0.10,0.1}{\rm{E^{1}}}}_{\\{\rm{O}\,;\,90.5\,;\,{\color[rgb]{0.80,0.10,0.1}23.1}\\}}$ |  ${\color[rgb]{0.0,0.0,1.0}{\rm{E^{1}}}}_{\\{\rm{MIS}\ ;28.1\,;\,{\color[rgb]{0.0,0.0,1.0}70.0}\\}}$
IVOOD |  Redesignated 65 ${Fighter\;Wing^{\rm{E^{1}}}_{\rm{[ORG]}}}$ on 24 July 1943. |  ${\color[rgb]{0.80,0.10,0.1}{\rm{E^{1}}}}_{\\{\rm{O}\,;\,99.2\,;\,{\color[rgb]{0.80,0.10,0.1}4.6}\\}}$ |  ${\color[rgb]{0.80,0.10,0.1}{\rm{E^{1}}}}_{\\{\rm{O}\,;\,51.3\,;\,{\color[rgb]{0.80,0.10,0.1}60.7}\\}}$
Table 10: Case study of Softmax and E-NER under the span-based paradigm. The
entities and their categories are already denoted in four sentences. The
predicted entities with confidence (%) and uncertainty (%) scores are also
presented. Incorrectly predicted entities are denoted by
“${\color[rgb]{0.80,0.10,0.1}{\rm{Red~{}E}}}$”, whereas
“${\color[rgb]{0.0,0.0,1.0}{\rm{Blue~{}E}}}$” represents correctly predicted
entities.

###  B.2 Case Study

As presented in Table 10, we conduct a case study by choosing four typical
cases, including ID, OOV, and OOD samples. The uncertainty of Softmax is
computed with entropy.

The first case contains two $\rm{MIS}$ entities. Softmax and E-NER both
wrongly predict the first entity to $\rm{O}$ category, with confidence scores
of 99.9% and 42.0%, respectively. This shows that Softmax is over-confident
even for error results. Yet E-NER can output a larger uncertainty score,
suggesting unsure towards the prediction. Then the second case describes two
entities. Softmax wrongly predicts the first $\rm{ORG}$ entity to $\rm{O}$
with large confidence, i.e. 98.8%. But E-NER can correctly detect the entity
category as $\rm{ORG}$.

Moreover, $Inthrnet$ in the third sentence is a $\rm{MIS}$ entity, which is
OOV due to misspelling. Softmax detects it as $\rm{O}$ with a confidence score
of 90.5%, showing over-confident for errors. On the contrary, E-NER assigns a
large uncertainty score for the OOV sample and correctly predicts the entity
category. Similarly, the last case describes an OOD entity. It can be observed
that E-NER outputs a large uncertainty score compared with Softmax.

Based on the cases and observations, we draw the following conclusions: 1)
Softmax is over-confident, even for error prediction, OOV and OOD samples; 2)
E-NER can recognize entities accurately and yield well-qualified uncertainties
towards error, OOV and OOD samples. This contributes to the reliability and
robustness of E-NER.

##  Appendix C Implementation Details

###  C.1 Model Parameters

In this paper, we implement three NER methods, including BERT-Tagger, SpanNER
and Seq2Seq. The testing set is evaluated by the best model chosen by the
development set. The implementation details are shown as follows.

BERT-Tagger. BERT-Tagger555https://github.com/google-research/bert adopts
BERT-large-cased as the base encoder Devlin et al. (2019). We set the dropout
rate as 0.2, the training batch size as 16, and the weight decay as 0.02. All
models in this paradigm use the Adam optimizer Kingma and Ba (2015) with a
learning rate of 2e-5. Sentences are truncated to a maximum length of 256. The
initial value for $\lambda_{0}$ is set to 1e-02.

SpanNER. Following the original SpanNER666https://github.com/neulab/spanner.
Fu et al. (2021), we adopt BERT-large-uncased as the base encoder Devlin et
al. (2019). The dropout rate is set to 0.2. All models in this paradigm are
trained using the AdamW optimizer Loshchilov and Hutter (2019) with a learning
rate of 1e-5, with the training batch size as 10. To improve training
efficiency, sentences are truncated to a maximum length of 128, and the
maximum length of span enumeration is set to 4. The sampling times for MC
dropout are set to 5 in the experiments. The initial value of $\lambda_{0}$ is
set to 1e-02. We use heuristic decoding and retain the highest probability
span for flattened entity recognition in span-based methods.

Seq2Seq. Following Yan et al. (2021), we exploit BART-Large
model777https://github.com/yhcc/BARTNER. BART model is fine-tuned with the
slanted triangular learning rate warmup. The warmup step is set to 0.01. The
training batch size is set to 16. The initial value of $\lambda_{0}$ is set to
1e-3.

###  C.2 Evaluation Metrics

ECE. It denotes the expected calibration error, which aims to evaluate the
expected difference between model prediction confidence and accuracy Guo et
al. (2017). Figure 6 depicts the difference in a geometric manner. The
concrete formulation is as follows:

$\displaystyle\mathrm{ECE}=\sum_{i=1}^{|B|}\frac{N_{i}}{N}\lvert{\mathrm{acc}(b_{i})-\mathrm{conf}(b_{i})}\rvert,$
(8)

where $b_{i}$ represents the $i$-th bin and $|B|$ represents the total number
of bins, setting to 10 in our experiment. $N$ denotes the number of total
samples. $N_{i}$ represents the number of samples in the $i$-th bin.
$\mathrm{acc}(b_{i})$ denotes the accuracy and $\mathrm{conf}(b_{i})$ denotes
the average of confidences in the $i$-th bin.

AUC. The area under the curve (AUC)888sklearn.metrics.auc.html. is a commonly
used metric for evaluating the performance of binary classifiers. The
formulation is as follows:

$\mathrm{AUC}(f)=\frac{\sum_{t_{0}\in\mathcal{D}^{0}}\sum_{t_{1}\in\mathcal{D}^{1}}\mathbf{1}[f(t_{0})<f(t_{1})]}{|\mathcal{D}^{0}|\cdot|\mathcal{D}^{1}|}$
(9)

where $\mathcal{D}^{0}$ is the set of negative examples, and $\mathcal{D}^{1}$
is the set of positive examples. $\mathbf{1}[f(t_{0})<f(t_{1})]$ denotes an
indicator function which returns 1 if $f(t_{0})<f(t_{1})$ otherwise return 0.

In this paper, we evaluate the performance of OOV/OOD detection using the AUC
metric. Specifically, we consider two settings for the AUC score:

  * •

Con. It uses confidence as a classifier. The correct entity recognition is a
positive example $\mathcal{D}^{1}$, and the entity recognition error is a
negative example $\mathcal{D}^{0}$.

  * •

Unc. It uses uncertainty as a classifier. Wrong prediction results of OOV/OOD
entities are considered positive examples, denoted as $\mathcal{D}^{1}$.
Correct prediction results of in-domain entities are considered negative
examples, recorded as $\mathcal{D}^{0}$. These metrics assess the classifier’s
capability in detecting OOV/OOD entities.

###  C.3 EDL Optimization Function

In this section, we give a detailed formulation of the EDL optimization
function. Eq. 1 introduces the density of the Dirichlet distribution. As the
classification loss item of EDL, its cross-entropy loss function is as
follows:

$\displaystyle\mathcal{L}^{(i)}_{CLS}$
$\displaystyle\\!=\\!\frac{\int\left[\sum_{c=1}^{C}\\!-\\!y_{c}^{(i)}{\rm{log}}(p_{c}^{(i)})\right]}{B(\boldsymbol{\alpha}^{(i)})}\prod_{c=1}^{C}p_{c}^{\alpha_{c}^{(i)}\\!-\\!1}d\mathbf{p}^{(i)}$
(10)
$\displaystyle\\!=\\!\sum_{c=1}^{C}y_{c}^{(i)}\left(\psi(S^{(i)})-\psi(\alpha_{c}^{(i)})\right).$

The KL divergence calculation function under the Dirichlet distribution takes
the following form and serves as the category penalty term in EDL:

$\displaystyle\mathcal{L}^{(i)}_{KL}$
$\displaystyle=KL[{\rm{Dir}}(\mathbf{p}^{(i)}|{\widetilde{\boldsymbol{\alpha}}^{(i)})}||{\rm{Dir}}(\mathbf{p}^{(i)}|\mathbf{1})]$
(11)
$\displaystyle={\rm{log}}\left(\frac{\Gamma(\sum_{c=1}^{C}\widetilde{\alpha}_{c}^{(i)})}{\Gamma(C)\prod_{c=1}^{C}\Gamma(\widetilde{\alpha}_{c}^{(i)})}\right)$
$\displaystyle+\sum_{c=1}^{C}(\widetilde{\alpha}_{c}^{(i)}-1)\left[\psi(S^{(i)})-\psi(\sum_{j=1}^{C}\widetilde{\alpha}^{(i)}_{j})\right].$

Finally, we get the loss function for overall EDL learning:

$\displaystyle\mathcal{L}_{EDL}\\!=\sum_{i=1}^{N}({\mathcal{L}^{(i)}_{CLS}+\mathcal{L}^{(i)}_{KL}})$
(12)


# The collider tests of a leptophilic scalar for the anomalous magnetic
moments

Ning Chen<EMAIL_ADDRESS>School of Physics, Nankai
University, Tianjin 300071, China  Bin Wang<EMAIL_ADDRESS>School of
Physics, Nankai University, Tianjin 300071, China  Chang-Yuan Yao
<EMAIL_ADDRESS>School of Physics, Nankai University, Tianjin 300071,
China

###### Abstract

We study the anomalous muon and electron magnetic moments by introducing a
scalar with CP-violating Yukawa couplings to the lepton sector. By fitting
these two magnetic moments with the recent experimental measurements, we find
that such a leptophilic scalar in the mass range of
$\mathcal{O}(10)-\mathcal{O}(1000)\,\rm GeV$ can be a possible source for the
current experimental deviations from the Standard Model (SM) predictions, with
$\mathcal{O}(0.1)-\mathcal{O}(1)$ Yukawa couplings. The current electron and
muon EDM constraints to the general CP-violating Yukawa couplings are
discussed. We propose to search such a leptophilic scalar mediated at the
future high-luminosity LHC (HL-LHC) runs, as well as the high-energy lepton
colliders, including the CEPC and the muon collider. Our results show that the
leptophilic scalar in the mass range of
$\mathcal{O}(10)-\mathcal{O}(1000)\,\rm GeV$ can be fully probed by the future
experimental searches at the HL-LHC and the lepton colliders at their early
stages.

##  I Introduction

Recently, there have been mild deviations from the precise measurements in the
leptonic sector, which include the both muon Bennett  _et al._ (2006) and
electron Hanneke  _et al._ (2008, 2011) anomalous magnetic moments of

$\displaystyle\Delta a_{\mu}$ $\displaystyle\equiv$ $\displaystyle
a_{\mu}^{\rm exp}-a_{\mu}^{\rm SM}=(274\pm 73)\times 10^{-11}\,,$ (1)
$\displaystyle\Delta a_{e}$ $\displaystyle\equiv$ $\displaystyle a_{e}^{\rm
exp}-a_{e}^{\rm SM}=(-87\pm 36)\times 10^{-14}\,.$ (2)

The combined results can be possible indications to the new physics beyond the
Standard Model (BSM) Czarnecki and Marciano (2001). A positive deviation of
the muon $g-2$ from the SM prediction of $a_{\mu}^{\rm
SM}=116~{}591~{}810(43)\times 10^{-11}$ Aoyama  _et al._ (2020) has been a
long-standing topics for phenomenological studies. Meanwhile, the newly
measured electron $g-2$, which is a $2.4\,\sigma$ discrepancy from the SM
prediction with a negative sign, brings additional challenge to the issue. In
the framework of the supersymmetric models, one has the absolute values of
$|\Delta a_{\mu}|\propto m_{\mu}^{2}$ and $|\Delta a_{e}|\propto m_{e}^{2}$.
Therefore, the mismatch of $\Delta a_{\mu}/\Delta a_{e}$ in Eqs. (1) (2) and
$|m_{\mu}/m_{e}|^{2}$ disfavors the explanation from the supersymmetric
particles.

Among the previous efforts where one tries to address the muon $g-2$ anomaly,
it is usually achieved by introducing spin-$0$ and/or spin-$1$ mediators
beyond the SM. The couplings of these mediators to the leptonic states can be
categorized as scalar, pseudo-scalar, vectorial, and axial ones 111The
estimation of the individual contributions to the $g-2$ was made in Ref
Leveille (1978).. It is known that the pseudo-scalar and/or axial couplings
can lead to negative contributions to the leptonic $g-2$, while the scalar
and/or vectorial couplings lead to positive contributions. Therefore, it is
meaningful to ask whether one may accommodate both anomalous magnetic moments
by minimally including spin-$0$ mediator. Through the experience in the SM, or
many versions of BSM new physics constructions, it is a general feature that
the Yukawa couplings between the scalar fields and the SM fermions can be
flavor-dependent.

Most of the previous studies focus on the simplified model where the scalar or
vector mediators are relatively light Chen  _et al._ (2016); Marciano  _et
al._ (2016); Davoudiasl and Marciano (2018); Liu  _et al._ (2019); Bauer  _et
al._ (2020); Kirpichnikov  _et al._ (2020), or with the lepton flavor-
violating couplings Altmannshofer  _et al._ (2016); Bauer  _et al._ (2020);
Dev  _et al._ (2020); Calibbi  _et al._ (2020). In addition, there have been
systematical studies of both anomalies in the “full model”, such as in the
two-Higgs-doublet model Han  _et al._ (2016); Wang  _et al._ (2019); Keus  _et
al._ (2018); Abe  _et al._ (2017); Han  _et al._ (2019); Chun  _et al._
(2019); Jana  _et al._ (2020); Botella  _et al._ (2020); Bian  _et al._
(2020); Li  _et al._ (2021); Delle Rose  _et al._ (2020), the supersymmetric
models Badziak and Sakurai (2019); Endo and Yin (2019), and others Batell  _et
al._ (2017); Cárcamo Hernández  _et al._ (2020); Haba  _et al._ (2020);
Keshavarzi  _et al._ (2020); Bigaran and Volkas (2020); Chen and Nomura
(2021). Note in passing, the recent efforts in addressing the muon and/or
electron $g-2$ anomalies include Refs. Crivellin  _et al._ (2018, 2020);
Colangelo  _et al._ (2021). Meanwhile with the proposals of the future lepton
colliders, such as the circular electron-positron collider (CEPC),
International Linear Collider (ILC), FCC-ee, and the muon colliders, it is
natural to ask whether one can directly look for the corresponding BSM
signals. Although there can be different configurations for the future
$e^{+}e^{-}$ colliders, it is commonly accepted that they will all run at the
center-of-mass energy of $\sqrt{s}=240\,\rm GeV$ or $\sqrt{s}=250\,\rm GeV$
for the precision measurements of the Higgs boson. As for the future muon
colliders, there has been a renewed interest to look for the physics cases
with the center-of-mass energies of $\mathcal{O}(1)-\mathcal{O}(100)\,\rm TeV$
Buttazzo  _et al._ (2018); Delahaye  _et al._ (2019); Costantini  _et al._
(2020); Capdevilla  _et al._ (2020); Han  _et al._ (2020a); Long  _et al._
(2020); Han  _et al._ (2020b, c); Liu and Xie (2021). A muon collider which
can operate at multiple TeV scale thus provides tremendous physical
opportunities beyond that of a Higgs factory operating at several hundred GeV.
Recently, discussions on the direct probes of the anomalous muon $g-2$ effects
at the future muon collider can be also found in Refs. Buttazzo and Paradisi
(2020); Yin and Yamaguchi (2020).

With a CP-violating scalar mediator in the leptonic sector, it is natural to
expect a prediction to the electric dipole moments (EDM) to the charged
leptons. The most recent upper bounds to the electron and muon EDMs were from
the ACME collaboration 222Another order of magnitude improvement to the
electron EDM precision can be expected from the future upgrade. Andreev  _et
al._ (2018) and the muon $g-2$ storage ring at the BNL Bennett  _et al._
(2009) as follows

$\displaystyle|d_{e}|<1.1\times 10^{-29}\,e\cdot{\rm
cm}\,,\qquad|d_{\mu}|<1.9\times 10^{-19}\,e\cdot{\rm cm}\,.$ (3)

The rest of the paper is organized as follows. Our model setup with a CP-
violating scalar coupling to the leptonic sector is described in Sec. II. The
combined fits to both electron and muon $g-2$ in Eqs. (1) and (2) are
performed, with the leptophilic scalar mass in the range of $(10\,,1000)\,\rm
GeV$. The CP-even Yukawa coupling to the muon and the CP-odd Yukawa coupling
to the electron will be used as the benchmark model points for the collider
searches. We also impose the electron and muon EDM constraints in Eq. (3) to
the benchmark models with the general CP-violating Yukawa couplings. In Sec.
III, we study the discovery potentials for the leptophilic scalar mediator at
the LHC, within the mass range of $(10\,,1000)\,\rm GeV$. In Sec. IV, we
further study the discovery potentials for the leptophilic scalar mediator at
the future high-energy lepton colliders, including both the CEPC running at
$\sqrt{s}=240\,\rm GeV$ and the muon collider running at $\sqrt{s}=3\,\rm
TeV$. We summarize the discovery potential of the leptophilic scalar $\phi$ at
the LHC, CEPC, and the muon collider in Sec. V.

##  II The model setup

The minimal model setup involves a CP-violating and leptophilic scalar $\phi$
as follows:

$\displaystyle\mathcal{L}$ $\displaystyle=$
$\displaystyle\frac{1}{2}(\partial\phi)^{2}-\frac{1}{2}m_{\phi}^{2}\phi^{2}-\sum_{\ell=e\,,\mu}\bar{\ell}(\lambda_{\ell}+\tilde{\lambda}_{\ell}i\gamma^{5})\ell\phi\,.$
(4)

The presence of the scalar and pseudo-scalar couplings of the $\phi$ field to
the SM leptons $\ell$ leads to both magnetic moments and electric dipole
moments. In our setup, we assume the CP-violating Yukawa couplings of
$(\lambda_{\ell}\,,\tilde{\lambda}_{\ell})$ to be flavor-dependent. Generally
speaking, such a CP-violating scalar can induce both the electric and magnetic
dipole operators as follows

$\displaystyle\mathcal{L}_{\rm eff}$ $\displaystyle=$
$\displaystyle\Big{(}-\frac{i}{2}d_{\ell}\bar{\ell}\sigma^{\mu\nu}\gamma_{5}\ell+\frac{e}{4m_{\ell}}a_{\ell}\bar{\ell}\sigma^{\mu\nu}\ell\Big{)}F_{\mu\nu}\,,$
(5)

through the one-loop radiative corrections.

###  II.1 The anomalous $g-2$

We estimate the size of the anomalous $g-2$ factor of $\Delta a_{\ell}$
according to the couplings in Eq. (4). The one-loop corrections to the lepton
$g-2$ from the leptophilic scalar read

$\displaystyle\Big{(}\Delta a_{\ell}\Big{)}_{{\rm spin}-0}$ $\displaystyle=$
$\displaystyle\lambda_{\ell}^{2}I_{S}(x)+\tilde{\lambda}_{\ell}^{2}I_{P}(x)\,,$
(6) $\displaystyle I_{S}(x)$ $\displaystyle=$
$\displaystyle\frac{x^{2}}{8\pi^{2}}\int_{0}^{1}dz\,\frac{(1+z)(1-z)^{2}}{x^{2}(1-z)^{2}+z}\,,$
(7) $\displaystyle I_{P}(x)$ $\displaystyle=$
$\displaystyle-\frac{x^{2}}{8\pi^{2}}\int_{0}^{1}dz\,\frac{(1-z)^{3}}{x^{2}(1-z)^{2}+z}\,,$
(8)

with $x\equiv m_{\ell}/m_{\phi}$. The integrals of $I_{S}(x)$ and $I_{P}(x)$
are for the scalar and pseudoscalar couplings respectively, and their values
are opposite. Therefore, one would expect a CP-violating scalar field coupling
to the leptons to be the most economical way. For a leptophilic scalar mass of
$m_{\phi}\sim\mathcal{O}(10)-\mathcal{O}(1000)\,\rm GeV$, the integrals of
$I_{S\,,P}(x)$ are in the range of
$\sim\pm\mathcal{O}(10^{-4})-\mathcal{O}(10^{-7})$ for muons and
$\sim\pm\mathcal{O}(10^{-8})-\mathcal{O}(10^{-12})$ for electrons,
respectively. The absolute values of both integrals vanish with the increasing
mediator mass of $m_{\phi}$.

Figure 1:  The $1\,\sigma$ region for the one-loop leptophilic scalar
contributions to the muon (left panel) and electron (right panel) $g-2$ in the
$(m_{\phi}\,,\lambda_{\mu})$ and $(m_{\phi}\,,\tilde{\lambda}_{e})$ planes.

To display the one-loop contributions to the muon and electron $g-2$, we vary
the scalar mediator mass in the range of $m_{\phi}\in(10\,,1000)\,\rm GeV$,
and take $\tilde{\lambda}_{\mu}=\lambda_{e}=0$ for simplicity. By only
assuming non-vanishing $\lambda_{\mu}$ and $\tilde{\lambda}_{e}$ in the
effective coupling of Eq. (4), we demonstrate the $1\,\sigma$ region to
accommodate the current experimental results of the muon and electron $g-2$ as
listed in Eqs. (1) and (2) in Fig. 1. With the leptophilic scalar mass
increasing from $10\,\rm GeV$ to $1000\,\rm GeV$, the corresponding Yukawa
couplings of $(\lambda_{\mu}\,,\tilde{\lambda}_{e})$ can be enhanced from
$\mathcal{O}(0.01)$ to $\mathcal{O}(1)$. This implies that for a scalar
mediator in such a mass range, the enhancements of the Yukawa couplings bring
it possible to probe the scalar mediator at the LHC and the future lepton
colliders, such as the CEPC and the muon colliders.

###  II.2 The EDM constraints

In order to accommodate the current $g-2$ measurements, we set
$\tilde{\lambda}_{\mu}=\lambda_{e}=0$ in Fig. 1 for simplicity. In turn, the
simultaneous existences of the CP-violating Yukawa couplings lead to the non-
vanishing predictions of the lepton EDMs. The one-loop contribution to the
lepton EDM is given by

$\displaystyle d_{\ell}$ $\displaystyle=$
$\displaystyle\frac{\lambda_{\ell}\tilde{\lambda}_{\ell}}{8\pi^{2}}\frac{e}{m_{\ell}}x^{2}\int_{0}^{1}dz\frac{(1-z)^{2}}{x^{2}(1-z)^{2}+z}\,.$
(9)

Note that it is also possible to induce the lepton EDMs via the two-loop Barr-
Zee diagrams. In practice, we find their contributions to be
$\sim\mathcal{O}(10^{-12})$ smaller than the one-loop contribution in Eq. (9).

Figure 2:  The bounds to the electron (left panel) and muon (right panel) EDMs
in the $(m_{\phi}\,,\lambda_{e}\tilde{\lambda}_{e})$ and
$(m_{\phi}\,,\lambda_{\mu}\tilde{\lambda}_{\mu})$ planes. The shaded regions
represent the electron and muon EDM exclusions according to Eq. (3).

According to the one-loop EDM result in Eq. (9), an upper bound to the lepton
EDM imposes constraint to the parameter of
$\lambda_{\ell}\tilde{\lambda}_{\ell}$. In Fig. 2, we display the electron and
muon EDM constraints in the $(m_{\phi}\,,\lambda_{e}\tilde{\lambda}_{e})$ and
$(m_{\phi}\,,\lambda_{\mu}\tilde{\lambda}_{\mu})$ planes. In the left panel of
Fig. 2, we display three different curves with fixed values of
$\lambda_{e}=(10^{-6}\,,10^{-8}\,,10^{-10})$, and together with the values of
$(m_{\phi}\,,\tilde{\lambda}_{e})$ from the benchmark models as in Fig. 1. For
the leptophilic scalar in the mass range of $(10\,,1000)\,\rm GeV$, the CP-
even Yukawa couplings of $\lambda_{e}\gtrsim 10^{-6}$ have been ruled out by
the electron EDM bound. In turn, this justifies our previous simplification of
$\lambda_{e}=0$ in fitting the electron $g-2$. Similarly, in the right panel
of Fig. 2 we display three different curves with fixed values of
$\tilde{\lambda}_{\mu}=(10\,,1\,,0.1)$, and together with the values of
$(m_{\phi}\,,\lambda_{\mu})$ from the benchmark models as in Fig. 1. With the
CP-odd Yukawa coupling of $\tilde{\lambda}_{\mu}$ in the range of
$(0.1\,,10)$, they can lead to negative contributions to the muon $g-2$. Since
the muon EDM bound is far less stringent comparing to the electron EDM, such
parameter ranges cannot be directly excluded. Thus, we assume that
$\tilde{\lambda}_{\mu}\ll\lambda_{\mu}$ in the collider searches to the
benchmark models that accommodate the current muon $g-2$ below.

##  III The LHC searches for the leptophilic mediator

###  III.1 The signal and the background processes at the LHC

In this section, we study the future collider searches for the leptophilic
scalar $\phi$ at the LHC. With the center-of-mass energy of $\sqrt{s}=14\,\rm
TeV$, and an integrated luminosity of $\mathcal{L}_{\rm int}=3000\,\rm
fb^{-1}$, we study the signal processes of

$\displaystyle
q\bar{q}\to\gamma^{*}/Z^{*}\to\ell^{+}\ell^{-}\phi(\to\ell^{+}\ell^{-})\,,$
(10)

at the parton level. The corresponding cross sections at the LHC $14\,\rm TeV$
runs are displayed in Fig. 3 for different final states of
$(4e\,,2e2\mu\,,4\mu)$, with $m_{\phi}\in(10\,,1000)\,\rm GeV$. Among these
three different channels, we find that the multiple leptonic final states of
$4e$ can have sizable cross sections of $\sim\mathcal{O}(1)\,\rm fb$ with
$m_{\phi}\sim\mathcal{O}(10)\,\rm GeV$. While for the $2e2\mu$ and the $4\mu$
final states, their cross sections are
$\sim\mathcal{O}(10^{-1})-\mathcal{O}(10^{-3})\,\rm fb$ and
$\sim\mathcal{O}(10^{-2})-\mathcal{O}(10^{-4})\,\rm fb$, respectively. Hence,
we focus on the $4e$ final states that are most promising at the LHC searches.

Figure 3:  The signal cross sections of the leptophilic scalar $\phi$ for
different final states of $(4e\,,4\mu\,,2e2\mu)$ at the LHC $14\,\rm TeV$ run.

The corresponding irreducible SM background processes include Sirunyan  _et
al._ (2019)

$\displaystyle q\bar{q}\to e^{+}e^{-}e^{+}e^{-}\,,\quad gg\to
e^{+}e^{-}e^{+}e^{-}\,.$ (11)

It turns out that the cross sections from the $gg\to 4e$ contributions are
$\sim\mathcal{O}(1)\,\%$ of the $q\bar{q}\to 4e$, and we have the cross
section of $\sigma(q\bar{q}\to e^{+}e^{-}e^{+}e^{-})=6.63\,\rm fb$ at the LHC
$14\,\rm TeV$ run. Given that the fake rate of muons to electrons can be as
low as $\sim\mathcal{O}(10^{-7})$ at the ATLAS detector Aad  _et al._ (2014),
we can safely ignore the possible contaminations from the SM background events
with the $2e2\mu$ and/or $4\mu$ final states.

###  III.2 The event generation and selections

In practice, we generate the signal and background events at the parton level
by Madgraph Alwall  _et al._ (2011). Sequentially, they are passed to Pythia8
Sjostrand  _et al._ (2008) for the parton showering and hadronizations, and to
Delphes de Favereau  _et al._ (2014) for the fast detector simulations. The
events containing four final-state electrons with two opposite-sign pairs are
selected. Afterwards, four invariant masses from the opposite-sign same-flavor
(OSSF) pairs are reconstructed. Among them, the one whose invariant mass of
$M_{ee}$ is mostly close to the scalar mediator mass of $m_{\phi}$ is
selected. The significances of $S/\sqrt{B}$ are obtained by imposing the
invariant mass cut of $|M_{ee}-m_{\phi}|<0.02\,m_{\phi}$.

###  III.3 The LHC search results

Figure 4:  The invariant mass distributions of $M_{ee}$ for both signals and
the SM background events at the LHC $14\,\rm TeV$ run, with
$m_{\phi}=(20\,,40\,,60)\,\rm GeV$.

We display the invariant mass distributions of $M_{ee}$ in Fig. 4, with
$M_{ee}$ reconstructed from four opposite-sign electrons. Since the invariant
mass of $M_{ee}$ is mostly close to the leptophilic scalar mass of $m_{\phi}$,
the distributions of the corresponding SM background events vary accordingly
for different cases. Here, three different samples with
$m_{\phi}=(20\,,40\,,60)\,\rm GeV$ are displayed. One can expect that for the
scalar mediator mass close to the $m_{Z}$, the corresponding distributions of
$M_{ee}$ suffer from large SM background events. Thus, we do not expect the
scalar mediator with mass of $m_{\phi}\sim 90\,\rm GeV$ to be discovered at
the LHC. The expected numbers of signal events with integrated luminosity of
$3000\,\rm fb^{-1}$ at LHC $14\,\rm TeV$ run are given in Tab. 1 for three
benchmark models with $m_{\phi}=(20,40,60)\,\rm GeV$. As expected, the SM
background is effectively reduced by the mass window cut applied on the
reconstructed leptophilic scalar. Furthermore, we can see clearly that the
significances of $S/\sqrt{B}$ drops rapidly when $m_{\phi}$ get close to
$m_{Z}$. The integrated luminosities for discovering the leptophilic scalar
$\phi$ are to be presented in Fig. 9. It turns out that such leptophilic
scalar can be discovered at the HL-LHC run for $m_{\phi}\in[10\,,200]\,\rm
GeV$, except for the $Z$-pole region.

| $m_{\phi}=20~{}\mathrm{GeV}$ | $m_{\phi}=40~{}\mathrm{GeV}$ | $m_{\phi}=60~{}\mathrm{GeV}$
---|---|---|---
| Sig. | Bkgd. | $S/\sqrt{B}$ | Sig. | Bkgd. | $S/\sqrt{B}$ | Sig. | Bkgd. | $S/\sqrt{B}$
Select OSSF $4e$ | 367.18 | 3690.68 | 6.04 | 345.20 | 3690.68 | 5.68 | 200.83 | 3690.68 | 3.31
$|M_{ee}-m_{\phi}|<0.02\,m_{\phi}$ | 216.40 | 34.23 | 36.99 | 216.87 | 117.82 | 19.98 | 133.35 | 173.15 | 10.13
Table 1:  The expected number of events for three benchmark models with
$m_{\phi}=(20,40,60)~{}\rm GeV$ and the corresponding SM background at the LHC
14 TeV run, with the integrated luminosity of $3000\,\rm fb^{-1}$.

##  IV The searches for the leptophilic mediator at the lepton colliders

###  IV.1 The signal and background processes at the lepton colliders

Figure 5:  The Feynman diagrams for the signal (top) and background (bottom)
processes at the CEPC and the muon collider runs.

In this section, we proceed to study the future collider searches for the
leptophilic scalar $\phi$ at the lepton colliders. We choose the
configurations of the CEPC run at $\sqrt{s}=240\,\rm GeV$ and the muon
collider run at $\sqrt{s}=3\,\rm TeV$. The following inclusive productions of

$\displaystyle
e^{+}e^{-}/\mu^{+}\mu^{-}\to\phi\ell^{+}\ell^{-}\to\ell^{+}\ell^{-}\ell^{+}\ell^{-}\,,$
(12)

for the leptophilic scalar $\phi$ will be studied. The corresponding SM
background processes are

$\displaystyle
e^{+}e^{-}/\mu^{+}\mu^{-}\to\ell^{+}\ell^{-}(\gamma^{*}/Z)\to\ell^{+}\ell^{-}\ell^{+}\ell^{-}\,,$
(13)

The related Feynman diagrams are depicted in Fig. 5. For the sake of
simplicity, we only demonstrated few typical diagrams, and the remaining
signal or background diagrams can be obtained by attaching the leptophilic
scalar $\phi$ or $\gamma^{*}/Z$ to three other leptons.

Figure 6:  The inclusive production cross sections of the leptophilic scalar
$\phi$ at the CEPC $240\,\rm GeV$ run (left panel) and the muon collider
$3\,\rm TeV$ run (right panel), for three different final states of
$(4e\,,4\mu\,,2e2\mu)$.

At both the CEPC and the muon collider, the final states of both signal and
background events can be categorized into: (i) $4e$, (ii) $4\mu$, and (iii)
$2e2\mu$. The inclusive production cross sections at the CEPC and the muon
collider runs are plotted in Fig. 6 for three different leptonic final states.
At the CEPC $240\,\rm GeV$ run, the signal cross sections increase from the
low-mass range and are peaked around $\sim\mathcal{O}(100)\,\rm fb$ for the
$4e/2e2\mu$ final states with $m_{\phi}\sim 100-200\,\rm GeV$, since the
leptophilic scalars are on-shell produced. For the leptophilic scalar in the
heavy-mass range, the signal cross sections decrease to
$\sim\mathcal{O}(1)\,\rm fb$ with $m_{\phi}\sim 1000\,\rm GeV$ for the
$4e/2e2\mu$ final states. The cross sections for the $4\mu$ final states are
smaller compared to the $4e/2e2\mu$ cases by an order of magnitude. At the
muon collider $3\,\rm TeV$ run, the leptophilic scalar $\phi$ is always on-
shell produced. The production cross sections for three different leptonic
final states all increase with the large leptophilic scalar masses of
$m_{\phi}$, and can be $\sim\mathcal{O}(10)-\mathcal{O}(100)\,\rm fb$ when
$m_{\phi}\sim 1000\,\rm GeV$. Accordingly, one expects that the muon collider
will have good search sensitivities for the leptophilic scalar $\phi$ at the
high-mass ranges.

Below, we carry out the analysis of the leptophilic scalar searches at both
the CEPC and the muon collider. The signal and background events are generated
at various levels in the same procedures as those in the LHC searches.
According to the production cross sections, we shall separate the search
strategies for the low-mass ranges of $m_{\phi}\in[50\,,200]\,\rm GeV$ at the
CEPC, and for the high-mass ranges of $m_{\phi}\in[200\,,1000]\,\rm GeV$ at
the muon collider. Afterwards, the events with two OSSF lepton pairs are
selected for each category. With four selected leptons, we record their
kinematic variables of $(p_{T}\,,\eta\,,\varphi)$ for the reconstructions of
the leptophilic scalar. For the $4e$ and/or $4\mu$ final states, we also order
the final-state electrons and/or muons as $(e_{1}^{\pm}\,,e_{2}^{\pm})$ and/or
$(\mu_{1}^{\pm}\,,\mu_{2}^{\pm})$ according to their $p_{T}$.

###  IV.2 The CEPC searches

Figure 7:  The distributions of: $\eta(e)$ (bottom left), $p_{T}(\phi)$
(bottom right), and the invariant mass $M_{ee}$ (top) for both the signal and
the SM background events from the $4e$ final states at the CEPC $240\,\rm GeV$
run. The $\eta(e)$ distributions are plotted for the $m_{\phi}=100\,\rm GeV$
case, while the $p_{T}(\phi)$ and the $M_{ee}$ distributions are plotted for
three different cases of $m_{\phi}=(50\,,100\,,200)\,\rm GeV$.

At the CEPC $240\,\rm GeV$ run, one only expects direct $\phi$ resonance
productions for $m_{\phi}\lesssim 240\,\rm GeV$. For the low-mass resonances
with $m_{\phi}\in[50\,,200]\,\rm GeV$, we obtain two/four invariant masses
from the OSSF lepton pairs for the $2e2\mu$/$(4e\,,4\mu)$ final states. Among
them, the one whose invariant mass of $M_{\ell\ell}(\ell=e\,,\mu)$ is mostly
close to the scalar mediator mass of $m_{\phi}$ is selected. One can expect
this to be the lepton pair from the on-shell productions of the leptophilic
scalar, and its $p_{T}$ and the invariant mass distribution will be used as
our kinematic variables to separate the signal events from the SM background
events.

In Fig. 7, we display the event fraction 333The event fraction is defined such
that the summation of the signal and the SM background event fraction is
unity. distributions of $(\eta(e)\,,p_{T}(\phi)\,,M_{ee})$ for different
samples of $m_{\phi}=(50\,,100\,,200)\,\rm GeV$ from the $4e$ final states.
For both the $4e$ and $2e2\mu$ final states, one finds relatively larger
values of $|\eta|$ distributions for the final-state electrons in the SM
background events. This is mostly due to the back-to-back productions of the
opposite-sign electrons in the background events, as depicted in the last
diagram in Fig. 5. Accordingly, the cuts we choose for the low-mass resonances
from the $4e$ and $2e2\mu$ final states are the following

  * •

$m_{\phi}\in[50,100]\,\rm GeV$:
$|M_{\ell\ell}-m_{\phi}|<0.02~{}m_{\phi}\,,~{}p_{T}(\phi)>60~{}\text{GeV},~{}\eta(e^{-})>-1,~{}\eta(e^{+})<1$,

  * •

$m_{\phi}=200$ GeV: $|M_{\ell\ell}-m_{\phi}|<0.02~{}m_{\phi},\quad
p_{T}(\phi)<60~{}\text{GeV},\quad\eta(e^{-})>-1,\quad\eta(e^{+})<1$.

The expected numbers of signal events with integrated luminosity of $1\,\rm
fb^{-1}$ at the CEPC $240\,\rm GeV$ run are given in Tab. 2 for three
benchmark models with $m_{\phi}=(50,100,200)\,\rm GeV$, respectively. We can
see that the background events are effectively suppressed by above cuts. For
the $4\mu$ final states, the cuts to the $M_{\ell\ell}$ and $p_{T}(\phi)$
similarly apply. However, we do not impose the cuts to the $\eta$ of the
final-state muons, since there are no back-to-back productions of muons at the
CEPC run. The final searches sensitivities of the leptophilic scalar $\phi$
via three different final states will be projected in Fig. 9.

| $m_{\phi}=50~{}\mathrm{GeV}$ | $m_{\phi}=100~{}\mathrm{GeV}$ | $m_{\phi}=200~{}\mathrm{GeV}$
---|---|---|---
| Sig. | Bkgd. | $S/\sqrt{B}$ | Sig. | Bkgd. | $S/\sqrt{B}$ | Sig. | Bkgd. | $S/\sqrt{B}$
Select OSSF $4e$ | 32.03 | 4.88 | 14.50 | 61.34 | 4.88 | 27.77 | 37.91 | 4.88 | 17.16
Kinematic Cuts | 7.31 | 0.014 | 61.89 | 14.90 | 0.025 | 93.49 | 15.79 | 0.019 | 115.36
Table 2:  The expected number of events for three benchmark models with
$m_{\phi}=(50,100,200)~{}\rm GeV$ and the corresponding SM background at the
CEPC 240 GeV run, with the integrated luminosity of $1\,\rm fb^{-1}$.

###  IV.3 The muon collider searches

Figure 8:  The distributions of: $\eta(\mu)$ (left panel) and the invariant
mass $M_{\mu\mu}$ (right panel) for both the signal and the SM background
events from the $4e$ final states at the muon collider $3\,\rm TeV$ run. The
$\eta(\mu)$ distributions are plotted for the $m_{\phi}=600\,\rm GeV$ case,
while the $M_{\mu\mu}$ distributions are plotted for three different cases of
$m_{\phi}=(300\,,600\,,900)\,\rm GeV$.

Next, we proceed to analyze the searches for the leptophilic scalar $\phi$ at
the muon collider $3\,\rm TeV$ run. The event selection criteria follow from
those in the CEPC case. At the muon collider, two/four invariant masses from
the OSSF lepton pairs for the $2e2\mu$/$(4e\,,4\mu)$ final states can be
similarly obtained. Afterwards, we select the one whose invariant mass of
$M_{\ell\ell}\,(\ell=e\,,\mu)$ mostly close to the leptophilic scalar mass of
$m_{\phi}$. In Fig. 8, we display the event fraction distributions of
$\eta(\mu)$ for $m_{\phi}=600\,\rm GeV$, and $M_{\mu\mu}$ for
$m_{\phi}=(300\,,600\,,900)\,\rm GeV$ from the $4\mu$ final states. For both
the $4\mu$ and $2e2\mu$ final states at the muon collider, one finds
relatively large values of $|\eta|$ distributions for the final-state muons in
the SM background events. At the muon collider here, there can be back-to-back
productions of the opposite-sign muons in the background events. Such a
feature does not exist for the $4e$ final states at the muon collider.
However, we find relatively small values of $p_{T}$ distributions for the
final-state $e_{1}^{-}$ and $e_{1}^{+}$ in the $4e$ signal events.
Accordingly, we choose the following cuts for the high-mass resonances from
the final states

  * •

$4\mu$ and $2e2\mu$:
$|M_{\ell\ell}-m_{\phi}|<0.1\,m_{\phi},\quad\eta(\mu_{1}^{-})>0,\quad\eta(\mu_{1}^{+})<0$,

  * •

$4e$: $|M_{ee}-m_{\phi}|<0.1\,m_{\phi},\quad p_{T}(e_{1}^{\pm})<900\,{\rm
GeV}$.

The expected numbers of signal events with integrated luminosity of $10\,\rm
fb^{-1}$ at muon collider $3\,\rm TeV$ run are given in Tab. 3 for three
benchmark models with $m_{\phi}=(300,600,900)\,\rm GeV$. We can see that the
background events are effectively suppressed by above cuts. The final search
sensitivities of the leptophilic scalar $\phi$ via three different final
states will be projected in Fig. 9.

| $m_{\phi}=300~{}\mathrm{GeV}$ | $m_{\phi}=600~{}\mathrm{GeV}$ | $m_{\phi}=900~{}\mathrm{GeV}$
---|---|---|---
| Sig. | Bkgd. | $S/\sqrt{B}$ | Sig. | Bkgd. | $S/\sqrt{B}$ | Sig. | Bkgd. | $S/\sqrt{B}$
Select OSSF $4\mu$ | 14.80 | 4.59 | 6.91 | 71.78 | 4.59 | 33.51 | 210.41 | 4.59 | 98.24
Kinematic Cuts | 3.08 | 0.011 | 29.31 | 8.97 | 0.018 | 66.25 | 19.65 | 0.024 | 127.57
Table 3:  The expected number of events for three benchmark models with
$m_{\phi}=(300,600,900)~{}\rm GeV$ and the corresponding SM background at the
muon collider 3 TeV run, with the integrated luminosity of $10\,\rm fb^{-1}$.

##  V Conclusion and Discussion

Figure 9:  The combined discovery potentials of the leptophilic scalar $\phi$
at the LHC $14\,\rm TeV$ run (brown solid curve), CEPC $240\,\rm GeV$ run
(solid curves), and the muon collider $3\,\rm TeV$ run (dashed curves).

In this work, we suggest a leptophilic scalar with CP-violating Yukawa
couplings so that the current discrepancies of the electron and muon $g-2$ can
be accommodated. In comparison to the previous efforts, we find that the
reasonable Yukawa couplings can be $\sim\mathcal{O}(0.1)-\mathcal{O}(1)$ when
the scalar mass is $\sim\mathcal{O}(10)-\mathcal{O}(1000)\,\rm GeV$. With the
sizable Yukawa couplings in this mass range, it is expected that such a
leptophilic scalar can be searched for at the current LHC experiments, as well
as the future high-energy lepton colliders, such as the CEPC and the muon
colliders running at the $\sqrt{s}\sim\mathcal{O}(1)\,\rm TeV$. The related
collider search strategy can be realized straightforwardly by constructing the
OSSF final-state lepton pairs and impose the invariant mass cuts accordingly.

Our final results of searching for such a leptophilic scalar $\phi$ are
summarized in Fig. 9 at three different collider runs. The benchmark models
with the leptophilic scalar $\phi$ in the mass range of $(10\,,1000)\,\rm GeV$
have been studied, where the integrated luminosities needed for the
$5\,\sigma$ discovery at the LHC $14\,\rm TeV$ run, the CEPC $240\,\rm GeV$
run, and the muon collider $3\,\rm TeV$ run are displayed. We find that the
HL-LHC run with the integrated luminosity of $3000\,\rm fb^{-1}$ can probe the
leptophilic scalar $\phi$ in the mass range of $[10\,,200]\,\rm GeV$ via the
$4e$ final state, except for the $Z$-pole region. The CEPC run can probe such
a leptophilic scalar via all three $(4e\,,4\mu\,,2e2\mu)$ final states with
integrated luminosities of $\sim\mathcal{O}(10^{-3})-\mathcal{O}(10^{-1})\,\rm
fb$, including the $Z$-pole region. For the leptophilic scalar in the high-
mass range of $[200\,,1000]\,\rm GeV$, we find that the muon collider run can
fully probe the benchmark models at early stage with integrated luminosities
of $\sim\mathcal{O}(10^{-4})-\mathcal{O}(1)\,\rm fb^{-1}$.

## ACKNOWLEDGMENTS

We would like to thank Yanwen Liu for very useful discussions and
communication during the preparation of this work. NC and BW are partially
supported by the National Natural Science Foundation of China (under Grant No.
11575176 and 12035008). CYY is supported in part by the National Natural
Science Foundation of China (under Grant No. 11975130, 12035008, 12047533), by
the National Key Research and Development Program of China (under Grant No.
2017YFA0402200), and the China Post-doctoral Science Foundation (under Grant
No. 2018M641621).

## References

  * Bennett  _et al._ (2006) G. Bennett _et al._ (Muon g-2), Phys. Rev. D 73, 072003 (2006), arXiv:hep-ex/0602035 .
  * Hanneke  _et al._ (2008) D. Hanneke, S. Fogwell, and G. Gabrielse, Phys. Rev. Lett. 100, 120801 (2008), arXiv:0801.1134 [physics.atom-ph] .
  * Hanneke  _et al._ (2011) D. Hanneke, S. Hoogerheide, and G. Gabrielse, Phys. Rev. A 83, 052122 (2011), arXiv:1009.4831 [physics.atom-ph] .
  * Czarnecki and Marciano (2001) A. Czarnecki and W. J. Marciano, Phys. Rev. D 64, 013014 (2001), arXiv:hep-ph/0102122 .
  * Aoyama  _et al._ (2020) T. Aoyama _et al._ , Phys. Rept. 887, 1 (2020), arXiv:2006.04822 [hep-ph] .
  * Leveille (1978) J. P. Leveille, Nucl. Phys. B 137, 63 (1978).
  * Chen  _et al._ (2016) C.-Y. Chen, H. Davoudiasl, W. J. Marciano, and C. Zhang, Phys. Rev. D 93, 035006 (2016), arXiv:1511.04715 [hep-ph] .
  * Marciano  _et al._ (2016) W. Marciano, A. Masiero, P. Paradisi, and M. Passera, Phys. Rev. D 94, 115033 (2016), arXiv:1607.01022 [hep-ph] .
  * Davoudiasl and Marciano (2018) H. Davoudiasl and W. J. Marciano, Phys. Rev. D 98, 075011 (2018), arXiv:1806.10252 [hep-ph] .
  * Liu  _et al._ (2019) J. Liu, C. E. Wagner, and X.-P. Wang, JHEP 03, 008 (2019), arXiv:1810.11028 [hep-ph] .
  * Bauer  _et al._ (2020) M. Bauer, M. Neubert, S. Renner, M. Schnubel, and A. Thamm, Phys. Rev. Lett. 124, 211803 (2020), arXiv:1908.00008 [hep-ph] .
  * Kirpichnikov  _et al._ (2020) D. Kirpichnikov, V. E. Lyubovitskij, and A. S. Zhevlakov, Phys. Rev. D 102, 095024 (2020), arXiv:2002.07496 [hep-ph] .
  * Altmannshofer  _et al._ (2016) W. Altmannshofer, C.-Y. Chen, P. Bhupal Dev, and A. Soni, Phys. Lett. B 762, 389 (2016), arXiv:1607.06832 [hep-ph] .
  * Dev  _et al._ (2020) P. B. Dev, W. Rodejohann, X.-J. Xu, and Y. Zhang, JHEP 05, 053 (2020), arXiv:2002.04822 [hep-ph] .
  * Calibbi  _et al._ (2020) L. Calibbi, M. López-Ibáñez, A. Melis, and O. Vives, JHEP 06, 087 (2020), arXiv:2003.06633 [hep-ph] .
  * Han  _et al._ (2016) T. Han, S. K. Kang, and J. Sayre, JHEP 02, 097 (2016), arXiv:1511.05162 [hep-ph] .
  * Wang  _et al._ (2019) L. Wang, J. M. Yang, M. Zhang, and Y. Zhang, Phys. Lett. B 788, 519 (2019), arXiv:1809.05857 [hep-ph] .
  * Keus  _et al._ (2018) V. Keus, N. Koivunen, and K. Tuominen, JHEP 09, 059 (2018), arXiv:1712.09613 [hep-ph] .
  * Abe  _et al._ (2017) T. Abe, R. Sato, and K. Yagyu, JHEP 07, 012 (2017), arXiv:1705.01469 [hep-ph] .
  * Han  _et al._ (2019) X.-F. Han, T. Li, L. Wang, and Y. Zhang, Phys. Rev. D 99, 095034 (2019), arXiv:1812.02449 [hep-ph] .
  * Chun  _et al._ (2019) E. J. Chun, J. Kim, and T. Mondal, JHEP 12, 068 (2019), arXiv:1906.00612 [hep-ph] .
  * Jana  _et al._ (2020) S. Jana, V. P. K., and S. Saad, Phys. Rev. D 101, 115037 (2020), arXiv:2003.03386 [hep-ph] .
  * Botella  _et al._ (2020) F. J. Botella, F. Cornet-Gomez, and M. Nebot, Phys. Rev. D 102, 035023 (2020), arXiv:2006.01934 [hep-ph] .
  * Bian  _et al._ (2020) L. Bian, H. M. Lee, and C. B. Park, (2020), arXiv:2008.03629 [hep-ph] .
  * Li  _et al._ (2021) S.-P. Li, X.-Q. Li, Y.-Y. Li, Y.-D. Yang, and X. Zhang, JHEP 01, 034 (2021), arXiv:2010.02799 [hep-ph] .
  * Delle Rose  _et al._ (2020) L. Delle Rose, S. Khalil, and S. Moretti, (2020), arXiv:2012.06911 [hep-ph] .
  * Badziak and Sakurai (2019) M. Badziak and K. Sakurai, JHEP 10, 024 (2019), arXiv:1908.03607 [hep-ph] .
  * Endo and Yin (2019) M. Endo and W. Yin, JHEP 08, 122 (2019), arXiv:1906.08768 [hep-ph] .
  * Batell  _et al._ (2017) B. Batell, N. Lange, D. McKeen, M. Pospelov, and A. Ritz, Phys. Rev. D 95, 075003 (2017), arXiv:1606.04943 [hep-ph] .
  * Cárcamo Hernández  _et al._ (2020) A. E. Cárcamo Hernández, Y. Hidalgo Velásquez, S. Kovalenko, H. N. Long, N. A. Pérez-Julve, and V. V. Vien, (2020), arXiv:2002.07347 [hep-ph] .
  * Haba  _et al._ (2020) N. Haba, Y. Shimizu, and T. Yamada, PTEP 2020, 093B05 (2020), arXiv:2002.10230 [hep-ph] .
  * Keshavarzi  _et al._ (2020) A. Keshavarzi, W. J. Marciano, M. Passera, and A. Sirlin, Phys. Rev. D 102, 033002 (2020), arXiv:2006.12666 [hep-ph] .
  * Bigaran and Volkas (2020) I. Bigaran and R. R. Volkas, Phys. Rev. D 102, 075037 (2020), arXiv:2002.12544 [hep-ph] .
  * Chen and Nomura (2021) C.-H. Chen and T. Nomura, Nucl. Phys. B 964, 115314 (2021), arXiv:2003.07638 [hep-ph] .
  * Crivellin  _et al._ (2018) A. Crivellin, M. Hoferichter, and P. Schmidt-Wellenburg, Phys. Rev. D 98, 113002 (2018), arXiv:1807.11484 [hep-ph] .
  * Crivellin  _et al._ (2020) A. Crivellin, M. Hoferichter, C. A. Manzari, and M. Montull, Phys. Rev. Lett. 125, 091801 (2020), arXiv:2003.04886 [hep-ph] .
  * Colangelo  _et al._ (2021) G. Colangelo, M. Hoferichter, and P. Stoffer, Phys. Lett. B 814, 136073 (2021), arXiv:2010.07943 [hep-ph] .
  * Buttazzo  _et al._ (2018) D. Buttazzo, D. Redigolo, F. Sala, and A. Tesi, JHEP 11, 144 (2018), arXiv:1807.04743 [hep-ph] .
  * Delahaye  _et al._ (2019) J. P. Delahaye, M. Diemoz, K. Long, B. Mansoulié, N. Pastrone, L. Rivkin, D. Schulte, A. Skrinsky, and A. Wulzer, (2019), arXiv:1901.06150 [physics.acc-ph] .
  * Costantini  _et al._ (2020) A. Costantini, F. De Lillo, F. Maltoni, L. Mantani, O. Mattelaer, R. Ruiz, and X. Zhao, JHEP 09, 080 (2020), arXiv:2005.10289 [hep-ph] .
  * Capdevilla  _et al._ (2020) R. Capdevilla, D. Curtin, Y. Kahn, and G. Krnjaic, (2020), arXiv:2006.16277 [hep-ph] .
  * Han  _et al._ (2020a) T. Han, Y. Ma, and K. Xie, (2020a), arXiv:2007.14300 [hep-ph] .
  * Long  _et al._ (2020) K. Long, D. Lucchesi, M. Palmer, N. Pastrone, D. Schulte, and V. Shiltsev, (2020), arXiv:2007.15684 [physics.acc-ph] .
  * Han  _et al._ (2020b) T. Han, D. Liu, I. Low, and X. Wang, (2020b), arXiv:2008.12204 [hep-ph] .
  * Han  _et al._ (2020c) T. Han, Z. Liu, L.-T. Wang, and X. Wang, (2020c), arXiv:2009.11287 [hep-ph] .
  * Liu and Xie (2021) W. Liu and K.-P. Xie, (2021), arXiv:2101.10469 [hep-ph] .
  * Buttazzo and Paradisi (2020) D. Buttazzo and P. Paradisi, (2020), arXiv:2012.02769 [hep-ph] .
  * Yin and Yamaguchi (2020) W. Yin and M. Yamaguchi, (2020), arXiv:2012.03928 [hep-ph] .
  * Andreev  _et al._ (2018) V. Andreev _et al._ (ACME), Nature 562, 355 (2018).
  * Bennett  _et al._ (2009) G. W. Bennett _et al._ (Muon (g-2)), Phys. Rev. D 80, 052008 (2009), arXiv:0811.1207 [hep-ex] .
  * Sirunyan  _et al._ (2019) A. M. Sirunyan _et al._ (CMS), Phys. Lett. B 792, 345 (2019), arXiv:1808.03684 [hep-ex] .
  * Aad  _et al._ (2014) G. Aad _et al._ (ATLAS), Phys. Rev. D 90, 072010 (2014), arXiv:1408.5774 [hep-ex] .
  * Alwall  _et al._ (2011) J. Alwall, M. Herquet, F. Maltoni, O. Mattelaer, and T. Stelzer, JHEP 06, 128 (2011), arXiv:1106.0522 [hep-ph] .
  * Sjostrand  _et al._ (2008) T. Sjostrand, S. Mrenna, and P. Z. Skands, Comput. Phys. Commun. 178, 852 (2008), arXiv:0710.3820 [hep-ph] .
  * de Favereau  _et al._ (2014) J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi (DELPHES 3), JHEP 02, 057 (2014), arXiv:1307.6346 [hep-ex] .