# GeneGPT: Augmenting Large Language Models with Domain Tools for Improved
Access to Biomedical Information

Qiao Jin♣, Yifan Yang♣♡, Qingyu Chen♣, Zhiyong Lu♣
♣ National Library of Medicine, National Institutes of Health
♡ University of Maryland, College Park
{qiao.jin, yifan.yang3, qingyu.chen<EMAIL_ADDRESS>
###### Abstract

While large language models (LLMs) have been successfully applied to various
tasks, they still face challenges with hallucinations. Augmenting LLMs with
domain-specific tools such as database utilities can facilitate easier and
more precise access to specialized knowledge. In this paper, we present
GeneGPT, a novel method for teaching LLMs to use the Web APIs of the National
Center for Biotechnology Information (NCBI) for answering genomics questions.
Specifically, we prompt Codex to solve the GeneTuring tests with NCBI Web APIs
by in-context learning and an augmented decoding algorithm that can detect and
execute API calls. Experimental results show that GeneGPT achieves state-of-
the-art performance on eight tasks in the GeneTuring benchmark with an average
score of 0.83, largely surpassing retrieval-augmented LLMs such as the new
Bing (0.44), biomedical LLMs such as BioMedLM (0.08) and BioGPT (0.04), as
well as GPT-3 (0.16) and ChatGPT (0.12). Our further analyses suggest that:
(1) API demonstrations have good cross-task generalizability and are more
useful than documentations for in-context learning; (2) GeneGPT can generalize
to longer chains of API calls and answer multi-hop questions in GeneHop, a
novel dataset introduced in this work; (3) Different types of errors are
enriched in different tasks, providing valuable insights for future
improvements.

##  1 Introduction

Large language models (LLMs) such as PaLM Chowdhery et al. (2022) and GPT-4
OpenAI (2023) have shown great success on a wide range of general-domain
Natural Language Processing (NLP) tasks. They also achieve state-of-the-art
(SOTA) performance on domain-specific tasks like biomedical question answering
Singhal et al. (2022); Liévin et al. (2022); Nori et al. (2023). However,
since there is no intrinsic mechanism for auto-regressive LLMs to “consult”
with any source of truth, they can generate plausible-sounding but incorrect
content Ji et al. (2023). To tackle the hallucination issue, various studies
have been proposed to augment LLMs Mialon et al. (2023) by either conditioning
them on retrieved relevant content Guu et al. (2020); Lewis et al. (2020);
Borgeaud et al. (2022) or allowing them to use other external tools such as
program APIs Gao et al. (2022); Parisi et al. (2022); Schick et al. (2023);
Qin et al. (2023).

In this work, we propose to teach LLMs to use the Web APIs of the National
Center for Biotechnology Information (NCBI). NCBI provides API access to its
entire biomedical databases and tools, including Entrez Programming Utilities
(E-utils) and Basic Local Alignment Search Tool (BLAST) URL API Altschul et
al. (1990); Schuler et al. (1996); Sayers et al. (2019). Enabling LLMs to use
NCBI Web APIs can provide easier and more precise access to biomedical
information, especially for users who are inexperienced with the database
systems. More importantly, Web APIs can relieve users from locally
implementing functionalities, maintaining large databases, and heavy
computation burdens because the only requirement for using Web APIs is an
internet connection.

Figure 1: Left: GeneGPT uses NCBI Web API documentations and demonstrations in
the prompt for in-context learning. Right: Examples of GeneGPT answering
GeneTuring and GeneHop questions with NCBI Web APIs.

We introduce GeneGPT, a novel method that prompts Codex Chen et al. (2021) to
use NCBI Web APIs by in-context learning Brown et al. (2020). GeneGPT consists
of two main modules: (a) a specifically designed prompt that consists of
documentations and demonstrations of API usage, and (b) an inference algorithm
that integrates API calls in the Codex decoding process. We evaluate GeneGPT
on GeneTuring Hou and Ji (2023), a question answering (QA) benchmark for
genomics, and compare GeneGPT to a variety of other LLMs such as the new
Bing111https://www.bing.com/new, ChatGPT222https://chat.openai.com/, and
BioGPT Luo et al. (2022). GeneGPT achieves the best performance on eight
GeneTuring tasks with an average score of 0.83, which is remarkably higher
than the previous SOTA (0.44 by New Bing). In addition, we systematically
characterize GeneGPT and find that: (1) API demonstrations are more useful
than documentations for in-context learning; (2) GeneGPT generalizes to longer
chains of subquestion decomposition and API calls with simple demonstrations;
(3) GeneGPT makes specific errors that are enriched for each task.

In summary, our contributions are three-fold:

  1. 1.

We introduce GeneGPT, a novel method that uses NCBI Web APIs to answer
biomedical questions. To the best of our knowledge, this is the first study on
augmenting LLMs with domain-specific Web API tools.

  2. 2.

GeneGPT achieves SOTA performance on 8 tasks in the GeneTuring benchmark,
largely outperforming previous best results by 88% (0.83 v.s. 0.44 set by the
new Bing).

  3. 3.

We conduct experiments to further characterize GeneGPT, including ablation,
probing, and error analyses. We also contribute a novel GeneHop dataset, and
use it to show that GeneGPT can perform chain-of-thought API calls to answer
multi-hop genomics questions.

##  2 GeneGPT

In this section, we first introduce the general functions and syntax of NCBI
Web APIs (§2.1). We then describe two key components of GeneGPT: its prompt
design for in-context learning (§2.2) and the inference algorithm (§2.3).

###  2.1 NCBI Web APIs

We utilize NCBI Web APIs of
E-utils333https://www.ncbi.nlm.nih.gov/books/NBK25501/ that provide access to
biomedical databases and the BLAST tool444https://ncbi.github.io/blast-
cloud/dev/api.html for DNA sequence alignment. Web API calls are implemented
by the urllib library in Python.

#### E-utils.

It is the API for accessing the Entrez portal Schuler et al. (1996), which is
a system that covers 38 NCBI databases of biomedical data such as genes and
proteins Sayers et al. (2019). The E-utils API provides a fixed URL syntax for
rapidly retrieving such biomedical information. Specifically, the base URL for
an E-utils request is
“https://eutils.ncbi.nlm.nih.gov/entrez/eutils/{function}.fcgi”, where
function can be esearch, efetch, or esummary. Typically, the user first calls
esearch to get the unique database identifiers of a given query term. Then,
efetch or esummary can be called to get the full records or text summaries of
a given list of identifiers returned by esearch. Important arguments in the
URL request include the search term or ids (term or id), the database to use
(db), the maximum number of returned items (retmax), and the return format
(retmode).

#### BLAST URL API.

BLAST takes as input a sequence of nucleotides or amino acids and finds the
most similar sequences in the database Altschul et al. (1990); Boratyn et al.
(2013). The results can be used to infer relationships between sequences or
identify members of gene families. The BLAST API allows users to submit
queries to find regions of similarities between nucleotide or protein
sequences to existing databases using the BLAST algorithm on NCBI servers. The
base URL for the BLAST URL API is
“https://blast.ncbi.nlm.nih.gov/blast/Blast.cgi”. By sending different
parameters to this API, the user can submit and retrieve queries that are
executed by NCBI web servers. Every call to the API must include a CMD
parameter that defines the type of the call. When submitting queries using
CMD=Put, the user can specify the querying database with the DATABASE
parameter, the searching program with the PROGRAM parameter, and the query
sequence with the QUERY parameter. The user will get an RID after the CMD=Put
API call, and can make another API call with the Get command and the returned
RID to retrieve its BLAST results.

Comp. | Documentation | Database | Function
---|---|---|---
Dc.1 | E-utils |  gene, snp, omim |  esearch, efetch, esummary
Dc.2 | BLAST | nt | blastn
Comp. | Demonstration | Database | Function
Dm.1 | Alias | gene |  esearch-> efetch
Dm.2 | Gene SNP | snp | esummary
Dm.3 | Gene disease | omim |  esearch-> esummary
Dm.4 | Alignment | nt | blastn
Table 1: Summary of API usage documentations (Dc.1 and Dc.2) and
demonstrations (Dm.1-4) in the GeneGPT prompt. Complete texts are shown in
Appendix A.

###  2.2 In-context learning

We teach an LLM to use NCBI Web APIs through in-context learning with an
engineered prompt. Figure 1 shows an example of the GeneGPT prompt, which is
composed of four modules: 1. an instruction; 2. API documentations; 3. API
demonstrations; 4. a test question. The first three parts are fixed for all
tasks, while the last one is task-specific.

  1. 1.

Instruction: The prompt starts with an overall task description (“Your task is
to use NCBI APIs to answer genomic questions.”). It is then followed by
documentations and demonstrations of API usage summarized in Table 1.

  2. 2.

Documentations (Dc.) provide natural language descriptions of the API
functionality, general syntax, and argument choices. We include one for the
E-utils API (Dc.1) and one for the BLAST tool (Dc.2).

  3. 3.

Demonstrations (Dm.) are concrete examples of using NCBI Web APIs to solve
questions. Based on questions in the GeneTuring tasks, we manually write four
demonstrations that cover four functions (esearch, efetch, esummary, blastn)
and four databases (gene, snp, omim, nt) of E-utils and BLAST. The API URLs
and the call results are marked up by “[ ]”, with a special “->” symbol
inserted in between that serves as an indicator for API calls.

  4. 4.

Test question: The specific test question is then appended to the end of the
prompt.

While the initial GeneGPT uses all documentations and demonstrations (denoted
as GeneGPT-full in Table 2), we find through analyses in §4.1 that GeneGPT can
work well with only two demonstrations (denoted as GeneGPT-slim) on all tasks.

###  2.3 Inference algorithm

Algorithm 1 GeneGPT inference algorithm

Input:  question

Model:  Codex (code-davinci-002)

Output:  answer

prompt $\leftarrow$ header $+$ demonstrations $+$ question

finished $\leftarrow$ False

while not finished do

next token $\leftarrow$ Codex(prompt)

prompt $\leftarrow$ prompt $+$ next token

if next token is "->" then

url $\leftarrow$ extractLastURL(prompt)

result $\leftarrow$ callWebAPI(url)

prompt $\leftarrow$ prompt $+$ result

else if next token is "\n\n" then

answer $\leftarrow$ extractAnswer(prompt)

finished $\leftarrow$ True

end if

end while

The GeneGPT inference algorithm is briefly shown in Algorithm 1. Specifically,
we first append the given question to the prompt (described in §2.2) and feed
the concatenated text to Codex (code-davinci-002, Chen et al. (2021)) with a
temperature of 0. We choose to use Codex for two reasons: (1) it is pre-
trained with code data and shows better code understanding abilities, which is
crucial in generating the URLs and interpreting the raw API results; (2) its
API has the longest (8k tokens) context length among all available models so
that we can fit the demonstrations in.

We discontinue the text generation process when the special “->” symbol is
detected, which is the indication for an API call request. Then we extract the
last URL and call the NCBI Web API with it. The raw execution results will be
appended to the generated text, and it will be fed to Codex to continue the
generation. When “\n\n”, an answer indicator used in the demonstrations, is
generated, we will stop the inference and extract the answer after the
generated “Answer: ”.

GeneTuring task | GPT-2 | BioGPT | BioMedLM | GPT-3 | ChatGPT | New Bing |  GeneGPT (ours)
---|---|---|---|---|---|---|---
-full | -slim
Nomenclature
Gene alias | 0.00 | 0.00 | 0.04 | 0.09 | 0.07 | 0.66 |  0.80* |  0.84*
Gene name conversion | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.85 | 1.00 | 1.00
Average | 0.00 | 0.00 | 0.02 | 0.05 | 0.04 | 0.76 | 0.90 | 0.92
Genomic location
Gene SNP association | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |  1.00* | 1.00
Gene location | 0.01 | 0.04 | 0.12 | 0.09 | 0.09 | 0.61 | 0.62 | 0.66
SNP location | 0.03 | 0.05 | 0.01 | 0.02 | 0.05 | 0.01 | 1.00 | 0.98
Average | 0.01 | 0.03 | 0.04 | 0.04 | 0.05 | 0.21 | 0.87 | 0.88
Functional analysis
Gene disease association | 0.00 | 0.02 | 0.16 | 0.34 | 0.31 | 0.84 |  0.76* | 0.66
Protein-coding genes | 0.00 | 0.18 | 0.37 | 0.70 | 0.54 | 0.97 | 0.76 | 1.00
Average | 0.00 | 0.10 | 0.27 | 0.52 | 0.43 | 0.91 | 0.76 | 0.84
Sequence alignment
DNA to human genome | 0.02 | 0.07 | 0.03 | 0.00 | 0.00 | 0.00 |  0.44* |  0.44*
DNA to multiple species | 0.02 | 0.00 | 0.00 | 0.20 | 0.00 | 0.00 | 0.86 | 0.88
Average | 0.02 | 0.04 | 0.02 | 0.10 | 0.00 | 0.00 | 0.65 | 0.66
Overall average | 0.00 | 0.04 | 0.08 | 0.16 | 0.12 | 0.44 | 0.80 | 0.83
Table 2: Performance of GeneGPT compared to other LLMs on the GeneTuring
benchmark. *One-shot learning for GeneGPT. Bolded and underlined numbers
denote the highest and second-highest performance, respectively.

##  3 Experiments

###  3.1 GeneTuring

The GeneTuring benchmark Hou and Ji (2023) contains 12 tasks, and each task
has 50 question-answer pairs. We use 9 GeneTuring tasks that are related to
NCBI resources to evaluate the proposed GeneGPT model, and the QA samples are
shown in Appendix B. The chosen tasks are classified into four modules and
briefly described in this section.

#### Nomenclature:

This is about gene names. We use the gene alias task and the gene name
conversion task, where the objective is to find the official gene symbols for
their non-official synonyms.

#### Genomics location:

The tasks are about the locations of genes, single-nucleotide polymorphism
(SNP), and their relations. We include the gene location, SNP location, and
gene SNP association tasks. The first two tasks ask for the chromosome
locations (e.g., “chr2”) of a gene or an SNP, and the last one asks for
related genes for a given SNP.

#### Functional analysis:

It asks for gene functions. We use the gene disease association task where the
goal is to return related genes for a given disease, and the protein-coding
genes task which asks whether a gene is a protein-coding gene or not.

#### Sequence alignment:

The tasks query specific DNA sequences. We use the DNA sequence alignment to
human genome task and the DNA sequence alignment to multiple species task. The
former maps an DNA sequence to a specific human chromosome, while the latter
maps an DNA sequence to a specific species (e.g. “zebrafish”).

###  3.2 Compared methods

We evaluate two settings of GeneGPT, a full setting (GeneGPT-full) where all
prompt components are used, as well as a slim setting (GeneGPT-slim) inspired
by our ablation and probing analyses (§4.1) where only Dm.1 and Dm.4 are used.

We compare GeneGPT with various baselines evaluated by Hou and Ji (2023),
including general-domain GPT-based Radford et al. (2018) LLMs such as GPT-2
Radford et al. (2019), GPT-3 (text-davinci-003) Brown et al. (2020), and
ChatGPT555https://chat.openai.com/ (Jan 31 version)., GPT-2-sized biomedical
domain-specific LLMs such as BioGPT Luo et al. (2022) and
BioMedLM666https://crfm.stanford.edu/2022/12/15/biomedlm.html, previously
known as PubMedGPT., as well as the new Bing777https://www.bing.com/new, a
retrieval-augmented LLM that has access to relevant web pages retrieved by
Bing.

###  3.3 Evaluation

For the performance of the compared methods, we directly use the results
reported in the original benchmark that are manually evaluated.

To evaluate our proposed GeneGPT method, we follow the general criteria but
perform automatic evaluations. Specifically, we only consider exact matches
between model predictions and the ground truth as correct predictions for all
nomenclature and genomics location tasks. For the gene disease association
task, we measure the recall as in the original dataset but based on exact
individual gene matches. For the protein-coding genes task and the DNA
sequence alignment to multiple species task, we also consider exact matches as
correct after applying a simple vocabulary mapping that converts model-
predicted “yes”/“no” to “TRUE”/“NA” and Latin species names to their informal
names (e.g., “Saccharomyces cerevisiae” to “yeast”), respectively. For the DNA
sequence alignment to human genome task, we give correct chromosome mapping
but incorrect position mapping a score of 0.5 (e.g., chr8:7081648-7081782 v.s.
chr8:1207812-1207946), since the original task does not specify a reference
genome. Overall, our evaluation of GeneGPT is more strict than the original
evaluation of other LLMs in Hou and Ji (2023), which performs manual
evaluation and might consider non-exact matches as correct.

###  3.4 Main results

Table 2 shows the performance of GeneGPT on the GeneTuring tasks in comparison
with other LLMs. For GeneGPT, tasks with “*” in Table 2 are one-shot where one
instance is used as API demonstration, and the other tasks are zero-shot. For
the compared LLMs, all tasks are zero-shot.

#### Nomenclature:

GeneGPT achieves state-of-the-art (SOTA) performance on both the one-shot gene
alias task with an accuracy of 0.84 and the zero-shot gene name conversion
task with an accuracy of 1.00. On average, GeneGPT outperforms New Bing by a
large margin (0.92 v.s. 0.76). All other GPT models have accuracy scores of
less than 0.10 on the nomenclature tasks.

#### Genomic location:

GeneGPT also achieves SOTA performance on all genomic location tasks,
including the gene SNP association task (1.00) gene location task (0.66) and
the SNP location task (1.00). While the New Bing is comparable to GeneGPT on
gene location (0.61 v.s. 0.66), its performance on the two SNP-related tasks
is close to 0. Again, most other LLMs score less than 0.10. Notably, while all
genomics location tasks are zero-shot for GeneGPT-slim, it performs comparably
to GeneGPT-full which uses one gene SNP association demonstration. This
indicates that API demonstrations have strong cross-task generalizability.

Figure 2: Performance changes of the ablation (left) and probing (right)
experiments as compared to GeneGPT-full.

#### Functional analysis:

The new Bing performs better functional analysis tasks than the proposed
GeneGPT (average score: 0.91 v.s. 0.84), which is probably because many web
pages related to gene functions can be retrieved by the Bing search engine. We
also note that other LLMs, especially GPT-3 and ChatGPT, perform moderately
well and much better than they perform on other tasks. This might also be due
to the fact that many gene-function-related texts are included in their pre-
training corpora.

#### Sequence alignment:

GeneGPT performs much better with an average score of 0.66 than all other
models including the new Bing (0.00), which essentially fails on the sequence
alignment tasks. This is not very surprising since sequence alignment is easy
with the BLAST tool, but almost impossible for an auto-regressive LLM even
with retrieval augmentation as the input sequences are too specific to be
indexed by a search engine.

Although evaluated under a more strict setting (§3.3), GeneGPT achieves a
macro-average performance of 0.83 which is much higher than other compared
LLMs including New Bing (0.44). Overall, GeneGPT achieves new SOTA performance
on all 2 one-shot tasks and 6 out of 7 zero-shot tasks and is outperformed by
New Bing only on the gene disease association task.

##  4 Discussions

We have shown that GeneGPT largely surpasses various LLMs on the GeneTuring
benchmark. In this section, we further characterize GeneGPT by studying three
research questions (RQ):

RQ1: What is the importance of each prompt component in GeneGPT?

RQ2: Can GeneGPT answer multi-hop questions by chain-of-thought API calls?

RQ3: What types of errors does GeneGPT make on each studied task?

###  4.1 RQ1: Component importance

We conduct ablation and probing experiments to study the importance of
individual prompt components, including 2 documentations (Dc.1, Dc.2) and 4
demonstrations (Dm.1-4) described in §2.2.

For ablation tests, we remove each component from GeneGPT-full and then
evaluate the prompt. The results are shown in Figure 2 (left). Notably, the
performance on the DNA to genome and species alignment tasks is only
significantly decreased without the BLAST demonstration (Dm.4), but not
affected by the ablation of the BLAST documentation (Dc.2). While the
ablations of other components decrease the performance, most only affect one
relevant task (e.g., Dm.1 and gene name conversion), which indicates a high
level of redundancy of the prompt components.

For the probing experiments, we evaluate GeneGPT with only one prompt
component to study the individual capability. The results are shown in Figure
2 (right). Overall, GeneGPT with only one documentation (Dc.1 or Dc.2) fails
on all tasks. Surprisingly, with only one demonstration of the gene alias task
(Dm.1) in the prompt, GeneGPT is able to perform comparably to GeneGPT-full on
all tasks except the alignment ones. On the other hand, GeneGPT with only the
BLAST demonstration (Dm.4) performs well on the two alignment tasks, which is
somehow expected. These results suggest that GeneGPT with only two
demonstrations (Dm.1 and Dm.4) in the prompt can generalize to all tasks in
the GeneTuring benchmark. We denote this as GeneGPT-slim, and results in Table
2 show that with only two demonstrations, it outperforms the GeneGPT-full and
achieves state-of-the-art overall results on GeneTuring.

###  4.2 RQ2: Multi-hop QA on GeneHop

Questions in the GeneTuring benchmark are single-hop and just require one step
of reasoning, e.g., “Which gene is SNP rs983419152 associated with?”. However,
many real-world biomedical questions are multi-hop that need more steps to
answer Jin et al. (2022). For example, to answer “What is the function of the
gene associated with SNP rs983419152?”, the model should first get the
associated gene name and then find its functions.

GeneHop question (SNP gene function):
What is the function of the gene associated with SNP rs1241371358? Let’s
decompose the question to sub-questions and solve them step by step.
GeneGPT output:
Sub-question 1: Which gene is SNP rs1241371358 associated with?
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=snp&retmax=10&retmode=json&id=1241371358]
->[API call results]
Answer: LRRC23
Sub-question 2: What is the function of LRRC23?
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=gene&retmax=10&retmode=json&id=10233]
->[API call results]
Answer: Predicted to be active in cytosol. (correct)

Figure 3: GeneGPT uses chain-of-thought API calls to answer a multi-hop
question in GeneHop.

To test GeneGPT’s capability of answering multi-hop questions, we present
GeneHop, a novel dataset that contains three new multi-hop QA tasks based on
the GeneTuring benchmark: (a) SNP gene function, which asks for the function
of the gene associated with a given SNP. (b) Disease gene location, where the
task is to list the chromosome locations of the genes associated with a given
disease. (c) Sequence gene alias, which asks for the aliases of the gene that
contains a specific DNA sequence. Each task in GeneHop contains 50 questions,
and the collection pipeline is detailed in Appendix C. For all tasks, we
append the chain-of-thought instruction “Let’s decompose the question to sub-
questions and solve them step by step.” after the test question Wei et al.
(2022b).

Figure 3 shows an example of GeneGPT to answer Task (a). In this case, GeneGPT
successfully decomposes the multi-hop question into two sub-questions, and the
sub-question 2 is based on the answer of the sub-question 1. Interestingly,
GeneGPT uses a shortcut to answer sub-question 2: instead of first calling
esearch and then calling esummary, GeneGPT finds the gene id in the API call
results of sub-question 1 and directly calls esummary. This capability is not
shown in the prompt but elicited by chain-of-thought API calls.

GeneHop question (Disease gene location):
List chromosome locations of the genes related to Cleft palate with
ankyloglossia. Let’s decompose the question to sub-questions and solve them
step by step.
GeneGPT output:
Sub-question 1: What is the OMIM id of Cleft palate with ankyloglossia?
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=omim&retmax=20&retmode=json&sort=relevance&term=Cleft+palate+with+ankyloglossia]->[API
call results]
Answer: 303400
Sub-question 2: What are genes related to Cleft palate with ankyloglossia?
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=omim&retmax=20&retmode=json&id=303400]
->[API call results]
Answer: CPX
Sub-question 3: What is the chromosome location of CPX?
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=gene&retmax=20&retmode=json&sort=relevance&term=CPX]->[API
call results]
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=gene&retmax=20&retmode=json&id=64877,1377,7758276,100818277,100166185,10948718,10849362,9700326,3180310,3180309,1192676,1371,50945,10682,12892,1037,4336951,100216262,380161,108700304]
->[API call results]
Answer: Xq21.1 (correct)

Figure 4: GeneGPT uses chain-of-thought API calls to answer a multi-hop
question in GeneHop.

Figure 4 shows another example of GeneGPT answering Task (b), where GeneGPT
successfully decomposes the multi-hop question and correctly calls the
required APIs. Notably, the answering chain involves 3 sub-questions and 4 API
calls, which are longer than all in-context demonstrations (1 single-hop
question and 2 API calls at most). This ability to generalize to longer chains
of thought is an important aspect of GeneGPT’s flexibility and usefulness for
real-world applications.

GeneHop Task | New Bing | GeneGPT
---|---|---
SNP gene function | 0.00 | 0.55
Disease gene location | 0.71 | 0.67
Sequence gene alias | 0.00 | 0.28
Average | 0.24 | 0.50
Table 3: Performance of multi-hop QA on GeneHop. We only compare GeneGPT with
New Bing since other LLMs cannot even answer single-hop questions well.

We manually evaluate the results predicted by GeneGPT and compare it to the
new Bing, which is the only baseline LLM that performs well on the single-hop
GeneTuring benchmark due to its retrieval augmentation feature. The evaluation
criteria are described in Appendix D. As shown in Table 3, while the new Bing
outperforms GeneGPT on the disease gene location task, it is mostly using
webpages that contain both the disease and location information without multi-
hop reasoning. The new Bing fails to perform the other 2 tasks since the input
information (SNP or sequence) is not indexed by Bing and can only be found in
specialized databases. GeneGPT, on the other hand, performs moderately well on
all 3 tasks, and achieves a much higher average score (0.50 v.s. 0.24).

###  4.3 RQ3: Error analysis

We manually study all errors made by GeneGPT and classify them into five
types. Table 4 shows the count of each error type on the evaluate tasks: E1:
using the wrong API or not using APIs, e.g., using the gene instead of the
omin database for diseases; E2: using the right API but wrong arguments, e.g.,
passing terms to id; E3: not extracting the answer in the API result, most
commonly seen in gene function extraction; E4: right API call but results do
not contain the answer, where the question is not answerable with NCBI
databases; and O includes other unclassified errors. Specific error examples
are shown in Appendix E.

GeneTuring Task | E1 | E2 | E3 | E4 | O
---|---|---|---|---|---
Gene alias | 0 | 0 | 2 | 6 | 0
Gene location | 0 | 0 | 0 | 17 | 0
SNP location | 0 | 1 | 0 | 0 | 0
Gene disease association | 15 | 0 | 0 | 3 | 2
DNA to human genome | 0 | 0 | 7 | 0 | 42
DNA to multiple species | 0 | 0 | 1 | 1 | 4
GeneHop Task | E1 | E2 | E3 | E4 | O
SNP gene function | 0 | 0 | 29 | 0 | 0
Disease gene location | 4 | 7 | 1 | 5 | 1
Sequence gene alias | 0 | 30 | 8 | 0 | 0
Table 4: Counts of GeneGPT errors on different tasks. E1: wrong API; E2: wrong
arguments; E3: wrong comprehension; E4: unanswerable with API; O: others.

Our results suggest that different tasks have specific and enriched error
types: simple tasks (alias and location) fail mostly because of E4; E1 only
happens in disease-related tasks; alignment tasks face more issues with BLAST
interfaces and reference genomes (O); multi-hop tasks in GeneHop tend to have
E2 and E3 in the reasoning chains.

##  5 Related work

#### Large language models:

Recent studies have shown that scaling pre-trained LMs leads to performance
improvement and potentially emergent abilities on various NLP tasks Brown et
al. (2020); Kaplan et al. (2020); Wei et al. (2022a); Chowdhery et al. (2022);
OpenAI (2023). However, such auto-regressive LLMs are still susceptible to
hallucinations and generate erroneous content Ji et al. (2023). Augmenting
LLMs with external tools is a possible solution to this issue Mialon et al.
(2023).

#### Tool augmentation:

Potential tools include: (1) search engines Guu et al. (2020); Lewis et al.
(2020); Borgeaud et al. (2022), also known as retrieval augmentation,
exemplified by New Bing; (2) program APIs by in-context learning Gao et al.
(2022); Schick et al. (2023) or fine-tuning Parisi et al. (2022); Schick et
al. (2023). We present the first study on the in-context learning abilities of
documentations and demonstrations of NCBI Web APIs.

#### Biomedical question answering:

It is an essential step in clinical decision support Ely et al. (2005) and
biomedical knowledge acquisition Jin et al. (2022). LLMs have been
successfully applied to various biomedical QA tasks that are knowledge\- or
reasoning-intensive Singhal et al. (2022); Liévin et al. (2022); Nori et al.
(2023). However, auto-regressive LLMs fail to perform data-intensive tasks
which require the model to precisely store and recite database entries, such
as the GeneTuring benchmark Hou and Ji (2023). Retrieval augmentation also
falls short since specialized databases are usually not indexed by commercial
search engines. GeneGPT solves this task by tool augmentation.

##  6 Conclusions

We present GeneGPT, a novel method that teaches LLMs to use NCBI Web APIs. It
achieves SOTA performance on 8 GeneTuring tasks and can perform chain-of-
thought API calls. Our results indicate that database utility tools might be
superior to relevant web pages for augmenting LLMs to faithfully serve various
biomedical information needs.

## References

  * Altschul et al. (1990) Stephen F Altschul, Warren Gish, Webb Miller, Eugene W Myers, and David J Lipman. 1990.  Basic local alignment search tool.  _Journal of molecular biology_ , 215(3):403–410.
  * Boratyn et al. (2013) Grzegorz M Boratyn, Christiam Camacho, Peter S Cooper, George Coulouris, Amelia Fong, Ning Ma, Thomas L Madden, Wayne T Matten, Scott D McGinnis, Yuri Merezhuk, et al. 2013.  Blast: a more efficient report with usability improvements.  _Nucleic acids research_ , 41(W1):W29–W33.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pages 2206–2240. PMLR.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Ely et al. (2005) John W Ely, Jerome A Osheroff, M Lee Chambliss, Mark H Ebell, and Marcy E Rosenbaum. 2005.  Answering physicians’ clinical questions: obstacles and potential solutions.  _Journal of the American Medical Informatics Association_ , 12(2):217–224.
  * Gao et al. (2022) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022.  Pal: Program-aided language models.  _arXiv preprint arXiv:2211.10435_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pages 3929–3938. PMLR.
  * Hou and Ji (2023) Wenpin Hou and Zhicheng Ji. 2023.  Geneturing tests gpt models in genomics.  _bioRxiv_ , pages 2023–03.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Jin et al. (2022) Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Huaiyuan Ying, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. 2022.  Biomedical question answering: a survey of approaches and challenges.  _ACM Computing Surveys (CSUR)_ , 55(2):1–36.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Liévin et al. (2022) Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2022.  Can large language models reason about medical questions?  _arXiv preprint arXiv:2207.08143_.
  * Luo et al. (2022) Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022.  Biogpt: generative pre-trained transformer for biomedical text generation and mining.  _Briefings in Bioinformatics_ , 23(6).
  * Mialon et al. (2023) Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023.  Augmented language models: a survey.  _arXiv preprint arXiv:2302.07842_.
  * Nori et al. (2023) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023.  Capabilities of gpt-4 on medical challenge problems.  _arXiv preprint arXiv:2303.13375_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Parisi et al. (2022) Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022.  Talm: Tool augmented language models.  _arXiv preprint arXiv:2205.12255_.
  * Qin et al. (2023) Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023.  Tool learning with foundation models.  _arXiv preprint arXiv:2304.08354_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Sayers et al. (2019) Eric W Sayers, Richa Agarwala, Evan E Bolton, J Rodney Brister, Kathi Canese, Karen Clark, Ryan Connor, Nicolas Fiorini, Kathryn Funk, Timothy Hefferon, et al. 2019.  Database resources of the national center for biotechnology information.  _Nucleic acids research_ , 47(Database issue):D23.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_.
  * Schuler et al. (1996) GD Schuler, JA Epstein, H Ohkawa, and JA Kans. 1996.  Entrez: molecular biology database and retrieval system.  _Methods in enzymology_ , 266:141–162.
  * Singhal et al. (2022) Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022\.  Large language models encode clinical knowledge.  _arXiv preprint arXiv:2212.13138_.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.

##  Appendix A GeneGPT prompt

Here we show the exact texts of each prompt component described in Table 1,
including two documentations Dc.1 (E-utils, Figure 5) and Dc.2 (BLAST, Figure
6) as well as four demonstrations Dm.1 (gene alias, Figure 7), Dm.2 (gene SNP
association, Figure 8), Dm.3 (gene disease association, Figure 9), and Dm.4
(DNA to human genome alignment, Figure 10).

Documentation 1 (Dc. 1)
You can call Eutils by:
"[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/{esearch|efetch|esummary}.fcgi?db={gene|snp|omim}&retmax={}&{term|id}={term|id}]".
esearch: input is a search term and output is database id(s).
efectch/esummary: input is database id(s) and output is full records or
summaries that contain name, chromosome location, and other information.
Normally, you need to first call esearch to get the database id(s) of the
search term, and then call efectch/esummary to get the information with the
database id(s).
Database: gene is for genes, snp is for SNPs, and omim is for genetic
diseases.

Figure 5: Documentation 1 (Dc.1) of the GeneGPT prompt. Dc.1 describes the
functionality, general syntax, and argument choices of the NCBI E-utils API.

Documentation 2 (Dc. 2)
For DNA sequences, you can use BLAST by:
"[https://blast.ncbi.nlm.nih.gov/blast/Blast.cgi?CMD={Put|Get}&PROGRAM=blastn&MEGABLAST=on&DATABASE=nt&FORMAT_TYPE={XML|Text}&QUERY={sequence}&HITLIST_SIZE={max_hit_size}]".
BLAST maps a specific DNA sequence to its chromosome location among different
species.
You need to first PUT the BLAST request and then GET the results using the RID
returned by PUT.

Figure 6: Documentation 2 (Dc.2) of the GeneGPT prompt. Dc.2 describes the
functionality, general syntax, and argument choices of the BLAST API.

Demonstration 1 (Dm. 1)
Question: What is the official gene symbol of LMP10?
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=gene&retmax=5&retmode=json&sort=relevance&term=LMP10]->[{"header":{"type":"esearch","version":"0.3"}
,"esearchresult":{"count":"3","retmax":"3","retstart":"0","idlist":["5699",
"8138","19171"],"translationset":[],"translationstack":[{"term":"LMP10[All
Fields]","field":"All Fields","count":"3","explode":"N"},"GROUP"],
"querytranslation":"LMP10[All Fields]"}}]
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=gene&retmax=5&retmode=json&id=19171,5699,8138]->[1\.
Psmb10 Official Symbol: Psmb10 and Name: proteasome (prosome, macropain)
subunit, beta type 10 [Mus musculus (house mouse)] Other Aliases: Mecl-1,
Mecl1 Other Designations: proteasome subunit beta type-10; low molecular mass
protein 10; macropain subunit MECl-1; multicatalytic endopeptidase complex
subunit MECl-1; prosome Mecl1; proteasome (prosomome, macropain) subunit, beta
type 10; proteasome MECl-1; proteasome subunit MECL1; proteasome subunit
beta-2i Chromosome: 8; Location: 8 53.06 cM Annotation: Chromosome 8
NC_000074.7 (106662360..106665024, complement) ID: 19171 2\. PSMB10 Official
Symbol: PSMB10 and Name: proteasome 20S subunit beta 10 [Homo sapiens (human)]
Other Aliases: LMP10, MECL1, PRAAS5, beta2i Other Designations: proteasome
subunit beta type-10; low molecular mass protein 10; macropain subunit MECl-1;
multicatalytic endopeptidase complex subunit MECl-1; proteasome (prosome,
macropain) subunit, beta type, 10; proteasome MECl-1; proteasome catalytic
subunit 2i; proteasome subunit MECL1; proteasome subunit beta 10; proteasome
subunit beta 7i; proteasome subunit beta-2i; proteasome subunit beta2i
Chromosome: 16; Location: 16q22.1 Annotation: Chromosome 16 NC_000016.10
(67934506..67936850, complement) MIM: 176847 ID: 5699 3\. MECL1 Proteosome
subunit MECL1 [Homo sapiens (human)] Other Aliases: LMP10, PSMB10 This record
was replaced with GeneID: 5699 ID: 8138]
Answer: PSMB10

Figure 7: Demonstration 1 (Dm.1) of the GeneGPT prompt. The instance is chosen
from the gene alias task in the GeneTuring benchmark. Links are actually
called Web API URLs. Readers can directly click the link and get the API call
result, which is inserted in the prompt.

Demonstration 2 (Dm. 2)
Question: Which gene is SNP rs1217074595 associated with?
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=snp&retmax=10&retmode=json&id=1217074595]->[{"header":{"type":"esummary","version":"0.3"},
"result":{"uids":["1217074595"],"1217074595":{"uid":"1217074595","snp_id":
1217074595,"allele_origin":"","global_mafs":[{"study":"GnomAD","freq":"A=
0.000007/1"},{"study":"TOPMED","freq":"A=0.000004/1"},{"study":"ALFA","freq":"A=
0./0"}],"global_population":"","global_samplesize":"","suspected":"","clinical
_significance":"","genes":[{"name":"LINC01270","gene_id":"284751"}],"acc":"NC
_000020.11","chr":"20","handle":"GNOMAD
,TOPMED","spdi":"NC_000020.11:50298394:G:A","fxn_class":"non_coding_transcript
_variant","validated":"by-frequency,by-alfa,by-
cluster","docsum":"HGVS=NC_000020
.11:g.50298395G>A,NC_000020.10:g.48914932G>A,NR_034124.1:n.351G>A,NM_001025463.1
:c.*4G>A|SEQ=[G/A]|LEN=1|GENE=LINC01270:284751","tax_id":9606,"orig_build"
:155,"upd_build":156,"createdate":"2017/11/09 09:55","updatedate":"2022/10/13
17:11","ss":"4354715686,5091242333","allele":"R","snp_class":"snv","chrpos":"20:
50298395","chrpos_prev_assm":"20:48914932","text":"","snp_id_sort":"1217074595",
"clinical_sort":"0","cited_sort":"","chrpos_sort":"0050298395","merged_sort":"0"
}}}]
Answer: LINC01270

Figure 8: Demonstration 2 (Dm.2) of the GeneGPT prompt. The instance is chosen
from the gene SNP association task in the GeneTuring benchmark. Links are
actually called Web API URLs. Readers can directly click the link and get the
API call result, which is inserted in the prompt.

Demonstration 3 (Dm. 3)
Question: What are genes related to Meesmann corneal dystrophy?
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=omim&retmax=20&retmode=json&sort=relevance&term=Meesmann+corneal+dystrophy]->[{"header":{"type":"esearch","version":"0.3"},"esearchresult":{"count":"5","ret
max":"5","retstart":"0","idlist":["122100","618767","300778","601687","148043"],
"translationset":[],"translationstack":[{"term":"Meesmann[All
Fields]","field":"All Fields","count":"5","explode":"N"},{"term":"corneal[All
Fields]","field":"All Fields","count":"561","explode":"N"},"AND",{"term":
"dystrophy[All Fields]","field":"All
Fields","count":"1326","explode":"N"},"AND"
,"GROUP"],"querytranslation":"Meesmann[All Fields] AND corneal[All Fields] AND
dystrophy[All Fields]"}}]
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=omim&retmax=20&retmode=json&id=618767,601687,300778,148043,122100]->[{"header":{"type":"esummary","version":"0.3"},"result":{"uids":["618767",
"601687","300778","148043","122100"],"618767":{"uid":"618767","oid":"#618767",
"title":"CORNEAL DYSTROPHY, MEESMANN, 2;
MECD2","alttitles":"","locus":"12q13.13"
},"601687":{"uid":"601687","oid":"*601687","title":"KERATIN 12, TYPE I;
KRT12","alttitles":"","locus":"17q21.2"},"300778":{"uid":"300778","oid":"%300778"
,"title":"CORNEAL DYSTROPHY, LISCH EPITHELIAL; LECD","alttitles":"","locus":
"Xp22.3"},"148043":{"uid":"148043","oid":"*148043","title":"KERATIN 3, TYPE
II;
KRT3","alttitles":"","locus":"12q13.13"},"122100":{"uid":"122100","oid":"#122100"
,"title":"CORNEAL DYSTROPHY, MEESMANN, 1;
MECD1","alttitles":"","locus":"17q21.2"
}}}]
Answer: KRT12, KRT3

Figure 9: Demonstration 3 (Dm.3) of the GeneGPT prompt. The instance is chosen
from the gene disease association task in the GeneTuring benchmark. Links are
actually called Web API URLs. Readers can directly click the link and get the
API call result, which is inserted in the prompt.

Demonstration 4 (Dm. 4)
Question: Align the DNA sequence to the human
genome:ATTCTGCCTTTAGTAATTTGATGACAGAGACTTCTTGGGAA
CCACAGCCAGGGAGCCACCCTTTACTCCACCAACAGGTGGCTTATATCCAATCTGAGAAAGAAAGAAAAAAAAA
AAAGTATTTCTCT
[https://blast.ncbi.nlm.nih.gov/blast/Blast.cgi?CMD=Put&PROGRAM=blastn&MEGABLAST=on&DATABASE=nt&FORMAT_TYPE=XML&QUERY=ATTCT...TCTCT&HITLIST_SIZE=5]->[5S8YKEBH016]
[https://blast.ncbi.nlm.nih.gov/blast/Blast.cgi?CMD=Get&FORMAT_TYPE=Text&RID=5S8YKEBH016]->[BLASTN
2.14.0+ [...] RID: 5S8YKEBH016 Database: Nucleotide collection (nt) 93,066,592
sequences; 1,138,553,367,010 total letters Query= Length=128 Score E Max
Sequences producing significant alignments: (Bits) Value Ident CP034493.1
Eukaryotic synthetic construct chromosome 15 237 3e-58 100% NG_132175.1 Homo
sapiens H3K27ac-H3K4me1 hESC enhancer GRCh37_... 237 3e-58 100% CP068263.2
Homo sapiens isolate CHM13 chromosome 15 237 3e-58 100% AP023475.1 Homo
sapiens DNA, chromosome 15, nearly complete ge... 237 3e-58 100% FJ515841.1
Homo sapiens isolate SLC3A1-VI-T solute carrier org... 237 3e-58 100%
ALIGNMENTS >CP034493.1 Eukaryotic synthetic construct chromosome 15 CP034518.1
Eukaryotic synthetic construct chromosome 15 Length=82521392 Score = 237 bits
(128), Expect = 3e-58 Identities = 128/128 (100%), Gaps = 0/128 (0%)
Strand=Plus/Plus Query 1
ATTCTGCCTTTAGTAATTTGATGACAGAGACTTCTTGGGAACCACAGCCAGGGAGCCACC 60
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Sbjct 72494035
ATTCTGCCTTTAGTAATTTGATGACAGAGACTTCTTGGGAACCACAGCCAGGGAGCCACC 72494094 Query 61
CTTTACTCCACCAACAGGTGGCTTATATCCAATCTGAGAAAGAAAGaaaaaaaaaaaaGT 120
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Sbjct 72494095
CTTTACTCCACCAACAGGTGGCTTATATCCAATCTGAGAAAGAAAGAAAAAAAAAAAAGT 72494154 Query
121 ATTTCTCT 128 |||||||| Sbjct 72494155 ATTTCTCT 72494162 >NG_132175.1 Homo
sapiens H3K27ac-H3K4me1 hESC enhancer GRCh37_chr15:92493309-92494181
(LOC127830695) on chromosome 15 Length=1073 Score = 237 bits (128), Expect =
3e-58 Identities = 128/128 (100%), Gaps = 0/128 (0%) Strand=Plus/Plus Query 1
ATTCTGCCTTTAGTAATTTGATGACAGAGACTTCTTGGGAACCACAGCCAGGGAGCCACC 60
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Sbjct 827
ATTCTGCCTTTAGTAATTTGATGACAGAGACTTCTTGGGAACCACAGCCAGGGAGCCACC 886 Query 61
CTTTACTCCACCAACAGGTGGCTTATATCCAATCTGAGAAAGAAAGaaaaaaaaaaaaGT 120
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Sbjct 887
CTTTACTCCACCAACAGGTGGCTTATATCCAATCTGAGAAAGAAAGAAAAAAAAAAAAGT 946 Query 121
ATTTCTCT 128 |||||||| Sbjct 947 ATTTCTCT 954 >CP068263.2 Homo sapiens isolate
CHM13 chromosome 15 Length=99753195 Score = 237 bits (128), Expect = 3e-58
Identities = 128/128 (100%), Gaps = 0/128 (0%) Strand=Plus/Plus Query 1
ATTCTGCCTTTAGTAATTTGATGACAGAGACTTCTTGGGAACCACAGCCAGGGAGCCACC 60
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Sbjct 89712558
ATTCTGCCTTTAGTAATTTGATGACAGAGACTTCTTGGGAACCACAGCCAGGGAGCCACC 89712617 Query 61
CTTTACTCCACCAACAGGTGGCTTATATCCAATCTGAGAAAGAAAGaaaaaaaaaaaaGT 120
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Sbjct 89712618
CTTTACTCCACCAACAGGTGGCTTATATCCAATCTGAGAAAGAAAGAAAAAAAAAAAAGT 89712677 Query
121 ATTTCTCT 128 |||||||| Sbjct 89712678 ATTTCTCT 89712685 >AP023475.1 Homo
sapiens DNA, chromosome 15, nearly complete genome Length=95537968 Score = 237
bits (128), Expect = 3e-58 Identities = 128/128 (100%), Gaps = 0/128 (0%)
Strand=Plus/Plus Query 1
ATTCTGCCTTTAGTAATTTGATGACAGAGACTTCTTGGGAACCACAGCCAGGGAGCCACC 60
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Sbjct 85572367
ATTCTGCCTTTAGTAATTTGATGACAGAGACTTCTTGGGAACCACAGCCAGGGAGCCACC 85572426 Query 61
CTTTACTCCACCAACAGGTGGCTTATATCCAATCTGAGAAAGAAAGaaaaaaaaaaaaGT 120
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| Sbjct 85572427
CTTTACTCCACCAACAGGTGGCTTATATCCAATCTGAGAAAGAAAGAAAAAAAAAAAAGT 85572486 Query
121 ATTTCTCT 128 |||||||| Sbjct 85572487 ATTTCTCT 85572494 >FJ515841.1 Homo
sapiens isolate SLC3A1-VI-T solute carrier organic anion transporter family
member 3A1 (SLCO3A1) gene, complete cds Length=315834 Score = 237 bits (128),
Expect = 3e-58 Identities = 128/128 (100%), Gaps = 0/128 (0%) Strand=Plus/Plus
[...]]
Answer: chr15:91950805-91950932

Figure 10: Demonstration 4 (Dm.4) of the GeneGPT prompt. The instance is
chosen from the DNA to human genome alignment task in the GeneTuring
benchmark. Links are actually called Web API URLs. Readers can directly click
the link and get the API call result, which is inserted in the prompt. Since
the BLAST Web API is about to be depricated and the first step returns an HTML
page, we use a regular expression to extract the PID and append it to the
GeneGPT input.

##  Appendix B GeneTuring samples

Table 5 shows some sample question-answer pairs from the GeneTuring benchmark.
The dataset is publicly available at
https://www.biorxiv.org/content/10.1101/2023.03.11.532238v1.supplementary-
material. We use the same questions sent to the new Bing as the test
questions.

GeneTuring task |  Question |  Answer
---|---|---
Nomenclature
Gene alias |  What is the official gene symbol of SNAT6? |  SLC38A6
Gene name conversion |  Convert ENSG00000215251 to official gene symbol. |  FASTKD5
Genomic location
Gene SNP association |  Which gene is SNP rs996319727 associated with? |  USP39
Gene location |  Which chromosome is FOXL2NB gene located on human genome? |  chr3
SNP location |  Which chromosome does SNP rs427884 locate on human genome? |  chr11
Functional analysis
Gene disease association |  What are genes related to Bile acid malabsorption? |  SLC10A2, SLC51B
Protein-coding genes |  Is ATP5F1EP2 a protein-coding gene? |  NA
Sequence alignment
DNA to human genome |  Align the DNA sequence to the human genome: AGGCCC TCACCT GGAAAT TACTTA CTCATG CTTCAT GACCCA GTTCAA ATTTTG TCACCT CTGTGA AACCTT CCCTGG GCCCCG TTGATC TCCTTG AAGGCA |  chr7:71368450-71368551
DNA to multiple species |  Which organism does the DNA sequence come from: AGGGGC AGCAAA CACCGG GACACA CCCATT CGTGCA CTAATC AGAAAC TTTTTT TTCTCA AATAAT TCAAAC AATCAA AATTGG TTTTTT CGAGCA AGGTGG GAAATT TTTCGAT |  worm
Table 5: Sample question-answer pairs of the GeneTuring tasks Hou and Ji
(2023).

##  Appendix C GeneHop collection

The GeneHop dataset contains three multi-hop tasks: SNP gene function, disease
gene location, and sequence gene alias. We describe the collection of these
tasks in this section. Table 6 shows several question-answer samples from the
GeneHop dataset.

GeneHop task |  Question |  Answer
---|---|---
SNP gene function |  What is the function of the gene associated with SNP rs1318850293? Let’s decompose the question to sub-questions and solve them step by step. |  Predicted to enable guanyl-nucleotide exchange facto r activity. Predicted to be involved in Rho protein signal transduction.
Disease gene location |  List chromosome locations of the genes related to Hemolytic anemia due to phosphofructokinase deficiency. Let’s decompose the question to sub-questions and solve them step by step. |  21q22.3
Sequence gene alias |  What are the aliases of the gene that contains this sequence: GTAGAT GGAACT GGTAGT CAGCTG GAGAGC AGCATG GAGGCG TCCTGG GGGAGC TTCAAC GCTGAG CGGGGC TGGTAT GTCTCT GTCCAG CAGCCT GAAGAA GCGGAG GCCGA. Let’s decompose the question to sub-questions and solve them step by step. |  SLC38A6, NAT-1, SNAT6
Table 6: Sample question-answer pairs of the GeneHop tasks (introduced in this
work).

#### SNP gene function:

The question template for this task is “What is the function of the gene
associated with SNP {snp}? Let’s decompose the question to sub-questions and
solve them step by step.”. We re-use the 50 {snp} from the gene SNP
association task in the original GeneTuring benchmark. The ground-truth answer
of the gene function is manually annotated: For each SNP, we first get its
corresponding gene from the annotations of the gene SNP association task. We
then check the gene information page888https://www.ncbi.nlm.nih.gov/gene/ and
select its functional summary as the ground-truth answer.

#### Disease gene location:

The question template for this task is “List chromosome locations of the genes
related to {disease}. Let’s decompose the question to sub-questions and solve
them step by step.”. Similarly, we re-use the 50 {disease} from the gene
disease association task in the original GeneTuring benchmark. The ground-
truth list of the chromosome locations is manually annotated: For each
disease, we first get its corresponding genes from the annotations of the gene
disease association task. We then check their NCBI gene information pages and
label the cytogenetics locations (e.g., 21q22.3).

#### Sequence gene alias:

The question template for this task is “What are the aliases of the gene that
contains this sequence: {sequence}. Let’s decompose the question to sub-
questions and solve them step by step.”. We find the information pages of the
50 genes used in the gene alias task in the original GeneTuring benchmark. We
manually crop part of the sequence with a similar length to the sequences in
the GeneTuring alignment tasks to serve as the {sequence}, and use the union
of its official name and the alias set as the ground-truth answer.

##  Appendix D GeneHop evaluation

#### SNP gene function:

We manually evaluate all answers predicted by GeneGPT and the New Bing against
the ground-truth gene functions. New Bing answers with “I’m sorry, but I
couldn’t find any information about the gene associated with SNP {snp}]. Would
you like to know more about SNPs in general?” for all questions, which we
simply 0. To evaluate GeneGPT’s results, we score 1 if (a) the predicted
answer exactly matches the ground-truth or (b) the gene is a non-coding, and
GeneGPT’s answer mentions it; we score 0.5 if there is a partial match, and 0
otherwise.

#### Disease gene location:

Following the evaluation of the gene disease association task in GeneTuring,
we measure the recall of ground-truth chromosome locations. We manually
evaluate all answers given by the New Bing, and consider partial match as
correct. For example, if the new Bing answers “17q21” and the ground-truth
answer is “17q21.2”, we still consider the prediction by new Bing correct. We
automatically evaluate GeneGPT’s prediction under a more strict setting where
we only consider exact matches of GeneGPT as correct.

#### Sequence gene alias:

We manually evaluate all answers predicted by GeneGPT and the New Bing, and
measure the recall of the ground-truth gene aliases. We only consider exact
matches between the predicted alias and a ground truth alias, but ignore the
case difference (e.g., ’Myc’ and ’MYC’ are still considered as a match).

##  Appendix E Error types

#### Error type 1 (E1):

Errors caused by using the wrong API or not using APIs. This only happens in
disease-related tasks where the model uses the gene instead of the omin
database. One example is shown in Figure 11.

Error type 1 (E1)

GeneTuring question (Gene disease association)
What are genes related to Immunodeficiency due to defect in MAPBP-interacting
protein?
Wrong API call
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=gene&retmax=5&retmode=json&sort=relevance&term=Immunodeficiency%20due%20to%20defect%20in%20MAPBP-
interacting%20protein]
Right API call
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=omim&retmax=5&retmode=json&sort=relevance&term=Immunodeficiency%20due%20to%20defect%20in%20MAPBP-
interacting%20protein]

Figure 11: An example of Error type 1 (E1). The wrong API call uses gene
database instead of the omin database used by the right API call.

#### Error type 2 (E2):

Errors caused by using the right API but wrong arguments, e.g., passing terms
to id. One example is shown in Figure 12.

Error type 2 (E2)

GeneTuring question (SNP location)
Which chromosome does SNP rs397784008 locate on human genome?
Wrong API call
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=snp&retmax=5&retmode=json&sort=relevance&id=rs397784008]
Right API call
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=snp&retmax=5&retmode=json&sort=relevance&id=397784008]

Figure 12: An example of Error type 2 (E2). The wrong API call passes all
rs397784008 to the id argument, while the right API should only pass 397784008
without rs to the id argument.

#### Error type 3 (E3):

Errors caused by not extracting the answer in the API result, most commonly
seen in gene function extraction. One example is shown in Figure 13.

Error type 3 (E3)

GeneHop question (SNP gene function)
What is the function of the gene associated with SNP rs1481036795? Let’s
decompose the question to sub-questions and solve them step by step.
Wrong answer (scored 0.5)
SEPTIN11 is a filament-forming cytoskeletal GTPase.

Right answer
SEPT11 belongs to the conserved septin family of filament-forming cytoskeletal
GTPases that are involved in a variety of cellular functions including
cytokinesis and vesicle trafficking (Hanai et al., 2004 [PubMed 15196925];
Nagata et al., 2004 [PubMed 15485874]).

Figure 13: An example of Error type 3 (E3). The wrong answer only contains the
protein family information without other details about its functions.

#### Error type 4 (E4):

The model makes the right API call, but the results do not contain the answer.
These questions are not answerable with the Web APIs. One example is shown in
Figure 13.

Error type 4 (E4)

GeneTuring Question (Gene location)
Which chromosome is AC093802.1 gene located on human genome?
Right API call returns no results
[https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=gene&retmax=5&retmode=json&sort=relevance&term=AC093802.1]

Figure 14: An example of Error type 4 (E4). The model makes the right API call
but there is no entry returned from the API. The gene AC093802.1 is not
indexed by the NCBI gene database.

#### Other errors (O):

Errors that cannot be classified into E1-4 are included in this category,
which are most commonly seen in BLAST-related tasks where the chromosomes are
right but the specific ranges are not matched (e.g., chr8:7081648-7081782 v.s.
chr8:1207812-1207946) because the original GeneTuring benchmark does not
specify a reference genome.


# Lessons from ATLAS and CMS measurements of Higgs boson decays to second
generation fermions

Yosef Nir<EMAIL_ADDRESS>Department of Particle Physics and
Astrophysics, Weizmann Institute of Science, Rehovot 7610001, Israel  Purvaash
Panduranghan Udhayashankar<EMAIL_ADDRESS>Department of Particle Physics and Astrophysics, Weizmann Institute of
Science, Rehovot 7610001, Israel

###### Abstract

There is now experimental evidence for Higgs boson decay into a pair of muons,
and significant constraints on the Higgs boson decay into a charm quark-
antiquark pair. The data on Higgs boson decays into second generation fermions
probes various extensions of the Standard Model. We analyze the implications
for the Standard Model effective field theory (SMEFT), without and with
minimal flavor violation (MFV), for two Higgs doublet models (2HDM) with
natural flavor conservation (NFC), for models with vector-like fermions, and
for specific models that predict significant modifications of the Yukawa
couplings to the light generations.

##  I Introduction

The ATLAS and CMS collaborations measured the decays of the Higgs boson into a
tau-lepton pair, $h\to\tau^{+}\tau^{-}$, and into bottom-quark pair, $h\to
b\bar{b}$, and the Higgs boson production in association with top-quark pair,
$pp\to t\bar{t}h$ CMS:2022dwd ; ATLAS:2022vkf . These measurements constitute
a discovery of the mechanism that gives masses to the third generation
fermions, as well as the discovery of Yukawa interactions among elementary
particles Nir:2020wxg . More recently, the ATLAS and CMS experiments searched
for the Higgs boson decays into second generation fermions.

The conventional way of reporting these measurements is by normalizing the
production cross-section $\sigma(i\to h)$ and the branching ratio of the decay
mode ${\rm BR}(h\to f\bar{f})$ to their Standard Model (SM) values:

$\mu_{f\bar{f}}^{i}=\frac{\sigma(i\to h){\rm BR}(h\to f\bar{f})}{\sigma(i\to
h)^{\rm SM}{\rm BR}(h\to f\bar{f})^{\rm SM}}.$ (1)

One can interpret such measurement in terms of the Yukawa coupling modifier,

$\kappa_{f}=Y_{f}/Y_{f}^{\rm SM}.$ (2)

Assuming that the production cross-section is not modified by new physics, and
that only a single Yukawa coupling $Y_{f}$ is modified, the relation is given
by

$\mu_{f\bar{f}}=\frac{\kappa_{f}^{2}}{1+{\rm BR}^{\rm SM}_{h\to
f\bar{f}}(\kappa_{f}^{2}-1)}.$ (3)

For the $h\to\mu^{+}\mu^{-}$ decay, the experiments report CMS:2020xwi ;
CMS:2022dwd ; ATLAS:2020fzp ; ATLAS:2022vkf

$\displaystyle\mu_{\mu^{+}\mu^{-}}$ $\displaystyle=$ $\displaystyle
1.21^{+0.45}_{-0.42}\ \ \ [{\rm CMS}],$ (4)
$\displaystyle\mu_{\mu^{+}\mu^{-}}$ $\displaystyle=$ $\displaystyle 1.2\pm
0.6\ \ [{\rm ATLAS}],$

with the average result PDG:2022

$\mu_{\mu^{+}\mu^{-}}=1.21\pm 0.35.$ (5)

This can be translated into

$0.92\leq\kappa_{\mu}\leq 1.25.$ (6)

For the $h\to c\bar{c}$ decay, the experiments report CMS:2022psv ;
ATLAS:2022ers

$\displaystyle\mu_{c\bar{c}}$ $\displaystyle=$ $\displaystyle
7.7^{+3.8}_{-3.5}\ \ \ [{\rm CMS}],$ (7) $\displaystyle\mu_{c\bar{c}}^{Vh}$
$\displaystyle=$ $\displaystyle-9\pm 15\ \ [{\rm ATLAS}],$

which translate into the following 95% C.L. bounds:

$\displaystyle 1.1<|\kappa_{c}|<5.5\ \ [{\rm CMS}],$ (8)
$\displaystyle|\kappa_{c}/\kappa_{b}|<4.5\ \ \ [{\rm ATLAS}].$

At $1\sigma$, the CMS measurement translates into

$\kappa_{c}=3.1\pm 1.0.$ (9)

There are two important discoveries implicit in these measurements:

  * •

The muon-related measurement by CMS in Eq. (4) confirms that the dominant
source of the muon mass is the Yukawa interaction with the field of the scalar
$h$.

  * •

The charm-related measurement by ATLAS in Eq. (8) confirms that
$Y_{c}/Y_{b}<1$, in accordance with the SM prediction that
$Y_{c}/Y_{b}=m_{c}/m_{b}$.

The measurements probe, however, also models that go beyond the SM (BSM). In
this work, we explore these implications.

The plan of this paper is as follows. Sections II-VI analyze the implications
of the $\mu_{\mu^{+}\mu^{-}}$ measurements in various relevant BSM frameworks:
The SMEFT in Section II, vector-like leptons in Section III, two Higgs doublet
models (2HDM) in Section IV, the Yukawa-less first-two-generation model in
Section V, and the Higgs-dependent Yukawa couplings model in Section VI.
Section VII studies the implications of $\mu_{c\bar{c}}$ for these models. Our
conclusions are summarized in Section VIII.

##  II The Standard Model effective field theory (SMEFT)

The SM effective field theory (SMEFT) is the most general presentation of all
BSM frameworks where the new degrees of freedom are much heavier than the
electroweak scale, $v\approx 246$ GeV. The SMEFT Lagrangian has the following
form:

${\cal L}_{\rm SMEFT}={\cal L}_{\rm SM}+\frac{1}{\Lambda}{\cal
O}_{d=5}+\frac{1}{\Lambda^{2}}{\cal O}_{d=6}+\ldots.$ (10)

Here, ${\cal L}_{\rm SM}$ is the Lagrangian of the renormalizable SM. The
operators ${\cal O}_{d=n}$ are made of products of SM fields, of overall mass
dimension $n$, contracted to a singlet of the SM gauge group. The scale
$\Lambda$ is the mass scale of the new degrees of freedom, and by assumption
$\Lambda\gg v$.

Beyond the dimension-four Yukawa interaction of the SM, the muon mass and
Yukawa coupling are modified by dimension-six terms:

${\cal L}_{\rm
Yuk}^{\mu}=y_{\mu}\overline{L_{L\mu}}\Phi\mu_{R}+\frac{1}{\Lambda^{2}}(X_{R}^{\mu}+iX_{I}^{\mu})|\Phi|^{2}\overline{L_{L\mu}}\Phi\mu_{R}+{\rm
h.c.}.$ (11)

Here, $L_{L\mu}=(\nu_{\mu}\ \mu_{L})^{T}$ is the left-handed
$SU(2)_{L}$-doublet whose $T_{3}=-1/2$ member is the muon mass eigenstate,
$\mu_{R}$ is the right-handed $SU(2)_{L}$-singlet muon mass eigenstate, and
$\Phi=(0\ (v+h)/\sqrt{2})^{T}$ is the scalar $SU(2)_{L}$ doublet in the
unitary gauge. Without loss of generality,we take the dimensionless coupling
$y_{\mu}$ to be real. It is convenient to define

$T_{R,I}^{\mu}=\frac{v^{2}}{2\Lambda^{2}}\frac{X_{R,I}^{\mu}}{y_{\mu}}.$ (12)

Then, the mass and the effective muon Yukawa coupling are given by

$\displaystyle m_{\mu}$ $\displaystyle=$
$\displaystyle\frac{y_{\mu}v}{\sqrt{2}}\sqrt{(1+T_{R}^{\mu})^{2}+(T_{I}^{\mu})^{2}},$
(13) $\displaystyle Y_{\mu}$ $\displaystyle=$
$\displaystyle\frac{y_{\mu}}{\sqrt{2}}\frac{1+4T_{R}^{\mu}+3(T_{R}^{\mu})^{2}+3(T_{I}^{\mu})^{2}+2iT_{I}^{\mu}}{\sqrt{(1+T_{R}^{\mu})^{2}+(T_{I}^{\mu})^{2}}}.$
(14)

With our assumptions above that the production cross-section is not modified
from the SM, that the muon Yukawa is the only one that is significnatly
modified, neglecting the change in the total Higgs width and the effect of
dimension-eight terms, we obtain Fuchs:2019ore

$\mu_{\mu^{+}\mu^{-}}=\frac{(1+3T_{R}^{\mu})^{2}+9(T_{I}^{\mu})^{2}}{(1+T_{R}^{\mu})^{2}+(T_{I}^{\mu})^{2}}.$
(15)

Thus, Eq. (5) provides an allowed ring in the $T_{R}^{\mu}-T_{I}^{\mu}$ plane,
as shown in Fig. 1.

Figure 1: The allowed region in the $T_{R}^{\mu}-T_{I}^{\mu}$ plane.

Ref. Fuchs:2019ore  estimated the upper bounds on the CP-violating
contributions of the dimension-six term to the baryon asymmetry of the
universe,

$\frac{|Y_{B}^{(\mu)}|}{2\times
10^{-11}}\approx\frac{T_{I}^{\mu}}{(1+T_{R}^{\mu})^{2}+(T_{I}^{\mu})^{2}}\leq\frac{\sqrt{\mu^{\rm
max}_{\mu^{+}\mu^{-}}}}{2},$ (16)

and to the electric dipole moment (EDM) of the electron:

$\frac{|d_{e}^{(\mu)}|}{1\times 10^{-30}\ e\ {\rm
cm}}\approx\frac{T_{I}^{\mu}}{(1+T_{R}^{\mu})^{2}+(T_{I}^{\mu})^{2}}\leq\frac{\sqrt{\mu^{\rm
max}_{\mu^{+}\mu^{-}}}}{2},$ (17)

where $\mu^{\rm max}_{\mu^{+}\mu^{-}}$ is the experimental upper bound on
$\mu_{\mu^{+}\mu^{-}}$. Given the observed value of the baryon asymmetry,
$Y_{B}^{\rm obs}=(8.63\pm 0.06)\times 10^{-11}$ PDG:2022 , a complex Yukawa
coupling of the muon can contribute at most (at the $1\sigma$ level)

$|Y_{B}^{(\mu)}/Y_{B}^{\rm obs}|\mathrel{\hbox to0.0pt{\lower
4.0pt\hbox{\hskip 1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$<$}}0.15.$ (18)

Given the experimental upper bound, $|d_{e}^{\rm max}|=4.1\times 10^{-30}\ e\
{\rm cm}$ Roussy:2022cmp  (see also ACME:2018yjb ), a complex Yukawa of the
muon is at most a factor of

$|d_{e}^{(\mu)}/d_{e}^{\rm max}|\mathrel{\hbox to0.0pt{\lower
4.0pt\hbox{\hskip 1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$<$}}0.15,$ (19)

below current experimental sensitivity.

The allowed range, which includes the SM value, $\mu_{\mu^{+}\mu^{-}}=1$, puts
a lower bound on the scale of new physics that suppresses the dimension-six
term. As long as $\mu_{\mu^{+}\mu^{-}}=1$ is within the experimental range,
the point $(T_{R}^{\mu},T_{I}^{\mu})=(-0.5,0)$, corresponding to
$|T^{\mu}|=0.5$, is allowed, which implies

$\frac{\Lambda}{\sqrt{|X^{\mu}|}}\mathrel{\hbox to0.0pt{\lower
4.0pt\hbox{\hskip 1.0pt$\sim$}\hss}\raise
1.0pt\hbox{$>$}}\frac{v}{\sqrt{2y_{\mu}^{\rm SM}|T^{\mu,{\rm max}}|}}\approx
10\ {\rm TeV},$ (20)

where $y_{\mu}^{\rm SM}=\sqrt{2}m_{\mu}/v\approx 6\times 10^{-4}$. Since
$|T|^{2}\propto 1/\Lambda^{4}$, small deviations from $\mu_{\mu^{+}\mu^{-}}=1$
do not change this bound significantly.

In our analysis so far we assumed no special flavor structure for $X_{R,I}$.
Imposing minimal flavor violation (MFV) implies that we expect
$X_{R,I}^{\mu}={\cal O}(y_{\mu})$ (and if we take MFV to imply that that
Yukawa spurion is the only souce of CP violation, $X_{I}^{\mu}=0$). In this
framework, $T_{R}^{\mu}={\cal O}(v^{2}/\Lambda^{2})$, and consequently the
bound on $\Lambda$ relaxes to $\Lambda\mathrel{\hbox to0.0pt{\lower
4.0pt\hbox{\hskip 1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$>$}}v/\sqrt{(\mu^{\rm
max}_{\mu^{+}\mu^{-}}-1)/2}\sim 0.5\ {\rm TeV}$.

##  III Vector-like leptons

Vector-like leptons affect the Higgs Yukawa couplings to pairs of the SM
leptons if there are Yukawa terms involving a vector-like lepton and a SM
lepton. There are five such vector-like representations, presented in Table 1.
We follow here the notations and analysis of Ref. Blum:2015rpa .

Names |  $SU(2)_{L}\times U(1)_{Y}$ rep’s | $2M_{X}^{2}(c_{Y_{e}})_{\mu\mu}$ | $\kappa_{\mu}-1=$
---|---|---|---
$E+E^{c}$ | $(1)_{-1}+(1)_{+1}$ | $y_{\mu}|Y_{\mu_{L}E^{c}}|^{2}$ | $-2\delta g_{A\mu}=-2\delta g_{V\mu}$
$L+L^{c}$ | $(2)_{-1/2}+(2)_{+1/2}$ | $y_{\mu}|Y_{L\mu_{R}}|^{2}$ | $-2\delta g_{A\mu}=+2\delta g_{V\mu}$
$L^{\prime}+L^{\prime c}$ | $(2)_{-3/2}+(2)_{+3/2}$ | $y_{\mu}|Y_{L^{\prime}\mu_{R}}|^{2}$ | $+2\delta g_{A\mu}=-2\delta g_{V\mu}$
$T+T^{c}$ | $(3)_{0}+(3)_{0}$ | $2y_{\mu}|Y_{\mu_{L}T^{c}}|^{2}$ | $+2\delta g_{A\mu}=+2\delta g_{V\mu}$
$T^{\prime}+T^{\prime c}$ | $(3)_{-1}+(3)_{+1}$ | $y_{\mu}|Y_{\mu_{L}T^{\prime c}}|^{2}$ | $-2\delta g_{A\mu}=-2\delta g_{V\mu}$
Table 1: The leptonic vector-like representations. All fields are $SU(3)_{C}$
singlets.

Taking the vector-like leptons to be much heavier than the electroweak scale,
we integrate out these fields. This leads to, among others, the dimension-six
term of Eq. (11), with a coefficient that we now denote by
$(c_{Y_{e}})_{\mu\mu}$. In the presence of a single vector-like
representation, the corresponding $(c_{Y_{e}})_{\mu\mu}$ is given in Table 1.
The dimension-six terms contribute to the mass and the Yukawa coupling of the
muon, and the relation between them is modified as follows:

$Y_{\mu}=\frac{\sqrt{2}m_{\mu}}{v}+v^{2}(c_{Y_{e}})_{\mu\mu}.$ (21)

Each of the five vector-like representations modifies also the vector
($g_{V\mu}$) and axial-vector ($g_{A\mu}$) $Z\mu^{+}\mu^{-}$ couplings. This
modification is related to the modification of the Yukawa coupling as given in
the fourth column ot Table 1 Blum:2015rpa . We denote the modifications of the
$Z$ couplings as follows:

$\delta g_{V\mu}=g_{V\mu}-g_{V\mu}^{\rm SM},\ \ \ \delta
g_{A\mu}=g_{A\mu}-g_{A\mu}^{\rm SM}.$ (22)

We use PDG:2022 ; ALEPH:2005ab

$\displaystyle g_{V\mu}$ $\displaystyle=$ $\displaystyle-0.0367\pm 0.0023,\ \
g_{V\mu}^{\rm SM}=-0.0371\pm 0.0003$ (23) $\displaystyle g_{A\mu}$
$\displaystyle=$ $\displaystyle-0.50120\pm 0.00054,\ \ g_{A\mu}^{\rm
SM}=-0.50127\pm 0.00020,$

and obtain

$\delta g_{V\mu}=+0.0004\pm 0.0023,\ \ \ \delta g_{A\mu}=+0.00007\pm 0.00060.$
(24)

Combining Table 1 and Eq. (24), we learn that in these five models,

$|\kappa_{\mu}-1|\mathrel{\hbox to0.0pt{\lower 4.0pt\hbox{\hskip
1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$<$}}0.0013,$ (25)

two orders of magnitude below the experimental sensitivity, as reflected in
Eq. (6).

A new ingredient is added in models that include two different vector-like
representations in each, where there is a Yukawa coupling involving these two
vector-like fields. These five models are presented in Table 2.

Model | Fields | $\kappa_{\mu}-1=$
---|---|---
LI | $L+L^{c}+E+E^{c}$ | $-2\delta g_{A\mu}+X_{\mu}|Y_{L^{c}E}|$
LII | $L^{\prime}+L^{\prime c}+E+E^{c}$ | $-2\delta g_{V\mu}-X_{\mu}|Y_{L^{\prime c}E}|$
LIII | $L+L^{c}+T^{\prime}+T^{\prime c}$ | $-2\delta g_{V\mu}-X_{\mu}|Y_{L^{c}T^{\prime}}|$
LIV | $L^{\prime}+L^{\prime c}+T^{\prime}+T^{\prime c}$ | $-2\delta g_{A\mu}+X_{\mu}|Y_{L^{\prime c}T^{\prime}}|$
LV | $L+L^{c}+T+T^{c}$ | $+2\delta g_{V\mu}+X_{\mu}|Y_{L^{c}T}|$
Table 2: Models with pairs of leptonic vector-like representations. $X_{\mu}$
is defined in Eq. (26).

In these models, the experimental ranges of $\delta g_{V\mu}$, $\delta
g_{A\mu}$ and $(\kappa_{\mu}-1)$ constrain the combination
$X_{\mu}Y_{F_{1}F_{2}}$, where $Y_{F_{1}F_{2}}$ is the Yukawa coupling between
two vector-like leptons, and

$X_{\mu}=(v/m_{\mu})\sqrt{2|\delta g_{V\mu}^{2}-\delta
g_{A\mu}^{2}|}\exp(i\phi),$ (26)

and $\phi$ is a convention-independent phase.

Using the relations in Table 2, and given that the allowed values of $\delta
g_{V\mu}$ and $\delta g_{A_{\mu}}$ are about two orders of magnitude smaller
than the bounds on $\kappa_{\mu}-1$, we obtain the following approximate
bound:

$Y_{F_{1}F_{2}}\sqrt{|\delta g_{V\mu}^{2}-\delta g_{A\mu}^{2}|}\mathrel{\hbox
to0.0pt{\lower 4.0pt\hbox{\hskip 1.0pt$\sim$}\hss}\raise
1.0pt\hbox{$<$}}3\times 10^{-4}\times|\kappa_{\mu}-1|_{\rm max}\sim 8\times
10^{-5},$ (27)

where we use the upper bound on $\kappa_{\mu}-1$ from Eq. (6). Comparing Eq.
(27) to Eq. (24), we learn that, in the models defined in Table 2, for
$Y_{F_{1}F_{2}}>{\cal O}(0.03)$, the $h\to\mu^{+}\mu^{-}$ measurement can be
as constraining as measurements of $Z\to\mu^{+}\mu^{-}$.

##  IV Two Higgs Doublet Models (2HDM)

We consider two Higgs doublet models (2HDM) with natural flavor conservation
(NFC). Common to all such models is the prediction that

$\mu_{\mu^{+}\mu^{-}}=\mu_{\tau^{+}\tau^{-}}.$ (28)

For the $h\to\tau^{+}\tau^{-}$ decay, the experiments report CMS:2022dwd ;
ATLAS:2022yrq

$\displaystyle\mu_{\tau^{+}\tau^{-}}$ $\displaystyle=$ $\displaystyle 0.85\pm
0.10\ \ \ [{\rm CMS}],$ (29) $\displaystyle\mu_{\tau^{+}\tau^{-}}$
$\displaystyle=$ $\displaystyle 0.93^{+0.13}_{-0.12}\ \ [{\rm ATLAS}],$

with the average result

$\mu_{\tau^{+}\tau^{-}}=0.88\pm 0.08,$ (30)

which translates into

$\kappa_{\tau}=0.935\pm 0.045.$ (31)

In this framework, we can combine Eq. (5) with Eq. (30) to obtain

$\mu_{\ell^{+}\ell^{-}}=0.90\pm 0.08.$ (32)

Comparing Eq. (30) to Eq. (32), it is clear that the effect of the
$\mu_{\mu^{+}\mu^{-}}$ measurement on the constraints on the model parameters
is minor.

In 2HDM with NFC, there are three convenient bases. The interaction basis
$(\Phi_{1},\Phi_{2})$, where each fermion sector ($u,d,e$) couples to one of
the two doublets, the Higgs basis $(\Phi_{M},\Phi_{A})$, defined via
$\langle\Phi_{M}\rangle=v$ and $\langle\Phi_{A}\rangle=0$, and the mass basis
$(\Phi_{h},\Phi_{H})$, where the neutral CP-even members of the two doublets
are the mass eigenstates $h$ and $H$. The angle $\beta$, defined via
$\tan\beta=v_{2}/v_{1}$, rotates from the interaction basis to the Higgs
basis, while the angle $\alpha$ rotates from the interaction basis to the mass
basis.

We demonstrate the smallness of the effect of the muon data in the lepton-
specific type of NFC, where the quark ($u,d$) sectors couple to $\Phi_{2}$ and
the lepton ($e$) sector couples to $\Phi_{1}$. Fig. 2 gives the excluded
region in the $\cos(\beta-\alpha)-\tan\beta$ plane, without (purple) and with
(purple plus pink) the muon constraint. We zoom in on the regions of
$\cos(\beta-\alpha)\approx 0$ (the decoupling limit) and
$\cos(\beta+\alpha)\approx 0$ (where the lepton Yukawa couplings are of
opposite sign to the SM values).

Figure 2: The excluded region in the $\cos(\beta-\alpha)-\tan\beta$ plane, in
the lepton-specifc NFC model. The purple-shaded region does not include the
$\mu_{\mu^{+}\mu^{-}}$ measurement, while the pink-shaded region is further
excluded by this measurement.

##  V The Yukawa-less first-two-generation model

The Yukawa-less first-two-generation model of Ref. Ghosh:2015gpa  was
proposed, first, to demonstrate that it is possible that the mechanism that
generates the masses of the first two generation fermions is different from
the one that generates the masses of the third generation fermions, second, to
provide an explanation for the lightness of the first two generations and,
third, to demonstrate that it is possible that the two different mechanisms
for fermion masses can be manifest in measurements of the Higgs boson decays
to the light generations.

Consider a 2HDM. One of the two doublets, $\Phi_{3}$, couples dominantly to
the third generation:

${\cal
L}_{3}^{Y}=-\overline{L_{Li}}(Y_{\tau}\delta_{i3}\delta_{j3}+\epsilon^{e}_{ij})\Phi_{3}E_{Rj}+{\rm
h.c.},$ (33)

where $|\epsilon^{e}_{ij}|\ll 1$ (and similarly in the quark sectors). The
second doublet, $\Phi_{12}$, couples to only the first two generations:

${\cal
L}_{12}^{Y}=-\sum_{i,j=1,2}\overline{L_{Li}}Y^{e12}_{ij}\Phi_{12}E_{Rj}+{\rm
h.c.}.$ (34)

The scalar potential is given by

$V=\mu_{1}^{2}|\Phi_{12}|^{2}+\mu_{2}^{2}|\Phi_{3}|^{2}+\mu^{2}(\Phi_{12}^{\dagger}\Phi_{3}+{\rm
h.c.})+\lambda_{1}|\Phi_{12}|^{4}+\lambda_{2}|\Phi_{3}|^{4}.$ (35)

The VEVs, $\langle\Phi_{3}\rangle=v_{3}$ and $\langle\Phi_{12}\rangle=v_{12}$,
are hierarchical:

$\tan\beta=v_{3}/v_{12}\gg 1,\ \ \ v_{3}^{2}+v_{12}^{2}=v^{2}.$ (36)

The scalar spectrum is given by

$\displaystyle m_{H^{\pm}}^{2}$ $\displaystyle=$ $\displaystyle
m_{A}^{2}=-\frac{\mu^{2}v^{2}}{v_{12}v_{3}},$ (37) $\displaystyle m^{2}_{h,H}$
$\displaystyle=$
$\displaystyle\lambda_{1}v_{12}^{2}+\lambda_{2}v_{3}^{2}-\frac{\mu^{2}v^{2}}{2v_{12}v_{3}}\mp\sqrt{\left(\lambda_{1}v_{12}^{2}-\lambda_{2}v_{3}^{2}+\frac{\mu^{2}(v_{12}^{2}-v_{3}^{2})}{2v_{12}v_{3}}\right)^{2}+(\mu^{2})^{2}},$

and the rotation angle from the $(\Phi_{12},\Phi_{3})$ basis to the
$(\Phi_{h},\Phi_{H})$ basis, $\alpha$, is given by

$\tan
2\alpha=-\frac{2m_{A}^{2}s_{\beta}c_{\beta}}{\sqrt{(m_{H}^{2}-m_{h}^{2})^{2}-4m_{A}^{4}s_{\beta}^{2}c_{\beta}^{2}}}.$
(38)

Clearly, in this model, third generation fermions acquire their masses from
interacting with the $\Phi_{3}$ field, while the first two generation fermions
acquire their masses from interacting with $\Phi_{12}$. The question remains
whether this situation will be manifest in the Yukawa couplings of the
lighter, CP-even neutral scalar $h$. In the limit of very large $\mu$ and, in
particular, $m_{A}^{2}\gg\lambda_{2}v^{2},\ \lambda_{1}v_{12}^{2}$, we are in
the decoupling limit, and $\kappa_{f}\simeq 1$ for all fermions, including the
light generations. Indeed, it is straightforward to show that, in this limit,
Eq. (38) implies that $s_{\alpha}\simeq-c_{\beta}$, and then
$\kappa_{\mu}=-s_{\alpha}/c_{\beta}\simeq 1$ and
$\kappa_{\tau}=c_{\alpha}/s_{\beta}\simeq 1$.

The situation is different in the case that $\lambda_{1}v^{2}c_{\beta}^{2}\gg
m_{A}^{2},\ \lambda_{2}v^{2}$ Ghosh:2015gpa . Given the smallness of
$c_{\beta}^{2}$, this hierarchy requires a very large $\lambda_{1}$. Ref.
Ghosh:2015gpa  argues that a natural embedding for this setup is a strongly
coupled theory like technicolor. Eqs. (37) and (38) give

$\displaystyle m_{h}^{2}$ $\displaystyle=$ $\displaystyle 2\lambda_{2}v^{2},$
(39) $\displaystyle m_{H}^{2}$ $\displaystyle=$ $\displaystyle
2\lambda_{1}v^{2}c_{\beta}^{2}+m_{A}^{2}\gg m_{A}^{2},$ $\displaystyle
s_{\alpha}$ $\displaystyle=$ $\displaystyle-(m_{A}^{2}/m_{H}^{2})c_{\beta}\ll
c_{\beta}.$

Consequently,

$\kappa_{\mu}\approx m_{A}^{2}/m_{H}^{2}\ll 1.$ (40)

The lower bound in Eq. (6) implies that, in contrast to Eq. (40),
$\kappa_{\mu}\not\ll 1$ which, in the context of this model, implies that
$\lambda_{1}v^{2}c_{\beta}^{2}\not\gg m_{A}^{2}$, and the Yukawa-less first-
two-generation model is excluded.

Related models are the flavorful 2HDMs of Ref. Altmannshofer:2018bch . In
their Type 1B 2HDM, the third generation fermions of all three sectors couple
to $\Phi$, which has Yukawa couplings only in the $Y^{f}_{33}$ entries, while
the first two generations couple to $\Phi^{\prime}$, which has small Yukawa
couplings that break the $[U(2)]^{5}$ symmetry acting on the first two
generations. Consequently, the measurement of $\mu_{\mu^{+}\mu^{-}}$ is the
first to probe the couplings of $\Phi^{\prime}$. Yet, the effect of this
measurement on the allowed region in the $\cos(\beta-\alpha)-\tan\beta$ plane
is small, as can be seen in Fig. 3.

Figure 3: The allowed region in the $\cos(\beta-\alpha)-\tan\beta$ plane, in
1B 2HDM model of Ref. Altmannshofer:2018bch . When adding the
$\mu_{\mu^{+}\mu^{-}}$ measurement to all previous measurements of Higgs
couplings, the previously allowed purple-shaded region is excluded, while the
yellow-shaded region opens up. The grey region remains unchanged.

##  VI Higgs dependent Yukawa couplings

The models of Refs. Giudice:2008uua ; Botella:2016krk  aim to explain the
hierarchy in fermion masses by having the light fermion masses dominated by
higher order temrs, e.g.,

${\cal L}^{\mu}_{\rm
Yuk}=c_{\mu\mu}\left(\frac{\Phi^{\dagger}\Phi}{\Lambda^{2}}\right)^{n_{\mu}}\overline{L_{L\mu}}\Phi\mu_{R}+{\rm
h.c.},$ (41)

where $c_{\mu\mu}$ is a dimensionless coefficient of ${\cal O}(1)$, $\Lambda$
is a cut-off scale of ${\cal O}({\rm TeV})$, and $n_{\mu}$ is an integer $\geq
1$. The muon mass and Yukawa coupling are given by

$\displaystyle m_{\mu}$ $\displaystyle=$ $\displaystyle
c_{\mu\mu}\left(\frac{v}{\sqrt{2}\Lambda}\right)^{2n_{\mu}}\frac{v}{\sqrt{2}},$
(42) $\displaystyle Y_{\mu}$ $\displaystyle=$ $\displaystyle
c_{\mu\mu}(2n_{\mu}+1)\left(\frac{v}{\sqrt{2}\Lambda}\right)^{2n_{\mu}}.$

Given that, experimentally, neither $\kappa_{b}\gg 1$, nor $\kappa_{\tau}\gg
1$, we assume that $n_{b}=n_{\tau}=0$. Then,

$\displaystyle\kappa_{\mu}$ $\displaystyle=$ $\displaystyle 2n_{\mu}+1,$ (43)
$\displaystyle\mu_{\mu^{+}\mu^{-}}$ $\displaystyle=$
$\displaystyle\frac{(2n_{\mu}+1)^{2}}{1+{\rm BR}_{c,{\rm
SM}}[(2n_{c}+1)^{2}-1]}.$

Given that $m_{c}\gg m_{\mu}$, we assume that $n_{c}\leq n_{\mu}$. Thus, the
minimal enhancement of $\mu_{\mu^{+}\mu^{-}}$ in this framework corresponds to
$n_{\mu}=n_{c}=1$:

$\mu_{\mu^{+}\mu^{-}}\geq\frac{9}{1+0.03\times 8}\approx 7.3.$

This value is clearly excluded by the experimental measurement of Eq. (5).

We conclude that the models of Higgs dependent Yukawa couplings of Refs.
Giudice:2008uua ; Botella:2016krk  are excluded by the measurement of
$\mu_{\mu^{+}\mu^{-}}$.

Another model where the Yukawa couplings are Higgs-dependent is the 2HDM model
of Ref. Bauer:2015fxa . We postpone the discussion of this model to Section
VII.4, in order to include the top and charm data.

##  VII Implications of $\mu_{c\bar{c}}$

The measurement of $\mu_{c\bar{c}}$ suffers from low statistics. One should
therefore be cautious about the SM value of $\kappa_{c}=1$ being outside the
$95\%$ C.L. interval. Of course, if, with higher statistics, $\kappa_{c}=1$
will be exlcuded with high enough C.L., it will have striking implications.
Here, however, we mention various implications for the BSM frameworks
discussed in the previous sections, taking Eq. (8) at face value.

###  VII.1 The SMEFT

We use notations similar to the discussion in section II (replacing $\mu$ with
$c$). There are several intriguing implications of the range of $\kappa_{c}$
in Eq. (8):

  * •

The range of $\kappa_{c}$ in Eq. (8) corresponds to

$1.3\ {\rm TeV}\mathrel{\hbox to0.0pt{\lower 4.0pt\hbox{\hskip
1.0pt$\sim$}\hss}\raise
1.0pt\hbox{$<$}}\frac{\Lambda}{\sqrt{|X^{c}|}}\mathrel{\hbox to0.0pt{\lower
4.0pt\hbox{\hskip 1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$<$}}9.1\ {\rm TeV}.$
(44)

Thus, first, since $\kappa_{c}=1$ is excluded, we obtain also an upper bound
on the scale $\Lambda$ and, second, the relevant new physics can be within the
direct reach of the LHC.

  * •

The value of $\kappa_{c}=3$ is within the allowed range. This value
corresponds to a situation where the contribution of the dimension-six term to
$m_{c}$ dominates over the SM, dimension-four contribution.

  * •

Imposing MFV implies that we expect $X_{R,I}^{c}={\cal O}(y_{c})$. In this
case, the scale of new physics is required to be rather low,
$\Lambda\mathrel{\hbox to0.0pt{\lower 4.0pt\hbox{\hskip
1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$<$}}0.8\ {\rm TeV}$.

###  VII.2 Vector-like up-type quarks

There are four representations of vector-like up-quarks that allow a Yukawa
coupling to SM fields, presented in Table 3. We follow here the notations and
analysis of Ref. Blum:2015rpa .

Names |  $SU(3)_{C}\times SU(2)_{L}\times U(1)_{Y}$ rep’s | $2M_{X}^{2}(c_{Y_{u}})_{cc}$ | $\kappa_{c}-1=$
---|---|---|---
$Q+Q^{c}$ | $(3,2)_{+1/6}+(\bar{3},2)_{-1/6}$ | $y_{c}|Y_{Qc_{R}}|^{2}$ | $+2\delta g_{Ac}=-2\delta g_{Vc}$
$Q^{\prime\prime}+Q^{\prime\prime c}$ | $(3,2)_{+7/6}+(\bar{3},2)_{-7/6}$ | $y_{c}|Y_{Q^{\prime\prime}c_{R}}|^{2}$ | $+2\delta g_{Ac}=-2\delta g_{Vc}$
$U+U^{c}$ | $(3,1)_{+2/3}+(\bar{3},1)_{-2/3}$ | $y_{c}|Y_{c_{L}U^{c}}|^{2}$ | $+2\delta g_{Ac}=+2\delta g_{Vc}$
$U^{\prime}+U^{\prime c}$ | $(3,3)_{+2/3}+(\bar{3},3)_{-2/3}$ | $y_{c}|Y_{c_{L}U^{\prime c}}|^{2}$ | $-2\delta g_{Ac}=-2\delta g_{Vc}$
Table 3: The up-quark vector-like representations.

Each of the four vector-like representations modifies also the vector
($g_{Vc}$) and axial-vector ($g_{Ac}$) $Zc\bar{c}$ couplings. This
modification is related to the modification of the Yukawa coupling as given in
the fourth column ot Table 3 Blum:2015rpa . We denote the modifications of the
$Z$ couplings as follows:

$\delta g_{Vc}=g_{Vc}-g_{Vc}^{\rm SM},\ \ \ \delta g_{Ac}=g_{Ac}-g_{Ac}^{\rm
SM}.$ (45)

We use ALEPH:2005ab

$\displaystyle g_{Vc}$ $\displaystyle=$ $\displaystyle+0.1873\pm 0.0070,\ \
g_{Vc}^{\rm SM}=+0.19204\pm 0.00023$ (46) $\displaystyle g_{Ac}$
$\displaystyle=$ $\displaystyle+0.5034\pm 0.0053,\ \ g_{Ac}^{\rm
SM}=+0.50144\pm 0.00020,$

and obtain

$\delta g_{Vc}=-0.0037\pm 0.0070,\ \ \ \delta g_{Ac}=+0.0020\pm 0.0053.$ (47)

Combining Table 3 and Eq. (47), we learn that in these four models,

$|\kappa_{c}-1|\mathrel{\hbox to0.0pt{\lower 4.0pt\hbox{\hskip
1.0pt$\sim$}\hss}\raise 1.0pt\hbox{$<$}}0.015,$ (48)

two orders of magnitude below the experimental sensitivity, as reflected in
Eq. (8).

A new ingredient is added in models that include two different vector-like
representations in each, where there is a Yukawa coupling involving these two
vector-like fields. These four models are presented in Table 4.

Model | Fields | $\kappa_{c}-1=$
---|---|---
UI | $Q+Q^{c}+U+Y^{c}$ | $+2\delta g_{Ac}-X_{c}|Y_{Q^{c}U}|$
UII | $Q^{\prime\prime}+Q^{\prime\prime c}+U+U^{c}$ | $+2\delta g_{Vc}+X_{c}|Y_{Q^{\prime\prime U}E}|$
UIII | $Q+Q^{c}+U^{\prime}+U^{\prime c}$ | $-2\delta g_{Vc}+X_{c}|Y_{Q^{c}U^{\prime}}|$
UIV | $Q^{\prime\prime}+Q^{\prime\prime c}+U^{\prime}+U^{\prime c}$ | $-2\delta g_{Ac}-X_{c}|Y_{Q^{\prime\prime c}U^{\prime}}|$
Table 4: Models with pairs of up-quark vector-like representations. $X_{c}$ is
defined in Eq. (49).

In these models, the experimental ranges of $\delta g_{Vc}$, $\delta g_{Ac}$
and $(\kappa_{c}-1)$ constrain the combination $X_{c}Y_{F_{1}F_{2}}$, where
$Y_{F_{1}F_{2}}$ is the Yukawa coupling of two vector-like up-quarks, and

$X_{c}=(v/m_{c})\sqrt{2|\delta g_{Vc}^{2}-\delta g_{Ac}^{2}|}\exp(i\phi),$
(49)

and $\phi$ is a convention-independent phase.

Using the relations in Table 4, and given that the allowed values of $\delta
g_{Vc}$ and $\delta g_{A_{c}}$ are about two orders of magnitude smaller than
the bounds on $\kappa_{c}-1$, we obtain the following approximate bound:

$Y_{F_{1}F_{2}}\sqrt{|\delta g_{Vc}^{2}-\delta g_{Ac}^{2}|}\mathrel{\hbox
to0.0pt{\lower 4.0pt\hbox{\hskip 1.0pt$\sim$}\hss}\raise
1.0pt\hbox{$<$}}4\times 10^{-3}\times|\kappa_{c}-1|_{\rm max}\sim 0.02,$ (50)

where we use the upper bound on $\kappa_{c}-1$ from Eq. (8). Comparing Eq.
(50) to Eq. (47), we learn that, in the models defined in Table 4, for
$Y_{F_{1}F_{2}}>{\cal O}(1)$, the $h\to c\bar{c}$ measurement can be as
constraining as measurements of $Z\to c\bar{c}$.

If $\kappa_{c}-1>0.1$ is established, then the models with vector-like up-
quarks can explain the deviation from the SM with $Y_{F_{1}F_{2}}\sqrt{|\delta
g_{Vc}^{2}-\delta g_{Ac}^{2}|}\sim 4\times 10^{-4}$.

###  VII.3 2HDM with NFC

Common to all 2HDMs with NFC is the prediction that

$\kappa_{c}=\kappa_{t}.$ (51)

For the $t\bar{t}h$ production, the experiments report CMS:2022dwd ;
CMS:2020mpn ; ATLAS:2017ztq

$\displaystyle\mu_{t\bar{t}h}$ $\displaystyle=$ $\displaystyle
0.94^{+0.20}_{-0.19}\ \ \ [{\rm CMS}],$ (52) $\displaystyle\mu_{t\bar{t}h}$
$\displaystyle=$ $\displaystyle 1.2\pm 0.3\ \ [{\rm ATLAS}],$

with the average result PDG:2022

$\mu_{t\bar{t}h}=1.10\pm 0.18,$ (53)

which translates into

$\kappa_{t}=1.05\pm 0.09.$ (54)

Taking the naive average of Eq. (9) and Eq. (54), we obtain

$\kappa_{u}=1.07\pm 0.09.$ (55)

Comparing Eq. (54) to Eq. (55), it is clear that the effect of the
$\mu_{c\bar{c}}$ measurement on the constraints on the model parameters is
minor.

###  VII.4 Models with significantly modified second generation Yukawa
couplings

In the Yukawa-less first-two-generations model of Ref. Ghosh:2015gpa , the
Yukawa couplings of the Higgs boson to the first two generations are strongly
suppressed compared to their SM values. In the Higgs-dependent Yukawa
couplings models of Refs. Giudice:2008uua ; Botella:2016krk , the Yukawa
couplings of the Higgs boson to the first two generations are strongly
enhanced compared to their SM values. As discussed in Sections V and VI, these
two models are excluded by the experimental value of $\mu_{\mu^{+}\mu^{-}}$
perez:2022 . Thus, we do not discuss them any further here.

Another model where the Yukawa couplings are Higgs-dependent is the 2HDM model
of Ref. Bauer:2015fxa . In this model, we have

$\displaystyle\kappa_{t}$ $\displaystyle=$ $\displaystyle
s_{\beta-\alpha}+c_{\beta-\alpha}/t_{\beta},$ (56)
$\displaystyle\kappa_{\tau}$ $\displaystyle=$
$\displaystyle\kappa_{b}=3s_{\beta-\alpha}-2c_{\beta-\alpha}t_{\beta}+c_{\beta-\alpha}/t_{\beta},$
$\displaystyle\kappa_{c}$ $\displaystyle=$ $\displaystyle
3s_{\beta-\alpha}-c_{\beta-\alpha}t_{\beta}+2c_{\beta-\alpha}/t_{\beta},$
$\displaystyle\kappa_{\mu}$ $\displaystyle=$ $\displaystyle
5s_{\beta-\alpha}-3c_{\beta-\alpha}t_{\beta}+2c_{\beta-\alpha}/t_{\beta}.$

We learn that $\kappa_{c}$ and $\kappa_{\mu}$ can be expressed in terms of
$\kappa_{t}$ and $\kappa_{\tau}$, and, consequently, the former are predicted
in terms of the latter:

$\displaystyle\kappa_{c}$ $\displaystyle=$
$\displaystyle\frac{1}{2}(\kappa_{\tau}+3\kappa_{t})=2.0\pm 0.2,$ (57)
$\displaystyle\kappa_{\mu}$ $\displaystyle=$
$\displaystyle\frac{1}{2}(3\kappa_{\tau}+\kappa_{t})=1.94\pm 0.08.$

While the value of $\kappa_{c}\sim 2$ is within the currently allowed range,
the value of $\kappa_{\mu}$ corresponds to $\mu_{\mu^{+}\mu^{-}}=3.5\pm 0.3$,
which is inconsistent with Eq. (5). We conclude that also this model is
excluded.

##  VIII Conclusions

The ATLAS and CMS experiments at the LHC have been measuring branching ratios
of the Higgs boson decays into second generation fermions,
$h\to\mu^{+}\mu^{-}$ and $h\to c\bar{c}$. These measurements probe various BSM
frameworks, where the SM relation between the Yukawa coupling and the mass,
$y_{f}=\sqrt{2}m_{f}/v$, is violated for second generation fermions.

We obtain the following lessons from the $\mu_{\mu^{+}\mu^{-}}=1.21\pm 0.35$
measurement:

  * •

Models where there is a strong suppression of the Higgs boson Yukawa coupling
to second generation fermions, such as the Yukawa-less first-two-generations
model of Ref. Ghosh:2015gpa , are excluded.

  * •

Models where there is a strong enhancement of the Higgs boson Yukawa coupling
to second generation fermions, such as the Higgs-dependent Yukawa couplings
models within either a single Higgs doublet Giudice:2008uua ; Botella:2016krk
or two Higgs doublets Bauer:2015fxa , are excluded.

  * •

In models where the SM fermion spectrum is extended by the addition of a
single leptonic vector-like representation, the Higgs boson decay constraints
are not competitive with the $Z$-boson decay constraints. In models with two
vector-like representations, the two sets of constraints are complementary.

  * •

In 2HDMs with natural flavor conservation, constraints from
$h\to\tau^{+}\tau^{-}$ are only slightly improved.

  * •

In the SMEFT, there is a lower bound on the scale that suppresses the relevant
dimension-six terms of order 10 TeV. (The bound will not get stronger with
better accuracy, as long as $\mu_{\mu^{+}\mu^{-}}=1$ remains within the
allowed range.)

The CMS measurement of $\mu_{c\bar{c}}=7.7^{+3.8}_{-3.5}$ does not include the
SM prediction, $\mu_{c\bar{c}}=1$, within the $95\%$ C.L. allowed range. Given
the limited statistics, one should be cautious about its interpretation. Taken
at face value, we draw the following new lessons;

  * •

Within the SMEFT, the scale of new physics should be within the ($1\sigma$)
range of $1.3-9.1$ TeV.

  * •

In particular, models with two vector-like up-quark representations, can
account from enhancement of the charm Yukawa coupling, if the Yukawa coupling
involving the two vector-like representations is large enough, and with
interesting implications for modifications of the $Zc\bar{c}$ coupling.

  * •

In 2HDM models with NFC, the constraints on the top Yukawa coupling from
$\mu_{t\bar{t}h}$ are in $\sim 2\sigma$ tension with the $\mu_{c\bar{c}}$
constraint.

One of the most intriguing open questions related to the Higgs boson has been
whether the second generation fermions acquire their masses through the Yukawa
interaction with the field of the observed scalar $h$. For the muon, the
$\mu_{\mu^{+}\mu^{-}}$ measurment now strongly supports that this is indeed
the dominant mechanism. For the charm quark, the $\mu_{c\bar{c}}$ measurement
still leaves open the possibility that this is not the case.

## Acknowledgements

YN is the Amos de-Shalit chair of theoretical physics, and is supported by
grants from the Israel Science Foundation (grant number 1124/20), the United
States-Israel Binational Science Foundation (BSF), Jerusalem, Israel (grant
number 2018257), by the Minerva Foundation (with funding from the Federal
Ministry for Education and Research), and by the Yeda-Sela (YeS) Center for
Basic Research.

## References

  * (1) A. Tumasyan et al. [CMS], “A portrait of the Higgs boson by the CMS experiment ten years after the discovery.,” Nature 607, no.7917, 60-68 (2022) [arXiv:2207.00043 [hep-ex]].
  * (2) G. Aad et al. [ATLAS], “A detailed map of Higgs boson interactions by the ATLAS experiment ten years after the discovery,” Nature 607, no.7917, 52-59 (2022) [erratum: Nature 612, no.7941, E24 (2022)] [arXiv:2207.00092 [hep-ex]].
  * (3) Y. Nir, “The three jewels in the crown of the LHC,” CERN Cour. 60, 41-45 (2020) [arXiv:2010.13126 [hep-ph]].
  * (4) A. M. Sirunyan et al. [CMS], “Evidence for Higgs boson decay to a pair of muons,” JHEP 01, 148 (2021) [arXiv:2009.04363 [hep-ex]].
  * (5) G. Aad et al. [ATLAS], “A search for the dimuon decay of the Standard Model Higgs boson with the ATLAS detector,” Phys. Lett. B 812, 135980 (2021) [arXiv:2007.07830 [hep-ex]].
  * (6) R.L.Workman et al. (ParticleDataGroup), Prog. Theor. Exp. Phys. 2022, 083C01 (2022) and 2023 update.
  * (7) A. Tumasyan et al. [CMS], “Search for Higgs Boson Decay to a Charm Quark-Antiquark Pair in Proton-Proton Collisions at s=13 TeV,” Phys. Rev. Lett. 131, no.6, 061801 (2023) [arXiv:2205.05550 [hep-ex]].
  * (8) G. Aad et al. [ATLAS], “Direct constraint on the Higgs-charm coupling from a search for Higgs boson decays into charm quarks with the ATLAS detector,” Eur. Phys. J. C 82, 717 (2022) [arXiv:2201.11428 [hep-ex]].
  * (9) E. Fuchs, M. Losada, Y. Nir and Y. Viernik, “Implications of the Upper Bound on $\boldsymbol{h\to\mu^{+}\mu^{-}}$ on the Baryon Asymmetry of the Universe,” Phys. Rev. Lett. 124, no.18, 181801 (2020) [arXiv:1911.08495 [hep-ph]]; “$CP$ violation from $\tau$, $t$ and $b$ dimension-6 Yukawa couplings - interplay of baryogenesis, EDM and Higgs physics,” JHEP 05, 056 (2020) [arXiv:2003.00099 [hep-ph]].
  * (10) T. S. Roussy, L. Caldwell, T. Wright, W. B. Cairncross, Y. Shagam, K. B. Ng, N. Schlossberger, S. Y. Park, A. Wang and J. Ye, et al. “An improved bound on the electron’s electric dipole moment,” Science 381, no.6653, adg4084 (2023) [arXiv:2212.11841 [physics.atom-ph]].
  * (11) V. Andreev et al. [ACME], “Improved limit on the electric dipole moment of the electron,” Nature 562, no.7727, 355-360 (2018)
  * (12) K. Blum, R. T. D’Agnolo and J. Fan, “Vacuum stability bounds on Higgs coupling deviations in the absence of new bosons,” JHEP 03, 166 (2015) [arXiv:1502.01045 [hep-ph]].
  * (13) S. Schael et al. [ALEPH, DELPHI, L3, OPAL, SLD, LEP Electroweak Working Group, SLD Electroweak Group and SLD Heavy Flavour Group], “Precision electroweak measurements on the $Z$ resonance,” Phys. Rept. 427, 257-454 (2006) [arXiv:hep-ex/0509008 [hep-ex]].
  * (14) G. Aad et al. [ATLAS], “Measurements of Higgs boson production cross-sections in the $H\to\tau^{+}\tau^{-}$ decay channel in pp collisions at $\sqrt{s}$ = 13 TeV with the ATLAS detector,” JHEP 08, 175 (2022) [arXiv:2201.08269 [hep-ex]].
  * (15) D. Ghosh, R. S. Gupta and G. Perez, “Is the Higgs Mechanism of Fermion Mass Generation a Fact? A Yukawa-less First-Two-Generation Model,” Phys. Lett. B 755, 504-508 (2016) [arXiv:1508.01501 [hep-ph]].
  * (16) W. Altmannshofer and B. Maddock, “Flavorful Two Higgs Doublet Models with a Twist,” Phys. Rev. D 98, no.7, 075005 (2018) [arXiv:1805.08659 [hep-ph]].
  * (17) G. F. Giudice and O. Lebedev, “Higgs-dependent Yukawa couplings,” Phys. Lett. B 665, 79-85 (2008) [arXiv:0804.1753 [hep-ph]].
  * (18) F. J. Botella, G. C. Branco, M. N. Rebelo and J. I. Silva-Marcos, “What if the masses of the first two quark families are not generated by the standard model Higgs boson?,” Phys. Rev. D 94, no.11, 115031 (2016) [arXiv:1602.08011 [hep-ph]].
  * (19) M. Bauer, M. Carena and K. Gemmler, “Flavor from the Electroweak Scale,” JHEP 11, 016 (2015) [arXiv:1506.01719 [hep-ph]]. “Creating the fermion mass hierarchies with multiple Higgs bosons,” Phys. Rev. D 94, no.11, 115030 (2016) [arXiv:1512.03458 [hep-ph]].
  * (20) A. M. Sirunyan et al. [CMS], “Measurement of the Higgs boson production rate in association with top quarks in final states with electrons, muons, and hadronically decaying tau leptons at $\sqrt{s}=$ 13 TeV,” Eur. Phys. J. C 81, no.4, 378 (2021) [arXiv:2011.03652 [hep-ex]].
  * (21) M. Aaboud et al. [ATLAS], “Evidence for the associated production of the Higgs boson and a top quark pair with the ATLAS detector,” Phys. Rev. D 97, no.7, 072003 (2018) [arXiv:1712.08891 [hep-ex]].
  * (22) G. Perez, “The origin of particle masses," CERN Cour. 62, 53-54 (2022).